2021-02-10T11:10:38.884 INFO:root:teuthology version: 1.1.0-14fad317 2021-02-10T11:10:38.885 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722 --description fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} --name khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba --owner scheduled_khiremat@teuthology -- /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/orig.config.yaml 2021-02-10T11:10:38.951 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-10T11:10:39.025 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722 branch: wip-khiremat-mgr-authid-bp-7-nautilus description: fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes} email: khiremat@redhat.com first_in_suite: false job_id: '5872722' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: gibba name: khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '7.9' overrides: admin_socket: branch: wip-khiremat-mgr-authid-bp-7-nautilus ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug client: 10 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(TOO_FEW_PGS\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-khiremat-mgr-authid-bp-7-nautilus sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 owner: scheduled_khiremat@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.1 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - client.2 - client.3 sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 sleep_before_teardown: 0 suite: fs suite_branch: wip-khiremat-mgr-authid-bp-7-nautilus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 targets: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvPiPCP+84JBxO1XaMH52v/h34wPsG8Lu6bQuua9H7e3EelLeBaTEUCub8H6EB8Ae+G23LYCaVnoEMQIgwwMZ8HOBX7quy4z9QBLocC+z7F13cR9wWevteezakfPUuug/DGwsAenCMa56jOwzrjaZMRunOg2JP6AXZQpVQPXT0929+hMWjizbsWZ/nG8dDzG2hPAgo7xolAj6rONMu0NowOJ2KDsCww6+ulTaJPh6hKSBqkcWITxTukahY4otQaD4AGvBfVRL/uhjxgn1Zhh4UySjZenQxEGe9EeApTdywgbF5OsSSUl/KmkyagcGO4VqRHjKP1Sbh4S1xuULCs26p gibba025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDA+sG5q84njl7vgBldbJqZbclIjZSgoPnPKk9H18ICtpMTl01ruuJH0JSQaW0P2v/fjthbPeU1QFI9yHvF8jinrVDeS8/vtV4KGbm2vehcSRT4zW1A68Q7NQkNziUMWGyz877vICgChjGllUS8ZTFJDMtLu/1wtSD0dkr2/QCjtuFVmFCMUOPHmcayDjHM/PIyVPK+qJESKrxoWSVNsgoOJi4lKHE8+8m/+f03XzDCZlZJw0gKJV2pz8KLWcn7h8/6LKtg8fEtuHZtb1wSjKYYW92HKny1SOkrDyA2EaZpRDvZJLGeSx9KufNI9OW1uVhyLB10WPWGB+VZeu7eTV+T tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes teuthology_branch: master timestamp: 2021-02-10_10:13:45 tube: gibba user: khiremat verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.gibba.29737 2021-02-10T11:10:39.026 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa; will attempt to use it 2021-02-10T11:10:39.032 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks 2021-02-10T11:10:39.033 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-10T11:10:39.034 INFO:teuthology.task.internal:Checking packages... 2021-02-10T11:10:39.087 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '962fac13b1a781b8c7d25c1eed50d93813b8f5f0' 2021-02-10T11:10:39.087 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-10T11:10:39.088 INFO:teuthology.packaging:ref: None 2021-02-10T11:10:39.088 INFO:teuthology.packaging:tag: None 2021-02-10T11:10:39.088 INFO:teuthology.packaging:branch: wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T11:10:39.088 INFO:teuthology.packaging:sha1: 962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:10:39.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-khiremat-mgr-authid-bp-7-nautilus 2021-02-10T11:10:39.264 INFO:teuthology.task.internal:Found packages for ceph version 14.2.16-280.g962fac13 2021-02-10T11:10:39.264 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-10T11:10:39.298 INFO:teuthology.task.internal:no buildpackages task found 2021-02-10T11:10:39.298 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-10T11:10:39.311 INFO:teuthology.task.internal:Saving configuration 2021-02-10T11:10:39.326 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-10T11:10:39.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-10T11:10:39.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 11:05:27.317987', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsSgD3RqMkF0yughkSIF3eRJ/b+ZROPQfi2kf2sIDSBU55EoiIhTzim6cnxtoxrTYSSextClVjOdeP7f+NqMN0=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba017.front.sepia.ceph.com'} 2021-02-10T11:10:39.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-10 11:05:27.318008', 'description': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722', 'locked_by': 'scheduled_khiremat@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgQWXj7F10LMg5xDVwLZT9hg2mQyHs/CUBG/vIBHLESXm3N5WtDiF7+iaN/tdZXpnsWQj6Ep2THSP6p93hvVPM=', 'os_version': '7.9', 'machine_type': 'gibba', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'gibba025.front.sepia.ceph.com'} 2021-02-10T11:10:39.372 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-10T11:10:39.386 INFO:teuthology.task.internal:roles: ubuntu@gibba017.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'] 2021-02-10T11:10:39.386 INFO:teuthology.task.internal:roles: ubuntu@gibba025.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3'] 2021-02-10T11:10:39.386 INFO:teuthology.run_tasks:Running task console_log... 2021-02-10T11:10:39.450 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b36645620>, signals=[15]) 2021-02-10T11:10:39.450 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-10T11:10:39.464 INFO:teuthology.task.internal:Opening connections... 2021-02-10T11:10:39.464 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:10:39.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:10:39.531 DEBUG:teuthology.task.internal:connecting to ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:10:39.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:10:39.599 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-10T11:10:39.620 DEBUG:teuthology.orchestra.run.gibba017:> uname -m 2021-02-10T11:10:39.988 INFO:teuthology.orchestra.run.gibba017.stdout:x86_64 2021-02-10T11:10:39.988 DEBUG:teuthology.orchestra.run.gibba017:> cat /etc/os-release 2021-02-10T11:10:40.323 INFO:teuthology.orchestra.run.gibba017.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T11:10:40.323 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION="7.9 (Maipo)" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:ID="rhel" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:ID_LIKE="fedora" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:VARIANT="Server" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:VARIANT_ID="server" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_ID="7.9" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:ANSI_COLOR="0;31" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T11:10:40.324 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T11:10:40.325 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T11:10:40.325 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-02-10T11:10:40.340 DEBUG:teuthology.orchestra.run.gibba025:> uname -m 2021-02-10T11:10:40.688 INFO:teuthology.orchestra.run.gibba025.stdout:x86_64 2021-02-10T11:10:40.689 DEBUG:teuthology.orchestra.run.gibba025:> cat /etc/os-release 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:NAME="Red Hat Enterprise Linux Server" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION="7.9 (Maipo)" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:ID="rhel" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:ID_LIKE="fedora" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:VARIANT="Server" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:VARIANT_ID="server" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION_ID="7.9" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:PRETTY_NAME="Employee SKU" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:ANSI_COLOR="0;31" 2021-02-10T11:10:41.022 INFO:teuthology.orchestra.run.gibba025.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server" 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:HOME_URL="https://www.redhat.com/" 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.9 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-10T11:10:41.023 INFO:teuthology.orchestra.run.gibba025.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.9" 2021-02-10T11:10:41.023 INFO:teuthology.lock.ops:Updating gibba025.front.sepia.ceph.com on lock server 2021-02-10T11:10:41.039 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-10T11:10:41.073 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-10T11:10:41.083 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-10T11:10:41.084 DEBUG:teuthology.orchestra.run.gibba017:> test '!' -e /home/ubuntu/cephtest 2021-02-10T11:10:41.088 DEBUG:teuthology.orchestra.run.gibba025:> test '!' -e /home/ubuntu/cephtest 2021-02-10T11:10:41.421 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-10T11:10:41.453 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-10T11:10:41.453 DEBUG:teuthology.orchestra.run.gibba017:> test -z $(ls -A /var/lib/ceph) 2021-02-10T11:10:41.457 DEBUG:teuthology.orchestra.run.gibba025:> test -z $(ls -A /var/lib/ceph) 2021-02-10T11:10:41.819 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-10T11:10:41.893 INFO:teuthology.run_tasks:Running task kernel... 2021-02-10T11:10:41.962 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}, 'client.3': {'kdb': True, 'sha1': 'distro'}} 2021-02-10T11:10:41.963 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-10T11:10:41.963 DEBUG:teuthology.orchestra.run.gibba017:> uname -r 2021-02-10T11:10:42.299 INFO:teuthology.orchestra.run.gibba017.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:42.299 INFO:teuthology.task.kernel:Running kernel on gibba017: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:42.299 DEBUG:teuthology.orchestra.run.gibba017:> sudo yum install -y kernel 2021-02-10T11:10:43.533 INFO:teuthology.orchestra.run.gibba017.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:10:43.533 INFO:teuthology.orchestra.run.gibba017.stdout: : disabled-repos, subscription-manager 2021-02-10T11:10:43.684 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:10:43.685 INFO:teuthology.orchestra.run.gibba017.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:10:43.685 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:10:43.714 INFO:teuthology.orchestra.run.gibba017.stdout:Determining fastest mirrors 2021-02-10T11:10:43.804 INFO:teuthology.orchestra.run.gibba017.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:10:44.730 INFO:teuthology.orchestra.run.gibba017.stdout:Nothing to do 2021-02-10T11:10:44.743 DEBUG:teuthology.orchestra.run.gibba017:> echo no | sudo yum reinstall kernel || true 2021-02-10T11:10:45.235 INFO:teuthology.orchestra.run.gibba017.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:10:45.235 INFO:teuthology.orchestra.run.gibba017.stdout: : disabled-repos, subscription-manager 2021-02-10T11:10:45.290 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:10:45.291 INFO:teuthology.orchestra.run.gibba017.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:10:45.291 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:10:45.356 INFO:teuthology.orchestra.run.gibba017.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:45.376 INFO:teuthology.orchestra.run.gibba017.stdout:Loading mirror speeds from cached hostfile 2021-02-10T11:10:45.376 INFO:teuthology.orchestra.run.gibba017.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:10:45.379 INFO:teuthology.orchestra.run.gibba017.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T11:10:45.380 INFO:teuthology.orchestra.run.gibba017.stderr:Error: Nothing to do 2021-02-10T11:10:45.392 DEBUG:teuthology.orchestra.run.gibba017:> rpm -q kernel --last | head -n 1 2021-02-10T11:10:45.842 INFO:teuthology.orchestra.run.gibba017.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T11:10:45.843 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:45.843 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T11:10:45.843 DEBUG:teuthology.orchestra.run.gibba025:> uname -r 2021-02-10T11:10:46.177 INFO:teuthology.orchestra.run.gibba025.stdout:3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:46.178 INFO:teuthology.task.kernel:Running kernel on gibba025: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:46.178 DEBUG:teuthology.orchestra.run.gibba025:> sudo yum install -y kernel 2021-02-10T11:10:47.382 INFO:teuthology.orchestra.run.gibba025.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:10:47.382 INFO:teuthology.orchestra.run.gibba025.stdout: : disabled-repos, subscription-manager 2021-02-10T11:10:47.559 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:10:47.559 INFO:teuthology.orchestra.run.gibba025.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:10:47.559 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:10:47.593 INFO:teuthology.orchestra.run.gibba025.stdout:Determining fastest mirrors 2021-02-10T11:10:48.638 INFO:teuthology.orchestra.run.gibba025.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:10:49.718 INFO:teuthology.orchestra.run.gibba025.stdout:Nothing to do 2021-02-10T11:10:49.730 DEBUG:teuthology.orchestra.run.gibba025:> echo no | sudo yum reinstall kernel || true 2021-02-10T11:10:50.182 INFO:teuthology.orchestra.run.gibba025.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:10:50.183 INFO:teuthology.orchestra.run.gibba025.stdout: : disabled-repos, subscription-manager 2021-02-10T11:10:50.241 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:10:50.242 INFO:teuthology.orchestra.run.gibba025.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-02-10T11:10:50.242 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:10:50.298 INFO:teuthology.orchestra.run.gibba025.stderr:Skipping the running kernel: kernel-3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:50.309 INFO:teuthology.orchestra.run.gibba025.stdout:Loading mirror speeds from cached hostfile 2021-02-10T11:10:50.310 INFO:teuthology.orchestra.run.gibba025.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:10:50.313 INFO:teuthology.orchestra.run.gibba025.stdout:Installed package kernel-3.10.0-1160.el7.x86_64 (from anaconda) not available. 2021-02-10T11:10:50.313 INFO:teuthology.orchestra.run.gibba025.stderr:Error: Nothing to do 2021-02-10T11:10:50.324 DEBUG:teuthology.orchestra.run.gibba025:> rpm -q kernel --last | head -n 1 2021-02-10T11:10:50.778 INFO:teuthology.orchestra.run.gibba025.stdout:kernel-3.10.0-1160.6.1.el7.x86_64 Wed 11 Nov 2020 03:14:13 PM UTC 2021-02-10T11:10:50.779 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1160.6.1.el7.x86_64 2021-02-10T11:10:50.779 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-02-10T11:10:50.779 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-02-10T11:10:50.779 DEBUG:teuthology.orchestra.run.gibba017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T11:10:51.128 INFO:teuthology.orchestra.run.gibba017.stdout:ttyS1 2021-02-10T11:10:51.142 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-02-10T11:10:51.142 DEBUG:teuthology.orchestra.run.gibba025:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-10T11:10:51.489 INFO:teuthology.orchestra.run.gibba025.stdout:ttyS1 2021-02-10T11:10:51.499 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-10T11:10:51.509 INFO:teuthology.task.internal:Creating test directory... 2021-02-10T11:10:51.510 DEBUG:teuthology.orchestra.run.gibba017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T11:10:51.513 DEBUG:teuthology.orchestra.run.gibba025:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-10T11:10:51.847 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-10T11:10:51.854 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-10T11:10:51.867 INFO:teuthology.task.internal:Creating archive directory... 2021-02-10T11:10:51.867 DEBUG:teuthology.orchestra.run.gibba017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T11:10:51.871 DEBUG:teuthology.orchestra.run.gibba025:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-10T11:10:52.215 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-10T11:10:52.223 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-10T11:10:52.223 DEBUG:teuthology.orchestra.run.gibba017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T11:10:52.227 DEBUG:teuthology.orchestra.run.gibba025:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-10T11:10:52.570 INFO:teuthology.orchestra.run.gibba025.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:10:52.571 INFO:teuthology.orchestra.run.gibba017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:10:52.580 INFO:teuthology.orchestra.run.gibba025.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:10:52.580 INFO:teuthology.orchestra.run.gibba017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-10T11:10:52.581 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-10T11:10:52.589 INFO:teuthology.task.internal:Configuring sudo... 2021-02-10T11:10:52.589 DEBUG:teuthology.orchestra.run.gibba017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T11:10:52.592 DEBUG:teuthology.orchestra.run.gibba025:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-10T11:10:52.938 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-10T11:10:52.949 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-10T11:10:52.949 DEBUG:teuthology.orchestra.run.gibba017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T11:10:52.952 DEBUG:teuthology.orchestra.run.gibba025:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-10T11:10:53.287 DEBUG:teuthology.orchestra.run.gibba017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:10:53.641 DEBUG:teuthology.orchestra.run.gibba017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:10:53.985 DEBUG:teuthology.orchestra.run.gibba017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:10:54.336 DEBUG:teuthology.orchestra.run.gibba017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:10:54.678 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:10:54.678 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T11:10:55.022 DEBUG:teuthology.orchestra.run.gibba025:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:10:55.375 DEBUG:teuthology.orchestra.run.gibba025:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-10T11:10:55.720 DEBUG:teuthology.orchestra.run.gibba025:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:10:56.070 DEBUG:teuthology.orchestra.run.gibba025:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-10T11:10:56.413 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:10:56.413 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-10T11:10:56.755 DEBUG:teuthology.orchestra.run.gibba017:> sudo service rsyslog restart 2021-02-10T11:10:56.760 DEBUG:teuthology.orchestra.run.gibba025:> sudo service rsyslog restart 2021-02-10T11:10:57.112 INFO:teuthology.orchestra.run.gibba017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:10:57.118 INFO:teuthology.orchestra.run.gibba025.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T11:10:57.135 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-10T11:10:57.152 INFO:teuthology.task.internal:Starting timer... 2021-02-10T11:10:57.152 INFO:teuthology.run_tasks:Running task pcp... 2021-02-10T11:10:57.167 INFO:teuthology.run_tasks:Running task selinux... 2021-02-10T11:10:57.182 DEBUG:teuthology.orchestra.run.gibba017:> sudo service auditd rotate 2021-02-10T11:10:57.550 INFO:teuthology.orchestra.run.gibba017.stdout:Rotating logs: [ OK ] 2021-02-10T11:10:57.551 DEBUG:teuthology.orchestra.run.gibba025:> sudo service auditd rotate 2021-02-10T11:10:57.926 INFO:teuthology.orchestra.run.gibba025.stdout:Rotating logs: [ OK ] 2021-02-10T11:10:57.928 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-10T11:10:57.928 DEBUG:teuthology.orchestra.run.gibba017:> /usr/sbin/getenforce 2021-02-10T11:10:58.267 INFO:teuthology.orchestra.run.gibba017.stdout:Permissive 2021-02-10T11:10:58.268 DEBUG:teuthology.orchestra.run.gibba025:> /usr/sbin/getenforce 2021-02-10T11:10:58.607 INFO:teuthology.orchestra.run.gibba025.stdout:Permissive 2021-02-10T11:10:58.608 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@gibba017.front.sepia.ceph.com': 'permissive', 'ubuntu@gibba025.front.sepia.ceph.com': 'permissive'} 2021-02-10T11:10:58.608 DEBUG:teuthology.orchestra.run.gibba017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:10:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:10:58.954 DEBUG:teuthology.orchestra.run.gibba025:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T11:10:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:10:59.298 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-10T11:10:59.299 DEBUG:teuthology.orchestra.run.gibba017:> sudo /usr/sbin/setenforce permissive 2021-02-10T11:10:59.643 DEBUG:teuthology.orchestra.run.gibba025:> sudo /usr/sbin/setenforce permissive 2021-02-10T11:10:59.998 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-10T11:11:00.009 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-10T11:11:00.009 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-10T11:11:00.024 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-10T11:11:00.027 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit gibba017.front.sepia.ceph.com,gibba025.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-10T11:11:00.356 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-10T11:11:01.231 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-10T11:11:03.824 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-10T11:11:03.843 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:03 +0000 (0:00:00.070) 0:00:00.070 **** 2021-02-10T11:11:03.858 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:03 +0000 (0:00:00.014) 0:00:00.085 **** 2021-02-10T11:11:06.486 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba025.front.sepia.ceph.com] 2021-02-10T11:11:06.554 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:06 +0000 (0:00:02.695) 0:00:02.780 **** 2021-02-10T11:11:07.122 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-10T11:11:10.637 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} Wednesday 10 February 2021 11:11:07 +0000 (0:00:00.600) 0:00:03.381 **** TASK [Gathering Facts] ********************************************************* ok: [gibba017.front.sepia.ceph.com] Wednesday 10 February 2021 11:11:07 +0000 (0:00:00.302) 0:00:03.684 **** TASK [ansible-managed : Create the ansible user.] ****************************** ok: [gibba025.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Wednesday 10 February 2021 11:11:07 +0000 (0:00:00.404) 0:00:04.088 **** TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:11:07 +0000 (0:00:00.057) 0:00:04.145 **** TASK [ansible-managed : Create the sudo group.] ******************************** ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} Wednesday 10 February 2021 11:11:07 +0000 (0:00:00.040) 0:00:04.185 **** TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [ansible-managed : Create the ansible user.] ****************************** ok: [gibba017.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Wednesday 10 February 2021 11:11:08 +0000 (0:00:00.597) 0:00:04.783 **** Wednesday 10 February 2021 11:11:08 +0000 (0:00:00.032 2021-02-10T11:11:10.638 INFO:teuthology.task.ansible.out:) 0:00:04.815 **** TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:11:08 +0000 (0:00:00.045) 0:00:04.860 **** TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Wednesday 10 February 2021 11:11:09 +0000 (0:00:00.477) 0:00:05.338 **** 2021-02-10T11:11:10.882 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-10T11:11:10.883 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-02-10T11:11:10.917 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:10 +0000 (0:00:01.805) 0:00:07.143 **** 2021-02-10T11:11:11.102 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "state": "file", "uid": 0} 2021-02-10T11:11:11.137 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:11 +0000 (0:00:00.220) 0:00:07.363 **** 2021-02-10T11:11:11.553 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-10T11:11:11.554 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4 2021-02-10T11:11:11.556 INFO:teuthology.task.ansible.out:r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-10T11:11:11.627 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4 2021-02-10T11:11:11.628 INFO:teuthology.task.ansible.out:r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-10T11:11:11.722 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2021-02-10T11:11:11.722 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-02-10T11:11:11.731 INFO:teuthology.task.ansible.out: 2021-02-10T11:11:11.733 INFO:teuthology.task.ansible.out:PLAY [testnodes] *************************************************************** 2021-02-10T11:11:11.909 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:11 +0000 (0:00:00.772) 0:00:08.136 **** 2021-02-10T11:11:11.941 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:11 +0000 (0:00:00.031) 0:00:08.167 **** 2021-02-10T11:11:13.021 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba025.front.sepia.ceph.com] 2021-02-10T11:11:13.099 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-10T11:11:13.100 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:13 +0000 (0:00:01.158) 0:00:09.326 **** 2021-02-10T11:11:13.564 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-10T11:11:13.566 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.145912", "end": "2021-02-10 11:11:13.537740", "rc": 0, "start": "2021-02-10 11:11:13.391828", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T11:11:13.599 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:13 +0000 (0:00:00.499) 0:00:09.826 **** 2021-02-10T11:11:13.992 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba017.front.sepia.ceph.com] 2021-02-10T11:11:14.072 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.473) 0:00:10.299 **** 2021-02-10T11:11:14.080 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-10T11:11:14.080 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.006744", "end": "2021-02-10 11:11:14.027749", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:11:14.021005", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T11:11:14.116 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.043) 0:00:10.342 **** 2021-02-10T11:11:14.143 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-10T11:11:14.144 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:14.176 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.060) 0:00:10.403 **** 2021-02-10T11:11:14.203 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-10T11:11:14.204 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:14.237 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.061) 0:00:10.464 **** 2021-02-10T11:11:14.273 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-10T11:11:14.275 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:14.309 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.071) 0:00:10.536 **** 2021-02-10T11:11:14.335 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-10T11:11:14.337 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:14.368 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.059) 0:00:10.595 **** 2021-02-10T11:11:14.396 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-10T11:11:14.397 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:14.429 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.061) 0:00:10.656 **** 2021-02-10T11:11:14.436 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-10T11:11:14.436 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.141767", "end": "2021-02-10 11:11:14.397060", "rc": 0, "start": "2021-02-10 11:11:14.255293", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-10T11:11:14.470 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.040) 0:00:10.697 **** 2021-02-10T11:11:14.871 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-10T11:11:14.872 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:11:14.905 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.434) 0:00:11.132 **** 2021-02-10T11:11:14.921 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-10T11:11:14.923 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011026", "end": "2021-02-10 11:11:14.903452", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:11:14.892426", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-10T11:11:14.956 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:14 +0000 (0:00:00.050) 0:00:11.183 **** 2021-02-10T11:11:14.982 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-10T11:11:14.983 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:15.015 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.058) 0:00:11.242 **** 2021-02-10T11:11:15.041 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-10T11:11:15.042 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:15.074 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.058) 0:00:11.301 **** 2021-02-10T11:11:15.100 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-10T11:11:15.102 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:15.134 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.060) 0:00:11.361 **** 2021-02-10T11:11:15.161 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:15.193 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.059) 0:00:11.420 **** 2021-02-10T11:11:15.219 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-10T11:11:15.221 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:11:15.252 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.058) 0:00:11.478 **** 2021-02-10T11:11:15.382 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T11:11:15.383 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T11:11:15.694 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-10T11:11:15.695 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:11:15.727 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:11:15 +0000 (0:00:00.475) 0:00:11.954 **** 2021-02-10T11:11:15.816 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T11:11:15.817 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T11:11:16.204 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-10T11:11:16.256 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T11:11:16.640 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-10T11:11:16.693 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T11:11:17.078 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-10T11:11:17.131 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T11:11:17.515 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-10T11:11:17.569 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T11:11:17.952 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-10T11:11:18.005 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T11:11:18.390 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-10T11:11:18.444 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T11:11:18.829 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-10T11:11:18.882 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T11:11:19.269 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-10T11:11:19.322 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T11:11:19.707 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-10T11:11:19.749 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T11:11:20.136 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-10T11:11:20.138 INFO:teuthology.task.ansible.out: 2021-02-10T11:11:20.189 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T11:11:20.586 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-10T11:11:20.639 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T11:11:21.017 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-10T11:11:21.081 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T11:11:21.455 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-10T11:11:21.519 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T11:11:21.896 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-10T11:11:21.959 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T11:11:22.340 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-10T11:11:22.341 INFO:teuthology.task.ansible.out: 2021-02-10T11:11:22.393 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T11:11:22.782 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-10T11:11:22.836 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T11:11:23.211 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-10T11:11:23.275 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T11:11:23.654 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-10T11:11:23.707 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T11:11:24.092 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-10T11:11:24.146 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T11:11:24.520 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-10T11:11:24.583 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T11:11:24.969 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-10T11:11:25.022 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T11:11:25.407 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-10T11:11:25.460 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T11:11:25.845 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-10T11:11:25.847 INFO:teuthology.task.ansible.out: 2021-02-10T11:11:25.898 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T11:11:26.282 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-10T11:11:26.335 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T11:11:26.719 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-10T11:11:26.784 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T11:11:27.158 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-10T11:11:27.222 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T11:11:27.610 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-10T11:11:27.914 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T11:11:28.050 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-10T11:11:28.103 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T11:11:28.490 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-10T11:11:28.544 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T11:11:28.918 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-10T11:11:28.981 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T11:11:29.354 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-10T11:11:29.418 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T11:11:29.795 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-10T11:11:29.848 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T11:11:30.236 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-10T11:11:30.289 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T11:11:30.674 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-10T11:11:30.727 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T11:11:31.112 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-10T11:11:31.165 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T11:11:31.569 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-10T11:11:31.620 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T11:11:32.016 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-10T11:11:32.079 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:11:32.465 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-10T11:11:32.549 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T11:11:32.933 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:11:33.008 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T11:11:33.383 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-10T11:11:33.463 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T11:11:33.833 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-10T11:11:33.907 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T11:11:34.281 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-10T11:11:34.344 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T11:11:34.715 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-10T11:11:34.778 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T11:11:35.150 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-10T11:11:35.224 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T11:11:35.589 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-10T11:11:35.653 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:11:36.028 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-10T11:11:36.092 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T11:11:36.456 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:11:36.530 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T11:11:36.896 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-10T11:11:36.970 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T11:11:37.343 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-10T11:11:37.417 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T11:11:37.780 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-10T11:11:37.854 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T11:11:38.221 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-10T11:11:38.295 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T11:11:38.661 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-10T11:11:38.735 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T11:11:39.096 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-10T11:11:39.170 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T11:11:39.534 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-10T11:11:39.597 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T11:11:39.971 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-10T11:11:40.035 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T11:11:40.416 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-10T11:11:40.479 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T11:11:40.864 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-10T11:11:40.916 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T11:11:41.296 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-10T11:11:41.361 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T11:11:41.736 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-10T11:11:41.790 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T11:11:42.176 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-10T11:11:42.229 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T11:11:42.616 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-10T11:11:42.679 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T11:11:43.057 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-10T11:11:43.110 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:11:43.496 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-10T11:11:43.550 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T11:11:43.929 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-10T11:11:43.993 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T11:11:44.375 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-10T11:11:44.429 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T11:11:44.809 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-10T11:11:44.863 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T11:11:45.253 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-10T11:11:45.307 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T11:11:45.686 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-10T11:11:45.751 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T11:11:46.131 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-10T11:11:46.184 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T11:11:46.567 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-10T11:11:46.631 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T11:11:47.002 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-10T11:11:47.076 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T11:11:47.478 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-10T11:11:47.530 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T11:11:47.920 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-10T11:11:47.973 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T11:11:48.363 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-10T11:11:48.417 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T11:11:48.796 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-10T11:11:48.860 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T11:11:49.236 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-10T11:11:49.649 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T11:11:49.675 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-10T11:11:49.741 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T11:11:50.122 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-10T11:11:50.188 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T11:11:50.577 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-10T11:11:50.620 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T11:11:51.013 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-10T11:11:51.056 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T11:11:51.460 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-10T11:11:51.503 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T11:11:51.893 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-10T11:11:51.936 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T11:11:52.333 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-10T11:11:52.375 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T11:11:52.771 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-10T11:11:52.814 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T11:11:53.223 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-10T11:11:53.255 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T11:11:53.665 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-10T11:11:53.708 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T11:11:54.133 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-10T11:11:54.155 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T11:11:54.576 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-10T11:11:54.609 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T11:11:55.020 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-10T11:11:55.053 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T11:11:55.461 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-10T11:11:55.493 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T11:11:55.900 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-10T11:11:55.933 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T11:11:56.337 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-10T11:11:56.380 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T11:11:56.775 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-10T11:11:56.817 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T11:11:57.203 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-10T11:11:57.257 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T11:11:57.643 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-10T11:11:57.706 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T11:11:58.083 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-10T11:11:58.146 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T11:11:58.521 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-10T11:11:58.586 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T11:11:58.964 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-10T11:11:59.017 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T11:11:59.401 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-10T11:11:59.465 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T11:11:59.842 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-10T11:11:59.894 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:12:00.281 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-10T11:12:00.333 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T11:12:00.719 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-10T11:12:00.772 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T11:12:01.155 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-10T11:12:01.207 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T11:12:01.594 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-10T11:12:01.648 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T11:12:02.027 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-10T11:12:02.091 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T11:12:02.467 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-10T11:12:02.531 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T11:12:02.909 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-10T11:12:02.962 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T11:12:03.338 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-10T11:12:03.339 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:03.401 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T11:12:03.779 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-10T11:12:03.843 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T11:12:04.217 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-10T11:12:04.281 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T11:12:04.659 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-10T11:12:04.712 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T11:12:05.099 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-10T11:12:05.152 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T11:12:05.540 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-10T11:12:05.594 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T11:12:05.970 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-10T11:12:06.044 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T11:12:06.411 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-10T11:12:06.486 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T11:12:06.839 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-10T11:12:06.923 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T11:12:07.277 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-10T11:12:07.371 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T11:12:07.727 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-10T11:12:07.801 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T11:12:08.158 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-10T11:12:08.242 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-10T11:12:08.608 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-10T11:12:10.273 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxT 2021-02-10T11:12:10.274 INFO:teuthology.task.ansible.out:iugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6ab 2021-02-10T11:12:10.276 INFO:teuthology.task.ansible.out:c4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T11:12:10.373 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-10T11:12:10.459 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T11:12:10.808 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-10T11:12:10.903 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T11:12:11.251 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-10T11:12:11.337 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T11:12:11.683 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-10T11:12:11.799 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T11:12:12.136 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-10T11:12:12.232 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T11:12:12.571 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-10T11:12:12.676 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T11:12:13.010 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-10T11:12:13.115 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T11:12:13.451 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-10T11:12:13.557 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T11:12:13.915 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-10T11:12:14.053 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T11:12:14.399 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-10T11:12:14.515 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T11:12:14.842 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-10T11:12:15.303 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-10T11:12:15.357 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T11:12:16.210 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T11:12:16.232 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-10T11:12:16.936 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T11:12:16.978 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-10T11:12:17.739 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T11:12:17.829 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-10T11:12:17.866 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:17 +0000 (0:01:02.138) 0:01:14.092 **** 2021-02-10T11:12:17.896 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-10T11:12:17.929 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:17 +0000 (0:00:00.063) 0:01:14.156 **** 2021-02-10T11:12:17.978 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-10T11:12:17.980 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T11:12:17.981 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T11:12:17.983 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T11:12:17.984 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T11:12:17.985 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T11:12:17.987 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T11:12:17.988 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5 2021-02-10T11:12:17.988 INFO:teuthology.task.ansible.out:d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T11:12:17.989 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T11:12:17.991 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T11:12:17.992 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T11:12:17.993 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": " 2021-02-10T11:12:17.993 INFO:teuthology.task.ansible.out:mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T11:12:17.995 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NIN 2021-02-10T11:12:17.996 INFO:teuthology.task.ansible.out:j0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T11:12:18.036 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:18 +0000 (0:00:00.107) 0:01:14.263 **** 2021-02-10T11:12:18.482 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-10T11:12:18.484 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-10T11:12:18.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:18 +0000 (0:00:00.585) 0:01:14.849 **** 2021-02-10T11:12:18.650 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-10T11:12:18.683 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:18 +0000 (0:00:00.060) 0:01:14.909 **** 2021-02-10T11:12:18.732 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-10T11:12:18.734 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora r 2021-02-10T11:12:18.735 INFO:teuthology.task.ansible.out:T6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c 2021-02-10T11:12:18.737 INFO:teuthology.task.ansible.out:63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich 2021-02-10T11:12:18.738 INFO:teuthology.task.ansible.out:", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiR 2021-02-10T11:12:18.740 INFO:teuthology.task.ansible.out:enVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdi 2021-02-10T11:12:18.740 INFO:teuthology.task.ansible.out:llaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumr 2021-02-10T11:12:18.741 INFO:teuthology.task.ansible.out:ao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69m 2021-02-10T11:12:18.742 INFO:teuthology.task.ansible.out:h95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d84 2021-02-10T11:12:18.744 INFO:teuthology.task.ansible.out:1a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan 2021-02-10T11:12:18.745 INFO:teuthology.task.ansible.out:@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOen 2021-02-10T11:12:18.746 INFO:teuthology.task.ansible.out:VV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9g 2021-02-10T11:12:18.747 INFO:teuthology.task.ansible.out:I46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": " 2021-02-10T11:12:18.749 INFO:teuthology.task.ansible.out:dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NIN 2021-02-10T11:12:18.750 INFO:teuthology.task.ansible.out:j0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-10T11:12:18.787 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:18 +0000 (0:00:00.103) 0:01:15.013 **** 2021-02-10T11:12:18.933 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-10T11:12:18.934 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T11:12:18.968 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:18 +0000 (0:00:00.181) 0:01:15.195 **** 2021-02-10T11:12:19.382 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-10T11:12:19.415 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:12:19 +0000 (0:00:00.447) 0:01:15.642 **** 2021-02-10T11:12:19.464 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-10T11:12:19.465 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T11:12:19.912 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-10T11:12:19.914 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T11:12:20.358 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-10T11:12:20.359 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T11:12:20.795 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T11:12:20.796 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T11:12:20.806 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-10T11:12:21.237 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T11:12:21.249 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZt 2021-02-10T11:12:21.250 INFO:teuthology.task.ansible.out:IEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-10T11:12:21.689 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T11:12:21.691 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmn 2021-02-10T11:12:21.692 INFO:teuthology.task.ansible.out:wkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-10T11:12:22.130 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T11:12:22.142 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o 2021-02-10T11:12:22.143 INFO:teuthology.task.ansible.out:8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-10T11:12:22.568 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T11:12:22.579 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-10T11:12:23.007 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T11:12:23.019 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:23.040 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-10T11:12:23.468 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorb 2021-02-10T11:12:23.469 INFO:teuthology.task.ansible.out:nuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T11:12:23.511 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-10T11:12:23.511 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:23.911 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemers 2021-02-10T11:12:23.912 INFO:teuthology.task.ansible.out:on", "validate_certs": true} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:23.922 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:23.965 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMorb 2021-02-10T11:12:23.966 INFO:teuthology.task.ansible.out:nuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-10T11:12:24.363 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T11:12:24.406 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemers 2021-02-10T11:12:24.407 INFO:teuthology.task.ansible.out:on", "validate_certs": true} 2021-02-10T11:12:24.417 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:24.428 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:12:24.814 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSo 2021-02-10T11:12:24.815 INFO:teuthology.task.ansible.out:AWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T11:12:24.867 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-10T11:12:25.252 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T11:12:25.304 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtSo 2021-02-10T11:12:25.306 INFO:teuthology.task.ansible.out:AWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-10T11:12:25.690 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T11:12:25.753 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-10T11:12:25.755 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:26.127 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T11:12:26.192 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-10T11:12:26.577 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T11:12:26.636 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-10T11:12:27.010 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T11:12:27.072 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-10T11:12:27.456 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9C 2021-02-10T11:12:27.457 INFO:teuthology.task.ansible.out:kxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T11:12:27.508 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-10T11:12:27.904 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T11:12:27.906 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T11:12:27.958 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9C 2021-02-10T11:12:27.959 INFO:teuthology.task.ansible.out:kxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-10T11:12:28.359 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T11:12:28.402 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBYB 2021-02-10T11:12:28.404 INFO:teuthology.task.ansible.out:0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-10T11:12:28.801 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T11:12:28.853 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-10T11:12:29.240 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T11:12:29.241 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T11:12:29.292 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-10T11:12:29.678 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T11:12:29.731 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46T 2021-02-10T11:12:29.733 INFO:teuthology.task.ansible.out:RJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-10T11:12:30.125 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T11:12:30.178 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-10T11:12:30.562 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T11:12:30.616 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-10T11:12:31.003 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T11:12:31.056 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-10T11:12:31.450 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T11:12:31.493 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-10T11:12:31.896 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T11:12:31.898 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T11:12:31.938 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-10T11:12:32.332 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T11:12:32.385 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQvU 2021-02-10T11:12:32.387 INFO:teuthology.task.ansible.out:xUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-10T11:12:32.786 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T11:12:32.866 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-10T11:12:33.240 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T11:12:33.241 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:33.273 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-10T11:12:33.275 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:33.682 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T11:12:33.725 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-10T11:12:34.137 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T11:12:34.159 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-10T11:12:34.577 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T11:12:34.599 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-10T11:12:35.020 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:12:35.041 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-10T11:12:35.460 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T11:12:35.483 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:12:35.906 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T11:12:35.928 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-10T11:12:36.361 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T11:12:36.362 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T11:12:36.382 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-10T11:12:36.831 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz 2021-02-10T11:12:36.833 INFO:teuthology.task.ansible.out:2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T11:12:36.844 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1O 2021-02-10T11:12:36.846 INFO:teuthology.task.ansible.out:MJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-10T11:12:37.305 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T11:12:37.306 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyKz 2021-02-10T11:12:37.308 INFO:teuthology.task.ansible.out:2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-10T11:12:37.756 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T11:12:37.756 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-10T11:12:38.208 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T11:12:38.209 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-10T11:12:38.654 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-10T11:12:38.655 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nul 2021-02-10T11:12:38.656 INFO:teuthology.task.ansible.out:l, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:12:39.107 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nul 2021-02-10T11:12:39.108 INFO:teuthology.task.ansible.out:l, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKcz 2021-02-10T11:12:39.110 INFO:teuthology.task.ansible.out:X+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeA 2021-02-10T11:12:39.111 INFO:teuthology.task.ansible.out:rkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T11:12:39.547 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T11:12:39.548 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjEf 2021-02-10T11:12:39.549 INFO:teuthology.task.ansible.out:dVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+dT 2021-02-10T11:12:39.551 INFO:teuthology.task.ansible.out:zNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-10T11:12:39.983 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T11:12:39.995 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-10T11:12:40.433 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-10T11:12:40.444 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T11:12:40.871 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-10T11:12:40.882 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T11:12:41.320 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-10T11:12:41.320 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T11:12:41.770 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/ 2021-02-10T11:12:41.772 INFO:teuthology.task.ansible.out:nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-10T11:12:42.213 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T11:12:42.214 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home/ 2021-02-10T11:12:42.215 INFO:teuthology.task.ansible.out:nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-10T11:12:42.654 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T11:12:42.666 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-10T11:12:42.667 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:43.093 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T11:12:43.095 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T11:12:43.105 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-10T11:12:43.532 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T11:12:43.553 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "g 2021-02-10T11:12:43.555 INFO:teuthology.task.ansible.out:abrioux", "validate_certs": true} 2021-02-10T11:12:43.987 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T11:12:43.999 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-10T11:12:44.426 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T11:12:44.428 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T11:12:44.460 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-10T11:12:44.869 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T11:12:44.901 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kkk 2021-02-10T11:12:44.903 INFO:teuthology.task.ansible.out:e6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-10T11:12:45.324 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T11:12:45.346 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-10T11:12:45.778 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T11:12:45.778 INFO:teuthology.task.ansible.out: 2021-02-10T11:12:45.789 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-10T11:12:46.227 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:12:46.238 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-10T11:12:46.666 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T11:12:46.667 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-10T11:12:47.106 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T11:12:47.118 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-10T11:12:47.554 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T11:12:47.555 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-10T11:12:48.000 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T11:12:48.011 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-10T11:12:48.448 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T11:12:48.449 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-10T11:12:48.882 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T11:12:48.894 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-10T11:12:49.322 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T11:12:49.345 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-10T11:12:49.764 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T11:12:49.786 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-10T11:12:50.204 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T11:12:50.236 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-10T11:12:50.645 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T11:12:50.677 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-10T11:12:51.098 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T11:12:51.120 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-10T11:12:51.539 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T11:12:51.561 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-10T11:12:51.995 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T11:12:52.006 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-10T11:12:52.445 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-10T11:12:52.457 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T11:12:52.895 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-10T11:12:52.896 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T11:12:53.334 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-10T11:12:53.335 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T11:12:53.776 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-10T11:12:53.777 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T11:12:53.778 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T11:12:54.215 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T11:12:54.226 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1l 2021-02-10T11:12:54.228 INFO:teuthology.task.ansible.out:XTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-10T11:12:54.663 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T11:12:54.664 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-10T11:12:55.097 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T11:12:55.109 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-10T11:12:55.541 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T11:12:55.552 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-10T11:12:55.989 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdo 2021-02-10T11:12:56.406 INFO:teuthology.task.ansible.out:main", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-10T11:12:56.443 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2 2021-02-10T11:12:56.444 INFO:teuthology.task.ansible.out:z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1f 2021-02-10T11:12:56.446 INFO:teuthology.task.ansible.out:XA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-02-10T11:12:56.886 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2 2021-02-10T11:12:56.887 INFO:teuthology.task.ansible.out:z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CR 2021-02-10T11:12:56.889 INFO:teuthology.task.ansible.out:BT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T11:12:57.327 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-10T11:12:57.328 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T11:12:57.799 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-10T11:12:57.821 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T11:12:58.260 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-10T11:12:58.294 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T11:12:58.722 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-10T11:12:58.742 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T11:12:59.180 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-10T11:12:59.202 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T11:12:59.627 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-10T11:12:59.649 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T11:13:00.069 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-10T11:13:00.102 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loca 2021-02-10T11:13:00.103 INFO:teuthology.task.ansible.out:l", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T11:13:00.519 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loca 2021-02-10T11:13:00.521 INFO:teuthology.task.ansible.out:l", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-10T11:13:00.541 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T11:13:00.955 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-10T11:13:00.987 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T11:13:01.399 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-10T11:13:01.422 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T11:13:01.832 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-10T11:13:01.875 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T11:13:02.277 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-10T11:13:02.309 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T11:13:02.720 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-10T11:13:02.763 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T11:13:03.159 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-10T11:13:03.202 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T11:13:03.203 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:13:03.610 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", " 2021-02-10T11:13:03.611 INFO:teuthology.task.ansible.out:manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-10T11:13:03.642 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T11:13:04.047 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-10T11:13:04.090 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T11:13:04.495 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-10T11:13:04.527 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T11:13:04.935 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-10T11:13:04.967 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T11:13:05.381 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-10T11:13:05.413 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T11:13:05.818 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-10T11:13:05.850 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T11:13:06.258 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-10T11:13:06.290 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T11:13:06.696 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-10T11:13:06.729 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T11:13:07.137 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-10T11:13:07.181 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T11:13:07.592 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-10T11:13:07.625 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T11:13:08.037 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-10T11:13:08.069 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T11:13:08.478 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-10T11:13:08.479 INFO:teuthology.task.ansible.out: 2021-02-10T11:13:08.510 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T11:13:08.923 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-10T11:13:08.957 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T11:13:09.375 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-10T11:13:09.397 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T11:13:09.819 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-10T11:13:09.852 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T11:13:10.263 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-10T11:13:10.296 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T11:13:10.706 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-10T11:13:10.763 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T11:13:11.152 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-10T11:13:11.215 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T11:13:11.602 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-10T11:13:11.665 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T11:13:11.666 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T11:13:12.040 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd 2021-02-10T11:13:12.042 INFO:teuthology.task.ansible.out:+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-10T11:13:12.105 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T11:13:12.489 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-10T11:13:12.552 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T11:13:12.927 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-10T11:13:12.990 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T11:13:13.367 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-10T11:13:13.440 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T11:13:13.815 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-10T11:13:13.888 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T11:13:14.257 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-10T11:13:14.331 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T11:13:14.705 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-10T11:13:14.778 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T11:13:15.143 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-10T11:13:15.218 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T11:13:15.597 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-10T11:13:15.661 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T11:13:16.040 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-10T11:13:16.104 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T11:13:16.474 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-10T11:13:16.553 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T11:13:16.919 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-10T11:13:17.006 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T11:13:17.366 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-10T11:13:17.453 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T11:13:17.819 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-10T11:13:17.893 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T11:13:18.263 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-10T11:13:18.337 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T11:13:18.708 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-10T11:13:18.793 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T11:13:19.169 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-10T11:13:19.265 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T11:13:19.620 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-10T11:13:19.714 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T11:13:19.805 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:19 +0000 (0:01:00.389) 0:02:16.031 **** 2021-02-10T11:13:19.853 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.854 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.864 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.875 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.876 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.887 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.898 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.898 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:19.909 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.067 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-10T11:13:20.068 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-10T11:13:20.145 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:20 +0000 (0:00:00.340) 0:02:16.372 **** 2021-02-10T11:13:20.183 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T11:13:20.185 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.195 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.205 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.206 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.217 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.228 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.228 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.239 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.250 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.345 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T11:13:20.355 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.366 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.689 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-10T11:13:20.701 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.701 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:20.806 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T11:13:21.140 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-10T11:13:21.246 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T11:13:21.257 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.258 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.269 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.280 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.280 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.291 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.303 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.303 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.315 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.326 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.327 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.338 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.350 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.361 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.372 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.374 INFO:teuthology.task.ansible.out: skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.384 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.396 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.396 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.408 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.408 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.419 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.430 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.431 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.442 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.453 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.464 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.466 INFO:teuthology.task.ansible.out: skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.476 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.488 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.489 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.499 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.510 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.511 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.522 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.533 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.544 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.556 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.556 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.567 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.578 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.579 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-10T11:13:21.590 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.591 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.603 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.603 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.605 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.616 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.618 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.630 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.631 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.632 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.634 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.645 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.646 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.647 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.658 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.658 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.670 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.671 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.672 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.673 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.683 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.684 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.695 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.697 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.707 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.708 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.709 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.719 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.720 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.731 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.732 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.743 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.744 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.745 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.756 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.756 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.757 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.767 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.768 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.769 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.780 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.780 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.782 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.792 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.793 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.804 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.804 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.805 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.816 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.816 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.827 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.828 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.829 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.840 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.840 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.851 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.851 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.853 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.863 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.863 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.865 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.875 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.887 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.899 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.900 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.901 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.902 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.913 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.914 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.925 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.926 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.936 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.938 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.949 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.950 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.951 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.962 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.962 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.974 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.976 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.987 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.988 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.999 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:21.999 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.001 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.012 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.013 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.014 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.024 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.025 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.026 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.036 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.037 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.048 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.048 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.049 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.060 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.061 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.062 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.072 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.073 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.084 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.084 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.085 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.096 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.096 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.098 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.108 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.108 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.119 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.120 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.122 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.122 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.123 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.134 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.135 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.136 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.147 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.148 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.149 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.151 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.161 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.161 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.163 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.173 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.173 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.174 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.184 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.185 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.186 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.196 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.197 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.198 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.209 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.210 INFO:teuthology.task.ansible.out: skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.220 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.221 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.232 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.233 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.234 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.245 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.246 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.258 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.259 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.260 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.271 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.271 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.291 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.325 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:22 +0000 (0:00:02.179) 0:02:18.552 **** 2021-02-10T11:13:22.332 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.333 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.334 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.334 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.336 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.349 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.349 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.351 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.389 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:22 +0000 (0:00:00.063) 0:02:18.615 **** 2021-02-10T11:13:22.396 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-10T11:13:22.397 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.399 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.410 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.411 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.422 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.423 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.435 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.446 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.447 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.458 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.458 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.469 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.480 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.491 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.493 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.503 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.515 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.527 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.528 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.538 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.549 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.560 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.571 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.571 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.582 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.593 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.594 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.605 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.616 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.617 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.628 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.639 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.693 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:22 +0000 (0:00:00.304) 0:02:18.920 **** 2021-02-10T11:13:22.721 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:22.753 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:22 +0000 (0:00:00.059) 0:02:18.980 **** 2021-02-10T11:13:22.865 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-10T11:13:22.865 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T11:13:23.221 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-10T11:13:23.296 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T11:13:23.661 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-10T11:13:23.734 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T11:13:24.122 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-10T11:13:24.217 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T11:13:24.594 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-10T11:13:24.668 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T11:13:25.033 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-10T11:13:25.120 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T11:13:25.473 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-10T11:13:25.567 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T11:13:25.915 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-10T11:13:26.021 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T11:13:26.356 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-10T11:13:26.460 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T11:13:26.793 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-10T11:13:26.897 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T11:13:27.231 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-10T11:13:27.336 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T11:13:27.669 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-10T11:13:27.774 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T11:13:28.108 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-10T11:13:28.213 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T11:13:28.549 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-10T11:13:28.655 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T11:13:28.977 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-10T11:13:29.096 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T11:13:29.418 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-10T11:13:29.532 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T11:13:29.856 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-10T11:13:29.960 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T11:13:30.294 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-10T11:13:30.399 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T11:13:30.732 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-10T11:13:30.839 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T11:13:31.162 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-10T11:13:31.268 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T11:13:31.600 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-10T11:13:31.705 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T11:13:32.037 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-10T11:13:32.142 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T11:13:32.465 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-10T11:13:32.581 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T11:13:32.905 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-10T11:13:33.011 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T11:13:33.338 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-10T11:13:33.454 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T11:13:33.773 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-10T11:13:33.888 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T11:13:34.224 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-10T11:13:34.330 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T11:13:34.665 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-10T11:13:34.780 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T11:13:35.093 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-10T11:13:35.219 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T11:13:35.534 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-10T11:13:35.651 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T11:13:35.972 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-10T11:13:36.088 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T11:13:36.411 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-10T11:13:36.413 INFO:teuthology.task.ansible.out: 2021-02-10T11:13:36.527 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T11:13:36.841 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-10T11:13:36.966 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T11:13:37.280 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-10T11:13:37.406 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T11:13:37.719 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-10T11:13:37.835 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T11:13:38.149 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-10T11:13:38.276 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T11:13:38.581 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-10T11:13:38.718 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T11:13:39.023 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-10T11:13:39.503 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T11:13:39.737 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-10T11:13:40.216 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T11:13:40.499 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-10T11:13:40.647 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T11:13:40.939 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-10T11:13:41.086 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T11:13:41.370 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-10T11:13:41.518 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T11:13:41.520 INFO:teuthology.task.ansible.out: 2021-02-10T11:13:41.812 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-10T11:13:41.951 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T11:13:42.246 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-10T11:13:42.393 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T11:13:42.688 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-10T11:13:42.825 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T11:13:43.120 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-10T11:13:43.262 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T11:13:43.554 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-10T11:13:43.702 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T11:13:43.996 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-10T11:13:44.142 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T11:13:44.436 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-10T11:13:44.573 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T11:13:44.626 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:44 +0000 (0:00:21.872) 0:02:40.853 **** 2021-02-10T11:13:44.874 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-10T11:13:44.966 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:44 +0000 (0:00:00.339) 0:02:41.193 **** 2021-02-10T11:13:47.521 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-10T11:13:47.522 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955624.6520894-172271802510538/source", "state": "file", "uid": 0} 2021-02-10T11:13:47.557 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:02.591) 0:02:43.784 **** 2021-02-10T11:13:47.584 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-10T11:13:47.585 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => { "msg": "Host gibba025.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T11:13:47.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.062) 0:02:43.847 **** 2021-02-10T11:13:47.627 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-10T11:13:47.629 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955624.9890187-124720883243555/source", "state": "file", "uid": 0} 2021-02-10T11:13:47.665 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.045) 0:02:43.892 **** 2021-02-10T11:13:47.674 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-10T11:13:47.676 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T11:13:47.709 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.043) 0:02:43.936 **** 2021-02-10T11:13:47.717 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-10T11:13:47.718 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => { "msg": "Host gibba017.front.sepia.ceph.com is running RedHat 7.9 (Maipo)" } 2021-02-10T11:13:47.752 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.042) 0:02:43.979 **** 2021-02-10T11:13:47.759 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-10T11:13:47.760 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:47.794 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.042) 0:02:44.021 **** 2021-02-10T11:13:47.800 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-10T11:13:47.802 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml) => {"ansible_facts": {"rhsm_repos": ["rhel-7-server-rpms", "rhel-7-server-optional-rpms", "rhel-7-server-extras-rpms", "rhel-ha-for-rhel-7-server-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_7.yml"} 2021-02-10T11:13:47.836 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.042) 0:02:44.063 **** 2021-02-10T11:13:47.863 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-10T11:13:47.865 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:47.896 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:47 +0000 (0:00:00.059) 0:02:44.123 **** 2021-02-10T11:13:48.319 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-10T11:13:48.981 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.067590", "end": "2021-02-10 11:13:48.300002", "rc": 0, "start": "2021-02-10 11:13:48.232412", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.462) 0:02:44.585 **** TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.064) 0:02:44.650 **** TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.067693", "end": "2021-02-10 11:13:48.393430", "rc": 0, "start": "2021-02-10 11:13:48.325737", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.044) 0:02:44.694 **** TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.043) 0:02:44.738 **** TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.041) 0:02:44.779 **** TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 ( 2021-02-10T11:13:48.983 INFO:teuthology.task.ansible.out:0:00:00.040) 0:02:44.820 **** TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.039) 0:02:44.860 **** TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.045) 0:02:44.906 **** TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.040) 0:02:44.946 **** TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.044) 0:02:44.991 **** TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.041) 0:02:45.032 **** TASK [common : Set entitlements_path] ****************************************** ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.039) 0:02:45.072 **** TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.041) 0:02:45 2021-02-10T11:13:48.984 INFO:teuthology.task.ansible.out:.114 **** TASK [common : Include Red Hat encrypted variables.] *************************** ok: [gibba025.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [gibba025.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.050) 0:02:45.164 **** TASK [common : Set entitlements_path] ****************************************** ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} Wednesday 10 February 2021 11:13:48 +0000 (0:00:00.040) 0:02:45.205 **** TASK [common : Set have_entitlements] ****************************************** 2021-02-10T11:13:48.985 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T11:13:49.021 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:49 +0000 (0:00:00.042) 0:02:45.247 **** 2021-02-10T11:13:49.037 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-10T11:13:49.039 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [gibba017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:13:49.075 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:49 +0000 (0:00:00.054) 0:02:45.302 **** 2021-02-10T11:13:49.114 INFO:teuthology.task.ansible.out: 2021-02-10T11:13:49.115 INFO:teuthology.task.ansible.out:TASK [common : Set have_entitlements] ****************************************** ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-02-10T11:13:49.148 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:49 +0000 (0:00:00.073) 0:02:45.375 **** 2021-02-10T11:13:50.733 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-10T11:13:50.735 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612955629.95-dcN6fJ/katello-ca-consumer-latest.noarchYJm95B.rpm is already installed"]} 2021-02-10T11:13:50.769 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:50 +0000 (0:00:01.619) 0:02:46.995 **** 2021-02-10T11:13:50.776 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["katello-ca-consumer-satellite.front.sepia.ceph.com-1.0-3.noarch providing /root/.ansible/tmp/ansible-moduletmp-1612955629.95-2HB1Ls/katello-ca-consumer-latest.noarchKK4IJp.rpm is already installed"]} 2021-02-10T11:13:50.814 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:50 +0000 (0:00:00.045) 0:02:47.041 **** 2021-02-10T11:13:51.673 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-10T11:13:51.675 INFO:teuthology.task.ansible.out:fatal: [gibba025.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T11:13:51.708 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:51 +0000 (0:00:00.893) 0:02:47.935 **** 2021-02-10T11:13:51.714 INFO:teuthology.task.ansible.out:fatal: [gibba017.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-02-10T11:13:51.756 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:51 +0000 (0:00:00.048) 0:02:47.983 **** 2021-02-10T11:13:51.762 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-10T11:13:51.764 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T11:13:51.798 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:51 +0000 (0:00:00.041) 0:02:48.024 **** 2021-02-10T11:13:51.803 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-02-10T11:13:51.838 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:51 +0000 (0:00:00.039) 0:02:48.064 **** 2021-02-10T11:13:57.716 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** changed: [gibba025.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:13:57.750 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:57 +0000 (0:00:05.912) 0:02:53.977 **** 2021-02-10T11:13:57.792 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T11:13:57.794 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T11:13:57.826 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:57 +0000 (0:00:00.076) 0:02:54.053 **** 2021-02-10T11:13:58.501 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T11:13:58.502 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.214611", "end": "2021-02-10 11:13:58.483865", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:13:58.269254", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T11:13:58.535 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.708) 0:02:54.762 **** 2021-02-10T11:13:58.542 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-10T11:13:58.542 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:13:58.578 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.042) 0:02:54.804 **** 2021-02-10T11:13:58.584 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T11:13:58.586 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:58.619 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.041) 0:02:54.846 **** 2021-02-10T11:13:58.625 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-10T11:13:58.626 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-02-10T11:13:58.666 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.046) 0:02:54.893 **** 2021-02-10T11:13:58.673 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T11:13:58.675 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:58.710 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.043) 0:02:54.937 **** 2021-02-10T11:13:58.738 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [gibba025.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:13:58.771 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.061) 0:02:54.998 **** 2021-02-10T11:13:58.810 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:13:58.811 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:58.843 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:58 +0000 (0:00:00.072) 0:02:55.070 **** 2021-02-10T11:13:59.362 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-10T11:13:59.364 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.234310", "end": "2021-02-10 11:13:59.339697", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:13:59.105387", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2021-02-10T11:13:59.396 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:59 +0000 (0:00:00.552) 0:02:55.623 **** 2021-02-10T11:13:59.435 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-10T11:13:59.436 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:59.471 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:59 +0000 (0:00:00.074) 0:02:55.698 **** 2021-02-10T11:13:59.522 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-10T11:13:59.524 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:59.554 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:59 +0000 (0:00:00.082) 0:02:55.781 **** 2021-02-10T11:13:59.580 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-02-10T11:13:59.582 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-10T11:13:59.612 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:59 +0000 (0:00:00.058) 0:02:55.839 **** 2021-02-10T11:13:59.649 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:13:59.651 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:13:59.681 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:13:59 +0000 (0:00:00.069) 0:02:55.908 **** 2021-02-10T11:14:00.648 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T11:14:00.649 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.346222", "end": "2021-02-10 11:14:00.626385", "rc": 0, "start": "2021-02-10 11:13:59.280163", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T11:14:00.685 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:00 +0000 (0:00:01.003) 0:02:56.912 **** 2021-02-10T11:14:00.746 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T11:14:00.747 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T11:14:00.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:00 +0000 (0:00:00.093) 0:02:57.006 **** 2021-02-10T11:14:00.818 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T11:14:00.820 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:00.855 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:00 +0000 (0:00:00.075) 0:02:57.082 **** 2021-02-10T11:14:00.894 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T11:14:00.895 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:14:00.930 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:00 +0000 (0:00:00.075) 0:02:57.157 **** 2021-02-10T11:14:00.968 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T11:14:00.968 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:14:01.001 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:00 +0000 (0:00:00.070) 0:02:57.228 **** 2021-02-10T11:14:01.632 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-02-10T11:14:01.633 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:01.496801", "end": "2021-02-10 11:14:01.618401", "rc": 0, "start": "2021-02-10 11:14:00.121600", "stderr": "", "stderr_lines": [], "stdout": "rhel-7-server-rpms", "stdout_lines": ["rhel-7-server-rpms"]} 2021-02-10T11:14:01.667 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:01 +0000 (0:00:00.665) 0:02:57.894 **** 2021-02-10T11:14:01.705 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-10T11:14:01.706 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-7-server-rpms"]}, "changed": false} 2021-02-10T11:14:01.739 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:01 +0000 (0:00:00.072) 0:02:57.966 **** 2021-02-10T11:14:01.766 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-10T11:14:01.767 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:01.802 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:01 +0000 (0:00:00.062) 0:02:58.029 **** 2021-02-10T11:14:01.839 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-10T11:14:01.841 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:14:01.873 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:01 +0000 (0:00:00.070) 0:02:58.099 **** 2021-02-10T11:14:01.911 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-10T11:14:01.913 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-02-10T11:14:01.946 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:01 +0000 (0:00:00.073) 0:02:58.173 **** 2021-02-10T11:14:03.199 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-10T11:14:03.200 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:14:03.233 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:03 +0000 (0:00:01.287) 0:02:59.460 **** 2021-02-10T11:14:04.192 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:14:04.227 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:04 +0000 (0:00:00.993) 0:03:00.454 **** 2021-02-10T11:14:05.770 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T11:14:05.771 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.079579", "end": "2021-02-10 11:14:05.750876", "rc": 0, "start": "2021-02-10 11:14:03.671297", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T11:14:05.807 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:05 +0000 (0:00:01.580) 0:03:02.034 **** 2021-02-10T11:14:06.266 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T11:14:06.268 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T11:14:06.300 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:06 +0000 (0:00:00.492) 0:03:02.526 **** 2021-02-10T11:14:06.752 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T11:14:06.754 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:14:06.787 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:06 +0000 (0:00:00.487) 0:03:03.014 **** 2021-02-10T11:14:06.793 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-10T11:14:06.794 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-7-server-rpms", "--enable=rhel-7-server-optional-rpms", "--enable=rhel-7-server-extras-rpms", "--enable=rhel-ha-for-rhel-7-server-rpms"], "delta": "0:00:02.104344", "end": "2021-02-10 11:14:06.771433", "rc": 0, "start": "2021-02-10 11:14:04.667089", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-optional-rpms' is enabled for this system.\nRepository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.\nRepository 'rhel-7-server-extras-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-optional-rpms' is enabled for this system.", "Repository 'rhel-ha-for-rhel-7-server-rpms' is enabled for this system.", "Repository 'rhel-7-server-extras-rpms' is enabled for this system."]} 2021-02-10T11:14:06.829 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:06 +0000 (0:00:00.041) 0:03:03.056 **** 2021-02-10T11:14:07.286 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-10T11:14:07.287 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-02-10T11:14:07.318 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:07 +0000 (0:00:00.489) 0:03:03.545 **** 2021-02-10T11:14:07.789 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-10T11:14:07.790 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:14:07.847 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:07 +0000 (0:00:00.528) 0:03:04.074 **** 2021-02-10T11:14:08.784 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [gibba025.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T11:14:09.859 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-10T11:14:10.703 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T11:14:10.739 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:10 +0000 (0:00:02.892) 0:03:06.966 **** 2021-02-10T11:14:11.768 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-10T11:14:11.808 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:11 +0000 (0:00:01.068) 0:03:08.034 **** 2021-02-10T11:14:12.675 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T11:14:12.676 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T11:14:13.746 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "state": "file", "uid": 0} 2021-02-10T11:14:14.594 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T11:14:14.630 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:02.822) 0:03:10.857 **** 2021-02-10T11:14:14.657 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:14:14.659 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.690 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:00.060) 0:03:10.917 **** 2021-02-10T11:14:14.717 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.751 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:00.060) 0:03:10.977 **** 2021-02-10T11:14:14.778 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-10T11:14:14.779 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.810 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:00.059) 0:03:11.037 **** 2021-02-10T11:14:14.838 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.872 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:00.061) 0:03:11.098 **** 2021-02-10T11:14:14.910 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T11:14:14.911 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.922 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.922 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:14.958 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:14 +0000 (0:00:00.085) 0:03:11.184 **** 2021-02-10T11:14:14.985 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:14:14.986 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.020 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.062) 0:03:11.246 **** 2021-02-10T11:14:15.047 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-02-10T11:14:15.048 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.080 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.060) 0:03:11.307 **** 2021-02-10T11:14:15.107 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.141 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.061) 0:03:11.368 **** 2021-02-10T11:14:15.171 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-10T11:14:15.173 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.062) 0:03:11.430 **** 2021-02-10T11:14:15.231 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T11:14:15.232 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.264 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.060) 0:03:11.491 **** 2021-02-10T11:14:15.659 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-10T11:14:15.660 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "state": "file", "uid": 0} 2021-02-10T11:14:15.699 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.435) 0:03:11.926 **** 2021-02-10T11:14:15.727 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-10T11:14:15.728 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.764 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.064) 0:03:11.991 **** 2021-02-10T11:14:15.792 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:14:15.794 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.829 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.064) 0:03:12.055 **** 2021-02-10T11:14:15.856 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.891 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.062) 0:03:12.118 **** 2021-02-10T11:14:15.920 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:15.953 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:15 +0000 (0:00:00.061) 0:03:12.180 **** 2021-02-10T11:14:15.991 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-10T11:14:15.993 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.004 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.006 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.041 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.088) 0:03:12.268 **** 2021-02-10T11:14:16.070 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-10T11:14:16.072 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.103 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.062) 0:03:12.330 **** 2021-02-10T11:14:16.131 INFO:teuthology.task.ansible.out: 2021-02-10T11:14:16.133 INFO:teuthology.task.ansible.out:TASK [common : Set the timezone] *********************************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.166 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.062) 0:03:12.393 **** 2021-02-10T11:14:16.195 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-10T11:14:16.197 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.235 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.068) 0:03:12.461 **** 2021-02-10T11:14:16.261 INFO:teuthology.task.ansible.out: 2021-02-10T11:14:16.263 INFO:teuthology.task.ansible.out:TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.301 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.066) 0:03:12.528 **** 2021-02-10T11:14:16.331 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-02-10T11:14:16.332 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:16.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.064) 0:03:12.592 **** 2021-02-10T11:14:16.516 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T11:14:16.517 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.815655", "end": "2021-02-10 11:14:16.500927", "rc": 0, "start": "2021-02-10 11:14:15.685272", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:14:16.556 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:16 +0000 (0:00:00.190) 0:03:12.783 **** 2021-02-10T11:14:17.387 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T11:14:17.388 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T11:14:17.424 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:17 +0000 (0:00:00.867) 0:03:13.651 **** 2021-02-10T11:14:17.452 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T11:14:17.454 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:17.492 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:17 +0000 (0:00:00.068) 0:03:13.719 **** 2021-02-10T11:14:17.520 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T11:14:17.521 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:17.553 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:17 +0000 (0:00:00.061) 0:03:13.780 **** 2021-02-10T11:14:17.580 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:17.616 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:17 +0000 (0:00:00.062) 0:03:13.842 **** 2021-02-10T11:14:17.643 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-10T11:14:17.645 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:17.680 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:17 +0000 (0:00:00.064) 0:03:13.907 **** 2021-02-10T11:14:18.510 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-10T11:14:18.512 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.700274", "end": "2021-02-10 11:14:18.500959", "rc": 0, "start": "2021-02-10 11:14:16.800685", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:14:18.546 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:18 +0000 (0:00:00.865) 0:03:14.772 **** 2021-02-10T11:14:19.374 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-10T11:14:19.375 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-50.el7.x86_64 providing krb5-workstation is already installed"]} 2021-02-10T11:14:19.407 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.861) 0:03:15.634 **** 2021-02-10T11:14:19.434 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-10T11:14:19.435 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:19.467 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.060) 0:03:15.694 **** 2021-02-10T11:14:19.496 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-10T11:14:19.498 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:19.529 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.061) 0:03:15.756 **** 2021-02-10T11:14:19.556 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-10T11:14:19.557 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:19.591 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.061) 0:03:15.817 **** 2021-02-10T11:14:19.607 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-10T11:14:19.608 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T11:14:19.644 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.053) 0:03:15.871 **** 2021-02-10T11:14:19.650 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-10T11:14:19.652 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:14:19.686 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:19 +0000 (0:00:00.041) 0:03:15.912 **** 2021-02-10T11:14:21.612 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-10T11:14:21.613 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-10T11:14:21.647 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:21 +0000 (0:00:01.961) 0:03:17.874 **** 2021-02-10T11:14:47.246 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-10T11:14:47.248 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-02-10T11:14:47.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:47 +0000 (0:00:25.634) 0:03:43.508 **** 2021-02-10T11:14:48.879 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2021-02-10T11:14:48.921 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:48 +0000 (0:00:01.640) 0:03:45.148 **** 2021-02-10T11:14:49.253 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T11:14:49.254 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T11:14:50.864 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-10T11:14:51.158 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T11:14:52.765 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-10T11:14:53.091 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T11:14:53.139 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:53 +0000 (0:00:04.218) 0:03:49.366 **** 2021-02-10T11:14:53.586 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T11:14:53.588 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:14:53.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:53 +0000 (0:00:00.480) 0:03:49.847 **** 2021-02-10T11:14:54.653 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-10T11:14:54.655 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-10T11:14:54.690 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:54 +0000 (0:00:01.069) 0:03:50.916 **** 2021-02-10T11:14:55.139 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-10T11:14:55.140 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:14:55.172 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:14:55 +0000 (0:00:00.481) 0:03:51.398 **** 2021-02-10T11:14:56.251 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T11:14:56.253 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955693.6453068-41517333796386/source", "state": "file", "uid": 0} 2021-02-10T11:14:58.195 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955695.1979604-160385398531784/source", "state": "file", "uid": 0} 2021-02-10T11:14:58.398 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T11:15:00.077 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-10T11:15:00.302 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T11:15:00.340 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:00 +0000 (0:00:05.167) 0:03:56.566 **** 2021-02-10T11:15:00.380 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T11:15:00.380 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:00.421 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:00 +0000 (0:00:00.081) 0:03:56.648 **** 2021-02-10T11:15:00.471 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-10T11:15:00.473 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:15:00.517 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:00 +0000 (0:00:00.095) 0:03:56.744 **** 2021-02-10T11:15:00.979 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-10T11:15:00.980 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:15:01.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:01 +0000 (0:00:00.494) 0:03:57.238 **** 2021-02-10T11:15:01.073 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [gibba025.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:01.094 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:01.104 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:01.115 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:01.148 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:01 +0000 (0:00:00.136) 0:03:57.375 **** 2021-02-10T11:15:01.975 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-10T11:15:01.976 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-10T11:15:02.013 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:02 +0000 (0:00:00.864) 0:03:58.240 **** 2021-02-10T11:15:02.052 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-10T11:15:02.053 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:02.087 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:02 +0000 (0:00:00.073) 0:03:58.313 **** 2021-02-10T11:15:02.125 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-10T11:15:02.127 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:15:02.159 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:02 +0000 (0:00:00.072) 0:03:58.385 **** 2021-02-10T11:15:02.609 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-10T11:15:02.611 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-10T11:15:02.645 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:02 +0000 (0:00:00.486) 0:03:58.872 **** 2021-02-10T11:15:02.694 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-10T11:15:02.696 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:02.717 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:02.749 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-10T11:15:04.801 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:02 +0000 (0:00:00.151) 0:03:59.023 **** 2021-02-10T11:15:06.118 INFO:teuthology.task.ansible.out: 2021-02-10T11:15:14.381 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} Wednesday 10 February 2021 11:15:06 +0000 (0:00:03.360) 0:04:02.384 **** TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:06 +0000 (0:00:00.095) 0:04:02.480 **** TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} Wednesday 10 February 2021 11:15:08 +0000 (0:00:01.766) 0:04:04.246 **** TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:08 +0000 (0:00:00.072) 0:04:04.319 **** TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": " 2021-02-10T11:15:14.383 INFO:teuthology.task.ansible.out:file", "uid": 0} Wednesday 10 February 2021 11:15:08 +0000 (0:00:00.128) 0:04:04.447 **** ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} Wednesday 10 February 2021 11:15:10 +0000 (0:00:01.851) 0:04:06.298 **** TASK [common : Upload nagios check_mem script] ********************************* ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} Wednesday 10 February 2021 11:15:10 +0000 (0:00:00.125) 0:04:06.424 **** TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [gibba025.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:10 +0000 (0:00:00.076) 0:04:06.500 **** TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Wednesday 10 February 2021 11:15:10 +0000 (0:00:00.493) 0:04:06.994 **** TASK [common : Check firewalld status] ***************************************** fatal: [gibba025.front.sepia.ceph.com]: FAILED! => {"censored": "the 2021-02-10T11:15:14.384 INFO:teuthology.task.ansible.out:output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring Wednesday 10 February 2021 11:15:11 +0000 (0:00:00.528) 0:04:07.522 **** TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:11 +0000 (0:00:00.059) 0:04:07.582 **** TASK [common : Upload nagios check_mem script] ********************************* ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} Wednesday 10 February 2021 11:15:12 +0000 (0:00:00.672) 0:04:08.255 **** TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [gibba017.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:12 +0000 (0:00:00.085) 0:04:08.340 **** TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Wednesday 10 February 2021 11:15:12 +0000 (0:00:00.488) 0:04:08.828 **** TASK [common : Check firewalld status] ***************************************** fatal: [gibba017.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to 2021-02-10T11:15:14.385 INFO:teuthology.task.ansible.out:the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring Wednesday 10 February 2021 11:15:13 +0000 (0:00:00.518) 0:04:09.347 **** TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 10 February 2021 11:15:13 +0000 (0:00:00.065) 0:04:09.412 **** TASK [common : Upload nagios nrpe config.] ************************************* ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} Wednesday 10 February 2021 11:15:13 +0000 (0:00:00.150) 0:04:09.563 **** TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ActiveEnterTimestampMonotonic": "19789266", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target basic.target systemd-journald.socket var-run.mount system.slice nss-lookup.target network-online.target network.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:09:44 UTC", "AssertTimestampMonotonic": "19776299", "Before": "getty@tty1.service shutdown.target multi-user.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanRel 2021-02-10T11:15:14.387 INFO:teuthology.task.ansible.out:oad": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ConditionTimestampMonotonic": "19776298", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1196", "ExecMainStartTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ExecMainStartTimestampMonotonic": "19789233", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 11:09:44 UTC] ; stop_time=[n/a] ; pid=1196 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:09:44 UTC", "InactiveExitTimestampMonotonic": "19789266", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744 2021-02-10T11:15:14.388 INFO:teuthology.task.ansible.out:073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1196", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled" 2021-02-10T11:15:14.389 INFO:teuthology.task.ansible.out:, "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 11:09:44 UTC", "WatchdogTimestampMonotonic": "19789255", "WatchdogUSec": "0"}} Wednesday 10 February 2021 11:15:14 +0000 (0:00:00.879) 0:04:10.442 **** 2021-02-10T11:15:14.667 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T11:15:14.668 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001384", "end": "2021-02-10 11:15:14.650070", "rc": 0, "start": "2021-02-10 11:15:14.648686", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T11:15:14.723 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:14 +0000 (0:00:00.508) 0:04:10.950 **** 2021-02-10T11:15:15.123 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-10T11:15:15.125 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-10T11:15:15.159 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:15 +0000 (0:00:00.435) 0:04:11.385 **** 2021-02-10T11:15:15.553 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-10T11:15:15.555 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T11:15:15.589 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:15 +0000 (0:00:00.430) 0:04:11.816 **** 2021-02-10T11:15:15.680 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-10T11:15:15.681 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ActiveEnterTimestampMonotonic": "19176411", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-journald.socket var-run.mount nss-lookup.target local-fs.target basic.target system.slice network.target network-online.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:10:05 UTC", "AssertTimestampMonotonic": "19175888", "Before": "getty@tty1.service shutdown.target xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ConditionTimestampMonotonic": "19175887", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1198", "ExecMainStartTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ExecMainStartTimestampMonotonic": "19176373", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /e 2021-02-10T11:15:15.683 INFO:teuthology.task.ansible.out:tc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Wed 2021-02-10 11:10:05 UTC] ; stop_time=[n/a] ; pid=1198 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:10:05 UTC", "InactiveExitTimestampMonotonic": "19176411", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "1198", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Re 2021-02-10T11:15:15.684 INFO:teuthology.task.ansible.out:sult": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-02-10 11:10:05 UTC", "WatchdogTimestampMonotonic": "19176388", "WatchdogUSec": "0"}} 2021-02-10T11:15:15.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:15 +0000 (0:00:00.129) 0:04:11.945 **** 2021-02-10T11:15:16.167 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-02-10T11:15:16.168 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001372", "end": "2021-02-10 11:15:16.150567", "rc": 0, "start": "2021-02-10 11:15:16.149195", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-10T11:15:16.201 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:16 +0000 (0:00:00.483) 0:04:12.428 **** 2021-02-10T11:15:17.024 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-10T11:15:17.026 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-34.el7.x86_64 providing policycoreutils-python is already installed"]} 2021-02-10T11:15:17.059 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:17 +0000 (0:00:00.857) 0:04:13.285 **** 2021-02-10T11:15:20.065 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T11:15:20.066 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T11:15:20.099 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:20 +0000 (0:00:03.040) 0:04:16.326 **** 2021-02-10T11:15:21.086 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T11:15:21.088 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T11:15:21.120 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:21 +0000 (0:00:01.020) 0:04:17.347 **** 2021-02-10T11:15:21.503 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-10T11:15:21.505 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing selinux-policy-targeted are up to date", ""]} 2021-02-10T11:15:21.540 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:21 +0000 (0:00:00.420) 0:04:17.767 **** 2021-02-10T11:15:22.395 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-10T11:15:22.397 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-10T11:15:22.442 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:22 +0000 (0:00:00.901) 0:04:18.669 **** 2021-02-10T11:15:27.171 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-10T11:15:27.172 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.593312", "end": "2021-02-10 11:15:27.146948", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:15:21.553636", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T11:15:27.224 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:27 +0000 (0:00:04.781) 0:04:23.451 **** 2021-02-10T11:15:28.559 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:05.658946", "end": "2021-02-10 11:15:28.544537", "failed_when_result": false, "rc": 0, "start": "2021-02-10 11:15:22.885591", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-10T11:15:28.598 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:28 +0000 (0:00:01.373) 0:04:24.825 **** 2021-02-10T11:15:29.166 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T11:15:29.167 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T11:15:29.203 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:29 +0000 (0:00:00.604) 0:04:25.429 **** 2021-02-10T11:15:29.686 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-10T11:15:29.687 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.007753", "end": "2021-02-10 11:15:29.673719", "rc": 0, "start": "2021-02-10 11:15:29.665966", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T11:15:29.723 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:29 +0000 (0:00:00.520) 0:04:25.949 **** 2021-02-10T11:15:30.196 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009638", "end": "2021-02-10 11:15:30.175680", "rc": 0, "start": "2021-02-10 11:15:30.166042", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:30.229 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:30 +0000 (0:00:00.506) 0:04:26.456 **** 2021-02-10T11:15:30.561 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-10T11:15:30.562 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-10T11:15:30.605 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:30 +0000 (0:00:00.376) 0:04:26.832 **** 2021-02-10T11:15:31.082 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.012937", "end": "2021-02-10 11:15:31.069214", "rc": 0, "start": "2021-02-10 11:15:31.056277", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2021-02-10T11:15:31.129 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:31 +0000 (0:00:00.523) 0:04:27.356 **** 2021-02-10T11:15:31.587 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-10T11:15:31.588 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.010537", "end": "2021-02-10 11:15:31.570192", "rc": 0, "start": "2021-02-10 11:15:31.559655", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:31.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:31 +0000 (0:00:00.490) 0:04:27.846 **** 2021-02-10T11:15:36.351 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T11:15:36.352 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.647756", "end": "2021-02-10 11:15:36.328735", "rc": 0, "start": "2021-02-10 11:15:30.680979", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:36.386 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:36 +0000 (0:00:04.766) 0:04:32.613 **** 2021-02-10T11:15:36.853 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T11:15:36.854 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T11:15:36.893 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:36 +0000 (0:00:00.506) 0:04:33.120 **** 2021-02-10T11:15:36.931 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T11:15:36.933 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:15:36.965 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:36 +0000 (0:00:00.071) 0:04:33.191 **** 2021-02-10T11:15:37.012 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T11:15:37.013 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:15:37.045 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:37 +0000 (0:00:00.080) 0:04:33.272 **** 2021-02-10T11:15:37.083 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T11:15:37.084 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T11:15:37.117 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:37 +0000 (0:00:00.072) 0:04:33.344 **** 2021-02-10T11:15:37.155 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T11:15:37.157 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:15:37.191 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:37 +0000 (0:00:00.073) 0:04:33.418 **** 2021-02-10T11:15:37.625 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-10T11:15:37.627 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:05.557143", "end": "2021-02-10 11:15:37.609026", "rc": 0, "start": "2021-02-10 11:15:32.051883", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:37.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:37 +0000 (0:00:00.468) 0:04:33.887 **** 2021-02-10T11:15:37.666 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T11:15:37.668 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:15:37.703 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:37 +0000 (0:00:00.043) 0:04:33.930 **** 2021-02-10T11:15:38.127 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-10T11:15:38.128 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-10T11:15:38.163 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.459) 0:04:34.390 **** 2021-02-10T11:15:38.170 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T11:15:38.170 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T11:15:38.206 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.043) 0:04:34.433 **** 2021-02-10T11:15:38.213 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-10T11:15:38.213 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2021-02-10T11:15:38.247 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.040) 0:04:34.474 **** 2021-02-10T11:15:38.254 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T11:15:38.254 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:38.296 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.048) 0:04:34.522 **** 2021-02-10T11:15:38.302 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-10T11:15:38.303 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:15:38.344 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.047) 0:04:34.570 **** 2021-02-10T11:15:38.350 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T11:15:38.352 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:38.395 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.051) 0:04:34.622 **** 2021-02-10T11:15:38.402 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-10T11:15:38.402 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}, "rhel-7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "epel_packages": ["cryptopp-devel", "cryptopp", "dbench", "fcgi", "fuse-sshfs", "perl-JSON-XS", "leveldb", "xmlstarlet"], "nfs_service": "nfs-server", "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "xfsprogs-devel", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "lvm2", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "ant", "lsof", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN", "podman", "podman-docker", "python3"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_7.yml"} 2021-02-10T11:15:38.444 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.048) 0:04:34.671 **** 2021-02-10T11:15:38.484 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-10T11:15:38.486 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-10T11:15:38.518 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.074) 0:04:34.745 **** 2021-02-10T11:15:38.857 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T11:15:38.858 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T11:15:38.892 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:38 +0000 (0:00:00.374) 0:04:35.119 **** 2021-02-10T11:15:38.963 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-10T11:15:38.965 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-10T11:15:39.015 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:39 +0000 (0:00:00.122) 0:04:35.242 **** 2021-02-10T11:15:39.464 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-10T11:15:39.465 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-10T11:15:39.496 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:39 +0000 (0:00:00.481) 0:04:35.723 **** 2021-02-10T11:15:39.525 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-10T11:15:39.526 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:39.558 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:39 +0000 (0:00:00.061) 0:04:35.784 **** 2021-02-10T11:15:39.584 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-10T11:15:39.586 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:39.618 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:39 +0000 (0:00:00.059) 0:04:35.844 **** 2021-02-10T11:15:40.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-10T11:15:40.075 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-10T11:15:40.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:40 +0000 (0:00:00.494) 0:04:36.339 **** 2021-02-10T11:15:40.909 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T11:15:40.910 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T11:15:40.945 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:40 +0000 (0:00:00.832) 0:04:37.172 **** 2021-02-10T11:15:40.982 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T11:15:40.984 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba025"}, "changed": false} 2021-02-10T11:15:41.015 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:41 +0000 (0:00:00.070) 0:04:37.242 **** 2021-02-10T11:15:42.050 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-10T11:15:42.052 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba025", "ansible_hostname": "gibba025", "ansible_nodename": "gibba025"}, "changed": false, "name": "gibba025"} 2021-02-10T11:15:42.087 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:42 +0000 (0:00:01.071) 0:04:38.313 **** 2021-02-10T11:15:42.115 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-10T11:15:42.117 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-02-10T11:15:42.155 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:42 +0000 (0:00:00.068) 0:04:38.382 **** 2021-02-10T11:15:42.194 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-02-10T11:15:42.195 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "gibba017"}, "changed": false} 2021-02-10T11:15:42.229 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:42 +0000 (0:00:00.074) 0:04:38.456 **** 2021-02-10T11:15:42.776 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-10T11:15:42.776 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "gibba017", "ansible_hostname": "gibba017", "ansible_nodename": "gibba017"}, "changed": false, "name": "gibba017"} 2021-02-10T11:15:42.810 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:42 +0000 (0:00:00.580) 0:04:39.037 **** 2021-02-10T11:15:44.005 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-10T11:15:44.007 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T11:15:44.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:44 +0000 (0:00:01.232) 0:04:40.270 **** 2021-02-10T11:15:44.722 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-10T11:15:44.761 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:44 +0000 (0:00:00.717) 0:04:40.988 **** 2021-02-10T11:15:45.959 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-10T11:15:45.961 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T11:15:45.994 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:45 +0000 (0:00:01.233) 0:04:42.221 **** 2021-02-10T11:15:46.658 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-10T11:15:46.691 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:46 +0000 (0:00:00.697) 0:04:42.918 **** 2021-02-10T11:15:46.760 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-10T11:15:46.762 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T11:15:46.795 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:46 +0000 (0:00:00.103) 0:04:43.022 **** 2021-02-10T11:15:47.454 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-10T11:15:47.489 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:47 +0000 (0:00:00.693) 0:04:43.716 **** 2021-02-10T11:15:48.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-10T11:15:48.711 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T11:15:48.746 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:48 +0000 (0:00:01.254) 0:04:44.970 **** 2021-02-10T11:15:49.197 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:15:49.199 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:15:49.232 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:49 +0000 (0:00:00.488) 0:04:45.459 **** 2021-02-10T11:15:49.414 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-10T11:15:49.415 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-10T11:15:49.450 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:49 +0000 (0:00:00.217) 0:04:45.677 **** 2021-02-10T11:15:49.678 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:15:49.751 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} Wednesday 10 February 2021 11:15:49 +0000 (0:00:00.262) 0:04:45.939 **** 2021-02-10T11:15:49.915 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:15:49.917 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:15:49.951 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:49 +0000 (0:00:00.239) 0:04:46.178 **** 2021-02-10T11:15:50.176 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:15:50.178 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T11:15:50.212 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:50 +0000 (0:00:00.261) 0:04:46.439 **** 2021-02-10T11:15:50.400 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:15:50.401 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "kvm", "state": "present", "system": false} 2021-02-10T11:15:50.437 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:50 +0000 (0:00:00.224) 0:04:46.663 **** 2021-02-10T11:15:50.894 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:15:50.896 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-10T11:15:50.929 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:50 +0000 (0:00:00.492) 0:04:47.155 **** 2021-02-10T11:15:52.199 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:15:52.200 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T11:15:52.234 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:52 +0000 (0:00:01.304) 0:04:48.460 **** 2021-02-10T11:15:52.882 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-10T11:15:52.916 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:52 +0000 (0:00:00.682) 0:04:49.143 **** 2021-02-10T11:15:54.179 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:15:54.181 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T11:15:54.219 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:54 +0000 (0:00:01.302) 0:04:50.446 **** 2021-02-10T11:15:54.846 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-10T11:15:54.880 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:54 +0000 (0:00:00.660) 0:04:51.106 **** 2021-02-10T11:15:56.263 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:15:56.263 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-02-10T11:15:56.265 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.591520", "end": "2021-02-10 11:15:56.246289", "rc": 0, "start": "2021-02-10 11:15:54.654769", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:56.306 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:56 +0000 (0:00:01.426) 0:04:52.533 **** 2021-02-10T11:15:56.334 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T11:15:56.336 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:56.372 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:56 +0000 (0:00:00.066) 0:04:52.599 **** 2021-02-10T11:15:56.898 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:15:56.898 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.563780", "end": "2021-02-10 11:15:56.884158", "rc": 0, "start": "2021-02-10 11:15:55.320378", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:15:56.932 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:56 +0000 (0:00:00.559) 0:04:53.159 **** 2021-02-10T11:15:56.960 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-10T11:15:56.961 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:15:56.994 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:15:56 +0000 (0:00:00.062) 0:04:53.221 **** 2021-02-10T11:15:58.306 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T11:15:58.307 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T11:15:58.933 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'key': 'rhel-7-fcgi-ceph', 'value': {'name': 'RHEL 7 Local fastcgi Repo', 'baseurl': 'http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e6aba9c9ab05e27255ed430f065502af284a869b", "dest": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "rhel-7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-rhel7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "RHEL 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/rhel-7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 245, "state": "file", "uid": 0} 2021-02-10T11:16:00.193 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T11:16:00.233 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:03.238) 0:04:56.460 **** 2021-02-10T11:16:00.261 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T11:16:00.301 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.067) 0:04:56.528 **** 2021-02-10T11:16:00.329 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T11:16:00.366 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.064) 0:04:56.593 **** 2021-02-10T11:16:00.394 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T11:16:00.395 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:00.434 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.068) 0:04:56.661 **** 2021-02-10T11:16:00.475 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T11:16:00.477 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:00.519 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.085) 0:04:56.746 **** 2021-02-10T11:16:00.548 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T11:16:00.549 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:00.586 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.066) 0:04:56.812 **** 2021-02-10T11:16:00.614 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:16:00.619 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:00.649 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.063) 0:04:56.876 **** 2021-02-10T11:16:00.824 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-10T11:16:00.825 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "e3b44e00721d55ceb436a2b7c0ec58cd5bc39dc9", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 207, "state": "file", "uid": 0} 2021-02-10T11:16:00.860 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.210) 0:04:57.087 **** 2021-02-10T11:16:00.887 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-10T11:16:00.920 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.060) 0:04:57.147 **** 2021-02-10T11:16:00.947 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-10T11:16:00.983 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:00 +0000 (0:00:00.062) 0:04:57.210 **** 2021-02-10T11:16:01.009 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-10T11:16:01.011 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:01.042 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:01 +0000 (0:00:00.059) 0:04:57.269 **** 2021-02-10T11:16:01.081 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-10T11:16:01.082 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:01.092 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:01.133 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:01 +0000 (0:00:00.090) 0:04:57.360 **** 2021-02-10T11:16:01.159 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-10T11:16:01.161 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:01.205 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:01 +0000 (0:00:00.072) 0:04:57.432 **** 2021-02-10T11:16:01.233 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:01.268 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:01 +0000 (0:00:00.062) 0:04:57.494 **** 2021-02-10T11:16:01.473 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-10T11:16:01.474 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T11:16:01.509 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:01 +0000 (0:00:00.240) 0:04:57.735 **** 2021-02-10T11:16:02.077 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-54.el7_8.noarch providing yum-utils is already installed"]} 2021-02-10T11:16:02.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:02 +0000 (0:00:00.603) 0:04:58.339 **** 2021-02-10T11:16:02.578 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T11:16:02.580 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.610515", "end": "2021-02-10 11:16:02.557185", "rc": 0, "start": "2021-02-10 11:16:01.946670", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T11:16:02.614 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:02 +0000 (0:00:00.501) 0:04:58.841 **** 2021-02-10T11:16:03.081 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T11:16:03.082 INFO:teuthology.task.ansible.out:fatal: [gibba025.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.017868", "end": "2021-02-10 11:16:03.067072", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:16:03.049204", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T11:16:03.116 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:03 +0000 (0:00:00.501) 0:04:59.343 **** 2021-02-10T11:16:03.155 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-10T11:16:03.156 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:03.188 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:03 +0000 (0:00:00.071) 0:04:59.414 **** 2021-02-10T11:16:03.194 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-10T11:16:03.194 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.602823", "end": "2021-02-10 11:16:03.155055", "rc": 0, "start": "2021-02-10 11:16:02.552232", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-", " : manager", "Loading mirror speeds from cached hostfile", " * epel: download-cc-rdu01.fedoraproject.org", "No unfinished transactions left."]} 2021-02-10T11:16:03.229 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:03 +0000 (0:00:00.041) 0:04:59.455 **** 2021-02-10T11:16:03.698 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-10T11:16:03.699 INFO:teuthology.task.ansible.out:fatal: [gibba017.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.018002", "end": "2021-02-10 11:16:03.685314", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:16:03.667312", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-10T11:16:03.732 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:03 +0000 (0:00:00.502) 0:04:59.958 **** 2021-02-10T11:16:03.769 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:03.802 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:03 +0000 (0:00:00.070) 0:05:00.029 **** 2021-02-10T11:16:12.440 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T11:16:12.441 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7- 2021-02-10T11:16:12.442 INFO:teuthology.task.ansible.out:fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 2021-02-10T11:16:12.444 INFO:teuthology.task.ansible.out:\n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} 2021-02-10T11:16:12.475 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:12 +0000 (0:00:08.672) 0:05:08.702 **** 2021-02-10T11:16:13.342 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @rhel-7-server-rpms 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 @rhel-7- 2021-02-10T11:16:13.343 INFO:teuthology.task.ansible.out:fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 2.3 M\n qemu-kvm x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.1 @rhel-7-server-rpms 755 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages (+3 Dependent packages)\n\nInstalled size: 22 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 1/6 \n Erasing : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 2/6 \n Erasing : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 3/6 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/6 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 6/6 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/6 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.1.x86_64 2/6 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 3/6 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.1.x86_64 4/6 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/6 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.1.x86_64 6/6 \n\nRemoved:\n librados2.x86_64 1:10.2.5-4.el7 librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-175.el7_9.1 2021-02-10T11:16:13.344 INFO:teuthology.task.ansible.out:\n qemu-kvm.x86_64 10:1.5.3-175.el7_9.1 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.1 \n\nComplete!\n"]} 2021-02-10T11:16:13.376 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:13 +0000 (0:00:00.901) 0:05:09.603 **** 2021-02-10T11:16:17.065 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:16:17.067 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T11:16:17.068 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:16:17.121 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:17 +0000 (0:00:03.745) 0:05:13.348 **** 2021-02-10T11:16:17.782 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "leveldb", "xmlstarlet", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be erased\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be erased\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-28.el7 @rhel-7-server-rpms 24 k\n leveldb x86_64 1.12.0-11.el7 @epel 439 k\n xmlstarlet x86_64 1.6.1-1.el7 @epel 192 k\n\nTransaction Summary\n================================================================================\nRemove 3 Packages\n\nInstalled size: 655 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : xmlstarlet-1.6.1-1.el7.x86_64 1/3 \n Erasing : boost-random-1.53.0-28.el7.x86_64 2/3 \n Erasing : leveldb-1.12.0-11.el7.x86_64 2021-02-10T11:16:17.784 INFO:teuthology.task.ansible.out: 3/3 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/3 \n Verifying : boost-random-1.53.0-28.el7.x86_64 2/3 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 3/3 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-28.el7 leveldb.x86_64 0:1.12.0-11.el7 \n xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:16:17.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:17 +0000 (0:00:00.701) 0:05:14.050 **** 2021-02-10T11:16:33.159 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:16:33.160 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T11:16:33.161 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T11:16:33.162 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T11:16:33.164 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 29 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T11:16:33.165 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Ve 2021-02-10T11:16:33.165 INFO:teuthology.task.ansible.out:rifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T11:16:33.201 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:33 +0000 (0:00:15.378) 0:05:29.428 **** 2021-02-10T11:16:33.730 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7"]}, "msg": "", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-54.el7_8.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-19.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-120.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-23.el7_8.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-44.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-28.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-22.el7.x86_64 providing xfsprogs is already installed", "xfsprogs-devel-4.5.0-22.el7.x86_64 providing xfsprogs-devel is already installed", "gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "parted-3.1-32.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "7:lvm2-2.02.187-6.el7.x86_64 providing lvm2 is already installed", "1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-65.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.15.0-11.el7.x86_64 providing valgrind is already installed", "python-nose-1 2021-02-10T11:16:33.732 INFO:teuthology.task.ansible.out:.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "lsof-4.87-6.el7.x86_64 providing lsof is already installed", "iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-3.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-65.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-15.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-97.el7_9.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-97.el7_9.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-97.el7_9.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-97.el7_9.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.68.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-d 2021-02-10T11:16:33.733 INFO:teuthology.task.ansible.out:evel is already installed", "python-devel-2.7.5-90.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-4.el7_8.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-297.el7.noarch providing perl-CPAN is already installed", "podman-1.6.4-26.el7_9.x86_64 providing podman is already installed", "podman-docker-1.6.4-26.el7_9.noarch providing podman-docker is already installed", "python3-3.6.8-18.el7.x86_64 providing python3 is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-175.el7_9.3 for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-175.el7_9.3 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-28.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n P 2021-02-10T11:16:33.734 INFO:teuthology.task.ansible.out:ackage Arch Version Repository Size\n================================================================================\nInstalling:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7 rhel-7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-28.el7 rhel-7-server-rpms 39 k\n librados2 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 rhel-7-server-rpms 2.4 M\n qemu-img x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 704 k\n qemu-kvm-common x86_64 10:1.5.3-175.el7_9.3 rhel-7-server-rpms 440 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+5 Dependent packages)\n\nTotal download size: 7.3 M\nInstalled size: 22 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 30 MB/s | 7.3 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-28.el7.x86_64 1/7 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/7 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/7 \n Installing : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 4/7 \n Installing : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 5/7 \n Installing : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 7/7 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : man 2021-02-10T11:16:33.735 INFO:teuthology.task.ansible.out:ager\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 2021-02-10T11:16:33.736 INFO:teuthology.task.ansible.out: 1/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.x86_64 2/7 \n Verifying : 10:qemu-img-1.5.3-175.el7_9.3.x86_64 3/7 \n Verifying : boost-random-1.53.0-28.el7.x86_64 4/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Verifying : 10:qemu-kvm-1.5.3-175.el7_9.3.x86_64 6/7 \n Verifying : 10:qemu-kvm-common-1.5.3-175.el7_9.3.x86_64 7/7 \n\nInstalled:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7 qemu-kvm.x86_64 10:1.5.3-175.el7_9.3 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-28.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-175.el7_9.3 \n qemu-kvm-common.x86_64 10:1.5.3-175.el7_9.3 \n\nComplete!\n"]} 2021-02-10T11:16:33.769 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:33 +0000 (0:00:00.568) 0:05:29.996 **** 2021-02-10T11:16:37.504 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-10T11:16:37.506 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 5.7 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-02-10T11:16:37.507 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:16:37.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:37 +0000 (0:00:03.769) 0:05:33.766 **** 2021-02-10T11:16:38.246 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": ["leveldb", "xmlstarlet"]}, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed", "Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed\n---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n leveldb x86_64 1.12.0-11.el7 epel 161 k\n xmlstarlet x86_64 1.6.1-1.el7 epel 62 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages\n\nTotal download size: 223 k\nInstalled size: 631 k\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 3.0 MB/s | 223 kB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : xmlstarlet-1.6.1-1.el7.x86_64 2021-02-10T11:16:38.248 INFO:teuthology.task.ansible.out: 1/2 \n Installing : leveldb-1.12.0-11.el7.x86_64 2/2 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : leveldb-1.12.0-11.el7.x86_64 1/2 \n Verifying : xmlstarlet-1.6.1-1.el7.x86_64 2/2 \n\nInstalled:\n leveldb.x86_64 0:1.12.0-11.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 \n\nComplete!\n"]} 2021-02-10T11:16:38.282 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:38 +0000 (0:00:00.742) 0:05:34.509 **** 2021-02-10T11:16:38.413 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:16:38.415 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T11:16:38.458 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:38 +0000 (0:00:00.175) 0:05:34.685 **** 2021-02-10T11:16:38.496 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:38.497 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:38.530 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:38 +0000 (0:00:00.072) 0:05:34.757 **** 2021-02-10T11:16:39.041 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T11:16:39.041 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service system.slice basic.target polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ipset.service ebtables.service ip6tables.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T11:16:39.043 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T11:16:39.044 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:16:39.076 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:39 +0000 (0:00:00.545) 0:05:35.302 **** 2021-02-10T11:16:39.166 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:16:39.168 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2021-02-10T11:16:39.204 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:39 +0000 (0:00:00.128) 0:05:35.431 **** 2021-02-10T11:16:39.242 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:39.244 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:39.276 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:39 +0000 (0:00:00.071) 0:05:35.502 **** 2021-02-10T11:16:39.577 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T11:16:39.579 INFO:teuthology.task.ansible.out:fatal: [gibba025.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T11:16:39.631 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:39 +0000 (0:00:00.355) 0:05:35.858 **** 2021-02-10T11:16:39.784 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-10T11:16:39.785 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice basic.target dbus.service polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "shutdown.target ipset.service ip6tables.service iptables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "Ign 2021-02-10T11:16:39.786 INFO:teuthology.task.ansible.out:oreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target system.slice", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", 2021-02-10T11:16:39.788 INFO:teuthology.task.ansible.out:"TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:16:39.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:39 +0000 (0:00:00.191) 0:05:36.050 **** 2021-02-10T11:16:40.315 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-10T11:16:40.316 INFO:teuthology.task.ansible.out:fatal: [gibba017.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-10T11:16:40.349 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.526) 0:05:36.576 **** 2021-02-10T11:16:40.548 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T11:16:40.549 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T11:16:40.583 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.233) 0:05:36.809 **** 2021-02-10T11:16:40.616 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.647 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.064) 0:05:36.874 **** 2021-02-10T11:16:40.674 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-10T11:16:40.675 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.706 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.059) 0:05:36.933 **** 2021-02-10T11:16:40.733 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-10T11:16:40.734 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.767 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.060) 0:05:36.994 **** 2021-02-10T11:16:40.793 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-10T11:16:40.795 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.827 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.060) 0:05:37.054 **** 2021-02-10T11:16:40.843 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-10T11:16:40.845 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-10T11:16:40.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.050) 0:05:37.104 **** 2021-02-10T11:16:40.885 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-10T11:16:40.886 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.923 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.044) 0:05:37.149 **** 2021-02-10T11:16:40.929 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-10T11:16:40.931 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:40.964 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:40 +0000 (0:00:00.041) 0:05:37.191 **** 2021-02-10T11:16:40.970 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-10T11:16:40.971 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.007 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:37.233 **** 2021-02-10T11:16:41.013 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-10T11:16:41.013 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.048 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:37.275 **** 2021-02-10T11:16:41.055 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T11:16:41.090 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:37.317 **** 2021-02-10T11:16:41.097 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-10T11:16:41.098 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.132 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:37.359 **** 2021-02-10T11:16:41.139 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-10T11:16:41.140 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.174 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:37.401 **** 2021-02-10T11:16:41.181 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-10T11:16:41.182 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.217 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:37.444 **** 2021-02-10T11:16:41.224 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T11:16:41.225 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.260 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.043) 0:05:37.487 **** 2021-02-10T11:16:41.266 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-10T11:16:41.268 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.301 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:37.528 **** 2021-02-10T11:16:41.307 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T11:16:41.309 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.345 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.043) 0:05:37.572 **** 2021-02-10T11:16:41.351 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-10T11:16:41.352 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.385 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.040) 0:05:37.612 **** 2021-02-10T11:16:41.392 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T11:16:41.393 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.427 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:37.654 **** 2021-02-10T11:16:41.434 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-10T11:16:41.471 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.043) 0:05:37.698 **** 2021-02-10T11:16:41.478 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T11:16:41.516 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.044) 0:05:37.742 **** 2021-02-10T11:16:41.522 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-02-10T11:16:41.522 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.560 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.044) 0:05:37.787 **** 2021-02-10T11:16:41.567 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T11:16:41.569 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.603 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:37.830 **** 2021-02-10T11:16:41.610 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-10T11:16:41.611 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.647 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.044) 0:05:37.874 **** 2021-02-10T11:16:41.654 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T11:16:41.655 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.689 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:37.916 **** 2021-02-10T11:16:41.695 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-10T11:16:41.697 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.707 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.708 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.742 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.052) 0:05:37.969 **** 2021-02-10T11:16:41.779 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.037) 0:05:38.006 **** 2021-02-10T11:16:41.786 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-10T11:16:41.787 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.821 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.041) 0:05:38.048 **** 2021-02-10T11:16:41.827 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:41.829 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.883 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.061) 0:05:38.110 **** 2021-02-10T11:16:41.892 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-10T11:16:41.926 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.042) 0:05:38.153 **** 2021-02-10T11:16:41.933 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T11:16:41.935 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:41.970 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:41 +0000 (0:00:00.044) 0:05:38.197 **** 2021-02-10T11:16:41.976 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-10T11:16:41.978 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.018 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.047) 0:05:38.245 **** 2021-02-10T11:16:42.025 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:16:42.026 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.059 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.286 **** 2021-02-10T11:16:42.065 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-10T11:16:42.067 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.102 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.042) 0:05:38.329 **** 2021-02-10T11:16:42.109 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.145 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.043) 0:05:38.372 **** 2021-02-10T11:16:42.152 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-10T11:16:42.153 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.188 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.043) 0:05:38.415 **** 2021-02-10T11:16:42.196 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T11:16:42.196 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.231 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.042) 0:05:38.458 **** 2021-02-10T11:16:42.237 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:42.239 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.271 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:38.498 **** 2021-02-10T11:16:42.278 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T11:16:42.279 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.316 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.044) 0:05:38.542 **** 2021-02-10T11:16:42.325 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-10T11:16:42.327 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.360 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.043) 0:05:38.586 **** 2021-02-10T11:16:42.366 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T11:16:42.367 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.401 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.628 **** 2021-02-10T11:16:42.408 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:16:42.409 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.443 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.670 **** 2021-02-10T11:16:42.451 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T11:16:42.452 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.486 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.043) 0:05:38.713 **** 2021-02-10T11:16:42.493 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-10T11:16:42.494 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.528 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.755 **** 2021-02-10T11:16:42.535 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T11:16:42.536 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.570 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.042) 0:05:38.797 **** 2021-02-10T11:16:42.577 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-10T11:16:42.577 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.611 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.838 **** 2021-02-10T11:16:42.617 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T11:16:42.618 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.660 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.049) 0:05:38.887 **** 2021-02-10T11:16:42.666 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-10T11:16:42.668 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.700 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:38.927 **** 2021-02-10T11:16:42.707 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T11:16:42.707 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.742 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:38.968 **** 2021-02-10T11:16:42.748 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-10T11:16:42.749 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.782 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:39.009 **** 2021-02-10T11:16:42.788 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:16:42.789 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.823 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:39.050 **** 2021-02-10T11:16:42.829 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-02-10T11:16:42.830 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.864 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.041) 0:05:39.091 **** 2021-02-10T11:16:42.871 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:16:42.872 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.905 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:39.132 **** 2021-02-10T11:16:42.911 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-02-10T11:16:42.912 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.946 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.040) 0:05:39.173 **** 2021-02-10T11:16:42.952 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:16:42.953 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:42.986 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:42 +0000 (0:00:00.039) 0:05:39.213 **** 2021-02-10T11:16:42.992 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-10T11:16:42.993 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.026 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.253 **** 2021-02-10T11:16:43.032 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:16:43.034 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.067 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.293 **** 2021-02-10T11:16:43.073 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-10T11:16:43.074 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.110 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.043) 0:05:39.336 **** 2021-02-10T11:16:43.116 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:16:43.118 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.152 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.042) 0:05:39.379 **** 2021-02-10T11:16:43.160 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-10T11:16:43.160 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.195 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.043) 0:05:39.422 **** 2021-02-10T11:16:43.203 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:16:43.204 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.238 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.042) 0:05:39.464 **** 2021-02-10T11:16:43.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-10T11:16:43.246 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.280 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.042) 0:05:39.507 **** 2021-02-10T11:16:43.286 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T11:16:43.287 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.321 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.547 **** 2021-02-10T11:16:43.327 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-10T11:16:43.328 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.363 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.042) 0:05:39.590 **** 2021-02-10T11:16:43.370 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:16:43.371 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.420 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.056) 0:05:39.647 **** 2021-02-10T11:16:43.427 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-10T11:16:43.429 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.461 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.688 **** 2021-02-10T11:16:43.466 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:16:43.468 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.504 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.042) 0:05:39.730 **** 2021-02-10T11:16:43.508 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-10T11:16:43.510 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.542 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.038) 0:05:39.769 **** 2021-02-10T11:16:43.548 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:16:43.549 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.581 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.038) 0:05:39.808 **** 2021-02-10T11:16:43.586 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-10T11:16:43.588 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.620 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.038) 0:05:39.847 **** 2021-02-10T11:16:43.628 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:16:43.629 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.665 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.044) 0:05:39.891 **** 2021-02-10T11:16:43.670 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-10T11:16:43.672 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.705 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.932 **** 2021-02-10T11:16:43.711 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:43.713 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.746 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.040) 0:05:39.973 **** 2021-02-10T11:16:43.758 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-10T11:16:43.758 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.797 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.051) 0:05:40.024 **** 2021-02-10T11:16:43.804 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:16:43.804 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.838 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.041) 0:05:40.065 **** 2021-02-10T11:16:43.845 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:16:43.846 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.039) 0:05:40.105 **** 2021-02-10T11:16:43.885 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:16:43.886 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.921 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.043) 0:05:40.148 **** 2021-02-10T11:16:43.927 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-10T11:16:43.928 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:43.960 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.039) 0:05:40.187 **** 2021-02-10T11:16:43.966 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:43.967 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.000 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:43 +0000 (0:00:00.039) 0:05:40.227 **** 2021-02-10T11:16:44.006 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-10T11:16:44.008 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.041 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.040) 0:05:40.268 **** 2021-02-10T11:16:44.048 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:16:44.049 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.080 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.039) 0:05:40.307 **** 2021-02-10T11:16:44.085 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-10T11:16:44.087 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.121 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.040) 0:05:40.348 **** 2021-02-10T11:16:44.150 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:16:44.151 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.182 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.060) 0:05:40.409 **** 2021-02-10T11:16:44.211 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:16:44.212 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.243 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.060) 0:05:40.470 **** 2021-02-10T11:16:44.270 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:44.272 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.303 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.059) 0:05:40.530 **** 2021-02-10T11:16:44.350 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:44.387 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:44 +0000 (0:00:00.084) 0:05:40.614 **** 2021-02-10T11:16:46.003 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:16:46.005 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T11:16:46.039 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:01.651) 0:05:42.266 **** 2021-02-10T11:16:46.323 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "secontext": "system_u:object_r:exports_t:s0", "size": 578, "state": "file", "uid": 0} 2021-02-10T11:16:46.359 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.319) 0:05:42.586 **** 2021-02-10T11:16:46.571 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:16:46.571 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ActiveEnterTimestampMonotonic": "20115796", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice -.mount systemd-journald.socket nfs-mountd.service tmp.mount nfs-config.service nfs-idmapd.service local-fs.target rpc-statd.service rpc-gssd.service proc-fs-nfsd.mount rpcbind.socket gssproxy.service network-online.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:09:44 UTC", "AssertTimestampMonotonic": "19952829", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-mountd.service nfs-idmapd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ConditionTimestampMonotonic": "19952828", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ExecMainExitTimestampMonotonic": "20093211", "ExecMainPID": "1252", "ExecMainStartTimestamp": "Wed 2021-02-10 11:09:44 UTC", "ExecMainStartTimestampMonotonic": "19961688", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T11:16:46.573 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:09:44 UTC", "InactiveExitTimestampMonotonic": "19953915", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T11:16:46.574 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "system.slice -.mount network.target proc-fs-nfsd.mount nfs-mountd.service", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "rpc-statd.service nfs-config.service network-online.target auth-rpcgss-module.service rpc-statd-notify.service rpcbind.socket nfs-idmapd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:16:46.613 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.253) 0:05:42.839 **** 2021-02-10T11:16:46.639 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:16:46.641 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:46.679 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.066) 0:05:42.906 **** 2021-02-10T11:16:46.706 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:16:46.707 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:46.740 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.060) 0:05:42.967 **** 2021-02-10T11:16:46.778 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:46.779 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:46.813 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.073) 0:05:43.040 **** 2021-02-10T11:16:46.842 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:46.879 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.065) 0:05:43.106 **** 2021-02-10T11:16:46.886 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:16:46.889 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nfs-server", "status": {"ActiveEnterTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ActiveEnterTimestampMonotonic": "19487419", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket rpc-gssd.service network-online.target nfs-config.service local-fs.target -.mount system.slice gssproxy.service rpc-statd.service nfs-idmapd.service rpcbind.socket proc-fs-nfsd.mount tmp.mount nfs-mountd.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-02-10 11:10:05 UTC", "AssertTimestampMonotonic": "19354730", "Before": "rpc-statd-notify.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BoundBy": "nfs-idmapd.service nfs-mountd.service", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ConditionTimestampMonotonic": "19354730", "ControlGroup": "/system.slice/nfs-server.service", "ControlPID": "0", "DefaultDependencies": "no", "Delegate": "no", "Description": "NFS server and services", "DevicePolicy": "auto", "DropInPaths": "/run/systemd/generator/nfs-server.service.d/order-with-mounts.conf", "EnvironmentFile": "/run/sysconfig/nfs-utils (ignore_errors=yes)", "ExecMainCode": "1", "ExecMainExitTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ExecMainExitTimestampMonotonic": "19477271", "ExecMainPID": "1460", "ExecMainStartTimestamp": "Wed 2021-02-10 11:10:05 UTC", "ExecMainStartTimestampMonotonic": "19361318", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes 2021-02-10T11:16:46.891 INFO:teuthology.task.ansible.out: ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd $RPCNFSDARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/bin/sh ; argv[]=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -r ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/rpc.nfsd ; argv[]=/usr/sbin/rpc.nfsd 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/usr/sbin/exportfs ; argv[]=/usr/sbin/exportfs -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nfs-server.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nfs-server.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-02-10 11:10:05 UTC", "InactiveExitTimestampMonotonic": "19355147", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "lo 2021-02-10T11:16:46.893 INFO:teuthology.task.ansible.out:aded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nfs-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "yes", "Requires": "nfs-mountd.service proc-fs-nfsd.mount -.mount system.slice network.target", "RequiresMountsFor": "/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "exited", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "0", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "oneshot", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "network-online.target rpc-statd-notify.service rpcbind.socket auth-rpcgss-module.service nfs-config.service rpc-statd.service nfs-idmapd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:16:46.924 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.044) 0:05:43.151 **** 2021-02-10T11:16:46.931 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:46.967 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:46 +0000 (0:00:00.043) 0:05:43.194 **** 2021-02-10T11:16:46.974 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-10T11:16:46.975 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.010 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.042) 0:05:43.236 **** 2021-02-10T11:16:47.016 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:47.018 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.053 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.043) 0:05:43.280 **** 2021-02-10T11:16:47.059 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:16:47.061 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.094 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.321 **** 2021-02-10T11:16:47.101 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:16:47.102 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.136 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.363 **** 2021-02-10T11:16:47.143 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:47.144 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.177 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.404 **** 2021-02-10T11:16:47.184 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T11:16:47.185 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.218 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.445 **** 2021-02-10T11:16:47.225 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:16:47.226 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.259 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.040) 0:05:43.485 **** 2021-02-10T11:16:47.266 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T11:16:47.267 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.305 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.046) 0:05:43.532 **** 2021-02-10T11:16:47.311 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-10T11:16:47.313 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.347 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.042) 0:05:43.574 **** 2021-02-10T11:16:47.354 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:16:47.356 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.403 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.056) 0:05:43.630 **** 2021-02-10T11:16:47.413 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-10T11:16:47.415 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.453 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.049) 0:05:43.680 **** 2021-02-10T11:16:47.460 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.499 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.045) 0:05:43.725 **** 2021-02-10T11:16:47.506 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-10T11:16:47.507 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.539 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.040) 0:05:43.766 **** 2021-02-10T11:16:47.547 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-10T11:16:47.548 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.581 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.808 **** 2021-02-10T11:16:47.587 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-10T11:16:47.589 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.622 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:43.849 **** 2021-02-10T11:16:47.629 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T11:16:47.630 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.663 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.040) 0:05:43.890 **** 2021-02-10T11:16:47.669 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-10T11:16:47.671 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.725 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.061) 0:05:43.952 **** 2021-02-10T11:16:47.735 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T11:16:47.736 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.774 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.048) 0:05:44.000 **** 2021-02-10T11:16:47.780 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-10T11:16:47.781 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.817 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.043) 0:05:44.044 **** 2021-02-10T11:16:47.823 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-10T11:16:47.824 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.857 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.040) 0:05:44.084 **** 2021-02-10T11:16:47.863 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-10T11:16:47.863 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.897 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.039) 0:05:44.124 **** 2021-02-10T11:16:47.903 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-10T11:16:47.905 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.938 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.041) 0:05:44.165 **** 2021-02-10T11:16:47.944 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-10T11:16:47.946 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:47.978 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:47 +0000 (0:00:00.039) 0:05:44.205 **** 2021-02-10T11:16:47.988 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-10T11:16:47.990 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.022 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.044) 0:05:44.249 **** 2021-02-10T11:16:48.029 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-10T11:16:48.030 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.064 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.041) 0:05:44.290 **** 2021-02-10T11:16:48.071 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T11:16:48.072 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.105 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.041) 0:05:44.332 **** 2021-02-10T11:16:48.112 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-10T11:16:48.113 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.146 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.040) 0:05:44.373 **** 2021-02-10T11:16:48.153 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-10T11:16:48.154 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.186 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.039) 0:05:44.413 **** 2021-02-10T11:16:48.194 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-10T11:16:48.195 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.227 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.041) 0:05:44.454 **** 2021-02-10T11:16:48.234 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-10T11:16:48.235 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.269 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.041) 0:05:44.496 **** 2021-02-10T11:16:48.276 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-10T11:16:48.277 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.310 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.041) 0:05:44.537 **** 2021-02-10T11:16:48.316 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:16:48.317 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T11:16:48.351 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.040) 0:05:44.578 **** 2021-02-10T11:16:48.357 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-10T11:16:48.358 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.392 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.040) 0:05:44.618 **** 2021-02-10T11:16:48.403 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T11:16:48.404 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [gibba025.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T11:16:48.415 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-10T11:16:48.415 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.416 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [gibba025.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.448 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.056) 0:05:44.675 **** 2021-02-10T11:16:48.457 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.458 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.460 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.470 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.471 INFO:teuthology.task.ansible.out: 2021-02-10T11:16:48.473 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.506 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.057) 0:05:44.732 **** 2021-02-10T11:16:48.544 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.038) 0:05:44.771 **** 2021-02-10T11:16:48.551 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-10T11:16:48.552 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.587 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.043) 0:05:44.814 **** 2021-02-10T11:16:48.629 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:16:48.631 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-10T11:16:48.662 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.074) 0:05:44.889 **** 2021-02-10T11:16:48.712 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-10T11:16:48.713 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.724 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-10T11:16:48.735 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.736 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.747 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.758 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.759 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.799 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.136) 0:05:45.025 **** 2021-02-10T11:16:48.809 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-10T11:16:48.810 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003217", "end": "2021-02-10 11:16:48.742326", "rc": 0, "start": "2021-02-10 11:16:48.739109", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T11:16:48.848 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.049) 0:05:45.075 **** 2021-02-10T11:16:48.875 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T11:16:48.876 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.912 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.063) 0:05:45.139 **** 2021-02-10T11:16:48.949 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-10T11:16:48.950 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:48.984 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:48 +0000 (0:00:00.072) 0:05:45.211 **** 2021-02-10T11:16:48.990 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-10T11:16:48.991 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003087", "end": "2021-02-10 11:16:48.965132", "rc": 0, "start": "2021-02-10 11:16:48.962045", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology-lock"]} 2021-02-10T11:16:49.029 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.044) 0:05:45.255 **** 2021-02-10T11:16:49.035 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T11:16:49.036 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.069 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.040) 0:05:45.296 **** 2021-02-10T11:16:49.074 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-10T11:16:49.076 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.043) 0:05:45.339 **** 2021-02-10T11:16:49.119 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-10T11:16:49.120 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.154 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.041) 0:05:45.381 **** 2021-02-10T11:16:49.159 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-10T11:16:49.160 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.195 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.041) 0:05:45.422 **** 2021-02-10T11:16:49.223 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-02-10T11:16:49.224 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.255 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.060) 0:05:45.482 **** 2021-02-10T11:16:49.282 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-10T11:16:49.283 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:49.315 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:49 +0000 (0:00:00.060) 0:05:45.542 **** 2021-02-10T11:16:50.025 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-10T11:16:50.027 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} 2021-02-10T11:16:50.060 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.744) 0:05:46.286 **** 2021-02-10T11:16:50.597 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [gibba025.front.sepia.ceph.com] => (item={'block_used': 13742188, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582296, 'size_available': 928081084416, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582296, "block_size": 4096, "block_total": 240324484, "block_used": 13742188, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081084416, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.086) 0:05:46.372 **** TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.170-6.el7.x86_64 providing device-mapper is already installed"]} Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.054) 0:05:46.427 **** TASK [testnode : Set root disk] ************************************************ ok: [gibba017.front.sepia.ceph.com] => (item={'block_used': 13742198, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582286, 'size_available': 928081043456, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_availa 2021-02-10T11:16:50.599 INFO:teuthology.task.ansible.out:ble': 60964811}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582286, "block_size": 4096, "block_total": 240324484, "block_used": 13742198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081043456, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}} Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.083) 0:05:46.511 **** TASK [testnode : Compile list of non-root partitions] ************************** changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004437", "end": "2021-02-10 11:16:50.582102", "rc": 0, "start": "2021-02-10 11:16:50.577665", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:16:50.632 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.347) 0:05:46.859 **** 2021-02-10T11:16:50.670 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T11:16:50.671 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'block_used': 13742188, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582296, 'size_available': 928081084416, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582296, "block_size": 4096, "block_total": 240324484, "block_used": 13742188, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081084416, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:50.713 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.081) 0:05:46.940 **** 2021-02-10T11:16:50.740 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-10T11:16:50.741 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004411", "end": "2021-02-10 11:16:50.720291", "rc": 0, "start": "2021-02-10 11:16:50.715880", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T11:16:50.775 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.062) 0:05:47.002 **** 2021-02-10T11:16:50.809 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.033) 0:05:47.036 **** 2021-02-10T11:16:50.816 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-10T11:16:50.817 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'block_used': 13742198, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582286, 'size_available': 928081043456, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582286, "block_size": 4096, "block_total": 240324484, "block_used": 13742198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081043456, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:16:50.856 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.046) 0:05:47.082 **** 2021-02-10T11:16:50.865 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T11:16:50.865 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF', 'wwn-0x50000399aba81668'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF-part1', 'wwn-0x50000399aba81668-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399aba81668', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I8K38IF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF", "wwn-0x50000399aba81668"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF-part1", "wwn-0x50000399aba81668-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I8K38IF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399aba81668"}}, "skip_reason": "Conditiona 2021-02-10T11:16:50.867 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:16:50.878 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-10T11:16:50.911 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:50 +0000 (0:00:00.056) 0:05:47.138 **** 2021-02-10T11:16:50.962 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-10T11:16:50.964 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF', 'wwn-0x50000399ab9813d2'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF-part1', 'wwn-0x50000399ab9813d2-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab9813d2', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I6K2K4F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF", "wwn-0x50000399ab9813d2"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF-part1", "wwn-0x50000399ab9813d2-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I6K2K4F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab9813d2"}}, "skip_reason": "Conditiona 2021-02-10T11:16:50.965 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:16:52.437 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-9nNRfG-vKQD-3rBy-479l-RldV-A1Wm-Ii5GDP', 'nvme-INTEL_SSDPEL1K375GA_PHKM92000092375A', 'nvme-nvme.8086-50484b4d393230303030393233373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM92000092375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.156329", "end": "2021-02-10 11:16:52.418380", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-9nNRfG-vKQD-3rBy-479l-RldV-A1Wm-Ii5GDP", "nvme-INTEL_SSDPEL1K375GA_PHKM92000092375A", "nvme-nvme.8086-50484b4d393230303030393233373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:51.262051", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T11:16:52.438 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:52.511 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YVwNhA-Z3eZ-2zeO-0puS-8hn4-60q9-egYyQZ', 'nvme-INTEL_SSDPEL1K375GA_PHKM913400KY375A', 'nvme-nvme.8086-50484b4d3931333430304b5933373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM913400KY375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.122439", "end": "2021-02-10 11:16:52.494247", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-YVwNhA-Z3eZ-2zeO-0puS-8hn4-60q9-egYyQZ", "nvme-INTEL_SSDPEL1K375GA_PHKM913400KY375A", "nvme-nvme.8086-50484b4d3931333430304b5933373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:51.371808", "stderr": "", "stderr_lines": [], "stdout": "Creating new 2021-02-10T11:16:52.513 INFO:teuthology.task.ansible.out:GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:53.932 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nn5kwPGHPOICSk0HQWsybbWrZoRxYUYvS'], 'uuids': ['272238a8-9846-442b-bdcb-8e22c68885b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.055098", "end": "2021-02-10 11:16:53.910275", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nn5kwPGHPOICSk0HQWsybbWrZoRxYUYvS"], "labels": [], "masters": [], "uuids": ["272238a8-9846-442b-bdcb-8e22c68885b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:52.855177", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:54.006 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYejf0wNuo30rBWjzThpXji5kyu03VU8J3l'], 'uuids': ['bb2425ea-1250-4737-813a-0ea7a523c0c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.059203", "end": "2021-02-10 11:16:53.989123", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYejf0wNuo30rBWjzThpXji5kyu03VU8J3l"], "labels": [], "masters": [], "uuids": ["bb2425ea-1250-4737-813a-0ea7a523c0c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:52.929920", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:55.417 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NrC1Z0X4Qlh5X370EURcrRXIZIZSnjEYJ'], 'uuids': ['f61fb47f-c2ba-4bdc-bf28-6247fda04769']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.054697", "end": "2021-02-10 11:16:55.401497", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NrC1Z0X4Qlh5X370EURcrRXIZIZSnjEYJ"], "labels": [], "masters": [], "uuids": ["f61fb47f-c2ba-4bdc-bf28-6247fda04769"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:54.346800", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:55.503 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYerLEVFCyK0bO3d0hSMwJlCX3vpxkocYk0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.053516", "end": "2021-02-10 11:16:55.479543", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYerLEVFCyK0bO3d0hSMwJlCX3vpxkocYk0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:54.426027", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:56.929 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nir2M3BaOSFxJwnm1AzF6wXt4QjYQ2pMO'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.062823", "end": "2021-02-10 11:16:56.910880", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nir2M3BaOSFxJwnm1AzF6wXt4QjYQ2pMO"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:55.848057", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:56.988 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeavujAckkewKvjBQeeNFGD3flLHJxpfky'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.052004", "end": "2021-02-10 11:16:56.972821", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeavujAckkewKvjBQeeNFGD3flLHJxpfky"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:55.920817", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:58.426 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NPBGlnBt1WnA9FkyOPCqcaIwKTVw5J5w0'], 'uuids': ['33ebd65a-2a92-4559-9352-443b50896445']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.056526", "end": "2021-02-10 11:16:58.404714", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NPBGlnBt1WnA9FkyOPCqcaIwKTVw5J5w0"], "labels": [], "masters": [], "uuids": ["33ebd65a-2a92-4559-9352-443b50896445"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:57.348188", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:58.480 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYem2rye3FdQH7HdVtBZIYahF0Du91XXvAY'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.057172", "end": "2021-02-10 11:16:58.465223", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYem2rye3FdQH7HdVtBZIYahF0Du91XXvAY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:57.408051", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:59.927 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NZ9Kfu3x5AN7RgZm1n9fF30p9owz9s4tm'], 'uuids': ['0f98a2f7-550a-49f6-8cf7-782db44a38db']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.065579", "end": "2021-02-10 11:16:59.904667", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NZ9Kfu3x5AN7RgZm1n9fF30p9owz9s4tm"], "labels": [], "masters": [], "uuids": ["0f98a2f7-550a-49f6-8cf7-782db44a38db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:58.839088", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:16:59.966 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:16:59 +0000 (0:00:09.054) 0:05:56.193 **** 2021-02-10T11:16:59.974 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeVR1idtM2y929CqWzrvPijKHzVFPts9de'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.052027", "end": "2021-02-10 11:16:59.948697", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeVR1idtM2y929CqWzrvPijKHzVFPts9de"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:16:58.896670", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-10T11:17:00.027 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:00 +0000 (0:00:00.060) 0:05:56.254 **** 2021-02-10T11:17:00.034 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-10T11:17:00.035 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF', 'wwn-0x50000399aba81668'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF-part1', 'wwn-0x50000399aba81668-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399aba81668', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I8K38IF6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF", "wwn-0x50000399aba81668"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I8K38IF6XF-part1", "wwn-0x50000399aba81668-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I8K38IF6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399aba81668"}}, "skip_reason": "Conditiona 2021-02-10T11:17:00.036 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:17:00.068 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF', 'wwn-0x50000399ab9813d2'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'links': {'masters': [], 'labels': [], 'ids': ['ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF-part1', 'wwn-0x50000399ab9813d2-part1'], 'uuids': ['aeb0602f-c6fc-4ca1-8920-0340080ffa5a']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x50000399ab9813d2', 'model': 'TOSHIBA MG04ACA1', 'serial': 'Y9I6K2K4F6XF', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)", "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF", "wwn-0x50000399ab9813d2"], "labels": [], "masters": [], "uuids": []}, "model": "TOSHIBA MG04ACA1", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-TOSHIBA_MG04ACA100N_Y9I6K2K4F6XF-part1", "wwn-0x50000399ab9813d2-part1"], "labels": [], "masters": [], "uuids": ["aeb0602f-c6fc-4ca1-8920-0340080ffa5a"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Y9I6K2K4F6XF", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x50000399ab9813d2"}}, "skip_reason": "Conditiona 2021-02-10T11:17:00.070 INFO:teuthology.task.ansible.out:l result was False"} 2021-02-10T11:17:00.467 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-9nNRfG-vKQD-3rBy-479l-RldV-A1Wm-Ii5GDP', 'nvme-INTEL_SSDPEL1K375GA_PHKM92000092375A', 'nvme-nvme.8086-50484b4d393230303030393233373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM92000092375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.026478", "end": "2021-02-10 11:17:00.444941", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-9nNRfG-vKQD-3rBy-479l-RldV-A1Wm-Ii5GDP", "nvme-INTEL_SSDPEL1K375GA_PHKM92000092375A", "nvme-nvme.8086-50484b4d393230303030393233373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:00.418463", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T11:17:00.468 INFO:teuthology.task.ansible.out:copied, 0.0243229 s, 4.7 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0243229 s, 4.7 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:00.531 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '732585168', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YVwNhA-Z3eZ-2zeO-0puS-8hn4-60q9-egYyQZ', 'nvme-INTEL_SSDPEL1K375GA_PHKM913400KY375A', 'nvme-nvme.8086-50484b4d3931333430304b5933373541-494e54454c2053534450454c314b3337354741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': ['vg_nvme-lv_1', 'vg_nvme-lv_2', 'vg_nvme-lv_3', 'vg_nvme-lv_4', 'vg_nvme-lv_5'], 'partitions': {}, 'model': 'INTEL SSDPEL1K375GA', 'serial': 'PHKM913400KY375A', 'size': '349.32 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.026240", "end": "2021-02-10 11:17:00.508078", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_1", "vg_nvme-lv_2", "vg_nvme-lv_3", "vg_nvme-lv_4", "vg_nvme-lv_5"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [Optane]", "links": {"ids": ["lvm-pv-uuid-YVwNhA-Z3eZ-2zeO-0puS-8hn4-60q9-egYyQZ", "nvme-INTEL_SSDPEL1K375GA_PHKM913400KY375A", "nvme-nvme.8086-50484b4d3931333430304b5933373541-494e54454c2053534450454c314b3337354741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEL1K375GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "732585168", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "349.32 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:00.481838", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) 2021-02-10T11:17:00.533 INFO:teuthology.task.ansible.out:copied, 0.0241218 s, 4.8 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0241218 s, 4.8 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:00.979 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nn5kwPGHPOICSk0HQWsybbWrZoRxYUYvS'], 'uuids': ['272238a8-9846-442b-bdcb-8e22c68885b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.083636", "end": "2021-02-10 11:17:00.957162", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nn5kwPGHPOICSk0HQWsybbWrZoRxYUYvS"], "labels": [], "masters": [], "uuids": ["272238a8-9846-442b-bdcb-8e22c68885b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:00.873526", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0815603 s, 1.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0815603 s, 1.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:01.043 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '27262976', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYejf0wNuo30rBWjzThpXji5kyu03VU8J3l'], 'uuids': ['bb2425ea-1250-4737-813a-0ea7a523c0c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '13.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.080055", "end": "2021-02-10 11:17:01.022874", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYejf0wNuo30rBWjzThpXji5kyu03VU8J3l"], "labels": [], "masters": [], "uuids": ["bb2425ea-1250-4737-813a-0ea7a523c0c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "27262976", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "13.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:00.942819", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0779426 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0779426 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:01.492 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NrC1Z0X4Qlh5X370EURcrRXIZIZSnjEYJ'], 'uuids': ['f61fb47f-c2ba-4bdc-bf28-6247fda04769']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.081051", "end": "2021-02-10 11:17:01.473621", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NrC1Z0X4Qlh5X370EURcrRXIZIZSnjEYJ"], "labels": [], "masters": [], "uuids": ["f61fb47f-c2ba-4bdc-bf28-6247fda04769"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:01.392570", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0789614 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0789614 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:01.556 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYerLEVFCyK0bO3d0hSMwJlCX3vpxkocYk0'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.081230", "end": "2021-02-10 11:17:01.538106", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYerLEVFCyK0bO3d0hSMwJlCX3vpxkocYk0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:01.456876", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0791382 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0791382 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:01.994 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nir2M3BaOSFxJwnm1AzF6wXt4QjYQ2pMO'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.069851", "end": "2021-02-10 11:17:01.974350", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0Nir2M3BaOSFxJwnm1AzF6wXt4QjYQ2pMO"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:01.904499", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0678525 s, 1.7 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0678525 s, 1.7 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:02.049 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeavujAckkewKvjBQeeNFGD3flLHJxpfky'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.060091", "end": "2021-02-10 11:17:02.031234", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeavujAckkewKvjBQeeNFGD3flLHJxpfky"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:01.971143", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0580378 s, 2.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0580378 s, 2.0 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:02.508 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NPBGlnBt1WnA9FkyOPCqcaIwKTVw5J5w0'], 'uuids': ['33ebd65a-2a92-4559-9352-443b50896445']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.079668", "end": "2021-02-10 11:17:02.484868", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NPBGlnBt1WnA9FkyOPCqcaIwKTVw5J5w0"], "labels": [], "masters": [], "uuids": ["33ebd65a-2a92-4559-9352-443b50896445"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:02.405200", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0776192 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0776192 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:02.561 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYem2rye3FdQH7HdVtBZIYahF0Du91XXvAY'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.080741", "end": "2021-02-10 11:17:02.539743", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYem2rye3FdQH7HdVtBZIYahF0Du91XXvAY"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:02.459002", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0786343 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0786343 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:03.019 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NZ9Kfu3x5AN7RgZm1n9fF30p9owz9s4tm'], 'uuids': ['0f98a2f7-550a-49f6-8cf7-782db44a38db']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM92000092375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.081055", "end": "2021-02-10 11:17:03.000096", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PbLOVP9Yfv4cfnKdjCZWEbJoeawwBS0NZ9Kfu3x5AN7RgZm1n9fF30p9owz9s4tm"], "labels": [], "masters": [], "uuids": ["0f98a2f7-550a-49f6-8cf7-782db44a38db"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM92000092375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:02.919041", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.078955 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.078955 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:03.060 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:03 +0000 (0:00:03.032) 0:05:59.287 **** 2021-02-10T11:17:03.077 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '176160768', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeVR1idtM2y929CqWzrvPijKHzVFPts9de'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'holders': [], 'partitions': {}, 'model': None, 'serial': 'PHKM913400KY375A', 'size': '84.00 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.080925", "end": "2021-02-10 11:17:03.053262", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-bZsQ5Jd73LB5tUgmEKFG0FB3mvFphbYeVR1idtM2y929CqWzrvPijKHzVFPts9de"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "176160768", "sectorsize": "512", "serial": "PHKM913400KY375A", "size": "84.00 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-10 11:17:02.972337", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0788403 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0788403 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:03.118 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:03 +0000 (0:00:00.058) 0:05:59.345 **** 2021-02-10T11:17:03.614 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-10T11:17:03.616 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.104140", "end": "2021-02-10 11:17:03.587995", "rc": 0, "start": "2021-02-10 11:17:03.483855", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:03.650 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:03 +0000 (0:00:00.531) 0:05:59.877 **** 2021-02-10T11:17:03.661 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.082479", "end": "2021-02-10 11:17:03.627711", "rc": 0, "start": "2021-02-10 11:17:03.545232", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:03.696 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:03 +0000 (0:00:00.046) 0:05:59.923 **** 2021-02-10T11:17:04.139 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-10T11:17:04.140 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043373", "end": "2021-02-10 11:17:04.122813", "rc": 0, "start": "2021-02-10 11:17:04.079440", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:04.173 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.477) 0:06:00.400 **** 2021-02-10T11:17:04.180 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.027954", "end": "2021-02-10 11:17:04.151800", "rc": 0, "start": "2021-02-10 11:17:04.123846", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:04.215 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.041) 0:06:00.442 **** 2021-02-10T11:17:04.221 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-10T11:17:04.257 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.041) 0:06:00.484 **** 2021-02-10T11:17:04.297 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.040) 0:06:00.524 **** 2021-02-10T11:17:04.303 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-02-10T11:17:04.305 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.340 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.043) 0:06:00.567 **** 2021-02-10T11:17:04.346 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.383 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.042) 0:06:00.610 **** 2021-02-10T11:17:04.389 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-10T11:17:04.391 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.424 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.041) 0:06:00.651 **** 2021-02-10T11:17:04.431 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.465 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.040) 0:06:00.691 **** 2021-02-10T11:17:04.470 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:17:04.471 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.046) 0:06:00.738 **** 2021-02-10T11:17:04.517 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.555 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.043) 0:06:00.781 **** 2021-02-10T11:17:04.561 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-10T11:17:04.563 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.595 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.040) 0:06:00.822 **** 2021-02-10T11:17:04.601 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:04.638 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:04 +0000 (0:00:00.042) 0:06:00.865 **** 2021-02-10T11:17:05.036 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-10T11:17:05.038 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:17:05.073 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.434) 0:06:01.300 **** 2021-02-10T11:17:05.090 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-10T11:17:05.125 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.052) 0:06:01.352 **** 2021-02-10T11:17:05.133 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-10T11:17:05.135 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'block_used': 13742188, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582296, 'size_available': 928081084416, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582296, "block_size": 4096, "block_total": 240324484, "block_used": 13742188, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081084416, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.171 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.045) 0:06:01.398 **** 2021-02-10T11:17:05.181 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'block_used': 13742198, 'uuid': 'aeb0602f-c6fc-4ca1-8920-0340080ffa5a', 'size_total': 984369086464, 'block_total': 240324484, 'mount': '/', 'block_available': 226582286, 'size_available': 928081043456, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,seclabel,relatime,data=ordered', 'device': '/dev/sda1', 'inode_used': 90165, 'block_size': 4096, 'inode_available': 60964811}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226582286, "block_size": 4096, "block_total": 240324484, "block_used": 13742198, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60964811, "inode_total": 61054976, "inode_used": 90165, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 928081043456, "size_total": 984369086464, "uuid": "aeb0602f-c6fc-4ca1-8920-0340080ffa5a"}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.223 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.052) 0:06:01.450 **** 2021-02-10T11:17:05.231 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-10T11:17:05.233 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.268 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.045) 0:06:01.495 **** 2021-02-10T11:17:05.276 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.313 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.044) 0:06:01.540 **** 2021-02-10T11:17:05.321 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:05.322 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.356 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.042) 0:06:01.583 **** 2021-02-10T11:17:05.364 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.398 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.041) 0:06:01.625 **** 2021-02-10T11:17:05.405 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:05.406 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.441 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.043) 0:06:01.668 **** 2021-02-10T11:17:05.447 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.482 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.041) 0:06:01.709 **** 2021-02-10T11:17:05.488 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-10T11:17:05.490 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.526 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.043) 0:06:01.753 **** 2021-02-10T11:17:05.533 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.573 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.047) 0:06:01.800 **** 2021-02-10T11:17:05.582 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-02-10T11:17:05.583 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.618 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.044) 0:06:01.845 **** 2021-02-10T11:17:05.625 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.673 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.054) 0:06:01.899 **** 2021-02-10T11:17:05.681 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-10T11:17:05.682 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.718 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.045) 0:06:01.944 **** 2021-02-10T11:17:05.724 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:05.760 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:05 +0000 (0:00:00.042) 0:06:01.987 **** 2021-02-10T11:17:06.559 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-10T11:17:06.561 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T11:17:06.616 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:06 +0000 (0:00:00.855) 0:06:02.843 **** 2021-02-10T11:17:06.628 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-10T11:17:06.678 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:06 +0000 (0:00:00.061) 0:06:02.905 **** 2021-02-10T11:17:07.438 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-10T11:17:07.439 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:07.963 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:07.963 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:08.474 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:08.485 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:08.998 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:09.009 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:09.523 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:09.526 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-10T11:17:09.565 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:09 +0000 (0:00:02.887) 0:06:05.792 **** 2021-02-10T11:17:09.604 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:09 +0000 (0:00:00.038) 0:06:05.831 **** 2021-02-10T11:17:10.009 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-10T11:17:10.010 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T11:17:10.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:10 +0000 (0:00:00.439) 0:06:06.270 **** 2021-02-10T11:17:10.063 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-10T11:17:10.097 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:10 +0000 (0:00:00.053) 0:06:06.324 **** 2021-02-10T11:17:10.511 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-10T11:17:10.512 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:10.565 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:10.942 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:10.995 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:11.373 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:11.437 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:11.813 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:11.825 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:11.863 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:11 +0000 (0:00:01.765) 0:06:08.090 **** 2021-02-10T11:17:11.872 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "84G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-10T11:17:11.873 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "13G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:11.911 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:11 +0000 (0:00:00.047) 0:06:08.137 **** 2021-02-10T11:17:11.917 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-10T11:17:11.918 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:11.955 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:11 +0000 (0:00:00.044) 0:06:08.181 **** 2021-02-10T11:17:11.963 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:12.000 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:11 +0000 (0:00:00.044) 0:06:08.226 **** 2021-02-10T11:17:12.007 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-10T11:17:12.008 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:12.044 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.044) 0:06:08.271 **** 2021-02-10T11:17:12.051 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:12.088 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.043) 0:06:08.314 **** 2021-02-10T11:17:12.492 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-10T11:17:12.493 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:12.526 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.438) 0:06:08.753 **** 2021-02-10T11:17:12.544 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:12.578 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.052) 0:06:08.805 **** 2021-02-10T11:17:12.585 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-10T11:17:12.586 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:12.623 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.044) 0:06:08.849 **** 2021-02-10T11:17:12.630 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:12.665 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:12 +0000 (0:00:00.041) 0:06:08.891 **** 2021-02-10T11:17:13.371 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-10T11:17:13.373 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true} changed: [gibba025.front.sepia.ceph.com] => {"changed": true} 2021-02-10T11:17:13.406 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:13 +0000 (0:00:00.741) 0:06:09.633 **** 2021-02-10T11:17:13.437 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:13 +0000 (0:00:00.031) 0:06:09.664 **** 2021-02-10T11:17:14.188 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-10T11:17:14.189 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-10T11:17:14.223 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.785) 0:06:10.450 **** 2021-02-10T11:17:14.254 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.031) 0:06:10.481 **** 2021-02-10T11:17:14.260 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-10T11:17:14.261 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.294 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.521 **** 2021-02-10T11:17:14.301 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.335 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.561 **** 2021-02-10T11:17:14.340 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-02-10T11:17:14.341 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.374 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.039) 0:06:10.601 **** 2021-02-10T11:17:14.381 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.415 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.641 **** 2021-02-10T11:17:14.420 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:17:14.421 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.454 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.039) 0:06:10.681 **** 2021-02-10T11:17:14.460 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.494 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.721 **** 2021-02-10T11:17:14.501 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-02-10T11:17:14.503 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.536 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.041) 0:06:10.763 **** 2021-02-10T11:17:14.542 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.577 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.041) 0:06:10.804 **** 2021-02-10T11:17:14.583 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-02-10T11:17:14.584 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.617 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.844 **** 2021-02-10T11:17:14.625 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.658 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.040) 0:06:10.884 **** 2021-02-10T11:17:14.663 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-02-10T11:17:14.663 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.697 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.039) 0:06:10.924 **** 2021-02-10T11:17:14.704 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.738 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.041) 0:06:10.965 **** 2021-02-10T11:17:14.743 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-02-10T11:17:14.745 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.782 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.043) 0:06:11.009 **** 2021-02-10T11:17:14.789 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.825 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.042) 0:06:11.052 **** 2021-02-10T11:17:14.832 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-02-10T11:17:14.834 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.869 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.044) 0:06:11.096 **** 2021-02-10T11:17:14.877 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.915 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.045) 0:06:11.142 **** 2021-02-10T11:17:14.922 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-02-10T11:17:14.923 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:14.958 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.042) 0:06:11.185 **** 2021-02-10T11:17:14.965 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:15.001 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:14 +0000 (0:00:00.042) 0:06:11.228 **** 2021-02-10T11:17:15.009 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-10T11:17:15.010 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:15.043 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:15 +0000 (0:00:00.041) 0:06:11.270 **** 2021-02-10T11:17:15.060 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:15.094 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:15 +0000 (0:00:00.051) 0:06:11.321 **** 2021-02-10T11:17:15.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-10T11:17:15.102 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:15.136 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:15 +0000 (0:00:00.041) 0:06:11.363 **** 2021-02-10T11:17:15.143 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:15.179 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:15 +0000 (0:00:00.042) 0:06:11.406 **** 2021-02-10T11:17:15.975 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-10T11:17:15.976 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T11:17:16.011 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.831) 0:06:12.237 **** 2021-02-10T11:17:16.018 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-29.el7_8.2.x86_64 providing ntp is already installed"]} 2021-02-10T11:17:16.053 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.042) 0:06:12.280 **** 2021-02-10T11:17:16.060 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-10T11:17:16.060 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:16.095 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.041) 0:06:12.321 **** 2021-02-10T11:17:16.103 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:16.136 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.041) 0:06:12.363 **** 2021-02-10T11:17:16.142 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-10T11:17:16.143 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:16.176 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.040) 0:06:12.403 **** 2021-02-10T11:17:16.183 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:16.217 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:16 +0000 (0:00:00.040) 0:06:12.444 **** 2021-02-10T11:17:18.113 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-02-10T11:17:18.115 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T11:17:18.153 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:01.936) 0:06:14.380 **** 2021-02-10T11:17:18.162 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2021-02-10T11:17:18.196 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.042) 0:06:14.423 **** 2021-02-10T11:17:18.202 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-10T11:17:18.204 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:18.238 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.041) 0:06:14.464 **** 2021-02-10T11:17:18.245 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:18.281 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.042) 0:06:14.507 **** 2021-02-10T11:17:18.770 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-10T11:17:18.770 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount system.slice sntp.service -.mount systemd-journald.socket syslog.target basic.target ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T11:17:18.772 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T11:17:18.773 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-10T11:17:18.807 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.526) 0:06:15.034 **** 2021-02-10T11:17:18.842 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "syslog.target systemd-journald.socket sntp.service -.mount ntpdate.service basic.target system.slice tmp.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "chronyd.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "Limi 2021-02-10T11:17:18.843 INFO:teuthology.task.ansible.out:tAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127560", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127560", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "system.slice basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "Timeou 2021-02-10T11:17:18.845 INFO:teuthology.task.ansible.out:tStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} TASK [testnode : set_fact] ***************************************************** ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T11:17:18.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.071) 0:06:15.105 **** 2021-02-10T11:17:18.914 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.035) 0:06:15.140 **** 2021-02-10T11:17:18.920 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:18.921 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:18.958 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:18 +0000 (0:00:00.043) 0:06:15.184 **** 2021-02-10T11:17:18.966 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:18.967 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-10T11:17:19.012 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:19 +0000 (0:00:00.054) 0:06:15.238 **** 2021-02-10T11:17:19.039 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:19.041 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:19.075 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:19 +0000 (0:00:00.063) 0:06:15.302 **** 2021-02-10T11:17:20.082 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T11:17:20.083 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} 2021-02-10T11:17:20.127 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:01.051) 0:06:16.354 **** 2021-02-10T11:17:20.156 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T11:17:20.158 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:20.205 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.077) 0:06:16.432 **** 2021-02-10T11:17:20.227 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-02-10T11:17:20.228 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-14.el7.noarch providing python-pip is already installed"]} TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:20.267 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.061) 0:06:16.493 **** 2021-02-10T11:17:20.312 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.044) 0:06:16.538 **** 2021-02-10T11:17:20.342 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-02-10T11:17:20.344 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:20.378 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.065) 0:06:16.604 **** 2021-02-10T11:17:20.405 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-10T11:17:20.406 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:20.444 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.065) 0:06:16.670 **** 2021-02-10T11:17:20.711 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-10T11:17:20.712 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T11:17:20.755 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.310) 0:06:16.981 **** 2021-02-10T11:17:20.899 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-10T11:17:20.945 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:20 +0000 (0:00:00.190) 0:06:17.172 **** 2021-02-10T11:17:22.692 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-10T11:17:22.693 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T11:17:22.732 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:22 +0000 (0:00:01.786) 0:06:18.958 **** 2021-02-10T11:17:22.874 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-10T11:17:22.910 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:22 +0000 (0:00:00.178) 0:06:19.137 **** 2021-02-10T11:17:24.194 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-10T11:17:24.195 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T11:17:24.230 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:01.320) 0:06:20.457 **** 2021-02-10T11:17:24.237 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-10T11:17:24.273 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.042) 0:06:20.499 **** 2021-02-10T11:17:24.279 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-10T11:17:24.280 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:24.315 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.041) 0:06:20.541 **** 2021-02-10T11:17:24.321 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:24.359 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.043) 0:06:20.585 **** 2021-02-10T11:17:24.367 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-10T11:17:24.368 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T11:17:24.404 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.045) 0:06:20.631 **** 2021-02-10T11:17:24.411 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp1s0f0"}, "changed": false} 2021-02-10T11:17:24.451 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.046) 0:06:20.678 **** 2021-02-10T11:17:24.843 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-10T11:17:24.845 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T11:17:24.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.426) 0:06:21.104 **** 2021-02-10T11:17:24.897 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-10T11:17:24.931 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.053) 0:06:21.158 **** 2021-02-10T11:17:24.939 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-10T11:17:24.939 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:24.982 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:24 +0000 (0:00:00.050) 0:06:21.209 **** 2021-02-10T11:17:24.990 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:25.025 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.042) 0:06:21.251 **** 2021-02-10T11:17:25.031 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-10T11:17:25.032 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T11:17:25.068 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.042) 0:06:21.294 **** 2021-02-10T11:17:25.074 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-10T11:17:25.109 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.041) 0:06:21.336 **** 2021-02-10T11:17:25.115 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-10T11:17:25.117 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:25.150 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.040) 0:06:21.377 **** 2021-02-10T11:17:25.157 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:25.193 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.042) 0:06:21.419 **** 2021-02-10T11:17:25.598 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-10T11:17:25.600 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T11:17:25.634 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.441) 0:06:21.861 **** 2021-02-10T11:17:25.641 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-10T11:17:25.681 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:25 +0000 (0:00:00.046) 0:06:21.908 **** 2021-02-10T11:17:26.077 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-10T11:17:26.079 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T11:17:26.112 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.431) 0:06:22.339 **** 2021-02-10T11:17:26.130 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-10T11:17:26.165 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.052) 0:06:22.392 **** 2021-02-10T11:17:26.172 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-10T11:17:26.173 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.209 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.044) 0:06:22.436 **** 2021-02-10T11:17:26.216 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.252 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.042) 0:06:22.478 **** 2021-02-10T11:17:26.259 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-10T11:17:26.261 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.262 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.272 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.309 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.057) 0:06:22.536 **** 2021-02-10T11:17:26.316 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.317 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.318 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=enp1s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item=enp1s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.355 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.045) 0:06:22.581 **** 2021-02-10T11:17:26.361 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-10T11:17:26.362 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.363 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [gibba017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.374 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.375 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.411 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.055) 0:06:22.637 **** 2021-02-10T11:17:26.418 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.418 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.420 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [gibba025.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.431 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp1s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp1s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.468 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.057) 0:06:22.695 **** 2021-02-10T11:17:26.474 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-10T11:17:26.476 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.511 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.043) 0:06:22.738 **** 2021-02-10T11:17:26.519 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.570 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.058) 0:06:22.797 **** 2021-02-10T11:17:26.579 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-10T11:17:26.581 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.631 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.061) 0:06:22.858 **** 2021-02-10T11:17:26.640 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.683 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.051) 0:06:22.910 **** 2021-02-10T11:17:26.691 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-10T11:17:26.692 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.733 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.049) 0:06:22.960 **** 2021-02-10T11:17:26.741 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.777 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.044) 0:06:23.004 **** 2021-02-10T11:17:26.784 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-10T11:17:26.785 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.832 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.054) 0:06:23.059 **** 2021-02-10T11:17:26.839 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.878 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.045) 0:06:23.105 **** 2021-02-10T11:17:26.887 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-10T11:17:26.887 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.933 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.055) 0:06:23.160 **** 2021-02-10T11:17:26.943 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:26.990 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:26 +0000 (0:00:00.056) 0:06:23.217 **** 2021-02-10T11:17:26.998 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:17:26.999 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.037 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.047) 0:06:23.264 **** 2021-02-10T11:17:27.044 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.084 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.046) 0:06:23.311 **** 2021-02-10T11:17:27.091 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-10T11:17:27.092 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.142 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.058) 0:06:23.369 **** 2021-02-10T11:17:27.149 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.185 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.043) 0:06:23.412 **** 2021-02-10T11:17:27.192 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-10T11:17:27.193 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.226 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.040) 0:06:23.453 **** 2021-02-10T11:17:27.233 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:27.269 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:27 +0000 (0:00:00.042) 0:06:23.496 **** 2021-02-10T11:17:27.708 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-10T11:17:27.710 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004542", "end": "2021-02-10 11:17:27.681970", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 11:17:27.677428", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:27.743 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004569", "end": "2021-02-10 11:17:27.710524", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-10 11:17:27.705955", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:28.140 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002284", "end": "2021-02-10 11:17:28.118054", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 11:17:28.115770", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:28.172 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002297", "end": "2021-02-10 11:17:28.148165", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-10 11:17:28.145868", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:28.586 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002081", "end": "2021-02-10 11:17:28.555271", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:17:28.553190", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:28.598 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002149", "end": "2021-02-10 11:17:28.578579", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-10 11:17:28.576430", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:29.031 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.022759", "end": "2021-02-10 11:17:29.009869", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 11:17:28.987110", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:29.068 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:01.798) 0:06:25.294 **** 2021-02-10T11:17:29.075 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.031252", "end": "2021-02-10 11:17:29.042319", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-10 11:17:29.011067", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:29.110 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:00.041) 0:06:25.336 **** 2021-02-10T11:17:29.116 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-10T11:17:29.154 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:00.044) 0:06:25.381 **** 2021-02-10T11:17:29.200 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:00.045) 0:06:25.426 **** 2021-02-10T11:17:29.597 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-10T11:17:29.598 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:17:29.669 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-10T11:17:29.879 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-10T11:17:29.910 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:00.710) 0:06:26.137 **** 2021-02-10T11:17:29.944 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:29 +0000 (0:00:00.033) 0:06:26.171 **** 2021-02-10T11:17:31.008 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-02-10T11:17:31.009 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] 2021-02-10T11:17:31.089 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:31 +0000 (0:00:01.145) 0:06:27.316 **** 2021-02-10T11:17:31.118 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:31.119 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:31.157 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:31 +0000 (0:00:00.067) 0:06:27.383 **** 2021-02-10T11:17:31.195 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:31.196 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T11:17:31.230 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:31 +0000 (0:00:00.073) 0:06:27.457 **** 2021-02-10T11:17:31.268 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T11:17:31.269 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T11:17:31.304 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:31 +0000 (0:00:00.073) 0:06:27.531 **** 2021-02-10T11:17:32.050 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [gibba025.front.sepia.ceph.com] 2021-02-10T11:17:32.126 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:32 +0000 (0:00:00.822) 0:06:28.353 **** 2021-02-10T11:17:32.158 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:32.159 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:32.192 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:32 +0000 (0:00:00.065) 0:06:28.419 **** 2021-02-10T11:17:32.229 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:32.230 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-10T11:17:32.263 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:32 +0000 (0:00:00.070) 0:06:28.490 **** 2021-02-10T11:17:32.301 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-10T11:17:32.302 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-10T11:17:32.336 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:32 +0000 (0:00:00.072) 0:06:28.563 **** 2021-02-10T11:17:41.540 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T11:17:41.541 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 74 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T11:17:41.542 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} 2021-02-10T11:17:41.579 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:41 +0000 (0:00:09.242) 0:06:37.805 **** 2021-02-10T11:17:41.607 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:41.608 INFO:teuthology.task.ansible.out:ok: [gibba017.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T11:17:41.644 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:41 +0000 (0:00:00.065) 0:06:37.870 **** 2021-02-10T11:17:41.671 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:41.673 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:41.708 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:41 +0000 (0:00:00.064) 0:06:37.934 **** 2021-02-10T11:17:42.183 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-10T11:17:42.185 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:42.632 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:42.671 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:42 +0000 (0:00:00.963) 0:06:38.898 **** 2021-02-10T11:17:42.773 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-10T11:17:42.774 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["podman", "1.6.4-27.el7_9.x86_64 from rhel-7-server-extras-rpms"], ["podman-docker", "1.6.4-27.el7_9.noarch from rhel-7-server-extras-rpms"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nLoading mirror speeds from cached hostfile\n * epel: download-cc-rdu01.fedoraproject.org\nResolving Dependencies\n--> Running transaction check\n---> Package podman.x86_64 0:1.6.4-26.el7_9 will be updated\n---> Package podman.x86_64 0:1.6.4-27.el7_9 will be an update\n---> Package podman-docker.noarch 0:1.6.4-26.el7_9 will be updated\n---> Package podman-docker.noarch 0:1.6.4-27.el7_9 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n podman x86_64 1.6.4-27.el7_9 rhel-7-server-extras-rpms 13 M\n podman-docker noarch 1.6.4-27.el7_9 rhel-7-server-extras-rpms 31 k\n\nTransaction Summary\n================================================================================\nUpgrade 2 Packages\n\nTotal download size: 13 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 70 MB/s | 13 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : podman-1.6.4-27.el7_9.x86_64 1/4 \n Updating : podman-docker-1.6.4-27.el7_9.noarch 2/4 \n Cleanup : podman-docke 2021-02-10T11:17:42.776 INFO:teuthology.task.ansible.out:r-1.6.4-26.el7_9.noarch 3/4 \n Cleanup : podman-1.6.4-26.el7_9.x86_64 4/4 \nLoaded plugins: fastestmirror, langpacks, priorities, product-id, subscription-\n : manager\n Verifying : podman-docker-1.6.4-27.el7_9.noarch 1/4 \n Verifying : podman-1.6.4-27.el7_9.x86_64 2/4 \n Verifying : podman-docker-1.6.4-26.el7_9.noarch 3/4 \n Verifying : podman-1.6.4-26.el7_9.x86_64 4/4 \n\nUpdated:\n podman.x86_64 0:1.6.4-27.el7_9 podman-docker.noarch 0:1.6.4-27.el7_9 \n\nComplete!\n"]} 2021-02-10T11:17:42.810 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:42 +0000 (0:00:00.139) 0:06:39.037 **** 2021-02-10T11:17:42.840 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:42.841 INFO:teuthology.task.ansible.out:ok: [gibba025.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-10T11:17:42.875 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:42 +0000 (0:00:00.064) 0:06:39.102 **** 2021-02-10T11:17:42.902 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-10T11:17:42.903 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:42.937 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:42 +0000 (0:00:00.061) 0:06:39.164 **** 2021-02-10T11:17:43.406 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-10T11:17:43.407 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:43.858 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-10T11:17:43.895 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:43 +0000 (0:00:00.958) 0:06:40.122 **** 2021-02-10T11:17:45.295 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-10T11:17:45.296 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955862.7171469-180754685149988/source", "state": "file", "uid": 0} 2021-02-10T11:17:46.522 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955863.9400778-139268845272922/source", "state": "file", "uid": 0} 2021-02-10T11:17:47.872 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955865.3026264-262229498358294/source", "state": "file", "uid": 0} 2021-02-10T11:17:47.908 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:47 +0000 (0:00:04.012) 0:06:44.135 **** 2021-02-10T11:17:49.092 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1612955866.5251923-119594118806759/source", "state": "file", "uid": 0} 2021-02-10T11:17:49.131 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:49 +0000 (0:00:01.222) 0:06:45.357 **** 2021-02-10T11:17:50.468 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-10T11:17:50.470 INFO:teuthology.task.ansible.out:changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ALyLfu-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-ALyLfu-build", "Co 2021-02-10T11:17:50.471 INFO:teuthology.task.ansible.out:llecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T11:17:50.507 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:50 +0000 (0:00:01.376) 0:06:46.734 **** 2021-02-10T11:17:50.534 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T11:17:50.536 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:50.572 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:50 +0000 (0:00:00.064) 0:06:46.799 **** 2021-02-10T11:17:50.599 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T11:17:50.600 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:50.636 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:50 +0000 (0:00:00.063) 0:06:46.862 **** 2021-02-10T11:17:51.147 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [gibba017.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.066518", "end": "2021-02-10 11:17:51.136342", "rc": 0, "start": "2021-02-10 11:17:51.069824", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:51.188 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:51 +0000 (0:00:00.552) 0:06:47.415 **** 2021-02-10T11:17:51.227 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T11:17:51.229 INFO:teuthology.task.ansible.out:skipping: [gibba017.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:51.756 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "You are using pip version 8.1.2, however version 21.0.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 21.0.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-KT5Iav-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nInstalling collected packages: toml, docopt, typing, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning 2021-02-10T11:17:51.758 INFO:teuthology.task.ansible.out:https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-KT5Iav-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Installing collected packages: toml, docopt, typing, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-02-10T11:17:51.790 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:51 +0000 (0:00:00.601) 0:06:48.017 **** 2021-02-10T11:17:51.821 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-10T11:17:51.822 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:51.857 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:51 +0000 (0:00:00.066) 0:06:48.083 **** 2021-02-10T11:17:51.883 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-10T11:17:51.885 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:51.918 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:51 +0000 (0:00:00.061) 0:06:48.145 **** 2021-02-10T11:17:52.433 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-10T11:17:52.434 INFO:teuthology.task.ansible.out:changed: [gibba025.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.071250", "end": "2021-02-10 11:17:52.420266", "rc": 0, "start": "2021-02-10 11:17:52.349016", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-10T11:17:52.475 INFO:teuthology.task.ansible.out:Wednesday 10 February 2021 11:17:52 +0000 (0:00:00.556) 0:06:48.702 **** 2021-02-10T11:17:52.502 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-10T11:17:52.504 INFO:teuthology.task.ansible.out:skipping: [gibba025.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-10T11:17:52.603 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-10T11:17:52.606 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2021-02-10T11:17:52.607 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2021-02-10T11:17:52.610 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-02-10T11:17:52.611 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* gibba017.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 gibba025.front.sepia.ceph.com : ok=141 changed=45 unreachable=0 failed=0 skipped=169 rescued=0 ignored=4 Wednesday 10 February 2021 11:17:52 +0000 (0:00:00.134) 0:06:48.837 **** =============================================================================== users : Create all admin users with sudo access. ----------------------- 62.14s users : Update authorized_keys using the keys repo --------------------- 60.39s common : Make sure smartmontools is installed -------------------------- 25.63s users : Remove revoked users ------------------------------------------- 21.87s testnode : Install packages -------------------------------------------- 15.38s container-host : Install container packages ----------------------------- 9.24s testnode : Zap all non-root disks --------------------------------------- 9.06s testnode : Ensure ceph packages are not present. ------------------------ 8.67s common : Register with subscription-manager. ---------------------------- 5.91s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 5.17s common : nrpe - Remove SELinux policy package --------------------------- 4.78s 2021-02-10T11:17:52.611 INFO:teuthology.task.ansible.out:common : nrpe - Load SELinux policy package ----------------------------- 4.77s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 4.22s container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert --- 4.01s testnode : Install epel packages ---------------------------------------- 3.77s testnode : Ensure ceph dependency packages are not present. ------------- 3.75s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 3.36s testnode : Configure common additional repos in /etc/yum.repos.d/ ------- 3.24s common : 2021-02-10T11:17:52.613 INFO:teuthology.task.ansible.out: nrpe - Ensure SELinux policy is up to date --------------------- 3.04s testnode : Blow away lingering OSD data and FSIDs ----------------------- 3.03s 2021-02-10T11:17:52.952 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@gibba017.front.sepia.ceph.com'), Remote(name='ubuntu@gibba025.front.sepia.ceph.com')] 2021-02-10T11:17:52.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T11:17:52.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:17:53.062 DEBUG:teuthology.orchestra.run.gibba017:> true 2021-02-10T11:17:53.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-10T11:17:53.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'gibba025.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-10T11:17:53.767 DEBUG:teuthology.orchestra.run.gibba025:> true 2021-02-10T11:17:54.265 INFO:teuthology.run_tasks:Running task clock... 2021-02-10T11:17:54.290 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-10T11:17:54.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T11:17:54.291 DEBUG:teuthology.orchestra.run.gibba017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:17:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T11:18:01.684 INFO:teuthology.orchestra.run.gibba017.stdout:ntpd: time slew -0.002568s 2021-02-10T11:18:01.696 INFO:teuthology.orchestra.run.gibba017.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T11:18:01.736 INFO:teuthology.orchestra.run.gibba017.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:18:01.737 INFO:teuthology.orchestra.run.gibba017.stdout:============================================================================== 2021-02-10T11:18:01.737 INFO:teuthology.orchestra.run.gibba017.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:01.737 INFO:teuthology.orchestra.run.gibba017.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:01.737 INFO:teuthology.orchestra.run.gibba017.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:01.737 INFO:teuthology.orchestra.run.gibba017.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:01.738 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-10T11:18:01.738 DEBUG:teuthology.orchestra.run.gibba025:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T11:18:02.091 INFO:teuthology.orchestra.run.gibba025.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-10T11:18:09.130 INFO:teuthology.orchestra.run.gibba025.stdout:ntpd: time slew -0.001976s 2021-02-10T11:18:09.140 INFO:teuthology.orchestra.run.gibba025.stderr:Failed to start ntp.service: Unit not found. 2021-02-10T11:18:09.200 INFO:teuthology.orchestra.run.gibba025.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T11:18:09.200 INFO:teuthology.orchestra.run.gibba025.stdout:============================================================================== 2021-02-10T11:18:09.200 INFO:teuthology.orchestra.run.gibba025.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:09.200 INFO:teuthology.orchestra.run.gibba025.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:09.201 INFO:teuthology.orchestra.run.gibba025.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:09.201 INFO:teuthology.orchestra.run.gibba025.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-02-10T11:18:09.201 INFO:teuthology.run_tasks:Running task install... 2021-02-10T11:18:09.212 DEBUG:teuthology.task.install:project ceph 2021-02-10T11:18:09.212 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}} 2021-02-10T11:18:09.212 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'} 2021-02-10T11:18:09.212 INFO:teuthology.task.install:Using flavor: basic 2021-02-10T11:18:09.216 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2021-02-10T11:18:09.216 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2021-02-10T11:18:09.217 DEBUG:teuthology.orchestra.run.gibba017:> sudo lsb_release -is 2021-02-10T11:18:09.577 INFO:teuthology.orchestra.run.gibba017.stdout:RedHatEnterpriseServer 2021-02-10T11:18:09.578 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:18:09.578 DEBUG:teuthology.orchestra.run.gibba025:> sudo lsb_release -is 2021-02-10T11:18:09.578 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T11:18:09.579 DEBUG:teuthology.orchestra.run.gibba017:> sudo lsb_release -is 2021-02-10T11:18:09.932 INFO:teuthology.orchestra.run.gibba017.stdout:RedHatEnterpriseServer 2021-02-10T11:18:09.933 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:18:09.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:18:09.937 INFO:teuthology.orchestra.run.gibba025.stdout:RedHatEnterpriseServer 2021-02-10T11:18:09.938 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:18:09.938 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'tag': None, 'wait_for_package': False} 2021-02-10T11:18:09.938 DEBUG:teuthology.orchestra.run.gibba025:> sudo lsb_release -is 2021-02-10T11:18:10.094 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T11:18:10.094 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T11:18:10.220 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T11:18:10.221 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:18:10.221 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T11:18:10.293 INFO:teuthology.orchestra.run.gibba025.stdout:RedHatEnterpriseServer 2021-02-10T11:18:10.294 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:18:10.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:18:10.450 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/ 2021-02-10T11:18:10.450 INFO:teuthology.task.install.rpm:Package version is 14.2.16-280.g962fac13 2021-02-10T11:18:10.564 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T11:18:10.564 DEBUG:teuthology.orchestra.run.gibba017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T11:18:10.584 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-khiremat-mgr-authid-bp-7-nautilus/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/centos/7/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-10T11:18:10.584 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:18:10.585 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-10T11:18:10.907 DEBUG:teuthology.orchestra.run.gibba017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T11:18:10.929 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2021-02-10T11:18:10.929 DEBUG:teuthology.orchestra.run.gibba025:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/962fac13b1a781b8c7d25c1eed50d93813b8f5f0/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-10T11:18:11.261 DEBUG:teuthology.orchestra.run.gibba017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T11:18:12.621 DEBUG:teuthology.orchestra.run.gibba025:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-10T11:18:12.963 INFO:teuthology.orchestra.run.gibba017.stdout:check_obsoletes = 1 2021-02-10T11:18:12.964 DEBUG:teuthology.orchestra.run.gibba017:> sudo yum clean all 2021-02-10T11:18:12.975 DEBUG:teuthology.orchestra.run.gibba025:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-10T11:18:13.316 INFO:teuthology.orchestra.run.gibba025.stdout:check_obsoletes = 1 2021-02-10T11:18:14.160 INFO:teuthology.orchestra.run.gibba017.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:18:14.160 INFO:teuthology.orchestra.run.gibba017.stdout: : disabled-repos, subscription-manager 2021-02-10T11:18:14.160 INFO:teuthology.orchestra.run.gibba017.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T11:18:14.160 INFO:teuthology.orchestra.run.gibba017.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T11:18:14.160 INFO:teuthology.orchestra.run.gibba017.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T11:18:14.161 INFO:teuthology.orchestra.run.gibba017.stdout:Cleaning up list of fastest mirrors 2021-02-10T11:18:14.161 DEBUG:teuthology.orchestra.run.gibba017:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T11:18:14.161 DEBUG:teuthology.orchestra.run.gibba025:> sudo yum clean all 2021-02-10T11:18:14.609 INFO:teuthology.orchestra.run.gibba017.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:18:14.609 INFO:teuthology.orchestra.run.gibba017.stdout: : disabled-repos, subscription-manager 2021-02-10T11:18:14.609 INFO:teuthology.orchestra.run.gibba025.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:18:14.609 INFO:teuthology.orchestra.run.gibba025.stdout: : disabled-repos, subscription-manager 2021-02-10T11:18:14.902 INFO:teuthology.orchestra.run.gibba017.stdout:Determining fastest mirrors 2021-02-10T11:18:14.911 INFO:teuthology.orchestra.run.gibba025.stdout:Cleaning repos: ceph ceph-noarch ceph-source epel lab-extras rhel-7-fcgi-ceph 2021-02-10T11:18:14.912 INFO:teuthology.orchestra.run.gibba025.stdout: : rhel-7-server-extras-rpms rhel-7-server-optional-rpms 2021-02-10T11:18:14.912 INFO:teuthology.orchestra.run.gibba025.stdout: : rhel-7-server-rpms rhel-ha-for-rhel-7-server-rpms 2021-02-10T11:18:15.062 INFO:teuthology.orchestra.run.gibba025.stdout:Cleaning up list of fastest mirrors 2021-02-10T11:18:15.074 DEBUG:teuthology.orchestra.run.gibba025:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2021-02-10T11:18:15.586 INFO:teuthology.orchestra.run.gibba025.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-02-10T11:18:15.586 INFO:teuthology.orchestra.run.gibba025.stdout: : disabled-repos, subscription-manager 2021-02-10T11:18:15.858 INFO:teuthology.orchestra.run.gibba025.stdout:Determining fastest mirrors 2021-02-10T11:18:15.931 INFO:teuthology.orchestra.run.gibba025.stdout: * epel: ftp.linux.ncsu.edu 2021-02-10T11:18:15.943 INFO:teuthology.orchestra.run.gibba017.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-02-10T11:18:41.307 INFO:teuthology.orchestra.run.gibba017.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:18:42.431 INFO:teuthology.orchestra.run.gibba025.stdout:29 packages excluded due to repository priority protections 2021-02-10T11:18:58.093 INFO:teuthology.orchestra.run.gibba017.stdout:Resolving Dependencies 2021-02-10T11:18:58.093 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.093 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:18:58.099 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.100 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.101 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.102 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.103 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.110 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.112 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.113 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.116 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.117 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.118 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.118 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.119 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.120 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:58.120 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:58.121 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.121 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.122 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:58.124 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.124 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:58.124 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.124 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:58.125 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.128 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.129 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.130 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.132 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.132 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.136 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.137 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.138 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.139 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.140 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:18:58.140 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:18:58.145 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:18:58.146 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.146 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.147 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.147 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:18:58.148 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:18:58.149 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.150 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:18:58.150 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:18:58.151 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:18:58.151 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:18:58.155 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:18:58.161 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.161 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.162 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.163 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.163 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.164 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.165 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.166 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.167 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.168 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.168 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.172 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.174 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.175 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.175 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.177 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.178 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:58.178 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.178 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.178 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.179 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.181 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.182 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:18:58.183 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:18:58.183 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:18:58.185 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:58.187 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:58.193 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:58.203 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:58.205 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:18:58.205 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:18:58.205 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.206 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.206 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.207 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:18:58.208 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:18:58.208 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:18:58.209 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:18:58.209 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:18:58.210 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:18:58.211 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.211 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:18:58.219 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:18:58.220 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:58.220 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:58.221 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:58.222 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:58.223 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:58.223 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.224 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.224 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:18:58.224 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:18:58.226 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.226 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:18:58.226 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:18:58.227 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:18:58.227 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:18:58.227 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:18:58.228 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:18:58.229 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:58.229 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:58.230 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:58.230 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:58.233 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:18:58.233 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:18:58.235 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.235 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.235 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:18:58.237 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:58.238 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:58.239 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:58.240 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:18:58.241 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:18:58.241 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:18:58.243 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.243 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:18:58.243 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:18:58.245 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:18:58.245 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:18:58.245 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:18:58.246 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:58.247 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:58.247 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:58.252 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:18:58.252 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:18:58.253 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:18:58.253 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:18:58.253 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:18:58.255 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:58.255 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:18:58.255 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:18:58.255 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:18:58.256 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:18:58.256 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:18:58.257 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:18:58.257 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:18:58.258 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:18:58.259 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:18:58.259 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:18:58.259 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:58.259 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:58.260 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:58.261 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:18:58.261 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:18:58.261 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:18:58.262 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:18:58.262 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:18:58.264 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:18:58.264 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:18:58.264 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:18:58.266 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:18:58.266 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.266 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:18:58.266 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:18:58.267 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:18:58.267 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:18:58.267 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:18:58.268 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:18:58.269 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:18:58.270 INFO:teuthology.orchestra.run.gibba017.stdout:--> Running transaction check 2021-02-10T11:18:58.271 INFO:teuthology.orchestra.run.gibba017.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:18:58.834 INFO:teuthology.orchestra.run.gibba017.stdout:--> Finished Dependency Resolution 2021-02-10T11:18:58.889 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:18:58.889 INFO:teuthology.orchestra.run.gibba017.stdout:Dependencies Resolved 2021-02-10T11:18:58.898 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:18:58.898 INFO:teuthology.orchestra.run.gibba017.stdout:================================================================================ 2021-02-10T11:18:58.898 INFO:teuthology.orchestra.run.gibba017.stdout: Package Arch Version Repository Size 2021-02-10T11:18:58.898 INFO:teuthology.orchestra.run.gibba017.stdout:================================================================================ 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout:Installing: 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:18:58.899 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: elfutils-libelf-devel 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout:Updating: 2021-02-10T11:18:58.900 INFO:teuthology.orchestra.run.gibba017.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout:Installing for dependencies: 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-grafana-dashboards 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: keyutils-libs-devel 2021-02-10T11:18:58.901 INFO:teuthology.orchestra.run.gibba017.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:18:58.902 INFO:teuthology.orchestra.run.gibba017.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-ceph-argparse 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:18:58.903 INFO:teuthology.orchestra.run.gibba017.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:18:58.904 INFO:teuthology.orchestra.run.gibba017.stdout: python-simplegeneric 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-singledispatch 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: 202 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-websocket-client 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: 66 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: 102 k 2021-02-10T11:18:58.905 INFO:teuthology.orchestra.run.gibba017.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: 562 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-google-auth 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-pyasn1-modules 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: 59 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:18:58.906 INFO:teuthology.orchestra.run.gibba017.stdout: python2-requests-oauthlib 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: python3-ceph-argparse 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:18:58.907 INFO:teuthology.orchestra.run.gibba017.stdout:Updating for dependencies: 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout:Transaction Summary 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout:================================================================================ 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:18:58.908 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:18:58.909 INFO:teuthology.orchestra.run.gibba017.stdout:Total download size: 130 M 2021-02-10T11:18:58.909 INFO:teuthology.orchestra.run.gibba017.stdout:Downloading packages: 2021-02-10T11:18:58.909 INFO:teuthology.orchestra.run.gibba017.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:18:59.268 INFO:teuthology.orchestra.run.gibba025.stdout:Resolving Dependencies 2021-02-10T11:18:59.268 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.268 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2021-02-10T11:18:59.274 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.276 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-mon = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.277 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-osd = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.278 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-mds = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.278 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.286 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.288 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-base = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.289 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.292 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.293 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.293 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.294 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.295 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.295 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:59.296 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:59.297 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.297 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.298 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:59.299 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.299 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:59.300 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.300 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 2021-02-10T11:18:59.301 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.304 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-selinux = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.305 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: librgw2 = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.306 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.307 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.308 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.311 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: ceph-common = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.313 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: jq for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.313 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: socat for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.315 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.316 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-5.el7 will be installed 2021-02-10T11:18:59.316 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-5.el7.x86_64 2021-02-10T11:18:59.320 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2021-02-10T11:18:59.321 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.321 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: librados-devel = 2:14.2.16-280.g962fac13.el7 for package: 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.322 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.323 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:18:59.323 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librados2.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:18:59.324 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.325 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2021-02-10T11:18:59.326 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librbd1.x86_64 2:14.2.16-280.g962fac13.el7 will be an update 2021-02-10T11:18:59.326 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-21.el7_9 will be installed 2021-02-10T11:18:59.326 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-21.el7_9 for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:18:59.380 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-21.el7_9.x86_64 2021-02-10T11:18:59.386 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.386 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.387 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-cephfs = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.387 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-rgw = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.388 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-rbd = 2:14.2.16-280.g962fac13.el7 for package: 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.389 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.390 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python3-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.391 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python3-rados = 2:14.2.16-280.g962fac13.el7 for package: 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.392 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.393 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.393 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.397 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.399 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-ceph-argparse = 2:14.2.16-280.g962fac13.el7 for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.400 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.401 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.402 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.403 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 2021-02-10T11:18:59.403 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.403 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.403 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.404 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.406 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.408 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package jq.x86_64 0:1.6-2.el7 will be installed 2021-02-10T11:18:59.408 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libonig.so.5()(64bit) for package: jq-1.6-2.el7.x86_64 2021-02-10T11:18:59.409 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package krb5-devel.x86_64 0:1.15.1-50.el7 will be installed 2021-02-10T11:18:59.411 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:59.413 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:59.418 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:59.428 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-50.el7.x86_64 2021-02-10T11:18:59.430 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2021-02-10T11:18:59.430 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librabbitmq.x86_64 0:0.8.0-3.el7 will be installed 2021-02-10T11:18:59.431 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.431 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.432 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librgw2.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.432 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2021-02-10T11:18:59.433 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:18:59.434 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2021-02-10T11:18:59.434 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:18:59.435 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-1.0.2k-19.el7.x86_64 2021-02-10T11:18:59.436 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:18:59.436 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.436 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2021-02-10T11:18:59.474 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2021-02-10T11:18:59.474 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:59.475 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:59.475 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:59.477 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:59.477 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2021-02-10T11:18:59.478 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.478 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.479 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-requests.noarch 0:2.6.0-10.el7 will be installed 2021-02-10T11:18:59.479 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-10.el7.noarch 2021-02-10T11:18:59.480 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.481 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2021-02-10T11:18:59.481 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2021-02-10T11:18:59.481 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2021-02-10T11:18:59.482 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2021-02-10T11:18:59.482 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2021-02-10T11:18:59.482 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-kubernetes.noarch 1:11.0.0-6.el7 will be installed 2021-02-10T11:18:59.483 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-certifi for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:59.484 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-google-auth for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:59.484 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-requests-oauthlib for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:59.485 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-websocket-client for package: 1:python2-kubernetes-11.0.0-6.el7.noarch 2021-02-10T11:18:59.488 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-remoto.noarch 0:1.0.0-1.el7 will be installed 2021-02-10T11:18:59.488 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python2-execnet >= 1.2.0 for package: python2-remoto-1.0.0-1.el7.noarch 2021-02-10T11:18:59.489 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.489 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.489 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2021-02-10T11:18:59.491 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:59.493 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:59.494 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2021-02-10T11:18:59.494 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2021-02-10T11:18:59.495 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package zlib-devel.x86_64 0:1.2.7-19.el7_9 will be installed 2021-02-10T11:18:59.495 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: zlib = 1.2.7-19.el7_9 for package: zlib-devel-1.2.7-19.el7_9.x86_64 2021-02-10T11:18:59.498 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.498 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2021-02-10T11:18:59.498 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2021-02-10T11:18:59.499 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-19.el7 will be installed 2021-02-10T11:18:59.499 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2021-02-10T11:18:59.500 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libselinux-devel.x86_64 0:2.5-15.el7 will be installed 2021-02-10T11:18:59.500 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:59.502 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:59.502 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-15.el7.x86_64 2021-02-10T11:18:59.506 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2021-02-10T11:18:59.507 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2021-02-10T11:18:59.507 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package oniguruma.x86_64 0:6.8.2-1.el7 will be installed 2021-02-10T11:18:59.507 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be updated 2021-02-10T11:18:59.508 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package openssl.x86_64 1:1.0.2k-21.el7_9 will be an update 2021-02-10T11:18:59.509 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 will be installed 2021-02-10T11:18:59.509 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-execnet.noarch 0:1.7.1-3.el7 will be installed 2021-02-10T11:18:59.510 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-apipkg for package: python-execnet-1.7.1-3.el7.noarch 2021-02-10T11:18:59.510 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2021-02-10T11:18:59.510 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:18:59.511 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2021-02-10T11:18:59.512 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2021-02-10T11:18:59.512 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2021-02-10T11:18:59.512 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2021-02-10T11:18:59.512 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2021-02-10T11:18:59.512 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2021-02-10T11:18:59.513 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:59.514 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:59.515 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2021-02-10T11:18:59.515 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2021-02-10T11:18:59.515 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2021-02-10T11:18:59.516 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2021-02-10T11:18:59.516 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2021-02-10T11:18:59.517 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:18:59.518 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2021-02-10T11:18:59.518 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2021-02-10T11:18:59.519 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated 2021-02-10T11:18:59.520 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package zlib.x86_64 0:1.2.7-19.el7_9 will be an update 2021-02-10T11:18:59.520 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.520 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2021-02-10T11:18:59.521 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2021-02-10T11:18:59.521 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2021-02-10T11:18:59.521 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2021-02-10T11:18:59.522 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2021-02-10T11:18:59.522 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2021-02-10T11:18:59.522 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-apipkg.noarch 0:1.5-5.el7 will be installed 2021-02-10T11:18:59.522 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2021-02-10T11:18:59.523 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python2-rsa.noarch 0:3.4.2-2.el7 will be installed 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2021-02-10T11:18:59.524 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2021-02-10T11:18:59.525 INFO:teuthology.orchestra.run.gibba025.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2021-02-10T11:18:59.525 INFO:teuthology.orchestra.run.gibba025.stdout:--> Running transaction check 2021-02-10T11:18:59.525 INFO:teuthology.orchestra.run.gibba025.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2021-02-10T11:19:00.086 INFO:teuthology.orchestra.run.gibba025.stdout:--> Finished Dependency Resolution 2021-02-10T11:19:00.141 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:19:00.142 INFO:teuthology.orchestra.run.gibba025.stdout:Dependencies Resolved 2021-02-10T11:19:00.150 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout:================================================================================ 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: Package Arch Version Repository Size 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout:================================================================================ 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout:Installing: 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: bison x86_64 3.0.4-2.el7 rhel-7-server-rpms 674 k 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.0 k 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 571 k 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.8 M 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-dashboard noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 4.0 M 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-diskprediction-cloud 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 83 k 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-diskprediction-local 2021-02-10T11:19:00.151 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 1.1 M 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-rook noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-ssh noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 21 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-radosgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.8 M 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-test x86_64 2:14.2.16-280.g962fac13.el7 ceph 34 M 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: elfutils-libelf-devel 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: x86_64 0.176-5.el7 rhel-7-server-rpms 40 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: flex x86_64 2.5.37-6.el7 rhel-7-server-rpms 293 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: libcephfs-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 17 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: libcephfs2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 500 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: openssl-devel x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.5 M 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: python-ceph-compat x86_64 2:14.2.16-280.g962fac13.el7 ceph 2.2 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: python3-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 151 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout: rbd-fuse x86_64 2:14.2.16-280.g962fac13.el7 ceph 69 k 2021-02-10T11:19:00.152 INFO:teuthology.orchestra.run.gibba025.stdout:Updating: 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: librados2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 3.4 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: librbd1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.7 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout:Installing for dependencies: 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-base x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.4 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-common x86_64 2:14.2.16-280.g962fac13.el7 ceph 19 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-grafana-dashboards 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 2:14.2.16-280.g962fac13.el7 ceph-noarch 19 k 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mds x86_64 2:14.2.16-280.g962fac13.el7 ceph 1.9 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mon x86_64 2:14.2.16-280.g962fac13.el7 ceph 4.3 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-osd x86_64 2:14.2.16-280.g962fac13.el7 ceph 16 M 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-selinux x86_64 2:14.2.16-280.g962fac13.el7 ceph 20 k 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: jq x86_64 1.6-2.el7 epel 167 k 2021-02-10T11:19:00.153 INFO:teuthology.orchestra.run.gibba025.stdout: keyutils-libs-devel 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: x86_64 1.5.8-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: krb5-devel x86_64 1.15.1-50.el7 rhel-7-server-rpms 273 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libcom_err-devel x86_64 1.42.9-19.el7 rhel-7-server-rpms 32 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: librabbitmq x86_64 0.8.0-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: librados-devel x86_64 2:14.2.16-280.g962fac13.el7 ceph 86 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libradosstriper1 x86_64 2:14.2.16-280.g962fac13.el7 ceph 364 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: librgw2 x86_64 2:14.2.16-280.g962fac13.el7 ceph 5.1 M 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libselinux-devel x86_64 2.5-15.el7 rhel-7-server-rpms 187 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libsepol-devel x86_64 2.5-10.el7 rhel-7-server-rpms 77 k 2021-02-10T11:19:00.154 INFO:teuthology.orchestra.run.gibba025.stdout: libverto-devel x86_64 0.2.5-4.el7 rhel-7-server-rpms 12 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 rhel-7-server-rpms 206 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: oniguruma x86_64 6.8.2-1.el7 epel 181 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: pcre-devel x86_64 8.32-17.el7 rhel-7-server-rpms 480 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-beaker noarch 1.5.4-10.el7 rhel-7-server-rpms 80 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-ceph-argparse 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-cephfs x86_64 2:14.2.16-280.g962fac13.el7 ceph 113 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-cherrypy noarch 3.2.2-4.el7 rhel-7-server-rpms 422 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-execnet noarch 1.7.1-3.el7 epel 260 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-jwcrypto noarch 0.4.2-1.el7 rhel-7-server-rpms 57 k 2021-02-10T11:19:00.155 INFO:teuthology.orchestra.run.gibba025.stdout: python-mako noarch 0.8.1-2.el7 rhel-7-server-rpms 307 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-markupsafe x86_64 0.11-10.el7 rhel-7-server-rpms 25 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: rhel-7-server-rpms 866 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-prettytable noarch 0.7.2-3.el7 rhel-7-server-rpms 37 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 199 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-rbd x86_64 2:14.2.16-280.g962fac13.el7 ceph 194 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-requests noarch 2.6.0-10.el7 rhel-7-server-rpms 95 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-rgw x86_64 2:14.2.16-280.g962fac13.el7 ceph 77 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-simplegeneric 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 0.8-7.el7 epel 12 k 2021-02-10T11:19:00.156 INFO:teuthology.orchestra.run.gibba025.stdout: python-singledispatch 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 3.4.0.2-2.el7 epel 18 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-tempita noarch 0.5.1-6.el7 rhel-7-server-rpms 33 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-urllib3 noarch 1.10.2-7.el7 rhel-7-server-rpms 103 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-webob noarch 1.2.3-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: 202 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-websocket-client 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 0.56.0-3.git3c25814.el7 rhel-7-server-extras-rpms 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: 66 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-webtest noarch 1.3.4-6.el7 rhel-7-server-optional-rpms 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: 102 k 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: python-werkzeug noarch 0.9.1-2.el7 rhel-7-server-extras-rpms 2021-02-10T11:19:00.157 INFO:teuthology.orchestra.run.gibba025.stdout: 562 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-apipkg noarch 1.5-5.el7 epel 17 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-google-auth 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 1:1.1.1-5.el7 epel 76 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-kubernetes noarch 1:11.0.0-6.el7 epel 1.1 M 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-oauthlib noarch 2.0.1-8.el7 rhel-7-server-rpms 146 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-pyasn1-modules 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 0.1.9-7.el7 rhel-7-server-optional-rpms 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: 59 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-remoto noarch 1.0.0-1.el7 epel 42 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-requests-oauthlib 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: noarch 0.8.0-5.el7 rhel-7-server-rpms 35 k 2021-02-10T11:19:00.158 INFO:teuthology.orchestra.run.gibba025.stdout: python2-rsa noarch 3.4.2-2.el7 epel 68 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: python3-ceph-argparse 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: x86_64 2:14.2.16-280.g962fac13.el7 ceph 35 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: python3-rados x86_64 2:14.2.16-280.g962fac13.el7 ceph 317 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: scipy x86_64 0.12.1-6.el7 rhel-7-server-rpms 9.3 M 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: socat x86_64 1.7.3.2-2.el7 rhel-7-server-rpms 290 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: suitesparse x86_64 4.0.2-10.el7 rhel-7-server-rpms 928 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: tbb x86_64 4.1-9.20130314.el7 rhel-7-server-rpms 124 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: zlib-devel x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 50 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout:Updating for dependencies: 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: openssl x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 493 k 2021-02-10T11:19:00.159 INFO:teuthology.orchestra.run.gibba025.stdout: openssl-libs x86_64 1:1.0.2k-21.el7_9 rhel-7-server-rpms 1.2 M 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout: zlib x86_64 1.2.7-19.el7_9 rhel-7-server-rpms 90 k 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout:Transaction Summary 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout:================================================================================ 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout:Install 19 Packages (+70 Dependent packages) 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout:Upgrade 2 Packages (+ 3 Dependent packages) 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:19:00.160 INFO:teuthology.orchestra.run.gibba025.stdout:Total download size: 130 M 2021-02-10T11:19:00.161 INFO:teuthology.orchestra.run.gibba025.stdout:Downloading packages: 2021-02-10T11:19:00.161 INFO:teuthology.orchestra.run.gibba025.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2021-02-10T11:19:12.465 INFO:teuthology.orchestra.run.gibba025.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:19:12.465 INFO:teuthology.orchestra.run.gibba025.stdout:Total 11 MB/s | 130 MB 00:12 2021-02-10T11:19:12.500 INFO:teuthology.orchestra.run.gibba025.stdout:Running transaction check 2021-02-10T11:19:12.526 INFO:teuthology.orchestra.run.gibba017.stdout:-------------------------------------------------------------------------------- 2021-02-10T11:19:12.526 INFO:teuthology.orchestra.run.gibba017.stdout:Total 9.5 MB/s | 130 MB 00:13 2021-02-10T11:19:12.561 INFO:teuthology.orchestra.run.gibba017.stdout:Running transaction check 2021-02-10T11:19:12.631 INFO:teuthology.orchestra.run.gibba025.stdout:Running transaction test 2021-02-10T11:19:12.689 INFO:teuthology.orchestra.run.gibba017.stdout:Running transaction test 2021-02-10T11:19:13.073 INFO:teuthology.orchestra.run.gibba025.stdout:Transaction test succeeded 2021-02-10T11:19:13.073 INFO:teuthology.orchestra.run.gibba025.stdout:Running transaction 2021-02-10T11:19:13.155 INFO:teuthology.orchestra.run.gibba017.stdout:Transaction test succeeded 2021-02-10T11:19:13.155 INFO:teuthology.orchestra.run.gibba017.stdout:Running transaction 2021-02-10T11:19:13.790 INFO:teuthology.orchestra.run.gibba025.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:19:13.842 INFO:teuthology.orchestra.run.gibba017.stdout: Updating : zlib-1.2.7-19.el7_9.x86_64 1/99 2021-02-10T11:19:14.118 INFO:teuthology.orchestra.run.gibba025.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:19:14.170 INFO:teuthology.orchestra.run.gibba017.stdout: Updating : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 2/99 2021-02-10T11:19:14.427 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:19:14.446 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/99 2021-02-10T11:19:14.688 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:19:14.724 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : librabbitmq-0.8.0-3.el7.x86_64 4/99 2021-02-10T11:19:14.984 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:19:15.036 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : zlib-devel-1.2.7-19.el7_9.x86_64 5/99 2021-02-10T11:19:15.180 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:19:15.232 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-six-1.9.0-0.el7.noarch 6/99 2021-02-10T11:19:15.474 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:19:15.527 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 7/99 2021-02-10T11:19:15.820 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:19:15.867 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 8/99 2021-02-10T11:19:16.107 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:19:16.180 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 9/99 2021-02-10T11:19:16.405 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:19:16.495 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-webob-1.2.3-7.el7.noarch 10/99 2021-02-10T11:19:16.708 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:19:16.764 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 11/99 2021-02-10T11:19:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:19:17.060 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-requests-2.6.0-10.el7.noarch 12/99 2021-02-10T11:19:17.258 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:19:17.297 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 13/99 2021-02-10T11:19:17.580 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:19:17.645 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 14/99 2021-02-10T11:19:17.889 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:19:17.963 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 15/99 2021-02-10T11:19:18.147 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:19:18.294 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 16/99 2021-02-10T11:19:18.420 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:19:18.609 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : oniguruma-6.8.2-1.el7.x86_64 17/99 2021-02-10T11:19:18.751 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:19:18.924 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : jq-1.6-2.el7.x86_64 18/99 2021-02-10T11:19:19.094 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:19:19.274 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 19/99 2021-02-10T11:19:19.435 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:19:19.650 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 20/99 2021-02-10T11:19:19.786 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:19:20.017 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 21/99 2021-02-10T11:19:20.050 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:19:20.282 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:19:20.340 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 22/99 2021-02-10T11:19:20.554 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:19:20.588 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 23/99 2021-02-10T11:19:20.805 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:19:20.853 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 24/99 2021-02-10T11:19:21.078 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 25/99 2021-02-10T11:19:21.093 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:19:21.366 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 26/99 2021-02-10T11:19:21.389 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:19:21.664 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 27/99 2021-02-10T11:19:21.674 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:19:21.901 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:19:21.932 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : libcom_err-devel-1.42.9-19.el7.x86_64 28/99 2021-02-10T11:19:22.199 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 29/99 2021-02-10T11:19:22.217 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:19:22.523 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:19:22.534 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/99 2021-02-10T11:19:22.751 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:19:22.845 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 31/99 2021-02-10T11:19:23.012 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:19:23.066 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 32/99 2021-02-10T11:19:23.286 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:19:23.309 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 33/99 2021-02-10T11:19:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:19:23.600 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 34/99 2021-02-10T11:19:23.850 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:19:23.856 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 35/99 2021-02-10T11:19:24.174 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:19:24.356 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 36/99 2021-02-10T11:19:24.703 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:19:25.038 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:19:25.322 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:19:25.559 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:19:25.620 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 38/99 2021-02-10T11:19:25.814 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:19:25.954 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 1:python2-kubernetes-11.0.0-6.el7.noarch 39/99 2021-02-10T11:19:26.059 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:19:26.177 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 40/99 2021-02-10T11:19:26.381 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:19:26.429 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 41/99 2021-02-10T11:19:26.703 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 42/99 2021-02-10T11:19:26.878 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:19:27.039 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 43/99 2021-02-10T11:19:27.192 INFO:teuthology.orchestra.run.gibba017.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:19:27.458 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:19:27.573 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 44/99 2021-02-10T11:19:27.813 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:19:27.899 INFO:teuthology.orchestra.run.gibba025.stdout: Updating : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:19:28.199 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 46/99 2021-02-10T11:19:28.446 INFO:teuthology.orchestra.run.gibba017.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:19:28.545 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:19:28.711 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:19:28.979 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:19:29.144 INFO:teuthology.orchestra.run.gibba025.stdout: Updating : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 48/99 2021-02-10T11:19:29.233 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:19:29.418 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 49/99 2021-02-10T11:19:29.533 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:19:29.711 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 50/99 2021-02-10T11:19:30.040 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 51/99 2021-02-10T11:19:30.323 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:19:31.548 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:19:32.070 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:19:32.346 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:19:32.726 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:19:32.894 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:19:33.298 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:19:39.996 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:19:40.683 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:19:40.774 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:19:41.231 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 57/99 2021-02-10T11:19:41.277 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:19:41.571 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:19:41.838 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:19:41.865 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 58/99 2021-02-10T11:19:42.044 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:19:42.196 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 59/99 2021-02-10T11:19:42.339 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:19:42.479 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:19:42.590 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:19:42.710 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:19:42.887 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:19:43.013 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 62/99 2021-02-10T11:19:43.273 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-routes-1.13-2.el7.noarch 63/99 2021-02-10T11:19:43.277 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:19:43.577 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 64/99 2021-02-10T11:19:43.755 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:19:43.935 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-mako-0.8.1-2.el7.noarch 65/99 2021-02-10T11:19:44.074 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:19:44.413 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:19:44.776 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:19:44.778 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 66/99 2021-02-10T11:19:45.095 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:19:45.380 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:19:45.458 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-apipkg-1.5-5.el7.noarch 68/99 2021-02-10T11:19:45.625 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:19:45.806 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python-execnet-1.7.1-3.el7.noarch 69/99 2021-02-10T11:19:45.935 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:19:46.172 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : python2-remoto-1.0.0-1.el7.noarch 70/99 2021-02-10T11:19:46.207 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:19:46.424 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 71/99 2021-02-10T11:19:46.538 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:19:46.768 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 72/99 2021-02-10T11:19:47.049 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : libselinux-devel-2.5-15.el7.x86_64 73/99 2021-02-10T11:19:47.406 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : krb5-devel-1.15.1-50.el7.x86_64 74/99 2021-02-10T11:19:47.800 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:19:48.394 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:19:48.689 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 75/99 2021-02-10T11:19:48.880 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:19:49.177 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:19:49.294 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : scipy-0.12.1-6.el7.x86_64 76/99 2021-02-10T11:19:49.419 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:19:49.666 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:19:49.796 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 77/99 2021-02-10T11:19:49.845 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:19:50.103 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 78/99 2021-02-10T11:19:50.353 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 79/99 2021-02-10T11:19:50.642 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 80/99 2021-02-10T11:19:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:19:50.829 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:19:51.163 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:19:51.412 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:19:51.598 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 82/99 2021-02-10T11:19:51.962 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 83/99 2021-02-10T11:19:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:19:52.236 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 84/99 2021-02-10T11:19:52.864 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 85/99 2021-02-10T11:19:54.833 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:19:55.300 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:19:55.497 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:19:55.653 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:19:55.739 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:19:55.945 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:19:55.956 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 87/99 2021-02-10T11:19:56.130 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 88/99 2021-02-10T11:19:56.281 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:19:56.372 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 89/99 2021-02-10T11:19:56.610 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 90/99 2021-02-10T11:19:56.732 INFO:teuthology.orchestra.run.gibba017.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:19:56.923 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : elfutils-libelf-devel-0.176-5.el7.x86_64 91/99 2021-02-10T11:19:57.062 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:19:57.316 INFO:teuthology.orchestra.run.gibba017.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:19:57.341 INFO:teuthology.orchestra.run.gibba025.stdout: Updating : 1:openssl-1.0.2k-21.el7_9.x86_64 92/99 2021-02-10T11:19:57.566 INFO:teuthology.orchestra.run.gibba017.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:19:57.686 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : bison-3.0.4-2.el7.x86_64 93/99 2021-02-10T11:19:57.859 INFO:teuthology.orchestra.run.gibba017.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:19:57.978 INFO:teuthology.orchestra.run.gibba025.stdout: Installing : flex-2.5.37-6.el7.x86_64 94/99 2021-02-10T11:19:58.116 INFO:teuthology.orchestra.run.gibba017.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:19:58.260 INFO:teuthology.orchestra.run.gibba025.stdout: Cleanup : 1:openssl-1.0.2k-19.el7.x86_64 95/99 2021-02-10T11:19:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:19:58.578 INFO:teuthology.orchestra.run.gibba025.stdout: Cleanup : 1:openssl-libs-1.0.2k-19.el7.x86_64 96/99 2021-02-10T11:19:58.773 INFO:teuthology.orchestra.run.gibba017.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:19:58.773 INFO:teuthology.orchestra.run.gibba017.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:19:58.773 INFO:teuthology.orchestra.run.gibba017.stdout: : manager 2021-02-10T11:19:58.853 INFO:teuthology.orchestra.run.gibba025.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 97/99 2021-02-10T11:19:59.128 INFO:teuthology.orchestra.run.gibba025.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 98/99 2021-02-10T11:19:59.795 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:19:59.912 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:19:59.952 INFO:teuthology.orchestra.run.gibba025.stdout: Cleanup : zlib-1.2.7-18.el7.x86_64 99/99 2021-02-10T11:19:59.952 INFO:teuthology.orchestra.run.gibba025.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, subscription- 2021-02-10T11:19:59.953 INFO:teuthology.orchestra.run.gibba025.stdout: : manager 2021-02-10T11:20:00.401 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:20:00.498 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:20:00.572 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:20:00.608 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:libradosstriper1-14.2.16-280.g962fac13.el7.x86_64 1/99 2021-02-10T11:20:00.682 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:20:00.693 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-selinux-14.2.16-280.g962fac13.el7.x86_64 2/99 2021-02-10T11:20:00.773 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:20:00.801 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-rgw-14.2.16-280.g962fac13.el7.x86_64 3/99 2021-02-10T11:20:00.892 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:20:00.909 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 4/99 2021-02-10T11:20:00.991 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:20:01.035 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : jq-1.6-2.el7.x86_64 5/99 2021-02-10T11:20:01.074 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:20:01.159 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:20:01.168 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-routes-1.13-2.el7.noarch 6/99 2021-02-10T11:20:01.259 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:20:01.294 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 7/99 2021-02-10T11:20:01.360 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:20:01.427 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 8/99 2021-02-10T11:20:01.477 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:20:01.561 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 9/99 2021-02-10T11:20:01.594 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:20:01.712 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:20:01.712 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 10/99 2021-02-10T11:20:01.829 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 11/99 2021-02-10T11:20:01.837 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:20:01.955 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:libcephfs2-14.2.16-280.g962fac13.el7.x86_64 12/99 2021-02-10T11:20:01.962 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:20:02.047 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-rbd-14.2.16-280.g962fac13.el7.x86_64 13/99 2021-02-10T11:20:02.071 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:20:02.139 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:openssl-1.0.2k-21.el7_9.x86_64 14/99 2021-02-10T11:20:02.163 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:20:02.223 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-apipkg-1.5-5.el7.noarch 15/99 2021-02-10T11:20:02.248 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:20:02.331 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 16/99 2021-02-10T11:20:02.364 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:20:02.424 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 17/99 2021-02-10T11:20:02.448 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:20:02.516 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 18/99 2021-02-10T11:20:02.532 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:20:02.607 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:20:02.608 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:openssl-libs-1.0.2k-21.el7_9.x86_64 19/99 2021-02-10T11:20:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:20:02.709 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-requests-2.6.0-10.el7.noarch 20/99 2021-02-10T11:20:02.758 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:20:02.826 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 21/99 2021-02-10T11:20:02.859 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:20:02.951 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 22/99 2021-02-10T11:20:02.960 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:20:03.068 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:20:03.068 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:python2-kubernetes-11.0.0-6.el7.noarch 23/99 2021-02-10T11:20:03.168 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:20:03.177 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 24/99 2021-02-10T11:20:03.286 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-osd-14.2.16-280.g962fac13.el7.x86_64 25/99 2021-02-10T11:20:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:20:03.395 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 26/99 2021-02-10T11:20:03.411 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:20:03.504 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-remoto-1.0.0-1.el7.noarch 27/99 2021-02-10T11:20:03.537 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:20:03.646 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 28/99 2021-02-10T11:20:03.654 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:20:03.755 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:20:03.763 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 29/99 2021-02-10T11:20:03.839 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:20:03.872 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-14.2.16-280.g962fac13.el 30/99 2021-02-10T11:20:03.922 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:20:03.955 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-fuse-14.2.16-280.g962fac13.el7.x86_64 31/99 2021-02-10T11:20:04.015 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:20:04.040 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-14.2.16-280.g962fac13.el7.x86_64 32/99 2021-02-10T11:20:04.098 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:20:04.115 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 33/99 2021-02-10T11:20:04.174 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:20:04.236 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 34/99 2021-02-10T11:20:04.274 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:20:04.324 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mds-14.2.16-280.g962fac13.el7.x86_64 35/99 2021-02-10T11:20:04.342 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:20:04.442 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:20:04.534 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:20:04.643 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:20:04.760 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:20:04.862 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:20:04.986 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:20:05.119 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 36/99 2021-02-10T11:20:05.228 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-rsa-3.4.2-2.el7.noarch 37/99 2021-02-10T11:20:05.345 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-ssh-14.2.16-280.g962fac13.el7.noarch 38/99 2021-02-10T11:20:05.462 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 39/99 2021-02-10T11:20:05.596 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 40/99 2021-02-10T11:20:05.714 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:20:05.721 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : elfutils-libelf-devel-0.176-5.el7.x86_64 41/99 2021-02-10T11:20:05.798 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:20:05.813 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 42/99 2021-02-10T11:20:05.882 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:20:05.897 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 43/99 2021-02-10T11:20:05.949 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:20:05.989 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : librabbitmq-0.8.0-3.el7.x86_64 44/99 2021-02-10T11:20:06.016 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:20:06.081 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-base-14.2.16-280.g962fac13.el7.x86_64 45/99 2021-02-10T11:20:06.117 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:20:06.174 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 46/99 2021-02-10T11:20:06.217 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:20:06.290 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 47/99 2021-02-10T11:20:06.318 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:20:06.358 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : libselinux-devel-2.5-15.el7.x86_64 48/99 2021-02-10T11:20:06.435 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:20:06.458 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : zlib-1.2.7-19.el7_9.x86_64 49/99 2021-02-10T11:20:06.535 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:20:06.559 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 50/99 2021-02-10T11:20:06.652 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:20:06.668 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 51/99 2021-02-10T11:20:06.777 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-radosgw-14.2.16-280.g962fac13.el7.x86_64 52/99 2021-02-10T11:20:06.778 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:20:06.866 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:20:06.919 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-cephfs-14.2.16-280.g962fac13.el7.x86_64 53/99 2021-02-10T11:20:06.967 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:20:07.036 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:libcephfs-devel-14.2.16-280.g962fac13.el7.x86_64 54/99 2021-02-10T11:20:07.075 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:20:07.153 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-rados-14.2.16-280.g962fac13.el7.x86_64 55/99 2021-02-10T11:20:07.176 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:20:07.270 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:librados2-14.2.16-280.g962fac13.el7.x86_64 56/99 2021-02-10T11:20:07.293 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:20:07.387 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 57/99 2021-02-10T11:20:07.410 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:20:07.505 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : flex-2.5.37-6.el7.x86_64 58/99 2021-02-10T11:20:07.536 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:20:07.630 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 59/99 2021-02-10T11:20:07.661 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:20:07.714 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python3-ceph-argparse-14.2.16-280.g962fac13.el7.x86_64 60/99 2021-02-10T11:20:07.778 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:20:07.798 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-14.2.16-280.g962fac13.el7.x86_64 61/99 2021-02-10T11:20:07.890 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:librgw2-14.2.16-280.g962fac13.el7.x86_64 62/99 2021-02-10T11:20:07.896 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:20:07.982 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : krb5-devel-1.15.1-50.el7.x86_64 63/99 2021-02-10T11:20:08.038 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-test-14.2.16-280.g962fac13.el7.x86_64 64/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 65/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : libcom_err-devel-1.42.9-19.el7.x86_64 66/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:rbd-fuse-14.2.16-280.g962fac13.el7.x86_64 67/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:librbd1-14.2.16-280.g962fac13.el7.x86_64 68/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 69/99 2021-02-10T11:20:08.661 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:20:08.662 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:20:08.711 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 70/99 2021-02-10T11:20:08.751 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:20:08.853 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-diskprediction-local-14.2.16-280.g962fac13.el 71/99 2021-02-10T11:20:08.868 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:20:08.970 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 72/99 2021-02-10T11:20:08.976 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:20:09.102 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:20:09.104 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 73/99 2021-02-10T11:20:09.228 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:20:09.238 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-rook-14.2.16-280.g962fac13.el7.noarch 74/99 2021-02-10T11:20:09.345 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:20:09.355 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:librados-devel-14.2.16-280.g962fac13.el7.x86_64 75/99 2021-02-10T11:20:09.470 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:20:09.472 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 76/99 2021-02-10T11:20:09.564 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mgr-dashboard-14.2.16-280.g962fac13.el7.noarch 77/99 2021-02-10T11:20:09.587 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:20:09.640 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python3-rados-14.2.16-280.g962fac13.el7.x86_64 78/99 2021-02-10T11:20:09.696 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:20:09.732 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 79/99 2021-02-10T11:20:09.797 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:20:09.824 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python-ceph-compat-14.2.16-280.g962fac13.el7.x86_64 80/99 2021-02-10T11:20:09.880 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:20:09.916 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-common-14.2.16-280.g962fac13.el7.x86_64 81/99 2021-02-10T11:20:09.998 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:20:10.082 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:20:10.165 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:20:10.200 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:python3-cephfs-14.2.16-280.g962fac13.el7.x86_64 82/99 2021-02-10T11:20:10.250 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:20:10.250 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:20:10.250 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:20:10.250 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:20:10.251 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:20:10.300 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-grafana-dashboards-14.2.16-280.g962fac13.el7.noar 83/99 2021-02-10T11:20:10.409 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-execnet-1.7.1-3.el7.noarch 84/99 2021-02-10T11:20:10.484 INFO:teuthology.orchestra.run.gibba017.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:20:10.484 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:10.484 INFO:teuthology.orchestra.run.gibba017.stdout:Installed: 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:20:10.485 INFO:teuthology.orchestra.run.gibba017.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout:Dependency Installed: 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.486 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.487 INFO:teuthology.orchestra.run.gibba017.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.488 INFO:teuthology.orchestra.run.gibba017.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:20:10.489 INFO:teuthology.orchestra.run.gibba017.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:20:10.490 INFO:teuthology.orchestra.run.gibba017.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:20:10.491 INFO:teuthology.orchestra.run.gibba017.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout:Updated: 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout:Dependency Updated: 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:20:10.492 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:10.493 INFO:teuthology.orchestra.run.gibba017.stdout:Complete! 2021-02-10T11:20:10.519 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 85/99 2021-02-10T11:20:10.617 DEBUG:teuthology.parallel:result is None 2021-02-10T11:20:10.627 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 2:ceph-mon-14.2.16-280.g962fac13.el7.x86_64 86/99 2021-02-10T11:20:10.737 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : bison-3.0.4-2.el7.x86_64 87/99 2021-02-10T11:20:10.870 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:openssl-devel-1.0.2k-21.el7_9.x86_64 88/99 2021-02-10T11:20:10.987 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 89/99 2021-02-10T11:20:11.106 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 90/99 2021-02-10T11:20:11.221 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : zlib-devel-1.2.7-19.el7_9.x86_64 91/99 2021-02-10T11:20:11.314 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 92/99 2021-02-10T11:20:11.414 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 93/99 2021-02-10T11:20:11.531 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : oniguruma-6.8.2-1.el7.x86_64 94/99 2021-02-10T11:20:11.532 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 95/99 2021-02-10T11:20:11.532 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : zlib-1.2.7-18.el7.x86_64 96/99 2021-02-10T11:20:11.532 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 97/99 2021-02-10T11:20:11.533 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 98/99 2021-02-10T11:20:11.757 INFO:teuthology.orchestra.run.gibba025.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 99/99 2021-02-10T11:20:11.758 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:11.758 INFO:teuthology.orchestra.run.gibba025.stdout:Installed: 2021-02-10T11:20:11.758 INFO:teuthology.orchestra.run.gibba025.stdout: bison.x86_64 0:3.0.4-2.el7 2021-02-10T11:20:11.758 INFO:teuthology.orchestra.run.gibba025.stdout: ceph.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.758 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-dashboard.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-diskprediction-cloud.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-diskprediction-local.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-rook.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.759 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mgr-ssh.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-radosgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-test.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: elfutils-libelf-devel.x86_64 0:0.176-5.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: flex.x86_64 0:2.5.37-6.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: libcephfs-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: libcephfs2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.760 INFO:teuthology.orchestra.run.gibba025.stdout: openssl-devel.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: python-ceph-compat.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: python3-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: rbd-fuse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout:Dependency Installed: 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-base.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.761 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-common.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-grafana-dashboards.noarch 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mds.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-mon.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-osd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: ceph-selinux.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: jq.x86_64 0:1.6-2.el7 2021-02-10T11:20:11.762 INFO:teuthology.orchestra.run.gibba025.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: krb5-devel.x86_64 0:1.15.1-50.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: libcom_err-devel.x86_64 0:1.42.9-19.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: liboath.x86_64 0:2.6.2-1.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: librabbitmq.x86_64 0:0.8.0-3.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: librados-devel.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: libradosstriper1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2021-02-10T11:20:11.763 INFO:teuthology.orchestra.run.gibba025.stdout: librgw2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: libselinux-devel.x86_64 0:2.5-15.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: oniguruma.x86_64 0:6.8.2-1.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: pcre-devel.x86_64 0:8.32-17.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: python-beaker.noarch 0:1.5.4-10.el7 2021-02-10T11:20:11.764 INFO:teuthology.orchestra.run.gibba025.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-cephfs.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-execnet.noarch 0:1.7.1-3.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-mako.noarch 0:0.8.1-2.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2021-02-10T11:20:11.765 INFO:teuthology.orchestra.run.gibba025.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2021-02-10T11:20:11.766 INFO:teuthology.orchestra.run.gibba025.stdout: python-pecan.noarch 0:0.4.5-2.el7 2021-02-10T11:20:11.766 INFO:teuthology.orchestra.run.gibba025.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2021-02-10T11:20:11.766 INFO:teuthology.orchestra.run.gibba025.stdout: python-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.766 INFO:teuthology.orchestra.run.gibba025.stdout: python-rbd.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.766 INFO:teuthology.orchestra.run.gibba025.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-requests.noarch 0:2.6.0-10.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-rgw.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-routes.noarch 0:1.13-2.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2021-02-10T11:20:11.767 INFO:teuthology.orchestra.run.gibba025.stdout: python-tempita.noarch 0:0.5.1-6.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python-webob.noarch 0:1.2.3-7.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python-webtest.noarch 0:1.3.4-6.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python2-apipkg.noarch 0:1.5-5.el7 2021-02-10T11:20:11.768 INFO:teuthology.orchestra.run.gibba025.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-kubernetes.noarch 1:11.0.0-6.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-remoto.noarch 0:1.0.0-1.el7 2021-02-10T11:20:11.769 INFO:teuthology.orchestra.run.gibba025.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: python2-rsa.noarch 0:3.4.2-2.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: python2-six.noarch 0:1.9.0-0.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: python3-ceph-argparse.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: python3-rados.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: scipy.x86_64 0:0.12.1-6.el7 2021-02-10T11:20:11.770 INFO:teuthology.orchestra.run.gibba025.stdout: socat.x86_64 0:1.7.3.2-2.el7 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: zlib-devel.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout:Updated: 2021-02-10T11:20:11.771 INFO:teuthology.orchestra.run.gibba025.stdout: librados2.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout: librbd1.x86_64 2:14.2.16-280.g962fac13.el7 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout:Dependency Updated: 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout: openssl.x86_64 1:1.0.2k-21.el7_9 openssl-libs.x86_64 1:1.0.2k-21.el7_9 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout: zlib.x86_64 0:1.2.7-19.el7_9 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:11.772 INFO:teuthology.orchestra.run.gibba025.stdout:Complete! 2021-02-10T11:20:11.871 DEBUG:teuthology.parallel:result is None 2021-02-10T11:20:11.872 DEBUG:teuthology.orchestra.run.gibba017:> sudo lsb_release -is 2021-02-10T11:20:12.252 INFO:teuthology.orchestra.run.gibba017.stdout:RedHatEnterpriseServer 2021-02-10T11:20:12.253 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:20:12.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:20:12.446 DEBUG:teuthology.orchestra.run.gibba017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:20:12.802 INFO:teuthology.orchestra.run.gibba017.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:20:12.803 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:20:12.803 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:20:12.803 DEBUG:teuthology.orchestra.run.gibba025:> sudo lsb_release -is 2021-02-10T11:20:13.185 INFO:teuthology.orchestra.run.gibba025.stdout:RedHatEnterpriseServer 2021-02-10T11:20:13.186 DEBUG:teuthology.misc:System to be installed: RedHatEnterpriseServer 2021-02-10T11:20:13.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=962fac13b1a781b8c7d25c1eed50d93813b8f5f0 2021-02-10T11:20:13.375 DEBUG:teuthology.orchestra.run.gibba025:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-10T11:20:13.731 INFO:teuthology.orchestra.run.gibba025.stdout:14.2.16-280.g962fac13.el7 2021-02-10T11:20:13.731 INFO:teuthology.packaging:The installed version of ceph is 14.2.16-280.g962fac13.el7 2021-02-10T11:20:13.731 INFO:teuthology.task.install:The correct ceph version 14.2.16-280.g962fac13 is installed. 2021-02-10T11:20:13.732 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-10T11:20:13.732 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:13.732 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:20:14.078 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:14.078 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-10T11:20:14.423 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-10T11:20:14.423 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:14.424 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:20:14.767 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:20:15.110 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:15.110 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/usr/bin/daemon-helper 2021-02-10T11:20:15.454 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-10T11:20:15.796 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-10T11:20:15.797 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:15.797 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:20:16.139 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:20:16.482 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:16.482 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-10T11:20:16.824 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-10T11:20:17.166 INFO:teuthology.run_tasks:Running task ceph... 2021-02-10T11:20:17.218 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-10T11:20:17.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 777 /var/log/ceph 2021-02-10T11:20:17.222 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 777 /var/log/ceph 2021-02-10T11:20:17.565 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-10T11:20:17.566 DEBUG:teuthology.orchestra.run.gibba017:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:20:17.569 DEBUG:teuthology.orchestra.run.gibba025:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-10T11:20:17.912 INFO:tasks.ceph:Creating extra log directories... 2021-02-10T11:20:17.913 DEBUG:teuthology.orchestra.run.gibba017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:20:17.916 DEBUG:teuthology.orchestra.run.gibba025:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-10T11:20:18.258 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2021-02-10T11:20:18.259 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2021-02-10T11:20:18.259 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2021-02-10T11:20:18.259 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:18.259 DEBUG:teuthology.orchestra.run.gibba017:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:20:18.592 DEBUG:teuthology.orchestra.run.gibba017:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:18.979 DEBUG:teuthology.orchestra.run.gibba017:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:19.322 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:19.322 DEBUG:teuthology.orchestra.run.gibba025:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2021-02-10T11:20:19.657 DEBUG:teuthology.orchestra.run.gibba025:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:20.483 DEBUG:teuthology.orchestra.run.gibba025:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:20.826 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-02-10T11:20:20.826 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-02-10T11:20:20.826 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722', 'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'description': 'fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_7} tasks/volumes}', 'email': 'khiremat@redhat.com', 'first_in_suite': False, 'job_id': '5872722', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'gibba', 'name': 'khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '7.9', 'overrides': {'admin_socket': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus'}, 'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': ''}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180}, 'mgr': {'debug client': 10, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(TOO_FEW_PGS\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, 'owner': 'scheduled_khiremat@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.1'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.2', 'client.3']], 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-khiremat-mgr-authid-bp-7-nautilus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0', 'targets': {'gibba017.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvPiPCP+84JBxO1XaMH52v/h34wPsG8Lu6bQuua9H7e3EelLeBaTEUCub8H6EB8Ae+G23LYCaVnoEMQIgwwMZ8HOBX7quy4z9QBLocC+z7F13cR9wWevteezakfPUuug/DGwsAenCMa56jOwzrjaZMRunOg2JP6AXZQpVQPXT0929+hMWjizbsWZ/nG8dDzG2hPAgo7xolAj6rONMu0NowOJ2KDsCww6+ulTaJPh6hKSBqkcWITxTukahY4otQaD4AGvBfVRL/uhjxgn1Zhh4UySjZenQxEGe9EeApTdywgbF5OsSSUl/KmkyagcGO4VqRHjKP1Sbh4S1xuULCs26p', 'gibba025.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDA+sG5q84njl7vgBldbJqZbclIjZSgoPnPKk9H18ICtpMTl01ruuJH0JSQaW0P2v/fjthbPeU1QFI9yHvF8jinrVDeS8/vtV4KGbm2vehcSRT4zW1A68Q7NQkNziUMWGyz877vICgChjGllUS8ZTFJDMtLu/1wtSD0dkr2/QCjtuFVmFCMUOPHmcayDjHM/PIyVPK+qJESKrxoWSVNsgoOJi4lKHE8+8m/+f03XzDCZlZJw0gKJV2pz8KLWcn7h8/6LKtg8fEtuHZtb1wSjKYYW92HKny1SOkrDyA2EaZpRDvZJLGeSx9KufNI9OW1uVhyLB10WPWGB+VZeu7eTV+T'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'sha1': '962fac13b1a781b8c7d25c1eed50d93813b8f5f0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes']}}], 'teuthology_branch': 'master', 'timestamp': '2021-02-10_10:13:45', 'tube': 'gibba', 'user': 'khiremat', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.gibba.29737'} 2021-02-10T11:20:20.827 DEBUG:teuthology.orchestra.run.gibba017:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:20:20.830 DEBUG:teuthology.orchestra.run.gibba025:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-10T11:20:21.164 DEBUG:teuthology.orchestra.run.gibba017:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:20:21.168 DEBUG:teuthology.orchestra.run.gibba025:> sudo install -d -m0777 -- /var/run/ceph 2021-02-10T11:20:21.511 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:21.512 DEBUG:teuthology.orchestra.run.gibba017:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:20:21.846 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:20:21.846 DEBUG:teuthology.orchestra.run.gibba017:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Device: 5h/5d Inode: 93767 Links: 1 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Access: 2021-02-10 11:19:52.256496645 +0000 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Modify: 2021-02-10 11:17:07.422428428 +0000 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout:Change: 2021-02-10 11:17:07.422428428 +0000 2021-02-10T11:20:22.180 INFO:teuthology.orchestra.run.gibba017.stdout: Birth: - 2021-02-10T11:20:22.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:20:22.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:20:22.189 INFO:teuthology.orchestra.run.gibba017.stderr:512 bytes (512 B) copied, 0.000212244 s, 2.4 MB/s 2021-02-10T11:20:22.191 DEBUG:teuthology.orchestra.run.gibba017:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:20:22.525 INFO:teuthology.orchestra.run.gibba017.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:20:22.525 INFO:teuthology.orchestra.run.gibba017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:22.525 INFO:teuthology.orchestra.run.gibba017.stdout:Device: 5h/5d Inode: 91608 Links: 1 2021-02-10T11:20:22.525 INFO:teuthology.orchestra.run.gibba017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:22.525 INFO:teuthology.orchestra.run.gibba017.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:22.526 INFO:teuthology.orchestra.run.gibba017.stdout:Access: 2021-02-10 11:19:52.256496645 +0000 2021-02-10T11:20:22.526 INFO:teuthology.orchestra.run.gibba017.stdout:Modify: 2021-02-10 11:17:07.945413354 +0000 2021-02-10T11:20:22.526 INFO:teuthology.orchestra.run.gibba017.stdout:Change: 2021-02-10 11:17:07.945413354 +0000 2021-02-10T11:20:22.526 INFO:teuthology.orchestra.run.gibba017.stdout: Birth: - 2021-02-10T11:20:22.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:20:22.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:20:22.534 INFO:teuthology.orchestra.run.gibba017.stderr:512 bytes (512 B) copied, 0.000188887 s, 2.7 MB/s 2021-02-10T11:20:22.536 DEBUG:teuthology.orchestra.run.gibba017:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Device: 5h/5d Inode: 84676 Links: 1 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Access: 2021-02-10 11:19:52.256496645 +0000 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Modify: 2021-02-10 11:17:08.463398425 +0000 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout:Change: 2021-02-10 11:17:08.463398425 +0000 2021-02-10T11:20:22.870 INFO:teuthology.orchestra.run.gibba017.stdout: Birth: - 2021-02-10T11:20:22.879 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:20:22.879 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:20:22.880 INFO:teuthology.orchestra.run.gibba017.stderr:512 bytes (512 B) copied, 0.000267558 s, 1.9 MB/s 2021-02-10T11:20:22.883 DEBUG:teuthology.orchestra.run.gibba017:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Device: 5h/5d Inode: 93792 Links: 1 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Access: 2021-02-10 11:19:52.256496645 +0000 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Modify: 2021-02-10 11:17:08.988383294 +0000 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout:Change: 2021-02-10 11:17:08.988383294 +0000 2021-02-10T11:20:23.216 INFO:teuthology.orchestra.run.gibba017.stdout: Birth: - 2021-02-10T11:20:23.225 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:20:23.225 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:20:23.225 INFO:teuthology.orchestra.run.gibba017.stderr:512 bytes (512 B) copied, 0.000135597 s, 3.8 MB/s 2021-02-10T11:20:23.227 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:23.227 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:23.227 DEBUG:teuthology.orchestra.run.gibba025:> dd if=/scratch_devs of=/dev/stdout 2021-02-10T11:20:23.560 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-10T11:20:23.561 DEBUG:teuthology.orchestra.run.gibba025:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-10T11:20:23.893 INFO:teuthology.orchestra.run.gibba025.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-0’ 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Device: 5h/5d Inode: 91609 Links: 1 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Access: 2021-02-10 11:19:53.065891923 +0000 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Modify: 2021-02-10 11:17:07.417009975 +0000 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout:Change: 2021-02-10 11:17:07.417009975 +0000 2021-02-10T11:20:23.894 INFO:teuthology.orchestra.run.gibba025.stdout: Birth: - 2021-02-10T11:20:23.903 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records in 2021-02-10T11:20:23.903 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records out 2021-02-10T11:20:23.903 INFO:teuthology.orchestra.run.gibba025.stderr:512 bytes (512 B) copied, 0.000205228 s, 2.5 MB/s 2021-02-10T11:20:23.905 DEBUG:teuthology.orchestra.run.gibba025:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-10T11:20:24.238 INFO:teuthology.orchestra.run.gibba025.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-1’ 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Device: 5h/5d Inode: 73512 Links: 1 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Access: 2021-02-10 11:19:53.065891923 +0000 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Modify: 2021-02-10 11:17:07.947003093 +0000 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout:Change: 2021-02-10 11:17:07.947003093 +0000 2021-02-10T11:20:24.239 INFO:teuthology.orchestra.run.gibba025.stdout: Birth: - 2021-02-10T11:20:24.248 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records in 2021-02-10T11:20:24.248 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records out 2021-02-10T11:20:24.248 INFO:teuthology.orchestra.run.gibba025.stderr:512 bytes (512 B) copied, 0.000192425 s, 2.7 MB/s 2021-02-10T11:20:24.250 DEBUG:teuthology.orchestra.run.gibba025:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-10T11:20:24.582 INFO:teuthology.orchestra.run.gibba025.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Device: 5h/5d Inode: 73539 Links: 1 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Access: 2021-02-10 11:19:53.065891923 +0000 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Modify: 2021-02-10 11:17:08.450996550 +0000 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout:Change: 2021-02-10 11:17:08.450996550 +0000 2021-02-10T11:20:24.583 INFO:teuthology.orchestra.run.gibba025.stdout: Birth: - 2021-02-10T11:20:24.592 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records in 2021-02-10T11:20:24.592 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records out 2021-02-10T11:20:24.592 INFO:teuthology.orchestra.run.gibba025.stderr:512 bytes (512 B) copied, 0.000184486 s, 2.8 MB/s 2021-02-10T11:20:24.594 DEBUG:teuthology.orchestra.run.gibba025:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-10T11:20:24.925 INFO:teuthology.orchestra.run.gibba025.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-3’ 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Device: 5h/5d Inode: 98490 Links: 1 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Context: system_u:object_r:device_t:s0 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Access: 2021-02-10 11:19:53.065891923 +0000 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Modify: 2021-02-10 11:17:08.977989707 +0000 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout:Change: 2021-02-10 11:17:08.977989707 +0000 2021-02-10T11:20:24.926 INFO:teuthology.orchestra.run.gibba025.stdout: Birth: - 2021-02-10T11:20:24.934 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records in 2021-02-10T11:20:24.935 INFO:teuthology.orchestra.run.gibba025.stderr:1+0 records out 2021-02-10T11:20:24.935 INFO:teuthology.orchestra.run.gibba025.stderr:512 bytes (512 B) copied, 0.000222441 s, 2.3 MB/s 2021-02-10T11:20:24.937 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:24.937 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@gibba017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba025.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:20:24.937 INFO:tasks.ceph:Generating config... 2021-02-10T11:20:24.938 INFO:tasks.ceph:[client] client mount timeout = 600 2021-02-10T11:20:24.938 INFO:tasks.ceph:[client] debug client = 20 2021-02-10T11:20:24.938 INFO:tasks.ceph:[client] debug ms = 1 2021-02-10T11:20:24.938 INFO:tasks.ceph:[client] pid file = 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] debug mds = 20 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] debug ms = 1 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds bal frag = True 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds bal merge size = 5 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds bal split bits = 3 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds bal split size = 100 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds debug frag = True 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds op complaint time = 180 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] mds verify scatter = True 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mds] osd op complaint time = 180 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mgr] debug client = 10 2021-02-10T11:20:24.939 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-10T11:20:24.940 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-10T11:20:24.940 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[mon] mon op complaint time = 120 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-02-10T11:20:24.940 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-02-10T11:20:24.941 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-02-10T11:20:24.941 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-02-10T11:20:24.941 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-02-10T11:20:24.941 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-02-10T11:20:24.941 INFO:tasks.ceph:[osd] osd op complaint time = 180 2021-02-10T11:20:24.941 INFO:tasks.ceph:Setting up mon.a... 2021-02-10T11:20:24.941 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-02-10T11:20:25.326 INFO:teuthology.orchestra.run.gibba017.stdout:creating /etc/ceph/ceph.keyring 2021-02-10T11:20:25.328 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-02-10T11:20:25.696 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:20:26.038 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.2.117'), ('mon.b', '172.21.2.125'), ('mon.c', '[v2:172.21.2.125:3301,v1:172.21.2.125:6790]')] 2021-02-10T11:20:26.038 DEBUG:teuthology.orchestra.run.gibba017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.2.117 --add b 172.21.2.125 --addv c '[v2:172.21.2.125:3301,v1:172.21.2.125:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:26.401 INFO:teuthology.orchestra.run.gibba017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:26.401 INFO:teuthology.orchestra.run.gibba017.stdout:monmaptool: generated fsid 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:epoch 0 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:fsid 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:last_changed 2021-02-10 11:20:26.403562 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:created 2021-02-10 11:20:26.403562 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:min_mon_release 0 (unknown) 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:0: [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] mon.a 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:1: [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] mon.b 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:2: [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] mon.c 2021-02-10T11:20:26.402 INFO:teuthology.orchestra.run.gibba017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-02-10T11:20:26.408 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 722ef60b-feda-4d6a-867f-a5d7f666055b... 2021-02-10T11:20:26.409 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:20:26.412 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:20:26.768 INFO:teuthology.orchestra.run.gibba017.stdout:[global] 2021-02-10T11:20:26.768 INFO:teuthology.orchestra.run.gibba017.stdout: chdir = "" 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: auth supported = cephx 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: filestore xattr use omap = true 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: mon clock drift allowed = 1.000 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: osd crush chooseleaf type = 0 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: auth debug = true 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on old message = true 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on bug = true 2021-02-10T11:20:26.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon pg warn max object skew = 0 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default size = 2 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow primary affinity = true 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow pg remap = true 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on no sortbitwise = false 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on too few osds = false 2021-02-10T11:20:26.770 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: osd default data pool replay window = 5 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: mon allow pool delete = true 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: mon cluster log file level = debug 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: debug asserts on shutdown = true 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: mon health detail to clog = false 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:20:26.771 INFO:teuthology.orchestra.run.gibba017.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout:[osd] 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd journal size = 100 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub load threshold = 5.0 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub max interval = 600 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd recover clone overlap = true 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd recovery max chunk = 1048576 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.772 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug shutdown = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug op order = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug verify stray on activate = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd open classes on start = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug pg log writeout = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: osd map max advance = 10 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: journal zero on create = true 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:20:26.773 INFO:teuthology.orchestra.run.gibba017.stdout: filestore apply finisher threads = 3 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: bdev debug aio = true 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug misdirected ops = true 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore block size = 96636764160 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore fsck on mount = True 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluefs = 20 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluestore = 20 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug filestore = 20 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug journal = 20 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 20 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug osd = 25 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: debug rocksdb = 10 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:20:26.774 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd full ratio = 0.9 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: osd objectstore = bluestore 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr] 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: debug mgr = 20 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:20:26.775 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 10 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout:[mon] 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: debug paxos = 20 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: mon data avail warn = 5 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: mon mgr mkfs grace = 240 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd reporter subtree level = osd 2021-02-10T11:20:26.776 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd prime pg temp = true 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: mon op complaint time = 120 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout:[client] 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: rgw cache enabled = true 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable ops log = true 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable usage log = true 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: client mount timeout = 600 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 20 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:20:26.777 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = "" 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.a] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.b] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.c] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout:[mds] 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: debug mds = 20 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal frag = True 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal fragment size max = 10000 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal merge size = 5 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split bits = 3 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split size = 100 2021-02-10T11:20:26.778 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug frag = True 2021-02-10T11:20:26.779 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug scatterstat = True 2021-02-10T11:20:26.779 INFO:teuthology.orchestra.run.gibba017.stdout: mds op complaint time = 180 2021-02-10T11:20:26.779 INFO:teuthology.orchestra.run.gibba017.stdout: mds verify scatter = True 2021-02-10T11:20:26.779 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout:[global] 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: chdir = "" 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: auth supported = cephx 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: filestore xattr use omap = true 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: mon clock drift allowed = 1.000 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: osd crush chooseleaf type = 0 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: auth debug = true 2021-02-10T11:20:26.780 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on old message = true 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on bug = true 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon pg warn max object skew = 0 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default size = 2 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow primary affinity = true 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow pg remap = true 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:20:26.781 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on no sortbitwise = false 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on too few osds = false 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: osd default data pool replay window = 5 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon allow pool delete = true 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: mon cluster log file level = debug 2021-02-10T11:20:26.782 INFO:teuthology.orchestra.run.gibba025.stdout: debug asserts on shutdown = true 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: mon health detail to clog = false 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout:[osd] 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: osd journal size = 100 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub load threshold = 5.0 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub max interval = 600 2021-02-10T11:20:26.783 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd recover clone overlap = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd recovery max chunk = 1048576 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug shutdown = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug op order = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug verify stray on activate = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd open classes on start = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug pg log writeout = true 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: osd map max advance = 10 2021-02-10T11:20:26.784 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: journal zero on create = true 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: filestore apply finisher threads = 3 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: bdev debug aio = true 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug misdirected ops = true 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore block size = 96636764160 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore fsck on mount = True 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluefs = 20 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluestore = 20 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: debug filestore = 20 2021-02-10T11:20:26.785 INFO:teuthology.orchestra.run.gibba025.stdout: debug journal = 20 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 20 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: debug osd = 25 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: debug rocksdb = 10 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd full ratio = 0.9 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: osd objectstore = bluestore 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr] 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:20:26.786 INFO:teuthology.orchestra.run.gibba025.stdout: debug mgr = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 10 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout:[mon] 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug paxos = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: mon data avail warn = 5 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: mon mgr mkfs grace = 240 2021-02-10T11:20:26.787 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd reporter subtree level = osd 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd prime pg temp = true 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: mon op complaint time = 120 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout:[client] 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: rgw cache enabled = true 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable ops log = true 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable usage log = true 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: client mount timeout = 600 2021-02-10T11:20:26.788 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 20 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = "" 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.a] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.b] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.c] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout:[mds] 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: debug mds = 20 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal frag = True 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal fragment size max = 10000 2021-02-10T11:20:26.789 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal merge size = 5 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split bits = 3 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split size = 100 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug frag = True 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug scatterstat = True 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds op complaint time = 180 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: mds verify scatter = True 2021-02-10T11:20:26.790 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:20:26.791 INFO:tasks.ceph:Creating admin key on mon.a... 2021-02-10T11:20:26.791 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-02-10T11:20:27.167 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-10T11:20:27.504 DEBUG:teuthology.orchestra.remote:gibba017:/etc/ceph/ceph.keyring is 206B 2021-02-10T11:20:28.179 DEBUG:teuthology.orchestra.remote:gibba017:/home/ubuntu/cephtest/ceph.monmap is 373B 2021-02-10T11:20:28.518 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:20:28.518 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:28.519 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:20:28.519 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:20:28.869 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:20:28.869 DEBUG:teuthology.orchestra.run.gibba017:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:29.202 INFO:tasks.ceph:Sending monmap to node ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:20:29.203 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:29.203 DEBUG:teuthology.orchestra.run.gibba025:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-10T11:20:29.203 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-10T11:20:29.556 DEBUG:teuthology.orchestra.run.gibba025:> set -ex 2021-02-10T11:20:29.556 DEBUG:teuthology.orchestra.run.gibba025:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:29.889 INFO:tasks.ceph:Setting up mon nodes... 2021-02-10T11:20:29.890 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-10T11:20:29.890 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:20:30.267 INFO:teuthology.orchestra.run.gibba017.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-02-10T11:20:30.269 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:20:30.669 INFO:teuthology.orchestra.run.gibba025.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-02-10T11:20:30.673 INFO:tasks.ceph:Setting up mds nodes... 2021-02-10T11:20:30.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:20:31.051 INFO:teuthology.orchestra.run.gibba017.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2021-02-10T11:20:31.053 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2021-02-10T11:20:31.395 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:20:31.772 INFO:teuthology.orchestra.run.gibba025.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2021-02-10T11:20:31.775 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2021-02-10T11:20:32.118 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-10T11:20:32.119 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-02-10T11:20:32.486 INFO:teuthology.orchestra.run.gibba017.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-02-10T11:20:32.646 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-02-10T11:20:33.016 INFO:teuthology.orchestra.run.gibba017.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-02-10T11:20:33.027 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2021-02-10T11:20:33.395 INFO:teuthology.orchestra.run.gibba025.stdout:creating /etc/ceph/ceph.client.2.keyring 2021-02-10T11:20:33.406 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2021-02-10T11:20:33.772 INFO:teuthology.orchestra.run.gibba025.stdout:creating /etc/ceph/ceph.client.3.keyring 2021-02-10T11:20:33.783 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-10T11:20:33.783 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@gibba017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@gibba025.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-02-10T11:20:33.783 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-02-10T11:20:34.126 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:34.127 INFO:tasks.ceph:role: osd.0 2021-02-10T11:20:34.127 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:20:34.127 DEBUG:teuthology.orchestra.run.gibba017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:20:34.515 INFO:teuthology.orchestra.run.gibba017.stdout:Discarding blocks...Done. 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:34.516 INFO:teuthology.orchestra.run.gibba017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:34.517 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba017.front.sepia.ceph.com -o noatime 2021-02-10T11:20:34.517 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-02-10T11:20:34.878 DEBUG:teuthology.orchestra.run.gibba017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-02-10T11:20:35.225 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-02-10T11:20:35.569 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:35.569 INFO:tasks.ceph:role: osd.1 2021-02-10T11:20:35.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:20:35.569 DEBUG:teuthology.orchestra.run.gibba017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:20:35.958 INFO:teuthology.orchestra.run.gibba017.stdout:Discarding blocks...Done. 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:35.959 INFO:teuthology.orchestra.run.gibba017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:35.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba017.front.sepia.ceph.com -o noatime 2021-02-10T11:20:35.960 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-02-10T11:20:36.316 DEBUG:teuthology.orchestra.run.gibba017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-02-10T11:20:36.660 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-02-10T11:20:37.004 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:37.004 INFO:tasks.ceph:role: osd.2 2021-02-10T11:20:37.004 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:20:37.005 DEBUG:teuthology.orchestra.run.gibba017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout:Discarding blocks...Done. 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:37.392 INFO:teuthology.orchestra.run.gibba017.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:37.393 INFO:teuthology.orchestra.run.gibba017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:37.393 INFO:teuthology.orchestra.run.gibba017.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:37.393 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:37.393 INFO:teuthology.orchestra.run.gibba017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:37.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba017.front.sepia.ceph.com -o noatime 2021-02-10T11:20:37.394 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-02-10T11:20:37.750 DEBUG:teuthology.orchestra.run.gibba017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-02-10T11:20:38.121 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-02-10T11:20:38.472 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:38.472 INFO:tasks.ceph:role: osd.3 2021-02-10T11:20:38.472 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T11:20:38.472 DEBUG:teuthology.orchestra.run.gibba017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:20:38.878 INFO:teuthology.orchestra.run.gibba017.stdout:Discarding blocks...Done. 2021-02-10T11:20:38.878 INFO:teuthology.orchestra.run.gibba017.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:38.878 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:38.878 INFO:teuthology.orchestra.run.gibba017.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:38.878 INFO:teuthology.orchestra.run.gibba017.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:38.879 INFO:teuthology.orchestra.run.gibba017.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:38.879 INFO:teuthology.orchestra.run.gibba017.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:38.879 INFO:teuthology.orchestra.run.gibba017.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:38.879 INFO:teuthology.orchestra.run.gibba017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:38.879 INFO:teuthology.orchestra.run.gibba017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:38.880 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba017.front.sepia.ceph.com -o noatime 2021-02-10T11:20:38.880 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-02-10T11:20:39.254 DEBUG:teuthology.orchestra.run.gibba017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-02-10T11:20:39.600 DEBUG:teuthology.orchestra.run.gibba017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:39.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.971 7fbde0cc6a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-02-10T11:20:39.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.971 7fbde0cc6a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-02-10T11:20:39.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.971 7fbde0cc6a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:20:39.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.971 7fbde0cc6a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:20:39.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.971 7fbde0cc6a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-10T11:20:39.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:39.972 7fbde0cc6a80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-02-10T11:20:42.261 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-10T11:20:42.606 DEBUG:teuthology.orchestra.run.gibba017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:42.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.976 7fe0932f0a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-02-10T11:20:42.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.977 7fe0932f0a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-02-10T11:20:42.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.977 7fe0932f0a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:20:42.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.977 7fe0932f0a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:20:42.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.977 7fe0932f0a80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-10T11:20:42.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:42.978 7fe0932f0a80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-02-10T11:20:45.271 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-02-10T11:20:45.614 DEBUG:teuthology.orchestra.run.gibba017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:45.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.985 7ff24314fa80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-02-10T11:20:45.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.986 7ff24314fa80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-02-10T11:20:45.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.986 7ff24314fa80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:20:45.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.986 7ff24314fa80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:20:45.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.986 7ff24314fa80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-10T11:20:45.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:45.987 7ff24314fa80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-02-10T11:20:48.273 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-02-10T11:20:48.617 DEBUG:teuthology.orchestra.run.gibba017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:48.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.988 7f166131ba80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-02-10T11:20:48.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.988 7f166131ba80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-02-10T11:20:48.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.988 7f166131ba80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:20:48.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.988 7f166131ba80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:20:48.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.988 7f166131ba80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-02-10T11:20:48.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:20:48.990 7f166131ba80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-02-10T11:20:50.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:50.832 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:20:51.278 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-02-10T11:20:51.622 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-02-10T11:20:51.967 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:51.967 INFO:tasks.ceph:role: osd.4 2021-02-10T11:20:51.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:20:51.968 DEBUG:teuthology.orchestra.run.gibba025:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-10T11:20:52.354 INFO:teuthology.orchestra.run.gibba025.stdout:Discarding blocks...Done. 2021-02-10T11:20:52.354 INFO:teuthology.orchestra.run.gibba025.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:52.355 INFO:teuthology.orchestra.run.gibba025.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:52.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@gibba025.front.sepia.ceph.com -o noatime 2021-02-10T11:20:52.356 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2021-02-10T11:20:52.715 DEBUG:teuthology.orchestra.run.gibba025:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-02-10T11:20:53.061 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-02-10T11:20:53.405 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:53.405 INFO:tasks.ceph:role: osd.5 2021-02-10T11:20:53.405 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:20:53.406 DEBUG:teuthology.orchestra.run.gibba025:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout:Discarding blocks...Done. 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:53.793 INFO:teuthology.orchestra.run.gibba025.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:53.794 INFO:teuthology.orchestra.run.gibba025.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:53.794 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:53.794 INFO:teuthology.orchestra.run.gibba025.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:53.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@gibba025.front.sepia.ceph.com -o noatime 2021-02-10T11:20:53.795 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2021-02-10T11:20:54.151 DEBUG:teuthology.orchestra.run.gibba025:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-02-10T11:20:54.495 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2021-02-10T11:20:54.838 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:54.838 INFO:tasks.ceph:role: osd.6 2021-02-10T11:20:54.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:20:54.838 DEBUG:teuthology.orchestra.run.gibba025:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-10T11:20:55.227 INFO:teuthology.orchestra.run.gibba025.stdout:Discarding blocks...Done. 2021-02-10T11:20:55.228 INFO:teuthology.orchestra.run.gibba025.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:55.228 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:55.228 INFO:teuthology.orchestra.run.gibba025.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:55.228 INFO:teuthology.orchestra.run.gibba025.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:55.228 INFO:teuthology.orchestra.run.gibba025.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:55.229 INFO:teuthology.orchestra.run.gibba025.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:55.229 INFO:teuthology.orchestra.run.gibba025.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:55.229 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:55.229 INFO:teuthology.orchestra.run.gibba025.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:55.230 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@gibba025.front.sepia.ceph.com -o noatime 2021-02-10T11:20:55.231 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2021-02-10T11:20:55.586 DEBUG:teuthology.orchestra.run.gibba025:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2021-02-10T11:20:55.931 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2021-02-10T11:20:56.275 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-02-10T11:20:56.275 INFO:tasks.ceph:role: osd.7 2021-02-10T11:20:56.275 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T11:20:56.275 DEBUG:teuthology.orchestra.run.gibba025:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-02-10T11:20:56.667 INFO:teuthology.orchestra.run.gibba025.stdout:Discarding blocks...Done. 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5505024 blks 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout: = crc=1 finobt=0, sparse=0 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout:data = bsize=4096 blocks=22020096, imaxpct=25 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout: = sunit=0 swidth=0 blks 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout:log =internal log bsize=4096 blocks=10752, version=2 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-10T11:20:56.668 INFO:teuthology.orchestra.run.gibba025.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-10T11:20:56.669 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@gibba025.front.sepia.ceph.com -o noatime 2021-02-10T11:20:56.669 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2021-02-10T11:20:57.027 DEBUG:teuthology.orchestra.run.gibba025:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2021-02-10T11:20:57.372 DEBUG:teuthology.orchestra.run.gibba025:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:20:57.743 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.743 7fe42b94ca80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-02-10T11:20:57.744 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.743 7fe42b94ca80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-02-10T11:20:57.744 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.743 7fe42b94ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:20:57.744 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.743 7fe42b94ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:20:57.744 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.743 7fe42b94ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-02-10T11:20:57.744 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:20:57.744 7fe42b94ca80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-02-10T11:21:00.034 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-02-10T11:21:00.378 DEBUG:teuthology.orchestra.run.gibba025:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.747 7fcf6624da80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.748 7fcf6624da80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.748 7fcf6624da80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.748 7fcf6624da80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.748 7fcf6624da80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-02-10T11:21:00.749 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:00.749 7fcf6624da80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-02-10T11:21:03.037 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-02-10T11:21:03.380 DEBUG:teuthology.orchestra.run.gibba025:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:21:03.751 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.751 7f900b0fba80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2021-02-10T11:21:03.751 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.751 7f900b0fba80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2021-02-10T11:21:03.751 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.751 7f900b0fba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:21:03.751 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.751 7f900b0fba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:21:03.752 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.751 7f900b0fba80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-02-10T11:21:03.752 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:03.752 7f900b0fba80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2021-02-10T11:21:06.043 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-02-10T11:21:06.388 DEBUG:teuthology.orchestra.run.gibba025:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:21:06.759 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.759 7f811c345a80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2021-02-10T11:21:06.759 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.759 7f811c345a80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2021-02-10T11:21:06.759 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.759 7f811c345a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:21:06.759 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.759 7f811c345a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:21:06.759 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.759 7f811c345a80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-02-10T11:21:06.760 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:21:06.760 7f811c345a80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2021-02-10T11:21:09.052 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2021-02-10T11:21:09.396 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-10T11:21:09.397 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:09.733 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.S6cLVKDc0N 2021-02-10T11:21:09.733 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.S6cLVKDc0N 2021-02-10T11:21:10.076 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.S6cLVKDc0N 2021-02-10T11:21:10.756 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.S6cLVKDc0N is 56B 2021-02-10T11:21:11.096 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.S6cLVKDc0N 2021-02-10T11:21:11.430 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:11.765 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.CoEA8oXl8G 2021-02-10T11:21:11.765 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.CoEA8oXl8G 2021-02-10T11:21:12.109 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.CoEA8oXl8G 2021-02-10T11:21:12.789 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.CoEA8oXl8G is 56B 2021-02-10T11:21:13.128 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.CoEA8oXl8G 2021-02-10T11:21:13.462 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:13.796 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.qu8Wlf5iP1 2021-02-10T11:21:13.796 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.qu8Wlf5iP1 2021-02-10T11:21:14.138 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.qu8Wlf5iP1 2021-02-10T11:21:14.821 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.qu8Wlf5iP1 is 56B 2021-02-10T11:21:15.159 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.qu8Wlf5iP1 2021-02-10T11:21:15.493 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:15.826 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.bQmVrjLDRK 2021-02-10T11:21:15.827 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.bQmVrjLDRK 2021-02-10T11:21:16.171 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.bQmVrjLDRK 2021-02-10T11:21:16.852 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.bQmVrjLDRK is 56B 2021-02-10T11:21:17.190 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.bQmVrjLDRK 2021-02-10T11:21:17.524 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:17.858 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.u20fJRgSPj 2021-02-10T11:21:17.859 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.u20fJRgSPj 2021-02-10T11:21:18.203 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.u20fJRgSPj 2021-02-10T11:21:18.882 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.u20fJRgSPj is 56B 2021-02-10T11:21:19.224 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.u20fJRgSPj 2021-02-10T11:21:19.558 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:19.892 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.ZUimkb61aY 2021-02-10T11:21:19.893 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.ZUimkb61aY 2021-02-10T11:21:20.237 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.ZUimkb61aY 2021-02-10T11:21:20.922 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.ZUimkb61aY is 56B 2021-02-10T11:21:21.223 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:21.226 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:21.261 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.ZUimkb61aY 2021-02-10T11:21:21.558 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:21.892 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.ASqk6lVu7o 2021-02-10T11:21:21.892 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.ASqk6lVu7o 2021-02-10T11:21:22.235 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.ASqk6lVu7o 2021-02-10T11:21:22.916 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.ASqk6lVu7o is 56B 2021-02-10T11:21:23.255 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.ASqk6lVu7o 2021-02-10T11:21:23.588 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:23.923 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.5meV1tb6Zs 2021-02-10T11:21:23.923 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.5meV1tb6Zs 2021-02-10T11:21:24.268 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.5meV1tb6Zs 2021-02-10T11:21:24.963 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.5meV1tb6Zs is 56B 2021-02-10T11:21:25.457 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.5meV1tb6Zs 2021-02-10T11:21:25.793 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:26.125 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.l6KbnqdZY4 2021-02-10T11:21:26.126 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.l6KbnqdZY4 2021-02-10T11:21:26.468 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.l6KbnqdZY4 2021-02-10T11:21:27.146 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.l6KbnqdZY4 is 56B 2021-02-10T11:21:27.485 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.l6KbnqdZY4 2021-02-10T11:21:27.820 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:28.153 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.5C1cMjBa05 2021-02-10T11:21:28.154 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.5C1cMjBa05 2021-02-10T11:21:28.498 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.5C1cMjBa05 2021-02-10T11:21:29.179 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.5C1cMjBa05 is 56B 2021-02-10T11:21:29.595 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.5C1cMjBa05 2021-02-10T11:21:29.951 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:30.296 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.RGTqGpnLYq 2021-02-10T11:21:30.296 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.RGTqGpnLYq 2021-02-10T11:21:30.651 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.RGTqGpnLYq 2021-02-10T11:21:31.356 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.RGTqGpnLYq is 56B 2021-02-10T11:21:31.698 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.RGTqGpnLYq 2021-02-10T11:21:32.032 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:21:32.365 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.GMl3PUutXz 2021-02-10T11:21:32.366 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.GMl3PUutXz 2021-02-10T11:21:32.709 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.GMl3PUutXz 2021-02-10T11:21:33.388 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.GMl3PUutXz is 56B 2021-02-10T11:21:33.729 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.GMl3PUutXz 2021-02-10T11:21:34.401 DEBUG:teuthology.orchestra.remote:gibba017:/etc/ceph/ceph.client.0.keyring is 59B 2021-02-10T11:21:35.077 DEBUG:teuthology.orchestra.remote:gibba017:/etc/ceph/ceph.client.1.keyring is 59B 2021-02-10T11:21:35.757 DEBUG:teuthology.orchestra.remote:gibba025:/etc/ceph/ceph.client.2.keyring is 59B 2021-02-10T11:21:36.434 DEBUG:teuthology.orchestra.remote:gibba025:/etc/ceph/ceph.client.3.keyring is 59B 2021-02-10T11:21:36.774 INFO:tasks.ceph:Adding keys to all mons... 2021-02-10T11:21:36.775 DEBUG:teuthology.orchestra.run.gibba017:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:21:36.778 DEBUG:teuthology.orchestra.run.gibba025:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr.y] 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg== 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg== 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.0] 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg== 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.1] 2021-02-10T11:21:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.2] 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.3] 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr.x] 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.4] 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA== 2021-02-10T11:21:37.119 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.5] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.6] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[osd.7] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[client.0] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[client.1] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[client.2] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g== 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout:[client.3] 2021-02-10T11:21:37.120 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ== 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr.y] 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg== 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg== 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.0] 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg== 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.1] 2021-02-10T11:21:37.122 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A== 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.2] 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ== 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.3] 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ== 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr.x] 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog== 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ== 2021-02-10T11:21:37.123 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.4] 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA== 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.5] 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w== 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.6] 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A== 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout:[osd.7] 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA== 2021-02-10T11:21:37.124 INFO:teuthology.orchestra.run.gibba025.stdout:[client.0] 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw== 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout:[client.1] 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg== 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout:[client.2] 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g== 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout:[client.3] 2021-02-10T11:21:37.125 INFO:teuthology.orchestra.run.gibba025.stdout: key = AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ== 2021-02-10T11:21:37.125 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:21:37.129 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:21:37.504 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:21:37.506 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:21:37.883 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:37.886 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:38.259 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:38.262 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:38.637 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:38.640 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:39.014 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:39.017 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:39.392 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:21:39.395 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-10T11:21:39.770 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:21:39.773 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2021-02-10T11:21:40.148 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:40.152 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:40.528 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:40.531 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:40.907 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:40.910 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:41.286 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:41.289 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-10T11:21:41.663 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:41.666 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.046 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.423 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.426 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.803 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:42.806 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-02-10T11:21:43.182 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-10T11:21:43.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-02-10T11:21:43.526 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:21:43.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-02-10T11:21:44.252 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-02-10T11:21:44.596 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:21:44.976 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-02-10T11:21:45.319 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-02-10T11:21:45.661 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-10T11:21:46.041 DEBUG:teuthology.orchestra.run.gibba025:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-02-10T11:21:46.384 DEBUG:teuthology.orchestra.run.gibba017:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:21:46.388 DEBUG:teuthology.orchestra.run.gibba025:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-10T11:21:46.721 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-02-10T11:21:46.722 INFO:tasks.ceph.mon.a:Restarting daemon 2021-02-10T11:21:46.722 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-02-10T11:21:46.725 INFO:tasks.ceph.mon.a:Started 2021-02-10T11:21:46.725 INFO:tasks.ceph.mon.b:Restarting daemon 2021-02-10T11:21:46.725 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-02-10T11:21:46.728 INFO:tasks.ceph.mon.b:Started 2021-02-10T11:21:46.728 INFO:tasks.ceph.mon.c:Restarting daemon 2021-02-10T11:21:46.729 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-02-10T11:21:46.731 INFO:tasks.ceph.mon.c:Started 2021-02-10T11:21:46.731 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-02-10T11:21:46.731 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-02-10T11:21:46.731 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-02-10T11:21:46.733 INFO:tasks.ceph.mgr.y:Started 2021-02-10T11:21:46.734 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-02-10T11:21:46.734 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-02-10T11:21:46.736 INFO:tasks.ceph.mgr.x:Started 2021-02-10T11:21:46.736 INFO:tasks.ceph:Setting crush tunables to default 2021-02-10T11:21:46.737 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --cluster ceph osd crush tunables default 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.140 7f2a5a535700 1 Processor -- start 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.140 7f2a5a535700 1 -- start start 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a5a535700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a5a535700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 0x7f2a541b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a5a535700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a54109ab0 con 0x7f2a541b3470 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a5a535700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a540d0ec0 con 0x7f2a540d0ac0 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a5a535700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540d10b0 con 0x7f2a540e51a0 2021-02-10T11:21:47.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a53fff700 1 -- >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 msgr2=0x7f2a541b2e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.117:3300/0 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a53fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.141 7f2a537fe700 1 -- >> v1:172.21.2.117:6789/0 conn(0x7f2a541b3470 legacy=0x7f2a541085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.2.117:6789/0 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.142 7f2a58b52700 1 -- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 msgr2=0x7f2a541b3870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.125:3301/0 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.142 7f2a58b52700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 0x7f2a541b3870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a53fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a537fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2a541b3470 0x7f2a541085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41068/0 (socket says 172.21.2.117:41068) 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a537fe700 1 -- 172.21.2.117:0/3717815939 learned_addr learned my addr 172.21.2.117:0/3717815939 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a53fff700 1 -- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 msgr2=0x7f2a541b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a53fff700 1 -- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 msgr2=0x7f2a541b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a53fff700 1 --2- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.342 7f2a53fff700 1 --2- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:21:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.343 7f2a58b52700 1 -- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 msgr2=0x7f2a541b3870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.2.125:3301/0 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.343 7f2a58b52700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 0x7f2a541b3870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226569430 0 0) 0x7f2a480035b0 con 0x7f2a541b3470 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a3c002fe0 con 0x7f2a541b3470 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 342509244 0 0) 0x7f2a480035b0 con 0x7f2a541b3470 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 msgr2=0x7f2a541b3870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a540e51a0 0x7f2a541b3870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 msgr2=0x7f2a541b2e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 --2- 172.21.2.117:0/3717815939 >> v2:172.21.2.117:3300/0 conn(0x7f2a540d0ac0 0x7f2a541b2e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5410b040 con 0x7f2a541b3470 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2a48003b30 con 0x7f2a541b3470 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 >> v1:172.21.2.117:6789/0 conn(0x7f2a541b3470 legacy=0x7f2a541085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a3c003a40 0x7f2a3c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a3c04a7a0 0x7f2a3c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a3c04d510 0x7f2a3c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0502e0 con 0x7f2a3c04d510 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c050500 con 0x7f2a3c003a40 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c050720 con 0x7f2a3c04a7a0 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a58b52700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a3c04d510 0x7f2a3c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.541 7f2a537fe700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a3c04a7a0 0x7f2a3c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a53fff700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a3c003a40 0x7f2a3c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 -- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a3c04a7a0 msgr2=0x7f2a3c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a3c04a7a0 0x7f2a3c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 -- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a3c003a40 msgr2=0x7f2a3c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a3c003a40 0x7f2a3c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 -- 172.21.2.117:0/3717815939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c0509c0 con 0x7f2a3c04d510 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a3c04d510 0x7f2a3c04fa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2a4c0038b0 tx=0x7f2a4c009dc0).ready entity=mon.0 client_cookie=a7c687b716d8992f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a4c006aa0 con 0x7f2a3c04d510 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c003690 con 0x7f2a3c04d510 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a52ffd700 1 -- 172.21.2.117:0/3717815939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a4c000f90 con 0x7f2a3c04d510 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a5a535700 1 -- 172.21.2.117:0/3717815939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a3c04d510 msgr2=0x7f2a3c04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a5a535700 1 --2- 172.21.2.117:0/3717815939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a3c04d510 0x7f2a3c04fa30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f2a4c0038b0 tx=0x7f2a4c009dc0).stop 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.542 7f2a58b52700 1 -- 172.21.2.117:0/3717815939 reap_dead start 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.543 7f2a5a535700 1 -- 172.21.2.117:0/3717815939 shutdown_connections 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.543 7f2a5a535700 1 -- 172.21.2.117:0/3717815939 >> 172.21.2.117:0/3717815939 conn(0x7f2a540ce410 msgr2=0x7f2a540d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.543 7f2a5a535700 1 -- 172.21.2.117:0/3717815939 shutdown_connections 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.543 7f2a5a535700 1 -- 172.21.2.117:0/3717815939 wait complete. 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 Processor -- start 2021-02-10T11:21:47.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 -- start start 2021-02-10T11:21:47.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 0x7f2a540bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a540911f0 0x7f2a540915f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540bc010 con 0x7f2a540bbc10 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540bc230 con 0x7f2a540911f0 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.559 7f2a5a535700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54092340 con 0x7f2a541a4850 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:40850/0 (socket says 172.21.2.117:40850) 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 -- 172.21.2.117:0/248775613 learned_addr learned my addr 172.21.2.117:0/248775613 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a58b52700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a540911f0 0x7f2a540915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a53fff700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 0x7f2a540bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 -- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 msgr2=0x7f2a540bb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 0x7f2a540bb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 -- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a540911f0 msgr2=0x7f2a540915f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a540911f0 0x7f2a540915f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 -- 172.21.2.117:0/248775613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54092560 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a53fff700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 0x7f2a540bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a537fe700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2a48007560 tx=0x7f2a4800a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a480035c0 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4800b070 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a5a535700 1 -- 172.21.2.117:0/248775613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a54092560 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a4800b500 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a5a535700 1 -- 172.21.2.117:0/248775613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a54092560 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 1) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2a4800ad10 con 0x7f2a540bbc10 2021-02-10T11:21:47.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.560 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2a4800b850 con 0x7f2a540bbc10 2021-02-10T11:21:47.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.592 7f2a5a535700 1 -- 172.21.2.117:0/248775613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a540eda30 con 0x7f2a540bbc10 2021-02-10T11:21:47.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.594 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66016 (secure 0 0 0) 0x7f2a480032f0 con 0x7f2a540bbc10 2021-02-10T11:21:47.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.665 7f2a5a535700 1 -- 172.21.2.117:0/248775613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f2a540eda30 con 0x7f2a540bbc10 2021-02-10T11:21:47.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.668 7f2a50ff9700 1 -- 172.21.2.117:0/248775613 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2a48003500 con 0x7f2a540bbc10 2021-02-10T11:21:47.665 INFO:teuthology.orchestra.run.gibba017.stderr:adjusted tunables profile to default 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 -- 172.21.2.117:0/248775613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 msgr2=0x7f2a54090bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f2a48007560 tx=0x7f2a4800a920).stop 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 -- 172.21.2.117:0/248775613 shutdown_connections 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a540bbc10 0x7f2a54090bd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a540911f0 0x7f2a540915f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 --2- 172.21.2.117:0/248775613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a541a4850 0x7f2a540bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:21:47.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 -- 172.21.2.117:0/248775613 >> 172.21.2.117:0/248775613 conn(0x7f2a540b5b80 msgr2=0x7f2a540edc60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:21:47.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 -- 172.21.2.117:0/248775613 shutdown_connections 2021-02-10T11:21:47.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:21:47.669 7f2a5a535700 1 -- 172.21.2.117:0/248775613 wait complete. 2021-02-10T11:21:47.676 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-02-10T11:21:47.676 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:48.010 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.XtEDCrUKD0 2021-02-10T11:21:48.010 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.XtEDCrUKD0 2021-02-10T11:21:48.353 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.XtEDCrUKD0 2021-02-10T11:21:49.032 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.XtEDCrUKD0 is 37B 2021-02-10T11:21:49.372 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.XtEDCrUKD0 2021-02-10T11:21:49.706 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:50.040 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.MqBABDYxey 2021-02-10T11:21:50.041 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.MqBABDYxey 2021-02-10T11:21:50.384 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.MqBABDYxey 2021-02-10T11:21:51.062 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.MqBABDYxey is 37B 2021-02-10T11:21:51.401 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.MqBABDYxey 2021-02-10T11:21:51.591 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:51.594 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:21:51.848 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:52.182 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.ao1mtHyNSy 2021-02-10T11:21:52.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.ao1mtHyNSy 2021-02-10T11:21:52.525 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.ao1mtHyNSy 2021-02-10T11:21:53.202 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.ao1mtHyNSy is 37B 2021-02-10T11:21:53.543 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.ao1mtHyNSy 2021-02-10T11:21:53.877 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T11:21:54.210 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.v5eGFJ8Ha0 2021-02-10T11:21:54.211 DEBUG:teuthology.orchestra.run.gibba017:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.v5eGFJ8Ha0 2021-02-10T11:21:54.554 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.v5eGFJ8Ha0 2021-02-10T11:21:55.254 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.v5eGFJ8Ha0 is 37B 2021-02-10T11:22:00.892 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.v5eGFJ8Ha0 2021-02-10T11:22:01.226 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:22:01.562 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.BYdUs2ZgQ8 2021-02-10T11:22:01.563 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.BYdUs2ZgQ8 2021-02-10T11:22:01.906 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.BYdUs2ZgQ8 2021-02-10T11:22:02.586 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.BYdUs2ZgQ8 is 37B 2021-02-10T11:22:02.926 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.BYdUs2ZgQ8 2021-02-10T11:22:03.261 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:22:03.595 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.vwcObS7fE3 2021-02-10T11:22:03.596 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.vwcObS7fE3 2021-02-10T11:22:03.939 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.vwcObS7fE3 2021-02-10T11:22:04.621 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.vwcObS7fE3 is 37B 2021-02-10T11:22:04.963 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.vwcObS7fE3 2021-02-10T11:22:05.297 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:22:05.631 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.a9TuS3r29k 2021-02-10T11:22:05.632 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.a9TuS3r29k 2021-02-10T11:22:05.978 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.a9TuS3r29k 2021-02-10T11:22:06.659 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.a9TuS3r29k is 37B 2021-02-10T11:22:06.999 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.a9TuS3r29k 2021-02-10T11:22:07.334 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T11:22:07.668 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.Bmxb9d1Q7P 2021-02-10T11:22:07.669 DEBUG:teuthology.orchestra.run.gibba025:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.Bmxb9d1Q7P 2021-02-10T11:22:08.012 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.Bmxb9d1Q7P 2021-02-10T11:22:08.692 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.Bmxb9d1Q7P is 37B 2021-02-10T11:22:09.030 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.Bmxb9d1Q7P 2021-02-10T11:22:09.366 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new b2b85857-0d25-4760-8348-b4e0feb34706 0 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.777 7fbfdf3d8700 1 Processor -- start 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 -- start start 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 0x7fbfd80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbfd80e4d80 0x7fbfd81b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbfd80e5180 0x7fbfd8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd80e5c00 con 0x7fbfd80e5180 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd8077a70 con 0x7fbfd80e4d80 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdf3d8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80777d0 con 0x7fbfd807da20 2021-02-10T11:22:09.794 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdc9f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbfd80e4d80 0x7fbfd81b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdd1f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 0x7fbfd80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdd1f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 0x7fbfd80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:55966/0 (socket says 172.21.2.125:55966) 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdc9f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbfd80e4d80 0x7fbfd81b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39320/0 (socket says 172.21.2.125:39320) 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.778 7fbfdd1f4700 1 -- 172.21.2.125:0/1580625755 learned_addr learned my addr 172.21.2.125:0/1580625755 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 -- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 msgr2=0x7fbfd80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 --2- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 0x7fbfd80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 -- 172.21.2.125:0/1580625755 >> v2:172.21.2.117:3300/0 conn(0x7fbfd80e5180 msgr2=0x7fbfd8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 --2- 172.21.2.125:0/1580625755 >> v2:172.21.2.117:3300/0 conn(0x7fbfd80e5180 0x7fbfd8077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 -- 172.21.2.125:0/1580625755 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd807de20 con 0x7fbfd80e4d80 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdc9f3700 1 --2- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80e4d80 0x7fbfd81b00d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbfc8006650 tx=0x7fbfc8009a80).ready entity=mon.1 client_cookie=ba01e3763bf42fbe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.795 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfcffff700 1 -- 172.21.2.125:0/1580625755 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfc80009b0 con 0x7fbfd80e4d80 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfcffff700 1 -- 172.21.2.125:0/1580625755 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc800c0c0 con 0x7fbfd80e4d80 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfcffff700 1 -- 172.21.2.125:0/1580625755 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfc800c5e0 con 0x7fbfd80e4d80 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdf3d8700 1 -- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80e4d80 msgr2=0x7fbfd81b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.779 7fbfdf3d8700 1 --2- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80e4d80 0x7fbfd81b00d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fbfc8006650 tx=0x7fbfc8009a80).stop 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 -- 172.21.2.125:0/1580625755 shutdown_connections 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 --2- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80e4d80 0x7fbfd81b00d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 --2- 172.21.2.125:0/1580625755 >> v2:172.21.2.117:3300/0 conn(0x7fbfd80e5180 0x7fbfd8077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 --2- 172.21.2.125:0/1580625755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd807da20 0x7fbfd80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 -- 172.21.2.125:0/1580625755 >> 172.21.2.125:0/1580625755 conn(0x7fbfd80ce430 msgr2=0x7fbfd80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 -- 172.21.2.125:0/1580625755 shutdown_connections 2021-02-10T11:22:09.796 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.780 7fbfdf3d8700 1 -- 172.21.2.125:0/1580625755 wait complete. 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 Processor -- start 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 -- start start 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd809e660 0x7fbfd809dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbfd8187bd0 0x7fbfd8187fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd8188a70 con 0x7fbfd8187bd0 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80a3180 con 0x7fbfd80bbe90 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.795 7fbfdf3d8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd81885f0 con 0x7fbfd809e660 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdd1f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd809e660 0x7fbfd809dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.798 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39326/0 (socket says 172.21.2.125:39326) 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 -- 172.21.2.125:0/2835729460 learned_addr learned my addr 172.21.2.125:0/2835729460 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 -- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd809e660 msgr2=0x7fbfd809dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd809e660 0x7fbfd809dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 -- 172.21.2.125:0/2835729460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbfd8187bd0 msgr2=0x7fbfd8187fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbfd8187bd0 0x7fbfd8187fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 -- 172.21.2.125:0/2835729460 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8188810 con 0x7fbfd80bbe90 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdc9f3700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbfc8009630 tx=0x7fbfc800b980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfc800bd70 con 0x7fbfd80bbe90 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc8000960 con 0x7fbfd80bbe90 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbfc80054d0 con 0x7fbfd80bbe90 2021-02-10T11:22:09.799 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd8188810 con 0x7fbfd80bbe90 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.796 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd81892e0 con 0x7fbfd80bbe90 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.797 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbfc800a000 con 0x7fbfd80bbe90 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.797 7fbfcdffb700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 0x7fbfb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.797 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbfc8031220 con 0x7fbfd80bbe90 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.797 7fbfdd1f4700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 0x7fbfb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:09.800 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.797 7fbfdd1f4700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 0x7fbfb801abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:09.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.828 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd8103300 con 0x7fbfd80bbe90 2021-02-10T11:22:09.830 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.830 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbfc8024fe0 con 0x7fbfd80bbe90 2021-02-10T11:22:09.967 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.968 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", "id": 0} v 0) v1 -- 0x7fbfd80a3180 con 0x7fbfd80bbe90 2021-02-10T11:22:09.972 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.972 7fbfcdffb700 1 -- 172.21.2.125:0/2835729460 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbfc8024fe0 con 0x7fbfd80bbe90 2021-02-10T11:22:09.974 INFO:teuthology.orchestra.run.gibba025.stdout:0 2021-02-10T11:22:09.975 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.976 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 msgr2=0x7fbfb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.976 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 0x7fbfb801abc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.976 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 msgr2=0x7fbfd80bc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.976 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fbfc8009630 tx=0x7fbfc800b980).stop 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 shutdown_connections 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbfb80186c0 0x7fbfb801abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbfd8187bd0 0x7fbfd8187fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:09.976 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbfd80bbe90 0x7fbfd80bc290 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.977 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 --2- 172.21.2.125:0/2835729460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbfd809e660 0x7fbfd809dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:09.977 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 >> 172.21.2.125:0/2835729460 conn(0x7fbfd819f550 msgr2=0x7fbfd8103530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:09.977 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 shutdown_connections 2021-02-10T11:22:09.977 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:09.977 7fbfdf3d8700 1 -- 172.21.2.125:0/2835729460 wait complete. 2021-02-10T11:22:09.984 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new 4bb369d2-72db-4361-97af-cd8850006a87 1 2021-02-10T11:22:10.411 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.394 7f20781a3700 1 Processor -- start 2021-02-10T11:22:10.411 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.394 7f20781a3700 1 -- start start 2021-02-10T11:22:10.411 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20781a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.411 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20781a3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20700e4d80 0x7f20701b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.411 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20781a3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070077a70 con 0x7f20700e5180 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20781a3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700e5c00 con 0x7f20700e4d80 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20781a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20700777d0 con 0x7f207007da20 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20700e4d80 0x7f20701b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20700e4d80 0x7f20701b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39332/0 (socket says 172.21.2.125:39332) 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f2075fbf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 -- 172.21.2.125:0/1916721947 learned_addr learned my addr 172.21.2.125:0/1916721947 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20767c0700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f20700e5180 0x7f2070077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.125:40792/0 (socket says 172.21.2.125:40792) 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 -- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 msgr2=0x7f20700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 -- 172.21.2.125:0/1916721947 >> v1:172.21.2.125:6789/0 conn(0x7f20700e5180 legacy=0x7f2070077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.412 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.395 7f20757be700 1 -- 172.21.2.125:0/1916721947 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207007de20 con 0x7f20700e4d80 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f2075fbf700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f20757be700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20700e4d80 0x7f20701b00d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f206c006700 tx=0x7f206c009b80).ready entity=mon.1 client_cookie=6694fd1064920817 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f2074fbd700 1 -- 172.21.2.125:0/1916721947 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f206c00c0c0 con 0x7f20700e4d80 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f2074fbd700 1 -- 172.21.2.125:0/1916721947 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c00c380 con 0x7f20700e4d80 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f2074fbd700 1 -- 172.21.2.125:0/1916721947 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f206c00c8c0 con 0x7f20700e4d80 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f20781a3700 1 -- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20700e4d80 msgr2=0x7f20701b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.396 7f20781a3700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20700e4d80 0x7f20701b00d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f206c006700 tx=0x7f206c009b80).stop 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 -- 172.21.2.125:0/1916721947 shutdown_connections 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20700e4d80 0x7f20701b00d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.413 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 --2- 172.21.2.125:0/1916721947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.414 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 -- 172.21.2.125:0/1916721947 >> 172.21.2.125:0/1916721947 conn(0x7f20700ce430 msgr2=0x7f20700e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:10.414 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 -- 172.21.2.125:0/1916721947 shutdown_connections 2021-02-10T11:22:10.414 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.397 7f20781a3700 1 -- 172.21.2.125:0/1916721947 wait complete. 2021-02-10T11:22:10.414 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 Processor -- start 2021-02-10T11:22:10.414 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 -- start start 2021-02-10T11:22:10.415 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20701903e0 0x7f2070090a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 0x7f20701000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20701907e0 con 0x7f20701903e0 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2070101040 con 0x7f2070091080 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20781a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2070101fe0 con 0x7f2070101be0 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20767c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 0x7f20701000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20757be700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20767c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 0x7f20701000d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:55984/0 (socket says 172.21.2.125:55984) 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20767c0700 1 -- 172.21.2.125:0/3742531192 learned_addr learned my addr 172.21.2.125:0/3742531192 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:10.416 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.412 7f20757be700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39342/0 (socket says 172.21.2.125:39342) 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 -- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 msgr2=0x7f20701000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 0x7f20701000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 -- 172.21.2.125:0/3742531192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20701903e0 msgr2=0x7f2070090a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20701903e0 0x7f2070090a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 -- 172.21.2.125:0/3742531192 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20701021c0 con 0x7f2070091080 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20757be700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f206c006cf0 tx=0x7f206c00b050).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f206c00b440 con 0x7f2070091080 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c00b680 con 0x7f2070091080 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f206c00bba0 con 0x7f2070091080 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20781a3700 1 -- 172.21.2.125:0/3742531192 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20701021c0 con 0x7f2070091080 2021-02-10T11:22:10.417 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.413 7f20781a3700 1 -- 172.21.2.125:0/3742531192 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20701021c0 con 0x7f2070091080 2021-02-10T11:22:10.418 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.414 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f206c000c90 con 0x7f2070091080 2021-02-10T11:22:10.418 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.414 7f20627fc700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 0x7f205801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:10.418 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.414 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 857+0+0 (secure 0 0 0) 0x7f206c031480 con 0x7f2070091080 2021-02-10T11:22:10.418 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.414 7f2075fbf700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 0x7f205801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:10.418 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.414 7f2075fbf700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 0x7f205801abd0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:10.444 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.445 7f20781a3700 1 -- 172.21.2.125:0/3742531192 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2070101040 con 0x7f2070091080 2021-02-10T11:22:10.446 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.447 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f206c00b440 con 0x7f2070091080 2021-02-10T11:22:10.586 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.587 7f20781a3700 1 -- 172.21.2.125:0/3742531192 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", "id": 1} v 0) v1 -- 0x7f2070101040 con 0x7f2070091080 2021-02-10T11:22:10.592 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.592 7f20627fc700 1 -- 172.21.2.125:0/3742531192 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f206c000c90 con 0x7f2070091080 2021-02-10T11:22:10.593 INFO:teuthology.orchestra.run.gibba025.stdout:1 2021-02-10T11:22:10.595 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 msgr2=0x7f205801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 0x7f205801abd0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 msgr2=0x7f20701015c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f206c006cf0 tx=0x7f206c00b050).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 shutdown_connections 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2070101be0 0x7f20701000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070091080 0x7f20701015c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20580186d0 0x7f205801abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 --2- 172.21.2.125:0/3742531192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20701903e0 0x7f2070090a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 >> 172.21.2.125:0/3742531192 conn(0x7f2070199470 msgr2=0x7f2070098e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 shutdown_connections 2021-02-10T11:22:10.596 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:10.596 7f20781a3700 1 -- 172.21.2.125:0/3742531192 wait complete. 2021-02-10T11:22:10.604 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new 63367d7e-7197-4d2e-90b2-2b5cea7b4304 2 2021-02-10T11:22:11.029 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 Processor -- start 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 -- start start 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f65d4079000 0x7f65d40e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f65d41b00d0 0x7f65d41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d4077a70 con 0x7f65d407da20 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40e5c00 con 0x7f65d4079000 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65db1a1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d40e53a0 con 0x7f65d41b00d0 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65d8fbd700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f65d407da20 0x7f65d40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.125:40802/0 (socket says 172.21.2.125:40802) 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65d8fbd700 1 -- 172.21.2.125:0/3451249203 learned_addr learned my addr 172.21.2.125:0/3451249203 (peer_addr_for_me v1:172.21.2.125:0/0) 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.013 7f65d97be700 1 --2- 172.21.2.125:0/3451249203 >> v2:172.21.2.125:3300/0 conn(0x7f65d41b00d0 0x7f65d41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d97be700 1 -- 172.21.2.125:0/3451249203 >> v2:172.21.2.117:3300/0 conn(0x7f65d4079000 msgr2=0x7f65d40e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:11.030 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d97be700 1 --2- 172.21.2.125:0/3451249203 >> v2:172.21.2.117:3300/0 conn(0x7f65d4079000 0x7f65d40e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d97be700 1 -- 172.21.2.125:0/3451249203 >> v1:172.21.2.125:6789/0 conn(0x7f65d407da20 legacy=0x7f65d40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d97be700 1 -- 172.21.2.125:0/3451249203 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d4079400 con 0x7f65d41b00d0 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d37fe700 1 -- 172.21.2.125:0/3451249203 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476860736 0 0) 0x7f65bc001180 con 0x7f65d407da20 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d97be700 1 --2- 172.21.2.125:0/3451249203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41b00d0 0x7f65d41b04d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f65cc006e30 tx=0x7f65cc009e60).ready entity=mon.1 client_cookie=a924c6ae364f32fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d37fe700 1 -- 172.21.2.125:0/3451249203 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65cc006a50 con 0x7f65d41b00d0 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d37fe700 1 -- 172.21.2.125:0/3451249203 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00b300 con 0x7f65d41b00d0 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65d37fe700 1 -- 172.21.2.125:0/3451249203 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65cc00b820 con 0x7f65d41b00d0 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65db1a1700 1 -- 172.21.2.125:0/3451249203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41b00d0 msgr2=0x7f65d41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.014 7f65db1a1700 1 --2- 172.21.2.125:0/3451249203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41b00d0 0x7f65d41b04d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f65cc006e30 tx=0x7f65cc009e60).stop 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 -- 172.21.2.125:0/3451249203 shutdown_connections 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 --2- 172.21.2.125:0/3451249203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41b00d0 0x7f65d41b04d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 --2- 172.21.2.125:0/3451249203 >> v2:172.21.2.117:3300/0 conn(0x7f65d4079000 0x7f65d40e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:11.031 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 -- 172.21.2.125:0/3451249203 >> 172.21.2.125:0/3451249203 conn(0x7f65d40ce430 msgr2=0x7f65d40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:11.032 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 -- 172.21.2.125:0/3451249203 shutdown_connections 2021-02-10T11:22:11.032 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.016 7f65db1a1700 1 -- 172.21.2.125:0/3451249203 wait complete. 2021-02-10T11:22:11.032 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.030 7f65db1a1700 1 Processor -- start 2021-02-10T11:22:11.032 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.030 7f65db1a1700 1 -- start start 2021-02-10T11:22:11.033 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.033 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41a5780 0x7f65d40b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.033 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65d40b1d50 0x7f65d419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d419d410 con 0x7f65d40b1d50 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d419fb70 con 0x7f65d41a5780 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65db1a1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40b2150 con 0x7f65d419a370 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65d8fbd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65d8fbd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:55998/0 (socket says 172.21.2.125:55998) 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65d8fbd700 1 -- 172.21.2.125:0/1972423967 learned_addr learned my addr 172.21.2.125:0/1972423967 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65d3fff700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41a5780 0x7f65d40b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.031 7f65d97be700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65d40b1d50 0x7f65d419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 -- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41a5780 msgr2=0x7f65d40b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41a5780 0x7f65d40b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 -- 172.21.2.125:0/1972423967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65d40b1d50 msgr2=0x7f65d419d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65d40b1d50 0x7f65d419d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.034 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 -- 172.21.2.125:0/1972423967 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40b2370 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d8fbd700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f65bc00a570 tx=0x7f65bc00a910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc00ad00 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc001180 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d40b2370 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65bc004590 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.032 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d40b2370 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.033 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f65bc0043a0 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.033 7f65d17fa700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 0x7f65b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.033 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 960+0+0 (secure 0 0 0) 0x7f65bc0315b0 con 0x7f65d419a370 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.033 7f65d3fff700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 0x7f65b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.035 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.033 7f65d3fff700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 0x7f65b801ac10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.062 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.063 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d4077a70 con 0x7f65d419a370 2021-02-10T11:22:11.064 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.066 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f65bc000c90 con 0x7f65d419a370 2021-02-10T11:22:11.203 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.203 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", "id": 2} v 0) v1 -- 0x7f65d419fb20 con 0x7f65d419a370 2021-02-10T11:22:11.207 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.208 7f65d17fa700 1 -- 172.21.2.125:0/1972423967 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f65bc001160 con 0x7f65d419a370 2021-02-10T11:22:11.210 INFO:teuthology.orchestra.run.gibba025.stdout:2 2021-02-10T11:22:11.211 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 msgr2=0x7f65b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.211 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 0x7f65b801ac10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.211 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 msgr2=0x7f65d41a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.211 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f65bc00a570 tx=0x7f65bc00a910).stop 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 shutdown_connections 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65b8018730 0x7f65b801ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65d40b1d50 0x7f65d419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65d41a5780 0x7f65d40b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 --2- 172.21.2.125:0/1972423967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65d419a370 0x7f65d41a5160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 >> 172.21.2.125:0/1972423967 conn(0x7f65d418cc20 msgr2=0x7f65d419a920 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 shutdown_connections 2021-02-10T11:22:11.212 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.212 7f65db1a1700 1 -- 172.21.2.125:0/1972423967 wait complete. 2021-02-10T11:22:11.219 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new 2c9e4d21-9bfc-4107-ba32-853e096e8eb9 3 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 Processor -- start 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 -- start start 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 0x7fa4f40e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa4f41b00d0 0x7fa4f41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f4077a70 con 0x7fa4f407da20 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f40e5c00 con 0x7fa4f4079000 2021-02-10T11:22:11.645 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.629 7fa4fc0f3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4f40e53a0 con 0x7fa4f41b00d0 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa4f41b00d0 0x7fa4f41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa4f41b00d0 0x7fa4f41b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39364/0 (socket says 172.21.2.125:39364) 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 -- 172.21.2.125:0/1386703641 learned_addr learned my addr 172.21.2.125:0/1386703641 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4f970e700 1 --2- 172.21.2.125:0/1386703641 >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 0x7fa4f40e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 -- 172.21.2.125:0/1386703641 >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 msgr2=0x7fa4f40e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 --2- 172.21.2.125:0/1386703641 >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 0x7fa4f40e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 -- 172.21.2.125:0/1386703641 >> v1:172.21.2.117:6789/0 conn(0x7fa4f407da20 legacy=0x7fa4f40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 -- 172.21.2.125:0/1386703641 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f4079400 con 0x7fa4f41b00d0 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4f970e700 1 --2- 172.21.2.125:0/1386703641 >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 0x7fa4f40e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:11.646 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.630 7fa4fa710700 1 --2- 172.21.2.125:0/1386703641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f41b00d0 0x7fa4f41b04d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa4ec006f00 tx=0x7fa4ec009f10).ready entity=mon.1 client_cookie=2d8948ce69f16933 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4f8f0d700 1 -- 172.21.2.125:0/1386703641 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ec00b3b0 con 0x7fa4f41b00d0 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4f8f0d700 1 -- 172.21.2.125:0/1386703641 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec00b630 con 0x7fa4f41b00d0 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4f8f0d700 1 -- 172.21.2.125:0/1386703641 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4ec00bb50 con 0x7fa4f41b00d0 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 -- 172.21.2.125:0/1386703641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f41b00d0 msgr2=0x7fa4f41b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 --2- 172.21.2.125:0/1386703641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f41b00d0 0x7fa4f41b04d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa4ec006f00 tx=0x7fa4ec009f10).stop 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 -- 172.21.2.125:0/1386703641 shutdown_connections 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 --2- 172.21.2.125:0/1386703641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f41b00d0 0x7fa4f41b04d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 --2- 172.21.2.125:0/1386703641 >> v2:172.21.2.117:3300/0 conn(0x7fa4f4079000 0x7fa4f40e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.631 7fa4fc0f3700 1 -- 172.21.2.125:0/1386703641 >> 172.21.2.125:0/1386703641 conn(0x7fa4f40ce430 msgr2=0x7fa4f40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.632 7fa4fc0f3700 1 -- 172.21.2.125:0/1386703641 shutdown_connections 2021-02-10T11:22:11.647 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.632 7fa4fc0f3700 1 -- 172.21.2.125:0/1386703641 wait complete. 2021-02-10T11:22:11.649 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 Processor -- start 2021-02-10T11:22:11.649 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 -- start start 2021-02-10T11:22:11.649 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f409e000 0x7fa4f41a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.649 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.649 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 0x7fa4f41a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f41a4360 con 0x7fa4f41a10d0 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f41a2860 con 0x7fa4f409e000 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.646 7fa4fc0f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f41a2240 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56012/0 (socket says 172.21.2.125:56012) 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 -- 172.21.2.125:0/2567522470 learned_addr learned my addr 172.21.2.125:0/2567522470 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f9f0f700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f409e000 0x7fa4f41a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4fa710700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 0x7fa4f41a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 -- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f409e000 msgr2=0x7fa4f41a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f409e000 0x7fa4f41a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 -- 172.21.2.125:0/2567522470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 msgr2=0x7fa4f41a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 0x7fa4f41a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.650 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 -- 172.21.2.125:0/2567522470 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f41a2420 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4fa710700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 0x7fa4f41a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.647 7fa4f970e700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa4f0006f20 tx=0x7fa4f000b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4f000bd80 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f0000c90 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f41a2420 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4f00022b0 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f41a2420 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa4f000a000 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 0x7fa4dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.648 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1063+0+0 (secure 0 0 0) 0x7fa4f00315f0 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.649 7fa4f9f0f700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 0x7fa4dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:11.651 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.649 7fa4f9f0f700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 0x7fa4dc01abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:11.678 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.680 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f4077a70 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.681 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.682 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa4f0006bb0 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.818 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.818 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", "id": 3} v 0) v1 -- 0x7fa4f4077a70 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.823 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.824 7fa4e67fc700 1 -- 172.21.2.125:0/2567522470 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa4f000bd80 con 0x7fa4f41a0cd0 2021-02-10T11:22:11.825 INFO:teuthology.orchestra.run.gibba025.stdout:3 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 msgr2=0x7fa4dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 0x7fa4dc01abc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 msgr2=0x7fa4f41a1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa4f0006f20 tx=0x7fa4f000b990).stop 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 shutdown_connections 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4dc018730 0x7fa4dc01abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4f41a10d0 0x7fa4f41a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.827 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4f41a0cd0 0x7fa4f41a1600 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.828 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 --2- 172.21.2.125:0/2567522470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4f409e000 0x7fa4f41a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:11.828 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 >> 172.21.2.125:0/2567522470 conn(0x7fa4f4199470 msgr2=0x7fa4f419f660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:11.828 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 shutdown_connections 2021-02-10T11:22:11.828 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:11.828 7fa4fc0f3700 1 -- 172.21.2.125:0/2567522470 wait complete. 2021-02-10T11:22:11.835 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new be969e8d-b93d-43e0-811f-7fb5288876a7 4 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.250 7f4b3e58b700 1 Processor -- start 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.250 7f4b3e58b700 1 -- start start 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.250 7f4b3e58b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b3807da20 0x7f4b380771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.250 7f4b3e58b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.250 7f4b3e58b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b380e5180 0x7f4b38077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b3e58b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b38077a70 con 0x7f4b380e5180 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b3e58b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b380777d0 con 0x7f4b3807da20 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b3e58b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3807de20 con 0x7f4b380e4d80 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b377fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.267 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b3807da20 0x7f4b380771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b377fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56020/0 (socket says 172.21.2.125:56020) 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b3807da20 0x7f4b380771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39374/0 (socket says 172.21.2.125:39374) 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b377fe700 1 -- 172.21.2.125:0/1767962344 learned_addr learned my addr 172.21.2.125:0/1767962344 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 -- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 msgr2=0x7f4b381b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 -- 172.21.2.125:0/1767962344 >> v2:172.21.2.117:3300/0 conn(0x7f4b380e5180 msgr2=0x7f4b38077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 --2- 172.21.2.125:0/1767962344 >> v2:172.21.2.117:3300/0 conn(0x7f4b380e5180 0x7f4b38077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 -- 172.21.2.125:0/1767962344 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b381b06f0 con 0x7f4b3807da20 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b377fe700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.251 7f4b37fff700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b3807da20 0x7f4b380771b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4b28006650 tx=0x7f4b28009a80).ready entity=mon.1 client_cookie=33d60b2d0397fa7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.252 7f4b36ffd700 1 -- 172.21.2.125:0/1767962344 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b280009b0 con 0x7f4b3807da20 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.252 7f4b36ffd700 1 -- 172.21.2.125:0/1767962344 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800c0c0 con 0x7f4b3807da20 2021-02-10T11:22:12.268 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.252 7f4b3e58b700 1 -- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b3807da20 msgr2=0x7f4b380771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.252 7f4b3e58b700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b3807da20 0x7f4b380771b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f4b28006650 tx=0x7f4b28009a80).stop 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.252 7f4b36ffd700 1 -- 172.21.2.125:0/1767962344 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2800c5e0 con 0x7f4b3807da20 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 -- 172.21.2.125:0/1767962344 shutdown_connections 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 --2- 172.21.2.125:0/1767962344 >> v2:172.21.2.117:3300/0 conn(0x7f4b380e5180 0x7f4b38077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b380e4d80 0x7f4b381b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 --2- 172.21.2.125:0/1767962344 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b3807da20 0x7f4b380771b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 -- 172.21.2.125:0/1767962344 >> 172.21.2.125:0/1767962344 conn(0x7f4b380ce430 msgr2=0x7f4b380e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 -- 172.21.2.125:0/1767962344 shutdown_connections 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.253 7f4b3e58b700 1 -- 172.21.2.125:0/1767962344 wait complete. 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.267 7f4b3e58b700 1 Processor -- start 2021-02-10T11:22:12.269 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 -- start start 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 0x7f4b38196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3818e4d0 con 0x7f4b38195c30 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38097e60 con 0x7f4b381b5fe0 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b3e58b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38098040 con 0x7f4b3818ea50 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b37fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b377fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.271 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b377fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39378/0 (socket says 172.21.2.125:39378) 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b37fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56024/0 (socket says 172.21.2.125:56024) 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.268 7f4b377fe700 1 -- 172.21.2.125:0/865182228 learned_addr learned my addr 172.21.2.125:0/865182228 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b3cba8700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 0x7f4b38196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 -- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 msgr2=0x7f4b3818de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 -- 172.21.2.125:0/865182228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 msgr2=0x7f4b38196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 0x7f4b38196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 -- 172.21.2.125:0/865182228 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38196650 con 0x7f4b381b5fe0 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b3cba8700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 0x7f4b38196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b37fff700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b377fe700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4b2c006ee0 tx=0x7f4b2c00b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2c00bd50 con 0x7f4b381b5fe0 2021-02-10T11:22:12.272 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c0052e0 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2c005740 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b38196650 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.269 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b38196650 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.270 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b2c0054c0 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.270 7f4b34ff9700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 0x7f4b1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.270 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1166+0+0 (secure 0 0 0) 0x7f4b2c0314f0 con 0x7f4b381b5fe0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.270 7f4b37fff700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 0x7f4b1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.273 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.270 7f4b37fff700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 0x7f4b1801ac10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.300 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.300 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b381b4f80 con 0x7f4b381b5fe0 2021-02-10T11:22:12.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.303 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b2c0054f0 con 0x7f4b381b5fe0 2021-02-10T11:22:12.438 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.439 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", "id": 4} v 0) v1 -- 0x7f4b38077a70 con 0x7f4b381b5fe0 2021-02-10T11:22:12.444 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.445 7f4b34ff9700 1 -- 172.21.2.125:0/865182228 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4b2c0052e0 con 0x7f4b381b5fe0 2021-02-10T11:22:12.445 INFO:teuthology.orchestra.run.gibba025.stdout:4 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 msgr2=0x7f4b1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 0x7f4b1801ac10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 msgr2=0x7f4b381b63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4b2c006ee0 tx=0x7f4b2c00b960).stop 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 shutdown_connections 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b18018730 0x7f4b1801ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.447 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b38195c30 0x7f4b38196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.447 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.448 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b381b5fe0 0x7f4b381b63e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.448 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.448 7f4b3e58b700 1 --2- 172.21.2.125:0/865182228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b3818ea50 0x7f4b3818de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.448 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.448 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 >> 172.21.2.125:0/865182228 conn(0x7f4b38199470 msgr2=0x7f4b381a55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:12.448 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.448 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 shutdown_connections 2021-02-10T11:22:12.448 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.448 7f4b3e58b700 1 -- 172.21.2.125:0/865182228 wait complete. 2021-02-10T11:22:12.455 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new 931344cc-ab1d-4eee-b067-08b04f0d8824 5 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 Processor -- start 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 -- start start 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 0x7f854c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 0x7f854c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c077a70 con 0x7f854c0e5180 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c0777d0 con 0x7f854c07da20 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.865 7f85520f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c07de20 con 0x7f854c0e4d80 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854bfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f854c0e5180 0x7f854c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.125:40846/0 (socket says 172.21.2.125:40846) 2021-02-10T11:22:12.881 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854bfff700 1 -- 172.21.2.125:0/2229626404 learned_addr learned my addr 172.21.2.125:0/2229626404 (peer_addr_for_me v1:172.21.2.125:0/0) 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 --2- 172.21.2.125:0/2229626404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 0x7f854c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854b7fe700 1 --2- 172.21.2.125:0/2229626404 >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 0x7f854c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854a7fc700 1 -- 172.21.2.125:0/2229626404 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800441397 0 0) 0x7f8544001500 con 0x7f854c0e5180 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 -- 172.21.2.125:0/2229626404 >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 msgr2=0x7f854c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 --2- 172.21.2.125:0/2229626404 >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 0x7f854c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 -- 172.21.2.125:0/2229626404 >> v1:172.21.2.125:6789/0 conn(0x7f854c0e5180 legacy=0x7f854c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 -- 172.21.2.125:0/2229626404 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c1b06f0 con 0x7f854c0e4d80 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854b7fe700 1 --2- 172.21.2.125:0/2229626404 >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 0x7f854c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.866 7f854affd700 1 --2- 172.21.2.125:0/2229626404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 0x7f854c1b00d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8540006650 tx=0x7f8540009a80).ready entity=mon.2 client_cookie=860d5944948c3059 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f854a7fc700 1 -- 172.21.2.125:0/2229626404 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f854000c0c0 con 0x7f854c0e4d80 2021-02-10T11:22:12.882 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f854a7fc700 1 -- 172.21.2.125:0/2229626404 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854000c340 con 0x7f854c0e4d80 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f854a7fc700 1 -- 172.21.2.125:0/2229626404 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f854000c860 con 0x7f854c0e4d80 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 -- 172.21.2.125:0/2229626404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 msgr2=0x7f854c1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 --2- 172.21.2.125:0/2229626404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 0x7f854c1b00d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8540006650 tx=0x7f8540009a80).stop 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 -- 172.21.2.125:0/2229626404 shutdown_connections 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 --2- 172.21.2.125:0/2229626404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c0e4d80 0x7f854c1b00d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 --2- 172.21.2.125:0/2229626404 >> v2:172.21.2.117:3300/0 conn(0x7f854c07da20 0x7f854c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 -- 172.21.2.125:0/2229626404 >> 172.21.2.125:0/2229626404 conn(0x7f854c0ce430 msgr2=0x7f854c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 -- 172.21.2.125:0/2229626404 shutdown_connections 2021-02-10T11:22:12.883 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.867 7f85520f9700 1 -- 172.21.2.125:0/2229626404 wait complete. 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 Processor -- start 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 -- start start 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f854c1a81c0 0x7f854c1a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f854c1a8be0 0x7f854c0fe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c095b80 con 0x7f854c1a81c0 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1077f0 con 0x7f854c1a8be0 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f85520f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1a8fe0 con 0x7f854c095ee0 2021-02-10T11:22:12.885 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f854b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f854b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56038/0 (socket says 172.21.2.125:56038) 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f854b7fe700 1 -- 172.21.2.125:0/2318715547 learned_addr learned my addr 172.21.2.125:0/2318715547 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.882 7f854bfff700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f854c1a8be0 0x7f854c0fe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 -- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f854c1a8be0 msgr2=0x7f854c0fe0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f854c1a8be0 0x7f854c0fe0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 -- 172.21.2.125:0/2318715547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f854c1a81c0 msgr2=0x7f854c1a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f854c1a81c0 0x7f854c1a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 -- 172.21.2.125:0/2318715547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c1a9200 con 0x7f854c095ee0 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f854b7fe700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f853c006f20 tx=0x7f853c00b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f853c00bd80 con 0x7f854c095ee0 2021-02-10T11:22:12.886 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c000c90 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f853c002220 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f85520f9700 1 -- 172.21.2.125:0/2318715547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c1a9200 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.883 7f85520f9700 1 -- 172.21.2.125:0/2318715547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c0fe9c0 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.884 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f853c00a000 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.884 7f8533fff700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 0x7f852c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.884 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1269+0+0 (secure 0 0 0) 0x7f853c0315a0 con 0x7f854c095ee0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.884 7f854affd700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 0x7f852c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:12.887 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.884 7f854affd700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 0x7f852c01abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:12.914 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.915 7f85520f9700 1 -- 172.21.2.125:0/2318715547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c107e00 con 0x7f854c095ee0 2021-02-10T11:22:12.917 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:12.919 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f853c0009b0 con 0x7f854c095ee0 2021-02-10T11:22:13.054 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.055 7f85520f9700 1 -- 172.21.2.125:0/2318715547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", "id": 5} v 0) v1 -- 0x7f854c107e00 con 0x7f854c095ee0 2021-02-10T11:22:13.059 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.059 7f8533fff700 1 -- 172.21.2.125:0/2318715547 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f853c000c90 con 0x7f854c095ee0 2021-02-10T11:22:13.061 INFO:teuthology.orchestra.run.gibba025.stdout:5 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 msgr2=0x7f852c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 0x7f852c01abc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 msgr2=0x7f854c095560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f853c006f20 tx=0x7f853c00b990).stop 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 shutdown_connections 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f852c018700 0x7f852c01abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f854c1a81c0 0x7f854c1a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:13.063 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f854c1a8be0 0x7f854c0fe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.064 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 --2- 172.21.2.125:0/2318715547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f854c095ee0 0x7f854c095560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.064 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 >> 172.21.2.125:0/2318715547 conn(0x7f854c1b6b90 msgr2=0x7f854c1024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.064 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 shutdown_connections 2021-02-10T11:22:13.064 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.064 7f85520f9700 1 -- 172.21.2.125:0/2318715547 wait complete. 2021-02-10T11:22:13.071 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new f0852634-0814-469a-9e5a-ae92485dd59e 6 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 Processor -- start 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 -- start start 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd8007da20 0x7fcd800771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 0x7fcd80079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd80077a70 con 0x7fcd800e4d80 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd800777d0 con 0x7fcd8007da20 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.483 7fcd8651c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801b08c0 con 0x7fcd800e5180 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd8007da20 0x7fcd800771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd84b39700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 0x7fcd80079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd8007da20 0x7fcd800771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39402/0 (socket says 172.21.2.125:39402) 2021-02-10T11:22:13.499 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd84b39700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 0x7fcd80079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56046/0 (socket says 172.21.2.125:56046) 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 -- 172.21.2.125:0/2132422637 learned_addr learned my addr 172.21.2.125:0/2132422637 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 -- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 msgr2=0x7fcd80079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 --2- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 0x7fcd80079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 -- 172.21.2.125:0/2132422637 >> v1:172.21.2.117:6789/0 conn(0x7fcd800e4d80 legacy=0x7fcd801b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 -- 172.21.2.125:0/2132422637 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8007de20 con 0x7fcd8007da20 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7ffff700 1 --2- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd8007da20 0x7fcd800771b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcd70006700 tx=0x7fcd70009b80).ready entity=mon.1 client_cookie=2ea87965b2d2baff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7effd700 1 -- 172.21.2.125:0/2132422637 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd700009b0 con 0x7fcd8007da20 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7effd700 1 -- 172.21.2.125:0/2132422637 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7000c0c0 con 0x7fcd8007da20 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd7effd700 1 -- 172.21.2.125:0/2132422637 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd7000c560 con 0x7fcd8007da20 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd8651c700 1 -- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd8007da20 msgr2=0x7fcd800771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.484 7fcd8651c700 1 --2- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd8007da20 0x7fcd800771b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcd70006700 tx=0x7fcd70009b80).stop 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 -- 172.21.2.125:0/2132422637 shutdown_connections 2021-02-10T11:22:13.500 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 --2- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800e5180 0x7fcd80079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.501 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 --2- 172.21.2.125:0/2132422637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd8007da20 0x7fcd800771b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.501 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 -- 172.21.2.125:0/2132422637 >> 172.21.2.125:0/2132422637 conn(0x7fcd800ce430 msgr2=0x7fcd800e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.501 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 -- 172.21.2.125:0/2132422637 shutdown_connections 2021-02-10T11:22:13.501 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.485 7fcd8651c700 1 -- 172.21.2.125:0/2132422637 wait complete. 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 Processor -- start 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 -- start start 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 0x7fcd801a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd801a85d0 0x7fcd801a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80091190 con 0x7fcd80090d90 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80091380 con 0x7fcd801a85d0 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd8651c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd801a89d0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.500 7fcd7ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.125:56050/0 (socket says 172.21.2.125:56050) 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 -- 172.21.2.125:0/3048463466 learned_addr learned my addr 172.21.2.125:0/3048463466 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:13.503 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd84b39700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd801a85d0 0x7fcd801a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 -- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd801a85d0 msgr2=0x7fcd801a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd801a85d0 0x7fcd801a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 -- 172.21.2.125:0/3048463466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 msgr2=0x7fcd801a7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7f7fe700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 0x7fcd801a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 0x7fcd801a7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 -- 172.21.2.125:0/3048463466 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd801a8bf0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7f7fe700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 0x7fcd801a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7ffff700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcd70006fc0 tx=0x7fcd700025a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd70002990 con 0x7fcd800aa7b0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.501 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd700052e0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd801a8bf0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd70005780 con 0x7fcd800aa7b0 2021-02-10T11:22:13.504 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd801a8bf0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.505 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd700054c0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.505 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd7cff9700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 0x7fcd6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:13.505 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1372+0+0 (secure 0 0 0) 0x7fcd70031740 con 0x7fcd800aa7b0 2021-02-10T11:22:13.505 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.502 7fcd7f7fe700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 0x7fcd6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:13.505 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.503 7fcd7f7fe700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 0x7fcd6001abc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:13.532 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.533 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd801b4f80 con 0x7fcd800aa7b0 2021-02-10T11:22:13.535 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.536 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd700054f0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.676 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.676 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", "id": 6} v 0) v1 -- 0x7fcd801b4f80 con 0x7fcd800aa7b0 2021-02-10T11:22:13.681 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.682 7fcd7cff9700 1 -- 172.21.2.125:0/3048463466 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd700052e0 con 0x7fcd800aa7b0 2021-02-10T11:22:13.683 INFO:teuthology.orchestra.run.gibba025.stdout:6 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 msgr2=0x7fcd6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 0x7fcd6001abc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 msgr2=0x7fcd80090770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcd70006fc0 tx=0x7fcd700025a0).stop 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 shutdown_connections 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd60018730 0x7fcd6001abc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd80090d90 0x7fcd801a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd801a85d0 0x7fcd801a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.685 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 --2- 172.21.2.125:0/3048463466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd800aa7b0 0x7fcd80090770 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:13.686 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.685 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 >> 172.21.2.125:0/3048463466 conn(0x7fcd80199470 msgr2=0x7fcd80098e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:13.686 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.686 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 shutdown_connections 2021-02-10T11:22:13.686 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:13.686 7fcd8651c700 1 -- 172.21.2.125:0/3048463466 wait complete. 2021-02-10T11:22:13.693 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --cluster ceph osd new 20bf9222-b6e5-4644-930f-5caff0fc379a 7 2021-02-10T11:22:14.119 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.103 7f5d45805700 1 Processor -- start 2021-02-10T11:22:14.119 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.103 7f5d45805700 1 -- start start 2021-02-10T11:22:14.119 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.103 7f5d45805700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5d40079000 0x7f5d400e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.119 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.103 7f5d45805700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d401b00d0 0x7f5d401b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d45805700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d40077a70 con 0x7f5d4007da20 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d45805700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d400e5c00 con 0x7f5d401b00d0 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d45805700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d400e53a0 con 0x7f5d40079000 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5d40079000 0x7f5d400e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5d40079000 0x7f5d400e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39412/0 (socket says 172.21.2.125:39412) 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 -- 172.21.2.125:0/588286013 learned_addr learned my addr 172.21.2.125:0/588286013 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3fd75700 1 --2- 172.21.2.125:0/588286013 >> v2:172.21.2.117:3300/0 conn(0x7f5d401b00d0 0x7f5d401b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 -- 172.21.2.125:0/588286013 >> v2:172.21.2.117:3300/0 conn(0x7f5d401b00d0 msgr2=0x7f5d401b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 --2- 172.21.2.125:0/588286013 >> v2:172.21.2.117:3300/0 conn(0x7f5d401b00d0 0x7f5d401b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 -- 172.21.2.125:0/588286013 >> v1:172.21.2.117:6789/0 conn(0x7f5d4007da20 legacy=0x7f5d400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 -- 172.21.2.125:0/588286013 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40079400 con 0x7f5d40079000 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.104 7f5d3ed73700 1 --2- 172.21.2.125:0/588286013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d40079000 0x7f5d400e4d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5d28006700 tx=0x7f5d28009b80).ready entity=mon.1 client_cookie=69c2615f0a057730 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:14.120 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.105 7f5d3e572700 1 -- 172.21.2.125:0/588286013 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d2800c0c0 con 0x7f5d40079000 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.105 7f5d3e572700 1 -- 172.21.2.125:0/588286013 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2800c440 con 0x7f5d40079000 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.105 7f5d3e572700 1 -- 172.21.2.125:0/588286013 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d2800c960 con 0x7f5d40079000 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.105 7f5d45805700 1 -- 172.21.2.125:0/588286013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d40079000 msgr2=0x7f5d400e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.105 7f5d45805700 1 --2- 172.21.2.125:0/588286013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d40079000 0x7f5d400e4d80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5d28006700 tx=0x7f5d28009b80).stop 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 -- 172.21.2.125:0/588286013 shutdown_connections 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 --2- 172.21.2.125:0/588286013 >> v2:172.21.2.117:3300/0 conn(0x7f5d401b00d0 0x7f5d401b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 --2- 172.21.2.125:0/588286013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d40079000 0x7f5d400e4d80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 -- 172.21.2.125:0/588286013 >> 172.21.2.125:0/588286013 conn(0x7f5d400ce430 msgr2=0x7f5d400e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 -- 172.21.2.125:0/588286013 shutdown_connections 2021-02-10T11:22:14.121 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.106 7f5d45805700 1 -- 172.21.2.125:0/588286013 wait complete. 2021-02-10T11:22:14.123 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.120 7f5d45805700 1 Processor -- start 2021-02-10T11:22:14.123 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.120 7f5d45805700 1 -- start start 2021-02-10T11:22:14.123 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.123 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 0x7f5d401a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 0x7f5d401a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40091190 con 0x7f5d401a85d0 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d40091380 con 0x7f5d400aa7b0 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d45805700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d401a89d0 con 0x7f5d40090d90 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3ed73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 0x7f5d401a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.125:39418/0 (socket says 172.21.2.125:39418) 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 -- 172.21.2.125:0/1278286660 learned_addr learned my addr 172.21.2.125:0/1278286660 (peer_addr_for_me v2:172.21.2.125:0/0) 2021-02-10T11:22:14.124 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 -- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 msgr2=0x7f5d401a7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 0x7f5d401a7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 -- 172.21.2.125:0/1278286660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 msgr2=0x7f5d401a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3fd75700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 0x7f5d401a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 0x7f5d401a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3f574700 1 -- 172.21.2.125:0/1278286660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d401a8bf0 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3ed73700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 0x7f5d401a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.121 7f5d3fd75700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 0x7f5d401a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d3f574700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5d30006ed0 tx=0x7f5d3000b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d30006240 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d300009b0 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d45805700 1 -- 172.21.2.125:0/1278286660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d401a8bf0 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d45805700 1 -- 172.21.2.125:0/1278286660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d401a9d90 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d30005730 con 0x7f5d400aa7b0 2021-02-10T11:22:14.125 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d30005c00 con 0x7f5d400aa7b0 2021-02-10T11:22:14.126 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 0x7f5d2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:14.126 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.122 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1475+0+0 (secure 0 0 0) 0x7f5d300317a0 con 0x7f5d400aa7b0 2021-02-10T11:22:14.126 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.124 7f5d3ed73700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 0x7f5d2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:14.126 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.124 7f5d3ed73700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 0x7f5d2401abc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:14.154 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.153 7f5d45805700 1 -- 172.21.2.125:0/1278286660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d401b4f80 con 0x7f5d400aa7b0 2021-02-10T11:22:14.155 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.156 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d300059f0 con 0x7f5d400aa7b0 2021-02-10T11:22:14.292 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.293 7f5d45805700 1 -- 172.21.2.125:0/1278286660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd new", "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", "id": 7} v 0) v1 -- 0x7f5d40077a70 con 0x7f5d400aa7b0 2021-02-10T11:22:14.297 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.298 7f5d37fff700 1 -- 172.21.2.125:0/1278286660 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d30005c00 con 0x7f5d400aa7b0 2021-02-10T11:22:14.299 INFO:teuthology.orchestra.run.gibba025.stdout:7 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 msgr2=0x7f5d2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 0x7f5d2401abc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 msgr2=0x7f5d40090770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5d30006ed0 tx=0x7f5d3000b960).stop 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 shutdown_connections 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d240186c0 0x7f5d2401abc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.302 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d401a85d0 0x7f5d401a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.303 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d40090d90 0x7f5d401a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.303 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 --2- 172.21.2.125:0/1278286660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d400aa7b0 0x7f5d40090770 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:14.303 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 >> 172.21.2.125:0/1278286660 conn(0x7f5d40199470 msgr2=0x7f5d40098e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:14.303 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 shutdown_connections 2021-02-10T11:22:14.303 INFO:teuthology.orchestra.run.gibba025.stderr:2021-02-10 11:22:14.303 7f5d45805700 1 -- 172.21.2.125:0/1278286660 wait complete. 2021-02-10T11:22:14.309 INFO:tasks.ceph.osd.0:Restarting daemon 2021-02-10T11:22:14.310 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-02-10T11:22:14.312 INFO:tasks.ceph.osd.0:Started 2021-02-10T11:22:14.312 INFO:tasks.ceph.osd.1:Restarting daemon 2021-02-10T11:22:14.313 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-02-10T11:22:14.315 INFO:tasks.ceph.osd.1:Started 2021-02-10T11:22:14.315 INFO:tasks.ceph.osd.2:Restarting daemon 2021-02-10T11:22:14.315 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-02-10T11:22:14.317 INFO:tasks.ceph.osd.2:Started 2021-02-10T11:22:14.317 INFO:tasks.ceph.osd.3:Restarting daemon 2021-02-10T11:22:14.317 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-02-10T11:22:14.320 INFO:tasks.ceph.osd.3:Started 2021-02-10T11:22:14.320 INFO:tasks.ceph.osd.4:Restarting daemon 2021-02-10T11:22:14.320 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-02-10T11:22:14.322 INFO:tasks.ceph.osd.4:Started 2021-02-10T11:22:14.322 INFO:tasks.ceph.osd.5:Restarting daemon 2021-02-10T11:22:14.323 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-02-10T11:22:14.325 INFO:tasks.ceph.osd.5:Started 2021-02-10T11:22:14.325 INFO:tasks.ceph.osd.6:Restarting daemon 2021-02-10T11:22:14.325 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2021-02-10T11:22:14.327 INFO:tasks.ceph.osd.6:Started 2021-02-10T11:22:14.328 INFO:tasks.ceph.osd.7:Restarting daemon 2021-02-10T11:22:14.328 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2021-02-10T11:22:14.330 INFO:tasks.ceph.osd.7:Started 2021-02-10T11:22:14.330 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-10T11:22:15.137 DEBUG:teuthology.orchestra.run.gibba017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:15.232 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:15.234 7fc5dc8dca80 -1 Falling back to public interface 2021-02-10T11:22:15.232 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:15.234 7fc5d59ff700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2021-02-10T11:22:15.233 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:15.234 7fc5d59ff700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2021-02-10T11:22:15.233 INFO:tasks.ceph.osd.1.gibba017.stderr:2021-02-10 11:22:15.234 7f2ae7ceea80 -1 Falling back to public interface 2021-02-10T11:22:15.233 INFO:tasks.ceph.osd.1.gibba017.stderr:2021-02-10 11:22:15.234 7f2ae0e11700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2021-02-10T11:22:15.233 INFO:tasks.ceph.osd.2.gibba017.stderr:2021-02-10 11:22:15.234 7f94240f2a80 -1 Falling back to public interface 2021-02-10T11:22:15.233 INFO:tasks.ceph.osd.2.gibba017.stderr:2021-02-10 11:22:15.234 7f941d215700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2021-02-10T11:22:15.238 INFO:tasks.ceph.osd.3.gibba017.stderr:2021-02-10 11:22:15.240 7fbbb4e29a80 -1 Falling back to public interface 2021-02-10T11:22:15.242 INFO:tasks.ceph.osd.4.gibba025.stderr:2021-02-10 11:22:15.244 7f3646ce6a80 -1 Falling back to public interface 2021-02-10T11:22:15.242 INFO:tasks.ceph.osd.7.gibba025.stderr:2021-02-10 11:22:15.244 7fc4a4bb4a80 -1 Falling back to public interface 2021-02-10T11:22:15.242 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:15.244 7f221c446a80 -1 Falling back to public interface 2021-02-10T11:22:15.242 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:15.244 7f2215569700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2021-02-10T11:22:15.243 INFO:tasks.ceph.osd.7.gibba025.stderr:2021-02-10 11:22:15.244 7fc49dcd7700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6807/0: (98) Address already in use 2021-02-10T11:22:15.243 INFO:tasks.ceph.osd.5.gibba025.stderr:2021-02-10 11:22:15.244 7fba8844ba80 -1 Falling back to public interface 2021-02-10T11:22:15.244 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:15.246 7f2215569700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6825/0: (98) Address already in use 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.541 7f92ccdef700 1 Processor -- start 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.541 7f92ccdef700 1 -- start start 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.541 7f92ccdef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 0x7f92c80e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.541 7f92ccdef700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92ccdef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c8077a30 con 0x7f92c807da40 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92ccdef700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c80e5410 con 0x7f92c81b0120 2021-02-10T11:22:15.556 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92ccdef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80793c0 con 0x7f92c8078fc0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c7314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 0x7f92c80e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f92c807da40 0x7f92c80771d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41134/0 (socket says 172.21.2.117:41134) 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c7314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:40884/0 (socket says 172.21.2.117:40884) 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6b13700 1 -- 172.21.2.117:0/3727882191 learned_addr learned my addr 172.21.2.117:0/3727882191 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 -- 172.21.2.117:0/3727882191 >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 msgr2=0x7f92c81b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 --2- 172.21.2.117:0/3727882191 >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 -- 172.21.2.117:0/3727882191 >> v1:172.21.2.117:6789/0 conn(0x7f92c807da40 legacy=0x7f92c80771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 -- 172.21.2.117:0/3727882191 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c807de40 con 0x7f92c8078fc0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c7314700 1 --2- 172.21.2.117:0/3727882191 >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.542 7f92c6312700 1 --2- 172.21.2.117:0/3727882191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 0x7f92c80e4df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f92bc006650 tx=0x7f92bc009a80).ready entity=mon.2 client_cookie=44eabc160a1aeda6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92c5b11700 1 -- 172.21.2.117:0/3727882191 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc0009b0 con 0x7f92c8078fc0 2021-02-10T11:22:15.557 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92c5b11700 1 -- 172.21.2.117:0/3727882191 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00c0c0 con 0x7f92c8078fc0 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92c5b11700 1 -- 172.21.2.117:0/3727882191 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92bc00c520 con 0x7f92c8078fc0 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 -- 172.21.2.117:0/3727882191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 msgr2=0x7f92c80e4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 --2- 172.21.2.117:0/3727882191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 0x7f92c80e4df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f92bc006650 tx=0x7f92bc009a80).stop 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 -- 172.21.2.117:0/3727882191 shutdown_connections 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 --2- 172.21.2.117:0/3727882191 >> v2:172.21.2.117:3300/0 conn(0x7f92c81b0120 0x7f92c81b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 --2- 172.21.2.117:0/3727882191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8078fc0 0x7f92c80e4df0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.543 7f92ccdef700 1 -- 172.21.2.117:0/3727882191 >> 172.21.2.117:0/3727882191 conn(0x7f92c80ce470 msgr2=0x7f92c80e42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.544 7f92ccdef700 1 -- 172.21.2.117:0/3727882191 shutdown_connections 2021-02-10T11:22:15.558 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.544 7f92ccdef700 1 -- 172.21.2.117:0/3727882191 wait complete. 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.558 7f92ccdef700 1 Processor -- start 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.558 7f92ccdef700 1 -- start start 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c81b6410 0x7f92c81b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c81b6e30 0x7f92c8093100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.560 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c81045f0 con 0x7f92c81a1ef0 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8101fb0 con 0x7f92c81b6410 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92ccdef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c81021d0 con 0x7f92c81b6e30 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:40892/0 (socket says 172.21.2.117:40892) 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 -- 172.21.2.117:0/1651125943 learned_addr learned my addr 172.21.2.117:0/1651125943 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6312700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c81b6410 0x7f92c81b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c7314700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c81b6e30 0x7f92c8093100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 -- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c81b6e30 msgr2=0x7f92c8093100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c81b6e30 0x7f92c8093100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 -- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c81b6410 msgr2=0x7f92c81b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c81b6410 0x7f92c81b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.561 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 -- 172.21.2.117:0/1651125943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c8093720 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.559 7f92c6b13700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f92b8001ec0 tx=0x7f92b800a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92b800ac90 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b80009b0 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c8093720 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92b8004590 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c8093720 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92b80043a0 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 0x7f92ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1578+0+0 (secure 0 0 0) 0x7f92b8031440 con 0x7f92c81a1ef0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.560 7f92c6312700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 0x7f92ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:15.562 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.561 7f92c6312700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 0x7f92ac01abc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:15.589 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.591 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c81b5210 con 0x7f92c81a1ef0 2021-02-10T11:22:15.591 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.594 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92b800ac90 con 0x7f92c81a1ef0 2021-02-10T11:22:15.717 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.719 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c8077a30 con 0x7f92c81a1ef0 2021-02-10T11:22:15.717 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.719 7f92b77fe700 1 -- 172.21.2.117:0/1651125943 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+5464 (secure 0 0 0) 0x7f92b801f500 con 0x7f92c81a1ef0 2021-02-10T11:22:15.718 INFO:teuthology.misc.health.gibba017.stdout: 2021-02-10T11:22:15.718 INFO:teuthology.misc.health.gibba017.stdout:{"epoch":10,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:14.297433","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":":/0","cluster_addr":":/0","heartbeat_back_addr":":/0","heartbeat_front_addr":":/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:15.719 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 msgr2=0x7f92ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 0x7f92ac01abc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 msgr2=0x7f92c81aa0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f92b8001ec0 tx=0x7f92b800a8a0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 shutdown_connections 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c81b6e30 0x7f92c8093100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c81b6410 0x7f92c81b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92ac0186c0 0x7f92ac01abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 --2- 172.21.2.117:0/1651125943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c81a1ef0 0x7f92c81aa0d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 >> 172.21.2.117:0/1651125943 conn(0x7f92c81960f0 msgr2=0x7f92c808a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.721 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 shutdown_connections 2021-02-10T11:22:15.720 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:15.722 7f92ccdef700 1 -- 172.21.2.117:0/1651125943 wait complete. 2021-02-10T11:22:15.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2021-02-10T11:22:16.020 INFO:tasks.ceph.osd.1.gibba017.stderr:2021-02-10 11:22:16.022 7f2ae7ceea80 -1 osd.1 0 log_to_monitors {default=true} 2021-02-10T11:22:16.036 INFO:tasks.ceph.osd.3.gibba017.stderr:2021-02-10 11:22:16.038 7fbbb4e29a80 -1 osd.3 0 log_to_monitors {default=true} 2021-02-10T11:22:16.265 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:16.267 7fc5dc8dca80 -1 osd.0 0 log_to_monitors {default=true} 2021-02-10T11:22:16.279 INFO:tasks.ceph.osd.2.gibba017.stderr:2021-02-10 11:22:16.281 7f94240f2a80 -1 osd.2 0 log_to_monitors {default=true} 2021-02-10T11:22:16.282 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:16.283 7f221c446a80 -1 osd.6 0 log_to_monitors {default=true} 2021-02-10T11:22:16.282 INFO:tasks.ceph.osd.7.gibba025.stderr:2021-02-10 11:22:16.283 7fc4a4bb4a80 -1 osd.7 0 log_to_monitors {default=true} 2021-02-10T11:22:16.284 INFO:tasks.ceph.osd.4.gibba025.stderr:2021-02-10 11:22:16.285 7f3646ce6a80 -1 osd.4 0 log_to_monitors {default=true} 2021-02-10T11:22:16.284 INFO:tasks.ceph.osd.5.gibba025.stderr:2021-02-10 11:22:16.286 7fba8844ba80 -1 osd.5 0 log_to_monitors {default=true} 2021-02-10T11:22:17.309 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:17.311 7fc5c2f55700 -1 osd.0 0 waiting for initial osdmap 2021-02-10T11:22:17.309 INFO:tasks.ceph.osd.2.gibba017.stderr:2021-02-10 11:22:17.311 7f940a76b700 -1 osd.2 0 waiting for initial osdmap 2021-02-10T11:22:17.309 INFO:tasks.ceph.osd.3.gibba017.stderr:2021-02-10 11:22:17.311 7fbb9b4a2700 -1 osd.3 0 waiting for initial osdmap 2021-02-10T11:22:17.310 INFO:tasks.ceph.osd.4.gibba025.stderr:2021-02-10 11:22:17.312 7f362d35f700 -1 osd.4 0 waiting for initial osdmap 2021-02-10T11:22:17.310 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:17.312 7f2202abf700 -1 osd.6 0 waiting for initial osdmap 2021-02-10T11:22:17.310 INFO:tasks.ceph.osd.5.gibba025.stderr:2021-02-10 11:22:17.312 7fba6eac4700 -1 osd.5 0 waiting for initial osdmap 2021-02-10T11:22:17.311 INFO:tasks.ceph.osd.1.gibba017.stderr:2021-02-10 11:22:17.312 7f2ace367700 -1 osd.1 0 waiting for initial osdmap 2021-02-10T11:22:17.311 INFO:tasks.ceph.osd.7.gibba025.stderr:2021-02-10 11:22:17.312 7fc48b22d700 -1 osd.7 0 waiting for initial osdmap 2021-02-10T11:22:17.312 INFO:tasks.ceph.osd.0.gibba017.stderr:2021-02-10 11:22:17.314 7fc5cf173700 -1 osd.0 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.312 INFO:tasks.ceph.osd.2.gibba017.stderr:2021-02-10 11:22:17.314 7f9416989700 -1 osd.2 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.313 INFO:tasks.ceph.osd.3.gibba017.stderr:2021-02-10 11:22:17.315 7fbba76c0700 -1 osd.3 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.313 INFO:tasks.ceph.osd.1.gibba017.stderr:2021-02-10 11:22:17.315 7f2ada585700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.313 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:22:17.315 7f220ecdd700 -1 osd.6 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.313 INFO:tasks.ceph.osd.4.gibba025.stderr:2021-02-10 11:22:17.315 7f363957d700 -1 osd.4 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.314 INFO:tasks.ceph.osd.5.gibba025.stderr:2021-02-10 11:22:17.315 7fba7ace2700 -1 osd.5 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:17.314 INFO:tasks.ceph.osd.7.gibba025.stderr:2021-02-10 11:22:17.316 7fc49744b700 -1 osd.7 12 set_numa_affinity unable to identify public interface 'enp1s0f0' numa node: (0) Success 2021-02-10T11:22:21.983 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:21.987 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:22.534 DEBUG:teuthology.orchestra.run.gibba017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:22.951 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.936 7fb8a7925700 1 Processor -- start 2021-02-10T11:22:22.951 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.936 7fb8a7925700 1 -- start start 2021-02-10T11:22:22.951 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a7925700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8a007da40 0x7fb8a00771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a7925700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00e4df0 0x7fb8a01b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a7925700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a0077a30 con 0x7fb8a00e51f0 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a7925700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a00777f0 con 0x7fb8a007da40 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a7925700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a007de40 con 0x7fb8a00e4df0 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a5741700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8a007da40 0x7fb8a00771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a5741700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8a007da40 0x7fb8a00771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41118/0 (socket says 172.21.2.117:41118) 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a5741700 1 -- 172.21.2.117:0/412032132 learned_addr learned my addr 172.21.2.117:0/412032132 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.937 7fb8a5f42700 1 --1- 172.21.2.117:0/412032132 >> v1:172.21.2.117:6789/0 conn(0x7fb8a00e51f0 0x7fb8a0077fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41372/0 (socket says 172.21.2.117:41372) 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a4f40700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00e4df0 0x7fb8a01b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a5741700 1 -- 172.21.2.117:0/412032132 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00e4df0 msgr2=0x7fb8a01b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:22.952 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a5741700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00e4df0 0x7fb8a01b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a5741700 1 -- 172.21.2.117:0/412032132 >> v1:172.21.2.117:6789/0 conn(0x7fb8a00e51f0 legacy=0x7fb8a0077fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a5741700 1 -- 172.21.2.117:0/412032132 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a01b0740 con 0x7fb8a007da40 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a5741700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a007da40 0x7fb8a00771d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb89c006700 tx=0x7fb89c009b00).ready entity=mon.0 client_cookie=996645aa2d38cbae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb897fff700 1 -- 172.21.2.117:0/412032132 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89c00c0c0 con 0x7fb8a007da40 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb897fff700 1 -- 172.21.2.117:0/412032132 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c00c340 con 0x7fb8a007da40 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb897fff700 1 -- 172.21.2.117:0/412032132 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89c00c820 con 0x7fb8a007da40 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a7925700 1 -- 172.21.2.117:0/412032132 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a007da40 msgr2=0x7fb8a00771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.938 7fb8a7925700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a007da40 0x7fb8a00771d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fb89c006700 tx=0x7fb89c009b00).stop 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 -- 172.21.2.117:0/412032132 shutdown_connections 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00e4df0 0x7fb8a01b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.953 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 --2- 172.21.2.117:0/412032132 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a007da40 0x7fb8a00771d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:22.954 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 -- 172.21.2.117:0/412032132 >> 172.21.2.117:0/412032132 conn(0x7fb8a00ce470 msgr2=0x7fb8a00e42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:22.954 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 -- 172.21.2.117:0/412032132 shutdown_connections 2021-02-10T11:22:22.954 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.939 7fb8a7925700 1 -- 172.21.2.117:0/412032132 wait complete. 2021-02-10T11:22:22.954 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.953 7fb8a7925700 1 Processor -- start 2021-02-10T11:22:22.954 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.953 7fb8a7925700 1 -- start start 2021-02-10T11:22:22.955 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8a00930e0 0x7fb8a0198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.955 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.955 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00916f0 0x7fb8a0091af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0098c90 con 0x7fb8a0199460 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0092ba0 con 0x7fb8a00930e0 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a7925700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0098700 con 0x7fb8a00916f0 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41124/0 (socket says 172.21.2.117:41124) 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 -- 172.21.2.117:0/1117736523 learned_addr learned my addr 172.21.2.117:0/1117736523 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 -- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00916f0 msgr2=0x7fb8a0091af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00916f0 0x7fb8a0091af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 -- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8a00930e0 msgr2=0x7fb8a0198e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:22.956 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8a00930e0 0x7fb8a0198e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 -- 172.21.2.117:0/1117736523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0092110 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.954 7fb8a4f40700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb890006f00 tx=0x7fb89000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb89000bd30 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb890002030 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb890002490 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a0092110 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a0092110 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb890000c90 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.955 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7fb890031f80 con 0x7fb8a0199460 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.956 7fb8a5741700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:22.957 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.956 7fb8a5741700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 0x7fb88001abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:22.985 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.987 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a01b5210 con 0x7fb8a0199460 2021-02-10T11:22:22.987 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:22.989 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb890030d50 con 0x7fb8a0199460 2021-02-10T11:22:23.110 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.112 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8a0077a30 con 0x7fb8a0199460 2021-02-10T11:22:23.111 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.113 7fb895ffb700 1 -- 172.21.2.117:0/1117736523 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9772 (secure 0 0 0) 0x7fb890000c90 con 0x7fb8a0199460 2021-02-10T11:22:23.111 INFO:teuthology.misc.health.gibba017.stdout: 2021-02-10T11:22:23.111 INFO:teuthology.misc.health.gibba017.stdout:{"epoch":13,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:18.312455","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.114 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 msgr2=0x7fb88001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.114 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 0x7fb88001abc0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.114 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 msgr2=0x7fb8a0199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.114 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb890006f00 tx=0x7fb89000b940).stop 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 shutdown_connections 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8800186c0 0x7fb88001abc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8a0199460 0x7fb8a0199860 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8a00916f0 0x7fb8a0091af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.113 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 --2- 172.21.2.117:0/1117736523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8a00930e0 0x7fb8a0198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.114 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 >> 172.21.2.117:0/1117736523 conn(0x7fb8a01a4e30 msgr2=0x7fb8a019da40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:23.114 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 shutdown_connections 2021-02-10T11:22:23.114 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:23.115 7fb8a7925700 1 -- 172.21.2.117:0/1117736523 wait complete. 2021-02-10T11:22:23.120 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:22:23.120 INFO:tasks.ceph:Creating RBD pool 2021-02-10T11:22:23.121 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --cluster ceph osd pool create rbd 8 2021-02-10T11:22:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.532 7f21d56ae700 1 Processor -- start 2021-02-10T11:22:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.532 7f21d56ae700 1 -- start start 2021-02-10T11:22:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21d56ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21d00d1530 0x7f21d00d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21d56ae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d0077a70 con 0x7f21d007da20 2021-02-10T11:22:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21d56ae700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d0079400 con 0x7f21d0079000 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21d56ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d007de20 con 0x7f21d00d1530 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cfac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21d00d1530 0x7f21d00d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cf2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f21d007da20 0x7f21d00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41382/0 (socket says 172.21.2.117:41382) 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cfac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21d00d1530 0x7f21d00d1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41136/0 (socket says 172.21.2.117:41136) 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cf2bf700 1 -- 172.21.2.117:0/907959800 learned_addr learned my addr 172.21.2.117:0/907959800 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cfac0700 1 -- 172.21.2.117:0/907959800 >> v1:172.21.2.125:6789/0 conn(0x7f21d0079000 legacy=0x7f21d00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21cfac0700 1 -- 172.21.2.117:0/907959800 >> v1:172.21.2.117:6789/0 conn(0x7f21d007da20 legacy=0x7f21d00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.533 7f21ce2bd700 1 -- 172.21.2.117:0/907959800 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435720822 0 0) 0x7f21c0001180 con 0x7f21d007da20 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21cfac0700 1 -- 172.21.2.117:0/907959800 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d01b00f0 con 0x7f21d00d1530 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21cfac0700 1 --2- 172.21.2.117:0/907959800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00d1530 0x7f21d00d1930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f21c8006e30 tx=0x7f21c8009de0).ready entity=mon.0 client_cookie=376e2ccb66c5d17b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21ce2bd700 1 -- 172.21.2.117:0/907959800 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c800b300 con 0x7f21d00d1530 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21ce2bd700 1 -- 172.21.2.117:0/907959800 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c800b580 con 0x7f21d00d1530 2021-02-10T11:22:23.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21ce2bd700 1 -- 172.21.2.117:0/907959800 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c800baa0 con 0x7f21d00d1530 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21d56ae700 1 -- 172.21.2.117:0/907959800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00d1530 msgr2=0x7f21d00d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.534 7f21d56ae700 1 --2- 172.21.2.117:0/907959800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00d1530 0x7f21d00d1930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f21c8006e30 tx=0x7f21c8009de0).stop 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.535 7f21d56ae700 1 -- 172.21.2.117:0/907959800 shutdown_connections 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.535 7f21d56ae700 1 --2- 172.21.2.117:0/907959800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00d1530 0x7f21d00d1930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.535 7f21d56ae700 1 -- 172.21.2.117:0/907959800 >> 172.21.2.117:0/907959800 conn(0x7f21d00ce450 msgr2=0x7f21d00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.535 7f21d56ae700 1 -- 172.21.2.117:0/907959800 shutdown_connections 2021-02-10T11:22:23.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.535 7f21d56ae700 1 -- 172.21.2.117:0/907959800 wait complete. 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 Processor -- start 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 -- start start 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d00929e0 0x7f21d009e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d00aeee0 0x7f21d00af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00af3e0 con 0x7f21d00bfd80 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d009e920 con 0x7f21d00929e0 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21d56ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d009eb00 con 0x7f21d00aeee0 2021-02-10T11:22:23.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21ceabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21ceabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41140/0 (socket says 172.21.2.117:41140) 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.550 7f21ceabe700 1 -- 172.21.2.117:0/1002587255 learned_addr learned my addr 172.21.2.117:0/1002587255 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 -- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d00aeee0 msgr2=0x7f21d00af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d00aeee0 0x7f21d00af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 -- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d00929e0 msgr2=0x7f21d009e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d00929e0 0x7f21d009e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 -- 172.21.2.117:0/1002587255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00aff80 con 0x7f21d00bfd80 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21ceabe700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f21c400b5b0 tx=0x7f21c400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c400bd40 con 0x7f21d00bfd80 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c4006240 con 0x7f21d00bfd80 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c4005720 con 0x7f21d00bfd80 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d00aff80 con 0x7f21d00bfd80 2021-02-10T11:22:23.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.551 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d00b04e0 con 0x7f21d00bfd80 2021-02-10T11:22:23.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.552 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21c40054a0 con 0x7f21d00bfd80 2021-02-10T11:22:23.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.552 7f21b7fff700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:23.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.552 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4400+0+0 (secure 0 0 0) 0x7f21c4032100 con 0x7f21d00bfd80 2021-02-10T11:22:23.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.552 7f21cf2bf700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:23.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.553 7f21cf2bf700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.583 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d00ff1f0 con 0x7f21d00bfd80 2021-02-10T11:22:23.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.585 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21c40264c0 con 0x7f21d00bfd80 2021-02-10T11:22:23.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.710 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f21d0077a70 con 0x7f21d00bfd80 2021-02-10T11:22:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.876 7f21b7fff700 1 -- 172.21.2.117:0/1002587255 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f21c40264c0 con 0x7f21d00bfd80 2021-02-10T11:22:23.923 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'rbd' created 2021-02-10T11:22:23.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 msgr2=0x7f21b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 msgr2=0x7f21d00ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f21c400b5b0 tx=0x7f21c400b950).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 shutdown_connections 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00bfd80 0x7f21d00ae8c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d00aeee0 0x7f21d00af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 --2- 172.21.2.117:0/1002587255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d00929e0 0x7f21d009e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.927 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 >> 172.21.2.117:0/1002587255 conn(0x7f21d00b4e50 msgr2=0x7f21d00be760 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.928 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 shutdown_connections 2021-02-10T11:22:23.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:23.928 7f21d56ae700 1 -- 172.21.2.117:0/1002587255 wait complete. 2021-02-10T11:22:23.933 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-10T11:22:24.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 Processor -- start 2021-02-10T11:22:24.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 -- start start 2021-02-10T11:22:24.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3d541b0130 0x7f3d541b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d54077a10 con 0x7f3d5407da20 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d540e5c60 con 0x7f3d54078fa0 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d59dcc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d540e5420 con 0x7f3d541b0130 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d52ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d53fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3d541b0130 0x7f3d541b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d52ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41146/0 (socket says 172.21.2.117:41146) 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d52ffd700 1 -- 172.21.2.117:0/2577660144 learned_addr learned my addr 172.21.2.117:0/2577660144 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.347 7f3d537fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3d5407da20 0x7f3d540771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41396/0 (socket says 172.21.2.117:41396) 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d527fc700 1 -- 172.21.2.117:0/2577660144 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079136758 0 0) 0x7f3d48001100 con 0x7f3d5407da20 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d53fff700 1 -- 172.21.2.117:0/2577660144 >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 msgr2=0x7f3d540e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:24.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d53fff700 1 --2- 172.21.2.117:0/2577660144 >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d53fff700 1 -- 172.21.2.117:0/2577660144 >> v1:172.21.2.117:6789/0 conn(0x7f3d5407da20 legacy=0x7f3d540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d53fff700 1 -- 172.21.2.117:0/2577660144 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540793a0 con 0x7f3d541b0130 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d52ffd700 1 --2- 172.21.2.117:0/2577660144 >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d53fff700 1 --2- 172.21.2.117:0/2577660144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d541b0130 0x7f3d541b0530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3d44006ad0 tx=0x7f3d44009ad0).ready entity=mon.1 client_cookie=e30c3763e1e3a84a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d527fc700 1 -- 172.21.2.117:0/2577660144 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d44006370 con 0x7f3d541b0130 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d527fc700 1 -- 172.21.2.117:0/2577660144 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d440022e0 con 0x7f3d541b0130 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d527fc700 1 -- 172.21.2.117:0/2577660144 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d44002800 con 0x7f3d541b0130 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d59dcc700 1 -- 172.21.2.117:0/2577660144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d541b0130 msgr2=0x7f3d541b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.348 7f3d59dcc700 1 --2- 172.21.2.117:0/2577660144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d541b0130 0x7f3d541b0530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f3d44006ad0 tx=0x7f3d44009ad0).stop 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 -- 172.21.2.117:0/2577660144 shutdown_connections 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 --2- 172.21.2.117:0/2577660144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d541b0130 0x7f3d541b0530 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 --2- 172.21.2.117:0/2577660144 >> v2:172.21.2.117:3300/0 conn(0x7f3d54078fa0 0x7f3d540e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 -- 172.21.2.117:0/2577660144 >> 172.21.2.117:0/2577660144 conn(0x7f3d540ce480 msgr2=0x7f3d540e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:24.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 -- 172.21.2.117:0/2577660144 shutdown_connections 2021-02-10T11:22:24.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.349 7f3d59dcc700 1 -- 172.21.2.117:0/2577660144 wait complete. 2021-02-10T11:22:24.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 Processor -- start 2021-02-10T11:22:24.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 -- start start 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d540c1410 0x7f3d540b99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 0x7f3d540ba010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540c1d20 con 0x7f3d541a0260 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540ba630 con 0x7f3d540c1810 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d59dcc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540c22a0 con 0x7f3d540c1410 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.364 7f3d537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41150/0 (socket says 172.21.2.117:41150) 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d537fe700 1 -- 172.21.2.117:0/2753795407 learned_addr learned my addr 172.21.2.117:0/2753795407 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 0x7f3d540ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d52ffd700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d540c1410 0x7f3d540b99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 -- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d540c1410 msgr2=0x7f3d540b99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d540c1410 0x7f3d540b99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 -- 172.21.2.117:0/2753795407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 msgr2=0x7f3d540c0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 -- 172.21.2.117:0/2753795407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540c24c0 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d537fe700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d53fff700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 0x7f3d540ba010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3d44006b00 tx=0x7f3d4400b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d44006b60 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d44000d10 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d540c24c0 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d540c2a50 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.365 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d44002220 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.366 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d440026f0 con 0x7f3d540c1810 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.366 7f3d3bfff700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 0x7f3d3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.366 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4742+0+0 (secure 0 0 0) 0x7f3d44032160 con 0x7f3d540c1810 2021-02-10T11:22:24.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.366 7f3d537fe700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 0x7f3d3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:24.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.367 7f3d537fe700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 0x7f3d3401abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.397 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d541051f0 con 0x7f3d540c1810 2021-02-10T11:22:24.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.400 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d440024e0 con 0x7f3d540c1810 2021-02-10T11:22:24.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.529 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3d54077a10 con 0x7f3d540c1810 2021-02-10T11:22:24.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.882 7f3d3bfff700 1 -- 172.21.2.117:0/2753795407 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3d440026f0 con 0x7f3d540c1810 2021-02-10T11:22:24.894 INFO:teuthology.orchestra.run.gibba017.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-10T11:22:24.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 msgr2=0x7f3d3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 0x7f3d3401abc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 msgr2=0x7f3d540ba010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:24.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 0x7f3d540ba010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f3d44006b00 tx=0x7f3d4400b910).stop 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 shutdown_connections 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d540c1810 0x7f3d540ba010 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d540c1410 0x7f3d540b99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d340186c0 0x7f3d3401abc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 --2- 172.21.2.117:0/2753795407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d541a0260 0x7f3d540c0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 >> 172.21.2.117:0/2753795407 conn(0x7f3d54090ed0 msgr2=0x7f3d5409ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:24.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 shutdown_connections 2021-02-10T11:22:24.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:24.898 7f3d59dcc700 1 -- 172.21.2.117:0/2753795407 wait complete. 2021-02-10T11:22:24.904 INFO:tasks.ceph:Setting up CephFS filesystem... 2021-02-10T11:22:24.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:25.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.323 7f58de8e0700 1 Processor -- start 2021-02-10T11:22:25.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.323 7f58de8e0700 1 -- start start 2021-02-10T11:22:25.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58de8e0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58d807da20 0x7f58d80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58de8e0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d80d1d10 con 0x7f58d80e5190 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58de8e0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d8077a70 con 0x7f58d80e4d90 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58de8e0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d80777d0 con 0x7f58d807da20 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58dcefd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f58d80e5190 0x7f58d80d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41408/0 (socket says 172.21.2.117:41408) 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.324 7f58dcefd700 1 -- 172.21.2.117:0/1929344393 learned_addr learned my addr 172.21.2.117:0/1929344393 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636825635 0 0) 0x7f58d0001500 con 0x7f58d80e5190 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c0002fe0 con 0x7f58d80e5190 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d7fff700 1 --2- 172.21.2.117:0/1929344393 >> v2:172.21.2.125:3300/0 conn(0x7f58d807da20 0x7f58d80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 759490650 0 0) 0x7f58d0001460 con 0x7f58d80e5190 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 >> v2:172.21.2.125:3300/0 conn(0x7f58d807da20 msgr2=0x7f58d80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 --2- 172.21.2.117:0/1929344393 >> v2:172.21.2.125:3300/0 conn(0x7f58d807da20 0x7f58d80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 >> v1:172.21.2.125:6789/0 conn(0x7f58d80e4d90 legacy=0x7f58d81b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d81b08d0 con 0x7f58d80e5190 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f58d0001980 con 0x7f58d80e5190 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 >> v1:172.21.2.117:6789/0 conn(0x7f58d80e5190 legacy=0x7f58d80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c0003a60 0x7f58c0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 0x7f58c0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c0009590 0x7f58c000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c000c300 con 0x7f58c0006790 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c000c520 con 0x7f58c0003a60 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c000c740 con 0x7f58c0009590 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d77fe700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 0x7f58c0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58d7fff700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c0003a60 0x7f58c0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.325 7f58dcefd700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c0009590 0x7f58c000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 -- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c0009590 msgr2=0x7f58c000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c0009590 0x7f58c000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 -- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c0003a60 msgr2=0x7f58c0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c0003a60 0x7f58c0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 -- 172.21.2.117:0/1929344393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00509b0 con 0x7f58c0006790 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58dcefd700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c0009590 0x7f58c000bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58dcefd700 1 -- 172.21.2.117:0/1929344393 reap_dead start 2021-02-10T11:22:25.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d77fe700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 0x7f58c0008c90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f58cc006ac0 tx=0x7f58cc009af0).ready entity=mon.0 client_cookie=36d04b18eea43bc0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc007280 con 0x7f58c0006790 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc0009b0 con 0x7f58c0006790 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58d6ffd700 1 -- 172.21.2.117:0/1929344393 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58cc007710 con 0x7f58c0006790 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58de8e0700 1 -- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 msgr2=0x7f58c0008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.326 7f58de8e0700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 0x7f58c0008c90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f58cc006ac0 tx=0x7f58cc009af0).stop 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.327 7f58de8e0700 1 -- 172.21.2.117:0/1929344393 shutdown_connections 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.327 7f58de8e0700 1 --2- 172.21.2.117:0/1929344393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c0006790 0x7f58c0008c90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.327 7f58de8e0700 1 -- 172.21.2.117:0/1929344393 >> 172.21.2.117:0/1929344393 conn(0x7f58d80ce440 msgr2=0x7f58d80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.327 7f58de8e0700 1 -- 172.21.2.117:0/1929344393 shutdown_connections 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.327 7f58de8e0700 1 -- 172.21.2.117:0/1929344393 wait complete. 2021-02-10T11:22:25.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.341 7f58de8e0700 1 Processor -- start 2021-02-10T11:22:25.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.341 7f58de8e0700 1 -- start start 2021-02-10T11:22:25.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58d8093b90 0x7f58d81ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58d81abab0 0x7f58d81abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d8105bf0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80bbfc0 con 0x7f58d81abab0 2021-02-10T11:22:25.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58de8e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d80bc1e0 con 0x7f58d8093b90 2021-02-10T11:22:25.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41172/0 (socket says 172.21.2.117:41172) 2021-02-10T11:22:25.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 -- 172.21.2.117:0/3319790371 learned_addr learned my addr 172.21.2.117:0/3319790371 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:25.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 -- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58d8093b90 msgr2=0x7f58d81ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58d8093b90 0x7f58d81ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 -- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58d81abab0 msgr2=0x7f58d81abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58d81abab0 0x7f58d81abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.342 7f58dcefd700 1 -- 172.21.2.117:0/3319790371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d80bc400 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58dcefd700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f58d000a8d0 tx=0x7f58d000ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58d00070d0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d00073d0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58d0004bb0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d80bc400 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.343 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d8106460 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.344 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58d0004720 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.344 7f58d4ff9700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 0x7f58b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.344 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f58d00327e0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.344 7f58d7fff700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 0x7f58b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.345 7f58d7fff700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 0x7f58b801abc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.373 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d81032e0 con 0x7f58d80bb5a0 2021-02-10T11:22:25.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.376 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58d0004720 con 0x7f58d80bb5a0 2021-02-10T11:22:25.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.498 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58d8077a70 con 0x7f58d80bb5a0 2021-02-10T11:22:25.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.498 7f58d4ff9700 1 -- 172.21.2.117:0/3319790371 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11155 (secure 0 0 0) 0x7f58d0001870 con 0x7f58d80bb5a0 2021-02-10T11:22:25.497 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:25.498 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":15,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:24.877072","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 msgr2=0x7f58b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 0x7f58b801abc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 msgr2=0x7f58d80bb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f58d000a8d0 tx=0x7f58d000ac70).stop 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 shutdown_connections 2021-02-10T11:22:25.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58b80186c0 0x7f58b801abc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58d80bb5a0 0x7f58d80bb9a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58d81abab0 0x7f58d81abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 --2- 172.21.2.117:0/3319790371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58d8093b90 0x7f58d81ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.500 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 >> 172.21.2.117:0/3319790371 conn(0x7f58d819c210 msgr2=0x7f58d809dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.501 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 shutdown_connections 2021-02-10T11:22:25.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.501 7f58de8e0700 1 -- 172.21.2.117:0/3319790371 wait complete. 2021-02-10T11:22:25.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:22:25.507 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:25.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 Processor -- start 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 -- start start 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 0x7fe9480771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9480e51a0 0x7fe9480d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe948077a70 con 0x7fe9480e4da0 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9480e5c20 con 0x7fe9480e51a0 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe94d83a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480777d0 con 0x7fe94807da20 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.924 7fe947d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9480e51a0 0x7fe9480d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9480e51a0 0x7fe9480d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41176/0 (socket says 172.21.2.117:41176) 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 -- 172.21.2.117:0/693478345 learned_addr learned my addr 172.21.2.117:0/693478345 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe946d73700 1 --1- 172.21.2.117:0/693478345 >> v1:172.21.2.117:6789/0 conn(0x7fe9480e4da0 0x7fe9481b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41426/0 (socket says 172.21.2.117:41426) 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947574700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 0x7fe9480771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 -- 172.21.2.117:0/693478345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 msgr2=0x7fe9480771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 0x7fe9480771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 -- 172.21.2.117:0/693478345 >> v1:172.21.2.117:6789/0 conn(0x7fe9480e4da0 legacy=0x7fe9481b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 -- 172.21.2.117:0/693478345 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9481b08e0 con 0x7fe9480e51a0 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947574700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 0x7fe9480771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe947d75700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480e51a0 0x7fe9480d0520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe940006f00 tx=0x7fe940009f10).ready entity=mon.0 client_cookie=759e42cd7bf038cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.925 7fe946572700 1 -- 172.21.2.117:0/693478345 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe940003030 con 0x7fe9480e51a0 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe946572700 1 -- 172.21.2.117:0/693478345 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94000b3b0 con 0x7fe9480e51a0 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe946572700 1 -- 172.21.2.117:0/693478345 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94000b8d0 con 0x7fe9480e51a0 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 -- 172.21.2.117:0/693478345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480e51a0 msgr2=0x7fe9480d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480e51a0 0x7fe9480d0520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe940006f00 tx=0x7fe940009f10).stop 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 -- 172.21.2.117:0/693478345 shutdown_connections 2021-02-10T11:22:25.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480e51a0 0x7fe9480d0520 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 --2- 172.21.2.117:0/693478345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe94807da20 0x7fe9480771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 -- 172.21.2.117:0/693478345 >> 172.21.2.117:0/693478345 conn(0x7fe9480ce450 msgr2=0x7fe9480e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 -- 172.21.2.117:0/693478345 shutdown_connections 2021-02-10T11:22:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.926 7fe94d83a700 1 -- 172.21.2.117:0/693478345 wait complete. 2021-02-10T11:22:25.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.941 7fe94d83a700 1 Processor -- start 2021-02-10T11:22:25.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.941 7fe94d83a700 1 -- start start 2021-02-10T11:22:25.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9480a41c0 0x7fe9480fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9480c2340 0x7fe9480c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480b8010 con 0x7fe9480b7390 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480b8550 con 0x7fe9480c2340 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe94d83a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480b8770 con 0x7fe9480a41c0 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41178/0 (socket says 172.21.2.117:41178) 2021-02-10T11:22:25.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 -- 172.21.2.117:0/3860730575 learned_addr learned my addr 172.21.2.117:0/3860730575 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:25.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe946d73700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9480c2340 0x7fe9480c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947574700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9480a41c0 0x7fe9480fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 -- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9480a41c0 msgr2=0x7fe9480fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9480a41c0 0x7fe9480fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 -- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9480c2340 msgr2=0x7fe9480c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9480c2340 0x7fe9480c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.942 7fe947d75700 1 -- 172.21.2.117:0/3860730575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480b8990 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe947d75700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe94000b940 tx=0x7fe94000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe940009450 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe940003030 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9480b8990 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9480b8f20 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.943 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94000c4e0 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.944 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe940001270 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.944 7fe92ffff700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 0x7fe92801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.944 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7fe940032620 con 0x7fe9480b7390 2021-02-10T11:22:25.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.944 7fe947574700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 0x7fe92801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:25.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.944 7fe947574700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 0x7fe92801abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:25.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.974 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9481b4f00 con 0x7fe9480b7390 2021-02-10T11:22:25.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:25.977 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe940009450 con 0x7fe9480b7390 2021-02-10T11:22:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.104 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe948077a70 con 0x7fe9480b7390 2021-02-10T11:22:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.105 7fe92ffff700 1 -- 172.21.2.117:0/3860730575 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe940001270 con 0x7fe9480b7390 2021-02-10T11:22:26.103 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 msgr2=0x7fe92801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 0x7fe92801abc0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 msgr2=0x7fe9480b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe94000b940 tx=0x7fe94000bce0).stop 2021-02-10T11:22:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 shutdown_connections 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9280186c0 0x7fe92801abc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9480b7390 0x7fe9480b7790 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9480c2340 0x7fe9480c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 --2- 172.21.2.117:0/3860730575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9480a41c0 0x7fe9480fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.106 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 >> 172.21.2.117:0/3860730575 conn(0x7fe94808bbc0 msgr2=0x7fe9480c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.107 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 shutdown_connections 2021-02-10T11:22:26.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.107 7fe94d83a700 1 -- 172.21.2.117:0/3860730575 wait complete. 2021-02-10T11:22:26.113 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:22:26.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.524 7f1229d35700 1 Processor -- start 2021-02-10T11:22:26.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.524 7f1229d35700 1 -- start start 2021-02-10T11:22:26.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1229d35700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f122407da20 0x7f12240771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1229d35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1229d35700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1224077a70 con 0x7f12240e5190 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1229d35700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12240777d0 con 0x7f122407da20 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1229d35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122407de20 con 0x7f12240e4d90 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1222ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f12237fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f122407da20 0x7f12240771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f12237fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f122407da20 0x7f12240771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:45570/0 (socket says 172.21.2.117:45570) 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1223fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f12240e5190 0x7f1224077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34366/0 (socket says 172.21.2.117:34366) 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1222ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:38866/0 (socket says 172.21.2.117:38866) 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.525 7f1223fff700 1 -- 172.21.2.117:0/1221486680 learned_addr learned my addr 172.21.2.117:0/1221486680 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12237fe700 1 -- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 msgr2=0x7f12241b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:26.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12237fe700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12237fe700 1 -- 172.21.2.117:0/1221486680 >> v1:172.21.2.125:6789/0 conn(0x7f12240e5190 legacy=0x7f1224077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12227fc700 1 -- 172.21.2.117:0/1221486680 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853331070 0 0) 0x7f121c001510 con 0x7f12240e5190 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12237fe700 1 -- 172.21.2.117:0/1221486680 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12241b0700 con 0x7f122407da20 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f1222ffd700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12237fe700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122407da20 0x7f12240771b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f120c006650 tx=0x7f120c009a80).ready entity=mon.1 client_cookie=6d18b75aaefa07cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12227fc700 1 -- 172.21.2.117:0/1221486680 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c0009b0 con 0x7f122407da20 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12227fc700 1 -- 172.21.2.117:0/1221486680 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00c0c0 con 0x7f122407da20 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f12227fc700 1 -- 172.21.2.117:0/1221486680 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c00c5e0 con 0x7f122407da20 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f1229d35700 1 -- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122407da20 msgr2=0x7f12240771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.526 7f1229d35700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122407da20 0x7f12240771b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f120c006650 tx=0x7f120c009a80).stop 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 -- 172.21.2.117:0/1221486680 shutdown_connections 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12240e4d90 0x7f12241b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 --2- 172.21.2.117:0/1221486680 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122407da20 0x7f12240771b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 -- 172.21.2.117:0/1221486680 >> 172.21.2.117:0/1221486680 conn(0x7f12240ce440 msgr2=0x7f12240e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 -- 172.21.2.117:0/1221486680 shutdown_connections 2021-02-10T11:22:26.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.527 7f1229d35700 1 -- 172.21.2.117:0/1221486680 wait complete. 2021-02-10T11:22:26.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 Processor -- start 2021-02-10T11:22:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 -- start start 2021-02-10T11:22:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f122408c080 0x7f122408c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122408caa0 0x7f122408f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122409d8a0 con 0x7f122409dd40 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12240b8bc0 con 0x7f122408caa0 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f1229d35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12240b8de0 con 0x7f122408c080 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f12237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f12237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41192/0 (socket says 172.21.2.117:41192) 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.542 7f12237fe700 1 -- 172.21.2.117:0/3908134501 learned_addr learned my addr 172.21.2.117:0/3908134501 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 -- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f122408c080 msgr2=0x7f122408c480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f122408c080 0x7f122408c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 -- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122408caa0 msgr2=0x7f122408f5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122408caa0 0x7f122408f5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 -- 172.21.2.117:0/3908134501 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122408cea0 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f12237fe700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f120c009630 tx=0x7f120c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c0093a0 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00bd70 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f1229d35700 1 -- 172.21.2.117:0/3908134501 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122408cea0 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f120c005560 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.543 7f1229d35700 1 -- 172.21.2.117:0/3908134501 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122408cea0 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.544 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f120c005770 con 0x7f122409dd40 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.544 7f121bfff700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 0x7f120801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.544 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f120c0321d0 con 0x7f122409dd40 2021-02-10T11:22:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.544 7f1222ffd700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 0x7f120801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.544 7f1222ffd700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 0x7f120801abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:26.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.575 7f1229d35700 1 -- 172.21.2.117:0/3908134501 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1224077a70 con 0x7f122409dd40 2021-02-10T11:22:26.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.577 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f120c005560 con 0x7f122409dd40 2021-02-10T11:22:26.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.703 7f1229d35700 1 -- 172.21.2.117:0/3908134501 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1224077a70 con 0x7f122409dd40 2021-02-10T11:22:26.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.704 7f121bfff700 1 -- 172.21.2.117:0/3908134501 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+64 (secure 0 0 0) 0x7f120c005770 con 0x7f122409dd40 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:26.702 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:22:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 msgr2=0x7f120801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 0x7f120801abc0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 msgr2=0x7f12240b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f120c009630 tx=0x7f120c00b980).stop 2021-02-10T11:22:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 shutdown_connections 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f122408caa0 0x7f122408f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f122408c080 0x7f122408c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12080186c0 0x7f120801abc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 --2- 172.21.2.117:0/3908134501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f122409dd40 0x7f12240b85a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 >> 172.21.2.117:0/3908134501 conn(0x7f122419f4c0 msgr2=0x7f1224094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 shutdown_connections 2021-02-10T11:22:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:26.706 7f1229d35700 1 -- 172.21.2.117:0/3908134501 wait complete. 2021-02-10T11:22:26.712 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:22:26.712 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:22:27.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 Processor -- start 2021-02-10T11:22:27.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 -- start start 2021-02-10T11:22:27.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e54079000 0x7f4e540e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e541b00f0 0x7f4e541b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e54077a70 con 0x7f4e5407da20 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e540e53c0 con 0x7f4e54079000 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e594e5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54079400 con 0x7f4e541b00f0 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e52abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e54079000 0x7f4e540e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e52abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e54079000 0x7f4e540e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41200/0 (socket says 172.21.2.117:41200) 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.126 7f4e52abe700 1 -- 172.21.2.117:0/3985614374 learned_addr learned my addr 172.21.2.117:0/3985614374 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e53ac0700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e541b00f0 0x7f4e541b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e52abe700 1 -- 172.21.2.117:0/3985614374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e541b00f0 msgr2=0x7f4e541b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e52abe700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e541b00f0 0x7f4e541b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e52abe700 1 -- 172.21.2.117:0/3985614374 >> v1:172.21.2.125:6789/0 conn(0x7f4e5407da20 legacy=0x7f4e540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e52abe700 1 -- 172.21.2.117:0/3985614374 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5407de20 con 0x7f4e54079000 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e52abe700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e54079000 0x7f4e540e4da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4e48006700 tx=0x7f4e48009b80).ready entity=mon.0 client_cookie=9636714ec7ba9376 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e522bd700 1 -- 172.21.2.117:0/3985614374 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e4800c0c0 con 0x7f4e54079000 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e522bd700 1 -- 172.21.2.117:0/3985614374 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4800c440 con 0x7f4e54079000 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e522bd700 1 -- 172.21.2.117:0/3985614374 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e4800c960 con 0x7f4e54079000 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e594e5700 1 -- 172.21.2.117:0/3985614374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e54079000 msgr2=0x7f4e540e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.127 7f4e594e5700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e54079000 0x7f4e540e4da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4e48006700 tx=0x7f4e48009b80).stop 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 -- 172.21.2.117:0/3985614374 shutdown_connections 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e541b00f0 0x7f4e541b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 --2- 172.21.2.117:0/3985614374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e54079000 0x7f4e540e4da0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 -- 172.21.2.117:0/3985614374 >> 172.21.2.117:0/3985614374 conn(0x7f4e540ce450 msgr2=0x7f4e540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 -- 172.21.2.117:0/3985614374 shutdown_connections 2021-02-10T11:22:27.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.128 7f4e594e5700 1 -- 172.21.2.117:0/3985614374 wait complete. 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.143 7f4e594e5700 1 Processor -- start 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.143 7f4e594e5700 1 -- start start 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 0x7f4e5408e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 0x7f4e5408e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540b1960 con 0x7f4e540990f0 2021-02-10T11:22:27.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e541a4670 con 0x7f4e540b1ee0 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e594e5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e541a4850 con 0x7f4e540b1360 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41206/0 (socket says 172.21.2.117:41206) 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 -- 172.21.2.117:0/2553367458 learned_addr learned my addr 172.21.2.117:0/2553367458 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e52abe700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 0x7f4e5408e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e53ac0700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 0x7f4e5408e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 -- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 msgr2=0x7f4e5408e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 0x7f4e5408e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 -- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 msgr2=0x7f4e5408e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 0x7f4e5408e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 -- 172.21.2.117:0/2553367458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54104300 con 0x7f4e540990f0 2021-02-10T11:22:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e52abe700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 0x7f4e5408e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e53ac0700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 0x7f4e5408e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.144 7f4e532bf700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4e4400b5b0 tx=0x7f4e4400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e4400bd40 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e44000c60 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e440062a0 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e54104300 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e54104890 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.145 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e44006020 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.146 7f4e3bfff700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 0x7f4e3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.146 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4753+0+0 (secure 0 0 0) 0x7f4e440322a0 con 0x7f4e540990f0 2021-02-10T11:22:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.146 7f4e52abe700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 0x7f4e3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:27.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.147 7f4e52abe700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 0x7f4e3401abc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:27.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.176 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e54103280 con 0x7f4e540990f0 2021-02-10T11:22:27.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.179 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e44000940 con 0x7f4e540990f0 2021-02-10T11:22:27.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.306 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f4e54077a70 con 0x7f4e540990f0 2021-02-10T11:22:27.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.898 7f4e3bfff700 1 -- 172.21.2.117:0/2553367458 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v17) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4e44006020 con 0x7f4e540990f0 2021-02-10T11:22:27.921 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:22:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 msgr2=0x7f4e3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 0x7f4e3401abc0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 msgr2=0x7f4e541a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4e4400b5b0 tx=0x7f4e4400b950).stop 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 shutdown_connections 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e540b1360 0x7f4e5408e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e540b1ee0 0x7f4e5408e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e340186c0 0x7f4e3401abc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 --2- 172.21.2.117:0/2553367458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e540990f0 0x7f4e541a4050 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.925 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 >> 172.21.2.117:0/2553367458 conn(0x7f4e540a1850 msgr2=0x7f4e540bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.926 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 shutdown_connections 2021-02-10T11:22:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:27.926 7f4e594e5700 1 -- 172.21.2.117:0/2553367458 wait complete. 2021-02-10T11:22:27.931 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2021-02-10T11:22:27.932 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.347 7fc6d700a700 1 Processor -- start 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.347 7fc6d700a700 1 -- start start 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d700a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc6d007da20 0x7fc6d00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d700a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6d00e4e00 0x7fc6d01b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d700a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d0077a10 con 0x7fc6d00e5200 2021-02-10T11:22:28.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d700a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d00777d0 con 0x7fc6d00e4e00 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d700a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d007de20 con 0x7fc6d007da20 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6d00e4e00 0x7fc6d01b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d5627700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc6d00e5200 0x7fc6d0077f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41462/0 (socket says 172.21.2.117:41462) 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6d5627700 1 -- 172.21.2.117:0/998782912 learned_addr learned my addr 172.21.2.117:0/998782912 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cffff700 1 -- 172.21.2.117:0/998782912 >> v2:172.21.2.125:3300/0 conn(0x7fc6d007da20 msgr2=0x7fc6d00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cffff700 1 --2- 172.21.2.117:0/998782912 >> v2:172.21.2.125:3300/0 conn(0x7fc6d007da20 0x7fc6d00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cffff700 1 -- 172.21.2.117:0/998782912 >> v1:172.21.2.117:6789/0 conn(0x7fc6d00e5200 legacy=0x7fc6d0077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cffff700 1 -- 172.21.2.117:0/998782912 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d01b0750 con 0x7fc6d00e4e00 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.348 7fc6cf7fe700 1 -- 172.21.2.117:0/998782912 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023193716 0 0) 0x7fc6c8001180 con 0x7fc6d00e5200 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6cffff700 1 --2- 172.21.2.117:0/998782912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d00e4e00 0x7fc6d01b0130 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc6c00069d0 tx=0x7fc6c0009dc0).ready entity=mon.0 client_cookie=413d766fcf60943f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6cf7fe700 1 -- 172.21.2.117:0/998782912 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6c000b510 con 0x7fc6d00e4e00 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6cf7fe700 1 -- 172.21.2.117:0/998782912 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c0006ac0 con 0x7fc6d00e4e00 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6cf7fe700 1 -- 172.21.2.117:0/998782912 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6c000bb10 con 0x7fc6d00e4e00 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6d700a700 1 -- 172.21.2.117:0/998782912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d00e4e00 msgr2=0x7fc6d01b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.349 7fc6d700a700 1 --2- 172.21.2.117:0/998782912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d00e4e00 0x7fc6d01b0130 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc6c00069d0 tx=0x7fc6c0009dc0).stop 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 -- 172.21.2.117:0/998782912 shutdown_connections 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 --2- 172.21.2.117:0/998782912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d00e4e00 0x7fc6d01b0130 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 --2- 172.21.2.117:0/998782912 >> v2:172.21.2.125:3300/0 conn(0x7fc6d007da20 0x7fc6d00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 -- 172.21.2.117:0/998782912 >> 172.21.2.117:0/998782912 conn(0x7fc6d00ce480 msgr2=0x7fc6d00e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 -- 172.21.2.117:0/998782912 shutdown_connections 2021-02-10T11:22:28.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.350 7fc6d700a700 1 -- 172.21.2.117:0/998782912 wait complete. 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.364 7fc6d700a700 1 Processor -- start 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.364 7fc6d700a700 1 -- start start 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6d009a630 0x7fc6d00c3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6d00c3940 0x7fc6d00c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d009a420 con 0x7fc6d0099d80 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d0190a90 con 0x7fc6d00c3940 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d700a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d01a6fc0 con 0x7fc6d009a630 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41216/0 (socket says 172.21.2.117:41216) 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 -- 172.21.2.117:0/2628961561 learned_addr learned my addr 172.21.2.117:0/2628961561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 -- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6d009a630 msgr2=0x7fc6d00c3320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6d009a630 0x7fc6d00c3320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 -- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6d00c3940 msgr2=0x7fc6d00c3d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6d00c3940 0x7fc6d00c3d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.365 7fc6d4e26700 1 -- 172.21.2.117:0/2628961561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d01a71e0 con 0x7fc6d0099d80 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6d4e26700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc6b8006f00 tx=0x7fc6b800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6b800bd30 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b8000c90 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6b80022a0 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d01a71e0 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d01a71e0 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6b8005310 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 0x7fc6b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.366 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5107+0+0 (secure 0 0 0) 0x7fc6b8032340 con 0x7fc6d0099d80 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.367 7fc6cffff700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 0x7fc6b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:28.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.368 7fc6cffff700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 0x7fc6b401abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:28.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.398 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d0190a60 con 0x7fc6d0099d80 2021-02-10T11:22:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.400 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6b8006b90 con 0x7fc6d0099d80 2021-02-10T11:22:28.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.534 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"} v 0) v1 -- 0x7fc6d0190a60 con 0x7fc6d0099d80 2021-02-10T11:22:28.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.902 7fc6cd7fa700 1 -- 172.21.2.117:0/2628961561 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"}]=0 v18) v1 ==== 154+0+0 (secure 0 0 0) 0x7fc6b8005310 con 0x7fc6d0099d80 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 msgr2=0x7fc6b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 0x7fc6b401abc0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 msgr2=0x7fc6d0190440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fc6b8006f00 tx=0x7fc6b800b940).stop 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 shutdown_connections 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6d00c3940 0x7fc6d00c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6d009a630 0x7fc6d00c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:28.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6b40186c0 0x7fc6b401abc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 --2- 172.21.2.117:0/2628961561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6d0099d80 0x7fc6d0190440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:28.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 >> 172.21.2.117:0/2628961561 conn(0x7fc6d00bff60 msgr2=0x7fc6d00bf330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:28.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 shutdown_connections 2021-02-10T11:22:28.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:28.953 7fc6d700a700 1 -- 172.21.2.117:0/2628961561 wait complete. 2021-02-10T11:22:28.959 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2021-02-10T11:22:29.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.372 7f25a88d3700 1 Processor -- start 2021-02-10T11:22:29.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.372 7f25a88d3700 1 -- start start 2021-02-10T11:22:29.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a88d3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f25a4078fa0 0x7f25a40e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a88d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25a41b0130 0x7f25a41b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a88d3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a4077a10 con 0x7f25a407da20 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a88d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a40e5c60 con 0x7f25a41b0130 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a88d3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a40e5420 con 0x7f25a4078fa0 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25a41b0130 0x7f25a41b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25a41b0130 0x7f25a41b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41228/0 (socket says 172.21.2.117:41228) 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 -- 172.21.2.117:0/1704648911 learned_addr learned my addr 172.21.2.117:0/1704648911 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a1e1b700 1 --2- 172.21.2.117:0/1704648911 >> v2:172.21.2.125:3300/0 conn(0x7f25a4078fa0 0x7f25a40e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 -- 172.21.2.117:0/1704648911 >> v2:172.21.2.125:3300/0 conn(0x7f25a4078fa0 msgr2=0x7f25a40e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 --2- 172.21.2.117:0/1704648911 >> v2:172.21.2.125:3300/0 conn(0x7f25a4078fa0 0x7f25a40e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 -- 172.21.2.117:0/1704648911 >> v1:172.21.2.117:6789/0 conn(0x7f25a407da20 legacy=0x7f25a40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.373 7f25a2e1d700 1 -- 172.21.2.117:0/1704648911 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a40793a0 con 0x7f25a41b0130 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a2e1d700 1 --2- 172.21.2.117:0/1704648911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a41b0130 0x7f25a41b0530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f259c006f00 tx=0x7f259c009f10).ready entity=mon.0 client_cookie=512da3c6d2018664 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a161a700 1 -- 172.21.2.117:0/1704648911 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259c00b3b0 con 0x7f25a41b0130 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a161a700 1 -- 172.21.2.117:0/1704648911 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c00b630 con 0x7f25a41b0130 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a161a700 1 -- 172.21.2.117:0/1704648911 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259c00bb50 con 0x7f25a41b0130 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a88d3700 1 -- 172.21.2.117:0/1704648911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a41b0130 msgr2=0x7f25a41b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a88d3700 1 --2- 172.21.2.117:0/1704648911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a41b0130 0x7f25a41b0530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f259c006f00 tx=0x7f259c009f10).stop 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a88d3700 1 -- 172.21.2.117:0/1704648911 shutdown_connections 2021-02-10T11:22:29.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a88d3700 1 --2- 172.21.2.117:0/1704648911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a41b0130 0x7f25a41b0530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.374 7f25a88d3700 1 --2- 172.21.2.117:0/1704648911 >> v2:172.21.2.125:3300/0 conn(0x7f25a4078fa0 0x7f25a40e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.375 7f25a88d3700 1 -- 172.21.2.117:0/1704648911 >> 172.21.2.117:0/1704648911 conn(0x7f25a40ce480 msgr2=0x7f25a40e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.375 7f25a88d3700 1 -- 172.21.2.117:0/1704648911 shutdown_connections 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.375 7f25a88d3700 1 -- 172.21.2.117:0/1704648911 wait complete. 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 Processor -- start 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 -- start start 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25a408e250 0x7f25a4098d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25a41b7560 0x7f25a41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40ff4a0 con 0x7f25a408e650 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41b7f80 con 0x7f25a408e250 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a88d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41b81a0 con 0x7f25a41b7560 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a1e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.389 7f25a1e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41234/0 (socket says 172.21.2.117:41234) 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 -- 172.21.2.117:0/2344913820 learned_addr learned my addr 172.21.2.117:0/2344913820 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 -- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25a41b7560 msgr2=0x7f25a41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25a41b7560 0x7f25a41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 -- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25a408e250 msgr2=0x7f25a4098d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:29.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25a408e250 0x7f25a4098d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 -- 172.21.2.117:0/2344913820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a41b83c0 con 0x7f25a408e650 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a1e1b700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2594006f00 tx=0x7f259400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f259400bd30 con 0x7f25a408e650 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594002030 con 0x7f25a408e650 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2594002550 con 0x7f25a408e650 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a41b83c0 con 0x7f25a408e650 2021-02-10T11:22:29.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.390 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a41b83c0 con 0x7f25a408e650 2021-02-10T11:22:29.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.391 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2594000c90 con 0x7f25a408e650 2021-02-10T11:22:29.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.391 7f259affd700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 0x7f258801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:29.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.391 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5363+0+0 (secure 0 0 0) 0x7f2594032520 con 0x7f25a408e650 2021-02-10T11:22:29.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.392 7f25a261c700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 0x7f258801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:29.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.392 7f25a261c700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 0x7f258801abc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:29.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.423 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4098830 con 0x7f25a408e650 2021-02-10T11:22:29.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.425 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f259401f4b0 con 0x7f25a408e650 2021-02-10T11:22:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:29.551 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"} v 0) v1 -- 0x7f25a4098830 con 0x7f25a408e650 2021-02-10T11:22:30.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.916 7f259affd700 1 -- 172.21.2.117:0/2344913820 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v20) v1 ==== 188+0+0 (secure 0 0 0) 0x7f259401f6c0 con 0x7f25a408e650 2021-02-10T11:22:30.921 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:22:30.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 msgr2=0x7f258801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 0x7f258801abc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 msgr2=0x7f25a4099380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:30.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2594006f00 tx=0x7f259400b940).stop 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 shutdown_connections 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2588018730 0x7f258801abc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25a408e650 0x7f25a4099380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25a41b7560 0x7f25a41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 --2- 172.21.2.117:0/2344913820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25a408e250 0x7f25a4098d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 >> 172.21.2.117:0/2344913820 conn(0x7f25a40ba5e0 msgr2=0x7f25a409ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 shutdown_connections 2021-02-10T11:22:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:30.925 7f25a88d3700 1 -- 172.21.2.117:0/2344913820 wait complete. 2021-02-10T11:22:30.931 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2021-02-10T11:22:31.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 Processor -- start 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 -- start start 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d807da20 0x7fe8d80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8d80e4df0 0x7fe8d81b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d8077a10 con 0x7fe8d80e51f0 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d80e5c50 con 0x7fe8d80e4df0 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.345 7fe8dddb6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80777d0 con 0x7fe8d807da20 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8d80e4df0 0x7fe8d81b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8d80e4df0 0x7fe8d81b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41238/0 (socket says 172.21.2.117:41238) 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 -- 172.21.2.117:0/2237785185 learned_addr learned my addr 172.21.2.117:0/2237785185 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d77fe700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d807da20 0x7fe8d80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 -- 172.21.2.117:0/2237785185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d807da20 msgr2=0x7fe8d80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d807da20 0x7fe8d80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 -- 172.21.2.117:0/2237785185 >> v1:172.21.2.125:6789/0 conn(0x7fe8d80e51f0 legacy=0x7fe8d8077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 -- 172.21.2.117:0/2237785185 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d807de20 con 0x7fe8d80e4df0 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.346 7fe8d6ffd700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80e4df0 0x7fe8d81b0120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fe8cc006700 tx=0x7fe8cc009b80).ready entity=mon.0 client_cookie=443af322e3763c0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8d67fc700 1 -- 172.21.2.117:0/2237785185 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc00c0c0 con 0x7fe8d80e4df0 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8d67fc700 1 -- 172.21.2.117:0/2237785185 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00c400 con 0x7fe8d80e4df0 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8d67fc700 1 -- 172.21.2.117:0/2237785185 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc00c920 con 0x7fe8d80e4df0 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 -- 172.21.2.117:0/2237785185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80e4df0 msgr2=0x7fe8d81b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80e4df0 0x7fe8d81b0120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fe8cc006700 tx=0x7fe8cc009b80).stop 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 -- 172.21.2.117:0/2237785185 shutdown_connections 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80e4df0 0x7fe8d81b0120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 --2- 172.21.2.117:0/2237785185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d807da20 0x7fe8d80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.347 7fe8dddb6700 1 -- 172.21.2.117:0/2237785185 >> 172.21.2.117:0/2237785185 conn(0x7fe8d80ce470 msgr2=0x7fe8d80e42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.348 7fe8dddb6700 1 -- 172.21.2.117:0/2237785185 shutdown_connections 2021-02-10T11:22:31.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.348 7fe8dddb6700 1 -- 172.21.2.117:0/2237785185 wait complete. 2021-02-10T11:22:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.362 7fe8dddb6700 1 Processor -- start 2021-02-10T11:22:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.362 7fe8dddb6700 1 -- start start 2021-02-10T11:22:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8d81b8480 0x7fe8d81b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d80b42d0 0x7fe8d8103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d80b49b0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d81b8ea0 con 0x7fe8d81b8480 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8dddb6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d8104040 con 0x7fe8d80b42d0 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41246/0 (socket says 172.21.2.117:41246) 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 -- 172.21.2.117:0/341758753 learned_addr learned my addr 172.21.2.117:0/341758753 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 -- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d80b42d0 msgr2=0x7fe8d8103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d80b42d0 0x7fe8d8103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 -- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8d81b8480 msgr2=0x7fe8d81b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8d81b8480 0x7fe8d81b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 -- 172.21.2.117:0/341758753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8104260 con 0x7fe8d80b38b0 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8d6ffd700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe8cc006420 tx=0x7fe8cc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.363 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc009440 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8cc00bd50 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8cc0054d0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d8104260 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d8104260 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8cc0052e0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8bffff700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 0x7fe8b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.364 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5815+0+0 (secure 0 0 0) 0x7fe8cc0325e0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.365 7fe8d77fe700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 0x7fe8b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.365 7fe8d77fe700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 0x7fe8b801abc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:31.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.395 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d81b8ea0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.398 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8cc0277e0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.524 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"} v 0) v1 -- 0x7fe8d81b8ea0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.918 7fe8bffff700 1 -- 172.21.2.117:0/341758753 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"}]=0 set pool 3 allow_ec_overwrites to true v21) v1 ==== 166+0+0 (secure 0 0 0) 0x7fe8cc0277e0 con 0x7fe8d80b38b0 2021-02-10T11:22:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:set pool 3 allow_ec_overwrites to true 2021-02-10T11:22:31.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.943 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 msgr2=0x7fe8b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.943 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 0x7fe8b801abc0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.943 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 msgr2=0x7fe8d80b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.943 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fe8cc006420 tx=0x7fe8cc00b960).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.943 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 shutdown_connections 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8b80186c0 0x7fe8b801abc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8d80b38b0 0x7fe8d80b3cb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8d80b42d0 0x7fe8d8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 --2- 172.21.2.117:0/341758753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8d81b8480 0x7fe8d81b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 >> 172.21.2.117:0/341758753 conn(0x7fe8d80b1480 msgr2=0x7fe8d81b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 shutdown_connections 2021-02-10T11:22:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:31.944 7fe8dddb6700 1 -- 172.21.2.117:0/341758753 wait complete. 2021-02-10T11:22:31.950 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.364 7f37cfdc9700 1 Processor -- start 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.364 7f37cfdc9700 1 -- start start 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f37c807da20 0x7f37c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c80e4e00 0x7f37c81b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c8077a10 con 0x7f37c80e5200 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c80777d0 con 0x7f37c807da20 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c807de20 con 0x7f37c80e4e00 2021-02-10T11:22:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f37c807da20 0x7f37c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f37c807da20 0x7f37c80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41250/0 (socket says 172.21.2.117:41250) 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 -- 172.21.2.117:0/1568399857 learned_addr learned my addr 172.21.2.117:0/1568399857 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cd3e4700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c80e4e00 0x7f37c81b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 -- 172.21.2.117:0/1568399857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c80e4e00 msgr2=0x7f37c81b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c80e4e00 0x7f37c81b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 -- 172.21.2.117:0/1568399857 >> v1:172.21.2.125:6789/0 conn(0x7f37c80e5200 legacy=0x7f37c8077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 -- 172.21.2.117:0/1568399857 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c81b0750 con 0x7f37c807da20 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cdbe5700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c807da20 0x7f37c80771b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f37c0006a80 tx=0x7f37c0009f00).ready entity=mon.0 client_cookie=7ba410d4320b61ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37ccbe3700 1 -- 172.21.2.117:0/1568399857 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c000b5c0 con 0x7f37c807da20 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37ccbe3700 1 -- 172.21.2.117:0/1568399857 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c000b840 con 0x7f37c807da20 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37ccbe3700 1 -- 172.21.2.117:0/1568399857 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37c000bd20 con 0x7f37c807da20 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 -- 172.21.2.117:0/1568399857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c807da20 msgr2=0x7f37c80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.365 7f37cfdc9700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c807da20 0x7f37c80771b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f37c0006a80 tx=0x7f37c0009f00).stop 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 -- 172.21.2.117:0/1568399857 shutdown_connections 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c80e4e00 0x7f37c81b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 --2- 172.21.2.117:0/1568399857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c807da20 0x7f37c80771b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 -- 172.21.2.117:0/1568399857 >> 172.21.2.117:0/1568399857 conn(0x7f37c80ce480 msgr2=0x7f37c80e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 -- 172.21.2.117:0/1568399857 shutdown_connections 2021-02-10T11:22:32.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.366 7f37cfdc9700 1 -- 172.21.2.117:0/1568399857 wait complete. 2021-02-10T11:22:32.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.380 7f37cfdc9700 1 Processor -- start 2021-02-10T11:22:32.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.380 7f37cfdc9700 1 -- start start 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c8106ce0 0x7f37c80bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37c8092230 0x7f37c8092630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8091cb0 con 0x7f37c80901f0 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8091970 con 0x7f37c8092230 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cfdc9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8092c50 con 0x7f37c8106ce0 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41258/0 (socket says 172.21.2.117:41258) 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 -- 172.21.2.117:0/26442136 learned_addr learned my addr 172.21.2.117:0/26442136 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37ce3e6700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37c8092230 0x7f37c8092630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cdbe5700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c8106ce0 0x7f37c80bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 -- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c8106ce0 msgr2=0x7f37c80bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c8106ce0 0x7f37c80bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 -- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37c8092230 msgr2=0x7f37c8092630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37c8092230 0x7f37c8092630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.381 7f37cd3e4700 1 -- 172.21.2.117:0/26442136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c8092e70 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37cd3e4700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f37b8006f00 tx=0x7f37b800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37b800bd30 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b8002030 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37b8002490 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c8092e70 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c8092e70 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37b8000c90 con 0x7f37c80901f0 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 0x7f37b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.382 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5815+0+0 (secure 0 0 0) 0x7f37b80324c0 con 0x7f37c80901f0 2021-02-10T11:22:32.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.383 7f37cdbe5700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 0x7f37b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.383 7f37cdbe5700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 0x7f37b001abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.414 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c8077a10 con 0x7f37c80901f0 2021-02-10T11:22:32.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.417 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37b8030d00 con 0x7f37c80901f0 2021-02-10T11:22:32.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.543 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f37c8077a10 con 0x7f37c80901f0 2021-02-10T11:22:32.542 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:22:32.544 7fa03f142700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:22:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.547 7f37be7fc700 1 -- 172.21.2.117:0/26442136 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 187+0+0 (secure 0 0 0) 0x7f37b8002420 con 0x7f37c80901f0 2021-02-10T11:22:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 2 and data pool 3 2021-02-10T11:22:32.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 msgr2=0x7f37b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 0x7f37b001abc0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 msgr2=0x7f37c80905f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f37b8006f00 tx=0x7f37b800b940).stop 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 shutdown_connections 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37b00186c0 0x7f37b001abc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37c80901f0 0x7f37c80905f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37c8092230 0x7f37c8092630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 --2- 172.21.2.117:0/26442136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37c8106ce0 0x7f37c80bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 >> 172.21.2.117:0/26442136 conn(0x7f37c80a5570 msgr2=0x7f37c8107b00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:32.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 shutdown_connections 2021-02-10T11:22:32.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.552 7f37cfdc9700 1 -- 172.21.2.117:0/26442136 wait complete. 2021-02-10T11:22:32.558 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.969 7f116f5bd700 1 Processor -- start 2021-02-10T11:22:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.969 7f116f5bd700 1 -- start start 2021-02-10T11:22:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116f5bd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116f5bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 0x7f11681b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116f5bd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168077a70 con 0x7f11680e5190 2021-02-10T11:22:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116f5bd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11680d33f0 con 0x7f116807da20 2021-02-10T11:22:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116f5bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11680777d0 con 0x7f11680e4d90 2021-02-10T11:22:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116d3d9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116d3d9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:45648/0 (socket says 172.21.2.117:45648) 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116d3d9700 1 -- 172.21.2.117:0/1043122608 learned_addr learned my addr 172.21.2.117:0/1043122608 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116dbda700 1 --1- 172.21.2.117:0/1043122608 >> v1:172.21.2.125:6789/0 conn(0x7f11680e5190 0x7f1168077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34442/0 (socket says 172.21.2.117:34442) 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 --2- 172.21.2.117:0/1043122608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 0x7f11681b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 -- 172.21.2.117:0/1043122608 >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 msgr2=0x7f11680771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 --2- 172.21.2.117:0/1043122608 >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 -- 172.21.2.117:0/1043122608 >> v1:172.21.2.125:6789/0 conn(0x7f11680e5190 legacy=0x7f1168077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 -- 172.21.2.117:0/1043122608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116807de20 con 0x7f11680e4d90 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116d3d9700 1 --2- 172.21.2.117:0/1043122608 >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.970 7f116cbd8700 1 --2- 172.21.2.117:0/1043122608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 0x7f11681b00e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1164006650 tx=0x7f1164009a80).ready entity=mon.2 client_cookie=dfd916a9cb5d013e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f115bfff700 1 -- 172.21.2.117:0/1043122608 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11640009b0 con 0x7f11680e4d90 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f115bfff700 1 -- 172.21.2.117:0/1043122608 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116400c0c0 con 0x7f11680e4d90 2021-02-10T11:22:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f115bfff700 1 -- 172.21.2.117:0/1043122608 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f116400c520 con 0x7f11680e4d90 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 -- 172.21.2.117:0/1043122608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 msgr2=0x7f11681b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 --2- 172.21.2.117:0/1043122608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 0x7f11681b00e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1164006650 tx=0x7f1164009a80).stop 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 -- 172.21.2.117:0/1043122608 shutdown_connections 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 --2- 172.21.2.117:0/1043122608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680e4d90 0x7f11681b00e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 --2- 172.21.2.117:0/1043122608 >> v2:172.21.2.125:3300/0 conn(0x7f116807da20 0x7f11680771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.971 7f116f5bd700 1 -- 172.21.2.117:0/1043122608 >> 172.21.2.117:0/1043122608 conn(0x7f11680ce440 msgr2=0x7f11680e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.972 7f116f5bd700 1 -- 172.21.2.117:0/1043122608 shutdown_connections 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.972 7f116f5bd700 1 -- 172.21.2.117:0/1043122608 wait complete. 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.986 7f116f5bd700 1 Processor -- start 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.986 7f116f5bd700 1 -- start start 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 0x7f1168192ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 0x7f1168193b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168193590 con 0x7f116818ebe0 2021-02-10T11:22:32.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168106ed0 con 0x7f1168106ad0 2021-02-10T11:22:32.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116f5bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11681931c0 con 0x7f11680b8bd0 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116dbda700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 0x7f1168193b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116cbd8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 0x7f1168192ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116dbda700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 0x7f1168193b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:45656/0 (socket says 172.21.2.117:45656) 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116cbd8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41272/0 (socket says 172.21.2.117:41272) 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116dbda700 1 -- 172.21.2.117:0/1708436416 learned_addr learned my addr 172.21.2.117:0/1708436416 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 -- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 msgr2=0x7f1168193b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 0x7f1168193b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 -- 172.21.2.117:0/1708436416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 msgr2=0x7f11681064b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116d3d9700 1 -- 172.21.2.117:0/1708436416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168194130 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.987 7f116cbd8700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f116d3d9700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 0x7f1168192ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f115c006b10 tx=0x7f115c00b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f115c00bcb0 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c006030 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f115c005560 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1168194130 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168194130 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f115c006210 con 0x7f11680b8bd0 2021-02-10T11:22:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 0x7f114801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:32.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.988 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f115c0324e0 con 0x7f11680b8bd0 2021-02-10T11:22:32.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.989 7f116cbd8700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 0x7f114801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:32.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:32.989 7f116cbd8700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 0x7f114801abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.020 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168077a70 con 0x7f11680b8bd0 2021-02-10T11:22:33.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.024 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f115c00bcb0 con 0x7f11680b8bd0 2021-02-10T11:22:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.147 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1168077a70 con 0x7f11680b8bd0 2021-02-10T11:22:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.148 7f1159ffb700 1 -- 172.21.2.117:0/1708436416 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f115c006210 con 0x7f11680b8bd0 2021-02-10T11:22:33.148 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:33.149 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:33.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.152 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 msgr2=0x7f114801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.152 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 0x7f114801abc0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.152 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 msgr2=0x7f1168192ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 0x7f1168192ba0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f115c006b10 tx=0x7f115c00b8c0).stop 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 shutdown_connections 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11480186c0 0x7f114801abc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f116818ebe0 0x7f11681064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1168106ad0 0x7f1168193b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 --2- 172.21.2.117:0/1708436416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11680b8bd0 0x7f1168192ba0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 >> 172.21.2.117:0/1708436416 conn(0x7f116819f510 msgr2=0x7f11681abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 shutdown_connections 2021-02-10T11:22:33.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.153 7f116f5bd700 1 -- 172.21.2.117:0/1708436416 wait complete. 2021-02-10T11:22:33.159 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.570 7fcacf785700 1 Processor -- start 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 -- start start 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 0x7fcac81b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 0x7fcac81b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac8109ab0 con 0x7fcac81085f0 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac80fde80 con 0x7fcac81b3470 2021-02-10T11:22:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.571 7fcacf785700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80d33f0 con 0x7fcac80d0ac0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacdda2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcac81085f0 0x7fcac81089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41532/0 (socket says 172.21.2.117:41532) 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacdda2700 1 -- 172.21.2.117:0/44879633 learned_addr learned my addr 172.21.2.117:0/44879633 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 --2- 172.21.2.117:0/44879633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 0x7fcac81b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcaccda0700 1 --2- 172.21.2.117:0/44879633 >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 0x7fcac81b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcabffff700 1 -- 172.21.2.117:0/44879633 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134494724 0 0) 0x7fcac0000d30 con 0x7fcac81085f0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcabffff700 1 -- 172.21.2.117:0/44879633 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab4002fe0 con 0x7fcac81085f0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 -- 172.21.2.117:0/44879633 >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 msgr2=0x7fcac81b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 --2- 172.21.2.117:0/44879633 >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 0x7fcac81b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 -- 172.21.2.117:0/44879633 >> v1:172.21.2.117:6789/0 conn(0x7fcac81085f0 legacy=0x7fcac81089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 -- 172.21.2.117:0/44879633 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80d0120 con 0x7fcac80d0ac0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcaccda0700 1 --2- 172.21.2.117:0/44879633 >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 0x7fcac81b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcacd5a1700 1 --2- 172.21.2.117:0/44879633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 0x7fcac81b2e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcab8006650 tx=0x7fcab8009a80).ready entity=mon.2 client_cookie=8be462a7ce8295f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.572 7fcabffff700 1 -- 172.21.2.117:0/44879633 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab800c0c0 con 0x7fcac80d0ac0 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcabffff700 1 -- 172.21.2.117:0/44879633 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab800c440 con 0x7fcac80d0ac0 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcabffff700 1 -- 172.21.2.117:0/44879633 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcab800c920 con 0x7fcac80d0ac0 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 -- 172.21.2.117:0/44879633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 msgr2=0x7fcac81b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 --2- 172.21.2.117:0/44879633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 0x7fcac81b2e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fcab8006650 tx=0x7fcab8009a80).stop 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 -- 172.21.2.117:0/44879633 shutdown_connections 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 --2- 172.21.2.117:0/44879633 >> v2:172.21.2.125:3300/0 conn(0x7fcac81b3470 0x7fcac81b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 --2- 172.21.2.117:0/44879633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80d0ac0 0x7fcac81b2e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 -- 172.21.2.117:0/44879633 >> 172.21.2.117:0/44879633 conn(0x7fcac80ce410 msgr2=0x7fcac80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 -- 172.21.2.117:0/44879633 shutdown_connections 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.573 7fcacf785700 1 -- 172.21.2.117:0/44879633 wait complete. 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 Processor -- start 2021-02-10T11:22:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 -- start start 2021-02-10T11:22:33.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80962f0 0x7fcac80966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcac819ff40 0x7fcac81a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80f0b30 con 0x7fcac8096d10 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac80ef030 con 0x7fcac819ff40 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.588 7fcacf785700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81a0960 con 0x7fcac80962f0 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41286/0 (socket says 172.21.2.117:41286) 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 -- 172.21.2.117:0/2092691332 learned_addr learned my addr 172.21.2.117:0/2092691332 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcacdda2700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcac819ff40 0x7fcac81a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 -- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80962f0 msgr2=0x7fcac80966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80962f0 0x7fcac80966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 -- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcac819ff40 msgr2=0x7fcac81a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcac819ff40 0x7fcac81a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 -- 172.21.2.117:0/2092691332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac80edb90 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcaccda0700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fcac4006f40 tx=0x7fcac400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcac400bd80 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4000c90 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcac4002220 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcacf785700 1 -- 172.21.2.117:0/2092691332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac80edb90 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.589 7fcacf785700 1 -- 172.21.2.117:0/2092691332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac80edb90 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.590 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcac400a000 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.590 7fcabdffb700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 0x7fcaa801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.590 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcac40328f0 con 0x7fcac8096d10 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.590 7fcacd5a1700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 0x7fcaa801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.591 7fcacd5a1700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 0x7fcaa801ac10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:33.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.621 7fcacf785700 1 -- 172.21.2.117:0/2092691332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac8109ab0 con 0x7fcac8096d10 2021-02-10T11:22:33.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.623 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcac4006b00 con 0x7fcac8096d10 2021-02-10T11:22:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.748 7fcacf785700 1 -- 172.21.2.117:0/2092691332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcac8109ab0 con 0x7fcac8096d10 2021-02-10T11:22:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.748 7fcabdffb700 1 -- 172.21.2.117:0/2092691332 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7fcac4000c90 con 0x7fcac8096d10 2021-02-10T11:22:33.747 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:33.747 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 -- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 msgr2=0x7fcaa801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 0x7fcaa801ac10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 -- 172.21.2.117:0/2092691332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 msgr2=0x7fcac819f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fcac4006f40 tx=0x7fcac400b990).stop 2021-02-10T11:22:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 -- 172.21.2.117:0/2092691332 shutdown_connections 2021-02-10T11:22:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaa8018730 0x7fcaa801ac10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcac8096d10 0x7fcac819f920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcac819ff40 0x7fcac81a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 --2- 172.21.2.117:0/2092691332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcac80962f0 0x7fcac80966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.750 7fcacf785700 1 -- 172.21.2.117:0/2092691332 >> 172.21.2.117:0/2092691332 conn(0x7fcac81b3690 msgr2=0x7fcac808c260 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.751 7fcacf785700 1 -- 172.21.2.117:0/2092691332 shutdown_connections 2021-02-10T11:22:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:33.751 7fcacf785700 1 -- 172.21.2.117:0/2092691332 wait complete. 2021-02-10T11:22:33.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:22:34.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.171 7fd2c6614700 1 Processor -- start 2021-02-10T11:22:34.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.171 7fd2c6614700 1 -- start start 2021-02-10T11:22:34.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c6614700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd2c007da20 0x7fd2c00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c6614700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2c00e51a0 0x7fd2c0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c6614700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c0077a70 con 0x7fd2c00e4da0 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c6614700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c00e5c20 con 0x7fd2c00e51a0 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c6614700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c00777d0 con 0x7fd2c007da20 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2c00e51a0 0x7fd2c0079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2c00e51a0 0x7fd2c0079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41292/0 (socket says 172.21.2.117:41292) 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 -- 172.21.2.117:0/556270169 learned_addr learned my addr 172.21.2.117:0/556270169 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 -- 172.21.2.117:0/556270169 >> v2:172.21.2.125:3300/0 conn(0x7fd2c007da20 msgr2=0x7fd2c00771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 --2- 172.21.2.117:0/556270169 >> v2:172.21.2.125:3300/0 conn(0x7fd2c007da20 0x7fd2c00771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 -- 172.21.2.117:0/556270169 >> v1:172.21.2.125:6789/0 conn(0x7fd2c00e4da0 legacy=0x7fd2c01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 -- 172.21.2.117:0/556270169 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c01b08e0 con 0x7fd2c00e51a0 2021-02-10T11:22:34.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2c4c31700 1 --2- 172.21.2.117:0/556270169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c00e51a0 0x7fd2c0079000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd2b8006f00 tx=0x7fd2b8009f10).ready entity=mon.0 client_cookie=400d587f5825da02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.172 7fd2beffd700 1 -- 172.21.2.117:0/556270169 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b800b3b0 con 0x7fd2c00e51a0 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2beffd700 1 -- 172.21.2.117:0/556270169 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b800b630 con 0x7fd2c00e51a0 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2beffd700 1 -- 172.21.2.117:0/556270169 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b800ba90 con 0x7fd2c00e51a0 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 -- 172.21.2.117:0/556270169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c00e51a0 msgr2=0x7fd2c0079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 --2- 172.21.2.117:0/556270169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c00e51a0 0x7fd2c0079000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd2b8006f00 tx=0x7fd2b8009f10).stop 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 -- 172.21.2.117:0/556270169 shutdown_connections 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 --2- 172.21.2.117:0/556270169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c00e51a0 0x7fd2c0079000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 --2- 172.21.2.117:0/556270169 >> v2:172.21.2.125:3300/0 conn(0x7fd2c007da20 0x7fd2c00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 -- 172.21.2.117:0/556270169 >> 172.21.2.117:0/556270169 conn(0x7fd2c00ce450 msgr2=0x7fd2c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 -- 172.21.2.117:0/556270169 shutdown_connections 2021-02-10T11:22:34.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.173 7fd2c6614700 1 -- 172.21.2.117:0/556270169 wait complete. 2021-02-10T11:22:34.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 Processor -- start 2021-02-10T11:22:34.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 -- start start 2021-02-10T11:22:34.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2c0093c80 0x7fd2c01a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2c01a7bf0 0x7fd2c0094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0095740 con 0x7fd2c0094080 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0095960 con 0x7fd2c0093c80 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c6614700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0095b80 con 0x7fd2c01a7bf0 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c4c31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c4c31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41300/0 (socket says 172.21.2.117:41300) 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.188 7fd2c4c31700 1 -- 172.21.2.117:0/3690831536 learned_addr learned my addr 172.21.2.117:0/3690831536 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 -- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2c01a7bf0 msgr2=0x7fd2c0094b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2c01a7bf0 0x7fd2c0094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 -- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2c0093c80 msgr2=0x7fd2c01a75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2c0093c80 0x7fd2c01a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 -- 172.21.2.117:0/3690831536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0095da0 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c4c31700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd2b8009390 tx=0x7fd2b800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b8003030 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b80068e0 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b800c570 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c0095da0 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.189 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c00ac7a0 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.190 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2b8001270 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.190 7fd2bcff9700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 0x7fd2a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.190 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fd2b8032ca0 con 0x7fd2c0094080 2021-02-10T11:22:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.190 7fd2bffff700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 0x7fd2a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.191 7fd2bffff700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 0x7fd2a401abc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:34.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.221 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c0103310 con 0x7fd2c0094080 2021-02-10T11:22:34.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.224 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2b8009450 con 0x7fd2c0094080 2021-02-10T11:22:34.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.359 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fd2c0077a70 con 0x7fd2c0094080 2021-02-10T11:22:34.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.931 7fd2bcff9700 1 -- 172.21.2.117:0/3690831536 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd2b80068e0 con 0x7fd2c0094080 2021-02-10T11:22:34.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 msgr2=0x7fd2a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 0x7fd2a401abc0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 msgr2=0x7fd2c0095120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:34.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd2b8009390 tx=0x7fd2b800bd40).stop 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 shutdown_connections 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a40186c0 0x7fd2a401abc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2c0094080 0x7fd2c0095120 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2c01a7bf0 0x7fd2c0094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 --2- 172.21.2.117:0/3690831536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2c0093c80 0x7fd2c01a75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 >> 172.21.2.117:0/3690831536 conn(0x7fd2c00a1880 msgr2=0x7fd2c00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 shutdown_connections 2021-02-10T11:22:34.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:34.980 7fd2c6614700 1 -- 172.21.2.117:0/3690831536 wait complete. 2021-02-10T11:22:34.986 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:35.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.398 7f5a7cc5c700 1 Processor -- start 2021-02-10T11:22:35.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.398 7f5a7cc5c700 1 -- start start 2021-02-10T11:22:35.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.398 7f5a7cc5c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5a78109600 0x7f5a781b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7cc5c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78109ab0 con 0x7f5a780e4190 2021-02-10T11:22:35.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7cc5c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a781b3c60 con 0x7f5a780d0ac0 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7cc5c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a780d0ec0 con 0x7f5a78109600 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5a78109600 0x7f5a781b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5a78109600 0x7f5a781b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41306/0 (socket says 172.21.2.117:41306) 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7705f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5a780e4190 0x7f5a780e4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41558/0 (socket says 172.21.2.117:41558) 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 -- 172.21.2.117:0/4144430053 learned_addr learned my addr 172.21.2.117:0/4144430053 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 -- 172.21.2.117:0/4144430053 >> v1:172.21.2.125:6789/0 conn(0x7f5a780d0ac0 legacy=0x7f5a781b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 -- 172.21.2.117:0/4144430053 >> v1:172.21.2.117:6789/0 conn(0x7f5a780e4190 legacy=0x7f5a780e4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 -- 172.21.2.117:0/4144430053 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7810b040 con 0x7f5a78109600 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7605d700 1 --2- 172.21.2.117:0/4144430053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a78109600 0x7f5a781b3640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5a68006700 tx=0x7f5a68009b00).ready entity=mon.0 client_cookie=7e8a6fdaeb433059 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.399 7f5a7585c700 1 -- 172.21.2.117:0/4144430053 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a680009b0 con 0x7f5a78109600 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7585c700 1 -- 172.21.2.117:0/4144430053 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a68006390 con 0x7f5a78109600 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7585c700 1 -- 172.21.2.117:0/4144430053 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6800c0c0 con 0x7f5a78109600 2021-02-10T11:22:35.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 -- 172.21.2.117:0/4144430053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a78109600 msgr2=0x7f5a781b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 --2- 172.21.2.117:0/4144430053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a78109600 0x7f5a781b3640 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5a68006700 tx=0x7f5a68009b00).stop 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 -- 172.21.2.117:0/4144430053 shutdown_connections 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 --2- 172.21.2.117:0/4144430053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a78109600 0x7f5a781b3640 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 -- 172.21.2.117:0/4144430053 >> 172.21.2.117:0/4144430053 conn(0x7f5a780ce410 msgr2=0x7f5a780d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 -- 172.21.2.117:0/4144430053 shutdown_connections 2021-02-10T11:22:35.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.400 7f5a7cc5c700 1 -- 172.21.2.117:0/4144430053 wait complete. 2021-02-10T11:22:35.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 Processor -- start 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 -- start start 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5a781c1b90 0x7f5a780ba1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5a781c1f90 0x7f5a780ba7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a780920f0 con 0x7f5a780b1b40 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a781b2720 con 0x7f5a781c1b90 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7cc5c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a781b2940 con 0x7f5a781c1f90 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41310/0 (socket says 172.21.2.117:41310) 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.415 7f5a7685e700 1 -- 172.21.2.117:0/478043565 learned_addr learned my addr 172.21.2.117:0/478043565 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 -- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5a781c1f90 msgr2=0x7f5a780ba7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:35.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5a781c1f90 0x7f5a780ba7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 -- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5a781c1b90 msgr2=0x7f5a780ba1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5a781c1b90 0x7f5a780ba1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 -- 172.21.2.117:0/478043565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a780badf0 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7685e700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5a6000b5b0 tx=0x7f5a6000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a6000bd40 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60006020 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5a600054a0 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a780badf0 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.416 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a780badf0 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.417 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5a60006200 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.417 7f5a6f7fe700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 0x7f5a5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.417 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f5a60032780 con 0x7f5a780b1b40 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.417 7f5a7605d700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 0x7f5a5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:35.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.418 7f5a7605d700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 0x7f5a5c01abc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:35.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.448 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a780920f0 con 0x7f5a780b1b40 2021-02-10T11:22:35.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.451 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5a60006230 con 0x7f5a780b1b40 2021-02-10T11:22:35.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.578 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a780920f0 con 0x7f5a780b1b40 2021-02-10T11:22:35.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.578 7f5a6f7fe700 1 -- 172.21.2.117:0/478043565 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1415 (secure 0 0 0) 0x7f5a60006000 con 0x7f5a780b1b40 2021-02-10T11:22:35.576 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:35.577 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:34.927666","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:22:35.577 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 3 2021-02-10T11:22:35.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 msgr2=0x7f5a5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 0x7f5a5c01abc0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 msgr2=0x7f5a78091ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:35.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5a6000b5b0 tx=0x7f5a6000b950).stop 2021-02-10T11:22:35.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 shutdown_connections 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5a781c1f90 0x7f5a780ba7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5a781c1b90 0x7f5a780ba1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5a5c018730 0x7f5a5c01abc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 --2- 172.21.2.117:0/478043565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5a780b1b40 0x7f5a78091ad0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 >> 172.21.2.117:0/478043565 conn(0x7f5a780c3b40 msgr2=0x7f5a7808e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 shutdown_connections 2021-02-10T11:22:35.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.581 7f5a7cc5c700 1 -- 172.21.2.117:0/478043565 wait complete. 2021-02-10T11:22:35.586 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 Processor -- start 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 -- start start 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f880d0ac0 0x7f7f881b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7f881b3470 0x7f7f881b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f88109ab0 con 0x7f7f881085f0 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f880fde80 con 0x7f7f881b3470 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8f599700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880d0ec0 con 0x7f7f880d0ac0 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8dbb6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7f881085f0 0x7f7f881089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41568/0 (socket says 172.21.2.117:41568) 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:35.999 7f7f8dbb6700 1 -- 172.21.2.117:0/3028095036 learned_addr learned my addr 172.21.2.117:0/3028095036 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:36.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113118704 0 0) 0x7f7f80001500 con 0x7f7f881085f0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74002fe0 con 0x7f7f881085f0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4071208086 0 0) 0x7f7f80001460 con 0x7f7f881085f0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f8cbb4700 1 --2- 172.21.2.117:0/3028095036 >> v2:172.21.2.125:3300/0 conn(0x7f7f881b3470 0x7f7f881b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f8d3b5700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f880d0ac0 0x7f7f881b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f880d0ac0 msgr2=0x7f7f881b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f880d0ac0 0x7f7f881b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 >> v2:172.21.2.125:3300/0 conn(0x7f7f881b3470 msgr2=0x7f7f881b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 --2- 172.21.2.117:0/3028095036 >> v2:172.21.2.125:3300/0 conn(0x7f7f881b3470 0x7f7f881b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f880e51a0 con 0x7f7f881085f0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7f80001980 con 0x7f7f881085f0 2021-02-10T11:22:36.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 >> v1:172.21.2.117:6789/0 conn(0x7f7f881085f0 legacy=0x7f7f881089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f74003b90 0x7f7f74006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f740068a0 0x7f7f74008da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f74009640 0x7f7f7400bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7400c410 con 0x7f7f74009640 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7400c630 con 0x7f7f740068a0 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7400c850 con 0x7f7f74003b90 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f8d3b5700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f880d0ac0 0x7f7f881b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:36.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f8dbb6700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f74009640 0x7f7f7400bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.000 7f7f8cbb4700 1 --2- 172.21.2.117:0/3028095036 >> v2:172.21.2.125:3300/0 conn(0x7f7f881b3470 0x7f7f881b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 -- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f74003b90 msgr2=0x7f7f74006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f74003b90 0x7f7f74006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 -- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f740068a0 msgr2=0x7f7f74008da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f740068a0 0x7f7f74008da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 -- 172.21.2.117:0/3028095036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7400caf0 con 0x7f7f74009640 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f74009640 0x7f7f7400bb60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7f80001ae0 tx=0x7f7f8000a310).ready entity=mon.0 client_cookie=c94b15071b7a07fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f800070c0 con 0x7f7f74009640 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80007400 con 0x7f7f74009640 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f7bfff700 1 -- 172.21.2.117:0/3028095036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f800049a0 con 0x7f7f74009640 2021-02-10T11:22:36.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 -- 172.21.2.117:0/3028095036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f74009640 msgr2=0x7f7f7400bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 --2- 172.21.2.117:0/3028095036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f74009640 0x7f7f7400bb60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7f80001ae0 tx=0x7f7f8000a310).stop 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8dbb6700 1 -- 172.21.2.117:0/3028095036 reap_dead start 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 -- 172.21.2.117:0/3028095036 shutdown_connections 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 -- 172.21.2.117:0/3028095036 >> 172.21.2.117:0/3028095036 conn(0x7f7f880ce410 msgr2=0x7f7f880d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 -- 172.21.2.117:0/3028095036 shutdown_connections 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.001 7f7f8f599700 1 -- 172.21.2.117:0/3028095036 wait complete. 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 Processor -- start 2021-02-10T11:22:36.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 -- start start 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f880905d0 0x7f7f88091060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f88091680 0x7f7f88091a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880f27c0 con 0x7f7f880f1b50 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88090ae0 con 0x7f7f880905d0 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.016 7f7f8f599700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f880f1f50 con 0x7f7f88091680 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41330/0 (socket says 172.21.2.117:41330) 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 -- 172.21.2.117:0/4208454935 learned_addr learned my addr 172.21.2.117:0/4208454935 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 -- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f88091680 msgr2=0x7f7f88091a80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:36.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f88091680 0x7f7f88091a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 -- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f880905d0 msgr2=0x7f7f88091060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f880905d0 0x7f7f88091060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 -- 172.21.2.117:0/4208454935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8818fa80 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8d3b5700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f84050530 tx=0x7f7f840508d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f84050cc0 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f840009e0 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f840052e0 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8818fa80 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.017 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f880920a0 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.018 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f84006030 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.018 7f7f79ffb700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 0x7f7f6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.018 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f7f840765b0 con 0x7f7f880f1b50 2021-02-10T11:22:36.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.018 7f7f8cbb4700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 0x7f7f6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.019 7f7f8cbb4700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 0x7f7f6801abc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.049 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f88109ab0 con 0x7f7f880f1b50 2021-02-10T11:22:36.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.052 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f84006bf0 con 0x7f7f880f1b50 2021-02-10T11:22:36.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.174 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f88109ab0 con 0x7f7f880f1b50 2021-02-10T11:22:36.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.175 7f7f79ffb700 1 -- 172.21.2.117:0/4208454935 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f7f84006030 con 0x7f7f880f1b50 2021-02-10T11:22:36.173 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:36.173 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.176 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 msgr2=0x7f7f6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.176 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 0x7f7f6801abc0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 msgr2=0x7f7f8818f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7f84050530 tx=0x7f7f840508d0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 shutdown_connections 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f88091680 0x7f7f88091a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f880905d0 0x7f7f88091060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f680186c0 0x7f7f6801abc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 --2- 172.21.2.117:0/4208454935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f880f1b50 0x7f7f8818f460 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 >> 172.21.2.117:0/4208454935 conn(0x7f7f880b5570 msgr2=0x7f7f880c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 shutdown_connections 2021-02-10T11:22:36.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.177 7f7f8f599700 1 -- 172.21.2.117:0/4208454935 wait complete. 2021-02-10T11:22:36.183 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-02-10T11:22:36.183 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:22:36.184 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:22:36.186 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:22:36.186 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:22:36.186 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:22:36.190 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:22:36.190 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:36.594 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:22:36.594 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:22:36.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.599 7f49a578f700 1 Processor -- start 2021-02-10T11:22:36.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.599 7f49a578f700 1 -- start start 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49a0109600 0x7f49a01b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a0109ab0 con 0x7f49a00d0ac0 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a01b3c60 con 0x7f49a0109600 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a00d0ec0 con 0x7f49a00e4190 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49a0109600 0x7f49a01b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499fd75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49a0109600 0x7f49a01b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41348/0 (socket says 172.21.2.117:41348) 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499f574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f49a00d0ac0 0x7f49a01b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41596/0 (socket says 172.21.2.117:41596) 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499fd75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:45734/0 (socket says 172.21.2.117:45734) 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 -- 172.21.2.117:0/4110284325 learned_addr learned my addr 172.21.2.117:0/4110284325 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:36.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 -- 172.21.2.117:0/4110284325 >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 msgr2=0x7f49a00e4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 --2- 172.21.2.117:0/4110284325 >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 -- 172.21.2.117:0/4110284325 >> v1:172.21.2.117:6789/0 conn(0x7f49a00d0ac0 legacy=0x7f49a01b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 -- 172.21.2.117:0/4110284325 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a010b040 con 0x7f49a0109600 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499ed73700 1 -- 172.21.2.117:0/4110284325 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323743504 0 0) 0x7f4988001100 con 0x7f49a00d0ac0 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499fd75700 1 --2- 172.21.2.117:0/4110284325 >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f4997fff700 1 --2- 172.21.2.117:0/4110284325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a0109600 0x7f49a01b3640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4990006a80 tx=0x7f4990009f00).ready entity=mon.0 client_cookie=c5be1f8c731d5f70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499ed73700 1 -- 172.21.2.117:0/4110284325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f499000b5c0 con 0x7f49a0109600 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499ed73700 1 -- 172.21.2.117:0/4110284325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499000b8c0 con 0x7f49a0109600 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f499ed73700 1 -- 172.21.2.117:0/4110284325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f499000bda0 con 0x7f49a0109600 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- 172.21.2.117:0/4110284325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a0109600 msgr2=0x7f49a01b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 --2- 172.21.2.117:0/4110284325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a0109600 0x7f49a01b3640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4990006a80 tx=0x7f4990009f00).stop 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- 172.21.2.117:0/4110284325 shutdown_connections 2021-02-10T11:22:36.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 --2- 172.21.2.117:0/4110284325 >> v2:172.21.2.125:3300/0 conn(0x7f49a00e4190 0x7f49a00e4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 --2- 172.21.2.117:0/4110284325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a0109600 0x7f49a01b3640 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- 172.21.2.117:0/4110284325 >> 172.21.2.117:0/4110284325 conn(0x7f49a00ce410 msgr2=0x7f49a00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- 172.21.2.117:0/4110284325 shutdown_connections 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.601 7f49a578f700 1 -- 172.21.2.117:0/4110284325 wait complete. 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 Processor -- start 2021-02-10T11:22:36.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 -- start start 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49a0191660 0x7f49a00ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49a0191a60 0x7f49a00ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a019e6f0 con 0x7f49a019e2f0 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00adad0 con 0x7f49a0191660 2021-02-10T11:22:36.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f49a578f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00adcf0 con 0x7f49a0191a60 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41354/0 (socket says 172.21.2.117:41354) 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 -- 172.21.2.117:0/3039940091 learned_addr learned my addr 172.21.2.117:0/3039940091 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f4997fff700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49a0191660 0x7f49a00ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 -- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49a0191a60 msgr2=0x7f49a00ad4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49a0191a60 0x7f49a00ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 -- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49a0191660 msgr2=0x7f49a00ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49a0191660 0x7f49a00ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.616 7f499f574700 1 -- 172.21.2.117:0/3039940091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0192560 con 0x7f49a019e2f0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499f574700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4988007fc0 tx=0x7f498800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4988001100 con 0x7f49a019e2f0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800ace0 con 0x7f49a019e2f0 2021-02-10T11:22:36.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f49a578f700 1 -- 172.21.2.117:0/3039940091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a0192560 con 0x7f49a019e2f0 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4988004590 con 0x7f49a019e2f0 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f49a578f700 1 -- 172.21.2.117:0/3039940091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a0192560 con 0x7f49a019e2f0 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49880045b0 con 0x7f49a019e2f0 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 0x7f498401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.617 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f49880327a0 con 0x7f49a019e2f0 2021-02-10T11:22:36.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.618 7f4997fff700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 0x7f498401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:36.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.618 7f4997fff700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 0x7f498401ac10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:36.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.648 7f49a578f700 1 -- 172.21.2.117:0/3039940091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a00ed930 con 0x7f49a019e2f0 2021-02-10T11:22:36.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.651 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49880043a0 con 0x7f49a019e2f0 2021-02-10T11:22:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.776 7f49a578f700 1 -- 172.21.2.117:0/3039940091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49a0109ab0 con 0x7f49a019e2f0 2021-02-10T11:22:36.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.777 7f499cd6f700 1 -- 172.21.2.117:0/3039940091 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f49880045b0 con 0x7f49a019e2f0 2021-02-10T11:22:36.775 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:36.776 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 msgr2=0x7f498401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 0x7f498401ac10 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 msgr2=0x7f49a0096d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4988007fc0 tx=0x7f498800a8f0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 shutdown_connections 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49a0191a60 0x7f49a00ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49a0191660 0x7f49a00ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4984018730 0x7f498401ac10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 --2- 172.21.2.117:0/3039940091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49a019e2f0 0x7f49a0096d40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:36.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 >> 172.21.2.117:0/3039940091 conn(0x7f49a008d410 msgr2=0x7f49a01b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:36.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 shutdown_connections 2021-02-10T11:22:36.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:36.779 7f49a578f700 1 -- 172.21.2.117:0/3039940091 wait complete. 2021-02-10T11:22:36.785 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:22:36.786 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:37.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.202 7fe961e6a700 1 Processor -- start 2021-02-10T11:22:37.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.202 7fe961e6a700 1 -- start start 2021-02-10T11:22:37.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe961e6a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe95c079000 0x7fe95c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe961e6a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c0d1d20 con 0x7fe95c07da20 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe961e6a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95c077a70 con 0x7fe95c1b00f0 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe961e6a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95c0e53c0 con 0x7fe95c079000 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe95c07da20 0x7fe95c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41610/0 (socket says 172.21.2.117:41610) 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95b7fe700 1 -- 172.21.2.117:0/2954184071 learned_addr learned my addr 172.21.2.117:0/2954184071 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086432240 0 0) 0x7fe9440010b0 con 0x7fe95c07da20 2021-02-10T11:22:37.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95affd700 1 --2- 172.21.2.117:0/2954184071 >> v2:172.21.2.125:3300/0 conn(0x7fe95c079000 0x7fe95c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe948002fe0 con 0x7fe95c07da20 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1425411243 0 0) 0x7fe9440010b0 con 0x7fe95c07da20 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 >> v2:172.21.2.125:3300/0 conn(0x7fe95c079000 msgr2=0x7fe95c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.203 7fe95a7fc700 1 --2- 172.21.2.117:0/2954184071 >> v2:172.21.2.125:3300/0 conn(0x7fe95c079000 0x7fe95c0e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 >> v1:172.21.2.125:6789/0 conn(0x7fe95c1b00f0 legacy=0x7fe95c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c079400 con 0x7fe95c07da20 2021-02-10T11:22:37.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe9440015b0 con 0x7fe95c07da20 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 >> v1:172.21.2.117:6789/0 conn(0x7fe95c07da20 legacy=0x7fe95c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 0x7fe948005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe948006780 0x7fe948008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe948009580 0x7fe94800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94800c2f0 con 0x7fe948003ac0 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94800c510 con 0x7fe948006780 2021-02-10T11:22:37.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94800c730 con 0x7fe948009580 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 0x7fe948005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95bfff700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe948009580 0x7fe94800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 -- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe948009580 msgr2=0x7fe94800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe948009580 0x7fe94800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 -- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe948006780 msgr2=0x7fe948008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe948006780 0x7fe948008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 -- 172.21.2.117:0/2954184071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480509a0 con 0x7fe948003ac0 2021-02-10T11:22:37.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95bfff700 1 -- 172.21.2.117:0/2954184071 reap_dead start 2021-02-10T11:22:37.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95b7fe700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 0x7fe948005f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe944007560 tx=0x7fe944007c90).ready entity=mon.0 client_cookie=401d1d586e6fcbcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe944001010 con 0x7fe948003ac0 2021-02-10T11:22:37.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94400a310 con 0x7fe948003ac0 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe95a7fc700 1 -- 172.21.2.117:0/2954184071 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94400a770 con 0x7fe948003ac0 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe961e6a700 1 -- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 msgr2=0x7fe948005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.204 7fe961e6a700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 0x7fe948005f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fe944007560 tx=0x7fe944007c90).stop 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.205 7fe961e6a700 1 -- 172.21.2.117:0/2954184071 shutdown_connections 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.205 7fe961e6a700 1 --2- 172.21.2.117:0/2954184071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe948003ac0 0x7fe948005f60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.205 7fe961e6a700 1 -- 172.21.2.117:0/2954184071 >> 172.21.2.117:0/2954184071 conn(0x7fe95c0ce450 msgr2=0x7fe95c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.205 7fe961e6a700 1 -- 172.21.2.117:0/2954184071 shutdown_connections 2021-02-10T11:22:37.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.205 7fe961e6a700 1 -- 172.21.2.117:0/2954184071 wait complete. 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 Processor -- start 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 -- start start 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe95c0a47d0 0x7fe95c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe95c0b77d0 0x7fe95c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c189970 con 0x7fe95c0a51f0 2021-02-10T11:22:37.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c190aa0 con 0x7fe95c0b77d0 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe961e6a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c0b81f0 con 0x7fe95c0a47d0 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe95affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe95affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41374/0 (socket says 172.21.2.117:41374) 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.220 7fe95affd700 1 -- 172.21.2.117:0/3177698928 learned_addr learned my addr 172.21.2.117:0/3177698928 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 -- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe95c0a47d0 msgr2=0x7fe95c0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe95c0a47d0 0x7fe95c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 -- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe95c0b77d0 msgr2=0x7fe95c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe95c0b77d0 0x7fe95c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 -- 172.21.2.117:0/3177698928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c0b8410 con 0x7fe95c0a51f0 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe95affd700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe94c00b580 tx=0x7fe94c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94c006b80 con 0x7fe95c0a51f0 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c00bd10 con 0x7fe95c0a51f0 2021-02-10T11:22:37.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe94c0022b0 con 0x7fe95c0a51f0 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95c0b8410 con 0x7fe95c0a51f0 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.221 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95c0b89a0 con 0x7fe95c0a51f0 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.222 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe94c006b00 con 0x7fe95c0a51f0 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.222 7fe953fff700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.222 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fe94c0326f0 con 0x7fe95c0a51f0 2021-02-10T11:22:37.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.222 7fe95b7fe700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.223 7fe95b7fe700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 0x7fe94001abc0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.252 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95c077a70 con 0x7fe95c0a51f0 2021-02-10T11:22:37.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.256 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe94c030dc0 con 0x7fe95c0a51f0 2021-02-10T11:22:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.382 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe95c077a70 con 0x7fe95c0a51f0 2021-02-10T11:22:37.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.383 7fe953fff700 1 -- 172.21.2.117:0/3177698928 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe94c006b00 con 0x7fe95c0a51f0 2021-02-10T11:22:37.381 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.384 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 msgr2=0x7fe94001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.384 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 0x7fe94001abc0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.384 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 msgr2=0x7fe95c0a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.384 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe94c00b580 tx=0x7fe94c00b920).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 shutdown_connections 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9400186c0 0x7fe94001abc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe95c0a51f0 0x7fe95c0a55f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe95c0b77d0 0x7fe95c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 --2- 172.21.2.117:0/3177698928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe95c0a47d0 0x7fe95c0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 >> 172.21.2.117:0/3177698928 conn(0x7fe95c0b5140 msgr2=0x7fe95c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 shutdown_connections 2021-02-10T11:22:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.385 7fe961e6a700 1 -- 172.21.2.117:0/3177698928 wait complete. 2021-02-10T11:22:37.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:22:37.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.808 7f8b19ea9700 1 Processor -- start 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.808 7f8b19ea9700 1 -- start start 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8b1407da20 0x7f8b140771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b140e4da0 0x7f8b141b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b140e51a0 0x7f8b14077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b14077a70 con 0x7f8b140e51a0 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140777d0 con 0x7f8b1407da20 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b19ea9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1407de20 con 0x7f8b140e4da0 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b140e51a0 0x7f8b14077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b140e51a0 0x7f8b14077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41380/0 (socket says 172.21.2.117:41380) 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 -- 172.21.2.117:0/253781669 learned_addr learned my addr 172.21.2.117:0/253781669 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 -- 172.21.2.117:0/253781669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b140e4da0 msgr2=0x7f8b141b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 --2- 172.21.2.117:0/253781669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b140e4da0 0x7f8b141b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 -- 172.21.2.117:0/253781669 >> v2:172.21.2.125:3300/0 conn(0x7f8b1407da20 msgr2=0x7f8b140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 --2- 172.21.2.117:0/253781669 >> v2:172.21.2.125:3300/0 conn(0x7f8b1407da20 0x7f8b140771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.809 7f8b13fff700 1 -- 172.21.2.117:0/253781669 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b141b0710 con 0x7f8b140e51a0 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b13fff700 1 --2- 172.21.2.117:0/253781669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140e51a0 0x7f8b14077ff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8b0c006f00 tx=0x7f8b0c009f10).ready entity=mon.0 client_cookie=f88df4eb7d611faa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b127fc700 1 -- 172.21.2.117:0/253781669 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b0c00b3b0 con 0x7f8b140e51a0 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b127fc700 1 -- 172.21.2.117:0/253781669 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00b630 con 0x7f8b140e51a0 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b127fc700 1 -- 172.21.2.117:0/253781669 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b0c00bb50 con 0x7f8b140e51a0 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b19ea9700 1 -- 172.21.2.117:0/253781669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140e51a0 msgr2=0x7f8b14077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.810 7f8b19ea9700 1 --2- 172.21.2.117:0/253781669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140e51a0 0x7f8b14077ff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8b0c006f00 tx=0x7f8b0c009f10).stop 2021-02-10T11:22:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 -- 172.21.2.117:0/253781669 shutdown_connections 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 --2- 172.21.2.117:0/253781669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140e51a0 0x7f8b14077ff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 --2- 172.21.2.117:0/253781669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b140e4da0 0x7f8b141b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 --2- 172.21.2.117:0/253781669 >> v2:172.21.2.125:3300/0 conn(0x7f8b1407da20 0x7f8b140771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 -- 172.21.2.117:0/253781669 >> 172.21.2.117:0/253781669 conn(0x7f8b140ce450 msgr2=0x7f8b140e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 -- 172.21.2.117:0/253781669 shutdown_connections 2021-02-10T11:22:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.811 7f8b19ea9700 1 -- 172.21.2.117:0/253781669 wait complete. 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.825 7f8b19ea9700 1 Processor -- start 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.825 7f8b19ea9700 1 -- start start 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 0x7f8b140b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b14090de0 0x7f8b140911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b14090600 con 0x7f8b140c33d0 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b14090860 con 0x7f8b140b3ff0 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b19ea9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140b3c70 con 0x7f8b14090de0 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b12ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b12ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41384/0 (socket says 172.21.2.117:41384) 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.826 7f8b12ffd700 1 -- 172.21.2.117:0/511758984 learned_addr learned my addr 172.21.2.117:0/511758984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:37.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b137fe700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 0x7f8b140b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b13fff700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b14090de0 0x7f8b140911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 -- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b14090de0 msgr2=0x7f8b140911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b14090de0 0x7f8b140911e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 -- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 msgr2=0x7f8b140b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 0x7f8b140b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 -- 172.21.2.117:0/511758984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b14091800 con 0x7f8b140c33d0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b137fe700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 0x7f8b140b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b12ffd700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8afc006f00 tx=0x7f8afc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8afc00bd30 con 0x7f8b140c33d0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc000c90 con 0x7f8b140c33d0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8afc0022a0 con 0x7f8b140c33d0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b14091800 con 0x7f8b140c33d0 2021-02-10T11:22:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.827 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b14091800 con 0x7f8b140c33d0 2021-02-10T11:22:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.828 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8afc005310 con 0x7f8b140c33d0 2021-02-10T11:22:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.828 7f8b0bfff700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 0x7f8af801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.828 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f8afc0328e0 con 0x7f8b140c33d0 2021-02-10T11:22:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.828 7f8b137fe700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 0x7f8af801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.829 7f8b137fe700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 0x7f8af801abc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:37.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.858 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b14103370 con 0x7f8b140c33d0 2021-02-10T11:22:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.861 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8afc006b90 con 0x7f8b140c33d0 2021-02-10T11:22:37.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.988 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f8b14077a70 con 0x7f8b140c33d0 2021-02-10T11:22:37.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.988 7f8b0bfff700 1 -- 172.21.2.117:0/511758984 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8afc00bd30 con 0x7f8b140c33d0 2021-02-10T11:22:37.985 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:37.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 msgr2=0x7f8af801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 0x7f8af801abc0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 msgr2=0x7f8b140b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:37.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8afc006f00 tx=0x7f8afc00b940).stop 2021-02-10T11:22:37.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 shutdown_connections 2021-02-10T11:22:37.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8af80186c0 0x7f8af801abc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b140c33d0 0x7f8b140b3650 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b14090de0 0x7f8b140911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 --2- 172.21.2.117:0/511758984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b140b3ff0 0x7f8b140b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 >> 172.21.2.117:0/511758984 conn(0x7f8b140a1420 msgr2=0x7f8b1409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 shutdown_connections 2021-02-10T11:22:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:37.990 7f8b19ea9700 1 -- 172.21.2.117:0/511758984 wait complete. 2021-02-10T11:22:37.995 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.411 7fd98de8d700 1 Processor -- start 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.411 7fd98de8d700 1 -- start start 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd98de8d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd988079000 0x7fd9880e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd98de8d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b00f0 0x7fd9881b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd98de8d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd988077a70 con 0x7fd98807da20 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd98de8d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9880e53c0 con 0x7fd988079000 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd98de8d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd988079400 con 0x7fd9881b00f0 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd986ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd988079000 0x7fd9880e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd986ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd988079000 0x7fd9880e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41392/0 (socket says 172.21.2.117:41392) 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd986ffd700 1 -- 172.21.2.117:0/1038567001 learned_addr learned my addr 172.21.2.117:0/1038567001 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd9877fe700 1 --1- 172.21.2.117:0/1038567001 >> v1:172.21.2.125:6789/0 conn(0x7fd98807da20 0x7fd9880771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34568/0 (socket says 172.21.2.117:34568) 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.412 7fd987fff700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b00f0 0x7fd9881b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd986ffd700 1 -- 172.21.2.117:0/1038567001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b00f0 msgr2=0x7fd9881b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd986ffd700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b00f0 0x7fd9881b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd986ffd700 1 -- 172.21.2.117:0/1038567001 >> v1:172.21.2.125:6789/0 conn(0x7fd98807da20 legacy=0x7fd9880771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd986ffd700 1 -- 172.21.2.117:0/1038567001 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98807de20 con 0x7fd988079000 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd986ffd700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd988079000 0x7fd9880e4da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd978006700 tx=0x7fd978009b80).ready entity=mon.0 client_cookie=e022d89f1eefb301 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd9867fc700 1 -- 172.21.2.117:0/1038567001 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd97800c0c0 con 0x7fd988079000 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd9867fc700 1 -- 172.21.2.117:0/1038567001 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97800c340 con 0x7fd988079000 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd9867fc700 1 -- 172.21.2.117:0/1038567001 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd97800c860 con 0x7fd988079000 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd98de8d700 1 -- 172.21.2.117:0/1038567001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd988079000 msgr2=0x7fd9880e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.413 7fd98de8d700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd988079000 0x7fd9880e4da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd978006700 tx=0x7fd978009b80).stop 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 -- 172.21.2.117:0/1038567001 shutdown_connections 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b00f0 0x7fd9881b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 --2- 172.21.2.117:0/1038567001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd988079000 0x7fd9880e4da0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 -- 172.21.2.117:0/1038567001 >> 172.21.2.117:0/1038567001 conn(0x7fd9880ce450 msgr2=0x7fd9880e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 -- 172.21.2.117:0/1038567001 shutdown_connections 2021-02-10T11:22:38.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.414 7fd98de8d700 1 -- 172.21.2.117:0/1038567001 wait complete. 2021-02-10T11:22:38.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.428 7fd98de8d700 1 Processor -- start 2021-02-10T11:22:38.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 -- start start 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9881b5130 0x7fd9881b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b5b50 0x7fd9880b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880b8da0 con 0x7fd9881b5f50 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880a4140 con 0x7fd9881b5130 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd98de8d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880a4320 con 0x7fd9881b5b50 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd987fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd987fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41400/0 (socket says 172.21.2.117:41400) 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.429 7fd987fff700 1 -- 172.21.2.117:0/3508360768 learned_addr learned my addr 172.21.2.117:0/3508360768 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 -- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b5b50 msgr2=0x7fd9880b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b5b50 0x7fd9880b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 -- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9881b5130 msgr2=0x7fd9881b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9881b5130 0x7fd9881b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 -- 172.21.2.117:0/3508360768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9880b62b0 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd987fff700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd98000b970 tx=0x7fd98000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd980000f90 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd980003030 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd980001400 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9880b62b0 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.430 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9880b6840 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.431 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd980006aa0 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.431 7fd97ffff700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 0x7fd96c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.431 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fd980032b10 con 0x7fd9881b5f50 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.431 7fd9877fe700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 0x7fd96c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:38.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.432 7fd9877fe700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 0x7fd96c01abc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:38.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.462 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9881b4f00 con 0x7fd9881b5f50 2021-02-10T11:22:38.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.465 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd980000f40 con 0x7fd9881b5f50 2021-02-10T11:22:38.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.591 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd988077a70 con 0x7fd9881b5f50 2021-02-10T11:22:38.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.592 7fd97ffff700 1 -- 172.21.2.117:0/3508360768 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd980006aa0 con 0x7fd9881b5f50 2021-02-10T11:22:38.589 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:38.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.593 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 msgr2=0x7fd96c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.593 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 0x7fd96c01abc0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.593 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 msgr2=0x7fd9880b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd98000b970 tx=0x7fd98000bd10).stop 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 shutdown_connections 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd96c018730 0x7fd96c01abc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9881b5f50 0x7fd9880b5c90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9881b5b50 0x7fd9880b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 --2- 172.21.2.117:0/3508360768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9881b5130 0x7fd9881b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 >> 172.21.2.117:0/3508360768 conn(0x7fd98808bbc0 msgr2=0x7fd9880c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:38.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 shutdown_connections 2021-02-10T11:22:38.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:38.594 7fd98de8d700 1 -- 172.21.2.117:0/3508360768 wait complete. 2021-02-10T11:22:38.599 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-02-10T11:22:38.599 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-02-10T11:22:38.599 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph mgr dump --format=json 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.011 7ff51da89700 1 Processor -- start 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.011 7ff51da89700 1 -- start start 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff51da89700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff5180d1520 0x7ff5180d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff51da89700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518077a70 con 0x7ff51807da20 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff51da89700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518079400 con 0x7ff518079000 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff51da89700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51807de20 con 0x7ff5180d1520 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff5177fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff51807da20 0x7ff5180771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41654/0 (socket says 172.21.2.117:41654) 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff5177fe700 1 -- 172.21.2.117:0/3217693943 learned_addr learned my addr 172.21.2.117:0/3217693943 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885535615 0 0) 0x7ff5000010b0 con 0x7ff51807da20 2021-02-10T11:22:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.012 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff504002fe0 con 0x7ff51807da20 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff517fff700 1 --2- 172.21.2.117:0/3217693943 >> v2:172.21.2.125:3300/0 conn(0x7ff5180d1520 0x7ff5180d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1536300960 0 0) 0x7ff5000010b0 con 0x7ff51807da20 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 >> v2:172.21.2.125:3300/0 conn(0x7ff5180d1520 msgr2=0x7ff5180d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 --2- 172.21.2.117:0/3217693943 >> v2:172.21.2.125:3300/0 conn(0x7ff5180d1520 0x7ff5180d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 >> v1:172.21.2.125:6789/0 conn(0x7ff518079000 legacy=0x7ff5180e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5181b00e0 con 0x7ff51807da20 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff5000015b0 con 0x7ff51807da20 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 >> v1:172.21.2.117:6789/0 conn(0x7ff51807da20 legacy=0x7ff5180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff504003b10 0x7ff504005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 0x7ff504008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff504009620 0x7ff50400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50400c390 con 0x7ff504006820 2021-02-10T11:22:39.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50400c5b0 con 0x7ff504003b10 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50400c7d0 con 0x7ff504009620 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff517fff700 1 --2- 172.21.2.117:0/3217693943 >> v2:172.21.2.125:3300/0 conn(0x7ff5180d1520 0x7ff5180d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff5177fe700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff504003b10 0x7ff504005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 0x7ff504008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff517fff700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff504009620 0x7ff50400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 -- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff504009620 msgr2=0x7ff50400bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff504009620 0x7ff50400bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 -- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff504003b10 msgr2=0x7ff504005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff504003b10 0x7ff504005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 -- 172.21.2.117:0/3217693943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50400ca70 con 0x7ff504006820 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff517fff700 1 -- 172.21.2.117:0/3217693943 reap_dead start 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.013 7ff516ffd700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 0x7ff504008d20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff508006ac0 tx=0x7ff508009af0).ready entity=mon.0 client_cookie=ba6faf958b250e24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff508007280 con 0x7ff504006820 2021-02-10T11:22:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508007600 con 0x7ff504006820 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff5167fc700 1 -- 172.21.2.117:0/3217693943 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff508007b20 con 0x7ff504006820 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 -- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 msgr2=0x7ff504008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 0x7ff504008d20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff508006ac0 tx=0x7ff508009af0).stop 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 -- 172.21.2.117:0/3217693943 shutdown_connections 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 --2- 172.21.2.117:0/3217693943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff504006820 0x7ff504008d20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 -- 172.21.2.117:0/3217693943 >> 172.21.2.117:0/3217693943 conn(0x7ff5180ce440 msgr2=0x7ff5180e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 -- 172.21.2.117:0/3217693943 shutdown_connections 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.014 7ff51da89700 1 -- 172.21.2.117:0/3217693943 wait complete. 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 Processor -- start 2021-02-10T11:22:39.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 -- start start 2021-02-10T11:22:39.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff518188930 0x7ff518189640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff518189c60 0x7ff51818a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518189140 con 0x7ff5180a3ee0 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518188d70 con 0x7ff518189c60 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.029 7ff51da89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5181aef50 con 0x7ff518188930 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41416/0 (socket says 172.21.2.117:41416) 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 -- 172.21.2.117:0/30733149 learned_addr learned my addr 172.21.2.117:0/30733149 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 -- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff518188930 msgr2=0x7ff518189640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff518188930 0x7ff518189640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 -- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff518189c60 msgr2=0x7ff51818a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff518189c60 0x7ff51818a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 -- 172.21.2.117:0/30733149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518199a50 con 0x7ff5180a3ee0 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff5177fe700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff500007650 tx=0x7ff50000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:39.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff50000acf0 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff500004350 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff500004870 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff51da89700 1 -- 172.21.2.117:0/30733149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff518199a50 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.030 7ff51da89700 1 -- 172.21.2.117:0/30733149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51818a970 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.031 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff500004530 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.031 7ff50ffff700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 0x7ff4fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.031 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7ff500032520 con 0x7ff5180a3ee0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.031 7ff516ffd700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 0x7ff4fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:39.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.032 7ff516ffd700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 0x7ff4fc01abc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:39.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.062 7ff51da89700 1 -- 172.21.2.117:0/30733149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5181aef30 con 0x7ff5180a3ee0 2021-02-10T11:22:39.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.064 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5000010b0 con 0x7ff5180a3ee0 2021-02-10T11:22:39.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.225 7ff51da89700 1 -- 172.21.2.117:0/30733149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff5181aef30 con 0x7ff5180a3ee0 2021-02-10T11:22:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.226 7ff50ffff700 1 -- 172.21.2.117:0/30733149 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+70186 (secure 0 0 0) 0x7ff500030cf0 con 0x7ff5180a3ee0 2021-02-10T11:22:39.227 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:39.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 msgr2=0x7ff4fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 0x7ff4fc01abc0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 msgr2=0x7ff5181ae7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:39.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7ff500007650 tx=0x7ff50000a900).stop 2021-02-10T11:22:39.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 shutdown_connections 2021-02-10T11:22:39.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff518189c60 0x7ff51818a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:39.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff518188930 0x7ff518189640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:39.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff4fc0186c0 0x7ff4fc01abc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 --2- 172.21.2.117:0/30733149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5180a3ee0 0x7ff5181ae7f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:39.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 >> 172.21.2.117:0/30733149 conn(0x7ff51819f510 msgr2=0x7ff51819a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:39.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 shutdown_connections 2021-02-10T11:22:39.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:39.230 7ff51da89700 1 -- 172.21.2.117:0/30733149 wait complete. 2021-02-10T11:22:39.235 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":3,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6800","nonce":34993},{"type":"v1","addr":"172.21.2.125:6801","nonce":34993}]},"active_addr":"172.21.2.125:6801/34993","active_change":"2021-02-10 11:21:48.805950","available":true,"standbys":[{"gid":4107,"name":"y","available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_url":{"name":"server_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_iterations":{"name":"upmap_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimization iterations","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{}},{"name":"iostat","can_run":true,"error_string":"","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout_min":{"name":"inventory_cache_timeout_min","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"]}} 2021-02-10T11:22:39.238 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-02-10T11:22:40.046 DEBUG:teuthology.orchestra.run.gibba017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.448 7fce666d4700 1 Processor -- start 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.448 7fce666d4700 1 -- start start 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce666d4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60079020 0x7fce600e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce666d4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600d1d20 con 0x7fce6007da40 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce666d4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce60077a90 con 0x7fce601b00f0 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce666d4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce600e5430 con 0x7fce60079020 2021-02-10T11:22:40.463 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce5ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fce6007da40 0x7fce600771d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41674/0 (socket says 172.21.2.117:41674) 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce5ffff700 1 -- 172.21.2.117:0/2203400605 learned_addr learned my addr 172.21.2.117:0/2203400605 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660940694 0 0) 0x7fce500010b0 con 0x7fce6007da40 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.449 7fce5effd700 1 -- 172.21.2.117:0/2203400605 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce48002fe0 con 0x7fce6007da40 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5f7fe700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60079020 0x7fce600e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2721734657 0 0) 0x7fce500010b0 con 0x7fce6007da40 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60079020 msgr2=0x7fce600e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60079020 0x7fce600e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 >> v1:172.21.2.125:6789/0 conn(0x7fce601b00f0 legacy=0x7fce600d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce60079420 con 0x7fce6007da40 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fce500015b0 con 0x7fce6007da40 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 >> v1:172.21.2.117:6789/0 conn(0x7fce6007da40 legacy=0x7fce600771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.464 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce48003b10 0x7fce48006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce480067a0 0x7fce48008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce48009510 0x7fce4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4800c2c0 con 0x7fce48009510 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4800c4e0 con 0x7fce48003b10 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5effd700 1 -- 172.21.2.117:0/2203400605 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4800c700 con 0x7fce480067a0 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce64cf1700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce48009510 0x7fce4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5f7fe700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce480067a0 0x7fce48008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.450 7fce5ffff700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce48003b10 0x7fce48006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.465 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 -- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce480067a0 msgr2=0x7fce48008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce480067a0 0x7fce48008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 -- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce48003b10 msgr2=0x7fce48006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce48003b10 0x7fce48006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 -- 172.21.2.117:0/2203400605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48050970 con 0x7fce48009510 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce5f7fe700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce480067a0 0x7fce48008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce48009510 0x7fce4800ba10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fce58006e40 tx=0x7fce58009e30).ready entity=mon.0 client_cookie=7cf6b08d5ba939d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce5ffff700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce48003b10 0x7fce48006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce5800b300 con 0x7fce48009510 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5800b640 con 0x7fce48009510 2021-02-10T11:22:40.466 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce5effd700 1 -- 172.21.2.117:0/2203400605 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce5800bb20 con 0x7fce48009510 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 -- 172.21.2.117:0/2203400605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce48009510 msgr2=0x7fce4800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 --2- 172.21.2.117:0/2203400605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce48009510 0x7fce4800ba10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fce58006e40 tx=0x7fce58009e30).stop 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce64cf1700 1 -- 172.21.2.117:0/2203400605 reap_dead start 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 -- 172.21.2.117:0/2203400605 shutdown_connections 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 -- 172.21.2.117:0/2203400605 >> 172.21.2.117:0/2203400605 conn(0x7fce600ce450 msgr2=0x7fce600e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 -- 172.21.2.117:0/2203400605 shutdown_connections 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.451 7fce666d4700 1 -- 172.21.2.117:0/2203400605 wait complete. 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 Processor -- start 2021-02-10T11:22:40.467 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 -- start start 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60097570 0x7fce601ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce600b78e0 0x7fce600983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60097e30 con 0x7fce601aed30 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce601af130 con 0x7fce600b78e0 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.466 7fce666d4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce60097970 con 0x7fce60097570 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.469 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41434/0 (socket says 172.21.2.117:41434) 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 -- 172.21.2.117:0/3718746565 learned_addr learned my addr 172.21.2.117:0/3718746565 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 -- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60097570 msgr2=0x7fce601ae710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60097570 0x7fce601ae710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 -- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce600b78e0 msgr2=0x7fce600983b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce600b78e0 0x7fce600983b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 -- 172.21.2.117:0/3718746565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600b7ce0 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5f7fe700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fce54006f40 tx=0x7fce5400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce540067f0 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400be10 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce666d4700 1 -- 172.21.2.117:0/3718746565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce600b7ce0 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce666d4700 1 -- 172.21.2.117:0/3718746565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce600989d0 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.467 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce54002030 con 0x7fce601aed30 2021-02-10T11:22:40.470 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.468 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce54000c90 con 0x7fce601aed30 2021-02-10T11:22:40.471 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.468 7fce5cff9700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 0x7fce4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:40.471 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.468 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fce54032940 con 0x7fce601aed30 2021-02-10T11:22:40.471 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.469 7fce5ffff700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 0x7fce4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:40.471 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.469 7fce5ffff700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 0x7fce4001abc0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:40.496 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.499 7fce666d4700 1 -- 172.21.2.117:0/3718746565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce60103240 con 0x7fce601aed30 2021-02-10T11:22:40.498 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.502 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fce54000940 con 0x7fce601aed30 2021-02-10T11:22:40.623 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.626 7fce666d4700 1 -- 172.21.2.117:0/3718746565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce60077a90 con 0x7fce601aed30 2021-02-10T11:22:40.624 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.627 7fce5cff9700 1 -- 172.21.2.117:0/3718746565 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7fce54000c90 con 0x7fce601aed30 2021-02-10T11:22:40.624 INFO:teuthology.misc.health.gibba017.stdout: 2021-02-10T11:22:40.624 INFO:teuthology.misc.health.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:40.625 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 msgr2=0x7fce4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.625 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 0x7fce4001abc0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 msgr2=0x7fce600b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fce54006f40 tx=0x7fce5400b990).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 shutdown_connections 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce400186c0 0x7fce4001abc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce601aed30 0x7fce600b72c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce600b78e0 0x7fce600983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 --2- 172.21.2.117:0/3718746565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce60097570 0x7fce601ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 >> 172.21.2.117:0/3718746565 conn(0x7fce6018cb10 msgr2=0x7fce601af500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:40.626 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 shutdown_connections 2021-02-10T11:22:40.627 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:40.629 7fce666d4700 1 -- 172.21.2.117:0/3718746565 wait complete. 2021-02-10T11:22:40.634 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-02-10T11:22:40.634 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:41.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.047 7fc49985a700 1 Processor -- start 2021-02-10T11:22:41.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.047 7fc49985a700 1 -- start start 2021-02-10T11:22:41.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc49985a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 0x7fc4940e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc49985a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4940d1d10 con 0x7fc4941b00e0 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc49985a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc494077a70 con 0x7fc49407da20 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc49985a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4940e5420 con 0x7fc494079000 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc493d75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc4941b00e0 0x7fc4940d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41696/0 (socket says 172.21.2.117:41696) 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc493d75700 1 -- 172.21.2.117:0/1763461255 learned_addr learned my addr 172.21.2.117:0/1763461255 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492d73700 1 --2- 172.21.2.117:0/1763461255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 0x7fc4940e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492572700 1 -- 172.21.2.117:0/1763461255 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497501166 0 0) 0x7fc48c000f90 con 0x7fc4941b00e0 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492572700 1 -- 172.21.2.117:0/1763461255 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc47c002fe0 con 0x7fc4941b00e0 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492d73700 1 -- 172.21.2.117:0/1763461255 >> v1:172.21.2.125:6789/0 conn(0x7fc49407da20 legacy=0x7fc4940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492d73700 1 -- 172.21.2.117:0/1763461255 >> v1:172.21.2.117:6789/0 conn(0x7fc4941b00e0 legacy=0x7fc4940d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492d73700 1 -- 172.21.2.117:0/1763461255 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc494079400 con 0x7fc494079000 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.048 7fc492d73700 1 --2- 172.21.2.117:0/1763461255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 0x7fc4940e4d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc488006650 tx=0x7fc488009a80).ready entity=mon.2 client_cookie=e851e9f292687809 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc492572700 1 -- 172.21.2.117:0/1763461255 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4880009b0 con 0x7fc494079000 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc492572700 1 -- 172.21.2.117:0/1763461255 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48800c0c0 con 0x7fc494079000 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc492572700 1 -- 172.21.2.117:0/1763461255 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48800c5e0 con 0x7fc494079000 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 -- 172.21.2.117:0/1763461255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 msgr2=0x7fc4940e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 --2- 172.21.2.117:0/1763461255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 0x7fc4940e4d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc488006650 tx=0x7fc488009a80).stop 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 -- 172.21.2.117:0/1763461255 shutdown_connections 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 --2- 172.21.2.117:0/1763461255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc494079000 0x7fc4940e4d90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 -- 172.21.2.117:0/1763461255 >> 172.21.2.117:0/1763461255 conn(0x7fc4940ce440 msgr2=0x7fc4940e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 -- 172.21.2.117:0/1763461255 shutdown_connections 2021-02-10T11:22:41.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.049 7fc49985a700 1 -- 172.21.2.117:0/1763461255 wait complete. 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 Processor -- start 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 -- start start 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49419e290 0x7fc4941af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 0x7fc4940b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4940932a0 con 0x7fc494092880 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc494195170 con 0x7fc4940b00b0 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.064 7fc49985a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc494195390 con 0x7fc49419e290 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41452/0 (socket says 172.21.2.117:41452) 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 -- 172.21.2.117:0/2687623043 learned_addr learned my addr 172.21.2.117:0/2687623043 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493574700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49419e290 0x7fc4941af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 -- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49419e290 msgr2=0x7fc4941af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49419e290 0x7fc4941af620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 -- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 msgr2=0x7fc4940b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc492d73700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 0x7fc4940b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 0x7fc4940b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 -- 172.21.2.117:0/2687623043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4940912c0 con 0x7fc494092880 2021-02-10T11:22:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc492d73700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 0x7fc4940b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.065 7fc493d75700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc48c00b8c0 tx=0x7fc48c00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48c00bda0 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c0060c0 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc49985a700 1 -- 172.21.2.117:0/2687623043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4940912c0 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48c0064d0 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc49985a700 1 -- 172.21.2.117:0/2687623043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc494091850 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc48c0062a0 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 0x7fc47401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.066 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fc48c032a70 con 0x7fc494092880 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.067 7fc493574700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 0x7fc47401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.067 7fc493574700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 0x7fc47401abc0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.098 7fc49985a700 1 -- 172.21.2.117:0/2687623043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc494077a70 con 0x7fc494092880 2021-02-10T11:22:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.100 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc48c000f70 con 0x7fc494092880 2021-02-10T11:22:41.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.226 7fc49985a700 1 -- 172.21.2.117:0/2687623043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc494077a70 con 0x7fc494092880 2021-02-10T11:22:41.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.227 7fc47bfff700 1 -- 172.21.2.117:0/2687623043 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7fc48c0062a0 con 0x7fc494092880 2021-02-10T11:22:41.224 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:41.224 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:41.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.228 7fc49985a700 1 -- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 msgr2=0x7fc47401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.228 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 0x7fc47401abc0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.228 7fc49985a700 1 -- 172.21.2.117:0/2687623043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 msgr2=0x7fc494092c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.228 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc48c00b8c0 tx=0x7fc48c00bc60).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 -- 172.21.2.117:0/2687623043 shutdown_connections 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4740186c0 0x7fc47401abc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc494092880 0x7fc494092c80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4940b00b0 0x7fc4940b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 --2- 172.21.2.117:0/2687623043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49419e290 0x7fc4941af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 -- 172.21.2.117:0/2687623043 >> 172.21.2.117:0/2687623043 conn(0x7fc4941a67a0 msgr2=0x7fc49418bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 -- 172.21.2.117:0/2687623043 shutdown_connections 2021-02-10T11:22:41.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.229 7fc49985a700 1 -- 172.21.2.117:0/2687623043 wait complete. 2021-02-10T11:22:41.234 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 Processor -- start 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 -- start start 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f6407da20 0x7f5f640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f640e4d80 0x7f5f641b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64077a70 con 0x7f5f640e5180 2021-02-10T11:22:41.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640e5c00 con 0x7f5f640e4d80 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.646 7f5f6b50f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640777d0 con 0x7f5f6407da20 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f640e4d80 0x7f5f641b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f69b2c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5f640e5180 0x7f5f64077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41710/0 (socket says 172.21.2.117:41710) 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f640e4d80 0x7f5f641b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41458/0 (socket says 172.21.2.117:41458) 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f69b2c700 1 -- 172.21.2.117:0/1646974814 learned_addr learned my addr 172.21.2.117:0/1646974814 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 -- 172.21.2.117:0/1646974814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f6407da20 msgr2=0x7f5f640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 --2- 172.21.2.117:0/1646974814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f6407da20 0x7f5f640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 -- 172.21.2.117:0/1646974814 >> v1:172.21.2.117:6789/0 conn(0x7f5f640e5180 legacy=0x7f5f64077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f5bfff700 1 -- 172.21.2.117:0/1646974814 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828409197 0 0) 0x7f5f5c000f90 con 0x7f5f640e5180 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 -- 172.21.2.117:0/1646974814 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6407de20 con 0x7f5f640e4d80 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.647 7f5f68b2a700 1 --2- 172.21.2.117:0/1646974814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f640e4d80 0x7f5f641b00d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5f54006650 tx=0x7f5f54009a00).ready entity=mon.0 client_cookie=8d07cda74280437a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f5bfff700 1 -- 172.21.2.117:0/1646974814 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f540009b0 con 0x7f5f640e4d80 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f5bfff700 1 -- 172.21.2.117:0/1646974814 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5400c0c0 con 0x7f5f640e4d80 2021-02-10T11:22:41.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f5bfff700 1 -- 172.21.2.117:0/1646974814 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f5400c5a0 con 0x7f5f640e4d80 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 -- 172.21.2.117:0/1646974814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f640e4d80 msgr2=0x7f5f641b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 --2- 172.21.2.117:0/1646974814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f640e4d80 0x7f5f641b00d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5f54006650 tx=0x7f5f54009a00).stop 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 -- 172.21.2.117:0/1646974814 shutdown_connections 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 --2- 172.21.2.117:0/1646974814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f640e4d80 0x7f5f641b00d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 --2- 172.21.2.117:0/1646974814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f6407da20 0x7f5f640771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 -- 172.21.2.117:0/1646974814 >> 172.21.2.117:0/1646974814 conn(0x7f5f640ce430 msgr2=0x7f5f640e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 -- 172.21.2.117:0/1646974814 shutdown_connections 2021-02-10T11:22:41.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.648 7f5f6b50f700 1 -- 172.21.2.117:0/1646974814 wait complete. 2021-02-10T11:22:41.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 Processor -- start 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 -- start start 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 0x7f5f641b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f641b2f00 0x7f5f641b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f641b61e0 0x7f5f641b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f641b3440 con 0x7f5f641b51a0 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f641b71d0 con 0x7f5f641b2f00 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6b50f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f641b73f0 con 0x7f5f641b61e0 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6932b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 0x7f5f641b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6932b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 0x7f5f641b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41464/0 (socket says 172.21.2.117:41464) 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.663 7f5f6932b700 1 -- 172.21.2.117:0/957655281 learned_addr learned my addr 172.21.2.117:0/957655281 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:41.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 -- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f641b61e0 msgr2=0x7f5f641b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f641b61e0 0x7f5f641b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 -- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f641b2f00 msgr2=0x7f5f641b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f641b2f00 0x7f5f641b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 -- 172.21.2.117:0/957655281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f641b7610 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6932b700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 0x7f5f641b55a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5f60006f00 tx=0x7f5f6000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f60006350 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f60006b90 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f60000c90 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f641b7610 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.664 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641b7610 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.665 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f60007000 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.665 7f5f59ffb700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f44018700 0x7f5f4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.665 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f5f60032750 con 0x7f5f641b51a0 2021-02-10T11:22:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.665 7f5f68b2a700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f44018700 0x7f5f4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.666 7f5f68b2a700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f44018700 0x7f5f4401abc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.696 7f5f6b50f700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7f5f64091420 0x7f5f6418ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.696 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 --> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f641b7e90 con 0x7f5f64091420 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.696 7f5f69b2c700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7f5f64091420 0x7f5f6418ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.697 7f5f69b2c700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7f5f64091420 0x7f5f6418ac60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.697 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== osd.0 v2:172.21.2.117:6805/36018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f5f5c000eb0 con 0x7f5f64091420 2021-02-10T11:22:41.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.702 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 --> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f641b7e90 con 0x7f5f64091420 2021-02-10T11:22:41.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.702 7f5f59ffb700 1 -- 172.21.2.117:0/957655281 <== osd.0 v2:172.21.2.117:6805/36018 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5f5c007dd0 con 0x7f5f64091420 2021-02-10T11:22:41.700 INFO:teuthology.orchestra.run.gibba017.stdout:55834574856 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7f5f64091420 msgr2=0x7f5f6418ac60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7f5f64091420 0x7f5f6418ac60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f44018700 msgr2=0x7f5f4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f44018700 0x7f5f4401abc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 msgr2=0x7f5f641b55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.703 7f5f6b50f700 1 --2- 172.21.2.117:0/957655281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f641b51a0 0x7f5f641b55a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5f60006f00 tx=0x7f5f6000b940).stop 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.704 7f5f69b2c700 1 -- 172.21.2.117:0/957655281 reap_dead start 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.704 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 shutdown_connections 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.704 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 >> 172.21.2.117:0/957655281 conn(0x7f5f6419c2a0 msgr2=0x7f5f64103580 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.704 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 shutdown_connections 2021-02-10T11:22:41.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:41.704 7f5f6b50f700 1 -- 172.21.2.117:0/957655281 wait complete. 2021-02-10T11:22:41.708 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.120 7fccfa2a1700 1 Processor -- start 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.120 7fccfa2a1700 1 -- start start 2021-02-10T11:22:42.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccfa2a1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf407da20 0x7fccf40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccfa2a1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fccf40e5180 0x7fccf40d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccfa2a1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf4077a70 con 0x7fccf40e4d80 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccfa2a1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf40e5c00 con 0x7fccf40e5180 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccfa2a1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40777d0 con 0x7fccf407da20 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fccf40e4d80 0x7fccf41b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41724/0 (socket says 172.21.2.117:41724) 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf37fe700 1 -- 172.21.2.117:0/939518213 learned_addr learned my addr 172.21.2.117:0/939518213 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050939063 0 0) 0x7fcce80009b0 con 0x7fccf40e4d80 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccdc002fe0 con 0x7fccf40e4d80 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf3fff700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf407da20 0x7fccf40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.121 7fccf88be700 1 --2- 172.21.2.117:0/939518213 >> v2:172.21.2.125:3300/0 conn(0x7fccf40e5180 0x7fccf40d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2028573359 0 0) 0x7fcce8001010 con 0x7fccf40e4d80 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf407da20 msgr2=0x7fccf40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf407da20 0x7fccf40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 >> v2:172.21.2.125:3300/0 conn(0x7fccf40e5180 msgr2=0x7fccf40d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 --2- 172.21.2.117:0/939518213 >> v2:172.21.2.125:3300/0 conn(0x7fccf40e5180 0x7fccf40d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf41b08c0 con 0x7fccf40e4d80 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fcce8001530 con 0x7fccf40e4d80 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 >> v1:172.21.2.117:6789/0 conn(0x7fccf40e4d80 legacy=0x7fccf41b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccdc003b70 0x7fccdc005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccdc04a7e0 0x7fccdc04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccdc04d510 0x7fccdc04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc0502e0 con 0x7fccdc003b70 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc050500 con 0x7fccdc04d510 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc050720 con 0x7fccdc04a7e0 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccdc003b70 0x7fccdc005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf88be700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccdc04d510 0x7fccdc04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf37fe700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccdc04a7e0 0x7fccdc04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 -- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccdc04a7e0 msgr2=0x7fccdc04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccdc04a7e0 0x7fccdc04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 -- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccdc04d510 msgr2=0x7fccdc04fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccdc04d510 0x7fccdc04fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf3fff700 1 -- 172.21.2.117:0/939518213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0509c0 con 0x7fccdc003b70 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf37fe700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccdc04a7e0 0x7fccdc04cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.122 7fccf88be700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccdc04d510 0x7fccdc04fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccf3fff700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccdc003b70 0x7fccdc005f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcce400c370 tx=0x7fcce400cb00).ready entity=mon.0 client_cookie=4635f9ac6a5ebffb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce40054f0 con 0x7fccdc003b70 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce4005870 con 0x7fccdc003b70 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccf2ffd700 1 -- 172.21.2.117:0/939518213 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce4005cc0 con 0x7fccdc003b70 2021-02-10T11:22:42.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 -- 172.21.2.117:0/939518213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccdc003b70 msgr2=0x7fccdc005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 --2- 172.21.2.117:0/939518213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccdc003b70 0x7fccdc005f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fcce400c370 tx=0x7fcce400cb00).stop 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccf88be700 1 -- 172.21.2.117:0/939518213 reap_dead start 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 -- 172.21.2.117:0/939518213 shutdown_connections 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 -- 172.21.2.117:0/939518213 >> 172.21.2.117:0/939518213 conn(0x7fccf40ce430 msgr2=0x7fccf40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 -- 172.21.2.117:0/939518213 shutdown_connections 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.123 7fccfa2a1700 1 -- 172.21.2.117:0/939518213 wait complete. 2021-02-10T11:22:42.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 Processor -- start 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 -- start start 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 0x7fccf40a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf40a0cd0 0x7fccf40a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf40a16f0 0x7fccf4092990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41a9850 con 0x7fccf40b8cc0 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf4092fb0 con 0x7fccf40a0cd0 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.138 7fccfa2a1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf40931d0 con 0x7fccf40a16f0 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 0x7fccf40a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 0x7fccf40a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41484/0 (socket says 172.21.2.117:41484) 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 -- 172.21.2.117:0/614956719 learned_addr learned my addr 172.21.2.117:0/614956719 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 -- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf40a16f0 msgr2=0x7fccf4092990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf40a16f0 0x7fccf4092990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 -- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf40a0cd0 msgr2=0x7fccf40a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf40a0cd0 0x7fccf40a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 -- 172.21.2.117:0/614956719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf40933f0 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf3fff700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 0x7fccf40a06b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcce4006d20 tx=0x7fcce400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce400bd80 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce4006240 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcce4005560 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf40933f0 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.139 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf4093980 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.140 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcce40054f0 con 0x7fccf40b8cc0 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.140 7fccf0ff9700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.140 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcce4032530 con 0x7fccf40b8cc0 2021-02-10T11:22:42.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.140 7fccf37fe700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.141 7fccf37fe700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.170 7fccfa2a1700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7fccf4062cd0 0x7fccf40630d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.170 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 --> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccf4093980 con 0x7fccf4062cd0 2021-02-10T11:22:42.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.171 7fccf88be700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7fccf4062cd0 0x7fccf40630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.171 7fccf88be700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7fccf4062cd0 0x7fccf40630d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.171 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== osd.1 v2:172.21.2.117:6800/36019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fccec005b20 con 0x7fccf4062cd0 2021-02-10T11:22:42.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.176 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 --> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fccf4092fb0 con 0x7fccf4062cd0 2021-02-10T11:22:42.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.177 7fccf0ff9700 1 -- 172.21.2.117:0/614956719 <== osd.1 v2:172.21.2.117:6800/36019 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fccec0066f0 con 0x7fccf4062cd0 2021-02-10T11:22:42.174 INFO:teuthology.orchestra.run.gibba017.stdout:55834574855 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7fccf4062cd0 msgr2=0x7fccf40630d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7fccf4062cd0 0x7fccf40630d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 msgr2=0x7fccd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 msgr2=0x7fccf40a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 --2- 172.21.2.117:0/614956719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf40b8cc0 0x7fccf40a06b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcce4006d20 tx=0x7fcce400b990).stop 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccf88be700 1 -- 172.21.2.117:0/614956719 reap_dead start 2021-02-10T11:22:42.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 shutdown_connections 2021-02-10T11:22:42.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 >> 172.21.2.117:0/614956719 conn(0x7fccf419f5a0 msgr2=0x7fccf41abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 shutdown_connections 2021-02-10T11:22:42.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.178 7fccfa2a1700 1 -- 172.21.2.117:0/614956719 wait complete. 2021-02-10T11:22:42.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 Processor -- start 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 -- start start 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac07da20 0x7f94ac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94ac0e4d80 0x7f94ac1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f94ac0e5180 0x7f94ac077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.593 7f94b1ec9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac0e5c00 con 0x7f94ac0e4d80 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94b1ec9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac077a70 con 0x7f94ac0e5180 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94b1ec9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0777d0 con 0x7f94ac07da20 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94ac0e4d80 0x7f94ac1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94ac0e4d80 0x7f94ac1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41494/0 (socket says 172.21.2.117:41494) 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 -- 172.21.2.117:0/1082300182 learned_addr learned my addr 172.21.2.117:0/1082300182 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 -- 172.21.2.117:0/1082300182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac07da20 msgr2=0x7f94ac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 --2- 172.21.2.117:0/1082300182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac07da20 0x7f94ac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 -- 172.21.2.117:0/1082300182 >> v2:172.21.2.125:3300/0 conn(0x7f94ac0e5180 msgr2=0x7f94ac077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 --2- 172.21.2.117:0/1082300182 >> v2:172.21.2.125:3300/0 conn(0x7f94ac0e5180 0x7f94ac077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 -- 172.21.2.117:0/1082300182 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac07de20 con 0x7f94ac0e4d80 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aaffd700 1 --2- 172.21.2.117:0/1082300182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac0e4d80 0x7f94ac1b00d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94a4006a80 tx=0x7f94a4009f00).ready entity=mon.0 client_cookie=a47e47faab58b3d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aa7fc700 1 -- 172.21.2.117:0/1082300182 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a400b3b0 con 0x7f94ac0e4d80 2021-02-10T11:22:42.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aa7fc700 1 -- 172.21.2.117:0/1082300182 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a400b6b0 con 0x7f94ac0e4d80 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94aa7fc700 1 -- 172.21.2.117:0/1082300182 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a400bbd0 con 0x7f94ac0e4d80 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94b1ec9700 1 -- 172.21.2.117:0/1082300182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac0e4d80 msgr2=0x7f94ac1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.594 7f94b1ec9700 1 --2- 172.21.2.117:0/1082300182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac0e4d80 0x7f94ac1b00d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94a4006a80 tx=0x7f94a4009f00).stop 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 -- 172.21.2.117:0/1082300182 shutdown_connections 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 --2- 172.21.2.117:0/1082300182 >> v2:172.21.2.125:3300/0 conn(0x7f94ac0e5180 0x7f94ac077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 --2- 172.21.2.117:0/1082300182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac0e4d80 0x7f94ac1b00d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 --2- 172.21.2.117:0/1082300182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac07da20 0x7f94ac0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 -- 172.21.2.117:0/1082300182 >> 172.21.2.117:0/1082300182 conn(0x7f94ac0ce430 msgr2=0x7f94ac0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 -- 172.21.2.117:0/1082300182 shutdown_connections 2021-02-10T11:22:42.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.595 7f94b1ec9700 1 -- 172.21.2.117:0/1082300182 wait complete. 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 Processor -- start 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 -- start start 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94ac097a60 0x7f94ac093c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 0x7f94ac095210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac094660 0x7f94ac095830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac094d10 con 0x7f94ac094260 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac097e60 con 0x7f94ac097a60 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94b1ec9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac098130 con 0x7f94ac094660 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 0x7f94ac095210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 0x7f94ac095210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41500/0 (socket says 172.21.2.117:41500) 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.610 7f94aaffd700 1 -- 172.21.2.117:0/4195810445 learned_addr learned my addr 172.21.2.117:0/4195810445 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:42.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 -- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac094660 msgr2=0x7f94ac095830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94ac094660 0x7f94ac095830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 -- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94ac097a60 msgr2=0x7f94ac093c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94ac097a60 0x7f94ac093c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 -- 172.21.2.117:0/4195810445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac095e50 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94aaffd700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 0x7f94ac095210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f94a4006fc0 tx=0x7f94a400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a40064c0 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a4005660 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac095e50 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac0963e0 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.611 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a4005b80 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.612 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94a4006030 con 0x7f94ac094260 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.612 7f9493fff700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f948c0186c0 0x7f948c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.612 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f94a4032a00 con 0x7f94ac094260 2021-02-10T11:22:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.613 7f94ab7fe700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f948c0186c0 0x7f948c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.613 7f94ab7fe700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f948c0186c0 0x7f948c01abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.643 7f94b1ec9700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f94ac0627c0 0x7f94ac062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:42.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.643 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 --> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94ac0963e0 con 0x7f94ac0627c0 2021-02-10T11:22:42.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.643 7f94abfff700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f94ac0627c0 0x7f94ac062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:42.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.644 7f94abfff700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f94ac0627c0 0x7f94ac062bc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:42.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.644 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== osd.2 v2:172.21.2.117:6801/36020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f949c006e50 con 0x7f94ac0627c0 2021-02-10T11:22:42.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.651 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 --> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94ac095e50 con 0x7f94ac0627c0 2021-02-10T11:22:42.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.652 7f9493fff700 1 -- 172.21.2.117:0/4195810445 <== osd.2 v2:172.21.2.117:6801/36020 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f949c006ac0 con 0x7f94ac0627c0 2021-02-10T11:22:42.649 INFO:teuthology.orchestra.run.gibba017.stdout:55834574856 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f94ac0627c0 msgr2=0x7f94ac062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f94ac0627c0 0x7f94ac062bc0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f948c0186c0 msgr2=0x7f948c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f948c0186c0 0x7f948c01abc0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 msgr2=0x7f94ac095210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 --2- 172.21.2.117:0/4195810445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94ac094260 0x7f94ac095210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f94a4006fc0 tx=0x7f94a400bd40).stop 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94abfff700 1 -- 172.21.2.117:0/4195810445 reap_dead start 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 shutdown_connections 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 >> 172.21.2.117:0/4195810445 conn(0x7f94ac1b9350 msgr2=0x7f94ac1b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 shutdown_connections 2021-02-10T11:22:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:42.653 7f94b1ec9700 1 -- 172.21.2.117:0/4195810445 wait complete. 2021-02-10T11:22:42.657 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-02-10T11:22:43.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.070 7f984a288700 1 Processor -- start 2021-02-10T11:22:43.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.070 7f984a288700 1 -- start start 2021-02-10T11:22:43.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f984a288700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f984407da20 0x7f98440771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f984a288700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98440e5180 0x7f9844079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f984a288700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9844077a70 con 0x7f98440e4d80 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f984a288700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98440777d0 con 0x7f984407da20 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f984a288700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98441b08c0 con 0x7f98440e5180 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f98437fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f98440e4d80 0x7f98441b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41760/0 (socket says 172.21.2.117:41760) 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f98437fe700 1 -- 172.21.2.117:0/1601355042 learned_addr learned my addr 172.21.2.117:0/1601355042 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 --2- 172.21.2.117:0/1601355042 >> v2:172.21.2.117:3300/0 conn(0x7f984407da20 0x7f98440771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 -- 172.21.2.117:0/1601355042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98440e5180 msgr2=0x7f9844079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 --2- 172.21.2.117:0/1601355042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98440e5180 0x7f9844079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 -- 172.21.2.117:0/1601355042 >> v1:172.21.2.117:6789/0 conn(0x7f98440e4d80 legacy=0x7f98441b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 -- 172.21.2.117:0/1601355042 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984407de20 con 0x7f984407da20 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.071 7f9843fff700 1 --2- 172.21.2.117:0/1601355042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f984407da20 0x7f98440771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9834006650 tx=0x7f9834009a80).ready entity=mon.0 client_cookie=7f50ce36a6e38bbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f9842ffd700 1 -- 172.21.2.117:0/1601355042 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98340009b0 con 0x7f984407da20 2021-02-10T11:22:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f9842ffd700 1 -- 172.21.2.117:0/1601355042 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983400c0c0 con 0x7f984407da20 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f9842ffd700 1 -- 172.21.2.117:0/1601355042 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f983400c5e0 con 0x7f984407da20 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 -- 172.21.2.117:0/1601355042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f984407da20 msgr2=0x7f98440771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 --2- 172.21.2.117:0/1601355042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f984407da20 0x7f98440771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9834006650 tx=0x7f9834009a80).stop 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 -- 172.21.2.117:0/1601355042 shutdown_connections 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 --2- 172.21.2.117:0/1601355042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98440e5180 0x7f9844079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 --2- 172.21.2.117:0/1601355042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f984407da20 0x7f98440771b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 -- 172.21.2.117:0/1601355042 >> 172.21.2.117:0/1601355042 conn(0x7f98440ce430 msgr2=0x7f98440e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 -- 172.21.2.117:0/1601355042 shutdown_connections 2021-02-10T11:22:43.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.072 7f984a288700 1 -- 172.21.2.117:0/1601355042 wait complete. 2021-02-10T11:22:43.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 Processor -- start 2021-02-10T11:22:43.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 -- start start 2021-02-10T11:22:43.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98440b4bf0 0x7f9844106340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98441aa790 0x7f9844106960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 0x7f98441973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844196e30 con 0x7f9844106f80 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9844107380 con 0x7f98440b4bf0 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f984a288700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984408ce40 con 0x7f98441aa790 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f98488a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 0x7f98441973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f98488a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 0x7f98441973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41512/0 (socket says 172.21.2.117:41512) 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.087 7f98488a5700 1 -- 172.21.2.117:0/1462488676 learned_addr learned my addr 172.21.2.117:0/1462488676 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 -- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98441aa790 msgr2=0x7f9844106960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98441aa790 0x7f9844106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 -- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98440b4bf0 msgr2=0x7f9844106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98440b4bf0 0x7f9844106340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 -- 172.21.2.117:0/1462488676 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98441979d0 con 0x7f9844106f80 2021-02-10T11:22:43.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f98488a5700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 0x7f98441973b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f983c009980 tx=0x7f983c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f983c005870 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c005af0 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f983c006030 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f984a288700 1 -- 172.21.2.117:0/1462488676 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98441979d0 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.088 7f984a288700 1 -- 172.21.2.117:0/1462488676 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98441979d0 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.089 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f983c005cd0 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.089 7f9840ff9700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98240186c0 0x7f982401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.089 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f983c032b50 con 0x7f9844106f80 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.089 7f9843fff700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98240186c0 0x7f982401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.090 7f9843fff700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98240186c0 0x7f982401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.120 7f984a288700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f98441037f0 0x7f98440627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.120 7f984a288700 1 -- 172.21.2.117:0/1462488676 --> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f984408cfb0 con 0x7f98441037f0 2021-02-10T11:22:43.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.120 7f98437fe700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f98441037f0 0x7f98440627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.121 7f98437fe700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f98441037f0 0x7f98440627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.121 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== osd.3 v2:172.21.2.117:6824/36021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f9838001590 con 0x7f98441037f0 2021-02-10T11:22:43.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.128 7f984a288700 1 -- 172.21.2.117:0/1462488676 --> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9844077a70 con 0x7f98441037f0 2021-02-10T11:22:43.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.129 7f9840ff9700 1 -- 172.21.2.117:0/1462488676 <== osd.3 v2:172.21.2.117:6824/36021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f98380009b0 con 0x7f98441037f0 2021-02-10T11:22:43.128 INFO:teuthology.orchestra.run.gibba017.stdout:55834574854 2021-02-10T11:22:43.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f98441037f0 msgr2=0x7f98440627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f98441037f0 0x7f98440627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98240186c0 msgr2=0x7f982401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98240186c0 0x7f982401abc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 msgr2=0x7f98441973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 --2- 172.21.2.117:0/1462488676 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9844106f80 0x7f98441973b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f983c009980 tx=0x7f983c00bcd0).stop 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f98488a5700 1 -- 172.21.2.117:0/1462488676 reap_dead start 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 shutdown_connections 2021-02-10T11:22:43.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 >> 172.21.2.117:0/1462488676 conn(0x7f9844107f00 msgr2=0x7f9844090030 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 shutdown_connections 2021-02-10T11:22:43.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.130 7f984a288700 1 -- 172.21.2.117:0/1462488676 wait complete. 2021-02-10T11:22:43.135 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 flush_pg_stats 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 Processor -- start 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 -- start start 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c807da20 0x7f00c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f00c80e5180 0x7f00c80d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c8077a70 con 0x7f00c80e4d80 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c80e5c00 con 0x7f00c80e5180 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ce637700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c80777d0 con 0x7f00c807da20 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00c77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f00c80e4d80 0x7f00c81b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41772/0 (socket says 172.21.2.117:41772) 2021-02-10T11:22:43.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00c77fe700 1 -- 172.21.2.117:0/1990952963 learned_addr learned my addr 172.21.2.117:0/1990952963 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.548 7f00ccc54700 1 --2- 172.21.2.117:0/1990952963 >> v2:172.21.2.117:3300/0 conn(0x7f00c80e5180 0x7f00c80d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00c6ffd700 1 -- 172.21.2.117:0/1990952963 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504761384 0 0) 0x7f00bc0010b0 con 0x7f00c80e4d80 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00c6ffd700 1 -- 172.21.2.117:0/1990952963 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b0002fe0 con 0x7f00c80e4d80 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ccc54700 1 -- 172.21.2.117:0/1990952963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c807da20 msgr2=0x7f00c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ccc54700 1 --2- 172.21.2.117:0/1990952963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c807da20 0x7f00c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ccc54700 1 -- 172.21.2.117:0/1990952963 >> v1:172.21.2.117:6789/0 conn(0x7f00c80e4d80 legacy=0x7f00c81b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ccc54700 1 -- 172.21.2.117:0/1990952963 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c81b08c0 con 0x7f00c80e5180 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ccc54700 1 --2- 172.21.2.117:0/1990952963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c80e5180 0x7f00c80d0500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f00c0006e30 tx=0x7f00c0009e60).ready entity=mon.0 client_cookie=d64fb632bd4a3010 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00c6ffd700 1 -- 172.21.2.117:0/1990952963 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c0006a50 con 0x7f00c80e5180 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00c6ffd700 1 -- 172.21.2.117:0/1990952963 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c000b300 con 0x7f00c80e5180 2021-02-10T11:22:43.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00c6ffd700 1 -- 172.21.2.117:0/1990952963 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c000b820 con 0x7f00c80e5180 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ce637700 1 -- 172.21.2.117:0/1990952963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c80e5180 msgr2=0x7f00c80d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.549 7f00ce637700 1 --2- 172.21.2.117:0/1990952963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c80e5180 0x7f00c80d0500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f00c0006e30 tx=0x7f00c0009e60).stop 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 -- 172.21.2.117:0/1990952963 shutdown_connections 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 --2- 172.21.2.117:0/1990952963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c80e5180 0x7f00c80d0500 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 --2- 172.21.2.117:0/1990952963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c807da20 0x7f00c80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 -- 172.21.2.117:0/1990952963 >> 172.21.2.117:0/1990952963 conn(0x7f00c80ce430 msgr2=0x7f00c80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 -- 172.21.2.117:0/1990952963 shutdown_connections 2021-02-10T11:22:43.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.550 7f00ce637700 1 -- 172.21.2.117:0/1990952963 wait complete. 2021-02-10T11:22:43.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.564 7f00ce637700 1 Processor -- start 2021-02-10T11:22:43.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 -- start start 2021-02-10T11:22:43.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00c80fe0b0 0x7f00c81b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c80fe4b0 0x7f00c81b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 0x7f00c81b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c8102070 con 0x7f00c81b7f10 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c8102290 con 0x7f00c80fe0b0 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ce637700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c81a75e0 con 0x7f00c80fe4b0 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 0x7f00c81b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 0x7f00c81b8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41524/0 (socket says 172.21.2.117:41524) 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 -- 172.21.2.117:0/1179267321 learned_addr learned my addr 172.21.2.117:0/1179267321 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 -- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c80fe4b0 msgr2=0x7f00c81b78f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00c80fe4b0 0x7f00c81b78f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 -- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00c80fe0b0 msgr2=0x7f00c81b89c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.565 7f00ccc54700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00c80fe0b0 0x7f00c81b89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00ccc54700 1 -- 172.21.2.117:0/1179267321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c81a7800 con 0x7f00c81b7f10 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00ccc54700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 0x7f00c81b8310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00c000b970 tx=0x7f00c000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c00093d0 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0000ff0 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00c0005a50 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00ce637700 1 -- 172.21.2.117:0/1179267321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c81a7800 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.566 7f00ce637700 1 -- 172.21.2.117:0/1179267321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c81a7800 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.567 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00c0005a30 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.567 7f00c4ff9700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00a80186c0 0x7f00a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.567 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f00c0032a80 con 0x7f00c81b7f10 2021-02-10T11:22:43.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.567 7f00c7fff700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00a80186c0 0x7f00a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.567 7f00c7fff700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00a80186c0 0x7f00a801abc0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.598 7f00ce637700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f00c81a8370 0x7f00c80627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:43.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.598 7f00ce637700 1 -- 172.21.2.117:0/1179267321 --> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00c81b4ee0 con 0x7f00c81a8370 2021-02-10T11:22:43.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.598 7f00c77fe700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f00c81a8370 0x7f00c80627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:43.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.598 7f00c77fe700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f00c81a8370 0x7f00c80627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:43.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.599 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== osd.4 v2:172.21.2.125:6803/36390 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f00bc001520 con 0x7f00c81a8370 2021-02-10T11:22:43.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.606 7f00ce637700 1 -- 172.21.2.117:0/1179267321 --> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f00c81b4ee0 con 0x7f00c81a8370 2021-02-10T11:22:43.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.606 7f00c4ff9700 1 -- 172.21.2.117:0/1179267321 <== osd.4 v2:172.21.2.125:6803/36390 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f00bc001570 con 0x7f00c81a8370 2021-02-10T11:22:43.603 INFO:teuthology.orchestra.run.gibba017.stdout:55834574855 2021-02-10T11:22:43.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 -- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f00c81a8370 msgr2=0x7f00c80627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f00c81a8370 0x7f00c80627c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 -- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00a80186c0 msgr2=0x7f00a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00a80186c0 0x7f00a801abc0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 -- 172.21.2.117:0/1179267321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 msgr2=0x7f00c81b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ce637700 1 --2- 172.21.2.117:0/1179267321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00c81b7f10 0x7f00c81b8310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f00c000b970 tx=0x7f00c000bd10).stop 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.607 7f00ccc54700 1 -- 172.21.2.117:0/1179267321 reap_dead start 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.608 7f00ce637700 1 -- 172.21.2.117:0/1179267321 shutdown_connections 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.608 7f00ce637700 1 -- 172.21.2.117:0/1179267321 >> 172.21.2.117:0/1179267321 conn(0x7f00c81a95f0 msgr2=0x7f00c8094350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.608 7f00ce637700 1 -- 172.21.2.117:0/1179267321 shutdown_connections 2021-02-10T11:22:43.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:43.608 7f00ce637700 1 -- 172.21.2.117:0/1179267321 wait complete. 2021-02-10T11:22:43.612 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 flush_pg_stats 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.022 7f34061c2700 1 Processor -- start 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.022 7f34061c2700 1 -- start start 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f34061c2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3400079000 0x7f34000e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f34061c2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001b00d0 0x7f34001b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f34061c2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3400077a70 con 0x7f340007da20 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f34061c2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34000e53a0 con 0x7f3400079000 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f34061c2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3400079400 con 0x7f34001b00d0 2021-02-10T11:22:44.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f33ff7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f340007da20 0x7f34000771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41784/0 (socket says 172.21.2.117:41784) 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f33ff7fe700 1 -- 172.21.2.117:0/11398133 learned_addr learned my addr 172.21.2.117:0/11398133 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f33fffff700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001b00d0 0x7f34001b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.023 7f33feffd700 1 --2- 172.21.2.117:0/11398133 >> v2:172.21.2.125:3300/0 conn(0x7f3400079000 0x7f34000e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33fe7fc700 1 -- 172.21.2.117:0/11398133 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137408676 0 0) 0x7f33f0000ce0 con 0x7f340007da20 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33fe7fc700 1 -- 172.21.2.117:0/11398133 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8002fe0 con 0x7f340007da20 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33feffd700 1 -- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001b00d0 msgr2=0x7f34001b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33feffd700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001b00d0 0x7f34001b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33feffd700 1 -- 172.21.2.117:0/11398133 >> v1:172.21.2.117:6789/0 conn(0x7f340007da20 legacy=0x7f34000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33feffd700 1 -- 172.21.2.117:0/11398133 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340007de20 con 0x7f3400079000 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33feffd700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3400079000 0x7f34000e4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f33f4006650 tx=0x7f33f4009a80).ready entity=mon.1 client_cookie=b7cd56b034ba2a3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33fe7fc700 1 -- 172.21.2.117:0/11398133 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f400c0c0 con 0x7f3400079000 2021-02-10T11:22:44.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33fe7fc700 1 -- 172.21.2.117:0/11398133 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f400c340 con 0x7f3400079000 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f33fe7fc700 1 -- 172.21.2.117:0/11398133 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f400c820 con 0x7f3400079000 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f34061c2700 1 -- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3400079000 msgr2=0x7f34000e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.024 7f34061c2700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3400079000 0x7f34000e4d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f33f4006650 tx=0x7f33f4009a80).stop 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 -- 172.21.2.117:0/11398133 shutdown_connections 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001b00d0 0x7f34001b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 --2- 172.21.2.117:0/11398133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3400079000 0x7f34000e4d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 -- 172.21.2.117:0/11398133 >> 172.21.2.117:0/11398133 conn(0x7f34000ce430 msgr2=0x7f34000e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 -- 172.21.2.117:0/11398133 shutdown_connections 2021-02-10T11:22:44.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.025 7f34061c2700 1 -- 172.21.2.117:0/11398133 wait complete. 2021-02-10T11:22:44.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 Processor -- start 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 -- start start 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 0x7f34000a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001a8350 0x7f34001a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34000a8650 0x7f34000a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001a8d70 con 0x7f34000b8cc0 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34000a9070 con 0x7f34000a8650 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f34061c2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34000a9250 con 0x7f34001a8350 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.040 7f33ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 0x7f34000a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 0x7f34000a9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41540/0 (socket says 172.21.2.117:41540) 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 -- 172.21.2.117:0/386246904 learned_addr learned my addr 172.21.2.117:0/386246904 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:44.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 -- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001a8350 msgr2=0x7f34001a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34001a8350 0x7f34001a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 -- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34000a8650 msgr2=0x7f34000a8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34000a8650 0x7f34000a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 -- 172.21.2.117:0/386246904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34000a9470 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.041 7f33ff7fe700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 0x7f34000a9720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f33f0001490 tx=0x7f33f000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f0000ce0 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f00043a0 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33f0004830 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f34061c2700 1 -- 172.21.2.117:0/386246904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34000a9470 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f34061c2700 1 -- 172.21.2.117:0/386246904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3400189de0 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33f0000ce0 con 0x7f34000b8cc0 2021-02-10T11:22:44.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33e00186c0 0x7f33e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.042 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f33f00326d0 con 0x7f34000b8cc0 2021-02-10T11:22:44.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.043 7f33feffd700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33e00186c0 0x7f33e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.043 7f33feffd700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33e00186c0 0x7f33e001abc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.073 7f34061c2700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7f340018a670 0x7f340018aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.073 7f34061c2700 1 -- 172.21.2.117:0/386246904 --> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3400189de0 con 0x7f340018a670 2021-02-10T11:22:44.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.074 7f33fffff700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7f340018a670 0x7f340018aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.075 7f33fffff700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7f340018a670 0x7f340018aa70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.075 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== osd.5 v2:172.21.2.125:6812/36389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f33f8007ce0 con 0x7f340018a670 2021-02-10T11:22:44.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.081 7f34061c2700 1 -- 172.21.2.117:0/386246904 --> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3400189de0 con 0x7f340018a670 2021-02-10T11:22:44.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.082 7f33e7fff700 1 -- 172.21.2.117:0/386246904 <== osd.5 v2:172.21.2.125:6812/36389 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f33f8006950 con 0x7f340018a670 2021-02-10T11:22:44.079 INFO:teuthology.orchestra.run.gibba017.stdout:55834574855 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.082 7f34061c2700 1 -- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7f340018a670 msgr2=0x7f340018aa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.082 7f34061c2700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7f340018a670 0x7f340018aa70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33e00186c0 msgr2=0x7f33e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33e00186c0 0x7f33e001abc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 msgr2=0x7f34000a9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 --2- 172.21.2.117:0/386246904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34000b8cc0 0x7f34000a9720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f33f0001490 tx=0x7f33f000a950).stop 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f33fffff700 1 -- 172.21.2.117:0/386246904 reap_dead start 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 shutdown_connections 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 >> 172.21.2.117:0/386246904 conn(0x7f3400107e80 msgr2=0x7f340019e850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 shutdown_connections 2021-02-10T11:22:44.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.083 7f34061c2700 1 -- 172.21.2.117:0/386246904 wait complete. 2021-02-10T11:22:44.087 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 flush_pg_stats 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.498 7f874f661700 1 Processor -- start 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.498 7f874f661700 1 -- start start 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.499 7f874f661700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 0x7f87480771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.499 7f874f661700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 0x7f87481b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.499 7f874f661700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8748077a70 con 0x7f87480e5180 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.499 7f874f661700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87480777d0 con 0x7f874807da20 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.499 7f874f661700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874807de20 con 0x7f87480e4d80 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874dc7e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f87480e5180 0x7f8748077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34726/0 (socket says 172.21.2.117:34726) 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874dc7e700 1 -- 172.21.2.117:0/1227017126 learned_addr learned my addr 172.21.2.117:0/1227017126 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874d47d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 0x7f87480771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874cc7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 0x7f87481b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874cc7c700 1 -- 172.21.2.117:0/1227017126 >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 msgr2=0x7f87480771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f874cc7c700 1 --2- 172.21.2.117:0/1227017126 >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 0x7f87480771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874cc7c700 1 -- 172.21.2.117:0/1227017126 >> v1:172.21.2.125:6789/0 conn(0x7f87480e5180 legacy=0x7f8748077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.500 7f873bfff700 1 -- 172.21.2.117:0/1227017126 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263366056 0 0) 0x7f8740001510 con 0x7f87480e5180 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874cc7c700 1 -- 172.21.2.117:0/1227017126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87481b06f0 con 0x7f87480e4d80 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874d47d700 1 --2- 172.21.2.117:0/1227017126 >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 0x7f87480771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874cc7c700 1 --2- 172.21.2.117:0/1227017126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 0x7f87481b00d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8744006650 tx=0x7f8744009a80).ready entity=mon.2 client_cookie=898f02077162c550 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f873bfff700 1 -- 172.21.2.117:0/1227017126 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874400c0c0 con 0x7f87480e4d80 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f873bfff700 1 -- 172.21.2.117:0/1227017126 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874400c340 con 0x7f87480e4d80 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f873bfff700 1 -- 172.21.2.117:0/1227017126 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874400c860 con 0x7f87480e4d80 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874f661700 1 -- 172.21.2.117:0/1227017126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 msgr2=0x7f87481b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.501 7f874f661700 1 --2- 172.21.2.117:0/1227017126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 0x7f87481b00d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8744006650 tx=0x7f8744009a80).stop 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 -- 172.21.2.117:0/1227017126 shutdown_connections 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 --2- 172.21.2.117:0/1227017126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87480e4d80 0x7f87481b00d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 --2- 172.21.2.117:0/1227017126 >> v2:172.21.2.125:3300/0 conn(0x7f874807da20 0x7f87480771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 -- 172.21.2.117:0/1227017126 >> 172.21.2.117:0/1227017126 conn(0x7f87480ce430 msgr2=0x7f87480e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 -- 172.21.2.117:0/1227017126 shutdown_connections 2021-02-10T11:22:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.502 7f874f661700 1 -- 172.21.2.117:0/1227017126 wait complete. 2021-02-10T11:22:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.516 7f874f661700 1 Processor -- start 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.516 7f874f661700 1 -- start start 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 0x7f87480a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f874818f3e0 0x7f87480a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f874818ff40 0x7f8748190340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480a99a0 con 0x7f87480a8860 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8748190960 con 0x7f874818ff40 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874f661700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8748190b80 con 0x7f874818f3e0 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874d47d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 0x7f87480a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874d47d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 0x7f87480a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41554/0 (socket says 172.21.2.117:41554) 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.517 7f874d47d700 1 -- 172.21.2.117:0/636560864 learned_addr learned my addr 172.21.2.117:0/636560864 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 -- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f874818f3e0 msgr2=0x7f87480a9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f874818f3e0 0x7f87480a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 -- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f874818ff40 msgr2=0x7f8748190340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f874818ff40 0x7f8748190340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 -- 172.21.2.117:0/636560864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748190da0 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874d47d700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 0x7f87480a8c60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f873c006f40 tx=0x7f873c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873c00bd80 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c000c90 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873c002220 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874f661700 1 -- 172.21.2.117:0/636560864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8748190da0 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.518 7f874f661700 1 -- 172.21.2.117:0/636560864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8748191330 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.519 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f873c00a000 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.519 7f8739ffb700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87280186c0 0x7f872801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.519 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f873c0326e0 con 0x7f87480a8860 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.519 7f874cc7c700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87280186c0 0x7f872801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.520 7f874cc7c700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87280186c0 0x7f872801abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.549 7f874f661700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7f87480627c0 0x7f8748062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.549 7f874f661700 1 -- 172.21.2.117:0/636560864 --> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8748191330 con 0x7f87480627c0 2021-02-10T11:22:44.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.550 7f874dc7e700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7f87480627c0 0x7f8748062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.551 7f874dc7e700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7f87480627c0 0x7f8748062bc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.551 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== osd.6 v2:172.21.2.125:6804/36388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f8740000f20 con 0x7f87480627c0 2021-02-10T11:22:44.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.557 7f874f661700 1 -- 172.21.2.117:0/636560864 --> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8748191330 con 0x7f87480627c0 2021-02-10T11:22:44.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.558 7f8739ffb700 1 -- 172.21.2.117:0/636560864 <== osd.6 v2:172.21.2.125:6804/36388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8740007ca0 con 0x7f87480627c0 2021-02-10T11:22:44.555 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.558 7f874f661700 1 -- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7f87480627c0 msgr2=0x7f8748062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.558 7f874f661700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7f87480627c0 0x7f8748062bc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87280186c0 msgr2=0x7f872801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87280186c0 0x7f872801abc0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 msgr2=0x7f87480a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 --2- 172.21.2.117:0/636560864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87480a8860 0x7f87480a8c60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f873c006f40 tx=0x7f873c00b990).stop 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874dc7e700 1 -- 172.21.2.117:0/636560864 reap_dead start 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 shutdown_connections 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 >> 172.21.2.117:0/636560864 conn(0x7f8748107e80 msgr2=0x7f874819e850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:44.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 shutdown_connections 2021-02-10T11:22:44.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.559 7f874f661700 1 -- 172.21.2.117:0/636560864 wait complete. 2021-02-10T11:22:44.563 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 flush_pg_stats 2021-02-10T11:22:44.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 Processor -- start 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 -- start start 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f740807da20 0x7f74080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080d1d00 con 0x7f74080e5180 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7408077a70 con 0x7f74080e4d80 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f740c868700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74080777d0 con 0x7f740807da20 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f7406e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f74080e5180 0x7f74080d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41814/0 (socket says 172.21.2.117:41814) 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.977 7f7406e1d700 1 -- 172.21.2.117:0/3838696313 learned_addr learned my addr 172.21.2.117:0/3838696313 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901680399 0 0) 0x7f7400001500 con 0x7f74080e5180 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4002fe0 con 0x7f74080e5180 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740661c700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f740807da20 0x7f74080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 862218118 0 0) 0x7f7400001460 con 0x7f74080e5180 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f740807da20 msgr2=0x7f74080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f740807da20 0x7f74080771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 >> v1:172.21.2.125:6789/0 conn(0x7f74080e4d80 legacy=0x7f74081b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74081b08c0 con 0x7f74080e5180 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7400001980 con 0x7f74080e5180 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 >> v1:172.21.2.117:6789/0 conn(0x7f74080e5180 legacy=0x7f74080d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73f4003ac0 0x7f73f4005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73f4006750 0x7f73f4008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73f4009510 0x7f73f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f400c2c0 con 0x7f73f4009510 2021-02-10T11:22:44.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f400c4e0 con 0x7f73f4006750 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740561a700 1 -- 172.21.2.117:0/3838696313 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f400c700 con 0x7f73f4003ac0 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f7406e1d700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73f4009510 0x7f73f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f7405e1b700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73f4006750 0x7f73f4008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.978 7f740661c700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73f4003ac0 0x7f73f4005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 -- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73f4003ac0 msgr2=0x7f73f4005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73f4003ac0 0x7f73f4005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 -- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73f4006750 msgr2=0x7f73f4008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73f4006750 0x7f73f4008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 -- 172.21.2.117:0/3838696313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4050970 con 0x7f73f4009510 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73f4009510 0x7f73f400ba10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f74000079d0 tx=0x7f740000a310).ready entity=mon.0 client_cookie=9ca4ef65ae30d1b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7400001460 con 0x7f73f4009510 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000070c0 con 0x7f73f4009510 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f740561a700 1 -- 172.21.2.117:0/3838696313 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7400007520 con 0x7f73f4009510 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f740c868700 1 -- 172.21.2.117:0/3838696313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73f4009510 msgr2=0x7f73f400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f740c868700 1 --2- 172.21.2.117:0/3838696313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73f4009510 0x7f73f400ba10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f74000079d0 tx=0x7f740000a310).stop 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.979 7f7406e1d700 1 -- 172.21.2.117:0/3838696313 reap_dead start 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.980 7f740c868700 1 -- 172.21.2.117:0/3838696313 shutdown_connections 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.980 7f740c868700 1 -- 172.21.2.117:0/3838696313 >> 172.21.2.117:0/3838696313 conn(0x7f74080ce430 msgr2=0x7f74080e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.980 7f740c868700 1 -- 172.21.2.117:0/3838696313 shutdown_connections 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.980 7f740c868700 1 -- 172.21.2.117:0/3838696313 wait complete. 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.994 7f740c868700 1 Processor -- start 2021-02-10T11:22:44.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.994 7f740c868700 1 -- start start 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74080b4930 0x7f74080a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 0x7f74080a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74080a2fe0 0x7f74081996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74081080f0 con 0x7f7408198c90 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7408199170 con 0x7f74080a2fe0 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740c868700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7408107c90 con 0x7f74080b4930 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 0x7f74080a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 0x7f74080a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41576/0 (socket says 172.21.2.117:41576) 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 -- 172.21.2.117:0/1749719439 learned_addr learned my addr 172.21.2.117:0/1749719439 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f740661c700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74080b4930 0x7f74080a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7406e1d700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74080a2fe0 0x7f74081996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 -- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74080b4930 msgr2=0x7f74080a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74080b4930 0x7f74080a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 -- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74080a2fe0 msgr2=0x7f74081996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74080a2fe0 0x7f74081996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.995 7f7405e1b700 1 -- 172.21.2.117:0/1749719439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408199ce0 con 0x7f7408198c90 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f7405e1b700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 0x7f74080a29c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f73f800b580 tx=0x7f73f800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f8002030 con 0x7f7408198c90 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80023b0 con 0x7f7408198c90 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f8002840 con 0x7f7408198c90 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f740c868700 1 -- 172.21.2.117:0/1749719439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7408199ce0 con 0x7f7408198c90 2021-02-10T11:22:44.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.996 7f740c868700 1 -- 172.21.2.117:0/1749719439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7408199ce0 con 0x7f7408198c90 2021-02-10T11:22:44.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.997 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73f8002590 con 0x7f7408198c90 2021-02-10T11:22:44.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.997 7f73feffd700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73ec0186c0 0x7f73ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:44.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.997 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f73f80326f0 con 0x7f7408198c90 2021-02-10T11:22:44.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.997 7f740661c700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73ec0186c0 0x7f73ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:44.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:44.997 7f740661c700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73ec0186c0 0x7f73ec01abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:45.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.027 7f740c868700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7f740819a850 0x7f7408062cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.027 7f740c868700 1 -- 172.21.2.117:0/1749719439 --> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7408077a70 con 0x7f740819a850 2021-02-10T11:22:45.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.028 7f7406e1d700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7f740819a850 0x7f7408062cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:45.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.029 7f7406e1d700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7f740819a850 0x7f7408062cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:45.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.030 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== osd.7 v2:172.21.2.125:6802/36387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f7400004720 con 0x7f740819a850 2021-02-10T11:22:45.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.035 7f740c868700 1 -- 172.21.2.117:0/1749719439 --> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f74081032a0 con 0x7f740819a850 2021-02-10T11:22:45.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.036 7f73feffd700 1 -- 172.21.2.117:0/1749719439 <== osd.7 v2:172.21.2.125:6802/36387 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7400000d00 con 0x7f740819a850 2021-02-10T11:22:45.033 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:45.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7f740819a850 msgr2=0x7f7408062cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7f740819a850 0x7f7408062cd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73ec0186c0 msgr2=0x7f73ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73ec0186c0 0x7f73ec01abc0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 msgr2=0x7f74080a29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 --2- 172.21.2.117:0/1749719439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7408198c90 0x7f74080a29c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f73f800b580 tx=0x7f73f800b920).stop 2021-02-10T11:22:45.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f7406e1d700 1 -- 172.21.2.117:0/1749719439 reap_dead start 2021-02-10T11:22:45.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 shutdown_connections 2021-02-10T11:22:45.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 >> 172.21.2.117:0/1749719439 conn(0x7f740819f5a0 msgr2=0x7f74081abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:45.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 shutdown_connections 2021-02-10T11:22:45.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.037 7f740c868700 1 -- 172.21.2.117:0/1749719439 wait complete. 2021-02-10T11:22:45.042 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-10T11:22:45.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 Processor -- start 2021-02-10T11:22:45.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 -- start start 2021-02-10T11:22:45.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 0x7f5c781b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 0x7f5c781b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c78109ab0 con 0x7f5c781085f0 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c780d0ec0 con 0x7f5c780d0ac0 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.453 7f5c8074f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780d10b0 con 0x7f5c781b3470 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7ed6c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f5c781085f0 0x7f5c781089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34760/0 (socket says 172.21.2.117:34760) 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7e56b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 0x7f5c781b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7dd6a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 0x7f5c781b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7ed6c700 1 -- 172.21.2.117:0/227736290 learned_addr learned my addr 172.21.2.117:0/227736290 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7dd6a700 1 -- 172.21.2.117:0/227736290 >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 msgr2=0x7f5c781b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7dd6a700 1 --2- 172.21.2.117:0/227736290 >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 0x7f5c781b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7dd6a700 1 -- 172.21.2.117:0/227736290 >> v1:172.21.2.125:6789/0 conn(0x7f5c781085f0 legacy=0x7f5c781089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7dd6a700 1 -- 172.21.2.117:0/227736290 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c780e51a0 con 0x7f5c781b3470 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.455 7f5c7e56b700 1 --2- 172.21.2.117:0/227736290 >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 0x7f5c781b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c7dd6a700 1 --2- 172.21.2.117:0/227736290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 0x7f5c781b3870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5c74006650 tx=0x7f5c74009a80).ready entity=mon.2 client_cookie=8157267a7be41f20 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c7d569700 1 -- 172.21.2.117:0/227736290 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c7400c0c0 con 0x7f5c781b3470 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c7d569700 1 -- 172.21.2.117:0/227736290 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400c340 con 0x7f5c781b3470 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c7d569700 1 -- 172.21.2.117:0/227736290 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c7400c860 con 0x7f5c781b3470 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c8074f700 1 -- 172.21.2.117:0/227736290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 msgr2=0x7f5c781b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.456 7f5c8074f700 1 --2- 172.21.2.117:0/227736290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 0x7f5c781b3870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5c74006650 tx=0x7f5c74009a80).stop 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 -- 172.21.2.117:0/227736290 shutdown_connections 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 --2- 172.21.2.117:0/227736290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781b3470 0x7f5c781b3870 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 --2- 172.21.2.117:0/227736290 >> v2:172.21.2.125:3300/0 conn(0x7f5c780d0ac0 0x7f5c781b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 -- 172.21.2.117:0/227736290 >> 172.21.2.117:0/227736290 conn(0x7f5c780ce410 msgr2=0x7f5c780d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:45.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 -- 172.21.2.117:0/227736290 shutdown_connections 2021-02-10T11:22:45.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.457 7f5c8074f700 1 -- 172.21.2.117:0/227736290 wait complete. 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 Processor -- start 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 -- start start 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c781a1410 0x7f5c780a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781adbd0 0x7f5c781adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781ad300 con 0x7f5c781a1010 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781a0be0 con 0x7f5c781a1410 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.471 7f5c8074f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781a0e00 con 0x7f5c781adbd0 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41588/0 (socket says 172.21.2.117:41588) 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 -- 172.21.2.117:0/579264140 learned_addr learned my addr 172.21.2.117:0/579264140 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 -- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781adbd0 msgr2=0x7f5c781adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781adbd0 0x7f5c781adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 -- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c781a1410 msgr2=0x7f5c780a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c781a1410 0x7f5c780a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 -- 172.21.2.117:0/579264140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c781a1940 con 0x7f5c781a1010 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c7e56b700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5c6c006f40 tx=0x7f5c6c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c6c00bd80 con 0x7f5c781a1010 2021-02-10T11:22:45.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c000c90 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c8074f700 1 -- 172.21.2.117:0/579264140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c781a1940 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.472 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c6c0022b0 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.473 7f5c8074f700 1 -- 172.21.2.117:0/579264140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c781a1940 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.473 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c6c006b00 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.473 7f5c6affd700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 0x7f5c6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.473 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f5c6c0325b0 con 0x7f5c781a1010 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.474 7f5c7dd6a700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 0x7f5c6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:45.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.474 7f5c7dd6a700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 0x7f5c6001abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:45.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.504 7f5c8074f700 1 -- 172.21.2.117:0/579264140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c780eda70 con 0x7f5c781a1010 2021-02-10T11:22:45.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.506 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c6c028ee0 con 0x7f5c781a1010 2021-02-10T11:22:45.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.646 7f5c8074f700 1 -- 172.21.2.117:0/579264140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5c78109ab0 con 0x7f5c781a1010 2021-02-10T11:22:45.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.646 7f5c6affd700 1 -- 172.21.2.117:0/579264140 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c6c028ee0 con 0x7f5c781a1010 2021-02-10T11:22:45.644 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:45.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.648 7f5c8074f700 1 -- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 msgr2=0x7f5c6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.648 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 0x7f5c6001abc0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.648 7f5c8074f700 1 -- 172.21.2.117:0/579264140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 msgr2=0x7f5c781acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:45.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.648 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5c6c006f40 tx=0x7f5c6c00b990).stop 2021-02-10T11:22:45.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 -- 172.21.2.117:0/579264140 shutdown_connections 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c781adbd0 0x7f5c781adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c781a1410 0x7f5c780a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c600186c0 0x7f5c6001abc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 --2- 172.21.2.117:0/579264140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c781a1010 0x7f5c781acc60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 -- 172.21.2.117:0/579264140 >> 172.21.2.117:0/579264140 conn(0x7f5c781b9e80 msgr2=0x7f5c781ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 -- 172.21.2.117:0/579264140 shutdown_connections 2021-02-10T11:22:45.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:45.649 7f5c8074f700 1 -- 172.21.2.117:0/579264140 wait complete. 2021-02-10T11:22:45.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.0 2021-02-10T11:22:45.654 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.064 7fc656a18700 1 Processor -- start 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.064 7fc656a18700 1 -- start start 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.064 7fc656a18700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc65007da20 0x7fc6500771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc656a18700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6500d1d00 con 0x7fc6500e5180 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc656a18700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc650077a70 con 0x7fc6500e4d80 2021-02-10T11:22:46.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc656a18700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6500777d0 con 0x7fc65007da20 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc655035700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc6500e5180 0x7fc6500d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41848/0 (socket says 172.21.2.117:41848) 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc655035700 1 -- 172.21.2.117:0/3224874290 learned_addr learned my addr 172.21.2.117:0/3224874290 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc654834700 1 --2- 172.21.2.117:0/3224874290 >> v2:172.21.2.125:3300/0 conn(0x7fc65007da20 0x7fc6500771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460634575 0 0) 0x7fc648001180 con 0x7fc6500e5180 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c002fe0 con 0x7fc6500e5180 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.065 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2915522214 0 0) 0x7fc648000ef0 con 0x7fc6500e5180 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 >> v2:172.21.2.125:3300/0 conn(0x7fc65007da20 msgr2=0x7fc6500771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 --2- 172.21.2.117:0/3224874290 >> v2:172.21.2.125:3300/0 conn(0x7fc65007da20 0x7fc6500771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 >> v1:172.21.2.125:6789/0 conn(0x7fc6500e4d80 legacy=0x7fc6501b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6501b08c0 con 0x7fc6500e5180 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fc6480015e0 con 0x7fc6500e5180 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 >> v1:172.21.2.117:6789/0 conn(0x7fc6500e5180 legacy=0x7fc6500d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 0x7fc63c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc63c006780 0x7fc63c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc63c0095a0 0x7fc63c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c00c2f0 con 0x7fc63c003a60 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c00c510 con 0x7fc63c0095a0 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c00c730 con 0x7fc63c006780 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 0x7fc63c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc655035700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc63c0095a0 0x7fc63c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 -- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc63c006780 msgr2=0x7fc63c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc63c006780 0x7fc63c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 -- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc63c0095a0 msgr2=0x7fc63c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc63c0095a0 0x7fc63c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 -- 172.21.2.117:0/3224874290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0509a0 con 0x7fc63c003a60 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc655035700 1 -- 172.21.2.117:0/3224874290 reap_dead start 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc654834700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 0x7fc63c005f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc63800c8f0 tx=0x7fc63800cc90).ready entity=mon.0 client_cookie=274ae9603114415a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc638000c90 con 0x7fc63c003a60 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc638009310 con 0x7fc63c003a60 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc64f7fe700 1 -- 172.21.2.117:0/3224874290 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc638009830 con 0x7fc63c003a60 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc656a18700 1 -- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 msgr2=0x7fc63c005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.066 7fc656a18700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 0x7fc63c005f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc63800c8f0 tx=0x7fc63800cc90).stop 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.067 7fc656a18700 1 -- 172.21.2.117:0/3224874290 shutdown_connections 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.067 7fc656a18700 1 --2- 172.21.2.117:0/3224874290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc63c003a60 0x7fc63c005f60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.067 7fc656a18700 1 -- 172.21.2.117:0/3224874290 >> 172.21.2.117:0/3224874290 conn(0x7fc6500ce430 msgr2=0x7fc6500e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.067 7fc656a18700 1 -- 172.21.2.117:0/3224874290 shutdown_connections 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.067 7fc656a18700 1 -- 172.21.2.117:0/3224874290 wait complete. 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.081 7fc656a18700 1 Processor -- start 2021-02-10T11:22:46.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 -- start start 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc650198c90 0x7fc6500a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 0x7fc6501996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501080f0 con 0x7fc6500b4930 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650199170 con 0x7fc6500a2fe0 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc656a18700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc650107c90 con 0x7fc650198c90 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc654834700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc654834700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41610/0 (socket says 172.21.2.117:41610) 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc654834700 1 -- 172.21.2.117:0/4056782258 learned_addr learned my addr 172.21.2.117:0/4056782258 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.082 7fc64ffff700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc650198c90 0x7fc6500a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc655035700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 0x7fc6501996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 -- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc650198c90 msgr2=0x7fc6500a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc650198c90 0x7fc6500a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 -- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 msgr2=0x7fc6501996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 0x7fc6501996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 -- 172.21.2.117:0/4056782258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650199ce0 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc655035700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 0x7fc6501996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc654834700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc638006f10 tx=0x7fc63800b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc638000c90 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63800c770 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc63800cc50 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc656a18700 1 -- 172.21.2.117:0/4056782258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc650199ce0 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.083 7fc656a18700 1 -- 172.21.2.117:0/4056782258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc650199ce0 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.084 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc638000c90 con 0x7fc6500b4930 2021-02-10T11:22:46.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.084 7fc64d7fa700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 0x7fc63401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.084 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fc6380327a0 con 0x7fc6500b4930 2021-02-10T11:22:46.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.084 7fc64ffff700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 0x7fc63401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.084 7fc64ffff700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 0x7fc63401abd0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.114 7fc656a18700 1 -- 172.21.2.117:0/4056782258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc650077a70 con 0x7fc6500b4930 2021-02-10T11:22:46.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.117 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc638000940 con 0x7fc6500b4930 2021-02-10T11:22:46.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.256 7fc656a18700 1 -- 172.21.2.117:0/4056782258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc650077a70 con 0x7fc6500b4930 2021-02-10T11:22:46.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.256 7fc64d7fa700 1 -- 172.21.2.117:0/4056782258 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc638000c90 con 0x7fc6500b4930 2021-02-10T11:22:46.254 INFO:teuthology.orchestra.run.gibba017.stdout:55834574856 2021-02-10T11:22:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.258 7fc656a18700 1 -- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 msgr2=0x7fc63401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.258 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 0x7fc63401abd0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.258 7fc656a18700 1 -- 172.21.2.117:0/4056782258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 msgr2=0x7fc6500a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.258 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc638006f10 tx=0x7fc63800b300).stop 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 -- 172.21.2.117:0/4056782258 shutdown_connections 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6500a2fe0 0x7fc6501996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc650198c90 0x7fc6500a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6340186d0 0x7fc63401abd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 --2- 172.21.2.117:0/4056782258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6500b4930 0x7fc6500a23a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 -- 172.21.2.117:0/4056782258 >> 172.21.2.117:0/4056782258 conn(0x7fc65019f5a0 msgr2=0x7fc6501abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 -- 172.21.2.117:0/4056782258 shutdown_connections 2021-02-10T11:22:46.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.259 7fc656a18700 1 -- 172.21.2.117:0/4056782258 wait complete. 2021-02-10T11:22:46.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.1 2021-02-10T11:22:46.263 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-10T11:22:46.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 Processor -- start 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 -- start start 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0cd0079000 0x7f0cd00e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00d1d00 con 0x7f0cd01b00d0 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd0077a70 con 0x7f0cd007da20 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd732f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd00e53a0 con 0x7f0cd0079000 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd594c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0cd01b00d0 0x7f0cd00d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41870/0 (socket says 172.21.2.117:41870) 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.677 7f0cd594c700 1 -- 172.21.2.117:0/4185808445 learned_addr learned my addr 172.21.2.117:0/4185808445 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158500390 0 0) 0x7f0cc8000f90 con 0x7f0cd01b00d0 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc002fe0 con 0x7f0cd01b00d0 2021-02-10T11:22:46.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4143672637 0 0) 0x7f0cc8001460 con 0x7f0cd01b00d0 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 >> v2:172.21.2.125:3300/0 conn(0x7f0cd0079000 msgr2=0x7f0cd00e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 --2- 172.21.2.117:0/4185808445 >> v2:172.21.2.125:3300/0 conn(0x7f0cd0079000 0x7f0cd00e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 >> v1:172.21.2.125:6789/0 conn(0x7f0cd007da20 legacy=0x7f0cd00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0079400 con 0x7f0cd01b00d0 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0cc8001980 con 0x7f0cd01b00d0 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 >> v1:172.21.2.117:6789/0 conn(0x7f0cd01b00d0 legacy=0x7f0cd00d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 0x7f0cbc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cbc006790 0x7f0cbc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cbc009590 0x7f0cbc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c300 con 0x7f0cbc003aa0 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c520 con 0x7f0cbc006790 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc00c740 con 0x7f0cbc009590 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cd514b700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 0x7f0cbc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.678 7f0cd494a700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cbc006790 0x7f0cbc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd594c700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cbc009590 0x7f0cbc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 -- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cbc009590 msgr2=0x7f0cbc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cbc009590 0x7f0cbc00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 -- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cbc006790 msgr2=0x7f0cbc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cbc006790 0x7f0cbc008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 -- 172.21.2.117:0/4185808445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc00c9e0 con 0x7f0cbc003aa0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd594c700 1 -- 172.21.2.117:0/4185808445 reap_dead start 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd514b700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 0x7f0cbc005ef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0cc4006ab0 tx=0x7f0cc4009b40).ready entity=mon.0 client_cookie=79ce789e5d69be81 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cc40066a0 con 0x7f0cbc003aa0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4000e20 con 0x7f0cbc003aa0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cc3fff700 1 -- 172.21.2.117:0/4185808445 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cc400b4d0 con 0x7f0cbc003aa0 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd732f700 1 -- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 msgr2=0x7f0cbc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.679 7f0cd732f700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 0x7f0cbc005ef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f0cc4006ab0 tx=0x7f0cc4009b40).stop 2021-02-10T11:22:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.680 7f0cd732f700 1 -- 172.21.2.117:0/4185808445 shutdown_connections 2021-02-10T11:22:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.680 7f0cd732f700 1 --2- 172.21.2.117:0/4185808445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cbc003aa0 0x7f0cbc005ef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.680 7f0cd732f700 1 -- 172.21.2.117:0/4185808445 >> 172.21.2.117:0/4185808445 conn(0x7f0cd00ce430 msgr2=0x7f0cd00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.680 7f0cd732f700 1 -- 172.21.2.117:0/4185808445 shutdown_connections 2021-02-10T11:22:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.680 7f0cd732f700 1 -- 172.21.2.117:0/4185808445 wait complete. 2021-02-10T11:22:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.694 7f0cd732f700 1 Processor -- start 2021-02-10T11:22:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.694 7f0cd732f700 1 -- start start 2021-02-10T11:22:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cd0198c90 0x7f0cd00a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cd00a2fe0 0x7f0cd01996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd01080f0 con 0x7f0cd00b4930 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0199170 con 0x7f0cd00a2fe0 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd732f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd0107c90 con 0x7f0cd0198c90 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41628/0 (socket says 172.21.2.117:41628) 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 -- 172.21.2.117:0/192556485 learned_addr learned my addr 172.21.2.117:0/192556485 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 -- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cd0198c90 msgr2=0x7f0cd00a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cd0198c90 0x7f0cd00a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 -- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cd00a2fe0 msgr2=0x7f0cd01996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cd00a2fe0 0x7f0cd01996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 -- 172.21.2.117:0/192556485 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0199ce0 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.695 7f0cd514b700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0cc400b630 tx=0x7f0cc400b9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cc4000940 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4002030 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0cc40022a0 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cd732f700 1 -- 172.21.2.117:0/192556485 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0199ce0 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cd732f700 1 -- 172.21.2.117:0/192556485 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd0199ce0 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0cc4002210 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 0x7f0cb001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.696 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f0cc4032810 con 0x7f0cd00b4930 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.697 7f0cd494a700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 0x7f0cb001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.697 7f0cd494a700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 0x7f0cb001abd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.727 7f0cd732f700 1 -- 172.21.2.117:0/192556485 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd0077a70 con 0x7f0cd00b4930 2021-02-10T11:22:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.729 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0cc4000ce0 con 0x7f0cd00b4930 2021-02-10T11:22:46.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.868 7f0cd732f700 1 -- 172.21.2.117:0/192556485 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0cd0077a70 con 0x7f0cd00b4930 2021-02-10T11:22:46.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.869 7f0cc1ffb700 1 -- 172.21.2.117:0/192556485 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cc400bdc0 con 0x7f0cd00b4930 2021-02-10T11:22:46.866 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:46.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 msgr2=0x7f0cb001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 0x7f0cb001abd0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 msgr2=0x7f0cd00a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:46.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0cc400b630 tx=0x7f0cc400b9d0).stop 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 shutdown_connections 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0cd00a2fe0 0x7f0cd01996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0cd0198c90 0x7f0cd00a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0cb00186d0 0x7f0cb001abd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 --2- 172.21.2.117:0/192556485 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0cd00b4930 0x7f0cd00a23a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 >> 172.21.2.117:0/192556485 conn(0x7f0cd019f5a0 msgr2=0x7f0cd01abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 shutdown_connections 2021-02-10T11:22:46.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:46.871 7f0cd732f700 1 -- 172.21.2.117:0/192556485 wait complete. 2021-02-10T11:22:46.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.2 2021-02-10T11:22:46.877 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.3 2021-02-10T11:22:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.288 7f0594367700 1 Processor -- start 2021-02-10T11:22:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.288 7f0594367700 1 -- start start 2021-02-10T11:22:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0594367700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c079000 0x7f058c0e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0594367700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f058c1b00d0 0x7f058c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0594367700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c077a70 con 0x7f058c07da20 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0594367700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c0e53a0 con 0x7f058c1b00d0 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0594367700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c079400 con 0x7f058c079000 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0592183700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f058c07da20 0x7f058c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41886/0 (socket says 172.21.2.117:41886) 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.289 7f0592183700 1 -- 172.21.2.117:0/2103299488 learned_addr learned my addr 172.21.2.117:0/2103299488 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001103672 0 0) 0x7f0580001180 con 0x7f058c07da20 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578002fe0 con 0x7f058c07da20 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591982700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c079000 0x7f058c0e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592984700 1 --2- 172.21.2.117:0/2103299488 >> v2:172.21.2.125:3300/0 conn(0x7f058c1b00d0 0x7f058c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4271796015 0 0) 0x7f05800010e0 con 0x7f058c07da20 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c079000 msgr2=0x7f058c0e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c079000 0x7f058c0e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 >> v2:172.21.2.125:3300/0 conn(0x7f058c1b00d0 msgr2=0x7f058c1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 --2- 172.21.2.117:0/2103299488 >> v2:172.21.2.125:3300/0 conn(0x7f058c1b00d0 0x7f058c1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c07de20 con 0x7f058c07da20 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0580001600 con 0x7f058c07da20 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 >> v1:172.21.2.117:6789/0 conn(0x7f058c07da20 legacy=0x7f058c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0578003af0 0x7f0578005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f057804a830 0x7f057804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f057804d5a0 0x7f057804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05780502e0 con 0x7f0578003af0 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578050500 con 0x7f057804a830 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0591181700 1 -- 172.21.2.117:0/2103299488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0578050720 con 0x7f057804d5a0 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0578003af0 0x7f0578005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 -- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f057804d5a0 msgr2=0x7f057804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f057804d5a0 0x7f057804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 -- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f057804a830 msgr2=0x7f057804cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f057804a830 0x7f057804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.290 7f0592183700 1 -- 172.21.2.117:0/2103299488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780509c0 con 0x7f0578003af0 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0592183700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0578003af0 0x7f0578005f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0580000e80 tx=0x7f0580007c70).ready entity=mon.0 client_cookie=ecdf99fae471b8aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05800070c0 con 0x7f0578003af0 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058000a520 con 0x7f0578003af0 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0591181700 1 -- 172.21.2.117:0/2103299488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f058000a9b0 con 0x7f0578003af0 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0594367700 1 -- 172.21.2.117:0/2103299488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0578003af0 msgr2=0x7f0578005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0594367700 1 --2- 172.21.2.117:0/2103299488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0578003af0 0x7f0578005f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0580000e80 tx=0x7f0580007c70).stop 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.291 7f0592984700 1 -- 172.21.2.117:0/2103299488 reap_dead start 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.292 7f0594367700 1 -- 172.21.2.117:0/2103299488 shutdown_connections 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.292 7f0594367700 1 -- 172.21.2.117:0/2103299488 >> 172.21.2.117:0/2103299488 conn(0x7f058c0ce430 msgr2=0x7f058c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.292 7f0594367700 1 -- 172.21.2.117:0/2103299488 shutdown_connections 2021-02-10T11:22:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.292 7f0594367700 1 -- 172.21.2.117:0/2103299488 wait complete. 2021-02-10T11:22:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.306 7f0594367700 1 Processor -- start 2021-02-10T11:22:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.306 7f0594367700 1 -- start start 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f058c0b8cc0 0x7f058c0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c0a16f0 0x7f058c092990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c1a9850 con 0x7f058c0a0cd0 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c092fb0 con 0x7f058c0b8cc0 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0594367700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0931d0 con 0x7f058c0a16f0 2021-02-10T11:22:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41648/0 (socket says 172.21.2.117:41648) 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 -- 172.21.2.117:0/3847598745 learned_addr learned my addr 172.21.2.117:0/3847598745 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 -- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c0a16f0 msgr2=0x7f058c092990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c0a16f0 0x7f058c092990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 -- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f058c0b8cc0 msgr2=0x7f058c0a06b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f058c0b8cc0 0x7f058c0a06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 -- 172.21.2.117:0/3847598745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0933f0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.307 7f0591982700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f058800b610 tx=0x7f058800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05880009b0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0588006030 con 0x7f058c0a0cd0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0588006360 con 0x7f058c0a0cd0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f0594367700 1 -- 172.21.2.117:0/3847598745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c0933f0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f0594367700 1 -- 172.21.2.117:0/3847598745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c0933f0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.308 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0588006210 con 0x7f058c0a0cd0 2021-02-10T11:22:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.309 7f057effd700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 0x7f057401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.309 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f05880326f0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.309 7f0592183700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 0x7f057401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.309 7f0592183700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 0x7f057401abc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.339 7f0594367700 1 -- 172.21.2.117:0/3847598745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c077a70 con 0x7f058c0a0cd0 2021-02-10T11:22:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.341 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0588006bb0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.478 7f0594367700 1 -- 172.21.2.117:0/3847598745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f058c077a70 con 0x7f058c0a0cd0 2021-02-10T11:22:47.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.478 7f057effd700 1 -- 172.21.2.117:0/3847598745 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f058800bda0 con 0x7f058c0a0cd0 2021-02-10T11:22:47.475 INFO:teuthology.orchestra.run.gibba017.stdout:55834574855 2021-02-10T11:22:47.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 msgr2=0x7f057401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 0x7f057401abc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 msgr2=0x7f058c0a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f058800b610 tx=0x7f058800b9b0).stop 2021-02-10T11:22:47.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 shutdown_connections 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05740186c0 0x7f057401abc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f058c0a0cd0 0x7f058c0a10d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f058c0a16f0 0x7f058c092990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 --2- 172.21.2.117:0/3847598745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f058c0b8cc0 0x7f058c0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 >> 172.21.2.117:0/3847598745 conn(0x7f058c19f5a0 msgr2=0x7f058c1abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 shutdown_connections 2021-02-10T11:22:47.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.480 7f0594367700 1 -- 172.21.2.117:0/3847598745 wait complete. 2021-02-10T11:22:47.485 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2021-02-10T11:22:47.485 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.4 2021-02-10T11:22:47.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.896 7fe5f1eed700 1 Processor -- start 2021-02-10T11:22:47.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.896 7fe5f1eed700 1 -- start start 2021-02-10T11:22:47.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5f1eed700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe5ec079000 0x7fe5ec0e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5f1eed700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec0d1d00 con 0x7fe5ec07da20 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5f1eed700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec077a70 con 0x7fe5ec1b00d0 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5f1eed700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5ec0e53a0 con 0x7fe5ec079000 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5eb7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe5ec07da20 0x7fe5ec0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41906/0 (socket says 172.21.2.117:41906) 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5eb7fe700 1 -- 172.21.2.117:0/2597085279 learned_addr learned my addr 172.21.2.117:0/2597085279 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5eaffd700 1 --2- 172.21.2.117:0/2597085279 >> v2:172.21.2.125:3300/0 conn(0x7fe5ec079000 0x7fe5ec0e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5ea7fc700 1 -- 172.21.2.117:0/2597085279 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516093282 0 0) 0x7fe5d40010b0 con 0x7fe5ec07da20 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5ea7fc700 1 -- 172.21.2.117:0/2597085279 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d8002fe0 con 0x7fe5ec07da20 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.897 7fe5eaffd700 1 -- 172.21.2.117:0/2597085279 >> v1:172.21.2.125:6789/0 conn(0x7fe5ec1b00d0 legacy=0x7fe5ec0d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5eaffd700 1 -- 172.21.2.117:0/2597085279 >> v1:172.21.2.117:6789/0 conn(0x7fe5ec07da20 legacy=0x7fe5ec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5eaffd700 1 -- 172.21.2.117:0/2597085279 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec079400 con 0x7fe5ec079000 2021-02-10T11:22:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5eaffd700 1 --2- 172.21.2.117:0/2597085279 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec079000 0x7fe5ec0e4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe5dc006650 tx=0x7fe5dc009a80).ready entity=mon.1 client_cookie=247567ca9e0d48bb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5ea7fc700 1 -- 172.21.2.117:0/2597085279 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc00c0c0 con 0x7fe5ec079000 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5ea7fc700 1 -- 172.21.2.117:0/2597085279 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc00c340 con 0x7fe5ec079000 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5ea7fc700 1 -- 172.21.2.117:0/2597085279 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc00c820 con 0x7fe5ec079000 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 -- 172.21.2.117:0/2597085279 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec079000 msgr2=0x7fe5ec0e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 --2- 172.21.2.117:0/2597085279 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec079000 0x7fe5ec0e4d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe5dc006650 tx=0x7fe5dc009a80).stop 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 -- 172.21.2.117:0/2597085279 shutdown_connections 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 --2- 172.21.2.117:0/2597085279 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec079000 0x7fe5ec0e4d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 -- 172.21.2.117:0/2597085279 >> 172.21.2.117:0/2597085279 conn(0x7fe5ec0ce430 msgr2=0x7fe5ec0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 -- 172.21.2.117:0/2597085279 shutdown_connections 2021-02-10T11:22:47.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.898 7fe5f1eed700 1 -- 172.21.2.117:0/2597085279 wait complete. 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.913 7fe5f1eed700 1 Processor -- start 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.913 7fe5f1eed700 1 -- start start 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.913 7fe5f1eed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe5ec0b0750 0x7fe5ec19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.913 7fe5f1eed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec09afa0 0x7fe5ec09b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec09a480 con 0x7fe5ec19aa30 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec09bb40 con 0x7fe5ec09afa0 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec1b2340 con 0x7fe5ec0b0750 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41660/0 (socket says 172.21.2.117:41660) 2021-02-10T11:22:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 -- 172.21.2.117:0/3211695191 learned_addr learned my addr 172.21.2.117:0/3211695191 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 -- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe5ec0b0750 msgr2=0x7fe5ec19a410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe5ec0b0750 0x7fe5ec19a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 -- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec09afa0 msgr2=0x7fe5ec09b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec09afa0 0x7fe5ec09b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 -- 172.21.2.117:0/3211695191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec1b2560 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5eaffd700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe5dc009650 tx=0x7fe5dc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc00bd50 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc006030 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec1b2560 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec1b2560 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.914 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe5dc005560 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.915 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe5dc0052e0 con 0x7fe5ec19aa30 2021-02-10T11:22:47.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.915 7fe5e3fff700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:47.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.915 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fe5dc0326b0 con 0x7fe5ec19aa30 2021-02-10T11:22:47.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.916 7fe5eb7fe700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:47.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.916 7fe5eb7fe700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:47.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.946 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec09bb40 con 0x7fe5ec19aa30 2021-02-10T11:22:47.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:47.949 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe5dc000940 con 0x7fe5ec19aa30 2021-02-10T11:22:48.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.086 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe5ec09bb40 con 0x7fe5ec19aa30 2021-02-10T11:22:48.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.087 7fe5e3fff700 1 -- 172.21.2.117:0/3211695191 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe5dc00bd50 con 0x7fe5ec19aa30 2021-02-10T11:22:48.084 INFO:teuthology.orchestra.run.gibba017.stdout:55834574856 2021-02-10T11:22:48.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 msgr2=0x7fe5d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 0x7fe5d001abc0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 msgr2=0x7fe5ec09a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe5dc009650 tx=0x7fe5dc00b960).stop 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 shutdown_connections 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe5d00186c0 0x7fe5d001abc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe5ec19aa30 0x7fe5ec09a980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe5ec09afa0 0x7fe5ec09b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 --2- 172.21.2.117:0/3211695191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe5ec0b0750 0x7fe5ec19a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 >> 172.21.2.117:0/3211695191 conn(0x7fe5ec1a2e20 msgr2=0x7fe5ec1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:48.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 shutdown_connections 2021-02-10T11:22:48.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.089 7fe5f1eed700 1 -- 172.21.2.117:0/3211695191 wait complete. 2021-02-10T11:22:48.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.4 2021-02-10T11:22:48.094 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.5 2021-02-10T11:22:48.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 Processor -- start 2021-02-10T11:22:48.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 -- start start 2021-02-10T11:22:48.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 0x7f2b140e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2b141b00d0 0x7f2b141b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b14077a70 con 0x7f2b1407da20 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140e53a0 con 0x7f2b141b00d0 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.504 7f2b1ba8f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14079400 con 0x7f2b14079000 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 0x7f2b140e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b198ab700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f2b1407da20 0x7f2b140771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34844/0 (socket says 172.21.2.117:34844) 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b198ab700 1 -- 172.21.2.117:0/523818090 learned_addr learned my addr 172.21.2.117:0/523818090 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b1a0ac700 1 --2- 172.21.2.117:0/523818090 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b00d0 0x7f2b141b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 -- 172.21.2.117:0/523818090 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b00d0 msgr2=0x7f2b141b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 --2- 172.21.2.117:0/523818090 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b00d0 0x7f2b141b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 -- 172.21.2.117:0/523818090 >> v1:172.21.2.125:6789/0 conn(0x7f2b1407da20 legacy=0x7f2b140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 -- 172.21.2.117:0/523818090 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1407de20 con 0x7f2b14079000 2021-02-10T11:22:48.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.506 7f2b190aa700 1 --2- 172.21.2.117:0/523818090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 0x7f2b140e4d80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2b10006650 tx=0x7f2b10009a80).ready entity=mon.2 client_cookie=4210d827377bbaa3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b188a9700 1 -- 172.21.2.117:0/523818090 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b1000c0c0 con 0x7f2b14079000 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b188a9700 1 -- 172.21.2.117:0/523818090 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1000c340 con 0x7f2b14079000 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b188a9700 1 -- 172.21.2.117:0/523818090 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b1000c860 con 0x7f2b14079000 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 -- 172.21.2.117:0/523818090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 msgr2=0x7f2b140e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 --2- 172.21.2.117:0/523818090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 0x7f2b140e4d80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f2b10006650 tx=0x7f2b10009a80).stop 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 -- 172.21.2.117:0/523818090 shutdown_connections 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 --2- 172.21.2.117:0/523818090 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b00d0 0x7f2b141b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 --2- 172.21.2.117:0/523818090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14079000 0x7f2b140e4d80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.507 7f2b1ba8f700 1 -- 172.21.2.117:0/523818090 >> 172.21.2.117:0/523818090 conn(0x7f2b140ce430 msgr2=0x7f2b140e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.508 7f2b1ba8f700 1 -- 172.21.2.117:0/523818090 shutdown_connections 2021-02-10T11:22:48.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.508 7f2b1ba8f700 1 -- 172.21.2.117:0/523818090 wait complete. 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 Processor -- start 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 -- start start 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14095360 0x7f2b140935b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b14093bd0 0x7f2b141a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141a76c0 con 0x7f2b140aa760 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14093fd0 con 0x7f2b14093bd0 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b1ba8f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141a81e0 con 0x7f2b14095360 2021-02-10T11:22:48.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b198ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b198ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41676/0 (socket says 172.21.2.117:41676) 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.522 7f2b198ab700 1 -- 172.21.2.117:0/1858838261 learned_addr learned my addr 172.21.2.117:0/1858838261 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 -- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14095360 msgr2=0x7f2b140935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14095360 0x7f2b140935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 -- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b14093bd0 msgr2=0x7f2b141a7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b14093bd0 0x7f2b141a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 -- 172.21.2.117:0/1858838261 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b141a8400 con 0x7f2b140aa760 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b198ab700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2b08007fa0 tx=0x7f2b0800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b080070d0 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0800ad40 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.523 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b141a8400 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.524 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b080045a0 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.524 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b141a8400 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.524 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b08000e20 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.524 7f2b067fc700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 0x7f2afc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.524 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2b08032930 con 0x7f2b140aa760 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.525 7f2b190aa700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 0x7f2afc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.525 7f2b190aa700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 0x7f2afc01abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:48.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.555 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b141b4ee0 con 0x7f2b140aa760 2021-02-10T11:22:48.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.557 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b08000c10 con 0x7f2b140aa760 2021-02-10T11:22:48.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.696 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2b14077a70 con 0x7f2b140aa760 2021-02-10T11:22:48.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.696 7f2b067fc700 1 -- 172.21.2.117:0/1858838261 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b0800ad40 con 0x7f2b140aa760 2021-02-10T11:22:48.693 INFO:teuthology.orchestra.run.gibba017.stdout:55834574856 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 msgr2=0x7f2afc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 0x7f2afc01abc0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 msgr2=0x7f2b141b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2b08007fa0 tx=0x7f2b0800a950).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 shutdown_connections 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b14093bd0 0x7f2b141a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b14095360 0x7f2b140935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2afc018730 0x7f2afc01abc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 --2- 172.21.2.117:0/1858838261 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140aa760 0x7f2b141b8b00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.698 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 >> 172.21.2.117:0/1858838261 conn(0x7f2b141a95f0 msgr2=0x7f2b14094350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.699 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 shutdown_connections 2021-02-10T11:22:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:48.699 7f2b1ba8f700 1 -- 172.21.2.117:0/1858838261 wait complete. 2021-02-10T11:22:48.703 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.5 2021-02-10T11:22:48.704 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.6 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.114 7ff624b2d700 1 Processor -- start 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.114 7ff624b2d700 1 -- start start 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff624b2d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff62007da20 0x7ff6200771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff624b2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200e5180 0x7ff620079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff624b2d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff620077a70 con 0x7ff6200e4d80 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff624b2d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200777d0 con 0x7ff62007da20 2021-02-10T11:22:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff624b2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6201b08c0 con 0x7ff6200e5180 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff61e05d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff6200e4d80 0x7ff6201b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41932/0 (socket says 172.21.2.117:41932) 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff61e05d700 1 -- 172.21.2.117:0/1498432352 learned_addr learned my addr 172.21.2.117:0/1498432352 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732971969 0 0) 0x7ff610001180 con 0x7ff6200e4d80 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.115 7ff61f05f700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200e5180 0x7ff620079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 --2- 172.21.2.117:0/1498432352 >> v2:172.21.2.125:3300/0 conn(0x7ff62007da20 0x7ff6200771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60c002fe0 con 0x7ff6200e4d80 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2742307242 0 0) 0x7ff610001180 con 0x7ff6200e4d80 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200e5180 msgr2=0x7ff620079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200e5180 0x7ff620079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 >> v2:172.21.2.125:3300/0 conn(0x7ff62007da20 msgr2=0x7ff6200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 --2- 172.21.2.117:0/1498432352 >> v2:172.21.2.125:3300/0 conn(0x7ff62007da20 0x7ff6200771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62007de20 con 0x7ff6200e4d80 2021-02-10T11:22:49.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff610001680 con 0x7ff6200e4d80 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 >> v1:172.21.2.117:6789/0 conn(0x7ff6200e4d80 legacy=0x7ff6201b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff60c003a60 0x7ff60c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff60c04a7a0 0x7ff60c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff60c04d510 0x7ff60c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0502e0 con 0x7ff60c003a60 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c050500 con 0x7ff60c04d510 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c050720 con 0x7ff60c04a7a0 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff60c003a60 0x7ff60c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e05d700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff60c04a7a0 0x7ff60c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61f05f700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff60c04d510 0x7ff60c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 -- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff60c04a7a0 msgr2=0x7ff60c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff60c04a7a0 0x7ff60c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 -- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff60c04d510 msgr2=0x7ff60c04fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff60c04d510 0x7ff60c04fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.116 7ff61e85e700 1 -- 172.21.2.117:0/1498432352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0509c0 con 0x7ff60c003a60 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff61e85e700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff60c003a60 0x7ff60c005ef0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7ff60800c8f0 tx=0x7ff60800cc90).ready entity=mon.0 client_cookie=bb8245125d033c60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff60800cdd0 con 0x7ff60c003a60 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608009310 con 0x7ff60c003a60 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff61d85c700 1 -- 172.21.2.117:0/1498432352 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6080097f0 con 0x7ff60c003a60 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff624b2d700 1 -- 172.21.2.117:0/1498432352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff60c003a60 msgr2=0x7ff60c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff624b2d700 1 --2- 172.21.2.117:0/1498432352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff60c003a60 0x7ff60c005ef0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7ff60800c8f0 tx=0x7ff60800cc90).stop 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.117 7ff61f05f700 1 -- 172.21.2.117:0/1498432352 reap_dead start 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.118 7ff624b2d700 1 -- 172.21.2.117:0/1498432352 shutdown_connections 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.118 7ff624b2d700 1 -- 172.21.2.117:0/1498432352 >> 172.21.2.117:0/1498432352 conn(0x7ff6200ce430 msgr2=0x7ff6200e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.118 7ff624b2d700 1 -- 172.21.2.117:0/1498432352 shutdown_connections 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.118 7ff624b2d700 1 -- 172.21.2.117:0/1498432352 wait complete. 2021-02-10T11:22:49.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.132 7ff624b2d700 1 Processor -- start 2021-02-10T11:22:49.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.132 7ff624b2d700 1 -- start start 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 0x7ff6200a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff620198c90 0x7ff6200a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6201080f0 con 0x7ff6200a2fe0 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff620199170 con 0x7ff620198c90 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff624b2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff620107c90 con 0x7ff6200b4930 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff61f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff61f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41696/0 (socket says 172.21.2.117:41696) 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff61f05f700 1 -- 172.21.2.117:0/3928078276 learned_addr learned my addr 172.21.2.117:0/3928078276 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff61e85e700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 0x7ff6200a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.133 7ff61e05d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff620198c90 0x7ff6200a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 -- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff620198c90 msgr2=0x7ff6200a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff620198c90 0x7ff6200a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 -- 172.21.2.117:0/3928078276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 msgr2=0x7ff6201996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 -- 172.21.2.117:0/3928078276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620199ce0 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61f05f700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff61e85e700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 0x7ff6200a23a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ff60800c1d0 tx=0x7ff60800b300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff60800c7c0 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608009e20 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff60800cc50 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff620199ce0 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.134 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff620199ce0 con 0x7ff6200b4930 2021-02-10T11:22:49.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.135 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff60800b6f0 con 0x7ff6200b4930 2021-02-10T11:22:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.135 7ff6177fe700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 0x7ff60401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.135 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7ff608032630 con 0x7ff6200b4930 2021-02-10T11:22:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.135 7ff61e05d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 0x7ff60401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.135 7ff61e05d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 0x7ff60401abc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.165 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff620107c90 con 0x7ff6200b4930 2021-02-10T11:22:49.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.168 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff608000c90 con 0x7ff6200b4930 2021-02-10T11:22:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.308 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff620199140 con 0x7ff6200b4930 2021-02-10T11:22:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.309 7ff6177fe700 1 -- 172.21.2.117:0/3928078276 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff60800b6f0 con 0x7ff6200b4930 2021-02-10T11:22:49.305 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 msgr2=0x7ff60401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 0x7ff60401abc0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 msgr2=0x7ff6200a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 0x7ff6200a23a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ff60800c1d0 tx=0x7ff60800b300).stop 2021-02-10T11:22:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 shutdown_connections 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6040186c0 0x7ff60401abc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6200a2fe0 0x7ff6201996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff620198c90 0x7ff6200a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 --2- 172.21.2.117:0/3928078276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6200b4930 0x7ff6200a23a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 >> 172.21.2.117:0/3928078276 conn(0x7ff62019f5a0 msgr2=0x7ff6201abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.310 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 shutdown_connections 2021-02-10T11:22:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.311 7ff624b2d700 1 -- 172.21.2.117:0/3928078276 wait complete. 2021-02-10T11:22:49.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2021-02-10T11:22:49.316 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd last-stat-seq osd.7 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 Processor -- start 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 -- start start 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 0x7f05300771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f05300e4d80 0x7f05301b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0530077a70 con 0x7f05300e5180 2021-02-10T11:22:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300e5c00 con 0x7f05300e4d80 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.727 7f053506d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300777d0 con 0x7f053007da20 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052f59e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f05300e5180 0x7f0530077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41956/0 (socket says 172.21.2.117:41956) 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052f59e700 1 -- 172.21.2.117:0/3623292550 learned_addr learned my addr 172.21.2.117:0/3623292550 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052ed9d700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 0x7f05300771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052e59c700 1 --2- 172.21.2.117:0/3623292550 >> v2:172.21.2.125:3300/0 conn(0x7f05300e4d80 0x7f05301b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187490198 0 0) 0x7f0528001500 con 0x7f05300e5180 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.728 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0518002fe0 con 0x7f05300e5180 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052e59c700 1 -- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 msgr2=0x7f05300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2290934352 0 0) 0x7f0528001500 con 0x7f05300e5180 2021-02-10T11:22:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052e59c700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 0x7f05300771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052e59c700 1 -- 172.21.2.117:0/3623292550 >> v1:172.21.2.117:6789/0 conn(0x7f05300e5180 legacy=0x7f0530077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052e59c700 1 -- 172.21.2.117:0/3623292550 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053007de20 con 0x7f05300e4d80 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052ed9d700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 0x7f05300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052e59c700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05300e4d80 0x7f05301b00d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0524006650 tx=0x7f0524009a80).ready entity=mon.1 client_cookie=44ce704a8723e236 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05240009b0 con 0x7f05300e4d80 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052400c0c0 con 0x7f05300e4d80 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f052dd9b700 1 -- 172.21.2.117:0/3623292550 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f052400c5e0 con 0x7f05300e4d80 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.729 7f053506d700 1 -- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05300e4d80 msgr2=0x7f05301b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05300e4d80 0x7f05301b00d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0524006650 tx=0x7f0524009a80).stop 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 -- 172.21.2.117:0/3623292550 shutdown_connections 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05300e4d80 0x7f05301b00d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 --2- 172.21.2.117:0/3623292550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053007da20 0x7f05300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 -- 172.21.2.117:0/3623292550 >> 172.21.2.117:0/3623292550 conn(0x7f05300ce430 msgr2=0x7f05300e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 -- 172.21.2.117:0/3623292550 shutdown_connections 2021-02-10T11:22:49.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.730 7f053506d700 1 -- 172.21.2.117:0/3623292550 wait complete. 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 Processor -- start 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 -- start start 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f053018f3e0 0x7f05300a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053018ff40 0x7f0530190340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300a99a0 con 0x7f05300a8860 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530190960 con 0x7f053018f3e0 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.745 7f053506d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0530190b80 con 0x7f053018ff40 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41708/0 (socket says 172.21.2.117:41708) 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 -- 172.21.2.117:0/677603488 learned_addr learned my addr 172.21.2.117:0/677603488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 -- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053018ff40 msgr2=0x7f0530190340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053018ff40 0x7f0530190340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 -- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f053018f3e0 msgr2=0x7f05300a9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f053018f3e0 0x7f05300a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 -- 172.21.2.117:0/677603488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0530190da0 con 0x7f05300a8860 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f052ed9d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0520006f40 tx=0x7f052000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.746 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f052000bd80 con 0x7f05300a8860 2021-02-10T11:22:49.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520000c90 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05200022b0 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f053506d700 1 -- 172.21.2.117:0/677603488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530190da0 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f053506d700 1 -- 172.21.2.117:0/677603488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530191330 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f052000a000 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f051f7fe700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 0x7f051401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.747 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f0520032670 con 0x7f05300a8860 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.748 7f052e59c700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 0x7f051401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.748 7f052e59c700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 0x7f051401abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:49.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.778 7f053506d700 1 -- 172.21.2.117:0/677603488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530103260 con 0x7f05300a8860 2021-02-10T11:22:49.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.780 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05200009b0 con 0x7f05300a8860 2021-02-10T11:22:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.919 7f053506d700 1 -- 172.21.2.117:0/677603488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0530077a70 con 0x7f05300a8860 2021-02-10T11:22:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.919 7f051f7fe700 1 -- 172.21.2.117:0/677603488 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f052000bd80 con 0x7f05300a8860 2021-02-10T11:22:49.916 INFO:teuthology.orchestra.run.gibba017.stdout:55834574857 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 msgr2=0x7f051401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 0x7f051401abc0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 msgr2=0x7f05300a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0520006f40 tx=0x7f052000b990).stop 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 shutdown_connections 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f053018ff40 0x7f0530190340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f053018f3e0 0x7f05300a9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05140186c0 0x7f051401abc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 --2- 172.21.2.117:0/677603488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05300a8860 0x7f05300a8c60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 >> 172.21.2.117:0/677603488 conn(0x7f0530107e80 msgr2=0x7f053019e850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 shutdown_connections 2021-02-10T11:22:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:49.921 7f053506d700 1 -- 172.21.2.117:0/677603488 wait complete. 2021-02-10T11:22:49.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2021-02-10T11:22:49.926 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-02-10T11:22:49.926 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.339 7f72bc17c700 1 Processor -- start 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.339 7f72bc17c700 1 -- start start 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72bc17c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f72b407da20 0x7f72b40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72bc17c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40e5190 0x7f72b4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72bc17c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b4077a70 con 0x7f72b40e4d90 2021-02-10T11:22:50.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72bc17c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40777d0 con 0x7f72b407da20 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72bc17c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41b08d0 con 0x7f72b40e5190 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72b9797700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f72b40e4d90 0x7f72b41b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41966/0 (socket says 172.21.2.117:41966) 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72b9797700 1 -- 172.21.2.117:0/2314078488 learned_addr learned my addr 172.21.2.117:0/2314078488 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204568894 0 0) 0x7f72a40009b0 con 0x7f72b40e4d90 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72ba799700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40e5190 0x7f72b4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a0002fe0 con 0x7f72b40e4d90 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.340 7f72b9f98700 1 --2- 172.21.2.117:0/2314078488 >> v2:172.21.2.125:3300/0 conn(0x7f72b407da20 0x7f72b40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3040252301 0 0) 0x7f72a40010b0 con 0x7f72b40e4d90 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40e5190 msgr2=0x7f72b4079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40e5190 0x7f72b4079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 >> v2:172.21.2.125:3300/0 conn(0x7f72b407da20 msgr2=0x7f72b40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 --2- 172.21.2.117:0/2314078488 >> v2:172.21.2.125:3300/0 conn(0x7f72b407da20 0x7f72b40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b407de20 con 0x7f72b40e4d90 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f72a40015b0 con 0x7f72b40e4d90 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 >> v1:172.21.2.117:6789/0 conn(0x7f72b40e4d90 legacy=0x7f72b41b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72a0003b10 0x7f72a0005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72a004a820 0x7f72a004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72a004d510 0x7f72a004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00502e0 con 0x7f72a004d510 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0050500 con 0x7f72a0003b10 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a0050720 con 0x7f72a004a820 2021-02-10T11:22:50.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72a004d510 0x7f72a004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b9f98700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72a0003b10 0x7f72a0005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72b9797700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72a004a820 0x7f72a004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 -- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72a004a820 msgr2=0x7f72a004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72a004a820 0x7f72a004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 -- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72a0003b10 msgr2=0x7f72a0005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72a0003b10 0x7f72a0005fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.341 7f72ba799700 1 -- 172.21.2.117:0/2314078488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00509c0 con 0x7f72a004d510 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72ba799700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72a004d510 0x7f72a004fa30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f72ac009b10 tx=0x7f72ac009eb0).ready entity=mon.0 client_cookie=b296409b828103d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72ac00b510 con 0x7f72a004d510 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac003030 con 0x7f72a004d510 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72b8f96700 1 -- 172.21.2.117:0/2314078488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72ac00b790 con 0x7f72a004d510 2021-02-10T11:22:50.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72bc17c700 1 -- 172.21.2.117:0/2314078488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72a004d510 msgr2=0x7f72a004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72bc17c700 1 --2- 172.21.2.117:0/2314078488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72a004d510 0x7f72a004fa30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f72ac009b10 tx=0x7f72ac009eb0).stop 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72ba799700 1 -- 172.21.2.117:0/2314078488 reap_dead start 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72bc17c700 1 -- 172.21.2.117:0/2314078488 shutdown_connections 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.342 7f72bc17c700 1 -- 172.21.2.117:0/2314078488 >> 172.21.2.117:0/2314078488 conn(0x7f72b40ce440 msgr2=0x7f72b40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.343 7f72bc17c700 1 -- 172.21.2.117:0/2314078488 shutdown_connections 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.343 7f72bc17c700 1 -- 172.21.2.117:0/2314078488 wait complete. 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.357 7f72bc17c700 1 Processor -- start 2021-02-10T11:22:50.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.357 7f72bc17c700 1 -- start start 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40b33b0 0x7f72b40b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72b418d6d0 0x7f72b418dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41ad6a0 con 0x7f72b418ccb0 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41ad8c0 con 0x7f72b418d6d0 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72bc17c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b41adae0 con 0x7f72b40b33b0 2021-02-10T11:22:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41726/0 (socket says 172.21.2.117:41726) 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 -- 172.21.2.117:0/4147565787 learned_addr learned my addr 172.21.2.117:0/4147565787 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 -- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40b33b0 msgr2=0x7f72b40b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40b33b0 0x7f72b40b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 -- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72b418d6d0 msgr2=0x7f72b418dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72b418d6d0 0x7f72b418dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 -- 172.21.2.117:0/4147565787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b41add00 con 0x7f72b418ccb0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.358 7f72b9797700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f72a4001c60 tx=0x7f72a400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a4001020 con 0x7f72b418ccb0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a40012a0 con 0x7f72b418ccb0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72a40043c0 con 0x7f72b418ccb0 2021-02-10T11:22:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b41add00 con 0x7f72b418ccb0 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b41add00 con 0x7f72b418ccb0 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72a4004140 con 0x7f72b418ccb0 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 0x7f729c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.359 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f72a40326d0 con 0x7f72b418ccb0 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.360 7f72b9f98700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 0x7f729c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.360 7f72b9f98700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 0x7f729c01abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.390 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4077a70 con 0x7f72b418ccb0 2021-02-10T11:22:50.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.392 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72a4020f90 con 0x7f72b418ccb0 2021-02-10T11:22:50.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.524 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f72b4077a70 con 0x7f729c0186c0 2021-02-10T11:22:50.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.529 7f72aa7fc700 1 -- 172.21.2.117:0/4147565787 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77333 (crc 0 0 0) 0x7f72b00052e0 con 0x7f729c0186c0 2021-02-10T11:22:50.529 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:50.530 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 msgr2=0x7f729c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 0x7f729c01abc0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 msgr2=0x7f72b418d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f72a4001c60 tx=0x7f72a400a8d0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 shutdown_connections 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f729c0186c0 0x7f729c01abc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72b418ccb0 0x7f72b418d0b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72b418d6d0 0x7f72b418dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 --2- 172.21.2.117:0/4147565787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72b40b33b0 0x7f72b40b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 >> 172.21.2.117:0/4147565787 conn(0x7f72b419f510 msgr2=0x7f72b41abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 shutdown_connections 2021-02-10T11:22:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.534 7f72bc17c700 1 -- 172.21.2.117:0/4147565787 wait complete. 2021-02-10T11:22:50.538 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2021-02-10 11:22:48.819985","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390528,"kb_used_data":1856,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584192,"statfs":{"total":773094113280,"available":764502212608,"internally_reserved":8589934592,"allocated":1900544,"data_stored":664500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001251"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921091","last_change":"2021-02-10 11:22:24.884900","last_active":"2021-02-10 11:22:31.921091","last_peered":"2021-02-10 11:22:31.921091","last_clean":"2021-02-10 11:22:31.921091","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:22:31.921091","last_undegraded":"2021-02-10 11:22:31.921091","last_fullsized":"2021-02-10 11:22:31.921091","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"22'8","reported_seq":"19","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.966869","last_change":"2021-02-10 11:22:28.908117","last_active":"2021-02-10 11:22:40.966869","last_peered":"2021-02-10 11:22:40.966869","last_clean":"2021-02-10 11:22:40.966869","last_became_active":"2021-02-10 11:22:28.907896","last_became_peered":"2021-02-10 11:22:28.907896","last_unstale":"2021-02-10 11:22:40.966869","last_undegraded":"2021-02-10 11:22:40.966869","last_fullsized":"2021-02-10 11:22:40.966869","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.922773","last_change":"2021-02-10 11:22:31.922394","last_active":"2021-02-10 11:22:31.922773","last_peered":"2021-02-10 11:22:31.922773","last_clean":"2021-02-10 11:22:31.922773","last_became_active":"2021-02-10 11:22:31.922009","last_became_peered":"2021-02-10 11:22:31.922009","last_unstale":"2021-02-10 11:22:31.922773","last_undegraded":"2021-02-10 11:22:31.922773","last_fullsized":"2021-02-10 11:22:31.922773","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921167","last_change":"2021-02-10 11:22:24.885206","last_active":"2021-02-10 11:22:31.921167","last_peered":"2021-02-10 11:22:31.921167","last_clean":"2021-02-10 11:22:31.921167","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:22:31.921167","last_undegraded":"2021-02-10 11:22:31.921167","last_fullsized":"2021-02-10 11:22:31.921167","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"22'6","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965769","last_change":"2021-02-10 11:22:28.906683","last_active":"2021-02-10 11:22:40.965769","last_peered":"2021-02-10 11:22:40.965769","last_clean":"2021-02-10 11:22:40.965769","last_became_active":"2021-02-10 11:22:28.906494","last_became_peered":"2021-02-10 11:22:28.906494","last_unstale":"2021-02-10 11:22:40.965769","last_undegraded":"2021-02-10 11:22:40.965769","last_fullsized":"2021-02-10 11:22:40.965769","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.923297","last_change":"2021-02-10 11:22:31.922650","last_active":"2021-02-10 11:22:31.923297","last_peered":"2021-02-10 11:22:31.923297","last_clean":"2021-02-10 11:22:31.923297","last_became_active":"2021-02-10 11:22:31.922394","last_became_peered":"2021-02-10 11:22:31.922394","last_unstale":"2021-02-10 11:22:31.923297","last_undegraded":"2021-02-10 11:22:31.923297","last_fullsized":"2021-02-10 11:22:31.923297","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.925252","last_change":"2021-02-10 11:22:31.924846","last_active":"2021-02-10 11:22:31.925252","last_peered":"2021-02-10 11:22:31.925252","last_clean":"2021-02-10 11:22:31.925252","last_became_active":"2021-02-10 11:22:31.924458","last_became_peered":"2021-02-10 11:22:31.924458","last_unstale":"2021-02-10 11:22:31.925252","last_undegraded":"2021-02-10 11:22:31.925252","last_fullsized":"2021-02-10 11:22:31.925252","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920971","last_change":"2021-02-10 11:22:24.885818","last_active":"2021-02-10 11:22:31.920971","last_peered":"2021-02-10 11:22:31.920971","last_clean":"2021-02-10 11:22:31.920971","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:22:31.920971","last_undegraded":"2021-02-10 11:22:31.920971","last_fullsized":"2021-02-10 11:22:31.920971","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.966621","last_change":"2021-02-10 11:22:28.907474","last_active":"2021-02-10 11:22:40.966621","last_peered":"2021-02-10 11:22:40.966621","last_clean":"2021-02-10 11:22:40.966621","last_became_active":"2021-02-10 11:22:28.907296","last_became_peered":"2021-02-10 11:22:28.907296","last_unstale":"2021-02-10 11:22:40.966621","last_undegraded":"2021-02-10 11:22:40.966621","last_fullsized":"2021-02-10 11:22:40.966621","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.925431","last_change":"2021-02-10 11:22:31.924942","last_active":"2021-02-10 11:22:31.925431","last_peered":"2021-02-10 11:22:31.925431","last_clean":"2021-02-10 11:22:31.925431","last_became_active":"2021-02-10 11:22:31.924505","last_became_peered":"2021-02-10 11:22:31.924505","last_unstale":"2021-02-10 11:22:31.925431","last_undegraded":"2021-02-10 11:22:31.925431","last_fullsized":"2021-02-10 11:22:31.925431","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"14","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920908","last_change":"2021-02-10 11:22:24.886002","last_active":"2021-02-10 11:22:31.920908","last_peered":"2021-02-10 11:22:31.920908","last_clean":"2021-02-10 11:22:31.920908","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:22:31.920908","last_undegraded":"2021-02-10 11:22:31.920908","last_fullsized":"2021-02-10 11:22:31.920908","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":"19","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965894","last_change":"2021-02-10 11:22:28.906256","last_active":"2021-02-10 11:22:40.965894","last_peered":"2021-02-10 11:22:40.965894","last_clean":"2021-02-10 11:22:40.965894","last_became_active":"2021-02-10 11:22:28.906045","last_became_peered":"2021-02-10 11:22:28.906045","last_unstale":"2021-02-10 11:22:40.965894","last_undegraded":"2021-02-10 11:22:40.965894","last_fullsized":"2021-02-10 11:22:40.965894","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921871","last_change":"2021-02-10 11:22:24.885674","last_active":"2021-02-10 11:22:31.921871","last_peered":"2021-02-10 11:22:31.921871","last_clean":"2021-02-10 11:22:31.921871","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:22:31.921871","last_undegraded":"2021-02-10 11:22:31.921871","last_fullsized":"2021-02-10 11:22:31.921871","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":"17","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.964993","last_change":"2021-02-10 11:22:28.905615","last_active":"2021-02-10 11:22:40.964993","last_peered":"2021-02-10 11:22:40.964993","last_clean":"2021-02-10 11:22:40.964993","last_became_active":"2021-02-10 11:22:28.905422","last_became_peered":"2021-02-10 11:22:28.905422","last_unstale":"2021-02-10 11:22:40.964993","last_undegraded":"2021-02-10 11:22:40.964993","last_fullsized":"2021-02-10 11:22:40.964993","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924944","last_change":"2021-02-10 11:22:31.924485","last_active":"2021-02-10 11:22:31.924944","last_peered":"2021-02-10 11:22:31.924944","last_clean":"2021-02-10 11:22:31.924944","last_became_active":"2021-02-10 11:22:31.924090","last_became_peered":"2021-02-10 11:22:31.924090","last_unstale":"2021-02-10 11:22:31.924944","last_undegraded":"2021-02-10 11:22:31.924944","last_fullsized":"2021-02-10 11:22:31.924944","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.919080","last_change":"2021-02-10 11:22:24.885203","last_active":"2021-02-10 11:22:31.919080","last_peered":"2021-02-10 11:22:31.919080","last_clean":"2021-02-10 11:22:31.919080","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:22:31.919080","last_undegraded":"2021-02-10 11:22:31.919080","last_fullsized":"2021-02-10 11:22:31.919080","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:32.548975","last_change":"2021-02-10 11:22:28.905680","last_active":"2021-02-10 11:22:32.548975","last_peered":"2021-02-10 11:22:32.548975","last_clean":"2021-02-10 11:22:32.548975","last_became_active":"2021-02-10 11:22:28.905492","last_became_peered":"2021-02-10 11:22:28.905492","last_unstale":"2021-02-10 11:22:32.548975","last_undegraded":"2021-02-10 11:22:32.548975","last_fullsized":"2021-02-10 11:22:32.548975","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.923600","last_change":"2021-02-10 11:22:31.923063","last_active":"2021-02-10 11:22:31.923600","last_peered":"2021-02-10 11:22:31.923600","last_clean":"2021-02-10 11:22:31.923600","last_became_active":"2021-02-10 11:22:31.922699","last_became_peered":"2021-02-10 11:22:31.922699","last_unstale":"2021-02-10 11:22:31.923600","last_undegraded":"2021-02-10 11:22:31.923600","last_fullsized":"2021-02-10 11:22:31.923600","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920500","last_change":"2021-02-10 11:22:28.906735","last_active":"2021-02-10 11:22:31.920500","last_peered":"2021-02-10 11:22:31.920500","last_clean":"2021-02-10 11:22:31.920500","last_became_active":"2021-02-10 11:22:28.906538","last_became_peered":"2021-02-10 11:22:28.906538","last_unstale":"2021-02-10 11:22:31.920500","last_undegraded":"2021-02-10 11:22:31.920500","last_fullsized":"2021-02-10 11:22:31.920500","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921848","last_change":"2021-02-10 11:22:24.885698","last_active":"2021-02-10 11:22:31.921848","last_peered":"2021-02-10 11:22:31.921848","last_clean":"2021-02-10 11:22:31.921848","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:22:31.921848","last_undegraded":"2021-02-10 11:22:31.921848","last_fullsized":"2021-02-10 11:22:31.921848","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924518","last_change":"2021-02-10 11:22:31.923835","last_active":"2021-02-10 11:22:31.924518","last_peered":"2021-02-10 11:22:31.924518","last_clean":"2021-02-10 11:22:31.924518","last_became_active":"2021-02-10 11:22:31.923471","last_became_peered":"2021-02-10 11:22:31.923471","last_unstale":"2021-02-10 11:22:31.924518","last_undegraded":"2021-02-10 11:22:31.924518","last_fullsized":"2021-02-10 11:22:31.924518","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"22'8","reported_seq":"20","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965939","last_change":"2021-02-10 11:22:28.906488","last_active":"2021-02-10 11:22:40.965939","last_peered":"2021-02-10 11:22:40.965939","last_clean":"2021-02-10 11:22:40.965939","last_became_active":"2021-02-10 11:22:28.906283","last_became_peered":"2021-02-10 11:22:28.906283","last_unstale":"2021-02-10 11:22:40.965939","last_undegraded":"2021-02-10 11:22:40.965939","last_fullsized":"2021-02-10 11:22:40.965939","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"14","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920236","last_change":"2021-02-10 11:22:24.886028","last_active":"2021-02-10 11:22:31.920236","last_peered":"2021-02-10 11:22:31.920236","last_clean":"2021-02-10 11:22:31.920236","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:22:31.920236","last_undegraded":"2021-02-10 11:22:31.920236","last_fullsized":"2021-02-10 11:22:31.920236","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924997","last_change":"2021-02-10 11:22:31.924599","last_active":"2021-02-10 11:22:31.924997","last_peered":"2021-02-10 11:22:31.924997","last_clean":"2021-02-10 11:22:31.924997","last_became_active":"2021-02-10 11:22:31.924231","last_became_peered":"2021-02-10 11:22:31.924231","last_unstale":"2021-02-10 11:22:31.924997","last_undegraded":"2021-02-10 11:22:31.924997","last_fullsized":"2021-02-10 11:22:31.924997","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048824,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323016,"statfs":{"total":96636764160,"available":95562768384,"internally_reserved":1073741824,"allocated":245760,"data_stored":83217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048816,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323024,"statfs":{"total":96636764160,"available":95562776576,"internally_reserved":1073741824,"allocated":237568,"data_stored":83057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":82525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":82491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048828,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323012,"statfs":{"total":96636764160,"available":95562764288,"internally_reserved":1073741824,"allocated":249856,"data_stored":84177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048820,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323020,"statfs":{"total":96636764160,"available":95562772480,"internally_reserved":1073741824,"allocated":241664,"data_stored":84017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":82525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":82491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]}]}} 2021-02-10T11:22:50.540 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:22:50.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.953 7fcf2181f700 1 Processor -- start 2021-02-10T11:22:50.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.953 7fcf2181f700 1 -- start start 2021-02-10T11:22:50.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf2181f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf1c079000 0x7fcf1c0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf2181f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 0x7fcf1c1b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf2181f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c077a70 con 0x7fcf1c07da20 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf2181f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c0e53b0 con 0x7fcf1c079000 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf2181f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c079400 con 0x7fcf1c1b00e0 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf1bd75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 0x7fcf1c1b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf1ad73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf1c079000 0x7fcf1c0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf1ad73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf1c079000 0x7fcf1c0e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:46118/0 (socket says 172.21.2.117:46118) 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf1b574700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcf1c07da20 0x7fcf1c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34910/0 (socket says 172.21.2.117:34910) 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.954 7fcf1ad73700 1 -- 172.21.2.117:0/841773049 learned_addr learned my addr 172.21.2.117:0/841773049 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1ad73700 1 -- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 msgr2=0x7fcf1c1b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:50.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1ad73700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 0x7fcf1c1b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1ad73700 1 -- 172.21.2.117:0/841773049 >> v1:172.21.2.125:6789/0 conn(0x7fcf1c07da20 legacy=0x7fcf1c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1ad73700 1 -- 172.21.2.117:0/841773049 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c07de20 con 0x7fcf1c079000 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1bd75700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 0x7fcf1c1b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1ad73700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c079000 0x7fcf1c0e4d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcf0c006700 tx=0x7fcf0c009b80).ready entity=mon.1 client_cookie=e103a0bfe6522ecc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1a572700 1 -- 172.21.2.117:0/841773049 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf0c00c0c0 con 0x7fcf1c079000 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1a572700 1 -- 172.21.2.117:0/841773049 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c00c340 con 0x7fcf1c079000 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf1a572700 1 -- 172.21.2.117:0/841773049 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf0c00c820 con 0x7fcf1c079000 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf2181f700 1 -- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c079000 msgr2=0x7fcf1c0e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.955 7fcf2181f700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c079000 0x7fcf1c0e4d90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcf0c006700 tx=0x7fcf0c009b80).stop 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 -- 172.21.2.117:0/841773049 shutdown_connections 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c1b00e0 0x7fcf1c1b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 --2- 172.21.2.117:0/841773049 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c079000 0x7fcf1c0e4d90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:50.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 -- 172.21.2.117:0/841773049 >> 172.21.2.117:0/841773049 conn(0x7fcf1c0ce440 msgr2=0x7fcf1c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:50.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 -- 172.21.2.117:0/841773049 shutdown_connections 2021-02-10T11:22:50.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.956 7fcf2181f700 1 -- 172.21.2.117:0/841773049 wait complete. 2021-02-10T11:22:50.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.970 7fcf2181f700 1 Processor -- start 2021-02-10T11:22:50.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.970 7fcf2181f700 1 -- start start 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c09d9a0 0x7fcf1c09dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c09e3c0 0x7fcf1c1a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c09e7c0 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1a3360 con 0x7fcf1c09d9a0 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf2181f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1a3580 con 0x7fcf1c09e3c0 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41738/0 (socket says 172.21.2.117:41738) 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 -- 172.21.2.117:0/1671202024 learned_addr learned my addr 172.21.2.117:0/1671202024 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:50.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 -- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c09e3c0 msgr2=0x7fcf1c1a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c09e3c0 0x7fcf1c1a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 -- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c09d9a0 msgr2=0x7fcf1c09dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c09d9a0 0x7fcf1c09dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.971 7fcf1b574700 1 -- 172.21.2.117:0/1671202024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c1a37a0 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf1b574700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcf04001c60 tx=0x7fcf0400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf0400acc0 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf04004140 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c1a37a0 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c1a3d30 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf04004620 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.972 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf04004320 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.973 7fcf13fff700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 0x7fcf0001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.973 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcf040328d0 con 0x7fcf1c1a1e50 2021-02-10T11:22:50.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.973 7fcf1ad73700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 0x7fcf0001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:50.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:50.973 7fcf1ad73700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 0x7fcf0001ac10 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.004 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c1032e0 con 0x7fcf1c1a1e50 2021-02-10T11:22:51.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.006 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf04000dd0 con 0x7fcf1c1a1e50 2021-02-10T11:22:51.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.141 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7fcf1c09e7c0 con 0x7fcf00018730 2021-02-10T11:22:51.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.143 7fcf13fff700 1 -- 172.21.2.117:0/1671202024 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+77225 (crc 0 0 0) 0x7fcf0c000970 con 0x7fcf00018730 2021-02-10T11:22:51.142 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:51.142 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T11:22:51.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 msgr2=0x7fcf0001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 0x7fcf0001ac10 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 msgr2=0x7fcf1c1a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcf04001c60 tx=0x7fcf0400a8d0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 shutdown_connections 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf1c09e3c0 0x7fcf1c1a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf1c09d9a0 0x7fcf1c09dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf00018730 0x7fcf0001ac10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 --2- 172.21.2.117:0/1671202024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf1c1a1e50 0x7fcf1c1a2250 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.146 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 >> 172.21.2.117:0/1671202024 conn(0x7fcf1c19f4f0 msgr2=0x7fcf1c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.147 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 shutdown_connections 2021-02-10T11:22:51.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.147 7fcf2181f700 1 -- 172.21.2.117:0/1671202024 wait complete. 2021-02-10T11:22:51.151 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2021-02-10 11:22:50.820510","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8390528,"kb_used_data":1856,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584192,"statfs":{"total":773094113280,"available":764502212608,"internally_reserved":8589934592,"allocated":1900544,"data_stored":664500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":760,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001775"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921091","last_change":"2021-02-10 11:22:24.884900","last_active":"2021-02-10 11:22:31.921091","last_peered":"2021-02-10 11:22:31.921091","last_clean":"2021-02-10 11:22:31.921091","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:22:31.921091","last_undegraded":"2021-02-10 11:22:31.921091","last_fullsized":"2021-02-10 11:22:31.921091","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"22'8","reported_seq":"19","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.966869","last_change":"2021-02-10 11:22:28.908117","last_active":"2021-02-10 11:22:40.966869","last_peered":"2021-02-10 11:22:40.966869","last_clean":"2021-02-10 11:22:40.966869","last_became_active":"2021-02-10 11:22:28.907896","last_became_peered":"2021-02-10 11:22:28.907896","last_unstale":"2021-02-10 11:22:40.966869","last_undegraded":"2021-02-10 11:22:40.966869","last_fullsized":"2021-02-10 11:22:40.966869","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.922773","last_change":"2021-02-10 11:22:31.922394","last_active":"2021-02-10 11:22:31.922773","last_peered":"2021-02-10 11:22:31.922773","last_clean":"2021-02-10 11:22:31.922773","last_became_active":"2021-02-10 11:22:31.922009","last_became_peered":"2021-02-10 11:22:31.922009","last_unstale":"2021-02-10 11:22:31.922773","last_undegraded":"2021-02-10 11:22:31.922773","last_fullsized":"2021-02-10 11:22:31.922773","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921167","last_change":"2021-02-10 11:22:24.885206","last_active":"2021-02-10 11:22:31.921167","last_peered":"2021-02-10 11:22:31.921167","last_clean":"2021-02-10 11:22:31.921167","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:22:31.921167","last_undegraded":"2021-02-10 11:22:31.921167","last_fullsized":"2021-02-10 11:22:31.921167","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"22'6","reported_seq":"18","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965769","last_change":"2021-02-10 11:22:28.906683","last_active":"2021-02-10 11:22:40.965769","last_peered":"2021-02-10 11:22:40.965769","last_clean":"2021-02-10 11:22:40.965769","last_became_active":"2021-02-10 11:22:28.906494","last_became_peered":"2021-02-10 11:22:28.906494","last_unstale":"2021-02-10 11:22:40.965769","last_undegraded":"2021-02-10 11:22:40.965769","last_fullsized":"2021-02-10 11:22:40.965769","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.923297","last_change":"2021-02-10 11:22:31.922650","last_active":"2021-02-10 11:22:31.923297","last_peered":"2021-02-10 11:22:31.923297","last_clean":"2021-02-10 11:22:31.923297","last_became_active":"2021-02-10 11:22:31.922394","last_became_peered":"2021-02-10 11:22:31.922394","last_unstale":"2021-02-10 11:22:31.923297","last_undegraded":"2021-02-10 11:22:31.923297","last_fullsized":"2021-02-10 11:22:31.923297","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.925252","last_change":"2021-02-10 11:22:31.924846","last_active":"2021-02-10 11:22:31.925252","last_peered":"2021-02-10 11:22:31.925252","last_clean":"2021-02-10 11:22:31.925252","last_became_active":"2021-02-10 11:22:31.924458","last_became_peered":"2021-02-10 11:22:31.924458","last_unstale":"2021-02-10 11:22:31.925252","last_undegraded":"2021-02-10 11:22:31.925252","last_fullsized":"2021-02-10 11:22:31.925252","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920971","last_change":"2021-02-10 11:22:24.885818","last_active":"2021-02-10 11:22:31.920971","last_peered":"2021-02-10 11:22:31.920971","last_clean":"2021-02-10 11:22:31.920971","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:22:31.920971","last_undegraded":"2021-02-10 11:22:31.920971","last_fullsized":"2021-02-10 11:22:31.920971","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'4","reported_seq":"15","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.966681","last_change":"2021-02-10 11:22:28.907474","last_active":"2021-02-10 11:22:40.966681","last_peered":"2021-02-10 11:22:40.966681","last_clean":"2021-02-10 11:22:40.966681","last_became_active":"2021-02-10 11:22:28.907296","last_became_peered":"2021-02-10 11:22:28.907296","last_unstale":"2021-02-10 11:22:40.966681","last_undegraded":"2021-02-10 11:22:40.966681","last_fullsized":"2021-02-10 11:22:40.966681","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.925431","last_change":"2021-02-10 11:22:31.924942","last_active":"2021-02-10 11:22:31.925431","last_peered":"2021-02-10 11:22:31.925431","last_clean":"2021-02-10 11:22:31.925431","last_became_active":"2021-02-10 11:22:31.924505","last_became_peered":"2021-02-10 11:22:31.924505","last_unstale":"2021-02-10 11:22:31.925431","last_undegraded":"2021-02-10 11:22:31.925431","last_fullsized":"2021-02-10 11:22:31.925431","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"14","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920908","last_change":"2021-02-10 11:22:24.886002","last_active":"2021-02-10 11:22:31.920908","last_peered":"2021-02-10 11:22:31.920908","last_clean":"2021-02-10 11:22:31.920908","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:22:31.920908","last_undegraded":"2021-02-10 11:22:31.920908","last_fullsized":"2021-02-10 11:22:31.920908","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'7","reported_seq":"19","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965894","last_change":"2021-02-10 11:22:28.906256","last_active":"2021-02-10 11:22:40.965894","last_peered":"2021-02-10 11:22:40.965894","last_clean":"2021-02-10 11:22:40.965894","last_became_active":"2021-02-10 11:22:28.906045","last_became_peered":"2021-02-10 11:22:28.906045","last_unstale":"2021-02-10 11:22:40.965894","last_undegraded":"2021-02-10 11:22:40.965894","last_fullsized":"2021-02-10 11:22:40.965894","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921871","last_change":"2021-02-10 11:22:24.885674","last_active":"2021-02-10 11:22:31.921871","last_peered":"2021-02-10 11:22:31.921871","last_clean":"2021-02-10 11:22:31.921871","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:22:31.921871","last_undegraded":"2021-02-10 11:22:31.921871","last_fullsized":"2021-02-10 11:22:31.921871","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":"17","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.964993","last_change":"2021-02-10 11:22:28.905615","last_active":"2021-02-10 11:22:40.964993","last_peered":"2021-02-10 11:22:40.964993","last_clean":"2021-02-10 11:22:40.964993","last_became_active":"2021-02-10 11:22:28.905422","last_became_peered":"2021-02-10 11:22:28.905422","last_unstale":"2021-02-10 11:22:40.964993","last_undegraded":"2021-02-10 11:22:40.964993","last_fullsized":"2021-02-10 11:22:40.964993","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924944","last_change":"2021-02-10 11:22:31.924485","last_active":"2021-02-10 11:22:31.924944","last_peered":"2021-02-10 11:22:31.924944","last_clean":"2021-02-10 11:22:31.924944","last_became_active":"2021-02-10 11:22:31.924090","last_became_peered":"2021-02-10 11:22:31.924090","last_unstale":"2021-02-10 11:22:31.924944","last_undegraded":"2021-02-10 11:22:31.924944","last_fullsized":"2021-02-10 11:22:31.924944","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.919080","last_change":"2021-02-10 11:22:24.885203","last_active":"2021-02-10 11:22:31.919080","last_peered":"2021-02-10 11:22:31.919080","last_clean":"2021-02-10 11:22:31.919080","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:22:31.919080","last_undegraded":"2021-02-10 11:22:31.919080","last_fullsized":"2021-02-10 11:22:31.919080","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":"13","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:32.548975","last_change":"2021-02-10 11:22:28.905680","last_active":"2021-02-10 11:22:32.548975","last_peered":"2021-02-10 11:22:32.548975","last_clean":"2021-02-10 11:22:32.548975","last_became_active":"2021-02-10 11:22:28.905492","last_became_peered":"2021-02-10 11:22:28.905492","last_unstale":"2021-02-10 11:22:32.548975","last_undegraded":"2021-02-10 11:22:32.548975","last_fullsized":"2021-02-10 11:22:32.548975","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.923600","last_change":"2021-02-10 11:22:31.923063","last_active":"2021-02-10 11:22:31.923600","last_peered":"2021-02-10 11:22:31.923600","last_clean":"2021-02-10 11:22:31.923600","last_became_active":"2021-02-10 11:22:31.922699","last_became_peered":"2021-02-10 11:22:31.922699","last_unstale":"2021-02-10 11:22:31.923600","last_undegraded":"2021-02-10 11:22:31.923600","last_fullsized":"2021-02-10 11:22:31.923600","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920500","last_change":"2021-02-10 11:22:28.906735","last_active":"2021-02-10 11:22:31.920500","last_peered":"2021-02-10 11:22:31.920500","last_clean":"2021-02-10 11:22:31.920500","last_became_active":"2021-02-10 11:22:28.906538","last_became_peered":"2021-02-10 11:22:28.906538","last_unstale":"2021-02-10 11:22:31.920500","last_undegraded":"2021-02-10 11:22:31.920500","last_fullsized":"2021-02-10 11:22:31.920500","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.921848","last_change":"2021-02-10 11:22:24.885698","last_active":"2021-02-10 11:22:31.921848","last_peered":"2021-02-10 11:22:31.921848","last_clean":"2021-02-10 11:22:31.921848","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:22:31.921848","last_undegraded":"2021-02-10 11:22:31.921848","last_fullsized":"2021-02-10 11:22:31.921848","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924518","last_change":"2021-02-10 11:22:31.923835","last_active":"2021-02-10 11:22:31.924518","last_peered":"2021-02-10 11:22:31.924518","last_clean":"2021-02-10 11:22:31.924518","last_became_active":"2021-02-10 11:22:31.923471","last_became_peered":"2021-02-10 11:22:31.923471","last_unstale":"2021-02-10 11:22:31.924518","last_undegraded":"2021-02-10 11:22:31.924518","last_fullsized":"2021-02-10 11:22:31.924518","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"22'8","reported_seq":"20","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-10 11:22:40.965939","last_change":"2021-02-10 11:22:28.906488","last_active":"2021-02-10 11:22:40.965939","last_peered":"2021-02-10 11:22:40.965939","last_clean":"2021-02-10 11:22:40.965939","last_became_active":"2021-02-10 11:22:28.906283","last_became_peered":"2021-02-10 11:22:28.906283","last_unstale":"2021-02-10 11:22:40.965939","last_undegraded":"2021-02-10 11:22:40.965939","last_fullsized":"2021-02-10 11:22:40.965939","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:27.892823","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:27.892823","last_clean_scrub_stamp":"2021-02-10 11:22:27.892823","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"14","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.920236","last_change":"2021-02-10 11:22:24.886028","last_active":"2021-02-10 11:22:31.920236","last_peered":"2021-02-10 11:22:31.920236","last_clean":"2021-02-10 11:22:31.920236","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:22:31.920236","last_undegraded":"2021-02-10 11:22:31.920236","last_fullsized":"2021-02-10 11:22:31.920236","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-10 11:22:31.924997","last_change":"2021-02-10 11:22:31.924599","last_active":"2021-02-10 11:22:31.924997","last_peered":"2021-02-10 11:22:31.924997","last_clean":"2021-02-10 11:22:31.924997","last_became_active":"2021-02-10 11:22:31.924231","last_became_peered":"2021-02-10 11:22:31.924231","last_unstale":"2021-02-10 11:22:31.924997","last_undegraded":"2021-02-10 11:22:31.924997","last_fullsized":"2021-02-10 11:22:31.924997","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:30.911788","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:30.911788","last_clean_scrub_stamp":"2021-02-10 11:22:30.911788","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048824,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323016,"statfs":{"total":96636764160,"available":95562768384,"internally_reserved":1073741824,"allocated":245760,"data_stored":83217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048816,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323024,"statfs":{"total":96636764160,"available":95562776576,"internally_reserved":1073741824,"allocated":237568,"data_stored":83057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1359999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":82525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":82491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048828,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323012,"statfs":{"total":96636764160,"available":95562764288,"internally_reserved":1073741824,"allocated":249856,"data_stored":84177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048820,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323020,"statfs":{"total":96636764160,"available":95562772480,"internally_reserved":1073741824,"allocated":241664,"data_stored":84017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":82525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":82491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]}]}]}} 2021-02-10T11:22:51.153 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph status --format=json-pretty 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 Processor -- start 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 -- start start 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f278807da20 0x7f27880771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2788077a10 con 0x7f27880e4df0 2021-02-10T11:22:51.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27881b0910 con 0x7f27880e51f0 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.565 7f278e96c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880777d0 con 0x7f278807da20 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2787fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f278807da20 0x7f27880771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f27877fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f27880e4df0 0x7f27881b0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:41998/0 (socket says 172.21.2.117:41998) 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f27877fe700 1 -- 172.21.2.117:0/2739765274 learned_addr learned my addr 172.21.2.117:0/2739765274 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2787fff700 1 -- 172.21.2.117:0/2739765274 >> v1:172.21.2.125:6789/0 conn(0x7f27880e51f0 legacy=0x7f27880d0540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2787fff700 1 -- 172.21.2.117:0/2739765274 >> v1:172.21.2.117:6789/0 conn(0x7f27880e4df0 legacy=0x7f27881b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2787fff700 1 -- 172.21.2.117:0/2739765274 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880d1d40 con 0x7f278807da20 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2787fff700 1 --2- 172.21.2.117:0/2739765274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278807da20 0x7f27880771b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2778006650 tx=0x7f2778009a80).ready entity=mon.0 client_cookie=c850cd4d406d1d38 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2786ffd700 1 -- 172.21.2.117:0/2739765274 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f277800c0c0 con 0x7f278807da20 2021-02-10T11:22:51.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2786ffd700 1 -- 172.21.2.117:0/2739765274 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277800c340 con 0x7f278807da20 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f2786ffd700 1 -- 172.21.2.117:0/2739765274 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f277800c860 con 0x7f278807da20 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f278e96c700 1 -- 172.21.2.117:0/2739765274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278807da20 msgr2=0x7f27880771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.566 7f278e96c700 1 --2- 172.21.2.117:0/2739765274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278807da20 0x7f27880771b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2778006650 tx=0x7f2778009a80).stop 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.567 7f278e96c700 1 -- 172.21.2.117:0/2739765274 shutdown_connections 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.567 7f278e96c700 1 --2- 172.21.2.117:0/2739765274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278807da20 0x7f27880771b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.567 7f278e96c700 1 -- 172.21.2.117:0/2739765274 >> 172.21.2.117:0/2739765274 conn(0x7f27880ce470 msgr2=0x7f27880e42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.567 7f278e96c700 1 -- 172.21.2.117:0/2739765274 shutdown_connections 2021-02-10T11:22:51.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.567 7f278e96c700 1 -- 172.21.2.117:0/2739765274 wait complete. 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 Processor -- start 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 -- start start 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27880b5b90 0x7f27880c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27880c3310 0x7f27881c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2788103ee0 con 0x7f27880b68b0 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880b5f90 con 0x7f27880c3310 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f278e96c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27880c3710 con 0x7f27880b5b90 2021-02-10T11:22:51.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f2787fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.582 7f2787fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41756/0 (socket says 172.21.2.117:41756) 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 -- 172.21.2.117:0/45223818 learned_addr learned my addr 172.21.2.117:0/45223818 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 -- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27880b5b90 msgr2=0x7f27880c2cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27880b5b90 0x7f27880c2cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 -- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27880c3310 msgr2=0x7f27881c4cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27880c3310 0x7f27881c4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 -- 172.21.2.117:0/45223818 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27881c5310 con 0x7f27880b68b0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2787fff700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2778009630 tx=0x7f277800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27780093a0 con 0x7f27880b68b0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277800bd70 con 0x7f27880b68b0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f278e96c700 1 -- 172.21.2.117:0/45223818 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27881c5310 con 0x7f27880b68b0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27780054d0 con 0x7f27880b68b0 2021-02-10T11:22:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.583 7f278e96c700 1 -- 172.21.2.117:0/45223818 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27881c58a0 con 0x7f27880b68b0 2021-02-10T11:22:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.584 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2778007020 con 0x7f27880b68b0 2021-02-10T11:22:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.584 7f2784ff9700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 0x7f276801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.584 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f27780327b0 con 0x7f27880b68b0 2021-02-10T11:22:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.584 7f27877fe700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 0x7f276801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.585 7f27877fe700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 0x7f276801ac10 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:51.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.615 7f278e96c700 1 -- 172.21.2.117:0/45223818 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2788102b00 con 0x7f27880b68b0 2021-02-10T11:22:51.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.618 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2778009340 con 0x7f27880b68b0 2021-02-10T11:22:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.749 7f278e96c700 1 -- 172.21.2.117:0/45223818 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f27880ce470 con 0x7f27880b68b0 2021-02-10T11:22:51.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.751 7f2784ff9700 1 -- 172.21.2.117:0/45223818 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+211028 (secure 0 0 0) 0x7f2778007000 con 0x7f27880b68b0 2021-02-10T11:22:51.751 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:51.751 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:22:51.751 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:22:51.751 INFO:teuthology.orchestra.run.gibba017.stdout: "health": { 2021-02-10T11:22:51.751 INFO:teuthology.orchestra.run.gibba017.stdout: "checks": {}, 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "status": "HEALTH_OK" 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "election_epoch": 4, 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "quorum": [ 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: 0, 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: 1, 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: 2 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "quorum_names": [ 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "a", 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "b", 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: "c" 2021-02-10T11:22:51.752 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "quorum_age": 64, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "monmap": { 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 1, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "min_mon_release_name": "14", 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "num_mons": 3 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap": { 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap": { 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 22, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "num_osds": 8, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "num_up_osds": 8, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "num_in_osds": 8, 2021-02-10T11:22:51.753 INFO:teuthology.orchestra.run.gibba017.stdout: "num_remapped_pgs": 0 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "pgmap": { 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "pgs_by_state": [ 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "state_name": "active+clean", 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "count": 24 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "num_pgs": 24, 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "num_pools": 3, 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "num_objects": 22, 2021-02-10T11:22:51.754 INFO:teuthology.orchestra.run.gibba017.stdout: "data_bytes": 2286, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_used": 8591900672, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_avail": 764502212608, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_total": 773094113280, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "write_bytes_sec": 1279, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "read_op_per_sec": 0, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "write_op_per_sec": 4 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "fsmap": { 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 6, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 1, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "max": 1, 2021-02-10T11:22:51.755 INFO:teuthology.orchestra.run.gibba017.stdout: "by_rank": [ 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "filesystem_id": 1, 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "rank": 0, 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "b", 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "status": "up:active", 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "gid": 4229 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "up:standby": 1 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "mgrmap": { 2021-02-10T11:22:51.756 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 3, 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "active_gid": 4100, 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "active_name": "x", 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "active_addrs": { 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6800", 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 34993 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6801", 2021-02-10T11:22:51.757 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 34993 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "active_addr": "172.21.2.125:6801/34993", 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "active_change": "2021-02-10 11:21:48.805950", 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "available": true, 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "standbys": [ 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "gid": 4107, 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "y", 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "available_modules": [ 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "alerts", 2021-02-10T11:22:51.758 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:22:51.759 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_destination": { 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_destination", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:22:51.760 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_from_name": { 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_from_name", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "Ceph", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.761 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Email From: name", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_host": { 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_host", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.762 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP server", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_password": { 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_password", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.763 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Password to authenticate with", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_port": { 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_port", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "465", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.764 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP port", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_sender": { 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_sender", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.765 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_ssl": { 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_ssl", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.766 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_user": { 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_user", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.767 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "User to authenticate as", 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.768 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ansible", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.769 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "server_url": { 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_url", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.770 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.771 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "verify_server": { 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "verify_server", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "balancer", 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "active": { 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "active", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "begin_time": { 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "begin_time", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0000", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:22:51.775 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "begin_weekday": { 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "begin_weekday", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "7", 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.776 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_max_iterations": { 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "25", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "1", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "250", 2021-02-10T11:22:51.777 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_metrics": { 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_metrics", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.778 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.779 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_step": { 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_step", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0.5", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0.001", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "0.999", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:22:51.780 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "end_time": { 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "end_time", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "2400", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.781 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "end_weekday": { 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "end_weekday", 2021-02-10T11:22:51.782 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "7", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "7", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.783 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "min_score": { 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "min_score", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:22:51.784 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mode": { 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "mode", 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.785 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "none", 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [ 2021-02-10T11:22:51.786 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-compat", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "none", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap" 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Balancer mode", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_ids": { 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pool_ids", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.787 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.788 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap_max_deviation": { 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "upmap_max_deviation", 2021-02-10T11:22:51.789 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "1", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap_max_iterations": { 2021-02-10T11:22:51.790 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "upmap_max_iterations", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "10", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.791 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crash", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "retain_interval": { 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "retain_interval", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "31536000", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.792 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "warn_recent_interval": { 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "warn_recent_interval", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "1209600", 2021-02-10T11:22:51.793 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "dashboard", 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.794 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.795 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.796 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.797 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.798 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.799 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.800 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.801 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.802 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.803 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.804 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.805 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:22:51.806 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_URL": { 2021-02-10T11:22:51.807 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:22:51.808 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.809 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.810 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.811 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.812 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.813 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "45", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.814 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.815 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.816 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_HOST": { 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_HOST", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.817 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_PORT": { 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_PORT", 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.818 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "80", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SCHEME": { 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.819 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "http", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:22:51.820 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:22:51.821 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.822 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_USER_ID": { 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.823 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "crt_file": { 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crt_file", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.824 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "debug": { 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "debug", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.825 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "jwt_token_ttl": { 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "jwt_token_ttl", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "28800", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.826 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "key_file": { 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "key_file", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.827 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.828 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "::", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.829 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8080", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.830 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl": { 2021-02-10T11:22:51.831 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.832 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl_server_port": { 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl_server_port", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8443", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.833 INFO:teuthology.orchestra.run.gibba017.stdout: "standby_behaviour": { 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "standby_behaviour", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "redirect", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [ 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "error", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "redirect" 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.834 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "standby_error_status_code": { 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "standby_error_status_code", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "500", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "400", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "599", 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.835 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "url_prefix": { 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "url_prefix", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.836 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.837 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "deepsea", 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.838 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_eauth": { 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_eauth", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "sharedsecret", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.839 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_password": { 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_password", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.840 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_url": { 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_url", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.841 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_username": { 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_username", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.842 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "devicehealth", 2021-02-10T11:22:51.843 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "enable_monitoring": { 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enable_monitoring", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "monitor device health metrics", 2021-02-10T11:22:51.844 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "mark_out_threshold": { 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "mark_out_threshold", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "2419200", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.845 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": { 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pool_name", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "device_health_metrics", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.846 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "retention_period": { 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "retention_period", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "15552000", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.847 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "scrape_frequency": { 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "scrape_frequency", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "86400", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.848 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "self_heal": { 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "self_heal", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.849 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.850 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "warn_threshold": { 2021-02-10T11:22:51.851 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "warn_threshold", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "7257600", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.852 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_cloud", 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_cert_context": { 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.853 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_default_authority": { 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:22:51.854 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "localhost", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.855 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_password": { 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_password", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.856 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_port": { 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_port", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "31400", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.857 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "43200", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.858 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_server": { 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_server", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.859 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "localhost", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.860 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.861 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "43200", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.862 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_user": { 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_user", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.863 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.864 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_local", 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.865 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "predict_interval": { 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "predict_interval", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "86400", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.866 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.867 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "influx", 2021-02-10T11:22:51.868 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": false, 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "batch_size": { 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "batch_size", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5000", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.869 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "database": { 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "database", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "ceph", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.870 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": { 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "hostname", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.871 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "30", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.872 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.873 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "port": { 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "port", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8086", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.874 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl": { 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "false", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.875 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "threads": { 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "threads", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.876 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.877 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "verify_ssl": { 2021-02-10T11:22:51.878 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "verify_ssl", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "true", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.879 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "insights", 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "iostat", 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.880 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "localpool", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "failure_domain": { 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "failure_domain", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "host", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.881 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": { 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "min_size", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.882 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "num_rep": { 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "num_rep", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "3", 2021-02-10T11:22:51.883 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": { 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pg_num", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "128", 2021-02-10T11:22:51.884 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "prefix": { 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "prefix", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.885 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "subtree": { 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "subtree", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.886 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "rack", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "orchestrator_cli", 2021-02-10T11:22:51.887 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "orchestrator": { 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "orchestrator", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.888 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pg_autoscaler", 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.889 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.890 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "progress", 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "enabled": { 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enabled", 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.891 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "max_completed_events": { 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "max_completed_events", 2021-02-10T11:22:51.892 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "50", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "persist_interval": { 2021-02-10T11:22:51.893 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "persist_interval", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.894 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "prometheus", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd_stats_pools": { 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_stats_pools", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.895 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "300", 2021-02-10T11:22:51.896 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "scrape_interval": { 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "scrape_interval", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.897 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.898 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.899 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "stale_cache_strategy": { 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "stale_cache_strategy", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.900 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.901 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_support", 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "restful", 2021-02-10T11:22:51.902 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "key_file": { 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "key_file", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.903 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.904 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.905 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.906 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rook", 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "selftest", 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "roption1": { 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "roption1", 2021-02-10T11:22:51.907 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "roption2": { 2021-02-10T11:22:51.908 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "roption2", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "xyz", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.909 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption1": { 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption1", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.910 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption2": { 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption2", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.911 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption3": { 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption3", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.912 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption4": { 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption4", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.913 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption5": { 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption5", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.914 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption6": { 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption6", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.915 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.916 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "testkey": { 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testkey", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.917 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "testlkey": { 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testlkey", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.918 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "testnewline": { 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testnewline", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.919 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssh", 2021-02-10T11:22:51.920 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.921 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "ssh_config_file": { 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssh_config_file", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.922 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "status", 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.923 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "telegraf", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "address": { 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "address", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.924 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "15", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.925 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "telemetry", 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.926 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_basic": { 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_basic", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:22:51.927 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_crash": { 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_crash", 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.928 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_device": { 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_device", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.929 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.930 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_ident": { 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_ident", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.931 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "contact": { 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "contact", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.932 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "description": { 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "description", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.933 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "device_url": { 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "device_url", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.934 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "enabled": { 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enabled", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.935 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "24", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "8", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.936 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "last_opt_revision": { 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "last_opt_revision", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "1", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.937 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "leaderboard": { 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "leaderboard", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.938 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "organization": { 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "organization", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.939 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.940 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "proxy": { 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "proxy", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.941 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "url": { 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "url", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.942 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "test_orchestrator", 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.943 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "volumes", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "max_concurrent_clones": { 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "max_concurrent_clones", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "4", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.944 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix", 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "identifier": { 2021-02-10T11:22:51.945 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "identifier", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.946 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.947 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_host": { 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_host", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.948 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_port": { 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_port", 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "10051", 2021-02-10T11:22:51.949 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_sender": { 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_sender", 2021-02-10T11:22:51.950 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.951 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: "modules": [ 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: "iostat", 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: "restful" 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.952 INFO:teuthology.orchestra.run.gibba017.stdout: "available_modules": [ 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "alerts", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.953 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "How frequently to reexamine health status", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_destination": { 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_destination", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.954 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Email address to send alerts to", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_from_name": { 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_from_name", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.955 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "Ceph", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Email From: name", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_host": { 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_host", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.956 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP server", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_password": { 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_password", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.957 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Password to authenticate with", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_port": { 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_port", 2021-02-10T11:22:51.958 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "465", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP port", 2021-02-10T11:22:51.959 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_sender": { 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_sender", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.960 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "SMTP envelope sender", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_ssl": { 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_ssl", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.961 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Use SSL to connect to SMTP server", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "smtp_user": { 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "smtp_user", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.962 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "User to authenticate as", 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ansible", 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.963 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.964 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "server_url": { 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_url", 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.965 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.966 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "verify_server": { 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "verify_server", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.967 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.968 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "balancer", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "active": { 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "active", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.969 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "automatically balance PGs across cluster", 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "begin_time": { 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "begin_time", 2021-02-10T11:22:51.970 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0000", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "beginning time of day to automatically balance", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "begin_weekday": { 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "begin_weekday", 2021-02-10T11:22:51.971 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "7", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_max_iterations": { 2021-02-10T11:22:51.972 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_max_iterations", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "25", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "1", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "250", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "maximum number of iterations to attempt optimization", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.973 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_metrics": { 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_metrics", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "pgs,objects,bytes", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "metrics with which to calculate OSD utilization", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.974 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_compat_step": { 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crush_compat_step", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0.5", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0.001", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "0.999", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "aggressiveness of optimization", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.975 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "end_time": { 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "end_time", 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "2400", 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.976 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "ending time of day to automatically balance", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "This is a time of day in the format HHMM.", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "end_weekday": { 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "end_weekday", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "7", 2021-02-10T11:22:51.977 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "0", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "7", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "0 or 7 = Sunday, 1 = Monday, etc.", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "min_score": { 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "min_score", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.978 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "0", 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "minimum score, below which no optimization is attempted", 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "mode": { 2021-02-10T11:22:51.979 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "mode", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "none", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [ 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-compat", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "none", 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap" 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:51.980 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Balancer mode", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_ids": { 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pool_ids", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.981 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "pools which the automatic balancing will be limited to", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.982 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to wake up and attempt optimization", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap_max_deviation": { 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "upmap_max_deviation", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:51.983 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "1", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "deviation below which no optimization is attempted", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "upmap_max_iterations": { 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "upmap_max_iterations", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "uint", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.984 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "10", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "maximum upmap optimization iterations", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crash", 2021-02-10T11:22:51.985 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "retain_interval": { 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "retain_interval", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "31536000", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how long to retain crashes before pruning them", 2021-02-10T11:22:51.986 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "warn_recent_interval": { 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "warn_recent_interval", 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "1209600", 2021-02-10T11:22:51.987 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "time interval in which to warn about recent crashes", 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.988 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "dashboard", 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "ALERTMANAGER_API_HOST": { 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ALERTMANAGER_API_HOST", 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.989 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "AUDIT_API_ENABLED": { 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "AUDIT_API_ENABLED", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.990 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "AUDIT_API_LOG_PAYLOAD": { 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.991 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "ENABLE_BROWSABLE_API": { 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ENABLE_BROWSABLE_API", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.992 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_cephfs": { 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_cephfs", 2021-02-10T11:22:51.993 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_iscsi": { 2021-02-10T11:22:51.994 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_iscsi", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.995 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_mirroring": { 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_mirroring", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_rbd": { 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_rbd", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.997 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "FEATURE_TOGGLE_rgw": { 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "FEATURE_TOGGLE_rgw", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:51.998 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:51.999 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_PASSWORD": { 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_PASSWORD", 2021-02-10T11:22:52.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.001 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_SSL_VERIFY": { 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.002 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_URL": { 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_URL", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.003 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_API_USERNAME": { 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_API_USERNAME", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.004 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.005 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "ISCSI_API_SSL_VERIFICATION": { 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.006 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "PROMETHEUS_API_HOST": { 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "PROMETHEUS_API_HOST", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.007 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "REST_REQUESTS_TIMEOUT": { 2021-02-10T11:22:52.008 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "REST_REQUESTS_TIMEOUT", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "45", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.009 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_ACCESS_KEY": { 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_ACCESS_KEY", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.010 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_ADMIN_RESOURCE": { 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "admin", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_HOST": { 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_HOST", 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_PORT": { 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_PORT", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "80", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SCHEME": { 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SCHEME", 2021-02-10T11:22:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "http", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SECRET_KEY": { 2021-02-10T11:22:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SECRET_KEY", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_SSL_VERIFY": { 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_SSL_VERIFY", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "RGW_API_USER_ID": { 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "RGW_API_USER_ID", 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "crt_file": { 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "crt_file", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "debug": { 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "debug", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Enable/disable debug options", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "jwt_token_ttl": { 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "jwt_token_ttl", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "28800", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "key_file": { 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "key_file", 2021-02-10T11:22:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "::", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8080", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl": { 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl_server_port": { 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl_server_port", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8443", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "standby_behaviour": { 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "standby_behaviour", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "redirect", 2021-02-10T11:22:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [ 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "error", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "redirect" 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "standby_error_status_code": { 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "standby_error_status_code", 2021-02-10T11:22:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "500", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "400", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "599", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "url_prefix": { 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "url_prefix", 2021-02-10T11:22:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "deepsea", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_eauth": { 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_eauth", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "sharedsecret", 2021-02-10T11:22:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_password": { 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_password", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_url": { 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_url", 2021-02-10T11:22:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "salt_api_username": { 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "salt_api_username", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "devicehealth", 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "enable_monitoring": { 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enable_monitoring", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "monitor device health metrics", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "mark_out_threshold": { 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "mark_out_threshold", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "2419200", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "automatically mark OSD if it may fail before this long", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": { 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pool_name", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "device_health_metrics", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "name of pool in which to store device health metrics", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "retention_period": { 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "retention_period", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "15552000", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how long to retain device health metrics", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "scrape_frequency": { 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "scrape_frequency", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "86400", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to scrape device health metrics", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "self_heal": { 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "self_heal", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "preemptively heal cluster around devices that may fail", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to wake up and check device health", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "warn_threshold": { 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "warn_threshold", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "7257600", 2021-02-10T11:22:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "raise health warning if OSD may fail before this long", 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_cloud", 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_cert_context": { 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_cert_context", 2021-02-10T11:22:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_default_authority": { 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_default_authority", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "localhost", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_password": { 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_password", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_port": { 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_port", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "31400", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_retrieve_prediction_interval": { 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_retrieve_prediction_interval", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "43200", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_server": { 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_server", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_ssl_target_name_override": { 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_ssl_target_name_override", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "localhost", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_upload_metrics_interval": { 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_upload_metrics_interval", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_upload_smart_interval": { 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_upload_smart_interval", 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "43200", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "diskprediction_user": { 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_user", 2021-02-10T11:22:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "diskprediction_local", 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "predict_interval": { 2021-02-10T11:22:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "predict_interval", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "86400", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "600", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "influx", 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": false, 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "influxdb python module not found", 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "batch_size": { 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "batch_size", 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5000", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "database": { 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "database", 2021-02-10T11:22:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "ceph", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": { 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "hostname", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "30", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "password": { 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "password", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "port": { 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "port", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "8086", 2021-02-10T11:22:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "ssl": { 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssl", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "false", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "threads": { 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "threads", 2021-02-10T11:22:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "username": { 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "username", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "verify_ssl": { 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "verify_ssl", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "true", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "insights", 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "iostat", 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "localpool", 2021-02-10T11:22:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "failure_domain": { 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "failure_domain", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "host", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "failure domain for any created local pool", 2021-02-10T11:22:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": { 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "min_size", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default min_size for any created local pool", 2021-02-10T11:22:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "num_rep": { 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "num_rep", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "3", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default replica count for any created local pool", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": { 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pg_num", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "128", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "default pg_num for any created local pool", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "prefix": { 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "prefix", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "name prefix for any created local pool", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "subtree": { 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "subtree", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "rack", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "CRUSH level for which to create a local pool", 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "orchestrator_cli", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "orchestrator": { 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "orchestrator", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.087 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "pg_autoscaler", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "sleep_interval": { 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "sleep_interval", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:52.088 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "progress", 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.089 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "enabled": { 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enabled", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.090 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "max_completed_events": { 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "max_completed_events", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "50", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "number of past completed events to remember", 2021-02-10T11:22:52.091 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "persist_interval": { 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "persist_interval", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "5", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.092 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "how frequently to persist completed events", 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "prometheus", 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd_stats_pools": { 2021-02-10T11:22:52.093 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_stats_pools", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.094 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd_stats_pools_refresh_interval": { 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_stats_pools_refresh_interval", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "300", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.095 INFO:teuthology.orchestra.run.gibba017.stdout: "scrape_interval": { 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "scrape_interval", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.096 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.097 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.098 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "stale_cache_strategy": { 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "stale_cache_strategy", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.099 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.100 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rbd_support", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "restful", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "key_file": { 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "key_file", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "server_addr": { 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_addr", 2021-02-10T11:22:52.102 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "server_port": { 2021-02-10T11:22:52.103 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "server_port", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.104 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rook", 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "selftest", 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "roption1": { 2021-02-10T11:22:52.105 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "roption1", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.106 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "roption2": { 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "roption2", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "xyz", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.107 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption1": { 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption1", 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.108 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption2": { 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption2", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.109 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.110 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption3": { 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption3", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "float", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.111 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption4": { 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption4", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.112 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption5": { 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption5", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.113 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "rwoption6": { 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "rwoption6", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.114 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "testkey": { 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testkey", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.115 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "testlkey": { 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testlkey", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.116 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "testnewline": { 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "testnewline", 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.117 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.118 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssh", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "inventory_cache_timeout_min": { 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "inventory_cache_timeout_min", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.119 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "ssh_config_file": { 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "ssh_config_file", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.120 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "status", 2021-02-10T11:22:52.121 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "telegraf", 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.122 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "address": { 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "address", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.123 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "15", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.124 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "telemetry", 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_basic": { 2021-02-10T11:22:52.125 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_basic", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Share basic cluster information (size, version)", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.126 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_crash": { 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_crash", 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.127 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_device": { 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_device", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.128 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "True", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "channel_ident": { 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "channel_ident", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.129 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "contact": { 2021-02-10T11:22:52.130 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "contact", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.131 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "description": { 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "description", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.132 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "device_url": { 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "device_url", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "https://telemetry.ceph.com/device", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.133 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "enabled": { 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "enabled", 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.134 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.135 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "24", 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "8", 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "last_opt_revision": { 2021-02-10T11:22:52.136 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "last_opt_revision", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "1", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.137 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "leaderboard": { 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "leaderboard", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "bool", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "False", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.138 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "organization": { 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "organization", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.139 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "proxy": { 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "proxy", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.140 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "url": { 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "url", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "https://telemetry.ceph.com/report", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.141 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "test_orchestrator", 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.142 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": {} 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "volumes", 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "max_concurrent_clones": { 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "max_concurrent_clones", 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.143 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "4", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "Number of asynchronous cloner threads", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.144 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "can_run": true, 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "error_string": "", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "module_options": { 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "identifier": { 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "identifier", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "", 2021-02-10T11:22:52.145 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "interval": { 2021-02-10T11:22:52.146 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "interval", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "secs", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "60", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.147 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_host": { 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_host", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "None", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.148 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_port": { 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_port", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "int", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "10051", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.149 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "zabbix_sender": { 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "zabbix_sender", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "str", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "level": "advanced", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 0, 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "default_value": "/usr/bin/zabbix_sender", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "min": "", 2021-02-10T11:22:52.150 INFO:teuthology.orchestra.run.gibba017.stdout: "max": "", 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "enum_allowed": [], 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "desc": "", 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "long_desc": "", 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "tags": [], 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "see_also": [] 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "services": {}, 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "always_on_modules": { 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "nautilus": [ 2021-02-10T11:22:52.151 INFO:teuthology.orchestra.run.gibba017.stdout: "balancer", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "crash", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "devicehealth", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "orchestrator_cli", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "progress", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd_support", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "status", 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "volumes" 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "servicemap": { 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 2, 2021-02-10T11:22:52.152 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:22:48.820001", 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "services": { 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": { 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "daemons": { 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "summary": "", 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "b": { 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "start_epoch": 0, 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "start_stamp": "0.000000", 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "gid": 0, 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": ":/0", 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": {}, 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "task_status": { 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: "scrub status": "idle" 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.153 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout: "progress_events": {} 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 -- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 msgr2=0x7f276801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 0x7f276801ac10 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 -- 172.21.2.117:0/45223818 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 msgr2=0x7f27880b5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2778009630 tx=0x7f277800b980).stop 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 -- 172.21.2.117:0/45223818 shutdown_connections 2021-02-10T11:22:52.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27880c3310 0x7f27881c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27880b5b90 0x7f27880c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2768018730 0x7f276801ac10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 --2- 172.21.2.117:0/45223818 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27880b68b0 0x7f27880b5570 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 -- 172.21.2.117:0/45223818 >> 172.21.2.117:0/45223818 conn(0x7f278809f860 msgr2=0x7f2788104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.754 7f278e96c700 1 -- 172.21.2.117:0/45223818 shutdown_connections 2021-02-10T11:22:52.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:51.755 7f278e96c700 1 -- 172.21.2.117:0/45223818 wait complete. 2021-02-10T11:22:52.158 INFO:tasks.ceph.ceph_manager.ceph:{'fsid': '722ef60b-feda-4d6a-867f-a5d7f666055b', 'health': {'checks': {}, 'status': 'HEALTH_OK'}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 64, 'monmap': {'epoch': 1, 'min_mon_release_name': '14', 'num_mons': 3}, 'osdmap': {'osdmap': {'epoch': 22, 'num_osds': 8, 'num_up_osds': 8, 'num_in_osds': 8, 'num_remapped_pgs': 0}}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 24}], 'num_pgs': 24, 'num_pools': 3, 'num_objects': 22, 'data_bytes': 2286, 'bytes_used': 8591900672, 'bytes_avail': 764502212608, 'bytes_total': 773094113280, 'write_bytes_sec': 1279, 'read_op_per_sec': 0, 'write_op_per_sec': 4}, 'fsmap': {'epoch': 6, 'id': 1, 'up': 1, 'in': 1, 'max': 1, 'by_rank': [{'filesystem_id': 1, 'rank': 0, 'name': 'b', 'status': 'up:active', 'gid': 4229}], 'up:standby': 1}, 'mgrmap': {'epoch': 3, 'active_gid': 4100, 'active_name': 'x', 'active_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6800', 'nonce': 34993}, {'type': 'v1', 'addr': '172.21.2.125:6801', 'nonce': 34993}]}, 'active_addr': '172.21.2.125:6801/34993', 'active_change': '2021-02-10 11:21:48.805950', 'available': True, 'standbys': [{'gid': 4107, 'name': 'y', 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}]}], 'modules': ['iostat', 'restful'], 'available_modules': [{'name': 'alerts', 'can_run': True, 'error_string': '', 'module_options': {'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'How frequently to reexamine health status', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_destination': {'name': 'smtp_destination', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email address to send alerts to', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_from_name': {'name': 'smtp_from_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'Ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Email From: name', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_host': {'name': 'smtp_host', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_password': {'name': 'smtp_password', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Password to authenticate with', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_port': {'name': 'smtp_port', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '465', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP port', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_sender': {'name': 'smtp_sender', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'SMTP envelope sender', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_ssl': {'name': 'smtp_ssl', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Use SSL to connect to SMTP server', 'long_desc': '', 'tags': [], 'see_also': []}, 'smtp_user': {'name': 'smtp_user', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'User to authenticate as', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ansible', 'can_run': True, 'error_string': '', 'module_options': {'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_url': {'name': 'server_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_server': {'name': 'verify_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'balancer', 'can_run': True, 'error_string': '', 'module_options': {'active': {'name': 'active', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically balance PGs across cluster', 'long_desc': '', 'tags': [], 'see_also': []}, 'begin_time': {'name': 'begin_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '0000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'beginning time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'begin_weekday': {'name': 'begin_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to this day of the week or later', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'crush_compat_max_iterations': {'name': 'crush_compat_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '25', 'min': '1', 'max': '250', 'enum_allowed': [], 'desc': 'maximum number of iterations to attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'crush_compat_metrics': {'name': 'crush_compat_metrics', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'pgs,objects,bytes', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'metrics with which to calculate OSD utilization', 'long_desc': 'Value is a list of one or more of "pgs", "objects", or "bytes", and indicates which metrics to use to balance utilization.', 'tags': [], 'see_also': []}, 'crush_compat_step': {'name': 'crush_compat_step', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0.5', 'min': '0.001', 'max': '0.999', 'enum_allowed': [], 'desc': 'aggressiveness of optimization', 'long_desc': '.99 is very aggressive, .01 is less aggressive', 'tags': [], 'see_also': []}, 'end_time': {'name': 'end_time', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '2400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'ending time of day to automatically balance', 'long_desc': 'This is a time of day in the format HHMM.', 'tags': [], 'see_also': []}, 'end_weekday': {'name': 'end_weekday', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '7', 'min': '0', 'max': '7', 'enum_allowed': [], 'desc': 'Restrict automatic balancing to days of the week earlier than this', 'long_desc': '0 or 7 = Sunday, 1 = Monday, etc.', 'tags': [], 'see_also': []}, 'min_score': {'name': 'min_score', 'type': 'float', 'level': 'advanced', 'flags': 1, 'default_value': '0', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'minimum score, below which no optimization is attempted', 'long_desc': '', 'tags': [], 'see_also': []}, 'mode': {'name': 'mode', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'none', 'min': '', 'max': '', 'enum_allowed': ['crush-compat', 'none', 'upmap'], 'desc': 'Balancer mode', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_ids': {'name': 'pool_ids', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'pools which the automatic balancing will be limited to', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and attempt optimization', 'long_desc': '', 'tags': [], 'see_also': []}, 'upmap_max_deviation': {'name': 'upmap_max_deviation', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '1', 'max': '', 'enum_allowed': [], 'desc': 'deviation below which no optimization is attempted', 'long_desc': 'If the number of PGs are within this count then no optimization is attempted', 'tags': [], 'see_also': []}, 'upmap_max_iterations': {'name': 'upmap_max_iterations', 'type': 'uint', 'level': 'advanced', 'flags': 1, 'default_value': '10', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'maximum upmap optimization iterations', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'crash', 'can_run': True, 'error_string': '', 'module_options': {'retain_interval': {'name': 'retain_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '31536000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain crashes before pruning them', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_recent_interval': {'name': 'warn_recent_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '1209600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'time interval in which to warn about recent crashes', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'dashboard', 'can_run': True, 'error_string': '', 'module_options': {'ALERTMANAGER_API_HOST': {'name': 'ALERTMANAGER_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_ENABLED': {'name': 'AUDIT_API_ENABLED', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'AUDIT_API_LOG_PAYLOAD': {'name': 'AUDIT_API_LOG_PAYLOAD', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ENABLE_BROWSABLE_API': {'name': 'ENABLE_BROWSABLE_API', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_cephfs': {'name': 'FEATURE_TOGGLE_cephfs', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_iscsi': {'name': 'FEATURE_TOGGLE_iscsi', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_mirroring': {'name': 'FEATURE_TOGGLE_mirroring', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rbd': {'name': 'FEATURE_TOGGLE_rbd', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'FEATURE_TOGGLE_rgw': {'name': 'FEATURE_TOGGLE_rgw', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE': {'name': 'GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_PASSWORD': {'name': 'GRAFANA_API_PASSWORD', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_SSL_VERIFY': {'name': 'GRAFANA_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_URL': {'name': 'GRAFANA_API_URL', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_API_USERNAME': {'name': 'GRAFANA_API_USERNAME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'GRAFANA_UPDATE_DASHBOARDS': {'name': 'GRAFANA_UPDATE_DASHBOARDS', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ISCSI_API_SSL_VERIFICATION': {'name': 'ISCSI_API_SSL_VERIFICATION', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'PROMETHEUS_API_HOST': {'name': 'PROMETHEUS_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'REST_REQUESTS_TIMEOUT': {'name': 'REST_REQUESTS_TIMEOUT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '45', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ACCESS_KEY': {'name': 'RGW_API_ACCESS_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_ADMIN_RESOURCE': {'name': 'RGW_API_ADMIN_RESOURCE', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'admin', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_HOST': {'name': 'RGW_API_HOST', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_PORT': {'name': 'RGW_API_PORT', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '80', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SCHEME': {'name': 'RGW_API_SCHEME', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'http', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SECRET_KEY': {'name': 'RGW_API_SECRET_KEY', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_SSL_VERIFY': {'name': 'RGW_API_SSL_VERIFY', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'RGW_API_USER_ID': {'name': 'RGW_API_USER_ID', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'crt_file': {'name': 'crt_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'debug': {'name': 'debug', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Enable/disable debug options', 'long_desc': '', 'tags': [], 'see_also': []}, 'jwt_token_ttl': {'name': 'jwt_token_ttl', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '28800', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '::', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8080', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl_server_port': {'name': 'ssl_server_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '8443', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_behaviour': {'name': 'standby_behaviour', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'redirect', 'min': '', 'max': '', 'enum_allowed': ['error', 'redirect'], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'standby_error_status_code': {'name': 'standby_error_status_code', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '500', 'min': '400', 'max': '599', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url_prefix': {'name': 'url_prefix', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'deepsea', 'can_run': True, 'error_string': '', 'module_options': {'salt_api_eauth': {'name': 'salt_api_eauth', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'sharedsecret', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_password': {'name': 'salt_api_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_url': {'name': 'salt_api_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'salt_api_username': {'name': 'salt_api_username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'devicehealth', 'can_run': True, 'error_string': '', 'module_options': {'enable_monitoring': {'name': 'enable_monitoring', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'monitor device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'mark_out_threshold': {'name': 'mark_out_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '2419200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'automatically mark OSD if it may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}, 'pool_name': {'name': 'pool_name', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'device_health_metrics', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name of pool in which to store device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'retention_period': {'name': 'retention_period', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '15552000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how long to retain device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_frequency': {'name': 'scrape_frequency', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to scrape device health metrics', 'long_desc': '', 'tags': [], 'see_also': []}, 'self_heal': {'name': 'self_heal', 'type': 'bool', 'level': 'advanced', 'flags': 1, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'preemptively heal cluster around devices that may fail', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to wake up and check device health', 'long_desc': '', 'tags': [], 'see_also': []}, 'warn_threshold': {'name': 'warn_threshold', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '7257600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'raise health warning if OSD may fail before this long', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_cloud', 'can_run': True, 'error_string': '', 'module_options': {'diskprediction_cert_context': {'name': 'diskprediction_cert_context', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_default_authority': {'name': 'diskprediction_default_authority', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_password': {'name': 'diskprediction_password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_port': {'name': 'diskprediction_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '31400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_retrieve_prediction_interval': {'name': 'diskprediction_retrieve_prediction_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_server': {'name': 'diskprediction_server', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_ssl_target_name_override': {'name': 'diskprediction_ssl_target_name_override', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'localhost', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_metrics_interval': {'name': 'diskprediction_upload_metrics_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_upload_smart_interval': {'name': 'diskprediction_upload_smart_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '43200', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'diskprediction_user': {'name': 'diskprediction_user', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'diskprediction_local', 'can_run': True, 'error_string': '', 'module_options': {'predict_interval': {'name': 'predict_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '86400', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '600', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'influx', 'can_run': False, 'error_string': 'influxdb python module not found', 'module_options': {'batch_size': {'name': 'batch_size', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5000', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'database': {'name': 'database', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'ceph', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'hostname': {'name': 'hostname', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '30', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'password': {'name': 'password', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'port': {'name': 'port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '8086', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssl': {'name': 'ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'false', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'threads': {'name': 'threads', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'username': {'name': 'username', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'verify_ssl': {'name': 'verify_ssl', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'true', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'insights', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'iostat', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'localpool', 'can_run': True, 'error_string': '', 'module_options': {'failure_domain': {'name': 'failure_domain', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'host', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'failure domain for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'min_size': {'name': 'min_size', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default min_size for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'num_rep': {'name': 'num_rep', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '3', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default replica count for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'pg_num': {'name': 'pg_num', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '128', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'default pg_num for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'prefix': {'name': 'prefix', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'name prefix for any created local pool', 'long_desc': '', 'tags': [], 'see_also': []}, 'subtree': {'name': 'subtree', 'type': 'str', 'level': 'advanced', 'flags': 1, 'default_value': 'rack', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'CRUSH level for which to create a local pool', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'orchestrator_cli', 'can_run': True, 'error_string': '', 'module_options': {'orchestrator': {'name': 'orchestrator', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'pg_autoscaler', 'can_run': True, 'error_string': '', 'module_options': {'sleep_interval': {'name': 'sleep_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'progress', 'can_run': True, 'error_string': '', 'module_options': {'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'max_completed_events': {'name': 'max_completed_events', 'type': 'int', 'level': 'advanced', 'flags': 1, 'default_value': '50', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'number of past completed events to remember', 'long_desc': '', 'tags': [], 'see_also': []}, 'persist_interval': {'name': 'persist_interval', 'type': 'secs', 'level': 'advanced', 'flags': 1, 'default_value': '5', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'how frequently to persist completed events', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'prometheus', 'can_run': True, 'error_string': '', 'module_options': {'rbd_stats_pools': {'name': 'rbd_stats_pools', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rbd_stats_pools_refresh_interval': {'name': 'rbd_stats_pools_refresh_interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '300', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'scrape_interval': {'name': 'scrape_interval', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'stale_cache_strategy': {'name': 'stale_cache_strategy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rbd_support', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'restful', 'can_run': True, 'error_string': '', 'module_options': {'key_file': {'name': 'key_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_addr': {'name': 'server_addr', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'server_port': {'name': 'server_port', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'rook', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'selftest', 'can_run': True, 'error_string': '', 'module_options': {'roption1': {'name': 'roption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'roption2': {'name': 'roption2', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'xyz', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption1': {'name': 'rwoption1', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption2': {'name': 'rwoption2', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption3': {'name': 'rwoption3', 'type': 'float', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption4': {'name': 'rwoption4', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption5': {'name': 'rwoption5', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'rwoption6': {'name': 'rwoption6', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testkey': {'name': 'testkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testlkey': {'name': 'testlkey', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'testnewline': {'name': 'testnewline', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'ssh', 'can_run': True, 'error_string': '', 'module_options': {'inventory_cache_timeout_min': {'name': 'inventory_cache_timeout_min', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'ssh_config_file': {'name': 'ssh_config_file', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'status', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'telegraf', 'can_run': True, 'error_string': '', 'module_options': {'address': {'name': 'address', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'unixgram:///tmp/telegraf.sock', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '15', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'telemetry', 'can_run': True, 'error_string': '', 'module_options': {'channel_basic': {'name': 'channel_basic', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Share basic cluster information (size, version)', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_crash': {'name': 'channel_crash', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_device': {'name': 'channel_device', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'True', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'channel_ident': {'name': 'channel_ident', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'contact': {'name': 'contact', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'description': {'name': 'description', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'device_url': {'name': 'device_url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/device', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'enabled': {'name': 'enabled', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '24', 'min': '8', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'last_opt_revision': {'name': 'last_opt_revision', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '1', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'leaderboard': {'name': 'leaderboard', 'type': 'bool', 'level': 'advanced', 'flags': 0, 'default_value': 'False', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'organization': {'name': 'organization', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'proxy': {'name': 'proxy', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'url': {'name': 'url', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'https://telemetry.ceph.com/report', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'test_orchestrator', 'can_run': True, 'error_string': '', 'module_options': {}}, {'name': 'volumes', 'can_run': True, 'error_string': '', 'module_options': {'max_concurrent_clones': {'name': 'max_concurrent_clones', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '4', 'min': '', 'max': '', 'enum_allowed': [], 'desc': 'Number of asynchronous cloner threads', 'long_desc': '', 'tags': [], 'see_also': []}}}, {'name': 'zabbix', 'can_run': True, 'error_string': '', 'module_options': {'identifier': {'name': 'identifier', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'interval': {'name': 'interval', 'type': 'secs', 'level': 'advanced', 'flags': 0, 'default_value': '60', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_host': {'name': 'zabbix_host', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': 'None', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_port': {'name': 'zabbix_port', 'type': 'int', 'level': 'advanced', 'flags': 0, 'default_value': '10051', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}, 'zabbix_sender': {'name': 'zabbix_sender', 'type': 'str', 'level': 'advanced', 'flags': 0, 'default_value': '/usr/bin/zabbix_sender', 'min': '', 'max': '', 'enum_allowed': [], 'desc': '', 'long_desc': '', 'tags': [], 'see_also': []}}}], 'services': {}, 'always_on_modules': {'nautilus': ['balancer', 'crash', 'devicehealth', 'orchestrator_cli', 'progress', 'rbd_support', 'status', 'volumes']}}, 'servicemap': {'epoch': 2, 'modified': '2021-02-10 11:22:48.820001', 'services': {'mds': {'daemons': {'summary': '', 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': ':/0', 'metadata': {}, 'task_status': {'scrub status': 'idle'}}}}}}, 'progress_events': {}} 2021-02-10T11:22:52.160 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-02-10T11:22:52.160 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-02-10T11:22:52.160 DEBUG:teuthology.orchestra.run.gibba017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph health 2021-02-10T11:22:52.353 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:52.356 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:22:52.575 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.562 7fc4b5895700 1 Processor -- start 2021-02-10T11:22:52.575 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.562 7fc4b5895700 1 -- start start 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4b5895700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4b00d1870 0x7fc4b00d7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4b5895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b00d9350 0x7fc4b00d9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4b5895700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b00f6160 con 0x7fc4b00d7e80 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4b5895700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b00d1c70 con 0x7fc4b00d1870 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4b5895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00d8280 con 0x7fc4b00d9350 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4b00d1870 0x7fc4b00d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4aed73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc4b00d7e80 0x7fc4b00d8b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42010/0 (socket says 172.21.2.117:42010) 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4aed73700 1 -- 172.21.2.117:0/1761985239 learned_addr learned my addr 172.21.2.117:0/1761985239 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4b00d1870 0x7fc4b00d7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41760/0 (socket says 172.21.2.117:41760) 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 -- 172.21.2.117:0/1761985239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b00d9350 msgr2=0x7fc4b00d9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 --2- 172.21.2.117:0/1761985239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b00d9350 0x7fc4b00d9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 -- 172.21.2.117:0/1761985239 >> v1:172.21.2.117:6789/0 conn(0x7fc4b00d7e80 legacy=0x7fc4b00d8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.576 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 -- 172.21.2.117:0/1761985239 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b01014a0 con 0x7fc4b00d1870 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.563 7fc4af574700 1 --2- 172.21.2.117:0/1761985239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b00d1870 0x7fc4b00d7860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc498006650 tx=0x7fc498009a80).ready entity=mon.0 client_cookie=c06ced015cc75b28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4ae572700 1 -- 172.21.2.117:0/1761985239 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4980009b0 con 0x7fc4b00d1870 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4ae572700 1 -- 172.21.2.117:0/1761985239 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49800c0c0 con 0x7fc4b00d1870 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4ae572700 1 -- 172.21.2.117:0/1761985239 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc49800c520 con 0x7fc4b00d1870 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 -- 172.21.2.117:0/1761985239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b00d1870 msgr2=0x7fc4b00d7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 --2- 172.21.2.117:0/1761985239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b00d1870 0x7fc4b00d7860 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc498006650 tx=0x7fc498009a80).stop 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 -- 172.21.2.117:0/1761985239 shutdown_connections 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 --2- 172.21.2.117:0/1761985239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b00d9350 0x7fc4b00d9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 --2- 172.21.2.117:0/1761985239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b00d1870 0x7fc4b00d7860 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 -- 172.21.2.117:0/1761985239 >> 172.21.2.117:0/1761985239 conn(0x7fc4b00cecf0 msgr2=0x7fc4b00cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 -- 172.21.2.117:0/1761985239 shutdown_connections 2021-02-10T11:22:52.577 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.564 7fc4b5895700 1 -- 172.21.2.117:0/1761985239 wait complete. 2021-02-10T11:22:52.579 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.579 7fc4b5895700 1 Processor -- start 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.579 7fc4b5895700 1 -- start start 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4b019e2a0 0x7fc4b0191490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 0x7fc4b0093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b019e7d0 con 0x7fc4b0192290 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b0191eb0 con 0x7fc4b019e2a0 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4b5895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00b0110 con 0x7fc4b0191ab0 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41766/0 (socket says 172.21.2.117:41766) 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 -- 172.21.2.117:0/2583875444 learned_addr learned my addr 172.21.2.117:0/2583875444 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:52.580 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4afd75700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 0x7fc4b0093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4aed73700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4b019e2a0 0x7fc4b0191490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 -- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 msgr2=0x7fc4b0093200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 0x7fc4b0093200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 -- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4b019e2a0 msgr2=0x7fc4b0191490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4b019e2a0 0x7fc4b0191490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4af574700 1 -- 172.21.2.117:0/2583875444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b0093820 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.580 7fc4afd75700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 0x7fc4b0093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4af574700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc498009630 tx=0x7fc49800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4980093a0 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49800bd70 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc4980054d0 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b0093820 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.581 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b0093820 con 0x7fc4b0192290 2021-02-10T11:22:52.581 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.582 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc4980054f0 con 0x7fc4b0192290 2021-02-10T11:22:52.582 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.582 7fc4a7fff700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 0x7fc49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:52.582 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.582 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fc498032930 con 0x7fc4b0192290 2021-02-10T11:22:52.582 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.582 7fc4aed73700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 0x7fc49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:52.582 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.582 7fc4aed73700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 0x7fc49401abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:52.609 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.612 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b00f6740 con 0x7fc4b0192290 2021-02-10T11:22:52.611 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.615 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc4980052e0 con 0x7fc4b0192290 2021-02-10T11:22:52.744 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.747 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7fc4b00f6740 con 0x7fc4b0192290 2021-02-10T11:22:52.744 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.747 7fc4a7fff700 1 -- 172.21.2.117:0/2583875444 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+10 (secure 0 0 0) 0x7fc49800bd70 con 0x7fc4b0192290 2021-02-10T11:22:52.744 INFO:teuthology.misc.health.gibba017.stdout:HEALTH_OK 2021-02-10T11:22:52.746 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 msgr2=0x7fc49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.746 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 0x7fc49401abc0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.746 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 msgr2=0x7fc4b019dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc498009630 tx=0x7fc49800b980).stop 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 shutdown_connections 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc4b0191ab0 0x7fc4b0093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc4b019e2a0 0x7fc4b0191490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4940186c0 0x7fc49401abc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.749 7fc4b5895700 1 --2- 172.21.2.117:0/2583875444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4b0192290 0x7fc4b019dc80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.750 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 >> 172.21.2.117:0/2583875444 conn(0x7fc4b019f5b0 msgr2=0x7fc4b01b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.750 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 shutdown_connections 2021-02-10T11:22:52.747 INFO:teuthology.misc.health.gibba017.stderr:2021-02-10 11:22:52.750 7fc4b5895700 1 -- 172.21.2.117:0/2583875444 wait complete. 2021-02-10T11:22:52.753 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2021-02-10T11:22:52.754 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:53.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 Processor -- start 2021-02-10T11:22:53.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 -- start start 2021-02-10T11:22:53.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58a407da20 0x7f58a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 0x7f58a41b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a4077a70 con 0x7f58a40e5190 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a40777d0 con 0x7f58a407da20 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.205 7f58abe4a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a407de20 con 0x7f58a40e4d90 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58a9c66700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58a407da20 0x7f58a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58a9465700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 0x7f58a41b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58a9465700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 0x7f58a41b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:39454/0 (socket says 172.21.2.117:39454) 2021-02-10T11:22:53.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58a9c66700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58a407da20 0x7f58a40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:46158/0 (socket says 172.21.2.117:46158) 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58a9465700 1 -- 172.21.2.117:0/401235495 learned_addr learned my addr 172.21.2.117:0/401235495 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.206 7f58aa467700 1 --1- 172.21.2.117:0/401235495 >> v1:172.21.2.125:6789/0 conn(0x7f58a40e5190 0x7f58a4077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:34954/0 (socket says 172.21.2.117:34954) 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a9c66700 1 -- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 msgr2=0x7f58a41b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a9c66700 1 --2- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 0x7f58a41b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a9c66700 1 -- 172.21.2.117:0/401235495 >> v1:172.21.2.125:6789/0 conn(0x7f58a40e5190 legacy=0x7f58a4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a9c66700 1 -- 172.21.2.117:0/401235495 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a41b0700 con 0x7f58a407da20 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a9c66700 1 --2- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a407da20 0x7f58a40771b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5898006650 tx=0x7f5898009a80).ready entity=mon.1 client_cookie=9de5816c15945c5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a8c64700 1 -- 172.21.2.117:0/401235495 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58980009b0 con 0x7f58a407da20 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a8c64700 1 -- 172.21.2.117:0/401235495 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589800c0c0 con 0x7f58a407da20 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.207 7f58a8c64700 1 -- 172.21.2.117:0/401235495 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589800c510 con 0x7f58a407da20 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 -- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a407da20 msgr2=0x7f58a40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 --2- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a407da20 0x7f58a40771b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5898006650 tx=0x7f5898009a80).stop 2021-02-10T11:22:53.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 -- 172.21.2.117:0/401235495 shutdown_connections 2021-02-10T11:22:53.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 --2- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a40e4d90 0x7f58a41b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 --2- 172.21.2.117:0/401235495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a407da20 0x7f58a40771b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 -- 172.21.2.117:0/401235495 >> 172.21.2.117:0/401235495 conn(0x7f58a40ce440 msgr2=0x7f58a40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 -- 172.21.2.117:0/401235495 shutdown_connections 2021-02-10T11:22:53.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.208 7f58abe4a700 1 -- 172.21.2.117:0/401235495 wait complete. 2021-02-10T11:22:53.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 Processor -- start 2021-02-10T11:22:53.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 -- start start 2021-02-10T11:22:53.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a41aeb00 0x7f58a41af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a41af630 0x7f58a4091b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4106c70 con 0x7f58a4107590 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4107990 con 0x7f58a41aeb00 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.223 7f58abe4a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4107bb0 con 0x7f58a41af630 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41780/0 (socket says 172.21.2.117:41780) 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 -- 172.21.2.117:0/3693571637 learned_addr learned my addr 172.21.2.117:0/3693571637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 -- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a41af630 msgr2=0x7f58a4091b80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a41af630 0x7f58a4091b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 -- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a41aeb00 msgr2=0x7f58a41af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a41aeb00 0x7f58a41af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 -- 172.21.2.117:0/3693571637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a40921a0 con 0x7f58a4107590 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.224 7f58a9c66700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5898009630 tx=0x7f589800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58980093a0 con 0x7f58a4107590 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589800bd70 con 0x7f58a4107590 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a40921a0 con 0x7f58a4107590 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5898005560 con 0x7f58a4107590 2021-02-10T11:22:53.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a40921a0 con 0x7f58a4107590 2021-02-10T11:22:53.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58980054f0 con 0x7f58a4107590 2021-02-10T11:22:53.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 0x7f588c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.225 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f58980328c0 con 0x7f58a4107590 2021-02-10T11:22:53.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.226 7f58a9465700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 0x7f588c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.226 7f58a9465700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 0x7f588c01ac10 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.255 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a41b4f60 con 0x7f58a4107590 2021-02-10T11:22:53.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.258 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58980052e0 con 0x7f58a4107590 2021-02-10T11:22:53.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.382 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a4077a70 con 0x7f58a4107590 2021-02-10T11:22:53.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.382 7f58967fc700 1 -- 172.21.2.117:0/3693571637 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f589800bd70 con 0x7f58a4107590 2021-02-10T11:22:53.379 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:53.380 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 msgr2=0x7f588c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 0x7f588c01ac10 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 msgr2=0x7f58a41ae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5898009630 tx=0x7f589800b980).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 shutdown_connections 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58a41af630 0x7f58a4091b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58a41aeb00 0x7f58a41af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f588c018730 0x7f588c01ac10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 --2- 172.21.2.117:0/3693571637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58a4107590 0x7f58a41ae4e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 >> 172.21.2.117:0/3693571637 conn(0x7f58a41993c0 msgr2=0x7f58a41997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 shutdown_connections 2021-02-10T11:22:53.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.384 7f58abe4a700 1 -- 172.21.2.117:0/3693571637 wait complete. 2021-02-10T11:22:53.389 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:22:53.390 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 Processor -- start 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 -- start start 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb69007da20 0x7fb6900771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 0x7fb690079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb690077a70 con 0x7fb6900e4da0 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6900777d0 con 0x7fb69007da20 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb694f96700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6901b08e0 con 0x7fb6900e51a0 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb68e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb6900e4da0 0x7fb6901b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42038/0 (socket says 172.21.2.117:42038) 2021-02-10T11:22:53.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.807 7fb68e59c700 1 -- 172.21.2.117:0/2501092364 learned_addr learned my addr 172.21.2.117:0/2501092364 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68f59e700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 0x7fb690079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68ed9d700 1 --2- 172.21.2.117:0/2501092364 >> v2:172.21.2.125:3300/0 conn(0x7fb69007da20 0x7fb6900771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044177671 0 0) 0x7fb678000ce0 con 0x7fb6900e4da0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c002fe0 con 0x7fb6900e4da0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68ed9d700 1 -- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 msgr2=0x7fb690079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68ed9d700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 0x7fb690079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68ed9d700 1 -- 172.21.2.117:0/2501092364 >> v1:172.21.2.117:6789/0 conn(0x7fb6900e4da0 legacy=0x7fb6901b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4024295379 0 0) 0x7fb6780010e0 con 0x7fb6900e4da0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.808 7fb68ed9d700 1 -- 172.21.2.117:0/2501092364 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69007de20 con 0x7fb69007da20 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb68f59e700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 0x7fb690079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb68ed9d700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb69007da20 0x7fb6900771b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb680006650 tx=0x7fb680009a80).ready entity=mon.1 client_cookie=e0c5b996a72361b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb68000c0c0 con 0x7fb69007da20 2021-02-10T11:22:53.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68000c3c0 con 0x7fb69007da20 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb68dd9b700 1 -- 172.21.2.117:0/2501092364 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb68000c8e0 con 0x7fb69007da20 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb694f96700 1 -- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb69007da20 msgr2=0x7fb6900771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.809 7fb694f96700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb69007da20 0x7fb6900771b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb680006650 tx=0x7fb680009a80).stop 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 -- 172.21.2.117:0/2501092364 shutdown_connections 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6900e51a0 0x7fb690079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 --2- 172.21.2.117:0/2501092364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb69007da20 0x7fb6900771b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 -- 172.21.2.117:0/2501092364 >> 172.21.2.117:0/2501092364 conn(0x7fb6900ce450 msgr2=0x7fb6900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 -- 172.21.2.117:0/2501092364 shutdown_connections 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.810 7fb694f96700 1 -- 172.21.2.117:0/2501092364 wait complete. 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.824 7fb694f96700 1 Processor -- start 2021-02-10T11:22:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.824 7fb694f96700 1 -- start start 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6901048f0 0x7fb690104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb690090710 0x7fb690090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6901054f0 con 0x7fb6900b7fc0 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb690195a30 con 0x7fb690090710 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb694f96700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb690195fb0 con 0x7fb6901048f0 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41794/0 (socket says 172.21.2.117:41794) 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 -- 172.21.2.117:0/2589172708 learned_addr learned my addr 172.21.2.117:0/2589172708 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 -- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6901048f0 msgr2=0x7fb690104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6901048f0 0x7fb690104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 -- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb690090710 msgr2=0x7fb690090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb690090710 0x7fb690090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.825 7fb68ed9d700 1 -- 172.21.2.117:0/2589172708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6900b7b60 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb68ed9d700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb680006e90 tx=0x7fb68000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb68000bd80 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6800063f0 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb680005560 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb694f96700 1 -- 172.21.2.117:0/2589172708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6900b7b60 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb694f96700 1 -- 172.21.2.117:0/2589172708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6900b7b60 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.826 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb680006030 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.827 7fb6877fe700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 0x7fb67401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.827 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fb6800327c0 con 0x7fb6900b7fc0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.827 7fb68e59c700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 0x7fb67401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:53.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.828 7fb68e59c700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 0x7fb67401abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:53.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.857 7fb694f96700 1 -- 172.21.2.117:0/2589172708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb690103280 con 0x7fb6900b7fc0 2021-02-10T11:22:53.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.859 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb68001f520 con 0x7fb6900b7fc0 2021-02-10T11:22:53.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.986 7fb694f96700 1 -- 172.21.2.117:0/2589172708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb690103280 con 0x7fb6900b7fc0 2021-02-10T11:22:53.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.986 7fb6877fe700 1 -- 172.21.2.117:0/2589172708 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb68000bd80 con 0x7fb6900b7fc0 2021-02-10T11:22:53.983 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 -- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 msgr2=0x7fb67401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 0x7fb67401abc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 -- 172.21.2.117:0/2589172708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 msgr2=0x7fb6901042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb680006e90 tx=0x7fb68000b990).stop 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 -- 172.21.2.117:0/2589172708 shutdown_connections 2021-02-10T11:22:53.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb690090710 0x7fb690090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6901048f0 0x7fb690104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6740186c0 0x7fb67401abc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 --2- 172.21.2.117:0/2589172708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6900b7fc0 0x7fb6901042d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 -- 172.21.2.117:0/2589172708 >> 172.21.2.117:0/2589172708 conn(0x7fb6900a1850 msgr2=0x7fb6900c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.988 7fb694f96700 1 -- 172.21.2.117:0/2589172708 shutdown_connections 2021-02-10T11:22:53.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:53.989 7fb694f96700 1 -- 172.21.2.117:0/2589172708 wait complete. 2021-02-10T11:22:53.993 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:22:54.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.408 7f1a07b74700 1 Processor -- start 2021-02-10T11:22:54.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.408 7f1a07b74700 1 -- start start 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a07b74700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a0007da20 0x7f1a000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a07b74700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1a000e51a0 0x7f1a00079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a07b74700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a00077a70 con 0x7f1a000e4da0 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a07b74700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a000e5c20 con 0x7f1a0007da20 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a07b74700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a000777d0 con 0x7f1a000e51a0 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a0007da20 0x7f1a000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a0007da20 0x7f1a000771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41802/0 (socket says 172.21.2.117:41802) 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 -- 172.21.2.117:0/1306928683 learned_addr learned my addr 172.21.2.117:0/1306928683 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a06191700 1 --2- 172.21.2.117:0/1306928683 >> v2:172.21.2.125:3300/0 conn(0x7f1a000e51a0 0x7f1a00079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 -- 172.21.2.117:0/1306928683 >> v2:172.21.2.125:3300/0 conn(0x7f1a000e51a0 msgr2=0x7f1a00079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:54.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 --2- 172.21.2.117:0/1306928683 >> v2:172.21.2.125:3300/0 conn(0x7f1a000e51a0 0x7f1a00079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 -- 172.21.2.117:0/1306928683 >> v1:172.21.2.125:6789/0 conn(0x7f1a000e4da0 legacy=0x7f1a001b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.409 7f1a05990700 1 -- 172.21.2.117:0/1306928683 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a001b08e0 con 0x7f1a0007da20 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a05990700 1 --2- 172.21.2.117:0/1306928683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0007da20 0x7f1a000771b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f19fc006700 tx=0x7f19fc009b80).ready entity=mon.0 client_cookie=fedbd30fa8900a2f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a0498e700 1 -- 172.21.2.117:0/1306928683 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19fc00c0c0 con 0x7f1a0007da20 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a0498e700 1 -- 172.21.2.117:0/1306928683 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc00c340 con 0x7f1a0007da20 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a0498e700 1 -- 172.21.2.117:0/1306928683 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19fc00c860 con 0x7f1a0007da20 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a07b74700 1 -- 172.21.2.117:0/1306928683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0007da20 msgr2=0x7f1a000771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.410 7f1a07b74700 1 --2- 172.21.2.117:0/1306928683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0007da20 0x7f1a000771b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f19fc006700 tx=0x7f19fc009b80).stop 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 -- 172.21.2.117:0/1306928683 shutdown_connections 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 --2- 172.21.2.117:0/1306928683 >> v2:172.21.2.125:3300/0 conn(0x7f1a000e51a0 0x7f1a00079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 --2- 172.21.2.117:0/1306928683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0007da20 0x7f1a000771b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 -- 172.21.2.117:0/1306928683 >> 172.21.2.117:0/1306928683 conn(0x7f1a000ce450 msgr2=0x7f1a000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:54.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 -- 172.21.2.117:0/1306928683 shutdown_connections 2021-02-10T11:22:54.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.411 7f1a07b74700 1 -- 172.21.2.117:0/1306928683 wait complete. 2021-02-10T11:22:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.425 7f1a07b74700 1 Processor -- start 2021-02-10T11:22:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.425 7f1a07b74700 1 -- start start 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a0008a180 0x7f1a0008a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a0008afa0 0x7f1a001b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a001b9cb0 con 0x7f1a0008aba0 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000c4050 con 0x7f1a0008a180 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a07b74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a001b9730 con 0x7f1a0008afa0 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41808/0 (socket says 172.21.2.117:41808) 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 -- 172.21.2.117:0/1806017940 learned_addr learned my addr 172.21.2.117:0/1806017940 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 -- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a0008afa0 msgr2=0x7f1a001b8a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a0008afa0 0x7f1a001b8a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 -- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a0008a180 msgr2=0x7f1a0008a580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a0008a180 0x7f1a0008a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 -- 172.21.2.117:0/1806017940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00106e90 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.426 7f1a0518f700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f19f4006ed0 tx=0x7f19f400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.427 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f4006240 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.427 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f40009b0 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.427 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a00106e90 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.427 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f4005730 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.427 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a00106e90 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.428 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19f4006030 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.428 7f19f27fc700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 0x7f19e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.428 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f19f4032960 con 0x7f1a0008aba0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.428 7f1a05990700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 0x7f19e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.429 7f1a05990700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 0x7f19e801ac10 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.459 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a00077a70 con 0x7f1a0008aba0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.462 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19f4000d00 con 0x7f1a0008aba0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.587 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1a000c4050 con 0x7f1a0008aba0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.587 7f19f27fc700 1 -- 172.21.2.117:0/1806017940 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f19f4006030 con 0x7f1a0008aba0 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 msgr2=0x7f19e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 0x7f19e801ac10 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 msgr2=0x7f1a001b8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f19f4006ed0 tx=0x7f19f400b960).stop 2021-02-10T11:22:54.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 shutdown_connections 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e8018730 0x7f19e801ac10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a0008aba0 0x7f1a001b8420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a0008afa0 0x7f1a001b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 --2- 172.21.2.117:0/1806017940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a0008a180 0x7f1a0008a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.589 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 >> 172.21.2.117:0/1806017940 conn(0x7f1a000a4180 msgr2=0x7f1a0009be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.590 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 shutdown_connections 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:54.590 7f1a07b74700 1 -- 172.21.2.117:0/1806017940 wait complete. 2021-02-10T11:22:54.627 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:54.627 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:22:55.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 Processor -- start 2021-02-10T11:22:55.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 -- start start 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcbbc0d1530 0x7fcbbc0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc077a70 con 0x7fcbbc079000 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc079400 con 0x7fcbbc07da20 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.042 7fcbc3d61700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc07de20 con 0x7fcbbc0d1530 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc137c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcbbc079000 0x7fcbbc0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42064/0 (socket says 172.21.2.117:42064) 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc137c700 1 -- 172.21.2.117:0/1139735750 learned_addr learned my addr 172.21.2.117:0/1139735750 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc237e700 1 --2- 172.21.2.117:0/1139735750 >> v2:172.21.2.125:3300/0 conn(0x7fcbbc0d1530 0x7fcbbc0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404153043 0 0) 0x7fcbb0001180 con 0x7fcbbc079000 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba8002fe0 con 0x7fcbbc079000 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943860018 0 0) 0x7fcbb8001080 con 0x7fcbbc07da20 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba80042c0 con 0x7fcbbc07da20 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc237e700 1 -- 172.21.2.117:0/1139735750 >> v1:172.21.2.125:6789/0 conn(0x7fcbbc07da20 legacy=0x7fcbbc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc237e700 1 -- 172.21.2.117:0/1139735750 >> v1:172.21.2.117:6789/0 conn(0x7fcbbc079000 legacy=0x7fcbbc0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.043 7fcbc237e700 1 -- 172.21.2.117:0/1139735750 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc1b00f0 con 0x7fcbbc0d1530 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc237e700 1 --2- 172.21.2.117:0/1139735750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc0d1530 0x7fcbbc0d1930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcbb4006e50 tx=0x7fcbb4009e70).ready entity=mon.1 client_cookie=5562f34ad11b12dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb4006a20 con 0x7fcbbc0d1530 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4006790 con 0x7fcbbc0d1530 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc0b7b700 1 -- 172.21.2.117:0/1139735750 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb400b700 con 0x7fcbbc0d1530 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc3d61700 1 -- 172.21.2.117:0/1139735750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc0d1530 msgr2=0x7fcbbc0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.044 7fcbc3d61700 1 --2- 172.21.2.117:0/1139735750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc0d1530 0x7fcbbc0d1930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcbb4006e50 tx=0x7fcbb4009e70).stop 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.045 7fcbc3d61700 1 -- 172.21.2.117:0/1139735750 shutdown_connections 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.045 7fcbc3d61700 1 --2- 172.21.2.117:0/1139735750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc0d1530 0x7fcbbc0d1930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.045 7fcbc3d61700 1 -- 172.21.2.117:0/1139735750 >> 172.21.2.117:0/1139735750 conn(0x7fcbbc0ce450 msgr2=0x7fcbbc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.045 7fcbc3d61700 1 -- 172.21.2.117:0/1139735750 shutdown_connections 2021-02-10T11:22:55.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.045 7fcbc3d61700 1 -- 172.21.2.117:0/1139735750 wait complete. 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.059 7fcbc3d61700 1 Processor -- start 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.059 7fcbc3d61700 1 -- start start 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbbc1b5b50 0x7fcbbc0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc1b5f50 0x7fcbbc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b8da0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0a4140 con 0x7fcbbc1b5f50 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0a4320 con 0x7fcbbc1b5b50 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41818/0 (socket says 172.21.2.117:41818) 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 -- 172.21.2.117:0/2248301864 learned_addr learned my addr 172.21.2.117:0/2248301864 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 -- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbbc1b5b50 msgr2=0x7fcbbc0b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbbc1b5b50 0x7fcbbc0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 -- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc1b5f50 msgr2=0x7fcbbc0b5c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc1b5f50 0x7fcbbc0b5c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 -- 172.21.2.117:0/2248301864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0b62b0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc1b7d700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcbb800b530 tx=0x7fcbb800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb800bcc0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb8000e20 con 0x7fcbbc1b5130 2021-02-10T11:22:55.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc0b62b0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.060 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbb80054f0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.061 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc0b62b0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.061 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbb8006240 con 0x7fcbbc1b5130 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.061 7fcbae7fc700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 0x7fcba401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.061 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcbb80327d0 con 0x7fcbbc1b5130 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.061 7fcbc137c700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 0x7fcba401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.062 7fcbc137c700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 0x7fcba401ac10 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.092 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc1b4f00 con 0x7fcbbc1b5130 2021-02-10T11:22:55.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.095 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbb8006030 con 0x7fcbbc1b5130 2021-02-10T11:22:55.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.223 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fcbbc077a70 con 0x7fcbbc1b5130 2021-02-10T11:22:55.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.223 7fcbae7fc700 1 -- 172.21.2.117:0/2248301864 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcbb8006240 con 0x7fcbbc1b5130 2021-02-10T11:22:55.220 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 msgr2=0x7fcba401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 0x7fcba401ac10 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 msgr2=0x7fcbbc1b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcbb800b530 tx=0x7fcbb800b8d0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 shutdown_connections 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbbc1b5f50 0x7fcbbc0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbbc1b5b50 0x7fcbbc0b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcba4018730 0x7fcba401ac10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 --2- 172.21.2.117:0/2248301864 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbbc1b5130 0x7fcbbc1b5530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 >> 172.21.2.117:0/2248301864 conn(0x7fcbbc08bbc0 msgr2=0x7fcbbc0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 shutdown_connections 2021-02-10T11:22:55.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.225 7fcbc3d61700 1 -- 172.21.2.117:0/2248301864 wait complete. 2021-02-10T11:22:55.230 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:55.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 Processor -- start 2021-02-10T11:22:55.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 -- start start 2021-02-10T11:22:55.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1020079000 0x7f10200e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f10201b00e0 0x7f10201b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1020077a70 con 0x7f102007da20 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10200e53b0 con 0x7f10201b00e0 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.643 7f1027d1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1020079400 con 0x7f1020079000 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f1025b3b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f102007da20 0x7f10200771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42076/0 (socket says 172.21.2.117:42076) 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f1025b3b700 1 -- 172.21.2.117:0/2190282546 learned_addr learned my addr 172.21.2.117:0/2190282546 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 --2- 172.21.2.117:0/2190282546 >> v2:172.21.2.117:3300/0 conn(0x7f10201b00e0 0x7f10201b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f1024b39700 1 -- 172.21.2.117:0/2190282546 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049435095 0 0) 0x7f1010001180 con 0x7f102007da20 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 -- 172.21.2.117:0/2190282546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1020079000 msgr2=0x7f10200e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:55.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 --2- 172.21.2.117:0/2190282546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1020079000 0x7f10200e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 -- 172.21.2.117:0/2190282546 >> v1:172.21.2.117:6789/0 conn(0x7f102007da20 legacy=0x7f10200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 -- 172.21.2.117:0/2190282546 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102007de20 con 0x7f10201b00e0 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.644 7f102633c700 1 --2- 172.21.2.117:0/2190282546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201b00e0 0x7f10201b04e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f101c006ab0 tx=0x7f101c009b40).ready entity=mon.0 client_cookie=4374c663c1478f0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1024b39700 1 -- 172.21.2.117:0/2190282546 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101c0022e0 con 0x7f10201b00e0 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1024b39700 1 -- 172.21.2.117:0/2190282546 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c002660 con 0x7f10201b00e0 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1024b39700 1 -- 172.21.2.117:0/2190282546 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101c002a70 con 0x7f10201b00e0 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 -- 172.21.2.117:0/2190282546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201b00e0 msgr2=0x7f10201b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 --2- 172.21.2.117:0/2190282546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201b00e0 0x7f10201b04e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f101c006ab0 tx=0x7f101c009b40).stop 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 -- 172.21.2.117:0/2190282546 shutdown_connections 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 --2- 172.21.2.117:0/2190282546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201b00e0 0x7f10201b04e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 --2- 172.21.2.117:0/2190282546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1020079000 0x7f10200e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.645 7f1027d1f700 1 -- 172.21.2.117:0/2190282546 >> 172.21.2.117:0/2190282546 conn(0x7f10200ce440 msgr2=0x7f10200e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.646 7f1027d1f700 1 -- 172.21.2.117:0/2190282546 shutdown_connections 2021-02-10T11:22:55.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.646 7f1027d1f700 1 -- 172.21.2.117:0/2190282546 wait complete. 2021-02-10T11:22:55.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.660 7f1027d1f700 1 Processor -- start 2021-02-10T11:22:55.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.660 7f1027d1f700 1 -- start start 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10201a1fd0 0x7f102019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f102019c030 0x7f102019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201a23d0 con 0x7f10201a5640 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10201a5a40 con 0x7f10201a1fd0 2021-02-10T11:22:55.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1027d1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102008a1b0 con 0x7f102019c030 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41832/0 (socket says 172.21.2.117:41832) 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 -- 172.21.2.117:0/767246617 learned_addr learned my addr 172.21.2.117:0/767246617 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 -- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f102019c030 msgr2=0x7f102019c430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f102019c030 0x7f102019c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 -- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10201a1fd0 msgr2=0x7f102019ba10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10201a1fd0 0x7f102019ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 -- 172.21.2.117:0/767246617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102019ca50 con 0x7f10201a5640 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.661 7f1025b3b700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f101000a570 tx=0x7f101000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f101000ad00 con 0x7f10201a5640 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010001180 con 0x7f10201a5640 2021-02-10T11:22:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1010004620 con 0x7f10201a5640 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f1027d1f700 1 -- 172.21.2.117:0/767246617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102019ca50 con 0x7f10201a5640 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f1027d1f700 1 -- 172.21.2.117:0/767246617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102019ca50 con 0x7f10201a5640 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10100043a0 con 0x7f10201a5640 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.662 7f10167fc700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 0x7f100801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.663 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f1010006fd0 con 0x7f10201a5640 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.663 7f102533a700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 0x7f100801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:55.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.663 7f102533a700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 0x7f100801ac10 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:55.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.694 7f1027d1f700 1 -- 172.21.2.117:0/767246617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1020108c10 con 0x7f10201a5640 2021-02-10T11:22:55.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.696 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1010001180 con 0x7f10201a5640 2021-02-10T11:22:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.824 7f1027d1f700 1 -- 172.21.2.117:0/767246617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1020077a70 con 0x7f10201a5640 2021-02-10T11:22:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.824 7f10167fc700 1 -- 172.21.2.117:0/767246617 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f1010006fd0 con 0x7f10201a5640 2021-02-10T11:22:55.822 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:55.822 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:22:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:22:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 -- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 msgr2=0x7f100801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 0x7f100801ac10 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 -- 172.21.2.117:0/767246617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 msgr2=0x7f10201a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f101000a570 tx=0x7f101000a910).stop 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 -- 172.21.2.117:0/767246617 shutdown_connections 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f102019c030 0x7f102019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10201a1fd0 0x7f102019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1008018730 0x7f100801ac10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.826 7f1027d1f700 1 --2- 172.21.2.117:0/767246617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10201a5640 0x7f10201a19b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.827 7f1027d1f700 1 -- 172.21.2.117:0/767246617 >> 172.21.2.117:0/767246617 conn(0x7f1020191110 msgr2=0x7f102019aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.827 7f1027d1f700 1 -- 172.21.2.117:0/767246617 shutdown_connections 2021-02-10T11:22:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:55.827 7f1027d1f700 1 -- 172.21.2.117:0/767246617 wait complete. 2021-02-10T11:22:56.032 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:32.545631', 'modified': '2021-02-10 11:22:41.961589', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4229': {'gid': 4229, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/921969069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 921969069}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 921969069}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:22:56.033 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:22:56.033 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:22:56.430 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:22:56.430 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:22:56.430 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4229, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 15.47299338, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 6, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 22, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 22, 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 19.834135648 2021-02-10T11:22:56.431 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:22:56.434 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4229, "want_state": "up:active", "state": "up:active", "rank_uptime": 15.47299338, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "uptime": 19.834135648 } 2021-02-10T11:22:56.434 INFO:teuthology.run_tasks:Running task ceph-fuse... 2021-02-10T11:22:56.452 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2021-02-10T11:22:56.452 INFO:tasks.ceph_fuse:config is None 2021-02-10T11:22:56.452 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None, 'client.3': None} 2021-02-10T11:22:56.452 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2021-02-10T11:22:56.452 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.866 7fa4ab4d3700 1 Processor -- start 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.866 7fa4ab4d3700 1 -- start start 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4ab4d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4a4079000 0x7fa4a40e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4ab4d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a41b00e0 0x7fa4a41b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4ab4d3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4077a70 con 0x7fa4a407da20 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4ab4d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a40e53b0 con 0x7fa4a4079000 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4ab4d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a4079400 con 0x7fa4a41b00e0 2021-02-10T11:22:56.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4a4079000 0x7fa4a40e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4a4079000 0x7fa4a40e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41840/0 (socket says 172.21.2.117:41840) 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 -- 172.21.2.117:0/4283395894 learned_addr learned my addr 172.21.2.117:0/4283395894 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 -- 172.21.2.117:0/4283395894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a41b00e0 msgr2=0x7fa4a41b04e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 --2- 172.21.2.117:0/4283395894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a41b00e0 0x7fa4a41b04e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 -- 172.21.2.117:0/4283395894 >> v1:172.21.2.125:6789/0 conn(0x7fa4a407da20 legacy=0x7fa4a40771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 -- 172.21.2.117:0/4283395894 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a407de20 con 0x7fa4a4079000 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa4a8aee700 1 --2- 172.21.2.117:0/4283395894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a4079000 0x7fa4a40e4d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa4a0006700 tx=0x7fa4a0009b80).ready entity=mon.0 client_cookie=72cb114d988f66e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.867 7fa497fff700 1 -- 172.21.2.117:0/4283395894 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a000c0c0 con 0x7fa4a4079000 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa497fff700 1 -- 172.21.2.117:0/4283395894 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a000c440 con 0x7fa4a4079000 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa497fff700 1 -- 172.21.2.117:0/4283395894 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa4a000c920 con 0x7fa4a4079000 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 -- 172.21.2.117:0/4283395894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a4079000 msgr2=0x7fa4a40e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 --2- 172.21.2.117:0/4283395894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a4079000 0x7fa4a40e4d90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa4a0006700 tx=0x7fa4a0009b80).stop 2021-02-10T11:22:56.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 -- 172.21.2.117:0/4283395894 shutdown_connections 2021-02-10T11:22:56.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 --2- 172.21.2.117:0/4283395894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a41b00e0 0x7fa4a41b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:56.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 --2- 172.21.2.117:0/4283395894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a4079000 0x7fa4a40e4d90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:56.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 -- 172.21.2.117:0/4283395894 >> 172.21.2.117:0/4283395894 conn(0x7fa4a40ce440 msgr2=0x7fa4a40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:56.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 -- 172.21.2.117:0/4283395894 shutdown_connections 2021-02-10T11:22:56.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.868 7fa4ab4d3700 1 -- 172.21.2.117:0/4283395894 wait complete. 2021-02-10T11:22:56.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 Processor -- start 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 -- start start 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a418ee60 0x7fa4a418f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4a418f880 0x7fa4a418fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41af0c0 con 0x7fa4a418cff0 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a418da10 con 0x7fa4a418f880 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4ab4d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a418dc30 con 0x7fa4a418ee60 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4a9af0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4a9af0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41848/0 (socket says 172.21.2.117:41848) 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.883 7fa4a9af0700 1 -- 172.21.2.117:0/2712473394 learned_addr learned my addr 172.21.2.117:0/2712473394 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 -- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a418ee60 msgr2=0x7fa4a418f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:56.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a418ee60 0x7fa4a418f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 -- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4a418f880 msgr2=0x7fa4a418fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4a418f880 0x7fa4a418fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 -- 172.21.2.117:0/2712473394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a418de50 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4a9af0700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa49c009960 tx=0x7fa49c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c003030 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c001310 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa49c001830 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a418de50 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.884 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a418e3e0 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.885 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa49c0014f0 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.885 7fa495ffb700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 0x7fa48401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.885 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fa49c032a50 con 0x7fa4a418cff0 2021-02-10T11:22:56.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.885 7fa4a92ef700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 0x7fa48401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:56.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.886 7fa4a92ef700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 0x7fa48401abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:56.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.916 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a418e930 con 0x7fa4a418cff0 2021-02-10T11:22:56.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:56.919 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa49c001700 con 0x7fa4a418cff0 2021-02-10T11:22:57.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.042 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4a418e930 con 0x7fa4a418cff0 2021-02-10T11:22:57.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.043 7fa495ffb700 1 -- 172.21.2.117:0/2712473394 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7fa49c0014f0 con 0x7fa4a418cff0 2021-02-10T11:22:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:57.041 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:22:57.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.044 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 msgr2=0x7fa48401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.044 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 0x7fa48401abc0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.044 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 msgr2=0x7fa4a418d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.044 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa49c009960 tx=0x7fa49c00bcf0).stop 2021-02-10T11:22:57.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 shutdown_connections 2021-02-10T11:22:57.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4840186c0 0x7fa48401abc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4a418cff0 0x7fa4a418d3f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4a418f880 0x7fa4a418fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:57.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 --2- 172.21.2.117:0/2712473394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4a418ee60 0x7fa4a418f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:57.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 >> 172.21.2.117:0/2712473394 conn(0x7fa4a418cba0 msgr2=0x7fa4a41a6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:57.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 shutdown_connections 2021-02-10T11:22:57.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.045 7fa4ab4d3700 1 -- 172.21.2.117:0/2712473394 wait complete. 2021-02-10T11:22:57.051 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:22:57.052 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.475 7f34e11df700 1 Processor -- start 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.475 7f34e11df700 1 -- start start 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.475 7f34e11df700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f34dc079000 0x7f34dc0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.475 7f34e11df700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34dc1b00f0 0x7f34dc1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34e11df700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34dc077a70 con 0x7f34dc07da20 2021-02-10T11:22:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34e11df700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc0e5c20 con 0x7f34dc1b00f0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34e11df700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34dc0e53c0 con 0x7f34dc079000 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34dc1b00f0 0x7f34dc1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34dc1b00f0 0x7f34dc1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41858/0 (socket says 172.21.2.117:41858) 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 -- 172.21.2.117:0/2839466869 learned_addr learned my addr 172.21.2.117:0/2839466869 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34da59c700 1 --2- 172.21.2.117:0/2839466869 >> v2:172.21.2.125:3300/0 conn(0x7f34dc079000 0x7f34dc0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 -- 172.21.2.117:0/2839466869 >> v2:172.21.2.125:3300/0 conn(0x7f34dc079000 msgr2=0x7f34dc0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 --2- 172.21.2.117:0/2839466869 >> v2:172.21.2.125:3300/0 conn(0x7f34dc079000 0x7f34dc0e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 -- 172.21.2.117:0/2839466869 >> v1:172.21.2.125:6789/0 conn(0x7f34dc07da20 legacy=0x7f34dc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 -- 172.21.2.117:0/2839466869 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc079400 con 0x7f34dc1b00f0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34db59e700 1 --2- 172.21.2.117:0/2839466869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc1b00f0 0x7f34dc1b04f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f34d4006f00 tx=0x7f34d4009f10).ready entity=mon.0 client_cookie=6491ac791d5f7561 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34d9d9b700 1 -- 172.21.2.117:0/2839466869 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34d400b3b0 con 0x7f34dc1b00f0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34d9d9b700 1 -- 172.21.2.117:0/2839466869 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d400b630 con 0x7f34dc1b00f0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.476 7f34d9d9b700 1 -- 172.21.2.117:0/2839466869 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34d400bb50 con 0x7f34dc1b00f0 2021-02-10T11:22:57.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 -- 172.21.2.117:0/2839466869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc1b00f0 msgr2=0x7f34dc1b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 --2- 172.21.2.117:0/2839466869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc1b00f0 0x7f34dc1b04f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f34d4006f00 tx=0x7f34d4009f10).stop 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 -- 172.21.2.117:0/2839466869 shutdown_connections 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 --2- 172.21.2.117:0/2839466869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc1b00f0 0x7f34dc1b04f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 --2- 172.21.2.117:0/2839466869 >> v2:172.21.2.125:3300/0 conn(0x7f34dc079000 0x7f34dc0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 -- 172.21.2.117:0/2839466869 >> 172.21.2.117:0/2839466869 conn(0x7f34dc0ce450 msgr2=0x7f34dc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 -- 172.21.2.117:0/2839466869 shutdown_connections 2021-02-10T11:22:57.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.477 7f34e11df700 1 -- 172.21.2.117:0/2839466869 wait complete. 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 Processor -- start 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 -- start start 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34dc08eb10 0x7f34dc0c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 0x7f34dc0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0af540 con 0x7f34dc0afac0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0a0c80 con 0x7f34dc08eb10 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.492 7f34e11df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc0b04e0 con 0x7f34dc08dfe0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34dad9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34dc08eb10 0x7f34dc0c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:39540/0 (socket says 172.21.2.117:39540) 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 -- 172.21.2.117:0/4219203439 learned_addr learned my addr 172.21.2.117:0/4219203439 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34db59e700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 0x7f34dc0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 -- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34dc08eb10 msgr2=0x7f34dc0c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34dc08eb10 0x7f34dc0c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 -- 172.21.2.117:0/4219203439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 msgr2=0x7f34dc0afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 0x7f34dc0afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 -- 172.21.2.117:0/4219203439 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0b0700 con 0x7f34dc08dfe0 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34db59e700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 0x7f34dc0afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:22:57.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34da59c700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f34c4006f60 tx=0x7f34c400b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c4006340 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c400bdf0 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34e11df700 1 -- 172.21.2.117:0/4219203439 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34dc0b0700 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34e11df700 1 -- 172.21.2.117:0/4219203439 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc0b0700 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.493 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34c4002030 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.494 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34c4000c90 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.494 7f34d37fe700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 0x7f34c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.494 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f34c4032960 con 0x7f34dc08dfe0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.494 7f34dad9d700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 0x7f34c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.495 7f34dad9d700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 0x7f34c001abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.525 7f34e11df700 1 -- 172.21.2.117:0/4219203439 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc103280 con 0x7f34dc08dfe0 2021-02-10T11:22:57.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.528 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34c4006b50 con 0x7f34dc08dfe0 2021-02-10T11:22:57.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.657 7f34e11df700 1 -- 172.21.2.117:0/4219203439 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f34dc077a70 con 0x7f34dc08dfe0 2021-02-10T11:22:57.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.658 7f34d37fe700 1 -- 172.21.2.117:0/4219203439 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f34c400bd60 con 0x7f34dc08dfe0 2021-02-10T11:22:57.657 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:57.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.661 7f34e11df700 1 -- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 msgr2=0x7f34c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.661 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 0x7f34c001abc0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.661 7f34e11df700 1 -- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 msgr2=0x7f34dc08e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:57.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.661 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f34c4006f60 tx=0x7f34c400b970).stop 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 -- 172.21.2.117:0/4219203439 shutdown_connections 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34c0018730 0x7f34c001abc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34dc0afac0 0x7f34dc0afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34dc08dfe0 0x7f34dc08e3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 --2- 172.21.2.117:0/4219203439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34dc08eb10 0x7f34dc0c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 -- 172.21.2.117:0/4219203439 >> 172.21.2.117:0/4219203439 conn(0x7f34dc0a1850 msgr2=0x7f34dc0bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 -- 172.21.2.117:0/4219203439 shutdown_connections 2021-02-10T11:22:57.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:57.662 7f34e11df700 1 -- 172.21.2.117:0/4219203439 wait complete. 2021-02-10T11:22:57.666 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:22:58.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.093 7f40e6950700 1 Processor -- start 2021-02-10T11:22:58.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.093 7f40e6950700 1 -- start start 2021-02-10T11:22:58.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e007da20 0x7f40e00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40e00e4da0 0x7f40e01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f40e00e51a0 0x7f40e0077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e00e5c20 con 0x7f40e00e4da0 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e0077a70 con 0x7f40e00e51a0 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40e6950700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00777d0 con 0x7f40e007da20 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40e00e4da0 0x7f40e01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40e00e4da0 0x7f40e01b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41870/0 (socket says 172.21.2.117:41870) 2021-02-10T11:22:58.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 -- 172.21.2.117:0/1412076225 learned_addr learned my addr 172.21.2.117:0/1412076225 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 -- 172.21.2.117:0/1412076225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e007da20 msgr2=0x7f40e00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 --2- 172.21.2.117:0/1412076225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e007da20 0x7f40e00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 -- 172.21.2.117:0/1412076225 >> v2:172.21.2.125:3300/0 conn(0x7f40e00e51a0 msgr2=0x7f40e0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 --2- 172.21.2.117:0/1412076225 >> v2:172.21.2.125:3300/0 conn(0x7f40e00e51a0 0x7f40e0077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 -- 172.21.2.117:0/1412076225 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e007de20 con 0x7f40e00e4da0 2021-02-10T11:22:58.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.094 7f40df7fe700 1 --2- 172.21.2.117:0/1412076225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00e4da0 0x7f40e01b00f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f40d4006700 tx=0x7f40d4009b00).ready entity=mon.0 client_cookie=4a2c0d0e7b6b845f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40deffd700 1 -- 172.21.2.117:0/1412076225 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40d400c0c0 con 0x7f40e00e4da0 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40deffd700 1 -- 172.21.2.117:0/1412076225 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d400c400 con 0x7f40e00e4da0 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40deffd700 1 -- 172.21.2.117:0/1412076225 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40d400c860 con 0x7f40e00e4da0 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 -- 172.21.2.117:0/1412076225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00e4da0 msgr2=0x7f40e01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 --2- 172.21.2.117:0/1412076225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00e4da0 0x7f40e01b00f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f40d4006700 tx=0x7f40d4009b00).stop 2021-02-10T11:22:58.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 -- 172.21.2.117:0/1412076225 shutdown_connections 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 --2- 172.21.2.117:0/1412076225 >> v2:172.21.2.125:3300/0 conn(0x7f40e00e51a0 0x7f40e0077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 --2- 172.21.2.117:0/1412076225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00e4da0 0x7f40e01b00f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 --2- 172.21.2.117:0/1412076225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e007da20 0x7f40e00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 -- 172.21.2.117:0/1412076225 >> 172.21.2.117:0/1412076225 conn(0x7f40e00ce450 msgr2=0x7f40e00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 -- 172.21.2.117:0/1412076225 shutdown_connections 2021-02-10T11:22:58.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.095 7f40e6950700 1 -- 172.21.2.117:0/1412076225 wait complete. 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 Processor -- start 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 -- start start 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e00c33d0 0x7f40e00b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40e0090de0 0x7f40e00911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0090600 con 0x7f40e00b3ff0 2021-02-10T11:22:58.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0090860 con 0x7f40e0090de0 2021-02-10T11:22:58.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40e6950700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00b3c70 con 0x7f40e00c33d0 2021-02-10T11:22:58.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40dffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40dffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41872/0 (socket says 172.21.2.117:41872) 2021-02-10T11:22:58.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.110 7f40dffff700 1 -- 172.21.2.117:0/3577351234 learned_addr learned my addr 172.21.2.117:0/3577351234 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:58.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 -- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e00c33d0 msgr2=0x7f40e00b3650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e00c33d0 0x7f40e00b3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 -- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40e0090de0 msgr2=0x7f40e00911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40e0090de0 0x7f40e00911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 -- 172.21.2.117:0/3577351234 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e0091800 con 0x7f40e00b3ff0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dffff700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f40d0006f00 tx=0x7f40d000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40d0006350 con 0x7f40e00b3ff0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d000bdc0 con 0x7f40e00b3ff0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40d0002020 con 0x7f40e00b3ff0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40e6950700 1 -- 172.21.2.117:0/3577351234 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e0091800 con 0x7f40e00b3ff0 2021-02-10T11:22:58.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.111 7f40e6950700 1 -- 172.21.2.117:0/3577351234 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e0091d90 con 0x7f40e00b3ff0 2021-02-10T11:22:58.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.112 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40d0007020 con 0x7f40e00b3ff0 2021-02-10T11:22:58.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.112 7f40dcff9700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 0x7f40c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.112 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f40d0032510 con 0x7f40e00b3ff0 2021-02-10T11:22:58.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.112 7f40df7fe700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 0x7f40c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.113 7f40df7fe700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 0x7f40c001abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.143 7f40e6950700 1 -- 172.21.2.117:0/3577351234 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e0103370 con 0x7f40e00b3ff0 2021-02-10T11:22:58.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.146 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40d0020fb0 con 0x7f40e00b3ff0 2021-02-10T11:22:58.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.272 7f40e6950700 1 -- 172.21.2.117:0/3577351234 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f40e0077a70 con 0x7f40e00b3ff0 2021-02-10T11:22:58.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.272 7f40dcff9700 1 -- 172.21.2.117:0/3577351234 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f40d0020fb0 con 0x7f40e00b3ff0 2021-02-10T11:22:58.269 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:58.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 msgr2=0x7f40c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 0x7f40c001abc0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 msgr2=0x7f40e00b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f40d0006f00 tx=0x7f40d000b940).stop 2021-02-10T11:22:58.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 shutdown_connections 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40e0090de0 0x7f40e00911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40e00c33d0 0x7f40e00b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40c00186c0 0x7f40c001abc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 --2- 172.21.2.117:0/3577351234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40e00b3ff0 0x7f40e00b3030 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 >> 172.21.2.117:0/3577351234 conn(0x7f40e00a1420 msgr2=0x7f40e009efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 shutdown_connections 2021-02-10T11:22:58.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.274 7f40e6950700 1 -- 172.21.2.117:0/3577351234 wait complete. 2021-02-10T11:22:58.280 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:22:58.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 Processor -- start 2021-02-10T11:22:58.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 -- start start 2021-02-10T11:22:58.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 0x7fc3c40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40d1d20 con 0x7fc3c40e4da0 2021-02-10T11:22:58.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c4077a70 con 0x7fc3c40e51a0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c9e1c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c40777d0 con 0x7fc3c407da20 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c2ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc3c40e4da0 0x7fc3c41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42130/0 (socket says 172.21.2.117:42130) 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c2ffd700 1 -- 172.21.2.117:0/824876707 learned_addr learned my addr 172.21.2.117:0/824876707 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.709 7fc3c3fff700 1 --1- 172.21.2.117:0/824876707 >> v1:172.21.2.125:6789/0 conn(0x7fc3c40e51a0 0x7fc3c40d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35056/0 (socket says 172.21.2.117:35056) 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c37fe700 1 --2- 172.21.2.117:0/824876707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 0x7fc3c40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289334584 0 0) 0x7fc3bc001440 con 0x7fc3c40e51a0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0002fe0 con 0x7fc3c40e51a0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085099856 0 0) 0x7fc3b4000ff0 con 0x7fc3c40e4da0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0003020 con 0x7fc3c40e4da0 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c37fe700 1 -- 172.21.2.117:0/824876707 >> v1:172.21.2.125:6789/0 conn(0x7fc3c40e51a0 legacy=0x7fc3c40d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c37fe700 1 -- 172.21.2.117:0/824876707 >> v1:172.21.2.117:6789/0 conn(0x7fc3c40e4da0 legacy=0x7fc3c41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c37fe700 1 -- 172.21.2.117:0/824876707 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c41b08e0 con 0x7fc3c407da20 2021-02-10T11:22:58.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c37fe700 1 --2- 172.21.2.117:0/824876707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 0x7fc3c40771b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc3ac006650 tx=0x7fc3ac009a80).ready entity=mon.2 client_cookie=cfbcf9ecc087a970 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.710 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac00c0c0 con 0x7fc3c407da20 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac00c340 con 0x7fc3c407da20 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c27fc700 1 -- 172.21.2.117:0/824876707 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac00c860 con 0x7fc3c407da20 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 -- 172.21.2.117:0/824876707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 msgr2=0x7fc3c40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 --2- 172.21.2.117:0/824876707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 0x7fc3c40771b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc3ac006650 tx=0x7fc3ac009a80).stop 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 -- 172.21.2.117:0/824876707 shutdown_connections 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 --2- 172.21.2.117:0/824876707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c407da20 0x7fc3c40771b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 -- 172.21.2.117:0/824876707 >> 172.21.2.117:0/824876707 conn(0x7fc3c40ce450 msgr2=0x7fc3c40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 -- 172.21.2.117:0/824876707 shutdown_connections 2021-02-10T11:22:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.711 7fc3c9e1c700 1 -- 172.21.2.117:0/824876707 wait complete. 2021-02-10T11:22:58.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 Processor -- start 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 -- start start 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c41b5b50 0x7fc3c40b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3c41b5f50 0x7fc3c40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c40b8da0 con 0x7fc3c41b5130 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c40a4140 con 0x7fc3c41b5f50 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c9e1c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c40a4320 con 0x7fc3c41b5b50 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41888/0 (socket says 172.21.2.117:41888) 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.726 7fc3c37fe700 1 -- 172.21.2.117:0/2498522335 learned_addr learned my addr 172.21.2.117:0/2498522335 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 -- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c41b5b50 msgr2=0x7fc3c40b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c41b5b50 0x7fc3c40b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 -- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3c41b5f50 msgr2=0x7fc3c40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3c41b5f50 0x7fc3c40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 -- 172.21.2.117:0/2498522335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c40b62b0 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c37fe700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc3ac009630 tx=0x7fc3ac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac0093a0 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac00bd70 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3ac005560 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c40b62b0 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.727 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c40b6840 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.728 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3ac007020 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.728 7fc3bbfff700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 0x7fc3a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.728 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fc3ac0327b0 con 0x7fc3c41b5130 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.728 7fc3c2ffd700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 0x7fc3a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:58.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.729 7fc3c2ffd700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 0x7fc3a801abd0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:58.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.759 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c41b4f00 con 0x7fc3c41b5130 2021-02-10T11:22:58.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.761 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc3ac009340 con 0x7fc3c41b5130 2021-02-10T11:22:58.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.888 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc3c4077a70 con 0x7fc3c41b5130 2021-02-10T11:22:58.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.888 7fc3bbfff700 1 -- 172.21.2.117:0/2498522335 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc3ac00bd70 con 0x7fc3c41b5130 2021-02-10T11:22:58.885 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 msgr2=0x7fc3a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 0x7fc3a801abd0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 msgr2=0x7fc3c41b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc3ac009630 tx=0x7fc3ac00b980).stop 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 shutdown_connections 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3c41b5f50 0x7fc3c40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3c41b5b50 0x7fc3c40b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:22:58.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3a80186d0 0x7fc3a801abd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 --2- 172.21.2.117:0/2498522335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3c41b5130 0x7fc3c41b5530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:58.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 >> 172.21.2.117:0/2498522335 conn(0x7fc3c408bbc0 msgr2=0x7fc3c40c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:58.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 shutdown_connections 2021-02-10T11:22:58.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:58.890 7fc3c9e1c700 1 -- 172.21.2.117:0/2498522335 wait complete. 2021-02-10T11:22:58.895 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:22:58.895 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:22:59.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.310 7f66bdb5b700 1 Processor -- start 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.310 7f66bdb5b700 1 -- start start 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66bdb5b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f66b8079000 0x7f66b80e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66bdb5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b81b00e0 0x7f66b81b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66bdb5b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b8077a70 con 0x7f66b807da20 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66bdb5b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b80e53b0 con 0x7f66b8079000 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66bdb5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8079400 con 0x7f66b81b00e0 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f66b807da20 0x7f66b80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42142/0 (socket says 172.21.2.117:42142) 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b77fe700 1 -- 172.21.2.117:0/3413867542 learned_addr learned my addr 172.21.2.117:0/3413867542 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170019195 0 0) 0x7f66a00010b0 con 0x7f66b807da20 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a4002fe0 con 0x7f66b807da20 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b6ffd700 1 --2- 172.21.2.117:0/3413867542 >> v2:172.21.2.125:3300/0 conn(0x7f66b8079000 0x7f66b80e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.311 7f66b7fff700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b81b00e0 0x7f66b81b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1380346872 0 0) 0x7f66a00010b0 con 0x7f66b807da20 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b81b00e0 msgr2=0x7f66b81b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b81b00e0 0x7f66b81b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 >> v2:172.21.2.125:3300/0 conn(0x7f66b8079000 msgr2=0x7f66b80e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 --2- 172.21.2.117:0/3413867542 >> v2:172.21.2.125:3300/0 conn(0x7f66b8079000 0x7f66b80e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b807de20 con 0x7f66b807da20 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f66a00015b0 con 0x7f66b807da20 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 >> v1:172.21.2.117:6789/0 conn(0x7f66b807da20 legacy=0x7f66b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a4003af0 0x7f66a4005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a4006760 0x7f66a4008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a4009520 0x7f66a400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40502e0 con 0x7f66a4006760 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4050500 con 0x7f66a4009520 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4050720 con 0x7f66a4003af0 2021-02-10T11:22:59.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b77fe700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a4003af0 0x7f66a4005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b7fff700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a4009520 0x7f66a400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a4006760 0x7f66a4008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 -- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a4003af0 msgr2=0x7f66a4005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a4003af0 0x7f66a4005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 -- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a4009520 msgr2=0x7f66a400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a4009520 0x7f66a400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 -- 172.21.2.117:0/3413867542 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40509c0 con 0x7f66a4006760 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b6ffd700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a4006760 0x7f66a4008c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f66a800c8f0 tx=0x7f66a800cc90).ready entity=mon.0 client_cookie=388d53322a6da168 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a800cdd0 con 0x7f66a4006760 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8009310 con 0x7f66a4006760 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.312 7f66b67fc700 1 -- 172.21.2.117:0/3413867542 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a80097f0 con 0x7f66a4006760 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 -- 172.21.2.117:0/3413867542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a4006760 msgr2=0x7f66a4008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 --2- 172.21.2.117:0/3413867542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a4006760 0x7f66a4008c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f66a800c8f0 tx=0x7f66a800cc90).stop 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66b7fff700 1 -- 172.21.2.117:0/3413867542 reap_dead start 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 -- 172.21.2.117:0/3413867542 shutdown_connections 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 -- 172.21.2.117:0/3413867542 >> 172.21.2.117:0/3413867542 conn(0x7f66b80ce440 msgr2=0x7f66b80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 -- 172.21.2.117:0/3413867542 shutdown_connections 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.313 7f66bdb5b700 1 -- 172.21.2.117:0/3413867542 wait complete. 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 Processor -- start 2021-02-10T11:22:59.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 -- start start 2021-02-10T11:22:59.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66b80b8bd0 0x7f66b8192ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 0x7f66b8193b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8193590 con 0x7f66b818ebe0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8106ed0 con 0x7f66b80b8bd0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.328 7f66bdb5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b81931c0 con 0x7f66b8106ad0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41904/0 (socket says 172.21.2.117:41904) 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b77fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66b80b8bd0 0x7f66b8192ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 -- 172.21.2.117:0/3586289748 learned_addr learned my addr 172.21.2.117:0/3586289748 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b7fff700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 0x7f66b8193b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 -- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 msgr2=0x7f66b8193b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 0x7f66b8193b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 -- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66b80b8bd0 msgr2=0x7f66b8192ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66b80b8bd0 0x7f66b8192ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 -- 172.21.2.117:0/3586289748 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b8194130 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b7fff700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 0x7f66b8193b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66b6ffd700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f66a8006d50 tx=0x7f66a800b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a800c7c0 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a800cb00 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b8194130 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b81946c0 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.329 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66a800b6f0 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.330 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66a800c7c0 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.330 7f66affff700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 0x7f669c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.330 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f66a80326e0 con 0x7f66b818ebe0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.330 7f66b77fe700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 0x7f669c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:22:59.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.331 7f66b77fe700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 0x7f669c01abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:22:59.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.361 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b8077a70 con 0x7f66b818ebe0 2021-02-10T11:22:59.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.363 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66a800c9d0 con 0x7f66b818ebe0 2021-02-10T11:22:59.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.491 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b8077a70 con 0x7f66b818ebe0 2021-02-10T11:22:59.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.491 7f66affff700 1 -- 172.21.2.117:0/3586289748 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f66a800c7c0 con 0x7f66b818ebe0 2021-02-10T11:22:59.488 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:22:59.489 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:22:59.489 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:22:59.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 msgr2=0x7f669c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 0x7f669c01abc0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 msgr2=0x7f66b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:22:59.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f66a8006d50 tx=0x7f66a800b300).stop 2021-02-10T11:22:59.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 shutdown_connections 2021-02-10T11:22:59.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f669c0186c0 0x7f669c01abc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66b818ebe0 0x7f66b81064b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66b8106ad0 0x7f66b8193b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 --2- 172.21.2.117:0/3586289748 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66b80b8bd0 0x7f66b8192ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.493 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 >> 172.21.2.117:0/3586289748 conn(0x7f66b819f510 msgr2=0x7f66b81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.494 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 shutdown_connections 2021-02-10T11:22:59.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:22:59.494 7f66bdb5b700 1 -- 172.21.2.117:0/3586289748 wait complete. 2021-02-10T11:22:59.700 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:32.545631', 'modified': '2021-02-10 11:22:41.961589', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4229': {'gid': 4229, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/921969069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 921969069}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 921969069}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:22:59.700 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:22:59.700 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:00.096 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:00.096 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4229, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 19.138808742999998, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 6, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 22, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 22, 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.499950971000001 2021-02-10T11:23:00.097 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:00.100 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4229, "want_state": "up:active", "state": "up:active", "rank_uptime": 19.138808742999998, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "uptime": 23.499950971000001 } 2021-02-10T11:23:00.100 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:00.100 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:23:00.100 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:23:00.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:00.101 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:00.436 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:23:00.804 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:00.804 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:01.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:01.149 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:23:01.483 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:23:01.484 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:01.486 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:23:01.833 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:01.834 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:01.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[39909]: starting ceph client 2021-02-10T11:23:01.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:23:01.902 7fba44771f80 -1 init, newargv = 0x559e0e624ee0 newargc=9 2021-02-10T11:23:01.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[39909]: starting fuse 2021-02-10T11:23:02.175 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:02.175 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:02.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:02.177 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:23:02.510 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:23:02.511 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:23:02.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:23:02.511 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:02.512 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:23:02.513 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:23:02.902 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.39909.asok 2021-02-10T11:23:02.904 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39909.asok 2021-02-10T11:23:02.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:02.905 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39909.asok status 2021-02-10T11:23:03.292 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:23:03.292 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "39909", 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 4557, 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:23:03.293 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 4557 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1550845593 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:23:03.294 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1550845593 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.4557 172.21.2.117:0/1550845593", 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1550845593", 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 6, 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 22, 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:23:03.295 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:23:03.297 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:03.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 Processor -- start 2021-02-10T11:23:03.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 -- start start 2021-02-10T11:23:03.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 0x7f09a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80e4d90 0x7f09a81b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a8077a70 con 0x7f09a80e5190 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a80777d0 con 0x7f09a80e4d90 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.708 7f09afde9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a807de20 con 0x7f09a807da20 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80e4d90 0x7f09a81b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80e4d90 0x7f09a81b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41926/0 (socket says 172.21.2.117:41926) 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 -- 172.21.2.117:0/3933658562 learned_addr learned my addr 172.21.2.117:0/3933658562 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09adc05700 1 --2- 172.21.2.117:0/3933658562 >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 0x7f09a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 -- 172.21.2.117:0/3933658562 >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 msgr2=0x7f09a80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 --2- 172.21.2.117:0/3933658562 >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 0x7f09a80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 -- 172.21.2.117:0/3933658562 >> v1:172.21.2.125:6789/0 conn(0x7f09a80e5190 legacy=0x7f09a8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 -- 172.21.2.117:0/3933658562 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a81b0700 con 0x7f09a80e4d90 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09adc05700 1 --2- 172.21.2.117:0/3933658562 >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 0x7f09a80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09ad404700 1 --2- 172.21.2.117:0/3933658562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80e4d90 0x7f09a81b00e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f09a4006700 tx=0x7f09a4009b80).ready entity=mon.0 client_cookie=1883ee2f19054162 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.709 7f09acc03700 1 -- 172.21.2.117:0/3933658562 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a400c0c0 con 0x7f09a80e4d90 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09acc03700 1 -- 172.21.2.117:0/3933658562 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a400c400 con 0x7f09a80e4d90 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09acc03700 1 -- 172.21.2.117:0/3933658562 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a400c8e0 con 0x7f09a80e4d90 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 -- 172.21.2.117:0/3933658562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80e4d90 msgr2=0x7f09a81b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 --2- 172.21.2.117:0/3933658562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80e4d90 0x7f09a81b00e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f09a4006700 tx=0x7f09a4009b80).stop 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 -- 172.21.2.117:0/3933658562 shutdown_connections 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 --2- 172.21.2.117:0/3933658562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80e4d90 0x7f09a81b00e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 --2- 172.21.2.117:0/3933658562 >> v2:172.21.2.125:3300/0 conn(0x7f09a807da20 0x7f09a80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.710 7f09afde9700 1 -- 172.21.2.117:0/3933658562 >> 172.21.2.117:0/3933658562 conn(0x7f09a80ce440 msgr2=0x7f09a80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.711 7f09afde9700 1 -- 172.21.2.117:0/3933658562 shutdown_connections 2021-02-10T11:23:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.711 7f09afde9700 1 -- 172.21.2.117:0/3933658562 wait complete. 2021-02-10T11:23:03.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.725 7f09afde9700 1 Processor -- start 2021-02-10T11:23:03.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.725 7f09afde9700 1 -- start start 2021-02-10T11:23:03.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a81a5640 0x7f09a81a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a819b930 0x7f09a819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80ab7d0 con 0x7f09a81a27c0 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a81a5a40 con 0x7f09a819b930 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09afde9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8199d40 con 0x7f09a81a5640 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41930/0 (socket says 172.21.2.117:41930) 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 -- 172.21.2.117:0/4033930196 learned_addr learned my addr 172.21.2.117:0/4033930196 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 -- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a81a5640 msgr2=0x7f09a81a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a81a5640 0x7f09a81a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 -- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a819b930 msgr2=0x7f09a819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a819b930 0x7f09a819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 -- 172.21.2.117:0/4033930196 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a819c350 con 0x7f09a81a27c0 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.726 7f09adc05700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f099c006f20 tx=0x7f099c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099c00bd80 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c000c90 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099c0022b0 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f09afde9700 1 -- 172.21.2.117:0/4033930196 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a819c350 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f09afde9700 1 -- 172.21.2.117:0/4033930196 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a819c350 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f099c00a000 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.727 7f099a7fc700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.728 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f099c0326e0 con 0x7f09a81a27c0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.728 7f09ad404700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.728 7f09ad404700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:03.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.758 7f09afde9700 1 -- 172.21.2.117:0/4033930196 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a8108c10 con 0x7f09a81a27c0 2021-02-10T11:23:03.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.761 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f099c006bb0 con 0x7f09a81a27c0 2021-02-10T11:23:03.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.884 7f09afde9700 1 -- 172.21.2.117:0/4033930196 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09a8077a70 con 0x7f09a81a27c0 2021-02-10T11:23:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.885 7f099a7fc700 1 -- 172.21.2.117:0/4033930196 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f099c00a000 con 0x7f09a81a27c0 2021-02-10T11:23:03.882 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:03.882 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 -- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 msgr2=0x7f099001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 -- 172.21.2.117:0/4033930196 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 msgr2=0x7f09a819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f099c006f20 tx=0x7f099c00b990).stop 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 -- 172.21.2.117:0/4033930196 shutdown_connections 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a819b930 0x7f09a819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a81a5640 0x7f09a81a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 --2- 172.21.2.117:0/4033930196 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a81a27c0 0x7f09a819a1e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.886 7f09afde9700 1 -- 172.21.2.117:0/4033930196 >> 172.21.2.117:0/4033930196 conn(0x7f09a8191110 msgr2=0x7f09a819aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.887 7f09afde9700 1 -- 172.21.2.117:0/4033930196 shutdown_connections 2021-02-10T11:23:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:03.887 7f09afde9700 1 -- 172.21.2.117:0/4033930196 wait complete. 2021-02-10T11:23:03.891 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:03.891 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:04.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.306 7fba2cc10700 1 Processor -- start 2021-02-10T11:23:04.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.306 7fba2cc10700 1 -- start start 2021-02-10T11:23:04.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.306 7fba2cc10700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba2807da20 0x7fba280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2cc10700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280d1d20 con 0x7fba280e4da0 2021-02-10T11:23:04.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2cc10700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba28077a70 con 0x7fba280e51a0 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2cc10700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba280777d0 con 0x7fba2807da20 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2605d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fba280e4da0 0x7fba281b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42190/0 (socket says 172.21.2.117:42190) 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2605d700 1 -- 172.21.2.117:0/2551001603 learned_addr learned my addr 172.21.2.117:0/2551001603 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2685e700 1 --2- 172.21.2.117:0/2551001603 >> v2:172.21.2.117:3300/0 conn(0x7fba2807da20 0x7fba280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2585c700 1 -- 172.21.2.117:0/2551001603 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270498504 0 0) 0x7fba100010b0 con 0x7fba280e4da0 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2585c700 1 -- 172.21.2.117:0/2551001603 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14002fe0 con 0x7fba280e4da0 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2685e700 1 -- 172.21.2.117:0/2551001603 >> v1:172.21.2.125:6789/0 conn(0x7fba280e51a0 legacy=0x7fba280d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2685e700 1 -- 172.21.2.117:0/2551001603 >> v1:172.21.2.117:6789/0 conn(0x7fba280e4da0 legacy=0x7fba281b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2685e700 1 -- 172.21.2.117:0/2551001603 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba281b08e0 con 0x7fba2807da20 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2685e700 1 --2- 172.21.2.117:0/2551001603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba2807da20 0x7fba280771b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fba18006650 tx=0x7fba18009a00).ready entity=mon.0 client_cookie=b22be972f92a1831 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2585c700 1 -- 172.21.2.117:0/2551001603 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba18009df0 con 0x7fba2807da20 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2585c700 1 -- 172.21.2.117:0/2551001603 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180009b0 con 0x7fba2807da20 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.307 7fba2585c700 1 -- 172.21.2.117:0/2551001603 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba1800c2b0 con 0x7fba2807da20 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 -- 172.21.2.117:0/2551001603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba2807da20 msgr2=0x7fba280771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 --2- 172.21.2.117:0/2551001603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba2807da20 0x7fba280771b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fba18006650 tx=0x7fba18009a00).stop 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 -- 172.21.2.117:0/2551001603 shutdown_connections 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 --2- 172.21.2.117:0/2551001603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba2807da20 0x7fba280771b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 -- 172.21.2.117:0/2551001603 >> 172.21.2.117:0/2551001603 conn(0x7fba280ce450 msgr2=0x7fba280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 -- 172.21.2.117:0/2551001603 shutdown_connections 2021-02-10T11:23:04.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.308 7fba2cc10700 1 -- 172.21.2.117:0/2551001603 wait complete. 2021-02-10T11:23:04.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 Processor -- start 2021-02-10T11:23:04.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 -- start start 2021-02-10T11:23:04.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba28190450 0x7fba28190850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba280a6c10 0x7fba280a7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28199d10 con 0x7fba280a61f0 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28104230 con 0x7fba28190450 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.323 7fba2cc10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba28104450 con 0x7fba280a6c10 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41944/0 (socket says 172.21.2.117:41944) 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 -- 172.21.2.117:0/3910098310 learned_addr learned my addr 172.21.2.117:0/3910098310 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2705f700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba280a6c10 0x7fba280a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2685e700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba28190450 0x7fba28190850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 -- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba280a6c10 msgr2=0x7fba280a7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba280a6c10 0x7fba280a7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 -- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba28190450 msgr2=0x7fba28190850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba28190450 0x7fba28190850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 -- 172.21.2.117:0/3910098310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28104670 con 0x7fba280a61f0 2021-02-10T11:23:04.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba2605d700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fba10001c60 tx=0x7fba1000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.324 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba1000acc0 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10004350 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28104670 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba100047b0 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba28104670 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba10007f30 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba1f7fe700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 0x7fba0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.325 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fba10032790 con 0x7fba280a61f0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.326 7fba2685e700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 0x7fba0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.326 7fba2685e700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 0x7fba0c01ac10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.356 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba28199d10 con 0x7fba280a61f0 2021-02-10T11:23:04.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.359 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba10004140 con 0x7fba280a61f0 2021-02-10T11:23:04.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.484 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fba28199d10 con 0x7fba280a61f0 2021-02-10T11:23:04.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.484 7fba1f7fe700 1 -- 172.21.2.117:0/3910098310 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fba10007f30 con 0x7fba280a61f0 2021-02-10T11:23:04.481 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 msgr2=0x7fba0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 0x7fba0c01ac10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 msgr2=0x7fba280a65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fba10001c60 tx=0x7fba1000a8d0).stop 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 shutdown_connections 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba0c018730 0x7fba0c01ac10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba280a61f0 0x7fba280a65f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba280a6c10 0x7fba280a7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 --2- 172.21.2.117:0/3910098310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba28190450 0x7fba28190850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.486 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 >> 172.21.2.117:0/3910098310 conn(0x7fba280ab190 msgr2=0x7fba280b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.487 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 shutdown_connections 2021-02-10T11:23:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.487 7fba2cc10700 1 -- 172.21.2.117:0/3910098310 wait complete. 2021-02-10T11:23:04.491 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:04.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.908 7fad609e0700 1 Processor -- start 2021-02-10T11:23:04.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.908 7fad609e0700 1 -- start start 2021-02-10T11:23:04.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad609e0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad5c079000 0x7fad5c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad609e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 0x7fad5c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad609e0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c077a70 con 0x7fad5c07da20 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad609e0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c0e53c0 con 0x7fad5c079000 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad609e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c079400 con 0x7fad5c1b00f0 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad59e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad5c079000 0x7fad5c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad59e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad5c079000 0x7fad5c0e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41952/0 (socket says 172.21.2.117:41952) 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad59e1b700 1 -- 172.21.2.117:0/271689324 learned_addr learned my addr 172.21.2.117:0/271689324 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad5ae1d700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 0x7fad5c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.909 7fad59e1b700 1 -- 172.21.2.117:0/271689324 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 msgr2=0x7fad5c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad59e1b700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 0x7fad5c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad59e1b700 1 -- 172.21.2.117:0/271689324 >> v1:172.21.2.125:6789/0 conn(0x7fad5c07da20 legacy=0x7fad5c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad59e1b700 1 -- 172.21.2.117:0/271689324 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c07de20 con 0x7fad5c079000 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad5ae1d700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 0x7fad5c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad59e1b700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c079000 0x7fad5c0e4da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fad4c006700 tx=0x7fad4c009b80).ready entity=mon.0 client_cookie=dddfc116a28009ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad5961a700 1 -- 172.21.2.117:0/271689324 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad4c00c0c0 con 0x7fad5c079000 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad5961a700 1 -- 172.21.2.117:0/271689324 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00c340 con 0x7fad5c079000 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad5961a700 1 -- 172.21.2.117:0/271689324 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad4c00c860 con 0x7fad5c079000 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad609e0700 1 -- 172.21.2.117:0/271689324 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c079000 msgr2=0x7fad5c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.910 7fad609e0700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c079000 0x7fad5c0e4da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fad4c006700 tx=0x7fad4c009b80).stop 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 -- 172.21.2.117:0/271689324 shutdown_connections 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c1b00f0 0x7fad5c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 --2- 172.21.2.117:0/271689324 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c079000 0x7fad5c0e4da0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 -- 172.21.2.117:0/271689324 >> 172.21.2.117:0/271689324 conn(0x7fad5c0ce450 msgr2=0x7fad5c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 -- 172.21.2.117:0/271689324 shutdown_connections 2021-02-10T11:23:04.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.911 7fad609e0700 1 -- 172.21.2.117:0/271689324 wait complete. 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.926 7fad609e0700 1 Processor -- start 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.926 7fad609e0700 1 -- start start 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad5c0b3ff0 0x7fad5c0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c0c33d0 0x7fad5c0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c090600 con 0x7fad5c090de0 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c090860 con 0x7fad5c0b3ff0 2021-02-10T11:23:04.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad609e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0b3c70 con 0x7fad5c0c33d0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41960/0 (socket says 172.21.2.117:41960) 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 -- 172.21.2.117:0/1999639727 learned_addr learned my addr 172.21.2.117:0/1999639727 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad59e1b700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c0c33d0 0x7fad5c0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5a61c700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad5c0b3ff0 0x7fad5c0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 -- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c0c33d0 msgr2=0x7fad5c0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c0c33d0 0x7fad5c0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 -- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad5c0b3ff0 msgr2=0x7fad5c0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad5c0b3ff0 0x7fad5c0b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.927 7fad5ae1d700 1 -- 172.21.2.117:0/1999639727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c091800 con 0x7fad5c090de0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad5ae1d700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fad54009980 tx=0x7fad5400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5400c2f0 con 0x7fad5c090de0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54006790 con 0x7fad5c090de0 2021-02-10T11:23:04.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad5400c870 con 0x7fad5c090de0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad609e0700 1 -- 172.21.2.117:0/1999639727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c091800 con 0x7fad5c090de0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.928 7fad609e0700 1 -- 172.21.2.117:0/1999639727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c091d90 con 0x7fad5c090de0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.929 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad540011f0 con 0x7fad5c090de0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.929 7fad52ffd700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 0x7fad4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.929 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fad54032af0 con 0x7fad5c090de0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.929 7fad5a61c700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 0x7fad4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.929 7fad5a61c700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 0x7fad4001abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:04.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.959 7fad609e0700 1 -- 172.21.2.117:0/1999639727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c103370 con 0x7fad5c090de0 2021-02-10T11:23:04.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:04.961 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad5402d310 con 0x7fad5c090de0 2021-02-10T11:23:05.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.087 7fad609e0700 1 -- 172.21.2.117:0/1999639727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fad5c077a70 con 0x7fad5c090de0 2021-02-10T11:23:05.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.088 7fad52ffd700 1 -- 172.21.2.117:0/1999639727 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fad5402d310 con 0x7fad5c090de0 2021-02-10T11:23:05.085 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 msgr2=0x7fad4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 0x7fad4001abc0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 msgr2=0x7fad5c0911e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fad54009980 tx=0x7fad5400bcd0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 shutdown_connections 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad400186c0 0x7fad4001abc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad5c090de0 0x7fad5c0911e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad5c0c33d0 0x7fad5c0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 --2- 172.21.2.117:0/1999639727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad5c0b3ff0 0x7fad5c0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 >> 172.21.2.117:0/1999639727 conn(0x7fad5c0a1420 msgr2=0x7fad5c09efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 shutdown_connections 2021-02-10T11:23:05.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.090 7fad609e0700 1 -- 172.21.2.117:0/1999639727 wait complete. 2021-02-10T11:23:05.095 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.509 7f87c6146700 1 Processor -- start 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.509 7f87c6146700 1 -- start start 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87c6146700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87c00d1530 0x7f87c00d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87c6146700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c0077a70 con 0x7f87c007da20 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87c6146700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c0079400 con 0x7f87c0079000 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87c6146700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c007de20 con 0x7f87c00d1530 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87bf7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f87c007da20 0x7f87c00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42216/0 (socket says 172.21.2.117:42216) 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87bf7fe700 1 -- 172.21.2.117:0/516774390 learned_addr learned my addr 172.21.2.117:0/516774390 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.510 7f87bffff700 1 --2- 172.21.2.117:0/516774390 >> v2:172.21.2.125:3300/0 conn(0x7f87c00d1530 0x7f87c00d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87be7fc700 1 -- 172.21.2.117:0/516774390 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219309073 0 0) 0x7f87b0000ce0 con 0x7f87c007da20 2021-02-10T11:23:05.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87be7fc700 1 -- 172.21.2.117:0/516774390 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ac002fe0 con 0x7f87c007da20 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87bffff700 1 -- 172.21.2.117:0/516774390 >> v1:172.21.2.125:6789/0 conn(0x7f87c0079000 legacy=0x7f87c00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87bffff700 1 -- 172.21.2.117:0/516774390 >> v1:172.21.2.117:6789/0 conn(0x7f87c007da20 legacy=0x7f87c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87bffff700 1 -- 172.21.2.117:0/516774390 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c01b00f0 con 0x7f87c00d1530 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87bffff700 1 --2- 172.21.2.117:0/516774390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00d1530 0x7f87c00d1930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f87b8006e50 tx=0x7f87b8009e70).ready entity=mon.1 client_cookie=7ab87bac609ecb7d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87be7fc700 1 -- 172.21.2.117:0/516774390 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b8006a20 con 0x7f87c00d1530 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87be7fc700 1 -- 172.21.2.117:0/516774390 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8006790 con 0x7f87c00d1530 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.511 7f87be7fc700 1 -- 172.21.2.117:0/516774390 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b800b700 con 0x7f87c00d1530 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 -- 172.21.2.117:0/516774390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00d1530 msgr2=0x7f87c00d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 --2- 172.21.2.117:0/516774390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00d1530 0x7f87c00d1930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f87b8006e50 tx=0x7f87b8009e70).stop 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 -- 172.21.2.117:0/516774390 shutdown_connections 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 --2- 172.21.2.117:0/516774390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00d1530 0x7f87c00d1930 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 -- 172.21.2.117:0/516774390 >> 172.21.2.117:0/516774390 conn(0x7f87c00ce450 msgr2=0x7f87c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 -- 172.21.2.117:0/516774390 shutdown_connections 2021-02-10T11:23:05.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.512 7f87c6146700 1 -- 172.21.2.117:0/516774390 wait complete. 2021-02-10T11:23:05.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 Processor -- start 2021-02-10T11:23:05.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 -- start start 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 0x7f87c0107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87c00b1340 0x7f87c018b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00b18c0 con 0x7f87c018b830 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c018ac90 con 0x7f87c00b0f40 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.527 7f87c6146700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c0107f50 con 0x7f87c00b1340 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41974/0 (socket says 172.21.2.117:41974) 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 -- 172.21.2.117:0/1937113017 learned_addr learned my addr 172.21.2.117:0/1937113017 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bf7fe700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 0x7f87c0107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 -- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87c00b1340 msgr2=0x7f87c018b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87c00b1340 0x7f87c018b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:05.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 -- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 msgr2=0x7f87c0107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 0x7f87c0107930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 -- 172.21.2.117:0/1937113017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0108130 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bf7fe700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 0x7f87c0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87bffff700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f87b8009f70 tx=0x7f87b800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.528 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b8001070 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b80012f0 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87b8001810 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87c6146700 1 -- 172.21.2.117:0/1937113017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c0108130 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87c6146700 1 -- 172.21.2.117:0/1937113017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c0188570 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87b8000f90 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87b7fff700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 0x7f87a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.529 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f87b8032a30 con 0x7f87c018b830 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.530 7f87bf7fe700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 0x7f87a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:05.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.530 7f87bf7fe700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 0x7f87a401abc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.560 7f87c6146700 1 -- 172.21.2.117:0/1937113017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c0077a70 con 0x7f87c018b830 2021-02-10T11:23:05.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.562 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87b80093d0 con 0x7f87c018b830 2021-02-10T11:23:05.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.689 7f87c6146700 1 -- 172.21.2.117:0/1937113017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f87c0077a70 con 0x7f87c018b830 2021-02-10T11:23:05.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.689 7f87b7fff700 1 -- 172.21.2.117:0/1937113017 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f87b8000f90 con 0x7f87c018b830 2021-02-10T11:23:05.686 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:05.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 msgr2=0x7f87a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 0x7f87a401abc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 msgr2=0x7f87c0187bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:05.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f87b8009f70 tx=0x7f87b800bcc0).stop 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 shutdown_connections 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87a40186c0 0x7f87a401abc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87c018b830 0x7f87c0187bf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87c00b1340 0x7f87c018b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 --2- 172.21.2.117:0/1937113017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87c00b0f40 0x7f87c0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 >> 172.21.2.117:0/1937113017 conn(0x7f87c00a1880 msgr2=0x7f87c00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:05.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 shutdown_connections 2021-02-10T11:23:05.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:05.691 7f87c6146700 1 -- 172.21.2.117:0/1937113017 wait complete. 2021-02-10T11:23:05.696 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:23:05.696 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.108 7f236c430700 1 Processor -- start 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.108 7f236c430700 1 -- start start 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f236407da20 0x7f23640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23640e5190 0x7f2364079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2364077a70 con 0x7f23640e4d90 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640e5c10 con 0x7f23640e5190 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23640777d0 con 0x7f236407da20 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23640e5190 0x7f2364079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23640e5190 0x7f2364079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41978/0 (socket says 172.21.2.117:41978) 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 -- 172.21.2.117:0/896501536 learned_addr learned my addr 172.21.2.117:0/896501536 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 -- 172.21.2.117:0/896501536 >> v2:172.21.2.125:3300/0 conn(0x7f236407da20 msgr2=0x7f23640771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 --2- 172.21.2.117:0/896501536 >> v2:172.21.2.125:3300/0 conn(0x7f236407da20 0x7f23640771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 -- 172.21.2.117:0/896501536 >> v1:172.21.2.125:6789/0 conn(0x7f23640e4d90 legacy=0x7f23641b00e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 -- 172.21.2.117:0/896501536 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23641b08d0 con 0x7f23640e5190 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236aa4d700 1 --2- 172.21.2.117:0/896501536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23640e5190 0x7f2364079000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f235c006f00 tx=0x7f235c009f10).ready entity=mon.0 client_cookie=98e5093e65f20ff1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236924a700 1 -- 172.21.2.117:0/896501536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235c00b3b0 con 0x7f23640e5190 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236924a700 1 -- 172.21.2.117:0/896501536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00b630 con 0x7f23640e5190 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236924a700 1 -- 172.21.2.117:0/896501536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235c00bb10 con 0x7f23640e5190 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 -- 172.21.2.117:0/896501536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23640e5190 msgr2=0x7f2364079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.109 7f236c430700 1 --2- 172.21.2.117:0/896501536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23640e5190 0x7f2364079000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f235c006f00 tx=0x7f235c009f10).stop 2021-02-10T11:23:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 -- 172.21.2.117:0/896501536 shutdown_connections 2021-02-10T11:23:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 --2- 172.21.2.117:0/896501536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23640e5190 0x7f2364079000 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 --2- 172.21.2.117:0/896501536 >> v2:172.21.2.125:3300/0 conn(0x7f236407da20 0x7f23640771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 -- 172.21.2.117:0/896501536 >> 172.21.2.117:0/896501536 conn(0x7f23640ce440 msgr2=0x7f23640e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 -- 172.21.2.117:0/896501536 shutdown_connections 2021-02-10T11:23:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.110 7f236c430700 1 -- 172.21.2.117:0/896501536 wait complete. 2021-02-10T11:23:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 Processor -- start 2021-02-10T11:23:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 -- start start 2021-02-10T11:23:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2364109bf0 0x7f2364109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f236409da70 0x7f236409de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236410a610 con 0x7f236419d3b0 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236409e490 con 0x7f236409da70 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236c430700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236409e6b0 con 0x7f2364109bf0 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:41982/0 (socket says 172.21.2.117:41982) 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 -- 172.21.2.117:0/289379850 learned_addr learned my addr 172.21.2.117:0/289379850 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 -- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2364109bf0 msgr2=0x7f2364109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2364109bf0 0x7f2364109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 -- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f236409da70 msgr2=0x7f236409de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f236409da70 0x7f236409de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.125 7f236a24c700 1 -- 172.21.2.117:0/289379850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236409e8d0 con 0x7f236419d3b0 2021-02-10T11:23:06.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f236a24c700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f235800b5c0 tx=0x7f235800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235800bd50 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2358006030 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23580054d0 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f236c430700 1 -- 172.21.2.117:0/289379850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236409e8d0 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f236c430700 1 -- 172.21.2.117:0/289379850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236409e8d0 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.126 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2358006210 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.127 7f2356ffd700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.127 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2358032560 con 0x7f236419d3b0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.127 7f2369a4b700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:06.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.127 7f2369a4b700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:06.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.158 7f236c430700 1 -- 172.21.2.117:0/289379850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2364108ba0 con 0x7f236419d3b0 2021-02-10T11:23:06.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.161 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2358025fb0 con 0x7f236419d3b0 2021-02-10T11:23:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.288 7f236c430700 1 -- 172.21.2.117:0/289379850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2364077a70 con 0x7f236419d3b0 2021-02-10T11:23:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.289 7f2356ffd700 1 -- 172.21.2.117:0/289379850 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7f2358025fb0 con 0x7f236419d3b0 2021-02-10T11:23:06.285 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:06.286 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:23:06.286 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 msgr2=0x7f234c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 msgr2=0x7f236419d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f235800b5c0 tx=0x7f235800b960).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 shutdown_connections 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f236409da70 0x7f236409de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2364109bf0 0x7f2364109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 --2- 172.21.2.117:0/289379850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236419d3b0 0x7f236419d7b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:06.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 >> 172.21.2.117:0/289379850 conn(0x7f236419a700 msgr2=0x7f2364199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:06.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 shutdown_connections 2021-02-10T11:23:06.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:06.290 7f236c430700 1 -- 172.21.2.117:0/289379850 wait complete. 2021-02-10T11:23:06.497 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:32.545631', 'modified': '2021-02-10 11:22:41.961589', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4229': {'gid': 4229, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/921969069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 921969069}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 921969069}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:06.497 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:06.498 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4229, 2021-02-10T11:23:06.895 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 25.937926959999999, 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 6, 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 22, 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 22, 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 30.299069475 2021-02-10T11:23:06.896 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:06.899 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4229, "want_state": "up:active", "state": "up:active", "rank_uptime": 25.937926959999999, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "uptime": 30.299069475 } 2021-02-10T11:23:06.899 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:06.899 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:23:06.899 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:23:06.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:06.900 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:07.234 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:23:07.577 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:07.577 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:07.920 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:07.920 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:07.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:07.921 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:23:08.256 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:23:08.256 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:23:08.256 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:08.259 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:23:08.603 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:08.603 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:08.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[40494]: starting ceph client 2021-02-10T11:23:08.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:23:08.671 7f836a90df80 -1 init, newargv = 0x55797d7dbf70 newargc=9 2021-02-10T11:23:08.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[40494]: starting fuse 2021-02-10T11:23:08.945 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:08.945 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:08.946 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:08.946 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:23:09.281 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:23:09.281 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:23:09.281 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:23:09.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:23:09.282 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:23:09.283 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:23:09.644 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.40494.asok 2021-02-10T11:23:09.647 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.40494.asok 2021-02-10T11:23:09.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:09.647 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.40494.asok status 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "40494", 2021-02-10T11:23:10.035 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 4596, 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 4596 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2602703055 2021-02-10T11:23:10.036 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2602703055 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.4596 172.21.2.117:0/2602703055", 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2602703055", 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 6, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 22, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:23:10.037 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:23:10.039 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:10.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.450 7ff1cbf89700 1 Processor -- start 2021-02-10T11:23:10.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.450 7ff1cbf89700 1 -- start start 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1cbf89700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff1c4079000 0x7ff1c40e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1cbf89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c41b00e0 0x7ff1c41b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1cbf89700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c4077a70 con 0x7ff1c407da20 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1cbf89700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40e53b0 con 0x7ff1c4079000 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1cbf89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4079400 con 0x7ff1c41b00e0 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1c9da5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff1c407da20 0x7ff1c40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42254/0 (socket says 172.21.2.117:42254) 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1c9da5700 1 -- 172.21.2.117:0/2764134142 learned_addr learned my addr 172.21.2.117:0/2764134142 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524440569 0 0) 0x7ff1c0000ce0 con 0x7ff1c407da20 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b0002fe0 con 0x7ff1c407da20 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1c95a4700 1 --2- 172.21.2.117:0/2764134142 >> v2:172.21.2.125:3300/0 conn(0x7ff1c4079000 0x7ff1c40e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.451 7ff1ca5a6700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c41b00e0 0x7ff1c41b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1693396314 0 0) 0x7ff1c00010e0 con 0x7ff1c407da20 2021-02-10T11:23:10.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c41b00e0 msgr2=0x7ff1c41b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c41b00e0 0x7ff1c41b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 >> v2:172.21.2.125:3300/0 conn(0x7ff1c4079000 msgr2=0x7ff1c40e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 --2- 172.21.2.117:0/2764134142 >> v2:172.21.2.125:3300/0 conn(0x7ff1c4079000 0x7ff1c40e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c407de20 con 0x7ff1c407da20 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff1c0001660 con 0x7ff1c407da20 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 >> v1:172.21.2.117:6789/0 conn(0x7ff1c407da20 legacy=0x7ff1c40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1b0003a70 0x7ff1b0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1b004a7b0 0x7ff1b004ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1b004d5b0 0x7ff1b004faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b00502f0 con 0x7ff1b004a7b0 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0050510 con 0x7ff1b0003a70 2021-02-10T11:23:10.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0050730 con 0x7ff1b004d5b0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c9da5700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1b0003a70 0x7ff1b0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1b004a7b0 0x7ff1b004ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1ca5a6700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1b004d5b0 0x7ff1b004faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 -- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1b004d5b0 msgr2=0x7ff1b004faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1b004d5b0 0x7ff1b004faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 -- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1b0003a70 msgr2=0x7ff1b0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1b0003a70 0x7ff1b0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 -- 172.21.2.117:0/2764134142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b00509d0 con 0x7ff1b004a7b0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c9da5700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1b0003a70 0x7ff1b0005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c95a4700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1b004a7b0 0x7ff1b004ccb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff1b400c8f0 tx=0x7ff1b400cc90).ready entity=mon.0 client_cookie=11a205ce81ceacee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.452 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b400cdd0 con 0x7ff1b004a7b0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b4009310 con 0x7ff1b004a7b0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1c8da3700 1 -- 172.21.2.117:0/2764134142 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b4009830 con 0x7ff1b004a7b0 2021-02-10T11:23:10.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 -- 172.21.2.117:0/2764134142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1b004a7b0 msgr2=0x7ff1b004ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 --2- 172.21.2.117:0/2764134142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1b004a7b0 0x7ff1b004ccb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff1b400c8f0 tx=0x7ff1b400cc90).stop 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1ca5a6700 1 -- 172.21.2.117:0/2764134142 reap_dead start 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 -- 172.21.2.117:0/2764134142 shutdown_connections 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 -- 172.21.2.117:0/2764134142 >> 172.21.2.117:0/2764134142 conn(0x7ff1c40ce440 msgr2=0x7ff1c40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 -- 172.21.2.117:0/2764134142 shutdown_connections 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.453 7ff1cbf89700 1 -- 172.21.2.117:0/2764134142 wait complete. 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 Processor -- start 2021-02-10T11:23:10.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 -- start start 2021-02-10T11:23:10.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40b8bd0 0x7ff1c4192ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c4106ad0 0x7ff1c4193b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4193590 con 0x7ff1c418ebe0 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c4106ed0 con 0x7ff1c4106ad0 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.468 7ff1cbf89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c41931c0 con 0x7ff1c40b8bd0 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42016/0 (socket says 172.21.2.117:42016) 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 -- 172.21.2.117:0/907461729 learned_addr learned my addr 172.21.2.117:0/907461729 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 -- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40b8bd0 msgr2=0x7ff1c4192ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40b8bd0 0x7ff1c4192ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 -- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c4106ad0 msgr2=0x7ff1c4193b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c4106ad0 0x7ff1c4193b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 -- 172.21.2.117:0/907461729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4194130 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1c95a4700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff1b4006de0 tx=0x7ff1b400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b4000c90 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b400c7c0 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b400cce0 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4194130 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.469 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c4194130 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.470 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1b4000c90 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.470 7ff1ba7fc700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 0x7ff1ac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.470 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7ff1b4032900 con 0x7ff1c418ebe0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.470 7ff1c9da5700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 0x7ff1ac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.470 7ff1c9da5700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 0x7ff1ac01abd0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:10.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.501 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c4077a70 con 0x7ff1c418ebe0 2021-02-10T11:23:10.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.503 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1b40069d0 con 0x7ff1c418ebe0 2021-02-10T11:23:10.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.627 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1c4077a70 con 0x7ff1c418ebe0 2021-02-10T11:23:10.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.628 7ff1ba7fc700 1 -- 172.21.2.117:0/907461729 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7ff1b4000c90 con 0x7ff1c418ebe0 2021-02-10T11:23:10.624 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:10.625 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.629 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 msgr2=0x7ff1ac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.629 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 0x7ff1ac01abd0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.629 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 msgr2=0x7ff1c41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.629 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff1b4006de0 tx=0x7ff1b400b300).stop 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 shutdown_connections 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac0186d0 0x7ff1ac01abd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c418ebe0 0x7ff1c41064b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:10.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c4106ad0 0x7ff1c4193b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 --2- 172.21.2.117:0/907461729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40b8bd0 0x7ff1c4192ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:10.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 >> 172.21.2.117:0/907461729 conn(0x7ff1c419f510 msgr2=0x7ff1c41abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:10.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 shutdown_connections 2021-02-10T11:23:10.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:10.630 7ff1cbf89700 1 -- 172.21.2.117:0/907461729 wait complete. 2021-02-10T11:23:10.634 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:10.635 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 Processor -- start 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 -- start start 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1424079000 0x7f14240e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14241b00f0 0x7f14241b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1424077a70 con 0x7f142407da20 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14240e53c0 con 0x7f14241b00f0 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1429c9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1424079400 con 0x7f1424079000 2021-02-10T11:23:11.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f1423fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14241b00f0 0x7f14241b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.051 7f14237fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f142407da20 0x7f14240771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42272/0 (socket says 172.21.2.117:42272) 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f14237fe700 1 -- 172.21.2.117:0/1781419322 learned_addr learned my addr 172.21.2.117:0/1781419322 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1422ffd700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1424079000 0x7f14240e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1423fff700 1 -- 172.21.2.117:0/1781419322 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1424079000 msgr2=0x7f14240e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1423fff700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1424079000 0x7f14240e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1423fff700 1 -- 172.21.2.117:0/1781419322 >> v1:172.21.2.117:6789/0 conn(0x7f142407da20 legacy=0x7f14240771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1423fff700 1 -- 172.21.2.117:0/1781419322 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142407de20 con 0x7f14241b00f0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1423fff700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14241b00f0 0x7f14241b04f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f141c006e30 tx=0x7f141c009e60).ready entity=mon.0 client_cookie=1dff4305cb15d228 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f14227fc700 1 -- 172.21.2.117:0/1781419322 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141c006a50 con 0x7f14241b00f0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f14227fc700 1 -- 172.21.2.117:0/1781419322 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c00b300 con 0x7f14241b00f0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f14227fc700 1 -- 172.21.2.117:0/1781419322 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141c00b7e0 con 0x7f14241b00f0 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1429c9b700 1 -- 172.21.2.117:0/1781419322 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14241b00f0 msgr2=0x7f14241b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.052 7f1429c9b700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14241b00f0 0x7f14241b04f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f141c006e30 tx=0x7f141c009e60).stop 2021-02-10T11:23:11.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 -- 172.21.2.117:0/1781419322 shutdown_connections 2021-02-10T11:23:11.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14241b00f0 0x7f14241b04f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 --2- 172.21.2.117:0/1781419322 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1424079000 0x7f14240e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 -- 172.21.2.117:0/1781419322 >> 172.21.2.117:0/1781419322 conn(0x7f14240ce450 msgr2=0x7f14240e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 -- 172.21.2.117:0/1781419322 shutdown_connections 2021-02-10T11:23:11.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.053 7f1429c9b700 1 -- 172.21.2.117:0/1781419322 wait complete. 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 Processor -- start 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 -- start start 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14240a41c0 0x7f14240fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14240c2340 0x7f14240c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240b8010 con 0x7f14240b7390 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14240b8550 con 0x7f14240a41c0 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1429c9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14240b8770 con 0x7f14240c2340 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1423fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1423fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42028/0 (socket says 172.21.2.117:42028) 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.068 7f1423fff700 1 -- 172.21.2.117:0/850099024 learned_addr learned my addr 172.21.2.117:0/850099024 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 -- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14240c2340 msgr2=0x7f14240c2740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14240c2340 0x7f14240c2740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 -- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14240a41c0 msgr2=0x7f14240fe0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14240a41c0 0x7f14240fe0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 -- 172.21.2.117:0/850099024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14240b8990 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1423fff700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f141c00b970 tx=0x7f141c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141c009380 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c005820 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141c005d40 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1429c9b700 1 -- 172.21.2.117:0/850099024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14240b8990 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.069 7f1429c9b700 1 -- 172.21.2.117:0/850099024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14240b8f20 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.070 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f141c005cd0 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.070 7f141bfff700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 0x7f140801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.070 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f141c032a80 con 0x7f14240b7390 2021-02-10T11:23:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.070 7f14237fe700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 0x7f140801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.071 7f14237fe700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 0x7f140801abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.102 7f1429c9b700 1 -- 172.21.2.117:0/850099024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14241b4f00 con 0x7f14240b7390 2021-02-10T11:23:11.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.104 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f141c005ac0 con 0x7f14240b7390 2021-02-10T11:23:11.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.230 7f1429c9b700 1 -- 172.21.2.117:0/850099024 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f1424077a70 con 0x7f14240b7390 2021-02-10T11:23:11.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.230 7f141bfff700 1 -- 172.21.2.117:0/850099024 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f141c005cd0 con 0x7f14240b7390 2021-02-10T11:23:11.227 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:11.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 -- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 msgr2=0x7f140801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 0x7f140801abc0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 -- 172.21.2.117:0/850099024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 msgr2=0x7f14240b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f141c00b970 tx=0x7f141c00bd10).stop 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 -- 172.21.2.117:0/850099024 shutdown_connections 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14080186c0 0x7f140801abc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14240b7390 0x7f14240b7790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14240c2340 0x7f14240c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 --2- 172.21.2.117:0/850099024 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14240a41c0 0x7f14240fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 -- 172.21.2.117:0/850099024 >> 172.21.2.117:0/850099024 conn(0x7f142408bbc0 msgr2=0x7f14240c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.232 7f1429c9b700 1 -- 172.21.2.117:0/850099024 shutdown_connections 2021-02-10T11:23:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.233 7f1429c9b700 1 -- 172.21.2.117:0/850099024 wait complete. 2021-02-10T11:23:11.237 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:11.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.653 7f2dc102e700 1 Processor -- start 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dc102e700 1 -- start start 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dc102e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2dbc07da20 0x7f2dbc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dc102e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0d1d20 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dc102e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc077a70 con 0x7f2dbc0e51a0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dc102e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dbc0777d0 con 0x7f2dbc07da20 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2db259c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2dbc0e4da0 0x7f2dbc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42286/0 (socket says 172.21.2.117:42286) 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2db259c700 1 -- 172.21.2.117:0/1306411512 learned_addr learned my addr 172.21.2.117:0/1306411512 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dbad9d700 1 --2- 172.21.2.117:0/1306411512 >> v2:172.21.2.125:3300/0 conn(0x7f2dbc07da20 0x7f2dbc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934192273 0 0) 0x7f2dac001180 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.654 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8002fe0 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2926550571 0 0) 0x7f2dac0010e0 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 >> v2:172.21.2.125:3300/0 conn(0x7f2dbc07da20 msgr2=0x7f2dbc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 --2- 172.21.2.117:0/1306411512 >> v2:172.21.2.125:3300/0 conn(0x7f2dbc07da20 0x7f2dbc0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 >> v1:172.21.2.125:6789/0 conn(0x7f2dbc0e51a0 legacy=0x7f2dbc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc1b08e0 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2dac001600 con 0x7f2dbc0e4da0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 >> v1:172.21.2.117:6789/0 conn(0x7f2dbc0e4da0 legacy=0x7f2dbc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2da8003a70 0x7f2da8005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2da80067a0 0x7f2da8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 0x7f2da800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da800c310 con 0x7f2da80095a0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da800c530 con 0x7f2da8003a70 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da800c750 con 0x7f2da80067a0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 0x7f2da800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2db259c700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2da80067a0 0x7f2da8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbad9d700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2da8003a70 0x7f2da8005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 -- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2da80067a0 msgr2=0x7f2da8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2da80067a0 0x7f2da8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 -- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2da8003a70 msgr2=0x7f2da8005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2da8003a70 0x7f2da8005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 -- 172.21.2.117:0/1306411512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8050990 con 0x7f2da80095a0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 -- 172.21.2.117:0/1306411512 reap_dead start 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.655 7f2dbb59e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 0x7f2da800bac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2db4006e40 tx=0x7f2db4009e30).ready entity=mon.0 client_cookie=6633cfd90e57d0ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2db4000f90 con 0x7f2da80095a0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db400b300 con 0x7f2da80095a0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dba59c700 1 -- 172.21.2.117:0/1306411512 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2db400b820 con 0x7f2da80095a0 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 -- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 msgr2=0x7f2da800bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 0x7f2da800bac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2db4006e40 tx=0x7f2db4009e30).stop 2021-02-10T11:23:11.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 -- 172.21.2.117:0/1306411512 shutdown_connections 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 --2- 172.21.2.117:0/1306411512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2da80095a0 0x7f2da800bac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 -- 172.21.2.117:0/1306411512 >> 172.21.2.117:0/1306411512 conn(0x7f2dbc0ce450 msgr2=0x7f2dbc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 -- 172.21.2.117:0/1306411512 shutdown_connections 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.656 7f2dc102e700 1 -- 172.21.2.117:0/1306411512 wait complete. 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.671 7f2dc102e700 1 Processor -- start 2021-02-10T11:23:11.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.671 7f2dc102e700 1 -- start start 2021-02-10T11:23:11.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2dbc1a46b0 0x7f2dbc190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2dbc0a46c0 0x7f2dbc0a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0c3040 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc190a20 con 0x7f2dbc1a46b0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dc102e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dbc0a54e0 con 0x7f2dbc0a46c0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42050/0 (socket says 172.21.2.117:42050) 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 -- 172.21.2.117:0/4102614158 learned_addr learned my addr 172.21.2.117:0/4102614158 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2db259c700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2dbc0a46c0 0x7f2dbc0a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbad9d700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2dbc1a46b0 0x7f2dbc190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 -- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2dbc0a46c0 msgr2=0x7f2dbc0a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2dbc0a46c0 0x7f2dbc0a4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 -- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2dbc1a46b0 msgr2=0x7f2dbc190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2dbc1a46b0 0x7f2dbc190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.672 7f2dbb59e700 1 -- 172.21.2.117:0/4102614158 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0a5700 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2dbb59e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2db400b970 tx=0x7f2db400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2db40093d0 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db4003020 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2db4005a20 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc0a5700 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc0a6990 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2db4005830 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 0x7f2da001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.673 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2db4032b00 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.674 7f2dbad9d700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 0x7f2da001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:11.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.674 7f2dbad9d700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 0x7f2da001abd0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:11.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.704 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc077a70 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.707 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2db4000f90 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.835 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2dbc077a70 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.835 7f2db3fff700 1 -- 172.21.2.117:0/4102614158 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2db4005830 con 0x7f2dbc0a50e0 2021-02-10T11:23:11.832 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 msgr2=0x7f2da001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 0x7f2da001abd0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 msgr2=0x7f2dbc0a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2db400b970 tx=0x7f2db400bd10).stop 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 shutdown_connections 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2da00186d0 0x7f2da001abd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2dbc0a50e0 0x7f2dbc0a6080 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2dbc0a46c0 0x7f2dbc0a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 --2- 172.21.2.117:0/4102614158 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2dbc1a46b0 0x7f2dbc190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:11.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.837 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 >> 172.21.2.117:0/4102614158 conn(0x7f2dbc0b5140 msgr2=0x7f2dbc0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:11.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.838 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 shutdown_connections 2021-02-10T11:23:11.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:11.838 7f2dc102e700 1 -- 172.21.2.117:0/4102614158 wait complete. 2021-02-10T11:23:11.842 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:12.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.257 7f88fdc26700 1 Processor -- start 2021-02-10T11:23:12.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.257 7f88fdc26700 1 -- start start 2021-02-10T11:23:12.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88fdc26700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f88f807da20 0x7f88f80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88fdc26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f80e4da0 0x7f88f81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88fdc26700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f8077a70 con 0x7f88f80e51a0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88fdc26700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f80777d0 con 0x7f88f807da20 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88fdc26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f807de20 con 0x7f88f80e4da0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f88f80e51a0 0x7f88f8077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42304/0 (socket says 172.21.2.117:42304) 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f7fff700 1 -- 172.21.2.117:0/2003489365 learned_addr learned my addr 172.21.2.117:0/2003489365 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439141801 0 0) 0x7f88f0001500 con 0x7f88f80e51a0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e0002fe0 con 0x7f88f80e51a0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f6ffd700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f80e4da0 0x7f88f81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f77fe700 1 --2- 172.21.2.117:0/2003489365 >> v2:172.21.2.125:3300/0 conn(0x7f88f807da20 0x7f88f80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.258 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3597596283 0 0) 0x7f88f0001460 con 0x7f88f80e51a0 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f80e4da0 msgr2=0x7f88f81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f80e4da0 0x7f88f81b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 >> v2:172.21.2.125:3300/0 conn(0x7f88f807da20 msgr2=0x7f88f80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 --2- 172.21.2.117:0/2003489365 >> v2:172.21.2.125:3300/0 conn(0x7f88f807da20 0x7f88f80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f81b0710 con 0x7f88f80e51a0 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f88f0001980 con 0x7f88f80e51a0 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 >> v1:172.21.2.117:6789/0 conn(0x7f88f80e51a0 legacy=0x7f88f8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88e0003b10 0x7f88e0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88e0006820 0x7f88e0008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88e00095b0 0x7f88e000bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e000c380 con 0x7f88e00095b0 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e000c5a0 con 0x7f88e0003b10 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88e00507d0 con 0x7f88e0006820 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f77fe700 1 --2- 172.21.2.117:0/2003489365 >> v2:172.21.2.125:3300/0 conn(0x7f88f807da20 0x7f88f80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88e0006820 0x7f88e0008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f7fff700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88e00095b0 0x7f88e000bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f77fe700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88e0003b10 0x7f88e0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 -- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88e0003b10 msgr2=0x7f88e0005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88e0003b10 0x7f88e0005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 -- 172.21.2.117:0/2003489365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88e00095b0 msgr2=0x7f88e000bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88e00095b0 0x7f88e000bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 -- 172.21.2.117:0/2003489365 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e0050a30 con 0x7f88e0006820 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f7fff700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88e00095b0 0x7f88e000bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f77fe700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88e0003b10 0x7f88e0005f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.259 7f88f6ffd700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88e0006820 0x7f88e0008d10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f88e8006ab0 tx=0x7f88e8009b40).ready entity=mon.2 client_cookie=64f6e79b87e53542 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e800c0c0 con 0x7f88e0006820 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e800c440 con 0x7f88e0006820 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88f67fc700 1 -- 172.21.2.117:0/2003489365 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88e800c8a0 con 0x7f88e0006820 2021-02-10T11:23:12.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 -- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88e0006820 msgr2=0x7f88e0008d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 --2- 172.21.2.117:0/2003489365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88e0006820 0x7f88e0008d10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f88e8006ab0 tx=0x7f88e8009b40).stop 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88f7fff700 1 -- 172.21.2.117:0/2003489365 reap_dead start 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 -- 172.21.2.117:0/2003489365 shutdown_connections 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 -- 172.21.2.117:0/2003489365 >> 172.21.2.117:0/2003489365 conn(0x7f88f80ce450 msgr2=0x7f88f80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 -- 172.21.2.117:0/2003489365 shutdown_connections 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.260 7f88fdc26700 1 -- 172.21.2.117:0/2003489365 wait complete. 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.275 7f88fdc26700 1 Processor -- start 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.275 7f88fdc26700 1 -- start start 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.275 7f88fdc26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f81a46b0 0x7f88f8190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.275 7f88fdc26700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88f80a46c0 0x7f88f80a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88fdc26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88fdc26700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80c3040 con 0x7f88f80a50e0 2021-02-10T11:23:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88fdc26700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f8190a20 con 0x7f88f80a46c0 2021-02-10T11:23:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88fdc26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80a54e0 con 0x7f88f81a46b0 2021-02-10T11:23:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42070/0 (socket says 172.21.2.117:42070) 2021-02-10T11:23:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 -- 172.21.2.117:0/3958862451 learned_addr learned my addr 172.21.2.117:0/3958862451 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:12.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 -- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f81a46b0 msgr2=0x7f88f8190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f81a46b0 0x7f88f8190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 -- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88f80a46c0 msgr2=0x7f88f80a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88f80a46c0 0x7f88f80a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 -- 172.21.2.117:0/3958862451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f80a5700 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88f7fff700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f88f000a900 tx=0x7f88f000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f0001480 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0004930 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.276 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f80a5700 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88f0004d80 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f80a5700 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88f000ade0 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88dffff700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 0x7f88d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f88f0032d10 con 0x7f88f80a50e0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.277 7f88f77fe700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 0x7f88d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.278 7f88f77fe700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 0x7f88d801abc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.307 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f8077a70 con 0x7f88f80a50e0 2021-02-10T11:23:12.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.310 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88f0004720 con 0x7f88f80a50e0 2021-02-10T11:23:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.437 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f88f8077a70 con 0x7f88f80a50e0 2021-02-10T11:23:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.438 7f88dffff700 1 -- 172.21.2.117:0/3958862451 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f88f0004930 con 0x7f88f80a50e0 2021-02-10T11:23:12.435 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.439 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 msgr2=0x7f88d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.439 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 0x7f88d801abc0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.439 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 msgr2=0x7f88f80a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.439 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f88f000a900 tx=0x7f88f000aca0).stop 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 shutdown_connections 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88d8018730 0x7f88d801abc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88f80a50e0 0x7f88f80a6080 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88f80a46c0 0x7f88f80a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 --2- 172.21.2.117:0/3958862451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88f81a46b0 0x7f88f8190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 >> 172.21.2.117:0/3958862451 conn(0x7f88f80b5140 msgr2=0x7f88f80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:12.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 shutdown_connections 2021-02-10T11:23:12.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.440 7f88fdc26700 1 -- 172.21.2.117:0/3958862451 wait complete. 2021-02-10T11:23:12.444 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:23:12.445 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:12.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.858 7fcb2d464700 1 Processor -- start 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.858 7fcb2d464700 1 -- start start 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.858 7fcb2d464700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb28079000 0x7fcb280e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb281b00e0 0x7fcb281b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb28077a70 con 0x7fcb2807da20 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb280e53b0 con 0x7fcb281b00e0 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb28079400 con 0x7fcb28079000 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb281b00e0 0x7fcb281b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb281b00e0 0x7fcb281b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42078/0 (socket says 172.21.2.117:42078) 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 -- 172.21.2.117:0/854100243 learned_addr learned my addr 172.21.2.117:0/854100243 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 -- 172.21.2.117:0/854100243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb28079000 msgr2=0x7fcb280e4d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:12.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 --2- 172.21.2.117:0/854100243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb28079000 0x7fcb280e4d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 -- 172.21.2.117:0/854100243 >> v1:172.21.2.125:6789/0 conn(0x7fcb2807da20 legacy=0x7fcb280771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 -- 172.21.2.117:0/854100243 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2807de20 con 0x7fcb281b00e0 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2787e700 1 --2- 172.21.2.117:0/854100243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb281b00e0 0x7fcb281b04e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fcb20006f00 tx=0x7fcb20009f10).ready entity=mon.0 client_cookie=48c3c1c1b4bd57ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2607b700 1 -- 172.21.2.117:0/854100243 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2000b3b0 con 0x7fcb281b00e0 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2607b700 1 -- 172.21.2.117:0/854100243 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2000b730 con 0x7fcb281b00e0 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2607b700 1 -- 172.21.2.117:0/854100243 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb2000bc50 con 0x7fcb281b00e0 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 -- 172.21.2.117:0/854100243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb281b00e0 msgr2=0x7fcb281b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.859 7fcb2d464700 1 --2- 172.21.2.117:0/854100243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb281b00e0 0x7fcb281b04e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fcb20006f00 tx=0x7fcb20009f10).stop 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 -- 172.21.2.117:0/854100243 shutdown_connections 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 --2- 172.21.2.117:0/854100243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb281b00e0 0x7fcb281b04e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 --2- 172.21.2.117:0/854100243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb28079000 0x7fcb280e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 -- 172.21.2.117:0/854100243 >> 172.21.2.117:0/854100243 conn(0x7fcb280ce440 msgr2=0x7fcb280e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 -- 172.21.2.117:0/854100243 shutdown_connections 2021-02-10T11:23:12.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.860 7fcb2d464700 1 -- 172.21.2.117:0/854100243 wait complete. 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.874 7fcb2d464700 1 Processor -- start 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 -- start start 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb280b7a90 0x7fcb280b7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb281096b0 0x7fcb2810a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb281ad470 con 0x7fcb28108c90 2021-02-10T11:23:12.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb28108170 con 0x7fcb280b7a90 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2d464700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb281acfb0 con 0x7fcb281096b0 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42084/0 (socket says 172.21.2.117:42084) 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2687c700 1 -- 172.21.2.117:0/3838971767 learned_addr learned my addr 172.21.2.117:0/3838971767 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2787e700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb281096b0 0x7fcb2810a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.875 7fcb2707d700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb280b7a90 0x7fcb280b7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 -- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb281096b0 msgr2=0x7fcb2810a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb281096b0 0x7fcb2810a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 -- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb280b7a90 msgr2=0x7fcb280b7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb280b7a90 0x7fcb280b7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 -- 172.21.2.117:0/3838971767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb28109ab0 con 0x7fcb28108c90 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2687c700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fcb1000b5c0 tx=0x7fcb1000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb1000bd50 con 0x7fcb28108c90 2021-02-10T11:23:12.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10006030 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb28109ab0 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb100054d0 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.876 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb28109ab0 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.877 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb10006210 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.877 7fcb1ffff700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 0x7fcb0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.877 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcb100327b0 con 0x7fcb28108c90 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.877 7fcb2707d700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 0x7fcb0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:12.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.878 7fcb2707d700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 0x7fcb0c01ac10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:12.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.908 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2810a050 con 0x7fcb28108c90 2021-02-10T11:23:12.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:12.911 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb10006b80 con 0x7fcb28108c90 2021-02-10T11:23:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.038 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb2810a050 con 0x7fcb28108c90 2021-02-10T11:23:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.039 7fcb1ffff700 1 -- 172.21.2.117:0/3838971767 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7fcb1000bd50 con 0x7fcb28108c90 2021-02-10T11:23:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:23:13.036 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:13.036 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:23:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 msgr2=0x7fcb0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 0x7fcb0c01ac10 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 msgr2=0x7fcb28109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fcb1000b5c0 tx=0x7fcb1000b960).stop 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 shutdown_connections 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb0c018730 0x7fcb0c01ac10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb28108c90 0x7fcb28109090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb281096b0 0x7fcb2810a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 --2- 172.21.2.117:0/3838971767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb280b7a90 0x7fcb280b7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.040 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 >> 172.21.2.117:0/3838971767 conn(0x7fcb281aaea0 msgr2=0x7fcb281923a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.041 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 shutdown_connections 2021-02-10T11:23:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:13.041 7fcb2d464700 1 -- 172.21.2.117:0/3838971767 wait complete. 2021-02-10T11:23:13.247 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:32.545631', 'modified': '2021-02-10 11:22:41.961589', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4229': {'gid': 4229, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/921969069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 921969069}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 921969069}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:13.247 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:13.247 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:13.649 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:13.649 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:23:13.649 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4229, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 32.692051915999997, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 6, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 22, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 22, 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 37.053194284999996 2021-02-10T11:23:13.650 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:13.653 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4229, "want_state": "up:active", "state": "up:active", "rank_uptime": 32.692051915999997, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "uptime": 37.053194284999996 } 2021-02-10T11:23:13.654 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:13.654 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2021-02-10T11:23:13.654 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@gibba025.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2021-02-10T11:23:13.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:13.654 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:13.988 DEBUG:teuthology.orchestra.run.gibba025:> sudo modprobe fuse 2021-02-10T11:23:14.361 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:14.361 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:14.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:14.707 DEBUG:teuthology.orchestra.run.gibba025:> ls /sys/fs/fuse/connections 2021-02-10T11:23:15.041 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:23:15.041 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:15.044 DEBUG:teuthology.orchestra.run.gibba025:> sudo modprobe fuse 2021-02-10T11:23:15.388 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:15.389 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:15.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba025.stderr:ceph-fuse[37347]: starting ceph client 2021-02-10T11:23:15.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba025.stderr:2021-02-10 11:23:15.455 7fa38a45cf80 -1 init, newargv = 0x555b85539af0 newargc=9 2021-02-10T11:23:15.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.gibba025.stderr:ceph-fuse[37347]: starting fuse 2021-02-10T11:23:15.730 INFO:teuthology.orchestra.run.gibba025.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:15.731 INFO:teuthology.orchestra.run.gibba025.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:15.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:15.732 DEBUG:teuthology.orchestra.run.gibba025:> ls /sys/fs/fuse/connections 2021-02-10T11:23:16.066 INFO:teuthology.orchestra.run.gibba025.stdout:42 2021-02-10T11:23:16.066 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:23:16.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:16.066 DEBUG:teuthology.orchestra.run.gibba025:> sudo python3 -c ' 2021-02-10T11:23:16.066 DEBUG:teuthology.orchestra.run.gibba025:> import glob 2021-02-10T11:23:16.066 DEBUG:teuthology.orchestra.run.gibba025:> import re 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> import os 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> import subprocess 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> def find_socket(client_name): 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> files = glob.glob(asok_path) 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> # Given a non-glob path, it better be there 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> if "*" not in asok_path: 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> assert(len(files) == 1) 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> return files[0] 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> for f in files: 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:16.067 DEBUG:teuthology.orchestra.run.gibba025:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:16.068 DEBUG:teuthology.orchestra.run.gibba025:> return f 2021-02-10T11:23:16.068 DEBUG:teuthology.orchestra.run.gibba025:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:16.068 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:16.068 DEBUG:teuthology.orchestra.run.gibba025:> print(find_socket("client.2")) 2021-02-10T11:23:16.068 DEBUG:teuthology.orchestra.run.gibba025:> ' 2021-02-10T11:23:16.438 INFO:teuthology.orchestra.run.gibba025.stdout:/var/run/ceph/ceph-client.2.37347.asok 2021-02-10T11:23:16.441 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.37347.asok 2021-02-10T11:23:16.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:16.441 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.37347.asok status 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout: "metadata": { 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "2", 2021-02-10T11:23:16.830 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "37347", 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "dentry_count": 0, 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4456, 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": { 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 4456 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:16.831 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "v1", 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 780192364 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "v1", 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 780192364 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "inst_str": "client.4456 172.21.2.125:0/780192364", 2021-02-10T11:23:16.832 INFO:teuthology.orchestra.run.gibba025.stdout: "addr_str": "172.21.2.125:0/780192364", 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout: "inode_count": 1, 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout: "mds_epoch": 6, 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout: "osd_epoch": 22, 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout: "blacklisted": false 2021-02-10T11:23:16.833 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:16.835 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.248 7f9beea6f700 1 Processor -- start 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.248 7f9beea6f700 1 -- start start 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9beea6f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9be807da20 0x7f9be80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9beea6f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80e4d90 0x7f9be81b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9beea6f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be8077a70 con 0x7f9be80e5190 2021-02-10T11:23:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9beea6f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80777d0 con 0x7f9be807da20 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9beea6f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be807de20 con 0x7f9be80e4d90 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9bed08c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9be80e5190 0x7f9be8077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42338/0 (socket says 172.21.2.117:42338) 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9bed08c700 1 -- 172.21.2.117:0/4085088116 learned_addr learned my addr 172.21.2.117:0/4085088116 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9bec88b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9be807da20 0x7f9be80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.249 7f9be7fff700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80e4d90 0x7f9be81b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9bec88b700 1 -- 172.21.2.117:0/4085088116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80e4d90 msgr2=0x7f9be81b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9bec88b700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80e4d90 0x7f9be81b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9bec88b700 1 -- 172.21.2.117:0/4085088116 >> v1:172.21.2.117:6789/0 conn(0x7f9be80e5190 legacy=0x7f9be8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9bec88b700 1 -- 172.21.2.117:0/4085088116 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be81b0700 con 0x7f9be807da20 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9bec88b700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be807da20 0x7f9be80771b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9bd0006650 tx=0x7f9bd0009a80).ready entity=mon.0 client_cookie=c63041fcfbbae5ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9be77fe700 1 -- 172.21.2.117:0/4085088116 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bd00009b0 con 0x7f9be807da20 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9be77fe700 1 -- 172.21.2.117:0/4085088116 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd000c0c0 con 0x7f9be807da20 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9be77fe700 1 -- 172.21.2.117:0/4085088116 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bd000c520 con 0x7f9be807da20 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9beea6f700 1 -- 172.21.2.117:0/4085088116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be807da20 msgr2=0x7f9be80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.250 7f9beea6f700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be807da20 0x7f9be80771b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9bd0006650 tx=0x7f9bd0009a80).stop 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 -- 172.21.2.117:0/4085088116 shutdown_connections 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80e4d90 0x7f9be81b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 --2- 172.21.2.117:0/4085088116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be807da20 0x7f9be80771b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 -- 172.21.2.117:0/4085088116 >> 172.21.2.117:0/4085088116 conn(0x7f9be80ce440 msgr2=0x7f9be80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 -- 172.21.2.117:0/4085088116 shutdown_connections 2021-02-10T11:23:17.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.251 7f9beea6f700 1 -- 172.21.2.117:0/4085088116 wait complete. 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 Processor -- start 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 -- start start 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80aa6c0 0x7f9be8188bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be818a490 0x7f9be818a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8189210 con 0x7f9be8189a70 2021-02-10T11:23:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8102e40 con 0x7f9be818a490 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.266 7f9beea6f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be819bc30 con 0x7f9be80aa6c0 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42092/0 (socket says 172.21.2.117:42092) 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 -- 172.21.2.117:0/134990792 learned_addr learned my addr 172.21.2.117:0/134990792 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 -- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80aa6c0 msgr2=0x7f9be8188bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80aa6c0 0x7f9be8188bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 -- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be818a490 msgr2=0x7f9be818a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be818a490 0x7f9be818a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 -- 172.21.2.117:0/134990792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be819be50 con 0x7f9be8189a70 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be7fff700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9bd8006f60 tx=0x7f9bd800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bd8006bb0 con 0x7f9be8189a70 2021-02-10T11:23:17.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8006340 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bd8002030 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9beea6f700 1 -- 172.21.2.117:0/134990792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be819be50 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.267 7f9beea6f700 1 -- 172.21.2.117:0/134990792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be819be50 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.268 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9bd8006b50 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.268 7f9be57fa700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 0x7f9bcc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.268 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f9bd8006340 con 0x7f9be8189a70 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.269 7f9bec88b700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 0x7f9bcc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.269 7f9bec88b700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 0x7f9bcc01abc0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.299 7f9beea6f700 1 -- 172.21.2.117:0/134990792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be8189210 con 0x7f9be8189a70 2021-02-10T11:23:17.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.301 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9bd8006b50 con 0x7f9be8189a70 2021-02-10T11:23:17.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.426 7f9beea6f700 1 -- 172.21.2.117:0/134990792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9be8189210 con 0x7f9be8189a70 2021-02-10T11:23:17.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.427 7f9be57fa700 1 -- 172.21.2.117:0/134990792 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f9bd8006340 con 0x7f9be8189a70 2021-02-10T11:23:17.424 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:17.424 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:17.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.428 7f9beea6f700 1 -- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 msgr2=0x7f9bcc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.428 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 0x7f9bcc01abc0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.428 7f9beea6f700 1 -- 172.21.2.117:0/134990792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 msgr2=0x7f9be8189e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.428 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9bd8006f60 tx=0x7f9bd800b970).stop 2021-02-10T11:23:17.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 -- 172.21.2.117:0/134990792 shutdown_connections 2021-02-10T11:23:17.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186c0 0x7f9bcc01abc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be8189a70 0x7f9be8189e70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be818a490 0x7f9be818a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 --2- 172.21.2.117:0/134990792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80aa6c0 0x7f9be8188bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 -- 172.21.2.117:0/134990792 >> 172.21.2.117:0/134990792 conn(0x7f9be8199390 msgr2=0x7f9be8198b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 -- 172.21.2.117:0/134990792 shutdown_connections 2021-02-10T11:23:17.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.429 7f9beea6f700 1 -- 172.21.2.117:0/134990792 wait complete. 2021-02-10T11:23:17.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:17.434 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 Processor -- start 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 -- start start 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c9407da20 0x7f2c940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2c940e51a0 0x7f2c94079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c94077a70 con 0x7f2c940e4da0 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c940e5c20 con 0x7f2c9407da20 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c9af6a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c940777d0 con 0x7f2c940e51a0 2021-02-10T11:23:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c98d86700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c9407da20 0x7f2c940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c98d86700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c9407da20 0x7f2c940771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42102/0 (socket says 172.21.2.117:42102) 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.851 7f2c98d86700 1 -- 172.21.2.117:0/3881521534 learned_addr learned my addr 172.21.2.117:0/3881521534 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c98d86700 1 -- 172.21.2.117:0/3881521534 >> v2:172.21.2.125:3300/0 conn(0x7f2c940e51a0 msgr2=0x7f2c94079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c98d86700 1 --2- 172.21.2.117:0/3881521534 >> v2:172.21.2.125:3300/0 conn(0x7f2c940e51a0 0x7f2c94079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c98d86700 1 -- 172.21.2.117:0/3881521534 >> v1:172.21.2.117:6789/0 conn(0x7f2c940e4da0 legacy=0x7f2c941b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c937fe700 1 -- 172.21.2.117:0/3881521534 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896413840 0 0) 0x7f2c84001060 con 0x7f2c940e4da0 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c98d86700 1 -- 172.21.2.117:0/3881521534 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c941b08e0 con 0x7f2c9407da20 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c98d86700 1 --2- 172.21.2.117:0/3881521534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9407da20 0x7f2c940771b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2c7c006700 tx=0x7f2c7c009b80).ready entity=mon.0 client_cookie=2bcbb9cbe8f7c931 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c937fe700 1 -- 172.21.2.117:0/3881521534 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c7c006340 con 0x7f2c9407da20 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c937fe700 1 -- 172.21.2.117:0/3881521534 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c00c0c0 con 0x7f2c9407da20 2021-02-10T11:23:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c937fe700 1 -- 172.21.2.117:0/3881521534 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c7c00c5e0 con 0x7f2c9407da20 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c9af6a700 1 -- 172.21.2.117:0/3881521534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9407da20 msgr2=0x7f2c940771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.852 7f2c9af6a700 1 --2- 172.21.2.117:0/3881521534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9407da20 0x7f2c940771b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2c7c006700 tx=0x7f2c7c009b80).stop 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 -- 172.21.2.117:0/3881521534 shutdown_connections 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 --2- 172.21.2.117:0/3881521534 >> v2:172.21.2.125:3300/0 conn(0x7f2c940e51a0 0x7f2c94079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 --2- 172.21.2.117:0/3881521534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9407da20 0x7f2c940771b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 -- 172.21.2.117:0/3881521534 >> 172.21.2.117:0/3881521534 conn(0x7f2c940ce450 msgr2=0x7f2c940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 -- 172.21.2.117:0/3881521534 shutdown_connections 2021-02-10T11:23:17.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.853 7f2c9af6a700 1 -- 172.21.2.117:0/3881521534 wait complete. 2021-02-10T11:23:17.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 Processor -- start 2021-02-10T11:23:17.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 -- start start 2021-02-10T11:23:17.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c941a3200 0x7f2c941a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c9408e590 0x7f2c9408e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94094460 con 0x7f2c9408efb0 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9408f3b0 con 0x7f2c9408e590 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c9af6a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940a6280 con 0x7f2c941a3200 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c99587700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c99587700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42110/0 (socket says 172.21.2.117:42110) 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.868 7f2c99587700 1 -- 172.21.2.117:0/2754380426 learned_addr learned my addr 172.21.2.117:0/2754380426 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 -- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c941a3200 msgr2=0x7f2c941a3600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c941a3200 0x7f2c941a3600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 -- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c9408e590 msgr2=0x7f2c9408e990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c9408e590 0x7f2c9408e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:17.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 -- 172.21.2.117:0/2754380426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940a64a0 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c99587700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2c8c006f10 tx=0x7f2c8c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c8c006240 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c000c10 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c8c005770 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c940a64a0 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.869 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c940a64a0 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.870 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c8c006030 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.870 7f2c917fa700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 0x7f2c7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.870 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2c8c032940 con 0x7f2c9408efb0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.870 7f2c98d86700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 0x7f2c7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:17.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.871 7f2c98d86700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 0x7f2c7801abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:17.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.901 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94103350 con 0x7f2c9408efb0 2021-02-10T11:23:17.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:17.904 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c8c000c10 con 0x7f2c9408efb0 2021-02-10T11:23:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.030 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2c94103350 con 0x7f2c9408efb0 2021-02-10T11:23:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.031 7f2c917fa700 1 -- 172.21.2.117:0/2754380426 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2c8c006030 con 0x7f2c9408efb0 2021-02-10T11:23:18.028 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.032 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 msgr2=0x7f2c7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.032 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 0x7f2c7801abc0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.032 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 msgr2=0x7f2c940a5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.032 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2c8c006f10 tx=0x7f2c8c00b990).stop 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 shutdown_connections 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c78018730 0x7f2c7801abc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c9408efb0 0x7f2c940a5c60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c9408e590 0x7f2c9408e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 --2- 172.21.2.117:0/2754380426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c941a3200 0x7f2c941a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 >> 172.21.2.117:0/2754380426 conn(0x7f2c940a19a0 msgr2=0x7f2c940b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 shutdown_connections 2021-02-10T11:23:18.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.033 7f2c9af6a700 1 -- 172.21.2.117:0/2754380426 wait complete. 2021-02-10T11:23:18.037 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:18.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 Processor -- start 2021-02-10T11:23:18.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 -- start start 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16440d1530 0x7f16440d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1644077a70 con 0x7f164407da20 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1644079400 con 0x7f1644079000 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.453 7f164ac10700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164407de20 con 0x7f16440d1530 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164922d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16440d1530 0x7f16440d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f1648a2c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f164407da20 0x7f16440771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42364/0 (socket says 172.21.2.117:42364) 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f1648a2c700 1 -- 172.21.2.117:0/337153390 learned_addr learned my addr 172.21.2.117:0/337153390 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164922d700 1 -- 172.21.2.117:0/337153390 >> v1:172.21.2.125:6789/0 conn(0x7f1644079000 legacy=0x7f16440e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164922d700 1 -- 172.21.2.117:0/337153390 >> v1:172.21.2.117:6789/0 conn(0x7f164407da20 legacy=0x7f16440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164922d700 1 -- 172.21.2.117:0/337153390 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16441b00f0 con 0x7f16440d1530 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164922d700 1 --2- 172.21.2.117:0/337153390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440d1530 0x7f16440d1930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f163c006e30 tx=0x7f163c009e60).ready entity=mon.0 client_cookie=d353ecb40865f14a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f16437fe700 1 -- 172.21.2.117:0/337153390 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f163c00b300 con 0x7f16440d1530 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f16437fe700 1 -- 172.21.2.117:0/337153390 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c00b580 con 0x7f16440d1530 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f16437fe700 1 -- 172.21.2.117:0/337153390 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f163c00ba60 con 0x7f16440d1530 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164ac10700 1 -- 172.21.2.117:0/337153390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440d1530 msgr2=0x7f16440d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.454 7f164ac10700 1 --2- 172.21.2.117:0/337153390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440d1530 0x7f16440d1930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f163c006e30 tx=0x7f163c009e60).stop 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.455 7f164ac10700 1 -- 172.21.2.117:0/337153390 shutdown_connections 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.455 7f164ac10700 1 --2- 172.21.2.117:0/337153390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440d1530 0x7f16440d1930 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.455 7f164ac10700 1 -- 172.21.2.117:0/337153390 >> 172.21.2.117:0/337153390 conn(0x7f16440ce450 msgr2=0x7f16440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.455 7f164ac10700 1 -- 172.21.2.117:0/337153390 shutdown_connections 2021-02-10T11:23:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.455 7f164ac10700 1 -- 172.21.2.117:0/337153390 wait complete. 2021-02-10T11:23:18.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.469 7f164ac10700 1 Processor -- start 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.469 7f164ac10700 1 -- start start 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16441b2430 0x7f164408a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164408aa90 0x7f164408ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440ab580 con 0x7f16440bc770 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16441b2830 con 0x7f16441b2430 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f164ac10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440bcb70 con 0x7f164408aa90 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42118/0 (socket says 172.21.2.117:42118) 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 -- 172.21.2.117:0/3416346792 learned_addr learned my addr 172.21.2.117:0/3416346792 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 -- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164408aa90 msgr2=0x7f164408ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164408aa90 0x7f164408ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 -- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16441b2430 msgr2=0x7f164408a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16441b2430 0x7f164408a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.470 7f1648a2c700 1 -- 172.21.2.117:0/3416346792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164408b4b0 con 0x7f16440bc770 2021-02-10T11:23:18.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f1648a2c700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f162c001c60 tx=0x7f162c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c00acc0 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c004140 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f164ac10700 1 -- 172.21.2.117:0/3416346792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164408b4b0 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f164ac10700 1 -- 172.21.2.117:0/3416346792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16441cd7a0 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f162c004660 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f162c004320 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.471 7f16417fa700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 0x7f162801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.472 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f162c032910 con 0x7f16440bc770 2021-02-10T11:23:18.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.472 7f1643fff700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 0x7f162801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:18.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.473 7f1643fff700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 0x7f162801abd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:18.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.503 7f164ac10700 1 -- 172.21.2.117:0/3416346792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16440ab580 con 0x7f16440bc770 2021-02-10T11:23:18.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.506 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f162c000dd0 con 0x7f16440bc770 2021-02-10T11:23:18.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.632 7f164ac10700 1 -- 172.21.2.117:0/3416346792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f16440ab580 con 0x7f16440bc770 2021-02-10T11:23:18.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.633 7f16417fa700 1 -- 172.21.2.117:0/3416346792 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f162c004140 con 0x7f16440bc770 2021-02-10T11:23:18.630 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:18.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 msgr2=0x7f162801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 0x7f162801abd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 msgr2=0x7f16441b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:18.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f162c001c60 tx=0x7f162c00a8d0).stop 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 shutdown_connections 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164408aa90 0x7f164408ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16441b2430 0x7f164408a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16280186d0 0x7f162801abd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 --2- 172.21.2.117:0/3416346792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16440bc770 0x7f16441b1e10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 >> 172.21.2.117:0/3416346792 conn(0x7f16440924b0 msgr2=0x7f16440a1970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 shutdown_connections 2021-02-10T11:23:18.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:18.635 7f164ac10700 1 -- 172.21.2.117:0/3416346792 wait complete. 2021-02-10T11:23:18.640 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:19.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 Processor -- start 2021-02-10T11:23:19.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 -- start start 2021-02-10T11:23:19.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fab4007da20 0x7fab400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400e51a0 0x7fab40079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40077a70 con 0x7fab400e4da0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab400777d0 con 0x7fab4007da20 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab4505c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab401b08e0 con 0x7fab400e51a0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab3e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fab400e4da0 0x7fab401b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42378/0 (socket says 172.21.2.117:42378) 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.056 7fab3e59c700 1 -- 172.21.2.117:0/1023010166 learned_addr learned my addr 172.21.2.117:0/1023010166 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119945451 0 0) 0x7fab280010b0 con 0x7fab400e4da0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab2c002fe0 con 0x7fab400e4da0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> v2:172.21.2.125:3300/0 conn(0x7fab4007da20 0x7fab400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4093182221 0 0) 0x7fab280010b0 con 0x7fab400e4da0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400e51a0 msgr2=0x7fab40079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3f59e700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400e51a0 0x7fab40079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400e51a0 0x7fab40079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 >> v2:172.21.2.125:3300/0 conn(0x7fab4007da20 msgr2=0x7fab400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 --2- 172.21.2.117:0/1023010166 >> v2:172.21.2.125:3300/0 conn(0x7fab4007da20 0x7fab400771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4007de20 con 0x7fab400e4da0 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fab280015b0 con 0x7fab400e4da0 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 >> v1:172.21.2.117:6789/0 conn(0x7fab400e4da0 legacy=0x7fab401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab2c003b90 0x7fab2c0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab2c0068d0 0x7fab2c008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab2c009660 0x7fab2c00bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c00c410 con 0x7fab2c003b90 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c00c630 con 0x7fab2c009660 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c00c850 con 0x7fab2c0068d0 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3f59e700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400e51a0 0x7fab40079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3e59c700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab2c0068d0 0x7fab2c008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> v2:172.21.2.125:3300/0 conn(0x7fab4007da20 0x7fab400771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:19.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab2c003b90 0x7fab2c0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.057 7fab3f59e700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab2c009660 0x7fab2c00bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 -- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab2c0068d0 msgr2=0x7fab2c008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab2c0068d0 0x7fab2c008dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 -- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab2c009660 msgr2=0x7fab2c00bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab2c009660 0x7fab2c00bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 -- 172.21.2.117:0/1023010166 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c00caf0 con 0x7fab2c003b90 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3e59c700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab2c0068d0 0x7fab2c008dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3ed9d700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab2c003b90 0x7fab2c0060b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fab30009970 tx=0x7fab30009d10).ready entity=mon.0 client_cookie=538714014449e936 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab300065d0 con 0x7fab2c003b90 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30000d20 con 0x7fab2c003b90 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3dd9b700 1 -- 172.21.2.117:0/1023010166 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab30002220 con 0x7fab2c003b90 2021-02-10T11:23:19.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab4505c700 1 -- 172.21.2.117:0/1023010166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab2c003b90 msgr2=0x7fab2c0060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab4505c700 1 --2- 172.21.2.117:0/1023010166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab2c003b90 0x7fab2c0060b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fab30009970 tx=0x7fab30009d10).stop 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.058 7fab3f59e700 1 -- 172.21.2.117:0/1023010166 reap_dead start 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.059 7fab4505c700 1 -- 172.21.2.117:0/1023010166 shutdown_connections 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.059 7fab4505c700 1 -- 172.21.2.117:0/1023010166 >> 172.21.2.117:0/1023010166 conn(0x7fab400ce450 msgr2=0x7fab400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.059 7fab4505c700 1 -- 172.21.2.117:0/1023010166 shutdown_connections 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.059 7fab4505c700 1 -- 172.21.2.117:0/1023010166 wait complete. 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 Processor -- start 2021-02-10T11:23:19.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 -- start start 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab400a47d0 0x7fab400a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400b77d0 0x7fab400b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40189970 con 0x7fab400a51f0 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab40190aa0 con 0x7fab400a47d0 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab4505c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab400b81f0 con 0x7fab400b77d0 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab3e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab3e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42140/0 (socket says 172.21.2.117:42140) 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab3e59c700 1 -- 172.21.2.117:0/2976590439 learned_addr learned my addr 172.21.2.117:0/2976590439 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:19.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.074 7fab3ed9d700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab400a47d0 0x7fab400a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3f59e700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400b77d0 0x7fab400b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 -- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400b77d0 msgr2=0x7fab400b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400b77d0 0x7fab400b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 -- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab400a47d0 msgr2=0x7fab400a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab400a47d0 0x7fab400a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 -- 172.21.2.117:0/2976590439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400b8410 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab3e59c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fab2800a540 tx=0x7fab2800a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab280009b0 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2800ad60 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab28004330 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab4505c700 1 -- 172.21.2.117:0/2976590439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab400b8410 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.075 7fab4505c700 1 -- 172.21.2.117:0/2976590439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab400b89a0 con 0x7fab400a51f0 2021-02-10T11:23:19.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.076 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab28004140 con 0x7fab400a51f0 2021-02-10T11:23:19.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.076 7fab377fe700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 0x7fab2401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.076 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fab280328d0 con 0x7fab400a51f0 2021-02-10T11:23:19.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.076 7fab3ed9d700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 0x7fab2401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.077 7fab3ed9d700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 0x7fab2401abd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.106 7fab4505c700 1 -- 172.21.2.117:0/2976590439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab40077a70 con 0x7fab400a51f0 2021-02-10T11:23:19.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.109 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab280009b0 con 0x7fab400a51f0 2021-02-10T11:23:19.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.235 7fab4505c700 1 -- 172.21.2.117:0/2976590439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fab40077a70 con 0x7fab400a51f0 2021-02-10T11:23:19.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.235 7fab377fe700 1 -- 172.21.2.117:0/2976590439 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fab28004140 con 0x7fab400a51f0 2021-02-10T11:23:19.232 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:19.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 msgr2=0x7fab2401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 0x7fab2401abd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 msgr2=0x7fab400a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fab2800a540 tx=0x7fab2800a8e0).stop 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 shutdown_connections 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab240186d0 0x7fab2401abd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab400a51f0 0x7fab400a55f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab400b77d0 0x7fab400b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 --2- 172.21.2.117:0/2976590439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab400a47d0 0x7fab400a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 >> 172.21.2.117:0/2976590439 conn(0x7fab400b5140 msgr2=0x7fab400a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 shutdown_connections 2021-02-10T11:23:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.237 7fab4505c700 1 -- 172.21.2.117:0/2976590439 wait complete. 2021-02-10T11:23:19.241 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:23:19.242 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 Processor -- start 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 -- start start 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff7f0109600 0x7ff7f01b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00e4190 0x7ff7f00e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7f0109ab0 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7f01b3c60 con 0x7ff7f0109600 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7f54bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00d0ec0 con 0x7ff7f00e4190 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7ef2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff7f00d0ac0 0x7ff7f01b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42396/0 (socket says 172.21.2.117:42396) 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.656 7ff7ef2bf700 1 -- 172.21.2.117:0/3982966868 learned_addr learned my addr 172.21.2.117:0/3982966868 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199097182 0 0) 0x7ff7e0001180 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7dc002fe0 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7eeabe700 1 --2- 172.21.2.117:0/3982966868 >> v2:172.21.2.125:3300/0 conn(0x7ff7f0109600 0x7ff7f01b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7efac0700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00e4190 0x7ff7f00e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1183924040 0 0) 0x7ff7e0001180 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00e4190 msgr2=0x7ff7f00e4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00e4190 0x7ff7f00e4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 >> v2:172.21.2.125:3300/0 conn(0x7ff7f0109600 msgr2=0x7ff7f01b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 --2- 172.21.2.117:0/3982966868 >> v2:172.21.2.125:3300/0 conn(0x7ff7f0109600 0x7ff7f01b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f010b040 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff7e0001680 con 0x7ff7f00d0ac0 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 >> v1:172.21.2.117:6789/0 conn(0x7ff7f00d0ac0 legacy=0x7ff7f01b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7dc003af0 0x7ff7dc005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7dc0067a0 0x7ff7dc008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7dc009580 0x7ff7dc00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc00c2c0 con 0x7ff7dc003af0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0504b0 con 0x7ff7dc0067a0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc0506d0 con 0x7ff7dc009580 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.657 7ff7ef2bf700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7dc003af0 0x7ff7dc005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 -- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7dc009580 msgr2=0x7ff7dc00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7dc009580 0x7ff7dc00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 -- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7dc0067a0 msgr2=0x7ff7dc008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7dc0067a0 0x7ff7dc008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 -- 172.21.2.117:0/3982966868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc050970 con 0x7ff7dc003af0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ef2bf700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7dc003af0 0x7ff7dc005f00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff7e0001340 tx=0x7ff7e0007cf0).ready entity=mon.0 client_cookie=1fadeacf276e70a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e0000ce0 con 0x7ff7dc003af0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e00070c0 con 0x7ff7dc003af0 2021-02-10T11:23:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7ee2bd700 1 -- 172.21.2.117:0/3982966868 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e000a7a0 con 0x7ff7dc003af0 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7f54bd700 1 -- 172.21.2.117:0/3982966868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7dc003af0 msgr2=0x7ff7dc005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7f54bd700 1 --2- 172.21.2.117:0/3982966868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7dc003af0 0x7ff7dc005f00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff7e0001340 tx=0x7ff7e0007cf0).stop 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7efac0700 1 -- 172.21.2.117:0/3982966868 reap_dead start 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7f54bd700 1 -- 172.21.2.117:0/3982966868 shutdown_connections 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.658 7ff7f54bd700 1 -- 172.21.2.117:0/3982966868 >> 172.21.2.117:0/3982966868 conn(0x7ff7f00ce410 msgr2=0x7ff7f00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.659 7ff7f54bd700 1 -- 172.21.2.117:0/3982966868 shutdown_connections 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.659 7ff7f54bd700 1 -- 172.21.2.117:0/3982966868 wait complete. 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.673 7ff7f54bd700 1 Processor -- start 2021-02-10T11:23:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.673 7ff7f54bd700 1 -- start start 2021-02-10T11:23:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00f1860 0x7ff7f00f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00f2280 0x7ff7f00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f018fb10 con 0x7ff7f018f690 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00f2680 con 0x7ff7f00f1860 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7f54bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00ac050 con 0x7ff7f00f2280 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42160/0 (socket says 172.21.2.117:42160) 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 -- 172.21.2.117:0/591897689 learned_addr learned my addr 172.21.2.117:0/591897689 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 -- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00f2280 msgr2=0x7ff7f00aef60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00f2280 0x7ff7f00aef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 -- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00f1860 msgr2=0x7ff7f00f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00f1860 0x7ff7f00f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.674 7ff7ef2bf700 1 -- 172.21.2.117:0/591897689 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f00af580 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7ef2bf700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff7e000a570 tx=0x7ff7e000a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e00070d0 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e0000940 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7e00043a0 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f00af580 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.675 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f00afb10 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.676 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7e00043a0 con 0x7ff7f018f690 2021-02-10T11:23:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.676 7ff7e7fff700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 0x7ff7d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.676 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7ff7e0032690 con 0x7ff7f018f690 2021-02-10T11:23:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.676 7ff7eeabe700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 0x7ff7d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.676 7ff7eeabe700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 0x7ff7d401abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:19.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.706 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f00afb10 con 0x7ff7f018f690 2021-02-10T11:23:19.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.709 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7e0001130 con 0x7ff7f018f690 2021-02-10T11:23:19.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.837 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7f00afb10 con 0x7ff7f018f690 2021-02-10T11:23:19.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.838 7ff7e7fff700 1 -- 172.21.2.117:0/591897689 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7ff7e001f4b0 con 0x7ff7f018f690 2021-02-10T11:23:19.835 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:19.835 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:23:19.835 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.839 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 msgr2=0x7ff7d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.839 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 0x7ff7d401abc0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.839 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 msgr2=0x7ff7f00aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.839 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff7e000a570 tx=0x7ff7e000a910).stop 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 shutdown_connections 2021-02-10T11:23:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f00f2280 0x7ff7f00aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00f1860 0x7ff7f00f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7d40186c0 0x7ff7d401abc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 --2- 172.21.2.117:0/591897689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f018f690 0x7ff7f00aba30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 >> 172.21.2.117:0/591897689 conn(0x7ff7f00b5570 msgr2=0x7ff7f00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 shutdown_connections 2021-02-10T11:23:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:19.840 7ff7f54bd700 1 -- 172.21.2.117:0/591897689 wait complete. 2021-02-10T11:23:20.046 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:22:32.545631', 'modified': '2021-02-10 11:22:41.961589', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4229': {'gid': 4229, 'name': 'b', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/921969069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 921969069}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 921969069}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:23:20.046 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:23:20.046 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:23:20.445 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:20.445 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4229, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 39.488068226000003, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 6, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 22, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 22, 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 43.849211001999997 2021-02-10T11:23:20.446 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:20.449 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4229, "want_state": "up:active", "state": "up:active", "rank_uptime": 39.488068226000003, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "uptime": 43.849211001999997 } 2021-02-10T11:23:20.450 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:23:20.450 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2021-02-10T11:23:20.450 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.3 at ubuntu@gibba025.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3... 2021-02-10T11:23:20.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:20.450 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:20.784 DEBUG:teuthology.orchestra.run.gibba025:> sudo modprobe fuse 2021-02-10T11:23:21.128 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:21.128 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:21.470 INFO:teuthology.orchestra.run.gibba025.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:21.470 INFO:teuthology.orchestra.run.gibba025.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:21.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:21.471 DEBUG:teuthology.orchestra.run.gibba025:> ls /sys/fs/fuse/connections 2021-02-10T11:23:21.805 INFO:teuthology.orchestra.run.gibba025.stdout:42 2021-02-10T11:23:21.806 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:23:21.806 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.3 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:21.808 DEBUG:teuthology.orchestra.run.gibba025:> sudo modprobe fuse 2021-02-10T11:23:22.151 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:23:22.152 DEBUG:teuthology.orchestra.run.gibba025:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:23:22.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba025.stderr:ceph-fuse[37680]: starting ceph client 2021-02-10T11:23:22.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba025.stderr:2021-02-10 11:23:22.219 7f2e24514f80 -1 init, newargv = 0x555ead6e8ee0 newargc=9 2021-02-10T11:23:22.223 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.gibba025.stderr:ceph-fuse[37680]: starting fuse 2021-02-10T11:23:22.493 INFO:teuthology.orchestra.run.gibba025.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:23:22.493 INFO:teuthology.orchestra.run.gibba025.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:23:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:23:22.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:22.494 DEBUG:teuthology.orchestra.run.gibba025:> ls /sys/fs/fuse/connections 2021-02-10T11:23:22.716 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:22.720 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:22.828 INFO:teuthology.orchestra.run.gibba025.stdout:42 2021-02-10T11:23:22.829 INFO:teuthology.orchestra.run.gibba025.stdout:43 2021-02-10T11:23:22.829 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:23:22.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:22.830 DEBUG:teuthology.orchestra.run.gibba025:> sudo python3 -c ' 2021-02-10T11:23:22.830 DEBUG:teuthology.orchestra.run.gibba025:> import glob 2021-02-10T11:23:22.830 DEBUG:teuthology.orchestra.run.gibba025:> import re 2021-02-10T11:23:22.830 DEBUG:teuthology.orchestra.run.gibba025:> import os 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> import subprocess 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> def find_socket(client_name): 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> files = glob.glob(asok_path) 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:22.831 DEBUG:teuthology.orchestra.run.gibba025:> # Given a non-glob path, it better be there 2021-02-10T11:23:22.832 DEBUG:teuthology.orchestra.run.gibba025:> if "*" not in asok_path: 2021-02-10T11:23:22.832 DEBUG:teuthology.orchestra.run.gibba025:> assert(len(files) == 1) 2021-02-10T11:23:22.832 DEBUG:teuthology.orchestra.run.gibba025:> return files[0] 2021-02-10T11:23:22.832 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:22.832 DEBUG:teuthology.orchestra.run.gibba025:> for f in files: 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> return f 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> 2021-02-10T11:23:22.833 DEBUG:teuthology.orchestra.run.gibba025:> print(find_socket("client.3")) 2021-02-10T11:23:22.834 DEBUG:teuthology.orchestra.run.gibba025:> ' 2021-02-10T11:23:23.198 INFO:teuthology.orchestra.run.gibba025.stdout:/var/run/ceph/ceph-client.3.37680.asok 2021-02-10T11:23:23.201 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.37680.asok 2021-02-10T11:23:23.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:23.201 DEBUG:teuthology.orchestra.run.gibba025:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.37680.asok status 2021-02-10T11:23:23.589 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:23:23.589 INFO:teuthology.orchestra.run.gibba025.stdout: "metadata": { 2021-02-10T11:23:23.589 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:23:23.589 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:23:23.589 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "3", 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "37680", 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "dentry_count": 0, 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "dentry_pinned_count": 0, 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4346, 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": { 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:23:23.590 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 4346 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "v1", 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 2865035248 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "v1", 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 2865035248 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "inst_str": "client.4346 172.21.2.125:0/2865035248", 2021-02-10T11:23:23.591 INFO:teuthology.orchestra.run.gibba025.stdout: "addr_str": "172.21.2.125:0/2865035248", 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout: "inode_count": 1, 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout: "mds_epoch": 6, 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout: "osd_epoch": 22, 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout: "blacklisted": false 2021-02-10T11:23:23.592 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:23:23.593 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:23.593 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:23.593 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:23.928 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:23:23.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:23.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:23.929 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:24.275 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:24.276 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:24.276 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:24.611 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:23:24.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:24.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:24.612 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:24.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:24.961 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:24.961 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:25.295 INFO:teuthology.orchestra.run.gibba025.stdout:fuseblk 2021-02-10T11:23:25.296 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:25.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:25.296 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:25.645 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:25.646 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:25.646 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:25.980 INFO:teuthology.orchestra.run.gibba025.stdout:fuseblk 2021-02-10T11:23:25.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:25.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:25.981 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:26.327 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2021-02-10T11:23:26.347 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:26.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.758 7fc14a6fb700 1 Processor -- start 2021-02-10T11:23:26.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.758 7fc14a6fb700 1 -- start start 2021-02-10T11:23:26.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc14a6fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc144079000 0x7fc1440e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc14a6fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1441b00e0 0x7fc1441b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc14a6fb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc144077a70 con 0x7fc14407da20 2021-02-10T11:23:26.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc14a6fb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1440e53b0 con 0x7fc1441b00e0 2021-02-10T11:23:26.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc14a6fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc144079400 con 0x7fc144079000 2021-02-10T11:23:26.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1441b00e0 0x7fc1441b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1441b00e0 0x7fc1441b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42168/0 (socket says 172.21.2.117:42168) 2021-02-10T11:23:26.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 -- 172.21.2.117:0/3268918226 learned_addr learned my addr 172.21.2.117:0/3268918226 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:26.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc1437fe700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc144079000 0x7fc1440e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 -- 172.21.2.117:0/3268918226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc144079000 msgr2=0x7fc1440e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc144079000 0x7fc1440e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 -- 172.21.2.117:0/3268918226 >> v1:172.21.2.117:6789/0 conn(0x7fc14407da20 legacy=0x7fc1440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 -- 172.21.2.117:0/3268918226 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14407de20 con 0x7fc1441b00e0 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc148d18700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1441b00e0 0x7fc1441b04e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc13c006f00 tx=0x7fc13c009e90).ready entity=mon.0 client_cookie=ba0dbdda50227cb2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.759 7fc142ffd700 1 -- 172.21.2.117:0/3268918226 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13c00b300 con 0x7fc1441b00e0 2021-02-10T11:23:26.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc142ffd700 1 -- 172.21.2.117:0/3268918226 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c00b580 con 0x7fc1441b00e0 2021-02-10T11:23:26.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc142ffd700 1 -- 172.21.2.117:0/3268918226 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13c00ba60 con 0x7fc1441b00e0 2021-02-10T11:23:26.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 -- 172.21.2.117:0/3268918226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1441b00e0 msgr2=0x7fc1441b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1441b00e0 0x7fc1441b04e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc13c006f00 tx=0x7fc13c009e90).stop 2021-02-10T11:23:26.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 -- 172.21.2.117:0/3268918226 shutdown_connections 2021-02-10T11:23:26.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1441b00e0 0x7fc1441b04e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 --2- 172.21.2.117:0/3268918226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc144079000 0x7fc1440e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 -- 172.21.2.117:0/3268918226 >> 172.21.2.117:0/3268918226 conn(0x7fc1440ce440 msgr2=0x7fc1440e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:26.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 -- 172.21.2.117:0/3268918226 shutdown_connections 2021-02-10T11:23:26.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.760 7fc14a6fb700 1 -- 172.21.2.117:0/3268918226 wait complete. 2021-02-10T11:23:26.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 Processor -- start 2021-02-10T11:23:26.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 -- start start 2021-02-10T11:23:26.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc14418e590 0x7fc1440b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1440b1c60 0x7fc14418edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14418e990 con 0x7fc1440b2060 2021-02-10T11:23:26.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14418fa10 con 0x7fc14418e590 2021-02-10T11:23:26.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.775 7fc14a6fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14418fc30 con 0x7fc1440b1c60 2021-02-10T11:23:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42174/0 (socket says 172.21.2.117:42174) 2021-02-10T11:23:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 -- 172.21.2.117:0/2783699925 learned_addr learned my addr 172.21.2.117:0/2783699925 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 -- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1440b1c60 msgr2=0x7fc14418edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1440b1c60 0x7fc14418edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 -- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc14418e590 msgr2=0x7fc1440b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc14418e590 0x7fc1440b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 -- 172.21.2.117:0/2783699925 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14418fe50 con 0x7fc1440b2060 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc148d18700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc13c00b910 tx=0x7fc13c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13c00bdf0 con 0x7fc1440b2060 2021-02-10T11:23:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13c005820 con 0x7fc1440b2060 2021-02-10T11:23:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc13c005c80 con 0x7fc1440b2060 2021-02-10T11:23:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14418fe50 con 0x7fc1440b2060 2021-02-10T11:23:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.776 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14418fe50 con 0x7fc1440b2060 2021-02-10T11:23:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.777 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc13c005a00 con 0x7fc1440b2060 2021-02-10T11:23:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.777 7fc140ff9700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 0x7fc12401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.777 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fc13c032b40 con 0x7fc1440b2060 2021-02-10T11:23:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.777 7fc143fff700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 0x7fc12401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.778 7fc143fff700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 0x7fc12401ac10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.808 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1441ac6e0 con 0x7fc1440b2060 2021-02-10T11:23:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.811 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc13c000cc0 con 0x7fc1440b2060 2021-02-10T11:23:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.934 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1441ac6e0 con 0x7fc1440b2060 2021-02-10T11:23:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.935 7fc140ff9700 1 -- 172.21.2.117:0/2783699925 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7fc13c005820 con 0x7fc1440b2060 2021-02-10T11:23:26.932 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:26.932 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:26.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 msgr2=0x7fc12401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 0x7fc12401ac10 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 msgr2=0x7fc14418f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:26.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc13c00b910 tx=0x7fc13c00bcb0).stop 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 shutdown_connections 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc124018730 0x7fc12401ac10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1440b2060 0x7fc14418f3f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1440b1c60 0x7fc14418edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 --2- 172.21.2.117:0/2783699925 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc14418e590 0x7fc1440b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 >> 172.21.2.117:0/2783699925 conn(0x7fc14419f9a0 msgr2=0x7fc14410acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 shutdown_connections 2021-02-10T11:23:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:26.937 7fc14a6fb700 1 -- 172.21.2.117:0/2783699925 wait complete. 2021-02-10T11:23:26.942 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:26.942 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.357 7f52f982b700 1 Processor -- start 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.357 7f52f982b700 1 -- start start 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f982b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f52f4079000 0x7f52f40e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f982b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f52f41b00f0 0x7f52f41b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f982b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f4077a70 con 0x7f52f407da20 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f982b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40e5c20 con 0x7f52f4079000 2021-02-10T11:23:27.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f982b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40e53c0 con 0x7f52f41b00f0 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f2d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f52f4079000 0x7f52f40e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f3574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f52f407da20 0x7f52f40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42430/0 (socket says 172.21.2.117:42430) 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f3574700 1 -- 172.21.2.117:0/245891926 learned_addr learned my addr 172.21.2.117:0/245891926 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f2d73700 1 -- 172.21.2.117:0/245891926 >> v2:172.21.2.125:3300/0 conn(0x7f52f41b00f0 msgr2=0x7f52f41b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f2d73700 1 --2- 172.21.2.117:0/245891926 >> v2:172.21.2.125:3300/0 conn(0x7f52f41b00f0 0x7f52f41b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f2d73700 1 -- 172.21.2.117:0/245891926 >> v1:172.21.2.117:6789/0 conn(0x7f52f407da20 legacy=0x7f52f40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.358 7f52f2d73700 1 -- 172.21.2.117:0/245891926 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f4079400 con 0x7f52f4079000 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f2d73700 1 --2- 172.21.2.117:0/245891926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f4079000 0x7f52f40e4da0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f52e4006650 tx=0x7f52e4009a00).ready entity=mon.0 client_cookie=b791825085c1a641 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f2572700 1 -- 172.21.2.117:0/245891926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e4009df0 con 0x7f52f4079000 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f2572700 1 -- 172.21.2.117:0/245891926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e40009b0 con 0x7f52f4079000 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f2572700 1 -- 172.21.2.117:0/245891926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52e400c2b0 con 0x7f52f4079000 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 -- 172.21.2.117:0/245891926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f4079000 msgr2=0x7f52f40e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 --2- 172.21.2.117:0/245891926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f4079000 0x7f52f40e4da0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f52e4006650 tx=0x7f52e4009a00).stop 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 -- 172.21.2.117:0/245891926 shutdown_connections 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 --2- 172.21.2.117:0/245891926 >> v2:172.21.2.125:3300/0 conn(0x7f52f41b00f0 0x7f52f41b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 --2- 172.21.2.117:0/245891926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f4079000 0x7f52f40e4da0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.359 7f52f982b700 1 -- 172.21.2.117:0/245891926 >> 172.21.2.117:0/245891926 conn(0x7f52f40ce450 msgr2=0x7f52f40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.360 7f52f982b700 1 -- 172.21.2.117:0/245891926 shutdown_connections 2021-02-10T11:23:27.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.360 7f52f982b700 1 -- 172.21.2.117:0/245891926 wait complete. 2021-02-10T11:23:27.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.374 7f52f982b700 1 Processor -- start 2021-02-10T11:23:27.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.374 7f52f982b700 1 -- start start 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52f40b0f40 0x7f52f4107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52f40b1340 0x7f52f418b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40b18c0 con 0x7f52f418b830 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f418ac90 con 0x7f52f40b1340 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f4107f50 con 0x7f52f40b0f40 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42186/0 (socket says 172.21.2.117:42186) 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 -- 172.21.2.117:0/691645233 learned_addr learned my addr 172.21.2.117:0/691645233 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 -- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52f40b0f40 msgr2=0x7f52f4107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52f40b0f40 0x7f52f4107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 -- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52f40b1340 msgr2=0x7f52f418b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52f40b1340 0x7f52f418b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 -- 172.21.2.117:0/691645233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f4108130 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f3d75700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f52ec009980 tx=0x7f52ec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ec003020 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec005a80 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52ec006030 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 -- 172.21.2.117:0/691645233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f4108130 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.375 7f52f982b700 1 -- 172.21.2.117:0/691645233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f4108130 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.376 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52ec006030 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.376 7f52ebfff700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 0x7f52d801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.376 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f52ec032ca0 con 0x7f52f418b830 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.377 7f52f3574700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 0x7f52d801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.377 7f52f3574700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 0x7f52d801abd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.408 7f52f982b700 1 -- 172.21.2.117:0/691645233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f4077a70 con 0x7f52f418b830 2021-02-10T11:23:27.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.410 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52ec005870 con 0x7f52f418b830 2021-02-10T11:23:27.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.537 7f52f982b700 1 -- 172.21.2.117:0/691645233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f52f4077a70 con 0x7f52f418b830 2021-02-10T11:23:27.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.537 7f52ebfff700 1 -- 172.21.2.117:0/691645233 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52ec005a80 con 0x7f52f418b830 2021-02-10T11:23:27.534 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:27.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 msgr2=0x7f52d801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 0x7f52d801abd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 msgr2=0x7f52f4187bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f52ec009980 tx=0x7f52ec00bcd0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 shutdown_connections 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52d80186d0 0x7f52d801abd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52f418b830 0x7f52f4187bf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52f40b1340 0x7f52f418b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 --2- 172.21.2.117:0/691645233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52f40b0f40 0x7f52f4107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 >> 172.21.2.117:0/691645233 conn(0x7f52f40a1880 msgr2=0x7f52f40b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 shutdown_connections 2021-02-10T11:23:27.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.539 7f52f982b700 1 -- 172.21.2.117:0/691645233 wait complete. 2021-02-10T11:23:27.543 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:27.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 Processor -- start 2021-02-10T11:23:27.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 -- start start 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a6c07da20 0x7f9a6c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9a6c0e4d80 0x7f9a6c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0e5180 0x7f9a6c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c077a70 con 0x7f9a6c07da20 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c0777d0 con 0x7f9a6c0e4d80 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a710ce700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c07de20 con 0x7f9a6c0e5180 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a6ad9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a6c07da20 0x7f9a6c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a6ad9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a6c07da20 0x7f9a6c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42194/0 (socket says 172.21.2.117:42194) 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.961 7f9a6ad9d700 1 -- 172.21.2.117:0/29302739 learned_addr learned my addr 172.21.2.117:0/29302739 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 -- 172.21.2.117:0/29302739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0e5180 msgr2=0x7f9a6c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 --2- 172.21.2.117:0/29302739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0e5180 0x7f9a6c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 -- 172.21.2.117:0/29302739 >> v2:172.21.2.125:3300/0 conn(0x7f9a6c0e4d80 msgr2=0x7f9a6c1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 --2- 172.21.2.117:0/29302739 >> v2:172.21.2.125:3300/0 conn(0x7f9a6c0e4d80 0x7f9a6c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 -- 172.21.2.117:0/29302739 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c1b06f0 con 0x7f9a6c07da20 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a6ad9d700 1 --2- 172.21.2.117:0/29302739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c07da20 0x7f9a6c0771b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f9a5c006700 tx=0x7f9a5c009b80).ready entity=mon.0 client_cookie=b6a26036e22ebd6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a69d9b700 1 -- 172.21.2.117:0/29302739 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a5c00c0c0 con 0x7f9a6c07da20 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a69d9b700 1 -- 172.21.2.117:0/29302739 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c00c340 con 0x7f9a6c07da20 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a69d9b700 1 -- 172.21.2.117:0/29302739 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a5c00c860 con 0x7f9a6c07da20 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a710ce700 1 -- 172.21.2.117:0/29302739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c07da20 msgr2=0x7f9a6c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.962 7f9a710ce700 1 --2- 172.21.2.117:0/29302739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c07da20 0x7f9a6c0771b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f9a5c006700 tx=0x7f9a5c009b80).stop 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 -- 172.21.2.117:0/29302739 shutdown_connections 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 --2- 172.21.2.117:0/29302739 >> v2:172.21.2.125:3300/0 conn(0x7f9a6c0e4d80 0x7f9a6c1b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 --2- 172.21.2.117:0/29302739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0e5180 0x7f9a6c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 --2- 172.21.2.117:0/29302739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c07da20 0x7f9a6c0771b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 -- 172.21.2.117:0/29302739 >> 172.21.2.117:0/29302739 conn(0x7f9a6c0ce430 msgr2=0x7f9a6c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:27.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 -- 172.21.2.117:0/29302739 shutdown_connections 2021-02-10T11:23:27.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.963 7f9a710ce700 1 -- 172.21.2.117:0/29302739 wait complete. 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.977 7f9a710ce700 1 Processor -- start 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 -- start start 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a6c0a3d70 0x7f9a6c092650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0932f0 0x7f9a6c1903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c092d70 con 0x7f9a6c0936f0 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c190a00 con 0x7f9a6c0a3d70 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a710ce700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0a4170 con 0x7f9a6c0932f0 2021-02-10T11:23:27.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42204/0 (socket says 172.21.2.117:42204) 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 -- 172.21.2.117:0/68894122 learned_addr learned my addr 172.21.2.117:0/68894122 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 -- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0932f0 msgr2=0x7f9a6c1903e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0932f0 0x7f9a6c1903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 -- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a6c0a3d70 msgr2=0x7f9a6c092650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a6c0a3d70 0x7f9a6c092650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.978 7f9a6b59e700 1 -- 172.21.2.117:0/68894122 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c08a7a0 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a6b59e700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f9a64009980 tx=0x7f9a6400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a64005870 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64005b30 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a64006030 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a710ce700 1 -- 172.21.2.117:0/68894122 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c08a7a0 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a710ce700 1 -- 172.21.2.117:0/68894122 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c08a7a0 con 0x7f9a6c0936f0 2021-02-10T11:23:27.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.979 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a64006030 con 0x7f9a6c0936f0 2021-02-10T11:23:27.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.980 7f9a637fe700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 0x7f9a5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:27.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.980 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f9a64032a60 con 0x7f9a6c0936f0 2021-02-10T11:23:27.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.980 7f9a6ad9d700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 0x7f9a5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:27.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:27.981 7f9a6ad9d700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 0x7f9a5001abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:28.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.011 7f9a710ce700 1 -- 172.21.2.117:0/68894122 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c103350 con 0x7f9a6c0936f0 2021-02-10T11:23:28.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.014 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a6400be10 con 0x7f9a6c0936f0 2021-02-10T11:23:28.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.140 7f9a710ce700 1 -- 172.21.2.117:0/68894122 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9a6c077a70 con 0x7f9a6c0936f0 2021-02-10T11:23:28.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.141 7f9a637fe700 1 -- 172.21.2.117:0/68894122 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9a64006f20 con 0x7f9a6c0936f0 2021-02-10T11:23:28.137 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:28.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 msgr2=0x7f9a5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 0x7f9a5001abc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 msgr2=0x7f9a6c08a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f9a64009980 tx=0x7f9a6400bcd0).stop 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 shutdown_connections 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a500186c0 0x7f9a5001abc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a6c0936f0 0x7f9a6c08a180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a6c0932f0 0x7f9a6c1903e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 --2- 172.21.2.117:0/68894122 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a6c0a3d70 0x7f9a6c092650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 >> 172.21.2.117:0/68894122 conn(0x7f9a6c0a1420 msgr2=0x7f9a6c093b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 shutdown_connections 2021-02-10T11:23:28.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.143 7f9a710ce700 1 -- 172.21.2.117:0/68894122 wait complete. 2021-02-10T11:23:28.147 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:28.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.564 7f2a8e6b0700 1 Processor -- start 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.564 7f2a8e6b0700 1 -- start start 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8e6b0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2a8807da20 0x7f2a880771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8e6b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 0x7f2a88079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8e6b0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a88077a70 con 0x7f2a880e4da0 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8e6b0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a880777d0 con 0x7f2a8807da20 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8e6b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a881b08e0 con 0x7f2a880e51a0 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a877fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f2a880e4da0 0x7f2a881b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35386/0 (socket says 172.21.2.117:35386) 2021-02-10T11:23:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a877fe700 1 -- 172.21.2.117:0/4000584262 learned_addr learned my addr 172.21.2.117:0/4000584262 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a8cccd700 1 --2- 172.21.2.117:0/4000584262 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 0x7f2a88079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.565 7f2a87fff700 1 --2- 172.21.2.117:0/4000584262 >> v2:172.21.2.125:3300/0 conn(0x7f2a8807da20 0x7f2a880771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a8cccd700 1 -- 172.21.2.117:0/4000584262 >> v2:172.21.2.125:3300/0 conn(0x7f2a8807da20 msgr2=0x7f2a880771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a8cccd700 1 --2- 172.21.2.117:0/4000584262 >> v2:172.21.2.125:3300/0 conn(0x7f2a8807da20 0x7f2a880771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a8cccd700 1 -- 172.21.2.117:0/4000584262 >> v1:172.21.2.125:6789/0 conn(0x7f2a880e4da0 legacy=0x7f2a881b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a86ffd700 1 -- 172.21.2.117:0/4000584262 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542124532 0 0) 0x7f2a7c0009b0 con 0x7f2a880e4da0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a8cccd700 1 -- 172.21.2.117:0/4000584262 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8807de20 con 0x7f2a880e51a0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.566 7f2a8cccd700 1 --2- 172.21.2.117:0/4000584262 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 0x7f2a88079000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2a80006e50 tx=0x7f2a80009e70).ready entity=mon.2 client_cookie=4e205b51a41670cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a86ffd700 1 -- 172.21.2.117:0/4000584262 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a800066f0 con 0x7f2a880e51a0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a86ffd700 1 -- 172.21.2.117:0/4000584262 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a80003030 con 0x7f2a880e51a0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a86ffd700 1 -- 172.21.2.117:0/4000584262 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a8000b700 con 0x7f2a880e51a0 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 -- 172.21.2.117:0/4000584262 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 msgr2=0x7f2a88079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 --2- 172.21.2.117:0/4000584262 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 0x7f2a88079000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f2a80006e50 tx=0x7f2a80009e70).stop 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 -- 172.21.2.117:0/4000584262 shutdown_connections 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 --2- 172.21.2.117:0/4000584262 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a880e51a0 0x7f2a88079000 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 --2- 172.21.2.117:0/4000584262 >> v2:172.21.2.125:3300/0 conn(0x7f2a8807da20 0x7f2a880771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 -- 172.21.2.117:0/4000584262 >> 172.21.2.117:0/4000584262 conn(0x7f2a880ce450 msgr2=0x7f2a880e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 -- 172.21.2.117:0/4000584262 shutdown_connections 2021-02-10T11:23:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.567 7f2a8e6b0700 1 -- 172.21.2.117:0/4000584262 wait complete. 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 Processor -- start 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 -- start start 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a881b5130 0x7f2a881b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a881b5b50 0x7f2a880b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880b8da0 con 0x7f2a881b5f50 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880a4140 con 0x7f2a881b5130 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.582 7f2a8e6b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880a4320 con 0x7f2a881b5b50 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42214/0 (socket says 172.21.2.117:42214) 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 -- 172.21.2.117:0/3373634825 learned_addr learned my addr 172.21.2.117:0/3373634825 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a87fff700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a881b5130 0x7f2a881b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a877fe700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a881b5b50 0x7f2a880b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 -- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a881b5b50 msgr2=0x7f2a880b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a881b5b50 0x7f2a880b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 -- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a881b5130 msgr2=0x7f2a881b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a881b5130 0x7f2a881b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 -- 172.21.2.117:0/3373634825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880b62b0 con 0x7f2a881b5f50 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8cccd700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2a8000b910 tx=0x7f2a8000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:28.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a80006c30 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a80001070 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a80001510 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.583 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a880b62b0 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.584 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a880b62b0 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.584 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a800056b0 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.584 7f2a84ff9700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 0x7f2a6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.584 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2a80032950 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.585 7f2a87fff700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 0x7f2a6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.585 7f2a87fff700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 0x7f2a6801abc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.615 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a881b4f00 con 0x7f2a881b5f50 2021-02-10T11:23:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.618 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a80001250 con 0x7f2a881b5f50 2021-02-10T11:23:28.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.745 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2a880b6b30 con 0x7f2a881b5f50 2021-02-10T11:23:28.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.745 7f2a84ff9700 1 -- 172.21.2.117:0/3373634825 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2a8001f870 con 0x7f2a881b5f50 2021-02-10T11:23:28.742 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:28.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 msgr2=0x7f2a6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 0x7f2a6801abc0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 msgr2=0x7f2a880b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f2a8000b910 tx=0x7f2a8000bcb0).stop 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 shutdown_connections 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a680186c0 0x7f2a6801abc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a881b5f50 0x7f2a880b5c90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a881b5b50 0x7f2a880b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.747 7f2a8e6b0700 1 --2- 172.21.2.117:0/3373634825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a881b5130 0x7f2a881b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.748 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 >> 172.21.2.117:0/3373634825 conn(0x7f2a8808bbc0 msgr2=0x7f2a880c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:28.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.748 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 shutdown_connections 2021-02-10T11:23:28.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:28.748 7f2a8e6b0700 1 -- 172.21.2.117:0/3373634825 wait complete. 2021-02-10T11:23:28.754 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:29.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 Processor -- start 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 -- start start 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 0x7f77ec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 0x7f77ec1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec077a70 con 0x7f77ec0e5190 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ec0e5c10 con 0x7f77ec0e4d90 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77f12f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0777d0 con 0x7f77ec07da20 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77eb87e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f77ec0e5190 0x7f77ec077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42474/0 (socket says 172.21.2.117:42474) 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.183 7f77eb87e700 1 -- 172.21.2.117:0/1996007615 learned_addr learned my addr 172.21.2.117:0/1996007615 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 --2- 172.21.2.117:0/1996007615 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 0x7f77ec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77ea87c700 1 --2- 172.21.2.117:0/1996007615 >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 0x7f77ec1b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77ea07b700 1 -- 172.21.2.117:0/1996007615 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765842201 0 0) 0x7f77e4001500 con 0x7f77ec0e5190 2021-02-10T11:23:29.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77ea07b700 1 -- 172.21.2.117:0/1996007615 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d8002fe0 con 0x7f77ec0e5190 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 -- 172.21.2.117:0/1996007615 >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 msgr2=0x7f77ec1b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 --2- 172.21.2.117:0/1996007615 >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 0x7f77ec1b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 -- 172.21.2.117:0/1996007615 >> v1:172.21.2.117:6789/0 conn(0x7f77ec0e5190 legacy=0x7f77ec077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 -- 172.21.2.117:0/1996007615 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec07de20 con 0x7f77ec07da20 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77ea87c700 1 --2- 172.21.2.117:0/1996007615 >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 0x7f77ec1b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.184 7f77eb07d700 1 --2- 172.21.2.117:0/1996007615 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 0x7f77ec0771b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f77d4006650 tx=0x7f77d4009a80).ready entity=mon.2 client_cookie=d61de70fb1e566bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77ea07b700 1 -- 172.21.2.117:0/1996007615 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77d40009b0 con 0x7f77ec07da20 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77ea07b700 1 -- 172.21.2.117:0/1996007615 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d400c0c0 con 0x7f77ec07da20 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77ea07b700 1 -- 172.21.2.117:0/1996007615 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77d400c520 con 0x7f77ec07da20 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 -- 172.21.2.117:0/1996007615 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 msgr2=0x7f77ec0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 --2- 172.21.2.117:0/1996007615 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 0x7f77ec0771b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f77d4006650 tx=0x7f77d4009a80).stop 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 -- 172.21.2.117:0/1996007615 shutdown_connections 2021-02-10T11:23:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 --2- 172.21.2.117:0/1996007615 >> v2:172.21.2.125:3300/0 conn(0x7f77ec0e4d90 0x7f77ec1b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 --2- 172.21.2.117:0/1996007615 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec07da20 0x7f77ec0771b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 -- 172.21.2.117:0/1996007615 >> 172.21.2.117:0/1996007615 conn(0x7f77ec0ce440 msgr2=0x7f77ec0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 -- 172.21.2.117:0/1996007615 shutdown_connections 2021-02-10T11:23:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.185 7f77f12f6700 1 -- 172.21.2.117:0/1996007615 wait complete. 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 Processor -- start 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 -- start start 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec09dd40 0x7f77ec0b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 0x7f77ec08c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec09d8a0 con 0x7f77ec08caa0 2021-02-10T11:23:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b8bc0 con 0x7f77ec08c080 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77f12f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0b8de0 con 0x7f77ec09dd40 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77eb87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77eb87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42230/0 (socket says 172.21.2.117:42230) 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.200 7f77eb87e700 1 -- 172.21.2.117:0/1651412500 learned_addr learned my addr 172.21.2.117:0/1651412500 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb07d700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec09dd40 0x7f77ec0b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77ea87c700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 0x7f77ec08c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 -- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec09dd40 msgr2=0x7f77ec0b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec09dd40 0x7f77ec0b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 -- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 msgr2=0x7f77ec08c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 0x7f77ec08c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 -- 172.21.2.117:0/1651412500 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec08cea0 con 0x7f77ec08caa0 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77ea87c700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 0x7f77ec08c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77eb87e700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f77e4001c10 tx=0x7f77e400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77e4007ce0 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e40070d0 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77e4004930 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec08cea0 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.201 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec08fed0 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.202 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77e400ade0 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.202 7f77e3fff700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 0x7f77d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.202 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f77e4032a60 con 0x7f77ec08caa0 2021-02-10T11:23:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.203 7f77eb07d700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 0x7f77d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.203 7f77eb07d700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 0x7f77d001abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.233 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec077a70 con 0x7f77ec08caa0 2021-02-10T11:23:29.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.236 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77e40194c0 con 0x7f77ec08caa0 2021-02-10T11:23:29.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.360 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77ec077a70 con 0x7f77ec08caa0 2021-02-10T11:23:29.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.360 7f77e3fff700 1 -- 172.21.2.117:0/1651412500 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f77e40194c0 con 0x7f77ec08caa0 2021-02-10T11:23:29.357 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:29.357 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:29.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 msgr2=0x7f77d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 0x7f77d001abc0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 msgr2=0x7f77ec08f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f77e4001c10 tx=0x7f77e400aca0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 shutdown_connections 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77d00186c0 0x7f77d001abc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77ec08caa0 0x7f77ec08f5c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77ec08c080 0x7f77ec08c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 --2- 172.21.2.117:0/1651412500 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77ec09dd40 0x7f77ec0b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.362 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 >> 172.21.2.117:0/1651412500 conn(0x7f77ec19f4c0 msgr2=0x7f77ec094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.363 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 shutdown_connections 2021-02-10T11:23:29.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.363 7f77f12f6700 1 -- 172.21.2.117:0/1651412500 wait complete. 2021-02-10T11:23:29.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:29.368 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:29.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.784 7faef4b31700 1 Processor -- start 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.784 7faef4b31700 1 -- start start 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faef4b31700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef007da20 0x7faef00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faef4b31700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faef00e4d80 0x7faef01b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faef4b31700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0077a70 con 0x7faef00e5180 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faef4b31700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00e5c00 con 0x7faef00e4d80 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faef4b31700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef00777d0 con 0x7faef007da20 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faeef05f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faef00e5180 0x7faef0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42488/0 (socket says 172.21.2.117:42488) 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faeef05f700 1 -- 172.21.2.117:0/2392485807 learned_addr learned my addr 172.21.2.117:0/2392485807 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576539344 0 0) 0x7faee8001500 con 0x7faef00e5180 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faeed85c700 1 -- 172.21.2.117:0/2392485807 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faedc002fe0 con 0x7faef00e5180 2021-02-10T11:23:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.785 7faeee85e700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef007da20 0x7faef00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee05d700 1 --2- 172.21.2.117:0/2392485807 >> v2:172.21.2.125:3300/0 conn(0x7faef00e4d80 0x7faef01b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 122749916 0 0) 0x7faee8001460 con 0x7faef00e5180 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef007da20 msgr2=0x7faef00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef007da20 0x7faef00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 >> v2:172.21.2.125:3300/0 conn(0x7faef00e4d80 msgr2=0x7faef01b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 --2- 172.21.2.117:0/2392485807 >> v2:172.21.2.125:3300/0 conn(0x7faef00e4d80 0x7faef01b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef007de20 con 0x7faef00e5180 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7faee8001980 con 0x7faef00e5180 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 >> v1:172.21.2.117:6789/0 conn(0x7faef00e5180 legacy=0x7faef0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc003a60 0x7faedc005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc04a7b0 0x7faedc04cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc04d590 0x7faedc04fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0502d0 con 0x7faedc003a60 2021-02-10T11:23:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0504f0 con 0x7faedc04a7b0 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeed85c700 1 -- 172.21.2.117:0/2392485807 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc050710 con 0x7faedc04d590 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc003a60 0x7faedc005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeef05f700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc04d590 0x7faedc04fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee05d700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc04a7b0 0x7faedc04cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 -- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc04d590 msgr2=0x7faedc04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc04d590 0x7faedc04fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 -- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc04a7b0 msgr2=0x7faedc04cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc04a7b0 0x7faedc04cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee85e700 1 -- 172.21.2.117:0/2392485807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0509b0 con 0x7faedc003a60 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.786 7faeee05d700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc04a7b0 0x7faedc04cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faeee85e700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc003a60 0x7faedc005f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7faed8006aa0 tx=0x7faed800cb50).ready entity=mon.0 client_cookie=102f0404bca7530 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed80090d0 con 0x7faedc003a60 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8009410 con 0x7faedc003a60 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faeed85c700 1 -- 172.21.2.117:0/2392485807 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed8009930 con 0x7faedc003a60 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faef4b31700 1 -- 172.21.2.117:0/2392485807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc003a60 msgr2=0x7faedc005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faef4b31700 1 --2- 172.21.2.117:0/2392485807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc003a60 0x7faedc005f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7faed8006aa0 tx=0x7faed800cb50).stop 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.787 7faeef05f700 1 -- 172.21.2.117:0/2392485807 reap_dead start 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.788 7faef4b31700 1 -- 172.21.2.117:0/2392485807 shutdown_connections 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.788 7faef4b31700 1 -- 172.21.2.117:0/2392485807 >> 172.21.2.117:0/2392485807 conn(0x7faef00ce430 msgr2=0x7faef00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.788 7faef4b31700 1 -- 172.21.2.117:0/2392485807 shutdown_connections 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.788 7faef4b31700 1 -- 172.21.2.117:0/2392485807 wait complete. 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.802 7faef4b31700 1 Processor -- start 2021-02-10T11:23:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.802 7faef4b31700 1 -- start start 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faef00b8160 0x7faef00b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef00b8630 0x7faef00b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a3290 con 0x7faef00b9050 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a34b0 con 0x7faef00b8160 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faef4b31700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef01a36d0 con 0x7faef00b8630 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42248/0 (socket says 172.21.2.117:42248) 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 -- 172.21.2.117:0/3750818238 learned_addr learned my addr 172.21.2.117:0/3750818238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 -- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef00b8630 msgr2=0x7faef00b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef00b8630 0x7faef00b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 -- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faef00b8160 msgr2=0x7faef00b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faef00b8160 0x7faef00b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 -- 172.21.2.117:0/3750818238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef01a38f0 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.803 7faeef05f700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7faee800b910 tx=0x7faee800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faee8007a40 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee8006020 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faee8006540 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faef4b31700 1 -- 172.21.2.117:0/3750818238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef01a38f0 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faef4b31700 1 -- 172.21.2.117:0/3750818238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef01a38f0 con 0x7faef00b9050 2021-02-10T11:23:29.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.804 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faee8006200 con 0x7faef00b9050 2021-02-10T11:23:29.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.805 7faee77fe700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 0x7faed401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:29.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.805 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7faee8032a10 con 0x7faef00b9050 2021-02-10T11:23:29.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.805 7faeee85e700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 0x7faed401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:29.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.805 7faeee85e700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 0x7faed401abc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:29.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.835 7faef4b31700 1 -- 172.21.2.117:0/3750818238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef0077a70 con 0x7faef00b9050 2021-02-10T11:23:29.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.837 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faee800bdf0 con 0x7faef00b9050 2021-02-10T11:23:29.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.964 7faef4b31700 1 -- 172.21.2.117:0/3750818238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7faef0077a70 con 0x7faef00b9050 2021-02-10T11:23:29.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.964 7faee77fe700 1 -- 172.21.2.117:0/3750818238 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7faee8005f10 con 0x7faef00b9050 2021-02-10T11:23:29.962 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.966 7faef4b31700 1 -- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 msgr2=0x7faed401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.966 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 0x7faed401abc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.966 7faef4b31700 1 -- 172.21.2.117:0/3750818238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 msgr2=0x7faef01a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.966 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7faee800b910 tx=0x7faee800bcb0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 -- 172.21.2.117:0/3750818238 shutdown_connections 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faed40186c0 0x7faed401abc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faef00b9050 0x7faef01a2bf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faef00b8630 0x7faef00b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 --2- 172.21.2.117:0/3750818238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faef00b8160 0x7faef00b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 -- 172.21.2.117:0/3750818238 >> 172.21.2.117:0/3750818238 conn(0x7faef00a90a0 msgr2=0x7faef00a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:29.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 -- 172.21.2.117:0/3750818238 shutdown_connections 2021-02-10T11:23:29.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:29.967 7faef4b31700 1 -- 172.21.2.117:0/3750818238 wait complete. 2021-02-10T11:23:29.972 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.386 7fabbde89700 1 Processor -- start 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.386 7fabbde89700 1 -- start start 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabbde89700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabb8079000 0x7fabb80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabbde89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb81b00f0 0x7fabb81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabbde89700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb8077a70 con 0x7fabb807da20 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabbde89700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb80e53c0 con 0x7fabb8079000 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabbde89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8079400 con 0x7fabb81b00f0 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabb8079000 0x7fabb80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:30.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabb8079000 0x7fabb80e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42254/0 (socket says 172.21.2.117:42254) 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 -- 172.21.2.117:0/379674722 learned_addr learned my addr 172.21.2.117:0/379674722 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 -- 172.21.2.117:0/379674722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb81b00f0 msgr2=0x7fabb81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 --2- 172.21.2.117:0/379674722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb81b00f0 0x7fabb81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 -- 172.21.2.117:0/379674722 >> v1:172.21.2.125:6789/0 conn(0x7fabb807da20 legacy=0x7fabb80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 -- 172.21.2.117:0/379674722 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb807de20 con 0x7fabb8079000 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.387 7fabb6ffd700 1 --2- 172.21.2.117:0/379674722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8079000 0x7fabb80e4da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fabac006700 tx=0x7fabac009b80).ready entity=mon.0 client_cookie=db13917c571e34b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabb67fc700 1 -- 172.21.2.117:0/379674722 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabac00c0c0 con 0x7fabb8079000 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabb67fc700 1 -- 172.21.2.117:0/379674722 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac00c400 con 0x7fabb8079000 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabb67fc700 1 -- 172.21.2.117:0/379674722 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabac00c920 con 0x7fabb8079000 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 -- 172.21.2.117:0/379674722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8079000 msgr2=0x7fabb80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 --2- 172.21.2.117:0/379674722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8079000 0x7fabb80e4da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fabac006700 tx=0x7fabac009b80).stop 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 -- 172.21.2.117:0/379674722 shutdown_connections 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 --2- 172.21.2.117:0/379674722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb81b00f0 0x7fabb81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 --2- 172.21.2.117:0/379674722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8079000 0x7fabb80e4da0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:30.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 -- 172.21.2.117:0/379674722 >> 172.21.2.117:0/379674722 conn(0x7fabb80ce450 msgr2=0x7fabb80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:30.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 -- 172.21.2.117:0/379674722 shutdown_connections 2021-02-10T11:23:30.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.388 7fabbde89700 1 -- 172.21.2.117:0/379674722 wait complete. 2021-02-10T11:23:30.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 Processor -- start 2021-02-10T11:23:30.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 -- start start 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb808a180 0x7fabb808a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabb81a7240 0x7fabb81a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb8108020 con 0x7fabb8107c20 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb81a7c60 con 0x7fabb81a7240 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.403 7fabbde89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb81a7e80 con 0x7fabb808a180 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42262/0 (socket says 172.21.2.117:42262) 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 -- 172.21.2.117:0/1780691858 learned_addr learned my addr 172.21.2.117:0/1780691858 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 -- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb808a180 msgr2=0x7fabb808a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb808a180 0x7fabb808a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 -- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabb81a7240 msgr2=0x7fabb81a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabb81a7240 0x7fabb81a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 -- 172.21.2.117:0/1780691858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb808b250 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabb77fe700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7faba800b530 tx=0x7faba800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba800bcc0 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba8000e20 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabbde89700 1 -- 172.21.2.117:0/1780691858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb808b250 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fabbde89700 1 -- 172.21.2.117:0/1780691858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb808b7e0 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.404 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba8006210 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.405 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faba8006020 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.405 7fab9ffff700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 0x7fab9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.405 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7faba80326f0 con 0x7fabb8107c20 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.405 7fabb6ffd700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 0x7fab9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:30.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.406 7fabb6ffd700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 0x7fab9801abc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:30.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.437 7fabbde89700 1 -- 172.21.2.117:0/1780691858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb8077a70 con 0x7fabb8107c20 2021-02-10T11:23:30.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.439 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faba80009b0 con 0x7fabb8107c20 2021-02-10T11:23:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.567 7fabbde89700 1 -- 172.21.2.117:0/1780691858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fabb8077a70 con 0x7fabb8107c20 2021-02-10T11:23:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.567 7fab9ffff700 1 -- 172.21.2.117:0/1780691858 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7faba8006000 con 0x7fabb8107c20 2021-02-10T11:23:30.565 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 -- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 msgr2=0x7fab9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 0x7fab9801abc0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 -- 172.21.2.117:0/1780691858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 msgr2=0x7fabb8096fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7faba800b530 tx=0x7faba800b8d0).stop 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 -- 172.21.2.117:0/1780691858 shutdown_connections 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabb81a7240 0x7fabb81a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabb808a180 0x7fabb808a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.569 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab980186c0 0x7fab9801abc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.570 7fabbde89700 1 --2- 172.21.2.117:0/1780691858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabb8107c20 0x7fabb8096fc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:30.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.570 7fabbde89700 1 -- 172.21.2.117:0/1780691858 >> 172.21.2.117:0/1780691858 conn(0x7fabb80a1880 msgr2=0x7fabb80b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:30.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.570 7fabbde89700 1 -- 172.21.2.117:0/1780691858 shutdown_connections 2021-02-10T11:23:30.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.570 7fabbde89700 1 -- 172.21.2.117:0/1780691858 wait complete. 2021-02-10T11:23:30.575 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:31.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.990 7f1534a2b700 1 Processor -- start 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.990 7f1534a2b700 1 -- start start 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f153007da20 0x7f15300771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15300e4d80 0x7f15301b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15300e5180 0x7f1530077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1530077a70 con 0x7f15300e5180 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300777d0 con 0x7f153007da20 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f1534a2b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153007de20 con 0x7f15300e4d80 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15300e5180 0x7f1530077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15300e5180 0x7f1530077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42270/0 (socket says 172.21.2.117:42270) 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 -- 172.21.2.117:0/864400541 learned_addr learned my addr 172.21.2.117:0/864400541 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152de1b700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15300e4d80 0x7f15301b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152e61c700 1 --2- 172.21.2.117:0/864400541 >> v2:172.21.2.125:3300/0 conn(0x7f153007da20 0x7f15300771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 -- 172.21.2.117:0/864400541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15300e4d80 msgr2=0x7f15301b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15300e4d80 0x7f15301b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 -- 172.21.2.117:0/864400541 >> v2:172.21.2.125:3300/0 conn(0x7f153007da20 msgr2=0x7f15300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 --2- 172.21.2.117:0/864400541 >> v2:172.21.2.125:3300/0 conn(0x7f153007da20 0x7f15300771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 -- 172.21.2.117:0/864400541 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15301b06f0 con 0x7f15300e5180 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152ee1d700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15300e5180 0x7f1530077ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1528006f00 tx=0x7f1528009e90).ready entity=mon.0 client_cookie=26e2e01a5642c0b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.991 7f152d61a700 1 -- 172.21.2.117:0/864400541 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f152800b300 con 0x7f15300e5180 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f152d61a700 1 -- 172.21.2.117:0/864400541 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152800b580 con 0x7f15300e5180 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f152d61a700 1 -- 172.21.2.117:0/864400541 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f152800ba60 con 0x7f15300e5180 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 -- 172.21.2.117:0/864400541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15300e5180 msgr2=0x7f1530077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15300e5180 0x7f1530077ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1528006f00 tx=0x7f1528009e90).stop 2021-02-10T11:23:31.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 -- 172.21.2.117:0/864400541 shutdown_connections 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15300e5180 0x7f1530077ff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 --2- 172.21.2.117:0/864400541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15300e4d80 0x7f15301b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 --2- 172.21.2.117:0/864400541 >> v2:172.21.2.125:3300/0 conn(0x7f153007da20 0x7f15300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 -- 172.21.2.117:0/864400541 >> 172.21.2.117:0/864400541 conn(0x7f15300ce430 msgr2=0x7f15300e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 -- 172.21.2.117:0/864400541 shutdown_connections 2021-02-10T11:23:31.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:30.992 7f1534a2b700 1 -- 172.21.2.117:0/864400541 wait complete. 2021-02-10T11:23:31.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 Processor -- start 2021-02-10T11:23:31.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 -- start start 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1530106c20 0x7f153008fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1530107020 0x7f1530090450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153018dc70 con 0x7f153018d870 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1530193070 con 0x7f1530107020 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f1534a2b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1530090a70 con 0x7f1530106c20 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f152e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f152e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42272/0 (socket says 172.21.2.117:42272) 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.007 7f152e61c700 1 -- 172.21.2.117:0/969050665 learned_addr learned my addr 172.21.2.117:0/969050665 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 -- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1530106c20 msgr2=0x7f153008fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1530106c20 0x7f153008fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 -- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1530107020 msgr2=0x7f1530090450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1530107020 0x7f1530090450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 -- 172.21.2.117:0/969050665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1530090c90 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f152e61c700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f1518006f00 tx=0x7f151800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f151800bd30 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518002030 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1518002550 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f1534a2b700 1 -- 172.21.2.117:0/969050665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1530090c90 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.008 7f1534a2b700 1 -- 172.21.2.117:0/969050665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1530090c90 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.009 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1518000c90 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.009 7f1526ffd700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 0x7f151401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.009 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f1518032710 con 0x7f153018d870 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.009 7f152de1b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 0x7f151401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.010 7f152de1b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 0x7f151401abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.041 7f1534a2b700 1 -- 172.21.2.117:0/969050665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153018dc70 con 0x7f153018d870 2021-02-10T11:23:31.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.043 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1518030d60 con 0x7f153018d870 2021-02-10T11:23:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.170 7f1534a2b700 1 -- 172.21.2.117:0/969050665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f153018dc70 con 0x7f153018d870 2021-02-10T11:23:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.170 7f1526ffd700 1 -- 172.21.2.117:0/969050665 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f151800bd30 con 0x7f153018d870 2021-02-10T11:23:31.168 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.172 7f1534a2b700 1 -- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 msgr2=0x7f151401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.172 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 0x7f151401abc0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.172 7f1534a2b700 1 -- 172.21.2.117:0/969050665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 msgr2=0x7f1530106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.172 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f1518006f00 tx=0x7f151800b940).stop 2021-02-10T11:23:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 -- 172.21.2.117:0/969050665 shutdown_connections 2021-02-10T11:23:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1530107020 0x7f1530090450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1530106c20 0x7f153008fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15140186c0 0x7f151401abc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 --2- 172.21.2.117:0/969050665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f153018d870 0x7f1530106600 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 -- 172.21.2.117:0/969050665 >> 172.21.2.117:0/969050665 conn(0x7f15300a18e0 msgr2=0x7f15301034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 -- 172.21.2.117:0/969050665 shutdown_connections 2021-02-10T11:23:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.173 7f1534a2b700 1 -- 172.21.2.117:0/969050665 wait complete. 2021-02-10T11:23:31.178 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:31.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.592 7f2077a0e700 1 Processor -- start 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.592 7f2077a0e700 1 -- start start 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2077a0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2077a0e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20700e5190 0x7f2070079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2077a0e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2070077a70 con 0x7f20700e4d90 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2077a0e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700e5c10 con 0x7f207007da20 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2077a0e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20700777d0 con 0x7f20700e5190 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2075029700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f20700e4d90 0x7f20701b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42530/0 (socket says 172.21.2.117:42530) 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2075029700 1 -- 172.21.2.117:0/3389405758 learned_addr learned my addr 172.21.2.117:0/3389405758 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207602b700 1 --2- 172.21.2.117:0/3389405758 >> v2:172.21.2.125:3300/0 conn(0x7f20700e5190 0x7f2070079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f2074828700 1 -- 172.21.2.117:0/3389405758 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533131006 0 0) 0x7f206c0009b0 con 0x7f20700e4d90 2021-02-10T11:23:31.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 -- 172.21.2.117:0/3389405758 >> v2:172.21.2.125:3300/0 conn(0x7f20700e5190 msgr2=0x7f2070079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 --2- 172.21.2.117:0/3389405758 >> v2:172.21.2.125:3300/0 conn(0x7f20700e5190 0x7f2070079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 -- 172.21.2.117:0/3389405758 >> v1:172.21.2.117:6789/0 conn(0x7f20700e4d90 legacy=0x7f20701b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 -- 172.21.2.117:0/3389405758 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20701b08d0 con 0x7f207007da20 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.593 7f207582a700 1 --2- 172.21.2.117:0/3389405758 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207007da20 0x7f20700771b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2064006650 tx=0x7f2064009a80).ready entity=mon.0 client_cookie=d9070fbc591b7118 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.594 7f2074828700 1 -- 172.21.2.117:0/3389405758 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20640009b0 con 0x7f207007da20 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.594 7f2074828700 1 -- 172.21.2.117:0/3389405758 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206400c0c0 con 0x7f207007da20 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.594 7f2074828700 1 -- 172.21.2.117:0/3389405758 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f206400c5e0 con 0x7f207007da20 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.594 7f2077a0e700 1 -- 172.21.2.117:0/3389405758 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207007da20 msgr2=0x7f20700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.594 7f2077a0e700 1 --2- 172.21.2.117:0/3389405758 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207007da20 0x7f20700771b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2064006650 tx=0x7f2064009a80).stop 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 -- 172.21.2.117:0/3389405758 shutdown_connections 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 --2- 172.21.2.117:0/3389405758 >> v2:172.21.2.125:3300/0 conn(0x7f20700e5190 0x7f2070079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 --2- 172.21.2.117:0/3389405758 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207007da20 0x7f20700771b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 -- 172.21.2.117:0/3389405758 >> 172.21.2.117:0/3389405758 conn(0x7f20700ce440 msgr2=0x7f20700e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:31.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 -- 172.21.2.117:0/3389405758 shutdown_connections 2021-02-10T11:23:31.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.595 7f2077a0e700 1 -- 172.21.2.117:0/3389405758 wait complete. 2021-02-10T11:23:31.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 Processor -- start 2021-02-10T11:23:31.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 -- start start 2021-02-10T11:23:31.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20701af390 0x7f207009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070090a20 0x7f2070090e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207009bbe0 con 0x7f207019a070 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2070091470 con 0x7f2070090a20 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2077a0e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2070102e40 con 0x7f20701af390 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f207602b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f207602b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42288/0 (socket says 172.21.2.117:42288) 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f207602b700 1 -- 172.21.2.117:0/3246205606 learned_addr learned my addr 172.21.2.117:0/3246205606 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f2075029700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070090a20 0x7f2070090e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.610 7f207582a700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20701af390 0x7f207009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 -- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20701af390 msgr2=0x7f207009b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20701af390 0x7f207009b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 -- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070090a20 msgr2=0x7f2070090e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070090a20 0x7f2070090e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 -- 172.21.2.117:0/3246205606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207019aa90 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f207602b700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f206800b970 tx=0x7f206800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2068000f90 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2068003030 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2068001400 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.611 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207019aa90 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.612 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207019aa90 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.612 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2068000df0 con 0x7f207019a070 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.612 7f20627fc700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 0x7f205801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.612 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f2068032af0 con 0x7f207019a070 2021-02-10T11:23:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.612 7f207582a700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 0x7f205801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.613 7f207582a700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 0x7f205801ac10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:31.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.643 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2070091440 con 0x7f207019a070 2021-02-10T11:23:31.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.645 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20680066f0 con 0x7f207019a070 2021-02-10T11:23:31.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.768 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2070091440 con 0x7f207019a070 2021-02-10T11:23:31.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.769 7f20627fc700 1 -- 172.21.2.117:0/3246205606 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f2068001ee0 con 0x7f207019a070 2021-02-10T11:23:31.767 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:31.767 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 msgr2=0x7f205801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 0x7f205801ac10 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 msgr2=0x7f207019a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f206800b970 tx=0x7f206800bd10).stop 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 shutdown_connections 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2058018730 0x7f205801ac10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f207019a070 0x7f207019a470 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2070090a20 0x7f2070090e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 --2- 172.21.2.117:0/3246205606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20701af390 0x7f207009b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:31.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 >> 172.21.2.117:0/3246205606 conn(0x7f2070199390 msgr2=0x7f2070198b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:31.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 shutdown_connections 2021-02-10T11:23:31.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:31.771 7f2077a0e700 1 -- 172.21.2.117:0/3246205606 wait complete. 2021-02-10T11:23:31.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:31.777 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:32.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 Processor -- start 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 -- start start 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fafa407da20 0x7fafa40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafa40e51a0 0x7fafa4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4077a70 con 0x7fafa40e4da0 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa40e5c20 con 0x7fafa40e51a0 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.192 7fafab60b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa40777d0 con 0x7fafa407da20 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafa40e51a0 0x7fafa4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafa40e51a0 0x7fafa4079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42292/0 (socket says 172.21.2.117:42292) 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 -- 172.21.2.117:0/1116257034 learned_addr learned my addr 172.21.2.117:0/1116257034 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9427700 1 --2- 172.21.2.117:0/1116257034 >> v2:172.21.2.125:3300/0 conn(0x7fafa407da20 0x7fafa40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 -- 172.21.2.117:0/1116257034 >> v2:172.21.2.125:3300/0 conn(0x7fafa407da20 msgr2=0x7fafa40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 --2- 172.21.2.117:0/1116257034 >> v2:172.21.2.125:3300/0 conn(0x7fafa407da20 0x7fafa40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 -- 172.21.2.117:0/1116257034 >> v1:172.21.2.117:6789/0 conn(0x7fafa40e4da0 legacy=0x7fafa41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 -- 172.21.2.117:0/1116257034 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa41b08e0 con 0x7fafa40e51a0 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7fafa9c28700 1 --2- 172.21.2.117:0/1116257034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa40e51a0 0x7fafa4079000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7faf9c006f00 tx=0x7faf9c009f10).ready entity=mon.0 client_cookie=aaa05e0162b782 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7faf97fff700 1 -- 172.21.2.117:0/1116257034 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf9c00b3b0 con 0x7fafa40e51a0 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7faf97fff700 1 -- 172.21.2.117:0/1116257034 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c00b730 con 0x7fafa40e51a0 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.193 7faf97fff700 1 -- 172.21.2.117:0/1116257034 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf9c00bc50 con 0x7fafa40e51a0 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 -- 172.21.2.117:0/1116257034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa40e51a0 msgr2=0x7fafa4079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 --2- 172.21.2.117:0/1116257034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa40e51a0 0x7fafa4079000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7faf9c006f00 tx=0x7faf9c009f10).stop 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 -- 172.21.2.117:0/1116257034 shutdown_connections 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 --2- 172.21.2.117:0/1116257034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa40e51a0 0x7fafa4079000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 --2- 172.21.2.117:0/1116257034 >> v2:172.21.2.125:3300/0 conn(0x7fafa407da20 0x7fafa40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 -- 172.21.2.117:0/1116257034 >> 172.21.2.117:0/1116257034 conn(0x7fafa40ce450 msgr2=0x7fafa40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:32.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 -- 172.21.2.117:0/1116257034 shutdown_connections 2021-02-10T11:23:32.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.194 7fafab60b700 1 -- 172.21.2.117:0/1116257034 wait complete. 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 Processor -- start 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 -- start start 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafa40b9e60 0x7fafa40a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafa40a3800 0x7fafa40a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40a4220 con 0x7fafa419a4a0 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40a4410 con 0x7fafa40b9e60 2021-02-10T11:23:32.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafab60b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa40ba910 con 0x7fafa40a3800 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafa9427700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafa9427700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42296/0 (socket says 172.21.2.117:42296) 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.209 7fafa9427700 1 -- 172.21.2.117:0/680390512 learned_addr learned my addr 172.21.2.117:0/680390512 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 -- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafa40a3800 msgr2=0x7fafa40a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafa40a3800 0x7fafa40a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 -- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafa40b9e60 msgr2=0x7fafa40a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafa40b9e60 0x7fafa40a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 -- 172.21.2.117:0/680390512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40bab30 con 0x7fafa419a4a0 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafa9427700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7faf98006f60 tx=0x7faf9800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf9800bd60 con 0x7fafa419a4a0 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf98000c90 con 0x7fafa419a4a0 2021-02-10T11:23:32.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faf98002220 con 0x7fafa419a4a0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafab60b700 1 -- 172.21.2.117:0/680390512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa40bab30 con 0x7fafa419a4a0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.210 7fafab60b700 1 -- 172.21.2.117:0/680390512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa40bab30 con 0x7fafa419a4a0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.211 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faf980052e0 con 0x7fafa419a4a0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.211 7faf95ffb700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 0x7faf8401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.211 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7faf980327a0 con 0x7fafa419a4a0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.212 7fafa8c26700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 0x7faf8401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.212 7fafa8c26700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 0x7faf8401ac10 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.242 7fafab60b700 1 -- 172.21.2.117:0/680390512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa41b4f00 con 0x7fafa419a4a0 2021-02-10T11:23:32.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.245 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faf980052e0 con 0x7fafa419a4a0 2021-02-10T11:23:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.373 7fafab60b700 1 -- 172.21.2.117:0/680390512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fafa41b4f00 con 0x7fafa419a4a0 2021-02-10T11:23:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.373 7faf95ffb700 1 -- 172.21.2.117:0/680390512 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7faf9800bd60 con 0x7fafa419a4a0 2021-02-10T11:23:32.371 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:32.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 msgr2=0x7faf8401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 0x7faf8401ac10 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 msgr2=0x7fafa40b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7faf98006f60 tx=0x7faf9800b970).stop 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 shutdown_connections 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafa40a3800 0x7fafa40a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafa40b9e60 0x7fafa40a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf84018730 0x7faf8401ac10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 --2- 172.21.2.117:0/680390512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafa419a4a0 0x7fafa40b9840 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 >> 172.21.2.117:0/680390512 conn(0x7fafa4107b60 msgr2=0x7fafa40c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 shutdown_connections 2021-02-10T11:23:32.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.375 7fafab60b700 1 -- 172.21.2.117:0/680390512 wait complete. 2021-02-10T11:23:32.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:32.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.794 7fa7723e4700 1 Processor -- start 2021-02-10T11:23:32.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.794 7fa7723e4700 1 -- start start 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa7723e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c07da20 0x7fa76c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa7723e4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa76c0e4da0 0x7fa76c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa7723e4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c077a70 con 0x7fa76c0e51a0 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa7723e4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c0e5c20 con 0x7fa76c0e4da0 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa7723e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c0777d0 con 0x7fa76c07da20 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa76c0e4da0 0x7fa76c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa76c0e4da0 0x7fa76c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42304/0 (socket says 172.21.2.117:42304) 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 -- 172.21.2.117:0/309185771 learned_addr learned my addr 172.21.2.117:0/309185771 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 -- 172.21.2.117:0/309185771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c07da20 msgr2=0x7fa76c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 --2- 172.21.2.117:0/309185771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c07da20 0x7fa76c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.795 7fa76b7fe700 1 -- 172.21.2.117:0/309185771 >> v1:172.21.2.125:6789/0 conn(0x7fa76c0e51a0 legacy=0x7fa76c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa76b7fe700 1 -- 172.21.2.117:0/309185771 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c07de20 con 0x7fa76c0e4da0 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa76b7fe700 1 --2- 172.21.2.117:0/309185771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0e4da0 0x7fa76c1b00f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fa760006700 tx=0x7fa760009b00).ready entity=mon.0 client_cookie=76fcec05d22259 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa76affd700 1 -- 172.21.2.117:0/309185771 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa76000c0c0 con 0x7fa76c0e4da0 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa76affd700 1 -- 172.21.2.117:0/309185771 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76000c400 con 0x7fa76c0e4da0 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa76affd700 1 -- 172.21.2.117:0/309185771 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa76000c8e0 con 0x7fa76c0e4da0 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa7723e4700 1 -- 172.21.2.117:0/309185771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0e4da0 msgr2=0x7fa76c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.796 7fa7723e4700 1 --2- 172.21.2.117:0/309185771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0e4da0 0x7fa76c1b00f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fa760006700 tx=0x7fa760009b00).stop 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 -- 172.21.2.117:0/309185771 shutdown_connections 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 --2- 172.21.2.117:0/309185771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0e4da0 0x7fa76c1b00f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 --2- 172.21.2.117:0/309185771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c07da20 0x7fa76c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 -- 172.21.2.117:0/309185771 >> 172.21.2.117:0/309185771 conn(0x7fa76c0ce450 msgr2=0x7fa76c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 -- 172.21.2.117:0/309185771 shutdown_connections 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.797 7fa7723e4700 1 -- 172.21.2.117:0/309185771 wait complete. 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.811 7fa7723e4700 1 Processor -- start 2021-02-10T11:23:32.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.811 7fa7723e4700 1 -- start start 2021-02-10T11:23:32.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa76c0b1340 0x7fa76c18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c18b830 0x7fa76c187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c0b18c0 con 0x7fa76c0b0f40 2021-02-10T11:23:32.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c18ac90 con 0x7fa76c0b1340 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa7723e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c107f50 con 0x7fa76c18b830 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42308/0 (socket says 172.21.2.117:42308) 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 -- 172.21.2.117:0/293877768 learned_addr learned my addr 172.21.2.117:0/293877768 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76b7fe700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa76c0b1340 0x7fa76c18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa770a01700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c18b830 0x7fa76c187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 -- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c18b830 msgr2=0x7fa76c187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c18b830 0x7fa76c187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 -- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa76c0b1340 msgr2=0x7fa76c18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa76c0b1340 0x7fa76c18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.812 7fa76bfff700 1 -- 172.21.2.117:0/293877768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c108130 con 0x7fa76c0b0f40 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa76bfff700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa75c006f00 tx=0x7fa75c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa75c006b90 con 0x7fa76c0b0f40 2021-02-10T11:23:32.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c000c90 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa75c002020 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa7723e4700 1 -- 172.21.2.117:0/293877768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c108130 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.813 7fa7723e4700 1 -- 172.21.2.117:0/293877768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c108130 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.814 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa75c005310 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.814 7fa768ff9700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 0x7fa74c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.814 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fa75c032900 con 0x7fa76c0b0f40 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.814 7fa76b7fe700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 0x7fa74c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:32.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.815 7fa76b7fe700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 0x7fa74c01abd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:32.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.845 7fa7723e4700 1 -- 172.21.2.117:0/293877768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76c077a70 con 0x7fa76c0b0f40 2021-02-10T11:23:32.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.847 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa75c006350 con 0x7fa76c0b0f40 2021-02-10T11:23:32.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.975 7fa7723e4700 1 -- 172.21.2.117:0/293877768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa76c077a70 con 0x7fa76c0b0f40 2021-02-10T11:23:32.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.975 7fa768ff9700 1 -- 172.21.2.117:0/293877768 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa75c000c90 con 0x7fa76c0b0f40 2021-02-10T11:23:32.973 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 msgr2=0x7fa74c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 0x7fa74c01abd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 msgr2=0x7fa76c107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa75c006f00 tx=0x7fa75c00b940).stop 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 shutdown_connections 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa76c18b830 0x7fa76c187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa76c0b1340 0x7fa76c18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa74c0186d0 0x7fa74c01abd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 --2- 172.21.2.117:0/293877768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa76c0b0f40 0x7fa76c107930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:32.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 >> 172.21.2.117:0/293877768 conn(0x7fa76c0a1880 msgr2=0x7fa76c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:32.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 shutdown_connections 2021-02-10T11:23:32.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:32.978 7fa7723e4700 1 -- 172.21.2.117:0/293877768 wait complete. 2021-02-10T11:23:32.983 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.398 7fcb1407c700 1 Processor -- start 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.398 7fcb1407c700 1 -- start start 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb1407c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcb0c079000 0x7fcb0c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb1407c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0d1d20 con 0x7fcb0c07da20 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb1407c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c077a70 con 0x7fcb0c1b00f0 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb1407c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c0e53c0 con 0x7fcb0c079000 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb11e98700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcb0c07da20 0x7fcb0c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42566/0 (socket says 172.21.2.117:42566) 2021-02-10T11:23:33.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb11e98700 1 -- 172.21.2.117:0/184396671 learned_addr learned my addr 172.21.2.117:0/184396671 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722746866 0 0) 0x7fcb08001180 con 0x7fcb0c07da20 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.399 7fcb10e96700 1 -- 172.21.2.117:0/184396671 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8002fe0 con 0x7fcb0c07da20 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1782936315 0 0) 0x7fcb08001180 con 0x7fcb0c07da20 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 >> v2:172.21.2.125:3300/0 conn(0x7fcb0c079000 msgr2=0x7fcb0c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 --2- 172.21.2.117:0/184396671 >> v2:172.21.2.125:3300/0 conn(0x7fcb0c079000 0x7fcb0c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 >> v1:172.21.2.125:6789/0 conn(0x7fcb0c1b00f0 legacy=0x7fcb0c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c079400 con 0x7fcb0c07da20 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fcb08001680 con 0x7fcb0c07da20 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 >> v1:172.21.2.117:6789/0 conn(0x7fcb0c07da20 legacy=0x7fcb0c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaf8003a60 0x7fcaf8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaf8006790 0x7fcaf8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 0x7fcaf800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800c300 con 0x7fcaf8009590 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800c520 con 0x7fcaf8003a60 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb10e96700 1 -- 172.21.2.117:0/184396671 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800c740 con 0x7fcaf8006790 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 0x7fcaf800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb11e98700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaf8003a60 0x7fcaf8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb11697700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaf8006790 0x7fcaf8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 -- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaf8006790 msgr2=0x7fcaf8008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaf8006790 0x7fcaf8008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 -- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaf8003a60 msgr2=0x7fcaf8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaf8003a60 0x7fcaf8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 -- 172.21.2.117:0/184396671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf800c9e0 con 0x7fcaf8009590 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb11e98700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaf8003a60 0x7fcaf8005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.400 7fcb12699700 1 -- 172.21.2.117:0/184396671 reap_dead start 2021-02-10T11:23:33.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb12699700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 0x7fcaf800bab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fcb04006e30 tx=0x7fcb0404fe50).ready entity=mon.0 client_cookie=e4488bac1c752004 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb04001400 con 0x7fcaf8009590 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb04006a70 con 0x7fcaf8009590 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb10e96700 1 -- 172.21.2.117:0/184396671 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb04001700 con 0x7fcaf8009590 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 -- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 msgr2=0x7fcaf800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 0x7fcaf800bab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fcb04006e30 tx=0x7fcb0404fe50).stop 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 -- 172.21.2.117:0/184396671 shutdown_connections 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 --2- 172.21.2.117:0/184396671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaf8009590 0x7fcaf800bab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 -- 172.21.2.117:0/184396671 >> 172.21.2.117:0/184396671 conn(0x7fcb0c0ce450 msgr2=0x7fcb0c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 -- 172.21.2.117:0/184396671 shutdown_connections 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.401 7fcb1407c700 1 -- 172.21.2.117:0/184396671 wait complete. 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 Processor -- start 2021-02-10T11:23:33.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 -- start start 2021-02-10T11:23:33.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb0c0a46c0 0x7fcb0c0a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb0c0a50e0 0x7fcb0c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0c3040 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c190a20 con 0x7fcb0c0a50e0 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb1407c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0a54e0 con 0x7fcb0c0a46c0 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb11e98700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb11e98700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42330/0 (socket says 172.21.2.117:42330) 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.416 7fcb11e98700 1 -- 172.21.2.117:0/3154684856 learned_addr learned my addr 172.21.2.117:0/3154684856 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 -- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb0c0a46c0 msgr2=0x7fcb0c0a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb0c0a46c0 0x7fcb0c0a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 -- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb0c0a50e0 msgr2=0x7fcb0c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb0c0a50e0 0x7fcb0c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 -- 172.21.2.117:0/3154684856 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0a5700 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb11e98700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fcb0800a570 tx=0x7fcb0800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb080070d0 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0800ad00 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c0a5700 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb08004830 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.417 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c0a5700 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.418 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb080043a0 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.418 7fcafe7fc700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 0x7fcaf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.418 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fcb080328e0 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.418 7fcb11697700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 0x7fcaf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.419 7fcb11697700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 0x7fcaf401abc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.449 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c077a70 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.451 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb08007030 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.577 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fcb0c077a70 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.578 7fcafe7fc700 1 -- 172.21.2.117:0/3154684856 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcb0800ad00 con 0x7fcb0c1a46b0 2021-02-10T11:23:33.575 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 msgr2=0x7fcaf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 0x7fcaf401abc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 msgr2=0x7fcb0c190400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fcb0800a570 tx=0x7fcb0800a910).stop 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 shutdown_connections 2021-02-10T11:23:33.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb0c0a50e0 0x7fcb0c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb0c0a46c0 0x7fcb0c0a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcaf4018730 0x7fcaf401abc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 --2- 172.21.2.117:0/3154684856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb0c1a46b0 0x7fcb0c190400 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 >> 172.21.2.117:0/3154684856 conn(0x7fcb0c0b5140 msgr2=0x7fcb0c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 shutdown_connections 2021-02-10T11:23:33.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.580 7fcb1407c700 1 -- 172.21.2.117:0/3154684856 wait complete. 2021-02-10T11:23:33.585 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.996 7f0fa89f2700 1 Processor -- start 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.996 7f0fa89f2700 1 -- start start 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa89f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa407da20 0x7f0fa40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa89f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0fa40e5190 0x7f0fa40d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa89f2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa4077a70 con 0x7f0fa40e4d90 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa89f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa40e5c10 con 0x7f0fa40e5190 2021-02-10T11:23:34.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa89f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa40777d0 con 0x7f0fa407da20 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0fa40e5190 0x7f0fa40d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0fa40e5190 0x7f0fa40d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42336/0 (socket says 172.21.2.117:42336) 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 -- 172.21.2.117:0/4013054026 learned_addr learned my addr 172.21.2.117:0/4013054026 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 -- 172.21.2.117:0/4013054026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa407da20 msgr2=0x7f0fa40771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 --2- 172.21.2.117:0/4013054026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa407da20 0x7f0fa40771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 -- 172.21.2.117:0/4013054026 >> v1:172.21.2.117:6789/0 conn(0x7f0fa40e4d90 legacy=0x7f0fa41b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 -- 172.21.2.117:0/4013054026 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa41b08d0 con 0x7f0fa40e5190 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.997 7f0fa2e1d700 1 --2- 172.21.2.117:0/4013054026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa40e5190 0x7f0fa40d0510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f9c006f00 tx=0x7f0f9c009f10).ready entity=mon.0 client_cookie=5f57e218081b4cfb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa161a700 1 -- 172.21.2.117:0/4013054026 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f9c00b3b0 con 0x7f0fa40e5190 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa161a700 1 -- 172.21.2.117:0/4013054026 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c00b730 con 0x7f0fa40e5190 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa161a700 1 -- 172.21.2.117:0/4013054026 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f9c00bc50 con 0x7f0fa40e5190 2021-02-10T11:23:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 -- 172.21.2.117:0/4013054026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa40e5190 msgr2=0x7f0fa40d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 --2- 172.21.2.117:0/4013054026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa40e5190 0x7f0fa40d0510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f9c006f00 tx=0x7f0f9c009f10).stop 2021-02-10T11:23:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 -- 172.21.2.117:0/4013054026 shutdown_connections 2021-02-10T11:23:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 --2- 172.21.2.117:0/4013054026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa40e5190 0x7f0fa40d0510 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 --2- 172.21.2.117:0/4013054026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa407da20 0x7f0fa40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 -- 172.21.2.117:0/4013054026 >> 172.21.2.117:0/4013054026 conn(0x7f0fa40ce440 msgr2=0x7f0fa40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 -- 172.21.2.117:0/4013054026 shutdown_connections 2021-02-10T11:23:34.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:33.998 7f0fa89f2700 1 -- 172.21.2.117:0/4013054026 wait complete. 2021-02-10T11:23:34.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 Processor -- start 2021-02-10T11:23:34.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 -- start start 2021-02-10T11:23:34.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa419b1a0 0x7f0fa40adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fa40ae550 0x7f0fa410a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa419b5a0 con 0x7f0fa410a6b0 2021-02-10T11:23:34.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa408af90 con 0x7f0fa40ae550 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa89f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa408b1b0 con 0x7f0fa419b1a0 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.013 7f0fa2e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42340/0 (socket says 172.21.2.117:42340) 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 -- 172.21.2.117:0/2078174510 learned_addr learned my addr 172.21.2.117:0/2078174510 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 -- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa419b1a0 msgr2=0x7f0fa40adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa419b1a0 0x7f0fa40adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 -- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fa40ae550 msgr2=0x7f0fa410a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fa40ae550 0x7f0fa410a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 -- 172.21.2.117:0/2078174510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa409a8c0 con 0x7f0fa410a6b0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa2e1d700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f0f9c009f90 tx=0x7f0f9c00bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f9c00bd80 con 0x7f0fa410a6b0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c009450 con 0x7f0fa410a6b0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f9c010980 con 0x7f0fa410a6b0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa409a8c0 con 0x7f0fa410a6b0 2021-02-10T11:23:34.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.014 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa409ae50 con 0x7f0fa410a6b0 2021-02-10T11:23:34.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.015 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f9c005820 con 0x7f0fa410a6b0 2021-02-10T11:23:34.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.015 7f0f9affd700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 0x7f0f8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.015 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f0f9c036eb0 con 0x7f0fa410a6b0 2021-02-10T11:23:34.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.015 7f0fa261c700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 0x7f0f8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.015 7f0fa261c700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 0x7f0f8801abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.047 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa4077a70 con 0x7f0fa410a6b0 2021-02-10T11:23:34.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.049 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f9c003030 con 0x7f0fa410a6b0 2021-02-10T11:23:34.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.172 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fa4077a70 con 0x7f0fa410a6b0 2021-02-10T11:23:34.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.173 7f0f9affd700 1 -- 172.21.2.117:0/2078174510 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14254 (secure 0 0 0) 0x7f0f9c005820 con 0x7f0fa410a6b0 2021-02-10T11:23:34.170 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:34.171 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:22:32.545531","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:22:27.324233","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2021-02-10 11:22:29.924823","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:34.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.174 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 msgr2=0x7f0f8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.174 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 0x7f0f8801abc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.174 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 msgr2=0x7f0fa409a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.174 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f0f9c009f90 tx=0x7f0f9c00bc40).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 shutdown_connections 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f880186c0 0x7f0f8801abc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fa410a6b0 0x7f0fa409a2a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fa40ae550 0x7f0fa410a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 --2- 172.21.2.117:0/2078174510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fa419b1a0 0x7f0fa40adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 >> 172.21.2.117:0/2078174510 conn(0x7f0fa419c250 msgr2=0x7f0fa410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 shutdown_connections 2021-02-10T11:23:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.175 7f0fa89f2700 1 -- 172.21.2.117:0/2078174510 wait complete. 2021-02-10T11:23:34.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:22:27.324233', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:22:29.924823', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:23:34.181 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 Processor -- start 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 -- start start 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 0x7f19b80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b80d1d20 con 0x7f19b81b00f0 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b8077a70 con 0x7f19b807da20 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.597 7f19bc82d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b80e5430 con 0x7f19b8079000 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b6e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f19b81b00f0 0x7f19b80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42602/0 (socket says 172.21.2.117:42602) 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b6e1d700 1 -- 172.21.2.117:0/402096898 learned_addr learned my addr 172.21.2.117:0/402096898 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b5e1b700 1 --2- 172.21.2.117:0/402096898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 0x7f19b80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b561a700 1 -- 172.21.2.117:0/402096898 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63780061 0 0) 0x7f19b0001500 con 0x7f19b81b00f0 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b561a700 1 -- 172.21.2.117:0/402096898 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a4002fe0 con 0x7f19b81b00f0 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b561a700 1 -- 172.21.2.117:0/402096898 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265874227 0 0) 0x7f19a8001080 con 0x7f19b807da20 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.598 7f19b561a700 1 -- 172.21.2.117:0/402096898 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a4003210 con 0x7f19b807da20 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b5e1b700 1 -- 172.21.2.117:0/402096898 >> v1:172.21.2.125:6789/0 conn(0x7f19b807da20 legacy=0x7f19b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b5e1b700 1 -- 172.21.2.117:0/402096898 >> v1:172.21.2.117:6789/0 conn(0x7f19b81b00f0 legacy=0x7f19b80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b5e1b700 1 -- 172.21.2.117:0/402096898 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b8079400 con 0x7f19b8079000 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b5e1b700 1 --2- 172.21.2.117:0/402096898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 0x7f19b80e4da0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f19a0006650 tx=0x7f19a0009a00).ready entity=mon.2 client_cookie=12b2dbbeee26d8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b561a700 1 -- 172.21.2.117:0/402096898 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a00009b0 con 0x7f19b8079000 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b561a700 1 -- 172.21.2.117:0/402096898 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a000c0c0 con 0x7f19b8079000 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19b561a700 1 -- 172.21.2.117:0/402096898 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a000c5e0 con 0x7f19b8079000 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19bc82d700 1 -- 172.21.2.117:0/402096898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 msgr2=0x7f19b80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.599 7f19bc82d700 1 --2- 172.21.2.117:0/402096898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 0x7f19b80e4da0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f19a0006650 tx=0x7f19a0009a00).stop 2021-02-10T11:23:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.600 7f19bc82d700 1 -- 172.21.2.117:0/402096898 shutdown_connections 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.600 7f19bc82d700 1 --2- 172.21.2.117:0/402096898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b8079000 0x7f19b80e4da0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.600 7f19bc82d700 1 -- 172.21.2.117:0/402096898 >> 172.21.2.117:0/402096898 conn(0x7f19b80ce450 msgr2=0x7f19b80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.600 7f19bc82d700 1 -- 172.21.2.117:0/402096898 shutdown_connections 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.600 7f19bc82d700 1 -- 172.21.2.117:0/402096898 wait complete. 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 Processor -- start 2021-02-10T11:23:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 -- start start 2021-02-10T11:23:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b819d5f0 0x7f19b819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b81b7370 0x7f19b81b7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b80c3d90 con 0x7f19b81b7d90 2021-02-10T11:23:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b80c3fb0 con 0x7f19b81b7370 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19bc82d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b80a2040 con 0x7f19b819d5f0 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19b6e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19b661c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b819d5f0 0x7f19b819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19b6e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42358/0 (socket says 172.21.2.117:42358) 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19b6e1d700 1 -- 172.21.2.117:0/1408110972 learned_addr learned my addr 172.21.2.117:0/1408110972 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.615 7f19b5e1b700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b81b7370 0x7f19b81b7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 -- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b819d5f0 msgr2=0x7f19b819d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b819d5f0 0x7f19b819d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 -- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b81b7370 msgr2=0x7f19b81b7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b81b7370 0x7f19b81b7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 -- 172.21.2.117:0/1408110972 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b80a2260 con 0x7f19b81b7d90 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19b6e1d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f19b0001c10 tx=0x7f19b000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19b0007ce0 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b00070d0 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19b0004930 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b80a2260 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.616 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b80a27f0 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.617 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19b00072b0 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.617 7f19aeffd700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 0x7f199c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.617 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f19b0032a30 con 0x7f19b81b7d90 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.618 7f19b661c700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 0x7f199c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:34.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.619 7f19b661c700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 0x7f199c01abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:34.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.647 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b8077a70 con 0x7f19b81b7d90 2021-02-10T11:23:34.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.650 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19b0000f40 con 0x7f19b81b7d90 2021-02-10T11:23:34.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.776 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f19b8077a70 con 0x7f19b81b7d90 2021-02-10T11:23:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.777 7f19aeffd700 1 -- 172.21.2.117:0/1408110972 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f19b0007290 con 0x7f19b81b7d90 2021-02-10T11:23:34.774 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 msgr2=0x7f199c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 0x7f199c01abc0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 msgr2=0x7f19b81b8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f19b0001c10 tx=0x7f19b000aca0).stop 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 shutdown_connections 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f199c0186c0 0x7f199c01abc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b81b7d90 0x7f19b81b8190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b81b7370 0x7f19b81b7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 --2- 172.21.2.117:0/1408110972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b819d5f0 0x7f19b819d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:34.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 >> 172.21.2.117:0/1408110972 conn(0x7f19b81b9780 msgr2=0x7f19b809be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:34.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 shutdown_connections 2021-02-10T11:23:34.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:34.779 7f19bc82d700 1 -- 172.21.2.117:0/1408110972 wait complete. 2021-02-10T11:23:34.784 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:23:35.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 Processor -- start 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 -- start start 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 0x7f539c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f539c0e4da0 0x7f539c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c077a70 con 0x7f539c0e51a0 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539c0e5c20 con 0x7f539c0e4da0 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.200 7f53a463d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0777d0 con 0x7f539c07da20 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.201 7f53a1c58700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f539c0e4da0 0x7f539c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.201 7f53a2459700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 0x7f539c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.201 7f53a2c5a700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f539c0e51a0 0x7f539c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35538/0 (socket says 172.21.2.117:35538) 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.201 7f53a2c5a700 1 -- 172.21.2.117:0/1975235298 learned_addr learned my addr 172.21.2.117:0/1975235298 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:35.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a2459700 1 -- 172.21.2.117:0/1975235298 >> v2:172.21.2.125:3300/0 conn(0x7f539c0e4da0 msgr2=0x7f539c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a2459700 1 --2- 172.21.2.117:0/1975235298 >> v2:172.21.2.125:3300/0 conn(0x7f539c0e4da0 0x7f539c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a2459700 1 -- 172.21.2.117:0/1975235298 >> v1:172.21.2.125:6789/0 conn(0x7f539c0e51a0 legacy=0x7f539c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a2459700 1 -- 172.21.2.117:0/1975235298 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c07de20 con 0x7f539c07da20 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a2459700 1 --2- 172.21.2.117:0/1975235298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 0x7f539c0771b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5398006650 tx=0x7f5398009a80).ready entity=mon.2 client_cookie=cfeebb15c4c1bc23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a1457700 1 -- 172.21.2.117:0/1975235298 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f539800c0c0 con 0x7f539c07da20 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a1457700 1 -- 172.21.2.117:0/1975235298 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539800c340 con 0x7f539c07da20 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.202 7f53a1457700 1 -- 172.21.2.117:0/1975235298 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f539800c860 con 0x7f539c07da20 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 -- 172.21.2.117:0/1975235298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 msgr2=0x7f539c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 --2- 172.21.2.117:0/1975235298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 0x7f539c0771b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5398006650 tx=0x7f5398009a80).stop 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 -- 172.21.2.117:0/1975235298 shutdown_connections 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 --2- 172.21.2.117:0/1975235298 >> v2:172.21.2.125:3300/0 conn(0x7f539c0e4da0 0x7f539c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 --2- 172.21.2.117:0/1975235298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c07da20 0x7f539c0771b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 -- 172.21.2.117:0/1975235298 >> 172.21.2.117:0/1975235298 conn(0x7f539c0ce450 msgr2=0x7f539c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 -- 172.21.2.117:0/1975235298 shutdown_connections 2021-02-10T11:23:35.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.203 7f53a463d700 1 -- 172.21.2.117:0/1975235298 wait complete. 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 Processor -- start 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 -- start start 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c0a41c0 0x7f539c0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f539c0b7390 0x7f539c0b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0b8010 con 0x7f539c0c2340 2021-02-10T11:23:35.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0b8550 con 0x7f539c0b7390 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a463d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0b8770 con 0x7f539c0a41c0 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a1c58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a2c5a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f539c0b7390 0x7f539c0b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a1c58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42366/0 (socket says 172.21.2.117:42366) 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a2459700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c0a41c0 0x7f539c0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.218 7f53a1c58700 1 -- 172.21.2.117:0/526309227 learned_addr learned my addr 172.21.2.117:0/526309227 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 -- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c0a41c0 msgr2=0x7f539c0fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c0a41c0 0x7f539c0fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 -- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f539c0b7390 msgr2=0x7f539c0b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f539c0b7390 0x7f539c0b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 -- 172.21.2.117:0/526309227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c0b8990 con 0x7f539c0c2340 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a1c58700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5390006f60 tx=0x7f539000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5390006bb0 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5390006340 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5390002030 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a463d700 1 -- 172.21.2.117:0/526309227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c0b8990 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.219 7f53a463d700 1 -- 172.21.2.117:0/526309227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c0b8f20 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.220 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5390006b50 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.220 7f538effd700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 0x7f538401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.220 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f5390032780 con 0x7f539c0c2340 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.220 7f53a2459700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 0x7f538401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.221 7f53a2459700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 0x7f538401abc0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.250 7f53a463d700 1 -- 172.21.2.117:0/526309227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c1b4f00 con 0x7f539c0c2340 2021-02-10T11:23:35.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.253 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5390000960 con 0x7f539c0c2340 2021-02-10T11:23:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.379 7f53a463d700 1 -- 172.21.2.117:0/526309227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f539c077a70 con 0x7f539c0c2340 2021-02-10T11:23:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.380 7f538effd700 1 -- 172.21.2.117:0/526309227 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5390000940 con 0x7f539c0c2340 2021-02-10T11:23:35.377 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:35.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 msgr2=0x7f538401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 0x7f538401abc0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 msgr2=0x7f539c0c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5390006f60 tx=0x7f539000b970).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 shutdown_connections 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53840186c0 0x7f538401abc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f539c0c2340 0x7f539c0c2740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f539c0b7390 0x7f539c0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 --2- 172.21.2.117:0/526309227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f539c0a41c0 0x7f539c0fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 >> 172.21.2.117:0/526309227 conn(0x7f539c08bbc0 msgr2=0x7f539c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 shutdown_connections 2021-02-10T11:23:35.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.382 7f53a463d700 1 -- 172.21.2.117:0/526309227 wait complete. 2021-02-10T11:23:35.388 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:23:35.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.803 7f38c7788700 1 Processor -- start 2021-02-10T11:23:35.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.803 7f38c7788700 1 -- start start 2021-02-10T11:23:35.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c7788700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c0079000 0x7f38c00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c7788700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c01b00f0 0x7f38c01b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c7788700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c0077a70 con 0x7f38c007da20 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c7788700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c00e53c0 con 0x7f38c01b00f0 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c7788700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0079400 con 0x7f38c0079000 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c01b00f0 0x7f38c01b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c01b00f0 0x7f38c01b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42378/0 (socket says 172.21.2.117:42378) 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 -- 172.21.2.117:0/4070092349 learned_addr learned my addr 172.21.2.117:0/4070092349 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 -- 172.21.2.117:0/4070092349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c0079000 msgr2=0x7f38c00e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 --2- 172.21.2.117:0/4070092349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c0079000 0x7f38c00e4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 -- 172.21.2.117:0/4070092349 >> v1:172.21.2.117:6789/0 conn(0x7f38c007da20 legacy=0x7f38c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 -- 172.21.2.117:0/4070092349 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c007de20 con 0x7f38c01b00f0 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.804 7f38c5da5700 1 --2- 172.21.2.117:0/4070092349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c01b00f0 0x7f38c01b04f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f38b8006f00 tx=0x7f38b8009f10).ready entity=mon.0 client_cookie=e07bcf983f09caa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38b3fff700 1 -- 172.21.2.117:0/4070092349 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38b800b3b0 con 0x7f38c01b00f0 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38b3fff700 1 -- 172.21.2.117:0/4070092349 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b800b730 con 0x7f38c01b00f0 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38b3fff700 1 -- 172.21.2.117:0/4070092349 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38b800bb90 con 0x7f38c01b00f0 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 -- 172.21.2.117:0/4070092349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c01b00f0 msgr2=0x7f38c01b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 --2- 172.21.2.117:0/4070092349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c01b00f0 0x7f38c01b04f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f38b8006f00 tx=0x7f38b8009f10).stop 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 -- 172.21.2.117:0/4070092349 shutdown_connections 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 --2- 172.21.2.117:0/4070092349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c01b00f0 0x7f38c01b04f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 --2- 172.21.2.117:0/4070092349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c0079000 0x7f38c00e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.805 7f38c7788700 1 -- 172.21.2.117:0/4070092349 >> 172.21.2.117:0/4070092349 conn(0x7f38c00ce450 msgr2=0x7f38c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.806 7f38c7788700 1 -- 172.21.2.117:0/4070092349 shutdown_connections 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.806 7f38c7788700 1 -- 172.21.2.117:0/4070092349 wait complete. 2021-02-10T11:23:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.820 7f38c7788700 1 Processor -- start 2021-02-10T11:23:35.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.820 7f38c7788700 1 -- start start 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00b0f40 0x7f38c0107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c018b830 0x7f38c0187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00b18c0 con 0x7f38c00b1340 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c018ac90 con 0x7f38c00b0f40 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0107f50 con 0x7f38c018b830 2021-02-10T11:23:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42380/0 (socket says 172.21.2.117:42380) 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 -- 172.21.2.117:0/3045187833 learned_addr learned my addr 172.21.2.117:0/3045187833 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 -- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c018b830 msgr2=0x7f38c0187bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c018b830 0x7f38c0187bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 -- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00b0f40 msgr2=0x7f38c0107930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00b0f40 0x7f38c0107930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 -- 172.21.2.117:0/3045187833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0108130 con 0x7f38c00b1340 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c4da3700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f38bc00b560 tx=0x7f38bc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38bc00bcf0 con 0x7f38c00b1340 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc006030 con 0x7f38c00b1340 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38bc0054d0 con 0x7f38c00b1340 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 -- 172.21.2.117:0/3045187833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c0108130 con 0x7f38c00b1340 2021-02-10T11:23:35.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.821 7f38c7788700 1 -- 172.21.2.117:0/3045187833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c0108130 con 0x7f38c00b1340 2021-02-10T11:23:35.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.822 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38bc006210 con 0x7f38c00b1340 2021-02-10T11:23:35.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.822 7f38b1ffb700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 0x7f38a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:35.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.822 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f38bc032780 con 0x7f38c00b1340 2021-02-10T11:23:35.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.822 7f38c55a4700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 0x7f38a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:35.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.823 7f38c55a4700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 0x7f38a001abc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:35.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.854 7f38c7788700 1 -- 172.21.2.117:0/3045187833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c0077a70 con 0x7f38c00b1340 2021-02-10T11:23:35.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.856 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38bc006b80 con 0x7f38c00b1340 2021-02-10T11:23:35.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.985 7f38c7788700 1 -- 172.21.2.117:0/3045187833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f38c0077a70 con 0x7f38c00b1340 2021-02-10T11:23:35.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.985 7f38b1ffb700 1 -- 172.21.2.117:0/3045187833 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f38bc006030 con 0x7f38c00b1340 2021-02-10T11:23:35.983 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:35.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.987 7f38c7788700 1 -- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 msgr2=0x7f38a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.987 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 0x7f38a001abc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.987 7f38c7788700 1 -- 172.21.2.117:0/3045187833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 msgr2=0x7f38c018b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:35.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.987 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f38bc00b560 tx=0x7f38bc00b900).stop 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 -- 172.21.2.117:0/3045187833 shutdown_connections 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a0018700 0x7f38a001abc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00b1340 0x7f38c018b210 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c018b830 0x7f38c0187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 --2- 172.21.2.117:0/3045187833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00b0f40 0x7f38c0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:35.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 -- 172.21.2.117:0/3045187833 >> 172.21.2.117:0/3045187833 conn(0x7f38c00a1880 msgr2=0x7f38c00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:35.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 -- 172.21.2.117:0/3045187833 shutdown_connections 2021-02-10T11:23:35.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:35.988 7f38c7788700 1 -- 172.21.2.117:0/3045187833 wait complete. 2021-02-10T11:23:36.029 INFO:tasks.cephfs_test_runner:Starting test: test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:23:36.029 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.442 7ff5beca7700 1 Processor -- start 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.442 7ff5beca7700 1 -- start start 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5beca7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b8079000 0x7ff5b80e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5beca7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80d1d10 con 0x7ff5b81b00e0 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5beca7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b8077a70 con 0x7ff5b807da20 2021-02-10T11:23:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5beca7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80e5420 con 0x7ff5b8079000 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5bd2c4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff5b81b00e0 0x7ff5b80d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42638/0 (socket says 172.21.2.117:42638) 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5bd2c4700 1 -- 172.21.2.117:0/3611169111 learned_addr learned my addr 172.21.2.117:0/3611169111 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277966345 0 0) 0x7ff5b0001180 con 0x7ff5b81b00e0 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a4002fe0 con 0x7ff5b81b00e0 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.443 7ff5b7fff700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b8079000 0x7ff5b80e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 988346135 0 0) 0x7ff5b00010e0 con 0x7ff5b81b00e0 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b8079000 msgr2=0x7ff5b80e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b8079000 0x7ff5b80e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 >> v1:172.21.2.125:6789/0 conn(0x7ff5b807da20 legacy=0x7ff5b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8079400 con 0x7ff5b81b00e0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff5b0001600 con 0x7ff5b81b00e0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 >> v1:172.21.2.117:6789/0 conn(0x7ff5b81b00e0 legacy=0x7ff5b80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5a4003ac0 0x7ff5a4005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5a40067b0 0x7ff5a4008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5a40095b0 0x7ff5a400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a400c2d0 con 0x7ff5a4003ac0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a400c4f0 con 0x7ff5a40067b0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a400c710 con 0x7ff5a40095b0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5a4003ac0 0x7ff5a4005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5b7fff700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5a40067b0 0x7ff5a4008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bd2c4700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5a40095b0 0x7ff5a400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 -- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5a40095b0 msgr2=0x7ff5a400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5a40095b0 0x7ff5a400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 -- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5a40067b0 msgr2=0x7ff5a4008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5a40067b0 0x7ff5a4008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.444 7ff5bcac3700 1 -- 172.21.2.117:0/3611169111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4050980 con 0x7ff5a4003ac0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5bcac3700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5a4003ac0 0x7ff5a4005f10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ff5a0006ab0 tx=0x7ff5a0009b40).ready entity=mon.0 client_cookie=209a82087d37ea3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a000b2e0 con 0x7ff5a4003ac0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a000b560 con 0x7ff5a4003ac0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5b77fe700 1 -- 172.21.2.117:0/3611169111 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a000ba80 con 0x7ff5a4003ac0 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5beca7700 1 -- 172.21.2.117:0/3611169111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5a4003ac0 msgr2=0x7ff5a4005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5beca7700 1 --2- 172.21.2.117:0/3611169111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5a4003ac0 0x7ff5a4005f10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ff5a0006ab0 tx=0x7ff5a0009b40).stop 2021-02-10T11:23:36.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.445 7ff5bd2c4700 1 -- 172.21.2.117:0/3611169111 reap_dead start 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.446 7ff5beca7700 1 -- 172.21.2.117:0/3611169111 shutdown_connections 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.446 7ff5beca7700 1 -- 172.21.2.117:0/3611169111 >> 172.21.2.117:0/3611169111 conn(0x7ff5b80ce440 msgr2=0x7ff5b80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.446 7ff5beca7700 1 -- 172.21.2.117:0/3611169111 shutdown_connections 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.446 7ff5beca7700 1 -- 172.21.2.117:0/3611169111 wait complete. 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 Processor -- start 2021-02-10T11:23:36.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 -- start start 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b80b8bd0 0x7ff5b8192ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5b818ebe0 0x7ff5b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8193590 con 0x7ff5b8106ad0 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b8106ed0 con 0x7ff5b818ebe0 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5beca7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81931c0 con 0x7ff5b80b8bd0 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5bd2c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5bd2c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42400/0 (socket says 172.21.2.117:42400) 2021-02-10T11:23:36.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.461 7ff5bd2c4700 1 -- 172.21.2.117:0/2901445282 learned_addr learned my addr 172.21.2.117:0/2901445282 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 -- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b80b8bd0 msgr2=0x7ff5b8192ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b80b8bd0 0x7ff5b8192ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 -- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5b818ebe0 msgr2=0x7ff5b81064b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5b818ebe0 0x7ff5b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 -- 172.21.2.117:0/2901445282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8194130 con 0x7ff5b8106ad0 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5bd2c4700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ff5b000b690 tx=0x7ff5b000ba30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5b0007b80 con 0x7ff5b8106ad0 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000be20 con 0x7ff5b8106ad0 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b8194130 con 0x7ff5b8106ad0 2021-02-10T11:23:36.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5b00043a0 con 0x7ff5b8106ad0 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.462 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b8194130 con 0x7ff5b8106ad0 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.463 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5b0007b80 con 0x7ff5b8106ad0 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.463 7ff5b57fa700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 0x7ff59c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.463 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7ff5b0032960 con 0x7ff5b8106ad0 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.463 7ff5bcac3700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 0x7ff59c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:36.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.464 7ff5bcac3700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 0x7ff59c01abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:36.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.492 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b8077a70 con 0x7ff5b8106ad0 2021-02-10T11:23:36.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.495 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5b0004ef0 con 0x7ff5b8106ad0 2021-02-10T11:23:36.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.625 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7ff5b8077a70 con 0x7ff5b8106ad0 2021-02-10T11:23:36.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.924 7ff5b57fa700 1 -- 172.21.2.117:0/2901445282 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v73) v1 ==== 143+0+0 (secure 0 0 0) 0x7ff5b0007f00 con 0x7ff5b8106ad0 2021-02-10T11:23:36.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 msgr2=0x7ff59c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 0x7ff59c01abc0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 msgr2=0x7ff5b8193b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:36.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ff5b000b690 tx=0x7ff5b000ba30).stop 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 shutdown_connections 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff59c018730 0x7ff59c01abc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5b8106ad0 0x7ff5b8193b10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5b818ebe0 0x7ff5b81064b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 --2- 172.21.2.117:0/2901445282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5b80b8bd0 0x7ff5b8192ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 >> 172.21.2.117:0/2901445282 conn(0x7ff5b819f510 msgr2=0x7ff5b81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 shutdown_connections 2021-02-10T11:23:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:36.944 7ff5beca7700 1 -- 172.21.2.117:0/2901445282 wait complete. 2021-02-10T11:23:36.949 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.364 7f3502f2a700 1 Processor -- start 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.364 7f3502f2a700 1 -- start start 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3502f2a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f34fc07da20 0x7f34fc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3502f2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 0x7f34fc1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3502f2a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34fc077a70 con 0x7f34fc0e5180 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3502f2a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc0777d0 con 0x7f34fc0e4d80 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3502f2a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34fc07de20 con 0x7f34fc07da20 2021-02-10T11:23:37.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f34fbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 0x7f34fc1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f34fbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 0x7f34fc1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42408/0 (socket says 172.21.2.117:42408) 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f34fbfff700 1 -- 172.21.2.117:0/3514588550 learned_addr learned my addr 172.21.2.117:0/3514588550 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3500d46700 1 --2- 172.21.2.117:0/3514588550 >> v2:172.21.2.125:3300/0 conn(0x7f34fc07da20 0x7f34fc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3500d46700 1 -- 172.21.2.117:0/3514588550 >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 msgr2=0x7f34fc1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3500d46700 1 --2- 172.21.2.117:0/3514588550 >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 0x7f34fc1b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.365 7f3500d46700 1 -- 172.21.2.117:0/3514588550 >> v1:172.21.2.125:6789/0 conn(0x7f34fc0e5180 legacy=0x7f34fc077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3500d46700 1 -- 172.21.2.117:0/3514588550 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc1b06f0 con 0x7f34fc07da20 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3500d46700 1 --2- 172.21.2.117:0/3514588550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc07da20 0x7f34fc0771b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f34f40069d0 tx=0x7f34f4009e40).ready entity=mon.1 client_cookie=36c10bfc9cf067f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f34fb7fe700 1 -- 172.21.2.117:0/3514588550 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34f400b510 con 0x7f34fc07da20 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f34fb7fe700 1 -- 172.21.2.117:0/3514588550 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400b790 con 0x7f34fc07da20 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f34fb7fe700 1 -- 172.21.2.117:0/3514588550 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34f400bcb0 con 0x7f34fc07da20 2021-02-10T11:23:37.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 -- 172.21.2.117:0/3514588550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc07da20 msgr2=0x7f34fc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 --2- 172.21.2.117:0/3514588550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc07da20 0x7f34fc0771b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f34f40069d0 tx=0x7f34f4009e40).stop 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 -- 172.21.2.117:0/3514588550 shutdown_connections 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 --2- 172.21.2.117:0/3514588550 >> v2:172.21.2.117:3300/0 conn(0x7f34fc0e4d80 0x7f34fc1b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 --2- 172.21.2.117:0/3514588550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc07da20 0x7f34fc0771b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.366 7f3502f2a700 1 -- 172.21.2.117:0/3514588550 >> 172.21.2.117:0/3514588550 conn(0x7f34fc0ce430 msgr2=0x7f34fc0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.367 7f3502f2a700 1 -- 172.21.2.117:0/3514588550 shutdown_connections 2021-02-10T11:23:37.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.367 7f3502f2a700 1 -- 172.21.2.117:0/3514588550 wait complete. 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.381 7f3502f2a700 1 Processor -- start 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.381 7f3502f2a700 1 -- start start 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc0a19b0 0x7f34fc102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34fc0bbac0 0x7f34fc0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0ad9e0 con 0x7f34fc0ae580 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0bbec0 con 0x7f34fc0a19b0 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3502f2a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc0bc0e0 con 0x7f34fc0bbac0 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42416/0 (socket says 172.21.2.117:42416) 2021-02-10T11:23:37.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 -- 172.21.2.117:0/3090032875 learned_addr learned my addr 172.21.2.117:0/3090032875 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f34fbfff700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34fc0bbac0 0x7f34fc0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3500d46700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc0a19b0 0x7f34fc102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 -- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34fc0bbac0 msgr2=0x7f34fc0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34fc0bbac0 0x7f34fc0adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 -- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc0a19b0 msgr2=0x7f34fc102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc0a19b0 0x7f34fc102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 -- 172.21.2.117:0/3090032875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0c27c0 con 0x7f34fc0ae580 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.382 7f3501547700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f34ec006ed0 tx=0x7f34ec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.383 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34ec00bd50 con 0x7f34fc0ae580 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.383 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec000c10 con 0x7f34fc0ae580 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.383 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34ec0062a0 con 0x7f34fc0ae580 2021-02-10T11:23:37.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.383 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34fc0c27c0 con 0x7f34fc0ae580 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.383 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34fc0c2d50 con 0x7f34fc0ae580 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.384 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34ec006020 con 0x7f34fc0ae580 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.384 7f34f97fa700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 0x7f34dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.384 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f34ec032690 con 0x7f34fc0ae580 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.384 7f3500d46700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 0x7f34dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:37.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.384 7f3500d46700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 0x7f34dc01abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.415 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34fc1b4ee0 con 0x7f34fc0ae580 2021-02-10T11:23:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.417 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34ec000940 con 0x7f34fc0ae580 2021-02-10T11:23:37.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.565 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f34fc077a70 con 0x7f34fc0ae580 2021-02-10T11:23:37.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.569 7f34f97fa700 1 -- 172.21.2.117:0/3090032875 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f34ec006020 con 0x7f34fc0ae580 2021-02-10T11:23:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 msgr2=0x7f34dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 0x7f34dc01abc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 msgr2=0x7f34fc0c21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f34ec006ed0 tx=0x7f34ec00b960).stop 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 shutdown_connections 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34dc0186c0 0x7f34dc01abc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34fc0ae580 0x7f34fc0c21a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34fc0bbac0 0x7f34fc0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 --2- 172.21.2.117:0/3090032875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34fc0a19b0 0x7f34fc102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 >> 172.21.2.117:0/3090032875 conn(0x7f34fc0a3820 msgr2=0x7f34fc0a3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.574 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 shutdown_connections 2021-02-10T11:23:37.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:37.575 7f3502f2a700 1 -- 172.21.2.117:0/3090032875 wait complete. 2021-02-10T11:23:37.579 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:23:37.579 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:37.936 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:23:38.363 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:23:38.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.364 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:38.364 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:38.699 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:23:38.699 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:38.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:38.700 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:38.700 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:39.035 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: Transport endpoint is not connected 2021-02-10T11:23:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:39.036 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2021-02-10T11:23:39.036 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:23:39.036 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:39.036 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:39.393 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:39.394 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:39.394 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:39.727 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:23:39.727 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:39.727 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:23:39.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:39.728 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:23:40.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:40.069 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:40.069 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:40.404 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:23:40.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:40.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:40.405 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:40.405 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:40.740 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: Transport endpoint is not connected 2021-02-10T11:23:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:40.740 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2021-02-10T11:23:40.741 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:23:40.741 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:40.741 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:41.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:41.093 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:41.093 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:41.427 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:23:41.428 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:41.428 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:23:41.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:41.428 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:23:41.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:41.764 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:41.764 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:42.101 INFO:teuthology.orchestra.run.gibba025.stdout:fuseblk 2021-02-10T11:23:42.101 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:42.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:42.101 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:42.102 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:42.434 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: Transport endpoint is not connected 2021-02-10T11:23:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:42.435 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2021-02-10T11:23:42.435 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba025.front.sepia.ceph.com... 2021-02-10T11:23:42.435 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:42.435 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:42.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:42.816 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:42.816 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:43.150 INFO:teuthology.orchestra.run.gibba025.stdout:ext2/ext3 2021-02-10T11:23:43.150 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:43.151 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:23:43.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:43.151 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:23:43.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:43.497 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:43.497 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:43.831 INFO:teuthology.orchestra.run.gibba025.stdout:fuseblk 2021-02-10T11:23:43.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:43.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:43.832 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:43.832 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:44.166 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: Transport endpoint is not connected 2021-02-10T11:23:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:23:44.167 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.3 2021-02-10T11:23:44.167 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba025.front.sepia.ceph.com... 2021-02-10T11:23:44.167 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:23:44.167 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:44.526 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:23:44.526 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:23:44.526 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:44.863 INFO:teuthology.orchestra.run.gibba025.stdout:ext2/ext3 2021-02-10T11:23:44.863 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:23:44.864 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:23:44.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:23:44.864 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:23:45.201 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:23:45.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 Processor -- start 2021-02-10T11:23:45.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 -- start start 2021-02-10T11:23:45.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f30a407da20 0x7f30a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f30a40e5190 0x7f30a4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a4077a70 con 0x7f30a40e4d90 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40e5c10 con 0x7f30a40e5190 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a98f5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40777d0 con 0x7f30a407da20 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a3d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f30a40e5190 0x7f30a4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a3d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f30a40e5190 0x7f30a4079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42420/0 (socket says 172.21.2.117:42420) 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.613 7f30a3d75700 1 -- 172.21.2.117:0/2316907439 learned_addr learned my addr 172.21.2.117:0/2316907439 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a3d75700 1 -- 172.21.2.117:0/2316907439 >> v2:172.21.2.125:3300/0 conn(0x7f30a407da20 msgr2=0x7f30a40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a3d75700 1 --2- 172.21.2.117:0/2316907439 >> v2:172.21.2.125:3300/0 conn(0x7f30a407da20 0x7f30a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a3d75700 1 -- 172.21.2.117:0/2316907439 >> v1:172.21.2.125:6789/0 conn(0x7f30a40e4d90 legacy=0x7f30a41b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a3d75700 1 -- 172.21.2.117:0/2316907439 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a41b08d0 con 0x7f30a40e5190 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a3d75700 1 --2- 172.21.2.117:0/2316907439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a40e5190 0x7f30a4079000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f309c006b80 tx=0x7f309c009b90).ready entity=mon.0 client_cookie=9332fa2922c1ad62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a2572700 1 -- 172.21.2.117:0/2316907439 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c0022e0 con 0x7f30a40e5190 2021-02-10T11:23:45.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a2572700 1 -- 172.21.2.117:0/2316907439 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c002560 con 0x7f30a40e5190 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a2572700 1 -- 172.21.2.117:0/2316907439 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309c002a80 con 0x7f30a40e5190 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a98f5700 1 -- 172.21.2.117:0/2316907439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a40e5190 msgr2=0x7f30a4079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.614 7f30a98f5700 1 --2- 172.21.2.117:0/2316907439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a40e5190 0x7f30a4079000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f309c006b80 tx=0x7f309c009b90).stop 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 -- 172.21.2.117:0/2316907439 shutdown_connections 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 --2- 172.21.2.117:0/2316907439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a40e5190 0x7f30a4079000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 --2- 172.21.2.117:0/2316907439 >> v2:172.21.2.125:3300/0 conn(0x7f30a407da20 0x7f30a40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 -- 172.21.2.117:0/2316907439 >> 172.21.2.117:0/2316907439 conn(0x7f30a40ce440 msgr2=0x7f30a40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 -- 172.21.2.117:0/2316907439 shutdown_connections 2021-02-10T11:23:45.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.615 7f30a98f5700 1 -- 172.21.2.117:0/2316907439 wait complete. 2021-02-10T11:23:45.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 Processor -- start 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 -- start start 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f30a419d920 0x7f30a4109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f30a41a8c60 0x7f30a41a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a419d3a0 con 0x7f30a410a210 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a41af340 con 0x7f30a41a8c60 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a98f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a419aea0 con 0x7f30a419d920 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42426/0 (socket says 172.21.2.117:42426) 2021-02-10T11:23:45.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.630 7f30a2d73700 1 -- 172.21.2.117:0/779944119 learned_addr learned my addr 172.21.2.117:0/779944119 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 -- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f30a419d920 msgr2=0x7f30a4109bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f30a419d920 0x7f30a4109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 -- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f30a41a8c60 msgr2=0x7f30a41a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f30a41a8c60 0x7f30a41a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 -- 172.21.2.117:0/779944119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a410a610 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a2d73700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f309400b5b0 tx=0x7f309400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f309400bd40 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3094000e20 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3094006210 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a98f5700 1 -- 172.21.2.117:0/779944119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a410a610 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.631 7f30a98f5700 1 -- 172.21.2.117:0/779944119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41a9970 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.632 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3094006020 con 0x7f30a410a210 2021-02-10T11:23:45.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.632 7f309bfff700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 0x7f308801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:45.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.632 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7f3094032540 con 0x7f30a410a210 2021-02-10T11:23:45.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.633 7f30a3574700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 0x7f308801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:45.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.633 7f30a3574700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 0x7f308801abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:45.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.663 7f30a98f5700 1 -- 172.21.2.117:0/779944119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4108ba0 con 0x7f30a410a210 2021-02-10T11:23:45.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.666 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f309402def0 con 0x7f30a410a210 2021-02-10T11:23:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.787 7f30a98f5700 1 -- 172.21.2.117:0/779944119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f30a4077a70 con 0x7f30a410a210 2021-02-10T11:23:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.788 7f309bfff700 1 -- 172.21.2.117:0/779944119 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v22) v1 ==== 81+0+30780 (secure 0 0 0) 0x7f309402def0 con 0x7f30a410a210 2021-02-10T11:23:45.785 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 22, 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:22:32.545531", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:23:45.786 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 3, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:23:45.787 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:23:45.788 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:45.789 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:23:45.790 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:45.791 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:45.792 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 2, 2021-02-10T11:23:45.793 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:27.324233", 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:45.794 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.795 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "22", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:23:45.796 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:23:45.797 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:23:45.798 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.799 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 3, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:29.924823", 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 5, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool,ec_overwrites", 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 3, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 4, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 3, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 1, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:23:45.800 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.801 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "22", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:23:45.802 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "cephfs_data", 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:23:45.803 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 8192, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:23:45.804 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:23:45.805 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 20, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.806 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.807 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.808 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.809 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:23:45.810 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.811 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 20, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.812 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.813 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:23:45.814 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.815 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.816 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.817 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 17, 2021-02-10T11:23:45.818 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:23:45.819 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.820 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.821 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.822 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.823 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.824 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 20, 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.825 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:23:45.826 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:23:45.827 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.828 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:23:45.829 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.830 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 20, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.831 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.832 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:23:45.833 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.834 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:23:45.835 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.836 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 20, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.837 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.838 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.839 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:23:45.840 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:23:45.841 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.842 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 17, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.843 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.844 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.845 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:23:45.846 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:23:45.847 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:23:45.848 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 17, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.849 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.850 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.851 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:23:45.852 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:23:45.853 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:23:45.854 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.856 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.857 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.858 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:23:45.859 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:23:45.860 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:23:45.861 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:23:45.862 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.789 7f30a98f5700 1 -- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 msgr2=0x7f308801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.789 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 0x7f308801abc0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.789 7f30a98f5700 1 -- 172.21.2.117:0/779944119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 msgr2=0x7f30a41a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.789 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f309400b5b0 tx=0x7f309400b950).stop 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 -- 172.21.2.117:0/779944119 shutdown_connections 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30880186c0 0x7f308801abc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f30a410a210 0x7f30a41a8640 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f30a41a8c60 0x7f30a41a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 --2- 172.21.2.117:0/779944119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f30a419d920 0x7f30a4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:45.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 -- 172.21.2.117:0/779944119 >> 172.21.2.117:0/779944119 conn(0x7f30a419a700 msgr2=0x7f30a4199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:45.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 -- 172.21.2.117:0/779944119 shutdown_connections 2021-02-10T11:23:45.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:45.790 7f30a98f5700 1 -- 172.21.2.117:0/779944119 wait complete. 2021-02-10T11:23:45.865 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:46.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.277 7fa0e17b8700 1 Processor -- start 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.277 7fa0e17b8700 1 -- start start 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.277 7fa0e17b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc07da20 0x7fa0dc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.277 7fa0e17b8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0e17b8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc077a70 con 0x7fa0dc0e5190 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0e17b8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc0e5c10 con 0x7fa0dc0e4d90 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0e17b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0777d0 con 0x7fa0dc07da20 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0dad73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0dad73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:46818/0 (socket says 172.21.2.117:46818) 2021-02-10T11:23:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0dad73700 1 -- 172.21.2.117:0/938237083 learned_addr learned my addr 172.21.2.117:0/938237083 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.278 7fa0db574700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc07da20 0x7fa0dc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0dad73700 1 -- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc07da20 msgr2=0x7fa0dc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0dad73700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc07da20 0x7fa0dc0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0dad73700 1 -- 172.21.2.117:0/938237083 >> v1:172.21.2.125:6789/0 conn(0x7fa0dc0e5190 legacy=0x7fa0dc077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0dad73700 1 -- 172.21.2.117:0/938237083 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc07de20 con 0x7fa0dc0e4d90 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0dad73700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa0cc006700 tx=0x7fa0cc009b80).ready entity=mon.1 client_cookie=1449f81937571a56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0da572700 1 -- 172.21.2.117:0/938237083 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0cc00c0c0 con 0x7fa0dc0e4d90 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0da572700 1 -- 172.21.2.117:0/938237083 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc00c340 con 0x7fa0dc0e4d90 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0da572700 1 -- 172.21.2.117:0/938237083 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0cc00c860 con 0x7fa0dc0e4d90 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0e17b8700 1 -- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc0e4d90 msgr2=0x7fa0dc1b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.279 7fa0e17b8700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa0cc006700 tx=0x7fa0cc009b80).stop 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 -- 172.21.2.117:0/938237083 shutdown_connections 2021-02-10T11:23:46.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc0e4d90 0x7fa0dc1b00e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 --2- 172.21.2.117:0/938237083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc07da20 0x7fa0dc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 -- 172.21.2.117:0/938237083 >> 172.21.2.117:0/938237083 conn(0x7fa0dc0ce440 msgr2=0x7fa0dc0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 -- 172.21.2.117:0/938237083 shutdown_connections 2021-02-10T11:23:46.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.280 7fa0e17b8700 1 -- 172.21.2.117:0/938237083 wait complete. 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 Processor -- start 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 -- start start 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 0x7fa0dc18e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc18ef10 0x7fa0dc18f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc18d930 con 0x7fa0dc18d530 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc18e230 con 0x7fa0dc18e4f0 2021-02-10T11:23:46.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.295 7fa0e17b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc192f30 con 0x7fa0dc18ef10 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42438/0 (socket says 172.21.2.117:42438) 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 -- 172.21.2.117:0/2186549516 learned_addr learned my addr 172.21.2.117:0/2186549516 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0dbd75700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc18ef10 0x7fa0dc18f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0dad73700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 0x7fa0dc18e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 -- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc18ef10 msgr2=0x7fa0dc18f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc18ef10 0x7fa0dc18f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 -- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 msgr2=0x7fa0dc18e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 0x7fa0dc18e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 -- 172.21.2.117:0/2186549516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc193150 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0dad73700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 0x7fa0dc18e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0db574700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa0c4006f60 tx=0x7fa0c400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c40009b0 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c400bdf0 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc193150 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c4002030 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.296 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc1936e0 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.297 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0c4000c90 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.297 7fa0d3fff700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 0x7fa0c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.297 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fa0c4032830 con 0x7fa0dc18d530 2021-02-10T11:23:46.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.297 7fa0dad73700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 0x7fa0c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.298 7fa0dad73700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 0x7fa0c001abc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.327 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc1b4f30 con 0x7fa0dc18d530 2021-02-10T11:23:46.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.330 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0c4000940 con 0x7fa0dc18d530 2021-02-10T11:23:46.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.459 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0dc077a70 con 0x7fa0dc18d530 2021-02-10T11:23:46.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.459 7fa0d3fff700 1 -- 172.21.2.117:0/2186549516 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2094 (secure 0 0 0) 0x7fa0c400bd60 con 0x7fa0dc18d530 2021-02-10T11:23:46.457 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:46.457 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:22:41.961589","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/921969069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":921969069},{"type":"v1","addr":"172.21.2.125:6835","nonce":921969069}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:23:46.457 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 6 2021-02-10T11:23:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 msgr2=0x7fa0c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 0x7fa0c001abc0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 msgr2=0x7fa0dc18dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa0c4006f60 tx=0x7fa0c400b970).stop 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 shutdown_connections 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0dc18ef10 0x7fa0dc18f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0dc18e4f0 0x7fa0dc18e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0c0018730 0x7fa0c001abc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 --2- 172.21.2.117:0/2186549516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0dc18d530 0x7fa0dc18dc10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.461 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 >> 172.21.2.117:0/2186549516 conn(0x7fa0dc1ace70 msgr2=0x7fa0dc1af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.462 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 shutdown_connections 2021-02-10T11:23:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.462 7fa0e17b8700 1 -- 172.21.2.117:0/2186549516 wait complete. 2021-02-10T11:23:46.467 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 Processor -- start 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 -- start start 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 0x7fd6a81b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6a80e4190 0x7fd6a80e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a8109ab0 con 0x7fd6a80d0ac0 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a81b3c60 con 0x7fd6a80e4190 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6ae5f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80d0ec0 con 0x7fd6a8109600 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6acc0f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6a80e4190 0x7fd6a80e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6a7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd6a80d0ac0 0x7fd6a81b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42696/0 (socket says 172.21.2.117:42696) 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.879 7fd6a7fff700 1 -- 172.21.2.117:0/3885896554 learned_addr learned my addr 172.21.2.117:0/3885896554 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6acc0f700 1 -- 172.21.2.117:0/3885896554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 msgr2=0x7fd6a81b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6a77fe700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 0x7fd6a81b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6acc0f700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 0x7fd6a81b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6acc0f700 1 -- 172.21.2.117:0/3885896554 >> v1:172.21.2.117:6789/0 conn(0x7fd6a80d0ac0 legacy=0x7fd6a81b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6acc0f700 1 -- 172.21.2.117:0/3885896554 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a810b040 con 0x7fd6a80e4190 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6a77fe700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 0x7fd6a81b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6acc0f700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a80e4190 0x7fd6a80e4590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd6a0006e30 tx=0x7fd6a0009e60).ready entity=mon.0 client_cookie=439a41a8fa31c1cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6a6ffd700 1 -- 172.21.2.117:0/3885896554 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6a0006a50 con 0x7fd6a80e4190 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6a6ffd700 1 -- 172.21.2.117:0/3885896554 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a000b300 con 0x7fd6a80e4190 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6a6ffd700 1 -- 172.21.2.117:0/3885896554 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6a000b7e0 con 0x7fd6a80e4190 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6ae5f2700 1 -- 172.21.2.117:0/3885896554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a80e4190 msgr2=0x7fd6a80e4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.880 7fd6ae5f2700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a80e4190 0x7fd6a80e4590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fd6a0006e30 tx=0x7fd6a0009e60).stop 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 -- 172.21.2.117:0/3885896554 shutdown_connections 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a80e4190 0x7fd6a80e4590 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 --2- 172.21.2.117:0/3885896554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a8109600 0x7fd6a81b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 -- 172.21.2.117:0/3885896554 >> 172.21.2.117:0/3885896554 conn(0x7fd6a80ce410 msgr2=0x7fd6a80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 -- 172.21.2.117:0/3885896554 shutdown_connections 2021-02-10T11:23:46.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.881 7fd6ae5f2700 1 -- 172.21.2.117:0/3885896554 wait complete. 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.896 7fd6ae5f2700 1 Processor -- start 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.896 7fd6ae5f2700 1 -- start start 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6a80adb50 0x7fd6a80901b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a80907d0 0x7fd6a8090bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81b9c00 con 0x7fd6a81b9800 2021-02-10T11:23:46.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a819b5c0 con 0x7fd6a80adb50 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6ae5f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a80911f0 con 0x7fd6a80907d0 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42454/0 (socket says 172.21.2.117:42454) 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 -- 172.21.2.117:0/1314376352 learned_addr learned my addr 172.21.2.117:0/1314376352 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6acc0f700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a80907d0 0x7fd6a8090bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a77fe700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6a80adb50 0x7fd6a80901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 -- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a80907d0 msgr2=0x7fd6a8090bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a80907d0 0x7fd6a8090bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 -- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6a80adb50 msgr2=0x7fd6a80901b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6a80adb50 0x7fd6a80901b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.897 7fd6a7fff700 1 -- 172.21.2.117:0/1314376352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a8091410 con 0x7fd6a81b9800 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a7fff700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fd69800a4f0 tx=0x7fd69800a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd69800ac80 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6980009b0 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd698004590 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a8091410 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a8091410 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd6980043a0 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 0x7fd68801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.898 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5931+0+0 (secure 0 0 0) 0x7fd698032580 con 0x7fd6a81b9800 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.899 7fd6a77fe700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 0x7fd68801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:46.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.899 7fd6a77fe700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 0x7fd68801abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:46.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.929 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a81b9c00 con 0x7fd6a81b9800 2021-02-10T11:23:46.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:46.932 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd69801c8a0 con 0x7fd6a81b9800 2021-02-10T11:23:47.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.060 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd6a81b9c00 con 0x7fd6a81b9800 2021-02-10T11:23:47.893 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:23:47.896 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:23:47.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.903 7fd6a4ff9700 1 -- 172.21.2.117:0/1314376352 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd69801c8a0 con 0x7fd6a81b9800 2021-02-10T11:23:47.923 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:23:47.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 msgr2=0x7fd68801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:47.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 0x7fd68801abc0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 msgr2=0x7fd6a80ad530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:47.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fd69800a4f0 tx=0x7fd69800a890).stop 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 shutdown_connections 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6a80907d0 0x7fd6a8090bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6a80adb50 0x7fd6a80901b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6880186c0 0x7fd68801abc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 --2- 172.21.2.117:0/1314376352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6a81b9800 0x7fd6a80ad530 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.928 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 >> 172.21.2.117:0/1314376352 conn(0x7fd6a81b61e0 msgr2=0x7fd6a81928e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.929 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 shutdown_connections 2021-02-10T11:23:47.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:47.929 7fd6ae5f2700 1 -- 172.21.2.117:0/1314376352 wait complete. 2021-02-10T11:23:47.934 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:23:47.945 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:23:48.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.346 7f13dce86700 1 Processor -- start 2021-02-10T11:23:48.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.346 7f13dce86700 1 -- start start 2021-02-10T11:23:48.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13dce86700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f13d80d0ac0 0x7f13d81b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13dce86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d81b3470 0x7f13d81b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13dce86700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d8109ab0 con 0x7f13d81085f0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13dce86700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80d0ec0 con 0x7f13d80d0ac0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13dce86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80d10b0 con 0x7f13d81b3470 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13d7314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f13d81085f0 0x7f13d81089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42710/0 (socket says 172.21.2.117:42710) 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13d7314700 1 -- 172.21.2.117:0/1487897736 learned_addr learned my addr 172.21.2.117:0/1487897736 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13d6312700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d81b3470 0x7f13d81b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> v2:172.21.2.125:3300/0 conn(0x7f13d80d0ac0 0x7f13d81b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.347 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294767290 0 0) 0x7f13d0001500 con 0x7f13d81085f0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0002fe0 con 0x7f13d81085f0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2003225548 0 0) 0x7f13d0001270 con 0x7f13d81085f0 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d81b3470 msgr2=0x7f13d81b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d81b3470 0x7f13d81b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 >> v2:172.21.2.125:3300/0 conn(0x7f13d80d0ac0 msgr2=0x7f13d81b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 --2- 172.21.2.117:0/1487897736 >> v2:172.21.2.125:3300/0 conn(0x7f13d80d0ac0 0x7f13d81b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80e51a0 con 0x7f13d81085f0 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f13d0001b50 con 0x7f13d81085f0 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 >> v1:172.21.2.117:6789/0 conn(0x7f13d81085f0 legacy=0x7f13d81089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c0003af0 0x7f13c0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c004a850 0x7f13c004cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c004d650 0x7f13c004fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c0050390 con 0x7f13c0003af0 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00505b0 con 0x7f13c004a850 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c00507d0 con 0x7f13c004d650 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> v2:172.21.2.125:3300/0 conn(0x7f13d80d0ac0 0x7f13d81b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c0003af0 0x7f13c0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d6312700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c004a850 0x7f13c004cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.348 7f13d7314700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c004d650 0x7f13c004fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 -- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c004d650 msgr2=0x7f13c004fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c004d650 0x7f13c004fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 -- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c004a850 msgr2=0x7f13c004cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c004a850 0x7f13c004cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 -- 172.21.2.117:0/1487897736 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c0050a70 con 0x7f13c0003af0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d7314700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c004d650 0x7f13c004fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d6b13700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c0003af0 0x7f13c0005fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f13c8000b80 tx=0x7f13c8009e80).ready entity=mon.0 client_cookie=ccba86092084bc17 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13c800c560 con 0x7f13c0003af0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c800c7a0 con 0x7f13c0003af0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d5b11700 1 -- 172.21.2.117:0/1487897736 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13c800ccc0 con 0x7f13c0003af0 2021-02-10T11:23:48.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13dce86700 1 -- 172.21.2.117:0/1487897736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c0003af0 msgr2=0x7f13c0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13dce86700 1 --2- 172.21.2.117:0/1487897736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c0003af0 0x7f13c0005fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f13c8000b80 tx=0x7f13c8009e80).stop 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.349 7f13d7314700 1 -- 172.21.2.117:0/1487897736 reap_dead start 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.350 7f13dce86700 1 -- 172.21.2.117:0/1487897736 shutdown_connections 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.350 7f13dce86700 1 -- 172.21.2.117:0/1487897736 >> 172.21.2.117:0/1487897736 conn(0x7f13d80ce410 msgr2=0x7f13d80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.350 7f13dce86700 1 -- 172.21.2.117:0/1487897736 shutdown_connections 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.350 7f13dce86700 1 -- 172.21.2.117:0/1487897736 wait complete. 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 Processor -- start 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 -- start start 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d808c400 0x7f13d80f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13d80f1ce0 0x7f13d808a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d808afa0 con 0x7f13d80f20e0 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d808c800 con 0x7f13d80f1ce0 2021-02-10T11:23:48.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.365 7f13dce86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80f25e0 con 0x7f13d808c400 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42472/0 (socket says 172.21.2.117:42472) 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 -- 172.21.2.117:0/4110934906 learned_addr learned my addr 172.21.2.117:0/4110934906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 -- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d808c400 msgr2=0x7f13d80f1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d808c400 0x7f13d80f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 -- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13d80f1ce0 msgr2=0x7f13d808a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13d80f1ce0 0x7f13d808a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 -- 172.21.2.117:0/4110934906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d808b520 con 0x7f13d80f20e0 2021-02-10T11:23:48.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13d7314700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f13d00069d0 tx=0x7f13d000bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13d0007d60 con 0x7f13d80f20e0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0000eb0 con 0x7f13d80f20e0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13dce86700 1 -- 172.21.2.117:0/4110934906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d808b520 con 0x7f13d80f20e0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13dce86700 1 -- 172.21.2.117:0/4110934906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d808bab0 con 0x7f13d80f20e0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.366 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13d0006220 con 0x7f13d80f20e0 2021-02-10T11:23:48.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.367 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13d0004720 con 0x7f13d80f20e0 2021-02-10T11:23:48.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.367 7f13c77fe700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 0x7f13bc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.367 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f13d0032d30 con 0x7f13d80f20e0 2021-02-10T11:23:48.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.368 7f13d6b13700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 0x7f13bc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.368 7f13d6b13700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 0x7f13bc01ac10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:48.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.397 7f13dce86700 1 -- 172.21.2.117:0/4110934906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d8109ab0 con 0x7f13d80f20e0 2021-02-10T11:23:48.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.400 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13d0007b20 con 0x7f13d80f20e0 2021-02-10T11:23:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.527 7f13dce86700 1 -- 172.21.2.117:0/4110934906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13d8109ab0 con 0x7f13d80f20e0 2021-02-10T11:23:48.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.527 7f13c77fe700 1 -- 172.21.2.117:0/4110934906 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+1753 (secure 0 0 0) 0x7f13d0004720 con 0x7f13d80f20e0 2021-02-10T11:23:48.525 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:48.525 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4320,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2595766190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2595766190},{"type":"v1","addr":"172.21.2.117:6833","nonce":2595766190}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":4}],"filesystems":[{"mdsmap":{"epoch":7,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:22:32.545631","modified":"2021-02-10 11:23:47.898057","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2021-02-10T11:23:48.525 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 7 2021-02-10T11:23:48.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 -- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 msgr2=0x7f13bc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 0x7f13bc01ac10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 -- 172.21.2.117:0/4110934906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 msgr2=0x7f13d808a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f13d00069d0 tx=0x7f13d000bc90).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 -- 172.21.2.117:0/4110934906 shutdown_connections 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.529 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13bc018730 0x7f13bc01ac10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13d80f20e0 0x7f13d808a7d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13d80f1ce0 0x7f13d808a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 --2- 172.21.2.117:0/4110934906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13d808c400 0x7f13d80f1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 -- 172.21.2.117:0/4110934906 >> 172.21.2.117:0/4110934906 conn(0x7f13d80bb7a0 msgr2=0x7f13d80c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 -- 172.21.2.117:0/4110934906 shutdown_connections 2021-02-10T11:23:48.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.530 7f13dce86700 1 -- 172.21.2.117:0/4110934906 wait complete. 2021-02-10T11:23:48.535 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:23:48.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.952 7f84e08d4700 1 Processor -- start 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.952 7f84e08d4700 1 -- start start 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84e08d4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84dc079000 0x7f84dc0e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84e08d4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc0d1d00 con 0x7f84dc1b00d0 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84e08d4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc077a70 con 0x7f84dc07da20 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84e08d4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc0e53a0 con 0x7f84dc079000 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84dae1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84dc1b00d0 0x7f84dc0d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42732/0 (socket says 172.21.2.117:42732) 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84d9e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84dc079000 0x7f84dc0e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84dae1d700 1 -- 172.21.2.117:0/1993248726 learned_addr learned my addr 172.21.2.117:0/1993248726 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84d961a700 1 -- 172.21.2.117:0/1993248726 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567166479 0 0) 0x7f84d4000f90 con 0x7f84dc1b00d0 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84d961a700 1 -- 172.21.2.117:0/1993248726 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c8002fe0 con 0x7f84dc1b00d0 2021-02-10T11:23:48.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.953 7f84d9e1b700 1 -- 172.21.2.117:0/1993248726 >> v1:172.21.2.125:6789/0 conn(0x7f84dc07da20 legacy=0x7f84dc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d9e1b700 1 -- 172.21.2.117:0/1993248726 >> v1:172.21.2.117:6789/0 conn(0x7f84dc1b00d0 legacy=0x7f84dc0d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d9e1b700 1 -- 172.21.2.117:0/1993248726 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc079400 con 0x7f84dc079000 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d9e1b700 1 --2- 172.21.2.117:0/1993248726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc079000 0x7f84dc0e4d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f84cc006650 tx=0x7f84cc009a80).ready entity=mon.0 client_cookie=6cbb4a0645062f82 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d961a700 1 -- 172.21.2.117:0/1993248726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84cc00c0c0 con 0x7f84dc079000 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d961a700 1 -- 172.21.2.117:0/1993248726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc00c340 con 0x7f84dc079000 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84d961a700 1 -- 172.21.2.117:0/1993248726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84cc00c860 con 0x7f84dc079000 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84e08d4700 1 -- 172.21.2.117:0/1993248726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc079000 msgr2=0x7f84dc0e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.954 7f84e08d4700 1 --2- 172.21.2.117:0/1993248726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc079000 0x7f84dc0e4d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f84cc006650 tx=0x7f84cc009a80).stop 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.955 7f84e08d4700 1 -- 172.21.2.117:0/1993248726 shutdown_connections 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.955 7f84e08d4700 1 --2- 172.21.2.117:0/1993248726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc079000 0x7f84dc0e4d80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.955 7f84e08d4700 1 -- 172.21.2.117:0/1993248726 >> 172.21.2.117:0/1993248726 conn(0x7f84dc0ce430 msgr2=0x7f84dc0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:48.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.955 7f84e08d4700 1 -- 172.21.2.117:0/1993248726 shutdown_connections 2021-02-10T11:23:48.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.955 7f84e08d4700 1 -- 172.21.2.117:0/1993248726 wait complete. 2021-02-10T11:23:48.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 Processor -- start 2021-02-10T11:23:48.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 -- start start 2021-02-10T11:23:48.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84dc09a050 0x7f84dc0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84dc09a450 0x7f84dc0a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0a3e20 con 0x7f84dc18d370 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0a4000 con 0x7f84dc09a050 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84e08d4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc18dd90 con 0x7f84dc09a450 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84dae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84dae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42486/0 (socket says 172.21.2.117:42486) 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.970 7f84dae1d700 1 -- 172.21.2.117:0/527411232 learned_addr learned my addr 172.21.2.117:0/527411232 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 -- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84dc09a450 msgr2=0x7f84dc0a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84dc09a450 0x7f84dc0a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 -- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84dc09a050 msgr2=0x7f84dc0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84dc09a050 0x7f84dc0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 -- 172.21.2.117:0/527411232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc18df70 con 0x7f84dc18d370 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84dae1d700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f84d4009980 tx=0x7f84d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:48.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d40060c0 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4006300 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84d4004720 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84e08d4700 1 -- 172.21.2.117:0/527411232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc18df70 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.971 7f84e08d4700 1 -- 172.21.2.117:0/527411232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc18e500 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.972 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84d4000e40 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.972 7f84d2ffd700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.972 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6017+0+0 (secure 0 0 0) 0x7f84d4032ae0 con 0x7f84dc18d370 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.972 7f84da61c700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:48.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:48.973 7f84da61c700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 0x7f84c001abc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:49.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.003 7f84e08d4700 1 -- 172.21.2.117:0/527411232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc103370 con 0x7f84dc18d370 2021-02-10T11:23:49.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.005 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84d4006f10 con 0x7f84dc18d370 2021-02-10T11:23:49.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.131 7f84e08d4700 1 -- 172.21.2.117:0/527411232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f84dc077a70 con 0x7f84dc18d370 2021-02-10T11:23:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.909 7f84d2ffd700 1 -- 172.21.2.117:0/527411232 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7f84d4000e40 con 0x7f84dc18d370 2021-02-10T11:23:49.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 msgr2=0x7f84c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:49.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 0x7f84c001abc0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 msgr2=0x7f84dc18d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:49.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f84d4009980 tx=0x7f84d400bcd0).stop 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 shutdown_connections 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84c00186c0 0x7f84c001abc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84dc18d370 0x7f84dc18d770 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84dc09a450 0x7f84dc0a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 --2- 172.21.2.117:0/527411232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84dc09a050 0x7f84dc0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 >> 172.21.2.117:0/527411232 conn(0x7f84dc19f610 msgr2=0x7f84dc09af00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 shutdown_connections 2021-02-10T11:23:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:49.952 7f84e08d4700 1 -- 172.21.2.117:0/527411232 wait complete. 2021-02-10T11:23:49.957 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:23:50.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.373 7fde2cf9f700 1 Processor -- start 2021-02-10T11:23:50.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.373 7fde2cf9f700 1 -- start start 2021-02-10T11:23:50.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde2807da20 0x7fde280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fde280e4e00 0x7fde281b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fde280e5200 0x7fde28077f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde280e5c60 con 0x7fde280e4e00 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde280777d0 con 0x7fde280e5200 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2cf9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde28077a10 con 0x7fde2807da20 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fde280e4e00 0x7fde281b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fde280e4e00 0x7fde281b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42494/0 (socket says 172.21.2.117:42494) 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 -- 172.21.2.117:0/3583781308 learned_addr learned my addr 172.21.2.117:0/3583781308 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 -- 172.21.2.117:0/3583781308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde2807da20 msgr2=0x7fde280771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 --2- 172.21.2.117:0/3583781308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde2807da20 0x7fde280771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 -- 172.21.2.117:0/3583781308 >> v2:172.21.2.125:3300/0 conn(0x7fde280e5200 msgr2=0x7fde28077f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 --2- 172.21.2.117:0/3583781308 >> v2:172.21.2.125:3300/0 conn(0x7fde280e5200 0x7fde28077f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 -- 172.21.2.117:0/3583781308 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2807de20 con 0x7fde280e4e00 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.374 7fde2659c700 1 --2- 172.21.2.117:0/3583781308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280e4e00 0x7fde281b0130 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fde18006700 tx=0x7fde18009b80).ready entity=mon.0 client_cookie=6d2a0cfc12cf321 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde25d9b700 1 -- 172.21.2.117:0/3583781308 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde1800c0c0 con 0x7fde280e4e00 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde25d9b700 1 -- 172.21.2.117:0/3583781308 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1800c400 con 0x7fde280e4e00 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde25d9b700 1 -- 172.21.2.117:0/3583781308 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde1800c920 con 0x7fde280e4e00 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 -- 172.21.2.117:0/3583781308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280e4e00 msgr2=0x7fde281b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 --2- 172.21.2.117:0/3583781308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280e4e00 0x7fde281b0130 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fde18006700 tx=0x7fde18009b80).stop 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 -- 172.21.2.117:0/3583781308 shutdown_connections 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 --2- 172.21.2.117:0/3583781308 >> v2:172.21.2.125:3300/0 conn(0x7fde280e5200 0x7fde28077f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 --2- 172.21.2.117:0/3583781308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280e4e00 0x7fde281b0130 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 --2- 172.21.2.117:0/3583781308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde2807da20 0x7fde280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 -- 172.21.2.117:0/3583781308 >> 172.21.2.117:0/3583781308 conn(0x7fde280ce480 msgr2=0x7fde280e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:50.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.375 7fde2cf9f700 1 -- 172.21.2.117:0/3583781308 shutdown_connections 2021-02-10T11:23:50.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.376 7fde2cf9f700 1 -- 172.21.2.117:0/3583781308 wait complete. 2021-02-10T11:23:50.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 Processor -- start 2021-02-10T11:23:50.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 -- start start 2021-02-10T11:23:50.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde28107340 0x7fde28107740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 0x7fde280b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2819d000 con 0x7fde280b5470 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde280b58a0 con 0x7fde28107340 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2cf9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde280c1880 con 0x7fde280b4a50 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42498/0 (socket says 172.21.2.117:42498) 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.390 7fde2759e700 1 -- 172.21.2.117:0/1280370700 learned_addr learned my addr 172.21.2.117:0/1280370700 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde26d9d700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde28107340 0x7fde28107740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2659c700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 0x7fde280b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 -- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 msgr2=0x7fde280b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 0x7fde280b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 -- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde28107340 msgr2=0x7fde28107740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde28107340 0x7fde28107740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 -- 172.21.2.117:0/1280370700 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde280c1aa0 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2659c700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 0x7fde280b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2759e700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fde20009960 tx=0x7fde2000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde20003030 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde20001310 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde20001830 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde280c1aa0 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.391 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde280c1aa0 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.392 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde200014f0 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.392 7fde1f7fe700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 0x7fde0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.392 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6017+0+0 (secure 0 0 0) 0x7fde20032ab0 con 0x7fde280b5470 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.393 7fde26d9d700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 0x7fde0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.393 7fde26d9d700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 0x7fde0c01abc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:50.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.424 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde281051e0 con 0x7fde280b5470 2021-02-10T11:23:50.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.427 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde2000be30 con 0x7fde280b5470 2021-02-10T11:23:50.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.553 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde28077a10 con 0x7fde280b5470 2021-02-10T11:23:50.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.914 7fde1f7fe700 1 -- 172.21.2.117:0/1280370700 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 185+0+0 (secure 0 0 0) 0x7fde200014f0 con 0x7fde280b5470 2021-02-10T11:23:50.916 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:23:50.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 msgr2=0x7fde0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 0x7fde0c01abc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 msgr2=0x7fde280c1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:50.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fde20009960 tx=0x7fde2000bcf0).stop 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 shutdown_connections 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde0c0186c0 0x7fde0c01abc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde280b5470 0x7fde280c1260 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde280b4a50 0x7fde280b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 --2- 172.21.2.117:0/1280370700 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde28107340 0x7fde28107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 >> 172.21.2.117:0/1280370700 conn(0x7fde280c0480 msgr2=0x7fde280bf360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 shutdown_connections 2021-02-10T11:23:50.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:50.921 7fde2cf9f700 1 -- 172.21.2.117:0/1280370700 wait complete. 2021-02-10T11:23:50.926 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.342 7fe4f0b7b700 1 Processor -- start 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.342 7fe4f0b7b700 1 -- start start 2021-02-10T11:23:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4f0b7b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe4ec07da20 0x7fe4ec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4f0b7b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec077a10 con 0x7fe4ec0e5200 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4f0b7b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ec1b0920 con 0x7fe4ec0e4e00 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4f0b7b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ec0777d0 con 0x7fe4ec07da20 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4eb05f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe4ec0e5200 0x7fe4ec0d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42756/0 (socket says 172.21.2.117:42756) 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4eb05f700 1 -- 172.21.2.117:0/89779348 learned_addr learned my addr 172.21.2.117:0/89779348 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4ea85e700 1 --2- 172.21.2.117:0/89779348 >> v2:172.21.2.125:3300/0 conn(0x7fe4ec07da20 0x7fe4ec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4e985c700 1 -- 172.21.2.117:0/89779348 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488455244 0 0) 0x7fe4e4000f90 con 0x7fe4ec0e5200 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.343 7fe4e985c700 1 -- 172.21.2.117:0/89779348 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4002fe0 con 0x7fe4ec0e5200 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4ea85e700 1 -- 172.21.2.117:0/89779348 >> v1:172.21.2.125:6789/0 conn(0x7fe4ec0e4e00 legacy=0x7fe4ec1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4ea85e700 1 -- 172.21.2.117:0/89779348 >> v1:172.21.2.117:6789/0 conn(0x7fe4ec0e5200 legacy=0x7fe4ec0d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4ea85e700 1 -- 172.21.2.117:0/89779348 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0d1d50 con 0x7fe4ec07da20 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4ea85e700 1 --2- 172.21.2.117:0/89779348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec07da20 0x7fe4ec0771b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe4dc006650 tx=0x7fe4dc009a80).ready entity=mon.1 client_cookie=f14c0d072aaa22ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4e985c700 1 -- 172.21.2.117:0/89779348 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4dc0009b0 con 0x7fe4ec07da20 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4e985c700 1 -- 172.21.2.117:0/89779348 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4dc006740 con 0x7fe4ec07da20 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4e985c700 1 -- 172.21.2.117:0/89779348 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4dc00c0c0 con 0x7fe4ec07da20 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4f0b7b700 1 -- 172.21.2.117:0/89779348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec07da20 msgr2=0x7fe4ec0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.344 7fe4f0b7b700 1 --2- 172.21.2.117:0/89779348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec07da20 0x7fe4ec0771b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe4dc006650 tx=0x7fe4dc009a80).stop 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.345 7fe4f0b7b700 1 -- 172.21.2.117:0/89779348 shutdown_connections 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.345 7fe4f0b7b700 1 --2- 172.21.2.117:0/89779348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec07da20 0x7fe4ec0771b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.345 7fe4f0b7b700 1 -- 172.21.2.117:0/89779348 >> 172.21.2.117:0/89779348 conn(0x7fe4ec0ce480 msgr2=0x7fe4ec0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.345 7fe4f0b7b700 1 -- 172.21.2.117:0/89779348 shutdown_connections 2021-02-10T11:23:51.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.345 7fe4f0b7b700 1 -- 172.21.2.117:0/89779348 wait complete. 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 Processor -- start 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 -- start start 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec0c3bb0 0x7fe4ec0c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4ec107630 0x7fe4ec190490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec190b80 con 0x7fe4ec107a30 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0c4020 con 0x7fe4ec0c3bb0 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4f0b7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0afbc0 con 0x7fe4ec107630 2021-02-10T11:23:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4eb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4eb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42510/0 (socket says 172.21.2.117:42510) 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.360 7fe4eb05f700 1 -- 172.21.2.117:0/3566518019 learned_addr learned my addr 172.21.2.117:0/3566518019 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 -- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4ec107630 msgr2=0x7fe4ec190490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4ec107630 0x7fe4ec190490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 -- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec0c3bb0 msgr2=0x7fe4ec0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec0c3bb0 0x7fe4ec0c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 -- 172.21.2.117:0/3566518019 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec0afde0 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4eb05f700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe4e4009980 tx=0x7fe4e400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e40060c0 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e4006340 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec0afde0 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec1a7260 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.361 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4e4004720 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.362 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4e4006520 con 0x7fe4ec107a30 2021-02-10T11:23:51.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.362 7fe4db7fe700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 0x7fe4d001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:51.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.362 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5579+0+0 (secure 0 0 0) 0x7fe4e4032b20 con 0x7fe4ec107a30 2021-02-10T11:23:51.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.362 7fe4ea85e700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 0x7fe4d001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:51.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.363 7fe4ea85e700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 0x7fe4d001ac10 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:51.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.393 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec1051e0 con 0x7fe4ec107a30 2021-02-10T11:23:51.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.396 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4e4000f40 con 0x7fe4ec107a30 2021-02-10T11:23:51.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.518 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4ec077a10 con 0x7fe4ec107a30 2021-02-10T11:23:51.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.920 7fe4db7fe700 1 -- 172.21.2.117:0/3566518019 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe4e4006f20 con 0x7fe4ec107a30 2021-02-10T11:23:51.933 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:23:51.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.937 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 msgr2=0x7fe4d001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.937 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 0x7fe4d001ac10 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 msgr2=0x7fe4ec0af5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe4e4009980 tx=0x7fe4e400bcd0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 shutdown_connections 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4d0018730 0x7fe4d001ac10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4ec107a30 0x7fe4ec0af5a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4ec107630 0x7fe4ec190490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 --2- 172.21.2.117:0/3566518019 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4ec0c3bb0 0x7fe4ec0c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 >> 172.21.2.117:0/3566518019 conn(0x7fe4ec0c0480 msgr2=0x7fe4ec0bf360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 shutdown_connections 2021-02-10T11:23:51.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:51.938 7fe4f0b7b700 1 -- 172.21.2.117:0/3566518019 wait complete. 2021-02-10T11:23:51.943 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:23:51.943 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:23:51.944 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:23:51.944 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:23:51.944 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:23:51.944 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:23:51.944 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:23:51.944 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:23:51.949 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:23:51.949 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:23:51.950 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:23:51.950 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:23:51.994 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:23:51.994 DEBUG:teuthology.parallel:result is None 2021-02-10T11:23:51.995 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:23:51.995 DEBUG:teuthology.parallel:result is None 2021-02-10T11:23:51.995 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:23:52.398 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:23:52.400 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:23:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.409 7f6b821ee700 1 Processor -- start 2021-02-10T11:23:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.409 7f6b821ee700 1 -- start start 2021-02-10T11:23:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 0x7f6b7c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7c077a70 con 0x7f6b7c0e5180 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c0777d0 con 0x7f6b7c0e4d80 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b7c0d33e0 con 0x7f6b7c07da20 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42530/0 (socket says 172.21.2.117:42530) 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 -- 172.21.2.117:0/1940108681 learned_addr learned my addr 172.21.2.117:0/1940108681 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b8080b700 1 --1- 172.21.2.117:0/1940108681 >> v1:172.21.2.117:6789/0 conn(0x7f6b7c0e5180 0x7f6b7c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42782/0 (socket says 172.21.2.117:42782) 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7bfff700 1 --2- 172.21.2.117:0/1940108681 >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 0x7f6b7c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 -- 172.21.2.117:0/1940108681 >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 msgr2=0x7f6b7c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 --2- 172.21.2.117:0/1940108681 >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 0x7f6b7c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 -- 172.21.2.117:0/1940108681 >> v1:172.21.2.117:6789/0 conn(0x7f6b7c0e5180 legacy=0x7f6b7c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 -- 172.21.2.117:0/1940108681 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c07de20 con 0x7f6b7c0e4d80 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7bfff700 1 --2- 172.21.2.117:0/1940108681 >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 0x7f6b7c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7b7fe700 1 --2- 172.21.2.117:0/1940108681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6b70006700 tx=0x7f6b70009b80).ready entity=mon.0 client_cookie=590a49da00c7563 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7affd700 1 -- 172.21.2.117:0/1940108681 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b7000c0c0 con 0x7f6b7c0e4d80 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7affd700 1 -- 172.21.2.117:0/1940108681 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7000c340 con 0x7f6b7c0e4d80 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b7affd700 1 -- 172.21.2.117:0/1940108681 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b7000c820 con 0x7f6b7c0e4d80 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 -- 172.21.2.117:0/1940108681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0e4d80 msgr2=0x7f6b7c1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.410 7f6b821ee700 1 --2- 172.21.2.117:0/1940108681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6b70006700 tx=0x7f6b70009b80).stop 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 -- 172.21.2.117:0/1940108681 shutdown_connections 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 --2- 172.21.2.117:0/1940108681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0e4d80 0x7f6b7c1b00d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 --2- 172.21.2.117:0/1940108681 >> v2:172.21.2.125:3300/0 conn(0x7f6b7c07da20 0x7f6b7c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 -- 172.21.2.117:0/1940108681 >> 172.21.2.117:0/1940108681 conn(0x7f6b7c0ce430 msgr2=0x7f6b7c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 -- 172.21.2.117:0/1940108681 shutdown_connections 2021-02-10T11:23:52.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.411 7f6b821ee700 1 -- 172.21.2.117:0/1940108681 wait complete. 2021-02-10T11:23:52.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.425 7f6b821ee700 1 Processor -- start 2021-02-10T11:23:52.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 -- start start 2021-02-10T11:23:52.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 0x7f6b7c1b89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b7c1b7f10 0x7f6b7c1b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c102070 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c102290 con 0x7f6b7c1b7f10 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b821ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7c1a75e0 con 0x7f6b7c0fe0b0 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b7b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b7b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42538/0 (socket says 172.21.2.117:42538) 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.426 7f6b7b7fe700 1 -- 172.21.2.117:0/3626514475 learned_addr learned my addr 172.21.2.117:0/3626514475 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b8080b700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b7c1b7f10 0x7f6b7c1b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7bfff700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 0x7f6b7c1b89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 -- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 msgr2=0x7f6b7c1b89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 0x7f6b7c1b89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 -- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b7c1b7f10 msgr2=0x7f6b7c1b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b7c1b7f10 0x7f6b7c1b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 -- 172.21.2.117:0/3626514475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c1a7800 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7bfff700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 0x7f6b7c1b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b7b7fe700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6b70000bd0 tx=0x7f6b7000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b70009100 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b700052e0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b70005740 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7c1a7800 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7c1a7d90 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.427 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b70006030 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.428 7f6b78ff9700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 0x7f6b5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.428 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f6b700322a0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.428 7f6b7bfff700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 0x7f6b5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:52.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.428 7f6b7bfff700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 0x7f6b5c01abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:52.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.459 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b7c1b90f0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.462 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b70021ee0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.598 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f6b7c1b90f0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.926 7f6b78ff9700 1 -- 172.21.2.117:0/3626514475 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v26) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6b70021ee0 con 0x7f6b7c0fe4b0 2021-02-10T11:23:52.962 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:23:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.966 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 msgr2=0x7f6b5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.966 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 0x7f6b5c01abc0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.966 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 msgr2=0x7f6b7c1b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.966 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6b70000bd0 tx=0x7f6b7000b960).stop 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 shutdown_connections 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b5c0186c0 0x7f6b5c01abc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b7c0fe4b0 0x7f6b7c1b78f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b7c1b7f10 0x7f6b7c1b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 --2- 172.21.2.117:0/3626514475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b7c0fe0b0 0x7f6b7c1b89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 >> 172.21.2.117:0/3626514475 conn(0x7f6b7c1a95f0 msgr2=0x7f6b7c094350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:52.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 shutdown_connections 2021-02-10T11:23:52.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:52.967 7f6b821ee700 1 -- 172.21.2.117:0/3626514475 wait complete. 2021-02-10T11:23:52.972 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:23:52.972 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:23:53.132 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:53.135 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.387 7f60571e4700 1 Processor -- start 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.387 7f60571e4700 1 -- start start 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f60571e4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6050079000 0x7f60500e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f60571e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501b00e0 0x7f60501b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f60571e4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6050077a70 con 0x7f605007da20 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f60571e4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60500e53b0 con 0x7f6050079000 2021-02-10T11:23:53.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f60571e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6050079400 con 0x7f60501b00e0 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f6055000700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f605007da20 0x7f60500771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42794/0 (socket says 172.21.2.117:42794) 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f6055000700 1 -- 172.21.2.117:0/1756924405 learned_addr learned my addr 172.21.2.117:0/1756924405 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604ffff700 1 --2- 172.21.2.117:0/1756924405 >> v2:172.21.2.117:3300/0 conn(0x7f6050079000 0x7f60500e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604f7fe700 1 -- 172.21.2.117:0/1756924405 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401476810 0 0) 0x7f60380009b0 con 0x7f605007da20 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604f7fe700 1 -- 172.21.2.117:0/1756924405 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f603c002fe0 con 0x7f605007da20 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604ffff700 1 -- 172.21.2.117:0/1756924405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501b00e0 msgr2=0x7f60501b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604ffff700 1 --2- 172.21.2.117:0/1756924405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501b00e0 0x7f60501b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604ffff700 1 -- 172.21.2.117:0/1756924405 >> v1:172.21.2.117:6789/0 conn(0x7f605007da20 legacy=0x7f60500771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.388 7f604ffff700 1 -- 172.21.2.117:0/1756924405 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605007de20 con 0x7f6050079000 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f604ffff700 1 --2- 172.21.2.117:0/1756924405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6050079000 0x7f60500e4d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6040006650 tx=0x7f6040009a80).ready entity=mon.0 client_cookie=ca036c062547ab70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f604f7fe700 1 -- 172.21.2.117:0/1756924405 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f604000c0c0 con 0x7f6050079000 2021-02-10T11:23:53.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f604f7fe700 1 -- 172.21.2.117:0/1756924405 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604000c340 con 0x7f6050079000 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f604f7fe700 1 -- 172.21.2.117:0/1756924405 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f604000c860 con 0x7f6050079000 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 -- 172.21.2.117:0/1756924405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6050079000 msgr2=0x7f60500e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 --2- 172.21.2.117:0/1756924405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6050079000 0x7f60500e4d90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6040006650 tx=0x7f6040009a80).stop 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 -- 172.21.2.117:0/1756924405 shutdown_connections 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 --2- 172.21.2.117:0/1756924405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501b00e0 0x7f60501b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 --2- 172.21.2.117:0/1756924405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6050079000 0x7f60500e4d90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 -- 172.21.2.117:0/1756924405 >> 172.21.2.117:0/1756924405 conn(0x7f60500ce440 msgr2=0x7f60500e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 -- 172.21.2.117:0/1756924405 shutdown_connections 2021-02-10T11:23:53.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.389 7f60571e4700 1 -- 172.21.2.117:0/1756924405 wait complete. 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.405 7f60571e4700 1 Processor -- start 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.405 7f60571e4700 1 -- start start 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60501af220 0x7f6050198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501995a0 0x7f60501999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6050199020 con 0x7f605018b8c0 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60501aecf0 con 0x7f60501af220 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f60571e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60501aeed0 con 0x7f60501995a0 2021-02-10T11:23:53.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42552/0 (socket says 172.21.2.117:42552) 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 -- 172.21.2.117:0/3000605307 learned_addr learned my addr 172.21.2.117:0/3000605307 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f6055000700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60501af220 0x7f6050198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f6055801700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501995a0 0x7f60501999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 -- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501995a0 msgr2=0x7f60501999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501995a0 0x7f60501999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 -- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60501af220 msgr2=0x7f6050198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60501af220 0x7f6050198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.406 7f604ffff700 1 -- 172.21.2.117:0/3000605307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050199fc0 con 0x7f605018b8c0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604ffff700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6040009630 tx=0x7f604000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60400093a0 con 0x7f605018b8c0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604000bd70 con 0x7f605018b8c0 2021-02-10T11:23:53.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6040005560 con 0x7f605018b8c0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f60571e4700 1 -- 172.21.2.117:0/3000605307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6050199fc0 con 0x7f605018b8c0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f60571e4700 1 -- 172.21.2.117:0/3000605307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6050199fc0 con 0x7f605018b8c0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6040005740 con 0x7f605018b8c0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 0x7f603401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.407 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f60400325f0 con 0x7f605018b8c0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.408 7f6055000700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 0x7f603401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:53.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.408 7f6055000700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 0x7f603401abd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:53.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.437 7f60571e4700 1 -- 172.21.2.117:0/3000605307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60501032e0 con 0x7f605018b8c0 2021-02-10T11:23:53.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.440 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6040005530 con 0x7f605018b8c0 2021-02-10T11:23:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.574 7f60571e4700 1 -- 172.21.2.117:0/3000605307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6050077a70 con 0x7f605018b8c0 2021-02-10T11:23:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.575 7f604d7fa700 1 -- 172.21.2.117:0/3000605307 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+5917 (secure 0 0 0) 0x7f6040005740 con 0x7f605018b8c0 2021-02-10T11:23:53.572 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:53.572 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.573 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.574 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.575 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:23:53.576 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.577 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:23:53.578 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.579 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow r", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow rw", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rwx" 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:23:53.580 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow r", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow rw", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rwx" 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow r", 2021-02-10T11:23:53.581 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow rw", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rwx" 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow r", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow rw", 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rwx" 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.582 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-mds", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNgWOWmHxAAl1YCu+Z3KFA1+rl13SFyTg==", 2021-02-10T11:23:53.583 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-mds" 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-mgr", 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNgme6mHxAAim22L6pbdqnPcID9JHOksw==", 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-mgr" 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-osd", 2021-02-10T11:23:53.584 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNgk/emHxAAh61mWwpHWj50ITkin5r57g==", 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-osd" 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-rbd", 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNgcQCnHxAAhaL3l78M2B8V6e715QfmMA==", 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-rbd" 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.585 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-rbd-mirror", 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNghwmnHxAAOb+L7fHeSIK3MR7nyWPnqQ==", 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.bootstrap-rgw", 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDLwSNgXRKnHxAAlYzzjHovdVvbGmRdnYwMCg==", 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile bootstrap-rgw" 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.586 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:23:53.587 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:23:53.588 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 msgr2=0x7f603401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 0x7f603401abd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 msgr2=0x7f60501ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6040009630 tx=0x7f604000b980).stop 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 shutdown_connections 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60340186d0 0x7f603401abd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f605018b8c0 0x7f60501ae6d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60501995a0 0x7f60501999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 --2- 172.21.2.117:0/3000605307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60501af220 0x7f6050198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:53.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 >> 172.21.2.117:0/3000605307 conn(0x7f605018bfd0 msgr2=0x7f605019aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:53.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 shutdown_connections 2021-02-10T11:23:53.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:53.577 7f60571e4700 1 -- 172.21.2.117:0/3000605307 wait complete. 2021-02-10T11:23:53.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mds 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 Processor -- start 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 -- start start 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1d7007da20 0x7f1d700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700e5180 0x7f1d70079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d70077a70 con 0x7f1d700e4d80 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d700777d0 con 0x7f1d7007da20 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.003 7f1d750df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d701b08c0 con 0x7f1d700e5180 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1d700e4d80 0x7f1d701b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42808/0 (socket says 172.21.2.117:42808) 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6e59c700 1 -- 172.21.2.117:0/2354110860 learned_addr learned my addr 172.21.2.117:0/2354110860 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6ed9d700 1 --2- 172.21.2.117:0/2354110860 >> v2:172.21.2.125:3300/0 conn(0x7f1d7007da20 0x7f1d700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6f59e700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700e5180 0x7f1d70079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196740909 0 0) 0x7f1d600010b0 con 0x7f1d700e4d80 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c002fe0 con 0x7f1d700e4d80 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6ed9d700 1 -- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700e5180 msgr2=0x7f1d70079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6ed9d700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700e5180 0x7f1d70079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2887015985 0 0) 0x7f1d600010b0 con 0x7f1d700e4d80 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6ed9d700 1 -- 172.21.2.117:0/2354110860 >> v1:172.21.2.117:6789/0 conn(0x7f1d700e4d80 legacy=0x7f1d701b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.004 7f1d6ed9d700 1 -- 172.21.2.117:0/2354110860 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7007de20 con 0x7f1d7007da20 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d6ed9d700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7007da20 0x7f1d700771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1d58006650 tx=0x7f1d58009a80).ready entity=mon.1 client_cookie=50694a701f882d74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d5800c0c0 con 0x7f1d7007da20 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5800c340 con 0x7f1d7007da20 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d6dd9b700 1 -- 172.21.2.117:0/2354110860 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d5800c790 con 0x7f1d7007da20 2021-02-10T11:23:54.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d750df700 1 -- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7007da20 msgr2=0x7f1d700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.005 7f1d750df700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7007da20 0x7f1d700771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f1d58006650 tx=0x7f1d58009a80).stop 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 -- 172.21.2.117:0/2354110860 shutdown_connections 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700e5180 0x7f1d70079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 --2- 172.21.2.117:0/2354110860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7007da20 0x7f1d700771b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 -- 172.21.2.117:0/2354110860 >> 172.21.2.117:0/2354110860 conn(0x7f1d700ce430 msgr2=0x7f1d700e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 -- 172.21.2.117:0/2354110860 shutdown_connections 2021-02-10T11:23:54.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.006 7f1d750df700 1 -- 172.21.2.117:0/2354110860 wait complete. 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.020 7f1d750df700 1 Processor -- start 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.020 7f1d750df700 1 -- start start 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700a8860 0x7f1d700a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7018f3e0 0x7f1d700a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d700a99a0 con 0x7f1d7018ff40 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d70190960 con 0x7f1d7018f3e0 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d750df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d70190b80 con 0x7f1d700a8860 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42566/0 (socket says 172.21.2.117:42566) 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 -- 172.21.2.117:0/686109923 learned_addr learned my addr 172.21.2.117:0/686109923 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6e59c700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7018f3e0 0x7f1d700a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6ed9d700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700a8860 0x7f1d700a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 -- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700a8860 msgr2=0x7f1d700a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700a8860 0x7f1d700a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 -- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7018f3e0 msgr2=0x7f1d700a9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7018f3e0 0x7f1d700a9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 -- 172.21.2.117:0/686109923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70190da0 con 0x7f1d7018ff40 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d6f59e700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1d6800b970 tx=0x7f1d6800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.021 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d68000f90 con 0x7f1d7018ff40 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68006aa0 con 0x7f1d7018ff40 2021-02-10T11:23:54.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d680015f0 con 0x7f1d7018ff40 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d750df700 1 -- 172.21.2.117:0/686109923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d70190da0 con 0x7f1d7018ff40 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d750df700 1 -- 172.21.2.117:0/686109923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d70191330 con 0x7f1d7018ff40 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d680011f0 con 0x7f1d7018ff40 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d677fe700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 0x7f1d5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.022 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f1d68032730 con 0x7f1d7018ff40 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.023 7f1d6ed9d700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 0x7f1d5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.023 7f1d6ed9d700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 0x7f1d5401abc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.054 7f1d750df700 1 -- 172.21.2.117:0/686109923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d70103260 con 0x7f1d7018ff40 2021-02-10T11:23:54.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.056 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d68000f40 con 0x7f1d7018ff40 2021-02-10T11:23:54.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.184 7f1d750df700 1 -- 172.21.2.117:0/686109923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1d70077a70 con 0x7f1d7018ff40 2021-02-10T11:23:54.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.188 7f1d677fe700 1 -- 172.21.2.117:0/686109923 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v5) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1d680011f0 con 0x7f1d7018ff40 2021-02-10T11:23:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.193 7f1d750df700 1 -- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 msgr2=0x7f1d5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.193 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 0x7f1d5401abc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.193 7f1d750df700 1 -- 172.21.2.117:0/686109923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 msgr2=0x7f1d70190340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f1d6800b970 tx=0x7f1d6800bd10).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 -- 172.21.2.117:0/686109923 shutdown_connections 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d540186c0 0x7f1d5401abc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d7018ff40 0x7f1d70190340 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d7018f3e0 0x7f1d700a9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 --2- 172.21.2.117:0/686109923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d700a8860 0x7f1d700a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 -- 172.21.2.117:0/686109923 >> 172.21.2.117:0/686109923 conn(0x7f1d70107e80 msgr2=0x7f1d7019e850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 -- 172.21.2.117:0/686109923 shutdown_connections 2021-02-10T11:23:54.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.194 7f1d750df700 1 -- 172.21.2.117:0/686109923 wait complete. 2021-02-10T11:23:54.199 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-mgr 2021-02-10T11:23:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.609 7fb67a00d700 1 Processor -- start 2021-02-10T11:23:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.609 7fb67a00d700 1 -- start start 2021-02-10T11:23:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.609 7fb67a00d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb67407da20 0x7fb6740771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.609 7fb67a00d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6740e5180 0x7fb674079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.609 7fb67a00d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb674077a70 con 0x7fb6740e4d80 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb67a00d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6740777d0 con 0x7fb67407da20 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb67a00d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741b08c0 con 0x7fb6740e5180 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb6737fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb67407da20 0x7fb6740771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb672ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb6740e4d80 0x7fb6741b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42820/0 (socket says 172.21.2.117:42820) 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb672ffd700 1 -- 172.21.2.117:0/2682018045 learned_addr learned my addr 172.21.2.117:0/2682018045 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb673fff700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6740e5180 0x7fb674079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb6737fe700 1 -- 172.21.2.117:0/2682018045 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6740e5180 msgr2=0x7fb674079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb6737fe700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6740e5180 0x7fb674079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb6737fe700 1 -- 172.21.2.117:0/2682018045 >> v1:172.21.2.117:6789/0 conn(0x7fb6740e4d80 legacy=0x7fb6741b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.610 7fb6737fe700 1 -- 172.21.2.117:0/2682018045 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67407de20 con 0x7fb67407da20 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb6737fe700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb67407da20 0x7fb6740771b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb664006650 tx=0x7fb664009a80).ready entity=mon.0 client_cookie=e019d818011076f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb6727fc700 1 -- 172.21.2.117:0/2682018045 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb66400c0c0 con 0x7fb67407da20 2021-02-10T11:23:54.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb6727fc700 1 -- 172.21.2.117:0/2682018045 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66400c340 con 0x7fb67407da20 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb6727fc700 1 -- 172.21.2.117:0/2682018045 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb66400c860 con 0x7fb67407da20 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb67a00d700 1 -- 172.21.2.117:0/2682018045 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb67407da20 msgr2=0x7fb6740771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.611 7fb67a00d700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb67407da20 0x7fb6740771b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb664006650 tx=0x7fb664009a80).stop 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 -- 172.21.2.117:0/2682018045 shutdown_connections 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb6740e5180 0x7fb674079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 --2- 172.21.2.117:0/2682018045 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb67407da20 0x7fb6740771b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 -- 172.21.2.117:0/2682018045 >> 172.21.2.117:0/2682018045 conn(0x7fb6740ce430 msgr2=0x7fb6740e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 -- 172.21.2.117:0/2682018045 shutdown_connections 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.612 7fb67a00d700 1 -- 172.21.2.117:0/2682018045 wait complete. 2021-02-10T11:23:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.626 7fb67a00d700 1 Processor -- start 2021-02-10T11:23:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.626 7fb67a00d700 1 -- start start 2021-02-10T11:23:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb674095360 0x7fb6740935b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb674093bd0 0x7fb6741a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a76c0 con 0x7fb6740aa760 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb674093fd0 con 0x7fb674095360 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb67a00d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6741a81e0 con 0x7fb674093bd0 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42572/0 (socket says 172.21.2.117:42572) 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 -- 172.21.2.117:0/4135914436 learned_addr learned my addr 172.21.2.117:0/4135914436 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 -- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb674093bd0 msgr2=0x7fb6741a7bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb674093bd0 0x7fb6741a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 -- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb674095360 msgr2=0x7fb6740935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.627 7fb6737fe700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb674095360 0x7fb6740935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb6737fe700 1 -- 172.21.2.117:0/4135914436 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6741a8400 con 0x7fb6740aa760 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb6737fe700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb664009630 tx=0x7fb66400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6640093a0 con 0x7fb6740aa760 2021-02-10T11:23:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66400bd70 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6741a8400 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6640054d0 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.628 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6741a8400 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.629 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb6640054f0 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.629 7fb66bfff700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 0x7fb65801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.629 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fb664032440 con 0x7fb6740aa760 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.629 7fb672ffd700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 0x7fb65801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.630 7fb672ffd700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 0x7fb65801abd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:54.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.659 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6741b4ee0 con 0x7fb6740aa760 2021-02-10T11:23:54.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.662 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb6640052e0 con 0x7fb6740aa760 2021-02-10T11:23:54.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.791 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fb674077a70 con 0x7fb6740aa760 2021-02-10T11:23:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.794 7fb66bfff700 1 -- 172.21.2.117:0/4135914436 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v6) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb6640054f0 con 0x7fb6740aa760 2021-02-10T11:23:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 msgr2=0x7fb65801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 0x7fb65801abd0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 msgr2=0x7fb6741b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fb664009630 tx=0x7fb66400b980).stop 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 shutdown_connections 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb674093bd0 0x7fb6741a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb674095360 0x7fb6740935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6580186d0 0x7fb65801abd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 --2- 172.21.2.117:0/4135914436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6740aa760 0x7fb6741b8b00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 >> 172.21.2.117:0/4135914436 conn(0x7fb6741a95f0 msgr2=0x7fb674094350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 shutdown_connections 2021-02-10T11:23:54.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:54.800 7fb67a00d700 1 -- 172.21.2.117:0/4135914436 wait complete. 2021-02-10T11:23:54.805 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-osd 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 Processor -- start 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 -- start start 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f52900d1510 0x7f52900d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5290077a70 con 0x7f529007da20 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5290079400 con 0x7f5290079000 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5296912700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529007de20 con 0x7f52900d1510 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.216 7f5294f2f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f52900d1510 0x7f52900d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f529007da20 0x7f52900771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42830/0 (socket says 172.21.2.117:42830) 2021-02-10T11:23:55.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528ffff700 1 -- 172.21.2.117:0/3000408389 learned_addr learned my addr 172.21.2.117:0/3000408389 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5287fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f5290079000 0x7f52900e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35756/0 (socket says 172.21.2.117:35756) 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5294f2f700 1 -- 172.21.2.117:0/3000408389 >> v1:172.21.2.125:6789/0 conn(0x7f5290079000 legacy=0x7f52900e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5294f2f700 1 -- 172.21.2.117:0/3000408389 >> v1:172.21.2.117:6789/0 conn(0x7f529007da20 legacy=0x7f52900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5294f2f700 1 -- 172.21.2.117:0/3000408389 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52901b00d0 con 0x7f52900d1510 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528f7fe700 1 -- 172.21.2.117:0/3000408389 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604022466 0 0) 0x7f5278000ce0 con 0x7f529007da20 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5294f2f700 1 --2- 172.21.2.117:0/3000408389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52900d1510 0x7f52900d1910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5288006e50 tx=0x7f5288009e70).ready entity=mon.1 client_cookie=bb9c5eabbfa82f5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528f7fe700 1 -- 172.21.2.117:0/3000408389 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5288006a20 con 0x7f52900d1510 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528f7fe700 1 -- 172.21.2.117:0/3000408389 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5288006790 con 0x7f52900d1510 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f528f7fe700 1 -- 172.21.2.117:0/3000408389 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5288000f90 con 0x7f52900d1510 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5296912700 1 -- 172.21.2.117:0/3000408389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52900d1510 msgr2=0x7f52900d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.217 7f5296912700 1 --2- 172.21.2.117:0/3000408389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52900d1510 0x7f52900d1910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5288006e50 tx=0x7f5288009e70).stop 2021-02-10T11:23:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.218 7f5296912700 1 -- 172.21.2.117:0/3000408389 shutdown_connections 2021-02-10T11:23:55.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.218 7f5296912700 1 --2- 172.21.2.117:0/3000408389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52900d1510 0x7f52900d1910 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.218 7f5296912700 1 -- 172.21.2.117:0/3000408389 >> 172.21.2.117:0/3000408389 conn(0x7f52900ce430 msgr2=0x7f52900e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.218 7f5296912700 1 -- 172.21.2.117:0/3000408389 shutdown_connections 2021-02-10T11:23:55.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.218 7f5296912700 1 -- 172.21.2.117:0/3000408389 wait complete. 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.232 7f5296912700 1 Processor -- start 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.232 7f5296912700 1 -- start start 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 0x7f5290099630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52900b4610 0x7f52900b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5290189660 con 0x7f529008e400 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52900901b0 con 0x7f529019f680 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5296912700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529008ee20 con 0x7f52900b4610 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5294f2f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5294f2f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42590/0 (socket says 172.21.2.117:42590) 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f5294f2f700 1 -- 172.21.2.117:0/2524081514 learned_addr learned my addr 172.21.2.117:0/2524081514 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:55.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.233 7f528ffff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 0x7f5290099630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f5287fff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52900b4610 0x7f52900b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 -- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52900b4610 msgr2=0x7f52900b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52900b4610 0x7f52900b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 -- 172.21.2.117:0/2524081514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 msgr2=0x7f529008e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 -- 172.21.2.117:0/2524081514 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290189be0 con 0x7f529019f680 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f5294f2f700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528ffff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 0x7f5290099630 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5278006e90 tx=0x7f527800b860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5278006030 con 0x7f529019f680 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278006330 con 0x7f529019f680 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5278004830 con 0x7f529019f680 2021-02-10T11:23:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f5296912700 1 -- 172.21.2.117:0/2524081514 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5290189be0 con 0x7f529019f680 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.234 7f5296912700 1 -- 172.21.2.117:0/2524081514 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529018a170 con 0x7f529019f680 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.235 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52780043a0 con 0x7f529019f680 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.235 7f528d7fa700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 0x7f527401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.235 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f5278032160 con 0x7f529019f680 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.235 7f5287fff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 0x7f527401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.236 7f5287fff700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 0x7f527401abc0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.266 7f5296912700 1 -- 172.21.2.117:0/2524081514 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52900ff1d0 con 0x7f529019f680 2021-02-10T11:23:55.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.269 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5278030c70 con 0x7f529019f680 2021-02-10T11:23:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.400 7f5296912700 1 -- 172.21.2.117:0/2524081514 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5290077a70 con 0x7f529019f680 2021-02-10T11:23:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.404 7f528d7fa700 1 -- 172.21.2.117:0/2524081514 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f52780043a0 con 0x7f529019f680 2021-02-10T11:23:55.404 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:55.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 msgr2=0x7f527401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 0x7f527401abc0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 msgr2=0x7f5290099630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 0x7f5290099630 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5278006e90 tx=0x7f527800b860).stop 2021-02-10T11:23:55.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 shutdown_connections 2021-02-10T11:23:55.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52740186c0 0x7f527401abc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f529008e400 0x7f529008e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52900b4610 0x7f52900b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 --2- 172.21.2.117:0/2524081514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f529019f680 0x7f5290099630 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 >> 172.21.2.117:0/2524081514 conn(0x7f5290107640 msgr2=0x7f529008a180 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 shutdown_connections 2021-02-10T11:23:55.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.409 7f5296912700 1 -- 172.21.2.117:0/2524081514 wait complete. 2021-02-10T11:23:55.414 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd 2021-02-10T11:23:55.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.825 7fec2edc9700 1 Processor -- start 2021-02-10T11:23:55.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.825 7fec2edc9700 1 -- start start 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fec2807da20 0x7fec280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec280e4d80 0x7fec281b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec280e5180 0x7fec28077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec28077a70 con 0x7fec280e4d80 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec280777d0 con 0x7fec2807da20 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec2edc9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2807de20 con 0x7fec280e5180 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec280e4d80 0x7fec281b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec280e4d80 0x7fec281b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42596/0 (socket says 172.21.2.117:42596) 2021-02-10T11:23:55.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 -- 172.21.2.117:0/1438534273 learned_addr learned my addr 172.21.2.117:0/1438534273 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 -- 172.21.2.117:0/1438534273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec280e5180 msgr2=0x7fec28077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 --2- 172.21.2.117:0/1438534273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec280e5180 0x7fec28077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 -- 172.21.2.117:0/1438534273 >> v2:172.21.2.125:3300/0 conn(0x7fec2807da20 msgr2=0x7fec280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 --2- 172.21.2.117:0/1438534273 >> v2:172.21.2.125:3300/0 conn(0x7fec2807da20 0x7fec280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 -- 172.21.2.117:0/1438534273 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec281b06f0 con 0x7fec280e4d80 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.826 7fec27fff700 1 --2- 172.21.2.117:0/1438534273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec280e4d80 0x7fec281b00d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fec18006a80 tx=0x7fec18009f00).ready entity=mon.0 client_cookie=833e7b87e9485dd5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec277fe700 1 -- 172.21.2.117:0/1438534273 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec1800b5c0 con 0x7fec280e4d80 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec277fe700 1 -- 172.21.2.117:0/1438534273 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1800b8c0 con 0x7fec280e4d80 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec277fe700 1 -- 172.21.2.117:0/1438534273 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec18000d30 con 0x7fec280e4d80 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 -- 172.21.2.117:0/1438534273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec280e4d80 msgr2=0x7fec281b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 --2- 172.21.2.117:0/1438534273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec280e4d80 0x7fec281b00d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fec18006a80 tx=0x7fec18009f00).stop 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 -- 172.21.2.117:0/1438534273 shutdown_connections 2021-02-10T11:23:55.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 --2- 172.21.2.117:0/1438534273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec280e5180 0x7fec28077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 --2- 172.21.2.117:0/1438534273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec280e4d80 0x7fec281b00d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:55.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 --2- 172.21.2.117:0/1438534273 >> v2:172.21.2.125:3300/0 conn(0x7fec2807da20 0x7fec280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 -- 172.21.2.117:0/1438534273 >> 172.21.2.117:0/1438534273 conn(0x7fec280ce430 msgr2=0x7fec280e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:55.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 -- 172.21.2.117:0/1438534273 shutdown_connections 2021-02-10T11:23:55.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.827 7fec2edc9700 1 -- 172.21.2.117:0/1438534273 wait complete. 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 Processor -- start 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 -- start start 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec28199d60 0x7fec2819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec2819e1e0 0x7fec2818ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2819a160 con 0x7fec28189be0 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2819a380 con 0x7fec28199d60 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.842 7fec2edc9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2818b240 con 0x7fec2819e1e0 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42600/0 (socket says 172.21.2.117:42600) 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 -- 172.21.2.117:0/2229764467 learned_addr learned my addr 172.21.2.117:0/2229764467 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 -- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec2819e1e0 msgr2=0x7fec2818ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec2819e1e0 0x7fec2818ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 -- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec28199d60 msgr2=0x7fec2819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec28199d60 0x7fec2819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 -- 172.21.2.117:0/2229764467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2818b460 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec27fff700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fec18009450 tx=0x7fec1800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec1800c2f0 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1800be60 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2818b460 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec1800c870 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.843 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2818b460 con 0x7fec28189be0 2021-02-10T11:23:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.844 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec18001010 con 0x7fec28189be0 2021-02-10T11:23:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.844 7fec257fa700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 0x7fec0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.844 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fec180325b0 con 0x7fec28189be0 2021-02-10T11:23:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.844 7fec2cbe5700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 0x7fec0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.845 7fec2cbe5700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 0x7fec0c01abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:55.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.875 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2818a0d0 con 0x7fec28189be0 2021-02-10T11:23:55.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:55.878 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec18001370 con 0x7fec28189be0 2021-02-10T11:23:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.005 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fec2818a0d0 con 0x7fec28189be0 2021-02-10T11:23:56.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.008 7fec257fa700 1 -- 172.21.2.117:0/2229764467 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7fec18001010 con 0x7fec28189be0 2021-02-10T11:23:56.005 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:56.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.009 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 msgr2=0x7fec0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.009 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 0x7fec0c01abc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.009 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 msgr2=0x7fec2818a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.009 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fec18009450 tx=0x7fec1800bd20).stop 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 shutdown_connections 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec0c0186c0 0x7fec0c01abc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec28189be0 0x7fec2818a600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec2819e1e0 0x7fec2818ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 --2- 172.21.2.117:0/2229764467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec28199d60 0x7fec2819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 >> 172.21.2.117:0/2229764467 conn(0x7fec2819f5a0 msgr2=0x7fec2819a710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:56.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 shutdown_connections 2021-02-10T11:23:56.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.010 7fec2edc9700 1 -- 172.21.2.117:0/2229764467 wait complete. 2021-02-10T11:23:56.015 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2021-02-10T11:23:56.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.429 7f1c50480700 1 Processor -- start 2021-02-10T11:23:56.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.429 7f1c50480700 1 -- start start 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c50480700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 0x7f1c481b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c50480700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 0x7f1c481b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c50480700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c48109ab0 con 0x7f1c481b3470 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c50480700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480e5be0 con 0x7f1c480e51a0 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c50480700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480fde80 con 0x7f1c480d0ac0 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4da9b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1c481b3470 0x7f1c481085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42856/0 (socket says 172.21.2.117:42856) 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4da9b700 1 -- 172.21.2.117:0/2323210899 learned_addr learned my addr 172.21.2.117:0/2323210899 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4e29c700 1 --2- 172.21.2.117:0/2323210899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 0x7f1c481b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4ea9d700 1 --2- 172.21.2.117:0/2323210899 >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 0x7f1c481b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303931579 0 0) 0x7f1c440010b0 con 0x7f1c481b3470 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c34002fe0 con 0x7f1c481b3470 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4e29c700 1 -- 172.21.2.117:0/2323210899 >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 msgr2=0x7f1c481b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4e29c700 1 --2- 172.21.2.117:0/2323210899 >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 0x7f1c481b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4e29c700 1 -- 172.21.2.117:0/2323210899 >> v1:172.21.2.117:6789/0 conn(0x7f1c481b3470 legacy=0x7f1c481085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.430 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 962372479 0 0) 0x7f1c44000e20 con 0x7f1c481b3470 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4e29c700 1 -- 172.21.2.117:0/2323210899 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480d0ec0 con 0x7f1c480d0ac0 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4ea9d700 1 --2- 172.21.2.117:0/2323210899 >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 0x7f1c481b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4e29c700 1 --2- 172.21.2.117:0/2323210899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 0x7f1c481b2e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1c3c006650 tx=0x7f1c3c009a80).ready entity=mon.2 client_cookie=939958c622c35e2d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c3c00c0c0 con 0x7f1c480d0ac0 2021-02-10T11:23:56.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3c00c340 con 0x7f1c480d0ac0 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c4d29a700 1 -- 172.21.2.117:0/2323210899 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c3c00c860 con 0x7f1c480d0ac0 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c50480700 1 -- 172.21.2.117:0/2323210899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 msgr2=0x7f1c481b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.431 7f1c50480700 1 --2- 172.21.2.117:0/2323210899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 0x7f1c481b2e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1c3c006650 tx=0x7f1c3c009a80).stop 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 -- 172.21.2.117:0/2323210899 shutdown_connections 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 --2- 172.21.2.117:0/2323210899 >> v2:172.21.2.125:3300/0 conn(0x7f1c480e51a0 0x7f1c481b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 --2- 172.21.2.117:0/2323210899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480d0ac0 0x7f1c481b2e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 -- 172.21.2.117:0/2323210899 >> 172.21.2.117:0/2323210899 conn(0x7f1c480ce410 msgr2=0x7f1c480d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 -- 172.21.2.117:0/2323210899 shutdown_connections 2021-02-10T11:23:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.432 7f1c50480700 1 -- 172.21.2.117:0/2323210899 wait complete. 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 Processor -- start 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 -- start start 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 0x7f1c480f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480905a0 0x7f1c48091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48090e90 con 0x7f1c480909a0 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480f2560 con 0x7f1c480f1510 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c50480700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480f1910 con 0x7f1c480905a0 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c4ea9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c4ea9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42614/0 (socket says 172.21.2.117:42614) 2021-02-10T11:23:56.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.447 7f1c4ea9d700 1 -- 172.21.2.117:0/1142091870 learned_addr learned my addr 172.21.2.117:0/1142091870 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 -- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480905a0 msgr2=0x7f1c48091410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480905a0 0x7f1c48091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4e29c700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 0x7f1c480f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 -- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 msgr2=0x7f1c480f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 0x7f1c480f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 -- 172.21.2.117:0/1142091870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48092050 con 0x7f1c480909a0 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4e29c700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 0x7f1c480f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c4ea9d700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1c4000b910 tx=0x7f1c4000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:56.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c400011f0 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40006790 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c50480700 1 -- 172.21.2.117:0/1142091870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c48092050 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c50480700 1 -- 172.21.2.117:0/1142091870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c480925e0 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.448 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c400016e0 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.449 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c400011f0 con 0x7f1c480909a0 2021-02-10T11:23:56.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.449 7f1c3affd700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 0x7f1c3001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:56.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.449 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f1c40032960 con 0x7f1c480909a0 2021-02-10T11:23:56.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.449 7f1c4e29c700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 0x7f1c3001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:56.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.450 7f1c4e29c700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 0x7f1c3001ac10 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:56.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.479 7f1c50480700 1 -- 172.21.2.117:0/1142091870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48109ab0 con 0x7f1c480909a0 2021-02-10T11:23:56.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.482 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c400011f0 con 0x7f1c480909a0 2021-02-10T11:23:56.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.611 7f1c50480700 1 -- 172.21.2.117:0/1142091870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f1c48109ab0 con 0x7f1c480909a0 2021-02-10T11:23:56.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.614 7f1c3affd700 1 -- 172.21.2.117:0/1142091870 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1c400066f0 con 0x7f1c480909a0 2021-02-10T11:23:56.615 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:56.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.619 7f1c50480700 1 -- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 msgr2=0x7f1c3001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.619 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 0x7f1c3001ac10 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.619 7f1c50480700 1 -- 172.21.2.117:0/1142091870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 msgr2=0x7f1c48091a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.619 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f1c4000b910 tx=0x7f1c4000bcb0).stop 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 -- 172.21.2.117:0/1142091870 shutdown_connections 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c30018730 0x7f1c3001ac10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1c480909a0 0x7f1c48091a30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1c480905a0 0x7f1c48091410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 --2- 172.21.2.117:0/1142091870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1c480f1510 0x7f1c480f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:56.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 -- 172.21.2.117:0/1142091870 >> 172.21.2.117:0/1142091870 conn(0x7f1c4809e110 msgr2=0x7f1c480b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:56.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 -- 172.21.2.117:0/1142091870 shutdown_connections 2021-02-10T11:23:56.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:56.620 7f1c50480700 1 -- 172.21.2.117:0/1142091870 wait complete. 2021-02-10T11:23:56.625 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth del client.bootstrap-rgw 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.037 7f334c6d0700 1 Processor -- start 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.037 7f334c6d0700 1 -- start start 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334c6d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d0ac0 0x7f33441b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334c6d0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33440e51a0 0x7f33441b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334c6d0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344109ab0 con 0x7f33441b3470 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334c6d0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33440e5be0 con 0x7f33440e51a0 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334c6d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33440fde80 con 0x7f33440d0ac0 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f3349ceb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f33441b3470 0x7f33441085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42872/0 (socket says 172.21.2.117:42872) 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f3349ceb700 1 -- 172.21.2.117:0/4164199753 learned_addr learned my addr 172.21.2.117:0/4164199753 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334aced700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33440e51a0 0x7f33441b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334a4ec700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d0ac0 0x7f33441b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f33494ea700 1 -- 172.21.2.117:0/4164199753 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905570861 0 0) 0x7f33340010b0 con 0x7f33441b3470 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f33494ea700 1 -- 172.21.2.117:0/4164199753 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3330002fe0 con 0x7f33441b3470 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334aced700 1 -- 172.21.2.117:0/4164199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d0ac0 msgr2=0x7f33441b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334aced700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d0ac0 0x7f33441b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334aced700 1 -- 172.21.2.117:0/4164199753 >> v1:172.21.2.117:6789/0 conn(0x7f33441b3470 legacy=0x7f33441085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.038 7f334aced700 1 -- 172.21.2.117:0/4164199753 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440d0ec0 con 0x7f33440e51a0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334aced700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440e51a0 0x7f33441b3870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f3340006ab0 tx=0x7f3340009b40).ready entity=mon.0 client_cookie=3fb6e82083a8c132 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f33494ea700 1 -- 172.21.2.117:0/4164199753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33400022e0 con 0x7f33440e51a0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f33494ea700 1 -- 172.21.2.117:0/4164199753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3340002560 con 0x7f33440e51a0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f33494ea700 1 -- 172.21.2.117:0/4164199753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33400029f0 con 0x7f33440e51a0 2021-02-10T11:23:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 -- 172.21.2.117:0/4164199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440e51a0 msgr2=0x7f33441b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440e51a0 0x7f33441b3870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f3340006ab0 tx=0x7f3340009b40).stop 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 -- 172.21.2.117:0/4164199753 shutdown_connections 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440e51a0 0x7f33441b3870 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 --2- 172.21.2.117:0/4164199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d0ac0 0x7f33441b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.039 7f334c6d0700 1 -- 172.21.2.117:0/4164199753 >> 172.21.2.117:0/4164199753 conn(0x7f33440ce410 msgr2=0x7f33440d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.040 7f334c6d0700 1 -- 172.21.2.117:0/4164199753 shutdown_connections 2021-02-10T11:23:57.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.040 7f334c6d0700 1 -- 172.21.2.117:0/4164199753 wait complete. 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.054 7f334c6d0700 1 Processor -- start 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.054 7f334c6d0700 1 -- start start 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440f1510 0x7f33440f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33440905a0 0x7f3344091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3344090e90 con 0x7f33440909a0 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440f2560 con 0x7f33440905a0 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334c6d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33440f1910 con 0x7f33440f1510 2021-02-10T11:23:57.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42630/0 (socket says 172.21.2.117:42630) 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 -- 172.21.2.117:0/2158651016 learned_addr learned my addr 172.21.2.117:0/2158651016 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f3349ceb700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33440905a0 0x7f3344091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334a4ec700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440f1510 0x7f33440f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 -- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440f1510 msgr2=0x7f33440f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440f1510 0x7f33440f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 -- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33440905a0 msgr2=0x7f3344091410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33440905a0 0x7f3344091410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 -- 172.21.2.117:0/2158651016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344092050 con 0x7f33440909a0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.055 7f334aced700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f334000b5e0 tx=0x7f334000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3340006a70 con 0x7f33440909a0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3340009070 con 0x7f33440909a0 2021-02-10T11:23:57.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3344092050 con 0x7f33440909a0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33440925e0 con 0x7f33440909a0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3340002470 con 0x7f33440909a0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.056 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3340006a20 con 0x7f33440909a0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.057 7f333affd700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 0x7f332c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.057 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f3340032600 con 0x7f33440909a0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.057 7f334a4ec700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 0x7f332c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.057 7f334a4ec700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 0x7f332c01ac10 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.088 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344109ab0 con 0x7f33440909a0 2021-02-10T11:23:57.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.091 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33400093e0 con 0x7f33440909a0 2021-02-10T11:23:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.220 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f3344109ab0 con 0x7f33440909a0 2021-02-10T11:23:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.223 7f333affd700 1 -- 172.21.2.117:0/2158651016 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3340006a20 con 0x7f33440909a0 2021-02-10T11:23:57.224 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:23:57.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 msgr2=0x7f332c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 0x7f332c01ac10 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 msgr2=0x7f3344091a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f334000b5e0 tx=0x7f334000b980).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 shutdown_connections 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f332c018730 0x7f332c01ac10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440909a0 0x7f3344091a30 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.229 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33440905a0 0x7f3344091410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.230 7f334c6d0700 1 --2- 172.21.2.117:0/2158651016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440f1510 0x7f33440f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.230 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 >> 172.21.2.117:0/2158651016 conn(0x7f334409e110 msgr2=0x7f33440b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.230 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 shutdown_connections 2021-02-10T11:23:57.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.230 7f334c6d0700 1 -- 172.21.2.117:0/2158651016 wait complete. 2021-02-10T11:23:57.235 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 Processor -- start 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 -- start start 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f103407da20 0x7f10340771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 0x7f1034079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1034077a70 con 0x7f10340e4d90 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10340e5c10 con 0x7f103407da20 2021-02-10T11:23:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.647 7f103a08e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10340777d0 con 0x7f10340e5190 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f103407da20 0x7f10340771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f1032ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f10340e4d90 0x7f10341b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42884/0 (socket says 172.21.2.117:42884) 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f103407da20 0x7f10340771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42634/0 (socket says 172.21.2.117:42634) 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f1032ffd700 1 -- 172.21.2.117:0/811523186 learned_addr learned my addr 172.21.2.117:0/811523186 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f1033fff700 1 --2- 172.21.2.117:0/811523186 >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 0x7f1034079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10327fc700 1 -- 172.21.2.117:0/811523186 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795517396 0 0) 0x7f10280010b0 con 0x7f10340e4d90 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10327fc700 1 -- 172.21.2.117:0/811523186 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c002fe0 con 0x7f10340e4d90 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 -- 172.21.2.117:0/811523186 >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 msgr2=0x7f1034079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 --2- 172.21.2.117:0/811523186 >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 0x7f1034079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 -- 172.21.2.117:0/811523186 >> v1:172.21.2.117:6789/0 conn(0x7f10340e4d90 legacy=0x7f10341b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 -- 172.21.2.117:0/811523186 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10341b08d0 con 0x7f103407da20 2021-02-10T11:23:57.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f1033fff700 1 --2- 172.21.2.117:0/811523186 >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 0x7f1034079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.648 7f10337fe700 1 --2- 172.21.2.117:0/811523186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103407da20 0x7f10340771b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1024006650 tx=0x7f1024009a80).ready entity=mon.0 client_cookie=830a10a0876d41da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.649 7f10327fc700 1 -- 172.21.2.117:0/811523186 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f102400c0c0 con 0x7f103407da20 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.649 7f10327fc700 1 -- 172.21.2.117:0/811523186 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102400c0c0 con 0x7f103407da20 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.649 7f10327fc700 1 -- 172.21.2.117:0/811523186 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f102400c440 con 0x7f103407da20 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.649 7f103a08e700 1 -- 172.21.2.117:0/811523186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103407da20 msgr2=0x7f10340771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.649 7f103a08e700 1 --2- 172.21.2.117:0/811523186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103407da20 0x7f10340771b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f1024006650 tx=0x7f1024009a80).stop 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 -- 172.21.2.117:0/811523186 shutdown_connections 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 --2- 172.21.2.117:0/811523186 >> v2:172.21.2.125:3300/0 conn(0x7f10340e5190 0x7f1034079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 --2- 172.21.2.117:0/811523186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103407da20 0x7f10340771b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 -- 172.21.2.117:0/811523186 >> 172.21.2.117:0/811523186 conn(0x7f10340ce440 msgr2=0x7f10340e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 -- 172.21.2.117:0/811523186 shutdown_connections 2021-02-10T11:23:57.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.650 7f103a08e700 1 -- 172.21.2.117:0/811523186 wait complete. 2021-02-10T11:23:57.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.664 7f103a08e700 1 Processor -- start 2021-02-10T11:23:57.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.664 7f103a08e700 1 -- start start 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f103419dfe0 0x7f103419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f103408d410 0x7f103408d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103408c080 con 0x7f103408cd50 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103408c2a0 con 0x7f103419dfe0 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f103a08e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103408c4c0 con 0x7f103408d410 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42640/0 (socket says 172.21.2.117:42640) 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 -- 172.21.2.117:0/1965074754 learned_addr learned my addr 172.21.2.117:0/1965074754 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:57.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f1033fff700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f103408d410 0x7f103408d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f1032ffd700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f103419dfe0 0x7f103419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 -- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f103408d410 msgr2=0x7f103408d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f103408d410 0x7f103408d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 -- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f103419dfe0 msgr2=0x7f103419e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f103419dfe0 0x7f103419e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 -- 172.21.2.117:0/1965074754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103408c6e0 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.665 7f10337fe700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1024006f70 tx=0x7f102400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f102400bda0 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024006470 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10240022b0 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f103a08e700 1 -- 172.21.2.117:0/1965074754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103408c6e0 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f103a08e700 1 -- 172.21.2.117:0/1965074754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103408c6e0 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1024000c90 con 0x7f103408cd50 2021-02-10T11:23:57.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 0x7f101401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:57.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.666 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f1024032610 con 0x7f103408cd50 2021-02-10T11:23:57.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.667 7f1032ffd700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 0x7f101401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:57.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.667 7f1032ffd700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 0x7f101401abd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:57.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.697 7f103a08e700 1 -- 172.21.2.117:0/1965074754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1034077a70 con 0x7f103408cd50 2021-02-10T11:23:57.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.700 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f102401f530 con 0x7f103408cd50 2021-02-10T11:23:57.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.823 7f103a08e700 1 -- 172.21.2.117:0/1965074754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1034077a70 con 0x7f103408cd50 2021-02-10T11:23:57.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.824 7f101bfff700 1 -- 172.21.2.117:0/1965074754 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+11377 (secure 0 0 0) 0x7f1024000c90 con 0x7f103408cd50 2021-02-10T11:23:57.821 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:57.822 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":27,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:23:56.409102","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.825 7f103a08e700 1 -- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 msgr2=0x7f101401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.825 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 0x7f101401abd0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.825 7f103a08e700 1 -- 172.21.2.117:0/1965074754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 msgr2=0x7f103419d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.825 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1024006f70 tx=0x7f102400b9b0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 -- 172.21.2.117:0/1965074754 shutdown_connections 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f103408d410 0x7f103408d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f103419dfe0 0x7f103419e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10140186d0 0x7f101401abd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 --2- 172.21.2.117:0/1965074754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f103408cd50 0x7f103419d330 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 -- 172.21.2.117:0/1965074754 >> 172.21.2.117:0/1965074754 conn(0x7f103419f4c0 msgr2=0x7f1034094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 -- 172.21.2.117:0/1965074754 shutdown_connections 2021-02-10T11:23:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:57.826 7f103a08e700 1 -- 172.21.2.117:0/1965074754 wait complete. 2021-02-10T11:23:57.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:23:57.832 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.248 7f2bdbba8700 1 Processor -- start 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.248 7f2bdbba8700 1 -- start start 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bdbba8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2bd407da20 0x7f2bd40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bdbba8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bd40e4da0 0x7f2bd41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bdbba8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bd4077a70 con 0x7f2bd40e51a0 2021-02-10T11:23:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bdbba8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd40777d0 con 0x7f2bd40e4da0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bdbba8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd407de20 con 0x7f2bd407da20 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bd40e4da0 0x7f2bd41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bd40e4da0 0x7f2bd41b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42646/0 (socket says 172.21.2.117:42646) 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 -- 172.21.2.117:0/3421440919 learned_addr learned my addr 172.21.2.117:0/3421440919 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 -- 172.21.2.117:0/3421440919 >> v2:172.21.2.125:3300/0 conn(0x7f2bd407da20 msgr2=0x7f2bd40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 --2- 172.21.2.117:0/3421440919 >> v2:172.21.2.125:3300/0 conn(0x7f2bd407da20 0x7f2bd40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 -- 172.21.2.117:0/3421440919 >> v1:172.21.2.117:6789/0 conn(0x7f2bd40e51a0 legacy=0x7f2bd4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 -- 172.21.2.117:0/3421440919 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd41b0710 con 0x7f2bd40e4da0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.249 7f2bd91c3700 1 --2- 172.21.2.117:0/3421440919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd40e4da0 0x7f2bd41b00f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2bc4006700 tx=0x7f2bc4009b80).ready entity=mon.0 client_cookie=2ca14cbbf0e5136c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bd89c2700 1 -- 172.21.2.117:0/3421440919 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc400c0c0 con 0x7f2bd40e4da0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bd89c2700 1 -- 172.21.2.117:0/3421440919 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc400c440 con 0x7f2bd40e4da0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bd89c2700 1 -- 172.21.2.117:0/3421440919 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc400c8a0 con 0x7f2bd40e4da0 2021-02-10T11:23:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 -- 172.21.2.117:0/3421440919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd40e4da0 msgr2=0x7f2bd41b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 --2- 172.21.2.117:0/3421440919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd40e4da0 0x7f2bd41b00f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2bc4006700 tx=0x7f2bc4009b80).stop 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 -- 172.21.2.117:0/3421440919 shutdown_connections 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 --2- 172.21.2.117:0/3421440919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd40e4da0 0x7f2bd41b00f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 --2- 172.21.2.117:0/3421440919 >> v2:172.21.2.125:3300/0 conn(0x7f2bd407da20 0x7f2bd40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.250 7f2bdbba8700 1 -- 172.21.2.117:0/3421440919 >> 172.21.2.117:0/3421440919 conn(0x7f2bd40ce450 msgr2=0x7f2bd40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.251 7f2bdbba8700 1 -- 172.21.2.117:0/3421440919 shutdown_connections 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.251 7f2bdbba8700 1 -- 172.21.2.117:0/3421440919 wait complete. 2021-02-10T11:23:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.265 7f2bdbba8700 1 Processor -- start 2021-02-10T11:23:58.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.265 7f2bdbba8700 1 -- start start 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bd41a7860 0x7f2bd40bd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bd41a7c60 0x7f2bd40bdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd418c950 con 0x7f2bd418c440 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40c3fb0 con 0x7f2bd41a7c60 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bdbba8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd40be250 con 0x7f2bd41a7860 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42648/0 (socket says 172.21.2.117:42648) 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 -- 172.21.2.117:0/3041292899 learned_addr learned my addr 172.21.2.117:0/3041292899 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 -- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bd41a7860 msgr2=0x7f2bd40bd610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bd41a7860 0x7f2bd40bd610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 -- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bd41a7c60 msgr2=0x7f2bd40bdc30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bd41a7c60 0x7f2bd40bdc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 -- 172.21.2.117:0/3041292899 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd40be470 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.266 7f2bd99c4700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2bd0006f00 tx=0x7f2bd000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bd0006b90 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd000bd30 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd40be470 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bd00020d0 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd40be470 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bd00022a0 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 0x7f2bbc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.267 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2bd0032440 con 0x7f2bd418c440 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.268 7f2bd91c3700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 0x7f2bbc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.268 7f2bd91c3700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 0x7f2bbc01ac10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.298 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd418ea20 con 0x7f2bd418c440 2021-02-10T11:23:58.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.301 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bd0002090 con 0x7f2bd418c440 2021-02-10T11:23:58.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.428 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2bd418ea20 con 0x7f2bd418c440 2021-02-10T11:23:58.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.428 7f2bca7fc700 1 -- 172.21.2.117:0/3041292899 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2bd000bd30 con 0x7f2bd418c440 2021-02-10T11:23:58.426 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:23:58.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 msgr2=0x7f2bbc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 0x7f2bbc01ac10 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 msgr2=0x7f2bd41a7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2bd0006f00 tx=0x7f2bd000b940).stop 2021-02-10T11:23:58.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 shutdown_connections 2021-02-10T11:23:58.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bd41a7c60 0x7f2bd40bdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bd41a7860 0x7f2bd40bd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bbc018730 0x7f2bbc01ac10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 --2- 172.21.2.117:0/3041292899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bd418c440 0x7f2bd41a7240 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.430 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 >> 172.21.2.117:0/3041292899 conn(0x7f2bd40bf160 msgr2=0x7f2bd40b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.431 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 shutdown_connections 2021-02-10T11:23:58.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.431 7f2bdbba8700 1 -- 172.21.2.117:0/3041292899 wait complete. 2021-02-10T11:23:58.436 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 Processor -- start 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 -- start start 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3f6407da20 0x7f3f640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640d1d10 con 0x7f3f640e5190 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f64077a70 con 0x7f3f640e4d90 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f6aaad700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f640777d0 con 0x7f3f6407da20 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f63fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f3f640e4d90 0x7f3f641b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35832/0 (socket says 172.21.2.117:35832) 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f63fff700 1 -- 172.21.2.117:0/1402084041 learned_addr learned my addr 172.21.2.117:0/1402084041 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:58.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f688c9700 1 --2- 172.21.2.117:0/1402084041 >> v2:172.21.2.125:3300/0 conn(0x7f3f6407da20 0x7f3f640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.848 7f3f690ca700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3f640e5190 0x7f3f640d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42906/0 (socket says 172.21.2.117:42906) 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f688c9700 1 -- 172.21.2.117:0/1402084041 >> v1:172.21.2.125:6789/0 conn(0x7f3f640e4d90 legacy=0x7f3f641b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f637fe700 1 -- 172.21.2.117:0/1402084041 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969651576 0 0) 0x7f3f54001430 con 0x7f3f640e4d90 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f688c9700 1 -- 172.21.2.117:0/1402084041 >> v1:172.21.2.117:6789/0 conn(0x7f3f640e5190 legacy=0x7f3f640d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f688c9700 1 -- 172.21.2.117:0/1402084041 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f641b08d0 con 0x7f3f6407da20 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f688c9700 1 --2- 172.21.2.117:0/1402084041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6407da20 0x7f3f640771b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3f4c006650 tx=0x7f3f4c009a80).ready entity=mon.1 client_cookie=b29328e96c7c145c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f637fe700 1 -- 172.21.2.117:0/1402084041 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c0009b0 con 0x7f3f6407da20 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f637fe700 1 -- 172.21.2.117:0/1402084041 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c00c0c0 con 0x7f3f6407da20 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.849 7f3f637fe700 1 -- 172.21.2.117:0/1402084041 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c00c520 con 0x7f3f6407da20 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 -- 172.21.2.117:0/1402084041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6407da20 msgr2=0x7f3f640771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 --2- 172.21.2.117:0/1402084041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6407da20 0x7f3f640771b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3f4c006650 tx=0x7f3f4c009a80).stop 2021-02-10T11:23:58.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 -- 172.21.2.117:0/1402084041 shutdown_connections 2021-02-10T11:23:58.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 --2- 172.21.2.117:0/1402084041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6407da20 0x7f3f640771b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:58.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 -- 172.21.2.117:0/1402084041 >> 172.21.2.117:0/1402084041 conn(0x7f3f640ce440 msgr2=0x7f3f640e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:58.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 -- 172.21.2.117:0/1402084041 shutdown_connections 2021-02-10T11:23:58.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.850 7f3f6aaad700 1 -- 172.21.2.117:0/1402084041 wait complete. 2021-02-10T11:23:58.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.864 7f3f6aaad700 1 Processor -- start 2021-02-10T11:23:58.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.864 7f3f6aaad700 1 -- start start 2021-02-10T11:23:58.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6419e6a0 0x7f3f6419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f641ac620 0x7f3f641aca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641ad110 con 0x7f3f6419e2a0 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f641ad8b0 con 0x7f3f6419e6a0 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f6aaad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419d970 con 0x7f3f641ac620 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42662/0 (socket says 172.21.2.117:42662) 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 -- 172.21.2.117:0/4221619281 learned_addr learned my addr 172.21.2.117:0/4221619281 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 -- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f641ac620 msgr2=0x7f3f641aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f641ac620 0x7f3f641aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 -- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6419e6a0 msgr2=0x7f3f6419f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6419e6a0 0x7f3f6419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 -- 172.21.2.117:0/4221619281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f641ad380 con 0x7f3f6419e2a0 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.865 7f3f688c9700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f3f4c009630 tx=0x7f3f4c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c0093a0 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c00bcf0 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c005560 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f641ad380 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f640b76b0 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f4c00a000 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 0x7f3f4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.866 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f3f4c032390 con 0x7f3f6419e2a0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.867 7f3f63fff700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 0x7f3f4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:58.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.867 7f3f63fff700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 0x7f3f4801abc0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:58.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.897 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f64104a20 con 0x7f3f6419e2a0 2021-02-10T11:23:58.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:58.900 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f4c026f50 con 0x7f3f6419e2a0 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.022 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3f64077a70 con 0x7f3f6419e2a0 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.022 7f3f617fa700 1 -- 172.21.2.117:0/4221619281 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v27) v1 ==== 84+0+64 (secure 0 0 0) 0x7f3f4c026f50 con 0x7f3f6419e2a0 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:23:59.020 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:23:59.021 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:23:59.021 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.024 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 msgr2=0x7f3f4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.024 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 0x7f3f4801abc0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.024 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 msgr2=0x7f3f6419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.024 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f3f4c009630 tx=0x7f3f4c00b900).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 shutdown_connections 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f641ac620 0x7f3f641aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f6419e6a0 0x7f3f6419f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f480186c0 0x7f3f4801abc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 --2- 172.21.2.117:0/4221619281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f6419e2a0 0x7f3f6419ebc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 >> 172.21.2.117:0/4221619281 conn(0x7f3f6409ab30 msgr2=0x7f3f64187be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:59.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 shutdown_connections 2021-02-10T11:23:59.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.025 7f3f6aaad700 1 -- 172.21.2.117:0/4221619281 wait complete. 2021-02-10T11:23:59.030 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:23:59.030 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 Processor -- start 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 -- start start 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb807da20 0x7f2eb80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2eb80e4d80 0x7f2eb81b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8077a70 con 0x7f2eb80e5180 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80e5c00 con 0x7f2eb80e4d80 2021-02-10T11:23:59.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.446 7f2ebf155700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb80777d0 con 0x7f2eb807da20 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2ebd772700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2eb80e5180 0x7f2eb8077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42918/0 (socket says 172.21.2.117:42918) 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2ebd772700 1 -- 172.21.2.117:0/2052348472 learned_addr learned my addr 172.21.2.117:0/2052348472 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049431869 0 0) 0x7f2eb0000f90 con 0x7f2eb80e5180 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4002fe0 con 0x7f2eb80e5180 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2ebcf71700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb807da20 0x7f2eb80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2eb7fff700 1 --2- 172.21.2.117:0/2052348472 >> v2:172.21.2.125:3300/0 conn(0x7f2eb80e4d80 0x7f2eb81b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.447 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4272930966 0 0) 0x7f2eb0001500 con 0x7f2eb80e5180 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb807da20 msgr2=0x7f2eb80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb807da20 0x7f2eb80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 >> v2:172.21.2.125:3300/0 conn(0x7f2eb80e4d80 msgr2=0x7f2eb81b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 --2- 172.21.2.117:0/2052348472 >> v2:172.21.2.125:3300/0 conn(0x7f2eb80e4d80 0x7f2eb81b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb807de20 con 0x7f2eb80e5180 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2eb0001a00 con 0x7f2eb80e5180 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 >> v1:172.21.2.117:6789/0 conn(0x7f2eb80e5180 legacy=0x7f2eb8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ea4003af0 0x7f2ea4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ea4006820 0x7f2ea4008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ea40095b0 0x7f2ea400bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea4050390 con 0x7f2ea4003af0 2021-02-10T11:23:59.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40505b0 con 0x7f2ea40095b0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea40507d0 con 0x7f2ea4006820 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb7fff700 1 --2- 172.21.2.117:0/2052348472 >> v2:172.21.2.125:3300/0 conn(0x7f2eb80e4d80 0x7f2eb81b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ea4003af0 0x7f2ea4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebd772700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ea40095b0 0x7f2ea400bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2eb7fff700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ea4006820 0x7f2ea4008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 -- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ea4006820 msgr2=0x7f2ea4008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ea4006820 0x7f2ea4008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 -- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ea40095b0 msgr2=0x7f2ea400bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ea40095b0 0x7f2ea400bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 -- 172.21.2.117:0/2052348472 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4050a70 con 0x7f2ea4003af0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.448 7f2ebcf71700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ea4003af0 0x7f2ea4005fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2ea0009310 tx=0x7f2ea0009a10).ready entity=mon.0 client_cookie=b97894d15105047f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea00009b0 con 0x7f2ea4003af0 2021-02-10T11:23:59.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0006400 con 0x7f2ea4003af0 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2eb77fe700 1 -- 172.21.2.117:0/2052348472 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ea000c2b0 con 0x7f2ea4003af0 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 -- 172.21.2.117:0/2052348472 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ea4003af0 msgr2=0x7f2ea4005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 --2- 172.21.2.117:0/2052348472 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ea4003af0 0x7f2ea4005fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2ea0009310 tx=0x7f2ea0009a10).stop 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebd772700 1 -- 172.21.2.117:0/2052348472 reap_dead start 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 -- 172.21.2.117:0/2052348472 shutdown_connections 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 -- 172.21.2.117:0/2052348472 >> 172.21.2.117:0/2052348472 conn(0x7f2eb80ce430 msgr2=0x7f2eb80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 -- 172.21.2.117:0/2052348472 shutdown_connections 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.449 7f2ebf155700 1 -- 172.21.2.117:0/2052348472 wait complete. 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 Processor -- start 2021-02-10T11:23:59.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 -- start start 2021-02-10T11:23:59.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2eb80b8160 0x7f2eb80b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb80b8630 0x7f2eb80b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81a3290 con 0x7f2eb80b9050 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81a34b0 con 0x7f2eb80b8160 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.464 7f2ebf155700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81a36d0 con 0x7f2eb80b8630 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42676/0 (socket says 172.21.2.117:42676) 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 -- 172.21.2.117:0/1943807878 learned_addr learned my addr 172.21.2.117:0/1943807878 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 -- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb80b8630 msgr2=0x7f2eb80b8a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb80b8630 0x7f2eb80b8a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 -- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2eb80b8160 msgr2=0x7f2eb80b78d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2eb80b8160 0x7f2eb80b78d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 -- 172.21.2.117:0/1943807878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb81a38f0 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebd772700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f2eb0001bf0 tx=0x7f2eb000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eb0007aa0 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0007160 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2eb0007530 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb81a38f0 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.465 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb81a38f0 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.466 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2eb0007340 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.466 7f2eb57fa700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 0x7f2e9c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.466 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2eb00327c0 con 0x7f2eb80b9050 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.466 7f2ebcf71700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 0x7f2e9c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:23:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.467 7f2ebcf71700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 0x7f2e9c01abd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:23:59.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.496 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8077a70 con 0x7f2eb80b9050 2021-02-10T11:23:59.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.499 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2eb0001270 con 0x7f2eb80b9050 2021-02-10T11:23:59.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:23:59.624 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2eb8077a70 con 0x7f2eb80b9050 2021-02-10T11:24:00.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.020 7f2eb57fa700 1 -- 172.21.2.117:0/1943807878 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v28) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2eb0007160 con 0x7f2eb80b9050 2021-02-10T11:24:00.037 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 msgr2=0x7f2e9c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 0x7f2e9c01abd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 msgr2=0x7f2eb81a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f2eb0001bf0 tx=0x7f2eb000ac80).stop 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 shutdown_connections 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e9c0186d0 0x7f2e9c01abd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2eb80b9050 0x7f2eb81a2bf0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2eb80b8630 0x7f2eb80b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 --2- 172.21.2.117:0/1943807878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2eb80b8160 0x7f2eb80b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:00.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 >> 172.21.2.117:0/1943807878 conn(0x7f2eb80a90a0 msgr2=0x7f2eb80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:00.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 shutdown_connections 2021-02-10T11:24:00.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.042 7f2ebf155700 1 -- 172.21.2.117:0/1943807878 wait complete. 2021-02-10T11:24:00.047 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.462 7f8f7fae2700 1 Processor -- start 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.462 7f8f7fae2700 1 -- start start 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7fae2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8f78079000 0x7f8f780e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7fae2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780d1d20 con 0x7f8f781b00f0 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7fae2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f78077a70 con 0x7f8f7807da20 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7fae2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f780e53c0 con 0x7f8f78079000 2021-02-10T11:24:00.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7e0ff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8f781b00f0 0x7f8f780d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42938/0 (socket says 172.21.2.117:42938) 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7e0ff700 1 -- 172.21.2.117:0/3029166241 learned_addr learned my addr 172.21.2.117:0/3029166241 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955812668 0 0) 0x7f8f70001500 con 0x7f8f781b00f0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.463 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f64002fe0 con 0x7f8f781b00f0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 --2- 172.21.2.117:0/3029166241 >> v2:172.21.2.125:3300/0 conn(0x7f8f78079000 0x7f8f780e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 775015411 0 0) 0x7f8f70001500 con 0x7f8f781b00f0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 >> v2:172.21.2.125:3300/0 conn(0x7f8f78079000 msgr2=0x7f8f780e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 --2- 172.21.2.117:0/3029166241 >> v2:172.21.2.125:3300/0 conn(0x7f8f78079000 0x7f8f780e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 >> v1:172.21.2.125:6789/0 conn(0x7f8f7807da20 legacy=0x7f8f780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78079400 con 0x7f8f781b00f0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8f70001a00 con 0x7f8f781b00f0 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 >> v1:172.21.2.117:6789/0 conn(0x7f8f781b00f0 legacy=0x7f8f780d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f64003b90 0x7f8f640060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 0x7f8f64008e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f64009670 0x7f8f6400bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6400c3e0 con 0x7f8f64006950 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6400c600 con 0x7f8f64003b90 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6400c820 con 0x7f8f64009670 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 0x7f8f64008e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7e0ff700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f64009670 0x7f8f6400bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d8fe700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f64003b90 0x7f8f640060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 -- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f64009670 msgr2=0x7f8f6400bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f64009670 0x7f8f6400bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 -- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f64003b90 msgr2=0x7f8f640060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f64003b90 0x7f8f640060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7d0fd700 1 -- 172.21.2.117:0/3029166241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640509e0 con 0x7f8f64006950 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.464 7f8f7e0ff700 1 -- 172.21.2.117:0/3029166241 reap_dead start 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7d0fd700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 0x7f8f64008e00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f8f6800c8f0 tx=0x7f8f6800cc90).ready entity=mon.0 client_cookie=e59541583d98a911 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f68009310 con 0x7f8f64006950 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f68009610 con 0x7f8f64006950 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7c8fc700 1 -- 172.21.2.117:0/3029166241 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f68009af0 con 0x7f8f64006950 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7fae2700 1 -- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 msgr2=0x7f8f64008e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7fae2700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 0x7f8f64008e00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f8f6800c8f0 tx=0x7f8f6800cc90).stop 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7fae2700 1 -- 172.21.2.117:0/3029166241 shutdown_connections 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7fae2700 1 --2- 172.21.2.117:0/3029166241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f64006950 0x7f8f64008e00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.465 7f8f7fae2700 1 -- 172.21.2.117:0/3029166241 >> 172.21.2.117:0/3029166241 conn(0x7f8f780ce450 msgr2=0x7f8f780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.466 7f8f7fae2700 1 -- 172.21.2.117:0/3029166241 shutdown_connections 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.466 7f8f7fae2700 1 -- 172.21.2.117:0/3029166241 wait complete. 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.480 7f8f7fae2700 1 Processor -- start 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.480 7f8f7fae2700 1 -- start start 2021-02-10T11:24:00.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f780a46c0 0x7f8f780a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f780a50e0 0x7f8f780a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780c3040 con 0x7f8f781a46b0 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78190a20 con 0x7f8f780a50e0 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7fae2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f780a54e0 con 0x7f8f780a46c0 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42696/0 (socket says 172.21.2.117:42696) 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 -- 172.21.2.117:0/3104958150 learned_addr learned my addr 172.21.2.117:0/3104958150 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d0fd700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f780a46c0 0x7f8f780a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7e0ff700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f780a50e0 0x7f8f780a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 -- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f780a46c0 msgr2=0x7f8f780a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f780a46c0 0x7f8f780a4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 -- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f780a50e0 msgr2=0x7f8f780a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f780a50e0 0x7f8f780a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 -- 172.21.2.117:0/3104958150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f780a5700 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.481 7f8f7d8fe700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8f74007fc0 tx=0x7f8f7400a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f74006470 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7400ace0 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f7400c350 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f780a5700 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f780a5700 con 0x7f8f781a46b0 2021-02-10T11:24:00.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f74008030 con 0x7f8f781a46b0 2021-02-10T11:24:00.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 0x7f8f6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:00.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.482 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f8f74032730 con 0x7f8f781a46b0 2021-02-10T11:24:00.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.483 7f8f7d0fd700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 0x7f8f6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:00.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.483 7f8f7d0fd700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 0x7f8f6001abc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:00.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.513 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78077a70 con 0x7f8f781a46b0 2021-02-10T11:24:00.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.515 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f74000c10 con 0x7f8f781a46b0 2021-02-10T11:24:00.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:00.643 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f8f78077a70 con 0x7f8f781a46b0 2021-02-10T11:24:01.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.023 7f8f6e7fc700 1 -- 172.21.2.117:0/3104958150 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v29) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8f74000c10 con 0x7f8f781a46b0 2021-02-10T11:24:01.055 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:24:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 msgr2=0x7f8f6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 0x7f8f6001abc0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 msgr2=0x7f8f78190400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8f74007fc0 tx=0x7f8f7400a8f0).stop 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 shutdown_connections 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f780a50e0 0x7f8f780a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f780a46c0 0x7f8f780a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f600186c0 0x7f8f6001abc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 --2- 172.21.2.117:0/3104958150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f781a46b0 0x7f8f78190400 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 >> 172.21.2.117:0/3104958150 conn(0x7f8f780b5140 msgr2=0x7f8f780a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 shutdown_connections 2021-02-10T11:24:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.060 7f8f7fae2700 1 -- 172.21.2.117:0/3104958150 wait complete. 2021-02-10T11:24:01.065 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:24:01.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.480 7f2c0daca700 1 Processor -- start 2021-02-10T11:24:01.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.480 7f2c0daca700 1 -- start start 2021-02-10T11:24:01.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c0daca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c0807da20 0x7f2c080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c0daca700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2c080e4e00 0x7f2c081b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c0daca700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c08077a10 con 0x7f2c080e5200 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c0daca700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c080777d0 con 0x7f2c0807da20 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c0daca700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0807de20 con 0x7f2c080e4e00 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c0807da20 0x7f2c080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2c0807da20 0x7f2c080771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42704/0 (socket says 172.21.2.117:42704) 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 -- 172.21.2.117:0/220366873 learned_addr learned my addr 172.21.2.117:0/220366873 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 -- 172.21.2.117:0/220366873 >> v2:172.21.2.125:3300/0 conn(0x7f2c080e4e00 msgr2=0x7f2c081b0130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:01.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 --2- 172.21.2.117:0/220366873 >> v2:172.21.2.125:3300/0 conn(0x7f2c080e4e00 0x7f2c081b0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 -- 172.21.2.117:0/220366873 >> v1:172.21.2.125:6789/0 conn(0x7f2c080e5200 legacy=0x7f2c08077f90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 -- 172.21.2.117:0/220366873 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c081b0750 con 0x7f2c0807da20 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.481 7f2c077fe700 1 --2- 172.21.2.117:0/220366873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c0807da20 0x7f2c080771b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2bf8006700 tx=0x7f2bf8009b80).ready entity=mon.0 client_cookie=66ae782675ec2033 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c067fc700 1 -- 172.21.2.117:0/220366873 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bf800c0c0 con 0x7f2c0807da20 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c067fc700 1 -- 172.21.2.117:0/220366873 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf800c440 con 0x7f2c0807da20 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c067fc700 1 -- 172.21.2.117:0/220366873 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bf800c8a0 con 0x7f2c0807da20 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 -- 172.21.2.117:0/220366873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c0807da20 msgr2=0x7f2c080771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 --2- 172.21.2.117:0/220366873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c0807da20 0x7f2c080771b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2bf8006700 tx=0x7f2bf8009b80).stop 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 -- 172.21.2.117:0/220366873 shutdown_connections 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 --2- 172.21.2.117:0/220366873 >> v2:172.21.2.125:3300/0 conn(0x7f2c080e4e00 0x7f2c081b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 --2- 172.21.2.117:0/220366873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c0807da20 0x7f2c080771b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 -- 172.21.2.117:0/220366873 >> 172.21.2.117:0/220366873 conn(0x7f2c080ce480 msgr2=0x7f2c080e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 -- 172.21.2.117:0/220366873 shutdown_connections 2021-02-10T11:24:01.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.482 7f2c0daca700 1 -- 172.21.2.117:0/220366873 wait complete. 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 Processor -- start 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 -- start start 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c08106c50 0x7f2c080b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c080c0e50 0x7f2c081a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08107050 con 0x7f2c080c1250 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c081a37d0 con 0x7f2c08106c50 2021-02-10T11:24:01.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.497 7f2c0daca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c081a39f0 con 0x7f2c080c0e50 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42710/0 (socket says 172.21.2.117:42710) 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 -- 172.21.2.117:0/1112877032 learned_addr learned my addr 172.21.2.117:0/1112877032 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 -- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c080c0e50 msgr2=0x7f2c081a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c080c0e50 0x7f2c081a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 -- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c08106c50 msgr2=0x7f2c080b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c08106c50 0x7f2c080b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 -- 172.21.2.117:0/1112877032 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c080c1c40 con 0x7f2c080c1250 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c07fff700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2c00009940 tx=0x7f2c0000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c00003030 con 0x7f2c080c1250 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c00001270 con 0x7f2c080c1250 2021-02-10T11:24:01.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c000016d0 con 0x7f2c080c1250 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c080c1c40 con 0x7f2c080c1250 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.498 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c080c1c40 con 0x7f2c080c1250 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.499 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c0000a000 con 0x7f2c080c1250 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.499 7f2bfffff700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 0x7f2bec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.499 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f2c00032a90 con 0x7f2c080c1250 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.499 7f2c077fe700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 0x7f2bec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.500 7f2c077fe700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 0x7f2bec01abc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:01.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.531 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c08107050 con 0x7f2c080c1250 2021-02-10T11:24:01.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.533 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c0000be30 con 0x7f2c080c1250 2021-02-10T11:24:01.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.660 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f2c08107050 con 0x7f2c080c1250 2021-02-10T11:24:01.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.664 7f2bfffff700 1 -- 172.21.2.117:0/1112877032 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 4 and data pool 5 v10) v1 ==== 187+0+0 (secure 0 0 0) 0x7f2c0000a000 con 0x7f2c080c1250 2021-02-10T11:24:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 4 and data pool 5 2021-02-10T11:24:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 msgr2=0x7f2bec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 0x7f2bec01abc0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 msgr2=0x7f2c081a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2c00009940 tx=0x7f2c0000bcf0).stop 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 shutdown_connections 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bec0186c0 0x7f2bec01abc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c080c1250 0x7f2c081a31b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c080c0e50 0x7f2c081a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 --2- 172.21.2.117:0/1112877032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c08106c50 0x7f2c080b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 >> 172.21.2.117:0/1112877032 conn(0x7f2c080c0520 msgr2=0x7f2c080ad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 shutdown_connections 2021-02-10T11:24:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:01.671 7f2c0daca700 1 -- 172.21.2.117:0/1112877032 wait complete. 2021-02-10T11:24:01.676 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:02.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.088 7f6e17b36700 1 Processor -- start 2021-02-10T11:24:02.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.088 7f6e17b36700 1 -- start start 2021-02-10T11:24:02.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e17b36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100d0ac0 0x7f6e101b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e17b36700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e100e51a0 0x7f6e101b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e17b36700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e10109ab0 con 0x7f6e101b3470 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e17b36700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100e5be0 con 0x7f6e100e51a0 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e17b36700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100fde80 con 0x7f6e100d0ac0 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e15151700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6e101b3470 0x7f6e101085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42966/0 (socket says 172.21.2.117:42966) 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e15151700 1 -- 172.21.2.117:0/3890646077 learned_addr learned my addr 172.21.2.117:0/3890646077 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620110604 0 0) 0x7f6e040010b0 con 0x7f6e101b3470 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e14950700 1 -- 172.21.2.117:0/3890646077 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dfc002fe0 con 0x7f6e101b3470 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e16153700 1 --2- 172.21.2.117:0/3890646077 >> v2:172.21.2.125:3300/0 conn(0x7f6e100e51a0 0x7f6e101b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e15952700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100d0ac0 0x7f6e101b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.089 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2795695706 0 0) 0x7f6e040010b0 con 0x7f6e101b3470 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100d0ac0 msgr2=0x7f6e101b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100d0ac0 0x7f6e101b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 >> v2:172.21.2.125:3300/0 conn(0x7f6e100e51a0 msgr2=0x7f6e101b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 --2- 172.21.2.117:0/3890646077 >> v2:172.21.2.125:3300/0 conn(0x7f6e100e51a0 0x7f6e101b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100d0ec0 con 0x7f6e101b3470 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6e040015b0 con 0x7f6e101b3470 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 >> v1:172.21.2.117:6789/0 conn(0x7f6e101b3470 legacy=0x7f6e101085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dfc003ae0 0x7f6dfc005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dfc006830 0x7f6dfc008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dfc0095a0 0x7f6dfc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc050380 con 0x7f6dfc0095a0 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0505a0 con 0x7f6dfc003ae0 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e14950700 1 -- 172.21.2.117:0/3890646077 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0507c0 con 0x7f6dfc006830 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e16153700 1 --2- 172.21.2.117:0/3890646077 >> v2:172.21.2.125:3300/0 conn(0x7f6e100e51a0 0x7f6e101b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dfc006830 0x7f6dfc008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15952700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dfc003ae0 0x7f6dfc005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e16153700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dfc0095a0 0x7f6dfc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 -- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dfc003ae0 msgr2=0x7f6dfc005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dfc003ae0 0x7f6dfc005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 -- 172.21.2.117:0/3890646077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dfc0095a0 msgr2=0x7f6dfc00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dfc0095a0 0x7f6dfc00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 -- 172.21.2.117:0/3890646077 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc050a60 con 0x7f6dfc006830 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.090 7f6e15151700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dfc006830 0x7f6dfc008d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f6e040017d0 tx=0x7f6e04007d30).ready entity=mon.2 client_cookie=49291a4dae58948b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e040013a0 con 0x7f6dfc006830 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0400a520 con 0x7f6dfc006830 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e14950700 1 -- 172.21.2.117:0/3890646077 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0400a970 con 0x7f6dfc006830 2021-02-10T11:24:02.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e17b36700 1 -- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dfc006830 msgr2=0x7f6dfc008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e17b36700 1 --2- 172.21.2.117:0/3890646077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dfc006830 0x7f6dfc008d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f6e040017d0 tx=0x7f6e04007d30).stop 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e16153700 1 -- 172.21.2.117:0/3890646077 reap_dead start 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e17b36700 1 -- 172.21.2.117:0/3890646077 shutdown_connections 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.091 7f6e17b36700 1 -- 172.21.2.117:0/3890646077 >> 172.21.2.117:0/3890646077 conn(0x7f6e100ce410 msgr2=0x7f6e100d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.092 7f6e17b36700 1 -- 172.21.2.117:0/3890646077 shutdown_connections 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.092 7f6e17b36700 1 -- 172.21.2.117:0/3890646077 wait complete. 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.106 7f6e17b36700 1 Processor -- start 2021-02-10T11:24:02.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 -- start start 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 0x7f6e101a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 0x7f6e101a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e101932a0 con 0x7f6e101a30e0 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10192e20 con 0x7f6e100b7940 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e17b36700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10193040 con 0x7f6e100b8360 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e16153700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 0x7f6e101a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e15952700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e16153700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 0x7f6e101a34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42728/0 (socket says 172.21.2.117:42728) 2021-02-10T11:24:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e15952700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:47110/0 (socket says 172.21.2.117:47110) 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e15151700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 0x7f6e101a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.107 7f6e16153700 1 -- 172.21.2.117:0/1102914275 learned_addr learned my addr 172.21.2.117:0/1102914275 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 -- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 msgr2=0x7f6e100b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 -- 172.21.2.117:0/1102914275 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 msgr2=0x7f6e101a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 0x7f6e101a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 -- 172.21.2.117:0/1102914275 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100b8eb0 con 0x7f6e100b8360 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15952700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e15151700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 0x7f6e101a2ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6e0400a4a0 tx=0x7f6e0400a840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0400ac30 con 0x7f6e100b8360 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e04004140 con 0x7f6e100b8360 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e04004660 con 0x7f6e100b8360 2021-02-10T11:24:02.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.108 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e100b8eb0 con 0x7f6e100b8360 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.109 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e100b8eb0 con 0x7f6e100b8360 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.109 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e04000940 con 0x7f6e100b8360 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.109 7f6e027fc700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 0x7f6df801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.109 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e040325a0 con 0x7f6e100b8360 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.110 7f6e15952700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 0x7f6df801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.110 7f6e15952700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 0x7f6df801abc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.139 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100b8980 con 0x7f6e100b8360 2021-02-10T11:24:02.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.144 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e04007970 con 0x7f6e100b8360 2021-02-10T11:24:02.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.271 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e100b8980 con 0x7f6e100b8360 2021-02-10T11:24:02.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.273 7f6e027fc700 1 -- 172.21.2.117:0/1102914275 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f6e04000940 con 0x7f6e100b8360 2021-02-10T11:24:02.271 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:02.271 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":30,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:01.662624","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 msgr2=0x7f6df801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 0x7f6df801abc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 msgr2=0x7f6e101a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 0x7f6e101a2ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6e0400a4a0 tx=0x7f6e0400a840).stop 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 shutdown_connections 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186c0 0x7f6df801abc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e101a30e0 0x7f6e101a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e100b8360 0x7f6e101a2ac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 --2- 172.21.2.117:0/1102914275 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e100b7940 0x7f6e100b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 >> 172.21.2.117:0/1102914275 conn(0x7f6e100bb680 msgr2=0x7f6e100c38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 shutdown_connections 2021-02-10T11:24:02.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.276 7f6e17b36700 1 -- 172.21.2.117:0/1102914275 wait complete. 2021-02-10T11:24:02.282 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.695 7f5047988700 1 Processor -- start 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.695 7f5047988700 1 -- start start 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5047988700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f504007da20 0x7f50400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5047988700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 0x7f5040079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5047988700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5040077a70 con 0x7f50400e4d90 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5047988700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50400e5c10 con 0x7f504007da20 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5047988700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50400777d0 con 0x7f50400e5190 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5045fa5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 0x7f5040079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5044fa3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f50400e4d90 0x7f50401b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42984/0 (socket says 172.21.2.117:42984) 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5044fa3700 1 -- 172.21.2.117:0/3319328898 learned_addr learned my addr 172.21.2.117:0/3319328898 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:02.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 --2- 172.21.2.117:0/3319328898 >> v2:172.21.2.117:3300/0 conn(0x7f504007da20 0x7f50400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5033fff700 1 -- 172.21.2.117:0/3319328898 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931735548 0 0) 0x7f50340009b0 con 0x7f50400e4d90 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5033fff700 1 -- 172.21.2.117:0/3319328898 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c002fe0 con 0x7f50400e4d90 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 -- 172.21.2.117:0/3319328898 >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 msgr2=0x7f5040079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 --2- 172.21.2.117:0/3319328898 >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 0x7f5040079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 -- 172.21.2.117:0/3319328898 >> v1:172.21.2.117:6789/0 conn(0x7f50400e4d90 legacy=0x7f50401b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 -- 172.21.2.117:0/3319328898 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50401b08d0 con 0x7f504007da20 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f5045fa5700 1 --2- 172.21.2.117:0/3319328898 >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 0x7f5040079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.696 7f50457a4700 1 --2- 172.21.2.117:0/3319328898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504007da20 0x7f50400771b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f503c006650 tx=0x7f503c009a80).ready entity=mon.0 client_cookie=c02895d4506a20e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5033fff700 1 -- 172.21.2.117:0/3319328898 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c0009b0 con 0x7f504007da20 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5033fff700 1 -- 172.21.2.117:0/3319328898 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c00c0c0 con 0x7f504007da20 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5033fff700 1 -- 172.21.2.117:0/3319328898 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c00c5e0 con 0x7f504007da20 2021-02-10T11:24:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 -- 172.21.2.117:0/3319328898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504007da20 msgr2=0x7f50400771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 --2- 172.21.2.117:0/3319328898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504007da20 0x7f50400771b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f503c006650 tx=0x7f503c009a80).stop 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 -- 172.21.2.117:0/3319328898 shutdown_connections 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 --2- 172.21.2.117:0/3319328898 >> v2:172.21.2.125:3300/0 conn(0x7f50400e5190 0x7f5040079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 --2- 172.21.2.117:0/3319328898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504007da20 0x7f50400771b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.697 7f5047988700 1 -- 172.21.2.117:0/3319328898 >> 172.21.2.117:0/3319328898 conn(0x7f50400ce440 msgr2=0x7f50400e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.698 7f5047988700 1 -- 172.21.2.117:0/3319328898 shutdown_connections 2021-02-10T11:24:02.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.698 7f5047988700 1 -- 172.21.2.117:0/3319328898 wait complete. 2021-02-10T11:24:02.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.712 7f5047988700 1 Processor -- start 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.712 7f5047988700 1 -- start start 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f504008cd50 0x7f504019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 0x7f504008d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504008c080 con 0x7f504019dfe0 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504008c2a0 con 0x7f504008cd50 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5047988700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504008c4c0 con 0x7f504008d410 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5044fa3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5044fa3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42740/0 (socket says 172.21.2.117:42740) 2021-02-10T11:24:02.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5044fa3700 1 -- 172.21.2.117:0/2384769331 learned_addr learned my addr 172.21.2.117:0/2384769331 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.713 7f5045fa5700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 0x7f504008d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 -- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 msgr2=0x7f504008d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 0x7f504008d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 -- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f504008cd50 msgr2=0x7f504019d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f504008cd50 0x7f504019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 -- 172.21.2.117:0/2384769331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504008c6e0 con 0x7f504019dfe0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5045fa5700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 0x7f504008d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5044fa3700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5034001c60 tx=0x7f503400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503400acc0 con 0x7f504019dfe0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034004350 con 0x7f504019dfe0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5047988700 1 -- 172.21.2.117:0/2384769331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504008c6e0 con 0x7f504019dfe0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5047988700 1 -- 172.21.2.117:0/2384769331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504008c6e0 con 0x7f504019dfe0 2021-02-10T11:24:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.714 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5034004870 con 0x7f504019dfe0 2021-02-10T11:24:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.715 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5034004530 con 0x7f504019dfe0 2021-02-10T11:24:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.715 7f5031ffb700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 0x7f502001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.715 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f50340326b0 con 0x7f504019dfe0 2021-02-10T11:24:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.715 7f50457a4700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 0x7f502001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.716 7f50457a4700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 0x7f502001abc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:02.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.745 7f5047988700 1 -- 172.21.2.117:0/2384769331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5040077a70 con 0x7f504019dfe0 2021-02-10T11:24:02.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.748 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5034004140 con 0x7f504019dfe0 2021-02-10T11:24:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.872 7f5047988700 1 -- 172.21.2.117:0/2384769331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5040077a70 con 0x7f504019dfe0 2021-02-10T11:24:02.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.872 7f5031ffb700 1 -- 172.21.2.117:0/2384769331 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f503401eab0 con 0x7f504019dfe0 2021-02-10T11:24:02.869 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:02.870 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":31,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:02.664299","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:02.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 msgr2=0x7f502001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 0x7f502001abc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 msgr2=0x7f504019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:02.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5034001c60 tx=0x7f503400a8d0).stop 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 shutdown_connections 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f50200186c0 0x7f502001abc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f504019dfe0 0x7f504019e3e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f504008d410 0x7f504008d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 --2- 172.21.2.117:0/2384769331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f504008cd50 0x7f504019d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 >> 172.21.2.117:0/2384769331 conn(0x7f504019f4c0 msgr2=0x7f5040094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 shutdown_connections 2021-02-10T11:24:02.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:02.874 7f5047988700 1 -- 172.21.2.117:0/2384769331 wait complete. 2021-02-10T11:24:02.880 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:24:03.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.295 7f22c2208700 1 Processor -- start 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.295 7f22c2208700 1 -- start start 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c2208700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 0x7f22bc0e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c2208700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 0x7f22bc1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c2208700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc077a70 con 0x7f22bc07da20 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c2208700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc0e53a0 con 0x7f22bc1b00d0 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c2208700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc079400 con 0x7f22bc079000 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f22bc07da20 0x7f22bc0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:42996/0 (socket says 172.21.2.117:42996) 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bbfff700 1 -- 172.21.2.117:0/2518869251 learned_addr learned my addr 172.21.2.117:0/2518869251 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bb7fe700 1 --2- 172.21.2.117:0/2518869251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 0x7f22bc0e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c0825700 1 --2- 172.21.2.117:0/2518869251 >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 0x7f22bc1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22baffd700 1 -- 172.21.2.117:0/2518869251 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986070627 0 0) 0x7f22ac000ce0 con 0x7f22bc07da20 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22baffd700 1 -- 172.21.2.117:0/2518869251 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4002fe0 con 0x7f22bc07da20 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bb7fe700 1 -- 172.21.2.117:0/2518869251 >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 msgr2=0x7f22bc1b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bb7fe700 1 --2- 172.21.2.117:0/2518869251 >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 0x7f22bc1b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bb7fe700 1 -- 172.21.2.117:0/2518869251 >> v1:172.21.2.117:6789/0 conn(0x7f22bc07da20 legacy=0x7f22bc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22bb7fe700 1 -- 172.21.2.117:0/2518869251 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc07de20 con 0x7f22bc079000 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.296 7f22c0825700 1 --2- 172.21.2.117:0/2518869251 >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 0x7f22bc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22bb7fe700 1 --2- 172.21.2.117:0/2518869251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 0x7f22bc0e4d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f22b0006650 tx=0x7f22b0009a80).ready entity=mon.2 client_cookie=166124f92e00b53b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22baffd700 1 -- 172.21.2.117:0/2518869251 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22b000c0c0 con 0x7f22bc079000 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22baffd700 1 -- 172.21.2.117:0/2518869251 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b000c340 con 0x7f22bc079000 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22c2208700 1 -- 172.21.2.117:0/2518869251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 msgr2=0x7f22bc0e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22c2208700 1 --2- 172.21.2.117:0/2518869251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 0x7f22bc0e4d80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f22b0006650 tx=0x7f22b0009a80).stop 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.297 7f22baffd700 1 -- 172.21.2.117:0/2518869251 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22b000c860 con 0x7f22bc079000 2021-02-10T11:24:03.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 -- 172.21.2.117:0/2518869251 shutdown_connections 2021-02-10T11:24:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 --2- 172.21.2.117:0/2518869251 >> v2:172.21.2.125:3300/0 conn(0x7f22bc1b00d0 0x7f22bc1b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 --2- 172.21.2.117:0/2518869251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc079000 0x7f22bc0e4d80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 -- 172.21.2.117:0/2518869251 >> 172.21.2.117:0/2518869251 conn(0x7f22bc0ce430 msgr2=0x7f22bc0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 -- 172.21.2.117:0/2518869251 shutdown_connections 2021-02-10T11:24:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.298 7f22c2208700 1 -- 172.21.2.117:0/2518869251 wait complete. 2021-02-10T11:24:03.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.312 7f22c2208700 1 Processor -- start 2021-02-10T11:24:03.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.312 7f22c2208700 1 -- start start 2021-02-10T11:24:03.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc0b8b60 0x7f22bc0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22bc095bc0 0x7f22bc095fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc091b90 con 0x7f22bc0b8f60 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0b2d00 con 0x7f22bc095bc0 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22c2208700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0916d0 con 0x7f22bc0b8b60 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42752/0 (socket says 172.21.2.117:42752) 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 -- 172.21.2.117:0/3607299584 learned_addr learned my addr 172.21.2.117:0/3607299584 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 -- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc0b8b60 msgr2=0x7f22bc0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc0b8b60 0x7f22bc0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 -- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22bc095bc0 msgr2=0x7f22bc095fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22bc095bc0 0x7f22bc095fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 -- 172.21.2.117:0/3607299584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0b8200 con 0x7f22bc0b8f60 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.313 7f22bb7fe700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f22b0009650 tx=0x7f22b000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22b0006030 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b000bcd0 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22b00054d0 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22c2208700 1 -- 172.21.2.117:0/3607299584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc0b8200 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22c2208700 1 -- 172.21.2.117:0/3607299584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc0968d0 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22b00052e0 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 0x7f229c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.314 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f22b0032660 con 0x7f22bc0b8f60 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.315 7f22bbfff700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 0x7f229c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.316 7f22bbfff700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 0x7f229c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:03.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.345 7f22c2208700 1 -- 172.21.2.117:0/3607299584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc0b2d00 con 0x7f22bc0b8f60 2021-02-10T11:24:03.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.348 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f22b0009300 con 0x7f22bc0b8f60 2021-02-10T11:24:03.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:03.482 7f22c2208700 1 -- 172.21.2.117:0/3607299584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f22bc0b2d00 con 0x7f22bc0b8f60 2021-02-10T11:24:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.039 7f22b8ff9700 1 -- 172.21.2.117:0/3607299584 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7f22b00052e0 con 0x7f22bc0b8f60 2021-02-10T11:24:04.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 -- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 msgr2=0x7f229c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 0x7f229c01abc0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 -- 172.21.2.117:0/3607299584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 msgr2=0x7f22bc0b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f22b0009650 tx=0x7f22b000b8e0).stop 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 -- 172.21.2.117:0/3607299584 shutdown_connections 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f229c0186c0 0x7f229c01abc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22bc0b8f60 0x7f22bc0b26e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22bc095bc0 0x7f22bc095fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 --2- 172.21.2.117:0/3607299584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22bc0b8b60 0x7f22bc0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.052 7f22c2208700 1 -- 172.21.2.117:0/3607299584 >> 172.21.2.117:0/3607299584 conn(0x7f22bc0a4170 msgr2=0x7f22bc09be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.053 7f22c2208700 1 -- 172.21.2.117:0/3607299584 shutdown_connections 2021-02-10T11:24:04.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.053 7f22c2208700 1 -- 172.21.2.117:0/3607299584 wait complete. 2021-02-10T11:24:04.057 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.469 7f09cb682700 1 Processor -- start 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.469 7f09cb682700 1 -- start start 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09cb682700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c40d1520 0x7f09c40d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09cb682700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c4077a70 con 0x7f09c4079000 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09cb682700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c4079400 con 0x7f09c407da20 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09cb682700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c407de20 con 0x7f09c40d1520 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09c8c9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09c4079000 0x7f09c40e4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43010/0 (socket says 172.21.2.117:43010) 2021-02-10T11:24:04.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.470 7f09c8c9d700 1 -- 172.21.2.117:0/4164758837 learned_addr learned my addr 172.21.2.117:0/4164758837 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439063930 0 0) 0x7f09b80010b0 con 0x7f09c4079000 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b0002fe0 con 0x7f09c4079000 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c9c9f700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c40d1520 0x7f09c40d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1811205024 0 0) 0x7f09b80010b0 con 0x7f09c4079000 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c40d1520 msgr2=0x7f09c40d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c40d1520 0x7f09c40d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 >> v1:172.21.2.125:6789/0 conn(0x7f09c407da20 legacy=0x7f09c40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c41b00e0 con 0x7f09c4079000 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f09b80015b0 con 0x7f09c4079000 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 >> v1:172.21.2.117:6789/0 conn(0x7f09c4079000 legacy=0x7f09c40e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09b0003b10 0x7f09b0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09b0006820 0x7f09b0008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09b00095c0 0x7f09b000ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b000c2e0 con 0x7f09b0003b10 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b000c500 con 0x7f09b0006820 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b000c720 con 0x7f09b00095c0 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09b0003b10 0x7f09b0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c8c9d700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09b0006820 0x7f09b0008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 -- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09b00095c0 msgr2=0x7f09b000ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09b00095c0 0x7f09b000ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 -- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09b0006820 msgr2=0x7f09b0008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09b0006820 0x7f09b0008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.471 7f09c949e700 1 -- 172.21.2.117:0/4164758837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b000c9c0 con 0x7f09b0003b10 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09c949e700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09b0003b10 0x7f09b0006010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f09c0006ae0 tx=0x7f09c004db00).ready entity=mon.0 client_cookie=9e54aa7f97bb36f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c004b260 con 0x7f09b0003b10 2021-02-10T11:24:04.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c004b460 con 0x7f09b0003b10 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09b7fff700 1 -- 172.21.2.117:0/4164758837 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09c004b870 con 0x7f09b0003b10 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09cb682700 1 -- 172.21.2.117:0/4164758837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09b0003b10 msgr2=0x7f09b0006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09cb682700 1 --2- 172.21.2.117:0/4164758837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09b0003b10 0x7f09b0006010 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f09c0006ae0 tx=0x7f09c004db00).stop 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09c9c9f700 1 -- 172.21.2.117:0/4164758837 reap_dead start 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09cb682700 1 -- 172.21.2.117:0/4164758837 shutdown_connections 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.472 7f09cb682700 1 -- 172.21.2.117:0/4164758837 >> 172.21.2.117:0/4164758837 conn(0x7f09c40ce440 msgr2=0x7f09c40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.473 7f09cb682700 1 -- 172.21.2.117:0/4164758837 shutdown_connections 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.473 7f09cb682700 1 -- 172.21.2.117:0/4164758837 wait complete. 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.487 7f09cb682700 1 Processor -- start 2021-02-10T11:24:04.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.487 7f09cb682700 1 -- start start 2021-02-10T11:24:04.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c40b33b0 0x7f09c40b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c418d6d0 0x7f09c418dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c41ad6a0 con 0x7f09c418ccb0 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c41ad8c0 con 0x7f09c40b33b0 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09cb682700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c41adae0 con 0x7f09c418d6d0 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42776/0 (socket says 172.21.2.117:42776) 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 -- 172.21.2.117:0/1254857006 learned_addr learned my addr 172.21.2.117:0/1254857006 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 -- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c418d6d0 msgr2=0x7f09c418dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c418d6d0 0x7f09c418dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 -- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c40b33b0 msgr2=0x7f09c40b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c40b33b0 0x7f09c40b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 -- 172.21.2.117:0/1254857006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c41add00 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.488 7f09c8c9d700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f09b8001c00 tx=0x7f09b800a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b80009b0 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800acd0 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b8004540 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09cb682700 1 -- 172.21.2.117:0/1254857006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c41add00 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09cb682700 1 -- 172.21.2.117:0/1254857006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c41ae290 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.489 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09b8004140 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.490 7f09b5ffb700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.490 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f09b8032680 con 0x7f09c418ccb0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.490 7f09c949e700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.490 7f09c949e700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:04.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.520 7f09cb682700 1 -- 172.21.2.117:0/1254857006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c4077a70 con 0x7f09c418ccb0 2021-02-10T11:24:04.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.522 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09b8000940 con 0x7f09c418ccb0 2021-02-10T11:24:04.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.649 7f09cb682700 1 -- 172.21.2.117:0/1254857006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09c4077a70 con 0x7f09c418ccb0 2021-02-10T11:24:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.650 7f09b5ffb700 1 -- 172.21.2.117:0/1254857006 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2097 (secure 0 0 0) 0x7f09b801f270 con 0x7f09c418ccb0 2021-02-10T11:24:04.647 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:04.647 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:24:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 12 2021-02-10T11:24:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 msgr2=0x7f09a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 msgr2=0x7f09c418d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f09b8001c00 tx=0x7f09b800a8e0).stop 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 shutdown_connections 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c418ccb0 0x7f09c418d0b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c418d6d0 0x7f09c418dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 --2- 172.21.2.117:0/1254857006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c40b33b0 0x7f09c40b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:04.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 >> 172.21.2.117:0/1254857006 conn(0x7f09c419f510 msgr2=0x7f09c41abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:04.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 shutdown_connections 2021-02-10T11:24:04.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:04.652 7f09cb682700 1 -- 172.21.2.117:0/1254857006 wait complete. 2021-02-10T11:24:04.657 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:05.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 Processor -- start 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 -- start start 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b6807da20 0x7f6b680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b680e4d90 0x7f6b681b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b68077a70 con 0x7f6b680e5190 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b680777d0 con 0x7f6b6807da20 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.071 7f6b6d2dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6807de20 con 0x7f6b680e4d90 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b6807da20 0x7f6b680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6787e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6b680e5190 0x7f6b68077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:35956/0 (socket says 172.21.2.117:35956) 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b6807da20 0x7f6b680771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42782/0 (socket says 172.21.2.117:42782) 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6787e700 1 -- 172.21.2.117:0/1623537313 learned_addr learned my addr 172.21.2.117:0/1623537313 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6687c700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b680e4d90 0x7f6b681b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6607b700 1 -- 172.21.2.117:0/1623537313 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950836243 0 0) 0x7f6b60001440 con 0x7f6b680e5190 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6607b700 1 -- 172.21.2.117:0/1623537313 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b54002fe0 con 0x7f6b680e5190 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 -- 172.21.2.117:0/1623537313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b680e4d90 msgr2=0x7f6b681b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b680e4d90 0x7f6b681b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 -- 172.21.2.117:0/1623537313 >> v1:172.21.2.125:6789/0 conn(0x7f6b680e5190 legacy=0x7f6b68077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.072 7f6b6707d700 1 -- 172.21.2.117:0/1623537313 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b681b0700 con 0x7f6b6807da20 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6707d700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6807da20 0x7f6b680771b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f6b58006650 tx=0x7f6b58009a80).ready entity=mon.0 client_cookie=5ff538bb38ebc1ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6607b700 1 -- 172.21.2.117:0/1623537313 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b580009b0 con 0x7f6b6807da20 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6607b700 1 -- 172.21.2.117:0/1623537313 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5800c0c0 con 0x7f6b6807da20 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6607b700 1 -- 172.21.2.117:0/1623537313 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5800c520 con 0x7f6b6807da20 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 -- 172.21.2.117:0/1623537313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6807da20 msgr2=0x7f6b680771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6807da20 0x7f6b680771b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f6b58006650 tx=0x7f6b58009a80).stop 2021-02-10T11:24:05.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 -- 172.21.2.117:0/1623537313 shutdown_connections 2021-02-10T11:24:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b680e4d90 0x7f6b681b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 --2- 172.21.2.117:0/1623537313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6807da20 0x7f6b680771b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 -- 172.21.2.117:0/1623537313 >> 172.21.2.117:0/1623537313 conn(0x7f6b680ce440 msgr2=0x7f6b680e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 -- 172.21.2.117:0/1623537313 shutdown_connections 2021-02-10T11:24:05.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.073 7f6b6d2dc700 1 -- 172.21.2.117:0/1623537313 wait complete. 2021-02-10T11:24:05.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 Processor -- start 2021-02-10T11:24:05.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 -- start start 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b681aea50 0x7f6b681917e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b681aee50 0x7f6b68191e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b681af250 con 0x7f6b681898a0 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68192590 con 0x7f6b681aea50 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.088 7f6b6d2dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68108030 con 0x7f6b681aee50 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42786/0 (socket says 172.21.2.117:42786) 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 -- 172.21.2.117:0/2616080784 learned_addr learned my addr 172.21.2.117:0/2616080784 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 -- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b681aee50 msgr2=0x7f6b68191e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b681aee50 0x7f6b68191e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 -- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b681aea50 msgr2=0x7f6b681917e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b681aea50 0x7f6b681917e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 -- 172.21.2.117:0/2616080784 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6818a2c0 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6787e700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6b60001b90 tx=0x7f6b6000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b600070d0 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60007450 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b60004b30 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.089 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6818a2c0 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.090 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6818a2c0 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.090 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b600046a0 con 0x7f6b681898a0 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.090 7f6b5ffff700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.090 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6b600328d0 con 0x7f6b681898a0 2021-02-10T11:24:05.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.091 7f6b6707d700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.091 7f6b6707d700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.121 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b681af250 con 0x7f6b681898a0 2021-02-10T11:24:05.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.124 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b600011f0 con 0x7f6b681898a0 2021-02-10T11:24:05.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.249 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b681af250 con 0x7f6b681898a0 2021-02-10T11:24:05.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.249 7f6b5ffff700 1 -- 172.21.2.117:0/2616080784 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f6b600046a0 con 0x7f6b681898a0 2021-02-10T11:24:05.248 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:05.248 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":31,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:02.664299","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:05.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 msgr2=0x7f6b4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 msgr2=0x7f6b68189ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6b60001b90 tx=0x7f6b6000ac50).stop 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 shutdown_connections 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b4c0186c0 0x7f6b4c01abc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b681898a0 0x7f6b68189ca0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b681aee50 0x7f6b68191e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 --2- 172.21.2.117:0/2616080784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b681aea50 0x7f6b681917e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.251 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 >> 172.21.2.117:0/2616080784 conn(0x7f6b68096860 msgr2=0x7f6b68188890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.252 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 shutdown_connections 2021-02-10T11:24:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.252 7f6b6d2dc700 1 -- 172.21.2.117:0/2616080784 wait complete. 2021-02-10T11:24:05.258 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 Processor -- start 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 -- start start 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc01807da20 0x7fc0180771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc0180e51a0 0x7fc018079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc018077a70 con 0x7fc0180e4da0 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180e5c20 con 0x7fc0180e51a0 2021-02-10T11:24:05.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.686 7fc01f252700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180777d0 con 0x7fc01807da20 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc0180e51a0 0x7fc018079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01c86d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc0180e4da0 0x7fc0181b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43044/0 (socket says 172.21.2.117:43044) 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01c86d700 1 -- 172.21.2.117:0/4121447587 learned_addr learned my addr 172.21.2.117:0/4121447587 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 -- 172.21.2.117:0/4121447587 >> v2:172.21.2.125:3300/0 conn(0x7fc01807da20 msgr2=0x7fc0180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 --2- 172.21.2.117:0/4121447587 >> v2:172.21.2.125:3300/0 conn(0x7fc01807da20 0x7fc0180771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 -- 172.21.2.117:0/4121447587 >> v1:172.21.2.117:6789/0 conn(0x7fc0180e4da0 legacy=0x7fc0181b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 -- 172.21.2.117:0/4121447587 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0181b08e0 con 0x7fc0180e51a0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01d86f700 1 --2- 172.21.2.117:0/4121447587 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0180e51a0 0x7fc018079000 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc010006e30 tx=0x7fc010009e60).ready entity=mon.0 client_cookie=73bb155f3005896a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc00bfff700 1 -- 172.21.2.117:0/4121447587 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc01000b300 con 0x7fc0180e51a0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc00bfff700 1 -- 172.21.2.117:0/4121447587 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010000d30 con 0x7fc0180e51a0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc00bfff700 1 -- 172.21.2.117:0/4121447587 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc01000b770 con 0x7fc0180e51a0 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01f252700 1 -- 172.21.2.117:0/4121447587 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0180e51a0 msgr2=0x7fc018079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.687 7fc01f252700 1 --2- 172.21.2.117:0/4121447587 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0180e51a0 0x7fc018079000 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc010006e30 tx=0x7fc010009e60).stop 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 -- 172.21.2.117:0/4121447587 shutdown_connections 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 --2- 172.21.2.117:0/4121447587 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0180e51a0 0x7fc018079000 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 --2- 172.21.2.117:0/4121447587 >> v2:172.21.2.125:3300/0 conn(0x7fc01807da20 0x7fc0180771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 -- 172.21.2.117:0/4121447587 >> 172.21.2.117:0/4121447587 conn(0x7fc0180ce450 msgr2=0x7fc0180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 -- 172.21.2.117:0/4121447587 shutdown_connections 2021-02-10T11:24:05.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.688 7fc01f252700 1 -- 172.21.2.117:0/4121447587 wait complete. 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 Processor -- start 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 -- start start 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc01809a7c0 0x7fc0181ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc01809abc0 0x7fc0181ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc018096900 con 0x7fc018096370 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181ace30 con 0x7fc01809abc0 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01f252700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181ad010 con 0x7fc01809a7c0 2021-02-10T11:24:05.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.703 7fc01d06e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42800/0 (socket says 172.21.2.117:42800) 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 -- 172.21.2.117:0/1279556623 learned_addr learned my addr 172.21.2.117:0/1279556623 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01c86d700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc01809a7c0 0x7fc0181ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d86f700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc01809abc0 0x7fc0181ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 -- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc01809a7c0 msgr2=0x7fc0181ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc01809a7c0 0x7fc0181ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 -- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc01809abc0 msgr2=0x7fc0181ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc01809abc0 0x7fc0181ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 -- 172.21.2.117:0/1279556623 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018098cf0 con 0x7fc018096370 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01d06e700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc01400b530 tx=0x7fc01400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc01400bcc0 con 0x7fc018096370 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014000c90 con 0x7fc018096370 2021-02-10T11:24:05.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0140022a0 con 0x7fc018096370 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.704 7fc01f252700 1 -- 172.21.2.117:0/1279556623 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc018098cf0 con 0x7fc018096370 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.705 7fc01f252700 1 -- 172.21.2.117:0/1279556623 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc018098cf0 con 0x7fc018096370 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.705 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc014005310 con 0x7fc018096370 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.705 7fc009ffb700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 0x7fbff801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.705 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc0140328c0 con 0x7fc018096370 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.706 7fc01c86d700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 0x7fbff801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:05.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.706 7fc01c86d700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 0x7fbff801abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:05.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.736 7fc01f252700 1 -- 172.21.2.117:0/1279556623 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0181ad010 con 0x7fc018096370 2021-02-10T11:24:05.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.738 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc014006b90 con 0x7fc018096370 2021-02-10T11:24:05.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.870 7fc01f252700 1 -- 172.21.2.117:0/1279556623 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc0181ad010 con 0x7fc018096370 2021-02-10T11:24:05.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.874 7fc009ffb700 1 -- 172.21.2.117:0/1279556623 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc014000c90 con 0x7fc018096370 2021-02-10T11:24:05.874 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 -- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 msgr2=0x7fbff801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 0x7fbff801abc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 -- 172.21.2.117:0/1279556623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 msgr2=0x7fc01809a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc01400b530 tx=0x7fc01400b8d0).stop 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 -- 172.21.2.117:0/1279556623 shutdown_connections 2021-02-10T11:24:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc01809abc0 0x7fc0181ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc01809a7c0 0x7fc0181ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbff80186c0 0x7fbff801abc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 --2- 172.21.2.117:0/1279556623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc018096370 0x7fc01809a1a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 -- 172.21.2.117:0/1279556623 >> 172.21.2.117:0/1279556623 conn(0x7fc01818cbd0 msgr2=0x7fc01818c000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.879 7fc01f252700 1 -- 172.21.2.117:0/1279556623 shutdown_connections 2021-02-10T11:24:05.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:05.880 7fc01f252700 1 -- 172.21.2.117:0/1279556623 wait complete. 2021-02-10T11:24:05.884 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:06.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.308 7ff5c6c83700 1 Processor -- start 2021-02-10T11:24:06.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.308 7ff5c6c83700 1 -- start start 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c6c83700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff5c0079000 0x7ff5c00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c6c83700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c00d1d20 con 0x7ff5c01b00f0 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c6c83700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c0077a70 con 0x7ff5c007da20 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c6c83700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c00e53c0 con 0x7ff5c0079000 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c52a0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff5c01b00f0 0x7ff5c00d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43062/0 (socket says 172.21.2.117:43062) 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5c52a0700 1 -- 172.21.2.117:0/4166479332 learned_addr learned my addr 172.21.2.117:0/4166479332 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989845535 0 0) 0x7ff5b8001180 con 0x7ff5c01b00f0 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.309 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ac002fe0 con 0x7ff5c01b00f0 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> v2:172.21.2.125:3300/0 conn(0x7ff5c0079000 0x7ff5c00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3213266004 0 0) 0x7ff5b80010e0 con 0x7ff5c01b00f0 2021-02-10T11:24:06.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 >> v2:172.21.2.125:3300/0 conn(0x7ff5c0079000 msgr2=0x7ff5c00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 --2- 172.21.2.117:0/4166479332 >> v2:172.21.2.125:3300/0 conn(0x7ff5c0079000 0x7ff5c00e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 >> v1:172.21.2.125:6789/0 conn(0x7ff5c007da20 legacy=0x7ff5c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c0079400 con 0x7ff5c01b00f0 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff5b8001600 con 0x7ff5c01b00f0 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> v2:172.21.2.125:3300/0 conn(0x7ff5c0079000 0x7ff5c00e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 >> v1:172.21.2.117:6789/0 conn(0x7ff5c01b00f0 legacy=0x7ff5c00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5ac003aa0 0x7ff5ac005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 0x7ff5ac008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5ac009590 0x7ff5ac00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac00c300 con 0x7ff5ac006790 2021-02-10T11:24:06.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac00c520 con 0x7ff5ac009590 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5ac00c740 con 0x7ff5ac003aa0 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 0x7ff5ac008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5c52a0700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5ac009590 0x7ff5ac00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5c4a9f700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5ac003aa0 0x7ff5ac005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 -- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5ac003aa0 msgr2=0x7ff5ac005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5ac003aa0 0x7ff5ac005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 -- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5ac009590 msgr2=0x7ff5ac00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5ac009590 0x7ff5ac00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 -- 172.21.2.117:0/4166479332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac00c9e0 con 0x7ff5ac006790 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5c52a0700 1 -- 172.21.2.117:0/4166479332 reap_dead start 2021-02-10T11:24:06.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bffff700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 0x7ff5ac008c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff5b0006b40 tx=0x7ff5b004f300).ready entity=mon.0 client_cookie=6ead85aac992268 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5b0006b80 con 0x7ff5ac006790 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b0006810 con 0x7ff5ac006790 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.310 7ff5bf7fe700 1 -- 172.21.2.117:0/4166479332 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5b004f8e0 con 0x7ff5ac006790 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 -- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 msgr2=0x7ff5ac008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 0x7ff5ac008c90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff5b0006b40 tx=0x7ff5b004f300).stop 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 -- 172.21.2.117:0/4166479332 shutdown_connections 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 --2- 172.21.2.117:0/4166479332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5ac006790 0x7ff5ac008c90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 -- 172.21.2.117:0/4166479332 >> 172.21.2.117:0/4166479332 conn(0x7ff5c00ce450 msgr2=0x7ff5c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 -- 172.21.2.117:0/4166479332 shutdown_connections 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.311 7ff5c6c83700 1 -- 172.21.2.117:0/4166479332 wait complete. 2021-02-10T11:24:06.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 Processor -- start 2021-02-10T11:24:06.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 -- start start 2021-02-10T11:24:06.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5c01a64e0 0x7ff5c01a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5c01a6bf0 0x7ff5c01a7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c01a8cf0 con 0x7ff5c00a7410 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c01a71f0 con 0x7ff5c01a64e0 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c6c83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c0191570 con 0x7ff5c01a6bf0 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.326 7ff5c4a9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42820/0 (socket says 172.21.2.117:42820) 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 -- 172.21.2.117:0/1735147147 learned_addr learned my addr 172.21.2.117:0/1735147147 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c52a0700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5c01a6bf0 0x7ff5c01a7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5bffff700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5c01a64e0 0x7ff5c01a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 -- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5c01a6bf0 msgr2=0x7ff5c01a7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5c01a6bf0 0x7ff5c01a7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 -- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5c01a64e0 msgr2=0x7ff5c01a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5c01a64e0 0x7ff5c01a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 -- 172.21.2.117:0/1735147147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c01a83b0 con 0x7ff5c00a7410 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c4a9f700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7ff5a8006f10 tx=0x7ff5a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a800bd80 con 0x7ff5c00a7410 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a8006030 con 0x7ff5c00a7410 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5a80052e0 con 0x7ff5c00a7410 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c01a83b0 con 0x7ff5c00a7410 2021-02-10T11:24:06.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.327 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c01a8940 con 0x7ff5c00a7410 2021-02-10T11:24:06.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.328 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5a8006210 con 0x7ff5c00a7410 2021-02-10T11:24:06.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.328 7ff5bd7fa700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 0x7ff5a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.328 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff5a80326d0 con 0x7ff5c00a7410 2021-02-10T11:24:06.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.329 7ff5bffff700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 0x7ff5a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.329 7ff5bffff700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 0x7ff5a401abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.359 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c0077a70 con 0x7ff5c00a7410 2021-02-10T11:24:06.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.362 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff5a8022e70 con 0x7ff5c00a7410 2021-02-10T11:24:06.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.489 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff5c0077a70 con 0x7ff5c00a7410 2021-02-10T11:24:06.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.493 7ff5bd7fa700 1 -- 172.21.2.117:0/1735147147 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v12) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff5a80061f0 con 0x7ff5c00a7410 2021-02-10T11:24:06.493 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:24:06.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 msgr2=0x7ff5a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 0x7ff5a401abc0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 msgr2=0x7ff5c01a5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7ff5a8006f10 tx=0x7ff5a800b990).stop 2021-02-10T11:24:06.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 shutdown_connections 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5c01a6bf0 0x7ff5c01a7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff5c01a64e0 0x7ff5c01a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff5a40186c0 0x7ff5a401abc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 --2- 172.21.2.117:0/1735147147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5c00a7410 0x7ff5c01a5ec0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.498 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 >> 172.21.2.117:0/1735147147 conn(0x7ff5c019f530 msgr2=0x7ff5c019bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.499 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 shutdown_connections 2021-02-10T11:24:06.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.499 7ff5c6c83700 1 -- 172.21.2.117:0/1735147147 wait complete. 2021-02-10T11:24:06.504 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.919 7f97345ba700 1 Processor -- start 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.919 7f97345ba700 1 -- start start 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97345ba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f972c07da20 0x7f972c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97345ba700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c0e4d80 0x7f972c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97345ba700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972c077a70 con 0x7f972c0e5180 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97345ba700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972c0777d0 con 0x7f972c07da20 2021-02-10T11:24:06.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97345ba700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c07de20 con 0x7f972c0e4d80 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97323d6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f972c07da20 0x7f972c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97323d6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f972c07da20 0x7f972c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42826/0 (socket says 172.21.2.117:42826) 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f97323d6700 1 -- 172.21.2.117:0/23352775 learned_addr learned my addr 172.21.2.117:0/23352775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.920 7f9731bd5700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c0e4d80 0x7f972c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97323d6700 1 -- 172.21.2.117:0/23352775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c0e4d80 msgr2=0x7f972c1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97323d6700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c0e4d80 0x7f972c1b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97323d6700 1 -- 172.21.2.117:0/23352775 >> v1:172.21.2.125:6789/0 conn(0x7f972c0e5180 legacy=0x7f972c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97323d6700 1 -- 172.21.2.117:0/23352775 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c1b06f0 con 0x7f972c07da20 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97323d6700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c07da20 0x7f972c0771b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9720006700 tx=0x7f9720009b80).ready entity=mon.0 client_cookie=fe0831911fbf9f45 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97313d4700 1 -- 172.21.2.117:0/23352775 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972000c0c0 con 0x7f972c07da20 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97313d4700 1 -- 172.21.2.117:0/23352775 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972000c340 con 0x7f972c07da20 2021-02-10T11:24:06.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97313d4700 1 -- 172.21.2.117:0/23352775 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972000c860 con 0x7f972c07da20 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97345ba700 1 -- 172.21.2.117:0/23352775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c07da20 msgr2=0x7f972c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.921 7f97345ba700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c07da20 0x7f972c0771b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f9720006700 tx=0x7f9720009b80).stop 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 -- 172.21.2.117:0/23352775 shutdown_connections 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c0e4d80 0x7f972c1b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 --2- 172.21.2.117:0/23352775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c07da20 0x7f972c0771b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 -- 172.21.2.117:0/23352775 >> 172.21.2.117:0/23352775 conn(0x7f972c0ce430 msgr2=0x7f972c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 -- 172.21.2.117:0/23352775 shutdown_connections 2021-02-10T11:24:06.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.922 7f97345ba700 1 -- 172.21.2.117:0/23352775 wait complete. 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 Processor -- start 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 -- start start 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f972c0ad660 0x7f972c0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c188650 0x7f972c188a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c1af0d0 con 0x7f972c187c30 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c1af2c0 con 0x7f972c0ad660 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97345ba700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c1af4f0 con 0x7f972c188650 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f9731bd5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f97323d6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f972c0ad660 0x7f972c0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f9731bd5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42832/0 (socket says 172.21.2.117:42832) 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f9731bd5700 1 -- 172.21.2.117:0/3316554823 learned_addr learned my addr 172.21.2.117:0/3316554823 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.937 7f9732bd7700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c188650 0x7f972c188a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 -- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c188650 msgr2=0x7f972c188a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c188650 0x7f972c188a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 -- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f972c0ad660 msgr2=0x7f972c0ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f972c0ad660 0x7f972c0ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 -- 172.21.2.117:0/3316554823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c1896f0 con 0x7f972c187c30 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f9731bd5700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f9728006f60 tx=0x7f972800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f972800bd60 con 0x7f972c187c30 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728000c90 con 0x7f972c187c30 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97280022b0 con 0x7f972c187c30 2021-02-10T11:24:06.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f97345ba700 1 -- 172.21.2.117:0/3316554823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c1896f0 con 0x7f972c187c30 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.938 7f97345ba700 1 -- 172.21.2.117:0/3316554823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c189c80 con 0x7f972c187c30 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.939 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9728006b00 con 0x7f972c187c30 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.939 7f971effd700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 0x7f971401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.939 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9728032670 con 0x7f972c187c30 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.939 7f97323d6700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 0x7f971401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:06.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.940 7f97323d6700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 0x7f971401abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:06.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.970 7f97345ba700 1 -- 172.21.2.117:0/3316554823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c077a70 con 0x7f972c187c30 2021-02-10T11:24:06.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:06.973 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9728000940 con 0x7f972c187c30 2021-02-10T11:24:07.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.102 7f97345ba700 1 -- 172.21.2.117:0/3316554823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f972c077a70 con 0x7f972c187c30 2021-02-10T11:24:07.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.105 7f971effd700 1 -- 172.21.2.117:0/3316554823 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7f972801f4f0 con 0x7f972c187c30 2021-02-10T11:24:07.106 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:24:07.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 msgr2=0x7f971401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 0x7f971401abc0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 msgr2=0x7f972c188030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f9728006f60 tx=0x7f972800b970).stop 2021-02-10T11:24:07.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 shutdown_connections 2021-02-10T11:24:07.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97140186c0 0x7f971401abc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f972c187c30 0x7f972c188030 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f972c188650 0x7f972c188a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 --2- 172.21.2.117:0/3316554823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f972c0ad660 0x7f972c0ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 >> 172.21.2.117:0/3316554823 conn(0x7f972c199730 msgr2=0x7f972c196f70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 shutdown_connections 2021-02-10T11:24:07.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.111 7f97345ba700 1 -- 172.21.2.117:0/3316554823 wait complete. 2021-02-10T11:24:07.116 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 Processor -- start 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 -- start start 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c07da20 0x7f380c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f380c0e4d80 0x7f380c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f380c0e5180 0x7f380c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c0e5c00 con 0x7f380c0e5180 2021-02-10T11:24:07.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380c077a70 con 0x7f380c0e4d80 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.532 7f3811a9d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c0777d0 con 0x7f380c07da20 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f380c0e5180 0x7f380c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f380c0e5180 0x7f380c077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42840/0 (socket says 172.21.2.117:42840) 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 -- 172.21.2.117:0/611020238 learned_addr learned my addr 172.21.2.117:0/611020238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 -- 172.21.2.117:0/611020238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c07da20 msgr2=0x7f380c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 --2- 172.21.2.117:0/611020238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c07da20 0x7f380c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 -- 172.21.2.117:0/611020238 >> v2:172.21.2.125:3300/0 conn(0x7f380c0e4d80 msgr2=0x7f380c1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 --2- 172.21.2.117:0/611020238 >> v2:172.21.2.125:3300/0 conn(0x7f380c0e4d80 0x7f380c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 -- 172.21.2.117:0/611020238 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c07de20 con 0x7f380c0e5180 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380bfff700 1 --2- 172.21.2.117:0/611020238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c0e5180 0x7f380c077ff0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f37fc006b80 tx=0x7f37fc009b90).ready entity=mon.0 client_cookie=6b5aca0dfda54869 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380a7fc700 1 -- 172.21.2.117:0/611020238 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37fc0022e0 con 0x7f380c0e5180 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380a7fc700 1 -- 172.21.2.117:0/611020238 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37fc002560 con 0x7f380c0e5180 2021-02-10T11:24:07.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f380a7fc700 1 -- 172.21.2.117:0/611020238 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37fc002a80 con 0x7f380c0e5180 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f3811a9d700 1 -- 172.21.2.117:0/611020238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c0e5180 msgr2=0x7f380c077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.533 7f3811a9d700 1 --2- 172.21.2.117:0/611020238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c0e5180 0x7f380c077ff0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f37fc006b80 tx=0x7f37fc009b90).stop 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 -- 172.21.2.117:0/611020238 shutdown_connections 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 --2- 172.21.2.117:0/611020238 >> v2:172.21.2.125:3300/0 conn(0x7f380c0e4d80 0x7f380c1b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 --2- 172.21.2.117:0/611020238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c0e5180 0x7f380c077ff0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 --2- 172.21.2.117:0/611020238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c07da20 0x7f380c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 -- 172.21.2.117:0/611020238 >> 172.21.2.117:0/611020238 conn(0x7f380c0ce430 msgr2=0x7f380c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 -- 172.21.2.117:0/611020238 shutdown_connections 2021-02-10T11:24:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.534 7f3811a9d700 1 -- 172.21.2.117:0/611020238 wait complete. 2021-02-10T11:24:07.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.549 7f3811a9d700 1 Processor -- start 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 -- start start 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c19e440 0x7f380c1b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f380c0aa830 0x7f380c18fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c18f890 con 0x7f380c18efe0 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c19e840 con 0x7f380c0aa830 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f3811a9d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c19ea20 con 0x7f380c19e440 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f380affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f380affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42846/0 (socket says 172.21.2.117:42846) 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f380affd700 1 -- 172.21.2.117:0/1052304232 learned_addr learned my addr 172.21.2.117:0/1052304232 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:07.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f380bfff700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f380c0aa830 0x7f380c18fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.550 7f380b7fe700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c19e440 0x7f380c1b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 -- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c19e440 msgr2=0x7f380c1b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c19e440 0x7f380c1b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 -- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f380c0aa830 msgr2=0x7f380c18fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f380c0aa830 0x7f380c18fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 -- 172.21.2.117:0/1052304232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c18f3e0 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f380affd700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3804009980 tx=0x7f380400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f380400c2f0 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380400c570 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c18f3e0 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c190720 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.551 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f380400ca90 con 0x7f380c18efe0 2021-02-10T11:24:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.552 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3804001010 con 0x7f380c18efe0 2021-02-10T11:24:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.552 7f37f3fff700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 0x7f37ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.552 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3804032c40 con 0x7f380c18efe0 2021-02-10T11:24:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.552 7f380b7fe700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 0x7f37ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.552 7f380b7fe700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 0x7f37ec01abc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:07.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.582 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c095390 con 0x7f380c18efe0 2021-02-10T11:24:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.585 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3804000cc0 con 0x7f380c18efe0 2021-02-10T11:24:07.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.715 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f380c095390 con 0x7f380c18efe0 2021-02-10T11:24:07.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.718 7f37f3fff700 1 -- 172.21.2.117:0/1052304232 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3804001010 con 0x7f380c18efe0 2021-02-10T11:24:07.718 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:24:07.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 msgr2=0x7f37ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 0x7f37ec01abc0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 msgr2=0x7f380c0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:07.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3804009980 tx=0x7f380400bcd0).stop 2021-02-10T11:24:07.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 shutdown_connections 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37ec0186c0 0x7f37ec01abc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f380c18efe0 0x7f380c0aa210 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f380c0aa830 0x7f380c18fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 --2- 172.21.2.117:0/1052304232 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f380c19e440 0x7f380c1b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.723 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 >> 172.21.2.117:0/1052304232 conn(0x7f380c19f530 msgr2=0x7f380c18bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.724 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 shutdown_connections 2021-02-10T11:24:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:07.724 7f3811a9d700 1 -- 172.21.2.117:0/1052304232 wait complete. 2021-02-10T11:24:07.728 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 Processor -- start 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 -- start start 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7c0079000 0x7fa7c00e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c00d1d10 con 0x7fa7c01b00e0 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c0077a70 con 0x7fa7c007da20 2021-02-10T11:24:08.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7c5783700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c00e53b0 con 0x7fa7c0079000 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7bed73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7c0079000 0x7fa7c00e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7bed73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7c0079000 0x7fa7c00e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42850/0 (socket says 172.21.2.117:42850) 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.141 7fa7bed73700 1 -- 172.21.2.117:0/2015984495 learned_addr learned my addr 172.21.2.117:0/2015984495 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7bed73700 1 -- 172.21.2.117:0/2015984495 >> v1:172.21.2.125:6789/0 conn(0x7fa7c007da20 legacy=0x7fa7c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7bed73700 1 -- 172.21.2.117:0/2015984495 >> v1:172.21.2.117:6789/0 conn(0x7fa7c01b00e0 legacy=0x7fa7c00d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7be572700 1 -- 172.21.2.117:0/2015984495 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775074584 0 0) 0x7fa7b8000f90 con 0x7fa7c01b00e0 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7bed73700 1 -- 172.21.2.117:0/2015984495 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0079400 con 0x7fa7c0079000 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7bed73700 1 --2- 172.21.2.117:0/2015984495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c0079000 0x7fa7c00e4d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa7b0006700 tx=0x7fa7b0009b00).ready entity=mon.0 client_cookie=f153aed094a90372 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7be572700 1 -- 172.21.2.117:0/2015984495 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7b000c0c0 con 0x7fa7c0079000 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7be572700 1 -- 172.21.2.117:0/2015984495 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b000c340 con 0x7fa7c0079000 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.142 7fa7be572700 1 -- 172.21.2.117:0/2015984495 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7b000c820 con 0x7fa7c0079000 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 -- 172.21.2.117:0/2015984495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c0079000 msgr2=0x7fa7c00e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 --2- 172.21.2.117:0/2015984495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c0079000 0x7fa7c00e4d90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fa7b0006700 tx=0x7fa7b0009b00).stop 2021-02-10T11:24:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 -- 172.21.2.117:0/2015984495 shutdown_connections 2021-02-10T11:24:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 --2- 172.21.2.117:0/2015984495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c0079000 0x7fa7c00e4d90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 -- 172.21.2.117:0/2015984495 >> 172.21.2.117:0/2015984495 conn(0x7fa7c00ce440 msgr2=0x7fa7c00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 -- 172.21.2.117:0/2015984495 shutdown_connections 2021-02-10T11:24:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.143 7fa7c5783700 1 -- 172.21.2.117:0/2015984495 wait complete. 2021-02-10T11:24:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 Processor -- start 2021-02-10T11:24:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 -- start start 2021-02-10T11:24:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7c0095f00 0x7fa7c00b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 0x7fa7c00b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c0096980 con 0x7fa7c008d2f0 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c01af8f0 con 0x7fa7c00b79d0 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7c5783700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c008ce40 con 0x7fa7c0095f00 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.158 7fa7bf574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42856/0 (socket says 172.21.2.117:42856) 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 -- 172.21.2.117:0/3378008556 learned_addr learned my addr 172.21.2.117:0/3378008556 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bed73700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7c0095f00 0x7fa7c00b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bfd75700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 0x7fa7c00b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 -- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7c0095f00 msgr2=0x7fa7c00b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7c0095f00 0x7fa7c00b73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 -- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 msgr2=0x7fa7c00b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 0x7fa7c00b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 -- 172.21.2.117:0/3378008556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c0097650 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bfd75700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 0x7fa7c00b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7bf574700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa7a800b5b0 tx=0x7fa7a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7a800bd40 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a8000e20 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c0097650 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.159 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7a80062a0 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.160 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c0097650 con 0x7fa7c008d2f0 2021-02-10T11:24:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.160 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7a800a000 con 0x7fa7c008d2f0 2021-02-10T11:24:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.160 7fa7b7fff700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 0x7fa7a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.160 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa7a8032920 con 0x7fa7c008d2f0 2021-02-10T11:24:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.161 7fa7bed73700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 0x7fa7a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.161 7fa7bed73700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 0x7fa7a401abc0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.191 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c0104a20 con 0x7fa7c008d2f0 2021-02-10T11:24:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.194 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa7a8000ce0 con 0x7fa7c008d2f0 2021-02-10T11:24:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.322 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7c0104a20 con 0x7fa7c008d2f0 2021-02-10T11:24:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.322 7fa7b7fff700 1 -- 172.21.2.117:0/3378008556 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2097 (secure 0 0 0) 0x7fa7a8000ef0 con 0x7fa7c008d2f0 2021-02-10T11:24:08.320 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:08.320 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:24:08.320 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 12 2021-02-10T11:24:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 msgr2=0x7fa7a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 0x7fa7a401abc0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 msgr2=0x7fa7c00958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fa7a800b5b0 tx=0x7fa7a800b950).stop 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 shutdown_connections 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7c00b79d0 0x7fa7c00b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7c0095f00 0x7fa7c00b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7a40186c0 0x7fa7a401abc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 --2- 172.21.2.117:0/3378008556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7c008d2f0 0x7fa7c00958e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 >> 172.21.2.117:0/3378008556 conn(0x7fa7c00970a0 msgr2=0x7fa7c010d060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 shutdown_connections 2021-02-10T11:24:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:08.324 7fa7c5783700 1 -- 172.21.2.117:0/3378008556 wait complete. 2021-02-10T11:24:08.532 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:01.662717', 'modified': '2021-02-10 11:24:04.035038', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4412': {'gid': 4412, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.125:6835/3266045438', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3266045438}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3266045438}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:08.532 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:08.532 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:08.929 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4412, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 7.269030002, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 12, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 31, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 31, 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.530491782999999 2021-02-10T11:24:08.930 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:24:08.934 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4412, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.269030002, "mdsmap_epoch": 12, "osdmap_epoch": 31, "osdmap_epoch_barrier": 31, "uptime": 16.530491782999999 } 2021-02-10T11:24:08.934 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.347 7fe2b6514700 1 Processor -- start 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.347 7fe2b6514700 1 -- start start 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b6514700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b0079000 0x7fe2b00e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b6514700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe2b01b00e0 0x7fe2b01b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b6514700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b0077a70 con 0x7fe2b007da20 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b6514700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b00e53b0 con 0x7fe2b01b00e0 2021-02-10T11:24:09.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b6514700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b0079400 con 0x7fe2b0079000 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2affff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe2b007da20 0x7fe2b00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43112/0 (socket says 172.21.2.117:43112) 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2affff700 1 -- 172.21.2.117:0/569946868 learned_addr learned my addr 172.21.2.117:0/569946868 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2b4b31700 1 --2- 172.21.2.117:0/569946868 >> v2:172.21.2.125:3300/0 conn(0x7fe2b01b00e0 0x7fe2b01b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.348 7fe2af7fe700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b0079000 0x7fe2b00e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2aeffd700 1 -- 172.21.2.117:0/569946868 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639120913 0 0) 0x7fe2a00010b0 con 0x7fe2b007da20 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2aeffd700 1 -- 172.21.2.117:0/569946868 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe298002fe0 con 0x7fe2b007da20 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b4b31700 1 -- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b0079000 msgr2=0x7fe2b00e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b4b31700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b0079000 0x7fe2b00e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b4b31700 1 -- 172.21.2.117:0/569946868 >> v1:172.21.2.117:6789/0 conn(0x7fe2b007da20 legacy=0x7fe2b00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b4b31700 1 -- 172.21.2.117:0/569946868 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b007de20 con 0x7fe2b01b00e0 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b4b31700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b01b00e0 0x7fe2b01b04e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fe2a8006e50 tx=0x7fe2a8009e70).ready entity=mon.1 client_cookie=8ae3777c0be757da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2aeffd700 1 -- 172.21.2.117:0/569946868 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a800b300 con 0x7fe2b01b00e0 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2aeffd700 1 -- 172.21.2.117:0/569946868 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a8003030 con 0x7fe2b01b00e0 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2aeffd700 1 -- 172.21.2.117:0/569946868 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a8000f90 con 0x7fe2b01b00e0 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b6514700 1 -- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b01b00e0 msgr2=0x7fe2b01b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.349 7fe2b6514700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b01b00e0 0x7fe2b01b04e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fe2a8006e50 tx=0x7fe2a8009e70).stop 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 -- 172.21.2.117:0/569946868 shutdown_connections 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b01b00e0 0x7fe2b01b04e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 --2- 172.21.2.117:0/569946868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b0079000 0x7fe2b00e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 -- 172.21.2.117:0/569946868 >> 172.21.2.117:0/569946868 conn(0x7fe2b00ce440 msgr2=0x7fe2b00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 -- 172.21.2.117:0/569946868 shutdown_connections 2021-02-10T11:24:09.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.350 7fe2b6514700 1 -- 172.21.2.117:0/569946868 wait complete. 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 Processor -- start 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 -- start start 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 0x7fe2b019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b008d410 0x7fe2b008d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b008c080 con 0x7fe2b008cd50 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b008c2a0 con 0x7fe2b019dfe0 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2b6514700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b008c4c0 con 0x7fe2b008d410 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2affff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2af7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 0x7fe2b019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2affff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42866/0 (socket says 172.21.2.117:42866) 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.365 7fe2affff700 1 -- 172.21.2.117:0/586556932 learned_addr learned my addr 172.21.2.117:0/586556932 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2b4b31700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b008d410 0x7fe2b008d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 -- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b008d410 msgr2=0x7fe2b008d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b008d410 0x7fe2b008d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 -- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 msgr2=0x7fe2b019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 0x7fe2b019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 -- 172.21.2.117:0/586556932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b008c6e0 con 0x7fe2b008cd50 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2af7fe700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 0x7fe2b019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2affff700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe2a0001c60 tx=0x7fe2a000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a000acc0 con 0x7fe2b008cd50 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a0004140 con 0x7fe2b008cd50 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2a0004620 con 0x7fe2b008cd50 2021-02-10T11:24:09.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2b6514700 1 -- 172.21.2.117:0/586556932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b008c6e0 con 0x7fe2b008cd50 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.366 7fe2b6514700 1 -- 172.21.2.117:0/586556932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b008c6e0 con 0x7fe2b008cd50 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.367 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2a0004380 con 0x7fe2b008cd50 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.367 7fe2acff9700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 0x7fe29001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.367 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe2a00327b0 con 0x7fe2b008cd50 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.367 7fe2af7fe700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 0x7fe29001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.368 7fe2af7fe700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 0x7fe29001abc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.398 7fe2b6514700 1 -- 172.21.2.117:0/586556932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b0077a70 con 0x7fe2b008cd50 2021-02-10T11:24:09.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.401 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe2a00009b0 con 0x7fe2b008cd50 2021-02-10T11:24:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.524 7fe2b6514700 1 -- 172.21.2.117:0/586556932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2b0077a70 con 0x7fe2b008cd50 2021-02-10T11:24:09.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.524 7fe2acff9700 1 -- 172.21.2.117:0/586556932 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14226 (secure 0 0 0) 0x7fe2a0004140 con 0x7fe2b008cd50 2021-02-10T11:24:09.521 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:09.522 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":31,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:02.664299","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:09.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 msgr2=0x7fe29001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 0x7fe29001abc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 msgr2=0x7fe2b019d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe2a0001c60 tx=0x7fe2a000a8d0).stop 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 shutdown_connections 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2b008d410 0x7fe2b008d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2b019dfe0 0x7fe2b019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe2900186c0 0x7fe29001abc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 --2- 172.21.2.117:0/586556932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2b008cd50 0x7fe2b019d330 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 >> 172.21.2.117:0/586556932 conn(0x7fe2b019f4c0 msgr2=0x7fe2b0094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:09.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 shutdown_connections 2021-02-10T11:24:09.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.526 7fe2b6514700 1 -- 172.21.2.117:0/586556932 wait complete. 2021-02-10T11:24:09.531 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:23:59.642031', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:00.660978', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:09.532 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 Processor -- start 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 -- start start 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9290079000 0x7f92900e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 0x7f92901b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9290077a70 con 0x7f929007da20 2021-02-10T11:24:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92900e53c0 con 0x7f9290079000 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9298607700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9290079400 con 0x7f92901b00f0 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9295c22700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9290079000 0x7f92900e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9295c22700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9290079000 0x7f92900e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42874/0 (socket says 172.21.2.117:42874) 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.949 7f9295c22700 1 -- 172.21.2.117:0/2988800999 learned_addr learned my addr 172.21.2.117:0/2988800999 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9296c24700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 0x7f92901b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295c22700 1 -- 172.21.2.117:0/2988800999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 msgr2=0x7f92901b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295c22700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 0x7f92901b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295c22700 1 -- 172.21.2.117:0/2988800999 >> v1:172.21.2.125:6789/0 conn(0x7f929007da20 legacy=0x7f92900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295c22700 1 -- 172.21.2.117:0/2988800999 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929007de20 con 0x7f9290079000 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9296c24700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 0x7f92901b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295c22700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9290079000 0x7f92900e4da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9280006700 tx=0x7f9280009b80).ready entity=mon.0 client_cookie=f404772fd24bd050 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295421700 1 -- 172.21.2.117:0/2988800999 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9280006340 con 0x7f9290079000 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295421700 1 -- 172.21.2.117:0/2988800999 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928000c0c0 con 0x7f9290079000 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9295421700 1 -- 172.21.2.117:0/2988800999 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f928000c5e0 con 0x7f9290079000 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9298607700 1 -- 172.21.2.117:0/2988800999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9290079000 msgr2=0x7f92900e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.950 7f9298607700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9290079000 0x7f92900e4da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f9280006700 tx=0x7f9280009b80).stop 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 -- 172.21.2.117:0/2988800999 shutdown_connections 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b00f0 0x7f92901b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 --2- 172.21.2.117:0/2988800999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9290079000 0x7f92900e4da0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 -- 172.21.2.117:0/2988800999 >> 172.21.2.117:0/2988800999 conn(0x7f92900ce450 msgr2=0x7f92900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 -- 172.21.2.117:0/2988800999 shutdown_connections 2021-02-10T11:24:09.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.951 7f9298607700 1 -- 172.21.2.117:0/2988800999 wait complete. 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 Processor -- start 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 -- start start 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92901b5130 0x7f92901b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b5f50 0x7f92900b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900b8da0 con 0x7f92901b5b50 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92900a4140 con 0x7f92901b5130 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9298607700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92900a4320 con 0x7f92901b5f50 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9295c22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9295c22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42880/0 (socket says 172.21.2.117:42880) 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.966 7f9295c22700 1 -- 172.21.2.117:0/3204372783 learned_addr learned my addr 172.21.2.117:0/3204372783 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 -- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b5f50 msgr2=0x7f92900b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b5f50 0x7f92900b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 -- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92901b5130 msgr2=0x7f92901b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92901b5130 0x7f92901b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 -- 172.21.2.117:0/3204372783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900b62b0 con 0x7f92901b5b50 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9295c22700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f928000b5d0 tx=0x7f928000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f928000bd60 con 0x7f92901b5b50 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9280002030 con 0x7f92901b5b50 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9298607700 1 -- 172.21.2.117:0/3204372783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92900b62b0 con 0x7f92901b5b50 2021-02-10T11:24:09.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9298607700 1 -- 172.21.2.117:0/3204372783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92900b6840 con 0x7f92901b5b50 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.967 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9280002550 con 0x7f92901b5b50 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.968 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9280000c90 con 0x7f92901b5b50 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.968 7f9286ffd700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 0x7f927801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.968 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9280032950 con 0x7f92901b5b50 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.968 7f9296423700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 0x7f927801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:09.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.969 7f9296423700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 0x7f927801abd0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:09.999 7f9298607700 1 -- 172.21.2.117:0/3204372783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92901b4f00 con 0x7f92901b5b50 2021-02-10T11:24:09.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.002 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92800009b0 con 0x7f92901b5b50 2021-02-10T11:24:10.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.128 7f9298607700 1 -- 172.21.2.117:0/3204372783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f9290077a70 con 0x7f92901b5b50 2021-02-10T11:24:10.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.128 7f9286ffd700 1 -- 172.21.2.117:0/3204372783 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f928000bd60 con 0x7f92901b5b50 2021-02-10T11:24:10.125 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:10.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.130 7f9298607700 1 -- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 msgr2=0x7f927801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.130 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 0x7f927801abd0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.130 7f9298607700 1 -- 172.21.2.117:0/3204372783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 msgr2=0x7f92900b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.130 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f928000b5d0 tx=0x7f928000b970).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 -- 172.21.2.117:0/3204372783 shutdown_connections 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92780186d0 0x7f927801abd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92901b5b50 0x7f92900b5670 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92901b5f50 0x7f92900b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 --2- 172.21.2.117:0/3204372783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92901b5130 0x7f92901b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 -- 172.21.2.117:0/3204372783 >> 172.21.2.117:0/3204372783 conn(0x7f929008bbc0 msgr2=0x7f92900c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 -- 172.21.2.117:0/3204372783 shutdown_connections 2021-02-10T11:24:10.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.131 7f9298607700 1 -- 172.21.2.117:0/3204372783 wait complete. 2021-02-10T11:24:10.136 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 Processor -- start 2021-02-10T11:24:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 -- start start 2021-02-10T11:24:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd36807da20 0x7fd3680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 0x7fd3681b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd368077a70 con 0x7fd3680e51a0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680777d0 con 0x7fd36807da20 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.551 7fd36dfae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36807de20 con 0x7fd3680e4da0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd367fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd3680e51a0 0x7fd368077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43138/0 (socket says 172.21.2.117:43138) 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd367fff700 1 -- 172.21.2.117:0/1053871434 learned_addr learned my addr 172.21.2.117:0/1053871434 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3677fe700 1 --2- 172.21.2.117:0/1053871434 >> v2:172.21.2.125:3300/0 conn(0x7fd36807da20 0x7fd3680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd366ffd700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 0x7fd3681b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3667fc700 1 -- 172.21.2.117:0/1053871434 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606142575 0 0) 0x7fd360000f90 con 0x7fd3680e51a0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3667fc700 1 -- 172.21.2.117:0/1053871434 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd354002fe0 con 0x7fd3680e51a0 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3677fe700 1 -- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 msgr2=0x7fd3681b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3677fe700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 0x7fd3681b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3677fe700 1 -- 172.21.2.117:0/1053871434 >> v1:172.21.2.117:6789/0 conn(0x7fd3680e51a0 legacy=0x7fd368077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd3677fe700 1 -- 172.21.2.117:0/1053871434 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3681b0710 con 0x7fd36807da20 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.552 7fd366ffd700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 0x7fd3681b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.553 7fd3677fe700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd36807da20 0x7fd3680771b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd350006650 tx=0x7fd350009a80).ready entity=mon.1 client_cookie=b44f58a4394e9332 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.553 7fd3667fc700 1 -- 172.21.2.117:0/1053871434 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35000c0c0 con 0x7fd36807da20 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.553 7fd3667fc700 1 -- 172.21.2.117:0/1053871434 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35000c340 con 0x7fd36807da20 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.553 7fd3667fc700 1 -- 172.21.2.117:0/1053871434 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd35000c860 con 0x7fd36807da20 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 -- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd36807da20 msgr2=0x7fd3680771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd36807da20 0x7fd3680771b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd350006650 tx=0x7fd350009a80).stop 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 -- 172.21.2.117:0/1053871434 shutdown_connections 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3680e4da0 0x7fd3681b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 --2- 172.21.2.117:0/1053871434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd36807da20 0x7fd3680771b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 -- 172.21.2.117:0/1053871434 >> 172.21.2.117:0/1053871434 conn(0x7fd3680ce450 msgr2=0x7fd3680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 -- 172.21.2.117:0/1053871434 shutdown_connections 2021-02-10T11:24:10.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.554 7fd36dfae700 1 -- 172.21.2.117:0/1053871434 wait complete. 2021-02-10T11:24:10.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 Processor -- start 2021-02-10T11:24:10.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 -- start start 2021-02-10T11:24:10.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd36818fe20 0x7fd3680bd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3680be9b0 0x7fd3680bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680be4b0 con 0x7fd3680bd990 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd368190260 con 0x7fd3680be9b0 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd36dfae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3680c1840 con 0x7fd36818fe20 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd366ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd366ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42894/0 (socket says 172.21.2.117:42894) 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.569 7fd366ffd700 1 -- 172.21.2.117:0/323141412 learned_addr learned my addr 172.21.2.117:0/323141412 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 -- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd36818fe20 msgr2=0x7fd3680bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd36818fe20 0x7fd3680bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 -- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3680be9b0 msgr2=0x7fd3680bedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3680be9b0 0x7fd3680bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 -- 172.21.2.117:0/323141412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680bf3d0 con 0x7fd3680bd990 2021-02-10T11:24:10.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd366ffd700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fd358006f40 tx=0x7fd35800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3580009b0 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35800bd80 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd36dfae700 1 -- 172.21.2.117:0/323141412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3680bf3d0 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd36dfae700 1 -- 172.21.2.117:0/323141412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3680bf960 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.570 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd358002220 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.571 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd358000c90 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.571 7fd35ffff700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 0x7fd34c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.571 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd358032930 con 0x7fd3680bd990 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.571 7fd3677fe700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 0x7fd34c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:10.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.572 7fd3677fe700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 0x7fd34c01abc0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:10.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.601 7fd36dfae700 1 -- 172.21.2.117:0/323141412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368077a70 con 0x7fd3680bd990 2021-02-10T11:24:10.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.604 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd358006c10 con 0x7fd3680bd990 2021-02-10T11:24:10.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.729 7fd36dfae700 1 -- 172.21.2.117:0/323141412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd3680be430 con 0x7fd3680bd990 2021-02-10T11:24:10.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.730 7fd35ffff700 1 -- 172.21.2.117:0/323141412 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v31) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd35800bd80 con 0x7fd3680bd990 2021-02-10T11:24:10.727 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:10.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 msgr2=0x7fd34c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 0x7fd34c01abc0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 msgr2=0x7fd3680bdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fd358006f40 tx=0x7fd35800b990).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 shutdown_connections 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd34c0186c0 0x7fd34c01abc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3680bd990 0x7fd3680bdd90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3680be9b0 0x7fd3680bedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 --2- 172.21.2.117:0/323141412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd36818fe20 0x7fd3680bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:10.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 >> 172.21.2.117:0/323141412 conn(0x7fd3680a1360 msgr2=0x7fd368091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:10.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 shutdown_connections 2021-02-10T11:24:10.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:10.732 7fd36dfae700 1 -- 172.21.2.117:0/323141412 wait complete. 2021-02-10T11:24:10.737 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 Processor -- start 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 -- start start 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fefcc0e4da0 0x7fefcc1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fefcc0e51a0 0x7fefcc077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0e5c20 con 0x7fefcc0e4da0 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc077a70 con 0x7fefcc0e51a0 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.152 7fefd3e2e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0777d0 con 0x7fefcc07da20 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fefcc0e4da0 0x7fefcc1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fefcc0e4da0 0x7fefcc1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42900/0 (socket says 172.21.2.117:42900) 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 -- 172.21.2.117:0/3994527503 learned_addr learned my addr 172.21.2.117:0/3994527503 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:11.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 -- 172.21.2.117:0/3994527503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc07da20 msgr2=0x7fefcc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 --2- 172.21.2.117:0/3994527503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 -- 172.21.2.117:0/3994527503 >> v2:172.21.2.125:3300/0 conn(0x7fefcc0e51a0 msgr2=0x7fefcc077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 --2- 172.21.2.117:0/3994527503 >> v2:172.21.2.125:3300/0 conn(0x7fefcc0e51a0 0x7fefcc077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 -- 172.21.2.117:0/3994527503 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc07de20 con 0x7fefcc0e4da0 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd1449700 1 --2- 172.21.2.117:0/3994527503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc0e4da0 0x7fefcc1b00f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fefbc006700 tx=0x7fefbc009b80).ready entity=mon.0 client_cookie=431a7e88f469fa08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd0c48700 1 -- 172.21.2.117:0/3994527503 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefbc0009b0 con 0x7fefcc0e4da0 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd0c48700 1 -- 172.21.2.117:0/3994527503 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc00c0c0 con 0x7fefcc0e4da0 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd0c48700 1 -- 172.21.2.117:0/3994527503 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefbc00c5e0 con 0x7fefcc0e4da0 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd3e2e700 1 -- 172.21.2.117:0/3994527503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc0e4da0 msgr2=0x7fefcc1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.153 7fefd3e2e700 1 --2- 172.21.2.117:0/3994527503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc0e4da0 0x7fefcc1b00f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fefbc006700 tx=0x7fefbc009b80).stop 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 -- 172.21.2.117:0/3994527503 shutdown_connections 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 --2- 172.21.2.117:0/3994527503 >> v2:172.21.2.125:3300/0 conn(0x7fefcc0e51a0 0x7fefcc077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 --2- 172.21.2.117:0/3994527503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc0e4da0 0x7fefcc1b00f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 --2- 172.21.2.117:0/3994527503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 -- 172.21.2.117:0/3994527503 >> 172.21.2.117:0/3994527503 conn(0x7fefcc0ce450 msgr2=0x7fefcc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 -- 172.21.2.117:0/3994527503 shutdown_connections 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.154 7fefd3e2e700 1 -- 172.21.2.117:0/3994527503 wait complete. 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 Processor -- start 2021-02-10T11:24:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 -- start start 2021-02-10T11:24:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc0b3ff0 0x7fefcc0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc0c33d0 0x7fefcc0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc090600 con 0x7fefcc090de0 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc090860 con 0x7fefcc0b3ff0 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd3e2e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc0b3c70 con 0x7fefcc0c33d0 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd244b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.169 7fefd244b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42906/0 (socket says 172.21.2.117:42906) 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 -- 172.21.2.117:0/1833892388 learned_addr learned my addr 172.21.2.117:0/1833892388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 -- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc0c33d0 msgr2=0x7fefcc0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc0c33d0 0x7fefcc0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 -- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc0b3ff0 msgr2=0x7fefcc0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc0b3ff0 0x7fefcc0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 -- 172.21.2.117:0/1833892388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc091800 con 0x7fefcc090de0 2021-02-10T11:24:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd244b700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fefc4009980 tx=0x7fefc400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc4003020 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4005a80 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc4006030 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.170 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc091800 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.171 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc091800 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.171 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fefc4006030 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.171 7fefc27fc700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 0x7fefb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.171 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fefc40328e0 con 0x7fefcc090de0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.172 7fefd1c4a700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 0x7fefb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.172 7fefd1c4a700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 0x7fefb401abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.203 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc103370 con 0x7fefcc090de0 2021-02-10T11:24:11.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.205 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fefc4005870 con 0x7fefcc090de0 2021-02-10T11:24:11.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.331 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fefcc090600 con 0x7fefcc090de0 2021-02-10T11:24:11.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.331 7fefc27fc700 1 -- 172.21.2.117:0/1833892388 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v31) v1 ==== 100+0+10 (secure 0 0 0) 0x7fefc4006030 con 0x7fefcc090de0 2021-02-10T11:24:11.328 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:11.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 msgr2=0x7fefb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 0x7fefb401abc0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 msgr2=0x7fefcc0911e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fefc4009980 tx=0x7fefc400bcd0).stop 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 shutdown_connections 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefb40186c0 0x7fefb401abc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc090de0 0x7fefcc0911e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc0c33d0 0x7fefcc0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 --2- 172.21.2.117:0/1833892388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc0b3ff0 0x7fefcc0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 >> 172.21.2.117:0/1833892388 conn(0x7fefcc0a1420 msgr2=0x7fefcc09efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 shutdown_connections 2021-02-10T11:24:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.333 7fefd3e2e700 1 -- 172.21.2.117:0/1833892388 wait complete. 2021-02-10T11:24:11.338 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:11.339 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.750 7f29b3b1e700 1 Processor -- start 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.750 7f29b3b1e700 1 -- start start 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f29ac07da20 0x7f29ac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29ac0e4d90 0x7f29ac1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac0e5190 0x7f29ac077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac077a70 con 0x7f29ac0e4d90 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29ac0777d0 con 0x7f29ac07da20 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b3b1e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac07de20 con 0x7f29ac0e5190 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29ac0e4d90 0x7f29ac1b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29ac0e4d90 0x7f29ac1b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42910/0 (socket says 172.21.2.117:42910) 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 -- 172.21.2.117:0/3325105431 learned_addr learned my addr 172.21.2.117:0/3325105431 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 -- 172.21.2.117:0/3325105431 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac0e5190 msgr2=0x7f29ac077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 --2- 172.21.2.117:0/3325105431 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac0e5190 0x7f29ac077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 -- 172.21.2.117:0/3325105431 >> v2:172.21.2.125:3300/0 conn(0x7f29ac07da20 msgr2=0x7f29ac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 --2- 172.21.2.117:0/3325105431 >> v2:172.21.2.125:3300/0 conn(0x7f29ac07da20 0x7f29ac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.751 7f29b1139700 1 -- 172.21.2.117:0/3325105431 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac1b0700 con 0x7f29ac0e4d90 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b1139700 1 --2- 172.21.2.117:0/3325105431 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac0e4d90 0x7f29ac1b00e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f29a8006700 tx=0x7f29a8009b80).ready entity=mon.0 client_cookie=b3b4b95a6d19a92f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b0938700 1 -- 172.21.2.117:0/3325105431 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a800c0c0 con 0x7f29ac0e4d90 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b0938700 1 -- 172.21.2.117:0/3325105431 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a800c400 con 0x7f29ac0e4d90 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b0938700 1 -- 172.21.2.117:0/3325105431 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a800c8e0 con 0x7f29ac0e4d90 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 -- 172.21.2.117:0/3325105431 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac0e4d90 msgr2=0x7f29ac1b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 --2- 172.21.2.117:0/3325105431 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac0e4d90 0x7f29ac1b00e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f29a8006700 tx=0x7f29a8009b80).stop 2021-02-10T11:24:11.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 -- 172.21.2.117:0/3325105431 shutdown_connections 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 --2- 172.21.2.117:0/3325105431 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac0e5190 0x7f29ac077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 --2- 172.21.2.117:0/3325105431 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac0e4d90 0x7f29ac1b00e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 --2- 172.21.2.117:0/3325105431 >> v2:172.21.2.125:3300/0 conn(0x7f29ac07da20 0x7f29ac0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.752 7f29b3b1e700 1 -- 172.21.2.117:0/3325105431 >> 172.21.2.117:0/3325105431 conn(0x7f29ac0ce440 msgr2=0x7f29ac0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.753 7f29b3b1e700 1 -- 172.21.2.117:0/3325105431 shutdown_connections 2021-02-10T11:24:11.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.753 7f29b3b1e700 1 -- 172.21.2.117:0/3325105431 wait complete. 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.767 7f29b3b1e700 1 Processor -- start 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.767 7f29b3b1e700 1 -- start start 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac1a5640 0x7f29ac1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29ac19c030 0x7f29ac19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac1a23d0 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac1a5a40 con 0x7f29ac19c030 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b3b1e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac08a1b0 con 0x7f29ac1a5640 2021-02-10T11:24:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42914/0 (socket says 172.21.2.117:42914) 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 -- 172.21.2.117:0/2698085268 learned_addr learned my addr 172.21.2.117:0/2698085268 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 -- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac1a5640 msgr2=0x7f29ac1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac1a5640 0x7f29ac1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 -- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29ac19c030 msgr2=0x7f29ac19c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29ac19c030 0x7f29ac19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 -- 172.21.2.117:0/2698085268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac19ca50 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f29b1139700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f29a80066a0 tx=0x7f29a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a8009440 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.768 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a80052e0 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac19ca50 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a8005800 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac19ca50 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f29a8006030 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f299e7fc700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f29a80329d0 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.769 7f29b193a700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.770 7f29b193a700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:11.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.801 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac108c10 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.803 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29a8009440 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.930 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29ac077a70 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.930 7f299e7fc700 1 -- 172.21.2.117:0/2698085268 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2097 (secure 0 0 0) 0x7f29a8006030 con 0x7f29ac1a1fd0 2021-02-10T11:24:11.928 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:11.928 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:24:11.928 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 12 2021-02-10T11:24:11.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 msgr2=0x7f299401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 msgr2=0x7f29ac19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:11.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f29a80066a0 tx=0x7f29a800b990).stop 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 shutdown_connections 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29ac1a1fd0 0x7f29ac19ba10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29ac19c030 0x7f29ac19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 --2- 172.21.2.117:0/2698085268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29ac1a5640 0x7f29ac1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 >> 172.21.2.117:0/2698085268 conn(0x7f29ac191110 msgr2=0x7f29ac19aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.932 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 shutdown_connections 2021-02-10T11:24:11.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:11.933 7f29b3b1e700 1 -- 172.21.2.117:0/2698085268 wait complete. 2021-02-10T11:24:12.139 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:01.662717', 'modified': '2021-02-10 11:24:04.035038', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4412': {'gid': 4412, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.125:6835/3266045438', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3266045438}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3266045438}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:12.139 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:12.140 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4412, 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:24:12.536 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.875108999, 2021-02-10T11:24:12.537 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 12, 2021-02-10T11:24:12.537 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 31, 2021-02-10T11:24:12.537 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 31, 2021-02-10T11:24:12.537 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 20.136571749000002 2021-02-10T11:24:12.537 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:24:12.540 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4412, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.875108999, "mdsmap_epoch": 12, "osdmap_epoch": 31, "osdmap_epoch_barrier": 31, "uptime": 20.136571749000002 } 2021-02-10T11:24:12.540 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:12.540 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:24:12.540 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:24:12.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:12.540 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:12.875 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:24:13.219 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:13.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:13.561 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:13.561 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:13.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:13.563 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:24:13.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:24:13.899 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:13.902 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:24:14.244 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:14.244 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:14.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[44954]: starting ceph client 2021-02-10T11:24:14.309 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:24:14.312 7fdb6e78cf80 -1 init, newargv = 0x55a03be92ee0 newargc=9 2021-02-10T11:24:14.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[44954]: starting fuse 2021-02-10T11:24:14.587 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:14.587 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:14.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:14.588 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:24:14.921 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:24:14.922 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:24:14.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:14.922 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:24:14.922 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:24:14.922 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:24:14.922 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:24:14.922 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:24:14.923 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:24:14.924 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:14.924 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:24:14.924 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:24:15.286 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.44954.asok 2021-02-10T11:24:15.294 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44954.asok 2021-02-10T11:24:15.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:15.294 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44954.asok status 2021-02-10T11:24:15.709 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:24:15.709 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "44954", 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5052, 2021-02-10T11:24:15.710 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 5052 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1572107307 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:24:15.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1572107307 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.5052 172.21.2.117:0/1572107307", 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1572107307", 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 12, 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 31, 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:24:15.712 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:24:15.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:15.714 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:15.714 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:16.062 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:16.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:16.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:16.063 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:16.408 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:24:16.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 Processor -- start 2021-02-10T11:24:16.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 -- start start 2021-02-10T11:24:16.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f34079000 0x7f3f340e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f340d1d10 con 0x7f3f3407da20 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f34077a70 con 0x7f3f341b00e0 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.821 7f3f3b303700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f340e53b0 con 0x7f3f34079000 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3911f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3f3407da20 0x7f3f340771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43184/0 (socket says 172.21.2.117:43184) 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3891e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f34079000 0x7f3f340e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3911f700 1 -- 172.21.2.117:0/1626895878 learned_addr learned my addr 172.21.2.117:0/1626895878 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3891e700 1 -- 172.21.2.117:0/1626895878 >> v1:172.21.2.125:6789/0 conn(0x7f3f341b00e0 legacy=0x7f3f340d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f27fff700 1 -- 172.21.2.117:0/1626895878 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281707878 0 0) 0x7f3f2c000f90 con 0x7f3f341b00e0 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3891e700 1 -- 172.21.2.117:0/1626895878 >> v1:172.21.2.117:6789/0 conn(0x7f3f3407da20 legacy=0x7f3f340771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.822 7f3f3891e700 1 -- 172.21.2.117:0/1626895878 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34079400 con 0x7f3f34079000 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3891e700 1 --2- 172.21.2.117:0/1626895878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f34079000 0x7f3f340e4d90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3f30006650 tx=0x7f3f30009a80).ready entity=mon.0 client_cookie=c0400ee85a29e6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f27fff700 1 -- 172.21.2.117:0/1626895878 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3000c0c0 con 0x7f3f34079000 2021-02-10T11:24:16.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f27fff700 1 -- 172.21.2.117:0/1626895878 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3000c340 con 0x7f3f34079000 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f27fff700 1 -- 172.21.2.117:0/1626895878 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3000c7a0 con 0x7f3f34079000 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3b303700 1 -- 172.21.2.117:0/1626895878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f34079000 msgr2=0x7f3f340e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3b303700 1 --2- 172.21.2.117:0/1626895878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f34079000 0x7f3f340e4d90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3f30006650 tx=0x7f3f30009a80).stop 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3b303700 1 -- 172.21.2.117:0/1626895878 shutdown_connections 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3b303700 1 --2- 172.21.2.117:0/1626895878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f34079000 0x7f3f340e4d90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.823 7f3f3b303700 1 -- 172.21.2.117:0/1626895878 >> 172.21.2.117:0/1626895878 conn(0x7f3f340ce440 msgr2=0x7f3f340e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.824 7f3f3b303700 1 -- 172.21.2.117:0/1626895878 shutdown_connections 2021-02-10T11:24:16.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.824 7f3f3b303700 1 -- 172.21.2.117:0/1626895878 wait complete. 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.838 7f3f3b303700 1 Processor -- start 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.838 7f3f3b303700 1 -- start start 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f340af200 0x7f3f340af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f341a82a0 0x7f3f341a86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f341983f0 con 0x7f3f341a8cc0 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34106940 con 0x7f3f340af200 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3b303700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34106b60 con 0x7f3f341a82a0 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42944/0 (socket says 172.21.2.117:42944) 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 -- 172.21.2.117:0/3355981844 learned_addr learned my addr 172.21.2.117:0/3355981844 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3891e700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f341a82a0 0x7f3f341a86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f3911f700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f340af200 0x7f3f340af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 -- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f341a82a0 msgr2=0x7f3f341a86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f341a82a0 0x7f3f341a86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 -- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f340af200 msgr2=0x7f3f340af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f340af200 0x7f3f340af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.839 7f3f39920700 1 -- 172.21.2.117:0/3355981844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34106d80 con 0x7f3f341a8cc0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f39920700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3f2c00b920 tx=0x7f3f2c00bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f2c005840 con 0x7f3f341a8cc0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c005ac0 con 0x7f3f341a8cc0 2021-02-10T11:24:16.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f2c006030 con 0x7f3f341a8cc0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f34106d80 con 0x7f3f341a8cc0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.840 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f34107310 con 0x7f3f341a8cc0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.841 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f2c006030 con 0x7f3f341a8cc0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.841 7f3f25ffb700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 0x7f3f1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.841 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f2c032a90 con 0x7f3f341a8cc0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.841 7f3f3911f700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 0x7f3f1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:16.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.842 7f3f3911f700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 0x7f3f1401abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:16.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.872 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f34104a20 con 0x7f3f341a8cc0 2021-02-10T11:24:16.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.874 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f2c006240 con 0x7f3f341a8cc0 2021-02-10T11:24:16.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.998 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f34077a70 con 0x7f3f341a8cc0 2021-02-10T11:24:16.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:16.999 7f3f25ffb700 1 -- 172.21.2.117:0/3355981844 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f3f2c006030 con 0x7f3f341a8cc0 2021-02-10T11:24:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:16.996 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":31,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:02.664299","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:24:16.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.000 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 msgr2=0x7f3f1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.000 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 0x7f3f1401abc0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.000 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 msgr2=0x7f3f341a90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:16.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.000 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3f2c00b920 tx=0x7f3f2c00bcc0).stop 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 shutdown_connections 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f140186c0 0x7f3f1401abc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f341a8cc0 0x7f3f341a90c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f341a82a0 0x7f3f341a86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 --2- 172.21.2.117:0/3355981844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f340af200 0x7f3f340af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 >> 172.21.2.117:0/3355981844 conn(0x7f3f3418cd60 msgr2=0x7f3f34187be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 shutdown_connections 2021-02-10T11:24:16.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.001 7f3f3b303700 1 -- 172.21.2.117:0/3355981844 wait complete. 2021-02-10T11:24:17.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:23:59.642031', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:00.660978', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:24:17.006 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:24:17.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.422 7f80decec700 1 Processor -- start 2021-02-10T11:24:17.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 -- start start 2021-02-10T11:24:17.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f80d8079000 0x7f80d80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81b00f0 0x7f80d81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d8077a70 con 0x7f80d807da20 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d80e53c0 con 0x7f80d8079000 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80decec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8079400 con 0x7f80d81b00f0 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80dcb08700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f80d807da20 0x7f80d80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43198/0 (socket says 172.21.2.117:43198) 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80dcb08700 1 -- 172.21.2.117:0/4045564054 learned_addr learned my addr 172.21.2.117:0/4045564054 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.423 7f80d7fff700 1 --2- 172.21.2.117:0/4045564054 >> v2:172.21.2.125:3300/0 conn(0x7f80d8079000 0x7f80d80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80dd309700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81b00f0 0x7f80d81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d77fe700 1 -- 172.21.2.117:0/4045564054 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316189692 0 0) 0x7f80c8001180 con 0x7f80d807da20 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d77fe700 1 -- 172.21.2.117:0/4045564054 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c4002fe0 con 0x7f80d807da20 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d7fff700 1 -- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81b00f0 msgr2=0x7f80d81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d7fff700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81b00f0 0x7f80d81b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d7fff700 1 -- 172.21.2.117:0/4045564054 >> v1:172.21.2.117:6789/0 conn(0x7f80d807da20 legacy=0x7f80d80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d7fff700 1 -- 172.21.2.117:0/4045564054 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d807de20 con 0x7f80d8079000 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d7fff700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8079000 0x7f80d80e4da0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f80c0006650 tx=0x7f80c0009a80).ready entity=mon.1 client_cookie=91d1e1a50d2509d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d77fe700 1 -- 172.21.2.117:0/4045564054 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c000c0c0 con 0x7f80d8079000 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d77fe700 1 -- 172.21.2.117:0/4045564054 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c000c340 con 0x7f80d8079000 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80d77fe700 1 -- 172.21.2.117:0/4045564054 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80c00009b0 con 0x7f80d8079000 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80decec700 1 -- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8079000 msgr2=0x7f80d80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.424 7f80decec700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8079000 0x7f80d80e4da0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f80c0006650 tx=0x7f80c0009a80).stop 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 -- 172.21.2.117:0/4045564054 shutdown_connections 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81b00f0 0x7f80d81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 --2- 172.21.2.117:0/4045564054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8079000 0x7f80d80e4da0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 -- 172.21.2.117:0/4045564054 >> 172.21.2.117:0/4045564054 conn(0x7f80d80ce450 msgr2=0x7f80d80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 -- 172.21.2.117:0/4045564054 shutdown_connections 2021-02-10T11:24:17.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.425 7f80decec700 1 -- 172.21.2.117:0/4045564054 wait complete. 2021-02-10T11:24:17.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 Processor -- start 2021-02-10T11:24:17.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 -- start start 2021-02-10T11:24:17.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81a0250 0x7f80d8106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8187e20 0x7f80d8188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d81067f0 con 0x7f80d8090420 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d8107350 con 0x7f80d8187e20 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.440 7f80decec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d81042d0 con 0x7f80d81a0250 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42958/0 (socket says 172.21.2.117:42958) 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 -- 172.21.2.117:0/3089095588 learned_addr learned my addr 172.21.2.117:0/3089095588 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 -- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81a0250 msgr2=0x7f80d8106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81a0250 0x7f80d8106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 -- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8187e20 msgr2=0x7f80d8188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8187e20 0x7f80d8188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 -- 172.21.2.117:0/3089095588 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d8090e40 con 0x7f80d8090420 2021-02-10T11:24:17.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.441 7f80dd309700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f80d000b970 tx=0x7f80d000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.442 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d0000f90 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.442 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0006aa0 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.442 7f80decec700 1 -- 172.21.2.117:0/3089095588 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d8090e40 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.442 7f80decec700 1 -- 172.21.2.117:0/3089095588 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d80913d0 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.442 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d0001680 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.443 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80d00011f0 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.443 7f80d57fa700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 0x7f80bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.443 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f80d0032a40 con 0x7f80d8090420 2021-02-10T11:24:17.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.443 7f80dcb08700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 0x7f80bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:17.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.444 7f80dcb08700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 0x7f80bc01abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:17.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.472 7f80decec700 1 -- 172.21.2.117:0/3089095588 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d8103280 con 0x7f80d8090420 2021-02-10T11:24:17.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.475 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80d002acd0 con 0x7f80d8090420 2021-02-10T11:24:17.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.601 7f80decec700 1 -- 172.21.2.117:0/3089095588 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f80d8077a70 con 0x7f80d8090420 2021-02-10T11:24:17.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.602 7f80d57fa700 1 -- 172.21.2.117:0/3089095588 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f80d002acd0 con 0x7f80d8090420 2021-02-10T11:24:17.599 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 msgr2=0x7f80bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 0x7f80bc01abc0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 msgr2=0x7f80d8090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f80d000b970 tx=0x7f80d000bd10).stop 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 shutdown_connections 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80bc0186c0 0x7f80bc01abc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80d8090420 0x7f80d8090820 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80d8187e20 0x7f80d8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 --2- 172.21.2.117:0/3089095588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80d81a0250 0x7f80d8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 >> 172.21.2.117:0/3089095588 conn(0x7f80d80a1850 msgr2=0x7f80d80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 shutdown_connections 2021-02-10T11:24:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:17.604 7f80decec700 1 -- 172.21.2.117:0/3089095588 wait complete. 2021-02-10T11:24:17.609 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.023 7f2872fdd700 1 Processor -- start 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.023 7f2872fdd700 1 -- start start 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2872fdd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c079000 0x7f286c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2872fdd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f286c1b00f0 0x7f286c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2872fdd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286c077a70 con 0x7f286c07da20 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2872fdd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286c0e53c0 con 0x7f286c1b00f0 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2872fdd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c079400 con 0x7f286c079000 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2870df9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f286c07da20 0x7f286c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43212/0 (socket says 172.21.2.117:43212) 2021-02-10T11:24:18.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f2870df9700 1 -- 172.21.2.117:0/3388510719 learned_addr learned my addr 172.21.2.117:0/3388510719 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913599143 0 0) 0x7f2854001180 con 0x7f286c07da20 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.024 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2858002fe0 con 0x7f286c07da20 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3365808139 0 0) 0x7f2854001180 con 0x7f286c07da20 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c079000 msgr2=0x7f286c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> v2:172.21.2.125:3300/0 conn(0x7f286c1b00f0 0x7f286c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286bfff700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c079000 0x7f286c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c079000 0x7f286c0e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 >> v2:172.21.2.125:3300/0 conn(0x7f286c1b00f0 msgr2=0x7f286c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 --2- 172.21.2.117:0/3388510719 >> v2:172.21.2.125:3300/0 conn(0x7f286c1b00f0 0x7f286c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c07de20 con 0x7f286c07da20 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2854001680 con 0x7f286c07da20 2021-02-10T11:24:18.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 >> v1:172.21.2.117:6789/0 conn(0x7f286c07da20 legacy=0x7f286c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2858003b90 0x7f2858006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28580068e0 0x7f2858008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2858009670 0x7f285800bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285800c420 con 0x7f2858009670 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285800c640 con 0x7f2858003b90 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285800c860 con 0x7f28580068e0 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286bfff700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c079000 0x7f286c0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> v2:172.21.2.125:3300/0 conn(0x7f286c1b00f0 0x7f286c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2858009670 0x7f285800bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f2870df9700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2858003b90 0x7f2858006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.025 7f286bfff700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28580068e0 0x7f2858008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 -- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28580068e0 msgr2=0x7f2858008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28580068e0 0x7f2858008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 -- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2858003b90 msgr2=0x7f2858006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2858003b90 0x7f2858006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 -- 172.21.2.117:0/3388510719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285800cb00 con 0x7f2858009670 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2858009670 0x7f285800bb70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f2864006cc0 tx=0x7f286400b300).ready entity=mon.0 client_cookie=b3cc05f8600aa8f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f286400b6f0 con 0x7f2858009670 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2864003030 con 0x7f2858009670 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f286b7fe700 1 -- 172.21.2.117:0/3388510719 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f286400bcb0 con 0x7f2858009670 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f2872fdd700 1 -- 172.21.2.117:0/3388510719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2858009670 msgr2=0x7f285800bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f2872fdd700 1 --2- 172.21.2.117:0/3388510719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2858009670 0x7f285800bb70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f2864006cc0 tx=0x7f286400b300).stop 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.026 7f28715fa700 1 -- 172.21.2.117:0/3388510719 reap_dead start 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.027 7f2872fdd700 1 -- 172.21.2.117:0/3388510719 shutdown_connections 2021-02-10T11:24:18.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.027 7f2872fdd700 1 -- 172.21.2.117:0/3388510719 >> 172.21.2.117:0/3388510719 conn(0x7f286c0ce450 msgr2=0x7f286c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.027 7f2872fdd700 1 -- 172.21.2.117:0/3388510719 shutdown_connections 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.027 7f2872fdd700 1 -- 172.21.2.117:0/3388510719 wait complete. 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.041 7f2872fdd700 1 Processor -- start 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 -- start start 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c192ed0 0x7f286c1a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 0x7f286c09f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1057f0 con 0x7f286c09f4c0 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1a46e0 con 0x7f286c0a0980 2021-02-10T11:24:18.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2872fdd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286c1a4900 con 0x7f286c192ed0 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2870df9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f28715fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 0x7f286c09f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f286bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c192ed0 0x7f286c1a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f28715fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 0x7f286c09f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42976/0 (socket says 172.21.2.117:42976) 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f2870df9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:47358/0 (socket says 172.21.2.117:47358) 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.042 7f28715fa700 1 -- 172.21.2.117:0/2372650563 learned_addr learned my addr 172.21.2.117:0/2372650563 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 -- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c192ed0 msgr2=0x7f286c1a40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c192ed0 0x7f286c1a40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 -- 172.21.2.117:0/2372650563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 msgr2=0x7f286c09f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 0x7f286c09f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 -- 172.21.2.117:0/2372650563 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c106130 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2870df9700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2854007fc0 tx=0x7f2854009890).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2854009c80 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28540073f0 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2854004590 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286c106130 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.043 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286c106130 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.044 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2854008000 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.044 7f28697fa700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 0x7f285001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.044 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28540323d0 con 0x7f286c0a0980 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.045 7f286bfff700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 0x7f285001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.045 7f286bfff700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 0x7f285001abc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.074 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286c1057f0 con 0x7f286c0a0980 2021-02-10T11:24:18.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.077 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2854000990 con 0x7f286c0a0980 2021-02-10T11:24:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.207 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f286c1057f0 con 0x7f286c0a0980 2021-02-10T11:24:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f28697fa700 1 -- 172.21.2.117:0/2372650563 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v31) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2854008000 con 0x7f286c0a0980 2021-02-10T11:24:18.204 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 msgr2=0x7f285001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 0x7f285001abc0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 msgr2=0x7f286c1051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2854007fc0 tx=0x7f2854009890).stop 2021-02-10T11:24:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 shutdown_connections 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.209 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28500186c0 0x7f285001abc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286c09f4c0 0x7f286c09f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f286c192ed0 0x7f286c1a40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 --2- 172.21.2.117:0/2372650563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f286c0a0980 0x7f286c1051d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 >> 172.21.2.117:0/2372650563 conn(0x7f286c09d920 msgr2=0x7f286c103820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 shutdown_connections 2021-02-10T11:24:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.210 7f2872fdd700 1 -- 172.21.2.117:0/2372650563 wait complete. 2021-02-10T11:24:18.214 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:24:18.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.630 7fd244116700 1 Processor -- start 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.630 7fd244116700 1 -- start start 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd244116700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd23c07da20 0x7fd23c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd244116700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c0e4da0 0x7fd23c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd244116700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c077a70 con 0x7fd23c0e51a0 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd244116700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd23c0777d0 con 0x7fd23c07da20 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd244116700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c07de20 con 0x7fd23c0e4da0 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd23c07da20 0x7fd23c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd23c07da20 0x7fd23c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42982/0 (socket says 172.21.2.117:42982) 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 -- 172.21.2.117:0/3456355791 learned_addr learned my addr 172.21.2.117:0/3456355791 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 -- 172.21.2.117:0/3456355791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c0e4da0 msgr2=0x7fd23c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 --2- 172.21.2.117:0/3456355791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c0e4da0 0x7fd23c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 -- 172.21.2.117:0/3456355791 >> v1:172.21.2.125:6789/0 conn(0x7fd23c0e51a0 legacy=0x7fd23c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.631 7fd241f32700 1 -- 172.21.2.117:0/3456355791 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c1b0710 con 0x7fd23c07da20 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd241f32700 1 --2- 172.21.2.117:0/3456355791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c07da20 0x7fd23c0771b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fd238006700 tx=0x7fd238009b80).ready entity=mon.0 client_cookie=ab50455398b0c34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd240f30700 1 -- 172.21.2.117:0/3456355791 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23800c0c0 con 0x7fd23c07da20 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd240f30700 1 -- 172.21.2.117:0/3456355791 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23800c400 con 0x7fd23c07da20 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd240f30700 1 -- 172.21.2.117:0/3456355791 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23800c920 con 0x7fd23c07da20 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd244116700 1 -- 172.21.2.117:0/3456355791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c07da20 msgr2=0x7fd23c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.632 7fd244116700 1 --2- 172.21.2.117:0/3456355791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c07da20 0x7fd23c0771b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fd238006700 tx=0x7fd238009b80).stop 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 -- 172.21.2.117:0/3456355791 shutdown_connections 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 --2- 172.21.2.117:0/3456355791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c0e4da0 0x7fd23c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 --2- 172.21.2.117:0/3456355791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c07da20 0x7fd23c0771b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 -- 172.21.2.117:0/3456355791 >> 172.21.2.117:0/3456355791 conn(0x7fd23c0ce450 msgr2=0x7fd23c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 -- 172.21.2.117:0/3456355791 shutdown_connections 2021-02-10T11:24:18.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.633 7fd244116700 1 -- 172.21.2.117:0/3456355791 wait complete. 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.647 7fd244116700 1 Processor -- start 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.647 7fd244116700 1 -- start start 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 0x7fd23c094f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c095560 0x7fd23c095960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0942a0 con 0x7fd23c1a4600 2021-02-10T11:24:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0ac4d0 con 0x7fd23c094b40 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd244116700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0ac6f0 con 0x7fd23c095560 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:42984/0 (socket says 172.21.2.117:42984) 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 -- 172.21.2.117:0/3795817764 learned_addr learned my addr 172.21.2.117:0/3795817764 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd242733700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c095560 0x7fd23c095960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241731700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 0x7fd23c094f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 -- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c095560 msgr2=0x7fd23c095960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c095560 0x7fd23c095960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 -- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 msgr2=0x7fd23c094f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 0x7fd23c094f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.648 7fd241f32700 1 -- 172.21.2.117:0/3795817764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c0ac910 con 0x7fd23c1a4600 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd241731700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 0x7fd23c094f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:18.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd241f32700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd238006420 tx=0x7fd23800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23800bd50 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2380090d0 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd244116700 1 -- 172.21.2.117:0/3795817764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c0ac910 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2380054d0 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.649 7fd244116700 1 -- 172.21.2.117:0/3795817764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c0ac910 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.650 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd238009440 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.650 7fd2327fc700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 0x7fd22401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.650 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd238032780 con 0x7fd23c1a4600 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.650 7fd241731700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 0x7fd22401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:18.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.651 7fd241731700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 0x7fd22401ac10 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:18.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.681 7fd244116700 1 -- 172.21.2.117:0/3795817764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23c103310 con 0x7fd23c1a4600 2021-02-10T11:24:18.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.683 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd238009030 con 0x7fd23c1a4600 2021-02-10T11:24:18.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.808 7fd244116700 1 -- 172.21.2.117:0/3795817764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd23c077a70 con 0x7fd23c1a4600 2021-02-10T11:24:18.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.809 7fd2327fc700 1 -- 172.21.2.117:0/3795817764 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v31) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd23800bd50 con 0x7fd23c1a4600 2021-02-10T11:24:18.806 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:24:18.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 msgr2=0x7fd22401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 0x7fd22401ac10 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 msgr2=0x7fd23c093c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:18.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fd238006420 tx=0x7fd23800b960).stop 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 shutdown_connections 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd23c095560 0x7fd23c095960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd23c094b40 0x7fd23c094f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd224018730 0x7fd22401ac10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 --2- 172.21.2.117:0/3795817764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd23c1a4600 0x7fd23c093c80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 >> 172.21.2.117:0/3795817764 conn(0x7fd23c0a1880 msgr2=0x7fd23c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 shutdown_connections 2021-02-10T11:24:18.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:18.811 7fd244116700 1 -- 172.21.2.117:0/3795817764 wait complete. 2021-02-10T11:24:18.816 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:24:18.816 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.227 7f9ddb39f700 1 Processor -- start 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.227 7f9ddb39f700 1 -- start start 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9ddb39f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 0x7f9dd40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9ddb39f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 0x7f9dd40d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9ddb39f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd4077a70 con 0x7f9dd40e4d90 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9ddb39f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40e5c10 con 0x7f9dd40e5190 2021-02-10T11:24:19.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9ddb39f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40777d0 con 0x7f9dd407da20 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9dd89ba700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9dd40e4d90 0x7f9dd41b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43242/0 (socket says 172.21.2.117:43242) 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9dd89ba700 1 -- 172.21.2.117:0/4060941829 learned_addr learned my addr 172.21.2.117:0/4060941829 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9dd99bc700 1 --2- 172.21.2.117:0/4060941829 >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 0x7f9dd40d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9dd91bb700 1 --2- 172.21.2.117:0/4060941829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 0x7f9dd40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.228 7f9dcbfff700 1 -- 172.21.2.117:0/4060941829 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680276649 0 0) 0x7f9dc40009b0 con 0x7f9dd40e4d90 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dcbfff700 1 -- 172.21.2.117:0/4060941829 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc0002fe0 con 0x7f9dd40e4d90 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd91bb700 1 -- 172.21.2.117:0/4060941829 >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 msgr2=0x7f9dd40d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd91bb700 1 --2- 172.21.2.117:0/4060941829 >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 0x7f9dd40d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd91bb700 1 -- 172.21.2.117:0/4060941829 >> v1:172.21.2.117:6789/0 conn(0x7f9dd40e4d90 legacy=0x7f9dd41b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd91bb700 1 -- 172.21.2.117:0/4060941829 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41b08d0 con 0x7f9dd407da20 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd99bc700 1 --2- 172.21.2.117:0/4060941829 >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 0x7f9dd40d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:19.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dd91bb700 1 --2- 172.21.2.117:0/4060941829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 0x7f9dd40771b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f9dd0006650 tx=0x7f9dd0009a80).ready entity=mon.2 client_cookie=df0af0a79252fdd7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dcbfff700 1 -- 172.21.2.117:0/4060941829 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd00009b0 con 0x7f9dd407da20 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dcbfff700 1 -- 172.21.2.117:0/4060941829 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd000c0c0 con 0x7f9dd407da20 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 -- 172.21.2.117:0/4060941829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 msgr2=0x7f9dd40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 --2- 172.21.2.117:0/4060941829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 0x7f9dd40771b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f9dd0006650 tx=0x7f9dd0009a80).stop 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.229 7f9dcbfff700 1 -- 172.21.2.117:0/4060941829 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dd000c5e0 con 0x7f9dd407da20 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 -- 172.21.2.117:0/4060941829 shutdown_connections 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 --2- 172.21.2.117:0/4060941829 >> v2:172.21.2.125:3300/0 conn(0x7f9dd40e5190 0x7f9dd40d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 --2- 172.21.2.117:0/4060941829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd407da20 0x7f9dd40771b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 -- 172.21.2.117:0/4060941829 >> 172.21.2.117:0/4060941829 conn(0x7f9dd40ce440 msgr2=0x7f9dd40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 -- 172.21.2.117:0/4060941829 shutdown_connections 2021-02-10T11:24:19.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.230 7f9ddb39f700 1 -- 172.21.2.117:0/4060941829 wait complete. 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.245 7f9ddb39f700 1 Processor -- start 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.245 7f9ddb39f700 1 -- start start 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.245 7f9ddb39f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd41a8760 0x7f9dd41a8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9ddb39f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 0x7f9dd41a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9ddb39f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9ddb39f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd41a91b0 con 0x7f9dd41a1a50 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9ddb39f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4106a00 con 0x7f9dd41a1650 2021-02-10T11:24:19.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9ddb39f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4106ec0 con 0x7f9dd41a8760 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43000/0 (socket says 172.21.2.117:43000) 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 -- 172.21.2.117:0/106447905 learned_addr learned my addr 172.21.2.117:0/106447905 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd89ba700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 0x7f9dd41a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 -- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd41a8760 msgr2=0x7f9dd41a8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd41a8760 0x7f9dd41a8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 -- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 msgr2=0x7f9dd41a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 0x7f9dd41a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 -- 172.21.2.117:0/106447905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41aa370 con 0x7f9dd41a1a50 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd89ba700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 0x7f9dd41a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.246 7f9dd99bc700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9dcc00b970 tx=0x7f9dcc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dcc000f90 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd41aa370 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00be50 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dcc006aa0 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd41aa370 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9dcc006aa0 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 0x7f9db401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.247 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9dcc032b10 con 0x7f9dd41a1a50 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.248 7f9dd91bb700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 0x7f9db401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:19.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.248 7f9dd91bb700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 0x7f9db401ac10 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:19.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.278 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd41b4f30 con 0x7f9dd41a1a50 2021-02-10T11:24:19.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.280 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9dcc01f880 con 0x7f9dd41a1a50 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.407 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd4077a70 con 0x7f9dd41a1a50 2021-02-10T11:24:19.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.408 7f9dc9ffb700 1 -- 172.21.2.117:0/106447905 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2097 (secure 0 0 0) 0x7f9dcc006aa0 con 0x7f9dd41a1a50 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:24:19.405 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 12 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.409 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 msgr2=0x7f9db401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.409 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 0x7f9db401ac10 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 msgr2=0x7f9dd41a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:19.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9dcc00b970 tx=0x7f9dcc00bd10).stop 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 shutdown_connections 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9db4018730 0x7f9db401ac10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9dd41a1a50 0x7f9dd41a9d50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9dd41a1650 0x7f9dd41a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 --2- 172.21.2.117:0/106447905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9dd41a8760 0x7f9dd41a8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 >> 172.21.2.117:0/106447905 conn(0x7f9dd418fed0 msgr2=0x7f9dd4107320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 shutdown_connections 2021-02-10T11:24:19.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:19.410 7f9ddb39f700 1 -- 172.21.2.117:0/106447905 wait complete. 2021-02-10T11:24:19.616 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:01.662717', 'modified': '2021-02-10 11:24:04.035038', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4412': {'gid': 4412, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.125:6835/3266045438', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3266045438}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3266045438}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:24:19.616 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:24:19.617 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4412, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.354363011, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 12, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 31, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 31, 2021-02-10T11:24:20.015 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 27.615823637999998 2021-02-10T11:24:20.016 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:24:20.019 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4412, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.354363011, "mdsmap_epoch": 12, "osdmap_epoch": 31, "osdmap_epoch_barrier": 31, "uptime": 27.615823637999998 } 2021-02-10T11:24:20.019 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:24:20.019 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:24:20.019 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:24:20.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:20.019 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:20.355 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:24:20.698 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:20.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:21.040 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:21.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:21.041 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:24:21.375 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:24:21.375 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:24:21.375 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:21.378 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:24:21.721 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:24:21.721 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:24:21.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[45594]: starting ceph client 2021-02-10T11:24:21.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:24:21.791 7f3a06a47f80 -1 init, newargv = 0x55c11570a700 newargc=9 2021-02-10T11:24:21.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[45594]: starting fuse 2021-02-10T11:24:22.063 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:24:22.064 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:24:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:24:22.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:22.065 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:24:22.398 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:24:22.398 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:24:22.399 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:24:22.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:22.399 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:24:22.400 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:24:22.401 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:24:22.760 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.45594.asok 2021-02-10T11:24:22.763 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45594.asok 2021-02-10T11:24:22.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:22.763 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45594.asok status 2021-02-10T11:24:23.151 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:24:23.151 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "45594", 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5088, 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:24:23.152 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 5088 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 383915257 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 383915257 2021-02-10T11:24:23.153 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.5088 172.21.2.117:0/383915257", 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/383915257", 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 12, 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 31, 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:24:23.154 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:24:23.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:23.156 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:23.156 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:23.491 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:23.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:24:23.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:23.492 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:24:23.501 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:23.538 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:23.838 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:24:24.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 Processor -- start 2021-02-10T11:24:24.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 -- start start 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faba4078fa0 0x7faba40e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faba41b0130 0x7faba41b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba4077a10 con 0x7faba407da20 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40e5c60 con 0x7faba41b0130 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7fabaa547700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40e5420 con 0x7faba4078fa0 2021-02-10T11:24:24.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faba41b0130 0x7faba41b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faba41b0130 0x7faba41b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43016/0 (socket says 172.21.2.117:43016) 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 -- 172.21.2.117:0/1802640776 learned_addr learned my addr 172.21.2.117:0/1802640776 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 -- 172.21.2.117:0/1802640776 >> v2:172.21.2.125:3300/0 conn(0x7faba4078fa0 msgr2=0x7faba40e4e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 --2- 172.21.2.117:0/1802640776 >> v2:172.21.2.125:3300/0 conn(0x7faba4078fa0 0x7faba40e4e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.254 7faba8b64700 1 -- 172.21.2.117:0/1802640776 >> v1:172.21.2.117:6789/0 conn(0x7faba407da20 legacy=0x7faba40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7faba8b64700 1 -- 172.21.2.117:0/1802640776 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40793a0 con 0x7faba41b0130 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7faba8b64700 1 --2- 172.21.2.117:0/1802640776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba41b0130 0x7faba41b0530 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fab9c006f00 tx=0x7fab9c009f10).ready entity=mon.0 client_cookie=f8e31116a3658715 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7faba2ffd700 1 -- 172.21.2.117:0/1802640776 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9c00b3b0 con 0x7faba41b0130 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7faba2ffd700 1 -- 172.21.2.117:0/1802640776 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c00b630 con 0x7faba41b0130 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7faba2ffd700 1 -- 172.21.2.117:0/1802640776 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9c00bb50 con 0x7faba41b0130 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 -- 172.21.2.117:0/1802640776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba41b0130 msgr2=0x7faba41b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 --2- 172.21.2.117:0/1802640776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba41b0130 0x7faba41b0530 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fab9c006f00 tx=0x7fab9c009f10).stop 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 -- 172.21.2.117:0/1802640776 shutdown_connections 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 --2- 172.21.2.117:0/1802640776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba41b0130 0x7faba41b0530 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 --2- 172.21.2.117:0/1802640776 >> v2:172.21.2.125:3300/0 conn(0x7faba4078fa0 0x7faba40e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.255 7fabaa547700 1 -- 172.21.2.117:0/1802640776 >> 172.21.2.117:0/1802640776 conn(0x7faba40ce480 msgr2=0x7faba40e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.256 7fabaa547700 1 -- 172.21.2.117:0/1802640776 shutdown_connections 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.256 7fabaa547700 1 -- 172.21.2.117:0/1802640776 wait complete. 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.270 7fabaa547700 1 Processor -- start 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.270 7fabaa547700 1 -- start start 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.270 7fabaa547700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faba40c3d40 0x7faba40bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.270 7fabaa547700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.270 7fabaa547700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faba40a8390 0x7faba40a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7fabaa547700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40c3260 con 0x7faba40c2e60 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7fabaa547700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40c3480 con 0x7faba40a8390 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7fabaa547700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a3130 con 0x7faba40c3d40 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43020/0 (socket says 172.21.2.117:43020) 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 -- 172.21.2.117:0/37161976 learned_addr learned my addr 172.21.2.117:0/37161976 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 -- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faba40c3d40 msgr2=0x7faba40bf5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faba40c3d40 0x7faba40bf5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 -- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faba40a8390 msgr2=0x7faba40a2b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faba40a8390 0x7faba40a2b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 -- 172.21.2.117:0/37161976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40a3350 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba37fe700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fab9800b5c0 tx=0x7fab9800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab9800bd50 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98006030 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab980054d0 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.271 7fabaa547700 1 -- 172.21.2.117:0/37161976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba40a3350 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.272 7fabaa547700 1 -- 172.21.2.117:0/37161976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba40a3350 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.272 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab98006210 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.272 7faba0ff9700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.272 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fab980326e0 con 0x7faba40c2e60 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.273 7faba3fff700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.273 7faba3fff700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 0x7fab8401abc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.304 7fabaa547700 1 -- 172.21.2.117:0/37161976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba40c3810 con 0x7faba40c2e60 2021-02-10T11:24:24.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.306 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab98000c40 con 0x7faba40c2e60 2021-02-10T11:24:24.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.442 7fabaa547700 1 -- 172.21.2.117:0/37161976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faba40c3810 con 0x7faba40c2e60 2021-02-10T11:24:24.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.496 7faba0ff9700 1 -- 172.21.2.117:0/37161976 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v13) v1 ==== 136+0+0 (secure 0 0 0) 0x7fab98006210 con 0x7faba40c2e60 2021-02-10T11:24:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 msgr2=0x7fab8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 0x7fab8401abc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 msgr2=0x7faba40a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fab9800b5c0 tx=0x7fab9800b960).stop 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 shutdown_connections 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab840186c0 0x7fab8401abc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faba40c2e60 0x7faba40a7d70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faba40a8390 0x7faba40a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 --2- 172.21.2.117:0/37161976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faba40c3d40 0x7faba40bf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 >> 172.21.2.117:0/37161976 conn(0x7faba40c0520 msgr2=0x7faba4090810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 shutdown_connections 2021-02-10T11:24:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.509 7fabaa547700 1 -- 172.21.2.117:0/37161976 wait complete. 2021-02-10T11:24:24.514 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 Processor -- start 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 -- start start 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe4000d1510 0x7fe4000d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe400077a70 con 0x7fe40007da20 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe400079400 con 0x7fe400079000 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.927 7fe4055a8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40007de20 con 0x7fe4000d1510 2021-02-10T11:24:24.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ff2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe40007da20 0x7fe4000771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43278/0 (socket says 172.21.2.117:43278) 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ff2bf700 1 -- 172.21.2.117:0/2763026720 learned_addr learned my addr 172.21.2.117:0/2763026720 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ffac0700 1 --2- 172.21.2.117:0/2763026720 >> v2:172.21.2.125:3300/0 conn(0x7fe4000d1510 0x7fe4000d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710888699 0 0) 0x7fe3f00009b0 con 0x7fe40007da20 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e8002fe0 con 0x7fe40007da20 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724044268 0 0) 0x7fe3f4000ff0 con 0x7fe400079000 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e80042c0 con 0x7fe400079000 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ffac0700 1 -- 172.21.2.117:0/2763026720 >> v1:172.21.2.125:6789/0 conn(0x7fe400079000 legacy=0x7fe4000e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ffac0700 1 -- 172.21.2.117:0/2763026720 >> v1:172.21.2.117:6789/0 conn(0x7fe40007da20 legacy=0x7fe4000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ffac0700 1 -- 172.21.2.117:0/2763026720 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4001b00d0 con 0x7fe4000d1510 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.928 7fe3ffac0700 1 --2- 172.21.2.117:0/2763026720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4000d1510 0x7fe4000d1910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe3f8006e50 tx=0x7fe3f8009df0).ready entity=mon.1 client_cookie=f0eba4df397f8eb8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3f800b510 con 0x7fe4000d1510 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f8003030 con 0x7fe4000d1510 2021-02-10T11:24:24.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe3fe2bd700 1 -- 172.21.2.117:0/2763026720 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3f800b7d0 con 0x7fe4000d1510 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe4055a8700 1 -- 172.21.2.117:0/2763026720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4000d1510 msgr2=0x7fe4000d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe4055a8700 1 --2- 172.21.2.117:0/2763026720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4000d1510 0x7fe4000d1910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe3f8006e50 tx=0x7fe3f8009df0).stop 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe4055a8700 1 -- 172.21.2.117:0/2763026720 shutdown_connections 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe4055a8700 1 --2- 172.21.2.117:0/2763026720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4000d1510 0x7fe4000d1910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.929 7fe4055a8700 1 -- 172.21.2.117:0/2763026720 >> 172.21.2.117:0/2763026720 conn(0x7fe4000ce430 msgr2=0x7fe4000e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.930 7fe4055a8700 1 -- 172.21.2.117:0/2763026720 shutdown_connections 2021-02-10T11:24:24.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.930 7fe4055a8700 1 -- 172.21.2.117:0/2763026720 wait complete. 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 Processor -- start 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 -- start start 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 0x7fe4000a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 0x7fe4001a8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4001a8d70 con 0x7fe4000a8650 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000a9070 con 0x7fe4001a8350 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.944 7fe4055a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4000a9250 con 0x7fe4000b8cc0 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43036/0 (socket says 172.21.2.117:43036) 2021-02-10T11:24:24.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 -- 172.21.2.117:0/951405347 learned_addr learned my addr 172.21.2.117:0/951405347 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ff2bf700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 0x7fe4000a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3feabe700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 0x7fe4001a8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 -- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 msgr2=0x7fe4000a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 0x7fe4000a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 -- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 msgr2=0x7fe4001a8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 0x7fe4001a8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 -- 172.21.2.117:0/951405347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4000a9470 con 0x7fe4000a8650 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ff2bf700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 0x7fe4000a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3feabe700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 0x7fe4001a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.945 7fe3ffac0700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fe3f8006fa0 tx=0x7fe3f800bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3f80096b0 con 0x7fe4000a8650 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f80082e0 con 0x7fe4000a8650 2021-02-10T11:24:24.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe4055a8700 1 -- 172.21.2.117:0/951405347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4000a9470 con 0x7fe4000a8650 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3f8008840 con 0x7fe4000a8650 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe4055a8700 1 -- 172.21.2.117:0/951405347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4000a9470 con 0x7fe4000a8650 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe3f80084c0 con 0x7fe4000a8650 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.946 7fe3e7fff700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 0x7fe3e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.947 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe3f80328e0 con 0x7fe4000a8650 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.947 7fe3ff2bf700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 0x7fe3e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:24.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.947 7fe3ff2bf700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 0x7fe3e001abc0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:24.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.977 7fe4055a8700 1 -- 172.21.2.117:0/951405347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe400103260 con 0x7fe4000a8650 2021-02-10T11:24:24.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:24.980 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe3f802ffb0 con 0x7fe4000a8650 2021-02-10T11:24:25.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.114 7fe4055a8700 1 -- 172.21.2.117:0/951405347 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fe400103260 con 0x7fe3e00186c0 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.115 7fe3e7fff700 1 -- 172.21.2.117:0/951405347 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fe3f0000940 con 0x7fe3e00186c0 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:24:25.111 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 msgr2=0x7fe3e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 0x7fe3e001abc0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 msgr2=0x7fe4000a8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fe3f8006fa0 tx=0x7fe3f800bd90).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 shutdown_connections 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3e00186c0 0x7fe3e001abc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4000a8650 0x7fe4000a8a50 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4001a8350 0x7fe4001a8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 --2- 172.21.2.117:0/951405347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4000b8cc0 0x7fe4000a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 >> 172.21.2.117:0/951405347 conn(0x7fe400107e80 msgr2=0x7fe40019e850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 shutdown_connections 2021-02-10T11:24:25.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.116 7fe4055a8700 1 -- 172.21.2.117:0/951405347 wait complete. 2021-02-10T11:24:25.121 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.537 7f36a7141700 1 Processor -- start 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.537 7f36a7141700 1 -- start start 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f36a007da20 0x7f36a00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00e4da0 0x7f36a01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a00e51a0 0x7f36a0077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a0077a70 con 0x7f36a00e51a0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a00777d0 con 0x7f36a007da20 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a7141700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a007de20 con 0x7f36a00e4da0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a00e51a0 0x7f36a0077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a00e51a0 0x7f36a0077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43040/0 (socket says 172.21.2.117:43040) 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 -- 172.21.2.117:0/3125063721 learned_addr learned my addr 172.21.2.117:0/3125063721 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 -- 172.21.2.117:0/3125063721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00e4da0 msgr2=0x7f36a01b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 --2- 172.21.2.117:0/3125063721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00e4da0 0x7f36a01b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 -- 172.21.2.117:0/3125063721 >> v2:172.21.2.125:3300/0 conn(0x7f36a007da20 msgr2=0x7f36a00771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 --2- 172.21.2.117:0/3125063721 >> v2:172.21.2.125:3300/0 conn(0x7f36a007da20 0x7f36a00771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 -- 172.21.2.117:0/3125063721 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a01b0710 con 0x7f36a00e51a0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f36a575e700 1 --2- 172.21.2.117:0/3125063721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00e51a0 0x7f36a0077ff0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3698006b80 tx=0x7f3698009b90).ready entity=mon.0 client_cookie=1114c2988f949cc1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.538 7f369f7fe700 1 -- 172.21.2.117:0/3125063721 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36980022e0 con 0x7f36a00e51a0 2021-02-10T11:24:25.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f369f7fe700 1 -- 172.21.2.117:0/3125063721 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698002560 con 0x7f36a00e51a0 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f369f7fe700 1 -- 172.21.2.117:0/3125063721 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3698002a40 con 0x7f36a00e51a0 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 -- 172.21.2.117:0/3125063721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00e51a0 msgr2=0x7f36a0077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 --2- 172.21.2.117:0/3125063721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00e51a0 0x7f36a0077ff0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3698006b80 tx=0x7f3698009b90).stop 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 -- 172.21.2.117:0/3125063721 shutdown_connections 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 --2- 172.21.2.117:0/3125063721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00e51a0 0x7f36a0077ff0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 --2- 172.21.2.117:0/3125063721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00e4da0 0x7f36a01b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 --2- 172.21.2.117:0/3125063721 >> v2:172.21.2.125:3300/0 conn(0x7f36a007da20 0x7f36a00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 -- 172.21.2.117:0/3125063721 >> 172.21.2.117:0/3125063721 conn(0x7f36a00ce450 msgr2=0x7f36a00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 -- 172.21.2.117:0/3125063721 shutdown_connections 2021-02-10T11:24:25.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.539 7f36a7141700 1 -- 172.21.2.117:0/3125063721 wait complete. 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 Processor -- start 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 -- start start 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a0093960 0x7f36a0093d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00ac730 0x7f36a00acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00adb40 con 0x7f36a00ad150 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00ad550 con 0x7f36a0093960 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a7141700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00ad730 con 0x7f36a00ac730 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.554 7f36a575e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43046/0 (socket says 172.21.2.117:43046) 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 -- 172.21.2.117:0/2689900423 learned_addr learned my addr 172.21.2.117:0/2689900423 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 -- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00ac730 msgr2=0x7f36a00acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00ac730 0x7f36a00acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 -- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a0093960 msgr2=0x7f36a0093d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a0093960 0x7f36a0093d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 -- 172.21.2.117:0/2689900423 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00ae6e0 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a575e700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f369800b5b0 tx=0x7f369800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369800bd40 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698002030 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a7141700 1 -- 172.21.2.117:0/2689900423 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a00ae6e0 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f36a7141700 1 -- 172.21.2.117:0/2689900423 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a00aec70 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.555 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36980024c0 con 0x7f36a00ad150 2021-02-10T11:24:25.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.556 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3698000940 con 0x7f36a00ad150 2021-02-10T11:24:25.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.556 7f369d7fa700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:25.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.556 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3698032710 con 0x7f36a00ad150 2021-02-10T11:24:25.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.556 7f36a4f5d700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:25.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.557 7f36a4f5d700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:25.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.588 7f36a7141700 1 -- 172.21.2.117:0/2689900423 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a0103370 con 0x7f36a00ad150 2021-02-10T11:24:25.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.591 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3698027430 con 0x7f36a00ad150 2021-02-10T11:24:25.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.737 7f36a7141700 1 -- 172.21.2.117:0/2689900423 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f36a0077a70 con 0x7f36a00ad150 2021-02-10T11:24:25.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.738 7f369d7fa700 1 -- 172.21.2.117:0/2689900423 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3698027430 con 0x7f36a00ad150 2021-02-10T11:24:25.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.739 7f36a7141700 1 -- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 msgr2=0x7f368401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.739 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.739 7f36a7141700 1 -- 172.21.2.117:0/2689900423 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 msgr2=0x7f36a00ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:25.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.739 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f369800b5b0 tx=0x7f369800b950).stop 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 -- 172.21.2.117:0/2689900423 shutdown_connections 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a00ad150 0x7f36a00ae0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a00ac730 0x7f36a00acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 --2- 172.21.2.117:0/2689900423 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a0093960 0x7f36a0093d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 -- 172.21.2.117:0/2689900423 >> 172.21.2.117:0/2689900423 conn(0x7f36a00bb5d0 msgr2=0x7f36a0190400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 -- 172.21.2.117:0/2689900423 shutdown_connections 2021-02-10T11:24:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:25.740 7f36a7141700 1 -- 172.21.2.117:0/2689900423 wait complete. 2021-02-10T11:24:25.745 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802434 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 Processor -- start 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 -- start start 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c07da20 0x7f724c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f724c0e4d80 0x7f724c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c077a70 con 0x7f724c0e5180 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f724c0e5c00 con 0x7f724c0e4d80 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.161 7f72529d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c0777d0 con 0x7f724c07da20 2021-02-10T11:24:26.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f724c0e4d80 0x7f724c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f724c0e4d80 0x7f724c1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43054/0 (socket says 172.21.2.117:43054) 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 -- 172.21.2.117:0/1288337743 learned_addr learned my addr 172.21.2.117:0/1288337743 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f7250ff0700 1 --1- 172.21.2.117:0/1288337743 >> v1:172.21.2.117:6789/0 conn(0x7f724c0e5180 0x7f724c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43306/0 (socket says 172.21.2.117:43306) 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 -- 172.21.2.117:0/1288337743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c07da20 msgr2=0x7f724c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 --2- 172.21.2.117:0/1288337743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c07da20 0x7f724c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 -- 172.21.2.117:0/1288337743 >> v1:172.21.2.117:6789/0 conn(0x7f724c0e5180 legacy=0x7f724c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 -- 172.21.2.117:0/1288337743 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c07de20 con 0x7f724c0e4d80 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724b7fe700 1 --2- 172.21.2.117:0/1288337743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c0e4d80 0x7f724c1b00d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7240006700 tx=0x7f7240009b80).ready entity=mon.0 client_cookie=63c9b02d0d9eed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724affd700 1 -- 172.21.2.117:0/1288337743 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f724000c0c0 con 0x7f724c0e4d80 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724affd700 1 -- 172.21.2.117:0/1288337743 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724000c340 con 0x7f724c0e4d80 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f724affd700 1 -- 172.21.2.117:0/1288337743 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f724000c860 con 0x7f724c0e4d80 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f72529d3700 1 -- 172.21.2.117:0/1288337743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c0e4d80 msgr2=0x7f724c1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.162 7f72529d3700 1 --2- 172.21.2.117:0/1288337743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c0e4d80 0x7f724c1b00d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7240006700 tx=0x7f7240009b80).stop 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 -- 172.21.2.117:0/1288337743 shutdown_connections 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 --2- 172.21.2.117:0/1288337743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c0e4d80 0x7f724c1b00d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 --2- 172.21.2.117:0/1288337743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c07da20 0x7f724c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 -- 172.21.2.117:0/1288337743 >> 172.21.2.117:0/1288337743 conn(0x7f724c0ce430 msgr2=0x7f724c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 -- 172.21.2.117:0/1288337743 shutdown_connections 2021-02-10T11:24:26.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.163 7f72529d3700 1 -- 172.21.2.117:0/1288337743 wait complete. 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 Processor -- start 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 -- start start 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 0x7f724c0a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c0a3db0 0x7f724c092650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c093070 con 0x7f724c092c70 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c093290 con 0x7f724c1a0270 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f72529d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c09cf60 con 0x7f724c0a3db0 2021-02-10T11:24:26.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f7250ff0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f7250ff0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43060/0 (socket says 172.21.2.117:43060) 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.178 7f7250ff0700 1 -- 172.21.2.117:0/3950261154 learned_addr learned my addr 172.21.2.117:0/3950261154 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 -- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c0a3db0 msgr2=0x7f724c092650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f724b7fe700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c0a3db0 0x7f724c092650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c0a3db0 0x7f724c092650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 -- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 msgr2=0x7f724c0a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f724bfff700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 0x7f724c0a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 0x7f724c0a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 -- 172.21.2.117:0/3950261154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c09d180 con 0x7f724c092c70 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f724bfff700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 0x7f724c0a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7250ff0700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f7244007f90 tx=0x7f724400abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7244001270 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7244007160 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7244004720 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f72529d3700 1 -- 172.21.2.117:0/3950261154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c09d180 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.179 7f72529d3700 1 -- 172.21.2.117:0/3950261154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c09d710 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.180 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7244007340 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.180 7f7248ff9700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 0x7f722c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.180 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7244032ad0 con 0x7f724c092c70 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.180 7f724bfff700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 0x7f722c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.181 7f724bfff700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 0x7f722c01ac10 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.211 7f72529d3700 1 -- 172.21.2.117:0/3950261154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c103350 con 0x7f724c092c70 2021-02-10T11:24:26.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.214 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7244001270 con 0x7f724c092c70 2021-02-10T11:24:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.338 7f72529d3700 1 -- 172.21.2.117:0/3950261154 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802434", "target": ["mgr", ""]}) v1 -- 0x7f724c0a42c0 con 0x7f722c018730 2021-02-10T11:24:26.338 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:24:26.340 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:26.339 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:24:26.340 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:26.339 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:24:26.340 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:26.339 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:24:26.340 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:26.339 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:24:26.340 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:24:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.373 7f7248ff9700 1 -- 172.21.2.117:0/3950261154 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f723c0009b0 con 0x7f722c018730 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 msgr2=0x7f722c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 0x7f722c01ac10 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 msgr2=0x7f724c09c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f7244007f90 tx=0x7f724400abd0).stop 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 shutdown_connections 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f722c018730 0x7f722c01ac10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f724c092c70 0x7f724c09c940 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f724c0a3db0 0x7f724c092650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 --2- 172.21.2.117:0/3950261154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f724c1a0270 0x7f724c0a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 >> 172.21.2.117:0/3950261154 conn(0x7f724c0a1420 msgr2=0x7f724c093b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 shutdown_connections 2021-02-10T11:24:26.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.403 7f72529d3700 1 -- 172.21.2.117:0/3950261154 wait complete. 2021-02-10T11:24:26.408 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802434 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 Processor -- start 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 -- start start 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a4007da20 0x7f3a400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3a400e4da0 0x7f3a401b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a40077a70 con 0x7f3a400e51a0 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400777d0 con 0x7f3a4007da20 2021-02-10T11:24:26.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a48563700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4007de20 con 0x7f3a400e4da0 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.824 7f3a4637f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a4007da20 0x7f3a400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a4007da20 0x7f3a400771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43066/0 (socket says 172.21.2.117:43066) 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 -- 172.21.2.117:0/1038347618 learned_addr learned my addr 172.21.2.117:0/1038347618 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a46b80700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3a400e51a0 0x7f3a40077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43320/0 (socket says 172.21.2.117:43320) 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 -- 172.21.2.117:0/1038347618 >> v2:172.21.2.125:3300/0 conn(0x7f3a400e4da0 msgr2=0x7f3a401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 --2- 172.21.2.117:0/1038347618 >> v2:172.21.2.125:3300/0 conn(0x7f3a400e4da0 0x7f3a401b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 -- 172.21.2.117:0/1038347618 >> v1:172.21.2.117:6789/0 conn(0x7f3a400e51a0 legacy=0x7f3a40077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 -- 172.21.2.117:0/1038347618 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a401b0710 con 0x7f3a4007da20 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4637f700 1 --2- 172.21.2.117:0/1038347618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4007da20 0x7f3a400771b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f3a34006700 tx=0x7f3a34009b80).ready entity=mon.0 client_cookie=69a834161d4804fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4537d700 1 -- 172.21.2.117:0/1038347618 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a3400c0c0 con 0x7f3a4007da20 2021-02-10T11:24:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4537d700 1 -- 172.21.2.117:0/1038347618 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3400c340 con 0x7f3a4007da20 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a4537d700 1 -- 172.21.2.117:0/1038347618 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a3400c860 con 0x7f3a4007da20 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a48563700 1 -- 172.21.2.117:0/1038347618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4007da20 msgr2=0x7f3a400771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.825 7f3a48563700 1 --2- 172.21.2.117:0/1038347618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4007da20 0x7f3a400771b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f3a34006700 tx=0x7f3a34009b80).stop 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 -- 172.21.2.117:0/1038347618 shutdown_connections 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 --2- 172.21.2.117:0/1038347618 >> v2:172.21.2.125:3300/0 conn(0x7f3a400e4da0 0x7f3a401b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 --2- 172.21.2.117:0/1038347618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4007da20 0x7f3a400771b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 -- 172.21.2.117:0/1038347618 >> 172.21.2.117:0/1038347618 conn(0x7f3a400ce450 msgr2=0x7f3a400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 -- 172.21.2.117:0/1038347618 shutdown_connections 2021-02-10T11:24:26.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.826 7f3a48563700 1 -- 172.21.2.117:0/1038347618 wait complete. 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 Processor -- start 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 -- start start 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a4009cb80 0x7f3a400c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a400b4250 0x7f3a400c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400bff40 con 0x7f3a4009c780 2021-02-10T11:24:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400bfb60 con 0x7f3a4009cb80 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.841 7f3a48563700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400bfd40 con 0x7f3a400b4250 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43072/0 (socket says 172.21.2.117:43072) 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 -- 172.21.2.117:0/2443142203 learned_addr learned my addr 172.21.2.117:0/2443142203 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 -- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a400b4250 msgr2=0x7f3a400c0ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a400b4250 0x7f3a400c0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 -- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a4009cb80 msgr2=0x7f3a400c04c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a4009cb80 0x7f3a400c04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 -- 172.21.2.117:0/2443142203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400c1100 con 0x7f3a4009c780 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a4637f700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f3a3400c620 tx=0x7f3a3400c9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a3400cdb0 con 0x7f3a4009c780 2021-02-10T11:24:26.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34009390 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a34002300 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a48563700 1 -- 172.21.2.117:0/2443142203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a400c1100 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.842 7f3a48563700 1 -- 172.21.2.117:0/2443142203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400c1100 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.843 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a34002020 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.843 7f3a32ffd700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 0x7f3a2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.843 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3a34032940 con 0x7f3a4009c780 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.844 7f3a45b7e700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 0x7f3a2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:26.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.844 7f3a45b7e700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 0x7f3a2801abc0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:26.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.874 7f3a48563700 1 -- 172.21.2.117:0/2443142203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a400ff170 con 0x7f3a4009c780 2021-02-10T11:24:26.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:26.877 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a34000c90 con 0x7f3a4009c780 2021-02-10T11:24:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.004 7f3a48563700 1 -- 172.21.2.117:0/2443142203 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802434", "target": ["mgr", ""]}) v1 -- 0x7f3a40077a70 con 0x7f3a280186c0 2021-02-10T11:24:27.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.008 7f3a32ffd700 1 -- 172.21.2.117:0/2443142203 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3a3c006970 con 0x7f3a280186c0 2021-02-10T11:24:27.008 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332 2021-02-10T11:24:27.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 -- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 msgr2=0x7f3a2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 0x7f3a2801abc0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 -- 172.21.2.117:0/2443142203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 msgr2=0x7f3a400b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f3a3400c620 tx=0x7f3a3400c9c0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 -- 172.21.2.117:0/2443142203 shutdown_connections 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a400b4250 0x7f3a400c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a4009cb80 0x7f3a400c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a280186c0 0x7f3a2801abc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 --2- 172.21.2.117:0/2443142203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a4009c780 0x7f3a400b3c30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.013 7f3a48563700 1 -- 172.21.2.117:0/2443142203 >> 172.21.2.117:0/2443142203 conn(0x7f3a4009d3b0 msgr2=0x7f3a400b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.014 7f3a48563700 1 -- 172.21.2.117:0/2443142203 shutdown_connections 2021-02-10T11:24:27.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:27.014 7f3a48563700 1 -- 172.21.2.117:0/2443142203 wait complete. 2021-02-10T11:24:27.018 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802434 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332 2021-02-10T11:24:27.018 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:27.018 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:27.018 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:27.353 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:27.353 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:27.353 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:27.734 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:27.735 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:27.735 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0268588 s, 39.0 MB/s 2021-02-10T11:24:27.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:27.736 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:27.736 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:28.069 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:28.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:28.070 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:28.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:28.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:28.440 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230695 s, 45.5 MB/s 2021-02-10T11:24:28.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:28.441 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:28.442 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:28.778 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:28.778 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:28.779 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:29.150 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:29.151 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:29.151 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221553 s, 47.3 MB/s 2021-02-10T11:24:29.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:29.153 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:29.153 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:29.488 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:29.489 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:29.489 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:29.861 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:29.861 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:29.861 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227032 s, 46.2 MB/s 2021-02-10T11:24:29.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:29.862 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:29.862 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:30.197 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:30.197 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:30.197 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:30.569 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:30.570 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:30.570 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225498 s, 46.5 MB/s 2021-02-10T11:24:30.571 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:30.571 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:30.571 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:30.906 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:30.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:30.923 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:31.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:31.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:31.298 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251299 s, 41.7 MB/s 2021-02-10T11:24:31.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:31.299 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:31.300 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:31.634 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:31.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:31.635 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:32.008 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:32.008 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:32.009 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229995 s, 45.6 MB/s 2021-02-10T11:24:32.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:32.010 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:32.010 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:32.344 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:32.345 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:32.345 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:32.718 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:32.718 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:32.718 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259537 s, 40.4 MB/s 2021-02-10T11:24:32.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:32.719 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:32.719 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:33.053 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:33.054 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:33.054 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:33.427 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:33.427 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:33.427 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241194 s, 43.5 MB/s 2021-02-10T11:24:33.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:33.428 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:33.428 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:33.763 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:33.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:33.764 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802434/ff8df3b5-0804-4497-b2c7-fb7714c52332/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0268646 s, 39.0 MB/s 2021-02-10T11:24:34.143 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802435 2021-02-10T11:24:34.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 Processor -- start 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 -- start start 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7904079000 0x7f79040e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041b00d0 0x7f79041b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7904077a70 con 0x7f790407da20 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040e53a0 con 0x7f7904079000 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.559 7f7909c26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7904079400 con 0x7f79041b00d0 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7904079000 0x7f79040e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f79037fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f790407da20 0x7f79040771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43340/0 (socket says 172.21.2.117:43340) 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7904079000 0x7f79040e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43090/0 (socket says 172.21.2.117:43090) 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f79037fe700 1 -- 172.21.2.117:0/2151490641 learned_addr learned my addr 172.21.2.117:0/2151490641 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:34.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 -- 172.21.2.117:0/2151490641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041b00d0 msgr2=0x7f79041b04d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 --2- 172.21.2.117:0/2151490641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041b00d0 0x7f79041b04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 -- 172.21.2.117:0/2151490641 >> v1:172.21.2.117:6789/0 conn(0x7f790407da20 legacy=0x7f79040771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 -- 172.21.2.117:0/2151490641 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790407de20 con 0x7f7904079000 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7902ffd700 1 --2- 172.21.2.117:0/2151490641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7904079000 0x7f79040e4d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f78ec006650 tx=0x7f78ec009a80).ready entity=mon.0 client_cookie=a7e47c7df3c32fd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f79027fc700 1 -- 172.21.2.117:0/2151490641 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78ec0009b0 con 0x7f7904079000 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f79027fc700 1 -- 172.21.2.117:0/2151490641 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec00c0c0 con 0x7f7904079000 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f79027fc700 1 -- 172.21.2.117:0/2151490641 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78ec00c510 con 0x7f7904079000 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7909c26700 1 -- 172.21.2.117:0/2151490641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7904079000 msgr2=0x7f79040e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.560 7f7909c26700 1 --2- 172.21.2.117:0/2151490641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7904079000 0x7f79040e4d80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f78ec006650 tx=0x7f78ec009a80).stop 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 -- 172.21.2.117:0/2151490641 shutdown_connections 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 --2- 172.21.2.117:0/2151490641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041b00d0 0x7f79041b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 --2- 172.21.2.117:0/2151490641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7904079000 0x7f79040e4d80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 -- 172.21.2.117:0/2151490641 >> 172.21.2.117:0/2151490641 conn(0x7f79040ce430 msgr2=0x7f79040e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:34.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 -- 172.21.2.117:0/2151490641 shutdown_connections 2021-02-10T11:24:34.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.561 7f7909c26700 1 -- 172.21.2.117:0/2151490641 wait complete. 2021-02-10T11:24:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.575 7f7909c26700 1 Processor -- start 2021-02-10T11:24:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 -- start start 2021-02-10T11:24:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041a0270 0x7f79040a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7904092c70 0x7f790409c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904093070 con 0x7f79040a3db0 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904093290 con 0x7f7904092c70 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7909c26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790409cf60 con 0x7f79041a0270 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43098/0 (socket says 172.21.2.117:43098) 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 -- 172.21.2.117:0/3180837582 learned_addr learned my addr 172.21.2.117:0/3180837582 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 -- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041a0270 msgr2=0x7f79040a3790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041a0270 0x7f79040a3790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 -- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7904092c70 msgr2=0x7f790409c940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7904092c70 0x7f790409c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.576 7f7902ffd700 1 -- 172.21.2.117:0/3180837582 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790409d180 con 0x7f79040a3db0 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f7902ffd700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f78ec009650 tx=0x7f78ec00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78ec00bd50 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec006030 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78ec005560 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f7909c26700 1 -- 172.21.2.117:0/3180837582 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790409d180 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f7909c26700 1 -- 172.21.2.117:0/3180837582 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790409d180 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.577 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78ec009300 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.578 7f78fbfff700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 0x7f78e801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.578 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78ec032910 con 0x7f79040a3db0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.578 7f79037fe700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 0x7f78e801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.579 7f79037fe700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 0x7f78e801ac10 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:34.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.609 7f7909c26700 1 -- 172.21.2.117:0/3180837582 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904103350 con 0x7f79040a3db0 2021-02-10T11:24:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.612 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78ec000940 con 0x7f79040a3db0 2021-02-10T11:24:34.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.738 7f7909c26700 1 -- 172.21.2.117:0/3180837582 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802435", "target": ["mgr", ""]}) v1 -- 0x7f7904077a70 con 0x7f78e8018730 2021-02-10T11:24:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.760 7f78fbfff700 1 -- 172.21.2.117:0/3180837582 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f78f4001490 con 0x7f78e8018730 2021-02-10T11:24:34.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.770 7f7909c26700 1 -- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 msgr2=0x7f78e801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 0x7f78e801ac10 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 -- 172.21.2.117:0/3180837582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 msgr2=0x7f7904092650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:34.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f78ec009650 tx=0x7f78ec00b960).stop 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 -- 172.21.2.117:0/3180837582 shutdown_connections 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78e8018730 0x7f78e801ac10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79040a3db0 0x7f7904092650 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7904092c70 0x7f790409c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 --2- 172.21.2.117:0/3180837582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79041a0270 0x7f79040a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 -- 172.21.2.117:0/3180837582 >> 172.21.2.117:0/3180837582 conn(0x7f79040a1420 msgr2=0x7f7904093b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 -- 172.21.2.117:0/3180837582 shutdown_connections 2021-02-10T11:24:34.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:34.771 7f7909c26700 1 -- 172.21.2.117:0/3180837582 wait complete. 2021-02-10T11:24:34.774 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802435 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 Processor -- start 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 -- start start 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c58079000 0x7f3c580e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c580d1d20 con 0x7f3c5807da20 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58077a70 con 0x7f3c581b00f0 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c5d790700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c580e53c0 con 0x7f3c58079000 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c56d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c58079000 0x7f3c580e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c57574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3c5807da20 0x7f3c580771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43352/0 (socket says 172.21.2.117:43352) 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c56d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c58079000 0x7f3c580e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43102/0 (socket says 172.21.2.117:43102) 2021-02-10T11:24:35.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.191 7f3c57574700 1 -- 172.21.2.117:0/4020060661 learned_addr learned my addr 172.21.2.117:0/4020060661 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56d73700 1 -- 172.21.2.117:0/4020060661 >> v1:172.21.2.125:6789/0 conn(0x7f3c581b00f0 legacy=0x7f3c580d0520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56d73700 1 -- 172.21.2.117:0/4020060661 >> v1:172.21.2.117:6789/0 conn(0x7f3c5807da20 legacy=0x7f3c580771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56d73700 1 -- 172.21.2.117:0/4020060661 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58079400 con 0x7f3c58079000 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56d73700 1 --2- 172.21.2.117:0/4020060661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c58079000 0x7f3c580e4da0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3c48006650 tx=0x7f3c48009a80).ready entity=mon.0 client_cookie=3d1e83a4c8303eb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56572700 1 -- 172.21.2.117:0/4020060661 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c4800c0c0 con 0x7f3c58079000 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56572700 1 -- 172.21.2.117:0/4020060661 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4800c340 con 0x7f3c58079000 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c56572700 1 -- 172.21.2.117:0/4020060661 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c4800c860 con 0x7f3c58079000 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c5d790700 1 -- 172.21.2.117:0/4020060661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c58079000 msgr2=0x7f3c580e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.192 7f3c5d790700 1 --2- 172.21.2.117:0/4020060661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c58079000 0x7f3c580e4da0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3c48006650 tx=0x7f3c48009a80).stop 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.193 7f3c5d790700 1 -- 172.21.2.117:0/4020060661 shutdown_connections 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.193 7f3c5d790700 1 --2- 172.21.2.117:0/4020060661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c58079000 0x7f3c580e4da0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.193 7f3c5d790700 1 -- 172.21.2.117:0/4020060661 >> 172.21.2.117:0/4020060661 conn(0x7f3c580ce450 msgr2=0x7f3c580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.193 7f3c5d790700 1 -- 172.21.2.117:0/4020060661 shutdown_connections 2021-02-10T11:24:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.193 7f3c5d790700 1 -- 172.21.2.117:0/4020060661 wait complete. 2021-02-10T11:24:35.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 Processor -- start 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 -- start start 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c580ba720 0x7f3c580bab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c58107760 0x7f3c58107b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580bbb60 con 0x7f3c580bb140 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580c20a0 con 0x7f3c58107760 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c5d790700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580b9dc0 con 0x7f3c580ba720 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c56d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c56d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43110/0 (socket says 172.21.2.117:43110) 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c56d73700 1 -- 172.21.2.117:0/1285914005 learned_addr learned my addr 172.21.2.117:0/1285914005 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.208 7f3c57574700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c580ba720 0x7f3c580bab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c57d75700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c58107760 0x7f3c58107b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:35.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 -- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c580ba720 msgr2=0x7f3c580bab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c580ba720 0x7f3c580bab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 -- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c58107760 msgr2=0x7f3c58107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c58107760 0x7f3c58107b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 -- 172.21.2.117:0/1285914005 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58108180 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c56d73700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3c48009630 tx=0x7f3c4800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c480093a0 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4800bcf0 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c48005560 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c58108180 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.209 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c58108180 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.210 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c48008240 con 0x7f3c580bb140 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.210 7f3c4ffff700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 0x7f3c3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:35.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.210 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3c48032780 con 0x7f3c580bb140 2021-02-10T11:24:35.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.210 7f3c57574700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 0x7f3c3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:35.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.211 7f3c57574700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 0x7f3c3c01ac10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:35.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.241 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c580ce450 con 0x7f3c580bb140 2021-02-10T11:24:35.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.244 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c48008030 con 0x7f3c580bb140 2021-02-10T11:24:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.372 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802435", "target": ["mgr", ""]}) v1 -- 0x7f3c580c20a0 con 0x7f3c3c018730 2021-02-10T11:24:35.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.375 7f3c4ffff700 1 -- 172.21.2.117:0/1285914005 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3c40001070 con 0x7f3c3c018730 2021-02-10T11:24:35.375 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179 2021-02-10T11:24:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.381 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 msgr2=0x7f3c3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 0x7f3c3c01ac10 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 msgr2=0x7f3c580bb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:35.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3c48009630 tx=0x7f3c4800b900).stop 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 shutdown_connections 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c3c018730 0x7f3c3c01ac10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c580bb140 0x7f3c580bb540 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c58107760 0x7f3c58107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 --2- 172.21.2.117:0/1285914005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c580ba720 0x7f3c580bab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 >> 172.21.2.117:0/1285914005 conn(0x7f3c5810d560 msgr2=0x7f3c58106a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 shutdown_connections 2021-02-10T11:24:35.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:35.382 7f3c5d790700 1 -- 172.21.2.117:0/1285914005 wait complete. 2021-02-10T11:24:35.385 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802435 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179 2021-02-10T11:24:35.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:35.386 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:35.386 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:35.721 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:35.721 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:35.722 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:36.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:36.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:36.106 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261901 s, 40.0 MB/s 2021-02-10T11:24:36.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:36.107 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:36.108 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:36.462 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:36.463 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:36.463 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:36.842 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:36.842 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:36.842 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232224 s, 45.2 MB/s 2021-02-10T11:24:36.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:36.844 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:36.844 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:37.179 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:37.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:37.180 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:37.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:37.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:37.550 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233031 s, 45.0 MB/s 2021-02-10T11:24:37.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:37.552 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:37.552 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:37.886 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:37.887 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:37.887 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225912 s, 46.4 MB/s 2021-02-10T11:24:38.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:38.261 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:38.261 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:38.596 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:38.596 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:38.596 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243246 s, 43.1 MB/s 2021-02-10T11:24:38.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:38.972 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:38.973 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:39.307 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:39.307 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:39.308 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:39.684 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:39.684 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:39.685 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260176 s, 40.3 MB/s 2021-02-10T11:24:39.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:39.686 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:39.686 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:40.021 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:40.021 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:40.022 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:40.397 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:40.397 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:40.397 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244372 s, 42.9 MB/s 2021-02-10T11:24:40.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:40.398 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:40.398 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:40.732 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:40.732 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:40.733 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:41.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:41.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:41.105 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231325 s, 45.3 MB/s 2021-02-10T11:24:41.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:41.106 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:41.106 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:41.441 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:41.441 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:41.441 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:41.816 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:41.817 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:41.817 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240651 s, 43.6 MB/s 2021-02-10T11:24:41.818 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:41.818 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:41.818 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:42.153 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:42.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:42.153 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802435/5483e21b-46d6-4a85-833c-70817c01c179/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:42.524 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:42.525 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:42.525 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248842 s, 42.1 MB/s 2021-02-10T11:24:42.526 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802436 2021-02-10T11:24:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.943 7f5c5f6c5700 1 Processor -- start 2021-02-10T11:24:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.943 7f5c5f6c5700 1 -- start start 2021-02-10T11:24:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5f6c5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5c5807da20 0x7f5c580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5f6c5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580d1d20 con 0x7f5c580e4da0 2021-02-10T11:24:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5f6c5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c58077a70 con 0x7f5c580e51a0 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5f6c5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c580777d0 con 0x7f5c5807da20 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5d4e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5c5807da20 0x7f5c580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5cce0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5c580e4da0 0x7f5c581b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43368/0 (socket says 172.21.2.117:43368) 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5d4e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5c5807da20 0x7f5c580771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43118/0 (socket says 172.21.2.117:43118) 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5cce0700 1 -- 172.21.2.117:0/1629746457 learned_addr learned my addr 172.21.2.117:0/1629746457 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5d4e1700 1 -- 172.21.2.117:0/1629746457 >> v1:172.21.2.125:6789/0 conn(0x7f5c580e51a0 legacy=0x7f5c580d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5d4e1700 1 -- 172.21.2.117:0/1629746457 >> v1:172.21.2.117:6789/0 conn(0x7f5c580e4da0 legacy=0x7f5c581b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c4ffff700 1 -- 172.21.2.117:0/1629746457 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783322756 0 0) 0x7f5c480010b0 con 0x7f5c580e4da0 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.944 7f5c5d4e1700 1 -- 172.21.2.117:0/1629746457 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c581b08e0 con 0x7f5c5807da20 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5d4e1700 1 --2- 172.21.2.117:0/1629746457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5807da20 0x7f5c580771b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5c54006650 tx=0x7f5c54009a80).ready entity=mon.0 client_cookie=46805b020fada37e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c4ffff700 1 -- 172.21.2.117:0/1629746457 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c540009b0 con 0x7f5c5807da20 2021-02-10T11:24:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c4ffff700 1 -- 172.21.2.117:0/1629746457 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5400c0c0 con 0x7f5c5807da20 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c4ffff700 1 -- 172.21.2.117:0/1629746457 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c5400c510 con 0x7f5c5807da20 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5f6c5700 1 -- 172.21.2.117:0/1629746457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5807da20 msgr2=0x7f5c580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5f6c5700 1 --2- 172.21.2.117:0/1629746457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5807da20 0x7f5c580771b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5c54006650 tx=0x7f5c54009a80).stop 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5f6c5700 1 -- 172.21.2.117:0/1629746457 shutdown_connections 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5f6c5700 1 --2- 172.21.2.117:0/1629746457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5807da20 0x7f5c580771b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.945 7f5c5f6c5700 1 -- 172.21.2.117:0/1629746457 >> 172.21.2.117:0/1629746457 conn(0x7f5c580ce450 msgr2=0x7f5c580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.946 7f5c5f6c5700 1 -- 172.21.2.117:0/1629746457 shutdown_connections 2021-02-10T11:24:42.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.946 7f5c5f6c5700 1 -- 172.21.2.117:0/1629746457 wait complete. 2021-02-10T11:24:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.960 7f5c5f6c5700 1 Processor -- start 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.960 7f5c5f6c5700 1 -- start start 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c5809cb80 0x7f5c580c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c580b4250 0x7f5c580c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c580bff40 con 0x7f5c5809c780 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c580bfb60 con 0x7f5c580b4250 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5f6c5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c580bfd40 con 0x7f5c5809cb80 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43120/0 (socket says 172.21.2.117:43120) 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 -- 172.21.2.117:0/1011213327 learned_addr learned my addr 172.21.2.117:0/1011213327 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 -- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c5809cb80 msgr2=0x7f5c580c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c5809cb80 0x7f5c580c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 -- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c580b4250 msgr2=0x7f5c580c0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c580b4250 0x7f5c580c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 -- 172.21.2.117:0/1011213327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c580c1100 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.961 7f5c5d4e1700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f5c54009630 tx=0x7f5c5400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c540093a0 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5400bd70 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5c54005560 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c580c1100 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c580c1690 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5c54007020 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.962 7f5c4dffb700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 0x7f5c3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.963 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5c54032510 con 0x7f5c5809c780 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.963 7f5c5cce0700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 0x7f5c3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:42.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.963 7f5c5cce0700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 0x7f5c3801abc0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:42.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.994 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c580ff170 con 0x7f5c5809c780 2021-02-10T11:24:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:42.997 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5c540183f0 con 0x7f5c5809c780 2021-02-10T11:24:43.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.122 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802436", "target": ["mgr", ""]}) v1 -- 0x7f5c580c1690 con 0x7f5c380186c0 2021-02-10T11:24:43.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.147 7f5c4dffb700 1 -- 172.21.2.117:0/1011213327 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5c48000940 con 0x7f5c380186c0 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 msgr2=0x7f5c3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 0x7f5c3801abc0 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 msgr2=0x7f5c580b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f5c54009630 tx=0x7f5c5400b980).stop 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 shutdown_connections 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c580b4250 0x7f5c580c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c5809cb80 0x7f5c580c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5c380186c0 0x7f5c3801abc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 --2- 172.21.2.117:0/1011213327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c5809c780 0x7f5c580b3c30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 >> 172.21.2.117:0/1011213327 conn(0x7f5c5809d3b0 msgr2=0x7f5c580b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:43.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 shutdown_connections 2021-02-10T11:24:43.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.156 7f5c5f6c5700 1 -- 172.21.2.117:0/1011213327 wait complete. 2021-02-10T11:24:43.159 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802436 2021-02-10T11:24:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.575 7fe29d23f700 1 Processor -- start 2021-02-10T11:24:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.575 7fe29d23f700 1 -- start start 2021-02-10T11:24:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29d23f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe29807da20 0x7fe2980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29d23f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 0x7fe298079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29d23f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe298077a70 con 0x7fe2980e4da0 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29d23f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2980777d0 con 0x7fe29807da20 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29d23f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2981b08e0 con 0x7fe2980e51a0 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe29807da20 0x7fe2980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe29807da20 0x7fe2980771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43128/0 (socket says 172.21.2.117:43128) 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 -- 172.21.2.117:0/3382778598 learned_addr learned my addr 172.21.2.117:0/3382778598 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29759e700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 0x7fe298079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 -- 172.21.2.117:0/3382778598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 msgr2=0x7fe298079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 0x7fe298079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 -- 172.21.2.117:0/3382778598 >> v1:172.21.2.117:6789/0 conn(0x7fe2980e4da0 legacy=0x7fe2981b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 -- 172.21.2.117:0/3382778598 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29807de20 con 0x7fe29807da20 2021-02-10T11:24:43.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe29759e700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 0x7fe298079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.576 7fe296d9d700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe29807da20 0x7fe2980771b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fe280006700 tx=0x7fe280009b80).ready entity=mon.0 client_cookie=469aa1f28993369e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe295d9b700 1 -- 172.21.2.117:0/3382778598 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe28000c0c0 con 0x7fe29807da20 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe295d9b700 1 -- 172.21.2.117:0/3382778598 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28000c340 con 0x7fe29807da20 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe295d9b700 1 -- 172.21.2.117:0/3382778598 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe28000c7a0 con 0x7fe29807da20 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 -- 172.21.2.117:0/3382778598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe29807da20 msgr2=0x7fe2980771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe29807da20 0x7fe2980771b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fe280006700 tx=0x7fe280009b80).stop 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 -- 172.21.2.117:0/3382778598 shutdown_connections 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2980e51a0 0x7fe298079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 --2- 172.21.2.117:0/3382778598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe29807da20 0x7fe2980771b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 -- 172.21.2.117:0/3382778598 >> 172.21.2.117:0/3382778598 conn(0x7fe2980ce450 msgr2=0x7fe2980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.577 7fe29d23f700 1 -- 172.21.2.117:0/3382778598 shutdown_connections 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.578 7fe29d23f700 1 -- 172.21.2.117:0/3382778598 wait complete. 2021-02-10T11:24:43.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 Processor -- start 2021-02-10T11:24:43.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 -- start start 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2981b5130 0x7fe2981b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2981b5b50 0x7fe2980b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980b8da0 con 0x7fe2981b5f50 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.592 7fe29d23f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980a4140 con 0x7fe2981b5130 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29d23f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2980a4320 con 0x7fe2981b5b50 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:43.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43134/0 (socket says 172.21.2.117:43134) 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 -- 172.21.2.117:0/2457997412 learned_addr learned my addr 172.21.2.117:0/2457997412 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 -- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2981b5b50 msgr2=0x7fe2980b5670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2981b5b50 0x7fe2980b5670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 -- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2981b5130 msgr2=0x7fe2981b5530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2981b5130 0x7fe2981b5530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 -- 172.21.2.117:0/2457997412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2980b62b0 con 0x7fe2981b5f50 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29759e700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fe290009980 tx=0x7fe29000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe29000c2f0 con 0x7fe2981b5f50 2021-02-10T11:24:43.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290006790 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe29000c760 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2980b62b0 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.593 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2980b62b0 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.594 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe2900011f0 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.594 7fe28f7fe700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 0x7fe27c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.594 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe290032c90 con 0x7fe2981b5f50 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.595 7fe296d9d700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 0x7fe27c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:43.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.595 7fe296d9d700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 0x7fe27c01abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:43.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.626 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2981b4f00 con 0x7fe2981b5f50 2021-02-10T11:24:43.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.628 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe290006770 con 0x7fe2981b5f50 2021-02-10T11:24:43.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.755 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802436", "target": ["mgr", ""]}) v1 -- 0x7fe2980b6b30 con 0x7fe27c0186c0 2021-02-10T11:24:43.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.759 7fe28f7fe700 1 -- 172.21.2.117:0/2457997412 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe280006500 con 0x7fe27c0186c0 2021-02-10T11:24:43.758 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c 2021-02-10T11:24:43.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 msgr2=0x7fe27c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 0x7fe27c01abc0 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 msgr2=0x7fe2980b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fe290009980 tx=0x7fe29000bcd0).stop 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 shutdown_connections 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe27c0186c0 0x7fe27c01abc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2981b5f50 0x7fe2980b5c90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2981b5b50 0x7fe2980b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 --2- 172.21.2.117:0/2457997412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2981b5130 0x7fe2981b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 >> 172.21.2.117:0/2457997412 conn(0x7fe29808bbc0 msgr2=0x7fe2980c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 shutdown_connections 2021-02-10T11:24:43.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:43.765 7fe29d23f700 1 -- 172.21.2.117:0/2457997412 wait complete. 2021-02-10T11:24:43.768 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802436 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c 2021-02-10T11:24:43.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:43.769 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:43.769 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:44.104 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:44.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:44.105 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:44.486 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:44.487 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:44.487 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250545 s, 41.9 MB/s 2021-02-10T11:24:44.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:44.488 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:44.488 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:44.823 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:44.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:44.824 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:45.201 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:45.201 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:45.201 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.028102 s, 37.3 MB/s 2021-02-10T11:24:45.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:45.202 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:45.202 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:45.538 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:45.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:45.539 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:45.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:45.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:45.914 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247476 s, 42.4 MB/s 2021-02-10T11:24:45.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:45.916 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:45.916 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:46.256 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:46.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:46.258 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:46.648 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:46.648 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:46.648 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253263 s, 41.4 MB/s 2021-02-10T11:24:46.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:46.649 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:46.649 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:46.988 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:46.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:46.988 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:47.362 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:47.362 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:47.363 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246871 s, 42.5 MB/s 2021-02-10T11:24:47.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:47.364 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:47.364 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:47.698 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:47.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:47.698 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:48.071 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:48.071 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:48.071 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245441 s, 42.7 MB/s 2021-02-10T11:24:48.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:48.072 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:48.072 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:48.406 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:48.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:48.407 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:48.781 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:48.781 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:48.781 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248191 s, 42.2 MB/s 2021-02-10T11:24:48.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:48.782 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:48.783 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:49.117 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:49.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:49.118 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:49.493 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:49.493 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:49.493 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257716 s, 40.7 MB/s 2021-02-10T11:24:49.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:49.494 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:49.494 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:49.829 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:49.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:49.829 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:50.205 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:50.205 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:50.205 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251858 s, 41.6 MB/s 2021-02-10T11:24:50.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:50.207 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:50.207 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:50.541 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:50.542 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:50.542 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802436/a376a8f3-f220-4ebc-a4b0-2d504cf08a3c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:50.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:50.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:50.916 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241654 s, 43.4 MB/s 2021-02-10T11:24:50.917 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802437 2021-02-10T11:24:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.333 7fcd210b0700 1 Processor -- start 2021-02-10T11:24:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.333 7fcd210b0700 1 -- start start 2021-02-10T11:24:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd210b0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd1c07da20 0x7fcd1c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd210b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd210b0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c077a70 con 0x7fcd1c0e51a0 2021-02-10T11:24:51.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd210b0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0777d0 con 0x7fcd1c07da20 2021-02-10T11:24:51.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd210b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c07de20 con 0x7fcd1c0e4da0 2021-02-10T11:24:51.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd1c07da20 0x7fcd1c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1a59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1a59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:40824/0 (socket says 172.21.2.117:40824) 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcd1c07da20 0x7fcd1c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:47528/0 (socket says 172.21.2.117:47528) 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1a59c700 1 -- 172.21.2.117:0/1019122424 learned_addr learned my addr 172.21.2.117:0/1019122424 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 -- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 msgr2=0x7fcd1c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 -- 172.21.2.117:0/1019122424 >> v1:172.21.2.125:6789/0 conn(0x7fcd1c0e51a0 legacy=0x7fcd1c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1ad9d700 1 -- 172.21.2.117:0/1019122424 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c1b0710 con 0x7fcd1c07da20 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.334 7fcd1a59c700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd1ad9d700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c07da20 0x7fcd1c0771b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcd0c006650 tx=0x7fcd0c009a80).ready entity=mon.1 client_cookie=dcae609905d9aa14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd19d9b700 1 -- 172.21.2.117:0/1019122424 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd0c00c0c0 con 0x7fcd1c07da20 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd19d9b700 1 -- 172.21.2.117:0/1019122424 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c00c340 con 0x7fcd1c07da20 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd19d9b700 1 -- 172.21.2.117:0/1019122424 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd0c00c860 con 0x7fcd1c07da20 2021-02-10T11:24:51.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd210b0700 1 -- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c07da20 msgr2=0x7fcd1c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.335 7fcd210b0700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c07da20 0x7fcd1c0771b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcd0c006650 tx=0x7fcd0c009a80).stop 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 -- 172.21.2.117:0/1019122424 shutdown_connections 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c0e4da0 0x7fcd1c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 --2- 172.21.2.117:0/1019122424 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c07da20 0x7fcd1c0771b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 -- 172.21.2.117:0/1019122424 >> 172.21.2.117:0/1019122424 conn(0x7fcd1c0ce450 msgr2=0x7fcd1c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 -- 172.21.2.117:0/1019122424 shutdown_connections 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.336 7fcd210b0700 1 -- 172.21.2.117:0/1019122424 wait complete. 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.350 7fcd210b0700 1 Processor -- start 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.350 7fcd210b0700 1 -- start start 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c1b5130 0x7fcd1c1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c1b5b50 0x7fcd1c0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0b8da0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0a4140 con 0x7fcd1c1b5b50 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd210b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0a4320 con 0x7fcd1c1b5130 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd1b59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd1b59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43148/0 (socket says 172.21.2.117:43148) 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd1b59e700 1 -- 172.21.2.117:0/4218149137 learned_addr learned my addr 172.21.2.117:0/4218149137 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd1ad9d700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c1b5130 0x7fcd1c1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.351 7fcd1a59c700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c1b5b50 0x7fcd1c0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 -- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c1b5130 msgr2=0x7fcd1c1b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c1b5130 0x7fcd1c1b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 -- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c1b5b50 msgr2=0x7fcd1c0b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c1b5b50 0x7fcd1c0b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 -- 172.21.2.117:0/4218149137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0b62b0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd1b59e700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fcd14009940 tx=0x7fcd1400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd14000ce0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1400c2f0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd1400c7d0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c0b62b0 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.352 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c0b6840 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.353 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd14001220 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.353 7fcd137fe700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 0x7fcd0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.353 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcd14032a40 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.353 7fcd1ad9d700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 0x7fcd0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.354 7fcd1ad9d700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 0x7fcd0001abc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:51.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.383 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c1b4f00 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.386 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd1400be30 con 0x7fcd1c1b5f50 2021-02-10T11:24:51.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.512 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802437", "target": ["mgr", ""]}) v1 -- 0x7fcd1c0b6b30 con 0x7fcd000186c0 2021-02-10T11:24:51.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.536 7fcd137fe700 1 -- 172.21.2.117:0/4218149137 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcd0c006420 con 0x7fcd000186c0 2021-02-10T11:24:51.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 msgr2=0x7fcd0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 0x7fcd0001abc0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 msgr2=0x7fcd1c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fcd14009940 tx=0x7fcd1400bcf0).stop 2021-02-10T11:24:51.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 shutdown_connections 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd000186c0 0x7fcd0001abc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd1c1b5f50 0x7fcd1c0b5c90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd1c1b5b50 0x7fcd1c0b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 --2- 172.21.2.117:0/4218149137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd1c1b5130 0x7fcd1c1b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 >> 172.21.2.117:0/4218149137 conn(0x7fcd1c08bbc0 msgr2=0x7fcd1c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 shutdown_connections 2021-02-10T11:24:51.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.545 7fcd210b0700 1 -- 172.21.2.117:0/4218149137 wait complete. 2021-02-10T11:24:51.549 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802437 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 Processor -- start 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 -- start start 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a807da20 0x7f64a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a80d1d20 con 0x7f64a80e4da0 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a8077a70 con 0x7f64a80e51a0 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64ad410700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a80777d0 con 0x7f64a807da20 2021-02-10T11:24:51.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64a707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a807da20 0x7f64a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64a687c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f64a80e4da0 0x7f64a81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43408/0 (socket says 172.21.2.117:43408) 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64a687c700 1 -- 172.21.2.117:0/4272400373 learned_addr learned my addr 172.21.2.117:0/4272400373 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.966 7f64a707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a807da20 0x7f64a80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43156/0 (socket says 172.21.2.117:43156) 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a707d700 1 -- 172.21.2.117:0/4272400373 >> v1:172.21.2.125:6789/0 conn(0x7f64a80e51a0 legacy=0x7f64a80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a707d700 1 -- 172.21.2.117:0/4272400373 >> v1:172.21.2.117:6789/0 conn(0x7f64a80e4da0 legacy=0x7f64a81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a707d700 1 -- 172.21.2.117:0/4272400373 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a81b08e0 con 0x7f64a807da20 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a707d700 1 --2- 172.21.2.117:0/4272400373 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a807da20 0x7f64a80771b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f64a00069d0 tx=0x7f64a0009e40).ready entity=mon.0 client_cookie=cfd914652a32f9e2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a607b700 1 -- 172.21.2.117:0/4272400373 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a000b510 con 0x7f64a807da20 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a607b700 1 -- 172.21.2.117:0/4272400373 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a000b890 con 0x7f64a807da20 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64a607b700 1 -- 172.21.2.117:0/4272400373 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64a000bcf0 con 0x7f64a807da20 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64ad410700 1 -- 172.21.2.117:0/4272400373 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a807da20 msgr2=0x7f64a80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.967 7f64ad410700 1 --2- 172.21.2.117:0/4272400373 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a807da20 0x7f64a80771b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f64a00069d0 tx=0x7f64a0009e40).stop 2021-02-10T11:24:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.968 7f64ad410700 1 -- 172.21.2.117:0/4272400373 shutdown_connections 2021-02-10T11:24:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.968 7f64ad410700 1 --2- 172.21.2.117:0/4272400373 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a807da20 0x7f64a80771b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.968 7f64ad410700 1 -- 172.21.2.117:0/4272400373 >> 172.21.2.117:0/4272400373 conn(0x7f64a80ce450 msgr2=0x7f64a80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.968 7f64ad410700 1 -- 172.21.2.117:0/4272400373 shutdown_connections 2021-02-10T11:24:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.968 7f64ad410700 1 -- 172.21.2.117:0/4272400373 wait complete. 2021-02-10T11:24:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.982 7f64ad410700 1 Processor -- start 2021-02-10T11:24:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.982 7f64ad410700 1 -- start start 2021-02-10T11:24:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a80b4720 0x7f64a80bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a80bfdd0 0x7f64a80c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b6200 con 0x7f64a80c2e80 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80c38a0 con 0x7f64a80bfdd0 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64ad410700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80c3b20 con 0x7f64a80b4720 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43166/0 (socket says 172.21.2.117:43166) 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 -- 172.21.2.117:0/113312645 learned_addr learned my addr 172.21.2.117:0/113312645 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a707d700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a80b4720 0x7f64a80bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a687c700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a80bfdd0 0x7f64a80c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 -- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a80b4720 msgr2=0x7f64a80bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a80b4720 0x7f64a80bf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 -- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a80bfdd0 msgr2=0x7f64a80c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a80bfdd0 0x7f64a80c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.983 7f64a787e700 1 -- 172.21.2.117:0/113312645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80b4b20 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f64a787e700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6498006ed0 tx=0x7f649800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f649800bd50 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6498006020 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f64980054b0 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f64ad410700 1 -- 172.21.2.117:0/113312645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a80b4b20 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f64ad410700 1 -- 172.21.2.117:0/113312645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a80b4b20 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6498006200 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 0x7f648801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.984 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f64980327b0 con 0x7f64a80c2e80 2021-02-10T11:24:51.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.985 7f64a707d700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 0x7f648801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:51.986 7f64a707d700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 0x7f648801abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:52.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.016 7f64ad410700 1 -- 172.21.2.117:0/113312645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a80ce450 con 0x7f64a80c2e80 2021-02-10T11:24:52.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.019 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6498006230 con 0x7f64a80c2e80 2021-02-10T11:24:52.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.147 7f64ad410700 1 -- 172.21.2.117:0/113312645 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802437", "target": ["mgr", ""]}) v1 -- 0x7f64a80c38a0 con 0x7f6488018730 2021-02-10T11:24:52.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.150 7f648ffff700 1 -- 172.21.2.117:0/113312645 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f64a0006c50 con 0x7f6488018730 2021-02-10T11:24:52.149 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561 2021-02-10T11:24:52.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 -- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 msgr2=0x7f648801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 0x7f648801abc0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 -- 172.21.2.117:0/113312645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 msgr2=0x7f64a80c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f6498006ed0 tx=0x7f649800b960).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 -- 172.21.2.117:0/113312645 shutdown_connections 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6488018730 0x7f648801abc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a80c2e80 0x7f64a80c3280 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a80bfdd0 0x7f64a80c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 --2- 172.21.2.117:0/113312645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a80b4720 0x7f64a80bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.156 7f64ad410700 1 -- 172.21.2.117:0/113312645 >> 172.21.2.117:0/113312645 conn(0x7f64a81116a0 msgr2=0x7f64a80af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:52.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.157 7f64ad410700 1 -- 172.21.2.117:0/113312645 shutdown_connections 2021-02-10T11:24:52.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:52.157 7f64ad410700 1 -- 172.21.2.117:0/113312645 wait complete. 2021-02-10T11:24:52.160 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802437 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561 2021-02-10T11:24:52.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:52.160 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:52.160 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:52.495 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:52.495 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:52.496 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:52.872 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:52.873 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:52.873 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247731 s, 42.3 MB/s 2021-02-10T11:24:52.874 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:52.874 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:52.874 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:53.208 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:53.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:53.208 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:53.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:53.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:53.583 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240294 s, 43.6 MB/s 2021-02-10T11:24:53.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:53.584 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:53.584 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:53.920 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:53.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:53.920 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:53.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:53.966 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:24:54.300 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:54.300 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:54.301 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0274143 s, 38.2 MB/s 2021-02-10T11:24:54.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:54.302 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:54.302 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:54.677 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:54.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:54.678 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:55.050 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:55.051 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:55.051 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022357 s, 46.9 MB/s 2021-02-10T11:24:55.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:55.052 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:55.053 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:55.387 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:55.387 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:55.387 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:55.762 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:55.763 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:55.763 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247615 s, 42.3 MB/s 2021-02-10T11:24:55.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:55.764 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:55.764 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:56.099 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:56.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:56.100 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:56.477 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:56.477 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:56.477 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0268864 s, 39.0 MB/s 2021-02-10T11:24:56.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:56.478 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:56.478 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:56.813 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:56.813 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:56.814 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253209 s, 41.4 MB/s 2021-02-10T11:24:57.189 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:57.189 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:57.190 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:57.523 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:57.524 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:57.524 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:57.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:57.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:57.894 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237992 s, 44.1 MB/s 2021-02-10T11:24:57.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:57.895 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:57.895 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:58.230 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:58.230 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:58.230 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:58.607 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:58.607 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:58.607 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251185 s, 41.7 MB/s 2021-02-10T11:24:58.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:24:58.608 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:24:58.608 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:24:58.944 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:24:58.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:24:58.944 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802437/bf018549-19f9-4d04-95d2-bf9917057561/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:24:59.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:24:59.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:24:59.318 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238286 s, 44.0 MB/s 2021-02-10T11:24:59.319 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802438 2021-02-10T11:24:59.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.736 7f7869cfe700 1 Processor -- start 2021-02-10T11:24:59.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.736 7f7869cfe700 1 -- start start 2021-02-10T11:24:59.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.736 7f7869cfe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7864079000 0x7f78640e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7869cfe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78641b00f0 0x7f78641b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7869cfe700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7864077a70 con 0x7f786407da20 2021-02-10T11:24:59.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7869cfe700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78640e53c0 con 0x7f78641b00f0 2021-02-10T11:24:59.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7869cfe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7864079400 con 0x7f7864079000 2021-02-10T11:24:59.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78641b00f0 0x7f78641b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78641b00f0 0x7f78641b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43174/0 (socket says 172.21.2.117:43174) 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 -- 172.21.2.117:0/852986832 learned_addr learned my addr 172.21.2.117:0/852986832 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 -- 172.21.2.117:0/852986832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7864079000 msgr2=0x7f78640e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 --2- 172.21.2.117:0/852986832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7864079000 0x7f78640e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 -- 172.21.2.117:0/852986832 >> v1:172.21.2.125:6789/0 conn(0x7f786407da20 legacy=0x7f78640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:24:59.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 -- 172.21.2.117:0/852986832 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786407de20 con 0x7f78641b00f0 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f7863fff700 1 --2- 172.21.2.117:0/852986832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78641b00f0 0x7f78641b04f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f785c006f00 tx=0x7f785c009e90).ready entity=mon.0 client_cookie=1f7f7ae356bc5239 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.737 7f78627fc700 1 -- 172.21.2.117:0/852986832 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c00b300 con 0x7f78641b00f0 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f78627fc700 1 -- 172.21.2.117:0/852986832 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c00b640 con 0x7f78641b00f0 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f78627fc700 1 -- 172.21.2.117:0/852986832 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c00baa0 con 0x7f78641b00f0 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 -- 172.21.2.117:0/852986832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78641b00f0 msgr2=0x7f78641b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:59.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 --2- 172.21.2.117:0/852986832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78641b00f0 0x7f78641b04f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f785c006f00 tx=0x7f785c009e90).stop 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 -- 172.21.2.117:0/852986832 shutdown_connections 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 --2- 172.21.2.117:0/852986832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78641b00f0 0x7f78641b04f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 --2- 172.21.2.117:0/852986832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7864079000 0x7f78640e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 -- 172.21.2.117:0/852986832 >> 172.21.2.117:0/852986832 conn(0x7f78640ce450 msgr2=0x7f78640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 -- 172.21.2.117:0/852986832 shutdown_connections 2021-02-10T11:24:59.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.738 7f7869cfe700 1 -- 172.21.2.117:0/852986832 wait complete. 2021-02-10T11:24:59.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.752 7f7869cfe700 1 Processor -- start 2021-02-10T11:24:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 -- start start 2021-02-10T11:24:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78641034d0 0x7f7864104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7864104740 0x7f7864104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7864103c20 con 0x7f7864105160 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7864103910 con 0x7f7864104740 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7869cfe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78640b7e00 con 0x7f78641034d0 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43176/0 (socket says 172.21.2.117:43176) 2021-02-10T11:24:59.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 -- 172.21.2.117:0/3663447607 learned_addr learned my addr 172.21.2.117:0/3663447607 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:24:59.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 -- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78641034d0 msgr2=0x7f7864104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:59.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78641034d0 0x7f7864104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 -- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7864104740 msgr2=0x7f7864104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.753 7f7863fff700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7864104740 0x7f7864104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f7863fff700 1 -- 172.21.2.117:0/3663447607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78640ab330 con 0x7f7864105160 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f7863fff700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f785c00b940 tx=0x7f785c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c000f90 con 0x7f7864105160 2021-02-10T11:24:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c003030 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c001460 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78640ab330 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.754 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78640ab8c0 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.755 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f785c005820 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.755 7f784bfff700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 0x7f784401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.755 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f785c032890 con 0x7f7864105160 2021-02-10T11:24:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.755 7f78637fe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 0x7f784401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:24:59.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.755 7f78637fe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 0x7f784401abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:24:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.787 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78641032a0 con 0x7f7864105160 2021-02-10T11:24:59.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.790 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f785c000f90 con 0x7f7864105160 2021-02-10T11:24:59.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.915 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802438", "target": ["mgr", ""]}) v1 -- 0x7f78640ab8c0 con 0x7f78440186c0 2021-02-10T11:24:59.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.939 7f784bfff700 1 -- 172.21.2.117:0/3663447607 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7854006a10 con 0x7f78440186c0 2021-02-10T11:24:59.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 msgr2=0x7f784401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:59.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 0x7f784401abc0 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 msgr2=0x7f78640aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f785c00b940 tx=0x7f785c00bce0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 shutdown_connections 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78440186c0 0x7f784401abc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7864105160 0x7f78640aad10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7864104740 0x7f7864104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 --2- 172.21.2.117:0/3663447607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78641034d0 0x7f7864104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 >> 172.21.2.117:0/3663447607 conn(0x7f78640a1140 msgr2=0x7f78640ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 shutdown_connections 2021-02-10T11:24:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:24:59.948 7f7869cfe700 1 -- 172.21.2.117:0/3663447607 wait complete. 2021-02-10T11:24:59.951 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802438 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 Processor -- start 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 -- start start 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac079000 0x7f4aac0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0d1d20 con 0x7f4aac07da20 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac077a70 con 0x7f4aac1b00f0 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4ab1b04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0e5430 con 0x7f4aac079000 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4aab7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4aac07da20 0x7f4aac0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43434/0 (socket says 172.21.2.117:43434) 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.370 7f4aab7fe700 1 -- 172.21.2.117:0/2886896815 learned_addr learned my addr 172.21.2.117:0/2886896815 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaaffd700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac079000 0x7f4aac0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264431165 0 0) 0x7f4a94001180 con 0x7f4aac07da20 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98002fe0 con 0x7f4aac07da20 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 344585193 0 0) 0x7f4a94000ce0 con 0x7f4aac07da20 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac079000 msgr2=0x7f4aac0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac079000 0x7f4aac0e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 >> v1:172.21.2.125:6789/0 conn(0x7f4aac1b00f0 legacy=0x7f4aac0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac079400 con 0x7f4aac07da20 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4a940017f0 con 0x7f4aac07da20 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 >> v1:172.21.2.117:6789/0 conn(0x7f4aac07da20 legacy=0x7f4aac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a98003af0 0x7f4a98005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a98006780 0x7f4a98008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a980094f0 0x7f4a9800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9800c2c0 con 0x7f4a98003af0 2021-02-10T11:25:00.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9800c4e0 con 0x7f4a980094f0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98050710 con 0x7f4a98006780 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a98003af0 0x7f4a98005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 -- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a98006780 msgr2=0x7f4a98008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aaaffd700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a98006780 0x7f4a98008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a98006780 0x7f4a98008c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 -- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a980094f0 msgr2=0x7f4a9800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a980094f0 0x7f4a9800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 -- 172.21.2.117:0/2886896815 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98050970 con 0x7f4a98003af0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.371 7f4aab7fe700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a98003af0 0x7f4a98005f00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4a94001610 tx=0x7f4a94007cb0).ready entity=mon.0 client_cookie=5de675d312eab08e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a94001a60 con 0x7f4a98003af0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9400a520 con 0x7f4a98003af0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4aaa7fc700 1 -- 172.21.2.117:0/2886896815 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a9400aa20 con 0x7f4a98003af0 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 -- 172.21.2.117:0/2886896815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a98003af0 msgr2=0x7f4a98005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 --2- 172.21.2.117:0/2886896815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a98003af0 0x7f4a98005f00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4a94001610 tx=0x7f4a94007cb0).stop 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4aabfff700 1 -- 172.21.2.117:0/2886896815 reap_dead start 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 -- 172.21.2.117:0/2886896815 shutdown_connections 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 -- 172.21.2.117:0/2886896815 >> 172.21.2.117:0/2886896815 conn(0x7f4aac0ce450 msgr2=0x7f4aac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 -- 172.21.2.117:0/2886896815 shutdown_connections 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.372 7f4ab1b04700 1 -- 172.21.2.117:0/2886896815 wait complete. 2021-02-10T11:25:00.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 Processor -- start 2021-02-10T11:25:00.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 -- start start 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4aac0a51f0 0x7f4aac0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac0b77d0 0x7f4aac0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac189970 con 0x7f4aac0a47d0 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac190aa0 con 0x7f4aac0a51f0 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.387 7f4ab1b04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0b81f0 con 0x7f4aac0b77d0 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43196/0 (socket says 172.21.2.117:43196) 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 -- 172.21.2.117:0/479400940 learned_addr learned my addr 172.21.2.117:0/479400940 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 -- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac0b77d0 msgr2=0x7f4aac0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac0b77d0 0x7f4aac0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 -- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4aac0a51f0 msgr2=0x7f4aac0a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4aac0a51f0 0x7f4aac0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 -- 172.21.2.117:0/479400940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0b8410 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aab7fe700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f4a94007fc0 tx=0x7f4a9400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a9400ad60 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a94000940 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac0b8410 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac0b8410 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.388 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a940043a0 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.389 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a940043a0 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.389 7f4aa3fff700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 0x7f4a9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.389 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a940327a0 con 0x7f4aac0a47d0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.389 7f4aaaffd700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 0x7f4a9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:00.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.390 7f4aaaffd700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 0x7f4a9001ac10 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:00.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.420 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac077a70 con 0x7f4aac0a47d0 2021-02-10T11:25:00.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.423 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a94007680 con 0x7f4aac0a47d0 2021-02-10T11:25:00.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.552 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802438", "target": ["mgr", ""]}) v1 -- 0x7f4aac077a70 con 0x7f4a90018730 2021-02-10T11:25:00.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.555 7f4aa3fff700 1 -- 172.21.2.117:0/479400940 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4a9c006a60 con 0x7f4a90018730 2021-02-10T11:25:00.550 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306 2021-02-10T11:25:00.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 msgr2=0x7f4a9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 0x7f4a9001ac10 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:00.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 msgr2=0x7f4aac0a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:00.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f4a94007fc0 tx=0x7f4a9400a970).stop 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 shutdown_connections 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4aac0b77d0 0x7f4aac0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4aac0a51f0 0x7f4aac0a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a90018730 0x7f4a9001ac10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 --2- 172.21.2.117:0/479400940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4aac0a47d0 0x7f4aac0a4bd0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 >> 172.21.2.117:0/479400940 conn(0x7f4aac0b5140 msgr2=0x7f4aac0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 shutdown_connections 2021-02-10T11:25:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:00.556 7f4ab1b04700 1 -- 172.21.2.117:0/479400940 wait complete. 2021-02-10T11:25:00.560 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802438 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306 2021-02-10T11:25:00.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:00.560 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:00.560 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:00.896 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:00.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:00.896 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:01.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:01.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:01.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248092 s, 42.3 MB/s 2021-02-10T11:25:01.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.278 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:01.279 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:01.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:01.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:01.613 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255697 s, 41.0 MB/s 2021-02-10T11:25:01.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:01.991 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:01.991 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:02.326 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:02.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:02.327 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:02.698 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:02.699 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:02.699 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257089 s, 40.8 MB/s 2021-02-10T11:25:02.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:02.700 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:02.700 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:03.034 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:03.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:03.035 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:03.412 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:03.413 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:03.413 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262631 s, 39.9 MB/s 2021-02-10T11:25:03.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:03.414 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:03.414 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:03.749 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:03.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:03.750 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:04.126 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:04.126 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:04.126 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256259 s, 40.9 MB/s 2021-02-10T11:25:04.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:04.127 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:04.127 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:04.462 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:04.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:04.462 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:04.838 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:04.838 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:04.838 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243547 s, 43.1 MB/s 2021-02-10T11:25:04.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:04.839 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:04.839 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:05.173 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:05.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:05.174 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:05.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:05.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:05.550 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024981 s, 42.0 MB/s 2021-02-10T11:25:05.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:05.551 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:05.551 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:05.885 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:05.885 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:05.886 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:06.257 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:06.257 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:06.257 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228176 s, 46.0 MB/s 2021-02-10T11:25:06.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:06.258 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:06.259 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:06.592 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:06.592 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:06.593 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:06.967 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:06.968 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:06.968 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251384 s, 41.7 MB/s 2021-02-10T11:25:06.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:06.970 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:06.970 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:07.317 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:07.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:07.317 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802438/3feba8aa-2e7a-4faa-a3fc-0f7a185e7306/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:07.687 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:07.688 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:07.688 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233035 s, 45.0 MB/s 2021-02-10T11:25:07.689 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802439 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 Processor -- start 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 -- start start 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9d0407da20 0x7f9d040771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040d1d20 con 0x7f9d040e51a0 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d04077a70 con 0x7f9d040e4da0 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.107 7f9d0c28f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040777d0 con 0x7f9d0407da20 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9d0407da20 0x7f9d040771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9d0407da20 0x7f9d040771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43204/0 (socket says 172.21.2.117:43204) 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 -- 172.21.2.117:0/1027176039 learned_addr learned my addr 172.21.2.117:0/1027176039 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 -- 172.21.2.117:0/1027176039 >> v1:172.21.2.125:6789/0 conn(0x7f9d040e4da0 legacy=0x7f9d041b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 -- 172.21.2.117:0/1027176039 >> v1:172.21.2.117:6789/0 conn(0x7f9d040e51a0 legacy=0x7f9d040d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 -- 172.21.2.117:0/1027176039 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d041b08e0 con 0x7f9d0407da20 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d0a0ab700 1 --2- 172.21.2.117:0/1027176039 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d0407da20 0x7f9d040771b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f9d00006700 tx=0x7f9d00009b80).ready entity=mon.0 client_cookie=dffb4ed157fd734c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.108 7f9d090a9700 1 -- 172.21.2.117:0/1027176039 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d0000c0c0 con 0x7f9d0407da20 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d090a9700 1 -- 172.21.2.117:0/1027176039 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0000c400 con 0x7f9d0407da20 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d090a9700 1 -- 172.21.2.117:0/1027176039 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d0000c8e0 con 0x7f9d0407da20 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 -- 172.21.2.117:0/1027176039 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d0407da20 msgr2=0x7f9d040771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 --2- 172.21.2.117:0/1027176039 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d0407da20 0x7f9d040771b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f9d00006700 tx=0x7f9d00009b80).stop 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 -- 172.21.2.117:0/1027176039 shutdown_connections 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 --2- 172.21.2.117:0/1027176039 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d0407da20 0x7f9d040771b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 -- 172.21.2.117:0/1027176039 >> 172.21.2.117:0/1027176039 conn(0x7f9d040ce450 msgr2=0x7f9d040e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 -- 172.21.2.117:0/1027176039 shutdown_connections 2021-02-10T11:25:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.109 7f9d0c28f700 1 -- 172.21.2.117:0/1027176039 wait complete. 2021-02-10T11:25:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 Processor -- start 2021-02-10T11:25:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 -- start start 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d040933f0 0x7f9d040937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d04107050 0x7f9d04107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04107a70 con 0x7f9d04093e10 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040c2c60 con 0x7f9d040933f0 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.124 7f9d0c28f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040c2e80 con 0x7f9d04107050 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43212/0 (socket says 172.21.2.117:43212) 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 -- 172.21.2.117:0/3361956924 learned_addr learned my addr 172.21.2.117:0/3361956924 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d0a0ab700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d040933f0 0x7f9d040937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d0a8ac700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d04107050 0x7f9d04107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 -- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d04107050 msgr2=0x7f9d04107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d04107050 0x7f9d04107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 -- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d040933f0 msgr2=0x7f9d040937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d040933f0 0x7f9d040937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 -- 172.21.2.117:0/3361956924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040c30a0 con 0x7f9d04093e10 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.125 7f9d098aa700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f9cf8006ed0 tx=0x7f9cf800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf8006020 con 0x7f9d04093e10 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf80063a0 con 0x7f9d04093e10 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d040c30a0 con 0x7f9d04093e10 2021-02-10T11:25:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9cf8005730 con 0x7f9d04093e10 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d040c3630 con 0x7f9d04093e10 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9cf8006020 con 0x7f9d04093e10 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 0x7f9cec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.126 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9cf8032930 con 0x7f9d04093e10 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.127 7f9d0a0ab700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 0x7f9cec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.128 7f9d0a0ab700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 0x7f9cec01ac10 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.157 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d040ce450 con 0x7f9d04093e10 2021-02-10T11:25:08.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.160 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9cf80063a0 con 0x7f9d04093e10 2021-02-10T11:25:08.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.284 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802439", "target": ["mgr", ""]}) v1 -- 0x7f9d040c3630 con 0x7f9cec018730 2021-02-10T11:25:08.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.310 7f9cf6ffd700 1 -- 172.21.2.117:0/3361956924 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9d00000940 con 0x7f9cec018730 2021-02-10T11:25:08.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 msgr2=0x7f9cec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 0x7f9cec01ac10 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 msgr2=0x7f9d04106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f9cf8006ed0 tx=0x7f9cf800b960).stop 2021-02-10T11:25:08.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 shutdown_connections 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9cec018730 0x7f9cec01ac10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d04093e10 0x7f9d04106a30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d04107050 0x7f9d04107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 --2- 172.21.2.117:0/3361956924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d040933f0 0x7f9d040937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 >> 172.21.2.117:0/3361956924 conn(0x7f9d041116a0 msgr2=0x7f9d040af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 shutdown_connections 2021-02-10T11:25:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.318 7f9d0c28f700 1 -- 172.21.2.117:0/3361956924 wait complete. 2021-02-10T11:25:08.321 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802439 2021-02-10T11:25:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.738 7f99592a8700 1 Processor -- start 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.738 7f99592a8700 1 -- start start 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f99592a8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f99540d1530 0x7f99540d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f99592a8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9954077a70 con 0x7f9954079000 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f99592a8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9954079400 con 0x7f995407da20 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f99592a8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995407de20 con 0x7f99540d1530 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f995287c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9954079000 0x7f99540e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43466/0 (socket says 172.21.2.117:43466) 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f995287c700 1 -- 172.21.2.117:0/2862530527 learned_addr learned my addr 172.21.2.117:0/2862530527 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150740915 0 0) 0x7f99480009b0 con 0x7f9954079000 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f995207b700 1 -- 172.21.2.117:0/2862530527 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c002fe0 con 0x7f9954079000 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.739 7f995387e700 1 --2- 172.21.2.117:0/2862530527 >> v2:172.21.2.125:3300/0 conn(0x7f99540d1530 0x7f99540d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1639058610 0 0) 0x7f99480010b0 con 0x7f9954079000 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 >> v2:172.21.2.125:3300/0 conn(0x7f99540d1530 msgr2=0x7f99540d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 --2- 172.21.2.117:0/2862530527 >> v2:172.21.2.125:3300/0 conn(0x7f99540d1530 0x7f99540d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 >> v1:172.21.2.125:6789/0 conn(0x7f995407da20 legacy=0x7f99540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99541b00f0 con 0x7f9954079000 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f99480015b0 con 0x7f9954079000 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 >> v1:172.21.2.117:6789/0 conn(0x7f9954079000 legacy=0x7f99540e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 0x7f993c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f993c006790 0x7f993c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f993c009590 0x7f993c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c00c300 con 0x7f993c003aa0 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c00c520 con 0x7f993c009590 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993c00c740 con 0x7f993c006790 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 0x7f993c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995287c700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f993c006790 0x7f993c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995387e700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f993c009590 0x7f993c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 -- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f993c006790 msgr2=0x7f993c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f993c006790 0x7f993c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 -- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f993c009590 msgr2=0x7f993c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f993c009590 0x7f993c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 -- 172.21.2.117:0/2862530527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c00c9e0 con 0x7f993c003aa0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995387e700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f993c009590 0x7f993c00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995387e700 1 -- 172.21.2.117:0/2862530527 reap_dead start 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995307d700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 0x7f993c005ef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f994404aad0 tx=0x7f994404da50).ready entity=mon.0 client_cookie=8b34a8ebafabb0e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994404b280 con 0x7f993c003aa0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994404b5c0 con 0x7f993c003aa0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f995207b700 1 -- 172.21.2.117:0/2862530527 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994404bae0 con 0x7f993c003aa0 2021-02-10T11:25:08.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f99592a8700 1 -- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 msgr2=0x7f993c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.740 7f99592a8700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 0x7f993c005ef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f994404aad0 tx=0x7f994404da50).stop 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.741 7f99592a8700 1 -- 172.21.2.117:0/2862530527 shutdown_connections 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.741 7f99592a8700 1 --2- 172.21.2.117:0/2862530527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f993c003aa0 0x7f993c005ef0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.741 7f99592a8700 1 -- 172.21.2.117:0/2862530527 >> 172.21.2.117:0/2862530527 conn(0x7f99540ce450 msgr2=0x7f99540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.741 7f99592a8700 1 -- 172.21.2.117:0/2862530527 shutdown_connections 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.741 7f99592a8700 1 -- 172.21.2.117:0/2862530527 wait complete. 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.755 7f99592a8700 1 Processor -- start 2021-02-10T11:25:08.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 -- start start 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 0x7f99540a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 0x7f99540b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9954189970 con 0x7f99540a51f0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9954190aa0 con 0x7f99540a47d0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f99592a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99540b81f0 con 0x7f99540b77d0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43226/0 (socket says 172.21.2.117:43226) 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 -- 172.21.2.117:0/2041991712 learned_addr learned my addr 172.21.2.117:0/2041991712 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995387e700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 0x7f99540b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995307d700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 0x7f99540a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 -- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 msgr2=0x7f99540b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 0x7f99540b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 -- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 msgr2=0x7f99540a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.756 7f995287c700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 0x7f99540a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f995287c700 1 -- 172.21.2.117:0/2041991712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540b8410 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f995387e700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 0x7f99540b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f995307d700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 0x7f99540a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f995287c700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9948007650 tx=0x7f994800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f994800acf0 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948004140 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f99592a8700 1 -- 172.21.2.117:0/2041991712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99540b8410 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f99592a8700 1 -- 172.21.2.117:0/2041991712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99540b8410 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.757 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9948004660 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.758 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9948006020 con 0x7f99540a51f0 2021-02-10T11:25:08.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.758 7f993bfff700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 0x7f993401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:08.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.758 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f99480326b0 con 0x7f99540a51f0 2021-02-10T11:25:08.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.758 7f995307d700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 0x7f993401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:08.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.759 7f995307d700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 0x7f993401abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:08.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.789 7f99592a8700 1 -- 172.21.2.117:0/2041991712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9954077a70 con 0x7f99540a51f0 2021-02-10T11:25:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.791 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9948025f90 con 0x7f99540a51f0 2021-02-10T11:25:08.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.921 7f99592a8700 1 -- 172.21.2.117:0/2041991712 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802439", "target": ["mgr", ""]}) v1 -- 0x7f9954077a70 con 0x7f99340186c0 2021-02-10T11:25:08.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.925 7f993bfff700 1 -- 172.21.2.117:0/2041991712 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f994404be00 con 0x7f99340186c0 2021-02-10T11:25:08.923 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd 2021-02-10T11:25:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 -- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 msgr2=0x7f993401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 0x7f993401abc0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 -- 172.21.2.117:0/2041991712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 msgr2=0x7f99540a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f9948007650 tx=0x7f994800a900).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 -- 172.21.2.117:0/2041991712 shutdown_connections 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99340186c0 0x7f993401abc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99540a51f0 0x7f99540a55f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99540b77d0 0x7f99540b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 --2- 172.21.2.117:0/2041991712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99540a47d0 0x7f99540a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.930 7f99592a8700 1 -- 172.21.2.117:0/2041991712 >> 172.21.2.117:0/2041991712 conn(0x7f99540b5140 msgr2=0x7f99540a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.931 7f99592a8700 1 -- 172.21.2.117:0/2041991712 shutdown_connections 2021-02-10T11:25:08.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:08.931 7f99592a8700 1 -- 172.21.2.117:0/2041991712 wait complete. 2021-02-10T11:25:08.933 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802439 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd 2021-02-10T11:25:08.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:08.934 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:08.934 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.267 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:09.268 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:09.268 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236856 s, 44.3 MB/s 2021-02-10T11:25:09.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:09.648 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:09.649 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:09.983 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:09.983 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:09.983 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240842 s, 43.5 MB/s 2021-02-10T11:25:10.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:10.358 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:10.359 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:10.694 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:10.694 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:10.694 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:11.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:11.069 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250056 s, 41.9 MB/s 2021-02-10T11:25:11.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:11.070 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:11.070 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:11.405 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:11.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:11.406 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248248 s, 42.2 MB/s 2021-02-10T11:25:11.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:11.781 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:11.782 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:12.116 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:12.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:12.117 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233913 s, 44.8 MB/s 2021-02-10T11:25:12.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:12.487 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:12.487 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:12.822 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:12.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:12.822 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:13.197 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:13.197 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:13.197 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235934 s, 44.4 MB/s 2021-02-10T11:25:13.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:13.198 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:13.199 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:13.534 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:13.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:13.534 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:13.909 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:13.909 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:13.909 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237806 s, 44.1 MB/s 2021-02-10T11:25:13.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:13.910 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:13.910 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:14.246 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:14.246 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:14.246 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:14.619 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:14.619 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:14.619 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230804 s, 45.4 MB/s 2021-02-10T11:25:14.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:14.620 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:14.620 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:14.954 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:14.955 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:14.955 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:15.332 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:15.332 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:15.332 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259456 s, 40.4 MB/s 2021-02-10T11:25:15.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:15.333 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:15.333 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:15.667 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:15.668 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:15.668 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802439/f9e4fd8a-4036-4841-bec5-6b68d0611bcd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:16.042 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:16.042 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:16.043 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023922 s, 43.8 MB/s 2021-02-10T11:25:16.044 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802440 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 Processor -- start 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 -- start start 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f06440d1510 0x7f06440d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0644077a70 con 0x7f064407da20 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0644079400 con 0x7f0644079000 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.461 7f064bca0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064407de20 con 0x7f06440d1510 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0649abc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f064407da20 0x7f06440771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43484/0 (socket says 172.21.2.117:43484) 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0649abc700 1 -- 172.21.2.117:0/78269108 learned_addr learned my addr 172.21.2.117:0/78269108 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:16.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706043671 0 0) 0x7f06400010b0 con 0x7f064407da20 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0630002fe0 con 0x7f064407da20 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f064a2bd700 1 --2- 172.21.2.117:0/78269108 >> v2:172.21.2.125:3300/0 conn(0x7f06440d1510 0x7f06440d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 773825776 0 0) 0x7f06400010b0 con 0x7f064407da20 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 >> v2:172.21.2.125:3300/0 conn(0x7f06440d1510 msgr2=0x7f06440d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 --2- 172.21.2.117:0/78269108 >> v2:172.21.2.125:3300/0 conn(0x7f06440d1510 0x7f06440d1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 >> v1:172.21.2.125:6789/0 conn(0x7f0644079000 legacy=0x7f06440e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06441b00d0 con 0x7f064407da20 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f06400015b0 con 0x7f064407da20 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 -- 172.21.2.117:0/78269108 >> v1:172.21.2.117:6789/0 conn(0x7f064407da20 legacy=0x7f06440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0630003b10 0x7f0630005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.462 7f0648aba700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 0x7f0630008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300095c0 0x7f063000bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063000c310 con 0x7f0630006840 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063000c530 con 0x7f0630003b10 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063000c750 con 0x7f06300095c0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 0x7f0630008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0649abc700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0630003b10 0x7f0630005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f064a2bd700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300095c0 0x7f063000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 -- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300095c0 msgr2=0x7f063000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300095c0 0x7f063000bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 -- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0630003b10 msgr2=0x7f0630005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0630003b10 0x7f0630005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 -- 172.21.2.117:0/78269108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06300509c0 con 0x7f0630006840 2021-02-10T11:25:16.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f064a2bd700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300095c0 0x7f063000bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f064a2bd700 1 -- 172.21.2.117:0/78269108 reap_dead start 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f06492bb700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 0x7f0630008d40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f0638006ac0 tx=0x7f0638009af0).ready entity=mon.0 client_cookie=7efcaa6c8040935d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0638007280 con 0x7f0630006840 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0638006760 con 0x7f0630006840 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f0648aba700 1 -- 172.21.2.117:0/78269108 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06380009b0 con 0x7f0630006840 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f064bca0700 1 -- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 msgr2=0x7f0630008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.463 7f064bca0700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 0x7f0630008d40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f0638006ac0 tx=0x7f0638009af0).stop 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.464 7f064bca0700 1 -- 172.21.2.117:0/78269108 shutdown_connections 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.464 7f064bca0700 1 --2- 172.21.2.117:0/78269108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0630006840 0x7f0630008d40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.464 7f064bca0700 1 -- 172.21.2.117:0/78269108 >> 172.21.2.117:0/78269108 conn(0x7f06440ce430 msgr2=0x7f06440e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.464 7f064bca0700 1 -- 172.21.2.117:0/78269108 shutdown_connections 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.464 7f064bca0700 1 -- 172.21.2.117:0/78269108 wait complete. 2021-02-10T11:25:16.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 Processor -- start 2021-02-10T11:25:16.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 -- start start 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06440959f0 0x7f0644094aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06440951c0 0x7f06441903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0644190a00 con 0x7f06440a4a20 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06440955c0 con 0x7f06440951c0 2021-02-10T11:25:16.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.479 7f064bca0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06440957e0 con 0x7f06440959f0 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43248/0 (socket says 172.21.2.117:43248) 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 -- 172.21.2.117:0/3892641682 learned_addr learned my addr 172.21.2.117:0/3892641682 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 -- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06440959f0 msgr2=0x7f0644094aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06440959f0 0x7f0644094aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 -- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06440951c0 msgr2=0x7f06441903e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06440951c0 0x7f06441903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 -- 172.21.2.117:0/3892641682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440a5440 con 0x7f06440a4a20 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064a2bd700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f063c00b910 tx=0x7f063c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f063c000f90 con 0x7f06440a4a20 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063c006720 con 0x7f06440a4a20 2021-02-10T11:25:16.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f063c001470 con 0x7f06440a4a20 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064bca0700 1 -- 172.21.2.117:0/3892641682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06440a5440 con 0x7f06440a4a20 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.480 7f064bca0700 1 -- 172.21.2.117:0/3892641682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06440a5440 con 0x7f06440a4a20 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.481 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f063c000df0 con 0x7f06440a4a20 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.481 7f06367fc700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 0x7f062c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.481 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f063c032cc0 con 0x7f06440a4a20 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.481 7f0649abc700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 0x7f062c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:16.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.482 7f0649abc700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 0x7f062c01ac10 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:16.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.511 7f064bca0700 1 -- 172.21.2.117:0/3892641682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06440b9290 con 0x7f06440a4a20 2021-02-10T11:25:16.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.514 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f063c00bdf0 con 0x7f06440a4a20 2021-02-10T11:25:16.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.638 7f064bca0700 1 -- 172.21.2.117:0/3892641682 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802440", "target": ["mgr", ""]}) v1 -- 0x7f06440b9290 con 0x7f062c018730 2021-02-10T11:25:16.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.663 7f06367fc700 1 -- 172.21.2.117:0/3892641682 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f06400009b0 con 0x7f062c018730 2021-02-10T11:25:16.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 msgr2=0x7f062c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 0x7f062c01ac10 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 msgr2=0x7f06440a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f063c00b910 tx=0x7f063c00bcb0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 shutdown_connections 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f062c018730 0x7f062c01ac10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06440a4a20 0x7f06440a4e20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06440951c0 0x7f06441903e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 --2- 172.21.2.117:0/3892641682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06440959f0 0x7f0644094aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 >> 172.21.2.117:0/3892641682 conn(0x7f06440a90a0 msgr2=0x7f06440a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 shutdown_connections 2021-02-10T11:25:16.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:16.672 7f064bca0700 1 -- 172.21.2.117:0/3892641682 wait complete. 2021-02-10T11:25:16.675 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802440 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.091 7f2d6f0d6700 1 Processor -- start 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.091 7f2d6f0d6700 1 -- start start 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.091 7f2d6f0d6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2d68079000 0x7f2d680e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.091 7f2d6f0d6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d680d1d20 con 0x7f2d681b00f0 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d6f0d6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d68077a70 con 0x7f2d6807da20 2021-02-10T11:25:17.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d6f0d6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680e53c0 con 0x7f2d68079000 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d6d6f3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2d681b00f0 0x7f2d680d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43502/0 (socket says 172.21.2.117:43502) 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d6d6f3700 1 -- 172.21.2.117:0/739128958 learned_addr learned my addr 172.21.2.117:0/739128958 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113862577 0 0) 0x7f2d60001180 con 0x7f2d681b00f0 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54002fe0 con 0x7f2d681b00f0 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2613341094 0 0) 0x7f2d60000ef0 con 0x7f2d681b00f0 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 >> v2:172.21.2.125:3300/0 conn(0x7f2d68079000 msgr2=0x7f2d680e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:17.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 --2- 172.21.2.117:0/739128958 >> v2:172.21.2.125:3300/0 conn(0x7f2d68079000 0x7f2d680e4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 >> v1:172.21.2.125:6789/0 conn(0x7f2d6807da20 legacy=0x7f2d680771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68079400 con 0x7f2d681b00f0 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2d600017d0 con 0x7f2d681b00f0 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 >> v1:172.21.2.117:6789/0 conn(0x7f2d681b00f0 legacy=0x7f2d680d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d54003a60 0x7f2d54005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d54006780 0x7f2d54008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 0x7f2d5400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5400c2f0 con 0x7f2d540095a0 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5400c510 con 0x7f2d54003a60 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d677fe700 1 -- 172.21.2.117:0/739128958 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5400c730 con 0x7f2d54006780 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.092 7f2d6d6f3700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 0x7f2d5400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d67fff700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d54006780 0x7f2d54008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6cef2700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d54003a60 0x7f2d54005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:17.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 -- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d54006780 msgr2=0x7f2d54008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d54006780 0x7f2d54008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 -- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d54003a60 msgr2=0x7f2d54005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d54003a60 0x7f2d54005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 -- 172.21.2.117:0/739128958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5400c9d0 con 0x7f2d540095a0 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6cef2700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d54003a60 0x7f2d54005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 -- 172.21.2.117:0/739128958 reap_dead start 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6d6f3700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 0x7f2d5400baa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2d60001450 tx=0x7f2d60007db0).ready entity=mon.0 client_cookie=b2e681a7aaff448 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d60001070 con 0x7f2d540095a0 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60007020 con 0x7f2d540095a0 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d677fe700 1 -- 172.21.2.117:0/739128958 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d600092e0 con 0x7f2d540095a0 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6f0d6700 1 -- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 msgr2=0x7f2d5400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.093 7f2d6f0d6700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 0x7f2d5400baa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2d60001450 tx=0x7f2d60007db0).stop 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.094 7f2d6f0d6700 1 -- 172.21.2.117:0/739128958 shutdown_connections 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.094 7f2d6f0d6700 1 --2- 172.21.2.117:0/739128958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d540095a0 0x7f2d5400baa0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.094 7f2d6f0d6700 1 -- 172.21.2.117:0/739128958 >> 172.21.2.117:0/739128958 conn(0x7f2d680ce450 msgr2=0x7f2d680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.094 7f2d6f0d6700 1 -- 172.21.2.117:0/739128958 shutdown_connections 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.094 7f2d6f0d6700 1 -- 172.21.2.117:0/739128958 wait complete. 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.108 7f2d6f0d6700 1 Processor -- start 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.108 7f2d6f0d6700 1 -- start start 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d680a7820 0x7f2d68092030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d68092650 0x7f2d68093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68092a50 con 0x7f2d680a7c20 2021-02-10T11:25:17.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68092c80 con 0x7f2d68092650 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d6f0d6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680a8340 con 0x7f2d680a7820 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43268/0 (socket says 172.21.2.117:43268) 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 -- 172.21.2.117:0/2966016513 learned_addr learned my addr 172.21.2.117:0/2966016513 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 -- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d680a7820 msgr2=0x7f2d68092030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d680a7820 0x7f2d68092030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 -- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d68092650 msgr2=0x7f2d68093820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d68092650 0x7f2d68093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.109 7f2d67fff700 1 -- 172.21.2.117:0/2966016513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680a8520 con 0x7f2d680a7c20 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d67fff700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2d58009980 tx=0x7f2d5800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d58003030 con 0x7f2d680a7c20 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d58001290 con 0x7f2d680a7c20 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d680a8520 con 0x7f2d680a7c20 2021-02-10T11:25:17.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d68094130 con 0x7f2d680a7c20 2021-02-10T11:25:17.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d58001720 con 0x7f2d680a7c20 2021-02-10T11:25:17.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d58001470 con 0x7f2d680a7c20 2021-02-10T11:25:17.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 0x7f2d4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:17.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.110 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2d58032a50 con 0x7f2d680a7c20 2021-02-10T11:25:17.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.111 7f2d6cef2700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 0x7f2d4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:17.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.112 7f2d6cef2700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 0x7f2d4c01abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:17.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.141 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d68092a50 con 0x7f2d680a7c20 2021-02-10T11:25:17.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.144 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d58020ed0 con 0x7f2d680a7c20 2021-02-10T11:25:17.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.271 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802440", "target": ["mgr", ""]}) v1 -- 0x7f2d68094130 con 0x7f2d4c0186c0 2021-02-10T11:25:17.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.275 7f2d657fa700 1 -- 172.21.2.117:0/2966016513 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2d50005330 con 0x7f2d4c0186c0 2021-02-10T11:25:17.273 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39 2021-02-10T11:25:17.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 msgr2=0x7f2d4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 0x7f2d4c01abc0 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 msgr2=0x7f2d68093200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2d58009980 tx=0x7f2d5800bcd0).stop 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 shutdown_connections 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d4c0186c0 0x7f2d4c01abc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d680a7c20 0x7f2d68093200 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d68092650 0x7f2d68093820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:17.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 --2- 172.21.2.117:0/2966016513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d680a7820 0x7f2d68092030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:17.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 >> 172.21.2.117:0/2966016513 conn(0x7f2d680a9210 msgr2=0x7f2d680a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:17.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 shutdown_connections 2021-02-10T11:25:17.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:17.281 7f2d6f0d6700 1 -- 172.21.2.117:0/2966016513 wait complete. 2021-02-10T11:25:17.284 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802440 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39 2021-02-10T11:25:17.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:17.284 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:17.284 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:17.629 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:17.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:17.630 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:18.006 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:18.007 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:18.007 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233613 s, 44.9 MB/s 2021-02-10T11:25:18.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.008 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.008 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:18.343 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:18.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:18.343 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240292 s, 43.6 MB/s 2021-02-10T11:25:18.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:18.717 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:18.718 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:19.052 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:19.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:19.053 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:19.426 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:19.426 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:19.426 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229036 s, 45.8 MB/s 2021-02-10T11:25:19.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:19.428 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:19.428 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:19.761 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:19.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:19.762 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022805 s, 46.0 MB/s 2021-02-10T11:25:20.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:20.136 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:20.136 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:20.470 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:20.471 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:20.471 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243032 s, 43.1 MB/s 2021-02-10T11:25:20.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:20.846 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:20.846 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:21.181 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:21.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:21.182 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:21.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:21.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:21.555 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230093 s, 45.6 MB/s 2021-02-10T11:25:21.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:21.556 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:21.556 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:21.892 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:21.892 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:21.892 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:22.265 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:22.265 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:22.265 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231011 s, 45.4 MB/s 2021-02-10T11:25:22.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:22.266 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:22.266 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:22.602 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:22.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:22.602 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:22.982 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:22.982 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:22.982 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252488 s, 41.5 MB/s 2021-02-10T11:25:22.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:22.983 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:22.983 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:23.318 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:23.319 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:23.319 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:23.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:23.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:23.696 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260501 s, 40.3 MB/s 2021-02-10T11:25:23.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:23.697 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:23.697 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:24.031 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:24.032 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:24.032 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802440/d16dea47-f62d-4205-91e5-012893d6ac39/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:24.330 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:24.333 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:24.404 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:24.404 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:24.404 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230792 s, 45.4 MB/s 2021-02-10T11:25:24.405 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802441 2021-02-10T11:25:24.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.750 7f7372846700 1 Processor -- start 2021-02-10T11:25:24.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.750 7f7372846700 1 -- start start 2021-02-10T11:25:24.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7372846700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c07da20 0x7f736c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7372846700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f736c0e4d80 0x7f736c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7372846700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736c077a70 con 0x7f736c0e5180 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7372846700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736c0e5c00 con 0x7f736c0e4d80 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7372846700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0777d0 con 0x7f736c07da20 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7370e63700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f736c0e5180 0x7f736c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43522/0 (socket says 172.21.2.117:43522) 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f7370e63700 1 -- 172.21.2.117:0/2457593422 learned_addr learned my addr 172.21.2.117:0/2457593422 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946846513 0 0) 0x7f7364001500 con 0x7f736c0e5180 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736bfff700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c07da20 0x7f736c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7354002fe0 con 0x7f736c0e5180 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736b7fe700 1 --2- 172.21.2.117:0/2457593422 >> v2:172.21.2.125:3300/0 conn(0x7f736c0e4d80 0x7f736c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 528219764 0 0) 0x7f7364001500 con 0x7f736c0e5180 2021-02-10T11:25:24.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c07da20 msgr2=0x7f736c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c07da20 0x7f736c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 >> v2:172.21.2.125:3300/0 conn(0x7f736c0e4d80 msgr2=0x7f736c1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 --2- 172.21.2.117:0/2457593422 >> v2:172.21.2.125:3300/0 conn(0x7f736c0e4d80 0x7f736c1b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c07de20 con 0x7f736c0e5180 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7364001a00 con 0x7f736c0e5180 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 -- 172.21.2.117:0/2457593422 >> v1:172.21.2.117:6789/0 conn(0x7f736c0e5180 legacy=0x7f736c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7354003af0 0x7f7354006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.751 7f736affd700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f735404a7b0 0x7f735404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f735404d590 0x7f735404fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540502d0 con 0x7f7354003af0 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540504f0 con 0x7f735404a7b0 2021-02-10T11:25:24.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7354050710 con 0x7f735404d590 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7354003af0 0x7f7354006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736b7fe700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f735404a7b0 0x7f735404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f7370e63700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f735404d590 0x7f735404fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 -- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f735404d590 msgr2=0x7f735404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f735404d590 0x7f735404fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 -- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f735404a7b0 msgr2=0x7f735404cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f735404a7b0 0x7f735404cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 -- 172.21.2.117:0/2457593422 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540509b0 con 0x7f7354003af0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f7370e63700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f735404d590 0x7f735404fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736bfff700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7354003af0 0x7f7354006010 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f735c006ac0 tx=0x7f735c009ab0).ready entity=mon.0 client_cookie=a5ea27eb481f824e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c00c0c0 con 0x7f7354003af0 2021-02-10T11:25:24.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c00c0c0 con 0x7f7354003af0 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f736affd700 1 -- 172.21.2.117:0/2457593422 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735c00c3c0 con 0x7f7354003af0 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f7372846700 1 -- 172.21.2.117:0/2457593422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7354003af0 msgr2=0x7f7354006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f7372846700 1 --2- 172.21.2.117:0/2457593422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7354003af0 0x7f7354006010 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f735c006ac0 tx=0x7f735c009ab0).stop 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.752 7f7370e63700 1 -- 172.21.2.117:0/2457593422 reap_dead start 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.753 7f7372846700 1 -- 172.21.2.117:0/2457593422 shutdown_connections 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.753 7f7372846700 1 -- 172.21.2.117:0/2457593422 >> 172.21.2.117:0/2457593422 conn(0x7f736c0ce430 msgr2=0x7f736c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.753 7f7372846700 1 -- 172.21.2.117:0/2457593422 shutdown_connections 2021-02-10T11:25:24.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.753 7f7372846700 1 -- 172.21.2.117:0/2457593422 wait complete. 2021-02-10T11:25:24.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 Processor -- start 2021-02-10T11:25:24.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 -- start start 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c0959f0 0x7f736c094aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c0951c0 0x7f736c1903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c190a00 con 0x7f736c0a4a20 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0955c0 con 0x7f736c0959f0 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.768 7f7372846700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0957e0 con 0x7f736c0951c0 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43288/0 (socket says 172.21.2.117:43288) 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 -- 172.21.2.117:0/2502821345 learned_addr learned my addr 172.21.2.117:0/2502821345 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 -- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c0951c0 msgr2=0x7f736c1903e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c0951c0 0x7f736c1903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 -- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c0959f0 msgr2=0x7f736c094aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c0959f0 0x7f736c094aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 -- 172.21.2.117:0/2502821345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c0a5440 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7370e63700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f736400b870 tx=0x7f736400bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7364007980 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7364007da0 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73640062a0 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7372846700 1 -- 172.21.2.117:0/2502821345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c0a5440 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.769 7f7372846700 1 -- 172.21.2.117:0/2502821345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c0a59d0 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.770 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7364001270 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.770 7f7368ff9700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 0x7f734c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.770 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7364032880 con 0x7f736c0a4a20 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.770 7f736bfff700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 0x7f734c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:24.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.770 7f736bfff700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 0x7f734c01abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:24.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.801 7f7372846700 1 -- 172.21.2.117:0/2502821345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c0b9290 con 0x7f736c0a4a20 2021-02-10T11:25:24.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.803 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7364007980 con 0x7f736c0a4a20 2021-02-10T11:25:24.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.930 7f7372846700 1 -- 172.21.2.117:0/2502821345 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802441", "target": ["mgr", ""]}) v1 -- 0x7f736c0a59d0 con 0x7f734c0186c0 2021-02-10T11:25:24.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.954 7f7368ff9700 1 -- 172.21.2.117:0/2502821345 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f735c000940 con 0x7f734c0186c0 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 -- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 msgr2=0x7f734c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 0x7f734c01abc0 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 -- 172.21.2.117:0/2502821345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 msgr2=0x7f736c0a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f736400b870 tx=0x7f736400bc10).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 -- 172.21.2.117:0/2502821345 shutdown_connections 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f734c0186c0 0x7f734c01abc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c0a4a20 0x7f736c0a4e20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c0951c0 0x7f736c1903e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 --2- 172.21.2.117:0/2502821345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c0959f0 0x7f736c094aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.963 7f7372846700 1 -- 172.21.2.117:0/2502821345 >> 172.21.2.117:0/2502821345 conn(0x7f736c0a90a0 msgr2=0x7f736c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.964 7f7372846700 1 -- 172.21.2.117:0/2502821345 shutdown_connections 2021-02-10T11:25:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:24.964 7f7372846700 1 -- 172.21.2.117:0/2502821345 wait complete. 2021-02-10T11:25:24.966 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802441 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 Processor -- start 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 -- start start 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c0d1d20 con 0x7f7c2c07da20 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c077a70 con 0x7f7c2c1b00f0 2021-02-10T11:25:25.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c33104700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0e5430 con 0x7f7c2c079000 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c30f20700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7c2c07da20 0x7f7c2c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43542/0 (socket says 172.21.2.117:43542) 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c30f20700 1 -- 172.21.2.117:0/371441940 learned_addr learned my addr 172.21.2.117:0/371441940 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.385 7f7c2bfff700 1 --2- 172.21.2.117:0/371441940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2b7fe700 1 -- 172.21.2.117:0/371441940 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231667591 0 0) 0x7f7c14001180 con 0x7f7c2c07da20 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2b7fe700 1 -- 172.21.2.117:0/371441940 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c18002fe0 con 0x7f7c2c07da20 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2bfff700 1 -- 172.21.2.117:0/371441940 >> v1:172.21.2.125:6789/0 conn(0x7f7c2c1b00f0 legacy=0x7f7c2c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2bfff700 1 -- 172.21.2.117:0/371441940 >> v1:172.21.2.117:6789/0 conn(0x7f7c2c07da20 legacy=0x7f7c2c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2bfff700 1 -- 172.21.2.117:0/371441940 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c079400 con 0x7f7c2c079000 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2bfff700 1 --2- 172.21.2.117:0/371441940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 0x7f7c2c0e4da0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7c1c006650 tx=0x7f7c1c009a00).ready entity=mon.2 client_cookie=a0d9a5925ea64462 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2b7fe700 1 -- 172.21.2.117:0/371441940 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c0009b0 con 0x7f7c2c079000 2021-02-10T11:25:25.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2b7fe700 1 -- 172.21.2.117:0/371441940 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c00c0c0 con 0x7f7c2c079000 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.386 7f7c2b7fe700 1 -- 172.21.2.117:0/371441940 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c1c00c5e0 con 0x7f7c2c079000 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 -- 172.21.2.117:0/371441940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 msgr2=0x7f7c2c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 --2- 172.21.2.117:0/371441940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 0x7f7c2c0e4da0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7c1c006650 tx=0x7f7c1c009a00).stop 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 -- 172.21.2.117:0/371441940 shutdown_connections 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 --2- 172.21.2.117:0/371441940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 -- 172.21.2.117:0/371441940 >> 172.21.2.117:0/371441940 conn(0x7f7c2c0ce450 msgr2=0x7f7c2c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 -- 172.21.2.117:0/371441940 shutdown_connections 2021-02-10T11:25:25.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.387 7f7c33104700 1 -- 172.21.2.117:0/371441940 wait complete. 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 Processor -- start 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 -- start start 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c0b1340 0x7f7c2c18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c18b830 0x7f7c2c187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0b18c0 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c18ac90 con 0x7f7c2c18b830 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c33104700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c107f50 con 0x7f7c2c0b1340 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c30f20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c30f20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43298/0 (socket says 172.21.2.117:43298) 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c30f20700 1 -- 172.21.2.117:0/2088259464 learned_addr learned my addr 172.21.2.117:0/2088259464 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c2bfff700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c0b1340 0x7f7c2c18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.402 7f7c31721700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c18b830 0x7f7c2c187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 -- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c0b1340 msgr2=0x7f7c2c18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c0b1340 0x7f7c2c18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 -- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c18b830 msgr2=0x7f7c2c187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c18b830 0x7f7c2c187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 -- 172.21.2.117:0/2088259464 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c108130 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c30f20700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7c14006f00 tx=0x7f7c1400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c14006020 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c14006320 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c14004620 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c33104700 1 -- 172.21.2.117:0/2088259464 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c108130 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.403 7f7c33104700 1 -- 172.21.2.117:0/2088259464 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c108130 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.404 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c1400a000 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.404 7f7c297fa700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:25.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.404 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7c14032930 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.404 7f7c2bfff700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:25.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.404 7f7c2bfff700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 0x7f7c1001abc0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.435 7f7c33104700 1 -- 172.21.2.117:0/2088259464 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c077a70 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.437 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c14006300 con 0x7f7c2c0b0f40 2021-02-10T11:25:25.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.564 7f7c33104700 1 -- 172.21.2.117:0/2088259464 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802441", "target": ["mgr", ""]}) v1 -- 0x7f7c2c077a70 con 0x7f7c100186c0 2021-02-10T11:25:25.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.568 7f7c297fa700 1 -- 172.21.2.117:0/2088259464 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7c1c0067b0 con 0x7f7c100186c0 2021-02-10T11:25:25.566 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0 2021-02-10T11:25:25.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 msgr2=0x7f7c1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 0x7f7c1001abc0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 msgr2=0x7f7c2c107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:25.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7c14006f00 tx=0x7f7c1400b940).stop 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 shutdown_connections 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c18b830 0x7f7c2c187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c0b1340 0x7f7c2c18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c100186c0 0x7f7c1001abc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 --2- 172.21.2.117:0/2088259464 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b0f40 0x7f7c2c107930 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 >> 172.21.2.117:0/2088259464 conn(0x7f7c2c0a1880 msgr2=0x7f7c2c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:25.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 shutdown_connections 2021-02-10T11:25:25.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:25.574 7f7c33104700 1 -- 172.21.2.117:0/2088259464 wait complete. 2021-02-10T11:25:25.577 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802441 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0 2021-02-10T11:25:25.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:25.578 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:25.578 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:25.913 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:25.913 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:25.913 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:26.294 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:26.294 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:26.294 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255396 s, 41.1 MB/s 2021-02-10T11:25:26.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:26.295 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:26.295 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:26.629 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:26.630 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:26.630 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:27.005 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:27.005 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:27.005 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023331 s, 44.9 MB/s 2021-02-10T11:25:27.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:27.006 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:27.006 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:27.341 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:27.342 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:27.342 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023402 s, 44.8 MB/s 2021-02-10T11:25:27.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:27.712 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:27.712 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:28.047 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:28.048 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:28.048 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:28.421 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:28.421 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:28.421 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022904 s, 45.8 MB/s 2021-02-10T11:25:28.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:28.422 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:28.422 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:28.756 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:28.757 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:28.757 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258214 s, 40.6 MB/s 2021-02-10T11:25:29.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:29.134 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:29.134 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:29.468 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:29.469 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:29.469 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:29.842 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:29.842 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:29.842 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230345 s, 45.5 MB/s 2021-02-10T11:25:29.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:29.843 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:29.843 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:30.179 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:30.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:30.179 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:30.553 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:30.553 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239499 s, 43.8 MB/s 2021-02-10T11:25:30.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:30.554 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:30.554 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:30.888 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:30.889 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:30.889 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:31.262 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:31.262 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:31.262 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229925 s, 45.6 MB/s 2021-02-10T11:25:31.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:31.263 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:31.263 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:31.597 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:31.598 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:31.598 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:31.971 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:31.971 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:31.971 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223937 s, 46.8 MB/s 2021-02-10T11:25:31.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:31.972 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:31.972 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:32.307 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:32.308 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:32.308 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802441/cfca44af-f433-4e8e-96af-8e70ee9589a0/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245724 s, 42.7 MB/s 2021-02-10T11:25:32.681 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802442 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 Processor -- start 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 -- start start 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6d90079000 0x7f6d900e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900d1d20 con 0x7f6d901b00f0 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90077a70 con 0x7f6d9007da20 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d94acb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d900e53c0 con 0x7f6d90079000 2021-02-10T11:25:33.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d8f05f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6d901b00f0 0x7f6d900d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43554/0 (socket says 172.21.2.117:43554) 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.098 7f6d8f05f700 1 -- 172.21.2.117:0/1632738881 learned_addr learned my addr 172.21.2.117:0/1632738881 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797218349 0 0) 0x7f6d88001180 con 0x7f6d901b00f0 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c002fe0 con 0x7f6d901b00f0 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1239320199 0 0) 0x7f6d880010e0 con 0x7f6d901b00f0 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 >> v2:172.21.2.125:3300/0 conn(0x7f6d90079000 msgr2=0x7f6d900e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 --2- 172.21.2.117:0/1632738881 >> v2:172.21.2.125:3300/0 conn(0x7f6d90079000 0x7f6d900e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 >> v1:172.21.2.125:6789/0 conn(0x7f6d9007da20 legacy=0x7f6d900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d90079400 con 0x7f6d901b00f0 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6d88001600 con 0x7f6d901b00f0 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 >> v1:172.21.2.117:6789/0 conn(0x7f6d901b00f0 legacy=0x7f6d900d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d7c003aa0 0x7f6d7c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 0x7f6d7c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d7c009590 0x7f6d7c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c00c300 con 0x7f6d7c006790 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c00c520 con 0x7f6d7c003aa0 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c00c740 con 0x7f6d7c009590 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.099 7f6d8605d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 0x7f6d7c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8e85e700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d7c003aa0 0x7f6d7c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8f05f700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d7c009590 0x7f6d7c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 -- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d7c009590 msgr2=0x7f6d7c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d7c009590 0x7f6d7c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 -- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d7c003aa0 msgr2=0x7f6d7c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d7c003aa0 0x7f6d7c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 -- 172.21.2.117:0/1632738881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c00c9e0 con 0x7f6d7c006790 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8f05f700 1 -- 172.21.2.117:0/1632738881 reap_dead start 2021-02-10T11:25:33.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8605d700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 0x7f6d7c008c90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6d78006bf0 tx=0x7f6d7804fcd0).ready entity=mon.0 client_cookie=6864d5ce468feadf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d780009b0 con 0x7f6d7c006790 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78002030 con 0x7f6d7c006790 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d8e05d700 1 -- 172.21.2.117:0/1632738881 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d78002550 con 0x7f6d7c006790 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d94acb700 1 -- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 msgr2=0x7f6d7c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.100 7f6d94acb700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 0x7f6d7c008c90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f6d78006bf0 tx=0x7f6d7804fcd0).stop 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.101 7f6d94acb700 1 -- 172.21.2.117:0/1632738881 shutdown_connections 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.101 7f6d94acb700 1 --2- 172.21.2.117:0/1632738881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d7c006790 0x7f6d7c008c90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.101 7f6d94acb700 1 -- 172.21.2.117:0/1632738881 >> 172.21.2.117:0/1632738881 conn(0x7f6d900ce450 msgr2=0x7f6d900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.101 7f6d94acb700 1 -- 172.21.2.117:0/1632738881 shutdown_connections 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.101 7f6d94acb700 1 -- 172.21.2.117:0/1632738881 wait complete. 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.115 7f6d94acb700 1 Processor -- start 2021-02-10T11:25:33.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.115 7f6d94acb700 1 -- start start 2021-02-10T11:25:33.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 0x7f6d90190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d900a50e0 0x7f6d900a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900c3040 con 0x7f6d900a46c0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d90190a20 con 0x7f6d901a46b0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d94acb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900a54e0 con 0x7f6d900a50e0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d8605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d8605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43314/0 (socket says 172.21.2.117:43314) 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d8605d700 1 -- 172.21.2.117:0/1561547831 learned_addr learned my addr 172.21.2.117:0/1561547831 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d8f05f700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d900a50e0 0x7f6d900a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.116 7f6d8e85e700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 0x7f6d90190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 -- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d900a50e0 msgr2=0x7f6d900a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d900a50e0 0x7f6d900a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 -- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 msgr2=0x7f6d90190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 0x7f6d90190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 -- 172.21.2.117:0/1561547831 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900a5700 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8e85e700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 0x7f6d90190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d8605d700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6d7804f540 tx=0x7f6d780099d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d78000c90 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78006020 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d900a5700 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d78006430 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.117 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d900a6990 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.118 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d78006200 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.118 7f6d87fff700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.118 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6d780766f0 con 0x7f6d900a46c0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.119 7f6d8e85e700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.120 7f6d8e85e700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.148 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d90077a70 con 0x7f6d900a46c0 2021-02-10T11:25:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.151 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d78009dc0 con 0x7f6d900a46c0 2021-02-10T11:25:33.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.276 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802442", "target": ["mgr", ""]}) v1 -- 0x7f6d900c3040 con 0x7f6d740186c0 2021-02-10T11:25:33.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.302 7f6d87fff700 1 -- 172.21.2.117:0/1561547831 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6d80006c40 con 0x7f6d740186c0 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 msgr2=0x7f6d7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 msgr2=0x7f6d900a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6d7804f540 tx=0x7f6d780099d0).stop 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 shutdown_connections 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d900a46c0 0x7f6d900a4ac0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d900a50e0 0x7f6d900a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 --2- 172.21.2.117:0/1561547831 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d901a46b0 0x7f6d90190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 >> 172.21.2.117:0/1561547831 conn(0x7f6d900b5140 msgr2=0x7f6d900a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 shutdown_connections 2021-02-10T11:25:33.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.310 7f6d94acb700 1 -- 172.21.2.117:0/1561547831 wait complete. 2021-02-10T11:25:33.313 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802442 2021-02-10T11:25:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.731 7fd2464fc700 1 Processor -- start 2021-02-10T11:25:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.731 7fd2464fc700 1 -- start start 2021-02-10T11:25:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd2464fc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd24007da20 0x7fd2400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd2464fc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2400d1d20 con 0x7fd2400e51a0 2021-02-10T11:25:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd2464fc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd240077a70 con 0x7fd2400e4da0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd2464fc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2400777d0 con 0x7fd24007da20 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd244b19700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd2400e51a0 0x7fd2400d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43572/0 (socket says 172.21.2.117:43572) 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd244b19700 1 -- 172.21.2.117:0/390678875 learned_addr learned my addr 172.21.2.117:0/390678875 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840617186 0 0) 0x7fd238001180 con 0x7fd2400e51a0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd23effd700 1 -- 172.21.2.117:0/390678875 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c002fe0 con 0x7fd2400e51a0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 484300161 0 0) 0x7fd238001180 con 0x7fd2400e51a0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd23effd700 1 -- 172.21.2.117:0/390678875 >> v2:172.21.2.125:3300/0 conn(0x7fd24007da20 msgr2=0x7fd2400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.732 7fd23ffff700 1 --2- 172.21.2.117:0/390678875 >> v2:172.21.2.125:3300/0 conn(0x7fd24007da20 0x7fd2400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 --2- 172.21.2.117:0/390678875 >> v2:172.21.2.125:3300/0 conn(0x7fd24007da20 0x7fd2400771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 >> v1:172.21.2.125:6789/0 conn(0x7fd2400e4da0 legacy=0x7fd2401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2401b08e0 con 0x7fd2400e51a0 2021-02-10T11:25:33.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd238001680 con 0x7fd2400e51a0 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 >> v1:172.21.2.117:6789/0 conn(0x7fd2400e51a0 legacy=0x7fd2400d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd22c003ae0 0x7fd22c006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 0x7fd22c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd22c009620 0x7fd22c00bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c00c390 con 0x7fd22c006820 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c00c5b0 con 0x7fd22c003ae0 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23effd700 1 -- 172.21.2.117:0/390678875 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c00c7d0 con 0x7fd22c009620 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23ffff700 1 --2- 172.21.2.117:0/390678875 >> v2:172.21.2.125:3300/0 conn(0x7fd24007da20 0x7fd2400771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 0x7fd22c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd244b19700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd22c009620 0x7fd22c00bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23ffff700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd22c003ae0 0x7fd22c006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 -- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd22c009620 msgr2=0x7fd22c00bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd22c009620 0x7fd22c00bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 -- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd22c003ae0 msgr2=0x7fd22c006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd22c003ae0 0x7fd22c006000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 -- 172.21.2.117:0/390678875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c00ca70 con 0x7fd22c006820 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd244b19700 1 -- 172.21.2.117:0/390678875 reap_dead start 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.733 7fd23f7fe700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 0x7fd22c008d20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd228006ac0 tx=0x7fd22804db00).ready entity=mon.0 client_cookie=63ffc669ca7905d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2280500c0 con 0x7fd22c006820 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd228050400 con 0x7fd22c006820 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd23effd700 1 -- 172.21.2.117:0/390678875 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2280508e0 con 0x7fd22c006820 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 -- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 msgr2=0x7fd22c008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 0x7fd22c008d20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd228006ac0 tx=0x7fd22804db00).stop 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 -- 172.21.2.117:0/390678875 shutdown_connections 2021-02-10T11:25:33.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 --2- 172.21.2.117:0/390678875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd22c006820 0x7fd22c008d20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 -- 172.21.2.117:0/390678875 >> 172.21.2.117:0/390678875 conn(0x7fd2400ce450 msgr2=0x7fd2400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 -- 172.21.2.117:0/390678875 shutdown_connections 2021-02-10T11:25:33.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.734 7fd2464fc700 1 -- 172.21.2.117:0/390678875 wait complete. 2021-02-10T11:25:33.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.749 7fd2464fc700 1 Processor -- start 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.749 7fd2464fc700 1 -- start start 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.749 7fd2464fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd240096ef0 0x7fd2400972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2400c13b0 0x7fd2400c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2400969f0 con 0x7fd24019bf70 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd240097950 con 0x7fd240096ef0 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd240097c60 con 0x7fd2400c13b0 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43332/0 (socket says 172.21.2.117:43332) 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 -- 172.21.2.117:0/1520327090 learned_addr learned my addr 172.21.2.117:0/1520327090 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd244b19700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2400c13b0 0x7fd2400c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23f7fe700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd240096ef0 0x7fd2400972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 -- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2400c13b0 msgr2=0x7fd2400c17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2400c13b0 0x7fd2400c17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 -- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd240096ef0 msgr2=0x7fd2400972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd240096ef0 0x7fd2400972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 -- 172.21.2.117:0/1520327090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400c1dd0 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23ffff700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd230009980 tx=0x7fd23000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23000c2f0 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230000d60 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2400c1dd0 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23000c670 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.750 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2400c2360 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.751 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd230001010 con 0x7fd24019bf70 2021-02-10T11:25:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.751 7fd23cff9700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 0x7fd22401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.751 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd230032b10 con 0x7fd24019bf70 2021-02-10T11:25:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.751 7fd23f7fe700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 0x7fd22401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.752 7fd23f7fe700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 0x7fd22401abd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:33.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.781 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2400c2bf0 con 0x7fd24019bf70 2021-02-10T11:25:33.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.784 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd230006900 con 0x7fd24019bf70 2021-02-10T11:25:33.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.912 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802442", "target": ["mgr", ""]}) v1 -- 0x7fd2400c2bf0 con 0x7fd2240186d0 2021-02-10T11:25:33.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.916 7fd23cff9700 1 -- 172.21.2.117:0/1520327090 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd2280056c0 con 0x7fd2240186d0 2021-02-10T11:25:33.914 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9 2021-02-10T11:25:33.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 msgr2=0x7fd22401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 0x7fd22401abd0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 msgr2=0x7fd2400b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:33.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd230009980 tx=0x7fd23000bcd0).stop 2021-02-10T11:25:33.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 shutdown_connections 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2400c13b0 0x7fd2400c17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd240096ef0 0x7fd2400972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2240186d0 0x7fd22401abd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 --2- 172.21.2.117:0/1520327090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24019bf70 0x7fd2400b5cd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 >> 172.21.2.117:0/1520327090 conn(0x7fd240107f20 msgr2=0x7fd240187bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 shutdown_connections 2021-02-10T11:25:33.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:33.922 7fd2464fc700 1 -- 172.21.2.117:0/1520327090 wait complete. 2021-02-10T11:25:33.924 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802442 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9 2021-02-10T11:25:33.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:33.924 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:33.925 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:34.260 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:34.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:34.261 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:34.643 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:34.643 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:34.643 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261739 s, 40.1 MB/s 2021-02-10T11:25:34.644 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:34.644 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:34.644 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:34.979 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:34.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:34.980 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:35.355 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:35.355 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:35.355 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256307 s, 40.9 MB/s 2021-02-10T11:25:35.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:35.356 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:35.357 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:35.691 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:35.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:35.692 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:36.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:36.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:36.069 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255818 s, 41.0 MB/s 2021-02-10T11:25:36.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:36.070 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:36.070 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:36.404 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:36.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:36.405 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:36.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:36.781 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:36.781 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253959 s, 41.3 MB/s 2021-02-10T11:25:36.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:36.782 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:36.782 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:37.116 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:37.116 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:37.117 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:37.489 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:37.489 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:37.489 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251119 s, 41.8 MB/s 2021-02-10T11:25:37.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:37.490 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:37.490 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:37.824 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:37.825 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:37.825 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239317 s, 43.8 MB/s 2021-02-10T11:25:38.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:38.201 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:38.201 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:38.535 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:38.536 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:38.536 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:38.913 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:38.913 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:38.913 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0266674 s, 39.3 MB/s 2021-02-10T11:25:38.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:38.914 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:38.915 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:39.250 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:39.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:39.251 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:39.624 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:39.625 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:39.625 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240241 s, 43.6 MB/s 2021-02-10T11:25:39.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:39.626 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:39.626 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:39.961 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:39.962 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:39.962 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:40.334 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:40.334 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:40.334 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230037 s, 45.6 MB/s 2021-02-10T11:25:40.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:40.335 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:40.335 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:40.669 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:40.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:40.669 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802442/69df7263-e906-49fe-a7cf-e8805a25b5f9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:41.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:41.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:41.043 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024984 s, 42.0 MB/s 2021-02-10T11:25:41.044 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802443 2021-02-10T11:25:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.462 7fccce8d7700 1 Processor -- start 2021-02-10T11:25:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.462 7fccce8d7700 1 -- start start 2021-02-10T11:25:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccce8d7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccc807da20 0x7fccc80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccce8d7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fccc80e5180 0x7fccc8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccce8d7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc8077a70 con 0x7fccc80e4d80 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccce8d7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80e5c00 con 0x7fccc807da20 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccce8d7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc80777d0 con 0x7fccc80e5180 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccc7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccc807da20 0x7fccc80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccc7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccc807da20 0x7fccc80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43340/0 (socket says 172.21.2.117:43340) 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccc7fff700 1 -- 172.21.2.117:0/3675209011 learned_addr learned my addr 172.21.2.117:0/3675209011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.463 7fccccef4700 1 --2- 172.21.2.117:0/3675209011 >> v2:172.21.2.125:3300/0 conn(0x7fccc80e5180 0x7fccc8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc7fff700 1 -- 172.21.2.117:0/3675209011 >> v2:172.21.2.125:3300/0 conn(0x7fccc80e5180 msgr2=0x7fccc8079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:41.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc7fff700 1 --2- 172.21.2.117:0/3675209011 >> v2:172.21.2.125:3300/0 conn(0x7fccc80e5180 0x7fccc8079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc7fff700 1 -- 172.21.2.117:0/3675209011 >> v1:172.21.2.125:6789/0 conn(0x7fccc80e4d80 legacy=0x7fccc81b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc7fff700 1 -- 172.21.2.117:0/3675209011 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc81b08c0 con 0x7fccc807da20 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc7fff700 1 --2- 172.21.2.117:0/3675209011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc807da20 0x7fccc80771b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fccb8006700 tx=0x7fccb8009b80).ready entity=mon.0 client_cookie=56870259193b9814 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc6ffd700 1 -- 172.21.2.117:0/3675209011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb800c0c0 con 0x7fccc807da20 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc6ffd700 1 -- 172.21.2.117:0/3675209011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800c440 con 0x7fccc807da20 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccc6ffd700 1 -- 172.21.2.117:0/3675209011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccb800c960 con 0x7fccc807da20 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccce8d7700 1 -- 172.21.2.117:0/3675209011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc807da20 msgr2=0x7fccc80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.464 7fccce8d7700 1 --2- 172.21.2.117:0/3675209011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc807da20 0x7fccc80771b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fccb8006700 tx=0x7fccb8009b80).stop 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 -- 172.21.2.117:0/3675209011 shutdown_connections 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 --2- 172.21.2.117:0/3675209011 >> v2:172.21.2.125:3300/0 conn(0x7fccc80e5180 0x7fccc8079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 --2- 172.21.2.117:0/3675209011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc807da20 0x7fccc80771b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 -- 172.21.2.117:0/3675209011 >> 172.21.2.117:0/3675209011 conn(0x7fccc80ce430 msgr2=0x7fccc80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 -- 172.21.2.117:0/3675209011 shutdown_connections 2021-02-10T11:25:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.465 7fccce8d7700 1 -- 172.21.2.117:0/3675209011 wait complete. 2021-02-10T11:25:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.479 7fccce8d7700 1 Processor -- start 2021-02-10T11:25:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.479 7fccce8d7700 1 -- start start 2021-02-10T11:25:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc819db50 0x7fccc80ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80ad040 con 0x7fccc80adb60 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8091f10 con 0x7fccc8091b10 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccce8d7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80ae580 con 0x7fccc819db50 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccc7fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccc7fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:47730/0 (socket says 172.21.2.117:47730) 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43350/0 (socket says 172.21.2.117:43350) 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccc7fff700 1 -- 172.21.2.117:0/1816560367 learned_addr learned my addr 172.21.2.117:0/1816560367 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccc77fe700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc819db50 0x7fccc80ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 -- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc819db50 msgr2=0x7fccc80ad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc819db50 0x7fccc80ad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 -- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 msgr2=0x7fccc819d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccccef4700 1 -- 172.21.2.117:0/1816560367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80ae7a0 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.480 7fccc7fff700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccccef4700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fccc000b8a0 tx=0x7fccc000bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc0000f90 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc00066f0 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc0001470 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc80ae7a0 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc80abbb0 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccc00011f0 con 0x7fccc80adb60 2021-02-10T11:25:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 0x7fcca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.481 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fccc0032970 con 0x7fccc80adb60 2021-02-10T11:25:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.482 7fccc7fff700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 0x7fcca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.482 7fccc7fff700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 0x7fcca801abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.513 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc81b4ee0 con 0x7fccc80adb60 2021-02-10T11:25:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.516 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccc0000f90 con 0x7fccc80adb60 2021-02-10T11:25:41.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.642 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802443", "target": ["mgr", ""]}) v1 -- 0x7fccc8077a70 con 0x7fcca80186c0 2021-02-10T11:25:41.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.670 7fccc4ff9700 1 -- 172.21.2.117:0/1816560367 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fccb80009b0 con 0x7fcca80186c0 2021-02-10T11:25:41.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 msgr2=0x7fcca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 0x7fcca801abc0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 msgr2=0x7fccc80adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:41.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fccc000b8a0 tx=0x7fccc000bc40).stop 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 shutdown_connections 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca80186c0 0x7fcca801abc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc80adb60 0x7fccc80adf60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc819db50 0x7fccc80ad540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 --2- 172.21.2.117:0/1816560367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc8091b10 0x7fccc819d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 >> 172.21.2.117:0/1816560367 conn(0x7fccc81b9670 msgr2=0x7fccc8107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 shutdown_connections 2021-02-10T11:25:41.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:41.676 7fccce8d7700 1 -- 172.21.2.117:0/1816560367 wait complete. 2021-02-10T11:25:41.678 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802443 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 Processor -- start 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 -- start start 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 0x7ff8180e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8180d1d20 con 0x7ff81807da20 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff818077a70 con 0x7ff8181b00f0 2021-02-10T11:25:42.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff81d540700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8180e5430 con 0x7ff818079000 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff8172bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff81807da20 0x7ff8180771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43604/0 (socket says 172.21.2.117:43604) 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.097 7ff8172bf700 1 -- 172.21.2.117:0/480580271 learned_addr learned my addr 172.21.2.117:0/480580271 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff816abe700 1 --2- 172.21.2.117:0/480580271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 0x7ff8180e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff8162bd700 1 -- 172.21.2.117:0/480580271 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858505137 0 0) 0x7ff808000ce0 con 0x7ff81807da20 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff8162bd700 1 -- 172.21.2.117:0/480580271 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff804002fe0 con 0x7ff81807da20 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff8162bd700 1 -- 172.21.2.117:0/480580271 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1139252022 0 0) 0x7ff8080010e0 con 0x7ff81807da20 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff816abe700 1 -- 172.21.2.117:0/480580271 >> v1:172.21.2.125:6789/0 conn(0x7ff8181b00f0 legacy=0x7ff8180d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff816abe700 1 -- 172.21.2.117:0/480580271 >> v1:172.21.2.117:6789/0 conn(0x7ff81807da20 legacy=0x7ff8180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff816abe700 1 -- 172.21.2.117:0/480580271 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818079400 con 0x7ff818079000 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.098 7ff816abe700 1 --2- 172.21.2.117:0/480580271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 0x7ff8180e4da0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff800006650 tx=0x7ff800009a80).ready entity=mon.2 client_cookie=c869030daaee2876 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff8162bd700 1 -- 172.21.2.117:0/480580271 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8000009b0 con 0x7ff818079000 2021-02-10T11:25:42.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff8162bd700 1 -- 172.21.2.117:0/480580271 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000c0c0 con 0x7ff818079000 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff8162bd700 1 -- 172.21.2.117:0/480580271 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8000009b0 con 0x7ff818079000 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 -- 172.21.2.117:0/480580271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 msgr2=0x7ff8180e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 --2- 172.21.2.117:0/480580271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 0x7ff8180e4da0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff800006650 tx=0x7ff800009a80).stop 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 -- 172.21.2.117:0/480580271 shutdown_connections 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 --2- 172.21.2.117:0/480580271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818079000 0x7ff8180e4da0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 -- 172.21.2.117:0/480580271 >> 172.21.2.117:0/480580271 conn(0x7ff8180ce450 msgr2=0x7ff8180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 -- 172.21.2.117:0/480580271 shutdown_connections 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.099 7ff81d540700 1 -- 172.21.2.117:0/480580271 wait complete. 2021-02-10T11:25:42.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 Processor -- start 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 -- start start 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8180b3ff0 0x7ff8180b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818090de0 0x7ff8180911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818090600 con 0x7ff8180c33d0 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818090860 con 0x7ff8180b3ff0 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.114 7ff81d540700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8180b3c70 con 0x7ff818090de0 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43362/0 (socket says 172.21.2.117:43362) 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 -- 172.21.2.117:0/1474595849 learned_addr learned my addr 172.21.2.117:0/1474595849 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:42.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff817ac0700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818090de0 0x7ff8180911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff8172bf700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8180b3ff0 0x7ff8180b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 -- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818090de0 msgr2=0x7ff8180911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818090de0 0x7ff8180911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 -- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8180b3ff0 msgr2=0x7ff8180b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8180b3ff0 0x7ff8180b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 -- 172.21.2.117:0/1474595849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818091800 con 0x7ff8180c33d0 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.115 7ff816abe700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff800009650 tx=0x7ff80000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8000093a0 con 0x7ff8180c33d0 2021-02-10T11:25:42.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000bde0 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8000052e0 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff81d540700 1 -- 172.21.2.117:0/1474595849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff818091800 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff81d540700 1 -- 172.21.2.117:0/1474595849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818091800 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff800006030 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 0x7ff7fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.116 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8000327b0 con 0x7ff8180c33d0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.117 7ff8172bf700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 0x7ff7fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.118 7ff8172bf700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 0x7ff7fc01ac10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.147 7ff81d540700 1 -- 172.21.2.117:0/1474595849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818103370 con 0x7ff8180c33d0 2021-02-10T11:25:42.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.149 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff800000940 con 0x7ff8180c33d0 2021-02-10T11:25:42.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.277 7ff81d540700 1 -- 172.21.2.117:0/1474595849 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802443", "target": ["mgr", ""]}) v1 -- 0x7ff818077a70 con 0x7ff7fc018730 2021-02-10T11:25:42.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.281 7ff80ffff700 1 -- 172.21.2.117:0/1474595849 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff808007180 con 0x7ff7fc018730 2021-02-10T11:25:42.279 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825 2021-02-10T11:25:42.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 -- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 msgr2=0x7ff7fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 0x7ff7fc01ac10 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 -- 172.21.2.117:0/1474595849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 msgr2=0x7ff8180b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff800009650 tx=0x7ff80000b960).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 -- 172.21.2.117:0/1474595849 shutdown_connections 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7fc018730 0x7ff7fc01ac10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8180c33d0 0x7ff8180b3650 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff818090de0 0x7ff8180911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 --2- 172.21.2.117:0/1474595849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8180b3ff0 0x7ff8180b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:42.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.286 7ff81d540700 1 -- 172.21.2.117:0/1474595849 >> 172.21.2.117:0/1474595849 conn(0x7ff8180a1420 msgr2=0x7ff81809efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:42.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.287 7ff81d540700 1 -- 172.21.2.117:0/1474595849 shutdown_connections 2021-02-10T11:25:42.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:42.287 7ff81d540700 1 -- 172.21.2.117:0/1474595849 wait complete. 2021-02-10T11:25:42.289 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802443 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825 2021-02-10T11:25:42.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:42.289 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:42.289 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:42.624 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:42.624 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:42.625 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:43.001 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:43.001 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:43.001 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233308 s, 44.9 MB/s 2021-02-10T11:25:43.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:43.002 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:43.002 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:43.338 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:43.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:43.339 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:43.712 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:43.713 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:43.713 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233372 s, 44.9 MB/s 2021-02-10T11:25:43.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:43.714 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:43.714 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:44.048 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:44.048 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:44.048 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262803 s, 39.9 MB/s 2021-02-10T11:25:44.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:44.425 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:44.425 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:44.760 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:44.760 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:44.761 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:45.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:45.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243605 s, 43.0 MB/s 2021-02-10T11:25:45.136 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:45.136 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:45.136 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:45.469 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:45.469 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:45.470 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:45.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:45.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:45.844 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244965 s, 42.8 MB/s 2021-02-10T11:25:45.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:45.845 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:45.845 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:46.179 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:46.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:46.180 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:46.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:46.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:46.556 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253345 s, 41.4 MB/s 2021-02-10T11:25:46.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:46.557 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:46.557 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:46.891 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:46.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:46.891 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:47.267 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:47.267 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:47.268 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252372 s, 41.5 MB/s 2021-02-10T11:25:47.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:47.269 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:47.269 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:47.602 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:47.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:47.603 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:47.975 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:47.976 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:47.976 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261524 s, 40.1 MB/s 2021-02-10T11:25:47.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:47.977 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:47.977 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:48.312 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:48.312 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:48.313 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:48.686 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:48.686 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:48.686 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233623 s, 44.9 MB/s 2021-02-10T11:25:48.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:48.687 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:48.687 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:49.021 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:49.021 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:49.022 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802443/4e642b76-2f16-4d96-a135-be92de88a825/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:49.395 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:49.395 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:49.395 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242735 s, 43.2 MB/s 2021-02-10T11:25:49.397 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802444 2021-02-10T11:25:49.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.815 7ff93cabe700 1 Processor -- start 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.815 7ff93cabe700 1 -- start start 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.815 7ff93cabe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93807da20 0x7ff9380771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.815 7ff93cabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9380e5180 0x7ff9380d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93cabe700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff938077a70 con 0x7ff9380e4d80 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93cabe700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9380e5c00 con 0x7ff9380e5180 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93cabe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380777d0 con 0x7ff93807da20 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9380e5180 0x7ff9380d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9380e5180 0x7ff9380d0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43370/0 (socket says 172.21.2.117:43370) 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 -- 172.21.2.117:0/2477628772 learned_addr learned my addr 172.21.2.117:0/2477628772 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 -- 172.21.2.117:0/2477628772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93807da20 msgr2=0x7ff9380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:49.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 --2- 172.21.2.117:0/2477628772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93807da20 0x7ff9380771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 -- 172.21.2.117:0/2477628772 >> v1:172.21.2.125:6789/0 conn(0x7ff9380e4d80 legacy=0x7ff9381b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 -- 172.21.2.117:0/2477628772 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9381b08c0 con 0x7ff9380e5180 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93705f700 1 --2- 172.21.2.117:0/2477628772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380e5180 0x7ff9380d0500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff930006b80 tx=0x7ff930009b90).ready entity=mon.0 client_cookie=60e06ccad88cbda6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93585c700 1 -- 172.21.2.117:0/2477628772 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9300022e0 con 0x7ff9380e5180 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93585c700 1 -- 172.21.2.117:0/2477628772 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9300025a0 con 0x7ff9380e5180 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93585c700 1 -- 172.21.2.117:0/2477628772 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff930002a80 con 0x7ff9380e5180 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93cabe700 1 -- 172.21.2.117:0/2477628772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380e5180 msgr2=0x7ff9380d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.816 7ff93cabe700 1 --2- 172.21.2.117:0/2477628772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380e5180 0x7ff9380d0500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff930006b80 tx=0x7ff930009b90).stop 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 -- 172.21.2.117:0/2477628772 shutdown_connections 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 --2- 172.21.2.117:0/2477628772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380e5180 0x7ff9380d0500 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 --2- 172.21.2.117:0/2477628772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93807da20 0x7ff9380771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 -- 172.21.2.117:0/2477628772 >> 172.21.2.117:0/2477628772 conn(0x7ff9380ce430 msgr2=0x7ff9380e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:49.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 -- 172.21.2.117:0/2477628772 shutdown_connections 2021-02-10T11:25:49.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.817 7ff93cabe700 1 -- 172.21.2.117:0/2477628772 wait complete. 2021-02-10T11:25:49.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 Processor -- start 2021-02-10T11:25:49.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 -- start start 2021-02-10T11:25:49.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 0x7ff9380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 0x7ff93809c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380a77c0 con 0x7ff9380a39b0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380a5360 con 0x7ff9380a3db0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93cabe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9380a5580 con 0x7ff93809c0c0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43374/0 (socket says 172.21.2.117:43374) 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.832 7ff93685e700 1 -- 172.21.2.117:0/28159077 learned_addr learned my addr 172.21.2.117:0/28159077 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 -- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 msgr2=0x7ff93809c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93705f700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 0x7ff93809c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 0x7ff93809c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93605d700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 0x7ff9380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:49.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 -- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 msgr2=0x7ff9380a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 0x7ff9380a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 -- 172.21.2.117:0/28159077 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93809cae0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93705f700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 0x7ff93809c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93605d700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 0x7ff9380a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93685e700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff920009980 tx=0x7ff92000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9200066d0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92000c2f0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff92000c780 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93cabe700 1 -- 172.21.2.117:0/28159077 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93809cae0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.833 7ff93cabe700 1 -- 172.21.2.117:0/28159077 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93809cae0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.834 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff920001010 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.834 7ff92f7fe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.834 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff920032ca0 con 0x7ff9380a39b0 2021-02-10T11:25:49.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.834 7ff93605d700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:49.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.835 7ff93605d700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 0x7ff91c01abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:49.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.865 7ff93cabe700 1 -- 172.21.2.117:0/28159077 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9380a5360 con 0x7ff9380a39b0 2021-02-10T11:25:49.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.868 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff92000be10 con 0x7ff9380a39b0 2021-02-10T11:25:49.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:49.993 7ff93cabe700 1 -- 172.21.2.117:0/28159077 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802444", "target": ["mgr", ""]}) v1 -- 0x7ff9380a77c0 con 0x7ff91c0186c0 2021-02-10T11:25:50.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.018 7ff92f7fe700 1 -- 172.21.2.117:0/28159077 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff928006b20 con 0x7ff91c0186c0 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 msgr2=0x7ff91c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 0x7ff91c01abc0 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 msgr2=0x7ff9380a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff920009980 tx=0x7ff92000bcd0).stop 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 shutdown_connections 2021-02-10T11:25:50.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff93809c0c0 0x7ff93809c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9380a3db0 0x7ff9380a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff91c0186c0 0x7ff91c01abc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 --2- 172.21.2.117:0/28159077 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9380a39b0 0x7ff9380a4720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 >> 172.21.2.117:0/28159077 conn(0x7ff9380bb240 msgr2=0x7ff9380a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 shutdown_connections 2021-02-10T11:25:50.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.026 7ff93cabe700 1 -- 172.21.2.117:0/28159077 wait complete. 2021-02-10T11:25:50.028 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802444 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.446 7fefd34c8700 1 Processor -- start 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd34c8700 1 -- start start 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd34c8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd34c8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc0d1d20 con 0x7fefcc0e4da0 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd34c8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefcc077a70 con 0x7fefcc0e51a0 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd34c8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefcc0777d0 con 0x7fefcc07da20 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd12e4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd0ae3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fefcc0e4da0 0x7fefcc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43630/0 (socket says 172.21.2.117:43630) 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.447 7fefd0ae3700 1 -- 172.21.2.117:0/232179749 learned_addr learned my addr 172.21.2.117:0/232179749 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefc3fff700 1 -- 172.21.2.117:0/232179749 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30851657 0 0) 0x7fefbc0009b0 con 0x7fefcc0e4da0 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefc3fff700 1 -- 172.21.2.117:0/232179749 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb8002fe0 con 0x7fefcc0e4da0 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd12e4700 1 -- 172.21.2.117:0/232179749 >> v1:172.21.2.125:6789/0 conn(0x7fefcc0e51a0 legacy=0x7fefcc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd12e4700 1 -- 172.21.2.117:0/232179749 >> v1:172.21.2.117:6789/0 conn(0x7fefcc0e4da0 legacy=0x7fefcc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd12e4700 1 -- 172.21.2.117:0/232179749 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc1b08e0 con 0x7fefcc07da20 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd12e4700 1 --2- 172.21.2.117:0/232179749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc07da20 0x7fefcc0771b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fefc8006650 tx=0x7fefc8009a80).ready entity=mon.0 client_cookie=671b09c3a3d18f04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefc3fff700 1 -- 172.21.2.117:0/232179749 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc800c0c0 con 0x7fefcc07da20 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefc3fff700 1 -- 172.21.2.117:0/232179749 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc800c340 con 0x7fefcc07da20 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefc3fff700 1 -- 172.21.2.117:0/232179749 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc800c860 con 0x7fefcc07da20 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd34c8700 1 -- 172.21.2.117:0/232179749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc07da20 msgr2=0x7fefcc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.448 7fefd34c8700 1 --2- 172.21.2.117:0/232179749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc07da20 0x7fefcc0771b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fefc8006650 tx=0x7fefc8009a80).stop 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.449 7fefd34c8700 1 -- 172.21.2.117:0/232179749 shutdown_connections 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.449 7fefd34c8700 1 --2- 172.21.2.117:0/232179749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc07da20 0x7fefcc0771b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.449 7fefd34c8700 1 -- 172.21.2.117:0/232179749 >> 172.21.2.117:0/232179749 conn(0x7fefcc0ce450 msgr2=0x7fefcc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.449 7fefd34c8700 1 -- 172.21.2.117:0/232179749 shutdown_connections 2021-02-10T11:25:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.449 7fefd34c8700 1 -- 172.21.2.117:0/232179749 wait complete. 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 Processor -- start 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 -- start start 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc08a180 0x7fefcc08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc1a7240 0x7fefcc1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc108020 con 0x7fefcc107c20 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc1a7c60 con 0x7fefcc1a7240 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd34c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc1a7e80 con 0x7fefcc08a180 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd12e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd12e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43384/0 (socket says 172.21.2.117:43384) 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.464 7fefd12e4700 1 -- 172.21.2.117:0/1013252447 learned_addr learned my addr 172.21.2.117:0/1013252447 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 -- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc08a180 msgr2=0x7fefcc08a580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc08a180 0x7fefcc08a580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 -- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc1a7240 msgr2=0x7fefcc1a7640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc1a7240 0x7fefcc1a7640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 -- 172.21.2.117:0/1013252447 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc08b250 con 0x7fefcc107c20 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd12e4700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fefc8009630 tx=0x7fefc800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc80093a0 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc800bd70 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fefc8005560 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc08b250 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.465 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc08b7e0 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.466 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fefc8007020 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.466 7fefc1ffb700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 0x7fefac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.466 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fefc80326d0 con 0x7fefcc107c20 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.466 7fefd0ae3700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 0x7fefac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:50.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.467 7fefd0ae3700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 0x7fefac01abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:50.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.497 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc077a70 con 0x7fefcc107c20 2021-02-10T11:25:50.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.500 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fefc802fbf0 con 0x7fefcc107c20 2021-02-10T11:25:50.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.629 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802444", "target": ["mgr", ""]}) v1 -- 0x7fefcc08b7e0 con 0x7fefac0186c0 2021-02-10T11:25:50.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.632 7fefc1ffb700 1 -- 172.21.2.117:0/1013252447 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fefbc000940 con 0x7fefac0186c0 2021-02-10T11:25:50.626 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9 2021-02-10T11:25:50.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 msgr2=0x7fefac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 0x7fefac01abc0 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 msgr2=0x7fefcc096fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fefc8009630 tx=0x7fefc800b980).stop 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 shutdown_connections 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fefcc1a7240 0x7fefcc1a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fefcc08a180 0x7fefcc08a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fefac0186c0 0x7fefac01abc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 --2- 172.21.2.117:0/1013252447 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fefcc107c20 0x7fefcc096fc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:50.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 >> 172.21.2.117:0/1013252447 conn(0x7fefcc0a1880 msgr2=0x7fefcc0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:50.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.633 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 shutdown_connections 2021-02-10T11:25:50.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:50.634 7fefd34c8700 1 -- 172.21.2.117:0/1013252447 wait complete. 2021-02-10T11:25:50.636 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802444 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9 2021-02-10T11:25:50.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:50.636 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:50.636 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:50.970 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:50.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:50.970 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248891 s, 42.1 MB/s 2021-02-10T11:25:51.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:51.353 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:51.353 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:51.688 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:51.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:51.689 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:52.060 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:52.060 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:52.060 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022514 s, 46.6 MB/s 2021-02-10T11:25:52.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:52.061 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:52.061 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:52.395 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:52.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:52.395 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:52.767 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:52.767 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:52.767 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224579 s, 46.7 MB/s 2021-02-10T11:25:52.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:52.768 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:52.768 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:53.102 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:53.102 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:53.103 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248925 s, 42.1 MB/s 2021-02-10T11:25:53.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:53.479 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:53.479 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:53.831 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:53.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:53.831 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:54.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:54.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:54.208 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233025 s, 45.0 MB/s 2021-02-10T11:25:54.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:54.209 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:54.209 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:54.543 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:54.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:54.544 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:54.713 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:54.716 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:25:54.917 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:54.917 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:54.917 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242317 s, 43.3 MB/s 2021-02-10T11:25:54.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:54.918 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:54.918 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:55.253 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:55.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:55.253 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026143 s, 40.1 MB/s 2021-02-10T11:25:55.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:55.631 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:55.631 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:55.966 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:55.966 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:55.967 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:56.341 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:56.341 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:56.341 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240408 s, 43.6 MB/s 2021-02-10T11:25:56.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:56.342 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:56.342 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:56.676 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:56.676 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:56.676 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238812 s, 43.9 MB/s 2021-02-10T11:25:57.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:57.052 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:57.052 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:57.387 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:57.387 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:57.387 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802444/a32842da-49c5-4e46-a598-1a50102f84e9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:57.760 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:57.760 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:57.760 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237305 s, 44.2 MB/s 2021-02-10T11:25:57.761 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802445 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.177 7f99aa97a700 1 Processor -- start 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.177 7f99aa97a700 1 -- start start 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99aa97a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a407da20 0x7f99a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99aa97a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99a40e51a0 0x7f99a40d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99aa97a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a4077a70 con 0x7f99a40e4da0 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99aa97a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40e5c20 con 0x7f99a40e51a0 2021-02-10T11:25:58.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99aa97a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40777d0 con 0x7f99a407da20 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a8f97700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99a40e51a0 0x7f99a40d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f99a40e4da0 0x7f99a41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43642/0 (socket says 172.21.2.117:43642) 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a37fe700 1 -- 172.21.2.117:0/163031186 learned_addr learned my addr 172.21.2.117:0/163031186 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a3fff700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a407da20 0x7f99a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a8f97700 1 -- 172.21.2.117:0/163031186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a407da20 msgr2=0x7f99a40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a8f97700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a407da20 0x7f99a40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a8f97700 1 -- 172.21.2.117:0/163031186 >> v1:172.21.2.117:6789/0 conn(0x7f99a40e4da0 legacy=0x7f99a41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.178 7f99a8f97700 1 -- 172.21.2.117:0/163031186 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a41b08e0 con 0x7f99a40e51a0 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99a8f97700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a40e51a0 0x7f99a40d0520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f999c006e30 tx=0x7f999c009e60).ready entity=mon.0 client_cookie=aaafe852dabe5d44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99a2ffd700 1 -- 172.21.2.117:0/163031186 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c006a50 con 0x7f99a40e51a0 2021-02-10T11:25:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99a2ffd700 1 -- 172.21.2.117:0/163031186 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00b300 con 0x7f99a40e51a0 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99a2ffd700 1 -- 172.21.2.117:0/163031186 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c00b7e0 con 0x7f99a40e51a0 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 -- 172.21.2.117:0/163031186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a40e51a0 msgr2=0x7f99a40d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a40e51a0 0x7f99a40d0520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f999c006e30 tx=0x7f999c009e60).stop 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 -- 172.21.2.117:0/163031186 shutdown_connections 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a40e51a0 0x7f99a40d0520 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 --2- 172.21.2.117:0/163031186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a407da20 0x7f99a40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.179 7f99aa97a700 1 -- 172.21.2.117:0/163031186 >> 172.21.2.117:0/163031186 conn(0x7f99a40ce450 msgr2=0x7f99a40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.180 7f99aa97a700 1 -- 172.21.2.117:0/163031186 shutdown_connections 2021-02-10T11:25:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.180 7f99aa97a700 1 -- 172.21.2.117:0/163031186 wait complete. 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.194 7f99aa97a700 1 Processor -- start 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.194 7f99aa97a700 1 -- start start 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a408a180 0x7f99a408a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99a408aba0 0x7f99a41b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41b9cb0 con 0x7f99a408afa0 2021-02-10T11:25:58.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40c4050 con 0x7f99a408aba0 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41b9730 con 0x7f99a408a180 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a37fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99a408aba0 0x7f99a41b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43400/0 (socket says 172.21.2.117:43400) 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 -- 172.21.2.117:0/628349807 learned_addr learned my addr 172.21.2.117:0/628349807 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a3fff700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a408a180 0x7f99a408a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 -- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a408a180 msgr2=0x7f99a408a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a408a180 0x7f99a408a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 -- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99a408aba0 msgr2=0x7f99a41b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99a408aba0 0x7f99a41b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 -- 172.21.2.117:0/628349807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4106e90 con 0x7f99a408afa0 2021-02-10T11:25:58.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a8f97700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f999c001070 tx=0x7f999c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c001270 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c0014f0 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f999c0019d0 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.195 7f99aa97a700 1 -- 172.21.2.117:0/628349807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a4106e90 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.196 7f99aa97a700 1 -- 172.21.2.117:0/628349807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a4106e90 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.196 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f999c0093d0 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.196 7f99a0ff9700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.196 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f999c032cc0 con 0x7f99a408afa0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.196 7f99a3fff700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.197 7f99a3fff700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 0x7f998401ac10 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.228 7f99aa97a700 1 -- 172.21.2.117:0/628349807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a4077a70 con 0x7f99a408afa0 2021-02-10T11:25:58.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.230 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f999c003020 con 0x7f99a408afa0 2021-02-10T11:25:58.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.354 7f99aa97a700 1 -- 172.21.2.117:0/628349807 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802445", "target": ["mgr", ""]}) v1 -- 0x7f99a40c4050 con 0x7f9984018730 2021-02-10T11:25:58.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.381 7f99a0ff9700 1 -- 172.21.2.117:0/628349807 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9994006a70 con 0x7f9984018730 2021-02-10T11:25:58.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 -- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 msgr2=0x7f998401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 0x7f998401ac10 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 -- 172.21.2.117:0/628349807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 msgr2=0x7f99a41b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f999c001070 tx=0x7f999c00bd40).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 -- 172.21.2.117:0/628349807 shutdown_connections 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9984018730 0x7f998401ac10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99a408afa0 0x7f99a41b8a40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99a408aba0 0x7f99a41b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 --2- 172.21.2.117:0/628349807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99a408a180 0x7f99a408a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.387 7f99aa97a700 1 -- 172.21.2.117:0/628349807 >> 172.21.2.117:0/628349807 conn(0x7f99a40a4180 msgr2=0x7f99a409be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.388 7f99aa97a700 1 -- 172.21.2.117:0/628349807 shutdown_connections 2021-02-10T11:25:58.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.388 7f99aa97a700 1 -- 172.21.2.117:0/628349807 wait complete. 2021-02-10T11:25:58.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802445 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.809 7f5af0e5b700 1 Processor -- start 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.809 7f5af0e5b700 1 -- start start 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5aec07da20 0x7f5aec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5aec0e4da0 0x7f5aec1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0e51a0 0x7f5aec077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec077a70 con 0x7f5aec0e4da0 2021-02-10T11:25:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec0777d0 con 0x7f5aec07da20 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5af0e5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec07de20 con 0x7f5aec0e51a0 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5aec0e4da0 0x7f5aec1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5aec0e4da0 0x7f5aec1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43406/0 (socket says 172.21.2.117:43406) 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 -- 172.21.2.117:0/4068776774 learned_addr learned my addr 172.21.2.117:0/4068776774 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aeb314700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0e51a0 0x7f5aec077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aeab13700 1 --2- 172.21.2.117:0/4068776774 >> v2:172.21.2.125:3300/0 conn(0x7f5aec07da20 0x7f5aec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 -- 172.21.2.117:0/4068776774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0e51a0 msgr2=0x7f5aec077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0e51a0 0x7f5aec077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 -- 172.21.2.117:0/4068776774 >> v2:172.21.2.125:3300/0 conn(0x7f5aec07da20 msgr2=0x7f5aec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 --2- 172.21.2.117:0/4068776774 >> v2:172.21.2.125:3300/0 conn(0x7f5aec07da20 0x7f5aec0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 -- 172.21.2.117:0/4068776774 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec1b0710 con 0x7f5aec0e4da0 2021-02-10T11:25:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.810 7f5aea312700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec0e4da0 0x7f5aec1b00f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f5adc006700 tx=0x7f5adc009b80).ready entity=mon.0 client_cookie=b6fb7b2f14bd7178 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5ae9b11700 1 -- 172.21.2.117:0/4068776774 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5adc00c0c0 con 0x7f5aec0e4da0 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5ae9b11700 1 -- 172.21.2.117:0/4068776774 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc00c380 con 0x7f5aec0e4da0 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5ae9b11700 1 -- 172.21.2.117:0/4068776774 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5adc00c880 con 0x7f5aec0e4da0 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 -- 172.21.2.117:0/4068776774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec0e4da0 msgr2=0x7f5aec1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec0e4da0 0x7f5aec1b00f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f5adc006700 tx=0x7f5adc009b80).stop 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 -- 172.21.2.117:0/4068776774 shutdown_connections 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0e51a0 0x7f5aec077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 --2- 172.21.2.117:0/4068776774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec0e4da0 0x7f5aec1b00f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 --2- 172.21.2.117:0/4068776774 >> v2:172.21.2.125:3300/0 conn(0x7f5aec07da20 0x7f5aec0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 -- 172.21.2.117:0/4068776774 >> 172.21.2.117:0/4068776774 conn(0x7f5aec0ce450 msgr2=0x7f5aec0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.811 7f5af0e5b700 1 -- 172.21.2.117:0/4068776774 shutdown_connections 2021-02-10T11:25:58.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.812 7f5af0e5b700 1 -- 172.21.2.117:0/4068776774 wait complete. 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 Processor -- start 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 -- start start 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0b7fc0 0x7f5aec1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5aec1048f0 0x7f5aec104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1054f0 con 0x7f5aec090710 2021-02-10T11:25:58.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.826 7f5af0e5b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec195a30 con 0x7f5aec1048f0 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5af0e5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec195fb0 con 0x7f5aec0b7fc0 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43414/0 (socket says 172.21.2.117:43414) 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 -- 172.21.2.117:0/2080268788 learned_addr learned my addr 172.21.2.117:0/2080268788 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 -- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0b7fc0 msgr2=0x7f5aec1042d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0b7fc0 0x7f5aec1042d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 -- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5aec1048f0 msgr2=0x7f5aec104cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5aec1048f0 0x7f5aec104cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 -- 172.21.2.117:0/2080268788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0b7b60 con 0x7f5aec090710 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5aeb314700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f5ae40099a0 tx=0x7f5ae400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae40011f0 con 0x7f5aec090710 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.827 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4000fc0 con 0x7f5aec090710 2021-02-10T11:25:58.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0b7b60 con 0x7f5aec090710 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae4001660 con 0x7f5aec090710 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec091420 con 0x7f5aec090710 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ae40011f0 con 0x7f5aec090710 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5ae37fe700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 0x7f5ad001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.828 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5ae40328c0 con 0x7f5aec090710 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.829 7f5aeab13700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 0x7f5ad001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:25:58.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.829 7f5aeab13700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 0x7f5ad001abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:25:58.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.860 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec103280 con 0x7f5aec090710 2021-02-10T11:25:58.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.862 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ae402ef50 con 0x7f5aec090710 2021-02-10T11:25:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.990 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802445", "target": ["mgr", ""]}) v1 -- 0x7f5aec091420 con 0x7f5ad00186c0 2021-02-10T11:25:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:58.994 7f5ae37fe700 1 -- 172.21.2.117:0/2080268788 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5ad4006450 con 0x7f5ad00186c0 2021-02-10T11:25:58.993 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 msgr2=0x7f5ad001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 0x7f5ad001abc0 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 msgr2=0x7f5aec090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f5ae40099a0 tx=0x7f5ae400bcd0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 shutdown_connections 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ad00186c0 0x7f5ad001abc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5aec090710 0x7f5aec090b10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5aec1048f0 0x7f5aec104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 --2- 172.21.2.117:0/2080268788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5aec0b7fc0 0x7f5aec1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:25:58.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 >> 172.21.2.117:0/2080268788 conn(0x7f5aec0a1850 msgr2=0x7f5aec0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 shutdown_connections 2021-02-10T11:25:58.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:25:59.000 7f5af0e5b700 1 -- 172.21.2.117:0/2080268788 wait complete. 2021-02-10T11:25:59.003 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802445 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609 2021-02-10T11:25:59.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:59.003 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:59.003 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:25:59.337 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:25:59.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:25:59.338 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:25:59.718 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:25:59.718 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:25:59.719 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236761 s, 44.3 MB/s 2021-02-10T11:25:59.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:25:59.720 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:25:59.720 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:00.056 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:00.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:00.056 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237091 s, 44.2 MB/s 2021-02-10T11:26:00.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:00.431 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:00.431 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:00.765 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:00.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:00.766 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:01.142 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:01.142 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:01.142 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259444 s, 40.4 MB/s 2021-02-10T11:26:01.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:01.143 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:01.143 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:01.477 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:01.477 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:01.477 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:01.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:01.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:01.853 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256582 s, 40.9 MB/s 2021-02-10T11:26:01.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:01.854 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:01.854 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:02.189 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:02.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:02.189 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:02.562 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:02.562 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:02.562 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262291 s, 40.0 MB/s 2021-02-10T11:26:02.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:02.564 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:02.564 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:02.899 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:02.899 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:02.899 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:03.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:03.276 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:03.276 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258938 s, 40.5 MB/s 2021-02-10T11:26:03.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:03.277 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:03.277 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:03.611 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:03.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:03.611 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:03.987 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:03.987 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:03.987 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257483 s, 40.7 MB/s 2021-02-10T11:26:03.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:03.988 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:03.989 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:04.323 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:04.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:04.323 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:04.697 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:04.697 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:04.697 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236484 s, 44.3 MB/s 2021-02-10T11:26:04.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:04.698 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:04.699 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:05.035 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:05.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:05.035 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:05.410 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:05.410 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:05.410 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259417 s, 40.4 MB/s 2021-02-10T11:26:05.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:05.411 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:05.412 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:05.746 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:05.746 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:05.746 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802445/9a60e8fd-4a4f-402d-ab86-4a0ca38b6609/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246663 s, 42.5 MB/s 2021-02-10T11:26:06.123 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802446 2021-02-10T11:26:06.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.542 7f49b10ab700 1 Processor -- start 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.542 7f49b10ab700 1 -- start start 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49b10ab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49ac0d1530 0x7f49ac0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49b10ab700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac077a70 con 0x7f49ac079000 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49b10ab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac079400 con 0x7f49ac07da20 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49b10ab700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac07de20 con 0x7f49ac0d1530 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49ac0d1530 0x7f49ac0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49ac0d1530 0x7f49ac0d1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43422/0 (socket says 172.21.2.117:43422) 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 -- 172.21.2.117:0/2915200136 learned_addr learned my addr 172.21.2.117:0/2915200136 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49aa59c700 1 --1- 172.21.2.117:0/2915200136 >> v1:172.21.2.117:6789/0 conn(0x7f49ac079000 0x7f49ac0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43670/0 (socket says 172.21.2.117:43670) 2021-02-10T11:26:06.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 -- 172.21.2.117:0/2915200136 >> v1:172.21.2.125:6789/0 conn(0x7f49ac07da20 legacy=0x7f49ac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 -- 172.21.2.117:0/2915200136 >> v1:172.21.2.117:6789/0 conn(0x7f49ac079000 legacy=0x7f49ac0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.543 7f49ab59e700 1 -- 172.21.2.117:0/2915200136 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac1b00f0 con 0x7f49ac0d1530 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49ab59e700 1 --2- 172.21.2.117:0/2915200136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0d1530 0x7f49ac0d1930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f49a4006f00 tx=0x7f49a4009f10).ready entity=mon.0 client_cookie=4816ca9100401175 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49a9d9b700 1 -- 172.21.2.117:0/2915200136 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49a400b3b0 con 0x7f49ac0d1530 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49a9d9b700 1 -- 172.21.2.117:0/2915200136 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a400b730 con 0x7f49ac0d1530 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49a9d9b700 1 -- 172.21.2.117:0/2915200136 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49a400bc50 con 0x7f49ac0d1530 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49b10ab700 1 -- 172.21.2.117:0/2915200136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0d1530 msgr2=0x7f49ac0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49b10ab700 1 --2- 172.21.2.117:0/2915200136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0d1530 0x7f49ac0d1930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f49a4006f00 tx=0x7f49a4009f10).stop 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49b10ab700 1 -- 172.21.2.117:0/2915200136 shutdown_connections 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49b10ab700 1 --2- 172.21.2.117:0/2915200136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0d1530 0x7f49ac0d1930 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.544 7f49b10ab700 1 -- 172.21.2.117:0/2915200136 >> 172.21.2.117:0/2915200136 conn(0x7f49ac0ce450 msgr2=0x7f49ac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.545 7f49b10ab700 1 -- 172.21.2.117:0/2915200136 shutdown_connections 2021-02-10T11:26:06.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.545 7f49b10ab700 1 -- 172.21.2.117:0/2915200136 wait complete. 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 Processor -- start 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 -- start start 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 0x7f49ac0bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49ac0bfdd0 0x7f49ac0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0b6200 con 0x7f49ac0c2e80 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c38a0 con 0x7f49ac0b4720 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.559 7f49b10ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c3b20 con 0x7f49ac0bfdd0 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43428/0 (socket says 172.21.2.117:43428) 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 -- 172.21.2.117:0/2057334051 learned_addr learned my addr 172.21.2.117:0/2057334051 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49aa59c700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49ac0bfdd0 0x7f49ac0c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49aad9d700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 0x7f49ac0bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 -- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49ac0bfdd0 msgr2=0x7f49ac0c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49ac0bfdd0 0x7f49ac0c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 -- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 msgr2=0x7f49ac0bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 0x7f49ac0bf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 -- 172.21.2.117:0/2057334051 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0b4b20 con 0x7f49ac0c2e80 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49aad9d700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 0x7f49ac0bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:06.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.560 7f49ab59e700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f49a4009970 tx=0x7f49a400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49a4009450 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4000cc0 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac0b4b20 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49a400c4e0 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac106d20 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.561 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49a4001270 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.562 7f499b7fe700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 0x7f499001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.562 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f49a4032a30 con 0x7f49ac0c2e80 2021-02-10T11:26:06.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.562 7f49aad9d700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 0x7f499001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:06.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.562 7f49aad9d700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 0x7f499001abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:06.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.593 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac0ce450 con 0x7f49ac0c2e80 2021-02-10T11:26:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.595 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49a4003080 con 0x7f49ac0c2e80 2021-02-10T11:26:06.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.721 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802446", "target": ["mgr", ""]}) v1 -- 0x7f49ac106d20 con 0x7f49900186c0 2021-02-10T11:26:06.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.746 7f499b7fe700 1 -- 172.21.2.117:0/2057334051 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f499c000960 con 0x7f49900186c0 2021-02-10T11:26:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 msgr2=0x7f499001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 0x7f499001abc0 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 msgr2=0x7f49ac0c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f49a4009970 tx=0x7f49a400bc40).stop 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 shutdown_connections 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49900186c0 0x7f499001abc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49ac0c2e80 0x7f49ac0c3280 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49ac0bfdd0 0x7f49ac0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 --2- 172.21.2.117:0/2057334051 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49ac0b4720 0x7f49ac0bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 >> 172.21.2.117:0/2057334051 conn(0x7f49ac1116a0 msgr2=0x7f49ac0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.754 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 shutdown_connections 2021-02-10T11:26:06.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:06.755 7f49b10ab700 1 -- 172.21.2.117:0/2057334051 wait complete. 2021-02-10T11:26:06.758 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802446 2021-02-10T11:26:07.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.174 7f86a8747700 1 Processor -- start 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.174 7f86a8747700 1 -- start start 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a8747700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f86a007da20 0x7f86a00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a8747700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f86a00e4da0 0x7f86a01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a8747700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a0077a70 con 0x7f86a00e51a0 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a8747700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a00777d0 con 0x7f86a00e4da0 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a8747700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a007de20 con 0x7f86a007da20 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f86a00e4da0 0x7f86a01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a6d64700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f86a00e51a0 0x7f86a0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43682/0 (socket says 172.21.2.117:43682) 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f86a00e4da0 0x7f86a01b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43432/0 (socket says 172.21.2.117:43432) 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a6d64700 1 -- 172.21.2.117:0/1556808379 learned_addr learned my addr 172.21.2.117:0/1556808379 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:07.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 -- 172.21.2.117:0/1556808379 >> v2:172.21.2.125:3300/0 conn(0x7f86a007da20 msgr2=0x7f86a00771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 --2- 172.21.2.117:0/1556808379 >> v2:172.21.2.125:3300/0 conn(0x7f86a007da20 0x7f86a00771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 -- 172.21.2.117:0/1556808379 >> v1:172.21.2.117:6789/0 conn(0x7f86a00e51a0 legacy=0x7f86a0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 -- 172.21.2.117:0/1556808379 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a01b0710 con 0x7f86a00e4da0 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.175 7f86a5d62700 1 --2- 172.21.2.117:0/1556808379 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00e4da0 0x7f86a01b00f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f869c006650 tx=0x7f869c009a00).ready entity=mon.0 client_cookie=f3639f7fd56e66cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a5561700 1 -- 172.21.2.117:0/1556808379 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f869c009df0 con 0x7f86a00e4da0 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a5561700 1 -- 172.21.2.117:0/1556808379 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c0009b0 con 0x7f86a00e4da0 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a5561700 1 -- 172.21.2.117:0/1556808379 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f869c00c2b0 con 0x7f86a00e4da0 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 -- 172.21.2.117:0/1556808379 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00e4da0 msgr2=0x7f86a01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 --2- 172.21.2.117:0/1556808379 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00e4da0 0x7f86a01b00f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f869c006650 tx=0x7f869c009a00).stop 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 -- 172.21.2.117:0/1556808379 shutdown_connections 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 --2- 172.21.2.117:0/1556808379 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00e4da0 0x7f86a01b00f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 --2- 172.21.2.117:0/1556808379 >> v2:172.21.2.125:3300/0 conn(0x7f86a007da20 0x7f86a00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 -- 172.21.2.117:0/1556808379 >> 172.21.2.117:0/1556808379 conn(0x7f86a00ce450 msgr2=0x7f86a00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:07.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 -- 172.21.2.117:0/1556808379 shutdown_connections 2021-02-10T11:26:07.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.176 7f86a8747700 1 -- 172.21.2.117:0/1556808379 wait complete. 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 Processor -- start 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 -- start start 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86a00bde30 0x7f86a00a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86a018e500 0x7f86a018e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0199d40 con 0x7f86a00bda30 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0107cb0 con 0x7f86a018e500 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a8747700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0107ed0 con 0x7f86a00bde30 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a6563700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a6563700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43436/0 (socket says 172.21.2.117:43436) 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.191 7f86a6563700 1 -- 172.21.2.117:0/3314440088 learned_addr learned my addr 172.21.2.117:0/3314440088 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 -- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86a00bde30 msgr2=0x7f86a00a1770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86a00bde30 0x7f86a00a1770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 -- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86a018e500 msgr2=0x7f86a018e900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86a018e500 0x7f86a018e900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 -- 172.21.2.117:0/3314440088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a018ef20 con 0x7f86a00bda30 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a6563700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f869400b5c0 tx=0x7f869400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8694006b80 con 0x7f86a00bda30 2021-02-10T11:26:07.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869400bd50 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86940054d0 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a8747700 1 -- 172.21.2.117:0/3314440088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a018ef20 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.192 7f86a8747700 1 -- 172.21.2.117:0/3314440088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a018f480 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.193 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8694006030 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.193 7f8692ffd700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 0x7f868801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.193 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8694032590 con 0x7f86a00bda30 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.193 7f86a5d62700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 0x7f868801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:07.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.194 7f86a5d62700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 0x7f868801abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:07.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.225 7f86a8747700 1 -- 172.21.2.117:0/3314440088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a0199d40 con 0x7f86a00bda30 2021-02-10T11:26:07.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.227 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f86940009b0 con 0x7f86a00bda30 2021-02-10T11:26:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.354 7f86a8747700 1 -- 172.21.2.117:0/3314440088 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802446", "target": ["mgr", ""]}) v1 -- 0x7f86a018f480 con 0x7f86880186c0 2021-02-10T11:26:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.357 7f8692ffd700 1 -- 172.21.2.117:0/3314440088 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f869c000a00 con 0x7f86880186c0 2021-02-10T11:26:07.352 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 msgr2=0x7f868801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 0x7f868801abc0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 msgr2=0x7f86a00a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f869400b5c0 tx=0x7f869400b960).stop 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 shutdown_connections 2021-02-10T11:26:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86a018e500 0x7f86a018e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86a00bde30 0x7f86a00a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86880186c0 0x7f868801abc0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 --2- 172.21.2.117:0/3314440088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86a00bda30 0x7f86a00a1150 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 >> 172.21.2.117:0/3314440088 conn(0x7f86a0095310 msgr2=0x7f86a0095830 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 shutdown_connections 2021-02-10T11:26:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:07.359 7f86a8747700 1 -- 172.21.2.117:0/3314440088 wait complete. 2021-02-10T11:26:07.362 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802446 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d 2021-02-10T11:26:07.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:07.363 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:07.363 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:07.698 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:07.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:07.698 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:08.076 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:08.076 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:08.077 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245298 s, 42.7 MB/s 2021-02-10T11:26:08.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:08.078 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:08.078 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:08.412 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:08.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:08.412 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251368 s, 41.7 MB/s 2021-02-10T11:26:08.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:08.789 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:08.789 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:09.123 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:09.123 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:09.123 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:09.498 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:09.498 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:09.498 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246127 s, 42.6 MB/s 2021-02-10T11:26:09.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:09.499 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:09.499 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:09.833 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:09.833 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:09.833 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:10.206 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:10.206 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:10.206 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229809 s, 45.6 MB/s 2021-02-10T11:26:10.207 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.207 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.207 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:10.543 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:10.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:10.543 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:10.917 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:10.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:10.918 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240545 s, 43.6 MB/s 2021-02-10T11:26:10.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:10.919 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:10.919 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:11.253 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:11.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:11.254 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:11.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:11.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:11.628 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248608 s, 42.2 MB/s 2021-02-10T11:26:11.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:11.629 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:11.629 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:11.964 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:11.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:11.964 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:12.337 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:12.338 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:12.338 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240959 s, 43.5 MB/s 2021-02-10T11:26:12.339 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:12.339 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:12.339 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:12.674 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:12.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:12.674 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:13.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:13.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:13.043 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229347 s, 45.7 MB/s 2021-02-10T11:26:13.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:13.045 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:13.045 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:13.379 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:13.380 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:13.380 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:13.755 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:13.755 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:13.755 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243821 s, 43.0 MB/s 2021-02-10T11:26:13.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:13.756 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:13.757 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:14.090 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:14.091 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:14.091 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802446/67181006-c0a4-446f-a89e-1d166caf0b3d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:14.464 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:14.464 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:14.464 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233227 s, 45.0 MB/s 2021-02-10T11:26:14.465 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802447 2021-02-10T11:26:14.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.882 7f19b7895700 1 Processor -- start 2021-02-10T11:26:14.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.882 7f19b7895700 1 -- start start 2021-02-10T11:26:14.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b7895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 0x7f19b00d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b7895700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b0077a70 con 0x7f19b007da20 2021-02-10T11:26:14.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b7895700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b0079400 con 0x7f19b0079000 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b7895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b007de20 con 0x7f19b00d1530 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b56b1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f19b007da20 0x7f19b00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43692/0 (socket says 172.21.2.117:43692) 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b56b1700 1 -- 172.21.2.117:0/2595921179 learned_addr learned my addr 172.21.2.117:0/2595921179 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.883 7f19b5eb2700 1 --2- 172.21.2.117:0/2595921179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 0x7f19b00d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601695649 0 0) 0x7f19a8001500 con 0x7f19b007da20 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c002fe0 con 0x7f19b007da20 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888724833 0 0) 0x7f19a0000ff0 con 0x7f19b0079000 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c003210 con 0x7f19b0079000 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b5eb2700 1 -- 172.21.2.117:0/2595921179 >> v1:172.21.2.125:6789/0 conn(0x7f19b0079000 legacy=0x7f19b00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b5eb2700 1 -- 172.21.2.117:0/2595921179 >> v1:172.21.2.117:6789/0 conn(0x7f19b007da20 legacy=0x7f19b00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b5eb2700 1 -- 172.21.2.117:0/2595921179 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b01b00f0 con 0x7f19b00d1530 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b5eb2700 1 --2- 172.21.2.117:0/2595921179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 0x7f19b00d1930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f19ac006ad0 tx=0x7f19ac009ad0).ready entity=mon.2 client_cookie=4de1cf0610374038 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ac0022e0 con 0x7f19b00d1530 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ac006410 con 0x7f19b00d1530 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19a7fff700 1 -- 172.21.2.117:0/2595921179 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ac0025e0 con 0x7f19b00d1530 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b7895700 1 -- 172.21.2.117:0/2595921179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 msgr2=0x7f19b00d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.884 7f19b7895700 1 --2- 172.21.2.117:0/2595921179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 0x7f19b00d1930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f19ac006ad0 tx=0x7f19ac009ad0).stop 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.885 7f19b7895700 1 -- 172.21.2.117:0/2595921179 shutdown_connections 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.885 7f19b7895700 1 --2- 172.21.2.117:0/2595921179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00d1530 0x7f19b00d1930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.885 7f19b7895700 1 -- 172.21.2.117:0/2595921179 >> 172.21.2.117:0/2595921179 conn(0x7f19b00ce450 msgr2=0x7f19b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.885 7f19b7895700 1 -- 172.21.2.117:0/2595921179 shutdown_connections 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.885 7f19b7895700 1 -- 172.21.2.117:0/2595921179 wait complete. 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.899 7f19b7895700 1 Processor -- start 2021-02-10T11:26:14.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.899 7f19b7895700 1 -- start start 2021-02-10T11:26:14.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b00a1090 0x7f19b019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00a20e0 0x7f19b00a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00c1360 con 0x7f19b019db70 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00a2b00 con 0x7f19b00a1090 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b7895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b00a2d20 con 0x7f19b00a20e0 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43452/0 (socket says 172.21.2.117:43452) 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 -- 172.21.2.117:0/1485954635 learned_addr learned my addr 172.21.2.117:0/1485954635 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 -- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00a20e0 msgr2=0x7f19b00a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b56b1700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b00a1090 0x7f19b019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00a20e0 0x7f19b00a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.900 7f19b4eb0700 1 -- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b00a1090 msgr2=0x7f19b019d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19b4eb0700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b00a1090 0x7f19b019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19b4eb0700 1 -- 172.21.2.117:0/1485954635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b0106040 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19b4eb0700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f19a0006ed0 tx=0x7f19a000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a000bcd0 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a0006240 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19a0005730 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19b7895700 1 -- 172.21.2.117:0/1485954635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b0106040 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.901 7f19b7895700 1 -- 172.21.2.117:0/1485954635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b01065d0 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.902 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19a00054b0 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.902 7f19a5ffb700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 0x7f199001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.902 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f19a00326d0 con 0x7f19b019db70 2021-02-10T11:26:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.902 7f19b56b1700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 0x7f199001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:14.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.903 7f19b56b1700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 0x7f199001abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.933 7f19b7895700 1 -- 172.21.2.117:0/1485954635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b01b4f00 con 0x7f19b019db70 2021-02-10T11:26:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:14.935 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19a002ef00 con 0x7f19b019db70 2021-02-10T11:26:15.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.060 7f19b7895700 1 -- 172.21.2.117:0/1485954635 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802447", "target": ["mgr", ""]}) v1 -- 0x7f19b0077a70 con 0x7f19900186c0 2021-02-10T11:26:15.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.086 7f19a5ffb700 1 -- 172.21.2.117:0/1485954635 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f19a8007af0 con 0x7f19900186c0 2021-02-10T11:26:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.093 7f19b7895700 1 -- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 msgr2=0x7f199001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.093 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 0x7f199001abc0 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.093 7f19b7895700 1 -- 172.21.2.117:0/1485954635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 msgr2=0x7f19b00a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.093 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f19a0006ed0 tx=0x7f19a000b8e0).stop 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 -- 172.21.2.117:0/1485954635 shutdown_connections 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19900186c0 0x7f199001abc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19b019db70 0x7f19b00a1ac0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19b00a20e0 0x7f19b00a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 --2- 172.21.2.117:0/1485954635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19b00a1090 0x7f19b019d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 -- 172.21.2.117:0/1485954635 >> 172.21.2.117:0/1485954635 conn(0x7f19b00ab050 msgr2=0x7f19b0107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:15.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 -- 172.21.2.117:0/1485954635 shutdown_connections 2021-02-10T11:26:15.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.094 7f19b7895700 1 -- 172.21.2.117:0/1485954635 wait complete. 2021-02-10T11:26:15.096 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802447 2021-02-10T11:26:15.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.513 7fa718610700 1 Processor -- start 2021-02-10T11:26:15.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.513 7fa718610700 1 -- start start 2021-02-10T11:26:15.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa718610700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100d1530 0x7fa7100d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa718610700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa710077a70 con 0x7fa71007da20 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa718610700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa710079400 con 0x7fa710079000 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa718610700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71007de20 con 0x7fa7100d1530 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa71642c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa71007da20 0x7fa7100771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43706/0 (socket says 172.21.2.117:43706) 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa71642c700 1 -- 172.21.2.117:0/2091497804 learned_addr learned my addr 172.21.2.117:0/2091497804 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963276185 0 0) 0x7fa704000ce0 con 0x7fa71007da20 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa71542a700 1 -- 172.21.2.117:0/2091497804 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6fc002fe0 con 0x7fa71007da20 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.514 7fa716c2d700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100d1530 0x7fa7100d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3964969458 0 0) 0x7fa7040010e0 con 0x7fa71007da20 2021-02-10T11:26:15.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100d1530 msgr2=0x7fa7100d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100d1530 0x7fa7100d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 >> v1:172.21.2.125:6789/0 conn(0x7fa710079000 legacy=0x7fa7100e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7101b00f0 con 0x7fa71007da20 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa704001660 con 0x7fa71007da20 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 >> v1:172.21.2.117:6789/0 conn(0x7fa71007da20 legacy=0x7fa7100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6fc003ac0 0x7fa6fc005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6fc006780 0x7fa6fc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6fc0095a0 0x7fa6fc00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc00c2c0 con 0x7fa6fc006780 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc00c4e0 con 0x7fa6fc003ac0 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71542a700 1 -- 172.21.2.117:0/2091497804 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc050710 con 0x7fa6fc0095a0 2021-02-10T11:26:15.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6fc006780 0x7fa6fc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa71642c700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6fc003ac0 0x7fa6fc005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa716c2d700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6fc0095a0 0x7fa6fc00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 -- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6fc0095a0 msgr2=0x7fa6fc00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6fc0095a0 0x7fa6fc00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 -- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6fc003ac0 msgr2=0x7fa6fc005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6fc003ac0 0x7fa6fc005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 -- 172.21.2.117:0/2091497804 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc050970 con 0x7fa6fc006780 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa716c2d700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6fc0095a0 0x7fa6fc00ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.515 7fa715c2b700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6fc006780 0x7fa6fc008ca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa70c006ab0 tx=0x7fa70c009b40).ready entity=mon.0 client_cookie=3789e8be9ff77ed5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70c0066d0 con 0x7fa6fc006780 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c00b2e0 con 0x7fa6fc006780 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa71542a700 1 -- 172.21.2.117:0/2091497804 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa70c00b7c0 con 0x7fa6fc006780 2021-02-10T11:26:15.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 -- 172.21.2.117:0/2091497804 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6fc006780 msgr2=0x7fa6fc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 --2- 172.21.2.117:0/2091497804 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6fc006780 0x7fa6fc008ca0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa70c006ab0 tx=0x7fa70c009b40).stop 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa716c2d700 1 -- 172.21.2.117:0/2091497804 reap_dead start 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 -- 172.21.2.117:0/2091497804 shutdown_connections 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 -- 172.21.2.117:0/2091497804 >> 172.21.2.117:0/2091497804 conn(0x7fa7100ce450 msgr2=0x7fa7100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 -- 172.21.2.117:0/2091497804 shutdown_connections 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.516 7fa718610700 1 -- 172.21.2.117:0/2091497804 wait complete. 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.531 7fa718610700 1 Processor -- start 2021-02-10T11:26:15.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.531 7fa718610700 1 -- start start 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.531 7fa718610700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7100a47d0 0x7fa7100a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.531 7fa718610700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100a51f0 0x7fa7100a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.531 7fa718610700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa718610700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710189970 con 0x7fa7100b77d0 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa718610700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa710190aa0 con 0x7fa7100a47d0 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa718610700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100b81f0 con 0x7fa7100a51f0 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43472/0 (socket says 172.21.2.117:43472) 2021-02-10T11:26:15.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 -- 172.21.2.117:0/1920054710 learned_addr learned my addr 172.21.2.117:0/1920054710 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 -- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100a51f0 msgr2=0x7fa7100a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100a51f0 0x7fa7100a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 -- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7100a47d0 msgr2=0x7fa7100a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7100a47d0 0x7fa7100a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 -- 172.21.2.117:0/1920054710 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100b8410 con 0x7fa7100b77d0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.532 7fa716c2d700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa708009a00 tx=0x7fa70800bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7080069e0 con 0x7fa7100b77d0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7080011f0 con 0x7fa7100b77d0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa718610700 1 -- 172.21.2.117:0/1920054710 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7100b8410 con 0x7fa7100b77d0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa718610700 1 -- 172.21.2.117:0/1920054710 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7100b89a0 con 0x7fa7100b77d0 2021-02-10T11:26:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa708001710 con 0x7fa7100b77d0 2021-02-10T11:26:15.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7080069e0 con 0x7fa7100b77d0 2021-02-10T11:26:15.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 0x7fa6f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:15.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.533 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa708032c70 con 0x7fa7100b77d0 2021-02-10T11:26:15.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.534 7fa71642c700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 0x7fa6f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:15.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.535 7fa71642c700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 0x7fa6f801ac10 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:15.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.564 7fa718610700 1 -- 172.21.2.117:0/1920054710 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa710077a70 con 0x7fa7100b77d0 2021-02-10T11:26:15.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.566 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa7080011f0 con 0x7fa7100b77d0 2021-02-10T11:26:15.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.694 7fa718610700 1 -- 172.21.2.117:0/1920054710 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802447", "target": ["mgr", ""]}) v1 -- 0x7fa710189970 con 0x7fa6f8018730 2021-02-10T11:26:15.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.698 7fa702ffd700 1 -- 172.21.2.117:0/1920054710 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa704000da0 con 0x7fa6f8018730 2021-02-10T11:26:15.697 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 msgr2=0x7fa6f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 0x7fa6f801ac10 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 msgr2=0x7fa7100b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa708009a00 tx=0x7fa70800bd00).stop 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 shutdown_connections 2021-02-10T11:26:15.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6f8018730 0x7fa6f801ac10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7100b77d0 0x7fa7100b7bd0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7100a51f0 0x7fa7100a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 --2- 172.21.2.117:0/1920054710 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7100a47d0 0x7fa7100a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 >> 172.21.2.117:0/1920054710 conn(0x7fa7100b5140 msgr2=0x7fa7100a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 shutdown_connections 2021-02-10T11:26:15.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:15.704 7fa718610700 1 -- 172.21.2.117:0/1920054710 wait complete. 2021-02-10T11:26:15.707 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802447 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816 2021-02-10T11:26:15.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:15.707 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:15.707 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:16.043 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:16.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:16.044 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:16.423 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:16.423 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:16.423 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239684 s, 43.7 MB/s 2021-02-10T11:26:16.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:16.424 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:16.424 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:16.759 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:16.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:16.760 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:17.133 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:17.133 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:17.133 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245997 s, 42.6 MB/s 2021-02-10T11:26:17.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.134 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:17.134 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:17.468 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:17.468 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:17.469 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:17.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:17.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:17.839 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242215 s, 43.3 MB/s 2021-02-10T11:26:17.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:17.840 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:17.841 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:18.175 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:18.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:18.176 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:18.551 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:18.551 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:18.552 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253083 s, 41.4 MB/s 2021-02-10T11:26:18.552 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:18.552 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:18.553 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:18.888 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:18.888 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:18.888 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:19.264 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:19.265 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:19.265 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026268 s, 39.9 MB/s 2021-02-10T11:26:19.266 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.266 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.266 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:19.599 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:19.600 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:19.600 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:19.975 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245158 s, 42.8 MB/s 2021-02-10T11:26:19.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:19.976 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:19.976 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:20.309 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:20.309 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:20.310 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:20.685 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:20.685 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:20.685 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242811 s, 43.2 MB/s 2021-02-10T11:26:20.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:20.686 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:20.686 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:21.022 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:21.022 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:21.022 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:21.397 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:21.397 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:21.397 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241805 s, 43.4 MB/s 2021-02-10T11:26:21.398 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:21.398 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:21.398 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:21.732 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:21.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:21.733 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:22.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:22.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:22.105 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227359 s, 46.1 MB/s 2021-02-10T11:26:22.106 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:22.106 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:22.106 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:22.439 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:22.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:22.440 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802447/cbb5292c-cdb6-42ce-b2d0-71dd399d4816/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:22.809 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:22.809 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:22.810 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240494 s, 43.6 MB/s 2021-02-10T11:26:22.811 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802448 2021-02-10T11:26:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.229 7fcf3e53e700 1 Processor -- start 2021-02-10T11:26:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.229 7fcf3e53e700 1 -- start start 2021-02-10T11:26:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.229 7fcf3e53e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 0x7fcf380771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.229 7fcf3e53e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf380e4da0 0x7fcf381b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf3e53e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf38077a70 con 0x7fcf380e51a0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf3e53e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf380e5c20 con 0x7fcf380e4da0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf3e53e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380777d0 con 0x7fcf3807da20 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf380e4da0 0x7fcf381b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf380e4da0 0x7fcf381b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43476/0 (socket says 172.21.2.117:43476) 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf3cb5b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcf380e51a0 0x7fcf38077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43726/0 (socket says 172.21.2.117:43726) 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 -- 172.21.2.117:0/1567662601 learned_addr learned my addr 172.21.2.117:0/1567662601 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf36ffd700 1 -- 172.21.2.117:0/1567662601 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849987144 0 0) 0x7fcf30001480 con 0x7fcf380e51a0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf36ffd700 1 -- 172.21.2.117:0/1567662601 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf20002fe0 con 0x7fcf380e51a0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 -- 172.21.2.117:0/1567662601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 msgr2=0x7fcf380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf37fff700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 0x7fcf380771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 0x7fcf380771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 -- 172.21.2.117:0/1567662601 >> v1:172.21.2.117:6789/0 conn(0x7fcf380e51a0 legacy=0x7fcf38077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 -- 172.21.2.117:0/1567662601 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3807de20 con 0x7fcf380e4da0 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf37fff700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 0x7fcf380771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf377fe700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf380e4da0 0x7fcf381b00f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fcf2c006700 tx=0x7fcf2c009b80).ready entity=mon.0 client_cookie=58d7957fffd62c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.230 7fcf36ffd700 1 -- 172.21.2.117:0/1567662601 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf2c00c0c0 con 0x7fcf380e4da0 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf36ffd700 1 -- 172.21.2.117:0/1567662601 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c00c400 con 0x7fcf380e4da0 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf36ffd700 1 -- 172.21.2.117:0/1567662601 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf2c00c8e0 con 0x7fcf380e4da0 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 -- 172.21.2.117:0/1567662601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf380e4da0 msgr2=0x7fcf381b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf380e4da0 0x7fcf381b00f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fcf2c006700 tx=0x7fcf2c009b80).stop 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 -- 172.21.2.117:0/1567662601 shutdown_connections 2021-02-10T11:26:23.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf380e4da0 0x7fcf381b00f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 --2- 172.21.2.117:0/1567662601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf3807da20 0x7fcf380771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 -- 172.21.2.117:0/1567662601 >> 172.21.2.117:0/1567662601 conn(0x7fcf380ce450 msgr2=0x7fcf380e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:23.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 -- 172.21.2.117:0/1567662601 shutdown_connections 2021-02-10T11:26:23.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.231 7fcf3e53e700 1 -- 172.21.2.117:0/1567662601 wait complete. 2021-02-10T11:26:23.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 Processor -- start 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 -- start start 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf381b5130 0x7fcf381b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf381b5f50 0x7fcf380b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380b8da0 con 0x7fcf381b5b50 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380a4140 con 0x7fcf381b5f50 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.246 7fcf3e53e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf380a4320 con 0x7fcf381b5130 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43486/0 (socket says 172.21.2.117:43486) 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 -- 172.21.2.117:0/976874975 learned_addr learned my addr 172.21.2.117:0/976874975 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 -- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf381b5130 msgr2=0x7fcf381b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf381b5130 0x7fcf381b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 -- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf381b5f50 msgr2=0x7fcf380b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf381b5f50 0x7fcf380b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 -- 172.21.2.117:0/976874975 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf380b62b0 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf377fe700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fcf2c0066a0 tx=0x7fcf2c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf2c00bd80 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c0052e0 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.247 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf380b62b0 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.248 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf2c0057c0 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.248 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf380b6840 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.248 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf2c006030 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.248 7fcf34ff9700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 0x7fcf1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.248 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcf2c0325f0 con 0x7fcf381b5b50 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.249 7fcf37fff700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 0x7fcf1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.249 7fcf37fff700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 0x7fcf1801abc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.279 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf381b4f00 con 0x7fcf381b5b50 2021-02-10T11:26:23.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.282 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf2c005a80 con 0x7fcf381b5b50 2021-02-10T11:26:23.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.407 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802448", "target": ["mgr", ""]}) v1 -- 0x7fcf380b8da0 con 0x7fcf180186c0 2021-02-10T11:26:23.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.432 7fcf34ff9700 1 -- 172.21.2.117:0/976874975 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcf280077a0 con 0x7fcf180186c0 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 msgr2=0x7fcf1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 0x7fcf1801abc0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 msgr2=0x7fcf380b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fcf2c0066a0 tx=0x7fcf2c00b990).stop 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 shutdown_connections 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf180186c0 0x7fcf1801abc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf381b5b50 0x7fcf380b5670 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf381b5f50 0x7fcf380b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 --2- 172.21.2.117:0/976874975 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf381b5130 0x7fcf381b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:23.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 >> 172.21.2.117:0/976874975 conn(0x7fcf3808bbc0 msgr2=0x7fcf380c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:23.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 shutdown_connections 2021-02-10T11:26:23.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.441 7fcf3e53e700 1 -- 172.21.2.117:0/976874975 wait complete. 2021-02-10T11:26:23.444 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802448 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 Processor -- start 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 -- start start 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd830079000 0x7fd8300e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8300d1d00 con 0x7fd83007da20 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd830077a70 con 0x7fd8301b00d0 2021-02-10T11:26:23.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd834a3c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300e53a0 con 0x7fd830079000 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd82de1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd830079000 0x7fd8300e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd82de1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd830079000 0x7fd8300e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43490/0 (socket says 172.21.2.117:43490) 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd82e61c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd83007da20 0x7fd8300771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43740/0 (socket says 172.21.2.117:43740) 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.862 7fd82de1b700 1 -- 172.21.2.117:0/239235086 learned_addr learned my addr 172.21.2.117:0/239235086 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82de1b700 1 -- 172.21.2.117:0/239235086 >> v1:172.21.2.125:6789/0 conn(0x7fd8301b00d0 legacy=0x7fd8300d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82de1b700 1 -- 172.21.2.117:0/239235086 >> v1:172.21.2.117:6789/0 conn(0x7fd83007da20 legacy=0x7fd8300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82de1b700 1 -- 172.21.2.117:0/239235086 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830079400 con 0x7fd830079000 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82de1b700 1 --2- 172.21.2.117:0/239235086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830079000 0x7fd8300e4d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fd818006700 tx=0x7fd818009b80).ready entity=mon.0 client_cookie=4eab947f0a974101 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82d61a700 1 -- 172.21.2.117:0/239235086 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd81800c0c0 con 0x7fd830079000 2021-02-10T11:26:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82d61a700 1 -- 172.21.2.117:0/239235086 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81800c340 con 0x7fd830079000 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd82d61a700 1 -- 172.21.2.117:0/239235086 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd81800c860 con 0x7fd830079000 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd834a3c700 1 -- 172.21.2.117:0/239235086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830079000 msgr2=0x7fd8300e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.863 7fd834a3c700 1 --2- 172.21.2.117:0/239235086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830079000 0x7fd8300e4d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fd818006700 tx=0x7fd818009b80).stop 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.864 7fd834a3c700 1 -- 172.21.2.117:0/239235086 shutdown_connections 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.864 7fd834a3c700 1 --2- 172.21.2.117:0/239235086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830079000 0x7fd8300e4d80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.864 7fd834a3c700 1 -- 172.21.2.117:0/239235086 >> 172.21.2.117:0/239235086 conn(0x7fd8300ce430 msgr2=0x7fd8300e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.864 7fd834a3c700 1 -- 172.21.2.117:0/239235086 shutdown_connections 2021-02-10T11:26:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.864 7fd834a3c700 1 -- 172.21.2.117:0/239235086 wait complete. 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 Processor -- start 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 -- start start 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd830093250 0x7fd83010ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd83010f8c0 0x7fd83009de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83010fd40 con 0x7fd830093650 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83009a060 con 0x7fd830093250 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd834a3c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83009e480 con 0x7fd83010f8c0 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd82de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd82de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43496/0 (socket says 172.21.2.117:43496) 2021-02-10T11:26:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd82de1b700 1 -- 172.21.2.117:0/3610635687 learned_addr learned my addr 172.21.2.117:0/3610635687 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd82ee1d700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd83010f8c0 0x7fd83009de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.879 7fd82e61c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd830093250 0x7fd83010ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 -- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd83010f8c0 msgr2=0x7fd83009de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd83010f8c0 0x7fd83009de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 -- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd830093250 msgr2=0x7fd83010ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd830093250 0x7fd83010ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 -- 172.21.2.117:0/3610635687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83009e6a0 con 0x7fd830093650 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd82de1b700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fd81800b5e0 tx=0x7fd81800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd81800bd70 con 0x7fd830093650 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818002030 con 0x7fd830093650 2021-02-10T11:26:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd818002550 con 0x7fd830093650 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83009e6a0 con 0x7fd830093650 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.880 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83009ec30 con 0x7fd830093650 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.881 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd818000c90 con 0x7fd830093650 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.881 7fd826ffd700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.881 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd8180325a0 con 0x7fd830093650 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.881 7fd82e61c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.882 7fd82e61c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 0x7fd81401abc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:23.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.912 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd830106060 con 0x7fd830093650 2021-02-10T11:26:23.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:23.915 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8180009b0 con 0x7fd830093650 2021-02-10T11:26:24.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.043 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802448", "target": ["mgr", ""]}) v1 -- 0x7fd83009a060 con 0x7fd8140186c0 2021-02-10T11:26:24.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.046 7fd826ffd700 1 -- 172.21.2.117:0/3610635687 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd8200069f0 con 0x7fd8140186c0 2021-02-10T11:26:24.040 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 msgr2=0x7fd81401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 0x7fd81401abc0 crc :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 msgr2=0x7fd83010f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fd81800b5e0 tx=0x7fd81800b980).stop 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 shutdown_connections 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8140186c0 0x7fd81401abc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:24.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd830093650 0x7fd83010f2a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:24.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd83010f8c0 0x7fd83009de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:24.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 --2- 172.21.2.117:0/3610635687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd830093250 0x7fd83010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:24.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 >> 172.21.2.117:0/3610635687 conn(0x7fd830108870 msgr2=0x7fd830093fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:24.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 shutdown_connections 2021-02-10T11:26:24.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:24.047 7fd834a3c700 1 -- 172.21.2.117:0/3610635687 wait complete. 2021-02-10T11:26:24.050 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802448 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53 2021-02-10T11:26:24.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:24.050 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:24.050 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:24.385 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:24.385 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:24.386 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:24.777 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:24.778 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:24.778 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244902 s, 42.8 MB/s 2021-02-10T11:26:24.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:24.779 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:24.779 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:25.076 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:25.080 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:25.116 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:25.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:25.117 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:25.451 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:25.451 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:25.451 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231792 s, 45.2 MB/s 2021-02-10T11:26:25.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:25.454 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:25.454 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:25.827 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:25.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:25.828 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:26.204 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:26.204 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:26.204 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247354 s, 42.4 MB/s 2021-02-10T11:26:26.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:26.205 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:26.205 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:26.540 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:26.541 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:26.541 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:26.915 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:26.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:26.916 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250193 s, 41.9 MB/s 2021-02-10T11:26:26.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:26.917 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:26.917 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:27.251 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:27.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:27.252 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:27.622 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:27.623 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:27.623 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227366 s, 46.1 MB/s 2021-02-10T11:26:27.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:27.624 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:27.624 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:27.958 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:27.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:27.959 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:28.334 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:28.334 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:28.334 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249627 s, 42.0 MB/s 2021-02-10T11:26:28.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:28.335 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:28.335 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:28.668 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:28.669 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:28.669 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:29.042 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:29.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:29.043 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236998 s, 44.2 MB/s 2021-02-10T11:26:29.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:29.044 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:29.044 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:29.378 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:29.378 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:29.378 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:29.753 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:29.753 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:29.754 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245461 s, 42.7 MB/s 2021-02-10T11:26:29.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:29.755 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:29.755 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:30.090 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:30.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:30.091 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:30.466 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:30.466 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:30.466 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247761 s, 42.3 MB/s 2021-02-10T11:26:30.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:30.468 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:30.468 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:30.803 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:30.804 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:30.804 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802448/4db3ae0b-b3f9-46b2-b8bf-a2dee812ad53/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:31.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:31.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:31.181 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247959 s, 42.3 MB/s 2021-02-10T11:26:31.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802449 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.602 7f7c33981700 1 Processor -- start 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.602 7f7c33981700 1 -- start start 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c077a70 con 0x7f7c2c07da20 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c0e5c20 con 0x7f7c2c1b00f0 2021-02-10T11:26:31.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c0e53c0 con 0x7f7c2c079000 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43508/0 (socket says 172.21.2.117:43508) 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 -- 172.21.2.117:0/2834273581 learned_addr learned my addr 172.21.2.117:0/2834273581 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 -- 172.21.2.117:0/2834273581 >> v2:172.21.2.125:3300/0 conn(0x7f7c2c079000 msgr2=0x7f7c2c0e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 --2- 172.21.2.117:0/2834273581 >> v2:172.21.2.125:3300/0 conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 -- 172.21.2.117:0/2834273581 >> v1:172.21.2.125:6789/0 conn(0x7f7c2c07da20 legacy=0x7f7c2c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 -- 172.21.2.117:0/2834273581 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c079400 con 0x7f7c2c1b00f0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c31f9e700 1 --2- 172.21.2.117:0/2834273581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7c24006f00 tx=0x7f7c24009f10).ready entity=mon.0 client_cookie=998acc4b68ac195d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c1ffff700 1 -- 172.21.2.117:0/2834273581 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2400b3b0 con 0x7f7c2c1b00f0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c1ffff700 1 -- 172.21.2.117:0/2834273581 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2400b730 con 0x7f7c2c1b00f0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c1ffff700 1 -- 172.21.2.117:0/2834273581 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2400bc10 con 0x7f7c2c1b00f0 2021-02-10T11:26:31.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 -- 172.21.2.117:0/2834273581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c1b00f0 msgr2=0x7f7c2c1b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.603 7f7c33981700 1 --2- 172.21.2.117:0/2834273581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f7c24006f00 tx=0x7f7c24009f10).stop 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 -- 172.21.2.117:0/2834273581 shutdown_connections 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 --2- 172.21.2.117:0/2834273581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c1b00f0 0x7f7c2c1b04f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 --2- 172.21.2.117:0/2834273581 >> v2:172.21.2.125:3300/0 conn(0x7f7c2c079000 0x7f7c2c0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 -- 172.21.2.117:0/2834273581 >> 172.21.2.117:0/2834273581 conn(0x7f7c2c0ce450 msgr2=0x7f7c2c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 -- 172.21.2.117:0/2834273581 shutdown_connections 2021-02-10T11:26:31.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.604 7f7c33981700 1 -- 172.21.2.117:0/2834273581 wait complete. 2021-02-10T11:26:31.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 Processor -- start 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 -- start start 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c09c780 0x7f7c2c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c09cb80 0x7f7c2c0c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0bff40 con 0x7f7c2c0b4250 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0bfb60 con 0x7f7c2c09c780 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c33981700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c0bfd40 con 0x7f7c2c09cb80 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c31f9e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c31f9e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43514/0 (socket says 172.21.2.117:43514) 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.619 7f7c31f9e700 1 -- 172.21.2.117:0/2341391484 learned_addr learned my addr 172.21.2.117:0/2341391484 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 -- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c09cb80 msgr2=0x7f7c2c0c04c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c09cb80 0x7f7c2c0c04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 -- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c09c780 msgr2=0x7f7c2c0b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c09c780 0x7f7c2c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 -- 172.21.2.117:0/2341391484 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0c1100 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c31f9e700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f7c24009390 tx=0x7f7c2400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c24003030 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c240068e0 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c33981700 1 -- 172.21.2.117:0/2341391484 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c0c1100 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c2400c570 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.620 7f7c33981700 1 -- 172.21.2.117:0/2341391484 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c0c1690 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.621 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c24001270 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.621 7f7c1dffb700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 0x7f7c0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.621 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7c24032c80 con 0x7f7c2c0b4250 2021-02-10T11:26:31.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.622 7f7c3179d700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 0x7f7c0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:31.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.622 7f7c3179d700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 0x7f7c0c01abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:31.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.653 7f7c33981700 1 -- 172.21.2.117:0/2341391484 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c0ff170 con 0x7f7c2c0b4250 2021-02-10T11:26:31.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.656 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c24009450 con 0x7f7c2c0b4250 2021-02-10T11:26:31.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.780 7f7c33981700 1 -- 172.21.2.117:0/2341391484 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802449", "target": ["mgr", ""]}) v1 -- 0x7f7c2c0c1690 con 0x7f7c0c0186c0 2021-02-10T11:26:31.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.807 7f7c1dffb700 1 -- 172.21.2.117:0/2341391484 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7c28006a10 con 0x7f7c0c0186c0 2021-02-10T11:26:31.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.813 7f7c33981700 1 -- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 msgr2=0x7f7c0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.813 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 0x7f7c0c01abc0 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.813 7f7c33981700 1 -- 172.21.2.117:0/2341391484 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 msgr2=0x7f7c2c0c0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:31.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.813 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f7c24009390 tx=0x7f7c2400bd40).stop 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 -- 172.21.2.117:0/2341391484 shutdown_connections 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c0c0186c0 0x7f7c0c01abc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c2c0b4250 0x7f7c2c0c0ae0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c2c09cb80 0x7f7c2c0c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 --2- 172.21.2.117:0/2341391484 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c2c09c780 0x7f7c2c0b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 -- 172.21.2.117:0/2341391484 >> 172.21.2.117:0/2341391484 conn(0x7f7c2c09d3b0 msgr2=0x7f7c2c0b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 -- 172.21.2.117:0/2341391484 shutdown_connections 2021-02-10T11:26:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:31.814 7f7c33981700 1 -- 172.21.2.117:0/2341391484 wait complete. 2021-02-10T11:26:31.817 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802449 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 Processor -- start 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 -- start start 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c07da20 0x7f709c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f709c0e4da0 0x7f709c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f709c077a70 con 0x7f709c0e51a0 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709c0e5c20 con 0x7f709c0e4da0 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.236 7f70a0c28700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709c0777d0 con 0x7f709c07da20 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f709c0e4da0 0x7f709c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f709c0e4da0 0x7f709c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43518/0 (socket says 172.21.2.117:43518) 2021-02-10T11:26:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 -- 172.21.2.117:0/490312767 learned_addr learned my addr 172.21.2.117:0/490312767 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 -- 172.21.2.117:0/490312767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c07da20 msgr2=0x7f709c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 --2- 172.21.2.117:0/490312767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c07da20 0x7f709c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 -- 172.21.2.117:0/490312767 >> v1:172.21.2.117:6789/0 conn(0x7f709c0e51a0 legacy=0x7f709c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 -- 172.21.2.117:0/490312767 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c07de20 con 0x7f709c0e4da0 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709a05d700 1 --2- 172.21.2.117:0/490312767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c0e4da0 0x7f709c1b00f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f7090006700 tx=0x7f7090009b80).ready entity=mon.0 client_cookie=e64af0c4eae2264e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709985c700 1 -- 172.21.2.117:0/490312767 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709000c0c0 con 0x7f709c0e4da0 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709985c700 1 -- 172.21.2.117:0/490312767 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709000c340 con 0x7f709c0e4da0 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f709985c700 1 -- 172.21.2.117:0/490312767 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709000c860 con 0x7f709c0e4da0 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f70a0c28700 1 -- 172.21.2.117:0/490312767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c0e4da0 msgr2=0x7f709c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.237 7f70a0c28700 1 --2- 172.21.2.117:0/490312767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c0e4da0 0x7f709c1b00f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f7090006700 tx=0x7f7090009b80).stop 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 -- 172.21.2.117:0/490312767 shutdown_connections 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 --2- 172.21.2.117:0/490312767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c0e4da0 0x7f709c1b00f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 --2- 172.21.2.117:0/490312767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c07da20 0x7f709c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 -- 172.21.2.117:0/490312767 >> 172.21.2.117:0/490312767 conn(0x7f709c0ce450 msgr2=0x7f709c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:32.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 -- 172.21.2.117:0/490312767 shutdown_connections 2021-02-10T11:26:32.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.238 7f70a0c28700 1 -- 172.21.2.117:0/490312767 wait complete. 2021-02-10T11:26:32.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 Processor -- start 2021-02-10T11:26:32.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 -- start start 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c107c20 0x7f709c096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f709c1a7240 0x7f709c1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709c108020 con 0x7f709c08a180 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f709c1a7c60 con 0x7f709c1a7240 2021-02-10T11:26:32.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f70a0c28700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f709c1a7e80 con 0x7f709c107c20 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f709a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f709a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43524/0 (socket says 172.21.2.117:43524) 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.253 7f709a05d700 1 -- 172.21.2.117:0/1518053496 learned_addr learned my addr 172.21.2.117:0/1518053496 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 -- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c107c20 msgr2=0x7f709c096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c107c20 0x7f709c096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 -- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f709c1a7240 msgr2=0x7f709c1a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f709c1a7240 0x7f709c1a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 -- 172.21.2.117:0/1518053496 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709c08b250 con 0x7f709c08a180 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f709a05d700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f709000b5e0 tx=0x7f709000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709000bd70 con 0x7f709c08a180 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090002030 con 0x7f709c08a180 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f709c08b250 con 0x7f709c08a180 2021-02-10T11:26:32.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709c08b7e0 con 0x7f709c08a180 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.254 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7090002550 con 0x7f709c08a180 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.255 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7090000c90 con 0x7f709c08a180 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.255 7f708b7fe700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.255 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7090032730 con 0x7f709c08a180 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.255 7f709a85e700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:32.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.256 7f709a85e700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 0x7f708001abc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:32.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.286 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709c077a70 con 0x7f709c08a180 2021-02-10T11:26:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.289 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70900009b0 con 0x7f709c08a180 2021-02-10T11:26:32.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.416 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802449", "target": ["mgr", ""]}) v1 -- 0x7f709c08b7e0 con 0x7f70800186c0 2021-02-10T11:26:32.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.419 7f708b7fe700 1 -- 172.21.2.117:0/1518053496 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f708c006a10 con 0x7f70800186c0 2021-02-10T11:26:32.414 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28 2021-02-10T11:26:32.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 msgr2=0x7f708001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 0x7f708001abc0 crc :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 msgr2=0x7f709c08a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f709000b5e0 tx=0x7f709000b980).stop 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 shutdown_connections 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f70800186c0 0x7f708001abc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f709c08a180 0x7f709c08a580 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f709c1a7240 0x7f709c1a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 --2- 172.21.2.117:0/1518053496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f709c107c20 0x7f709c096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:32.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 >> 172.21.2.117:0/1518053496 conn(0x7f709c0a1880 msgr2=0x7f709c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:32.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 shutdown_connections 2021-02-10T11:26:32.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:32.421 7f70a0c28700 1 -- 172.21.2.117:0/1518053496 wait complete. 2021-02-10T11:26:32.424 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802449 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28 2021-02-10T11:26:32.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:32.425 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:32.425 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:32.760 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:32.760 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:32.761 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:33.139 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:33.139 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:33.139 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246158 s, 42.6 MB/s 2021-02-10T11:26:33.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:33.141 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:33.141 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:33.475 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:33.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:33.476 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:33.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:33.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:33.851 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233721 s, 44.9 MB/s 2021-02-10T11:26:33.852 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:33.852 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:33.852 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:34.189 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:34.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:34.189 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:34.567 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:34.567 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:34.567 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254556 s, 41.2 MB/s 2021-02-10T11:26:34.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:34.569 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:34.569 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:34.904 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:34.905 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:34.905 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:35.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:35.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:35.281 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249941 s, 42.0 MB/s 2021-02-10T11:26:35.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:35.282 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:35.283 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:35.616 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:35.617 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:35.617 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:35.991 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:35.991 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:35.991 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233266 s, 45.0 MB/s 2021-02-10T11:26:35.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:35.992 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:35.992 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:36.327 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:36.327 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:36.328 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:36.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:36.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:36.705 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025193 s, 41.6 MB/s 2021-02-10T11:26:36.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:36.706 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:36.706 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:37.039 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:37.040 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:37.040 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0277079 s, 37.8 MB/s 2021-02-10T11:26:37.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:37.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:37.419 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:37.753 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:37.753 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:37.753 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:38.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:38.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:38.127 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248626 s, 42.2 MB/s 2021-02-10T11:26:38.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:38.128 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:38.128 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:38.463 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:38.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:38.464 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:38.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:38.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:38.839 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252438 s, 41.5 MB/s 2021-02-10T11:26:38.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:38.841 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:38.841 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:39.175 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:39.175 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:39.176 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802449/bf2e16f6-c7a2-4d38-99ce-7534fcc12d28/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:39.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:39.550 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:39.551 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236209 s, 44.4 MB/s 2021-02-10T11:26:39.552 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802450 2021-02-10T11:26:39.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 Processor -- start 2021-02-10T11:26:39.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 -- start start 2021-02-10T11:26:39.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 0x7f77840e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f77841b00f0 0x7f77841b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7784077a70 con 0x7f778407da20 2021-02-10T11:26:39.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840e53c0 con 0x7f77841b00f0 2021-02-10T11:26:39.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778c329700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7784079400 con 0x7f7784079000 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778a145700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f778407da20 0x7f77840771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43780/0 (socket says 172.21.2.117:43780) 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.970 7f778a145700 1 -- 172.21.2.117:0/2736140990 learned_addr learned my addr 172.21.2.117:0/2736140990 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 --2- 172.21.2.117:0/2736140990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 0x7f77840e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f778a946700 1 --2- 172.21.2.117:0/2736140990 >> v2:172.21.2.125:3300/0 conn(0x7f77841b00f0 0x7f77841b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789143700 1 -- 172.21.2.117:0/2736140990 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644899796 0 0) 0x7f77800010b0 con 0x7f778407da20 2021-02-10T11:26:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789143700 1 -- 172.21.2.117:0/2736140990 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7770002fe0 con 0x7f778407da20 2021-02-10T11:26:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 -- 172.21.2.117:0/2736140990 >> v2:172.21.2.125:3300/0 conn(0x7f77841b00f0 msgr2=0x7f77841b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 --2- 172.21.2.117:0/2736140990 >> v2:172.21.2.125:3300/0 conn(0x7f77841b00f0 0x7f77841b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 -- 172.21.2.117:0/2736140990 >> v1:172.21.2.117:6789/0 conn(0x7f778407da20 legacy=0x7f77840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 -- 172.21.2.117:0/2736140990 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778407de20 con 0x7f7784079000 2021-02-10T11:26:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.971 7f7789944700 1 --2- 172.21.2.117:0/2736140990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 0x7f77840e4da0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f7778006650 tx=0x7f7778009a80).ready entity=mon.2 client_cookie=573971409b901e36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f7789143700 1 -- 172.21.2.117:0/2736140990 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77780009b0 con 0x7f7784079000 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f7789143700 1 -- 172.21.2.117:0/2736140990 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777800c0c0 con 0x7f7784079000 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f7789143700 1 -- 172.21.2.117:0/2736140990 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f777800c550 con 0x7f7784079000 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 -- 172.21.2.117:0/2736140990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 msgr2=0x7f77840e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 --2- 172.21.2.117:0/2736140990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 0x7f77840e4da0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f7778006650 tx=0x7f7778009a80).stop 2021-02-10T11:26:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 -- 172.21.2.117:0/2736140990 shutdown_connections 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 --2- 172.21.2.117:0/2736140990 >> v2:172.21.2.125:3300/0 conn(0x7f77841b00f0 0x7f77841b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 --2- 172.21.2.117:0/2736140990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784079000 0x7f77840e4da0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 -- 172.21.2.117:0/2736140990 >> 172.21.2.117:0/2736140990 conn(0x7f77840ce450 msgr2=0x7f77840e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 -- 172.21.2.117:0/2736140990 shutdown_connections 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.972 7f778c329700 1 -- 172.21.2.117:0/2736140990 wait complete. 2021-02-10T11:26:39.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 Processor -- start 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 -- start start 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77841a0250 0x7f7784106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 0x7f7784090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77841067f0 con 0x7f7784187e20 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784107350 con 0x7f77841a0250 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.987 7f778c329700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77841042d0 con 0x7f7784090420 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43538/0 (socket says 172.21.2.117:43538) 2021-02-10T11:26:39.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 -- 172.21.2.117:0/114184329 learned_addr learned my addr 172.21.2.117:0/114184329 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f778a145700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77841a0250 0x7f7784106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f778a946700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 0x7f7784090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 -- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 msgr2=0x7f7784090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 0x7f7784090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 -- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77841a0250 msgr2=0x7f7784106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:39.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77841a0250 0x7f7784106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 -- 172.21.2.117:0/114184329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7784090e40 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f778a946700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 0x7f7784090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7789944700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7778009630 tx=0x7f777800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77780093a0 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777800bd70 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f778c329700 1 -- 172.21.2.117:0/114184329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7784090e40 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f778c329700 1 -- 172.21.2.117:0/114184329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77840913d0 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.988 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7778005560 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.989 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f777800a000 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.989 7f7776ffd700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 0x7f776c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.989 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7778032950 con 0x7f7784187e20 2021-02-10T11:26:39.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.990 7f778a145700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 0x7f776c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:39.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:39.990 7f778a145700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 0x7f776c01ac10 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.020 7f778c329700 1 -- 172.21.2.117:0/114184329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7784103280 con 0x7f7784187e20 2021-02-10T11:26:40.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.023 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7778009340 con 0x7f7784187e20 2021-02-10T11:26:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.148 7f778c329700 1 -- 172.21.2.117:0/114184329 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802450", "target": ["mgr", ""]}) v1 -- 0x7f7784091c60 con 0x7f776c018730 2021-02-10T11:26:40.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.174 7f7776ffd700 1 -- 172.21.2.117:0/114184329 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7780000ec0 con 0x7f776c018730 2021-02-10T11:26:40.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.181 7f778c329700 1 -- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 msgr2=0x7f776c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.181 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 0x7f776c01ac10 crc :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.181 7f778c329700 1 -- 172.21.2.117:0/114184329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 msgr2=0x7f7784188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.181 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7778009630 tx=0x7f777800b980).stop 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 -- 172.21.2.117:0/114184329 shutdown_connections 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f776c018730 0x7f776c01ac10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7784187e20 0x7f7784188220 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7784090420 0x7f7784090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 --2- 172.21.2.117:0/114184329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77841a0250 0x7f7784106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 -- 172.21.2.117:0/114184329 >> 172.21.2.117:0/114184329 conn(0x7f77840a1850 msgr2=0x7f77840c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 -- 172.21.2.117:0/114184329 shutdown_connections 2021-02-10T11:26:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.182 7f778c329700 1 -- 172.21.2.117:0/114184329 wait complete. 2021-02-10T11:26:40.184 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802450 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 Processor -- start 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 -- start start 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f59f0079000 0x7f59f00e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f01b00d0 0x7f59f01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f0077a70 con 0x7f59f007da20 2021-02-10T11:26:40.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f00e53a0 con 0x7f59f0079000 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59f55e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f0079400 con 0x7f59f01b00d0 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59ef2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f59f007da20 0x7f59f00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43794/0 (socket says 172.21.2.117:43794) 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.603 7f59ef2bf700 1 -- 172.21.2.117:0/3784170988 learned_addr learned my addr 172.21.2.117:0/3784170988 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505511954 0 0) 0x7f59d8001180 con 0x7f59f007da20 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59dc002fe0 con 0x7f59f007da20 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59efac0700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f01b00d0 0x7f59f01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59eeabe700 1 --2- 172.21.2.117:0/3784170988 >> v2:172.21.2.125:3300/0 conn(0x7f59f0079000 0x7f59f00e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4100475499 0 0) 0x7f59d8001180 con 0x7f59f007da20 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f01b00d0 msgr2=0x7f59f01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f01b00d0 0x7f59f01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 >> v2:172.21.2.125:3300/0 conn(0x7f59f0079000 msgr2=0x7f59f00e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 --2- 172.21.2.117:0/3784170988 >> v2:172.21.2.125:3300/0 conn(0x7f59f0079000 0x7f59f00e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f007de20 con 0x7f59f007da20 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f59d8001680 con 0x7f59f007da20 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 >> v1:172.21.2.117:6789/0 conn(0x7f59f007da20 legacy=0x7f59f00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59dc003ae0 0x7f59dc005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59dc0067f0 0x7f59dc008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59dc009590 0x7f59dc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.604 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc00c360 con 0x7f59dc0067f0 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc050550 con 0x7f59dc009590 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc050770 con 0x7f59dc003ae0 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59dc0067f0 0x7f59dc008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59efac0700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f01b00d0 0x7f59f01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ef2bf700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59dc003ae0 0x7f59dc005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59efac0700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59dc009590 0x7f59dc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 -- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59dc003ae0 msgr2=0x7f59dc005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59dc003ae0 0x7f59dc005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 -- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59dc009590 msgr2=0x7f59dc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59dc009590 0x7f59dc00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 -- 172.21.2.117:0/3784170988 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc050a10 con 0x7f59dc0067f0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59efac0700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59dc009590 0x7f59dc00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59eeabe700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59dc0067f0 0x7f59dc008cf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f59e0006360 tx=0x7f59e0009c30).ready entity=mon.0 client_cookie=bc546e29a4ca2483 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e00009b0 con 0x7f59dc0067f0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0000c90 con 0x7f59dc0067f0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59ee2bd700 1 -- 172.21.2.117:0/3784170988 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e00009b0 con 0x7f59dc0067f0 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59f55e4700 1 -- 172.21.2.117:0/3784170988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59dc0067f0 msgr2=0x7f59dc008cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59f55e4700 1 --2- 172.21.2.117:0/3784170988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59dc0067f0 0x7f59dc008cf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f59e0006360 tx=0x7f59e0009c30).stop 2021-02-10T11:26:40.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.605 7f59efac0700 1 -- 172.21.2.117:0/3784170988 reap_dead start 2021-02-10T11:26:40.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.606 7f59f55e4700 1 -- 172.21.2.117:0/3784170988 shutdown_connections 2021-02-10T11:26:40.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.606 7f59f55e4700 1 -- 172.21.2.117:0/3784170988 >> 172.21.2.117:0/3784170988 conn(0x7f59f00ce430 msgr2=0x7f59f00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:40.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.606 7f59f55e4700 1 -- 172.21.2.117:0/3784170988 shutdown_connections 2021-02-10T11:26:40.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.606 7f59f55e4700 1 -- 172.21.2.117:0/3784170988 wait complete. 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.621 7f59f55e4700 1 Processor -- start 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.621 7f59f55e4700 1 -- start start 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f00bc470 0x7f59f00bc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 0x7f59f00bb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00bbec0 con 0x7f59f008e820 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00bc0e0 con 0x7f59f00bb3d0 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59f55e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f00b5cd0 con 0x7f59f00bc470 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59ef2bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f00bc470 0x7f59f00bc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43558/0 (socket says 172.21.2.117:43558) 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 -- 172.21.2.117:0/1157820240 learned_addr learned my addr 172.21.2.117:0/1157820240 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:40.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59efac0700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 0x7f59f00bb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 -- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f00bc470 msgr2=0x7f59f00bc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f00bc470 0x7f59f00bc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 -- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 msgr2=0x7f59f00bb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 0x7f59f00bb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 -- 172.21.2.117:0/1157820240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f00b5ef0 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59efac0700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 0x7f59f00bb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.622 7f59eeabe700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f59e0009430 tx=0x7f59e000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e0002030 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0000980 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f00b5ef0 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f00b5ef0 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59e00024e0 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59e0000c90 con 0x7f59f008e820 2021-02-10T11:26:40.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 0x7f59d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.624 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f59e0032940 con 0x7f59f008e820 2021-02-10T11:26:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.625 7f59ef2bf700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 0x7f59d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.625 7f59ef2bf700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 0x7f59d401ac10 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:40.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.653 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f0077a70 con 0x7f59f008e820 2021-02-10T11:26:40.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.656 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59e001f530 con 0x7f59f008e820 2021-02-10T11:26:40.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.784 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802450", "target": ["mgr", ""]}) v1 -- 0x7f59f0077a70 con 0x7f59d4018730 2021-02-10T11:26:40.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.788 7f59e7fff700 1 -- 172.21.2.117:0/1157820240 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f59d8007ca0 con 0x7f59d4018730 2021-02-10T11:26:40.786 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 msgr2=0x7f59d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 0x7f59d401ac10 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 msgr2=0x7f59f008ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f59e0009430 tx=0x7f59e000b920).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 shutdown_connections 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59d4018730 0x7f59d401ac10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59f008e820 0x7f59f008ec20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f59f00bb3d0 0x7f59f00bb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 --2- 172.21.2.117:0/1157820240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59f00bc470 0x7f59f00bc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:40.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.793 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 >> 172.21.2.117:0/1157820240 conn(0x7f59f00be670 msgr2=0x7f59f01903e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:40.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.794 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 shutdown_connections 2021-02-10T11:26:40.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:40.794 7f59f55e4700 1 -- 172.21.2.117:0/1157820240 wait complete. 2021-02-10T11:26:40.796 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802450 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983 2021-02-10T11:26:40.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:40.796 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:40.796 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:41.131 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:41.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:41.132 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244127 s, 43.0 MB/s 2021-02-10T11:26:41.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:41.513 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:41.513 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:41.848 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:41.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:41.849 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:42.223 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:42.223 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023036 s, 45.5 MB/s 2021-02-10T11:26:42.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:42.224 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:42.224 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:42.560 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:42.561 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:42.561 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236667 s, 44.3 MB/s 2021-02-10T11:26:42.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:42.933 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:42.933 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:43.267 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:43.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:43.267 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:43.641 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:43.641 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233613 s, 44.9 MB/s 2021-02-10T11:26:43.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:43.642 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:43.642 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:43.977 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:43.977 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:43.978 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:44.354 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:44.354 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:44.354 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257232 s, 40.8 MB/s 2021-02-10T11:26:44.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:44.356 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:44.356 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:44.691 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:44.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:44.691 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:45.067 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:45.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:45.068 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255431 s, 41.1 MB/s 2021-02-10T11:26:45.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:45.069 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:45.069 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:45.403 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:45.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:45.403 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:45.778 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:45.778 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251057 s, 41.8 MB/s 2021-02-10T11:26:45.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:45.780 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:45.780 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:46.116 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:46.116 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:46.117 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:46.491 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:46.491 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:46.491 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248561 s, 42.2 MB/s 2021-02-10T11:26:46.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:46.493 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:46.493 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:46.829 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:46.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:46.829 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232733 s, 45.1 MB/s 2021-02-10T11:26:47.204 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:47.204 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:47.204 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:47.539 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:47.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:47.540 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802450/859ab296-200b-49b9-8f79-43316f57d983/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:47.912 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:47.912 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:47.912 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254238 s, 41.2 MB/s 2021-02-10T11:26:47.913 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802451 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 Processor -- start 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 -- start start 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe108079000 0x7fe1080e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1081b00f0 0x7fe1081b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe108077a70 con 0x7fe10807da20 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1080e53c0 con 0x7fe108079000 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10fb43700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe108079400 con 0x7fe1081b00f0 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10d95f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe10807da20 0x7fe1080771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43814/0 (socket says 172.21.2.117:43814) 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.332 7fe10d95f700 1 -- 172.21.2.117:0/2702784544 learned_addr learned my addr 172.21.2.117:0/2702784544 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876754533 0 0) 0x7fe1040010b0 con 0x7fe10807da20 2021-02-10T11:26:48.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f4002fe0 con 0x7fe10807da20 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10e160700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1081b00f0 0x7fe1081b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10d15e700 1 --2- 172.21.2.117:0/2702784544 >> v2:172.21.2.125:3300/0 conn(0x7fe108079000 0x7fe1080e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2664778281 0 0) 0x7fe1040010b0 con 0x7fe10807da20 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1081b00f0 msgr2=0x7fe1081b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1081b00f0 0x7fe1081b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 >> v2:172.21.2.125:3300/0 conn(0x7fe108079000 msgr2=0x7fe1080e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 --2- 172.21.2.117:0/2702784544 >> v2:172.21.2.125:3300/0 conn(0x7fe108079000 0x7fe1080e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10807de20 con 0x7fe10807da20 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe1040015b0 con 0x7fe10807da20 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 >> v1:172.21.2.117:6789/0 conn(0x7fe10807da20 legacy=0x7fe1080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0f4003a40 0x7fe0f4005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0f4006750 0x7fe0f4008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0f40094f0 0x7fe0f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f400c2c0 con 0x7fe0f4006750 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4050480 con 0x7fe0f40094f0 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.333 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40506a0 con 0x7fe0f4003a40 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0f4006750 0x7fe0f4008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d95f700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0f4003a40 0x7fe0f4005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10e160700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0f40094f0 0x7fe0f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 -- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0f4003a40 msgr2=0x7fe0f4005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0f4003a40 0x7fe0f4005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 -- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0f40094f0 msgr2=0x7fe0f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0f40094f0 0x7fe0f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 -- 172.21.2.117:0/2702784544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4050940 con 0x7fe0f4006750 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10d15e700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0f4006750 0x7fe0f4008c50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fe0f8006360 tx=0x7fe0f8009bb0).ready entity=mon.0 client_cookie=967486d11eb01ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0f8006560 con 0x7fe0f4006750 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f800c150 con 0x7fe0f4006750 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10c95d700 1 -- 172.21.2.117:0/2702784544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0f800c5a0 con 0x7fe0f4006750 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10fb43700 1 -- 172.21.2.117:0/2702784544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0f4006750 msgr2=0x7fe0f4008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10fb43700 1 --2- 172.21.2.117:0/2702784544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0f4006750 0x7fe0f4008c50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fe0f8006360 tx=0x7fe0f8009bb0).stop 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.334 7fe10e160700 1 -- 172.21.2.117:0/2702784544 reap_dead start 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.335 7fe10fb43700 1 -- 172.21.2.117:0/2702784544 shutdown_connections 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.335 7fe10fb43700 1 -- 172.21.2.117:0/2702784544 >> 172.21.2.117:0/2702784544 conn(0x7fe1080ce450 msgr2=0x7fe1080e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.335 7fe10fb43700 1 -- 172.21.2.117:0/2702784544 shutdown_connections 2021-02-10T11:26:48.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.335 7fe10fb43700 1 -- 172.21.2.117:0/2702784544 wait complete. 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 Processor -- start 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 -- start start 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10819bf70 0x7fe1080b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe108096ef0 0x7fe1080972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1080969f0 con 0x7fe1080c13b0 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe108097950 con 0x7fe10819bf70 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10fb43700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe108097c60 con 0x7fe108096ef0 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10d15e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe108096ef0 0x7fe1080972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10e160700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10e160700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43578/0 (socket says 172.21.2.117:43578) 2021-02-10T11:26:48.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.350 7fe10e160700 1 -- 172.21.2.117:0/2417233574 learned_addr learned my addr 172.21.2.117:0/2417233574 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:48.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10d95f700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10819bf70 0x7fe1080b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 -- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe108096ef0 msgr2=0x7fe1080972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe108096ef0 0x7fe1080972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 -- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10819bf70 msgr2=0x7fe1080b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10819bf70 0x7fe1080b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 -- 172.21.2.117:0/2417233574 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1080c1dd0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10e160700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe1000099e0 tx=0x7fe10000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe100006030 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1000062b0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1000056b0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.351 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1080c1dd0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.352 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1080c1dd0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.352 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1000056b0 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.352 7fe0fe7fc700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 0x7fe0f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.352 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe100032870 con 0x7fe1080c13b0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.352 7fe10d95f700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 0x7fe0f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.353 7fe10d95f700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 0x7fe0f001abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.382 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108077a70 con 0x7fe1080c13b0 2021-02-10T11:26:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.385 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe10002cf20 con 0x7fe1080c13b0 2021-02-10T11:26:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.511 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802451", "target": ["mgr", ""]}) v1 -- 0x7fe108077a70 con 0x7fe0f00186c0 2021-02-10T11:26:48.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.538 7fe0fe7fc700 1 -- 172.21.2.117:0/2417233574 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe104000e20 con 0x7fe0f00186c0 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 msgr2=0x7fe0f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 0x7fe0f001abc0 crc :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 msgr2=0x7fe1080c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fe1000099e0 tx=0x7fe10000bcf0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 shutdown_connections 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f00186c0 0x7fe0f001abc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1080c13b0 0x7fe1080c17b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe108096ef0 0x7fe1080972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 --2- 172.21.2.117:0/2417233574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10819bf70 0x7fe1080b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 >> 172.21.2.117:0/2417233574 conn(0x7fe108107f20 msgr2=0x7fe108187bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 shutdown_connections 2021-02-10T11:26:48.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.544 7fe10fb43700 1 -- 172.21.2.117:0/2417233574 wait complete. 2021-02-10T11:26:48.547 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802451 2021-02-10T11:26:48.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.968 7f9017102700 1 Processor -- start 2021-02-10T11:26:48.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.968 7f9017102700 1 -- start start 2021-02-10T11:26:48.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9017102700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f901007da20 0x7f90100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9017102700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90100e4da0 0x7f90101b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9017102700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9010077a70 con 0x7f90100e51a0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9017102700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90100e5c20 con 0x7f90100e4da0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9017102700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90100777d0 con 0x7f901007da20 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f901571f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f90100e51a0 0x7f9010077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43832/0 (socket says 172.21.2.117:43832) 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f901571f700 1 -- 172.21.2.117:0/3036299936 learned_addr learned my addr 172.21.2.117:0/3036299936 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900ffff700 1 --2- 172.21.2.117:0/3036299936 >> v2:172.21.2.117:3300/0 conn(0x7f90100e4da0 0x7f90101b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f9014f1e700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f901007da20 0x7f90100771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900f7fe700 1 -- 172.21.2.117:0/3036299936 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838327607 0 0) 0x7f9008001180 con 0x7f90100e51a0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900f7fe700 1 -- 172.21.2.117:0/3036299936 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc002fe0 con 0x7f90100e51a0 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900ffff700 1 -- 172.21.2.117:0/3036299936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f901007da20 msgr2=0x7f90100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900ffff700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f901007da20 0x7f90100771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900ffff700 1 -- 172.21.2.117:0/3036299936 >> v1:172.21.2.117:6789/0 conn(0x7f90100e51a0 legacy=0x7f9010077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.969 7f900ffff700 1 -- 172.21.2.117:0/3036299936 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f901007de20 con 0x7f90100e4da0 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f900ffff700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90100e4da0 0x7f90101b00f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f90000069d0 tx=0x7f9000009e40).ready entity=mon.0 client_cookie=7770fecd590e92aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f900f7fe700 1 -- 172.21.2.117:0/3036299936 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f900000b510 con 0x7f90100e4da0 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f900f7fe700 1 -- 172.21.2.117:0/3036299936 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900000b790 con 0x7f90100e4da0 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f900f7fe700 1 -- 172.21.2.117:0/3036299936 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f900000bcb0 con 0x7f90100e4da0 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f9017102700 1 -- 172.21.2.117:0/3036299936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90100e4da0 msgr2=0x7f90101b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.970 7f9017102700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90100e4da0 0x7f90101b00f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f90000069d0 tx=0x7f9000009e40).stop 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 -- 172.21.2.117:0/3036299936 shutdown_connections 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90100e4da0 0x7f90101b00f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 --2- 172.21.2.117:0/3036299936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f901007da20 0x7f90100771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 -- 172.21.2.117:0/3036299936 >> 172.21.2.117:0/3036299936 conn(0x7f90100ce450 msgr2=0x7f90100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 -- 172.21.2.117:0/3036299936 shutdown_connections 2021-02-10T11:26:48.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.971 7f9017102700 1 -- 172.21.2.117:0/3036299936 wait complete. 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.985 7f9017102700 1 Processor -- start 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.985 7f9017102700 1 -- start start 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 0x7f9010188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9010090420 0x7f9010090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90101067f0 con 0x7f90101a0250 2021-02-10T11:26:48.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9010107350 con 0x7f9010187e20 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9017102700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90101042d0 con 0x7f9010090420 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43588/0 (socket says 172.21.2.117:43588) 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 -- 172.21.2.117:0/2384523374 learned_addr learned my addr 172.21.2.117:0/2384523374 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f901571f700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9010090420 0x7f9010090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f900ffff700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 0x7f9010188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 -- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9010090420 msgr2=0x7f9010090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9010090420 0x7f9010090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 -- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 msgr2=0x7f9010188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 0x7f9010188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 -- 172.21.2.117:0/2384523374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010090e40 con 0x7f90101a0250 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f900ffff700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 0x7f9010188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:26:48.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.986 7f9014f1e700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8ff800b560 tx=0x7f8ff800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ff800bcf0 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff8000c90 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ff8002220 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f9017102700 1 -- 172.21.2.117:0/2384523374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9010090e40 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f9017102700 1 -- 172.21.2.117:0/2384523374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90100913d0 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ff8006b00 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 0x7f8ff401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.987 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ff8032560 con 0x7f90101a0250 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.988 7f900ffff700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 0x7f8ff401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:48.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:48.988 7f900ffff700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 0x7f8ff401abc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:49.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.018 7f9017102700 1 -- 172.21.2.117:0/2384523374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9010103280 con 0x7f90101a0250 2021-02-10T11:26:49.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.021 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ff8000940 con 0x7f90101a0250 2021-02-10T11:26:49.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.148 7f9017102700 1 -- 172.21.2.117:0/2384523374 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802451", "target": ["mgr", ""]}) v1 -- 0x7f90100913d0 con 0x7f8ff40186c0 2021-02-10T11:26:49.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.153 7f900d7fa700 1 -- 172.21.2.117:0/2384523374 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9000000d30 con 0x7f8ff40186c0 2021-02-10T11:26:49.150 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23 2021-02-10T11:26:49.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.157 7f9017102700 1 -- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 msgr2=0x7f8ff401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:49.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.157 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 0x7f8ff401abc0 crc :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 -- 172.21.2.117:0/2384523374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 msgr2=0x7f9010106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:49.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8ff800b560 tx=0x7f8ff800b900).stop 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 -- 172.21.2.117:0/2384523374 shutdown_connections 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9010090420 0x7f9010090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9010187e20 0x7f9010188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186c0 0x7f8ff401abc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 --2- 172.21.2.117:0/2384523374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90101a0250 0x7f9010106cf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 -- 172.21.2.117:0/2384523374 >> 172.21.2.117:0/2384523374 conn(0x7f90100a1850 msgr2=0x7f90100c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 -- 172.21.2.117:0/2384523374 shutdown_connections 2021-02-10T11:26:49.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:49.158 7f9017102700 1 -- 172.21.2.117:0/2384523374 wait complete. 2021-02-10T11:26:49.160 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802451 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23 2021-02-10T11:26:49.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:49.160 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:49.161 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:49.495 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:49.496 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:49.496 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:49.876 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:49.876 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:49.876 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237753 s, 44.1 MB/s 2021-02-10T11:26:49.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:49.877 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:49.877 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:50.211 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:50.212 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:50.212 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:50.588 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:50.589 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:50.589 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257226 s, 40.8 MB/s 2021-02-10T11:26:50.590 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:50.590 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:50.590 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:50.925 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:50.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:50.925 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:51.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:51.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:51.301 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245025 s, 42.8 MB/s 2021-02-10T11:26:51.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:51.302 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:51.302 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:51.637 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:51.637 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:51.637 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:52.012 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:52.012 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:52.012 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238598 s, 43.9 MB/s 2021-02-10T11:26:52.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:52.013 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:52.013 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:52.348 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:52.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:52.349 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:52.720 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:52.720 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:52.720 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245339 s, 42.7 MB/s 2021-02-10T11:26:52.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:52.721 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:52.721 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:53.055 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:53.055 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:53.056 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:53.430 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:53.430 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:53.430 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258677 s, 40.5 MB/s 2021-02-10T11:26:53.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:53.431 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:53.431 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:53.767 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:53.767 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:53.768 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260796 s, 40.2 MB/s 2021-02-10T11:26:54.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:54.145 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:54.145 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:54.480 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:54.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:54.480 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:54.853 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223754 s, 46.9 MB/s 2021-02-10T11:26:54.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:54.854 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:54.854 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:55.188 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:55.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:55.189 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:55.443 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:55.452 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:26:55.565 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:55.565 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:55.565 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238405 s, 44.0 MB/s 2021-02-10T11:26:55.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:55.566 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:55.566 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:55.904 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:55.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:55.904 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802451/d9694928-76bc-49c3-a7bd-1143d2b7bf23/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:56.278 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:56.279 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:56.279 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249652 s, 42.0 MB/s 2021-02-10T11:26:56.280 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802452 2021-02-10T11:26:56.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.698 7f09b0e9d700 1 Processor -- start 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.698 7f09b0e9d700 1 -- start start 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09b0e9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09ac07da20 0x7f09ac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09b0e9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 0x7f09ac079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09b0e9d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac077a70 con 0x7f09ac0e4da0 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09b0e9d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0e5c20 con 0x7f09ac07da20 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09b0e9d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0777d0 con 0x7f09ac0e51a0 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09aab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09ac07da20 0x7f09ac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09aa312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09ac0e4da0 0x7f09ac1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43844/0 (socket says 172.21.2.117:43844) 2021-02-10T11:26:56.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09aab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09ac07da20 0x7f09ac0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43594/0 (socket says 172.21.2.117:43594) 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09aa312700 1 -- 172.21.2.117:0/3679198319 learned_addr learned my addr 172.21.2.117:0/3679198319 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09ab314700 1 --2- 172.21.2.117:0/3679198319 >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 0x7f09ac079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.699 7f09a9b11700 1 -- 172.21.2.117:0/3679198319 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351579615 0 0) 0x7f0994000ce0 con 0x7f09ac0e4da0 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09a9b11700 1 -- 172.21.2.117:0/3679198319 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0998002fe0 con 0x7f09ac0e4da0 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09aab13700 1 -- 172.21.2.117:0/3679198319 >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 msgr2=0x7f09ac079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09aab13700 1 --2- 172.21.2.117:0/3679198319 >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 0x7f09ac079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09aab13700 1 -- 172.21.2.117:0/3679198319 >> v1:172.21.2.117:6789/0 conn(0x7f09ac0e4da0 legacy=0x7f09ac1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09aab13700 1 -- 172.21.2.117:0/3679198319 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac1b08e0 con 0x7f09ac07da20 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09ab314700 1 --2- 172.21.2.117:0/3679198319 >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 0x7f09ac079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09aab13700 1 --2- 172.21.2.117:0/3679198319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac07da20 0x7f09ac0771b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f099c006650 tx=0x7f099c009a80).ready entity=mon.0 client_cookie=d8f86e935868e02f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09a9b11700 1 -- 172.21.2.117:0/3679198319 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099c00c0c0 con 0x7f09ac07da20 2021-02-10T11:26:56.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09a9b11700 1 -- 172.21.2.117:0/3679198319 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c00c340 con 0x7f09ac07da20 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09a9b11700 1 -- 172.21.2.117:0/3679198319 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099c00c860 con 0x7f09ac07da20 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09b0e9d700 1 -- 172.21.2.117:0/3679198319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac07da20 msgr2=0x7f09ac0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.700 7f09b0e9d700 1 --2- 172.21.2.117:0/3679198319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac07da20 0x7f09ac0771b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f099c006650 tx=0x7f099c009a80).stop 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 -- 172.21.2.117:0/3679198319 shutdown_connections 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 --2- 172.21.2.117:0/3679198319 >> v2:172.21.2.125:3300/0 conn(0x7f09ac0e51a0 0x7f09ac079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 --2- 172.21.2.117:0/3679198319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac07da20 0x7f09ac0771b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 -- 172.21.2.117:0/3679198319 >> 172.21.2.117:0/3679198319 conn(0x7f09ac0ce450 msgr2=0x7f09ac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 -- 172.21.2.117:0/3679198319 shutdown_connections 2021-02-10T11:26:56.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.701 7f09b0e9d700 1 -- 172.21.2.117:0/3679198319 wait complete. 2021-02-10T11:26:56.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 Processor -- start 2021-02-10T11:26:56.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 -- start start 2021-02-10T11:26:56.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09ac08a180 0x7f09ac09b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09ac190110 0x7f09ac1a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac08a720 con 0x7f09ac1a7860 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0bb540 con 0x7f09ac08a180 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09b0e9d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0bb9e0 con 0x7f09ac190110 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09ab314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09ab314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43602/0 (socket says 172.21.2.117:43602) 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.716 7f09ab314700 1 -- 172.21.2.117:0/2582514579 learned_addr learned my addr 172.21.2.117:0/2582514579 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09aab13700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09ac08a180 0x7f09ac09b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09aa312700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09ac190110 0x7f09ac1a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 -- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09ac190110 msgr2=0x7f09ac1a7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09ac190110 0x7f09ac1a7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 -- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09ac08a180 msgr2=0x7f09ac09b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09ac08a180 0x7f09ac09b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 -- 172.21.2.117:0/2582514579 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac1a7c60 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09ab314700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f09a400b970 tx=0x7f09a400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a4000f90 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4003030 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.717 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac1a7c60 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.718 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac08b240 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.718 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09a40015f0 con 0x7f09ac1a7860 2021-02-10T11:26:56.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.718 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09a40011f0 con 0x7f09ac1a7860 2021-02-10T11:26:56.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.719 7f09a37fe700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 0x7f099001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:56.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.719 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f09a4032c80 con 0x7f09ac1a7860 2021-02-10T11:26:56.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.719 7f09aab13700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 0x7f099001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:56.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.719 7f09aab13700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 0x7f099001ac10 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:56.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.749 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac077a70 con 0x7f09ac1a7860 2021-02-10T11:26:56.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.751 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09a4000f70 con 0x7f09ac1a7860 2021-02-10T11:26:56.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.879 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802452", "target": ["mgr", ""]}) v1 -- 0x7f09ac08b240 con 0x7f0990018730 2021-02-10T11:26:56.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.906 7f09a37fe700 1 -- 172.21.2.117:0/2582514579 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f099c006420 con 0x7f0990018730 2021-02-10T11:26:56.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 msgr2=0x7f099001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 0x7f099001ac10 crc :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 msgr2=0x7f09ac08ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f09a400b970 tx=0x7f09a400bd10).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 shutdown_connections 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0990018730 0x7f099001ac10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09ac1a7860 0x7f09ac08ac20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09ac190110 0x7f09ac1a7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 --2- 172.21.2.117:0/2582514579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09ac08a180 0x7f09ac09b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 >> 172.21.2.117:0/2582514579 conn(0x7f09ac0a1360 msgr2=0x7f09ac091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 shutdown_connections 2021-02-10T11:26:56.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:56.912 7f09b0e9d700 1 -- 172.21.2.117:0/2582514579 wait complete. 2021-02-10T11:26:56.914 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802452 2021-02-10T11:26:57.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 Processor -- start 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 -- start start 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe62c07da20 0x7fe62c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe62c0e4da0 0x7fe62c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c0e51a0 0x7fe62c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c077a70 con 0x7fe62c07da20 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0777d0 con 0x7fe62c0e4da0 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.334 7fe631e2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c07de20 con 0x7fe62c0e51a0 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe62c07da20 0x7fe62c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe62c07da20 0x7fe62c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43608/0 (socket says 172.21.2.117:43608) 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 -- 172.21.2.117:0/2372068312 learned_addr learned my addr 172.21.2.117:0/2372068312 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 -- 172.21.2.117:0/2372068312 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c0e51a0 msgr2=0x7fe62c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 --2- 172.21.2.117:0/2372068312 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c0e51a0 0x7fe62c077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 -- 172.21.2.117:0/2372068312 >> v2:172.21.2.125:3300/0 conn(0x7fe62c0e4da0 msgr2=0x7fe62c1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 --2- 172.21.2.117:0/2372068312 >> v2:172.21.2.125:3300/0 conn(0x7fe62c0e4da0 0x7fe62c1b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 -- 172.21.2.117:0/2372068312 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c1b0710 con 0x7fe62c07da20 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62b7fe700 1 --2- 172.21.2.117:0/2372068312 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c07da20 0x7fe62c0771b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fe61c006700 tx=0x7fe61c009b80).ready entity=mon.0 client_cookie=fb23016681d9aede server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62a7fc700 1 -- 172.21.2.117:0/2372068312 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe61c00c0c0 con 0x7fe62c07da20 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62a7fc700 1 -- 172.21.2.117:0/2372068312 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c00c440 con 0x7fe62c07da20 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe62a7fc700 1 -- 172.21.2.117:0/2372068312 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe61c00c960 con 0x7fe62c07da20 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe631e2d700 1 -- 172.21.2.117:0/2372068312 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c07da20 msgr2=0x7fe62c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.335 7fe631e2d700 1 --2- 172.21.2.117:0/2372068312 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c07da20 0x7fe62c0771b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fe61c006700 tx=0x7fe61c009b80).stop 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 -- 172.21.2.117:0/2372068312 shutdown_connections 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 --2- 172.21.2.117:0/2372068312 >> v2:172.21.2.125:3300/0 conn(0x7fe62c0e4da0 0x7fe62c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 --2- 172.21.2.117:0/2372068312 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c0e51a0 0x7fe62c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:26:57.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 --2- 172.21.2.117:0/2372068312 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c07da20 0x7fe62c0771b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 -- 172.21.2.117:0/2372068312 >> 172.21.2.117:0/2372068312 conn(0x7fe62c0ce450 msgr2=0x7fe62c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:57.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 -- 172.21.2.117:0/2372068312 shutdown_connections 2021-02-10T11:26:57.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.336 7fe631e2d700 1 -- 172.21.2.117:0/2372068312 wait complete. 2021-02-10T11:26:57.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 Processor -- start 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 -- start start 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe62c09bbe0 0x7fe62c092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c09bfe0 0x7fe62c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0927d0 con 0x7fe62c093370 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0b08d0 con 0x7fe62c09bbe0 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe631e2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c190400 con 0x7fe62c09bfe0 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe62bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe62bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43612/0 (socket says 172.21.2.117:43612) 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.351 7fe62bfff700 1 -- 172.21.2.117:0/1291357905 learned_addr learned my addr 172.21.2.117:0/1291357905 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62b7fe700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe62c09bbe0 0x7fe62c092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62affd700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c09bfe0 0x7fe62c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 -- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c09bfe0 msgr2=0x7fe62c0afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c09bfe0 0x7fe62c0afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 -- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe62c09bbe0 msgr2=0x7fe62c092d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe62c09bbe0 0x7fe62c092d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 -- 172.21.2.117:0/1291357905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c1908a0 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe62bfff700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fe624009fc0 tx=0x7fe62400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe624003020 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624005850 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c1908a0 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe624005d70 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.352 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c1abb00 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.353 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe624005a30 con 0x7fe62c093370 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.353 7fe613fff700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 0x7fe60c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:26:57.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.353 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe624032c70 con 0x7fe62c093370 2021-02-10T11:26:57.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.354 7fe62b7fe700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 0x7fe60c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:26:57.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.354 7fe62b7fe700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 0x7fe60c01abc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:26:57.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.384 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c103370 con 0x7fe62c093370 2021-02-10T11:26:57.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.387 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe624000f90 con 0x7fe62c093370 2021-02-10T11:26:57.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.518 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802452", "target": ["mgr", ""]}) v1 -- 0x7fe62c1abb00 con 0x7fe60c0186c0 2021-02-10T11:26:57.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.521 7fe613fff700 1 -- 172.21.2.117:0/1291357905 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe61c006430 con 0x7fe60c0186c0 2021-02-10T11:26:57.519 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04 2021-02-10T11:26:57.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 msgr2=0x7fe60c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 0x7fe60c01abc0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 msgr2=0x7fe62c0b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:26:57.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fe624009fc0 tx=0x7fe62400bc40).stop 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 shutdown_connections 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe60c0186c0 0x7fe60c01abc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe62c093370 0x7fe62c0b02b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe62c09bfe0 0x7fe62c0afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 --2- 172.21.2.117:0/1291357905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe62c09bbe0 0x7fe62c092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 >> 172.21.2.117:0/1291357905 conn(0x7fe62c0a1420 msgr2=0x7fe62c09efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:26:57.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 shutdown_connections 2021-02-10T11:26:57.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:26:57.527 7fe631e2d700 1 -- 172.21.2.117:0/1291357905 wait complete. 2021-02-10T11:26:57.529 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802452 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04 2021-02-10T11:26:57.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:57.530 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:57.530 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:57.864 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:57.864 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:57.865 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:58.243 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:58.243 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:58.243 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235112 s, 44.6 MB/s 2021-02-10T11:26:58.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:58.244 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:58.244 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:58.579 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:58.579 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:58.580 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:58.956 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:58.956 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:58.956 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242932 s, 43.2 MB/s 2021-02-10T11:26:58.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:58.957 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:58.957 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:26:59.292 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:26:59.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:26:59.293 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:26:59.668 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:26:59.669 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:26:59.669 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256808 s, 40.8 MB/s 2021-02-10T11:26:59.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:26:59.670 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:26:59.670 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:00.006 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:00.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:00.006 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:00.382 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247806 s, 42.3 MB/s 2021-02-10T11:27:00.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:00.383 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:00.383 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:00.720 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:00.720 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:00.720 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:01.094 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:01.094 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:01.095 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248019 s, 42.3 MB/s 2021-02-10T11:27:01.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:01.096 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:01.096 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:01.429 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:01.430 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:01.430 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:01.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:01.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:01.805 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244589 s, 42.9 MB/s 2021-02-10T11:27:01.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:01.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:01.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:02.142 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:02.142 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:02.142 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:02.512 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:02.512 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:02.512 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233404 s, 44.9 MB/s 2021-02-10T11:27:02.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:02.513 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:02.513 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:02.849 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:02.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:02.849 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:03.223 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024466 s, 42.9 MB/s 2021-02-10T11:27:03.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:03.225 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:03.225 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:03.560 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:03.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:03.560 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:03.932 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:03.932 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:03.933 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232375 s, 45.1 MB/s 2021-02-10T11:27:03.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:03.934 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:03.934 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:04.269 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:04.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:04.269 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802452/874bdecd-c161-4a7f-8424-8d525ab84c04/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:04.642 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:04.642 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:04.642 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231546 s, 45.3 MB/s 2021-02-10T11:27:04.643 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802453 2021-02-10T11:27:05.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 Processor -- start 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 -- start start 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff184079000 0x7ff1840e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840d1d20 con 0x7ff18407da20 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184077a70 con 0x7ff1841b00f0 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.062 7ff18c74d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840e5430 con 0x7ff184079000 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff18a569700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff18407da20 0x7ff1840771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43870/0 (socket says 172.21.2.117:43870) 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff18a569700 1 -- 172.21.2.117:0/373664822 learned_addr learned my addr 172.21.2.117:0/373664822 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970772248 0 0) 0x7ff178001180 con 0x7ff18407da20 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170002fe0 con 0x7ff18407da20 2021-02-10T11:27:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189d68700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff184079000 0x7ff1840e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3935699489 0 0) 0x7ff178001180 con 0x7ff18407da20 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff184079000 msgr2=0x7ff1840e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff184079000 0x7ff1840e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 >> v1:172.21.2.125:6789/0 conn(0x7ff1841b00f0 legacy=0x7ff1840d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff184079400 con 0x7ff18407da20 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff178001680 con 0x7ff18407da20 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 -- 172.21.2.117:0/373664822 >> v1:172.21.2.117:6789/0 conn(0x7ff18407da20 legacy=0x7ff1840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.063 7ff189567700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff170003a70 0x7ff170005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff170006780 0x7ff170008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff170009520 0x7ff17000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000c2d0 con 0x7ff170006780 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000c4f0 con 0x7ff170009520 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000c710 con 0x7ff170003a70 2021-02-10T11:27:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff170006780 0x7ff170008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff18a569700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff170003a70 0x7ff170005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff18ad6a700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff170009520 0x7ff17000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 -- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff170003a70 msgr2=0x7ff170005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff170003a70 0x7ff170005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 -- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff170009520 msgr2=0x7ff17000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff170009520 0x7ff17000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 -- 172.21.2.117:0/373664822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170050980 con 0x7ff170006780 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189d68700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff170006780 0x7ff170008c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff180006ac0 tx=0x7ff180009ab0).ready entity=mon.0 client_cookie=e3c1ad197193d1a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18000c0c0 con 0x7ff170006780 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000c380 con 0x7ff170006780 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff189567700 1 -- 172.21.2.117:0/373664822 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff18000c8a0 con 0x7ff170006780 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff18c74d700 1 -- 172.21.2.117:0/373664822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff170006780 msgr2=0x7ff170008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff18c74d700 1 --2- 172.21.2.117:0/373664822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff170006780 0x7ff170008c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7ff180006ac0 tx=0x7ff180009ab0).stop 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.064 7ff18ad6a700 1 -- 172.21.2.117:0/373664822 reap_dead start 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.065 7ff18c74d700 1 -- 172.21.2.117:0/373664822 shutdown_connections 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.065 7ff18c74d700 1 -- 172.21.2.117:0/373664822 >> 172.21.2.117:0/373664822 conn(0x7ff1840ce450 msgr2=0x7ff1840e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.065 7ff18c74d700 1 -- 172.21.2.117:0/373664822 shutdown_connections 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.065 7ff18c74d700 1 -- 172.21.2.117:0/373664822 wait complete. 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.079 7ff18c74d700 1 Processor -- start 2021-02-10T11:27:05.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.079 7ff18c74d700 1 -- start start 2021-02-10T11:27:05.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 0x7ff184190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 0x7ff1840a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1840a50e0 0x7ff1840a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840c3040 con 0x7ff1841a46b0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff184190a20 con 0x7ff1840a50e0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18c74d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840a54e0 con 0x7ff1840a46c0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18a569700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 0x7ff184190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18a569700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 0x7ff184190400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43632/0 (socket says 172.21.2.117:43632) 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18a569700 1 -- 172.21.2.117:0/3525706943 learned_addr learned my addr 172.21.2.117:0/3525706943 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 0x7ff1840a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff18ad6a700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1840a50e0 0x7ff1840a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 -- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1840a50e0 msgr2=0x7ff1840a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1840a50e0 0x7ff1840a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 -- 172.21.2.117:0/3525706943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 msgr2=0x7ff184190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 0x7ff184190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.080 7ff189d68700 1 -- 172.21.2.117:0/3525706943 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840a5700 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff189d68700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 0x7ff1840a4ac0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff18000b4e0 tx=0x7ff18000b880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff180009360 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff180002030 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff180002550 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1840a5700 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.081 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840a5700 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.082 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff180000c90 con 0x7ff1840a46c0 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.082 7ff176ffd700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 0x7ff16c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.082 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff1800325e0 con 0x7ff1840a46c0 2021-02-10T11:27:05.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.082 7ff18a569700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 0x7ff16c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.082 7ff18a569700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 0x7ff16c01abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.112 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184077a70 con 0x7ff1840a46c0 2021-02-10T11:27:05.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.118 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff180009360 con 0x7ff1840a46c0 2021-02-10T11:27:05.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.243 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802453", "target": ["mgr", ""]}) v1 -- 0x7ff1840c3040 con 0x7ff16c0186c0 2021-02-10T11:27:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.270 7ff176ffd700 1 -- 172.21.2.117:0/3525706943 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff178000ce0 con 0x7ff16c0186c0 2021-02-10T11:27:05.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 msgr2=0x7ff16c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 0x7ff16c01abc0 crc :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 msgr2=0x7ff1840a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 0x7ff1840a4ac0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff18000b4e0 tx=0x7ff18000b880).stop 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 shutdown_connections 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1840a50e0 0x7ff1840a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1840a46c0 0x7ff1840a4ac0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff16c0186c0 0x7ff16c01abc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 --2- 172.21.2.117:0/3525706943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1841a46b0 0x7ff184190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 >> 172.21.2.117:0/3525706943 conn(0x7ff1840b5140 msgr2=0x7ff1840a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 shutdown_connections 2021-02-10T11:27:05.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.277 7ff18c74d700 1 -- 172.21.2.117:0/3525706943 wait complete. 2021-02-10T11:27:05.279 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802453 2021-02-10T11:27:05.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 Processor -- start 2021-02-10T11:27:05.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 -- start start 2021-02-10T11:27:05.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b7007da20 0x7f4b700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b700e51a0 0x7f4b70079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b70077a70 con 0x7f4b700e4da0 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700e5c20 con 0x7f4b7007da20 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.699 7f4b76ed1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700777d0 con 0x7f4b700e51a0 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b7007da20 0x7f4b700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4b700e4da0 0x7f4b701b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43888/0 (socket says 172.21.2.117:43888) 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6ffff700 1 -- 172.21.2.117:0/1812754184 learned_addr learned my addr 172.21.2.117:0/1812754184 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6f7fe700 1 -- 172.21.2.117:0/1812754184 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161597672 0 0) 0x7f4b60000ce0 con 0x7f4b700e4da0 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 -- 172.21.2.117:0/1812754184 >> v2:172.21.2.125:3300/0 conn(0x7f4b700e51a0 msgr2=0x7f4b70079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 --2- 172.21.2.117:0/1812754184 >> v2:172.21.2.125:3300/0 conn(0x7f4b700e51a0 0x7f4b70079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 -- 172.21.2.117:0/1812754184 >> v1:172.21.2.117:6789/0 conn(0x7f4b700e4da0 legacy=0x7f4b701b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 -- 172.21.2.117:0/1812754184 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b701b08e0 con 0x7f4b7007da20 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b74ced700 1 --2- 172.21.2.117:0/1812754184 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007da20 0x7f4b700771b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4b58006650 tx=0x7f4b58009a00).ready entity=mon.0 client_cookie=7636581afacd0a4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6f7fe700 1 -- 172.21.2.117:0/1812754184 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b58009df0 con 0x7f4b7007da20 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6f7fe700 1 -- 172.21.2.117:0/1812754184 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b580009b0 con 0x7f4b7007da20 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b6f7fe700 1 -- 172.21.2.117:0/1812754184 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b5800c2b0 con 0x7f4b7007da20 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b76ed1700 1 -- 172.21.2.117:0/1812754184 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007da20 msgr2=0x7f4b700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.700 7f4b76ed1700 1 --2- 172.21.2.117:0/1812754184 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007da20 0x7f4b700771b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f4b58006650 tx=0x7f4b58009a00).stop 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 -- 172.21.2.117:0/1812754184 shutdown_connections 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 --2- 172.21.2.117:0/1812754184 >> v2:172.21.2.125:3300/0 conn(0x7f4b700e51a0 0x7f4b70079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 --2- 172.21.2.117:0/1812754184 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007da20 0x7f4b700771b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 -- 172.21.2.117:0/1812754184 >> 172.21.2.117:0/1812754184 conn(0x7f4b700ce450 msgr2=0x7f4b700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 -- 172.21.2.117:0/1812754184 shutdown_connections 2021-02-10T11:27:05.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.701 7f4b76ed1700 1 -- 172.21.2.117:0/1812754184 wait complete. 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.715 7f4b76ed1700 1 Processor -- start 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.715 7f4b76ed1700 1 -- start start 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b7008e590 0x7f4b7008e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7008efb0 0x7f4b700a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70094460 con 0x7f4b701a3200 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7008f3b0 con 0x7f4b7008e590 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b76ed1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700a6280 con 0x7f4b7008efb0 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43646/0 (socket says 172.21.2.117:43646) 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 -- 172.21.2.117:0/3662879806 learned_addr learned my addr 172.21.2.117:0/3662879806 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 -- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7008efb0 msgr2=0x7f4b700a5c60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7008efb0 0x7f4b700a5c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 -- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b7008e590 msgr2=0x7f4b7008e990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b7008e590 0x7f4b7008e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.716 7f4b74ced700 1 -- 172.21.2.117:0/3662879806 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700a64a0 con 0x7f4b701a3200 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b74ced700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b5800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b580093a0 con 0x7f4b701a3200 2021-02-10T11:27:05.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b580067f0 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b58002220 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b700a64a0 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b700a6a30 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.717 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b580052e0 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.718 7f4b6d7fa700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.718 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b58032750 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.718 7f4b6ffff700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.719 7f4b6ffff700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.749 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b70103350 con 0x7f4b701a3200 2021-02-10T11:27:05.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.752 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b580093a0 con 0x7f4b701a3200 2021-02-10T11:27:05.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.878 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802453", "target": ["mgr", ""]}) v1 -- 0x7f4b7008f5d0 con 0x7f4b54018730 2021-02-10T11:27:05.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.882 7f4b6d7fa700 1 -- 172.21.2.117:0/3662879806 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4b60000ce0 con 0x7f4b54018730 2021-02-10T11:27:05.880 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682 2021-02-10T11:27:05.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.887 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 msgr2=0x7f4b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.887 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.887 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 msgr2=0x7f4b701a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.887 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b5800b9b0).stop 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 shutdown_connections 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7008efb0 0x7f4b700a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b7008e590 0x7f4b7008e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 --2- 172.21.2.117:0/3662879806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701a3200 0x7f4b701a3600 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 >> 172.21.2.117:0/3662879806 conn(0x7f4b700a19a0 msgr2=0x7f4b700b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 shutdown_connections 2021-02-10T11:27:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:05.888 7f4b76ed1700 1 -- 172.21.2.117:0/3662879806 wait complete. 2021-02-10T11:27:05.890 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802453 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682 2021-02-10T11:27:05.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:05.890 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:05.890 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:06.225 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:06.225 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:06.225 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:06.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:06.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:06.606 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258836 s, 40.5 MB/s 2021-02-10T11:27:06.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:06.608 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:06.608 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:06.941 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:06.942 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:06.942 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:07.317 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:07.317 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:07.317 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244104 s, 43.0 MB/s 2021-02-10T11:27:07.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:07.318 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:07.318 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:07.653 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:07.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:07.653 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:08.026 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:08.026 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:08.026 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259084 s, 40.5 MB/s 2021-02-10T11:27:08.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:08.027 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:08.027 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:08.362 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:08.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:08.362 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:08.739 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:08.739 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:08.740 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253017 s, 41.4 MB/s 2021-02-10T11:27:08.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:08.741 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:08.741 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:09.076 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:09.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:09.076 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:09.450 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:09.451 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:09.451 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240535 s, 43.6 MB/s 2021-02-10T11:27:09.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:09.452 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:09.452 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:09.786 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:09.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:09.786 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:10.161 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:10.162 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:10.162 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242155 s, 43.3 MB/s 2021-02-10T11:27:10.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:10.163 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:10.163 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:10.498 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:10.498 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:10.498 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:10.871 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023261 s, 45.1 MB/s 2021-02-10T11:27:10.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:10.873 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:10.873 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:11.209 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:11.209 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:11.209 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:11.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:11.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:11.583 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234659 s, 44.7 MB/s 2021-02-10T11:27:11.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:11.585 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:11.585 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:11.920 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:11.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:11.920 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249686 s, 42.0 MB/s 2021-02-10T11:27:12.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:12.297 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:12.297 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:12.632 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:12.632 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:12.632 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802453/ae05e664-1249-410c-8173-be00fc047682/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:13.005 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253052 s, 41.4 MB/s 2021-02-10T11:27:13.006 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802454 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.425 7f18459c7700 1 Processor -- start 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.425 7f18459c7700 1 -- start start 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f18459c7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1840079000 0x7f18400e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f18459c7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f18401b00f0 0x7f18401b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f18459c7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1840077a70 con 0x7f184007da20 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f18459c7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18400e53c0 con 0x7f18401b00f0 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f18459c7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1840079400 con 0x7f1840079000 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183f574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f184007da20 0x7f18400771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43900/0 (socket says 172.21.2.117:43900) 2021-02-10T11:27:13.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183f574700 1 -- 172.21.2.117:0/3281421099 learned_addr learned my addr 172.21.2.117:0/3281421099 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1840079000 0x7f18400e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183fd75700 1 --2- 172.21.2.117:0/3281421099 >> v2:172.21.2.125:3300/0 conn(0x7f18401b00f0 0x7f18401b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091386211 0 0) 0x7f18300010b0 con 0x7f184007da20 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.426 7f183e572700 1 -- 172.21.2.117:0/3281421099 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828002fe0 con 0x7f184007da20 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2971824966 0 0) 0x7f18300010b0 con 0x7f184007da20 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1840079000 msgr2=0x7f18400e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1840079000 0x7f18400e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 >> v2:172.21.2.125:3300/0 conn(0x7f18401b00f0 msgr2=0x7f18401b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 --2- 172.21.2.117:0/3281421099 >> v2:172.21.2.125:3300/0 conn(0x7f18401b00f0 0x7f18401b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184007de20 con 0x7f184007da20 2021-02-10T11:27:13.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f18300015b0 con 0x7f184007da20 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 >> v1:172.21.2.117:6789/0 conn(0x7f184007da20 legacy=0x7f18400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828003b90 0x7f1828006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f182804a8f0 0x7f182804cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f182804d6f0 0x7f182804fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1828050430 con 0x7f182804a8f0 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1828050650 con 0x7f1828003b90 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183e572700 1 -- 172.21.2.117:0/3281421099 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1828050870 con 0x7f182804d6f0 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183fd75700 1 --2- 172.21.2.117:0/3281421099 >> v2:172.21.2.125:3300/0 conn(0x7f18401b00f0 0x7f18401b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1840079000 0x7f18400e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183f574700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828003b90 0x7f1828006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f182804a8f0 0x7f182804cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183fd75700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f182804d6f0 0x7f182804fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183ed73700 1 -- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f182804d6f0 msgr2=0x7f182804fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.427 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f182804d6f0 0x7f182804fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183ed73700 1 -- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828003b90 msgr2=0x7f1828006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828003b90 0x7f1828006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183ed73700 1 -- 172.21.2.117:0/3281421099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828050b10 con 0x7f182804a8f0 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183fd75700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f182804d6f0 0x7f182804fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183ed73700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f182804a8f0 0x7f182804cdf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f18340063a0 tx=0x7f1834009aa0).ready entity=mon.0 client_cookie=fbfc6709ffb3bb80 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f183400c340 con 0x7f182804a8f0 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183400c6c0 con 0x7f182804a8f0 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183e572700 1 -- 172.21.2.117:0/3281421099 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f183400cbe0 con 0x7f182804a8f0 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f18459c7700 1 -- 172.21.2.117:0/3281421099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f182804a8f0 msgr2=0x7f182804cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f18459c7700 1 --2- 172.21.2.117:0/3281421099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f182804a8f0 0x7f182804cdf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f18340063a0 tx=0x7f1834009aa0).stop 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f183fd75700 1 -- 172.21.2.117:0/3281421099 reap_dead start 2021-02-10T11:27:13.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f18459c7700 1 -- 172.21.2.117:0/3281421099 shutdown_connections 2021-02-10T11:27:13.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f18459c7700 1 -- 172.21.2.117:0/3281421099 >> 172.21.2.117:0/3281421099 conn(0x7f18400ce450 msgr2=0x7f18400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:13.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.428 7f18459c7700 1 -- 172.21.2.117:0/3281421099 shutdown_connections 2021-02-10T11:27:13.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.429 7f18459c7700 1 -- 172.21.2.117:0/3281421099 wait complete. 2021-02-10T11:27:13.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.443 7f18459c7700 1 Processor -- start 2021-02-10T11:27:13.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.443 7f18459c7700 1 -- start start 2021-02-10T11:27:13.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18400a47d0 0x7f18400a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18400a51f0 0x7f18400a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1840189970 con 0x7f18400b77d0 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1840190aa0 con 0x7f18400a47d0 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f18459c7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18400b81f0 con 0x7f18400a51f0 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43666/0 (socket says 172.21.2.117:43666) 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 -- 172.21.2.117:0/2084913671 learned_addr learned my addr 172.21.2.117:0/2084913671 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 -- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18400a51f0 msgr2=0x7f18400a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18400a51f0 0x7f18400a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 -- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18400a47d0 msgr2=0x7f18400a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18400a47d0 0x7f18400a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.444 7f183fd75700 1 -- 172.21.2.117:0/2084913671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400b8410 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f183fd75700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f18380099c0 tx=0x7f183800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1838006ae0 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838003030 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f18459c7700 1 -- 172.21.2.117:0/2084913671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18400b8410 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f18459c7700 1 -- 172.21.2.117:0/2084913671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18400b89a0 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18380015f0 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.445 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f18380056b0 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.446 7f1827fff700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 0x7f182001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.446 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1838032c90 con 0x7f18400b77d0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.446 7f183f574700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 0x7f182001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:13.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.446 7f183f574700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 0x7f182001abc0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:13.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.476 7f18459c7700 1 -- 172.21.2.117:0/2084913671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1840077a70 con 0x7f18400b77d0 2021-02-10T11:27:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.478 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1838003010 con 0x7f18400b77d0 2021-02-10T11:27:13.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.603 7f18459c7700 1 -- 172.21.2.117:0/2084913671 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802454", "target": ["mgr", ""]}) v1 -- 0x7f1840189970 con 0x7f18200186c0 2021-02-10T11:27:13.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.629 7f1827fff700 1 -- 172.21.2.117:0/2084913671 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f18300009b0 con 0x7f18200186c0 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 -- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 msgr2=0x7f182001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 0x7f182001abc0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 -- 172.21.2.117:0/2084913671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 msgr2=0x7f18400b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f18380099c0 tx=0x7f183800bd40).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 -- 172.21.2.117:0/2084913671 shutdown_connections 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18200186c0 0x7f182001abc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18400b77d0 0x7f18400b7bd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18400a51f0 0x7f18400a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 --2- 172.21.2.117:0/2084913671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18400a47d0 0x7f18400a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.636 7f18459c7700 1 -- 172.21.2.117:0/2084913671 >> 172.21.2.117:0/2084913671 conn(0x7f18400b5140 msgr2=0x7f18400a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:13.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.637 7f18459c7700 1 -- 172.21.2.117:0/2084913671 shutdown_connections 2021-02-10T11:27:13.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:13.637 7f18459c7700 1 -- 172.21.2.117:0/2084913671 wait complete. 2021-02-10T11:27:13.639 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802454 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.059 7f2000432700 1 Processor -- start 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.059 7f2000432700 1 -- start start 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f2000432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 0x7f1ff80e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f2000432700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff80d1d00 con 0x7f1ff807da20 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f2000432700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff8077a70 con 0x7f1ff81b00d0 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f2000432700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80e5410 con 0x7f1ff8079000 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f1ffe24e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1ff807da20 0x7f1ff80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:43920/0 (socket says 172.21.2.117:43920) 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.060 7f1ffe24e700 1 -- 172.21.2.117:0/3540977563 learned_addr learned my addr 172.21.2.117:0/3540977563 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffda4d700 1 --2- 172.21.2.117:0/3540977563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 0x7f1ff80e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffd24c700 1 -- 172.21.2.117:0/3540977563 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701355056 0 0) 0x7f1fec0010b0 con 0x7f1ff807da20 2021-02-10T11:27:14.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffd24c700 1 -- 172.21.2.117:0/3540977563 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe4002fe0 con 0x7f1ff807da20 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffda4d700 1 -- 172.21.2.117:0/3540977563 >> v1:172.21.2.125:6789/0 conn(0x7f1ff81b00d0 legacy=0x7f1ff80d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffda4d700 1 -- 172.21.2.117:0/3540977563 >> v1:172.21.2.117:6789/0 conn(0x7f1ff807da20 legacy=0x7f1ff80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffda4d700 1 -- 172.21.2.117:0/3540977563 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8079400 con 0x7f1ff8079000 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.061 7f1ffda4d700 1 --2- 172.21.2.117:0/3540977563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 0x7f1ff80e4d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1ff4006650 tx=0x7f1ff4009a80).ready entity=mon.2 client_cookie=7a21c59c95e3fa42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f1ffd24c700 1 -- 172.21.2.117:0/3540977563 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff400c0c0 con 0x7f1ff8079000 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f1ffd24c700 1 -- 172.21.2.117:0/3540977563 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff400c340 con 0x7f1ff8079000 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f1ffd24c700 1 -- 172.21.2.117:0/3540977563 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff400c860 con 0x7f1ff8079000 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 -- 172.21.2.117:0/3540977563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 msgr2=0x7f1ff80e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 --2- 172.21.2.117:0/3540977563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 0x7f1ff80e4d80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1ff4006650 tx=0x7f1ff4009a80).stop 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 -- 172.21.2.117:0/3540977563 shutdown_connections 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 --2- 172.21.2.117:0/3540977563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff8079000 0x7f1ff80e4d80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 -- 172.21.2.117:0/3540977563 >> 172.21.2.117:0/3540977563 conn(0x7f1ff80ce430 msgr2=0x7f1ff80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 -- 172.21.2.117:0/3540977563 shutdown_connections 2021-02-10T11:27:14.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.063 7f2000432700 1 -- 172.21.2.117:0/3540977563 wait complete. 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 Processor -- start 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 -- start start 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff80a46c0 0x7f1ff80b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ff819fd10 0x7f1ff80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8189be0 con 0x7f1ff81a0110 2021-02-10T11:27:14.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80a4be0 con 0x7f1ff819fd10 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.077 7f2000432700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80b44e0 con 0x7f1ff80a46c0 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43678/0 (socket says 172.21.2.117:43678) 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 -- 172.21.2.117:0/2285652802 learned_addr learned my addr 172.21.2.117:0/2285652802 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 -- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff80a46c0 msgr2=0x7f1ff80b3ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff80a46c0 0x7f1ff80b3ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 -- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ff819fd10 msgr2=0x7f1ff80a5160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ff819fd10 0x7f1ff80a5160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 -- 172.21.2.117:0/2285652802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80b4830 con 0x7f1ff81a0110 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1ffea4f700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f1ff0009940 tx=0x7f1ff000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff0000ce0 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff000c2f0 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff000c7d0 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f2000432700 1 -- 172.21.2.117:0/2285652802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff80b4830 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.078 7f2000432700 1 -- 172.21.2.117:0/2285652802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff80b4830 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.079 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ff0001220 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.079 7f1feaffd700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 0x7f1fe001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.079 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1ff0032ca0 con 0x7f1ff81a0110 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.079 7f1ffe24e700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 0x7f1fe001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:14.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.080 7f1ffe24e700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 0x7f1fe001abc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.110 7f2000432700 1 -- 172.21.2.117:0/2285652802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8189be0 con 0x7f1ff81a0110 2021-02-10T11:27:14.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.112 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ff000be30 con 0x7f1ff81a0110 2021-02-10T11:27:14.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.240 7f2000432700 1 -- 172.21.2.117:0/2285652802 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802454", "target": ["mgr", ""]}) v1 -- 0x7f1ff80a4be0 con 0x7f1fe00186c0 2021-02-10T11:27:14.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.244 7f1feaffd700 1 -- 172.21.2.117:0/2285652802 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1fec001090 con 0x7f1fe00186c0 2021-02-10T11:27:14.241 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.249 7f2000432700 1 -- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 msgr2=0x7f1fe001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.249 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 0x7f1fe001abc0 crc :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.249 7f2000432700 1 -- 172.21.2.117:0/2285652802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 msgr2=0x7f1ff80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.249 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f1ff0009940 tx=0x7f1ff000bcf0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 -- 172.21.2.117:0/2285652802 shutdown_connections 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe00186c0 0x7f1fe001abc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ff81a0110 0x7f1ff80a5780 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ff819fd10 0x7f1ff80a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 --2- 172.21.2.117:0/2285652802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ff80a46c0 0x7f1ff80b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 -- 172.21.2.117:0/2285652802 >> 172.21.2.117:0/2285652802 conn(0x7f1ff80b5140 msgr2=0x7f1ff80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:14.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 -- 172.21.2.117:0/2285652802 shutdown_connections 2021-02-10T11:27:14.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:14.250 7f2000432700 1 -- 172.21.2.117:0/2285652802 wait complete. 2021-02-10T11:27:14.251 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802454 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1 2021-02-10T11:27:14.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:14.252 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:14.252 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:14.588 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:14.588 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:14.588 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:14.969 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:14.969 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:14.969 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238532 s, 44.0 MB/s 2021-02-10T11:27:14.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:14.970 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:14.970 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:15.305 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:15.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:15.305 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244688 s, 42.9 MB/s 2021-02-10T11:27:15.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:15.682 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:15.682 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:16.018 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:16.018 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:16.018 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:16.408 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:16.408 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:16.408 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249603 s, 42.0 MB/s 2021-02-10T11:27:16.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:16.409 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:16.410 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:16.750 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:16.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:16.751 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235504 s, 44.5 MB/s 2021-02-10T11:27:17.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:17.125 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:17.125 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:17.460 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:17.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:17.461 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:17.834 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:17.834 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:17.834 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025485 s, 41.1 MB/s 2021-02-10T11:27:17.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:17.835 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:17.835 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:18.170 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:18.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:18.171 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:18.545 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:18.545 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:18.545 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238188 s, 44.0 MB/s 2021-02-10T11:27:18.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:18.546 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:18.546 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:18.880 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:18.881 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:18.881 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:19.255 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:19.255 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:19.255 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233562 s, 44.9 MB/s 2021-02-10T11:27:19.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:19.256 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:19.256 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:19.592 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:19.592 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:19.592 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242521 s, 43.2 MB/s 2021-02-10T11:27:19.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:19.969 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:19.969 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:20.304 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:20.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:20.305 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232789 s, 45.0 MB/s 2021-02-10T11:27:20.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:20.680 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:20.680 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:21.014 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:21.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:21.015 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802454/aa483ccc-1ad2-4835-af06-7f1b2b137ac1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:21.390 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:21.390 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:21.390 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245756 s, 42.7 MB/s 2021-02-10T11:27:21.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802455 2021-02-10T11:27:21.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 Processor -- start 2021-02-10T11:27:21.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 -- start start 2021-02-10T11:27:21.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e58079000 0x7f9e580e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e581b00f0 0x7f9e581b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e58077a70 con 0x7f9e5807da20 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e580e53c0 con 0x7f9e58079000 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e5dc0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58079400 con 0x7f9e581b00f0 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e56ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e58079000 0x7f9e580e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e56ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e58079000 0x7f9e580e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43684/0 (socket says 172.21.2.117:43684) 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e56ffd700 1 -- 172.21.2.117:0/4142315740 learned_addr learned my addr 172.21.2.117:0/4142315740 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.811 7f9e57fff700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e581b00f0 0x7f9e581b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e56ffd700 1 -- 172.21.2.117:0/4142315740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e581b00f0 msgr2=0x7f9e581b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e56ffd700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e581b00f0 0x7f9e581b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e56ffd700 1 -- 172.21.2.117:0/4142315740 >> v1:172.21.2.117:6789/0 conn(0x7f9e5807da20 legacy=0x7f9e580771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e56ffd700 1 -- 172.21.2.117:0/4142315740 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5807de20 con 0x7f9e58079000 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e56ffd700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e58079000 0x7f9e580e4da0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9e48006700 tx=0x7f9e48009b80).ready entity=mon.0 client_cookie=1383f659ab35580a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:21.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e567fc700 1 -- 172.21.2.117:0/4142315740 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800c0c0 con 0x7f9e58079000 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e567fc700 1 -- 172.21.2.117:0/4142315740 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4800c340 con 0x7f9e58079000 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e567fc700 1 -- 172.21.2.117:0/4142315740 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800c7a0 con 0x7f9e58079000 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e5dc0f700 1 -- 172.21.2.117:0/4142315740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e58079000 msgr2=0x7f9e580e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.812 7f9e5dc0f700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e58079000 0x7f9e580e4da0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9e48006700 tx=0x7f9e48009b80).stop 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 -- 172.21.2.117:0/4142315740 shutdown_connections 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e581b00f0 0x7f9e581b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 --2- 172.21.2.117:0/4142315740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e58079000 0x7f9e580e4da0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 -- 172.21.2.117:0/4142315740 >> 172.21.2.117:0/4142315740 conn(0x7f9e580ce450 msgr2=0x7f9e580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 -- 172.21.2.117:0/4142315740 shutdown_connections 2021-02-10T11:27:21.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.813 7f9e5dc0f700 1 -- 172.21.2.117:0/4142315740 wait complete. 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 Processor -- start 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 -- start start 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e58105870 0x7f9e5819d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e5819dbc0 0x7f9e581b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58107920 con 0x7f9e5819d7c0 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5819dfc0 con 0x7f9e58105870 2021-02-10T11:27:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.828 7f9e5dc0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58105e20 con 0x7f9e5819dbc0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43692/0 (socket says 172.21.2.117:43692) 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 -- 172.21.2.117:0/2719943385 learned_addr learned my addr 172.21.2.117:0/2719943385 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e577fe700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e58105870 0x7f9e5819d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e57fff700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e5819dbc0 0x7f9e581b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 -- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e5819dbc0 msgr2=0x7f9e581b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e5819dbc0 0x7f9e581b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 -- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e58105870 msgr2=0x7f9e5819d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e58105870 0x7f9e5819d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 -- 172.21.2.117:0/2719943385 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e580b3fc0 con 0x7f9e5819d7c0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.829 7f9e56ffd700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f9e4800b5e0 tx=0x7f9e4800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e4800bd70 con 0x7f9e5819d7c0 2021-02-10T11:27:21.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e580b3fc0 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4800bd70 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e581b6a50 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e48002030 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.830 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e4800a000 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.831 7f9e4ffff700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 0x7f9e3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.831 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e480326e0 con 0x7f9e5819d7c0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.831 7f9e577fe700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 0x7f9e3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:21.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.831 7f9e577fe700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 0x7f9e3c01abc0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:21.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.861 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e30002fd0 con 0x7f9e5819d7c0 2021-02-10T11:27:21.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.864 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e48030dc0 con 0x7f9e5819d7c0 2021-02-10T11:27:21.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:21.989 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802455", "target": ["mgr", ""]}) v1 -- 0x7f9e30002f80 con 0x7f9e3c0186c0 2021-02-10T11:27:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.016 7f9e4ffff700 1 -- 172.21.2.117:0/2719943385 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9e400055c0 con 0x7f9e3c0186c0 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 msgr2=0x7f9e3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 0x7f9e3c01abc0 crc :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 msgr2=0x7f9e581b5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f9e4800b5e0 tx=0x7f9e4800b980).stop 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 shutdown_connections 2021-02-10T11:27:22.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e3c0186c0 0x7f9e3c01abc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e5819d7c0 0x7f9e581b5e10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e5819dbc0 0x7f9e581b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 --2- 172.21.2.117:0/2719943385 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e58105870 0x7f9e5819d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.022 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 >> 172.21.2.117:0/2719943385 conn(0x7f9e580ffe30 msgr2=0x7f9e580c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.023 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 shutdown_connections 2021-02-10T11:27:22.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.023 7f9e5dc0f700 1 -- 172.21.2.117:0/2719943385 wait complete. 2021-02-10T11:27:22.024 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802455 2021-02-10T11:27:22.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.447 7f9b992d0700 1 Processor -- start 2021-02-10T11:27:22.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.447 7f9b992d0700 1 -- start start 2021-02-10T11:27:22.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b992d0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b94079000 0x7f9b940e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b992d0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 0x7f9b941b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b992d0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b94077a70 con 0x7f9b9407da20 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b992d0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940e5c20 con 0x7f9b94079000 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b992d0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b940e53c0 con 0x7f9b941b00f0 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b94079000 0x7f9b940e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b94079000 0x7f9b940e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43700/0 (socket says 172.21.2.117:43700) 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 -- 172.21.2.117:0/224605716 learned_addr learned my addr 172.21.2.117:0/224605716 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9387e700 1 --2- 172.21.2.117:0/224605716 >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 0x7f9b941b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 -- 172.21.2.117:0/224605716 >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 msgr2=0x7f9b941b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 --2- 172.21.2.117:0/224605716 >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 0x7f9b941b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 -- 172.21.2.117:0/224605716 >> v1:172.21.2.125:6789/0 conn(0x7f9b9407da20 legacy=0x7f9b940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9287c700 1 -- 172.21.2.117:0/224605716 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94079400 con 0x7f9b94079000 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.448 7f9b9387e700 1 --2- 172.21.2.117:0/224605716 >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 0x7f9b941b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b9287c700 1 --2- 172.21.2.117:0/224605716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b94079000 0x7f9b940e4da0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f9b84006a80 tx=0x7f9b84009f00).ready entity=mon.0 client_cookie=52a9b10d3aa871cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:22.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b9207b700 1 -- 172.21.2.117:0/224605716 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b8400b5c0 con 0x7f9b94079000 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b9207b700 1 -- 172.21.2.117:0/224605716 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400b8c0 con 0x7f9b94079000 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b9207b700 1 -- 172.21.2.117:0/224605716 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b8400bde0 con 0x7f9b94079000 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b992d0700 1 -- 172.21.2.117:0/224605716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b94079000 msgr2=0x7f9b940e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.449 7f9b992d0700 1 --2- 172.21.2.117:0/224605716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b94079000 0x7f9b940e4da0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f9b84006a80 tx=0x7f9b84009f00).stop 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 -- 172.21.2.117:0/224605716 shutdown_connections 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 --2- 172.21.2.117:0/224605716 >> v2:172.21.2.125:3300/0 conn(0x7f9b941b00f0 0x7f9b941b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 --2- 172.21.2.117:0/224605716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b94079000 0x7f9b940e4da0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 -- 172.21.2.117:0/224605716 >> 172.21.2.117:0/224605716 conn(0x7f9b940ce450 msgr2=0x7f9b940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 -- 172.21.2.117:0/224605716 shutdown_connections 2021-02-10T11:27:22.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.450 7f9b992d0700 1 -- 172.21.2.117:0/224605716 wait complete. 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 Processor -- start 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 -- start start 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b94187e20 0x7f9b94188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b94090420 0x7f9b94090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b941067f0 con 0x7f9b941a0250 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94107350 con 0x7f9b94187e20 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b992d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b941042d0 con 0x7f9b94090420 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b9307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:22.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b9307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43704/0 (socket says 172.21.2.117:43704) 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.465 7f9b9307d700 1 -- 172.21.2.117:0/3922489735 learned_addr learned my addr 172.21.2.117:0/3922489735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 -- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b94090420 msgr2=0x7f9b94090820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b94090420 0x7f9b94090820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 -- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b94187e20 msgr2=0x7f9b94188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b94187e20 0x7f9b94188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 -- 172.21.2.117:0/3922489735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94090e40 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b9307d700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f9b8c00b5b0 tx=0x7f9b8c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b8c00bd40 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c006240 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b8c005720 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b94090e40 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b940913d0 con 0x7f9b941a0250 2021-02-10T11:27:22.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b8c0054a0 con 0x7f9b941a0250 2021-02-10T11:27:22.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 0x7f9b7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:22.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.466 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9b8c032740 con 0x7f9b941a0250 2021-02-10T11:27:22.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.467 7f9b9287c700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 0x7f9b7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:22.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.467 7f9b9287c700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 0x7f9b7401abc0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:22.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.497 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b94103280 con 0x7f9b941a0250 2021-02-10T11:27:22.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.500 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b8c025410 con 0x7f9b941a0250 2021-02-10T11:27:22.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.629 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802455", "target": ["mgr", ""]}) v1 -- 0x7f9b94091c60 con 0x7f9b740186c0 2021-02-10T11:27:22.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.634 7f9b7bfff700 1 -- 172.21.2.117:0/3922489735 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9b84006710 con 0x7f9b740186c0 2021-02-10T11:27:22.630 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b 2021-02-10T11:27:22.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 msgr2=0x7f9b7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 0x7f9b7401abc0 crc :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 msgr2=0x7f9b94106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f9b8c00b5b0 tx=0x7f9b8c00b950).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 shutdown_connections 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b94090420 0x7f9b94090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b94187e20 0x7f9b94188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b740186c0 0x7f9b7401abc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 --2- 172.21.2.117:0/3922489735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b941a0250 0x7f9b94106cf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:22.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 >> 172.21.2.117:0/3922489735 conn(0x7f9b940a1850 msgr2=0x7f9b940c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:22.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 shutdown_connections 2021-02-10T11:27:22.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:22.638 7f9b992d0700 1 -- 172.21.2.117:0/3922489735 wait complete. 2021-02-10T11:27:22.640 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802455 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b 2021-02-10T11:27:22.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:22.640 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:22.640 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:22.975 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:22.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:22.976 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:23.350 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:23.350 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:23.350 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0222021 s, 47.2 MB/s 2021-02-10T11:27:23.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:23.351 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:23.351 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:23.686 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:23.687 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:23.687 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023115 s, 45.4 MB/s 2021-02-10T11:27:24.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:24.061 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:24.061 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:24.396 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:24.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:24.397 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:24.772 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:24.772 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:24.772 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228313 s, 45.9 MB/s 2021-02-10T11:27:24.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:24.773 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:24.773 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:25.108 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:25.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:25.109 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:25.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:25.483 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:25.483 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226681 s, 46.3 MB/s 2021-02-10T11:27:25.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:25.484 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:25.484 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:25.819 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:25.819 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:25.819 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:25.820 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:25.823 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:26.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:26.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:26.199 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261252 s, 40.1 MB/s 2021-02-10T11:27:26.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:26.200 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:26.200 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:26.573 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:26.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:26.574 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223113 s, 47.0 MB/s 2021-02-10T11:27:26.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:26.986 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:26.986 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:27.323 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:27.324 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:27.324 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:27.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:27.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:27.696 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254091 s, 41.3 MB/s 2021-02-10T11:27:27.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:27.697 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:27.697 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:28.032 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:28.033 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:28.033 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:28.407 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:28.407 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:28.407 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023004 s, 45.6 MB/s 2021-02-10T11:27:28.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:28.408 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:28.408 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:28.743 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:28.744 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:28.744 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:29.117 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:29.117 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:29.117 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237256 s, 44.2 MB/s 2021-02-10T11:27:29.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:29.118 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:29.119 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:29.454 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:29.454 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:29.454 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802455/9ace6d64-a24a-42cb-9a5c-c2131eb1cf4b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:29.830 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:29.830 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:29.830 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236413 s, 44.4 MB/s 2021-02-10T11:27:29.831 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802456 2021-02-10T11:27:30.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.251 7fc72e7a8700 1 Processor -- start 2021-02-10T11:27:30.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.251 7fc72e7a8700 1 -- start start 2021-02-10T11:27:30.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc72e7a8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc72807da20 0x7fc7280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc72e7a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc72e7a8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc728077a70 con 0x7fc7280e51a0 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc72e7a8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7280777d0 con 0x7fc72807da20 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc72e7a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72807de20 con 0x7fc7280e4da0 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.252 7fc7277fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:41396/0 (socket says 172.21.2.117:41396) 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 -- 172.21.2.117:0/2152919568 learned_addr learned my addr 172.21.2.117:0/2152919568 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc727fff700 1 --2- 172.21.2.117:0/2152919568 >> v2:172.21.2.125:3300/0 conn(0x7fc72807da20 0x7fc7280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 -- 172.21.2.117:0/2152919568 >> v2:172.21.2.125:3300/0 conn(0x7fc72807da20 msgr2=0x7fc7280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 --2- 172.21.2.117:0/2152919568 >> v2:172.21.2.125:3300/0 conn(0x7fc72807da20 0x7fc7280771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 -- 172.21.2.117:0/2152919568 >> v1:172.21.2.125:6789/0 conn(0x7fc7280e51a0 legacy=0x7fc728077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 -- 172.21.2.117:0/2152919568 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7281b0710 con 0x7fc7280e4da0 2021-02-10T11:27:30.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc7277fe700 1 --2- 172.21.2.117:0/2152919568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc71c006700 tx=0x7fc71c009b80).ready entity=mon.2 client_cookie=94a31faff0609ce2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc726ffd700 1 -- 172.21.2.117:0/2152919568 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc71c00c0c0 con 0x7fc7280e4da0 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc726ffd700 1 -- 172.21.2.117:0/2152919568 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c00c340 con 0x7fc7280e4da0 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.253 7fc726ffd700 1 -- 172.21.2.117:0/2152919568 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc71c00c860 con 0x7fc7280e4da0 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 -- 172.21.2.117:0/2152919568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 msgr2=0x7fc7281b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 --2- 172.21.2.117:0/2152919568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc71c006700 tx=0x7fc71c009b80).stop 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 -- 172.21.2.117:0/2152919568 shutdown_connections 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 --2- 172.21.2.117:0/2152919568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7280e4da0 0x7fc7281b00f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 --2- 172.21.2.117:0/2152919568 >> v2:172.21.2.125:3300/0 conn(0x7fc72807da20 0x7fc7280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 -- 172.21.2.117:0/2152919568 >> 172.21.2.117:0/2152919568 conn(0x7fc7280ce450 msgr2=0x7fc7280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 -- 172.21.2.117:0/2152919568 shutdown_connections 2021-02-10T11:27:30.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.254 7fc72e7a8700 1 -- 172.21.2.117:0/2152919568 wait complete. 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 Processor -- start 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 -- start start 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7280b7fc0 0x7fc7281042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7281048f0 0x7fc728104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281054f0 con 0x7fc728090710 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc728195a30 con 0x7fc7280b7fc0 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72e7a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc728195fb0 con 0x7fc7281048f0 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72cdc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72cdc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43720/0 (socket says 172.21.2.117:43720) 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.269 7fc72cdc5700 1 -- 172.21.2.117:0/1131666349 learned_addr learned my addr 172.21.2.117:0/1131666349 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 -- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7281048f0 msgr2=0x7fc728104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7281048f0 0x7fc728104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 -- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7280b7fc0 msgr2=0x7fc7281042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7280b7fc0 0x7fc7281042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 -- 172.21.2.117:0/1131666349 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280b7b60 con 0x7fc728090710 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72cdc5700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fc720009940 tx=0x7fc72000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc720000f40 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72000c2f0 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc72000c7d0 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7280b7b60 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.270 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc728091420 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.271 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc720001220 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.271 7fc724ff9700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 0x7fc70801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.271 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc7200329f0 con 0x7fc728090710 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.271 7fc727fff700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 0x7fc70801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.272 7fc727fff700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 0x7fc70801abc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.302 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc728103280 con 0x7fc728090710 2021-02-10T11:27:30.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.305 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc720000f40 con 0x7fc728090710 2021-02-10T11:27:30.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.432 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802456", "target": ["mgr", ""]}) v1 -- 0x7fc728091420 con 0x7fc7080186c0 2021-02-10T11:27:30.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.459 7fc724ff9700 1 -- 172.21.2.117:0/1131666349 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc7180009b0 con 0x7fc7080186c0 2021-02-10T11:27:30.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 msgr2=0x7fc70801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 0x7fc70801abc0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 msgr2=0x7fc728090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fc720009940 tx=0x7fc72000bcf0).stop 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 shutdown_connections 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7080186c0 0x7fc70801abc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc728090710 0x7fc728090b10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7281048f0 0x7fc728104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 --2- 172.21.2.117:0/1131666349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7280b7fc0 0x7fc7281042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.465 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 >> 172.21.2.117:0/1131666349 conn(0x7fc7280a1850 msgr2=0x7fc7280c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.466 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 shutdown_connections 2021-02-10T11:27:30.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.466 7fc72e7a8700 1 -- 172.21.2.117:0/1131666349 wait complete. 2021-02-10T11:27:30.467 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802456 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 Processor -- start 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 -- start start 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f19f8079000 0x7f19f80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f19f81b00f0 0x7f19f81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f8077a70 con 0x7f19f807da20 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80e5c20 con 0x7f19f8079000 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.887 7f19fe530700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f80e53c0 con 0x7f19f81b00f0 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f19f8079000 0x7f19f80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f19f8079000 0x7f19f80e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43728/0 (socket says 172.21.2.117:43728) 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 -- 172.21.2.117:0/3076480019 learned_addr learned my addr 172.21.2.117:0/3076480019 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 -- 172.21.2.117:0/3076480019 >> v2:172.21.2.125:3300/0 conn(0x7f19f81b00f0 msgr2=0x7f19f81b04f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 --2- 172.21.2.117:0/3076480019 >> v2:172.21.2.125:3300/0 conn(0x7f19f81b00f0 0x7f19f81b04f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 -- 172.21.2.117:0/3076480019 >> v1:172.21.2.125:6789/0 conn(0x7f19f807da20 legacy=0x7f19f80771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 -- 172.21.2.117:0/3076480019 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8079400 con 0x7f19f8079000 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f77fe700 1 --2- 172.21.2.117:0/3076480019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f8079000 0x7f19f80e4da0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f19ec006700 tx=0x7f19ec009b00).ready entity=mon.0 client_cookie=4622731266d8c7bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f6ffd700 1 -- 172.21.2.117:0/3076480019 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec00c0c0 con 0x7f19f8079000 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f6ffd700 1 -- 172.21.2.117:0/3076480019 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec00c400 con 0x7f19f8079000 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19f6ffd700 1 -- 172.21.2.117:0/3076480019 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec00c920 con 0x7f19f8079000 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19fe530700 1 -- 172.21.2.117:0/3076480019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f8079000 msgr2=0x7f19f80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:30.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.888 7f19fe530700 1 --2- 172.21.2.117:0/3076480019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f8079000 0x7f19f80e4da0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f19ec006700 tx=0x7f19ec009b00).stop 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 -- 172.21.2.117:0/3076480019 shutdown_connections 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 --2- 172.21.2.117:0/3076480019 >> v2:172.21.2.125:3300/0 conn(0x7f19f81b00f0 0x7f19f81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 --2- 172.21.2.117:0/3076480019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f8079000 0x7f19f80e4da0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 -- 172.21.2.117:0/3076480019 >> 172.21.2.117:0/3076480019 conn(0x7f19f80ce450 msgr2=0x7f19f80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 -- 172.21.2.117:0/3076480019 shutdown_connections 2021-02-10T11:27:30.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.889 7f19fe530700 1 -- 172.21.2.117:0/3076480019 wait complete. 2021-02-10T11:27:30.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.903 7f19fe530700 1 Processor -- start 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.903 7f19fe530700 1 -- start start 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19f80b0f40 0x7f19f8107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19f818b830 0x7f19f8187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f80b18c0 con 0x7f19f80b1340 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f818ac90 con 0x7f19f80b0f40 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19fe530700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f8107f50 con 0x7f19f818b830 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43730/0 (socket says 172.21.2.117:43730) 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 -- 172.21.2.117:0/748817894 learned_addr learned my addr 172.21.2.117:0/748817894 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:30.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 -- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19f818b830 msgr2=0x7f19f8187bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19f818b830 0x7f19f8187bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 -- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19f80b0f40 msgr2=0x7f19f8107930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19f80b0f40 0x7f19f8107930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 -- 172.21.2.117:0/748817894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8108130 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.904 7f19f77fe700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f19ec00b660 tx=0x7f19ec00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec009440 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec006390 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19ec002030 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19fe530700 1 -- 172.21.2.117:0/748817894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f8108130 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19fe530700 1 -- 172.21.2.117:0/748817894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f8108130 con 0x7f19f80b1340 2021-02-10T11:27:30.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19ec0093e0 con 0x7f19f80b1340 2021-02-10T11:27:30.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 0x7f19d801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:30.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.905 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f19ec0327b0 con 0x7f19f80b1340 2021-02-10T11:27:30.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.906 7f19f7fff700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 0x7f19d801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:30.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.906 7f19f7fff700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 0x7f19d801abd0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:30.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.937 7f19fe530700 1 -- 172.21.2.117:0/748817894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f8077a70 con 0x7f19f80b1340 2021-02-10T11:27:30.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:30.941 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19ec006370 con 0x7f19f80b1340 2021-02-10T11:27:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.068 7f19fe530700 1 -- 172.21.2.117:0/748817894 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802456", "target": ["mgr", ""]}) v1 -- 0x7f19f8077a70 con 0x7f19d80186d0 2021-02-10T11:27:31.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.073 7f19f4ff9700 1 -- 172.21.2.117:0/748817894 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f19e8000c10 con 0x7f19d80186d0 2021-02-10T11:27:31.069 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df 2021-02-10T11:27:31.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 msgr2=0x7f19d801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:31.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 0x7f19d801abd0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 msgr2=0x7f19f818b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f19ec00b660 tx=0x7f19ec00ba00).stop 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 shutdown_connections 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19d80186d0 0x7f19d801abd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19f80b1340 0x7f19f818b210 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19f818b830 0x7f19f8187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 --2- 172.21.2.117:0/748817894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19f80b0f40 0x7f19f8107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 >> 172.21.2.117:0/748817894 conn(0x7f19f80a1880 msgr2=0x7f19f80b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:31.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 shutdown_connections 2021-02-10T11:27:31.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:31.078 7f19fe530700 1 -- 172.21.2.117:0/748817894 wait complete. 2021-02-10T11:27:31.079 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802456 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df 2021-02-10T11:27:31.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:31.080 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:31.080 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:31.416 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:31.416 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:31.417 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:31.796 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:31.796 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:31.796 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234366 s, 44.7 MB/s 2021-02-10T11:27:31.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:31.797 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:31.797 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:32.132 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:32.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:32.133 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:32.503 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:32.503 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:32.504 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250885 s, 41.8 MB/s 2021-02-10T11:27:32.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:32.505 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:32.505 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:32.838 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:32.839 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:32.839 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:33.212 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:33.212 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:33.212 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232634 s, 45.1 MB/s 2021-02-10T11:27:33.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:33.213 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:33.213 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:33.548 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:33.549 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:33.549 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:33.923 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:33.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:33.924 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230611 s, 45.5 MB/s 2021-02-10T11:27:33.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:33.925 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:33.926 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:34.261 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:34.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:34.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:34.637 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:34.638 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:34.638 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242736 s, 43.2 MB/s 2021-02-10T11:27:34.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:34.639 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:34.639 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:34.974 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:34.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:34.975 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251995 s, 41.6 MB/s 2021-02-10T11:27:35.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:35.352 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:35.352 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:35.687 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:35.687 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:35.688 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:36.060 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:36.061 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:36.061 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238422 s, 44.0 MB/s 2021-02-10T11:27:36.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:36.062 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:36.062 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:36.395 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:36.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:36.396 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:36.767 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228039 s, 46.0 MB/s 2021-02-10T11:27:36.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:36.769 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:36.769 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:37.104 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:37.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:37.105 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:37.496 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:37.496 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240446 s, 43.6 MB/s 2021-02-10T11:27:37.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:37.497 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:37.497 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:37.842 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:37.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:37.843 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802456/3600cc3e-8831-4ba1-841f-4ffda776e3df/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:38.216 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:38.217 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:38.217 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240004 s, 43.7 MB/s 2021-02-10T11:27:38.218 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802457 2021-02-10T11:27:38.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 Processor -- start 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 -- start start 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c0077a70 con 0x7fa5c007da20 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00e53a0 con 0x7fa5c01b00d0 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.638 7fa5c760c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0079400 con 0x7fa5c0079000 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c5c29700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c5428700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fa5c007da20 0x7fa5c00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:36914/0 (socket says 172.21.2.117:36914) 2021-02-10T11:27:38.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:41420/0 (socket says 172.21.2.117:41420) 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c5c29700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:48126/0 (socket says 172.21.2.117:48126) 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c5428700 1 -- 172.21.2.117:0/1188113728 learned_addr learned my addr 172.21.2.117:0/1188113728 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5b7fff700 1 -- 172.21.2.117:0/1188113728 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254103885 0 0) 0x7fa5bc000c60 con 0x7fa5c007da20 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 -- 172.21.2.117:0/1188113728 >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 msgr2=0x7fa5c01b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 --2- 172.21.2.117:0/1188113728 >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 -- 172.21.2.117:0/1188113728 >> v1:172.21.2.125:6789/0 conn(0x7fa5c007da20 legacy=0x7fa5c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 -- 172.21.2.117:0/1188113728 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c007de20 con 0x7fa5c0079000 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c5c29700 1 --2- 172.21.2.117:0/1188113728 >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.639 7fa5c4c27700 1 --2- 172.21.2.117:0/1188113728 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa5b0006650 tx=0x7fa5b0009a80).ready entity=mon.2 client_cookie=1feda0fbd411df3f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.640 7fa5b7fff700 1 -- 172.21.2.117:0/1188113728 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b00009b0 con 0x7fa5c0079000 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.640 7fa5b7fff700 1 -- 172.21.2.117:0/1188113728 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b000c0c0 con 0x7fa5c0079000 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.640 7fa5b7fff700 1 -- 172.21.2.117:0/1188113728 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b000c5e0 con 0x7fa5c0079000 2021-02-10T11:27:38.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.640 7fa5c760c700 1 -- 172.21.2.117:0/1188113728 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 msgr2=0x7fa5c00e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.640 7fa5c760c700 1 --2- 172.21.2.117:0/1188113728 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa5b0006650 tx=0x7fa5b0009a80).stop 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 -- 172.21.2.117:0/1188113728 shutdown_connections 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 --2- 172.21.2.117:0/1188113728 >> v2:172.21.2.125:3300/0 conn(0x7fa5c01b00d0 0x7fa5c01b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 --2- 172.21.2.117:0/1188113728 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c0079000 0x7fa5c00e4d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 -- 172.21.2.117:0/1188113728 >> 172.21.2.117:0/1188113728 conn(0x7fa5c00ce430 msgr2=0x7fa5c00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 -- 172.21.2.117:0/1188113728 shutdown_connections 2021-02-10T11:27:38.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.641 7fa5c760c700 1 -- 172.21.2.117:0/1188113728 wait complete. 2021-02-10T11:27:38.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.655 7fa5c760c700 1 Processor -- start 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.655 7fa5c760c700 1 -- start start 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5c00a1310 0x7fa5c018a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c00b78d0 0x7fa5c00b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00ba970 con 0x7fa5c00b82f0 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0104b90 con 0x7fa5c00a1310 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c760c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0104d70 con 0x7fa5c00b78d0 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43744/0 (socket says 172.21.2.117:43744) 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 -- 172.21.2.117:0/1013087137 learned_addr learned my addr 172.21.2.117:0/1013087137 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 -- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c00b78d0 msgr2=0x7fa5c00b7cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:38.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c00b78d0 0x7fa5c00b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 -- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5c00a1310 msgr2=0x7fa5c018a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5c00a1310 0x7fa5c018a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 -- 172.21.2.117:0/1013087137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0104f90 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.656 7fa5c5c29700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fa5b800b970 tx=0x7fa5b800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.657 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b8000f90 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.657 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8006790 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.657 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5b80015f0 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.657 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c0104f90 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.657 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c0104f90 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.658 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5b8000df0 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.658 7fa5b5ffb700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 0x7fa5a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.658 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa5b8032a50 con 0x7fa5c00b82f0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.658 7fa5c5428700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 0x7fa5a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:38.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.658 7fa5c5428700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 0x7fa5a001abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:38.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.688 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c0077a70 con 0x7fa5c00b82f0 2021-02-10T11:27:38.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.691 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5b80011f0 con 0x7fa5c00b82f0 2021-02-10T11:27:38.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.816 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802457", "target": ["mgr", ""]}) v1 -- 0x7fa5c0077a70 con 0x7fa5a00186c0 2021-02-10T11:27:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.841 7fa5b5ffb700 1 -- 172.21.2.117:0/1013087137 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa5bc000f00 con 0x7fa5a00186c0 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.849 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 msgr2=0x7fa5a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.849 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 0x7fa5a001abc0 crc :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.849 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 msgr2=0x7fa5c0104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.849 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fa5b800b970 tx=0x7fa5b800bd10).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 shutdown_connections 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5a00186c0 0x7fa5a001abc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5c00b82f0 0x7fa5c0104570 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5c00b78d0 0x7fa5c00b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 --2- 172.21.2.117:0/1013087137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5c00a1310 0x7fa5c018a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 >> 172.21.2.117:0/1013087137 conn(0x7fa5c00bacb0 msgr2=0x7fa5c008e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:38.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 shutdown_connections 2021-02-10T11:27:38.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:38.850 7fa5c760c700 1 -- 172.21.2.117:0/1013087137 wait complete. 2021-02-10T11:27:38.850 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802457 2021-02-10T11:27:39.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.270 7f4089902700 1 Processor -- start 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.270 7f4089902700 1 -- start start 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4089902700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4084079000 0x7f40840e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4089902700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841b00f0 0x7f40841b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4089902700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4084077a70 con 0x7f408407da20 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4089902700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40840e53c0 con 0x7f4084079000 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4089902700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4084079400 con 0x7f40841b00f0 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4084079000 0x7f40840e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4083574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f408407da20 0x7f40840771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44002/0 (socket says 172.21.2.117:44002) 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4083574700 1 -- 172.21.2.117:0/2393939060 learned_addr learned my addr 172.21.2.117:0/2393939060 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 -- 172.21.2.117:0/2393939060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841b00f0 msgr2=0x7f40841b04f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 --2- 172.21.2.117:0/2393939060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841b00f0 0x7f40841b04f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 -- 172.21.2.117:0/2393939060 >> v1:172.21.2.117:6789/0 conn(0x7f408407da20 legacy=0x7f40840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 -- 172.21.2.117:0/2393939060 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408407de20 con 0x7f4084079000 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082d73700 1 --2- 172.21.2.117:0/2393939060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084079000 0x7f40840e4da0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4074006650 tx=0x7f4074009a80).ready entity=mon.0 client_cookie=c1e0be940c802a24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.271 7f4082572700 1 -- 172.21.2.117:0/2393939060 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40740009b0 con 0x7f4084079000 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4082572700 1 -- 172.21.2.117:0/2393939060 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407400c0c0 con 0x7f4084079000 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4082572700 1 -- 172.21.2.117:0/2393939060 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f407400c510 con 0x7f4084079000 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 -- 172.21.2.117:0/2393939060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084079000 msgr2=0x7f40840e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 --2- 172.21.2.117:0/2393939060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084079000 0x7f40840e4da0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f4074006650 tx=0x7f4074009a80).stop 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 -- 172.21.2.117:0/2393939060 shutdown_connections 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 --2- 172.21.2.117:0/2393939060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841b00f0 0x7f40841b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 --2- 172.21.2.117:0/2393939060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084079000 0x7f40840e4da0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 -- 172.21.2.117:0/2393939060 >> 172.21.2.117:0/2393939060 conn(0x7f40840ce450 msgr2=0x7f40840e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 -- 172.21.2.117:0/2393939060 shutdown_connections 2021-02-10T11:27:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.272 7f4089902700 1 -- 172.21.2.117:0/2393939060 wait complete. 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.286 7f4089902700 1 Processor -- start 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.286 7f4089902700 1 -- start start 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4084093c80 0x7f40841a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841a7bf0 0x7f4084094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084095740 con 0x7f4084094080 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084095960 con 0x7f4084093c80 2021-02-10T11:27:39.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4089902700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4084095b80 con 0x7f40841a7bf0 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43760/0 (socket says 172.21.2.117:43760) 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 -- 172.21.2.117:0/3081459593 learned_addr learned my addr 172.21.2.117:0/3081459593 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 -- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841a7bf0 msgr2=0x7f4084094b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841a7bf0 0x7f4084094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 -- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4084093c80 msgr2=0x7f40841a75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4084093c80 0x7f40841a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 -- 172.21.2.117:0/3081459593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084095da0 con 0x7f4084094080 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.287 7f4083d75700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f407c009960 tx=0x7f407c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f407c003030 con 0x7f4084094080 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c001310 con 0x7f4084094080 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f4089902700 1 -- 172.21.2.117:0/3081459593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4084095da0 con 0x7f4084094080 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f4089902700 1 -- 172.21.2.117:0/3081459593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4084095da0 con 0x7f4084094080 2021-02-10T11:27:39.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f407c001830 con 0x7f4084094080 2021-02-10T11:27:39.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f407c000d10 con 0x7f4084094080 2021-02-10T11:27:39.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 0x7f406801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:39.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.288 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f407c032b10 con 0x7f4084094080 2021-02-10T11:27:39.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.289 7f4083574700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 0x7f406801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:39.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.289 7f4083574700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 0x7f406801ac10 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:39.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.320 7f4089902700 1 -- 172.21.2.117:0/3081459593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4084103310 con 0x7f4084094080 2021-02-10T11:27:39.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.323 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f407c003020 con 0x7f4084094080 2021-02-10T11:27:39.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.451 7f4089902700 1 -- 172.21.2.117:0/3081459593 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802457", "target": ["mgr", ""]}) v1 -- 0x7f4084103310 con 0x7f4068018730 2021-02-10T11:27:39.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.455 7f407bfff700 1 -- 172.21.2.117:0/3081459593 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f406c001530 con 0x7f4068018730 2021-02-10T11:27:39.451 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 msgr2=0x7f406801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 0x7f406801ac10 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 msgr2=0x7f4084095120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f407c009960 tx=0x7f407c00bcf0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 shutdown_connections 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4068018730 0x7f406801ac10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4084094080 0x7f4084095120 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40841a7bf0 0x7f4084094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 --2- 172.21.2.117:0/3081459593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4084093c80 0x7f40841a75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:39.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 >> 172.21.2.117:0/3081459593 conn(0x7f40840a1880 msgr2=0x7f40840b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:39.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 shutdown_connections 2021-02-10T11:27:39.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:39.460 7f4089902700 1 -- 172.21.2.117:0/3081459593 wait complete. 2021-02-10T11:27:39.461 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802457 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af 2021-02-10T11:27:39.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:39.461 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:39.461 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:39.796 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:39.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:39.797 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:40.177 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:40.178 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024606 s, 42.6 MB/s 2021-02-10T11:27:40.178 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:40.179 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:40.179 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:40.514 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:40.514 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:40.514 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227496 s, 46.1 MB/s 2021-02-10T11:27:40.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:40.888 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:40.888 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:41.224 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:41.225 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:41.225 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:41.600 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:41.600 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:41.600 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249244 s, 42.1 MB/s 2021-02-10T11:27:41.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:41.601 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:41.601 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:41.936 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:41.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:41.937 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:42.311 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:42.312 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:42.312 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023576 s, 44.5 MB/s 2021-02-10T11:27:42.313 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:42.313 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:42.313 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:42.647 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:42.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:42.648 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:43.019 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:43.019 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:43.019 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233687 s, 44.9 MB/s 2021-02-10T11:27:43.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:43.020 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:43.020 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:43.355 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:43.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:43.356 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:43.731 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:43.732 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:43.732 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025436 s, 41.2 MB/s 2021-02-10T11:27:43.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:43.733 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:43.733 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:44.067 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:44.068 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:44.068 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:44.442 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:44.442 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:44.442 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248649 s, 42.2 MB/s 2021-02-10T11:27:44.443 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:44.443 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:44.444 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:44.778 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:44.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:44.779 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:45.155 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:45.155 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:45.155 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252915 s, 41.5 MB/s 2021-02-10T11:27:45.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:45.157 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:45.157 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:45.490 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:45.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:45.490 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:45.864 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229699 s, 45.6 MB/s 2021-02-10T11:27:45.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:45.866 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:45.866 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:46.201 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:46.201 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:46.201 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802457/b28786b1-241f-40ce-9ebe-df37081fa4af/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:46.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:46.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:46.576 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234138 s, 44.8 MB/s 2021-02-10T11:27:46.577 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802458 2021-02-10T11:27:47.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 Processor -- start 2021-02-10T11:27:47.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 -- start start 2021-02-10T11:27:47.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36b0079000 0x7f36b00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00d1d20 con 0x7f36b01b00f0 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b0077a70 con 0x7f36b007da20 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b78b8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b00e53c0 con 0x7f36b0079000 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b4ed3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36b0079000 0x7f36b00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b5ed5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f36b01b00f0 0x7f36b00d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44018/0 (socket says 172.21.2.117:44018) 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.998 7f36b5ed5700 1 -- 172.21.2.117:0/3918475738 learned_addr learned my addr 172.21.2.117:0/3918475738 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b4ed3700 1 -- 172.21.2.117:0/3918475738 >> v1:172.21.2.125:6789/0 conn(0x7f36b007da20 legacy=0x7f36b00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b4ed3700 1 -- 172.21.2.117:0/3918475738 >> v1:172.21.2.117:6789/0 conn(0x7f36b01b00f0 legacy=0x7f36b00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b4ed3700 1 -- 172.21.2.117:0/3918475738 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0079400 con 0x7f36b0079000 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36a3fff700 1 -- 172.21.2.117:0/3918475738 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480181126 0 0) 0x7f36a8001500 con 0x7f36b01b00f0 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b4ed3700 1 --2- 172.21.2.117:0/3918475738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b0079000 0x7f36b00e4da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f36a4006650 tx=0x7f36a4009a00).ready entity=mon.0 client_cookie=a04fc3a6a82f7342 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36a3fff700 1 -- 172.21.2.117:0/3918475738 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a4009df0 con 0x7f36b0079000 2021-02-10T11:27:47.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36a3fff700 1 -- 172.21.2.117:0/3918475738 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a40009b0 con 0x7f36b0079000 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36a3fff700 1 -- 172.21.2.117:0/3918475738 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a400c2b0 con 0x7f36b0079000 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b78b8700 1 -- 172.21.2.117:0/3918475738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b0079000 msgr2=0x7f36b00e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:46.999 7f36b78b8700 1 --2- 172.21.2.117:0/3918475738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b0079000 0x7f36b00e4da0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f36a4006650 tx=0x7f36a4009a00).stop 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.000 7f36b78b8700 1 -- 172.21.2.117:0/3918475738 shutdown_connections 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.000 7f36b78b8700 1 --2- 172.21.2.117:0/3918475738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b0079000 0x7f36b00e4da0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.000 7f36b78b8700 1 -- 172.21.2.117:0/3918475738 >> 172.21.2.117:0/3918475738 conn(0x7f36b00ce450 msgr2=0x7f36b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.000 7f36b78b8700 1 -- 172.21.2.117:0/3918475738 shutdown_connections 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.000 7f36b78b8700 1 -- 172.21.2.117:0/3918475738 wait complete. 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 Processor -- start 2021-02-10T11:27:47.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 -- start start 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36b009c280 0x7f36b009c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36b009cca0 0x7f36b00aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0107df0 con 0x7f36b00bc290 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00bc690 con 0x7f36b009c280 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.015 7f36b78b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b009d2b0 con 0x7f36b009cca0 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43772/0 (socket says 172.21.2.117:43772) 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b5ed5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36b009cca0 0x7f36b00aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 -- 172.21.2.117:0/554254021 learned_addr learned my addr 172.21.2.117:0/554254021 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:47.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b4ed3700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36b009c280 0x7f36b009c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 -- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36b009cca0 msgr2=0x7f36b00aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36b009cca0 0x7f36b00aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 -- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36b009c280 msgr2=0x7f36b009c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36b009c280 0x7f36b009c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 -- 172.21.2.117:0/554254021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b00bbed0 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b56d4700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f36ac006f10 tx=0x7f36ac00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36ac00bd80 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac001120 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36ac0015b0 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b78b8700 1 -- 172.21.2.117:0/554254021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b00bbed0 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.016 7f36b78b8700 1 -- 172.21.2.117:0/554254021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b00bbed0 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.017 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36ac000d60 con 0x7f36b00bc290 2021-02-10T11:27:47.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.017 7f36a1ffb700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 0x7f369001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.017 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f36ac0328e0 con 0x7f36b00bc290 2021-02-10T11:27:47.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.017 7f36b4ed3700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 0x7f369001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.018 7f36b4ed3700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 0x7f369001abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.048 7f36b78b8700 1 -- 172.21.2.117:0/554254021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b00ff170 con 0x7f36b00bc290 2021-02-10T11:27:47.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.050 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f36ac006ba0 con 0x7f36b00bc290 2021-02-10T11:27:47.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.175 7f36b78b8700 1 -- 172.21.2.117:0/554254021 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802458", "target": ["mgr", ""]}) v1 -- 0x7f36b00bc690 con 0x7f3690018730 2021-02-10T11:27:47.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.201 7f36a1ffb700 1 -- 172.21.2.117:0/554254021 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f36a4006740 con 0x7f3690018730 2021-02-10T11:27:47.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 -- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 msgr2=0x7f369001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 0x7f369001abc0 crc :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 -- 172.21.2.117:0/554254021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 msgr2=0x7f36b019feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f36ac006f10 tx=0x7f36ac00b990).stop 2021-02-10T11:27:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 -- 172.21.2.117:0/554254021 shutdown_connections 2021-02-10T11:27:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36b009cca0 0x7f36b00aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36b009c280 0x7f36b009c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3690018730 0x7f369001abc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 --2- 172.21.2.117:0/554254021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36b00bc290 0x7f36b019feb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 -- 172.21.2.117:0/554254021 >> 172.21.2.117:0/554254021 conn(0x7f36b00b28f0 msgr2=0x7f36b0093430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.208 7f36b78b8700 1 -- 172.21.2.117:0/554254021 shutdown_connections 2021-02-10T11:27:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.209 7f36b78b8700 1 -- 172.21.2.117:0/554254021 wait complete. 2021-02-10T11:27:47.209 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802458 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.630 7f1a3e6dd700 1 Processor -- start 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.630 7f1a3e6dd700 1 -- start start 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a3e6dd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1a38079000 0x7f1a380e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a3e6dd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a380d1d20 con 0x7f1a3807da20 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a3e6dd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a38077a70 con 0x7f1a381b00f0 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a3e6dd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380e53c0 con 0x7f1a38079000 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a37fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1a3807da20 0x7f1a380771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44028/0 (socket says 172.21.2.117:44028) 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.631 7f1a37fff700 1 -- 172.21.2.117:0/3529472992 learned_addr learned my addr 172.21.2.117:0/3529472992 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:47.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560327412 0 0) 0x7f1a28000ce0 con 0x7f1a3807da20 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a20002fe0 con 0x7f1a3807da20 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a377fe700 1 --2- 172.21.2.117:0/3529472992 >> v2:172.21.2.125:3300/0 conn(0x7f1a38079000 0x7f1a380e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1234253029 0 0) 0x7f1a28001180 con 0x7f1a3807da20 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 >> v2:172.21.2.125:3300/0 conn(0x7f1a38079000 msgr2=0x7f1a380e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 --2- 172.21.2.117:0/3529472992 >> v2:172.21.2.125:3300/0 conn(0x7f1a38079000 0x7f1a380e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 >> v1:172.21.2.125:6789/0 conn(0x7f1a381b00f0 legacy=0x7f1a380d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a38079400 con 0x7f1a3807da20 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1a28001680 con 0x7f1a3807da20 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 >> v1:172.21.2.117:6789/0 conn(0x7f1a3807da20 legacy=0x7f1a380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 0x7f1a20005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a20006790 0x7f1a20008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a20009590 0x7f1a2000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2000c300 con 0x7f1a20003a60 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2000c520 con 0x7f1a20006790 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2000c740 con 0x7f1a20009590 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.632 7f1a37fff700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 0x7f1a20005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 -- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a20009590 msgr2=0x7f1a2000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a20009590 0x7f1a2000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 -- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a20006790 msgr2=0x7f1a20008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a20006790 0x7f1a20008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 -- 172.21.2.117:0/3529472992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200509b0 con 0x7f1a20003a60 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a3ccfa700 1 -- 172.21.2.117:0/3529472992 reap_dead start 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a37fff700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 0x7f1a20005ef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f1a28001870 tx=0x7f1a28007d20).ready entity=mon.0 client_cookie=7205d912f3c735b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a28000c90 con 0x7f1a20003a60 2021-02-10T11:27:47.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a280070c0 con 0x7f1a20003a60 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a36ffd700 1 -- 172.21.2.117:0/3529472992 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a2800a710 con 0x7f1a20003a60 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a3e6dd700 1 -- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 msgr2=0x7f1a20005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.633 7f1a3e6dd700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 0x7f1a20005ef0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f1a28001870 tx=0x7f1a28007d20).stop 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.634 7f1a3e6dd700 1 -- 172.21.2.117:0/3529472992 shutdown_connections 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.634 7f1a3e6dd700 1 --2- 172.21.2.117:0/3529472992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a20003a60 0x7f1a20005ef0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.634 7f1a3e6dd700 1 -- 172.21.2.117:0/3529472992 >> 172.21.2.117:0/3529472992 conn(0x7f1a380ce450 msgr2=0x7f1a380e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.634 7f1a3e6dd700 1 -- 172.21.2.117:0/3529472992 shutdown_connections 2021-02-10T11:27:47.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.634 7f1a3e6dd700 1 -- 172.21.2.117:0/3529472992 wait complete. 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 Processor -- start 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 -- start start 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 0x7f1a380a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a380b77d0 0x7f1a380b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38189970 con 0x7f1a380a51f0 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38190aa0 con 0x7f1a380a47d0 2021-02-10T11:27:47.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.649 7f1a3e6dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380b81f0 con 0x7f1a380b77d0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43792/0 (socket says 172.21.2.117:43792) 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 -- 172.21.2.117:0/2168079482 learned_addr learned my addr 172.21.2.117:0/2168079482 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a3ccfa700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a380b77d0 0x7f1a380b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a37fff700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 0x7f1a380a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 -- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a380b77d0 msgr2=0x7f1a380b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a380b77d0 0x7f1a380b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 -- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 msgr2=0x7f1a380a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 0x7f1a380a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 -- 172.21.2.117:0/2168079482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a380b8410 con 0x7f1a380a51f0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a37fff700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 0x7f1a380a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a377fe700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f1a2c006f40 tx=0x7f1a2c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a2c00bd80 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c000c90 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a2c0022b0 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.650 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a380b8410 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.651 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a380b8410 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.651 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a2c006b00 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.651 7f1a34ff9700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 0x7f1a1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.651 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1a2c032790 con 0x7f1a380a51f0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.651 7f1a37fff700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 0x7f1a1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:47.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.652 7f1a37fff700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 0x7f1a1801ac10 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.681 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a38077a70 con 0x7f1a380a51f0 2021-02-10T11:27:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.683 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a2c000c90 con 0x7f1a380a51f0 2021-02-10T11:27:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.809 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802458", "target": ["mgr", ""]}) v1 -- 0x7f1a38077a70 con 0x7f1a18018730 2021-02-10T11:27:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.813 7f1a34ff9700 1 -- 172.21.2.117:0/2168079482 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1a28000c90 con 0x7f1a18018730 2021-02-10T11:27:47.809 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6 2021-02-10T11:27:47.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 msgr2=0x7f1a1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 0x7f1a1801ac10 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 msgr2=0x7f1a380a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:47.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f1a2c006f40 tx=0x7f1a2c00b990).stop 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 shutdown_connections 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a18018730 0x7f1a1801ac10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a380a51f0 0x7f1a380a55f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a380b77d0 0x7f1a380b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 --2- 172.21.2.117:0/2168079482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a380a47d0 0x7f1a380a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 >> 172.21.2.117:0/2168079482 conn(0x7f1a380b5140 msgr2=0x7f1a380a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 shutdown_connections 2021-02-10T11:27:47.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:47.819 7f1a3e6dd700 1 -- 172.21.2.117:0/2168079482 wait complete. 2021-02-10T11:27:47.820 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802458 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6 2021-02-10T11:27:47.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:47.820 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:47.820 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:48.155 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:48.156 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:48.156 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236602 s, 44.3 MB/s 2021-02-10T11:27:48.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:48.536 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:48.536 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:48.870 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:48.870 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:48.871 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:49.244 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:49.244 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:49.244 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231774 s, 45.2 MB/s 2021-02-10T11:27:49.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:49.245 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:49.246 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:49.580 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:49.580 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:49.581 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:49.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:49.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:49.955 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238977 s, 43.9 MB/s 2021-02-10T11:27:49.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:49.957 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:49.957 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:50.291 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:50.291 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:50.292 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:50.668 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:50.669 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:50.669 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254974 s, 41.1 MB/s 2021-02-10T11:27:50.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:50.670 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:50.670 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:51.005 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:51.005 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:51.006 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236834 s, 44.3 MB/s 2021-02-10T11:27:51.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:51.381 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:51.381 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:51.716 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:51.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:51.716 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:52.092 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:52.092 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:52.092 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257118 s, 40.8 MB/s 2021-02-10T11:27:52.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:52.094 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:52.094 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:52.428 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:52.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:52.428 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:52.801 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:52.801 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241622 s, 43.4 MB/s 2021-02-10T11:27:52.802 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:52.802 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:52.802 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:53.137 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:53.137 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:53.137 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:53.511 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:53.512 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:53.512 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238048 s, 44.0 MB/s 2021-02-10T11:27:53.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:53.513 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:53.513 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:53.847 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:53.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:53.848 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:54.224 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:54.225 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:54.225 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255064 s, 41.1 MB/s 2021-02-10T11:27:54.226 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:54.226 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:54.226 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:54.560 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:54.560 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:54.561 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802458/27ba2c34-1fbc-4a68-a417-4a2e7cae04a6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:54.935 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:54.936 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:54.936 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232957 s, 45.0 MB/s 2021-02-10T11:27:54.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802459 2021-02-10T11:27:55.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.357 7f36ad398700 1 Processor -- start 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.357 7f36ad398700 1 -- start start 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f36a807da20 0x7f36a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a80e4da0 0x7f36a81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a80e51a0 0x7f36a8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a8077a70 con 0x7f36a80e51a0 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a80777d0 con 0x7f36a807da20 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36ad398700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a807de20 con 0x7f36a80e4da0 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a80e51a0 0x7f36a8077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36a80e51a0 0x7f36a8077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43798/0 (socket says 172.21.2.117:43798) 2021-02-10T11:27:55.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 -- 172.21.2.117:0/4098786863 learned_addr learned my addr 172.21.2.117:0/4098786863 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 -- 172.21.2.117:0/4098786863 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a80e4da0 msgr2=0x7f36a81b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 --2- 172.21.2.117:0/4098786863 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a80e4da0 0x7f36a81b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 -- 172.21.2.117:0/4098786863 >> v2:172.21.2.125:3300/0 conn(0x7f36a807da20 msgr2=0x7f36a80771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 --2- 172.21.2.117:0/4098786863 >> v2:172.21.2.125:3300/0 conn(0x7f36a807da20 0x7f36a80771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.358 7f36a787e700 1 -- 172.21.2.117:0/4098786863 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a81b0710 con 0x7f36a80e51a0 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36a787e700 1 --2- 172.21.2.117:0/4098786863 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a80e51a0 0x7f36a8077ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f36a0006f00 tx=0x7f36a0009f10).ready entity=mon.0 client_cookie=8fd999846e778ce6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36a607b700 1 -- 172.21.2.117:0/4098786863 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a000b3b0 con 0x7f36a80e51a0 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36a607b700 1 -- 172.21.2.117:0/4098786863 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a000b730 con 0x7f36a80e51a0 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36a607b700 1 -- 172.21.2.117:0/4098786863 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36a000bc50 con 0x7f36a80e51a0 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 -- 172.21.2.117:0/4098786863 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a80e51a0 msgr2=0x7f36a8077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 --2- 172.21.2.117:0/4098786863 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a80e51a0 0x7f36a8077ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f36a0006f00 tx=0x7f36a0009f10).stop 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 -- 172.21.2.117:0/4098786863 shutdown_connections 2021-02-10T11:27:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 --2- 172.21.2.117:0/4098786863 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a80e51a0 0x7f36a8077ff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 --2- 172.21.2.117:0/4098786863 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a80e4da0 0x7f36a81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 --2- 172.21.2.117:0/4098786863 >> v2:172.21.2.125:3300/0 conn(0x7f36a807da20 0x7f36a80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 -- 172.21.2.117:0/4098786863 >> 172.21.2.117:0/4098786863 conn(0x7f36a80ce450 msgr2=0x7f36a80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 -- 172.21.2.117:0/4098786863 shutdown_connections 2021-02-10T11:27:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.359 7f36ad398700 1 -- 172.21.2.117:0/4098786863 wait complete. 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 Processor -- start 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 -- start start 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a818b350 0x7f36a81054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a8105b00 0x7f36a818bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a818b7f0 con 0x7f36a818af50 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a809d610 con 0x7f36a818b350 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36ad398700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819e160 con 0x7f36a8105b00 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36a707d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36a707d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43800/0 (socket says 172.21.2.117:43800) 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.374 7f36a707d700 1 -- 172.21.2.117:0/2584417607 learned_addr learned my addr 172.21.2.117:0/2584417607 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 -- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a8105b00 msgr2=0x7f36a818bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a8105b00 0x7f36a818bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 -- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a818b350 msgr2=0x7f36a81054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a818b350 0x7f36a81054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 -- 172.21.2.117:0/2584417607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8105f00 con 0x7f36a818af50 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36a707d700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f369800b5c0 tx=0x7f369800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369800bd50 con 0x7f36a818af50 2021-02-10T11:27:55.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698006030 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36980054d0 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36ad398700 1 -- 172.21.2.117:0/2584417607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a8105f00 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.375 7f36ad398700 1 -- 172.21.2.117:0/2584417607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a818c390 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.376 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3698006210 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.376 7f369ffff700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 0x7f368c01eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.376 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3698032910 con 0x7f36a818af50 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.376 7f36a687c700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 0x7f368c01eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:55.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.377 7f36a687c700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 0x7f368c01eff0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.408 7f36ad398700 1 -- 172.21.2.117:0/2584417607 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a809db20 con 0x7f36a818af50 2021-02-10T11:27:55.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.411 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f36980009b0 con 0x7f36a818af50 2021-02-10T11:27:55.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.536 7f36ad398700 1 -- 172.21.2.117:0/2584417607 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802459", "target": ["mgr", ""]}) v1 -- 0x7f36a809db20 con 0x7f368c01cad0 2021-02-10T11:27:55.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.562 7f369ffff700 1 -- 172.21.2.117:0/2584417607 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3690006e60 con 0x7f368c01cad0 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.569 7f36ad398700 1 -- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 msgr2=0x7f368c01eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.569 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 0x7f368c01eff0 crc :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.569 7f36ad398700 1 -- 172.21.2.117:0/2584417607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 msgr2=0x7f36a819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.569 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f369800b5c0 tx=0x7f369800b960).stop 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 -- 172.21.2.117:0/2584417607 shutdown_connections 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a8105b00 0x7f36a818bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a818b350 0x7f36a81054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f368c01cad0 0x7f368c01eff0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 --2- 172.21.2.117:0/2584417607 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a818af50 0x7f36a819db40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 -- 172.21.2.117:0/2584417607 >> 172.21.2.117:0/2584417607 conn(0x7f36a8096500 msgr2=0x7f36a80b8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 -- 172.21.2.117:0/2584417607 shutdown_connections 2021-02-10T11:27:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.570 7f36ad398700 1 -- 172.21.2.117:0/2584417607 wait complete. 2021-02-10T11:27:55.571 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802459 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.990 7f2dff693700 1 Processor -- start 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.990 7f2dff693700 1 -- start start 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dff693700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2df807da20 0x7f2df80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dff693700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80d1d20 con 0x7f2df80e51a0 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dff693700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df8077a70 con 0x7f2df80e4da0 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dff693700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df80777d0 con 0x7f2df807da20 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfd4af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2df807da20 0x7f2df80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfdcb0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2df80e51a0 0x7f2df80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44058/0 (socket says 172.21.2.117:44058) 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfdcb0700 1 -- 172.21.2.117:0/1195145795 learned_addr learned my addr 172.21.2.117:0/1195145795 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:27:55.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfd4af700 1 -- 172.21.2.117:0/1195145795 >> v1:172.21.2.125:6789/0 conn(0x7f2df80e4da0 legacy=0x7f2df81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfd4af700 1 -- 172.21.2.117:0/1195145795 >> v1:172.21.2.117:6789/0 conn(0x7f2df80e51a0 legacy=0x7f2df80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.991 7f2dfd4af700 1 -- 172.21.2.117:0/1195145795 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df81b08e0 con 0x7f2df807da20 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dfd4af700 1 --2- 172.21.2.117:0/1195145795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df807da20 0x7f2df80771b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f2dec006650 tx=0x7f2dec009a00).ready entity=mon.0 client_cookie=8fdeaa49878334dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2debfff700 1 -- 172.21.2.117:0/1195145795 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dec0009b0 con 0x7f2df807da20 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2debfff700 1 -- 172.21.2.117:0/1195145795 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec00c0c0 con 0x7f2df807da20 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2debfff700 1 -- 172.21.2.117:0/1195145795 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dec00c5a0 con 0x7f2df807da20 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 -- 172.21.2.117:0/1195145795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df807da20 msgr2=0x7f2df80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 --2- 172.21.2.117:0/1195145795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df807da20 0x7f2df80771b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f2dec006650 tx=0x7f2dec009a00).stop 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 -- 172.21.2.117:0/1195145795 shutdown_connections 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 --2- 172.21.2.117:0/1195145795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df807da20 0x7f2df80771b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 -- 172.21.2.117:0/1195145795 >> 172.21.2.117:0/1195145795 conn(0x7f2df80ce450 msgr2=0x7f2df80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 -- 172.21.2.117:0/1195145795 shutdown_connections 2021-02-10T11:27:56.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:55.992 7f2dff693700 1 -- 172.21.2.117:0/1195145795 wait complete. 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 Processor -- start 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 -- start start 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2df80933f0 0x7f2df80937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2df8093e10 0x7f2df8106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8107a70 con 0x7f2df8107050 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80c2c60 con 0x7f2df8093e10 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dff693700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80c2e80 con 0x7f2df80933f0 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.007 7f2dfdcb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43812/0 (socket says 172.21.2.117:43812) 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 -- 172.21.2.117:0/4128622412 learned_addr learned my addr 172.21.2.117:0/4128622412 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 -- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2df80933f0 msgr2=0x7f2df80937f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2df80933f0 0x7f2df80937f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 -- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2df8093e10 msgr2=0x7f2df8106a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2df8093e10 0x7f2df8106a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 -- 172.21.2.117:0/4128622412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df80c30a0 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dfdcb0700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f2df000a940 tx=0x7f2df000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df0001270 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df00070b0 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2df0007480 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dff693700 1 -- 172.21.2.117:0/4128622412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df80c30a0 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.008 7f2dff693700 1 -- 172.21.2.117:0/4128622412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df80c3630 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.009 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2df0007290 con 0x7f2df8107050 2021-02-10T11:27:56.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.009 7f2de9ffb700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 0x7f2dd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:27:56.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.009 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2df0032a10 con 0x7f2df8107050 2021-02-10T11:27:56.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.009 7f2dfd4af700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 0x7f2dd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:27:56.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.010 7f2dfd4af700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 0x7f2dd801abc0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:27:56.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.041 7f2dff693700 1 -- 172.21.2.117:0/4128622412 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df8107e30 con 0x7f2df8107050 2021-02-10T11:27:56.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.043 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2df002d270 con 0x7f2df8107050 2021-02-10T11:27:56.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.169 7f2dff693700 1 -- 172.21.2.117:0/4128622412 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802459", "target": ["mgr", ""]}) v1 -- 0x7f2df80c3630 con 0x7f2dd80186c0 2021-02-10T11:27:56.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.173 7f2de9ffb700 1 -- 172.21.2.117:0/4128622412 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2dec000c90 con 0x7f2dd80186c0 2021-02-10T11:27:56.169 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 -- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 msgr2=0x7f2dd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 0x7f2dd801abc0 crc :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 -- 172.21.2.117:0/4128622412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 msgr2=0x7f2df8107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f2df000a940 tx=0x7f2df000ace0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 -- 172.21.2.117:0/4128622412 shutdown_connections 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dd80186c0 0x7f2dd801abc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2df8107050 0x7f2df8107450 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2df8093e10 0x7f2df8106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 --2- 172.21.2.117:0/4128622412 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2df80933f0 0x7f2df80937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 -- 172.21.2.117:0/4128622412 >> 172.21.2.117:0/4128622412 conn(0x7f2df81116a0 msgr2=0x7f2df80af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.179 7f2dff693700 1 -- 172.21.2.117:0/4128622412 shutdown_connections 2021-02-10T11:27:56.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:27:56.180 7f2dff693700 1 -- 172.21.2.117:0/4128622412 wait complete. 2021-02-10T11:27:56.179 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802459 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146 2021-02-10T11:27:56.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:56.180 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:56.180 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:56.189 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:56.225 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:27:56.515 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:56.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:56.516 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:56.896 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:56.896 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:56.896 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238136 s, 44.0 MB/s 2021-02-10T11:27:56.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:56.897 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:56.897 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:57.232 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:57.232 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:57.232 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:57.601 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:57.602 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:57.602 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226405 s, 46.3 MB/s 2021-02-10T11:27:57.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:57.603 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:57.603 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:57.939 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:57.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:57.940 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:58.334 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:58.335 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:58.335 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241117 s, 43.5 MB/s 2021-02-10T11:27:58.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:58.336 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:58.336 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:58.670 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:58.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:58.670 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:59.044 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:59.045 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:59.045 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228025 s, 46.0 MB/s 2021-02-10T11:27:59.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:59.046 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:59.046 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:27:59.381 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:27:59.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:27:59.381 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:27:59.753 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:27:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:27:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0220707 s, 47.5 MB/s 2021-02-10T11:27:59.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:27:59.755 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:27:59.755 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:00.091 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:00.091 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:00.092 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252115 s, 41.6 MB/s 2021-02-10T11:28:00.470 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:00.470 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:00.470 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:00.806 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:00.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:00.806 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:01.180 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:01.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:01.181 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248028 s, 42.3 MB/s 2021-02-10T11:28:01.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:01.182 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:01.182 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:01.518 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:01.518 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:01.518 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:01.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:01.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:01.891 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232435 s, 45.1 MB/s 2021-02-10T11:28:01.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:01.893 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:01.893 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:02.228 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:02.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:02.229 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:02.602 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:02.602 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:02.602 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239898 s, 43.7 MB/s 2021-02-10T11:28:02.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:02.603 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:02.603 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:02.938 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:02.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:02.938 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802459/c8000f60-32b1-42ff-a23a-b4180bb51146/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:03.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:03.340 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241121 s, 43.5 MB/s 2021-02-10T11:28:03.342 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802460 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.763 7f8c1ffac700 1 Processor -- start 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.763 7f8c1ffac700 1 -- start start 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1ffac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c18079000 0x7f8c180e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1ffac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c181b00f0 0x7f8c181b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1ffac700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c18077a70 con 0x7f8c1807da20 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1ffac700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180e53c0 con 0x7f8c181b00f0 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1ffac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18079400 con 0x7f8c18079000 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c181b00f0 0x7f8c181b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c181b00f0 0x7f8c181b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43824/0 (socket says 172.21.2.117:43824) 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 -- 172.21.2.117:0/489644584 learned_addr learned my addr 172.21.2.117:0/489644584 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:03.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1d5c7700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c18079000 0x7f8c180e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 -- 172.21.2.117:0/489644584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c18079000 msgr2=0x7f8c180e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c18079000 0x7f8c180e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 -- 172.21.2.117:0/489644584 >> v1:172.21.2.117:6789/0 conn(0x7f8c1807da20 legacy=0x7f8c180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.764 7f8c1e5c9700 1 -- 172.21.2.117:0/489644584 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1807de20 con 0x7f8c181b00f0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1e5c9700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b00f0 0x7f8c181b04f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f8c10006f00 tx=0x7f8c10009f10).ready entity=mon.0 client_cookie=3a5dcd235f1735ee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1cdc6700 1 -- 172.21.2.117:0/489644584 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c10003030 con 0x7f8c181b00f0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1cdc6700 1 -- 172.21.2.117:0/489644584 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1000b3b0 con 0x7f8c181b00f0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1cdc6700 1 -- 172.21.2.117:0/489644584 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c1000b8d0 con 0x7f8c181b00f0 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 -- 172.21.2.117:0/489644584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b00f0 msgr2=0x7f8c181b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b00f0 0x7f8c181b04f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f8c10006f00 tx=0x7f8c10009f10).stop 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 -- 172.21.2.117:0/489644584 shutdown_connections 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b00f0 0x7f8c181b04f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 --2- 172.21.2.117:0/489644584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c18079000 0x7f8c180e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 -- 172.21.2.117:0/489644584 >> 172.21.2.117:0/489644584 conn(0x7f8c180ce450 msgr2=0x7f8c180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 -- 172.21.2.117:0/489644584 shutdown_connections 2021-02-10T11:28:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.765 7f8c1ffac700 1 -- 172.21.2.117:0/489644584 wait complete. 2021-02-10T11:28:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 Processor -- start 2021-02-10T11:28:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 -- start start 2021-02-10T11:28:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c181b5130 0x7f8c181b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c181b5b50 0x7f8c180b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180b8da0 con 0x7f8c181b5f50 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180a4140 con 0x7f8c181b5b50 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1ffac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180a4320 con 0x7f8c181b5130 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.780 7f8c1e5c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43826/0 (socket says 172.21.2.117:43826) 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 -- 172.21.2.117:0/4029423226 learned_addr learned my addr 172.21.2.117:0/4029423226 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 -- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c181b5130 msgr2=0x7f8c181b5530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c181b5130 0x7f8c181b5530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 -- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c181b5b50 msgr2=0x7f8c180b5670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c181b5b50 0x7f8c180b5670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 -- 172.21.2.117:0/4029423226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180b62b0 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1e5c9700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8c1000b940 tx=0x7f8c1000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c10009450 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10003030 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c1000c570 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180b62b0 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.781 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c180b62b0 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.782 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c10001270 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.782 7f8c0a7fc700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 0x7f8c0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.782 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8c10032a50 con 0x7f8c181b5f50 2021-02-10T11:28:03.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.783 7f8c1ddc8700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 0x7f8c0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.784 7f8c1ddc8700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 0x7f8c0001abc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.814 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c181b4f00 con 0x7f8c181b5f50 2021-02-10T11:28:03.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.817 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c100083c0 con 0x7f8c181b5f50 2021-02-10T11:28:03.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.942 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802460", "target": ["mgr", ""]}) v1 -- 0x7f8c18077a70 con 0x7f8c000186c0 2021-02-10T11:28:03.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.968 7f8c0a7fc700 1 -- 172.21.2.117:0/4029423226 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8c0c006db0 con 0x7f8c000186c0 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 msgr2=0x7f8c0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 0x7f8c0001abc0 crc :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 msgr2=0x7f8c180b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8c1000b940 tx=0x7f8c1000bce0).stop 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 shutdown_connections 2021-02-10T11:28:03.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c000186c0 0x7f8c0001abc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c181b5f50 0x7f8c180b5c90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c181b5b50 0x7f8c180b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 --2- 172.21.2.117:0/4029423226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c181b5130 0x7f8c181b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 >> 172.21.2.117:0/4029423226 conn(0x7f8c1808bbc0 msgr2=0x7f8c180c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 shutdown_connections 2021-02-10T11:28:03.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:03.975 7f8c1ffac700 1 -- 172.21.2.117:0/4029423226 wait complete. 2021-02-10T11:28:03.975 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802460 2021-02-10T11:28:04.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.397 7f3f59119700 1 Processor -- start 2021-02-10T11:28:04.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.397 7f3f59119700 1 -- start start 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f59119700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f5407da20 0x7f3f540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f59119700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f540d1d20 con 0x7f3f540e51a0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f59119700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f54077a70 con 0x7f3f540e4da0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f59119700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540777d0 con 0x7f3f5407da20 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f5359e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3f540e51a0 0x7f3f540d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44086/0 (socket says 172.21.2.117:44086) 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f5359e700 1 -- 172.21.2.117:0/3620436073 learned_addr learned my addr 172.21.2.117:0/3620436073 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f52d9d700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f5407da20 0x7f3f540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447285423 0 0) 0x7f3f4c001500 con 0x7f3f540e51a0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f40002fe0 con 0x7f3f540e51a0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1758036274 0 0) 0x7f3f4c001460 con 0x7f3f540e51a0 2021-02-10T11:28:04.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f5407da20 msgr2=0x7f3f540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f5407da20 0x7f3f540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 >> v1:172.21.2.125:6789/0 conn(0x7f3f540e4da0 legacy=0x7f3f541b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f541b08e0 con 0x7f3f540e51a0 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3f4c001980 con 0x7f3f540e51a0 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 >> v1:172.21.2.117:6789/0 conn(0x7f3f540e51a0 legacy=0x7f3f540d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f40003a60 0x7f3f40005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f40006790 0x7f3f40008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f40009500 0x7f3f4000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4000c2d0 con 0x7f3f40009500 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4000c4f0 con 0x7f3f40003a60 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f40050720 con 0x7f3f40006790 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.398 7f3f5359e700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f40009500 0x7f3f4000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5259c700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f40006790 0x7f3f40008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f52d9d700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f40003a60 0x7f3f40005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 -- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f40006790 msgr2=0x7f3f40008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f40006790 0x7f3f40008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 -- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f40003a60 msgr2=0x7f3f40005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f40003a60 0x7f3f40005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 -- 172.21.2.117:0/3620436073 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40050980 con 0x7f3f40009500 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5259c700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f40006790 0x7f3f40008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f40009500 0x7f3f4000ba20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3f4c000ed0 tx=0x7f3f4c00a310).ready entity=mon.0 client_cookie=7b6267a8857ef703 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c0070c0 con 0x7f3f40009500 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4c0073c0 con 0x7f3f40009500 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f51d9b700 1 -- 172.21.2.117:0/3620436073 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4c00ab90 con 0x7f3f40009500 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f59119700 1 -- 172.21.2.117:0/3620436073 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f40009500 msgr2=0x7f3f4000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f59119700 1 --2- 172.21.2.117:0/3620436073 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f40009500 0x7f3f4000ba20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3f4c000ed0 tx=0x7f3f4c00a310).stop 2021-02-10T11:28:04.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.399 7f3f5359e700 1 -- 172.21.2.117:0/3620436073 reap_dead start 2021-02-10T11:28:04.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.400 7f3f59119700 1 -- 172.21.2.117:0/3620436073 shutdown_connections 2021-02-10T11:28:04.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.400 7f3f59119700 1 -- 172.21.2.117:0/3620436073 >> 172.21.2.117:0/3620436073 conn(0x7f3f540ce450 msgr2=0x7f3f540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:04.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.400 7f3f59119700 1 -- 172.21.2.117:0/3620436073 shutdown_connections 2021-02-10T11:28:04.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.400 7f3f59119700 1 -- 172.21.2.117:0/3620436073 wait complete. 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.451 7f3f59119700 1 Processor -- start 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 -- start start 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f540a46c0 0x7f3f540a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f540a50e0 0x7f3f540a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540c3040 con 0x7f3f541a46b0 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f54190a20 con 0x7f3f540a50e0 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f59119700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540a54e0 con 0x7f3f540a46c0 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43848/0 (socket says 172.21.2.117:43848) 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 -- 172.21.2.117:0/341170454 learned_addr learned my addr 172.21.2.117:0/341170454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 -- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f540a46c0 msgr2=0x7f3f540a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f540a46c0 0x7f3f540a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 -- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f540a50e0 msgr2=0x7f3f540a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.452 7f3f52d9d700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f540a50e0 0x7f3f540a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f52d9d700 1 -- 172.21.2.117:0/341170454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540a5700 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f52d9d700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f3f44006f20 tx=0x7f3f4400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f4400bd80 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44000c90 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f440022b0 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f59119700 1 -- 172.21.2.117:0/341170454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f540a5700 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.453 7f3f59119700 1 -- 172.21.2.117:0/341170454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f540a6990 con 0x7f3f541a46b0 2021-02-10T11:28:04.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.454 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f4400a000 con 0x7f3f541a46b0 2021-02-10T11:28:04.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.454 7f3f4b7fe700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 0x7f3f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:04.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.454 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f44032790 con 0x7f3f541a46b0 2021-02-10T11:28:04.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.454 7f3f5259c700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 0x7f3f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:04.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.455 7f3f5259c700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 0x7f3f3801abc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:04.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.498 7f3f59119700 1 -- 172.21.2.117:0/341170454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54077a70 con 0x7f3f541a46b0 2021-02-10T11:28:04.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.500 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f44000940 con 0x7f3f541a46b0 2021-02-10T11:28:04.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.630 7f3f59119700 1 -- 172.21.2.117:0/341170454 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802460", "target": ["mgr", ""]}) v1 -- 0x7f3f540a6990 con 0x7f3f380186c0 2021-02-10T11:28:04.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.634 7f3f4b7fe700 1 -- 172.21.2.117:0/341170454 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3f3c0054b0 con 0x7f3f380186c0 2021-02-10T11:28:04.630 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.639 7f3f59119700 1 -- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 msgr2=0x7f3f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.639 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 0x7f3f3801abc0 crc :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.639 7f3f59119700 1 -- 172.21.2.117:0/341170454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 msgr2=0x7f3f54190400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.639 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f3f44006f20 tx=0x7f3f4400b990).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 -- 172.21.2.117:0/341170454 shutdown_connections 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f540a50e0 0x7f3f540a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f540a46c0 0x7f3f540a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f380186c0 0x7f3f3801abc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 --2- 172.21.2.117:0/341170454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f541a46b0 0x7f3f54190400 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 -- 172.21.2.117:0/341170454 >> 172.21.2.117:0/341170454 conn(0x7f3f540b5140 msgr2=0x7f3f540a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:04.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 -- 172.21.2.117:0/341170454 shutdown_connections 2021-02-10T11:28:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:04.640 7f3f59119700 1 -- 172.21.2.117:0/341170454 wait complete. 2021-02-10T11:28:04.640 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802460 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6 2021-02-10T11:28:04.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:04.641 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:04.641 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:04.976 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:04.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:04.976 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:05.357 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:05.358 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:05.358 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239643 s, 43.8 MB/s 2021-02-10T11:28:05.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:05.359 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:05.359 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:05.694 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:05.695 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:05.695 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246698 s, 42.5 MB/s 2021-02-10T11:28:06.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:06.072 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:06.072 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:06.406 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:06.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:06.406 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:06.782 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:06.782 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:06.782 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240096 s, 43.7 MB/s 2021-02-10T11:28:06.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:06.783 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:06.783 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.119 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:07.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:07.119 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:07.490 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:07.490 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:07.490 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234726 s, 44.7 MB/s 2021-02-10T11:28:07.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:07.491 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:07.491 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:07.826 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:07.826 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:07.826 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:08.203 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:08.309 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:08.309 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262198 s, 40.0 MB/s 2021-02-10T11:28:08.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:08.310 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:08.311 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:08.657 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:08.657 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:08.657 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:09.035 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:09.035 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:09.035 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255621 s, 41.0 MB/s 2021-02-10T11:28:09.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:09.037 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:09.037 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:09.371 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:09.372 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:09.372 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:09.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:09.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:09.749 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258499 s, 40.6 MB/s 2021-02-10T11:28:09.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:09.750 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:09.750 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:10.085 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:10.085 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:10.085 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:10.460 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:10.460 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:10.460 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236743 s, 44.3 MB/s 2021-02-10T11:28:10.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:10.461 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:10.461 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:10.795 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:10.796 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:10.796 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:11.172 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253302 s, 41.4 MB/s 2021-02-10T11:28:11.173 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:11.173 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:11.173 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:11.507 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:11.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:11.508 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802460/3cb0e926-291b-4061-8175-59139d251df6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0264632 s, 39.6 MB/s 2021-02-10T11:28:11.886 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802461 2021-02-10T11:28:12.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 Processor -- start 2021-02-10T11:28:12.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 -- start start 2021-02-10T11:28:12.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa6b007da20 0x7fa6b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa6b00e4da0 0x7fa6b01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b0077a70 con 0x7fa6b00e51a0 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b00777d0 con 0x7fa6b00e4da0 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.308 7fa6b5812700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b007de20 con 0x7fa6b007da20 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6aed73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa6b00e4da0 0x7fa6b01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6afd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa6b00e51a0 0x7fa6b0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44104/0 (socket says 172.21.2.117:44104) 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6afd75700 1 -- 172.21.2.117:0/2440879145 learned_addr learned my addr 172.21.2.117:0/2440879145 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 --2- 172.21.2.117:0/2440879145 >> v2:172.21.2.125:3300/0 conn(0x7fa6b007da20 0x7fa6b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 -- 172.21.2.117:0/2440879145 >> v2:172.21.2.117:3300/0 conn(0x7fa6b00e4da0 msgr2=0x7fa6b01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 --2- 172.21.2.117:0/2440879145 >> v2:172.21.2.117:3300/0 conn(0x7fa6b00e4da0 0x7fa6b01b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 -- 172.21.2.117:0/2440879145 >> v1:172.21.2.117:6789/0 conn(0x7fa6b00e51a0 legacy=0x7fa6b0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6ae572700 1 -- 172.21.2.117:0/2440879145 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916038522 0 0) 0x7fa6a0001180 con 0x7fa6b00e51a0 2021-02-10T11:28:12.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 -- 172.21.2.117:0/2440879145 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b01b0710 con 0x7fa6b007da20 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6af574700 1 --2- 172.21.2.117:0/2440879145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b007da20 0x7fa6b00771b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa698006650 tx=0x7fa698009a80).ready entity=mon.1 client_cookie=8a587211bff24702 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.309 7fa6ae572700 1 -- 172.21.2.117:0/2440879145 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6980009b0 con 0x7fa6b007da20 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6ae572700 1 -- 172.21.2.117:0/2440879145 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698006740 con 0x7fa6b007da20 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6ae572700 1 -- 172.21.2.117:0/2440879145 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa69800c0c0 con 0x7fa6b007da20 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 -- 172.21.2.117:0/2440879145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b007da20 msgr2=0x7fa6b00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 --2- 172.21.2.117:0/2440879145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b007da20 0x7fa6b00771b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa698006650 tx=0x7fa698009a80).stop 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 -- 172.21.2.117:0/2440879145 shutdown_connections 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 --2- 172.21.2.117:0/2440879145 >> v2:172.21.2.117:3300/0 conn(0x7fa6b00e4da0 0x7fa6b01b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 --2- 172.21.2.117:0/2440879145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b007da20 0x7fa6b00771b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 -- 172.21.2.117:0/2440879145 >> 172.21.2.117:0/2440879145 conn(0x7fa6b00ce450 msgr2=0x7fa6b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 -- 172.21.2.117:0/2440879145 shutdown_connections 2021-02-10T11:28:12.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.310 7fa6b5812700 1 -- 172.21.2.117:0/2440879145 wait complete. 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 Processor -- start 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 -- start start 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6b01b5b50 0x7fa6b00b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b01b5f50 0x7fa6b00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00b8da0 con 0x7fa6b01b5130 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00a4140 con 0x7fa6b01b5f50 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6b5812700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00a4320 con 0x7fa6b01b5b50 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6af574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6af574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43856/0 (socket says 172.21.2.117:43856) 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.325 7fa6af574700 1 -- 172.21.2.117:0/565282753 learned_addr learned my addr 172.21.2.117:0/565282753 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6afd75700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b01b5f50 0x7fa6b00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6aed73700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6b01b5b50 0x7fa6b00b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 -- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6b01b5b50 msgr2=0x7fa6b00b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6b01b5b50 0x7fa6b00b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 -- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b01b5f50 msgr2=0x7fa6b00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b01b5f50 0x7fa6b00b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 -- 172.21.2.117:0/565282753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00b62b0 con 0x7fa6b01b5130 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6af574700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fa69800b660 tx=0x7fa69800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6980009b0 con 0x7fa6b01b5130 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698006760 con 0x7fa6b01b5130 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa698002030 con 0x7fa6b01b5130 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6b5812700 1 -- 172.21.2.117:0/565282753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00b62b0 con 0x7fa6b01b5130 2021-02-10T11:28:12.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.326 7fa6b5812700 1 -- 172.21.2.117:0/565282753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b00b6840 con 0x7fa6b01b5130 2021-02-10T11:28:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.327 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa698000c90 con 0x7fa6b01b5130 2021-02-10T11:28:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.327 7fa6a7fff700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 0x7fa69401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.327 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa698032700 con 0x7fa6b01b5130 2021-02-10T11:28:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.327 7fa6aed73700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 0x7fa69401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.328 7fa6aed73700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 0x7fa69401abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.358 7fa6b5812700 1 -- 172.21.2.117:0/565282753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b01b4f00 con 0x7fa6b01b5130 2021-02-10T11:28:12.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.361 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6980250a0 con 0x7fa6b01b5130 2021-02-10T11:28:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.495 7fa6b5812700 1 -- 172.21.2.117:0/565282753 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802461", "target": ["mgr", ""]}) v1 -- 0x7fa6b00b6840 con 0x7fa6940186c0 2021-02-10T11:28:12.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.518 7fa6a7fff700 1 -- 172.21.2.117:0/565282753 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa6a8006ce0 con 0x7fa6940186c0 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 -- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 msgr2=0x7fa69401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 0x7fa69401abc0 crc :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 -- 172.21.2.117:0/565282753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 msgr2=0x7fa6b01b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fa69800b660 tx=0x7fa69800ba00).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 -- 172.21.2.117:0/565282753 shutdown_connections 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6b01b5f50 0x7fa6b00b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6b01b5b50 0x7fa6b00b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6940186c0 0x7fa69401abc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 --2- 172.21.2.117:0/565282753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6b01b5130 0x7fa6b01b5530 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.527 7fa6b5812700 1 -- 172.21.2.117:0/565282753 >> 172.21.2.117:0/565282753 conn(0x7fa6b008bbc0 msgr2=0x7fa6b00c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:12.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.528 7fa6b5812700 1 -- 172.21.2.117:0/565282753 shutdown_connections 2021-02-10T11:28:12.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.528 7fa6b5812700 1 -- 172.21.2.117:0/565282753 wait complete. 2021-02-10T11:28:12.528 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802461 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 Processor -- start 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 -- start start 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f622c07da20 0x7f622c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c0d1d20 con 0x7f622c0e51a0 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f622c077a70 con 0x7f622c0e4da0 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.948 7f6231bc3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622c0777d0 con 0x7f622c07da20 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f622c0e51a0 0x7f622c0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44114/0 (socket says 172.21.2.117:44114) 2021-02-10T11:28:12.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622bfff700 1 -- 172.21.2.117:0/4020847235 learned_addr learned my addr 172.21.2.117:0/4020847235 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956144095 0 0) 0x7f6224001500 con 0x7f622c0e51a0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6214002fe0 con 0x7f622c0e51a0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622b7fe700 1 --2- 172.21.2.117:0/4020847235 >> v2:172.21.2.125:3300/0 conn(0x7f622c07da20 0x7f622c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2305427000 0 0) 0x7f6224001460 con 0x7f622c0e51a0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 >> v2:172.21.2.125:3300/0 conn(0x7f622c07da20 msgr2=0x7f622c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 --2- 172.21.2.117:0/4020847235 >> v2:172.21.2.125:3300/0 conn(0x7f622c07da20 0x7f622c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 >> v1:172.21.2.125:6789/0 conn(0x7f622c0e4da0 legacy=0x7f622c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.949 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c1b08e0 con 0x7f622c0e51a0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6224001980 con 0x7f622c0e51a0 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 >> v1:172.21.2.117:6789/0 conn(0x7f622c0e51a0 legacy=0x7f622c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6214003a40 0x7f6214005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 0x7f6214008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6214009590 0x7f621400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621400c300 con 0x7f6214006790 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621400c520 con 0x7f6214009590 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621400c740 con 0x7f6214003a40 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 0x7f6214008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 -- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6214003a40 msgr2=0x7f6214005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6214003a40 0x7f6214005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 -- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6214009590 msgr2=0x7f621400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6214009590 0x7f621400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 -- 172.21.2.117:0/4020847235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621400c9e0 con 0x7f6214006790 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622bfff700 1 -- 172.21.2.117:0/4020847235 reap_dead start 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622affd700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 0x7f6214008c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f622004aad0 tx=0x7f622004dad0).ready entity=mon.0 client_cookie=d873cb7352985867 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622004f2e0 con 0x7f6214006790 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622004a6c0 con 0x7f6214006790 2021-02-10T11:28:12.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f622a7fc700 1 -- 172.21.2.117:0/4020847235 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622004f8a0 con 0x7f6214006790 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f6231bc3700 1 -- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 msgr2=0x7f6214008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.950 7f6231bc3700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 0x7f6214008c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f622004aad0 tx=0x7f622004dad0).stop 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.951 7f6231bc3700 1 -- 172.21.2.117:0/4020847235 shutdown_connections 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.951 7f6231bc3700 1 --2- 172.21.2.117:0/4020847235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6214006790 0x7f6214008c90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.951 7f6231bc3700 1 -- 172.21.2.117:0/4020847235 >> 172.21.2.117:0/4020847235 conn(0x7f622c0ce450 msgr2=0x7f622c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.951 7f6231bc3700 1 -- 172.21.2.117:0/4020847235 shutdown_connections 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.951 7f6231bc3700 1 -- 172.21.2.117:0/4020847235 wait complete. 2021-02-10T11:28:12.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.966 7f6231bc3700 1 Processor -- start 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.966 7f6231bc3700 1 -- start start 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.966 7f6231bc3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f622c1a46b0 0x7f622c190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f6231bc3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f6231bc3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 0x7f622c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f6231bc3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0c3040 con 0x7f622c0a46c0 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f6231bc3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c190a20 con 0x7f622c0a50e0 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f6231bc3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622c0a54e0 con 0x7f622c1a46b0 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43876/0 (socket says 172.21.2.117:43876) 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 -- 172.21.2.117:0/978824355 learned_addr learned my addr 172.21.2.117:0/978824355 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622b7fe700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f622c1a46b0 0x7f622c190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622bfff700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 0x7f622c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 -- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f622c1a46b0 msgr2=0x7f622c190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f622c1a46b0 0x7f622c190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 -- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 msgr2=0x7f622c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 0x7f622c0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 -- 172.21.2.117:0/978824355 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622c0a5700 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622bfff700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 0x7f622c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.967 7f622affd700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f622004aa00 tx=0x7f622004f9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622004fdb0 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622004abf0 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622004a2a0 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6231bc3700 1 -- 172.21.2.117:0/978824355 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c0a5700 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6231bc3700 1 -- 172.21.2.117:0/978824355 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c0a6990 con 0x7f622c0a46c0 2021-02-10T11:28:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f622004a030 con 0x7f622c0a46c0 2021-02-10T11:28:12.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 0x7f620c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:12.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.968 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6220076750 con 0x7f622c0a46c0 2021-02-10T11:28:12.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.969 7f622b7fe700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 0x7f620c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:12.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.970 7f622b7fe700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 0x7f620c01abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:12.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:12.998 7f6231bc3700 1 -- 172.21.2.117:0/978824355 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c077a70 con 0x7f622c0a46c0 2021-02-10T11:28:12.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.001 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f622004d0d0 con 0x7f622c0a46c0 2021-02-10T11:28:13.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.128 7f6231bc3700 1 -- 172.21.2.117:0/978824355 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802461", "target": ["mgr", ""]}) v1 -- 0x7f622c0a54e0 con 0x7f620c0186c0 2021-02-10T11:28:13.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.132 7f6213fff700 1 -- 172.21.2.117:0/978824355 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f621c000960 con 0x7f620c0186c0 2021-02-10T11:28:13.124 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b 2021-02-10T11:28:13.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 msgr2=0x7f620c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:13.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 0x7f620c01abc0 crc :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:13.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 msgr2=0x7f622c0a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:13.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f622004aa00 tx=0x7f622004f9c0).stop 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 shutdown_connections 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f620c0186c0 0x7f620c01abc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f622c0a46c0 0x7f622c0a4ac0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f622c0a50e0 0x7f622c0a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 --2- 172.21.2.117:0/978824355 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f622c1a46b0 0x7f622c190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 >> 172.21.2.117:0/978824355 conn(0x7f622c0b5140 msgr2=0x7f622c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 shutdown_connections 2021-02-10T11:28:13.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:13.134 7f6231bc3700 1 -- 172.21.2.117:0/978824355 wait complete. 2021-02-10T11:28:13.134 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802461 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b 2021-02-10T11:28:13.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:13.134 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:13.134 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:13.469 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:13.469 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:13.469 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:13.849 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:13.849 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:13.849 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02247 s, 46.7 MB/s 2021-02-10T11:28:13.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:13.850 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:13.850 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:14.185 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:14.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:14.185 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:14.558 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:14.559 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:14.559 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229926 s, 45.6 MB/s 2021-02-10T11:28:14.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:14.560 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:14.560 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:14.894 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:14.894 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:14.895 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:15.268 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:15.269 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:15.269 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243713 s, 43.0 MB/s 2021-02-10T11:28:15.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:15.270 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:15.270 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:15.605 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:15.605 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:15.605 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:15.979 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:15.979 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:15.979 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232733 s, 45.1 MB/s 2021-02-10T11:28:15.980 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:15.981 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:15.981 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:16.316 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:16.316 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:16.317 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:16.691 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:16.691 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:16.692 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023847 s, 44.0 MB/s 2021-02-10T11:28:16.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:16.693 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:16.693 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:17.028 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:17.028 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:17.028 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:17.401 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:17.401 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:17.401 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231756 s, 45.2 MB/s 2021-02-10T11:28:17.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:17.402 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:17.402 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:17.737 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:17.737 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:17.737 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254968 s, 41.1 MB/s 2021-02-10T11:28:18.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:18.111 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:18.111 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:18.445 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:18.446 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:18.446 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:18.821 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:18.821 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:18.821 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231293 s, 45.3 MB/s 2021-02-10T11:28:18.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:18.822 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:18.822 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:19.157 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:19.158 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:19.158 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:19.532 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:19.532 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:19.532 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234929 s, 44.6 MB/s 2021-02-10T11:28:19.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:19.533 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:19.533 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:19.869 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:19.869 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:19.869 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802461/02a78f14-5903-46f2-8a26-cd512fbe011b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:20.246 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:20.246 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:20.246 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257993 s, 40.6 MB/s 2021-02-10T11:28:20.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802462 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.669 7fe090aaf700 1 Processor -- start 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.669 7fe090aaf700 1 -- start start 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe090aaf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe090aaf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe08c0e51a0 0x7fe08c0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe090aaf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c077a70 con 0x7fe08c0e4da0 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe090aaf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08c0e5c20 con 0x7fe08c0e51a0 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe090aaf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0777d0 con 0x7fe08c07da20 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:20.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:41560/0 (socket says 172.21.2.117:41560) 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 -- 172.21.2.117:0/202188704 learned_addr learned my addr 172.21.2.117:0/202188704 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08b05f700 1 --2- 172.21.2.117:0/202188704 >> v2:172.21.2.125:3300/0 conn(0x7fe08c0e51a0 0x7fe08c0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 -- 172.21.2.117:0/202188704 >> v2:172.21.2.125:3300/0 conn(0x7fe08c0e51a0 msgr2=0x7fe08c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 --2- 172.21.2.117:0/202188704 >> v2:172.21.2.125:3300/0 conn(0x7fe08c0e51a0 0x7fe08c0d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 -- 172.21.2.117:0/202188704 >> v1:172.21.2.125:6789/0 conn(0x7fe08c0e4da0 legacy=0x7fe08c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.670 7fe08a85e700 1 -- 172.21.2.117:0/202188704 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c1b08e0 con 0x7fe08c07da20 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe08a85e700 1 --2- 172.21.2.117:0/202188704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe080006700 tx=0x7fe080009b80).ready entity=mon.2 client_cookie=5e19ed9678141820 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe08985c700 1 -- 172.21.2.117:0/202188704 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08000c0c0 con 0x7fe08c07da20 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe08985c700 1 -- 172.21.2.117:0/202188704 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08000c440 con 0x7fe08c07da20 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe08985c700 1 -- 172.21.2.117:0/202188704 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe08000c960 con 0x7fe08c07da20 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe090aaf700 1 -- 172.21.2.117:0/202188704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 msgr2=0x7fe08c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.671 7fe090aaf700 1 --2- 172.21.2.117:0/202188704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe080006700 tx=0x7fe080009b80).stop 2021-02-10T11:28:20.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 -- 172.21.2.117:0/202188704 shutdown_connections 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 --2- 172.21.2.117:0/202188704 >> v2:172.21.2.125:3300/0 conn(0x7fe08c0e51a0 0x7fe08c0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 --2- 172.21.2.117:0/202188704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c07da20 0x7fe08c0771b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 -- 172.21.2.117:0/202188704 >> 172.21.2.117:0/202188704 conn(0x7fe08c0ce450 msgr2=0x7fe08c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 -- 172.21.2.117:0/202188704 shutdown_connections 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.672 7fe090aaf700 1 -- 172.21.2.117:0/202188704 wait complete. 2021-02-10T11:28:20.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.686 7fe090aaf700 1 Processor -- start 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.686 7fe090aaf700 1 -- start start 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c0b87c0 0x7fe08c1a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe08c1a4660 0x7fe08c0a9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0aa990 con 0x7fe08c0aa450 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0c26b0 con 0x7fe08c1a4660 2021-02-10T11:28:20.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe090aaf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0c2b20 con 0x7fe08c0b87c0 2021-02-10T11:28:20.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:20.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43890/0 (socket says 172.21.2.117:43890) 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 -- 172.21.2.117:0/3309012353 learned_addr learned my addr 172.21.2.117:0/3309012353 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 -- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c0b87c0 msgr2=0x7fe08c1a4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c0b87c0 0x7fe08c1a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 -- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe08c1a4660 msgr2=0x7fe08c0a9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe08c1a4660 0x7fe08c0a9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 -- 172.21.2.117:0/3309012353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c08c970 con 0x7fe08c0aa450 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe08b05f700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fe07c00b580 tx=0x7fe07c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.687 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe07c006370 con 0x7fe08c0aa450 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c00bd10 con 0x7fe08c0aa450 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe07c0024d0 con 0x7fe08c0aa450 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c08c970 con 0x7fe08c0aa450 2021-02-10T11:28:20.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c08c970 con 0x7fe08c0aa450 2021-02-10T11:28:20.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe07c005330 con 0x7fe08c0aa450 2021-02-10T11:28:20.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe07b7fe700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 0x7fe07001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:20.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe07c0325e0 con 0x7fe08c0aa450 2021-02-10T11:28:20.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.688 7fe08a85e700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 0x7fe07001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:20.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.689 7fe08a85e700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 0x7fe07001abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:20.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.720 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c103280 con 0x7fe08c0aa450 2021-02-10T11:28:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.722 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe07c0067f0 con 0x7fe08c0aa450 2021-02-10T11:28:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.848 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802462", "target": ["mgr", ""]}) v1 -- 0x7fe08c103280 con 0x7fe0700186c0 2021-02-10T11:28:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.875 7fe07b7fe700 1 -- 172.21.2.117:0/3309012353 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe080000990 con 0x7fe0700186c0 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 msgr2=0x7fe07001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 0x7fe07001abc0 crc :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 msgr2=0x7fe08c08c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7fe07c00b580 tx=0x7fe07c00b920).stop 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 shutdown_connections 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0700186c0 0x7fe07001abc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe08c0aa450 0x7fe08c08c350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:20.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe08c1a4660 0x7fe08c0a9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:20.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 --2- 172.21.2.117:0/3309012353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe08c0b87c0 0x7fe08c1a4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:20.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 >> 172.21.2.117:0/3309012353 conn(0x7fe08c0aac00 msgr2=0x7fe08c0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:20.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 shutdown_connections 2021-02-10T11:28:20.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:20.882 7fe090aaf700 1 -- 172.21.2.117:0/3309012353 wait complete. 2021-02-10T11:28:20.882 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802462 2021-02-10T11:28:21.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.304 7f48a2549700 1 Processor -- start 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.304 7f48a2549700 1 -- start start 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f48a2549700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f489c079000 0x7f489c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f48a2549700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f489c1b00f0 0x7f489c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f48a2549700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c077a70 con 0x7f489c07da20 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f48a2549700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c0e5c20 con 0x7f489c079000 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f48a2549700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489c0e53c0 con 0x7f489c1b00f0 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f489c079000 0x7f489c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f489c079000 0x7f489c0e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43894/0 (socket says 172.21.2.117:43894) 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 -- 172.21.2.117:0/2176810317 learned_addr learned my addr 172.21.2.117:0/2176810317 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 -- 172.21.2.117:0/2176810317 >> v2:172.21.2.125:3300/0 conn(0x7f489c1b00f0 msgr2=0x7f489c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 --2- 172.21.2.117:0/2176810317 >> v2:172.21.2.125:3300/0 conn(0x7f489c1b00f0 0x7f489c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 -- 172.21.2.117:0/2176810317 >> v1:172.21.2.125:6789/0 conn(0x7f489c07da20 legacy=0x7f489c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 -- 172.21.2.117:0/2176810317 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c079400 con 0x7f489c079000 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.305 7f489b7fe700 1 --2- 172.21.2.117:0/2176810317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c079000 0x7f489c0e4da0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f4888006700 tx=0x7f4888009b80).ready entity=mon.0 client_cookie=36115ed3848c5e65 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f489affd700 1 -- 172.21.2.117:0/2176810317 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488800c0c0 con 0x7f489c079000 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f489affd700 1 -- 172.21.2.117:0/2176810317 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488800c340 con 0x7f489c079000 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f489affd700 1 -- 172.21.2.117:0/2176810317 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488800c860 con 0x7f489c079000 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 -- 172.21.2.117:0/2176810317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c079000 msgr2=0x7f489c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 --2- 172.21.2.117:0/2176810317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c079000 0x7f489c0e4da0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f4888006700 tx=0x7f4888009b80).stop 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 -- 172.21.2.117:0/2176810317 shutdown_connections 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 --2- 172.21.2.117:0/2176810317 >> v2:172.21.2.125:3300/0 conn(0x7f489c1b00f0 0x7f489c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 --2- 172.21.2.117:0/2176810317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c079000 0x7f489c0e4da0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:21.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.306 7f48a2549700 1 -- 172.21.2.117:0/2176810317 >> 172.21.2.117:0/2176810317 conn(0x7f489c0ce450 msgr2=0x7f489c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:21.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.307 7f48a2549700 1 -- 172.21.2.117:0/2176810317 shutdown_connections 2021-02-10T11:28:21.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.307 7f48a2549700 1 -- 172.21.2.117:0/2176810317 wait complete. 2021-02-10T11:28:21.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.321 7f48a2549700 1 Processor -- start 2021-02-10T11:28:21.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.321 7f48a2549700 1 -- start start 2021-02-10T11:28:21.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c094d90 0x7f489c095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c1a7290 0x7f489c1a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0a9af0 con 0x7f489c095a60 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0957b0 con 0x7f489c1a7290 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f48a2549700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c1a7cb0 con 0x7f489c094d90 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43900/0 (socket says 172.21.2.117:43900) 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 -- 172.21.2.117:0/3484504544 learned_addr learned my addr 172.21.2.117:0/3484504544 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 -- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c094d90 msgr2=0x7f489c095190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c094d90 0x7f489c095190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 -- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c1a7290 msgr2=0x7f489c1a7690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c1a7290 0x7f489c1a7690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 -- 172.21.2.117:0/3484504544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c1a7ed0 con 0x7f489c095a60 2021-02-10T11:28:21.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f489bfff700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f488c00b5b0 tx=0x7f488c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.322 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c00bd40 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c006020 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c0054a0 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f48a2549700 1 -- 172.21.2.117:0/3484504544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c1a7ed0 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f48a2549700 1 -- 172.21.2.117:0/3484504544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0aa360 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f488c006200 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f4898ff9700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.323 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f488c032710 con 0x7f489c095a60 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.324 7f489b7fe700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:21.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.325 7f489b7fe700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:21.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.355 7f48a2549700 1 -- 172.21.2.117:0/3484504544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c1032d0 con 0x7f489c095a60 2021-02-10T11:28:21.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.357 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f488c02cf40 con 0x7f489c095a60 2021-02-10T11:28:21.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.483 7f48a2549700 1 -- 172.21.2.117:0/3484504544 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802462", "target": ["mgr", ""]}) v1 -- 0x7f489c0aa360 con 0x7f48780186c0 2021-02-10T11:28:21.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.486 7f4898ff9700 1 -- 172.21.2.117:0/3484504544 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4888006500 con 0x7f48780186c0 2021-02-10T11:28:21.483 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175 2021-02-10T11:28:21.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 -- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 msgr2=0x7f487801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:21.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 0x7f487801abc0 crc :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 -- 172.21.2.117:0/3484504544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 msgr2=0x7f489c190560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f488c00b5b0 tx=0x7f488c00b950).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 -- 172.21.2.117:0/3484504544 shutdown_connections 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c1a7290 0x7f489c1a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c094d90 0x7f489c095190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48780186c0 0x7f487801abc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 --2- 172.21.2.117:0/3484504544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c095a60 0x7f489c190560 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.492 7f48a2549700 1 -- 172.21.2.117:0/3484504544 >> 172.21.2.117:0/3484504544 conn(0x7f489c0b56c0 msgr2=0x7f489c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.493 7f48a2549700 1 -- 172.21.2.117:0/3484504544 shutdown_connections 2021-02-10T11:28:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:21.493 7f48a2549700 1 -- 172.21.2.117:0/3484504544 wait complete. 2021-02-10T11:28:21.492 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802462 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175 2021-02-10T11:28:21.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:21.492 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:21.492 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:21.828 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:21.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:21.828 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:22.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:22.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:22.207 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223448 s, 46.9 MB/s 2021-02-10T11:28:22.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:22.208 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:22.208 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:22.545 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:22.545 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:22.545 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:22.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:22.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:22.918 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241003 s, 43.5 MB/s 2021-02-10T11:28:22.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:22.919 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:22.919 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:23.255 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:23.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:23.255 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:23.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:23.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229713 s, 45.6 MB/s 2021-02-10T11:28:23.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:23.631 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:23.631 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:23.972 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:23.973 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:23.973 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:24.349 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:24.349 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:24.349 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244014 s, 43.0 MB/s 2021-02-10T11:28:24.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:24.350 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:24.350 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:24.685 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:24.685 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:24.685 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:25.061 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:25.061 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:25.061 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244108 s, 43.0 MB/s 2021-02-10T11:28:25.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:25.062 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:25.062 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:25.396 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:25.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:25.397 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243065 s, 43.1 MB/s 2021-02-10T11:28:25.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:25.773 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:25.773 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:26.106 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:26.107 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:26.107 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:26.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:26.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:26.482 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248648 s, 42.2 MB/s 2021-02-10T11:28:26.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:26.484 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:26.484 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:26.607 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:26.610 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:26.819 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:26.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:26.820 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:27.194 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:27.194 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:27.194 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235302 s, 44.6 MB/s 2021-02-10T11:28:27.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:27.195 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:27.196 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:27.531 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:27.531 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:27.531 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:27.902 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:27.902 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:27.902 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239724 s, 43.7 MB/s 2021-02-10T11:28:27.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:27.904 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:27.904 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:28.239 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:28.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:28.240 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802462/8557cb5c-3e35-455a-a45e-86d598ddb175/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:28.614 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:28.614 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:28.614 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239692 s, 43.7 MB/s 2021-02-10T11:28:28.615 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802463 2021-02-10T11:28:29.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.037 7fe7707dd700 1 Processor -- start 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.037 7fe7707dd700 1 -- start start 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe7707dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe768079000 0x7fe7680e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe7707dd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe7681b00f0 0x7fe7681b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe7707dd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe768077a70 con 0x7fe76807da20 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe7707dd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7680e5c20 con 0x7fe768079000 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe7707dd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7680e53c0 con 0x7fe7681b00f0 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe768079000 0x7fe7680e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe768079000 0x7fe7680e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43908/0 (socket says 172.21.2.117:43908) 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 -- 172.21.2.117:0/4088658033 learned_addr learned my addr 172.21.2.117:0/4088658033 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 -- 172.21.2.117:0/4088658033 >> v2:172.21.2.125:3300/0 conn(0x7fe7681b00f0 msgr2=0x7fe7681b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 --2- 172.21.2.117:0/4088658033 >> v2:172.21.2.125:3300/0 conn(0x7fe7681b00f0 0x7fe7681b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 -- 172.21.2.117:0/4088658033 >> v1:172.21.2.125:6789/0 conn(0x7fe76807da20 legacy=0x7fe7680771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 -- 172.21.2.117:0/4088658033 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768079400 con 0x7fe768079000 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.038 7fe76ddf8700 1 --2- 172.21.2.117:0/4088658033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768079000 0x7fe7680e4da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fe758006700 tx=0x7fe758009b80).ready entity=mon.0 client_cookie=2c3896cc3d86aba0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe76d5f7700 1 -- 172.21.2.117:0/4088658033 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75800c0c0 con 0x7fe768079000 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe76d5f7700 1 -- 172.21.2.117:0/4088658033 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75800c440 con 0x7fe768079000 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe76d5f7700 1 -- 172.21.2.117:0/4088658033 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75800c920 con 0x7fe768079000 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 -- 172.21.2.117:0/4088658033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768079000 msgr2=0x7fe7680e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 --2- 172.21.2.117:0/4088658033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768079000 0x7fe7680e4da0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fe758006700 tx=0x7fe758009b80).stop 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 -- 172.21.2.117:0/4088658033 shutdown_connections 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 --2- 172.21.2.117:0/4088658033 >> v2:172.21.2.125:3300/0 conn(0x7fe7681b00f0 0x7fe7681b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 --2- 172.21.2.117:0/4088658033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768079000 0x7fe7680e4da0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.039 7fe7707dd700 1 -- 172.21.2.117:0/4088658033 >> 172.21.2.117:0/4088658033 conn(0x7fe7680ce450 msgr2=0x7fe7680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.040 7fe7707dd700 1 -- 172.21.2.117:0/4088658033 shutdown_connections 2021-02-10T11:28:29.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.040 7fe7707dd700 1 -- 172.21.2.117:0/4088658033 wait complete. 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 Processor -- start 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 -- start start 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7681034d0 0x7fe768104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe768105160 0x7fe7680aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe768103c20 con 0x7fe768104740 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe768103910 con 0x7fe7681034d0 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.054 7fe7707dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680b7e00 con 0x7fe768105160 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43912/0 (socket says 172.21.2.117:43912) 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 -- 172.21.2.117:0/602563816 learned_addr learned my addr 172.21.2.117:0/602563816 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 -- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe768105160 msgr2=0x7fe7680aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe768105160 0x7fe7680aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 -- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7681034d0 msgr2=0x7fe768104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7681034d0 0x7fe768104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 -- 172.21.2.117:0/602563816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680ab330 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe76ddf8700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fe75800b5e0 tx=0x7fe75800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe75800bd70 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.055 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758002030 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe758002550 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe7707dd700 1 -- 172.21.2.117:0/602563816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7680ab330 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe7707dd700 1 -- 172.21.2.117:0/602563816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7680ab8c0 con 0x7fe768104740 2021-02-10T11:28:29.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe758000c90 con 0x7fe768104740 2021-02-10T11:28:29.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe75effd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 0x7fe75001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.056 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe758032790 con 0x7fe768104740 2021-02-10T11:28:29.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.057 7fe76e5f9700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 0x7fe75001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.057 7fe76e5f9700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 0x7fe75001abc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.088 7fe7707dd700 1 -- 172.21.2.117:0/602563816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7680b7de0 con 0x7fe768104740 2021-02-10T11:28:29.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.090 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe758030df0 con 0x7fe768104740 2021-02-10T11:28:29.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.215 7fe7707dd700 1 -- 172.21.2.117:0/602563816 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802463", "target": ["mgr", ""]}) v1 -- 0x7fe7680b7de0 con 0x7fe7500186c0 2021-02-10T11:28:29.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.241 7fe75effd700 1 -- 172.21.2.117:0/602563816 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe764006a10 con 0x7fe7500186c0 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 msgr2=0x7fe75001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 0x7fe75001abc0 crc :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 msgr2=0x7fe768104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fe75800b5e0 tx=0x7fe75800b980).stop 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 shutdown_connections 2021-02-10T11:28:29.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7500186c0 0x7fe75001abc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe768104740 0x7fe768104b40 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe768105160 0x7fe7680aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 --2- 172.21.2.117:0/602563816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7681034d0 0x7fe768104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 >> 172.21.2.117:0/602563816 conn(0x7fe7680a1140 msgr2=0x7fe7680ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 shutdown_connections 2021-02-10T11:28:29.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.248 7fe7707dd700 1 -- 172.21.2.117:0/602563816 wait complete. 2021-02-10T11:28:29.248 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802463 2021-02-10T11:28:29.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.670 7ffb6b5f1700 1 Processor -- start 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.670 7ffb6b5f1700 1 -- start start 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb6b5f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb64079000 0x7ffb640e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb6b5f1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640d1d20 con 0x7ffb641b00f0 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb6b5f1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb64077a70 con 0x7ffb6407da20 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb6b5f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640e5430 con 0x7ffb64079000 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb69c0e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ffb641b00f0 0x7ffb640d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44174/0 (socket says 172.21.2.117:44174) 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb69c0e700 1 -- 172.21.2.117:0/2570150931 learned_addr learned my addr 172.21.2.117:0/2570150931 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654791796 0 0) 0x7ffb5c001500 con 0x7ffb641b00f0 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50002fe0 con 0x7ffb641b00f0 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb68c0c700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb64079000 0x7ffb640e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2824989026 0 0) 0x7ffb5c001460 con 0x7ffb641b00f0 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.671 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb64079000 msgr2=0x7ffb640e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb64079000 0x7ffb640e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 >> v1:172.21.2.125:6789/0 conn(0x7ffb6407da20 legacy=0x7ffb640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64079400 con 0x7ffb641b00f0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ffb5c001980 con 0x7ffb641b00f0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 >> v1:172.21.2.117:6789/0 conn(0x7ffb641b00f0 legacy=0x7ffb640d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb50003ac0 0x7ffb50005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb50006750 0x7ffb50008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb50009510 0x7ffb5000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c2e0 con 0x7ffb50009510 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c500 con 0x7ffb50006750 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c720 con 0x7ffb50003ac0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb50009510 0x7ffb5000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb68c0c700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb50006750 0x7ffb50008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb6940d700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb50003ac0 0x7ffb50005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 -- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb50003ac0 msgr2=0x7ffb50005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb50003ac0 0x7ffb50005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 -- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb50006750 msgr2=0x7ffb50008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb50006750 0x7ffb50008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 -- 172.21.2.117:0/2570150931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5000c9c0 con 0x7ffb50009510 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb69c0e700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb50009510 0x7ffb5000ba30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7ffb5c001140 tx=0x7ffb5c04e320).ready entity=mon.0 client_cookie=45fd7e7ff3035f64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5c000da0 con 0x7ffb50009510 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.672 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c0070d0 con 0x7ffb50009510 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb57fff700 1 -- 172.21.2.117:0/2570150931 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5c004720 con 0x7ffb50009510 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 -- 172.21.2.117:0/2570150931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb50009510 msgr2=0x7ffb5000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 --2- 172.21.2.117:0/2570150931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb50009510 0x7ffb5000ba30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7ffb5c001140 tx=0x7ffb5c04e320).stop 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb69c0e700 1 -- 172.21.2.117:0/2570150931 reap_dead start 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 -- 172.21.2.117:0/2570150931 shutdown_connections 2021-02-10T11:28:29.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 -- 172.21.2.117:0/2570150931 >> 172.21.2.117:0/2570150931 conn(0x7ffb640ce450 msgr2=0x7ffb640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:29.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 -- 172.21.2.117:0/2570150931 shutdown_connections 2021-02-10T11:28:29.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.673 7ffb6b5f1700 1 -- 172.21.2.117:0/2570150931 wait complete. 2021-02-10T11:28:29.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 Processor -- start 2021-02-10T11:28:29.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 -- start start 2021-02-10T11:28:29.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb640a51f0 0x7ffb640a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb640b77d0 0x7ffb640b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb64189970 con 0x7ffb640a47d0 2021-02-10T11:28:29.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb64190aa0 con 0x7ffb640a51f0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6b5f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640b81f0 con 0x7ffb640b77d0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.688 7ffb6940d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43934/0 (socket says 172.21.2.117:43934) 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 -- 172.21.2.117:0/3936094652 learned_addr learned my addr 172.21.2.117:0/3936094652 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 -- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb640b77d0 msgr2=0x7ffb640b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb640b77d0 0x7ffb640b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 -- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb640a51f0 msgr2=0x7ffb640a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb640a51f0 0x7ffb640a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 -- 172.21.2.117:0/3936094652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640b8410 con 0x7ffb640a47d0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6940d700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ffb58005a20 tx=0x7ffb5800a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5800ad70 con 0x7ffb640a47d0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800b070 con 0x7ffb640a47d0 2021-02-10T11:28:29.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb640b8410 con 0x7ffb640a47d0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb640b8410 con 0x7ffb640a47d0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.689 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb5800b590 con 0x7ffb640a47d0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.690 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb5800ad70 con 0x7ffb640a47d0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.690 7ffb55ffb700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 0x7ffb4401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.690 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffb58032950 con 0x7ffb640a47d0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.690 7ffb68c0c700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 0x7ffb4401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:29.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.691 7ffb68c0c700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 0x7ffb4401ac10 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:29.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.721 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb64077a70 con 0x7ffb640a47d0 2021-02-10T11:28:29.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.723 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb5800b070 con 0x7ffb640a47d0 2021-02-10T11:28:29.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.852 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802463", "target": ["mgr", ""]}) v1 -- 0x7ffb64077a70 con 0x7ffb44018730 2021-02-10T11:28:29.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.855 7ffb55ffb700 1 -- 172.21.2.117:0/3936094652 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ffb60007df0 con 0x7ffb44018730 2021-02-10T11:28:29.847 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75 2021-02-10T11:28:29.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 msgr2=0x7ffb4401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 0x7ffb4401ac10 crc :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 msgr2=0x7ffb640a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:29.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7ffb58005a20 tx=0x7ffb5800a980).stop 2021-02-10T11:28:29.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 shutdown_connections 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb640b77d0 0x7ffb640b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb640a51f0 0x7ffb640a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb44018730 0x7ffb4401ac10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 --2- 172.21.2.117:0/3936094652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb640a47d0 0x7ffb640a4bd0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 >> 172.21.2.117:0/3936094652 conn(0x7ffb640b5140 msgr2=0x7ffb640a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.856 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 shutdown_connections 2021-02-10T11:28:29.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:29.857 7ffb6b5f1700 1 -- 172.21.2.117:0/3936094652 wait complete. 2021-02-10T11:28:29.856 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802463 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75 2021-02-10T11:28:29.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:29.856 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:29.856 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:30.192 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:30.193 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:30.193 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:30.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:30.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:30.575 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246449 s, 42.5 MB/s 2021-02-10T11:28:30.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:30.576 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:30.576 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:30.911 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:30.911 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:30.911 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233744 s, 44.9 MB/s 2021-02-10T11:28:31.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:31.286 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:31.286 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:31.621 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:31.622 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:31.622 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:31.996 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:31.996 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:31.996 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226659 s, 46.3 MB/s 2021-02-10T11:28:31.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:31.997 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:31.997 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:32.332 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:32.333 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:32.333 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:32.705 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:32.705 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:32.705 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248434 s, 42.2 MB/s 2021-02-10T11:28:32.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:32.706 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:32.706 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:33.041 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:33.042 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:33.042 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:33.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:33.419 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242715 s, 43.2 MB/s 2021-02-10T11:28:33.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:33.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:33.419 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:33.753 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:33.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:33.754 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:34.129 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:34.130 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:34.130 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254889 s, 41.1 MB/s 2021-02-10T11:28:34.131 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:34.131 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:34.131 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:34.479 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:34.479 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:34.479 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:34.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:34.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231023 s, 45.4 MB/s 2021-02-10T11:28:34.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:34.855 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:34.855 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:35.190 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:35.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:35.191 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:35.565 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:35.565 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:35.565 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228083 s, 46.0 MB/s 2021-02-10T11:28:35.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:35.566 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:35.566 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:35.902 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:35.903 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:35.903 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02332 s, 45.0 MB/s 2021-02-10T11:28:36.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:36.279 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:36.279 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:36.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:36.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:36.614 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802463/3c1fbf0a-cc4c-4fdb-9c51-f9db15cdfc75/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:36.988 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:36.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:36.989 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240562 s, 43.6 MB/s 2021-02-10T11:28:36.990 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802464 2021-02-10T11:28:37.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.410 7f99ff34d700 1 Processor -- start 2021-02-10T11:28:37.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.410 7f99ff34d700 1 -- start start 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99ff34d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f99f807da20 0x7f99f80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99ff34d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 0x7f99f8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99ff34d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f8077a70 con 0x7f99f80e4da0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99ff34d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80777d0 con 0x7f99f807da20 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99ff34d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81b08e0 con 0x7f99f80e51a0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fc968700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f99f80e4da0 0x7f99f81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44190/0 (socket says 172.21.2.117:44190) 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fc968700 1 -- 172.21.2.117:0/1385870398 learned_addr learned my addr 172.21.2.117:0/1385870398 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd96a700 1 --2- 172.21.2.117:0/1385870398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 0x7f99f8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd169700 1 --2- 172.21.2.117:0/1385870398 >> v2:172.21.2.125:3300/0 conn(0x7f99f807da20 0x7f99f80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99effff700 1 -- 172.21.2.117:0/1385870398 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461343682 0 0) 0x7f99e8000ce0 con 0x7f99f80e4da0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99effff700 1 -- 172.21.2.117:0/1385870398 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4002fe0 con 0x7f99f80e4da0 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd96a700 1 -- 172.21.2.117:0/1385870398 >> v2:172.21.2.125:3300/0 conn(0x7f99f807da20 msgr2=0x7f99f80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:37.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd96a700 1 --2- 172.21.2.117:0/1385870398 >> v2:172.21.2.125:3300/0 conn(0x7f99f807da20 0x7f99f80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd96a700 1 -- 172.21.2.117:0/1385870398 >> v1:172.21.2.117:6789/0 conn(0x7f99f80e4da0 legacy=0x7f99f81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.411 7f99fd96a700 1 -- 172.21.2.117:0/1385870398 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f807de20 con 0x7f99f80e51a0 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99fd96a700 1 --2- 172.21.2.117:0/1385870398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 0x7f99f8079000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f99f0006e50 tx=0x7f99f0009e70).ready entity=mon.2 client_cookie=3a77e86d4299f76d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99effff700 1 -- 172.21.2.117:0/1385870398 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f000b510 con 0x7f99f80e51a0 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99effff700 1 -- 172.21.2.117:0/1385870398 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f0006790 con 0x7f99f80e51a0 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99effff700 1 -- 172.21.2.117:0/1385870398 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f000b940 con 0x7f99f80e51a0 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 -- 172.21.2.117:0/1385870398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 msgr2=0x7f99f8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 --2- 172.21.2.117:0/1385870398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 0x7f99f8079000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f99f0006e50 tx=0x7f99f0009e70).stop 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 -- 172.21.2.117:0/1385870398 shutdown_connections 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 --2- 172.21.2.117:0/1385870398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f80e51a0 0x7f99f8079000 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 --2- 172.21.2.117:0/1385870398 >> v2:172.21.2.125:3300/0 conn(0x7f99f807da20 0x7f99f80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.412 7f99ff34d700 1 -- 172.21.2.117:0/1385870398 >> 172.21.2.117:0/1385870398 conn(0x7f99f80ce450 msgr2=0x7f99f80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:37.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.413 7f99ff34d700 1 -- 172.21.2.117:0/1385870398 shutdown_connections 2021-02-10T11:28:37.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.413 7f99ff34d700 1 -- 172.21.2.117:0/1385870398 wait complete. 2021-02-10T11:28:37.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.428 7f99ff34d700 1 Processor -- start 2021-02-10T11:28:37.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.428 7f99ff34d700 1 -- start start 2021-02-10T11:28:37.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 0x7f99f8188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 0x7f99f8090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81067f0 con 0x7f99f81a0250 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8107350 con 0x7f99f8090420 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99ff34d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f81042d0 con 0x7f99f8187e20 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99fd169700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99fd169700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43944/0 (socket says 172.21.2.117:43944) 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99fd169700 1 -- 172.21.2.117:0/2721439669 learned_addr learned my addr 172.21.2.117:0/2721439669 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99fd96a700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 0x7f99f8090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.429 7f99fc968700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 0x7f99f8188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 -- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 msgr2=0x7f99f8188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 0x7f99f8188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 -- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 msgr2=0x7f99f8090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 0x7f99f8090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 -- 172.21.2.117:0/2721439669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8090e40 con 0x7f99f81a0250 2021-02-10T11:28:37.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd96a700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 0x7f99f8090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fc968700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 0x7f99f8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99fd169700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f99f4006f60 tx=0x7f99f400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f40009b0 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4006bb0 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99f4002030 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f8090e40 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.430 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f8090e40 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.431 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99f4000c90 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.431 7f99edffb700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 0x7f99d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.431 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f99f40327d0 con 0x7f99f81a0250 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.431 7f99fc968700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 0x7f99d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:37.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.432 7f99fc968700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 0x7f99d801ac10 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:37.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.461 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f8103280 con 0x7f99f81a0250 2021-02-10T11:28:37.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.463 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99f4006340 con 0x7f99f81a0250 2021-02-10T11:28:37.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.590 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802464", "target": ["mgr", ""]}) v1 -- 0x7f99f8107310 con 0x7f99d8018730 2021-02-10T11:28:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.614 7f99edffb700 1 -- 172.21.2.117:0/2721439669 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f99e80094c0 con 0x7f99d8018730 2021-02-10T11:28:37.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 msgr2=0x7f99d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 0x7f99d801ac10 crc :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 msgr2=0x7f99f8106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f99f4006f60 tx=0x7f99f400b970).stop 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 shutdown_connections 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99f8090420 0x7f99f8090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99f8187e20 0x7f99f8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99d8018730 0x7f99d801ac10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 --2- 172.21.2.117:0/2721439669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99f81a0250 0x7f99f8106cf0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:37.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 >> 172.21.2.117:0/2721439669 conn(0x7f99f80a1850 msgr2=0x7f99f80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:37.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 shutdown_connections 2021-02-10T11:28:37.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:37.623 7f99ff34d700 1 -- 172.21.2.117:0/2721439669 wait complete. 2021-02-10T11:28:37.623 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802464 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.044 7fb8f489c700 1 Processor -- start 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.044 7fb8f489c700 1 -- start start 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8f489c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f007da20 0x7fb8f00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8f489c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb8f00e4da0 0x7fb8f01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8f489c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f0077a70 con 0x7fb8f00e51a0 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8f489c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f00e5c20 con 0x7fb8f00e4da0 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8f489c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00777d0 con 0x7fb8f007da20 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8eee1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb8f00e51a0 0x7fb8f0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44202/0 (socket says 172.21.2.117:44202) 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8eee1d700 1 -- 172.21.2.117:0/1259342926 learned_addr learned my addr 172.21.2.117:0/1259342926 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8ede1b700 1 --2- 172.21.2.117:0/1259342926 >> v2:172.21.2.125:3300/0 conn(0x7fb8f00e4da0 0x7fb8f01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8ee61c700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f007da20 0x7fb8f00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8ed61a700 1 -- 172.21.2.117:0/1259342926 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064115396 0 0) 0x7fb8e8001500 con 0x7fb8f00e51a0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.045 7fb8ed61a700 1 -- 172.21.2.117:0/1259342926 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d8002fe0 con 0x7fb8f00e51a0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ede1b700 1 -- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f007da20 msgr2=0x7fb8f00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ede1b700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f007da20 0x7fb8f00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ede1b700 1 -- 172.21.2.117:0/1259342926 >> v1:172.21.2.117:6789/0 conn(0x7fb8f00e51a0 legacy=0x7fb8f0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ede1b700 1 -- 172.21.2.117:0/1259342926 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f007de20 con 0x7fb8f00e4da0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ede1b700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f00e4da0 0x7fb8f01b00f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb8e4006650 tx=0x7fb8e4009a80).ready entity=mon.1 client_cookie=91e0b034a5202dda server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ed61a700 1 -- 172.21.2.117:0/1259342926 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e400c0c0 con 0x7fb8f00e4da0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ed61a700 1 -- 172.21.2.117:0/1259342926 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e400c0c0 con 0x7fb8f00e4da0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8ed61a700 1 -- 172.21.2.117:0/1259342926 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e400c340 con 0x7fb8f00e4da0 2021-02-10T11:28:38.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8f489c700 1 -- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f00e4da0 msgr2=0x7fb8f01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.046 7fb8f489c700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f00e4da0 0x7fb8f01b00f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fb8e4006650 tx=0x7fb8e4009a80).stop 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 -- 172.21.2.117:0/1259342926 shutdown_connections 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f00e4da0 0x7fb8f01b00f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 --2- 172.21.2.117:0/1259342926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f007da20 0x7fb8f00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 -- 172.21.2.117:0/1259342926 >> 172.21.2.117:0/1259342926 conn(0x7fb8f00ce450 msgr2=0x7fb8f00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 -- 172.21.2.117:0/1259342926 shutdown_connections 2021-02-10T11:28:38.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.047 7fb8f489c700 1 -- 172.21.2.117:0/1259342926 wait complete. 2021-02-10T11:28:38.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.061 7fb8f489c700 1 Processor -- start 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 -- start start 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f008dfe0 0x7fb8f008e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f00afac0 0x7fb8f00afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00af540 con 0x7fb8f008eb10 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00a0c80 con 0x7fb8f008dfe0 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8f489c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00b04e0 con 0x7fb8f00afac0 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43958/0 (socket says 172.21.2.117:43958) 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 -- 172.21.2.117:0/2335280924 learned_addr learned my addr 172.21.2.117:0/2335280924 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 -- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f00afac0 msgr2=0x7fb8f00afec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:38.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f00afac0 0x7fb8f00afec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 -- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f008dfe0 msgr2=0x7fb8f008e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f008dfe0 0x7fb8f008e3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.062 7fb8ee61c700 1 -- 172.21.2.117:0/2335280924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00b0700 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8ee61c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fb8e0006f60 tx=0x7fb8e000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e000bd60 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e0000c90 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f00b0700 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f00b0c90 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e0002220 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8e00052e0 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 0x7fb8d401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.063 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb8e0032900 con 0x7fb8f008eb10 2021-02-10T11:28:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.064 7fb8ede1b700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 0x7fb8d401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.065 7fb8ede1b700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 0x7fb8d401ac10 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.094 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0103280 con 0x7fb8f008eb10 2021-02-10T11:28:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.097 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8e0000960 con 0x7fb8f008eb10 2021-02-10T11:28:38.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.224 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802464", "target": ["mgr", ""]}) v1 -- 0x7fb8f00b0c90 con 0x7fb8d4018730 2021-02-10T11:28:38.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.228 7fb8deffd700 1 -- 172.21.2.117:0/2335280924 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb8e4000940 con 0x7fb8d4018730 2021-02-10T11:28:38.223 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c 2021-02-10T11:28:38.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 msgr2=0x7fb8d401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:38.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 0x7fb8d401ac10 crc :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 msgr2=0x7fb8f00c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:38.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7fb8e0006f60 tx=0x7fb8e000b970).stop 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 shutdown_connections 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8f00afac0 0x7fb8f00afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8f008dfe0 0x7fb8f008e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8d4018730 0x7fb8d401ac10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 --2- 172.21.2.117:0/2335280924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8f008eb10 0x7fb8f00c3b60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:38.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 >> 172.21.2.117:0/2335280924 conn(0x7fb8f00a1850 msgr2=0x7fb8f00bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:38.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 shutdown_connections 2021-02-10T11:28:38.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:38.234 7fb8f489c700 1 -- 172.21.2.117:0/2335280924 wait complete. 2021-02-10T11:28:38.234 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802464 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c 2021-02-10T11:28:38.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:38.235 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:38.235 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:38.570 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:38.570 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:38.571 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:38.951 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:38.952 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:38.952 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244158 s, 42.9 MB/s 2021-02-10T11:28:38.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:38.953 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:38.953 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:39.288 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:39.288 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:39.288 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226969 s, 46.2 MB/s 2021-02-10T11:28:39.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:39.662 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:39.662 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:39.997 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:39.998 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:39.998 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:40.372 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:40.373 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:40.373 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229943 s, 45.6 MB/s 2021-02-10T11:28:40.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:40.374 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:40.374 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:40.709 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:40.709 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:40.710 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:41.082 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:41.082 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:41.082 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022627 s, 46.3 MB/s 2021-02-10T11:28:41.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:41.083 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:41.083 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:41.417 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:41.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:41.418 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:41.796 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025587 s, 41.0 MB/s 2021-02-10T11:28:41.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:41.797 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:41.797 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:42.131 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:42.131 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:42.131 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:42.504 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:42.504 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:42.504 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256826 s, 40.8 MB/s 2021-02-10T11:28:42.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:42.505 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:42.505 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:42.841 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:42.842 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:42.842 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237818 s, 44.1 MB/s 2021-02-10T11:28:43.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:43.217 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:43.217 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:43.553 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:43.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:43.554 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:43.930 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:43.930 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:43.930 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248243 s, 42.2 MB/s 2021-02-10T11:28:43.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:43.931 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:43.931 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:44.266 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:44.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:44.267 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:44.641 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:44.642 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:44.642 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249465 s, 42.0 MB/s 2021-02-10T11:28:44.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:44.643 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:44.643 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:44.978 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:44.978 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:44.979 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802464/8e044207-456b-42d3-a4ae-939e7df0fb6c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:45.352 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:45.352 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:45.352 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226428 s, 46.3 MB/s 2021-02-10T11:28:45.353 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802465 2021-02-10T11:28:45.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.773 7fa75a231700 1 Processor -- start 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.773 7fa75a231700 1 -- start start 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa75a231700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa7540d1530 0x7fa7540d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa75a231700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754077a70 con 0x7fa754079000 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa75a231700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754079400 con 0x7fa75407da20 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa75a231700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75407de20 con 0x7fa7540d1530 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa7537fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa754079000 0x7fa7540e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44216/0 (socket says 172.21.2.117:44216) 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa7537fe700 1 -- 172.21.2.117:0/780804720 learned_addr learned my addr 172.21.2.117:0/780804720 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516751532 0 0) 0x7fa7480009b0 con 0x7fa754079000 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa752ffd700 1 -- 172.21.2.117:0/780804720 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa73c002fe0 con 0x7fa754079000 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.774 7fa75884e700 1 --2- 172.21.2.117:0/780804720 >> v2:172.21.2.125:3300/0 conn(0x7fa7540d1530 0x7fa7540d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3304077693 0 0) 0x7fa748001010 con 0x7fa754079000 2021-02-10T11:28:45.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 >> v2:172.21.2.125:3300/0 conn(0x7fa7540d1530 msgr2=0x7fa7540d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 --2- 172.21.2.117:0/780804720 >> v2:172.21.2.125:3300/0 conn(0x7fa7540d1530 0x7fa7540d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 >> v1:172.21.2.125:6789/0 conn(0x7fa75407da20 legacy=0x7fa7540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7541b00f0 con 0x7fa754079000 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa748001590 con 0x7fa754079000 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 >> v1:172.21.2.117:6789/0 conn(0x7fa754079000 legacy=0x7fa7540e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa73c003a60 0x7fa73c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa73c006770 0x7fa73c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 0x7fa73c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c00c300 con 0x7fa73c009590 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c00c520 con 0x7fa73c003a60 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa752ffd700 1 -- 172.21.2.117:0/780804720 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c050750 con 0x7fa73c006770 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 0x7fa73c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa753fff700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa73c003a60 0x7fa73c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa7537fe700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa73c006770 0x7fa73c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 -- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa73c006770 msgr2=0x7fa73c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa73c006770 0x7fa73c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 -- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa73c003a60 msgr2=0x7fa73c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa73c003a60 0x7fa73c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 -- 172.21.2.117:0/780804720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c0509b0 con 0x7fa73c009590 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 -- 172.21.2.117:0/780804720 reap_dead start 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.775 7fa75884e700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 0x7fa73c00bab0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fa74c006a40 tx=0x7fa74c009f40).ready entity=mon.0 client_cookie=33ee18e7d992a3a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74c006a80 con 0x7fa73c009590 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c007070 con 0x7fa73c009590 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa752ffd700 1 -- 172.21.2.117:0/780804720 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74c0074d0 con 0x7fa73c009590 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 -- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 msgr2=0x7fa73c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 0x7fa73c00bab0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fa74c006a40 tx=0x7fa74c009f40).stop 2021-02-10T11:28:45.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 -- 172.21.2.117:0/780804720 shutdown_connections 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 --2- 172.21.2.117:0/780804720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa73c009590 0x7fa73c00bab0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 -- 172.21.2.117:0/780804720 >> 172.21.2.117:0/780804720 conn(0x7fa7540ce450 msgr2=0x7fa7540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 -- 172.21.2.117:0/780804720 shutdown_connections 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.776 7fa75a231700 1 -- 172.21.2.117:0/780804720 wait complete. 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.791 7fa75a231700 1 Processor -- start 2021-02-10T11:28:45.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.791 7fa75a231700 1 -- start start 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7540a47d0 0x7fa7540a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7540b77d0 0x7fa7540b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa754189970 con 0x7fa7540a51f0 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa754190aa0 con 0x7fa7540b77d0 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75a231700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540b81f0 con 0x7fa7540a47d0 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43976/0 (socket says 172.21.2.117:43976) 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 -- 172.21.2.117:0/1658593332 learned_addr learned my addr 172.21.2.117:0/1658593332 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa753fff700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7540a47d0 0x7fa7540a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa75884e700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7540b77d0 0x7fa7540b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 -- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7540a47d0 msgr2=0x7fa7540a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7540a47d0 0x7fa7540a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 -- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7540b77d0 msgr2=0x7fa7540b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7540b77d0 0x7fa7540b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 -- 172.21.2.117:0/1658593332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540b8410 con 0x7fa7540a51f0 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.792 7fa7537fe700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fa748001bf0 tx=0x7fa74800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7480012d0 con 0x7fa7540a51f0 2021-02-10T11:28:45.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74800ad80 con 0x7fa7540a51f0 2021-02-10T11:28:45.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa748004330 con 0x7fa7540a51f0 2021-02-10T11:28:45.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa75a231700 1 -- 172.21.2.117:0/1658593332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7540b8410 con 0x7fa7540a51f0 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa75a231700 1 -- 172.21.2.117:0/1658593332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7540b8410 con 0x7fa7540a51f0 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa748004140 con 0x7fa7540a51f0 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.793 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa7480327a0 con 0x7fa7540a51f0 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.794 7fa753fff700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:45.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.794 7fa753fff700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:45.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.823 7fa75a231700 1 -- 172.21.2.117:0/1658593332 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa754077a70 con 0x7fa7540a51f0 2021-02-10T11:28:45.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.826 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa748001030 con 0x7fa7540a51f0 2021-02-10T11:28:45.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.951 7fa75a231700 1 -- 172.21.2.117:0/1658593332 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802465", "target": ["mgr", ""]}) v1 -- 0x7fa754077a70 con 0x7fa7340186c0 2021-02-10T11:28:45.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.976 7fa750ff9700 1 -- 172.21.2.117:0/1658593332 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa744007d50 con 0x7fa7340186c0 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 -- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 msgr2=0x7fa73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 -- 172.21.2.117:0/1658593332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 msgr2=0x7fa7540a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fa748001bf0 tx=0x7fa74800a900).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 -- 172.21.2.117:0/1658593332 shutdown_connections 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7540a51f0 0x7fa7540a55f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7540b77d0 0x7fa7540b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 --2- 172.21.2.117:0/1658593332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7540a47d0 0x7fa7540a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:45.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.984 7fa75a231700 1 -- 172.21.2.117:0/1658593332 >> 172.21.2.117:0/1658593332 conn(0x7fa7540b5140 msgr2=0x7fa7540a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:45.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.985 7fa75a231700 1 -- 172.21.2.117:0/1658593332 shutdown_connections 2021-02-10T11:28:45.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:45.985 7fa75a231700 1 -- 172.21.2.117:0/1658593332 wait complete. 2021-02-10T11:28:45.984 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802465 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 Processor -- start 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 -- start start 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f569007da20 0x7f56900771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56900e4da0 0x7f56901b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900e51a0 0x7f5690077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5690077a70 con 0x7f56900e4da0 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56900777d0 con 0x7f569007da20 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f5694be5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569007de20 con 0x7f56900e51a0 2021-02-10T11:28:46.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f568e05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56900e4da0 0x7f56901b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f568e05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56900e4da0 0x7f56901b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43982/0 (socket says 172.21.2.117:43982) 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.406 7f568e05d700 1 -- 172.21.2.117:0/822907675 learned_addr learned my addr 172.21.2.117:0/822907675 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 -- 172.21.2.117:0/822907675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900e51a0 msgr2=0x7f5690077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 --2- 172.21.2.117:0/822907675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900e51a0 0x7f5690077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 -- 172.21.2.117:0/822907675 >> v2:172.21.2.125:3300/0 conn(0x7f569007da20 msgr2=0x7f56900771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 --2- 172.21.2.117:0/822907675 >> v2:172.21.2.125:3300/0 conn(0x7f569007da20 0x7f56900771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 -- 172.21.2.117:0/822907675 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56901b0710 con 0x7f56900e4da0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568e05d700 1 --2- 172.21.2.117:0/822907675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56900e4da0 0x7f56901b00f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f5684006700 tx=0x7f5684009b80).ready entity=mon.0 client_cookie=962444fd75cc0f54 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568d85c700 1 -- 172.21.2.117:0/822907675 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f568400c0c0 con 0x7f56900e4da0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568d85c700 1 -- 172.21.2.117:0/822907675 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568400c340 con 0x7f56900e4da0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f568d85c700 1 -- 172.21.2.117:0/822907675 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f568400c820 con 0x7f56900e4da0 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f5694be5700 1 -- 172.21.2.117:0/822907675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56900e4da0 msgr2=0x7f56901b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.407 7f5694be5700 1 --2- 172.21.2.117:0/822907675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56900e4da0 0x7f56901b00f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f5684006700 tx=0x7f5684009b80).stop 2021-02-10T11:28:46.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 -- 172.21.2.117:0/822907675 shutdown_connections 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 --2- 172.21.2.117:0/822907675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900e51a0 0x7f5690077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 --2- 172.21.2.117:0/822907675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56900e4da0 0x7f56901b00f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 --2- 172.21.2.117:0/822907675 >> v2:172.21.2.125:3300/0 conn(0x7f569007da20 0x7f56900771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 -- 172.21.2.117:0/822907675 >> 172.21.2.117:0/822907675 conn(0x7f56900ce450 msgr2=0x7f56900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 -- 172.21.2.117:0/822907675 shutdown_connections 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.408 7f5694be5700 1 -- 172.21.2.117:0/822907675 wait complete. 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.422 7f5694be5700 1 Processor -- start 2021-02-10T11:28:46.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.422 7f5694be5700 1 -- start start 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 0x7f56900acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900ad150 0x7f56900ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56900adb40 con 0x7f5690093960 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56900ad550 con 0x7f56900ac730 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f5694be5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56900ad730 con 0x7f56900ad150 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43986/0 (socket says 172.21.2.117:43986) 2021-02-10T11:28:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 -- 172.21.2.117:0/272975407 learned_addr learned my addr 172.21.2.117:0/272975407 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e05d700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 0x7f56900acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 -- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900ad150 msgr2=0x7f56900ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900ad150 0x7f56900ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.423 7f568e85e700 1 -- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 msgr2=0x7f56900acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f568e85e700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 0x7f56900acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f568e85e700 1 -- 172.21.2.117:0/272975407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900ae6e0 con 0x7f5690093960 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f568e05d700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 0x7f56900acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f568e85e700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f568000b560 tx=0x7f568000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f568000bcf0 con 0x7f5690093960 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5680006030 con 0x7f5690093960 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56800054d0 con 0x7f5690093960 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f5694be5700 1 -- 172.21.2.117:0/272975407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56900ae6e0 con 0x7f5690093960 2021-02-10T11:28:46.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f5694be5700 1 -- 172.21.2.117:0/272975407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56900ae6e0 con 0x7f5690093960 2021-02-10T11:28:46.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.424 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5680006210 con 0x7f5690093960 2021-02-10T11:28:46.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.425 7f567f7fe700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 0x7f567401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:46.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.425 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f56800328e0 con 0x7f5690093960 2021-02-10T11:28:46.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.425 7f568e05d700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 0x7f567401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:46.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.425 7f568e05d700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 0x7f567401abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:46.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.456 7f5694be5700 1 -- 172.21.2.117:0/272975407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5690103370 con 0x7f5690093960 2021-02-10T11:28:46.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.459 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5680006b80 con 0x7f5690093960 2021-02-10T11:28:46.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.586 7f5694be5700 1 -- 172.21.2.117:0/272975407 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802465", "target": ["mgr", ""]}) v1 -- 0x7f5690103370 con 0x7f56740186c0 2021-02-10T11:28:46.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.590 7f567f7fe700 1 -- 172.21.2.117:0/272975407 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f5684006340 con 0x7f56740186c0 2021-02-10T11:28:46.585 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f 2021-02-10T11:28:46.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 msgr2=0x7f567401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:46.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 0x7f567401abc0 crc :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 msgr2=0x7f5690093d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f568000b560 tx=0x7f568000b900).stop 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 shutdown_connections 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56900ad150 0x7f56900ae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56900ac730 0x7f56900acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56740186c0 0x7f567401abc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 --2- 172.21.2.117:0/272975407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5690093960 0x7f5690093d60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:46.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 >> 172.21.2.117:0/272975407 conn(0x7f56900bb5d0 msgr2=0x7f5690190400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:46.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 shutdown_connections 2021-02-10T11:28:46.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:46.596 7f5694be5700 1 -- 172.21.2.117:0/272975407 wait complete. 2021-02-10T11:28:46.596 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802465 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f 2021-02-10T11:28:46.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:46.596 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:46.597 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:46.933 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:46.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:46.933 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:47.313 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:47.313 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:47.313 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223238 s, 47.0 MB/s 2021-02-10T11:28:47.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:47.315 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:47.315 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:47.649 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:47.649 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:47.649 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:48.021 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:48.021 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:48.022 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247849 s, 42.3 MB/s 2021-02-10T11:28:48.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:48.023 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:48.023 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:48.358 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:48.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:48.359 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:48.735 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:48.735 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:48.735 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249038 s, 42.1 MB/s 2021-02-10T11:28:48.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:48.736 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:48.736 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:49.071 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:49.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:49.071 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:49.448 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:49.448 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:49.448 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257497 s, 40.7 MB/s 2021-02-10T11:28:49.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:49.450 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:49.450 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:49.785 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:49.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:49.786 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:50.161 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:50.161 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:50.162 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239005 s, 43.9 MB/s 2021-02-10T11:28:50.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:50.163 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:50.163 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:50.498 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:50.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:50.499 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:50.872 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:50.872 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:50.872 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225771 s, 46.4 MB/s 2021-02-10T11:28:50.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:50.873 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:50.873 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:51.207 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:51.207 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:51.207 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:51.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:51.584 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247363 s, 42.4 MB/s 2021-02-10T11:28:51.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:51.585 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:51.585 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:51.919 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:51.919 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:51.919 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:52.295 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:52.296 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:52.296 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250481 s, 41.9 MB/s 2021-02-10T11:28:52.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:52.297 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:52.297 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:52.631 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:52.632 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:52.632 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:53.003 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:53.003 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:53.004 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233717 s, 44.9 MB/s 2021-02-10T11:28:53.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:53.005 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:53.005 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:53.340 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:53.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:53.340 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802465/d8afe3d4-7020-4301-b952-2fbd598c902f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:53.715 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:53.716 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:53.716 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249747 s, 42.0 MB/s 2021-02-10T11:28:53.717 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802466 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.138 7f6db6803700 1 Processor -- start 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.138 7f6db6803700 1 -- start start 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6db6803700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 0x7f6db00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6db6803700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6db00e4da0 0x7f6db01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6db6803700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0077a70 con 0x7f6db00e51a0 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6db6803700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db00777d0 con 0x7f6db00e4da0 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6db6803700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db007de20 con 0x7f6db007da20 2021-02-10T11:28:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6db00e4da0 0x7f6db01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6db00e4da0 0x7f6db01b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43992/0 (socket says 172.21.2.117:43992) 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 -- 172.21.2.117:0/3378112537 learned_addr learned my addr 172.21.2.117:0/3378112537 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daeffd700 1 -- 172.21.2.117:0/3378112537 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262361779 0 0) 0x7f6da8000f90 con 0x7f6db00e51a0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 -- 172.21.2.117:0/3378112537 >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 msgr2=0x7f6db00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daffff700 1 --2- 172.21.2.117:0/3378112537 >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 0x7f6db00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 --2- 172.21.2.117:0/3378112537 >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 0x7f6db00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 -- 172.21.2.117:0/3378112537 >> v1:172.21.2.117:6789/0 conn(0x7f6db00e51a0 legacy=0x7f6db0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daf7fe700 1 -- 172.21.2.117:0/3378112537 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db01b0710 con 0x7f6db00e4da0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.139 7f6daffff700 1 --2- 172.21.2.117:0/3378112537 >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 0x7f6db00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6daf7fe700 1 --2- 172.21.2.117:0/3378112537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00e4da0 0x7f6db01b00f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f6d9c006700 tx=0x7f6d9c009b80).ready entity=mon.0 client_cookie=859a90213670e120 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6daeffd700 1 -- 172.21.2.117:0/3378112537 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9c00c0c0 con 0x7f6db00e4da0 2021-02-10T11:28:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6daeffd700 1 -- 172.21.2.117:0/3378112537 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c00c400 con 0x7f6db00e4da0 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6daeffd700 1 -- 172.21.2.117:0/3378112537 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9c00c8e0 con 0x7f6db00e4da0 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 -- 172.21.2.117:0/3378112537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00e4da0 msgr2=0x7f6db01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 --2- 172.21.2.117:0/3378112537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00e4da0 0x7f6db01b00f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f6d9c006700 tx=0x7f6d9c009b80).stop 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 -- 172.21.2.117:0/3378112537 shutdown_connections 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 --2- 172.21.2.117:0/3378112537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00e4da0 0x7f6db01b00f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 --2- 172.21.2.117:0/3378112537 >> v2:172.21.2.125:3300/0 conn(0x7f6db007da20 0x7f6db00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 -- 172.21.2.117:0/3378112537 >> 172.21.2.117:0/3378112537 conn(0x7f6db00ce450 msgr2=0x7f6db00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 -- 172.21.2.117:0/3378112537 shutdown_connections 2021-02-10T11:28:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.140 7f6db6803700 1 -- 172.21.2.117:0/3378112537 wait complete. 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 Processor -- start 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 -- start start 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6db00a1090 0x7f6db019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6db019db70 0x7f6db00a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00c1360 con 0x7f6db00a20e0 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00a2b00 con 0x7f6db019db70 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.155 7f6db6803700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00a2d20 con 0x7f6db00a1090 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:43998/0 (socket says 172.21.2.117:43998) 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 -- 172.21.2.117:0/1951085139 learned_addr learned my addr 172.21.2.117:0/1951085139 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6daf7fe700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6db019db70 0x7f6db00a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6daffff700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6db00a1090 0x7f6db019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 -- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6db00a1090 msgr2=0x7f6db019d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6db00a1090 0x7f6db019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 -- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6db019db70 msgr2=0x7f6db00a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6db019db70 0x7f6db00a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 -- 172.21.2.117:0/1951085139 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0106040 con 0x7f6db00a20e0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db4e20700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6da8009980 tx=0x7f6da800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6da8005a60 con 0x7f6db00a20e0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8005ce0 con 0x7f6db00a20e0 2021-02-10T11:28:54.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6da80062b0 con 0x7f6db00a20e0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db6803700 1 -- 172.21.2.117:0/1951085139 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db0106040 con 0x7f6db00a20e0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.156 7f6db6803700 1 -- 172.21.2.117:0/1951085139 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db01065d0 con 0x7f6db00a20e0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.157 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6da8005870 con 0x7f6db00a20e0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.157 7f6dacff9700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.157 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6da8032ab0 con 0x7f6db00a20e0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.158 7f6daffff700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.158 7f6daffff700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.189 7f6db6803700 1 -- 172.21.2.117:0/1951085139 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db01b4f00 con 0x7f6db00a20e0 2021-02-10T11:28:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.192 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6da8006570 con 0x7f6db00a20e0 2021-02-10T11:28:54.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.316 7f6db6803700 1 -- 172.21.2.117:0/1951085139 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802466", "target": ["mgr", ""]}) v1 -- 0x7f6db0077a70 con 0x7f6d8c0186c0 2021-02-10T11:28:54.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.341 7f6dacff9700 1 -- 172.21.2.117:0/1951085139 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6da0006e50 con 0x7f6d8c0186c0 2021-02-10T11:28:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 msgr2=0x7f6d8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 crc :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 msgr2=0x7f6db00a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6da8009980 tx=0x7f6da800bcd0).stop 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 shutdown_connections 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6db00a20e0 0x7f6db00a24e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6db019db70 0x7f6db00a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 --2- 172.21.2.117:0/1951085139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6db00a1090 0x7f6db019d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 >> 172.21.2.117:0/1951085139 conn(0x7f6db00ab050 msgr2=0x7f6db0107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 shutdown_connections 2021-02-10T11:28:54.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.349 7f6db6803700 1 -- 172.21.2.117:0/1951085139 wait complete. 2021-02-10T11:28:54.348 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802466 2021-02-10T11:28:54.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 Processor -- start 2021-02-10T11:28:54.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 -- start start 2021-02-10T11:28:54.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4ea807da20 0x7f4ea80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ea80e4da0 0x7f4ea81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80e51a0 0x7f4ea8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea8077a70 con 0x7f4ea80e4da0 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea80777d0 con 0x7f4ea807da20 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.772 7f4ead5c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea807de20 con 0x7f4ea80e51a0 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ea80e4da0 0x7f4ea81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ea80e4da0 0x7f4ea81b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44004/0 (socket says 172.21.2.117:44004) 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 -- 172.21.2.117:0/4287922264 learned_addr learned my addr 172.21.2.117:0/4287922264 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 -- 172.21.2.117:0/4287922264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80e51a0 msgr2=0x7f4ea8077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 --2- 172.21.2.117:0/4287922264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80e51a0 0x7f4ea8077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 -- 172.21.2.117:0/4287922264 >> v2:172.21.2.125:3300/0 conn(0x7f4ea807da20 msgr2=0x7f4ea80771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 --2- 172.21.2.117:0/4287922264 >> v2:172.21.2.125:3300/0 conn(0x7f4ea807da20 0x7f4ea80771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 -- 172.21.2.117:0/4287922264 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea81b0710 con 0x7f4ea80e4da0 2021-02-10T11:28:54.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea6abe700 1 --2- 172.21.2.117:0/4287922264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea80e4da0 0x7f4ea81b00f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4e98006700 tx=0x7f4e98009b00).ready entity=mon.0 client_cookie=6697ae49c609af39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea62bd700 1 -- 172.21.2.117:0/4287922264 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e9800c0c0 con 0x7f4ea80e4da0 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea62bd700 1 -- 172.21.2.117:0/4287922264 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9800c400 con 0x7f4ea80e4da0 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ea62bd700 1 -- 172.21.2.117:0/4287922264 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e9800c8e0 con 0x7f4ea80e4da0 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ead5c8700 1 -- 172.21.2.117:0/4287922264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea80e4da0 msgr2=0x7f4ea81b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.773 7f4ead5c8700 1 --2- 172.21.2.117:0/4287922264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea80e4da0 0x7f4ea81b00f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f4e98006700 tx=0x7f4e98009b00).stop 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 -- 172.21.2.117:0/4287922264 shutdown_connections 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 --2- 172.21.2.117:0/4287922264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80e51a0 0x7f4ea8077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 --2- 172.21.2.117:0/4287922264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea80e4da0 0x7f4ea81b00f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 --2- 172.21.2.117:0/4287922264 >> v2:172.21.2.125:3300/0 conn(0x7f4ea807da20 0x7f4ea80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 -- 172.21.2.117:0/4287922264 >> 172.21.2.117:0/4287922264 conn(0x7f4ea80ce450 msgr2=0x7f4ea80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 -- 172.21.2.117:0/4287922264 shutdown_connections 2021-02-10T11:28:54.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.774 7f4ead5c8700 1 -- 172.21.2.117:0/4287922264 wait complete. 2021-02-10T11:28:54.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.788 7f4ead5c8700 1 Processor -- start 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.788 7f4ead5c8700 1 -- start start 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80b3320 0x7f4ea80912f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ea81a78e0 0x7f4ea80925a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8091d90 con 0x7f4ea8091910 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80b3720 con 0x7f4ea81a78e0 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80b5bc0 con 0x7f4ea80b3320 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44010/0 (socket says 172.21.2.117:44010) 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 -- 172.21.2.117:0/3227445646 learned_addr learned my addr 172.21.2.117:0/3227445646 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 -- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80b3320 msgr2=0x7f4ea80912f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:54.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80b3320 0x7f4ea80912f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 -- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ea81a78e0 msgr2=0x7f4ea80925a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ea81a78e0 0x7f4ea80925a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 -- 172.21.2.117:0/3227445646 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea81a7ce0 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ea6abe700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f4e98006f20 tx=0x7f4e9800b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e9800bdd0 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98009160 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e980052e0 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea81a7ce0 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.789 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea81a7ce0 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.790 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e98006030 con 0x7f4ea8091910 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.790 7f4e9ffff700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:28:54.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.790 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4e98032930 con 0x7f4ea8091910 2021-02-10T11:28:54.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.791 7f4ea72bf700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:28:54.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.791 7f4ea72bf700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:28:54.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.823 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea8103370 con 0x7f4ea8091910 2021-02-10T11:28:54.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.825 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e98006440 con 0x7f4ea8091910 2021-02-10T11:28:54.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.954 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802466", "target": ["mgr", ""]}) v1 -- 0x7f4ea8103370 con 0x7f4e8c0186c0 2021-02-10T11:28:54.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.957 7f4e9ffff700 1 -- 172.21.2.117:0/3227445646 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4e90006990 con 0x7f4e8c0186c0 2021-02-10T11:28:54.953 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761 2021-02-10T11:28:54.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.963 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 msgr2=0x7f4e8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.963 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 crc :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.963 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 msgr2=0x7f4ea81a72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.963 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f4e98006f20 tx=0x7f4e9800b9e0).stop 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 shutdown_connections 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e8c0186c0 0x7f4e8c01abc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ea8091910 0x7f4ea81a72c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ea81a78e0 0x7f4ea80925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 --2- 172.21.2.117:0/3227445646 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ea80b3320 0x7f4ea80912f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:28:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 >> 172.21.2.117:0/3227445646 conn(0x7f4ea80bb580 msgr2=0x7f4ea8095d00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:28:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 shutdown_connections 2021-02-10T11:28:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:28:54.964 7f4ead5c8700 1 -- 172.21.2.117:0/3227445646 wait complete. 2021-02-10T11:28:54.963 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802466 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761 2021-02-10T11:28:54.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:54.963 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:54.963 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:55.305 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:55.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:55.305 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:55.690 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:55.690 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:55.690 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247943 s, 42.3 MB/s 2021-02-10T11:28:55.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:55.691 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:55.691 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:56.026 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:56.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:56.027 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:56.401 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:56.401 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:56.401 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247047 s, 42.4 MB/s 2021-02-10T11:28:56.402 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:56.402 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:56.403 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:56.738 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:56.738 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:56.739 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:56.974 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:56.978 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:28:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0276015 s, 38.0 MB/s 2021-02-10T11:28:57.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:57.117 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:57.117 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:57.451 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:57.452 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:57.452 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247032 s, 42.4 MB/s 2021-02-10T11:28:57.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:57.824 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:57.824 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:58.160 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:58.160 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:58.161 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260314 s, 40.3 MB/s 2021-02-10T11:28:58.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:58.540 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:58.540 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:58.875 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:58.875 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:58.875 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:59.250 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:59.250 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:59.250 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024599 s, 42.6 MB/s 2021-02-10T11:28:59.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:59.251 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:59.251 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:28:59.586 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:28:59.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:28:59.587 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:28:59.962 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:28:59.962 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:28:59.963 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252727 s, 41.5 MB/s 2021-02-10T11:28:59.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:28:59.964 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:28:59.964 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:00.298 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:00.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:00.298 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:00.671 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:00.671 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:00.671 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226816 s, 46.2 MB/s 2021-02-10T11:29:00.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:00.672 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:00.672 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:01.008 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:01.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:01.008 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:01.385 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:01.386 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:01.386 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257538 s, 40.7 MB/s 2021-02-10T11:29:01.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:01.387 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:01.387 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:01.721 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:01.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:01.722 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802466/f06fb8f7-52b3-469b-853b-1872b31b7761/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:02.097 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:02.097 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:02.097 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254096 s, 41.3 MB/s 2021-02-10T11:29:02.098 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802467 2021-02-10T11:29:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 Processor -- start 2021-02-10T11:29:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 -- start start 2021-02-10T11:29:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f175c07da20 0x7f175c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c0e4da0 0x7f175c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f175c0e51a0 0x7f175c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175c077a70 con 0x7f175c0e51a0 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175c0777d0 con 0x7f175c07da20 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.521 7f1764179700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175c07de20 con 0x7f175c0e4da0 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f175c0e51a0 0x7f175c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f175c0e51a0 0x7f175c077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44022/0 (socket says 172.21.2.117:44022) 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 -- 172.21.2.117:0/3334909116 learned_addr learned my addr 172.21.2.117:0/3334909116 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1761794700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c0e4da0 0x7f175c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1761f95700 1 --2- 172.21.2.117:0/3334909116 >> v2:172.21.2.125:3300/0 conn(0x7f175c07da20 0x7f175c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 -- 172.21.2.117:0/3334909116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c0e4da0 msgr2=0x7f175c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c0e4da0 0x7f175c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 -- 172.21.2.117:0/3334909116 >> v2:172.21.2.125:3300/0 conn(0x7f175c07da20 msgr2=0x7f175c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 --2- 172.21.2.117:0/3334909116 >> v2:172.21.2.125:3300/0 conn(0x7f175c07da20 0x7f175c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 -- 172.21.2.117:0/3334909116 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c1b0710 con 0x7f175c0e51a0 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1762796700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0e51a0 0x7f175c077ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f1754006f00 tx=0x7f1754009f10).ready entity=mon.0 client_cookie=3282cdf90936ac16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1760f93700 1 -- 172.21.2.117:0/3334909116 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1754003030 con 0x7f175c0e51a0 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1760f93700 1 -- 172.21.2.117:0/3334909116 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175400b3b0 con 0x7f175c0e51a0 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1760f93700 1 -- 172.21.2.117:0/3334909116 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175400b8d0 con 0x7f175c0e51a0 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1764179700 1 -- 172.21.2.117:0/3334909116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0e51a0 msgr2=0x7f175c077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.522 7f1764179700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0e51a0 0x7f175c077ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f1754006f00 tx=0x7f1754009f10).stop 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 -- 172.21.2.117:0/3334909116 shutdown_connections 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0e51a0 0x7f175c077ff0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 --2- 172.21.2.117:0/3334909116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c0e4da0 0x7f175c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 --2- 172.21.2.117:0/3334909116 >> v2:172.21.2.125:3300/0 conn(0x7f175c07da20 0x7f175c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 -- 172.21.2.117:0/3334909116 >> 172.21.2.117:0/3334909116 conn(0x7f175c0ce450 msgr2=0x7f175c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:02.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 -- 172.21.2.117:0/3334909116 shutdown_connections 2021-02-10T11:29:02.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.523 7f1764179700 1 -- 172.21.2.117:0/3334909116 wait complete. 2021-02-10T11:29:02.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 Processor -- start 2021-02-10T11:29:02.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 -- start start 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c1048f0 0x7f175c104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f175c090710 0x7f175c090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c1054f0 con 0x7f175c0b7fc0 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c195a30 con 0x7f175c090710 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.538 7f1764179700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175c195fb0 con 0x7f175c1048f0 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44026/0 (socket says 172.21.2.117:44026) 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 -- 172.21.2.117:0/1051926578 learned_addr learned my addr 172.21.2.117:0/1051926578 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:02.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 -- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c1048f0 msgr2=0x7f175c104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c1048f0 0x7f175c104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 -- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f175c090710 msgr2=0x7f175c090b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f175c090710 0x7f175c090b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 -- 172.21.2.117:0/1051926578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175c0b7b60 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1761f95700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f175000b530 tx=0x7f175000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175000bcc0 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1750002030 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1750002490 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1764179700 1 -- 172.21.2.117:0/1051926578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175c0b7b60 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.539 7f1764179700 1 -- 172.21.2.117:0/1051926578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c0b7b60 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.540 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1750000c90 con 0x7f175c0b7fc0 2021-02-10T11:29:02.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.540 7f174e7fc700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 0x7f174401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:02.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.540 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f17500326d0 con 0x7f175c0b7fc0 2021-02-10T11:29:02.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.540 7f1761794700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 0x7f174401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:02.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.540 7f1761794700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 0x7f174401abc0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:02.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.570 7f1764179700 1 -- 172.21.2.117:0/1051926578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175c103280 con 0x7f175c0b7fc0 2021-02-10T11:29:02.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.573 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f175002d3e0 con 0x7f175c0b7fc0 2021-02-10T11:29:02.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.698 7f1764179700 1 -- 172.21.2.117:0/1051926578 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802467", "target": ["mgr", ""]}) v1 -- 0x7f175c103280 con 0x7f17440186c0 2021-02-10T11:29:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.723 7f174e7fc700 1 -- 172.21.2.117:0/1051926578 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f17580009b0 con 0x7f17440186c0 2021-02-10T11:29:02.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.731 7f1764179700 1 -- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 msgr2=0x7f174401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.731 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 0x7f174401abc0 crc :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.731 7f1764179700 1 -- 172.21.2.117:0/1051926578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 msgr2=0x7f175c1042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:02.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.731 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f175000b530 tx=0x7f175000b8d0).stop 2021-02-10T11:29:02.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 -- 172.21.2.117:0/1051926578 shutdown_connections 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f175c090710 0x7f175c090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f175c1048f0 0x7f175c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17440186c0 0x7f174401abc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 --2- 172.21.2.117:0/1051926578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f175c0b7fc0 0x7f175c1042d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 -- 172.21.2.117:0/1051926578 >> 172.21.2.117:0/1051926578 conn(0x7f175c0a1850 msgr2=0x7f175c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 -- 172.21.2.117:0/1051926578 shutdown_connections 2021-02-10T11:29:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:02.732 7f1764179700 1 -- 172.21.2.117:0/1051926578 wait complete. 2021-02-10T11:29:02.731 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802467 2021-02-10T11:29:03.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.152 7f8abacba700 1 Processor -- start 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.152 7f8abacba700 1 -- start start 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8abacba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ab40d1530 0x7f8ab40d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8abacba700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab4077a70 con 0x7f8ab4079000 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8abacba700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab4079400 con 0x7f8ab407da20 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8abacba700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ab407de20 con 0x7f8ab40d1530 2021-02-10T11:29:03.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8ab4079000 0x7f8ab40e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44282/0 (socket says 172.21.2.117:44282) 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab3fff700 1 -- 172.21.2.117:0/2181600075 learned_addr learned my addr 172.21.2.117:0/2181600075 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab92d7700 1 --2- 172.21.2.117:0/2181600075 >> v2:172.21.2.117:3300/0 conn(0x7f8ab40d1530 0x7f8ab40d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab37fe700 1 -- 172.21.2.117:0/2181600075 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058459842 0 0) 0x7f8aa4001180 con 0x7f8ab4079000 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab37fe700 1 -- 172.21.2.117:0/2181600075 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa0002fe0 con 0x7f8ab4079000 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab92d7700 1 -- 172.21.2.117:0/2181600075 >> v1:172.21.2.125:6789/0 conn(0x7f8ab407da20 legacy=0x7f8ab40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab92d7700 1 -- 172.21.2.117:0/2181600075 >> v1:172.21.2.117:6789/0 conn(0x7f8ab4079000 legacy=0x7f8ab40e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab92d7700 1 -- 172.21.2.117:0/2181600075 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab41b00f0 con 0x7f8ab40d1530 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.153 7f8ab92d7700 1 --2- 172.21.2.117:0/2181600075 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab40d1530 0x7f8ab40d1930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f8aac006e30 tx=0x7f8aac009e60).ready entity=mon.0 client_cookie=5b6d22eb466bfef4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8ab37fe700 1 -- 172.21.2.117:0/2181600075 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aac006a50 con 0x7f8ab40d1530 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8ab37fe700 1 -- 172.21.2.117:0/2181600075 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aac00b300 con 0x7f8ab40d1530 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8ab37fe700 1 -- 172.21.2.117:0/2181600075 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aac00b7e0 con 0x7f8ab40d1530 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8abacba700 1 -- 172.21.2.117:0/2181600075 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab40d1530 msgr2=0x7f8ab40d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8abacba700 1 --2- 172.21.2.117:0/2181600075 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab40d1530 0x7f8ab40d1930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f8aac006e30 tx=0x7f8aac009e60).stop 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8abacba700 1 -- 172.21.2.117:0/2181600075 shutdown_connections 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8abacba700 1 --2- 172.21.2.117:0/2181600075 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab40d1530 0x7f8ab40d1930 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.154 7f8abacba700 1 -- 172.21.2.117:0/2181600075 >> 172.21.2.117:0/2181600075 conn(0x7f8ab40ce450 msgr2=0x7f8ab40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.155 7f8abacba700 1 -- 172.21.2.117:0/2181600075 shutdown_connections 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.155 7f8abacba700 1 -- 172.21.2.117:0/2181600075 wait complete. 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.169 7f8abacba700 1 Processor -- start 2021-02-10T11:29:03.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.169 7f8abacba700 1 -- start start 2021-02-10T11:29:03.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 0x7f8ab40a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:03.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:03.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 0x7f8ab40b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:03.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab4189be0 con 0x7f8ab418a0e0 2021-02-10T11:29:03.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b83e0 con 0x7f8ab40b79c0 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8abacba700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab40b8600 con 0x7f8ab40a9890 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44038/0 (socket says 172.21.2.117:44038) 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 -- 172.21.2.117:0/787182727 learned_addr learned my addr 172.21.2.117:0/787182727 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 -- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 msgr2=0x7f8ab40a9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab8ad6700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 0x7f8ab40a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:03.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab92d7700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 0x7f8ab40b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 0x7f8ab40a9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 -- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 msgr2=0x7f8ab40b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 0x7f8ab40b7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab3fff700 1 -- 172.21.2.117:0/787182727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab40b8820 con 0x7f8ab418a0e0 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab8ad6700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 0x7f8ab40a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.170 7f8ab92d7700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 0x7f8ab40b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8ab3fff700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f8aa4006f00 tx=0x7f8aa400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa4006020 con 0x7f8ab418a0e0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa4006360 con 0x7f8ab418a0e0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa4004620 con 0x7f8ab418a0e0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8abacba700 1 -- 172.21.2.117:0/787182727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab40b8820 con 0x7f8ab418a0e0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8abacba700 1 -- 172.21.2.117:0/787182727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab40b8db0 con 0x7f8ab418a0e0 2021-02-10T11:29:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.171 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8aa400a000 con 0x7f8ab418a0e0 2021-02-10T11:29:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.172 7f8ab17fa700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 0x7f8a9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.172 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8aa4032500 con 0x7f8ab418a0e0 2021-02-10T11:29:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.172 7f8ab8ad6700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 0x7f8a9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.172 7f8ab8ad6700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 0x7f8a9801abc0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:03.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.202 7f8abacba700 1 -- 172.21.2.117:0/787182727 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab41032d0 con 0x7f8ab418a0e0 2021-02-10T11:29:03.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.205 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8aa40009b0 con 0x7f8ab418a0e0 2021-02-10T11:29:03.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.331 7f8abacba700 1 -- 172.21.2.117:0/787182727 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802467", "target": ["mgr", ""]}) v1 -- 0x7f8ab40b8db0 con 0x7f8a980186c0 2021-02-10T11:29:03.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.334 7f8ab17fa700 1 -- 172.21.2.117:0/787182727 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8a9c000960 con 0x7f8a980186c0 2021-02-10T11:29:03.329 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 -- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 msgr2=0x7f8a9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 0x7f8a9801abc0 crc :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 -- 172.21.2.117:0/787182727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 msgr2=0x7f8ab418a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f8aa4006f00 tx=0x7f8aa400b940).stop 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 -- 172.21.2.117:0/787182727 shutdown_connections 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a980186c0 0x7f8a9801abc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab418a0e0 0x7f8ab418a4e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab40b79c0 0x7f8ab40b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 --2- 172.21.2.117:0/787182727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab40a9890 0x7f8ab40a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:03.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.340 7f8abacba700 1 -- 172.21.2.117:0/787182727 >> 172.21.2.117:0/787182727 conn(0x7f8ab40b56c0 msgr2=0x7f8ab40a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:03.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.341 7f8abacba700 1 -- 172.21.2.117:0/787182727 shutdown_connections 2021-02-10T11:29:03.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:03.341 7f8abacba700 1 -- 172.21.2.117:0/787182727 wait complete. 2021-02-10T11:29:03.340 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802467 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e 2021-02-10T11:29:03.340 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:03.340 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:03.340 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:03.676 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:03.676 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:03.676 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:04.058 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:04.058 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:04.058 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232535 s, 45.1 MB/s 2021-02-10T11:29:04.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:04.059 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:04.059 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:04.394 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:04.395 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:04.395 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228814 s, 45.8 MB/s 2021-02-10T11:29:04.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:04.770 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:04.770 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:05.105 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:05.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:05.106 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:05.484 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251883 s, 41.6 MB/s 2021-02-10T11:29:05.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:05.486 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:05.487 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:05.830 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:05.830 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:05.830 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:06.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:06.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:06.208 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254531 s, 41.2 MB/s 2021-02-10T11:29:06.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:06.209 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:06.209 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:06.544 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:06.544 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:06.544 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:06.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:06.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:06.916 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226123 s, 46.4 MB/s 2021-02-10T11:29:06.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:06.917 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:06.917 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:07.253 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:07.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:07.253 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250037 s, 41.9 MB/s 2021-02-10T11:29:07.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:07.627 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:07.627 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:07.963 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:07.963 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:07.964 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:08.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:08.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:08.340 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024885 s, 42.1 MB/s 2021-02-10T11:29:08.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:08.341 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:08.341 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:08.676 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:08.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:08.677 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:09.051 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:09.052 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:09.052 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240422 s, 43.6 MB/s 2021-02-10T11:29:09.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:09.053 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:09.053 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:09.387 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:09.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:09.388 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:09.764 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:09.764 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:09.765 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251552 s, 41.7 MB/s 2021-02-10T11:29:09.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:09.766 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:09.766 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:10.101 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:10.101 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:10.101 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802467/0e62970e-6c83-4e94-aab3-78d22176305e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:10.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:10.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:10.478 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249852 s, 42.0 MB/s 2021-02-10T11:29:10.479 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802468 2021-02-10T11:29:10.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.901 7fbaa574c700 1 Processor -- start 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.901 7fbaa574c700 1 -- start start 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fbaa574c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa007da20 0x7fbaa00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fbaa574c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbaa00e4da0 0x7fbaa01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fbaa574c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa0077a70 con 0x7fbaa00e51a0 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fbaa574c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa00e5c20 con 0x7fbaa00e4da0 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fbaa574c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00777d0 con 0x7fbaa007da20 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbaa00e4da0 0x7fbaa01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbaa00e4da0 0x7fbaa01b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44046/0 (socket says 172.21.2.117:44046) 2021-02-10T11:29:10.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 -- 172.21.2.117:0/4244240343 learned_addr learned my addr 172.21.2.117:0/4244240343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9f549700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa007da20 0x7fbaa00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 -- 172.21.2.117:0/4244240343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa007da20 msgr2=0x7fbaa00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa007da20 0x7fbaa00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 -- 172.21.2.117:0/4244240343 >> v1:172.21.2.117:6789/0 conn(0x7fbaa00e51a0 legacy=0x7fbaa0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.902 7fba9ed48700 1 -- 172.21.2.117:0/4244240343 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa007de20 con 0x7fbaa00e4da0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fba9ed48700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa00e4da0 0x7fbaa01b00f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fba90006700 tx=0x7fba90009b80).ready entity=mon.0 client_cookie=a4c518ba7ca34b9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fba9e547700 1 -- 172.21.2.117:0/4244240343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9000c0c0 con 0x7fbaa00e4da0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fba9e547700 1 -- 172.21.2.117:0/4244240343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9000c340 con 0x7fbaa00e4da0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fba9e547700 1 -- 172.21.2.117:0/4244240343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9000c860 con 0x7fbaa00e4da0 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fbaa574c700 1 -- 172.21.2.117:0/4244240343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa00e4da0 msgr2=0x7fbaa01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.903 7fbaa574c700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa00e4da0 0x7fbaa01b00f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fba90006700 tx=0x7fba90009b80).stop 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 -- 172.21.2.117:0/4244240343 shutdown_connections 2021-02-10T11:29:10.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa00e4da0 0x7fbaa01b00f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 --2- 172.21.2.117:0/4244240343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa007da20 0x7fbaa00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 -- 172.21.2.117:0/4244240343 >> 172.21.2.117:0/4244240343 conn(0x7fbaa00ce450 msgr2=0x7fbaa00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 -- 172.21.2.117:0/4244240343 shutdown_connections 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.904 7fbaa574c700 1 -- 172.21.2.117:0/4244240343 wait complete. 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 Processor -- start 2021-02-10T11:29:10.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 -- start start 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa0099820 0x7fbaa0098dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa0105870 0x7fbaa0105c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.919 7fbaa574c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0106a40 con 0x7fbaa0106290 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fbaa574c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0099430 con 0x7fbaa0099820 2021-02-10T11:29:10.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fbaa574c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0106690 con 0x7fbaa0105870 2021-02-10T11:29:10.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44050/0 (socket says 172.21.2.117:44050) 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 -- 172.21.2.117:0/3390570388 learned_addr learned my addr 172.21.2.117:0/3390570388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 -- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa0105870 msgr2=0x7fbaa0105c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa0105870 0x7fbaa0105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 -- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa0099820 msgr2=0x7fbaa0098dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa0099820 0x7fbaa0098dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 -- 172.21.2.117:0/3390570388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0100280 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba9fd4a700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fba98009980 tx=0x7fba9800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba98005870 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98005a70 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba98006030 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa0100280 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.920 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa0107850 con 0x7fbaa0106290 2021-02-10T11:29:10.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.921 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba98005c50 con 0x7fbaa0106290 2021-02-10T11:29:10.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.921 7fba97fff700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 0x7fba8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:10.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.921 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fba98032a70 con 0x7fbaa0106290 2021-02-10T11:29:10.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.921 7fba9f549700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 0x7fba8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:10.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.922 7fba9f549700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 0x7fba8401abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:10.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.952 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa01b4f00 con 0x7fbaa0106290 2021-02-10T11:29:10.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:10.954 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba98025f30 con 0x7fbaa0106290 2021-02-10T11:29:11.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.082 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802468", "target": ["mgr", ""]}) v1 -- 0x7fbaa0106690 con 0x7fba840186c0 2021-02-10T11:29:11.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.107 7fba97fff700 1 -- 172.21.2.117:0/3390570388 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba880063c0 con 0x7fba840186c0 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 msgr2=0x7fba8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 0x7fba8401abc0 crc :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 msgr2=0x7fbaa0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fba98009980 tx=0x7fba9800bcd0).stop 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 shutdown_connections 2021-02-10T11:29:11.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba840186c0 0x7fba8401abc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa0106290 0x7fbaa0106f40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa0105870 0x7fbaa0105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 --2- 172.21.2.117:0/3390570388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa0099820 0x7fbaa0098dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 >> 172.21.2.117:0/3390570388 conn(0x7fbaa00ffe30 msgr2=0x7fbaa00c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 shutdown_connections 2021-02-10T11:29:11.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.115 7fbaa574c700 1 -- 172.21.2.117:0/3390570388 wait complete. 2021-02-10T11:29:11.114 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802468 2021-02-10T11:29:11.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.538 7f3cedd7b700 1 Processor -- start 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.538 7f3cedd7b700 1 -- start start 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3cedd7b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3ce807da20 0x7f3ce80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3cedd7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80e4d80 0x7f3ce81b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3cedd7b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce8077a70 con 0x7f3ce80e5180 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3cedd7b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce80777d0 con 0x7f3ce807da20 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3cedd7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce807de20 con 0x7f3ce80e4d80 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3ce80e5180 0x7f3ce8077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44308/0 (socket says 172.21.2.117:44308) 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce7fff700 1 -- 172.21.2.117:0/2387806416 learned_addr learned my addr 172.21.2.117:0/2387806416 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86760673 0 0) 0x7f3ce0000f90 con 0x7f3ce80e5180 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd0002fe0 con 0x7f3ce80e5180 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce77fe700 1 --2- 172.21.2.117:0/2387806416 >> v2:172.21.2.125:3300/0 conn(0x7f3ce807da20 0x7f3ce80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.539 7f3ce6ffd700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80e4d80 0x7f3ce81b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2983037814 0 0) 0x7f3ce0001270 con 0x7f3ce80e5180 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80e4d80 msgr2=0x7f3ce81b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80e4d80 0x7f3ce81b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 >> v2:172.21.2.125:3300/0 conn(0x7f3ce807da20 msgr2=0x7f3ce80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 --2- 172.21.2.117:0/2387806416 >> v2:172.21.2.125:3300/0 conn(0x7f3ce807da20 0x7f3ce80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce81b06f0 con 0x7f3ce80e5180 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3ce0001980 con 0x7f3ce80e5180 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 >> v1:172.21.2.117:6789/0 conn(0x7f3ce80e5180 legacy=0x7f3ce8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3cd0003a60 0x7f3cd0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3cd004a7a0 0x7f3cd004cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3cd004d5a0 0x7f3cd004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00502e0 con 0x7f3cd004a7a0 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0050500 con 0x7f3cd0003a60 2021-02-10T11:29:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0050720 con 0x7f3cd004d5a0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce6ffd700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3cd004a7a0 0x7f3cd004cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce7fff700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3cd004d5a0 0x7f3cd004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce77fe700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3cd0003a60 0x7f3cd0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce6ffd700 1 -- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3cd004d5a0 msgr2=0x7f3cd004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce6ffd700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3cd004d5a0 0x7f3cd004fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.540 7f3ce6ffd700 1 -- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3cd0003a60 msgr2=0x7f3cd0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce6ffd700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3cd0003a60 0x7f3cd0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce6ffd700 1 -- 172.21.2.117:0/2387806416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00509c0 con 0x7f3cd004a7a0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce77fe700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3cd0003a60 0x7f3cd0005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce6ffd700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3cd004a7a0 0x7f3cd004cca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f3cdc006ac0 tx=0x7f3cdc009ab0).ready entity=mon.0 client_cookie=6170207b0b0a00bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc00c0c0 con 0x7f3cd004a7a0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc00c3c0 con 0x7f3cd004a7a0 2021-02-10T11:29:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce67fc700 1 -- 172.21.2.117:0/2387806416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc00c820 con 0x7f3cd004a7a0 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3cedd7b700 1 -- 172.21.2.117:0/2387806416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3cd004a7a0 msgr2=0x7f3cd004cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3cedd7b700 1 --2- 172.21.2.117:0/2387806416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3cd004a7a0 0x7f3cd004cca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f3cdc006ac0 tx=0x7f3cdc009ab0).stop 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.541 7f3ce7fff700 1 -- 172.21.2.117:0/2387806416 reap_dead start 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.542 7f3cedd7b700 1 -- 172.21.2.117:0/2387806416 shutdown_connections 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.542 7f3cedd7b700 1 -- 172.21.2.117:0/2387806416 >> 172.21.2.117:0/2387806416 conn(0x7f3ce80ce430 msgr2=0x7f3ce80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.542 7f3cedd7b700 1 -- 172.21.2.117:0/2387806416 shutdown_connections 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.542 7f3cedd7b700 1 -- 172.21.2.117:0/2387806416 wait complete. 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.556 7f3cedd7b700 1 Processor -- start 2021-02-10T11:29:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.556 7f3cedd7b700 1 -- start start 2021-02-10T11:29:11.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80959f0 0x7f3ce8094aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce80a4a20 0x7f3ce80a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8190a00 con 0x7f3ce80951c0 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce80955c0 con 0x7f3ce80a4a20 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3cedd7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce80957e0 con 0x7f3ce80959f0 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44068/0 (socket says 172.21.2.117:44068) 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 -- 172.21.2.117:0/3795496136 learned_addr learned my addr 172.21.2.117:0/3795496136 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 -- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80959f0 msgr2=0x7f3ce8094aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80959f0 0x7f3ce8094aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 -- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce80a4a20 msgr2=0x7f3ce80a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce80a4a20 0x7f3ce80a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 -- 172.21.2.117:0/3795496136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce80a5440 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.557 7f3ce6ffd700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f3cdc0097a0 tx=0x7f3cdc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc00bd60 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc000940 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cdc0022b0 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce80a5440 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce80a59d0 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3cdc000c90 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 0x7f3cc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.558 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3cdc0326b0 con 0x7f3ce80951c0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.559 7f3ce77fe700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 0x7f3cc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:11.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.559 7f3ce77fe700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 0x7f3cc801abc0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:11.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.589 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce8077a70 con 0x7f3ce80951c0 2021-02-10T11:29:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.591 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3cdc030d10 con 0x7f3ce80951c0 2021-02-10T11:29:11.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.718 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802468", "target": ["mgr", ""]}) v1 -- 0x7f3ce80a59d0 con 0x7f3cc80186c0 2021-02-10T11:29:11.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.722 7f3ccffff700 1 -- 172.21.2.117:0/3795496136 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3cd8000960 con 0x7f3cc80186c0 2021-02-10T11:29:11.717 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 msgr2=0x7f3cc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 0x7f3cc801abc0 crc :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 msgr2=0x7f3ce81903e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f3cdc0097a0 tx=0x7f3cdc00b970).stop 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 shutdown_connections 2021-02-10T11:29:11.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc80186c0 0x7f3cc801abc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce80951c0 0x7f3ce81903e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce80a4a20 0x7f3ce80a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 --2- 172.21.2.117:0/3795496136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce80959f0 0x7f3ce8094aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 >> 172.21.2.117:0/3795496136 conn(0x7f3ce80a90a0 msgr2=0x7f3ce80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 shutdown_connections 2021-02-10T11:29:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:11.728 7f3cedd7b700 1 -- 172.21.2.117:0/3795496136 wait complete. 2021-02-10T11:29:11.727 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802468 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca 2021-02-10T11:29:11.727 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:11.727 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:11.727 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:12.062 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:12.063 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:12.063 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245398 s, 42.7 MB/s 2021-02-10T11:29:12.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:12.445 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:12.446 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:12.780 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:12.780 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:12.780 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:13.156 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:13.156 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248666 s, 42.2 MB/s 2021-02-10T11:29:13.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:13.157 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:13.157 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:13.492 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:13.492 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:13.492 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:13.870 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:13.870 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:13.870 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259526 s, 40.4 MB/s 2021-02-10T11:29:13.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:13.871 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:13.871 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:14.208 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:14.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:14.208 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:14.584 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:14.584 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:14.584 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242537 s, 43.2 MB/s 2021-02-10T11:29:14.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:14.585 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:14.585 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:14.921 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:14.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:14.921 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:15.297 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:15.297 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:15.297 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246197 s, 42.6 MB/s 2021-02-10T11:29:15.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:15.299 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:15.299 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:15.633 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:15.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:15.634 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:16.009 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:16.009 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:16.009 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232772 s, 45.0 MB/s 2021-02-10T11:29:16.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:16.010 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:16.010 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:16.345 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:16.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:16.346 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:16.720 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:16.721 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:16.721 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252367 s, 41.5 MB/s 2021-02-10T11:29:16.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:16.722 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:16.722 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:17.056 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:17.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:17.057 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:17.431 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:17.432 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:17.432 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253438 s, 41.4 MB/s 2021-02-10T11:29:17.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:17.433 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:17.433 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:17.769 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:17.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:17.769 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:18.142 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:18.143 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:18.143 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255432 s, 41.1 MB/s 2021-02-10T11:29:18.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:18.144 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:18.144 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:18.480 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:18.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:18.480 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802468/c75f8532-547d-4aa2-8196-76ec006498ca/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:18.854 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241522 s, 43.4 MB/s 2021-02-10T11:29:18.856 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802469 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 Processor -- start 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 -- start start 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12807da20 0x7fb1280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb1280e4da0 0x7fb1281b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1280e51a0 0x7fb128077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280e5c20 con 0x7fb1280e51a0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb128077a70 con 0x7fb1280e4da0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12f891700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280777d0 con 0x7fb12807da20 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12deae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1280e51a0 0x7fb128077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12deae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1280e51a0 0x7fb128077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44074/0 (socket says 172.21.2.117:44074) 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.278 7fb12deae700 1 -- 172.21.2.117:0/3005529704 learned_addr learned my addr 172.21.2.117:0/3005529704 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12ceac700 1 --2- 172.21.2.117:0/3005529704 >> v2:172.21.2.125:3300/0 conn(0x7fb1280e4da0 0x7fb1281b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12d6ad700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12807da20 0x7fb1280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 -- 172.21.2.117:0/3005529704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12807da20 msgr2=0x7fb1280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12807da20 0x7fb1280771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 -- 172.21.2.117:0/3005529704 >> v2:172.21.2.125:3300/0 conn(0x7fb1280e4da0 msgr2=0x7fb1281b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 --2- 172.21.2.117:0/3005529704 >> v2:172.21.2.125:3300/0 conn(0x7fb1280e4da0 0x7fb1281b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 -- 172.21.2.117:0/3005529704 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12807de20 con 0x7fb1280e51a0 2021-02-10T11:29:19.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12deae700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1280e51a0 0x7fb128077ff0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fb120006f00 tx=0x7fb120009f10).ready entity=mon.0 client_cookie=b56c1330bf126673 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb11bfff700 1 -- 172.21.2.117:0/3005529704 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12000b3b0 con 0x7fb1280e51a0 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb11bfff700 1 -- 172.21.2.117:0/3005529704 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12000b630 con 0x7fb1280e51a0 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb11bfff700 1 -- 172.21.2.117:0/3005529704 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb12000bb50 con 0x7fb1280e51a0 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12f891700 1 -- 172.21.2.117:0/3005529704 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1280e51a0 msgr2=0x7fb128077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.279 7fb12f891700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1280e51a0 0x7fb128077ff0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fb120006f00 tx=0x7fb120009f10).stop 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 -- 172.21.2.117:0/3005529704 shutdown_connections 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 --2- 172.21.2.117:0/3005529704 >> v2:172.21.2.125:3300/0 conn(0x7fb1280e4da0 0x7fb1281b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1280e51a0 0x7fb128077ff0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 --2- 172.21.2.117:0/3005529704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12807da20 0x7fb1280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 -- 172.21.2.117:0/3005529704 >> 172.21.2.117:0/3005529704 conn(0x7fb1280ce450 msgr2=0x7fb1280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 -- 172.21.2.117:0/3005529704 shutdown_connections 2021-02-10T11:29:19.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.280 7fb12f891700 1 -- 172.21.2.117:0/3005529704 wait complete. 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 Processor -- start 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 -- start start 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12808eb10 0x7fb1280c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1280afac0 0x7fb1280afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280af540 con 0x7fb12808dfe0 2021-02-10T11:29:19.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280a0c80 con 0x7fb1280afac0 2021-02-10T11:29:19.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12f891700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280b04e0 con 0x7fb12808eb10 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12ceac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12ceac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44080/0 (socket says 172.21.2.117:44080) 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.295 7fb12ceac700 1 -- 172.21.2.117:0/632992743 learned_addr learned my addr 172.21.2.117:0/632992743 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 -- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12808eb10 msgr2=0x7fb1280c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12808eb10 0x7fb1280c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 -- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1280afac0 msgr2=0x7fb1280afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1280afac0 0x7fb1280afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 -- 172.21.2.117:0/632992743 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280b0700 con 0x7fb12808dfe0 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12ceac700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fb124006f60 tx=0x7fb12400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1240009b0 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12400bdf0 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb124002030 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12f891700 1 -- 172.21.2.117:0/632992743 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1280b0700 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.296 7fb12f891700 1 -- 172.21.2.117:0/632992743 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1280b0c90 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.297 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb124006b00 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.297 7fb119ffb700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 0x7fb10801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.297 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb124032560 con 0x7fb12808dfe0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.297 7fb12d6ad700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 0x7fb10801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.298 7fb12d6ad700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 0x7fb10801abc0 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.328 7fb12f891700 1 -- 172.21.2.117:0/632992743 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb128103280 con 0x7fb12808dfe0 2021-02-10T11:29:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.331 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb1240009b0 con 0x7fb12808dfe0 2021-02-10T11:29:19.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.457 7fb12f891700 1 -- 172.21.2.117:0/632992743 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802469", "target": ["mgr", ""]}) v1 -- 0x7fb1280b0c90 con 0x7fb1080186c0 2021-02-10T11:29:19.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.483 7fb119ffb700 1 -- 172.21.2.117:0/632992743 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb11c0009b0 con 0x7fb1080186c0 2021-02-10T11:29:19.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.490 7fb12f891700 1 -- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 msgr2=0x7fb10801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.490 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 0x7fb10801abc0 crc :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.490 7fb12f891700 1 -- 172.21.2.117:0/632992743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 msgr2=0x7fb12808e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.490 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fb124006f60 tx=0x7fb12400b970).stop 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 -- 172.21.2.117:0/632992743 shutdown_connections 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1080186c0 0x7fb10801abc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb12808dfe0 0x7fb12808e3e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1280afac0 0x7fb1280afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 --2- 172.21.2.117:0/632992743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb12808eb10 0x7fb1280c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 -- 172.21.2.117:0/632992743 >> 172.21.2.117:0/632992743 conn(0x7fb1280a1850 msgr2=0x7fb1280bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:19.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 -- 172.21.2.117:0/632992743 shutdown_connections 2021-02-10T11:29:19.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.491 7fb12f891700 1 -- 172.21.2.117:0/632992743 wait complete. 2021-02-10T11:29:19.492 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802469 2021-02-10T11:29:19.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.914 7f4dd3053700 1 Processor -- start 2021-02-10T11:29:19.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.914 7f4dd3053700 1 -- start start 2021-02-10T11:29:19.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd3053700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4dcc07da20 0x7f4dcc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd3053700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc0d1d20 con 0x7f4dcc0e51a0 2021-02-10T11:29:19.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd3053700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc077a70 con 0x7f4dcc0e4da0 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd3053700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc0777d0 con 0x7f4dcc07da20 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd0e6f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4dcc07da20 0x7f4dcc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd1670700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4dcc0e51a0 0x7f4dcc0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44336/0 (socket says 172.21.2.117:44336) 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd1670700 1 -- 172.21.2.117:0/2449709857 learned_addr learned my addr 172.21.2.117:0/2449709857 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.915 7f4dd0e6f700 1 --2- 172.21.2.117:0/2449709857 >> v2:172.21.2.117:3300/0 conn(0x7f4dcc07da20 0x7f4dcc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44086/0 (socket says 172.21.2.117:44086) 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd0e6f700 1 -- 172.21.2.117:0/2449709857 >> v1:172.21.2.125:6789/0 conn(0x7f4dcc0e4da0 legacy=0x7f4dcc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd0e6f700 1 -- 172.21.2.117:0/2449709857 >> v1:172.21.2.117:6789/0 conn(0x7f4dcc0e51a0 legacy=0x7f4dcc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dcb7fe700 1 -- 172.21.2.117:0/2449709857 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581671200 0 0) 0x7f4dc4001180 con 0x7f4dcc0e51a0 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd0e6f700 1 -- 172.21.2.117:0/2449709857 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc1b08e0 con 0x7f4dcc07da20 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd0e6f700 1 --2- 172.21.2.117:0/2449709857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc07da20 0x7f4dcc0771b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f4db4006650 tx=0x7f4db4009a00).ready entity=mon.0 client_cookie=4fad271a2310f3d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dcb7fe700 1 -- 172.21.2.117:0/2449709857 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db40009b0 con 0x7f4dcc07da20 2021-02-10T11:29:19.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dcb7fe700 1 -- 172.21.2.117:0/2449709857 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db400c0c0 con 0x7f4dcc07da20 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dcb7fe700 1 -- 172.21.2.117:0/2449709857 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db400c5a0 con 0x7f4dcc07da20 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd3053700 1 -- 172.21.2.117:0/2449709857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc07da20 msgr2=0x7f4dcc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.916 7f4dd3053700 1 --2- 172.21.2.117:0/2449709857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc07da20 0x7f4dcc0771b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f4db4006650 tx=0x7f4db4009a00).stop 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.917 7f4dd3053700 1 -- 172.21.2.117:0/2449709857 shutdown_connections 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.917 7f4dd3053700 1 --2- 172.21.2.117:0/2449709857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc07da20 0x7f4dcc0771b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.917 7f4dd3053700 1 -- 172.21.2.117:0/2449709857 >> 172.21.2.117:0/2449709857 conn(0x7f4dcc0ce450 msgr2=0x7f4dcc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.917 7f4dd3053700 1 -- 172.21.2.117:0/2449709857 shutdown_connections 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.917 7f4dd3053700 1 -- 172.21.2.117:0/2449709857 wait complete. 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 Processor -- start 2021-02-10T11:29:19.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 -- start start 2021-02-10T11:29:19.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dcc09cb80 0x7f4dcc0c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dcc0b4250 0x7f4dcc0c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0bff40 con 0x7f4dcc09c780 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0bfb60 con 0x7f4dcc09cb80 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.932 7f4dd3053700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc0bfd40 con 0x7f4dcc0b4250 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44092/0 (socket says 172.21.2.117:44092) 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 -- 172.21.2.117:0/3255623450 learned_addr learned my addr 172.21.2.117:0/3255623450 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 -- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dcc0b4250 msgr2=0x7f4dcc0c0ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dcc0b4250 0x7f4dcc0c0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 -- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dcc09cb80 msgr2=0x7f4dcc0c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dcc09cb80 0x7f4dcc0c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 -- 172.21.2.117:0/3255623450 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc0c1100 con 0x7f4dcc09c780 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.933 7f4dd0e6f700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f4db400b590 tx=0x7f4db400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db400bd20 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4000c90 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dcc0c1100 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4db40022b0 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc0c1100 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4db4006a10 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 0x7f4db001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.934 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4db40328f0 con 0x7f4dcc09c780 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.935 7f4dcbfff700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 0x7f4db001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.935 7f4dcbfff700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 0x7f4db001ac10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:19.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.965 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc0ff170 con 0x7f4dcc09c780 2021-02-10T11:29:19.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:19.968 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4db4000c90 con 0x7f4dcc09c780 2021-02-10T11:29:20.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.094 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802469", "target": ["mgr", ""]}) v1 -- 0x7f4dcc077a70 con 0x7f4db0018730 2021-02-10T11:29:20.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.098 7f4dc97fa700 1 -- 172.21.2.117:0/3255623450 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4dbc0058a0 con 0x7f4db0018730 2021-02-10T11:29:20.093 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee 2021-02-10T11:29:20.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 msgr2=0x7f4db001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:20.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 0x7f4db001ac10 crc :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:20.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 msgr2=0x7f4dcc0b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:20.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f4db400b590 tx=0x7f4db400b930).stop 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 shutdown_connections 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dcc0b4250 0x7f4dcc0c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dcc09cb80 0x7f4dcc0c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4db0018730 0x7f4db001ac10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 --2- 172.21.2.117:0/3255623450 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dcc09c780 0x7f4dcc0b3c30 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.104 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 >> 172.21.2.117:0/3255623450 conn(0x7f4dcc09d3b0 msgr2=0x7f4dcc0b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.105 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 shutdown_connections 2021-02-10T11:29:20.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:20.105 7f4dd3053700 1 -- 172.21.2.117:0/3255623450 wait complete. 2021-02-10T11:29:20.103 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802469 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee 2021-02-10T11:29:20.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:20.104 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:20.104 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:20.439 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:20.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:20.439 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250827 s, 41.8 MB/s 2021-02-10T11:29:20.822 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:20.822 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:20.823 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:21.159 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:21.159 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:21.159 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:21.533 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:21.598 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:21.598 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228915 s, 45.8 MB/s 2021-02-10T11:29:21.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:21.599 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:21.600 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:21.946 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:21.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:21.946 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:22.326 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248983 s, 42.1 MB/s 2021-02-10T11:29:22.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:22.327 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:22.327 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:22.671 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:22.672 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:22.672 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:23.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:23.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:23.048 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232729 s, 45.1 MB/s 2021-02-10T11:29:23.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:23.049 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:23.049 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:23.392 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:23.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:23.393 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:23.769 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:23.769 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:23.769 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024926 s, 42.1 MB/s 2021-02-10T11:29:23.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:23.770 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:23.770 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:24.104 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:24.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:24.105 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:24.480 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:24.480 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:24.480 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244829 s, 42.8 MB/s 2021-02-10T11:29:24.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:24.481 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:24.481 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:24.816 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:24.816 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:24.816 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:25.191 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:25.191 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:25.191 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230687 s, 45.5 MB/s 2021-02-10T11:29:25.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:25.192 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:25.193 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:25.529 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:25.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:25.529 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:25.904 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:25.904 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:25.904 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231255 s, 45.3 MB/s 2021-02-10T11:29:25.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:25.905 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:25.905 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:26.240 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:26.241 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:26.241 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:26.617 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:26.617 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:26.617 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025299 s, 41.4 MB/s 2021-02-10T11:29:26.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:26.619 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:26.619 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:26.953 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:26.954 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:26.954 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802469/7d370075-7aa6-4212-8324-99b0f4c986ee/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:27.331 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:27.331 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:27.331 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245427 s, 42.7 MB/s 2021-02-10T11:29:27.332 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802470 2021-02-10T11:29:27.343 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:27.376 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.756 7f7e909bb700 1 Processor -- start 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.756 7f7e909bb700 1 -- start start 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e909bb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c07da20 0x7f7e8c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e909bb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8c0d1d20 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e909bb700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e8c077a70 con 0x7f7e8c0e4da0 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e909bb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0777d0 con 0x7f7e8c07da20 2021-02-10T11:29:27.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8ae1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7e8c0e51a0 0x7f7e8c0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44348/0 (socket says 172.21.2.117:44348) 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8ae1d700 1 -- 172.21.2.117:0/1308628193 learned_addr learned my addr 172.21.2.117:0/1308628193 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8a61c700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c07da20 0x7f7e8c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310358117 0 0) 0x7f7e84001500 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e78002fe0 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 823803241 0 0) 0x7f7e84001460 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c07da20 msgr2=0x7f7e8c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c07da20 0x7f7e8c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 >> v1:172.21.2.125:6789/0 conn(0x7f7e8c0e4da0 legacy=0x7f7e8c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c1b08e0 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7e84001980 con 0x7f7e8c0e51a0 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 >> v1:172.21.2.117:6789/0 conn(0x7f7e8c0e51a0 legacy=0x7f7e8c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e78003a70 0x7f7e78005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e78006780 0x7f7e78008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e780095a0 0x7f7e7800ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7800c2e0 con 0x7f7e78006780 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.757 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7800c500 con 0x7f7e78003a70 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78050730 con 0x7f7e780095a0 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e78006780 0x7f7e78008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8a61c700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e78003a70 0x7f7e78005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8ae1d700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e780095a0 0x7f7e7800ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 -- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e780095a0 msgr2=0x7f7e7800ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e780095a0 0x7f7e7800ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 -- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e78003a70 msgr2=0x7f7e78005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e78003a70 0x7f7e78005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 -- 172.21.2.117:0/1308628193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78050990 con 0x7f7e78006780 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8a61c700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e78003a70 0x7f7e78005ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e89e1b700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e78006780 0x7f7e78008ca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f7e7c006ac0 tx=0x7f7e7c009af0).ready entity=mon.0 client_cookie=d4e0e1849c6b68a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e7c007280 con 0x7f7e78006780 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c007580 con 0x7f7e78006780 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8961a700 1 -- 172.21.2.117:0/1308628193 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e7c0079e0 con 0x7f7e78006780 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e909bb700 1 -- 172.21.2.117:0/1308628193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e78006780 msgr2=0x7f7e78008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e909bb700 1 --2- 172.21.2.117:0/1308628193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e78006780 0x7f7e78008ca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f7e7c006ac0 tx=0x7f7e7c009af0).stop 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.758 7f7e8ae1d700 1 -- 172.21.2.117:0/1308628193 reap_dead start 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.759 7f7e909bb700 1 -- 172.21.2.117:0/1308628193 shutdown_connections 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.759 7f7e909bb700 1 -- 172.21.2.117:0/1308628193 >> 172.21.2.117:0/1308628193 conn(0x7f7e8c0ce450 msgr2=0x7f7e8c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.759 7f7e909bb700 1 -- 172.21.2.117:0/1308628193 shutdown_connections 2021-02-10T11:29:27.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.759 7f7e909bb700 1 -- 172.21.2.117:0/1308628193 wait complete. 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 Processor -- start 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 -- start start 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c0a47d0 0x7f7e8c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e8c0b77d0 0x7f7e8c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c189970 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c190aa0 con 0x7f7e8c0b77d0 2021-02-10T11:29:27.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e909bb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c0b81f0 con 0x7f7e8c0a47d0 2021-02-10T11:29:27.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44108/0 (socket says 172.21.2.117:44108) 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 -- 172.21.2.117:0/3987053853 learned_addr learned my addr 172.21.2.117:0/3987053853 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 -- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c0a47d0 msgr2=0x7f7e8c0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c0a47d0 0x7f7e8c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 -- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e8c0b77d0 msgr2=0x7f7e8c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e8c0b77d0 0x7f7e8c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.774 7f7e89e1b700 1 -- 172.21.2.117:0/3987053853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c0b8410 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e89e1b700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f7e7c00a680 tx=0x7f7e7c00c960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e7c006a80 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c006e00 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c0b8410 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c0b89a0 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e7c007210 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e7c006a80 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 0x7f7e7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.775 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7e7c032900 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.776 7f7e8a61c700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 0x7f7e7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.776 7f7e8a61c700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 0x7f7e7001ac10 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:27.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.807 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c077a70 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.810 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e7c0074d0 con 0x7f7e8c0a51f0 2021-02-10T11:29:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.934 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802470", "target": ["mgr", ""]}) v1 -- 0x7f7e8c0b81f0 con 0x7f7e70018730 2021-02-10T11:29:27.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.958 7f7e82ffd700 1 -- 172.21.2.117:0/3987053853 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7e74006e50 con 0x7f7e70018730 2021-02-10T11:29:27.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 msgr2=0x7f7e7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 0x7f7e7001ac10 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 msgr2=0x7f7e8c0a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f7e7c00a680 tx=0x7f7e7c00c960).stop 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 shutdown_connections 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e70018730 0x7f7e7001ac10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e8c0a51f0 0x7f7e8c0a55f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e8c0b77d0 0x7f7e8c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 --2- 172.21.2.117:0/3987053853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e8c0a47d0 0x7f7e8c0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 >> 172.21.2.117:0/3987053853 conn(0x7f7e8c0b5140 msgr2=0x7f7e8c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.967 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 shutdown_connections 2021-02-10T11:29:27.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:27.968 7f7e909bb700 1 -- 172.21.2.117:0/3987053853 wait complete. 2021-02-10T11:29:27.966 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802470 2021-02-10T11:29:28.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 Processor -- start 2021-02-10T11:29:28.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 -- start start 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 0x7f62440771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 0x7f62441b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6244077a70 con 0x7f62440e51a0 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62440777d0 con 0x7f624407da20 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.389 7f62494f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624407de20 con 0x7f62440e4da0 2021-02-10T11:29:28.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6243ac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f62440e51a0 0x7f6244077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44368/0 (socket says 172.21.2.117:44368) 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6243ac0700 1 -- 172.21.2.117:0/4074492249 learned_addr learned my addr 172.21.2.117:0/4074492249 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 --2- 172.21.2.117:0/4074492249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 0x7f62441b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62432bf700 1 --2- 172.21.2.117:0/4074492249 >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 0x7f62440771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62422bd700 1 -- 172.21.2.117:0/4074492249 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458627649 0 0) 0x7f6234001180 con 0x7f62440e51a0 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62422bd700 1 -- 172.21.2.117:0/4074492249 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6230002fe0 con 0x7f62440e51a0 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 -- 172.21.2.117:0/4074492249 >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 msgr2=0x7f62440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 --2- 172.21.2.117:0/4074492249 >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 0x7f62440771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 -- 172.21.2.117:0/4074492249 >> v1:172.21.2.117:6789/0 conn(0x7f62440e51a0 legacy=0x7f6244077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 -- 172.21.2.117:0/4074492249 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62441b0710 con 0x7f62440e4da0 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62432bf700 1 --2- 172.21.2.117:0/4074492249 >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 0x7f62440771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f6242abe700 1 --2- 172.21.2.117:0/4074492249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 0x7f62441b00f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f622c006650 tx=0x7f622c009a80).ready entity=mon.2 client_cookie=c172f475be58f39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62422bd700 1 -- 172.21.2.117:0/4074492249 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622c00c0c0 con 0x7f62440e4da0 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62422bd700 1 -- 172.21.2.117:0/4074492249 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c00c340 con 0x7f62440e4da0 2021-02-10T11:29:28.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62494f2700 1 -- 172.21.2.117:0/4074492249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 msgr2=0x7f62441b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62494f2700 1 --2- 172.21.2.117:0/4074492249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 0x7f62441b00f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f622c006650 tx=0x7f622c009a80).stop 2021-02-10T11:29:28.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.390 7f62422bd700 1 -- 172.21.2.117:0/4074492249 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622c00c7a0 con 0x7f62440e4da0 2021-02-10T11:29:28.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 -- 172.21.2.117:0/4074492249 shutdown_connections 2021-02-10T11:29:28.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 --2- 172.21.2.117:0/4074492249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62440e4da0 0x7f62441b00f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 --2- 172.21.2.117:0/4074492249 >> v2:172.21.2.125:3300/0 conn(0x7f624407da20 0x7f62440771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 -- 172.21.2.117:0/4074492249 >> 172.21.2.117:0/4074492249 conn(0x7f62440ce450 msgr2=0x7f62440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:28.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 -- 172.21.2.117:0/4074492249 shutdown_connections 2021-02-10T11:29:28.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.391 7f62494f2700 1 -- 172.21.2.117:0/4074492249 wait complete. 2021-02-10T11:29:28.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.405 7f62494f2700 1 Processor -- start 2021-02-10T11:29:28.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.405 7f62494f2700 1 -- start start 2021-02-10T11:29:28.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62441b5b50 0x7f62440b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62441b5f50 0x7f62440b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62440b8da0 con 0x7f62441b5130 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62440a4140 con 0x7f62441b5b50 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62494f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62440a4320 con 0x7f62441b5f50 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44124/0 (socket says 172.21.2.117:44124) 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 -- 172.21.2.117:0/2786517435 learned_addr learned my addr 172.21.2.117:0/2786517435 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f6242abe700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62441b5b50 0x7f62440b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f6243ac0700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62441b5f50 0x7f62440b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 -- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62441b5f50 msgr2=0x7f62440b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62441b5f50 0x7f62440b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 -- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62441b5b50 msgr2=0x7f62440b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62441b5b50 0x7f62440b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.406 7f62432bf700 1 -- 172.21.2.117:0/2786517435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62440b62b0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f62432bf700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f623c00b910 tx=0x7f623c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f623c00c2f0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c00c530 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f62494f2700 1 -- 172.21.2.117:0/2786517435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62440b62b0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f623c00ca10 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f62494f2700 1 -- 172.21.2.117:0/2786517435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62440b62b0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.407 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f623c009fe0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.408 7f623bfff700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 0x7f622801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.408 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f623c0328f0 con 0x7f62441b5130 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.408 7f6242abe700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 0x7f622801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:28.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.408 7f6242abe700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 0x7f622801abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:28.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.439 7f62494f2700 1 -- 172.21.2.117:0/2786517435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62441b4f00 con 0x7f62441b5130 2021-02-10T11:29:28.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.442 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f623c02dff0 con 0x7f62441b5130 2021-02-10T11:29:28.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.569 7f62494f2700 1 -- 172.21.2.117:0/2786517435 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802470", "target": ["mgr", ""]}) v1 -- 0x7f6244077a70 con 0x7f62280186c0 2021-02-10T11:29:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.572 7f623bfff700 1 -- 172.21.2.117:0/2786517435 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f622c009180 con 0x7f62280186c0 2021-02-10T11:29:28.567 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78 2021-02-10T11:29:28.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.578 7f62494f2700 1 -- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 msgr2=0x7f622801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.578 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 0x7f622801abc0 crc :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.578 7f62494f2700 1 -- 172.21.2.117:0/2786517435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 msgr2=0x7f62441b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:28.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.578 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f623c00b910 tx=0x7f623c00bcb0).stop 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 -- 172.21.2.117:0/2786517435 shutdown_connections 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62441b5f50 0x7f62440b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62441b5b50 0x7f62440b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62280186c0 0x7f622801abc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 --2- 172.21.2.117:0/2786517435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62441b5130 0x7f62441b5530 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 -- 172.21.2.117:0/2786517435 >> 172.21.2.117:0/2786517435 conn(0x7f624408bbc0 msgr2=0x7f62440c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 -- 172.21.2.117:0/2786517435 shutdown_connections 2021-02-10T11:29:28.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:28.579 7f62494f2700 1 -- 172.21.2.117:0/2786517435 wait complete. 2021-02-10T11:29:28.577 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802470 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78 2021-02-10T11:29:28.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:28.577 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:28.578 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:28.914 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:28.915 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:28.915 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:29.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:29.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:29.298 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250684 s, 41.8 MB/s 2021-02-10T11:29:29.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:29.299 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:29.299 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:29.634 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:29.635 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:29.635 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:30.009 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:30.009 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:30.009 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243287 s, 43.1 MB/s 2021-02-10T11:29:30.010 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:30.010 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:30.010 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:30.345 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:30.346 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:30.346 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247744 s, 42.3 MB/s 2021-02-10T11:29:30.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:30.723 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:30.723 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:31.059 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:31.060 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:31.060 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:31.434 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:31.434 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:31.434 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232631 s, 45.1 MB/s 2021-02-10T11:29:31.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:31.435 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:31.435 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:31.769 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:31.769 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:31.770 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:32.145 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:32.145 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:32.145 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250904 s, 41.8 MB/s 2021-02-10T11:29:32.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:32.146 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:32.146 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:32.481 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:32.481 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:32.482 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:32.852 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:32.852 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:32.852 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228228 s, 45.9 MB/s 2021-02-10T11:29:32.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:32.853 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:32.853 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:33.188 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:33.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:33.189 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:33.565 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:33.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:33.566 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255453 s, 41.0 MB/s 2021-02-10T11:29:33.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:33.567 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:33.567 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:33.901 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:33.901 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:33.901 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:34.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:34.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:34.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251873 s, 41.6 MB/s 2021-02-10T11:29:34.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:34.278 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:34.278 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:34.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:34.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:34.614 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:34.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:34.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:34.989 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242097 s, 43.3 MB/s 2021-02-10T11:29:34.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:34.991 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:34.991 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:35.325 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:35.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:35.326 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802470/e6d2cc32-2742-463d-84b8-7e31d5aacd78/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246631 s, 42.5 MB/s 2021-02-10T11:29:35.703 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802471 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 Processor -- start 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 -- start start 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c6807da20 0x7f7c680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7c680e4da0 0x7f7c681b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c68077a70 con 0x7f7c680e51a0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c680777d0 con 0x7f7c6807da20 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.128 7f7c6fe9c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c6807de20 con 0x7f7c680e4da0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c6807da20 0x7f7c680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7c6807da20 0x7f7c680771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44130/0 (socket says 172.21.2.117:44130) 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 -- 172.21.2.117:0/3672280477 learned_addr learned my addr 172.21.2.117:0/3672280477 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6d4b7700 1 --2- 172.21.2.117:0/3672280477 >> v2:172.21.2.125:3300/0 conn(0x7f7c680e4da0 0x7f7c681b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 -- 172.21.2.117:0/3672280477 >> v2:172.21.2.125:3300/0 conn(0x7f7c680e4da0 msgr2=0x7f7c681b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 --2- 172.21.2.117:0/3672280477 >> v2:172.21.2.125:3300/0 conn(0x7f7c680e4da0 0x7f7c681b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 -- 172.21.2.117:0/3672280477 >> v1:172.21.2.125:6789/0 conn(0x7f7c680e51a0 legacy=0x7f7c68077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 -- 172.21.2.117:0/3672280477 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c681b0710 con 0x7f7c6807da20 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6dcb8700 1 --2- 172.21.2.117:0/3672280477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c6807da20 0x7f7c680771b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f7c60006a80 tx=0x7f7c60009f00).ready entity=mon.0 client_cookie=97123deb80f7cf14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6ccb6700 1 -- 172.21.2.117:0/3672280477 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c6000b5c0 con 0x7f7c6807da20 2021-02-10T11:29:36.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6ccb6700 1 -- 172.21.2.117:0/3672280477 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6000b840 con 0x7f7c6807da20 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6ccb6700 1 -- 172.21.2.117:0/3672280477 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c6000bd60 con 0x7f7c6807da20 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6fe9c700 1 -- 172.21.2.117:0/3672280477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c6807da20 msgr2=0x7f7c680771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.129 7f7c6fe9c700 1 --2- 172.21.2.117:0/3672280477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c6807da20 0x7f7c680771b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f7c60006a80 tx=0x7f7c60009f00).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 -- 172.21.2.117:0/3672280477 shutdown_connections 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 --2- 172.21.2.117:0/3672280477 >> v2:172.21.2.125:3300/0 conn(0x7f7c680e4da0 0x7f7c681b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 --2- 172.21.2.117:0/3672280477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c6807da20 0x7f7c680771b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 -- 172.21.2.117:0/3672280477 >> 172.21.2.117:0/3672280477 conn(0x7f7c680ce450 msgr2=0x7f7c680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 -- 172.21.2.117:0/3672280477 shutdown_connections 2021-02-10T11:29:36.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.130 7f7c6fe9c700 1 -- 172.21.2.117:0/3672280477 wait complete. 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.145 7f7c6fe9c700 1 Processor -- start 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.145 7f7c6fe9c700 1 -- start start 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.145 7f7c6fe9c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c68187bf0 0x7f7c680a7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.145 7f7c6fe9c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c680a7f40 0x7f7c680a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6fe9c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6fe9c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68187ff0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6fe9c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6819beb0 con 0x7f7c68187bf0 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6fe9c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681902d0 con 0x7f7c680a7f40 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44140/0 (socket says 172.21.2.117:44140) 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 -- 172.21.2.117:0/291266042 learned_addr learned my addr 172.21.2.117:0/291266042 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:36.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6d4b7700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c680a7f40 0x7f7c680a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6dcb8700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c68187bf0 0x7f7c680a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 -- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c680a7f40 msgr2=0x7f7c680a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c680a7f40 0x7f7c680a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 -- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c68187bf0 msgr2=0x7f7c680a7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c68187bf0 0x7f7c680a7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 -- 172.21.2.117:0/291266042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680a72e0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c6e4b9700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f7c58006ed0 tx=0x7f7c5800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.146 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c5800bd50 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c58006020 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7c580054b0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c680a72e0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c680a72e0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7c58006200 con 0x7f7c680a6ee0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c667fc700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 0x7f7c4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.147 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7c580326d0 con 0x7f7c680a6ee0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.148 7f7c6dcb8700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 0x7f7c4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.148 7f7c6dcb8700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 0x7f7c4801abc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.178 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68077a70 con 0x7f7c680a6ee0 2021-02-10T11:29:36.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.181 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7c5801c010 con 0x7f7c680a6ee0 2021-02-10T11:29:36.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.306 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802471", "target": ["mgr", ""]}) v1 -- 0x7f7c68077a70 con 0x7f7c480186c0 2021-02-10T11:29:36.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.333 7f7c667fc700 1 -- 172.21.2.117:0/291266042 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7c60000d30 con 0x7f7c480186c0 2021-02-10T11:29:36.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 msgr2=0x7f7c4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 0x7f7c4801abc0 crc :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 msgr2=0x7f7c6818f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f7c58006ed0 tx=0x7f7c5800b960).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 shutdown_connections 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7c480186c0 0x7f7c4801abc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7c680a6ee0 0x7f7c6818f010 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7c680a7f40 0x7f7c680a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 --2- 172.21.2.117:0/291266042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7c68187bf0 0x7f7c680a7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 >> 172.21.2.117:0/291266042 conn(0x7f7c680a1360 msgr2=0x7f7c6808d560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 shutdown_connections 2021-02-10T11:29:36.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.340 7f7c6fe9c700 1 -- 172.21.2.117:0/291266042 wait complete. 2021-02-10T11:29:36.338 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802471 2021-02-10T11:29:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.761 7f3faa41a700 1 Processor -- start 2021-02-10T11:29:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 -- start start 2021-02-10T11:29:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fa407da20 0x7f3fa40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3fa40e4da0 0x7f3fa41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa4077a70 con 0x7f3fa40e51a0 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa40777d0 con 0x7f3fa407da20 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3faa41a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa407de20 con 0x7f3fa40e4da0 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3fa3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fa407da20 0x7f3fa40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3fa3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fa407da20 0x7f3fa40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44144/0 (socket says 172.21.2.117:44144) 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3fa3fff700 1 -- 172.21.2.117:0/2589882974 learned_addr learned my addr 172.21.2.117:0/2589882974 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.762 7f3fa37fe700 1 --2- 172.21.2.117:0/2589882974 >> v2:172.21.2.125:3300/0 conn(0x7f3fa40e4da0 0x7f3fa41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa3fff700 1 -- 172.21.2.117:0/2589882974 >> v2:172.21.2.125:3300/0 conn(0x7f3fa40e4da0 msgr2=0x7f3fa41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa3fff700 1 --2- 172.21.2.117:0/2589882974 >> v2:172.21.2.125:3300/0 conn(0x7f3fa40e4da0 0x7f3fa41b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa3fff700 1 -- 172.21.2.117:0/2589882974 >> v1:172.21.2.125:6789/0 conn(0x7f3fa40e51a0 legacy=0x7f3fa4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa3fff700 1 -- 172.21.2.117:0/2589882974 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa41b0710 con 0x7f3fa407da20 2021-02-10T11:29:36.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa3fff700 1 --2- 172.21.2.117:0/2589882974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa407da20 0x7f3fa40771b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f3f94006700 tx=0x7f3f94009b00).ready entity=mon.0 client_cookie=515bb4e67215982e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa2ffd700 1 -- 172.21.2.117:0/2589882974 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9400c0c0 con 0x7f3fa407da20 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa2ffd700 1 -- 172.21.2.117:0/2589882974 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9400c340 con 0x7f3fa407da20 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3fa2ffd700 1 -- 172.21.2.117:0/2589882974 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9400c820 con 0x7f3fa407da20 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3faa41a700 1 -- 172.21.2.117:0/2589882974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa407da20 msgr2=0x7f3fa40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.763 7f3faa41a700 1 --2- 172.21.2.117:0/2589882974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa407da20 0x7f3fa40771b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f3f94006700 tx=0x7f3f94009b00).stop 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 -- 172.21.2.117:0/2589882974 shutdown_connections 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 --2- 172.21.2.117:0/2589882974 >> v2:172.21.2.125:3300/0 conn(0x7f3fa40e4da0 0x7f3fa41b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 --2- 172.21.2.117:0/2589882974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa407da20 0x7f3fa40771b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 -- 172.21.2.117:0/2589882974 >> 172.21.2.117:0/2589882974 conn(0x7f3fa40ce450 msgr2=0x7f3fa40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 -- 172.21.2.117:0/2589882974 shutdown_connections 2021-02-10T11:29:36.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.764 7f3faa41a700 1 -- 172.21.2.117:0/2589882974 wait complete. 2021-02-10T11:29:36.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.778 7f3faa41a700 1 Processor -- start 2021-02-10T11:29:36.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.778 7f3faa41a700 1 -- start start 2021-02-10T11:29:36.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fa4187e20 0x7f3fa4188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 0x7f3fa4090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa41067f0 con 0x7f3fa41a0250 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4107350 con 0x7f3fa4090420 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa41042d0 con 0x7f3fa4187e20 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44150/0 (socket says 172.21.2.117:44150) 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 -- 172.21.2.117:0/1610365556 learned_addr learned my addr 172.21.2.117:0/1610365556 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa37fe700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fa4187e20 0x7f3fa4188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa8a37700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 0x7f3fa4090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 -- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fa4187e20 msgr2=0x7f3fa4188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fa4187e20 0x7f3fa4188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 -- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 msgr2=0x7f3fa4090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 0x7f3fa4090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 -- 172.21.2.117:0/1610365556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa4090e40 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa8a37700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 0x7f3fa4090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa3fff700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f3f94006f20 tx=0x7f3f9400b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f9400bd00 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f94006390 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.779 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa4090e40 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f94005560 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa4090e40 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f940052e0 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3fa0ff9700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 0x7f3f8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f940328f0 con 0x7f3fa41a0250 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.780 7f3fa37fe700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 0x7f3f8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:36.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.781 7f3fa37fe700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 0x7f3f8401abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:36.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.812 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa4103280 con 0x7f3fa41a0250 2021-02-10T11:29:36.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.815 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f940052e0 con 0x7f3fa41a0250 2021-02-10T11:29:36.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.942 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802471", "target": ["mgr", ""]}) v1 -- 0x7f3fa4107310 con 0x7f3f84018730 2021-02-10T11:29:36.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.946 7f3fa0ff9700 1 -- 172.21.2.117:0/1610365556 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3f98006c80 con 0x7f3f84018730 2021-02-10T11:29:36.940 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b 2021-02-10T11:29:36.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 msgr2=0x7f3f8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 0x7f3f8401abc0 crc :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 msgr2=0x7f3fa4106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f3f94006f20 tx=0x7f3f9400b910).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 shutdown_connections 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fa4090420 0x7f3fa4090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fa4187e20 0x7f3fa4188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f84018730 0x7f3f8401abc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 --2- 172.21.2.117:0/1610365556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fa41a0250 0x7f3fa4106cf0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:36.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 >> 172.21.2.117:0/1610365556 conn(0x7f3fa40a1850 msgr2=0x7f3fa40c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:36.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 shutdown_connections 2021-02-10T11:29:36.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:36.951 7f3faa41a700 1 -- 172.21.2.117:0/1610365556 wait complete. 2021-02-10T11:29:36.950 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802471 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b 2021-02-10T11:29:36.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:36.950 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:36.950 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:37.286 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:37.287 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:37.287 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:37.667 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:37.667 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:37.667 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262057 s, 40.0 MB/s 2021-02-10T11:29:37.668 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:37.668 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:37.668 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:38.002 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:38.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:38.003 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:38.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:38.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:38.379 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256401 s, 40.9 MB/s 2021-02-10T11:29:38.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:38.380 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:38.380 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:38.716 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:38.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:38.717 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:39.095 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:39.095 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:39.095 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257279 s, 40.8 MB/s 2021-02-10T11:29:39.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:39.096 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:39.096 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:39.430 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:39.430 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:39.430 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:39.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:39.806 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:39.806 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023596 s, 44.4 MB/s 2021-02-10T11:29:39.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:39.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:39.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:40.142 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:40.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:40.143 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:40.518 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:40.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:40.519 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250622 s, 41.8 MB/s 2021-02-10T11:29:40.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:40.520 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:40.520 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:40.854 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:40.855 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:40.855 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:41.229 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:41.229 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:41.229 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232599 s, 45.1 MB/s 2021-02-10T11:29:41.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:41.230 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:41.230 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:41.564 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:41.565 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:41.565 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:41.940 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:41.940 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:41.940 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237519 s, 44.1 MB/s 2021-02-10T11:29:41.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:41.941 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:41.941 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:42.277 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:42.277 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:42.277 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:42.649 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:42.650 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248447 s, 42.2 MB/s 2021-02-10T11:29:42.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:42.651 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:42.651 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:42.987 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:42.987 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:42.988 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:43.364 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:43.364 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:43.364 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250611 s, 41.8 MB/s 2021-02-10T11:29:43.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:43.365 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:43.365 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:43.701 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:43.702 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:43.702 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802471/2d82711a-18e2-4e32-9e6c-37d828a0376b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:44.078 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:44.078 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:44.078 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253735 s, 41.3 MB/s 2021-02-10T11:29:44.079 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802472 2021-02-10T11:29:44.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.503 7f440668c700 1 Processor -- start 2021-02-10T11:29:44.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.503 7f440668c700 1 -- start start 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f440668c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f440007da20 0x7f44000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f440668c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44000d1d20 con 0x7f44000e4da0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f440668c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4400077a70 con 0x7f44000e51a0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f440668c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000777d0 con 0x7f440007da20 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f43ff7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f44000e4da0 0x7f44001b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44408/0 (socket says 172.21.2.117:44408) 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f43ff7fe700 1 -- 172.21.2.117:0/3879807299 learned_addr learned my addr 172.21.2.117:0/3879807299 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f43fffff700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f440007da20 0x7f44000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061114211 0 0) 0x7f43f40010b0 con 0x7f44000e4da0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.504 7f43feffd700 1 -- 172.21.2.117:0/3879807299 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8002fe0 con 0x7f44000e4da0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3306126018 0 0) 0x7f43f40010b0 con 0x7f44000e4da0 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f440007da20 msgr2=0x7f44000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f440007da20 0x7f44000771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 >> v1:172.21.2.125:6789/0 conn(0x7f44000e51a0 legacy=0x7f44000d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44001b08e0 con 0x7f44000e4da0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f43f40015b0 con 0x7f44000e4da0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 >> v1:172.21.2.117:6789/0 conn(0x7f44000e4da0 legacy=0x7f44001b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e8003b20 0x7f43e8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e80067b0 0x7f43e8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e8009590 0x7f43e800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e800c2d0 con 0x7f43e80067b0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e800c4f0 con 0x7f43e8003b20 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e800c710 con 0x7f43e8009590 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e80067b0 0x7f43e8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f4404ca9700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e8009590 0x7f43e800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43fffff700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e8003b20 0x7f43e8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 -- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e8009590 msgr2=0x7f43e800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e8009590 0x7f43e800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 -- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e8003b20 msgr2=0x7f43e8005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e8003b20 0x7f43e8005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 -- 172.21.2.117:0/3879807299 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8050980 con 0x7f43e80067b0 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43ff7fe700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e80067b0 0x7f43e8008c90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f43f4001380 tx=0x7f43f4007ce0).ready entity=mon.0 client_cookie=1349e36062b670d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.505 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f40010b0 con 0x7f43e80067b0 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f400a520 con 0x7f43e80067b0 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f43feffd700 1 -- 172.21.2.117:0/3879807299 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f400a9c0 con 0x7f43e80067b0 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 -- 172.21.2.117:0/3879807299 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e80067b0 msgr2=0x7f43e8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 --2- 172.21.2.117:0/3879807299 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e80067b0 0x7f43e8008c90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f43f4001380 tx=0x7f43f4007ce0).stop 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f4404ca9700 1 -- 172.21.2.117:0/3879807299 reap_dead start 2021-02-10T11:29:44.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 -- 172.21.2.117:0/3879807299 shutdown_connections 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 -- 172.21.2.117:0/3879807299 >> 172.21.2.117:0/3879807299 conn(0x7f44000ce450 msgr2=0x7f44000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 -- 172.21.2.117:0/3879807299 shutdown_connections 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.506 7f440668c700 1 -- 172.21.2.117:0/3879807299 wait complete. 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 Processor -- start 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 -- start start 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44000a47d0 0x7f44000a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44000b77d0 0x7f44000b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400189970 con 0x7f44000a51f0 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400190aa0 con 0x7f44000b77d0 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.521 7f440668c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44000b81f0 con 0x7f44000a47d0 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44170/0 (socket says 172.21.2.117:44170) 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 -- 172.21.2.117:0/2633852570 learned_addr learned my addr 172.21.2.117:0/2633852570 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43fffff700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44000a47d0 0x7f44000a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f4404ca9700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44000b77d0 0x7f44000b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 -- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44000a47d0 msgr2=0x7f44000a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44000a47d0 0x7f44000a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 -- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44000b77d0 msgr2=0x7f44000b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44000b77d0 0x7f44000b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 -- 172.21.2.117:0/2633852570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44000b8410 con 0x7f44000a51f0 2021-02-10T11:29:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43ff7fe700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f43f4001c10 tx=0x7f43f400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f400acc0 con 0x7f44000a51f0 2021-02-10T11:29:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f4004140 con 0x7f44000a51f0 2021-02-10T11:29:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f440668c700 1 -- 172.21.2.117:0/2633852570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44000b8410 con 0x7f44000a51f0 2021-02-10T11:29:44.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f440668c700 1 -- 172.21.2.117:0/2633852570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44000b89a0 con 0x7f44000a51f0 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.522 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43f4004660 con 0x7f44000a51f0 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.523 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43f4004380 con 0x7f44000a51f0 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.523 7f43fcff9700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 0x7f43e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.523 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f43f4032910 con 0x7f44000a51f0 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.523 7f43fffff700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 0x7f43e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:44.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.524 7f43fffff700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 0x7f43e001abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:44.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.554 7f440668c700 1 -- 172.21.2.117:0/2633852570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400077a70 con 0x7f44000a51f0 2021-02-10T11:29:44.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.557 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43f4004140 con 0x7f44000a51f0 2021-02-10T11:29:44.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.682 7f440668c700 1 -- 172.21.2.117:0/2633852570 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802472", "target": ["mgr", ""]}) v1 -- 0x7f4400189970 con 0x7f43e00186c0 2021-02-10T11:29:44.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.709 7f43fcff9700 1 -- 172.21.2.117:0/2633852570 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f43f00009b0 con 0x7f43e00186c0 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.715 7f440668c700 1 -- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 msgr2=0x7f43e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.715 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 0x7f43e001abc0 crc :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 -- 172.21.2.117:0/2633852570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 msgr2=0x7f44000a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f43f4001c10 tx=0x7f43f400a8d0).stop 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 -- 172.21.2.117:0/2633852570 shutdown_connections 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43e00186c0 0x7f43e001abc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44000a51f0 0x7f44000a55f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44000b77d0 0x7f44000b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 --2- 172.21.2.117:0/2633852570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44000a47d0 0x7f44000a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:44.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 -- 172.21.2.117:0/2633852570 >> 172.21.2.117:0/2633852570 conn(0x7f44000b5140 msgr2=0x7f44000a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:44.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 -- 172.21.2.117:0/2633852570 shutdown_connections 2021-02-10T11:29:44.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:44.716 7f440668c700 1 -- 172.21.2.117:0/2633852570 wait complete. 2021-02-10T11:29:44.714 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802472 2021-02-10T11:29:45.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 Processor -- start 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 -- start start 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f454007da20 0x7f45400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 0x7f4540079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4540077a70 con 0x7f45400e4da0 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45400e5c20 con 0x7f454007da20 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4546f6e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45400777d0 con 0x7f45400e51a0 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f4544d8a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f454007da20 0x7f45400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f453ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f45400e4da0 0x7f45401b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44426/0 (socket says 172.21.2.117:44426) 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f453ffff700 1 -- 172.21.2.117:0/1953804252 learned_addr learned my addr 172.21.2.117:0/1953804252 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:29:45.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.138 7f454558b700 1 --2- 172.21.2.117:0/1953804252 >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 0x7f4540079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f453f7fe700 1 -- 172.21.2.117:0/1953804252 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212528471 0 0) 0x7f4530001060 con 0x7f45400e4da0 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f453f7fe700 1 -- 172.21.2.117:0/1953804252 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c002fe0 con 0x7f45400e4da0 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4544d8a700 1 -- 172.21.2.117:0/1953804252 >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 msgr2=0x7f4540079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4544d8a700 1 --2- 172.21.2.117:0/1953804252 >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 0x7f4540079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4544d8a700 1 -- 172.21.2.117:0/1953804252 >> v1:172.21.2.117:6789/0 conn(0x7f45400e4da0 legacy=0x7f45401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4544d8a700 1 -- 172.21.2.117:0/1953804252 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45401b08e0 con 0x7f454007da20 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f454558b700 1 --2- 172.21.2.117:0/1953804252 >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 0x7f4540079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4544d8a700 1 --2- 172.21.2.117:0/1953804252 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f454007da20 0x7f45400771b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f4528006650 tx=0x7f4528009a80).ready entity=mon.0 client_cookie=635ce1b6405b0ec6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f453f7fe700 1 -- 172.21.2.117:0/1953804252 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45280009b0 con 0x7f454007da20 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f453f7fe700 1 -- 172.21.2.117:0/1953804252 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452800c0c0 con 0x7f454007da20 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f453f7fe700 1 -- 172.21.2.117:0/1953804252 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f452800c520 con 0x7f454007da20 2021-02-10T11:29:45.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4546f6e700 1 -- 172.21.2.117:0/1953804252 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f454007da20 msgr2=0x7f45400771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.139 7f4546f6e700 1 --2- 172.21.2.117:0/1953804252 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f454007da20 0x7f45400771b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f4528006650 tx=0x7f4528009a80).stop 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 -- 172.21.2.117:0/1953804252 shutdown_connections 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 --2- 172.21.2.117:0/1953804252 >> v2:172.21.2.125:3300/0 conn(0x7f45400e51a0 0x7f4540079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 --2- 172.21.2.117:0/1953804252 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f454007da20 0x7f45400771b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 -- 172.21.2.117:0/1953804252 >> 172.21.2.117:0/1953804252 conn(0x7f45400ce450 msgr2=0x7f45400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 -- 172.21.2.117:0/1953804252 shutdown_connections 2021-02-10T11:29:45.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.140 7f4546f6e700 1 -- 172.21.2.117:0/1953804252 wait complete. 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.154 7f4546f6e700 1 Processor -- start 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.154 7f4546f6e700 1 -- start start 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4540195b40 0x7f4540195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4540105870 0x7f4540105c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4540106500 con 0x7f4540104690 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45401b63f0 con 0x7f4540105870 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4546f6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45401b65d0 con 0x7f4540195b40 2021-02-10T11:29:45.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44184/0 (socket says 172.21.2.117:44184) 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 -- 172.21.2.117:0/694600108 learned_addr learned my addr 172.21.2.117:0/694600108 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 -- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4540195b40 msgr2=0x7f4540195f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4540195b40 0x7f4540195f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 -- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4540105870 msgr2=0x7f4540105c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.155 7f4544d8a700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4540105870 0x7f4540105c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f4544d8a700 1 -- 172.21.2.117:0/694600108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540106a80 con 0x7f4540104690 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f4544d8a700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f4528009630 tx=0x7f452800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45280093a0 con 0x7f4540104690 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452800bd70 con 0x7f4540104690 2021-02-10T11:29:45.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45280054d0 con 0x7f4540104690 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f4546f6e700 1 -- 172.21.2.117:0/694600108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4540106a80 con 0x7f4540104690 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f4546f6e700 1 -- 172.21.2.117:0/694600108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4540107010 con 0x7f4540104690 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.156 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45280052e0 con 0x7f4540104690 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.157 7f453d7fa700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 0x7f452401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.157 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f45280326d0 con 0x7f4540104690 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.157 7f453ffff700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 0x7f452401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:45.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.157 7f453ffff700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 0x7f452401abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:45.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.188 7f4546f6e700 1 -- 172.21.2.117:0/694600108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45401b4f00 con 0x7f4540104690 2021-02-10T11:29:45.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.191 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4528009340 con 0x7f4540104690 2021-02-10T11:29:45.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.316 7f4546f6e700 1 -- 172.21.2.117:0/694600108 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802472", "target": ["mgr", ""]}) v1 -- 0x7f4540077a70 con 0x7f45240186c0 2021-02-10T11:29:45.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.320 7f453d7fa700 1 -- 172.21.2.117:0/694600108 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4530001010 con 0x7f45240186c0 2021-02-10T11:29:45.314 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7 2021-02-10T11:29:45.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.325 7f4546f6e700 1 -- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 msgr2=0x7f452401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.325 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 0x7f452401abc0 crc :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.325 7f4546f6e700 1 -- 172.21.2.117:0/694600108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 msgr2=0x7f45401b67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.325 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f4528009630 tx=0x7f452800b980).stop 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 -- 172.21.2.117:0/694600108 shutdown_connections 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4540105870 0x7f4540105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:45.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4540195b40 0x7f4540195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:45.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45240186c0 0x7f452401abc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 --2- 172.21.2.117:0/694600108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4540104690 0x7f45401b67e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:45.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 -- 172.21.2.117:0/694600108 >> 172.21.2.117:0/694600108 conn(0x7f45400ffe30 msgr2=0x7f45400c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:45.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 -- 172.21.2.117:0/694600108 shutdown_connections 2021-02-10T11:29:45.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:45.326 7f4546f6e700 1 -- 172.21.2.117:0/694600108 wait complete. 2021-02-10T11:29:45.324 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802472 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7 2021-02-10T11:29:45.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:45.324 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:45.324 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:45.660 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:45.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:45.660 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:46.044 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:46.044 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:46.044 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259204 s, 40.5 MB/s 2021-02-10T11:29:46.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:46.045 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:46.046 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:46.381 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:46.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:46.382 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258047 s, 40.6 MB/s 2021-02-10T11:29:46.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:46.760 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:46.760 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:47.094 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:47.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:47.094 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247425 s, 42.4 MB/s 2021-02-10T11:29:47.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:47.467 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:47.467 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:47.802 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:47.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:47.803 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:48.178 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:48.179 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:48.179 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245376 s, 42.7 MB/s 2021-02-10T11:29:48.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:48.180 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:48.180 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:48.514 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:48.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:48.515 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:48.889 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:48.889 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:48.890 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232241 s, 45.2 MB/s 2021-02-10T11:29:48.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:48.891 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:48.891 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:49.225 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:49.226 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:49.226 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:49.601 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:49.601 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:49.601 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243687 s, 43.0 MB/s 2021-02-10T11:29:49.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:49.602 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:49.602 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:49.937 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:49.937 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:49.938 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:50.311 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:50.311 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:50.311 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231974 s, 45.2 MB/s 2021-02-10T11:29:50.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:50.312 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:50.312 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:50.647 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:50.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:50.647 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:51.023 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:51.023 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:51.023 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238009 s, 44.1 MB/s 2021-02-10T11:29:51.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:51.025 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:51.025 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:51.361 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:51.361 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:51.362 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:51.739 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:51.739 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258531 s, 40.6 MB/s 2021-02-10T11:29:51.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:51.740 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:51.740 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:52.074 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:52.074 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:52.075 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802472/6c9f3069-5c69-40a1-878f-235a706823c7/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:52.447 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:52.448 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:52.448 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240994 s, 43.5 MB/s 2021-02-10T11:29:52.449 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802473 2021-02-10T11:29:52.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.876 7f7d433bd700 1 Processor -- start 2021-02-10T11:29:52.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.876 7f7d433bd700 1 -- start start 2021-02-10T11:29:52.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c07da20 0x7f7d3c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7d3c0e51a0 0x7f7d3c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c0e5c20 con 0x7f7d3c0e4da0 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c077a70 con 0x7f7d3c0e51a0 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d433bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0777d0 con 0x7f7d3c07da20 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44188/0 (socket says 172.21.2.117:44188) 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 -- 172.21.2.117:0/199833091 learned_addr learned my addr 172.21.2.117:0/199833091 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 -- 172.21.2.117:0/199833091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c07da20 msgr2=0x7f7d3c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 --2- 172.21.2.117:0/199833091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c07da20 0x7f7d3c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 -- 172.21.2.117:0/199833091 >> v2:172.21.2.125:3300/0 conn(0x7f7d3c0e51a0 msgr2=0x7f7d3c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 --2- 172.21.2.117:0/199833091 >> v2:172.21.2.125:3300/0 conn(0x7f7d3c0e51a0 0x7f7d3c077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 -- 172.21.2.117:0/199833091 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c07de20 con 0x7f7d3c0e4da0 2021-02-10T11:29:52.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.877 7f7d409d8700 1 --2- 172.21.2.117:0/199833091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f7d30006700 tx=0x7f7d30009b00).ready entity=mon.0 client_cookie=f848728f57945ccb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d2ffff700 1 -- 172.21.2.117:0/199833091 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3000c0c0 con 0x7f7d3c0e4da0 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d2ffff700 1 -- 172.21.2.117:0/199833091 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000c400 con 0x7f7d3c0e4da0 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d2ffff700 1 -- 172.21.2.117:0/199833091 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3000c8e0 con 0x7f7d3c0e4da0 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 -- 172.21.2.117:0/199833091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0e4da0 msgr2=0x7f7d3c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 --2- 172.21.2.117:0/199833091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f7d30006700 tx=0x7f7d30009b00).stop 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 -- 172.21.2.117:0/199833091 shutdown_connections 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 --2- 172.21.2.117:0/199833091 >> v2:172.21.2.125:3300/0 conn(0x7f7d3c0e51a0 0x7f7d3c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 --2- 172.21.2.117:0/199833091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0e4da0 0x7f7d3c1b00f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 --2- 172.21.2.117:0/199833091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c07da20 0x7f7d3c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.878 7f7d433bd700 1 -- 172.21.2.117:0/199833091 >> 172.21.2.117:0/199833091 conn(0x7f7d3c0ce450 msgr2=0x7f7d3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.879 7f7d433bd700 1 -- 172.21.2.117:0/199833091 shutdown_connections 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.879 7f7d433bd700 1 -- 172.21.2.117:0/199833091 wait complete. 2021-02-10T11:29:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 Processor -- start 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 -- start start 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d3c18a2a0 0x7f7d3c1a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c18a6a0 0x7f7d3c1a3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c190970 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0a4ac0 con 0x7f7d3c18a2a0 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d433bd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0a4ce0 con 0x7f7d3c18a6a0 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.893 7f7d411d9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44192/0 (socket says 172.21.2.117:44192) 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 -- 172.21.2.117:0/3801196694 learned_addr learned my addr 172.21.2.117:0/3801196694 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d419da700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c18a6a0 0x7f7d3c1a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d409d8700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d3c18a2a0 0x7f7d3c1a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 -- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c18a6a0 msgr2=0x7f7d3c1a3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c18a6a0 0x7f7d3c1a3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 -- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d3c18a2a0 msgr2=0x7f7d3c1a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d3c18a2a0 0x7f7d3c1a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 -- 172.21.2.117:0/3801196694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c1a3850 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d411d9700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f7d3800b5c0 tx=0x7f7d3800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d380009b0 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38000c80 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d38006240 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c1a3850 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.894 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c1a3850 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.895 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d38006030 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.895 7f7d2dffb700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.895 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7d38032910 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.895 7f7d409d8700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.896 7f7d409d8700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:52.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.927 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c103340 con 0x7f7d3c0a46c0 2021-02-10T11:29:52.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:52.929 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d38006240 con 0x7f7d3c0a46c0 2021-02-10T11:29:53.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.055 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802473", "target": ["mgr", ""]}) v1 -- 0x7f7d3c0a4ac0 con 0x7f7d1c0186c0 2021-02-10T11:29:53.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.080 7f7d2dffb700 1 -- 172.21.2.117:0/3801196694 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7d30006410 con 0x7f7d1c0186c0 2021-02-10T11:29:53.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 msgr2=0x7f7d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 crc :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 msgr2=0x7f7d3c189c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f7d3800b5c0 tx=0x7f7d3800b960).stop 2021-02-10T11:29:53.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 shutdown_connections 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d3c18a6a0 0x7f7d3c1a3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d3c18a2a0 0x7f7d3c1a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d1c0186c0 0x7f7d1c01abc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 --2- 172.21.2.117:0/3801196694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d3c0a46c0 0x7f7d3c189c80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 >> 172.21.2.117:0/3801196694 conn(0x7f7d3c0a90a0 msgr2=0x7f7d3c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 shutdown_connections 2021-02-10T11:29:53.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.088 7f7d433bd700 1 -- 172.21.2.117:0/3801196694 wait complete. 2021-02-10T11:29:53.087 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802473 2021-02-10T11:29:53.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 Processor -- start 2021-02-10T11:29:53.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 -- start start 2021-02-10T11:29:53.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8758079000 0x7f87580e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87581b00f0 0x7f87581b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8758077a70 con 0x7f875807da20 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580e5c20 con 0x7f8758079000 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f875e839700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580e53c0 con 0x7f87581b00f0 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f87577fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8758079000 0x7f87580e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.514 7f87577fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8758079000 0x7f87580e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44202/0 (socket says 172.21.2.117:44202) 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 -- 172.21.2.117:0/2169186547 learned_addr learned my addr 172.21.2.117:0/2169186547 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:53.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 -- 172.21.2.117:0/2169186547 >> v2:172.21.2.125:3300/0 conn(0x7f87581b00f0 msgr2=0x7f87581b04f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 --2- 172.21.2.117:0/2169186547 >> v2:172.21.2.125:3300/0 conn(0x7f87581b00f0 0x7f87581b04f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 -- 172.21.2.117:0/2169186547 >> v1:172.21.2.125:6789/0 conn(0x7f875807da20 legacy=0x7f87580771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 -- 172.21.2.117:0/2169186547 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758079400 con 0x7f8758079000 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f87577fe700 1 --2- 172.21.2.117:0/2169186547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8758079000 0x7f87580e4da0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f874c006700 tx=0x7f874c009b80).ready entity=mon.0 client_cookie=148547752217dd85 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f8756ffd700 1 -- 172.21.2.117:0/2169186547 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874c00c0c0 con 0x7f8758079000 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f8756ffd700 1 -- 172.21.2.117:0/2169186547 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874c00c440 con 0x7f8758079000 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f8756ffd700 1 -- 172.21.2.117:0/2169186547 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874c00c960 con 0x7f8758079000 2021-02-10T11:29:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f875e839700 1 -- 172.21.2.117:0/2169186547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8758079000 msgr2=0x7f87580e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.515 7f875e839700 1 --2- 172.21.2.117:0/2169186547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8758079000 0x7f87580e4da0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f874c006700 tx=0x7f874c009b80).stop 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 -- 172.21.2.117:0/2169186547 shutdown_connections 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 --2- 172.21.2.117:0/2169186547 >> v2:172.21.2.125:3300/0 conn(0x7f87581b00f0 0x7f87581b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 --2- 172.21.2.117:0/2169186547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8758079000 0x7f87580e4da0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 -- 172.21.2.117:0/2169186547 >> 172.21.2.117:0/2169186547 conn(0x7f87580ce450 msgr2=0x7f87580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 -- 172.21.2.117:0/2169186547 shutdown_connections 2021-02-10T11:29:53.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.516 7f875e839700 1 -- 172.21.2.117:0/2169186547 wait complete. 2021-02-10T11:29:53.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 Processor -- start 2021-02-10T11:29:53.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 -- start start 2021-02-10T11:29:53.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87580b1340 0x7f875818b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f875818b830 0x7f8758187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580b18c0 con 0x7f87580b0f40 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875818ac90 con 0x7f875818b830 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.531 7f875e839700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8758107f50 con 0x7f87580b1340 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44206/0 (socket says 172.21.2.117:44206) 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 -- 172.21.2.117:0/983181012 learned_addr learned my addr 172.21.2.117:0/983181012 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:29:53.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 -- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87580b1340 msgr2=0x7f875818b210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87580b1340 0x7f875818b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 -- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f875818b830 msgr2=0x7f8758187bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f875818b830 0x7f8758187bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 -- 172.21.2.117:0/983181012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758108130 con 0x7f87580b0f40 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8757fff700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f874800b5b0 tx=0x7f874800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f874800bd40 con 0x7f87580b0f40 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.532 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8748006020 con 0x7f87580b0f40 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87480054a0 con 0x7f87580b0f40 2021-02-10T11:29:53.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f875e839700 1 -- 172.21.2.117:0/983181012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8758108130 con 0x7f87580b0f40 2021-02-10T11:29:53.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f875e839700 1 -- 172.21.2.117:0/983181012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8758188570 con 0x7f87580b0f40 2021-02-10T11:29:53.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8748006200 con 0x7f87580b0f40 2021-02-10T11:29:53.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f8754ff9700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 0x7f873801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:29:53.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.533 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f87480326c0 con 0x7f87580b0f40 2021-02-10T11:29:53.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.534 7f87577fe700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 0x7f873801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:29:53.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.535 7f87577fe700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 0x7f873801abc0 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:29:53.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.565 7f875e839700 1 -- 172.21.2.117:0/983181012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8758077a70 con 0x7f87580b0f40 2021-02-10T11:29:53.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.567 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8748000c60 con 0x7f87580b0f40 2021-02-10T11:29:53.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.692 7f875e839700 1 -- 172.21.2.117:0/983181012 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802473", "target": ["mgr", ""]}) v1 -- 0x7f8758108130 con 0x7f87380186c0 2021-02-10T11:29:53.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.696 7f8754ff9700 1 -- 172.21.2.117:0/983181012 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f874c006500 con 0x7f87380186c0 2021-02-10T11:29:53.690 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 msgr2=0x7f873801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 0x7f873801abc0 crc :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 msgr2=0x7f8758107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f874800b5b0 tx=0x7f874800b950).stop 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 shutdown_connections 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f875818b830 0x7f8758187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87580b1340 0x7f875818b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:29:53.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87380186c0 0x7f873801abc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 --2- 172.21.2.117:0/983181012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87580b0f40 0x7f8758107930 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:29:53.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 >> 172.21.2.117:0/983181012 conn(0x7f87580a1880 msgr2=0x7f87580b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:29:53.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 shutdown_connections 2021-02-10T11:29:53.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:29:53.702 7f875e839700 1 -- 172.21.2.117:0/983181012 wait complete. 2021-02-10T11:29:53.700 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802473 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c 2021-02-10T11:29:53.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:53.700 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:53.700 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:54.040 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:54.040 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:54.041 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:54.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:54.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:54.424 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024558 s, 42.7 MB/s 2021-02-10T11:29:54.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:54.425 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:54.425 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:54.762 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:54.762 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:54.763 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:55.139 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:55.139 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:55.139 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235645 s, 44.5 MB/s 2021-02-10T11:29:55.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:55.140 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:55.140 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:55.476 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:55.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:55.477 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:55.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:55.854 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:55.854 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249052 s, 42.1 MB/s 2021-02-10T11:29:55.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:55.855 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:55.855 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:56.191 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:56.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:56.191 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:56.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:56.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:56.566 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234552 s, 44.7 MB/s 2021-02-10T11:29:56.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:56.567 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:56.567 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:56.901 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:56.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:56.902 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238511 s, 44.0 MB/s 2021-02-10T11:29:57.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:57.277 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:57.277 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:57.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:57.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:57.613 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:57.738 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:57.741 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:29:57.985 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:57.985 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:57.985 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232259 s, 45.1 MB/s 2021-02-10T11:29:57.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:57.987 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:57.987 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:58.322 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:58.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:58.322 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:58.700 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:58.700 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:58.700 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250295 s, 41.9 MB/s 2021-02-10T11:29:58.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:58.701 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:58.701 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:59.037 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:59.037 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:59.037 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:29:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:29:59.415 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:29:59.415 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249606 s, 42.0 MB/s 2021-02-10T11:29:59.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:29:59.416 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:29:59.416 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:29:59.751 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:29:59.751 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:29:59.751 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023307 s, 45.0 MB/s 2021-02-10T11:30:00.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:00.127 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:00.127 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:00.462 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:00.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:00.463 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802473/7c61be12-d01f-4305-8d48-16433a130f2c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:00.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:00.840 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:00.840 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252647 s, 41.5 MB/s 2021-02-10T11:30:00.841 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802474 2021-02-10T11:30:01.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 Processor -- start 2021-02-10T11:30:01.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 -- start start 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd70079000 0x7fbd700e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbd701b00f0 0x7fbd701b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd70077a70 con 0x7fbd7007da20 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd700e53c0 con 0x7fbd701b00f0 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.265 7fbd76e27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70079400 con 0x7fbd70079000 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd74c43700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbd7007da20 0x7fbd700771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44464/0 (socket says 172.21.2.117:44464) 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd74c43700 1 -- 172.21.2.117:0/927737308 learned_addr learned my addr 172.21.2.117:0/927737308 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168099752 0 0) 0x7fbd580009b0 con 0x7fbd7007da20 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5c002fe0 con 0x7fbd7007da20 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6ffff700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd70079000 0x7fbd700e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd75444700 1 --2- 172.21.2.117:0/927737308 >> v2:172.21.2.125:3300/0 conn(0x7fbd701b00f0 0x7fbd701b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2143485813 0 0) 0x7fbd58001010 con 0x7fbd7007da20 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd70079000 msgr2=0x7fbd700e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd70079000 0x7fbd700e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 >> v2:172.21.2.125:3300/0 conn(0x7fbd701b00f0 msgr2=0x7fbd701b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 --2- 172.21.2.117:0/927737308 >> v2:172.21.2.125:3300/0 conn(0x7fbd701b00f0 0x7fbd701b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7007de20 con 0x7fbd7007da20 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fbd58001530 con 0x7fbd7007da20 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 >> v1:172.21.2.117:6789/0 conn(0x7fbd7007da20 legacy=0x7fbd700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd5c003a40 0x7fbd5c005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd5c006730 0x7fbd5c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd5c0094f0 0x7fbd5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c00c2c0 con 0x7fbd5c0094f0 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c050480 con 0x7fbd5c006730 2021-02-10T11:30:01.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0506a0 con 0x7fbd5c003a40 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.266 7fbd75444700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd5c0094f0 0x7fbd5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd74c43700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd5c003a40 0x7fbd5c005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd6ffff700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd5c006730 0x7fbd5c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 -- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd5c003a40 msgr2=0x7fbd5c005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd5c003a40 0x7fbd5c005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 -- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd5c006730 msgr2=0x7fbd5c008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd5c006730 0x7fbd5c008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 -- 172.21.2.117:0/927737308 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c050940 con 0x7fbd5c0094f0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd5c0094f0 0x7fbd5c00ba10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fbd600090f0 tx=0x7fbd6000b050).ready entity=mon.0 client_cookie=397b32a8a889de1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd60000f90 con 0x7fbd5c0094f0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd60006240 con 0x7fbd5c0094f0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd6f7fe700 1 -- 172.21.2.117:0/927737308 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd600056b0 con 0x7fbd5c0094f0 2021-02-10T11:30:01.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd76e27700 1 -- 172.21.2.117:0/927737308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd5c0094f0 msgr2=0x7fbd5c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd76e27700 1 --2- 172.21.2.117:0/927737308 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd5c0094f0 0x7fbd5c00ba10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fbd600090f0 tx=0x7fbd6000b050).stop 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd75444700 1 -- 172.21.2.117:0/927737308 reap_dead start 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd76e27700 1 -- 172.21.2.117:0/927737308 shutdown_connections 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.267 7fbd76e27700 1 -- 172.21.2.117:0/927737308 >> 172.21.2.117:0/927737308 conn(0x7fbd700ce450 msgr2=0x7fbd700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.268 7fbd76e27700 1 -- 172.21.2.117:0/927737308 shutdown_connections 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.268 7fbd76e27700 1 -- 172.21.2.117:0/927737308 wait complete. 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.282 7fbd76e27700 1 Processor -- start 2021-02-10T11:30:01.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.282 7fbd76e27700 1 -- start start 2021-02-10T11:30:01.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd700a47d0 0x7fbd700a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd700a51f0 0x7fbd700a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70189970 con 0x7fbd700b77d0 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70190aa0 con 0x7fbd700a47d0 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd76e27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700b81f0 con 0x7fbd700a51f0 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44224/0 (socket says 172.21.2.117:44224) 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 -- 172.21.2.117:0/3700640197 learned_addr learned my addr 172.21.2.117:0/3700640197 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 -- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd700a51f0 msgr2=0x7fbd700a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd700a51f0 0x7fbd700a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 -- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd700a47d0 msgr2=0x7fbd700a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd700a47d0 0x7fbd700a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 -- 172.21.2.117:0/3700640197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700b8410 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.283 7fbd75444700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fbd600090e0 tx=0x7fbd6000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd60009910 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd60006240 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd700b8410 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd700b89a0 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd600056b0 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd60009910 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 0x7fbd5401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.284 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbd60032c90 con 0x7fbd700b77d0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.285 7fbd74c43700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 0x7fbd5401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.285 7fbd74c43700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 0x7fbd5401ac10 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.314 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd70077a70 con 0x7fbd700b77d0 2021-02-10T11:30:01.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.317 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd60006610 con 0x7fbd700b77d0 2021-02-10T11:30:01.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.444 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802474", "target": ["mgr", ""]}) v1 -- 0x7fbd70189970 con 0x7fbd54018730 2021-02-10T11:30:01.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.471 7fbd6d7fa700 1 -- 172.21.2.117:0/3700640197 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbd580014c0 con 0x7fbd54018730 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 msgr2=0x7fbd5401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 0x7fbd5401ac10 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 msgr2=0x7fbd700b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fbd600090e0 tx=0x7fbd6000bd10).stop 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 shutdown_connections 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd54018730 0x7fbd5401ac10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd700b77d0 0x7fbd700b7bd0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd700a51f0 0x7fbd700a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 --2- 172.21.2.117:0/3700640197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd700a47d0 0x7fbd700a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 >> 172.21.2.117:0/3700640197 conn(0x7fbd700b5140 msgr2=0x7fbd700a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 shutdown_connections 2021-02-10T11:30:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.478 7fbd76e27700 1 -- 172.21.2.117:0/3700640197 wait complete. 2021-02-10T11:30:01.476 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802474 2021-02-10T11:30:01.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 Processor -- start 2021-02-10T11:30:01.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 -- start start 2021-02-10T11:30:01.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c079000 0x7f7f6c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c077a70 con 0x7f7f6c07da20 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c0e53c0 con 0x7f7f6c1b00f0 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f71d1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c079400 con 0x7f7f6c079000 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f6bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f6bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44232/0 (socket says 172.21.2.117:44232) 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.907 7f7f6bfff700 1 -- 172.21.2.117:0/199560475 learned_addr learned my addr 172.21.2.117:0/199560475 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6bfff700 1 -- 172.21.2.117:0/199560475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c079000 msgr2=0x7f7f6c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6bfff700 1 --2- 172.21.2.117:0/199560475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c079000 0x7f7f6c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6bfff700 1 -- 172.21.2.117:0/199560475 >> v1:172.21.2.117:6789/0 conn(0x7f7f6c07da20 legacy=0x7f7f6c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6bfff700 1 -- 172.21.2.117:0/199560475 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c07de20 con 0x7f7f6c1b00f0 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6bfff700 1 --2- 172.21.2.117:0/199560475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f7f5c006b80 tx=0x7f7f5c009b90).ready entity=mon.0 client_cookie=2b5cd34788f358f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6a7fc700 1 -- 172.21.2.117:0/199560475 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f5c0022e0 con 0x7f7f6c1b00f0 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6a7fc700 1 -- 172.21.2.117:0/199560475 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c0009b0 con 0x7f7f6c1b00f0 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f6a7fc700 1 -- 172.21.2.117:0/199560475 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f5c0027e0 con 0x7f7f6c1b00f0 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f71d1a700 1 -- 172.21.2.117:0/199560475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c1b00f0 msgr2=0x7f7f6c1b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.908 7f7f71d1a700 1 --2- 172.21.2.117:0/199560475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f7f5c006b80 tx=0x7f7f5c009b90).stop 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 -- 172.21.2.117:0/199560475 shutdown_connections 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 --2- 172.21.2.117:0/199560475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c1b00f0 0x7f7f6c1b04f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 --2- 172.21.2.117:0/199560475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c079000 0x7f7f6c0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 -- 172.21.2.117:0/199560475 >> 172.21.2.117:0/199560475 conn(0x7f7f6c0ce450 msgr2=0x7f7f6c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:01.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 -- 172.21.2.117:0/199560475 shutdown_connections 2021-02-10T11:30:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.909 7f7f71d1a700 1 -- 172.21.2.117:0/199560475 wait complete. 2021-02-10T11:30:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 Processor -- start 2021-02-10T11:30:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 -- start start 2021-02-10T11:30:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c1034d0 0x7f7f6c104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f6c105160 0x7f7f6c0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c103c20 con 0x7f7f6c104740 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c103910 con 0x7f7f6c105160 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f71d1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0b7e00 con 0x7f7f6c1034d0 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f6affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f6affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44238/0 (socket says 172.21.2.117:44238) 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.924 7f7f6affd700 1 -- 172.21.2.117:0/2645146370 learned_addr learned my addr 172.21.2.117:0/2645146370 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 -- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c1034d0 msgr2=0x7f7f6c104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c1034d0 0x7f7f6c104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 -- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f6c105160 msgr2=0x7f7f6c0aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f6c105160 0x7f7f6c0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 -- 172.21.2.117:0/2645146370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c0ab330 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f6affd700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f7f54006f00 tx=0x7f7f5400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f5400bd30 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54000c90 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f540022a0 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c0ab330 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.925 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c0ab330 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.926 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f54005310 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.926 7f7f63fff700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 0x7f7f5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.926 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7f540328d0 con 0x7f7f6c104740 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.926 7f7f6b7fe700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 0x7f7f5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:01.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.927 7f7f6b7fe700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 0x7f7f5001abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:01.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.958 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c0b7de0 con 0x7f7f6c104740 2021-02-10T11:30:01.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:01.960 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f54006b90 con 0x7f7f6c104740 2021-02-10T11:30:02.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.087 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802474", "target": ["mgr", ""]}) v1 -- 0x7f7f6c0b7de0 con 0x7f7f500186c0 2021-02-10T11:30:02.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.091 7f7f63fff700 1 -- 172.21.2.117:0/2645146370 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7f640058c0 con 0x7f7f500186c0 2021-02-10T11:30:02.085 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 msgr2=0x7f7f5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 0x7f7f5001abc0 crc :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 msgr2=0x7f7f6c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f7f54006f00 tx=0x7f7f5400b940).stop 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 shutdown_connections 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f500186c0 0x7f7f5001abc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f6c104740 0x7f7f6c104b40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:02.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f6c105160 0x7f7f6c0aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:02.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 --2- 172.21.2.117:0/2645146370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f6c1034d0 0x7f7f6c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:02.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 >> 172.21.2.117:0/2645146370 conn(0x7f7f6c0a1140 msgr2=0x7f7f6c0ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:02.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 shutdown_connections 2021-02-10T11:30:02.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:02.097 7f7f71d1a700 1 -- 172.21.2.117:0/2645146370 wait complete. 2021-02-10T11:30:02.096 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802474 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15 2021-02-10T11:30:02.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:02.096 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:02.097 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:02.432 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:02.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:02.433 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:02.812 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:02.812 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:02.812 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260058 s, 40.3 MB/s 2021-02-10T11:30:02.813 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:02.813 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:02.813 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:03.149 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:03.149 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:03.149 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:03.523 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:03.524 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:03.524 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234867 s, 44.6 MB/s 2021-02-10T11:30:03.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:03.526 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:03.526 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:03.862 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:03.862 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:03.862 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:04.235 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:04.236 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:04.236 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228161 s, 46.0 MB/s 2021-02-10T11:30:04.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:04.237 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:04.237 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:04.572 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:04.572 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:04.572 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:04.947 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:04.947 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234074 s, 44.8 MB/s 2021-02-10T11:30:04.948 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:04.948 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:04.948 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:05.284 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:05.284 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:05.284 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:05.658 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:05.658 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:05.659 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230784 s, 45.4 MB/s 2021-02-10T11:30:05.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:05.660 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:05.660 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:05.996 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:05.996 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:05.996 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:06.370 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:06.370 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:06.370 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023592 s, 44.4 MB/s 2021-02-10T11:30:06.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:06.371 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:06.372 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:06.708 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:06.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:06.708 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:07.083 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:07.083 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:07.083 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221715 s, 47.3 MB/s 2021-02-10T11:30:07.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:07.084 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:07.084 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:07.420 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:07.421 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:07.421 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:07.794 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:07.794 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:07.794 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248319 s, 42.2 MB/s 2021-02-10T11:30:07.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:07.795 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:07.795 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:08.132 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:08.132 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:08.132 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243237 s, 43.1 MB/s 2021-02-10T11:30:08.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:08.508 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:08.509 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:08.843 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:08.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:08.844 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802474/df0f13e7-0179-4a46-937f-bd9825409e15/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:09.219 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:09.219 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:09.219 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242385 s, 43.3 MB/s 2021-02-10T11:30:09.220 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802475 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 Processor -- start 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 -- start start 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 0x7f516c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f516c0e4da0 0x7f516c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 0x7f516c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c077a70 con 0x7f516c0e4da0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f516c0777d0 con 0x7f516c07da20 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.645 7f51743b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c07de20 con 0x7f516c0e51a0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f516c0e4da0 0x7f516c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f516c0e4da0 0x7f516c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44242/0 (socket says 172.21.2.117:44242) 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 -- 172.21.2.117:0/860763559 learned_addr learned my addr 172.21.2.117:0/860763559 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51729d3700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 0x7f516c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51721d2700 1 --2- 172.21.2.117:0/860763559 >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 0x7f516c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 -- 172.21.2.117:0/860763559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 msgr2=0x7f516c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 0x7f516c077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 -- 172.21.2.117:0/860763559 >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 msgr2=0x7f516c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 --2- 172.21.2.117:0/860763559 >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 0x7f516c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51719d1700 1 -- 172.21.2.117:0/860763559 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c1b0710 con 0x7f516c0e4da0 2021-02-10T11:30:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51729d3700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 0x7f516c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.646 7f51721d2700 1 --2- 172.21.2.117:0/860763559 >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 0x7f516c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51719d1700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0e4da0 0x7f516c1b00f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f515c006700 tx=0x7f515c009b80).ready entity=mon.0 client_cookie=eab82cc6982a47c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51711d0700 1 -- 172.21.2.117:0/860763559 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c00c0c0 con 0x7f516c0e4da0 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51711d0700 1 -- 172.21.2.117:0/860763559 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00c340 con 0x7f516c0e4da0 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51711d0700 1 -- 172.21.2.117:0/860763559 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f515c00c820 con 0x7f516c0e4da0 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51743b6700 1 -- 172.21.2.117:0/860763559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0e4da0 msgr2=0x7f516c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.647 7f51743b6700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0e4da0 0x7f516c1b00f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f515c006700 tx=0x7f515c009b80).stop 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 -- 172.21.2.117:0/860763559 shutdown_connections 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c0e51a0 0x7f516c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 --2- 172.21.2.117:0/860763559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0e4da0 0x7f516c1b00f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 --2- 172.21.2.117:0/860763559 >> v2:172.21.2.125:3300/0 conn(0x7f516c07da20 0x7f516c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 -- 172.21.2.117:0/860763559 >> 172.21.2.117:0/860763559 conn(0x7f516c0ce450 msgr2=0x7f516c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 -- 172.21.2.117:0/860763559 shutdown_connections 2021-02-10T11:30:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.648 7f51743b6700 1 -- 172.21.2.117:0/860763559 wait complete. 2021-02-10T11:30:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.662 7f51743b6700 1 Processor -- start 2021-02-10T11:30:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.662 7f51743b6700 1 -- start start 2021-02-10T11:30:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f516c1048f0 0x7f516c104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c090710 0x7f516c090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c1054f0 con 0x7f516c0b7fc0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516c195a30 con 0x7f516c1048f0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51743b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f516c195fb0 con 0x7f516c090710 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44246/0 (socket says 172.21.2.117:44246) 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 -- 172.21.2.117:0/1141367371 learned_addr learned my addr 172.21.2.117:0/1141367371 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 -- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c090710 msgr2=0x7f516c090b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c090710 0x7f516c090b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 -- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f516c1048f0 msgr2=0x7f516c104cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f516c1048f0 0x7f516c104cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.663 7f51721d2700 1 -- 172.21.2.117:0/1141367371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c0b7b60 con 0x7f516c0b7fc0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f51721d2700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f5168006f40 tx=0x7f516800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f516800bd80 con 0x7f516c0b7fc0 2021-02-10T11:30:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168000c90 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f51743b6700 1 -- 172.21.2.117:0/1141367371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c0b7b60 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f51743b6700 1 -- 172.21.2.117:0/1141367371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516c091420 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51680022b0 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f516800a000 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 0x7f515401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.664 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5168032950 con 0x7f516c0b7fc0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.665 7f51719d1700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 0x7f515401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.666 7f51719d1700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 0x7f515401ac10 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:09.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.695 7f51743b6700 1 -- 172.21.2.117:0/1141367371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c103280 con 0x7f516c0b7fc0 2021-02-10T11:30:09.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.698 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51680009b0 con 0x7f516c0b7fc0 2021-02-10T11:30:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.824 7f51743b6700 1 -- 172.21.2.117:0/1141367371 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802475", "target": ["mgr", ""]}) v1 -- 0x7f516c091420 con 0x7f5154018730 2021-02-10T11:30:09.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.851 7f5162ffd700 1 -- 172.21.2.117:0/1141367371 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f515c009100 con 0x7f5154018730 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 msgr2=0x7f515401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 0x7f515401ac10 crc :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 msgr2=0x7f516c1042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f5168006f40 tx=0x7f516800b990).stop 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 shutdown_connections 2021-02-10T11:30:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f516c090710 0x7f516c090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f516c1048f0 0x7f516c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5154018730 0x7f515401ac10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 --2- 172.21.2.117:0/1141367371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516c0b7fc0 0x7f516c1042d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 >> 172.21.2.117:0/1141367371 conn(0x7f516c0a1850 msgr2=0x7f516c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 shutdown_connections 2021-02-10T11:30:09.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:09.858 7f51743b6700 1 -- 172.21.2.117:0/1141367371 wait complete. 2021-02-10T11:30:09.856 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802475 2021-02-10T11:30:10.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.280 7fde76fa8700 1 Processor -- start 2021-02-10T11:30:10.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde76fa8700 1 -- start start 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde76fa8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 0x7fde700e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde76fa8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde700d1d20 con 0x7fde7007da20 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde76fa8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde70077a70 con 0x7fde701b00f0 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde76fa8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700e5430 con 0x7fde70079000 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde74dc4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fde7007da20 0x7fde700771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44504/0 (socket says 172.21.2.117:44504) 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde74dc4700 1 -- 172.21.2.117:0/1596742040 learned_addr learned my addr 172.21.2.117:0/1596742040 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.281 7fde6ffff700 1 --2- 172.21.2.117:0/1596742040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 0x7fde700e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195155965 0 0) 0x7fde68001060 con 0x7fde7007da20 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde58002fe0 con 0x7fde7007da20 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019590610 0 0) 0x7fde60000c10 con 0x7fde701b00f0 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde58003210 con 0x7fde701b00f0 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6ffff700 1 -- 172.21.2.117:0/1596742040 >> v1:172.21.2.125:6789/0 conn(0x7fde701b00f0 legacy=0x7fde700d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6ffff700 1 -- 172.21.2.117:0/1596742040 >> v1:172.21.2.117:6789/0 conn(0x7fde7007da20 legacy=0x7fde700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1096807573 0 0) 0x7fde68001250 con 0x7fde7007da20 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6ffff700 1 -- 172.21.2.117:0/1596742040 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70079400 con 0x7fde70079000 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6ffff700 1 --2- 172.21.2.117:0/1596742040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 0x7fde700e4da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fde64006650 tx=0x7fde64009a00).ready entity=mon.2 client_cookie=18446246820ed66c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde640009b0 con 0x7fde70079000 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64009df0 con 0x7fde70079000 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde6f7fe700 1 -- 172.21.2.117:0/1596742040 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde6400c0c0 con 0x7fde70079000 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde76fa8700 1 -- 172.21.2.117:0/1596742040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 msgr2=0x7fde700e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.282 7fde76fa8700 1 --2- 172.21.2.117:0/1596742040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 0x7fde700e4da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fde64006650 tx=0x7fde64009a00).stop 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.283 7fde76fa8700 1 -- 172.21.2.117:0/1596742040 shutdown_connections 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.283 7fde76fa8700 1 --2- 172.21.2.117:0/1596742040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde70079000 0x7fde700e4da0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.283 7fde76fa8700 1 -- 172.21.2.117:0/1596742040 >> 172.21.2.117:0/1596742040 conn(0x7fde700ce450 msgr2=0x7fde700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.283 7fde76fa8700 1 -- 172.21.2.117:0/1596742040 shutdown_connections 2021-02-10T11:30:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.283 7fde76fa8700 1 -- 172.21.2.117:0/1596742040 wait complete. 2021-02-10T11:30:10.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 Processor -- start 2021-02-10T11:30:10.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 -- start start 2021-02-10T11:30:10.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde700acf20 0x7fde700b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde700b6880 0x7fde700b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700ad320 con 0x7fde700b2640 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde700b6c80 con 0x7fde700b6880 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.298 7fde76fa8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde700b6e60 con 0x7fde700acf20 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44260/0 (socket says 172.21.2.117:44260) 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 -- 172.21.2.117:0/2894721143 learned_addr learned my addr 172.21.2.117:0/2894721143 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde74dc4700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde700acf20 0x7fde700b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde755c5700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde700b6880 0x7fde700b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 -- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde700acf20 msgr2=0x7fde700b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde700acf20 0x7fde700b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 -- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde700b6880 msgr2=0x7fde700b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde700b6880 0x7fde700b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 -- 172.21.2.117:0/2894721143 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700b31c0 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6ffff700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fde640096a0 tx=0x7fde6400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde64000c90 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64002030 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde700b31c0 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde64002550 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.299 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde700b31c0 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.300 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde64002210 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.300 7fde6d7fa700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 0x7fde5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.300 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fde64032800 con 0x7fde700b2640 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.300 7fde74dc4700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 0x7fde5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:10.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.300 7fde74dc4700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 0x7fde5001ac10 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.330 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70077a70 con 0x7fde700b2640 2021-02-10T11:30:10.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.333 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde640009a0 con 0x7fde700b2640 2021-02-10T11:30:10.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.460 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802475", "target": ["mgr", ""]}) v1 -- 0x7fde70077a70 con 0x7fde50018730 2021-02-10T11:30:10.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.464 7fde6d7fa700 1 -- 172.21.2.117:0/2894721143 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fde68001580 con 0x7fde50018730 2021-02-10T11:30:10.454 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.465 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 msgr2=0x7fde5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.465 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 0x7fde5001ac10 crc :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.465 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 msgr2=0x7fde700b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.465 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fde640096a0 tx=0x7fde6400b990).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 shutdown_connections 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde50018730 0x7fde5001ac10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde700b2640 0x7fde700b6260 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde700b6880 0x7fde700b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 --2- 172.21.2.117:0/2894721143 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde700acf20 0x7fde700b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 >> 172.21.2.117:0/2894721143 conn(0x7fde700aca80 msgr2=0x7fde700aaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 shutdown_connections 2021-02-10T11:30:10.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:10.466 7fde76fa8700 1 -- 172.21.2.117:0/2894721143 wait complete. 2021-02-10T11:30:10.463 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802475 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10 2021-02-10T11:30:10.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:10.464 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:10.464 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:10.799 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:10.800 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:10.800 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:11.182 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:11.182 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:11.182 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245523 s, 42.7 MB/s 2021-02-10T11:30:11.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:11.183 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:11.183 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:11.518 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:11.519 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:11.519 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:11.895 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:11.895 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:11.896 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252781 s, 41.5 MB/s 2021-02-10T11:30:11.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:11.897 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:11.897 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:12.233 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:12.233 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:12.233 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:12.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:12.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:12.606 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254076 s, 41.3 MB/s 2021-02-10T11:30:12.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:12.607 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:12.607 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:12.943 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:12.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:12.943 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:13.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:13.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:13.318 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232668 s, 45.1 MB/s 2021-02-10T11:30:13.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:13.319 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:13.319 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:13.655 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:13.656 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:13.656 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:14.029 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:14.030 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238041 s, 44.1 MB/s 2021-02-10T11:30:14.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:14.031 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:14.031 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:14.367 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:14.367 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:14.367 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:14.741 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:14.741 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:14.741 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229416 s, 45.7 MB/s 2021-02-10T11:30:14.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:14.743 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:14.743 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:15.078 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:15.079 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:15.079 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:15.453 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:15.453 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:15.453 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227116 s, 46.2 MB/s 2021-02-10T11:30:15.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:15.455 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:15.455 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:15.789 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:15.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:15.790 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:16.164 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:16.164 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:16.164 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0220402 s, 47.6 MB/s 2021-02-10T11:30:16.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:16.165 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:16.165 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:16.502 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:16.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:16.503 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:16.877 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:16.878 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:16.878 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231446 s, 45.3 MB/s 2021-02-10T11:30:16.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:16.879 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:16.879 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:17.216 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:17.216 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:17.216 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802475/57705797-e8eb-404f-ad17-8bb8dc5c8e10/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:17.590 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:17.591 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:17.591 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256766 s, 40.8 MB/s 2021-02-10T11:30:17.592 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802476 2021-02-10T11:30:18.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 Processor -- start 2021-02-10T11:30:18.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 -- start start 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb3c07da20 0x7fcb3c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcb3c0e51a0 0x7fcb3c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3c077a70 con 0x7fcb3c0e4da0 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c0e5c20 con 0x7fcb3c07da20 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.016 7fcb44172700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3c0777d0 con 0x7fcb3c0e51a0 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb3c07da20 0x7fcb3c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb3c07da20 0x7fcb3c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44266/0 (socket says 172.21.2.117:44266) 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 -- 172.21.2.117:0/2310116829 learned_addr learned my addr 172.21.2.117:0/2310116829 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 -- 172.21.2.117:0/2310116829 >> v2:172.21.2.125:3300/0 conn(0x7fcb3c0e51a0 msgr2=0x7fcb3c079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 --2- 172.21.2.117:0/2310116829 >> v2:172.21.2.125:3300/0 conn(0x7fcb3c0e51a0 0x7fcb3c079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 -- 172.21.2.117:0/2310116829 >> v1:172.21.2.125:6789/0 conn(0x7fcb3c0e4da0 legacy=0x7fcb3c1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 -- 172.21.2.117:0/2310116829 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c1b08e0 con 0x7fcb3c07da20 2021-02-10T11:30:18.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb41f8e700 1 --2- 172.21.2.117:0/2310116829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c07da20 0x7fcb3c0771b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fcb30006700 tx=0x7fcb30009b80).ready entity=mon.0 client_cookie=7efd2d66436c5da2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb40f8c700 1 -- 172.21.2.117:0/2310116829 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3000c0c0 con 0x7fcb3c07da20 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb40f8c700 1 -- 172.21.2.117:0/2310116829 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3000c340 con 0x7fcb3c07da20 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb40f8c700 1 -- 172.21.2.117:0/2310116829 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3000c860 con 0x7fcb3c07da20 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb44172700 1 -- 172.21.2.117:0/2310116829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c07da20 msgr2=0x7fcb3c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.017 7fcb44172700 1 --2- 172.21.2.117:0/2310116829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c07da20 0x7fcb3c0771b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fcb30006700 tx=0x7fcb30009b80).stop 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.018 7fcb44172700 1 -- 172.21.2.117:0/2310116829 shutdown_connections 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.018 7fcb44172700 1 --2- 172.21.2.117:0/2310116829 >> v2:172.21.2.125:3300/0 conn(0x7fcb3c0e51a0 0x7fcb3c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.018 7fcb44172700 1 --2- 172.21.2.117:0/2310116829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c07da20 0x7fcb3c0771b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.018 7fcb44172700 1 -- 172.21.2.117:0/2310116829 >> 172.21.2.117:0/2310116829 conn(0x7fcb3c0ce450 msgr2=0x7fcb3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.019 7fcb44172700 1 -- 172.21.2.117:0/2310116829 shutdown_connections 2021-02-10T11:30:18.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.019 7fcb44172700 1 -- 172.21.2.117:0/2310116829 wait complete. 2021-02-10T11:30:18.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 Processor -- start 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 -- start start 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb3c0a9890 0x7fcb3c0a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb3c18a0e0 0x7fcb3c18a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c189be0 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0b83e0 con 0x7fcb3c0a9890 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb44172700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3c0b8600 con 0x7fcb3c18a0e0 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb4278f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb4278f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44270/0 (socket says 172.21.2.117:44270) 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.033 7fcb4278f700 1 -- 172.21.2.117:0/657643176 learned_addr learned my addr 172.21.2.117:0/657643176 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 -- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb3c18a0e0 msgr2=0x7fcb3c18a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:18.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb3c18a0e0 0x7fcb3c18a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 -- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb3c0a9890 msgr2=0x7fcb3c0a9c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb3c0a9890 0x7fcb3c0a9c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 -- 172.21.2.117:0/657643176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c0b8820 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb4278f700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fcb34009960 tx=0x7fcb3400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb34003030 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb44172700 1 -- 172.21.2.117:0/657643176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3c0b8820 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb34003030 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb34001310 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.034 7fcb44172700 1 -- 172.21.2.117:0/657643176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3c0b8820 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.035 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb34000de0 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.035 7fcb2e7fc700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 0x7fcb2401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.035 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb34032c90 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.036 7fcb41f8e700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 0x7fcb2401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.036 7fcb41f8e700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 0x7fcb2401ac10 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.067 7fcb44172700 1 -- 172.21.2.117:0/657643176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3c1032d0 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.069 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb3400be30 con 0x7fcb3c0b79c0 2021-02-10T11:30:18.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.194 7fcb44172700 1 -- 172.21.2.117:0/657643176 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802476", "target": ["mgr", ""]}) v1 -- 0x7fcb3c1032d0 con 0x7fcb24018730 2021-02-10T11:30:18.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.221 7fcb2e7fc700 1 -- 172.21.2.117:0/657643176 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcb30006500 con 0x7fcb24018730 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 msgr2=0x7fcb2401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 0x7fcb2401ac10 crc :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 msgr2=0x7fcb3c0b7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fcb34009960 tx=0x7fcb3400bcf0).stop 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 shutdown_connections 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb24018730 0x7fcb2401ac10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb3c0b79c0 0x7fcb3c0b7dc0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb3c18a0e0 0x7fcb3c18a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 --2- 172.21.2.117:0/657643176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb3c0a9890 0x7fcb3c0a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 >> 172.21.2.117:0/657643176 conn(0x7fcb3c0b56c0 msgr2=0x7fcb3c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 shutdown_connections 2021-02-10T11:30:18.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.227 7fcb44172700 1 -- 172.21.2.117:0/657643176 wait complete. 2021-02-10T11:30:18.421 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802476 2021-02-10T11:30:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.845 7faae475e700 1 Processor -- start 2021-02-10T11:30:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.845 7faae475e700 1 -- start start 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc07da20 0x7faadc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faadc0e4da0 0x7faadc1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faadc0e51a0 0x7faadc077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faadc0e5c20 con 0x7faadc0e51a0 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faadc077a70 con 0x7faadc0e4da0 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae475e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0777d0 con 0x7faadc07da20 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faadc0e51a0 0x7faadc077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faadc0e51a0 0x7faadc077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44278/0 (socket says 172.21.2.117:44278) 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 -- 172.21.2.117:0/2302530291 learned_addr learned my addr 172.21.2.117:0/2302530291 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 -- 172.21.2.117:0/2302530291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc07da20 msgr2=0x7faadc0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 --2- 172.21.2.117:0/2302530291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc07da20 0x7faadc0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 -- 172.21.2.117:0/2302530291 >> v2:172.21.2.125:3300/0 conn(0x7faadc0e4da0 msgr2=0x7faadc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 --2- 172.21.2.117:0/2302530291 >> v2:172.21.2.125:3300/0 conn(0x7faadc0e4da0 0x7faadc1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 -- 172.21.2.117:0/2302530291 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc07de20 con 0x7faadc0e51a0 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.846 7faae2d7b700 1 --2- 172.21.2.117:0/2302530291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc0e51a0 0x7faadc077ff0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7faad4006f00 tx=0x7faad4009f10).ready entity=mon.0 client_cookie=bac3d7cd340d56ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae1578700 1 -- 172.21.2.117:0/2302530291 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad400b3b0 con 0x7faadc0e51a0 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae1578700 1 -- 172.21.2.117:0/2302530291 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad400b730 con 0x7faadc0e51a0 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae1578700 1 -- 172.21.2.117:0/2302530291 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad400bc50 con 0x7faadc0e51a0 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 -- 172.21.2.117:0/2302530291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc0e51a0 msgr2=0x7faadc077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 --2- 172.21.2.117:0/2302530291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc0e51a0 0x7faadc077ff0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7faad4006f00 tx=0x7faad4009f10).stop 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 -- 172.21.2.117:0/2302530291 shutdown_connections 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 --2- 172.21.2.117:0/2302530291 >> v2:172.21.2.125:3300/0 conn(0x7faadc0e4da0 0x7faadc1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 --2- 172.21.2.117:0/2302530291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc0e51a0 0x7faadc077ff0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 --2- 172.21.2.117:0/2302530291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc07da20 0x7faadc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.847 7faae475e700 1 -- 172.21.2.117:0/2302530291 >> 172.21.2.117:0/2302530291 conn(0x7faadc0ce450 msgr2=0x7faadc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:18.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.848 7faae475e700 1 -- 172.21.2.117:0/2302530291 shutdown_connections 2021-02-10T11:30:18.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.848 7faae475e700 1 -- 172.21.2.117:0/2302530291 wait complete. 2021-02-10T11:30:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 Processor -- start 2021-02-10T11:30:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 -- start start 2021-02-10T11:30:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faadc09bbe0 0x7faadc092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc093370 0x7faadc0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0927d0 con 0x7faadc09bfe0 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc0b08d0 con 0x7faadc09bbe0 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.862 7faae475e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faadc190400 con 0x7faadc093370 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44284/0 (socket says 172.21.2.117:44284) 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 -- 172.21.2.117:0/2141695458 learned_addr learned my addr 172.21.2.117:0/2141695458 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 -- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc093370 msgr2=0x7faadc0b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc093370 0x7faadc0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 -- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faadc09bbe0 msgr2=0x7faadc092d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faadc09bbe0 0x7faadc092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 -- 172.21.2.117:0/2141695458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc1908a0 con 0x7faadc09bfe0 2021-02-10T11:30:18.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae1d79700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7faad8006f00 tx=0x7faad800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad8006b90 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad800bd30 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae475e700 1 -- 172.21.2.117:0/2141695458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc1908a0 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad80020d0 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.863 7faae475e700 1 -- 172.21.2.117:0/2141695458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc1abb00 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.864 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faad8007020 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.864 7faaceffd700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 0x7faac401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.864 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faad80328c0 con 0x7faadc09bfe0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.864 7faae257a700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 0x7faac401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:18.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.865 7faae257a700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 0x7faac401abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:18.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.896 7faae475e700 1 -- 172.21.2.117:0/2141695458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc103370 con 0x7faadc09bfe0 2021-02-10T11:30:18.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:18.898 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faad8007ef0 con 0x7faadc09bfe0 2021-02-10T11:30:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.027 7faae475e700 1 -- 172.21.2.117:0/2141695458 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802476", "target": ["mgr", ""]}) v1 -- 0x7faadc1abb00 con 0x7faac4018730 2021-02-10T11:30:19.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.031 7faaceffd700 1 -- 172.21.2.117:0/2141695458 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7faad0006e60 con 0x7faac4018730 2021-02-10T11:30:19.024 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 msgr2=0x7faac401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 0x7faac401abc0 crc :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 msgr2=0x7faadc0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7faad8006f00 tx=0x7faad800b940).stop 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 shutdown_connections 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faac4018730 0x7faac401abc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faadc09bfe0 0x7faadc0afc90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:19.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faadc093370 0x7faadc0b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 --2- 172.21.2.117:0/2141695458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faadc09bbe0 0x7faadc092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:19.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 >> 172.21.2.117:0/2141695458 conn(0x7faadc0a1420 msgr2=0x7faadc09efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:19.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 shutdown_connections 2021-02-10T11:30:19.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:19.036 7faae475e700 1 -- 172.21.2.117:0/2141695458 wait complete. 2021-02-10T11:30:19.034 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802476 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49 2021-02-10T11:30:19.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:19.034 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:19.034 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:19.371 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:19.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:19.371 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:19.751 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:19.751 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:19.751 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230986 s, 45.4 MB/s 2021-02-10T11:30:19.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:19.752 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:19.752 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:20.089 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:20.089 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:20.089 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233533 s, 44.9 MB/s 2021-02-10T11:30:20.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:20.466 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:20.466 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:20.801 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:20.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:20.802 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241946 s, 43.3 MB/s 2021-02-10T11:30:21.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:21.179 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:21.179 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:21.514 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:21.515 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:21.515 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:21.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:21.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:21.894 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026788 s, 39.1 MB/s 2021-02-10T11:30:21.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:21.895 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:21.895 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:22.235 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:22.235 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:22.235 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:22.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:22.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:22.606 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243028 s, 43.1 MB/s 2021-02-10T11:30:22.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:22.607 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:22.608 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:22.943 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:22.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:22.944 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:23.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:23.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:23.322 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249329 s, 42.1 MB/s 2021-02-10T11:30:23.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:23.323 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:23.323 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:23.660 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:23.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:23.660 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:24.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:24.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:24.048 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241899 s, 43.3 MB/s 2021-02-10T11:30:24.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:24.049 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:24.049 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:24.393 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:24.394 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:24.394 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:24.773 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:24.773 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:24.773 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253943 s, 41.3 MB/s 2021-02-10T11:30:24.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:24.774 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:24.774 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:25.109 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:25.109 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:25.110 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:25.484 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:25.485 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:25.485 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023443 s, 44.7 MB/s 2021-02-10T11:30:25.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:25.486 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:25.486 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:25.822 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:25.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:25.823 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802476/096bf17b-aab5-4714-bfa6-c507e82d5c49/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:26.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:26.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:26.198 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249321 s, 42.1 MB/s 2021-02-10T11:30:26.199 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802477 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 Processor -- start 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 -- start start 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e1007da20 0x7f6e100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e100e4da0 0x7f6e101b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e10077a70 con 0x7f6e100e51a0 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100777d0 con 0x7f6e100e4da0 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e1744c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1007de20 con 0x7f6e1007da20 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e14a67700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e100e4da0 0x7f6e101b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e14a67700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e100e4da0 0x7f6e101b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44290/0 (socket says 172.21.2.117:44290) 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.622 7f6e14a67700 1 -- 172.21.2.117:0/1861990815 learned_addr learned my addr 172.21.2.117:0/1861990815 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e14a67700 1 -- 172.21.2.117:0/1861990815 >> v2:172.21.2.125:3300/0 conn(0x7f6e1007da20 msgr2=0x7f6e100771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e14a67700 1 --2- 172.21.2.117:0/1861990815 >> v2:172.21.2.125:3300/0 conn(0x7f6e1007da20 0x7f6e100771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e14a67700 1 -- 172.21.2.117:0/1861990815 >> v1:172.21.2.125:6789/0 conn(0x7f6e100e51a0 legacy=0x7f6e10077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e14a67700 1 -- 172.21.2.117:0/1861990815 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e101b0710 con 0x7f6e100e4da0 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e14a67700 1 --2- 172.21.2.117:0/1861990815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e100e4da0 0x7f6e101b00f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f6e0c006700 tx=0x7f6e0c009b00).ready entity=mon.0 client_cookie=9827756bac809cd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e03fff700 1 -- 172.21.2.117:0/1861990815 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c00c0c0 con 0x7f6e100e4da0 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e03fff700 1 -- 172.21.2.117:0/1861990815 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c00c400 con 0x7f6e100e4da0 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e03fff700 1 -- 172.21.2.117:0/1861990815 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c00c8e0 con 0x7f6e100e4da0 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e1744c700 1 -- 172.21.2.117:0/1861990815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e100e4da0 msgr2=0x7f6e101b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.623 7f6e1744c700 1 --2- 172.21.2.117:0/1861990815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e100e4da0 0x7f6e101b00f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f6e0c006700 tx=0x7f6e0c009b00).stop 2021-02-10T11:30:26.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 -- 172.21.2.117:0/1861990815 shutdown_connections 2021-02-10T11:30:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 --2- 172.21.2.117:0/1861990815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e100e4da0 0x7f6e101b00f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 --2- 172.21.2.117:0/1861990815 >> v2:172.21.2.125:3300/0 conn(0x7f6e1007da20 0x7f6e100771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 -- 172.21.2.117:0/1861990815 >> 172.21.2.117:0/1861990815 conn(0x7f6e100ce450 msgr2=0x7f6e100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 -- 172.21.2.117:0/1861990815 shutdown_connections 2021-02-10T11:30:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.624 7f6e1744c700 1 -- 172.21.2.117:0/1861990815 wait complete. 2021-02-10T11:30:26.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.638 7f6e1744c700 1 Processor -- start 2021-02-10T11:30:26.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.638 7f6e1744c700 1 -- start start 2021-02-10T11:30:26.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e101034d0 0x7f6e10104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e10105160 0x7f6e100aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10103c20 con 0x7f6e10104740 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10103910 con 0x7f6e101034d0 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e1744c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100b7e00 con 0x7f6e10105160 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44298/0 (socket says 172.21.2.117:44298) 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 -- 172.21.2.117:0/388816655 learned_addr learned my addr 172.21.2.117:0/388816655 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 -- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e10105160 msgr2=0x7f6e100aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e10105160 0x7f6e100aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 -- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e101034d0 msgr2=0x7f6e10104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e101034d0 0x7f6e10104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 -- 172.21.2.117:0/388816655 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100ab330 con 0x7f6e10104740 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.639 7f6e14a67700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f6e0c006f20 tx=0x7f6e0c00b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.640 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c0090d0 con 0x7f6e10104740 2021-02-10T11:30:26.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.640 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c00bd00 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.640 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c005560 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.640 7f6e1744c700 1 -- 172.21.2.117:0/388816655 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e100ab330 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.640 7f6e1744c700 1 -- 172.21.2.117:0/388816655 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e100ab330 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.641 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e0c00a000 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.641 7f6e01ffb700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 0x7f6df001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.641 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e0c032690 con 0x7f6e10104740 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.641 7f6e15268700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 0x7f6df001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:26.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.642 7f6e15268700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 0x7f6df001abc0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:26.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.672 7f6e1744c700 1 -- 172.21.2.117:0/388816655 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e100b7de0 con 0x7f6e10104740 2021-02-10T11:30:26.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.674 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e0c0090d0 con 0x7f6e10104740 2021-02-10T11:30:26.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.801 7f6e1744c700 1 -- 172.21.2.117:0/388816655 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802477", "target": ["mgr", ""]}) v1 -- 0x7f6e100b7de0 con 0x7f6df00186c0 2021-02-10T11:30:26.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.828 7f6e01ffb700 1 -- 172.21.2.117:0/388816655 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6e04006e60 con 0x7f6df00186c0 2021-02-10T11:30:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.834 7f6e1744c700 1 -- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 msgr2=0x7f6df001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.834 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 0x7f6df001abc0 crc :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.834 7f6e1744c700 1 -- 172.21.2.117:0/388816655 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 msgr2=0x7f6e10104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.834 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f6e0c006f20 tx=0x7f6e0c00b910).stop 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 -- 172.21.2.117:0/388816655 shutdown_connections 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df00186c0 0x7f6df001abc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e10104740 0x7f6e10104b40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e10105160 0x7f6e100aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 --2- 172.21.2.117:0/388816655 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e101034d0 0x7f6e10104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 -- 172.21.2.117:0/388816655 >> 172.21.2.117:0/388816655 conn(0x7f6e100a1140 msgr2=0x7f6e100ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 -- 172.21.2.117:0/388816655 shutdown_connections 2021-02-10T11:30:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:26.835 7f6e1744c700 1 -- 172.21.2.117:0/388816655 wait complete. 2021-02-10T11:30:26.833 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802477 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 Processor -- start 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 -- start start 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f08b40d1510 0x7f08b40d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b4077a70 con 0x7f08b407da20 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b4079400 con 0x7f08b4079000 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.257 7f08bc4aa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b407de20 con 0x7f08b40d1510 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f08b40d1510 0x7f08b40d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f08b40d1510 0x7f08b40d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44308/0 (socket says 172.21.2.117:44308) 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 -- 172.21.2.117:0/1332125255 learned_addr learned my addr 172.21.2.117:0/1332125255 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 -- 172.21.2.117:0/1332125255 >> v1:172.21.2.125:6789/0 conn(0x7f08b4079000 legacy=0x7f08b40e4d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:30:27.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 -- 172.21.2.117:0/1332125255 >> v1:172.21.2.117:6789/0 conn(0x7f08b407da20 legacy=0x7f08b40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 -- 172.21.2.117:0/1332125255 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b41b00d0 con 0x7f08b40d1510 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08baac7700 1 --2- 172.21.2.117:0/1332125255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b40d1510 0x7f08b40d1910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f08ac006f00 tx=0x7f08ac009f10).ready entity=mon.0 client_cookie=57d493be2b0e9142 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08b92c4700 1 -- 172.21.2.117:0/1332125255 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08ac00b3b0 con 0x7f08b40d1510 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08b92c4700 1 -- 172.21.2.117:0/1332125255 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac00b630 con 0x7f08b40d1510 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08b92c4700 1 -- 172.21.2.117:0/1332125255 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08ac00bb50 con 0x7f08b40d1510 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08bc4aa700 1 -- 172.21.2.117:0/1332125255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b40d1510 msgr2=0x7f08b40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.258 7f08bc4aa700 1 --2- 172.21.2.117:0/1332125255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b40d1510 0x7f08b40d1910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f08ac006f00 tx=0x7f08ac009f10).stop 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.259 7f08bc4aa700 1 -- 172.21.2.117:0/1332125255 shutdown_connections 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.259 7f08bc4aa700 1 --2- 172.21.2.117:0/1332125255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b40d1510 0x7f08b40d1910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.259 7f08bc4aa700 1 -- 172.21.2.117:0/1332125255 >> 172.21.2.117:0/1332125255 conn(0x7f08b40ce430 msgr2=0x7f08b40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.259 7f08bc4aa700 1 -- 172.21.2.117:0/1332125255 shutdown_connections 2021-02-10T11:30:27.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.259 7f08bc4aa700 1 -- 172.21.2.117:0/1332125255 wait complete. 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.273 7f08bc4aa700 1 Processor -- start 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.273 7f08bc4aa700 1 -- start start 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f08b4097ae0 0x7f08b4107270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f08b40c3170 0x7f08b4098ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4098340 con 0x7f08b4107890 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4107da0 con 0x7f08b40c3170 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08bc4aa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b4097ee0 con 0x7f08b4097ae0 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44312/0 (socket says 172.21.2.117:44312) 2021-02-10T11:30:27.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 -- 172.21.2.117:0/2570016633 learned_addr learned my addr 172.21.2.117:0/2570016633 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 -- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f08b4097ae0 msgr2=0x7f08b4107270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f08b4097ae0 0x7f08b4107270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 -- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f08b40c3170 msgr2=0x7f08b4098ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f08b40c3170 0x7f08b4098ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 -- 172.21.2.117:0/2570016633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b4098100 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.274 7f08b9ac5700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f08a800b5b0 tx=0x7f08a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a800bd40 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a8006240 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f08a8005720 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b4098100 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b40997f0 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f08a80054a0 con 0x7f08b4107890 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 0x7f089c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:27.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.275 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f08a8032570 con 0x7f08b4107890 2021-02-10T11:30:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.276 7f08ba2c6700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 0x7f089c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.276 7f08ba2c6700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 0x7f089c01abc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:27.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.308 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4107d50 con 0x7f08b4107890 2021-02-10T11:30:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.310 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f08a8030d50 con 0x7f08b4107890 2021-02-10T11:30:27.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.439 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802477", "target": ["mgr", ""]}) v1 -- 0x7f08b4107d50 con 0x7f089c0186c0 2021-02-10T11:30:27.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.443 7f08a6ffd700 1 -- 172.21.2.117:0/2570016633 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f08b0006960 con 0x7f089c0186c0 2021-02-10T11:30:27.433 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 msgr2=0x7f089c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 0x7f089c01abc0 crc :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 msgr2=0x7f08b40988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f08a800b5b0 tx=0x7f08a800b950).stop 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 shutdown_connections 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f089c0186c0 0x7f089c01abc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f08b4107890 0x7f08b40988c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f08b40c3170 0x7f08b4098ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 --2- 172.21.2.117:0/2570016633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f08b4097ae0 0x7f08b4107270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:27.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 >> 172.21.2.117:0/2570016633 conn(0x7f08b40bd740 msgr2=0x7f08b40bdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:27.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 shutdown_connections 2021-02-10T11:30:27.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:27.444 7f08bc4aa700 1 -- 172.21.2.117:0/2570016633 wait complete. 2021-02-10T11:30:27.443 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802477 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe 2021-02-10T11:30:27.443 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:27.443 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:27.444 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:27.779 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:27.780 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:27.780 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:28.105 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:28.108 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:28.158 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:28.158 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:28.158 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228257 s, 45.9 MB/s 2021-02-10T11:30:28.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:28.159 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:28.159 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:28.441 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:28.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:28.442 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:28.818 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228552 s, 45.9 MB/s 2021-02-10T11:30:28.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:28.819 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:28.819 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:29.153 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:29.154 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:29.154 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:29.529 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:29.529 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:29.529 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235048 s, 44.6 MB/s 2021-02-10T11:30:29.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:29.530 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:29.530 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:29.865 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:29.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:29.866 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:30.240 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:30.240 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:30.240 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236803 s, 44.3 MB/s 2021-02-10T11:30:30.241 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:30.242 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:30.242 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:30.577 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:30.577 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:30.578 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:30.951 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:30.951 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:30.951 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224679 s, 46.7 MB/s 2021-02-10T11:30:30.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:30.953 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:30.953 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:31.289 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:31.289 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:31.289 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:31.665 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:31.666 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:31.666 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024876 s, 42.2 MB/s 2021-02-10T11:30:31.667 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:31.667 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:31.667 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:32.003 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:32.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:32.004 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234759 s, 44.7 MB/s 2021-02-10T11:30:32.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:32.380 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:32.380 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:32.714 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:32.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:32.715 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:33.086 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:33.087 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:33.087 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240908 s, 43.5 MB/s 2021-02-10T11:30:33.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:33.088 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:33.088 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:33.423 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:33.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:33.424 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:33.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:33.800 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024165 s, 43.4 MB/s 2021-02-10T11:30:33.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:33.801 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:33.801 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:34.138 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:34.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:34.139 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802477/cbef569a-4285-47fe-be02-ca6ba19253fe/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:34.514 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:34.514 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:34.514 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243794 s, 43.0 MB/s 2021-02-10T11:30:34.515 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802478 2021-02-10T11:30:34.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.939 7f9e73f83700 1 Processor -- start 2021-02-10T11:30:34.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.939 7f9e73f83700 1 -- start start 2021-02-10T11:30:34.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e73f83700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e6c07da20 0x7f9e6c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e73f83700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e6c0e5180 0x7f9e6c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e73f83700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c077a70 con 0x7f9e6c0e4d80 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e73f83700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0e5c00 con 0x7f9e6c0e5180 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e73f83700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0777d0 con 0x7f9e6c07da20 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e6c0e5180 0x7f9e6c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e6c0e5180 0x7f9e6c079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44318/0 (socket says 172.21.2.117:44318) 2021-02-10T11:30:34.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 -- 172.21.2.117:0/2638503930 learned_addr learned my addr 172.21.2.117:0/2638503930 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e71d9f700 1 --2- 172.21.2.117:0/2638503930 >> v2:172.21.2.125:3300/0 conn(0x7f9e6c07da20 0x7f9e6c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 -- 172.21.2.117:0/2638503930 >> v2:172.21.2.125:3300/0 conn(0x7f9e6c07da20 msgr2=0x7f9e6c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 --2- 172.21.2.117:0/2638503930 >> v2:172.21.2.125:3300/0 conn(0x7f9e6c07da20 0x7f9e6c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 -- 172.21.2.117:0/2638503930 >> v1:172.21.2.125:6789/0 conn(0x7f9e6c0e4d80 legacy=0x7f9e6c1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.940 7f9e725a0700 1 -- 172.21.2.117:0/2638503930 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c1b08c0 con 0x7f9e6c0e5180 2021-02-10T11:30:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e725a0700 1 --2- 172.21.2.117:0/2638503930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c0e5180 0x7f9e6c079000 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f9e64006f00 tx=0x7f9e64009f10).ready entity=mon.0 client_cookie=471a4cdf02f53c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e70d9d700 1 -- 172.21.2.117:0/2638503930 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e6400b3b0 con 0x7f9e6c0e5180 2021-02-10T11:30:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e70d9d700 1 -- 172.21.2.117:0/2638503930 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6400b630 con 0x7f9e6c0e5180 2021-02-10T11:30:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e70d9d700 1 -- 172.21.2.117:0/2638503930 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e6400bb10 con 0x7f9e6c0e5180 2021-02-10T11:30:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e73f83700 1 -- 172.21.2.117:0/2638503930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c0e5180 msgr2=0x7f9e6c079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.941 7f9e73f83700 1 --2- 172.21.2.117:0/2638503930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c0e5180 0x7f9e6c079000 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f9e64006f00 tx=0x7f9e64009f10).stop 2021-02-10T11:30:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 -- 172.21.2.117:0/2638503930 shutdown_connections 2021-02-10T11:30:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 --2- 172.21.2.117:0/2638503930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c0e5180 0x7f9e6c079000 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 --2- 172.21.2.117:0/2638503930 >> v2:172.21.2.125:3300/0 conn(0x7f9e6c07da20 0x7f9e6c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 -- 172.21.2.117:0/2638503930 >> 172.21.2.117:0/2638503930 conn(0x7f9e6c0ce430 msgr2=0x7f9e6c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 -- 172.21.2.117:0/2638503930 shutdown_connections 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.942 7f9e73f83700 1 -- 172.21.2.117:0/2638503930 wait complete. 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.956 7f9e73f83700 1 Processor -- start 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.956 7f9e73f83700 1 -- start start 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e6c091a50 0x7f9e6c0b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e6c199750 0x7f9e6c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c091ea0 con 0x7f9e6c187bd0 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c187fd0 con 0x7f9e6c199750 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e73f83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c1881f0 con 0x7f9e6c091a50 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44326/0 (socket says 172.21.2.117:44326) 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 -- 172.21.2.117:0/2352930877 learned_addr learned my addr 172.21.2.117:0/2352930877 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 -- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e6c091a50 msgr2=0x7f9e6c0b8cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e6c091a50 0x7f9e6c0b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 -- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e6c199750 msgr2=0x7f9e6c0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e6c199750 0x7f9e6c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.957 7f9e7159e700 1 -- 172.21.2.117:0/2352930877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c0abee0 con 0x7f9e6c187bd0 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e7159e700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f9e5c006ed0 tx=0x7f9e5c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e5c006240 con 0x7f9e6c187bd0 2021-02-10T11:30:34.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0054b0 con 0x7f9e6c187bd0 2021-02-10T11:30:34.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e5c005990 con 0x7f9e6c187bd0 2021-02-10T11:30:34.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c0abee0 con 0x7f9e6c187bd0 2021-02-10T11:30:34.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.958 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c0abee0 con 0x7f9e6c187bd0 2021-02-10T11:30:34.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.959 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e5c006030 con 0x7f9e6c187bd0 2021-02-10T11:30:34.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.959 7f9e627fc700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 0x7f9e5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:34.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.959 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e5c032690 con 0x7f9e6c187bd0 2021-02-10T11:30:34.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.959 7f9e71d9f700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 0x7f9e5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:34.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.960 7f9e71d9f700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 0x7f9e5401abc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:34.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.989 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c077a70 con 0x7f9e6c187bd0 2021-02-10T11:30:34.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:34.992 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e5c000de0 con 0x7f9e6c187bd0 2021-02-10T11:30:35.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.118 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802478", "target": ["mgr", ""]}) v1 -- 0x7f9e6c187fd0 con 0x7f9e540186c0 2021-02-10T11:30:35.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.143 7f9e627fc700 1 -- 172.21.2.117:0/2352930877 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9e680009b0 con 0x7f9e540186c0 2021-02-10T11:30:35.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 msgr2=0x7f9e5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 0x7f9e5401abc0 crc :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 msgr2=0x7f9e6c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f9e5c006ed0 tx=0x7f9e5c00b960).stop 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 shutdown_connections 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e540186c0 0x7f9e5401abc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e6c187bd0 0x7f9e6c199130 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e6c199750 0x7f9e6c0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 --2- 172.21.2.117:0/2352930877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e6c091a50 0x7f9e6c0b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 >> 172.21.2.117:0/2352930877 conn(0x7f9e6c0b5320 msgr2=0x7f9e6c1b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 shutdown_connections 2021-02-10T11:30:35.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.151 7f9e73f83700 1 -- 172.21.2.117:0/2352930877 wait complete. 2021-02-10T11:30:35.150 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802478 2021-02-10T11:30:35.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.577 7f7d3fdda700 1 Processor -- start 2021-02-10T11:30:35.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.577 7f7d3fdda700 1 -- start start 2021-02-10T11:30:35.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3fdda700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7d38079000 0x7f7d380e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3fdda700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381b00f0 0x7f7d381b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3fdda700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38077a70 con 0x7f7d3807da20 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3fdda700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d380e53c0 con 0x7f7d38079000 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3fdda700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38079400 con 0x7f7d381b00f0 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3dbf6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7d3807da20 0x7f7d380771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44580/0 (socket says 172.21.2.117:44580) 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3dbf6700 1 -- 172.21.2.117:0/3384902003 learned_addr learned my addr 172.21.2.117:0/3384902003 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616555367 0 0) 0x7f7d2c0010b0 con 0x7f7d3807da20 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24002fe0 con 0x7f7d3807da20 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3e3f7700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381b00f0 0x7f7d381b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3d3f5700 1 --2- 172.21.2.117:0/3384902003 >> v2:172.21.2.125:3300/0 conn(0x7f7d38079000 0x7f7d380e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.578 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2282200978 0 0) 0x7f7d2c0010b0 con 0x7f7d3807da20 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381b00f0 msgr2=0x7f7d381b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381b00f0 0x7f7d381b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 >> v2:172.21.2.125:3300/0 conn(0x7f7d38079000 msgr2=0x7f7d380e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 --2- 172.21.2.117:0/3384902003 >> v2:172.21.2.125:3300/0 conn(0x7f7d38079000 0x7f7d380e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3807de20 con 0x7f7d3807da20 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7d2c0015b0 con 0x7f7d3807da20 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 >> v1:172.21.2.117:6789/0 conn(0x7f7d3807da20 legacy=0x7f7d380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d24003af0 0x7f7d24005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d2404a7a0 0x7f7d2404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d2404d500 0x7f7d2404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240502d0 con 0x7f7d2404a7a0 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d240504f0 con 0x7f7d2404d500 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d24050710 con 0x7f7d24003af0 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d2404a7a0 0x7f7d2404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3dbf6700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d24003af0 0x7f7d24005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3e3f7700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d2404d500 0x7f7d2404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 -- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d24003af0 msgr2=0x7f7d24005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d24003af0 0x7f7d24005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 -- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d2404d500 msgr2=0x7f7d2404fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d2404d500 0x7f7d2404fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 -- 172.21.2.117:0/3384902003 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d240509b0 con 0x7f7d2404a7a0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3d3f5700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d2404a7a0 0x7f7d2404cc60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f7d3400c8f0 tx=0x7f7d3400cc90).ready entity=mon.0 client_cookie=92a92c33f687c503 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400cdd0 con 0x7f7d2404a7a0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34009310 con 0x7f7d2404a7a0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3cbf4700 1 -- 172.21.2.117:0/3384902003 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d34009830 con 0x7f7d2404a7a0 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3fdda700 1 -- 172.21.2.117:0/3384902003 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d2404a7a0 msgr2=0x7f7d2404cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3fdda700 1 --2- 172.21.2.117:0/3384902003 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d2404a7a0 0x7f7d2404cc60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f7d3400c8f0 tx=0x7f7d3400cc90).stop 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.579 7f7d3e3f7700 1 -- 172.21.2.117:0/3384902003 reap_dead start 2021-02-10T11:30:35.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.580 7f7d3fdda700 1 -- 172.21.2.117:0/3384902003 shutdown_connections 2021-02-10T11:30:35.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.580 7f7d3fdda700 1 -- 172.21.2.117:0/3384902003 >> 172.21.2.117:0/3384902003 conn(0x7f7d380ce450 msgr2=0x7f7d380e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.580 7f7d3fdda700 1 -- 172.21.2.117:0/3384902003 shutdown_connections 2021-02-10T11:30:35.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.580 7f7d3fdda700 1 -- 172.21.2.117:0/3384902003 wait complete. 2021-02-10T11:30:35.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 Processor -- start 2021-02-10T11:30:35.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 -- start start 2021-02-10T11:30:35.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381a46b0 0x7f7d38190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d380a50e0 0x7f7d380a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c3040 con 0x7f7d380a46c0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38190a20 con 0x7f7d380a50e0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.595 7f7d3fdda700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380a54e0 con 0x7f7d381a46b0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44342/0 (socket says 172.21.2.117:44342) 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 -- 172.21.2.117:0/2762074310 learned_addr learned my addr 172.21.2.117:0/2762074310 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3dbf6700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381a46b0 0x7f7d38190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3e3f7700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d380a50e0 0x7f7d380a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 -- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381a46b0 msgr2=0x7f7d38190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381a46b0 0x7f7d38190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 -- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d380a50e0 msgr2=0x7f7d380a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d380a50e0 0x7f7d380a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 -- 172.21.2.117:0/2762074310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380a5700 con 0x7f7d380a46c0 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3d3f5700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f7d34009d10 tx=0x7f7d3400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400c1e0 con 0x7f7d380a46c0 2021-02-10T11:30:35.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3400c8f0 con 0x7f7d380a46c0 2021-02-10T11:30:35.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3400b6f0 con 0x7f7d380a46c0 2021-02-10T11:30:35.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.596 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380a5700 con 0x7f7d380a46c0 2021-02-10T11:30:35.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.597 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d380a5700 con 0x7f7d380a46c0 2021-02-10T11:30:35.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.597 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d3400b6f0 con 0x7f7d380a46c0 2021-02-10T11:30:35.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.597 7f7d2a7fc700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:35.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.597 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7d340327a0 con 0x7f7d380a46c0 2021-02-10T11:30:35.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.597 7f7d3dbf6700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:35.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.599 7f7d3dbf6700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 0x7f7d2001abc0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:35.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.628 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38077a70 con 0x7f7d380a46c0 2021-02-10T11:30:35.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.630 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d34000c90 con 0x7f7d380a46c0 2021-02-10T11:30:35.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.759 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802478", "target": ["mgr", ""]}) v1 -- 0x7f7d38077a70 con 0x7f7d200186c0 2021-02-10T11:30:35.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.763 7f7d2a7fc700 1 -- 172.21.2.117:0/2762074310 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7d2c001420 con 0x7f7d200186c0 2021-02-10T11:30:35.757 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 msgr2=0x7f7d2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 0x7f7d2001abc0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 msgr2=0x7f7d380a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f7d34009d10 tx=0x7f7d3400b300).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 shutdown_connections 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d200186c0 0x7f7d2001abc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d380a46c0 0x7f7d380a4ac0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d380a50e0 0x7f7d380a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 --2- 172.21.2.117:0/2762074310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d381a46b0 0x7f7d38190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 >> 172.21.2.117:0/2762074310 conn(0x7f7d380b5140 msgr2=0x7f7d380a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:35.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 shutdown_connections 2021-02-10T11:30:35.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:35.769 7f7d3fdda700 1 -- 172.21.2.117:0/2762074310 wait complete. 2021-02-10T11:30:35.767 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802478 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f 2021-02-10T11:30:35.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:35.768 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:35.768 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:36.103 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:36.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:36.104 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:36.485 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:36.486 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:36.486 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251221 s, 41.7 MB/s 2021-02-10T11:30:36.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:36.487 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:36.487 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:36.822 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:36.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:36.823 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:37.199 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:37.199 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:37.199 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259387 s, 40.4 MB/s 2021-02-10T11:30:37.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:37.200 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:37.200 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:37.535 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:37.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:37.536 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239337 s, 43.8 MB/s 2021-02-10T11:30:37.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:37.909 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:37.909 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:38.244 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:38.244 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:38.245 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:38.619 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:38.620 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:38.620 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239795 s, 43.7 MB/s 2021-02-10T11:30:38.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:38.621 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:38.621 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:38.957 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:38.958 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:38.958 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:39.332 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:39.333 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:39.333 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232354 s, 45.1 MB/s 2021-02-10T11:30:39.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:39.334 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:39.334 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:39.670 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:39.671 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:39.671 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:40.047 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:40.047 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:40.047 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259503 s, 40.4 MB/s 2021-02-10T11:30:40.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:40.048 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:40.048 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:40.384 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:40.384 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:40.385 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:40.759 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:40.759 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:40.760 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237711 s, 44.1 MB/s 2021-02-10T11:30:40.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:40.761 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:40.761 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:41.096 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:41.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:41.097 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:41.473 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262348 s, 40.0 MB/s 2021-02-10T11:30:41.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:41.474 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:41.475 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:41.810 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:41.810 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:41.811 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255369 s, 41.1 MB/s 2021-02-10T11:30:42.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:42.188 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:42.188 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:42.523 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:42.523 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:42.524 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802478/79f5d933-b547-480d-b95f-1c4680b0da5f/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:42.895 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:42.895 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:42.895 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236176 s, 44.4 MB/s 2021-02-10T11:30:42.896 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802479 2021-02-10T11:30:43.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.320 7f1004796700 1 Processor -- start 2021-02-10T11:30:43.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.320 7f1004796700 1 -- start start 2021-02-10T11:30:43.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1004796700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ffc0d1530 0x7f0ffc0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1004796700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc077a70 con 0x7f0ffc079000 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1004796700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ffc079400 con 0x7f0ffc07da20 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1004796700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ffc07de20 con 0x7f0ffc0d1530 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1002db3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ffc0d1530 0x7f0ffc0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1001db1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0ffc079000 0x7f0ffc0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44600/0 (socket says 172.21.2.117:44600) 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1001db1700 1 -- 172.21.2.117:0/2611823538 learned_addr learned my addr 172.21.2.117:0/2611823538 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1002db3700 1 -- 172.21.2.117:0/2611823538 >> v1:172.21.2.125:6789/0 conn(0x7f0ffc07da20 legacy=0x7f0ffc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.321 7f1002db3700 1 -- 172.21.2.117:0/2611823538 >> v1:172.21.2.117:6789/0 conn(0x7f0ffc079000 legacy=0x7f0ffc0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f1002db3700 1 -- 172.21.2.117:0/2611823538 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc1b00f0 con 0x7f0ffc0d1530 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f1002db3700 1 --2- 172.21.2.117:0/2611823538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0d1530 0x7f0ffc0d1930 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f0ff4006e30 tx=0x7f0ff4009de0).ready entity=mon.0 client_cookie=4521ba7d3e2f2ca4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f10015b0700 1 -- 172.21.2.117:0/2611823538 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ff400b300 con 0x7f0ffc0d1530 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f10015b0700 1 -- 172.21.2.117:0/2611823538 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400b580 con 0x7f0ffc0d1530 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f10015b0700 1 -- 172.21.2.117:0/2611823538 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ff400baa0 con 0x7f0ffc0d1530 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f1004796700 1 -- 172.21.2.117:0/2611823538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0d1530 msgr2=0x7f0ffc0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.322 7f1004796700 1 --2- 172.21.2.117:0/2611823538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0d1530 0x7f0ffc0d1930 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f0ff4006e30 tx=0x7f0ff4009de0).stop 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.323 7f1004796700 1 -- 172.21.2.117:0/2611823538 shutdown_connections 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.323 7f1004796700 1 --2- 172.21.2.117:0/2611823538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0d1530 0x7f0ffc0d1930 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.323 7f1004796700 1 -- 172.21.2.117:0/2611823538 >> 172.21.2.117:0/2611823538 conn(0x7f0ffc0ce450 msgr2=0x7f0ffc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.323 7f1004796700 1 -- 172.21.2.117:0/2611823538 shutdown_connections 2021-02-10T11:30:43.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.323 7f1004796700 1 -- 172.21.2.117:0/2611823538 wait complete. 2021-02-10T11:30:43.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.337 7f1004796700 1 Processor -- start 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.337 7f1004796700 1 -- start start 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ffc0bfdd0 0x7f0ffc0c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ffc0c2e80 0x7f0ffc0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0b6200 con 0x7f0ffc0b4720 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0c38a0 con 0x7f0ffc0bfdd0 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f1004796700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ffc0c3b20 con 0x7f0ffc0c2e80 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44352/0 (socket says 172.21.2.117:44352) 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 -- 172.21.2.117:0/1316368343 learned_addr learned my addr 172.21.2.117:0/1316368343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 -- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ffc0c2e80 msgr2=0x7f0ffc0c3280 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ffc0c2e80 0x7f0ffc0c3280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 -- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ffc0bfdd0 msgr2=0x7f0ffc0c2860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ffc0bfdd0 0x7f0ffc0c2860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.338 7f10025b2700 1 -- 172.21.2.117:0/1316368343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0b4b20 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f10025b2700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f0ff800b5b0 tx=0x7f0ff800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ff800bd40 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff8006240 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ff8005720 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f1004796700 1 -- 172.21.2.117:0/1316368343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ffc0b4b20 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f1004796700 1 -- 172.21.2.117:0/1316368343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ffc0b4b20 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ff80054a0 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 0x7f0fe401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.339 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0ff8032900 con 0x7f0ffc0b4720 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.340 7f1001db1700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 0x7f0fe401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.340 7f1001db1700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 0x7f0fe401abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.371 7f1004796700 1 -- 172.21.2.117:0/1316368343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ffc0ce450 con 0x7f0ffc0b4720 2021-02-10T11:30:43.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.373 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ff80009b0 con 0x7f0ffc0b4720 2021-02-10T11:30:43.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.499 7f1004796700 1 -- 172.21.2.117:0/1316368343 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802479", "target": ["mgr", ""]}) v1 -- 0x7f0ffc0c38a0 con 0x7f0fe40186c0 2021-02-10T11:30:43.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.526 7f0ff2ffd700 1 -- 172.21.2.117:0/1316368343 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0fec007690 con 0x7f0fe40186c0 2021-02-10T11:30:43.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.532 7f1004796700 1 -- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 msgr2=0x7f0fe401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.532 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 0x7f0fe401abc0 crc :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.532 7f1004796700 1 -- 172.21.2.117:0/1316368343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 msgr2=0x7f0ffc0bf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.532 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f0ff800b5b0 tx=0x7f0ff800b950).stop 2021-02-10T11:30:43.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 -- 172.21.2.117:0/1316368343 shutdown_connections 2021-02-10T11:30:43.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ffc0c2e80 0x7f0ffc0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ffc0bfdd0 0x7f0ffc0c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fe40186c0 0x7f0fe401abc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 --2- 172.21.2.117:0/1316368343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ffc0b4720 0x7f0ffc0bf7b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 -- 172.21.2.117:0/1316368343 >> 172.21.2.117:0/1316368343 conn(0x7f0ffc1116a0 msgr2=0x7f0ffc0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 -- 172.21.2.117:0/1316368343 shutdown_connections 2021-02-10T11:30:43.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.533 7f1004796700 1 -- 172.21.2.117:0/1316368343 wait complete. 2021-02-10T11:30:43.531 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802479 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.955 7fcaa58d1700 1 Processor -- start 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.955 7fcaa58d1700 1 -- start start 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fcaa58d1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcaa0079000 0x7fcaa00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fcaa58d1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa00d1d20 con 0x7fcaa007da20 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fcaa58d1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa0077a70 con 0x7fcaa01b00f0 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fcaa58d1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa00e53c0 con 0x7fcaa0079000 2021-02-10T11:30:43.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fca9f574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcaa007da20 0x7fcaa00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44610/0 (socket says 172.21.2.117:44610) 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fca9f574700 1 -- 172.21.2.117:0/3011698588 learned_addr learned my addr 172.21.2.117:0/3011698588 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fca9ed73700 1 --2- 172.21.2.117:0/3011698588 >> v2:172.21.2.125:3300/0 conn(0x7fcaa0079000 0x7fcaa00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fca9e572700 1 -- 172.21.2.117:0/3011698588 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631358371 0 0) 0x7fca900010b0 con 0x7fcaa007da20 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.956 7fca9e572700 1 -- 172.21.2.117:0/3011698588 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca8c002fe0 con 0x7fcaa007da20 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9ed73700 1 -- 172.21.2.117:0/3011698588 >> v1:172.21.2.125:6789/0 conn(0x7fcaa01b00f0 legacy=0x7fcaa00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9e572700 1 -- 172.21.2.117:0/3011698588 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305068188 0 0) 0x7fca98001400 con 0x7fcaa01b00f0 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9ed73700 1 -- 172.21.2.117:0/3011698588 >> v1:172.21.2.117:6789/0 conn(0x7fcaa007da20 legacy=0x7fcaa00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9ed73700 1 -- 172.21.2.117:0/3011698588 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa0079400 con 0x7fcaa0079000 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9ed73700 1 --2- 172.21.2.117:0/3011698588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa0079000 0x7fcaa00e4da0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fca88006650 tx=0x7fca88009a80).ready entity=mon.1 client_cookie=8430e6b93c35af9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9e572700 1 -- 172.21.2.117:0/3011698588 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca8800c0c0 con 0x7fcaa0079000 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9e572700 1 -- 172.21.2.117:0/3011698588 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8800c340 con 0x7fcaa0079000 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fca9e572700 1 -- 172.21.2.117:0/3011698588 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca8800c860 con 0x7fcaa0079000 2021-02-10T11:30:43.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fcaa58d1700 1 -- 172.21.2.117:0/3011698588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa0079000 msgr2=0x7fcaa00e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.957 7fcaa58d1700 1 --2- 172.21.2.117:0/3011698588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa0079000 0x7fcaa00e4da0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fca88006650 tx=0x7fca88009a80).stop 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.958 7fcaa58d1700 1 -- 172.21.2.117:0/3011698588 shutdown_connections 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.958 7fcaa58d1700 1 --2- 172.21.2.117:0/3011698588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa0079000 0x7fcaa00e4da0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.958 7fcaa58d1700 1 -- 172.21.2.117:0/3011698588 >> 172.21.2.117:0/3011698588 conn(0x7fcaa00ce450 msgr2=0x7fcaa00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.958 7fcaa58d1700 1 -- 172.21.2.117:0/3011698588 shutdown_connections 2021-02-10T11:30:43.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.958 7fcaa58d1700 1 -- 172.21.2.117:0/3011698588 wait complete. 2021-02-10T11:30:43.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 Processor -- start 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 -- start start 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaa0107c20 0x7fcaa0096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa008a180 0x7fcaa008a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa0108020 con 0x7fcaa01a7240 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa01a7c60 con 0x7fcaa008a180 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.973 7fcaa58d1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa01a7e80 con 0x7fcaa0107c20 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44368/0 (socket says 172.21.2.117:44368) 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 -- 172.21.2.117:0/1970628368 learned_addr learned my addr 172.21.2.117:0/1970628368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 -- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaa0107c20 msgr2=0x7fcaa0096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaa0107c20 0x7fcaa0096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 -- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa008a180 msgr2=0x7fcaa008a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa008a180 0x7fcaa008a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 -- 172.21.2.117:0/1970628368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa008b250 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca9fd75700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fca98009940 tx=0x7fca9800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9800c2f0 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9800c530 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9800ca10 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa008b250 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.974 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa008b7e0 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.975 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca9800c710 con 0x7fcaa01a7240 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.975 7fca97fff700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 0x7fca8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:43.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.975 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fca98032ab0 con 0x7fcaa01a7240 2021-02-10T11:30:43.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.976 7fca9f574700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 0x7fca8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:43.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:43.976 7fca9f574700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 0x7fca8401abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:43.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.006 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa0077a70 con 0x7fcaa01a7240 2021-02-10T11:30:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.009 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca9801f8e0 con 0x7fcaa01a7240 2021-02-10T11:30:44.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.136 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802479", "target": ["mgr", ""]}) v1 -- 0x7fcaa008b7e0 con 0x7fca840186c0 2021-02-10T11:30:44.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.140 7fca97fff700 1 -- 172.21.2.117:0/1970628368 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fca90006e20 con 0x7fca840186c0 2021-02-10T11:30:44.134 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.145 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 msgr2=0x7fca8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.145 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 0x7fca8401abc0 crc :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.145 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 msgr2=0x7fcaa01a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.145 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fca98009940 tx=0x7fca9800bcf0).stop 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 shutdown_connections 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca840186c0 0x7fca8401abc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaa01a7240 0x7fcaa01a7640 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaa008a180 0x7fcaa008a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:44.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 --2- 172.21.2.117:0/1970628368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaa0107c20 0x7fcaa0096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:44.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 >> 172.21.2.117:0/1970628368 conn(0x7fcaa00a1880 msgr2=0x7fcaa00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:44.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 shutdown_connections 2021-02-10T11:30:44.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:44.146 7fcaa58d1700 1 -- 172.21.2.117:0/1970628368 wait complete. 2021-02-10T11:30:44.144 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802479 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2 2021-02-10T11:30:44.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:44.145 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:44.145 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:44.481 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:44.481 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:44.482 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:44.865 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:44.865 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:44.865 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251385 s, 41.7 MB/s 2021-02-10T11:30:44.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:44.866 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:44.866 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:45.201 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:45.201 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:45.201 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233052 s, 45.0 MB/s 2021-02-10T11:30:45.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:45.576 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:45.576 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:45.910 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:45.910 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:45.911 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:46.287 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:46.287 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024351 s, 43.1 MB/s 2021-02-10T11:30:46.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:46.288 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:46.288 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:46.623 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:46.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:46.623 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:46.999 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:46.999 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:46.999 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241531 s, 43.4 MB/s 2021-02-10T11:30:47.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:47.000 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:47.000 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:47.335 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:47.335 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:47.335 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:47.708 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256559 s, 40.9 MB/s 2021-02-10T11:30:47.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:47.881 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:47.881 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:48.217 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:48.217 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:48.217 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:48.594 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:48.594 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:48.594 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249052 s, 42.1 MB/s 2021-02-10T11:30:48.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:48.595 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:48.595 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:48.930 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:48.930 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:48.931 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230704 s, 45.5 MB/s 2021-02-10T11:30:49.305 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:49.305 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:49.305 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:49.642 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:49.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:49.643 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:50.019 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:50.019 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:50.019 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024414 s, 42.9 MB/s 2021-02-10T11:30:50.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:50.020 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:50.020 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:50.356 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:50.356 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:50.356 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:50.729 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:50.730 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:50.730 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237428 s, 44.2 MB/s 2021-02-10T11:30:50.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:50.731 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:50.731 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:51.066 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:51.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:51.067 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802479/cfba26b1-3db3-4a08-ae28-61c2766ee4c2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:51.441 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:51.441 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:51.441 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239159 s, 43.8 MB/s 2021-02-10T11:30:51.442 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802480 2021-02-10T11:30:51.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.867 7fd699940700 1 Processor -- start 2021-02-10T11:30:51.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.867 7fd699940700 1 -- start start 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd69407da20 0x7fd6940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6940e4da0 0x7fd6941b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6940e51a0 0x7fd694077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd694077a70 con 0x7fd6940e4da0 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6940777d0 con 0x7fd69407da20 2021-02-10T11:30:51.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69407de20 con 0x7fd6940e51a0 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6940e4da0 0x7fd6941b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6940e4da0 0x7fd6941b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44374/0 (socket says 172.21.2.117:44374) 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 -- 172.21.2.117:0/197868879 learned_addr learned my addr 172.21.2.117:0/197868879 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 -- 172.21.2.117:0/197868879 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6940e51a0 msgr2=0x7fd694077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 --2- 172.21.2.117:0/197868879 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6940e51a0 0x7fd694077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 -- 172.21.2.117:0/197868879 >> v2:172.21.2.125:3300/0 conn(0x7fd69407da20 msgr2=0x7fd6940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:51.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 --2- 172.21.2.117:0/197868879 >> v2:172.21.2.125:3300/0 conn(0x7fd69407da20 0x7fd6940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 -- 172.21.2.117:0/197868879 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6941b0710 con 0x7fd6940e4da0 2021-02-10T11:30:51.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692d73700 1 --2- 172.21.2.117:0/197868879 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940e4da0 0x7fd6941b00f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fd67c006700 tx=0x7fd67c009b00).ready entity=mon.0 client_cookie=489cdd9cc2c075d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692572700 1 -- 172.21.2.117:0/197868879 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd67c00c0c0 con 0x7fd6940e4da0 2021-02-10T11:30:51.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692572700 1 -- 172.21.2.117:0/197868879 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c00c400 con 0x7fd6940e4da0 2021-02-10T11:30:51.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd692572700 1 -- 172.21.2.117:0/197868879 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd67c00c8e0 con 0x7fd6940e4da0 2021-02-10T11:30:51.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 -- 172.21.2.117:0/197868879 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940e4da0 msgr2=0x7fd6941b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.868 7fd699940700 1 --2- 172.21.2.117:0/197868879 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940e4da0 0x7fd6941b00f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7fd67c006700 tx=0x7fd67c009b00).stop 2021-02-10T11:30:51.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 -- 172.21.2.117:0/197868879 shutdown_connections 2021-02-10T11:30:51.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 --2- 172.21.2.117:0/197868879 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6940e51a0 0x7fd694077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 --2- 172.21.2.117:0/197868879 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940e4da0 0x7fd6941b00f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:51.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 --2- 172.21.2.117:0/197868879 >> v2:172.21.2.125:3300/0 conn(0x7fd69407da20 0x7fd6940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 -- 172.21.2.117:0/197868879 >> 172.21.2.117:0/197868879 conn(0x7fd6940ce450 msgr2=0x7fd6940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:51.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 -- 172.21.2.117:0/197868879 shutdown_connections 2021-02-10T11:30:51.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.869 7fd699940700 1 -- 172.21.2.117:0/197868879 wait complete. 2021-02-10T11:30:51.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.883 7fd699940700 1 Processor -- start 2021-02-10T11:30:51.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 -- start start 2021-02-10T11:30:51.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6940c33d0 0x7fd6940b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd694090de0 0x7fd6940911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694090600 con 0x7fd6940b3ff0 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd694090860 con 0x7fd6940c33d0 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd699940700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6940b3c70 con 0x7fd694090de0 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44382/0 (socket says 172.21.2.117:44382) 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 -- 172.21.2.117:0/3345681612 learned_addr learned my addr 172.21.2.117:0/3345681612 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 -- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd694090de0 msgr2=0x7fd6940911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd694090de0 0x7fd6940911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 -- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6940c33d0 msgr2=0x7fd6940b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6940c33d0 0x7fd6940b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.884 7fd693574700 1 -- 172.21.2.117:0/3345681612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd694091800 con 0x7fd6940b3ff0 2021-02-10T11:30:51.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd693574700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fd684006f00 tx=0x7fd68400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd68400bd30 con 0x7fd6940b3ff0 2021-02-10T11:30:51.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684000c90 con 0x7fd6940b3ff0 2021-02-10T11:30:51.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6840022a0 con 0x7fd6940b3ff0 2021-02-10T11:30:51.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd699940700 1 -- 172.21.2.117:0/3345681612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd694091800 con 0x7fd6940b3ff0 2021-02-10T11:30:51.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd699940700 1 -- 172.21.2.117:0/3345681612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd694091800 con 0x7fd6940b3ff0 2021-02-10T11:30:51.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd684005310 con 0x7fd6940b3ff0 2021-02-10T11:30:51.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 0x7fd67801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:51.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.885 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd684032750 con 0x7fd6940b3ff0 2021-02-10T11:30:51.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.886 7fd692d73700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 0x7fd67801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:51.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.886 7fd692d73700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 0x7fd67801abd0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:51.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.917 7fd699940700 1 -- 172.21.2.117:0/3345681612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd694103370 con 0x7fd6940b3ff0 2021-02-10T11:30:51.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:51.920 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd684006b90 con 0x7fd6940b3ff0 2021-02-10T11:30:52.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.044 7fd699940700 1 -- 172.21.2.117:0/3345681612 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802480", "target": ["mgr", ""]}) v1 -- 0x7fd694077a70 con 0x7fd6780186d0 2021-02-10T11:30:52.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.073 7fd68bfff700 1 -- 172.21.2.117:0/3345681612 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd67c0091c0 con 0x7fd6780186d0 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 -- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 msgr2=0x7fd67801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 0x7fd67801abd0 crc :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 -- 172.21.2.117:0/3345681612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 msgr2=0x7fd6940b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fd684006f00 tx=0x7fd68400b940).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 -- 172.21.2.117:0/3345681612 shutdown_connections 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd694090de0 0x7fd6940911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6940c33d0 0x7fd6940b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6780186d0 0x7fd67801abd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 --2- 172.21.2.117:0/3345681612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6940b3ff0 0x7fd6940b3030 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.078 7fd699940700 1 -- 172.21.2.117:0/3345681612 >> 172.21.2.117:0/3345681612 conn(0x7fd6940a1420 msgr2=0x7fd69409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.079 7fd699940700 1 -- 172.21.2.117:0/3345681612 shutdown_connections 2021-02-10T11:30:52.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.079 7fd699940700 1 -- 172.21.2.117:0/3345681612 wait complete. 2021-02-10T11:30:52.077 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802480 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 Processor -- start 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 -- start start 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcbb40d1510 0x7fcbb40d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb4077a70 con 0x7fcbb4079000 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb4079400 con 0x7fcbb407da20 2021-02-10T11:30:52.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.506 7fcbba2c3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb407de20 con 0x7fcbb40d1510 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcbb40d1510 0x7fcbb40d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcbb40d1510 0x7fcbb40d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44386/0 (socket says 172.21.2.117:44386) 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 -- 172.21.2.117:0/4063978427 learned_addr learned my addr 172.21.2.117:0/4063978427 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 -- 172.21.2.117:0/4063978427 >> v1:172.21.2.125:6789/0 conn(0x7fcbb407da20 legacy=0x7fcbb40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 -- 172.21.2.117:0/4063978427 >> v1:172.21.2.117:6789/0 conn(0x7fcbb4079000 legacy=0x7fcbb40e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 -- 172.21.2.117:0/4063978427 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb41b00d0 con 0x7fcbb40d1510 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb88e0700 1 --2- 172.21.2.117:0/4063978427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40d1510 0x7fcbb40d1910 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fcbac006f00 tx=0x7fcbac009f10).ready entity=mon.0 client_cookie=1cb650e19817666f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb2ffd700 1 -- 172.21.2.117:0/4063978427 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbac00b3b0 con 0x7fcbb40d1510 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb2ffd700 1 -- 172.21.2.117:0/4063978427 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac00b730 con 0x7fcbb40d1510 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbb2ffd700 1 -- 172.21.2.117:0/4063978427 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbac00bb90 con 0x7fcbb40d1510 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbba2c3700 1 -- 172.21.2.117:0/4063978427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40d1510 msgr2=0x7fcbb40d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.507 7fcbba2c3700 1 --2- 172.21.2.117:0/4063978427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40d1510 0x7fcbb40d1910 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fcbac006f00 tx=0x7fcbac009f10).stop 2021-02-10T11:30:52.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.508 7fcbba2c3700 1 -- 172.21.2.117:0/4063978427 shutdown_connections 2021-02-10T11:30:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.508 7fcbba2c3700 1 --2- 172.21.2.117:0/4063978427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40d1510 0x7fcbb40d1910 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.508 7fcbba2c3700 1 -- 172.21.2.117:0/4063978427 >> 172.21.2.117:0/4063978427 conn(0x7fcbb40ce430 msgr2=0x7fcbb40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.508 7fcbba2c3700 1 -- 172.21.2.117:0/4063978427 shutdown_connections 2021-02-10T11:30:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.508 7fcbba2c3700 1 -- 172.21.2.117:0/4063978427 wait complete. 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 Processor -- start 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 -- start start 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbb40bf960 0x7fcbb40c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbb40bfd60 0x7fcbb40c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40b6cf0 con 0x7fcbb40adc70 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40ae690 con 0x7fcbb40bf960 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.523 7fcbba2c3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41116c0 con 0x7fcbb40bfd60 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44396/0 (socket says 172.21.2.117:44396) 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 -- 172.21.2.117:0/3564254753 learned_addr learned my addr 172.21.2.117:0/3564254753 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb3fff700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbb40bf960 0x7fcbb40c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb37fe700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbb40bfd60 0x7fcbb40c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 -- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbb40bfd60 msgr2=0x7fcbb40c0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbb40bfd60 0x7fcbb40c0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 -- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbb40bf960 msgr2=0x7fcbb40c0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbb40bf960 0x7fcbb40c0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 -- 172.21.2.117:0/3564254753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4110420 con 0x7fcbb40adc70 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb88e0700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fcbac009390 tx=0x7fcbac00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbac003030 con 0x7fcbb40adc70 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac0068e0 con 0x7fcbb40adc70 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb4110420 con 0x7fcbb40adc70 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbac00c570 con 0x7fcbb40adc70 2021-02-10T11:30:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.524 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb41109b0 con 0x7fcbb40adc70 2021-02-10T11:30:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.525 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbac001270 con 0x7fcbb40adc70 2021-02-10T11:30:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.525 7fcbb0ff9700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 0x7fcb9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:30:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.525 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcbac0068e0 con 0x7fcbb40adc70 2021-02-10T11:30:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.525 7fcbb3fff700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 0x7fcb9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:30:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.525 7fcbb3fff700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 0x7fcb9401abc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:30:52.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.556 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb40ce430 con 0x7fcbb40adc70 2021-02-10T11:30:52.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.558 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbac009430 con 0x7fcbb40adc70 2021-02-10T11:30:52.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.685 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802480", "target": ["mgr", ""]}) v1 -- 0x7fcbb41109b0 con 0x7fcb940186c0 2021-02-10T11:30:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.688 7fcbb0ff9700 1 -- 172.21.2.117:0/3564254753 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fcba4006560 con 0x7fcb940186c0 2021-02-10T11:30:52.682 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 msgr2=0x7fcb9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 0x7fcb9401abc0 crc :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 msgr2=0x7fcbb40ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fcbac009390 tx=0x7fcbac00bd40).stop 2021-02-10T11:30:52.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 shutdown_connections 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb940186c0 0x7fcb9401abc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbb40adc70 0x7fcbb40ae070 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbb40bfd60 0x7fcbb40c0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 --2- 172.21.2.117:0/3564254753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbb40bf960 0x7fcbb40c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.694 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 >> 172.21.2.117:0/3564254753 conn(0x7fcbb41088c0 msgr2=0x7fcbb410f510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.695 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 shutdown_connections 2021-02-10T11:30:52.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:30:52.695 7fcbba2c3700 1 -- 172.21.2.117:0/3564254753 wait complete. 2021-02-10T11:30:52.692 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802480 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311 2021-02-10T11:30:52.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:52.693 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:52.693 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:53.035 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:53.035 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:53.035 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:53.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:53.419 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226922 s, 46.2 MB/s 2021-02-10T11:30:53.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:53.420 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:53.420 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:53.759 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:53.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:53.760 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237573 s, 44.1 MB/s 2021-02-10T11:30:54.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:54.137 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:54.137 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:54.473 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:54.473 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:54.473 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:54.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:54.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:54.851 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255318 s, 41.1 MB/s 2021-02-10T11:30:54.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:54.852 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:54.852 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:55.187 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:55.188 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:55.188 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024599 s, 42.6 MB/s 2021-02-10T11:30:55.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:55.565 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:55.565 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:55.901 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:55.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:55.902 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:56.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:56.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:56.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241887 s, 43.3 MB/s 2021-02-10T11:30:56.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:56.278 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:56.278 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:56.615 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:56.615 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:56.615 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:56.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:56.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:56.990 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240111 s, 43.7 MB/s 2021-02-10T11:30:56.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:56.991 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:56.991 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:57.328 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:57.328 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:57.328 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:57.703 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:57.703 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:57.703 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026629 s, 39.4 MB/s 2021-02-10T11:30:57.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:57.705 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:57.705 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:58.041 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:58.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:58.041 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:58.417 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:58.417 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:58.417 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233605 s, 44.9 MB/s 2021-02-10T11:30:58.418 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:58.418 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:58.419 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:58.475 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:58.479 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:30:58.755 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:58.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:58.755 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:59.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:59.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:59.132 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246307 s, 42.6 MB/s 2021-02-10T11:30:59.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:30:59.134 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:30:59.134 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:30:59.468 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:30:59.469 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:30:59.469 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802480/9f8a17ef-ab71-4561-b591-308dbc0aa311/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:30:59.845 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:30:59.845 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:30:59.846 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246343 s, 42.6 MB/s 2021-02-10T11:30:59.847 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802481 2021-02-10T11:31:00.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 Processor -- start 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 -- start start 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b280e51a0 0x7f4b280d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b28077a70 con 0x7f4b280e4da0 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280e5c20 con 0x7f4b280e51a0 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.271 7f4b2c857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280777d0 con 0x7f4b2807da20 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b280e51a0 0x7f4b280d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b280e51a0 0x7f4b280d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44400/0 (socket says 172.21.2.117:44400) 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 -- 172.21.2.117:0/2005282617 learned_addr learned my addr 172.21.2.117:0/2005282617 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2661c700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 -- 172.21.2.117:0/2005282617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b2807da20 msgr2=0x7f4b280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 -- 172.21.2.117:0/2005282617 >> v1:172.21.2.117:6789/0 conn(0x7f4b280e4da0 legacy=0x7f4b281b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 -- 172.21.2.117:0/2005282617 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b281b08e0 con 0x7f4b280e51a0 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b26e1d700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280e51a0 0x7f4b280d0520 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f4b20006f00 tx=0x7f4b20009f10).ready entity=mon.0 client_cookie=8b5c2a3e72448c0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2561a700 1 -- 172.21.2.117:0/2005282617 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2000b3b0 con 0x7f4b280e51a0 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2561a700 1 -- 172.21.2.117:0/2005282617 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b20000d30 con 0x7f4b280e51a0 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2561a700 1 -- 172.21.2.117:0/2005282617 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2000b8e0 con 0x7f4b280e51a0 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2c857700 1 -- 172.21.2.117:0/2005282617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280e51a0 msgr2=0x7f4b280d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.272 7f4b2c857700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280e51a0 0x7f4b280d0520 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f4b20006f00 tx=0x7f4b20009f10).stop 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 -- 172.21.2.117:0/2005282617 shutdown_connections 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280e51a0 0x7f4b280d0520 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 --2- 172.21.2.117:0/2005282617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 -- 172.21.2.117:0/2005282617 >> 172.21.2.117:0/2005282617 conn(0x7f4b280ce450 msgr2=0x7f4b280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 -- 172.21.2.117:0/2005282617 shutdown_connections 2021-02-10T11:31:00.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.273 7f4b2c857700 1 -- 172.21.2.117:0/2005282617 wait complete. 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 Processor -- start 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 -- start start 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281b5130 0x7f4b281b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b281b5b50 0x7f4b280b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280b8da0 con 0x7f4b281b5f50 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280a4140 con 0x7f4b281b5b50 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b2c857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280a4320 con 0x7f4b281b5130 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b26e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b26e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44406/0 (socket says 172.21.2.117:44406) 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.288 7f4b26e1d700 1 -- 172.21.2.117:0/332385572 learned_addr learned my addr 172.21.2.117:0/332385572 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 -- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281b5130 msgr2=0x7f4b281b5530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281b5130 0x7f4b281b5530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 -- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b281b5b50 msgr2=0x7f4b280b5670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b281b5b50 0x7f4b280b5670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 -- 172.21.2.117:0/332385572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280b62b0 con 0x7f4b281b5f50 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b26e1d700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f4b20002960 tx=0x7f4b20002d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b20000d30 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b20002e40 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b2c857700 1 -- 172.21.2.117:0/332385572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b280b62b0 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b20005a10 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b2c857700 1 -- 172.21.2.117:0/332385572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b280b62b0 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.289 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b2000a240 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.290 7f4b1effd700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 0x7f4b0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.290 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b20032b10 con 0x7f4b281b5f50 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.290 7f4b2661c700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 0x7f4b0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.290 7f4b2661c700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 0x7f4b0c01ac10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.321 7f4b2c857700 1 -- 172.21.2.117:0/332385572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b281b4f00 con 0x7f4b281b5f50 2021-02-10T11:31:00.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.324 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b2000a030 con 0x7f4b281b5f50 2021-02-10T11:31:00.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.448 7f4b2c857700 1 -- 172.21.2.117:0/332385572 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802481", "target": ["mgr", ""]}) v1 -- 0x7f4b280b6b30 con 0x7f4b0c018730 2021-02-10T11:31:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.475 7f4b1effd700 1 -- 172.21.2.117:0/332385572 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4b180009b0 con 0x7f4b0c018730 2021-02-10T11:31:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 -- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 msgr2=0x7f4b0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 0x7f4b0c01ac10 crc :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 -- 172.21.2.117:0/332385572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 msgr2=0x7f4b280b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f4b20002960 tx=0x7f4b20002d00).stop 2021-02-10T11:31:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 -- 172.21.2.117:0/332385572 shutdown_connections 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b0c018730 0x7f4b0c01ac10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b281b5f50 0x7f4b280b5c90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b281b5b50 0x7f4b280b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 --2- 172.21.2.117:0/332385572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281b5130 0x7f4b281b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.481 7f4b2c857700 1 -- 172.21.2.117:0/332385572 >> 172.21.2.117:0/332385572 conn(0x7f4b2808bbc0 msgr2=0x7f4b280c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.482 7f4b2c857700 1 -- 172.21.2.117:0/332385572 shutdown_connections 2021-02-10T11:31:00.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.482 7f4b2c857700 1 -- 172.21.2.117:0/332385572 wait complete. 2021-02-10T11:31:00.479 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802481 2021-02-10T11:31:00.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.903 7fc0a0d2d700 1 Processor -- start 2021-02-10T11:31:00.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc0a0d2d700 1 -- start start 2021-02-10T11:31:00.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc0a0d2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c07da20 0x7fc09c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc0a0d2d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c0d1d20 con 0x7fc09c0e4da0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc0a0d2d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09c077a70 con 0x7fc09c0e51a0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc0a0d2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0777d0 con 0x7fc09c07da20 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc09a312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc09c0e4da0 0x7fc09c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44666/0 (socket says 172.21.2.117:44666) 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.904 7fc09a312700 1 -- 172.21.2.117:0/1878898389 learned_addr learned my addr 172.21.2.117:0/1878898389 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910218869 0 0) 0x7fc08c0010b0 con 0x7fc09c0e4da0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088002fe0 con 0x7fc09c0e4da0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc09ab13700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c07da20 0x7fc09c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3773163489 0 0) 0x7fc08c001010 con 0x7fc09c0e4da0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c07da20 msgr2=0x7fc09c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c07da20 0x7fc09c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 >> v1:172.21.2.125:6789/0 conn(0x7fc09c0e51a0 legacy=0x7fc09c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c1b08e0 con 0x7fc09c0e4da0 2021-02-10T11:31:00.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fc08c001530 con 0x7fc09c0e4da0 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 >> v1:172.21.2.117:6789/0 conn(0x7fc09c0e4da0 legacy=0x7fc09c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc088003aa0 0x7fc088005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc088006790 0x7fc088008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc088009500 0x7fc08800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800c2d0 con 0x7fc088009500 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800c4f0 con 0x7fc088003aa0 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc099b11700 1 -- 172.21.2.117:0/1878898389 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08800c710 con 0x7fc088006790 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc09b314700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc088009500 0x7fc08800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc09ab13700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc088003aa0 0x7fc088005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.905 7fc09a312700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc088006790 0x7fc088008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 -- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc088006790 msgr2=0x7fc088008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc088006790 0x7fc088008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 -- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc088003aa0 msgr2=0x7fc088005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc088003aa0 0x7fc088005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 -- 172.21.2.117:0/1878898389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08800c9b0 con 0x7fc088009500 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc088009500 0x7fc08800ba20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc09404ae40 tx=0x7fc09404de20).ready entity=mon.0 client_cookie=4df41bb272bdafa8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc094001400 con 0x7fc088009500 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc094001680 con 0x7fc088009500 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc099b11700 1 -- 172.21.2.117:0/1878898389 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc094001ba0 con 0x7fc088009500 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc0a0d2d700 1 -- 172.21.2.117:0/1878898389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc088009500 msgr2=0x7fc08800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc0a0d2d700 1 --2- 172.21.2.117:0/1878898389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc088009500 0x7fc08800ba20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc09404ae40 tx=0x7fc09404de20).stop 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc09b314700 1 -- 172.21.2.117:0/1878898389 reap_dead start 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc0a0d2d700 1 -- 172.21.2.117:0/1878898389 shutdown_connections 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.906 7fc0a0d2d700 1 -- 172.21.2.117:0/1878898389 >> 172.21.2.117:0/1878898389 conn(0x7fc09c0ce450 msgr2=0x7fc09c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:00.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.907 7fc0a0d2d700 1 -- 172.21.2.117:0/1878898389 shutdown_connections 2021-02-10T11:31:00.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.907 7fc0a0d2d700 1 -- 172.21.2.117:0/1878898389 wait complete. 2021-02-10T11:31:00.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.921 7fc0a0d2d700 1 Processor -- start 2021-02-10T11:31:00.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.921 7fc0a0d2d700 1 -- start start 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c0a47d0 0x7fc09c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc09c0b77d0 0x7fc09c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c189970 con 0x7fc09c0a51f0 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c190aa0 con 0x7fc09c0b77d0 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c0b81f0 con 0x7fc09c0a47d0 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44428/0 (socket says 172.21.2.117:44428) 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 -- 172.21.2.117:0/2845969697 learned_addr learned my addr 172.21.2.117:0/2845969697 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 -- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c0a47d0 msgr2=0x7fc09c0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c0a47d0 0x7fc09c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 -- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc09c0b77d0 msgr2=0x7fc09c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc09c0b77d0 0x7fc09c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 -- 172.21.2.117:0/2845969697 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0b8410 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc09a312700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fc08c00a4f0 tx=0x7fc08c00a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc08c00ac80 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c004350 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc08c004870 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.922 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c0b8410 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.923 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c0b8410 con 0x7fc09c0a51f0 2021-02-10T11:31:00.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.923 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc08c000db0 con 0x7fc09c0a51f0 2021-02-10T11:31:00.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.923 7fc0937fe700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 0x7fc08001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:00.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.923 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc08c032720 con 0x7fc09c0a51f0 2021-02-10T11:31:00.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.924 7fc09ab13700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 0x7fc08001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:00.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.924 7fc09ab13700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 0x7fc08001abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:00.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.954 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c077a70 con 0x7fc09c0a51f0 2021-02-10T11:31:00.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:00.956 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc08c004140 con 0x7fc09c0a51f0 2021-02-10T11:31:01.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.086 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802481", "target": ["mgr", ""]}) v1 -- 0x7fc09c077a70 con 0x7fc0800186c0 2021-02-10T11:31:01.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.089 7fc0937fe700 1 -- 172.21.2.117:0/2845969697 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fc084007df0 con 0x7fc0800186c0 2021-02-10T11:31:01.083 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b 2021-02-10T11:31:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.094 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 msgr2=0x7fc08001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.094 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 0x7fc08001abc0 crc :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.094 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 msgr2=0x7fc09c0a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.094 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fc08c00a4f0 tx=0x7fc08c00a890).stop 2021-02-10T11:31:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 shutdown_connections 2021-02-10T11:31:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc0800186c0 0x7fc08001abc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc09c0a51f0 0x7fc09c0a55f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc09c0b77d0 0x7fc09c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 --2- 172.21.2.117:0/2845969697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc09c0a47d0 0x7fc09c0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 >> 172.21.2.117:0/2845969697 conn(0x7fc09c0b5140 msgr2=0x7fc09c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 shutdown_connections 2021-02-10T11:31:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:01.095 7fc0a0d2d700 1 -- 172.21.2.117:0/2845969697 wait complete. 2021-02-10T11:31:01.095 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802481 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b 2021-02-10T11:31:01.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:01.095 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:01.095 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:01.431 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:01.431 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:01.431 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:01.814 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:01.814 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:01.814 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245973 s, 42.6 MB/s 2021-02-10T11:31:01.815 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:01.815 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:01.815 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:02.150 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:02.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:02.150 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251568 s, 41.7 MB/s 2021-02-10T11:31:02.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:02.524 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:02.525 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:02.859 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:02.859 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:02.859 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:03.236 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:03.236 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:03.236 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024931 s, 42.1 MB/s 2021-02-10T11:31:03.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:03.237 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:03.237 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:03.574 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:03.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:03.575 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:03.948 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:03.949 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:03.949 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231426 s, 45.3 MB/s 2021-02-10T11:31:03.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:03.950 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:03.950 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:04.284 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:04.285 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:04.285 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:04.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:04.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:04.660 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225806 s, 46.4 MB/s 2021-02-10T11:31:04.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:04.662 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:04.662 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:04.996 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:04.997 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:04.997 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:05.371 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:05.371 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:05.371 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223719 s, 46.9 MB/s 2021-02-10T11:31:05.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:05.372 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:05.372 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:05.708 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:05.709 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:05.709 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:06.084 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:06.084 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:06.084 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232279 s, 45.1 MB/s 2021-02-10T11:31:06.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:06.085 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:06.085 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:06.421 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:06.421 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:06.421 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:06.798 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:06.798 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:06.799 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257625 s, 40.7 MB/s 2021-02-10T11:31:06.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:06.800 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:06.800 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:07.134 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:07.135 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:07.135 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234319 s, 44.7 MB/s 2021-02-10T11:31:07.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:07.507 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:07.507 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:07.842 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:07.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:07.843 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802481/8beac807-0846-4449-8e18-c3a67302eb9b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:08.217 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:08.217 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:08.217 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237747 s, 44.1 MB/s 2021-02-10T11:31:08.218 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802482 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.642 7f91c526c700 1 Processor -- start 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.642 7f91c526c700 1 -- start start 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91c526c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 0x7f91c00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91c526c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c00d1d20 con 0x7f91c007da20 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91c526c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c0077a70 con 0x7f91c01b00f0 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91c526c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00e5430 con 0x7f91c0079000 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91bf07d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f91c007da20 0x7f91c00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44684/0 (socket says 172.21.2.117:44684) 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91bf07d700 1 -- 172.21.2.117:0/1124321774 learned_addr learned my addr 172.21.2.117:0/1124321774 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91be87c700 1 --2- 172.21.2.117:0/1124321774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 0x7f91c00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91be87c700 1 -- 172.21.2.117:0/1124321774 >> v1:172.21.2.125:6789/0 conn(0x7f91c01b00f0 legacy=0x7f91c00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be87c700 1 -- 172.21.2.117:0/1124321774 >> v1:172.21.2.117:6789/0 conn(0x7f91c007da20 legacy=0x7f91c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be87c700 1 -- 172.21.2.117:0/1124321774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c0079400 con 0x7f91c0079000 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.643 7f91be07b700 1 -- 172.21.2.117:0/1124321774 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576186923 0 0) 0x7f91b8001400 con 0x7f91c01b00f0 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be87c700 1 --2- 172.21.2.117:0/1124321774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 0x7f91c00e4da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f91b4006650 tx=0x7f91b4009a00).ready entity=mon.2 client_cookie=91b43aadd822614c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be07b700 1 -- 172.21.2.117:0/1124321774 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91b4009df0 con 0x7f91c0079000 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be07b700 1 -- 172.21.2.117:0/1124321774 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b40009b0 con 0x7f91c0079000 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91be07b700 1 -- 172.21.2.117:0/1124321774 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91b400c2b0 con 0x7f91c0079000 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91c526c700 1 -- 172.21.2.117:0/1124321774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 msgr2=0x7f91c00e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91c526c700 1 --2- 172.21.2.117:0/1124321774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 0x7f91c00e4da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f91b4006650 tx=0x7f91b4009a00).stop 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91c526c700 1 -- 172.21.2.117:0/1124321774 shutdown_connections 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91c526c700 1 --2- 172.21.2.117:0/1124321774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c0079000 0x7f91c00e4da0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.644 7f91c526c700 1 -- 172.21.2.117:0/1124321774 >> 172.21.2.117:0/1124321774 conn(0x7f91c00ce450 msgr2=0x7f91c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.645 7f91c526c700 1 -- 172.21.2.117:0/1124321774 shutdown_connections 2021-02-10T11:31:08.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.645 7f91c526c700 1 -- 172.21.2.117:0/1124321774 wait complete. 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.659 7f91c526c700 1 Processor -- start 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.659 7f91c526c700 1 -- start start 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c009c780 0x7f91c00b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00b4250 0x7f91c00c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00bff40 con 0x7f91c009cb80 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00bfb60 con 0x7f91c009c780 2021-02-10T11:31:08.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c00bfd40 con 0x7f91c00b4250 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44440/0 (socket says 172.21.2.117:44440) 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 -- 172.21.2.117:0/1585075098 learned_addr learned my addr 172.21.2.117:0/1585075098 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 -- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00b4250 msgr2=0x7f91c00c0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00b4250 0x7f91c00c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 -- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c009c780 msgr2=0x7f91c00b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c009c780 0x7f91c00b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 -- 172.21.2.117:0/1585075098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00c1100 con 0x7f91c009cb80 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91be87c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f91b40096a0 tx=0x7f91b400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91b4000c90 con 0x7f91c009cb80 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b4002030 con 0x7f91c009cb80 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91b4002490 con 0x7f91c009cb80 2021-02-10T11:31:08.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 -- 172.21.2.117:0/1585075098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c00c1100 con 0x7f91c009cb80 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.660 7f91c526c700 1 -- 172.21.2.117:0/1585075098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c00c1690 con 0x7f91c009cb80 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.661 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91b4002210 con 0x7f91c009cb80 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.661 7f91a7fff700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 0x7f91a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.661 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f91b40326c0 con 0x7f91c009cb80 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.661 7f91bf07d700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 0x7f91a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:08.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.662 7f91bf07d700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 0x7f91a001abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:08.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.693 7f91c526c700 1 -- 172.21.2.117:0/1585075098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c00ff170 con 0x7f91c009cb80 2021-02-10T11:31:08.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.695 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91b4018fd0 con 0x7f91c009cb80 2021-02-10T11:31:08.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.822 7f91c526c700 1 -- 172.21.2.117:0/1585075098 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802482", "target": ["mgr", ""]}) v1 -- 0x7f91c00bff40 con 0x7f91a00186c0 2021-02-10T11:31:08.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.848 7f91a7fff700 1 -- 172.21.2.117:0/1585075098 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f91b00077a0 con 0x7f91a00186c0 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 msgr2=0x7f91a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 0x7f91a001abc0 crc :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 msgr2=0x7f91c00c04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f91b40096a0 tx=0x7f91b400b990).stop 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 shutdown_connections 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91a00186c0 0x7f91a001abc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c009cb80 0x7f91c00c04c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:08.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00b4250 0x7f91c00c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 --2- 172.21.2.117:0/1585075098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c009c780 0x7f91c00b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:08.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 >> 172.21.2.117:0/1585075098 conn(0x7f91c009d3b0 msgr2=0x7f91c00b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:08.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 shutdown_connections 2021-02-10T11:31:08.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:08.855 7f91c526c700 1 -- 172.21.2.117:0/1585075098 wait complete. 2021-02-10T11:31:08.854 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802482 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.278 7f07888b4700 1 Processor -- start 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.278 7f07888b4700 1 -- start start 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f07888b4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0784079000 0x7f07840e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f07888b4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07840d1d00 con 0x7f078407da20 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f07888b4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0784077a70 con 0x7f07841b00d0 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f07888b4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07840e53a0 con 0x7f0784079000 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0784079000 0x7f07840e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0784079000 0x7f07840e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44448/0 (socket says 172.21.2.117:44448) 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 -- 172.21.2.117:0/3524644222 learned_addr learned my addr 172.21.2.117:0/3524644222 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 -- 172.21.2.117:0/3524644222 >> v1:172.21.2.125:6789/0 conn(0x7f07841b00d0 legacy=0x7f07840d0500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 -- 172.21.2.117:0/3524644222 >> v1:172.21.2.117:6789/0 conn(0x7f078407da20 legacy=0x7f07840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 -- 172.21.2.117:0/3524644222 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784079400 con 0x7f0784079000 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.279 7f0781e1b700 1 --2- 172.21.2.117:0/3524644222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0784079000 0x7f07840e4d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f0774006700 tx=0x7f0774009b80).ready entity=mon.0 client_cookie=afe8f80810468bf1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.280 7f078161a700 1 -- 172.21.2.117:0/3524644222 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077400c0c0 con 0x7f0784079000 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.280 7f078161a700 1 -- 172.21.2.117:0/3524644222 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077400c440 con 0x7f0784079000 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.280 7f078161a700 1 -- 172.21.2.117:0/3524644222 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077400c960 con 0x7f0784079000 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.280 7f07888b4700 1 -- 172.21.2.117:0/3524644222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0784079000 msgr2=0x7f07840e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.280 7f07888b4700 1 --2- 172.21.2.117:0/3524644222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0784079000 0x7f07840e4d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f0774006700 tx=0x7f0774009b80).stop 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.281 7f07888b4700 1 -- 172.21.2.117:0/3524644222 shutdown_connections 2021-02-10T11:31:09.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.281 7f07888b4700 1 --2- 172.21.2.117:0/3524644222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0784079000 0x7f07840e4d80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.281 7f07888b4700 1 -- 172.21.2.117:0/3524644222 >> 172.21.2.117:0/3524644222 conn(0x7f07840ce430 msgr2=0x7f07840e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:09.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.281 7f07888b4700 1 -- 172.21.2.117:0/3524644222 shutdown_connections 2021-02-10T11:31:09.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.281 7f07888b4700 1 -- 172.21.2.117:0/3524644222 wait complete. 2021-02-10T11:31:09.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 Processor -- start 2021-02-10T11:31:09.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 -- start start 2021-02-10T11:31:09.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078408c9a0 0x7f07840c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078419e290 0x7f07840c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07840c3310 con 0x7f078419de90 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078410c7e0 con 0x7f078419e290 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.295 7f07888b4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07840c3530 con 0x7f078408c9a0 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44456/0 (socket says 172.21.2.117:44456) 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f078261c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078408c9a0 0x7f07840c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 -- 172.21.2.117:0/1022119714 learned_addr learned my addr 172.21.2.117:0/1022119714 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0782e1d700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078419e290 0x7f07840c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 -- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078408c9a0 msgr2=0x7f07840c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078408c9a0 0x7f07840c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 -- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078419e290 msgr2=0x7f07840c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078419e290 0x7f07840c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 -- 172.21.2.117:0/1022119714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784096e10 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f0781e1b700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f077400b580 tx=0x7f077400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077400bd10 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774002030 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f07888b4700 1 -- 172.21.2.117:0/1022119714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0784096e10 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0774002510 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.296 7f07888b4700 1 -- 172.21.2.117:0/1022119714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0784097360 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.297 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0774000c90 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.297 7f077affd700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 0x7f076801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.297 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f07740326f0 con 0x7f078419de90 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.297 7f078261c700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 0x7f076801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:09.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.298 7f078261c700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 0x7f076801abc0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:09.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.329 7f07888b4700 1 -- 172.21.2.117:0/1022119714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07840ce430 con 0x7f078419de90 2021-02-10T11:31:09.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.332 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f077401b920 con 0x7f078419de90 2021-02-10T11:31:09.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.462 7f07888b4700 1 -- 172.21.2.117:0/1022119714 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802482", "target": ["mgr", ""]}) v1 -- 0x7f0784097360 con 0x7f07680186c0 2021-02-10T11:31:09.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.466 7f077affd700 1 -- 172.21.2.117:0/1022119714 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f076c006a90 con 0x7f07680186c0 2021-02-10T11:31:09.455 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82 2021-02-10T11:31:09.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.467 7f07888b4700 1 -- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 msgr2=0x7f076801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.467 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 0x7f076801abc0 crc :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.467 7f07888b4700 1 -- 172.21.2.117:0/1022119714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 msgr2=0x7f07840c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:09.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.467 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f077400b580 tx=0x7f077400b920).stop 2021-02-10T11:31:09.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 -- 172.21.2.117:0/1022119714 shutdown_connections 2021-02-10T11:31:09.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07680186c0 0x7f076801abc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078419de90 0x7f07840c26d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078419e290 0x7f07840c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 --2- 172.21.2.117:0/1022119714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078408c9a0 0x7f07840c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:09.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 -- 172.21.2.117:0/1022119714 >> 172.21.2.117:0/1022119714 conn(0x7f07840adf10 msgr2=0x7f07840bf960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:09.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 -- 172.21.2.117:0/1022119714 shutdown_connections 2021-02-10T11:31:09.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:09.468 7f07888b4700 1 -- 172.21.2.117:0/1022119714 wait complete. 2021-02-10T11:31:09.466 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802482 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82 2021-02-10T11:31:09.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:09.466 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:09.467 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:09.803 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:09.804 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:09.804 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:10.184 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:10.184 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:10.184 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233484 s, 44.9 MB/s 2021-02-10T11:31:10.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:10.185 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:10.185 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:10.521 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:10.522 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:10.522 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:10.896 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:10.896 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:10.896 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234442 s, 44.7 MB/s 2021-02-10T11:31:10.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:10.897 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:10.897 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:11.234 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:11.234 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:11.234 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234416 s, 44.7 MB/s 2021-02-10T11:31:11.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:11.609 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:11.609 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:11.946 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:11.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:11.946 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:12.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:12.323 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:12.323 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248945 s, 42.1 MB/s 2021-02-10T11:31:12.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:12.324 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:12.324 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:12.659 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:12.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:12.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252844 s, 41.5 MB/s 2021-02-10T11:31:13.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:13.035 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:13.035 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:13.369 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:13.370 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:13.370 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:13.747 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:13.747 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:13.747 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237632 s, 44.1 MB/s 2021-02-10T11:31:13.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:13.748 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:13.748 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:14.083 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:14.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:14.084 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:14.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:14.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:14.462 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248885 s, 42.1 MB/s 2021-02-10T11:31:14.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:14.463 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:14.463 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:14.799 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:14.799 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:14.799 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:15.171 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:15.171 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:15.171 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225182 s, 46.6 MB/s 2021-02-10T11:31:15.172 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:15.172 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:15.172 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:15.507 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:15.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:15.508 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:15.882 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:15.883 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:15.883 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229555 s, 45.7 MB/s 2021-02-10T11:31:15.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:15.884 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:15.884 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:16.219 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:16.219 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:16.220 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802482/a50b41d5-4b1d-4309-806e-d8e71924ba82/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:16.598 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:16.598 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:16.598 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256561 s, 40.9 MB/s 2021-02-10T11:31:16.600 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802483 2021-02-10T11:31:17.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.025 7fd248eb2700 1 Processor -- start 2021-02-10T11:31:17.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.025 7fd248eb2700 1 -- start start 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd248eb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd244079000 0x7fd2440e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd248eb2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440d1d20 con 0x7fd24407da20 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd248eb2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd244077a70 con 0x7fd2441b00f0 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd248eb2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2440e53c0 con 0x7fd244079000 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd244079000 0x7fd2440e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd24407da20 0x7fd2440771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44712/0 (socket says 172.21.2.117:44712) 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242b13700 1 -- 172.21.2.117:0/893092770 learned_addr learned my addr 172.21.2.117:0/893092770 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242312700 1 -- 172.21.2.117:0/893092770 >> v1:172.21.2.125:6789/0 conn(0x7fd2441b00f0 legacy=0x7fd2440d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242312700 1 -- 172.21.2.117:0/893092770 >> v1:172.21.2.117:6789/0 conn(0x7fd24407da20 legacy=0x7fd2440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242312700 1 -- 172.21.2.117:0/893092770 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd244079400 con 0x7fd244079000 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.026 7fd242312700 1 --2- 172.21.2.117:0/893092770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd244079000 0x7fd2440e4da0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fd234006650 tx=0x7fd234009a80).ready entity=mon.0 client_cookie=b629b235fe41eda7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.027 7fd241b11700 1 -- 172.21.2.117:0/893092770 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23400c0c0 con 0x7fd244079000 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.027 7fd241b11700 1 -- 172.21.2.117:0/893092770 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23400c340 con 0x7fd244079000 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.027 7fd241b11700 1 -- 172.21.2.117:0/893092770 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23400c820 con 0x7fd244079000 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.027 7fd248eb2700 1 -- 172.21.2.117:0/893092770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd244079000 msgr2=0x7fd2440e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.027 7fd248eb2700 1 --2- 172.21.2.117:0/893092770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd244079000 0x7fd2440e4da0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fd234006650 tx=0x7fd234009a80).stop 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.028 7fd248eb2700 1 -- 172.21.2.117:0/893092770 shutdown_connections 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.028 7fd248eb2700 1 --2- 172.21.2.117:0/893092770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd244079000 0x7fd2440e4da0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.028 7fd248eb2700 1 -- 172.21.2.117:0/893092770 >> 172.21.2.117:0/893092770 conn(0x7fd2440ce450 msgr2=0x7fd2440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.028 7fd248eb2700 1 -- 172.21.2.117:0/893092770 shutdown_connections 2021-02-10T11:31:17.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.028 7fd248eb2700 1 -- 172.21.2.117:0/893092770 wait complete. 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.042 7fd248eb2700 1 Processor -- start 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.042 7fd248eb2700 1 -- start start 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2441c22d0 0x7fd2440b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2441b0770 0x7fd2441b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441b0f50 con 0x7fd2440b6190 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440b6590 con 0x7fd2441c22d0 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441b0b70 con 0x7fd2441b0770 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44468/0 (socket says 172.21.2.117:44468) 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 -- 172.21.2.117:0/2059627384 learned_addr learned my addr 172.21.2.117:0/2059627384 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 -- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2441b0770 msgr2=0x7fd2441b14d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2441b0770 0x7fd2441b14d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 -- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2441c22d0 msgr2=0x7fd2440b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2441c22d0 0x7fd2440b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 -- 172.21.2.117:0/2059627384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2441b1af0 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd242312700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fd234009650 tx=0x7fd23400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2340093a0 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234006100 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2340052e0 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2441b1af0 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.043 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2441b1af0 con 0x7fd2440b6190 2021-02-10T11:31:17.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.044 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd234009300 con 0x7fd2440b6190 2021-02-10T11:31:17.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.044 7fd23b7fe700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 0x7fd22801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.044 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd2340328d0 con 0x7fd2440b6190 2021-02-10T11:31:17.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.045 7fd242b13700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 0x7fd22801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.045 7fd242b13700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 0x7fd22801abc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.076 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2440ce450 con 0x7fd2440b6190 2021-02-10T11:31:17.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.079 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2340009c0 con 0x7fd2440b6190 2021-02-10T11:31:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.203 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802483", "target": ["mgr", ""]}) v1 -- 0x7fd2441b0f50 con 0x7fd228018730 2021-02-10T11:31:17.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.231 7fd23b7fe700 1 -- 172.21.2.117:0/2059627384 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd22c007ca0 con 0x7fd228018730 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 msgr2=0x7fd22801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 0x7fd22801abc0 crc :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 msgr2=0x7fd2441b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fd234009650 tx=0x7fd23400b8e0).stop 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 shutdown_connections 2021-02-10T11:31:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd228018730 0x7fd22801abc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2440b6190 0x7fd2441b0150 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2441b0770 0x7fd2441b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 --2- 172.21.2.117:0/2059627384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2441c22d0 0x7fd2440b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 >> 172.21.2.117:0/2059627384 conn(0x7fd2441116c0 msgr2=0x7fd244108870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 shutdown_connections 2021-02-10T11:31:17.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.236 7fd248eb2700 1 -- 172.21.2.117:0/2059627384 wait complete. 2021-02-10T11:31:17.235 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802483 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.660 7f9f1d94f700 1 Processor -- start 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.660 7f9f1d94f700 1 -- start start 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f1d94f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f1807da20 0x7f9f180771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f1d94f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f180e4da0 0x7f9f181b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f1d94f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f18077a70 con 0x7f9f180e51a0 2021-02-10T11:31:17.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f1d94f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f180e5c20 con 0x7f9f180e4da0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f1d94f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180777d0 con 0x7f9f1807da20 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f180e4da0 0x7f9f181b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f180e4da0 0x7f9f181b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44476/0 (socket says 172.21.2.117:44476) 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 -- 172.21.2.117:0/1224800336 learned_addr learned my addr 172.21.2.117:0/1224800336 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 -- 172.21.2.117:0/1224800336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f1807da20 msgr2=0x7f9f180771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 --2- 172.21.2.117:0/1224800336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f1807da20 0x7f9f180771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 -- 172.21.2.117:0/1224800336 >> v1:172.21.2.125:6789/0 conn(0x7f9f180e51a0 legacy=0x7f9f18077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 -- 172.21.2.117:0/1224800336 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1807de20 con 0x7f9f180e4da0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.661 7f9f16d73700 1 --2- 172.21.2.117:0/1224800336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180e4da0 0x7f9f181b00f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f9f08006700 tx=0x7f9f08009b00).ready entity=mon.0 client_cookie=bce81c94e1bb5108 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f16572700 1 -- 172.21.2.117:0/1224800336 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f0800c0c0 con 0x7f9f180e4da0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f16572700 1 -- 172.21.2.117:0/1224800336 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0800c340 con 0x7f9f180e4da0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f16572700 1 -- 172.21.2.117:0/1224800336 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f0800c820 con 0x7f9f180e4da0 2021-02-10T11:31:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 -- 172.21.2.117:0/1224800336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180e4da0 msgr2=0x7f9f181b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 --2- 172.21.2.117:0/1224800336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180e4da0 0x7f9f181b00f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f9f08006700 tx=0x7f9f08009b00).stop 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 -- 172.21.2.117:0/1224800336 shutdown_connections 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 --2- 172.21.2.117:0/1224800336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180e4da0 0x7f9f181b00f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 --2- 172.21.2.117:0/1224800336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f1807da20 0x7f9f180771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 -- 172.21.2.117:0/1224800336 >> 172.21.2.117:0/1224800336 conn(0x7f9f180ce450 msgr2=0x7f9f180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 -- 172.21.2.117:0/1224800336 shutdown_connections 2021-02-10T11:31:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.662 7f9f1d94f700 1 -- 172.21.2.117:0/1224800336 wait complete. 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 Processor -- start 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 -- start start 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 0x7f9f180c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f18104840 0x7f9f18104c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b8a00 con 0x7f9f180c2220 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b8f80 con 0x7f9f180b84b0 2021-02-10T11:31:17.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f1d94f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b9160 con 0x7f9f18104840 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f16d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f16d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44482/0 (socket says 172.21.2.117:44482) 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f16d73700 1 -- 172.21.2.117:0/2044000136 learned_addr learned my addr 172.21.2.117:0/2044000136 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f17574700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 0x7f9f180c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.677 7f9f17d75700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f18104840 0x7f9f18104c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 -- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f18104840 msgr2=0x7f9f18104c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f18104840 0x7f9f18104c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 -- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 msgr2=0x7f9f180c1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 0x7f9f180c1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 -- 172.21.2.117:0/2044000136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180b9380 con 0x7f9f180c2220 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f17574700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 0x7f9f180c1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f16d73700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f9f08006f20 tx=0x7f9f0800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f0800bd80 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f08009100 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f180b9380 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f180b9380 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.678 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f080054d0 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.679 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f0800a000 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.679 7f9f0ffff700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 0x7f9efc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.679 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9f080328d0 con 0x7f9f180c2220 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.679 7f9f17574700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 0x7f9efc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.679 7f9f17574700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 0x7f9efc01ac10 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:17.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.711 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180b8f20 con 0x7f9f180c2220 2021-02-10T11:31:17.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.714 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f08009100 con 0x7f9f180c2220 2021-02-10T11:31:17.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.841 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802483", "target": ["mgr", ""]}) v1 -- 0x7f9f180b8f20 con 0x7f9efc018730 2021-02-10T11:31:17.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.845 7f9f0ffff700 1 -- 172.21.2.117:0/2044000136 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9f00000940 con 0x7f9efc018730 2021-02-10T11:31:17.838 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a 2021-02-10T11:31:17.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 msgr2=0x7f9efc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 0x7f9efc01ac10 crc :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 msgr2=0x7f9f18104220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f9f08006f20 tx=0x7f9f0800b990).stop 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 shutdown_connections 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9efc018730 0x7f9efc01ac10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f180c2220 0x7f9f18104220 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f18104840 0x7f9f18104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 --2- 172.21.2.117:0/2044000136 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f180b84b0 0x7f9f180c1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:17.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 >> 172.21.2.117:0/2044000136 conn(0x7f9f1808a180 msgr2=0x7f9f1808ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:17.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 shutdown_connections 2021-02-10T11:31:17.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:17.851 7f9f1d94f700 1 -- 172.21.2.117:0/2044000136 wait complete. 2021-02-10T11:31:17.849 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802483 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a 2021-02-10T11:31:17.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:17.849 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:17.849 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:18.184 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:18.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:18.185 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:18.564 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:18.564 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:18.564 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230411 s, 45.5 MB/s 2021-02-10T11:31:18.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:18.566 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:18.566 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:18.901 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:18.902 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:18.902 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:19.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:19.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:19.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245859 s, 42.6 MB/s 2021-02-10T11:31:19.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:19.278 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:19.278 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:19.614 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:19.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:19.615 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:19.992 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:19.992 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:19.992 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250568 s, 41.8 MB/s 2021-02-10T11:31:19.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:19.993 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:19.993 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:20.330 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:20.330 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:20.330 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:20.703 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:20.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:20.704 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227494 s, 46.1 MB/s 2021-02-10T11:31:20.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:20.705 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:20.705 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:21.041 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:21.041 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:21.042 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:21.415 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:21.415 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:21.415 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221878 s, 47.3 MB/s 2021-02-10T11:31:21.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:21.417 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:21.417 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:21.752 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:21.753 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:21.753 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232056 s, 45.2 MB/s 2021-02-10T11:31:22.128 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:22.128 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:22.128 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:22.463 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:22.463 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:22.463 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:22.833 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:22.833 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:22.833 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227389 s, 46.1 MB/s 2021-02-10T11:31:22.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:22.835 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:22.835 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:23.172 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:23.173 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:23.173 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:23.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:23.555 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:23.555 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259825 s, 40.4 MB/s 2021-02-10T11:31:23.556 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:23.556 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:23.556 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:23.895 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:23.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:23.896 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:24.275 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250507 s, 41.9 MB/s 2021-02-10T11:31:24.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:24.276 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:24.276 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:24.611 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:24.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:24.612 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802483/de4ba7ee-f0e1-4b1d-bd68-2355cb43396a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:24.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:24.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:24.989 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244916 s, 42.8 MB/s 2021-02-10T11:31:24.990 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802484 2021-02-10T11:31:25.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 Processor -- start 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 -- start start 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f69f407da20 0x7f69f40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69f40e51a0 0x7f69f4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f4077a70 con 0x7f69f40e4da0 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40e5c20 con 0x7f69f40e51a0 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.416 7f69fb3a0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40777d0 con 0x7f69f407da20 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69f40e51a0 0x7f69f4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69f40e51a0 0x7f69f4079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44488/0 (socket says 172.21.2.117:44488) 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 -- 172.21.2.117:0/1465852852 learned_addr learned my addr 172.21.2.117:0/1465852852 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f89bb700 1 --1- 172.21.2.117:0/1465852852 >> v1:172.21.2.117:6789/0 conn(0x7f69f40e4da0 0x7f69f41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44738/0 (socket says 172.21.2.117:44738) 2021-02-10T11:31:25.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f91bc700 1 --2- 172.21.2.117:0/1465852852 >> v2:172.21.2.125:3300/0 conn(0x7f69f407da20 0x7f69f40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 -- 172.21.2.117:0/1465852852 >> v2:172.21.2.125:3300/0 conn(0x7f69f407da20 msgr2=0x7f69f40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 --2- 172.21.2.117:0/1465852852 >> v2:172.21.2.125:3300/0 conn(0x7f69f407da20 0x7f69f40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 -- 172.21.2.117:0/1465852852 >> v1:172.21.2.117:6789/0 conn(0x7f69f40e4da0 legacy=0x7f69f41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 -- 172.21.2.117:0/1465852852 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f41b08e0 con 0x7f69f40e51a0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69f99bd700 1 --2- 172.21.2.117:0/1465852852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f40e51a0 0x7f69f4079000 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f69ec006f00 tx=0x7f69ec009f10).ready entity=mon.0 client_cookie=7e4de1eb5d209a60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69e7fff700 1 -- 172.21.2.117:0/1465852852 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00b3b0 con 0x7f69f40e51a0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69e7fff700 1 -- 172.21.2.117:0/1465852852 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec00b630 con 0x7f69f40e51a0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.417 7f69e7fff700 1 -- 172.21.2.117:0/1465852852 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00bb50 con 0x7f69f40e51a0 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 -- 172.21.2.117:0/1465852852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f40e51a0 msgr2=0x7f69f4079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 --2- 172.21.2.117:0/1465852852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f40e51a0 0x7f69f4079000 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f69ec006f00 tx=0x7f69ec009f10).stop 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 -- 172.21.2.117:0/1465852852 shutdown_connections 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 --2- 172.21.2.117:0/1465852852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f40e51a0 0x7f69f4079000 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 --2- 172.21.2.117:0/1465852852 >> v2:172.21.2.125:3300/0 conn(0x7f69f407da20 0x7f69f40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.418 7f69fb3a0700 1 -- 172.21.2.117:0/1465852852 >> 172.21.2.117:0/1465852852 conn(0x7f69f40ce450 msgr2=0x7f69f40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.419 7f69fb3a0700 1 -- 172.21.2.117:0/1465852852 shutdown_connections 2021-02-10T11:31:25.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.419 7f69fb3a0700 1 -- 172.21.2.117:0/1465852852 wait complete. 2021-02-10T11:31:25.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.433 7f69fb3a0700 1 Processor -- start 2021-02-10T11:31:25.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.433 7f69fb3a0700 1 -- start start 2021-02-10T11:31:25.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 0x7f69f408a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f408aba0 0x7f69f41b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 0x7f69f41b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41b9cb0 con 0x7f69f408aba0 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40c4050 con 0x7f69f408afa0 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69fb3a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41b9730 con 0x7f69f408a180 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f89bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f408aba0 0x7f69f41b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f91bc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 0x7f69f408a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f91bc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 0x7f69f408a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:42172/0 (socket says 172.21.2.117:42172) 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f91bc700 1 -- 172.21.2.117:0/2131026248 learned_addr learned my addr 172.21.2.117:0/2131026248 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 0x7f69f41b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 -- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 msgr2=0x7f69f408a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 0x7f69f408a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 -- 172.21.2.117:0/2131026248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f408aba0 msgr2=0x7f69f41b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f408aba0 0x7f69f41b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.434 7f69f99bd700 1 -- 172.21.2.117:0/2131026248 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4106e90 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69f99bd700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 0x7f69f41b8a40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f69ec006f30 tx=0x7f69ec00bca0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00bde0 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec000f90 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f4106e90 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f41b9350 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.435 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69ec00c570 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.436 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69ec00a000 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.436 7f69e5ffb700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 0x7f69d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.436 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f69ec032950 con 0x7f69f408afa0 2021-02-10T11:31:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.436 7f69f91bc700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 0x7f69d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.436 7f69f91bc700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 0x7f69d401abc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:25.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.467 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f41b9c50 con 0x7f69f408afa0 2021-02-10T11:31:25.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.470 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69ec009450 con 0x7f69f408afa0 2021-02-10T11:31:25.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.596 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802484", "target": ["mgr", ""]}) v1 -- 0x7f69f41b9350 con 0x7f69d40186c0 2021-02-10T11:31:25.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.623 7f69e5ffb700 1 -- 172.21.2.117:0/2131026248 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f69e8006b20 con 0x7f69d40186c0 2021-02-10T11:31:25.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 msgr2=0x7f69d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 0x7f69d401abc0 crc :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 msgr2=0x7f69f41b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 0x7f69f41b8a40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f69ec006f30 tx=0x7f69ec00bca0).stop 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 shutdown_connections 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69d40186c0 0x7f69d401abc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69f408aba0 0x7f69f41b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69f408afa0 0x7f69f41b8a40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 --2- 172.21.2.117:0/2131026248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69f408a180 0x7f69f408a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:25.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 >> 172.21.2.117:0/2131026248 conn(0x7f69f40a4180 msgr2=0x7f69f409be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:25.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 shutdown_connections 2021-02-10T11:31:25.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:25.630 7f69fb3a0700 1 -- 172.21.2.117:0/2131026248 wait complete. 2021-02-10T11:31:25.628 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802484 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.054 7f1a5fbbd700 1 Processor -- start 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.054 7f1a5fbbd700 1 -- start start 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5fbbd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1a5807da20 0x7f1a580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5fbbd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a580d1d20 con 0x7f1a580e51a0 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5fbbd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58077a70 con 0x7f1a580e4da0 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5fbbd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a580777d0 con 0x7f1a5807da20 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5e1da700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1a580e51a0 0x7f1a580d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44750/0 (socket says 172.21.2.117:44750) 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5e1da700 1 -- 172.21.2.117:0/4263047750 learned_addr learned my addr 172.21.2.117:0/4263047750 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:26.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5d9d9700 1 --2- 172.21.2.117:0/4263047750 >> v2:172.21.2.125:3300/0 conn(0x7f1a5807da20 0x7f1a580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646449535 0 0) 0x7f1a50001500 con 0x7f1a580e51a0 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a44002fe0 con 0x7f1a580e51a0 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5d9d9700 1 -- 172.21.2.117:0/4263047750 >> v1:172.21.2.125:6789/0 conn(0x7f1a580e4da0 legacy=0x7f1a581b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2952829221 0 0) 0x7f1a50001460 con 0x7f1a580e51a0 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5d9d9700 1 -- 172.21.2.117:0/4263047750 >> v1:172.21.2.117:6789/0 conn(0x7f1a580e51a0 legacy=0x7f1a580d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.055 7f1a5d9d9700 1 -- 172.21.2.117:0/4263047750 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a581b08e0 con 0x7f1a5807da20 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5d9d9700 1 --2- 172.21.2.117:0/4263047750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5807da20 0x7f1a580771b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f1a4c006650 tx=0x7f1a4c009a80).ready entity=mon.1 client_cookie=3b01043d93777219 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4c00c0c0 con 0x7f1a5807da20 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a4c00c0c0 con 0x7f1a5807da20 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5c9d7700 1 -- 172.21.2.117:0/4263047750 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a4c00c340 con 0x7f1a5807da20 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5fbbd700 1 -- 172.21.2.117:0/4263047750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5807da20 msgr2=0x7f1a580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:26.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.056 7f1a5fbbd700 1 --2- 172.21.2.117:0/4263047750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5807da20 0x7f1a580771b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f1a4c006650 tx=0x7f1a4c009a80).stop 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.057 7f1a5fbbd700 1 -- 172.21.2.117:0/4263047750 shutdown_connections 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.057 7f1a5fbbd700 1 --2- 172.21.2.117:0/4263047750 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5807da20 0x7f1a580771b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.057 7f1a5fbbd700 1 -- 172.21.2.117:0/4263047750 >> 172.21.2.117:0/4263047750 conn(0x7f1a580ce450 msgr2=0x7f1a580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.057 7f1a5fbbd700 1 -- 172.21.2.117:0/4263047750 shutdown_connections 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.057 7f1a5fbbd700 1 -- 172.21.2.117:0/4263047750 wait complete. 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.071 7f1a5fbbd700 1 Processor -- start 2021-02-10T11:31:26.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.071 7f1a5fbbd700 1 -- start start 2021-02-10T11:31:26.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a580a46c0 0x7f1a58189c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:26.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5818a2a0 0x7f1a581a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:26.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a58190970 con 0x7f1a5818a6a0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580a4ac0 con 0x7f1a5818a2a0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5fbbd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580a4ce0 con 0x7f1a580a46c0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44504/0 (socket says 172.21.2.117:44504) 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 -- 172.21.2.117:0/3541501553 learned_addr learned my addr 172.21.2.117:0/3541501553 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 -- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a580a46c0 msgr2=0x7f1a58189c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a580a46c0 0x7f1a58189c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 -- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5818a2a0 msgr2=0x7f1a581a2c10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5818a2a0 0x7f1a581a2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 -- 172.21.2.117:0/3541501553 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a581a3850 con 0x7f1a5818a6a0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a5e1da700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f1a50009980 tx=0x7f1a5000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:26.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.072 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a50006030 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50006270 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a50004720 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a581a3850 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a581a3850 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a50006450 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a4a7fc700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 0x7f1a4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.073 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1a500328e0 con 0x7f1a5818a6a0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.074 7f1a5d9d9700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 0x7f1a4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:26.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.074 7f1a5d9d9700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 0x7f1a4001abc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:26.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.104 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58103340 con 0x7f1a5818a6a0 2021-02-10T11:31:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.107 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a50024f50 con 0x7f1a5818a6a0 2021-02-10T11:31:26.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.235 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802484", "target": ["mgr", ""]}) v1 -- 0x7f1a58103340 con 0x7f1a400186c0 2021-02-10T11:31:26.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.238 7f1a4a7fc700 1 -- 172.21.2.117:0/3541501553 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1a4c0093a0 con 0x7f1a400186c0 2021-02-10T11:31:26.232 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78 2021-02-10T11:31:26.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 msgr2=0x7f1a4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 0x7f1a4001abc0 crc :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 msgr2=0x7f1a581a3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f1a50009980 tx=0x7f1a5000bcd0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 shutdown_connections 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a400186c0 0x7f1a4001abc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a5818a6a0 0x7f1a581a3230 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a5818a2a0 0x7f1a581a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 --2- 172.21.2.117:0/3541501553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a580a46c0 0x7f1a58189c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 >> 172.21.2.117:0/3541501553 conn(0x7f1a580a90a0 msgr2=0x7f1a580a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 shutdown_connections 2021-02-10T11:31:26.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:26.244 7f1a5fbbd700 1 -- 172.21.2.117:0/3541501553 wait complete. 2021-02-10T11:31:26.241 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802484 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78 2021-02-10T11:31:26.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:26.242 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:26.242 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:26.578 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:26.578 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:26.578 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:26.960 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:26.960 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:26.960 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024619 s, 42.6 MB/s 2021-02-10T11:31:26.961 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:26.961 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:26.961 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:27.297 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:27.298 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:27.298 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:27.670 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:27.671 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:27.671 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244969 s, 42.8 MB/s 2021-02-10T11:31:27.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:27.672 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:27.672 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:28.007 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:28.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:28.008 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228582 s, 45.9 MB/s 2021-02-10T11:31:28.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:28.384 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:28.384 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:28.720 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:28.721 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:28.721 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:28.842 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:28.845 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:29.095 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:29.096 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:29.096 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245068 s, 42.8 MB/s 2021-02-10T11:31:29.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:29.097 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:29.097 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:29.433 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:29.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:29.434 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:29.808 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:29.809 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:29.809 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02337 s, 44.9 MB/s 2021-02-10T11:31:29.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:29.810 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:29.810 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:30.144 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:30.144 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:30.144 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:30.518 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:30.518 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:30.518 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232763 s, 45.0 MB/s 2021-02-10T11:31:30.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:30.519 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:30.519 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:30.854 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:30.854 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:30.855 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239205 s, 43.8 MB/s 2021-02-10T11:31:31.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:31.232 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:31.232 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:31.566 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:31.566 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:31.567 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:31.943 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:31.943 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:31.943 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252468 s, 41.5 MB/s 2021-02-10T11:31:31.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:31.945 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:31.945 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:32.279 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:32.279 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:32.280 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:32.652 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:32.652 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:32.652 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247405 s, 42.4 MB/s 2021-02-10T11:31:32.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:32.653 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:32.653 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:32.989 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:32.989 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:32.990 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802484/26e637fa-5ee1-4198-8e13-09d93f1d2d78/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:33.365 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:33.366 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:33.366 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236276 s, 44.4 MB/s 2021-02-10T11:31:33.367 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802485 2021-02-10T11:31:33.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.790 7f6ce01b2700 1 Processor -- start 2021-02-10T11:31:33.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.790 7f6ce01b2700 1 -- start start 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6ce01b2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd807da20 0x7f6cd80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6ce01b2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6cd80e5180 0x7f6cd80d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6ce01b2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8077a70 con 0x7f6cd80e4d80 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6ce01b2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd80e5c00 con 0x7f6cd80e5180 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6ce01b2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80777d0 con 0x7f6cd807da20 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6cd80e5180 0x7f6cd80d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6cd80e5180 0x7f6cd80d0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44516/0 (socket says 172.21.2.117:44516) 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 -- 172.21.2.117:0/1220541987 learned_addr learned my addr 172.21.2.117:0/1220541987 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:33.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cddfce700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd807da20 0x7f6cd80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 -- 172.21.2.117:0/1220541987 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd807da20 msgr2=0x7f6cd80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd807da20 0x7f6cd80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 -- 172.21.2.117:0/1220541987 >> v1:172.21.2.117:6789/0 conn(0x7f6cd80e4d80 legacy=0x7f6cd81b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.791 7f6cde7cf700 1 -- 172.21.2.117:0/1220541987 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd81b08c0 con 0x7f6cd80e5180 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6cde7cf700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd80e5180 0x7f6cd80d0500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f6cd0006f00 tx=0x7f6cd0009f10).ready entity=mon.0 client_cookie=a86634c1fa8d4001 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6cdcfcc700 1 -- 172.21.2.117:0/1220541987 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd000b3b0 con 0x7f6cd80e5180 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6cdcfcc700 1 -- 172.21.2.117:0/1220541987 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000b630 con 0x7f6cd80e5180 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6cdcfcc700 1 -- 172.21.2.117:0/1220541987 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd000bb50 con 0x7f6cd80e5180 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 -- 172.21.2.117:0/1220541987 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd80e5180 msgr2=0x7f6cd80d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd80e5180 0x7f6cd80d0500 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f6cd0006f00 tx=0x7f6cd0009f10).stop 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 -- 172.21.2.117:0/1220541987 shutdown_connections 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd80e5180 0x7f6cd80d0500 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 --2- 172.21.2.117:0/1220541987 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd807da20 0x7f6cd80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.792 7f6ce01b2700 1 -- 172.21.2.117:0/1220541987 >> 172.21.2.117:0/1220541987 conn(0x7f6cd80ce430 msgr2=0x7f6cd80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.793 7f6ce01b2700 1 -- 172.21.2.117:0/1220541987 shutdown_connections 2021-02-10T11:31:33.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.793 7f6ce01b2700 1 -- 172.21.2.117:0/1220541987 wait complete. 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.807 7f6ce01b2700 1 Processor -- start 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.807 7f6ce01b2700 1 -- start start 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cd8187bd0 0x7f6cd8199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd8199750 0x7f6cd80ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8091ea0 con 0x7f6cd8091a50 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8187fd0 con 0x7f6cd8187bd0 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6ce01b2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd81881f0 con 0x7f6cd8199750 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44520/0 (socket says 172.21.2.117:44520) 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 -- 172.21.2.117:0/1731020264 learned_addr learned my addr 172.21.2.117:0/1731020264 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:33.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 -- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd8199750 msgr2=0x7f6cd80ab8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd8199750 0x7f6cd80ab8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 -- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cd8187bd0 msgr2=0x7f6cd8199130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cd8187bd0 0x7f6cd8199130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 -- 172.21.2.117:0/1731020264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80abee0 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.808 7f6cddfce700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f6cd4006f00 tx=0x7f6cd400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd400bd30 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4002030 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cd4002550 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd80abee0 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd80abee0 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cd4000c90 con 0x7f6cd8091a50 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 0x7f6cc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:33.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.809 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6cd40326e0 con 0x7f6cd8091a50 2021-02-10T11:31:33.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.810 7f6cdd7cd700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 0x7f6cc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:33.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.811 7f6cdd7cd700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 0x7f6cc001abc0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:33.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.841 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8077a70 con 0x7f6cd8091a50 2021-02-10T11:31:33.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.843 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cd400bd30 con 0x7f6cd8091a50 2021-02-10T11:31:33.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.969 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802485", "target": ["mgr", ""]}) v1 -- 0x7f6cd8077a70 con 0x7f6cc00186c0 2021-02-10T11:31:33.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:33.997 7f6cca7fc700 1 -- 172.21.2.117:0/1731020264 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6ccc006db0 con 0x7f6cc00186c0 2021-02-10T11:31:33.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 msgr2=0x7f6cc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 0x7f6cc001abc0 crc :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 msgr2=0x7f6cd80b8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:33.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f6cd4006f00 tx=0x7f6cd400b940).stop 2021-02-10T11:31:33.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 shutdown_connections 2021-02-10T11:31:33.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cd8199750 0x7f6cd80ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cd8187bd0 0x7f6cd8199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:33.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6cc00186c0 0x7f6cc001abc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 --2- 172.21.2.117:0/1731020264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cd8091a50 0x7f6cd80b8cf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:33.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 >> 172.21.2.117:0/1731020264 conn(0x7f6cd80b5320 msgr2=0x7f6cd81b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:33.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 shutdown_connections 2021-02-10T11:31:33.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.003 7f6ce01b2700 1 -- 172.21.2.117:0/1731020264 wait complete. 2021-02-10T11:31:34.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802485 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 Processor -- start 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 -- start start 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdd000d1510 0x7fdd000d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd00077a70 con 0x7fdd0007da20 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd00079400 con 0x7fdd00079000 2021-02-10T11:31:34.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdd05571700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0007de20 con 0x7fdd000d1510 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdcff2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdd0007da20 0x7fdd000771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44776/0 (socket says 172.21.2.117:44776) 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.426 7fdcff2bf700 1 -- 172.21.2.117:0/3543241997 learned_addr learned my addr 172.21.2.117:0/3543241997 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcffac0700 1 --2- 172.21.2.117:0/3543241997 >> v2:172.21.2.125:3300/0 conn(0x7fdd000d1510 0x7fdd000d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534384110 0 0) 0x7fdcf0001180 con 0x7fdd0007da20 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcec002fe0 con 0x7fdd0007da20 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564728757 0 0) 0x7fdce8001400 con 0x7fdd00079000 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcec004370 con 0x7fdd00079000 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcffac0700 1 -- 172.21.2.117:0/3543241997 >> v1:172.21.2.125:6789/0 conn(0x7fdd00079000 legacy=0x7fdd000e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcffac0700 1 -- 172.21.2.117:0/3543241997 >> v1:172.21.2.117:6789/0 conn(0x7fdd0007da20 legacy=0x7fdd000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcffac0700 1 -- 172.21.2.117:0/3543241997 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd001b00d0 con 0x7fdd000d1510 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcffac0700 1 --2- 172.21.2.117:0/3543241997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd000d1510 0x7fdd000d1910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fdcf8006ad0 tx=0x7fdcf8009a50).ready entity=mon.1 client_cookie=12da9144cb84a1f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.427 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf80066a0 con 0x7fdd000d1510 2021-02-10T11:31:34.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8006410 con 0x7fdd000d1510 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdcfe2bd700 1 -- 172.21.2.117:0/3543241997 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf80024d0 con 0x7fdd000d1510 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 -- 172.21.2.117:0/3543241997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd000d1510 msgr2=0x7fdd000d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 --2- 172.21.2.117:0/3543241997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd000d1510 0x7fdd000d1910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fdcf8006ad0 tx=0x7fdcf8009a50).stop 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 -- 172.21.2.117:0/3543241997 shutdown_connections 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 --2- 172.21.2.117:0/3543241997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd000d1510 0x7fdd000d1910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 -- 172.21.2.117:0/3543241997 >> 172.21.2.117:0/3543241997 conn(0x7fdd000ce430 msgr2=0x7fdd000e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 -- 172.21.2.117:0/3543241997 shutdown_connections 2021-02-10T11:31:34.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.428 7fdd05571700 1 -- 172.21.2.117:0/3543241997 wait complete. 2021-02-10T11:31:34.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 Processor -- start 2021-02-10T11:31:34.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 -- start start 2021-02-10T11:31:34.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd001071e0 0x7fdd001076b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 0x7fdd000aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000aaa60 con 0x7fdd000aaf60 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000ab360 con 0x7fdd001071e0 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.443 7fdd05571700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000ab580 con 0x7fdd00107eb0 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44534/0 (socket says 172.21.2.117:44534) 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 -- 172.21.2.117:0/3210696193 learned_addr learned my addr 172.21.2.117:0/3210696193 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcfeabe700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 0x7fdd000aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcff2bf700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd001071e0 0x7fdd001076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 -- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 msgr2=0x7fdd000aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 0x7fdd000aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 -- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd001071e0 msgr2=0x7fdd001076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd001071e0 0x7fdd001076b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 -- 172.21.2.117:0/3210696193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd000a4ce0 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcfeabe700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 0x7fdd000aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.444 7fdcffac0700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdcf8009660 tx=0x7fdcf800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf8002030 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf80022b0 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcf80027d0 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdd05571700 1 -- 172.21.2.117:0/3210696193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd000a4ce0 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdd05571700 1 -- 172.21.2.117:0/3210696193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd000a5270 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdcf8002490 con 0x7fdd000aaf60 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 0x7fdce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:34.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.445 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdcf8032660 con 0x7fdd000aaf60 2021-02-10T11:31:34.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.446 7fdcff2bf700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 0x7fdce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:34.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.446 7fdcff2bf700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 0x7fdce401abc0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:34.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.476 7fdd05571700 1 -- 172.21.2.117:0/3210696193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd00077a70 con 0x7fdd000aaf60 2021-02-10T11:31:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.479 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdcf8000d20 con 0x7fdd000aaf60 2021-02-10T11:31:34.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.604 7fdd05571700 1 -- 172.21.2.117:0/3210696193 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802485", "target": ["mgr", ""]}) v1 -- 0x7fdd00077a70 con 0x7fdce40186c0 2021-02-10T11:31:34.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.609 7fdcf7fff700 1 -- 172.21.2.117:0/3210696193 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fdcf0007770 con 0x7fdce40186c0 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 -- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 msgr2=0x7fdce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 0x7fdce401abc0 crc :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 -- 172.21.2.117:0/3210696193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 msgr2=0x7fdd000a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdcf8009660 tx=0x7fdcf800b990).stop 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 -- 172.21.2.117:0/3210696193 shutdown_connections 2021-02-10T11:31:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdce40186c0 0x7fdce401abc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd000aaf60 0x7fdd000a46c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd00107eb0 0x7fdd000aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 --2- 172.21.2.117:0/3210696193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd001071e0 0x7fdd001076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.614 7fdd05571700 1 -- 172.21.2.117:0/3210696193 >> 172.21.2.117:0/3210696193 conn(0x7fdd000ab8d0 msgr2=0x7fdd0009be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.615 7fdd05571700 1 -- 172.21.2.117:0/3210696193 shutdown_connections 2021-02-10T11:31:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:34.615 7fdd05571700 1 -- 172.21.2.117:0/3210696193 wait complete. 2021-02-10T11:31:34.612 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802485 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6 2021-02-10T11:31:34.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:34.612 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:34.612 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:34.947 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:34.947 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:34.948 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:35.330 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:35.330 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:35.331 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254987 s, 41.1 MB/s 2021-02-10T11:31:35.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:35.332 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:35.332 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:35.667 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:35.667 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:35.668 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:36.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:36.043 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:36.043 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022611 s, 46.4 MB/s 2021-02-10T11:31:36.044 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:36.044 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:36.044 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:36.381 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:36.381 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:36.382 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:36.757 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:36.757 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:36.757 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226371 s, 46.3 MB/s 2021-02-10T11:31:36.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:36.758 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:36.758 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:37.093 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:37.093 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:37.093 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:37.465 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:37.465 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:37.465 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240713 s, 43.6 MB/s 2021-02-10T11:31:37.466 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:37.466 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:37.466 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:37.800 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:37.800 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:37.801 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:38.179 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:38.179 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:38.179 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0264766 s, 39.6 MB/s 2021-02-10T11:31:38.180 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:38.180 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:38.180 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:38.516 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:38.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:38.517 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:38.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:38.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:38.891 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233704 s, 44.9 MB/s 2021-02-10T11:31:38.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:38.892 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:38.892 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:39.228 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:39.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:39.229 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239939 s, 43.7 MB/s 2021-02-10T11:31:39.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:39.610 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:39.610 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:39.946 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:39.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:39.947 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:40.324 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:40.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:40.325 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256067 s, 40.9 MB/s 2021-02-10T11:31:40.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:40.326 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:40.326 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:40.661 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:40.662 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:40.662 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:41.037 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:41.038 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:41.038 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236453 s, 44.3 MB/s 2021-02-10T11:31:41.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:41.039 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:41.039 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:41.373 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:41.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:41.373 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802485/f24d9161-b06e-420b-9af7-efd22eb6a8b6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227455 s, 46.1 MB/s 2021-02-10T11:31:41.749 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802486 2021-02-10T11:31:42.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 Processor -- start 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 -- start start 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080d1530 0x7fc6080d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc608077a70 con 0x7fc608079000 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc608079400 con 0x7fc60807da20 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc610257700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60807de20 con 0x7fc6080d1530 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc60d872700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc608079000 0x7fc6080e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44788/0 (socket says 172.21.2.117:44788) 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.173 7fc60d872700 1 -- 172.21.2.117:0/3643317286 learned_addr learned my addr 172.21.2.117:0/3643317286 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60e874700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080d1530 0x7fc6080d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280126022 0 0) 0x7fc6040010b0 con 0x7fc608079000 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f4002fe0 con 0x7fc608079000 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4103881367 0 0) 0x7fc604000e20 con 0x7fc608079000 2021-02-10T11:31:42.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080d1530 msgr2=0x7fc6080d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080d1530 0x7fc6080d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 >> v1:172.21.2.125:6789/0 conn(0x7fc60807da20 legacy=0x7fc6080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6081b00f0 con 0x7fc608079000 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fc604001700 con 0x7fc608079000 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 >> v1:172.21.2.117:6789/0 conn(0x7fc608079000 legacy=0x7fc6080e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5f4003a60 0x7fc5f4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5f4006790 0x7fc5f4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5f4009590 0x7fc5f400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f400c2d0 con 0x7fc5f4006790 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f400c4f0 con 0x7fc5f4003a60 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d071700 1 -- 172.21.2.117:0/3643317286 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f400c710 con 0x7fc5f4009590 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5f4006790 0x7fc5f4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60e874700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5f4009590 0x7fc5f400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60e073700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5f4003a60 0x7fc5f4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 -- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5f4009590 msgr2=0x7fc5f400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5f4009590 0x7fc5f400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 -- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5f4003a60 msgr2=0x7fc5f4005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5f4003a60 0x7fc5f4005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.174 7fc60d872700 1 -- 172.21.2.117:0/3643317286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f4050980 con 0x7fc5f4006790 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc60d872700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5f4006790 0x7fc5f4008c90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fc604001440 tx=0x7fc604007d40).ready entity=mon.0 client_cookie=a88571b07fc2e892 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc60400a520 con 0x7fc5f4006790 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60400a760 con 0x7fc5f4006790 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc60d071700 1 -- 172.21.2.117:0/3643317286 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc60400abf0 con 0x7fc5f4006790 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc610257700 1 -- 172.21.2.117:0/3643317286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5f4006790 msgr2=0x7fc5f4008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc610257700 1 --2- 172.21.2.117:0/3643317286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5f4006790 0x7fc5f4008c90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fc604001440 tx=0x7fc604007d40).stop 2021-02-10T11:31:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc60e874700 1 -- 172.21.2.117:0/3643317286 reap_dead start 2021-02-10T11:31:42.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc610257700 1 -- 172.21.2.117:0/3643317286 shutdown_connections 2021-02-10T11:31:42.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.175 7fc610257700 1 -- 172.21.2.117:0/3643317286 >> 172.21.2.117:0/3643317286 conn(0x7fc6080ce450 msgr2=0x7fc6080e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.176 7fc610257700 1 -- 172.21.2.117:0/3643317286 shutdown_connections 2021-02-10T11:31:42.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.176 7fc610257700 1 -- 172.21.2.117:0/3643317286 wait complete. 2021-02-10T11:31:42.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.190 7fc610257700 1 Processor -- start 2021-02-10T11:31:42.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.190 7fc610257700 1 -- start start 2021-02-10T11:31:42.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6080a7820 0x7fc608092030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080a7c20 0x7fc608093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc608092a50 con 0x7fc608092650 2021-02-10T11:31:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc608092c80 con 0x7fc6080a7820 2021-02-10T11:31:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc610257700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6080a8340 con 0x7fc6080a7c20 2021-02-10T11:31:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44554/0 (socket says 172.21.2.117:44554) 2021-02-10T11:31:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 -- 172.21.2.117:0/3081865552 learned_addr learned my addr 172.21.2.117:0/3081865552 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 -- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080a7c20 msgr2=0x7fc608093200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080a7c20 0x7fc608093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 -- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6080a7820 msgr2=0x7fc608092030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6080a7820 0x7fc608092030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.191 7fc60e874700 1 -- 172.21.2.117:0/3081865552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080a8520 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc60e874700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fc600009a30 tx=0x7fc60000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc600001400 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000be60 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6000019c0 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc610257700 1 -- 172.21.2.117:0/3081865552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6080a8520 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc610257700 1 -- 172.21.2.117:0/3081865552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc608094130 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6000011f0 con 0x7fc608092650 2021-02-10T11:31:42.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.192 7fc5faffd700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 0x7fc5f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.193 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc600032aa0 con 0x7fc608092650 2021-02-10T11:31:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.193 7fc60e073700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 0x7fc5f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.193 7fc60e073700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 0x7fc5f001abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.223 7fc610257700 1 -- 172.21.2.117:0/3081865552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc608092a50 con 0x7fc608092650 2021-02-10T11:31:42.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.226 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc600006a80 con 0x7fc608092650 2021-02-10T11:31:42.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.351 7fc610257700 1 -- 172.21.2.117:0/3081865552 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802486", "target": ["mgr", ""]}) v1 -- 0x7fc608094130 con 0x7fc5f00186c0 2021-02-10T11:31:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.376 7fc5faffd700 1 -- 172.21.2.117:0/3081865552 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc5fc0069f0 con 0x7fc5f00186c0 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 msgr2=0x7fc5f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 0x7fc5f001abc0 crc :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 msgr2=0x7fc608093820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7fc600009a30 tx=0x7fc60000bd20).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 shutdown_connections 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5f00186c0 0x7fc5f001abc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc608092650 0x7fc608093820 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6080a7c20 0x7fc608093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 --2- 172.21.2.117:0/3081865552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6080a7820 0x7fc608092030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:42.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 >> 172.21.2.117:0/3081865552 conn(0x7fc6080a9210 msgr2=0x7fc6080a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 shutdown_connections 2021-02-10T11:31:42.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.385 7fc610257700 1 -- 172.21.2.117:0/3081865552 wait complete. 2021-02-10T11:31:42.383 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802486 2021-02-10T11:31:42.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 Processor -- start 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 -- start start 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15407da20 0x7ff1540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff1540e4da0 0x7ff1541b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff1540e51a0 0x7ff154077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540e5c20 con 0x7ff1540e51a0 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff154077a70 con 0x7ff1540e4da0 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff15996a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540777d0 con 0x7ff15407da20 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff153d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff1540e51a0 0x7ff154077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff153d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff1540e51a0 0x7ff154077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44558/0 (socket says 172.21.2.117:44558) 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.808 7ff153d75700 1 -- 172.21.2.117:0/4110056054 learned_addr learned my addr 172.21.2.117:0/4110056054 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:42.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff152d73700 1 --2- 172.21.2.117:0/4110056054 >> v2:172.21.2.125:3300/0 conn(0x7ff1540e4da0 0x7ff1541b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153574700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15407da20 0x7ff1540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 -- 172.21.2.117:0/4110056054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15407da20 msgr2=0x7ff1540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15407da20 0x7ff1540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 -- 172.21.2.117:0/4110056054 >> v2:172.21.2.125:3300/0 conn(0x7ff1540e4da0 msgr2=0x7ff1541b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 --2- 172.21.2.117:0/4110056054 >> v2:172.21.2.125:3300/0 conn(0x7ff1540e4da0 0x7ff1541b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 -- 172.21.2.117:0/4110056054 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15407de20 con 0x7ff1540e51a0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff153d75700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540e51a0 0x7ff154077ff0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7ff14c006f00 tx=0x7ff14c009e90).ready entity=mon.0 client_cookie=ab1287a2eec65ac9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff152572700 1 -- 172.21.2.117:0/4110056054 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff14c00b300 con 0x7ff1540e51a0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff152572700 1 -- 172.21.2.117:0/4110056054 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c00b580 con 0x7ff1540e51a0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff152572700 1 -- 172.21.2.117:0/4110056054 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff14c00b9d0 con 0x7ff1540e51a0 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff15996a700 1 -- 172.21.2.117:0/4110056054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540e51a0 msgr2=0x7ff154077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.809 7ff15996a700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540e51a0 0x7ff154077ff0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7ff14c006f00 tx=0x7ff14c009e90).stop 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 -- 172.21.2.117:0/4110056054 shutdown_connections 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 --2- 172.21.2.117:0/4110056054 >> v2:172.21.2.125:3300/0 conn(0x7ff1540e4da0 0x7ff1541b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540e51a0 0x7ff154077ff0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 --2- 172.21.2.117:0/4110056054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15407da20 0x7ff1540771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 -- 172.21.2.117:0/4110056054 >> 172.21.2.117:0/4110056054 conn(0x7ff1540ce450 msgr2=0x7ff1540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 -- 172.21.2.117:0/4110056054 shutdown_connections 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.810 7ff15996a700 1 -- 172.21.2.117:0/4110056054 wait complete. 2021-02-10T11:31:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.824 7ff15996a700 1 Processor -- start 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.824 7ff15996a700 1 -- start start 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 0x7ff1540bd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1540bd990 0x7ff1540bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540be4b0 con 0x7ff1540be9b0 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff154190260 con 0x7ff1540bd990 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff15996a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540c1840 con 0x7ff15418fe20 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44562/0 (socket says 172.21.2.117:44562) 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153d75700 1 -- 172.21.2.117:0/1884361731 learned_addr learned my addr 172.21.2.117:0/1884361731 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff152d73700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1540bd990 0x7ff1540bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153574700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 0x7ff1540bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153d75700 1 -- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 msgr2=0x7ff1540bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.825 7ff153d75700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 0x7ff1540bd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff153d75700 1 -- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1540bd990 msgr2=0x7ff1540bdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff153d75700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1540bd990 0x7ff1540bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff153d75700 1 -- 172.21.2.117:0/1884361731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540bf3d0 con 0x7ff1540be9b0 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff153574700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 0x7ff1540bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff153d75700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7ff14c00b910 tx=0x7ff14c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff14c00bdf0 con 0x7ff1540be9b0 2021-02-10T11:31:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c005820 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff15996a700 1 -- 172.21.2.117:0/1884361731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1540bf3d0 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff14c005c70 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.826 7ff15996a700 1 -- 172.21.2.117:0/1884361731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1540bf3d0 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.827 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff14c005a00 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.827 7ff14bfff700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 0x7ff13801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.827 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff14c032850 con 0x7ff1540be9b0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.827 7ff153574700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 0x7ff13801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.828 7ff153574700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 0x7ff13801abc0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:42.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.858 7ff15996a700 1 -- 172.21.2.117:0/1884361731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff154077a70 con 0x7ff1540be9b0 2021-02-10T11:31:42.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.861 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff14c02ced0 con 0x7ff1540be9b0 2021-02-10T11:31:42.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.989 7ff15996a700 1 -- 172.21.2.117:0/1884361731 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802486", "target": ["mgr", ""]}) v1 -- 0x7ff154077a70 con 0x7ff1380186c0 2021-02-10T11:31:42.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.993 7ff14bfff700 1 -- 172.21.2.117:0/1884361731 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7ff13c006060 con 0x7ff1380186c0 2021-02-10T11:31:42.981 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369 2021-02-10T11:31:42.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 -- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 msgr2=0x7ff13801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 0x7ff13801abc0 crc :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 -- 172.21.2.117:0/1884361731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 msgr2=0x7ff1540bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:42.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7ff14c00b910 tx=0x7ff14c00bcb0).stop 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 -- 172.21.2.117:0/1884361731 shutdown_connections 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1380186c0 0x7ff13801abc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1540be9b0 0x7ff1540bedb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1540bd990 0x7ff1540bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 --2- 172.21.2.117:0/1884361731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff15418fe20 0x7ff1540bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 -- 172.21.2.117:0/1884361731 >> 172.21.2.117:0/1884361731 conn(0x7ff1540a1360 msgr2=0x7ff154091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.994 7ff15996a700 1 -- 172.21.2.117:0/1884361731 shutdown_connections 2021-02-10T11:31:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:42.995 7ff15996a700 1 -- 172.21.2.117:0/1884361731 wait complete. 2021-02-10T11:31:42.992 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802486 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369 2021-02-10T11:31:42.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:42.992 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:42.992 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:43.329 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:43.329 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:43.330 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:43.709 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:43.709 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:43.710 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250566 s, 41.8 MB/s 2021-02-10T11:31:43.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:43.711 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:43.711 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:44.047 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:44.047 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:44.048 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244613 s, 42.9 MB/s 2021-02-10T11:31:44.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:44.424 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:44.424 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:44.760 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:44.761 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:44.761 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:45.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:45.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:45.134 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022519 s, 46.6 MB/s 2021-02-10T11:31:45.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:45.135 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:45.135 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:45.470 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:45.471 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:45.471 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:45.849 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:45.849 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:45.850 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252954 s, 41.5 MB/s 2021-02-10T11:31:45.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:45.851 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:45.851 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:46.186 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:46.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:46.186 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:46.561 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:46.561 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:46.561 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02347 s, 44.7 MB/s 2021-02-10T11:31:46.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:46.562 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:46.562 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:46.898 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:46.898 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:46.898 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:47.276 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:47.276 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:47.276 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254547 s, 41.2 MB/s 2021-02-10T11:31:47.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:47.277 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:47.277 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:47.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:47.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:47.614 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:47.986 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:47.986 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:47.986 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227157 s, 46.2 MB/s 2021-02-10T11:31:47.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:47.987 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:47.987 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:48.323 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:48.323 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:48.323 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242817 s, 43.2 MB/s 2021-02-10T11:31:48.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:48.701 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:48.701 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:49.037 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:49.037 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:49.037 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:49.411 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:49.411 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:49.411 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235454 s, 44.5 MB/s 2021-02-10T11:31:49.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:49.412 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:49.412 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:49.749 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:49.749 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:49.749 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802486/c0d59ebd-e3d1-4bb3-8a95-03dca3d24369/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:50.127 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025007 s, 41.9 MB/s 2021-02-10T11:31:50.128 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802487 2021-02-10T11:31:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 Processor -- start 2021-02-10T11:31:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 -- start start 2021-02-10T11:31:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f0079000 0x7f68f00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68f01b00f0 0x7f68f01b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f0077a70 con 0x7f68f007da20 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f00e53c0 con 0x7f68f01b00f0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f7696700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0079400 con 0x7f68f0079000 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.555 7f68f5cb3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68f01b00f0 0x7f68f01b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68f01b00f0 0x7f68f01b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44572/0 (socket says 172.21.2.117:44572) 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 -- 172.21.2.117:0/775630364 learned_addr learned my addr 172.21.2.117:0/775630364 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 -- 172.21.2.117:0/775630364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f0079000 msgr2=0x7f68f00e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 --2- 172.21.2.117:0/775630364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f0079000 0x7f68f00e4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 -- 172.21.2.117:0/775630364 >> v1:172.21.2.125:6789/0 conn(0x7f68f007da20 legacy=0x7f68f00771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 -- 172.21.2.117:0/775630364 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f007de20 con 0x7f68f01b00f0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f5cb3700 1 --2- 172.21.2.117:0/775630364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f01b00f0 0x7f68f01b04f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f68e8006f00 tx=0x7f68e8009f10).ready entity=mon.0 client_cookie=aa4fb65b855e44d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68e3fff700 1 -- 172.21.2.117:0/775630364 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e800b3b0 con 0x7f68f01b00f0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68e3fff700 1 -- 172.21.2.117:0/775630364 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e800b630 con 0x7f68f01b00f0 2021-02-10T11:31:50.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68e3fff700 1 -- 172.21.2.117:0/775630364 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e800bb10 con 0x7f68f01b00f0 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f7696700 1 -- 172.21.2.117:0/775630364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f01b00f0 msgr2=0x7f68f01b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.556 7f68f7696700 1 --2- 172.21.2.117:0/775630364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f01b00f0 0x7f68f01b04f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f68e8006f00 tx=0x7f68e8009f10).stop 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 -- 172.21.2.117:0/775630364 shutdown_connections 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 --2- 172.21.2.117:0/775630364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f01b00f0 0x7f68f01b04f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 --2- 172.21.2.117:0/775630364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f0079000 0x7f68f00e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 -- 172.21.2.117:0/775630364 >> 172.21.2.117:0/775630364 conn(0x7f68f00ce450 msgr2=0x7f68f00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 -- 172.21.2.117:0/775630364 shutdown_connections 2021-02-10T11:31:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.557 7f68f7696700 1 -- 172.21.2.117:0/775630364 wait complete. 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.571 7f68f7696700 1 Processor -- start 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.571 7f68f7696700 1 -- start start 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68f00b0f40 0x7f68f0107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f018b830 0x7f68f0187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00b18c0 con 0x7f68f00b1340 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f018ac90 con 0x7f68f00b0f40 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f7696700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0107f50 con 0x7f68f018b830 2021-02-10T11:31:50.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44576/0 (socket says 172.21.2.117:44576) 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 -- 172.21.2.117:0/1108093812 learned_addr learned my addr 172.21.2.117:0/1108093812 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 -- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f018b830 msgr2=0x7f68f0187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f018b830 0x7f68f0187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 -- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68f00b0f40 msgr2=0x7f68f0107930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68f00b0f40 0x7f68f0107930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.572 7f68f4cb1700 1 -- 172.21.2.117:0/1108093812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0108130 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68f4cb1700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f68e400b560 tx=0x7f68e400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e400bcf0 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e4006030 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68e40054d0 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68f7696700 1 -- 172.21.2.117:0/1108093812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f0108130 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68f7696700 1 -- 172.21.2.117:0/1108093812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f0108130 con 0x7f68f00b1340 2021-02-10T11:31:50.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.573 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68e4006210 con 0x7f68f00b1340 2021-02-10T11:31:50.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.574 7f68e1ffb700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 0x7f68d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:50.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.574 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f68e40326f0 con 0x7f68f00b1340 2021-02-10T11:31:50.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.574 7f68f54b2700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 0x7f68d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:50.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.574 7f68f54b2700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 0x7f68d001abc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:50.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.605 7f68f7696700 1 -- 172.21.2.117:0/1108093812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0077a70 con 0x7f68f00b1340 2021-02-10T11:31:50.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.608 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68e4006b80 con 0x7f68f00b1340 2021-02-10T11:31:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.733 7f68f7696700 1 -- 172.21.2.117:0/1108093812 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802487", "target": ["mgr", ""]}) v1 -- 0x7f68f0077a70 con 0x7f68d00186c0 2021-02-10T11:31:50.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.757 7f68e1ffb700 1 -- 172.21.2.117:0/1108093812 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f68ec000c60 con 0x7f68d00186c0 2021-02-10T11:31:50.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 msgr2=0x7f68d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 0x7f68d001abc0 crc :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 msgr2=0x7f68f018b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:50.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f68e400b560 tx=0x7f68e400b900).stop 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 shutdown_connections 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68d00186c0 0x7f68d001abc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68f00b1340 0x7f68f018b210 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68f018b830 0x7f68f0187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 --2- 172.21.2.117:0/1108093812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68f00b0f40 0x7f68f0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 >> 172.21.2.117:0/1108093812 conn(0x7f68f00a1880 msgr2=0x7f68f00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 shutdown_connections 2021-02-10T11:31:50.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:50.766 7f68f7696700 1 -- 172.21.2.117:0/1108093812 wait complete. 2021-02-10T11:31:50.763 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802487 2021-02-10T11:31:51.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.187 7f56c2955700 1 Processor -- start 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.187 7f56c2955700 1 -- start start 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c2955700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc07da20 0x7f56bc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c2955700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56bc0e51a0 0x7f56bc0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c2955700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56bc077a70 con 0x7f56bc0e4da0 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c2955700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56bc0e5c20 con 0x7f56bc0e51a0 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c2955700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0777d0 con 0x7f56bc07da20 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56bc0e51a0 0x7f56bc0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f56bc0e51a0 0x7f56bc0d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44582/0 (socket says 172.21.2.117:44582) 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 -- 172.21.2.117:0/105662513 learned_addr learned my addr 172.21.2.117:0/105662513 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:51.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 -- 172.21.2.117:0/105662513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc07da20 msgr2=0x7f56bc0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 --2- 172.21.2.117:0/105662513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc07da20 0x7f56bc0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 -- 172.21.2.117:0/105662513 >> v1:172.21.2.125:6789/0 conn(0x7f56bc0e4da0 legacy=0x7f56bc1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 -- 172.21.2.117:0/105662513 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc1b08e0 con 0x7f56bc0e51a0 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56c0f72700 1 --2- 172.21.2.117:0/105662513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc0e51a0 0x7f56bc0d0520 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f56b4006f00 tx=0x7f56b4009f10).ready entity=mon.0 client_cookie=33ae1e2c09c5f507 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.188 7f56baffd700 1 -- 172.21.2.117:0/105662513 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56b400b3b0 con 0x7f56bc0e51a0 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56baffd700 1 -- 172.21.2.117:0/105662513 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b400b730 con 0x7f56bc0e51a0 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56baffd700 1 -- 172.21.2.117:0/105662513 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56b400bc10 con 0x7f56bc0e51a0 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 -- 172.21.2.117:0/105662513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc0e51a0 msgr2=0x7f56bc0d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 --2- 172.21.2.117:0/105662513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc0e51a0 0x7f56bc0d0520 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f56b4006f00 tx=0x7f56b4009f10).stop 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 -- 172.21.2.117:0/105662513 shutdown_connections 2021-02-10T11:31:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 --2- 172.21.2.117:0/105662513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc0e51a0 0x7f56bc0d0520 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 --2- 172.21.2.117:0/105662513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc07da20 0x7f56bc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 -- 172.21.2.117:0/105662513 >> 172.21.2.117:0/105662513 conn(0x7f56bc0ce450 msgr2=0x7f56bc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 -- 172.21.2.117:0/105662513 shutdown_connections 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.189 7f56c2955700 1 -- 172.21.2.117:0/105662513 wait complete. 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.203 7f56c2955700 1 Processor -- start 2021-02-10T11:31:51.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 -- start start 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc0a4ca0 0x7f56bc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56bc19f570 0x7f56bc102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0a50a0 con 0x7f56bc19f170 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc0a6910 con 0x7f56bc19f570 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56c2955700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc102e60 con 0x7f56bc0a4ca0 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44588/0 (socket says 172.21.2.117:44588) 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 -- 172.21.2.117:0/909881445 learned_addr learned my addr 172.21.2.117:0/909881445 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 -- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc0a4ca0 msgr2=0x7f56bc19eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc0a4ca0 0x7f56bc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 -- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56bc19f570 msgr2=0x7f56bc102840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.204 7f56bb7fe700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56bc19f570 0x7f56bc102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56bb7fe700 1 -- 172.21.2.117:0/909881445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc103080 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56bb7fe700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f56b000b5b0 tx=0x7f56b000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56b000bd40 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0006240 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56c2955700 1 -- 172.21.2.117:0/909881445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc103080 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56b0005720 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56c2955700 1 -- 172.21.2.117:0/909881445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc103080 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.205 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56b00054a0 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.206 7f56b8ff9700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 0x7f569c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.206 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f56b0032570 con 0x7f56bc19f170 2021-02-10T11:31:51.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.206 7f56bbfff700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 0x7f569c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:51.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.206 7f56bbfff700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 0x7f569c01abc0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:51.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.238 7f56c2955700 1 -- 172.21.2.117:0/909881445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc0ff160 con 0x7f56bc19f170 2021-02-10T11:31:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.240 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56b0006030 con 0x7f56bc19f170 2021-02-10T11:31:51.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.366 7f56c2955700 1 -- 172.21.2.117:0/909881445 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802487", "target": ["mgr", ""]}) v1 -- 0x7f56bc0a6b30 con 0x7f569c0186c0 2021-02-10T11:31:51.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.370 7f56b8ff9700 1 -- 172.21.2.117:0/909881445 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f56ac006e60 con 0x7f569c0186c0 2021-02-10T11:31:51.363 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 msgr2=0x7f569c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 0x7f569c01abc0 crc :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 msgr2=0x7f56bc102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f56b000b5b0 tx=0x7f56b000b950).stop 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 shutdown_connections 2021-02-10T11:31:51.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f569c0186c0 0x7f569c01abc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56bc19f170 0x7f56bc102220 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56bc19f570 0x7f56bc102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 --2- 172.21.2.117:0/909881445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56bc0a4ca0 0x7f56bc19eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 >> 172.21.2.117:0/909881445 conn(0x7f56bc0b6d80 msgr2=0x7f56bc0c3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 shutdown_connections 2021-02-10T11:31:51.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:51.376 7f56c2955700 1 -- 172.21.2.117:0/909881445 wait complete. 2021-02-10T11:31:51.373 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802487 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a 2021-02-10T11:31:51.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:51.373 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:51.373 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:51.709 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:51.709 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:51.710 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:52.089 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:52.089 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227195 s, 46.2 MB/s 2021-02-10T11:31:52.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:52.090 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:52.090 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:52.425 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:52.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:52.425 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261694 s, 40.1 MB/s 2021-02-10T11:31:52.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:52.801 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:52.801 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:53.136 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:53.136 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:53.136 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:53.511 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:53.511 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:53.511 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248211 s, 42.2 MB/s 2021-02-10T11:31:53.512 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:53.512 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:53.512 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:53.848 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:53.848 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:53.848 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:54.226 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:54.226 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:54.227 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251761 s, 41.6 MB/s 2021-02-10T11:31:54.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:54.228 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:54.228 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:54.573 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:54.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:54.574 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:54.962 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:54.963 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:54.963 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226796 s, 46.2 MB/s 2021-02-10T11:31:54.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:54.964 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:54.964 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:55.301 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:55.301 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:55.302 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:55.677 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:55.677 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:55.677 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241576 s, 43.4 MB/s 2021-02-10T11:31:55.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:55.678 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:55.678 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:56.014 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:56.014 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:56.014 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237204 s, 44.2 MB/s 2021-02-10T11:31:56.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:56.390 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:56.390 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:56.725 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:56.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:56.726 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:57.103 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:57.103 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:57.104 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261869 s, 40.0 MB/s 2021-02-10T11:31:57.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:57.105 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:57.105 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:57.439 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:57.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:57.440 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:57.813 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:57.813 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:57.814 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248843 s, 42.1 MB/s 2021-02-10T11:31:57.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:57.815 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:57.815 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:31:58.150 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:31:58.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:31:58.150 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802487/31f8b915-635c-4dd8-9cf8-da4cab4a844a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:31:58.525 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:31:58.525 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:31:58.525 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240235 s, 43.6 MB/s 2021-02-10T11:31:58.526 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802488 2021-02-10T11:31:58.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.952 7f0e6a2f9700 1 Processor -- start 2021-02-10T11:31:58.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.952 7f0e6a2f9700 1 -- start start 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e6a2f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 0x7f0e640d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e6a2f9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e64077a70 con 0x7f0e6407da20 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e6a2f9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e64079400 con 0x7f0e64079000 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e6a2f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6407de20 con 0x7f0e640d1530 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e63fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0e6407da20 0x7f0e640771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44846/0 (socket says 172.21.2.117:44846) 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e63fff700 1 -- 172.21.2.117:0/2468020886 learned_addr learned my addr 172.21.2.117:0/2468020886 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e68916700 1 --2- 172.21.2.117:0/2468020886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 0x7f0e640d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.953 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299776809 0 0) 0x7f0e54001430 con 0x7f0e6407da20 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e50002fe0 con 0x7f0e6407da20 2021-02-10T11:31:58.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008157130 0 0) 0x7f0e4c000c60 con 0x7f0e64079000 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e50003210 con 0x7f0e64079000 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e68916700 1 -- 172.21.2.117:0/2468020886 >> v1:172.21.2.125:6789/0 conn(0x7f0e64079000 legacy=0x7f0e640e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e68916700 1 -- 172.21.2.117:0/2468020886 >> v1:172.21.2.117:6789/0 conn(0x7f0e6407da20 legacy=0x7f0e640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e68916700 1 -- 172.21.2.117:0/2468020886 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e641b00f0 con 0x7f0e640d1530 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e68916700 1 --2- 172.21.2.117:0/2468020886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 0x7f0e640d1930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0e5c006ad0 tx=0x7f0e5c009a50).ready entity=mon.2 client_cookie=6428ef47e6487026 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e5c00c0c0 con 0x7f0e640d1530 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c00c340 con 0x7f0e640d1530 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e62ffd700 1 -- 172.21.2.117:0/2468020886 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e5c00c790 con 0x7f0e640d1530 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e6a2f9700 1 -- 172.21.2.117:0/2468020886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 msgr2=0x7f0e640d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.954 7f0e6a2f9700 1 --2- 172.21.2.117:0/2468020886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 0x7f0e640d1930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0e5c006ad0 tx=0x7f0e5c009a50).stop 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.955 7f0e6a2f9700 1 -- 172.21.2.117:0/2468020886 shutdown_connections 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.955 7f0e6a2f9700 1 --2- 172.21.2.117:0/2468020886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640d1530 0x7f0e640d1930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.955 7f0e6a2f9700 1 -- 172.21.2.117:0/2468020886 >> 172.21.2.117:0/2468020886 conn(0x7f0e640ce450 msgr2=0x7f0e640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:58.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.955 7f0e6a2f9700 1 -- 172.21.2.117:0/2468020886 shutdown_connections 2021-02-10T11:31:58.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.955 7f0e6a2f9700 1 -- 172.21.2.117:0/2468020886 wait complete. 2021-02-10T11:31:58.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 Processor -- start 2021-02-10T11:31:58.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 -- start start 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e640b9e60 0x7f0e640a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640a3800 0x7f0e640a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640a4220 con 0x7f0e6419a4a0 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640a4410 con 0x7f0e640b9e60 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e6a2f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e640ba910 con 0x7f0e640a3800 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e63fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e63fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44600/0 (socket says 172.21.2.117:44600) 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.970 7f0e63fff700 1 -- 172.21.2.117:0/989698165 learned_addr learned my addr 172.21.2.117:0/989698165 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 -- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640a3800 msgr2=0x7f0e640a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640a3800 0x7f0e640a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 -- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e640b9e60 msgr2=0x7f0e640a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e640b9e60 0x7f0e640a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 -- 172.21.2.117:0/989698165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e640bab30 con 0x7f0e6419a4a0 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e63fff700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f0e54001910 tx=0x7f0e5400ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e54007020 con 0x7f0e6419a4a0 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e540072a0 con 0x7f0e6419a4a0 2021-02-10T11:31:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e540048c0 con 0x7f0e6419a4a0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e640bab30 con 0x7f0e6419a4a0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.971 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e640bb0c0 con 0x7f0e6419a4a0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.972 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e5400ad90 con 0x7f0e6419a4a0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.972 7f0e60ff9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 0x7f0e4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.972 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0e54032a80 con 0x7f0e6419a4a0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.972 7f0e637fe700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 0x7f0e4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:58.973 7f0e637fe700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 0x7f0e4801abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:58.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.002 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e641b4f00 con 0x7f0e6419a4a0 2021-02-10T11:31:58.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.005 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e54000d30 con 0x7f0e6419a4a0 2021-02-10T11:31:59.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.129 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802488", "target": ["mgr", ""]}) v1 -- 0x7f0e640bb0c0 con 0x7f0e480186c0 2021-02-10T11:31:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.153 7f0e60ff9700 1 -- 172.21.2.117:0/989698165 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0e4c000e20 con 0x7f0e480186c0 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.162 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 msgr2=0x7f0e4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.162 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 0x7f0e4801abc0 crc :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.162 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 msgr2=0x7f0e640b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.162 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f0e54001910 tx=0x7f0e5400ac50).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 shutdown_connections 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e640a3800 0x7f0e640a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e640b9e60 0x7f0e640a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e480186c0 0x7f0e4801abc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 --2- 172.21.2.117:0/989698165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e6419a4a0 0x7f0e640b9840 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 >> 172.21.2.117:0/989698165 conn(0x7f0e64107b60 msgr2=0x7f0e640c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:59.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 shutdown_connections 2021-02-10T11:31:59.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.163 7f0e6a2f9700 1 -- 172.21.2.117:0/989698165 wait complete. 2021-02-10T11:31:59.160 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802488 2021-02-10T11:31:59.234 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:59.238 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:31:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 Processor -- start 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 -- start start 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f360007da20 0x7f36000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36000e4da0 0x7f36001b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f36000e51a0 0x7f3600077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36000e5c20 con 0x7f36000e4da0 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3600077a70 con 0x7f36000e51a0 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.588 7f3606c48700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36000777d0 con 0x7f360007da20 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36000e4da0 0x7f36001b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36000e4da0 0x7f36001b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44610/0 (socket says 172.21.2.117:44610) 2021-02-10T11:31:59.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 -- 172.21.2.117:0/2985437906 learned_addr learned my addr 172.21.2.117:0/2985437906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f3604a64700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f360007da20 0x7f36000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f3605265700 1 --2- 172.21.2.117:0/2985437906 >> v2:172.21.2.125:3300/0 conn(0x7f36000e51a0 0x7f3600077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 -- 172.21.2.117:0/2985437906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f360007da20 msgr2=0x7f36000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f360007da20 0x7f36000771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 -- 172.21.2.117:0/2985437906 >> v2:172.21.2.125:3300/0 conn(0x7f36000e51a0 msgr2=0x7f3600077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 --2- 172.21.2.117:0/2985437906 >> v2:172.21.2.125:3300/0 conn(0x7f36000e51a0 0x7f3600077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 -- 172.21.2.117:0/2985437906 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360007de20 con 0x7f36000e4da0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35fffff700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36000e4da0 0x7f36001b00f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f35f0006700 tx=0x7f35f0009b80).ready entity=mon.0 client_cookie=627d47f7140bdfab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35ff7fe700 1 -- 172.21.2.117:0/2985437906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35f0000c90 con 0x7f36000e4da0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35ff7fe700 1 -- 172.21.2.117:0/2985437906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f000c0c0 con 0x7f36000e4da0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f35ff7fe700 1 -- 172.21.2.117:0/2985437906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35f000c5e0 con 0x7f36000e4da0 2021-02-10T11:31:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f3606c48700 1 -- 172.21.2.117:0/2985437906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36000e4da0 msgr2=0x7f36001b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.589 7f3606c48700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36000e4da0 0x7f36001b00f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f35f0006700 tx=0x7f35f0009b80).stop 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 -- 172.21.2.117:0/2985437906 shutdown_connections 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 --2- 172.21.2.117:0/2985437906 >> v2:172.21.2.125:3300/0 conn(0x7f36000e51a0 0x7f3600077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36000e4da0 0x7f36001b00f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 --2- 172.21.2.117:0/2985437906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f360007da20 0x7f36000771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 -- 172.21.2.117:0/2985437906 >> 172.21.2.117:0/2985437906 conn(0x7f36000ce450 msgr2=0x7f36000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 -- 172.21.2.117:0/2985437906 shutdown_connections 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.590 7f3606c48700 1 -- 172.21.2.117:0/2985437906 wait complete. 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 Processor -- start 2021-02-10T11:31:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 -- start start 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3600187e20 0x7f3600188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3600090420 0x7f3600090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36001067f0 con 0x7f36001a0250 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600107350 con 0x7f3600187e20 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3606c48700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36001042d0 con 0x7f3600090420 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3604a64700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3604a64700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44616/0 (socket says 172.21.2.117:44616) 2021-02-10T11:31:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.605 7f3604a64700 1 -- 172.21.2.117:0/1121663973 learned_addr learned my addr 172.21.2.117:0/1121663973 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 -- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3600090420 msgr2=0x7f3600090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3600090420 0x7f3600090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 -- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3600187e20 msgr2=0x7f3600188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3600187e20 0x7f3600188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 -- 172.21.2.117:0/1121663973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3600090e40 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3604a64700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f35e800b530 tx=0x7f35e800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e800bcc0 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8002030 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e8002510 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3606c48700 1 -- 172.21.2.117:0/1121663973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3600090e40 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.606 7f3606c48700 1 -- 172.21.2.117:0/1121663973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36000913d0 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.607 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35e8000c90 con 0x7f36001a0250 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.607 7f35fd7fa700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 0x7f35e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:31:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.607 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f35e8032670 con 0x7f36001a0250 2021-02-10T11:31:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.607 7f35fffff700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 0x7f35e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:31:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.608 7f35fffff700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 0x7f35e401abc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:31:59.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.638 7f3606c48700 1 -- 172.21.2.117:0/1121663973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3600103280 con 0x7f36001a0250 2021-02-10T11:31:59.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.641 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35e8024b30 con 0x7f36001a0250 2021-02-10T11:31:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.768 7f3606c48700 1 -- 172.21.2.117:0/1121663973 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802488", "target": ["mgr", ""]}) v1 -- 0x7f36000913d0 con 0x7f35e40186c0 2021-02-10T11:31:59.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.772 7f35fd7fa700 1 -- 172.21.2.117:0/1121663973 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f35f0000940 con 0x7f35e40186c0 2021-02-10T11:31:59.765 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9 2021-02-10T11:31:59.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 -- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 msgr2=0x7f35e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 0x7f35e401abc0 crc :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 -- 172.21.2.117:0/1121663973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 msgr2=0x7f3600106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:31:59.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f35e800b530 tx=0x7f35e800b8d0).stop 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 -- 172.21.2.117:0/1121663973 shutdown_connections 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3600090420 0x7f3600090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3600187e20 0x7f3600188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35e40186c0 0x7f35e401abc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 --2- 172.21.2.117:0/1121663973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36001a0250 0x7f3600106cf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.778 7f3606c48700 1 -- 172.21.2.117:0/1121663973 >> 172.21.2.117:0/1121663973 conn(0x7f36000a1850 msgr2=0x7f36000c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.779 7f3606c48700 1 -- 172.21.2.117:0/1121663973 shutdown_connections 2021-02-10T11:31:59.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:31:59.779 7f3606c48700 1 -- 172.21.2.117:0/1121663973 wait complete. 2021-02-10T11:31:59.775 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802488 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9 2021-02-10T11:31:59.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:31:59.776 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:31:59.776 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:00.110 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:00.111 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:00.111 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:00.493 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:00.494 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:00.494 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251303 s, 41.7 MB/s 2021-02-10T11:32:00.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:00.495 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:00.495 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:00.831 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:00.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:00.831 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:01.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250969 s, 41.8 MB/s 2021-02-10T11:32:01.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.209 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:01.209 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:01.545 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:01.545 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:01.545 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239891 s, 43.7 MB/s 2021-02-10T11:32:01.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:01.923 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:01.923 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:02.259 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:02.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:02.260 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:02.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:02.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:02.632 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223792 s, 46.9 MB/s 2021-02-10T11:32:02.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:02.633 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:02.633 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:02.969 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:02.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:02.970 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:03.346 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:03.346 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:03.346 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251487 s, 41.7 MB/s 2021-02-10T11:32:03.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:03.347 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:03.347 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:03.683 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:03.683 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:03.683 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:04.057 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:04.058 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:04.058 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231802 s, 45.2 MB/s 2021-02-10T11:32:04.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:04.060 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:04.060 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:04.396 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:04.396 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:04.396 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235398 s, 44.5 MB/s 2021-02-10T11:32:04.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:04.773 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:04.773 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:05.107 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:05.108 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:05.108 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:05.484 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:05.484 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:05.484 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246567 s, 42.5 MB/s 2021-02-10T11:32:05.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:05.485 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:05.485 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:05.820 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:05.821 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:05.821 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:06.197 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:06.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:06.198 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241156 s, 43.5 MB/s 2021-02-10T11:32:06.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:06.199 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:06.199 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:06.536 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:06.536 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:06.536 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802488/6d71aac6-5228-49ba-a8d2-0edaa1f885c9/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:06.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:06.916 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:06.917 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0266848 s, 39.3 MB/s 2021-02-10T11:32:06.918 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802489 2021-02-10T11:32:07.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.343 7f0515189700 1 Processor -- start 2021-02-10T11:32:07.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.343 7f0515189700 1 -- start start 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.343 7f0515189700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f051007da20 0x7f05100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f0515189700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f05100e5180 0x7f0510079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f0515189700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510077a70 con 0x7f05100e4d80 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f0515189700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05100e5c00 con 0x7f051007da20 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f0515189700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05100777d0 con 0x7f05100e5180 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f051007da20 0x7f05100771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f051007da20 0x7f05100771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44620/0 (socket says 172.21.2.117:44620) 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 -- 172.21.2.117:0/1121975848 learned_addr learned my addr 172.21.2.117:0/1121975848 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 -- 172.21.2.117:0/1121975848 >> v2:172.21.2.125:3300/0 conn(0x7f05100e5180 msgr2=0x7f0510079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 --2- 172.21.2.117:0/1121975848 >> v2:172.21.2.125:3300/0 conn(0x7f05100e5180 0x7f0510079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 -- 172.21.2.117:0/1121975848 >> v1:172.21.2.125:6789/0 conn(0x7f05100e4d80 legacy=0x7f05101b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 -- 172.21.2.117:0/1121975848 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05101b08c0 con 0x7f051007da20 2021-02-10T11:32:07.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.344 7f050ed9d700 1 --2- 172.21.2.117:0/1121975848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f051007da20 0x7f05100771b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f0500006700 tx=0x7f0500009b80).ready entity=mon.0 client_cookie=33213746c6e40745 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f050dd9b700 1 -- 172.21.2.117:0/1121975848 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f050000c0c0 con 0x7f051007da20 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f050dd9b700 1 -- 172.21.2.117:0/1121975848 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050000c380 con 0x7f051007da20 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f050dd9b700 1 -- 172.21.2.117:0/1121975848 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f050000c8a0 con 0x7f051007da20 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 -- 172.21.2.117:0/1121975848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f051007da20 msgr2=0x7f05100771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 --2- 172.21.2.117:0/1121975848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f051007da20 0x7f05100771b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f0500006700 tx=0x7f0500009b80).stop 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 -- 172.21.2.117:0/1121975848 shutdown_connections 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 --2- 172.21.2.117:0/1121975848 >> v2:172.21.2.125:3300/0 conn(0x7f05100e5180 0x7f0510079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 --2- 172.21.2.117:0/1121975848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f051007da20 0x7f05100771b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 -- 172.21.2.117:0/1121975848 >> 172.21.2.117:0/1121975848 conn(0x7f05100ce430 msgr2=0x7f05100e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 -- 172.21.2.117:0/1121975848 shutdown_connections 2021-02-10T11:32:07.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.345 7f0515189700 1 -- 172.21.2.117:0/1121975848 wait complete. 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 Processor -- start 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 -- start start 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05100a46c0 0x7f05100a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05100a02a0 0x7f05100a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05101a13f0 con 0x7f05100a0cc0 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05100a52e0 con 0x7f05100a02a0 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.360 7f0515189700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05100a10c0 con 0x7f05100a46c0 2021-02-10T11:32:07.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44628/0 (socket says 172.21.2.117:44628) 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 -- 172.21.2.117:0/3317027128 learned_addr learned my addr 172.21.2.117:0/3317027128 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 -- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05100a46c0 msgr2=0x7f05100a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05100a46c0 0x7f05100a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 -- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05100a02a0 msgr2=0x7f05100a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05100a02a0 0x7f05100a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 -- 172.21.2.117:0/3317027128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05100a12a0 con 0x7f05100a0cc0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f050f59e700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f0508009980 tx=0x7f050800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0508003020 con 0x7f05100a0cc0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0508005a80 con 0x7f05100a0cc0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f0515189700 1 -- 172.21.2.117:0/3317027128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05100a12a0 con 0x7f05100a0cc0 2021-02-10T11:32:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f0515189700 1 -- 172.21.2.117:0/3317027128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05100a6170 con 0x7f05100a0cc0 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.361 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0508006030 con 0x7f05100a0cc0 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.362 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0508006500 con 0x7f05100a0cc0 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.362 7f04ff7fe700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 0x7f04f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.362 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0508032b70 con 0x7f05100a0cc0 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.362 7f050ed9d700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 0x7f04f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.363 7f050ed9d700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 0x7f04f401ac10 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.394 7f0515189700 1 -- 172.21.2.117:0/3317027128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05100a52e0 con 0x7f05100a0cc0 2021-02-10T11:32:07.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.396 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05080062f0 con 0x7f05100a0cc0 2021-02-10T11:32:07.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.525 7f0515189700 1 -- 172.21.2.117:0/3317027128 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802489", "target": ["mgr", ""]}) v1 -- 0x7f05100a6170 con 0x7f04f4018730 2021-02-10T11:32:07.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.547 7f04ff7fe700 1 -- 172.21.2.117:0/3317027128 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0500000990 con 0x7f04f4018730 2021-02-10T11:32:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.558 7f0515189700 1 -- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 msgr2=0x7f04f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.558 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 0x7f04f401ac10 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.558 7f0515189700 1 -- 172.21.2.117:0/3317027128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 msgr2=0x7f05100a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.558 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f0508009980 tx=0x7f050800bcd0).stop 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 -- 172.21.2.117:0/3317027128 shutdown_connections 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04f4018730 0x7f04f401ac10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05100a0cc0 0x7f05100a5860 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05100a02a0 0x7f05100a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 --2- 172.21.2.117:0/3317027128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05100a46c0 0x7f05100a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 -- 172.21.2.117:0/3317027128 >> 172.21.2.117:0/3317027128 conn(0x7f05100a8d10 msgr2=0x7f0510107a90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 -- 172.21.2.117:0/3317027128 shutdown_connections 2021-02-10T11:32:07.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.559 7f0515189700 1 -- 172.21.2.117:0/3317027128 wait complete. 2021-02-10T11:32:07.556 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802489 2021-02-10T11:32:07.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.982 7f3a59584700 1 Processor -- start 2021-02-10T11:32:07.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.982 7f3a59584700 1 -- start start 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.982 7f3a59584700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3a5407da20 0x7f3a540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.982 7f3a59584700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a540e4da0 0x7f3a541b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a540e51a0 0x7f3a54077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a54077a70 con 0x7f3a540e4da0 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540777d0 con 0x7f3a5407da20 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5407de20 con 0x7f3a540e51a0 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a540e4da0 0x7f3a541b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a540e4da0 0x7f3a541b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44632/0 (socket says 172.21.2.117:44632) 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 -- 172.21.2.117:0/2935846509 learned_addr learned my addr 172.21.2.117:0/2935846509 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a53ac0700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a540e51a0 0x7f3a54077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a532bf700 1 --2- 172.21.2.117:0/2935846509 >> v2:172.21.2.125:3300/0 conn(0x7f3a5407da20 0x7f3a540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 -- 172.21.2.117:0/2935846509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a540e51a0 msgr2=0x7f3a54077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a540e51a0 0x7f3a54077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 -- 172.21.2.117:0/2935846509 >> v2:172.21.2.125:3300/0 conn(0x7f3a5407da20 msgr2=0x7f3a540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 --2- 172.21.2.117:0/2935846509 >> v2:172.21.2.125:3300/0 conn(0x7f3a5407da20 0x7f3a540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 -- 172.21.2.117:0/2935846509 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a541b0710 con 0x7f3a540e4da0 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a52abe700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a540e4da0 0x7f3a541b00f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f3a44006700 tx=0x7f3a44009b00).ready entity=mon.0 client_cookie=be16f435b19d09c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a522bd700 1 -- 172.21.2.117:0/2935846509 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a4400c0c0 con 0x7f3a540e4da0 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a522bd700 1 -- 172.21.2.117:0/2935846509 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4400c340 con 0x7f3a540e4da0 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a522bd700 1 -- 172.21.2.117:0/2935846509 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a4400c820 con 0x7f3a540e4da0 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 -- 172.21.2.117:0/2935846509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a540e4da0 msgr2=0x7f3a541b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.983 7f3a59584700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a540e4da0 0x7f3a541b00f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f3a44006700 tx=0x7f3a44009b00).stop 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 -- 172.21.2.117:0/2935846509 shutdown_connections 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a540e51a0 0x7f3a54077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 --2- 172.21.2.117:0/2935846509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a540e4da0 0x7f3a541b00f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 --2- 172.21.2.117:0/2935846509 >> v2:172.21.2.125:3300/0 conn(0x7f3a5407da20 0x7f3a540771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 -- 172.21.2.117:0/2935846509 >> 172.21.2.117:0/2935846509 conn(0x7f3a540ce450 msgr2=0x7f3a540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 -- 172.21.2.117:0/2935846509 shutdown_connections 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.984 7f3a59584700 1 -- 172.21.2.117:0/2935846509 wait complete. 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 Processor -- start 2021-02-10T11:32:07.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 -- start start 2021-02-10T11:32:07.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a541a0250 0x7f3a54106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a54090420 0x7f3a54090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541067f0 con 0x7f3a54187e20 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a54107350 con 0x7f3a54090420 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a59584700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a541042d0 con 0x7f3a541a0250 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a52abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a52abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44640/0 (socket says 172.21.2.117:44640) 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:07.999 7f3a52abe700 1 -- 172.21.2.117:0/2273908726 learned_addr learned my addr 172.21.2.117:0/2273908726 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 -- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a541a0250 msgr2=0x7f3a54106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a541a0250 0x7f3a54106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 -- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a54090420 msgr2=0x7f3a54090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a54090420 0x7f3a54090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 -- 172.21.2.117:0/2273908726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54090e40 con 0x7f3a54187e20 2021-02-10T11:32:07.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a52abe700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f3a44006f20 tx=0x7f3a4400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a44000c90 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a44009100 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a44002220 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a59584700 1 -- 172.21.2.117:0/2273908726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a54090e40 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.000 7f3a59584700 1 -- 172.21.2.117:0/2273908726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a54090e40 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.001 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a4400a000 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.001 7f3a4bfff700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 0x7f3a3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.001 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3a44032530 con 0x7f3a54187e20 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.001 7f3a532bf700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 0x7f3a3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:07.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.002 7f3a532bf700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 0x7f3a3801abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:08.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.032 7f3a59584700 1 -- 172.21.2.117:0/2273908726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a54103280 con 0x7f3a54187e20 2021-02-10T11:32:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.035 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a4402cee0 con 0x7f3a54187e20 2021-02-10T11:32:08.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.163 7f3a59584700 1 -- 172.21.2.117:0/2273908726 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802489", "target": ["mgr", ""]}) v1 -- 0x7f3a54103280 con 0x7f3a380186c0 2021-02-10T11:32:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.168 7f3a4bfff700 1 -- 172.21.2.117:0/2273908726 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3a3c006a20 con 0x7f3a380186c0 2021-02-10T11:32:08.160 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 msgr2=0x7f3a3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 0x7f3a3801abc0 crc :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 msgr2=0x7f3a54188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f3a44006f20 tx=0x7f3a4400b990).stop 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 shutdown_connections 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a380186c0 0x7f3a3801abc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a54187e20 0x7f3a54188220 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a54090420 0x7f3a54090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 --2- 172.21.2.117:0/2273908726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a541a0250 0x7f3a54106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 >> 172.21.2.117:0/2273908726 conn(0x7f3a540a1850 msgr2=0x7f3a540c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 shutdown_connections 2021-02-10T11:32:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:08.173 7f3a59584700 1 -- 172.21.2.117:0/2273908726 wait complete. 2021-02-10T11:32:08.170 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802489 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1 2021-02-10T11:32:08.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:08.170 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:08.170 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:08.505 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:08.506 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:08.506 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:08.885 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:08.885 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:08.885 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230711 s, 45.4 MB/s 2021-02-10T11:32:08.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:08.886 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:08.886 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:09.222 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:09.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:09.223 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232636 s, 45.1 MB/s 2021-02-10T11:32:09.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:09.598 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:09.599 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:09.935 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:09.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:09.935 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226936 s, 46.2 MB/s 2021-02-10T11:32:10.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:10.311 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:10.311 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:10.646 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:10.646 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:10.646 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:11.023 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:11.023 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:11.023 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251228 s, 41.7 MB/s 2021-02-10T11:32:11.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:11.025 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:11.025 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:11.361 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:11.362 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:11.362 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:11.735 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:11.736 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:11.736 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235802 s, 44.5 MB/s 2021-02-10T11:32:11.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:11.737 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:11.737 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:12.072 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:12.072 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:12.072 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:12.446 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:12.446 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:12.447 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025679 s, 40.8 MB/s 2021-02-10T11:32:12.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:12.448 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:12.448 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:12.784 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:12.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:12.784 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232502 s, 45.1 MB/s 2021-02-10T11:32:13.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:13.160 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:13.160 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:13.495 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:13.496 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:13.496 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:13.871 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:13.871 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:13.871 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236064 s, 44.4 MB/s 2021-02-10T11:32:13.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:13.872 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:13.872 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:14.208 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:14.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:14.208 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:14.585 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:14.585 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:14.585 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252382 s, 41.5 MB/s 2021-02-10T11:32:14.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:14.587 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:14.587 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:14.923 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:14.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:14.923 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802489/8d4b816a-7a0c-467a-a4a9-bee58e7376f1/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255618 s, 41.0 MB/s 2021-02-10T11:32:15.301 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802490 2021-02-10T11:32:15.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.727 7f9f17cb9700 1 Processor -- start 2021-02-10T11:32:15.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.727 7f9f17cb9700 1 -- start start 2021-02-10T11:32:15.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f17cb9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f100d1530 0x7f9f100d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f17cb9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f10077a70 con 0x7f9f1007da20 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f17cb9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f10079400 con 0x7f9f10079000 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f17cb9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1007de20 con 0x7f9f100d1530 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f15ad5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9f1007da20 0x7f9f100771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44894/0 (socket says 172.21.2.117:44894) 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f15ad5700 1 -- 172.21.2.117:0/4063982338 learned_addr learned my addr 172.21.2.117:0/4063982338 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018190485 0 0) 0x7f9f0c000ce0 con 0x7f9f1007da20 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f162d6700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f100d1530 0x7f9f100d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.728 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc002fe0 con 0x7f9f1007da20 2021-02-10T11:32:15.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3500126593 0 0) 0x7f9f0c0010e0 con 0x7f9f1007da20 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f100d1530 msgr2=0x7f9f100d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f100d1530 0x7f9f100d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 >> v1:172.21.2.125:6789/0 conn(0x7f9f10079000 legacy=0x7f9f100e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f101b00f0 con 0x7f9f1007da20 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f9f0c001660 con 0x7f9f1007da20 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 >> v1:172.21.2.117:6789/0 conn(0x7f9f1007da20 legacy=0x7f9f100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc003a60 0x7f9efc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc006790 0x7f9efc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc009590 0x7f9efc00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc00c2d0 con 0x7f9efc003a60 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc00c4f0 con 0x7f9efc006790 2021-02-10T11:32:15.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc00c710 con 0x7f9efc009590 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc003a60 0x7f9efc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f152d4700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc006790 0x7f9efc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f162d6700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc009590 0x7f9efc00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 -- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc009590 msgr2=0x7f9efc00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc009590 0x7f9efc00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 -- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc006790 msgr2=0x7f9efc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc006790 0x7f9efc008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.729 7f9f15ad5700 1 -- 172.21.2.117:0/4063982338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc050980 con 0x7f9efc003a60 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f15ad5700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc003a60 0x7f9efc005ef0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f9f0c0016f0 tx=0x7f9f0c007d00).ready entity=mon.0 client_cookie=a9246cc30831f40e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f0c0070c0 con 0x7f9efc003a60 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c0043a0 con 0x7f9efc003a60 2021-02-10T11:32:15.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f14ad3700 1 -- 172.21.2.117:0/4063982338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f0c004770 con 0x7f9efc003a60 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f17cb9700 1 -- 172.21.2.117:0/4063982338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc003a60 msgr2=0x7f9efc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f17cb9700 1 --2- 172.21.2.117:0/4063982338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc003a60 0x7f9efc005ef0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f9f0c0016f0 tx=0x7f9f0c007d00).stop 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f162d6700 1 -- 172.21.2.117:0/4063982338 reap_dead start 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f17cb9700 1 -- 172.21.2.117:0/4063982338 shutdown_connections 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.730 7f9f17cb9700 1 -- 172.21.2.117:0/4063982338 >> 172.21.2.117:0/4063982338 conn(0x7f9f100ce450 msgr2=0x7f9f100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.731 7f9f17cb9700 1 -- 172.21.2.117:0/4063982338 shutdown_connections 2021-02-10T11:32:15.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.731 7f9f17cb9700 1 -- 172.21.2.117:0/4063982338 wait complete. 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 Processor -- start 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 -- start start 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f101a46b0 0x7f9f10190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f100a46c0 0x7f9f100a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100c3040 con 0x7f9f100a50e0 2021-02-10T11:32:15.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f10190a20 con 0x7f9f100a46c0 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.745 7f9f17cb9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f100a54e0 con 0x7f9f101a46b0 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44654/0 (socket says 172.21.2.117:44654) 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 -- 172.21.2.117:0/2679348626 learned_addr learned my addr 172.21.2.117:0/2679348626 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 -- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f101a46b0 msgr2=0x7f9f10190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f101a46b0 0x7f9f10190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 -- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f100a46c0 msgr2=0x7f9f100a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f100a46c0 0x7f9f100a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 -- 172.21.2.117:0/2679348626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100a5700 con 0x7f9f100a50e0 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f162d6700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f9f0800b910 tx=0x7f9f0800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f080011f0 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f08006bf0 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f080017b0 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f100a5700 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.746 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f100a5700 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.747 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f080011f0 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.747 7f9f067fc700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 0x7f9ef801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.747 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9f08032b10 con 0x7f9f100a50e0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.748 7f9f15ad5700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 0x7f9ef801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:15.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.748 7f9f15ad5700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 0x7f9ef801abd0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:15.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.778 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f10077a70 con 0x7f9f100a50e0 2021-02-10T11:32:15.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.780 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f0800bdf0 con 0x7f9f100a50e0 2021-02-10T11:32:15.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.904 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802490", "target": ["mgr", ""]}) v1 -- 0x7f9f10077a70 con 0x7f9ef80186d0 2021-02-10T11:32:15.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.930 7f9f067fc700 1 -- 172.21.2.117:0/2679348626 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9f0c0074c0 con 0x7f9ef80186d0 2021-02-10T11:32:15.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.937 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 msgr2=0x7f9ef801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.937 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 0x7f9ef801abd0 crc :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.937 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 msgr2=0x7f9f100a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.937 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f9f0800b910 tx=0x7f9f0800bcb0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 shutdown_connections 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ef80186d0 0x7f9ef801abd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f100a50e0 0x7f9f100a6080 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f100a46c0 0x7f9f100a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 --2- 172.21.2.117:0/2679348626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f101a46b0 0x7f9f10190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 >> 172.21.2.117:0/2679348626 conn(0x7f9f100b5140 msgr2=0x7f9f100a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 shutdown_connections 2021-02-10T11:32:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:15.938 7f9f17cb9700 1 -- 172.21.2.117:0/2679348626 wait complete. 2021-02-10T11:32:15.934 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802490 2021-02-10T11:32:16.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.360 7f6e55e7b700 1 Processor -- start 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.360 7f6e55e7b700 1 -- start start 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e55e7b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e50079000 0x7f6e500e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e55e7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 0x7f6e501b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e55e7b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e50077a70 con 0x7f6e5007da20 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e55e7b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e500e53c0 con 0x7f6e50079000 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e55e7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50079400 con 0x7f6e501b00f0 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6e5007da20 0x7f6e500771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44912/0 (socket says 172.21.2.117:44912) 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4f7fe700 1 -- 172.21.2.117:0/2528541073 learned_addr learned my addr 172.21.2.117:0/2528541073 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4ffff700 1 --2- 172.21.2.117:0/2528541073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 0x7f6e501b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4effd700 1 --2- 172.21.2.117:0/2528541073 >> v2:172.21.2.125:3300/0 conn(0x7f6e50079000 0x7f6e500e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4e7fc700 1 -- 172.21.2.117:0/2528541073 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240518466 0 0) 0x7f6e40001180 con 0x7f6e5007da20 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4e7fc700 1 -- 172.21.2.117:0/2528541073 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e3c002fe0 con 0x7f6e5007da20 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4ffff700 1 -- 172.21.2.117:0/2528541073 >> v2:172.21.2.125:3300/0 conn(0x7f6e50079000 msgr2=0x7f6e500e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4ffff700 1 --2- 172.21.2.117:0/2528541073 >> v2:172.21.2.125:3300/0 conn(0x7f6e50079000 0x7f6e500e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4ffff700 1 -- 172.21.2.117:0/2528541073 >> v1:172.21.2.117:6789/0 conn(0x7f6e5007da20 legacy=0x7f6e500771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.361 7f6e4ffff700 1 -- 172.21.2.117:0/2528541073 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5007de20 con 0x7f6e501b00f0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e4ffff700 1 --2- 172.21.2.117:0/2528541073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 0x7f6e501b04f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6e48006e50 tx=0x7f6e48009e70).ready entity=mon.2 client_cookie=9331df9f91dba522 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e4e7fc700 1 -- 172.21.2.117:0/2528541073 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e48000f90 con 0x7f6e501b00f0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e4e7fc700 1 -- 172.21.2.117:0/2528541073 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e480066f0 con 0x7f6e501b00f0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e4e7fc700 1 -- 172.21.2.117:0/2528541073 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e4800b700 con 0x7f6e501b00f0 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 -- 172.21.2.117:0/2528541073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 msgr2=0x7f6e501b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 --2- 172.21.2.117:0/2528541073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 0x7f6e501b04f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f6e48006e50 tx=0x7f6e48009e70).stop 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 -- 172.21.2.117:0/2528541073 shutdown_connections 2021-02-10T11:32:16.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 --2- 172.21.2.117:0/2528541073 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b00f0 0x7f6e501b04f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 --2- 172.21.2.117:0/2528541073 >> v2:172.21.2.125:3300/0 conn(0x7f6e50079000 0x7f6e500e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 -- 172.21.2.117:0/2528541073 >> 172.21.2.117:0/2528541073 conn(0x7f6e500ce450 msgr2=0x7f6e500e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 -- 172.21.2.117:0/2528541073 shutdown_connections 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.362 7f6e55e7b700 1 -- 172.21.2.117:0/2528541073 wait complete. 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 Processor -- start 2021-02-10T11:32:16.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 -- start start 2021-02-10T11:32:16.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b5130 0x7f6e501b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e501b5f50 0x7f6e500b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e500b8da0 con 0x7f6e501b5b50 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e500a4140 con 0x7f6e501b5f50 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e55e7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e500a4320 con 0x7f6e501b5130 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e4effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e4effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44672/0 (socket says 172.21.2.117:44672) 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e4effd700 1 -- 172.21.2.117:0/1318357604 learned_addr learned my addr 172.21.2.117:0/1318357604 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.377 7f6e4ffff700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e501b5f50 0x7f6e500b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4f7fe700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b5130 0x7f6e501b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 -- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b5130 msgr2=0x7f6e501b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b5130 0x7f6e501b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 -- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e501b5f50 msgr2=0x7f6e500b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e501b5f50 0x7f6e500b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 -- 172.21.2.117:0/1318357604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e500b62b0 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e4effd700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f6e38006f60 tx=0x7f6e3800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e38006bb0 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38006340 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e38002030 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e500b62b0 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.378 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e500b6840 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.379 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e38006b50 con 0x7f6e501b5b50 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.379 7f6e47fff700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 0x7f6e3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:16.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.379 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e38032780 con 0x7f6e501b5b50 2021-02-10T11:32:16.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.379 7f6e4f7fe700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 0x7f6e3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:16.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.380 7f6e4f7fe700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 0x7f6e3401abc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:16.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.411 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e501b4f00 con 0x7f6e501b5b50 2021-02-10T11:32:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.413 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e38000f00 con 0x7f6e501b5b50 2021-02-10T11:32:16.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.540 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802490", "target": ["mgr", ""]}) v1 -- 0x7f6e500b8da0 con 0x7f6e34018730 2021-02-10T11:32:16.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.544 7f6e47fff700 1 -- 172.21.2.117:0/1318357604 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6e400076e0 con 0x7f6e34018730 2021-02-10T11:32:16.536 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844 2021-02-10T11:32:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 msgr2=0x7f6e3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 0x7f6e3401abc0 crc :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 msgr2=0x7f6e500b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f6e38006f60 tx=0x7f6e3800b970).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 shutdown_connections 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e34018730 0x7f6e3401abc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e501b5b50 0x7f6e500b5670 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e501b5f50 0x7f6e500b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 --2- 172.21.2.117:0/1318357604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e501b5130 0x7f6e501b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.549 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 >> 172.21.2.117:0/1318357604 conn(0x7f6e5008bbc0 msgr2=0x7f6e500c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:16.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.550 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 shutdown_connections 2021-02-10T11:32:16.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:16.550 7f6e55e7b700 1 -- 172.21.2.117:0/1318357604 wait complete. 2021-02-10T11:32:16.546 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802490 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844 2021-02-10T11:32:16.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:16.547 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:16.547 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:16.883 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:16.883 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:16.884 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:17.262 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:17.263 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225499 s, 46.5 MB/s 2021-02-10T11:32:17.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:17.264 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:17.264 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:17.600 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:17.601 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:17.601 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:17.973 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:17.974 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:17.974 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247647 s, 42.3 MB/s 2021-02-10T11:32:17.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:17.975 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:17.975 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:18.311 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:18.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:18.311 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:18.687 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:18.687 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:18.687 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246337 s, 42.6 MB/s 2021-02-10T11:32:18.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:18.688 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:18.688 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:19.024 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:19.024 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:19.025 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223908 s, 46.8 MB/s 2021-02-10T11:32:19.399 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:19.399 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:19.399 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:19.736 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:19.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:19.736 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247841 s, 42.3 MB/s 2021-02-10T11:32:20.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:20.114 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:20.114 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:20.448 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:20.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:20.449 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:20.823 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:20.823 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:20.823 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226602 s, 46.3 MB/s 2021-02-10T11:32:20.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:20.824 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:20.824 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:21.160 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:21.161 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:21.161 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231093 s, 45.4 MB/s 2021-02-10T11:32:21.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:21.537 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:21.538 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:21.874 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:21.875 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:21.875 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:22.249 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:22.249 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:22.250 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231486 s, 45.3 MB/s 2021-02-10T11:32:22.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:22.251 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:22.251 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:22.585 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:22.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:22.585 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:22.956 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:22.956 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:22.956 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234946 s, 44.6 MB/s 2021-02-10T11:32:22.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:22.957 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:22.958 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:23.294 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:23.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:23.295 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802490/590c7c1e-b4e4-46d1-b437-96de40fcd844/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:23.670 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:23.671 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:23.671 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239414 s, 43.8 MB/s 2021-02-10T11:32:23.672 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802491 2021-02-10T11:32:24.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.099 7f72d7623700 1 Processor -- start 2021-02-10T11:32:24.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d7623700 1 -- start start 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d7623700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72d007da20 0x7f72d00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d7623700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d00d1d20 con 0x7f72d00e51a0 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d7623700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d0077a70 con 0x7f72d00e4da0 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d7623700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d00777d0 con 0x7f72d007da20 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d543f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72d007da20 0x7f72d00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d5c40700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f72d00e51a0 0x7f72d00d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44926/0 (socket says 172.21.2.117:44926) 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d5c40700 1 -- 172.21.2.117:0/3548019909 learned_addr learned my addr 172.21.2.117:0/3548019909 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d543f700 1 -- 172.21.2.117:0/3548019909 >> v1:172.21.2.125:6789/0 conn(0x7f72d00e4da0 legacy=0x7f72d01b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d543f700 1 -- 172.21.2.117:0/3548019909 >> v1:172.21.2.117:6789/0 conn(0x7f72d00e51a0 legacy=0x7f72d00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.100 7f72d543f700 1 -- 172.21.2.117:0/3548019909 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d01b08e0 con 0x7f72d007da20 2021-02-10T11:32:24.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72d543f700 1 --2- 172.21.2.117:0/3548019909 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d007da20 0x7f72d00771b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f72c4006650 tx=0x7f72c4009a00).ready entity=mon.0 client_cookie=b4fb9b8f91582cd0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72c3fff700 1 -- 172.21.2.117:0/3548019909 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72c4009df0 con 0x7f72d007da20 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72c3fff700 1 -- 172.21.2.117:0/3548019909 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c40009b0 con 0x7f72d007da20 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72c3fff700 1 -- 172.21.2.117:0/3548019909 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72c400c2b0 con 0x7f72d007da20 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72d7623700 1 -- 172.21.2.117:0/3548019909 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d007da20 msgr2=0x7f72d00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.101 7f72d7623700 1 --2- 172.21.2.117:0/3548019909 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d007da20 0x7f72d00771b0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f72c4006650 tx=0x7f72c4009a00).stop 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.102 7f72d7623700 1 -- 172.21.2.117:0/3548019909 shutdown_connections 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.102 7f72d7623700 1 --2- 172.21.2.117:0/3548019909 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d007da20 0x7f72d00771b0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.102 7f72d7623700 1 -- 172.21.2.117:0/3548019909 >> 172.21.2.117:0/3548019909 conn(0x7f72d00ce450 msgr2=0x7f72d00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.102 7f72d7623700 1 -- 172.21.2.117:0/3548019909 shutdown_connections 2021-02-10T11:32:24.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.102 7f72d7623700 1 -- 172.21.2.117:0/3548019909 wait complete. 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.116 7f72d7623700 1 Processor -- start 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.116 7f72d7623700 1 -- start start 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72d00933f0 0x7f72d00937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72d0107050 0x7f72d0107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d0107a70 con 0x7f72d0093e10 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00c2c60 con 0x7f72d0107050 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d00c2e80 con 0x7f72d00933f0 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44682/0 (socket says 172.21.2.117:44682) 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 -- 172.21.2.117:0/344150671 learned_addr learned my addr 172.21.2.117:0/344150671 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 -- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72d00933f0 msgr2=0x7f72d00937f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72d00933f0 0x7f72d00937f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 -- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72d0107050 msgr2=0x7f72d0107450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72d0107050 0x7f72d0107450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 -- 172.21.2.117:0/344150671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d00c30a0 con 0x7f72d0093e10 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d4c3e700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f72cc00b5b0 tx=0x7f72cc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72cc00bd40 con 0x7f72d0093e10 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc006020 con 0x7f72d0093e10 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 -- 172.21.2.117:0/344150671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d00c30a0 con 0x7f72d0093e10 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72cc0054a0 con 0x7f72d0093e10 2021-02-10T11:32:24.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.117 7f72d7623700 1 -- 172.21.2.117:0/344150671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d00c3630 con 0x7f72d0093e10 2021-02-10T11:32:24.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.118 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72cc006200 con 0x7f72d0093e10 2021-02-10T11:32:24.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.118 7f72c1ffb700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 0x7f72b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.118 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f72cc032900 con 0x7f72d0093e10 2021-02-10T11:32:24.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.118 7f72d543f700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 0x7f72b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.119 7f72d543f700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 0x7f72b001ac10 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.150 7f72d7623700 1 -- 172.21.2.117:0/344150671 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d00ce450 con 0x7f72d0093e10 2021-02-10T11:32:24.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.153 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72cc000940 con 0x7f72d0093e10 2021-02-10T11:32:24.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.276 7f72d7623700 1 -- 172.21.2.117:0/344150671 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802491", "target": ["mgr", ""]}) v1 -- 0x7f72d00c3630 con 0x7f72b0018730 2021-02-10T11:32:24.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.301 7f72c1ffb700 1 -- 172.21.2.117:0/344150671 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f72c4000a00 con 0x7f72b0018730 2021-02-10T11:32:24.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.309 7f72d7623700 1 -- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 msgr2=0x7f72b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.309 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 0x7f72b001ac10 crc :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.309 7f72d7623700 1 -- 172.21.2.117:0/344150671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 msgr2=0x7f72d0106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.309 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f72cc00b5b0 tx=0x7f72cc00b950).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 -- 172.21.2.117:0/344150671 shutdown_connections 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72b0018730 0x7f72b001ac10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72d0093e10 0x7f72d0106a30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72d0107050 0x7f72d0107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 --2- 172.21.2.117:0/344150671 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72d00933f0 0x7f72d00937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 -- 172.21.2.117:0/344150671 >> 172.21.2.117:0/344150671 conn(0x7f72d01116a0 msgr2=0x7f72d00af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 -- 172.21.2.117:0/344150671 shutdown_connections 2021-02-10T11:32:24.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.310 7f72d7623700 1 -- 172.21.2.117:0/344150671 wait complete. 2021-02-10T11:32:24.306 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802491 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.733 7f1a70f5f700 1 Processor -- start 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.733 7f1a70f5f700 1 -- start start 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a6c07da20 0x7f1a6c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1a6c0e4da0 0x7f1a6c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c0e51a0 0x7f1a6c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c077a70 con 0x7f1a6c07da20 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6c0777d0 con 0x7f1a6c0e4da0 2021-02-10T11:32:24.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a70f5f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c07de20 con 0x7f1a6c0e51a0 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a6ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a6c07da20 0x7f1a6c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a6ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1a6c07da20 0x7f1a6c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44688/0 (socket says 172.21.2.117:44688) 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.734 7f1a6ab13700 1 -- 172.21.2.117:0/2413437398 learned_addr learned my addr 172.21.2.117:0/2413437398 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 -- 172.21.2.117:0/2413437398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c0e51a0 msgr2=0x7f1a6c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 --2- 172.21.2.117:0/2413437398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c0e51a0 0x7f1a6c077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 -- 172.21.2.117:0/2413437398 >> v2:172.21.2.125:3300/0 conn(0x7f1a6c0e4da0 msgr2=0x7f1a6c1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 --2- 172.21.2.117:0/2413437398 >> v2:172.21.2.125:3300/0 conn(0x7f1a6c0e4da0 0x7f1a6c1b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 -- 172.21.2.117:0/2413437398 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c1b0710 con 0x7f1a6c07da20 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a6ab13700 1 --2- 172.21.2.117:0/2413437398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c07da20 0x7f1a6c0771b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f1a5c006700 tx=0x7f1a5c009b80).ready entity=mon.0 client_cookie=b49fa077e5449127 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a69b11700 1 -- 172.21.2.117:0/2413437398 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5c00c0c0 con 0x7f1a6c07da20 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a69b11700 1 -- 172.21.2.117:0/2413437398 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c00c440 con 0x7f1a6c07da20 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a69b11700 1 -- 172.21.2.117:0/2413437398 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a5c00c960 con 0x7f1a6c07da20 2021-02-10T11:32:24.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a70f5f700 1 -- 172.21.2.117:0/2413437398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c07da20 msgr2=0x7f1a6c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.735 7f1a70f5f700 1 --2- 172.21.2.117:0/2413437398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c07da20 0x7f1a6c0771b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f1a5c006700 tx=0x7f1a5c009b80).stop 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 -- 172.21.2.117:0/2413437398 shutdown_connections 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 --2- 172.21.2.117:0/2413437398 >> v2:172.21.2.125:3300/0 conn(0x7f1a6c0e4da0 0x7f1a6c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 --2- 172.21.2.117:0/2413437398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c0e51a0 0x7f1a6c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 --2- 172.21.2.117:0/2413437398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c07da20 0x7f1a6c0771b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 -- 172.21.2.117:0/2413437398 >> 172.21.2.117:0/2413437398 conn(0x7f1a6c0ce450 msgr2=0x7f1a6c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 -- 172.21.2.117:0/2413437398 shutdown_connections 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.736 7f1a70f5f700 1 -- 172.21.2.117:0/2413437398 wait complete. 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.750 7f1a70f5f700 1 Processor -- start 2021-02-10T11:32:24.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 -- start start 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c19bf40 0x7f1a6c0bb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 0x7f1a6c08c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c091510 con 0x7f1a6c08c810 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c091a90 con 0x7f1a6c18f8e0 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a70f5f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c091c70 con 0x7f1a6c19bf40 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a6b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a6b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44690/0 (socket says 172.21.2.117:44690) 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a6b314700 1 -- 172.21.2.117:0/4020092755 learned_addr learned my addr 172.21.2.117:0/4020092755 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.751 7f1a6ab13700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c19bf40 0x7f1a6c0bb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6a312700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 0x7f1a6c08c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 -- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c19bf40 msgr2=0x7f1a6c0bb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c19bf40 0x7f1a6c0bb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 -- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 msgr2=0x7f1a6c08c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 0x7f1a6c08c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 -- 172.21.2.117:0/4020092755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c091e90 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6a312700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 0x7f1a6c08c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a6b314700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f1a64009fc0 tx=0x7f1a6400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a64000d10 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a64003020 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1a64005ad0 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c091e90 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.752 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c091e90 con 0x7f1a6c08c810 2021-02-10T11:32:24.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.753 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1a64000d10 con 0x7f1a6c08c810 2021-02-10T11:32:24.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.753 7f1a5b7fe700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 0x7f1a5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:24.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.753 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1a64032b40 con 0x7f1a6c08c810 2021-02-10T11:32:24.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.753 7f1a6ab13700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 0x7f1a5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:24.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.754 7f1a6ab13700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 0x7f1a5001ac10 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:24.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.784 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c103370 con 0x7f1a6c08c810 2021-02-10T11:32:24.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.787 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1a6400bd80 con 0x7f1a6c08c810 2021-02-10T11:32:24.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.914 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802491", "target": ["mgr", ""]}) v1 -- 0x7f1a6c103370 con 0x7f1a50018730 2021-02-10T11:32:24.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.917 7f1a5b7fe700 1 -- 172.21.2.117:0/4020092755 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1a5c000940 con 0x7f1a50018730 2021-02-10T11:32:24.906 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405 2021-02-10T11:32:24.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 msgr2=0x7f1a5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 0x7f1a5001ac10 crc :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 msgr2=0x7f1a6c08cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f1a64009fc0 tx=0x7f1a6400bc40).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 shutdown_connections 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1a50018730 0x7f1a5001ac10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a6c08c810 0x7f1a6c08cc10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a6c18f8e0 0x7f1a6c08c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 --2- 172.21.2.117:0/4020092755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a6c19bf40 0x7f1a6c0bb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 >> 172.21.2.117:0/4020092755 conn(0x7f1a6c0bef40 msgr2=0x7f1a6c09e080 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 shutdown_connections 2021-02-10T11:32:24.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:24.919 7f1a70f5f700 1 -- 172.21.2.117:0/4020092755 wait complete. 2021-02-10T11:32:24.916 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802491 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405 2021-02-10T11:32:24.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:24.916 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:24.917 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:25.257 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:25.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:25.258 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:25.640 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:25.640 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:25.640 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239412 s, 43.8 MB/s 2021-02-10T11:32:25.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:25.641 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:25.641 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:25.984 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:25.984 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:25.985 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:26.361 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:26.361 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:26.361 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230463 s, 45.5 MB/s 2021-02-10T11:32:26.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:26.362 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:26.362 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:26.697 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:26.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:26.698 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:27.073 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:27.074 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:27.074 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243627 s, 43.0 MB/s 2021-02-10T11:32:27.075 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:27.075 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:27.075 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:27.412 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:27.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:27.413 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:27.786 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:27.786 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:27.786 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252018 s, 41.6 MB/s 2021-02-10T11:32:27.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:27.787 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:27.787 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:28.122 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:28.122 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:28.123 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:28.500 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:28.500 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:28.500 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251337 s, 41.7 MB/s 2021-02-10T11:32:28.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:28.501 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:28.501 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:28.835 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:28.835 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:28.836 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:29.211 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:29.211 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:29.211 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240085 s, 43.7 MB/s 2021-02-10T11:32:29.212 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:29.212 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:29.212 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:29.548 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:29.548 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:29.548 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:29.621 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:29.625 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:32:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0265484 s, 39.5 MB/s 2021-02-10T11:32:29.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:29.929 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:29.929 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:30.264 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:30.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:30.265 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:30.644 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:30.644 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:30.644 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0263804 s, 39.7 MB/s 2021-02-10T11:32:30.645 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:30.645 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:30.645 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:30.980 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:30.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:30.981 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:31.354 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:31.355 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:31.355 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247503 s, 42.4 MB/s 2021-02-10T11:32:31.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:31.356 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:31.356 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:31.691 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:31.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:31.692 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802491/19e4aca3-451d-45f3-8114-3bbfdb819405/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:32.070 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:32.070 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245238 s, 42.8 MB/s 2021-02-10T11:32:32.071 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802492 2021-02-10T11:32:32.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.500 7f4a38676700 1 Processor -- start 2021-02-10T11:32:32.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 -- start start 2021-02-10T11:32:32.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a30079000 0x7f4a300e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4a301b00f0 0x7f4a301b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a30077a70 con 0x7f4a3007da20 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a300e5c20 con 0x7f4a30079000 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a38676700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a300e53c0 con 0x7f4a301b00f0 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a36492700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4a3007da20 0x7f4a300771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44948/0 (socket says 172.21.2.117:44948) 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a36492700 1 -- 172.21.2.117:0/937592592 learned_addr learned my addr 172.21.2.117:0/937592592 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a35c91700 1 --2- 172.21.2.117:0/937592592 >> v2:172.21.2.117:3300/0 conn(0x7f4a30079000 0x7f4a300e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.501 7f4a36c93700 1 --2- 172.21.2.117:0/937592592 >> v2:172.21.2.125:3300/0 conn(0x7f4a301b00f0 0x7f4a301b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35490700 1 -- 172.21.2.117:0/937592592 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601380266 0 0) 0x7f4a28001500 con 0x7f4a3007da20 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35490700 1 -- 172.21.2.117:0/937592592 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a1c002fe0 con 0x7f4a3007da20 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35c91700 1 -- 172.21.2.117:0/937592592 >> v2:172.21.2.125:3300/0 conn(0x7f4a301b00f0 msgr2=0x7f4a301b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35c91700 1 --2- 172.21.2.117:0/937592592 >> v2:172.21.2.125:3300/0 conn(0x7f4a301b00f0 0x7f4a301b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35c91700 1 -- 172.21.2.117:0/937592592 >> v1:172.21.2.117:6789/0 conn(0x7f4a3007da20 legacy=0x7f4a300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35c91700 1 -- 172.21.2.117:0/937592592 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30079400 con 0x7f4a30079000 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35c91700 1 --2- 172.21.2.117:0/937592592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30079000 0x7f4a300e4da0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f4a20006650 tx=0x7f4a20009a80).ready entity=mon.0 client_cookie=3ffc56c79f5a32bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35490700 1 -- 172.21.2.117:0/937592592 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2000c0c0 con 0x7f4a30079000 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35490700 1 -- 172.21.2.117:0/937592592 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2000c340 con 0x7f4a30079000 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a35490700 1 -- 172.21.2.117:0/937592592 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2000c860 con 0x7f4a30079000 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a38676700 1 -- 172.21.2.117:0/937592592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30079000 msgr2=0x7f4a300e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.502 7f4a38676700 1 --2- 172.21.2.117:0/937592592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30079000 0x7f4a300e4da0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f4a20006650 tx=0x7f4a20009a80).stop 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 -- 172.21.2.117:0/937592592 shutdown_connections 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 --2- 172.21.2.117:0/937592592 >> v2:172.21.2.125:3300/0 conn(0x7f4a301b00f0 0x7f4a301b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 --2- 172.21.2.117:0/937592592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30079000 0x7f4a300e4da0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 -- 172.21.2.117:0/937592592 >> 172.21.2.117:0/937592592 conn(0x7f4a300ce450 msgr2=0x7f4a300e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 -- 172.21.2.117:0/937592592 shutdown_connections 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.503 7f4a38676700 1 -- 172.21.2.117:0/937592592 wait complete. 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 Processor -- start 2021-02-10T11:32:32.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 -- start start 2021-02-10T11:32:32.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a300b7fc0 0x7f4a301042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a301048f0 0x7f4a30104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a301054f0 con 0x7f4a30090710 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30195a30 con 0x7f4a300b7fc0 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a38676700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a30195fb0 con 0x7f4a301048f0 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a36c93700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a36c93700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44708/0 (socket says 172.21.2.117:44708) 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a36c93700 1 -- 172.21.2.117:0/1739957851 learned_addr learned my addr 172.21.2.117:0/1739957851 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a35c91700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a301048f0 0x7f4a30104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.518 7f4a36492700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a300b7fc0 0x7f4a301042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 -- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a301048f0 msgr2=0x7f4a30104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a301048f0 0x7f4a30104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 -- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a300b7fc0 msgr2=0x7f4a301042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a300b7fc0 0x7f4a301042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 -- 172.21.2.117:0/1739957851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a300b7b60 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a36c93700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f4a2c00b5e0 tx=0x7f4a2c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2c000c10 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00bd70 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a2c0024d0 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a38676700 1 -- 172.21.2.117:0/1739957851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a300b7b60 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a38676700 1 -- 172.21.2.117:0/1739957851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a30091420 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.519 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a2c006720 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.520 7f4a26ffd700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 0x7f4a1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.520 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a2c032730 con 0x7f4a30090710 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.520 7f4a36492700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 0x7f4a1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:32.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.520 7f4a36492700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 0x7f4a1801abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:32.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.550 7f4a38676700 1 -- 172.21.2.117:0/1739957851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a30103280 con 0x7f4a30090710 2021-02-10T11:32:32.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.553 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a2c006370 con 0x7f4a30090710 2021-02-10T11:32:32.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.680 7f4a38676700 1 -- 172.21.2.117:0/1739957851 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802492", "target": ["mgr", ""]}) v1 -- 0x7f4a30103280 con 0x7f4a180186c0 2021-02-10T11:32:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.703 7f4a26ffd700 1 -- 172.21.2.117:0/1739957851 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4a28001060 con 0x7f4a180186c0 2021-02-10T11:32:32.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 -- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 msgr2=0x7f4a1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 0x7f4a1801abc0 crc :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 -- 172.21.2.117:0/1739957851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 msgr2=0x7f4a30090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f4a2c00b5e0 tx=0x7f4a2c00b980).stop 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 -- 172.21.2.117:0/1739957851 shutdown_connections 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a180186c0 0x7f4a1801abc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a30090710 0x7f4a30090b10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a301048f0 0x7f4a30104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 --2- 172.21.2.117:0/1739957851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a300b7fc0 0x7f4a301042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 -- 172.21.2.117:0/1739957851 >> 172.21.2.117:0/1739957851 conn(0x7f4a300a1850 msgr2=0x7f4a300c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.713 7f4a38676700 1 -- 172.21.2.117:0/1739957851 shutdown_connections 2021-02-10T11:32:32.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:32.714 7f4a38676700 1 -- 172.21.2.117:0/1739957851 wait complete. 2021-02-10T11:32:32.710 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802492 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.135 7f35b227e700 1 Processor -- start 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.135 7f35b227e700 1 -- start start 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b227e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac07da20 0x7f35ac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b227e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35ac0e4da0 0x7f35ac1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b227e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac077a70 con 0x7f35ac0e51a0 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b227e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac0e5c20 con 0x7f35ac0e4da0 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b227e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0777d0 con 0x7f35ac07da20 2021-02-10T11:32:33.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35ac0e4da0 0x7f35ac1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b089b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f35ac0e51a0 0x7f35ac077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44962/0 (socket says 172.21.2.117:44962) 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35ac0e4da0 0x7f35ac1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44712/0 (socket says 172.21.2.117:44712) 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35b089b700 1 -- 172.21.2.117:0/1206287439 learned_addr learned my addr 172.21.2.117:0/1206287439 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 -- 172.21.2.117:0/1206287439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac07da20 msgr2=0x7f35ac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 --2- 172.21.2.117:0/1206287439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac07da20 0x7f35ac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 -- 172.21.2.117:0/1206287439 >> v1:172.21.2.117:6789/0 conn(0x7f35ac0e51a0 legacy=0x7f35ac077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.136 7f35ab7fe700 1 -- 172.21.2.117:0/1206287439 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac07de20 con 0x7f35ac0e4da0 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35ab7fe700 1 --2- 172.21.2.117:0/1206287439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac0e4da0 0x7f35ac1b00f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f35a0006650 tx=0x7f35a0009a80).ready entity=mon.0 client_cookie=3b9496f942361109 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35aaffd700 1 -- 172.21.2.117:0/1206287439 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35a00009b0 con 0x7f35ac0e4da0 2021-02-10T11:32:33.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35aaffd700 1 -- 172.21.2.117:0/1206287439 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a000c0c0 con 0x7f35ac0e4da0 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35aaffd700 1 -- 172.21.2.117:0/1206287439 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35a000c5e0 con 0x7f35ac0e4da0 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35b227e700 1 -- 172.21.2.117:0/1206287439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac0e4da0 msgr2=0x7f35ac1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35b227e700 1 --2- 172.21.2.117:0/1206287439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac0e4da0 0x7f35ac1b00f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f35a0006650 tx=0x7f35a0009a80).stop 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35b227e700 1 -- 172.21.2.117:0/1206287439 shutdown_connections 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35b227e700 1 --2- 172.21.2.117:0/1206287439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac0e4da0 0x7f35ac1b00f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.137 7f35b227e700 1 --2- 172.21.2.117:0/1206287439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac07da20 0x7f35ac0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.138 7f35b227e700 1 -- 172.21.2.117:0/1206287439 >> 172.21.2.117:0/1206287439 conn(0x7f35ac0ce450 msgr2=0x7f35ac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.138 7f35b227e700 1 -- 172.21.2.117:0/1206287439 shutdown_connections 2021-02-10T11:32:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.138 7f35b227e700 1 -- 172.21.2.117:0/1206287439 wait complete. 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.152 7f35b227e700 1 Processor -- start 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.152 7f35b227e700 1 -- start start 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35ac0aa220 0x7f35ac0aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac0aac40 0x7f35ac0ab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0a9ca0 con 0x7f35ac18f490 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0b7610 con 0x7f35ac0aa220 2021-02-10T11:32:33.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac0b77f0 con 0x7f35ac0aac40 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44718/0 (socket says 172.21.2.117:44718) 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 -- 172.21.2.117:0/3208782630 learned_addr learned my addr 172.21.2.117:0/3208782630 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 -- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac0aac40 msgr2=0x7f35ac0ab040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac0aac40 0x7f35ac0ab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 -- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35ac0aa220 msgr2=0x7f35ac0aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35ac0aa220 0x7f35ac0aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 -- 172.21.2.117:0/3208782630 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0a9890 con 0x7f35ac18f490 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b089b700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f35a400a940 tx=0x7f35a400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35a4000dc0 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a40070b0 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35a4007540 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 -- 172.21.2.117:0/3208782630 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac0a9890 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.153 7f35b227e700 1 -- 172.21.2.117:0/3208782630 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac0a9890 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.154 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35a4007290 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.154 7f35a8ff9700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 0x7f358c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.154 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f35a4032ca0 con 0x7f35ac18f490 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.155 7f35abfff700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 0x7f358c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.155 7f35abfff700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 0x7f358c01abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:33.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.186 7f35b227e700 1 -- 172.21.2.117:0/3208782630 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac103330 con 0x7f35ac18f490 2021-02-10T11:32:33.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.188 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35a4007cb0 con 0x7f35ac18f490 2021-02-10T11:32:33.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.315 7f35b227e700 1 -- 172.21.2.117:0/3208782630 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802492", "target": ["mgr", ""]}) v1 -- 0x7f35ac077a70 con 0x7f358c0186c0 2021-02-10T11:32:33.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.319 7f35a8ff9700 1 -- 172.21.2.117:0/3208782630 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f359c006a10 con 0x7f358c0186c0 2021-02-10T11:32:33.311 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 -- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 msgr2=0x7f358c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 0x7f358c01abc0 crc :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 -- 172.21.2.117:0/3208782630 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 msgr2=0x7f35ac18f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f35a400a940 tx=0x7f35a400ace0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 -- 172.21.2.117:0/3208782630 shutdown_connections 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f358c0186c0 0x7f358c01abc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35ac18f490 0x7f35ac18f890 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35ac0aac40 0x7f35ac0ab040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 --2- 172.21.2.117:0/3208782630 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35ac0aa220 0x7f35ac0aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:33.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.324 7f35b227e700 1 -- 172.21.2.117:0/3208782630 >> 172.21.2.117:0/3208782630 conn(0x7f35ac0b56c0 msgr2=0x7f35ac0a8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:33.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.325 7f35b227e700 1 -- 172.21.2.117:0/3208782630 shutdown_connections 2021-02-10T11:32:33.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:33.325 7f35b227e700 1 -- 172.21.2.117:0/3208782630 wait complete. 2021-02-10T11:32:33.321 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802492 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28 2021-02-10T11:32:33.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:33.322 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:33.322 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:33.658 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:33.658 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:33.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:34.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:34.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:34.039 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236589 s, 44.3 MB/s 2021-02-10T11:32:34.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:34.040 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:34.040 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:34.375 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:34.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:34.376 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:34.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:34.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:34.748 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224532 s, 46.7 MB/s 2021-02-10T11:32:34.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:34.750 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:34.750 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:35.086 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:35.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:35.086 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:35.460 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:35.461 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:35.461 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247752 s, 42.3 MB/s 2021-02-10T11:32:35.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:35.462 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:35.462 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:35.799 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:35.799 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:35.799 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:36.176 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:36.176 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:36.176 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252081 s, 41.6 MB/s 2021-02-10T11:32:36.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:36.177 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:36.177 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:36.513 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:36.513 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:36.514 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:36.890 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:36.890 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:36.890 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245417 s, 42.7 MB/s 2021-02-10T11:32:36.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:36.892 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:36.892 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:37.228 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:37.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:37.228 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257653 s, 40.7 MB/s 2021-02-10T11:32:37.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:37.604 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:37.604 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:37.938 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:37.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:37.938 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258783 s, 40.5 MB/s 2021-02-10T11:32:38.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:38.316 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:38.316 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:38.652 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:38.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:38.653 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:39.028 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024081 s, 43.5 MB/s 2021-02-10T11:32:39.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:39.030 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:39.030 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:39.366 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:39.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:39.367 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:39.743 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:39.743 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:39.743 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245657 s, 42.7 MB/s 2021-02-10T11:32:39.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:39.744 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:39.744 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:40.078 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:40.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:40.079 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802492/eb82341d-9b23-41a4-a330-f2446e296f28/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:40.451 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:40.451 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:40.452 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227449 s, 46.1 MB/s 2021-02-10T11:32:40.453 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802493 2021-02-10T11:32:40.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.880 7ff9338af700 1 Processor -- start 2021-02-10T11:32:40.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.880 7ff9338af700 1 -- start start 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff9338af700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c07da20 0x7ff92c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff9338af700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0d1d20 con 0x7ff92c0e51a0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff9338af700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c077a70 con 0x7ff92c0e4da0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff9338af700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0777d0 con 0x7ff92c07da20 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff931ecc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff92c0e51a0 0x7ff92c0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:44972/0 (socket says 172.21.2.117:44972) 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff931ecc700 1 -- 172.21.2.117:0/4237727780 learned_addr learned my addr 172.21.2.117:0/4237727780 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224690095 0 0) 0x7ff924001500 con 0x7ff92c0e51a0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918002fe0 con 0x7ff92c0e51a0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.881 7ff9316cb700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c07da20 0x7ff92c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3352391601 0 0) 0x7ff924001460 con 0x7ff92c0e51a0 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c07da20 msgr2=0x7ff92c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:40.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c07da20 0x7ff92c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 >> v1:172.21.2.125:6789/0 conn(0x7ff92c0e4da0 legacy=0x7ff92c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c1b08e0 con 0x7ff92c0e51a0 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff924001980 con 0x7ff92c0e51a0 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 >> v1:172.21.2.117:6789/0 conn(0x7ff92c0e51a0 legacy=0x7ff92c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff918003a70 0x7ff918005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff918006740 0x7ff918008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff918009500 0x7ff91800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800c2d0 con 0x7ff918009500 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800c4f0 con 0x7ff918006740 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800c710 con 0x7ff918003a70 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff918009500 0x7ff91800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff9316cb700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff918003a70 0x7ff918005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff930eca700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff918006740 0x7ff918008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 -- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff918003a70 msgr2=0x7ff918005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff918003a70 0x7ff918005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 -- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff918006740 msgr2=0x7ff918008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff918006740 0x7ff918008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 -- 172.21.2.117:0/4237727780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91800c9b0 con 0x7ff918009500 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.882 7ff931ecc700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff918009500 0x7ff91800ba20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7ff924001140 tx=0x7ff92404e320).ready entity=mon.0 client_cookie=a567fb3f85a2d5b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff924001460 con 0x7ff918009500 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9240070c0 con 0x7ff918009500 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff91ffff700 1 -- 172.21.2.117:0/4237727780 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff924004720 con 0x7ff918009500 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 -- 172.21.2.117:0/4237727780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff918009500 msgr2=0x7ff91800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 --2- 172.21.2.117:0/4237727780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff918009500 0x7ff91800ba20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7ff924001140 tx=0x7ff92404e320).stop 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff931ecc700 1 -- 172.21.2.117:0/4237727780 reap_dead start 2021-02-10T11:32:40.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 -- 172.21.2.117:0/4237727780 shutdown_connections 2021-02-10T11:32:40.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 -- 172.21.2.117:0/4237727780 >> 172.21.2.117:0/4237727780 conn(0x7ff92c0ce450 msgr2=0x7ff92c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:40.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 -- 172.21.2.117:0/4237727780 shutdown_connections 2021-02-10T11:32:40.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.883 7ff9338af700 1 -- 172.21.2.117:0/4237727780 wait complete. 2021-02-10T11:32:40.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.898 7ff9338af700 1 Processor -- start 2021-02-10T11:32:40.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.898 7ff9338af700 1 -- start start 2021-02-10T11:32:40.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c0a47d0 0x7ff92c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff92c0a51f0 0x7ff92c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c189970 con 0x7ff92c0b77d0 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c190aa0 con 0x7ff92c0a51f0 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff9338af700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b81f0 con 0x7ff92c0a47d0 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44736/0 (socket says 172.21.2.117:44736) 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 -- 172.21.2.117:0/2727485367 learned_addr learned my addr 172.21.2.117:0/2727485367 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 -- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c0a47d0 msgr2=0x7ff92c0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c0a47d0 0x7ff92c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 -- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff92c0a51f0 msgr2=0x7ff92c0a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff92c0a51f0 0x7ff92c0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 -- 172.21.2.117:0/2727485367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0b8410 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.899 7ff931ecc700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7ff92404f8d0 tx=0x7ff92404fc70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff924007da0 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff924007020 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff924007540 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff9338af700 1 -- 172.21.2.117:0/2727485367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c0b8410 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff9338af700 1 -- 172.21.2.117:0/2727485367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c0b89a0 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.900 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff92404e020 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.901 7ff91dffb700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 0x7ff90c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.901 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff924076a60 con 0x7ff92c0b77d0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.901 7ff9316cb700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 0x7ff90c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:40.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.902 7ff9316cb700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 0x7ff90c01abc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:40.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.931 7ff9338af700 1 -- 172.21.2.117:0/2727485367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c077a70 con 0x7ff92c0b77d0 2021-02-10T11:32:40.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:40.933 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9240072c0 con 0x7ff92c0b77d0 2021-02-10T11:32:41.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.058 7ff9338af700 1 -- 172.21.2.117:0/2727485367 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802493", "target": ["mgr", ""]}) v1 -- 0x7ff92c189970 con 0x7ff90c0186c0 2021-02-10T11:32:41.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.083 7ff91dffb700 1 -- 172.21.2.117:0/2727485367 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff928007df0 con 0x7ff90c0186c0 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 msgr2=0x7ff90c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 0x7ff90c01abc0 crc :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 msgr2=0x7ff92c0b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7ff92404f8d0 tx=0x7ff92404fc70).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 shutdown_connections 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff90c0186c0 0x7ff90c01abc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff92c0b77d0 0x7ff92c0b7bd0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff92c0a51f0 0x7ff92c0a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 --2- 172.21.2.117:0/2727485367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff92c0a47d0 0x7ff92c0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 >> 172.21.2.117:0/2727485367 conn(0x7ff92c0b5140 msgr2=0x7ff92c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 shutdown_connections 2021-02-10T11:32:41.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.091 7ff9338af700 1 -- 172.21.2.117:0/2727485367 wait complete. 2021-02-10T11:32:41.088 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802493 2021-02-10T11:32:41.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.516 7f8742ff3700 1 Processor -- start 2021-02-10T11:32:41.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.516 7f8742ff3700 1 -- start start 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c07da20 0x7f873c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f873c0e51a0 0x7f873c0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c077a70 con 0x7f873c0e4da0 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c0e5c20 con 0x7f873c0e51a0 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0777d0 con 0x7f873c07da20 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f873c0e51a0 0x7f873c0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f873c0e51a0 0x7f873c0d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44742/0 (socket says 172.21.2.117:44742) 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 -- 172.21.2.117:0/742474114 learned_addr learned my addr 172.21.2.117:0/742474114 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 -- 172.21.2.117:0/742474114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c07da20 msgr2=0x7f873c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 --2- 172.21.2.117:0/742474114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c07da20 0x7f873c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 -- 172.21.2.117:0/742474114 >> v1:172.21.2.125:6789/0 conn(0x7f873c0e4da0 legacy=0x7f873c1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 -- 172.21.2.117:0/742474114 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c1b08e0 con 0x7f873c0e51a0 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8741610700 1 --2- 172.21.2.117:0/742474114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0e51a0 0x7f873c0d0520 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8734006f00 tx=0x7f8734009f10).ready entity=mon.0 client_cookie=950d6b31f3bdf6e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f873b7fe700 1 -- 172.21.2.117:0/742474114 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873400b3b0 con 0x7f873c0e51a0 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f873b7fe700 1 -- 172.21.2.117:0/742474114 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873400b730 con 0x7f873c0e51a0 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f873b7fe700 1 -- 172.21.2.117:0/742474114 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873400bc10 con 0x7f873c0e51a0 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 -- 172.21.2.117:0/742474114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0e51a0 msgr2=0x7f873c0d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.517 7f8742ff3700 1 --2- 172.21.2.117:0/742474114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0e51a0 0x7f873c0d0520 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8734006f00 tx=0x7f8734009f10).stop 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 -- 172.21.2.117:0/742474114 shutdown_connections 2021-02-10T11:32:41.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 --2- 172.21.2.117:0/742474114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0e51a0 0x7f873c0d0520 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 --2- 172.21.2.117:0/742474114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c07da20 0x7f873c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 -- 172.21.2.117:0/742474114 >> 172.21.2.117:0/742474114 conn(0x7f873c0ce450 msgr2=0x7f873c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 -- 172.21.2.117:0/742474114 shutdown_connections 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.518 7f8742ff3700 1 -- 172.21.2.117:0/742474114 wait complete. 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.532 7f8742ff3700 1 Processor -- start 2021-02-10T11:32:41.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.532 7f8742ff3700 1 -- start start 2021-02-10T11:32:41.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873c1a0500 0x7f873c0be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c0a4ac0 0x7f873c0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0a6cb0 con 0x7f873c0a46c0 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0be980 con 0x7f873c1a0500 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f8742ff3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0a7230 con 0x7f873c0a4ac0 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44746/0 (socket says 172.21.2.117:44746) 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 -- 172.21.2.117:0/2904289575 learned_addr learned my addr 172.21.2.117:0/2904289575 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 -- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c0a4ac0 msgr2=0x7f873c0b3f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c0a4ac0 0x7f873c0b3f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 -- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873c1a0500 msgr2=0x7f873c0be360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:32:41.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873c1a0500 0x7f873c0be360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.533 7f873bfff700 1 -- 172.21.2.117:0/2904289575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0a7450 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f873bfff700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f872c00b5b0 tx=0x7f872c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f872c00bd40 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c006240 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f872c005720 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c0a7450 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0a7450 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f872c0054a0 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.534 7f87397fa700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 0x7f872001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.535 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f872c0327c0 con 0x7f873c0a46c0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.536 7f8740e0f700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 0x7f872001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:41.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.537 7f8740e0f700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 0x7f872001abd0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:41.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.566 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c0ff170 con 0x7f873c0a46c0 2021-02-10T11:32:41.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.569 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f872c006030 con 0x7f873c0a46c0 2021-02-10T11:32:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.696 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802493", "target": ["mgr", ""]}) v1 -- 0x7f873c077a70 con 0x7f87200186d0 2021-02-10T11:32:41.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.701 7f87397fa700 1 -- 172.21.2.117:0/2904289575 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8724006990 con 0x7f87200186d0 2021-02-10T11:32:41.691 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2 2021-02-10T11:32:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 msgr2=0x7f872001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 0x7f872001abd0 crc :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 msgr2=0x7f873c0b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:41.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f872c00b5b0 tx=0x7f872c00b950).stop 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 shutdown_connections 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87200186d0 0x7f872001abd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f873c0a46c0 0x7f873c0b3940 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873c0a4ac0 0x7f873c0b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 --2- 172.21.2.117:0/2904289575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873c1a0500 0x7f873c0be360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 >> 172.21.2.117:0/2904289575 conn(0x7f873c09d3b0 msgr2=0x7f873c0b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.705 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 shutdown_connections 2021-02-10T11:32:41.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:41.706 7f8742ff3700 1 -- 172.21.2.117:0/2904289575 wait complete. 2021-02-10T11:32:41.702 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802493 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2 2021-02-10T11:32:41.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:41.702 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:41.702 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:42.038 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:42.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:42.039 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:42.421 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:42.421 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:42.421 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247255 s, 42.4 MB/s 2021-02-10T11:32:42.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:42.422 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:42.422 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:42.757 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:42.757 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:42.757 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:43.131 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:43.131 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:43.131 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250052 s, 41.9 MB/s 2021-02-10T11:32:43.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:43.133 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:43.133 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:43.469 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:43.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:43.470 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:43.846 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:43.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:43.847 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024747 s, 42.4 MB/s 2021-02-10T11:32:43.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:43.848 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:43.848 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:44.183 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:44.183 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:44.183 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:44.559 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:44.559 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:44.559 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02478 s, 42.3 MB/s 2021-02-10T11:32:44.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:44.560 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:44.561 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:44.897 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:44.897 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:44.897 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:45.271 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:45.272 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:45.272 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226351 s, 46.3 MB/s 2021-02-10T11:32:45.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:45.273 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:45.273 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:45.610 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:45.610 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:45.610 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:45.988 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:45.988 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:45.988 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244486 s, 42.9 MB/s 2021-02-10T11:32:45.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:45.989 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:45.989 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:46.326 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:46.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:46.327 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228547 s, 45.9 MB/s 2021-02-10T11:32:46.701 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:46.701 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:46.702 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:47.037 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:47.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:47.038 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:47.414 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:47.415 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:47.415 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024148 s, 43.4 MB/s 2021-02-10T11:32:47.416 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:47.416 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:47.416 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:47.750 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:47.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:47.750 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:48.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:48.125 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:48.125 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232367 s, 45.1 MB/s 2021-02-10T11:32:48.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:48.126 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:48.126 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:48.462 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:48.462 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:48.462 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802493/1c1a3f6e-eb93-4cfb-bb1b-271e1ab808f2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:48.837 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:48.837 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:48.837 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225136 s, 46.6 MB/s 2021-02-10T11:32:48.838 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802494 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 Processor -- start 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 -- start start 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f573c07da20 0x7f573c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f573c0e4da0 0x7f573c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c077a70 con 0x7f573c0e51a0 2021-02-10T11:32:49.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0777d0 con 0x7f573c07da20 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.268 7f5744552700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c07de20 con 0x7f573c0e4da0 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f5742b6f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f573c0e51a0 0x7f573c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45008/0 (socket says 172.21.2.117:45008) 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f5741b6d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f573c0e4da0 0x7f573c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f573c07da20 0x7f573c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f5742b6f700 1 -- 172.21.2.117:0/1113547821 learned_addr learned my addr 172.21.2.117:0/1113547821 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 -- 172.21.2.117:0/1113547821 >> v2:172.21.2.125:3300/0 conn(0x7f573c0e4da0 msgr2=0x7f573c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 --2- 172.21.2.117:0/1113547821 >> v2:172.21.2.125:3300/0 conn(0x7f573c0e4da0 0x7f573c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 -- 172.21.2.117:0/1113547821 >> v1:172.21.2.117:6789/0 conn(0x7f573c0e51a0 legacy=0x7f573c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 -- 172.21.2.117:0/1113547821 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c1b0710 con 0x7f573c07da20 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574236e700 1 --2- 172.21.2.117:0/1113547821 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c07da20 0x7f573c0771b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f5738006650 tx=0x7f5738009a80).ready entity=mon.0 client_cookie=5eacbc484248ee2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574136c700 1 -- 172.21.2.117:0/1113547821 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57380009b0 con 0x7f573c07da20 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574136c700 1 -- 172.21.2.117:0/1113547821 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800c0c0 con 0x7f573c07da20 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f574136c700 1 -- 172.21.2.117:0/1113547821 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f573800c510 con 0x7f573c07da20 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f5744552700 1 -- 172.21.2.117:0/1113547821 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c07da20 msgr2=0x7f573c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.269 7f5744552700 1 --2- 172.21.2.117:0/1113547821 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c07da20 0x7f573c0771b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f5738006650 tx=0x7f5738009a80).stop 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 -- 172.21.2.117:0/1113547821 shutdown_connections 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 --2- 172.21.2.117:0/1113547821 >> v2:172.21.2.125:3300/0 conn(0x7f573c0e4da0 0x7f573c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 --2- 172.21.2.117:0/1113547821 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c07da20 0x7f573c0771b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 -- 172.21.2.117:0/1113547821 >> 172.21.2.117:0/1113547821 conn(0x7f573c0ce450 msgr2=0x7f573c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 -- 172.21.2.117:0/1113547821 shutdown_connections 2021-02-10T11:32:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.270 7f5744552700 1 -- 172.21.2.117:0/1113547821 wait complete. 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.285 7f5744552700 1 Processor -- start 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.285 7f5744552700 1 -- start start 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f573c0c2340 0x7f573c0c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f573c0b7390 0x7f573c0b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b8010 con 0x7f573c0a41c0 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b8550 con 0x7f573c0b7390 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5744552700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0b8770 con 0x7f573c0c2340 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44764/0 (socket says 172.21.2.117:44764) 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 -- 172.21.2.117:0/3195042669 learned_addr learned my addr 172.21.2.117:0/3195042669 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5741b6d700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f573c0c2340 0x7f573c0c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f5742b6f700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f573c0b7390 0x7f573c0b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 -- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f573c0c2340 msgr2=0x7f573c0c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f573c0c2340 0x7f573c0c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 -- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f573c0b7390 msgr2=0x7f573c0b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f573c0b7390 0x7f573c0b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.286 7f574236e700 1 -- 172.21.2.117:0/3195042669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0b8990 con 0x7f573c0a41c0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f574236e700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f5738009650 tx=0x7f573800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5738009360 con 0x7f573c0a41c0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573800bde0 con 0x7f573c0a41c0 2021-02-10T11:32:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57380052e0 con 0x7f573c0a41c0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f5744552700 1 -- 172.21.2.117:0/3195042669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0b8990 con 0x7f573c0a41c0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f5744552700 1 -- 172.21.2.117:0/3195042669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c0b8f20 con 0x7f573c0a41c0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.287 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5738006030 con 0x7f573c0a41c0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.288 7f572effd700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 0x7f572401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.288 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5738032510 con 0x7f573c0a41c0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.288 7f5741b6d700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 0x7f572401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.288 7f5741b6d700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 0x7f572401abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.317 7f5744552700 1 -- 172.21.2.117:0/3195042669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c1b4f00 con 0x7f573c0a41c0 2021-02-10T11:32:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.320 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5738009360 con 0x7f573c0a41c0 2021-02-10T11:32:49.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.451 7f5744552700 1 -- 172.21.2.117:0/3195042669 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802494", "target": ["mgr", ""]}) v1 -- 0x7f573c077a70 con 0x7f57240186c0 2021-02-10T11:32:49.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.476 7f572effd700 1 -- 172.21.2.117:0/3195042669 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5730006a70 con 0x7f57240186c0 2021-02-10T11:32:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.484 7f5744552700 1 -- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 msgr2=0x7f572401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.484 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 0x7f572401abc0 crc :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.484 7f5744552700 1 -- 172.21.2.117:0/3195042669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 msgr2=0x7f573c0fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.484 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f5738009650 tx=0x7f573800b960).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 -- 172.21.2.117:0/3195042669 shutdown_connections 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f573c0b7390 0x7f573c0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f573c0c2340 0x7f573c0c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57240186c0 0x7f572401abc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 --2- 172.21.2.117:0/3195042669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f573c0a41c0 0x7f573c0fe0d0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 -- 172.21.2.117:0/3195042669 >> 172.21.2.117:0/3195042669 conn(0x7f573c08bbc0 msgr2=0x7f573c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:49.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 -- 172.21.2.117:0/3195042669 shutdown_connections 2021-02-10T11:32:49.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.485 7f5744552700 1 -- 172.21.2.117:0/3195042669 wait complete. 2021-02-10T11:32:49.481 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802494 2021-02-10T11:32:49.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 Processor -- start 2021-02-10T11:32:49.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 -- start start 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbba807da20 0x7fbba80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 0x7fbba81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba8077a70 con 0x7fbba80e51a0 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba80777d0 con 0x7fbba807da20 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbb0031700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba807de20 con 0x7fbba80e4da0 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbade4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbba807da20 0x7fbba80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.909 7fbbade4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbba807da20 0x7fbba80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44780/0 (socket says 172.21.2.117:44780) 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 -- 172.21.2.117:0/2780786006 learned_addr learned my addr 172.21.2.117:0/2780786006 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:49.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 -- 172.21.2.117:0/2780786006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 msgr2=0x7fbba81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbad64c700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 0x7fbba81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 0x7fbba81b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 -- 172.21.2.117:0/2780786006 >> v1:172.21.2.125:6789/0 conn(0x7fbba80e51a0 legacy=0x7fbba8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 -- 172.21.2.117:0/2780786006 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba81b0710 con 0x7fbba807da20 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbad64c700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 0x7fbba81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbade4d700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba807da20 0x7fbba80771b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fbba4006700 tx=0x7fbba4009b80).ready entity=mon.0 client_cookie=c5c11e74fc6c0828 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.910 7fbbace4b700 1 -- 172.21.2.117:0/2780786006 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba400c0c0 con 0x7fbba807da20 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbace4b700 1 -- 172.21.2.117:0/2780786006 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba400c3c0 con 0x7fbba807da20 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbace4b700 1 -- 172.21.2.117:0/2780786006 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba400c8a0 con 0x7fbba807da20 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 -- 172.21.2.117:0/2780786006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba807da20 msgr2=0x7fbba80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba807da20 0x7fbba80771b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fbba4006700 tx=0x7fbba4009b80).stop 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 -- 172.21.2.117:0/2780786006 shutdown_connections 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80e4da0 0x7fbba81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 --2- 172.21.2.117:0/2780786006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba807da20 0x7fbba80771b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.911 7fbbb0031700 1 -- 172.21.2.117:0/2780786006 >> 172.21.2.117:0/2780786006 conn(0x7fbba80ce450 msgr2=0x7fbba80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.912 7fbbb0031700 1 -- 172.21.2.117:0/2780786006 shutdown_connections 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.912 7fbbb0031700 1 -- 172.21.2.117:0/2780786006 wait complete. 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.926 7fbbb0031700 1 Processor -- start 2021-02-10T11:32:49.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.926 7fbbb0031700 1 -- start start 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbba80b5490 0x7fbba80b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80b5890 0x7fbba80b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba80b9180 con 0x7fbba80b8d00 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8092df0 con 0x7fbba80b5490 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbb0031700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8093010 con 0x7fbba80b5890 2021-02-10T11:32:49.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44786/0 (socket says 172.21.2.117:44786) 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 -- 172.21.2.117:0/3714786681 learned_addr learned my addr 172.21.2.117:0/3714786681 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbad64c700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80b5890 0x7fbba80b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbade4d700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbba80b5490 0x7fbba80b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 -- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80b5890 msgr2=0x7fbba80b86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80b5890 0x7fbba80b86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 -- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbba80b5490 msgr2=0x7fbba80b80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbba80b5490 0x7fbba80b80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.927 7fbbae64e700 1 -- 172.21.2.117:0/3714786681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba8093230 con 0x7fbba80b8d00 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbbae64e700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fbba0009940 tx=0x7fbba000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba0000f90 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba000c2f0 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbba000c7d0 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8093230 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba8093230 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.928 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbba0001220 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.929 7fbb9a7fc700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 0x7fbb9001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.929 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbba0032b20 con 0x7fbba80b8d00 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.929 7fbbade4d700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 0x7fbb9001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.929 7fbbade4d700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 0x7fbb9001abd0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:49.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.959 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba8093b30 con 0x7fbba80b8d00 2021-02-10T11:32:49.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:49.962 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbba000be30 con 0x7fbba80b8d00 2021-02-10T11:32:50.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.091 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802494", "target": ["mgr", ""]}) v1 -- 0x7fbba8093b30 con 0x7fbb900186d0 2021-02-10T11:32:50.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.094 7fbb9a7fc700 1 -- 172.21.2.117:0/3714786681 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbba40090e0 con 0x7fbb900186d0 2021-02-10T11:32:50.083 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 msgr2=0x7fbb9001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 0x7fbb9001abd0 crc :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 msgr2=0x7fbba80927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fbba0009940 tx=0x7fbba000bcf0).stop 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 shutdown_connections 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb900186d0 0x7fbb9001abd0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:50.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbba80b8d00 0x7fbba80927d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:50.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbba80b5890 0x7fbba80b86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:50.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 --2- 172.21.2.117:0/3714786681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbba80b5490 0x7fbba80b80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:50.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 >> 172.21.2.117:0/3714786681 conn(0x7fbba8099ec0 msgr2=0x7fbba8190400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:50.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 shutdown_connections 2021-02-10T11:32:50.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:50.096 7fbbb0031700 1 -- 172.21.2.117:0/3714786681 wait complete. 2021-02-10T11:32:50.092 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802494 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4 2021-02-10T11:32:50.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:50.092 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:50.092 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:50.430 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:50.430 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:50.430 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:50.811 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:50.811 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:50.811 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226601 s, 46.3 MB/s 2021-02-10T11:32:50.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:50.812 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:50.812 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:51.150 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:51.150 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:51.150 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:51.527 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:51.527 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:51.527 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252823 s, 41.5 MB/s 2021-02-10T11:32:51.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:51.528 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:51.528 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:51.863 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:51.863 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:51.863 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0264881 s, 39.6 MB/s 2021-02-10T11:32:52.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.242 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:52.242 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:52.578 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:52.578 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:52.578 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:52.951 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:52.951 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:52.951 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254309 s, 41.2 MB/s 2021-02-10T11:32:52.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:52.953 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:52.953 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:53.290 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:53.290 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:53.290 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:53.667 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:53.667 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:53.668 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226041 s, 46.4 MB/s 2021-02-10T11:32:53.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:53.669 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:53.669 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:54.006 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:54.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:54.006 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:54.380 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:54.380 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:54.380 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237789 s, 44.1 MB/s 2021-02-10T11:32:54.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:54.382 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:54.382 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:54.717 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:54.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:54.718 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:55.092 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:55.093 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:55.093 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022519 s, 46.6 MB/s 2021-02-10T11:32:55.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:55.094 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:55.094 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:55.429 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:55.429 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:55.429 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:55.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:55.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:55.805 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228347 s, 45.9 MB/s 2021-02-10T11:32:55.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:55.806 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:55.806 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:56.152 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:56.153 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:56.153 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:56.530 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:56.530 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:56.530 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238638 s, 43.9 MB/s 2021-02-10T11:32:56.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:56.531 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:56.531 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:56.867 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:56.867 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:56.868 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802494/0e500f3a-0116-4370-b5e5-ac8dcb1a92a4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:57.245 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:57.246 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:57.246 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235419 s, 44.5 MB/s 2021-02-10T11:32:57.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802495 2021-02-10T11:32:57.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.674 7f87ef607700 1 Processor -- start 2021-02-10T11:32:57.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.674 7f87ef607700 1 -- start start 2021-02-10T11:32:57.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ef607700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 0x7f87e80e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ef607700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87e81b00d0 0x7f87e81b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ef607700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e8077a70 con 0x7f87e807da20 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ef607700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e80e53a0 con 0x7f87e81b00d0 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ef607700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e8079400 con 0x7f87e8079000 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ed423700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f87e807da20 0x7f87e80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45040/0 (socket says 172.21.2.117:45040) 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ed423700 1 -- 172.21.2.117:0/1516671454 learned_addr learned my addr 172.21.2.117:0/1516671454 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87edc24700 1 --2- 172.21.2.117:0/1516671454 >> v2:172.21.2.125:3300/0 conn(0x7f87e81b00d0 0x7f87e81b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87ecc22700 1 --2- 172.21.2.117:0/1516671454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 0x7f87e80e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87dbfff700 1 -- 172.21.2.117:0/1516671454 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471826697 0 0) 0x7f87dc0010b0 con 0x7f87e807da20 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.675 7f87dbfff700 1 -- 172.21.2.117:0/1516671454 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d4002fe0 con 0x7f87e807da20 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ecc22700 1 -- 172.21.2.117:0/1516671454 >> v2:172.21.2.125:3300/0 conn(0x7f87e81b00d0 msgr2=0x7f87e81b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ecc22700 1 --2- 172.21.2.117:0/1516671454 >> v2:172.21.2.125:3300/0 conn(0x7f87e81b00d0 0x7f87e81b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ecc22700 1 -- 172.21.2.117:0/1516671454 >> v1:172.21.2.117:6789/0 conn(0x7f87e807da20 legacy=0x7f87e80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ecc22700 1 -- 172.21.2.117:0/1516671454 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e807de20 con 0x7f87e8079000 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ecc22700 1 --2- 172.21.2.117:0/1516671454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 0x7f87e80e4d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f87e4006650 tx=0x7f87e4009a80).ready entity=mon.2 client_cookie=516d28c06569a344 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87dbfff700 1 -- 172.21.2.117:0/1516671454 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e400c0c0 con 0x7f87e8079000 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87dbfff700 1 -- 172.21.2.117:0/1516671454 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e400c340 con 0x7f87e8079000 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87dbfff700 1 -- 172.21.2.117:0/1516671454 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e400c820 con 0x7f87e8079000 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ef607700 1 -- 172.21.2.117:0/1516671454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 msgr2=0x7f87e80e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.676 7f87ef607700 1 --2- 172.21.2.117:0/1516671454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 0x7f87e80e4d80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f87e4006650 tx=0x7f87e4009a80).stop 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 -- 172.21.2.117:0/1516671454 shutdown_connections 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 --2- 172.21.2.117:0/1516671454 >> v2:172.21.2.125:3300/0 conn(0x7f87e81b00d0 0x7f87e81b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 --2- 172.21.2.117:0/1516671454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8079000 0x7f87e80e4d80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 -- 172.21.2.117:0/1516671454 >> 172.21.2.117:0/1516671454 conn(0x7f87e80ce430 msgr2=0x7f87e80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 -- 172.21.2.117:0/1516671454 shutdown_connections 2021-02-10T11:32:57.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.677 7f87ef607700 1 -- 172.21.2.117:0/1516671454 wait complete. 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 Processor -- start 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 -- start start 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8104ea0 0x7f87e81052a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87e80b2140 0x7f87e80b2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80a4330 con 0x7f87e80a3910 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80a21e0 con 0x7f87e80b2140 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.692 7f87ef607700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e80a2400 con 0x7f87e8104ea0 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44800/0 (socket says 172.21.2.117:44800) 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 -- 172.21.2.117:0/886641174 learned_addr learned my addr 172.21.2.117:0/886641174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 -- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8104ea0 msgr2=0x7f87e81052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8104ea0 0x7f87e81052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 -- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87e80b2140 msgr2=0x7f87e80b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87e80b2140 0x7f87e80b2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 -- 172.21.2.117:0/886641174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e80a2620 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87edc24700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f87e000b970 tx=0x7f87e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e0000f90 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0003030 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e0001400 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.693 7f87ef607700 1 -- 172.21.2.117:0/886641174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e80a2620 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.694 7f87ef607700 1 -- 172.21.2.117:0/886641174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e80a2620 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.694 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87e0000eb0 con 0x7f87e80a3910 2021-02-10T11:32:57.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.694 7f87d9ffb700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 0x7f87c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:57.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.694 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f87e0032a60 con 0x7f87e80a3910 2021-02-10T11:32:57.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.695 7f87ed423700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 0x7f87c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:57.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.695 7f87ed423700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 0x7f87c801abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:57.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.724 7f87ef607700 1 -- 172.21.2.117:0/886641174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e8077a70 con 0x7f87e80a3910 2021-02-10T11:32:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.727 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87e00066f0 con 0x7f87e80a3910 2021-02-10T11:32:57.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.854 7f87ef607700 1 -- 172.21.2.117:0/886641174 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802495", "target": ["mgr", ""]}) v1 -- 0x7f87e8077a70 con 0x7f87c80186c0 2021-02-10T11:32:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.879 7f87d9ffb700 1 -- 172.21.2.117:0/886641174 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f87dc0009b0 con 0x7f87c80186c0 2021-02-10T11:32:57.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 -- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 msgr2=0x7f87c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 0x7f87c801abc0 crc :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 -- 172.21.2.117:0/886641174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 msgr2=0x7f87e80a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:57.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f87e000b970 tx=0x7f87e000bd10).stop 2021-02-10T11:32:57.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 -- 172.21.2.117:0/886641174 shutdown_connections 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87c80186c0 0x7f87c801abc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87e80a3910 0x7f87e80a3d10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87e80b2140 0x7f87e80b2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 --2- 172.21.2.117:0/886641174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87e8104ea0 0x7f87e81052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.887 7f87ef607700 1 -- 172.21.2.117:0/886641174 >> 172.21.2.117:0/886641174 conn(0x7f87e80a18e0 msgr2=0x7f87e8104300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.888 7f87ef607700 1 -- 172.21.2.117:0/886641174 shutdown_connections 2021-02-10T11:32:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:57.888 7f87ef607700 1 -- 172.21.2.117:0/886641174 wait complete. 2021-02-10T11:32:57.883 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802495 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 Processor -- start 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 -- start start 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43280d1530 0x7f43280d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4328077a70 con 0x7f432807da20 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4328079400 con 0x7f4328079000 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.310 7f432d777700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432807de20 con 0x7f43280d1530 2021-02-10T11:32:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43280d1530 0x7f43280d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43280d1530 0x7f43280d1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44808/0 (socket says 172.21.2.117:44808) 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 -- 172.21.2.117:0/471922025 learned_addr learned my addr 172.21.2.117:0/471922025 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 -- 172.21.2.117:0/471922025 >> v1:172.21.2.125:6789/0 conn(0x7f4328079000 legacy=0x7f43280e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 -- 172.21.2.117:0/471922025 >> v1:172.21.2.117:6789/0 conn(0x7f432807da20 legacy=0x7f43280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 -- 172.21.2.117:0/471922025 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43281b00f0 con 0x7f43280d1530 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4327d75700 1 --2- 172.21.2.117:0/471922025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43280d1530 0x7f43280d1930 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f4320006f00 tx=0x7f4320009e90).ready entity=mon.0 client_cookie=586ff2fd59b55717 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4326572700 1 -- 172.21.2.117:0/471922025 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432000b300 con 0x7f43280d1530 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4326572700 1 -- 172.21.2.117:0/471922025 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432000b640 con 0x7f43280d1530 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f4326572700 1 -- 172.21.2.117:0/471922025 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f432000bb20 con 0x7f43280d1530 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f432d777700 1 -- 172.21.2.117:0/471922025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43280d1530 msgr2=0x7f43280d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.311 7f432d777700 1 --2- 172.21.2.117:0/471922025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43280d1530 0x7f43280d1930 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f4320006f00 tx=0x7f4320009e90).stop 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.312 7f432d777700 1 -- 172.21.2.117:0/471922025 shutdown_connections 2021-02-10T11:32:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.312 7f432d777700 1 --2- 172.21.2.117:0/471922025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43280d1530 0x7f43280d1930 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.312 7f432d777700 1 -- 172.21.2.117:0/471922025 >> 172.21.2.117:0/471922025 conn(0x7f43280ce450 msgr2=0x7f43280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.312 7f432d777700 1 -- 172.21.2.117:0/471922025 shutdown_connections 2021-02-10T11:32:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.312 7f432d777700 1 -- 172.21.2.117:0/471922025 wait complete. 2021-02-10T11:32:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.326 7f432d777700 1 Processor -- start 2021-02-10T11:32:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 -- start start 2021-02-10T11:32:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43280933f0 0x7f43280937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 0x7f4328107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328107a70 con 0x7f4328093e10 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c2c60 con 0x7f4328107050 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f432d777700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c2e80 con 0x7f43280933f0 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44814/0 (socket says 172.21.2.117:44814) 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 -- 172.21.2.117:0/2566252210 learned_addr learned my addr 172.21.2.117:0/2566252210 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4327574700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43280933f0 0x7f43280937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4327d75700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 0x7f4328107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 -- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43280933f0 msgr2=0x7f43280937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43280933f0 0x7f43280937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 -- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 msgr2=0x7f4328107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.327 7f4326d73700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 0x7f4328107450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f4326d73700 1 -- 172.21.2.117:0/2566252210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280c30a0 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f4327d75700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 0x7f4328107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f4326d73700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f431800b5b0 tx=0x7f431800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f431800bd40 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43180009b0 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43180062a0 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f432d777700 1 -- 172.21.2.117:0/2566252210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43280c30a0 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.328 7f432d777700 1 -- 172.21.2.117:0/2566252210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43280c3630 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.329 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4318006020 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.329 7f431ffff700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 0x7f430c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.329 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4318032690 con 0x7f4328093e10 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.329 7f4327574700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 0x7f430c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:32:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.329 7f4327574700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 0x7f430c01abc0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:32:58.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.360 7f432d777700 1 -- 172.21.2.117:0/2566252210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43280ce450 con 0x7f4328093e10 2021-02-10T11:32:58.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.363 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4318006230 con 0x7f4328093e10 2021-02-10T11:32:58.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.489 7f432d777700 1 -- 172.21.2.117:0/2566252210 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802495", "target": ["mgr", ""]}) v1 -- 0x7f43280c3630 con 0x7f430c0186c0 2021-02-10T11:32:58.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.493 7f431ffff700 1 -- 172.21.2.117:0/2566252210 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f43100068e0 con 0x7f430c0186c0 2021-02-10T11:32:58.480 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 msgr2=0x7f430c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 0x7f430c01abc0 crc :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 msgr2=0x7f4328106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f431800b5b0 tx=0x7f431800b950).stop 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 shutdown_connections 2021-02-10T11:32:58.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f430c0186c0 0x7f430c01abc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4328093e10 0x7f4328106a30 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4328107050 0x7f4328107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 --2- 172.21.2.117:0/2566252210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43280933f0 0x7f43280937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 >> 172.21.2.117:0/2566252210 conn(0x7f43281116a0 msgr2=0x7f43280af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 shutdown_connections 2021-02-10T11:32:58.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:32:58.494 7f432d777700 1 -- 172.21.2.117:0/2566252210 wait complete. 2021-02-10T11:32:58.490 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802495 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59 2021-02-10T11:32:58.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:58.490 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:58.490 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:58.826 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:58.826 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:58.826 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:59.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:59.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:59.207 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234846 s, 44.6 MB/s 2021-02-10T11:32:59.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:59.209 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:59.209 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:32:59.545 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:32:59.545 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:32:59.545 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:32:59.922 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:32:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:32:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255864 s, 41.0 MB/s 2021-02-10T11:32:59.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:32:59.924 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:32:59.924 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:00.006 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:00.009 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:00.260 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:00.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:00.261 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:00.636 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:00.636 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:00.636 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237093 s, 44.2 MB/s 2021-02-10T11:33:00.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:00.637 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:00.637 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:00.972 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:00.973 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:00.973 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:01.350 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:01.350 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:01.350 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237608 s, 44.1 MB/s 2021-02-10T11:33:01.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:01.351 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:01.351 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:01.689 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:01.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:01.689 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248502 s, 42.2 MB/s 2021-02-10T11:33:02.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:02.067 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:02.067 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:02.403 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:02.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:02.403 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:02.775 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:02.776 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:02.776 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229217 s, 45.7 MB/s 2021-02-10T11:33:02.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:02.777 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:02.777 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:03.110 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:03.110 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:03.111 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:03.487 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:03.487 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:03.487 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023282 s, 45.0 MB/s 2021-02-10T11:33:03.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:03.488 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:03.488 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:03.824 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:03.825 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:03.825 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:04.201 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:04.201 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:04.201 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246652 s, 42.5 MB/s 2021-02-10T11:33:04.202 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:04.202 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:04.202 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:04.539 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:04.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:04.539 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232706 s, 45.1 MB/s 2021-02-10T11:33:04.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:04.916 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:04.916 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:05.250 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:05.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:05.251 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802495/e30d6c3f-16c2-458e-8710-b0a197fdbd59/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:05.627 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:05.627 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:05.627 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223612 s, 46.9 MB/s 2021-02-10T11:33:05.628 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802496 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 Processor -- start 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 -- start start 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43e007da20 0x7f43e00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f43e00e4da0 0x7f43e01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e0077a70 con 0x7f43e00e51a0 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e00777d0 con 0x7f43e007da20 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.055 7f43e5b27700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e007de20 con 0x7f43e00e4da0 2021-02-10T11:33:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43e007da20 0x7f43e00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43e007da20 0x7f43e00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44818/0 (socket says 172.21.2.117:44818) 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 -- 172.21.2.117:0/4206270827 learned_addr learned my addr 172.21.2.117:0/4206270827 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 -- 172.21.2.117:0/4206270827 >> v2:172.21.2.125:3300/0 conn(0x7f43e00e4da0 msgr2=0x7f43e01b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 --2- 172.21.2.117:0/4206270827 >> v2:172.21.2.125:3300/0 conn(0x7f43e00e4da0 0x7f43e01b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 -- 172.21.2.117:0/4206270827 >> v1:172.21.2.125:6789/0 conn(0x7f43e00e51a0 legacy=0x7f43e0077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 -- 172.21.2.117:0/4206270827 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e01b0710 con 0x7f43e007da20 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43df7fe700 1 --2- 172.21.2.117:0/4206270827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e007da20 0x7f43e00771b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f43d0006700 tx=0x7f43d0009b00).ready entity=mon.0 client_cookie=c0c39a53d0866b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43de7fc700 1 -- 172.21.2.117:0/4206270827 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d00009b0 con 0x7f43e007da20 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43de7fc700 1 -- 172.21.2.117:0/4206270827 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d000c0c0 con 0x7f43e007da20 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43de7fc700 1 -- 172.21.2.117:0/4206270827 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43d000c550 con 0x7f43e007da20 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43e5b27700 1 -- 172.21.2.117:0/4206270827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e007da20 msgr2=0x7f43e00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.056 7f43e5b27700 1 --2- 172.21.2.117:0/4206270827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e007da20 0x7f43e00771b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f43d0006700 tx=0x7f43d0009b00).stop 2021-02-10T11:33:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 -- 172.21.2.117:0/4206270827 shutdown_connections 2021-02-10T11:33:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 --2- 172.21.2.117:0/4206270827 >> v2:172.21.2.125:3300/0 conn(0x7f43e00e4da0 0x7f43e01b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 --2- 172.21.2.117:0/4206270827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e007da20 0x7f43e00771b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 -- 172.21.2.117:0/4206270827 >> 172.21.2.117:0/4206270827 conn(0x7f43e00ce450 msgr2=0x7f43e00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 -- 172.21.2.117:0/4206270827 shutdown_connections 2021-02-10T11:33:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.057 7f43e5b27700 1 -- 172.21.2.117:0/4206270827 wait complete. 2021-02-10T11:33:06.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.071 7f43e5b27700 1 Processor -- start 2021-02-10T11:33:06.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 -- start start 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e00a9890 0x7f43e00a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e00b79c0 0x7f43e00b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e0189be0 con 0x7f43e018a0e0 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00b83e0 con 0x7f43e00a9890 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43e5b27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e00b8600 con 0x7f43e00b79c0 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43deffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43deffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44824/0 (socket says 172.21.2.117:44824) 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.072 7f43deffd700 1 -- 172.21.2.117:0/3390489506 learned_addr learned my addr 172.21.2.117:0/3390489506 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 -- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e00b79c0 msgr2=0x7f43e00b7dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e00b79c0 0x7f43e00b7dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 -- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e00a9890 msgr2=0x7f43e00a9c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e00a9890 0x7f43e00a9c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 -- 172.21.2.117:0/3390489506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e00b8820 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43deffd700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f43c800b560 tx=0x7f43c800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c800bcf0 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c8006030 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43c80054d0 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e00b8820 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.073 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e00b8820 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.074 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43c8006210 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.074 7f43d7fff700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 0x7f43c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.074 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f43c80328e0 con 0x7f43e018a0e0 2021-02-10T11:33:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.075 7f43df7fe700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 0x7f43c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.075 7f43df7fe700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 0x7f43c401ac10 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.106 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e01032d0 con 0x7f43e018a0e0 2021-02-10T11:33:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.108 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43c8006b80 con 0x7f43e018a0e0 2021-02-10T11:33:06.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.234 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802496", "target": ["mgr", ""]}) v1 -- 0x7f43e00b83e0 con 0x7f43c4018730 2021-02-10T11:33:06.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.257 7f43d7fff700 1 -- 172.21.2.117:0/3390489506 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f43d0009440 con 0x7f43c4018730 2021-02-10T11:33:06.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 msgr2=0x7f43c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 0x7f43c401ac10 crc :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 msgr2=0x7f43e018a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f43c800b560 tx=0x7f43c800b900).stop 2021-02-10T11:33:06.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 shutdown_connections 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43c4018730 0x7f43c401ac10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43e018a0e0 0x7f43e018a4e0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43e00b79c0 0x7f43e00b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 --2- 172.21.2.117:0/3390489506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43e00a9890 0x7f43e00a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.267 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 >> 172.21.2.117:0/3390489506 conn(0x7f43e00b56c0 msgr2=0x7f43e00a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.268 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 shutdown_connections 2021-02-10T11:33:06.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.268 7f43e5b27700 1 -- 172.21.2.117:0/3390489506 wait complete. 2021-02-10T11:33:06.263 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802496 2021-02-10T11:33:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.691 7f4a6029c700 1 Processor -- start 2021-02-10T11:33:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.691 7f4a6029c700 1 -- start start 2021-02-10T11:33:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a6029c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a5807da20 0x7f4a580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a6029c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4a580e4d80 0x7f4a581b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a6029c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a58077a70 con 0x7f4a580e5180 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a6029c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a580777d0 con 0x7f4a5807da20 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a6029c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5807de20 con 0x7f4a580e4d80 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a5807da20 0x7f4a580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a5807da20 0x7f4a580771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44830/0 (socket says 172.21.2.117:44830) 2021-02-10T11:33:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 -- 172.21.2.117:0/1541802775 learned_addr learned my addr 172.21.2.117:0/1541802775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 -- 172.21.2.117:0/1541802775 >> v2:172.21.2.125:3300/0 conn(0x7f4a580e4d80 msgr2=0x7f4a581b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 --2- 172.21.2.117:0/1541802775 >> v2:172.21.2.125:3300/0 conn(0x7f4a580e4d80 0x7f4a581b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 -- 172.21.2.117:0/1541802775 >> v1:172.21.2.125:6789/0 conn(0x7f4a580e5180 legacy=0x7f4a58077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 -- 172.21.2.117:0/1541802775 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a581b06f0 con 0x7f4a5807da20 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5e0b8700 1 --2- 172.21.2.117:0/1541802775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5807da20 0x7f4a580771b0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f4a4c006700 tx=0x7f4a4c009b80).ready entity=mon.0 client_cookie=96f2f13173a87330 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5d0b6700 1 -- 172.21.2.117:0/1541802775 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4c00c0c0 con 0x7f4a5807da20 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.692 7f4a5d0b6700 1 -- 172.21.2.117:0/1541802775 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c00c340 con 0x7f4a5807da20 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a5d0b6700 1 -- 172.21.2.117:0/1541802775 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4c00c7a0 con 0x7f4a5807da20 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 -- 172.21.2.117:0/1541802775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5807da20 msgr2=0x7f4a580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 --2- 172.21.2.117:0/1541802775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5807da20 0x7f4a580771b0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f4a4c006700 tx=0x7f4a4c009b80).stop 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 -- 172.21.2.117:0/1541802775 shutdown_connections 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 --2- 172.21.2.117:0/1541802775 >> v2:172.21.2.125:3300/0 conn(0x7f4a580e4d80 0x7f4a581b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 --2- 172.21.2.117:0/1541802775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5807da20 0x7f4a580771b0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 -- 172.21.2.117:0/1541802775 >> 172.21.2.117:0/1541802775 conn(0x7f4a580ce430 msgr2=0x7f4a580e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 -- 172.21.2.117:0/1541802775 shutdown_connections 2021-02-10T11:33:06.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.693 7f4a6029c700 1 -- 172.21.2.117:0/1541802775 wait complete. 2021-02-10T11:33:06.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.708 7f4a6029c700 1 Processor -- start 2021-02-10T11:33:06.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.708 7f4a6029c700 1 -- start start 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a580a39b0 0x7f4a580a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a580a3db0 0x7f4a580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580a77c0 con 0x7f4a5809c0c0 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580a5360 con 0x7f4a580a39b0 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580a5580 con 0x7f4a580a3db0 2021-02-10T11:33:06.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44836/0 (socket says 172.21.2.117:44836) 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 -- 172.21.2.117:0/1315172681 learned_addr learned my addr 172.21.2.117:0/1315172681 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e0b8700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a580a39b0 0x7f4a580a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5d8b7700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a580a3db0 0x7f4a580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 -- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a580a3db0 msgr2=0x7f4a580a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a580a3db0 0x7f4a580a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 -- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a580a39b0 msgr2=0x7f4a580a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a580a39b0 0x7f4a580a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 -- 172.21.2.117:0/1315172681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5809cae0 con 0x7f4a5809c0c0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a5e8b9700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f4a50009940 tx=0x7f4a5000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a50000d30 con 0x7f4a5809c0c0 2021-02-10T11:33:06.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a50000f90 con 0x7f4a5809c0c0 2021-02-10T11:33:06.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.709 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5809cae0 con 0x7f4a5809c0c0 2021-02-10T11:33:06.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a5000c570 con 0x7f4a5809c0c0 2021-02-10T11:33:06.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5809cae0 con 0x7f4a5809c0c0 2021-02-10T11:33:06.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a50001220 con 0x7f4a5809c0c0 2021-02-10T11:33:06.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a4affd700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:06.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a50032cf0 con 0x7f4a5809c0c0 2021-02-10T11:33:06.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.710 7f4a5e0b8700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:06.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.711 7f4a5e0b8700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 0x7f4a4001ac10 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:06.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.741 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a580a5360 con 0x7f4a5809c0c0 2021-02-10T11:33:06.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.744 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a50000f70 con 0x7f4a5809c0c0 2021-02-10T11:33:06.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.871 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802496", "target": ["mgr", ""]}) v1 -- 0x7f4a580a77c0 con 0x7f4a40018730 2021-02-10T11:33:06.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.875 7f4a4affd700 1 -- 172.21.2.117:0/1315172681 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4a4c006430 con 0x7f4a40018730 2021-02-10T11:33:06.866 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5 2021-02-10T11:33:06.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 msgr2=0x7f4a4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 0x7f4a4001ac10 crc :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 msgr2=0x7f4a5809c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f4a50009940 tx=0x7f4a5000bcf0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 shutdown_connections 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a40018730 0x7f4a4001ac10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5809c0c0 0x7f4a5809c4c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a580a3db0 0x7f4a580a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 --2- 172.21.2.117:0/1315172681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a580a39b0 0x7f4a580a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:06.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 >> 172.21.2.117:0/1315172681 conn(0x7f4a580bb240 msgr2=0x7f4a580a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:06.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 shutdown_connections 2021-02-10T11:33:06.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:06.880 7f4a6029c700 1 -- 172.21.2.117:0/1315172681 wait complete. 2021-02-10T11:33:06.877 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802496 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5 2021-02-10T11:33:06.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:06.877 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:06.877 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:07.215 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:07.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:07.216 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:07.592 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:07.592 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:07.593 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024103 s, 43.5 MB/s 2021-02-10T11:33:07.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:07.594 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:07.595 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:07.930 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:07.931 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:07.931 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:08.306 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:08.306 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:08.306 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237755 s, 44.1 MB/s 2021-02-10T11:33:08.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:08.307 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:08.307 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:08.643 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:08.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:08.644 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:09.022 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:09.022 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:09.022 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0263292 s, 39.8 MB/s 2021-02-10T11:33:09.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:09.023 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:09.023 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:09.357 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:09.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:09.358 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:09.734 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:09.734 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:09.734 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251796 s, 41.6 MB/s 2021-02-10T11:33:09.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:09.735 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:09.735 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:10.071 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:10.072 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:10.072 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:10.450 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:10.450 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:10.450 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247446 s, 42.4 MB/s 2021-02-10T11:33:10.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:10.451 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:10.451 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:10.786 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:10.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:10.787 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241773 s, 43.4 MB/s 2021-02-10T11:33:11.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:11.165 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:11.165 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:11.502 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:11.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:11.502 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:11.876 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:11.876 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:11.876 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232904 s, 45.0 MB/s 2021-02-10T11:33:11.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:11.878 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:11.878 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:12.212 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:12.213 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:12.213 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:12.584 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:12.585 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:12.585 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230586 s, 45.5 MB/s 2021-02-10T11:33:12.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:12.586 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:12.586 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:12.923 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:12.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:12.924 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:13.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:13.298 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:13.298 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223511 s, 46.9 MB/s 2021-02-10T11:33:13.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:13.299 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:13.299 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:13.634 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:13.635 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:13.635 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802496/0a37e965-5ed9-40d0-b1cf-be32c172fcf5/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:14.011 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:14.011 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:14.011 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255677 s, 41.0 MB/s 2021-02-10T11:33:14.012 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802497 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.439 7fc6ec4a4700 1 Processor -- start 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.439 7fc6ec4a4700 1 -- start start 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ec4a4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6e407da20 0x7fc6e40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ec4a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40e4da0 0x7fc6e41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ec4a4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e4077a70 con 0x7fc6e40e51a0 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ec4a4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e40777d0 con 0x7fc6e407da20 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ec4a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e407de20 con 0x7fc6e40e4da0 2021-02-10T11:33:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ea2c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6e407da20 0x7fc6e40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ea2c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6e407da20 0x7fc6e40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44842/0 (socket says 172.21.2.117:44842) 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6ea2c0700 1 -- 172.21.2.117:0/1313236756 learned_addr learned my addr 172.21.2.117:0/1313236756 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.440 7fc6e9abf700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40e4da0 0x7fc6e41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ea2c0700 1 -- 172.21.2.117:0/1313236756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40e4da0 msgr2=0x7fc6e41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ea2c0700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40e4da0 0x7fc6e41b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ea2c0700 1 -- 172.21.2.117:0/1313236756 >> v1:172.21.2.117:6789/0 conn(0x7fc6e40e51a0 legacy=0x7fc6e4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ea2c0700 1 -- 172.21.2.117:0/1313236756 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e41b0710 con 0x7fc6e407da20 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ea2c0700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e407da20 0x7fc6e40771b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fc6e0006700 tx=0x7fc6e0009b80).ready entity=mon.0 client_cookie=f01bbbf3193575ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6e92be700 1 -- 172.21.2.117:0/1313236756 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e000c0c0 con 0x7fc6e407da20 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6e92be700 1 -- 172.21.2.117:0/1313236756 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e000c440 con 0x7fc6e407da20 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6e92be700 1 -- 172.21.2.117:0/1313236756 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e000c960 con 0x7fc6e407da20 2021-02-10T11:33:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ec4a4700 1 -- 172.21.2.117:0/1313236756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e407da20 msgr2=0x7fc6e40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.441 7fc6ec4a4700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e407da20 0x7fc6e40771b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fc6e0006700 tx=0x7fc6e0009b80).stop 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 -- 172.21.2.117:0/1313236756 shutdown_connections 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40e4da0 0x7fc6e41b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 --2- 172.21.2.117:0/1313236756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e407da20 0x7fc6e40771b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 -- 172.21.2.117:0/1313236756 >> 172.21.2.117:0/1313236756 conn(0x7fc6e40ce450 msgr2=0x7fc6e40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 -- 172.21.2.117:0/1313236756 shutdown_connections 2021-02-10T11:33:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.442 7fc6ec4a4700 1 -- 172.21.2.117:0/1313236756 wait complete. 2021-02-10T11:33:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 Processor -- start 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 -- start start 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6e408dfe0 0x7fc6e408e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40afac0 0x7fc6e40afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40af540 con 0x7fc6e408eb10 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40a0c80 con 0x7fc6e408dfe0 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ec4a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40b04e0 con 0x7fc6e40afac0 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ea2c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ea2c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44846/0 (socket says 172.21.2.117:44846) 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6ea2c0700 1 -- 172.21.2.117:0/1244868394 learned_addr learned my addr 172.21.2.117:0/1244868394 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6e9abf700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6e408dfe0 0x7fc6e408e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.457 7fc6eaac1700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40afac0 0x7fc6e40afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 -- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40afac0 msgr2=0x7fc6e40afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40afac0 0x7fc6e40afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 -- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6e408dfe0 msgr2=0x7fc6e408e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6e408dfe0 0x7fc6e408e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 -- 172.21.2.117:0/1244868394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e40b0700 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ea2c0700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fc6e000b5e0 tx=0x7fc6e000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e000bd70 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0002030 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6e0002550 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e40b0700 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.458 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e40b0c90 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.459 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6e0000c90 con 0x7fc6e408eb10 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.459 7fc6d6ffd700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 0x7fc6cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:14.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.459 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc6e0032960 con 0x7fc6e408eb10 2021-02-10T11:33:14.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.459 7fc6e9abf700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 0x7fc6cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:14.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.459 7fc6e9abf700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 0x7fc6cc01abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:14.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.490 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4103280 con 0x7fc6e408eb10 2021-02-10T11:33:14.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.492 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6e00009b0 con 0x7fc6e408eb10 2021-02-10T11:33:14.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.619 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802497", "target": ["mgr", ""]}) v1 -- 0x7fc6e40b0c90 con 0x7fc6cc018700 2021-02-10T11:33:14.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.645 7fc6d6ffd700 1 -- 172.21.2.117:0/1244868394 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc6d8006a70 con 0x7fc6cc018700 2021-02-10T11:33:14.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 msgr2=0x7fc6cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 0x7fc6cc01abc0 crc :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 msgr2=0x7fc6e40c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7fc6e000b5e0 tx=0x7fc6e000b980).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 shutdown_connections 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6e40afac0 0x7fc6e40afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6e408dfe0 0x7fc6e408e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6cc018700 0x7fc6cc01abc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 --2- 172.21.2.117:0/1244868394 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6e408eb10 0x7fc6e40c3b60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 >> 172.21.2.117:0/1244868394 conn(0x7fc6e40a1850 msgr2=0x7fc6e40bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 shutdown_connections 2021-02-10T11:33:14.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:14.652 7fc6ec4a4700 1 -- 172.21.2.117:0/1244868394 wait complete. 2021-02-10T11:33:14.647 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802497 2021-02-10T11:33:15.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 Processor -- start 2021-02-10T11:33:15.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 -- start start 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f780807da20 0x7f78080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78080d1d20 con 0x7f78080e4da0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7808077a70 con 0x7f78080e51a0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f780e571700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080777d0 con 0x7f780807da20 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f78077fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f78080e4da0 0x7f78081b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45104/0 (socket says 172.21.2.117:45104) 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.076 7f78077fe700 1 -- 172.21.2.117:0/488383569 learned_addr learned my addr 172.21.2.117:0/488383569 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607080919 0 0) 0x7f77fc0010b0 con 0x7f78080e4da0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7807fff700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f780807da20 0x7f78080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f0002fe0 con 0x7f78080e4da0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3728344232 0 0) 0x7f77fc001010 con 0x7f78080e4da0 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f780807da20 msgr2=0x7f78080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:15.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f780807da20 0x7f78080771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 >> v1:172.21.2.125:6789/0 conn(0x7f78080e51a0 legacy=0x7f78080d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78081b08e0 con 0x7f78080e4da0 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f77fc001530 con 0x7f78080e4da0 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 >> v1:172.21.2.117:6789/0 conn(0x7f78080e4da0 legacy=0x7f78081b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77f0003aa0 0x7f77f0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77f0006790 0x7f77f0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77f0009590 0x7f77f000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000c2d0 con 0x7f77f0003aa0 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f000c4f0 con 0x7f77f0006790 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.077 7f7806ffd700 1 -- 172.21.2.117:0/488383569 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f0050720 con 0x7f77f0009590 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77f0003aa0 0x7f77f0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f78077fe700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77f0006790 0x7f77f0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f780cb8e700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77f0009590 0x7f77f000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 -- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77f0009590 msgr2=0x7f77f000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77f0009590 0x7f77f000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 -- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77f0006790 msgr2=0x7f77f0008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77f0006790 0x7f77f0008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 -- 172.21.2.117:0/488383569 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0050950 con 0x7f77f0003aa0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7807fff700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77f0003aa0 0x7f77f0005ef0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f77f8006ac0 tx=0x7f77f8009ab0).ready entity=mon.0 client_cookie=492d818b7073267c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f800c0c0 con 0x7f77f0003aa0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f800c3c0 con 0x7f77f0003aa0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f7806ffd700 1 -- 172.21.2.117:0/488383569 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77f800c8a0 con 0x7f77f0003aa0 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f780e571700 1 -- 172.21.2.117:0/488383569 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77f0003aa0 msgr2=0x7f77f0005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f780e571700 1 --2- 172.21.2.117:0/488383569 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77f0003aa0 0x7f77f0005ef0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f77f8006ac0 tx=0x7f77f8009ab0).stop 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.078 7f780cb8e700 1 -- 172.21.2.117:0/488383569 reap_dead start 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.079 7f780e571700 1 -- 172.21.2.117:0/488383569 shutdown_connections 2021-02-10T11:33:15.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.079 7f780e571700 1 -- 172.21.2.117:0/488383569 >> 172.21.2.117:0/488383569 conn(0x7f78080ce450 msgr2=0x7f78080e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.079 7f780e571700 1 -- 172.21.2.117:0/488383569 shutdown_connections 2021-02-10T11:33:15.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.079 7f780e571700 1 -- 172.21.2.117:0/488383569 wait complete. 2021-02-10T11:33:15.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 Processor -- start 2021-02-10T11:33:15.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 -- start start 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78080a47d0 0x7f78080a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78080b77d0 0x7f78080b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7808189970 con 0x7f78080a51f0 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7808190aa0 con 0x7f78080b77d0 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f780e571700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78080b81f0 con 0x7f78080a47d0 2021-02-10T11:33:15.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f78077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f78077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44868/0 (socket says 172.21.2.117:44868) 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.094 7f78077fe700 1 -- 172.21.2.117:0/3216489414 learned_addr learned my addr 172.21.2.117:0/3216489414 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 -- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78080a47d0 msgr2=0x7f78080a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78080a47d0 0x7f78080a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 -- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78080b77d0 msgr2=0x7f78080b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78080b77d0 0x7f78080b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 -- 172.21.2.117:0/3216489414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080b8410 con 0x7f78080a51f0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f78077fe700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f77fc001c00 tx=0x7f77fc00a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc0009b0 con 0x7f78080a51f0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc00acd0 con 0x7f78080a51f0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc004350 con 0x7f78080a51f0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f780e571700 1 -- 172.21.2.117:0/3216489414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78080b8410 con 0x7f78080a51f0 2021-02-10T11:33:15.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f780e571700 1 -- 172.21.2.117:0/3216489414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78080b8410 con 0x7f78080a51f0 2021-02-10T11:33:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.095 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77fc004140 con 0x7f78080a51f0 2021-02-10T11:33:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.096 7f7804ff9700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 0x7f77e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.096 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f77fc032630 con 0x7f78080a51f0 2021-02-10T11:33:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.096 7f7807fff700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 0x7f77e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:15.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.096 7f7807fff700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 0x7f77e801abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:15.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.126 7f780e571700 1 -- 172.21.2.117:0/3216489414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7808077a70 con 0x7f78080a51f0 2021-02-10T11:33:15.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.129 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77fc02bf90 con 0x7f78080a51f0 2021-02-10T11:33:15.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.257 7f780e571700 1 -- 172.21.2.117:0/3216489414 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802497", "target": ["mgr", ""]}) v1 -- 0x7f7808077a70 con 0x7f77e80186c0 2021-02-10T11:33:15.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.262 7f7804ff9700 1 -- 172.21.2.117:0/3216489414 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f77f8006e20 con 0x7f77e80186c0 2021-02-10T11:33:15.252 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 msgr2=0x7f77e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 0x7f77e801abc0 crc :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 msgr2=0x7f78080a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f77fc001c00 tx=0x7f77fc00a8e0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 shutdown_connections 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77e80186c0 0x7f77e801abc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78080a51f0 0x7f78080a55f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78080b77d0 0x7f78080b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 --2- 172.21.2.117:0/3216489414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78080a47d0 0x7f78080a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:15.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 >> 172.21.2.117:0/3216489414 conn(0x7f78080b5140 msgr2=0x7f78080a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:15.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 shutdown_connections 2021-02-10T11:33:15.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:15.266 7f780e571700 1 -- 172.21.2.117:0/3216489414 wait complete. 2021-02-10T11:33:15.262 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802497 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8 2021-02-10T11:33:15.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:15.262 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:15.262 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:15.599 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:15.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:15.600 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238863 s, 43.9 MB/s 2021-02-10T11:33:15.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:15.983 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:15.983 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:16.319 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:16.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:16.320 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:16.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:16.696 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:16.696 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235669 s, 44.5 MB/s 2021-02-10T11:33:16.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:16.697 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:16.697 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:17.033 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:17.033 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:17.034 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:17.411 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:17.411 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:17.411 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026125 s, 40.1 MB/s 2021-02-10T11:33:17.412 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:17.412 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:17.412 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:17.749 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:17.749 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:17.749 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:18.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:18.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:18.125 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254627 s, 41.2 MB/s 2021-02-10T11:33:18.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:18.126 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:18.126 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:18.460 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:18.461 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:18.461 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:18.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:18.839 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:18.839 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251775 s, 41.6 MB/s 2021-02-10T11:33:18.840 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:18.840 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:18.840 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:19.175 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:19.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:19.176 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:19.549 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:19.549 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:19.549 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230366 s, 45.5 MB/s 2021-02-10T11:33:19.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:19.551 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:19.551 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:19.885 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:19.886 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:19.886 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:20.261 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:20.261 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:20.261 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237153 s, 44.2 MB/s 2021-02-10T11:33:20.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:20.262 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:20.262 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:20.598 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:20.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:20.599 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:20.973 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:20.973 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:20.973 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227222 s, 46.1 MB/s 2021-02-10T11:33:20.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:20.974 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:20.974 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:21.312 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:21.312 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:21.312 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240118 s, 43.7 MB/s 2021-02-10T11:33:21.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:21.690 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:21.690 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:22.026 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:22.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:22.027 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802497/641e18f3-67e1-4303-b712-23899b69caf8/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:22.402 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:22.403 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242583 s, 43.2 MB/s 2021-02-10T11:33:22.404 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802498 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.834 7f3ce69f3700 1 Processor -- start 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.834 7f3ce69f3700 1 -- start start 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce69f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce007da20 0x7f3ce00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce69f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3ce00e4da0 0x7f3ce01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce69f3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce0077a70 con 0x7f3ce00e51a0 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce69f3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce00e5c20 con 0x7f3ce00e4da0 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce69f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00777d0 con 0x7f3ce007da20 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce5010700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3ce00e51a0 0x7f3ce0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45122/0 (socket says 172.21.2.117:45122) 2021-02-10T11:33:22.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3ce5010700 1 -- 172.21.2.117:0/316278849 learned_addr learned my addr 172.21.2.117:0/316278849 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638214522 0 0) 0x7f3cd8000c10 con 0x7f3ce00e51a0 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.835 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc002fe0 con 0x7f3ce00e51a0 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 877610315 0 0) 0x7f3cd80010e0 con 0x7f3ce00e51a0 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce007da20 msgr2=0x7f3ce00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce007da20 0x7f3ce00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 >> v2:172.21.2.125:3300/0 conn(0x7f3ce00e4da0 msgr2=0x7f3ce01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 --2- 172.21.2.117:0/316278849 >> v2:172.21.2.125:3300/0 conn(0x7f3ce00e4da0 0x7f3ce01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce007de20 con 0x7f3ce00e51a0 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3cd8001600 con 0x7f3ce00e51a0 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 >> v1:172.21.2.117:6789/0 conn(0x7f3ce00e51a0 legacy=0x7f3ce0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ccc003b10 0x7f3ccc005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ccc0067a0 0x7f3ccc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ccc0094f0 0x7f3ccc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00c2c0 con 0x7f3ccc0094f0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00c4e0 con 0x7f3ccc003b10 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00c700 con 0x7f3ccc0067a0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce480f700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ccc003b10 0x7f3ccc005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ccc0094f0 0x7f3ccc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdffff700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ccc0067a0 0x7f3ccc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 -- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ccc0067a0 msgr2=0x7f3ccc008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ccc0067a0 0x7f3ccc008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 -- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ccc003b10 msgr2=0x7f3ccc005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ccc003b10 0x7f3ccc005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 -- 172.21.2.117:0/316278849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc00c9a0 con 0x7f3ccc0094f0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce480f700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ccc003b10 0x7f3ccc005f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3ce5010700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ccc0094f0 0x7f3ccc00ba10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f3cd8000de0 tx=0x7f3cd8007cf0).ready entity=mon.0 client_cookie=3ef9d73a8d5c2bf4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cd80070c0 con 0x7f3ccc0094f0 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd80043a0 con 0x7f3ccc0094f0 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.836 7f3cdf7fe700 1 -- 172.21.2.117:0/316278849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cd8004830 con 0x7f3ccc0094f0 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 -- 172.21.2.117:0/316278849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ccc0094f0 msgr2=0x7f3ccc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 --2- 172.21.2.117:0/316278849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ccc0094f0 0x7f3ccc00ba10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f3cd8000de0 tx=0x7f3cd8007cf0).stop 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce5010700 1 -- 172.21.2.117:0/316278849 reap_dead start 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 -- 172.21.2.117:0/316278849 shutdown_connections 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 -- 172.21.2.117:0/316278849 >> 172.21.2.117:0/316278849 conn(0x7f3ce00ce450 msgr2=0x7f3ce00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 -- 172.21.2.117:0/316278849 shutdown_connections 2021-02-10T11:33:22.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.837 7f3ce69f3700 1 -- 172.21.2.117:0/316278849 wait complete. 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 Processor -- start 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 -- start start 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce00a47d0 0x7f3ce00a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce00a51f0 0x7f3ce00a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce0189970 con 0x7f3ce00b77d0 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce0190aa0 con 0x7f3ce00a47d0 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.852 7f3ce69f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00b81f0 con 0x7f3ce00a51f0 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44884/0 (socket says 172.21.2.117:44884) 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 -- 172.21.2.117:0/2294559082 learned_addr learned my addr 172.21.2.117:0/2294559082 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 -- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce00a51f0 msgr2=0x7f3ce00a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce00a51f0 0x7f3ce00a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 -- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce00a47d0 msgr2=0x7f3ce00a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce00a47d0 0x7f3ce00a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 -- 172.21.2.117:0/2294559082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00b8410 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce5010700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f3cd8001c10 tx=0x7f3cd800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cd800acc0 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd8000b10 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3cd8004830 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce00b8410 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.853 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce00b8410 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.854 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3cd8007f60 con 0x7f3ce00b77d0 2021-02-10T11:33:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.854 7f3cdd7fa700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 0x7f3cc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:22.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.854 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3cd8032660 con 0x7f3ce00b77d0 2021-02-10T11:33:22.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.855 7f3ce480f700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 0x7f3cc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:22.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.856 7f3ce480f700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 0x7f3cc401abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:22.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.885 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce0077a70 con 0x7f3ce00b77d0 2021-02-10T11:33:22.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:22.888 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3cd8030d40 con 0x7f3ce00b77d0 2021-02-10T11:33:23.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.013 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802498", "target": ["mgr", ""]}) v1 -- 0x7f3ce0077a70 con 0x7f3cc40186c0 2021-02-10T11:33:23.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.040 7f3cdd7fa700 1 -- 172.21.2.117:0/2294559082 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3cc80009b0 con 0x7f3cc40186c0 2021-02-10T11:33:23.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.046 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 msgr2=0x7f3cc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.046 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 0x7f3cc401abc0 crc :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.046 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 msgr2=0x7f3ce00b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.046 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f3cd8001c10 tx=0x7f3cd800a8d0).stop 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 shutdown_connections 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3cc40186c0 0x7f3cc401abc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ce00b77d0 0x7f3ce00b7bd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ce00a51f0 0x7f3ce00a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 --2- 172.21.2.117:0/2294559082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ce00a47d0 0x7f3ce00a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 >> 172.21.2.117:0/2294559082 conn(0x7f3ce00b5140 msgr2=0x7f3ce00a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 shutdown_connections 2021-02-10T11:33:23.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.047 7f3ce69f3700 1 -- 172.21.2.117:0/2294559082 wait complete. 2021-02-10T11:33:23.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802498 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.470 7fbab912a700 1 Processor -- start 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.470 7fbab912a700 1 -- start start 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab912a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbab4079000 0x7fbab40e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab912a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab41b00f0 0x7fbab41b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab912a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab4077a70 con 0x7fbab407da20 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab912a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab40e53c0 con 0x7fbab4079000 2021-02-10T11:33:23.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab912a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4079400 con 0x7fbab41b00f0 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbab4079000 0x7fbab40e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbab4079000 0x7fbab40e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44892/0 (socket says 172.21.2.117:44892) 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 -- 172.21.2.117:0/2948380800 learned_addr learned my addr 172.21.2.117:0/2948380800 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab359e700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab41b00f0 0x7fbab41b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 -- 172.21.2.117:0/2948380800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab41b00f0 msgr2=0x7fbab41b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab41b00f0 0x7fbab41b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 -- 172.21.2.117:0/2948380800 >> v1:172.21.2.125:6789/0 conn(0x7fbab407da20 legacy=0x7fbab40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 -- 172.21.2.117:0/2948380800 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab407de20 con 0x7fbab4079000 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.471 7fbab259c700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab4079000 0x7fbab40e4da0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fbaa4006700 tx=0x7fbaa4009b80).ready entity=mon.0 client_cookie=6d6616975d13c6c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab1d9b700 1 -- 172.21.2.117:0/2948380800 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa400c0c0 con 0x7fbab4079000 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab1d9b700 1 -- 172.21.2.117:0/2948380800 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa400c440 con 0x7fbab4079000 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab1d9b700 1 -- 172.21.2.117:0/2948380800 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbaa400c920 con 0x7fbab4079000 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 -- 172.21.2.117:0/2948380800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab4079000 msgr2=0x7fbab40e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab4079000 0x7fbab40e4da0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fbaa4006700 tx=0x7fbaa4009b80).stop 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 -- 172.21.2.117:0/2948380800 shutdown_connections 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab41b00f0 0x7fbab41b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 --2- 172.21.2.117:0/2948380800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab4079000 0x7fbab40e4da0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 -- 172.21.2.117:0/2948380800 >> 172.21.2.117:0/2948380800 conn(0x7fbab40ce450 msgr2=0x7fbab40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 -- 172.21.2.117:0/2948380800 shutdown_connections 2021-02-10T11:33:23.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.472 7fbab912a700 1 -- 172.21.2.117:0/2948380800 wait complete. 2021-02-10T11:33:23.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 Processor -- start 2021-02-10T11:33:23.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 -- start start 2021-02-10T11:33:23.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbab4187e20 0x7fbab4188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab4090420 0x7fbab4090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41067f0 con 0x7fbab41a0250 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4107350 con 0x7fbab4187e20 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab912a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab41042d0 con 0x7fbab4090420 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab2d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab2d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44898/0 (socket says 172.21.2.117:44898) 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.487 7fbab2d9d700 1 -- 172.21.2.117:0/2203299503 learned_addr learned my addr 172.21.2.117:0/2203299503 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 -- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab4090420 msgr2=0x7fbab4090820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab4090420 0x7fbab4090820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 -- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbab4187e20 msgr2=0x7fbab4188220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbab4187e20 0x7fbab4188220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 -- 172.21.2.117:0/2203299503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4090e40 con 0x7fbab41a0250 2021-02-10T11:33:23.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab2d9d700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fba9c00b5b0 tx=0x7fba9c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9c00bd40 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9c006020 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab912a700 1 -- 172.21.2.117:0/2203299503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab4090e40 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbab912a700 1 -- 172.21.2.117:0/2203299503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab40913d0 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.488 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9c0054a0 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.489 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba9c006200 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.489 7fbaab7fe700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 0x7fba9801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.489 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fba9c032940 con 0x7fbab41a0250 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.489 7fbab259c700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 0x7fba9801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:23.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.491 7fbab259c700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 0x7fba9801ac10 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:23.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.521 7fbab912a700 1 -- 172.21.2.117:0/2203299503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4103280 con 0x7fbab41a0250 2021-02-10T11:33:23.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.523 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba9c000c10 con 0x7fbab41a0250 2021-02-10T11:33:23.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.652 7fbab912a700 1 -- 172.21.2.117:0/2203299503 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802498", "target": ["mgr", ""]}) v1 -- 0x7fbab40913d0 con 0x7fba98018730 2021-02-10T11:33:23.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.656 7fbaab7fe700 1 -- 172.21.2.117:0/2203299503 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbaa4009100 con 0x7fba98018730 2021-02-10T11:33:23.647 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6 2021-02-10T11:33:23.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 msgr2=0x7fba9801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 0x7fba9801ac10 crc :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 msgr2=0x7fbab4106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fba9c00b5b0 tx=0x7fba9c00b950).stop 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 shutdown_connections 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbab4090420 0x7fbab4090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbab4187e20 0x7fbab4188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba98018730 0x7fba9801ac10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 --2- 172.21.2.117:0/2203299503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbab41a0250 0x7fbab4106cf0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:23.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 >> 172.21.2.117:0/2203299503 conn(0x7fbab40a1850 msgr2=0x7fbab40c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:23.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 shutdown_connections 2021-02-10T11:33:23.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:23.661 7fbab912a700 1 -- 172.21.2.117:0/2203299503 wait complete. 2021-02-10T11:33:23.657 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802498 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6 2021-02-10T11:33:23.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:23.657 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:23.658 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:23.995 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:23.995 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:23.996 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:24.376 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:24.376 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:24.377 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244746 s, 42.8 MB/s 2021-02-10T11:33:24.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:24.378 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:24.378 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:24.714 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:24.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:24.714 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:25.092 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:25.092 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:25.092 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262221 s, 40.0 MB/s 2021-02-10T11:33:25.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:25.094 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:25.094 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:25.429 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:25.429 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:25.430 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:25.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:25.806 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:25.806 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242234 s, 43.3 MB/s 2021-02-10T11:33:25.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:25.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:25.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:26.143 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:26.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:26.144 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:26.520 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:26.520 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:26.520 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245046 s, 42.8 MB/s 2021-02-10T11:33:26.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:26.522 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:26.522 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:26.858 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:26.859 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:26.859 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:27.243 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:27.244 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:27.244 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0306604 s, 34.2 MB/s 2021-02-10T11:33:27.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:27.245 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:27.245 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:27.582 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:27.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:27.583 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:27.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:27.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:27.955 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233578 s, 44.9 MB/s 2021-02-10T11:33:27.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:27.956 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:27.956 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:28.292 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:28.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:28.293 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:28.670 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:28.671 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:28.671 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025385 s, 41.3 MB/s 2021-02-10T11:33:28.672 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:28.672 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:28.672 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:29.008 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:29.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:29.008 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:29.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:29.382 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:29.382 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231214 s, 45.4 MB/s 2021-02-10T11:33:29.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:29.383 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:29.383 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:29.718 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:29.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:29.719 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:30.096 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:30.096 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:30.096 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246123 s, 42.6 MB/s 2021-02-10T11:33:30.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:30.097 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:30.097 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:30.375 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:30.378 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:33:30.433 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:30.433 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:30.434 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802498/c2a581ab-b7de-41f3-809e-4b6187fbefe6/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:30.752 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:30.752 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:30.752 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248027 s, 42.3 MB/s 2021-02-10T11:33:30.753 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802499 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 Processor -- start 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 -- start start 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f844c0d1530 0x7f844c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c077a70 con 0x7f844c079000 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c079400 con 0x7f844c07da20 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.221 7f8452209700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844c07de20 con 0x7f844c0d1530 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f844c0d1530 0x7f844c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f844c0d1530 0x7f844c0d1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44904/0 (socket says 172.21.2.117:44904) 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 -- 172.21.2.117:0/1896197388 learned_addr learned my addr 172.21.2.117:0/1896197388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 -- 172.21.2.117:0/1896197388 >> v1:172.21.2.125:6789/0 conn(0x7f844c07da20 legacy=0x7f844c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 -- 172.21.2.117:0/1896197388 >> v1:172.21.2.117:6789/0 conn(0x7f844c079000 legacy=0x7f844c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 -- 172.21.2.117:0/1896197388 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c1b00f0 con 0x7f844c0d1530 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8450826700 1 --2- 172.21.2.117:0/1896197388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0d1530 0x7f844c0d1930 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f8444006f00 tx=0x7f8444009e90).ready entity=mon.0 client_cookie=1ba481ca2bf8820b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f844affd700 1 -- 172.21.2.117:0/1896197388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f844400b300 con 0x7f844c0d1530 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f844affd700 1 -- 172.21.2.117:0/1896197388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844400b580 con 0x7f844c0d1530 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f844affd700 1 -- 172.21.2.117:0/1896197388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f844400ba60 con 0x7f844c0d1530 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8452209700 1 -- 172.21.2.117:0/1896197388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0d1530 msgr2=0x7f844c0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.222 7f8452209700 1 --2- 172.21.2.117:0/1896197388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0d1530 0x7f844c0d1930 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f8444006f00 tx=0x7f8444009e90).stop 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.223 7f8452209700 1 -- 172.21.2.117:0/1896197388 shutdown_connections 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.223 7f8452209700 1 --2- 172.21.2.117:0/1896197388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0d1530 0x7f844c0d1930 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.223 7f8452209700 1 -- 172.21.2.117:0/1896197388 >> 172.21.2.117:0/1896197388 conn(0x7f844c0ce450 msgr2=0x7f844c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.223 7f8452209700 1 -- 172.21.2.117:0/1896197388 shutdown_connections 2021-02-10T11:33:31.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.223 7f8452209700 1 -- 172.21.2.117:0/1896197388 wait complete. 2021-02-10T11:33:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.237 7f8452209700 1 Processor -- start 2021-02-10T11:33:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 -- start start 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f844c093e10 0x7f844c106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f844c107050 0x7f844c107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c107a70 con 0x7f844c0933f0 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0c2c60 con 0x7f844c107050 2021-02-10T11:33:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f8452209700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844c0c2e80 con 0x7f844c093e10 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44910/0 (socket says 172.21.2.117:44910) 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 -- 172.21.2.117:0/2229741870 learned_addr learned my addr 172.21.2.117:0/2229741870 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 -- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f844c093e10 msgr2=0x7f844c106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f844c093e10 0x7f844c106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 -- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f844c107050 msgr2=0x7f844c107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f844c107050 0x7f844c107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 -- 172.21.2.117:0/2229741870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0c30a0 con 0x7f844c0933f0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.238 7f844bfff700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f843c00b5b0 tx=0x7f843c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f843c00bd40 con 0x7f844c0933f0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c006240 con 0x7f844c0933f0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8452209700 1 -- 172.21.2.117:0/2229741870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c0c30a0 con 0x7f844c0933f0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8452209700 1 -- 172.21.2.117:0/2229741870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c0c3630 con 0x7f844c0933f0 2021-02-10T11:33:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f843c005720 con 0x7f844c0933f0 2021-02-10T11:33:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f843c0054a0 con 0x7f844c0933f0 2021-02-10T11:33:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.239 7f8448ff9700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 0x7f842c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.240 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f843c032940 con 0x7f844c0933f0 2021-02-10T11:33:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.240 7f844b7fe700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 0x7f842c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.240 7f844b7fe700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 0x7f842c01ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.271 7f8452209700 1 -- 172.21.2.117:0/2229741870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844c0ce450 con 0x7f844c0933f0 2021-02-10T11:33:31.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.274 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f843c000c60 con 0x7f844c0933f0 2021-02-10T11:33:31.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.397 7f8452209700 1 -- 172.21.2.117:0/2229741870 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802499", "target": ["mgr", ""]}) v1 -- 0x7f844c0c3630 con 0x7f842c018730 2021-02-10T11:33:31.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.422 7f8448ff9700 1 -- 172.21.2.117:0/2229741870 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8440006e40 con 0x7f842c018730 2021-02-10T11:33:31.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 msgr2=0x7f842c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 0x7f842c01ac10 crc :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 msgr2=0x7f844c0937f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f843c00b5b0 tx=0x7f843c00b950).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 shutdown_connections 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f844c107050 0x7f844c107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f844c093e10 0x7f844c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f842c018730 0x7f842c01ac10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 --2- 172.21.2.117:0/2229741870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f844c0933f0 0x7f844c0937f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 >> 172.21.2.117:0/2229741870 conn(0x7f844c1116a0 msgr2=0x7f844c0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 shutdown_connections 2021-02-10T11:33:31.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.430 7f8452209700 1 -- 172.21.2.117:0/2229741870 wait complete. 2021-02-10T11:33:31.425 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802499 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 Processor -- start 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 -- start start 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e0807da20 0x7f9e080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e080e4da0 0x7f9e081b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e08077a70 con 0x7f9e080e51a0 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080e5c20 con 0x7f9e080e4da0 2021-02-10T11:33:31.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e106f4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080777d0 con 0x7f9e0807da20 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e0ed11700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9e080e51a0 0x7f9e08077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45166/0 (socket says 172.21.2.117:45166) 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.854 7f9e0ed11700 1 -- 172.21.2.117:0/3155057664 learned_addr learned my addr 172.21.2.117:0/3155057664 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364053358 0 0) 0x7f9e00001500 con 0x7f9e080e51a0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df4002fe0 con 0x7f9e080e51a0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e0807da20 0x7f9e080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0dd0f700 1 --2- 172.21.2.117:0/3155057664 >> v2:172.21.2.125:3300/0 conn(0x7f9e080e4da0 0x7f9e081b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1788183368 0 0) 0x7f9e00001460 con 0x7f9e080e51a0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e0807da20 msgr2=0x7f9e080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e0807da20 0x7f9e080771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 >> v2:172.21.2.125:3300/0 conn(0x7f9e080e4da0 msgr2=0x7f9e081b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 --2- 172.21.2.117:0/3155057664 >> v2:172.21.2.125:3300/0 conn(0x7f9e080e4da0 0x7f9e081b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0807de20 con 0x7f9e080e51a0 2021-02-10T11:33:31.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f9e00001980 con 0x7f9e080e51a0 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 >> v1:172.21.2.117:6789/0 conn(0x7f9e080e51a0 legacy=0x7f9e08077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9df4003b90 0x7f9df4006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9df40068e0 0x7f9df4008db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9df4009650 0x7f9df400bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df400c420 con 0x7f9df4003b90 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4050610 con 0x7f9df4009650 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4050830 con 0x7f9df40068e0 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0dd0f700 1 --2- 172.21.2.117:0/3155057664 >> v2:172.21.2.125:3300/0 conn(0x7f9e080e4da0 0x7f9e081b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e0807da20 0x7f9e080771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9df4003b90 0x7f9df4006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0dd0f700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9df40068e0 0x7f9df4008db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.855 7f9e0ed11700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9df4009650 0x7f9df400bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 -- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9df40068e0 msgr2=0x7f9df4008db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9df40068e0 0x7f9df4008db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 -- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9df4009650 msgr2=0x7f9df400bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9df4009650 0x7f9df400bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 -- 172.21.2.117:0/3155057664 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4050ad0 con 0x7f9df4003b90 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0e510700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9df4003b90 0x7f9df4006040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f9dfc006890 tx=0x7f9dfc009aa0).ready entity=mon.0 client_cookie=7921256b79bc9811 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc00c0c0 con 0x7f9df4003b90 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc00c3c0 con 0x7f9df4003b90 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0d50e700 1 -- 172.21.2.117:0/3155057664 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9dfc00c8e0 con 0x7f9df4003b90 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e106f4700 1 -- 172.21.2.117:0/3155057664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9df4003b90 msgr2=0x7f9df4006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e106f4700 1 --2- 172.21.2.117:0/3155057664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9df4003b90 0x7f9df4006040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f9dfc006890 tx=0x7f9dfc009aa0).stop 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.856 7f9e0ed11700 1 -- 172.21.2.117:0/3155057664 reap_dead start 2021-02-10T11:33:31.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.857 7f9e106f4700 1 -- 172.21.2.117:0/3155057664 shutdown_connections 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.857 7f9e106f4700 1 -- 172.21.2.117:0/3155057664 >> 172.21.2.117:0/3155057664 conn(0x7f9e080ce450 msgr2=0x7f9e080e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.857 7f9e106f4700 1 -- 172.21.2.117:0/3155057664 shutdown_connections 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.857 7f9e106f4700 1 -- 172.21.2.117:0/3155057664 wait complete. 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.871 7f9e106f4700 1 Processor -- start 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.871 7f9e106f4700 1 -- start start 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 0x7f9e080b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 0x7f9e08094f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080945a0 con 0x7f9e08093870 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08094290 con 0x7f9e080afed0 2021-02-10T11:33:31.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e106f4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08093170 con 0x7f9e08094b20 2021-02-10T11:33:31.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44926/0 (socket says 172.21.2.117:44926) 2021-02-10T11:33:31.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 -- 172.21.2.117:0/3090165289 learned_addr learned my addr 172.21.2.117:0/3090165289 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0e510700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 0x7f9e080b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 -- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 msgr2=0x7f9e08094f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0ed11700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 0x7f9e08094f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 0x7f9e08094f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 -- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 msgr2=0x7f9e080b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 0x7f9e080b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 -- 172.21.2.117:0/3090165289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08092d00 con 0x7f9e08093870 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0ed11700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 0x7f9e08094f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:31.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0e510700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 0x7f9e080b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.872 7f9e0dd0f700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f9e04006f70 tx=0x7f9e0400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e04006bc0 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0400bd50 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e04005560 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08092d00 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08092d00 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.873 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e04006030 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.874 7f9dfaffd700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 0x7f9df001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.874 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e040326c0 con 0x7f9e08093870 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.874 7f9e0e510700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 0x7f9df001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:31.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.874 7f9e0e510700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 0x7f9df001abc0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:31.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.904 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08077a70 con 0x7f9e08093870 2021-02-10T11:33:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:31.907 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e04030d20 con 0x7f9e08093870 2021-02-10T11:33:32.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.034 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802499", "target": ["mgr", ""]}) v1 -- 0x7f9e08094290 con 0x7f9df00186c0 2021-02-10T11:33:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.038 7f9dfaffd700 1 -- 172.21.2.117:0/3090165289 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9dfc006770 con 0x7f9df00186c0 2021-02-10T11:33:32.024 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 msgr2=0x7f9df001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 0x7f9df001abc0 crc :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 msgr2=0x7f9e08093c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f9e04006f70 tx=0x7f9e0400b960).stop 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 shutdown_connections 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9df00186c0 0x7f9df001abc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e08093870 0x7f9e08093c70 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e08094b20 0x7f9e08094f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 --2- 172.21.2.117:0/3090165289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e080afed0 0x7f9e080b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 >> 172.21.2.117:0/3090165289 conn(0x7f9e080a1030 msgr2=0x7f9e080a3440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 shutdown_connections 2021-02-10T11:33:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:32.039 7f9e106f4700 1 -- 172.21.2.117:0/3090165289 wait complete. 2021-02-10T11:33:32.035 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802499 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228 2021-02-10T11:33:32.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:32.035 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:32.035 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:32.370 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:32.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:32.371 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:32.750 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:32.750 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:32.750 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247752 s, 42.3 MB/s 2021-02-10T11:33:32.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:32.751 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:32.751 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:33.087 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:33.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:33.087 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:33.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:33.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:33.462 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230006 s, 45.6 MB/s 2021-02-10T11:33:33.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:33.464 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:33.464 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:33.800 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:33.800 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:33.800 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:34.175 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241724 s, 43.4 MB/s 2021-02-10T11:33:34.177 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:34.177 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:34.177 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:34.513 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:34.513 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:34.513 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:34.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:34.891 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:34.891 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249664 s, 42.0 MB/s 2021-02-10T11:33:34.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:34.892 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:34.892 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:35.228 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:35.229 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:35.229 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:35.605 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:35.605 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:35.605 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244869 s, 42.8 MB/s 2021-02-10T11:33:35.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:35.607 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:35.607 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:35.941 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:35.942 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:35.942 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:36.316 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:36.316 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:36.316 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229223 s, 45.7 MB/s 2021-02-10T11:33:36.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:36.317 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:36.318 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:36.653 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:36.653 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:36.653 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241288 s, 43.5 MB/s 2021-02-10T11:33:37.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:37.030 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:37.031 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:37.367 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:37.367 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:37.367 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:37.743 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:37.743 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:37.744 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260899 s, 40.2 MB/s 2021-02-10T11:33:37.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:37.745 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:37.745 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:38.081 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:38.081 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:38.081 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234678 s, 44.7 MB/s 2021-02-10T11:33:38.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:38.457 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:38.457 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:38.792 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:38.792 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:38.792 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802499/7435e180-9f0c-44c3-add4-fdda51a2f228/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:39.168 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:39.168 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:39.168 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242485 s, 43.2 MB/s 2021-02-10T11:33:39.169 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802500 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.598 7fe8cf167700 1 Processor -- start 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.598 7fe8cf167700 1 -- start start 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe8c807da20 0x7fe8c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8c80e51a0 0x7fe8c8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c8077a70 con 0x7fe8c80e4da0 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c80e5c20 con 0x7fe8c80e51a0 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c80777d0 con 0x7fe8c807da20 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8c80e51a0 0x7fe8c8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8c80e51a0 0x7fe8c8079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44938/0 (socket says 172.21.2.117:44938) 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 -- 172.21.2.117:0/2137235731 learned_addr learned my addr 172.21.2.117:0/2137235731 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8ccf83700 1 --2- 172.21.2.117:0/2137235731 >> v2:172.21.2.125:3300/0 conn(0x7fe8c807da20 0x7fe8c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 -- 172.21.2.117:0/2137235731 >> v2:172.21.2.125:3300/0 conn(0x7fe8c807da20 msgr2=0x7fe8c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 --2- 172.21.2.117:0/2137235731 >> v2:172.21.2.125:3300/0 conn(0x7fe8c807da20 0x7fe8c80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 -- 172.21.2.117:0/2137235731 >> v1:172.21.2.125:6789/0 conn(0x7fe8c80e4da0 legacy=0x7fe8c81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 -- 172.21.2.117:0/2137235731 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c81b08e0 con 0x7fe8c80e51a0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cd784700 1 --2- 172.21.2.117:0/2137235731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c80e51a0 0x7fe8c8079000 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fe8c0006b80 tx=0x7fe8c0009b90).ready entity=mon.0 client_cookie=d81de4e2c4c7c0ca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8c77fe700 1 -- 172.21.2.117:0/2137235731 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c000c0c0 con 0x7fe8c80e51a0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8c77fe700 1 -- 172.21.2.117:0/2137235731 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c000c340 con 0x7fe8c80e51a0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8c77fe700 1 -- 172.21.2.117:0/2137235731 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8c000c820 con 0x7fe8c80e51a0 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 -- 172.21.2.117:0/2137235731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c80e51a0 msgr2=0x7fe8c8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.599 7fe8cf167700 1 --2- 172.21.2.117:0/2137235731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c80e51a0 0x7fe8c8079000 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7fe8c0006b80 tx=0x7fe8c0009b90).stop 2021-02-10T11:33:39.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 -- 172.21.2.117:0/2137235731 shutdown_connections 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 --2- 172.21.2.117:0/2137235731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c80e51a0 0x7fe8c8079000 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 --2- 172.21.2.117:0/2137235731 >> v2:172.21.2.125:3300/0 conn(0x7fe8c807da20 0x7fe8c80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 -- 172.21.2.117:0/2137235731 >> 172.21.2.117:0/2137235731 conn(0x7fe8c80ce450 msgr2=0x7fe8c80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 -- 172.21.2.117:0/2137235731 shutdown_connections 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.600 7fe8cf167700 1 -- 172.21.2.117:0/2137235731 wait complete. 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 Processor -- start 2021-02-10T11:33:39.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 -- start start 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8c80c2420 0x7fe8c80c2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8c80a5c60 0x7fe8c80a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80a77c0 con 0x7fe8c81a4690 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80a66b0 con 0x7fe8c80c2420 2021-02-10T11:33:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8cf167700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80a68d0 con 0x7fe8c80a5c60 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8ccf83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8ccf83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44944/0 (socket says 172.21.2.117:44944) 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.615 7fe8ccf83700 1 -- 172.21.2.117:0/3902694665 learned_addr learned my addr 172.21.2.117:0/3902694665 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 -- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8c80a5c60 msgr2=0x7fe8c80a6060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8c80a5c60 0x7fe8c80a6060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 -- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8c80c2420 msgr2=0x7fe8c80c2820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8c80c2420 0x7fe8c80c2820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 -- 172.21.2.117:0/3902694665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c80a6af0 con 0x7fe8c81a4690 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8ccf83700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fe8b0006f00 tx=0x7fe8b000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b000bd30 con 0x7fe8c81a4690 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b0002030 con 0x7fe8c81a4690 2021-02-10T11:33:39.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c80a6af0 con 0x7fe8c81a4690 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c80a6af0 con 0x7fe8c81a4690 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b0002590 con 0x7fe8c81a4690 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8b0002210 con 0x7fe8c81a4690 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.616 7fe8c57fa700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 0x7fe8ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.617 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe8b0032760 con 0x7fe8c81a4690 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.617 7fe8c7fff700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 0x7fe8ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.618 7fe8c7fff700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 0x7fe8ac01ac10 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:39.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.648 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c8103350 con 0x7fe8c81a4690 2021-02-10T11:33:39.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.651 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8b0002240 con 0x7fe8c81a4690 2021-02-10T11:33:39.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.777 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802500", "target": ["mgr", ""]}) v1 -- 0x7fe8c8103350 con 0x7fe8ac018730 2021-02-10T11:33:39.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.803 7fe8c57fa700 1 -- 172.21.2.117:0/3902694665 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe8b8000fe0 con 0x7fe8ac018730 2021-02-10T11:33:39.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 msgr2=0x7fe8ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 0x7fe8ac01ac10 crc :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 msgr2=0x7fe8c80a7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fe8b0006f00 tx=0x7fe8b000b940).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 shutdown_connections 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8c80a5c60 0x7fe8c80a6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8c80c2420 0x7fe8c80c2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8ac018730 0x7fe8ac01ac10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 --2- 172.21.2.117:0/3902694665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8c81a4690 0x7fe8c80a7090 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 >> 172.21.2.117:0/3902694665 conn(0x7fe8c80a19a0 msgr2=0x7fe8c80b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.810 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 shutdown_connections 2021-02-10T11:33:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:39.811 7fe8cf167700 1 -- 172.21.2.117:0/3902694665 wait complete. 2021-02-10T11:33:39.806 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802500 2021-02-10T11:33:40.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.234 7f6973ca3700 1 Processor -- start 2021-02-10T11:33:40.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.234 7f6973ca3700 1 -- start start 2021-02-10T11:33:40.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6973ca3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f696c07da20 0x7f696c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6973ca3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c0e51a0 0x7f696c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6973ca3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c077a70 con 0x7f696c0e4da0 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6973ca3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0777d0 con 0x7f696c07da20 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6973ca3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1b08e0 con 0x7f696c0e51a0 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f696c07da20 0x7f696c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f696c07da20 0x7f696c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44948/0 (socket says 172.21.2.117:44948) 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 -- 172.21.2.117:0/1741566368 learned_addr learned my addr 172.21.2.117:0/1741566368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 -- 172.21.2.117:0/1741566368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c0e51a0 msgr2=0x7f696c079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 --2- 172.21.2.117:0/1741566368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c0e51a0 0x7f696c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 -- 172.21.2.117:0/1741566368 >> v1:172.21.2.125:6789/0 conn(0x7f696c0e4da0 legacy=0x7f696c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.235 7f6971abf700 1 -- 172.21.2.117:0/1741566368 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c07de20 con 0x7f696c07da20 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6971abf700 1 --2- 172.21.2.117:0/1741566368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c07da20 0x7f696c0771b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6968006700 tx=0x7f6968009b80).ready entity=mon.0 client_cookie=5e0a1482d76a5ded server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6970abd700 1 -- 172.21.2.117:0/1741566368 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f696800c0c0 con 0x7f696c07da20 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6970abd700 1 -- 172.21.2.117:0/1741566368 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696800c440 con 0x7f696c07da20 2021-02-10T11:33:40.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6970abd700 1 -- 172.21.2.117:0/1741566368 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f696800c960 con 0x7f696c07da20 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6973ca3700 1 -- 172.21.2.117:0/1741566368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c07da20 msgr2=0x7f696c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.236 7f6973ca3700 1 --2- 172.21.2.117:0/1741566368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c07da20 0x7f696c0771b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f6968006700 tx=0x7f6968009b80).stop 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 -- 172.21.2.117:0/1741566368 shutdown_connections 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 --2- 172.21.2.117:0/1741566368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c0e51a0 0x7f696c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 --2- 172.21.2.117:0/1741566368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c07da20 0x7f696c0771b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 -- 172.21.2.117:0/1741566368 >> 172.21.2.117:0/1741566368 conn(0x7f696c0ce450 msgr2=0x7f696c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 -- 172.21.2.117:0/1741566368 shutdown_connections 2021-02-10T11:33:40.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.237 7f6973ca3700 1 -- 172.21.2.117:0/1741566368 wait complete. 2021-02-10T11:33:40.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.251 7f6973ca3700 1 Processor -- start 2021-02-10T11:33:40.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.251 7f6973ca3700 1 -- start start 2021-02-10T11:33:40.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f696c107c20 0x7f696c096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c08a180 0x7f696c08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c108020 con 0x7f696c1a7240 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1a7c60 con 0x7f696c107c20 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6973ca3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1a7e80 con 0x7f696c08a180 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f69722c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f69722c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44956/0 (socket says 172.21.2.117:44956) 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f69722c0700 1 -- 172.21.2.117:0/130181716 learned_addr learned my addr 172.21.2.117:0/130181716 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:40.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f69712be700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c08a180 0x7f696c08a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.252 7f6971abf700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f696c107c20 0x7f696c096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 -- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c08a180 msgr2=0x7f696c08a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c08a180 0x7f696c08a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 -- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f696c107c20 msgr2=0x7f696c096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f696c107c20 0x7f696c096fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 -- 172.21.2.117:0/130181716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c08b250 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69722c0700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f6964009980 tx=0x7f696400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6964005af0 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6964005d70 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69640062b0 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f6973ca3700 1 -- 172.21.2.117:0/130181716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c08b250 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.253 7f6973ca3700 1 -- 172.21.2.117:0/130181716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c08b250 con 0x7f696c1a7240 2021-02-10T11:33:40.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.254 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6964006030 con 0x7f696c1a7240 2021-02-10T11:33:40.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.254 7f69627fc700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 0x7f695401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:40.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.254 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6964032a50 con 0x7f696c1a7240 2021-02-10T11:33:40.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.255 7f6971abf700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 0x7f695401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:40.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.255 7f6971abf700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 0x7f695401abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:40.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.285 7f6973ca3700 1 -- 172.21.2.117:0/130181716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c077a70 con 0x7f696c1a7240 2021-02-10T11:33:40.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.287 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6964005af0 con 0x7f696c1a7240 2021-02-10T11:33:40.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.415 7f6973ca3700 1 -- 172.21.2.117:0/130181716 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802500", "target": ["mgr", ""]}) v1 -- 0x7f696c077a70 con 0x7f69540186c0 2021-02-10T11:33:40.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.420 7f69627fc700 1 -- 172.21.2.117:0/130181716 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6968006430 con 0x7f69540186c0 2021-02-10T11:33:40.410 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618 2021-02-10T11:33:40.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 msgr2=0x7f695401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 0x7f695401abc0 crc :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 msgr2=0x7f696c1a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:40.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f6964009980 tx=0x7f696400bcd0).stop 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 shutdown_connections 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69540186c0 0x7f695401abc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f696c1a7240 0x7f696c1a7640 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f696c08a180 0x7f696c08a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 --2- 172.21.2.117:0/130181716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f696c107c20 0x7f696c096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 >> 172.21.2.117:0/130181716 conn(0x7f696c0a1880 msgr2=0x7f696c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 shutdown_connections 2021-02-10T11:33:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:40.425 7f6973ca3700 1 -- 172.21.2.117:0/130181716 wait complete. 2021-02-10T11:33:40.420 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802500 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618 2021-02-10T11:33:40.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:40.420 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:40.420 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:40.757 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:40.757 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:40.758 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:41.140 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:41.141 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:41.141 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251114 s, 41.8 MB/s 2021-02-10T11:33:41.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:41.143 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:41.143 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:41.479 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:41.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:41.480 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:41.857 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:41.858 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:41.858 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02512 s, 41.7 MB/s 2021-02-10T11:33:41.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:41.859 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:41.859 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:42.193 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:42.194 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:42.194 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:42.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:42.567 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:42.567 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241868 s, 43.4 MB/s 2021-02-10T11:33:42.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:42.568 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:42.568 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:42.904 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:42.905 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:42.905 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230829 s, 45.4 MB/s 2021-02-10T11:33:43.281 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:43.282 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:43.282 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:43.617 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:43.617 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:43.617 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:43.991 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:43.992 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:43.992 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235044 s, 44.6 MB/s 2021-02-10T11:33:43.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:43.993 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:43.993 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:44.328 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:44.328 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:44.328 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:44.706 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261978 s, 40.0 MB/s 2021-02-10T11:33:44.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:44.707 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:44.707 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:45.043 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:45.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:45.044 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:45.417 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:45.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:45.418 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231749 s, 45.2 MB/s 2021-02-10T11:33:45.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:45.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:45.420 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:45.755 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:45.756 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:45.756 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:46.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:46.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:46.133 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240068 s, 43.7 MB/s 2021-02-10T11:33:46.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:46.134 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:46.134 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:46.469 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:46.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:46.470 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259013 s, 40.5 MB/s 2021-02-10T11:33:46.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:46.849 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:46.849 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:47.184 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:47.184 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:47.184 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802500/06cc419a-4926-438e-9a40-9d2688a11618/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:47.557 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245953 s, 42.6 MB/s 2021-02-10T11:33:47.558 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802501 2021-02-10T11:33:47.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 Processor -- start 2021-02-10T11:33:47.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 -- start start 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fba6c07da20 0x7fba6c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba6c0e4da0 0x7fba6c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6c077a70 con 0x7fba6c0e51a0 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c0777d0 con 0x7fba6c0e4da0 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba722a2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6c07de20 con 0x7fba6c07da20 2021-02-10T11:33:47.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba6b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba6c0e4da0 0x7fba6c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba6b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba6c0e4da0 0x7fba6c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44964/0 (socket says 172.21.2.117:44964) 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.986 7fba6b7fe700 1 -- 172.21.2.117:0/2857041823 learned_addr learned my addr 172.21.2.117:0/2857041823 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6b7fe700 1 -- 172.21.2.117:0/2857041823 >> v2:172.21.2.125:3300/0 conn(0x7fba6c07da20 msgr2=0x7fba6c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6b7fe700 1 --2- 172.21.2.117:0/2857041823 >> v2:172.21.2.125:3300/0 conn(0x7fba6c07da20 0x7fba6c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6b7fe700 1 -- 172.21.2.117:0/2857041823 >> v1:172.21.2.125:6789/0 conn(0x7fba6c0e51a0 legacy=0x7fba6c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6b7fe700 1 -- 172.21.2.117:0/2857041823 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c1b0710 con 0x7fba6c0e4da0 2021-02-10T11:33:47.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6b7fe700 1 --2- 172.21.2.117:0/2857041823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0e4da0 0x7fba6c1b00f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fba60006700 tx=0x7fba60009b80).ready entity=mon.0 client_cookie=3b831c3ecb5b278e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6affd700 1 -- 172.21.2.117:0/2857041823 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba6000c0c0 con 0x7fba6c0e4da0 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6affd700 1 -- 172.21.2.117:0/2857041823 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6000c340 con 0x7fba6c0e4da0 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba6affd700 1 -- 172.21.2.117:0/2857041823 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba6000c860 con 0x7fba6c0e4da0 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba722a2700 1 -- 172.21.2.117:0/2857041823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0e4da0 msgr2=0x7fba6c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.987 7fba722a2700 1 --2- 172.21.2.117:0/2857041823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0e4da0 0x7fba6c1b00f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fba60006700 tx=0x7fba60009b80).stop 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 -- 172.21.2.117:0/2857041823 shutdown_connections 2021-02-10T11:33:47.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 --2- 172.21.2.117:0/2857041823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0e4da0 0x7fba6c1b00f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 --2- 172.21.2.117:0/2857041823 >> v2:172.21.2.125:3300/0 conn(0x7fba6c07da20 0x7fba6c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 -- 172.21.2.117:0/2857041823 >> 172.21.2.117:0/2857041823 conn(0x7fba6c0ce450 msgr2=0x7fba6c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 -- 172.21.2.117:0/2857041823 shutdown_connections 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:47.988 7fba722a2700 1 -- 172.21.2.117:0/2857041823 wait complete. 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.002 7fba722a2700 1 Processor -- start 2021-02-10T11:33:47.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.002 7fba722a2700 1 -- start start 2021-02-10T11:33:47.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba6c0b0f40 0x7fba6c107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba6c18b830 0x7fba6c187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c0b18c0 con 0x7fba6c0b1340 2021-02-10T11:33:47.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c18ac90 con 0x7fba6c0b0f40 2021-02-10T11:33:47.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba722a2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c107f50 con 0x7fba6c18b830 2021-02-10T11:33:47.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44968/0 (socket says 172.21.2.117:44968) 2021-02-10T11:33:47.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 -- 172.21.2.117:0/3991141878 learned_addr learned my addr 172.21.2.117:0/3991141878 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba708bf700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba6c18b830 0x7fba6c187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 -- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba6c18b830 msgr2=0x7fba6c187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba6c18b830 0x7fba6c187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 -- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba6c0b0f40 msgr2=0x7fba6c107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba6c0b0f40 0x7fba6c107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 -- 172.21.2.117:0/3991141878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c108130 con 0x7fba6c0b1340 2021-02-10T11:33:47.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba6b7fe700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fba6000b5e0 tx=0x7fba6000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba6000bd70 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.003 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60002030 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba722a2700 1 -- 172.21.2.117:0/3991141878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c108130 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba60002550 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba722a2700 1 -- 172.21.2.117:0/3991141878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c188570 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba60002210 con 0x7fba6c0b1340 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba68ff9700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 0x7fba4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:47.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.004 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fba60032770 con 0x7fba6c0b1340 2021-02-10T11:33:48.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.005 7fba6bfff700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 0x7fba4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.005 7fba6bfff700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 0x7fba4c01abc0 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.036 7fba722a2700 1 -- 172.21.2.117:0/3991141878 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c188a50 con 0x7fba6c0b1340 2021-02-10T11:33:48.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.039 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba6001f4e0 con 0x7fba6c0b1340 2021-02-10T11:33:48.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.162 7fba722a2700 1 -- 172.21.2.117:0/3991141878 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802501", "target": ["mgr", ""]}) v1 -- 0x7fba6c188a50 con 0x7fba4c0186c0 2021-02-10T11:33:48.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.189 7fba68ff9700 1 -- 172.21.2.117:0/3991141878 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba5c006e60 con 0x7fba4c0186c0 2021-02-10T11:33:48.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 msgr2=0x7fba4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 0x7fba4c01abc0 crc :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 msgr2=0x7fba6c18b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fba6000b5e0 tx=0x7fba6000b980).stop 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 shutdown_connections 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba4c0186c0 0x7fba4c01abc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba6c0b1340 0x7fba6c18b210 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba6c18b830 0x7fba6c187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 --2- 172.21.2.117:0/3991141878 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba6c0b0f40 0x7fba6c107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:48.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 >> 172.21.2.117:0/3991141878 conn(0x7fba6c0a1880 msgr2=0x7fba6c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 shutdown_connections 2021-02-10T11:33:48.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.196 7fba722a2700 1 -- 172.21.2.117:0/3991141878 wait complete. 2021-02-10T11:33:48.192 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802501 2021-02-10T11:33:48.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.621 7fb100f64700 1 Processor -- start 2021-02-10T11:33:48.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.621 7fb100f64700 1 -- start start 2021-02-10T11:33:48.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.621 7fb100f64700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb0fc07da20 0x7fb0fc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb100f64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0e51a0 0x7fb0fc079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb100f64700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc077a70 con 0x7fb0fc0e4da0 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb100f64700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc0777d0 con 0x7fb0fc07da20 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb100f64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc1b08e0 con 0x7fb0fc0e51a0 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb0fc07da20 0x7fb0fc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb0fc07da20 0x7fb0fc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44974/0 (socket says 172.21.2.117:44974) 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 -- 172.21.2.117:0/823218665 learned_addr learned my addr 172.21.2.117:0/823218665 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 -- 172.21.2.117:0/823218665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0e51a0 msgr2=0x7fb0fc079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 --2- 172.21.2.117:0/823218665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0e51a0 0x7fb0fc079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 -- 172.21.2.117:0/823218665 >> v1:172.21.2.117:6789/0 conn(0x7fb0fc0e4da0 legacy=0x7fb0fc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 -- 172.21.2.117:0/823218665 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc07de20 con 0x7fb0fc07da20 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0fab13700 1 --2- 172.21.2.117:0/823218665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc07da20 0x7fb0fc0771b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fb0ec006700 tx=0x7fb0ec009b80).ready entity=mon.0 client_cookie=eb7f2849a4630571 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.622 7fb0f9b11700 1 -- 172.21.2.117:0/823218665 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0ec00c0c0 con 0x7fb0fc07da20 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb0f9b11700 1 -- 172.21.2.117:0/823218665 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ec00c440 con 0x7fb0fc07da20 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb0f9b11700 1 -- 172.21.2.117:0/823218665 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0ec00c960 con 0x7fb0fc07da20 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 -- 172.21.2.117:0/823218665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc07da20 msgr2=0x7fb0fc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 --2- 172.21.2.117:0/823218665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc07da20 0x7fb0fc0771b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fb0ec006700 tx=0x7fb0ec009b80).stop 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 -- 172.21.2.117:0/823218665 shutdown_connections 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 --2- 172.21.2.117:0/823218665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0e51a0 0x7fb0fc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 --2- 172.21.2.117:0/823218665 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc07da20 0x7fb0fc0771b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 -- 172.21.2.117:0/823218665 >> 172.21.2.117:0/823218665 conn(0x7fb0fc0ce450 msgr2=0x7fb0fc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 -- 172.21.2.117:0/823218665 shutdown_connections 2021-02-10T11:33:48.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.623 7fb100f64700 1 -- 172.21.2.117:0/823218665 wait complete. 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 Processor -- start 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 -- start start 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 0x7fb0fc107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 0x7fb0fc18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0b18c0 con 0x7fb0fc18b830 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc18ac90 con 0x7fb0fc0b0f40 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb100f64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc107f50 con 0x7fb0fc0b1340 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb0fb314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb0fb314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44982/0 (socket says 172.21.2.117:44982) 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.638 7fb0fb314700 1 -- 172.21.2.117:0/1726238036 learned_addr learned my addr 172.21.2.117:0/1726238036 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fa312700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 0x7fb0fc18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fab13700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 0x7fb0fc107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 -- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 msgr2=0x7fb0fc18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 0x7fb0fc18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 -- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 msgr2=0x7fb0fc107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 0x7fb0fc107930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 -- 172.21.2.117:0/1726238036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc108130 con 0x7fb0fc18b830 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fa312700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 0x7fb0fc18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:48.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fb314700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fb0f4009980 tx=0x7fb0f400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0fab13700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 0x7fb0fc107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0f4003020 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f4005a80 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb100f64700 1 -- 172.21.2.117:0/1726238036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc108130 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0f4006030 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.639 7fb100f64700 1 -- 172.21.2.117:0/1726238036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc108130 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.640 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb0f4005c60 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.640 7fb0f37fe700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 0x7fb0e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.640 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb0f4032b30 con 0x7fb0fc18b830 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.640 7fb0fab13700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 0x7fb0e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:48.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.640 7fb0fab13700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 0x7fb0e001abc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:48.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.671 7fb100f64700 1 -- 172.21.2.117:0/1726238036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc077a70 con 0x7fb0fc18b830 2021-02-10T11:33:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.674 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb0f4005870 con 0x7fb0fc18b830 2021-02-10T11:33:48.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.800 7fb100f64700 1 -- 172.21.2.117:0/1726238036 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802501", "target": ["mgr", ""]}) v1 -- 0x7fb0fc077a70 con 0x7fb0e0018730 2021-02-10T11:33:48.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.806 7fb0f37fe700 1 -- 172.21.2.117:0/1726238036 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb0ec0009b0 con 0x7fb0e0018730 2021-02-10T11:33:48.795 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 msgr2=0x7fb0e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 0x7fb0e001abc0 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 msgr2=0x7fb0fc187bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fb0f4009980 tx=0x7fb0f400bcd0).stop 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 shutdown_connections 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb0e0018730 0x7fb0e001abc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb0fc18b830 0x7fb0fc187bf0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0fc0b1340 0x7fb0fc18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 --2- 172.21.2.117:0/1726238036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0fc0b0f40 0x7fb0fc107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:48.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 >> 172.21.2.117:0/1726238036 conn(0x7fb0fc0a1880 msgr2=0x7fb0fc0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:48.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 shutdown_connections 2021-02-10T11:33:48.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:48.810 7fb100f64700 1 -- 172.21.2.117:0/1726238036 wait complete. 2021-02-10T11:33:48.808 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802501 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955 2021-02-10T11:33:48.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:48.808 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:48.808 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:49.145 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:49.145 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:49.145 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:49.527 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024781 s, 42.3 MB/s 2021-02-10T11:33:49.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:49.529 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:49.529 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:49.865 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:49.866 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:49.866 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:50.241 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:50.242 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:50.242 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235109 s, 44.6 MB/s 2021-02-10T11:33:50.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:50.243 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:50.243 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:50.579 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:50.579 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:50.579 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250919 s, 41.8 MB/s 2021-02-10T11:33:50.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:50.956 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:50.956 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:51.292 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:51.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:51.292 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:51.668 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:51.669 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:51.669 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234706 s, 44.7 MB/s 2021-02-10T11:33:51.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:51.670 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:51.670 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:52.006 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:52.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:52.006 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:52.385 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:52.385 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:52.385 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255745 s, 41.0 MB/s 2021-02-10T11:33:52.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:52.386 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:52.386 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:52.722 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:52.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:52.722 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:53.094 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:53.094 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:53.095 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023311 s, 45.0 MB/s 2021-02-10T11:33:53.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:53.096 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:53.096 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:53.431 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:53.431 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:53.431 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:53.807 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:53.807 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:53.807 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235058 s, 44.6 MB/s 2021-02-10T11:33:53.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:53.808 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:53.808 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:54.145 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:54.145 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:54.145 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:54.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:54.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:54.519 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227387 s, 46.1 MB/s 2021-02-10T11:33:54.520 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:54.520 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:54.520 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:54.857 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:54.857 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:54.857 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:55.234 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248858 s, 42.1 MB/s 2021-02-10T11:33:55.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:55.235 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:55.235 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:55.571 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:55.571 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:55.572 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802501/96834a77-fc7d-4087-b561-7a2602db3955/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262257 s, 40.0 MB/s 2021-02-10T11:33:55.951 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802502 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 Processor -- start 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 -- start start 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b2007da20 0x7f3b200771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200e51a0 0x7f3b20079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b20077a70 con 0x7f3b200e4da0 2021-02-10T11:33:56.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b200777d0 con 0x7f3b2007da20 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b24acd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b201b08e0 con 0x7f3b200e51a0 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b1e85e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b2007da20 0x7f3b200771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b1e85e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b2007da20 0x7f3b200771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44986/0 (socket says 172.21.2.117:44986) 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.380 7f3b1e85e700 1 -- 172.21.2.117:0/2170935368 learned_addr learned my addr 172.21.2.117:0/2170935368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1f05f700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200e51a0 0x7f3b20079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1e85e700 1 -- 172.21.2.117:0/2170935368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200e51a0 msgr2=0x7f3b20079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1e85e700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200e51a0 0x7f3b20079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1e85e700 1 -- 172.21.2.117:0/2170935368 >> v1:172.21.2.125:6789/0 conn(0x7f3b200e4da0 legacy=0x7f3b201b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1e85e700 1 -- 172.21.2.117:0/2170935368 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2007de20 con 0x7f3b2007da20 2021-02-10T11:33:56.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1e85e700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b2007da20 0x7f3b200771b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f3b08006700 tx=0x7f3b08009b80).ready entity=mon.0 client_cookie=1a21fe24f2e0c227 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1d85c700 1 -- 172.21.2.117:0/2170935368 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b0800c0c0 con 0x7f3b2007da20 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1d85c700 1 -- 172.21.2.117:0/2170935368 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0800c340 con 0x7f3b2007da20 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b1d85c700 1 -- 172.21.2.117:0/2170935368 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b0800c860 con 0x7f3b2007da20 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b24acd700 1 -- 172.21.2.117:0/2170935368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b2007da20 msgr2=0x7f3b200771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.381 7f3b24acd700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b2007da20 0x7f3b200771b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f3b08006700 tx=0x7f3b08009b80).stop 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 -- 172.21.2.117:0/2170935368 shutdown_connections 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200e51a0 0x7f3b20079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 --2- 172.21.2.117:0/2170935368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b2007da20 0x7f3b200771b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 -- 172.21.2.117:0/2170935368 >> 172.21.2.117:0/2170935368 conn(0x7f3b200ce450 msgr2=0x7f3b200e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 -- 172.21.2.117:0/2170935368 shutdown_connections 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.382 7f3b24acd700 1 -- 172.21.2.117:0/2170935368 wait complete. 2021-02-10T11:33:56.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.396 7f3b24acd700 1 Processor -- start 2021-02-10T11:33:56.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.396 7f3b24acd700 1 -- start start 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b200c2340 0x7f3b200c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200b7390 0x7f3b200b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200b8010 con 0x7f3b200a41c0 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200b8550 con 0x7f3b200c2340 2021-02-10T11:33:56.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b24acd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200b8770 con 0x7f3b200b7390 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:44996/0 (socket says 172.21.2.117:44996) 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 -- 172.21.2.117:0/1168653050 learned_addr learned my addr 172.21.2.117:0/1168653050 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 -- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200b7390 msgr2=0x7f3b200b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200b7390 0x7f3b200b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 -- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b200c2340 msgr2=0x7f3b200c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b200c2340 0x7f3b200c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 -- 172.21.2.117:0/1168653050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b200b8990 con 0x7f3b200a41c0 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.397 7f3b1e85e700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f3b0800b5e0 tx=0x7f3b0800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b0800bd70 con 0x7f3b200a41c0 2021-02-10T11:33:56.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b200b8990 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b200b8f20 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0800bd70 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b08002030 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b08000c90 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.398 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b08032930 con 0x7f3b200a41c0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.399 7f3b1e05d700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:56.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.399 7f3b1e05d700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 0x7f3b0401ac10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:56.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.430 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b201b4f00 con 0x7f3b200a41c0 2021-02-10T11:33:56.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.433 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b080022f0 con 0x7f3b200a41c0 2021-02-10T11:33:56.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.556 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802502", "target": ["mgr", ""]}) v1 -- 0x7f3b20077a70 con 0x7f3b04018730 2021-02-10T11:33:56.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.582 7f3b177fe700 1 -- 172.21.2.117:0/1168653050 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3b10006a20 con 0x7f3b04018730 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 msgr2=0x7f3b0401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 0x7f3b0401ac10 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 msgr2=0x7f3b200fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f3b0800b5e0 tx=0x7f3b0800b980).stop 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 shutdown_connections 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b200b7390 0x7f3b200b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b200c2340 0x7f3b200c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:33:56.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b04018730 0x7f3b0401ac10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 --2- 172.21.2.117:0/1168653050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b200a41c0 0x7f3b200fe0d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:56.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 >> 172.21.2.117:0/1168653050 conn(0x7f3b2008bbc0 msgr2=0x7f3b200c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:56.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 shutdown_connections 2021-02-10T11:33:56.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:56.590 7f3b24acd700 1 -- 172.21.2.117:0/1168653050 wait complete. 2021-02-10T11:33:56.585 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802502 2021-02-10T11:33:57.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 Processor -- start 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 -- start start 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 0x7f15880e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15880d1d20 con 0x7f15881b00f0 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1588077a70 con 0x7f158807da20 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f158d5a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15880e5430 con 0x7f1588079000 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f1587ac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f15881b00f0 0x7f15880d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45254/0 (socket says 172.21.2.117:45254) 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f1587ac0700 1 -- 172.21.2.117:0/794488611 learned_addr learned my addr 172.21.2.117:0/794488611 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.015 7f1586abe700 1 --2- 172.21.2.117:0/794488611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 0x7f15880e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f15862bd700 1 -- 172.21.2.117:0/794488611 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659554325 0 0) 0x7f1580001500 con 0x7f15881b00f0 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f15862bd700 1 -- 172.21.2.117:0/794488611 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1574002fe0 con 0x7f15881b00f0 2021-02-10T11:33:57.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f1586abe700 1 -- 172.21.2.117:0/794488611 >> v1:172.21.2.125:6789/0 conn(0x7f158807da20 legacy=0x7f15880771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f1586abe700 1 -- 172.21.2.117:0/794488611 >> v1:172.21.2.117:6789/0 conn(0x7f15881b00f0 legacy=0x7f15880d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f1586abe700 1 -- 172.21.2.117:0/794488611 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588079400 con 0x7f1588079000 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f1586abe700 1 --2- 172.21.2.117:0/794488611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 0x7f15880e4da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1578006650 tx=0x7f1578009a80).ready entity=mon.2 client_cookie=fef1908f4ed10fa0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f15862bd700 1 -- 172.21.2.117:0/794488611 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157800c0c0 con 0x7f1588079000 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f15862bd700 1 -- 172.21.2.117:0/794488611 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157800c340 con 0x7f1588079000 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f15862bd700 1 -- 172.21.2.117:0/794488611 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157800c860 con 0x7f1588079000 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f158d5a3700 1 -- 172.21.2.117:0/794488611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 msgr2=0x7f15880e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.016 7f158d5a3700 1 --2- 172.21.2.117:0/794488611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 0x7f15880e4da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1578006650 tx=0x7f1578009a80).stop 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.017 7f158d5a3700 1 -- 172.21.2.117:0/794488611 shutdown_connections 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.017 7f158d5a3700 1 --2- 172.21.2.117:0/794488611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1588079000 0x7f15880e4da0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.017 7f158d5a3700 1 -- 172.21.2.117:0/794488611 >> 172.21.2.117:0/794488611 conn(0x7f15880ce450 msgr2=0x7f15880e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.017 7f158d5a3700 1 -- 172.21.2.117:0/794488611 shutdown_connections 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.017 7f158d5a3700 1 -- 172.21.2.117:0/794488611 wait complete. 2021-02-10T11:33:57.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 Processor -- start 2021-02-10T11:33:57.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 -- start start 2021-02-10T11:33:57.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 0x7f1588107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 0x7f1588187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15880b18c0 con 0x7f15880b1340 2021-02-10T11:33:57.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158818ac90 con 0x7f158818b830 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f158d5a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1588107f50 con 0x7f15880b0f40 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f1586abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f1586abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45006/0 (socket says 172.21.2.117:45006) 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.032 7f1586abe700 1 -- 172.21.2.117:0/2350402651 learned_addr learned my addr 172.21.2.117:0/2350402651 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 -- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 msgr2=0x7f1588107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f15872bf700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 0x7f1588107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1587ac0700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 0x7f1588187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 0x7f1588107930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 -- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 msgr2=0x7f1588187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 0x7f1588187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 -- 172.21.2.117:0/2350402651 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588108130 con 0x7f15880b1340 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f15872bf700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 0x7f1588107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1587ac0700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 0x7f1588187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:33:57.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f1586abe700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f1578009650 tx=0x7f157800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157800bd50 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1578006030 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1588108130 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1578005560 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.033 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1588108130 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.034 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15780052e0 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.034 7f157ffff700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 0x7f156c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.034 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1578032790 con 0x7f15880b1340 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.034 7f15872bf700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 0x7f156c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:33:57.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.034 7f15872bf700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 0x7f156c01abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:33:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.065 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1588077a70 con 0x7f15880b1340 2021-02-10T11:33:57.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.068 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15780009a0 con 0x7f15880b1340 2021-02-10T11:33:57.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.193 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802502", "target": ["mgr", ""]}) v1 -- 0x7f1588077a70 con 0x7f156c018730 2021-02-10T11:33:57.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.198 7f157ffff700 1 -- 172.21.2.117:0/2350402651 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1570006b10 con 0x7f156c018730 2021-02-10T11:33:57.188 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf 2021-02-10T11:33:57.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 msgr2=0x7f156c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 0x7f156c01abc0 crc :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 msgr2=0x7f158818b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f1578009650 tx=0x7f157800b960).stop 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 shutdown_connections 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c018730 0x7f156c01abc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880b1340 0x7f158818b210 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158818b830 0x7f1588187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 --2- 172.21.2.117:0/2350402651 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880b0f40 0x7f1588107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:33:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 >> 172.21.2.117:0/2350402651 conn(0x7f15880a1880 msgr2=0x7f15880b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:33:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 shutdown_connections 2021-02-10T11:33:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:33:57.203 7f158d5a3700 1 -- 172.21.2.117:0/2350402651 wait complete. 2021-02-10T11:33:57.198 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802502 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf 2021-02-10T11:33:57.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:57.198 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:57.198 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:57.536 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:57.536 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:57.536 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:57.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:57.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:57.918 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025457 s, 41.2 MB/s 2021-02-10T11:33:57.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:57.919 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:57.919 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:58.254 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:58.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:58.255 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:58.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:58.629 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:58.629 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238337 s, 44.0 MB/s 2021-02-10T11:33:58.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:58.630 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:58.631 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:58.965 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:58.966 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:58.966 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:33:59.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:33:59.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:33:59.340 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229669 s, 45.7 MB/s 2021-02-10T11:33:59.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:33:59.341 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:33:59.341 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:33:59.677 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:33:59.677 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:33:59.678 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:00.053 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:00.053 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:00.053 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235509 s, 44.5 MB/s 2021-02-10T11:34:00.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:00.055 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:00.055 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:00.391 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:00.392 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:00.392 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:00.742 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:00.746 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:00.770 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:00.770 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:00.770 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262655 s, 39.9 MB/s 2021-02-10T11:34:00.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:00.772 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:00.772 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:01.084 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:01.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:01.084 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:01.459 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:01.459 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:01.459 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235463 s, 44.5 MB/s 2021-02-10T11:34:01.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:01.460 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:01.460 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:01.796 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:01.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:01.797 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:02.173 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:02.173 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:02.174 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261021 s, 40.2 MB/s 2021-02-10T11:34:02.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:02.175 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:02.175 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:02.510 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:02.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:02.510 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:02.884 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:02.885 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:02.885 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258735 s, 40.5 MB/s 2021-02-10T11:34:02.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:02.886 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:02.886 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:03.223 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:03.223 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:03.223 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:03.598 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:03.598 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:03.598 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237708 s, 44.1 MB/s 2021-02-10T11:34:03.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:03.599 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:03.599 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:03.934 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:03.935 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:03.935 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802502/216e7ece-b3ad-457e-8d33-b5f0af47cebf/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025851 s, 40.6 MB/s 2021-02-10T11:34:04.314 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802503 2021-02-10T11:34:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.743 7fb9e19cd700 1 Processor -- start 2021-02-10T11:34:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.743 7fb9e19cd700 1 -- start start 2021-02-10T11:34:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9e19cd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0d1530 0x7fb9dc0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9e19cd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc077a70 con 0x7fb9dc079000 2021-02-10T11:34:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9e19cd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc079400 con 0x7fb9dc07da20 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9e19cd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc07de20 con 0x7fb9dc0d1530 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9dad73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb9dc079000 0x7fb9dc0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45264/0 (socket says 172.21.2.117:45264) 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9dad73700 1 -- 172.21.2.117:0/2987054419 learned_addr learned my addr 172.21.2.117:0/2987054419 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169017121 0 0) 0x7fb9c4000ce0 con 0x7fb9dc079000 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c8002fe0 con 0x7fb9dc079000 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9dbd75700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0d1530 0x7fb9dc0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2691046273 0 0) 0x7fb9c40010e0 con 0x7fb9dc079000 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0d1530 msgr2=0x7fb9dc0d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0d1530 0x7fb9dc0d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 >> v1:172.21.2.125:6789/0 conn(0x7fb9dc07da20 legacy=0x7fb9dc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc07dfd0 con 0x7fb9dc079000 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fb9c4001600 con 0x7fb9dc079000 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.744 7fb9da572700 1 -- 172.21.2.117:0/2987054419 >> v1:172.21.2.117:6789/0 conn(0x7fb9dc079000 legacy=0x7fb9dc0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9c8003aa0 0x7fb9c8005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9c8006770 0x7fb9c8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9c8009590 0x7fb9c800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c800c2d0 con 0x7fb9c8006770 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c800c4f0 con 0x7fb9c8003aa0 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c800c710 con 0x7fb9c8009590 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9c8006770 0x7fb9c8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9db574700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9c8003aa0 0x7fb9c8005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dbd75700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9c8009590 0x7fb9c800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 -- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9c8009590 msgr2=0x7fb9c800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9c8009590 0x7fb9c800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 -- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9c8003aa0 msgr2=0x7fb9c8005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9c8003aa0 0x7fb9c8005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 -- 172.21.2.117:0/2987054419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c800c9b0 con 0x7fb9c8006770 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dbd75700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9c8009590 0x7fb9c800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dad73700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9c8006770 0x7fb9c8008c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7fb9c40017c0 tx=0x7fb9c404bd00).ready entity=mon.0 client_cookie=1c5477eee9ca56cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c40043a0 con 0x7fb9c8006770 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4004660 con 0x7fb9c8006770 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9da572700 1 -- 172.21.2.117:0/2987054419 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c4004ac0 con 0x7fb9c8006770 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9e19cd700 1 -- 172.21.2.117:0/2987054419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9c8006770 msgr2=0x7fb9c8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9e19cd700 1 --2- 172.21.2.117:0/2987054419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9c8006770 0x7fb9c8008c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7fb9c40017c0 tx=0x7fb9c404bd00).stop 2021-02-10T11:34:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.745 7fb9dbd75700 1 -- 172.21.2.117:0/2987054419 reap_dead start 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.746 7fb9e19cd700 1 -- 172.21.2.117:0/2987054419 shutdown_connections 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.746 7fb9e19cd700 1 -- 172.21.2.117:0/2987054419 >> 172.21.2.117:0/2987054419 conn(0x7fb9dc0ce450 msgr2=0x7fb9dc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.746 7fb9e19cd700 1 -- 172.21.2.117:0/2987054419 shutdown_connections 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.746 7fb9e19cd700 1 -- 172.21.2.117:0/2987054419 wait complete. 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 Processor -- start 2021-02-10T11:34:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 -- start start 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9dc0a47d0 0x7fb9dc0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0a51f0 0x7fb9dc0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc189970 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc190aa0 con 0x7fb9dc0a47d0 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9e19cd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0b81f0 con 0x7fb9dc0a51f0 2021-02-10T11:34:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9dbd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9dbd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45030/0 (socket says 172.21.2.117:45030) 2021-02-10T11:34:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.761 7fb9dbd75700 1 -- 172.21.2.117:0/844724561 learned_addr learned my addr 172.21.2.117:0/844724561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 -- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0a51f0 msgr2=0x7fb9dc0a55f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0a51f0 0x7fb9dc0a55f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 -- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9dc0a47d0 msgr2=0x7fb9dc0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9dc0a47d0 0x7fb9dc0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 -- 172.21.2.117:0/844724561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0b8410 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9dbd75700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb9d400b940 tx=0x7fb9d400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9d4006cd0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d400c2f0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc0b8410 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9d400c810 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.762 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc0b89a0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.763 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9d4006cb0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.763 7fb9d3fff700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 0x7fb9c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:04.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.763 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb9d4032ce0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.763 7fb9db574700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 0x7fb9c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:04.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.764 7fb9db574700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 0x7fb9c001abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:04.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.793 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc077a70 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.796 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9d4006cb0 con 0x7fb9dc0b77d0 2021-02-10T11:34:04.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.921 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802503", "target": ["mgr", ""]}) v1 -- 0x7fb9dc189970 con 0x7fb9c00186c0 2021-02-10T11:34:04.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.949 7fb9d3fff700 1 -- 172.21.2.117:0/844724561 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9cc006a40 con 0x7fb9c00186c0 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 msgr2=0x7fb9c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 0x7fb9c001abc0 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 msgr2=0x7fb9dc0b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb9d400b940 tx=0x7fb9d400bce0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 shutdown_connections 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9c00186c0 0x7fb9c001abc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9dc0b77d0 0x7fb9dc0b7bd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9dc0a51f0 0x7fb9dc0a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 --2- 172.21.2.117:0/844724561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9dc0a47d0 0x7fb9dc0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.954 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 >> 172.21.2.117:0/844724561 conn(0x7fb9dc0b5140 msgr2=0x7fb9dc0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.955 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 shutdown_connections 2021-02-10T11:34:04.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:04.955 7fb9e19cd700 1 -- 172.21.2.117:0/844724561 wait complete. 2021-02-10T11:34:04.949 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802503 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.377 7fd27f8b6700 1 Processor -- start 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.377 7fd27f8b6700 1 -- start start 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27f8b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd27807da20 0x7fd2780771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27f8b6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd2780e51a0 0x7fd2780d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27f8b6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd278077a70 con 0x7fd2780e4da0 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27f8b6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2780e5c20 con 0x7fd2780e51a0 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27f8b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780777d0 con 0x7fd27807da20 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27ced1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd2780e4da0 0x7fd2781b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45284/0 (socket says 172.21.2.117:45284) 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd27ced1700 1 -- 172.21.2.117:0/3944372287 learned_addr learned my addr 172.21.2.117:0/3944372287 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:05.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596834473 0 0) 0x7fd2740010b0 con 0x7fd2780e4da0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.378 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd264002fe0 con 0x7fd2780e4da0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27d6d2700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd27807da20 0x7fd2780771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ded3700 1 --2- 172.21.2.117:0/3944372287 >> v2:172.21.2.125:3300/0 conn(0x7fd2780e51a0 0x7fd2780d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2386185338 0 0) 0x7fd2740010b0 con 0x7fd2780e4da0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd27807da20 msgr2=0x7fd2780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd27807da20 0x7fd2780771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 >> v2:172.21.2.125:3300/0 conn(0x7fd2780e51a0 msgr2=0x7fd2780d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 --2- 172.21.2.117:0/3944372287 >> v2:172.21.2.125:3300/0 conn(0x7fd2780e51a0 0x7fd2780d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2781b08e0 con 0x7fd2780e4da0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd2740015b0 con 0x7fd2780e4da0 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 >> v1:172.21.2.117:6789/0 conn(0x7fd2780e4da0 legacy=0x7fd2781b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd264003a60 0x7fd264005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd264006750 0x7fd264008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd264009510 0x7fd26400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26400c2c0 con 0x7fd264006750 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264050480 con 0x7fd264009510 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2640506a0 con 0x7fd264003a60 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd264006750 0x7fd264008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27d6d2700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd264003a60 0x7fd264005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ded3700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd264009510 0x7fd26400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 -- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd264003a60 msgr2=0x7fd264005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd264003a60 0x7fd264005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 -- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd264009510 msgr2=0x7fd26400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd264009510 0x7fd26400ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 -- 172.21.2.117:0/3944372287 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264050940 con 0x7fd264006750 2021-02-10T11:34:05.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd27ced1700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd264006750 0x7fd264008c70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fd274007990 tx=0x7fd274007d30).ready entity=mon.0 client_cookie=59cd472300bedbd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2740009b0 con 0x7fd264006750 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27400a310 con 0x7fd264006750 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.379 7fd26bfff700 1 -- 172.21.2.117:0/3944372287 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd27400a830 con 0x7fd264006750 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 -- 172.21.2.117:0/3944372287 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd264006750 msgr2=0x7fd264008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 --2- 172.21.2.117:0/3944372287 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd264006750 0x7fd264008c70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7fd274007990 tx=0x7fd274007d30).stop 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27ded3700 1 -- 172.21.2.117:0/3944372287 reap_dead start 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 -- 172.21.2.117:0/3944372287 shutdown_connections 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 -- 172.21.2.117:0/3944372287 >> 172.21.2.117:0/3944372287 conn(0x7fd2780ce450 msgr2=0x7fd2780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 -- 172.21.2.117:0/3944372287 shutdown_connections 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.380 7fd27f8b6700 1 -- 172.21.2.117:0/3944372287 wait complete. 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.395 7fd27f8b6700 1 Processor -- start 2021-02-10T11:34:05.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.395 7fd27f8b6700 1 -- start start 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2780a47d0 0x7fd2780a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2780a51f0 0x7fd2780a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd278189970 con 0x7fd2780b77d0 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd278190aa0 con 0x7fd2780a51f0 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780b81f0 con 0x7fd2780a47d0 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45048/0 (socket says 172.21.2.117:45048) 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 -- 172.21.2.117:0/1617857887 learned_addr learned my addr 172.21.2.117:0/1617857887 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 -- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2780a47d0 msgr2=0x7fd2780a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2780a47d0 0x7fd2780a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 -- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2780a51f0 msgr2=0x7fd2780a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2780a51f0 0x7fd2780a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 -- 172.21.2.117:0/1617857887 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2780b8410 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27ded3700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fd27000b8c0 tx=0x7fd27000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd270006ae0 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270006030 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2780b8410 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.396 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2780b89a0 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.397 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2700056b0 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.397 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2700068f0 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.397 7fd269ffb700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 0x7fd25801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.397 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd270032b40 con 0x7fd2780b77d0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.397 7fd27d6d2700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 0x7fd25801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:05.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.398 7fd27d6d2700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 0x7fd25801abd0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:05.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.428 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd250002fd0 con 0x7fd2780b77d0 2021-02-10T11:34:05.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.430 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd270006030 con 0x7fd2780b77d0 2021-02-10T11:34:05.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.559 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802503", "target": ["mgr", ""]}) v1 -- 0x7fd250002f80 con 0x7fd2580186d0 2021-02-10T11:34:05.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.563 7fd269ffb700 1 -- 172.21.2.117:0/1617857887 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd26c0069f0 con 0x7fd2580186d0 2021-02-10T11:34:05.553 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2 2021-02-10T11:34:05.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.568 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 msgr2=0x7fd25801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.568 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 0x7fd25801abd0 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.568 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 msgr2=0x7fd2780b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:05.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.568 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7fd27000b8c0 tx=0x7fd27000bc60).stop 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 shutdown_connections 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2580186d0 0x7fd25801abd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2780b77d0 0x7fd2780b7bd0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2780a51f0 0x7fd2780a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 --2- 172.21.2.117:0/1617857887 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2780a47d0 0x7fd2780a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 >> 172.21.2.117:0/1617857887 conn(0x7fd2780b5140 msgr2=0x7fd2780a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 shutdown_connections 2021-02-10T11:34:05.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:05.569 7fd27f8b6700 1 -- 172.21.2.117:0/1617857887 wait complete. 2021-02-10T11:34:05.563 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802503 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2 2021-02-10T11:34:05.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:05.564 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:05.564 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:05.900 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:05.901 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:05.901 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:06.283 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:06.283 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:06.283 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248726 s, 42.2 MB/s 2021-02-10T11:34:06.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:06.285 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:06.285 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:06.620 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:06.621 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:06.621 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235185 s, 44.6 MB/s 2021-02-10T11:34:06.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:06.997 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:06.997 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:07.332 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:07.333 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:07.333 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:07.706 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:07.706 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:07.706 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257629 s, 40.7 MB/s 2021-02-10T11:34:07.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:07.707 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:07.707 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:08.043 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:08.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:08.043 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:08.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:08.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:08.418 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239487 s, 43.8 MB/s 2021-02-10T11:34:08.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:08.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:08.419 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:08.755 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:08.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:08.756 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:09.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:09.132 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:09.132 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239268 s, 43.8 MB/s 2021-02-10T11:34:09.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:09.133 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:09.133 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:09.470 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:09.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:09.471 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:09.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255138 s, 41.1 MB/s 2021-02-10T11:34:09.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:09.849 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:09.849 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:10.185 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:10.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:10.186 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242959 s, 43.2 MB/s 2021-02-10T11:34:10.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:10.563 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:10.563 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:10.898 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:10.899 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:10.899 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:11.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:11.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:11.275 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245123 s, 42.8 MB/s 2021-02-10T11:34:11.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:11.276 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:11.276 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:11.612 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:11.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:11.613 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023621 s, 44.4 MB/s 2021-02-10T11:34:11.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:11.990 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:11.990 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:12.325 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:12.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:12.326 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802503/0724dc60-beaf-401b-8f03-bff5bcf28dd2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:12.699 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:12.700 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:12.700 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257984 s, 40.6 MB/s 2021-02-10T11:34:12.701 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802504 2021-02-10T11:34:13.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 Processor -- start 2021-02-10T11:34:13.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 -- start start 2021-02-10T11:34:13.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb318079000 0x7fb3180e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb3181b00f0 0x7fb3181b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb318077a70 con 0x7fb31807da20 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180e53c0 con 0x7fb3181b00f0 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb31daf0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb318079400 con 0x7fb318079000 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.130 7fb317fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb3181b00f0 0x7fb3181b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb3181b00f0 0x7fb3181b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45056/0 (socket says 172.21.2.117:45056) 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 -- 172.21.2.117:0/814516160 learned_addr learned my addr 172.21.2.117:0/814516160 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb316ffd700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb318079000 0x7fb3180e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 -- 172.21.2.117:0/814516160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb318079000 msgr2=0x7fb3180e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb318079000 0x7fb3180e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 -- 172.21.2.117:0/814516160 >> v1:172.21.2.117:6789/0 conn(0x7fb31807da20 legacy=0x7fb3180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 -- 172.21.2.117:0/814516160 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31807de20 con 0x7fb3181b00f0 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb317fff700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b00f0 0x7fb3181b04f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb310006f00 tx=0x7fb310009f10).ready entity=mon.0 client_cookie=af5759da623d89b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.131 7fb3167fc700 1 -- 172.21.2.117:0/814516160 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb31000b3b0 con 0x7fb3181b00f0 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb3167fc700 1 -- 172.21.2.117:0/814516160 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31000b630 con 0x7fb3181b00f0 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb3167fc700 1 -- 172.21.2.117:0/814516160 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb31000ba90 con 0x7fb3181b00f0 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 -- 172.21.2.117:0/814516160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b00f0 msgr2=0x7fb3181b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b00f0 0x7fb3181b04f0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb310006f00 tx=0x7fb310009f10).stop 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 -- 172.21.2.117:0/814516160 shutdown_connections 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b00f0 0x7fb3181b04f0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 --2- 172.21.2.117:0/814516160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb318079000 0x7fb3180e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 -- 172.21.2.117:0/814516160 >> 172.21.2.117:0/814516160 conn(0x7fb3180ce450 msgr2=0x7fb3180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 -- 172.21.2.117:0/814516160 shutdown_connections 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.132 7fb31daf0700 1 -- 172.21.2.117:0/814516160 wait complete. 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.147 7fb31daf0700 1 Processor -- start 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 -- start start 2021-02-10T11:34:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3180952f0 0x7fb3181b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 0x7fb31819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181b7c80 con 0x7fb3181b7760 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318094260 con 0x7fb3181b8a00 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb31daf0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181b8e00 con 0x7fb3180952f0 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb316ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb316ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45062/0 (socket says 172.21.2.117:45062) 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb316ffd700 1 -- 172.21.2.117:0/1454385050 learned_addr learned my addr 172.21.2.117:0/1454385050 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb3177fe700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3180952f0 0x7fb3181b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.148 7fb317fff700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 0x7fb31819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 -- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3180952f0 msgr2=0x7fb3181b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3180952f0 0x7fb3181b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 -- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 msgr2=0x7fb31819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 0x7fb31819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 -- 172.21.2.117:0/1454385050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180c0980 con 0x7fb3181b7760 2021-02-10T11:34:13.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb317fff700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 0x7fb31819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb316ffd700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7fb308006f60 tx=0x7fb30800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb30800bd60 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308000c90 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb308002220 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3180c0980 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.149 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3180c0980 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.150 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3080052e0 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.150 7fb30ffff700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 0x7fb2fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.150 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb3080328d0 con 0x7fb3181b7760 2021-02-10T11:34:13.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.150 7fb3177fe700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 0x7fb2fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.151 7fb3177fe700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 0x7fb2fc01abc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.180 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3181b4f00 con 0x7fb3181b7760 2021-02-10T11:34:13.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.183 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb308000c90 con 0x7fb3181b7760 2021-02-10T11:34:13.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.309 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802504", "target": ["mgr", ""]}) v1 -- 0x7fb318077a70 con 0x7fb2fc0186c0 2021-02-10T11:34:13.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.335 7fb30ffff700 1 -- 172.21.2.117:0/1454385050 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb3000055c0 con 0x7fb2fc0186c0 2021-02-10T11:34:13.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 msgr2=0x7fb2fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 0x7fb2fc01abc0 crc :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 msgr2=0x7fb3181b83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7fb308006f60 tx=0x7fb30800b970).stop 2021-02-10T11:34:13.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 shutdown_connections 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb2fc0186c0 0x7fb2fc01abc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3181b7760 0x7fb3181b83e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3181b8a00 0x7fb31819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 --2- 172.21.2.117:0/1454385050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3180952f0 0x7fb3181b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.342 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 >> 172.21.2.117:0/1454385050 conn(0x7fb318099ba0 msgr2=0x7fb3180b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.343 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 shutdown_connections 2021-02-10T11:34:13.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.343 7fb31daf0700 1 -- 172.21.2.117:0/1454385050 wait complete. 2021-02-10T11:34:13.337 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802504 2021-02-10T11:34:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 Processor -- start 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 -- start start 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa1f4079000 0x7fa1f40e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f40d1d20 con 0x7fa1f407da20 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f4077a70 con 0x7fa1f41b00f0 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.770 7fa1fa2ce700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f40e53c0 con 0x7fa1f4079000 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa1f407da20 0x7fa1f40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45316/0 (socket says 172.21.2.117:45316) 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f3fff700 1 -- 172.21.2.117:0/2217064404 learned_addr learned my addr 172.21.2.117:0/2217064404 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564311824 0 0) 0x7fa1e4000ce0 con 0x7fa1f407da20 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1dc002fe0 con 0x7fa1f407da20 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2150351511 0 0) 0x7fa1e40010e0 con 0x7fa1f407da20 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 >> v2:172.21.2.125:3300/0 conn(0x7fa1f4079000 msgr2=0x7fa1f40e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 --2- 172.21.2.117:0/2217064404 >> v2:172.21.2.125:3300/0 conn(0x7fa1f4079000 0x7fa1f40e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:13.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 >> v1:172.21.2.125:6789/0 conn(0x7fa1f41b00f0 legacy=0x7fa1f40d0520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4079400 con 0x7fa1f407da20 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa1e4001600 con 0x7fa1f407da20 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 >> v1:172.21.2.117:6789/0 conn(0x7fa1f407da20 legacy=0x7fa1f40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 0x7fa1dc005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1dc006790 0x7fa1dc008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1dc009590 0x7fa1dc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc00c300 con 0x7fa1dc006790 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc00c520 con 0x7fa1dc003a60 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc00c740 con 0x7fa1dc009590 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f3fff700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 0x7fa1dc005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.771 7fa1f37fe700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1dc006790 0x7fa1dc008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f88eb700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1dc009590 0x7fa1dc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 -- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1dc009590 msgr2=0x7fa1dc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:13.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1dc009590 0x7fa1dc00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 -- 172.21.2.117:0/2217064404 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1dc006790 msgr2=0x7fa1dc008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1dc006790 0x7fa1dc008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 -- 172.21.2.117:0/2217064404 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc00c9e0 con 0x7fa1dc003a60 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f88eb700 1 -- 172.21.2.117:0/2217064404 reap_dead start 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f3fff700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 0x7fa1dc005ef0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa1e40014f0 tx=0x7fa1e4007d80).ready entity=mon.1 client_cookie=6c9d4c6de3961974 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e4001810 con 0x7fa1dc003a60 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40070c0 con 0x7fa1dc003a60 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1f2ffd700 1 -- 172.21.2.117:0/2217064404 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e4004620 con 0x7fa1dc003a60 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1fa2ce700 1 -- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 msgr2=0x7fa1dc005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.772 7fa1fa2ce700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 0x7fa1dc005ef0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa1e40014f0 tx=0x7fa1e4007d80).stop 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.773 7fa1fa2ce700 1 -- 172.21.2.117:0/2217064404 shutdown_connections 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.773 7fa1fa2ce700 1 --2- 172.21.2.117:0/2217064404 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1dc003a60 0x7fa1dc005ef0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.773 7fa1fa2ce700 1 -- 172.21.2.117:0/2217064404 >> 172.21.2.117:0/2217064404 conn(0x7fa1f40ce450 msgr2=0x7fa1f40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.773 7fa1fa2ce700 1 -- 172.21.2.117:0/2217064404 shutdown_connections 2021-02-10T11:34:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.773 7fa1fa2ce700 1 -- 172.21.2.117:0/2217064404 wait complete. 2021-02-10T11:34:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 Processor -- start 2021-02-10T11:34:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 -- start start 2021-02-10T11:34:13.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1f40a47d0 0x7fa1f40a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1f40a51f0 0x7fa1f40a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4189970 con 0x7fa1f40b77d0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4190aa0 con 0x7fa1f40a51f0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.788 7fa1fa2ce700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f40b81f0 con 0x7fa1f40a47d0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45082/0 (socket says 172.21.2.117:45082) 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 -- 172.21.2.117:0/2912557097 learned_addr learned my addr 172.21.2.117:0/2912557097 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f3fff700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1f40a47d0 0x7fa1f40a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f37fe700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1f40a51f0 0x7fa1f40a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 -- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1f40a47d0 msgr2=0x7fa1f40a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1f40a47d0 0x7fa1f40a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 -- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1f40a51f0 msgr2=0x7fa1f40a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1f40a51f0 0x7fa1f40a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 -- 172.21.2.117:0/2912557097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40b8410 con 0x7fa1f40b77d0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f88eb700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fa1ec0099a0 tx=0x7fa1ec00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1ec0066f0 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec003030 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1ec0013e0 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f40b8410 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.789 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f40b8410 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.790 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa1ec0011f0 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.790 7fa1f0ff9700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 0x7fa1d401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.790 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa1ec032cf0 con 0x7fa1f40b77d0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.790 7fa1f3fff700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 0x7fa1d401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:13.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.791 7fa1f3fff700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 0x7fa1d401abd0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:13.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.820 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f4077a70 con 0x7fa1f40b77d0 2021-02-10T11:34:13.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.823 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa1ec00be30 con 0x7fa1f40b77d0 2021-02-10T11:34:13.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.952 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802504", "target": ["mgr", ""]}) v1 -- 0x7fa1f4077a70 con 0x7fa1d40186d0 2021-02-10T11:34:13.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.956 7fa1f0ff9700 1 -- 172.21.2.117:0/2912557097 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa1e4000c90 con 0x7fa1d40186d0 2021-02-10T11:34:13.946 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752 2021-02-10T11:34:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.961 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 msgr2=0x7fa1d401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.961 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 0x7fa1d401abd0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 msgr2=0x7fa1f40b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7fa1ec0099a0 tx=0x7fa1ec00bcf0).stop 2021-02-10T11:34:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 shutdown_connections 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1d40186d0 0x7fa1d401abd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1f40b77d0 0x7fa1f40b7bd0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1f40a51f0 0x7fa1f40a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 --2- 172.21.2.117:0/2912557097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1f40a47d0 0x7fa1f40a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 >> 172.21.2.117:0/2912557097 conn(0x7fa1f40b5140 msgr2=0x7fa1f40a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 shutdown_connections 2021-02-10T11:34:13.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:13.962 7fa1fa2ce700 1 -- 172.21.2.117:0/2912557097 wait complete. 2021-02-10T11:34:13.957 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802504 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752 2021-02-10T11:34:13.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:13.957 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:13.957 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:14.295 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:14.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:14.296 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:14.677 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:14.677 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:14.677 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231853 s, 45.2 MB/s 2021-02-10T11:34:14.678 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:14.678 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:14.678 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:15.014 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:15.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:15.019 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243292 s, 43.1 MB/s 2021-02-10T11:34:15.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:15.397 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:15.397 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:15.733 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:15.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:15.733 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:16.111 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:16.111 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:16.111 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256578 s, 40.9 MB/s 2021-02-10T11:34:16.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:16.112 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:16.112 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:16.447 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:16.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:16.448 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:16.825 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:16.825 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:16.825 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251223 s, 41.7 MB/s 2021-02-10T11:34:16.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:16.826 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:16.826 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:17.162 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:17.162 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:17.163 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:17.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:17.534 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:17.534 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235824 s, 44.5 MB/s 2021-02-10T11:34:17.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:17.535 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:17.535 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:17.870 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:17.871 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:17.871 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:18.247 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:18.247 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:18.247 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251653 s, 41.7 MB/s 2021-02-10T11:34:18.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:18.248 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:18.248 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:18.584 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:18.584 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:18.585 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:18.961 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:18.961 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:18.961 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024826 s, 42.2 MB/s 2021-02-10T11:34:18.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:18.962 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:18.962 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:19.299 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:19.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:19.300 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236408 s, 44.4 MB/s 2021-02-10T11:34:19.675 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:19.676 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:19.676 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:20.012 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:20.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:20.013 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:20.389 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:20.389 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:20.389 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239877 s, 43.7 MB/s 2021-02-10T11:34:20.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:20.390 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:20.390 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:20.726 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:20.726 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:20.727 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802504/eec07d43-0c56-43ab-980e-c93888a9e752/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253949 s, 41.3 MB/s 2021-02-10T11:34:21.105 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802505 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 Processor -- start 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 -- start start 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffbac07da20 0x7ffbac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e4d80 0x7ffbac1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac077a70 con 0x7ffbac0e5180 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac0777d0 con 0x7ffbac07da20 2021-02-10T11:34:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.533 7ffbb3cf1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac07de20 con 0x7ffbac0e4d80 2021-02-10T11:34:21.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffbac07da20 0x7ffbac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffbac07da20 0x7ffbac0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45086/0 (socket says 172.21.2.117:45086) 2021-02-10T11:34:21.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 -- 172.21.2.117:0/1766082355 learned_addr learned my addr 172.21.2.117:0/1766082355 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:21.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb130c700 1 --2- 172.21.2.117:0/1766082355 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e4d80 0x7ffbac1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 -- 172.21.2.117:0/1766082355 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e4d80 msgr2=0x7ffbac1b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 --2- 172.21.2.117:0/1766082355 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e4d80 0x7ffbac1b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 -- 172.21.2.117:0/1766082355 >> v1:172.21.2.125:6789/0 conn(0x7ffbac0e5180 legacy=0x7ffbac077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 -- 172.21.2.117:0/1766082355 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac1b06f0 con 0x7ffbac07da20 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.534 7ffbb1b0d700 1 --2- 172.21.2.117:0/1766082355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac07da20 0x7ffbac0771b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7ffba8006700 tx=0x7ffba8009b80).ready entity=mon.0 client_cookie=b609bc5b9a364e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb0b0b700 1 -- 172.21.2.117:0/1766082355 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba800c0c0 con 0x7ffbac07da20 2021-02-10T11:34:21.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb0b0b700 1 -- 172.21.2.117:0/1766082355 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba800c440 con 0x7ffbac07da20 2021-02-10T11:34:21.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb0b0b700 1 -- 172.21.2.117:0/1766082355 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba800c920 con 0x7ffbac07da20 2021-02-10T11:34:21.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 -- 172.21.2.117:0/1766082355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac07da20 msgr2=0x7ffbac0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 --2- 172.21.2.117:0/1766082355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac07da20 0x7ffbac0771b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7ffba8006700 tx=0x7ffba8009b80).stop 2021-02-10T11:34:21.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 -- 172.21.2.117:0/1766082355 shutdown_connections 2021-02-10T11:34:21.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 --2- 172.21.2.117:0/1766082355 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e4d80 0x7ffbac1b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 --2- 172.21.2.117:0/1766082355 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac07da20 0x7ffbac0771b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.535 7ffbb3cf1700 1 -- 172.21.2.117:0/1766082355 >> 172.21.2.117:0/1766082355 conn(0x7ffbac0ce430 msgr2=0x7ffbac0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:21.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.536 7ffbb3cf1700 1 -- 172.21.2.117:0/1766082355 shutdown_connections 2021-02-10T11:34:21.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.536 7ffbb3cf1700 1 -- 172.21.2.117:0/1766082355 wait complete. 2021-02-10T11:34:21.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.550 7ffbb3cf1700 1 Processor -- start 2021-02-10T11:34:21.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.550 7ffbb3cf1700 1 -- start start 2021-02-10T11:34:21.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.550 7ffbb3cf1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.550 7ffbb3cf1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0b78d0 0x7ffbac0b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb3cf1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac0b82f0 0x7ffbac104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb3cf1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0ba970 con 0x7ffbac0a1310 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb3cf1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac104b90 con 0x7ffbac0b82f0 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb3cf1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac104d70 con 0x7ffbac0b78d0 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45090/0 (socket says 172.21.2.117:45090) 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 -- 172.21.2.117:0/2827559150 learned_addr learned my addr 172.21.2.117:0/2827559150 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 -- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0b78d0 msgr2=0x7ffbac0b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0b78d0 0x7ffbac0b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 -- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac0b82f0 msgr2=0x7ffbac104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac0b82f0 0x7ffbac104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 -- 172.21.2.117:0/2827559150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac104f90 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb1b0d700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7ffba800b650 tx=0x7ffba800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba800bde0 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8002030 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac104f90 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.551 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba80024d0 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.552 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac104f90 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.552 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffba8000c90 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.552 7ffba27fc700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 0x7ffb9401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.552 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffba8000940 con 0x7ffbac0a1310 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.553 7ffbb130c700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 0x7ffb9401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:21.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.553 7ffbb130c700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 0x7ffb9401ac10 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:21.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.584 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac077a70 con 0x7ffbac0a1310 2021-02-10T11:34:21.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.587 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffba8000940 con 0x7ffbac0a1310 2021-02-10T11:34:21.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.709 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802505", "target": ["mgr", ""]}) v1 -- 0x7ffbac077a70 con 0x7ffb94018730 2021-02-10T11:34:21.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.738 7ffba27fc700 1 -- 172.21.2.117:0/2827559150 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ffb9c0009b0 con 0x7ffb94018730 2021-02-10T11:34:21.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 msgr2=0x7ffb9401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 0x7ffb9401ac10 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 msgr2=0x7ffbac18a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:21.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7ffba800b650 tx=0x7ffba800b9f0).stop 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 shutdown_connections 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac0b82f0 0x7ffbac104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0b78d0 0x7ffbac0b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb94018730 0x7ffb9401ac10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 --2- 172.21.2.117:0/2827559150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac0a1310 0x7ffbac18a090 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 >> 172.21.2.117:0/2827559150 conn(0x7ffbac0bacb0 msgr2=0x7ffbac08e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 shutdown_connections 2021-02-10T11:34:21.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:21.743 7ffbb3cf1700 1 -- 172.21.2.117:0/2827559150 wait complete. 2021-02-10T11:34:21.738 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802505 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.167 7f0b94aa9700 1 Processor -- start 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.167 7f0b94aa9700 1 -- start start 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b94aa9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0b90079000 0x7f0b900e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b94aa9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900d1d20 con 0x7f0b901b00f0 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b94aa9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b90077a70 con 0x7f0b9007da20 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b94aa9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b900e53c0 con 0x7f0b90079000 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b8f05f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0b901b00f0 0x7f0b900d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45352/0 (socket says 172.21.2.117:45352) 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b8f05f700 1 -- 172.21.2.117:0/10358772 learned_addr learned my addr 172.21.2.117:0/10358772 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:22.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291782337 0 0) 0x7f0b88001500 con 0x7f0b901b00f0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c002fe0 con 0x7f0b901b00f0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.168 7f0b8e05d700 1 --2- 172.21.2.117:0/10358772 >> v2:172.21.2.125:3300/0 conn(0x7f0b90079000 0x7f0b900e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3818801633 0 0) 0x7f0b88001500 con 0x7f0b901b00f0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 >> v2:172.21.2.125:3300/0 conn(0x7f0b90079000 msgr2=0x7f0b900e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 --2- 172.21.2.117:0/10358772 >> v2:172.21.2.125:3300/0 conn(0x7f0b90079000 0x7f0b900e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 >> v1:172.21.2.125:6789/0 conn(0x7f0b9007da20 legacy=0x7f0b900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90079400 con 0x7f0b901b00f0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0b88001a00 con 0x7f0b901b00f0 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 >> v1:172.21.2.117:6789/0 conn(0x7f0b901b00f0 legacy=0x7f0b900d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 0x7f0b7c005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0067a0 0x7f0b7c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c009580 0x7f0b7c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c00c2f0 con 0x7f0b7c003af0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c00c510 con 0x7f0b7c0067a0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c00c730 con 0x7f0b7c009580 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 0x7f0b7c005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e05d700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0067a0 0x7f0b7c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8f05f700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c009580 0x7f0b7c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 -- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c009580 msgr2=0x7f0b7c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c009580 0x7f0b7c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 -- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0067a0 msgr2=0x7f0b7c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0067a0 0x7f0b7c008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 -- 172.21.2.117:0/10358772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0509a0 con 0x7f0b7c003af0 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8f05f700 1 -- 172.21.2.117:0/10358772 reap_dead start 2021-02-10T11:34:22.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.169 7f0b8e85e700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 0x7f0b7c005f20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f0b78006ab0 tx=0x7f0b78009b40).ready entity=mon.0 client_cookie=498f10c778f24238 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.170 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7800b2e0 con 0x7f0b7c003af0 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.170 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7800b660 con 0x7f0b7c003af0 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.170 7f0b8d85c700 1 -- 172.21.2.117:0/10358772 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7800bb80 con 0x7f0b7c003af0 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.170 7f0b94aa9700 1 -- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 msgr2=0x7f0b7c005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.170 7f0b94aa9700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 0x7f0b7c005f20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f0b78006ab0 tx=0x7f0b78009b40).stop 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.171 7f0b94aa9700 1 -- 172.21.2.117:0/10358772 shutdown_connections 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.171 7f0b94aa9700 1 --2- 172.21.2.117:0/10358772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c003af0 0x7f0b7c005f20 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.171 7f0b94aa9700 1 -- 172.21.2.117:0/10358772 >> 172.21.2.117:0/10358772 conn(0x7f0b900ce450 msgr2=0x7f0b900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.171 7f0b94aa9700 1 -- 172.21.2.117:0/10358772 shutdown_connections 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.171 7f0b94aa9700 1 -- 172.21.2.117:0/10358772 wait complete. 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.185 7f0b94aa9700 1 Processor -- start 2021-02-10T11:34:22.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 -- start start 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b900a51f0 0x7f0b900a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b900b77d0 0x7f0b900b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90189970 con 0x7f0b900a47d0 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90190aa0 con 0x7f0b900b77d0 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b94aa9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900b81f0 con 0x7f0b900a51f0 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b8e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b8e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45112/0 (socket says 172.21.2.117:45112) 2021-02-10T11:34:22.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.186 7f0b8e85e700 1 -- 172.21.2.117:0/3122778366 learned_addr learned my addr 172.21.2.117:0/3122778366 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 -- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b900a51f0 msgr2=0x7f0b900a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b900a51f0 0x7f0b900a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 -- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b900b77d0 msgr2=0x7f0b900b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b900b77d0 0x7f0b900b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 -- 172.21.2.117:0/3122778366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900b8410 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b8e85e700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f0b780093a0 tx=0x7f0b7800b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7800bdb0 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b78000a10 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b780054a0 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b900b8410 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.187 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b900b89a0 con 0x7f0b900a47d0 2021-02-10T11:34:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.188 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b78006030 con 0x7f0b900a47d0 2021-02-10T11:34:22.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.188 7f0b877fe700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 0x7f0b7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:22.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.188 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0b780326b0 con 0x7f0b900a47d0 2021-02-10T11:34:22.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.188 7f0b8e05d700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 0x7f0b7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:22.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.189 7f0b8e05d700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 0x7f0b7401abc0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.218 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b90077a70 con 0x7f0b900a47d0 2021-02-10T11:34:22.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.220 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b780093e0 con 0x7f0b900a47d0 2021-02-10T11:34:22.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.348 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802505", "target": ["mgr", ""]}) v1 -- 0x7f0b90189970 con 0x7f0b740186c0 2021-02-10T11:34:22.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.353 7f0b877fe700 1 -- 172.21.2.117:0/3122778366 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0b80000960 con 0x7f0b740186c0 2021-02-10T11:34:22.342 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20 2021-02-10T11:34:22.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 msgr2=0x7f0b7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 0x7f0b7401abc0 crc :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 msgr2=0x7f0b900a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f0b780093a0 tx=0x7f0b7800b9c0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 shutdown_connections 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b900b77d0 0x7f0b900b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b900a51f0 0x7f0b900a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b740186c0 0x7f0b7401abc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 --2- 172.21.2.117:0/3122778366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b900a47d0 0x7f0b900a4bd0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 >> 172.21.2.117:0/3122778366 conn(0x7f0b900b5140 msgr2=0x7f0b900a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 shutdown_connections 2021-02-10T11:34:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:22.357 7f0b94aa9700 1 -- 172.21.2.117:0/3122778366 wait complete. 2021-02-10T11:34:22.351 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802505 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20 2021-02-10T11:34:22.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:22.352 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:22.352 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:22.688 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:22.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:22.688 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:23.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:23.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:23.068 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023466 s, 44.7 MB/s 2021-02-10T11:34:23.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:23.069 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:23.069 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:23.405 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:23.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:23.406 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:23.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:23.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:23.780 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235257 s, 44.6 MB/s 2021-02-10T11:34:23.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:23.782 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:23.782 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:24.117 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:24.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:24.118 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:24.491 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:24.492 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:24.492 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230973 s, 45.4 MB/s 2021-02-10T11:34:24.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:24.493 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:24.493 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:24.829 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:24.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:24.829 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:25.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:25.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:25.207 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262277 s, 40.0 MB/s 2021-02-10T11:34:25.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:25.208 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:25.208 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:25.543 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:25.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:25.544 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:25.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:25.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:25.918 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235333 s, 44.6 MB/s 2021-02-10T11:34:25.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:25.919 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:25.919 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:26.255 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:26.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:26.255 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:26.631 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244968 s, 42.8 MB/s 2021-02-10T11:34:26.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:26.632 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:26.632 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:26.967 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:26.967 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:26.967 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:27.343 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247058 s, 42.4 MB/s 2021-02-10T11:34:27.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:27.345 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:27.345 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:27.680 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:27.680 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:27.680 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257684 s, 40.7 MB/s 2021-02-10T11:34:28.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:28.056 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:28.057 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:28.391 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:28.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:28.392 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:28.768 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:28.768 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:28.768 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241752 s, 43.4 MB/s 2021-02-10T11:34:28.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:28.769 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:28.769 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:29.104 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:29.104 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:29.104 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802505/65bd6685-7476-4da8-838e-1debecebce20/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:29.479 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:29.479 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:29.479 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245452 s, 42.7 MB/s 2021-02-10T11:34:29.480 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802506 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 Processor -- start 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 -- start start 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f543c07da20 0x7f543c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f543c0e4da0 0x7f543c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f543c077a70 con 0x7f543c0e51a0 2021-02-10T11:34:29.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c0777d0 con 0x7f543c0e4da0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f5444211700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f543c07de20 con 0x7f543c07da20 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f544182c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f543c0e4da0 0x7f543c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f544182c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f543c0e4da0 0x7f543c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45118/0 (socket says 172.21.2.117:45118) 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f544182c700 1 -- 172.21.2.117:0/1628354763 learned_addr learned my addr 172.21.2.117:0/1628354763 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f544282e700 1 --1- 172.21.2.117:0/1628354763 >> v1:172.21.2.117:6789/0 conn(0x7f543c0e51a0 0x7f543c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45368/0 (socket says 172.21.2.117:45368) 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.909 7f544202d700 1 --2- 172.21.2.117:0/1628354763 >> v2:172.21.2.125:3300/0 conn(0x7f543c07da20 0x7f543c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544182c700 1 -- 172.21.2.117:0/1628354763 >> v2:172.21.2.125:3300/0 conn(0x7f543c07da20 msgr2=0x7f543c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544182c700 1 --2- 172.21.2.117:0/1628354763 >> v2:172.21.2.125:3300/0 conn(0x7f543c07da20 0x7f543c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544182c700 1 -- 172.21.2.117:0/1628354763 >> v1:172.21.2.117:6789/0 conn(0x7f543c0e51a0 legacy=0x7f543c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544182c700 1 -- 172.21.2.117:0/1628354763 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c1b0710 con 0x7f543c0e4da0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544182c700 1 --2- 172.21.2.117:0/1628354763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c0e4da0 0x7f543c1b00f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f5438006700 tx=0x7f5438009b00).ready entity=mon.0 client_cookie=ae2e640e969f178c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544102b700 1 -- 172.21.2.117:0/1628354763 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543800c0c0 con 0x7f543c0e4da0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544102b700 1 -- 172.21.2.117:0/1628354763 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543800c340 con 0x7f543c0e4da0 2021-02-10T11:34:29.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f544102b700 1 -- 172.21.2.117:0/1628354763 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543800c820 con 0x7f543c0e4da0 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f5444211700 1 -- 172.21.2.117:0/1628354763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c0e4da0 msgr2=0x7f543c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.910 7f5444211700 1 --2- 172.21.2.117:0/1628354763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c0e4da0 0x7f543c1b00f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7f5438006700 tx=0x7f5438009b00).stop 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 -- 172.21.2.117:0/1628354763 shutdown_connections 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 --2- 172.21.2.117:0/1628354763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c0e4da0 0x7f543c1b00f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 --2- 172.21.2.117:0/1628354763 >> v2:172.21.2.125:3300/0 conn(0x7f543c07da20 0x7f543c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 -- 172.21.2.117:0/1628354763 >> 172.21.2.117:0/1628354763 conn(0x7f543c0ce450 msgr2=0x7f543c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 -- 172.21.2.117:0/1628354763 shutdown_connections 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.911 7f5444211700 1 -- 172.21.2.117:0/1628354763 wait complete. 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 Processor -- start 2021-02-10T11:34:29.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 -- start start 2021-02-10T11:34:29.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f543c0b9e60 0x7f543c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f543c0a3800 0x7f543c0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0a4220 con 0x7f543c19a4a0 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0a4410 con 0x7f543c0a3800 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f5444211700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543c0ba910 con 0x7f543c0b9e60 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f544202d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f544202d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45124/0 (socket says 172.21.2.117:45124) 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.926 7f544202d700 1 -- 172.21.2.117:0/2314162753 learned_addr learned my addr 172.21.2.117:0/2314162753 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 -- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f543c0b9e60 msgr2=0x7f543c0a31e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f543c0b9e60 0x7f543c0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 -- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f543c0a3800 msgr2=0x7f543c0a3c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f543c0a3800 0x7f543c0a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 -- 172.21.2.117:0/2314162753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0bab30 con 0x7f543c19a4a0 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f544202d700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f5430006be0 tx=0x7f543000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f543000bd80 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5430006820 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5430002030 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f5444211700 1 -- 172.21.2.117:0/2314162753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f543c0bab30 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.927 7f5444211700 1 -- 172.21.2.117:0/2314162753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f543c0bb0c0 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.928 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5430000c90 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.928 7f542effd700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 0x7f542401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.928 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f54300326b0 con 0x7f543c19a4a0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.928 7f544182c700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 0x7f542401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:29.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.928 7f544182c700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 0x7f542401abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:29.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.959 7f5444211700 1 -- 172.21.2.117:0/2314162753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f543c1b4f00 con 0x7f543c19a4a0 2021-02-10T11:34:29.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:29.961 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5430006cb0 con 0x7f543c19a4a0 2021-02-10T11:34:30.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.087 7f5444211700 1 -- 172.21.2.117:0/2314162753 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802506", "target": ["mgr", ""]}) v1 -- 0x7f543c0bb0c0 con 0x7f54240186c0 2021-02-10T11:34:30.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.111 7f542effd700 1 -- 172.21.2.117:0/2314162753 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5438000940 con 0x7f54240186c0 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.120 7f5444211700 1 -- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 msgr2=0x7f542401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.120 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 0x7f542401abc0 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.120 7f5444211700 1 -- 172.21.2.117:0/2314162753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 msgr2=0x7f543c0b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.120 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f5430006be0 tx=0x7f543000b990).stop 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 -- 172.21.2.117:0/2314162753 shutdown_connections 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f543c0a3800 0x7f543c0a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f543c0b9e60 0x7f543c0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54240186c0 0x7f542401abc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 --2- 172.21.2.117:0/2314162753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f543c19a4a0 0x7f543c0b9840 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 -- 172.21.2.117:0/2314162753 >> 172.21.2.117:0/2314162753 conn(0x7f543c107b60 msgr2=0x7f543c0c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 -- 172.21.2.117:0/2314162753 shutdown_connections 2021-02-10T11:34:30.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.121 7f5444211700 1 -- 172.21.2.117:0/2314162753 wait complete. 2021-02-10T11:34:30.115 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802506 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.543 7fac9cc2e700 1 Processor -- start 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.543 7fac9cc2e700 1 -- start start 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9cc2e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fac9807da20 0x7fac980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9cc2e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 0x7fac981b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9cc2e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98077a70 con 0x7fac980e51a0 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9cc2e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980777d0 con 0x7fac9807da20 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9cc2e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9807de20 con 0x7fac980e4da0 2021-02-10T11:34:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9685e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fac9807da20 0x7fac980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9685e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fac9807da20 0x7fac980771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45130/0 (socket says 172.21.2.117:45130) 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.544 7fac9685e700 1 -- 172.21.2.117:0/280020056 learned_addr learned my addr 172.21.2.117:0/280020056 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9605d700 1 --2- 172.21.2.117:0/280020056 >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 0x7fac981b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9685e700 1 -- 172.21.2.117:0/280020056 >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 msgr2=0x7fac981b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9685e700 1 --2- 172.21.2.117:0/280020056 >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 0x7fac981b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9685e700 1 -- 172.21.2.117:0/280020056 >> v1:172.21.2.125:6789/0 conn(0x7fac980e51a0 legacy=0x7fac98077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9685e700 1 -- 172.21.2.117:0/280020056 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac981b0710 con 0x7fac9807da20 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9605d700 1 --2- 172.21.2.117:0/280020056 >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 0x7fac981b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9685e700 1 --2- 172.21.2.117:0/280020056 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac9807da20 0x7fac980771b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fac88006700 tx=0x7fac88009b80).ready entity=mon.0 client_cookie=ce06b0dc7a73590f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9585c700 1 -- 172.21.2.117:0/280020056 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac8800c0c0 con 0x7fac9807da20 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9585c700 1 -- 172.21.2.117:0/280020056 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8800c3c0 con 0x7fac9807da20 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9585c700 1 -- 172.21.2.117:0/280020056 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac8800c8a0 con 0x7fac9807da20 2021-02-10T11:34:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9cc2e700 1 -- 172.21.2.117:0/280020056 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac9807da20 msgr2=0x7fac980771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.545 7fac9cc2e700 1 --2- 172.21.2.117:0/280020056 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac9807da20 0x7fac980771b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fac88006700 tx=0x7fac88009b80).stop 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 -- 172.21.2.117:0/280020056 shutdown_connections 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 --2- 172.21.2.117:0/280020056 >> v2:172.21.2.125:3300/0 conn(0x7fac980e4da0 0x7fac981b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 --2- 172.21.2.117:0/280020056 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac9807da20 0x7fac980771b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 -- 172.21.2.117:0/280020056 >> 172.21.2.117:0/280020056 conn(0x7fac980ce450 msgr2=0x7fac980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 -- 172.21.2.117:0/280020056 shutdown_connections 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.546 7fac9cc2e700 1 -- 172.21.2.117:0/280020056 wait complete. 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 Processor -- start 2021-02-10T11:34:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 -- start start 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 0x7fac981b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fac981b5b50 0x7fac980b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980b8da0 con 0x7fac981b5f50 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980a4140 con 0x7fac981b5b50 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.561 7fac9cc2e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980a4320 con 0x7fac981b5130 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45140/0 (socket says 172.21.2.117:45140) 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 -- 172.21.2.117:0/2850674052 learned_addr learned my addr 172.21.2.117:0/2850674052 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9605d700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fac981b5b50 0x7fac980b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9685e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 0x7fac981b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 -- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 msgr2=0x7fac981b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 0x7fac981b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 -- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fac981b5b50 msgr2=0x7fac980b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fac981b5b50 0x7fac980b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 -- 172.21.2.117:0/2850674052 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac980b62b0 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9685e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 0x7fac981b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac9705f700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fac90009940 tx=0x7fac9000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.562 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac90000f90 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9000c2f0 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac980b62b0 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fac9000c750 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac980b6840 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fac90001220 con 0x7fac981b5f50 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac877fe700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 0x7fac7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:30.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.563 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fac90032a80 con 0x7fac981b5f50 2021-02-10T11:34:30.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.564 7fac9685e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 0x7fac7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:30.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.564 7fac9685e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 0x7fac7c01abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:30.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.594 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac981b4f00 con 0x7fac981b5f50 2021-02-10T11:34:30.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.596 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fac90002ef0 con 0x7fac981b5f50 2021-02-10T11:34:30.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.725 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802506", "target": ["mgr", ""]}) v1 -- 0x7fac980b6840 con 0x7fac7c0186c0 2021-02-10T11:34:30.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.729 7fac877fe700 1 -- 172.21.2.117:0/2850674052 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fac88006060 con 0x7fac7c0186c0 2021-02-10T11:34:30.719 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2 2021-02-10T11:34:30.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.734 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 msgr2=0x7fac7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.734 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 0x7fac7c01abc0 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.734 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 msgr2=0x7fac980b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.734 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fac90009940 tx=0x7fac9000bcf0).stop 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 shutdown_connections 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac7c0186c0 0x7fac7c01abc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fac981b5f50 0x7fac980b5c90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fac981b5b50 0x7fac980b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 --2- 172.21.2.117:0/2850674052 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fac981b5130 0x7fac981b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 >> 172.21.2.117:0/2850674052 conn(0x7fac9808bbc0 msgr2=0x7fac980c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 shutdown_connections 2021-02-10T11:34:30.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:30.735 7fac9cc2e700 1 -- 172.21.2.117:0/2850674052 wait complete. 2021-02-10T11:34:30.729 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802506 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2 2021-02-10T11:34:30.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:30.729 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:30.729 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:31.066 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:31.066 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:31.067 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:31.115 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:31.118 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:34:31.449 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:31.449 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:31.450 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242636 s, 43.2 MB/s 2021-02-10T11:34:31.451 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:31.451 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:31.451 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:31.786 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:31.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:31.787 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:32.163 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:32.163 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:32.163 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244381 s, 42.9 MB/s 2021-02-10T11:34:32.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:32.164 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:32.164 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:32.499 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:32.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:32.500 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025453 s, 41.2 MB/s 2021-02-10T11:34:32.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:32.875 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:32.875 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:33.212 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:33.212 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:33.213 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:33.586 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237898 s, 44.1 MB/s 2021-02-10T11:34:33.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:33.589 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:33.589 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:33.924 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:33.924 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:33.924 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:34.300 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:34.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:34.301 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239254 s, 43.8 MB/s 2021-02-10T11:34:34.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:34.303 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:34.303 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:34.639 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:34.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:34.639 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238918 s, 43.9 MB/s 2021-02-10T11:34:35.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:35.016 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:35.016 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:35.352 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:35.352 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:35.352 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:35.727 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:35.727 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:35.727 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228863 s, 45.8 MB/s 2021-02-10T11:34:35.728 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:35.728 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:35.729 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:36.065 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:36.065 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:36.065 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:36.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:36.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:36.440 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237886 s, 44.1 MB/s 2021-02-10T11:34:36.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:36.441 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:36.441 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:36.777 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:36.777 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:36.777 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:37.153 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256898 s, 40.8 MB/s 2021-02-10T11:34:37.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:37.155 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:37.155 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:37.490 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:37.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:37.491 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802506/5233818f-17d5-4c51-8b88-1121fdcd4bd2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:37.863 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:37.863 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:37.863 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237367 s, 44.2 MB/s 2021-02-10T11:34:37.864 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802507 2021-02-10T11:34:38.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.293 7f47b23da700 1 Processor -- start 2021-02-10T11:34:38.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.293 7f47b23da700 1 -- start start 2021-02-10T11:34:38.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b23da700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 0x7f47ac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b23da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 0x7f47ac079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b23da700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47ac077a70 con 0x7f47ac0e4da0 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b23da700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ac0777d0 con 0x7f47ac07da20 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b23da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac1b08e0 con 0x7f47ac0e51a0 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47b09f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 0x7f47ac079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.294 7f47abfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 0x7f47ac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47ab7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f47ac0e4da0 0x7f47ac1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38320/0 (socket says 172.21.2.117:38320) 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47ab7fe700 1 -- 172.21.2.117:0/1144237918 learned_addr learned my addr 172.21.2.117:0/1144237918 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47b09f7700 1 -- 172.21.2.117:0/1144237918 >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 msgr2=0x7f47ac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47b09f7700 1 --2- 172.21.2.117:0/1144237918 >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 0x7f47ac0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47b09f7700 1 -- 172.21.2.117:0/1144237918 >> v1:172.21.2.125:6789/0 conn(0x7f47ac0e4da0 legacy=0x7f47ac1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47b09f7700 1 -- 172.21.2.117:0/1144237918 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac07de20 con 0x7f47ac0e51a0 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47abfff700 1 --2- 172.21.2.117:0/1144237918 >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 0x7f47ac0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47b09f7700 1 --2- 172.21.2.117:0/1144237918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 0x7f47ac079000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f47a4006e50 tx=0x7f47a4009e70).ready entity=mon.2 client_cookie=7c24687e8400b7a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.295 7f47aaffd700 1 -- 172.21.2.117:0/1144237918 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4006a20 con 0x7f47ac0e51a0 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47aaffd700 1 -- 172.21.2.117:0/1144237918 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a4006790 con 0x7f47ac0e51a0 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47aaffd700 1 -- 172.21.2.117:0/1144237918 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a400b700 con 0x7f47ac0e51a0 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 -- 172.21.2.117:0/1144237918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 msgr2=0x7f47ac079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 --2- 172.21.2.117:0/1144237918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 0x7f47ac079000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f47a4006e50 tx=0x7f47a4009e70).stop 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 -- 172.21.2.117:0/1144237918 shutdown_connections 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 --2- 172.21.2.117:0/1144237918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac0e51a0 0x7f47ac079000 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 --2- 172.21.2.117:0/1144237918 >> v2:172.21.2.125:3300/0 conn(0x7f47ac07da20 0x7f47ac0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 -- 172.21.2.117:0/1144237918 >> 172.21.2.117:0/1144237918 conn(0x7f47ac0ce450 msgr2=0x7f47ac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 -- 172.21.2.117:0/1144237918 shutdown_connections 2021-02-10T11:34:38.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.296 7f47b23da700 1 -- 172.21.2.117:0/1144237918 wait complete. 2021-02-10T11:34:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 Processor -- start 2021-02-10T11:34:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 -- start start 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac08a180 0x7f47ac08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 0x7f47ac1b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac1b9cb0 con 0x7f47ac08afa0 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac0c4050 con 0x7f47ac08aba0 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b23da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ac1b9730 con 0x7f47ac08a180 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.311 7f47b09f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45154/0 (socket says 172.21.2.117:45154) 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 -- 172.21.2.117:0/2452664243 learned_addr learned my addr 172.21.2.117:0/2452664243 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47abfff700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac08a180 0x7f47ac08a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47ab7fe700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 0x7f47ac1b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 -- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac08a180 msgr2=0x7f47ac08a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac08a180 0x7f47ac08a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 -- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 msgr2=0x7f47ac1b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 0x7f47ac1b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 -- 172.21.2.117:0/2452664243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac106e90 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47ab7fe700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 0x7f47ac1b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47b09f7700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f47a40099f0 tx=0x7f47a400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.312 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a40093d0 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a40090c0 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4001420 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47b23da700 1 -- 172.21.2.117:0/2452664243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ac106e90 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47b23da700 1 -- 172.21.2.117:0/2452664243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ac1b9350 con 0x7f47ac08afa0 2021-02-10T11:34:38.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47a4000df0 con 0x7f47ac08afa0 2021-02-10T11:34:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47a8ff9700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.313 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f47a4032a50 con 0x7f47ac08afa0 2021-02-10T11:34:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.315 7f47abfff700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.315 7f47abfff700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.344 7f47b23da700 1 -- 172.21.2.117:0/2452664243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ac1b9c50 con 0x7f47ac08afa0 2021-02-10T11:34:38.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.347 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47a40093d0 con 0x7f47ac08afa0 2021-02-10T11:34:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.471 7f47b23da700 1 -- 172.21.2.117:0/2452664243 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802507", "target": ["mgr", ""]}) v1 -- 0x7f47ac1b9350 con 0x7f478c0186c0 2021-02-10T11:34:38.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.496 7f47a8ff9700 1 -- 172.21.2.117:0/2452664243 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f479c006a30 con 0x7f478c0186c0 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 msgr2=0x7f478c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 crc :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 msgr2=0x7f47ac1b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f47a40099f0 tx=0x7f47a400bd10).stop 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 shutdown_connections 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47ac08afa0 0x7f47ac1b8a40 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47ac08aba0 0x7f47ac1b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 --2- 172.21.2.117:0/2452664243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47ac08a180 0x7f47ac08a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 >> 172.21.2.117:0/2452664243 conn(0x7f47ac0a4180 msgr2=0x7f47ac09be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 shutdown_connections 2021-02-10T11:34:38.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.505 7f47b23da700 1 -- 172.21.2.117:0/2452664243 wait complete. 2021-02-10T11:34:38.499 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802507 2021-02-10T11:34:38.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.929 7fbe06414700 1 Processor -- start 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.929 7fbe06414700 1 -- start start 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe06414700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe0007da20 0x7fbe000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe06414700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe000e4da0 0x7fbe001b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe06414700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00077a70 con 0x7fbe000e51a0 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe06414700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe000e5c20 con 0x7fbe000e4da0 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe06414700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000777d0 con 0x7fbe0007da20 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe000e4da0 0x7fbe001b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe04a31700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbe000e51a0 0x7fbe00077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45408/0 (socket says 172.21.2.117:45408) 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe000e4da0 0x7fbe001b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45158/0 (socket says 172.21.2.117:45158) 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbe04a31700 1 -- 172.21.2.117:0/1876307881 learned_addr learned my addr 172.21.2.117:0/1876307881 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdfffff700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe0007da20 0x7fbe000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 -- 172.21.2.117:0/1876307881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe0007da20 msgr2=0x7fbe000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe0007da20 0x7fbe000771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 -- 172.21.2.117:0/1876307881 >> v1:172.21.2.117:6789/0 conn(0x7fbe000e51a0 legacy=0x7fbe00077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 -- 172.21.2.117:0/1876307881 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0007de20 con 0x7fbe000e4da0 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.930 7fbdff7fe700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe000e4da0 0x7fbe001b00f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7fbdf4006650 tx=0x7fbdf4009a80).ready entity=mon.0 client_cookie=befc9c82791f9b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbdfeffd700 1 -- 172.21.2.117:0/1876307881 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf400c0c0 con 0x7fbe000e4da0 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbdfeffd700 1 -- 172.21.2.117:0/1876307881 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf400c340 con 0x7fbe000e4da0 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbdfeffd700 1 -- 172.21.2.117:0/1876307881 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf400c860 con 0x7fbe000e4da0 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 -- 172.21.2.117:0/1876307881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe000e4da0 msgr2=0x7fbe001b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe000e4da0 0x7fbe001b00f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7fbdf4006650 tx=0x7fbdf4009a80).stop 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 -- 172.21.2.117:0/1876307881 shutdown_connections 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe000e4da0 0x7fbe001b00f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 --2- 172.21.2.117:0/1876307881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe0007da20 0x7fbe000771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 -- 172.21.2.117:0/1876307881 >> 172.21.2.117:0/1876307881 conn(0x7fbe000ce450 msgr2=0x7fbe000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 -- 172.21.2.117:0/1876307881 shutdown_connections 2021-02-10T11:34:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.931 7fbe06414700 1 -- 172.21.2.117:0/1876307881 wait complete. 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.946 7fbe06414700 1 Processor -- start 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.946 7fbe06414700 1 -- start start 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe001b5130 0x7fbe001b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe001b5b50 0x7fbe000b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000b8da0 con 0x7fbe001b5f50 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000a4140 con 0x7fbe001b5b50 2021-02-10T11:34:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe06414700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe000a4320 con 0x7fbe001b5130 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45162/0 (socket says 172.21.2.117:45162) 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 -- 172.21.2.117:0/819502434 learned_addr learned my addr 172.21.2.117:0/819502434 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 -- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe001b5130 msgr2=0x7fbe001b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe001b5130 0x7fbe001b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 -- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe001b5b50 msgr2=0x7fbe000b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe001b5b50 0x7fbe000b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.947 7fbe04a31700 1 -- 172.21.2.117:0/819502434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe000b62b0 con 0x7fbe001b5f50 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbe04a31700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fbdf800a940 tx=0x7fbdf800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf80070d0 con 0x7fbe001b5f50 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf8000f90 con 0x7fbe001b5f50 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdf8004720 con 0x7fbe001b5f50 2021-02-10T11:34:38.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbe06414700 1 -- 172.21.2.117:0/819502434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe000b62b0 con 0x7fbe001b5f50 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.948 7fbe06414700 1 -- 172.21.2.117:0/819502434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe000b62b0 con 0x7fbe001b5f50 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.949 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbdf8004720 con 0x7fbe001b5f50 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.949 7fbdfcff9700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 0x7fbde001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.949 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbdf8032900 con 0x7fbe001b5f50 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.949 7fbdfffff700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 0x7fbde001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.949 7fbdfffff700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 0x7fbde001abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:38.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.979 7fbe06414700 1 -- 172.21.2.117:0/819502434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe001b4f00 con 0x7fbe001b5f50 2021-02-10T11:34:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:38.982 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbdf8007cb0 con 0x7fbe001b5f50 2021-02-10T11:34:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.110 7fbe06414700 1 -- 172.21.2.117:0/819502434 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802507", "target": ["mgr", ""]}) v1 -- 0x7fbe000b6b30 con 0x7fbde00186c0 2021-02-10T11:34:39.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.115 7fbdfcff9700 1 -- 172.21.2.117:0/819502434 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbdf0006a20 con 0x7fbde00186c0 2021-02-10T11:34:39.104 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 msgr2=0x7fbde001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 0x7fbde001abc0 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 msgr2=0x7fbe000b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7fbdf800a940 tx=0x7fbdf800ace0).stop 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 shutdown_connections 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbde00186c0 0x7fbde001abc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe001b5f50 0x7fbe000b5c90 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:39.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe001b5b50 0x7fbe000b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 --2- 172.21.2.117:0/819502434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe001b5130 0x7fbe001b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 >> 172.21.2.117:0/819502434 conn(0x7fbe0008bbc0 msgr2=0x7fbe000c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 shutdown_connections 2021-02-10T11:34:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:39.120 7fbe06414700 1 -- 172.21.2.117:0/819502434 wait complete. 2021-02-10T11:34:39.113 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802507 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a 2021-02-10T11:34:39.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:39.114 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:39.114 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:39.449 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:39.449 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:39.450 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:39.829 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:39.829 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:39.829 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235015 s, 44.6 MB/s 2021-02-10T11:34:39.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:39.830 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:39.830 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:40.164 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:40.165 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:40.165 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:40.541 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:40.542 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:40.542 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231306 s, 45.3 MB/s 2021-02-10T11:34:40.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:40.543 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:40.543 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:40.878 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:40.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:40.879 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:41.252 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:41.252 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:41.252 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225287 s, 46.5 MB/s 2021-02-10T11:34:41.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:41.253 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:41.253 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:41.588 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:41.589 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:41.589 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:41.966 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:41.966 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:41.966 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253592 s, 41.3 MB/s 2021-02-10T11:34:41.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:41.967 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:41.967 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:42.304 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:42.304 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:42.304 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:42.675 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:42.675 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:42.675 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233506 s, 44.9 MB/s 2021-02-10T11:34:42.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:42.677 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:42.677 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:43.013 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:43.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:43.013 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:43.391 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:43.391 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:43.392 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0261331 s, 40.1 MB/s 2021-02-10T11:34:43.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:43.393 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:43.393 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:43.728 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:43.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:43.729 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:44.105 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:44.106 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:44.106 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244561 s, 42.9 MB/s 2021-02-10T11:34:44.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:44.107 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:44.107 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:44.451 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:44.451 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:44.451 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257086 s, 40.8 MB/s 2021-02-10T11:34:44.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:44.832 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:44.832 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:45.168 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:45.169 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:45.169 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:45.546 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:45.546 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:45.546 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250084 s, 41.9 MB/s 2021-02-10T11:34:45.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:45.547 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:45.547 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:45.882 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:45.883 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:45.883 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802507/2a9f76af-4dbc-4de8-b182-5e98c93a7b9a/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:46.259 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:46.260 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:46.260 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243877 s, 43.0 MB/s 2021-02-10T11:34:46.261 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802508 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 Processor -- start 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 -- start start 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2e7c079000 0x7f2e7c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7c077a70 con 0x7f2e7c07da20 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0e5c20 con 0x7f2e7c1b00f0 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e82e6e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7c0e53c0 con 0x7f2e7c079000 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e8148b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e8148b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45172/0 (socket says 172.21.2.117:45172) 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.690 7f2e8148b700 1 -- 172.21.2.117:0/3113776338 learned_addr learned my addr 172.21.2.117:0/3113776338 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e8148b700 1 -- 172.21.2.117:0/3113776338 >> v2:172.21.2.125:3300/0 conn(0x7f2e7c079000 msgr2=0x7f2e7c0e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e8148b700 1 --2- 172.21.2.117:0/3113776338 >> v2:172.21.2.125:3300/0 conn(0x7f2e7c079000 0x7f2e7c0e4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e8148b700 1 -- 172.21.2.117:0/3113776338 >> v1:172.21.2.125:6789/0 conn(0x7f2e7c07da20 legacy=0x7f2e7c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e8148b700 1 -- 172.21.2.117:0/3113776338 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c079400 con 0x7f2e7c1b00f0 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e8148b700 1 --2- 172.21.2.117:0/3113776338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f2e74006f00 tx=0x7f2e74009f10).ready entity=mon.0 client_cookie=e542ae9c64a40a64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e7b7fe700 1 -- 172.21.2.117:0/3113776338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e74003030 con 0x7f2e7c1b00f0 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e7b7fe700 1 -- 172.21.2.117:0/3113776338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7400b3b0 con 0x7f2e7c1b00f0 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e7b7fe700 1 -- 172.21.2.117:0/3113776338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e7400b8d0 con 0x7f2e7c1b00f0 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e82e6e700 1 -- 172.21.2.117:0/3113776338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1b00f0 msgr2=0x7f2e7c1b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.691 7f2e82e6e700 1 --2- 172.21.2.117:0/3113776338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f2e74006f00 tx=0x7f2e74009f10).stop 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 -- 172.21.2.117:0/3113776338 shutdown_connections 2021-02-10T11:34:46.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 --2- 172.21.2.117:0/3113776338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1b00f0 0x7f2e7c1b04f0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 --2- 172.21.2.117:0/3113776338 >> v2:172.21.2.125:3300/0 conn(0x7f2e7c079000 0x7f2e7c0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 -- 172.21.2.117:0/3113776338 >> 172.21.2.117:0/3113776338 conn(0x7f2e7c0ce450 msgr2=0x7f2e7c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 -- 172.21.2.117:0/3113776338 shutdown_connections 2021-02-10T11:34:46.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.692 7f2e82e6e700 1 -- 172.21.2.117:0/3113776338 wait complete. 2021-02-10T11:34:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.706 7f2e82e6e700 1 Processor -- start 2021-02-10T11:34:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.706 7f2e82e6e700 1 -- start start 2021-02-10T11:34:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e7c093c80 0x7f2e7c1a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e7c094080 0x7f2e7c095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c095740 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c095960 con 0x7f2e7c094080 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e82e6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c095b80 con 0x7f2e7c093c80 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45174/0 (socket says 172.21.2.117:45174) 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 -- 172.21.2.117:0/3830425475 learned_addr learned my addr 172.21.2.117:0/3830425475 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 -- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e7c093c80 msgr2=0x7f2e7c1a75d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e7c093c80 0x7f2e7c1a75d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 -- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e7c094080 msgr2=0x7f2e7c095120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e7c094080 0x7f2e7c095120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 -- 172.21.2.117:0/3830425475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c095da0 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.707 7f2e7bfff700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f2e6c00b5c0 tx=0x7f2e6c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e6c00bd50 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c006030 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e6c0054d0 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c095da0 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c095da0 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.708 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e6c006210 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.709 7f2e797fa700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 0x7f2e6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.709 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2e6c032570 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.709 7f2e80c8a700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 0x7f2e6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.709 7f2e80c8a700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 0x7f2e6001abc0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.740 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7c103310 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.743 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e6c006b80 con 0x7f2e7c1a7bf0 2021-02-10T11:34:46.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.868 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802508", "target": ["mgr", ""]}) v1 -- 0x7f2e7c103310 con 0x7f2e600186c0 2021-02-10T11:34:46.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.895 7f2e797fa700 1 -- 172.21.2.117:0/3830425475 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2e640009b0 con 0x7f2e600186c0 2021-02-10T11:34:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.901 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 msgr2=0x7f2e6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.901 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 0x7f2e6001abc0 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 msgr2=0x7f2e7c094b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f2e6c00b5c0 tx=0x7f2e6c00b960).stop 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 shutdown_connections 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e600186c0 0x7f2e6001abc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e7c1a7bf0 0x7f2e7c094b00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e7c094080 0x7f2e7c095120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 --2- 172.21.2.117:0/3830425475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e7c093c80 0x7f2e7c1a75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 >> 172.21.2.117:0/3830425475 conn(0x7f2e7c0a1880 msgr2=0x7f2e7c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:46.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 shutdown_connections 2021-02-10T11:34:46.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:46.902 7f2e82e6e700 1 -- 172.21.2.117:0/3830425475 wait complete. 2021-02-10T11:34:46.896 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802508 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 Processor -- start 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 -- start start 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faad807da20 0x7faad80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faad80e51a0 0x7faad8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad8077a70 con 0x7faad80e4da0 2021-02-10T11:34:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad80e5c20 con 0x7faad80e51a0 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.327 7faadc82b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad80777d0 con 0x7faad807da20 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faad80e51a0 0x7faad8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faad80e51a0 0x7faad8079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45182/0 (socket says 172.21.2.117:45182) 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 -- 172.21.2.117:0/3751365734 learned_addr learned my addr 172.21.2.117:0/3751365734 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 -- 172.21.2.117:0/3751365734 >> v2:172.21.2.125:3300/0 conn(0x7faad807da20 msgr2=0x7faad80771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 --2- 172.21.2.117:0/3751365734 >> v2:172.21.2.125:3300/0 conn(0x7faad807da20 0x7faad80771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 -- 172.21.2.117:0/3751365734 >> v1:172.21.2.117:6789/0 conn(0x7faad80e4da0 legacy=0x7faad81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 -- 172.21.2.117:0/3751365734 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad81b08e0 con 0x7faad80e51a0 2021-02-10T11:34:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad6e1d700 1 --2- 172.21.2.117:0/3751365734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad80e51a0 0x7faad8079000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7faad0006f00 tx=0x7faad0009f10).ready entity=mon.0 client_cookie=329bdd2e633d7227 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad561a700 1 -- 172.21.2.117:0/3751365734 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad000b3b0 con 0x7faad80e51a0 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad561a700 1 -- 172.21.2.117:0/3751365734 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad000b630 con 0x7faad80e51a0 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faad561a700 1 -- 172.21.2.117:0/3751365734 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faad000bb50 con 0x7faad80e51a0 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faadc82b700 1 -- 172.21.2.117:0/3751365734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad80e51a0 msgr2=0x7faad8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.328 7faadc82b700 1 --2- 172.21.2.117:0/3751365734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad80e51a0 0x7faad8079000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7faad0006f00 tx=0x7faad0009f10).stop 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 -- 172.21.2.117:0/3751365734 shutdown_connections 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 --2- 172.21.2.117:0/3751365734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad80e51a0 0x7faad8079000 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 --2- 172.21.2.117:0/3751365734 >> v2:172.21.2.125:3300/0 conn(0x7faad807da20 0x7faad80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 -- 172.21.2.117:0/3751365734 >> 172.21.2.117:0/3751365734 conn(0x7faad80ce450 msgr2=0x7faad80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 -- 172.21.2.117:0/3751365734 shutdown_connections 2021-02-10T11:34:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.329 7faadc82b700 1 -- 172.21.2.117:0/3751365734 wait complete. 2021-02-10T11:34:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 Processor -- start 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 -- start start 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 0x7faad80a7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 0x7faad808d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad808cf40 con 0x7faad808a870 2021-02-10T11:34:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad808dae0 con 0x7faad80a8170 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faadc82b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad808dcc0 con 0x7faad808a470 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faad5e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faad5e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45190/0 (socket says 172.21.2.117:45190) 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faad5e1b700 1 -- 172.21.2.117:0/189098888 learned_addr learned my addr 172.21.2.117:0/189098888 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.344 7faad661c700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 0x7faad80a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad6e1d700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 0x7faad808d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 -- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 msgr2=0x7faad80a7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 0x7faad80a7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 -- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 msgr2=0x7faad808d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 0x7faad808d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 -- 172.21.2.117:0/189098888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad808dee0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad661c700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 0x7faad80a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad6e1d700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 0x7faad808d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faad5e1b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7faac0006ed0 tx=0x7faac000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faac0006020 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac00009b0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faac00054b0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faadc82b700 1 -- 172.21.2.117:0/189098888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad808dee0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.345 7faadc82b700 1 -- 172.21.2.117:0/189098888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad808dee0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.346 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faac00062e0 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.346 7faaceffd700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 0x7faabc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.346 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faac0032730 con 0x7faad808a870 2021-02-10T11:34:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.346 7faad661c700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 0x7faabc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.347 7faad661c700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 0x7faabc01abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:47.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.377 7faadc82b700 1 -- 172.21.2.117:0/189098888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad81032d0 con 0x7faad808a870 2021-02-10T11:34:47.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.380 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faac002df40 con 0x7faad808a870 2021-02-10T11:34:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.510 7faadc82b700 1 -- 172.21.2.117:0/189098888 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802508", "target": ["mgr", ""]}) v1 -- 0x7faad8077a70 con 0x7faabc0186c0 2021-02-10T11:34:47.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.514 7faaceffd700 1 -- 172.21.2.117:0/189098888 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7faac8006580 con 0x7faabc0186c0 2021-02-10T11:34:47.499 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba 2021-02-10T11:34:47.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 -- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 msgr2=0x7faabc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 0x7faabc01abc0 crc :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 -- 172.21.2.117:0/189098888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 msgr2=0x7faad80a7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:47.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7faac0006ed0 tx=0x7faac000b960).stop 2021-02-10T11:34:47.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 -- 172.21.2.117:0/189098888 shutdown_connections 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faabc0186c0 0x7faabc01abc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faad808a870 0x7faad80a7b50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faad80a8170 0x7faad808d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 --2- 172.21.2.117:0/189098888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faad808a470 0x7faad80a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.515 7faadc82b700 1 -- 172.21.2.117:0/189098888 >> 172.21.2.117:0/189098888 conn(0x7faad80a9210 msgr2=0x7faad80a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.516 7faadc82b700 1 -- 172.21.2.117:0/189098888 shutdown_connections 2021-02-10T11:34:47.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:47.516 7faadc82b700 1 -- 172.21.2.117:0/189098888 wait complete. 2021-02-10T11:34:47.509 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802508 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba 2021-02-10T11:34:47.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:47.509 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:47.510 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:47.847 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:47.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:47.847 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:48.227 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:48.227 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:48.227 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024151 s, 43.4 MB/s 2021-02-10T11:34:48.228 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:48.229 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:48.229 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:48.564 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:48.564 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:48.565 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:48.938 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:48.938 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:48.938 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223705 s, 46.9 MB/s 2021-02-10T11:34:48.939 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:48.939 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:48.939 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:49.273 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:49.274 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:49.274 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:49.651 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:49.651 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:49.651 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245888 s, 42.6 MB/s 2021-02-10T11:34:49.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:49.652 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:49.652 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:49.988 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:49.988 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:49.988 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:50.362 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229412 s, 45.7 MB/s 2021-02-10T11:34:50.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:50.364 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:50.364 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:50.698 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:50.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:50.699 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:51.075 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:51.075 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:51.075 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236704 s, 44.3 MB/s 2021-02-10T11:34:51.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:51.077 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:51.077 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:51.411 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:51.412 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:51.412 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:51.788 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:51.788 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:51.788 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233867 s, 44.8 MB/s 2021-02-10T11:34:51.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:51.789 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:51.790 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:52.127 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:52.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:52.127 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:52.499 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:52.500 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:52.500 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244593 s, 42.9 MB/s 2021-02-10T11:34:52.500 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:52.501 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:52.501 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:52.836 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:52.837 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:52.837 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:53.212 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:53.213 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:53.213 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025557 s, 41.0 MB/s 2021-02-10T11:34:53.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:53.214 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:53.214 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:53.551 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:53.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:53.552 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:53.930 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:53.930 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:53.930 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252708 s, 41.5 MB/s 2021-02-10T11:34:53.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:53.931 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:53.931 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:54.267 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:54.267 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:54.267 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802508/b40f9da4-dbff-4db6-9d08-5b8ef8c784ba/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249759 s, 42.0 MB/s 2021-02-10T11:34:54.646 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802509 2021-02-10T11:34:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.077 7fecfbc7c700 1 Processor -- start 2021-02-10T11:34:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.077 7fecfbc7c700 1 -- start start 2021-02-10T11:34:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfbc7c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fecf40d1530 0x7fecf40d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfbc7c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf4077a70 con 0x7fecf407da20 2021-02-10T11:34:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfbc7c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf4079400 con 0x7fecf4079000 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfbc7c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf407de20 con 0x7fecf40d1530 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfa299700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fecf40d1530 0x7fecf40d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf9a98700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fecf407da20 0x7fecf40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45444/0 (socket says 172.21.2.117:45444) 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf9a98700 1 -- 172.21.2.117:0/2202725651 learned_addr learned my addr 172.21.2.117:0/2202725651 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfa299700 1 -- 172.21.2.117:0/2202725651 >> v1:172.21.2.125:6789/0 conn(0x7fecf4079000 legacy=0x7fecf40e4da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfa299700 1 -- 172.21.2.117:0/2202725651 >> v1:172.21.2.117:6789/0 conn(0x7fecf407da20 legacy=0x7fecf40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfa299700 1 -- 172.21.2.117:0/2202725651 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf41b00f0 con 0x7fecf40d1530 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf8a96700 1 -- 172.21.2.117:0/2202725651 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737211455 0 0) 0x7fecf0001180 con 0x7fecf407da20 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecfa299700 1 --2- 172.21.2.117:0/2202725651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40d1530 0x7fecf40d1930 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fecec006e30 tx=0x7fecec009de0).ready entity=mon.0 client_cookie=93fdd185a997c13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf8a96700 1 -- 172.21.2.117:0/2202725651 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec00b300 con 0x7fecf40d1530 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf8a96700 1 -- 172.21.2.117:0/2202725651 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec00b580 con 0x7fecf40d1530 2021-02-10T11:34:55.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.078 7fecf8a96700 1 -- 172.21.2.117:0/2202725651 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec00ba60 con 0x7fecf40d1530 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 -- 172.21.2.117:0/2202725651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40d1530 msgr2=0x7fecf40d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 --2- 172.21.2.117:0/2202725651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40d1530 0x7fecf40d1930 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fecec006e30 tx=0x7fecec009de0).stop 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 -- 172.21.2.117:0/2202725651 shutdown_connections 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 --2- 172.21.2.117:0/2202725651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40d1530 0x7fecf40d1930 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 -- 172.21.2.117:0/2202725651 >> 172.21.2.117:0/2202725651 conn(0x7fecf40ce450 msgr2=0x7fecf40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 -- 172.21.2.117:0/2202725651 shutdown_connections 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.079 7fecfbc7c700 1 -- 172.21.2.117:0/2202725651 wait complete. 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.093 7fecfbc7c700 1 Processor -- start 2021-02-10T11:34:55.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 -- start start 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fecf409c780 0x7fecf40b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fecf409cb80 0x7fecf40c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40bff40 con 0x7fecf40b4250 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40bfb60 con 0x7fecf409cb80 2021-02-10T11:34:55.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfbc7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf40bfd40 con 0x7fecf409c780 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfa299700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfa299700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45204/0 (socket says 172.21.2.117:45204) 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecfa299700 1 -- 172.21.2.117:0/4218309329 learned_addr learned my addr 172.21.2.117:0/4218309329 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecf9a98700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fecf409c780 0x7fecf40b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.094 7fecf9297700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fecf409cb80 0x7fecf40c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 -- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fecf409c780 msgr2=0x7fecf40b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fecf409c780 0x7fecf40b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 -- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fecf409cb80 msgr2=0x7fecf40c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fecf409cb80 0x7fecf40c04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 -- 172.21.2.117:0/4218309329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40c1100 con 0x7fecf40b4250 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfa299700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7fecec00b970 tx=0x7fecec00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec009380 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec005820 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecec005c80 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40c1100 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.095 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf40c1100 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.096 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fecec005c10 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.096 7fecea7fc700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 0x7fecdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.096 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fecec032a50 con 0x7fecf40b4250 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.096 7fecf9a98700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 0x7fecdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.096 7fecf9a98700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 0x7fecdc01abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.128 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf40ff170 con 0x7fecf40b4250 2021-02-10T11:34:55.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.131 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fecec005a00 con 0x7fecf40b4250 2021-02-10T11:34:55.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.258 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802509", "target": ["mgr", ""]}) v1 -- 0x7fecf40c1a00 con 0x7fecdc0186c0 2021-02-10T11:34:55.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.284 7fecea7fc700 1 -- 172.21.2.117:0/4218309329 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fecf0001460 con 0x7fecdc0186c0 2021-02-10T11:34:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 msgr2=0x7fecdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 0x7fecdc01abc0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 msgr2=0x7fecf40c0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7fecec00b970 tx=0x7fecec00bd10).stop 2021-02-10T11:34:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 shutdown_connections 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecdc0186c0 0x7fecdc01abc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fecf40b4250 0x7fecf40c0ae0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fecf409cb80 0x7fecf40c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 --2- 172.21.2.117:0/4218309329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fecf409c780 0x7fecf40b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.291 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 >> 172.21.2.117:0/4218309329 conn(0x7fecf409d3b0 msgr2=0x7fecf40b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.292 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 shutdown_connections 2021-02-10T11:34:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.292 7fecfbc7c700 1 -- 172.21.2.117:0/4218309329 wait complete. 2021-02-10T11:34:55.285 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802509 2021-02-10T11:34:55.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 Processor -- start 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 -- start start 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280d1d20 con 0x7f26280e4da0 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2628077a70 con 0x7f26280e51a0 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.715 7f262e593700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280777d0 con 0x7f262807da20 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f26277fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f26280e4da0 0x7f26281b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45458/0 (socket says 172.21.2.117:45458) 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f26277fe700 1 -- 172.21.2.117:0/63985910 learned_addr learned my addr 172.21.2.117:0/63985910 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286664996 0 0) 0x7f261c0010b0 con 0x7f26280e4da0 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610002fe0 con 0x7f26280e4da0 2021-02-10T11:34:55.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2627fff700 1 --2- 172.21.2.117:0/63985910 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 810119963 0 0) 0x7f261c001010 con 0x7f26280e4da0 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 msgr2=0x7f26280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 --2- 172.21.2.117:0/63985910 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 >> v1:172.21.2.125:6789/0 conn(0x7f26280e51a0 legacy=0x7f26280d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26281b08e0 con 0x7f26280e4da0 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f261c001530 con 0x7f26280e4da0 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.716 7f2626ffd700 1 -- 172.21.2.117:0/63985910 >> v1:172.21.2.117:6789/0 conn(0x7f26280e4da0 legacy=0x7f26281b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2610003b20 0x7f2610005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2610006830 0x7f2610008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 0x7f261000bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 -- 172.21.2.117:0/63985910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261000c3a0 con 0x7f2610003b20 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 -- 172.21.2.117:0/63985910 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261000c5c0 con 0x7f2610009630 2021-02-10T11:34:55.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2626ffd700 1 -- 172.21.2.117:0/63985910 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261000c7e0 con 0x7f2610006830 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2627fff700 1 --2- 172.21.2.117:0/63985910 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2627fff700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2610003b20 0x7f2610005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f26277fe700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2610006830 0x7f2610008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 0x7f261000bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 -- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2610006830 msgr2=0x7f2610008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2610006830 0x7f2610008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 -- 172.21.2.117:0/63985910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2610003b20 msgr2=0x7f2610005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2610003b20 0x7f2610005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 -- 172.21.2.117:0/63985910 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261000ca80 con 0x7f2610009630 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f26277fe700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2610006830 0x7f2610008d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f2627fff700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2610003b20 0x7f2610005f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.717 7f262cbb0700 1 -- 172.21.2.117:0/63985910 reap_dead start 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262cbb0700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 0x7f261000bb50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f26200069e0 tx=0x7f2620009e10).ready entity=mon.1 client_cookie=23d32705e801c022 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262000b790 con 0x7f2610009630 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262000bad0 con 0x7f2610009630 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f2626ffd700 1 -- 172.21.2.117:0/63985910 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262000b510 con 0x7f2610009630 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 -- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 msgr2=0x7f261000bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 0x7f261000bb50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f26200069e0 tx=0x7f2620009e10).stop 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 -- 172.21.2.117:0/63985910 shutdown_connections 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 --2- 172.21.2.117:0/63985910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2610009630 0x7f261000bb50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 -- 172.21.2.117:0/63985910 >> 172.21.2.117:0/63985910 conn(0x7f26280ce450 msgr2=0x7f26280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 -- 172.21.2.117:0/63985910 shutdown_connections 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.718 7f262e593700 1 -- 172.21.2.117:0/63985910 wait complete. 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 Processor -- start 2021-02-10T11:34:55.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 -- start start 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26280a7820 0x7f2628092030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26280a7c20 0x7f2628093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628092a50 con 0x7f2628092650 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628092c80 con 0x7f26280a7820 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.733 7f262e593700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26280a8340 con 0x7f26280a7c20 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45222/0 (socket says 172.21.2.117:45222) 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 -- 172.21.2.117:0/828686192 learned_addr learned my addr 172.21.2.117:0/828686192 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f2627fff700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26280a7820 0x7f2628092030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f26277fe700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26280a7c20 0x7f2628093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 -- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26280a7c20 msgr2=0x7f2628093200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26280a7c20 0x7f2628093200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 -- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26280a7820 msgr2=0x7f2628092030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26280a7820 0x7f2628092030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 -- 172.21.2.117:0/828686192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26280a8520 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262cbb0700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f26200031a0 tx=0x7f262000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26200093d0 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26200031e0 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.734 7f262e593700 1 -- 172.21.2.117:0/828686192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26280a8520 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f262e593700 1 -- 172.21.2.117:0/828686192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2628094130 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f262000c5f0 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f262000a000 con 0x7f2628092650 2021-02-10T11:34:55.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f2624ff9700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 0x7f260801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:34:55.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2620032c90 con 0x7f2628092650 2021-02-10T11:34:55.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.735 7f2627fff700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 0x7f260801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:34:55.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.736 7f2627fff700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 0x7f260801ac10 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:34:55.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.765 7f262e593700 1 -- 172.21.2.117:0/828686192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628092a50 con 0x7f2628092650 2021-02-10T11:34:55.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.769 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26200093d0 con 0x7f2628092650 2021-02-10T11:34:55.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.893 7f262e593700 1 -- 172.21.2.117:0/828686192 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802509", "target": ["mgr", ""]}) v1 -- 0x7f26280a82f0 con 0x7f2608018730 2021-02-10T11:34:55.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.898 7f2624ff9700 1 -- 172.21.2.117:0/828686192 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f26180009f0 con 0x7f2608018730 2021-02-10T11:34:55.886 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 msgr2=0x7f260801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 0x7f260801ac10 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 msgr2=0x7f2628093820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f26200031a0 tx=0x7f262000bd10).stop 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 shutdown_connections 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2608018730 0x7f260801ac10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2628092650 0x7f2628093820 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26280a7c20 0x7f2628093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 --2- 172.21.2.117:0/828686192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26280a7820 0x7f2628092030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:34:55.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 >> 172.21.2.117:0/828686192 conn(0x7f26280a9210 msgr2=0x7f26280a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:34:55.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 shutdown_connections 2021-02-10T11:34:55.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:34:55.903 7f262e593700 1 -- 172.21.2.117:0/828686192 wait complete. 2021-02-10T11:34:55.896 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802509 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e 2021-02-10T11:34:55.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:55.896 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:55.897 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:56.233 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:56.234 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:56.234 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:56.615 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:56.615 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:56.616 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249843 s, 42.0 MB/s 2021-02-10T11:34:56.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:56.617 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:56.617 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:56.951 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:56.952 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:56.952 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:57.330 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:57.330 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:57.330 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253744 s, 41.3 MB/s 2021-02-10T11:34:57.331 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:57.332 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:57.332 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:57.667 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:57.668 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:57.668 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:58.042 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:58.042 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:58.042 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241886 s, 43.4 MB/s 2021-02-10T11:34:58.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.043 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:58.043 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:58.379 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:58.380 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:58.380 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:58.759 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:58.759 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:58.759 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251201 s, 41.7 MB/s 2021-02-10T11:34:58.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:58.760 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:58.761 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:59.095 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:59.095 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:59.095 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:34:59.468 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224984 s, 46.6 MB/s 2021-02-10T11:34:59.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:34:59.470 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:34:59.470 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:34:59.805 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:34:59.806 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:34:59.806 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:00.183 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:00.183 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:00.183 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247257 s, 42.4 MB/s 2021-02-10T11:35:00.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:00.184 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:00.184 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:00.521 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:00.522 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:00.522 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:00.899 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:00.899 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:00.899 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249462 s, 42.0 MB/s 2021-02-10T11:35:00.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:00.900 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:00.900 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:01.236 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:01.237 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:01.237 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:01.536 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:01.539 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:01.614 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:01.614 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:01.615 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252388 s, 41.5 MB/s 2021-02-10T11:35:01.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:01.616 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:01.616 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:01.873 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:01.874 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:01.874 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:02.250 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:02.251 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:02.251 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235318 s, 44.6 MB/s 2021-02-10T11:35:02.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:02.252 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:02.252 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:02.587 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:02.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:02.588 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802509/27fdd4b7-29b2-4bb0-9436-c92cf09fc28e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:02.962 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:02.962 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:02.962 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025083 s, 41.8 MB/s 2021-02-10T11:35:02.963 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802510 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 Processor -- start 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 -- start start 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd01807da20 0x7fd0180771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 0x7fd0181b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd018077a70 con 0x7fd0180e51a0 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0180777d0 con 0x7fd01807da20 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.394 7fd01db00700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01807de20 con 0x7fd0180e4da0 2021-02-10T11:35:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd01807da20 0x7fd0180771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd01807da20 0x7fd0180771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45226/0 (socket says 172.21.2.117:45226) 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 -- 172.21.2.117:0/2533567481 learned_addr learned my addr 172.21.2.117:0/2533567481 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd016ffd700 1 --2- 172.21.2.117:0/2533567481 >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 0x7fd0181b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 -- 172.21.2.117:0/2533567481 >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 msgr2=0x7fd0181b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 --2- 172.21.2.117:0/2533567481 >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 0x7fd0181b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 -- 172.21.2.117:0/2533567481 >> v1:172.21.2.117:6789/0 conn(0x7fd0180e51a0 legacy=0x7fd018077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 -- 172.21.2.117:0/2533567481 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0181b0710 con 0x7fd01807da20 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd016ffd700 1 --2- 172.21.2.117:0/2533567481 >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 0x7fd0181b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0177fe700 1 --2- 172.21.2.117:0/2533567481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd01807da20 0x7fd0180771b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fd000006700 tx=0x7fd000009b80).ready entity=mon.0 client_cookie=5fbf39dc87e439b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0167fc700 1 -- 172.21.2.117:0/2533567481 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00000c0c0 con 0x7fd01807da20 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0167fc700 1 -- 172.21.2.117:0/2533567481 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00000c340 con 0x7fd01807da20 2021-02-10T11:35:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.395 7fd0167fc700 1 -- 172.21.2.117:0/2533567481 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00000c860 con 0x7fd01807da20 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 -- 172.21.2.117:0/2533567481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd01807da20 msgr2=0x7fd0180771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 --2- 172.21.2.117:0/2533567481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd01807da20 0x7fd0180771b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fd000006700 tx=0x7fd000009b80).stop 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 -- 172.21.2.117:0/2533567481 shutdown_connections 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 --2- 172.21.2.117:0/2533567481 >> v2:172.21.2.125:3300/0 conn(0x7fd0180e4da0 0x7fd0181b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 --2- 172.21.2.117:0/2533567481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd01807da20 0x7fd0180771b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 -- 172.21.2.117:0/2533567481 >> 172.21.2.117:0/2533567481 conn(0x7fd0180ce450 msgr2=0x7fd0180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 -- 172.21.2.117:0/2533567481 shutdown_connections 2021-02-10T11:35:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.396 7fd01db00700 1 -- 172.21.2.117:0/2533567481 wait complete. 2021-02-10T11:35:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 Processor -- start 2021-02-10T11:35:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 -- start start 2021-02-10T11:35:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd01818fdd0 0x7fd018098990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0180c2a90 0x7fd0180c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180997e0 con 0x7fd0181901d0 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180c34b0 con 0x7fd01818fdd0 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd01db00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180c36a0 con 0x7fd0180c2a90 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd016ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd016ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45234/0 (socket says 172.21.2.117:45234) 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.411 7fd016ffd700 1 -- 172.21.2.117:0/4123829754 learned_addr learned my addr 172.21.2.117:0/4123829754 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 -- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0180c2a90 msgr2=0x7fd0180c2e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0180c2a90 0x7fd0180c2e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 -- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd01818fdd0 msgr2=0x7fd018098990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd01818fdd0 0x7fd018098990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 -- 172.21.2.117:0/4123829754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0180c38c0 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd016ffd700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fd008006f20 tx=0x7fd00800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00800bd80 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd008000c90 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0080022b0 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd01db00700 1 -- 172.21.2.117:0/4123829754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0180c38c0 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.412 7fd01db00700 1 -- 172.21.2.117:0/4123829754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0180c18d0 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.413 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd00800a000 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.413 7fd00ffff700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.413 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd008032910 con 0x7fd0181901d0 2021-02-10T11:35:03.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.415 7fd0177fe700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:03.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.415 7fd0177fe700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 0x7fcffc01abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:03.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.444 7fd01db00700 1 -- 172.21.2.117:0/4123829754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd018103320 con 0x7fd0181901d0 2021-02-10T11:35:03.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.448 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0080009b0 con 0x7fd0181901d0 2021-02-10T11:35:03.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.571 7fd01db00700 1 -- 172.21.2.117:0/4123829754 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802510", "target": ["mgr", ""]}) v1 -- 0x7fd018077a70 con 0x7fcffc0186c0 2021-02-10T11:35:03.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.596 7fd00ffff700 1 -- 172.21.2.117:0/4123829754 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd000006520 con 0x7fcffc0186c0 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 msgr2=0x7fcffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 0x7fcffc01abc0 crc :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 msgr2=0x7fd018098fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fd008006f20 tx=0x7fd00800b990).stop 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 shutdown_connections 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcffc0186c0 0x7fcffc01abc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0181901d0 0x7fd018098fb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0180c2a90 0x7fd0180c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 --2- 172.21.2.117:0/4123829754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd01818fdd0 0x7fd018098990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 >> 172.21.2.117:0/4123829754 conn(0x7fd0180a1360 msgr2=0x7fd018091810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 shutdown_connections 2021-02-10T11:35:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:03.605 7fd01db00700 1 -- 172.21.2.117:0/4123829754 wait complete. 2021-02-10T11:35:03.598 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802510 2021-02-10T11:35:04.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.029 7f8649729700 1 Processor -- start 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.029 7f8649729700 1 -- start start 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f8649729700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8644079000 0x7f86440e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f8649729700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440d1d20 con 0x7f864407da20 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f8649729700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8644077a70 con 0x7f86441b00f0 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f8649729700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86440e53c0 con 0x7f8644079000 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f86432bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f864407da20 0x7f86440771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45488/0 (socket says 172.21.2.117:45488) 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f86432bf700 1 -- 172.21.2.117:0/155687432 learned_addr learned my addr 172.21.2.117:0/155687432 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515479911 0 0) 0x7f8634001180 con 0x7f864407da20 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f86422bd700 1 -- 172.21.2.117:0/155687432 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c002fe0 con 0x7f864407da20 2021-02-10T11:35:04.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f8642abe700 1 --2- 172.21.2.117:0/155687432 >> v2:172.21.2.125:3300/0 conn(0x7f8644079000 0x7f86440e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.030 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2248780805 0 0) 0x7f8634001180 con 0x7f864407da20 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 >> v2:172.21.2.125:3300/0 conn(0x7f8644079000 msgr2=0x7f86440e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 --2- 172.21.2.117:0/155687432 >> v2:172.21.2.125:3300/0 conn(0x7f8644079000 0x7f86440e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 >> v1:172.21.2.125:6789/0 conn(0x7f86441b00f0 legacy=0x7f86440d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644079400 con 0x7f864407da20 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8634001680 con 0x7f864407da20 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 >> v1:172.21.2.117:6789/0 conn(0x7f864407da20 legacy=0x7f86440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 0x7f862c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f862c006790 0x7f862c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f862c009590 0x7f862c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00c300 con 0x7f862c003ac0 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00c520 con 0x7f862c006790 2021-02-10T11:35:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86422bd700 1 -- 172.21.2.117:0/155687432 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00c740 con 0x7f862c009590 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 0x7f862c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f8642abe700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f862c006790 0x7f862c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f8643ac0700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f862c009590 0x7f862c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 -- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f862c009590 msgr2=0x7f862c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f862c009590 0x7f862c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 -- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f862c006790 msgr2=0x7f862c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f862c006790 0x7f862c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 -- 172.21.2.117:0/155687432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c00c9e0 con 0x7f862c003ac0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f8643ac0700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f862c009590 0x7f862c00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f8643ac0700 1 -- 172.21.2.117:0/155687432 reap_dead start 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.031 7f86432bf700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 0x7f862c005ef0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f8634001340 tx=0x7f8634007cf0).ready entity=mon.0 client_cookie=881880eb61101db8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8634000ce0 con 0x7f862c003ac0 2021-02-10T11:35:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86340070c0 con 0x7f862c003ac0 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f86422bd700 1 -- 172.21.2.117:0/155687432 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86340043a0 con 0x7f862c003ac0 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 -- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 msgr2=0x7f862c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 0x7f862c005ef0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f8634001340 tx=0x7f8634007cf0).stop 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 -- 172.21.2.117:0/155687432 shutdown_connections 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 --2- 172.21.2.117:0/155687432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f862c003ac0 0x7f862c005ef0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 -- 172.21.2.117:0/155687432 >> 172.21.2.117:0/155687432 conn(0x7f86440ce450 msgr2=0x7f86440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 -- 172.21.2.117:0/155687432 shutdown_connections 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.032 7f8649729700 1 -- 172.21.2.117:0/155687432 wait complete. 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 Processor -- start 2021-02-10T11:35:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 -- start start 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8644104530 0x7f8644104930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86440a2c40 0x7f86440a3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86440c2760 con 0x7f86440a3660 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864409d5f0 con 0x7f86440a2c40 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.047 7f8649729700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8644104f50 con 0x7f8644104530 2021-02-10T11:35:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45250/0 (socket says 172.21.2.117:45250) 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 -- 172.21.2.117:0/896852366 learned_addr learned my addr 172.21.2.117:0/896852366 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 -- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8644104530 msgr2=0x7f8644104930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8644104530 0x7f8644104930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 -- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86440a2c40 msgr2=0x7f86440a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86440a2c40 0x7f86440a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 -- 172.21.2.117:0/896852366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86440a3a60 con 0x7f86440a3660 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f8643ac0700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f863c0099a0 tx=0x7f863c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.048 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863c000ce0 con 0x7f86440a3660 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00c2f0 con 0x7f86440a3660 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f8649729700 1 -- 172.21.2.117:0/896852366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86440a3a60 con 0x7f86440a3660 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f863c00c810 con 0x7f86440a3660 2021-02-10T11:35:04.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f8649729700 1 -- 172.21.2.117:0/896852366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86440c3570 con 0x7f86440a3660 2021-02-10T11:35:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f863c00a000 con 0x7f86440a3660 2021-02-10T11:35:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f862bfff700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 0x7f862401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.049 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f863c032ae0 con 0x7f86440a3660 2021-02-10T11:35:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.050 7f86432bf700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 0x7f862401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.050 7f86432bf700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 0x7f862401abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:04.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.080 7f8649729700 1 -- 172.21.2.117:0/896852366 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644077a70 con 0x7f86440a3660 2021-02-10T11:35:04.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.083 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f863c00be30 con 0x7f86440a3660 2021-02-10T11:35:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.207 7f8649729700 1 -- 172.21.2.117:0/896852366 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802510", "target": ["mgr", ""]}) v1 -- 0x7f8644077a70 con 0x7f86240186c0 2021-02-10T11:35:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.212 7f862bfff700 1 -- 172.21.2.117:0/896852366 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8634000d10 con 0x7f86240186c0 2021-02-10T11:35:04.200 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 msgr2=0x7f862401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 0x7f862401abc0 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 msgr2=0x7f86440c2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f863c0099a0 tx=0x7f863c00bcf0).stop 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 shutdown_connections 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86240186c0 0x7f862401abc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86440a3660 0x7f86440c2c60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86440a2c40 0x7f86440a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 --2- 172.21.2.117:0/896852366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8644104530 0x7f8644104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:04.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 >> 172.21.2.117:0/896852366 conn(0x7f86440bfa60 msgr2=0x7f8644094470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:04.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 shutdown_connections 2021-02-10T11:35:04.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:04.217 7f8649729700 1 -- 172.21.2.117:0/896852366 wait complete. 2021-02-10T11:35:04.209 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802510 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4 2021-02-10T11:35:04.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:04.210 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:04.210 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:04.546 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:04.547 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:04.547 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:04.928 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:04.929 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236015 s, 44.4 MB/s 2021-02-10T11:35:04.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:04.930 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:04.930 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:05.265 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:05.266 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:05.266 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:05.642 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:05.642 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:05.642 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023046 s, 45.5 MB/s 2021-02-10T11:35:05.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:05.643 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:05.643 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:05.980 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:05.980 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:05.981 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:06.356 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:06.357 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:06.357 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227227 s, 46.1 MB/s 2021-02-10T11:35:06.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:06.358 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:06.358 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:06.693 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:06.694 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:06.694 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:07.069 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:07.069 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:07.069 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227797 s, 46.0 MB/s 2021-02-10T11:35:07.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:07.071 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:07.071 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:07.406 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:07.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:07.407 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:07.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:07.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:07.780 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237169 s, 44.2 MB/s 2021-02-10T11:35:07.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:07.781 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:07.781 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:08.117 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:08.118 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:08.118 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:08.494 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:08.494 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:08.494 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236627 s, 44.3 MB/s 2021-02-10T11:35:08.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:08.495 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:08.496 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:08.831 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:08.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:08.832 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:09.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:09.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:09.207 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231413 s, 45.3 MB/s 2021-02-10T11:35:09.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:09.208 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:09.208 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:09.545 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:09.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:09.546 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:09.922 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:09.923 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:09.923 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240671 s, 43.6 MB/s 2021-02-10T11:35:09.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:09.924 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:09.924 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:10.260 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:10.261 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:10.261 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:10.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:10.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:10.639 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259722 s, 40.4 MB/s 2021-02-10T11:35:10.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:10.640 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:10.640 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:10.975 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:10.975 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:10.975 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802510/a5744571-f667-4b6e-b0ed-36e6006d68b4/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:11.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:11.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:11.351 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239127 s, 43.9 MB/s 2021-02-10T11:35:11.352 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802511 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.783 7f776829b700 1 Processor -- start 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.783 7f776829b700 1 -- start start 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f776829b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f776007da20 0x7f77600771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f776829b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77600e51a0 0x7f7760079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f776829b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7760077a70 con 0x7f77600e4da0 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f776829b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77600e5c20 con 0x7f77600e51a0 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f776829b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77600777d0 con 0x7f776007da20 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77600e51a0 0x7f7760079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77600e51a0 0x7f7760079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45256/0 (socket says 172.21.2.117:45256) 2021-02-10T11:35:11.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 -- 172.21.2.117:0/3943602428 learned_addr learned my addr 172.21.2.117:0/3943602428 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 -- 172.21.2.117:0/3943602428 >> v2:172.21.2.125:3300/0 conn(0x7f776007da20 msgr2=0x7f77600771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 --2- 172.21.2.117:0/3943602428 >> v2:172.21.2.125:3300/0 conn(0x7f776007da20 0x7f77600771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 -- 172.21.2.117:0/3943602428 >> v1:172.21.2.125:6789/0 conn(0x7f77600e4da0 legacy=0x7f77601b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 -- 172.21.2.117:0/3943602428 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77601b08e0 con 0x7f77600e51a0 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.784 7f77668b8700 1 --2- 172.21.2.117:0/3943602428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600e51a0 0x7f7760079000 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f7758006f00 tx=0x7f7758009f10).ready entity=mon.0 client_cookie=7558a4d647fec14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f77650b5700 1 -- 172.21.2.117:0/3943602428 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775800b3b0 con 0x7f77600e51a0 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f77650b5700 1 -- 172.21.2.117:0/3943602428 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775800b630 con 0x7f77600e51a0 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f77650b5700 1 -- 172.21.2.117:0/3943602428 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775800bb50 con 0x7f77600e51a0 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 -- 172.21.2.117:0/3943602428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600e51a0 msgr2=0x7f7760079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 --2- 172.21.2.117:0/3943602428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600e51a0 0x7f7760079000 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f7758006f00 tx=0x7f7758009f10).stop 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 -- 172.21.2.117:0/3943602428 shutdown_connections 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 --2- 172.21.2.117:0/3943602428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600e51a0 0x7f7760079000 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 --2- 172.21.2.117:0/3943602428 >> v2:172.21.2.125:3300/0 conn(0x7f776007da20 0x7f77600771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:11.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 -- 172.21.2.117:0/3943602428 >> 172.21.2.117:0/3943602428 conn(0x7f77600ce450 msgr2=0x7f77600e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 -- 172.21.2.117:0/3943602428 shutdown_connections 2021-02-10T11:35:11.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.785 7f776829b700 1 -- 172.21.2.117:0/3943602428 wait complete. 2021-02-10T11:35:11.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 Processor -- start 2021-02-10T11:35:11.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 -- start start 2021-02-10T11:35:11.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601a7bf0 0x7f7760094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7760094080 0x7f7760095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7760095740 con 0x7f7760093c80 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7760095960 con 0x7f7760094080 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f776829b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7760095b80 con 0x7f77601a7bf0 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f77660b7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f77660b7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45260/0 (socket says 172.21.2.117:45260) 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f77660b7700 1 -- 172.21.2.117:0/1898322915 learned_addr learned my addr 172.21.2.117:0/1898322915 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f77668b8700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7760094080 0x7f7760095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.800 7f77658b6700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601a7bf0 0x7f7760094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 -- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601a7bf0 msgr2=0x7f7760094b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601a7bf0 0x7f7760094b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 -- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7760094080 msgr2=0x7f7760095120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7760094080 0x7f7760095120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 -- 172.21.2.117:0/1898322915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760095da0 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f77660b7700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f775400b5c0 tx=0x7f775400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775400bd50 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7754006030 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77540054d0 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f776829b700 1 -- 172.21.2.117:0/1898322915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7760095da0 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.801 7f776829b700 1 -- 172.21.2.117:0/1898322915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7760095da0 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.802 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7754006210 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.802 7f7752ffd700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 0x7f774801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.802 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f77540328f0 con 0x7f7760093c80 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.802 7f77658b6700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 0x7f774801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:11.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.802 7f77658b6700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 0x7f774801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:11.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.834 7f776829b700 1 -- 172.21.2.117:0/1898322915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7760103310 con 0x7f7760093c80 2021-02-10T11:35:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.837 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7754006b80 con 0x7f7760093c80 2021-02-10T11:35:11.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.963 7f776829b700 1 -- 172.21.2.117:0/1898322915 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802511", "target": ["mgr", ""]}) v1 -- 0x7f7760103310 con 0x7f77480186c0 2021-02-10T11:35:11.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.989 7f7752ffd700 1 -- 172.21.2.117:0/1898322915 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f775c006550 con 0x7f77480186c0 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 msgr2=0x7f774801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 0x7f774801abc0 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 msgr2=0x7f77601a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f775400b5c0 tx=0x7f775400b960).stop 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 shutdown_connections 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7760094080 0x7f7760095120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601a7bf0 0x7f7760094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77480186c0 0x7f774801abc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 --2- 172.21.2.117:0/1898322915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7760093c80 0x7f77601a75d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 >> 172.21.2.117:0/1898322915 conn(0x7f77600a1880 msgr2=0x7f77600b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 shutdown_connections 2021-02-10T11:35:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:11.996 7f776829b700 1 -- 172.21.2.117:0/1898322915 wait complete. 2021-02-10T11:35:11.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802511 2021-02-10T11:35:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 Processor -- start 2021-02-10T11:35:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 -- start start 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900d1530 0x7f71900d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7190077a70 con 0x7f7190079000 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7190079400 con 0x7f719007da20 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.420 7f7196b98700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719007de20 con 0x7f71900d1530 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7190079000 0x7f71900e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45516/0 (socket says 172.21.2.117:45516) 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718ffff700 1 -- 172.21.2.117:0/2595395820 learned_addr learned my addr 172.21.2.117:0/2595395820 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823688711 0 0) 0x7f7180001060 con 0x7f7190079000 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f717c002fe0 con 0x7f7190079000 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f71951b5700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900d1530 0x7f71900d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 983286619 0 0) 0x7f7180001250 con 0x7f7190079000 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900d1530 msgr2=0x7f71900d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900d1530 0x7f71900d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 >> v1:172.21.2.125:6789/0 conn(0x7f719007da20 legacy=0x7f71900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71901b00f0 con 0x7f7190079000 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7180001a20 con 0x7f7190079000 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f71951b5700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900d1530 0x7f71900d1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 >> v1:172.21.2.117:6789/0 conn(0x7f7190079000 legacy=0x7f71900e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f717c003a60 0x7f717c005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f717c006750 0x7f717c008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f717c0094f0 0x7f717c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c00c2c0 con 0x7f717c006750 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717c00c4e0 con 0x7f717c0094f0 2021-02-10T11:35:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717c00c700 con 0x7f717c003a60 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.421 7f718ffff700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f717c006750 0x7f717c008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f71949b4700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f717c003a60 0x7f717c005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 -- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f717c003a60 msgr2=0x7f717c005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f717c003a60 0x7f717c005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 -- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f717c0094f0 msgr2=0x7f717c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f717c0094f0 0x7f717c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 -- 172.21.2.117:0/2595395820 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c00c9a0 con 0x7f717c006750 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f71949b4700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f717c003a60 0x7f717c005eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718ffff700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f717c006750 0x7f717c008c50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f718004bb90 tx=0x7f718004e310).ready entity=mon.0 client_cookie=11e961df1a8bbb7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7180001d50 con 0x7f717c006750 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71800062e0 con 0x7f717c006750 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f718f7fe700 1 -- 172.21.2.117:0/2595395820 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718004e980 con 0x7f717c006750 2021-02-10T11:35:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f7196b98700 1 -- 172.21.2.117:0/2595395820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f717c006750 msgr2=0x7f717c008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f7196b98700 1 --2- 172.21.2.117:0/2595395820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f717c006750 0x7f717c008c50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f718004bb90 tx=0x7f718004e310).stop 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.422 7f71951b5700 1 -- 172.21.2.117:0/2595395820 reap_dead start 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.423 7f7196b98700 1 -- 172.21.2.117:0/2595395820 shutdown_connections 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.423 7f7196b98700 1 -- 172.21.2.117:0/2595395820 >> 172.21.2.117:0/2595395820 conn(0x7f71900ce450 msgr2=0x7f71900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.423 7f7196b98700 1 -- 172.21.2.117:0/2595395820 shutdown_connections 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.423 7f7196b98700 1 -- 172.21.2.117:0/2595395820 wait complete. 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.441 7f7196b98700 1 Processor -- start 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.441 7f7196b98700 1 -- start start 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71900b77d0 0x7f71900b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7190189970 con 0x7f71900a51f0 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7190190aa0 con 0x7f71900b77d0 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f7196b98700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71900b81f0 con 0x7f71900a47d0 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f71949b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f718ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f71949b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:42956/0 (socket says 172.21.2.117:42956) 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f718ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45280/0 (socket says 172.21.2.117:45280) 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f71949b4700 1 -- 172.21.2.117:0/607997620 learned_addr learned my addr 172.21.2.117:0/607997620 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.442 7f71951b5700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71900b77d0 0x7f71900b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 -- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 msgr2=0x7f71900a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 -- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71900b77d0 msgr2=0x7f71900b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71900b77d0 0x7f71900b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 -- 172.21.2.117:0/607997620 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900b8410 con 0x7f71900a51f0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f71949b4700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718ffff700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f7180000f30 tx=0x7f718004ec30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7180000cc0 con 0x7f71900a51f0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180006030 con 0x7f71900a51f0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71800064c0 con 0x7f71900a51f0 2021-02-10T11:35:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f7196b98700 1 -- 172.21.2.117:0/607997620 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71900b8410 con 0x7f71900a51f0 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.443 7f7196b98700 1 -- 172.21.2.117:0/607997620 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71900b8410 con 0x7f71900a51f0 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.444 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7180006210 con 0x7f71900a51f0 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.444 7f718d7fa700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 0x7f717401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.444 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7180076c00 con 0x7f71900a51f0 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.444 7f71949b4700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 0x7f717401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:12.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.444 7f71949b4700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 0x7f717401abd0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.470 7f7196b98700 1 -- 172.21.2.117:0/607997620 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7190077a70 con 0x7f71900a51f0 2021-02-10T11:35:12.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.473 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7180006300 con 0x7f71900a51f0 2021-02-10T11:35:12.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.602 7f7196b98700 1 -- 172.21.2.117:0/607997620 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802511", "target": ["mgr", ""]}) v1 -- 0x7f7190077a70 con 0x7f71740186d0 2021-02-10T11:35:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.606 7f718d7fa700 1 -- 172.21.2.117:0/607997620 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f7178000d20 con 0x7f71740186d0 2021-02-10T11:35:12.594 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8 2021-02-10T11:35:12.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 -- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 msgr2=0x7f717401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 0x7f717401abd0 crc :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 -- 172.21.2.117:0/607997620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 msgr2=0x7f71900a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f7180000f30 tx=0x7f718004ec30).stop 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 -- 172.21.2.117:0/607997620 shutdown_connections 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186d0 0x7f717401abd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900a51f0 0x7f71900a55f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71900b77d0 0x7f71900b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 --2- 172.21.2.117:0/607997620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900a47d0 0x7f71900a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:12.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.611 7f7196b98700 1 -- 172.21.2.117:0/607997620 >> 172.21.2.117:0/607997620 conn(0x7f71900b5140 msgr2=0x7f71900a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:12.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.612 7f7196b98700 1 -- 172.21.2.117:0/607997620 shutdown_connections 2021-02-10T11:35:12.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:12.612 7f7196b98700 1 -- 172.21.2.117:0/607997620 wait complete. 2021-02-10T11:35:12.604 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802511 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8 2021-02-10T11:35:12.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:12.604 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:12.604 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:12.939 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:12.940 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:12.940 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:13.320 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:13.320 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:13.320 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253014 s, 41.4 MB/s 2021-02-10T11:35:13.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:13.321 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:13.321 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:13.656 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:13.656 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:13.657 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:14.034 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:14.034 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:14.034 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252588 s, 41.5 MB/s 2021-02-10T11:35:14.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:14.035 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:14.036 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:14.372 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:14.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:14.373 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:14.750 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:14.750 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233746 s, 44.9 MB/s 2021-02-10T11:35:14.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:14.751 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:14.751 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:15.086 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:15.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:15.087 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:15.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:15.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:15.462 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238295 s, 44.0 MB/s 2021-02-10T11:35:15.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:15.463 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:15.463 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:15.802 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:15.803 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:15.803 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:16.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:16.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:16.181 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248214 s, 42.2 MB/s 2021-02-10T11:35:16.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:16.182 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:16.182 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:16.518 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:16.518 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:16.519 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:16.892 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:16.893 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:16.893 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227498 s, 46.1 MB/s 2021-02-10T11:35:16.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:16.894 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:16.894 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:17.230 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:17.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:17.231 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233216 s, 45.0 MB/s 2021-02-10T11:35:17.605 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:17.605 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:17.605 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:17.941 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:17.941 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:17.942 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:18.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:18.318 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:18.318 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242065 s, 43.3 MB/s 2021-02-10T11:35:18.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:18.319 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:18.319 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:18.655 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:18.655 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:18.655 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:19.031 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:19.031 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:19.031 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235828 s, 44.5 MB/s 2021-02-10T11:35:19.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:19.032 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:19.032 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:19.368 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:19.369 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:19.369 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802511/465f793a-0f95-4301-b273-86a5f1f39ac8/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:19.749 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0276788 s, 37.9 MB/s 2021-02-10T11:35:19.750 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802512 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 Processor -- start 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 -- start start 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 0x7f6ff80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff8077a70 con 0x7f6ff80e4da0 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ff80777d0 con 0x7f6ff807da20 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.182 7f6ffe772700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81b08e0 con 0x7f6ff80e51a0 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ff7fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 0x7f6ff80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ff77fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6ff80e4da0 0x7f6ff81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38460/0 (socket says 172.21.2.117:38460) 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:42964/0 (socket says 172.21.2.117:42964) 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ff77fe700 1 -- 172.21.2.117:0/2546728383 learned_addr learned my addr 172.21.2.117:0/2546728383 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 -- 172.21.2.117:0/2546728383 >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 msgr2=0x7f6ff80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 --2- 172.21.2.117:0/2546728383 >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 0x7f6ff80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 -- 172.21.2.117:0/2546728383 >> v1:172.21.2.125:6789/0 conn(0x7f6ff80e4da0 legacy=0x7f6ff81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ffcd8f700 1 -- 172.21.2.117:0/2546728383 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff807de20 con 0x7f6ff80e51a0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ff6ffd700 1 -- 172.21.2.117:0/2546728383 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528775904 0 0) 0x7f6fec0010b0 con 0x7f6ff80e4da0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.183 7f6ff7fff700 1 --2- 172.21.2.117:0/2546728383 >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 0x7f6ff80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ffcd8f700 1 --2- 172.21.2.117:0/2546728383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6ff0006e50 tx=0x7f6ff0009e70).ready entity=mon.2 client_cookie=69218e768f9b85d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ff6ffd700 1 -- 172.21.2.117:0/2546728383 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff000b510 con 0x7f6ff80e51a0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ff6ffd700 1 -- 172.21.2.117:0/2546728383 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff0006790 con 0x7f6ff80e51a0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ff6ffd700 1 -- 172.21.2.117:0/2546728383 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff000b940 con 0x7f6ff80e51a0 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ffe772700 1 -- 172.21.2.117:0/2546728383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 msgr2=0x7f6ff8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.184 7f6ffe772700 1 --2- 172.21.2.117:0/2546728383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6ff0006e50 tx=0x7f6ff0009e70).stop 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 -- 172.21.2.117:0/2546728383 shutdown_connections 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 --2- 172.21.2.117:0/2546728383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff80e51a0 0x7f6ff8079000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 --2- 172.21.2.117:0/2546728383 >> v2:172.21.2.125:3300/0 conn(0x7f6ff807da20 0x7f6ff80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 -- 172.21.2.117:0/2546728383 >> 172.21.2.117:0/2546728383 conn(0x7f6ff80ce450 msgr2=0x7f6ff80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 -- 172.21.2.117:0/2546728383 shutdown_connections 2021-02-10T11:35:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.185 7f6ffe772700 1 -- 172.21.2.117:0/2546728383 wait complete. 2021-02-10T11:35:20.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 Processor -- start 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 -- start start 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ff80b7fc0 0x7f6ff81042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 0x7f6ff8104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff81054f0 con 0x7f6ff8090710 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8195a30 con 0x7f6ff80b7fc0 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.199 7f6ffe772700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8195fb0 con 0x7f6ff81048f0 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45290/0 (socket says 172.21.2.117:45290) 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 -- 172.21.2.117:0/1832807289 learned_addr learned my addr 172.21.2.117:0/1832807289 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ff7fff700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ff80b7fc0 0x7f6ff81042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ff77fe700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 0x7f6ff8104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 -- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 msgr2=0x7f6ff8104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 0x7f6ff8104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 -- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ff80b7fc0 msgr2=0x7f6ff81042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ff80b7fc0 0x7f6ff81042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 -- 172.21.2.117:0/1832807289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80b7b60 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ff77fe700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 0x7f6ff8104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.200 7f6ffcd8f700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f6ff00099f0 tx=0x7f6ff000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff00093d0 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00090c0 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ff00015a0 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff80b7b60 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8091420 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ff00011a0 con 0x7f6ff8090710 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 0x7f6fd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.201 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ff0032a20 con 0x7f6ff8090710 2021-02-10T11:35:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.202 7f6ff7fff700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 0x7f6fd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.202 7f6ff7fff700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 0x7f6fd801abc0 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.232 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8103280 con 0x7f6ff8090710 2021-02-10T11:35:20.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.234 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ff0028c50 con 0x7f6ff8090710 2021-02-10T11:35:20.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.359 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802512", "target": ["mgr", ""]}) v1 -- 0x7f6ff8091420 con 0x7f6fd80186c0 2021-02-10T11:35:20.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.385 7f6ff4ff9700 1 -- 172.21.2.117:0/1832807289 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6fe8006a30 con 0x7f6fd80186c0 2021-02-10T11:35:20.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 msgr2=0x7f6fd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 0x7f6fd801abc0 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 msgr2=0x7f6ff8090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f6ff00099f0 tx=0x7f6ff000bd10).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 shutdown_connections 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fd80186c0 0x7f6fd801abc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ff8090710 0x7f6ff8090b10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ff81048f0 0x7f6ff8104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 --2- 172.21.2.117:0/1832807289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ff80b7fc0 0x7f6ff81042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 >> 172.21.2.117:0/1832807289 conn(0x7f6ff80a1850 msgr2=0x7f6ff80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 shutdown_connections 2021-02-10T11:35:20.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.393 7f6ffe772700 1 -- 172.21.2.117:0/1832807289 wait complete. 2021-02-10T11:35:20.385 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802512 2021-02-10T11:35:20.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.816 7f3ac668f700 1 Processor -- start 2021-02-10T11:35:20.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.816 7f3ac668f700 1 -- start start 2021-02-10T11:35:20.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ac00d1510 0x7f3ac00d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac0077a70 con 0x7f3ac0079000 2021-02-10T11:35:20.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac0079400 con 0x7f3ac007da20 2021-02-10T11:35:20.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac007de20 con 0x7f3ac00d1510 2021-02-10T11:35:20.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ac00d1510 0x7f3ac00d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ac00d1510 0x7f3ac00d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45296/0 (socket says 172.21.2.117:45296) 2021-02-10T11:35:20.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 -- 172.21.2.117:0/1924093016 learned_addr learned my addr 172.21.2.117:0/1924093016 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:20.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3abf7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3ac0079000 0x7f3ac00e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45544/0 (socket says 172.21.2.117:45544) 2021-02-10T11:35:20.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 -- 172.21.2.117:0/1924093016 >> v1:172.21.2.125:6789/0 conn(0x7f3ac007da20 legacy=0x7f3ac00771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:20.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 -- 172.21.2.117:0/1924093016 >> v1:172.21.2.117:6789/0 conn(0x7f3ac0079000 legacy=0x7f3ac00e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 -- 172.21.2.117:0/1924093016 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac01b00d0 con 0x7f3ac00d1510 2021-02-10T11:35:20.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac4cac700 1 --2- 172.21.2.117:0/1924093016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac00d1510 0x7f3ac00d1910 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f3ab8006f00 tx=0x7f3ab8009f10).ready entity=mon.0 client_cookie=ee56458b0a89c03 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3abeffd700 1 -- 172.21.2.117:0/1924093016 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab800b3b0 con 0x7f3ac00d1510 2021-02-10T11:35:20.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3abeffd700 1 -- 172.21.2.117:0/1924093016 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab8000d30 con 0x7f3ac00d1510 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3abeffd700 1 -- 172.21.2.117:0/1924093016 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab800b820 con 0x7f3ac00d1510 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 -- 172.21.2.117:0/1924093016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac00d1510 msgr2=0x7f3ac00d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.817 7f3ac668f700 1 --2- 172.21.2.117:0/1924093016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac00d1510 0x7f3ac00d1910 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f3ab8006f00 tx=0x7f3ab8009f10).stop 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.818 7f3ac668f700 1 -- 172.21.2.117:0/1924093016 shutdown_connections 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.818 7f3ac668f700 1 --2- 172.21.2.117:0/1924093016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac00d1510 0x7f3ac00d1910 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.818 7f3ac668f700 1 -- 172.21.2.117:0/1924093016 >> 172.21.2.117:0/1924093016 conn(0x7f3ac00ce430 msgr2=0x7f3ac00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.818 7f3ac668f700 1 -- 172.21.2.117:0/1924093016 shutdown_connections 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.818 7f3ac668f700 1 -- 172.21.2.117:0/1924093016 wait complete. 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.832 7f3ac668f700 1 Processor -- start 2021-02-10T11:35:20.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 -- start start 2021-02-10T11:35:20.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ac0098500 0x7f3ac00bf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ac00bf860 0x7f3ac00bfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00c33c0 con 0x7f3ac0097ae0 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac00c2e00 con 0x7f3ac0098500 2021-02-10T11:35:20.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3ac668f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0098900 con 0x7f3ac00bf860 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3abffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3abffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45300/0 (socket says 172.21.2.117:45300) 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.833 7f3abffff700 1 -- 172.21.2.117:0/4195782307 learned_addr learned my addr 172.21.2.117:0/4195782307 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 -- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ac00bf860 msgr2=0x7f3ac00bfc60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ac00bf860 0x7f3ac00bfc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 -- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ac0098500 msgr2=0x7f3ac00bf240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:20.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ac0098500 0x7f3ac00bf240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 -- 172.21.2.117:0/4195782307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00c0280 con 0x7f3ac0097ae0 2021-02-10T11:35:20.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abffff700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f3ab000b5b0 tx=0x7f3ab000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab000bd40 con 0x7f3ac0097ae0 2021-02-10T11:35:20.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0006020 con 0x7f3ac0097ae0 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ab00054a0 con 0x7f3ac0097ae0 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac00c0280 con 0x7f3ac0097ae0 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.834 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac00c0810 con 0x7f3ac0097ae0 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.835 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ab0006200 con 0x7f3ac0097ae0 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.835 7f3abcff9700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:20.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.835 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3ab0032710 con 0x7f3ac0097ae0 2021-02-10T11:35:20.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.835 7f3abf7fe700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:20.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.836 7f3abf7fe700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 0x7f3aa001abc0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:20.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.866 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac0106040 con 0x7f3ac0097ae0 2021-02-10T11:35:20.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.869 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ab0006300 con 0x7f3ac0097ae0 2021-02-10T11:35:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.995 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802512", "target": ["mgr", ""]}) v1 -- 0x7f3ac00c0810 con 0x7f3aa00186c0 2021-02-10T11:35:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:20.998 7f3abcff9700 1 -- 172.21.2.117:0/4195782307 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f3ab4001490 con 0x7f3aa00186c0 2021-02-10T11:35:20.982 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d 2021-02-10T11:35:20.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 msgr2=0x7f3aa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 0x7f3aa001abc0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 msgr2=0x7f3ac0097ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:20.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f3ab000b5b0 tx=0x7f3ab000b950).stop 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 shutdown_connections 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ac00bf860 0x7f3ac00bfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ac0098500 0x7f3ac00bf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aa00186c0 0x7f3aa001abc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 --2- 172.21.2.117:0/4195782307 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ac0097ae0 0x7f3ac0097ee0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 >> 172.21.2.117:0/4195782307 conn(0x7f3ac00bd740 msgr2=0x7f3ac00bdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 shutdown_connections 2021-02-10T11:35:20.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:21.000 7f3ac668f700 1 -- 172.21.2.117:0/4195782307 wait complete. 2021-02-10T11:35:20.992 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802512 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d 2021-02-10T11:35:20.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:20.993 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:20.993 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:21.328 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:21.328 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:21.329 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:21.712 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:21.712 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:21.712 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251267 s, 41.7 MB/s 2021-02-10T11:35:21.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:21.713 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:21.713 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:22.049 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:22.049 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:22.049 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:22.427 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:22.427 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:22.427 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259474 s, 40.4 MB/s 2021-02-10T11:35:22.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:22.428 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:22.429 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:22.763 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:22.764 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:22.764 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:23.137 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:23.137 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:23.137 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253192 s, 41.4 MB/s 2021-02-10T11:35:23.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:23.138 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:23.138 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:23.473 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:23.473 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:23.473 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:23.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:23.850 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:23.851 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024741 s, 42.4 MB/s 2021-02-10T11:35:23.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:23.852 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:23.852 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:24.187 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:24.187 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:24.187 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:24.563 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:24.564 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:24.564 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243474 s, 43.1 MB/s 2021-02-10T11:35:24.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:24.565 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:24.565 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:24.900 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:24.900 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:24.900 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:25.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:25.277 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:25.277 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250072 s, 41.9 MB/s 2021-02-10T11:35:25.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:25.278 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:25.278 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:25.615 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:25.615 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:25.615 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:25.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:25.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:25.990 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230038 s, 45.6 MB/s 2021-02-10T11:35:25.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:25.992 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:25.992 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:26.328 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:26.328 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:26.328 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244942 s, 42.8 MB/s 2021-02-10T11:35:26.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:26.705 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:26.706 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:27.042 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:27.042 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:27.042 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:27.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:27.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:27.418 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250737 s, 41.8 MB/s 2021-02-10T11:35:27.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:27.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:27.419 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:27.755 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:27.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:27.756 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802512/a3e89cc1-7eb4-4eca-ada9-7d5a0225454d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:28.128 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:28.129 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:28.129 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240839 s, 43.5 MB/s 2021-02-10T11:35:28.130 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802513 2021-02-10T11:35:28.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.561 7efe2326e700 1 Processor -- start 2021-02-10T11:35:28.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.561 7efe2326e700 1 -- start start 2021-02-10T11:35:28.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe2326e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efe1c0d1530 0x7efe1c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe2326e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c077a70 con 0x7efe1c079000 2021-02-10T11:35:28.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe2326e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c079400 con 0x7efe1c07da20 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe2326e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c07de20 con 0x7efe1c0d1530 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe20889700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efe1c079000 0x7efe1c0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45556/0 (socket says 172.21.2.117:45556) 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe20889700 1 -- 172.21.2.117:0/3870217537 learned_addr learned my addr 172.21.2.117:0/3870217537 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe2188b700 1 --2- 172.21.2.117:0/3870217537 >> v2:172.21.2.125:3300/0 conn(0x7efe1c0d1530 0x7efe1c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.562 7efe0ffff700 1 -- 172.21.2.117:0/3870217537 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564028854 0 0) 0x7efe18000ce0 con 0x7efe1c079000 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe0ffff700 1 -- 172.21.2.117:0/3870217537 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08002fe0 con 0x7efe1c079000 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2188b700 1 -- 172.21.2.117:0/3870217537 >> v1:172.21.2.125:6789/0 conn(0x7efe1c07da20 legacy=0x7efe1c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2188b700 1 -- 172.21.2.117:0/3870217537 >> v1:172.21.2.117:6789/0 conn(0x7efe1c079000 legacy=0x7efe1c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2188b700 1 -- 172.21.2.117:0/3870217537 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c07dfd0 con 0x7efe1c0d1530 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2188b700 1 --2- 172.21.2.117:0/3870217537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c0d1530 0x7efe1c0d1930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7efe14006e50 tx=0x7efe14009e70).ready entity=mon.1 client_cookie=6c87e1b467c8533f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe0ffff700 1 -- 172.21.2.117:0/3870217537 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe14006a20 con 0x7efe1c0d1530 2021-02-10T11:35:28.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe0ffff700 1 -- 172.21.2.117:0/3870217537 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14006790 con 0x7efe1c0d1530 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2326e700 1 -- 172.21.2.117:0/3870217537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c0d1530 msgr2=0x7efe1c0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2326e700 1 --2- 172.21.2.117:0/3870217537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c0d1530 0x7efe1c0d1930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7efe14006e50 tx=0x7efe14009e70).stop 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2326e700 1 -- 172.21.2.117:0/3870217537 shutdown_connections 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2326e700 1 --2- 172.21.2.117:0/3870217537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c0d1530 0x7efe1c0d1930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.563 7efe2326e700 1 -- 172.21.2.117:0/3870217537 >> 172.21.2.117:0/3870217537 conn(0x7efe1c0ce450 msgr2=0x7efe1c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.564 7efe2326e700 1 -- 172.21.2.117:0/3870217537 shutdown_connections 2021-02-10T11:35:28.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.564 7efe2326e700 1 -- 172.21.2.117:0/3870217537 wait complete. 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.578 7efe2326e700 1 Processor -- start 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.578 7efe2326e700 1 -- start start 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe1c0933f0 0x7efe1c0937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c107050 0x7efe1c107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c107a70 con 0x7efe1c093e10 2021-02-10T11:35:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0c2c60 con 0x7efe1c107050 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe2326e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0c2e80 con 0x7efe1c0933f0 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe20889700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe20889700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45312/0 (socket says 172.21.2.117:45312) 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.579 7efe20889700 1 -- 172.21.2.117:0/2963766358 learned_addr learned my addr 172.21.2.117:0/2963766358 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 -- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe1c0933f0 msgr2=0x7efe1c0937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe1c0933f0 0x7efe1c0937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 -- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c107050 msgr2=0x7efe1c107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c107050 0x7efe1c107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 -- 172.21.2.117:0/2963766358 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0c30a0 con 0x7efe1c093e10 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe20889700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7efe1800b530 tx=0x7efe1800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe1800bcc0 con 0x7efe1c093e10 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe18006030 con 0x7efe1c093e10 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe2326e700 1 -- 172.21.2.117:0/2963766358 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c0c30a0 con 0x7efe1c093e10 2021-02-10T11:35:28.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe2326e700 1 -- 172.21.2.117:0/2963766358 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c0c30a0 con 0x7efe1c093e10 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.580 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe18004620 con 0x7efe1c093e10 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.581 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe180043a0 con 0x7efe1c093e10 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.581 7efe0dffb700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 0x7efdfc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.581 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efe180328c0 con 0x7efe1c093e10 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.581 7efe2108a700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 0x7efdfc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:28.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.582 7efe2108a700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 0x7efdfc01abd0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:28.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.611 7efe2326e700 1 -- 172.21.2.117:0/2963766358 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c0ce450 con 0x7efe1c093e10 2021-02-10T11:35:28.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.614 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe18006b90 con 0x7efe1c093e10 2021-02-10T11:35:28.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.739 7efe2326e700 1 -- 172.21.2.117:0/2963766358 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802513", "target": ["mgr", ""]}) v1 -- 0x7efe1c0ce450 con 0x7efdfc0186d0 2021-02-10T11:35:28.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.765 7efe0dffb700 1 -- 172.21.2.117:0/2963766358 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efe10000960 con 0x7efdfc0186d0 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.772 7efe2326e700 1 -- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 msgr2=0x7efdfc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.772 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 0x7efdfc01abd0 crc :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.772 7efe2326e700 1 -- 172.21.2.117:0/2963766358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 msgr2=0x7efe1c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.772 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7efe1800b530 tx=0x7efe1800b8d0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 -- 172.21.2.117:0/2963766358 shutdown_connections 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdfc0186d0 0x7efdfc01abd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe1c093e10 0x7efe1c106a30 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe1c107050 0x7efe1c107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 --2- 172.21.2.117:0/2963766358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe1c0933f0 0x7efe1c0937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 -- 172.21.2.117:0/2963766358 >> 172.21.2.117:0/2963766358 conn(0x7efe1c1116a0 msgr2=0x7efe1c0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:28.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 -- 172.21.2.117:0/2963766358 shutdown_connections 2021-02-10T11:35:28.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:28.773 7efe2326e700 1 -- 172.21.2.117:0/2963766358 wait complete. 2021-02-10T11:35:28.765 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802513 2021-02-10T11:35:29.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.197 7fb21ec53700 1 Processor -- start 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.197 7fb21ec53700 1 -- start start 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21ec53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 0x7fb2180e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21ec53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2181b00f0 0x7fb2181b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21ec53700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218077a70 con 0x7fb21807da20 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21ec53700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2180e53c0 con 0x7fb2181b00f0 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21ec53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb218079400 con 0x7fb218079000 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2181b00f0 0x7fb2181b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2181b00f0 0x7fb2181b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45322/0 (socket says 172.21.2.117:45322) 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 -- 172.21.2.117:0/1452633647 learned_addr learned my addr 172.21.2.117:0/1452633647 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb217fff700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 0x7fb2180e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:29.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 -- 172.21.2.117:0/1452633647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 msgr2=0x7fb2180e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 0x7fb2180e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 -- 172.21.2.117:0/1452633647 >> v1:172.21.2.125:6789/0 conn(0x7fb21807da20 legacy=0x7fb2180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 -- 172.21.2.117:0/1452633647 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21807de20 con 0x7fb2181b00f0 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb217fff700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 0x7fb2180e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb21d270700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2181b00f0 0x7fb2181b04f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fb210006f00 tx=0x7fb210009f10).ready entity=mon.0 client_cookie=c97bebbda7777c70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.198 7fb2177fe700 1 -- 172.21.2.117:0/1452633647 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb21000b3b0 con 0x7fb2181b00f0 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb2177fe700 1 -- 172.21.2.117:0/1452633647 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000b730 con 0x7fb2181b00f0 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb2177fe700 1 -- 172.21.2.117:0/1452633647 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb21000bc10 con 0x7fb2181b00f0 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 -- 172.21.2.117:0/1452633647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2181b00f0 msgr2=0x7fb2181b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2181b00f0 0x7fb2181b04f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7fb210006f00 tx=0x7fb210009f10).stop 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 -- 172.21.2.117:0/1452633647 shutdown_connections 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2181b00f0 0x7fb2181b04f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 --2- 172.21.2.117:0/1452633647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb218079000 0x7fb2180e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 -- 172.21.2.117:0/1452633647 >> 172.21.2.117:0/1452633647 conn(0x7fb2180ce450 msgr2=0x7fb2180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 -- 172.21.2.117:0/1452633647 shutdown_connections 2021-02-10T11:35:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.199 7fb21ec53700 1 -- 172.21.2.117:0/1452633647 wait complete. 2021-02-10T11:35:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.213 7fb21ec53700 1 Processor -- start 2021-02-10T11:35:29.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 -- start start 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb21819a4a0 0x7fb2180b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2180a3800 0x7fb2180a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180a4220 con 0x7fb2180b9e60 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180a4410 con 0x7fb21819a4a0 2021-02-10T11:35:29.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb21ec53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2180ba910 con 0x7fb2180a3800 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45324/0 (socket says 172.21.2.117:45324) 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 -- 172.21.2.117:0/3448783999 learned_addr learned my addr 172.21.2.117:0/3448783999 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 -- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2180a3800 msgr2=0x7fb2180a3c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2180a3800 0x7fb2180a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 -- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb21819a4a0 msgr2=0x7fb2180b9840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb21819a4a0 0x7fb2180b9840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 -- 172.21.2.117:0/3448783999 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2180bab30 con 0x7fb2180b9e60 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.214 7fb217fff700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fb208006f20 tx=0x7fb20800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb20800bd80 con 0x7fb2180b9e60 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208000c90 con 0x7fb2180b9e60 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2080022b0 con 0x7fb2180b9e60 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2180bab30 con 0x7fb2180b9e60 2021-02-10T11:35:29.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2180bab30 con 0x7fb2180b9e60 2021-02-10T11:35:29.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb20800a000 con 0x7fb2180b9e60 2021-02-10T11:35:29.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 0x7fb1fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:29.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.215 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb2080327d0 con 0x7fb2180b9e60 2021-02-10T11:35:29.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.216 7fb21ca6f700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 0x7fb1fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:29.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.216 7fb21ca6f700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 0x7fb1fc01ac10 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:29.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.247 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2181b4f00 con 0x7fb2180b9e60 2021-02-10T11:35:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.250 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb208006bb0 con 0x7fb2180b9e60 2021-02-10T11:35:29.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.375 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802513", "target": ["mgr", ""]}) v1 -- 0x7fb2181b4f00 con 0x7fb1fc018730 2021-02-10T11:35:29.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.380 7fb2157fa700 1 -- 172.21.2.117:0/3448783999 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb200006970 con 0x7fb1fc018730 2021-02-10T11:35:29.367 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031 2021-02-10T11:35:29.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 msgr2=0x7fb1fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:29.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 0x7fb1fc01ac10 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 msgr2=0x7fb2180a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:29.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7fb208006f20 tx=0x7fb20800b990).stop 2021-02-10T11:35:29.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 shutdown_connections 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1fc018730 0x7fb1fc01ac10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2180b9e60 0x7fb2180a31e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2180a3800 0x7fb2180a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 --2- 172.21.2.117:0/3448783999 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb21819a4a0 0x7fb2180b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 >> 172.21.2.117:0/3448783999 conn(0x7fb218107b60 msgr2=0x7fb2180c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 shutdown_connections 2021-02-10T11:35:29.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:29.385 7fb21ec53700 1 -- 172.21.2.117:0/3448783999 wait complete. 2021-02-10T11:35:29.377 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802513 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031 2021-02-10T11:35:29.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:29.377 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:29.377 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:29.714 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:29.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:29.714 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245405 s, 42.7 MB/s 2021-02-10T11:35:30.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:30.099 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:30.099 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:30.436 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:30.436 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:30.436 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:30.815 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:30.815 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:30.815 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258725 s, 40.5 MB/s 2021-02-10T11:35:30.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:30.816 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:30.816 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:31.151 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:31.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:31.152 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:31.527 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:31.527 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:31.528 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242134 s, 43.3 MB/s 2021-02-10T11:35:31.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:31.529 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:31.529 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:31.865 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:31.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:31.865 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:31.907 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:31.911 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:35:32.242 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240084 s, 43.7 MB/s 2021-02-10T11:35:32.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.244 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:32.244 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:32.579 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:32.580 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:32.580 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:32.953 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:32.953 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:32.953 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247881 s, 42.3 MB/s 2021-02-10T11:35:32.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:32.954 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:32.955 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:33.289 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:33.290 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:33.290 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:33.665 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:33.665 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:33.666 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023546 s, 44.5 MB/s 2021-02-10T11:35:33.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:33.667 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:33.667 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:34.003 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:34.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:34.003 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:34.379 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:34.380 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:34.380 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249696 s, 42.0 MB/s 2021-02-10T11:35:34.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:34.381 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:34.381 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:34.716 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:34.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:34.716 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:35.089 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:35.089 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:35.089 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229238 s, 45.7 MB/s 2021-02-10T11:35:35.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:35.091 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:35.091 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:35.428 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:35.428 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:35.428 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240675 s, 43.6 MB/s 2021-02-10T11:35:35.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:35.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:35.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:36.144 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:36.144 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:36.144 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802513/028c9c1e-1210-4e2f-8330-1c7cad272031/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:36.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:36.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:36.519 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235593 s, 44.5 MB/s 2021-02-10T11:35:36.521 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802514 2021-02-10T11:35:36.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.953 7f589b684700 1 Processor -- start 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.953 7f589b684700 1 -- start start 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f589b684700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f589407da20 0x7f58940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f589b684700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58940e51a0 0x7f5894079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f589b684700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5894077a70 con 0x7f58940e4da0 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f589b684700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58940e5c20 con 0x7f589407da20 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f589b684700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58940777d0 con 0x7f58940e51a0 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f58994a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f589407da20 0x7f58940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f58994a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f589407da20 0x7f58940771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45332/0 (socket says 172.21.2.117:45332) 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f58994a0700 1 -- 172.21.2.117:0/4070211095 learned_addr learned my addr 172.21.2.117:0/4070211095 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.954 7f5899ca1700 1 --2- 172.21.2.117:0/4070211095 >> v2:172.21.2.125:3300/0 conn(0x7f58940e51a0 0x7f5894079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f58994a0700 1 -- 172.21.2.117:0/4070211095 >> v2:172.21.2.125:3300/0 conn(0x7f58940e51a0 msgr2=0x7f5894079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:36.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f58994a0700 1 --2- 172.21.2.117:0/4070211095 >> v2:172.21.2.125:3300/0 conn(0x7f58940e51a0 0x7f5894079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f58994a0700 1 -- 172.21.2.117:0/4070211095 >> v1:172.21.2.125:6789/0 conn(0x7f58940e4da0 legacy=0x7f58941b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f58994a0700 1 -- 172.21.2.117:0/4070211095 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58941b08e0 con 0x7f589407da20 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f58994a0700 1 --2- 172.21.2.117:0/4070211095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589407da20 0x7f58940771b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f5888006700 tx=0x7f5888009b80).ready entity=mon.0 client_cookie=49c249df3df7c9b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f5887fff700 1 -- 172.21.2.117:0/4070211095 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f588800c0c0 con 0x7f589407da20 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f5887fff700 1 -- 172.21.2.117:0/4070211095 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800c440 con 0x7f589407da20 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f5887fff700 1 -- 172.21.2.117:0/4070211095 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f588800c960 con 0x7f589407da20 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f589b684700 1 -- 172.21.2.117:0/4070211095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589407da20 msgr2=0x7f58940771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.955 7f589b684700 1 --2- 172.21.2.117:0/4070211095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589407da20 0x7f58940771b0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f5888006700 tx=0x7f5888009b80).stop 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 -- 172.21.2.117:0/4070211095 shutdown_connections 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 --2- 172.21.2.117:0/4070211095 >> v2:172.21.2.125:3300/0 conn(0x7f58940e51a0 0x7f5894079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 --2- 172.21.2.117:0/4070211095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589407da20 0x7f58940771b0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 -- 172.21.2.117:0/4070211095 >> 172.21.2.117:0/4070211095 conn(0x7f58940ce450 msgr2=0x7f58940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:36.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 -- 172.21.2.117:0/4070211095 shutdown_connections 2021-02-10T11:35:36.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.956 7f589b684700 1 -- 172.21.2.117:0/4070211095 wait complete. 2021-02-10T11:35:36.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 Processor -- start 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 -- start start 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 0x7f58940b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58940a3800 0x7f58940a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940a4220 con 0x7f58940b9e60 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58940a4410 con 0x7f58940a3800 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.971 7f589b684700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58940ba910 con 0x7f589419a4a0 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5898c9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5898c9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45334/0 (socket says 172.21.2.117:45334) 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5898c9f700 1 -- 172.21.2.117:0/1609729782 learned_addr learned my addr 172.21.2.117:0/1609729782 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 0x7f58940b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5899ca1700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58940a3800 0x7f58940a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 -- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58940a3800 msgr2=0x7f58940a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58940a3800 0x7f58940a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 -- 172.21.2.117:0/1609729782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 msgr2=0x7f58940a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 -- 172.21.2.117:0/1609729782 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940bab30 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5898c9f700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f58994a0700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 0x7f58940b9840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f58880096d0 tx=0x7f588800b8c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.972 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58880009b0 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800bcb0 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58880022b0 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f589b684700 1 -- 172.21.2.117:0/1609729782 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58940bab30 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f589b684700 1 -- 172.21.2.117:0/1609729782 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58940bab30 con 0x7f589419a4a0 2021-02-10T11:35:36.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5888009fd0 con 0x7f589419a4a0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f5885ffb700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 0x7f587401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.973 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f58880325c0 con 0x7f589419a4a0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.974 7f5898c9f700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 0x7f587401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:36.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:36.974 7f5898c9f700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 0x7f587401abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:36.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.004 7f589b684700 1 -- 172.21.2.117:0/1609729782 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58941b4f00 con 0x7f589419a4a0 2021-02-10T11:35:36.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.007 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58880064f0 con 0x7f589419a4a0 2021-02-10T11:35:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.136 7f589b684700 1 -- 172.21.2.117:0/1609729782 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802514", "target": ["mgr", ""]}) v1 -- 0x7f58941b4f00 con 0x7f58740186c0 2021-02-10T11:35:37.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.161 7f5885ffb700 1 -- 172.21.2.117:0/1609729782 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5890000de0 con 0x7f58740186c0 2021-02-10T11:35:37.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 -- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 msgr2=0x7f587401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 0x7f587401abc0 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 -- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 msgr2=0x7f58940b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 0x7f58940b9840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f58880096d0 tx=0x7f588800b8c0).stop 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 -- 172.21.2.117:0/1609729782 shutdown_connections 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58740186c0 0x7f587401abc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58940b9e60 0x7f58940a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58940a3800 0x7f58940a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 --2- 172.21.2.117:0/1609729782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589419a4a0 0x7f58940b9840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.169 7f589b684700 1 -- 172.21.2.117:0/1609729782 >> 172.21.2.117:0/1609729782 conn(0x7f5894107b60 msgr2=0x7f58940c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.170 7f589b684700 1 -- 172.21.2.117:0/1609729782 shutdown_connections 2021-02-10T11:35:37.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.170 7f589b684700 1 -- 172.21.2.117:0/1609729782 wait complete. 2021-02-10T11:35:37.161 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802514 2021-02-10T11:35:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.593 7f46cfa84700 1 Processor -- start 2021-02-10T11:35:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.593 7f46cfa84700 1 -- start start 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cfa84700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46c807da20 0x7f46c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cfa84700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f46c80e4da0 0x7f46c81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cfa84700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c8077a70 con 0x7f46c80e51a0 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cfa84700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c80777d0 con 0x7f46c807da20 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cfa84700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c807de20 con 0x7f46c80e4da0 2021-02-10T11:35:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46c807da20 0x7f46c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46c807da20 0x7f46c80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45342/0 (socket says 172.21.2.117:45342) 2021-02-10T11:35:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 -- 172.21.2.117:0/1101048926 learned_addr learned my addr 172.21.2.117:0/1101048926 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 -- 172.21.2.117:0/1101048926 >> v2:172.21.2.125:3300/0 conn(0x7f46c80e4da0 msgr2=0x7f46c81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 --2- 172.21.2.117:0/1101048926 >> v2:172.21.2.125:3300/0 conn(0x7f46c80e4da0 0x7f46c81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 -- 172.21.2.117:0/1101048926 >> v1:172.21.2.125:6789/0 conn(0x7f46c80e51a0 legacy=0x7f46c8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 -- 172.21.2.117:0/1101048926 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c81b0710 con 0x7f46c807da20 2021-02-10T11:35:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.594 7f46cd8a0700 1 --2- 172.21.2.117:0/1101048926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c807da20 0x7f46c80771b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f46bc006700 tx=0x7f46bc009b80).ready entity=mon.0 client_cookie=a0eae02f349f8cd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cc89e700 1 -- 172.21.2.117:0/1101048926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46bc00c0c0 con 0x7f46c807da20 2021-02-10T11:35:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cc89e700 1 -- 172.21.2.117:0/1101048926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc00c340 con 0x7f46c807da20 2021-02-10T11:35:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cc89e700 1 -- 172.21.2.117:0/1101048926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46bc00c860 con 0x7f46c807da20 2021-02-10T11:35:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 -- 172.21.2.117:0/1101048926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c807da20 msgr2=0x7f46c80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 --2- 172.21.2.117:0/1101048926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c807da20 0x7f46c80771b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f46bc006700 tx=0x7f46bc009b80).stop 2021-02-10T11:35:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 -- 172.21.2.117:0/1101048926 shutdown_connections 2021-02-10T11:35:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 --2- 172.21.2.117:0/1101048926 >> v2:172.21.2.125:3300/0 conn(0x7f46c80e4da0 0x7f46c81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 --2- 172.21.2.117:0/1101048926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c807da20 0x7f46c80771b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 -- 172.21.2.117:0/1101048926 >> 172.21.2.117:0/1101048926 conn(0x7f46c80ce450 msgr2=0x7f46c80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 -- 172.21.2.117:0/1101048926 shutdown_connections 2021-02-10T11:35:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.595 7f46cfa84700 1 -- 172.21.2.117:0/1101048926 wait complete. 2021-02-10T11:35:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.610 7f46cfa84700 1 Processor -- start 2021-02-10T11:35:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.610 7f46cfa84700 1 -- start start 2021-02-10T11:35:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46c8107c20 0x7f46c8096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46c81a7240 0x7f46c81a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8108020 con 0x7f46c808a180 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81a7c60 con 0x7f46c8107c20 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c81a7e80 con 0x7f46c81a7240 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45344/0 (socket says 172.21.2.117:45344) 2021-02-10T11:35:37.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 -- 172.21.2.117:0/4089347435 learned_addr learned my addr 172.21.2.117:0/4089347435 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:37.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 -- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46c81a7240 msgr2=0x7f46c81a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46c81a7240 0x7f46c81a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 -- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46c8107c20 msgr2=0x7f46c8096fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:37.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46c8107c20 0x7f46c8096fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 -- 172.21.2.117:0/4089347435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c808b250 con 0x7f46c808a180 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cd09f700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f46c400b530 tx=0x7f46c400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46c400bcc0 con 0x7f46c808a180 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4000c90 con 0x7f46c808a180 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46c40022a0 con 0x7f46c808a180 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.611 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c808b250 con 0x7f46c808a180 2021-02-10T11:35:37.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.612 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c808b250 con 0x7f46c808a180 2021-02-10T11:35:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.612 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f46c4005310 con 0x7f46c808a180 2021-02-10T11:35:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.612 7f46ba7fc700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 0x7f46b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.612 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f46c40328e0 con 0x7f46c808a180 2021-02-10T11:35:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.613 7f46cd8a0700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 0x7f46b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:37.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.613 7f46cd8a0700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 0x7f46b001ac10 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:37.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.644 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c8077a70 con 0x7f46c808a180 2021-02-10T11:35:37.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.646 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46c4006b90 con 0x7f46c808a180 2021-02-10T11:35:37.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.774 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802514", "target": ["mgr", ""]}) v1 -- 0x7f46c8077a70 con 0x7f46b0018730 2021-02-10T11:35:37.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.777 7f46ba7fc700 1 -- 172.21.2.117:0/4089347435 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f46bc006500 con 0x7f46b0018730 2021-02-10T11:35:37.767 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 msgr2=0x7f46b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 0x7f46b001ac10 crc :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 msgr2=0x7f46c808a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f46c400b530 tx=0x7f46c400b8d0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 shutdown_connections 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.783 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46b0018730 0x7f46b001ac10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46c808a180 0x7f46c808a580 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46c81a7240 0x7f46c81a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 --2- 172.21.2.117:0/4089347435 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46c8107c20 0x7f46c8096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:37.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 >> 172.21.2.117:0/4089347435 conn(0x7f46c80a1880 msgr2=0x7f46c80b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:37.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 shutdown_connections 2021-02-10T11:35:37.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:37.784 7f46cfa84700 1 -- 172.21.2.117:0/4089347435 wait complete. 2021-02-10T11:35:37.775 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802514 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de 2021-02-10T11:35:37.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:37.776 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:37.776 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.113 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:38.113 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:38.113 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247886 s, 42.3 MB/s 2021-02-10T11:35:38.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:38.496 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:38.496 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:38.831 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:38.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:38.832 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:39.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:39.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:39.208 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248102 s, 42.3 MB/s 2021-02-10T11:35:39.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:39.210 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:39.210 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:39.546 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:39.547 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:39.547 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:39.923 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:39.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:39.924 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234109 s, 44.8 MB/s 2021-02-10T11:35:39.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:39.925 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:39.925 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:40.262 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:40.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:40.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:40.640 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:40.640 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:40.641 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0263158 s, 39.8 MB/s 2021-02-10T11:35:40.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:40.642 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:40.642 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:40.978 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:40.978 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:40.978 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:41.353 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:41.353 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:41.353 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0226049 s, 46.4 MB/s 2021-02-10T11:35:41.354 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:41.354 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:41.354 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:41.690 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:41.690 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:41.690 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:42.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:42.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:42.068 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256427 s, 40.9 MB/s 2021-02-10T11:35:42.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:42.069 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:42.069 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:42.405 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:42.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:42.406 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:42.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:42.780 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:42.780 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246064 s, 42.6 MB/s 2021-02-10T11:35:42.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:42.781 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:42.781 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:43.117 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:43.118 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:43.118 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:43.494 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:43.494 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:43.494 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249745 s, 42.0 MB/s 2021-02-10T11:35:43.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:43.495 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:43.495 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:43.831 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:43.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:43.832 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:44.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:44.208 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:44.208 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249694 s, 42.0 MB/s 2021-02-10T11:35:44.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:44.209 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:44.210 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:44.546 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:44.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:44.546 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802514/01db24c5-e7d5-4aae-b0cb-f42b161961de/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:44.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:44.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:44.925 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259847 s, 40.4 MB/s 2021-02-10T11:35:44.926 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802515 2021-02-10T11:35:45.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 Processor -- start 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 -- start start 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fca3c07da20 0x7fca3c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0d1d20 con 0x7fca3c0e4da0 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c077a70 con 0x7fca3c0e51a0 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca440fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c0777d0 con 0x7fca3c07da20 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca41715700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fca3c0e4da0 0x7fca3c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45602/0 (socket says 172.21.2.117:45602) 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.356 7fca41715700 1 -- 172.21.2.117:0/3726910803 learned_addr learned my addr 172.21.2.117:0/3726910803 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443384040 0 0) 0x7fca300010b0 con 0x7fca3c0e4da0 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca28002fe0 con 0x7fca3c0e4da0 2021-02-10T11:35:45.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 567942240 0 0) 0x7fca300010b0 con 0x7fca3c0e4da0 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 >> v2:172.21.2.125:3300/0 conn(0x7fca3c07da20 msgr2=0x7fca3c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 --2- 172.21.2.117:0/3726910803 >> v2:172.21.2.125:3300/0 conn(0x7fca3c07da20 0x7fca3c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 >> v1:172.21.2.125:6789/0 conn(0x7fca3c0e51a0 legacy=0x7fca3c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c1b08e0 con 0x7fca3c0e4da0 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fca300015b0 con 0x7fca3c0e4da0 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 >> v1:172.21.2.117:6789/0 conn(0x7fca3c0e4da0 legacy=0x7fca3c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 0x7fca28006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca28006830 0x7fca28008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca280095b0 0x7fca2800bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2800c320 con 0x7fca28003af0 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2800c540 con 0x7fca28006830 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca40f14700 1 -- 172.21.2.117:0/3726910803 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2800c760 con 0x7fca280095b0 2021-02-10T11:35:45.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca41f16700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 0x7fca28006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca41715700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca28006830 0x7fca28008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.357 7fca42717700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca280095b0 0x7fca2800bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 -- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca280095b0 msgr2=0x7fca2800bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca280095b0 0x7fca2800bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 -- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca28006830 msgr2=0x7fca28008d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca28006830 0x7fca28008d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 -- 172.21.2.117:0/3726910803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2800ca00 con 0x7fca28003af0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca42717700 1 -- 172.21.2.117:0/3726910803 reap_dead start 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca41f16700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 0x7fca28006010 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fca38006bf0 tx=0x7fca3804fcd0).ready entity=mon.0 client_cookie=68ace5c6bb9449a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca380009b0 con 0x7fca28003af0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38002030 con 0x7fca28003af0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca40f14700 1 -- 172.21.2.117:0/3726910803 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca38002510 con 0x7fca28003af0 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca440fa700 1 -- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 msgr2=0x7fca28006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.358 7fca440fa700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 0x7fca28006010 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7fca38006bf0 tx=0x7fca3804fcd0).stop 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.359 7fca440fa700 1 -- 172.21.2.117:0/3726910803 shutdown_connections 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.359 7fca440fa700 1 --2- 172.21.2.117:0/3726910803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca28003af0 0x7fca28006010 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.359 7fca440fa700 1 -- 172.21.2.117:0/3726910803 >> 172.21.2.117:0/3726910803 conn(0x7fca3c0ce450 msgr2=0x7fca3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.359 7fca440fa700 1 -- 172.21.2.117:0/3726910803 shutdown_connections 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.359 7fca440fa700 1 -- 172.21.2.117:0/3726910803 wait complete. 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 Processor -- start 2021-02-10T11:35:45.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 -- start start 2021-02-10T11:35:45.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca3c0a51f0 0x7fca3c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca3c0b77d0 0x7fca3c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c189970 con 0x7fca3c0a47d0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c190aa0 con 0x7fca3c0a51f0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca440fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0b81f0 con 0x7fca3c0b77d0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca41f16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca41f16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45362/0 (socket says 172.21.2.117:45362) 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.374 7fca41f16700 1 -- 172.21.2.117:0/103342144 learned_addr learned my addr 172.21.2.117:0/103342144 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 -- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca3c0b77d0 msgr2=0x7fca3c0b7bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca3c0b77d0 0x7fca3c0b7bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 -- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca3c0a51f0 msgr2=0x7fca3c0a55f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca3c0a51f0 0x7fca3c0a55f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 -- 172.21.2.117:0/103342144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0b8410 con 0x7fca3c0a47d0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca41f16700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fca3804f540 tx=0x7fca380099d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca38000c90 con 0x7fca3c0a47d0 2021-02-10T11:35:45.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38006020 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca380055e0 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca440fa700 1 -- 172.21.2.117:0/103342144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c0b8410 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.375 7fca440fa700 1 -- 172.21.2.117:0/103342144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c0b8410 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.376 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca38000c90 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.376 7fca2e7fc700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 0x7fca2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.376 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fca38076730 con 0x7fca3c0a47d0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.377 7fca41715700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 0x7fca2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.378 7fca41715700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 0x7fca2401abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.406 7fca440fa700 1 -- 172.21.2.117:0/103342144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c0b9000 con 0x7fca3c0a47d0 2021-02-10T11:35:45.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.409 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca38065f20 con 0x7fca3c0a47d0 2021-02-10T11:35:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.536 7fca440fa700 1 -- 172.21.2.117:0/103342144 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802515", "target": ["mgr", ""]}) v1 -- 0x7fca3c0b9000 con 0x7fca240186c0 2021-02-10T11:35:45.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.561 7fca2e7fc700 1 -- 172.21.2.117:0/103342144 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fca300012d0 con 0x7fca240186c0 2021-02-10T11:35:45.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 msgr2=0x7fca2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 0x7fca2401abc0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 msgr2=0x7fca3c0a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fca3804f540 tx=0x7fca380099d0).stop 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 shutdown_connections 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca3c0b77d0 0x7fca3c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca3c0a51f0 0x7fca3c0a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca240186c0 0x7fca2401abc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 --2- 172.21.2.117:0/103342144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca3c0a47d0 0x7fca3c0a4bd0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 >> 172.21.2.117:0/103342144 conn(0x7fca3c0b5140 msgr2=0x7fca3c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 shutdown_connections 2021-02-10T11:35:45.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.569 7fca440fa700 1 -- 172.21.2.117:0/103342144 wait complete. 2021-02-10T11:35:45.561 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802515 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.992 7fd961260700 1 Processor -- start 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.992 7fd961260700 1 -- start start 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd961260700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c07da20 0x7fd95c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd961260700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd95c0e4da0 0x7fd95c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd961260700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c077a70 con 0x7fd95c0e51a0 2021-02-10T11:35:45.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd961260700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0e5c20 con 0x7fd95c0e4da0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd961260700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0777d0 con 0x7fd95c07da20 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd95c0e4da0 0x7fd95c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd95c0e4da0 0x7fd95c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45372/0 (socket says 172.21.2.117:45372) 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 -- 172.21.2.117:0/2540210974 learned_addr learned my addr 172.21.2.117:0/2540210974 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 -- 172.21.2.117:0/2540210974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c07da20 msgr2=0x7fd95c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 --2- 172.21.2.117:0/2540210974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c07da20 0x7fd95c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 -- 172.21.2.117:0/2540210974 >> v1:172.21.2.125:6789/0 conn(0x7fd95c0e51a0 legacy=0x7fd95c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 -- 172.21.2.117:0/2540210974 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c07de20 con 0x7fd95c0e4da0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.993 7fd95a59c700 1 --2- 172.21.2.117:0/2540210974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0e4da0 0x7fd95c1b00f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fd944006700 tx=0x7fd944009b80).ready entity=mon.0 client_cookie=74c1233594da4df8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd959d9b700 1 -- 172.21.2.117:0/2540210974 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94400c0c0 con 0x7fd95c0e4da0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd959d9b700 1 -- 172.21.2.117:0/2540210974 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94400c340 con 0x7fd95c0e4da0 2021-02-10T11:35:45.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd959d9b700 1 -- 172.21.2.117:0/2540210974 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94400c860 con 0x7fd95c0e4da0 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 -- 172.21.2.117:0/2540210974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0e4da0 msgr2=0x7fd95c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 --2- 172.21.2.117:0/2540210974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0e4da0 0x7fd95c1b00f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7fd944006700 tx=0x7fd944009b80).stop 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 -- 172.21.2.117:0/2540210974 shutdown_connections 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 --2- 172.21.2.117:0/2540210974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0e4da0 0x7fd95c1b00f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 --2- 172.21.2.117:0/2540210974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c07da20 0x7fd95c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 -- 172.21.2.117:0/2540210974 >> 172.21.2.117:0/2540210974 conn(0x7fd95c0ce450 msgr2=0x7fd95c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 -- 172.21.2.117:0/2540210974 shutdown_connections 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:45.994 7fd961260700 1 -- 172.21.2.117:0/2540210974 wait complete. 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 Processor -- start 2021-02-10T11:35:45.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 -- start start 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd95c0b1340 0x7fd95c18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c18b830 0x7fd95c187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0b18c0 con 0x7fd95c0b0f40 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c18ac90 con 0x7fd95c0b1340 2021-02-10T11:35:45.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd961260700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c107f50 con 0x7fd95c18b830 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd95ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd95ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45376/0 (socket says 172.21.2.117:45376) 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.009 7fd95ad9d700 1 -- 172.21.2.117:0/4072865853 learned_addr learned my addr 172.21.2.117:0/4072865853 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95b59e700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c18b830 0x7fd95c187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95a59c700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd95c0b1340 0x7fd95c18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 -- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c18b830 msgr2=0x7fd95c187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c18b830 0x7fd95c187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 -- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd95c0b1340 msgr2=0x7fd95c18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd95c0b1340 0x7fd95c18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 -- 172.21.2.117:0/4072865853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c108130 con 0x7fd95c0b0f40 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd95ad9d700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fd94c00b5c0 tx=0x7fd94c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:46.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c00bd50 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c006030 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd94c0054d0 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd961260700 1 -- 172.21.2.117:0/4072865853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c108130 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.010 7fd961260700 1 -- 172.21.2.117:0/4072865853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c108130 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.011 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd94c006210 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.011 7fd9537fe700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 0x7fd94001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.011 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd94c0326f0 con 0x7fd95c0b0f40 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.012 7fd95a59c700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 0x7fd94001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:46.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.012 7fd95a59c700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 0x7fd94001abc0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:46.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.043 7fd961260700 1 -- 172.21.2.117:0/4072865853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c077a70 con 0x7fd95c0b0f40 2021-02-10T11:35:46.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.045 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd94c025fd0 con 0x7fd95c0b0f40 2021-02-10T11:35:46.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.173 7fd961260700 1 -- 172.21.2.117:0/4072865853 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802515", "target": ["mgr", ""]}) v1 -- 0x7fd95c077a70 con 0x7fd9400186c0 2021-02-10T11:35:46.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.176 7fd9537fe700 1 -- 172.21.2.117:0/4072865853 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd9440090b0 con 0x7fd9400186c0 2021-02-10T11:35:46.164 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2 2021-02-10T11:35:46.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.182 7fd961260700 1 -- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 msgr2=0x7fd94001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.182 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 0x7fd94001abc0 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.182 7fd961260700 1 -- 172.21.2.117:0/4072865853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 msgr2=0x7fd95c107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:46.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.182 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7fd94c00b5c0 tx=0x7fd94c00b960).stop 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 -- 172.21.2.117:0/4072865853 shutdown_connections 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd95c18b830 0x7fd95c187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd95c0b1340 0x7fd95c18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9400186c0 0x7fd94001abc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 --2- 172.21.2.117:0/4072865853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd95c0b0f40 0x7fd95c107930 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 -- 172.21.2.117:0/4072865853 >> 172.21.2.117:0/4072865853 conn(0x7fd95c0a1880 msgr2=0x7fd95c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 -- 172.21.2.117:0/4072865853 shutdown_connections 2021-02-10T11:35:46.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:46.183 7fd961260700 1 -- 172.21.2.117:0/4072865853 wait complete. 2021-02-10T11:35:46.174 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802515 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2 2021-02-10T11:35:46.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:46.175 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:46.175 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:46.538 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:46.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:46.539 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:46.921 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:46.921 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:46.922 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238347 s, 44.0 MB/s 2021-02-10T11:35:46.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:46.923 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:46.923 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:47.258 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:47.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:47.259 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:47.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:47.631 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:47.631 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240626 s, 43.6 MB/s 2021-02-10T11:35:47.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:47.632 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:47.633 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:47.968 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:47.969 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:47.969 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:48.346 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235908 s, 44.4 MB/s 2021-02-10T11:35:48.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:48.347 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:48.347 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:48.683 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:48.684 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:48.684 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:49.061 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:49.062 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:49.062 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0264249 s, 39.7 MB/s 2021-02-10T11:35:49.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:49.063 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:49.063 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:49.398 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:49.399 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:49.399 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:49.778 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:49.779 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:49.779 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254583 s, 41.2 MB/s 2021-02-10T11:35:49.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:49.779 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:49.779 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:50.115 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:50.115 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:50.116 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:50.491 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:50.491 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:50.491 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232814 s, 45.0 MB/s 2021-02-10T11:35:50.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:50.492 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:50.492 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:50.827 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:50.828 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:50.828 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:51.204 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:51.205 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:51.205 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255231 s, 41.1 MB/s 2021-02-10T11:35:51.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:51.206 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:51.206 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:51.542 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:51.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:51.543 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:51.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:51.918 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:51.918 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230432 s, 45.5 MB/s 2021-02-10T11:35:51.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:51.919 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:51.919 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:52.256 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:52.256 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:52.256 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:52.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:52.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:52.628 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230713 s, 45.4 MB/s 2021-02-10T11:35:52.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:52.629 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:52.629 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:52.964 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:52.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:52.964 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802515/af01afd9-62ce-40b1-b278-2ed3efc3f0b2/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:53.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:53.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:53.341 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246672 s, 42.5 MB/s 2021-02-10T11:35:53.341 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802516 2021-02-10T11:35:53.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.773 7f229f646700 1 Processor -- start 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.773 7f229f646700 1 -- start start 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229f646700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f22980d1530 0x7f22980d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229f646700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298077a70 con 0x7f2298079000 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229f646700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298079400 con 0x7f229807da20 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229f646700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f229807de20 con 0x7f22980d1530 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229cc61700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2298079000 0x7f22980e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45634/0 (socket says 172.21.2.117:45634) 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229cc61700 1 -- 172.21.2.117:0/4248380321 learned_addr learned my addr 172.21.2.117:0/4248380321 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098325640 0 0) 0x7f22940010b0 con 0x7f2298079000 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f228bfff700 1 -- 172.21.2.117:0/4248380321 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2284002fe0 con 0x7f2298079000 2021-02-10T11:35:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.774 7f229dc63700 1 --2- 172.21.2.117:0/4248380321 >> v2:172.21.2.125:3300/0 conn(0x7f22980d1530 0x7f22980d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 91074100 0 0) 0x7f22940010b0 con 0x7f2298079000 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 >> v2:172.21.2.125:3300/0 conn(0x7f22980d1530 msgr2=0x7f22980d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 --2- 172.21.2.117:0/4248380321 >> v2:172.21.2.125:3300/0 conn(0x7f22980d1530 0x7f22980d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 >> v1:172.21.2.125:6789/0 conn(0x7f229807da20 legacy=0x7f22980771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22981b00f0 con 0x7f2298079000 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f22940015b0 con 0x7f2298079000 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 >> v1:172.21.2.117:6789/0 conn(0x7f2298079000 legacy=0x7f22980e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2284003a70 0x7f2284005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 0x7f2284008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22840095c0 0x7f228400bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228400c330 con 0x7f2284006810 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228400c550 con 0x7f2284003a70 2021-02-10T11:35:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f228bfff700 1 -- 172.21.2.117:0/4248380321 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228400c770 con 0x7f22840095c0 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 0x7f2284008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229d462700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2284003a70 0x7f2284005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229dc63700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22840095c0 0x7f228400bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 -- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22840095c0 msgr2=0x7f228400bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22840095c0 0x7f228400bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 -- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2284003a70 msgr2=0x7f2284005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2284003a70 0x7f2284005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229dc63700 1 -- 172.21.2.117:0/4248380321 reap_dead start 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 -- 172.21.2.117:0/4248380321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840509b0 con 0x7f2284006810 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.775 7f229cc61700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 0x7f2284008cc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f2294007560 tx=0x7f2294007c60).ready entity=mon.0 client_cookie=3c80ebeef6004172 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22940009b0 con 0x7f2284006810 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229400a310 con 0x7f2284006810 2021-02-10T11:35:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f228bfff700 1 -- 172.21.2.117:0/4248380321 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f229400a830 con 0x7f2284006810 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 -- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 msgr2=0x7f2284008cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 0x7f2284008cc0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7f2294007560 tx=0x7f2294007c60).stop 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 -- 172.21.2.117:0/4248380321 shutdown_connections 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 --2- 172.21.2.117:0/4248380321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2284006810 0x7f2284008cc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 -- 172.21.2.117:0/4248380321 >> 172.21.2.117:0/4248380321 conn(0x7f22980ce450 msgr2=0x7f22980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 -- 172.21.2.117:0/4248380321 shutdown_connections 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.776 7f229f646700 1 -- 172.21.2.117:0/4248380321 wait complete. 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.791 7f229f646700 1 Processor -- start 2021-02-10T11:35:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.791 7f229f646700 1 -- start start 2021-02-10T11:35:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22980a51f0 0x7f22980a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22980b77d0 0x7f22980b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298189970 con 0x7f22980a47d0 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2298190aa0 con 0x7f22980a51f0 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229f646700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22980b81f0 con 0x7f22980b77d0 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45396/0 (socket says 172.21.2.117:45396) 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 -- 172.21.2.117:0/2553314735 learned_addr learned my addr 172.21.2.117:0/2553314735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229dc63700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22980b77d0 0x7f22980b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 -- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22980b77d0 msgr2=0x7f22980b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22980b77d0 0x7f22980b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 -- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22980a51f0 msgr2=0x7f22980a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22980a51f0 0x7f22980a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:53.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 -- 172.21.2.117:0/2553314735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980b8410 con 0x7f22980a47d0 2021-02-10T11:35:53.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.792 7f229d462700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f228c00b560 tx=0x7f228c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c006b00 con 0x7f22980a47d0 2021-02-10T11:35:53.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c002030 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c002550 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f229f646700 1 -- 172.21.2.117:0/2553314735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22980b8410 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f229f646700 1 -- 172.21.2.117:0/2553314735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22980b89a0 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f228c006b00 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 0x7f227801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.793 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f228c0326c0 con 0x7f22980a47d0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.794 7f229cc61700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 0x7f227801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:53.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.794 7f229cc61700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 0x7f227801abc0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:53.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.824 7f229f646700 1 -- 172.21.2.117:0/2553314735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2298077a70 con 0x7f22980a47d0 2021-02-10T11:35:53.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.826 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f228c02eef0 con 0x7f22980a47d0 2021-02-10T11:35:53.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.951 7f229f646700 1 -- 172.21.2.117:0/2553314735 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802516", "target": ["mgr", ""]}) v1 -- 0x7f22980b81f0 con 0x7f22780186c0 2021-02-10T11:35:53.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.977 7f2289ffb700 1 -- 172.21.2.117:0/2553314735 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2294007520 con 0x7f22780186c0 2021-02-10T11:35:53.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 msgr2=0x7f227801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 0x7f227801abc0 crc :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 msgr2=0x7f22980a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:53.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f228c00b560 tx=0x7f228c00b900).stop 2021-02-10T11:35:53.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 shutdown_connections 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22980b77d0 0x7f22980b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22980a51f0 0x7f22980a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22780186c0 0x7f227801abc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 --2- 172.21.2.117:0/2553314735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22980a47d0 0x7f22980a4bd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 >> 172.21.2.117:0/2553314735 conn(0x7f22980b5140 msgr2=0x7f22980a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 shutdown_connections 2021-02-10T11:35:53.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:53.984 7f229f646700 1 -- 172.21.2.117:0/2553314735 wait complete. 2021-02-10T11:35:53.976 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802516 2021-02-10T11:35:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 Processor -- start 2021-02-10T11:35:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 -- start start 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb5407da20 0x7fbb540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbb540e5180 0x7fbb540d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb54077a70 con 0x7fbb540e4d80 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb540e5c00 con 0x7fbb540e5180 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb5b379700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540777d0 con 0x7fbb5407da20 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb59996700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbb540e5180 0x7fbb540d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.409 7fbb58994700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbb540e4d80 0x7fbb541b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45656/0 (socket says 172.21.2.117:45656) 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb58994700 1 -- 172.21.2.117:0/3763010418 learned_addr learned my addr 172.21.2.117:0/3763010418 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59195700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb5407da20 0x7fbb540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59996700 1 -- 172.21.2.117:0/3763010418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb5407da20 msgr2=0x7fbb540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59996700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb5407da20 0x7fbb540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59996700 1 -- 172.21.2.117:0/3763010418 >> v1:172.21.2.117:6789/0 conn(0x7fbb540e4d80 legacy=0x7fbb541b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59996700 1 -- 172.21.2.117:0/3763010418 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb541b08c0 con 0x7fbb540e5180 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb59996700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb540e5180 0x7fbb540d0500 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7fbb50006ab0 tx=0x7fbb50009b40).ready entity=mon.0 client_cookie=9e2ad56923991ac8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb4bfff700 1 -- 172.21.2.117:0/3763010418 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb500022e0 con 0x7fbb540e5180 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb4bfff700 1 -- 172.21.2.117:0/3763010418 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb50002560 con 0x7fbb540e5180 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb4bfff700 1 -- 172.21.2.117:0/3763010418 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb500029f0 con 0x7fbb540e5180 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb5b379700 1 -- 172.21.2.117:0/3763010418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb540e5180 msgr2=0x7fbb540d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.410 7fbb5b379700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb540e5180 0x7fbb540d0500 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7fbb50006ab0 tx=0x7fbb50009b40).stop 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 -- 172.21.2.117:0/3763010418 shutdown_connections 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb540e5180 0x7fbb540d0500 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 --2- 172.21.2.117:0/3763010418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb5407da20 0x7fbb540771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 -- 172.21.2.117:0/3763010418 >> 172.21.2.117:0/3763010418 conn(0x7fbb540ce430 msgr2=0x7fbb540e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 -- 172.21.2.117:0/3763010418 shutdown_connections 2021-02-10T11:35:54.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.411 7fbb5b379700 1 -- 172.21.2.117:0/3763010418 wait complete. 2021-02-10T11:35:54.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 Processor -- start 2021-02-10T11:35:54.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 -- start start 2021-02-10T11:35:54.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb54091a50 0x7fbb540b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbb54199750 0x7fbb540ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54091ea0 con 0x7fbb54187bd0 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54187fd0 con 0x7fbb54199750 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.426 7fbb5b379700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541881f0 con 0x7fbb54091a50 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45410/0 (socket says 172.21.2.117:45410) 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 -- 172.21.2.117:0/2108993932 learned_addr learned my addr 172.21.2.117:0/2108993932 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb59195700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb54091a50 0x7fbb540b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 -- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb54091a50 msgr2=0x7fbb540b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb54091a50 0x7fbb540b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 -- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbb54199750 msgr2=0x7fbb540ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbb54199750 0x7fbb540ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 -- 172.21.2.117:0/2108993932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb540abee0 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb58994700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fbb44001c60 tx=0x7fbb4400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb4400acc0 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb44004140 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbb44004660 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb540abee0 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.427 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb540abee0 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.428 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbb44000d90 con 0x7fbb54187bd0 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.428 7fbb49ffb700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 0x7fbb3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:35:54.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.428 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbb44032520 con 0x7fbb54187bd0 2021-02-10T11:35:54.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.428 7fbb59195700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 0x7fbb3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:35:54.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.429 7fbb59195700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 0x7fbb3401abc0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:35:54.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.459 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb54077a70 con 0x7fbb54187bd0 2021-02-10T11:35:54.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.461 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbb440009b0 con 0x7fbb54187bd0 2021-02-10T11:35:54.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.589 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802516", "target": ["mgr", ""]}) v1 -- 0x7fbb54187fd0 con 0x7fbb340186c0 2021-02-10T11:35:54.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.592 7fbb49ffb700 1 -- 172.21.2.117:0/2108993932 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fbb4c000cc0 con 0x7fbb340186c0 2021-02-10T11:35:54.576 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203 2021-02-10T11:35:54.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 msgr2=0x7fbb3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 0x7fbb3401abc0 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 msgr2=0x7fbb54199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fbb44001c60 tx=0x7fbb4400a8d0).stop 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 shutdown_connections 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbb340186c0 0x7fbb3401abc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbb54187bd0 0x7fbb54199130 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbb54199750 0x7fbb540ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:35:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 --2- 172.21.2.117:0/2108993932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbb54091a50 0x7fbb540b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:35:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 >> 172.21.2.117:0/2108993932 conn(0x7fbb540b5320 msgr2=0x7fbb541b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:35:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 shutdown_connections 2021-02-10T11:35:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:35:54.594 7fbb5b379700 1 -- 172.21.2.117:0/2108993932 wait complete. 2021-02-10T11:35:54.586 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802516 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203 2021-02-10T11:35:54.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:54.586 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:54.586 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:54.922 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:54.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:54.923 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254945 s, 41.1 MB/s 2021-02-10T11:35:55.307 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:55.307 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:55.307 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:55.642 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:55.643 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:55.643 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:56.021 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:56.021 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:56.021 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249547 s, 42.0 MB/s 2021-02-10T11:35:56.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:56.022 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:56.022 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:56.359 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:56.359 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:56.360 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:56.737 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:56.737 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:56.737 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249879 s, 42.0 MB/s 2021-02-10T11:35:56.738 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:56.739 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:56.739 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:57.075 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:57.075 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:57.075 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230062 s, 45.6 MB/s 2021-02-10T11:35:57.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:57.448 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:57.448 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:57.784 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:57.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:57.785 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:58.160 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:58.160 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:58.160 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221809 s, 47.3 MB/s 2021-02-10T11:35:58.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:58.162 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:58.163 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:58.499 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:58.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:58.499 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:58.875 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:58.875 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:58.875 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237078 s, 44.2 MB/s 2021-02-10T11:35:58.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:58.877 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:58.877 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:59.211 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:59.212 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:59.212 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:35:59.585 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:35:59.586 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:35:59.586 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230804 s, 45.4 MB/s 2021-02-10T11:35:59.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:35:59.587 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:35:59.587 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:35:59.923 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:35:59.924 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:35:59.924 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231116 s, 45.4 MB/s 2021-02-10T11:36:00.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:00.303 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:00.303 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:00.639 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:00.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:00.640 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:01.013 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:01.013 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:01.014 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0222287 s, 47.2 MB/s 2021-02-10T11:36:01.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:01.015 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:01.015 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:01.350 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:01.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:01.351 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802516/1f5665ac-e856-432e-9d7e-549a1cefa203/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:01.729 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:01.729 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:01.729 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250634 s, 41.8 MB/s 2021-02-10T11:36:01.730 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802517 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 Processor -- start 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 -- start start 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e24077a70 con 0x7f9e240e4da0 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e240777d0 con 0x7f9e2407da20 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.162 7f9e294d9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e241b08e0 con 0x7f9e240e51a0 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e232bf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43098/0 (socket says 172.21.2.117:43098) 2021-02-10T11:36:02.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 -- 172.21.2.117:0/2994703747 learned_addr learned my addr 172.21.2.117:0/2994703747 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e232bf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:49802/0 (socket says 172.21.2.117:49802) 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e22abe700 1 --1- 172.21.2.117:0/2994703747 >> v1:172.21.2.125:6789/0 conn(0x7f9e240e4da0 0x7f9e241b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38594/0 (socket says 172.21.2.117:38594) 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 -- 172.21.2.117:0/2994703747 >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 msgr2=0x7f9e240771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 --2- 172.21.2.117:0/2994703747 >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 -- 172.21.2.117:0/2994703747 >> v1:172.21.2.125:6789/0 conn(0x7f9e240e4da0 legacy=0x7f9e241b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e222bd700 1 -- 172.21.2.117:0/2994703747 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234504453 0 0) 0x7f9e140010b0 con 0x7f9e240e4da0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 -- 172.21.2.117:0/2994703747 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2407de20 con 0x7f9e240e51a0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e232bf700 1 --2- 172.21.2.117:0/2994703747 >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.164 7f9e23ac0700 1 --2- 172.21.2.117:0/2994703747 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f9e1c006aa0 tx=0x7f9e1c009f20).ready entity=mon.2 client_cookie=90f5528670270e9d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e222bd700 1 -- 172.21.2.117:0/2994703747 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c00b5c0 con 0x7f9e240e51a0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e222bd700 1 -- 172.21.2.117:0/2994703747 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c003030 con 0x7f9e240e51a0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e222bd700 1 -- 172.21.2.117:0/2994703747 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e1c00b9f0 con 0x7f9e240e51a0 2021-02-10T11:36:02.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 -- 172.21.2.117:0/2994703747 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 msgr2=0x7f9e24079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 --2- 172.21.2.117:0/2994703747 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f9e1c006aa0 tx=0x7f9e1c009f20).stop 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 -- 172.21.2.117:0/2994703747 shutdown_connections 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 --2- 172.21.2.117:0/2994703747 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e240e51a0 0x7f9e24079000 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 --2- 172.21.2.117:0/2994703747 >> v2:172.21.2.125:3300/0 conn(0x7f9e2407da20 0x7f9e240771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 -- 172.21.2.117:0/2994703747 >> 172.21.2.117:0/2994703747 conn(0x7f9e240ce450 msgr2=0x7f9e240e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 -- 172.21.2.117:0/2994703747 shutdown_connections 2021-02-10T11:36:02.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.165 7f9e294d9700 1 -- 172.21.2.117:0/2994703747 wait complete. 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 Processor -- start 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 -- start start 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e241048f0 0x7f9e24104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e24090710 0x7f9e24090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e241054f0 con 0x7f9e240b7fc0 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24195a30 con 0x7f9e24090710 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e294d9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24195fb0 con 0x7f9e241048f0 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e232bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e232bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45424/0 (socket says 172.21.2.117:45424) 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.180 7f9e232bf700 1 -- 172.21.2.117:0/2928721677 learned_addr learned my addr 172.21.2.117:0/2928721677 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:02.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 -- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e241048f0 msgr2=0x7f9e24104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e241048f0 0x7f9e24104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 -- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e24090710 msgr2=0x7f9e24090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e24090710 0x7f9e24090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 -- 172.21.2.117:0/2928721677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240b7b60 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e232bf700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f9e0c006f40 tx=0x7f9e0c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e0c00bd80 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0c000c90 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e0c002220 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e240b7b60 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.181 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e24091420 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.182 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e0c00a000 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.182 7f9e1bfff700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 0x7f9e0801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.182 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e0c032790 con 0x7f9e240b7fc0 2021-02-10T11:36:02.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.183 7f9e22abe700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 0x7f9e0801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.183 7f9e22abe700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 0x7f9e0801ac10 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.212 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24103280 con 0x7f9e240b7fc0 2021-02-10T11:36:02.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.215 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e0c000940 con 0x7f9e240b7fc0 2021-02-10T11:36:02.279 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:02.282 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:02.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.342 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802517", "target": ["mgr", ""]}) v1 -- 0x7f9e24091420 con 0x7f9e08018730 2021-02-10T11:36:02.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.368 7f9e1bfff700 1 -- 172.21.2.117:0/2928721677 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9e14000940 con 0x7f9e08018730 2021-02-10T11:36:02.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 msgr2=0x7f9e0801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 0x7f9e0801ac10 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 msgr2=0x7f9e241042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f9e0c006f40 tx=0x7f9e0c00b990).stop 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 shutdown_connections 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e24090710 0x7f9e24090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e241048f0 0x7f9e24104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e08018730 0x7f9e0801ac10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 --2- 172.21.2.117:0/2928721677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e240b7fc0 0x7f9e241042d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 >> 172.21.2.117:0/2928721677 conn(0x7f9e240a1850 msgr2=0x7f9e240c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 shutdown_connections 2021-02-10T11:36:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.376 7f9e294d9700 1 -- 172.21.2.117:0/2928721677 wait complete. 2021-02-10T11:36:02.368 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802517 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 Processor -- start 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 -- start start 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7007da20 0x7f1e700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1e700e51a0 0x7f1e70079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e70077a70 con 0x7f1e700e4da0 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e700e5c20 con 0x7f1e7007da20 2021-02-10T11:36:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e74806700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e700777d0 con 0x7f1e700e51a0 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e6e61c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7007da20 0x7f1e700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e6e61c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7007da20 0x7f1e700771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45430/0 (socket says 172.21.2.117:45430) 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.714 7f1e6e61c700 1 -- 172.21.2.117:0/2131765918 learned_addr learned my addr 172.21.2.117:0/2131765918 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6e61c700 1 -- 172.21.2.117:0/2131765918 >> v2:172.21.2.125:3300/0 conn(0x7f1e700e51a0 msgr2=0x7f1e70079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6e61c700 1 --2- 172.21.2.117:0/2131765918 >> v2:172.21.2.125:3300/0 conn(0x7f1e700e51a0 0x7f1e70079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6e61c700 1 -- 172.21.2.117:0/2131765918 >> v1:172.21.2.125:6789/0 conn(0x7f1e700e4da0 legacy=0x7f1e701b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6e61c700 1 -- 172.21.2.117:0/2131765918 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e701b08e0 con 0x7f1e7007da20 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6e61c700 1 --2- 172.21.2.117:0/2131765918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7007da20 0x7f1e700771b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f1e60006700 tx=0x7f1e60009b80).ready entity=mon.0 client_cookie=291c8499fa19cffc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6d61a700 1 -- 172.21.2.117:0/2131765918 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6000c0c0 con 0x7f1e7007da20 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6d61a700 1 -- 172.21.2.117:0/2131765918 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6000c340 con 0x7f1e7007da20 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e6d61a700 1 -- 172.21.2.117:0/2131765918 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6000c860 con 0x7f1e7007da20 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e74806700 1 -- 172.21.2.117:0/2131765918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7007da20 msgr2=0x7f1e700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.715 7f1e74806700 1 --2- 172.21.2.117:0/2131765918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7007da20 0x7f1e700771b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f1e60006700 tx=0x7f1e60009b80).stop 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 -- 172.21.2.117:0/2131765918 shutdown_connections 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 --2- 172.21.2.117:0/2131765918 >> v2:172.21.2.125:3300/0 conn(0x7f1e700e51a0 0x7f1e70079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 --2- 172.21.2.117:0/2131765918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7007da20 0x7f1e700771b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 -- 172.21.2.117:0/2131765918 >> 172.21.2.117:0/2131765918 conn(0x7f1e700ce450 msgr2=0x7f1e700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 -- 172.21.2.117:0/2131765918 shutdown_connections 2021-02-10T11:36:02.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.716 7f1e74806700 1 -- 172.21.2.117:0/2131765918 wait complete. 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 Processor -- start 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 -- start start 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 0x7f1e7018b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 0x7f1e70187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e700b18c0 con 0x7f1e700b0f40 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7018ac90 con 0x7f1e7018b830 2021-02-10T11:36:02.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e74806700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e70107f50 con 0x7f1e700b1340 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.731 7f1e6e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45440/0 (socket says 172.21.2.117:45440) 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 -- 172.21.2.117:0/3728689327 learned_addr learned my addr 172.21.2.117:0/3728689327 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6de1b700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 0x7f1e7018b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 -- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 msgr2=0x7f1e7018b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 0x7f1e7018b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 -- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 msgr2=0x7f1e70187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6ee1d700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 0x7f1e70187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 0x7f1e70187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 -- 172.21.2.117:0/3728689327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e70108130 con 0x7f1e700b0f40 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6de1b700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 0x7f1e7018b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6ee1d700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 0x7f1e70187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:36:02.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e6e61c700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f1e6000b5e0 tx=0x7f1e6000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6000bd70 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60002030 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.732 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e60002550 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.733 7f1e74806700 1 -- 172.21.2.117:0/3728689327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e70108130 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.733 7f1e74806700 1 -- 172.21.2.117:0/3728689327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e70108130 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.733 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e60000c90 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.733 7f1e66ffd700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 0x7f1e5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.733 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1e60032980 con 0x7f1e700b0f40 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.734 7f1e6de1b700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 0x7f1e5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:02.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.734 7f1e6de1b700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 0x7f1e5401abc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.764 7f1e74806700 1 -- 172.21.2.117:0/3728689327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e70077a70 con 0x7f1e700b0f40 2021-02-10T11:36:02.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.766 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e6001f570 con 0x7f1e700b0f40 2021-02-10T11:36:02.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.894 7f1e74806700 1 -- 172.21.2.117:0/3728689327 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802517", "target": ["mgr", ""]}) v1 -- 0x7f1e70077a70 con 0x7f1e540186c0 2021-02-10T11:36:02.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.897 7f1e66ffd700 1 -- 172.21.2.117:0/3728689327 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1e58000960 con 0x7f1e540186c0 2021-02-10T11:36:02.885 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd 2021-02-10T11:36:02.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 msgr2=0x7f1e5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 0x7f1e5401abc0 crc :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 msgr2=0x7f1e70107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f1e6000b5e0 tx=0x7f1e6000b980).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 shutdown_connections 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7018b830 0x7f1e70187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e700b1340 0x7f1e7018b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e540186c0 0x7f1e5401abc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 --2- 172.21.2.117:0/3728689327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e700b0f40 0x7f1e70107930 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 >> 172.21.2.117:0/3728689327 conn(0x7f1e700a1880 msgr2=0x7f1e700b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 shutdown_connections 2021-02-10T11:36:02.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:02.903 7f1e74806700 1 -- 172.21.2.117:0/3728689327 wait complete. 2021-02-10T11:36:02.895 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802517 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd 2021-02-10T11:36:02.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:02.895 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:02.895 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:03.231 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:03.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:03.231 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:03.610 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:03.610 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:03.610 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231875 s, 45.2 MB/s 2021-02-10T11:36:03.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:03.611 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:03.611 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:03.948 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:03.948 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:03.949 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:04.326 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:04.326 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:04.326 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247901 s, 42.3 MB/s 2021-02-10T11:36:04.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:04.327 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:04.327 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:04.662 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:04.662 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:04.663 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:05.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:05.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:05.039 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231456 s, 45.3 MB/s 2021-02-10T11:36:05.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:05.040 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:05.040 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:05.376 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:05.376 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:05.377 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:05.756 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:05.756 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:05.757 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255125 s, 41.1 MB/s 2021-02-10T11:36:05.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:05.758 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:05.758 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:06.094 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:06.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:06.095 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:06.470 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:06.471 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:06.471 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023829 s, 44.0 MB/s 2021-02-10T11:36:06.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:06.472 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:06.472 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:06.807 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:06.807 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:06.807 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:07.182 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:07.182 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:07.182 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245187 s, 42.8 MB/s 2021-02-10T11:36:07.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:07.183 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:07.184 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:07.519 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:07.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:07.520 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:07.893 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:07.893 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:07.893 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246442 s, 42.5 MB/s 2021-02-10T11:36:07.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:07.894 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:07.894 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:08.231 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:08.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:08.232 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:08.606 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:08.607 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:08.607 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233119 s, 45.0 MB/s 2021-02-10T11:36:08.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:08.608 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:08.608 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:08.944 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:08.944 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:08.945 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:09.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:09.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:09.322 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254792 s, 41.2 MB/s 2021-02-10T11:36:09.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:09.323 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:09.324 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:09.659 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:09.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:09.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802517/929a22f3-528a-4877-811a-8f2dbdf8aacd/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251859 s, 41.6 MB/s 2021-02-10T11:36:10.037 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802518 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.470 7eff755aa700 1 Processor -- start 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.470 7eff755aa700 1 -- start start 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff755aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7007da20 0x7eff700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff755aa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff700e4da0 0x7eff701b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff755aa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff70077a70 con 0x7eff700e51a0 2021-02-10T11:36:10.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff755aa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff700e5c20 con 0x7eff700e4da0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff755aa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff700777d0 con 0x7eff7007da20 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff700e4da0 0x7eff701b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff700e4da0 0x7eff701b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45444/0 (socket says 172.21.2.117:45444) 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 -- 172.21.2.117:0/2265297764 learned_addr learned my addr 172.21.2.117:0/2265297764 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6f2bf700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7007da20 0x7eff700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 -- 172.21.2.117:0/2265297764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7007da20 msgr2=0x7eff700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7007da20 0x7eff700771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 -- 172.21.2.117:0/2265297764 >> v1:172.21.2.125:6789/0 conn(0x7eff700e51a0 legacy=0x7eff70077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 -- 172.21.2.117:0/2265297764 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7007de20 con 0x7eff700e4da0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.471 7eff6eabe700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700e4da0 0x7eff701b00f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7eff60006700 tx=0x7eff60009b80).ready entity=mon.0 client_cookie=e05e6cad2ceba94a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff6e2bd700 1 -- 172.21.2.117:0/2265297764 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff60000c90 con 0x7eff700e4da0 2021-02-10T11:36:10.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff6e2bd700 1 -- 172.21.2.117:0/2265297764 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6000c0c0 con 0x7eff700e4da0 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff6e2bd700 1 -- 172.21.2.117:0/2265297764 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6000c5e0 con 0x7eff700e4da0 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 -- 172.21.2.117:0/2265297764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700e4da0 msgr2=0x7eff701b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700e4da0 0x7eff701b00f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7eff60006700 tx=0x7eff60009b80).stop 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 -- 172.21.2.117:0/2265297764 shutdown_connections 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700e4da0 0x7eff701b00f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 --2- 172.21.2.117:0/2265297764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7007da20 0x7eff700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 -- 172.21.2.117:0/2265297764 >> 172.21.2.117:0/2265297764 conn(0x7eff700ce450 msgr2=0x7eff700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 -- 172.21.2.117:0/2265297764 shutdown_connections 2021-02-10T11:36:10.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.472 7eff755aa700 1 -- 172.21.2.117:0/2265297764 wait complete. 2021-02-10T11:36:10.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.487 7eff755aa700 1 Processor -- start 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.487 7eff755aa700 1 -- start start 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7019a4a0 0x7eff700b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff700b9e60 0x7eff700a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700a4220 con 0x7eff700a3800 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700a4410 con 0x7eff700b9e60 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff700ba910 con 0x7eff7019a4a0 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45450/0 (socket says 172.21.2.117:45450) 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 -- 172.21.2.117:0/550687468 learned_addr learned my addr 172.21.2.117:0/550687468 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:10.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 -- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7019a4a0 msgr2=0x7eff700b9840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7019a4a0 0x7eff700b9840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 -- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff700b9e60 msgr2=0x7eff700a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff700b9e60 0x7eff700a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 -- 172.21.2.117:0/550687468 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700bab30 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff6fac0700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7eff68009940 tx=0x7eff6800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff68000ce0 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff68000f90 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff6800c570 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 -- 172.21.2.117:0/550687468 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff700bab30 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.488 7eff755aa700 1 -- 172.21.2.117:0/550687468 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff700bb0c0 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.489 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff68001220 con 0x7eff700a3800 2021-02-10T11:36:10.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.489 7eff67fff700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 0x7eff5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:10.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.489 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7eff68032a50 con 0x7eff700a3800 2021-02-10T11:36:10.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.490 7eff6f2bf700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 0x7eff5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:10.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.490 7eff6f2bf700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 0x7eff5401abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:10.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.521 7eff755aa700 1 -- 172.21.2.117:0/550687468 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff701b4f00 con 0x7eff700a3800 2021-02-10T11:36:10.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.524 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff68000cc0 con 0x7eff700a3800 2021-02-10T11:36:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.651 7eff755aa700 1 -- 172.21.2.117:0/550687468 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802518", "target": ["mgr", ""]}) v1 -- 0x7eff700bb0c0 con 0x7eff540186c0 2021-02-10T11:36:10.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.677 7eff67fff700 1 -- 172.21.2.117:0/550687468 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7eff580009b0 con 0x7eff540186c0 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 msgr2=0x7eff5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 0x7eff5401abc0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 msgr2=0x7eff700a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7eff68009940 tx=0x7eff6800bcf0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 shutdown_connections 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff540186c0 0x7eff5401abc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff700a3800 0x7eff700a3c00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff700b9e60 0x7eff700a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 --2- 172.21.2.117:0/550687468 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff7019a4a0 0x7eff700b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 >> 172.21.2.117:0/550687468 conn(0x7eff70107b60 msgr2=0x7eff700c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:10.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 shutdown_connections 2021-02-10T11:36:10.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:10.684 7eff755aa700 1 -- 172.21.2.117:0/550687468 wait complete. 2021-02-10T11:36:10.675 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802518 2021-02-10T11:36:11.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 Processor -- start 2021-02-10T11:36:11.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 -- start start 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd13c07da20 0x7fd13c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0d1d20 con 0x7fd13c0e51a0 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c077a70 con 0x7fd13c0e4da0 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd143b19700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13c0777d0 con 0x7fd13c07da20 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd141935700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd13c07da20 0x7fd13c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd142136700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd13c0e51a0 0x7fd13c0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45704/0 (socket says 172.21.2.117:45704) 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.109 7fd142136700 1 -- 172.21.2.117:0/2702158817 learned_addr learned my addr 172.21.2.117:0/2702158817 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd141935700 1 -- 172.21.2.117:0/2702158817 >> v1:172.21.2.125:6789/0 conn(0x7fd13c0e4da0 legacy=0x7fd13c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd141935700 1 -- 172.21.2.117:0/2702158817 >> v1:172.21.2.117:6789/0 conn(0x7fd13c0e51a0 legacy=0x7fd13c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd141935700 1 -- 172.21.2.117:0/2702158817 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c1b08e0 con 0x7fd13c07da20 2021-02-10T11:36:11.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd141935700 1 --2- 172.21.2.117:0/2702158817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c07da20 0x7fd13c0771b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fd138006650 tx=0x7fd138009a00).ready entity=mon.0 client_cookie=f193ac1965c495ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd140933700 1 -- 172.21.2.117:0/2702158817 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd138009df0 con 0x7fd13c07da20 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd140933700 1 -- 172.21.2.117:0/2702158817 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1380009b0 con 0x7fd13c07da20 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd140933700 1 -- 172.21.2.117:0/2702158817 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd13800c2b0 con 0x7fd13c07da20 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd143b19700 1 -- 172.21.2.117:0/2702158817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c07da20 msgr2=0x7fd13c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.110 7fd143b19700 1 --2- 172.21.2.117:0/2702158817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c07da20 0x7fd13c0771b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fd138006650 tx=0x7fd138009a00).stop 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.111 7fd143b19700 1 -- 172.21.2.117:0/2702158817 shutdown_connections 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.111 7fd143b19700 1 --2- 172.21.2.117:0/2702158817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c07da20 0x7fd13c0771b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.111 7fd143b19700 1 -- 172.21.2.117:0/2702158817 >> 172.21.2.117:0/2702158817 conn(0x7fd13c0ce450 msgr2=0x7fd13c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.111 7fd143b19700 1 -- 172.21.2.117:0/2702158817 shutdown_connections 2021-02-10T11:36:11.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.111 7fd143b19700 1 -- 172.21.2.117:0/2702158817 wait complete. 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.125 7fd143b19700 1 Processor -- start 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 -- start start 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd13c093e10 0x7fd13c106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd13c107050 0x7fd13c107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c107a70 con 0x7fd13c0933f0 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0c2c60 con 0x7fd13c093e10 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd143b19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13c0c2e80 con 0x7fd13c107050 2021-02-10T11:36:11.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd141935700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd141935700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45460/0 (socket says 172.21.2.117:45460) 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.126 7fd141935700 1 -- 172.21.2.117:0/2212389602 learned_addr learned my addr 172.21.2.117:0/2212389602 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 -- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd13c107050 msgr2=0x7fd13c107450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd13c107050 0x7fd13c107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 -- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd13c093e10 msgr2=0x7fd13c106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd13c093e10 0x7fd13c106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 -- 172.21.2.117:0/2212389602 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0c30a0 con 0x7fd13c0933f0 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd141935700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7fd1380096a0 tx=0x7fd13800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1380093a0 con 0x7fd13c0933f0 2021-02-10T11:36:11.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1380067f0 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd143b19700 1 -- 172.21.2.117:0/2212389602 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c0c30a0 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd138002220 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.127 7fd143b19700 1 -- 172.21.2.117:0/2212389602 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c0c30a0 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.128 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1380052e0 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.128 7fd1327fc700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 0x7fd12401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.128 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd138032780 con 0x7fd13c0933f0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.128 7fd141134700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 0x7fd12401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:11.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.129 7fd141134700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 0x7fd12401ac10 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:11.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.159 7fd143b19700 1 -- 172.21.2.117:0/2212389602 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13c0ce450 con 0x7fd13c0933f0 2021-02-10T11:36:11.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.162 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd138000c90 con 0x7fd13c0933f0 2021-02-10T11:36:11.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.288 7fd143b19700 1 -- 172.21.2.117:0/2212389602 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802518", "target": ["mgr", ""]}) v1 -- 0x7fd13c0ce450 con 0x7fd124018730 2021-02-10T11:36:11.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.292 7fd1327fc700 1 -- 172.21.2.117:0/2212389602 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fd12c006a10 con 0x7fd124018730 2021-02-10T11:36:11.279 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b 2021-02-10T11:36:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 msgr2=0x7fd12401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 0x7fd12401ac10 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 msgr2=0x7fd13c0937f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7fd1380096a0 tx=0x7fd13800b9b0).stop 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 shutdown_connections 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd13c107050 0x7fd13c107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd13c093e10 0x7fd13c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd124018730 0x7fd12401ac10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 --2- 172.21.2.117:0/2212389602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd13c0933f0 0x7fd13c0937f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 >> 172.21.2.117:0/2212389602 conn(0x7fd13c1116a0 msgr2=0x7fd13c0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 shutdown_connections 2021-02-10T11:36:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:11.298 7fd143b19700 1 -- 172.21.2.117:0/2212389602 wait complete. 2021-02-10T11:36:11.290 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802518 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b 2021-02-10T11:36:11.290 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:11.290 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:11.290 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:11.625 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:11.626 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:11.626 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254303 s, 41.2 MB/s 2021-02-10T11:36:12.011 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:12.011 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:12.011 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:12.348 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:12.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:12.348 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:12.721 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:12.721 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:12.721 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247021 s, 42.4 MB/s 2021-02-10T11:36:12.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:12.722 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:12.722 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:13.058 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:13.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:13.058 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:13.435 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:13.435 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:13.436 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251519 s, 41.7 MB/s 2021-02-10T11:36:13.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:13.437 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:13.437 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:13.773 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:13.773 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:13.774 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240794 s, 43.5 MB/s 2021-02-10T11:36:14.150 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:14.150 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:14.150 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:14.486 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:14.486 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:14.486 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0220625 s, 47.5 MB/s 2021-02-10T11:36:14.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:14.863 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:14.863 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:15.198 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:15.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:15.199 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:15.574 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:15.575 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:15.575 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230668 s, 45.5 MB/s 2021-02-10T11:36:15.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:15.576 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:15.576 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:15.911 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:15.911 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:15.912 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:16.288 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:16.288 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:16.288 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239218 s, 43.8 MB/s 2021-02-10T11:36:16.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:16.289 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:16.289 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:16.625 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:16.625 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:16.625 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:17.003 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:17.003 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:17.003 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.026237 s, 40.0 MB/s 2021-02-10T11:36:17.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:17.004 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:17.004 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:17.344 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:17.344 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:17.344 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:17.714 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:17.715 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:17.715 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228104 s, 46.0 MB/s 2021-02-10T11:36:17.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:17.716 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:17.716 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:18.052 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:18.052 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:18.053 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802518/3b84214e-5c2f-4bbc-8d21-13b95513ae1b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259423 s, 40.4 MB/s 2021-02-10T11:36:18.432 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802519 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.864 7f899fc40700 1 Processor -- start 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.864 7f899fc40700 1 -- start start 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.865 7f899fc40700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f899807da20 0x7f89980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.865 7f899fc40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 0x7f8998079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.865 7f899fc40700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8998077a70 con 0x7f89980e4da0 2021-02-10T11:36:18.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.865 7f899fc40700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980777d0 con 0x7f899807da20 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.865 7f899fc40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89981b08e0 con 0x7f89980e51a0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899e25d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 0x7f8998079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899da5c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f899807da20 0x7f89980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899d25b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f89980e4da0 0x7f89981b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38644/0 (socket says 172.21.2.117:38644) 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899d25b700 1 -- 172.21.2.117:0/3443936545 learned_addr learned my addr 172.21.2.117:0/3443936545 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899e25d700 1 -- 172.21.2.117:0/3443936545 >> v2:172.21.2.125:3300/0 conn(0x7f899807da20 msgr2=0x7f89980771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899e25d700 1 --2- 172.21.2.117:0/3443936545 >> v2:172.21.2.125:3300/0 conn(0x7f899807da20 0x7f89980771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899e25d700 1 -- 172.21.2.117:0/3443936545 >> v1:172.21.2.125:6789/0 conn(0x7f89980e4da0 legacy=0x7f89981b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.866 7f899e25d700 1 -- 172.21.2.117:0/3443936545 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899807de20 con 0x7f89980e51a0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899e25d700 1 --2- 172.21.2.117:0/3443936545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 0x7f8998079000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8990006e50 tx=0x7f8990009e70).ready entity=mon.2 client_cookie=bad7b38e633e9f89 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899ca5a700 1 -- 172.21.2.117:0/3443936545 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f899000b510 con 0x7f89980e51a0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899ca5a700 1 -- 172.21.2.117:0/3443936545 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990006790 con 0x7f89980e51a0 2021-02-10T11:36:18.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899ca5a700 1 -- 172.21.2.117:0/3443936545 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f899000b940 con 0x7f89980e51a0 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899fc40700 1 -- 172.21.2.117:0/3443936545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 msgr2=0x7f8998079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.867 7f899fc40700 1 --2- 172.21.2.117:0/3443936545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 0x7f8998079000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f8990006e50 tx=0x7f8990009e70).stop 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 -- 172.21.2.117:0/3443936545 shutdown_connections 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 --2- 172.21.2.117:0/3443936545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980e51a0 0x7f8998079000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 --2- 172.21.2.117:0/3443936545 >> v2:172.21.2.125:3300/0 conn(0x7f899807da20 0x7f89980771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 -- 172.21.2.117:0/3443936545 >> 172.21.2.117:0/3443936545 conn(0x7f89980ce450 msgr2=0x7f89980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 -- 172.21.2.117:0/3443936545 shutdown_connections 2021-02-10T11:36:18.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.868 7f899fc40700 1 -- 172.21.2.117:0/3443936545 wait complete. 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.882 7f899fc40700 1 Processor -- start 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.882 7f899fc40700 1 -- start start 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980a41c0 0x7f89980fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 0x7f89980c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b8010 con 0x7f89980b7390 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b8550 con 0x7f89980c2340 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899fc40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980b8770 con 0x7f89980a41c0 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899e25d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899e25d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45476/0 (socket says 172.21.2.117:45476) 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 0x7f89980c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899e25d700 1 -- 172.21.2.117:0/269509281 learned_addr learned my addr 172.21.2.117:0/269509281 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:18.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899da5c700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980a41c0 0x7f89980fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 -- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980a41c0 msgr2=0x7f89980fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980a41c0 0x7f89980fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 -- 172.21.2.117:0/269509281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 msgr2=0x7f89980b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.883 7f899d25b700 1 -- 172.21.2.117:0/269509281 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980b8990 con 0x7f89980c2340 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f899e25d700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f899d25b700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 0x7f89980c2740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f899400a4c0 tx=0x7f899400a860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f899400ac50 con 0x7f89980c2340 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994004350 con 0x7f89980c2340 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89940047b0 con 0x7f89980c2340 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f899fc40700 1 -- 172.21.2.117:0/269509281 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89980b8990 con 0x7f89980c2340 2021-02-10T11:36:18.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.884 7f899fc40700 1 -- 172.21.2.117:0/269509281 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89980b8990 con 0x7f89980c2340 2021-02-10T11:36:18.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.885 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8994004530 con 0x7f89980c2340 2021-02-10T11:36:18.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.885 7f898a7fc700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 0x7f898001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:18.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.885 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8994032570 con 0x7f89980c2340 2021-02-10T11:36:18.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.885 7f899da5c700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 0x7f898001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:18.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.885 7f899da5c700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 0x7f898001abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:18.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.914 7f899fc40700 1 -- 172.21.2.117:0/269509281 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89981b4f00 con 0x7f89980c2340 2021-02-10T11:36:18.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:18.918 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8994004140 con 0x7f89980c2340 2021-02-10T11:36:19.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.046 7f899fc40700 1 -- 172.21.2.117:0/269509281 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802519", "target": ["mgr", ""]}) v1 -- 0x7f89981b4f00 con 0x7f89800186c0 2021-02-10T11:36:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.073 7f898a7fc700 1 -- 172.21.2.117:0/269509281 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f898c006a70 con 0x7f89800186c0 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 -- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 msgr2=0x7f898001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 0x7f898001abc0 crc :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 -- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 msgr2=0x7f89980c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 0x7f89980c2740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f899400a4c0 tx=0x7f899400a860).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 -- 172.21.2.117:0/269509281 shutdown_connections 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89800186c0 0x7f898001abc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89980b7390 0x7f89980b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89980c2340 0x7f89980c2740 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 --2- 172.21.2.117:0/269509281 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89980a41c0 0x7f89980fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.079 7f899fc40700 1 -- 172.21.2.117:0/269509281 >> 172.21.2.117:0/269509281 conn(0x7f899808bbc0 msgr2=0x7f89980c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:19.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.080 7f899fc40700 1 -- 172.21.2.117:0/269509281 shutdown_connections 2021-02-10T11:36:19.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.080 7f899fc40700 1 -- 172.21.2.117:0/269509281 wait complete. 2021-02-10T11:36:19.071 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802519 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.503 7fa370181700 1 Processor -- start 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.503 7fa370181700 1 -- start start 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.504 7fa370181700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa36807da20 0x7fa3680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.504 7fa370181700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.504 7fa370181700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa368077a70 con 0x7fa3680e4da0 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.504 7fa370181700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3680777d0 con 0x7fa36807da20 2021-02-10T11:36:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.504 7fa370181700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3681b08e0 con 0x7fa3680e51a0 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.505 7fa36e79e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.505 7fa36e79e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43160/0 (socket says 172.21.2.117:43160) 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.505 7fa36e79e700 1 -- 172.21.2.117:0/1447724988 learned_addr learned my addr 172.21.2.117:0/1447724988 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.505 7fa36df9d700 1 --2- 172.21.2.117:0/1447724988 >> v2:172.21.2.125:3300/0 conn(0x7fa36807da20 0x7fa3680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36df9d700 1 -- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 msgr2=0x7fa368079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36df9d700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36df9d700 1 -- 172.21.2.117:0/1447724988 >> v1:172.21.2.125:6789/0 conn(0x7fa3680e4da0 legacy=0x7fa3681b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36df9d700 1 -- 172.21.2.117:0/1447724988 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36807de20 con 0x7fa36807da20 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36cf9b700 1 -- 172.21.2.117:0/1447724988 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762823600 0 0) 0x7fa364000ff0 con 0x7fa3680e4da0 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36e79e700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36df9d700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa36807da20 0x7fa3680771b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa35c006650 tx=0x7fa35c009a80).ready entity=mon.1 client_cookie=1a9b7d85422df609 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36cf9b700 1 -- 172.21.2.117:0/1447724988 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c0009b0 con 0x7fa36807da20 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36cf9b700 1 -- 172.21.2.117:0/1447724988 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c00c0c0 con 0x7fa36807da20 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa36cf9b700 1 -- 172.21.2.117:0/1447724988 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c00c5e0 con 0x7fa36807da20 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa370181700 1 -- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa36807da20 msgr2=0x7fa3680771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.506 7fa370181700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa36807da20 0x7fa3680771b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa35c006650 tx=0x7fa35c009a80).stop 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 -- 172.21.2.117:0/1447724988 shutdown_connections 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3680e51a0 0x7fa368079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 --2- 172.21.2.117:0/1447724988 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa36807da20 0x7fa3680771b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 -- 172.21.2.117:0/1447724988 >> 172.21.2.117:0/1447724988 conn(0x7fa3680ce450 msgr2=0x7fa3680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 -- 172.21.2.117:0/1447724988 shutdown_connections 2021-02-10T11:36:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.507 7fa370181700 1 -- 172.21.2.117:0/1447724988 wait complete. 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.521 7fa370181700 1 Processor -- start 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.521 7fa370181700 1 -- start start 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa368190110 0x7fa3681a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3681a7860 0x7fa36808ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36808a720 con 0x7fa36808a180 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680bb540 con 0x7fa3681a7860 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa370181700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3680bb9e0 con 0x7fa368190110 2021-02-10T11:36:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45484/0 (socket says 172.21.2.117:45484) 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 -- 172.21.2.117:0/396578869 learned_addr learned my addr 172.21.2.117:0/396578869 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 -- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa368190110 msgr2=0x7fa3681a7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa368190110 0x7fa3681a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 -- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3681a7860 msgr2=0x7fa36808ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3681a7860 0x7fa36808ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.522 7fa36df9d700 1 -- 172.21.2.117:0/396578869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3681a7c60 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa36df9d700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fa35c009630 tx=0x7fa35c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c0093a0 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c00bd70 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c005560 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa370181700 1 -- 172.21.2.117:0/396578869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3681a7c60 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa370181700 1 -- 172.21.2.117:0/396578869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3681a7c60 con 0x7fa36808a180 2021-02-10T11:36:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.523 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa35c00a000 con 0x7fa36808a180 2021-02-10T11:36:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.524 7fa35a7fc700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 0x7fa35001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.524 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa35c0327d0 con 0x7fa36808a180 2021-02-10T11:36:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.524 7fa36d79c700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 0x7fa35001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.524 7fa36d79c700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 0x7fa35001abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.554 7fa370181700 1 -- 172.21.2.117:0/396578869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa368077a70 con 0x7fa36808a180 2021-02-10T11:36:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.556 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa35c01f540 con 0x7fa36808a180 2021-02-10T11:36:19.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.684 7fa370181700 1 -- 172.21.2.117:0/396578869 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802519", "target": ["mgr", ""]}) v1 -- 0x7fa3680bb540 con 0x7fa3500186c0 2021-02-10T11:36:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.686 7fa35a7fc700 1 -- 172.21.2.117:0/396578869 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa364007e00 con 0x7fa3500186c0 2021-02-10T11:36:19.670 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 msgr2=0x7fa35001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 0x7fa35001abc0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 msgr2=0x7fa36809b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7fa35c009630 tx=0x7fa35c00b980).stop 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 shutdown_connections 2021-02-10T11:36:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3681a7860 0x7fa36808ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa368190110 0x7fa3681a7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3500186c0 0x7fa35001abc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 --2- 172.21.2.117:0/396578869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa36808a180 0x7fa36809b600 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 >> 172.21.2.117:0/396578869 conn(0x7fa3680a1360 msgr2=0x7fa368091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 shutdown_connections 2021-02-10T11:36:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:19.689 7fa370181700 1 -- 172.21.2.117:0/396578869 wait complete. 2021-02-10T11:36:19.680 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802519 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e 2021-02-10T11:36:19.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:19.680 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:19.680 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:20.016 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:20.016 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:20.016 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232241 s, 45.2 MB/s 2021-02-10T11:36:20.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:20.400 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:20.400 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:20.735 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:20.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:20.736 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:21.112 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:21.113 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:21.113 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249232 s, 42.1 MB/s 2021-02-10T11:36:21.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:21.114 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:21.114 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:21.450 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:21.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:21.450 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:21.825 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229282 s, 45.7 MB/s 2021-02-10T11:36:21.826 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:21.827 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:21.827 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:22.163 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:22.163 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:22.163 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:22.535 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:22.535 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:22.535 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250184 s, 41.9 MB/s 2021-02-10T11:36:22.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:22.536 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:22.537 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:22.872 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:22.872 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:22.872 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:23.247 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:23.247 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:23.247 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022393 s, 46.8 MB/s 2021-02-10T11:36:23.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:23.249 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:23.249 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:23.585 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:23.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:23.585 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:23.964 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:23.965 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:23.965 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257016 s, 40.8 MB/s 2021-02-10T11:36:23.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:23.967 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:23.967 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:24.304 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:24.304 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:24.304 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:24.679 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:24.679 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:24.679 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231673 s, 45.3 MB/s 2021-02-10T11:36:24.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:24.680 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:24.680 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:25.015 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:25.015 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:25.015 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:25.390 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:25.390 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:25.390 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223069 s, 47.0 MB/s 2021-02-10T11:36:25.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:25.391 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:25.391 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:25.727 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:25.727 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:25.727 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:26.104 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250039 s, 41.9 MB/s 2021-02-10T11:36:26.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:26.105 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:26.105 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:26.440 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:26.441 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:26.441 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802519/c30c530f-7328-40af-971b-beb858971a0e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:26.815 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:26.815 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:26.816 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244146 s, 42.9 MB/s 2021-02-10T11:36:26.816 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802520 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 Processor -- start 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 -- start start 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 0x7f8edc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 0x7f8edc0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8edc077a70 con 0x7f8edc0e4da0 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc0e5c20 con 0x7f8edc0e51a0 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.248 7f8ee0a26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0777d0 con 0x7f8edc07da20 2021-02-10T11:36:27.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8edae1d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 0x7f8edc0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 0x7f8edc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8ed9e1b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f8edc0e4da0 0x7f8edc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38668/0 (socket says 172.21.2.117:38668) 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8ed9e1b700 1 -- 172.21.2.117:0/3935826079 learned_addr learned my addr 172.21.2.117:0/3935826079 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 -- 172.21.2.117:0/3935826079 >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 msgr2=0x7f8edc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 --2- 172.21.2.117:0/3935826079 >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 0x7f8edc0d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 -- 172.21.2.117:0/3935826079 >> v1:172.21.2.125:6789/0 conn(0x7f8edc0e4da0 legacy=0x7f8edc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 -- 172.21.2.117:0/3935826079 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc1b08e0 con 0x7f8edc07da20 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8edae1d700 1 --2- 172.21.2.117:0/3935826079 >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 0x7f8edc0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.250 7f8eda61c700 1 --2- 172.21.2.117:0/3935826079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 0x7f8edc0771b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8ec4006650 tx=0x7f8ec4009a80).ready entity=mon.2 client_cookie=1415a442614fd8fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.251 7f8ed961a700 1 -- 172.21.2.117:0/3935826079 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ec40009b0 con 0x7f8edc07da20 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.251 7f8ed961a700 1 -- 172.21.2.117:0/3935826079 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec400c0c0 con 0x7f8edc07da20 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.251 7f8ed961a700 1 -- 172.21.2.117:0/3935826079 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ec400c5e0 con 0x7f8edc07da20 2021-02-10T11:36:27.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.251 7f8ee0a26700 1 -- 172.21.2.117:0/3935826079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 msgr2=0x7f8edc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.251 7f8ee0a26700 1 --2- 172.21.2.117:0/3935826079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 0x7f8edc0771b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8ec4006650 tx=0x7f8ec4009a80).stop 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 -- 172.21.2.117:0/3935826079 shutdown_connections 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 --2- 172.21.2.117:0/3935826079 >> v2:172.21.2.125:3300/0 conn(0x7f8edc0e51a0 0x7f8edc0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 --2- 172.21.2.117:0/3935826079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc07da20 0x7f8edc0771b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 -- 172.21.2.117:0/3935826079 >> 172.21.2.117:0/3935826079 conn(0x7f8edc0ce450 msgr2=0x7f8edc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 -- 172.21.2.117:0/3935826079 shutdown_connections 2021-02-10T11:36:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.252 7f8ee0a26700 1 -- 172.21.2.117:0/3935826079 wait complete. 2021-02-10T11:36:27.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.266 7f8ee0a26700 1 Processor -- start 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.266 7f8ee0a26700 1 -- start start 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc1b5130 0x7f8edc1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8edc1b5b50 0x7f8edc0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0b8da0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0a4140 con 0x7f8edc1b5b50 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0a4320 con 0x7f8edc1b5130 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45498/0 (socket says 172.21.2.117:45498) 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 -- 172.21.2.117:0/3085341618 learned_addr learned my addr 172.21.2.117:0/3085341618 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 -- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc1b5130 msgr2=0x7f8edc1b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc1b5130 0x7f8edc1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 -- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8edc1b5b50 msgr2=0x7f8edc0b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8edc1b5b50 0x7f8edc0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 -- 172.21.2.117:0/3085341618 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0b62b0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8edae1d700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f8ed400b970 tx=0x7f8ed400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ed4000f90 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4006790 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ed4000f90 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.267 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc0b62b0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.268 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc0b62b0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.268 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ed4006aa0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.268 7f8ed2ffd700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 0x7f8ec001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.268 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ed40328e0 con 0x7f8edc1b5f50 2021-02-10T11:36:27.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.268 7f8eda61c700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 0x7f8ec001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.269 7f8eda61c700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 0x7f8ec001abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.298 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc1b4f00 con 0x7f8edc1b5f50 2021-02-10T11:36:27.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.301 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ed402df80 con 0x7f8edc1b5f50 2021-02-10T11:36:27.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.427 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802520", "target": ["mgr", ""]}) v1 -- 0x7f8edc0b6b30 con 0x7f8ec00186c0 2021-02-10T11:36:27.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.452 7f8ed2ffd700 1 -- 172.21.2.117:0/3085341618 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8ec4000940 con 0x7f8ec00186c0 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 msgr2=0x7f8ec001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 0x7f8ec001abc0 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 msgr2=0x7f8edc0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0x7f8ed400b970 tx=0x7f8ed400bd10).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 shutdown_connections 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ec00186c0 0x7f8ec001abc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8edc1b5f50 0x7f8edc0b5c90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8edc1b5b50 0x7f8edc0b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 --2- 172.21.2.117:0/3085341618 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8edc1b5130 0x7f8edc1b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.459 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 >> 172.21.2.117:0/3085341618 conn(0x7f8edc08bbc0 msgr2=0x7f8edc0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.460 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 shutdown_connections 2021-02-10T11:36:27.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.460 7f8ee0a26700 1 -- 172.21.2.117:0/3085341618 wait complete. 2021-02-10T11:36:27.451 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802520 2021-02-10T11:36:27.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.882 7f9f036c1700 1 Processor -- start 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.882 7f9f036c1700 1 -- start start 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f036c1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9efc07da20 0x7f9efc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f036c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc0e51a0 0x7f9efc079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f036c1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc077a70 con 0x7f9efc0e4da0 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f036c1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc0777d0 con 0x7f9efc07da20 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f036c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc1b08e0 con 0x7f9efc0e51a0 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9efc07da20 0x7f9efc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9efc07da20 0x7f9efc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45504/0 (socket says 172.21.2.117:45504) 2021-02-10T11:36:27.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 -- 172.21.2.117:0/522813504 learned_addr learned my addr 172.21.2.117:0/522813504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 -- 172.21.2.117:0/522813504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc0e51a0 msgr2=0x7f9efc079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 --2- 172.21.2.117:0/522813504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc0e51a0 0x7f9efc079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 -- 172.21.2.117:0/522813504 >> v1:172.21.2.125:6789/0 conn(0x7f9efc0e4da0 legacy=0x7f9efc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.883 7f9f014dd700 1 -- 172.21.2.117:0/522813504 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc07de20 con 0x7f9efc07da20 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9f014dd700 1 --2- 172.21.2.117:0/522813504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc07da20 0x7f9efc0771b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f9ef0006700 tx=0x7f9ef0009b80).ready entity=mon.0 client_cookie=5bd79d0ee1b37824 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9eeffff700 1 -- 172.21.2.117:0/522813504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef000c0c0 con 0x7f9efc07da20 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9eeffff700 1 -- 172.21.2.117:0/522813504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef000c440 con 0x7f9efc07da20 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9eeffff700 1 -- 172.21.2.117:0/522813504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef000c960 con 0x7f9efc07da20 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9f036c1700 1 -- 172.21.2.117:0/522813504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc07da20 msgr2=0x7f9efc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.884 7f9f036c1700 1 --2- 172.21.2.117:0/522813504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc07da20 0x7f9efc0771b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f9ef0006700 tx=0x7f9ef0009b80).stop 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 -- 172.21.2.117:0/522813504 shutdown_connections 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 --2- 172.21.2.117:0/522813504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc0e51a0 0x7f9efc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 --2- 172.21.2.117:0/522813504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc07da20 0x7f9efc0771b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:27.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 -- 172.21.2.117:0/522813504 >> 172.21.2.117:0/522813504 conn(0x7f9efc0ce450 msgr2=0x7f9efc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:27.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 -- 172.21.2.117:0/522813504 shutdown_connections 2021-02-10T11:36:27.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.885 7f9f036c1700 1 -- 172.21.2.117:0/522813504 wait complete. 2021-02-10T11:36:27.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 Processor -- start 2021-02-10T11:36:27.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 -- start start 2021-02-10T11:36:27.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc097dd0 0x7f9efc1a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc1a7dd0 0x7f9efc08f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.899 7f9f036c1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc090a90 con 0x7f9efc08ff50 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f036c1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc090cb0 con 0x7f9efc097dd0 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f036c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0b71e0 con 0x7f9efc1a7dd0 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45508/0 (socket says 172.21.2.117:45508) 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 -- 172.21.2.117:0/2920164768 learned_addr learned my addr 172.21.2.117:0/2920164768 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 -- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc1a7dd0 msgr2=0x7f9efc08f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc1a7dd0 0x7f9efc08f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 -- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc097dd0 msgr2=0x7f9efc1a77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc097dd0 0x7f9efc1a77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 -- 172.21.2.117:0/2920164768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0b7400 con 0x7f9efc08ff50 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f01cde700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f9ef4009980 tx=0x7f9ef400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef4005870 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef4005b30 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ef4006030 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.900 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0b7400 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.901 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc0b7400 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.901 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ef4005d10 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.901 7f9eedffb700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 0x7f9edc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.901 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9ef4032b70 con 0x7f9efc08ff50 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.902 7f9f014dd700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 0x7f9edc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:27.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.902 7f9f014dd700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 0x7f9edc01ac10 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:27.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.933 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc1032a0 con 0x7f9efc08ff50 2021-02-10T11:36:27.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:27.935 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ef400be10 con 0x7f9efc08ff50 2021-02-10T11:36:28.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.063 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802520", "target": ["mgr", ""]}) v1 -- 0x7f9efc1032a0 con 0x7f9edc018730 2021-02-10T11:36:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.066 7f9eedffb700 1 -- 172.21.2.117:0/2920164768 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9ef0000970 con 0x7f9edc018730 2021-02-10T11:36:28.054 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c 2021-02-10T11:36:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.072 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 msgr2=0x7f9edc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.072 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 0x7f9edc01ac10 crc :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.072 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 msgr2=0x7f9efc090350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.072 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f9ef4009980 tx=0x7f9ef400bcd0).stop 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 shutdown_connections 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9edc018730 0x7f9edc01ac10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9efc08ff50 0x7f9efc090350 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9efc1a7dd0 0x7f9efc08f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 --2- 172.21.2.117:0/2920164768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9efc097dd0 0x7f9efc1a77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 >> 172.21.2.117:0/2920164768 conn(0x7f9efc0c3c10 msgr2=0x7f9efc0b4670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 shutdown_connections 2021-02-10T11:36:28.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:28.073 7f9f036c1700 1 -- 172.21.2.117:0/2920164768 wait complete. 2021-02-10T11:36:28.063 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802520 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c 2021-02-10T11:36:28.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:28.064 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:28.064 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:28.401 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:28.401 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:28.401 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:28.783 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:28.783 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:28.783 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023308 s, 45.0 MB/s 2021-02-10T11:36:28.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:28.784 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:28.784 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:29.120 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:29.120 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:29.121 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:29.497 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:29.497 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:29.497 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253666 s, 41.3 MB/s 2021-02-10T11:36:29.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:29.498 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:29.498 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:29.835 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:29.835 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:29.835 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:30.210 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023798 s, 44.1 MB/s 2021-02-10T11:36:30.212 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:30.212 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:30.212 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:30.547 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:30.548 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:30.548 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:30.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:30.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:30.925 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246414 s, 42.6 MB/s 2021-02-10T11:36:30.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:30.926 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:30.926 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:31.261 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:31.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:31.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:31.638 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:31.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:31.639 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240119 s, 43.7 MB/s 2021-02-10T11:36:31.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:31.640 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:31.640 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:31.974 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:31.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:31.975 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:32.353 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:32.483 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:32.483 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249357 s, 42.1 MB/s 2021-02-10T11:36:32.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:32.489 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:32.490 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:32.675 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:32.682 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:36:32.829 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:32.830 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:32.830 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:33.206 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:33.207 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:33.207 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248309 s, 42.2 MB/s 2021-02-10T11:36:33.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.208 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.208 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:33.546 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:33.546 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:33.546 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:33.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:33.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:33.925 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024243 s, 43.3 MB/s 2021-02-10T11:36:33.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:33.926 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:33.927 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:34.262 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:34.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:34.263 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:34.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:34.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:34.640 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240461 s, 43.6 MB/s 2021-02-10T11:36:34.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:34.641 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:34.641 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:34.976 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:34.977 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:34.977 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802520/a0cf8331-51e0-472e-be3b-ad501011284c/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:35.353 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:35.353 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:35.353 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254728 s, 41.2 MB/s 2021-02-10T11:36:35.354 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802521 2021-02-10T11:36:35.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 Processor -- start 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 -- start start 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88ec07da20 0x7f88ec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f88ec0e51a0 0x7f88ec079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec077a70 con 0x7f88ec0e4da0 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0e5c20 con 0x7f88ec07da20 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88f2828700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0777d0 con 0x7f88ec0e51a0 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88ebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88ec07da20 0x7f88ec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88ebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88ec07da20 0x7f88ec0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45516/0 (socket says 172.21.2.117:45516) 2021-02-10T11:36:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.786 7f88ebfff700 1 -- 172.21.2.117:0/3220819416 learned_addr learned my addr 172.21.2.117:0/3220819416 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88ebfff700 1 -- 172.21.2.117:0/3220819416 >> v2:172.21.2.125:3300/0 conn(0x7f88ec0e51a0 msgr2=0x7f88ec079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88ebfff700 1 --2- 172.21.2.117:0/3220819416 >> v2:172.21.2.125:3300/0 conn(0x7f88ec0e51a0 0x7f88ec079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88ebfff700 1 -- 172.21.2.117:0/3220819416 >> v1:172.21.2.125:6789/0 conn(0x7f88ec0e4da0 legacy=0x7f88ec1b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88ebfff700 1 -- 172.21.2.117:0/3220819416 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec1b08e0 con 0x7f88ec07da20 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88ebfff700 1 --2- 172.21.2.117:0/3220819416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec07da20 0x7f88ec0771b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f88dc006700 tx=0x7f88dc009b80).ready entity=mon.0 client_cookie=35d0fffc8168eefd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88eaffd700 1 -- 172.21.2.117:0/3220819416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88dc00c0c0 con 0x7f88ec07da20 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88eaffd700 1 -- 172.21.2.117:0/3220819416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc00c440 con 0x7f88ec07da20 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88eaffd700 1 -- 172.21.2.117:0/3220819416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88dc00c8a0 con 0x7f88ec07da20 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88f2828700 1 -- 172.21.2.117:0/3220819416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec07da20 msgr2=0x7f88ec0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.787 7f88f2828700 1 --2- 172.21.2.117:0/3220819416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec07da20 0x7f88ec0771b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0x7f88dc006700 tx=0x7f88dc009b80).stop 2021-02-10T11:36:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 -- 172.21.2.117:0/3220819416 shutdown_connections 2021-02-10T11:36:35.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 --2- 172.21.2.117:0/3220819416 >> v2:172.21.2.125:3300/0 conn(0x7f88ec0e51a0 0x7f88ec079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 --2- 172.21.2.117:0/3220819416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec07da20 0x7f88ec0771b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 -- 172.21.2.117:0/3220819416 >> 172.21.2.117:0/3220819416 conn(0x7f88ec0ce450 msgr2=0x7f88ec0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:35.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 -- 172.21.2.117:0/3220819416 shutdown_connections 2021-02-10T11:36:35.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.788 7f88f2828700 1 -- 172.21.2.117:0/3220819416 wait complete. 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.802 7f88f2828700 1 Processor -- start 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.802 7f88f2828700 1 -- start start 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88ec08a870 0x7f88ec0a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88ec0a8170 0x7f88ec08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec08cf40 con 0x7f88ec08a470 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec08dae0 con 0x7f88ec08a870 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88f2828700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec08dcc0 con 0x7f88ec0a8170 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45522/0 (socket says 172.21.2.117:45522) 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 -- 172.21.2.117:0/2774297180 learned_addr learned my addr 172.21.2.117:0/2774297180 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 -- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88ec0a8170 msgr2=0x7f88ec08d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88ec0a8170 0x7f88ec08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 -- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88ec08a870 msgr2=0x7f88ec0a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88ec08a870 0x7f88ec0a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 -- 172.21.2.117:0/2774297180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec08dee0 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.803 7f88ebfff700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f88dc00b5e0 tx=0x7f88dc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88dc00bd70 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc002030 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88dc002550 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88f2828700 1 -- 172.21.2.117:0/2774297180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec08dee0 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88f2828700 1 -- 172.21.2.117:0/2774297180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec08e470 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88dc000c90 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 0x7f88cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.804 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f88dc0326d0 con 0x7f88ec08a470 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.805 7f88eb7fe700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 0x7f88cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:35.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.805 7f88eb7fe700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 0x7f88cc01abc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:35.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.836 7f88f2828700 1 -- 172.21.2.117:0/2774297180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec1032d0 con 0x7f88ec08a470 2021-02-10T11:36:35.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.839 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88dc030d30 con 0x7f88ec08a470 2021-02-10T11:36:35.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.964 7f88f2828700 1 -- 172.21.2.117:0/2774297180 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802521", "target": ["mgr", ""]}) v1 -- 0x7f88ec08e470 con 0x7f88cc0186c0 2021-02-10T11:36:35.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.990 7f88e8ff9700 1 -- 172.21.2.117:0/2774297180 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f88e00009b0 con 0x7f88cc0186c0 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 msgr2=0x7f88cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 0x7f88cc01abc0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 msgr2=0x7f88ec0a7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f88dc00b5e0 tx=0x7f88dc00b980).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 shutdown_connections 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88ec0a8170 0x7f88ec08d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88ec08a870 0x7f88ec0a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88cc0186c0 0x7f88cc01abc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 --2- 172.21.2.117:0/2774297180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88ec08a470 0x7f88ec0a7530 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:35.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 >> 172.21.2.117:0/2774297180 conn(0x7f88ec0a9210 msgr2=0x7f88ec0a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:35.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 shutdown_connections 2021-02-10T11:36:35.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:35.997 7f88f2828700 1 -- 172.21.2.117:0/2774297180 wait complete. 2021-02-10T11:36:35.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802521 2021-02-10T11:36:36.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 Processor -- start 2021-02-10T11:36:36.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 -- start start 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8bac07da20 0x7f8bac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac0d1d20 con 0x7f8bac0e51a0 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac077a70 con 0x7f8bac0e4da0 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.423 7f8bb212e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0777d0 con 0x7f8bac07da20 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8bac07da20 0x7f8bac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8babfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8bac0e51a0 0x7f8bac0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45776/0 (socket says 172.21.2.117:45776) 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8bac07da20 0x7f8bac0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45526/0 (socket says 172.21.2.117:45526) 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8babfff700 1 -- 172.21.2.117:0/3239537295 learned_addr learned my addr 172.21.2.117:0/3239537295 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 -- 172.21.2.117:0/3239537295 >> v1:172.21.2.125:6789/0 conn(0x7f8bac0e4da0 legacy=0x7f8bac1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 -- 172.21.2.117:0/3239537295 >> v1:172.21.2.117:6789/0 conn(0x7f8bac0e51a0 legacy=0x7f8bac0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 -- 172.21.2.117:0/3239537295 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac1b08e0 con 0x7f8bac07da20 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.424 7f8bab7fe700 1 --2- 172.21.2.117:0/3239537295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac07da20 0x7f8bac0771b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f8b9c006650 tx=0x7f8b9c009a80).ready entity=mon.0 client_cookie=ede1503f0f3700f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8baa7fc700 1 -- 172.21.2.117:0/3239537295 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b9c00c0c0 con 0x7f8bac07da20 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8baa7fc700 1 -- 172.21.2.117:0/3239537295 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c00c340 con 0x7f8bac07da20 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8baa7fc700 1 -- 172.21.2.117:0/3239537295 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b9c00c860 con 0x7f8bac07da20 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8bb212e700 1 -- 172.21.2.117:0/3239537295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac07da20 msgr2=0x7f8bac0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8bb212e700 1 --2- 172.21.2.117:0/3239537295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac07da20 0x7f8bac0771b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f8b9c006650 tx=0x7f8b9c009a80).stop 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8bb212e700 1 -- 172.21.2.117:0/3239537295 shutdown_connections 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8bb212e700 1 --2- 172.21.2.117:0/3239537295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac07da20 0x7f8bac0771b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.425 7f8bb212e700 1 -- 172.21.2.117:0/3239537295 >> 172.21.2.117:0/3239537295 conn(0x7f8bac0ce450 msgr2=0x7f8bac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.426 7f8bb212e700 1 -- 172.21.2.117:0/3239537295 shutdown_connections 2021-02-10T11:36:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.426 7f8bb212e700 1 -- 172.21.2.117:0/3239537295 wait complete. 2021-02-10T11:36:36.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 Processor -- start 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 -- start start 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8bac0b6190 0x7f8bac1b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8bac1b0770 0x7f8bac1b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac1b0f50 con 0x7f8bac1c22d0 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0b6590 con 0x7f8bac1b0770 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.440 7f8bb212e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac1b0b70 con 0x7f8bac0b6190 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45530/0 (socket says 172.21.2.117:45530) 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 -- 172.21.2.117:0/610927829 learned_addr learned my addr 172.21.2.117:0/610927829 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 -- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8bac0b6190 msgr2=0x7f8bac1b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8bac0b6190 0x7f8bac1b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 -- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8bac1b0770 msgr2=0x7f8bac1b14d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8bac1b0770 0x7f8bac1b14d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 -- 172.21.2.117:0/610927829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac1b1af0 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8bab7fe700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8b9c009630 tx=0x7f8b9c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.441 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b9c0093a0 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c00bd70 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8bb212e700 1 -- 172.21.2.117:0/610927829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac1b1af0 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8bb212e700 1 -- 172.21.2.117:0/610927829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac1b2080 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b9c005560 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b9c00a000 con 0x7f8bac1c22d0 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8ba3fff700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 0x7f8b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:36.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.442 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b9c032670 con 0x7f8bac1c22d0 2021-02-10T11:36:36.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.443 7f8baaffd700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 0x7f8b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:36.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.444 7f8baaffd700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 0x7f8b9001abc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:36.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.473 7f8bb212e700 1 -- 172.21.2.117:0/610927829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac0ce450 con 0x7f8bac1c22d0 2021-02-10T11:36:36.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.476 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b9c009340 con 0x7f8bac1c22d0 2021-02-10T11:36:36.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.602 7f8bb212e700 1 -- 172.21.2.117:0/610927829 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802521", "target": ["mgr", ""]}) v1 -- 0x7f8bac1b2080 con 0x7f8b900186c0 2021-02-10T11:36:36.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.605 7f8ba3fff700 1 -- 172.21.2.117:0/610927829 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f8b94006a10 con 0x7f8b900186c0 2021-02-10T11:36:36.592 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5 2021-02-10T11:36:36.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.611 7f8bb212e700 1 -- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 msgr2=0x7f8b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.611 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 0x7f8b9001abc0 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.611 7f8bb212e700 1 -- 172.21.2.117:0/610927829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 msgr2=0x7f8bac0b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:36.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.611 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f8b9c009630 tx=0x7f8b9c00b980).stop 2021-02-10T11:36:36.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 -- 172.21.2.117:0/610927829 shutdown_connections 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8bac1b0770 0x7f8bac1b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8bac0b6190 0x7f8bac1b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b900186c0 0x7f8b9001abc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 --2- 172.21.2.117:0/610927829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8bac1c22d0 0x7f8bac0b5b70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 -- 172.21.2.117:0/610927829 >> 172.21.2.117:0/610927829 conn(0x7f8bac1116c0 msgr2=0x7f8bac108870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 -- 172.21.2.117:0/610927829 shutdown_connections 2021-02-10T11:36:36.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:36.612 7f8bb212e700 1 -- 172.21.2.117:0/610927829 wait complete. 2021-02-10T11:36:36.602 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802521 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5 2021-02-10T11:36:36.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:36.602 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:36.602 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:36.939 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:36.939 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:36.940 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:37.321 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:37.321 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:37.321 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230662 s, 45.5 MB/s 2021-02-10T11:36:37.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:37.322 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:37.323 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:37.658 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:37.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:37.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:38.032 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:38.032 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:38.032 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251318 s, 41.7 MB/s 2021-02-10T11:36:38.033 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:38.033 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:38.033 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:38.368 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:38.369 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:38.369 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:38.744 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:38.744 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:38.745 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240594 s, 43.6 MB/s 2021-02-10T11:36:38.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:38.746 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:38.747 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:39.081 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:39.081 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:39.082 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:39.458 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:39.458 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:39.458 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238535 s, 44.0 MB/s 2021-02-10T11:36:39.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:39.459 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:39.459 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:39.795 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:39.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:39.795 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:40.170 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:40.170 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:40.170 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238107 s, 44.0 MB/s 2021-02-10T11:36:40.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:40.171 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:40.171 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:40.507 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:40.508 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:40.508 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:40.882 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:40.882 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:40.883 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221816 s, 47.3 MB/s 2021-02-10T11:36:40.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:40.884 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:40.884 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:41.220 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:41.220 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:41.221 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:41.596 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:41.596 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:41.596 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232969 s, 45.0 MB/s 2021-02-10T11:36:41.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:41.597 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:41.597 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:41.932 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:41.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:41.933 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257115 s, 40.8 MB/s 2021-02-10T11:36:42.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:42.311 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:42.312 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:42.646 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:42.647 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:42.647 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:43.020 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:43.020 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:43.020 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244726 s, 42.8 MB/s 2021-02-10T11:36:43.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:43.021 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:43.022 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:43.357 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:43.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:43.357 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802521/57e97db2-a0df-40e5-9512-1845e9a5bab5/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:43.734 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:43.734 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:43.734 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250633 s, 41.8 MB/s 2021-02-10T11:36:43.735 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802522 2021-02-10T11:36:44.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 Processor -- start 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 -- start start 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99d807da20 0x7f99d80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f99d80e4da0 0x7f99d81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d8077a70 con 0x7f99d80e51a0 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d80777d0 con 0x7f99d807da20 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99dd085700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d807de20 con 0x7f99d80e4da0 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.168 7f99d6d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99d807da20 0x7f99d80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99d807da20 0x7f99d80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45538/0 (socket says 172.21.2.117:45538) 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 -- 172.21.2.117:0/2206926434 learned_addr learned my addr 172.21.2.117:0/2206926434 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 -- 172.21.2.117:0/2206926434 >> v2:172.21.2.125:3300/0 conn(0x7f99d80e4da0 msgr2=0x7f99d81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 --2- 172.21.2.117:0/2206926434 >> v2:172.21.2.125:3300/0 conn(0x7f99d80e4da0 0x7f99d81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 -- 172.21.2.117:0/2206926434 >> v1:172.21.2.125:6789/0 conn(0x7f99d80e51a0 legacy=0x7f99d8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 -- 172.21.2.117:0/2206926434 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d81b0710 con 0x7f99d807da20 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d6d9d700 1 --2- 172.21.2.117:0/2206926434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d807da20 0x7f99d80771b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f99c8006700 tx=0x7f99c8009b80).ready entity=mon.0 client_cookie=bd259a10eab9951 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d5d9b700 1 -- 172.21.2.117:0/2206926434 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c800c0c0 con 0x7f99d807da20 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d5d9b700 1 -- 172.21.2.117:0/2206926434 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800c340 con 0x7f99d807da20 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99d5d9b700 1 -- 172.21.2.117:0/2206926434 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c800c860 con 0x7f99d807da20 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99dd085700 1 -- 172.21.2.117:0/2206926434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d807da20 msgr2=0x7f99d80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.169 7f99dd085700 1 --2- 172.21.2.117:0/2206926434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d807da20 0x7f99d80771b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f99c8006700 tx=0x7f99c8009b80).stop 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 -- 172.21.2.117:0/2206926434 shutdown_connections 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 --2- 172.21.2.117:0/2206926434 >> v2:172.21.2.125:3300/0 conn(0x7f99d80e4da0 0x7f99d81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 --2- 172.21.2.117:0/2206926434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d807da20 0x7f99d80771b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 -- 172.21.2.117:0/2206926434 >> 172.21.2.117:0/2206926434 conn(0x7f99d80ce450 msgr2=0x7f99d80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 -- 172.21.2.117:0/2206926434 shutdown_connections 2021-02-10T11:36:44.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.170 7f99dd085700 1 -- 172.21.2.117:0/2206926434 wait complete. 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.184 7f99dd085700 1 Processor -- start 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.184 7f99dd085700 1 -- start start 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d818a0e0 0x7f99d818a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d80b79c0 0x7f99d80b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d8189be0 con 0x7f99d80a9890 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80b83e0 con 0x7f99d818a0e0 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99dd085700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80b8600 con 0x7f99d80b79c0 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45544/0 (socket says 172.21.2.117:45544) 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 -- 172.21.2.117:0/4201529789 learned_addr learned my addr 172.21.2.117:0/4201529789 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 -- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d80b79c0 msgr2=0x7f99d80b7dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d80b79c0 0x7f99d80b7dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 -- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d818a0e0 msgr2=0x7f99d818a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d818a0e0 0x7f99d818a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 -- 172.21.2.117:0/4201529789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d80b8820 con 0x7f99d80a9890 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.185 7f99d6d9d700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f99c800b5e0 tx=0x7f99c800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c800bd70 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c8002030 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c8002550 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99dd085700 1 -- 172.21.2.117:0/4201529789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d80b8820 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99dd085700 1 -- 172.21.2.117:0/4201529789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d80b8820 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99c8000c90 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 0x7f99bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.186 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f99c8032800 con 0x7f99d80a9890 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.187 7f99d659c700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 0x7f99bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.187 7f99d659c700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 0x7f99bc01abc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.218 7f99dd085700 1 -- 172.21.2.117:0/4201529789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d81032d0 con 0x7f99d80a9890 2021-02-10T11:36:44.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.221 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99c801f570 con 0x7f99d80a9890 2021-02-10T11:36:44.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.345 7f99dd085700 1 -- 172.21.2.117:0/4201529789 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802522", "target": ["mgr", ""]}) v1 -- 0x7f99d81032d0 con 0x7f99bc0186c0 2021-02-10T11:36:44.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.370 7f99cf7fe700 1 -- 172.21.2.117:0/4201529789 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f99c00009b0 con 0x7f99bc0186c0 2021-02-10T11:36:44.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 msgr2=0x7f99bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 0x7f99bc01abc0 crc :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 msgr2=0x7f99d80a9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f99c800b5e0 tx=0x7f99c800b980).stop 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 shutdown_connections 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d80b79c0 0x7f99d80b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d818a0e0 0x7f99d818a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99bc0186c0 0x7f99bc01abc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 --2- 172.21.2.117:0/4201529789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d80a9890 0x7f99d80a9c90 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 >> 172.21.2.117:0/4201529789 conn(0x7f99d80b56c0 msgr2=0x7f99d80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 shutdown_connections 2021-02-10T11:36:44.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.379 7f99dd085700 1 -- 172.21.2.117:0/4201529789 wait complete. 2021-02-10T11:36:44.370 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802522 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.804 7f6b9e4b8700 1 Processor -- start 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.804 7f6b9e4b8700 1 -- start start 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9e4b8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6b9807da20 0x7f6b980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9e4b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b980e4da0 0x7f6b981b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9e4b8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b98077a70 con 0x7f6b980e51a0 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9e4b8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b980777d0 con 0x7f6b980e4da0 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9e4b8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9807de20 con 0x7f6b9807da20 2021-02-10T11:36:44.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b977fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b980e4da0 0x7f6b981b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9cad5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6b980e51a0 0x7f6b98077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45802/0 (socket says 172.21.2.117:45802) 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b9cad5700 1 -- 172.21.2.117:0/1073020930 learned_addr learned my addr 172.21.2.117:0/1073020930 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b97fff700 1 --2- 172.21.2.117:0/1073020930 >> v2:172.21.2.125:3300/0 conn(0x7f6b9807da20 0x7f6b980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.805 7f6b97fff700 1 -- 172.21.2.117:0/1073020930 >> v2:172.21.2.117:3300/0 conn(0x7f6b980e4da0 msgr2=0x7f6b981b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b97fff700 1 --2- 172.21.2.117:0/1073020930 >> v2:172.21.2.117:3300/0 conn(0x7f6b980e4da0 0x7f6b981b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b97fff700 1 -- 172.21.2.117:0/1073020930 >> v1:172.21.2.117:6789/0 conn(0x7f6b980e51a0 legacy=0x7f6b98077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b97fff700 1 -- 172.21.2.117:0/1073020930 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b981b0710 con 0x7f6b9807da20 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b97fff700 1 --2- 172.21.2.117:0/1073020930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b9807da20 0x7f6b980771b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6b88006650 tx=0x7f6b88009a80).ready entity=mon.1 client_cookie=3e280ed07729bcd3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b96ffd700 1 -- 172.21.2.117:0/1073020930 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8800c0c0 con 0x7f6b9807da20 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b96ffd700 1 -- 172.21.2.117:0/1073020930 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8800c340 con 0x7f6b9807da20 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b96ffd700 1 -- 172.21.2.117:0/1073020930 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8800c7a0 con 0x7f6b9807da20 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b9e4b8700 1 -- 172.21.2.117:0/1073020930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b9807da20 msgr2=0x7f6b980771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.806 7f6b9e4b8700 1 --2- 172.21.2.117:0/1073020930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b9807da20 0x7f6b980771b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f6b88006650 tx=0x7f6b88009a80).stop 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 -- 172.21.2.117:0/1073020930 shutdown_connections 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 --2- 172.21.2.117:0/1073020930 >> v2:172.21.2.117:3300/0 conn(0x7f6b980e4da0 0x7f6b981b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 --2- 172.21.2.117:0/1073020930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b9807da20 0x7f6b980771b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 -- 172.21.2.117:0/1073020930 >> 172.21.2.117:0/1073020930 conn(0x7f6b980ce450 msgr2=0x7f6b980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 -- 172.21.2.117:0/1073020930 shutdown_connections 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.807 7f6b9e4b8700 1 -- 172.21.2.117:0/1073020930 wait complete. 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.821 7f6b9e4b8700 1 Processor -- start 2021-02-10T11:36:44.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 -- start start 2021-02-10T11:36:44.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 0x7f6b98106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b98090420 0x7f6b98090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981067f0 con 0x7f6b98187e20 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98107350 con 0x7f6b981a0250 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9e4b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981042d0 con 0x7f6b98090420 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45558/0 (socket says 172.21.2.117:45558) 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b97fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 0x7f6b98106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b977fe700 1 -- 172.21.2.117:0/2197849766 learned_addr learned my addr 172.21.2.117:0/2197849766 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.822 7f6b9cad5700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b98090420 0x7f6b98090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 -- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b98090420 msgr2=0x7f6b98090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b98090420 0x7f6b98090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 -- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 msgr2=0x7f6b98106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 0x7f6b98106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 -- 172.21.2.117:0/2197849766 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98090e40 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b97fff700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 0x7f6b98106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b977fe700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f6b8c006f60 tx=0x7f6b8c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8c00bd60 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c000c90 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b8c002220 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b98090e40 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.823 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b98090e40 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.824 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b8c0052e0 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.824 7f6b94ff9700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.824 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6b8c0328d0 con 0x7f6b98187e20 2021-02-10T11:36:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.824 7f6b97fff700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:44.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.824 7f6b97fff700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 0x7f6b7801abc0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:44.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.855 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b98103280 con 0x7f6b98187e20 2021-02-10T11:36:44.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.857 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6b8c0052e0 con 0x7f6b98187e20 2021-02-10T11:36:44.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.984 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802522", "target": ["mgr", ""]}) v1 -- 0x7f6b98103280 con 0x7f6b780186c0 2021-02-10T11:36:44.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.987 7f6b94ff9700 1 -- 172.21.2.117:0/2197849766 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f6b88000960 con 0x7f6b780186c0 2021-02-10T11:36:44.974 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73 2021-02-10T11:36:44.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.993 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 msgr2=0x7f6b7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.993 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 0x7f6b7801abc0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.993 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 msgr2=0x7f6b98188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.993 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7f6b8c006f60 tx=0x7f6b8c00b970).stop 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 shutdown_connections 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b780186c0 0x7f6b7801abc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b98187e20 0x7f6b98188220 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b98090420 0x7f6b98090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 --2- 172.21.2.117:0/2197849766 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b981a0250 0x7f6b98106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 >> 172.21.2.117:0/2197849766 conn(0x7f6b980a1850 msgr2=0x7f6b980c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 shutdown_connections 2021-02-10T11:36:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:44.994 7f6b9e4b8700 1 -- 172.21.2.117:0/2197849766 wait complete. 2021-02-10T11:36:44.984 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802522 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73 2021-02-10T11:36:44.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:44.984 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:44.984 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:45.321 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:45.322 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:45.322 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:45.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252885 s, 41.5 MB/s 2021-02-10T11:36:45.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:45.706 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:45.706 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:46.042 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:46.043 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:46.043 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:46.418 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232246 s, 45.1 MB/s 2021-02-10T11:36:46.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:46.419 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:46.420 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:46.755 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:46.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:46.756 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:47.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:47.134 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:47.134 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244259 s, 42.9 MB/s 2021-02-10T11:36:47.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:47.135 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:47.135 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:47.470 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:47.470 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:47.470 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:47.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:47.844 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:47.844 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246392 s, 42.6 MB/s 2021-02-10T11:36:47.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:47.845 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:47.845 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:48.182 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:48.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:48.183 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:48.557 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:48.558 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:48.558 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022757 s, 46.1 MB/s 2021-02-10T11:36:48.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:48.559 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:48.559 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:48.894 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:48.894 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:48.894 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:49.269 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224105 s, 46.8 MB/s 2021-02-10T11:36:49.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:49.270 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:49.270 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:49.605 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:49.605 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:49.605 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:49.982 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:49.982 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:49.983 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248329 s, 42.2 MB/s 2021-02-10T11:36:49.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:49.984 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:49.984 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:50.319 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:50.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:50.320 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:50.695 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:50.695 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:50.695 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230694 s, 45.5 MB/s 2021-02-10T11:36:50.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:50.696 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:50.696 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:51.033 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:51.033 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:51.034 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:51.409 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:51.410 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:51.410 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239269 s, 43.8 MB/s 2021-02-10T11:36:51.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:51.411 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:51.411 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:51.746 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:51.746 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:51.747 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802522/8c1bb692-449e-4554-9771-70e6d42abb73/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:52.121 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:52.121 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:52.122 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224865 s, 46.6 MB/s 2021-02-10T11:36:52.123 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802523 2021-02-10T11:36:52.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.555 7efee02f9700 1 Processor -- start 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.555 7efee02f9700 1 -- start start 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efed807da20 0x7efed80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efed80e51a0 0x7efed8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed8077a70 con 0x7efed80e4da0 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80e5c20 con 0x7efed80e51a0 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed80777d0 con 0x7efed807da20 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efed80e51a0 0x7efed8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efed80e51a0 0x7efed8079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45566/0 (socket says 172.21.2.117:45566) 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 -- 172.21.2.117:0/2356195936 learned_addr learned my addr 172.21.2.117:0/2356195936 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:52.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede115700 1 --2- 172.21.2.117:0/2356195936 >> v2:172.21.2.125:3300/0 conn(0x7efed807da20 0x7efed80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 -- 172.21.2.117:0/2356195936 >> v2:172.21.2.125:3300/0 conn(0x7efed807da20 msgr2=0x7efed80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 --2- 172.21.2.117:0/2356195936 >> v2:172.21.2.125:3300/0 conn(0x7efed807da20 0x7efed80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 -- 172.21.2.117:0/2356195936 >> v1:172.21.2.117:6789/0 conn(0x7efed80e4da0 legacy=0x7efed81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 -- 172.21.2.117:0/2356195936 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed81b08e0 con 0x7efed80e51a0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efede916700 1 --2- 172.21.2.117:0/2356195936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80e51a0 0x7efed8079000 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7efed0006f00 tx=0x7efed0009f10).ready entity=mon.0 client_cookie=20df333637ba14a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efedd113700 1 -- 172.21.2.117:0/2356195936 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed000b3b0 con 0x7efed80e51a0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efedd113700 1 -- 172.21.2.117:0/2356195936 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000b630 con 0x7efed80e51a0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efedd113700 1 -- 172.21.2.117:0/2356195936 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed000bb50 con 0x7efed80e51a0 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 -- 172.21.2.117:0/2356195936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80e51a0 msgr2=0x7efed8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.556 7efee02f9700 1 --2- 172.21.2.117:0/2356195936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80e51a0 0x7efed8079000 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7efed0006f00 tx=0x7efed0009f10).stop 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 -- 172.21.2.117:0/2356195936 shutdown_connections 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 --2- 172.21.2.117:0/2356195936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80e51a0 0x7efed8079000 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 --2- 172.21.2.117:0/2356195936 >> v2:172.21.2.125:3300/0 conn(0x7efed807da20 0x7efed80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 -- 172.21.2.117:0/2356195936 >> 172.21.2.117:0/2356195936 conn(0x7efed80ce450 msgr2=0x7efed80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 -- 172.21.2.117:0/2356195936 shutdown_connections 2021-02-10T11:36:52.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.557 7efee02f9700 1 -- 172.21.2.117:0/2356195936 wait complete. 2021-02-10T11:36:52.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.571 7efee02f9700 1 Processor -- start 2021-02-10T11:36:52.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.571 7efee02f9700 1 -- start start 2021-02-10T11:36:52.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efed819a4a0 0x7efed80b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efed80b9e60 0x7efed80a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed80a4220 con 0x7efed80a3800 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed80a4410 con 0x7efed819a4a0 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efee02f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed80ba910 con 0x7efed80b9e60 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45576/0 (socket says 172.21.2.117:45576) 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 -- 172.21.2.117:0/3506823011 learned_addr learned my addr 172.21.2.117:0/3506823011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede115700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efed819a4a0 0x7efed80b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efedd914700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efed80b9e60 0x7efed80a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 -- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efed80b9e60 msgr2=0x7efed80a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efed80b9e60 0x7efed80a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 -- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efed819a4a0 msgr2=0x7efed80b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efed819a4a0 0x7efed80b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.572 7efede916700 1 -- 172.21.2.117:0/3506823011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed80bab30 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efede916700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7efed000b9b0 tx=0x7efed000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed0005820 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed0005aa0 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efee02f9700 1 -- 172.21.2.117:0/3506823011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed80bab30 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efee02f9700 1 -- 172.21.2.117:0/3506823011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed80bb0c0 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.573 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed0006030 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.574 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efed0006500 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.574 7efecaffd700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 0x7efec001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.574 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efed0032c90 con 0x7efed80a3800 2021-02-10T11:36:52.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.574 7efede115700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 0x7efec001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:52.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.574 7efede115700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 0x7efec001ac10 crc :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:52.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.605 7efee02f9700 1 -- 172.21.2.117:0/3506823011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed80ba3e0 con 0x7efed80a3800 2021-02-10T11:36:52.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.608 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efed00062f0 con 0x7efed80a3800 2021-02-10T11:36:52.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.734 7efee02f9700 1 -- 172.21.2.117:0/3506823011 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802523", "target": ["mgr", ""]}) v1 -- 0x7efed80ba3e0 con 0x7efec0018730 2021-02-10T11:36:52.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.759 7efecaffd700 1 -- 172.21.2.117:0/3506823011 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efed40009b0 con 0x7efec0018730 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 -- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 msgr2=0x7efec001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 0x7efec001ac10 crc :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 -- 172.21.2.117:0/3506823011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 msgr2=0x7efed80a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0x7efed000b9b0 tx=0x7efed000bd50).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 -- 172.21.2.117:0/3506823011 shutdown_connections 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efec0018730 0x7efec001ac10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efed80a3800 0x7efed80a3c00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efed80b9e60 0x7efed80a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 --2- 172.21.2.117:0/3506823011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efed819a4a0 0x7efed80b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:52.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.767 7efee02f9700 1 -- 172.21.2.117:0/3506823011 >> 172.21.2.117:0/3506823011 conn(0x7efed8107b60 msgr2=0x7efed80c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:52.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.768 7efee02f9700 1 -- 172.21.2.117:0/3506823011 shutdown_connections 2021-02-10T11:36:52.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:52.768 7efee02f9700 1 -- 172.21.2.117:0/3506823011 wait complete. 2021-02-10T11:36:52.758 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802523 2021-02-10T11:36:53.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 Processor -- start 2021-02-10T11:36:53.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 -- start start 2021-02-10T11:36:53.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f985c079000 0x7f985c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 0x7f985c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c077a70 con 0x7f985c07da20 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c0e5c20 con 0x7f985c079000 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.190 7f986329e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c0e53c0 con 0x7f985c1b00f0 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f985c079000 0x7f985c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98610ba700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f985c07da20 0x7f985c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45830/0 (socket says 172.21.2.117:45830) 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98610ba700 1 -- 172.21.2.117:0/654378938 learned_addr learned my addr 172.21.2.117:0/654378938 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98618bb700 1 --2- 172.21.2.117:0/654378938 >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 0x7f985c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 -- 172.21.2.117:0/654378938 >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 msgr2=0x7f985c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 --2- 172.21.2.117:0/654378938 >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 0x7f985c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 -- 172.21.2.117:0/654378938 >> v1:172.21.2.117:6789/0 conn(0x7f985c07da20 legacy=0x7f985c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 -- 172.21.2.117:0/654378938 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c079400 con 0x7f985c079000 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98618bb700 1 --2- 172.21.2.117:0/654378938 >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 0x7f985c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f98608b9700 1 --2- 172.21.2.117:0/654378938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c079000 0x7f985c0e4da0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f9858006650 tx=0x7f9858009a80).ready entity=mon.0 client_cookie=33ff62d637670e24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f984ffff700 1 -- 172.21.2.117:0/654378938 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98580009b0 con 0x7f985c079000 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f984ffff700 1 -- 172.21.2.117:0/654378938 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985800c0c0 con 0x7f985c079000 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.191 7f984ffff700 1 -- 172.21.2.117:0/654378938 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f985800c520 con 0x7f985c079000 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 -- 172.21.2.117:0/654378938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c079000 msgr2=0x7f985c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 --2- 172.21.2.117:0/654378938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c079000 0x7f985c0e4da0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f9858006650 tx=0x7f9858009a80).stop 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 -- 172.21.2.117:0/654378938 shutdown_connections 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 --2- 172.21.2.117:0/654378938 >> v2:172.21.2.125:3300/0 conn(0x7f985c1b00f0 0x7f985c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 --2- 172.21.2.117:0/654378938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c079000 0x7f985c0e4da0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 -- 172.21.2.117:0/654378938 >> 172.21.2.117:0/654378938 conn(0x7f985c0ce450 msgr2=0x7f985c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 -- 172.21.2.117:0/654378938 shutdown_connections 2021-02-10T11:36:53.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.192 7f986329e700 1 -- 172.21.2.117:0/654378938 wait complete. 2021-02-10T11:36:53.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 Processor -- start 2021-02-10T11:36:53.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 -- start start 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f985c1a0250 0x7f985c106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 0x7f985c188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1067f0 con 0x7f985c090420 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c107350 con 0x7f985c187e20 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f986329e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1042d0 con 0x7f985c1a0250 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.207 7f98618bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45586/0 (socket says 172.21.2.117:45586) 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 -- 172.21.2.117:0/1044110690 learned_addr learned my addr 172.21.2.117:0/1044110690 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98610ba700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f985c1a0250 0x7f985c106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98608b9700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 0x7f985c188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 -- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f985c1a0250 msgr2=0x7f985c106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f985c1a0250 0x7f985c106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 -- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 msgr2=0x7f985c188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 0x7f985c188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 -- 172.21.2.117:0/1044110690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c090e40 con 0x7f985c090420 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98608b9700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 0x7f985c188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f98618bb700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f985400b970 tx=0x7f985400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9854000f90 con 0x7f985c090420 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854006aa0 con 0x7f985c090420 2021-02-10T11:36:53.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f986329e700 1 -- 172.21.2.117:0/1044110690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c090e40 con 0x7f985c090420 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f986329e700 1 -- 172.21.2.117:0/1044110690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c090e40 con 0x7f985c090420 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.208 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98540015f0 con 0x7f985c090420 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.209 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9854000e70 con 0x7f985c090420 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.209 7f984dffb700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 0x7f983c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.209 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9854032c90 con 0x7f985c090420 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.209 7f98610ba700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 0x7f983c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:36:53.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.209 7f98610ba700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 0x7f983c01ac10 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:36:53.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.240 7f986329e700 1 -- 172.21.2.117:0/1044110690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c103280 con 0x7f985c090420 2021-02-10T11:36:53.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.243 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9854006aa0 con 0x7f985c090420 2021-02-10T11:36:53.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.371 7f986329e700 1 -- 172.21.2.117:0/1044110690 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802523", "target": ["mgr", ""]}) v1 -- 0x7f985c103280 con 0x7f983c018730 2021-02-10T11:36:53.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.374 7f984dffb700 1 -- 172.21.2.117:0/1044110690 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9850001a00 con 0x7f983c018730 2021-02-10T11:36:53.356 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 -- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 msgr2=0x7f983c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 0x7f983c01ac10 crc :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 -- 172.21.2.117:0/1044110690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 msgr2=0x7f985c090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f985400b970 tx=0x7f985400bd10).stop 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 -- 172.21.2.117:0/1044110690 shutdown_connections 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f983c018730 0x7f983c01ac10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f985c090420 0x7f985c090820 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f985c187e20 0x7f985c188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 --2- 172.21.2.117:0/1044110690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f985c1a0250 0x7f985c106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:36:53.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.375 7f986329e700 1 -- 172.21.2.117:0/1044110690 >> 172.21.2.117:0/1044110690 conn(0x7f985c0a1850 msgr2=0x7f985c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:36:53.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.376 7f986329e700 1 -- 172.21.2.117:0/1044110690 shutdown_connections 2021-02-10T11:36:53.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:36:53.376 7f986329e700 1 -- 172.21.2.117:0/1044110690 wait complete. 2021-02-10T11:36:53.366 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802523 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e 2021-02-10T11:36:53.366 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:53.366 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:53.366 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:53.702 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:53.703 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:53.703 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:54.086 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:54.086 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:54.087 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024599 s, 42.6 MB/s 2021-02-10T11:36:54.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:54.088 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:54.088 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:54.424 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:54.424 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:54.424 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:54.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:54.800 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:54.800 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232225 s, 45.2 MB/s 2021-02-10T11:36:54.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:54.801 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:54.802 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:55.138 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:55.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:55.139 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:55.515 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:55.515 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:55.516 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240652 s, 43.6 MB/s 2021-02-10T11:36:55.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:55.516 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:55.517 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:55.852 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:55.852 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:55.853 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:56.229 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:56.229 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:56.229 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238709 s, 43.9 MB/s 2021-02-10T11:36:56.231 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:56.231 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:56.231 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:56.567 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:56.567 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:56.568 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:56.946 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:56.946 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:56.946 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240287 s, 43.6 MB/s 2021-02-10T11:36:56.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:56.947 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:56.947 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:57.283 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:57.283 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:57.283 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:57.661 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0281765 s, 37.2 MB/s 2021-02-10T11:36:57.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:57.662 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:57.662 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:57.997 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:57.997 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:57.997 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:58.375 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:58.375 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:58.375 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250539 s, 41.9 MB/s 2021-02-10T11:36:58.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:58.376 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:58.376 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:58.713 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:58.713 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:58.714 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:59.090 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234542 s, 44.7 MB/s 2021-02-10T11:36:59.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:59.092 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:59.092 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:36:59.427 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:36:59.427 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:36:59.428 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:36:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:36:59.806 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:36:59.806 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254327 s, 41.2 MB/s 2021-02-10T11:36:59.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:36:59.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:36:59.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:00.143 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:00.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:00.143 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802523/41979c35-7be2-4929-8452-f0c3cddb248e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:00.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:00.519 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:00.519 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235133 s, 44.6 MB/s 2021-02-10T11:37:00.520 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802524 2021-02-10T11:37:00.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.953 7f5625717700 1 Processor -- start 2021-02-10T11:37:00.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.953 7f5625717700 1 -- start start 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f5625717700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 0x7f56200771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f5625717700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56200d1d20 con 0x7f56200e51a0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f5625717700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620077a70 con 0x7f56200e4da0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f5625717700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56200777d0 con 0x7f562007da20 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f561fac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f56200e51a0 0x7f56200d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45844/0 (socket says 172.21.2.117:45844) 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f561fac0700 1 -- 172.21.2.117:0/2404634847 learned_addr learned my addr 172.21.2.117:0/2404634847 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.954 7f561f2bf700 1 --2- 172.21.2.117:0/2404634847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 0x7f56200771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568540967 0 0) 0x7f5618001500 con 0x7f56200e51a0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5608002fe0 con 0x7f56200e51a0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210398577 0 0) 0x7f5614000ff0 con 0x7f56200e4da0 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561f2bf700 1 -- 172.21.2.117:0/2404634847 >> v1:172.21.2.125:6789/0 conn(0x7f56200e4da0 legacy=0x7f56201b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561f2bf700 1 -- 172.21.2.117:0/2404634847 >> v1:172.21.2.117:6789/0 conn(0x7f56200e51a0 legacy=0x7f56200d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561f2bf700 1 -- 172.21.2.117:0/2404634847 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56201b08e0 con 0x7f562007da20 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561f2bf700 1 --2- 172.21.2.117:0/2404634847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 0x7f56200771b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5610006650 tx=0x7f5610009a80).ready entity=mon.2 client_cookie=c1b03d172c37c4d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56100009b0 con 0x7f562007da20 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561000c0c0 con 0x7f562007da20 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.955 7f561e2bd700 1 -- 172.21.2.117:0/2404634847 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561000c5e0 con 0x7f562007da20 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 -- 172.21.2.117:0/2404634847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 msgr2=0x7f56200771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 --2- 172.21.2.117:0/2404634847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 0x7f56200771b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5610006650 tx=0x7f5610009a80).stop 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 -- 172.21.2.117:0/2404634847 shutdown_connections 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 --2- 172.21.2.117:0/2404634847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562007da20 0x7f56200771b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 -- 172.21.2.117:0/2404634847 >> 172.21.2.117:0/2404634847 conn(0x7f56200ce450 msgr2=0x7f56200e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 -- 172.21.2.117:0/2404634847 shutdown_connections 2021-02-10T11:37:00.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.956 7f5625717700 1 -- 172.21.2.117:0/2404634847 wait complete. 2021-02-10T11:37:00.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.970 7f5625717700 1 Processor -- start 2021-02-10T11:37:00.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 -- start start 2021-02-10T11:37:00.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56200c33d0 0x7f56200b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5620090de0 0x7f56200911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5620090600 con 0x7f56200b3ff0 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5620090860 con 0x7f56200c33d0 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f5625717700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56200b3c70 con 0x7f5620090de0 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45598/0 (socket says 172.21.2.117:45598) 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 -- 172.21.2.117:0/2946263209 learned_addr learned my addr 172.21.2.117:0/2946263209 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 -- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5620090de0 msgr2=0x7f56200911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5620090de0 0x7f56200911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 -- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56200c33d0 msgr2=0x7f56200b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56200c33d0 0x7f56200b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.971 7f561f2bf700 1 -- 172.21.2.117:0/2946263209 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5620091800 con 0x7f56200b3ff0 2021-02-10T11:37:00.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f561f2bf700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f56100096a0 tx=0x7f561000b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5610000940 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56100093a0 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5610002220 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5625717700 1 -- 172.21.2.117:0/2946263209 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5620091800 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5625717700 1 -- 172.21.2.117:0/2946263209 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5620091800 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56100052e0 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 0x7f560001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.972 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5610032660 con 0x7f56200b3ff0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.973 7f561eabe700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 0x7f560001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:00.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:00.973 7f561eabe700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 0x7f560001abc0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:00.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.004 7f5625717700 1 -- 172.21.2.117:0/2946263209 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5620103370 con 0x7f56200b3ff0 2021-02-10T11:37:00.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.006 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5610000c90 con 0x7f56200b3ff0 2021-02-10T11:37:01.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.132 7f5625717700 1 -- 172.21.2.117:0/2946263209 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802524", "target": ["mgr", ""]}) v1 -- 0x7f5620077a70 con 0x7f56000186c0 2021-02-10T11:37:01.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.158 7f5607fff700 1 -- 172.21.2.117:0/2946263209 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5614007e00 con 0x7f56000186c0 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 -- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 msgr2=0x7f560001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 0x7f560001abc0 crc :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 -- 172.21.2.117:0/2946263209 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 msgr2=0x7f56200b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f56100096a0 tx=0x7f561000b9b0).stop 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 -- 172.21.2.117:0/2946263209 shutdown_connections 2021-02-10T11:37:01.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5620090de0 0x7f56200911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56200c33d0 0x7f56200b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56000186c0 0x7f560001abc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 --2- 172.21.2.117:0/2946263209 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56200b3ff0 0x7f56200b3030 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.165 7f5625717700 1 -- 172.21.2.117:0/2946263209 >> 172.21.2.117:0/2946263209 conn(0x7f56200a1420 msgr2=0x7f562009efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.166 7f5625717700 1 -- 172.21.2.117:0/2946263209 shutdown_connections 2021-02-10T11:37:01.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.166 7f5625717700 1 -- 172.21.2.117:0/2946263209 wait complete. 2021-02-10T11:37:01.156 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802524 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 Processor -- start 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 -- start start 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac4077a70 con 0x7f0ac40e4da0 2021-02-10T11:37:01.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac40e5c20 con 0x7f0ac40e51a0 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0acbbac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac40777d0 con 0x7f0ac407da20 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0aca1c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0ac99c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0aca1c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45606/0 (socket says 172.21.2.117:45606) 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0ac99c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43286/0 (socket says 172.21.2.117:43286) 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.591 7f0aca1c9700 1 -- 172.21.2.117:0/1005606484 learned_addr learned my addr 172.21.2.117:0/1005606484 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac99c8700 1 -- 172.21.2.117:0/1005606484 >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 msgr2=0x7f0ac40d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac99c8700 1 --2- 172.21.2.117:0/1005606484 >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac99c8700 1 -- 172.21.2.117:0/1005606484 >> v1:172.21.2.117:6789/0 conn(0x7f0ac40e4da0 legacy=0x7f0ac41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac99c8700 1 -- 172.21.2.117:0/1005606484 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac41b08e0 con 0x7f0ac407da20 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0aca1c9700 1 --2- 172.21.2.117:0/1005606484 >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac99c8700 1 --2- 172.21.2.117:0/1005606484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0ac0006650 tx=0x7f0ac0009a80).ready entity=mon.2 client_cookie=e2e8d262baf8f2dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac89c6700 1 -- 172.21.2.117:0/1005606484 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac00009b0 con 0x7f0ac407da20 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac89c6700 1 -- 172.21.2.117:0/1005606484 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac000c0c0 con 0x7f0ac407da20 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.592 7f0ac89c6700 1 -- 172.21.2.117:0/1005606484 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac000c510 con 0x7f0ac407da20 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 -- 172.21.2.117:0/1005606484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 msgr2=0x7f0ac40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 --2- 172.21.2.117:0/1005606484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0ac0006650 tx=0x7f0ac0009a80).stop 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 -- 172.21.2.117:0/1005606484 shutdown_connections 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 --2- 172.21.2.117:0/1005606484 >> v2:172.21.2.117:3300/0 conn(0x7f0ac40e51a0 0x7f0ac40d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 --2- 172.21.2.117:0/1005606484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac407da20 0x7f0ac40771b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 -- 172.21.2.117:0/1005606484 >> 172.21.2.117:0/1005606484 conn(0x7f0ac40ce450 msgr2=0x7f0ac40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 -- 172.21.2.117:0/1005606484 shutdown_connections 2021-02-10T11:37:01.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.593 7f0acbbac700 1 -- 172.21.2.117:0/1005606484 wait complete. 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 Processor -- start 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 -- start start 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac41b5130 0x7f0ac41b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ac41b5b50 0x7f0ac40b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac40b8da0 con 0x7f0ac41b5f50 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac40a4140 con 0x7f0ac41b5b50 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.608 7f0acbbac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac40a4320 con 0x7f0ac41b5130 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45610/0 (socket says 172.21.2.117:45610) 2021-02-10T11:37:01.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 -- 172.21.2.117:0/1644457124 learned_addr learned my addr 172.21.2.117:0/1644457124 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 -- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac41b5130 msgr2=0x7f0ac41b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac41b5130 0x7f0ac41b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 -- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ac41b5b50 msgr2=0x7f0ac40b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ac41b5b50 0x7f0ac40b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 -- 172.21.2.117:0/1644457124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac40b62b0 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0aca1c9700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f0abc00b940 tx=0x7f0abc00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0abc000f90 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc003030 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac40b62b0 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0abc001460 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.609 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac40b62b0 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.610 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0abc001270 con 0x7f0ac41b5f50 2021-02-10T11:37:01.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.610 7f0ab67fc700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 0x7f0aac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:01.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.610 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0abc032cc0 con 0x7f0ac41b5f50 2021-02-10T11:37:01.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.610 7f0ac99c8700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 0x7f0aac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:01.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.611 7f0ac99c8700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 0x7f0aac01abd0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:01.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.641 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac41b4f00 con 0x7f0ac41b5f50 2021-02-10T11:37:01.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.644 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0abc003010 con 0x7f0ac41b5f50 2021-02-10T11:37:01.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.771 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802524", "target": ["mgr", ""]}) v1 -- 0x7f0ac4077a70 con 0x7f0aac0186d0 2021-02-10T11:37:01.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.774 7f0ab67fc700 1 -- 172.21.2.117:0/1644457124 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f0ac0006060 con 0x7f0aac0186d0 2021-02-10T11:37:01.761 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.780 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 msgr2=0x7f0aac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 0x7f0aac01abd0 crc :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 msgr2=0x7f0ac40b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f0abc00b940 tx=0x7f0abc00bce0).stop 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 shutdown_connections 2021-02-10T11:37:01.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0aac0186d0 0x7f0aac01abd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ac41b5f50 0x7f0ac40b5c90 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ac41b5b50 0x7f0ac40b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 --2- 172.21.2.117:0/1644457124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ac41b5130 0x7f0ac41b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 >> 172.21.2.117:0/1644457124 conn(0x7f0ac408bbc0 msgr2=0x7f0ac40c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 shutdown_connections 2021-02-10T11:37:01.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:01.781 7f0acbbac700 1 -- 172.21.2.117:0/1644457124 wait complete. 2021-02-10T11:37:01.771 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802524 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b 2021-02-10T11:37:01.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:01.772 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:01.772 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:02.108 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:02.108 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:02.108 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:02.483 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:02.483 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:02.483 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224697 s, 46.7 MB/s 2021-02-10T11:37:02.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:02.484 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:02.484 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:02.820 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:02.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:02.820 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:03.046 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:03.049 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:03.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255178 s, 41.1 MB/s 2021-02-10T11:37:03.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:03.265 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:03.265 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:03.611 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:03.611 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:03.612 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:03.998 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:03.999 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:03.999 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251385 s, 41.7 MB/s 2021-02-10T11:37:04.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:04.000 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:04.000 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:04.336 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:04.336 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:04.336 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:04.712 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:04.712 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:04.712 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025006 s, 41.9 MB/s 2021-02-10T11:37:04.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:04.713 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:04.713 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:05.048 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:05.048 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:05.048 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:05.426 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:05.426 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:05.427 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024846 s, 42.2 MB/s 2021-02-10T11:37:05.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:05.428 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:05.428 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:05.765 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:05.765 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:05.766 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:06.140 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:06.140 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:06.141 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230283 s, 45.5 MB/s 2021-02-10T11:37:06.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.142 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:06.142 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:06.476 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:06.477 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:06.477 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236102 s, 44.4 MB/s 2021-02-10T11:37:06.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:06.854 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:06.854 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:07.190 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:07.190 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:07.190 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:07.559 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:07.560 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:07.560 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231037 s, 45.4 MB/s 2021-02-10T11:37:07.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:07.561 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:07.561 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:07.896 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:07.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:07.896 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:08.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:08.275 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:08.275 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260372 s, 40.3 MB/s 2021-02-10T11:37:08.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:08.276 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:08.276 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:08.613 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:08.613 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:08.614 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802524/e37e282e-10d5-4dfd-9050-d7e20b800e8b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225307 s, 46.5 MB/s 2021-02-10T11:37:08.990 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802525 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.423 7fe17db7a700 1 Processor -- start 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 -- start start 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 0x7fe1780e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe1781b00f0 0x7fe1781b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe178077a70 con 0x7fe17807da20 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1780e53c0 con 0x7fe1781b00f0 2021-02-10T11:37:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe17db7a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe178079400 con 0x7fe178079000 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe1777fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe17807da20 0x7fe1780771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45866/0 (socket says 172.21.2.117:45866) 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe1777fe700 1 -- 172.21.2.117:0/3468054252 learned_addr learned my addr 172.21.2.117:0/3468054252 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe176ffd700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 0x7fe1780e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.424 7fe177fff700 1 --2- 172.21.2.117:0/3468054252 >> v2:172.21.2.125:3300/0 conn(0x7fe1781b00f0 0x7fe1781b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe1767fc700 1 -- 172.21.2.117:0/3468054252 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550431826 0 0) 0x7fe168000ce0 con 0x7fe17807da20 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe1767fc700 1 -- 172.21.2.117:0/3468054252 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164002fe0 con 0x7fe17807da20 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe177fff700 1 -- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 msgr2=0x7fe1780e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe177fff700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 0x7fe1780e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe177fff700 1 -- 172.21.2.117:0/3468054252 >> v1:172.21.2.117:6789/0 conn(0x7fe17807da20 legacy=0x7fe1780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe177fff700 1 -- 172.21.2.117:0/3468054252 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17807de20 con 0x7fe1781b00f0 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe176ffd700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 0x7fe1780e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.425 7fe177fff700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1781b00f0 0x7fe1781b04f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe170006e50 tx=0x7fe170009e70).ready entity=mon.1 client_cookie=9e7c1276cd52a982 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe1767fc700 1 -- 172.21.2.117:0/3468054252 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17000b510 con 0x7fe1781b00f0 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe1767fc700 1 -- 172.21.2.117:0/3468054252 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe170006790 con 0x7fe1781b00f0 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe1767fc700 1 -- 172.21.2.117:0/3468054252 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17000b940 con 0x7fe1781b00f0 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 -- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1781b00f0 msgr2=0x7fe1781b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1781b00f0 0x7fe1781b04f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe170006e50 tx=0x7fe170009e70).stop 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 -- 172.21.2.117:0/3468054252 shutdown_connections 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1781b00f0 0x7fe1781b04f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 --2- 172.21.2.117:0/3468054252 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe178079000 0x7fe1780e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 -- 172.21.2.117:0/3468054252 >> 172.21.2.117:0/3468054252 conn(0x7fe1780ce450 msgr2=0x7fe1780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 -- 172.21.2.117:0/3468054252 shutdown_connections 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.426 7fe17db7a700 1 -- 172.21.2.117:0/3468054252 wait complete. 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 Processor -- start 2021-02-10T11:37:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 -- start start 2021-02-10T11:37:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe17808a180 0x7fe17809b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 0x7fe17808ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17808a720 con 0x7fe178190110 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780bb540 con 0x7fe17808a180 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe17db7a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1780bb9e0 con 0x7fe1781a7860 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe176ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.441 7fe176ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45626/0 (socket says 172.21.2.117:45626) 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 -- 172.21.2.117:0/653256538 learned_addr learned my addr 172.21.2.117:0/653256538 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe177fff700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 0x7fe17808ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 -- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 msgr2=0x7fe17808ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 0x7fe17808ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 -- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe17808a180 msgr2=0x7fe17809b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe17808a180 0x7fe17809b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 -- 172.21.2.117:0/653256538 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1781a7c60 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe177fff700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 0x7fe17808ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe176ffd700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe160006f40 tx=0x7fe16000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe16000bd80 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160000c90 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe160002220 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe17db7a700 1 -- 172.21.2.117:0/653256538 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1781a7c60 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.442 7fe17db7a700 1 -- 172.21.2.117:0/653256538 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17808b240 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.443 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe16000a000 con 0x7fe178190110 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.443 7fe16ffff700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 0x7fe15c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:09.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.443 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe1600326f0 con 0x7fe178190110 2021-02-10T11:37:09.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.444 7fe1777fe700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 0x7fe15c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:09.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.445 7fe1777fe700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 0x7fe15c01abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:09.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.474 7fe17db7a700 1 -- 172.21.2.117:0/653256538 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178077a70 con 0x7fe178190110 2021-02-10T11:37:09.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.477 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe160000940 con 0x7fe178190110 2021-02-10T11:37:09.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.600 7fe17db7a700 1 -- 172.21.2.117:0/653256538 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802525", "target": ["mgr", ""]}) v1 -- 0x7fe17808b240 con 0x7fe15c0186c0 2021-02-10T11:37:09.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.627 7fe16ffff700 1 -- 172.21.2.117:0/653256538 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe1680077a0 con 0x7fe15c0186c0 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 msgr2=0x7fe15c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 0x7fe15c01abc0 crc :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 msgr2=0x7fe1781a7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fe160006f40 tx=0x7fe16000b990).stop 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 shutdown_connections 2021-02-10T11:37:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe15c0186c0 0x7fe15c01abc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe178190110 0x7fe1781a7240 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1781a7860 0x7fe17808ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 --2- 172.21.2.117:0/653256538 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe17808a180 0x7fe17809b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 >> 172.21.2.117:0/653256538 conn(0x7fe1780a1360 msgr2=0x7fe178091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 shutdown_connections 2021-02-10T11:37:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:09.634 7fe17db7a700 1 -- 172.21.2.117:0/653256538 wait complete. 2021-02-10T11:37:09.624 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802525 2021-02-10T11:37:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 Processor -- start 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 -- start start 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38e80d1510 0x7f38e80d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e8077a70 con 0x7f38e807da20 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e8079400 con 0x7f38e8079000 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38efd89700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e807de20 con 0x7f38e80d1510 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38ee3a6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38e80d1510 0x7f38e80d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38ee3a6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38e80d1510 0x7f38e80d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45634/0 (socket says 172.21.2.117:45634) 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.059 7f38ee3a6700 1 -- 172.21.2.117:0/3351531149 learned_addr learned my addr 172.21.2.117:0/3351531149 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ee3a6700 1 -- 172.21.2.117:0/3351531149 >> v1:172.21.2.125:6789/0 conn(0x7f38e8079000 legacy=0x7f38e80e4d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ee3a6700 1 -- 172.21.2.117:0/3351531149 >> v1:172.21.2.117:6789/0 conn(0x7f38e807da20 legacy=0x7f38e80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ee3a6700 1 -- 172.21.2.117:0/3351531149 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e81b00d0 con 0x7f38e80d1510 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ee3a6700 1 --2- 172.21.2.117:0/3351531149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80d1510 0x7f38e80d1910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f38e0006f00 tx=0x7f38e0009f10).ready entity=mon.0 client_cookie=a31b816db93126a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ecba3700 1 -- 172.21.2.117:0/3351531149 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38e0003030 con 0x7f38e80d1510 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ecba3700 1 -- 172.21.2.117:0/3351531149 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e000b3b0 con 0x7f38e80d1510 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38ecba3700 1 -- 172.21.2.117:0/3351531149 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38e000b8d0 con 0x7f38e80d1510 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38efd89700 1 -- 172.21.2.117:0/3351531149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80d1510 msgr2=0x7f38e80d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38efd89700 1 --2- 172.21.2.117:0/3351531149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80d1510 0x7f38e80d1910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f38e0006f00 tx=0x7f38e0009f10).stop 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38efd89700 1 -- 172.21.2.117:0/3351531149 shutdown_connections 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38efd89700 1 --2- 172.21.2.117:0/3351531149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80d1510 0x7f38e80d1910 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.060 7f38efd89700 1 -- 172.21.2.117:0/3351531149 >> 172.21.2.117:0/3351531149 conn(0x7f38e80ce430 msgr2=0x7f38e80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.061 7f38efd89700 1 -- 172.21.2.117:0/3351531149 shutdown_connections 2021-02-10T11:37:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.061 7f38efd89700 1 -- 172.21.2.117:0/3351531149 wait complete. 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 Processor -- start 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 -- start start 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e810f510 0x7f38e810f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e810ff30 0x7f38e80c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80ae900 con 0x7f38e80c03a0 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8110330 con 0x7f38e810f510 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38efd89700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80c2180 con 0x7f38e810ff30 2021-02-10T11:37:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38edba5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38edba5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45636/0 (socket says 172.21.2.117:45636) 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.075 7f38edba5700 1 -- 172.21.2.117:0/1330518703 learned_addr learned my addr 172.21.2.117:0/1330518703 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 -- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e810ff30 msgr2=0x7f38e80c0900 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e810ff30 0x7f38e80c0900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 -- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e810f510 msgr2=0x7f38e810f910 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e810f510 0x7f38e810f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 -- 172.21.2.117:0/1330518703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e80c0f20 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38edba5700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f38dc006ed0 tx=0x7f38dc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc006020 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0062a0 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc005730 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38efd89700 1 -- 172.21.2.117:0/1330518703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e80c0f20 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.076 7f38efd89700 1 -- 172.21.2.117:0/1330518703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e80c0f20 con 0x7f38e80c03a0 2021-02-10T11:37:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.077 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38dc0054b0 con 0x7f38e80c03a0 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.077 7f38da7fc700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 0x7f38d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.077 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f38dc032550 con 0x7f38e80c03a0 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.077 7f38efd89700 1 -- 172.21.2.117:0/1330518703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e80ce430 con 0x7f38e80c03a0 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.077 7f38ed3a4700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 0x7f38d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.078 7f38ed3a4700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 0x7f38d001abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:10.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.080 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38dc006020 con 0x7f38e80c03a0 2021-02-10T11:37:10.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.206 7f38efd89700 1 -- 172.21.2.117:0/1330518703 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802525", "target": ["mgr", ""]}) v1 -- 0x7f38e80ce430 con 0x7f38d00186c0 2021-02-10T11:37:10.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.210 7f38da7fc700 1 -- 172.21.2.117:0/1330518703 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f38e4000960 con 0x7f38d00186c0 2021-02-10T11:37:10.196 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 msgr2=0x7f38d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 0x7f38d001abc0 crc :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 msgr2=0x7f38e80ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f38dc006ed0 tx=0x7f38dc00b960).stop 2021-02-10T11:37:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 shutdown_connections 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e810ff30 0x7f38e80c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e810f510 0x7f38e810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38d00186c0 0x7f38d001abc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 --2- 172.21.2.117:0/1330518703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e80c03a0 0x7f38e80ae100 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 >> 172.21.2.117:0/1330518703 conn(0x7f38e81114d0 msgr2=0x7f38e809b130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 shutdown_connections 2021-02-10T11:37:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:10.216 7f38efd89700 1 -- 172.21.2.117:0/1330518703 wait complete. 2021-02-10T11:37:10.206 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802525 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db 2021-02-10T11:37:10.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:10.206 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:10.206 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:10.543 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:10.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:10.544 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:10.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:10.926 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:10.926 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239817 s, 43.7 MB/s 2021-02-10T11:37:10.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:10.927 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:10.927 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:11.262 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:11.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:11.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:11.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:11.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:11.639 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242117 s, 43.3 MB/s 2021-02-10T11:37:11.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:11.640 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:11.640 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:11.975 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:11.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:11.976 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:12.352 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:12.352 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:12.352 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023323 s, 45.0 MB/s 2021-02-10T11:37:12.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:12.353 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:12.353 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:12.689 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:12.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:12.689 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:13.064 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:13.064 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:13.064 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257279 s, 40.8 MB/s 2021-02-10T11:37:13.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:13.066 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:13.066 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:13.403 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:13.403 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:13.404 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:13.778 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233026 s, 45.0 MB/s 2021-02-10T11:37:13.779 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:13.779 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:13.779 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:14.117 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:14.117 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:14.118 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:14.496 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:14.496 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:14.496 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252473 s, 41.5 MB/s 2021-02-10T11:37:14.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:14.497 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:14.497 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:14.834 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:14.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:14.835 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:15.211 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:15.211 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:15.211 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02396 s, 43.8 MB/s 2021-02-10T11:37:15.212 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:15.212 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:15.212 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:15.548 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:15.548 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:15.548 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:15.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:15.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:15.925 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232802 s, 45.0 MB/s 2021-02-10T11:37:15.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:15.926 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:15.926 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:16.262 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:16.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:16.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:16.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:16.639 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:16.639 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240514 s, 43.6 MB/s 2021-02-10T11:37:16.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:16.640 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:16.640 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:16.976 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:16.977 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:16.977 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802525/02d9cf24-638f-41b0-afdf-76003ad824db/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:17.354 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:17.354 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:17.354 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244567 s, 42.9 MB/s 2021-02-10T11:37:17.355 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802526 2021-02-10T11:37:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.787 7fc299090700 1 Processor -- start 2021-02-10T11:37:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.787 7fc299090700 1 -- start start 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.788 7fc299090700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 0x7fc2940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.788 7fc299090700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 0x7fc294079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.788 7fc299090700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc294077a70 con 0x7fc2940e4da0 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.788 7fc299090700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2940777d0 con 0x7fc29407da20 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.788 7fc299090700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2941b08e0 con 0x7fc2940e51a0 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.789 7fc29259c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fc2940e4da0 0x7fc2941b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38822/0 (socket says 172.21.2.117:38822) 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.789 7fc29359e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 0x7fc294079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.789 7fc292d9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 0x7fc2940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.789 7fc29259c700 1 -- 172.21.2.117:0/2895325883 learned_addr learned my addr 172.21.2.117:0/2895325883 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc29359e700 1 -- 172.21.2.117:0/2895325883 >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 msgr2=0x7fc2940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc29359e700 1 --2- 172.21.2.117:0/2895325883 >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 0x7fc2940771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc29359e700 1 -- 172.21.2.117:0/2895325883 >> v1:172.21.2.125:6789/0 conn(0x7fc2940e4da0 legacy=0x7fc2941b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc29359e700 1 -- 172.21.2.117:0/2895325883 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29407de20 con 0x7fc2940e51a0 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc292d9d700 1 --2- 172.21.2.117:0/2895325883 >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 0x7fc2940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc29359e700 1 --2- 172.21.2.117:0/2895325883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 0x7fc294079000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc28c006e50 tx=0x7fc28c009e70).ready entity=mon.2 client_cookie=c163d9c47f4fd729 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc291d9b700 1 -- 172.21.2.117:0/2895325883 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc28c006aa0 con 0x7fc2940e51a0 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc291d9b700 1 -- 172.21.2.117:0/2895325883 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c006790 con 0x7fc2940e51a0 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc291d9b700 1 -- 172.21.2.117:0/2895325883 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc28c00b700 con 0x7fc2940e51a0 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc299090700 1 -- 172.21.2.117:0/2895325883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 msgr2=0x7fc294079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.790 7fc299090700 1 --2- 172.21.2.117:0/2895325883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 0x7fc294079000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc28c006e50 tx=0x7fc28c009e70).stop 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 -- 172.21.2.117:0/2895325883 shutdown_connections 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 --2- 172.21.2.117:0/2895325883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2940e51a0 0x7fc294079000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 --2- 172.21.2.117:0/2895325883 >> v2:172.21.2.125:3300/0 conn(0x7fc29407da20 0x7fc2940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 -- 172.21.2.117:0/2895325883 >> 172.21.2.117:0/2895325883 conn(0x7fc2940ce450 msgr2=0x7fc2940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 -- 172.21.2.117:0/2895325883 shutdown_connections 2021-02-10T11:37:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.791 7fc299090700 1 -- 172.21.2.117:0/2895325883 wait complete. 2021-02-10T11:37:17.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 Processor -- start 2021-02-10T11:37:17.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 -- start start 2021-02-10T11:37:17.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2941b5130 0x7fc2941b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2941b5f50 0x7fc2940b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940b8da0 con 0x7fc2941b5b50 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940a4140 con 0x7fc2941b5f50 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc299090700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940a4320 con 0x7fc2941b5130 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc29259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc29259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45650/0 (socket says 172.21.2.117:45650) 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.806 7fc29259c700 1 -- 172.21.2.117:0/3043023230 learned_addr learned my addr 172.21.2.117:0/3043023230 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29359e700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2941b5f50 0x7fc2940b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc292d9d700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2941b5130 0x7fc2941b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 -- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2941b5130 msgr2=0x7fc2941b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2941b5130 0x7fc2941b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 -- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2941b5f50 msgr2=0x7fc2940b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2941b5f50 0x7fc2940b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 -- 172.21.2.117:0/3043023230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940b62b0 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc29259c700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fc27c001c60 tx=0x7fc27c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c0010b0 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c004140 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc27c004640 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc299090700 1 -- 172.21.2.117:0/3043023230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2940b62b0 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.807 7fc299090700 1 -- 172.21.2.117:0/3043023230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2940b6840 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.808 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc27c0045b0 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.808 7fc28b7fe700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 0x7fc27801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.808 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc27c0326f0 con 0x7fc2941b5b50 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.808 7fc292d9d700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 0x7fc27801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:17.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.809 7fc292d9d700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 0x7fc27801abc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:17.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.837 7fc299090700 1 -- 172.21.2.117:0/3043023230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2941b4f00 con 0x7fc2941b5b50 2021-02-10T11:37:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.840 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc27c004380 con 0x7fc2941b5b50 2021-02-10T11:37:17.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.966 7fc299090700 1 -- 172.21.2.117:0/3043023230 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802526", "target": ["mgr", ""]}) v1 -- 0x7fc2940b8da0 con 0x7fc2780186c0 2021-02-10T11:37:17.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.993 7fc28b7fe700 1 -- 172.21.2.117:0/3043023230 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc284006a30 con 0x7fc2780186c0 2021-02-10T11:37:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 msgr2=0x7fc27801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 0x7fc27801abc0 crc :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 msgr2=0x7fc2940b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7fc27c001c60 tx=0x7fc27c00a8d0).stop 2021-02-10T11:37:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 shutdown_connections 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2780186c0 0x7fc27801abc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2941b5b50 0x7fc2940b5670 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2941b5f50 0x7fc2940b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 --2- 172.21.2.117:0/3043023230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2941b5130 0x7fc2941b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 >> 172.21.2.117:0/3043023230 conn(0x7fc29408bbc0 msgr2=0x7fc2940c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 shutdown_connections 2021-02-10T11:37:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:17.999 7fc299090700 1 -- 172.21.2.117:0/3043023230 wait complete. 2021-02-10T11:37:17.989 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802526 2021-02-10T11:37:18.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.424 7f9a1610b700 1 Processor -- start 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.424 7f9a1610b700 1 -- start start 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a1610b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9a10079000 0x7f9a100e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a1610b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a101b00d0 0x7f9a101b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a1610b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10077a70 con 0x7f9a1007da20 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a1610b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a100e53a0 con 0x7f9a10079000 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a1610b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a10079400 con 0x7f9a101b00d0 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a0effd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9a10079000 0x7f9a100e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a0ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a101b00d0 0x7f9a101b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a0effd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9a10079000 0x7f9a100e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50046/0 (socket says 172.21.2.117:50046) 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.425 7f9a0effd700 1 -- 172.21.2.117:0/1657072040 learned_addr learned my addr 172.21.2.117:0/1657072040 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:18.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0effd700 1 -- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a101b00d0 msgr2=0x7f9a101b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0effd700 1 --2- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a101b00d0 0x7f9a101b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0effd700 1 -- 172.21.2.117:0/1657072040 >> v1:172.21.2.125:6789/0 conn(0x7f9a1007da20 legacy=0x7f9a100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0effd700 1 -- 172.21.2.117:0/1657072040 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1007de20 con 0x7f9a10079000 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0effd700 1 --2- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10079000 0x7f9a100e4d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f9a04006700 tx=0x7f9a04009b80).ready entity=mon.1 client_cookie=e515cc02dfb5fd57 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0e7fc700 1 -- 172.21.2.117:0/1657072040 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a0400c0c0 con 0x7f9a10079000 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0e7fc700 1 -- 172.21.2.117:0/1657072040 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0400c440 con 0x7f9a10079000 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a0e7fc700 1 -- 172.21.2.117:0/1657072040 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a0400c920 con 0x7f9a10079000 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a1610b700 1 -- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10079000 msgr2=0x7f9a100e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.426 7f9a1610b700 1 --2- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10079000 0x7f9a100e4d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f9a04006700 tx=0x7f9a04009b80).stop 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 -- 172.21.2.117:0/1657072040 shutdown_connections 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 --2- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a101b00d0 0x7f9a101b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 --2- 172.21.2.117:0/1657072040 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10079000 0x7f9a100e4d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 -- 172.21.2.117:0/1657072040 >> 172.21.2.117:0/1657072040 conn(0x7f9a100ce430 msgr2=0x7f9a100e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 -- 172.21.2.117:0/1657072040 shutdown_connections 2021-02-10T11:37:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.427 7f9a1610b700 1 -- 172.21.2.117:0/1657072040 wait complete. 2021-02-10T11:37:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.441 7f9a1610b700 1 Processor -- start 2021-02-10T11:37:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.441 7f9a1610b700 1 -- start start 2021-02-10T11:37:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a1008dc10 0x7f9a10105980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10105fa0 0x7f9a100b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a100b9090 con 0x7f9a100b8670 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a100b92b0 con 0x7f9a10105fa0 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a1610b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a100b6520 con 0x7f9a1008dc10 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45666/0 (socket says 172.21.2.117:45666) 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 -- 172.21.2.117:0/2948328504 learned_addr learned my addr 172.21.2.117:0/2948328504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 -- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a1008dc10 msgr2=0x7f9a10105980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a1008dc10 0x7f9a10105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 -- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10105fa0 msgr2=0x7f9a100b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10105fa0 0x7f9a100b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 -- 172.21.2.117:0/2948328504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a100b6740 con 0x7f9a100b8670 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.442 7f9a0ffff700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f9a0800b970 tx=0x7f9a0800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a08000f90 con 0x7f9a100b8670 2021-02-10T11:37:18.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a08001400 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a080018e0 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a100b6740 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a100b6740 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a08000df0 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 0x7f99f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.443 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9a08032890 con 0x7f9a100b8670 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.444 7f9a0f7fe700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 0x7f99f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.444 7f9a0f7fe700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 0x7f99f001abc0 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:18.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.474 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a10103260 con 0x7f9a100b8670 2021-02-10T11:37:18.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.477 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a080011f0 con 0x7f9a100b8670 2021-02-10T11:37:18.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.605 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802526", "target": ["mgr", ""]}) v1 -- 0x7f9a10077a70 con 0x7f99f00186c0 2021-02-10T11:37:18.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.609 7f99f7fff700 1 -- 172.21.2.117:0/2948328504 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f9a00006a10 con 0x7f99f00186c0 2021-02-10T11:37:18.594 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94 2021-02-10T11:37:18.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.613 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 msgr2=0x7f99f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.613 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 0x7f99f001abc0 crc :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.613 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 msgr2=0x7f9a100b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.613 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0x7f9a0800b970 tx=0x7f9a0800bd10).stop 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 shutdown_connections 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99f00186c0 0x7f99f001abc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a100b8670 0x7f9a100b8a70 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:18.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a10105fa0 0x7f9a100b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:18.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 --2- 172.21.2.117:0/2948328504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a1008dc10 0x7f9a10105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:18.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 >> 172.21.2.117:0/2948328504 conn(0x7f9a1008a180 msgr2=0x7f9a1008ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:18.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 shutdown_connections 2021-02-10T11:37:18.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:18.614 7f9a1610b700 1 -- 172.21.2.117:0/2948328504 wait complete. 2021-02-10T11:37:18.604 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802526 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94 2021-02-10T11:37:18.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:18.604 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:18.604 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:18.941 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:18.941 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:18.941 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242818 s, 43.2 MB/s 2021-02-10T11:37:19.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:19.327 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:19.327 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:19.664 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:19.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:19.664 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:20.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:20.039 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:20.039 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236286 s, 44.4 MB/s 2021-02-10T11:37:20.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:20.040 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:20.040 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:20.376 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:20.377 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:20.377 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:20.753 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:20.754 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:20.754 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246372 s, 42.6 MB/s 2021-02-10T11:37:20.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:20.755 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:20.755 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:21.090 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:21.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:21.090 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:21.464 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:21.464 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:21.465 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0221661 s, 47.3 MB/s 2021-02-10T11:37:21.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:21.466 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:21.466 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:21.802 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:21.802 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:21.802 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:22.180 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:22.180 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:22.180 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250698 s, 41.8 MB/s 2021-02-10T11:37:22.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:22.181 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:22.181 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:22.516 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:22.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:22.517 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:22.890 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:22.890 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:22.890 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236152 s, 44.4 MB/s 2021-02-10T11:37:22.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:22.891 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:22.891 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:23.227 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:23.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:23.228 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:23.605 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:23.605 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:23.605 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246191 s, 42.6 MB/s 2021-02-10T11:37:23.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:23.607 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:23.607 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:23.943 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:23.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:23.943 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:24.321 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:24.322 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:24.322 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0260698 s, 40.2 MB/s 2021-02-10T11:37:24.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:24.323 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:24.323 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:24.658 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:24.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:24.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:25.034 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:25.035 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:25.035 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235942 s, 44.4 MB/s 2021-02-10T11:37:25.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:25.036 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:25.036 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:25.371 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:25.372 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:25.372 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802526/eff2737d-df45-4c01-9cf7-ab0a656aaf94/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.025117 s, 41.7 MB/s 2021-02-10T11:37:25.751 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802527 2021-02-10T11:37:26.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.185 7fc1b8c3b700 1 Processor -- start 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.185 7fc1b8c3b700 1 -- start start 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b8c3b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1b407da20 0x7fc1b40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b8c3b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b40e4da0 0x7fc1b41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b8c3b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b4077a70 con 0x7fc1b40e51a0 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b8c3b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40777d0 con 0x7fc1b407da20 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b8c3b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b407de20 con 0x7fc1b40e4da0 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1b407da20 0x7fc1b40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1b407da20 0x7fc1b40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45678/0 (socket says 172.21.2.117:45678) 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 -- 172.21.2.117:0/3202502338 learned_addr learned my addr 172.21.2.117:0/3202502338 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:26.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 -- 172.21.2.117:0/3202502338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b40e4da0 msgr2=0x7fc1b41b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 --2- 172.21.2.117:0/3202502338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b40e4da0 0x7fc1b41b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 -- 172.21.2.117:0/3202502338 >> v1:172.21.2.125:6789/0 conn(0x7fc1b40e51a0 legacy=0x7fc1b4077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.186 7fc1b285e700 1 -- 172.21.2.117:0/3202502338 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b41b0710 con 0x7fc1b407da20 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b285e700 1 --2- 172.21.2.117:0/3202502338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b407da20 0x7fc1b40771b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fc1a4006700 tx=0x7fc1a4009b80).ready entity=mon.0 client_cookie=f72b073b7727595e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b185c700 1 -- 172.21.2.117:0/3202502338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a400c0c0 con 0x7fc1b407da20 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b185c700 1 -- 172.21.2.117:0/3202502338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a400c440 con 0x7fc1b407da20 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b185c700 1 -- 172.21.2.117:0/3202502338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a400c920 con 0x7fc1b407da20 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b8c3b700 1 -- 172.21.2.117:0/3202502338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b407da20 msgr2=0x7fc1b40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.187 7fc1b8c3b700 1 --2- 172.21.2.117:0/3202502338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b407da20 0x7fc1b40771b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fc1a4006700 tx=0x7fc1a4009b80).stop 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 -- 172.21.2.117:0/3202502338 shutdown_connections 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 --2- 172.21.2.117:0/3202502338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b40e4da0 0x7fc1b41b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 --2- 172.21.2.117:0/3202502338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b407da20 0x7fc1b40771b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 -- 172.21.2.117:0/3202502338 >> 172.21.2.117:0/3202502338 conn(0x7fc1b40ce450 msgr2=0x7fc1b40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 -- 172.21.2.117:0/3202502338 shutdown_connections 2021-02-10T11:37:26.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.188 7fc1b8c3b700 1 -- 172.21.2.117:0/3202502338 wait complete. 2021-02-10T11:37:26.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.202 7fc1b8c3b700 1 Processor -- start 2021-02-10T11:37:26.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.202 7fc1b8c3b700 1 -- start start 2021-02-10T11:37:26.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 0x7fc1b4104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 0x7fc1b40aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4103c20 con 0x7fc1b41034d0 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4103910 con 0x7fc1b4105160 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b8c3b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40b7e00 con 0x7fc1b4104740 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45684/0 (socket says 172.21.2.117:45684) 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 -- 172.21.2.117:0/1338265980 learned_addr learned my addr 172.21.2.117:0/1338265980 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b205d700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 0x7fc1b4104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b305f700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 0x7fc1b40aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 -- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 msgr2=0x7fc1b4104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 0x7fc1b4104b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 -- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 msgr2=0x7fc1b40aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 0x7fc1b40aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 -- 172.21.2.117:0/1338265980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40ab330 con 0x7fc1b41034d0 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b305f700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 0x7fc1b40aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b205d700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 0x7fc1b4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.203 7fc1b285e700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fc1a400b5e0 tx=0x7fc1a400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a400bd70 con 0x7fc1b41034d0 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4002030 con 0x7fc1b41034d0 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1a4002510 con 0x7fc1b41034d0 2021-02-10T11:37:26.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b40ab330 con 0x7fc1b41034d0 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b40ab8c0 con 0x7fc1b41034d0 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1a4000c90 con 0x7fc1b41034d0 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.204 7fc1ab7fe700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 0x7fc19801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.205 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc1a4032770 con 0x7fc1b41034d0 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.205 7fc1b205d700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 0x7fc19801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.205 7fc1b205d700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 0x7fc19801abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.236 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b40b7de0 con 0x7fc1b41034d0 2021-02-10T11:37:26.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.239 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc1a4030dd0 con 0x7fc1b41034d0 2021-02-10T11:37:26.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.364 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802527", "target": ["mgr", ""]}) v1 -- 0x7fc1b40ab8c0 con 0x7fc1980186c0 2021-02-10T11:37:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.388 7fc1ab7fe700 1 -- 172.21.2.117:0/1338265980 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc19c0009b0 con 0x7fc1980186c0 2021-02-10T11:37:26.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 msgr2=0x7fc19801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 0x7fc19801abc0 crc :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 msgr2=0x7fc1b4104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7fc1a400b5e0 tx=0x7fc1a400b980).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 shutdown_connections 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1b4105160 0x7fc1b40aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.397 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1b4104740 0x7fc1b4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.398 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1980186c0 0x7fc19801abc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.398 7fc1b8c3b700 1 --2- 172.21.2.117:0/1338265980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1b41034d0 0x7fc1b4104120 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.398 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 >> 172.21.2.117:0/1338265980 conn(0x7fc1b40a1140 msgr2=0x7fc1b40ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.398 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 shutdown_connections 2021-02-10T11:37:26.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.398 7fc1b8c3b700 1 -- 172.21.2.117:0/1338265980 wait complete. 2021-02-10T11:37:26.387 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802527 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 Processor -- start 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 -- start start 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5e407da20 0x7fa5e40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40e4da0 0x7fa5e41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5e40e51a0 0x7fa5e4077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e4077a70 con 0x7fa5e407da20 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e40777d0 con 0x7fa5e40e51a0 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.822 7fa5e8a35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e407de20 con 0x7fa5e40e4da0 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5e407da20 0x7fa5e40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5e407da20 0x7fa5e40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45692/0 (socket says 172.21.2.117:45692) 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 -- 172.21.2.117:0/2884268967 learned_addr learned my addr 172.21.2.117:0/2884268967 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 -- 172.21.2.117:0/2884268967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40e4da0 msgr2=0x7fa5e41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 --2- 172.21.2.117:0/2884268967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40e4da0 0x7fa5e41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 -- 172.21.2.117:0/2884268967 >> v2:172.21.2.125:3300/0 conn(0x7fa5e40e51a0 msgr2=0x7fa5e4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 --2- 172.21.2.117:0/2884268967 >> v2:172.21.2.125:3300/0 conn(0x7fa5e40e51a0 0x7fa5e4077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 -- 172.21.2.117:0/2884268967 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e41b0710 con 0x7fa5e407da20 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e261c700 1 --2- 172.21.2.117:0/2884268967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e407da20 0x7fa5e40771b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7fa5d4006700 tx=0x7fa5d4009b80).ready entity=mon.0 client_cookie=7030067501317824 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.823 7fa5e161a700 1 -- 172.21.2.117:0/2884268967 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5d400c0c0 con 0x7fa5e407da20 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e161a700 1 -- 172.21.2.117:0/2884268967 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d400c340 con 0x7fa5e407da20 2021-02-10T11:37:26.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e161a700 1 -- 172.21.2.117:0/2884268967 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5d400c860 con 0x7fa5e407da20 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 -- 172.21.2.117:0/2884268967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e407da20 msgr2=0x7fa5e40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 --2- 172.21.2.117:0/2884268967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e407da20 0x7fa5e40771b0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7fa5d4006700 tx=0x7fa5d4009b80).stop 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 -- 172.21.2.117:0/2884268967 shutdown_connections 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 --2- 172.21.2.117:0/2884268967 >> v2:172.21.2.125:3300/0 conn(0x7fa5e40e51a0 0x7fa5e4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 --2- 172.21.2.117:0/2884268967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40e4da0 0x7fa5e41b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 --2- 172.21.2.117:0/2884268967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e407da20 0x7fa5e40771b0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 -- 172.21.2.117:0/2884268967 >> 172.21.2.117:0/2884268967 conn(0x7fa5e40ce450 msgr2=0x7fa5e40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 -- 172.21.2.117:0/2884268967 shutdown_connections 2021-02-10T11:37:26.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.824 7fa5e8a35700 1 -- 172.21.2.117:0/2884268967 wait complete. 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 Processor -- start 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 -- start start 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5e40b3ff0 0x7fa5e40b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40c33d0 0x7fa5e40b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e4090600 con 0x7fa5e4090de0 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e4090860 con 0x7fa5e40b3ff0 2021-02-10T11:37:26.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e8a35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40b3c70 con 0x7fa5e40c33d0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e2e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e2e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45698/0 (socket says 172.21.2.117:45698) 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.839 7fa5e2e1d700 1 -- 172.21.2.117:0/1231665036 learned_addr learned my addr 172.21.2.117:0/1231665036 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e1e1b700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40c33d0 0x7fa5e40b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e261c700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5e40b3ff0 0x7fa5e40b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 -- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40c33d0 msgr2=0x7fa5e40b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40c33d0 0x7fa5e40b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 -- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5e40b3ff0 msgr2=0x7fa5e40b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5e40b3ff0 0x7fa5e40b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 -- 172.21.2.117:0/1231665036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4091800 con 0x7fa5e4090de0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e2e1d700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fa5dc00b580 tx=0x7fa5dc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5dc006240 con 0x7fa5e4090de0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc000c10 con 0x7fa5e4090de0 2021-02-10T11:37:26.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e4091800 con 0x7fa5e4090de0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e4091d90 con 0x7fa5e4090de0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.840 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5dc0054f0 con 0x7fa5e4090de0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.841 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5dc006030 con 0x7fa5e4090de0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.841 7fa5daffd700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 0x7fa5c801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.841 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa5dc032920 con 0x7fa5e4090de0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.841 7fa5e261c700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 0x7fa5c801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:26.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.842 7fa5e261c700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 0x7fa5c801ac10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:26.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.872 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e4103370 con 0x7fa5e4090de0 2021-02-10T11:37:26.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:26.875 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5dc000bc0 con 0x7fa5e4090de0 2021-02-10T11:37:26.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.002 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802527", "target": ["mgr", ""]}) v1 -- 0x7fa5e4091d90 con 0x7fa5c8018730 2021-02-10T11:37:26.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.005 7fa5daffd700 1 -- 172.21.2.117:0/1231665036 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fa5d4006520 con 0x7fa5c8018730 2021-02-10T11:37:26.987 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 msgr2=0x7fa5c801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 0x7fa5c801ac10 crc :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 msgr2=0x7fa5e40911e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fa5dc00b580 tx=0x7fa5dc00b920).stop 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 shutdown_connections 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5c8018730 0x7fa5c801ac10 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5e4090de0 0x7fa5e40911e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5e40c33d0 0x7fa5e40b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 --2- 172.21.2.117:0/1231665036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5e40b3ff0 0x7fa5e40b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:26.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 >> 172.21.2.117:0/1231665036 conn(0x7fa5e40a1420 msgr2=0x7fa5e409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:26.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 shutdown_connections 2021-02-10T11:37:26.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:27.006 7fa5e8a35700 1 -- 172.21.2.117:0/1231665036 wait complete. 2021-02-10T11:37:26.996 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802527 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77 2021-02-10T11:37:26.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:26.997 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:26.997 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:27.333 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:27.334 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:27.334 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223885 s, 46.8 MB/s 2021-02-10T11:37:27.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:27.712 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:27.712 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:28.048 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:28.048 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:28.048 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:28.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:28.424 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:28.424 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245189 s, 42.8 MB/s 2021-02-10T11:37:28.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:28.425 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:28.425 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:28.760 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:28.761 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:28.761 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:29.137 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:29.137 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230497 s, 45.5 MB/s 2021-02-10T11:37:29.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:29.138 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:29.138 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:29.474 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:29.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:29.475 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:29.851 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:29.852 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:29.852 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022798 s, 46.0 MB/s 2021-02-10T11:37:29.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:29.853 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:29.853 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:30.190 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:30.190 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:30.190 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:30.567 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242583 s, 43.2 MB/s 2021-02-10T11:37:30.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:30.568 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:30.568 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:30.902 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:30.903 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:30.903 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257194 s, 40.8 MB/s 2021-02-10T11:37:31.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:31.282 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:31.282 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:31.618 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:31.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:31.618 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:31.994 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:31.994 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:31.994 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232903 s, 45.0 MB/s 2021-02-10T11:37:31.995 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:31.995 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:31.995 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:32.331 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:32.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:32.332 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:32.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:32.704 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:32.705 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023588 s, 44.5 MB/s 2021-02-10T11:37:32.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:32.706 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:32.706 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:33.045 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:33.045 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:33.045 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:33.409 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:33.412 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:37:33.421 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:33.422 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:33.422 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248994 s, 42.1 MB/s 2021-02-10T11:37:33.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:33.423 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:33.423 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:33.748 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:33.748 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:33.748 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802527/c4dd34c3-4651-423d-affc-df1513870d77/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:34.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:34.124 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:34.124 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241483 s, 43.4 MB/s 2021-02-10T11:37:34.125 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802528 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 Processor -- start 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 -- start start 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 0x7fc4340771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4340e51a0 0x7fc434079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc434077a70 con 0x7fc4340e4da0 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4340e5c20 con 0x7fc4340e51a0 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc438858700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4340777d0 con 0x7fc43407da20 2021-02-10T11:37:34.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.560 7fc432e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4340e51a0 0x7fc434079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc431e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc4340e4da0 0x7fc4341b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45954/0 (socket says 172.21.2.117:45954) 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc4340e51a0 0x7fc434079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45706/0 (socket says 172.21.2.117:45706) 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc431e1b700 1 -- 172.21.2.117:0/1401571903 learned_addr learned my addr 172.21.2.117:0/1401571903 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc43261c700 1 --2- 172.21.2.117:0/1401571903 >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 0x7fc4340771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 -- 172.21.2.117:0/1401571903 >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 msgr2=0x7fc4340771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 --2- 172.21.2.117:0/1401571903 >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 0x7fc4340771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 -- 172.21.2.117:0/1401571903 >> v1:172.21.2.117:6789/0 conn(0x7fc4340e4da0 legacy=0x7fc4341b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 -- 172.21.2.117:0/1401571903 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4341b08e0 con 0x7fc4340e51a0 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc43261c700 1 --2- 172.21.2.117:0/1401571903 >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 0x7fc4340771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc432e1d700 1 --2- 172.21.2.117:0/1401571903 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4340e51a0 0x7fc434079000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7fc42c006e30 tx=0x7fc42c009e60).ready entity=mon.0 client_cookie=ad686b428f14c7c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc43161a700 1 -- 172.21.2.117:0/1401571903 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc42c006a50 con 0x7fc4340e51a0 2021-02-10T11:37:34.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc43161a700 1 -- 172.21.2.117:0/1401571903 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c00b300 con 0x7fc4340e51a0 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc43161a700 1 -- 172.21.2.117:0/1401571903 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc42c00b820 con 0x7fc4340e51a0 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc438858700 1 -- 172.21.2.117:0/1401571903 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4340e51a0 msgr2=0x7fc434079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.561 7fc438858700 1 --2- 172.21.2.117:0/1401571903 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4340e51a0 0x7fc434079000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7fc42c006e30 tx=0x7fc42c009e60).stop 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 -- 172.21.2.117:0/1401571903 shutdown_connections 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 --2- 172.21.2.117:0/1401571903 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc4340e51a0 0x7fc434079000 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 --2- 172.21.2.117:0/1401571903 >> v2:172.21.2.125:3300/0 conn(0x7fc43407da20 0x7fc4340771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 -- 172.21.2.117:0/1401571903 >> 172.21.2.117:0/1401571903 conn(0x7fc4340ce450 msgr2=0x7fc4340e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 -- 172.21.2.117:0/1401571903 shutdown_connections 2021-02-10T11:37:34.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.562 7fc438858700 1 -- 172.21.2.117:0/1401571903 wait complete. 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.576 7fc438858700 1 Processor -- start 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.576 7fc438858700 1 -- start start 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc434104690 0x7fc4341b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc434105870 0x7fc434105c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc434106500 con 0x7fc434195b40 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341b63f0 con 0x7fc434104690 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc438858700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4341b65d0 con 0x7fc434105870 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc431e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc431e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45710/0 (socket says 172.21.2.117:45710) 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc431e1b700 1 -- 172.21.2.117:0/3939795438 learned_addr learned my addr 172.21.2.117:0/3939795438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc432e1d700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc434105870 0x7fc434105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.577 7fc43261c700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc434104690 0x7fc4341b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 -- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc434105870 msgr2=0x7fc434105c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc434105870 0x7fc434105c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 -- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc434104690 msgr2=0x7fc4341b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc434104690 0x7fc4341b67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 -- 172.21.2.117:0/3939795438 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434106a80 con 0x7fc434195b40 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc431e1b700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fc41c001c60 tx=0x7fc41c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc41c00acc0 con 0x7fc434195b40 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c004140 con 0x7fc434195b40 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc41c004510 con 0x7fc434195b40 2021-02-10T11:37:34.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc438858700 1 -- 172.21.2.117:0/3939795438 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc434106a80 con 0x7fc434195b40 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.578 7fc438858700 1 -- 172.21.2.117:0/3939795438 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc434106a80 con 0x7fc434195b40 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.579 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc41c000ce0 con 0x7fc434195b40 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.579 7fc42affd700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 0x7fc41801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.579 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc41c032790 con 0x7fc434195b40 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.579 7fc43261c700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 0x7fc41801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:34.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.579 7fc43261c700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 0x7fc41801abd0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:34.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.610 7fc438858700 1 -- 172.21.2.117:0/3939795438 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4341b4f00 con 0x7fc434195b40 2021-02-10T11:37:34.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.613 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc41c0009b0 con 0x7fc434195b40 2021-02-10T11:37:34.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.738 7fc438858700 1 -- 172.21.2.117:0/3939795438 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802528", "target": ["mgr", ""]}) v1 -- 0x7fc4341b4f00 con 0x7fc4180186d0 2021-02-10T11:37:34.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.765 7fc42affd700 1 -- 172.21.2.117:0/3939795438 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc424006540 con 0x7fc4180186d0 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 -- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 msgr2=0x7fc41801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 0x7fc41801abd0 crc :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 -- 172.21.2.117:0/3939795438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 msgr2=0x7fc434195f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fc41c001c60 tx=0x7fc41c00a8d0).stop 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 -- 172.21.2.117:0/3939795438 shutdown_connections 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4180186d0 0x7fc41801abd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc434195b40 0x7fc434195f40 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc434105870 0x7fc434105c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 --2- 172.21.2.117:0/3939795438 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc434104690 0x7fc4341b67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 -- 172.21.2.117:0/3939795438 >> 172.21.2.117:0/3939795438 conn(0x7fc4340ffe30 msgr2=0x7fc4340c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.771 7fc438858700 1 -- 172.21.2.117:0/3939795438 shutdown_connections 2021-02-10T11:37:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:34.772 7fc438858700 1 -- 172.21.2.117:0/3939795438 wait complete. 2021-02-10T11:37:34.761 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802528 2021-02-10T11:37:35.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.198 7f4f84754700 1 Processor -- start 2021-02-10T11:37:35.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f84754700 1 -- start start 2021-02-10T11:37:35.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f84754700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0d1530 0x7f4f7c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f84754700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c077a70 con 0x7f4f7c079000 2021-02-10T11:37:35.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f84754700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c079400 con 0x7f4f7c07da20 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f84754700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c07de20 con 0x7f4f7c0d1530 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f81d6f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4f7c079000 0x7f4f7c0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45966/0 (socket says 172.21.2.117:45966) 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.199 7f4f81d6f700 1 -- 172.21.2.117:0/2086909724 learned_addr learned my addr 172.21.2.117:0/2086909724 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394449396 0 0) 0x7f4f70001180 con 0x7f4f7c079000 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f68002fe0 con 0x7f4f7c079000 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f82d71700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0d1530 0x7f4f7c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 250369425 0 0) 0x7f4f70001180 con 0x7f4f7c079000 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0d1530 msgr2=0x7f4f7c0d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0d1530 0x7f4f7c0d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 >> v1:172.21.2.125:6789/0 conn(0x7f4f7c07da20 legacy=0x7f4f7c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c1b00f0 con 0x7f4f7c079000 2021-02-10T11:37:35.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4f70001680 con 0x7f4f7c079000 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 >> v1:172.21.2.117:6789/0 conn(0x7f4f7c079000 legacy=0x7f4f7c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f68003b10 0x7f4f68006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f68006830 0x7f4f68008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f680095d0 0x7f4f6800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6800c2f0 con 0x7f4f68006830 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6800c510 con 0x7f4f68003b10 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6800c730 con 0x7f4f680095d0 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.200 7f4f81d6f700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f68006830 0x7f4f68008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f82570700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f68003b10 0x7f4f68006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f82d71700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f680095d0 0x7f4f6800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 -- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f680095d0 msgr2=0x7f4f6800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:35.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f680095d0 0x7f4f6800baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 -- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f68003b10 msgr2=0x7f4f68006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f68003b10 0x7f4f68006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 -- 172.21.2.117:0/2086909724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68050970 con 0x7f4f68006830 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f81d6f700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f68006830 0x7f4f68008d50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f4f70000b60 tx=0x7f4f70007da0).ready entity=mon.0 client_cookie=d43fb2a972d9cbb3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f700070c0 con 0x7f4f68006830 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f700043a0 con 0x7f4f68006830 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f8156e700 1 -- 172.21.2.117:0/2086909724 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f70004830 con 0x7f4f68006830 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f84754700 1 -- 172.21.2.117:0/2086909724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f68006830 msgr2=0x7f4f68008d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f84754700 1 --2- 172.21.2.117:0/2086909724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f68006830 0x7f4f68008d50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f4f70000b60 tx=0x7f4f70007da0).stop 2021-02-10T11:37:35.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.201 7f4f82d71700 1 -- 172.21.2.117:0/2086909724 reap_dead start 2021-02-10T11:37:35.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.202 7f4f84754700 1 -- 172.21.2.117:0/2086909724 shutdown_connections 2021-02-10T11:37:35.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.202 7f4f84754700 1 -- 172.21.2.117:0/2086909724 >> 172.21.2.117:0/2086909724 conn(0x7f4f7c0ce450 msgr2=0x7f4f7c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.202 7f4f84754700 1 -- 172.21.2.117:0/2086909724 shutdown_connections 2021-02-10T11:37:35.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.202 7f4f84754700 1 -- 172.21.2.117:0/2086909724 wait complete. 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 Processor -- start 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 -- start start 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f7c0a47d0 0x7f4f7c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0b77d0 0x7f4f7c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c189970 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c190aa0 con 0x7f4f7c0a47d0 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f84754700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c0b81f0 con 0x7f4f7c0b77d0 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f81d6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f81d6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45730/0 (socket says 172.21.2.117:45730) 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.217 7f4f81d6f700 1 -- 172.21.2.117:0/1801555482 learned_addr learned my addr 172.21.2.117:0/1801555482 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:35.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 -- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0b77d0 msgr2=0x7f4f7c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0b77d0 0x7f4f7c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 -- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f7c0a47d0 msgr2=0x7f4f7c0a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f7c0a47d0 0x7f4f7c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 -- 172.21.2.117:0/1801555482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0b8410 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f81d6f700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f4f7000a590 tx=0x7f4f7000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f7000ad20 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f70007030 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f700043a0 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f84754700 1 -- 172.21.2.117:0/1801555482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c0b8410 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.218 7f4f84754700 1 -- 172.21.2.117:0/1801555482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c0b89a0 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.219 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f70007210 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.219 7f4f6effd700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 0x7f4f6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:35.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.219 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4f70032680 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.219 7f4f82570700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 0x7f4f6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:35.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.220 7f4f82570700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 0x7f4f6401abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:35.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.249 7f4f84754700 1 -- 172.21.2.117:0/1801555482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c077a70 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.252 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f7002dff0 con 0x7f4f7c0a51f0 2021-02-10T11:37:35.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.379 7f4f84754700 1 -- 172.21.2.117:0/1801555482 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802528", "target": ["mgr", ""]}) v1 -- 0x7f4f7c189970 con 0x7f4f640186c0 2021-02-10T11:37:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.381 7f4f6effd700 1 -- 172.21.2.117:0/1801555482 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f4f78007a00 con 0x7f4f640186c0 2021-02-10T11:37:35.363 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 -- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 msgr2=0x7f4f6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 0x7f4f6401abc0 crc :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 -- 172.21.2.117:0/1801555482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 msgr2=0x7f4f7c0a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f4f7000a590 tx=0x7f4f7000a930).stop 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 -- 172.21.2.117:0/1801555482 shutdown_connections 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f640186c0 0x7f4f6401abc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f7c0a51f0 0x7f4f7c0a55f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f7c0b77d0 0x7f4f7c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 --2- 172.21.2.117:0/1801555482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f7c0a47d0 0x7f4f7c0a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.383 7f4f84754700 1 -- 172.21.2.117:0/1801555482 >> 172.21.2.117:0/1801555482 conn(0x7f4f7c0b5140 msgr2=0x7f4f7c0a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.384 7f4f84754700 1 -- 172.21.2.117:0/1801555482 shutdown_connections 2021-02-10T11:37:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:35.384 7f4f84754700 1 -- 172.21.2.117:0/1801555482 wait complete. 2021-02-10T11:37:35.373 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802528 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e 2021-02-10T11:37:35.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:35.373 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:35.374 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:35.710 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:35.711 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:35.711 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:36.093 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:36.093 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:36.093 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243811 s, 43.0 MB/s 2021-02-10T11:37:36.094 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:36.094 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:36.094 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:36.430 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:36.430 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:36.431 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:36.807 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:36.807 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:36.807 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239048 s, 43.9 MB/s 2021-02-10T11:37:36.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:36.809 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:36.809 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:37.144 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:37.144 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:37.145 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:37.517 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:37.517 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:37.517 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246 s, 42.6 MB/s 2021-02-10T11:37:37.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:37.518 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:37.518 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:37.854 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:37.854 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:37.855 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:38.231 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:38.231 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:38.231 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245314 s, 42.7 MB/s 2021-02-10T11:37:38.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:38.232 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:38.233 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:38.568 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:38.568 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:38.569 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0232288 s, 45.1 MB/s 2021-02-10T11:37:38.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:38.946 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:38.946 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:39.282 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:39.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:39.283 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023313 s, 45.0 MB/s 2021-02-10T11:37:39.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:39.660 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:39.660 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:39.995 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:39.995 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:39.996 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:40.371 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:40.372 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:40.372 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237346 s, 44.2 MB/s 2021-02-10T11:37:40.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:40.373 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:40.373 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:40.708 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:40.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:40.709 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:41.084 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:41.084 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:41.084 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229313 s, 45.7 MB/s 2021-02-10T11:37:41.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:41.085 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:41.085 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:41.422 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:41.423 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:41.423 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:41.802 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:41.802 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253386 s, 41.4 MB/s 2021-02-10T11:37:41.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:41.804 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:41.804 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:42.139 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:42.140 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:42.140 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802528/154d8b08-9d3f-450a-b6f9-25b8af15b77e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:42.510 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:42.510 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:42.510 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233994 s, 44.8 MB/s 2021-02-10T11:37:42.511 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802529 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.946 7ffa6cae4700 1 Processor -- start 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.946 7ffa6cae4700 1 -- start start 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6cae4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa6807da20 0x7ffa680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6cae4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffa680e5180 0x7ffa680d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6cae4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa68077a70 con 0x7ffa680e4d80 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6cae4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680e5c00 con 0x7ffa680e5180 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6cae4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680777d0 con 0x7ffa6807da20 2021-02-10T11:37:42.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6605d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ffa680e4d80 0x7ffa681b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:45984/0 (socket says 172.21.2.117:45984) 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6605d700 1 -- 172.21.2.117:0/1445896981 learned_addr learned my addr 172.21.2.117:0/1445896981 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025076305 0 0) 0x7ffa5c0010b0 con 0x7ffa680e4d80 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa50002fe0 con 0x7ffa680e4d80 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6685e700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa6807da20 0x7ffa680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.947 7ffa6705f700 1 --2- 172.21.2.117:0/1445896981 >> v2:172.21.2.125:3300/0 conn(0x7ffa680e5180 0x7ffa680d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3038269271 0 0) 0x7ffa5c0010b0 con 0x7ffa680e4d80 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa6807da20 msgr2=0x7ffa680771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa6807da20 0x7ffa680771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 >> v2:172.21.2.125:3300/0 conn(0x7ffa680e5180 msgr2=0x7ffa680d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 --2- 172.21.2.117:0/1445896981 >> v2:172.21.2.125:3300/0 conn(0x7ffa680e5180 0x7ffa680d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa681b08c0 con 0x7ffa680e4d80 2021-02-10T11:37:42.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ffa5c0015b0 con 0x7ffa680e4d80 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 >> v1:172.21.2.117:6789/0 conn(0x7ffa680e4d80 legacy=0x7ffa681b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa50003a60 0x7ffa50005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa5004a740 0x7ffa5004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa5004d500 0x7ffa5004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500502d0 con 0x7ffa5004d500 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500504f0 con 0x7ffa5004a740 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50050710 con 0x7ffa50003a60 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa5004d500 0x7ffa5004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6685e700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa50003a60 0x7ffa50005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6605d700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa5004a740 0x7ffa5004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 -- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa50003a60 msgr2=0x7ffa50005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa50003a60 0x7ffa50005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 -- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa5004a740 msgr2=0x7ffa5004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa5004a740 0x7ffa5004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.948 7ffa6705f700 1 -- 172.21.2.117:0/1445896981 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500509b0 con 0x7ffa5004d500 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6705f700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa5004d500 0x7ffa5004fa20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7ffa60006a40 tx=0x7ffa6000c0c0).ready entity=mon.0 client_cookie=306a8662018fb34a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa6000c4b0 con 0x7ffa5004d500 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6000c7b0 con 0x7ffa5004d500 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6585c700 1 -- 172.21.2.117:0/1445896981 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa6000cc90 con 0x7ffa5004d500 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6cae4700 1 -- 172.21.2.117:0/1445896981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa5004d500 msgr2=0x7ffa5004fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6cae4700 1 --2- 172.21.2.117:0/1445896981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa5004d500 0x7ffa5004fa20 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7ffa60006a40 tx=0x7ffa6000c0c0).stop 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.949 7ffa6705f700 1 -- 172.21.2.117:0/1445896981 reap_dead start 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.950 7ffa6cae4700 1 -- 172.21.2.117:0/1445896981 shutdown_connections 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.950 7ffa6cae4700 1 -- 172.21.2.117:0/1445896981 >> 172.21.2.117:0/1445896981 conn(0x7ffa680ce430 msgr2=0x7ffa680e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.950 7ffa6cae4700 1 -- 172.21.2.117:0/1445896981 shutdown_connections 2021-02-10T11:37:42.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.950 7ffa6cae4700 1 -- 172.21.2.117:0/1445896981 wait complete. 2021-02-10T11:37:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.964 7ffa6cae4700 1 Processor -- start 2021-02-10T11:37:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.964 7ffa6cae4700 1 -- start start 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa680b8160 0x7ffa680b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa680b8630 0x7ffa680b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681a3290 con 0x7ffa680b9050 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681a34b0 con 0x7ffa680b8630 2021-02-10T11:37:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6cae4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa681a36d0 con 0x7ffa680b8160 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45744/0 (socket says 172.21.2.117:45744) 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 -- 172.21.2.117:0/1311803976 learned_addr learned my addr 172.21.2.117:0/1311803976 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 -- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa680b8160 msgr2=0x7ffa680b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa680b8160 0x7ffa680b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 -- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa680b8630 msgr2=0x7ffa680b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa680b8630 0x7ffa680b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 -- 172.21.2.117:0/1311803976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa681a38f0 con 0x7ffa680b9050 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.965 7ffa6705f700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7ffa6000b8e0 tx=0x7ffa6000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa60000df0 con 0x7ffa680b9050 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa60009c30 con 0x7ffa680b9050 2021-02-10T11:37:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa681a38f0 con 0x7ffa680b9050 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa680a4b00 con 0x7ffa680b9050 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa6000c4e0 con 0x7ffa680b9050 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa600011a0 con 0x7ffa680b9050 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 0x7ffa4c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.966 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffa60032c90 con 0x7ffa680b9050 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.967 7ffa6685e700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 0x7ffa4c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:42.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.967 7ffa6685e700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 0x7ffa4c01ac10 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:42.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.996 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa68077a70 con 0x7ffa680b9050 2021-02-10T11:37:42.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:42.999 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa60009500 con 0x7ffa680b9050 2021-02-10T11:37:43.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.124 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802529", "target": ["mgr", ""]}) v1 -- 0x7ffa680a4b00 con 0x7ffa4c018730 2021-02-10T11:37:43.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.151 7ffa577fe700 1 -- 172.21.2.117:0/1311803976 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ffa58006e70 con 0x7ffa4c018730 2021-02-10T11:37:43.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 msgr2=0x7ffa4c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 0x7ffa4c01ac10 crc :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 msgr2=0x7ffa681a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7ffa6000b8e0 tx=0x7ffa6000bc80).stop 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 shutdown_connections 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa4c018730 0x7ffa4c01ac10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa680b9050 0x7ffa681a2bf0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa680b8630 0x7ffa680b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 --2- 172.21.2.117:0/1311803976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa680b8160 0x7ffa680b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 >> 172.21.2.117:0/1311803976 conn(0x7ffa680a90a0 msgr2=0x7ffa680a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:43.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 shutdown_connections 2021-02-10T11:37:43.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.158 7ffa6cae4700 1 -- 172.21.2.117:0/1311803976 wait complete. 2021-02-10T11:37:43.147 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802529 2021-02-10T11:37:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.583 7fb069887700 1 Processor -- start 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.583 7fb069887700 1 -- start start 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb069887700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb069887700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb0640e51a0 0x7fb064079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb069887700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb064077a70 con 0x7fb0640e4da0 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb069887700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640e5c20 con 0x7fb06407da20 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb069887700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640777d0 con 0x7fb0640e51a0 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45752/0 (socket says 172.21.2.117:45752) 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063574700 1 -- 172.21.2.117:0/589712467 learned_addr learned my addr 172.21.2.117:0/589712467 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063d75700 1 --2- 172.21.2.117:0/589712467 >> v2:172.21.2.125:3300/0 conn(0x7fb0640e51a0 0x7fb064079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063d75700 1 -- 172.21.2.117:0/589712467 >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 msgr2=0x7fb0640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063d75700 1 --2- 172.21.2.117:0/589712467 >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.584 7fb063d75700 1 -- 172.21.2.117:0/589712467 >> v1:172.21.2.125:6789/0 conn(0x7fb0640e4da0 legacy=0x7fb0641b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb063d75700 1 -- 172.21.2.117:0/589712467 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0641b08e0 con 0x7fb0640e51a0 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb063574700 1 --2- 172.21.2.117:0/589712467 >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb063d75700 1 --2- 172.21.2.117:0/589712467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0640e51a0 0x7fb064079000 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb05c006e50 tx=0x7fb05c009e70).ready entity=mon.1 client_cookie=cf9c26656acc6f49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb062572700 1 -- 172.21.2.117:0/589712467 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c0066f0 con 0x7fb0640e51a0 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb062572700 1 -- 172.21.2.117:0/589712467 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c003030 con 0x7fb0640e51a0 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb062572700 1 -- 172.21.2.117:0/589712467 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c00b700 con 0x7fb0640e51a0 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb069887700 1 -- 172.21.2.117:0/589712467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0640e51a0 msgr2=0x7fb064079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.585 7fb069887700 1 --2- 172.21.2.117:0/589712467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0640e51a0 0x7fb064079000 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb05c006e50 tx=0x7fb05c009e70).stop 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 -- 172.21.2.117:0/589712467 shutdown_connections 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 --2- 172.21.2.117:0/589712467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0640e51a0 0x7fb064079000 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 --2- 172.21.2.117:0/589712467 >> v2:172.21.2.117:3300/0 conn(0x7fb06407da20 0x7fb0640771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 -- 172.21.2.117:0/589712467 >> 172.21.2.117:0/589712467 conn(0x7fb0640ce450 msgr2=0x7fb0640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:43.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 -- 172.21.2.117:0/589712467 shutdown_connections 2021-02-10T11:37:43.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.586 7fb069887700 1 -- 172.21.2.117:0/589712467 wait complete. 2021-02-10T11:37:43.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.600 7fb069887700 1 Processor -- start 2021-02-10T11:37:43.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.600 7fb069887700 1 -- start start 2021-02-10T11:37:43.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0640b7fc0 0x7fb0641042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0641048f0 0x7fb064104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641054f0 con 0x7fb064090710 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb064195a30 con 0x7fb0641048f0 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb069887700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb064195fb0 con 0x7fb0640b7fc0 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45758/0 (socket says 172.21.2.117:45758) 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 -- 172.21.2.117:0/1815848896 learned_addr learned my addr 172.21.2.117:0/1815848896 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 -- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0640b7fc0 msgr2=0x7fb0641042d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0640b7fc0 0x7fb0641042d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 -- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0641048f0 msgr2=0x7fb064104cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0641048f0 0x7fb064104cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 -- 172.21.2.117:0/1815848896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0640b7b60 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.601 7fb063d75700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fb05c00b910 tx=0x7fb05c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c00c2f0 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb069887700 1 -- 172.21.2.117:0/1815848896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0640b7b60 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c00c2f0 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb069887700 1 -- 172.21.2.117:0/1815848896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0640b7b60 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05c009020 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb05c0093d0 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 0x7fb04801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.602 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb05c032cd0 con 0x7fb064090710 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.603 7fb063574700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 0x7fb04801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:43.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.603 7fb063574700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 0x7fb04801abc0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.634 7fb069887700 1 -- 172.21.2.117:0/1815848896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb064103280 con 0x7fb064090710 2021-02-10T11:37:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.636 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb05c009700 con 0x7fb064090710 2021-02-10T11:37:43.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.763 7fb069887700 1 -- 172.21.2.117:0/1815848896 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802529", "target": ["mgr", ""]}) v1 -- 0x7fb064103280 con 0x7fb048018730 2021-02-10T11:37:43.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.767 7fb05bfff700 1 -- 172.21.2.117:0/1815848896 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fb04c000940 con 0x7fb048018730 2021-02-10T11:37:43.752 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735 2021-02-10T11:37:43.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 msgr2=0x7fb04801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 0x7fb04801abc0 crc :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 msgr2=0x7fb064090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fb05c00b910 tx=0x7fb05c00bcb0).stop 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 shutdown_connections 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb048018730 0x7fb04801abc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb064090710 0x7fb064090b10 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:43.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0641048f0 0x7fb064104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 --2- 172.21.2.117:0/1815848896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0640b7fc0 0x7fb0641042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:43.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 >> 172.21.2.117:0/1815848896 conn(0x7fb0640a1850 msgr2=0x7fb0640c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:43.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 shutdown_connections 2021-02-10T11:37:43.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:43.773 7fb069887700 1 -- 172.21.2.117:0/1815848896 wait complete. 2021-02-10T11:37:43.762 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802529 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735 2021-02-10T11:37:43.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:43.763 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:43.763 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:44.098 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:44.098 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:44.098 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:44.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:44.482 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:44.482 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0246265 s, 42.6 MB/s 2021-02-10T11:37:44.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:44.483 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:44.483 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:44.820 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:44.820 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:44.820 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:45.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:45.198 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:45.198 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0254067 s, 41.3 MB/s 2021-02-10T11:37:45.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:45.199 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:45.200 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:45.534 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:45.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:45.535 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:45.911 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:45.911 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:45.911 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237602 s, 44.1 MB/s 2021-02-10T11:37:45.912 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:45.912 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:45.912 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:46.248 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:46.248 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:46.249 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:46.627 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:46.627 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:46.628 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240012 s, 43.7 MB/s 2021-02-10T11:37:46.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:46.629 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:46.629 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:46.964 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:46.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:46.964 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:47.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:47.340 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:47.340 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023819 s, 44.0 MB/s 2021-02-10T11:37:47.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:47.341 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:47.341 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:47.677 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:47.678 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:47.678 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:48.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:48.048 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:48.048 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0223961 s, 46.8 MB/s 2021-02-10T11:37:48.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:48.049 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:48.049 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:48.385 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:48.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:48.386 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:48.762 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:48.762 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:48.762 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235163 s, 44.6 MB/s 2021-02-10T11:37:48.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:48.764 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:48.764 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:49.099 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:49.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:49.099 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:49.474 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:49.475 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:49.475 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244968 s, 42.8 MB/s 2021-02-10T11:37:49.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:49.476 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:49.476 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:49.813 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:49.813 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:49.814 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:50.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:50.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:50.189 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248084 s, 42.3 MB/s 2021-02-10T11:37:50.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:50.191 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:50.191 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:50.527 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:50.528 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:50.528 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802529/6f1b8009-691b-4879-96f0-0d5a4ecec735/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:50.907 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:50.907 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:50.907 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253455 s, 41.4 MB/s 2021-02-10T11:37:50.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802530 2021-02-10T11:37:51.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.341 7f89318e4700 1 Processor -- start 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.341 7f89318e4700 1 -- start start 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f89318e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 0x7f892c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f89318e4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 0x7f892c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f89318e4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c077a70 con 0x7f892c07da20 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f89318e4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892c0e53c0 con 0x7f892c1b00f0 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f89318e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c079400 con 0x7f892c079000 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f892b574700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f892c07da20 0x7f892c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:38942/0 (socket says 172.21.2.117:38942) 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f892b574700 1 -- 172.21.2.117:0/3050640296 learned_addr learned my addr 172.21.2.117:0/3050640296 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f892ad73700 1 --2- 172.21.2.117:0/3050640296 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 0x7f892c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.342 7f892bd75700 1 --2- 172.21.2.117:0/3050640296 >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 0x7f892c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892ad73700 1 -- 172.21.2.117:0/3050640296 >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 msgr2=0x7f892c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892a572700 1 -- 172.21.2.117:0/3050640296 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528744525 0 0) 0x7f891c0010b0 con 0x7f892c07da20 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892ad73700 1 --2- 172.21.2.117:0/3050640296 >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 0x7f892c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892ad73700 1 -- 172.21.2.117:0/3050640296 >> v1:172.21.2.125:6789/0 conn(0x7f892c07da20 legacy=0x7f892c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892ad73700 1 -- 172.21.2.117:0/3050640296 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c07de20 con 0x7f892c079000 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892bd75700 1 --2- 172.21.2.117:0/3050640296 >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 0x7f892c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892ad73700 1 --2- 172.21.2.117:0/3050640296 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 0x7f892c0e4da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8920006650 tx=0x7f8920009a80).ready entity=mon.2 client_cookie=a97414a86222ed37 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.343 7f892a572700 1 -- 172.21.2.117:0/3050640296 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89200009b0 con 0x7f892c079000 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f892a572700 1 -- 172.21.2.117:0/3050640296 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892000c0c0 con 0x7f892c079000 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f892a572700 1 -- 172.21.2.117:0/3050640296 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f892000c510 con 0x7f892c079000 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 -- 172.21.2.117:0/3050640296 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 msgr2=0x7f892c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 --2- 172.21.2.117:0/3050640296 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 0x7f892c0e4da0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8920006650 tx=0x7f8920009a80).stop 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 -- 172.21.2.117:0/3050640296 shutdown_connections 2021-02-10T11:37:51.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 --2- 172.21.2.117:0/3050640296 >> v2:172.21.2.125:3300/0 conn(0x7f892c1b00f0 0x7f892c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 --2- 172.21.2.117:0/3050640296 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c079000 0x7f892c0e4da0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 -- 172.21.2.117:0/3050640296 >> 172.21.2.117:0/3050640296 conn(0x7f892c0ce450 msgr2=0x7f892c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 -- 172.21.2.117:0/3050640296 shutdown_connections 2021-02-10T11:37:51.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.344 7f89318e4700 1 -- 172.21.2.117:0/3050640296 wait complete. 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 Processor -- start 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 -- start start 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 0x7f892c1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f892c090710 0x7f892c090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c1054f0 con 0x7f892c1048f0 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892c195a30 con 0x7f892c090710 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f89318e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892c195fb0 con 0x7f892c0b7fc0 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f892ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f892ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45770/0 (socket says 172.21.2.117:45770) 2021-02-10T11:37:51.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.359 7f892ad73700 1 -- 172.21.2.117:0/486618943 learned_addr learned my addr 172.21.2.117:0/486618943 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892bd75700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f892c090710 0x7f892c090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892b574700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 0x7f892c1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 -- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 msgr2=0x7f892c1042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 0x7f892c1042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 -- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f892c090710 msgr2=0x7f892c090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f892c090710 0x7f892c090b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 -- 172.21.2.117:0/486618943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0b7b60 con 0x7f892c1048f0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892b574700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 0x7f892c1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f892ad73700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8920009630 tx=0x7f892000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89200093a0 con 0x7f892c1048f0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892000bd70 con 0x7f892c1048f0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8920005560 con 0x7f892c1048f0 2021-02-10T11:37:51.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f89318e4700 1 -- 172.21.2.117:0/486618943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c0b7b60 con 0x7f892c1048f0 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.360 7f89318e4700 1 -- 172.21.2.117:0/486618943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c0b7b60 con 0x7f892c1048f0 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.361 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8920000c90 con 0x7f892c1048f0 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.361 7f8913fff700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 0x7f890c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.361 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f89200326a0 con 0x7f892c1048f0 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.361 7f892b574700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 0x7f890c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.361 7f892b574700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 0x7f890c01abc0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.392 7f89318e4700 1 -- 172.21.2.117:0/486618943 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c103280 con 0x7f892c1048f0 2021-02-10T11:37:51.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.394 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8920030d00 con 0x7f892c1048f0 2021-02-10T11:37:51.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.518 7f89318e4700 1 -- 172.21.2.117:0/486618943 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802530", "target": ["mgr", ""]}) v1 -- 0x7f892c103280 con 0x7f890c0186c0 2021-02-10T11:37:51.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.546 7f8913fff700 1 -- 172.21.2.117:0/486618943 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f891c0009b0 con 0x7f890c0186c0 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 msgr2=0x7f890c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 0x7f890c01abc0 crc :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 msgr2=0x7f892c104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f8920009630 tx=0x7f892000b980).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 shutdown_connections 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f890c0186c0 0x7f890c01abc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f892c1048f0 0x7f892c104cf0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f892c090710 0x7f892c090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 --2- 172.21.2.117:0/486618943 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f892c0b7fc0 0x7f892c1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 >> 172.21.2.117:0/486618943 conn(0x7f892c0a1850 msgr2=0x7f892c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 shutdown_connections 2021-02-10T11:37:51.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.552 7f89318e4700 1 -- 172.21.2.117:0/486618943 wait complete. 2021-02-10T11:37:51.542 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802530 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.976 7fcda8dfa700 1 Processor -- start 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.976 7fcda8dfa700 1 -- start start 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.976 7fcda8dfa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcda407da20 0x7fcda40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.976 7fcda8dfa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40e4d80 0x7fcda41b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda8dfa700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda4077a70 con 0x7fcda40e5180 2021-02-10T11:37:51.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda8dfa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda40777d0 con 0x7fcda407da20 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda8dfa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda407de20 con 0x7fcda40e4d80 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcda407da20 0x7fcda40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcda407da20 0x7fcda40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45778/0 (socket says 172.21.2.117:45778) 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 -- 172.21.2.117:0/2212725727 learned_addr learned my addr 172.21.2.117:0/2212725727 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 -- 172.21.2.117:0/2212725727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40e4d80 msgr2=0x7fcda41b00d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 --2- 172.21.2.117:0/2212725727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40e4d80 0x7fcda41b00d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 -- 172.21.2.117:0/2212725727 >> v1:172.21.2.117:6789/0 conn(0x7fcda40e5180 legacy=0x7fcda4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 -- 172.21.2.117:0/2212725727 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda41b06f0 con 0x7fcda407da20 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda2b13700 1 --2- 172.21.2.117:0/2212725727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda407da20 0x7fcda40771b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fcd8c006700 tx=0x7fcd8c009b00).ready entity=mon.0 client_cookie=4691d2a004f0113c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.977 7fcda1b11700 1 -- 172.21.2.117:0/2212725727 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd8c00c0c0 con 0x7fcda407da20 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda1b11700 1 -- 172.21.2.117:0/2212725727 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c00c400 con 0x7fcda407da20 2021-02-10T11:37:51.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda1b11700 1 -- 172.21.2.117:0/2212725727 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd8c00c8e0 con 0x7fcda407da20 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 -- 172.21.2.117:0/2212725727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda407da20 msgr2=0x7fcda40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 --2- 172.21.2.117:0/2212725727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda407da20 0x7fcda40771b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fcd8c006700 tx=0x7fcd8c009b00).stop 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 -- 172.21.2.117:0/2212725727 shutdown_connections 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 --2- 172.21.2.117:0/2212725727 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40e4d80 0x7fcda41b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 --2- 172.21.2.117:0/2212725727 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda407da20 0x7fcda40771b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 -- 172.21.2.117:0/2212725727 >> 172.21.2.117:0/2212725727 conn(0x7fcda40ce430 msgr2=0x7fcda40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 -- 172.21.2.117:0/2212725727 shutdown_connections 2021-02-10T11:37:51.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.978 7fcda8dfa700 1 -- 172.21.2.117:0/2212725727 wait complete. 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 Processor -- start 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 -- start start 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 0x7fcda40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcda409c0c0 0x7fcda409c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40a77c0 con 0x7fcda40a3db0 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40a5360 con 0x7fcda409c0c0 2021-02-10T11:37:51.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.993 7fcda8dfa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40a5580 con 0x7fcda40a39b0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2b13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 0x7fcda40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda3314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcda409c0c0 0x7fcda409c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45788/0 (socket says 172.21.2.117:45788) 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2b13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 0x7fcda40a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43464/0 (socket says 172.21.2.117:43464) 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2b13700 1 -- 172.21.2.117:0/1170366354 learned_addr learned my addr 172.21.2.117:0/1170366354 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 -- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 msgr2=0x7fcda40a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 0x7fcda40a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 -- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcda409c0c0 msgr2=0x7fcda409c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcda409c0c0 0x7fcda409c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 -- 172.21.2.117:0/1170366354 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda409cae0 con 0x7fcda40a3db0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda2312700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fcd94006e80 tx=0x7fcd9400b870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:51.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd940009b0 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94006240 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.994 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda409cae0 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd940054d0 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda409cae0 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd940059a0 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcd9b7fe700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 0x7fcd8801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcd94032860 con 0x7fcda40a3db0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.995 7fcda2b13700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 0x7fcd8801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:51.996 7fcda2b13700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 0x7fcd8801abd0 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:52.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.026 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda40a5360 con 0x7fcda40a3db0 2021-02-10T11:37:52.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.029 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd94006130 con 0x7fcda40a3db0 2021-02-10T11:37:52.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.157 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802530", "target": ["mgr", ""]}) v1 -- 0x7fcda40a77c0 con 0x7fcd880186d0 2021-02-10T11:37:52.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.161 7fcd9b7fe700 1 -- 172.21.2.117:0/1170366354 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fcd8c009290 con 0x7fcd880186d0 2021-02-10T11:37:52.146 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e 2021-02-10T11:37:52.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 msgr2=0x7fcd8801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 0x7fcd8801abd0 crc :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 msgr2=0x7fcda40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fcd94006e80 tx=0x7fcd9400b870).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 shutdown_connections 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd880186d0 0x7fcd8801abd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcda40a3db0 0x7fcda40a4d40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcda409c0c0 0x7fcda409c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 --2- 172.21.2.117:0/1170366354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcda40a39b0 0x7fcda40a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 >> 172.21.2.117:0/1170366354 conn(0x7fcda40bb240 msgr2=0x7fcda40a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 shutdown_connections 2021-02-10T11:37:52.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:52.167 7fcda8dfa700 1 -- 172.21.2.117:0/1170366354 wait complete. 2021-02-10T11:37:52.156 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802530 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e 2021-02-10T11:37:52.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:52.157 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:52.157 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:52.493 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:52.493 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:52.493 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:52.871 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:52.871 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:52.871 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237713 s, 44.1 MB/s 2021-02-10T11:37:52.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:52.873 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:52.873 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:53.209 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:53.209 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:53.209 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:53.584 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:53.585 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:53.585 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225851 s, 46.4 MB/s 2021-02-10T11:37:53.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:53.586 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:53.586 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:53.922 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:53.922 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:53.922 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:54.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:54.301 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:54.301 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256992 s, 40.8 MB/s 2021-02-10T11:37:54.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:54.302 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:54.302 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:54.639 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:54.639 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:54.639 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:55.015 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:55.016 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:55.016 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235483 s, 44.5 MB/s 2021-02-10T11:37:55.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:55.017 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:55.017 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:55.354 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:55.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:55.355 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:55.730 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:55.730 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:55.730 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0228061 s, 46.0 MB/s 2021-02-10T11:37:55.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:55.731 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:55.731 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:56.067 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:56.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:56.068 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230942 s, 45.4 MB/s 2021-02-10T11:37:56.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:56.445 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:56.446 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:56.780 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:56.781 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:56.781 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:57.157 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:57.157 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:57.157 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248498 s, 42.2 MB/s 2021-02-10T11:37:57.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:57.158 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:57.158 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:57.494 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:57.494 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:57.495 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:57.869 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:57.869 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:57.869 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0247377 s, 42.4 MB/s 2021-02-10T11:37:57.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:57.871 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:57.871 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:58.206 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:58.206 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:58.207 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:58.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:58.583 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:58.583 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235794 s, 44.5 MB/s 2021-02-10T11:37:58.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:37:58.584 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:37:58.584 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:37:58.920 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:37:58.920 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:37:58.920 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802530/c60a58a0-3d8f-4426-bc6d-a07e5698608e/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:37:59.297 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:37:59.297 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:37:59.297 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240949 s, 43.5 MB/s 2021-02-10T11:37:59.298 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802531 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.730 7fc67f0bf700 1 Processor -- start 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.730 7fc67f0bf700 1 -- start start 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.731 7fc67f0bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.731 7fc67f0bf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc6781b00f0 0x7fc6781b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.731 7fc67f0bf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc678077a70 con 0x7fc67807da20 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.731 7fc67f0bf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6780e53c0 con 0x7fc6781b00f0 2021-02-10T11:37:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.731 7fc67f0bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678079400 con 0x7fc678079000 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc677fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc677fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43472/0 (socket says 172.21.2.117:43472) 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc677fff700 1 -- 172.21.2.117:0/1046685690 learned_addr learned my addr 172.21.2.117:0/1046685690 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 --2- 172.21.2.117:0/1046685690 >> v2:172.21.2.125:3300/0 conn(0x7fc6781b00f0 0x7fc6781b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 -- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 msgr2=0x7fc6780e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 -- 172.21.2.117:0/1046685690 >> v1:172.21.2.125:6789/0 conn(0x7fc67807da20 legacy=0x7fc6780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 -- 172.21.2.117:0/1046685690 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67807de20 con 0x7fc6781b00f0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc677fff700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.732 7fc67d6dc700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b00f0 0x7fc6781b04f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc670006e50 tx=0x7fc670009e70).ready entity=mon.1 client_cookie=ac5016794aa927ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.733 7fc6777fe700 1 -- 172.21.2.117:0/1046685690 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc670006a20 con 0x7fc6781b00f0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.733 7fc6777fe700 1 -- 172.21.2.117:0/1046685690 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670006790 con 0x7fc6781b00f0 2021-02-10T11:37:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.733 7fc6777fe700 1 -- 172.21.2.117:0/1046685690 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc67000b700 con 0x7fc6781b00f0 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.733 7fc67f0bf700 1 -- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b00f0 msgr2=0x7fc6781b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.733 7fc67f0bf700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b00f0 0x7fc6781b04f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc670006e50 tx=0x7fc670009e70).stop 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 -- 172.21.2.117:0/1046685690 shutdown_connections 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b00f0 0x7fc6781b04f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 --2- 172.21.2.117:0/1046685690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc678079000 0x7fc6780e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 -- 172.21.2.117:0/1046685690 >> 172.21.2.117:0/1046685690 conn(0x7fc6780ce450 msgr2=0x7fc6780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 -- 172.21.2.117:0/1046685690 shutdown_connections 2021-02-10T11:37:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.734 7fc67f0bf700 1 -- 172.21.2.117:0/1046685690 wait complete. 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.748 7fc67f0bf700 1 Processor -- start 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.748 7fc67f0bf700 1 -- start start 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 0x7fc6781b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6781b6b20 0x7fc678106a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678106540 con 0x7fc678105870 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678192ea0 con 0x7fc6781b6100 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67f0bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678104690 con 0x7fc6781b6b20 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67cedb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67cedb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45796/0 (socket says 172.21.2.117:45796) 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67cedb700 1 -- 172.21.2.117:0/491629378 learned_addr learned my addr 172.21.2.117:0/491629378 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:37:59.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc67d6dc700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6781b6b20 0x7fc678106a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.749 7fc677fff700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 0x7fc6781b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 -- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6781b6b20 msgr2=0x7fc678106a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6781b6b20 0x7fc678106a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 -- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 msgr2=0x7fc6781b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 0x7fc6781b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 -- 172.21.2.117:0/491629378 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc678107060 con 0x7fc678105870 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc677fff700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 0x7fc6781b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67cedb700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc66000b5b0 tx=0x7fc66000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc660006240 con 0x7fc678105870 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6600054a0 con 0x7fc678105870 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6600058f0 con 0x7fc678105870 2021-02-10T11:37:59.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc678107060 con 0x7fc678105870 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.750 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6781075f0 con 0x7fc678105870 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.751 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc660006030 con 0x7fc678105870 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.751 7fc6757fa700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 0x7fc65c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.751 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc6600328d0 con 0x7fc678105870 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.751 7fc677fff700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 0x7fc65c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:37:59.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.751 7fc677fff700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 0x7fc65c01abc0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:37:59.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.780 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc678107dc0 con 0x7fc678105870 2021-02-10T11:37:59.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.783 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc660000c90 con 0x7fc678105870 2021-02-10T11:37:59.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.908 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802531", "target": ["mgr", ""]}) v1 -- 0x7fc678107dc0 con 0x7fc65c0186c0 2021-02-10T11:37:59.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.934 7fc6757fa700 1 -- 172.21.2.117:0/491629378 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc668006e40 con 0x7fc65c0186c0 2021-02-10T11:37:59.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.940 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 msgr2=0x7fc65c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.940 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 0x7fc65c01abc0 crc :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.940 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 msgr2=0x7fc678105c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:37:59.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.940 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fc66000b5b0 tx=0x7fc66000b950).stop 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 shutdown_connections 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6781b6b20 0x7fc678106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6781b6100 0x7fc6781b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc65c0186c0 0x7fc65c01abc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 --2- 172.21.2.117:0/491629378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc678105870 0x7fc678105c70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 >> 172.21.2.117:0/491629378 conn(0x7fc6780ffe30 msgr2=0x7fc6780c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 shutdown_connections 2021-02-10T11:37:59.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:37:59.941 7fc67f0bf700 1 -- 172.21.2.117:0/491629378 wait complete. 2021-02-10T11:37:59.930 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802531 2021-02-10T11:38:00.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 Processor -- start 2021-02-10T11:38:00.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 -- start start 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f158807da20 0x7f15880771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880e51a0 0x7f1588079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1588077a70 con 0x7f15880e4da0 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15880777d0 con 0x7f158807da20 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158d2a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15881b08e0 con 0x7f15880e51a0 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f158807da20 0x7f15880771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158687c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f15880e4da0 0x7f15881b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46054/0 (socket says 172.21.2.117:46054) 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.363 7f158687c700 1 -- 172.21.2.117:0/113453059 learned_addr learned my addr 172.21.2.117:0/113453059 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158707d700 1 -- 172.21.2.117:0/113453059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880e51a0 msgr2=0x7f1588079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158707d700 1 --2- 172.21.2.117:0/113453059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880e51a0 0x7f1588079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158707d700 1 -- 172.21.2.117:0/113453059 >> v1:172.21.2.117:6789/0 conn(0x7f15880e4da0 legacy=0x7f15881b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158707d700 1 -- 172.21.2.117:0/113453059 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158807de20 con 0x7f158807da20 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158707d700 1 --2- 172.21.2.117:0/113453059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f158807da20 0x7f15880771b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f1578006650 tx=0x7f1578009a80).ready entity=mon.0 client_cookie=3a92f5d322470905 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158607b700 1 -- 172.21.2.117:0/113453059 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157800c0c0 con 0x7f158807da20 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158607b700 1 -- 172.21.2.117:0/113453059 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157800c340 con 0x7f158807da20 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158607b700 1 -- 172.21.2.117:0/113453059 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f157800c860 con 0x7f158807da20 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158d2a3700 1 -- 172.21.2.117:0/113453059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f158807da20 msgr2=0x7f15880771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.364 7f158d2a3700 1 --2- 172.21.2.117:0/113453059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f158807da20 0x7f15880771b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f1578006650 tx=0x7f1578009a80).stop 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 -- 172.21.2.117:0/113453059 shutdown_connections 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 --2- 172.21.2.117:0/113453059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15880e51a0 0x7f1588079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 --2- 172.21.2.117:0/113453059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f158807da20 0x7f15880771b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 -- 172.21.2.117:0/113453059 >> 172.21.2.117:0/113453059 conn(0x7f15880ce450 msgr2=0x7f15880e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:00.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 -- 172.21.2.117:0/113453059 shutdown_connections 2021-02-10T11:38:00.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.365 7f158d2a3700 1 -- 172.21.2.117:0/113453059 wait complete. 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.379 7f158d2a3700 1 Processor -- start 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 -- start start 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f158808a470 0x7f15880a7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158808a870 0x7f15880a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158808cf40 con 0x7f15880a8170 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158808dae0 con 0x7f158808a870 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158d2a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158808dcc0 con 0x7f158808a470 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45808/0 (socket says 172.21.2.117:45808) 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 -- 172.21.2.117:0/3626018560 learned_addr learned my addr 172.21.2.117:0/3626018560 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 -- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f158808a470 msgr2=0x7f15880a7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f158808a470 0x7f15880a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 -- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158808a870 msgr2=0x7f15880a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158808a870 0x7f15880a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.380 7f158787e700 1 -- 172.21.2.117:0/3626018560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158808dee0 con 0x7f15880a8170 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f158787e700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f1580009960 tx=0x7f158000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1580003030 con 0x7f15880a8170 2021-02-10T11:38:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1580001310 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158808dee0 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1580001770 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.381 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158808dee0 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.382 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1580000d10 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.382 7f157ffff700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 0x7f156c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.382 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1580032b00 con 0x7f15880a8170 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.382 7f158707d700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 0x7f156c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.383 7f158707d700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 0x7f156c01abc0 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.413 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15881032d0 con 0x7f15880a8170 2021-02-10T11:38:00.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.416 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f158000be30 con 0x7f15880a8170 2021-02-10T11:38:00.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.545 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802531", "target": ["mgr", ""]}) v1 -- 0x7f158808cf40 con 0x7f156c0186c0 2021-02-10T11:38:00.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.548 7f157ffff700 1 -- 172.21.2.117:0/3626018560 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f1578006060 con 0x7f156c0186c0 2021-02-10T11:38:00.530 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 msgr2=0x7f156c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 0x7f156c01abc0 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 msgr2=0x7f158808d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f1580009960 tx=0x7f158000bcf0).stop 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 shutdown_connections 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f156c0186c0 0x7f156c01abc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15880a8170 0x7f158808d4c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f158808a870 0x7f15880a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 --2- 172.21.2.117:0/3626018560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f158808a470 0x7f15880a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:00.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.550 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 >> 172.21.2.117:0/3626018560 conn(0x7f15880a9210 msgr2=0x7f15880a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:00.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.551 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 shutdown_connections 2021-02-10T11:38:00.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:00.551 7f158d2a3700 1 -- 172.21.2.117:0/3626018560 wait complete. 2021-02-10T11:38:00.539 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802531 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415 2021-02-10T11:38:00.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:00.540 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:00.540 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:00.873 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:00.874 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:00.874 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:01.256 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:01.256 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:01.256 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255539 s, 41.0 MB/s 2021-02-10T11:38:01.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:01.257 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:01.257 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:01.591 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:01.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:01.592 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:01.966 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:01.967 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:01.967 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0258459 s, 40.6 MB/s 2021-02-10T11:38:01.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:01.968 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:01.968 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:02.302 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:02.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:02.302 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253232 s, 41.4 MB/s 2021-02-10T11:38:02.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:02.675 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:02.675 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:03.009 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:03.009 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:03.009 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:03.386 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:03.386 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:03.387 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0262621 s, 39.9 MB/s 2021-02-10T11:38:03.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:03.388 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:03.388 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:03.722 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:03.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:03.722 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:03.796 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:03.799 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:04.096 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:04.096 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:04.096 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243173 s, 43.1 MB/s 2021-02-10T11:38:04.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:04.097 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:04.097 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:04.471 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:04.471 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:04.472 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:04.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:04.847 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:04.847 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249234 s, 42.1 MB/s 2021-02-10T11:38:04.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:04.848 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:04.848 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:05.184 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:05.184 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:05.185 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:05.569 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:05.569 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:05.569 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253368 s, 41.4 MB/s 2021-02-10T11:38:05.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:05.570 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:05.571 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:05.905 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:05.905 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:05.905 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:06.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:06.281 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:06.281 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256265 s, 40.9 MB/s 2021-02-10T11:38:06.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:06.282 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:06.282 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:06.616 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:06.617 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:06.617 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:06.990 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224279 s, 46.8 MB/s 2021-02-10T11:38:06.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:06.992 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:06.992 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:07.326 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:07.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:07.327 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802531/5e501aad-a608-4000-8e3a-4b38ded97415/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:07.698 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:07.698 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:07.698 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248325 s, 42.2 MB/s 2021-02-10T11:38:07.699 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802532 2021-02-10T11:38:08.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.129 7fb56b439700 1 Processor -- start 2021-02-10T11:38:08.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 -- start start 2021-02-10T11:38:08.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb56407da20 0x7fb5640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb5640e51a0 0x7fb564079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb564077a70 con 0x7fb5640e4da0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5640e5c20 con 0x7fb56407da20 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb56b439700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5640777d0 con 0x7fb5640e51a0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb569255700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb56407da20 0x7fb5640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb568a54700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb5640e4da0 0x7fb5641b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46066/0 (socket says 172.21.2.117:46066) 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.130 7fb568a54700 1 -- 172.21.2.117:0/654573528 learned_addr learned my addr 172.21.2.117:0/654573528 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb55bfff700 1 -- 172.21.2.117:0/654573528 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167247234 0 0) 0x7fb5540009b0 con 0x7fb5640e4da0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb55bfff700 1 -- 172.21.2.117:0/654573528 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb550002fe0 con 0x7fb5640e4da0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb569255700 1 -- 172.21.2.117:0/654573528 >> v2:172.21.2.125:3300/0 conn(0x7fb5640e51a0 msgr2=0x7fb564079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb569255700 1 --2- 172.21.2.117:0/654573528 >> v2:172.21.2.125:3300/0 conn(0x7fb5640e51a0 0x7fb564079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb569255700 1 -- 172.21.2.117:0/654573528 >> v1:172.21.2.117:6789/0 conn(0x7fb5640e4da0 legacy=0x7fb5641b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb569255700 1 -- 172.21.2.117:0/654573528 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5641b08e0 con 0x7fb56407da20 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb569255700 1 --2- 172.21.2.117:0/654573528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56407da20 0x7fb5640771b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fb560006650 tx=0x7fb560009a80).ready entity=mon.0 client_cookie=dab0abe09f0942da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb55bfff700 1 -- 172.21.2.117:0/654573528 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5600009b0 con 0x7fb56407da20 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb55bfff700 1 -- 172.21.2.117:0/654573528 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56000c0c0 con 0x7fb56407da20 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb55bfff700 1 -- 172.21.2.117:0/654573528 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb56000c5e0 con 0x7fb56407da20 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb56b439700 1 -- 172.21.2.117:0/654573528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56407da20 msgr2=0x7fb5640771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.131 7fb56b439700 1 --2- 172.21.2.117:0/654573528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56407da20 0x7fb5640771b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7fb560006650 tx=0x7fb560009a80).stop 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 -- 172.21.2.117:0/654573528 shutdown_connections 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 --2- 172.21.2.117:0/654573528 >> v2:172.21.2.125:3300/0 conn(0x7fb5640e51a0 0x7fb564079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 --2- 172.21.2.117:0/654573528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56407da20 0x7fb5640771b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 -- 172.21.2.117:0/654573528 >> 172.21.2.117:0/654573528 conn(0x7fb5640ce450 msgr2=0x7fb5640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 -- 172.21.2.117:0/654573528 shutdown_connections 2021-02-10T11:38:08.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.132 7fb56b439700 1 -- 172.21.2.117:0/654573528 wait complete. 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.146 7fb56b439700 1 Processor -- start 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 -- start start 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 0x7fb5641b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 0x7fb5640b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640b8da0 con 0x7fb5641b5b50 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640a4140 con 0x7fb5641b5130 2021-02-10T11:38:08.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb56b439700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5640a4320 con 0x7fb5641b5f50 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb568a54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb568a54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45822/0 (socket says 172.21.2.117:45822) 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.147 7fb568a54700 1 -- 172.21.2.117:0/1651789202 learned_addr learned my addr 172.21.2.117:0/1651789202 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb569a56700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 0x7fb5640b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb569255700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 0x7fb5641b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 -- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 msgr2=0x7fb5640b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 0x7fb5640b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 -- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 msgr2=0x7fb5641b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 0x7fb5641b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 -- 172.21.2.117:0/1651789202 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640b62b0 con 0x7fb5641b5b50 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb569a56700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 0x7fb5640b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb569255700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 0x7fb5641b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb568a54700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fb554001c60 tx=0x7fb55400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb554000e00 con 0x7fb5641b5b50 2021-02-10T11:38:08.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.148 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55400acc0 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5540045d0 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb56b439700 1 -- 172.21.2.117:0/1651789202 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5640b62b0 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb56b439700 1 -- 172.21.2.117:0/1651789202 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5640b6840 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb554004140 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb559ffb700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 0x7fb54401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.149 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb554032640 con 0x7fb5641b5b50 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.150 7fb569255700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 0x7fb54401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.150 7fb569255700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 0x7fb54401abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.180 7fb56b439700 1 -- 172.21.2.117:0/1651789202 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5641b4f00 con 0x7fb5641b5b50 2021-02-10T11:38:08.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.182 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb554024fe0 con 0x7fb5641b5b50 2021-02-10T11:38:08.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.306 7fb56b439700 1 -- 172.21.2.117:0/1651789202 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802532", "target": ["mgr", ""]}) v1 -- 0x7fb5640b6840 con 0x7fb5440186c0 2021-02-10T11:38:08.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.333 7fb559ffb700 1 -- 172.21.2.117:0/1651789202 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb560006310 con 0x7fb5440186c0 2021-02-10T11:38:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.339 7fb56b439700 1 -- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 msgr2=0x7fb54401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.339 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 0x7fb54401abc0 crc :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.339 7fb56b439700 1 -- 172.21.2.117:0/1651789202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 msgr2=0x7fb5640b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.339 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fb554001c60 tx=0x7fb55400a8d0).stop 2021-02-10T11:38:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 -- 172.21.2.117:0/1651789202 shutdown_connections 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5440186c0 0x7fb54401abc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5641b5b50 0x7fb5640b5670 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5641b5f50 0x7fb5640b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 --2- 172.21.2.117:0/1651789202 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5641b5130 0x7fb5641b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 -- 172.21.2.117:0/1651789202 >> 172.21.2.117:0/1651789202 conn(0x7fb56408bbc0 msgr2=0x7fb5640c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 -- 172.21.2.117:0/1651789202 shutdown_connections 2021-02-10T11:38:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.340 7fb56b439700 1 -- 172.21.2.117:0/1651789202 wait complete. 2021-02-10T11:38:08.329 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802532 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 Processor -- start 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 -- start start 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fec580d1530 0x7fec580d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec58077a70 con 0x7fec58079000 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec58079400 con 0x7fec5807da20 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.761 7fec5db78700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5807de20 con 0x7fec580d1530 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec56ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fec58079000 0x7fec580e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46080/0 (socket says 172.21.2.117:46080) 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec56ffd700 1 -- 172.21.2.117:0/4099261935 learned_addr learned my addr 172.21.2.117:0/4099261935 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:08.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823988035 0 0) 0x7fec48001180 con 0x7fec58079000 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec57fff700 1 --2- 172.21.2.117:0/4099261935 >> v2:172.21.2.125:3300/0 conn(0x7fec580d1530 0x7fec580d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec44002fe0 con 0x7fec58079000 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 499272997 0 0) 0x7fec48001180 con 0x7fec58079000 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 >> v2:172.21.2.125:3300/0 conn(0x7fec580d1530 msgr2=0x7fec580d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 --2- 172.21.2.117:0/4099261935 >> v2:172.21.2.125:3300/0 conn(0x7fec580d1530 0x7fec580d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 >> v1:172.21.2.125:6789/0 conn(0x7fec5807da20 legacy=0x7fec580771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec581b00f0 con 0x7fec58079000 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fec48001680 con 0x7fec58079000 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 >> v1:172.21.2.117:6789/0 conn(0x7fec58079000 legacy=0x7fec580e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec44003a60 0x7fec44005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 0x7fec44008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec440095b0 0x7fec4400bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4400c320 con 0x7fec44006800 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4400c540 con 0x7fec440095b0 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.762 7fec567fc700 1 -- 172.21.2.117:0/4099261935 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4400c760 con 0x7fec44003a60 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 0x7fec44008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec577fe700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec44003a60 0x7fec44005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec57fff700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec440095b0 0x7fec4400bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 -- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec44003a60 msgr2=0x7fec44005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec44003a60 0x7fec44005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 -- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec440095b0 msgr2=0x7fec4400bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec440095b0 0x7fec4400bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 -- 172.21.2.117:0/4099261935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440509a0 con 0x7fec44006800 2021-02-10T11:38:08.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec57fff700 1 -- 172.21.2.117:0/4099261935 reap_dead start 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec56ffd700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 0x7fec44008cb0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fec480078f0 tx=0x7fec48007c90).ready entity=mon.0 client_cookie=34fb86de06b94fd1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec48000ce0 con 0x7fec44006800 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48007150 con 0x7fec44006800 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec567fc700 1 -- 172.21.2.117:0/4099261935 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec48004590 con 0x7fec44006800 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec5db78700 1 -- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 msgr2=0x7fec44008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.763 7fec5db78700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 0x7fec44008cb0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fec480078f0 tx=0x7fec48007c90).stop 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.764 7fec5db78700 1 -- 172.21.2.117:0/4099261935 shutdown_connections 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.764 7fec5db78700 1 --2- 172.21.2.117:0/4099261935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec44006800 0x7fec44008cb0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.764 7fec5db78700 1 -- 172.21.2.117:0/4099261935 >> 172.21.2.117:0/4099261935 conn(0x7fec580ce450 msgr2=0x7fec580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.764 7fec5db78700 1 -- 172.21.2.117:0/4099261935 shutdown_connections 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.764 7fec5db78700 1 -- 172.21.2.117:0/4099261935 wait complete. 2021-02-10T11:38:08.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.778 7fec5db78700 1 Processor -- start 2021-02-10T11:38:08.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 -- start start 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec580a7820 0x7fec58092030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec580a7c20 0x7fec58093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec58092a50 con 0x7fec58092650 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec58092c80 con 0x7fec580a7820 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec5db78700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec580a8340 con 0x7fec580a7c20 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45840/0 (socket says 172.21.2.117:45840) 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.779 7fec57fff700 1 -- 172.21.2.117:0/290101561 learned_addr learned my addr 172.21.2.117:0/290101561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:08.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 -- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec580a7c20 msgr2=0x7fec58093200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec580a7c20 0x7fec58093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 -- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec580a7820 msgr2=0x7fec58092030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec580a7820 0x7fec58092030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 -- 172.21.2.117:0/290101561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec580a8520 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec57fff700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7fec5000b970 tx=0x7fec5000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec50000f90 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec500011f0 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec50001690 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec5db78700 1 -- 172.21.2.117:0/290101561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec580a8520 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.780 7fec5db78700 1 -- 172.21.2.117:0/290101561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec58094130 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.781 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec50000df0 con 0x7fec58092650 2021-02-10T11:38:08.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.781 7fec4ffff700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:08.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.781 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fec50032aa0 con 0x7fec58092650 2021-02-10T11:38:08.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.781 7fec577fe700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:08.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.781 7fec577fe700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:08.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.811 7fec5db78700 1 -- 172.21.2.117:0/290101561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58092a50 con 0x7fec58092650 2021-02-10T11:38:08.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.814 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec50006720 con 0x7fec58092650 2021-02-10T11:38:08.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.941 7fec5db78700 1 -- 172.21.2.117:0/290101561 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802532", "target": ["mgr", ""]}) v1 -- 0x7fec58092c80 con 0x7fec3c0186c0 2021-02-10T11:38:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.945 7fec4ffff700 1 -- 172.21.2.117:0/290101561 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fec400069f0 con 0x7fec3c0186c0 2021-02-10T11:38:08.930 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b 2021-02-10T11:38:08.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 -- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 msgr2=0x7fec3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 -- 172.21.2.117:0/290101561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 msgr2=0x7fec58093820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7fec5000b970 tx=0x7fec5000bd10).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 -- 172.21.2.117:0/290101561 shutdown_connections 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec58092650 0x7fec58093820 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec580a7c20 0x7fec58093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 --2- 172.21.2.117:0/290101561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec580a7820 0x7fec58092030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:08.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.950 7fec5db78700 1 -- 172.21.2.117:0/290101561 >> 172.21.2.117:0/290101561 conn(0x7fec580a9210 msgr2=0x7fec580a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:08.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.951 7fec5db78700 1 -- 172.21.2.117:0/290101561 shutdown_connections 2021-02-10T11:38:08.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:08.951 7fec5db78700 1 -- 172.21.2.117:0/290101561 wait complete. 2021-02-10T11:38:08.940 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802532 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b 2021-02-10T11:38:08.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:08.940 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:08.940 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:09.274 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:09.274 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:09.275 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249271 s, 42.1 MB/s 2021-02-10T11:38:09.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:09.660 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:09.661 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:09.995 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:09.995 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:09.996 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:10.369 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:10.370 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:10.370 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243682 s, 43.0 MB/s 2021-02-10T11:38:10.371 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:10.371 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:10.371 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:10.706 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:10.706 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:10.706 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:11.082 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:11.082 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:11.083 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242704 s, 43.2 MB/s 2021-02-10T11:38:11.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:11.084 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:11.084 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:11.419 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:11.420 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:11.420 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:11.796 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245814 s, 42.7 MB/s 2021-02-10T11:38:11.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:11.797 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:11.798 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:12.133 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:12.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:12.133 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:12.505 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:12.505 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:12.505 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245772 s, 42.7 MB/s 2021-02-10T11:38:12.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:12.506 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:12.506 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:12.839 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:12.840 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:12.840 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:13.216 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:13.216 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:13.216 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0248436 s, 42.2 MB/s 2021-02-10T11:38:13.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:13.217 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:13.218 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:13.552 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:13.552 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:13.552 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:13.925 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:13.926 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:13.926 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.022839 s, 45.9 MB/s 2021-02-10T11:38:13.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:13.927 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:13.927 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:14.262 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:14.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:14.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:14.635 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:14.635 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:14.635 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225673 s, 46.5 MB/s 2021-02-10T11:38:14.636 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:14.637 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:14.637 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:14.972 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:14.972 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:14.973 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:15.346 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:15.347 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:15.347 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0227544 s, 46.1 MB/s 2021-02-10T11:38:15.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:15.348 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:15.348 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:15.682 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:15.682 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:15.682 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802532/e54ef9ca-cf9c-4219-8547-f5b93da42a2b/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:16.057 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:16.057 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:16.057 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234914 s, 44.6 MB/s 2021-02-10T11:38:16.058 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000802533 2021-02-10T11:38:16.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 Processor -- start 2021-02-10T11:38:16.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 -- start start 2021-02-10T11:38:16.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155807da20 0x7f15580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f15580e4da0 0x7f15581b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1558077a70 con 0x7f15580e51a0 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580777d0 con 0x7f155807da20 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f155e723700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155807de20 con 0x7f15580e4da0 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f1557fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155807da20 0x7f15580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f1557fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155807da20 0x7f15580771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45848/0 (socket says 172.21.2.117:45848) 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.488 7f1557fff700 1 -- 172.21.2.117:0/3996020279 learned_addr learned my addr 172.21.2.117:0/3996020279 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1557fff700 1 -- 172.21.2.117:0/3996020279 >> v2:172.21.2.125:3300/0 conn(0x7f15580e4da0 msgr2=0x7f15581b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1557fff700 1 --2- 172.21.2.117:0/3996020279 >> v2:172.21.2.125:3300/0 conn(0x7f15580e4da0 0x7f15581b00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1557fff700 1 -- 172.21.2.117:0/3996020279 >> v1:172.21.2.125:6789/0 conn(0x7f15580e51a0 legacy=0x7f1558077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:16.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1557fff700 1 -- 172.21.2.117:0/3996020279 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15581b0710 con 0x7f155807da20 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1557fff700 1 --2- 172.21.2.117:0/3996020279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155807da20 0x7f15580771b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f1548006700 tx=0x7f1548009b00).ready entity=mon.0 client_cookie=5cf7b49da10875a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1556ffd700 1 -- 172.21.2.117:0/3996020279 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f154800c0c0 con 0x7f155807da20 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1556ffd700 1 -- 172.21.2.117:0/3996020279 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154800c340 con 0x7f155807da20 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f1556ffd700 1 -- 172.21.2.117:0/3996020279 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f154800c820 con 0x7f155807da20 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f155e723700 1 -- 172.21.2.117:0/3996020279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155807da20 msgr2=0x7f15580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.489 7f155e723700 1 --2- 172.21.2.117:0/3996020279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155807da20 0x7f15580771b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7f1548006700 tx=0x7f1548009b00).stop 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 -- 172.21.2.117:0/3996020279 shutdown_connections 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 --2- 172.21.2.117:0/3996020279 >> v2:172.21.2.125:3300/0 conn(0x7f15580e4da0 0x7f15581b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 --2- 172.21.2.117:0/3996020279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155807da20 0x7f15580771b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 -- 172.21.2.117:0/3996020279 >> 172.21.2.117:0/3996020279 conn(0x7f15580ce450 msgr2=0x7f15580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 -- 172.21.2.117:0/3996020279 shutdown_connections 2021-02-10T11:38:16.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.490 7f155e723700 1 -- 172.21.2.117:0/3996020279 wait complete. 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.504 7f155e723700 1 Processor -- start 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 -- start start 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15580b1340 0x7f155818b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155818b830 0x7f1558187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580b18c0 con 0x7f15580b0f40 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155818ac90 con 0x7f155818b830 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155e723700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1558107f50 con 0x7f15580b1340 2021-02-10T11:38:16.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f1557fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f1557fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45854/0 (socket says 172.21.2.117:45854) 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f1557fff700 1 -- 172.21.2.117:0/311096521 learned_addr learned my addr 172.21.2.117:0/311096521 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f15577fe700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15580b1340 0x7f155818b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.505 7f155cd40700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155818b830 0x7f1558187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 -- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15580b1340 msgr2=0x7f155818b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15580b1340 0x7f155818b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 -- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155818b830 msgr2=0x7f1558187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155818b830 0x7f1558187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 -- 172.21.2.117:0/311096521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558108130 con 0x7f15580b0f40 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1557fff700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f1548006f20 tx=0x7f154800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1548000c90 con 0x7f15580b0f40 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548009100 con 0x7f15580b0f40 2021-02-10T11:38:16.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f155e723700 1 -- 172.21.2.117:0/311096521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1558108130 con 0x7f15580b0f40 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1548002220 con 0x7f15580b0f40 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.506 7f155e723700 1 -- 172.21.2.117:0/311096521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1558108130 con 0x7f15580b0f40 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.507 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f154800a000 con 0x7f15580b0f40 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.507 7f1554ff9700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 0x7f153801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.507 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1548032770 con 0x7f15580b0f40 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.507 7f15577fe700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 0x7f153801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:16.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.507 7f15577fe700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 0x7f153801ac10 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:16.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.538 7f155e723700 1 -- 172.21.2.117:0/311096521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1558077a70 con 0x7f15580b0f40 2021-02-10T11:38:16.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.540 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1548000c90 con 0x7f15580b0f40 2021-02-10T11:38:16.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.665 7f155e723700 1 -- 172.21.2.117:0/311096521 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802533", "target": ["mgr", ""]}) v1 -- 0x7f1558077a70 con 0x7f1538018730 2021-02-10T11:38:16.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.691 7f1554ff9700 1 -- 172.21.2.117:0/311096521 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f154c000ce0 con 0x7f1538018730 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 -- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 msgr2=0x7f153801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 0x7f153801ac10 crc :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 -- 172.21.2.117:0/311096521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 msgr2=0x7f1558107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f1548006f20 tx=0x7f154800b990).stop 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 -- 172.21.2.117:0/311096521 shutdown_connections 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155818b830 0x7f1558187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15580b1340 0x7f155818b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1538018730 0x7f153801ac10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 --2- 172.21.2.117:0/311096521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15580b0f40 0x7f1558107930 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:16.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.698 7f155e723700 1 -- 172.21.2.117:0/311096521 >> 172.21.2.117:0/311096521 conn(0x7f15580a1880 msgr2=0x7f15580b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:16.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.699 7f155e723700 1 -- 172.21.2.117:0/311096521 shutdown_connections 2021-02-10T11:38:16.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:16.699 7f155e723700 1 -- 172.21.2.117:0/311096521 wait complete. 2021-02-10T11:38:16.689 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000802533 2021-02-10T11:38:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.118 7fe2172f1700 1 Processor -- start 2021-02-10T11:38:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.118 7fe2172f1700 1 -- start start 2021-02-10T11:38:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe21007da20 0x7fe2100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100e4da0 0x7fe2101b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2100e51a0 0x7fe210077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe210077a70 con 0x7fe2100e51a0 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2100777d0 con 0x7fe21007da20 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe2172f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21007de20 con 0x7fe2100e4da0 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe21590e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2100e51a0 0x7fe210077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe21510d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe21007da20 0x7fe2100771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe21590e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2100e51a0 0x7fe210077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45862/0 (socket says 172.21.2.117:45862) 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe21590e700 1 -- 172.21.2.117:0/1173083142 learned_addr learned my addr 172.21.2.117:0/1173083142 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.119 7fe21490c700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100e4da0 0x7fe2101b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 -- 172.21.2.117:0/1173083142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100e4da0 msgr2=0x7fe2101b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100e4da0 0x7fe2101b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 -- 172.21.2.117:0/1173083142 >> v2:172.21.2.125:3300/0 conn(0x7fe21007da20 msgr2=0x7fe2100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 --2- 172.21.2.117:0/1173083142 >> v2:172.21.2.125:3300/0 conn(0x7fe21007da20 0x7fe2100771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 -- 172.21.2.117:0/1173083142 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2101b0710 con 0x7fe2100e51a0 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe21590e700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100e51a0 0x7fe210077ff0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fe20c006b80 tx=0x7fe20c009b90).ready entity=mon.0 client_cookie=356ca6820a9d61ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe207fff700 1 -- 172.21.2.117:0/1173083142 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe20c0022e0 con 0x7fe2100e51a0 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe207fff700 1 -- 172.21.2.117:0/1173083142 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c002620 con 0x7fe2100e51a0 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe207fff700 1 -- 172.21.2.117:0/1173083142 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe20c002b00 con 0x7fe2100e51a0 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe2172f1700 1 -- 172.21.2.117:0/1173083142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100e51a0 msgr2=0x7fe210077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.120 7fe2172f1700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100e51a0 0x7fe210077ff0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fe20c006b80 tx=0x7fe20c009b90).stop 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 -- 172.21.2.117:0/1173083142 shutdown_connections 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100e51a0 0x7fe210077ff0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 --2- 172.21.2.117:0/1173083142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100e4da0 0x7fe2101b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 --2- 172.21.2.117:0/1173083142 >> v2:172.21.2.125:3300/0 conn(0x7fe21007da20 0x7fe2100771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 -- 172.21.2.117:0/1173083142 >> 172.21.2.117:0/1173083142 conn(0x7fe2100ce450 msgr2=0x7fe2100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:17.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 -- 172.21.2.117:0/1173083142 shutdown_connections 2021-02-10T11:38:17.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.121 7fe2172f1700 1 -- 172.21.2.117:0/1173083142 wait complete. 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.135 7fe2172f1700 1 Processor -- start 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.135 7fe2172f1700 1 -- start start 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2100b1ee0 0x7fe21008e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100b1360 0x7fe21008e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100b1960 con 0x7fe2100990f0 2021-02-10T11:38:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101a4670 con 0x7fe2100b1ee0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe2172f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2101a4850 con 0x7fe2100b1360 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe21510d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe21510d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45866/0 (socket says 172.21.2.117:45866) 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe21510d700 1 -- 172.21.2.117:0/2444678391 learned_addr learned my addr 172.21.2.117:0/2444678391 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe21590e700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100b1360 0x7fe21008e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.136 7fe21490c700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2100b1ee0 0x7fe21008e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 -- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100b1360 msgr2=0x7fe21008e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100b1360 0x7fe21008e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 -- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2100b1ee0 msgr2=0x7fe21008e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2100b1ee0 0x7fe21008e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 -- 172.21.2.117:0/2444678391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210104300 con 0x7fe2100990f0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe21510d700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fe20000b560 tx=0x7fe20000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe2000009b0 con 0x7fe2100990f0 2021-02-10T11:38:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20000bdc0 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe200002030 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe210104300 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.137 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe210104300 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.138 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe200000c90 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.138 7fe205ffb700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.138 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe200032720 con 0x7fe2100990f0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.138 7fe21490c700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:17.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.139 7fe21490c700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 0x7fe1f001abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:17.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.169 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe210103280 con 0x7fe2100990f0 2021-02-10T11:38:17.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.172 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe20002fab0 con 0x7fe2100990f0 2021-02-10T11:38:17.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.299 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802533", "target": ["mgr", ""]}) v1 -- 0x7fe2101a4670 con 0x7fe1f00186c0 2021-02-10T11:38:17.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.303 7fe205ffb700 1 -- 172.21.2.117:0/2444678391 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7fe2080090e0 con 0x7fe1f00186c0 2021-02-10T11:38:17.289 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d 2021-02-10T11:38:17.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 msgr2=0x7fe1f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 0x7fe1f001abc0 crc :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 msgr2=0x7fe2101a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:17.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0x7fe20000b560 tx=0x7fe20000b900).stop 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 shutdown_connections 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2100b1360 0x7fe21008e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe2100b1ee0 0x7fe21008e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1f00186c0 0x7fe1f001abc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 --2- 172.21.2.117:0/2444678391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2100990f0 0x7fe2101a4050 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 >> 172.21.2.117:0/2444678391 conn(0x7fe2100a1850 msgr2=0x7fe2100bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 shutdown_connections 2021-02-10T11:38:17.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:17.309 7fe2172f1700 1 -- 172.21.2.117:0/2444678391 wait complete. 2021-02-10T11:38:17.299 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000802533 with 10 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d 2021-02-10T11:38:17.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:17.299 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:17.300 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:17.634 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:17.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:17.634 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:18.013 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:18.014 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:18.014 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024993 s, 42.0 MB/s 2021-02-10T11:38:18.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:18.015 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:18.015 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:18.349 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:18.349 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:18.349 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:18.724 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:18.725 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:18.725 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023677 s, 44.3 MB/s 2021-02-10T11:38:18.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:18.726 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:18.726 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:19.061 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:19.061 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:19.061 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:19.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:19.440 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:19.440 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0264372 s, 39.7 MB/s 2021-02-10T11:38:19.441 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:19.441 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:19.441 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:19.776 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:19.776 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:19.777 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:20.152 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:20.152 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:20.152 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253658 s, 41.3 MB/s 2021-02-10T11:38:20.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:20.153 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:20.154 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:20.487 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:20.488 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:20.488 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:20.861 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:20.862 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:20.862 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230924 s, 45.4 MB/s 2021-02-10T11:38:20.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:20.864 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:20.864 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:21.199 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:21.199 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:21.200 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:21.576 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:21.576 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:21.576 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244889 s, 42.8 MB/s 2021-02-10T11:38:21.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:21.578 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:21.578 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:21.910 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:21.911 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:21.911 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:22.282 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:22.283 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:22.283 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239596 s, 43.8 MB/s 2021-02-10T11:38:22.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:22.284 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:22.284 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:22.618 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:22.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:22.618 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:22.991 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:22.991 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:22.991 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0259368 s, 40.4 MB/s 2021-02-10T11:38:22.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:22.992 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:22.992 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:23.326 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:23.326 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:23.326 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:23.701 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:23.701 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:23.701 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235196 s, 44.6 MB/s 2021-02-10T11:38:23.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:23.702 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:23.703 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:24.038 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:24.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:24.039 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000802533/d4ae84bf-009d-4d39-95fb-95232771214d/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:38:24.414 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:38:24.414 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:38:24.414 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0255644 s, 41.0 MB/s 2021-02-10T11:38:24.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:24.415 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:24.415 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:24.751 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:38:24.751 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:38:24.751 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:38:24.751 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:38:24.751 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:25.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[44954]: fuse finished with error 0 and tester_r 0 2021-02-10T11:38:25.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:38:25.162 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:38:25.162 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:25.495 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:38:25.495 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:38:25.495 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:38:31.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:38:31.598 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:38:31.933 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802434 2021-02-10T11:38:32.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 Processor -- start 2021-02-10T11:38:32.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 -- start start 2021-02-10T11:38:32.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49cc07da20 0x7f49cc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0d1d20 con 0x7f49cc0e51a0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc077a70 con 0x7f49cc0e4da0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.366 7f49d4318700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc0777d0 con 0x7f49cc07da20 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2935700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f49cc0e51a0 0x7f49cc0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46122/0 (socket says 172.21.2.117:46122) 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2935700 1 -- 172.21.2.117:0/1986220240 learned_addr learned my addr 172.21.2.117:0/1986220240 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2134700 1 --2- 172.21.2.117:0/1986220240 >> v2:172.21.2.117:3300/0 conn(0x7f49cc07da20 0x7f49cc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d1132700 1 -- 172.21.2.117:0/1986220240 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814462474 0 0) 0x7f49c4001500 con 0x7f49cc0e51a0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d1132700 1 -- 172.21.2.117:0/1986220240 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b8002fe0 con 0x7f49cc0e51a0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2134700 1 -- 172.21.2.117:0/1986220240 >> v1:172.21.2.125:6789/0 conn(0x7f49cc0e4da0 legacy=0x7f49cc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2134700 1 -- 172.21.2.117:0/1986220240 >> v1:172.21.2.117:6789/0 conn(0x7f49cc0e51a0 legacy=0x7f49cc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2134700 1 -- 172.21.2.117:0/1986220240 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc1b08e0 con 0x7f49cc07da20 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.367 7f49d2134700 1 --2- 172.21.2.117:0/1986220240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc07da20 0x7f49cc0771b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f49c0006650 tx=0x7f49c0009a80).ready entity=mon.0 client_cookie=19275a96e8bbd01a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.368 7f49d1132700 1 -- 172.21.2.117:0/1986220240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c00009b0 con 0x7f49cc07da20 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.368 7f49d1132700 1 -- 172.21.2.117:0/1986220240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000c0c0 con 0x7f49cc07da20 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.368 7f49d1132700 1 -- 172.21.2.117:0/1986220240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c000c520 con 0x7f49cc07da20 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.368 7f49d4318700 1 -- 172.21.2.117:0/1986220240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc07da20 msgr2=0x7f49cc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.368 7f49d4318700 1 --2- 172.21.2.117:0/1986220240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc07da20 0x7f49cc0771b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f49c0006650 tx=0x7f49c0009a80).stop 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.369 7f49d4318700 1 -- 172.21.2.117:0/1986220240 shutdown_connections 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.369 7f49d4318700 1 --2- 172.21.2.117:0/1986220240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc07da20 0x7f49cc0771b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.369 7f49d4318700 1 -- 172.21.2.117:0/1986220240 >> 172.21.2.117:0/1986220240 conn(0x7f49cc0ce450 msgr2=0x7f49cc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.369 7f49d4318700 1 -- 172.21.2.117:0/1986220240 shutdown_connections 2021-02-10T11:38:32.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.369 7f49d4318700 1 -- 172.21.2.117:0/1986220240 wait complete. 2021-02-10T11:38:32.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.383 7f49d4318700 1 Processor -- start 2021-02-10T11:38:32.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.383 7f49d4318700 1 -- start start 2021-02-10T11:38:32.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49cc0b0f40 0x7f49cc107930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 0x7f49cc18b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0b18c0 con 0x7f49cc18b830 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc18ac90 con 0x7f49cc0b0f40 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d4318700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc107f50 con 0x7f49cc0b1340 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d2935700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d2935700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45878/0 (socket says 172.21.2.117:45878) 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d2935700 1 -- 172.21.2.117:0/131890703 learned_addr learned my addr 172.21.2.117:0/131890703 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d2134700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49cc0b0f40 0x7f49cc107930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.384 7f49d1933700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 0x7f49cc18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 -- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 msgr2=0x7f49cc18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 0x7f49cc18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 -- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49cc0b0f40 msgr2=0x7f49cc107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49cc0b0f40 0x7f49cc107930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 -- 172.21.2.117:0/131890703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc108130 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d1933700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 0x7f49cc18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d2935700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f49c4009980 tx=0x7f49c400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c400be10 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c4006080 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d4318700 1 -- 172.21.2.117:0/131890703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc108130 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49d4318700 1 -- 172.21.2.117:0/131890703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc188570 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.385 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c4004720 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.386 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49c4006260 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.386 7f49beffd700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 0x7f49b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.386 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f49c4032c80 con 0x7f49cc18b830 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.386 7f49d2134700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 0x7f49b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.387 7f49d2134700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 0x7f49b401ac10 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.416 7f49d4318700 1 -- 172.21.2.117:0/131890703 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc077a70 con 0x7f49cc18b830 2021-02-10T11:38:32.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.419 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49c4000f20 con 0x7f49cc18b830 2021-02-10T11:38:32.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.549 7f49d4318700 1 -- 172.21.2.117:0/131890703 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802434", "target": ["mgr", ""]}) v1 -- 0x7f49cc108130 con 0x7f49b4018730 2021-02-10T11:38:32.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.562 7f49beffd700 1 -- 172.21.2.117:0/131890703 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f49c0006420 con 0x7f49b4018730 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 -- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 msgr2=0x7f49b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 0x7f49b401ac10 crc :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 -- 172.21.2.117:0/131890703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 msgr2=0x7f49cc187bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f49c4009980 tx=0x7f49c400bcd0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 -- 172.21.2.117:0/131890703 shutdown_connections 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49b4018730 0x7f49b401ac10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49cc18b830 0x7f49cc187bf0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49cc0b1340 0x7f49cc18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 --2- 172.21.2.117:0/131890703 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49cc0b0f40 0x7f49cc107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.565 7f49d4318700 1 -- 172.21.2.117:0/131890703 >> 172.21.2.117:0/131890703 conn(0x7f49cc0a1880 msgr2=0x7f49cc0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.566 7f49d4318700 1 -- 172.21.2.117:0/131890703 shutdown_connections 2021-02-10T11:38:32.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.566 7f49d4318700 1 -- 172.21.2.117:0/131890703 wait complete. 2021-02-10T11:38:32.556 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802435 2021-02-10T11:38:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 Processor -- start 2021-02-10T11:38:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 -- start start 2021-02-10T11:38:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff8079000 0x7efff80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efff81b00f0 0x7efff81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff8077a70 con 0x7efff807da20 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff80e53c0 con 0x7efff81b00f0 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7effff8bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff8079400 con 0x7efff8079000 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7efffd6db700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efff807da20 0x7efff80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46136/0 (socket says 172.21.2.117:46136) 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.986 7efffd6db700 1 -- 172.21.2.117:0/1487806130 learned_addr learned my addr 172.21.2.117:0/1487806130 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176050913 0 0) 0x7effec001180 con 0x7efff807da20 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe4002fe0 con 0x7efff807da20 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7efffceda700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff8079000 0x7efff80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7efffdedc700 1 --2- 172.21.2.117:0/1487806130 >> v2:172.21.2.125:3300/0 conn(0x7efff81b00f0 0x7efff81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2241546612 0 0) 0x7effec001180 con 0x7efff807da20 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff8079000 msgr2=0x7efff80e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff8079000 0x7efff80e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 >> v2:172.21.2.125:3300/0 conn(0x7efff81b00f0 msgr2=0x7efff81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 --2- 172.21.2.117:0/1487806130 >> v2:172.21.2.125:3300/0 conn(0x7efff81b00f0 0x7efff81b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff807de20 con 0x7efff807da20 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7effec001680 con 0x7efff807da20 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 >> v1:172.21.2.117:6789/0 conn(0x7efff807da20 legacy=0x7efff80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effe4003b10 0x7effe4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effe404a870 0x7effe404cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effe404d5a0 0x7effe404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.987 7effebfff700 1 -- 172.21.2.117:0/1487806130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe40502e0 con 0x7effe4003b10 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effebfff700 1 -- 172.21.2.117:0/1487806130 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe4050500 con 0x7effe404a870 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effebfff700 1 -- 172.21.2.117:0/1487806130 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe4050720 con 0x7effe404d5a0 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effe4003b10 0x7effe4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffceda700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effe404a870 0x7effe404cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffdedc700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effe404d5a0 0x7effe404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 -- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effe404d5a0 msgr2=0x7effe404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effe404d5a0 0x7effe404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 -- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effe404a870 msgr2=0x7effe404cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effe404a870 0x7effe404cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 -- 172.21.2.117:0/1487806130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40509c0 con 0x7effe4003b10 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffdedc700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effe404d5a0 0x7effe404fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7efffd6db700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effe4003b10 0x7effe4006010 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7effec007970 tx=0x7effec007d10).ready entity=mon.0 client_cookie=2a7ce59c3199e1f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effec0070c0 con 0x7effe4003b10 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec0043a0 con 0x7effe4003b10 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effebfff700 1 -- 172.21.2.117:0/1487806130 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effec0048c0 con 0x7effe4003b10 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effff8bf700 1 -- 172.21.2.117:0/1487806130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effe4003b10 msgr2=0x7effe4006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.988 7effff8bf700 1 --2- 172.21.2.117:0/1487806130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effe4003b10 0x7effe4006010 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7effec007970 tx=0x7effec007d10).stop 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.989 7efffdedc700 1 -- 172.21.2.117:0/1487806130 reap_dead start 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.989 7effff8bf700 1 -- 172.21.2.117:0/1487806130 shutdown_connections 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.989 7effff8bf700 1 -- 172.21.2.117:0/1487806130 >> 172.21.2.117:0/1487806130 conn(0x7efff80ce450 msgr2=0x7efff80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.989 7effff8bf700 1 -- 172.21.2.117:0/1487806130 shutdown_connections 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:32.989 7effff8bf700 1 -- 172.21.2.117:0/1487806130 wait complete. 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 Processor -- start 2021-02-10T11:38:32.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 -- start start 2021-02-10T11:38:32.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff80a47d0 0x7efff80a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efff80b77d0 0x7efff80b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff8189970 con 0x7efff80a51f0 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff8190aa0 con 0x7efff80b77d0 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7effff8bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff80b81f0 con 0x7efff80a47d0 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7efffceda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7efffceda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45898/0 (socket says 172.21.2.117:45898) 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.004 7efffceda700 1 -- 172.21.2.117:0/3578442788 learned_addr learned my addr 172.21.2.117:0/3578442788 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 -- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff80a47d0 msgr2=0x7efff80a4bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff80a47d0 0x7efff80a4bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 -- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efff80b77d0 msgr2=0x7efff80b7bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efff80b77d0 0x7efff80b7bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:32.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 -- 172.21.2.117:0/3578442788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80b8410 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7efffceda700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7efff400b610 tx=0x7efff400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff400bda0 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff4000c90 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efff4002220 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7effff8bf700 1 -- 172.21.2.117:0/3578442788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff80b8410 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.005 7effff8bf700 1 -- 172.21.2.117:0/3578442788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff80b8410 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.006 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efff40052e0 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.006 7effe9ffb700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 0x7effd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.006 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efff40326b0 con 0x7efff80a51f0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.006 7efffd6db700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 0x7effd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:32.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.007 7efffd6db700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 0x7effd801abc0 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.036 7effff8bf700 1 -- 172.21.2.117:0/3578442788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff8077a70 con 0x7efff80a51f0 2021-02-10T11:38:33.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.039 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efff4006bb0 con 0x7efff80a51f0 2021-02-10T11:38:33.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.164 7effff8bf700 1 -- 172.21.2.117:0/3578442788 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802435", "target": ["mgr", ""]}) v1 -- 0x7efff8077a70 con 0x7effd80186c0 2021-02-10T11:38:33.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.176 7effe9ffb700 1 -- 172.21.2.117:0/3578442788 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7effec0018c0 con 0x7effd80186c0 2021-02-10T11:38:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 msgr2=0x7effd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 0x7effd801abc0 crc :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 msgr2=0x7efff80a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7efff400b610 tx=0x7efff400b9b0).stop 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 shutdown_connections 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effd80186c0 0x7effd801abc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efff80a51f0 0x7efff80a55f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efff80b77d0 0x7efff80b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 --2- 172.21.2.117:0/3578442788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efff80a47d0 0x7efff80a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 >> 172.21.2.117:0/3578442788 conn(0x7efff80b5140 msgr2=0x7efff80a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 shutdown_connections 2021-02-10T11:38:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.182 7effff8bf700 1 -- 172.21.2.117:0/3578442788 wait complete. 2021-02-10T11:38:33.173 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802436 2021-02-10T11:38:33.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.602 7fef83ec2700 1 Processor -- start 2021-02-10T11:38:33.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.602 7fef83ec2700 1 -- start start 2021-02-10T11:38:33.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef83ec2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef7c07da20 0x7fef7c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef83ec2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fef7c0e4da0 0x7fef7c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef83ec2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c077a70 con 0x7fef7c0e51a0 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef83ec2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c0777d0 con 0x7fef7c07da20 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef83ec2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7c07de20 con 0x7fef7c0e4da0 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef824df700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fef7c0e51a0 0x7fef7c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46156/0 (socket says 172.21.2.117:46156) 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef7c07da20 0x7fef7c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef824df700 1 -- 172.21.2.117:0/850568894 learned_addr learned my addr 172.21.2.117:0/850568894 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef814dd700 1 --2- 172.21.2.117:0/850568894 >> v2:172.21.2.125:3300/0 conn(0x7fef7c0e4da0 0x7fef7c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 -- 172.21.2.117:0/850568894 >> v2:172.21.2.125:3300/0 conn(0x7fef7c0e4da0 msgr2=0x7fef7c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 --2- 172.21.2.117:0/850568894 >> v2:172.21.2.125:3300/0 conn(0x7fef7c0e4da0 0x7fef7c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 -- 172.21.2.117:0/850568894 >> v1:172.21.2.117:6789/0 conn(0x7fef7c0e51a0 legacy=0x7fef7c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef80cdc700 1 -- 172.21.2.117:0/850568894 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588474607 0 0) 0x7fef74001500 con 0x7fef7c0e51a0 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 -- 172.21.2.117:0/850568894 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c1b0710 con 0x7fef7c07da20 2021-02-10T11:38:33.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.603 7fef81cde700 1 --2- 172.21.2.117:0/850568894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c07da20 0x7fef7c0771b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fef70006650 tx=0x7fef70009a00).ready entity=mon.0 client_cookie=53281fef91f225a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef80cdc700 1 -- 172.21.2.117:0/850568894 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef700009b0 con 0x7fef7c07da20 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef80cdc700 1 -- 172.21.2.117:0/850568894 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7000c0c0 con 0x7fef7c07da20 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef80cdc700 1 -- 172.21.2.117:0/850568894 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef7000c5a0 con 0x7fef7c07da20 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 -- 172.21.2.117:0/850568894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c07da20 msgr2=0x7fef7c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 --2- 172.21.2.117:0/850568894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c07da20 0x7fef7c0771b0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fef70006650 tx=0x7fef70009a00).stop 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 -- 172.21.2.117:0/850568894 shutdown_connections 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 --2- 172.21.2.117:0/850568894 >> v2:172.21.2.125:3300/0 conn(0x7fef7c0e4da0 0x7fef7c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 --2- 172.21.2.117:0/850568894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c07da20 0x7fef7c0771b0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.604 7fef83ec2700 1 -- 172.21.2.117:0/850568894 >> 172.21.2.117:0/850568894 conn(0x7fef7c0ce450 msgr2=0x7fef7c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.605 7fef83ec2700 1 -- 172.21.2.117:0/850568894 shutdown_connections 2021-02-10T11:38:33.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.605 7fef83ec2700 1 -- 172.21.2.117:0/850568894 wait complete. 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.619 7fef83ec2700 1 Processor -- start 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.619 7fef83ec2700 1 -- start start 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 0x7fef7c106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef7c187e20 0x7fef7c188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1067f0 con 0x7fef7c090420 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c107350 con 0x7fef7c1a0250 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef83ec2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c1042d0 con 0x7fef7c187e20 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef824df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef824df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45914/0 (socket says 172.21.2.117:45914) 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef824df700 1 -- 172.21.2.117:0/3578375913 learned_addr learned my addr 172.21.2.117:0/3578375913 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 0x7fef7c106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef814dd700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef7c187e20 0x7fef7c188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 -- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef7c187e20 msgr2=0x7fef7c188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef7c187e20 0x7fef7c188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 -- 172.21.2.117:0/3578375913 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 msgr2=0x7fef7c090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.620 7fef81cde700 1 -- 172.21.2.117:0/3578375913 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c090e40 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef824df700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef81cde700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 0x7fef7c106cf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fef70006f10 tx=0x7fef7000b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef70006390 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef70002030 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef70002490 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c090e40 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.621 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c090e40 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.622 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef70000c90 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.622 7fef6e7fc700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 0x7fef6401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.622 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef700326f0 con 0x7fef7c1a0250 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.622 7fef814dd700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 0x7fef6401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.622 7fef814dd700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 0x7fef6401ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:33.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.652 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c103280 con 0x7fef7c1a0250 2021-02-10T11:38:33.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.656 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef70009020 con 0x7fef7c1a0250 2021-02-10T11:38:33.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.791 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802436", "target": ["mgr", ""]}) v1 -- 0x7fef7c106770 con 0x7fef64018730 2021-02-10T11:38:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.798 7fef6e7fc700 1 -- 172.21.2.117:0/3578375913 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fef78006a70 con 0x7fef64018730 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.807 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 msgr2=0x7fef6401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.807 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 0x7fef6401ac10 crc :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.807 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 msgr2=0x7fef7c106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.807 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 0x7fef7c106cf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fef70006f10 tx=0x7fef7000b8c0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 shutdown_connections 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef64018730 0x7fef6401ac10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef7c090420 0x7fef7c090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef7c187e20 0x7fef7c188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 --2- 172.21.2.117:0/3578375913 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef7c1a0250 0x7fef7c106cf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:33.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 >> 172.21.2.117:0/3578375913 conn(0x7fef7c0a1850 msgr2=0x7fef7c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:33.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 shutdown_connections 2021-02-10T11:38:33.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:33.808 7fef83ec2700 1 -- 172.21.2.117:0/3578375913 wait complete. 2021-02-10T11:38:33.799 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802437 2021-02-10T11:38:34.168 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:34.171 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:38:34.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.232 7f79efcf6700 1 Processor -- start 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.232 7f79efcf6700 1 -- start start 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79efcf6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79efcf6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 0x7f79e81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79efcf6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e8077a70 con 0x7f79e80e51a0 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79efcf6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80e5c20 con 0x7f79e80e4da0 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79efcf6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80777d0 con 0x7f79e807da20 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79edb12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79ed311700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 0x7f79e81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79edb12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43598/0 (socket says 172.21.2.117:43598) 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79ed311700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 0x7f79e81b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45918/0 (socket says 172.21.2.117:45918) 2021-02-10T11:38:34.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79edb12700 1 -- 172.21.2.117:0/2504006808 learned_addr learned my addr 172.21.2.117:0/2504006808 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79ecb10700 1 -- 172.21.2.117:0/2504006808 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876056466 0 0) 0x7f79e0000f90 con 0x7f79e80e51a0 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79edb12700 1 -- 172.21.2.117:0/2504006808 >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 msgr2=0x7f79e81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.233 7f79edb12700 1 --2- 172.21.2.117:0/2504006808 >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 0x7f79e81b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79edb12700 1 -- 172.21.2.117:0/2504006808 >> v1:172.21.2.125:6789/0 conn(0x7f79e80e51a0 legacy=0x7f79e8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79edb12700 1 -- 172.21.2.117:0/2504006808 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e807de20 con 0x7f79e807da20 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79edb12700 1 --2- 172.21.2.117:0/2504006808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f79e4006700 tx=0x7f79e4009b80).ready entity=mon.2 client_cookie=2f9c43c2b9db20c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79ecb10700 1 -- 172.21.2.117:0/2504006808 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e400c0c0 con 0x7f79e807da20 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79ecb10700 1 -- 172.21.2.117:0/2504006808 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e400c340 con 0x7f79e807da20 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79ecb10700 1 -- 172.21.2.117:0/2504006808 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e400c860 con 0x7f79e807da20 2021-02-10T11:38:34.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79efcf6700 1 -- 172.21.2.117:0/2504006808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 msgr2=0x7f79e80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.234 7f79efcf6700 1 --2- 172.21.2.117:0/2504006808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f79e4006700 tx=0x7f79e4009b80).stop 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 -- 172.21.2.117:0/2504006808 shutdown_connections 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 --2- 172.21.2.117:0/2504006808 >> v2:172.21.2.117:3300/0 conn(0x7f79e80e4da0 0x7f79e81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 --2- 172.21.2.117:0/2504006808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e807da20 0x7f79e80771b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 -- 172.21.2.117:0/2504006808 >> 172.21.2.117:0/2504006808 conn(0x7f79e80ce450 msgr2=0x7f79e80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 -- 172.21.2.117:0/2504006808 shutdown_connections 2021-02-10T11:38:34.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.235 7f79efcf6700 1 -- 172.21.2.117:0/2504006808 wait complete. 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 Processor -- start 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 -- start start 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e808dfe0 0x7f79e808e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e80afac0 0x7f79e80afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80af540 con 0x7f79e808eb10 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80a0c80 con 0x7f79e80afac0 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.250 7f79efcf6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80b04e0 con 0x7f79e808dfe0 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45922/0 (socket says 172.21.2.117:45922) 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 -- 172.21.2.117:0/2918263199 learned_addr learned my addr 172.21.2.117:0/2918263199 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79ed311700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e808dfe0 0x7f79e808e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79ee313700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e80afac0 0x7f79e80afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 -- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e808dfe0 msgr2=0x7f79e808e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e808dfe0 0x7f79e808e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 -- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e80afac0 msgr2=0x7f79e80afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e80afac0 0x7f79e80afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 -- 172.21.2.117:0/2918263199 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80b0700 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79edb12700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f79e400b5e0 tx=0x7f79e400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e400bd70 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e4002030 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79e40024d0 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.251 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e80b0700 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.252 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e80b0700 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.252 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79e4000c90 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.252 7f79da7fc700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 0x7f79d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.252 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f79e40325d0 con 0x7f79e808eb10 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.253 7f79ed311700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 0x7f79d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.253 7f79ed311700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 0x7f79d001abc0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.282 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8103280 con 0x7f79e808eb10 2021-02-10T11:38:34.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.284 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79e4030db0 con 0x7f79e808eb10 2021-02-10T11:38:34.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.413 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802437", "target": ["mgr", ""]}) v1 -- 0x7f79e8103280 con 0x7f79d00186c0 2021-02-10T11:38:34.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.424 7f79da7fc700 1 -- 172.21.2.117:0/2918263199 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f79dc006a70 con 0x7f79d00186c0 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 msgr2=0x7f79d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 0x7f79d001abc0 crc :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 msgr2=0x7f79e80c3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f79e400b5e0 tx=0x7f79e400b980).stop 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 shutdown_connections 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e80afac0 0x7f79e80afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e808dfe0 0x7f79e808e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79d00186c0 0x7f79d001abc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 --2- 172.21.2.117:0/2918263199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e808eb10 0x7f79e80c3b60 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 >> 172.21.2.117:0/2918263199 conn(0x7f79e80a1850 msgr2=0x7f79e80bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 shutdown_connections 2021-02-10T11:38:34.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.431 7f79efcf6700 1 -- 172.21.2.117:0/2918263199 wait complete. 2021-02-10T11:38:34.421 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802438 2021-02-10T11:38:34.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.854 7facbe1cc700 1 Processor -- start 2021-02-10T11:38:34.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.854 7facbe1cc700 1 -- start start 2021-02-10T11:38:34.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facbe1cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 0x7facb80e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facbe1cc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80d1d00 con 0x7facb807da20 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facbe1cc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb8077a70 con 0x7facb81b00d0 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facbe1cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80e5410 con 0x7facb8079000 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7facb807da20 0x7facb80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46180/0 (socket says 172.21.2.117:46180) 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb77fe700 1 -- 172.21.2.117:0/733759659 learned_addr learned my addr 172.21.2.117:0/733759659 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb6ffd700 1 --2- 172.21.2.117:0/733759659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 0x7facb80e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb6ffd700 1 -- 172.21.2.117:0/733759659 >> v1:172.21.2.125:6789/0 conn(0x7facb81b00d0 legacy=0x7facb80d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb6ffd700 1 -- 172.21.2.117:0/733759659 >> v1:172.21.2.117:6789/0 conn(0x7facb807da20 legacy=0x7facb80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb67fc700 1 -- 172.21.2.117:0/733759659 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929939234 0 0) 0x7faca80010b0 con 0x7facb807da20 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.855 7facb6ffd700 1 -- 172.21.2.117:0/733759659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb8079400 con 0x7facb8079000 2021-02-10T11:38:34.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facb6ffd700 1 --2- 172.21.2.117:0/733759659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 0x7facb80e4d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7facac006650 tx=0x7facac009a80).ready entity=mon.2 client_cookie=92ce5a8d07feb5e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facb67fc700 1 -- 172.21.2.117:0/733759659 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facac00c0c0 con 0x7facb8079000 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facb67fc700 1 -- 172.21.2.117:0/733759659 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac00c340 con 0x7facb8079000 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facb67fc700 1 -- 172.21.2.117:0/733759659 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facac00c860 con 0x7facb8079000 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facbe1cc700 1 -- 172.21.2.117:0/733759659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 msgr2=0x7facb80e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.856 7facbe1cc700 1 --2- 172.21.2.117:0/733759659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 0x7facb80e4d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7facac006650 tx=0x7facac009a80).stop 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.857 7facbe1cc700 1 -- 172.21.2.117:0/733759659 shutdown_connections 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.857 7facbe1cc700 1 --2- 172.21.2.117:0/733759659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb8079000 0x7facb80e4d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.857 7facbe1cc700 1 -- 172.21.2.117:0/733759659 >> 172.21.2.117:0/733759659 conn(0x7facb80ce430 msgr2=0x7facb80e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.857 7facbe1cc700 1 -- 172.21.2.117:0/733759659 shutdown_connections 2021-02-10T11:38:34.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.857 7facbe1cc700 1 -- 172.21.2.117:0/733759659 wait complete. 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 Processor -- start 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 -- start start 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 0x7facb8101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb80b30b0 0x7facb80bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80b3f60 con 0x7facb80b34b0 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb80a0dc0 con 0x7facb81a04a0 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facbe1cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80a0fe0 con 0x7facb80b30b0 2021-02-10T11:38:34.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facb7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facb7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45938/0 (socket says 172.21.2.117:45938) 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facb7fff700 1 -- 172.21.2.117:0/1917064207 learned_addr learned my addr 172.21.2.117:0/1917064207 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facb6ffd700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb80b30b0 0x7facb80bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.872 7facb77fe700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 0x7facb8101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 -- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb80b30b0 msgr2=0x7facb80bdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb80b30b0 0x7facb80bdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 -- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 msgr2=0x7facb8101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 0x7facb8101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 -- 172.21.2.117:0/1917064207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80a1200 con 0x7facb80b34b0 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb77fe700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 0x7facb8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facb7fff700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7facb000b8e0 tx=0x7facb000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb000c2f0 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb000c570 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb80a1200 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb80a1790 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.873 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7facb000ca50 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.874 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7facb000c750 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.874 7fac9ffff700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 0x7fac9801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.874 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7facb0032cb0 con 0x7facb80b34b0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.874 7facb77fe700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 0x7fac9801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:34.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.874 7facb77fe700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 0x7fac9801abd0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:34.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.904 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac90002fd0 con 0x7facb80b34b0 2021-02-10T11:38:34.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:34.907 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7facb000c780 con 0x7facb80b34b0 2021-02-10T11:38:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.036 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802438", "target": ["mgr", ""]}) v1 -- 0x7fac90002f80 con 0x7fac980186d0 2021-02-10T11:38:35.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.048 7fac9ffff700 1 -- 172.21.2.117:0/1917064207 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faca80009b0 con 0x7fac980186d0 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 msgr2=0x7fac9801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 0x7fac9801abd0 crc :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 msgr2=0x7facb80be110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7facb000b8e0 tx=0x7facb000bc80).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 shutdown_connections 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fac980186d0 0x7fac9801abd0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7facb80b34b0 0x7facb80be110 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7facb80b30b0 0x7facb80bdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 --2- 172.21.2.117:0/1917064207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7facb81a04a0 0x7facb8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.053 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 >> 172.21.2.117:0/1917064207 conn(0x7facb80b4e50 msgr2=0x7facb80b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.054 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 shutdown_connections 2021-02-10T11:38:35.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.054 7facbe1cc700 1 -- 172.21.2.117:0/1917064207 wait complete. 2021-02-10T11:38:35.049 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802439 2021-02-10T11:38:35.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.479 7fdd60227700 1 Processor -- start 2021-02-10T11:38:35.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.479 7fdd60227700 1 -- start start 2021-02-10T11:38:35.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd60227700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 0x7fdd580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd60227700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdd580e4da0 0x7fdd581b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd60227700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd58077a70 con 0x7fdd580e51a0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd60227700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd580e5c20 con 0x7fdd580e4da0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd60227700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580777d0 con 0x7fdd5807da20 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd5e844700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdd580e51a0 0x7fdd58077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46192/0 (socket says 172.21.2.117:46192) 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd5e844700 1 -- 172.21.2.117:0/2986656903 learned_addr learned my addr 172.21.2.117:0/2986656903 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd5e043700 1 --2- 172.21.2.117:0/2986656903 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 0x7fdd580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd5d842700 1 --2- 172.21.2.117:0/2986656903 >> v2:172.21.2.125:3300/0 conn(0x7fdd580e4da0 0x7fdd581b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.480 7fdd5d041700 1 -- 172.21.2.117:0/2986656903 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79984937 0 0) 0x7fdd50000f90 con 0x7fdd580e51a0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5d041700 1 -- 172.21.2.117:0/2986656903 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd44002fe0 con 0x7fdd580e51a0 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5e043700 1 -- 172.21.2.117:0/2986656903 >> v2:172.21.2.125:3300/0 conn(0x7fdd580e4da0 msgr2=0x7fdd581b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:35.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5e043700 1 --2- 172.21.2.117:0/2986656903 >> v2:172.21.2.125:3300/0 conn(0x7fdd580e4da0 0x7fdd581b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5e043700 1 -- 172.21.2.117:0/2986656903 >> v1:172.21.2.117:6789/0 conn(0x7fdd580e51a0 legacy=0x7fdd58077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5e043700 1 -- 172.21.2.117:0/2986656903 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5807de20 con 0x7fdd5807da20 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5e043700 1 --2- 172.21.2.117:0/2986656903 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 0x7fdd580771b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd4c006650 tx=0x7fdd4c009a80).ready entity=mon.2 client_cookie=8538cec4143c64da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5d041700 1 -- 172.21.2.117:0/2986656903 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c00c0c0 con 0x7fdd5807da20 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5d041700 1 -- 172.21.2.117:0/2986656903 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c00c3c0 con 0x7fdd5807da20 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd60227700 1 -- 172.21.2.117:0/2986656903 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 msgr2=0x7fdd580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd5d041700 1 -- 172.21.2.117:0/2986656903 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c0009b0 con 0x7fdd5807da20 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.481 7fdd60227700 1 --2- 172.21.2.117:0/2986656903 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 0x7fdd580771b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd4c006650 tx=0x7fdd4c009a80).stop 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 -- 172.21.2.117:0/2986656903 shutdown_connections 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 --2- 172.21.2.117:0/2986656903 >> v2:172.21.2.125:3300/0 conn(0x7fdd580e4da0 0x7fdd581b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 --2- 172.21.2.117:0/2986656903 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5807da20 0x7fdd580771b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 -- 172.21.2.117:0/2986656903 >> 172.21.2.117:0/2986656903 conn(0x7fdd580ce450 msgr2=0x7fdd580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 -- 172.21.2.117:0/2986656903 shutdown_connections 2021-02-10T11:38:35.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.482 7fdd60227700 1 -- 172.21.2.117:0/2986656903 wait complete. 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 Processor -- start 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 -- start start 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 0x7fdd581b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd581b5b50 0x7fdd580b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580b8da0 con 0x7fdd581b5f50 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580a4140 con 0x7fdd581b5130 2021-02-10T11:38:35.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd60227700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd580a4320 con 0x7fdd581b5b50 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd5e844700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd5e844700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45950/0 (socket says 172.21.2.117:45950) 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd5e844700 1 -- 172.21.2.117:0/2026587129 learned_addr learned my addr 172.21.2.117:0/2026587129 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd5d842700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd581b5b50 0x7fdd580b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.497 7fdd5e043700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 0x7fdd581b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 -- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd581b5b50 msgr2=0x7fdd580b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd581b5b50 0x7fdd580b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 -- 172.21.2.117:0/2026587129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 msgr2=0x7fdd580b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 -- 172.21.2.117:0/2026587129 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580b62b0 con 0x7fdd581b5130 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e844700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd5e043700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 0x7fdd581b5530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fdd4c009440 tx=0x7fdd4c00b840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c00bc30 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c002080 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd4c0009b0 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd60227700 1 -- 172.21.2.117:0/2026587129 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd580b62b0 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd60227700 1 -- 172.21.2.117:0/2026587129 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd580b62b0 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd4c002260 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.498 7fdd4affd700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.499 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdd4c032630 con 0x7fdd581b5130 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.499 7fdd5d842700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:35.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.499 7fdd5d842700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 0x7fdd4001abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:35.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.530 7fdd60227700 1 -- 172.21.2.117:0/2026587129 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd581b4f00 con 0x7fdd581b5130 2021-02-10T11:38:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.533 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd4c000c90 con 0x7fdd581b5130 2021-02-10T11:38:35.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.661 7fdd60227700 1 -- 172.21.2.117:0/2026587129 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802439", "target": ["mgr", ""]}) v1 -- 0x7fdd58077a70 con 0x7fdd400186c0 2021-02-10T11:38:35.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.675 7fdd4affd700 1 -- 172.21.2.117:0/2026587129 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdd54006a70 con 0x7fdd400186c0 2021-02-10T11:38:35.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 msgr2=0x7fdd4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 0x7fdd4001abc0 crc :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 msgr2=0x7fdd581b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 0x7fdd581b5530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fdd4c009440 tx=0x7fdd4c00b840).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 shutdown_connections 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd400186c0 0x7fdd4001abc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd581b5f50 0x7fdd580b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd581b5b50 0x7fdd580b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 --2- 172.21.2.117:0/2026587129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd581b5130 0x7fdd581b5530 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 >> 172.21.2.117:0/2026587129 conn(0x7fdd5808bbc0 msgr2=0x7fdd580c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 shutdown_connections 2021-02-10T11:38:35.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:35.679 7fdd60227700 1 -- 172.21.2.117:0/2026587129 wait complete. 2021-02-10T11:38:35.670 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802440 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.099 7ff58cd77700 1 Processor -- start 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.099 7ff58cd77700 1 -- start start 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff58cd77700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff588079000 0x7ff5880e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff58cd77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 0x7ff5881b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff58cd77700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff588077a70 con 0x7ff58807da20 2021-02-10T11:38:36.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff58cd77700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5880e53c0 con 0x7ff588079000 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff58cd77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff588079400 con 0x7ff5881b00f0 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff588079000 0x7ff5880e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff58807da20 0x7ff5880771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46204/0 (socket says 172.21.2.117:46204) 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586b13700 1 -- 172.21.2.117:0/500072323 learned_addr learned my addr 172.21.2.117:0/500072323 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff587314700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 0x7ff5881b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586312700 1 -- 172.21.2.117:0/500072323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 msgr2=0x7ff5881b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586312700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 0x7ff5881b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586312700 1 -- 172.21.2.117:0/500072323 >> v1:172.21.2.117:6789/0 conn(0x7ff58807da20 legacy=0x7ff5880771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.100 7ff586312700 1 -- 172.21.2.117:0/500072323 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58807de20 con 0x7ff588079000 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff587314700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 0x7ff5881b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:36.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff586312700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff588079000 0x7ff5880e4da0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7ff570006650 tx=0x7ff570009a80).ready entity=mon.0 client_cookie=72aa8ab56a48b6d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff585b11700 1 -- 172.21.2.117:0/500072323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff57000c0c0 con 0x7ff588079000 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff585b11700 1 -- 172.21.2.117:0/500072323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57000c340 con 0x7ff588079000 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff585b11700 1 -- 172.21.2.117:0/500072323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff57000c860 con 0x7ff588079000 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff58cd77700 1 -- 172.21.2.117:0/500072323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff588079000 msgr2=0x7ff5880e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff58cd77700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff588079000 0x7ff5880e4da0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7ff570006650 tx=0x7ff570009a80).stop 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.101 7ff58cd77700 1 -- 172.21.2.117:0/500072323 shutdown_connections 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.102 7ff58cd77700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5881b00f0 0x7ff5881b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.102 7ff58cd77700 1 --2- 172.21.2.117:0/500072323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff588079000 0x7ff5880e4da0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.102 7ff58cd77700 1 -- 172.21.2.117:0/500072323 >> 172.21.2.117:0/500072323 conn(0x7ff5880ce450 msgr2=0x7ff5880e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.102 7ff58cd77700 1 -- 172.21.2.117:0/500072323 shutdown_connections 2021-02-10T11:38:36.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.102 7ff58cd77700 1 -- 172.21.2.117:0/500072323 wait complete. 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.116 7ff58cd77700 1 Processor -- start 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.116 7ff58cd77700 1 -- start start 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5880b3ff0 0x7ff5880b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff588090de0 0x7ff5880911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588090600 con 0x7ff5880c33d0 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588090860 con 0x7ff588090de0 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff58cd77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880b3c70 con 0x7ff5880b3ff0 2021-02-10T11:38:36.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45960/0 (socket says 172.21.2.117:45960) 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 -- 172.21.2.117:0/2641454466 learned_addr learned my addr 172.21.2.117:0/2641454466 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff587314700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff588090de0 0x7ff5880911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586b13700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5880b3ff0 0x7ff5880b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 -- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5880b3ff0 msgr2=0x7ff5880b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5880b3ff0 0x7ff5880b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 -- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff588090de0 msgr2=0x7ff5880911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff588090de0 0x7ff5880911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 -- 172.21.2.117:0/2641454466 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588091800 con 0x7ff5880c33d0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.117 7ff586312700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7ff570009630 tx=0x7ff57000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.118 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5700093a0 con 0x7ff5880c33d0 2021-02-10T11:38:36.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.118 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57000bd70 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.118 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff570005560 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.118 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff588091800 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.118 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff588091d90 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.119 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff570000c90 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.119 7ff57f7fe700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 0x7ff56c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.119 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff570032700 con 0x7ff5880c33d0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.119 7ff586b13700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 0x7ff56c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.119 7ff586b13700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 0x7ff56c01abc0 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.150 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff588103370 con 0x7ff5880c33d0 2021-02-10T11:38:36.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.153 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff570030d60 con 0x7ff5880c33d0 2021-02-10T11:38:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.278 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802440", "target": ["mgr", ""]}) v1 -- 0x7ff588091d90 con 0x7ff56c0186c0 2021-02-10T11:38:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.291 7ff57f7fe700 1 -- 172.21.2.117:0/2641454466 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff578001050 con 0x7ff56c0186c0 2021-02-10T11:38:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 msgr2=0x7ff56c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 0x7ff56c01abc0 crc :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 msgr2=0x7ff5880b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7ff570009630 tx=0x7ff57000b980).stop 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 shutdown_connections 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff56c0186c0 0x7ff56c01abc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff5880c33d0 0x7ff5880b3650 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff588090de0 0x7ff5880911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 --2- 172.21.2.117:0/2641454466 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff5880b3ff0 0x7ff5880b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 >> 172.21.2.117:0/2641454466 conn(0x7ff5880a1420 msgr2=0x7ff58809efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 shutdown_connections 2021-02-10T11:38:36.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.296 7ff58cd77700 1 -- 172.21.2.117:0/2641454466 wait complete. 2021-02-10T11:38:36.286 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802441 2021-02-10T11:38:36.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.717 7f86574a0700 1 Processor -- start 2021-02-10T11:38:36.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.717 7f86574a0700 1 -- start start 2021-02-10T11:38:36.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86574a0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f865007da20 0x7f86500771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86574a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500e4da0 0x7f86501b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86574a0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8650077a70 con 0x7f86500e51a0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86574a0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86500777d0 con 0x7f865007da20 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86574a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865007de20 con 0x7f86500e4da0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f8655abd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f86500e51a0 0x7f8650077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46218/0 (socket says 172.21.2.117:46218) 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f8655abd700 1 -- 172.21.2.117:0/152365614 learned_addr learned my addr 172.21.2.117:0/152365614 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014956527 0 0) 0x7f8648001500 con 0x7f86500e51a0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f8647fff700 1 -- 172.21.2.117:0/152365614 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f863c002fe0 con 0x7f86500e51a0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f8654abb700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500e4da0 0x7f86501b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.718 7f86552bc700 1 --2- 172.21.2.117:0/152365614 >> v2:172.21.2.125:3300/0 conn(0x7f865007da20 0x7f86500771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2402867522 0 0) 0x7f8648001500 con 0x7f86500e51a0 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500e4da0 msgr2=0x7f86501b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500e4da0 0x7f86501b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 >> v2:172.21.2.125:3300/0 conn(0x7f865007da20 msgr2=0x7f86500771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 --2- 172.21.2.117:0/152365614 >> v2:172.21.2.125:3300/0 conn(0x7f865007da20 0x7f86500771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86501b0710 con 0x7f86500e51a0 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8648001a00 con 0x7f86500e51a0 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 >> v1:172.21.2.117:6789/0 conn(0x7f86500e51a0 legacy=0x7f8650077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f863c003b10 0x7f863c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f863c006830 0x7f863c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f863c0095c0 0x7f863c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c050380 con 0x7f863c0095c0 2021-02-10T11:38:36.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c0505a0 con 0x7f863c003b10 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863c0507c0 con 0x7f863c006830 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f86552bc700 1 --2- 172.21.2.117:0/152365614 >> v2:172.21.2.125:3300/0 conn(0x7f865007da20 0x7f86500771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f863c0095c0 0x7f863c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8654abb700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f863c006830 0x7f863c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f86552bc700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f863c003b10 0x7f863c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 -- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f863c006830 msgr2=0x7f863c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f863c006830 0x7f863c008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 -- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f863c003b10 msgr2=0x7f863c006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f863c003b10 0x7f863c006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 -- 172.21.2.117:0/152365614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c050a60 con 0x7f863c0095c0 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f86552bc700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f863c003b10 0x7f863c006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8655abd700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f863c0095c0 0x7f863c00bac0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f8648007a70 tx=0x7f864800a310).ready entity=mon.0 client_cookie=469eda7333b54d47 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86480070c0 con 0x7f863c0095c0 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.719 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8648007440 con 0x7f863c0095c0 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f8647fff700 1 -- 172.21.2.117:0/152365614 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8648004720 con 0x7f863c0095c0 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 -- 172.21.2.117:0/152365614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f863c0095c0 msgr2=0x7f863c00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 --2- 172.21.2.117:0/152365614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f863c0095c0 0x7f863c00bac0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7f8648007a70 tx=0x7f864800a310).stop 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f8655abd700 1 -- 172.21.2.117:0/152365614 reap_dead start 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 -- 172.21.2.117:0/152365614 shutdown_connections 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 -- 172.21.2.117:0/152365614 >> 172.21.2.117:0/152365614 conn(0x7f86500ce450 msgr2=0x7f86500e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 -- 172.21.2.117:0/152365614 shutdown_connections 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.720 7f86574a0700 1 -- 172.21.2.117:0/152365614 wait complete. 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 Processor -- start 2021-02-10T11:38:36.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 -- start start 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500a47d0 0x7f86500a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 0x7f86500a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650189970 con 0x7f86500b77d0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8650190aa0 con 0x7f86500a51f0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.735 7f86574a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86500b81f0 con 0x7f86500a47d0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8655abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8655abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45978/0 (socket says 172.21.2.117:45978) 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8655abd700 1 -- 172.21.2.117:0/2843874233 learned_addr learned my addr 172.21.2.117:0/2843874233 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f86552bc700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500a47d0 0x7f86500a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 0x7f86500a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 -- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500a47d0 msgr2=0x7f86500a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500a47d0 0x7f86500a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 -- 172.21.2.117:0/2843874233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 msgr2=0x7f86500b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 -- 172.21.2.117:0/2843874233 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86500b8410 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8655abd700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8654abb700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 0x7f86500a55f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f864000b560 tx=0x7f864000b900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.736 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f864000bcf0 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640006ae0 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f86400022b0 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f86574a0700 1 -- 172.21.2.117:0/2843874233 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86500b8410 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f86574a0700 1 -- 172.21.2.117:0/2843874233 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86500b8410 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8640000c90 con 0x7f86500a51f0 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f8645ffb700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 0x7f863001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:36.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8640032830 con 0x7f86500a51f0 2021-02-10T11:38:36.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.737 7f86552bc700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 0x7f863001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:36.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.738 7f86552bc700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 0x7f863001abc0 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.767 7f86574a0700 1 -- 172.21.2.117:0/2843874233 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8650077a70 con 0x7f86500a51f0 2021-02-10T11:38:36.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.771 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f864001f420 con 0x7f86500a51f0 2021-02-10T11:38:36.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.900 7f86574a0700 1 -- 172.21.2.117:0/2843874233 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802441", "target": ["mgr", ""]}) v1 -- 0x7f8650190aa0 con 0x7f86300186c0 2021-02-10T11:38:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.913 7f8645ffb700 1 -- 172.21.2.117:0/2843874233 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f864c006dd0 con 0x7f86300186c0 2021-02-10T11:38:36.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.916 7f86574a0700 1 -- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 msgr2=0x7f863001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.916 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 0x7f863001abc0 crc :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.916 7f86574a0700 1 -- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 msgr2=0x7f86500a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:36.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.916 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 0x7f86500a55f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f864000b560 tx=0x7f864000b900).stop 2021-02-10T11:38:36.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 -- 172.21.2.117:0/2843874233 shutdown_connections 2021-02-10T11:38:36.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f86300186c0 0x7f863001abc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f86500b77d0 0x7f86500b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f86500a51f0 0x7f86500a55f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 --2- 172.21.2.117:0/2843874233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f86500a47d0 0x7f86500a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 -- 172.21.2.117:0/2843874233 >> 172.21.2.117:0/2843874233 conn(0x7f86500b5140 msgr2=0x7f86500a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 -- 172.21.2.117:0/2843874233 shutdown_connections 2021-02-10T11:38:36.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:36.917 7f86574a0700 1 -- 172.21.2.117:0/2843874233 wait complete. 2021-02-10T11:38:36.907 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802442 2021-02-10T11:38:37.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.336 7f1f5ce79700 1 Processor -- start 2021-02-10T11:38:37.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.336 7f1f5ce79700 1 -- start start 2021-02-10T11:38:37.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f5ce79700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1f58079000 0x7f1f580e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f5ce79700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580d1d20 con 0x7f1f5807da20 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f5ce79700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f58077a70 con 0x7f1f581b00f0 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f5ce79700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f580e53c0 con 0x7f1f58079000 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f56b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1f5807da20 0x7f1f580771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46236/0 (socket says 172.21.2.117:46236) 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f56b13700 1 -- 172.21.2.117:0/105636071 learned_addr learned my addr 172.21.2.117:0/105636071 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431815096 0 0) 0x7f1f48001180 con 0x7f1f5807da20 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.337 7f1f55b11700 1 -- 172.21.2.117:0/105636071 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40002fe0 con 0x7f1f5807da20 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f56312700 1 --2- 172.21.2.117:0/105636071 >> v2:172.21.2.125:3300/0 conn(0x7f1f58079000 0x7f1f580e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3383933587 0 0) 0x7f1f48001180 con 0x7f1f5807da20 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 >> v2:172.21.2.125:3300/0 conn(0x7f1f58079000 msgr2=0x7f1f580e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 --2- 172.21.2.117:0/105636071 >> v2:172.21.2.125:3300/0 conn(0x7f1f58079000 0x7f1f580e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 >> v1:172.21.2.125:6789/0 conn(0x7f1f581b00f0 legacy=0x7f1f580d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58079400 con 0x7f1f5807da20 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1f48001680 con 0x7f1f5807da20 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 >> v1:172.21.2.117:6789/0 conn(0x7f1f5807da20 legacy=0x7f1f580771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 0x7f1f40005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f40006790 0x7f1f40008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f40009590 0x7f1f4000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000c300 con 0x7f1f40006790 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000c520 con 0x7f1f40003a60 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f55b11700 1 -- 172.21.2.117:0/105636071 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4000c740 con 0x7f1f40009590 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f56312700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f40006790 0x7f1f40008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f56b13700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 0x7f1f40005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.338 7f1f57314700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f40009590 0x7f1f4000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 -- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f40009590 msgr2=0x7f1f4000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f40009590 0x7f1f4000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 -- 172.21.2.117:0/105636071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f40006790 msgr2=0x7f1f40008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f40006790 0x7f1f40008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 -- 172.21.2.117:0/105636071 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400509b0 con 0x7f1f40003a60 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f57314700 1 -- 172.21.2.117:0/105636071 reap_dead start 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56312700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f40006790 0x7f1f40008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f56b13700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 0x7f1f40005ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1f48007970 tx=0x7f1f48007d10).ready entity=mon.1 client_cookie=8022d5f65d910a16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f480070c0 con 0x7f1f40003a60 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f480043a0 con 0x7f1f40003a60 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f55b11700 1 -- 172.21.2.117:0/105636071 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f48004770 con 0x7f1f40003a60 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f5ce79700 1 -- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 msgr2=0x7f1f40005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.339 7f1f5ce79700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 0x7f1f40005ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1f48007970 tx=0x7f1f48007d10).stop 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.340 7f1f5ce79700 1 -- 172.21.2.117:0/105636071 shutdown_connections 2021-02-10T11:38:37.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.340 7f1f5ce79700 1 --2- 172.21.2.117:0/105636071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f40003a60 0x7f1f40005ef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.340 7f1f5ce79700 1 -- 172.21.2.117:0/105636071 >> 172.21.2.117:0/105636071 conn(0x7f1f580ce450 msgr2=0x7f1f580e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.340 7f1f5ce79700 1 -- 172.21.2.117:0/105636071 shutdown_connections 2021-02-10T11:38:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.340 7f1f5ce79700 1 -- 172.21.2.117:0/105636071 wait complete. 2021-02-10T11:38:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.355 7f1f5ce79700 1 Processor -- start 2021-02-10T11:38:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.355 7f1f5ce79700 1 -- start start 2021-02-10T11:38:37.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f580a51f0 0x7f1f580a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f580b77d0 0x7f1f580b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58189970 con 0x7f1f580a47d0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58190aa0 con 0x7f1f580a51f0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f5ce79700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f580b81f0 con 0x7f1f580b77d0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:45998/0 (socket says 172.21.2.117:45998) 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 -- 172.21.2.117:0/1900303774 learned_addr learned my addr 172.21.2.117:0/1900303774 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f57314700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f580b77d0 0x7f1f580b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56312700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f580a51f0 0x7f1f580a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 -- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f580b77d0 msgr2=0x7f1f580b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f580b77d0 0x7f1f580b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 -- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f580a51f0 msgr2=0x7f1f580a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f580a51f0 0x7f1f580a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 -- 172.21.2.117:0/1900303774 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580b8410 con 0x7f1f580a47d0 2021-02-10T11:38:37.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.356 7f1f56b13700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f1f4800a550 tx=0x7f1f4800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f4800ace0 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f48001390 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f580b8410 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f48004590 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f580b8410 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f480043a0 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 0x7f1f3c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1f48032750 con 0x7f1f580a47d0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.357 7f1f56312700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 0x7f1f3c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.358 7f1f56312700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 0x7f1f3c01ac10 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.387 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f58077a70 con 0x7f1f580a47d0 2021-02-10T11:38:37.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.389 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f48000ec0 con 0x7f1f580a47d0 2021-02-10T11:38:37.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.515 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802442", "target": ["mgr", ""]}) v1 -- 0x7f1f58077a70 con 0x7f1f3c018730 2021-02-10T11:38:37.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.526 7f1f477fe700 1 -- 172.21.2.117:0/1900303774 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f4c0009b0 con 0x7f1f3c018730 2021-02-10T11:38:37.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 msgr2=0x7f1f3c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 0x7f1f3c01ac10 crc :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 msgr2=0x7f1f580a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f1f4800a550 tx=0x7f1f4800a8f0).stop 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 shutdown_connections 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f580b77d0 0x7f1f580b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f580a51f0 0x7f1f580a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f3c018730 0x7f1f3c01ac10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 --2- 172.21.2.117:0/1900303774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f580a47d0 0x7f1f580a4bd0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 >> 172.21.2.117:0/1900303774 conn(0x7f1f580b5140 msgr2=0x7f1f580a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 shutdown_connections 2021-02-10T11:38:37.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.532 7f1f5ce79700 1 -- 172.21.2.117:0/1900303774 wait complete. 2021-02-10T11:38:37.523 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802443 2021-02-10T11:38:37.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.953 7fd643a6c700 1 Processor -- start 2021-02-10T11:38:37.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 -- start start 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c07da20 0x7fd63c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd63c0e4d80 0x7fd63c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd63c0e5180 0x7fd63c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c0e5c00 con 0x7fd63c0e4d80 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c077a70 con 0x7fd63c0e5180 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd643a6c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c0777d0 con 0x7fd63c07da20 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd641087700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd63c0e4d80 0x7fd63c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd641087700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd63c0e4d80 0x7fd63c1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46006/0 (socket says 172.21.2.117:46006) 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.954 7fd641087700 1 -- 172.21.2.117:0/2054523628 learned_addr learned my addr 172.21.2.117:0/2054523628 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 -- 172.21.2.117:0/2054523628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c07da20 msgr2=0x7fd63c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 --2- 172.21.2.117:0/2054523628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c07da20 0x7fd63c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 -- 172.21.2.117:0/2054523628 >> v2:172.21.2.125:3300/0 conn(0x7fd63c0e5180 msgr2=0x7fd63c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 --2- 172.21.2.117:0/2054523628 >> v2:172.21.2.125:3300/0 conn(0x7fd63c0e5180 0x7fd63c077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 -- 172.21.2.117:0/2054523628 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c07de20 con 0x7fd63c0e4d80 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd641087700 1 --2- 172.21.2.117:0/2054523628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c0e4d80 0x7fd63c1b00d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fd62c006700 tx=0x7fd62c009b00).ready entity=mon.0 client_cookie=3b2c0698a21af339 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd640886700 1 -- 172.21.2.117:0/2054523628 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd62c00c0c0 con 0x7fd63c0e4d80 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd640886700 1 -- 172.21.2.117:0/2054523628 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c00c340 con 0x7fd63c0e4d80 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd640886700 1 -- 172.21.2.117:0/2054523628 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd62c00c820 con 0x7fd63c0e4d80 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd643a6c700 1 -- 172.21.2.117:0/2054523628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c0e4d80 msgr2=0x7fd63c1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.955 7fd643a6c700 1 --2- 172.21.2.117:0/2054523628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c0e4d80 0x7fd63c1b00d0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7fd62c006700 tx=0x7fd62c009b00).stop 2021-02-10T11:38:37.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 -- 172.21.2.117:0/2054523628 shutdown_connections 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 --2- 172.21.2.117:0/2054523628 >> v2:172.21.2.125:3300/0 conn(0x7fd63c0e5180 0x7fd63c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 --2- 172.21.2.117:0/2054523628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c0e4d80 0x7fd63c1b00d0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 --2- 172.21.2.117:0/2054523628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c07da20 0x7fd63c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 -- 172.21.2.117:0/2054523628 >> 172.21.2.117:0/2054523628 conn(0x7fd63c0ce430 msgr2=0x7fd63c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 -- 172.21.2.117:0/2054523628 shutdown_connections 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.956 7fd643a6c700 1 -- 172.21.2.117:0/2054523628 wait complete. 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.970 7fd643a6c700 1 Processor -- start 2021-02-10T11:38:37.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.970 7fd643a6c700 1 -- start start 2021-02-10T11:38:37.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd63c09f170 0x7fd63c09f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c107070 0x7fd63c107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c105e30 con 0x7fd63c106650 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c09f570 con 0x7fd63c09f170 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd643a6c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c0a0710 con 0x7fd63c107070 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641888700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd63c09f170 0x7fd63c09f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46010/0 (socket says 172.21.2.117:46010) 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 -- 172.21.2.117:0/3394291291 learned_addr learned my addr 172.21.2.117:0/3394291291 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd642089700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c107070 0x7fd63c107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 -- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c107070 msgr2=0x7fd63c107470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c107070 0x7fd63c107470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 -- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd63c09f170 msgr2=0x7fd63c09f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd63c09f170 0x7fd63c09f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.971 7fd641087700 1 -- 172.21.2.117:0/3394291291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c09fe40 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd641087700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fd62c006f20 tx=0x7fd62c00b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd62c00bd00 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c000c90 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c09fe40 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd62c0022b0 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.972 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c09fe40 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.973 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd62c00a000 con 0x7fd63c106650 2021-02-10T11:38:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.973 7fd6327fc700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 0x7fd62401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.973 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd62c032980 con 0x7fd63c106650 2021-02-10T11:38:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.973 7fd641888700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 0x7fd62401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:37.973 7fd641888700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 0x7fd62401ac10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:37.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.003 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c103060 con 0x7fd63c106650 2021-02-10T11:38:37.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.006 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd62c006390 con 0x7fd63c106650 2021-02-10T11:38:38.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.132 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802443", "target": ["mgr", ""]}) v1 -- 0x7fd63c103060 con 0x7fd624018730 2021-02-10T11:38:38.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.143 7fd6327fc700 1 -- 172.21.2.117:0/3394291291 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd638000ce0 con 0x7fd624018730 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 msgr2=0x7fd62401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 0x7fd62401ac10 crc :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 msgr2=0x7fd63c106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fd62c006f20 tx=0x7fd62c00b910).stop 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 shutdown_connections 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd624018730 0x7fd62401ac10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd63c106650 0x7fd63c106a50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd63c107070 0x7fd63c107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 --2- 172.21.2.117:0/3394291291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd63c09f170 0x7fd63c09f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 >> 172.21.2.117:0/3394291291 conn(0x7fd63c103940 msgr2=0x7fd63c103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 shutdown_connections 2021-02-10T11:38:38.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.150 7fd643a6c700 1 -- 172.21.2.117:0/3394291291 wait complete. 2021-02-10T11:38:38.141 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802444 2021-02-10T11:38:38.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 Processor -- start 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 -- start start 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f8407da20 0x7f1f840771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1f840e4da0 0x7f1f841b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f84077a70 con 0x7f1f840e51a0 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840777d0 con 0x7f1f8407da20 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.572 7f1f890d8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8407de20 con 0x7f1f840e4da0 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f8407da20 0x7f1f840771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f8359e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f1f840e51a0 0x7f1f84077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39192/0 (socket says 172.21.2.117:39192) 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f8359e700 1 -- 172.21.2.117:0/4057799787 learned_addr learned my addr 172.21.2.117:0/4057799787 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f8259c700 1 --2- 172.21.2.117:0/4057799787 >> v2:172.21.2.125:3300/0 conn(0x7f1f840e4da0 0x7f1f841b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 -- 172.21.2.117:0/4057799787 >> v2:172.21.2.125:3300/0 conn(0x7f1f840e4da0 msgr2=0x7f1f841b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 --2- 172.21.2.117:0/4057799787 >> v2:172.21.2.125:3300/0 conn(0x7f1f840e4da0 0x7f1f841b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 -- 172.21.2.117:0/4057799787 >> v1:172.21.2.125:6789/0 conn(0x7f1f840e51a0 legacy=0x7f1f84077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 -- 172.21.2.117:0/4057799787 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f841b0710 con 0x7f1f8407da20 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f82d9d700 1 --2- 172.21.2.117:0/4057799787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f8407da20 0x7f1f840771b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f1f7c0069d0 tx=0x7f1f7c009e40).ready entity=mon.0 client_cookie=140748bd045413d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f81d9b700 1 -- 172.21.2.117:0/4057799787 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c00b510 con 0x7f1f8407da20 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f81d9b700 1 -- 172.21.2.117:0/4057799787 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c00b890 con 0x7f1f8407da20 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.573 7f1f81d9b700 1 -- 172.21.2.117:0/4057799787 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f7c00bcf0 con 0x7f1f8407da20 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 -- 172.21.2.117:0/4057799787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f8407da20 msgr2=0x7f1f840771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 --2- 172.21.2.117:0/4057799787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f8407da20 0x7f1f840771b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f1f7c0069d0 tx=0x7f1f7c009e40).stop 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 -- 172.21.2.117:0/4057799787 shutdown_connections 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 --2- 172.21.2.117:0/4057799787 >> v2:172.21.2.125:3300/0 conn(0x7f1f840e4da0 0x7f1f841b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 --2- 172.21.2.117:0/4057799787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f8407da20 0x7f1f840771b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 -- 172.21.2.117:0/4057799787 >> 172.21.2.117:0/4057799787 conn(0x7f1f840ce450 msgr2=0x7f1f840e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 -- 172.21.2.117:0/4057799787 shutdown_connections 2021-02-10T11:38:38.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.574 7f1f890d8700 1 -- 172.21.2.117:0/4057799787 wait complete. 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 Processor -- start 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 -- start start 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f840a41c0 0x7f1f840fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f840b7390 0x7f1f840b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840b8010 con 0x7f1f840b7390 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840b8550 con 0x7f1f840c2340 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.589 7f1f890d8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840b8770 con 0x7f1f840a41c0 2021-02-10T11:38:38.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f840b7390 0x7f1f840b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50406/0 (socket says 172.21.2.117:50406) 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 -- 172.21.2.117:0/3085184345 learned_addr learned my addr 172.21.2.117:0/3085184345 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f82d9d700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f840a41c0 0x7f1f840fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 -- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f840a41c0 msgr2=0x7f1f840fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f840a41c0 0x7f1f840fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 -- 172.21.2.117:0/3085184345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f840b7390 msgr2=0x7f1f840b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f840b7390 0x7f1f840b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 -- 172.21.2.117:0/3085184345 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840b8990 con 0x7f1f840c2340 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f8259c700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1f74006c00 tx=0x7f1f7400b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.590 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f74006d00 con 0x7f1f840c2340 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f74000c90 con 0x7f1f840c2340 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f840b8990 con 0x7f1f840c2340 2021-02-10T11:38:38.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f74002300 con 0x7f1f840c2340 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f840b8f20 con 0x7f1f840c2340 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f74002020 con 0x7f1f840c2340 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f7b7fe700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 0x7f1f6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.591 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1f74032540 con 0x7f1f840c2340 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.592 7f1f82d9d700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 0x7f1f6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:38.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.592 7f1f82d9d700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 0x7f1f6801abc0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:38.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.622 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f841b4f00 con 0x7f1f840c2340 2021-02-10T11:38:38.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.628 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f74030d20 con 0x7f1f840c2340 2021-02-10T11:38:38.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.757 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802444", "target": ["mgr", ""]}) v1 -- 0x7f1f840b8f20 con 0x7f1f680186c0 2021-02-10T11:38:38.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.768 7f1f7b7fe700 1 -- 172.21.2.117:0/3085184345 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1f7c000d30 con 0x7f1f680186c0 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 msgr2=0x7f1f6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 0x7f1f6801abc0 crc :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 msgr2=0x7f1f840c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1f74006c00 tx=0x7f1f7400b960).stop 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 shutdown_connections 2021-02-10T11:38:38.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f680186c0 0x7f1f6801abc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f840b7390 0x7f1f840b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f840c2340 0x7f1f840c2740 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 --2- 172.21.2.117:0/3085184345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f840a41c0 0x7f1f840fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 >> 172.21.2.117:0/3085184345 conn(0x7f1f8408bbc0 msgr2=0x7f1f840c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 shutdown_connections 2021-02-10T11:38:38.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:38.775 7f1f890d8700 1 -- 172.21.2.117:0/3085184345 wait complete. 2021-02-10T11:38:38.765 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802445 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.196 7f7ac7ce3700 1 Processor -- start 2021-02-10T11:38:39.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.196 7f7ac7ce3700 1 -- start start 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac7ce3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7ac007da20 0x7f7ac00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac7ce3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00d1d00 con 0x7f7ac00e5180 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac7ce3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac0077a70 con 0x7f7ac00e4d80 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac7ce3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00777d0 con 0x7f7ac007da20 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7ac007da20 0x7f7ac00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7ac007da20 0x7f7ac00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50414/0 (socket says 172.21.2.117:50414) 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 -- 172.21.2.117:0/1078776366 learned_addr learned my addr 172.21.2.117:0/1078776366 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 -- 172.21.2.117:0/1078776366 >> v1:172.21.2.125:6789/0 conn(0x7f7ac00e4d80 legacy=0x7f7ac01b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 -- 172.21.2.117:0/1078776366 >> v1:172.21.2.117:6789/0 conn(0x7f7ac00e5180 legacy=0x7f7ac00d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.197 7f7ac5aff700 1 -- 172.21.2.117:0/1078776366 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac01b08c0 con 0x7f7ac007da20 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac5aff700 1 --2- 172.21.2.117:0/1078776366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac007da20 0x7f7ac00771b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7ab0006700 tx=0x7f7ab0009b80).ready entity=mon.1 client_cookie=91657ef0774e23b4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac4afd700 1 -- 172.21.2.117:0/1078776366 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab000c0c0 con 0x7f7ac007da20 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac4afd700 1 -- 172.21.2.117:0/1078776366 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab000c440 con 0x7f7ac007da20 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac4afd700 1 -- 172.21.2.117:0/1078776366 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab000c8a0 con 0x7f7ac007da20 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac7ce3700 1 -- 172.21.2.117:0/1078776366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac007da20 msgr2=0x7f7ac00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac7ce3700 1 --2- 172.21.2.117:0/1078776366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac007da20 0x7f7ac00771b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7ab0006700 tx=0x7f7ab0009b80).stop 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac7ce3700 1 -- 172.21.2.117:0/1078776366 shutdown_connections 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac7ce3700 1 --2- 172.21.2.117:0/1078776366 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac007da20 0x7f7ac00771b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.198 7f7ac7ce3700 1 -- 172.21.2.117:0/1078776366 >> 172.21.2.117:0/1078776366 conn(0x7f7ac00ce430 msgr2=0x7f7ac00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.199 7f7ac7ce3700 1 -- 172.21.2.117:0/1078776366 shutdown_connections 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.199 7f7ac7ce3700 1 -- 172.21.2.117:0/1078776366 wait complete. 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.213 7f7ac7ce3700 1 Processor -- start 2021-02-10T11:38:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.213 7f7ac7ce3700 1 -- start start 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ac00bf570 0x7f7ac0096340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ac0096960 0x7f7ac0095140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0095dc0 con 0x7f7ac00bf570 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0107da0 con 0x7f7ac0107560 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac7ce3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac0107960 con 0x7f7ac0096960 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50420/0 (socket says 172.21.2.117:50420) 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 -- 172.21.2.117:0/3945916006 learned_addr learned my addr 172.21.2.117:0/3945916006 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac6300700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ac0096960 0x7f7ac0095140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac52fe700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ac00bf570 0x7f7ac0096340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 -- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ac0096960 msgr2=0x7f7ac0095140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ac0096960 0x7f7ac0095140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 -- 172.21.2.117:0/3945916006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ac00bf570 msgr2=0x7f7ac0096340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ac00bf570 0x7f7ac0096340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.214 7f7ac5aff700 1 -- 172.21.2.117:0/3945916006 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0095760 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ac5aff700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f7ab000b5e0 tx=0x7f7ab000b980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab000bd70 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab0002030 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ab00024d0 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0095760 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac0095760 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ab0000c90 con 0x7f7ac0107560 2021-02-10T11:38:39.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.215 7f7ab67fc700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 0x7f7aa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.216 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7ab00326f0 con 0x7f7ac0107560 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.216 7f7ac52fe700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 0x7f7aa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.216 7f7ac52fe700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 0x7f7aa801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.246 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac00ce430 con 0x7f7ac0107560 2021-02-10T11:38:39.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.250 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ab0019940 con 0x7f7ac0107560 2021-02-10T11:38:39.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.379 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802445", "target": ["mgr", ""]}) v1 -- 0x7f7ac00ce430 con 0x7f7aa80186c0 2021-02-10T11:38:39.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.390 7f7ab67fc700 1 -- 172.21.2.117:0/3945916006 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7abc006a40 con 0x7f7aa80186c0 2021-02-10T11:38:39.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.396 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 msgr2=0x7f7aa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.396 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 0x7f7aa801abc0 crc :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.396 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 msgr2=0x7f7ac00bef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.396 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f7ab000b5e0 tx=0x7f7ab000b980).stop 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 shutdown_connections 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7aa80186c0 0x7f7aa801abc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ac00bf570 0x7f7ac0096340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ac0096960 0x7f7ac0095140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 --2- 172.21.2.117:0/3945916006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ac0107560 0x7f7ac00bef50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 >> 172.21.2.117:0/3945916006 conn(0x7f7ac00bd740 msgr2=0x7f7ac00bdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 shutdown_connections 2021-02-10T11:38:39.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.397 7f7ac7ce3700 1 -- 172.21.2.117:0/3945916006 wait complete. 2021-02-10T11:38:39.388 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802446 2021-02-10T11:38:39.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 Processor -- start 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 -- start start 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 0x7f7f8c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c0e4da0 0x7f7f8c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8c077a70 con 0x7f7f8c0e51a0 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8c0777d0 con 0x7f7f8c07da20 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f934b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c07de20 con 0x7f7f8c0e4da0 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f91ad0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f91ad0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46046/0 (socket says 172.21.2.117:46046) 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f91ad0700 1 -- 172.21.2.117:0/75554036 learned_addr learned my addr 172.21.2.117:0/75554036 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.822 7f7f912cf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 0x7f7f8c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f90ace700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c0e4da0 0x7f7f8c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 -- 172.21.2.117:0/75554036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c0e4da0 msgr2=0x7f7f8c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c0e4da0 0x7f7f8c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 -- 172.21.2.117:0/75554036 >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 msgr2=0x7f7f8c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 --2- 172.21.2.117:0/75554036 >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 0x7f7f8c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 -- 172.21.2.117:0/75554036 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c1b0710 con 0x7f7f8c0e51a0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f912cf700 1 --2- 172.21.2.117:0/75554036 >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 0x7f7f8c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f91ad0700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f7f84006f00 tx=0x7f7f84009f10).ready entity=mon.0 client_cookie=6cca18f3dd31ca5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f8bfff700 1 -- 172.21.2.117:0/75554036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8400b3b0 con 0x7f7f8c0e51a0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f8bfff700 1 -- 172.21.2.117:0/75554036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f84000d30 con 0x7f7f8c0e51a0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f8bfff700 1 -- 172.21.2.117:0/75554036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f8400b920 con 0x7f7f8c0e51a0 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f934b3700 1 -- 172.21.2.117:0/75554036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0e51a0 msgr2=0x7f7f8c077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.823 7f7f934b3700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f7f84006f00 tx=0x7f7f84009f10).stop 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 -- 172.21.2.117:0/75554036 shutdown_connections 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0e51a0 0x7f7f8c077ff0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 --2- 172.21.2.117:0/75554036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c0e4da0 0x7f7f8c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 --2- 172.21.2.117:0/75554036 >> v2:172.21.2.125:3300/0 conn(0x7f7f8c07da20 0x7f7f8c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 -- 172.21.2.117:0/75554036 >> 172.21.2.117:0/75554036 conn(0x7f7f8c0ce450 msgr2=0x7f7f8c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 -- 172.21.2.117:0/75554036 shutdown_connections 2021-02-10T11:38:39.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.824 7f7f934b3700 1 -- 172.21.2.117:0/75554036 wait complete. 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 Processor -- start 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 -- start start 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c18fe20 0x7f7f8c0bd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 0x7f7f8c0bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c0be4b0 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c190260 con 0x7f7f8c0bd990 2021-02-10T11:38:39.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.839 7f7f934b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c0c1840 con 0x7f7f8c18fe20 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46052/0 (socket says 172.21.2.117:46052) 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 -- 172.21.2.117:0/389302730 learned_addr learned my addr 172.21.2.117:0/389302730 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f912cf700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c18fe20 0x7f7f8c0bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f90ace700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 0x7f7f8c0bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 -- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c18fe20 msgr2=0x7f7f8c0bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c18fe20 0x7f7f8c0bd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 -- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 msgr2=0x7f7f8c0bdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 0x7f7f8c0bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 -- 172.21.2.117:0/389302730 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c0bf3d0 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f90ace700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 0x7f7f8c0bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:39.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f91ad0700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f7f84006f30 tx=0x7f7f84002d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.840 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f84000d30 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f84005a30 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7f84006020 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f934b3700 1 -- 172.21.2.117:0/389302730 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c0bf3d0 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f934b3700 1 -- 172.21.2.117:0/389302730 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c0bf960 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7f84005c10 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f89ffb700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 0x7f7f6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.841 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7f840328f0 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.842 7f7f912cf700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 0x7f7f6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:39.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.842 7f7f912cf700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 0x7f7f6c01abc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.872 7f7f934b3700 1 -- 172.21.2.117:0/389302730 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c077a70 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:39.874 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7f84000d30 con 0x7f7f8c0be9b0 2021-02-10T11:38:39.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.003 7f7f934b3700 1 -- 172.21.2.117:0/389302730 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802446", "target": ["mgr", ""]}) v1 -- 0x7f7f8c0bf960 con 0x7f7f6c0186c0 2021-02-10T11:38:39.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.015 7f7f89ffb700 1 -- 172.21.2.117:0/389302730 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7f7c006a30 con 0x7f7f6c0186c0 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 -- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 msgr2=0x7f7f6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 0x7f7f6c01abc0 crc :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 -- 172.21.2.117:0/389302730 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 msgr2=0x7f7f8c0bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f7f84006f30 tx=0x7f7f84002d20).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 -- 172.21.2.117:0/389302730 shutdown_connections 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7f6c0186c0 0x7f7f6c01abc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7f8c0be9b0 0x7f7f8c0bedb0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7f8c0bd990 0x7f7f8c0bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 --2- 172.21.2.117:0/389302730 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7f8c18fe20 0x7f7f8c0bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.021 7f7f934b3700 1 -- 172.21.2.117:0/389302730 >> 172.21.2.117:0/389302730 conn(0x7f7f8c0a1360 msgr2=0x7f7f8c091840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.022 7f7f934b3700 1 -- 172.21.2.117:0/389302730 shutdown_connections 2021-02-10T11:38:40.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.022 7f7f934b3700 1 -- 172.21.2.117:0/389302730 wait complete. 2021-02-10T11:38:40.012 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802447 2021-02-10T11:38:40.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.443 7fcd6827e700 1 Processor -- start 2021-02-10T11:38:40.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.443 7fcd6827e700 1 -- start start 2021-02-10T11:38:40.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6827e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd6007da20 0x7fcd600771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6827e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcd600e4da0 0x7fcd601b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6827e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd60077a70 con 0x7fcd600e51a0 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6827e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd600e5c20 con 0x7fcd600e4da0 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6827e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600777d0 con 0x7fcd6007da20 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd65899700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcd600e4da0 0x7fcd601b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6689b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcd600e51a0 0x7fcd60077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39232/0 (socket says 172.21.2.117:39232) 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6689b700 1 -- 172.21.2.117:0/1442062809 learned_addr learned my addr 172.21.2.117:0/1442062809 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd6609a700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd6007da20 0x7fcd600771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd65899700 1 -- 172.21.2.117:0/1442062809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd6007da20 msgr2=0x7fcd600771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd65899700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd6007da20 0x7fcd600771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd65899700 1 -- 172.21.2.117:0/1442062809 >> v1:172.21.2.125:6789/0 conn(0x7fcd600e51a0 legacy=0x7fcd60077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.444 7fcd65899700 1 -- 172.21.2.117:0/1442062809 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6007de20 con 0x7fcd600e4da0 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd65899700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd600e4da0 0x7fcd601b00f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fcd54006650 tx=0x7fcd54009a80).ready entity=mon.0 client_cookie=99ca2241e3f8165f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd65098700 1 -- 172.21.2.117:0/1442062809 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd540009b0 con 0x7fcd600e4da0 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd65098700 1 -- 172.21.2.117:0/1442062809 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5400c0c0 con 0x7fcd600e4da0 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd65098700 1 -- 172.21.2.117:0/1442062809 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5400c510 con 0x7fcd600e4da0 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd6827e700 1 -- 172.21.2.117:0/1442062809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd600e4da0 msgr2=0x7fcd601b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.445 7fcd6827e700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd600e4da0 0x7fcd601b00f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fcd54006650 tx=0x7fcd54009a80).stop 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 -- 172.21.2.117:0/1442062809 shutdown_connections 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd600e4da0 0x7fcd601b00f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 --2- 172.21.2.117:0/1442062809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd6007da20 0x7fcd600771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 -- 172.21.2.117:0/1442062809 >> 172.21.2.117:0/1442062809 conn(0x7fcd600ce450 msgr2=0x7fcd600e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 -- 172.21.2.117:0/1442062809 shutdown_connections 2021-02-10T11:38:40.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.446 7fcd6827e700 1 -- 172.21.2.117:0/1442062809 wait complete. 2021-02-10T11:38:40.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 Processor -- start 2021-02-10T11:38:40.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 -- start start 2021-02-10T11:38:40.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 0x7fcd600a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 0x7fcd600a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600a4220 con 0x7fcd6019a4a0 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600a4410 con 0x7fcd600b9e60 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6827e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd600ba910 con 0x7fcd600a3800 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6609a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6689b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 0x7fcd600a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6609a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46060/0 (socket says 172.21.2.117:46060) 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6609a700 1 -- 172.21.2.117:0/232488984 learned_addr learned my addr 172.21.2.117:0/232488984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6689b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 0x7fcd600a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43740/0 (socket says 172.21.2.117:43740) 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd65899700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 0x7fcd600a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.461 7fcd6609a700 1 -- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 msgr2=0x7fcd600a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6609a700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 0x7fcd600a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6609a700 1 -- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 msgr2=0x7fcd600a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6609a700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 0x7fcd600a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6609a700 1 -- 172.21.2.117:0/232488984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd600bab30 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd65899700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 0x7fcd600a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6609a700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fcd5c006f60 tx=0x7fcd5c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c0009b0 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c00bdf0 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcd5c002030 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6827e700 1 -- 172.21.2.117:0/232488984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd600bab30 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.462 7fcd6827e700 1 -- 172.21.2.117:0/232488984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd600bab30 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.463 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcd5c006b00 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.463 7fcd52ffd700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 0x7fcd4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.463 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcd5c032910 con 0x7fcd6019a4a0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.463 7fcd65899700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 0x7fcd4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.463 7fcd65899700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 0x7fcd4801abc0 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:40.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.493 7fcd6827e700 1 -- 172.21.2.117:0/232488984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd600ba3e0 con 0x7fcd6019a4a0 2021-02-10T11:38:40.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.495 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcd5c006340 con 0x7fcd6019a4a0 2021-02-10T11:38:40.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.623 7fcd6827e700 1 -- 172.21.2.117:0/232488984 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802447", "target": ["mgr", ""]}) v1 -- 0x7fcd600ba3e0 con 0x7fcd48018730 2021-02-10T11:38:40.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.635 7fcd52ffd700 1 -- 172.21.2.117:0/232488984 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcd54006390 con 0x7fcd48018730 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 -- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 msgr2=0x7fcd4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 0x7fcd4801abc0 crc :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 -- 172.21.2.117:0/232488984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 msgr2=0x7fcd600b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7fcd5c006f60 tx=0x7fcd5c00b970).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 -- 172.21.2.117:0/232488984 shutdown_connections 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcd600a3800 0x7fcd600a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcd600b9e60 0x7fcd600a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcd48018730 0x7fcd4801abc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 --2- 172.21.2.117:0/232488984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcd6019a4a0 0x7fcd600b9840 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:40.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.640 7fcd6827e700 1 -- 172.21.2.117:0/232488984 >> 172.21.2.117:0/232488984 conn(0x7fcd60107b60 msgr2=0x7fcd600c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:40.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.641 7fcd6827e700 1 -- 172.21.2.117:0/232488984 shutdown_connections 2021-02-10T11:38:40.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:40.641 7fcd6827e700 1 -- 172.21.2.117:0/232488984 wait complete. 2021-02-10T11:38:40.631 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802448 2021-02-10T11:38:41.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.060 7f8f619f8700 1 Processor -- start 2021-02-10T11:38:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.060 7f8f619f8700 1 -- start start 2021-02-10T11:38:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f619f8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f619f8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 0x7f8f5c1b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f619f8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c077a70 con 0x7f8f5c07da20 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f619f8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5c0e53a0 con 0x7f8f5c079000 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f619f8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c079400 con 0x7f8f5c1b00d0 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46068/0 (socket says 172.21.2.117:46068) 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5affd700 1 -- 172.21.2.117:0/1376976161 learned_addr learned my addr 172.21.2.117:0/1376976161 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:41.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5bfff700 1 --2- 172.21.2.117:0/1376976161 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 0x7f8f5c1b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5bfff700 1 -- 172.21.2.117:0/1376976161 >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 msgr2=0x7f8f5c0e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5bfff700 1 --2- 172.21.2.117:0/1376976161 >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5bfff700 1 -- 172.21.2.117:0/1376976161 >> v1:172.21.2.125:6789/0 conn(0x7f8f5c07da20 legacy=0x7f8f5c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.061 7f8f5bfff700 1 -- 172.21.2.117:0/1376976161 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c07de20 con 0x7f8f5c1b00d0 2021-02-10T11:38:41.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f5affd700 1 --2- 172.21.2.117:0/1376976161 >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:41.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f5bfff700 1 --2- 172.21.2.117:0/1376976161 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 0x7f8f5c1b04d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8f54006e50 tx=0x7f8f54009e70).ready entity=mon.2 client_cookie=118ca8338cbb4d0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f5a7fc700 1 -- 172.21.2.117:0/1376976161 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f54006a20 con 0x7f8f5c1b00d0 2021-02-10T11:38:41.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f5a7fc700 1 -- 172.21.2.117:0/1376976161 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54006790 con 0x7f8f5c1b00d0 2021-02-10T11:38:41.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f5a7fc700 1 -- 172.21.2.117:0/1376976161 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f5400b700 con 0x7f8f5c1b00d0 2021-02-10T11:38:41.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f619f8700 1 -- 172.21.2.117:0/1376976161 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 msgr2=0x7f8f5c1b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.062 7f8f619f8700 1 --2- 172.21.2.117:0/1376976161 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 0x7f8f5c1b04d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8f54006e50 tx=0x7f8f54009e70).stop 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 -- 172.21.2.117:0/1376976161 shutdown_connections 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 --2- 172.21.2.117:0/1376976161 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c1b00d0 0x7f8f5c1b04d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 --2- 172.21.2.117:0/1376976161 >> v2:172.21.2.117:3300/0 conn(0x7f8f5c079000 0x7f8f5c0e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 -- 172.21.2.117:0/1376976161 >> 172.21.2.117:0/1376976161 conn(0x7f8f5c0ce430 msgr2=0x7f8f5c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 -- 172.21.2.117:0/1376976161 shutdown_connections 2021-02-10T11:38:41.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.063 7f8f619f8700 1 -- 172.21.2.117:0/1376976161 wait complete. 2021-02-10T11:38:41.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 Processor -- start 2021-02-10T11:38:41.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 -- start start 2021-02-10T11:38:41.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f5c091a50 0x7f8f5c0b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 0x7f8f5c0ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c091ea0 con 0x7f8f5c187bd0 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c187fd0 con 0x7f8f5c091a50 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f619f8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c1881f0 con 0x7f8f5c199750 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 0x7f8f5c0ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 0x7f8f5c0ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43754/0 (socket says 172.21.2.117:43754) 2021-02-10T11:38:41.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46076/0 (socket says 172.21.2.117:46076) 2021-02-10T11:38:41.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5bfff700 1 -- 172.21.2.117:0/1117698632 learned_addr learned my addr 172.21.2.117:0/1117698632 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:41.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.078 7f8f5b7fe700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f5c091a50 0x7f8f5c0b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 -- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 msgr2=0x7f8f5c0ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 0x7f8f5c0ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 -- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f5c091a50 msgr2=0x7f8f5c0b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f5c091a50 0x7f8f5c0b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 -- 172.21.2.117:0/1117698632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0abee0 con 0x7f8f5c187bd0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f5affd700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f8f4c006ed0 tx=0x7f8f4c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f4c00bd30 con 0x7f8f5c187bd0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c000c90 con 0x7f8f5c187bd0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f4c0022b0 con 0x7f8f5c187bd0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c0abee0 con 0x7f8f5c187bd0 2021-02-10T11:38:41.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c0ac430 con 0x7f8f5c187bd0 2021-02-10T11:38:41.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.079 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f4c002240 con 0x7f8f5c187bd0 2021-02-10T11:38:41.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.080 7f8f53fff700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 0x7f8f4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.080 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8f4c032620 con 0x7f8f5c187bd0 2021-02-10T11:38:41.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.080 7f8f5b7fe700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 0x7f8f4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.080 7f8f5b7fe700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 0x7f8f4001abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.111 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c077a70 con 0x7f8f5c187bd0 2021-02-10T11:38:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.113 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f4c002030 con 0x7f8f5c187bd0 2021-02-10T11:38:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.238 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802448", "target": ["mgr", ""]}) v1 -- 0x7f8f5c0ac430 con 0x7f8f400186c0 2021-02-10T11:38:41.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.249 7f8f53fff700 1 -- 172.21.2.117:0/1117698632 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8f440009b0 con 0x7f8f400186c0 2021-02-10T11:38:41.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 msgr2=0x7f8f4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 0x7f8f4001abc0 crc :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 msgr2=0x7f8f5c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f8f4c006ed0 tx=0x7f8f4c00b940).stop 2021-02-10T11:38:41.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 shutdown_connections 2021-02-10T11:38:41.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f400186c0 0x7f8f4001abc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f5c187bd0 0x7f8f5c199130 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f5c199750 0x7f8f5c0ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 --2- 172.21.2.117:0/1117698632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f5c091a50 0x7f8f5c0b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 >> 172.21.2.117:0/1117698632 conn(0x7f8f5c0b5320 msgr2=0x7f8f5c1b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 shutdown_connections 2021-02-10T11:38:41.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.256 7f8f619f8700 1 -- 172.21.2.117:0/1117698632 wait complete. 2021-02-10T11:38:41.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802449 2021-02-10T11:38:41.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 Processor -- start 2021-02-10T11:38:41.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 -- start start 2021-02-10T11:38:41.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8738079000 0x7f87380e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87381b00f0 0x7f87381b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8738077a70 con 0x7f873807da20 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380e53c0 con 0x7f87381b00f0 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873e7da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8738079400 con 0x7f8738079000 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f8737fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f873807da20 0x7f87380771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46332/0 (socket says 172.21.2.117:46332) 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f8737fff700 1 -- 172.21.2.117:0/3090016609 learned_addr learned my addr 172.21.2.117:0/3090016609 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f873cdf7700 1 --2- 172.21.2.117:0/3090016609 >> v2:172.21.2.125:3300/0 conn(0x7f87381b00f0 0x7f87381b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.680 7f87377fe700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8738079000 0x7f87380e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f8736ffd700 1 -- 172.21.2.117:0/3090016609 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651729125 0 0) 0x7f8728001180 con 0x7f873807da20 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f8736ffd700 1 -- 172.21.2.117:0/3090016609 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8720002fe0 con 0x7f873807da20 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873cdf7700 1 -- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8738079000 msgr2=0x7f87380e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873cdf7700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8738079000 0x7f87380e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873cdf7700 1 -- 172.21.2.117:0/3090016609 >> v1:172.21.2.117:6789/0 conn(0x7f873807da20 legacy=0x7f87380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873cdf7700 1 -- 172.21.2.117:0/3090016609 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873807de20 con 0x7f87381b00f0 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873cdf7700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87381b00f0 0x7f87381b04f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8730006e50 tx=0x7f8730009e70).ready entity=mon.1 client_cookie=1a4fd5a8f5699a7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f8736ffd700 1 -- 172.21.2.117:0/3090016609 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f873000b510 con 0x7f87381b00f0 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f8736ffd700 1 -- 172.21.2.117:0/3090016609 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730006790 con 0x7f87381b00f0 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f8736ffd700 1 -- 172.21.2.117:0/3090016609 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8730000f90 con 0x7f87381b00f0 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873e7da700 1 -- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87381b00f0 msgr2=0x7f87381b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.681 7f873e7da700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87381b00f0 0x7f87381b04f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8730006e50 tx=0x7f8730009e70).stop 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 -- 172.21.2.117:0/3090016609 shutdown_connections 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87381b00f0 0x7f87381b04f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 --2- 172.21.2.117:0/3090016609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8738079000 0x7f87380e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 -- 172.21.2.117:0/3090016609 >> 172.21.2.117:0/3090016609 conn(0x7f87380ce450 msgr2=0x7f87380e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 -- 172.21.2.117:0/3090016609 shutdown_connections 2021-02-10T11:38:41.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.682 7f873e7da700 1 -- 172.21.2.117:0/3090016609 wait complete. 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.696 7f873e7da700 1 Processor -- start 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.696 7f873e7da700 1 -- start start 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873808eb10 0x7f87380c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873808dfe0 0x7f873808e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380af540 con 0x7f87380afac0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380a0c80 con 0x7f873808dfe0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873e7da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b04e0 con 0x7f873808eb10 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46090/0 (socket says 172.21.2.117:46090) 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f87377fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873808dfe0 0x7f873808e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f8737fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873808eb10 0x7f87380c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 -- 172.21.2.117:0/1976196851 learned_addr learned my addr 172.21.2.117:0/1976196851 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 -- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873808eb10 msgr2=0x7f87380c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873808eb10 0x7f87380c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 -- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873808dfe0 msgr2=0x7f873808e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873808dfe0 0x7f873808e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.697 7f873cdf7700 1 -- 172.21.2.117:0/1976196851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380b0700 con 0x7f87380afac0 2021-02-10T11:38:41.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f873cdf7700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f873000ba40 tx=0x7f873000bde0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87300013b0 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87300015b0 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8730001a90 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f873e7da700 1 -- 172.21.2.117:0/1976196851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87380b0700 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f873e7da700 1 -- 172.21.2.117:0/1976196851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87380b0700 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87300011a0 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 0x7f871801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.698 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8730032b70 con 0x7f87380afac0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.699 7f8737fff700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 0x7f871801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:41.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.699 7f8737fff700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 0x7f871801ac10 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:41.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.730 7f873e7da700 1 -- 172.21.2.117:0/1976196851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8738103280 con 0x7f87380afac0 2021-02-10T11:38:41.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.732 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8730009320 con 0x7f87380afac0 2021-02-10T11:38:41.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.862 7f873e7da700 1 -- 172.21.2.117:0/1976196851 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802449", "target": ["mgr", ""]}) v1 -- 0x7f87380a0c80 con 0x7f8718018730 2021-02-10T11:38:41.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.873 7f8734ff9700 1 -- 172.21.2.117:0/1976196851 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8728000ce0 con 0x7f8718018730 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 -- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 msgr2=0x7f871801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 0x7f871801ac10 crc :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 -- 172.21.2.117:0/1976196851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 msgr2=0x7f87380afec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f873000ba40 tx=0x7f873000bde0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 -- 172.21.2.117:0/1976196851 shutdown_connections 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8718018730 0x7f871801ac10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87380afac0 0x7f87380afec0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f873808dfe0 0x7f873808e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 --2- 172.21.2.117:0/1976196851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f873808eb10 0x7f87380c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 -- 172.21.2.117:0/1976196851 >> 172.21.2.117:0/1976196851 conn(0x7f87380a1850 msgr2=0x7f87380bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.879 7f873e7da700 1 -- 172.21.2.117:0/1976196851 shutdown_connections 2021-02-10T11:38:41.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:41.880 7f873e7da700 1 -- 172.21.2.117:0/1976196851 wait complete. 2021-02-10T11:38:41.870 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802450 2021-02-10T11:38:42.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.300 7f17f3550700 1 Processor -- start 2021-02-10T11:38:42.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.300 7f17f3550700 1 -- start start 2021-02-10T11:38:42.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 0x7f17ec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 0x7f17ec1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17ec0e51a0 0x7f17ec077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec077a70 con 0x7f17ec07da20 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec0777d0 con 0x7f17ec0e51a0 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f3550700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec07de20 con 0x7f17ec0e4da0 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f136c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 0x7f17ec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 0x7f17ec1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f136c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 0x7f17ec0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46098/0 (socket says 172.21.2.117:46098) 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f136c700 1 -- 172.21.2.117:0/210310313 learned_addr learned my addr 172.21.2.117:0/210310313 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f1b6d700 1 --2- 172.21.2.117:0/210310313 >> v2:172.21.2.125:3300/0 conn(0x7f17ec0e51a0 0x7f17ec077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 -- 172.21.2.117:0/210310313 >> v2:172.21.2.125:3300/0 conn(0x7f17ec0e51a0 msgr2=0x7f17ec077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 --2- 172.21.2.117:0/210310313 >> v2:172.21.2.125:3300/0 conn(0x7f17ec0e51a0 0x7f17ec077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 -- 172.21.2.117:0/210310313 >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 msgr2=0x7f17ec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 --2- 172.21.2.117:0/210310313 >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 0x7f17ec0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.301 7f17f0b6b700 1 -- 172.21.2.117:0/210310313 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec1b0710 con 0x7f17ec0e4da0 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17f0b6b700 1 --2- 172.21.2.117:0/210310313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 0x7f17ec1b00f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f17e40069d0 tx=0x7f17e4009e40).ready entity=mon.2 client_cookie=41a1b460b80ccf50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17e3fff700 1 -- 172.21.2.117:0/210310313 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400b510 con 0x7f17ec0e4da0 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17e3fff700 1 -- 172.21.2.117:0/210310313 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400b790 con 0x7f17ec0e4da0 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17e3fff700 1 -- 172.21.2.117:0/210310313 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400bc70 con 0x7f17ec0e4da0 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17f3550700 1 -- 172.21.2.117:0/210310313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 msgr2=0x7f17ec1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.302 7f17f3550700 1 --2- 172.21.2.117:0/210310313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 0x7f17ec1b00f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f17e40069d0 tx=0x7f17e4009e40).stop 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 -- 172.21.2.117:0/210310313 shutdown_connections 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 --2- 172.21.2.117:0/210310313 >> v2:172.21.2.125:3300/0 conn(0x7f17ec0e51a0 0x7f17ec077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 --2- 172.21.2.117:0/210310313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec0e4da0 0x7f17ec1b00f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 --2- 172.21.2.117:0/210310313 >> v2:172.21.2.117:3300/0 conn(0x7f17ec07da20 0x7f17ec0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 -- 172.21.2.117:0/210310313 >> 172.21.2.117:0/210310313 conn(0x7f17ec0ce450 msgr2=0x7f17ec0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 -- 172.21.2.117:0/210310313 shutdown_connections 2021-02-10T11:38:42.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.303 7f17f3550700 1 -- 172.21.2.117:0/210310313 wait complete. 2021-02-10T11:38:42.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 Processor -- start 2021-02-10T11:38:42.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 -- start start 2021-02-10T11:38:42.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec1a0250 0x7f17ec106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec090420 0x7f17ec090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec1067f0 con 0x7f17ec187e20 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec107350 con 0x7f17ec090420 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f3550700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec1042d0 con 0x7f17ec1a0250 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f0b6b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f0b6b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46104/0 (socket says 172.21.2.117:46104) 2021-02-10T11:38:42.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.318 7f17f0b6b700 1 -- 172.21.2.117:0/2179385359 learned_addr learned my addr 172.21.2.117:0/2179385359 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f136c700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec1a0250 0x7f17ec106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f1b6d700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec090420 0x7f17ec090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 -- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec1a0250 msgr2=0x7f17ec106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec1a0250 0x7f17ec106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 -- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec090420 msgr2=0x7f17ec090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec090420 0x7f17ec090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 -- 172.21.2.117:0/2179385359 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec090e40 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f0b6b700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f17e40099e0 tx=0x7f17e400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e4006240 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400be60 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e4005660 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f3550700 1 -- 172.21.2.117:0/2179385359 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec090e40 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.319 7f17f3550700 1 -- 172.21.2.117:0/2179385359 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec090e40 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.320 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17e400a000 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.320 7f17e1ffb700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.320 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f17e4032aa0 con 0x7f17ec187e20 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.320 7f17f136c700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.321 7f17f136c700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 0x7f17cc01abc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.350 7f17f3550700 1 -- 172.21.2.117:0/2179385359 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec103280 con 0x7f17ec187e20 2021-02-10T11:38:42.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.353 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17e402e020 con 0x7f17ec187e20 2021-02-10T11:38:42.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.484 7f17f3550700 1 -- 172.21.2.117:0/2179385359 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802450", "target": ["mgr", ""]}) v1 -- 0x7f17ec103280 con 0x7f17cc0186c0 2021-02-10T11:38:42.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.495 7f17e1ffb700 1 -- 172.21.2.117:0/2179385359 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f17dc006a80 con 0x7f17cc0186c0 2021-02-10T11:38:42.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 msgr2=0x7f17cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 0x7f17cc01abc0 crc :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 msgr2=0x7f17ec188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f17e40099e0 tx=0x7f17e400bd20).stop 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 shutdown_connections 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17cc0186c0 0x7f17cc01abc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec187e20 0x7f17ec188220 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec090420 0x7f17ec090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 --2- 172.21.2.117:0/2179385359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec1a0250 0x7f17ec106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 >> 172.21.2.117:0/2179385359 conn(0x7f17ec0a1850 msgr2=0x7f17ec0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 shutdown_connections 2021-02-10T11:38:42.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.501 7f17f3550700 1 -- 172.21.2.117:0/2179385359 wait complete. 2021-02-10T11:38:42.492 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802451 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 Processor -- start 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 -- start start 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc1840e4da0 0x7fc1841b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc184077a70 con 0x7fc1840e51a0 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840e5c20 con 0x7fc1840e4da0 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.924 7fc1891a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840777d0 con 0x7fc18407da20 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc182d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc182d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43790/0 (socket says 172.21.2.117:43790) 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc182d9d700 1 -- 172.21.2.117:0/3830024953 learned_addr learned my addr 172.21.2.117:0/3830024953 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 --2- 172.21.2.117:0/3830024953 >> v2:172.21.2.125:3300/0 conn(0x7fc1840e4da0 0x7fc1841b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 -- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 msgr2=0x7fc1840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 -- 172.21.2.117:0/3830024953 >> v1:172.21.2.125:6789/0 conn(0x7fc1840e51a0 legacy=0x7fc184077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 -- 172.21.2.117:0/3830024953 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18407de20 con 0x7fc1840e4da0 2021-02-10T11:38:42.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc182d9d700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.925 7fc18259c700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1840e4da0 0x7fc1841b00f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc17c0069d0 tx=0x7fc17c009e40).ready entity=mon.1 client_cookie=58f9959aefb4f1ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc181d9b700 1 -- 172.21.2.117:0/3830024953 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17c0066c0 con 0x7fc1840e4da0 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc181d9b700 1 -- 172.21.2.117:0/3830024953 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c00b510 con 0x7fc1840e4da0 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc181d9b700 1 -- 172.21.2.117:0/3830024953 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17c00ba30 con 0x7fc1840e4da0 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 -- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1840e4da0 msgr2=0x7fc1841b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1840e4da0 0x7fc1841b00f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc17c0069d0 tx=0x7fc17c009e40).stop 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 -- 172.21.2.117:0/3830024953 shutdown_connections 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1840e4da0 0x7fc1841b00f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 --2- 172.21.2.117:0/3830024953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc18407da20 0x7fc1840771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 -- 172.21.2.117:0/3830024953 >> 172.21.2.117:0/3830024953 conn(0x7fc1840ce450 msgr2=0x7fc1840e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:42.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 -- 172.21.2.117:0/3830024953 shutdown_connections 2021-02-10T11:38:42.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.927 7fc1891a4700 1 -- 172.21.2.117:0/3830024953 wait complete. 2021-02-10T11:38:42.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 Processor -- start 2021-02-10T11:38:42.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 -- start start 2021-02-10T11:38:42.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1841b8130 0x7fc1841b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1841b9240 0x7fc1841b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841b86a0 con 0x7fc184187ee0 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18408a180 con 0x7fc1841b8130 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc1891a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841b9c60 con 0x7fc1841b9240 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc182d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc182d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46114/0 (socket says 172.21.2.117:46114) 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc182d9d700 1 -- 172.21.2.117:0/772591673 learned_addr learned my addr 172.21.2.117:0/772591673 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.942 7fc18359e700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1841b9240 0x7fc1841b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc18259c700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1841b8130 0x7fc1841b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 -- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1841b9240 msgr2=0x7fc1841b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1841b9240 0x7fc1841b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 -- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1841b8130 msgr2=0x7fc1841b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1841b8130 0x7fc1841b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 -- 172.21.2.117:0/772591673 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840c3fc0 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc182d9d700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fc178006ee0 tx=0x7fc17800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc17800bd50 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1780052e0 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc178005840 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1891a4700 1 -- 172.21.2.117:0/772591673 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1840c3fc0 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1891a4700 1 -- 172.21.2.117:0/772591673 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1840c3fc0 con 0x7fc184187ee0 2021-02-10T11:38:42.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.943 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1780054c0 con 0x7fc184187ee0 2021-02-10T11:38:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.944 7fc1737fe700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 0x7fc16801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.944 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc178032550 con 0x7fc184187ee0 2021-02-10T11:38:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.944 7fc18259c700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 0x7fc16801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:42.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.944 7fc18259c700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 0x7fc16801abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.974 7fc1891a4700 1 -- 172.21.2.117:0/772591673 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1841b9c60 con 0x7fc184187ee0 2021-02-10T11:38:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:42.976 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc178030d30 con 0x7fc184187ee0 2021-02-10T11:38:43.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.102 7fc1891a4700 1 -- 172.21.2.117:0/772591673 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802451", "target": ["mgr", ""]}) v1 -- 0x7fc1841b9c60 con 0x7fc1680186c0 2021-02-10T11:38:43.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.114 7fc1737fe700 1 -- 172.21.2.117:0/772591673 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc17c000d30 con 0x7fc1680186c0 2021-02-10T11:38:43.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 msgr2=0x7fc16801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 0x7fc16801abc0 crc :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 msgr2=0x7fc1841882e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fc178006ee0 tx=0x7fc17800b960).stop 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 shutdown_connections 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1841b9240 0x7fc1841b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1841b8130 0x7fc1841b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1680186c0 0x7fc16801abc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 --2- 172.21.2.117:0/772591673 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc184187ee0 0x7fc1841882e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 >> 172.21.2.117:0/772591673 conn(0x7fc1840a4180 msgr2=0x7fc18409be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 shutdown_connections 2021-02-10T11:38:43.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.120 7fc1891a4700 1 -- 172.21.2.117:0/772591673 wait complete. 2021-02-10T11:38:43.111 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802452 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 Processor -- start 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 -- start start 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5214079000 0x7f52140e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5214077a70 con 0x7f521407da20 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52140e53a0 con 0x7f5214079000 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.543 7f521af36700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5214079400 con 0x7f52141b00d0 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5219553700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5218d52700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f521407da20 0x7f52140771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39298/0 (socket says 172.21.2.117:39298) 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5218d52700 1 -- 172.21.2.117:0/2950028139 learned_addr learned my addr 172.21.2.117:0/2950028139 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5219553700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43802/0 (socket says 172.21.2.117:43802) 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 --2- 172.21.2.117:0/2950028139 >> v2:172.21.2.117:3300/0 conn(0x7f5214079000 0x7f52140e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 -- 172.21.2.117:0/2950028139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 msgr2=0x7f52141b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 -- 172.21.2.117:0/2950028139 >> v1:172.21.2.125:6789/0 conn(0x7f521407da20 legacy=0x7f52140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 -- 172.21.2.117:0/2950028139 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521407de20 con 0x7f5214079000 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5219553700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f5213fff700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5214079000 0x7f52140e4d80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f52040069d0 tx=0x7f5204009e40).ready entity=mon.0 client_cookie=7c242509e1689a79 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.544 7f52137fe700 1 -- 172.21.2.117:0/2950028139 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f520400b510 con 0x7f5214079000 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f52137fe700 1 -- 172.21.2.117:0/2950028139 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520400b890 con 0x7f5214079000 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f52137fe700 1 -- 172.21.2.117:0/2950028139 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f520400bdb0 con 0x7f5214079000 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 -- 172.21.2.117:0/2950028139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5214079000 msgr2=0x7f52140e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5214079000 0x7f52140e4d80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f52040069d0 tx=0x7f5204009e40).stop 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 -- 172.21.2.117:0/2950028139 shutdown_connections 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52141b00d0 0x7f52141b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 --2- 172.21.2.117:0/2950028139 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5214079000 0x7f52140e4d80 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 -- 172.21.2.117:0/2950028139 >> 172.21.2.117:0/2950028139 conn(0x7f52140ce430 msgr2=0x7f52140e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 -- 172.21.2.117:0/2950028139 shutdown_connections 2021-02-10T11:38:43.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.545 7f521af36700 1 -- 172.21.2.117:0/2950028139 wait complete. 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 Processor -- start 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 -- start start 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 0x7f521418a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52140b78d0 0x7f52140b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 0x7f5214104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52140ba970 con 0x7f52140b78d0 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214104b90 con 0x7f52140b82f0 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.560 7f521af36700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5214104d70 con 0x7f52140a1310 2021-02-10T11:38:43.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5219553700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 0x7f5214104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 0x7f521418a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5219553700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 0x7f5214104570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50512/0 (socket says 172.21.2.117:50512) 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5219553700 1 -- 172.21.2.117:0/3025088693 learned_addr learned my addr 172.21.2.117:0/3025088693 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5213fff700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52140b78d0 0x7f52140b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 -- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 msgr2=0x7f5214104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 0x7f5214104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 -- 172.21.2.117:0/3025088693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52140b78d0 msgr2=0x7f52140b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52140b78d0 0x7f52140b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 -- 172.21.2.117:0/3025088693 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214104f90 con 0x7f52140a1310 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f5218d52700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 0x7f521418a090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f51fc001440 tx=0x7f51fc00a8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51fc00ac90 con 0x7f52140a1310 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f521af36700 1 -- 172.21.2.117:0/3025088693 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5214104f90 con 0x7f52140a1310 2021-02-10T11:38:43.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f521af36700 1 -- 172.21.2.117:0/3025088693 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5214105520 con 0x7f52140a1310 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc00ac90 con 0x7f52140a1310 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.561 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51fc0070d0 con 0x7f52140a1310 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.562 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51fc004530 con 0x7f52140a1310 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.562 7f52117fa700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 0x7f51f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.562 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f51fc0326b0 con 0x7f52140a1310 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.562 7f5213fff700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 0x7f51f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:43.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.562 7f5213fff700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 0x7f51f801ac10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.592 7f521af36700 1 -- 172.21.2.117:0/3025088693 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214077a70 con 0x7f52140a1310 2021-02-10T11:38:43.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.596 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51fc004320 con 0x7f52140a1310 2021-02-10T11:38:43.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.720 7f521af36700 1 -- 172.21.2.117:0/3025088693 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802452", "target": ["mgr", ""]}) v1 -- 0x7f5214077a70 con 0x7f51f8018730 2021-02-10T11:38:43.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.731 7f52117fa700 1 -- 172.21.2.117:0/3025088693 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5204006ac0 con 0x7f51f8018730 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 -- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 msgr2=0x7f51f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 0x7f51f801ac10 crc :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 -- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 msgr2=0x7f521418a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 0x7f521418a090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f51fc001440 tx=0x7f51fc00a8a0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 -- 172.21.2.117:0/3025088693 shutdown_connections 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51f8018730 0x7f51f801ac10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.737 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52140b78d0 0x7f52140b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.738 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52140b82f0 0x7f5214104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.738 7f521af36700 1 --2- 172.21.2.117:0/3025088693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52140a1310 0x7f521418a090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.738 7f521af36700 1 -- 172.21.2.117:0/3025088693 >> 172.21.2.117:0/3025088693 conn(0x7f52140bacb0 msgr2=0x7f521408e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.738 7f521af36700 1 -- 172.21.2.117:0/3025088693 shutdown_connections 2021-02-10T11:38:43.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:43.738 7f521af36700 1 -- 172.21.2.117:0/3025088693 wait complete. 2021-02-10T11:38:43.728 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802453 2021-02-10T11:38:44.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.158 7f2cc8a31700 1 Processor -- start 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.158 7f2cc8a31700 1 -- start start 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc8a31700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2cc407da20 0x7f2cc40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc8a31700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2cc40e4da0 0x7f2cc41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc8a31700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc4077a70 con 0x7f2cc40e51a0 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc8a31700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc40777d0 con 0x7f2cc40e4da0 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc8a31700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc407de20 con 0x7f2cc407da20 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc2e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2cc40e51a0 0x7f2cc4077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46384/0 (socket says 172.21.2.117:46384) 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc2e1d700 1 -- 172.21.2.117:0/563703493 learned_addr learned my addr 172.21.2.117:0/563703493 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:44.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 --2- 172.21.2.117:0/563703493 >> v2:172.21.2.117:3300/0 conn(0x7f2cc40e4da0 0x7f2cc41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc161a700 1 -- 172.21.2.117:0/563703493 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778538788 0 0) 0x7f2cbc001500 con 0x7f2cc40e51a0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc161a700 1 -- 172.21.2.117:0/563703493 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb0002fe0 con 0x7f2cc40e51a0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 -- 172.21.2.117:0/563703493 >> v2:172.21.2.125:3300/0 conn(0x7f2cc407da20 msgr2=0x7f2cc40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 --2- 172.21.2.117:0/563703493 >> v2:172.21.2.125:3300/0 conn(0x7f2cc407da20 0x7f2cc40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 -- 172.21.2.117:0/563703493 >> v1:172.21.2.117:6789/0 conn(0x7f2cc40e51a0 legacy=0x7f2cc4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 -- 172.21.2.117:0/563703493 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc41b0710 con 0x7f2cc40e4da0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.159 7f2cc1e1b700 1 --2- 172.21.2.117:0/563703493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40e4da0 0x7f2cc41b00f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f2cac006650 tx=0x7f2cac009a80).ready entity=mon.0 client_cookie=faeaf758c481de3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc161a700 1 -- 172.21.2.117:0/563703493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cac0009b0 con 0x7f2cc40e4da0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc161a700 1 -- 172.21.2.117:0/563703493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac00c0c0 con 0x7f2cc40e4da0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc161a700 1 -- 172.21.2.117:0/563703493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cac00c520 con 0x7f2cc40e4da0 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 -- 172.21.2.117:0/563703493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40e4da0 msgr2=0x7f2cc41b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 --2- 172.21.2.117:0/563703493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40e4da0 0x7f2cc41b00f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f2cac006650 tx=0x7f2cac009a80).stop 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 -- 172.21.2.117:0/563703493 shutdown_connections 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 --2- 172.21.2.117:0/563703493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40e4da0 0x7f2cc41b00f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 --2- 172.21.2.117:0/563703493 >> v2:172.21.2.125:3300/0 conn(0x7f2cc407da20 0x7f2cc40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 -- 172.21.2.117:0/563703493 >> 172.21.2.117:0/563703493 conn(0x7f2cc40ce450 msgr2=0x7f2cc40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.160 7f2cc8a31700 1 -- 172.21.2.117:0/563703493 shutdown_connections 2021-02-10T11:38:44.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.161 7f2cc8a31700 1 -- 172.21.2.117:0/563703493 wait complete. 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 Processor -- start 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 -- start start 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc40a41c0 0x7f2cc40fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc40b7390 0x7f2cc40b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b8010 con 0x7f2cc40c2340 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b8550 con 0x7f2cc40b7390 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.175 7f2cc8a31700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc40b8770 con 0x7f2cc40a41c0 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46138/0 (socket says 172.21.2.117:46138) 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 -- 172.21.2.117:0/3640253944 learned_addr learned my addr 172.21.2.117:0/3640253944 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc261c700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc40a41c0 0x7f2cc40fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc2e1d700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc40b7390 0x7f2cc40b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 -- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc40a41c0 msgr2=0x7f2cc40fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc40a41c0 0x7f2cc40fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 -- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc40b7390 msgr2=0x7f2cc40b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc40b7390 0x7f2cc40b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 -- 172.21.2.117:0/3640253944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc40b8990 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.176 7f2cc1e1b700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f2cac009630 tx=0x7f2cac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cac0093a0 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac00bd70 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cac005560 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc40b8990 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc40b8f20 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cac00a000 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.177 7f2cbaffd700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 0x7f2ca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.178 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2cac0326a0 con 0x7f2cc40c2340 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.178 7f2cc261c700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 0x7f2ca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.178 7f2cc261c700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 0x7f2ca801abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.209 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc41b4f00 con 0x7f2cc40c2340 2021-02-10T11:38:44.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.211 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cac02a610 con 0x7f2cc40c2340 2021-02-10T11:38:44.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.336 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802453", "target": ["mgr", ""]}) v1 -- 0x7f2cc4077a70 con 0x7f2ca80186c0 2021-02-10T11:38:44.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.350 7f2cbaffd700 1 -- 172.21.2.117:0/3640253944 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2cb4006550 con 0x7f2ca80186c0 2021-02-10T11:38:44.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 msgr2=0x7f2ca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 0x7f2ca801abc0 crc :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 msgr2=0x7f2cc40c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f2cac009630 tx=0x7f2cac00b980).stop 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 shutdown_connections 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca80186c0 0x7f2ca801abc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc40c2340 0x7f2cc40c2740 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc40b7390 0x7f2cc40b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 --2- 172.21.2.117:0/3640253944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc40a41c0 0x7f2cc40fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 >> 172.21.2.117:0/3640253944 conn(0x7f2cc408bbc0 msgr2=0x7f2cc40c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 shutdown_connections 2021-02-10T11:38:44.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.354 7f2cc8a31700 1 -- 172.21.2.117:0/3640253944 wait complete. 2021-02-10T11:38:44.345 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802454 2021-02-10T11:38:44.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.777 7fb8c649d700 1 Processor -- start 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.777 7fb8c649d700 1 -- start start 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.777 7fb8c649d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb8c0079000 0x7fb8c00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8c01b00f0 0x7fb8c01b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c0077a70 con 0x7fb8c007da20 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c00e5c20 con 0x7fb8c01b00f0 2021-02-10T11:38:44.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8c00e53c0 con 0x7fb8c0079000 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb8c0079000 0x7fb8c00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c4aba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8c01b00f0 0x7fb8c01b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb8c0079000 0x7fb8c00e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50532/0 (socket says 172.21.2.117:50532) 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 -- 172.21.2.117:0/2585957917 learned_addr learned my addr 172.21.2.117:0/2585957917 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8bffff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fb8c007da20 0x7fb8c00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39322/0 (socket says 172.21.2.117:39322) 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 -- 172.21.2.117:0/2585957917 >> v2:172.21.2.117:3300/0 conn(0x7fb8c01b00f0 msgr2=0x7fb8c01b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 --2- 172.21.2.117:0/2585957917 >> v2:172.21.2.117:3300/0 conn(0x7fb8c01b00f0 0x7fb8c01b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 -- 172.21.2.117:0/2585957917 >> v1:172.21.2.125:6789/0 conn(0x7fb8c007da20 legacy=0x7fb8c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8bf7fe700 1 -- 172.21.2.117:0/2585957917 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086435501 0 0) 0x7fb8b0000c60 con 0x7fb8c007da20 2021-02-10T11:38:44.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 -- 172.21.2.117:0/2585957917 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c00d3400 con 0x7fb8c0079000 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8b7fff700 1 --2- 172.21.2.117:0/2585957917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0079000 0x7fb8c00e4da0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb8a8006700 tx=0x7fb8a8009b80).ready entity=mon.1 client_cookie=f598f4e7438d0f7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8bf7fe700 1 -- 172.21.2.117:0/2585957917 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8a800c0c0 con 0x7fb8c0079000 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8bf7fe700 1 -- 172.21.2.117:0/2585957917 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a800c400 con 0x7fb8c0079000 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8bf7fe700 1 -- 172.21.2.117:0/2585957917 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8a80009b0 con 0x7fb8c0079000 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 -- 172.21.2.117:0/2585957917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0079000 msgr2=0x7fb8c00e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.778 7fb8c649d700 1 --2- 172.21.2.117:0/2585957917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0079000 0x7fb8c00e4da0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb8a8006700 tx=0x7fb8a8009b80).stop 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 -- 172.21.2.117:0/2585957917 shutdown_connections 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 --2- 172.21.2.117:0/2585957917 >> v2:172.21.2.117:3300/0 conn(0x7fb8c01b00f0 0x7fb8c01b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 --2- 172.21.2.117:0/2585957917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0079000 0x7fb8c00e4da0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 -- 172.21.2.117:0/2585957917 >> 172.21.2.117:0/2585957917 conn(0x7fb8c00ce450 msgr2=0x7fb8c00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 -- 172.21.2.117:0/2585957917 shutdown_connections 2021-02-10T11:38:44.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.779 7fb8c649d700 1 -- 172.21.2.117:0/2585957917 wait complete. 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.794 7fb8c649d700 1 Processor -- start 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.794 7fb8c649d700 1 -- start start 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8c0187e20 0x7fb8c0188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0090420 0x7fb8c0090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01067f0 con 0x7fb8c01a0250 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0107350 con 0x7fb8c0090420 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c649d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c01042d0 con 0x7fb8c0187e20 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8bffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8bffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46152/0 (socket says 172.21.2.117:46152) 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8bffff700 1 -- 172.21.2.117:0/1077024834 learned_addr learned my addr 172.21.2.117:0/1077024834 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8c4aba700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0090420 0x7fb8c0090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.795 7fb8b7fff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8c0187e20 0x7fb8c0188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 -- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8c0187e20 msgr2=0x7fb8c0188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8c0187e20 0x7fb8c0188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 -- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0090420 msgr2=0x7fb8c0090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0090420 0x7fb8c0090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 -- 172.21.2.117:0/1077024834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0090e40 con 0x7fb8c01a0250 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bffff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fb8b000a5d0 tx=0x7fb8b000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b0000c10 con 0x7fb8c01a0250 2021-02-10T11:38:44.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b00070d0 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c0090e40 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c00913d0 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.796 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8b00047d0 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.797 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8b0000c10 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.797 7fb8bd7fa700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 0x7fb8a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.797 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb8b00326c0 con 0x7fb8c01a0250 2021-02-10T11:38:44.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.797 7fb8b7fff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 0x7fb8a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:44.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.798 7fb8b7fff700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 0x7fb8a401abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:44.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.825 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c0103280 con 0x7fb8c01a0250 2021-02-10T11:38:44.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.828 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8b0006fd0 con 0x7fb8c01a0250 2021-02-10T11:38:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.958 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802454", "target": ["mgr", ""]}) v1 -- 0x7fb8c0091c60 con 0x7fb8a40186c0 2021-02-10T11:38:44.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.969 7fb8bd7fa700 1 -- 172.21.2.117:0/1077024834 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb8a8009100 con 0x7fb8a40186c0 2021-02-10T11:38:44.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 msgr2=0x7fb8a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 0x7fb8a401abc0 crc :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 msgr2=0x7fb8c0106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:44.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7fb8b000a5d0 tx=0x7fb8b000a970).stop 2021-02-10T11:38:44.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 shutdown_connections 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8c0090420 0x7fb8c0090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8c0187e20 0x7fb8c0188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8a40186c0 0x7fb8a401abc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 --2- 172.21.2.117:0/1077024834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8c01a0250 0x7fb8c0106cf0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 >> 172.21.2.117:0/1077024834 conn(0x7fb8c00a1850 msgr2=0x7fb8c00c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 shutdown_connections 2021-02-10T11:38:44.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:44.975 7fb8c649d700 1 -- 172.21.2.117:0/1077024834 wait complete. 2021-02-10T11:38:44.966 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802455 2021-02-10T11:38:45.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.396 7fae6ac0f700 1 Processor -- start 2021-02-10T11:38:45.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.396 7fae6ac0f700 1 -- start start 2021-02-10T11:38:45.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6ac0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae6407da20 0x7fae640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6ac0f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fae640e51a0 0x7fae640d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6ac0f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae64077a70 con 0x7fae640e4da0 2021-02-10T11:38:45.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6ac0f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640e5c20 con 0x7fae640e51a0 2021-02-10T11:38:45.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6ac0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640777d0 con 0x7fae6407da20 2021-02-10T11:38:45.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae63fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fae640e4da0 0x7fae641b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46412/0 (socket says 172.21.2.117:46412) 2021-02-10T11:38:45.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae63fff700 1 -- 172.21.2.117:0/3067345789 learned_addr learned my addr 172.21.2.117:0/3067345789 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:45.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae6922c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fae640e51a0 0x7fae640d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae68a2b700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae6407da20 0x7fae640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.397 7fae637fe700 1 -- 172.21.2.117:0/3067345789 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427319577 0 0) 0x7fae540009b0 con 0x7fae640e4da0 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae637fe700 1 -- 172.21.2.117:0/3067345789 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae50002fe0 con 0x7fae640e4da0 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6922c700 1 -- 172.21.2.117:0/3067345789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae6407da20 msgr2=0x7fae640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6922c700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae6407da20 0x7fae640771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6922c700 1 -- 172.21.2.117:0/3067345789 >> v1:172.21.2.117:6789/0 conn(0x7fae640e4da0 legacy=0x7fae641b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6922c700 1 -- 172.21.2.117:0/3067345789 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641b08e0 con 0x7fae640e51a0 2021-02-10T11:38:45.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6922c700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640e51a0 0x7fae640d0520 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fae5c006e30 tx=0x7fae5c009e60).ready entity=mon.0 client_cookie=39487ea9a7c65933 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae637fe700 1 -- 172.21.2.117:0/3067345789 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5c006a50 con 0x7fae640e51a0 2021-02-10T11:38:45.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae637fe700 1 -- 172.21.2.117:0/3067345789 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c00b300 con 0x7fae640e51a0 2021-02-10T11:38:45.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae637fe700 1 -- 172.21.2.117:0/3067345789 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae5c00b7e0 con 0x7fae640e51a0 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 -- 172.21.2.117:0/3067345789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640e51a0 msgr2=0x7fae640d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640e51a0 0x7fae640d0520 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fae5c006e30 tx=0x7fae5c009e60).stop 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 -- 172.21.2.117:0/3067345789 shutdown_connections 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640e51a0 0x7fae640d0520 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 --2- 172.21.2.117:0/3067345789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae6407da20 0x7fae640771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 -- 172.21.2.117:0/3067345789 >> 172.21.2.117:0/3067345789 conn(0x7fae640ce450 msgr2=0x7fae640e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 -- 172.21.2.117:0/3067345789 shutdown_connections 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.399 7fae6ac0f700 1 -- 172.21.2.117:0/3067345789 wait complete. 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.413 7fae6ac0f700 1 Processor -- start 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.413 7fae6ac0f700 1 -- start start 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae640c2340 0x7fae640c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae640b7390 0x7fae640b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b8010 con 0x7fae640a41c0 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b8550 con 0x7fae640b7390 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b8770 con 0x7fae640c2340 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46166/0 (socket says 172.21.2.117:46166) 2021-02-10T11:38:45.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 -- 172.21.2.117:0/3537531514 learned_addr learned my addr 172.21.2.117:0/3537531514 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae63fff700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae640c2340 0x7fae640c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6922c700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae640b7390 0x7fae640b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 -- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae640c2340 msgr2=0x7fae640c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae640c2340 0x7fae640c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 -- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae640b7390 msgr2=0x7fae640b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae640b7390 0x7fae640b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 -- 172.21.2.117:0/3537531514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640b8990 con 0x7fae640a41c0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae68a2b700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fae4c006f60 tx=0x7fae4c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4c0009b0 con 0x7fae640a41c0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c00bd60 con 0x7fae640a41c0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae640b8990 con 0x7fae640a41c0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae640b8f20 con 0x7fae640a41c0 2021-02-10T11:38:45.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.415 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4c002030 con 0x7fae640a41c0 2021-02-10T11:38:45.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.416 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae4c000c90 con 0x7fae640a41c0 2021-02-10T11:38:45.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.416 7fae617fa700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 0x7fae4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:45.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.416 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fae4c032910 con 0x7fae640a41c0 2021-02-10T11:38:45.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.416 7fae63fff700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 0x7fae4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:45.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.416 7fae63fff700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 0x7fae4801abc0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:45.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.447 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae641b4f00 con 0x7fae640a41c0 2021-02-10T11:38:45.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.449 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae4c000960 con 0x7fae640a41c0 2021-02-10T11:38:45.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.573 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802455", "target": ["mgr", ""]}) v1 -- 0x7fae64077a70 con 0x7fae480186c0 2021-02-10T11:38:45.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.586 7fae617fa700 1 -- 172.21.2.117:0/3537531514 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fae540009b0 con 0x7fae480186c0 2021-02-10T11:38:45.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 msgr2=0x7fae4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 0x7fae4801abc0 crc :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 msgr2=0x7fae640fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fae4c006f60 tx=0x7fae4c00b970).stop 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 shutdown_connections 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae640b7390 0x7fae640b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae640c2340 0x7fae640c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae480186c0 0x7fae4801abc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 --2- 172.21.2.117:0/3537531514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae640a41c0 0x7fae640fe0d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 >> 172.21.2.117:0/3537531514 conn(0x7fae6408bbc0 msgr2=0x7fae640c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 shutdown_connections 2021-02-10T11:38:45.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:45.592 7fae6ac0f700 1 -- 172.21.2.117:0/3537531514 wait complete. 2021-02-10T11:38:45.582 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802456 2021-02-10T11:38:46.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.014 7ff25dc44700 1 Processor -- start 2021-02-10T11:38:46.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.014 7ff25dc44700 1 -- start start 2021-02-10T11:38:46.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.014 7ff25dc44700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 0x7ff2580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.014 7ff25dc44700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2580d1d00 con 0x7ff2580e5180 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff25dc44700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff258077a70 con 0x7ff2580e4d80 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff25dc44700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580777d0 con 0x7ff25807da20 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff257fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff2580e5180 0x7ff2580d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46424/0 (socket says 172.21.2.117:46424) 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff257fff700 1 -- 172.21.2.117:0/1452455091 learned_addr learned my addr 172.21.2.117:0/1452455091 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2577fe700 1 --2- 172.21.2.117:0/1452455091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 0x7ff2580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915071384 0 0) 0x7ff250000d30 con 0x7ff2580e5180 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff244002fe0 con 0x7ff2580e5180 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054838600 0 0) 0x7ff248000ff0 con 0x7ff2580e4d80 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff244003210 con 0x7ff2580e4d80 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 296942921 0 0) 0x7ff250001270 con 0x7ff2580e5180 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2577fe700 1 -- 172.21.2.117:0/1452455091 >> v1:172.21.2.125:6789/0 conn(0x7ff2580e4d80 legacy=0x7ff2581b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.015 7ff2577fe700 1 -- 172.21.2.117:0/1452455091 >> v1:172.21.2.117:6789/0 conn(0x7ff2580e5180 legacy=0x7ff2580d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff2577fe700 1 -- 172.21.2.117:0/1452455091 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2581b08c0 con 0x7ff25807da20 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff2577fe700 1 --2- 172.21.2.117:0/1452455091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 0x7ff2580771b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7ff240006650 tx=0x7ff240009a80).ready entity=mon.2 client_cookie=39baae4bb728047c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff24000c0c0 con 0x7ff25807da20 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24000c340 con 0x7ff25807da20 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff2567fc700 1 -- 172.21.2.117:0/1452455091 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff24000c860 con 0x7ff25807da20 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff25dc44700 1 -- 172.21.2.117:0/1452455091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 msgr2=0x7ff2580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.016 7ff25dc44700 1 --2- 172.21.2.117:0/1452455091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 0x7ff2580771b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7ff240006650 tx=0x7ff240009a80).stop 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.017 7ff25dc44700 1 -- 172.21.2.117:0/1452455091 shutdown_connections 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.017 7ff25dc44700 1 --2- 172.21.2.117:0/1452455091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff25807da20 0x7ff2580771b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.017 7ff25dc44700 1 -- 172.21.2.117:0/1452455091 >> 172.21.2.117:0/1452455091 conn(0x7ff2580ce430 msgr2=0x7ff2580e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.017 7ff25dc44700 1 -- 172.21.2.117:0/1452455091 shutdown_connections 2021-02-10T11:38:46.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.017 7ff25dc44700 1 -- 172.21.2.117:0/1452455091 wait complete. 2021-02-10T11:38:46.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.032 7ff25dc44700 1 Processor -- start 2021-02-10T11:38:46.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.032 7ff25dc44700 1 -- start start 2021-02-10T11:38:46.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2580c1620 0x7ff25818a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2580c3c40 0x7ff2580b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b8480 con 0x7ff2580b7ef0 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b9020 con 0x7ff2580c3c40 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2580b9200 con 0x7ff2580c1620 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46178/0 (socket says 172.21.2.117:46178) 2021-02-10T11:38:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 -- 172.21.2.117:0/972419433 learned_addr learned my addr 172.21.2.117:0/972419433 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff256ffd700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2580c3c40 0x7ff2580b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 -- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2580c1620 msgr2=0x7ff25818a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2580c1620 0x7ff25818a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 -- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2580c3c40 msgr2=0x7ff2580b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2580c3c40 0x7ff2580b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 -- 172.21.2.117:0/972419433 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580b9420 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff257fff700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7ff250007f70 tx=0x7ff25000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff250007020 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2500072e0 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 -- 172.21.2.117:0/972419433 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2580b9420 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff25dc44700 1 -- 172.21.2.117:0/972419433 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2580b99b0 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.033 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff250004720 con 0x7ff2580b7ef0 2021-02-10T11:38:46.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.034 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff25000ade0 con 0x7ff2580b7ef0 2021-02-10T11:38:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.034 7ff24ffff700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 0x7ff23c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.034 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff250032ca0 con 0x7ff2580b7ef0 2021-02-10T11:38:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.034 7ff2577fe700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 0x7ff23c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.035 7ff2577fe700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 0x7ff23c01ac10 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.064 7ff25dc44700 1 -- 172.21.2.117:0/972419433 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff258077a70 con 0x7ff2580b7ef0 2021-02-10T11:38:46.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.067 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff250000d30 con 0x7ff2580b7ef0 2021-02-10T11:38:46.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.194 7ff25dc44700 1 -- 172.21.2.117:0/972419433 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802456", "target": ["mgr", ""]}) v1 -- 0x7ff2580b99b0 con 0x7ff23c018730 2021-02-10T11:38:46.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.205 7ff24ffff700 1 -- 172.21.2.117:0/972419433 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff240000940 con 0x7ff23c018730 2021-02-10T11:38:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 -- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 msgr2=0x7ff23c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 0x7ff23c01ac10 crc :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 -- 172.21.2.117:0/972419433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 msgr2=0x7ff2580b8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7ff250007f70 tx=0x7ff25000aca0).stop 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 -- 172.21.2.117:0/972419433 shutdown_connections 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff23c018730 0x7ff23c01ac10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2580b7ef0 0x7ff2580b8a00 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2580c3c40 0x7ff2580b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 --2- 172.21.2.117:0/972419433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2580c1620 0x7ff25818a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:46.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 -- 172.21.2.117:0/972419433 >> 172.21.2.117:0/972419433 conn(0x7ff2580bacb0 msgr2=0x7ff25808e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.212 7ff25dc44700 1 -- 172.21.2.117:0/972419433 shutdown_connections 2021-02-10T11:38:46.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.213 7ff25dc44700 1 -- 172.21.2.117:0/972419433 wait complete. 2021-02-10T11:38:46.202 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802457 2021-02-10T11:38:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.634 7ff5a1d10700 1 Processor -- start 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.634 7ff5a1d10700 1 -- start start 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff5a1d10700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff59c07da20 0x7ff59c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff5a1d10700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff59c0e4da0 0x7ff59c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff5a1d10700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c077a70 con 0x7ff59c0e51a0 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff5a1d10700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c0777d0 con 0x7ff59c07da20 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff5a1d10700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c07de20 con 0x7ff59c0e4da0 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff59b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff59c07da20 0x7ff59c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff59bfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7ff59c0e51a0 0x7ff59c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39364/0 (socket says 172.21.2.117:39364) 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff59affd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff59c0e4da0 0x7ff59c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff59b7fe700 1 --2- 172.21.2.117:0/3900696989 >> v2:172.21.2.117:3300/0 conn(0x7ff59c07da20 0x7ff59c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46186/0 (socket says 172.21.2.117:46186) 2021-02-10T11:38:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.635 7ff59bfff700 1 -- 172.21.2.117:0/3900696989 learned_addr learned my addr 172.21.2.117:0/3900696989 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59b7fe700 1 -- 172.21.2.117:0/3900696989 >> v2:172.21.2.125:3300/0 conn(0x7ff59c0e4da0 msgr2=0x7ff59c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59b7fe700 1 --2- 172.21.2.117:0/3900696989 >> v2:172.21.2.125:3300/0 conn(0x7ff59c0e4da0 0x7ff59c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59b7fe700 1 -- 172.21.2.117:0/3900696989 >> v1:172.21.2.125:6789/0 conn(0x7ff59c0e51a0 legacy=0x7ff59c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59b7fe700 1 -- 172.21.2.117:0/3900696989 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c1b0710 con 0x7ff59c07da20 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59b7fe700 1 --2- 172.21.2.117:0/3900696989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c07da20 0x7ff59c0771b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff58c006650 tx=0x7ff58c009a80).ready entity=mon.0 client_cookie=21197778197f95f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59a7fc700 1 -- 172.21.2.117:0/3900696989 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58c00c0c0 con 0x7ff59c07da20 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59a7fc700 1 -- 172.21.2.117:0/3900696989 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c00c440 con 0x7ff59c07da20 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff59a7fc700 1 -- 172.21.2.117:0/3900696989 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff58c00c960 con 0x7ff59c07da20 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 -- 172.21.2.117:0/3900696989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c07da20 msgr2=0x7ff59c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 --2- 172.21.2.117:0/3900696989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c07da20 0x7ff59c0771b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff58c006650 tx=0x7ff58c009a80).stop 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 -- 172.21.2.117:0/3900696989 shutdown_connections 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 --2- 172.21.2.117:0/3900696989 >> v2:172.21.2.125:3300/0 conn(0x7ff59c0e4da0 0x7ff59c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 --2- 172.21.2.117:0/3900696989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c07da20 0x7ff59c0771b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.636 7ff5a1d10700 1 -- 172.21.2.117:0/3900696989 >> 172.21.2.117:0/3900696989 conn(0x7ff59c0ce450 msgr2=0x7ff59c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.637 7ff5a1d10700 1 -- 172.21.2.117:0/3900696989 shutdown_connections 2021-02-10T11:38:46.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.637 7ff5a1d10700 1 -- 172.21.2.117:0/3900696989 wait complete. 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.651 7ff5a1d10700 1 Processor -- start 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.651 7ff5a1d10700 1 -- start start 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff59c1a4640 0x7ff59c09ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 0x7ff59c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 0x7ff59c0c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0c1250 con 0x7ff59c0c0610 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c18bb30 con 0x7ff59c1a4640 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff5a1d10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0c1750 con 0x7ff59c09b140 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 0x7ff59c0c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 0x7ff59c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 0x7ff59c0c0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46194/0 (socket says 172.21.2.117:46194) 2021-02-10T11:38:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59bfff700 1 -- 172.21.2.117:0/1866815547 learned_addr learned my addr 172.21.2.117:0/1866815547 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59b7fe700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff59c1a4640 0x7ff59c09ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 -- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff59c1a4640 msgr2=0x7ff59c09ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff59c1a4640 0x7ff59c09ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 -- 172.21.2.117:0/1866815547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 msgr2=0x7ff59c0c0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 0x7ff59c0c0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 -- 172.21.2.117:0/1866815547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0c1970 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.652 7ff59affd700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 0x7ff59c0bfff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff590009540 tx=0x7ff59000b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff59000bcd0 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590006740 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c0c1970 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff5900009b0 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c0c1970 con 0x7ff59c09b140 2021-02-10T11:38:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff5900052e0 con 0x7ff59c09b140 2021-02-10T11:38:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 0x7ff57c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff5900326d0 con 0x7ff59c09b140 2021-02-10T11:38:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.653 7ff59b7fe700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 0x7ff57c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.654 7ff59b7fe700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 0x7ff57c01ac10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:46.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.684 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c18bb30 con 0x7ff59c09b140 2021-02-10T11:38:46.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.687 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff590006ae0 con 0x7ff59c09b140 2021-02-10T11:38:46.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.821 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802457", "target": ["mgr", ""]}) v1 -- 0x7ff59c18bb30 con 0x7ff57c018730 2021-02-10T11:38:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.829 7ff583fff700 1 -- 172.21.2.117:0/1866815547 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff58c0009b0 con 0x7ff57c018730 2021-02-10T11:38:46.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 msgr2=0x7ff57c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 0x7ff57c01ac10 crc :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 msgr2=0x7ff59c0bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:46.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 0x7ff59c0bfff0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff590009540 tx=0x7ff59000b8e0).stop 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 shutdown_connections 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff57c018730 0x7ff57c01ac10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff59c0c0610 0x7ff59c0c0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff59c09b140 0x7ff59c0bfff0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 --2- 172.21.2.117:0/1866815547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff59c1a4640 0x7ff59c09ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 >> 172.21.2.117:0/1866815547 conn(0x7ff59c08a180 msgr2=0x7ff59c0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 shutdown_connections 2021-02-10T11:38:46.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:46.838 7ff5a1d10700 1 -- 172.21.2.117:0/1866815547 wait complete. 2021-02-10T11:38:46.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802458 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 Processor -- start 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 -- start start 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd50c0d1530 0x7fd50c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c077a70 con 0x7fd50c07da20 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50c079400 con 0x7fd50c079000 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd51181b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50c07de20 con 0x7fd50c0d1530 2021-02-10T11:38:47.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd50b574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd50c07da20 0x7fd50c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46450/0 (socket says 172.21.2.117:46450) 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.261 7fd50b574700 1 -- 172.21.2.117:0/3795035403 learned_addr learned my addr 172.21.2.117:0/3795035403 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50bd75700 1 --2- 172.21.2.117:0/3795035403 >> v2:172.21.2.125:3300/0 conn(0x7fd50c0d1530 0x7fd50c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50a572700 1 -- 172.21.2.117:0/3795035403 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793304353 0 0) 0x7fd4fc000ce0 con 0x7fd50c07da20 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50a572700 1 -- 172.21.2.117:0/3795035403 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8002fe0 con 0x7fd50c07da20 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50a572700 1 -- 172.21.2.117:0/3795035403 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181303121 0 0) 0x7fd4f4001080 con 0x7fd50c079000 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50a572700 1 -- 172.21.2.117:0/3795035403 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8003210 con 0x7fd50c079000 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50bd75700 1 -- 172.21.2.117:0/3795035403 >> v1:172.21.2.125:6789/0 conn(0x7fd50c079000 legacy=0x7fd50c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50bd75700 1 -- 172.21.2.117:0/3795035403 >> v1:172.21.2.117:6789/0 conn(0x7fd50c07da20 legacy=0x7fd50c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50bd75700 1 -- 172.21.2.117:0/3795035403 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c1b00f0 con 0x7fd50c0d1530 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50bd75700 1 --2- 172.21.2.117:0/3795035403 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0d1530 0x7fd50c0d1930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd504006e50 tx=0x7fd504009df0).ready entity=mon.1 client_cookie=6fad4d3a769efd35 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.262 7fd50a572700 1 -- 172.21.2.117:0/3795035403 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd50400b510 con 0x7fd50c0d1530 2021-02-10T11:38:47.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd50a572700 1 -- 172.21.2.117:0/3795035403 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504006790 con 0x7fd50c0d1530 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd50a572700 1 -- 172.21.2.117:0/3795035403 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd504000f90 con 0x7fd50c0d1530 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 -- 172.21.2.117:0/3795035403 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0d1530 msgr2=0x7fd50c0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 --2- 172.21.2.117:0/3795035403 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0d1530 0x7fd50c0d1930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd504006e50 tx=0x7fd504009df0).stop 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 -- 172.21.2.117:0/3795035403 shutdown_connections 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 --2- 172.21.2.117:0/3795035403 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0d1530 0x7fd50c0d1930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 -- 172.21.2.117:0/3795035403 >> 172.21.2.117:0/3795035403 conn(0x7fd50c0ce450 msgr2=0x7fd50c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 -- 172.21.2.117:0/3795035403 shutdown_connections 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.263 7fd51181b700 1 -- 172.21.2.117:0/3795035403 wait complete. 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 Processor -- start 2021-02-10T11:38:47.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 -- start start 2021-02-10T11:38:47.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd50c0c2340 0x7fd50c0c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 0x7fd50c0b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c0b8010 con 0x7fd50c0a41c0 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c0b8550 con 0x7fd50c0b7390 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.278 7fd51181b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c0b8770 con 0x7fd50c0c2340 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46210/0 (socket says 172.21.2.117:46210) 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 -- 172.21.2.117:0/2964666012 learned_addr learned my addr 172.21.2.117:0/2964666012 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50ad73700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd50c0c2340 0x7fd50c0c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50bd75700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 0x7fd50c0b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 -- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd50c0c2340 msgr2=0x7fd50c0c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd50c0c2340 0x7fd50c0c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 -- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 msgr2=0x7fd50c0b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 0x7fd50c0b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 -- 172.21.2.117:0/2964666012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0b8990 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50bd75700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 0x7fd50c0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.279 7fd50b574700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7fd4fc006f00 tx=0x7fd4fc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc00bd30 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc006030 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4fc004620 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd51181b700 1 -- 172.21.2.117:0/2964666012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c0b8990 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd51181b700 1 -- 172.21.2.117:0/2964666012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c0b8f20 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4fc0043a0 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.280 7fd503fff700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 0x7fd4f001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.281 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd4fc032740 con 0x7fd50c0a41c0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.281 7fd50ad73700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 0x7fd4f001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.281 7fd50ad73700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 0x7fd4f001ac10 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.310 7fd51181b700 1 -- 172.21.2.117:0/2964666012 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c1b4f00 con 0x7fd50c0a41c0 2021-02-10T11:38:47.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.313 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4fc006030 con 0x7fd50c0a41c0 2021-02-10T11:38:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.441 7fd51181b700 1 -- 172.21.2.117:0/2964666012 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802458", "target": ["mgr", ""]}) v1 -- 0x7fd50c077a70 con 0x7fd4f0018730 2021-02-10T11:38:47.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.451 7fd503fff700 1 -- 172.21.2.117:0/2964666012 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd4f4006550 con 0x7fd4f0018730 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 msgr2=0x7fd4f001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 0x7fd4f001ac10 crc :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 msgr2=0x7fd50c0fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7fd4fc006f00 tx=0x7fd4fc00b940).stop 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 shutdown_connections 2021-02-10T11:38:47.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd50c0b7390 0x7fd50c0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd50c0c2340 0x7fd50c0c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4f0018730 0x7fd4f001ac10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 --2- 172.21.2.117:0/2964666012 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd50c0a41c0 0x7fd50c0fe0d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 >> 172.21.2.117:0/2964666012 conn(0x7fd50c08bbc0 msgr2=0x7fd50c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 shutdown_connections 2021-02-10T11:38:47.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.458 7fd51181b700 1 -- 172.21.2.117:0/2964666012 wait complete. 2021-02-10T11:38:47.447 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802459 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 Processor -- start 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 -- start start 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 0x7fead80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead80d1d20 con 0x7fead81b00f0 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead8077a70 con 0x7fead807da20 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.879 7feadc84b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead80e5430 con 0x7fead8079000 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead6e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fead81b00f0 0x7fead80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46468/0 (socket says 172.21.2.117:46468) 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead6e1d700 1 -- 172.21.2.117:0/2806056665 learned_addr learned my addr 172.21.2.117:0/2806056665 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead5e1b700 1 --2- 172.21.2.117:0/2806056665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 0x7fead80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190296255 0 0) 0x7fead0001500 con 0x7fead81b00f0 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4002fe0 con 0x7fead81b00f0 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377290238 0 0) 0x7feac8000f30 con 0x7fead807da20 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4002fe0 con 0x7fead807da20 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead5e1b700 1 -- 172.21.2.117:0/2806056665 >> v1:172.21.2.125:6789/0 conn(0x7fead807da20 legacy=0x7fead80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead5e1b700 1 -- 172.21.2.117:0/2806056665 >> v1:172.21.2.117:6789/0 conn(0x7fead81b00f0 legacy=0x7fead80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead5e1b700 1 -- 172.21.2.117:0/2806056665 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8079400 con 0x7fead8079000 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead5e1b700 1 --2- 172.21.2.117:0/2806056665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 0x7fead80e4da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7feac0006650 tx=0x7feac0009a00).ready entity=mon.2 client_cookie=bc3ab40f731ec93f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac00009b0 con 0x7fead8079000 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac000c0c0 con 0x7fead8079000 2021-02-10T11:38:47.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7fead561a700 1 -- 172.21.2.117:0/2806056665 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac000c510 con 0x7fead8079000 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7feadc84b700 1 -- 172.21.2.117:0/2806056665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 msgr2=0x7fead80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.880 7feadc84b700 1 --2- 172.21.2.117:0/2806056665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 0x7fead80e4da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7feac0006650 tx=0x7feac0009a00).stop 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.881 7feadc84b700 1 -- 172.21.2.117:0/2806056665 shutdown_connections 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.881 7feadc84b700 1 --2- 172.21.2.117:0/2806056665 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead8079000 0x7fead80e4da0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.881 7feadc84b700 1 -- 172.21.2.117:0/2806056665 >> 172.21.2.117:0/2806056665 conn(0x7fead80ce450 msgr2=0x7fead80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.881 7feadc84b700 1 -- 172.21.2.117:0/2806056665 shutdown_connections 2021-02-10T11:38:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.881 7feadc84b700 1 -- 172.21.2.117:0/2806056665 wait complete. 2021-02-10T11:38:47.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.896 7feadc84b700 1 Processor -- start 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.896 7feadc84b700 1 -- start start 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 0x7fead81b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 0x7fead80b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 0x7fead80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead80b8da0 con 0x7fead81b5f50 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead80a4140 con 0x7fead81b5b50 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7feadc84b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead80a4320 con 0x7fead81b5130 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead6e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 0x7fead80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead6e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 0x7fead80b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46220/0 (socket says 172.21.2.117:46220) 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead6e1d700 1 -- 172.21.2.117:0/832325057 learned_addr learned my addr 172.21.2.117:0/832325057 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 0x7fead81b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead5e1b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 0x7fead80b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 -- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 msgr2=0x7fead80b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 0x7fead80b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 -- 172.21.2.117:0/832325057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 msgr2=0x7fead80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 0x7fead80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 -- 172.21.2.117:0/832325057 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead80b62b0 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead5e1b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 0x7fead80b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.897 7fead661c700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 0x7fead81b5530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7feac800b510 tx=0x7feac800b8b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac800bca0 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8000c90 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feadc84b700 1 -- 172.21.2.117:0/832325057 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead80b62b0 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feac8005470 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feadc84b700 1 -- 172.21.2.117:0/832325057 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead80b6840 con 0x7fead81b5130 2021-02-10T11:38:47.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feac8006030 con 0x7fead81b5130 2021-02-10T11:38:47.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 0x7feabc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:47.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.898 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feac8032720 con 0x7fead81b5130 2021-02-10T11:38:47.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.899 7fead5e1b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 0x7feabc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:47.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.899 7fead5e1b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 0x7feabc01abd0 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:47.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.929 7feadc84b700 1 -- 172.21.2.117:0/832325057 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead81b4f00 con 0x7fead81b5130 2021-02-10T11:38:47.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:47.932 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feac8006240 con 0x7fead81b5130 2021-02-10T11:38:48.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.063 7feadc84b700 1 -- 172.21.2.117:0/832325057 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802459", "target": ["mgr", ""]}) v1 -- 0x7fead80b8da0 con 0x7feabc0186d0 2021-02-10T11:38:48.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.073 7feaceffd700 1 -- 172.21.2.117:0/832325057 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7feac00009b0 con 0x7feabc0186d0 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 -- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 msgr2=0x7feabc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 0x7feabc01abd0 crc :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 -- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 msgr2=0x7fead81b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 0x7fead81b5530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7feac800b510 tx=0x7feac800b8b0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 -- 172.21.2.117:0/832325057 shutdown_connections 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feabc0186d0 0x7feabc01abd0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fead81b5f50 0x7fead80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fead81b5b50 0x7fead80b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 --2- 172.21.2.117:0/832325057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fead81b5130 0x7fead81b5530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.079 7feadc84b700 1 -- 172.21.2.117:0/832325057 >> 172.21.2.117:0/832325057 conn(0x7fead808bbc0 msgr2=0x7fead80c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.080 7feadc84b700 1 -- 172.21.2.117:0/832325057 shutdown_connections 2021-02-10T11:38:48.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.080 7feadc84b700 1 -- 172.21.2.117:0/832325057 wait complete. 2021-02-10T11:38:48.069 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802460 2021-02-10T11:38:48.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 Processor -- start 2021-02-10T11:38:48.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 -- start start 2021-02-10T11:38:48.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6bbc079000 0x7f6bbc0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc077a70 con 0x7f6bbc07da20 2021-02-10T11:38:48.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc0e5c20 con 0x7f6bbc1b00f0 2021-02-10T11:38:48.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc29d9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc0e53c0 con 0x7f6bbc079000 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc0ff6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc0ff6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46230/0 (socket says 172.21.2.117:46230) 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc0ff6700 1 -- 172.21.2.117:0/1960674939 learned_addr learned my addr 172.21.2.117:0/1960674939 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bb3fff700 1 --2- 172.21.2.117:0/1960674939 >> v2:172.21.2.125:3300/0 conn(0x7f6bbc079000 0x7f6bbc0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc0ff6700 1 -- 172.21.2.117:0/1960674939 >> v2:172.21.2.125:3300/0 conn(0x7f6bbc079000 msgr2=0x7f6bbc0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:48.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.501 7f6bc0ff6700 1 --2- 172.21.2.117:0/1960674939 >> v2:172.21.2.125:3300/0 conn(0x7f6bbc079000 0x7f6bbc0e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bc0ff6700 1 -- 172.21.2.117:0/1960674939 >> v1:172.21.2.125:6789/0 conn(0x7f6bbc07da20 legacy=0x7f6bbc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bc0ff6700 1 -- 172.21.2.117:0/1960674939 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0d3400 con 0x7f6bbc1b00f0 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bc0ff6700 1 --2- 172.21.2.117:0/1960674939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f6bb4006f00 tx=0x7f6bb4009f10).ready entity=mon.0 client_cookie=459863c9192091a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bbb7fe700 1 -- 172.21.2.117:0/1960674939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bb400b3b0 con 0x7f6bbc1b00f0 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bbb7fe700 1 -- 172.21.2.117:0/1960674939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb400b730 con 0x7f6bbc1b00f0 2021-02-10T11:38:48.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bbb7fe700 1 -- 172.21.2.117:0/1960674939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bb400bc10 con 0x7f6bbc1b00f0 2021-02-10T11:38:48.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bc29d9700 1 -- 172.21.2.117:0/1960674939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b00f0 msgr2=0x7f6bbc1b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.502 7f6bc29d9700 1 --2- 172.21.2.117:0/1960674939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f6bb4006f00 tx=0x7f6bb4009f10).stop 2021-02-10T11:38:48.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 -- 172.21.2.117:0/1960674939 shutdown_connections 2021-02-10T11:38:48.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 --2- 172.21.2.117:0/1960674939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b00f0 0x7f6bbc1b04f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 --2- 172.21.2.117:0/1960674939 >> v2:172.21.2.125:3300/0 conn(0x7f6bbc079000 0x7f6bbc0e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 -- 172.21.2.117:0/1960674939 >> 172.21.2.117:0/1960674939 conn(0x7f6bbc0ce450 msgr2=0x7f6bbc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 -- 172.21.2.117:0/1960674939 shutdown_connections 2021-02-10T11:38:48.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.503 7f6bc29d9700 1 -- 172.21.2.117:0/1960674939 wait complete. 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.517 7f6bc29d9700 1 Processor -- start 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 -- start start 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 0x7f6bbc0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6bbc1b5f50 0x7f6bbc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0b8da0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0a4140 con 0x7f6bbc1b5b50 2021-02-10T11:38:48.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc29d9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0a4320 con 0x7f6bbc1b5f50 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bbbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bbbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46234/0 (socket says 172.21.2.117:46234) 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bbbfff700 1 -- 172.21.2.117:0/548815390 learned_addr learned my addr 172.21.2.117:0/548815390 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bc0ff6700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6bbc1b5f50 0x7f6bbc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.518 7f6bb3fff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 0x7f6bbc0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 -- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6bbc1b5f50 msgr2=0x7f6bbc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6bbc1b5f50 0x7f6bbc0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 -- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 msgr2=0x7f6bbc0b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 0x7f6bbc0b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 -- 172.21.2.117:0/548815390 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0b62b0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bb3fff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 0x7f6bbc0b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bbbfff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f6ba400b5b0 tx=0x7f6ba400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba400bd40 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4006030 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc0b62b0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ba40054a0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.519 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0b62b0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.520 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ba4006210 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.520 7f6bb97fa700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 0x7f6ba001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.520 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ba40327b0 con 0x7f6bbc1b5130 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.520 7f6bb3fff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 0x7f6ba001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:48.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.520 7f6bb3fff700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 0x7f6ba001ac10 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:48.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.550 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc1b4f00 con 0x7f6bbc1b5130 2021-02-10T11:38:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.553 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ba4000c60 con 0x7f6bbc1b5130 2021-02-10T11:38:48.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.677 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802460", "target": ["mgr", ""]}) v1 -- 0x7f6bbc077a70 con 0x7f6ba0018730 2021-02-10T11:38:48.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.687 7f6bb97fa700 1 -- 172.21.2.117:0/548815390 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6bac0009b0 con 0x7f6ba0018730 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 msgr2=0x7f6ba001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 0x7f6ba001ac10 crc :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 msgr2=0x7f6bbc1b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f6ba400b5b0 tx=0x7f6ba400b950).stop 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 shutdown_connections 2021-02-10T11:38:48.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6bbc1b5f50 0x7f6bbc0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6bbc1b5b50 0x7f6bbc0b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ba0018730 0x7f6ba001ac10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 --2- 172.21.2.117:0/548815390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6bbc1b5130 0x7f6bbc1b5530 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.694 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 >> 172.21.2.117:0/548815390 conn(0x7f6bbc08bbc0 msgr2=0x7f6bbc0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.695 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 shutdown_connections 2021-02-10T11:38:48.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:48.695 7f6bc29d9700 1 -- 172.21.2.117:0/548815390 wait complete. 2021-02-10T11:38:48.684 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802461 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 Processor -- start 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 -- start start 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f007da20 0x7ff3f00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3f00e4da0 0x7ff3f01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff3f00e51a0 0x7ff3f0077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f00e5c20 con 0x7ff3f00e4da0 2021-02-10T11:38:49.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f0077a70 con 0x7ff3f00e51a0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3f55dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00777d0 con 0x7ff3f007da20 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3eeabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3f00e4da0 0x7ff3f01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3eeabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3f00e4da0 0x7ff3f01b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46242/0 (socket says 172.21.2.117:46242) 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3eeabe700 1 -- 172.21.2.117:0/3574306001 learned_addr learned my addr 172.21.2.117:0/3574306001 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.117 7ff3ef2bf700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f007da20 0x7ff3f00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3efac0700 1 --2- 172.21.2.117:0/3574306001 >> v2:172.21.2.125:3300/0 conn(0x7ff3f00e51a0 0x7ff3f0077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 -- 172.21.2.117:0/3574306001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f007da20 msgr2=0x7ff3f00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f007da20 0x7ff3f00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 -- 172.21.2.117:0/3574306001 >> v2:172.21.2.125:3300/0 conn(0x7ff3f00e51a0 msgr2=0x7ff3f0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 --2- 172.21.2.117:0/3574306001 >> v2:172.21.2.125:3300/0 conn(0x7ff3f00e51a0 0x7ff3f0077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 -- 172.21.2.117:0/3574306001 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f007de20 con 0x7ff3f00e4da0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3eeabe700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f00e4da0 0x7ff3f01b00f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7ff3e8006a80 tx=0x7ff3e8009f00).ready entity=mon.0 client_cookie=601aff9098967c18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3ee2bd700 1 -- 172.21.2.117:0/3574306001 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e800b5c0 con 0x7ff3f00e4da0 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3ee2bd700 1 -- 172.21.2.117:0/3574306001 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e800b8c0 con 0x7ff3f00e4da0 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3ee2bd700 1 -- 172.21.2.117:0/3574306001 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e800bd20 con 0x7ff3f00e4da0 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3f55dd700 1 -- 172.21.2.117:0/3574306001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f00e4da0 msgr2=0x7ff3f01b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.118 7ff3f55dd700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f00e4da0 0x7ff3f01b00f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7ff3e8006a80 tx=0x7ff3e8009f00).stop 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 -- 172.21.2.117:0/3574306001 shutdown_connections 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 --2- 172.21.2.117:0/3574306001 >> v2:172.21.2.125:3300/0 conn(0x7ff3f00e51a0 0x7ff3f0077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f00e4da0 0x7ff3f01b00f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 --2- 172.21.2.117:0/3574306001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f007da20 0x7ff3f00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 -- 172.21.2.117:0/3574306001 >> 172.21.2.117:0/3574306001 conn(0x7ff3f00ce450 msgr2=0x7ff3f00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 -- 172.21.2.117:0/3574306001 shutdown_connections 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.119 7ff3f55dd700 1 -- 172.21.2.117:0/3574306001 wait complete. 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 Processor -- start 2021-02-10T11:38:49.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 -- start start 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 0x7ff3f0188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 0x7ff3f0090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f01067f0 con 0x7ff3f01a0250 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0107350 con 0x7ff3f0090420 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.134 7ff3f55dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f01042d0 con 0x7ff3f0187e20 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3ef2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3ef2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46248/0 (socket says 172.21.2.117:46248) 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3ef2bf700 1 -- 172.21.2.117:0/2668097670 learned_addr learned my addr 172.21.2.117:0/2668097670 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 0x7ff3f0188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3efac0700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 0x7ff3f0090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 -- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 msgr2=0x7ff3f0090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 0x7ff3f0090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 -- 172.21.2.117:0/2668097670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 msgr2=0x7ff3f0106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 -- 172.21.2.117:0/2668097670 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0090e40 con 0x7ff3f0187e20 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3ef2bf700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3efac0700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 0x7ff3f0090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.135 7ff3eeabe700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 0x7ff3f0188220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff3e80090e0 tx=0x7ff3e800bc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.136 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e8009700 con 0x7ff3f0187e20 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.136 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e800c2f0 con 0x7ff3f0187e20 2021-02-10T11:38:49.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.136 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e800c7d0 con 0x7ff3f0187e20 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.136 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f0090e40 con 0x7ff3f0187e20 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.136 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f00913d0 con 0x7ff3f0187e20 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.137 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3e8001010 con 0x7ff3f0187e20 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.137 7ff3d7fff700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.137 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff3e8032830 con 0x7ff3f0187e20 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.137 7ff3ef2bf700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.137 7ff3ef2bf700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 0x7ff3d001abc0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.166 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f0091c20 con 0x7ff3f0187e20 2021-02-10T11:38:49.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.170 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3e8000d30 con 0x7ff3f0187e20 2021-02-10T11:38:49.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.299 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802461", "target": ["mgr", ""]}) v1 -- 0x7ff3f00913d0 con 0x7ff3d00186c0 2021-02-10T11:38:49.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.310 7ff3d7fff700 1 -- 172.21.2.117:0/2668097670 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff3e4006400 con 0x7ff3d00186c0 2021-02-10T11:38:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.316 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 msgr2=0x7ff3d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.316 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 0x7ff3d001abc0 crc :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.316 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 msgr2=0x7ff3f0188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.316 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 0x7ff3f0188220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff3e80090e0 tx=0x7ff3e800bc90).stop 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 shutdown_connections 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3f0090420 0x7ff3f0090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3f0187e20 0x7ff3f0188220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3d00186c0 0x7ff3d001abc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 --2- 172.21.2.117:0/2668097670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3f01a0250 0x7ff3f0106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 >> 172.21.2.117:0/2668097670 conn(0x7ff3f00a1850 msgr2=0x7ff3f00c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 shutdown_connections 2021-02-10T11:38:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.317 7ff3f55dd700 1 -- 172.21.2.117:0/2668097670 wait complete. 2021-02-10T11:38:49.307 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802462 2021-02-10T11:38:49.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.739 7f66ca019700 1 Processor -- start 2021-02-10T11:38:49.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 -- start start 2021-02-10T11:38:49.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c407da20 0x7f66c40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66c40e51a0 0x7f66c40d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c4077a70 con 0x7f66c40e4da0 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c40e5c20 con 0x7f66c40e51a0 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66ca019700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40777d0 con 0x7f66c407da20 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66c3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66c40e51a0 0x7f66c40d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66c2ffd700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f66c40e4da0 0x7f66c41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39428/0 (socket says 172.21.2.117:39428) 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66c3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66c40e51a0 0x7f66c40d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46252/0 (socket says 172.21.2.117:46252) 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66c37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c407da20 0x7f66c40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.740 7f66c2ffd700 1 -- 172.21.2.117:0/1780109416 learned_addr learned my addr 172.21.2.117:0/1780109416 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c3fff700 1 -- 172.21.2.117:0/1780109416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c407da20 msgr2=0x7f66c40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c3fff700 1 --2- 172.21.2.117:0/1780109416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c407da20 0x7f66c40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c3fff700 1 -- 172.21.2.117:0/1780109416 >> v1:172.21.2.125:6789/0 conn(0x7f66c40e4da0 legacy=0x7f66c41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c3fff700 1 -- 172.21.2.117:0/1780109416 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c41b08e0 con 0x7f66c40e51a0 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c3fff700 1 --2- 172.21.2.117:0/1780109416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c40e51a0 0x7f66c40d0520 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f66bc006e30 tx=0x7f66bc009e60).ready entity=mon.0 client_cookie=f6a9a4651c2d0358 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c27fc700 1 -- 172.21.2.117:0/1780109416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc006a50 con 0x7f66c40e51a0 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c27fc700 1 -- 172.21.2.117:0/1780109416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc00b300 con 0x7f66c40e51a0 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66c27fc700 1 -- 172.21.2.117:0/1780109416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc00b820 con 0x7f66c40e51a0 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66ca019700 1 -- 172.21.2.117:0/1780109416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c40e51a0 msgr2=0x7f66c40d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.741 7f66ca019700 1 --2- 172.21.2.117:0/1780109416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c40e51a0 0x7f66c40d0520 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f66bc006e30 tx=0x7f66bc009e60).stop 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 -- 172.21.2.117:0/1780109416 shutdown_connections 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 --2- 172.21.2.117:0/1780109416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c40e51a0 0x7f66c40d0520 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 --2- 172.21.2.117:0/1780109416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c407da20 0x7f66c40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 -- 172.21.2.117:0/1780109416 >> 172.21.2.117:0/1780109416 conn(0x7f66c40ce450 msgr2=0x7f66c40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 -- 172.21.2.117:0/1780109416 shutdown_connections 2021-02-10T11:38:49.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.742 7f66ca019700 1 -- 172.21.2.117:0/1780109416 wait complete. 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 Processor -- start 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 -- start start 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66c4187e20 0x7f66c4188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 0x7f66c4090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41067f0 con 0x7f66c4090420 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c4107350 con 0x7f66c4187e20 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66ca019700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41042d0 con 0x7f66c41a0250 2021-02-10T11:38:49.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66c37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66c3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 0x7f66c4090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66c37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43936/0 (socket says 172.21.2.117:43936) 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66c37fe700 1 -- 172.21.2.117:0/1041556636 learned_addr learned my addr 172.21.2.117:0/1041556636 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.757 7f66c2ffd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66c4187e20 0x7f66c4188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 -- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 msgr2=0x7f66c4106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 -- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66c4187e20 msgr2=0x7f66c4188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66c4187e20 0x7f66c4188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 -- 172.21.2.117:0/1041556636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c4090e40 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c37fe700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66c3fff700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 0x7f66c4090820 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f66bc00b8f0 tx=0x7f66bc00bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc0093d0 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66bc005820 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66ca019700 1 -- 172.21.2.117:0/1041556636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c4090e40 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66ca019700 1 -- 172.21.2.117:0/1041556636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c40913d0 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.758 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66bc005d40 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.759 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66bc009380 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.759 7f66bbfff700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 0x7f66a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.759 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f66bc032bf0 con 0x7f66c4090420 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.759 7f66c37fe700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 0x7f66a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.760 7f66c37fe700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 0x7f66a801ac10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:49.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.789 7f66ca019700 1 -- 172.21.2.117:0/1041556636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c4103280 con 0x7f66c4090420 2021-02-10T11:38:49.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.792 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f66bc005820 con 0x7f66c4090420 2021-02-10T11:38:49.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.921 7f66ca019700 1 -- 172.21.2.117:0/1041556636 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802462", "target": ["mgr", ""]}) v1 -- 0x7f66c4091c60 con 0x7f66a8018730 2021-02-10T11:38:49.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.932 7f66bbfff700 1 -- 172.21.2.117:0/1041556636 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f66ac006400 con 0x7f66a8018730 2021-02-10T11:38:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 msgr2=0x7f66a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 0x7f66a801ac10 crc :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 msgr2=0x7f66c4090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:49.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 0x7f66c4090820 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f66bc00b8f0 tx=0x7f66bc00bc90).stop 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 shutdown_connections 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66a8018730 0x7f66a801ac10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66c4090420 0x7f66c4090820 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66c4187e20 0x7f66c4188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 --2- 172.21.2.117:0/1041556636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66c41a0250 0x7f66c4106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 >> 172.21.2.117:0/1041556636 conn(0x7f66c40a1850 msgr2=0x7f66c40c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 shutdown_connections 2021-02-10T11:38:49.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:49.939 7f66ca019700 1 -- 172.21.2.117:0/1041556636 wait complete. 2021-02-10T11:38:49.929 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802463 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 Processor -- start 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 -- start start 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 0x7f698c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 0x7f698c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698c077a70 con 0x7f698c07da20 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c0e53c0 con 0x7f698c1b00f0 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.361 7f6991fe8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c079400 con 0x7f698c079000 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 0x7f698c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 0x7f698c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f698c07da20 0x7f698c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46512/0 (socket says 172.21.2.117:46512) 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698b7fe700 1 -- 172.21.2.117:0/1689670135 learned_addr learned my addr 172.21.2.117:0/1689670135 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 -- 172.21.2.117:0/1689670135 >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 msgr2=0x7f698c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 --2- 172.21.2.117:0/1689670135 >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 0x7f698c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 -- 172.21.2.117:0/1689670135 >> v1:172.21.2.117:6789/0 conn(0x7f698c07da20 legacy=0x7f698c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 -- 172.21.2.117:0/1689670135 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c07de20 con 0x7f698c079000 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698bfff700 1 --2- 172.21.2.117:0/1689670135 >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 0x7f698c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698affd700 1 --2- 172.21.2.117:0/1689670135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 0x7f698c0e4da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f697c006650 tx=0x7f697c009a80).ready entity=mon.2 client_cookie=baf12161edcb9946 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698a7fc700 1 -- 172.21.2.117:0/1689670135 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f697c0009b0 con 0x7f698c079000 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698a7fc700 1 -- 172.21.2.117:0/1689670135 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c00c0c0 con 0x7f698c079000 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f698a7fc700 1 -- 172.21.2.117:0/1689670135 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f697c00c5e0 con 0x7f698c079000 2021-02-10T11:38:50.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.362 7f6991fe8700 1 -- 172.21.2.117:0/1689670135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 msgr2=0x7f698c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 --2- 172.21.2.117:0/1689670135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 0x7f698c0e4da0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f697c006650 tx=0x7f697c009a80).stop 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 -- 172.21.2.117:0/1689670135 shutdown_connections 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 --2- 172.21.2.117:0/1689670135 >> v2:172.21.2.117:3300/0 conn(0x7f698c1b00f0 0x7f698c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 --2- 172.21.2.117:0/1689670135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c079000 0x7f698c0e4da0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 -- 172.21.2.117:0/1689670135 >> 172.21.2.117:0/1689670135 conn(0x7f698c0ce450 msgr2=0x7f698c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 -- 172.21.2.117:0/1689670135 shutdown_connections 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.363 7f6991fe8700 1 -- 172.21.2.117:0/1689670135 wait complete. 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 Processor -- start 2021-02-10T11:38:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 -- start start 2021-02-10T11:38:50.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c08a180 0x7f698c08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 0x7f698c1b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c1b9cb0 con 0x7f698c08afa0 2021-02-10T11:38:50.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c0c4050 con 0x7f698c08aba0 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.378 7f6991fe8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c1b9730 con 0x7f698c08a180 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 0x7f698c1b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c08a180 0x7f698c08a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50654/0 (socket says 172.21.2.117:50654) 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 -- 172.21.2.117:0/534271705 learned_addr learned my addr 172.21.2.117:0/534271705 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 -- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c08a180 msgr2=0x7f698c08a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c08a180 0x7f698c08a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 -- 172.21.2.117:0/534271705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 msgr2=0x7f698c1b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 0x7f698c1b8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 -- 172.21.2.117:0/534271705 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c106e90 con 0x7f698c08aba0 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698bfff700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 0x7f698c1b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f698affd700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f697c009630 tx=0x7f697c00b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.379 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f697c00bd60 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c006740 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f697c002300 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6991fe8700 1 -- 172.21.2.117:0/534271705 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c106e90 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6991fe8700 1 -- 172.21.2.117:0/534271705 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c106e90 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f697c000c90 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6983fff700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 0x7f697001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f697c032790 con 0x7f698c08aba0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.380 7f698b7fe700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 0x7f697001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.381 7f698b7fe700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 0x7f697001ac10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.411 7f6991fe8700 1 -- 172.21.2.117:0/534271705 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c077a70 con 0x7f698c08aba0 2021-02-10T11:38:50.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.414 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f697c01f500 con 0x7f698c08aba0 2021-02-10T11:38:50.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.545 7f6991fe8700 1 -- 172.21.2.117:0/534271705 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802463", "target": ["mgr", ""]}) v1 -- 0x7f698c1b9730 con 0x7f6970018730 2021-02-10T11:38:50.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.555 7f6983fff700 1 -- 172.21.2.117:0/534271705 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6974001a00 con 0x7f6970018730 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 msgr2=0x7f697001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 0x7f697001ac10 crc :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 msgr2=0x7f698c1b8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f697c009630 tx=0x7f697c00b970).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 shutdown_connections 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6970018730 0x7f697001ac10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c08afa0 0x7f698c1b8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08aba0 0x7f698c1b8420 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 --2- 172.21.2.117:0/534271705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c08a180 0x7f698c08a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 >> 172.21.2.117:0/534271705 conn(0x7f698c0a4180 msgr2=0x7f698c09be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:50.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 shutdown_connections 2021-02-10T11:38:50.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.563 7f6991fe8700 1 -- 172.21.2.117:0/534271705 wait complete. 2021-02-10T11:38:50.553 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802464 2021-02-10T11:38:50.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.986 7fc4a23e3700 1 Processor -- start 2021-02-10T11:38:50.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.986 7fc4a23e3700 1 -- start start 2021-02-10T11:38:50.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a23e3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc49c07da20 0x7fc49c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a23e3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c0d1d20 con 0x7fc49c0e51a0 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a23e3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49c077a70 con 0x7fc49c0e4da0 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a23e3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc49c0777d0 con 0x7fc49c07da20 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a0a00700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc49c0e51a0 0x7fc49c0d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46526/0 (socket says 172.21.2.117:46526) 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc4a0a00700 1 -- 172.21.2.117:0/2306479364 learned_addr learned my addr 172.21.2.117:0/2306479364 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc49bfff700 1 --2- 172.21.2.117:0/2306479364 >> v2:172.21.2.117:3300/0 conn(0x7fc49c07da20 0x7fc49c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc49bfff700 1 -- 172.21.2.117:0/2306479364 >> v1:172.21.2.125:6789/0 conn(0x7fc49c0e4da0 legacy=0x7fc49c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc49bfff700 1 -- 172.21.2.117:0/2306479364 >> v1:172.21.2.117:6789/0 conn(0x7fc49c0e51a0 legacy=0x7fc49c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.987 7fc49bfff700 1 -- 172.21.2.117:0/2306479364 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c1b08e0 con 0x7fc49c07da20 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc49bfff700 1 --2- 172.21.2.117:0/2306479364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c07da20 0x7fc49c0771b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7fc48c006650 tx=0x7fc48c009a00).ready entity=mon.0 client_cookie=b8a7f847d49d676a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc49affd700 1 -- 172.21.2.117:0/2306479364 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48c0009b0 con 0x7fc49c07da20 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc49affd700 1 -- 172.21.2.117:0/2306479364 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c00c0c0 con 0x7fc49c07da20 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc49affd700 1 -- 172.21.2.117:0/2306479364 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc48c00c5a0 con 0x7fc49c07da20 2021-02-10T11:38:50.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc4a23e3700 1 -- 172.21.2.117:0/2306479364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c07da20 msgr2=0x7fc49c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.988 7fc4a23e3700 1 --2- 172.21.2.117:0/2306479364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c07da20 0x7fc49c0771b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7fc48c006650 tx=0x7fc48c009a00).stop 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.989 7fc4a23e3700 1 -- 172.21.2.117:0/2306479364 shutdown_connections 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.989 7fc4a23e3700 1 --2- 172.21.2.117:0/2306479364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c07da20 0x7fc49c0771b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.989 7fc4a23e3700 1 -- 172.21.2.117:0/2306479364 >> 172.21.2.117:0/2306479364 conn(0x7fc49c0ce450 msgr2=0x7fc49c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.989 7fc4a23e3700 1 -- 172.21.2.117:0/2306479364 shutdown_connections 2021-02-10T11:38:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:50.989 7fc4a23e3700 1 -- 172.21.2.117:0/2306479364 wait complete. 2021-02-10T11:38:50.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 Processor -- start 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 -- start start 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc49c1c22d0 0x7fc49c0b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49c1b0770 0x7fc49c1b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c1b0f50 con 0x7fc49c0b6190 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c0b6590 con 0x7fc49c1c22d0 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a23e3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c1b0b70 con 0x7fc49c1b0770 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc49b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc49b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46280/0 (socket says 172.21.2.117:46280) 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc49b7fe700 1 -- 172.21.2.117:0/1805867524 learned_addr learned my addr 172.21.2.117:0/1805867524 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc49bfff700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc49c1c22d0 0x7fc49c0b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.004 7fc4a0a00700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49c1b0770 0x7fc49c1b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 -- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49c1b0770 msgr2=0x7fc49c1b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49c1b0770 0x7fc49c1b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 -- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc49c1c22d0 msgr2=0x7fc49c0b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc49c1c22d0 0x7fc49c0b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 -- 172.21.2.117:0/1805867524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c1b1af0 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc49b7fe700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fc490006ed0 tx=0x7fc49000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc490006240 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4900054b0 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc490005990 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c1b1af0 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c1b1af0 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.005 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc490006030 con 0x7fc49c0b6190 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.006 7fc498ff9700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 0x7fc47c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:50.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.006 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc490032910 con 0x7fc49c0b6190 2021-02-10T11:38:50.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.006 7fc49bfff700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 0x7fc47c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:50.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.006 7fc49bfff700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 0x7fc47c01abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.036 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c0ce450 con 0x7fc49c0b6190 2021-02-10T11:38:51.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.039 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc490000de0 con 0x7fc49c0b6190 2021-02-10T11:38:51.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.164 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802464", "target": ["mgr", ""]}) v1 -- 0x7fc49c1b0f50 con 0x7fc47c018730 2021-02-10T11:38:51.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.176 7fc498ff9700 1 -- 172.21.2.117:0/1805867524 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc48c006450 con 0x7fc47c018730 2021-02-10T11:38:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.181 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 msgr2=0x7fc47c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.181 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 0x7fc47c01abc0 crc :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.181 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 msgr2=0x7fc49c1b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.181 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7fc490006ed0 tx=0x7fc49000b960).stop 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 shutdown_connections 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc47c018730 0x7fc47c01abc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc49c0b6190 0x7fc49c1b0150 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc49c1b0770 0x7fc49c1b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 --2- 172.21.2.117:0/1805867524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc49c1c22d0 0x7fc49c0b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 >> 172.21.2.117:0/1805867524 conn(0x7fc49c1116c0 msgr2=0x7fc49c108870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 shutdown_connections 2021-02-10T11:38:51.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.182 7fc4a23e3700 1 -- 172.21.2.117:0/1805867524 wait complete. 2021-02-10T11:38:51.172 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802465 2021-02-10T11:38:51.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 Processor -- start 2021-02-10T11:38:51.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 -- start start 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f02c407da20 0x7f02c40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02c40e4da0 0x7f02c41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c4077a70 con 0x7f02c40e51a0 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c40777d0 con 0x7f02c40e4da0 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02c8994700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c407de20 con 0x7f02c407da20 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.605 7f02bbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02c40e4da0 0x7f02c41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02c40e4da0 0x7f02c41b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46288/0 (socket says 172.21.2.117:46288) 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 -- 172.21.2.117:0/2657207718 learned_addr learned my addr 172.21.2.117:0/2657207718 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c261c700 1 --2- 172.21.2.117:0/2657207718 >> v2:172.21.2.125:3300/0 conn(0x7f02c407da20 0x7f02c40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 -- 172.21.2.117:0/2657207718 >> v2:172.21.2.125:3300/0 conn(0x7f02c407da20 msgr2=0x7f02c40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 --2- 172.21.2.117:0/2657207718 >> v2:172.21.2.125:3300/0 conn(0x7f02c407da20 0x7f02c40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 -- 172.21.2.117:0/2657207718 >> v1:172.21.2.125:6789/0 conn(0x7f02c40e51a0 legacy=0x7f02c4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:51.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 -- 172.21.2.117:0/2657207718 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c41b0710 con 0x7f02c40e4da0 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02bbfff700 1 --2- 172.21.2.117:0/2657207718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40e4da0 0x7f02c41b00f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f02ac006700 tx=0x7f02ac009b80).ready entity=mon.0 client_cookie=6441ad279698f300 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c1e1b700 1 -- 172.21.2.117:0/2657207718 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02ac00c0c0 con 0x7f02c40e4da0 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c1e1b700 1 -- 172.21.2.117:0/2657207718 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac00c440 con 0x7f02c40e4da0 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c1e1b700 1 -- 172.21.2.117:0/2657207718 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02ac00c960 con 0x7f02c40e4da0 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c8994700 1 -- 172.21.2.117:0/2657207718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40e4da0 msgr2=0x7f02c41b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.606 7f02c8994700 1 --2- 172.21.2.117:0/2657207718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40e4da0 0x7f02c41b00f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f02ac006700 tx=0x7f02ac009b80).stop 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 -- 172.21.2.117:0/2657207718 shutdown_connections 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 --2- 172.21.2.117:0/2657207718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40e4da0 0x7f02c41b00f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 --2- 172.21.2.117:0/2657207718 >> v2:172.21.2.125:3300/0 conn(0x7f02c407da20 0x7f02c40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 -- 172.21.2.117:0/2657207718 >> 172.21.2.117:0/2657207718 conn(0x7f02c40ce450 msgr2=0x7f02c40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 -- 172.21.2.117:0/2657207718 shutdown_connections 2021-02-10T11:38:51.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.607 7f02c8994700 1 -- 172.21.2.117:0/2657207718 wait complete. 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 Processor -- start 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 -- start start 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 0x7f02c40c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02c40b7390 0x7f02c40b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40b8010 con 0x7f02c40a41c0 2021-02-10T11:38:51.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40b8550 con 0x7f02c40b7390 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c8994700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40b8770 con 0x7f02c40c2340 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02bbfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 0x7f02c40c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46294/0 (socket says 172.21.2.117:46294) 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02c261c700 1 -- 172.21.2.117:0/1356571933 learned_addr learned my addr 172.21.2.117:0/1356571933 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.622 7f02bbfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 0x7f02c40c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43976/0 (socket says 172.21.2.117:43976) 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c2e1d700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02c40b7390 0x7f02c40b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 -- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 msgr2=0x7f02c40c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 0x7f02c40c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 -- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02c40b7390 msgr2=0x7f02c40b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02c40b7390 0x7f02c40b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 -- 172.21.2.117:0/1356571933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c40b8990 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c261c700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f02b4006be0 tx=0x7f02b400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b40064d0 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b400bd80 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c8994700 1 -- 172.21.2.117:0/1356571933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c40b8990 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02b4002030 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.623 7f02c8994700 1 -- 172.21.2.117:0/1356571933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c40b8990 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.624 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02b4006c90 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.624 7f02baffd700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 0x7f02a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.624 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f02b40328f0 con 0x7f02c40a41c0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.624 7f02bbfff700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 0x7f02a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:51.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.625 7f02bbfff700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 0x7f02a801abc0 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.655 7f02c8994700 1 -- 172.21.2.117:0/1356571933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c41b4f00 con 0x7f02c40a41c0 2021-02-10T11:38:51.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.658 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02b4006820 con 0x7f02c40a41c0 2021-02-10T11:38:51.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.787 7f02c8994700 1 -- 172.21.2.117:0/1356571933 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802465", "target": ["mgr", ""]}) v1 -- 0x7f02c41b4f00 con 0x7f02a80186c0 2021-02-10T11:38:51.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.797 7f02baffd700 1 -- 172.21.2.117:0/1356571933 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f02ac006430 con 0x7f02a80186c0 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.804 7f02c8994700 1 -- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 msgr2=0x7f02a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.804 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 0x7f02a801abc0 crc :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.804 7f02c8994700 1 -- 172.21.2.117:0/1356571933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 msgr2=0x7f02c40fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.804 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f02b4006be0 tx=0x7f02b400b990).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 -- 172.21.2.117:0/1356571933 shutdown_connections 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02c40b7390 0x7f02c40b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02c40c2340 0x7f02c40c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02a80186c0 0x7f02a801abc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 --2- 172.21.2.117:0/1356571933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02c40a41c0 0x7f02c40fe0d0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 -- 172.21.2.117:0/1356571933 >> 172.21.2.117:0/1356571933 conn(0x7f02c408bbc0 msgr2=0x7f02c40c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:51.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 -- 172.21.2.117:0/1356571933 shutdown_connections 2021-02-10T11:38:51.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:51.805 7f02c8994700 1 -- 172.21.2.117:0/1356571933 wait complete. 2021-02-10T11:38:51.794 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802466 2021-02-10T11:38:52.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 Processor -- start 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 -- start start 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f39fc07da20 0x7f39fc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 0x7f39fc079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39fc077a70 con 0x7f39fc0e4da0 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39fc0777d0 con 0x7f39fc07da20 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0355e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc1b08e0 con 0x7f39fc0e51a0 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a0137a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f39fc07da20 0x7f39fc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.229 7f3a01b7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 0x7f39fc079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f39fc07da20 0x7f39fc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50688/0 (socket says 172.21.2.117:50688) 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 -- 172.21.2.117:0/3379303059 learned_addr learned my addr 172.21.2.117:0/3379303059 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a01b7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 0x7f39fc079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:43986/0 (socket says 172.21.2.117:43986) 2021-02-10T11:38:52.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a00b79700 1 --1- 172.21.2.117:0/3379303059 >> v1:172.21.2.125:6789/0 conn(0x7f39fc0e4da0 0x7f39fc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39478/0 (socket says 172.21.2.117:39478) 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 -- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 msgr2=0x7f39fc079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 --2- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 0x7f39fc079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 -- 172.21.2.117:0/3379303059 >> v1:172.21.2.125:6789/0 conn(0x7f39fc0e4da0 legacy=0x7f39fc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 -- 172.21.2.117:0/3379303059 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc07de20 con 0x7f39fc07da20 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0137a700 1 --2- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc07da20 0x7f39fc0771b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f39ec006700 tx=0x7f39ec009b80).ready entity=mon.1 client_cookie=1a169393980bc132 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f39f3fff700 1 -- 172.21.2.117:0/3379303059 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ec00c0c0 con 0x7f39fc07da20 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f39f3fff700 1 -- 172.21.2.117:0/3379303059 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec00c440 con 0x7f39fc07da20 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f39f3fff700 1 -- 172.21.2.117:0/3379303059 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ec0009b0 con 0x7f39fc07da20 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0355e700 1 -- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc07da20 msgr2=0x7f39fc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.230 7f3a0355e700 1 --2- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc07da20 0x7f39fc0771b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f39ec006700 tx=0x7f39ec009b80).stop 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 -- 172.21.2.117:0/3379303059 shutdown_connections 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 --2- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc0e51a0 0x7f39fc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 --2- 172.21.2.117:0/3379303059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc07da20 0x7f39fc0771b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 -- 172.21.2.117:0/3379303059 >> 172.21.2.117:0/3379303059 conn(0x7f39fc0ce450 msgr2=0x7f39fc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 -- 172.21.2.117:0/3379303059 shutdown_connections 2021-02-10T11:38:52.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.231 7f3a0355e700 1 -- 172.21.2.117:0/3379303059 wait complete. 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.246 7f3a0355e700 1 Processor -- start 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.246 7f3a0355e700 1 -- start start 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc187e20 0x7f39fc188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39fc090420 0x7f39fc090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc1067f0 con 0x7f39fc090420 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc107350 con 0x7f39fc1a0250 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0355e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc1042d0 con 0x7f39fc187e20 2021-02-10T11:38:52.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a01b7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39fc090420 0x7f39fc090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50692/0 (socket says 172.21.2.117:50692) 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 -- 172.21.2.117:0/3057029773 learned_addr learned my addr 172.21.2.117:0/3057029773 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a00b79700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc187e20 0x7f39fc188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 -- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc187e20 msgr2=0x7f39fc188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc187e20 0x7f39fc188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 -- 172.21.2.117:0/3057029773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39fc090420 msgr2=0x7f39fc090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39fc090420 0x7f39fc090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 -- 172.21.2.117:0/3057029773 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc090e40 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.247 7f3a0137a700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f39ec00b650 tx=0x7f39ec00b9f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.248 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ec00bde0 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.248 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec000c90 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.248 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39fc090e40 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.248 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39ec002220 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.248 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39fc090e40 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.249 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39ec002020 con 0x7f39fc1a0250 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.249 7f39f1ffb700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 0x7f39dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.249 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f39ec032720 con 0x7f39fc1a0250 2021-02-10T11:38:52.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.249 7f3a00b79700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 0x7f39dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.250 7f3a00b79700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 0x7f39dc01abc0 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.278 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39fc103280 con 0x7f39fc1a0250 2021-02-10T11:38:52.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.281 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39ec0091e0 con 0x7f39fc1a0250 2021-02-10T11:38:52.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.414 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802466", "target": ["mgr", ""]}) v1 -- 0x7f39fc106770 con 0x7f39dc0186c0 2021-02-10T11:38:52.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.424 7f39f1ffb700 1 -- 172.21.2.117:0/3057029773 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f39f8007cc0 con 0x7f39dc0186c0 2021-02-10T11:38:52.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.431 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 msgr2=0x7f39dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.431 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 0x7f39dc01abc0 crc :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.431 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 msgr2=0x7f39fc106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.431 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f39ec00b650 tx=0x7f39ec00b9f0).stop 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 shutdown_connections 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39dc0186c0 0x7f39dc01abc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39fc090420 0x7f39fc090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39fc187e20 0x7f39fc188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 --2- 172.21.2.117:0/3057029773 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39fc1a0250 0x7f39fc106cf0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 >> 172.21.2.117:0/3057029773 conn(0x7f39fc0a1850 msgr2=0x7f39fc0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 shutdown_connections 2021-02-10T11:38:52.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.432 7f3a0355e700 1 -- 172.21.2.117:0/3057029773 wait complete. 2021-02-10T11:38:52.422 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802467 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.854 7faaab1c7700 1 Processor -- start 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.854 7faaab1c7700 1 -- start start 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaab1c7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faaa4079000 0x7faaa40e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaab1c7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faaa41b00d0 0x7faaa41b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaab1c7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa4077a70 con 0x7faaa407da20 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaab1c7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa40e5c00 con 0x7faaa4079000 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaab1c7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa40e53a0 con 0x7faaa41b00d0 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faaa4079000 0x7faaa40e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faaa4079000 0x7faaa40e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46318/0 (socket says 172.21.2.117:46318) 2021-02-10T11:38:52.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 -- 172.21.2.117:0/964136827 learned_addr learned my addr 172.21.2.117:0/964136827 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa8fe3700 1 --1- 172.21.2.117:0/964136827 >> v1:172.21.2.125:6789/0 conn(0x7faaa407da20 0x7faaa40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39492/0 (socket says 172.21.2.117:39492) 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa97e4700 1 --2- 172.21.2.117:0/964136827 >> v2:172.21.2.125:3300/0 conn(0x7faaa41b00d0 0x7faaa41b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 -- 172.21.2.117:0/964136827 >> v2:172.21.2.125:3300/0 conn(0x7faaa41b00d0 msgr2=0x7faaa41b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 --2- 172.21.2.117:0/964136827 >> v2:172.21.2.125:3300/0 conn(0x7faaa41b00d0 0x7faaa41b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 -- 172.21.2.117:0/964136827 >> v1:172.21.2.125:6789/0 conn(0x7faaa407da20 legacy=0x7faaa40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 -- 172.21.2.117:0/964136827 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4079400 con 0x7faaa4079000 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa37fe700 1 -- 172.21.2.117:0/964136827 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205668960 0 0) 0x7faa8c0010c0 con 0x7faaa407da20 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.855 7faaa3fff700 1 --2- 172.21.2.117:0/964136827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa4079000 0x7faaa40e4d80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7faa94006a80 tx=0x7faa94009f00).ready entity=mon.0 client_cookie=745bd3449fa8dee6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.856 7faaa37fe700 1 -- 172.21.2.117:0/964136827 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa9400b5c0 con 0x7faaa4079000 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.856 7faaa37fe700 1 -- 172.21.2.117:0/964136827 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9400b8c0 con 0x7faaa4079000 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.856 7faaa37fe700 1 -- 172.21.2.117:0/964136827 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa9400bde0 con 0x7faaa4079000 2021-02-10T11:38:52.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.856 7faaab1c7700 1 -- 172.21.2.117:0/964136827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa4079000 msgr2=0x7faaa40e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.856 7faaab1c7700 1 --2- 172.21.2.117:0/964136827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa4079000 0x7faaa40e4d80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7faa94006a80 tx=0x7faa94009f00).stop 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 -- 172.21.2.117:0/964136827 shutdown_connections 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 --2- 172.21.2.117:0/964136827 >> v2:172.21.2.125:3300/0 conn(0x7faaa41b00d0 0x7faaa41b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 --2- 172.21.2.117:0/964136827 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa4079000 0x7faaa40e4d80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 -- 172.21.2.117:0/964136827 >> 172.21.2.117:0/964136827 conn(0x7faaa40ce430 msgr2=0x7faaa40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 -- 172.21.2.117:0/964136827 shutdown_connections 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.857 7faaab1c7700 1 -- 172.21.2.117:0/964136827 wait complete. 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 Processor -- start 2021-02-10T11:38:52.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 -- start start 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faaa40b78d0 0x7faaa40b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faaa40b82f0 0x7faaa4104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa40ba970 con 0x7faaa40a1310 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4104b90 con 0x7faaa40b82f0 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaab1c7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa4104d70 con 0x7faaa40b78d0 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaa8fe3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.872 7faaa8fe3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46326/0 (socket says 172.21.2.117:46326) 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 -- 172.21.2.117:0/2779319231 learned_addr learned my addr 172.21.2.117:0/2779319231 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:52.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa97e4700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faaa40b82f0 0x7faaa4104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa3fff700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faaa40b78d0 0x7faaa40b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 -- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faaa40b78d0 msgr2=0x7faaa40b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faaa40b78d0 0x7faaa40b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 -- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faaa40b82f0 msgr2=0x7faaa4104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faaa40b82f0 0x7faaa4104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 -- 172.21.2.117:0/2779319231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4104f90 con 0x7faaa40a1310 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa8fe3700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7faa8c00a590 tx=0x7faa8c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa8c007560 con 0x7faaa40a1310 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c00ad20 con 0x7faaa40a1310 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa4104f90 con 0x7faaa40a1310 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa4105520 con 0x7faaa40a1310 2021-02-10T11:38:52.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.873 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa8c004530 con 0x7faaa40a1310 2021-02-10T11:38:52.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.874 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa8c004320 con 0x7faaa40a1310 2021-02-10T11:38:52.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.874 7faaa17fa700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 0x7faa8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:52.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.874 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faa8c0328e0 con 0x7faaa40a1310 2021-02-10T11:38:52.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.874 7faaa3fff700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 0x7faa8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:52.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.874 7faaa3fff700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 0x7faa8801abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:52.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.904 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa4077a70 con 0x7faaa40a1310 2021-02-10T11:38:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:52.907 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa8c000e70 con 0x7faaa40a1310 2021-02-10T11:38:53.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.036 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802467", "target": ["mgr", ""]}) v1 -- 0x7faaa40ba970 con 0x7faa880186c0 2021-02-10T11:38:53.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.047 7faaa17fa700 1 -- 172.21.2.117:0/2779319231 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7faa94000d30 con 0x7faa880186c0 2021-02-10T11:38:53.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 msgr2=0x7faa8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 0x7faa8801abc0 crc :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 msgr2=0x7faaa418a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7faa8c00a590 tx=0x7faa8c00a930).stop 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 shutdown_connections 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faaa40b82f0 0x7faaa4104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faaa40b78d0 0x7faaa40b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa880186c0 0x7faa8801abc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 --2- 172.21.2.117:0/2779319231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faaa40a1310 0x7faaa418a090 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 >> 172.21.2.117:0/2779319231 conn(0x7faaa40bacb0 msgr2=0x7faaa408e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 shutdown_connections 2021-02-10T11:38:53.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.053 7faaab1c7700 1 -- 172.21.2.117:0/2779319231 wait complete. 2021-02-10T11:38:53.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802468 2021-02-10T11:38:53.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.476 7f12053b8700 1 Processor -- start 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.476 7f12053b8700 1 -- start start 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f12053b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f12000d1530 0x7f12000d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f12053b8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1200077a70 con 0x7f1200079000 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f12053b8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1200079400 con 0x7f120007da20 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f12053b8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120007de20 con 0x7f12000d1530 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11fe87c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1200079000 0x7f12000e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46580/0 (socket says 172.21.2.117:46580) 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11fe87c700 1 -- 172.21.2.117:0/1815931892 learned_addr learned my addr 172.21.2.117:0/1815931892 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11ff87e700 1 --2- 172.21.2.117:0/1815931892 >> v2:172.21.2.117:3300/0 conn(0x7f12000d1530 0x7f12000d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11fe07b700 1 -- 172.21.2.117:0/1815931892 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704757984 0 0) 0x7f11f8001060 con 0x7f1200079000 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11fe07b700 1 -- 172.21.2.117:0/1815931892 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ec002fe0 con 0x7f1200079000 2021-02-10T11:38:53.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11ff87e700 1 -- 172.21.2.117:0/1815931892 >> v1:172.21.2.125:6789/0 conn(0x7f120007da20 legacy=0x7f12000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11ff87e700 1 -- 172.21.2.117:0/1815931892 >> v1:172.21.2.117:6789/0 conn(0x7f1200079000 legacy=0x7f12000e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.477 7f11ff87e700 1 -- 172.21.2.117:0/1815931892 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12001b00f0 con 0x7f12000d1530 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f11ff87e700 1 --2- 172.21.2.117:0/1815931892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12000d1530 0x7f12000d1930 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f11f0006ab0 tx=0x7f11f0009b40).ready entity=mon.0 client_cookie=e2fcdeae030f2344 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f11fe07b700 1 -- 172.21.2.117:0/1815931892 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f00022e0 con 0x7f12000d1530 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f11fe07b700 1 -- 172.21.2.117:0/1815931892 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0002660 con 0x7f12000d1530 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f11fe07b700 1 -- 172.21.2.117:0/1815931892 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f0002ac0 con 0x7f12000d1530 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 -- 172.21.2.117:0/1815931892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12000d1530 msgr2=0x7f12000d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 --2- 172.21.2.117:0/1815931892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12000d1530 0x7f12000d1930 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f11f0006ab0 tx=0x7f11f0009b40).stop 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 -- 172.21.2.117:0/1815931892 shutdown_connections 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 --2- 172.21.2.117:0/1815931892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12000d1530 0x7f12000d1930 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 -- 172.21.2.117:0/1815931892 >> 172.21.2.117:0/1815931892 conn(0x7f12000ce450 msgr2=0x7f12000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 -- 172.21.2.117:0/1815931892 shutdown_connections 2021-02-10T11:38:53.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.478 7f12053b8700 1 -- 172.21.2.117:0/1815931892 wait complete. 2021-02-10T11:38:53.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 Processor -- start 2021-02-10T11:38:53.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 -- start start 2021-02-10T11:38:53.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 0x7f1200096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120008a180 0x7f120008a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200108020 con 0x7f12001a7240 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12001a7c60 con 0x7f1200107c20 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.493 7f12053b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12001a7e80 con 0x7f120008a180 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11fe87c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120008a180 0x7f120008a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46334/0 (socket says 172.21.2.117:46334) 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 -- 172.21.2.117:0/1681142675 learned_addr learned my addr 172.21.2.117:0/1681142675 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff07d700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 0x7f1200096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 -- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120008a180 msgr2=0x7f120008a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120008a180 0x7f120008a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 -- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 msgr2=0x7f1200096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 0x7f1200096fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 -- 172.21.2.117:0/1681142675 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120008b250 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff07d700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 0x7f1200096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.494 7f11ff87e700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f11f000b5e0 tx=0x7f11f000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f0006b40 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f0009070 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11f0002470 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f12053b8700 1 -- 172.21.2.117:0/1681142675 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120008b250 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f12053b8700 1 -- 172.21.2.117:0/1681142675 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120008b7e0 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11f0006a20 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 0x7f11e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.495 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f11f00326a0 con 0x7f12001a7240 2021-02-10T11:38:53.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.496 7f11ff07d700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 0x7f11e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:53.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.496 7f11ff07d700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 0x7f11e401abc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:53.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.527 7f12053b8700 1 -- 172.21.2.117:0/1681142675 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200077a70 con 0x7f12001a7240 2021-02-10T11:38:53.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.529 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11f0030d00 con 0x7f12001a7240 2021-02-10T11:38:53.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.656 7f12053b8700 1 -- 172.21.2.117:0/1681142675 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802468", "target": ["mgr", ""]}) v1 -- 0x7f120008b7e0 con 0x7f11e40186c0 2021-02-10T11:38:53.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.666 7f11f7fff700 1 -- 172.21.2.117:0/1681142675 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f11e8000960 con 0x7f11e40186c0 2021-02-10T11:38:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 -- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 msgr2=0x7f11e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 0x7f11e401abc0 crc :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 -- 172.21.2.117:0/1681142675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 msgr2=0x7f12001a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f11f000b5e0 tx=0x7f11f000b980).stop 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 -- 172.21.2.117:0/1681142675 shutdown_connections 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11e40186c0 0x7f11e401abc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12001a7240 0x7f12001a7640 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120008a180 0x7f120008a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 --2- 172.21.2.117:0/1681142675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1200107c20 0x7f1200096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.674 7f12053b8700 1 -- 172.21.2.117:0/1681142675 >> 172.21.2.117:0/1681142675 conn(0x7f12000a1880 msgr2=0x7f12000b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:53.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.675 7f12053b8700 1 -- 172.21.2.117:0/1681142675 shutdown_connections 2021-02-10T11:38:53.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:53.675 7f12053b8700 1 -- 172.21.2.117:0/1681142675 wait complete. 2021-02-10T11:38:53.664 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802469 2021-02-10T11:38:54.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.096 7f4b2e959700 1 Processor -- start 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.096 7f4b2e959700 1 -- start start 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 0x7f4b281b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b280e5180 0x7f4b28077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b28077a70 con 0x7f4b2807da20 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280777d0 con 0x7f4b280e5180 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2e959700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2807de20 con 0x7f4b280e4d80 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46342/0 (socket says 172.21.2.117:46342) 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 -- 172.21.2.117:0/1334604123 learned_addr learned my addr 172.21.2.117:0/1334604123 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b277fe700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 0x7f4b281b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b2cf76700 1 --2- 172.21.2.117:0/1334604123 >> v2:172.21.2.125:3300/0 conn(0x7f4b280e5180 0x7f4b28077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 -- 172.21.2.117:0/1334604123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 msgr2=0x7f4b281b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 0x7f4b281b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 -- 172.21.2.117:0/1334604123 >> v2:172.21.2.125:3300/0 conn(0x7f4b280e5180 msgr2=0x7f4b28077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 --2- 172.21.2.117:0/1334604123 >> v2:172.21.2.125:3300/0 conn(0x7f4b280e5180 0x7f4b28077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 -- 172.21.2.117:0/1334604123 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b281b06f0 con 0x7f4b2807da20 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b277fe700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 0x7f4b281b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.097 7f4b27fff700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b2807da20 0x7f4b280771b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f4b20006a80 tx=0x7f4b20009f00).ready entity=mon.0 client_cookie=d4a72cd80448e90e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b26ffd700 1 -- 172.21.2.117:0/1334604123 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2000b5c0 con 0x7f4b2807da20 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b26ffd700 1 -- 172.21.2.117:0/1334604123 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2000b8c0 con 0x7f4b2807da20 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b26ffd700 1 -- 172.21.2.117:0/1334604123 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2000bde0 con 0x7f4b2807da20 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 -- 172.21.2.117:0/1334604123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b2807da20 msgr2=0x7f4b280771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b2807da20 0x7f4b280771b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f4b20006a80 tx=0x7f4b20009f00).stop 2021-02-10T11:38:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 -- 172.21.2.117:0/1334604123 shutdown_connections 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 --2- 172.21.2.117:0/1334604123 >> v2:172.21.2.125:3300/0 conn(0x7f4b280e5180 0x7f4b28077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280e4d80 0x7f4b281b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 --2- 172.21.2.117:0/1334604123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b2807da20 0x7f4b280771b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 -- 172.21.2.117:0/1334604123 >> 172.21.2.117:0/1334604123 conn(0x7f4b280ce430 msgr2=0x7f4b280e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 -- 172.21.2.117:0/1334604123 shutdown_connections 2021-02-10T11:38:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.098 7f4b2e959700 1 -- 172.21.2.117:0/1334604123 wait complete. 2021-02-10T11:38:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 Processor -- start 2021-02-10T11:38:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 -- start start 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a1310 0x7f4b2818a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280b78d0 0x7f4b280b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280ba970 con 0x7f4b280b82f0 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b28104b90 con 0x7f4b280a1310 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.113 7f4b2e959700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b28104d70 con 0x7f4b280b78d0 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b277fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280b78d0 0x7f4b280b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46348/0 (socket says 172.21.2.117:46348) 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 -- 172.21.2.117:0/1827560371 learned_addr learned my addr 172.21.2.117:0/1827560371 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b27fff700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a1310 0x7f4b2818a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 -- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280b78d0 msgr2=0x7f4b280b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280b78d0 0x7f4b280b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 -- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a1310 msgr2=0x7f4b2818a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a1310 0x7f4b2818a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 -- 172.21.2.117:0/1827560371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28104f90 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2cf76700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f4b1800b580 tx=0x7f4b1800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b18006240 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18006640 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b18005330 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b28104f90 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.114 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b28104f90 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.115 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b18006030 con 0x7f4b280b82f0 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.115 7f4b24ff9700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 0x7f4b0801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.115 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b180327a0 con 0x7f4b280b82f0 2021-02-10T11:38:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.115 7f4b27fff700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 0x7f4b0801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.115 7f4b27fff700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 0x7f4b0801ac10 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.146 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b28077a70 con 0x7f4b280b82f0 2021-02-10T11:38:54.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.149 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b18006030 con 0x7f4b280b82f0 2021-02-10T11:38:54.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.273 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802469", "target": ["mgr", ""]}) v1 -- 0x7f4b28077a70 con 0x7f4b08018730 2021-02-10T11:38:54.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.283 7f4b24ff9700 1 -- 172.21.2.117:0/1827560371 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4b20000d30 con 0x7f4b08018730 2021-02-10T11:38:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 msgr2=0x7f4b0801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 0x7f4b0801ac10 crc :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 msgr2=0x7f4b28104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7f4b1800b580 tx=0x7f4b1800b920).stop 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 shutdown_connections 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b08018730 0x7f4b0801ac10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280b82f0 0x7f4b28104570 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b280b78d0 0x7f4b280b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 --2- 172.21.2.117:0/1827560371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a1310 0x7f4b2818a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.290 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 >> 172.21.2.117:0/1827560371 conn(0x7f4b280bacb0 msgr2=0x7f4b2808e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.291 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 shutdown_connections 2021-02-10T11:38:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.291 7f4b2e959700 1 -- 172.21.2.117:0/1827560371 wait complete. 2021-02-10T11:38:54.280 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802470 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.714 7f343c710700 1 Processor -- start 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 -- start start 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 0x7f34340771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f34340e5180 0x7f34340d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3434077a70 con 0x7f34340e4d80 2021-02-10T11:38:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34340e5c00 con 0x7f34340e5180 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343c710700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34340777d0 con 0x7f343407da20 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f3439d2b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f34340e4d80 0x7f34341b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46606/0 (socket says 172.21.2.117:46606) 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f3439d2b700 1 -- 172.21.2.117:0/3230392146 learned_addr learned my addr 172.21.2.117:0/3230392146 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343ad2d700 1 --2- 172.21.2.117:0/3230392146 >> v2:172.21.2.125:3300/0 conn(0x7f34340e5180 0x7f34340d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.715 7f343a52c700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 0x7f34340771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343952a700 1 -- 172.21.2.117:0/3230392146 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370995254 0 0) 0x7f34280010b0 con 0x7f34340e4d80 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343952a700 1 -- 172.21.2.117:0/3230392146 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3420002fe0 con 0x7f34340e4d80 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343ad2d700 1 -- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 msgr2=0x7f34340771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343ad2d700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 0x7f34340771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343ad2d700 1 -- 172.21.2.117:0/3230392146 >> v1:172.21.2.117:6789/0 conn(0x7f34340e4d80 legacy=0x7f34341b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343ad2d700 1 -- 172.21.2.117:0/3230392146 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34341b08c0 con 0x7f34340e5180 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343a52c700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 0x7f34340771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.716 7f343ad2d700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340e5180 0x7f34340d0500 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f342c006e50 tx=0x7f342c009e70).ready entity=mon.1 client_cookie=255a376ff676972 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.717 7f343952a700 1 -- 172.21.2.117:0/3230392146 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f342c006a20 con 0x7f34340e5180 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.717 7f343952a700 1 -- 172.21.2.117:0/3230392146 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c006790 con 0x7f34340e5180 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.717 7f343952a700 1 -- 172.21.2.117:0/3230392146 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f342c00b700 con 0x7f34340e5180 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.717 7f343c710700 1 -- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340e5180 msgr2=0x7f34340d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.717 7f343c710700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340e5180 0x7f34340d0500 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f342c006e50 tx=0x7f342c009e70).stop 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 -- 172.21.2.117:0/3230392146 shutdown_connections 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340e5180 0x7f34340d0500 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 --2- 172.21.2.117:0/3230392146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f343407da20 0x7f34340771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 -- 172.21.2.117:0/3230392146 >> 172.21.2.117:0/3230392146 conn(0x7f34340ce430 msgr2=0x7f34340e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 -- 172.21.2.117:0/3230392146 shutdown_connections 2021-02-10T11:38:54.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.718 7f343c710700 1 -- 172.21.2.117:0/3230392146 wait complete. 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.732 7f343c710700 1 Processor -- start 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.732 7f343c710700 1 -- start start 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340b7ef0 0x7f34340b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b8480 con 0x7f34340c1620 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b9020 con 0x7f34340b7ef0 2021-02-10T11:38:54.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343c710700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34340b9200 con 0x7f34340c3c40 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343a52c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f3439d2b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343a52c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46360/0 (socket says 172.21.2.117:46360) 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f3439d2b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44038/0 (socket says 172.21.2.117:44038) 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343a52c700 1 -- 172.21.2.117:0/4210259775 learned_addr learned my addr 172.21.2.117:0/4210259775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.733 7f343ad2d700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340b7ef0 0x7f34340b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 -- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 msgr2=0x7f34340b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 -- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340b7ef0 msgr2=0x7f34340b8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340b7ef0 0x7f34340b8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 -- 172.21.2.117:0/4210259775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34340b9420 con 0x7f34340c1620 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f3439d2b700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:54.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343a52c700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3430006f40 tx=0x7f343000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3430006bd0 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000bd80 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34300022b0 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343c710700 1 -- 172.21.2.117:0/4210259775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34340b9420 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.734 7f343c710700 1 -- 172.21.2.117:0/4210259775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34340b9420 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.735 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3430002020 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.735 7f3426ffd700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 0x7f341c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.735 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3430032710 con 0x7f34340c1620 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.735 7f3439d2b700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 0x7f341c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:54.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.735 7f3439d2b700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 0x7f341c01abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:54.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.764 7f343c710700 1 -- 172.21.2.117:0/4210259775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434077a70 con 0x7f34340c1620 2021-02-10T11:38:54.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.767 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3430030d70 con 0x7f34340c1620 2021-02-10T11:38:54.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.895 7f343c710700 1 -- 172.21.2.117:0/4210259775 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802470", "target": ["mgr", ""]}) v1 -- 0x7f3434077a70 con 0x7f341c0186c0 2021-02-10T11:38:54.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.905 7f3426ffd700 1 -- 172.21.2.117:0/4210259775 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f34280009b0 con 0x7f341c0186c0 2021-02-10T11:38:54.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 msgr2=0x7f341c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 0x7f341c01abc0 crc :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 msgr2=0x7f343418a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:54.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f3430006f40 tx=0x7f343000b990).stop 2021-02-10T11:38:54.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 shutdown_connections 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34340b7ef0 0x7f34340b8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34340c3c40 0x7f34340b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f341c0186c0 0x7f341c01abc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 --2- 172.21.2.117:0/4210259775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34340c1620 0x7f343418a090 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 >> 172.21.2.117:0/4210259775 conn(0x7f34340bacb0 msgr2=0x7f343408e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 shutdown_connections 2021-02-10T11:38:54.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:54.912 7f343c710700 1 -- 172.21.2.117:0/4210259775 wait complete. 2021-02-10T11:38:54.902 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802471 2021-02-10T11:38:55.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.334 7f8e28c26700 1 Processor -- start 2021-02-10T11:38:55.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.334 7f8e28c26700 1 -- start start 2021-02-10T11:38:55.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e28c26700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8e2407da20 0x7f8e240771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e28c26700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240d1d00 con 0x7f8e240e4d80 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e28c26700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e24077a70 con 0x7f8e240e5180 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e28c26700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e240777d0 con 0x7f8e2407da20 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e2285e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8e2407da20 0x7f8e240771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e2205d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8e240e4d80 0x7f8e241b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46618/0 (socket says 172.21.2.117:46618) 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e2205d700 1 -- 172.21.2.117:0/1126487812 learned_addr learned my addr 172.21.2.117:0/1126487812 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.335 7f8e2285e700 1 --2- 172.21.2.117:0/1126487812 >> v2:172.21.2.117:3300/0 conn(0x7f8e2407da20 0x7f8e240771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46366/0 (socket says 172.21.2.117:46366) 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2285e700 1 -- 172.21.2.117:0/1126487812 >> v1:172.21.2.125:6789/0 conn(0x7f8e240e5180 legacy=0x7f8e240d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2285e700 1 -- 172.21.2.117:0/1126487812 >> v1:172.21.2.117:6789/0 conn(0x7f8e240e4d80 legacy=0x7f8e241b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2285e700 1 -- 172.21.2.117:0/1126487812 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e241b08c0 con 0x7f8e2407da20 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2285e700 1 --2- 172.21.2.117:0/1126487812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2407da20 0x7f8e240771b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f8e1c0069d0 tx=0x7f8e1c009e40).ready entity=mon.0 client_cookie=2e67f0719a164835 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2185c700 1 -- 172.21.2.117:0/1126487812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c00b510 con 0x7f8e2407da20 2021-02-10T11:38:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2185c700 1 -- 172.21.2.117:0/1126487812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c00b790 con 0x7f8e2407da20 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e2185c700 1 -- 172.21.2.117:0/1126487812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c00bcb0 con 0x7f8e2407da20 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e28c26700 1 -- 172.21.2.117:0/1126487812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2407da20 msgr2=0x7f8e240771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.336 7f8e28c26700 1 --2- 172.21.2.117:0/1126487812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2407da20 0x7f8e240771b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7f8e1c0069d0 tx=0x7f8e1c009e40).stop 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.337 7f8e28c26700 1 -- 172.21.2.117:0/1126487812 shutdown_connections 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.337 7f8e28c26700 1 --2- 172.21.2.117:0/1126487812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2407da20 0x7f8e240771b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.337 7f8e28c26700 1 -- 172.21.2.117:0/1126487812 >> 172.21.2.117:0/1126487812 conn(0x7f8e240ce430 msgr2=0x7f8e240e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.337 7f8e28c26700 1 -- 172.21.2.117:0/1126487812 shutdown_connections 2021-02-10T11:38:55.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.337 7f8e28c26700 1 -- 172.21.2.117:0/1126487812 wait complete. 2021-02-10T11:38:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 Processor -- start 2021-02-10T11:38:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 -- start start 2021-02-10T11:38:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e240bf960 0x7f8e240c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 0x7f8e240ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240b6cf0 con 0x7f8e240bfd60 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240ae690 con 0x7f8e240adc70 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e28c26700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241116c0 con 0x7f8e240bf960 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.352 7f8e2205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46374/0 (socket says 172.21.2.117:46374) 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 -- 172.21.2.117:0/814825521 learned_addr learned my addr 172.21.2.117:0/814825521 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 -- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e240bf960 msgr2=0x7f8e240c0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e240bf960 0x7f8e240c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 -- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 msgr2=0x7f8e240ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2305f700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 0x7f8e240ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 0x7f8e240ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 -- 172.21.2.117:0/814825521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24110420 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2305f700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 0x7f8e240ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e2205d700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f8e1000a590 tx=0x7f8e1000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e10000ef0 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1000ad20 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e10004620 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e28c26700 1 -- 172.21.2.117:0/814825521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e24110420 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.353 7f8e28c26700 1 -- 172.21.2.117:0/814825521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24110420 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.354 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e100043a0 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.354 7f8e1b7fe700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.354 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8e10032930 con 0x7f8e240bfd60 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.355 7f8e2285e700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.355 7f8e2285e700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 0x7f8e0001abc0 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.384 7f8e28c26700 1 -- 172.21.2.117:0/814825521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e240ce430 con 0x7f8e240bfd60 2021-02-10T11:38:55.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.387 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e10007070 con 0x7f8e240bfd60 2021-02-10T11:38:55.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.513 7f8e28c26700 1 -- 172.21.2.117:0/814825521 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802471", "target": ["mgr", ""]}) v1 -- 0x7f8e240ae690 con 0x7f8e000186c0 2021-02-10T11:38:55.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.523 7f8e1b7fe700 1 -- 172.21.2.117:0/814825521 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8e1c006e60 con 0x7f8e000186c0 2021-02-10T11:38:55.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 msgr2=0x7f8e0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 0x7f8e0001abc0 crc :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 msgr2=0x7f8e240c0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7f8e1000a590 tx=0x7f8e1000a930).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 shutdown_connections 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e000186c0 0x7f8e0001abc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e240bfd60 0x7f8e240c0f20 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e240adc70 0x7f8e240ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 --2- 172.21.2.117:0/814825521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e240bf960 0x7f8e240c0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 >> 172.21.2.117:0/814825521 conn(0x7f8e241088c0 msgr2=0x7f8e2410f510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 shutdown_connections 2021-02-10T11:38:55.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.530 7f8e28c26700 1 -- 172.21.2.117:0/814825521 wait complete. 2021-02-10T11:38:55.520 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802472 2021-02-10T11:38:55.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.951 7f3d3c612700 1 Processor -- start 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.951 7f3d3c612700 1 -- start start 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d3c612700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3d340d1510 0x7f3d340d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d3c612700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d34077a70 con 0x7f3d34079000 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d3c612700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d34079400 con 0x7f3d3407da20 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d3c612700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3407de20 con 0x7f3d340d1510 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d39c2d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3d34079000 0x7f3d340e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46630/0 (socket says 172.21.2.117:46630) 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d39c2d700 1 -- 172.21.2.117:0/569689574 learned_addr learned my addr 172.21.2.117:0/569689574 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.952 7f3d3ac2f700 1 --2- 172.21.2.117:0/569689574 >> v2:172.21.2.125:3300/0 conn(0x7f3d340d1510 0x7f3d340d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042064407 0 0) 0x7f3d30001180 con 0x7f3d34079000 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20002fe0 con 0x7f3d34079000 2021-02-10T11:38:55.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067099805 0 0) 0x7f3d28000ff0 con 0x7f3d3407da20 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20004370 con 0x7f3d3407da20 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3ac2f700 1 -- 172.21.2.117:0/569689574 >> v1:172.21.2.125:6789/0 conn(0x7f3d3407da20 legacy=0x7f3d340771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3ac2f700 1 -- 172.21.2.117:0/569689574 >> v1:172.21.2.117:6789/0 conn(0x7f3d34079000 legacy=0x7f3d340e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3ac2f700 1 -- 172.21.2.117:0/569689574 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d341b00d0 con 0x7f3d340d1510 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3ac2f700 1 --2- 172.21.2.117:0/569689574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d340d1510 0x7f3d340d1910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3d2c006e50 tx=0x7f3d2c009df0).ready entity=mon.1 client_cookie=55e3bb34dca19b05 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c00b510 con 0x7f3d340d1510 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c003030 con 0x7f3d340d1510 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3c612700 1 -- 172.21.2.117:0/569689574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d340d1510 msgr2=0x7f3d340d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3942c700 1 -- 172.21.2.117:0/569689574 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c000f90 con 0x7f3d340d1510 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.953 7f3d3c612700 1 --2- 172.21.2.117:0/569689574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d340d1510 0x7f3d340d1910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3d2c006e50 tx=0x7f3d2c009df0).stop 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.954 7f3d3c612700 1 -- 172.21.2.117:0/569689574 shutdown_connections 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.954 7f3d3c612700 1 --2- 172.21.2.117:0/569689574 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d340d1510 0x7f3d340d1910 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.954 7f3d3c612700 1 -- 172.21.2.117:0/569689574 >> 172.21.2.117:0/569689574 conn(0x7f3d340ce430 msgr2=0x7f3d340e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:55.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.954 7f3d3c612700 1 -- 172.21.2.117:0/569689574 shutdown_connections 2021-02-10T11:38:55.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.954 7f3d3c612700 1 -- 172.21.2.117:0/569689574 wait complete. 2021-02-10T11:38:55.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 Processor -- start 2021-02-10T11:38:55.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 -- start start 2021-02-10T11:38:55.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d341a0270 0x7f3d340a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d340a3db0 0x7f3d34092650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34093070 con 0x7f3d34092c70 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34093290 con 0x7f3d341a0270 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.969 7f3d3c612700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3409cf60 con 0x7f3d340a3db0 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46386/0 (socket says 172.21.2.117:46386) 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 -- 172.21.2.117:0/3440721034 learned_addr learned my addr 172.21.2.117:0/3440721034 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d39c2d700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d340a3db0 0x7f3d34092650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3a42e700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d341a0270 0x7f3d340a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 -- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d340a3db0 msgr2=0x7f3d34092650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d340a3db0 0x7f3d34092650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 -- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d341a0270 msgr2=0x7f3d340a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d341a0270 0x7f3d340a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 -- 172.21.2.117:0/3440721034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3409d180 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3ac2f700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f3d2c009fc0 tx=0x7f3d2c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c0013b0 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c001630 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3409d180 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d2c001a90 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.970 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3409d180 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.971 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d2c009520 con 0x7f3d34092c70 2021-02-10T11:38:55.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.971 7f3d26ffd700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 0x7f3d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:55.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.971 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3d2c032b40 con 0x7f3d34092c70 2021-02-10T11:38:55.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.971 7f3d3a42e700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 0x7f3d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:55.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:55.971 7f3d3a42e700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 0x7f3d1c01abc0 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:55.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.002 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d34103350 con 0x7f3d34092c70 2021-02-10T11:38:55.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.004 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d2c001a90 con 0x7f3d34092c70 2021-02-10T11:38:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.133 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802472", "target": ["mgr", ""]}) v1 -- 0x7f3d34077a70 con 0x7f3d1c0186c0 2021-02-10T11:38:56.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.144 7f3d26ffd700 1 -- 172.21.2.117:0/3440721034 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3d28006550 con 0x7f3d1c0186c0 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 msgr2=0x7f3d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 0x7f3d1c01abc0 crc :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 msgr2=0x7f3d3409c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f3d2c009fc0 tx=0x7f3d2c00bd40).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 shutdown_connections 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d1c0186c0 0x7f3d1c01abc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d34092c70 0x7f3d3409c940 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d340a3db0 0x7f3d34092650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 --2- 172.21.2.117:0/3440721034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d341a0270 0x7f3d340a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 >> 172.21.2.117:0/3440721034 conn(0x7f3d340a1420 msgr2=0x7f3d34093b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 shutdown_connections 2021-02-10T11:38:56.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.151 7f3d3c612700 1 -- 172.21.2.117:0/3440721034 wait complete. 2021-02-10T11:38:56.140 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802473 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.574 7f2f4e3ec700 1 Processor -- start 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.574 7f2f4e3ec700 1 -- start start 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4e3ec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f4807da20 0x7f2f480771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4e3ec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2f480e51a0 0x7f2f48079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4e3ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f48077a70 con 0x7f2f480e4da0 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4e3ec700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f480e5c20 con 0x7f2f4807da20 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4e3ec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f480777d0 con 0x7f2f480e51a0 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f4807da20 0x7f2f480771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f4807da20 0x7f2f480771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46394/0 (socket says 172.21.2.117:46394) 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 -- 172.21.2.117:0/2282625806 learned_addr learned my addr 172.21.2.117:0/2282625806 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f4ca09700 1 --2- 172.21.2.117:0/2282625806 >> v2:172.21.2.125:3300/0 conn(0x7f2f480e51a0 0x7f2f48079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 -- 172.21.2.117:0/2282625806 >> v2:172.21.2.125:3300/0 conn(0x7f2f480e51a0 msgr2=0x7f2f48079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 --2- 172.21.2.117:0/2282625806 >> v2:172.21.2.125:3300/0 conn(0x7f2f480e51a0 0x7f2f48079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 -- 172.21.2.117:0/2282625806 >> v1:172.21.2.125:6789/0 conn(0x7f2f480e4da0 legacy=0x7f2f481b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.575 7f2f47fff700 1 -- 172.21.2.117:0/2282625806 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f481b08e0 con 0x7f2f4807da20 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f47fff700 1 --2- 172.21.2.117:0/2282625806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f4807da20 0x7f2f480771b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f2f38006700 tx=0x7f2f38009b80).ready entity=mon.0 client_cookie=a3b8dba4ebd7602d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f46ffd700 1 -- 172.21.2.117:0/2282625806 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f3800c0c0 con 0x7f2f4807da20 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f46ffd700 1 -- 172.21.2.117:0/2282625806 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800c440 con 0x7f2f4807da20 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f46ffd700 1 -- 172.21.2.117:0/2282625806 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f3800c960 con 0x7f2f4807da20 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 -- 172.21.2.117:0/2282625806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f4807da20 msgr2=0x7f2f480771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 --2- 172.21.2.117:0/2282625806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f4807da20 0x7f2f480771b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f2f38006700 tx=0x7f2f38009b80).stop 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 -- 172.21.2.117:0/2282625806 shutdown_connections 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 --2- 172.21.2.117:0/2282625806 >> v2:172.21.2.125:3300/0 conn(0x7f2f480e51a0 0x7f2f48079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 --2- 172.21.2.117:0/2282625806 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f4807da20 0x7f2f480771b0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 -- 172.21.2.117:0/2282625806 >> 172.21.2.117:0/2282625806 conn(0x7f2f480ce450 msgr2=0x7f2f480e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.576 7f2f4e3ec700 1 -- 172.21.2.117:0/2282625806 shutdown_connections 2021-02-10T11:38:56.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.577 7f2f4e3ec700 1 -- 172.21.2.117:0/2282625806 wait complete. 2021-02-10T11:38:56.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.591 7f2f4e3ec700 1 Processor -- start 2021-02-10T11:38:56.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.591 7f2f4e3ec700 1 -- start start 2021-02-10T11:38:56.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 0x7f2f481b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f481b5b50 0x7f2f480b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480b8da0 con 0x7f2f481b5f50 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480a4140 con 0x7f2f481b5130 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4e3ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f480a4320 con 0x7f2f481b5b50 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46402/0 (socket says 172.21.2.117:46402) 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 -- 172.21.2.117:0/2227529357 learned_addr learned my addr 172.21.2.117:0/2227529357 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f477fe700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f481b5b50 0x7f2f480b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f47fff700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 0x7f2f481b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 -- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f481b5b50 msgr2=0x7f2f480b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f481b5b50 0x7f2f480b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 -- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 msgr2=0x7f2f481b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 0x7f2f481b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 -- 172.21.2.117:0/2227529357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f480b62b0 con 0x7f2f481b5f50 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f47fff700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 0x7f2f481b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:56.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.592 7f2f4ca09700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f2f4000b970 tx=0x7f2f4000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f40000f90 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f400066f0 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f400015f0 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f480b62b0 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f480b6840 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f400011f0 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 0x7f2f2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.593 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2f40032a50 con 0x7f2f481b5f50 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.594 7f2f47fff700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 0x7f2f2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:56.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.594 7f2f47fff700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 0x7f2f2801abc0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:56.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.624 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f481b4f00 con 0x7f2f481b5f50 2021-02-10T11:38:56.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.627 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f400293d0 con 0x7f2f481b5f50 2021-02-10T11:38:56.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.754 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802473", "target": ["mgr", ""]}) v1 -- 0x7f2f480b6840 con 0x7f2f280186c0 2021-02-10T11:38:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.766 7f2f44ff9700 1 -- 172.21.2.117:0/2227529357 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2f380009b0 con 0x7f2f280186c0 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 msgr2=0x7f2f2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 0x7f2f2801abc0 crc :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 msgr2=0x7f2f480b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7f2f4000b970 tx=0x7f2f4000bd10).stop 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 shutdown_connections 2021-02-10T11:38:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f280186c0 0x7f2f2801abc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f481b5f50 0x7f2f480b5c90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f481b5b50 0x7f2f480b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 --2- 172.21.2.117:0/2227529357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f481b5130 0x7f2f481b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 >> 172.21.2.117:0/2227529357 conn(0x7f2f4808bbc0 msgr2=0x7f2f480c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 shutdown_connections 2021-02-10T11:38:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:56.772 7f2f4e3ec700 1 -- 172.21.2.117:0/2227529357 wait complete. 2021-02-10T11:38:56.762 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802474 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.193 7fdea4d33700 1 Processor -- start 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.193 7fdea4d33700 1 -- start start 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fdea4d33700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdea007da20 0x7fdea00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fdea4d33700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fdea4d33700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea0077a70 con 0x7fdea00e4d80 2021-02-10T11:38:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fdea4d33700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea00777d0 con 0x7fdea007da20 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fdea4d33700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea01b08c0 con 0x7fdea00e5180 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fde9f314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fde9f314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44094/0 (socket says 172.21.2.117:44094) 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fde9eb13700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdea007da20 0x7fdea00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.194 7fde9f314700 1 -- 172.21.2.117:0/184835759 learned_addr learned my addr 172.21.2.117:0/184835759 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9db11700 1 -- 172.21.2.117:0/184835759 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94583159 0 0) 0x7fde90000ff0 con 0x7fdea00e4d80 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9db11700 1 -- 172.21.2.117:0/184835759 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde8c002fe0 con 0x7fdea00e4d80 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9eb13700 1 -- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 msgr2=0x7fdea0079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9eb13700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9eb13700 1 -- 172.21.2.117:0/184835759 >> v1:172.21.2.125:6789/0 conn(0x7fdea00e4d80 legacy=0x7fdea01b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9eb13700 1 -- 172.21.2.117:0/184835759 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea007de20 con 0x7fdea007da20 2021-02-10T11:38:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9f314700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9eb13700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea007da20 0x7fdea00771b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fde88006650 tx=0x7fde88009a80).ready entity=mon.1 client_cookie=1a60fc80c13f5d00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9db11700 1 -- 172.21.2.117:0/184835759 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde8800c0c0 con 0x7fdea007da20 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9db11700 1 -- 172.21.2.117:0/184835759 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8800c0c0 con 0x7fdea007da20 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fde9db11700 1 -- 172.21.2.117:0/184835759 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde8800c340 con 0x7fdea007da20 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fdea4d33700 1 -- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea007da20 msgr2=0x7fdea00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.195 7fdea4d33700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea007da20 0x7fdea00771b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fde88006650 tx=0x7fde88009a80).stop 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 -- 172.21.2.117:0/184835759 shutdown_connections 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea00e5180 0x7fdea0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 --2- 172.21.2.117:0/184835759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea007da20 0x7fdea00771b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 -- 172.21.2.117:0/184835759 >> 172.21.2.117:0/184835759 conn(0x7fdea00ce430 msgr2=0x7fdea00e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 -- 172.21.2.117:0/184835759 shutdown_connections 2021-02-10T11:38:57.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.196 7fdea4d33700 1 -- 172.21.2.117:0/184835759 wait complete. 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.210 7fdea4d33700 1 Processor -- start 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 -- start start 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea0189f00 0x7fdea01893b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdea01062c0 0x7fdea01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea018a300 con 0x7fdea01062c0 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea01899d0 con 0x7fdea00bc940 2021-02-10T11:38:57.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fdea4d33700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea00bd360 con 0x7fdea0189f00 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fde9f314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fde9e312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdea01062c0 0x7fdea01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fde9f314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50800/0 (socket says 172.21.2.117:50800) 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fde9f314700 1 -- 172.21.2.117:0/3599838206 learned_addr learned my addr 172.21.2.117:0/3599838206 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.211 7fde9eb13700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea0189f00 0x7fdea01893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 -- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea0189f00 msgr2=0x7fdea01893b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea0189f00 0x7fdea01893b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 -- 172.21.2.117:0/3599838206 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdea01062c0 msgr2=0x7fdea01066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdea01062c0 0x7fdea01066c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 -- 172.21.2.117:0/3599838206 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea00bd580 con 0x7fdea00bc940 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde9f314700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fde98009940 tx=0x7fde9800bcf0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde98000f90 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde98006790 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde9800c570 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea00bd580 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.212 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea00bd580 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.213 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde980011f0 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.213 7fde977fe700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 0x7fde8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.213 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fde980328d0 con 0x7fdea00bc940 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.213 7fde9eb13700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 0x7fde8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.213 7fde9eb13700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 0x7fde8401abc0 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.244 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea0103260 con 0x7fdea00bc940 2021-02-10T11:38:57.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.247 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde98029210 con 0x7fdea00bc940 2021-02-10T11:38:57.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.374 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802474", "target": ["mgr", ""]}) v1 -- 0x7fdea0077a70 con 0x7fde840186c0 2021-02-10T11:38:57.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.384 7fde977fe700 1 -- 172.21.2.117:0/3599838206 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fde88006420 con 0x7fde840186c0 2021-02-10T11:38:57.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.391 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 msgr2=0x7fde8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.391 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 0x7fde8401abc0 crc :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.391 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 msgr2=0x7fdea00bcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.391 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fde98009940 tx=0x7fde9800bcf0).stop 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 shutdown_connections 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde840186c0 0x7fde8401abc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdea01062c0 0x7fdea01066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdea00bc940 0x7fdea00bcd40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 --2- 172.21.2.117:0/3599838206 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdea0189f00 0x7fdea01893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 >> 172.21.2.117:0/3599838206 conn(0x7fdea018a760 msgr2=0x7fdea0190320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 shutdown_connections 2021-02-10T11:38:57.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.392 7fdea4d33700 1 -- 172.21.2.117:0/3599838206 wait complete. 2021-02-10T11:38:57.383 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802475 2021-02-10T11:38:57.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.819 7f0366259700 1 Processor -- start 2021-02-10T11:38:57.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.819 7f0366259700 1 -- start start 2021-02-10T11:38:57.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0366259700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f036007da20 0x7f03600771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0366259700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03600e4d80 0x7f03601b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0366259700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0360077a70 con 0x7f03600e5180 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0366259700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03600e5c00 con 0x7f03600e4d80 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0366259700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03600777d0 con 0x7f036007da20 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03600e4d80 0x7f03601b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0364876700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f03600e5180 0x7f0360077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39600/0 (socket says 172.21.2.117:39600) 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f036007da20 0x7f03600771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f0364876700 1 -- 172.21.2.117:0/2944687776 learned_addr learned my addr 172.21.2.117:0/2944687776 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035effd700 1 -- 172.21.2.117:0/2944687776 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706295742 0 0) 0x7f0358001510 con 0x7f03600e5180 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035effd700 1 -- 172.21.2.117:0/2944687776 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0348002fe0 con 0x7f03600e5180 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035f7fe700 1 -- 172.21.2.117:0/2944687776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f036007da20 msgr2=0x7f03600771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035f7fe700 1 --2- 172.21.2.117:0/2944687776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f036007da20 0x7f03600771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.820 7f035f7fe700 1 -- 172.21.2.117:0/2944687776 >> v1:172.21.2.125:6789/0 conn(0x7f03600e5180 legacy=0x7f0360077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f035f7fe700 1 -- 172.21.2.117:0/2944687776 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036007de20 con 0x7f03600e4d80 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f035f7fe700 1 --2- 172.21.2.117:0/2944687776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03600e4d80 0x7f03601b00d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f0354006650 tx=0x7f0354009a80).ready entity=mon.0 client_cookie=f91299e084222e04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f035effd700 1 -- 172.21.2.117:0/2944687776 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03540009b0 con 0x7f03600e4d80 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f035effd700 1 -- 172.21.2.117:0/2944687776 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035400c0c0 con 0x7f03600e4d80 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f035effd700 1 -- 172.21.2.117:0/2944687776 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f035400c5e0 con 0x7f03600e4d80 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f0366259700 1 -- 172.21.2.117:0/2944687776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03600e4d80 msgr2=0x7f03601b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.821 7f0366259700 1 --2- 172.21.2.117:0/2944687776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03600e4d80 0x7f03601b00d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0x7f0354006650 tx=0x7f0354009a80).stop 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 -- 172.21.2.117:0/2944687776 shutdown_connections 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 --2- 172.21.2.117:0/2944687776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03600e4d80 0x7f03601b00d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 --2- 172.21.2.117:0/2944687776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f036007da20 0x7f03600771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 -- 172.21.2.117:0/2944687776 >> 172.21.2.117:0/2944687776 conn(0x7f03600ce430 msgr2=0x7f03600e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 -- 172.21.2.117:0/2944687776 shutdown_connections 2021-02-10T11:38:57.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.822 7f0366259700 1 -- 172.21.2.117:0/2944687776 wait complete. 2021-02-10T11:38:57.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.836 7f0366259700 1 Processor -- start 2021-02-10T11:38:57.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 -- start start 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 0x7f03600b8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 0x7f0360199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0360091ea0 con 0x7f0360199750 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0360187fd0 con 0x7f0360187bd0 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0366259700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03601881f0 con 0x7f0360091a50 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f035f7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 0x7f0360199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0364876700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f035ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 0x7f03600b8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0364876700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46434/0 (socket says 172.21.2.117:46434) 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.837 7f0364876700 1 -- 172.21.2.117:0/4645561 learned_addr learned my addr 172.21.2.117:0/4645561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 -- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 msgr2=0x7f03600b8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 0x7f03600b8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 -- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 msgr2=0x7f0360199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 0x7f0360199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 -- 172.21.2.117:0/4645561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03600abee0 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f035ffff700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 0x7f03600b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f035f7fe700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 0x7f0360199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0364876700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f035800a940 tx=0x7f035800ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03580048b0 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0358004b70 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03580070d0 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0366259700 1 -- 172.21.2.117:0/4645561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03600abee0 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.838 7f0366259700 1 -- 172.21.2.117:0/4645561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03600ac430 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.839 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03580046a0 con 0x7f0360199750 2021-02-10T11:38:57.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.839 7f035cff9700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 0x7f034001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:57.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.839 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0358032a80 con 0x7f0360199750 2021-02-10T11:38:57.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.839 7f035ffff700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 0x7f034001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:57.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.839 7f035ffff700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 0x7f034001abc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:57.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.869 7f0366259700 1 -- 172.21.2.117:0/4645561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0360077a70 con 0x7f0360199750 2021-02-10T11:38:57.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:57.872 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0358004a50 con 0x7f0360199750 2021-02-10T11:38:57.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.000 7f0366259700 1 -- 172.21.2.117:0/4645561 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802475", "target": ["mgr", ""]}) v1 -- 0x7f03600ac430 con 0x7f03400186c0 2021-02-10T11:38:57.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.010 7f035cff9700 1 -- 172.21.2.117:0/4645561 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f03500009b0 con 0x7f03400186c0 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 -- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 msgr2=0x7f034001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 0x7f034001abc0 crc :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 -- 172.21.2.117:0/4645561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 msgr2=0x7f03600ab8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f035800a940 tx=0x7f035800ace0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 -- 172.21.2.117:0/4645561 shutdown_connections 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03400186c0 0x7f034001abc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0360199750 0x7f03600ab8c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0360187bd0 0x7f0360199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 --2- 172.21.2.117:0/4645561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0360091a50 0x7f03600b8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.017 7f0366259700 1 -- 172.21.2.117:0/4645561 >> 172.21.2.117:0/4645561 conn(0x7f03600b5320 msgr2=0x7f03601b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.018 7f0366259700 1 -- 172.21.2.117:0/4645561 shutdown_connections 2021-02-10T11:38:57.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.018 7f0366259700 1 -- 172.21.2.117:0/4645561 wait complete. 2021-02-10T11:38:58.007 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802476 2021-02-10T11:38:58.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.441 7fcdd3348700 1 Processor -- start 2021-02-10T11:38:58.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.441 7fcdd3348700 1 -- start start 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd3348700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 0x7fcdcc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd3348700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd3348700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdcc077a70 con 0x7fcdcc0e51a0 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd3348700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdcc0e5c20 con 0x7fcdcc0e4da0 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd3348700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0777d0 con 0x7fcdcc07da20 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd1164700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 0x7fcdcc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd0963700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd1965700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcdcc0e51a0 0x7fcdcc077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39616/0 (socket says 172.21.2.117:39616) 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd0963700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50822/0 (socket says 172.21.2.117:50822) 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd1164700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 0x7fcdcc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44118/0 (socket says 172.21.2.117:44118) 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.442 7fcdd0963700 1 -- 172.21.2.117:0/3246532968 learned_addr learned my addr 172.21.2.117:0/3246532968 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:58.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd0963700 1 -- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 msgr2=0x7fcdcc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd0963700 1 --2- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 0x7fcdcc0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd0963700 1 -- 172.21.2.117:0/3246532968 >> v1:172.21.2.125:6789/0 conn(0x7fcdcc0e51a0 legacy=0x7fcdcc077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd0963700 1 -- 172.21.2.117:0/3246532968 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc07de20 con 0x7fcdcc0e4da0 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd0963700 1 --2- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fcdbc006700 tx=0x7fcdbc009b80).ready entity=mon.1 client_cookie=c4b9eb33811cb1ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdcbfff700 1 -- 172.21.2.117:0/3246532968 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdbc00c0c0 con 0x7fcdcc0e4da0 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdcbfff700 1 -- 172.21.2.117:0/3246532968 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc00c340 con 0x7fcdcc0e4da0 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdcbfff700 1 -- 172.21.2.117:0/3246532968 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdbc00c860 con 0x7fcdcc0e4da0 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd3348700 1 -- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0e4da0 msgr2=0x7fcdcc1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.443 7fcdd3348700 1 --2- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fcdbc006700 tx=0x7fcdbc009b80).stop 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 -- 172.21.2.117:0/3246532968 shutdown_connections 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 --2- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0e4da0 0x7fcdcc1b00f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 --2- 172.21.2.117:0/3246532968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc07da20 0x7fcdcc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 -- 172.21.2.117:0/3246532968 >> 172.21.2.117:0/3246532968 conn(0x7fcdcc0ce450 msgr2=0x7fcdcc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 -- 172.21.2.117:0/3246532968 shutdown_connections 2021-02-10T11:38:58.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.444 7fcdd3348700 1 -- 172.21.2.117:0/3246532968 wait complete. 2021-02-10T11:38:58.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.458 7fcdd3348700 1 Processor -- start 2021-02-10T11:38:58.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 -- start start 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0a3010 0x7fcdcc0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 0x7fcdcc0aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 0x7fcdcc0a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0a2ab0 con 0x7fcdcc0a9d70 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0aa790 con 0x7fcdcc0a3010 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd3348700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdcc0a3410 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd0963700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 0x7fcdcc0aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd0963700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 0x7fcdcc0aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46442/0 (socket says 172.21.2.117:46442) 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd0963700 1 -- 172.21.2.117:0/3301502316 learned_addr learned my addr 172.21.2.117:0/3301502316 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd1965700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 0x7fcdcc0a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.459 7fcdd1164700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0a3010 0x7fcdcc0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 -- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0a3010 msgr2=0x7fcdcc0fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0a3010 0x7fcdcc0fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 -- 172.21.2.117:0/3301502316 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 msgr2=0x7fcdcc0aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 0x7fcdcc0aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 -- 172.21.2.117:0/3301502316 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0a92f0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd1965700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 0x7fcdcc0a8cd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fcdc4001b10 tx=0x7fcdc400abd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdc4000f00 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc40070d0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc0a92f0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdcc0a92f0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.460 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdc40072b0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.461 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcdc4000eb0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.461 7fcdc9ffb700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 0x7fcdac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:58.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.461 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcdc40327f0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.461 7fcdd1164700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 0x7fcdac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:58.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.461 7fcdd1164700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 0x7fcdac01abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:58.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.491 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdcc0a9590 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.496 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcdc40046a0 con 0x7fcdcc0a88d0 2021-02-10T11:38:58.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.626 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802476", "target": ["mgr", ""]}) v1 -- 0x7fcdcc1b4f00 con 0x7fcdac0186c0 2021-02-10T11:38:58.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.637 7fcdc9ffb700 1 -- 172.21.2.117:0/3301502316 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcdc0006e70 con 0x7fcdac0186c0 2021-02-10T11:38:58.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.643 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 msgr2=0x7fcdac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.643 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 0x7fcdac01abc0 crc :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.643 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 msgr2=0x7fcdcc0a8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:58.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.643 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 0x7fcdcc0a8cd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fcdc4001b10 tx=0x7fcdc400abd0).stop 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 shutdown_connections 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdac0186c0 0x7fcdac01abc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcdcc0a9d70 0x7fcdcc0aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcdcc0a88d0 0x7fcdcc0a8cd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 --2- 172.21.2.117:0/3301502316 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcdcc0a3010 0x7fcdcc0fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:58.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 >> 172.21.2.117:0/3301502316 conn(0x7fcdcc0a3a70 msgr2=0x7fcdcc107450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:58.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 shutdown_connections 2021-02-10T11:38:58.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:58.644 7fcdd3348700 1 -- 172.21.2.117:0/3301502316 wait complete. 2021-02-10T11:38:58.633 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802477 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.064 7f19816ea700 1 Processor -- start 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.064 7f19816ea700 1 -- start start 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f19816ea700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f197c07da20 0x7f197c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f19816ea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f197c0e4da0 0x7f197c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f19816ea700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197c077a70 con 0x7f197c0e51a0 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f19816ea700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c0777d0 con 0x7f197c0e4da0 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f19816ea700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197c07de20 con 0x7f197c07da20 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f197c0e4da0 0x7f197c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f197c0e4da0 0x7f197c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46450/0 (socket says 172.21.2.117:46450) 2021-02-10T11:38:59.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 -- 172.21.2.117:0/1736293812 learned_addr learned my addr 172.21.2.117:0/1736293812 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 -- 172.21.2.117:0/1736293812 >> v2:172.21.2.125:3300/0 conn(0x7f197c07da20 msgr2=0x7f197c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 --2- 172.21.2.117:0/1736293812 >> v2:172.21.2.125:3300/0 conn(0x7f197c07da20 0x7f197c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 -- 172.21.2.117:0/1736293812 >> v1:172.21.2.125:6789/0 conn(0x7f197c0e51a0 legacy=0x7f197c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.065 7f197aabe700 1 -- 172.21.2.117:0/1736293812 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c1b0710 con 0x7f197c0e4da0 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f197aabe700 1 --2- 172.21.2.117:0/1736293812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c0e4da0 0x7f197c1b00f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f1964006700 tx=0x7f1964009b80).ready entity=mon.0 client_cookie=e4939649189488b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f197a2bd700 1 -- 172.21.2.117:0/1736293812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196400c0c0 con 0x7f197c0e4da0 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f197a2bd700 1 -- 172.21.2.117:0/1736293812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196400c340 con 0x7f197c0e4da0 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f197a2bd700 1 -- 172.21.2.117:0/1736293812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196400c860 con 0x7f197c0e4da0 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 -- 172.21.2.117:0/1736293812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c0e4da0 msgr2=0x7f197c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 --2- 172.21.2.117:0/1736293812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c0e4da0 0x7f197c1b00f0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f1964006700 tx=0x7f1964009b80).stop 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 -- 172.21.2.117:0/1736293812 shutdown_connections 2021-02-10T11:38:59.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 --2- 172.21.2.117:0/1736293812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c0e4da0 0x7f197c1b00f0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 --2- 172.21.2.117:0/1736293812 >> v2:172.21.2.125:3300/0 conn(0x7f197c07da20 0x7f197c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:38:59.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 -- 172.21.2.117:0/1736293812 >> 172.21.2.117:0/1736293812 conn(0x7f197c0ce450 msgr2=0x7f197c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.066 7f19816ea700 1 -- 172.21.2.117:0/1736293812 shutdown_connections 2021-02-10T11:38:59.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.067 7f19816ea700 1 -- 172.21.2.117:0/1736293812 wait complete. 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.081 7f19816ea700 1 Processor -- start 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.081 7f19816ea700 1 -- start start 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f197c1a7bf0 0x7f197c094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f197c094080 0x7f197c095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c095740 con 0x7f197c093c80 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c095960 con 0x7f197c094080 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f19816ea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c095b80 con 0x7f197c1a7bf0 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197bac0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f197c094080 0x7f197c095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197aabe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f197c1a7bf0 0x7f197c094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46456/0 (socket says 172.21.2.117:46456) 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 -- 172.21.2.117:0/2044973938 learned_addr learned my addr 172.21.2.117:0/2044973938 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 -- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f197c1a7bf0 msgr2=0x7f197c094b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f197c1a7bf0 0x7f197c094b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 -- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f197c094080 msgr2=0x7f197c095120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f197c094080 0x7f197c095120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 -- 172.21.2.117:0/2044973938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c095da0 con 0x7f197c093c80 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.082 7f197b2bf700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f196c006f00 tx=0x7f196c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196c006350 con 0x7f197c093c80 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196c00be00 con 0x7f197c093c80 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f19816ea700 1 -- 172.21.2.117:0/2044973938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c095da0 con 0x7f197c093c80 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f19816ea700 1 -- 172.21.2.117:0/2044973938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c0ac7a0 con 0x7f197c093c80 2021-02-10T11:38:59.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196c002020 con 0x7f197c093c80 2021-02-10T11:38:59.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f196c005310 con 0x7f197c093c80 2021-02-10T11:38:59.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 0x7f196001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.083 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f196c0328d0 con 0x7f197c093c80 2021-02-10T11:38:59.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.084 7f197aabe700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 0x7f196001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.084 7f197aabe700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 0x7f196001ac10 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.114 7f19816ea700 1 -- 172.21.2.117:0/2044973938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c103310 con 0x7f197c093c80 2021-02-10T11:38:59.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.117 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f196c000960 con 0x7f197c093c80 2021-02-10T11:38:59.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.244 7f19816ea700 1 -- 172.21.2.117:0/2044973938 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802477", "target": ["mgr", ""]}) v1 -- 0x7f197c095740 con 0x7f1960018730 2021-02-10T11:38:59.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.255 7f1973fff700 1 -- 172.21.2.117:0/2044973938 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1964006430 con 0x7f1960018730 2021-02-10T11:38:59.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 -- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 msgr2=0x7f196001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 0x7f196001ac10 crc :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 -- 172.21.2.117:0/2044973938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 msgr2=0x7f197c1a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f196c006f00 tx=0x7f196c00b940).stop 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 -- 172.21.2.117:0/2044973938 shutdown_connections 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f197c094080 0x7f197c095120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f197c1a7bf0 0x7f197c094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1960018730 0x7f196001ac10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 --2- 172.21.2.117:0/2044973938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f197c093c80 0x7f197c1a75d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.261 7f19816ea700 1 -- 172.21.2.117:0/2044973938 >> 172.21.2.117:0/2044973938 conn(0x7f197c0a1880 msgr2=0x7f197c0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.262 7f19816ea700 1 -- 172.21.2.117:0/2044973938 shutdown_connections 2021-02-10T11:38:59.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.262 7f19816ea700 1 -- 172.21.2.117:0/2044973938 wait complete. 2021-02-10T11:38:59.251 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802478 2021-02-10T11:38:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.682 7f3e5b9ca700 1 Processor -- start 2021-02-10T11:38:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.682 7f3e5b9ca700 1 -- start start 2021-02-10T11:38:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e5b9ca700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 0x7f3e540e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e5b9ca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e5b9ca700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e54077a70 con 0x7f3e5407da20 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e5b9ca700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540e53c0 con 0x7f3e54079000 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e5b9ca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54079400 con 0x7f3e541b00f0 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44144/0 (socket says 172.21.2.117:44144) 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 -- 172.21.2.117:0/405270228 learned_addr learned my addr 172.21.2.117:0/405270228 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e58fe5700 1 --2- 172.21.2.117:0/405270228 >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 0x7f3e540e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 -- 172.21.2.117:0/405270228 >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 msgr2=0x7f3e540e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 --2- 172.21.2.117:0/405270228 >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 0x7f3e540e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 -- 172.21.2.117:0/405270228 >> v1:172.21.2.125:6789/0 conn(0x7f3e5407da20 legacy=0x7f3e540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.683 7f3e59fe7700 1 -- 172.21.2.117:0/405270228 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5407de20 con 0x7f3e541b00f0 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e58fe5700 1 --2- 172.21.2.117:0/405270228 >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 0x7f3e540e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e59fe7700 1 --2- 172.21.2.117:0/405270228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3e4c006aa0 tx=0x7f3e4c009f20).ready entity=mon.2 client_cookie=745dda04a9b31268 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e4bfff700 1 -- 172.21.2.117:0/405270228 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c000f90 con 0x7f3e541b00f0 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e4bfff700 1 -- 172.21.2.117:0/405270228 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c003030 con 0x7f3e541b00f0 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e4bfff700 1 -- 172.21.2.117:0/405270228 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c00b7b0 con 0x7f3e541b00f0 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e5b9ca700 1 -- 172.21.2.117:0/405270228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 msgr2=0x7f3e541b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.684 7f3e5b9ca700 1 --2- 172.21.2.117:0/405270228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3e4c006aa0 tx=0x7f3e4c009f20).stop 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 -- 172.21.2.117:0/405270228 shutdown_connections 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 --2- 172.21.2.117:0/405270228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b00f0 0x7f3e541b04f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 --2- 172.21.2.117:0/405270228 >> v2:172.21.2.125:3300/0 conn(0x7f3e54079000 0x7f3e540e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 -- 172.21.2.117:0/405270228 >> 172.21.2.117:0/405270228 conn(0x7f3e540ce450 msgr2=0x7f3e540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 -- 172.21.2.117:0/405270228 shutdown_connections 2021-02-10T11:38:59.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.685 7f3e5b9ca700 1 -- 172.21.2.117:0/405270228 wait complete. 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 Processor -- start 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 -- start start 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e541b5130 0x7f3e541b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e541b5b50 0x7f3e540b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b8da0 con 0x7f3e541b5130 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540a4140 con 0x7f3e541b5b50 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e5b9ca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540a4320 con 0x7f3e541b5f50 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e59fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e597e6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e541b5130 0x7f3e541b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e59fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44150/0 (socket says 172.21.2.117:44150) 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e59fe7700 1 -- 172.21.2.117:0/3110392253 learned_addr learned my addr 172.21.2.117:0/3110392253 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.700 7f3e58fe5700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e541b5b50 0x7f3e540b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 -- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e541b5b50 msgr2=0x7f3e540b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e541b5b50 0x7f3e540b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 -- 172.21.2.117:0/3110392253 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e541b5130 msgr2=0x7f3e541b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e541b5130 0x7f3e541b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 -- 172.21.2.117:0/3110392253 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540b62b0 con 0x7f3e541b5f50 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e59fe7700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3e4c009980 tx=0x7f3e4c00bd10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c000f90 con 0x7f3e541b5f50 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c003030 con 0x7f3e541b5f50 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c0015a0 con 0x7f3e541b5f50 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e540b62b0 con 0x7f3e541b5f50 2021-02-10T11:38:59.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e540b62b0 con 0x7f3e541b5f50 2021-02-10T11:38:59.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.701 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e4c0013b0 con 0x7f3e541b5f50 2021-02-10T11:38:59.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.702 7f3e49ffb700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 0x7f3e3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:38:59.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.702 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3e4c032cc0 con 0x7f3e541b5f50 2021-02-10T11:38:59.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.702 7f3e597e6700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 0x7f3e3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:38:59.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.702 7f3e597e6700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 0x7f3e3401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:38:59.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.732 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e541b4f00 con 0x7f3e541b5f50 2021-02-10T11:38:59.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.735 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e4c0011a0 con 0x7f3e541b5f50 2021-02-10T11:38:59.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.864 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802478", "target": ["mgr", ""]}) v1 -- 0x7f3e540b6b30 con 0x7f3e34018730 2021-02-10T11:38:59.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.873 7f3e49ffb700 1 -- 172.21.2.117:0/3110392253 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3e50006a40 con 0x7f3e34018730 2021-02-10T11:38:59.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.880 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 msgr2=0x7f3e3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.880 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 0x7f3e3401abc0 crc :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.880 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 msgr2=0x7f3e540b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.880 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3e4c009980 tx=0x7f3e4c00bd10).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 shutdown_connections 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e541b5f50 0x7f3e540b5c90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e541b5b50 0x7f3e540b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e34018730 0x7f3e3401abc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 --2- 172.21.2.117:0/3110392253 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e541b5130 0x7f3e541b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 >> 172.21.2.117:0/3110392253 conn(0x7f3e5408bbc0 msgr2=0x7f3e540c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 shutdown_connections 2021-02-10T11:38:59.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:38:59.881 7f3e5b9ca700 1 -- 172.21.2.117:0/3110392253 wait complete. 2021-02-10T11:38:59.870 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802479 2021-02-10T11:39:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.303 7fabcdf0d700 1 Processor -- start 2021-02-10T11:39:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.303 7fabcdf0d700 1 -- start start 2021-02-10T11:39:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabcdf0d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 0x7fabc80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabcdf0d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fabc81b00f0 0x7fabc81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabcdf0d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc8077a70 con 0x7fabc807da20 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabcdf0d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc80e53c0 con 0x7fabc81b00f0 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabcdf0d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc8079400 con 0x7fabc8079000 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fabc807da20 0x7fabc80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46728/0 (socket says 172.21.2.117:46728) 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc77fe700 1 -- 172.21.2.117:0/799600361 learned_addr learned my addr 172.21.2.117:0/799600361 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc7fff700 1 --2- 172.21.2.117:0/799600361 >> v2:172.21.2.125:3300/0 conn(0x7fabc81b00f0 0x7fabc81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc6ffd700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 0x7fabc80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc67fc700 1 -- 172.21.2.117:0/799600361 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099452674 0 0) 0x7fabb8000ce0 con 0x7fabc807da20 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc67fc700 1 -- 172.21.2.117:0/799600361 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0002fe0 con 0x7fabc807da20 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc7fff700 1 -- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 msgr2=0x7fabc80e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc7fff700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 0x7fabc80e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc7fff700 1 -- 172.21.2.117:0/799600361 >> v1:172.21.2.117:6789/0 conn(0x7fabc807da20 legacy=0x7fabc80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc7fff700 1 -- 172.21.2.117:0/799600361 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc807de20 con 0x7fabc81b00f0 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.304 7fabc6ffd700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 0x7fabc80e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabc7fff700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc81b00f0 0x7fabc81b04f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fabc0006e50 tx=0x7fabc0009e70).ready entity=mon.1 client_cookie=7a8ad1b1f5707965 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabc67fc700 1 -- 172.21.2.117:0/799600361 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc0006a20 con 0x7fabc81b00f0 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabc67fc700 1 -- 172.21.2.117:0/799600361 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc0006790 con 0x7fabc81b00f0 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabc67fc700 1 -- 172.21.2.117:0/799600361 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc000b700 con 0x7fabc81b00f0 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabcdf0d700 1 -- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc81b00f0 msgr2=0x7fabc81b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.305 7fabcdf0d700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc81b00f0 0x7fabc81b04f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fabc0006e50 tx=0x7fabc0009e70).stop 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 -- 172.21.2.117:0/799600361 shutdown_connections 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc81b00f0 0x7fabc81b04f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 --2- 172.21.2.117:0/799600361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc8079000 0x7fabc80e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 -- 172.21.2.117:0/799600361 >> 172.21.2.117:0/799600361 conn(0x7fabc80ce450 msgr2=0x7fabc80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 -- 172.21.2.117:0/799600361 shutdown_connections 2021-02-10T11:39:00.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.306 7fabcdf0d700 1 -- 172.21.2.117:0/799600361 wait complete. 2021-02-10T11:39:00.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 Processor -- start 2021-02-10T11:39:00.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 -- start start 2021-02-10T11:39:00.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 0x7fabc80fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80c2340 0x7fabc80c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80b8010 con 0x7fabc80b7390 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80b8550 con 0x7fabc80c2340 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabcdf0d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80b8770 con 0x7fabc80a41c0 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabc7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabc7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46486/0 (socket says 172.21.2.117:46486) 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabc7fff700 1 -- 172.21.2.117:0/3484599895 learned_addr learned my addr 172.21.2.117:0/3484599895 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabc6ffd700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80c2340 0x7fabc80c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.321 7fabc77fe700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 0x7fabc80fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 -- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 msgr2=0x7fabc80fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 0x7fabc80fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 -- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80c2340 msgr2=0x7fabc80c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80c2340 0x7fabc80c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 -- 172.21.2.117:0/3484599895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80b8990 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc77fe700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 0x7fabc80fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabc7fff700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fabc0006e60 tx=0x7fabc000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc0000df0 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc00091c0 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabc0001390 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc80b8990 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.322 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc80b8f20 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.323 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fabc00011a0 con 0x7fabc80b7390 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.323 7fabaffff700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.323 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fabc00328d0 con 0x7fabc80b7390 2021-02-10T11:39:00.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.323 7fabc77fe700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.323 7fabc77fe700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.353 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc81b4f00 con 0x7fabc80b7390 2021-02-10T11:39:00.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.356 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fabc002e5a0 con 0x7fabc80b7390 2021-02-10T11:39:00.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.481 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802479", "target": ["mgr", ""]}) v1 -- 0x7fabc80b8f20 con 0x7faba80186c0 2021-02-10T11:39:00.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.492 7fabaffff700 1 -- 172.21.2.117:0/3484599895 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fabb80094c0 con 0x7faba80186c0 2021-02-10T11:39:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.498 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 msgr2=0x7faba801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.498 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 crc :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.498 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 msgr2=0x7fabc80b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fabc0006e60 tx=0x7fabc000bcf0).stop 2021-02-10T11:39:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 shutdown_connections 2021-02-10T11:39:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80b7390 0x7fabc80b7790 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80c2340 0x7fabc80c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 --2- 172.21.2.117:0/3484599895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc80a41c0 0x7fabc80fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 >> 172.21.2.117:0/3484599895 conn(0x7fabc808bbc0 msgr2=0x7fabc80c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 shutdown_connections 2021-02-10T11:39:00.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.499 7fabcdf0d700 1 -- 172.21.2.117:0/3484599895 wait complete. 2021-02-10T11:39:00.489 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802480 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.922 7f5c0e224700 1 Processor -- start 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.922 7f5c0e224700 1 -- start start 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0e224700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5c0807da20 0x7f5c080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0e224700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5c080e4da0 0x7f5c081b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0e224700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08077a70 con 0x7f5c080e51a0 2021-02-10T11:39:00.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0e224700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c080777d0 con 0x7f5c0807da20 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0e224700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0807de20 con 0x7f5c080e4da0 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0c841700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5c080e51a0 0x7f5c08077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46744/0 (socket says 172.21.2.117:46744) 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c0c841700 1 -- 172.21.2.117:0/1540781583 learned_addr learned my addr 172.21.2.117:0/1540781583 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5c0807da20 0x7f5c080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c077fe700 1 --2- 172.21.2.117:0/1540781583 >> v2:172.21.2.125:3300/0 conn(0x7f5c080e4da0 0x7f5c081b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 -- 172.21.2.117:0/1540781583 >> v2:172.21.2.125:3300/0 conn(0x7f5c080e4da0 msgr2=0x7f5c081b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 --2- 172.21.2.117:0/1540781583 >> v2:172.21.2.125:3300/0 conn(0x7f5c080e4da0 0x7f5c081b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 -- 172.21.2.117:0/1540781583 >> v1:172.21.2.117:6789/0 conn(0x7f5c080e51a0 legacy=0x7f5c08077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c06ffd700 1 -- 172.21.2.117:0/1540781583 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440233146 0 0) 0x7f5c00001500 con 0x7f5c080e51a0 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 -- 172.21.2.117:0/1540781583 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c081b0710 con 0x7f5c0807da20 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.923 7f5c07fff700 1 --2- 172.21.2.117:0/1540781583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c0807da20 0x7f5c080771b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f5bf8006650 tx=0x7f5bf8009a80).ready entity=mon.0 client_cookie=3a05c38fb8bdf671 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c06ffd700 1 -- 172.21.2.117:0/1540781583 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bf80009b0 con 0x7f5c0807da20 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c06ffd700 1 -- 172.21.2.117:0/1540781583 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf800c0c0 con 0x7f5c0807da20 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c06ffd700 1 -- 172.21.2.117:0/1540781583 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bf800c520 con 0x7f5c0807da20 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 -- 172.21.2.117:0/1540781583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c0807da20 msgr2=0x7f5c080771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 --2- 172.21.2.117:0/1540781583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c0807da20 0x7f5c080771b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7f5bf8006650 tx=0x7f5bf8009a80).stop 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 -- 172.21.2.117:0/1540781583 shutdown_connections 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 --2- 172.21.2.117:0/1540781583 >> v2:172.21.2.125:3300/0 conn(0x7f5c080e4da0 0x7f5c081b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 --2- 172.21.2.117:0/1540781583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c0807da20 0x7f5c080771b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 -- 172.21.2.117:0/1540781583 >> 172.21.2.117:0/1540781583 conn(0x7f5c080ce450 msgr2=0x7f5c080e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 -- 172.21.2.117:0/1540781583 shutdown_connections 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.924 7f5c0e224700 1 -- 172.21.2.117:0/1540781583 wait complete. 2021-02-10T11:39:00.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.939 7f5c0e224700 1 Processor -- start 2021-02-10T11:39:00.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.939 7f5c0e224700 1 -- start start 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 0x7f5c08107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c08097ad0 0x7f5c08092430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08107720 con 0x7f5c080976d0 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08196110 con 0x7f5c08107320 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08196330 con 0x7f5c08097ad0 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46502/0 (socket says 172.21.2.117:46502) 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 -- 172.21.2.117:0/4029567997 learned_addr learned my addr 172.21.2.117:0/4029567997 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0c841700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c08097ad0 0x7f5c08092430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c07fff700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 0x7f5c08107b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 -- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c08097ad0 msgr2=0x7f5c08092430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c08097ad0 0x7f5c08092430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 -- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 msgr2=0x7f5c08107b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 0x7f5c08107b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 -- 172.21.2.117:0/4029567997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08092a50 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c07fff700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 0x7f5c08107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c077fe700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f5bfc006f00 tx=0x7f5bfc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bfc00bd30 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc002030 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c08092a50 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c08092a50 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.940 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5bfc002550 con 0x7f5c080976d0 2021-02-10T11:39:00.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.941 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5bfc000c90 con 0x7f5c080976d0 2021-02-10T11:39:00.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.941 7f5c04ff9700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 0x7f5be801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:00.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.941 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5bfc032930 con 0x7f5c080976d0 2021-02-10T11:39:00.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.941 7f5c07fff700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 0x7f5be801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:00.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.941 7f5c07fff700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 0x7f5be801abc0 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:00.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.972 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c081b4f00 con 0x7f5c080976d0 2021-02-10T11:39:00.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:00.975 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5bfc00bd30 con 0x7f5c080976d0 2021-02-10T11:39:01.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.101 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802480", "target": ["mgr", ""]}) v1 -- 0x7f5c081b4f00 con 0x7f5be8018730 2021-02-10T11:39:01.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.112 7f5c04ff9700 1 -- 172.21.2.117:0/4029567997 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5bf8006390 con 0x7f5be8018730 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 msgr2=0x7f5be801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 0x7f5be801abc0 crc :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 msgr2=0x7f5c08195af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7f5bfc006f00 tx=0x7f5bfc00b940).stop 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 shutdown_connections 2021-02-10T11:39:01.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5be8018730 0x7f5be801abc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5c080976d0 0x7f5c08195af0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5c08097ad0 0x7f5c08092430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 --2- 172.21.2.117:0/4029567997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5c08107320 0x7f5c08107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.118 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 >> 172.21.2.117:0/4029567997 conn(0x7f5c080ffe30 msgr2=0x7f5c080c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.119 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 shutdown_connections 2021-02-10T11:39:01.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.119 7f5c0e224700 1 -- 172.21.2.117:0/4029567997 wait complete. 2021-02-10T11:39:01.108 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802481 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.540 7f750a182700 1 Processor -- start 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.540 7f750a182700 1 -- start start 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f750a182700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 0x7f75040771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f750a182700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f75040e4da0 0x7f75041b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f750a182700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7504077a70 con 0x7f75040e51a0 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f750a182700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75040e5c20 con 0x7f75040e4da0 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f750a182700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75040777d0 con 0x7f750407da20 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f7503fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f75040e51a0 0x7f7504077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46756/0 (socket says 172.21.2.117:46756) 2021-02-10T11:39:01.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f7503fff700 1 -- 172.21.2.117:0/3193071183 learned_addr learned my addr 172.21.2.117:0/3193071183 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f75037fe700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 0x7f75040771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.541 7f7502ffd700 1 --2- 172.21.2.117:0/3193071183 >> v2:172.21.2.125:3300/0 conn(0x7f75040e4da0 0x7f75041b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f75027fc700 1 -- 172.21.2.117:0/3193071183 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856819505 0 0) 0x7f74fc001500 con 0x7f75040e51a0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f7502ffd700 1 -- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 msgr2=0x7f75040771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f7502ffd700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 0x7f75040771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f7502ffd700 1 -- 172.21.2.117:0/3193071183 >> v1:172.21.2.117:6789/0 conn(0x7f75040e51a0 legacy=0x7f7504077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f7502ffd700 1 -- 172.21.2.117:0/3193071183 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750407de20 con 0x7f75040e4da0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f75037fe700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 0x7f75040771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f7502ffd700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75040e4da0 0x7f75041b00f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f74f8006650 tx=0x7f74f8009a80).ready entity=mon.1 client_cookie=4cbbe071d22a8c43 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f75027fc700 1 -- 172.21.2.117:0/3193071183 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f80009b0 con 0x7f75040e4da0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f75027fc700 1 -- 172.21.2.117:0/3193071183 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f800c0c0 con 0x7f75040e4da0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f75027fc700 1 -- 172.21.2.117:0/3193071183 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f800c5e0 con 0x7f75040e4da0 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f750a182700 1 -- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75040e4da0 msgr2=0x7f75041b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.542 7f750a182700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75040e4da0 0x7f75041b00f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f74f8006650 tx=0x7f74f8009a80).stop 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 -- 172.21.2.117:0/3193071183 shutdown_connections 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75040e4da0 0x7f75041b00f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 --2- 172.21.2.117:0/3193071183 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f750407da20 0x7f75040771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 -- 172.21.2.117:0/3193071183 >> 172.21.2.117:0/3193071183 conn(0x7f75040ce450 msgr2=0x7f75040e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 -- 172.21.2.117:0/3193071183 shutdown_connections 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.543 7f750a182700 1 -- 172.21.2.117:0/3193071183 wait complete. 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 Processor -- start 2021-02-10T11:39:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 -- start start 2021-02-10T11:39:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f750408eb10 0x7f75040c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 0x7f75040afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75040af540 con 0x7f750408dfe0 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75040a0c80 con 0x7f750408eb10 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.558 7f750a182700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75040b04e0 con 0x7f75040afac0 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46516/0 (socket says 172.21.2.117:46516) 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 -- 172.21.2.117:0/3770068479 learned_addr learned my addr 172.21.2.117:0/3770068479 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f75037fe700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f750408eb10 0x7f75040c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7503fff700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 0x7f75040afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 -- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 msgr2=0x7f75040afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 0x7f75040afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 -- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f750408eb10 msgr2=0x7f75040c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f750408eb10 0x7f75040c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 -- 172.21.2.117:0/3770068479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75040b0700 con 0x7f750408dfe0 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7503fff700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 0x7f75040afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f7502ffd700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f74f8009630 tx=0x7f74f800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f80093a0 con 0x7f750408dfe0 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.559 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f800bd70 con 0x7f750408dfe0 2021-02-10T11:39:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f750a182700 1 -- 172.21.2.117:0/3770068479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75040b0700 con 0x7f750408dfe0 2021-02-10T11:39:01.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f750a182700 1 -- 172.21.2.117:0/3770068479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75040b0c90 con 0x7f750408dfe0 2021-02-10T11:39:01.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f8005560 con 0x7f750408dfe0 2021-02-10T11:39:01.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74f800a000 con 0x7f750408dfe0 2021-02-10T11:39:01.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f74ebfff700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 0x7f74e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:01.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.560 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f74f80326c0 con 0x7f750408dfe0 2021-02-10T11:39:01.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.561 7f75037fe700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 0x7f74e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:01.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.561 7f75037fe700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 0x7f74e401abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:01.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.591 7f750a182700 1 -- 172.21.2.117:0/3770068479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7504103280 con 0x7f750408dfe0 2021-02-10T11:39:01.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.593 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74f8025fd0 con 0x7f750408dfe0 2021-02-10T11:39:01.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.719 7f750a182700 1 -- 172.21.2.117:0/3770068479 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802481", "target": ["mgr", ""]}) v1 -- 0x7f75040b0c90 con 0x7f74e40186c0 2021-02-10T11:39:01.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.730 7f74ebfff700 1 -- 172.21.2.117:0/3770068479 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f74f4006a30 con 0x7f74e40186c0 2021-02-10T11:39:01.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 msgr2=0x7f74e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 0x7f74e401abc0 crc :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 msgr2=0x7f750408e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f74f8009630 tx=0x7f74f800b980).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 shutdown_connections 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74e40186c0 0x7f74e401abc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f750408dfe0 0x7f750408e3e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75040afac0 0x7f75040afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 --2- 172.21.2.117:0/3770068479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f750408eb10 0x7f75040c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 >> 172.21.2.117:0/3770068479 conn(0x7f75040a1850 msgr2=0x7f75040bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 shutdown_connections 2021-02-10T11:39:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:01.737 7f750a182700 1 -- 172.21.2.117:0/3770068479 wait complete. 2021-02-10T11:39:01.726 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802482 2021-02-10T11:39:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.157 7f9edfc2c700 1 Processor -- start 2021-02-10T11:39:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.157 7f9edfc2c700 1 -- start start 2021-02-10T11:39:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edfc2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edfc2c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edfc2c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed8077a70 con 0x7f9ed80e4da0 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edfc2c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80e5c20 con 0x7f9ed80e51a0 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edfc2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80777d0 con 0x7f9ed807da20 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9ede249700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44200/0 (socket says 172.21.2.117:44200) 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9ede249700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:50904/0 (socket says 172.21.2.117:50904) 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 -- 172.21.2.117:0/1256816543 learned_addr learned my addr 172.21.2.117:0/1256816543 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 -- 172.21.2.117:0/1256816543 >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 msgr2=0x7f9ed80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 --2- 172.21.2.117:0/1256816543 >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 -- 172.21.2.117:0/1256816543 >> v1:172.21.2.125:6789/0 conn(0x7f9ed80e4da0 legacy=0x7f9ed81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9edda48700 1 -- 172.21.2.117:0/1256816543 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed81b08e0 con 0x7f9ed807da20 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.158 7f9ede249700 1 --2- 172.21.2.117:0/1256816543 >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edda48700 1 --2- 172.21.2.117:0/1256816543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9ecc006700 tx=0x7f9ecc009b80).ready entity=mon.2 client_cookie=6227d3c2c0e16aa6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edca46700 1 -- 172.21.2.117:0/1256816543 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ecc00c0c0 con 0x7f9ed807da20 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edca46700 1 -- 172.21.2.117:0/1256816543 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00c340 con 0x7f9ed807da20 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edca46700 1 -- 172.21.2.117:0/1256816543 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ecc00c860 con 0x7f9ed807da20 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edfc2c700 1 -- 172.21.2.117:0/1256816543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 msgr2=0x7f9ed80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.159 7f9edfc2c700 1 --2- 172.21.2.117:0/1256816543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9ecc006700 tx=0x7f9ecc009b80).stop 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 -- 172.21.2.117:0/1256816543 shutdown_connections 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 --2- 172.21.2.117:0/1256816543 >> v2:172.21.2.125:3300/0 conn(0x7f9ed80e51a0 0x7f9ed80d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 --2- 172.21.2.117:0/1256816543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed807da20 0x7f9ed80771b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 -- 172.21.2.117:0/1256816543 >> 172.21.2.117:0/1256816543 conn(0x7f9ed80ce450 msgr2=0x7f9ed80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 -- 172.21.2.117:0/1256816543 shutdown_connections 2021-02-10T11:39:02.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.160 7f9edfc2c700 1 -- 172.21.2.117:0/1256816543 wait complete. 2021-02-10T11:39:02.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 Processor -- start 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 -- start start 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed81b5b50 0x7f9ed80b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 0x7f9ed80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80b8da0 con 0x7f9ed81b5130 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80a4140 con 0x7f9ed81b5f50 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edfc2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80a4320 con 0x7f9ed81b5b50 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edda48700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edda48700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46526/0 (socket says 172.21.2.117:46526) 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9edda48700 1 -- 172.21.2.117:0/3915642170 learned_addr learned my addr 172.21.2.117:0/3915642170 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:02.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.175 7f9ede249700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 0x7f9ed80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edd247700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed81b5b50 0x7f9ed80b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 -- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed81b5b50 msgr2=0x7f9ed80b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed81b5b50 0x7f9ed80b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 -- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 msgr2=0x7f9ed80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 0x7f9ed80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 -- 172.21.2.117:0/3915642170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed80b62b0 con 0x7f9ed81b5130 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9ede249700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 0x7f9ed80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edda48700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f9ecc00b5e0 tx=0x7f9ecc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ecc00bd70 con 0x7f9ed81b5130 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc002030 con 0x7f9ed81b5130 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed80b62b0 con 0x7f9ed81b5130 2021-02-10T11:39:02.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.176 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ecc002550 con 0x7f9ed81b5130 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.177 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed80b62b0 con 0x7f9ed81b5130 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.177 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ecc000c90 con 0x7f9ed81b5130 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.177 7f9eca7fc700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 0x7f9ec001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.177 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9ecc032800 con 0x7f9ed81b5130 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.177 7f9edd247700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 0x7f9ec001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.178 7f9edd247700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 0x7f9ec001ac10 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.208 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed81b4f00 con 0x7f9ed81b5130 2021-02-10T11:39:02.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.210 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ecc01f570 con 0x7f9ed81b5130 2021-02-10T11:39:02.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.335 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802482", "target": ["mgr", ""]}) v1 -- 0x7f9ed8077a70 con 0x7f9ec0018730 2021-02-10T11:39:02.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.346 7f9eca7fc700 1 -- 172.21.2.117:0/3915642170 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9ed4006a90 con 0x7f9ec0018730 2021-02-10T11:39:02.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 msgr2=0x7f9ec001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 0x7f9ec001ac10 crc :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 msgr2=0x7f9ed81b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0x7f9ecc00b5e0 tx=0x7f9ecc00b980).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 shutdown_connections 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ed81b5f50 0x7f9ed80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ed81b5b50 0x7f9ed80b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ec0018730 0x7f9ec001ac10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 --2- 172.21.2.117:0/3915642170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ed81b5130 0x7f9ed81b5530 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 >> 172.21.2.117:0/3915642170 conn(0x7f9ed808bbc0 msgr2=0x7f9ed80c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 shutdown_connections 2021-02-10T11:39:02.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.353 7f9edfc2c700 1 -- 172.21.2.117:0/3915642170 wait complete. 2021-02-10T11:39:02.344 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802483 2021-02-10T11:39:02.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 Processor -- start 2021-02-10T11:39:02.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 -- start start 2021-02-10T11:39:02.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17b007da20 0x7f17b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00e51a0 0x7f17b0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b0077a70 con 0x7f17b00e4da0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b00777d0 con 0x7f17b007da20 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.777 7f17b4db3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b01b08e0 con 0x7f17b00e51a0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17a6312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17b00e4da0 0x7f17b01b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46786/0 (socket says 172.21.2.117:46786) 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17a6312700 1 -- 172.21.2.117:0/1336397372 learned_addr learned my addr 172.21.2.117:0/1336397372 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17aeb13700 1 --2- 172.21.2.117:0/1336397372 >> v2:172.21.2.125:3300/0 conn(0x7f17b007da20 0x7f17b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00e51a0 0x7f17b0079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42067706 0 0) 0x7f17980009b0 con 0x7f17b00e4da0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179c002fe0 con 0x7f17b00e4da0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2606420583 0 0) 0x7f1798001010 con 0x7f17b00e4da0 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00e51a0 msgr2=0x7f17b0079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00e51a0 0x7f17b0079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 >> v2:172.21.2.125:3300/0 conn(0x7f17b007da20 msgr2=0x7f17b00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 --2- 172.21.2.117:0/1336397372 >> v2:172.21.2.125:3300/0 conn(0x7f17b007da20 0x7f17b00771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b007de20 con 0x7f17b00e4da0 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1798001530 con 0x7f17b00e4da0 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 >> v1:172.21.2.117:6789/0 conn(0x7f17b00e4da0 legacy=0x7f17b01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f179c003ae0 0x7f179c005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f179c04a800 0x7f179c04cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f179c04d5a0 0x7f179c04fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c050370 con 0x7f179c04d5a0 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c050590 con 0x7f179c04a800 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17ae312700 1 -- 172.21.2.117:0/1336397372 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c0507b0 con 0x7f179c003ae0 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17aeb13700 1 --2- 172.21.2.117:0/1336397372 >> v2:172.21.2.125:3300/0 conn(0x7f17b007da20 0x7f17b00771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f179c04d5a0 0x7f179c04fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17a6312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f179c04a800 0x7f179c04cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17aeb13700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f179c003ae0 0x7f179c005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 -- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f179c003ae0 msgr2=0x7f179c005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f179c003ae0 0x7f179c005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 -- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f179c04a800 msgr2=0x7f179c04cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f179c04a800 0x7f179c04cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17af314700 1 -- 172.21.2.117:0/1336397372 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c050a50 con 0x7f179c04d5a0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17a6312700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f179c04a800 0x7f179c04cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.778 7f17aeb13700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f179c003ae0 0x7f179c005f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17af314700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f179c04d5a0 0x7f179c04fac0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f17a8006af0 tx=0x7f17a8009ad0).ready entity=mon.0 client_cookie=e6870127b8a14a84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17a80022e0 con 0x7f179c04d5a0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a80066c0 con 0x7f179c04d5a0 2021-02-10T11:39:02.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17ae312700 1 -- 172.21.2.117:0/1336397372 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17a8002810 con 0x7f179c04d5a0 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 -- 172.21.2.117:0/1336397372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f179c04d5a0 msgr2=0x7f179c04fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 --2- 172.21.2.117:0/1336397372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f179c04d5a0 0x7f179c04fac0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f17a8006af0 tx=0x7f17a8009ad0).stop 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17af314700 1 -- 172.21.2.117:0/1336397372 reap_dead start 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 -- 172.21.2.117:0/1336397372 shutdown_connections 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 -- 172.21.2.117:0/1336397372 >> 172.21.2.117:0/1336397372 conn(0x7f17b00ce450 msgr2=0x7f17b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 -- 172.21.2.117:0/1336397372 shutdown_connections 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.779 7f17b4db3700 1 -- 172.21.2.117:0/1336397372 wait complete. 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.794 7f17b4db3700 1 Processor -- start 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.794 7f17b4db3700 1 -- start start 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17b00a47d0 0x7f17b00a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00b77d0 0x7f17b00b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b0189970 con 0x7f17b00a51f0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b0190aa0 con 0x7f17b00a47d0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17b4db3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00b81f0 con 0x7f17b00b77d0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46546/0 (socket says 172.21.2.117:46546) 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 -- 172.21.2.117:0/2828286954 learned_addr learned my addr 172.21.2.117:0/2828286954 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17aeb13700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17b00a47d0 0x7f17b00a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17af314700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00b77d0 0x7f17b00b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 -- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00b77d0 msgr2=0x7f17b00b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00b77d0 0x7f17b00b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 -- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17b00a47d0 msgr2=0x7f17b00a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17b00a47d0 0x7f17b00a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 -- 172.21.2.117:0/2828286954 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b00b8410 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a6312700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f17980012e0 tx=0x7f179800a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.795 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f179800ad70 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1798004350 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b00b8410 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1798004870 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b00b89a0 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1798004530 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17a7fff700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 0x7f179401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1798032770 con 0x7f17b00a51f0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.796 7f17aeb13700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 0x7f179401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:02.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.797 7f17aeb13700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 0x7f179401abc0 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:02.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.825 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b0077a70 con 0x7f17b00a51f0 2021-02-10T11:39:02.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.828 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17980018f0 con 0x7f17b00a51f0 2021-02-10T11:39:02.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.958 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802483", "target": ["mgr", ""]}) v1 -- 0x7f17b00b81f0 con 0x7f17940186c0 2021-02-10T11:39:02.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.969 7f17a7fff700 1 -- 172.21.2.117:0/2828286954 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f17a0005660 con 0x7f17940186c0 2021-02-10T11:39:02.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 msgr2=0x7f179401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 0x7f179401abc0 crc :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 msgr2=0x7f17b00a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 rx=0x7f17980012e0 tx=0x7f179800a980).stop 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 shutdown_connections 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17940186c0 0x7f179401abc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17b00a51f0 0x7f17b00a55f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17b00b77d0 0x7f17b00b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 --2- 172.21.2.117:0/2828286954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17b00a47d0 0x7f17b00a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:02.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.975 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 >> 172.21.2.117:0/2828286954 conn(0x7f17b00b5140 msgr2=0x7f17b00a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:02.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.976 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 shutdown_connections 2021-02-10T11:39:02.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:02.976 7f17b4db3700 1 -- 172.21.2.117:0/2828286954 wait complete. 2021-02-10T11:39:02.965 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802484 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.396 7f7bddd5d700 1 Processor -- start 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.396 7f7bddd5d700 1 -- start start 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bddd5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 0x7f7bd80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bddd5d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7bd80e4da0 0x7f7bd81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bddd5d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bd8077a70 con 0x7f7bd80e51a0 2021-02-10T11:39:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bddd5d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bd80e5c20 con 0x7f7bd80e4da0 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bddd5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd80777d0 con 0x7f7bd807da20 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7bd80e51a0 0x7f7bd8077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46806/0 (socket says 172.21.2.117:46806) 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd7fff700 1 -- 172.21.2.117:0/2421346751 learned_addr learned my addr 172.21.2.117:0/2421346751 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd77fe700 1 --2- 172.21.2.117:0/2421346751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 0x7f7bd80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd6ffd700 1 --2- 172.21.2.117:0/2421346751 >> v2:172.21.2.125:3300/0 conn(0x7f7bd80e4da0 0x7f7bd81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd67fc700 1 -- 172.21.2.117:0/2421346751 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019575774 0 0) 0x7f7bd0000f90 con 0x7f7bd80e51a0 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd67fc700 1 -- 172.21.2.117:0/2421346751 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc0002fe0 con 0x7f7bd80e51a0 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd77fe700 1 -- 172.21.2.117:0/2421346751 >> v2:172.21.2.125:3300/0 conn(0x7f7bd80e4da0 msgr2=0x7f7bd81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd77fe700 1 --2- 172.21.2.117:0/2421346751 >> v2:172.21.2.125:3300/0 conn(0x7f7bd80e4da0 0x7f7bd81b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd77fe700 1 -- 172.21.2.117:0/2421346751 >> v1:172.21.2.117:6789/0 conn(0x7f7bd80e51a0 legacy=0x7f7bd8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.397 7f7bd77fe700 1 -- 172.21.2.117:0/2421346751 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd807de20 con 0x7f7bd807da20 2021-02-10T11:39:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bd77fe700 1 --2- 172.21.2.117:0/2421346751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 0x7f7bd80771b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7bc8006650 tx=0x7f7bc8009a80).ready entity=mon.2 client_cookie=48f1bc664949f8e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bd67fc700 1 -- 172.21.2.117:0/2421346751 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bc80009b0 con 0x7f7bd807da20 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bd67fc700 1 -- 172.21.2.117:0/2421346751 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc800c0c0 con 0x7f7bd807da20 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bd67fc700 1 -- 172.21.2.117:0/2421346751 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bc800c510 con 0x7f7bd807da20 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 -- 172.21.2.117:0/2421346751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 msgr2=0x7f7bd80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 --2- 172.21.2.117:0/2421346751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 0x7f7bd80771b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7bc8006650 tx=0x7f7bc8009a80).stop 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 -- 172.21.2.117:0/2421346751 shutdown_connections 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 --2- 172.21.2.117:0/2421346751 >> v2:172.21.2.125:3300/0 conn(0x7f7bd80e4da0 0x7f7bd81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 --2- 172.21.2.117:0/2421346751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd807da20 0x7f7bd80771b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 -- 172.21.2.117:0/2421346751 >> 172.21.2.117:0/2421346751 conn(0x7f7bd80ce450 msgr2=0x7f7bd80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 -- 172.21.2.117:0/2421346751 shutdown_connections 2021-02-10T11:39:03.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.398 7f7bddd5d700 1 -- 172.21.2.117:0/2421346751 wait complete. 2021-02-10T11:39:03.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.413 7f7bddd5d700 1 Processor -- start 2021-02-10T11:39:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.413 7f7bddd5d700 1 -- start start 2021-02-10T11:39:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.413 7f7bddd5d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 0x7f7bd8106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.413 7f7bddd5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 0x7f7bd8188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.413 7f7bddd5d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bddd5d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81067f0 con 0x7f7bd8090420 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bddd5d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd8107350 con 0x7f7bd81a0250 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bddd5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd81042d0 con 0x7f7bd8187e20 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46562/0 (socket says 172.21.2.117:46562) 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 -- 172.21.2.117:0/4055437551 learned_addr learned my addr 172.21.2.117:0/4055437551 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd77fe700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 0x7f7bd8106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd6ffd700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 0x7f7bd8188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 -- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 msgr2=0x7f7bd8188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 0x7f7bd8188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 -- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 msgr2=0x7f7bd8106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 0x7f7bd8106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 -- 172.21.2.117:0/4055437551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8090e40 con 0x7f7bd8090420 2021-02-10T11:39:03.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd6ffd700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 0x7f7bd8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd77fe700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 0x7f7bd8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bd7fff700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f7bd0009980 tx=0x7f7bd000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bd000be10 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd0006080 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7bd00064e0 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.414 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd8090e40 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.415 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd8090e40 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.415 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7bd0006260 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.415 7f7bbffff700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 0x7f7bb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.415 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7bd0032c80 con 0x7f7bd8090420 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.415 7f7bd77fe700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 0x7f7bb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:03.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.416 7f7bd77fe700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 0x7f7bb801abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:03.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.446 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd8103280 con 0x7f7bd8090420 2021-02-10T11:39:03.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.449 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7bd0006290 con 0x7f7bd8090420 2021-02-10T11:39:03.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.575 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802484", "target": ["mgr", ""]}) v1 -- 0x7f7bd8103280 con 0x7f7bb80186c0 2021-02-10T11:39:03.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.585 7f7bbffff700 1 -- 172.21.2.117:0/4055437551 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7bc8000c90 con 0x7f7bb80186c0 2021-02-10T11:39:03.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 msgr2=0x7f7bb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 0x7f7bb801abc0 crc :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 msgr2=0x7f7bd8090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0x7f7bd0009980 tx=0x7f7bd000bcd0).stop 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 shutdown_connections 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.592 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7bb80186c0 0x7f7bb801abc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7bd8090420 0x7f7bd8090820 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7bd8187e20 0x7f7bd8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 --2- 172.21.2.117:0/4055437551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7bd81a0250 0x7f7bd8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:03.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 >> 172.21.2.117:0/4055437551 conn(0x7f7bd80a1850 msgr2=0x7f7bd80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:03.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 shutdown_connections 2021-02-10T11:39:03.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:03.593 7f7bddd5d700 1 -- 172.21.2.117:0/4055437551 wait complete. 2021-02-10T11:39:03.582 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802485 2021-02-10T11:39:04.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 Processor -- start 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 -- start start 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f527807da20 0x7f52780771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52780d1d20 con 0x7f52780e51a0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5278077a70 con 0x7f52780e4da0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.016 7f52804c2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780777d0 con 0x7f527807da20 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527eadf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f52780e51a0 0x7f52780d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46816/0 (socket says 172.21.2.117:46816) 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527eadf700 1 -- 172.21.2.117:0/1462054495 learned_addr learned my addr 172.21.2.117:0/1462054495 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527e2de700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f527807da20 0x7f52780771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270761829 0 0) 0x7f5270001500 con 0x7f52780e51a0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5264002fe0 con 0x7f52780e51a0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3638183224 0 0) 0x7f5270001270 con 0x7f52780e51a0 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f527807da20 msgr2=0x7f52780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:04.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f527807da20 0x7f52780771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 >> v1:172.21.2.125:6789/0 conn(0x7f52780e4da0 legacy=0x7f52781b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52781b08e0 con 0x7f52780e51a0 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5270001b50 con 0x7f52780e51a0 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 >> v1:172.21.2.117:6789/0 conn(0x7f52780e51a0 legacy=0x7f52780d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5264003a70 0x7f5264005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5264006780 0x7f5264008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52640095a0 0x7f526400ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526400c2e0 con 0x7f5264003a70 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526400c500 con 0x7f5264006780 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.017 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f526400c720 con 0x7f52640095a0 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5264003a70 0x7f5264005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527eadf700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52640095a0 0x7f526400ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527dadd700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5264006780 0x7f5264008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 -- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52640095a0 msgr2=0x7f526400ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52640095a0 0x7f526400ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 -- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5264006780 msgr2=0x7f5264008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5264006780 0x7f5264008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 -- 172.21.2.117:0/1462054495 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5264050990 con 0x7f5264003a70 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527dadd700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5264006780 0x7f5264008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527e2de700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5264003a70 0x7f5264005f00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f5268006ac0 tx=0x7f5268009ab0).ready entity=mon.0 client_cookie=13f360444e9d376b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f526800c0c0 con 0x7f5264003a70 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526800c380 con 0x7f5264003a70 2021-02-10T11:39:04.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527d2dc700 1 -- 172.21.2.117:0/1462054495 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f526800c8a0 con 0x7f5264003a70 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f52804c2700 1 -- 172.21.2.117:0/1462054495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5264003a70 msgr2=0x7f5264005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f52804c2700 1 --2- 172.21.2.117:0/1462054495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5264003a70 0x7f5264005f00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f5268006ac0 tx=0x7f5268009ab0).stop 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.018 7f527eadf700 1 -- 172.21.2.117:0/1462054495 reap_dead start 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.019 7f52804c2700 1 -- 172.21.2.117:0/1462054495 shutdown_connections 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.019 7f52804c2700 1 -- 172.21.2.117:0/1462054495 >> 172.21.2.117:0/1462054495 conn(0x7f52780ce450 msgr2=0x7f52780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.019 7f52804c2700 1 -- 172.21.2.117:0/1462054495 shutdown_connections 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.019 7f52804c2700 1 -- 172.21.2.117:0/1462054495 wait complete. 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.033 7f52804c2700 1 Processor -- start 2021-02-10T11:39:04.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 -- start start 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52780a47d0 0x7f52780a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278189970 con 0x7f52780b77d0 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5278190aa0 con 0x7f52780a47d0 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f52804c2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52780b81f0 con 0x7f52780a51f0 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527eadf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527dadd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527dadd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44256/0 (socket says 172.21.2.117:44256) 2021-02-10T11:39:04.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527eadf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46580/0 (socket says 172.21.2.117:46580) 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527dadd700 1 -- 172.21.2.117:0/2636121364 learned_addr learned my addr 172.21.2.117:0/2636121364 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.034 7f527e2de700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52780a47d0 0x7f52780a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 -- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 msgr2=0x7f52780a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 -- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52780a47d0 msgr2=0x7f52780a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52780a47d0 0x7f52780a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 -- 172.21.2.117:0/2636121364 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52780b8410 con 0x7f52780b77d0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527dadd700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f527eadf700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f5270001960 tx=0x7f527000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5270001270 con 0x7f52780b77d0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270000f50 con 0x7f52780b77d0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f52804c2700 1 -- 172.21.2.117:0/2636121364 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52780b8410 con 0x7f52780b77d0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52700074f0 con 0x7f52780b77d0 2021-02-10T11:39:04.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.035 7f52804c2700 1 -- 172.21.2.117:0/2636121364 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52780b89a0 con 0x7f52780b77d0 2021-02-10T11:39:04.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.036 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5270004720 con 0x7f52780b77d0 2021-02-10T11:39:04.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.036 7f526effd700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 0x7f526001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.036 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5270032ba0 con 0x7f52780b77d0 2021-02-10T11:39:04.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.036 7f527e2de700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 0x7f526001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.036 7f527e2de700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 0x7f526001abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.066 7f52804c2700 1 -- 172.21.2.117:0/2636121364 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5278077a70 con 0x7f52780b77d0 2021-02-10T11:39:04.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.068 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5270002040 con 0x7f52780b77d0 2021-02-10T11:39:04.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.194 7f52804c2700 1 -- 172.21.2.117:0/2636121364 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802485", "target": ["mgr", ""]}) v1 -- 0x7f5278189970 con 0x7f5260018730 2021-02-10T11:39:04.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.206 7f526effd700 1 -- 172.21.2.117:0/2636121364 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5268006d60 con 0x7f5260018730 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 msgr2=0x7f526001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 0x7f526001abc0 crc :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 msgr2=0x7f52780b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f5270001960 tx=0x7f527000abd0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 shutdown_connections 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5260018730 0x7f526001abc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52780b77d0 0x7f52780b7bd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52780a51f0 0x7f52780a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 --2- 172.21.2.117:0/2636121364 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52780a47d0 0x7f52780a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 >> 172.21.2.117:0/2636121364 conn(0x7f52780b5140 msgr2=0x7f52780a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 shutdown_connections 2021-02-10T11:39:04.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.212 7f52804c2700 1 -- 172.21.2.117:0/2636121364 wait complete. 2021-02-10T11:39:04.201 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802486 2021-02-10T11:39:04.535 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:04.538 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.635 7f015b806700 1 Processor -- start 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 -- start start 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0154079000 0x7f01540e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01541b00f0 0x7f01541b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0154077a70 con 0x7f015407da20 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540e5c20 con 0x7f0154079000 2021-02-10T11:39:04.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f015b806700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540e53c0 con 0x7f01541b00f0 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f0158e21700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0154079000 0x7f01540e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f0159622700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f015407da20 0x7f01540771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46834/0 (socket says 172.21.2.117:46834) 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f0159622700 1 -- 172.21.2.117:0/1945485155 learned_addr learned my addr 172.21.2.117:0/1945485155 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.636 7f0159e23700 1 --2- 172.21.2.117:0/1945485155 >> v2:172.21.2.125:3300/0 conn(0x7f01541b00f0 0x7f01541b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0147fff700 1 -- 172.21.2.117:0/1945485155 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619847669 0 0) 0x7f0148000d30 con 0x7f015407da20 2021-02-10T11:39:04.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0147fff700 1 -- 172.21.2.117:0/1945485155 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0140002fe0 con 0x7f015407da20 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0158e21700 1 -- 172.21.2.117:0/1945485155 >> v2:172.21.2.125:3300/0 conn(0x7f01541b00f0 msgr2=0x7f01541b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0158e21700 1 --2- 172.21.2.117:0/1945485155 >> v2:172.21.2.125:3300/0 conn(0x7f01541b00f0 0x7f01541b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0158e21700 1 -- 172.21.2.117:0/1945485155 >> v1:172.21.2.117:6789/0 conn(0x7f015407da20 legacy=0x7f01540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0158e21700 1 -- 172.21.2.117:0/1945485155 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154079400 con 0x7f0154079000 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0158e21700 1 --2- 172.21.2.117:0/1945485155 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0154079000 0x7f01540e4da0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f0150006650 tx=0x7f0150009a80).ready entity=mon.0 client_cookie=957bd9c3963e61e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0147fff700 1 -- 172.21.2.117:0/1945485155 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01500009b0 con 0x7f0154079000 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0147fff700 1 -- 172.21.2.117:0/1945485155 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000c0c0 con 0x7f0154079000 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f0147fff700 1 -- 172.21.2.117:0/1945485155 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c5e0 con 0x7f0154079000 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f015b806700 1 -- 172.21.2.117:0/1945485155 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0154079000 msgr2=0x7f01540e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.637 7f015b806700 1 --2- 172.21.2.117:0/1945485155 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0154079000 0x7f01540e4da0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7f0150006650 tx=0x7f0150009a80).stop 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 -- 172.21.2.117:0/1945485155 shutdown_connections 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 --2- 172.21.2.117:0/1945485155 >> v2:172.21.2.125:3300/0 conn(0x7f01541b00f0 0x7f01541b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 --2- 172.21.2.117:0/1945485155 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0154079000 0x7f01540e4da0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 -- 172.21.2.117:0/1945485155 >> 172.21.2.117:0/1945485155 conn(0x7f01540ce450 msgr2=0x7f01540e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 -- 172.21.2.117:0/1945485155 shutdown_connections 2021-02-10T11:39:04.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.638 7f015b806700 1 -- 172.21.2.117:0/1945485155 wait complete. 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 Processor -- start 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 -- start start 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 0x7f0154106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 0x7f0154188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0154090420 0x7f0154090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01541067f0 con 0x7f01541a0250 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0154107350 con 0x7f0154090420 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f015b806700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01541042d0 con 0x7f0154187e20 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f0159622700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 0x7f0154106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f0159622700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 0x7f0154106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46588/0 (socket says 172.21.2.117:46588) 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f0159622700 1 -- 172.21.2.117:0/996920481 learned_addr learned my addr 172.21.2.117:0/996920481 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.653 7f0158e21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 0x7f0154188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0159e23700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0154090420 0x7f0154090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 -- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0154090420 msgr2=0x7f0154090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0154090420 0x7f0154090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 -- 172.21.2.117:0/996920481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 msgr2=0x7f0154106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 0x7f0154106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 -- 172.21.2.117:0/996920481 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154090e40 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0158e21700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 0x7f0154188220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f015000b530 tx=0x7f015000b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0150000a10 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000bcc0 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f015b806700 1 -- 172.21.2.117:0/996920481 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154090e40 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f015b806700 1 -- 172.21.2.117:0/996920481 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0154090e40 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.654 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0150000a10 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.655 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0150009280 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.655 7f0145ffb700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 0x7f013401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.655 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0150032870 con 0x7f0154187e20 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.655 7f0159622700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 0x7f013401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.656 7f0159622700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 0x7f013401ac10 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:04.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.685 7f015b806700 1 -- 172.21.2.117:0/996920481 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0154103280 con 0x7f0154187e20 2021-02-10T11:39:04.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.689 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0150000c90 con 0x7f0154187e20 2021-02-10T11:39:04.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.821 7f015b806700 1 -- 172.21.2.117:0/996920481 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802486", "target": ["mgr", ""]}) v1 -- 0x7f0154103280 con 0x7f0134018730 2021-02-10T11:39:04.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.830 7f0145ffb700 1 -- 172.21.2.117:0/996920481 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0148000d30 con 0x7f0134018730 2021-02-10T11:39:04.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.838 7f015b806700 1 -- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 msgr2=0x7f013401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.838 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 0x7f013401ac10 crc :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.838 7f015b806700 1 -- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 msgr2=0x7f0154188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:04.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.838 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 0x7f0154188220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f015000b530 tx=0x7f015000b8d0).stop 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 -- 172.21.2.117:0/996920481 shutdown_connections 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0154090420 0x7f0154090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0154187e20 0x7f0154188220 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0134018730 0x7f013401ac10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 --2- 172.21.2.117:0/996920481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01541a0250 0x7f0154106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 -- 172.21.2.117:0/996920481 >> 172.21.2.117:0/996920481 conn(0x7f01540a1850 msgr2=0x7f01540c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 -- 172.21.2.117:0/996920481 shutdown_connections 2021-02-10T11:39:04.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:04.839 7f015b806700 1 -- 172.21.2.117:0/996920481 wait complete. 2021-02-10T11:39:04.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802487 2021-02-10T11:39:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.260 7fdd85989700 1 Processor -- start 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.260 7fdd85989700 1 -- start start 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd85989700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd8007da20 0x7fdd800771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd85989700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdd800e4da0 0x7fdd801b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd85989700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd80077a70 con 0x7fdd800e51a0 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd85989700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd800e5c20 con 0x7fdd800e4da0 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd85989700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800777d0 con 0x7fdd8007da20 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7fd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdd800e51a0 0x7fdd80077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46848/0 (socket says 172.21.2.117:46848) 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7fd75700 1 -- 172.21.2.117:0/1313792410 learned_addr learned my addr 172.21.2.117:0/1313792410 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 --2- 172.21.2.117:0/1313792410 >> v2:172.21.2.117:3300/0 conn(0x7fdd800e4da0 0x7fdd801b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7f574700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd8007da20 0x7fdd800771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7e572700 1 -- 172.21.2.117:0/1313792410 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609734771 0 0) 0x7fdd78001500 con 0x7fdd800e51a0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7e572700 1 -- 172.21.2.117:0/1313792410 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd68002fe0 con 0x7fdd800e51a0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 -- 172.21.2.117:0/1313792410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd8007da20 msgr2=0x7fdd800771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd8007da20 0x7fdd800771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 -- 172.21.2.117:0/1313792410 >> v1:172.21.2.117:6789/0 conn(0x7fdd800e51a0 legacy=0x7fdd80077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 -- 172.21.2.117:0/1313792410 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8007de20 con 0x7fdd800e4da0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7ed73700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd800e4da0 0x7fdd801b00f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fdd74006650 tx=0x7fdd74009a80).ready entity=mon.0 client_cookie=8e630fee59cc040a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7e572700 1 -- 172.21.2.117:0/1313792410 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd7400c0c0 con 0x7fdd800e4da0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.261 7fdd7e572700 1 -- 172.21.2.117:0/1313792410 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7400c340 con 0x7fdd800e4da0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd7e572700 1 -- 172.21.2.117:0/1313792410 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd7400c860 con 0x7fdd800e4da0 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 -- 172.21.2.117:0/1313792410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd800e4da0 msgr2=0x7fdd801b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd800e4da0 0x7fdd801b00f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fdd74006650 tx=0x7fdd74009a80).stop 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 -- 172.21.2.117:0/1313792410 shutdown_connections 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd800e4da0 0x7fdd801b00f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 --2- 172.21.2.117:0/1313792410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd8007da20 0x7fdd800771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 -- 172.21.2.117:0/1313792410 >> 172.21.2.117:0/1313792410 conn(0x7fdd800ce450 msgr2=0x7fdd800e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 -- 172.21.2.117:0/1313792410 shutdown_connections 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.262 7fdd85989700 1 -- 172.21.2.117:0/1313792410 wait complete. 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 Processor -- start 2021-02-10T11:39:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 -- start start 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd8008eb10 0x7fdd800c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd800afac0 0x7fdd800afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800af540 con 0x7fdd8008dfe0 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800a0c80 con 0x7fdd8008eb10 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd85989700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd800b04e0 con 0x7fdd800afac0 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd7ed73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd7ed73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46602/0 (socket says 172.21.2.117:46602) 2021-02-10T11:39:05.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd7ed73700 1 -- 172.21.2.117:0/1550069302 learned_addr learned my addr 172.21.2.117:0/1550069302 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd7fd75700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd800afac0 0x7fdd800afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.277 7fdd7f574700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd8008eb10 0x7fdd800c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 -- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd800afac0 msgr2=0x7fdd800afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd800afac0 0x7fdd800afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 -- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd8008eb10 msgr2=0x7fdd800c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd8008eb10 0x7fdd800c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 -- 172.21.2.117:0/1550069302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd800b0700 con 0x7fdd8008dfe0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd7ed73700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fdd74009630 tx=0x7fdd7400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd740093a0 con 0x7fdd8008dfe0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7400bd70 con 0x7fdd8008dfe0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd740054d0 con 0x7fdd8008dfe0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd85989700 1 -- 172.21.2.117:0/1550069302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd800b0700 con 0x7fdd8008dfe0 2021-02-10T11:39:05.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.278 7fdd85989700 1 -- 172.21.2.117:0/1550069302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd800b0700 con 0x7fdd8008dfe0 2021-02-10T11:39:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.279 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd7400a000 con 0x7fdd8008dfe0 2021-02-10T11:39:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.279 7fdd67fff700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 0x7fdd6001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.279 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdd74032770 con 0x7fdd8008dfe0 2021-02-10T11:39:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.279 7fdd7f574700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 0x7fdd6001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.280 7fdd7f574700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 0x7fdd6001ac10 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.310 7fdd85989700 1 -- 172.21.2.117:0/1550069302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd80103280 con 0x7fdd8008dfe0 2021-02-10T11:39:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.313 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd7401f4e0 con 0x7fdd8008dfe0 2021-02-10T11:39:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.440 7fdd85989700 1 -- 172.21.2.117:0/1550069302 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802487", "target": ["mgr", ""]}) v1 -- 0x7fdd80103280 con 0x7fdd60018730 2021-02-10T11:39:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.452 7fdd67fff700 1 -- 172.21.2.117:0/1550069302 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdd70006a70 con 0x7fdd60018730 2021-02-10T11:39:05.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.457 7fdd85989700 1 -- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 msgr2=0x7fdd6001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.457 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 0x7fdd6001ac10 crc :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.457 7fdd85989700 1 -- 172.21.2.117:0/1550069302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 msgr2=0x7fdd8008e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.457 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0x7fdd74009630 tx=0x7fdd7400b980).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 -- 172.21.2.117:0/1550069302 shutdown_connections 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd60018730 0x7fdd6001ac10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd8008dfe0 0x7fdd8008e3e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd800afac0 0x7fdd800afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 --2- 172.21.2.117:0/1550069302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd8008eb10 0x7fdd800c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 -- 172.21.2.117:0/1550069302 >> 172.21.2.117:0/1550069302 conn(0x7fdd800a1850 msgr2=0x7fdd800bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 -- 172.21.2.117:0/1550069302 shutdown_connections 2021-02-10T11:39:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.458 7fdd85989700 1 -- 172.21.2.117:0/1550069302 wait complete. 2021-02-10T11:39:05.451 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802488 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 Processor -- start 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 -- start start 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 0x7f5b140d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b14077a70 con 0x7f5b14079000 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b14079400 con 0x7f5b1407da20 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.884 7f5b1a2de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1407de20 con 0x7f5b140d1510 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b137fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5b14079000 0x7f5b140e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46860/0 (socket says 172.21.2.117:46860) 2021-02-10T11:39:05.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b137fe700 1 -- 172.21.2.117:0/674481611 learned_addr learned my addr 172.21.2.117:0/674481611 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b188fb700 1 --2- 172.21.2.117:0/674481611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 0x7f5b140d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207930801 0 0) 0x7f5b04001180 con 0x7f5b14079000 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b00002fe0 con 0x7f5b14079000 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557201001 0 0) 0x7f5afc000ff0 con 0x7f5b1407da20 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b00004370 con 0x7f5b1407da20 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b188fb700 1 -- 172.21.2.117:0/674481611 >> v1:172.21.2.125:6789/0 conn(0x7f5b1407da20 legacy=0x7f5b140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b188fb700 1 -- 172.21.2.117:0/674481611 >> v1:172.21.2.117:6789/0 conn(0x7f5b14079000 legacy=0x7f5b140e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b188fb700 1 -- 172.21.2.117:0/674481611 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b141b00d0 con 0x7f5b140d1510 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b188fb700 1 --2- 172.21.2.117:0/674481611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 0x7f5b140d1910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5b0c006e50 tx=0x7f5b0c009df0).ready entity=mon.2 client_cookie=f82f3c93e9ab0f7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0c00b510 con 0x7f5b140d1510 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0c003030 con 0x7f5b140d1510 2021-02-10T11:39:05.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b12ffd700 1 -- 172.21.2.117:0/674481611 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b0c000f90 con 0x7f5b140d1510 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b1a2de700 1 -- 172.21.2.117:0/674481611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 msgr2=0x7f5b140d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.885 7f5b1a2de700 1 --2- 172.21.2.117:0/674481611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 0x7f5b140d1910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f5b0c006e50 tx=0x7f5b0c009df0).stop 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.886 7f5b1a2de700 1 -- 172.21.2.117:0/674481611 shutdown_connections 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.886 7f5b1a2de700 1 --2- 172.21.2.117:0/674481611 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b140d1510 0x7f5b140d1910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.886 7f5b1a2de700 1 -- 172.21.2.117:0/674481611 >> 172.21.2.117:0/674481611 conn(0x7f5b140ce430 msgr2=0x7f5b140e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.886 7f5b1a2de700 1 -- 172.21.2.117:0/674481611 shutdown_connections 2021-02-10T11:39:05.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.886 7f5b1a2de700 1 -- 172.21.2.117:0/674481611 wait complete. 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.901 7f5b1a2de700 1 Processor -- start 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.901 7f5b1a2de700 1 -- start start 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b14091b10 0x7f5b1419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b140adb60 0x7f5b140adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140ad040 con 0x7f5b1419db50 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b14091f10 con 0x7f5b140adb60 2021-02-10T11:39:05.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b1a2de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b140ae580 con 0x7f5b14091b10 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46616/0 (socket says 172.21.2.117:46616) 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 -- 172.21.2.117:0/512550126 learned_addr learned my addr 172.21.2.117:0/512550126 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b188fb700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b140adb60 0x7f5b140adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b13fff700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b14091b10 0x7f5b1419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 -- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b14091b10 msgr2=0x7f5b1419d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b14091b10 0x7f5b1419d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 -- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b140adb60 msgr2=0x7f5b140adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b140adb60 0x7f5b140adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 -- 172.21.2.117:0/512550126 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b140ae7a0 con 0x7f5b1419db50 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.902 7f5b137fe700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f5b0400a5d0 tx=0x7f5b0400a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b04000ce0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b040043a0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b140ae7a0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b140abbb0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b040048c0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b04000ce0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 0x7f5af801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5b040328e0 con 0x7f5b1419db50 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.903 7f5b13fff700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 0x7f5af801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:05.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.904 7f5b13fff700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 0x7f5af801ac10 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:05.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.933 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec002fd0 con 0x7f5b1419db50 2021-02-10T11:39:05.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:05.936 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b0401f4d0 con 0x7f5b1419db50 2021-02-10T11:39:06.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.064 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802488", "target": ["mgr", ""]}) v1 -- 0x7f5aec002f80 con 0x7f5af8018730 2021-02-10T11:39:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.074 7f5b10ff9700 1 -- 172.21.2.117:0/512550126 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5afc0009f0 con 0x7f5af8018730 2021-02-10T11:39:06.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 msgr2=0x7f5af801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 0x7f5af801ac10 crc :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 msgr2=0x7f5b140ad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f5b0400a5d0 tx=0x7f5b0400a970).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 shutdown_connections 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5af8018730 0x7f5af801ac10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b1419db50 0x7f5b140ad540 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b140adb60 0x7f5b140adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 --2- 172.21.2.117:0/512550126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b14091b10 0x7f5b1419d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.081 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 >> 172.21.2.117:0/512550126 conn(0x7f5b141b9670 msgr2=0x7f5b14107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.082 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 shutdown_connections 2021-02-10T11:39:06.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.082 7f5b1a2de700 1 -- 172.21.2.117:0/512550126 wait complete. 2021-02-10T11:39:06.071 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802489 2021-02-10T11:39:06.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 Processor -- start 2021-02-10T11:39:06.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 -- start start 2021-02-10T11:39:06.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 0x7fef200d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef20077a70 con 0x7fef20079000 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef20079400 con 0x7fef2007da20 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.504 7fef25ee4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2007de20 con 0x7fef200d1510 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1effd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fef20079000 0x7fef200e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46874/0 (socket says 172.21.2.117:46874) 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 0x7fef200d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1effd700 1 -- 172.21.2.117:0/4073922142 learned_addr learned my addr 172.21.2.117:0/4073922142 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1e7fc700 1 -- 172.21.2.117:0/4073922142 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394792133 0 0) 0x7fef080010b0 con 0x7fef20079000 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1ffff700 1 -- 172.21.2.117:0/4073922142 >> v1:172.21.2.125:6789/0 conn(0x7fef2007da20 legacy=0x7fef200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1ffff700 1 -- 172.21.2.117:0/4073922142 >> v1:172.21.2.117:6789/0 conn(0x7fef20079000 legacy=0x7fef200e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1ffff700 1 -- 172.21.2.117:0/4073922142 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef201b00d0 con 0x7fef200d1510 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1ffff700 1 --2- 172.21.2.117:0/4073922142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 0x7fef200d1910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fef18006e50 tx=0x7fef18009e70).ready entity=mon.2 client_cookie=9afb873a82ea70ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1e7fc700 1 -- 172.21.2.117:0/4073922142 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1800b510 con 0x7fef200d1510 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1e7fc700 1 -- 172.21.2.117:0/4073922142 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18006790 con 0x7fef200d1510 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.505 7fef1e7fc700 1 -- 172.21.2.117:0/4073922142 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1800b940 con 0x7fef200d1510 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 -- 172.21.2.117:0/4073922142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 msgr2=0x7fef200d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 --2- 172.21.2.117:0/4073922142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 0x7fef200d1910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fef18006e50 tx=0x7fef18009e70).stop 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 -- 172.21.2.117:0/4073922142 shutdown_connections 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 --2- 172.21.2.117:0/4073922142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200d1510 0x7fef200d1910 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 -- 172.21.2.117:0/4073922142 >> 172.21.2.117:0/4073922142 conn(0x7fef200ce430 msgr2=0x7fef200e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 -- 172.21.2.117:0/4073922142 shutdown_connections 2021-02-10T11:39:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.506 7fef25ee4700 1 -- 172.21.2.117:0/4073922142 wait complete. 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 Processor -- start 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 -- start start 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 0x7fef2019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef2019f6e0 0x7fef201023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2008ea40 con 0x7fef20102a10 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef200accb0 con 0x7fef2019f6e0 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef25ee4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2008e4d0 con 0x7fef200ac8b0 2021-02-10T11:39:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef1ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef1ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46628/0 (socket says 172.21.2.117:46628) 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef1ffff700 1 -- 172.21.2.117:0/841236705 learned_addr learned my addr 172.21.2.117:0/841236705 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef1f7fe700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 0x7fef2019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.521 7fef1effd700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef2019f6e0 0x7fef201023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 -- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 msgr2=0x7fef2019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 0x7fef2019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 -- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef2019f6e0 msgr2=0x7fef201023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef2019f6e0 0x7fef201023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 -- 172.21.2.117:0/841236705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef20102e10 con 0x7fef20102a10 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1f7fe700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 0x7fef2019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef1ffff700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fef18009a90 tx=0x7fef1800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef18009390 con 0x7fef20102a10 2021-02-10T11:39:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18002f10 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1800c570 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef25ee4700 1 -- 172.21.2.117:0/841236705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef20102e10 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.522 7fef25ee4700 1 -- 172.21.2.117:0/841236705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef20102e10 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.523 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef180091c0 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.523 7fef17fff700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 0x7fef0401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.523 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef18032aa0 con 0x7fef20102a10 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.523 7fef1f7fe700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 0x7fef0401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:06.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.523 7fef1f7fe700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 0x7fef0401abc0 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:06.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.554 7fef25ee4700 1 -- 172.21.2.117:0/841236705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef200ff1d0 con 0x7fef20102a10 2021-02-10T11:39:06.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.557 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef1800be30 con 0x7fef20102a10 2021-02-10T11:39:06.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.682 7fef25ee4700 1 -- 172.21.2.117:0/841236705 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802489", "target": ["mgr", ""]}) v1 -- 0x7fef200ff1d0 con 0x7fef040186c0 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.692 7fef17fff700 1 -- 172.21.2.117:0/841236705 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fef10000da0 con 0x7fef040186c0 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 msgr2=0x7fef0401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 0x7fef0401abc0 crc :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 msgr2=0x7fef200ad560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fef18009a90 tx=0x7fef1800bcf0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 shutdown_connections 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef040186c0 0x7fef0401abc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef20102a10 0x7fef200ad560 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef2019f6e0 0x7fef201023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 --2- 172.21.2.117:0/841236705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef200ac8b0 0x7fef2019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 >> 172.21.2.117:0/841236705 conn(0x7fef200b4e50 msgr2=0x7fef200be780 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:06.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 shutdown_connections 2021-02-10T11:39:06.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:06.700 7fef25ee4700 1 -- 172.21.2.117:0/841236705 wait complete. 2021-02-10T11:39:06.818 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802490 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.248 7fbcda21f700 1 Processor -- start 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.248 7fbcda21f700 1 -- start start 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcda21f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbcd4079000 0x7fbcd40e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcda21f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd40d1d20 con 0x7fbcd41b00f0 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcda21f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd4077a70 con 0x7fbcd407da20 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcda21f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd40e53c0 con 0x7fbcd4079000 2021-02-10T11:39:07.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd883c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbcd41b00f0 0x7fbcd40d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46886/0 (socket says 172.21.2.117:46886) 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd883c700 1 -- 172.21.2.117:0/397840223 learned_addr learned my addr 172.21.2.117:0/397840223 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd37fe700 1 --2- 172.21.2.117:0/397840223 >> v2:172.21.2.117:3300/0 conn(0x7fbcd4079000 0x7fbcd40e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd37fe700 1 -- 172.21.2.117:0/397840223 >> v1:172.21.2.125:6789/0 conn(0x7fbcd407da20 legacy=0x7fbcd40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd37fe700 1 -- 172.21.2.117:0/397840223 >> v1:172.21.2.117:6789/0 conn(0x7fbcd41b00f0 legacy=0x7fbcd40d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.249 7fbcd37fe700 1 -- 172.21.2.117:0/397840223 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd4079400 con 0x7fbcd4079000 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcd37fe700 1 --2- 172.21.2.117:0/397840223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd4079000 0x7fbcd40e4da0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fbcc8006650 tx=0x7fbcc8009a80).ready entity=mon.0 client_cookie=767b939cfc43c429 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcd2ffd700 1 -- 172.21.2.117:0/397840223 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcc80009b0 con 0x7fbcd4079000 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcd2ffd700 1 -- 172.21.2.117:0/397840223 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800c0c0 con 0x7fbcd4079000 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcd2ffd700 1 -- 172.21.2.117:0/397840223 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcc800c5e0 con 0x7fbcd4079000 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcda21f700 1 -- 172.21.2.117:0/397840223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd4079000 msgr2=0x7fbcd40e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.250 7fbcda21f700 1 --2- 172.21.2.117:0/397840223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd4079000 0x7fbcd40e4da0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fbcc8006650 tx=0x7fbcc8009a80).stop 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.251 7fbcda21f700 1 -- 172.21.2.117:0/397840223 shutdown_connections 2021-02-10T11:39:07.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.251 7fbcda21f700 1 --2- 172.21.2.117:0/397840223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd4079000 0x7fbcd40e4da0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.251 7fbcda21f700 1 -- 172.21.2.117:0/397840223 >> 172.21.2.117:0/397840223 conn(0x7fbcd40ce450 msgr2=0x7fbcd40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.251 7fbcda21f700 1 -- 172.21.2.117:0/397840223 shutdown_connections 2021-02-10T11:39:07.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.251 7fbcda21f700 1 -- 172.21.2.117:0/397840223 wait complete. 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.265 7fbcda21f700 1 Processor -- start 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 -- start start 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 0x7fbcd40bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbcd40c2e80 0x7fbcd40c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40b6200 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40c38a0 con 0x7fbcd40b4720 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcda21f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd40c3b20 con 0x7fbcd40c2e80 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcd37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcd37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46640/0 (socket says 172.21.2.117:46640) 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcd37fe700 1 -- 172.21.2.117:0/3323537573 learned_addr learned my addr 172.21.2.117:0/3323537573 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.266 7fbcd883c700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbcd40c2e80 0x7fbcd40c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd3fff700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 0x7fbcd40bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 -- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbcd40c2e80 msgr2=0x7fbcd40c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbcd40c2e80 0x7fbcd40c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 -- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 msgr2=0x7fbcd40bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 0x7fbcd40bf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 -- 172.21.2.117:0/3323537573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40b4b20 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd3fff700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 0x7fbcd40bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd37fe700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fbcc8009630 tx=0x7fbcc800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcc80093a0 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800bd70 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbcc80054d0 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd40b4b20 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.267 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd40b4b20 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.268 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbcc800a000 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.268 7fbcd0ff9700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 0x7fbcb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.268 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbcc80327e0 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.268 7fbcd3fff700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 0x7fbcb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.269 7fbcd3fff700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 0x7fbcb401ac10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.299 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd40ce450 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.301 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbcc801f550 con 0x7fbcd40bfdd0 2021-02-10T11:39:07.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.429 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802490", "target": ["mgr", ""]}) v1 -- 0x7fbcd40b4b20 con 0x7fbcb4018730 2021-02-10T11:39:07.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.449 7fbcd0ff9700 1 -- 172.21.2.117:0/3323537573 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbcc4006b20 con 0x7fbcb4018730 2021-02-10T11:39:07.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 msgr2=0x7fbcb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 0x7fbcb401ac10 crc :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 msgr2=0x7fbcd40c2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fbcc8009630 tx=0x7fbcc800b980).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 shutdown_connections 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbcb4018730 0x7fbcb401ac10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbcd40bfdd0 0x7fbcd40c2860 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbcd40c2e80 0x7fbcd40c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 --2- 172.21.2.117:0/3323537573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbcd40b4720 0x7fbcd40bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.462 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 >> 172.21.2.117:0/3323537573 conn(0x7fbcd41116a0 msgr2=0x7fbcd40af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.463 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 shutdown_connections 2021-02-10T11:39:07.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.463 7fbcda21f700 1 -- 172.21.2.117:0/3323537573 wait complete. 2021-02-10T11:39:07.452 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802491 2021-02-10T11:39:07.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 Processor -- start 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 -- start start 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1b3c0d1530 0x7f1b3c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c077a70 con 0x7f1b3c07da20 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c079400 con 0x7f1b3c079000 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b42074700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c07de20 con 0x7f1b3c0d1530 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b3b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1b3c07da20 0x7f1b3c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46898/0 (socket says 172.21.2.117:46898) 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.885 7f1b3b7fe700 1 -- 172.21.2.117:0/4074275384 learned_addr learned my addr 172.21.2.117:0/4074275384 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3bfff700 1 --2- 172.21.2.117:0/4074275384 >> v2:172.21.2.125:3300/0 conn(0x7f1b3c0d1530 0x7f1b3c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828786529 0 0) 0x7f1b240010b0 con 0x7f1b3c07da20 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28002fe0 con 0x7f1b3c07da20 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 583962027 0 0) 0x7f1b240010b0 con 0x7f1b3c07da20 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3bfff700 1 -- 172.21.2.117:0/4074275384 >> v1:172.21.2.125:6789/0 conn(0x7f1b3c079000 legacy=0x7f1b3c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3bfff700 1 -- 172.21.2.117:0/4074275384 >> v1:172.21.2.117:6789/0 conn(0x7f1b3c07da20 legacy=0x7f1b3c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3bfff700 1 -- 172.21.2.117:0/4074275384 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c1b00f0 con 0x7f1b3c0d1530 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3bfff700 1 --2- 172.21.2.117:0/4074275384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0d1530 0x7f1b3c0d1930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1b34006e50 tx=0x7f1b34009e70).ready entity=mon.1 client_cookie=7af2261ff3832b9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b3400b510 con 0x7f1b3c0d1530 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b34006790 con 0x7f1b3c0d1530 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b3a7fc700 1 -- 172.21.2.117:0/4074275384 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b3400b940 con 0x7f1b3c0d1530 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.886 7f1b42074700 1 -- 172.21.2.117:0/4074275384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0d1530 msgr2=0x7f1b3c0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 --2- 172.21.2.117:0/4074275384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0d1530 0x7f1b3c0d1930 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f1b34006e50 tx=0x7f1b34009e70).stop 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 -- 172.21.2.117:0/4074275384 shutdown_connections 2021-02-10T11:39:07.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 --2- 172.21.2.117:0/4074275384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0d1530 0x7f1b3c0d1930 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:07.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 -- 172.21.2.117:0/4074275384 >> 172.21.2.117:0/4074275384 conn(0x7f1b3c0ce450 msgr2=0x7f1b3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:07.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 -- 172.21.2.117:0/4074275384 shutdown_connections 2021-02-10T11:39:07.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.887 7f1b42074700 1 -- 172.21.2.117:0/4074275384 wait complete. 2021-02-10T11:39:07.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.902 7f1b42074700 1 Processor -- start 2021-02-10T11:39:07.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.902 7f1b42074700 1 -- start start 2021-02-10T11:39:07.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c18fdd0 0x7f1b3c098990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c1901d0 0x7f1b3c098fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0997e0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0c34b0 con 0x7f1b3c1901d0 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b42074700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0c36a0 con 0x7f1b3c18fdd0 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46654/0 (socket says 172.21.2.117:46654) 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 -- 172.21.2.117:0/367968876 learned_addr learned my addr 172.21.2.117:0/367968876 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 -- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c18fdd0 msgr2=0x7f1b3c098990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c18fdd0 0x7f1b3c098990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 -- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c1901d0 msgr2=0x7f1b3c098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c1901d0 0x7f1b3c098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 -- 172.21.2.117:0/367968876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0c38c0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b3bfff700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f1b34006c50 tx=0x7f1b3400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.903 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b340093d0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3400be60 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b42074700 1 -- 172.21.2.117:0/367968876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c0c38c0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b3400c4e0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b42074700 1 -- 172.21.2.117:0/367968876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c0c18d0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b340093d0 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b33fff700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1b34032930 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.904 7f1b3b7fe700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:07.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.905 7f1b3b7fe700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:07.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.934 7f1b42074700 1 -- 172.21.2.117:0/367968876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c103320 con 0x7f1b3c0c2a90 2021-02-10T11:39:07.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:07.937 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b3401f700 con 0x7f1b3c0c2a90 2021-02-10T11:39:08.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.063 7f1b42074700 1 -- 172.21.2.117:0/367968876 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802491", "target": ["mgr", ""]}) v1 -- 0x7f1b3c077a70 con 0x7f1b200186c0 2021-02-10T11:39:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.074 7f1b33fff700 1 -- 172.21.2.117:0/367968876 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1b240009b0 con 0x7f1b200186c0 2021-02-10T11:39:08.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.080 7f1b42074700 1 -- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 msgr2=0x7f1b2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.080 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 crc :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.080 7f1b42074700 1 -- 172.21.2.117:0/367968876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 msgr2=0x7f1b3c0c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.080 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f1b34006c50 tx=0x7f1b3400bd20).stop 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 -- 172.21.2.117:0/367968876 shutdown_connections 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c0c2a90 0x7f1b3c0c2e90 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c1901d0 0x7f1b3c098fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 --2- 172.21.2.117:0/367968876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c18fdd0 0x7f1b3c098990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 -- 172.21.2.117:0/367968876 >> 172.21.2.117:0/367968876 conn(0x7f1b3c0a1360 msgr2=0x7f1b3c091810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 -- 172.21.2.117:0/367968876 shutdown_connections 2021-02-10T11:39:08.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.081 7f1b42074700 1 -- 172.21.2.117:0/367968876 wait complete. 2021-02-10T11:39:08.070 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802492 2021-02-10T11:39:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.502 7f49749e0700 1 Processor -- start 2021-02-10T11:39:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.502 7f49749e0700 1 -- start start 2021-02-10T11:39:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f49749e0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4970079000 0x7f49700e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f49749e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49701b00f0 0x7f49701b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f49749e0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4970077a70 con 0x7f497007da20 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f49749e0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49700e53c0 con 0x7f4970079000 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f49749e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4970079400 con 0x7f49701b00f0 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f496de1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4970079000 0x7f49700e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f496de1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4970079000 0x7f49700e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51044/0 (socket says 172.21.2.117:51044) 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f496de1b700 1 -- 172.21.2.117:0/2809080145 learned_addr learned my addr 172.21.2.117:0/2809080145 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.503 7f496ee1d700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49701b00f0 0x7f49701b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496de1b700 1 -- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49701b00f0 msgr2=0x7f49701b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496de1b700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49701b00f0 0x7f49701b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496de1b700 1 -- 172.21.2.117:0/2809080145 >> v1:172.21.2.125:6789/0 conn(0x7f497007da20 legacy=0x7f49700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496d61a700 1 -- 172.21.2.117:0/2809080145 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235226485 0 0) 0x7f4960000ff0 con 0x7f497007da20 2021-02-10T11:39:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496de1b700 1 -- 172.21.2.117:0/2809080145 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497007de20 con 0x7f4970079000 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496de1b700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4970079000 0x7f49700e4da0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f4958006700 tx=0x7f4958009b00).ready entity=mon.1 client_cookie=a9e0a5a60fb68f6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496d61a700 1 -- 172.21.2.117:0/2809080145 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f495800c0c0 con 0x7f4970079000 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496d61a700 1 -- 172.21.2.117:0/2809080145 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495800c340 con 0x7f4970079000 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f496d61a700 1 -- 172.21.2.117:0/2809080145 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49580009b0 con 0x7f4970079000 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f49749e0700 1 -- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4970079000 msgr2=0x7f49700e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.504 7f49749e0700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4970079000 0x7f49700e4da0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f4958006700 tx=0x7f4958009b00).stop 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 -- 172.21.2.117:0/2809080145 shutdown_connections 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49701b00f0 0x7f49701b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 --2- 172.21.2.117:0/2809080145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4970079000 0x7f49700e4da0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 -- 172.21.2.117:0/2809080145 >> 172.21.2.117:0/2809080145 conn(0x7f49700ce450 msgr2=0x7f49700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 -- 172.21.2.117:0/2809080145 shutdown_connections 2021-02-10T11:39:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.505 7f49749e0700 1 -- 172.21.2.117:0/2809080145 wait complete. 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.519 7f49749e0700 1 Processor -- start 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.519 7f49749e0700 1 -- start start 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49700a41c0 0x7f49700fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 0x7f49700c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49700b8010 con 0x7f49700b7390 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49700b8550 con 0x7f49700c2340 2021-02-10T11:39:08.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f49749e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49700b8770 con 0x7f49700a41c0 2021-02-10T11:39:08.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496de1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 0x7f49700c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496de1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 0x7f49700c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51048/0 (socket says 172.21.2.117:51048) 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46668/0 (socket says 172.21.2.117:46668) 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496de1b700 1 -- 172.21.2.117:0/25714759 learned_addr learned my addr 172.21.2.117:0/25714759 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496e61c700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49700a41c0 0x7f49700fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 -- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49700a41c0 msgr2=0x7f49700fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49700a41c0 0x7f49700fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 -- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 msgr2=0x7f49700c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 0x7f49700c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.520 7f496ee1d700 1 -- 172.21.2.117:0/25714759 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49700b8990 con 0x7f49700b7390 2021-02-10T11:39:08.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f496ee1d700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f496800b8a0 tx=0x7f496800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4968000f90 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49680066f0 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49680013e0 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f49749e0700 1 -- 172.21.2.117:0/25714759 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49700b8990 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.521 7f49749e0700 1 -- 172.21.2.117:0/25714759 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49700b8f20 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.522 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49680056b0 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.522 7f4966ffd700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 0x7f495401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.522 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4968032930 con 0x7f49700b7390 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.522 7f496e61c700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 0x7f495401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:08.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.523 7f496e61c700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 0x7f495401abc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:08.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.553 7f49749e0700 1 -- 172.21.2.117:0/25714759 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49701b4f00 con 0x7f49700b7390 2021-02-10T11:39:08.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.555 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4968000f70 con 0x7f49700b7390 2021-02-10T11:39:08.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.681 7f49749e0700 1 -- 172.21.2.117:0/25714759 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802492", "target": ["mgr", ""]}) v1 -- 0x7f4970077a70 con 0x7f49540186c0 2021-02-10T11:39:08.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.691 7f4966ffd700 1 -- 172.21.2.117:0/25714759 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f49600054b0 con 0x7f49540186c0 2021-02-10T11:39:08.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 -- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 msgr2=0x7f495401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 0x7f495401abc0 crc :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 -- 172.21.2.117:0/25714759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 msgr2=0x7f49700b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:08.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f496800b8a0 tx=0x7f496800bc40).stop 2021-02-10T11:39:08.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 -- 172.21.2.117:0/25714759 shutdown_connections 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49540186c0 0x7f495401abc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49700b7390 0x7f49700b7790 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49700c2340 0x7f49700c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 --2- 172.21.2.117:0/25714759 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49700a41c0 0x7f49700fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.698 7f49749e0700 1 -- 172.21.2.117:0/25714759 >> 172.21.2.117:0/25714759 conn(0x7f497008bbc0 msgr2=0x7f49700c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.699 7f49749e0700 1 -- 172.21.2.117:0/25714759 shutdown_connections 2021-02-10T11:39:08.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:08.699 7f49749e0700 1 -- 172.21.2.117:0/25714759 wait complete. 2021-02-10T11:39:08.688 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802493 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.120 7fd054227700 1 Processor -- start 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.120 7fd054227700 1 -- start start 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd054227700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 0x7fd04c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd054227700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 0x7fd04c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd054227700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd04c077a70 con 0x7fd04c0e4da0 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd054227700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04c0777d0 con 0x7fd04c07da20 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd054227700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1b08e0 con 0x7fd04c0e51a0 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd051842700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd04c0e4da0 0x7fd04c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46922/0 (socket says 172.21.2.117:46922) 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd051842700 1 -- 172.21.2.117:0/1085526043 learned_addr learned my addr 172.21.2.117:0/1085526043 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd052043700 1 --2- 172.21.2.117:0/1085526043 >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 0x7fd04c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd052844700 1 --2- 172.21.2.117:0/1085526043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 0x7fd04c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd051041700 1 -- 172.21.2.117:0/1085526043 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803738020 0 0) 0x7fd0480010b0 con 0x7fd04c0e4da0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd051041700 1 -- 172.21.2.117:0/1085526043 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd038002fe0 con 0x7fd04c0e4da0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.121 7fd052844700 1 -- 172.21.2.117:0/1085526043 >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 msgr2=0x7fd04c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd052844700 1 --2- 172.21.2.117:0/1085526043 >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 0x7fd04c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd052844700 1 -- 172.21.2.117:0/1085526043 >> v1:172.21.2.117:6789/0 conn(0x7fd04c0e4da0 legacy=0x7fd04c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd052844700 1 -- 172.21.2.117:0/1085526043 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c07de20 con 0x7fd04c0e51a0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd052043700 1 --2- 172.21.2.117:0/1085526043 >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 0x7fd04c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd052844700 1 --2- 172.21.2.117:0/1085526043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 0x7fd04c079000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd044006e50 tx=0x7fd044009e70).ready entity=mon.2 client_cookie=ab5f863feb524b7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd051041700 1 -- 172.21.2.117:0/1085526043 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd04400b510 con 0x7fd04c0e51a0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd051041700 1 -- 172.21.2.117:0/1085526043 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044006790 con 0x7fd04c0e51a0 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd054227700 1 -- 172.21.2.117:0/1085526043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 msgr2=0x7fd04c079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd054227700 1 --2- 172.21.2.117:0/1085526043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 0x7fd04c079000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd044006e50 tx=0x7fd044009e70).stop 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.122 7fd051041700 1 -- 172.21.2.117:0/1085526043 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd044000f90 con 0x7fd04c0e51a0 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 -- 172.21.2.117:0/1085526043 shutdown_connections 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 --2- 172.21.2.117:0/1085526043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c0e51a0 0x7fd04c079000 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 --2- 172.21.2.117:0/1085526043 >> v2:172.21.2.125:3300/0 conn(0x7fd04c07da20 0x7fd04c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 -- 172.21.2.117:0/1085526043 >> 172.21.2.117:0/1085526043 conn(0x7fd04c0ce450 msgr2=0x7fd04c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 -- 172.21.2.117:0/1085526043 shutdown_connections 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.123 7fd054227700 1 -- 172.21.2.117:0/1085526043 wait complete. 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 Processor -- start 2021-02-10T11:39:09.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 -- start start 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 0x7fd04c1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 0x7fd04c0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0b8da0 con 0x7fd04c1b5b50 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0a4140 con 0x7fd04c1b5130 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.138 7fd054227700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0a4320 con 0x7fd04c1b5f50 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052844700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052844700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44356/0 (socket says 172.21.2.117:44356) 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052844700 1 -- 172.21.2.117:0/1589999557 learned_addr learned my addr 172.21.2.117:0/1589999557 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 0x7fd04c0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052043700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 0x7fd04c1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 -- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 msgr2=0x7fd04c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 -- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 msgr2=0x7fd04c1b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 0x7fd04c1b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 -- 172.21.2.117:0/1589999557 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c0b62b0 con 0x7fd04c1b5b50 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052844700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd052043700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 0x7fd04c1b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd051842700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 0x7fd04c0b5670 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fd048001c60 tx=0x7fd04800a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd048000e00 con 0x7fd04c1b5b50 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04800ac60 con 0x7fd04c1b5b50 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.139 7fd054227700 1 -- 172.21.2.117:0/1589999557 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c0b62b0 con 0x7fd04c1b5b50 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0480045d0 con 0x7fd04c1b5b50 2021-02-10T11:39:09.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd054227700 1 -- 172.21.2.117:0/1589999557 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c0b62b0 con 0x7fd04c1b5b50 2021-02-10T11:39:09.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd048000e00 con 0x7fd04c1b5b50 2021-02-10T11:39:09.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd03effd700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 0x7fd03401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd048032810 con 0x7fd04c1b5b50 2021-02-10T11:39:09.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.140 7fd052043700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 0x7fd03401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.141 7fd052043700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 0x7fd03401ac10 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.171 7fd054227700 1 -- 172.21.2.117:0/1589999557 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c0a4320 con 0x7fd04c1b5b50 2021-02-10T11:39:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.173 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd048000e00 con 0x7fd04c1b5b50 2021-02-10T11:39:09.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.298 7fd054227700 1 -- 172.21.2.117:0/1589999557 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802493", "target": ["mgr", ""]}) v1 -- 0x7fd04c0a4320 con 0x7fd034018730 2021-02-10T11:39:09.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.309 7fd03effd700 1 -- 172.21.2.117:0/1589999557 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd0400009b0 con 0x7fd034018730 2021-02-10T11:39:09.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 msgr2=0x7fd03401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 0x7fd03401ac10 crc :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 msgr2=0x7fd04c0b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 0x7fd04c0b5670 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7fd048001c60 tx=0x7fd04800a870).stop 2021-02-10T11:39:09.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 shutdown_connections 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd034018730 0x7fd03401ac10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd04c1b5b50 0x7fd04c0b5670 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd04c1b5f50 0x7fd04c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 --2- 172.21.2.117:0/1589999557 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd04c1b5130 0x7fd04c1b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 >> 172.21.2.117:0/1589999557 conn(0x7fd04c08bbc0 msgr2=0x7fd04c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 shutdown_connections 2021-02-10T11:39:09.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.315 7fd054227700 1 -- 172.21.2.117:0/1589999557 wait complete. 2021-02-10T11:39:09.304 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802494 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.737 7fccf66f3700 1 Processor -- start 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.737 7fccf66f3700 1 -- start start 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf66f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fccf007da20 0x7fccf00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf66f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 0x7fccf0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf66f3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf0077a70 con 0x7fccf00e4da0 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf66f3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf00777d0 con 0x7fccf007da20 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf66f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf01b08e0 con 0x7fccf00e51a0 2021-02-10T11:39:09.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcceffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fccf007da20 0x7fccf00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcce7fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fccf00e4da0 0x7fccf01b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39860/0 (socket says 172.21.2.117:39860) 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcce7fff700 1 -- 172.21.2.117:0/3056046094 learned_addr learned my addr 172.21.2.117:0/3056046094 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fccf4d10700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 0x7fccf0079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcceffff700 1 -- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 msgr2=0x7fccf0079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcceffff700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 0x7fccf0079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.738 7fcceffff700 1 -- 172.21.2.117:0/3056046094 >> v1:172.21.2.125:6789/0 conn(0x7fccf00e4da0 legacy=0x7fccf01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fcceffff700 1 -- 172.21.2.117:0/3056046094 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf007de20 con 0x7fccf007da20 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccf4d10700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 0x7fccf0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fcceffff700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf007da20 0x7fccf00771b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fccd8006650 tx=0x7fccd8009a80).ready entity=mon.1 client_cookie=1194927d1be8a2d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccef7fe700 1 -- 172.21.2.117:0/3056046094 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccd80009b0 con 0x7fccf007da20 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccef7fe700 1 -- 172.21.2.117:0/3056046094 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd800c0c0 con 0x7fccf007da20 2021-02-10T11:39:09.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccef7fe700 1 -- 172.21.2.117:0/3056046094 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccd800c520 con 0x7fccf007da20 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccf66f3700 1 -- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf007da20 msgr2=0x7fccf00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.739 7fccf66f3700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf007da20 0x7fccf00771b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fccd8006650 tx=0x7fccd8009a80).stop 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 -- 172.21.2.117:0/3056046094 shutdown_connections 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf00e51a0 0x7fccf0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 --2- 172.21.2.117:0/3056046094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf007da20 0x7fccf00771b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 -- 172.21.2.117:0/3056046094 >> 172.21.2.117:0/3056046094 conn(0x7fccf00ce450 msgr2=0x7fccf00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 -- 172.21.2.117:0/3056046094 shutdown_connections 2021-02-10T11:39:09.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.740 7fccf66f3700 1 -- 172.21.2.117:0/3056046094 wait complete. 2021-02-10T11:39:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.754 7fccf66f3700 1 Processor -- start 2021-02-10T11:39:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 -- start start 2021-02-10T11:39:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf00976d0 0x7fccf0195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf0097ad0 0x7fccf0092430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0107720 con 0x7fccf0107320 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0196110 con 0x7fccf00976d0 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fccf66f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0196330 con 0x7fccf0097ad0 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fcceffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fcceffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46686/0 (socket says 172.21.2.117:46686) 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.755 7fcceffff700 1 -- 172.21.2.117:0/3124374443 learned_addr learned my addr 172.21.2.117:0/3124374443 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fccf4d10700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf0097ad0 0x7fccf0092430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcce7fff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf00976d0 0x7fccf0195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 -- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf0097ad0 msgr2=0x7fccf0092430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf0097ad0 0x7fccf0092430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 -- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf00976d0 msgr2=0x7fccf0195af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf00976d0 0x7fccf0195af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 -- 172.21.2.117:0/3124374443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf0092a50 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcceffff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fccd8009630 tx=0x7fccd800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccd800bd70 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd8000960 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.756 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccd80054d0 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf0092a50 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf0092fe0 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccd800a000 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fcced7fa700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fccd8032670 con 0x7fccf0107320 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.757 7fcce7fff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.758 7fcce7fff700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:09.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.786 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf01b4f00 con 0x7fccf0107320 2021-02-10T11:39:09.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.789 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccd8024a00 con 0x7fccf0107320 2021-02-10T11:39:09.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.919 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802494", "target": ["mgr", ""]}) v1 -- 0x7fccf0092fe0 con 0x7fccd40186c0 2021-02-10T11:39:09.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.929 7fcced7fa700 1 -- 172.21.2.117:0/3124374443 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcce0007840 con 0x7fccd40186c0 2021-02-10T11:39:09.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 msgr2=0x7fccd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 crc :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 msgr2=0x7fccf0107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fccd8009630 tx=0x7fccd800b980).stop 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 shutdown_connections 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccf0097ad0 0x7fccf0092430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccf00976d0 0x7fccf0195af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fccd40186c0 0x7fccd401abc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 --2- 172.21.2.117:0/3124374443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccf0107320 0x7fccf0107b70 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 >> 172.21.2.117:0/3124374443 conn(0x7fccf00ffe30 msgr2=0x7fccf00c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 shutdown_connections 2021-02-10T11:39:09.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:09.935 7fccf66f3700 1 -- 172.21.2.117:0/3124374443 wait complete. 2021-02-10T11:39:09.925 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802495 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.357 7f5ee69f0700 1 Processor -- start 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.357 7f5ee69f0700 1 -- start start 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee69f0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ee00d1530 0x7f5ee00d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee69f0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0077a70 con 0x7f5ee007da20 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee69f0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0079400 con 0x7f5ee0079000 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee69f0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee007de20 con 0x7f5ee00d1530 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee500d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ee00d1530 0x7f5ee00d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee480c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5ee007da20 0x7f5ee00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46944/0 (socket says 172.21.2.117:46944) 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee480c700 1 -- 172.21.2.117:0/777670145 learned_addr learned my addr 172.21.2.117:0/777670145 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5edf7fe700 1 -- 172.21.2.117:0/777670145 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214471106 0 0) 0x7f5ed0000f30 con 0x7f5ee0079000 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5edf7fe700 1 -- 172.21.2.117:0/777670145 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc002fe0 con 0x7f5ee0079000 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee500d700 1 -- 172.21.2.117:0/777670145 >> v1:172.21.2.125:6789/0 conn(0x7f5ee0079000 legacy=0x7f5ee00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee500d700 1 -- 172.21.2.117:0/777670145 >> v1:172.21.2.117:6789/0 conn(0x7f5ee007da20 legacy=0x7f5ee00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.358 7f5ee500d700 1 -- 172.21.2.117:0/777670145 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee01b00f0 con 0x7f5ee00d1530 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee500d700 1 --2- 172.21.2.117:0/777670145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee00d1530 0x7f5ee00d1930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f5ed8006e30 tx=0x7f5ed8009e60).ready entity=mon.0 client_cookie=53f36e1755b53601 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5edf7fe700 1 -- 172.21.2.117:0/777670145 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ed8006a50 con 0x7f5ee00d1530 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5edf7fe700 1 -- 172.21.2.117:0/777670145 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed800b300 con 0x7f5ee00d1530 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5edf7fe700 1 -- 172.21.2.117:0/777670145 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ed800b820 con 0x7f5ee00d1530 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 -- 172.21.2.117:0/777670145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee00d1530 msgr2=0x7f5ee00d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 --2- 172.21.2.117:0/777670145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee00d1530 0x7f5ee00d1930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f5ed8006e30 tx=0x7f5ed8009e60).stop 2021-02-10T11:39:10.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 -- 172.21.2.117:0/777670145 shutdown_connections 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 --2- 172.21.2.117:0/777670145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee00d1530 0x7f5ee00d1930 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 -- 172.21.2.117:0/777670145 >> 172.21.2.117:0/777670145 conn(0x7f5ee00ce450 msgr2=0x7f5ee00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 -- 172.21.2.117:0/777670145 shutdown_connections 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.359 7f5ee69f0700 1 -- 172.21.2.117:0/777670145 wait complete. 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.374 7f5ee69f0700 1 Processor -- start 2021-02-10T11:39:10.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.374 7f5ee69f0700 1 -- start start 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 0x7f5ee0104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee0105160 0x7f5ee00aad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0103c20 con 0x7f5ee01034d0 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0103910 con 0x7f5ee0104740 2021-02-10T11:39:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee69f0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00b7e00 con 0x7f5ee0105160 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5edffff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 0x7f5ee0104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46702/0 (socket says 172.21.2.117:46702) 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 -- 172.21.2.117:0/1446835560 learned_addr learned my addr 172.21.2.117:0/1446835560 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee500d700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee0105160 0x7f5ee00aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 -- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee0105160 msgr2=0x7f5ee00aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee0105160 0x7f5ee00aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 -- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 msgr2=0x7f5ee0104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 0x7f5ee0104b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.375 7f5ee480c700 1 -- 172.21.2.117:0/1446835560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00ab330 con 0x7f5ee01034d0 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5edffff700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 0x7f5ee0104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5ee480c700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f5ec8001c60 tx=0x7f5ec800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:10.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ec800acc0 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec8004140 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ec8004620 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee00ab330 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.376 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee00ab8c0 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.377 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ec8004320 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.377 7f5edd7fa700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 0x7f5ec401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.377 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5ec8032670 con 0x7f5ee01034d0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.377 7f5edffff700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 0x7f5ec401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.377 7f5edffff700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 0x7f5ec401abc0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:10.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.407 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee01032a0 con 0x7f5ee01034d0 2021-02-10T11:39:10.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.410 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ec801a400 con 0x7f5ee01034d0 2021-02-10T11:39:10.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.536 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802495", "target": ["mgr", ""]}) v1 -- 0x7f5ee00ab8c0 con 0x7f5ec40186c0 2021-02-10T11:39:10.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.547 7f5edd7fa700 1 -- 172.21.2.117:0/1446835560 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5ed00068d0 con 0x7f5ec40186c0 2021-02-10T11:39:10.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 msgr2=0x7f5ec401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 0x7f5ec401abc0 crc :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 msgr2=0x7f5ee0104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f5ec8001c60 tx=0x7f5ec800a8d0).stop 2021-02-10T11:39:10.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 shutdown_connections 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee0105160 0x7f5ee00aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee0104740 0x7f5ee0104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec40186c0 0x7f5ec401abc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 --2- 172.21.2.117:0/1446835560 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee01034d0 0x7f5ee0104120 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.554 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 >> 172.21.2.117:0/1446835560 conn(0x7f5ee00a1140 msgr2=0x7f5ee00ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.555 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 shutdown_connections 2021-02-10T11:39:10.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.555 7f5ee69f0700 1 -- 172.21.2.117:0/1446835560 wait complete. 2021-02-10T11:39:10.544 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802496 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.977 7f09971da700 1 Processor -- start 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.977 7f09971da700 1 -- start start 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09971da700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f099007da20 0x7f09900771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09971da700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09900e4da0 0x7f09901b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09971da700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0990077a70 con 0x7f09900e51a0 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09971da700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09900777d0 con 0x7f09900e4da0 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09971da700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099007de20 con 0x7f099007da20 2021-02-10T11:39:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09957f7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09900e51a0 0x7f0990077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:46960/0 (socket says 172.21.2.117:46960) 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f09957f7700 1 -- 172.21.2.117:0/2975345653 learned_addr learned my addr 172.21.2.117:0/2975345653 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f098ffff700 1 --2- 172.21.2.117:0/2975345653 >> v2:172.21.2.117:3300/0 conn(0x7f09900e4da0 0x7f09901b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f0994ff6700 1 --2- 172.21.2.117:0/2975345653 >> v2:172.21.2.125:3300/0 conn(0x7f099007da20 0x7f09900771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f098ffff700 1 -- 172.21.2.117:0/2975345653 >> v2:172.21.2.125:3300/0 conn(0x7f099007da20 msgr2=0x7f09900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f098ffff700 1 --2- 172.21.2.117:0/2975345653 >> v2:172.21.2.125:3300/0 conn(0x7f099007da20 0x7f09900771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f098ffff700 1 -- 172.21.2.117:0/2975345653 >> v1:172.21.2.117:6789/0 conn(0x7f09900e51a0 legacy=0x7f0990077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.978 7f098ffff700 1 -- 172.21.2.117:0/2975345653 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09901b0710 con 0x7f09900e4da0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f098ffff700 1 --2- 172.21.2.117:0/2975345653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09900e4da0 0x7f09901b00f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f09800069d0 tx=0x7f0980009e40).ready entity=mon.0 client_cookie=35c1ceaf36ca47ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f098f7fe700 1 -- 172.21.2.117:0/2975345653 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f098000b510 con 0x7f09900e4da0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f098f7fe700 1 -- 172.21.2.117:0/2975345653 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098000b790 con 0x7f09900e4da0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f098f7fe700 1 -- 172.21.2.117:0/2975345653 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f098000bcb0 con 0x7f09900e4da0 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f09971da700 1 -- 172.21.2.117:0/2975345653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09900e4da0 msgr2=0x7f09901b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.979 7f09971da700 1 --2- 172.21.2.117:0/2975345653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09900e4da0 0x7f09901b00f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f09800069d0 tx=0x7f0980009e40).stop 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 -- 172.21.2.117:0/2975345653 shutdown_connections 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 --2- 172.21.2.117:0/2975345653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09900e4da0 0x7f09901b00f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 --2- 172.21.2.117:0/2975345653 >> v2:172.21.2.125:3300/0 conn(0x7f099007da20 0x7f09900771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 -- 172.21.2.117:0/2975345653 >> 172.21.2.117:0/2975345653 conn(0x7f09900ce450 msgr2=0x7f09900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 -- 172.21.2.117:0/2975345653 shutdown_connections 2021-02-10T11:39:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.980 7f09971da700 1 -- 172.21.2.117:0/2975345653 wait complete. 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.994 7f09971da700 1 Processor -- start 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.994 7f09971da700 1 -- start start 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09900b7fc0 0x7f09901042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09901048f0 0x7f0990104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09901054f0 con 0x7f0990090710 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0990195a30 con 0x7f09900b7fc0 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09971da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0990195fb0 con 0x7f09901048f0 2021-02-10T11:39:10.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46714/0 (socket says 172.21.2.117:46714) 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 -- 172.21.2.117:0/1046724937 learned_addr learned my addr 172.21.2.117:0/1046724937 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f0994ff6700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09900b7fc0 0x7f09901042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f098ffff700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09901048f0 0x7f0990104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 -- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09901048f0 msgr2=0x7f0990104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09901048f0 0x7f0990104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 -- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09900b7fc0 msgr2=0x7f09901042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09900b7fc0 0x7f09901042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.995 7f09957f7700 1 -- 172.21.2.117:0/1046724937 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09900b7b60 con 0x7f0990090710 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f09957f7700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f0988001ec0 tx=0x7f098800a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0988000ef0 con 0x7f0990090710 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098800ac90 con 0x7f0990090710 2021-02-10T11:39:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f09971da700 1 -- 172.21.2.117:0/1046724937 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09900b7b60 con 0x7f0990090710 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f09971da700 1 -- 172.21.2.117:0/1046724937 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09900b7b60 con 0x7f0990090710 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.996 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0988004620 con 0x7f0990090710 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.997 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09880043a0 con 0x7f0990090710 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.997 7f098d7fa700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 0x7f097401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.997 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0988032930 con 0x7f0990090710 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.997 7f0994ff6700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 0x7f097401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:10.997 7f0994ff6700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 0x7f097401ac10 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:11.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.027 7f09971da700 1 -- 172.21.2.117:0/1046724937 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0990103280 con 0x7f0990090710 2021-02-10T11:39:11.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.030 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0988007070 con 0x7f0990090710 2021-02-10T11:39:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.156 7f09971da700 1 -- 172.21.2.117:0/1046724937 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802496", "target": ["mgr", ""]}) v1 -- 0x7f0990103280 con 0x7f0974018730 2021-02-10T11:39:11.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.168 7f098d7fa700 1 -- 172.21.2.117:0/1046724937 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0978006a70 con 0x7f0974018730 2021-02-10T11:39:11.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 msgr2=0x7f097401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 0x7f097401ac10 crc :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 msgr2=0x7f0990090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f0988001ec0 tx=0x7f098800a8a0).stop 2021-02-10T11:39:11.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 shutdown_connections 2021-02-10T11:39:11.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0974018730 0x7f097401ac10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0990090710 0x7f0990090b10 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09901048f0 0x7f0990104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 --2- 172.21.2.117:0/1046724937 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09900b7fc0 0x7f09901042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 >> 172.21.2.117:0/1046724937 conn(0x7f09900a1850 msgr2=0x7f09900c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 shutdown_connections 2021-02-10T11:39:11.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.174 7f09971da700 1 -- 172.21.2.117:0/1046724937 wait complete. 2021-02-10T11:39:11.163 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802497 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.595 7ff43d72b700 1 Processor -- start 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.595 7ff43d72b700 1 -- start start 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff43d72b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff438079000 0x7ff4380e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff43d72b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff4381b00d0 0x7ff4381b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff43d72b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438077a70 con 0x7ff43807da20 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff43d72b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4380e53a0 con 0x7ff438079000 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff43d72b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff438079400 con 0x7ff4381b00d0 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff438079000 0x7ff4380e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff438079000 0x7ff4380e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46726/0 (socket says 172.21.2.117:46726) 2021-02-10T11:39:11.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 -- 172.21.2.117:0/2468620544 learned_addr learned my addr 172.21.2.117:0/2468620544 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff437ac0700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff4381b00d0 0x7ff4381b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 -- 172.21.2.117:0/2468620544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff4381b00d0 msgr2=0x7ff4381b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff4381b00d0 0x7ff4381b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 -- 172.21.2.117:0/2468620544 >> v1:172.21.2.125:6789/0 conn(0x7ff43807da20 legacy=0x7ff4380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.596 7ff42eabe700 1 -- 172.21.2.117:0/2468620544 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43807de20 con 0x7ff438079000 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff42eabe700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff438079000 0x7ff4380e4d80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7ff420006700 tx=0x7ff420009b80).ready entity=mon.0 client_cookie=5d5bbd1cee4be7be server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff436abe700 1 -- 172.21.2.117:0/2468620544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42000c0c0 con 0x7ff438079000 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff436abe700 1 -- 172.21.2.117:0/2468620544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42000c400 con 0x7ff438079000 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff436abe700 1 -- 172.21.2.117:0/2468620544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42000c920 con 0x7ff438079000 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff43d72b700 1 -- 172.21.2.117:0/2468620544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff438079000 msgr2=0x7ff4380e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.597 7ff43d72b700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff438079000 0x7ff4380e4d80 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7ff420006700 tx=0x7ff420009b80).stop 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 -- 172.21.2.117:0/2468620544 shutdown_connections 2021-02-10T11:39:11.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff4381b00d0 0x7ff4381b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 --2- 172.21.2.117:0/2468620544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff438079000 0x7ff4380e4d80 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 -- 172.21.2.117:0/2468620544 >> 172.21.2.117:0/2468620544 conn(0x7ff4380ce430 msgr2=0x7ff4380e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:11.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 -- 172.21.2.117:0/2468620544 shutdown_connections 2021-02-10T11:39:11.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.598 7ff43d72b700 1 -- 172.21.2.117:0/2468620544 wait complete. 2021-02-10T11:39:11.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.612 7ff43d72b700 1 Processor -- start 2021-02-10T11:39:11.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.612 7ff43d72b700 1 -- start start 2021-02-10T11:39:11.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff438189f00 0x7ff4381893b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff4380bc940 0x7ff4380bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43818a300 con 0x7ff4381062c0 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4381899d0 con 0x7ff4380bc940 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff43d72b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4380bd360 con 0x7ff438189f00 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46728/0 (socket says 172.21.2.117:46728) 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 -- 172.21.2.117:0/1291131174 learned_addr learned my addr 172.21.2.117:0/1291131174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff437ac0700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff4380bc940 0x7ff4380bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff4372bf700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff438189f00 0x7ff4381893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 -- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff438189f00 msgr2=0x7ff4381893b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff438189f00 0x7ff4381893b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 -- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff4380bc940 msgr2=0x7ff4380bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff4380bc940 0x7ff4380bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.613 7ff42eabe700 1 -- 172.21.2.117:0/1291131174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380bd580 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff42eabe700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7ff4200066a0 tx=0x7ff42000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff42000bd80 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4200052e0 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff420005800 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4380bd580 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.614 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4380bdb10 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.615 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff420006030 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.615 7ff434aba700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 0x7ff41c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.615 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff420032720 con 0x7ff4381062c0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.615 7ff4372bf700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 0x7ff41c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:11.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.615 7ff4372bf700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 0x7ff41c01abc0 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:11.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.646 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff438103260 con 0x7ff4381062c0 2021-02-10T11:39:11.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.648 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff420030d80 con 0x7ff4381062c0 2021-02-10T11:39:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.777 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802497", "target": ["mgr", ""]}) v1 -- 0x7ff43818a300 con 0x7ff41c0186c0 2021-02-10T11:39:11.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.787 7ff434aba700 1 -- 172.21.2.117:0/1291131174 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff428006550 con 0x7ff41c0186c0 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 msgr2=0x7ff41c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 0x7ff41c01abc0 crc :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 msgr2=0x7ff4381066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7ff4200066a0 tx=0x7ff42000b990).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 shutdown_connections 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff41c0186c0 0x7ff41c01abc0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff4381062c0 0x7ff4381066c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff4380bc940 0x7ff4380bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 --2- 172.21.2.117:0/1291131174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff438189f00 0x7ff4381893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 >> 172.21.2.117:0/1291131174 conn(0x7ff43818a760 msgr2=0x7ff438190320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 shutdown_connections 2021-02-10T11:39:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:11.794 7ff43d72b700 1 -- 172.21.2.117:0/1291131174 wait complete. 2021-02-10T11:39:11.784 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802498 2021-02-10T11:39:12.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.230 7f976093f700 1 Processor -- start 2021-02-10T11:39:12.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.230 7f976093f700 1 -- start start 2021-02-10T11:39:12.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f976093f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f975c07da20 0x7f975c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f976093f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 0x7f975c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f976093f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c077a70 con 0x7f975c0e51a0 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f976093f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975c0777d0 con 0x7f975c07da20 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f976093f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c07de20 con 0x7f975c0e4da0 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f975a61c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f975c07da20 0x7f975c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f975ae1d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f975c0e51a0 0x7f975c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:39916/0 (socket says 172.21.2.117:39916) 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f975ae1d700 1 -- 172.21.2.117:0/4192956801 learned_addr learned my addr 172.21.2.117:0/4192956801 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.231 7f9759e1b700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 0x7f975c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975a61c700 1 -- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 msgr2=0x7f975c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975a61c700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 0x7f975c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975a61c700 1 -- 172.21.2.117:0/4192956801 >> v1:172.21.2.125:6789/0 conn(0x7f975c0e51a0 legacy=0x7f975c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975a61c700 1 -- 172.21.2.117:0/4192956801 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c1b0710 con 0x7f975c07da20 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f9759e1b700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 0x7f975c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975a61c700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c07da20 0x7f975c0771b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9750006650 tx=0x7f9750009a80).ready entity=mon.1 client_cookie=374ca35f33c1d421 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975961a700 1 -- 172.21.2.117:0/4192956801 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97500009b0 con 0x7f975c07da20 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975961a700 1 -- 172.21.2.117:0/4192956801 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975000c0c0 con 0x7f975c07da20 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.232 7f975961a700 1 -- 172.21.2.117:0/4192956801 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f975000c5e0 con 0x7f975c07da20 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 -- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c07da20 msgr2=0x7f975c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c07da20 0x7f975c0771b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f9750006650 tx=0x7f9750009a80).stop 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 -- 172.21.2.117:0/4192956801 shutdown_connections 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0e4da0 0x7f975c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 --2- 172.21.2.117:0/4192956801 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c07da20 0x7f975c0771b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 -- 172.21.2.117:0/4192956801 >> 172.21.2.117:0/4192956801 conn(0x7f975c0ce450 msgr2=0x7f975c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 -- 172.21.2.117:0/4192956801 shutdown_connections 2021-02-10T11:39:12.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.233 7f976093f700 1 -- 172.21.2.117:0/4192956801 wait complete. 2021-02-10T11:39:12.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 Processor -- start 2021-02-10T11:39:12.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 -- start start 2021-02-10T11:39:12.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0a1090 0x7f975c19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c19db70 0x7f975c0a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0c1360 con 0x7f975c0a20e0 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0a2b00 con 0x7f975c19db70 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f976093f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0a2d20 con 0x7f975c0a1090 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f975ae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f975ae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46742/0 (socket says 172.21.2.117:46742) 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.248 7f975ae1d700 1 -- 172.21.2.117:0/3177794046 learned_addr learned my addr 172.21.2.117:0/3177794046 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975a61c700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0a1090 0x7f975c19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f9759e1b700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c19db70 0x7f975c0a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 -- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0a1090 msgr2=0x7f975c19d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0a1090 0x7f975c19d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 -- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c19db70 msgr2=0x7f975c0a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c19db70 0x7f975c0a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 -- 172.21.2.117:0/3177794046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c106040 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f975ae1d700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f975400a8a0 tx=0x7f975400ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9754007020 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754007220 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f976093f700 1 -- 172.21.2.117:0/3177794046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c106040 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f976093f700 1 -- 172.21.2.117:0/3177794046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c106040 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.249 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97540046a0 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.250 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9754007400 con 0x7f975c0a20e0 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.250 7f974affd700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 0x7f974001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.250 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9754032c70 con 0x7f975c0a20e0 2021-02-10T11:39:12.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.250 7f975a61c700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 0x7f974001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.251 7f975a61c700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 0x7f974001ac10 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.281 7f976093f700 1 -- 172.21.2.117:0/3177794046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c1b4f00 con 0x7f975c0a20e0 2021-02-10T11:39:12.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.283 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9754007db0 con 0x7f975c0a20e0 2021-02-10T11:39:12.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.407 7f976093f700 1 -- 172.21.2.117:0/3177794046 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802498", "target": ["mgr", ""]}) v1 -- 0x7f975c1b4f00 con 0x7f9740018730 2021-02-10T11:39:12.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.418 7f974affd700 1 -- 172.21.2.117:0/3177794046 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9750006420 con 0x7f9740018730 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 msgr2=0x7f974001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 0x7f974001ac10 crc :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 msgr2=0x7f975c0a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0x7f975400a8a0 tx=0x7f975400ac40).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 shutdown_connections 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9740018730 0x7f974001ac10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f975c0a20e0 0x7f975c0a24e0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f975c19db70 0x7f975c0a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 --2- 172.21.2.117:0/3177794046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f975c0a1090 0x7f975c19d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 >> 172.21.2.117:0/3177794046 conn(0x7f975c0ab050 msgr2=0x7f975c107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:12.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 shutdown_connections 2021-02-10T11:39:12.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.425 7f976093f700 1 -- 172.21.2.117:0/3177794046 wait complete. 2021-02-10T11:39:12.414 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802499 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 Processor -- start 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 -- start start 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 0x7f16c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 0x7f16c81b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16c80e51a0 0x7f16c8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c8077a70 con 0x7f16c80e51a0 2021-02-10T11:39:12.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c80777d0 con 0x7f16c807da20 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16cd800700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c807de20 con 0x7f16c80e4da0 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.853 7f16c7d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16c80e51a0 0x7f16c8077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16c80e51a0 0x7f16c8077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46750/0 (socket says 172.21.2.117:46750) 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 -- 172.21.2.117:0/2820986992 learned_addr learned my addr 172.21.2.117:0/2820986992 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c6d73700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 0x7f16c81b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7574700 1 --2- 172.21.2.117:0/2820986992 >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 0x7f16c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 -- 172.21.2.117:0/2820986992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 msgr2=0x7f16c81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 0x7f16c81b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 -- 172.21.2.117:0/2820986992 >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 msgr2=0x7f16c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 --2- 172.21.2.117:0/2820986992 >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 0x7f16c80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 -- 172.21.2.117:0/2820986992 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c81b0710 con 0x7f16c80e51a0 2021-02-10T11:39:12.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7574700 1 --2- 172.21.2.117:0/2820986992 >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 0x7f16c80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c6d73700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 0x7f16c81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c7d75700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80e51a0 0x7f16c8077ff0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f16c0006f00 tx=0x7f16c0009f10).ready entity=mon.0 client_cookie=70bd3639e97316a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.854 7f16c6572700 1 -- 172.21.2.117:0/2820986992 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c000b3b0 con 0x7f16c80e51a0 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16c6572700 1 -- 172.21.2.117:0/2820986992 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c0000d30 con 0x7f16c80e51a0 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16c6572700 1 -- 172.21.2.117:0/2820986992 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c000b8e0 con 0x7f16c80e51a0 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 -- 172.21.2.117:0/2820986992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80e51a0 msgr2=0x7f16c8077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80e51a0 0x7f16c8077ff0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f16c0006f00 tx=0x7f16c0009f10).stop 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 -- 172.21.2.117:0/2820986992 shutdown_connections 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80e51a0 0x7f16c8077ff0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 --2- 172.21.2.117:0/2820986992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c80e4da0 0x7f16c81b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 --2- 172.21.2.117:0/2820986992 >> v2:172.21.2.125:3300/0 conn(0x7f16c807da20 0x7f16c80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 -- 172.21.2.117:0/2820986992 >> 172.21.2.117:0/2820986992 conn(0x7f16c80ce450 msgr2=0x7f16c80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:12.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 -- 172.21.2.117:0/2820986992 shutdown_connections 2021-02-10T11:39:12.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.855 7f16cd800700 1 -- 172.21.2.117:0/2820986992 wait complete. 2021-02-10T11:39:12.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.870 7f16cd800700 1 Processor -- start 2021-02-10T11:39:12.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.870 7f16cd800700 1 -- start start 2021-02-10T11:39:12.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16c81048f0 0x7f16c8104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c81054f0 con 0x7f16c80b7fc0 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8195a30 con 0x7f16c81048f0 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16cd800700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8195fb0 con 0x7f16c8090710 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44432/0 (socket says 172.21.2.117:44432) 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46754/0 (socket says 172.21.2.117:46754) 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 -- 172.21.2.117:0/1474245552 learned_addr learned my addr 172.21.2.117:0/1474245552 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c6d73700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16c81048f0 0x7f16c8104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 -- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16c81048f0 msgr2=0x7f16c8104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16c81048f0 0x7f16c8104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 -- 172.21.2.117:0/1474245552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 msgr2=0x7f16c81042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 -- 172.21.2.117:0/1474245552 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c80b7b60 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7574700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.871 7f16c7d75700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f16c0006c50 tx=0x7f16c000acd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c0000d30 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c000c300 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16cd800700 1 -- 172.21.2.117:0/1474245552 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c80b7b60 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16cd800700 1 -- 172.21.2.117:0/1474245552 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c80b7b60 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16c000c820 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16c0000d30 con 0x7f16c8090710 2021-02-10T11:39:12.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 0x7f16a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:12.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f16c0032a00 con 0x7f16c8090710 2021-02-10T11:39:12.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.872 7f16c7574700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 0x7f16a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:12.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.873 7f16c7574700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 0x7f16a401abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:12.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.903 7f16cd800700 1 -- 172.21.2.117:0/1474245552 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c8103280 con 0x7f16c8090710 2021-02-10T11:39:12.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:12.907 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16c0003070 con 0x7f16c8090710 2021-02-10T11:39:13.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.038 7f16cd800700 1 -- 172.21.2.117:0/1474245552 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802499", "target": ["mgr", ""]}) v1 -- 0x7f16c8103280 con 0x7f16a40186c0 2021-02-10T11:39:13.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.048 7f16bffff700 1 -- 172.21.2.117:0/1474245552 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f16b80009b0 con 0x7f16a40186c0 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 msgr2=0x7f16a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 0x7f16a401abc0 crc :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 msgr2=0x7f16c8090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f16c0006c50 tx=0x7f16c000acd0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 shutdown_connections 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16c8090710 0x7f16c8090b10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16c81048f0 0x7f16c8104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16a40186c0 0x7f16a401abc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 --2- 172.21.2.117:0/1474245552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16c80b7fc0 0x7f16c81042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 >> 172.21.2.117:0/1474245552 conn(0x7f16c80a1850 msgr2=0x7f16c80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 shutdown_connections 2021-02-10T11:39:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.055 7f16cd800700 1 -- 172.21.2.117:0/1474245552 wait complete. 2021-02-10T11:39:13.044 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802500 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 Processor -- start 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 -- start start 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff67407da20 0x7ff6740771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff6740e51a0 0x7ff674079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff674077a70 con 0x7ff6740e4da0 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6740e5c20 con 0x7ff6740e51a0 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67bb73700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6740777d0 con 0x7ff67407da20 2021-02-10T11:39:13.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67a190700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff6740e51a0 0x7ff674079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67a190700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff6740e51a0 0x7ff674079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46760/0 (socket says 172.21.2.117:46760) 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67a190700 1 -- 172.21.2.117:0/1616004720 learned_addr learned my addr 172.21.2.117:0/1616004720 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.477 7ff67998f700 1 --2- 172.21.2.117:0/1616004720 >> v2:172.21.2.125:3300/0 conn(0x7ff67407da20 0x7ff6740771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67a190700 1 -- 172.21.2.117:0/1616004720 >> v2:172.21.2.125:3300/0 conn(0x7ff67407da20 msgr2=0x7ff6740771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67a190700 1 --2- 172.21.2.117:0/1616004720 >> v2:172.21.2.125:3300/0 conn(0x7ff67407da20 0x7ff6740771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67a190700 1 -- 172.21.2.117:0/1616004720 >> v1:172.21.2.125:6789/0 conn(0x7ff6740e4da0 legacy=0x7ff6741b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67a190700 1 -- 172.21.2.117:0/1616004720 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6741b08e0 con 0x7ff6740e51a0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67a190700 1 --2- 172.21.2.117:0/1616004720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6740e51a0 0x7ff674079000 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7ff66c006f00 tx=0x7ff66c009f10).ready entity=mon.0 client_cookie=50777b21e68d56f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67898d700 1 -- 172.21.2.117:0/1616004720 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff66c00b3b0 con 0x7ff6740e51a0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67898d700 1 -- 172.21.2.117:0/1616004720 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c000d30 con 0x7ff6740e51a0 2021-02-10T11:39:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67898d700 1 -- 172.21.2.117:0/1616004720 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff66c00b8a0 con 0x7ff6740e51a0 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67bb73700 1 -- 172.21.2.117:0/1616004720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6740e51a0 msgr2=0x7ff674079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.478 7ff67bb73700 1 --2- 172.21.2.117:0/1616004720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6740e51a0 0x7ff674079000 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7ff66c006f00 tx=0x7ff66c009f10).stop 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 -- 172.21.2.117:0/1616004720 shutdown_connections 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 --2- 172.21.2.117:0/1616004720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6740e51a0 0x7ff674079000 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 --2- 172.21.2.117:0/1616004720 >> v2:172.21.2.125:3300/0 conn(0x7ff67407da20 0x7ff6740771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 -- 172.21.2.117:0/1616004720 >> 172.21.2.117:0/1616004720 conn(0x7ff6740ce450 msgr2=0x7ff6740e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 -- 172.21.2.117:0/1616004720 shutdown_connections 2021-02-10T11:39:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.479 7ff67bb73700 1 -- 172.21.2.117:0/1616004720 wait complete. 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.493 7ff67bb73700 1 Processor -- start 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 -- start start 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6740b7fc0 0x7ff6741042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 0x7ff674104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 0x7ff674090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6741054f0 con 0x7ff674090710 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674195a30 con 0x7ff6740b7fc0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67bb73700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff674195fb0 con 0x7ff6741048f0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 0x7ff674104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67a190700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 0x7ff674090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67a190700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 0x7ff674090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46764/0 (socket says 172.21.2.117:46764) 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67a190700 1 -- 172.21.2.117:0/3892605989 learned_addr learned my addr 172.21.2.117:0/3892605989 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67998f700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6740b7fc0 0x7ff6741042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 -- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6740b7fc0 msgr2=0x7ff6741042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6740b7fc0 0x7ff6741042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 -- 172.21.2.117:0/3892605989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 msgr2=0x7ff674090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 0x7ff674090b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.494 7ff67918e700 1 -- 172.21.2.117:0/3892605989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6740b7b60 con 0x7ff6741048f0 2021-02-10T11:39:13.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff67918e700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 0x7ff674104cf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff668006e80 tx=0x7ff66800b8b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6680080d0 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff668008410 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6740b7b60 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff668008870 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.495 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6740b7b60 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.496 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff6680085f0 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.496 7ff6667fc700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 0x7ff65c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.496 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff6680325d0 con 0x7ff6741048f0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.496 7ff67998f700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 0x7ff65c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:13.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.496 7ff67998f700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 0x7ff65c01abc0 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:13.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.527 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff674103280 con 0x7ff6741048f0 2021-02-10T11:39:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.530 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff668006b50 con 0x7ff6741048f0 2021-02-10T11:39:13.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.660 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802500", "target": ["mgr", ""]}) v1 -- 0x7ff674103280 con 0x7ff65c0186c0 2021-02-10T11:39:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.672 7ff6667fc700 1 -- 172.21.2.117:0/3892605989 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff670006a70 con 0x7ff65c0186c0 2021-02-10T11:39:13.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 msgr2=0x7ff65c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 0x7ff65c01abc0 crc :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 msgr2=0x7ff674104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:13.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 0x7ff674104cf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff668006e80 tx=0x7ff66800b8b0).stop 2021-02-10T11:39:13.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 shutdown_connections 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff65c0186c0 0x7ff65c01abc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff674090710 0x7ff674090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6741048f0 0x7ff674104cf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 --2- 172.21.2.117:0/3892605989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6740b7fc0 0x7ff6741042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 >> 172.21.2.117:0/3892605989 conn(0x7ff6740a1850 msgr2=0x7ff6740c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 shutdown_connections 2021-02-10T11:39:13.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:13.678 7ff67bb73700 1 -- 172.21.2.117:0/3892605989 wait complete. 2021-02-10T11:39:13.667 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802501 2021-02-10T11:39:14.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.100 7f28cd8b7700 1 Processor -- start 2021-02-10T11:39:14.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.100 7f28cd8b7700 1 -- start start 2021-02-10T11:39:14.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28cd8b7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c8079000 0x7f28c80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28cd8b7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28c81b00f0 0x7f28c81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28cd8b7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c8077a70 con 0x7f28c807da20 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28cd8b7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c80e53c0 con 0x7f28c81b00f0 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28cd8b7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8079400 con 0x7f28c8079000 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f28c807da20 0x7f28c80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47022/0 (socket says 172.21.2.117:47022) 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7574700 1 -- 172.21.2.117:0/2939258097 learned_addr learned my addr 172.21.2.117:0/2939258097 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7d75700 1 --2- 172.21.2.117:0/2939258097 >> v2:172.21.2.117:3300/0 conn(0x7f28c81b00f0 0x7f28c81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c6d73700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c8079000 0x7f28c80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7d75700 1 -- 172.21.2.117:0/2939258097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c8079000 msgr2=0x7f28c80e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7d75700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c8079000 0x7f28c80e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7d75700 1 -- 172.21.2.117:0/2939258097 >> v1:172.21.2.117:6789/0 conn(0x7f28c807da20 legacy=0x7f28c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.101 7f28c7d75700 1 -- 172.21.2.117:0/2939258097 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c807de20 con 0x7f28c81b00f0 2021-02-10T11:39:14.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28c7d75700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c81b00f0 0x7f28c81b04f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f28b8006ab0 tx=0x7f28b8009b40).ready entity=mon.0 client_cookie=789157e039fd0c50 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28c6572700 1 -- 172.21.2.117:0/2939258097 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b800c0c0 con 0x7f28c81b00f0 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28c6572700 1 -- 172.21.2.117:0/2939258097 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b800c3c0 con 0x7f28c81b00f0 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28c6572700 1 -- 172.21.2.117:0/2939258097 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b800c820 con 0x7f28c81b00f0 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 -- 172.21.2.117:0/2939258097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c81b00f0 msgr2=0x7f28c81b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c81b00f0 0x7f28c81b04f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f28b8006ab0 tx=0x7f28b8009b40).stop 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 -- 172.21.2.117:0/2939258097 shutdown_connections 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c81b00f0 0x7f28c81b04f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 --2- 172.21.2.117:0/2939258097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c8079000 0x7f28c80e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 -- 172.21.2.117:0/2939258097 >> 172.21.2.117:0/2939258097 conn(0x7f28c80ce450 msgr2=0x7f28c80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 -- 172.21.2.117:0/2939258097 shutdown_connections 2021-02-10T11:39:14.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.102 7f28cd8b7700 1 -- 172.21.2.117:0/2939258097 wait complete. 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 Processor -- start 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 -- start start 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80a41c0 0x7f28c80fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 0x7f28c80c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 0x7f28c80b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80b8010 con 0x7f28c80c2340 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80b8550 con 0x7f28c80a41c0 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28cd8b7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80b8770 con 0x7f28c80b7390 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28c6d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 0x7f28c80c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28c7d75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 0x7f28c80b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28c7d75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 0x7f28c80b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44458/0 (socket says 172.21.2.117:44458) 2021-02-10T11:39:14.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28c7d75700 1 -- 172.21.2.117:0/185513994 learned_addr learned my addr 172.21.2.117:0/185513994 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.117 7f28c7574700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80a41c0 0x7f28c80fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 -- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 msgr2=0x7f28c80b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 0x7f28c80b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 -- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80a41c0 msgr2=0x7f28c80fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80a41c0 0x7f28c80fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 -- 172.21.2.117:0/185513994 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80b8990 con 0x7f28c80c2340 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28c6d73700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 0x7f28c80c2740 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f28c0009920 tx=0x7f28c000bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28c0006240 con 0x7f28c80c2340 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c0005660 con 0x7f28c80c2340 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28c0005b40 con 0x7f28c80c2340 2021-02-10T11:39:14.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c80b8990 con 0x7f28c80c2340 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.118 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c80b8990 con 0x7f28c80c2340 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.119 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28c0005840 con 0x7f28c80c2340 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.119 7f28affff700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 0x7f28a801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.119 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28c0032b90 con 0x7f28c80c2340 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.119 7f28c7574700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 0x7f28a801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.120 7f28c7574700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 0x7f28a801ac10 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.150 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c81b4f00 con 0x7f28c80c2340 2021-02-10T11:39:14.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.153 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28c0005870 con 0x7f28c80c2340 2021-02-10T11:39:14.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.278 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802501", "target": ["mgr", ""]}) v1 -- 0x7f28c81b4f00 con 0x7f28a8018730 2021-02-10T11:39:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.289 7f28affff700 1 -- 172.21.2.117:0/185513994 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f28bc007740 con 0x7f28a8018730 2021-02-10T11:39:14.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.295 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 msgr2=0x7f28a801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.295 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 0x7f28a801ac10 crc :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.295 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 msgr2=0x7f28c80c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.295 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 0x7f28c80c2740 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f28c0009920 tx=0x7f28c000bc10).stop 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 shutdown_connections 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28a8018730 0x7f28a801ac10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c80c2340 0x7f28c80c2740 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b7390 0x7f28c80b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 --2- 172.21.2.117:0/185513994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80a41c0 0x7f28c80fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 >> 172.21.2.117:0/185513994 conn(0x7f28c808bbc0 msgr2=0x7f28c80c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 shutdown_connections 2021-02-10T11:39:14.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.296 7f28cd8b7700 1 -- 172.21.2.117:0/185513994 wait complete. 2021-02-10T11:39:14.285 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802502 2021-02-10T11:39:14.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 Processor -- start 2021-02-10T11:39:14.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 -- start start 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f730c07da20 0x7f730c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 0x7f730c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 0x7f730c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c077a70 con 0x7f730c07da20 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c0777d0 con 0x7f730c0e51a0 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.718 7f731406e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c07de20 con 0x7f730c0e4da0 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f730c07da20 0x7f730c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311689700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 0x7f730c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f730c07da20 0x7f730c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46788/0 (socket says 172.21.2.117:46788) 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311689700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 0x7f730c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44466/0 (socket says 172.21.2.117:44466) 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 -- 172.21.2.117:0/1280849202 learned_addr learned my addr 172.21.2.117:0/1280849202 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:14.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f731268b700 1 --2- 172.21.2.117:0/1280849202 >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 0x7f730c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 -- 172.21.2.117:0/1280849202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 msgr2=0x7f730c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 --2- 172.21.2.117:0/1280849202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 0x7f730c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 -- 172.21.2.117:0/1280849202 >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 msgr2=0x7f730c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 --2- 172.21.2.117:0/1280849202 >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 0x7f730c077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 -- 172.21.2.117:0/1280849202 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c1b0710 con 0x7f730c07da20 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f731268b700 1 --2- 172.21.2.117:0/1280849202 >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 0x7f730c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7311e8a700 1 --2- 172.21.2.117:0/1280849202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c07da20 0x7f730c0771b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f72fc006700 tx=0x7f72fc009b80).ready entity=mon.0 client_cookie=a056713814ef13d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.719 7f7310e88700 1 -- 172.21.2.117:0/1280849202 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72fc00c0c0 con 0x7f730c07da20 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f7310e88700 1 -- 172.21.2.117:0/1280849202 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc00c440 con 0x7f730c07da20 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f7310e88700 1 -- 172.21.2.117:0/1280849202 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72fc00c8a0 con 0x7f730c07da20 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 -- 172.21.2.117:0/1280849202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c07da20 msgr2=0x7f730c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 --2- 172.21.2.117:0/1280849202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c07da20 0x7f730c0771b0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f72fc006700 tx=0x7f72fc009b80).stop 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 -- 172.21.2.117:0/1280849202 shutdown_connections 2021-02-10T11:39:14.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 --2- 172.21.2.117:0/1280849202 >> v2:172.21.2.125:3300/0 conn(0x7f730c0e51a0 0x7f730c077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 --2- 172.21.2.117:0/1280849202 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c0e4da0 0x7f730c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 --2- 172.21.2.117:0/1280849202 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c07da20 0x7f730c0771b0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 -- 172.21.2.117:0/1280849202 >> 172.21.2.117:0/1280849202 conn(0x7f730c0ce450 msgr2=0x7f730c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 -- 172.21.2.117:0/1280849202 shutdown_connections 2021-02-10T11:39:14.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.720 7f731406e700 1 -- 172.21.2.117:0/1280849202 wait complete. 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 Processor -- start 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 -- start start 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f730c187e20 0x7f730c188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c090420 0x7f730c090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c1067f0 con 0x7f730c1a0250 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c107350 con 0x7f730c187e20 2021-02-10T11:39:14.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.735 7f731406e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c1042d0 con 0x7f730c090420 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46792/0 (socket says 172.21.2.117:46792) 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 -- 172.21.2.117:0/3015943164 learned_addr learned my addr 172.21.2.117:0/3015943164 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f731268b700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c090420 0x7f730c090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311689700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f730c187e20 0x7f730c188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 -- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c090420 msgr2=0x7f730c090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c090420 0x7f730c090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 -- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f730c187e20 msgr2=0x7f730c188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f730c187e20 0x7f730c188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 -- 172.21.2.117:0/3015943164 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c090e40 con 0x7f730c1a0250 2021-02-10T11:39:14.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.736 7f7311e8a700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f72fc00b5e0 tx=0x7f72fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72fc00bd70 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc002030 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72fc002550 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f731406e700 1 -- 172.21.2.117:0/3015943164 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c090e40 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f731406e700 1 -- 172.21.2.117:0/3015943164 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c090e40 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72fc000c90 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 0x7f72f401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.737 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f72fc032800 con 0x7f730c1a0250 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.738 7f7311689700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 0x7f72f401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:14.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.738 7f7311689700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 0x7f72f401ac10 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:14.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.768 7f731406e700 1 -- 172.21.2.117:0/3015943164 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c103280 con 0x7f730c1a0250 2021-02-10T11:39:14.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.771 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72fc01f570 con 0x7f730c1a0250 2021-02-10T11:39:14.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.900 7f731406e700 1 -- 172.21.2.117:0/3015943164 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802502", "target": ["mgr", ""]}) v1 -- 0x7f730c107310 con 0x7f72f4018730 2021-02-10T11:39:14.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.911 7f73027fc700 1 -- 172.21.2.117:0/3015943164 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f7304006ce0 con 0x7f72f4018730 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 msgr2=0x7f72f401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 0x7f72f401ac10 crc :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 msgr2=0x7f730c106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f72fc00b5e0 tx=0x7f72fc00b980).stop 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 shutdown_connections 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f730c090420 0x7f730c090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f730c187e20 0x7f730c188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72f4018730 0x7f72f401ac10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 --2- 172.21.2.117:0/3015943164 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f730c1a0250 0x7f730c106cf0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:14.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 >> 172.21.2.117:0/3015943164 conn(0x7f730c0a1850 msgr2=0x7f730c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:14.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 shutdown_connections 2021-02-10T11:39:14.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:14.918 7f731406e700 1 -- 172.21.2.117:0/3015943164 wait complete. 2021-02-10T11:39:14.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802503 2021-02-10T11:39:15.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 Processor -- start 2021-02-10T11:39:15.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 -- start start 2021-02-10T11:39:15.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f687007da20 0x7f68700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 0x7f68701b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6870077a70 con 0x7f68700e5180 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700777d0 con 0x7f68700e4d80 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f6875246700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687007de20 con 0x7f687007da20 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.340 7f686f59e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f68700e5180 0x7f6870077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47054/0 (socket says 172.21.2.117:47054) 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686f59e700 1 -- 172.21.2.117:0/2354629624 learned_addr learned my addr 172.21.2.117:0/2354629624 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686e59c700 1 --2- 172.21.2.117:0/2354629624 >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 0x7f68701b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 --2- 172.21.2.117:0/2354629624 >> v2:172.21.2.125:3300/0 conn(0x7f687007da20 0x7f68700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 -- 172.21.2.117:0/2354629624 >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 msgr2=0x7f68701b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 --2- 172.21.2.117:0/2354629624 >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 0x7f68701b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 -- 172.21.2.117:0/2354629624 >> v1:172.21.2.117:6789/0 conn(0x7f68700e5180 legacy=0x7f6870077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 -- 172.21.2.117:0/2354629624 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68701b06f0 con 0x7f687007da20 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686e59c700 1 --2- 172.21.2.117:0/2354629624 >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 0x7f68701b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:15.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686ed9d700 1 --2- 172.21.2.117:0/2354629624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f687007da20 0x7f68700771b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f6860006650 tx=0x7f6860009a80).ready entity=mon.1 client_cookie=b421d37e6103a407 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686dd9b700 1 -- 172.21.2.117:0/2354629624 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686000c0c0 con 0x7f687007da20 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686dd9b700 1 -- 172.21.2.117:0/2354629624 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686000c340 con 0x7f687007da20 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f686dd9b700 1 -- 172.21.2.117:0/2354629624 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686000c820 con 0x7f687007da20 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f6875246700 1 -- 172.21.2.117:0/2354629624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f687007da20 msgr2=0x7f68700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.341 7f6875246700 1 --2- 172.21.2.117:0/2354629624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f687007da20 0x7f68700771b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f6860006650 tx=0x7f6860009a80).stop 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 -- 172.21.2.117:0/2354629624 shutdown_connections 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 --2- 172.21.2.117:0/2354629624 >> v2:172.21.2.117:3300/0 conn(0x7f68700e4d80 0x7f68701b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 --2- 172.21.2.117:0/2354629624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f687007da20 0x7f68700771b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 -- 172.21.2.117:0/2354629624 >> 172.21.2.117:0/2354629624 conn(0x7f68700ce430 msgr2=0x7f68700e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 -- 172.21.2.117:0/2354629624 shutdown_connections 2021-02-10T11:39:15.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.342 7f6875246700 1 -- 172.21.2.117:0/2354629624 wait complete. 2021-02-10T11:39:15.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 Processor -- start 2021-02-10T11:39:15.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 -- start start 2021-02-10T11:39:15.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68700b8b60 0x7f68700b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68700b8f60 0x7f68700b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870091b90 con 0x7f6870095bc0 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700b2d00 con 0x7f68700b8b60 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f6875246700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700916d0 con 0x7f68700b8f60 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f686f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f686f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46810/0 (socket says 172.21.2.117:46810) 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f686f59e700 1 -- 172.21.2.117:0/3061611552 learned_addr learned my addr 172.21.2.117:0/3061611552 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f686e59c700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68700b8f60 0x7f68700b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.357 7f686ed9d700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68700b8b60 0x7f68700b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 -- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68700b8f60 msgr2=0x7f68700b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68700b8f60 0x7f68700b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 -- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68700b8b60 msgr2=0x7f68700b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68700b8b60 0x7f68700b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 -- 172.21.2.117:0/3061611552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700b8200 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f686f59e700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f6868001400 tx=0x7f686800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68680009b0 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686800acf0 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6868004620 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f6875246700 1 -- 172.21.2.117:0/3061611552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700b8200 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.358 7f6875246700 1 -- 172.21.2.117:0/3061611552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700968d0 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.359 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68680009b0 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.359 7f68677fe700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 0x7f685401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.359 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f68680326c0 con 0x7f6870095bc0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.359 7f686ed9d700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 0x7f685401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.360 7f686ed9d700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 0x7f685401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.390 7f6875246700 1 -- 172.21.2.117:0/3061611552 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6870091b90 con 0x7f6870095bc0 2021-02-10T11:39:15.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.393 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6868026a70 con 0x7f6870095bc0 2021-02-10T11:39:15.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.519 7f6875246700 1 -- 172.21.2.117:0/3061611552 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802503", "target": ["mgr", ""]}) v1 -- 0x7f6870091b90 con 0x7f68540186c0 2021-02-10T11:39:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.528 7f68677fe700 1 -- 172.21.2.117:0/3061611552 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6860006420 con 0x7f68540186c0 2021-02-10T11:39:15.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 msgr2=0x7f685401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 0x7f685401abc0 crc :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 msgr2=0x7f6870095fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f6868001400 tx=0x7f686800a900).stop 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 shutdown_connections 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68540186c0 0x7f685401abc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6870095bc0 0x7f6870095fc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68700b8f60 0x7f68700b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 --2- 172.21.2.117:0/3061611552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68700b8b60 0x7f68700b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 >> 172.21.2.117:0/3061611552 conn(0x7f68700a4170 msgr2=0x7f687009be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 shutdown_connections 2021-02-10T11:39:15.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.536 7f6875246700 1 -- 172.21.2.117:0/3061611552 wait complete. 2021-02-10T11:39:15.526 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802504 2021-02-10T11:39:15.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.956 7fc7bcf90700 1 Processor -- start 2021-02-10T11:39:15.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 -- start start 2021-02-10T11:39:15.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 0x7fc7b80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 0x7fc7b81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b8077a70 con 0x7fc7b807da20 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b80e53c0 con 0x7fc7b81b00f0 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7bcf90700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b8079400 con 0x7fc7b8079000 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7b6d9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc7b807da20 0x7fc7b80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47064/0 (socket says 172.21.2.117:47064) 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.957 7fc7b6d9d700 1 -- 172.21.2.117:0/3275535085 learned_addr learned my addr 172.21.2.117:0/3275535085 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 --2- 172.21.2.117:0/3275535085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 0x7fc7b80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b759e700 1 --2- 172.21.2.117:0/3275535085 >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 0x7fc7b81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b5d9b700 1 -- 172.21.2.117:0/3275535085 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780165420 0 0) 0x7fc7ac001180 con 0x7fc7b807da20 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b5d9b700 1 -- 172.21.2.117:0/3275535085 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a0002fe0 con 0x7fc7b807da20 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 -- 172.21.2.117:0/3275535085 >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 msgr2=0x7fc7b81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 --2- 172.21.2.117:0/3275535085 >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 0x7fc7b81b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 -- 172.21.2.117:0/3275535085 >> v1:172.21.2.117:6789/0 conn(0x7fc7b807da20 legacy=0x7fc7b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 -- 172.21.2.117:0/3275535085 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b807de20 con 0x7fc7b8079000 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b759e700 1 --2- 172.21.2.117:0/3275535085 >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 0x7fc7b81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b659c700 1 --2- 172.21.2.117:0/3275535085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 0x7fc7b80e4da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc7b00069d0 tx=0x7fc7b0009e40).ready entity=mon.2 client_cookie=b6ce883187677b83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b5d9b700 1 -- 172.21.2.117:0/3275535085 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7b00066c0 con 0x7fc7b8079000 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b5d9b700 1 -- 172.21.2.117:0/3275535085 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b000b510 con 0x7fc7b8079000 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.958 7fc7b5d9b700 1 -- 172.21.2.117:0/3275535085 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7b000ba30 con 0x7fc7b8079000 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 -- 172.21.2.117:0/3275535085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 msgr2=0x7fc7b80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 --2- 172.21.2.117:0/3275535085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 0x7fc7b80e4da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc7b00069d0 tx=0x7fc7b0009e40).stop 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 -- 172.21.2.117:0/3275535085 shutdown_connections 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 --2- 172.21.2.117:0/3275535085 >> v2:172.21.2.125:3300/0 conn(0x7fc7b81b00f0 0x7fc7b81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 --2- 172.21.2.117:0/3275535085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8079000 0x7fc7b80e4da0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 -- 172.21.2.117:0/3275535085 >> 172.21.2.117:0/3275535085 conn(0x7fc7b80ce450 msgr2=0x7fc7b80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 -- 172.21.2.117:0/3275535085 shutdown_connections 2021-02-10T11:39:15.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.959 7fc7bcf90700 1 -- 172.21.2.117:0/3275535085 wait complete. 2021-02-10T11:39:15.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 Processor -- start 2021-02-10T11:39:15.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 -- start start 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7b81a0250 0x7fc7b8106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 0x7fc7b8188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b81067f0 con 0x7fc7b8090420 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b8107350 con 0x7fc7b81a0250 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7bcf90700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b81042d0 con 0x7fc7b8187e20 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7b759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7b759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46822/0 (socket says 172.21.2.117:46822) 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7b759e700 1 -- 172.21.2.117:0/419396444 learned_addr learned my addr 172.21.2.117:0/419396444 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.974 7fc7b6d9d700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7b81a0250 0x7fc7b8106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b659c700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 0x7fc7b8188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 -- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 msgr2=0x7fc7b8188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 0x7fc7b8188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 -- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7b81a0250 msgr2=0x7fc7b8106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7b81a0250 0x7fc7b8106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 -- 172.21.2.117:0/419396444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8090e40 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b659c700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 0x7fc7b8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7b759e700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7fc7a800b5e0 tx=0x7fc7a800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a800bd70 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a8006410 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b8090e40 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7a80024d0 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.975 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b80913d0 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.976 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc7a80020d0 con 0x7fc7b8090420 2021-02-10T11:39:15.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.976 7fc7a77fe700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 0x7fc79c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:15.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.976 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc7a8032780 con 0x7fc7b8090420 2021-02-10T11:39:15.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.976 7fc7b6d9d700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 0x7fc79c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:15.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:15.977 7fc7b6d9d700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 0x7fc79c01abc0 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:15.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.007 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b8103280 con 0x7fc7b8090420 2021-02-10T11:39:15.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.009 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc7a8000c10 con 0x7fc7b8090420 2021-02-10T11:39:16.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.138 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802504", "target": ["mgr", ""]}) v1 -- 0x7fc7b8091c60 con 0x7fc79c018730 2021-02-10T11:39:16.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.148 7fc7a77fe700 1 -- 172.21.2.117:0/419396444 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc7ac000ce0 con 0x7fc79c018730 2021-02-10T11:39:16.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 msgr2=0x7fc79c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 0x7fc79c01abc0 crc :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 msgr2=0x7fc7b8090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7fc7a800b5e0 tx=0x7fc7a800b980).stop 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 shutdown_connections 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc79c018730 0x7fc79c01abc0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7b8090420 0x7fc7b8090820 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7b8187e20 0x7fc7b8188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 --2- 172.21.2.117:0/419396444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7b81a0250 0x7fc7b8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 >> 172.21.2.117:0/419396444 conn(0x7fc7b80a1850 msgr2=0x7fc7b80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.154 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 shutdown_connections 2021-02-10T11:39:16.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.155 7fc7bcf90700 1 -- 172.21.2.117:0/419396444 wait complete. 2021-02-10T11:39:16.144 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802505 2021-02-10T11:39:16.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 Processor -- start 2021-02-10T11:39:16.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 -- start start 2021-02-10T11:39:16.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 0x7fe3700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 0x7fe3701b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe370077a70 con 0x7fe3700e51a0 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3700e5c20 con 0x7fe3700e4da0 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe375dfd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700777d0 con 0x7fe37007da20 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe36effd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 0x7fe3701b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe36ffff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fe3700e51a0 0x7fe370077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40006/0 (socket says 172.21.2.117:40006) 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe36ffff700 1 -- 172.21.2.117:0/1221411597 learned_addr learned my addr 172.21.2.117:0/1221411597 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.576 7fe36f7fe700 1 --2- 172.21.2.117:0/1221411597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 0x7fe3700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36e7fc700 1 -- 172.21.2.117:0/1221411597 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777182554 0 0) 0x7fe368001510 con 0x7fe3700e51a0 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36e7fc700 1 -- 172.21.2.117:0/1221411597 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c002fe0 con 0x7fe3700e51a0 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36f7fe700 1 -- 172.21.2.117:0/1221411597 >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 msgr2=0x7fe3701b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36f7fe700 1 --2- 172.21.2.117:0/1221411597 >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 0x7fe3701b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36f7fe700 1 -- 172.21.2.117:0/1221411597 >> v1:172.21.2.125:6789/0 conn(0x7fe3700e51a0 legacy=0x7fe370077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36f7fe700 1 -- 172.21.2.117:0/1221411597 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37007de20 con 0x7fe37007da20 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36effd700 1 --2- 172.21.2.117:0/1221411597 >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 0x7fe3701b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36f7fe700 1 --2- 172.21.2.117:0/1221411597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 0x7fe3700771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe358006650 tx=0x7fe358009a80).ready entity=mon.2 client_cookie=573cede2e5594cb3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36e7fc700 1 -- 172.21.2.117:0/1221411597 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3580009b0 con 0x7fe37007da20 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36e7fc700 1 -- 172.21.2.117:0/1221411597 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358006390 con 0x7fe37007da20 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe375dfd700 1 -- 172.21.2.117:0/1221411597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 msgr2=0x7fe3700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe375dfd700 1 --2- 172.21.2.117:0/1221411597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 0x7fe3700771b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe358006650 tx=0x7fe358009a80).stop 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.577 7fe36e7fc700 1 -- 172.21.2.117:0/1221411597 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35800c0c0 con 0x7fe37007da20 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 -- 172.21.2.117:0/1221411597 shutdown_connections 2021-02-10T11:39:16.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 --2- 172.21.2.117:0/1221411597 >> v2:172.21.2.125:3300/0 conn(0x7fe3700e4da0 0x7fe3701b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 --2- 172.21.2.117:0/1221411597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe37007da20 0x7fe3700771b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 -- 172.21.2.117:0/1221411597 >> 172.21.2.117:0/1221411597 conn(0x7fe3700ce450 msgr2=0x7fe3700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 -- 172.21.2.117:0/1221411597 shutdown_connections 2021-02-10T11:39:16.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.578 7fe375dfd700 1 -- 172.21.2.117:0/1221411597 wait complete. 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 Processor -- start 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 -- start start 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 0x7fe3701b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 0x7fe3700b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700b8da0 con 0x7fe3701b5130 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700a4140 con 0x7fe3701b5f50 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.593 7fe375dfd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3700a4320 con 0x7fe3701b5b50 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 0x7fe3701b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36effd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36effd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44516/0 (socket says 172.21.2.117:44516) 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36effd700 1 -- 172.21.2.117:0/1611263746 learned_addr learned my addr 172.21.2.117:0/1611263746 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36ffff700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 0x7fe3700b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 -- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 msgr2=0x7fe3700b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 -- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 msgr2=0x7fe3700b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 0x7fe3700b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 -- 172.21.2.117:0/1611263746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3700b62b0 con 0x7fe3701b5130 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36effd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36ffff700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 0x7fe3700b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe36f7fe700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 0x7fe3701b5530 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fe358006e90 tx=0x7fe35800b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3580009b0 con 0x7fe3701b5130 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358006740 con 0x7fe3701b5130 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3700b62b0 con 0x7fe3701b5130 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3580052e0 con 0x7fe3701b5130 2021-02-10T11:39:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.594 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3700b6840 con 0x7fe3701b5130 2021-02-10T11:39:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.595 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe35800a000 con 0x7fe3701b5130 2021-02-10T11:39:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.595 7fe367fff700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.595 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe3580324e0 con 0x7fe3701b5130 2021-02-10T11:39:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.595 7fe36effd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.596 7fe36effd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:16.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.626 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3701b4f00 con 0x7fe3701b5130 2021-02-10T11:39:16.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.629 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe3580009b0 con 0x7fe3701b5130 2021-02-10T11:39:16.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.753 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802505", "target": ["mgr", ""]}) v1 -- 0x7fe3700b6840 con 0x7fe3540186c0 2021-02-10T11:39:16.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.763 7fe367fff700 1 -- 172.21.2.117:0/1611263746 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe3600009b0 con 0x7fe3540186c0 2021-02-10T11:39:16.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 msgr2=0x7fe35401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 crc :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 msgr2=0x7fe3701b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 0x7fe3701b5530 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0x7fe358006e90 tx=0x7fe35800b8c0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 shutdown_connections 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3701b5f50 0x7fe3700b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3701b5b50 0x7fe3700b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 --2- 172.21.2.117:0/1611263746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3701b5130 0x7fe3701b5530 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:16.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 >> 172.21.2.117:0/1611263746 conn(0x7fe37008bbc0 msgr2=0x7fe3700c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:16.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 shutdown_connections 2021-02-10T11:39:16.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:16.771 7fe375dfd700 1 -- 172.21.2.117:0/1611263746 wait complete. 2021-02-10T11:39:16.761 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802506 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.193 7fb5f6575700 1 Processor -- start 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.193 7fb5f6575700 1 -- start start 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.193 7fb5f6575700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 0x7fb5f00e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5f6575700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00d1d20 con 0x7fb5f007da20 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5f6575700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f0077a70 con 0x7fb5f01b00f0 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5f6575700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00e5430 con 0x7fb5f0079000 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5effff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb5f007da20 0x7fb5f00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47090/0 (socket says 172.21.2.117:47090) 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5effff700 1 -- 172.21.2.117:0/3568265532 learned_addr learned my addr 172.21.2.117:0/3568265532 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:17.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5ef7fe700 1 --2- 172.21.2.117:0/3568265532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 0x7fb5f00e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616140260 0 0) 0x7fb5e0000ce0 con 0x7fb5f007da20 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8002fe0 con 0x7fb5f007da20 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386805781 0 0) 0x7fb5e8000f90 con 0x7fb5f01b00f0 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5ef7fe700 1 -- 172.21.2.117:0/3568265532 >> v1:172.21.2.125:6789/0 conn(0x7fb5f01b00f0 legacy=0x7fb5f00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5ef7fe700 1 -- 172.21.2.117:0/3568265532 >> v1:172.21.2.117:6789/0 conn(0x7fb5f007da20 legacy=0x7fb5f00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.194 7fb5ef7fe700 1 -- 172.21.2.117:0/3568265532 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0079400 con 0x7fb5f0079000 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5ef7fe700 1 --2- 172.21.2.117:0/3568265532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 0x7fb5f00e4da0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb5e4006650 tx=0x7fb5e4009a00).ready entity=mon.2 client_cookie=562a894d2c8fb7e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e4009df0 con 0x7fb5f0079000 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40009b0 con 0x7fb5f0079000 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5eeffd700 1 -- 172.21.2.117:0/3568265532 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e400c2b0 con 0x7fb5f0079000 2021-02-10T11:39:17.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5f6575700 1 -- 172.21.2.117:0/3568265532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 msgr2=0x7fb5f00e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.195 7fb5f6575700 1 --2- 172.21.2.117:0/3568265532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 0x7fb5f00e4da0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb5e4006650 tx=0x7fb5e4009a00).stop 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.196 7fb5f6575700 1 -- 172.21.2.117:0/3568265532 shutdown_connections 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.196 7fb5f6575700 1 --2- 172.21.2.117:0/3568265532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0079000 0x7fb5f00e4da0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.196 7fb5f6575700 1 -- 172.21.2.117:0/3568265532 >> 172.21.2.117:0/3568265532 conn(0x7fb5f00ce450 msgr2=0x7fb5f00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.196 7fb5f6575700 1 -- 172.21.2.117:0/3568265532 shutdown_connections 2021-02-10T11:39:17.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.196 7fb5f6575700 1 -- 172.21.2.117:0/3568265532 wait complete. 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.211 7fb5f6575700 1 Processor -- start 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.211 7fb5f6575700 1 -- start start 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5f01051f0 0x7fb5f01055f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0105c10 0x7fb5f018b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f018fbe0 con 0x7fb5f0106010 2021-02-10T11:39:17.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00c2740 con 0x7fb5f01051f0 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f6575700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00c2c00 con 0x7fb5f0105c10 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46848/0 (socket says 172.21.2.117:46848) 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 -- 172.21.2.117:0/3591952250 learned_addr learned my addr 172.21.2.117:0/3591952250 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5ef7fe700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0105c10 0x7fb5f018b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5effff700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5f01051f0 0x7fb5f01055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 -- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0105c10 msgr2=0x7fb5f018b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0105c10 0x7fb5f018b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 -- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5f01051f0 msgr2=0x7fb5f01055f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.212 7fb5f4b92700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5f01051f0 0x7fb5f01055f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5f4b92700 1 -- 172.21.2.117:0/3591952250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f018bca0 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5f4b92700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7fb5e8009940 tx=0x7fb5e800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e800c2f0 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e800c570 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5e800c9d0 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f018bca0 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.213 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f018c230 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.214 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5e800c750 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.214 7fb5ecff9700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.214 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb5e8032910 con 0x7fb5f0106010 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.214 7fb5effff700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.214 7fb5effff700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 0x7fb5d001abc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.243 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f0103370 con 0x7fb5f0106010 2021-02-10T11:39:17.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.246 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5e800c450 con 0x7fb5f0106010 2021-02-10T11:39:17.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.374 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802506", "target": ["mgr", ""]}) v1 -- 0x7fb5f0077a70 con 0x7fb5d00186c0 2021-02-10T11:39:17.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.386 7fb5ecff9700 1 -- 172.21.2.117:0/3591952250 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb5e0006b40 con 0x7fb5d00186c0 2021-02-10T11:39:17.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 msgr2=0x7fb5d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 0x7fb5d001abc0 crc :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 msgr2=0x7fb5f018b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7fb5e8009940 tx=0x7fb5e800bcf0).stop 2021-02-10T11:39:17.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 shutdown_connections 2021-02-10T11:39:17.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5d00186c0 0x7fb5d001abc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5f0106010 0x7fb5f018b680 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5f0105c10 0x7fb5f018b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 --2- 172.21.2.117:0/3591952250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5f01051f0 0x7fb5f01055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 >> 172.21.2.117:0/3591952250 conn(0x7fb5f0096500 msgr2=0x7fb5f00b8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 shutdown_connections 2021-02-10T11:39:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.392 7fb5f6575700 1 -- 172.21.2.117:0/3591952250 wait complete. 2021-02-10T11:39:17.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802507 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.814 7f8b4bba0700 1 Processor -- start 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.814 7f8b4bba0700 1 -- start start 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4bba0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8b44079000 0x7f8b440e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4bba0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b441b00f0 0x7f8b441b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4bba0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44077a70 con 0x7f8b4407da20 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4bba0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b440e5c20 con 0x7f8b441b00f0 2021-02-10T11:39:17.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4bba0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b440e53c0 con 0x7f8b44079000 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b491bb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8b44079000 0x7f8b440e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b499bc700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f8b4407da20 0x7f8b440771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40028/0 (socket says 172.21.2.117:40028) 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b499bc700 1 -- 172.21.2.117:0/2916006765 learned_addr learned my addr 172.21.2.117:0/2916006765 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b4a1bd700 1 --2- 172.21.2.117:0/2916006765 >> v2:172.21.2.117:3300/0 conn(0x7f8b441b00f0 0x7f8b441b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b491bb700 1 -- 172.21.2.117:0/2916006765 >> v2:172.21.2.117:3300/0 conn(0x7f8b441b00f0 msgr2=0x7f8b441b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b491bb700 1 --2- 172.21.2.117:0/2916006765 >> v2:172.21.2.117:3300/0 conn(0x7f8b441b00f0 0x7f8b441b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b491bb700 1 -- 172.21.2.117:0/2916006765 >> v1:172.21.2.125:6789/0 conn(0x7f8b4407da20 legacy=0x7f8b440771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.815 7f8b491bb700 1 -- 172.21.2.117:0/2916006765 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44079400 con 0x7f8b44079000 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b491bb700 1 --2- 172.21.2.117:0/2916006765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b44079000 0x7f8b440e4da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8b34006650 tx=0x7f8b34009a80).ready entity=mon.1 client_cookie=6f311f34c72cde9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b489ba700 1 -- 172.21.2.117:0/2916006765 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400c0c0 con 0x7f8b44079000 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b489ba700 1 -- 172.21.2.117:0/2916006765 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3400c340 con 0x7f8b44079000 2021-02-10T11:39:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b489ba700 1 -- 172.21.2.117:0/2916006765 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400c860 con 0x7f8b44079000 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 -- 172.21.2.117:0/2916006765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b44079000 msgr2=0x7f8b440e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 --2- 172.21.2.117:0/2916006765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b44079000 0x7f8b440e4da0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8b34006650 tx=0x7f8b34009a80).stop 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 -- 172.21.2.117:0/2916006765 shutdown_connections 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 --2- 172.21.2.117:0/2916006765 >> v2:172.21.2.117:3300/0 conn(0x7f8b441b00f0 0x7f8b441b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 --2- 172.21.2.117:0/2916006765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b44079000 0x7f8b440e4da0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.816 7f8b4bba0700 1 -- 172.21.2.117:0/2916006765 >> 172.21.2.117:0/2916006765 conn(0x7f8b440ce450 msgr2=0x7f8b440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.817 7f8b4bba0700 1 -- 172.21.2.117:0/2916006765 shutdown_connections 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.817 7f8b4bba0700 1 -- 172.21.2.117:0/2916006765 wait complete. 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.831 7f8b4bba0700 1 Processor -- start 2021-02-10T11:39:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.831 7f8b4bba0700 1 -- start start 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b440b7fc0 0x7f8b441042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b44090710 0x7f8b44090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441054f0 con 0x7f8b441048f0 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44195a30 con 0x7f8b440b7fc0 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4bba0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44195fb0 con 0x7f8b44090710 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46862/0 (socket says 172.21.2.117:46862) 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 -- 172.21.2.117:0/1935922769 learned_addr learned my addr 172.21.2.117:0/1935922769 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b4a1bd700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b44090710 0x7f8b44090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b499bc700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b440b7fc0 0x7f8b441042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 -- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b44090710 msgr2=0x7f8b44090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b44090710 0x7f8b44090b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 -- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b440b7fc0 msgr2=0x7f8b441042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b440b7fc0 0x7f8b441042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.832 7f8b491bb700 1 -- 172.21.2.117:0/1935922769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440b7b60 con 0x7f8b441048f0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b491bb700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f8b34009650 tx=0x7f8b3400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b3400bd50 con 0x7f8b441048f0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34006030 con 0x7f8b441048f0 2021-02-10T11:39:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440b7b60 con 0x7f8b441048f0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b34005560 con 0x7f8b441048f0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.833 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440b7b60 con 0x7f8b441048f0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.834 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b340052e0 con 0x7f8b441048f0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.834 7f8b3a7fc700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 0x7f8b2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.834 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b34032760 con 0x7f8b441048f0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.834 7f8b499bc700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 0x7f8b2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.834 7f8b499bc700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 0x7f8b2c01ac10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:17.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.865 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44103280 con 0x7f8b441048f0 2021-02-10T11:39:17.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.867 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b340009c0 con 0x7f8b441048f0 2021-02-10T11:39:17.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:17.994 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802507", "target": ["mgr", ""]}) v1 -- 0x7f8b44103280 con 0x7f8b2c018730 2021-02-10T11:39:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.005 7f8b3a7fc700 1 -- 172.21.2.117:0/1935922769 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8b400077a0 con 0x7f8b2c018730 2021-02-10T11:39:17.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 msgr2=0x7f8b2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 0x7f8b2c01ac10 crc :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 msgr2=0x7f8b44104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f8b34009650 tx=0x7f8b3400b960).stop 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 shutdown_connections 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b2c018730 0x7f8b2c01ac10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b441048f0 0x7f8b44104cf0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b44090710 0x7f8b44090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 --2- 172.21.2.117:0/1935922769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b440b7fc0 0x7f8b441042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:17.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 >> 172.21.2.117:0/1935922769 conn(0x7f8b440a1850 msgr2=0x7f8b440c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:17.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 shutdown_connections 2021-02-10T11:39:17.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.011 7f8b4bba0700 1 -- 172.21.2.117:0/1935922769 wait complete. 2021-02-10T11:39:18.000 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802508 2021-02-10T11:39:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.432 7f6ef654e700 1 Processor -- start 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.432 7f6ef654e700 1 -- start start 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef654e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 0x7f6ef00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef654e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6ef00e4da0 0x7f6ef01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef654e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef0077a70 con 0x7f6ef00e51a0 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef654e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef00e5c20 con 0x7f6ef00e4da0 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef654e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef00777d0 con 0x7f6ef007da20 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef4b6b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6ef00e51a0 0x7f6ef0077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47116/0 (socket says 172.21.2.117:47116) 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.433 7f6ef4b6b700 1 -- 172.21.2.117:0/1176930241 learned_addr learned my addr 172.21.2.117:0/1176930241 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 --2- 172.21.2.117:0/1176930241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 0x7f6ef00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eef7fe700 1 --2- 172.21.2.117:0/1176930241 >> v2:172.21.2.125:3300/0 conn(0x7f6ef00e4da0 0x7f6ef01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368708788 0 0) 0x7f6ee8001500 con 0x7f6ef00e51a0 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed8002fe0 con 0x7f6ef00e51a0 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 -- 172.21.2.117:0/1176930241 >> v2:172.21.2.125:3300/0 conn(0x7f6ef00e4da0 msgr2=0x7f6ef01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 --2- 172.21.2.117:0/1176930241 >> v2:172.21.2.125:3300/0 conn(0x7f6ef00e4da0 0x7f6ef01b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3297094480 0 0) 0x7f6ee8001500 con 0x7f6ef00e51a0 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 -- 172.21.2.117:0/1176930241 >> v1:172.21.2.117:6789/0 conn(0x7f6ef00e51a0 legacy=0x7f6ef0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 -- 172.21.2.117:0/1176930241 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef007de20 con 0x7f6ef007da20 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.434 7f6eeffff700 1 --2- 172.21.2.117:0/1176930241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 0x7f6ef00771b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6ee0006650 tx=0x7f6ee0009a80).ready entity=mon.2 client_cookie=af6a568b1f08e64d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee000c0c0 con 0x7f6ef007da20 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee000c340 con 0x7f6ef007da20 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6eeeffd700 1 -- 172.21.2.117:0/1176930241 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee000c860 con 0x7f6ef007da20 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 -- 172.21.2.117:0/1176930241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 msgr2=0x7f6ef00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 --2- 172.21.2.117:0/1176930241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 0x7f6ef00771b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6ee0006650 tx=0x7f6ee0009a80).stop 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 -- 172.21.2.117:0/1176930241 shutdown_connections 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 --2- 172.21.2.117:0/1176930241 >> v2:172.21.2.125:3300/0 conn(0x7f6ef00e4da0 0x7f6ef01b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 --2- 172.21.2.117:0/1176930241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef007da20 0x7f6ef00771b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 -- 172.21.2.117:0/1176930241 >> 172.21.2.117:0/1176930241 conn(0x7f6ef00ce450 msgr2=0x7f6ef00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 -- 172.21.2.117:0/1176930241 shutdown_connections 2021-02-10T11:39:18.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.435 7f6ef654e700 1 -- 172.21.2.117:0/1176930241 wait complete. 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.450 7f6ef654e700 1 Processor -- start 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.450 7f6ef654e700 1 -- start start 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef01a4660 0x7f6ef00978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 0x7f6ef0107650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 0x7f6ef0104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0108070 con 0x7f6ef0097ef0 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef01047b0 con 0x7f6ef0107c70 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef654e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef01049d0 con 0x7f6ef01a4660 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6eef7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 0x7f6ef0107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6eef7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 0x7f6ef0107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46868/0 (socket says 172.21.2.117:46868) 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6eef7fe700 1 -- 172.21.2.117:0/801888853 learned_addr learned my addr 172.21.2.117:0/801888853 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:18.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6ef4b6b700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 0x7f6ef0104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.451 7f6eeffff700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef01a4660 0x7f6ef00978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 -- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef01a4660 msgr2=0x7f6ef00978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef01a4660 0x7f6ef00978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 -- 172.21.2.117:0/801888853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 msgr2=0x7f6ef0107650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 0x7f6ef0107650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 -- 172.21.2.117:0/801888853 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0104bf0 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef4b6b700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 0x7f6ef0104190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6ee800b890 tx=0x7f6ee800bc30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee800bd70 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee80060c0 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ee80065e0 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef654e700 1 -- 172.21.2.117:0/801888853 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef0104bf0 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.452 7f6ef654e700 1 -- 172.21.2.117:0/801888853 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef0104bf0 con 0x7f6ef0107c70 2021-02-10T11:39:18.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.453 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ee800bd70 con 0x7f6ef0107c70 2021-02-10T11:39:18.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.453 7f6eecff9700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 0x7f6ed001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:18.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.453 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ee8032bf0 con 0x7f6ef0107c70 2021-02-10T11:39:18.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.453 7f6eeffff700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 0x7f6ed001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:18.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.453 7f6eeffff700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 0x7f6ed001abd0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:18.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.483 7f6ef654e700 1 -- 172.21.2.117:0/801888853 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef00a6b90 con 0x7f6ef0107c70 2021-02-10T11:39:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.487 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ee80060a0 con 0x7f6ef0107c70 2021-02-10T11:39:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.617 7f6ef654e700 1 -- 172.21.2.117:0/801888853 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802508", "target": ["mgr", ""]}) v1 -- 0x7f6ef00a6b90 con 0x7f6ed00186d0 2021-02-10T11:39:18.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.629 7f6eecff9700 1 -- 172.21.2.117:0/801888853 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6ee0006420 con 0x7f6ed00186d0 2021-02-10T11:39:18.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.634 7f6ef654e700 1 -- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 msgr2=0x7f6ed001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.634 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 0x7f6ed001abd0 crc :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.634 7f6ef654e700 1 -- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 msgr2=0x7f6ef0104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.634 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 0x7f6ef0104190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6ee800b890 tx=0x7f6ee800bc30).stop 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 -- 172.21.2.117:0/801888853 shutdown_connections 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ed00186d0 0x7f6ed001abd0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ef0097ef0 0x7f6ef0107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ef0107c70 0x7f6ef0104190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 --2- 172.21.2.117:0/801888853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ef01a4660 0x7f6ef00978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 -- 172.21.2.117:0/801888853 >> 172.21.2.117:0/801888853 conn(0x7f6ef008a180 msgr2=0x7f6ef00b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 -- 172.21.2.117:0/801888853 shutdown_connections 2021-02-10T11:39:18.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:18.635 7f6ef654e700 1 -- 172.21.2.117:0/801888853 wait complete. 2021-02-10T11:39:18.624 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802509 2021-02-10T11:39:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 Processor -- start 2021-02-10T11:39:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 -- start start 2021-02-10T11:39:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8d8079000 0x7ff8d80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d80d1d20 con 0x7ff8d81b00f0 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d8077a70 con 0x7ff8d807da20 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8dcf8e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d80e53c0 con 0x7ff8d8079000 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8d759e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8d81b00f0 0x7ff8d80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47130/0 (socket says 172.21.2.117:47130) 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.057 7ff8d759e700 1 -- 172.21.2.117:0/1827017619 learned_addr learned my addr 172.21.2.117:0/1827017619 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d659c700 1 --2- 172.21.2.117:0/1827017619 >> v2:172.21.2.125:3300/0 conn(0x7ff8d8079000 0x7ff8d80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925522684 0 0) 0x7ff8c0000c60 con 0x7ff8d807da20 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c4002fe0 con 0x7ff8d807da20 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838041189 0 0) 0x7ff8d0000f90 con 0x7ff8d81b00f0 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c4002fe0 con 0x7ff8d81b00f0 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d659c700 1 -- 172.21.2.117:0/1827017619 >> v1:172.21.2.125:6789/0 conn(0x7ff8d807da20 legacy=0x7ff8d80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d659c700 1 -- 172.21.2.117:0/1827017619 >> v1:172.21.2.117:6789/0 conn(0x7ff8d81b00f0 legacy=0x7ff8d80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d659c700 1 -- 172.21.2.117:0/1827017619 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8079400 con 0x7ff8d8079000 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d659c700 1 --2- 172.21.2.117:0/1827017619 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d8079000 0x7ff8d80e4da0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff8c8006650 tx=0x7ff8c8009a00).ready entity=mon.1 client_cookie=596f70d060f2b01b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8c80009b0 con 0x7ff8d8079000 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c800c0c0 con 0x7ff8d8079000 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.058 7ff8d5d9b700 1 -- 172.21.2.117:0/1827017619 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8c800c5a0 con 0x7ff8d8079000 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 -- 172.21.2.117:0/1827017619 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d8079000 msgr2=0x7ff8d80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 --2- 172.21.2.117:0/1827017619 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d8079000 0x7ff8d80e4da0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff8c8006650 tx=0x7ff8c8009a00).stop 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 -- 172.21.2.117:0/1827017619 shutdown_connections 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 --2- 172.21.2.117:0/1827017619 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d8079000 0x7ff8d80e4da0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 -- 172.21.2.117:0/1827017619 >> 172.21.2.117:0/1827017619 conn(0x7ff8d80ce450 msgr2=0x7ff8d80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 -- 172.21.2.117:0/1827017619 shutdown_connections 2021-02-10T11:39:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.059 7ff8dcf8e700 1 -- 172.21.2.117:0/1827017619 wait complete. 2021-02-10T11:39:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 Processor -- start 2021-02-10T11:39:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 -- start start 2021-02-10T11:39:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d8187ee0 0x7ff8d81882e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d81b8130 0x7ff8d81b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81b86a0 con 0x7ff8d81b9240 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d808a180 con 0x7ff8d81b8130 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8dcf8e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81b9c60 con 0x7ff8d8187ee0 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.074 7ff8d759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46882/0 (socket says 172.21.2.117:46882) 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 -- 172.21.2.117:0/819019575 learned_addr learned my addr 172.21.2.117:0/819019575 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d659c700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d81b8130 0x7ff8d81b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d6d9d700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d8187ee0 0x7ff8d81882e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 -- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d8187ee0 msgr2=0x7ff8d81882e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d8187ee0 0x7ff8d81882e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 -- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d81b8130 msgr2=0x7ff8d81b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d81b8130 0x7ff8d81b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 -- 172.21.2.117:0/819019575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80c3fc0 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8d759e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7ff8d0001c10 tx=0x7ff8d000aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d0007ce0 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d00070d0 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d0007520 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d80c3fc0 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.075 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d80c3fc0 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.076 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8d0000e10 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.076 7ff8cf7fe700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 0x7ff8bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.076 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8d0032c70 con 0x7ff8d81b9240 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.076 7ff8d6d9d700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 0x7ff8bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.077 7ff8d6d9d700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 0x7ff8bc01abd0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.106 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d81b9c60 con 0x7ff8d81b9240 2021-02-10T11:39:19.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.109 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8d0007ce0 con 0x7ff8d81b9240 2021-02-10T11:39:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.236 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802509", "target": ["mgr", ""]}) v1 -- 0x7ff8d81b9c60 con 0x7ff8bc0186d0 2021-02-10T11:39:19.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.246 7ff8cf7fe700 1 -- 172.21.2.117:0/819019575 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff8c0000e20 con 0x7ff8bc0186d0 2021-02-10T11:39:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 msgr2=0x7ff8bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 0x7ff8bc01abd0 crc :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 msgr2=0x7ff8d81b9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7ff8d0001c10 tx=0x7ff8d000aca0).stop 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 shutdown_connections 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8bc0186d0 0x7ff8bc01abd0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d81b9240 0x7ff8d81b9640 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d81b8130 0x7ff8d81b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 --2- 172.21.2.117:0/819019575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d8187ee0 0x7ff8d81882e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.253 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 >> 172.21.2.117:0/819019575 conn(0x7ff8d80a4180 msgr2=0x7ff8d809be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.254 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 shutdown_connections 2021-02-10T11:39:19.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.254 7ff8dcf8e700 1 -- 172.21.2.117:0/819019575 wait complete. 2021-02-10T11:39:19.243 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802510 2021-02-10T11:39:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.673 7fcf36b0e700 1 Processor -- start 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 -- start start 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf3007da20 0x7fcf300771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf300e5180 0x7fcf30079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf30077a70 con 0x7fcf300e4d80 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300e5c00 con 0x7fcf300e5180 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf36b0e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf300777d0 con 0x7fcf3007da20 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf2ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcf300e4d80 0x7fcf301b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47140/0 (socket says 172.21.2.117:47140) 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf2ffff700 1 -- 172.21.2.117:0/800058383 learned_addr learned my addr 172.21.2.117:0/800058383 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf3512b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf300e5180 0x7fcf30079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.674 7fcf3492a700 1 --2- 172.21.2.117:0/800058383 >> v2:172.21.2.125:3300/0 conn(0x7fcf3007da20 0x7fcf300771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf2f7fe700 1 -- 172.21.2.117:0/800058383 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858052879 0 0) 0x7fcf20000ce0 con 0x7fcf300e4d80 2021-02-10T11:39:19.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf2f7fe700 1 -- 172.21.2.117:0/800058383 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c002fe0 con 0x7fcf300e4d80 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf3512b700 1 -- 172.21.2.117:0/800058383 >> v2:172.21.2.125:3300/0 conn(0x7fcf3007da20 msgr2=0x7fcf300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf3512b700 1 --2- 172.21.2.117:0/800058383 >> v2:172.21.2.125:3300/0 conn(0x7fcf3007da20 0x7fcf300771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf3512b700 1 -- 172.21.2.117:0/800058383 >> v1:172.21.2.117:6789/0 conn(0x7fcf300e4d80 legacy=0x7fcf301b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf3512b700 1 -- 172.21.2.117:0/800058383 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf301b08c0 con 0x7fcf300e5180 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf3512b700 1 --2- 172.21.2.117:0/800058383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300e5180 0x7fcf30079000 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcf28006e30 tx=0x7fcf28009e60).ready entity=mon.0 client_cookie=3a782cc465a63d93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf2f7fe700 1 -- 172.21.2.117:0/800058383 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf28000f90 con 0x7fcf300e5180 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf2f7fe700 1 -- 172.21.2.117:0/800058383 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2800b300 con 0x7fcf300e5180 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf2f7fe700 1 -- 172.21.2.117:0/800058383 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf2800b820 con 0x7fcf300e5180 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf36b0e700 1 -- 172.21.2.117:0/800058383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300e5180 msgr2=0x7fcf30079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.675 7fcf36b0e700 1 --2- 172.21.2.117:0/800058383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300e5180 0x7fcf30079000 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcf28006e30 tx=0x7fcf28009e60).stop 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 -- 172.21.2.117:0/800058383 shutdown_connections 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 --2- 172.21.2.117:0/800058383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300e5180 0x7fcf30079000 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 --2- 172.21.2.117:0/800058383 >> v2:172.21.2.125:3300/0 conn(0x7fcf3007da20 0x7fcf300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 -- 172.21.2.117:0/800058383 >> 172.21.2.117:0/800058383 conn(0x7fcf300ce430 msgr2=0x7fcf300e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 -- 172.21.2.117:0/800058383 shutdown_connections 2021-02-10T11:39:19.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.676 7fcf36b0e700 1 -- 172.21.2.117:0/800058383 wait complete. 2021-02-10T11:39:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.691 7fcf36b0e700 1 Processor -- start 2021-02-10T11:39:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.691 7fcf36b0e700 1 -- start start 2021-02-10T11:39:19.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 0x7fcf301893b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf301062c0 0x7fcf301066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3018a300 con 0x7fcf300bc940 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf301899d0 con 0x7fcf301062c0 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf36b0e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf300bd360 con 0x7fcf30189f00 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46896/0 (socket says 172.21.2.117:46896) 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 -- 172.21.2.117:0/3185847151 learned_addr learned my addr 172.21.2.117:0/3185847151 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3492a700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 0x7fcf301893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf2ffff700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf301062c0 0x7fcf301066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 -- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 msgr2=0x7fcf301893b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 0x7fcf301893b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 -- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf301062c0 msgr2=0x7fcf301066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf301062c0 0x7fcf301066c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3512b700 1 -- 172.21.2.117:0/3185847151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf300bd580 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.692 7fcf3492a700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 0x7fcf301893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf3512b700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7fcf28009fc0 tx=0x7fcf2800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf28003030 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf28001480 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf300bd580 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf300bdb10 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.693 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf280019a0 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.694 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf28001660 con 0x7fcf300bc940 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.694 7fcf2d7fa700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 0x7fcf1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:19.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.694 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcf28032a70 con 0x7fcf300bc940 2021-02-10T11:39:19.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.694 7fcf3492a700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 0x7fcf1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:19.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.694 7fcf3492a700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 0x7fcf1401abc0 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:19.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.724 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf30103260 con 0x7fcf300bc940 2021-02-10T11:39:19.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.726 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf28000f90 con 0x7fcf300bc940 2021-02-10T11:39:19.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.855 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802510", "target": ["mgr", ""]}) v1 -- 0x7fcf300bdb10 con 0x7fcf140186c0 2021-02-10T11:39:19.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.865 7fcf2d7fa700 1 -- 172.21.2.117:0/3185847151 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcf180009b0 con 0x7fcf140186c0 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 msgr2=0x7fcf1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 0x7fcf1401abc0 crc :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 msgr2=0x7fcf300bcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7fcf28009fc0 tx=0x7fcf2800bd40).stop 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 shutdown_connections 2021-02-10T11:39:19.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf140186c0 0x7fcf1401abc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf300bc940 0x7fcf300bcd40 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf301062c0 0x7fcf301066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 --2- 172.21.2.117:0/3185847151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf30189f00 0x7fcf301893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 >> 172.21.2.117:0/3185847151 conn(0x7fcf3018a760 msgr2=0x7fcf30190320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 shutdown_connections 2021-02-10T11:39:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:19.872 7fcf36b0e700 1 -- 172.21.2.117:0/3185847151 wait complete. 2021-02-10T11:39:19.861 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802511 2021-02-10T11:39:20.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.294 7f1b421dc700 1 Processor -- start 2021-02-10T11:39:20.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.294 7f1b421dc700 1 -- start start 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b421dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 0x7f1b3c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b421dc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b421dc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c077a70 con 0x7f1b3c0e51a0 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b421dc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c0e5c20 con 0x7f1b3c0e4da0 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b421dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0777d0 con 0x7f1b3c07da20 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b3affd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b3affd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51286/0 (socket says 172.21.2.117:51286) 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b3b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 0x7f1b3c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.295 7f1b3affd700 1 -- 172.21.2.117:0/4272074911 learned_addr learned my addr 172.21.2.117:0/4272074911 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3affd700 1 -- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 msgr2=0x7f1b3c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3affd700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 0x7f1b3c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3affd700 1 -- 172.21.2.117:0/4272074911 >> v1:172.21.2.125:6789/0 conn(0x7f1b3c0e51a0 legacy=0x7f1b3c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3affd700 1 -- 172.21.2.117:0/4272074911 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c07de20 con 0x7f1b3c0e4da0 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3b7fe700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 0x7f1b3c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3affd700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1b24006700 tx=0x7f1b24009b80).ready entity=mon.1 client_cookie=9142d461b001a93a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3a7fc700 1 -- 172.21.2.117:0/4272074911 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2400c0c0 con 0x7f1b3c0e4da0 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3a7fc700 1 -- 172.21.2.117:0/4272074911 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2400c3c0 con 0x7f1b3c0e4da0 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b3a7fc700 1 -- 172.21.2.117:0/4272074911 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2400c8e0 con 0x7f1b3c0e4da0 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b421dc700 1 -- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0e4da0 msgr2=0x7f1b3c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.296 7f1b421dc700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1b24006700 tx=0x7f1b24009b80).stop 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 -- 172.21.2.117:0/4272074911 shutdown_connections 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0e4da0 0x7f1b3c1b00f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 --2- 172.21.2.117:0/4272074911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c07da20 0x7f1b3c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 -- 172.21.2.117:0/4272074911 >> 172.21.2.117:0/4272074911 conn(0x7f1b3c0ce450 msgr2=0x7f1b3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 -- 172.21.2.117:0/4272074911 shutdown_connections 2021-02-10T11:39:20.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.297 7f1b421dc700 1 -- 172.21.2.117:0/4272074911 wait complete. 2021-02-10T11:39:20.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 Processor -- start 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 -- start start 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 0x7f1b3c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c0a3800 0x7f1b3c0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a4220 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a4410 con 0x7f1b3c0b9e60 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.312 7f1b421dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0ba910 con 0x7f1b3c0a3800 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46906/0 (socket says 172.21.2.117:46906) 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 -- 172.21.2.117:0/1122340070 learned_addr learned my addr 172.21.2.117:0/1122340070 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3affd700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 0x7f1b3c0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3bfff700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c0a3800 0x7f1b3c0a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 -- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c0a3800 msgr2=0x7f1b3c0a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c0a3800 0x7f1b3c0a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 -- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 msgr2=0x7f1b3c0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 0x7f1b3c0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 -- 172.21.2.117:0/1122340070 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0bab30 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3affd700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 0x7f1b3c0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.313 7f1b3b7fe700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1b2c00b970 tx=0x7f1b2c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2c000d30 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c006020 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1b2c006540 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c0bab30 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c0bb0c0 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1b2c006200 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.314 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1b2c032a30 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.315 7f1b3affd700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.315 7f1b3affd700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.344 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c0ba3e0 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.347 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1b2c000d30 con 0x7f1b3c19a4a0 2021-02-10T11:39:20.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.473 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802511", "target": ["mgr", ""]}) v1 -- 0x7f1b3c0ba3e0 con 0x7f1b200186c0 2021-02-10T11:39:20.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.483 7f1b33fff700 1 -- 172.21.2.117:0/1122340070 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1b24000c90 con 0x7f1b200186c0 2021-02-10T11:39:20.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.490 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 msgr2=0x7f1b2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.490 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 crc :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.490 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 msgr2=0x7f1b3c0b9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.490 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f1b2c00b970 tx=0x7f1b2c00bd10).stop 2021-02-10T11:39:20.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 shutdown_connections 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1b3c0a3800 0x7f1b3c0a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1b3c0b9e60 0x7f1b3c0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1b200186c0 0x7f1b2001abc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 --2- 172.21.2.117:0/1122340070 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1b3c19a4a0 0x7f1b3c0b9840 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 >> 172.21.2.117:0/1122340070 conn(0x7f1b3c107b60 msgr2=0x7f1b3c0c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 shutdown_connections 2021-02-10T11:39:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.491 7f1b421dc700 1 -- 172.21.2.117:0/1122340070 wait complete. 2021-02-10T11:39:20.480 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802512 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.914 7fc8e33fb700 1 Processor -- start 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 -- start start 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8dc07da20 0x7fc8dc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc8dc0e51a0 0x7fc8dc079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc077a70 con 0x7fc8dc0e4da0 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0e5c20 con 0x7fc8dc07da20 2021-02-10T11:39:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e33fb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0777d0 con 0x7fc8dc0e51a0 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8dc07da20 0x7fc8dc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8dc07da20 0x7fc8dc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46914/0 (socket says 172.21.2.117:46914) 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1a18700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc8dc0e51a0 0x7fc8dc079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 -- 172.21.2.117:0/1384740511 learned_addr learned my addr 172.21.2.117:0/1384740511 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 -- 172.21.2.117:0/1384740511 >> v2:172.21.2.125:3300/0 conn(0x7fc8dc0e51a0 msgr2=0x7fc8dc079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 --2- 172.21.2.117:0/1384740511 >> v2:172.21.2.125:3300/0 conn(0x7fc8dc0e51a0 0x7fc8dc079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 -- 172.21.2.117:0/1384740511 >> v1:172.21.2.125:6789/0 conn(0x7fc8dc0e4da0 legacy=0x7fc8dc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.915 7fc8e1217700 1 -- 172.21.2.117:0/1384740511 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc1b08e0 con 0x7fc8dc07da20 2021-02-10T11:39:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8e1217700 1 --2- 172.21.2.117:0/1384740511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc07da20 0x7fc8dc0771b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fc8d8006700 tx=0x7fc8d8009b80).ready entity=mon.0 client_cookie=4f50cabfc51c3375 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8d3fff700 1 -- 172.21.2.117:0/1384740511 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8d800c0c0 con 0x7fc8dc07da20 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8d3fff700 1 -- 172.21.2.117:0/1384740511 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d800c340 con 0x7fc8dc07da20 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8d3fff700 1 -- 172.21.2.117:0/1384740511 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8d800c7a0 con 0x7fc8dc07da20 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8e33fb700 1 -- 172.21.2.117:0/1384740511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc07da20 msgr2=0x7fc8dc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.916 7fc8e33fb700 1 --2- 172.21.2.117:0/1384740511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc07da20 0x7fc8dc0771b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7fc8d8006700 tx=0x7fc8d8009b80).stop 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 -- 172.21.2.117:0/1384740511 shutdown_connections 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 --2- 172.21.2.117:0/1384740511 >> v2:172.21.2.125:3300/0 conn(0x7fc8dc0e51a0 0x7fc8dc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 --2- 172.21.2.117:0/1384740511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc07da20 0x7fc8dc0771b0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 -- 172.21.2.117:0/1384740511 >> 172.21.2.117:0/1384740511 conn(0x7fc8dc0ce450 msgr2=0x7fc8dc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 -- 172.21.2.117:0/1384740511 shutdown_connections 2021-02-10T11:39:20.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.917 7fc8e33fb700 1 -- 172.21.2.117:0/1384740511 wait complete. 2021-02-10T11:39:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 Processor -- start 2021-02-10T11:39:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 -- start start 2021-02-10T11:39:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 0x7fc8dc106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 0x7fc8dc090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1067f0 con 0x7fc8dc187e20 2021-02-10T11:39:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc107350 con 0x7fc8dc090420 2021-02-10T11:39:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e33fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1042d0 con 0x7fc8dc1a0250 2021-02-10T11:39:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e0a16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e0a16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46924/0 (socket says 172.21.2.117:46924) 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e0a16700 1 -- 172.21.2.117:0/511147875 learned_addr learned my addr 172.21.2.117:0/511147875 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.932 7fc8e1a18700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 0x7fc8dc090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 -- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 msgr2=0x7fc8dc106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e1217700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 0x7fc8dc106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 0x7fc8dc106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 -- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 msgr2=0x7fc8dc090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 0x7fc8dc090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 -- 172.21.2.117:0/511147875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc090e40 con 0x7fc8dc187e20 2021-02-10T11:39:20.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e1217700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 0x7fc8dc106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e1a18700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 0x7fc8dc090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e0a16700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fc8cc00b5b0 tx=0x7fc8cc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8cc0009b0 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc00bd40 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8cc006210 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc090e40 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc090e40 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8cc006230 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.933 7fc8d1ffb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 0x7fc8bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.934 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc8cc032960 con 0x7fc8dc187e20 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.934 7fc8e1217700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 0x7fc8bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:20.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.934 7fc8e1217700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 0x7fc8bc01abd0 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:20.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.964 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc103280 con 0x7fc8dc187e20 2021-02-10T11:39:20.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:20.967 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8cc006000 con 0x7fc8dc187e20 2021-02-10T11:39:21.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.092 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802512", "target": ["mgr", ""]}) v1 -- 0x7fc8dc103280 con 0x7fc8bc0186d0 2021-02-10T11:39:21.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.103 7fc8d1ffb700 1 -- 172.21.2.117:0/511147875 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc8d80009b0 con 0x7fc8bc0186d0 2021-02-10T11:39:21.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 msgr2=0x7fc8bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 0x7fc8bc01abd0 crc :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 msgr2=0x7fc8dc188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0x7fc8cc00b5b0 tx=0x7fc8cc00b950).stop 2021-02-10T11:39:21.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 shutdown_connections 2021-02-10T11:39:21.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.109 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8bc0186d0 0x7fc8bc01abd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8dc187e20 0x7fc8dc188220 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8dc090420 0x7fc8dc090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 --2- 172.21.2.117:0/511147875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8dc1a0250 0x7fc8dc106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 >> 172.21.2.117:0/511147875 conn(0x7fc8dc0a1850 msgr2=0x7fc8dc0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 shutdown_connections 2021-02-10T11:39:21.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.110 7fc8e33fb700 1 -- 172.21.2.117:0/511147875 wait complete. 2021-02-10T11:39:21.099 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802513 2021-02-10T11:39:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.531 7f0f30895700 1 Processor -- start 2021-02-10T11:39:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.531 7f0f30895700 1 -- start start 2021-02-10T11:39:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f30895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 0x7f0f2c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f30895700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 0x7f0f2c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f30895700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c077a70 con 0x7f0f2c0e51a0 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f30895700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0e5c20 con 0x7f0f2c0e4da0 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f30895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0777d0 con 0x7f0f2c07da20 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f2ae1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0f2c0e51a0 0x7f0f2c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47178/0 (socket says 172.21.2.117:47178) 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f2ae1d700 1 -- 172.21.2.117:0/3460745388 learned_addr learned my addr 172.21.2.117:0/3460745388 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f29e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 0x7f0f2c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f2a61c700 1 --2- 172.21.2.117:0/3460745388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 0x7f0f2c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.532 7f0f2a61c700 1 -- 172.21.2.117:0/3460745388 >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 msgr2=0x7f0f2c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2a61c700 1 --2- 172.21.2.117:0/3460745388 >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 0x7f0f2c1b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2a61c700 1 -- 172.21.2.117:0/3460745388 >> v1:172.21.2.117:6789/0 conn(0x7f0f2c0e51a0 legacy=0x7f0f2c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2a61c700 1 -- 172.21.2.117:0/3460745388 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c07de20 con 0x7f0f2c07da20 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2961a700 1 -- 172.21.2.117:0/3460745388 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807799848 0 0) 0x7f0f24000f90 con 0x7f0f2c0e51a0 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f29e1b700 1 --2- 172.21.2.117:0/3460745388 >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 0x7f0f2c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2a61c700 1 --2- 172.21.2.117:0/3460745388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 0x7f0f2c0771b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0f14006650 tx=0x7f0f14009a80).ready entity=mon.2 client_cookie=f5174e87c5f018da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2961a700 1 -- 172.21.2.117:0/3460745388 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f140009b0 con 0x7f0f2c07da20 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2961a700 1 -- 172.21.2.117:0/3460745388 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400c0c0 con 0x7f0f2c07da20 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f2961a700 1 -- 172.21.2.117:0/3460745388 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f1400c5e0 con 0x7f0f2c07da20 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f30895700 1 -- 172.21.2.117:0/3460745388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 msgr2=0x7f0f2c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.533 7f0f30895700 1 --2- 172.21.2.117:0/3460745388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 0x7f0f2c0771b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f0f14006650 tx=0x7f0f14009a80).stop 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 -- 172.21.2.117:0/3460745388 shutdown_connections 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 --2- 172.21.2.117:0/3460745388 >> v2:172.21.2.117:3300/0 conn(0x7f0f2c0e4da0 0x7f0f2c1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 --2- 172.21.2.117:0/3460745388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c07da20 0x7f0f2c0771b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 -- 172.21.2.117:0/3460745388 >> 172.21.2.117:0/3460745388 conn(0x7f0f2c0ce450 msgr2=0x7f0f2c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 -- 172.21.2.117:0/3460745388 shutdown_connections 2021-02-10T11:39:21.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.534 7f0f30895700 1 -- 172.21.2.117:0/3460745388 wait complete. 2021-02-10T11:39:21.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.549 7f0f30895700 1 Processor -- start 2021-02-10T11:39:21.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.549 7f0f30895700 1 -- start start 2021-02-10T11:39:21.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c187e20 0x7f0f2c188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f2c090420 0x7f0f2c090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1067f0 con 0x7f0f2c1a0250 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c107350 con 0x7f0f2c090420 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f30895700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1042d0 con 0x7f0f2c187e20 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46932/0 (socket says 172.21.2.117:46932) 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 -- 172.21.2.117:0/3962172022 learned_addr learned my addr 172.21.2.117:0/3962172022 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f29e1b700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c187e20 0x7f0f2c188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2ae1d700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f2c090420 0x7f0f2c090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 -- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c187e20 msgr2=0x7f0f2c188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c187e20 0x7f0f2c188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 -- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f2c090420 msgr2=0x7f0f2c090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f2c090420 0x7f0f2c090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 -- 172.21.2.117:0/3962172022 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c090e40 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.550 7f0f2a61c700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f0f14009630 tx=0x7f0f1400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f140093a0 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400bcf0 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f14005560 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f30895700 1 -- 172.21.2.117:0/3962172022 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c090e40 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f30895700 1 -- 172.21.2.117:0/3962172022 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c090e40 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f14008240 con 0x7f0f2c1a0250 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 0x7f0f1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:21.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.551 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0f14032780 con 0x7f0f2c1a0250 2021-02-10T11:39:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.552 7f0f29e1b700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 0x7f0f1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:21.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.552 7f0f29e1b700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 0x7f0f1001abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:21.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.581 7f0f30895700 1 -- 172.21.2.117:0/3962172022 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c103280 con 0x7f0f2c1a0250 2021-02-10T11:39:21.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.583 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f14008010 con 0x7f0f2c1a0250 2021-02-10T11:39:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.707 7f0f30895700 1 -- 172.21.2.117:0/3962172022 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802513", "target": ["mgr", ""]}) v1 -- 0x7f0f2c107310 con 0x7f0f100186c0 2021-02-10T11:39:21.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.718 7f0f22ffd700 1 -- 172.21.2.117:0/3962172022 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0f1c0009b0 con 0x7f0f100186c0 2021-02-10T11:39:21.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.724 7f0f30895700 1 -- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 msgr2=0x7f0f1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.724 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 0x7f0f1001abc0 crc :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.724 7f0f30895700 1 -- 172.21.2.117:0/3962172022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 msgr2=0x7f0f2c106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.724 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f0f14009630 tx=0x7f0f1400b900).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 -- 172.21.2.117:0/3962172022 shutdown_connections 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f2c090420 0x7f0f2c090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f2c187e20 0x7f0f2c188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f100186c0 0x7f0f1001abc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 --2- 172.21.2.117:0/3962172022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f2c1a0250 0x7f0f2c106cf0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:21.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 -- 172.21.2.117:0/3962172022 >> 172.21.2.117:0/3962172022 conn(0x7f0f2c0a1850 msgr2=0x7f0f2c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:21.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 -- 172.21.2.117:0/3962172022 shutdown_connections 2021-02-10T11:39:21.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:21.725 7f0f30895700 1 -- 172.21.2.117:0/3962172022 wait complete. 2021-02-10T11:39:21.715 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802514 2021-02-10T11:39:22.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 Processor -- start 2021-02-10T11:39:22.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 -- start start 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 0x7fa3e40d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e4077a70 con 0x7fa3e4079000 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e4079400 con 0x7fa3e407da20 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.147 7fa3eb406700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e407de20 con 0x7fa3e40d1530 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e8a21700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa3e4079000 0x7fa3e40e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47190/0 (socket says 172.21.2.117:47190) 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e8a21700 1 -- 172.21.2.117:0/2542148321 learned_addr learned my addr 172.21.2.117:0/2542148321 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e9a23700 1 --2- 172.21.2.117:0/2542148321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 0x7fa3e40d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3dbfff700 1 -- 172.21.2.117:0/2542148321 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107513279 0 0) 0x7fa3d40010b0 con 0x7fa3e4079000 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3dbfff700 1 -- 172.21.2.117:0/2542148321 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d0002fe0 con 0x7fa3e4079000 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e9a23700 1 -- 172.21.2.117:0/2542148321 >> v1:172.21.2.125:6789/0 conn(0x7fa3e407da20 legacy=0x7fa3e40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e9a23700 1 -- 172.21.2.117:0/2542148321 >> v1:172.21.2.117:6789/0 conn(0x7fa3e4079000 legacy=0x7fa3e40e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.148 7fa3e9a23700 1 -- 172.21.2.117:0/2542148321 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e41b00f0 con 0x7fa3e40d1530 2021-02-10T11:39:22.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3e9a23700 1 --2- 172.21.2.117:0/2542148321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 0x7fa3e40d1930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa3dc006e50 tx=0x7fa3dc009e70).ready entity=mon.2 client_cookie=bd66292f869e05b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3dbfff700 1 -- 172.21.2.117:0/2542148321 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3dc006a20 con 0x7fa3e40d1530 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3dbfff700 1 -- 172.21.2.117:0/2542148321 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc006790 con 0x7fa3e40d1530 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3dbfff700 1 -- 172.21.2.117:0/2542148321 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3dc00b700 con 0x7fa3e40d1530 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3eb406700 1 -- 172.21.2.117:0/2542148321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 msgr2=0x7fa3e40d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.149 7fa3eb406700 1 --2- 172.21.2.117:0/2542148321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 0x7fa3e40d1930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fa3dc006e50 tx=0x7fa3dc009e70).stop 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.150 7fa3eb406700 1 -- 172.21.2.117:0/2542148321 shutdown_connections 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.150 7fa3eb406700 1 --2- 172.21.2.117:0/2542148321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e40d1530 0x7fa3e40d1930 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.150 7fa3eb406700 1 -- 172.21.2.117:0/2542148321 >> 172.21.2.117:0/2542148321 conn(0x7fa3e40ce450 msgr2=0x7fa3e40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.150 7fa3eb406700 1 -- 172.21.2.117:0/2542148321 shutdown_connections 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.150 7fa3eb406700 1 -- 172.21.2.117:0/2542148321 wait complete. 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 Processor -- start 2021-02-10T11:39:22.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 -- start start 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e4095120 0x7fa3e40b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3e40b77f0 0x7fa3e40b9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4199cf0 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40b8670 con 0x7fa3e40b77f0 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.165 7fa3eb406700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e40b7e20 con 0x7fa3e4095120 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e8a21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e4095120 0x7fa3e40b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46944/0 (socket says 172.21.2.117:46944) 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 -- 172.21.2.117:0/3437508289 learned_addr learned my addr 172.21.2.117:0/3437508289 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:22.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9a23700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3e40b77f0 0x7fa3e40b9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 -- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e4095120 msgr2=0x7fa3e40b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e4095120 0x7fa3e40b8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 -- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3e40b77f0 msgr2=0x7fa3e40b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3e40b77f0 0x7fa3e40b9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 -- 172.21.2.117:0/3437508289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e40b9830 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3e9222700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fa3e000b5b0 tx=0x7fa3e000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e0006240 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e00054a0 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.166 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e40b9830 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e40b9dc0 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e0005980 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3e0006030 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3d9ffb700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 0x7fa3c401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa3e0032940 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.167 7fa3e8a21700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 0x7fa3c401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.168 7fa3e8a21700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 0x7fa3c401abd0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.197 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e4199cf0 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.199 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3e00054a0 con 0x7fa3e40b6dd0 2021-02-10T11:39:22.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.325 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802514", "target": ["mgr", ""]}) v1 -- 0x7fa3e40b9dc0 con 0x7fa3c40186d0 2021-02-10T11:39:22.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.337 7fa3d9ffb700 1 -- 172.21.2.117:0/3437508289 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa3d4000ec0 con 0x7fa3c40186d0 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 msgr2=0x7fa3c401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 0x7fa3c401abd0 crc :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 msgr2=0x7fa3e40b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7fa3e000b5b0 tx=0x7fa3e000b950).stop 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 shutdown_connections 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3e40b77f0 0x7fa3e40b9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3e4095120 0x7fa3e40b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3c40186d0 0x7fa3c401abd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 --2- 172.21.2.117:0/3437508289 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3e40b6dd0 0x7fa3e40b71d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 >> 172.21.2.117:0/3437508289 conn(0x7fa3e408a180 msgr2=0x7fa3e408ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 shutdown_connections 2021-02-10T11:39:22.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.343 7fa3eb406700 1 -- 172.21.2.117:0/3437508289 wait complete. 2021-02-10T11:39:22.332 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802515 2021-02-10T11:39:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.763 7fb9d47a5700 1 Processor -- start 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.763 7fb9d47a5700 1 -- start start 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d47a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 0x7fb9cc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d47a5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 0x7fb9cc0d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d47a5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc077a70 con 0x7fb9cc0e4da0 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d47a5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9cc0e5c20 con 0x7fb9cc0e51a0 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d47a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0777d0 con 0x7fb9cc07da20 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d2dc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 0x7fb9cc0d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d2dc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 0x7fb9cc0d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46952/0 (socket says 172.21.2.117:46952) 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d2dc2700 1 -- 172.21.2.117:0/2549462666 learned_addr learned my addr 172.21.2.117:0/2549462666 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.764 7fb9d25c1700 1 --2- 172.21.2.117:0/2549462666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 0x7fb9cc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d25c1700 1 -- 172.21.2.117:0/2549462666 >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 msgr2=0x7fb9cc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d25c1700 1 --2- 172.21.2.117:0/2549462666 >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 0x7fb9cc0d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d25c1700 1 -- 172.21.2.117:0/2549462666 >> v1:172.21.2.125:6789/0 conn(0x7fb9cc0e4da0 legacy=0x7fb9cc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d25c1700 1 -- 172.21.2.117:0/2549462666 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc1b08e0 con 0x7fb9cc07da20 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d25c1700 1 --2- 172.21.2.117:0/2549462666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 0x7fb9cc0771b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb9c8006650 tx=0x7fb9c8009a80).ready entity=mon.2 client_cookie=3aea1bebd3f695f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d15bf700 1 -- 172.21.2.117:0/2549462666 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c80009b0 con 0x7fb9cc07da20 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d15bf700 1 -- 172.21.2.117:0/2549462666 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c800c0c0 con 0x7fb9cc07da20 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d15bf700 1 -- 172.21.2.117:0/2549462666 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c800c5e0 con 0x7fb9cc07da20 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d47a5700 1 -- 172.21.2.117:0/2549462666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 msgr2=0x7fb9cc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.765 7fb9d47a5700 1 --2- 172.21.2.117:0/2549462666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 0x7fb9cc0771b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb9c8006650 tx=0x7fb9c8009a80).stop 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 -- 172.21.2.117:0/2549462666 shutdown_connections 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 --2- 172.21.2.117:0/2549462666 >> v2:172.21.2.117:3300/0 conn(0x7fb9cc0e51a0 0x7fb9cc0d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 --2- 172.21.2.117:0/2549462666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc07da20 0x7fb9cc0771b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 -- 172.21.2.117:0/2549462666 >> 172.21.2.117:0/2549462666 conn(0x7fb9cc0ce450 msgr2=0x7fb9cc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 -- 172.21.2.117:0/2549462666 shutdown_connections 2021-02-10T11:39:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.766 7fb9d47a5700 1 -- 172.21.2.117:0/2549462666 wait complete. 2021-02-10T11:39:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.783 7fb9d47a5700 1 Processor -- start 2021-02-10T11:39:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.783 7fb9d47a5700 1 -- start start 2021-02-10T11:39:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc193910 0x7fb9cc100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9cc193d10 0x7fb9cc101220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc100700 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1080e0 con 0x7fb9cc193d10 2021-02-10T11:39:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1b6be0 con 0x7fb9cc193910 2021-02-10T11:39:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46954/0 (socket says 172.21.2.117:46954) 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 -- 172.21.2.117:0/2313813194 learned_addr learned my addr 172.21.2.117:0/2313813194 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d2dc2700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9cc193d10 0x7fb9cc101220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d1dc0700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc193910 0x7fb9cc100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 -- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc193910 msgr2=0x7fb9cc100c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc193910 0x7fb9cc100c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 -- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9cc193d10 msgr2=0x7fb9cc101220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9cc193d10 0x7fb9cc101220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 -- 172.21.2.117:0/2313813194 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc091f10 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d25c1700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fb9c8009650 tx=0x7fb9c800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c80093a0 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c800bd50 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9c80052e0 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.784 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc091f10 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.785 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc100280 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.785 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9c8006030 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.785 7fb9beffd700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 0x7fb9b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.785 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb9c80327b0 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.785 7fb9d1dc0700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 0x7fb9b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:22.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.786 7fb9d1dc0700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 0x7fb9b401ac10 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:22.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.814 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc1b4f00 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.816 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9c8009300 con 0x7fb9cc1b67e0 2021-02-10T11:39:22.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.945 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802515", "target": ["mgr", ""]}) v1 -- 0x7fb9cc091f10 con 0x7fb9b4018730 2021-02-10T11:39:22.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.955 7fb9beffd700 1 -- 172.21.2.117:0/2313813194 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9c0006a90 con 0x7fb9b4018730 2021-02-10T11:39:22.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.961 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 msgr2=0x7fb9b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.961 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 0x7fb9b401ac10 crc :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.961 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 msgr2=0x7fb9cc107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.961 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fb9c8009650 tx=0x7fb9c800b960).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 shutdown_connections 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9cc193d10 0x7fb9cc101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9cc193910 0x7fb9cc100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9b4018730 0x7fb9b401ac10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 --2- 172.21.2.117:0/2313813194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9cc1b67e0 0x7fb9cc107a80 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 >> 172.21.2.117:0/2313813194 conn(0x7fb9cc0ffe30 msgr2=0x7fb9cc0c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 shutdown_connections 2021-02-10T11:39:22.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:22.962 7fb9d47a5700 1 -- 172.21.2.117:0/2313813194 wait complete. 2021-02-10T11:39:22.951 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802516 2021-02-10T11:39:23.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.384 7f0587bad700 1 Processor -- start 2021-02-10T11:39:23.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.384 7f0587bad700 1 -- start start 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f05800e51a0 0x7f0580077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0580077a70 con 0x7f05800e51a0 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800777d0 con 0x7f058007da20 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f0587bad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058007de20 con 0x7f05800e4da0 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05851c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05861ca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f05800e51a0 0x7f0580077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05859c9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05851c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44642/0 (socket says 172.21.2.117:44642) 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05861ca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f05800e51a0 0x7f0580077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46966/0 (socket says 172.21.2.117:46966) 2021-02-10T11:39:23.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05859c9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51346/0 (socket says 172.21.2.117:51346) 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.385 7f05851c8700 1 -- 172.21.2.117:0/1582599236 learned_addr learned my addr 172.21.2.117:0/1582599236 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 -- 172.21.2.117:0/1582599236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 msgr2=0x7f05801b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 -- 172.21.2.117:0/1582599236 >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 msgr2=0x7f05800771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 --2- 172.21.2.117:0/1582599236 >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 -- 172.21.2.117:0/1582599236 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05801b0710 con 0x7f05800e51a0 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05851c8700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05859c9700 1 --2- 172.21.2.117:0/1582599236 >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05861ca700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800e51a0 0x7f0580077ff0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f0570006ab0 tx=0x7f0570009b40).ready entity=mon.0 client_cookie=7e544f3c28eca6d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05849c7700 1 -- 172.21.2.117:0/1582599236 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05700066d0 con 0x7f05800e51a0 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05849c7700 1 -- 172.21.2.117:0/1582599236 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05700500d0 con 0x7f05800e51a0 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f05849c7700 1 -- 172.21.2.117:0/1582599236 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05700505f0 con 0x7f05800e51a0 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f0587bad700 1 -- 172.21.2.117:0/1582599236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800e51a0 msgr2=0x7f0580077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.386 7f0587bad700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800e51a0 0x7f0580077ff0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f0570006ab0 tx=0x7f0570009b40).stop 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 -- 172.21.2.117:0/1582599236 shutdown_connections 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800e51a0 0x7f0580077ff0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 --2- 172.21.2.117:0/1582599236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800e4da0 0x7f05801b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 --2- 172.21.2.117:0/1582599236 >> v2:172.21.2.125:3300/0 conn(0x7f058007da20 0x7f05800771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 -- 172.21.2.117:0/1582599236 >> 172.21.2.117:0/1582599236 conn(0x7f05800ce450 msgr2=0x7f05800e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 -- 172.21.2.117:0/1582599236 shutdown_connections 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.387 7f0587bad700 1 -- 172.21.2.117:0/1582599236 wait complete. 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 Processor -- start 2021-02-10T11:39:23.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 -- start start 2021-02-10T11:39:23.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 0x7f05800a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05800b1640 0x7f05800b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 0x7f05800b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b3700 con 0x7f05800b2060 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b10c0 con 0x7f05800b1640 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f0587bad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b7b80 con 0x7f05800b3300 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f05861ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 0x7f05800b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f05861ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 0x7f05800b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46968/0 (socket says 172.21.2.117:46968) 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f05861ca700 1 -- 172.21.2.117:0/1216887194 learned_addr learned my addr 172.21.2.117:0/1216887194 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f05859c9700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 0x7f05800a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.402 7f05851c8700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05800b1640 0x7f05800b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 -- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05800b1640 msgr2=0x7f05800b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05800b1640 0x7f05800b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 -- 172.21.2.117:0/1216887194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 msgr2=0x7f05800b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 0x7f05800b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 -- 172.21.2.117:0/1216887194 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800b0740 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05859c9700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 0x7f05800a3ee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0578009540 tx=0x7f057800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f057800bcd0 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578006740 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0578002220 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f0587bad700 1 -- 172.21.2.117:0/1216887194 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800b0740 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.403 7f0587bad700 1 -- 172.21.2.117:0/1216887194 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800b0740 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.404 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0578009fe0 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.404 7f05767fc700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 0x7f056801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.404 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f05780326d0 con 0x7f05800b3300 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.404 7f05851c8700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 0x7f056801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:23.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.404 7f05851c8700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 0x7f056801abd0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:23.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.435 7f0587bad700 1 -- 172.21.2.117:0/1216887194 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580102460 con 0x7f05800b3300 2021-02-10T11:39:23.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.438 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0578006740 con 0x7f05800b3300 2021-02-10T11:39:23.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.567 7f0587bad700 1 -- 172.21.2.117:0/1216887194 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802516", "target": ["mgr", ""]}) v1 -- 0x7f0580102460 con 0x7f05680186d0 2021-02-10T11:39:23.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.578 7f05767fc700 1 -- 172.21.2.117:0/1216887194 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f057c006bf0 con 0x7f05680186d0 2021-02-10T11:39:23.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 -- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 msgr2=0x7f056801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 0x7f056801abd0 crc :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 -- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 msgr2=0x7f05800a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 0x7f05800a3ee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f0578009540 tx=0x7f057800b8e0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 -- 172.21.2.117:0/1216887194 shutdown_connections 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05680186d0 0x7f056801abd0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05800b2060 0x7f05800b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05800b1640 0x7f05800b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 --2- 172.21.2.117:0/1216887194 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05800b3300 0x7f05800a3ee0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:23.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.584 7f0587bad700 1 -- 172.21.2.117:0/1216887194 >> 172.21.2.117:0/1216887194 conn(0x7f0580104070 msgr2=0x7f05800aced0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:23.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.585 7f0587bad700 1 -- 172.21.2.117:0/1216887194 shutdown_connections 2021-02-10T11:39:23.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:23.585 7f0587bad700 1 -- 172.21.2.117:0/1216887194 wait complete. 2021-02-10T11:39:23.574 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802517 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 Processor -- start 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 -- start start 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb94007da20 0x7fb9400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9400e51a0 0x7fb940079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb940077a70 con 0x7fb9400e4da0 2021-02-10T11:39:24.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.007 7fb945266700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9400e5c20 con 0x7fb9400e51a0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb945266700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9400777d0 con 0x7fb94007da20 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9400e51a0 0x7fb940079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9400e51a0 0x7fb940079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46976/0 (socket says 172.21.2.117:46976) 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 -- 172.21.2.117:0/1847649589 learned_addr learned my addr 172.21.2.117:0/1847649589 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f07d700 1 --2- 172.21.2.117:0/1847649589 >> v2:172.21.2.125:3300/0 conn(0x7fb94007da20 0x7fb9400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 -- 172.21.2.117:0/1847649589 >> v2:172.21.2.125:3300/0 conn(0x7fb94007da20 msgr2=0x7fb9400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 --2- 172.21.2.117:0/1847649589 >> v2:172.21.2.125:3300/0 conn(0x7fb94007da20 0x7fb9400771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 -- 172.21.2.117:0/1847649589 >> v1:172.21.2.125:6789/0 conn(0x7fb9400e4da0 legacy=0x7fb9401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 -- 172.21.2.117:0/1847649589 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9401b08e0 con 0x7fb9400e51a0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93f87e700 1 --2- 172.21.2.117:0/1847649589 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9400e51a0 0x7fb940079000 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7fb938006f00 tx=0x7fb938009f10).ready entity=mon.0 client_cookie=cc97cba15117e389 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93e07b700 1 -- 172.21.2.117:0/1847649589 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93800b3b0 con 0x7fb9400e51a0 2021-02-10T11:39:24.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93e07b700 1 -- 172.21.2.117:0/1847649589 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93800b630 con 0x7fb9400e51a0 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.008 7fb93e07b700 1 -- 172.21.2.117:0/1847649589 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93800bb50 con 0x7fb9400e51a0 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 -- 172.21.2.117:0/1847649589 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9400e51a0 msgr2=0x7fb940079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 --2- 172.21.2.117:0/1847649589 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9400e51a0 0x7fb940079000 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7fb938006f00 tx=0x7fb938009f10).stop 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 -- 172.21.2.117:0/1847649589 shutdown_connections 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 --2- 172.21.2.117:0/1847649589 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9400e51a0 0x7fb940079000 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 --2- 172.21.2.117:0/1847649589 >> v2:172.21.2.125:3300/0 conn(0x7fb94007da20 0x7fb9400771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 -- 172.21.2.117:0/1847649589 >> 172.21.2.117:0/1847649589 conn(0x7fb9400ce450 msgr2=0x7fb9400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 -- 172.21.2.117:0/1847649589 shutdown_connections 2021-02-10T11:39:24.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.009 7fb945266700 1 -- 172.21.2.117:0/1847649589 wait complete. 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 Processor -- start 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 -- start start 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9401a0250 0x7fb940106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb940187e20 0x7fb940188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9401067f0 con 0x7fb940090420 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb940107350 con 0x7fb9401a0250 2021-02-10T11:39:24.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb945266700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9401042d0 con 0x7fb940187e20 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb93f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb93e87c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb940187e20 0x7fb940188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb93f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46984/0 (socket says 172.21.2.117:46984) 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb93f87e700 1 -- 172.21.2.117:0/953080085 learned_addr learned my addr 172.21.2.117:0/953080085 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.024 7fb93f07d700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9401a0250 0x7fb940106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 -- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb940187e20 msgr2=0x7fb940188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb940187e20 0x7fb940188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 -- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9401a0250 msgr2=0x7fb940106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9401a0250 0x7fb940106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 -- 172.21.2.117:0/953080085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940090e40 con 0x7fb940090420 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb93f87e700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7fb93800b9b0 tx=0x7fb93800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb938003030 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93800c2f0 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb93800c810 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb945266700 1 -- 172.21.2.117:0/953080085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb940090e40 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.025 7fb945266700 1 -- 172.21.2.117:0/953080085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb940090e40 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.026 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb938009450 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.026 7fb937fff700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.026 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb938032a90 con 0x7fb940090420 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.026 7fb93f07d700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.027 7fb93f07d700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 0x7fb92401abc0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.057 7fb945266700 1 -- 172.21.2.117:0/953080085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb940103280 con 0x7fb940090420 2021-02-10T11:39:24.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.059 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb93802cfd0 con 0x7fb940090420 2021-02-10T11:39:24.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.186 7fb945266700 1 -- 172.21.2.117:0/953080085 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802517", "target": ["mgr", ""]}) v1 -- 0x7fb940103280 con 0x7fb9240186c0 2021-02-10T11:39:24.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.196 7fb937fff700 1 -- 172.21.2.117:0/953080085 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb928006a70 con 0x7fb9240186c0 2021-02-10T11:39:24.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 msgr2=0x7fb92401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 0x7fb92401abc0 crc :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 msgr2=0x7fb940090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7fb93800b9b0 tx=0x7fb93800bd50).stop 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 shutdown_connections 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9240186c0 0x7fb92401abc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb940090420 0x7fb940090820 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb940187e20 0x7fb940188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 --2- 172.21.2.117:0/953080085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9401a0250 0x7fb940106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 >> 172.21.2.117:0/953080085 conn(0x7fb9400a1850 msgr2=0x7fb9400c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 shutdown_connections 2021-02-10T11:39:24.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.204 7fb945266700 1 -- 172.21.2.117:0/953080085 wait complete. 2021-02-10T11:39:24.193 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802518 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 Processor -- start 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 -- start start 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fafb407da20 0x7fafb40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 0x7fafb41b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb4077a70 con 0x7fafb40e5180 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40777d0 con 0x7fafb407da20 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafba918700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb407de20 con 0x7fafb40e4d80 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb3fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fafb407da20 0x7fafb40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb8f35700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 0x7fafb41b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb3fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fafb407da20 0x7fafb40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51372/0 (socket says 172.21.2.117:51372) 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb8f35700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46988/0 (socket says 172.21.2.117:46988) 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 0x7fafb41b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44670/0 (socket says 172.21.2.117:44670) 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.625 7fafb3fff700 1 -- 172.21.2.117:0/106668959 learned_addr learned my addr 172.21.2.117:0/106668959 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 -- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 msgr2=0x7fafb41b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:24.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 --2- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 0x7fafb41b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 -- 172.21.2.117:0/106668959 >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 msgr2=0x7fafb4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 --2- 172.21.2.117:0/106668959 >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 -- 172.21.2.117:0/106668959 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb41b06f0 con 0x7fafb407da20 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb8f35700 1 --2- 172.21.2.117:0/106668959 >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb3fff700 1 --2- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb407da20 0x7fafb40771b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fafa4006700 tx=0x7fafa4009b80).ready entity=mon.1 client_cookie=75fb024a8cbc8a4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb2ffd700 1 -- 172.21.2.117:0/106668959 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa400c0c0 con 0x7fafb407da20 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb2ffd700 1 -- 172.21.2.117:0/106668959 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa400c440 con 0x7fafb407da20 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafb2ffd700 1 -- 172.21.2.117:0/106668959 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa400c920 con 0x7fafb407da20 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafba918700 1 -- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb407da20 msgr2=0x7fafb40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.626 7fafba918700 1 --2- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb407da20 0x7fafb40771b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fafa4006700 tx=0x7fafa4009b80).stop 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 -- 172.21.2.117:0/106668959 shutdown_connections 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 --2- 172.21.2.117:0/106668959 >> v2:172.21.2.117:3300/0 conn(0x7fafb40e5180 0x7fafb4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 --2- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb40e4d80 0x7fafb41b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 --2- 172.21.2.117:0/106668959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb407da20 0x7fafb40771b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 -- 172.21.2.117:0/106668959 >> 172.21.2.117:0/106668959 conn(0x7fafb40ce430 msgr2=0x7fafb40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 -- 172.21.2.117:0/106668959 shutdown_connections 2021-02-10T11:39:24.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.627 7fafba918700 1 -- 172.21.2.117:0/106668959 wait complete. 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.641 7fafba918700 1 Processor -- start 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 -- start start 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 0x7fafb40978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 0x7fafb40ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb41059a0 con 0x7fafb41a4430 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4097f00 con 0x7fafb41a4030 2021-02-10T11:39:24.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafba918700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40ac510 con 0x7fafb4105ea0 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafb8f35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafb8f35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:46992/0 (socket says 172.21.2.117:46992) 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafb8f35700 1 -- 172.21.2.117:0/3483633267 learned_addr learned my addr 172.21.2.117:0/3483633267 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafb3fff700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 0x7fafb40978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.642 7fafb37fe700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 0x7fafb40ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 -- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 msgr2=0x7fafb40978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 0x7fafb40978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 -- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 msgr2=0x7fafb40ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 0x7fafb40ab8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 -- 172.21.2.117:0/3483633267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb408dc10 con 0x7fafb41a4430 2021-02-10T11:39:24.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb37fe700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 0x7fafb40ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb3fff700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 0x7fafb40978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb8f35700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fafac006230 tx=0x7fafac00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafac000e50 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac00c300 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafba918700 1 -- 172.21.2.117:0/3483633267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb408dc10 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafac00c7e0 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.643 7fafba918700 1 -- 172.21.2.117:0/3483633267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb408dc10 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.644 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fafac001270 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.644 7fafb0ff9700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 0x7faf9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.644 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fafac032b30 con 0x7fafb41a4430 2021-02-10T11:39:24.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.644 7fafb3fff700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 0x7faf9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:24.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.644 7fafb3fff700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 0x7faf9401abc0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.675 7fafba918700 1 -- 172.21.2.117:0/3483633267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb4097f00 con 0x7fafb41a4430 2021-02-10T11:39:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.677 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fafac000e50 con 0x7fafb41a4430 2021-02-10T11:39:24.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.807 7fafba918700 1 -- 172.21.2.117:0/3483633267 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802518", "target": ["mgr", ""]}) v1 -- 0x7fafb4097f00 con 0x7faf94018730 2021-02-10T11:39:24.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.816 7fafb0ff9700 1 -- 172.21.2.117:0/3483633267 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fafa4006e80 con 0x7faf94018730 2021-02-10T11:39:24.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 -- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 msgr2=0x7faf9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 0x7faf9401abc0 crc :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 -- 172.21.2.117:0/3483633267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 msgr2=0x7fafb40abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fafac006230 tx=0x7fafac00ac80).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 -- 172.21.2.117:0/3483633267 shutdown_connections 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf94018730 0x7faf9401abc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb41a4430 0x7fafb40abef0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb41a4030 0x7fafb40ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 --2- 172.21.2.117:0/3483633267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb4105ea0 0x7fafb40978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.824 7fafba918700 1 -- 172.21.2.117:0/3483633267 >> 172.21.2.117:0/3483633267 conn(0x7fafb408a180 msgr2=0x7fafb408ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.825 7fafba918700 1 -- 172.21.2.117:0/3483633267 shutdown_connections 2021-02-10T11:39:24.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:24.825 7fafba918700 1 -- 172.21.2.117:0/3483633267 wait complete. 2021-02-10T11:39:24.815 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802519 2021-02-10T11:39:25.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 Processor -- start 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 -- start start 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c07da20 0x7f005c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f005c0e4da0 0x7f005c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c077a70 con 0x7f005c0e51a0 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c0e5c20 con 0x7f005c0e4da0 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.246 7f006271f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0777d0 con 0x7f005c07da20 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f005c0e4da0 0x7f005c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f005c0e4da0 0x7f005c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47000/0 (socket says 172.21.2.117:47000) 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 -- 172.21.2.117:0/790066644 learned_addr learned my addr 172.21.2.117:0/790066644 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005bfff700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c07da20 0x7f005c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 -- 172.21.2.117:0/790066644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c07da20 msgr2=0x7f005c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c07da20 0x7f005c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 -- 172.21.2.117:0/790066644 >> v1:172.21.2.125:6789/0 conn(0x7f005c0e51a0 legacy=0x7f005c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 -- 172.21.2.117:0/790066644 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c07de20 con 0x7f005c0e4da0 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.247 7f005b7fe700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0e4da0 0x7f005c1b00f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f0050006700 tx=0x7f0050009b80).ready entity=mon.0 client_cookie=a4b98c45106c911a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f005affd700 1 -- 172.21.2.117:0/790066644 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005000c0c0 con 0x7f005c0e4da0 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f005affd700 1 -- 172.21.2.117:0/790066644 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005000c400 con 0x7f005c0e4da0 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f005affd700 1 -- 172.21.2.117:0/790066644 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005000c940 con 0x7f005c0e4da0 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 -- 172.21.2.117:0/790066644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0e4da0 msgr2=0x7f005c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0e4da0 0x7f005c1b00f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f0050006700 tx=0x7f0050009b80).stop 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 -- 172.21.2.117:0/790066644 shutdown_connections 2021-02-10T11:39:25.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0e4da0 0x7f005c1b00f0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 --2- 172.21.2.117:0/790066644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c07da20 0x7f005c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 -- 172.21.2.117:0/790066644 >> 172.21.2.117:0/790066644 conn(0x7f005c0ce450 msgr2=0x7f005c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 -- 172.21.2.117:0/790066644 shutdown_connections 2021-02-10T11:39:25.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.248 7f006271f700 1 -- 172.21.2.117:0/790066644 wait complete. 2021-02-10T11:39:25.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.263 7f006271f700 1 Processor -- start 2021-02-10T11:39:25.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.263 7f006271f700 1 -- start start 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.263 7f006271f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f006271f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0c2340 0x7f005c0c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f006271f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 0x7f005c0b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f006271f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0b8010 con 0x7f005c0b7390 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f006271f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0b8550 con 0x7f005c0a41c0 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f006271f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0b8770 con 0x7f005c0c2340 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f0060d3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 0x7f005c0b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51390/0 (socket says 172.21.2.117:51390) 2021-02-10T11:39:25.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f0060d3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 0x7f005c0b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47008/0 (socket says 172.21.2.117:47008) 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 -- 172.21.2.117:0/3353374501 learned_addr learned my addr 172.21.2.117:0/3353374501 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005b7fe700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0c2340 0x7f005c0c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 -- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0c2340 msgr2=0x7f005c0c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0c2340 0x7f005c0c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 -- 172.21.2.117:0/3353374501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 msgr2=0x7f005c0b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 0x7f005c0b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 -- 172.21.2.117:0/3353374501 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0b8990 con 0x7f005c0a41c0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.264 7f005bfff700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0054009fc0 tx=0x7f005400bcd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0054006030 con 0x7f005c0a41c0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0054006330 con 0x7f005c0a41c0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00540058e0 con 0x7f005c0a41c0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f006271f700 1 -- 172.21.2.117:0/3353374501 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c0b8990 con 0x7f005c0a41c0 2021-02-10T11:39:25.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f006271f700 1 -- 172.21.2.117:0/3353374501 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c0b8990 con 0x7f005c0a41c0 2021-02-10T11:39:25.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0054006030 con 0x7f005c0a41c0 2021-02-10T11:39:25.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 0x7f003c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0054032d50 con 0x7f005c0a41c0 2021-02-10T11:39:25.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.265 7f005b7fe700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 0x7f003c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.266 7f005b7fe700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 0x7f003c01ac10 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.296 7f006271f700 1 -- 172.21.2.117:0/3353374501 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c1b4f00 con 0x7f005c0a41c0 2021-02-10T11:39:25.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.299 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f005400be10 con 0x7f005c0a41c0 2021-02-10T11:39:25.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.429 7f006271f700 1 -- 172.21.2.117:0/3353374501 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802519", "target": ["mgr", ""]}) v1 -- 0x7f005c1b4f00 con 0x7f003c018730 2021-02-10T11:39:25.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.439 7f0058ff9700 1 -- 172.21.2.117:0/3353374501 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f0050000c90 con 0x7f003c018730 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 -- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 msgr2=0x7f003c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 0x7f003c01ac10 crc :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 -- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 msgr2=0x7f005c0fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0054009fc0 tx=0x7f005400bcd0).stop 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 -- 172.21.2.117:0/3353374501 shutdown_connections 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c018730 0x7f003c01ac10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0b7390 0x7f005c0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0c2340 0x7f005c0c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 --2- 172.21.2.117:0/3353374501 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a41c0 0x7f005c0fe0d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.446 7f006271f700 1 -- 172.21.2.117:0/3353374501 >> 172.21.2.117:0/3353374501 conn(0x7f005c08bbc0 msgr2=0x7f005c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.447 7f006271f700 1 -- 172.21.2.117:0/3353374501 shutdown_connections 2021-02-10T11:39:25.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.447 7f006271f700 1 -- 172.21.2.117:0/3353374501 wait complete. 2021-02-10T11:39:25.437 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802520 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.871 7f897e834700 1 Processor -- start 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.871 7f897e834700 1 -- start start 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897e834700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f897807da20 0x7f89780771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897e834700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89780d1d20 con 0x7f89780e51a0 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897e834700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8978077a70 con 0x7f89780e4da0 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897e834700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89780777d0 con 0x7f897807da20 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897ce51700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f89780e51a0 0x7f89780d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47264/0 (socket says 172.21.2.117:47264) 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f897ce51700 1 -- 172.21.2.117:0/2979503004 learned_addr learned my addr 172.21.2.117:0/2979503004 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8977fff700 1 --2- 172.21.2.117:0/2979503004 >> v2:172.21.2.125:3300/0 conn(0x7f897807da20 0x7f89780771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100260532 0 0) 0x7f8970001500 con 0x7f89780e51a0 2021-02-10T11:39:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8960002fe0 con 0x7f89780e51a0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502626229 0 0) 0x7f896c000ff0 con 0x7f89780e4da0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89600042c0 con 0x7f89780e4da0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.1 v1:172.21.2.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 867677608 0 0) 0x7f896c000fd0 con 0x7f89780e4da0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 >> v2:172.21.2.125:3300/0 conn(0x7f897807da20 msgr2=0x7f89780771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 --2- 172.21.2.117:0/2979503004 >> v2:172.21.2.125:3300/0 conn(0x7f897807da20 0x7f89780771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 >> v1:172.21.2.117:6789/0 conn(0x7f89780e51a0 legacy=0x7f89780d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> v1:172.21.2.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89781b08e0 con 0x7f89780e4da0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.1 v1:172.21.2.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f896c001510 con 0x7f89780e4da0 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 >> v1:172.21.2.125:6789/0 conn(0x7f89780e4da0 legacy=0x7f89781b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89600037f0 0x7f8960003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8960006830 0x7f8960008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 0x7f896000bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896000c3a0 con 0x7f8960009630 2021-02-10T11:39:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.872 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8960050590 con 0x7f89600037f0 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89600507b0 con 0x7f8960006830 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8977fff700 1 --2- 172.21.2.117:0/2979503004 >> v2:172.21.2.125:3300/0 conn(0x7f897807da20 0x7f89780771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 0x7f896000bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f89777fe700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8960006830 0x7f8960008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8977fff700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89600037f0 0x7f8960003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 -- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8960006830 msgr2=0x7f8960008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8960006830 0x7f8960008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 -- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89600037f0 msgr2=0x7f8960003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89600037f0 0x7f8960003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 -- 172.21.2.117:0/2979503004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8960050a50 con 0x7f8960009630 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 -- 172.21.2.117:0/2979503004 reap_dead start 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897ce51700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 0x7f896000bb50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8970007ce0 tx=0x7f8970002140).ready entity=mon.0 client_cookie=c6a7aaef38f31258 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8970007d70 con 0x7f8960009630 2021-02-10T11:39:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8970007180 con 0x7f8960009630 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f8976ffd700 1 -- 172.21.2.117:0/2979503004 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8970004720 con 0x7f8960009630 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897e834700 1 -- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 msgr2=0x7f896000bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.873 7f897e834700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 0x7f896000bb50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7f8970007ce0 tx=0x7f8970002140).stop 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.874 7f897e834700 1 -- 172.21.2.117:0/2979503004 shutdown_connections 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.874 7f897e834700 1 --2- 172.21.2.117:0/2979503004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8960009630 0x7f896000bb50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.874 7f897e834700 1 -- 172.21.2.117:0/2979503004 >> 172.21.2.117:0/2979503004 conn(0x7f89780ce450 msgr2=0x7f89780e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.874 7f897e834700 1 -- 172.21.2.117:0/2979503004 shutdown_connections 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.874 7f897e834700 1 -- 172.21.2.117:0/2979503004 wait complete. 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 Processor -- start 2021-02-10T11:39:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 -- start start 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89780a47d0 0x7f89780a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89780b77d0 0x7f89780b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978189970 con 0x7f89780a51f0 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8978190aa0 con 0x7f89780b77d0 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.889 7f897e834700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89780b81f0 con 0x7f89780a47d0 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47024/0 (socket says 172.21.2.117:47024) 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 -- 172.21.2.117:0/14669098 learned_addr learned my addr 172.21.2.117:0/14669098 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f8977fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89780a47d0 0x7f89780a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f897ce51700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89780b77d0 0x7f89780b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 -- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89780a47d0 msgr2=0x7f89780a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89780a47d0 0x7f89780a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 -- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89780b77d0 msgr2=0x7f89780b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89780b77d0 0x7f89780b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 -- 172.21.2.117:0/14669098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89780b8410 con 0x7f89780a51f0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f89777fe700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f896c0095c0 tx=0x7f896c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f896c00bd80 con 0x7f89780a51f0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896c002080 con 0x7f89780a51f0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f896c002520 con 0x7f89780a51f0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f897e834700 1 -- 172.21.2.117:0/14669098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89780b8410 con 0x7f89780a51f0 2021-02-10T11:39:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.890 7f897e834700 1 -- 172.21.2.117:0/14669098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89780b89a0 con 0x7f89780a51f0 2021-02-10T11:39:25.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.891 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f896c002260 con 0x7f89780a51f0 2021-02-10T11:39:25.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.891 7f8974ff9700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 0x7f895801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:25.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.891 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f896c032550 con 0x7f89780a51f0 2021-02-10T11:39:25.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.891 7f8977fff700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 0x7f895801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:25.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.891 7f8977fff700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 0x7f895801abc0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:25.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.921 7f897e834700 1 -- 172.21.2.117:0/14669098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978077a70 con 0x7f89780a51f0 2021-02-10T11:39:25.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:25.923 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f896c009690 con 0x7f89780a51f0 2021-02-10T11:39:26.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.051 7f897e834700 1 -- 172.21.2.117:0/14669098 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802520", "target": ["mgr", ""]}) v1 -- 0x7f8978189970 con 0x7f89580186c0 2021-02-10T11:39:26.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.061 7f8974ff9700 1 -- 172.21.2.117:0/14669098 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8968006af0 con 0x7f89580186c0 2021-02-10T11:39:26.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.067 7f897e834700 1 -- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 msgr2=0x7f895801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.067 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 0x7f895801abc0 crc :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.067 7f897e834700 1 -- 172.21.2.117:0/14669098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 msgr2=0x7f89780a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.067 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f896c0095c0 tx=0x7f896c00b990).stop 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 -- 172.21.2.117:0/14669098 shutdown_connections 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89580186c0 0x7f895801abc0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89780a51f0 0x7f89780a55f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89780b77d0 0x7f89780b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 --2- 172.21.2.117:0/14669098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89780a47d0 0x7f89780a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 -- 172.21.2.117:0/14669098 >> 172.21.2.117:0/14669098 conn(0x7f89780b5140 msgr2=0x7f89780a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 -- 172.21.2.117:0/14669098 shutdown_connections 2021-02-10T11:39:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.068 7f897e834700 1 -- 172.21.2.117:0/14669098 wait complete. 2021-02-10T11:39:26.058 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802521 2021-02-10T11:39:26.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 Processor -- start 2021-02-10T11:39:26.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 -- start start 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f170007da20 0x7f17000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 0x7f1700079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1700077a70 con 0x7f17000e4da0 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17000e5c20 con 0x7f170007da20 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.489 7f170533f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17000777d0 con 0x7f17000e51a0 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16fe87c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17000e4da0 0x7f17001b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47282/0 (socket says 172.21.2.117:47282) 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f170007da20 0x7f17000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16fe87c700 1 -- 172.21.2.117:0/1742713711 learned_addr learned my addr 172.21.2.117:0/1742713711 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff87e700 1 --2- 172.21.2.117:0/1742713711 >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 0x7f1700079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 -- 172.21.2.117:0/1742713711 >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 msgr2=0x7f1700079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:26.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 --2- 172.21.2.117:0/1742713711 >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 0x7f1700079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 -- 172.21.2.117:0/1742713711 >> v1:172.21.2.117:6789/0 conn(0x7f17000e4da0 legacy=0x7f17001b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 -- 172.21.2.117:0/1742713711 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17001b08e0 con 0x7f170007da20 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff87e700 1 --2- 172.21.2.117:0/1742713711 >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 0x7f1700079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.490 7f16ff07d700 1 --2- 172.21.2.117:0/1742713711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f170007da20 0x7f17000771b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f16f0006650 tx=0x7f16f0009a80).ready entity=mon.0 client_cookie=cc647fe0027ff8e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f16fe07b700 1 -- 172.21.2.117:0/1742713711 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f00009b0 con 0x7f170007da20 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f16fe07b700 1 -- 172.21.2.117:0/1742713711 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f000c0c0 con 0x7f170007da20 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f16fe07b700 1 -- 172.21.2.117:0/1742713711 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f000c5e0 con 0x7f170007da20 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 -- 172.21.2.117:0/1742713711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f170007da20 msgr2=0x7f17000771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 --2- 172.21.2.117:0/1742713711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f170007da20 0x7f17000771b0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0x7f16f0006650 tx=0x7f16f0009a80).stop 2021-02-10T11:39:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 -- 172.21.2.117:0/1742713711 shutdown_connections 2021-02-10T11:39:26.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 --2- 172.21.2.117:0/1742713711 >> v2:172.21.2.125:3300/0 conn(0x7f17000e51a0 0x7f1700079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 --2- 172.21.2.117:0/1742713711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f170007da20 0x7f17000771b0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.491 7f170533f700 1 -- 172.21.2.117:0/1742713711 >> 172.21.2.117:0/1742713711 conn(0x7f17000ce450 msgr2=0x7f17000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.492 7f170533f700 1 -- 172.21.2.117:0/1742713711 shutdown_connections 2021-02-10T11:39:26.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.492 7f170533f700 1 -- 172.21.2.117:0/1742713711 wait complete. 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.506 7f170533f700 1 Processor -- start 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.506 7f170533f700 1 -- start start 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.506 7f170533f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17001a0250 0x7f1700106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f170533f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1700187e20 0x7f1700188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f170533f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f170533f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001067f0 con 0x7f17001a0250 2021-02-10T11:39:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f170533f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700107350 con 0x7f1700090420 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f170533f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17001042d0 con 0x7f1700187e20 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17001a0250 0x7f1700106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51418/0 (socket says 172.21.2.117:51418) 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 -- 172.21.2.117:0/1320432566 learned_addr learned my addr 172.21.2.117:0/1320432566 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16fe87c700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1700187e20 0x7f1700188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 -- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1700187e20 msgr2=0x7f1700188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1700187e20 0x7f1700188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 -- 172.21.2.117:0/1320432566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17001a0250 msgr2=0x7f1700106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17001a0250 0x7f1700106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 -- 172.21.2.117:0/1320432566 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1700090e40 con 0x7f1700090420 2021-02-10T11:39:26.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16ff87e700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f16f800b970 tx=0x7f16f800bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.507 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f8000f90 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f8006790 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16f80015f0 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f170533f700 1 -- 172.21.2.117:0/1320432566 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700090e40 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f170533f700 1 -- 172.21.2.117:0/1320432566 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17000913d0 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16f80011f0 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f16e7fff700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 0x7f16e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.508 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f16f8032a70 con 0x7f1700090420 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.509 7f16ff07d700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 0x7f16e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:26.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.509 7f16ff07d700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 0x7f16e001abc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:26.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.539 7f170533f700 1 -- 172.21.2.117:0/1320432566 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700103280 con 0x7f1700090420 2021-02-10T11:39:26.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.543 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16f802f460 con 0x7f1700090420 2021-02-10T11:39:26.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.672 7f170533f700 1 -- 172.21.2.117:0/1320432566 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802521", "target": ["mgr", ""]}) v1 -- 0x7f1700091c60 con 0x7f16e00186c0 2021-02-10T11:39:26.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.682 7f16e7fff700 1 -- 172.21.2.117:0/1320432566 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f16f0006420 con 0x7f16e00186c0 2021-02-10T11:39:26.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.689 7f170533f700 1 -- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 msgr2=0x7f16e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.689 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 0x7f16e001abc0 crc :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.689 7f170533f700 1 -- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 msgr2=0x7f1700090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:26.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f16f800b970 tx=0x7f16f800bd10).stop 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 -- 172.21.2.117:0/1320432566 shutdown_connections 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1700090420 0x7f1700090820 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1700187e20 0x7f1700188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16e00186c0 0x7f16e001abc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 --2- 172.21.2.117:0/1320432566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17001a0250 0x7f1700106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 -- 172.21.2.117:0/1320432566 >> 172.21.2.117:0/1320432566 conn(0x7f17000a1850 msgr2=0x7f17000c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:26.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 -- 172.21.2.117:0/1320432566 shutdown_connections 2021-02-10T11:39:26.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:26.690 7f170533f700 1 -- 172.21.2.117:0/1320432566 wait complete. 2021-02-10T11:39:26.680 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802522 2021-02-10T11:39:27.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.111 7f85de649700 1 Processor -- start 2021-02-10T11:39:27.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.111 7f85de649700 1 -- start start 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85de649700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85de649700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f85d80e51a0 0x7f85d8079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85de649700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d8077a70 con 0x7f85d80e4da0 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85de649700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80e5c20 con 0x7f85d807da20 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85de649700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80777d0 con 0x7f85d80e51a0 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f85d80e51a0 0x7f85d8079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85d7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f85d80e51a0 0x7f85d8079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51428/0 (socket says 172.21.2.117:51428) 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85d7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47042/0 (socket says 172.21.2.117:47042) 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 -- 172.21.2.117:0/3371384840 learned_addr learned my addr 172.21.2.117:0/3371384840 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85d77fe700 1 --1- 172.21.2.117:0/3371384840 >> v1:172.21.2.125:6789/0 conn(0x7f85d80e4da0 0x7f85d81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40222/0 (socket says 172.21.2.117:40222) 2021-02-10T11:39:27.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 -- 172.21.2.117:0/3371384840 >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 msgr2=0x7f85d80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 --2- 172.21.2.117:0/3371384840 >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 -- 172.21.2.117:0/3371384840 >> v1:172.21.2.125:6789/0 conn(0x7f85d80e4da0 legacy=0x7f85d81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 -- 172.21.2.117:0/3371384840 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d81b08e0 con 0x7f85d80e51a0 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85d7fff700 1 --2- 172.21.2.117:0/3371384840 >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.112 7f85dcc66700 1 --2- 172.21.2.117:0/3371384840 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d80e51a0 0x7f85d8079000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f85d0006aa0 tx=0x7f85d0009f20).ready entity=mon.1 client_cookie=80b6f506d44776f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85d6ffd700 1 -- 172.21.2.117:0/3371384840 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85d000b3b0 con 0x7f85d80e51a0 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85d6ffd700 1 -- 172.21.2.117:0/3371384840 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0003030 con 0x7f85d80e51a0 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85d6ffd700 1 -- 172.21.2.117:0/3371384840 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85d000b970 con 0x7f85d80e51a0 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 -- 172.21.2.117:0/3371384840 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d80e51a0 msgr2=0x7f85d8079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 --2- 172.21.2.117:0/3371384840 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d80e51a0 0x7f85d8079000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f85d0006aa0 tx=0x7f85d0009f20).stop 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 -- 172.21.2.117:0/3371384840 shutdown_connections 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 --2- 172.21.2.117:0/3371384840 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d80e51a0 0x7f85d8079000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 --2- 172.21.2.117:0/3371384840 >> v2:172.21.2.117:3300/0 conn(0x7f85d807da20 0x7f85d80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 -- 172.21.2.117:0/3371384840 >> 172.21.2.117:0/3371384840 conn(0x7f85d80ce450 msgr2=0x7f85d80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.113 7f85de649700 1 -- 172.21.2.117:0/3371384840 shutdown_connections 2021-02-10T11:39:27.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.114 7f85de649700 1 -- 172.21.2.117:0/3371384840 wait complete. 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.128 7f85de649700 1 Processor -- start 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.128 7f85de649700 1 -- start start 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85d81a0250 0x7f85d8106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d8090420 0x7f85d8090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81067f0 con 0x7f85d8187e20 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d8107350 con 0x7f85d8090420 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85de649700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81042d0 con 0x7f85d81a0250 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47050/0 (socket says 172.21.2.117:47050) 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 -- 172.21.2.117:0/2109716489 learned_addr learned my addr 172.21.2.117:0/2109716489 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d7fff700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85d81a0250 0x7f85d8106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85dcc66700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d8090420 0x7f85d8090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 -- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85d81a0250 msgr2=0x7f85d8106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85d81a0250 0x7f85d8106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 -- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d8090420 msgr2=0x7f85d8090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d8090420 0x7f85d8090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 -- 172.21.2.117:0/2109716489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8090e40 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.129 7f85d77fe700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f85cc00a4c0 tx=0x7f85cc00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85cc00ac50 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc004140 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85cc004660 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85de649700 1 -- 172.21.2.117:0/2109716489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d8090e40 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85de649700 1 -- 172.21.2.117:0/2109716489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d8090e40 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f85cc000ce0 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 0x7f85b801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.130 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f85cc032920 con 0x7f85d8187e20 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.131 7f85d7fff700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 0x7f85b801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.131 7f85d7fff700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 0x7f85b801abd0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.161 7f85de649700 1 -- 172.21.2.117:0/2109716489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d8103280 con 0x7f85d8187e20 2021-02-10T11:39:27.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.164 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f85cc0009b0 con 0x7f85d8187e20 2021-02-10T11:39:27.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.289 7f85de649700 1 -- 172.21.2.117:0/2109716489 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802522", "target": ["mgr", ""]}) v1 -- 0x7f85d8107310 con 0x7f85b80186d0 2021-02-10T11:39:27.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.300 7f85d4ff9700 1 -- 172.21.2.117:0/2109716489 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f85c8006a30 con 0x7f85b80186d0 2021-02-10T11:39:27.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.306 7f85de649700 1 -- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 msgr2=0x7f85b801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 0x7f85b801abd0 crc :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 -- 172.21.2.117:0/2109716489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 msgr2=0x7f85d8188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f85cc00a4c0 tx=0x7f85cc00a860).stop 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 -- 172.21.2.117:0/2109716489 shutdown_connections 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f85b80186d0 0x7f85b801abd0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85d8187e20 0x7f85d8188220 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f85d8090420 0x7f85d8090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 --2- 172.21.2.117:0/2109716489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85d81a0250 0x7f85d8106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 -- 172.21.2.117:0/2109716489 >> 172.21.2.117:0/2109716489 conn(0x7f85d80a1850 msgr2=0x7f85d80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 -- 172.21.2.117:0/2109716489 shutdown_connections 2021-02-10T11:39:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.307 7f85de649700 1 -- 172.21.2.117:0/2109716489 wait complete. 2021-02-10T11:39:27.297 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802523 2021-02-10T11:39:27.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.728 7f74feade700 1 Processor -- start 2021-02-10T11:39:27.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.728 7f74feade700 1 -- start start 2021-02-10T11:39:27.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74feade700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f74f8079000 0x7f74f80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74feade700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80d1d20 con 0x7f74f81b00f0 2021-02-10T11:39:27.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74feade700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8077a70 con 0x7f74f807da20 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74feade700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f80e53c0 con 0x7f74f8079000 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74fd0fb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f74f81b00f0 0x7f74f80d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47306/0 (socket says 172.21.2.117:47306) 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74fd0fb700 1 -- 172.21.2.117:0/58823408 learned_addr learned my addr 172.21.2.117:0/58823408 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.729 7f74f7fff700 1 --2- 172.21.2.117:0/58823408 >> v2:172.21.2.117:3300/0 conn(0x7f74f8079000 0x7f74f80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f77fe700 1 -- 172.21.2.117:0/58823408 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344155546 0 0) 0x7f74f0000c10 con 0x7f74f81b00f0 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f77fe700 1 -- 172.21.2.117:0/58823408 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e4002fe0 con 0x7f74f81b00f0 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f7fff700 1 -- 172.21.2.117:0/58823408 >> v1:172.21.2.125:6789/0 conn(0x7f74f807da20 legacy=0x7f74f80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f7fff700 1 -- 172.21.2.117:0/58823408 >> v1:172.21.2.117:6789/0 conn(0x7f74f81b00f0 legacy=0x7f74f80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f7fff700 1 -- 172.21.2.117:0/58823408 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f8079400 con 0x7f74f8079000 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f7fff700 1 --2- 172.21.2.117:0/58823408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f8079000 0x7f74f80e4da0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f74e80069d0 tx=0x7f74e8009dc0).ready entity=mon.0 client_cookie=4687ee7d725677b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f77fe700 1 -- 172.21.2.117:0/58823408 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e800b510 con 0x7f74f8079000 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f77fe700 1 -- 172.21.2.117:0/58823408 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e8006ac0 con 0x7f74f8079000 2021-02-10T11:39:27.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74f77fe700 1 -- 172.21.2.117:0/58823408 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74e800ba80 con 0x7f74f8079000 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74feade700 1 -- 172.21.2.117:0/58823408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f8079000 msgr2=0x7f74f80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.730 7f74feade700 1 --2- 172.21.2.117:0/58823408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f8079000 0x7f74f80e4da0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f74e80069d0 tx=0x7f74e8009dc0).stop 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.731 7f74feade700 1 -- 172.21.2.117:0/58823408 shutdown_connections 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.731 7f74feade700 1 --2- 172.21.2.117:0/58823408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f8079000 0x7f74f80e4da0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.731 7f74feade700 1 -- 172.21.2.117:0/58823408 >> 172.21.2.117:0/58823408 conn(0x7f74f80ce450 msgr2=0x7f74f80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.731 7f74feade700 1 -- 172.21.2.117:0/58823408 shutdown_connections 2021-02-10T11:39:27.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.731 7f74feade700 1 -- 172.21.2.117:0/58823408 wait complete. 2021-02-10T11:39:27.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 Processor -- start 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 -- start start 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74f81a47d0 0x7f74f808a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 0x7f74f80a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 0x7f74f80a7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f808cfd0 con 0x7f74f80a7be0 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f808d550 con 0x7f74f808a7a0 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.746 7f74feade700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f808d730 con 0x7f74f81a47d0 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74f7fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 0x7f74f80a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 0x7f74f80a7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74f7fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 0x7f74f80a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51446/0 (socket says 172.21.2.117:51446) 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74f7fff700 1 -- 172.21.2.117:0/3480680083 learned_addr learned my addr 172.21.2.117:0/3480680083 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:27.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fc8fa700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74f81a47d0 0x7f74f808a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 -- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74f81a47d0 msgr2=0x7f74f808a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74f81a47d0 0x7f74f808a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 -- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 msgr2=0x7f74f80a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 0x7f74f80a75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 -- 172.21.2.117:0/3480680083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f808d950 con 0x7f74f80a7be0 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74fd0fb700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 0x7f74f80a7fe0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f74f000a4d0 tx=0x7f74f000a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.747 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f00070d0 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f000ac60 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74f0004620 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74feade700 1 -- 172.21.2.117:0/3480680083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f808d950 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74feade700 1 -- 172.21.2.117:0/3480680083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f808dee0 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74f00043a0 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74f57fa700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 0x7f74dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.748 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f74f0032670 con 0x7f74f80a7be0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.749 7f74fc8fa700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 0x7f74dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:27.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.749 7f74fc8fa700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 0x7f74dc01abc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:27.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.779 7f74feade700 1 -- 172.21.2.117:0/3480680083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f81032d0 con 0x7f74f80a7be0 2021-02-10T11:39:27.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.781 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74f0026f00 con 0x7f74f80a7be0 2021-02-10T11:39:27.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.911 7f74feade700 1 -- 172.21.2.117:0/3480680083 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802523", "target": ["mgr", ""]}) v1 -- 0x7f74f808d510 con 0x7f74dc0186c0 2021-02-10T11:39:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.923 7f74f57fa700 1 -- 172.21.2.117:0/3480680083 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f74e0006a90 con 0x7f74dc0186c0 2021-02-10T11:39:27.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.928 7f74feade700 1 -- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 msgr2=0x7f74dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.928 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 0x7f74dc01abc0 crc :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.928 7f74feade700 1 -- 172.21.2.117:0/3480680083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 msgr2=0x7f74f80a7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:27.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.928 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 0x7f74f80a7fe0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f74f000a4d0 tx=0x7f74f000a870).stop 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 -- 172.21.2.117:0/3480680083 shutdown_connections 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74dc0186c0 0x7f74dc01abc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74f80a7be0 0x7f74f80a7fe0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74f808a7a0 0x7f74f80a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 --2- 172.21.2.117:0/3480680083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74f81a47d0 0x7f74f808a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 -- 172.21.2.117:0/3480680083 >> 172.21.2.117:0/3480680083 conn(0x7f74f80a9210 msgr2=0x7f74f80a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 -- 172.21.2.117:0/3480680083 shutdown_connections 2021-02-10T11:39:27.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:27.929 7f74feade700 1 -- 172.21.2.117:0/3480680083 wait complete. 2021-02-10T11:39:27.919 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802524 2021-02-10T11:39:28.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.351 7efd7713e700 1 Processor -- start 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.351 7efd7713e700 1 -- start start 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7713e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 0x7efd700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7713e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efd700e4da0 0x7efd701b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7713e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd70077a70 con 0x7efd700e51a0 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7713e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd700e5c20 con 0x7efd700e4da0 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7713e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd700777d0 con 0x7efd7007da20 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7575b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7efd700e51a0 0x7efd70077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40242/0 (socket says 172.21.2.117:40242) 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd7575b700 1 -- 172.21.2.117:0/1413874280 learned_addr learned my addr 172.21.2.117:0/1413874280 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd6ffff700 1 --2- 172.21.2.117:0/1413874280 >> v2:172.21.2.125:3300/0 conn(0x7efd700e4da0 0x7efd701b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd74f5a700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 0x7efd700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.352 7efd6f7fe700 1 -- 172.21.2.117:0/1413874280 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166519520 0 0) 0x7efd68001190 con 0x7efd700e51a0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6f7fe700 1 -- 172.21.2.117:0/1413874280 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5c002fe0 con 0x7efd700e51a0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6ffff700 1 -- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 msgr2=0x7efd700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6ffff700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 0x7efd700771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6ffff700 1 -- 172.21.2.117:0/1413874280 >> v1:172.21.2.125:6789/0 conn(0x7efd700e51a0 legacy=0x7efd70077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6ffff700 1 -- 172.21.2.117:0/1413874280 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7007de20 con 0x7efd700e4da0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd74f5a700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 0x7efd700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6ffff700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd700e4da0 0x7efd701b00f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7efd600069d0 tx=0x7efd60009e40).ready entity=mon.1 client_cookie=445be07d467027d9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6f7fe700 1 -- 172.21.2.117:0/1413874280 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6000b510 con 0x7efd700e4da0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6f7fe700 1 -- 172.21.2.117:0/1413874280 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000b790 con 0x7efd700e4da0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd6f7fe700 1 -- 172.21.2.117:0/1413874280 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd6000bcb0 con 0x7efd700e4da0 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd7713e700 1 -- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd700e4da0 msgr2=0x7efd701b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.353 7efd7713e700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd700e4da0 0x7efd701b00f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7efd600069d0 tx=0x7efd60009e40).stop 2021-02-10T11:39:28.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 -- 172.21.2.117:0/1413874280 shutdown_connections 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd700e4da0 0x7efd701b00f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 --2- 172.21.2.117:0/1413874280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd7007da20 0x7efd700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 -- 172.21.2.117:0/1413874280 >> 172.21.2.117:0/1413874280 conn(0x7efd700ce450 msgr2=0x7efd700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 -- 172.21.2.117:0/1413874280 shutdown_connections 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.354 7efd7713e700 1 -- 172.21.2.117:0/1413874280 wait complete. 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 Processor -- start 2021-02-10T11:39:28.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 -- start start 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd701a0250 0x7efd70106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 0x7efd70188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701067f0 con 0x7efd70187e20 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70107350 con 0x7efd70090420 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.369 7efd7713e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd701042d0 con 0x7efd701a0250 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 0x7efd70188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 0x7efd70188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47070/0 (socket says 172.21.2.117:47070) 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:51456/0 (socket says 172.21.2.117:51456) 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6ffff700 1 -- 172.21.2.117:0/2372204959 learned_addr learned my addr 172.21.2.117:0/2372204959 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd74f5a700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd701a0250 0x7efd70106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 -- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd701a0250 msgr2=0x7efd70106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd701a0250 0x7efd70106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 -- 172.21.2.117:0/2372204959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 msgr2=0x7efd70188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 0x7efd70188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 -- 172.21.2.117:0/2372204959 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd70090e40 con 0x7efd70090420 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7575b700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7efd6800a4a0 tx=0x7efd6800a840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd68007400 con 0x7efd70090420 2021-02-10T11:39:28.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd680070c0 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7713e700 1 -- 172.21.2.117:0/2372204959 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd70090e40 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd7713e700 1 -- 172.21.2.117:0/2372204959 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd700913d0 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.370 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd68004320 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.371 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd68004320 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.371 7efd6d7fa700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 0x7efd5401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.371 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efd68032810 con 0x7efd70090420 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.371 7efd74f5a700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 0x7efd5401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.372 7efd74f5a700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 0x7efd5401abd0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.401 7efd7713e700 1 -- 172.21.2.117:0/2372204959 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd70103280 con 0x7efd70090420 2021-02-10T11:39:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.403 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd68000bc0 con 0x7efd70090420 2021-02-10T11:39:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.528 7efd7713e700 1 -- 172.21.2.117:0/2372204959 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802524", "target": ["mgr", ""]}) v1 -- 0x7efd70091c60 con 0x7efd540186d0 2021-02-10T11:39:28.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.540 7efd6d7fa700 1 -- 172.21.2.117:0/2372204959 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efd58006a30 con 0x7efd540186d0 2021-02-10T11:39:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 msgr2=0x7efd5401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 0x7efd5401abd0 crc :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 msgr2=0x7efd70090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7efd6800a4a0 tx=0x7efd6800a840).stop 2021-02-10T11:39:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 shutdown_connections 2021-02-10T11:39:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd540186d0 0x7efd5401abd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd70187e20 0x7efd70188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd70090420 0x7efd70090820 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 --2- 172.21.2.117:0/2372204959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd701a0250 0x7efd70106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 >> 172.21.2.117:0/2372204959 conn(0x7efd700a1850 msgr2=0x7efd700c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 shutdown_connections 2021-02-10T11:39:28.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.546 7efd7713e700 1 -- 172.21.2.117:0/2372204959 wait complete. 2021-02-10T11:39:28.536 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802525 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.968 7f58e3088700 1 Processor -- start 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.968 7f58e3088700 1 -- start start 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e3088700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58dc07da20 0x7f58dc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e3088700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58dc0e4da0 0x7f58dc1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e3088700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc077a70 con 0x7f58dc0e51a0 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e3088700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc0777d0 con 0x7f58dc07da20 2021-02-10T11:39:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e3088700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc07de20 con 0x7f58dc0e4da0 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e0ea4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58dc07da20 0x7f58dc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e0ea4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58dc07da20 0x7f58dc0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47076/0 (socket says 172.21.2.117:47076) 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.969 7f58e0ea4700 1 -- 172.21.2.117:0/3700416833 learned_addr learned my addr 172.21.2.117:0/3700416833 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e0ea4700 1 -- 172.21.2.117:0/3700416833 >> v2:172.21.2.125:3300/0 conn(0x7f58dc0e4da0 msgr2=0x7f58dc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e0ea4700 1 --2- 172.21.2.117:0/3700416833 >> v2:172.21.2.125:3300/0 conn(0x7f58dc0e4da0 0x7f58dc1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e0ea4700 1 -- 172.21.2.117:0/3700416833 >> v1:172.21.2.125:6789/0 conn(0x7f58dc0e51a0 legacy=0x7f58dc077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e0ea4700 1 -- 172.21.2.117:0/3700416833 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc1b0710 con 0x7f58dc07da20 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e0ea4700 1 --2- 172.21.2.117:0/3700416833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc07da20 0x7f58dc0771b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f58c4006700 tx=0x7f58c4009b80).ready entity=mon.0 client_cookie=eebb6273e58f988c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58db7fe700 1 -- 172.21.2.117:0/3700416833 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c400c0c0 con 0x7f58dc07da20 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58db7fe700 1 -- 172.21.2.117:0/3700416833 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c400c380 con 0x7f58dc07da20 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58db7fe700 1 -- 172.21.2.117:0/3700416833 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c400c8c0 con 0x7f58dc07da20 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e3088700 1 -- 172.21.2.117:0/3700416833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc07da20 msgr2=0x7f58dc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.970 7f58e3088700 1 --2- 172.21.2.117:0/3700416833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc07da20 0x7f58dc0771b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f58c4006700 tx=0x7f58c4009b80).stop 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 -- 172.21.2.117:0/3700416833 shutdown_connections 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 --2- 172.21.2.117:0/3700416833 >> v2:172.21.2.125:3300/0 conn(0x7f58dc0e4da0 0x7f58dc1b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 --2- 172.21.2.117:0/3700416833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc07da20 0x7f58dc0771b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 -- 172.21.2.117:0/3700416833 >> 172.21.2.117:0/3700416833 conn(0x7f58dc0ce450 msgr2=0x7f58dc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 -- 172.21.2.117:0/3700416833 shutdown_connections 2021-02-10T11:39:28.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.971 7f58e3088700 1 -- 172.21.2.117:0/3700416833 wait complete. 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 Processor -- start 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 -- start start 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 0x7f58dc096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58dc08a180 0x7f58dc08a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc108020 con 0x7f58dc1a7240 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1a7c60 con 0x7f58dc107c20 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e3088700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1a7e80 con 0x7f58dc08a180 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e16a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e16a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47084/0 (socket says 172.21.2.117:47084) 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e16a5700 1 -- 172.21.2.117:0/3814822071 learned_addr learned my addr 172.21.2.117:0/3814822071 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:28.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58dbfff700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58dc08a180 0x7f58dc08a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.986 7f58e0ea4700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 0x7f58dc096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 -- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58dc08a180 msgr2=0x7f58dc08a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58dc08a180 0x7f58dc08a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 -- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 msgr2=0x7f58dc096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 0x7f58dc096fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 -- 172.21.2.117:0/3814822071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc08b250 con 0x7f58dc1a7240 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e0ea4700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 0x7f58dc096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e16a5700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f58d4009940 tx=0x7f58d400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58d4000f90 con 0x7f58dc1a7240 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d400c2f0 con 0x7f58dc1a7240 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e3088700 1 -- 172.21.2.117:0/3814822071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc08b250 con 0x7f58dc1a7240 2021-02-10T11:39:28.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58e3088700 1 -- 172.21.2.117:0/3814822071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc08b7e0 con 0x7f58dc1a7240 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.987 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58d400c810 con 0x7f58dc1a7240 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.988 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58d4001220 con 0x7f58dc1a7240 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.988 7f58d97fa700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 0x7f58c001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.988 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f58d4032c70 con 0x7f58dc1a7240 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.988 7f58e0ea4700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 0x7f58c001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:28.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:28.988 7f58e0ea4700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 0x7f58c001abd0 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.018 7f58e3088700 1 -- 172.21.2.117:0/3814822071 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc077a70 con 0x7f58dc1a7240 2021-02-10T11:39:29.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.021 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58d4000f40 con 0x7f58dc1a7240 2021-02-10T11:39:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.148 7f58e3088700 1 -- 172.21.2.117:0/3814822071 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802525", "target": ["mgr", ""]}) v1 -- 0x7f58dc08b7e0 con 0x7f58c00186d0 2021-02-10T11:39:29.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.159 7f58d97fa700 1 -- 172.21.2.117:0/3814822071 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f58c40092c0 con 0x7f58c00186d0 2021-02-10T11:39:29.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 msgr2=0x7f58c001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 0x7f58c001abd0 crc :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 msgr2=0x7f58dc1a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f58d4009940 tx=0x7f58d400bcf0).stop 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 shutdown_connections 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58c00186d0 0x7f58c001abd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58dc1a7240 0x7f58dc1a7640 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58dc08a180 0x7f58dc08a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 --2- 172.21.2.117:0/3814822071 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58dc107c20 0x7f58dc096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 >> 172.21.2.117:0/3814822071 conn(0x7f58dc0a1880 msgr2=0x7f58dc0b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 shutdown_connections 2021-02-10T11:39:29.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.165 7f58e3088700 1 -- 172.21.2.117:0/3814822071 wait complete. 2021-02-10T11:39:29.155 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802526 2021-02-10T11:39:29.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 Processor -- start 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 -- start start 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 0x7fabfc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc0d1d20 con 0x7fabfc0e4da0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabfc077a70 con 0x7fabfc0e51a0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fac008a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc0777d0 con 0x7fabfc07da20 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fabf9e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fabfc0e4da0 0x7fabfc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47338/0 (socket says 172.21.2.117:47338) 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fabf9e1b700 1 -- 172.21.2.117:0/2517963868 learned_addr learned my addr 172.21.2.117:0/2517963868 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.588 7fabfa61c700 1 --2- 172.21.2.117:0/2517963868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 0x7fabfc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696533844 0 0) 0x7fabec001500 con 0x7fabfc0e4da0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4002fe0 con 0x7fabfc0e4da0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142553911 0 0) 0x7fabf4001080 con 0x7fabfc0e51a0 2021-02-10T11:39:29.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe4003210 con 0x7fabfc0e51a0 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 453065519 0 0) 0x7fabec001460 con 0x7fabfc0e4da0 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabfa61c700 1 -- 172.21.2.117:0/2517963868 >> v1:172.21.2.125:6789/0 conn(0x7fabfc0e51a0 legacy=0x7fabfc0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabfa61c700 1 -- 172.21.2.117:0/2517963868 >> v1:172.21.2.117:6789/0 conn(0x7fabfc0e4da0 legacy=0x7fabfc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabfa61c700 1 -- 172.21.2.117:0/2517963868 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc1b08e0 con 0x7fabfc07da20 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabfa61c700 1 --2- 172.21.2.117:0/2517963868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 0x7fabfc0771b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fabf0006650 tx=0x7fabf0009a00).ready entity=mon.2 client_cookie=22c20230d3341a78 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf00009b0 con 0x7fabfc07da20 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf000c0c0 con 0x7fabfc07da20 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fabf961a700 1 -- 172.21.2.117:0/2517963868 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabf000c5a0 con 0x7fabfc07da20 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fac008a3700 1 -- 172.21.2.117:0/2517963868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 msgr2=0x7fabfc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.589 7fac008a3700 1 --2- 172.21.2.117:0/2517963868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 0x7fabfc0771b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fabf0006650 tx=0x7fabf0009a00).stop 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.590 7fac008a3700 1 -- 172.21.2.117:0/2517963868 shutdown_connections 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.590 7fac008a3700 1 --2- 172.21.2.117:0/2517963868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc07da20 0x7fabfc0771b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.590 7fac008a3700 1 -- 172.21.2.117:0/2517963868 >> 172.21.2.117:0/2517963868 conn(0x7fabfc0ce450 msgr2=0x7fabfc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.590 7fac008a3700 1 -- 172.21.2.117:0/2517963868 shutdown_connections 2021-02-10T11:39:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.590 7fac008a3700 1 -- 172.21.2.117:0/2517963868 wait complete. 2021-02-10T11:39:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 Processor -- start 2021-02-10T11:39:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 -- start start 2021-02-10T11:39:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc0b7fc0 0x7fabfc1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 0x7fabfc104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 0x7fabfc090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc1054f0 con 0x7fabfc090710 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc195a30 con 0x7fabfc1048f0 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fac008a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabfc195fb0 con 0x7fabfc0b7fc0 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fabfae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 0x7fabfc090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fabf9e1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 0x7fabfc104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fabfae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 0x7fabfc090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47094/0 (socket says 172.21.2.117:47094) 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fabfae1d700 1 -- 172.21.2.117:0/905190738 learned_addr learned my addr 172.21.2.117:0/905190738 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.605 7fabfa61c700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc0b7fc0 0x7fabfc1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 -- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc0b7fc0 msgr2=0x7fabfc1042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc0b7fc0 0x7fabfc1042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 -- 172.21.2.117:0/905190738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 msgr2=0x7fabfc090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 0x7fabfc090b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 -- 172.21.2.117:0/905190738 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc0b7b60 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabf9e1b700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 0x7fabfc104cf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fabec007cc0 tx=0x7fabec00ac30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabec004930 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabec004930 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabec001e70 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fac008a3700 1 -- 172.21.2.117:0/905190738 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc0b7b60 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.606 7fac008a3700 1 -- 172.21.2.117:0/905190738 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabfc0b7b60 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.607 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fabec001e70 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.607 7fabeaffd700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 0x7fabe001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.607 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fabec0329a0 con 0x7fabfc1048f0 2021-02-10T11:39:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.607 7fabfa61c700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 0x7fabe001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.607 7fabfa61c700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 0x7fabe001abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:29.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.638 7fac008a3700 1 -- 172.21.2.117:0/905190738 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabfc103280 con 0x7fabfc1048f0 2021-02-10T11:39:29.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.643 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fabec004720 con 0x7fabfc1048f0 2021-02-10T11:39:29.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.770 7fac008a3700 1 -- 172.21.2.117:0/905190738 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802526", "target": ["mgr", ""]}) v1 -- 0x7fabfc103280 con 0x7fabe00186c0 2021-02-10T11:39:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.780 7fabeaffd700 1 -- 172.21.2.117:0/905190738 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fabf00009b0 con 0x7fabe00186c0 2021-02-10T11:39:29.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 -- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 msgr2=0x7fabe001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 0x7fabe001abc0 crc :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 -- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 msgr2=0x7fabfc104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 0x7fabfc104cf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fabec007cc0 tx=0x7fabec00ac30).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 -- 172.21.2.117:0/905190738 shutdown_connections 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fabe00186c0 0x7fabe001abc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabfc090710 0x7fabfc090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabfc1048f0 0x7fabfc104cf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 --2- 172.21.2.117:0/905190738 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabfc0b7fc0 0x7fabfc1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.787 7fac008a3700 1 -- 172.21.2.117:0/905190738 >> 172.21.2.117:0/905190738 conn(0x7fabfc0a1850 msgr2=0x7fabfc0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:29.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.788 7fac008a3700 1 -- 172.21.2.117:0/905190738 shutdown_connections 2021-02-10T11:39:29.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:29.788 7fac008a3700 1 -- 172.21.2.117:0/905190738 wait complete. 2021-02-10T11:39:29.778 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802527 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.208 7f9355490700 1 Processor -- start 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.208 7f9355490700 1 -- start start 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f9355490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 0x7f93500e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f9355490700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 0x7f93501b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f9355490700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350077a70 con 0x7f935007da20 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f9355490700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500e53c0 con 0x7f93501b00f0 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f9355490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9350079400 con 0x7f9350079000 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f934f07d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f935007da20 0x7f93500771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40278/0 (socket says 172.21.2.117:40278) 2021-02-10T11:39:30.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f934e87c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 0x7f93500e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f934f07d700 1 -- 172.21.2.117:0/3188868326 learned_addr learned my addr 172.21.2.117:0/3188868326 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.209 7f934f87e700 1 --2- 172.21.2.117:0/3188868326 >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 0x7f93501b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e87c700 1 -- 172.21.2.117:0/3188868326 >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 msgr2=0x7f93501b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e87c700 1 --2- 172.21.2.117:0/3188868326 >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 0x7f93501b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e87c700 1 -- 172.21.2.117:0/3188868326 >> v1:172.21.2.125:6789/0 conn(0x7f935007da20 legacy=0x7f93500771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e87c700 1 -- 172.21.2.117:0/3188868326 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500d3400 con 0x7f9350079000 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934f87e700 1 --2- 172.21.2.117:0/3188868326 >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 0x7f93501b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e87c700 1 --2- 172.21.2.117:0/3188868326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 0x7f93500e4da0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9338006650 tx=0x7f9338009a80).ready entity=mon.2 client_cookie=7fef919412c0dde7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e07b700 1 -- 172.21.2.117:0/3188868326 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f933800c0c0 con 0x7f9350079000 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e07b700 1 -- 172.21.2.117:0/3188868326 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800c3c0 con 0x7f9350079000 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f934e07b700 1 -- 172.21.2.117:0/3188868326 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f933800c8e0 con 0x7f9350079000 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f9355490700 1 -- 172.21.2.117:0/3188868326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 msgr2=0x7f93500e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.210 7f9355490700 1 --2- 172.21.2.117:0/3188868326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 0x7f93500e4da0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9338006650 tx=0x7f9338009a80).stop 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 -- 172.21.2.117:0/3188868326 shutdown_connections 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 --2- 172.21.2.117:0/3188868326 >> v2:172.21.2.125:3300/0 conn(0x7f93501b00f0 0x7f93501b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 --2- 172.21.2.117:0/3188868326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9350079000 0x7f93500e4da0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 -- 172.21.2.117:0/3188868326 >> 172.21.2.117:0/3188868326 conn(0x7f93500ce450 msgr2=0x7f93500e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 -- 172.21.2.117:0/3188868326 shutdown_connections 2021-02-10T11:39:30.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.211 7f9355490700 1 -- 172.21.2.117:0/3188868326 wait complete. 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.225 7f9355490700 1 Processor -- start 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 -- start start 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 0x7f93500fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93500b7390 0x7f93500b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b8010 con 0x7f93500c2340 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b8550 con 0x7f93500b7390 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f9355490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b8770 con 0x7f93500a41c0 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f934e87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f934e87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47108/0 (socket says 172.21.2.117:47108) 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f934e87c700 1 -- 172.21.2.117:0/748855532 learned_addr learned my addr 172.21.2.117:0/748855532 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f934f87e700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93500b7390 0x7f93500b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.226 7f934f07d700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 0x7f93500fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 -- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 msgr2=0x7f93500fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 0x7f93500fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 -- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93500b7390 msgr2=0x7f93500b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93500b7390 0x7f93500b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 -- 172.21.2.117:0/748855532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500b8990 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934f07d700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 0x7f93500fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f934e87c700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f9338006e90 tx=0x7f933800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93380009b0 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800bd80 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f9355490700 1 -- 172.21.2.117:0/748855532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93500b8990 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9338005560 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.227 7f9355490700 1 -- 172.21.2.117:0/748855532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93500b8990 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.228 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9338006030 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.228 7f9347fff700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 0x7f933401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.228 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f93380327b0 con 0x7f93500c2340 2021-02-10T11:39:30.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.228 7f934f07d700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 0x7f933401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.228 7f934f07d700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 0x7f933401ac10 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.259 7f9355490700 1 -- 172.21.2.117:0/748855532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93501b4f00 con 0x7f93500c2340 2021-02-10T11:39:30.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.261 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9338000960 con 0x7f93500c2340 2021-02-10T11:39:30.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.387 7f9355490700 1 -- 172.21.2.117:0/748855532 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802527", "target": ["mgr", ""]}) v1 -- 0x7f93501b4f00 con 0x7f9334018730 2021-02-10T11:39:30.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.398 7f9347fff700 1 -- 172.21.2.117:0/748855532 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9340007d50 con 0x7f9334018730 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 msgr2=0x7f933401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 0x7f933401ac10 crc :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 msgr2=0x7f93500c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f9338006e90 tx=0x7f933800b990).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 shutdown_connections 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9334018730 0x7f933401ac10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93500c2340 0x7f93500c2740 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93500b7390 0x7f93500b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 --2- 172.21.2.117:0/748855532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93500a41c0 0x7f93500fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 >> 172.21.2.117:0/748855532 conn(0x7f935008bbc0 msgr2=0x7f93500c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 shutdown_connections 2021-02-10T11:39:30.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.404 7f9355490700 1 -- 172.21.2.117:0/748855532 wait complete. 2021-02-10T11:39:30.394 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802528 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.828 7f3f1ad96700 1 Processor -- start 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 -- start start 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3f140e4d80 0x7f3f141b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f14077a70 con 0x7f3f140e5180 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f140777d0 con 0x7f3f1407da20 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f1ad96700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1407de20 con 0x7f3f140e4d80 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f18bb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f18bb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47116/0 (socket says 172.21.2.117:47116) 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f18bb2700 1 -- 172.21.2.117:0/99141977 learned_addr learned my addr 172.21.2.117:0/99141977 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.829 7f3f13fff700 1 --2- 172.21.2.117:0/99141977 >> v2:172.21.2.125:3300/0 conn(0x7f3f140e4d80 0x7f3f141b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f13fff700 1 -- 172.21.2.117:0/99141977 >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 msgr2=0x7f3f140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f13fff700 1 --2- 172.21.2.117:0/99141977 >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f13fff700 1 -- 172.21.2.117:0/99141977 >> v1:172.21.2.125:6789/0 conn(0x7f3f140e5180 legacy=0x7f3f14077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f13fff700 1 -- 172.21.2.117:0/99141977 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f141b06f0 con 0x7f3f140e4d80 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f18bb2700 1 --2- 172.21.2.117:0/99141977 >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:30.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f13fff700 1 --2- 172.21.2.117:0/99141977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f140e4d80 0x7f3f141b00d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f3f04006650 tx=0x7f3f04009a80).ready entity=mon.1 client_cookie=f1c8b3f7981f7630 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f137fe700 1 -- 172.21.2.117:0/99141977 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f040009b0 con 0x7f3f140e4d80 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f137fe700 1 -- 172.21.2.117:0/99141977 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04006740 con 0x7f3f140e4d80 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f137fe700 1 -- 172.21.2.117:0/99141977 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f0400c0c0 con 0x7f3f140e4d80 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f1ad96700 1 -- 172.21.2.117:0/99141977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f140e4d80 msgr2=0x7f3f141b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.830 7f3f1ad96700 1 --2- 172.21.2.117:0/99141977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f140e4d80 0x7f3f141b00d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f3f04006650 tx=0x7f3f04009a80).stop 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 -- 172.21.2.117:0/99141977 shutdown_connections 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 --2- 172.21.2.117:0/99141977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f140e4d80 0x7f3f141b00d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 --2- 172.21.2.117:0/99141977 >> v2:172.21.2.117:3300/0 conn(0x7f3f1407da20 0x7f3f140771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 -- 172.21.2.117:0/99141977 >> 172.21.2.117:0/99141977 conn(0x7f3f140ce430 msgr2=0x7f3f140e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 -- 172.21.2.117:0/99141977 shutdown_connections 2021-02-10T11:39:30.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.831 7f3f1ad96700 1 -- 172.21.2.117:0/99141977 wait complete. 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 Processor -- start 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 -- start start 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 0x7f3f140becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 0x7f3f140bf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 0x7f3f1408ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f14091b90 con 0x7f3f1408aa50 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1408a180 con 0x7f3f14091790 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.846 7f3f1ad96700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1408c860 con 0x7f3f140bf310 2021-02-10T11:39:30.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f193b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 0x7f3f1408ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f13fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 0x7f3f140bf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f13fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 0x7f3f140bf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44802/0 (socket says 172.21.2.117:44802) 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f193b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 0x7f3f1408ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47126/0 (socket says 172.21.2.117:47126) 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f13fff700 1 -- 172.21.2.117:0/3690519915 learned_addr learned my addr 172.21.2.117:0/3690519915 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 0x7f3f140becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 -- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 msgr2=0x7f3f140bf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 0x7f3f140bf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 -- 172.21.2.117:0/3690519915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 msgr2=0x7f3f1408ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 0x7f3f1408ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 -- 172.21.2.117:0/3690519915 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1408b470 con 0x7f3f14091790 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f18bb2700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 0x7f3f140becf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3efc006b10 tx=0x7f3efc00b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.847 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc00bcb0 con 0x7f3f14091790 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc006030 con 0x7f3f14091790 2021-02-10T11:39:30.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1408b470 con 0x7f3f14091790 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3efc0054d0 con 0x7f3f14091790 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1408b470 con 0x7f3f14091790 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3efc006210 con 0x7f3f14091790 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f117fa700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 0x7f3ef801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3efc032880 con 0x7f3f14091790 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.848 7f3f13fff700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 0x7f3ef801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:30.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.849 7f3f13fff700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 0x7f3ef801ac10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:30.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.878 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f14077a70 con 0x7f3f14091790 2021-02-10T11:39:30.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:30.882 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3efc00bcb0 con 0x7f3f14091790 2021-02-10T11:39:30.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.012 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802528", "target": ["mgr", ""]}) v1 -- 0x7f3f14077a70 con 0x7f3ef8018730 2021-02-10T11:39:31.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.023 7f3f117fa700 1 -- 172.21.2.117:0/3690519915 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3f040067c0 con 0x7f3ef8018730 2021-02-10T11:39:31.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 msgr2=0x7f3ef801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 0x7f3ef801ac10 crc :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 msgr2=0x7f3f140becf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 0x7f3f140becf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3efc006b10 tx=0x7f3efc00b8c0).stop 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 shutdown_connections 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ef8018730 0x7f3ef801ac10 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1408aa50 0x7f3f1408ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f140bf310 0x7f3f140bf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 --2- 172.21.2.117:0/3690519915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f14091790 0x7f3f140becf0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 >> 172.21.2.117:0/3690519915 conn(0x7f3f140a4170 msgr2=0x7f3f1409be70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:31.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 shutdown_connections 2021-02-10T11:39:31.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.030 7f3f1ad96700 1 -- 172.21.2.117:0/3690519915 wait complete. 2021-02-10T11:39:31.020 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802529 2021-02-10T11:39:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.452 7fd9772d5700 1 Processor -- start 2021-02-10T11:39:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.452 7fd9772d5700 1 -- start start 2021-02-10T11:39:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9772d5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd97007da20 0x7fd9700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9772d5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9700e4da0 0x7fd9701b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9772d5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd970077a70 con 0x7fd9700e51a0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9772d5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700777d0 con 0x7fd9700e4da0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9772d5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97007de20 con 0x7fd97007da20 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9748f0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9700e4da0 0x7fd9701b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9748f0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9700e4da0 0x7fd9701b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47130/0 (socket says 172.21.2.117:47130) 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9748f0700 1 -- 172.21.2.117:0/1544558149 learned_addr learned my addr 172.21.2.117:0/1544558149 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.453 7fd9750f1700 1 --2- 172.21.2.117:0/1544558149 >> v2:172.21.2.125:3300/0 conn(0x7fd97007da20 0x7fd9700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9748f0700 1 -- 172.21.2.117:0/1544558149 >> v2:172.21.2.125:3300/0 conn(0x7fd97007da20 msgr2=0x7fd9700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9748f0700 1 --2- 172.21.2.117:0/1544558149 >> v2:172.21.2.125:3300/0 conn(0x7fd97007da20 0x7fd9700771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9748f0700 1 -- 172.21.2.117:0/1544558149 >> v1:172.21.2.125:6789/0 conn(0x7fd9700e51a0 legacy=0x7fd970077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9748f0700 1 -- 172.21.2.117:0/1544558149 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9701b0710 con 0x7fd9700e4da0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9748f0700 1 --2- 172.21.2.117:0/1544558149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9700e4da0 0x7fd9701b00f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7fd96c006700 tx=0x7fd96c009b00).ready entity=mon.0 client_cookie=eba440caaaa61de1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd963fff700 1 -- 172.21.2.117:0/1544558149 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c00c0c0 con 0x7fd9700e4da0 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd963fff700 1 -- 172.21.2.117:0/1544558149 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00c400 con 0x7fd9700e4da0 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd963fff700 1 -- 172.21.2.117:0/1544558149 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c00c860 con 0x7fd9700e4da0 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9772d5700 1 -- 172.21.2.117:0/1544558149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9700e4da0 msgr2=0x7fd9701b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.454 7fd9772d5700 1 --2- 172.21.2.117:0/1544558149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9700e4da0 0x7fd9701b00f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7fd96c006700 tx=0x7fd96c009b00).stop 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 -- 172.21.2.117:0/1544558149 shutdown_connections 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 --2- 172.21.2.117:0/1544558149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9700e4da0 0x7fd9701b00f0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 --2- 172.21.2.117:0/1544558149 >> v2:172.21.2.125:3300/0 conn(0x7fd97007da20 0x7fd9700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 -- 172.21.2.117:0/1544558149 >> 172.21.2.117:0/1544558149 conn(0x7fd9700ce450 msgr2=0x7fd9700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 -- 172.21.2.117:0/1544558149 shutdown_connections 2021-02-10T11:39:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.455 7fd9772d5700 1 -- 172.21.2.117:0/1544558149 wait complete. 2021-02-10T11:39:31.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 Processor -- start 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 -- start start 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd97008eb10 0x7fd9700c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9700afac0 0x7fd9700afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700af540 con 0x7fd97008dfe0 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700a0c80 con 0x7fd97008eb10 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9772d5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700b04e0 con 0x7fd9700afac0 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9748f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.470 7fd9758f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9700afac0 0x7fd9700afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47138/0 (socket says 172.21.2.117:47138) 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 -- 172.21.2.117:0/4012745471 learned_addr learned my addr 172.21.2.117:0/4012745471 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:31.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9750f1700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd97008eb10 0x7fd9700c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 -- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9700afac0 msgr2=0x7fd9700afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9700afac0 0x7fd9700afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 -- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd97008eb10 msgr2=0x7fd9700c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd97008eb10 0x7fd9700c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 -- 172.21.2.117:0/4012745471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700b0700 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9748f0700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7fd96c00b660 tx=0x7fd96c00ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c009440 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c000c90 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd96c002030 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9700b0700 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.471 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9700b0c90 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.472 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd96c0093e0 con 0x7fd97008dfe0 2021-02-10T11:39:31.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.472 7fd961ffb700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 0x7fd95001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:31.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.472 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd96c032540 con 0x7fd97008dfe0 2021-02-10T11:39:31.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.472 7fd9750f1700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 0x7fd95001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:31.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.473 7fd9750f1700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 0x7fd95001abc0 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:31.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.502 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970103280 con 0x7fd97008dfe0 2021-02-10T11:39:31.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.505 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd96c030d20 con 0x7fd97008dfe0 2021-02-10T11:39:31.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.631 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802529", "target": ["mgr", ""]}) v1 -- 0x7fd9700b0c90 con 0x7fd9500186c0 2021-02-10T11:39:31.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.642 7fd961ffb700 1 -- 172.21.2.117:0/4012745471 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd964006a70 con 0x7fd9500186c0 2021-02-10T11:39:31.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 msgr2=0x7fd95001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 0x7fd95001abc0 crc :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 msgr2=0x7fd97008e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:31.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7fd96c00b660 tx=0x7fd96c00ba00).stop 2021-02-10T11:39:31.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 shutdown_connections 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9500186c0 0x7fd95001abc0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd97008dfe0 0x7fd97008e3e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9700afac0 0x7fd9700afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 --2- 172.21.2.117:0/4012745471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd97008eb10 0x7fd9700c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 >> 172.21.2.117:0/4012745471 conn(0x7fd9700a1850 msgr2=0x7fd9700bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.647 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 shutdown_connections 2021-02-10T11:39:31.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:31.648 7fd9772d5700 1 -- 172.21.2.117:0/4012745471 wait complete. 2021-02-10T11:39:31.638 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802530 2021-02-10T11:39:32.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.069 7f1683046700 1 Processor -- start 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.069 7f1683046700 1 -- start start 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1683046700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 0x7f167c0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1683046700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c077a70 con 0x7f167c079000 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1683046700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c079400 con 0x7f167c07da20 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1683046700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c07de20 con 0x7f167c0d1530 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f167bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f167c079000 0x7f167c0e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47394/0 (socket says 172.21.2.117:47394) 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f167bfff700 1 -- 172.21.2.117:0/2651051630 learned_addr learned my addr 172.21.2.117:0/2651051630 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1681663700 1 --2- 172.21.2.117:0/2651051630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 0x7f167c0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756071419 0 0) 0x7f166c001500 con 0x7f167c079000 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1668002fe0 con 0x7f167c079000 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961103867 0 0) 0x7f1664000c60 con 0x7f167c07da20 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1681663700 1 -- 172.21.2.117:0/2651051630 >> v1:172.21.2.125:6789/0 conn(0x7f167c07da20 legacy=0x7f167c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1681663700 1 -- 172.21.2.117:0/2651051630 >> v1:172.21.2.117:6789/0 conn(0x7f167c079000 legacy=0x7f167c0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.070 7f1681663700 1 -- 172.21.2.117:0/2651051630 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c1b00f0 con 0x7f167c0d1530 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1681663700 1 --2- 172.21.2.117:0/2651051630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 0x7f167c0d1930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1674006ad0 tx=0x7f1674009ad0).ready entity=mon.2 client_cookie=2e19671653190297 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16740022e0 con 0x7f167c0d1530 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674006410 con 0x7f167c0d1530 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f167b7fe700 1 -- 172.21.2.117:0/2651051630 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1674002620 con 0x7f167c0d1530 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 -- 172.21.2.117:0/2651051630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 msgr2=0x7f167c0d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 --2- 172.21.2.117:0/2651051630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 0x7f167c0d1930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1674006ad0 tx=0x7f1674009ad0).stop 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 -- 172.21.2.117:0/2651051630 shutdown_connections 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 --2- 172.21.2.117:0/2651051630 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c0d1530 0x7f167c0d1930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 -- 172.21.2.117:0/2651051630 >> 172.21.2.117:0/2651051630 conn(0x7f167c0ce450 msgr2=0x7f167c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.071 7f1683046700 1 -- 172.21.2.117:0/2651051630 shutdown_connections 2021-02-10T11:39:32.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.072 7f1683046700 1 -- 172.21.2.117:0/2651051630 wait complete. 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.086 7f1683046700 1 Processor -- start 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.086 7f1683046700 1 -- start start 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c09bbe0 0x7f167c092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f167c093370 0x7f167c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0927d0 con 0x7f167c09bfe0 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0b08d0 con 0x7f167c093370 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1683046700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c190400 con 0x7f167c09bbe0 2021-02-10T11:39:32.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47152/0 (socket says 172.21.2.117:47152) 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 -- 172.21.2.117:0/1091321314 learned_addr learned my addr 172.21.2.117:0/1091321314 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1680e62700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c09bbe0 0x7f167c092d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f1681663700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f167c093370 0x7f167c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 -- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c09bbe0 msgr2=0x7f167c092d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c09bbe0 0x7f167c092d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 -- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f167c093370 msgr2=0x7f167c0b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f167c093370 0x7f167c0b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.087 7f167bfff700 1 -- 172.21.2.117:0/1091321314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c1908a0 con 0x7f167c09bfe0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f167bfff700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f166c009980 tx=0x7f166c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f166c006030 con 0x7f167c09bfe0 2021-02-10T11:39:32.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c006270 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f166c004720 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f1683046700 1 -- 172.21.2.117:0/1091321314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c1908a0 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f1683046700 1 -- 172.21.2.117:0/1091321314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c1abb00 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.088 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f166c004720 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.089 7f16797fa700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 0x7f166001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.089 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f166c032a10 con 0x7f167c09bfe0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.089 7f1680e62700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 0x7f166001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.089 7f1680e62700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 0x7f166001abc0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.119 7f1683046700 1 -- 172.21.2.117:0/1091321314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c103370 con 0x7f167c09bfe0 2021-02-10T11:39:32.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.122 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f166c006450 con 0x7f167c09bfe0 2021-02-10T11:39:32.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.249 7f1683046700 1 -- 172.21.2.117:0/1091321314 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802530", "target": ["mgr", ""]}) v1 -- 0x7f167c1abb00 con 0x7f16600186c0 2021-02-10T11:39:32.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.259 7f16797fa700 1 -- 172.21.2.117:0/1091321314 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1664006550 con 0x7f16600186c0 2021-02-10T11:39:32.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 msgr2=0x7f166001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 0x7f166001abc0 crc :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 msgr2=0x7f167c0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f166c009980 tx=0x7f166c00bcd0).stop 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 shutdown_connections 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16600186c0 0x7f166001abc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f167c09bfe0 0x7f167c0afc90 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f167c093370 0x7f167c0b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 --2- 172.21.2.117:0/1091321314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f167c09bbe0 0x7f167c092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 >> 172.21.2.117:0/1091321314 conn(0x7f167c0a1420 msgr2=0x7f167c09efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 shutdown_connections 2021-02-10T11:39:32.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.266 7f1683046700 1 -- 172.21.2.117:0/1091321314 wait complete. 2021-02-10T11:39:32.257 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802531 2021-02-10T11:39:32.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.688 7f8c535e8700 1 Processor -- start 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.688 7f8c535e8700 1 -- start start 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c535e8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c4c07da20 0x7f8c4c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c535e8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8c4c0e51a0 0x7f8c4c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c535e8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c077a70 con 0x7f8c4c0e4da0 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c535e8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0e5c20 con 0x7f8c4c07da20 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c535e8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0777d0 con 0x7f8c4c0e51a0 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c51404700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c4c07da20 0x7f8c4c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c50c03700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f8c4c0e4da0 0x7f8c4c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40334/0 (socket says 172.21.2.117:40334) 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c50c03700 1 -- 172.21.2.117:0/1955648616 learned_addr learned my addr 172.21.2.117:0/1955648616 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.689 7f8c51c05700 1 --2- 172.21.2.117:0/1955648616 >> v2:172.21.2.125:3300/0 conn(0x7f8c4c0e51a0 0x7f8c4c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c51404700 1 -- 172.21.2.117:0/1955648616 >> v2:172.21.2.125:3300/0 conn(0x7f8c4c0e51a0 msgr2=0x7f8c4c079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:32.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c51404700 1 --2- 172.21.2.117:0/1955648616 >> v2:172.21.2.125:3300/0 conn(0x7f8c4c0e51a0 0x7f8c4c079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c51404700 1 -- 172.21.2.117:0/1955648616 >> v1:172.21.2.125:6789/0 conn(0x7f8c4c0e4da0 legacy=0x7f8c4c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c51404700 1 -- 172.21.2.117:0/1955648616 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c1b08e0 con 0x7f8c4c07da20 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c51404700 1 --2- 172.21.2.117:0/1955648616 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c07da20 0x7f8c4c0771b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f8c48006650 tx=0x7f8c48009a80).ready entity=mon.0 client_cookie=917d2f051d4145a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c3ffff700 1 -- 172.21.2.117:0/1955648616 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c480009b0 con 0x7f8c4c07da20 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c3ffff700 1 -- 172.21.2.117:0/1955648616 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800c0c0 con 0x7f8c4c07da20 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c3ffff700 1 -- 172.21.2.117:0/1955648616 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4800c510 con 0x7f8c4c07da20 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c535e8700 1 -- 172.21.2.117:0/1955648616 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c07da20 msgr2=0x7f8c4c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.690 7f8c535e8700 1 --2- 172.21.2.117:0/1955648616 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c07da20 0x7f8c4c0771b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f8c48006650 tx=0x7f8c48009a80).stop 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 -- 172.21.2.117:0/1955648616 shutdown_connections 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 --2- 172.21.2.117:0/1955648616 >> v2:172.21.2.125:3300/0 conn(0x7f8c4c0e51a0 0x7f8c4c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 --2- 172.21.2.117:0/1955648616 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c07da20 0x7f8c4c0771b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 -- 172.21.2.117:0/1955648616 >> 172.21.2.117:0/1955648616 conn(0x7f8c4c0ce450 msgr2=0x7f8c4c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 -- 172.21.2.117:0/1955648616 shutdown_connections 2021-02-10T11:39:32.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.691 7f8c535e8700 1 -- 172.21.2.117:0/1955648616 wait complete. 2021-02-10T11:39:32.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.705 7f8c535e8700 1 Processor -- start 2021-02-10T11:39:32.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.705 7f8c535e8700 1 -- start start 2021-02-10T11:39:32.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c4c1b5130 0x7f8c4c1b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 0x7f8c4c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0b8da0 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0a4140 con 0x7f8c4c1b5f50 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0a4320 con 0x7f8c4c1b5130 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47168/0 (socket says 172.21.2.117:47168) 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 -- 172.21.2.117:0/31087636 learned_addr learned my addr 172.21.2.117:0/31087636 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c51404700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c4c1b5130 0x7f8c4c1b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 -- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c4c1b5130 msgr2=0x7f8c4c1b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c4c1b5130 0x7f8c4c1b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 -- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 msgr2=0x7f8c4c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c51c05700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 0x7f8c4c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 0x7f8c4c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 -- 172.21.2.117:0/31087636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0b62b0 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c51c05700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 0x7f8c4c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c50c03700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8c4000a4c0 tx=0x7f8c4000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c4000ac50 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c40004140 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c400045a0 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.706 7f8c535e8700 1 -- 172.21.2.117:0/31087636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c0b62b0 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.707 7f8c535e8700 1 -- 172.21.2.117:0/31087636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c0b6840 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.707 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c40009030 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.707 7f8c3dffb700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 0x7f8c2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.707 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8c400326c0 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.707 7f8c51404700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 0x7f8c2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.708 7f8c51404700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 0x7f8c2c01abc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:32.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.739 7f8c535e8700 1 -- 172.21.2.117:0/31087636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c1b4f00 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.742 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c40030d20 con 0x7f8c4c1b5b50 2021-02-10T11:39:32.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.873 7f8c535e8700 1 -- 172.21.2.117:0/31087636 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802531", "target": ["mgr", ""]}) v1 -- 0x7f8c4c0b6840 con 0x7f8c2c0186c0 2021-02-10T11:39:32.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.884 7f8c3dffb700 1 -- 172.21.2.117:0/31087636 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f8c48000c90 con 0x7f8c2c0186c0 2021-02-10T11:39:32.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 -- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 msgr2=0x7f8c2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 0x7f8c2c01abc0 crc :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 -- 172.21.2.117:0/31087636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 msgr2=0x7f8c4c0b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f8c4000a4c0 tx=0x7f8c4000a860).stop 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 -- 172.21.2.117:0/31087636 shutdown_connections 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c2c0186c0 0x7f8c2c01abc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c4c1b5b50 0x7f8c4c0b5670 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c4c1b5f50 0x7f8c4c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 --2- 172.21.2.117:0/31087636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c4c1b5130 0x7f8c4c1b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.890 7f8c535e8700 1 -- 172.21.2.117:0/31087636 >> 172.21.2.117:0/31087636 conn(0x7f8c4c08bbc0 msgr2=0x7f8c4c0c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.891 7f8c535e8700 1 -- 172.21.2.117:0/31087636 shutdown_connections 2021-02-10T11:39:32.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:32.891 7f8c535e8700 1 -- 172.21.2.117:0/31087636 wait complete. 2021-02-10T11:39:32.880 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802532 2021-02-10T11:39:33.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 Processor -- start 2021-02-10T11:39:33.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 -- start start 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e3c07da20 0x7f5e3c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e3c0e51a0 0x7f5e3c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c077a70 con 0x7f5e3c0e4da0 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.312 7f5e41af1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c0e5c20 con 0x7f5e3c07da20 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e41af1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c0777d0 con 0x7f5e3c0e51a0 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e3c07da20 0x7f5e3c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3affd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5e3c0e4da0 0x7f5e3c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47424/0 (socket says 172.21.2.117:47424) 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3affd700 1 -- 172.21.2.117:0/3086219650 learned_addr learned my addr 172.21.2.117:0/3086219650 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3bfff700 1 --2- 172.21.2.117:0/3086219650 >> v2:172.21.2.125:3300/0 conn(0x7f5e3c0e51a0 0x7f5e3c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 -- 172.21.2.117:0/3086219650 >> v2:172.21.2.125:3300/0 conn(0x7f5e3c0e51a0 msgr2=0x7f5e3c079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 --2- 172.21.2.117:0/3086219650 >> v2:172.21.2.125:3300/0 conn(0x7f5e3c0e51a0 0x7f5e3c079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 -- 172.21.2.117:0/3086219650 >> v1:172.21.2.117:6789/0 conn(0x7f5e3c0e4da0 legacy=0x7f5e3c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 -- 172.21.2.117:0/3086219650 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c1b08e0 con 0x7f5e3c07da20 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.313 7f5e3b7fe700 1 --2- 172.21.2.117:0/3086219650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c07da20 0x7f5e3c0771b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f5e340069d0 tx=0x7f5e34009e40).ready entity=mon.0 client_cookie=7d9ebea68b745a60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.314 7f5e3a7fc700 1 -- 172.21.2.117:0/3086219650 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3400b510 con 0x7f5e3c07da20 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.314 7f5e3a7fc700 1 -- 172.21.2.117:0/3086219650 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3400b790 con 0x7f5e3c07da20 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.314 7f5e3a7fc700 1 -- 172.21.2.117:0/3086219650 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3400bcb0 con 0x7f5e3c07da20 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.314 7f5e41af1700 1 -- 172.21.2.117:0/3086219650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c07da20 msgr2=0x7f5e3c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.314 7f5e41af1700 1 --2- 172.21.2.117:0/3086219650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c07da20 0x7f5e3c0771b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0x7f5e340069d0 tx=0x7f5e34009e40).stop 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 -- 172.21.2.117:0/3086219650 shutdown_connections 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 --2- 172.21.2.117:0/3086219650 >> v2:172.21.2.125:3300/0 conn(0x7f5e3c0e51a0 0x7f5e3c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 --2- 172.21.2.117:0/3086219650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c07da20 0x7f5e3c0771b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 -- 172.21.2.117:0/3086219650 >> 172.21.2.117:0/3086219650 conn(0x7f5e3c0ce450 msgr2=0x7f5e3c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 -- 172.21.2.117:0/3086219650 shutdown_connections 2021-02-10T11:39:33.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.315 7f5e41af1700 1 -- 172.21.2.117:0/3086219650 wait complete. 2021-02-10T11:39:33.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.329 7f5e41af1700 1 Processor -- start 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.329 7f5e41af1700 1 -- start start 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 0x7f5e3c0a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 0x7f5e3c0a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0c1360 con 0x7f5e3c0a1090 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0a2b00 con 0x7f5e3c19db70 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e41af1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0a2d20 con 0x7f5e3c0a20e0 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47178/0 (socket says 172.21.2.117:47178) 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3b7fe700 1 -- 172.21.2.117:0/1436264654 learned_addr learned my addr 172.21.2.117:0/1436264654 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 0x7f5e3c0a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3bfff700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 0x7f5e3c0a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 -- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 msgr2=0x7f5e3c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 0x7f5e3c0a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 -- 172.21.2.117:0/1436264654 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 msgr2=0x7f5e3c19d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 -- 172.21.2.117:0/1436264654 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c106040 con 0x7f5e3c19db70 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3bfff700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 0x7f5e3c0a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3b7fe700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.330 7f5e3affd700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 0x7f5e3c0a1ac0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5e3000a4c0 tx=0x7f5e3000a860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e30000960 con 0x7f5e3c19db70 2021-02-10T11:39:33.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3000ac50 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c106040 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e300045d0 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c106040 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e30000da0 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 0x7f5e1c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.331 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e30032830 con 0x7f5e3c19db70 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.332 7f5e3b7fe700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 0x7f5e1c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.332 7f5e3b7fe700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 0x7f5e1c01abd0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.362 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c1b4f00 con 0x7f5e3c19db70 2021-02-10T11:39:33.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.367 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e3001f420 con 0x7f5e3c19db70 2021-02-10T11:39:33.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.495 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802532", "target": ["mgr", ""]}) v1 -- 0x7f5e3c1b4f00 con 0x7f5e1c0186d0 2021-02-10T11:39:33.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.506 7f5e23fff700 1 -- 172.21.2.117:0/1436264654 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5e34000d30 con 0x7f5e1c0186d0 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 msgr2=0x7f5e1c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 0x7f5e1c01abd0 crc :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 msgr2=0x7f5e3c0a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 0x7f5e3c0a1ac0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5e3000a4c0 tx=0x7f5e3000a860).stop 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 shutdown_connections 2021-02-10T11:39:33.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e3c0a20e0 0x7f5e3c0a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e3c19db70 0x7f5e3c0a1ac0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e1c0186d0 0x7f5e1c01abd0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 --2- 172.21.2.117:0/1436264654 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e3c0a1090 0x7f5e3c19d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.511 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 >> 172.21.2.117:0/1436264654 conn(0x7f5e3c0ab050 msgr2=0x7f5e3c107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.512 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 shutdown_connections 2021-02-10T11:39:33.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.512 7f5e41af1700 1 -- 172.21.2.117:0/1436264654 wait complete. 2021-02-10T11:39:33.502 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000802533 2021-02-10T11:39:33.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.935 7f4a48650700 1 Processor -- start 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.935 7f4a48650700 1 -- start start 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a48650700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a4007da20 0x7f4a400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a48650700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4a400e51a0 0x7f4a40079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a48650700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a40077a70 con 0x7f4a400e4da0 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a48650700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a400e5c20 con 0x7f4a4007da20 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a48650700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a400777d0 con 0x7f4a400e51a0 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a45c6b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4a400e4da0 0x7f4a401b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47436/0 (socket says 172.21.2.117:47436) 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a45c6b700 1 -- 172.21.2.117:0/1470258084 learned_addr learned my addr 172.21.2.117:0/1470258084 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4646c700 1 --2- 172.21.2.117:0/1470258084 >> v2:172.21.2.117:3300/0 conn(0x7f4a4007da20 0x7f4a400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a46c6d700 1 --2- 172.21.2.117:0/1470258084 >> v2:172.21.2.125:3300/0 conn(0x7f4a400e51a0 0x7f4a40079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4546a700 1 -- 172.21.2.117:0/1470258084 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303868779 0 0) 0x7f4a3c001180 con 0x7f4a400e4da0 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4546a700 1 -- 172.21.2.117:0/1470258084 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c002fe0 con 0x7f4a400e4da0 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4646c700 1 -- 172.21.2.117:0/1470258084 >> v2:172.21.2.125:3300/0 conn(0x7f4a400e51a0 msgr2=0x7f4a40079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4646c700 1 --2- 172.21.2.117:0/1470258084 >> v2:172.21.2.125:3300/0 conn(0x7f4a400e51a0 0x7f4a40079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4646c700 1 -- 172.21.2.117:0/1470258084 >> v1:172.21.2.117:6789/0 conn(0x7f4a400e4da0 legacy=0x7f4a401b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.936 7f4a4646c700 1 -- 172.21.2.117:0/1470258084 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a401b08e0 con 0x7f4a4007da20 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a4646c700 1 --2- 172.21.2.117:0/1470258084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a4007da20 0x7f4a400771b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f4a380069d0 tx=0x7f4a38009e40).ready entity=mon.0 client_cookie=eb56a34a5c9e1692 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a4546a700 1 -- 172.21.2.117:0/1470258084 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a3800b510 con 0x7f4a4007da20 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a4546a700 1 -- 172.21.2.117:0/1470258084 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3800b790 con 0x7f4a4007da20 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a4546a700 1 -- 172.21.2.117:0/1470258084 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a3800bcb0 con 0x7f4a4007da20 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 -- 172.21.2.117:0/1470258084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a4007da20 msgr2=0x7f4a400771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 --2- 172.21.2.117:0/1470258084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a4007da20 0x7f4a400771b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7f4a380069d0 tx=0x7f4a38009e40).stop 2021-02-10T11:39:33.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 -- 172.21.2.117:0/1470258084 shutdown_connections 2021-02-10T11:39:33.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 --2- 172.21.2.117:0/1470258084 >> v2:172.21.2.125:3300/0 conn(0x7f4a400e51a0 0x7f4a40079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 --2- 172.21.2.117:0/1470258084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a4007da20 0x7f4a400771b0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 -- 172.21.2.117:0/1470258084 >> 172.21.2.117:0/1470258084 conn(0x7f4a400ce450 msgr2=0x7f4a400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:33.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 -- 172.21.2.117:0/1470258084 shutdown_connections 2021-02-10T11:39:33.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.937 7f4a48650700 1 -- 172.21.2.117:0/1470258084 wait complete. 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.952 7f4a48650700 1 Processor -- start 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.952 7f4a48650700 1 -- start start 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a400b7fc0 0x7f4a401042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a401048f0 0x7f4a40104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401054f0 con 0x7f4a40090710 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a40195a30 con 0x7f4a401048f0 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a40195fb0 con 0x7f4a400b7fc0 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47192/0 (socket says 172.21.2.117:47192) 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 -- 172.21.2.117:0/3940953383 learned_addr learned my addr 172.21.2.117:0/3940953383 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a45c6b700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a401048f0 0x7f4a40104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a4646c700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a400b7fc0 0x7f4a401042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 -- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a400b7fc0 msgr2=0x7f4a401042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a400b7fc0 0x7f4a401042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 -- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a401048f0 msgr2=0x7f4a40104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:33.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a401048f0 0x7f4a40104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 -- 172.21.2.117:0/3940953383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a400b7b60 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a46c6d700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f4a3000b5e0 tx=0x7f4a3000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a30000c10 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3000bd70 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a300024d0 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.953 7f4a48650700 1 -- 172.21.2.117:0/3940953383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a400b7b60 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.954 7f4a48650700 1 -- 172.21.2.117:0/3940953383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a40091420 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.954 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a30005330 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.954 7f4a36ffd700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 0x7f4a2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.954 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a30032730 con 0x7f4a40090710 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.954 7f4a4646c700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 0x7f4a2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:33.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.955 7f4a4646c700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 0x7f4a2801abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.985 7f4a48650700 1 -- 172.21.2.117:0/3940953383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a40103280 con 0x7f4a40090710 2021-02-10T11:39:33.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:33.988 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a30000ef0 con 0x7f4a40090710 2021-02-10T11:39:34.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.115 7f4a48650700 1 -- 172.21.2.117:0/3940953383 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000802533", "target": ["mgr", ""]}) v1 -- 0x7f4a40091420 con 0x7f4a280186c0 2021-02-10T11:39:34.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.125 7f4a36ffd700 1 -- 172.21.2.117:0/3940953383 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4a38000d30 con 0x7f4a280186c0 2021-02-10T11:39:34.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.132 7f4a48650700 1 -- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 msgr2=0x7f4a2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.132 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 0x7f4a2801abc0 crc :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.132 7f4a48650700 1 -- 172.21.2.117:0/3940953383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 msgr2=0x7f4a40090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.132 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7f4a3000b5e0 tx=0x7f4a3000b980).stop 2021-02-10T11:39:34.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 -- 172.21.2.117:0/3940953383 shutdown_connections 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a280186c0 0x7f4a2801abc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a40090710 0x7f4a40090b10 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a401048f0 0x7f4a40104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 --2- 172.21.2.117:0/3940953383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a400b7fc0 0x7f4a401042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 -- 172.21.2.117:0/3940953383 >> 172.21.2.117:0/3940953383 conn(0x7f4a400a1850 msgr2=0x7f4a400c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 -- 172.21.2.117:0/3940953383 shutdown_connections 2021-02-10T11:39:34.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.133 7f4a48650700 1 -- 172.21.2.117:0/3940953383 wait complete. 2021-02-10T11:39:34.123 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:34.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 Processor -- start 2021-02-10T11:39:34.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 -- start start 2021-02-10T11:39:34.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbc300e5190 0x7fbc300d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc30077a70 con 0x7fbc300e4d90 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc300e5c10 con 0x7fbc300e5190 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.553 7fbc38757700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300777d0 con 0x7fbc3007da20 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44878/0 (socket says 172.21.2.117:44878) 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 -- 172.21.2.117:0/3263881092 learned_addr learned my addr 172.21.2.117:0/3263881092 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36d74700 1 --2- 172.21.2.117:0/3263881092 >> v2:172.21.2.117:3300/0 conn(0x7fbc300e5190 0x7fbc300d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 -- 172.21.2.117:0/3263881092 >> v2:172.21.2.117:3300/0 conn(0x7fbc300e5190 msgr2=0x7fbc300d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 --2- 172.21.2.117:0/3263881092 >> v2:172.21.2.117:3300/0 conn(0x7fbc300e5190 0x7fbc300d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 -- 172.21.2.117:0/3263881092 >> v1:172.21.2.125:6789/0 conn(0x7fbc300e4d90 legacy=0x7fbc301b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 -- 172.21.2.117:0/3263881092 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc301b08d0 con 0x7fbc3007da20 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc36573700 1 --2- 172.21.2.117:0/3263881092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbc28006a80 tx=0x7fbc28009f00).ready entity=mon.2 client_cookie=3f9c126740890cf7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.554 7fbc35571700 1 -- 172.21.2.117:0/3263881092 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc2800b5c0 con 0x7fbc3007da20 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.555 7fbc35571700 1 -- 172.21.2.117:0/3263881092 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2800b8c0 con 0x7fbc3007da20 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.555 7fbc35571700 1 -- 172.21.2.117:0/3263881092 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc2800bda0 con 0x7fbc3007da20 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.555 7fbc38757700 1 -- 172.21.2.117:0/3263881092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 msgr2=0x7fbc300771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.555 7fbc38757700 1 --2- 172.21.2.117:0/3263881092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbc28006a80 tx=0x7fbc28009f00).stop 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 -- 172.21.2.117:0/3263881092 shutdown_connections 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 --2- 172.21.2.117:0/3263881092 >> v2:172.21.2.117:3300/0 conn(0x7fbc300e5190 0x7fbc300d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 --2- 172.21.2.117:0/3263881092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc3007da20 0x7fbc300771b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 -- 172.21.2.117:0/3263881092 >> 172.21.2.117:0/3263881092 conn(0x7fbc300ce440 msgr2=0x7fbc300e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 -- 172.21.2.117:0/3263881092 shutdown_connections 2021-02-10T11:39:34.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.556 7fbc38757700 1 -- 172.21.2.117:0/3263881092 wait complete. 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 Processor -- start 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 -- start start 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc3009a780 0x7fbc301a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc301a07a0 0x7fbc301a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3009afa0 con 0x7fbc301a07a0 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc301a0d60 con 0x7fbc3009a780 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc38757700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc301048d0 con 0x7fbc301a1900 2021-02-10T11:39:34.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.570 7fbc36d74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44884/0 (socket says 172.21.2.117:44884) 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 -- 172.21.2.117:0/3994579272 learned_addr learned my addr 172.21.2.117:0/3994579272 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc35d72700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc301a07a0 0x7fbc301a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36573700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc3009a780 0x7fbc301a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 -- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc3009a780 msgr2=0x7fbc301a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc3009a780 0x7fbc301a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 -- 172.21.2.117:0/3994579272 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc301a07a0 msgr2=0x7fbc301a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc301a07a0 0x7fbc301a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 -- 172.21.2.117:0/3994579272 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc301a2320 con 0x7fbc301a1900 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc36d74700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbc2c00b580 tx=0x7fbc2c00b920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc2c00bd10 con 0x7fbc301a1900 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c0022e0 con 0x7fbc301a1900 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc38757700 1 -- 172.21.2.117:0/3994579272 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc301a2320 con 0x7fbc301a1900 2021-02-10T11:39:34.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc38757700 1 -- 172.21.2.117:0/3994579272 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc301a2320 con 0x7fbc301a1900 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.571 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbc2c002800 con 0x7fbc301a1900 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.572 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbc2c000ab0 con 0x7fbc301a1900 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.572 7fbc26ffd700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 0x7fbc1801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.572 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbc2c032970 con 0x7fbc301a1900 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.572 7fbc36573700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 0x7fbc1801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:34.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.572 7fbc36573700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 0x7fbc1801ac10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:34.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.603 7fbc38757700 1 -- 172.21.2.117:0/3994579272 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc301b4f60 con 0x7fbc301a1900 2021-02-10T11:39:34.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.607 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbc2c01f560 con 0x7fbc301a1900 2021-02-10T11:39:34.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.732 7fbc38757700 1 -- 172.21.2.117:0/3994579272 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc301a2320 con 0x7fbc301a1900 2021-02-10T11:39:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.734 7fbc26ffd700 1 -- 172.21.2.117:0/3994579272 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+14226 (secure 0 0 0) 0x7fbc2c000ab0 con 0x7fbc301a1900 2021-02-10T11:39:34.717 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:34.718 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":31,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:24:02.664299","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:23:59.642031","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data","create_time":"2021-02-10 11:24:00.660978","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:39:34.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.737 7fbc38757700 1 -- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 msgr2=0x7fbc1801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.737 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 0x7fbc1801ac10 crc :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.737 7fbc38757700 1 -- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 msgr2=0x7fbc301a1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:34.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.737 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fbc2c00b580 tx=0x7fbc2c00b920).stop 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 -- 172.21.2.117:0/3994579272 shutdown_connections 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbc18018730 0x7fbc1801ac10 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc301a07a0 0x7fbc301a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc301a1900 0x7fbc301a1d00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 --2- 172.21.2.117:0/3994579272 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc3009a780 0x7fbc301a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 -- 172.21.2.117:0/3994579272 >> 172.21.2.117:0/3994579272 conn(0x7fbc301960b0 msgr2=0x7fbc301ace50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 -- 172.21.2.117:0/3994579272 shutdown_connections 2021-02-10T11:39:34.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:34.738 7fbc38757700 1 -- 172.21.2.117:0/3994579272 wait complete. 2021-02-10T11:39:34.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:23:59.642031', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:24:00.660978', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:39:34.729 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:39:34.903 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:34.907 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:39:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 Processor -- start 2021-02-10T11:39:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 -- start start 2021-02-10T11:39:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c07da20 0x7f9b0c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c077a70 con 0x7f9b0c0e51a0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c0e5c20 con 0x7f9b0c0e4da0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b13e1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0777d0 con 0x7f9b0c07da20 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b1143a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b1143a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47212/0 (socket says 172.21.2.117:47212) 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.160 7f9b1143a700 1 -- 172.21.2.117:0/2853057674 learned_addr learned my addr 172.21.2.117:0/2853057674 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b11c3b700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c07da20 0x7f9b0c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b1143a700 1 -- 172.21.2.117:0/2853057674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c07da20 msgr2=0x7f9b0c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b1143a700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c07da20 0x7f9b0c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b1143a700 1 -- 172.21.2.117:0/2853057674 >> v1:172.21.2.125:6789/0 conn(0x7f9b0c0e51a0 legacy=0x7f9b0c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b1143a700 1 -- 172.21.2.117:0/2853057674 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c07de20 con 0x7f9b0c0e4da0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b1143a700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f9b08006700 tx=0x7f9b08009b80).ready entity=mon.0 client_cookie=663d2dc7dc1e0348 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b10c39700 1 -- 172.21.2.117:0/2853057674 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b0800c0c0 con 0x7f9b0c0e4da0 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b10c39700 1 -- 172.21.2.117:0/2853057674 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0800c440 con 0x7f9b0c0e4da0 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b10c39700 1 -- 172.21.2.117:0/2853057674 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b0800c960 con 0x7f9b0c0e4da0 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b13e1f700 1 -- 172.21.2.117:0/2853057674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0e4da0 msgr2=0x7f9b0c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.161 7f9b13e1f700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f9b08006700 tx=0x7f9b08009b80).stop 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 -- 172.21.2.117:0/2853057674 shutdown_connections 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0e4da0 0x7f9b0c1b00f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 --2- 172.21.2.117:0/2853057674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c07da20 0x7f9b0c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 -- 172.21.2.117:0/2853057674 >> 172.21.2.117:0/2853057674 conn(0x7f9b0c0ce450 msgr2=0x7f9b0c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 -- 172.21.2.117:0/2853057674 shutdown_connections 2021-02-10T11:39:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.162 7f9b13e1f700 1 -- 172.21.2.117:0/2853057674 wait complete. 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 Processor -- start 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 -- start start 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c08eb10 0x7f9b0c0c3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b0c08dfe0 0x7f9b0c08e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0af540 con 0x7f9b0c0afac0 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0a0c80 con 0x7f9b0c08dfe0 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b13e1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0b04e0 con 0x7f9b0c08eb10 2021-02-10T11:39:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b1243c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b1243c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47214/0 (socket says 172.21.2.117:47214) 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b1243c700 1 -- 172.21.2.117:0/2057216046 learned_addr learned my addr 172.21.2.117:0/2057216046 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b11c3b700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c08eb10 0x7f9b0c0c3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.177 7f9b1143a700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b0c08dfe0 0x7f9b0c08e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 -- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c08eb10 msgr2=0x7f9b0c0c3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c08eb10 0x7f9b0c0c3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 -- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b0c08dfe0 msgr2=0x7f9b0c08e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b0c08dfe0 0x7f9b0c08e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 -- 172.21.2.117:0/2057216046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0b0700 con 0x7f9b0c0afac0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b1243c700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f9b04009940 tx=0x7f9b0400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b04000ce0 con 0x7f9b0c0afac0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04000f90 con 0x7f9b0c0afac0 2021-02-10T11:39:35.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b0400c4e0 con 0x7f9b0c0afac0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c0b0700 con 0x7f9b0c0afac0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.178 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c0b0700 con 0x7f9b0c0afac0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.179 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b04001220 con 0x7f9b0c0afac0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.179 7f9afe7fc700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 0x7f9af401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.179 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9b04032b30 con 0x7f9b0c0afac0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.179 7f9b11c3b700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 0x7f9af401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.179 7f9b11c3b700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 0x7f9af401ac10 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.210 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c103280 con 0x7f9b0c0afac0 2021-02-10T11:39:35.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.212 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b0400be30 con 0x7f9b0c0afac0 2021-02-10T11:39:35.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.342 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f9b0c103280 con 0x7f9b0c0afac0 2021-02-10T11:39:35.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.342 7f9afe7fc700 1 -- 172.21.2.117:0/2057216046 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9b0401f8a0 con 0x7f9b0c0afac0 2021-02-10T11:39:35.324 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:39:35.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 msgr2=0x7f9af401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 0x7f9af401ac10 crc :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 msgr2=0x7f9b0c0afec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7f9b04009940 tx=0x7f9b0400bcf0).stop 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 shutdown_connections 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9af4018730 0x7f9af401ac10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b0c0afac0 0x7f9b0c0afec0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b0c08dfe0 0x7f9b0c08e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 --2- 172.21.2.117:0/2057216046 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b0c08eb10 0x7f9b0c0c3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 >> 172.21.2.117:0/2057216046 conn(0x7f9b0c0a1850 msgr2=0x7f9b0c0bcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 shutdown_connections 2021-02-10T11:39:35.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.344 7f9b13e1f700 1 -- 172.21.2.117:0/2057216046 wait complete. 2021-02-10T11:39:35.335 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:39:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 Processor -- start 2021-02-10T11:39:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 -- start start 2021-02-10T11:39:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efe00079000 0x7efe000e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe00077a70 con 0x7efe0007da20 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe000e53c0 con 0x7efe00079000 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.805 7efe0647f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe00079400 con 0x7efe001b00f0 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:44906/0 (socket says 172.21.2.117:44906) 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 -- 172.21.2.117:0/2288783267 learned_addr learned my addr 172.21.2.117:0/2288783267 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efdff7fe700 1 --2- 172.21.2.117:0/2288783267 >> v2:172.21.2.125:3300/0 conn(0x7efe00079000 0x7efe000e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 -- 172.21.2.117:0/2288783267 >> v2:172.21.2.125:3300/0 conn(0x7efe00079000 msgr2=0x7efe000e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 --2- 172.21.2.117:0/2288783267 >> v2:172.21.2.125:3300/0 conn(0x7efe00079000 0x7efe000e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 -- 172.21.2.117:0/2288783267 >> v1:172.21.2.125:6789/0 conn(0x7efe0007da20 legacy=0x7efe000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 -- 172.21.2.117:0/2288783267 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0007de20 con 0x7efe001b00f0 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe04a9c700 1 --2- 172.21.2.117:0/2288783267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efdf8006aa0 tx=0x7efdf8009f20).ready entity=mon.2 client_cookie=ce78b0e2f16aaee5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efdfeffd700 1 -- 172.21.2.117:0/2288783267 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf8000f90 con 0x7efe001b00f0 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efdfeffd700 1 -- 172.21.2.117:0/2288783267 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf8003030 con 0x7efe001b00f0 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efdfeffd700 1 -- 172.21.2.117:0/2288783267 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf800b7b0 con 0x7efe001b00f0 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe0647f700 1 -- 172.21.2.117:0/2288783267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 msgr2=0x7efe001b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.806 7efe0647f700 1 --2- 172.21.2.117:0/2288783267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efdf8006aa0 tx=0x7efdf8009f20).stop 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 -- 172.21.2.117:0/2288783267 shutdown_connections 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 --2- 172.21.2.117:0/2288783267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe001b00f0 0x7efe001b04f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 --2- 172.21.2.117:0/2288783267 >> v2:172.21.2.125:3300/0 conn(0x7efe00079000 0x7efe000e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 -- 172.21.2.117:0/2288783267 >> 172.21.2.117:0/2288783267 conn(0x7efe000ce450 msgr2=0x7efe000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 -- 172.21.2.117:0/2288783267 shutdown_connections 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.807 7efe0647f700 1 -- 172.21.2.117:0/2288783267 wait complete. 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.821 7efe0647f700 1 Processor -- start 2021-02-10T11:39:35.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 -- start start 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe001a0250 0x7efe00106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe00187e20 0x7efe00188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001067f0 con 0x7efe00090420 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe00107350 con 0x7efe001a0250 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe0647f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe001042d0 con 0x7efe00187e20 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe04a9c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe04a9c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47230/0 (socket says 172.21.2.117:47230) 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efe04a9c700 1 -- 172.21.2.117:0/2829655990 learned_addr learned my addr 172.21.2.117:0/2829655990 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efdfffff700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe001a0250 0x7efe00106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.822 7efdff7fe700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe00187e20 0x7efe00188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 -- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe00187e20 msgr2=0x7efe00188220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe00187e20 0x7efe00188220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 -- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe001a0250 msgr2=0x7efe00106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe001a0250 0x7efe00106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 -- 172.21.2.117:0/2829655990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe00090e40 con 0x7efe00090420 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe04a9c700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7efdf8009670 tx=0x7efdf800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf8006030 con 0x7efe00090420 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf8006330 con 0x7efe00090420 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdf80056b0 con 0x7efe00090420 2021-02-10T11:39:35.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe0647f700 1 -- 172.21.2.117:0/2829655990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe00090e40 con 0x7efe00090420 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.823 7efe0647f700 1 -- 172.21.2.117:0/2829655990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe00090e40 con 0x7efe00090420 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.824 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efdf8006030 con 0x7efe00090420 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.824 7efdfcff9700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 0x7efddc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.824 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efdf8032ad0 con 0x7efe00090420 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.824 7efdfffff700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 0x7efddc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.824 7efdfffff700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 0x7efddc01abd0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:35.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.855 7efe0647f700 1 -- 172.21.2.117:0/2829655990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe00103280 con 0x7efe00090420 2021-02-10T11:39:35.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.858 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efdf800be00 con 0x7efe00090420 2021-02-10T11:39:35.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.986 7efe0647f700 1 -- 172.21.2.117:0/2829655990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7efe00103280 con 0x7efe00090420 2021-02-10T11:39:35.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.986 7efdfcff9700 1 -- 172.21.2.117:0/2829655990 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v31) v1 ==== 104+0+10 (secure 0 0 0) 0x7efdf8006030 con 0x7efe00090420 2021-02-10T11:39:35.968 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 msgr2=0x7efddc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 0x7efddc01abd0 crc :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 msgr2=0x7efe00090820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7efdf8009670 tx=0x7efdf800bcc0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 shutdown_connections 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efddc0186d0 0x7efddc01abd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe00090420 0x7efe00090820 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe00187e20 0x7efe00188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 --2- 172.21.2.117:0/2829655990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe001a0250 0x7efe00106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:35.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 >> 172.21.2.117:0/2829655990 conn(0x7efe000a1850 msgr2=0x7efe000c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:35.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 shutdown_connections 2021-02-10T11:39:35.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:35.988 7efe0647f700 1 -- 172.21.2.117:0/2829655990 wait complete. 2021-02-10T11:39:35.978 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 Processor -- start 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 -- start start 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f119007da20 0x7f11900771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11900d1d20 con 0x7f11900e4da0 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1190077a70 con 0x7f11900e51a0 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f11950bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900777d0 con 0x7f119007da20 2021-02-10T11:39:36.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f118e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f11900e4da0 0x7f11901b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47488/0 (socket says 172.21.2.117:47488) 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.409 7f118e59c700 1 -- 172.21.2.117:0/1861298069 learned_addr learned my addr 172.21.2.117:0/1861298069 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194225399 0 0) 0x7f11780010b0 con 0x7f11900e4da0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c002fe0 con 0x7f11900e4da0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118ed9d700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f119007da20 0x7f11900771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2183834999 0 0) 0x7f11780010b0 con 0x7f11900e4da0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f119007da20 msgr2=0x7f11900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f119007da20 0x7f11900771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 >> v1:172.21.2.125:6789/0 conn(0x7f11900e51a0 legacy=0x7f11900d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11901b08e0 con 0x7f11900e4da0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f11780015b0 con 0x7f11900e4da0 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 >> v1:172.21.2.117:6789/0 conn(0x7f11900e4da0 legacy=0x7f11901b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f117c003b10 0x7f117c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f117c0067b0 0x7f117c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f117c0095b0 0x7f117c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c2d0 con 0x7f117c0067b0 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c4f0 con 0x7f117c003b10 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c710 con 0x7f117c0095b0 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118e59c700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f117c0067b0 0x7f117c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.410 7f118ed9d700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f117c003b10 0x7f117c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118f59e700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f117c0095b0 0x7f117c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 -- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f117c0095b0 msgr2=0x7f117c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f117c0095b0 0x7f117c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 -- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f117c003b10 msgr2=0x7f117c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f117c003b10 0x7f117c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 -- 172.21.2.117:0/1861298069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c050980 con 0x7f117c0067b0 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118ed9d700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f117c003b10 0x7f117c005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118e59c700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f117c0067b0 0x7f117c008cb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f1178007990 tx=0x7f1178007d30).ready entity=mon.0 client_cookie=746426b064c14027 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11780009b0 con 0x7f117c0067b0 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117800a310 con 0x7f117c0067b0 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118dd9b700 1 -- 172.21.2.117:0/1861298069 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f117800a830 con 0x7f117c0067b0 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f11950bf700 1 -- 172.21.2.117:0/1861298069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f117c0067b0 msgr2=0x7f117c008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f11950bf700 1 --2- 172.21.2.117:0/1861298069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f117c0067b0 0x7f117c008cb0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 rx=0x7f1178007990 tx=0x7f1178007d30).stop 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.411 7f118f59e700 1 -- 172.21.2.117:0/1861298069 reap_dead start 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.412 7f11950bf700 1 -- 172.21.2.117:0/1861298069 shutdown_connections 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.412 7f11950bf700 1 -- 172.21.2.117:0/1861298069 >> 172.21.2.117:0/1861298069 conn(0x7f11900ce450 msgr2=0x7f11900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.412 7f11950bf700 1 -- 172.21.2.117:0/1861298069 shutdown_connections 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.412 7f11950bf700 1 -- 172.21.2.117:0/1861298069 wait complete. 2021-02-10T11:39:36.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 Processor -- start 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 -- start start 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11901a46b0 0x7f1190190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 0x7f11900a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11900c3040 con 0x7f11900a50e0 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1190190a20 con 0x7f11900a46c0 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f11950bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11900a54e0 con 0x7f11901a46b0 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f118f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f118f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47252/0 (socket says 172.21.2.117:47252) 2021-02-10T11:39:36.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.427 7f118f59e700 1 -- 172.21.2.117:0/1275833475 learned_addr learned my addr 172.21.2.117:0/1275833475 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:36.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118ed9d700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11901a46b0 0x7f1190190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118e59c700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 0x7f11900a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 -- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11901a46b0 msgr2=0x7f1190190400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11901a46b0 0x7f1190190400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 -- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 msgr2=0x7f11900a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 0x7f11900a4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 -- 172.21.2.117:0/1275833475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11900a5700 con 0x7f11900a50e0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118e59c700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 0x7f11900a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f118f59e700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f1188006900 tx=0x7f118800bca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1188006030 con 0x7f11900a50e0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1188000d90 con 0x7f11900a50e0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11880056a0 con 0x7f11900a50e0 2021-02-10T11:39:36.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f11950bf700 1 -- 172.21.2.117:0/1275833475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11900a5700 con 0x7f11900a50e0 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.428 7f11950bf700 1 -- 172.21.2.117:0/1275833475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11900a6990 con 0x7f11900a50e0 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.429 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1188006330 con 0x7f11900a50e0 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.429 7f11877fe700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 0x7f117401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.429 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1188032a40 con 0x7f11900a50e0 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.429 7f118ed9d700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 0x7f117401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.430 7f118ed9d700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 0x7f117401abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:36.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.459 7f11950bf700 1 -- 172.21.2.117:0/1275833475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1190077a70 con 0x7f11900a50e0 2021-02-10T11:39:36.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.461 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1188006a30 con 0x7f11900a50e0 2021-02-10T11:39:36.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.589 7f11950bf700 1 -- 172.21.2.117:0/1275833475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f1190077a70 con 0x7f11900a50e0 2021-02-10T11:39:36.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.590 7f11877fe700 1 -- 172.21.2.117:0/1275833475 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v31) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1188006330 con 0x7f11900a50e0 2021-02-10T11:39:36.572 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 msgr2=0x7f117401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 0x7f117401abc0 crc :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 msgr2=0x7f11900a6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f1188006900 tx=0x7f118800bca0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 shutdown_connections 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11740186c0 0x7f117401abc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11900a50e0 0x7f11900a6080 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11900a46c0 0x7f11900a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 --2- 172.21.2.117:0/1275833475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11901a46b0 0x7f1190190400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 >> 172.21.2.117:0/1275833475 conn(0x7f11900b5140 msgr2=0x7f11900a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 shutdown_connections 2021-02-10T11:39:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:36.592 7f11950bf700 1 -- 172.21.2.117:0/1275833475 wait complete. 2021-02-10T11:39:36.582 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:39:36.583 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:37.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.008 7f63179fd700 1 Processor -- start 2021-02-10T11:39:37.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.008 7f63179fd700 1 -- start start 2021-02-10T11:39:37.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f63179fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 0x7f63100e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f63179fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63101b00e0 0x7f63101b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f63179fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6310077a70 con 0x7f631007da20 2021-02-10T11:39:37.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f63179fd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63100e53b0 con 0x7f63101b00e0 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f63179fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6310079400 con 0x7f6310079000 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63101b00e0 0x7f63101b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63101b00e0 0x7f63101b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47260/0 (socket says 172.21.2.117:47260) 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 -- 172.21.2.117:0/2378582801 learned_addr learned my addr 172.21.2.117:0/2378582801 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f6315018700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 0x7f63100e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 -- 172.21.2.117:0/2378582801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 msgr2=0x7f63100e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:37.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 0x7f63100e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 -- 172.21.2.117:0/2378582801 >> v1:172.21.2.125:6789/0 conn(0x7f631007da20 legacy=0x7f63100771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f631601a700 1 -- 172.21.2.117:0/2378582801 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631007de20 con 0x7f63101b00e0 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.009 7f6315018700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 0x7f63100e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f631601a700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63101b00e0 0x7f63101b04e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f6308006b80 tx=0x7f6308009b90).ready entity=mon.0 client_cookie=21b696332da4299f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f6314817700 1 -- 172.21.2.117:0/2378582801 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63080022e0 con 0x7f63101b00e0 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f6314817700 1 -- 172.21.2.117:0/2378582801 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308002560 con 0x7f63101b00e0 2021-02-10T11:39:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f6314817700 1 -- 172.21.2.117:0/2378582801 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6308002a80 con 0x7f63101b00e0 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f63179fd700 1 -- 172.21.2.117:0/2378582801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63101b00e0 msgr2=0x7f63101b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.010 7f63179fd700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63101b00e0 0x7f63101b04e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f6308006b80 tx=0x7f6308009b90).stop 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 -- 172.21.2.117:0/2378582801 shutdown_connections 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63101b00e0 0x7f63101b04e0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 --2- 172.21.2.117:0/2378582801 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6310079000 0x7f63100e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 -- 172.21.2.117:0/2378582801 >> 172.21.2.117:0/2378582801 conn(0x7f63100ce440 msgr2=0x7f63100e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:37.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 -- 172.21.2.117:0/2378582801 shutdown_connections 2021-02-10T11:39:37.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.011 7f63179fd700 1 -- 172.21.2.117:0/2378582801 wait complete. 2021-02-10T11:39:37.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.025 7f63179fd700 1 Processor -- start 2021-02-10T11:39:37.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.025 7f63179fd700 1 -- start start 2021-02-10T11:39:37.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63101a5640 0x7f63101a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63101a1fd0 0x7f631019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63101a23d0 con 0x7f631019c030 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63101a5a40 con 0x7f63101a5640 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f63179fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631008a1b0 con 0x7f63101a1fd0 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47264/0 (socket says 172.21.2.117:47264) 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 -- 172.21.2.117:0/3114610722 learned_addr learned my addr 172.21.2.117:0/3114610722 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 -- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63101a1fd0 msgr2=0x7f631019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63101a1fd0 0x7f631019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 -- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63101a5640 msgr2=0x7f63101a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.026 7f631601a700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63101a5640 0x7f63101a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f631601a700 1 -- 172.21.2.117:0/3114610722 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631019ca50 con 0x7f631019c030 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f631601a700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7f6308009640 tx=0x7f630800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6308009100 con 0x7f631019c030 2021-02-10T11:39:37.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630800bde0 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6308006430 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f63179fd700 1 -- 172.21.2.117:0/3114610722 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631019ca50 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.027 7f63179fd700 1 -- 172.21.2.117:0/3114610722 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631019ca50 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.028 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6308006240 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.028 7f63027fc700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 0x7f62f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.028 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f63080326d0 con 0x7f631019c030 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.028 7f6315819700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 0x7f62f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:37.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.029 7f6315819700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 0x7f62f801abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:37.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.059 7f63179fd700 1 -- 172.21.2.117:0/3114610722 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6310108c10 con 0x7f631019c030 2021-02-10T11:39:37.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.061 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f630802dc50 con 0x7f631019c030 2021-02-10T11:39:37.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.189 7f63179fd700 1 -- 172.21.2.117:0/3114610722 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6310077a70 con 0x7f631019c030 2021-02-10T11:39:37.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.189 7f63027fc700 1 -- 172.21.2.117:0/3114610722 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2095 (secure 0 0 0) 0x7f630802dc50 con 0x7f631019c030 2021-02-10T11:39:37.171 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:37.172 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:39:37.172 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 13 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 msgr2=0x7f62f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 0x7f62f801abc0 crc :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 msgr2=0x7f631019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7f6308009640 tx=0x7f630800b960).stop 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 shutdown_connections 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62f80186c0 0x7f62f801abc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f631019c030 0x7f631019c430 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:37.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63101a1fd0 0x7f631019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 --2- 172.21.2.117:0/3114610722 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63101a5640 0x7f63101a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:37.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 >> 172.21.2.117:0/3114610722 conn(0x7f6310191110 msgr2=0x7f631019aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:37.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 shutdown_connections 2021-02-10T11:39:37.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:37.191 7f63179fd700 1 -- 172.21.2.117:0/3114610722 wait complete. 2021-02-10T11:39:37.383 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:24:01.662717', 'modified': '2021-02-10 11:24:04.035038', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4412}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4412': {'gid': 4412, 'name': 'b', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.2.125:6835/3266045438', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3266045438}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3266045438}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:39:37.383 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:39:37.384 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:39:37.783 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 4412, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 936.13319724600001, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 12, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 31, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 31, 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 945.39465973699998 2021-02-10T11:39:37.784 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:39:37.788 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 4412, "want_state": "up:active", "state": "up:active", "rank_uptime": 936.13319724600001, "mdsmap_epoch": 12, "osdmap_epoch": 31, "osdmap_epoch_barrier": 31, "uptime": 945.39465973699998 } 2021-02-10T11:39:37.788 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:39:37.788 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:39:37.789 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:39:37.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:37.789 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:38.124 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:39:38.470 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:38.470 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:38.813 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:38.813 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:38.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:38.814 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:39:39.149 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:39:39.149 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2021-02-10T11:39:39.149 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:39.152 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:39:39.496 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:39:39.497 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:39:39.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[118517]: starting ceph client 2021-02-10T11:39:39.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:39:39.581 7f1c35bb1f80 -1 init, newargv = 0x563ca5e49f00 newargc=9 2021-02-10T11:39:39.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[118517]: starting fuse 2021-02-10T11:39:39.838 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:39:39.838 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:39:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:39:39.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:39.840 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:39:40.175 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:39:40.175 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:39:40.175 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:39:40.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:39:40.176 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:39:40.177 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:39:40.537 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.118517.asok 2021-02-10T11:39:40.539 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.118517.asok 2021-02-10T11:39:40.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:40.540 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.118517.asok status 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "118517", 2021-02-10T11:39:40.929 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 7134, 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 7134 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:39:40.930 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 440075317 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 440075317 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.7134 172.21.2.117:0/440075317", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/440075317", 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 12, 2021-02-10T11:39:40.931 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 31, 2021-02-10T11:39:40.932 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:39:40.932 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:39:40.932 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:39:40.933 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:39:41.281 INFO:teuthology.orchestra.run.gibba017.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:39:41.281 INFO:teuthology.orchestra.run.gibba017.stdout:0 2021-02-10T11:39:41.282 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:39:41.282 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:41.282 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:41.628 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:41.972 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:41.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:41.972 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:41.972 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:42.308 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:39:42.309 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:42.309 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:39:42.309 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:42.309 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:42.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[118517]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:42.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:42.665 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:42.665 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:42.998 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:39:42.998 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:42.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:42.999 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:43.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:43.333 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:43.333 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:43.669 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:39:43.670 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:43.670 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:39:43.670 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:39:43.670 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:44.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[45594]: fuse finished with error 0 and tester_r 0 2021-02-10T11:39:44.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:44.020 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:44.020 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:44.353 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:39:44.354 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:39:44.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:44.354 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:44.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:44.690 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:44.691 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:45.027 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:45.027 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:45.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:45.028 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:45.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:45.362 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:45.362 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:45.695 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:45.695 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:39:45.695 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:46.030 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:39:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 Processor -- start 2021-02-10T11:39:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 -- start start 2021-02-10T11:39:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7664079000 0x7f76640e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f76641b00d0 0x7f76641b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7664077a70 con 0x7f766407da20 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76640e5c00 con 0x7f7664079000 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f766950c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76640e53a0 con 0x7f76641b00d0 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f7662abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7664079000 0x7f76640e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f7662abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7664079000 0x7f76640e4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47284/0 (socket says 172.21.2.117:47284) 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.462 7f7662abe700 1 -- 172.21.2.117:0/405461454 learned_addr learned my addr 172.21.2.117:0/405461454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f7662abe700 1 -- 172.21.2.117:0/405461454 >> v2:172.21.2.125:3300/0 conn(0x7f76641b00d0 msgr2=0x7f76641b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f7662abe700 1 --2- 172.21.2.117:0/405461454 >> v2:172.21.2.125:3300/0 conn(0x7f76641b00d0 0x7f76641b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f7662abe700 1 -- 172.21.2.117:0/405461454 >> v1:172.21.2.125:6789/0 conn(0x7f766407da20 legacy=0x7f76640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f7662abe700 1 -- 172.21.2.117:0/405461454 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664079400 con 0x7f7664079000 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f7662abe700 1 --2- 172.21.2.117:0/405461454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7664079000 0x7f76640e4d80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f764c006700 tx=0x7f764c009b00).ready entity=mon.0 client_cookie=63fbd3f0c44103d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f76622bd700 1 -- 172.21.2.117:0/405461454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764c00c0c0 con 0x7f7664079000 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f76622bd700 1 -- 172.21.2.117:0/405461454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c00c340 con 0x7f7664079000 2021-02-10T11:39:46.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f76622bd700 1 -- 172.21.2.117:0/405461454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764c00c820 con 0x7f7664079000 2021-02-10T11:39:46.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f766950c700 1 -- 172.21.2.117:0/405461454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7664079000 msgr2=0x7f76640e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.463 7f766950c700 1 --2- 172.21.2.117:0/405461454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7664079000 0x7f76640e4d80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f764c006700 tx=0x7f764c009b00).stop 2021-02-10T11:39:46.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 -- 172.21.2.117:0/405461454 shutdown_connections 2021-02-10T11:39:46.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 --2- 172.21.2.117:0/405461454 >> v2:172.21.2.125:3300/0 conn(0x7f76641b00d0 0x7f76641b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 --2- 172.21.2.117:0/405461454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7664079000 0x7f76640e4d80 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 -- 172.21.2.117:0/405461454 >> 172.21.2.117:0/405461454 conn(0x7f76640ce430 msgr2=0x7f76640e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 -- 172.21.2.117:0/405461454 shutdown_connections 2021-02-10T11:39:46.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.464 7f766950c700 1 -- 172.21.2.117:0/405461454 wait complete. 2021-02-10T11:39:46.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 Processor -- start 2021-02-10T11:39:46.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 -- start start 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76641a5c00 0x7f76641a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76641a50c0 0x7f76640911f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76641a5680 con 0x7f76641a2600 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76641abea0 con 0x7f76641a5c00 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.479 7f766950c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7664091810 con 0x7f76641a50c0 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47292/0 (socket says 172.21.2.117:47292) 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 -- 172.21.2.117:0/2166514947 learned_addr learned my addr 172.21.2.117:0/2166514947 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 -- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76641a50c0 msgr2=0x7f76640911f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76641a50c0 0x7f76640911f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 -- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76641a5c00 msgr2=0x7f76641a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76641a5c00 0x7f76641a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 -- 172.21.2.117:0/2166514947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664091a30 con 0x7f76641a2600 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f7662abe700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f764c006f20 tx=0x7f764c00b910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764c00bd00 con 0x7f76641a2600 2021-02-10T11:39:46.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c000c90 con 0x7f76641a2600 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f764c0022b0 con 0x7f76641a2600 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f766950c700 1 -- 172.21.2.117:0/2166514947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664091a30 con 0x7f76641a2600 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.480 7f766950c700 1 -- 172.21.2.117:0/2166514947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7664091fc0 con 0x7f76641a2600 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.481 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f764c00a000 con 0x7f76641a2600 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.481 7f765bfff700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 0x7f764801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:46.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.481 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f764c032540 con 0x7f76641a2600 2021-02-10T11:39:46.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.482 7f76632bf700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 0x7f764801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:46.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.482 7f76632bf700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 0x7f764801abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:46.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.512 7f766950c700 1 -- 172.21.2.117:0/2166514947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7664077a70 con 0x7f76641a2600 2021-02-10T11:39:46.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.514 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f764c026020 con 0x7f76641a2600 2021-02-10T11:39:46.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.665 7f766950c700 1 -- 172.21.2.117:0/2166514947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7664077a70 con 0x7f76641a2600 2021-02-10T11:39:46.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.669 7f765bfff700 1 -- 172.21.2.117:0/2166514947 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7f764c026020 con 0x7f76641a2600 2021-02-10T11:39:46.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 msgr2=0x7f764801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 0x7f764801abc0 crc :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 msgr2=0x7f76641a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f764c006f20 tx=0x7f764c00b910).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 shutdown_connections 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76480186c0 0x7f764801abc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76641a2600 0x7f76641a4aa0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76641a50c0 0x7f76640911f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 --2- 172.21.2.117:0/2166514947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76641a5c00 0x7f76641a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:46.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 >> 172.21.2.117:0/2166514947 conn(0x7f7664199af0 msgr2=0x7f7664099550 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:46.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 shutdown_connections 2021-02-10T11:39:46.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:46.674 7f766950c700 1 -- 172.21.2.117:0/2166514947 wait complete. 2021-02-10T11:39:46.665 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm' 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.091 7f4e7c18e700 1 Processor -- start 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.091 7f4e7c18e700 1 -- start start 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e7c18e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 0x7f4e740d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e7c18e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e74077a70 con 0x7f4e74079000 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e7c18e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e74079400 con 0x7f4e7407da20 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e7c18e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7407de20 con 0x7f4e740d1520 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e797a9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4e74079000 0x7f4e740e4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47546/0 (socket says 172.21.2.117:47546) 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e797a9700 1 -- 172.21.2.117:0/2237535376 learned_addr learned my addr 172.21.2.117:0/2237535376 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:47.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e7a7ab700 1 --2- 172.21.2.117:0/2237535376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 0x7f4e740d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.092 7f4e78fa8700 1 -- 172.21.2.117:0/2237535376 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494316947 0 0) 0x7f4e700010b0 con 0x7f4e74079000 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e78fa8700 1 -- 172.21.2.117:0/2237535376 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e60002fe0 con 0x7f4e74079000 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7a7ab700 1 -- 172.21.2.117:0/2237535376 >> v1:172.21.2.125:6789/0 conn(0x7f4e7407da20 legacy=0x7f4e740771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7a7ab700 1 -- 172.21.2.117:0/2237535376 >> v1:172.21.2.117:6789/0 conn(0x7f4e74079000 legacy=0x7f4e740e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7a7ab700 1 -- 172.21.2.117:0/2237535376 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e741b00e0 con 0x7f4e740d1520 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7a7ab700 1 --2- 172.21.2.117:0/2237535376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 0x7f4e740d1920 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4e6c006e50 tx=0x7f4e6c009df0).ready entity=mon.2 client_cookie=e059c69cca993700 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e78fa8700 1 -- 172.21.2.117:0/2237535376 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6c00b510 con 0x7f4e740d1520 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e78fa8700 1 -- 172.21.2.117:0/2237535376 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c003030 con 0x7f4e740d1520 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e78fa8700 1 -- 172.21.2.117:0/2237535376 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6c00b7d0 con 0x7f4e740d1520 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7c18e700 1 -- 172.21.2.117:0/2237535376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 msgr2=0x7f4e740d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.093 7f4e7c18e700 1 --2- 172.21.2.117:0/2237535376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 0x7f4e740d1920 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4e6c006e50 tx=0x7f4e6c009df0).stop 2021-02-10T11:39:47.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.094 7f4e7c18e700 1 -- 172.21.2.117:0/2237535376 shutdown_connections 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.094 7f4e7c18e700 1 --2- 172.21.2.117:0/2237535376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e740d1520 0x7f4e740d1920 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.094 7f4e7c18e700 1 -- 172.21.2.117:0/2237535376 >> 172.21.2.117:0/2237535376 conn(0x7f4e740ce440 msgr2=0x7f4e740e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.094 7f4e7c18e700 1 -- 172.21.2.117:0/2237535376 shutdown_connections 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.094 7f4e7c18e700 1 -- 172.21.2.117:0/2237535376 wait complete. 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 Processor -- start 2021-02-10T11:39:47.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 -- start start 2021-02-10T11:39:47.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e741b3420 0x7f4e740aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e740af130 0x7f4e740af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7418be80 con 0x7f4e7418b910 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741aef00 con 0x7f4e740af130 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.109 7f4e7c18e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741aec70 con 0x7f4e741b3420 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47302/0 (socket says 172.21.2.117:47302) 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 -- 172.21.2.117:0/2992746684 learned_addr learned my addr 172.21.2.117:0/2992746684 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 -- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e741b3420 msgr2=0x7f4e740aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e741b3420 0x7f4e740aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e7a7ab700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e740af130 0x7f4e740af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 -- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e740af130 msgr2=0x7f4e740af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e740af130 0x7f4e740af530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 -- 172.21.2.117:0/2992746684 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e741b3820 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e79faa700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f4e68006ed0 tx=0x7f4e6800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e68006020 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e680062a0 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.110 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e741b3820 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.111 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e68005730 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.111 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e741b3820 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.111 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e680056c0 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.111 7f4e667fc700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 0x7f4e5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.111 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4e680326e0 con 0x7f4e7418b910 2021-02-10T11:39:47.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.112 7f4e797a9700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 0x7f4e5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:47.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.112 7f4e797a9700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 0x7f4e5c01abc0 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:47.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.142 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74108ba0 con 0x7f4e7418b910 2021-02-10T11:39:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.144 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e680054b0 con 0x7f4e7418b910 2021-02-10T11:39:47.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.275 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]} v 0) v1 -- 0x7f4e74077a70 con 0x7f4e7418b910 2021-02-10T11:39:47.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.724 7f4e667fc700 1 -- 172.21.2.117:0/2992746684 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_async_subvolume_rm"]}]=0 v207) v1 ==== 140+0+0 (secure 0 0 0) 0x7f4e680056c0 con 0x7f4e7418b910 2021-02-10T11:39:47.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 msgr2=0x7f4e5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 0x7f4e5c01abc0 crc :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 msgr2=0x7f4e741b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:47.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f4e68006ed0 tx=0x7f4e6800b960).stop 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 shutdown_connections 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e740af130 0x7f4e740af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e741b3420 0x7f4e740aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e5c0186c0 0x7f4e5c01abc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 --2- 172.21.2.117:0/2992746684 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7418b910 0x7f4e741b2e00 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 >> 172.21.2.117:0/2992746684 conn(0x7f4e7418cba0 msgr2=0x7f4e7418cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.745 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 shutdown_connections 2021-02-10T11:39:47.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:47.746 7f4e7c18e700 1 -- 172.21.2.117:0/2992746684 wait complete. 2021-02-10T11:39:47.735 INFO:tasks.cephfs_test_runner:test_async_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:39:47.736 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:39:47.736 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:39:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 Processor -- start 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 -- start start 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f320c079000 0x7f320c0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c0d1d10 con 0x7f320c1b00e0 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c077a70 con 0x7f320c07da20 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f32121c0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c0e53b0 con 0x7f320c079000 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f320bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f320c1b00e0 0x7f320c0d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47564/0 (socket says 172.21.2.117:47564) 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.163 7f320bfff700 1 -- 172.21.2.117:0/1284873331 learned_addr learned my addr 172.21.2.117:0/1284873331 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305818527 0 0) 0x7f3204001500 con 0x7f320c1b00e0 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8002fe0 con 0x7f320c1b00e0 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320affd700 1 --2- 172.21.2.117:0/1284873331 >> v2:172.21.2.125:3300/0 conn(0x7f320c079000 0x7f320c0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3295410205 0 0) 0x7f3204001500 con 0x7f320c1b00e0 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 >> v2:172.21.2.125:3300/0 conn(0x7f320c079000 msgr2=0x7f320c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 --2- 172.21.2.117:0/1284873331 >> v2:172.21.2.125:3300/0 conn(0x7f320c079000 0x7f320c0e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 >> v1:172.21.2.125:6789/0 conn(0x7f320c07da20 legacy=0x7f320c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c079400 con 0x7f320c1b00e0 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3204001a00 con 0x7f320c1b00e0 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 >> v1:172.21.2.117:6789/0 conn(0x7f320c1b00e0 legacy=0x7f320c0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 0x7f31f8005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f8006810 0x7f31f8008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f8009630 0x7f31f800bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800c320 con 0x7f31f8003a70 2021-02-10T11:39:48.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800c540 con 0x7f31f8006810 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f800c760 con 0x7f31f8009630 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.164 7f320b7fe700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 0x7f31f8005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320affd700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f8006810 0x7f31f8008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320bfff700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f8009630 0x7f31f800bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 -- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f8009630 msgr2=0x7f31f800bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f8009630 0x7f31f800bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 -- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f8006810 msgr2=0x7f31f8008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f8006810 0x7f31f8008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 -- 172.21.2.117:0/1284873331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80509d0 con 0x7f31f8003a70 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320bfff700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f8009630 0x7f31f800bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320bfff700 1 -- 172.21.2.117:0/1284873331 reap_dead start 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320b7fe700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 0x7f31f8005f70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f31fc006ab0 tx=0x7f31fc009b40).ready entity=mon.0 client_cookie=ad81b72d883dfad1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31fc00b2e0 con 0x7f31f8003a70 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc00b520 con 0x7f31f8003a70 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f320a7fc700 1 -- 172.21.2.117:0/1284873331 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31fc00b970 con 0x7f31f8003a70 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f32121c0700 1 -- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 msgr2=0x7f31f8005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.165 7f32121c0700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 0x7f31f8005f70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f31fc006ab0 tx=0x7f31fc009b40).stop 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.166 7f32121c0700 1 -- 172.21.2.117:0/1284873331 shutdown_connections 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.166 7f32121c0700 1 --2- 172.21.2.117:0/1284873331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f8003a70 0x7f31f8005f70 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.166 7f32121c0700 1 -- 172.21.2.117:0/1284873331 >> 172.21.2.117:0/1284873331 conn(0x7f320c0ce440 msgr2=0x7f320c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.166 7f32121c0700 1 -- 172.21.2.117:0/1284873331 shutdown_connections 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.166 7f32121c0700 1 -- 172.21.2.117:0/1284873331 wait complete. 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 Processor -- start 2021-02-10T11:39:48.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 -- start start 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f320c1aeb50 0x7f320c0b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f320c1aef50 0x7f320c1895b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c199a50 con 0x7f320c093cc0 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0b8bf0 con 0x7f320c1aeb50 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f32121c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0b8e10 con 0x7f320c1aef50 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f320b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f320b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47324/0 (socket says 172.21.2.117:47324) 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.181 7f320b7fe700 1 -- 172.21.2.117:0/3686002449 learned_addr learned my addr 172.21.2.117:0/3686002449 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:48.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 -- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f320c1aef50 msgr2=0x7f320c1895b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f320c1aef50 0x7f320c1895b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 -- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f320c1aeb50 msgr2=0x7f320c0b85d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f320c1aeb50 0x7f320c0b85d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 -- 172.21.2.117:0/3686002449 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c189bd0 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f320b7fe700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f31fc0093a0 tx=0x7f31fc00b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31fc00bdb0 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc000a10 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31fc0054a0 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f32121c0700 1 -- 172.21.2.117:0/3686002449 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c189bd0 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.182 7f32121c0700 1 -- 172.21.2.117:0/3686002449 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c189bd0 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.183 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31fc00a000 con 0x7f320c093cc0 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.183 7f3203fff700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 0x7f31f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:48.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.183 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f31fc0326b0 con 0x7f320c093cc0 2021-02-10T11:39:48.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.183 7f320affd700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 0x7f31f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:48.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.184 7f320affd700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 0x7f31f001abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:48.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.213 7f32121c0700 1 -- 172.21.2.117:0/3686002449 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c077a70 con 0x7f320c093cc0 2021-02-10T11:39:48.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.215 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31fc030d10 con 0x7f320c093cc0 2021-02-10T11:39:48.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.348 7f32121c0700 1 -- 172.21.2.117:0/3686002449 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7f320c077a70 con 0x7f320c093cc0 2021-02-10T11:39:48.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.728 7f3203fff700 1 -- 172.21.2.117:0/3686002449 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v208) v1 ==== 159+0+0 (secure 0 0 0) 0x7f31fc00a000 con 0x7f320c093cc0 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 -- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 msgr2=0x7f31f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 0x7f31f001abc0 crc :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 -- 172.21.2.117:0/3686002449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 msgr2=0x7f320c188930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f31fc0093a0 tx=0x7f31fc00b9c0).stop 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 -- 172.21.2.117:0/3686002449 shutdown_connections 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f320c1aef50 0x7f320c1895b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f320c1aeb50 0x7f320c0b85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:48.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31f00186c0 0x7f31f001abc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 --2- 172.21.2.117:0/3686002449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f320c093cc0 0x7f320c188930 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:48.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 -- 172.21.2.117:0/3686002449 >> 172.21.2.117:0/3686002449 conn(0x7f320c19f510 msgr2=0x7f320c187be0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:48.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.768 7f32121c0700 1 -- 172.21.2.117:0/3686002449 shutdown_connections 2021-02-10T11:39:48.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:48.769 7f32121c0700 1 -- 172.21.2.117:0/3686002449 wait complete. 2021-02-10T11:39:48.758 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 Processor -- start 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 -- start start 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdab407da20 0x7fdab40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40d1d20 con 0x7fdab40e51a0 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4077a70 con 0x7fdab40e4da0 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdaba2fd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab40777d0 con 0x7fdab407da20 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdab3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdab407da20 0x7fdab40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdab3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdab407da20 0x7fdab40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47330/0 (socket says 172.21.2.117:47330) 2021-02-10T11:39:49.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdab891a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdab40e51a0 0x7fdab40d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47580/0 (socket says 172.21.2.117:47580) 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.191 7fdab3fff700 1 -- 172.21.2.117:0/2233913735 learned_addr learned my addr 172.21.2.117:0/2233913735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab3fff700 1 -- 172.21.2.117:0/2233913735 >> v1:172.21.2.125:6789/0 conn(0x7fdab40e4da0 legacy=0x7fdab41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab3fff700 1 -- 172.21.2.117:0/2233913735 >> v1:172.21.2.117:6789/0 conn(0x7fdab40e51a0 legacy=0x7fdab40d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab3fff700 1 -- 172.21.2.117:0/2233913735 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab41b08e0 con 0x7fdab407da20 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab3fff700 1 --2- 172.21.2.117:0/2233913735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab407da20 0x7fdab40771b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fdaa4006700 tx=0x7fdaa4009b80).ready entity=mon.0 client_cookie=e7714176db404f4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab2ffd700 1 -- 172.21.2.117:0/2233913735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdaa400c0c0 con 0x7fdab407da20 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab2ffd700 1 -- 172.21.2.117:0/2233913735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa400c340 con 0x7fdab407da20 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdab2ffd700 1 -- 172.21.2.117:0/2233913735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdaa400c860 con 0x7fdab407da20 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdaba2fd700 1 -- 172.21.2.117:0/2233913735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab407da20 msgr2=0x7fdab40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.192 7fdaba2fd700 1 --2- 172.21.2.117:0/2233913735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab407da20 0x7fdab40771b0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fdaa4006700 tx=0x7fdaa4009b80).stop 2021-02-10T11:39:49.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.193 7fdaba2fd700 1 -- 172.21.2.117:0/2233913735 shutdown_connections 2021-02-10T11:39:49.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.193 7fdaba2fd700 1 --2- 172.21.2.117:0/2233913735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab407da20 0x7fdab40771b0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.193 7fdaba2fd700 1 -- 172.21.2.117:0/2233913735 >> 172.21.2.117:0/2233913735 conn(0x7fdab40ce450 msgr2=0x7fdab40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:49.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.193 7fdaba2fd700 1 -- 172.21.2.117:0/2233913735 shutdown_connections 2021-02-10T11:39:49.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.193 7fdaba2fd700 1 -- 172.21.2.117:0/2233913735 wait complete. 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.207 7fdaba2fd700 1 Processor -- start 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 -- start start 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 0x7fdab40b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdab40b6190 0x7fdab41b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 0x7fdab41b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab41b0f50 con 0x7fdab41b0770 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab40b6590 con 0x7fdab40b6190 2021-02-10T11:39:49.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdaba2fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab41b0b70 con 0x7fdab41c22d0 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdab3fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 0x7fdab40b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdab891a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 0x7fdab41b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdab3fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 0x7fdab40b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:45014/0 (socket says 172.21.2.117:45014) 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdab3fff700 1 -- 172.21.2.117:0/1038694931 learned_addr learned my addr 172.21.2.117:0/1038694931 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.208 7fdab37fe700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdab40b6190 0x7fdab41b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 -- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 msgr2=0x7fdab40b5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 0x7fdab40b5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 -- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdab40b6190 msgr2=0x7fdab41b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdab40b6190 0x7fdab41b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 -- 172.21.2.117:0/1038694931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab41b1af0 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab891a700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 0x7fdab41b14d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fdaac001b90 tx=0x7fdaac00abe0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdaac0070d0 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac007390 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab41b1af0 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab41b2080 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.209 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdaac004910 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.210 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdaac004720 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.210 7fdab0ff9700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 0x7fda9401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.210 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdaac032b90 con 0x7fdab41b0770 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.210 7fdab3fff700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 0x7fda9401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:49.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.210 7fdab3fff700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 0x7fda9401ac10 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:49.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.241 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab41b2850 con 0x7fdab41b0770 2021-02-10T11:39:49.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.243 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdaac000ec0 con 0x7fdab41b0770 2021-02-10T11:39:49.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.392 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdab41b2850 con 0x7fdab41b0770 2021-02-10T11:39:49.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.395 7fdab0ff9700 1 -- 172.21.2.117:0/1038694931 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdaac004720 con 0x7fdab41b0770 2021-02-10T11:39:49.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 msgr2=0x7fda9401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 0x7fda9401ac10 crc :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 msgr2=0x7fdab41b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:49.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 0x7fdab41b14d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fdaac001b90 tx=0x7fdaac00abe0).stop 2021-02-10T11:39:49.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 shutdown_connections 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda94018730 0x7fda9401ac10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdab41b0770 0x7fdab41b14d0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdab40b6190 0x7fdab41b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 --2- 172.21.2.117:0/1038694931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdab41c22d0 0x7fdab40b5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.400 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 >> 172.21.2.117:0/1038694931 conn(0x7fdab41116c0 msgr2=0x7fdab4108870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.401 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 shutdown_connections 2021-02-10T11:39:49.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:49.401 7fdaba2fd700 1 -- 172.21.2.117:0/1038694931 wait complete. 2021-02-10T11:39:49.391 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:39:49.392 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:49.736 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:39:50.081 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:39:50.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:50.082 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:50.082 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:39:50.416 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:39:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:50.416 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:39:50.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:50.417 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:50.417 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:39:50.752 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:39:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:50.752 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:39:50.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:50.752 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:50.753 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:39:51.087 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:39:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:51.088 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:39:51.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:39:51.088 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:39:51.088 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:39:51.422 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:39:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:39:51.423 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:39:51.423 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.851 7f07c0234700 1 Processor -- start 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.851 7f07c0234700 1 -- start start 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b807da20 0x7f07b80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f07b80e4d90 0x7f07b81b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f07b80e5190 0x7f07b8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80e5c10 con 0x7f07b80e4d90 2021-02-10T11:39:51.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b8077a70 con 0x7f07b80e5190 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07c0234700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80777d0 con 0x7f07b807da20 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f07b80e4d90 0x7f07b81b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f07b80e4d90 0x7f07b81b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47342/0 (socket says 172.21.2.117:47342) 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 -- 172.21.2.117:0/2798882664 learned_addr learned my addr 172.21.2.117:0/2798882664 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 -- 172.21.2.117:0/2798882664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b807da20 msgr2=0x7f07b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 --2- 172.21.2.117:0/2798882664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b807da20 0x7f07b80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 -- 172.21.2.117:0/2798882664 >> v2:172.21.2.125:3300/0 conn(0x7f07b80e5190 msgr2=0x7f07b8077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 --2- 172.21.2.117:0/2798882664 >> v2:172.21.2.125:3300/0 conn(0x7f07b80e5190 0x7f07b8077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.852 7f07bd84f700 1 -- 172.21.2.117:0/2798882664 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b807de20 con 0x7f07b80e4d90 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07bd84f700 1 --2- 172.21.2.117:0/2798882664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b80e4d90 0x7f07b81b00e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f07b4006700 tx=0x7f07b4009b80).ready entity=mon.0 client_cookie=6b50fdbb77420106 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07bd04e700 1 -- 172.21.2.117:0/2798882664 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b40009b0 con 0x7f07b80e4d90 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07bd04e700 1 -- 172.21.2.117:0/2798882664 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b400c0c0 con 0x7f07b80e4d90 2021-02-10T11:39:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07bd04e700 1 -- 172.21.2.117:0/2798882664 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b400c600 con 0x7f07b80e4d90 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07c0234700 1 -- 172.21.2.117:0/2798882664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b80e4d90 msgr2=0x7f07b81b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.853 7f07c0234700 1 --2- 172.21.2.117:0/2798882664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b80e4d90 0x7f07b81b00e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f07b4006700 tx=0x7f07b4009b80).stop 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 -- 172.21.2.117:0/2798882664 shutdown_connections 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 --2- 172.21.2.117:0/2798882664 >> v2:172.21.2.125:3300/0 conn(0x7f07b80e5190 0x7f07b8077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 --2- 172.21.2.117:0/2798882664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b80e4d90 0x7f07b81b00e0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 --2- 172.21.2.117:0/2798882664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b807da20 0x7f07b80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 -- 172.21.2.117:0/2798882664 >> 172.21.2.117:0/2798882664 conn(0x7f07b80ce440 msgr2=0x7f07b80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 -- 172.21.2.117:0/2798882664 shutdown_connections 2021-02-10T11:39:51.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.854 7f07c0234700 1 -- 172.21.2.117:0/2798882664 wait complete. 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 Processor -- start 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 -- start start 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f07b81a27c0 0x7f07b819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 0x7f07b819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80ab7d0 con 0x7f07b81a5640 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81a5a40 con 0x7f07b81a27c0 2021-02-10T11:39:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07c0234700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8199d40 con 0x7f07b819b930 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07bd84f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07bd84f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47348/0 (socket says 172.21.2.117:47348) 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.869 7f07bd84f700 1 -- 172.21.2.117:0/3974985714 learned_addr learned my addr 172.21.2.117:0/3974985714 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07be851700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 0x7f07b819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 -- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 msgr2=0x7f07b819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 0x7f07b819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 -- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f07b81a27c0 msgr2=0x7f07b819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f07b81a27c0 0x7f07b819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 -- 172.21.2.117:0/3974985714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b819c350 con 0x7f07b81a5640 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07be851700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 0x7f07b819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07bd84f700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f07b4009240 tx=0x7f07b400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b4006580 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b4002030 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07b4002550 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07c0234700 1 -- 172.21.2.117:0/3974985714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b819c350 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.870 7f07c0234700 1 -- 172.21.2.117:0/3974985714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b819c350 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.871 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f07b4000c90 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.871 7f07aaffd700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 0x7f07a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.871 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f07b40326f0 con 0x7f07b81a5640 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.871 7f07be050700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 0x7f07a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:51.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.872 7f07be050700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 0x7f07a001abc0 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:51.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.902 7f07c0234700 1 -- 172.21.2.117:0/3974985714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b80ab7d0 con 0x7f07b81a5640 2021-02-10T11:39:51.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:51.905 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f07b4001f30 con 0x7f07b81a5640 2021-02-10T11:39:52.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.028 7f07c0234700 1 -- 172.21.2.117:0/3974985714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f07b80ab7d0 con 0x7f07b81a5640 2021-02-10T11:39:52.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.029 7f07aaffd700 1 -- 172.21.2.117:0/3974985714 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v31) v1 ==== 81+0+30752 (secure 0 0 0) 0x7f07b4007570 con 0x7f07b81a5640 2021-02-10T11:39:52.011 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:52.011 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 31, 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:24:02.664299", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:39:52.012 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 5, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.013 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:52.014 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:39:52.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:39:52.016 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:52.017 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:39:52.018 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 4, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:23:59.642031", 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:39:52.019 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:52.020 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "30", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:52.021 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:39:52.022 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:52.023 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:39:52.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 5, 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:24:00.660978", 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:39:52.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:39:52.026 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "31", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:39:52.027 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:39:52.028 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:39:52.029 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:39:52.030 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.031 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.032 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.033 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:39:52.034 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.036 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 28, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.037 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.039 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.040 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:39:52.041 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.042 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.043 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.046 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.047 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:39:52.048 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.050 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.053 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:39:52.054 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.056 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.058 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:39:52.059 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.060 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.061 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.063 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.064 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.065 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:39:52.066 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.067 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.069 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.070 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:39:52.071 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.072 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:39:52.073 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.074 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:39:52.075 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.076 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.077 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:39:52.078 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.079 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.080 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:39:52.081 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.082 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:39:52.083 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:39:52.084 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:39:52.085 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:39:52.086 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 msgr2=0x7f07a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 0x7f07a001abc0 crc :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 msgr2=0x7f07b81a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f07b4009240 tx=0x7f07b400b960).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 shutdown_connections 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07a00186c0 0x7f07a001abc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f07b81a5640 0x7f07b81a19b0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f07b819b930 0x7f07b819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 --2- 172.21.2.117:0/3974985714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f07b81a27c0 0x7f07b819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 >> 172.21.2.117:0/3974985714 conn(0x7f07b8191110 msgr2=0x7f07b819aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 shutdown_connections 2021-02-10T11:39:52.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.031 7f07c0234700 1 -- 172.21.2.117:0/3974985714 wait complete. 2021-02-10T11:39:52.089 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.517 7f3f21590700 1 Processor -- start 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.517 7f3f21590700 1 -- start start 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f21590700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f1c0d1520 0x7f3f1c0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f21590700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c077a70 con 0x7f3f1c079000 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f21590700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1c079400 con 0x7f3f1c07da20 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f21590700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1c07de20 con 0x7f3f1c0d1520 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f1aabe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3f1c079000 0x7f3f1c0e4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47602/0 (socket says 172.21.2.117:47602) 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f1bac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f1c0d1520 0x7f3f1c0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f1aabe700 1 -- 172.21.2.117:0/1224865428 learned_addr learned my addr 172.21.2.117:0/1224865428 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:52.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f1a2bd700 1 -- 172.21.2.117:0/1224865428 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673822952 0 0) 0x7f3f0c000ce0 con 0x7f3f1c079000 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.518 7f3f1a2bd700 1 -- 172.21.2.117:0/1224865428 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f08002fe0 con 0x7f3f1c079000 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1bac0700 1 -- 172.21.2.117:0/1224865428 >> v1:172.21.2.125:6789/0 conn(0x7f3f1c07da20 legacy=0x7f3f1c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1bac0700 1 -- 172.21.2.117:0/1224865428 >> v1:172.21.2.117:6789/0 conn(0x7f3f1c079000 legacy=0x7f3f1c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1bac0700 1 -- 172.21.2.117:0/1224865428 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c1b00e0 con 0x7f3f1c0d1520 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1bac0700 1 --2- 172.21.2.117:0/1224865428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c0d1520 0x7f3f1c0d1920 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f3f14006e30 tx=0x7f3f14009de0).ready entity=mon.0 client_cookie=ebf9cd690a55df8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1a2bd700 1 -- 172.21.2.117:0/1224865428 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f1400b300 con 0x7f3f1c0d1520 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1a2bd700 1 -- 172.21.2.117:0/1224865428 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1400b580 con 0x7f3f1c0d1520 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f1a2bd700 1 -- 172.21.2.117:0/1224865428 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f1400baa0 con 0x7f3f1c0d1520 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f21590700 1 -- 172.21.2.117:0/1224865428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c0d1520 msgr2=0x7f3f1c0d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.519 7f3f21590700 1 --2- 172.21.2.117:0/1224865428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c0d1520 0x7f3f1c0d1920 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f3f14006e30 tx=0x7f3f14009de0).stop 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.520 7f3f21590700 1 -- 172.21.2.117:0/1224865428 shutdown_connections 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.520 7f3f21590700 1 --2- 172.21.2.117:0/1224865428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c0d1520 0x7f3f1c0d1920 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.520 7f3f21590700 1 -- 172.21.2.117:0/1224865428 >> 172.21.2.117:0/1224865428 conn(0x7f3f1c0ce440 msgr2=0x7f3f1c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.520 7f3f21590700 1 -- 172.21.2.117:0/1224865428 shutdown_connections 2021-02-10T11:39:52.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.520 7f3f21590700 1 -- 172.21.2.117:0/1224865428 wait complete. 2021-02-10T11:39:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.534 7f3f21590700 1 Processor -- start 2021-02-10T11:39:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 -- start start 2021-02-10T11:39:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f1c19d3b0 0x7f3f1c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f1c109bf0 0x7f3f1c109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c10a610 con 0x7f3f1c09da70 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c09e490 con 0x7f3f1c19d3b0 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f21590700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1c09e6b0 con 0x7f3f1c109bf0 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f1bac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f1bac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47360/0 (socket says 172.21.2.117:47360) 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f1bac0700 1 -- 172.21.2.117:0/3321358524 learned_addr learned my addr 172.21.2.117:0/3321358524 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f1aabe700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f1c109bf0 0x7f3f1c109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.535 7f3f1b2bf700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f1c19d3b0 0x7f3f1c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 -- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f1c109bf0 msgr2=0x7f3f1c109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f1c109bf0 0x7f3f1c109ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 -- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f1c19d3b0 msgr2=0x7f3f1c19d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f1c19d3b0 0x7f3f1c19d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 -- 172.21.2.117:0/3321358524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c09e8d0 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f1bac0700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f3f1400b910 tx=0x7f3f1400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f1400bdf0 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f14005820 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f21590700 1 -- 172.21.2.117:0/3321358524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1c09e8d0 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f14005c80 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.536 7f3f21590700 1 -- 172.21.2.117:0/3321358524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1c09e8d0 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.537 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f14005a00 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.537 7f3f13fff700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 0x7f3f0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.537 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f14032a40 con 0x7f3f1c09da70 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.537 7f3f1b2bf700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 0x7f3f0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:52.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.538 7f3f1b2bf700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 0x7f3f0001abc0 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:52.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.568 7f3f21590700 1 -- 172.21.2.117:0/3321358524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1c108ba0 con 0x7f3f1c09da70 2021-02-10T11:39:52.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.570 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f1400bdf0 con 0x7f3f1c09da70 2021-02-10T11:39:52.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.697 7f3f21590700 1 -- 172.21.2.117:0/3321358524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f1c077a70 con 0x7f3f1c09da70 2021-02-10T11:39:52.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.698 7f3f13fff700 1 -- 172.21.2.117:0/3321358524 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2095 (secure 0 0 0) 0x7f3f1402dba0 con 0x7f3f1c09da70 2021-02-10T11:39:52.680 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:52.680 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:24:04.035038","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4412},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4412":{"gid":4412,"name":"b","rank":0,"incarnation":10,"state":"up:active","state_seq":4,"addr":"172.21.2.125:6835/3266045438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3266045438},{"type":"v1","addr":"172.21.2.125:6835","nonce":3266045438}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:39:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 13 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 -- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 msgr2=0x7f3f0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 0x7f3f0001abc0 crc :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 -- 172.21.2.117:0/3321358524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 msgr2=0x7f3f1c09de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f3f1400b910 tx=0x7f3f1400bcb0).stop 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 -- 172.21.2.117:0/3321358524 shutdown_connections 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f000186c0 0x7f3f0001abc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f1c09da70 0x7f3f1c09de70 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f1c109bf0 0x7f3f1c109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 --2- 172.21.2.117:0/3321358524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f1c19d3b0 0x7f3f1c19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.699 7f3f21590700 1 -- 172.21.2.117:0/3321358524 >> 172.21.2.117:0/3321358524 conn(0x7f3f1c19a700 msgr2=0x7f3f1c199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.700 7f3f21590700 1 -- 172.21.2.117:0/3321358524 shutdown_connections 2021-02-10T11:39:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:52.700 7f3f21590700 1 -- 172.21.2.117:0/3321358524 wait complete. 2021-02-10T11:39:52.690 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.118 7fe9328c0700 1 Processor -- start 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.118 7fe9328c0700 1 -- start start 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe9328c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c07da20 0x7fe92c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe9328c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe92c0e4d80 0x7fe92c1b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe9328c0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c077a70 con 0x7fe92c0e5180 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe9328c0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0e5c00 con 0x7fe92c0e4d80 2021-02-10T11:39:53.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe9328c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0777d0 con 0x7fe92c07da20 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe92c0e4d80 0x7fe92c1b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe92c0e4d80 0x7fe92c1b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47366/0 (socket says 172.21.2.117:47366) 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 -- 172.21.2.117:0/2819157820 learned_addr learned my addr 172.21.2.117:0/2819157820 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 -- 172.21.2.117:0/2819157820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c07da20 msgr2=0x7fe92c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 --2- 172.21.2.117:0/2819157820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c07da20 0x7fe92c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 -- 172.21.2.117:0/2819157820 >> v1:172.21.2.117:6789/0 conn(0x7fe92c0e5180 legacy=0x7fe92c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 -- 172.21.2.117:0/2819157820 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c07de20 con 0x7fe92c0e4d80 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.119 7fe92b7fe700 1 --2- 172.21.2.117:0/2819157820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c0e4d80 0x7fe92c1b00d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fe918006700 tx=0x7fe918009b80).ready entity=mon.0 client_cookie=9d5f03639c3593e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe92affd700 1 -- 172.21.2.117:0/2819157820 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe91800c0c0 con 0x7fe92c0e4d80 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe92affd700 1 -- 172.21.2.117:0/2819157820 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91800c440 con 0x7fe92c0e4d80 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe92affd700 1 -- 172.21.2.117:0/2819157820 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe91800c8a0 con 0x7fe92c0e4d80 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 -- 172.21.2.117:0/2819157820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c0e4d80 msgr2=0x7fe92c1b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 --2- 172.21.2.117:0/2819157820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c0e4d80 0x7fe92c1b00d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7fe918006700 tx=0x7fe918009b80).stop 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 -- 172.21.2.117:0/2819157820 shutdown_connections 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 --2- 172.21.2.117:0/2819157820 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c0e4d80 0x7fe92c1b00d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 --2- 172.21.2.117:0/2819157820 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c07da20 0x7fe92c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 -- 172.21.2.117:0/2819157820 >> 172.21.2.117:0/2819157820 conn(0x7fe92c0ce430 msgr2=0x7fe92c0e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 -- 172.21.2.117:0/2819157820 shutdown_connections 2021-02-10T11:39:53.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.120 7fe9328c0700 1 -- 172.21.2.117:0/2819157820 wait complete. 2021-02-10T11:39:53.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 Processor -- start 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 -- start start 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe92c09a050 0x7fe92c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c09a450 0x7fe92c0a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0a3e20 con 0x7fe92c18d370 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0a4000 con 0x7fe92c09a050 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe9328c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c18dd90 con 0x7fe92c09a450 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.135 7fe930edd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47368/0 (socket says 172.21.2.117:47368) 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 -- 172.21.2.117:0/282548938 learned_addr learned my addr 172.21.2.117:0/282548938 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 -- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c09a450 msgr2=0x7fe92c0a3800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c09a450 0x7fe92c0a3800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 -- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe92c09a050 msgr2=0x7fe92c0a31e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe92c09a050 0x7fe92c0a31e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 -- 172.21.2.117:0/282548938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c18df70 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe930edd700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe924009980 tx=0x7fe92400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe924005870 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe924005a70 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe924006030 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe9328c0700 1 -- 172.21.2.117:0/282548938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c18df70 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.136 7fe9328c0700 1 -- 172.21.2.117:0/282548938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c18e500 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.137 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe924005c50 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.137 7fe928ff9700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 0x7fe90801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.137 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe924032a30 con 0x7fe92c18d370 2021-02-10T11:39:53.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.138 7fe92bfff700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 0x7fe90801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:53.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.139 7fe92bfff700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 0x7fe90801abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:53.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.169 7fe9328c0700 1 -- 172.21.2.117:0/282548938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c103370 con 0x7fe92c18d370 2021-02-10T11:39:53.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.171 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe92400be10 con 0x7fe92c18d370 2021-02-10T11:39:53.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.299 7fe9328c0700 1 -- 172.21.2.117:0/282548938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe92c077a70 con 0x7fe92c18d370 2021-02-10T11:39:53.722 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:39:53.740 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:39:53.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.746 7fe928ff9700 1 -- 172.21.2.117:0/282548938 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe924005c50 con 0x7fe92c18d370 2021-02-10T11:39:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 msgr2=0x7fe90801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 0x7fe90801abc0 crc :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 msgr2=0x7fe92c18d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe924009980 tx=0x7fe92400bcd0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 shutdown_connections 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9080186c0 0x7fe90801abc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe92c18d370 0x7fe92c18d770 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe92c09a450 0x7fe92c0a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 --2- 172.21.2.117:0/282548938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe92c09a050 0x7fe92c0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 >> 172.21.2.117:0/282548938 conn(0x7fe92c19f610 msgr2=0x7fe92c09af00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 shutdown_connections 2021-02-10T11:39:53.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:53.767 7fe9328c0700 1 -- 172.21.2.117:0/282548938 wait complete. 2021-02-10T11:39:53.757 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:39:53.780 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:39:54.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 Processor -- start 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 -- start start 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcb4007da20 0x7fcb400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb400e5190 0x7fcb40079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40077a70 con 0x7fcb400e4d90 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb400e5c10 con 0x7fcb400e5190 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb454d7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb400777d0 con 0x7fcb4007da20 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb3fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb400e5190 0x7fcb40079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb3fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb400e5190 0x7fcb40079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47376/0 (socket says 172.21.2.117:47376) 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.185 7fcb3fac0700 1 -- 172.21.2.117:0/2549026135 learned_addr learned my addr 172.21.2.117:0/2549026135 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3fac0700 1 -- 172.21.2.117:0/2549026135 >> v2:172.21.2.125:3300/0 conn(0x7fcb4007da20 msgr2=0x7fcb400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3fac0700 1 --2- 172.21.2.117:0/2549026135 >> v2:172.21.2.125:3300/0 conn(0x7fcb4007da20 0x7fcb400771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3fac0700 1 -- 172.21.2.117:0/2549026135 >> v1:172.21.2.125:6789/0 conn(0x7fcb400e4d90 legacy=0x7fcb401b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3fac0700 1 -- 172.21.2.117:0/2549026135 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb401b08d0 con 0x7fcb400e5190 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3fac0700 1 --2- 172.21.2.117:0/2549026135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb400e5190 0x7fcb40079000 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fcb38006f00 tx=0x7fcb38009f10).ready entity=mon.0 client_cookie=5c3d6c0a180bffc0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3e2bd700 1 -- 172.21.2.117:0/2549026135 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3800b3b0 con 0x7fcb400e5190 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3e2bd700 1 -- 172.21.2.117:0/2549026135 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3800b630 con 0x7fcb400e5190 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb3e2bd700 1 -- 172.21.2.117:0/2549026135 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3800bb50 con 0x7fcb400e5190 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb454d7700 1 -- 172.21.2.117:0/2549026135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb400e5190 msgr2=0x7fcb40079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.186 7fcb454d7700 1 --2- 172.21.2.117:0/2549026135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb400e5190 0x7fcb40079000 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fcb38006f00 tx=0x7fcb38009f10).stop 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 -- 172.21.2.117:0/2549026135 shutdown_connections 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 --2- 172.21.2.117:0/2549026135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb400e5190 0x7fcb40079000 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 --2- 172.21.2.117:0/2549026135 >> v2:172.21.2.125:3300/0 conn(0x7fcb4007da20 0x7fcb400771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 -- 172.21.2.117:0/2549026135 >> 172.21.2.117:0/2549026135 conn(0x7fcb400ce440 msgr2=0x7fcb400e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 -- 172.21.2.117:0/2549026135 shutdown_connections 2021-02-10T11:39:54.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.187 7fcb454d7700 1 -- 172.21.2.117:0/2549026135 wait complete. 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.201 7fcb454d7700 1 Processor -- start 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.201 7fcb454d7700 1 -- start start 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 0x7fcb4018f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cff0 0x7fcb4018d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb401af0c0 con 0x7fcb4018f880 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4018da10 con 0x7fcb4018cff0 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb454d7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4018dc30 con 0x7fcb4018ee60 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47380/0 (socket says 172.21.2.117:47380) 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 -- 172.21.2.117:0/996823353 learned_addr learned my addr 172.21.2.117:0/996823353 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3f2bf700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 0x7fcb4018f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 -- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 msgr2=0x7fcb4018f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 0x7fcb4018f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 -- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cff0 msgr2=0x7fcb4018d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cff0 0x7fcb4018d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 -- 172.21.2.117:0/996823353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4018de50 con 0x7fcb4018f880 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3f2bf700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 0x7fcb4018f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:54.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.202 7fcb3eabe700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7fcb3000b5b0 tx=0x7fcb3000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3000bd40 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb30000ce0 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb454d7700 1 -- 172.21.2.117:0/996823353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4018de50 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb454d7700 1 -- 172.21.2.117:0/996823353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4018de50 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb300054a0 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb30006240 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 0x7fcb2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.203 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fcb30032760 con 0x7fcb4018f880 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.204 7fcb3f2bf700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 0x7fcb2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.204 7fcb3f2bf700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 0x7fcb2401abc0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.235 7fcb454d7700 1 -- 172.21.2.117:0/996823353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4018e930 con 0x7fcb4018f880 2021-02-10T11:39:54.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.238 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb3002cf30 con 0x7fcb4018f880 2021-02-10T11:39:54.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.365 7fcb454d7700 1 -- 172.21.2.117:0/996823353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb4018e930 con 0x7fcb4018f880 2021-02-10T11:39:54.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.366 7fcb37fff700 1 -- 172.21.2.117:0/996823353 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+1750 (secure 0 0 0) 0x7fcb3002cf30 con 0x7fcb4018f880 2021-02-10T11:39:54.348 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 14 2021-02-10T11:39:54.348 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:54.348 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4848,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/233938727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":233938727},{"type":"v1","addr":"172.21.2.117:6833","nonce":233938727}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":9}],"filesystems":[{"mdsmap":{"epoch":14,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:24:01.662717","modified":"2021-02-10 11:39:53.742074","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 msgr2=0x7fcb2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 0x7fcb2401abc0 crc :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 msgr2=0x7fcb4018fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7fcb3000b5b0 tx=0x7fcb3000b950).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 shutdown_connections 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb240186c0 0x7fcb2401abc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4018f880 0x7fcb4018fc80 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cff0 0x7fcb4018d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 --2- 172.21.2.117:0/996823353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb4018ee60 0x7fcb4018f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 >> 172.21.2.117:0/996823353 conn(0x7fcb4018cba0 msgr2=0x7fcb401a6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 shutdown_connections 2021-02-10T11:39:54.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.368 7fcb454d7700 1 -- 172.21.2.117:0/996823353 wait complete. 2021-02-10T11:39:54.358 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.792 7fc786d3a700 1 Processor -- start 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.792 7fc786d3a700 1 -- start start 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc786d3a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc78007da20 0x7fc7800771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc786d3a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc7800e4d80 0x7fc7801b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc786d3a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc780077a70 con 0x7fc7800e5180 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc786d3a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7800777d0 con 0x7fc78007da20 2021-02-10T11:39:54.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc786d3a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78007de20 con 0x7fc7800e4d80 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc784b56700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc78007da20 0x7fc7800771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc785357700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc7800e5180 0x7fc780077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47636/0 (socket says 172.21.2.117:47636) 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc785357700 1 -- 172.21.2.117:0/290990738 learned_addr learned my addr 172.21.2.117:0/290990738 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc77ffff700 1 --2- 172.21.2.117:0/290990738 >> v2:172.21.2.125:3300/0 conn(0x7fc7800e4d80 0x7fc7801b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc784b56700 1 -- 172.21.2.117:0/290990738 >> v2:172.21.2.125:3300/0 conn(0x7fc7800e4d80 msgr2=0x7fc7801b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc784b56700 1 --2- 172.21.2.117:0/290990738 >> v2:172.21.2.125:3300/0 conn(0x7fc7800e4d80 0x7fc7801b00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc784b56700 1 -- 172.21.2.117:0/290990738 >> v1:172.21.2.117:6789/0 conn(0x7fc7800e5180 legacy=0x7fc780077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.793 7fc784b56700 1 -- 172.21.2.117:0/290990738 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7801b06f0 con 0x7fc78007da20 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc784b56700 1 --2- 172.21.2.117:0/290990738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78007da20 0x7fc7800771b0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fc768006650 tx=0x7fc768009a80).ready entity=mon.0 client_cookie=af1ddd8a6702aa5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc77f7fe700 1 -- 172.21.2.117:0/290990738 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc76800c0c0 con 0x7fc78007da20 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc77f7fe700 1 -- 172.21.2.117:0/290990738 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76800c340 con 0x7fc78007da20 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc77f7fe700 1 -- 172.21.2.117:0/290990738 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc76800c860 con 0x7fc78007da20 2021-02-10T11:39:54.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc786d3a700 1 -- 172.21.2.117:0/290990738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78007da20 msgr2=0x7fc7800771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.794 7fc786d3a700 1 --2- 172.21.2.117:0/290990738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78007da20 0x7fc7800771b0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fc768006650 tx=0x7fc768009a80).stop 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 -- 172.21.2.117:0/290990738 shutdown_connections 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 --2- 172.21.2.117:0/290990738 >> v2:172.21.2.125:3300/0 conn(0x7fc7800e4d80 0x7fc7801b00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 --2- 172.21.2.117:0/290990738 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78007da20 0x7fc7800771b0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 -- 172.21.2.117:0/290990738 >> 172.21.2.117:0/290990738 conn(0x7fc7800ce430 msgr2=0x7fc7800e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 -- 172.21.2.117:0/290990738 shutdown_connections 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.795 7fc786d3a700 1 -- 172.21.2.117:0/290990738 wait complete. 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.809 7fc786d3a700 1 Processor -- start 2021-02-10T11:39:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.809 7fc786d3a700 1 -- start start 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7801a0cd0 0x7fc7801a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a10d0 0x7fc7801a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801a4360 con 0x7fc7801a0cd0 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801a2860 con 0x7fc7801a10d0 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc786d3a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801a2240 con 0x7fc78009e000 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:45074/0 (socket says 172.21.2.117:45074) 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 -- 172.21.2.117:0/1736516628 learned_addr learned my addr 172.21.2.117:0/1736516628 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc77ffff700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7801a0cd0 0x7fc7801a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc785357700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a10d0 0x7fc7801a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 -- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a10d0 msgr2=0x7fc7801a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a10d0 0x7fc7801a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 -- 172.21.2.117:0/1736516628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7801a0cd0 msgr2=0x7fc7801a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7801a0cd0 0x7fc7801a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.810 7fc784b56700 1 -- 172.21.2.117:0/1736516628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7801a2420 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc784b56700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc768009630 tx=0x7fc76800b970).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc76800bd60 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc768006740 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc768002300 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7801a2420 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.811 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc780187ec0 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.812 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc768000c90 con 0x7fc78009e000 2021-02-10T11:39:54.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.812 7fc77d7fa700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.812 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fc768032740 con 0x7fc78009e000 2021-02-10T11:39:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.812 7fc77ffff700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:54.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.812 7fc77ffff700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:54.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.842 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc780077a70 con 0x7fc78009e000 2021-02-10T11:39:54.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.845 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc76802ddc0 con 0x7fc78009e000 2021-02-10T11:39:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:54.975 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc780077a70 con 0x7fc78009e000 2021-02-10T11:39:55.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.751 7fc77d7fa700 1 -- 172.21.2.117:0/1736516628 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc76802ddc0 con 0x7fc78009e000 2021-02-10T11:39:55.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.796 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 msgr2=0x7fc76401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.796 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 crc :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.796 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 msgr2=0x7fc7801a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.796 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc768009630 tx=0x7fc76800b970).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 shutdown_connections 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7801a0cd0 0x7fc7801a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a10d0 0x7fc7801a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 --2- 172.21.2.117:0/1736516628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009e000 0x7fc7801a06b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 >> 172.21.2.117:0/1736516628 conn(0x7fc780199470 msgr2=0x7fc78019f660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 shutdown_connections 2021-02-10T11:39:55.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:55.797 7fc786d3a700 1 -- 172.21.2.117:0/1736516628 wait complete. 2021-02-10T11:39:55.787 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:39:56.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.219 7fee52424700 1 Processor -- start 2021-02-10T11:39:56.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.219 7fee52424700 1 -- start start 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee52424700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fee4c078fa0 0x7fee4c0e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee52424700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fee4c1b0130 0x7fee4c1b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee52424700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4c077a10 con 0x7fee4c07da20 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee52424700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c0e5c60 con 0x7fee4c078fa0 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee52424700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee4c0e5420 con 0x7fee4c1b0130 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fee4c078fa0 0x7fee4c0e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fee4c078fa0 0x7fee4c0e4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47400/0 (socket says 172.21.2.117:47400) 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 -- 172.21.2.117:0/1188234403 learned_addr learned my addr 172.21.2.117:0/1188234403 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 -- 172.21.2.117:0/1188234403 >> v2:172.21.2.125:3300/0 conn(0x7fee4c1b0130 msgr2=0x7fee4c1b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 --2- 172.21.2.117:0/1188234403 >> v2:172.21.2.125:3300/0 conn(0x7fee4c1b0130 0x7fee4c1b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 -- 172.21.2.117:0/1188234403 >> v1:172.21.2.125:6789/0 conn(0x7fee4c07da20 legacy=0x7fee4c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:56.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.220 7fee4b7fe700 1 -- 172.21.2.117:0/1188234403 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0793a0 con 0x7fee4c078fa0 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee4b7fe700 1 --2- 172.21.2.117:0/1188234403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c078fa0 0x7fee4c0e4e00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7fee40006700 tx=0x7fee40009b80).ready entity=mon.0 client_cookie=8d2b0115d002443 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee4affd700 1 -- 172.21.2.117:0/1188234403 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4000c0c0 con 0x7fee4c078fa0 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee4affd700 1 -- 172.21.2.117:0/1188234403 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4000c440 con 0x7fee4c078fa0 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee4affd700 1 -- 172.21.2.117:0/1188234403 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4000c960 con 0x7fee4c078fa0 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee52424700 1 -- 172.21.2.117:0/1188234403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c078fa0 msgr2=0x7fee4c0e4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.221 7fee52424700 1 --2- 172.21.2.117:0/1188234403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c078fa0 0x7fee4c0e4e00 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0x7fee40006700 tx=0x7fee40009b80).stop 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 -- 172.21.2.117:0/1188234403 shutdown_connections 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 --2- 172.21.2.117:0/1188234403 >> v2:172.21.2.125:3300/0 conn(0x7fee4c1b0130 0x7fee4c1b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 --2- 172.21.2.117:0/1188234403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c078fa0 0x7fee4c0e4e00 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 -- 172.21.2.117:0/1188234403 >> 172.21.2.117:0/1188234403 conn(0x7fee4c0ce480 msgr2=0x7fee4c0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 -- 172.21.2.117:0/1188234403 shutdown_connections 2021-02-10T11:39:56.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.222 7fee52424700 1 -- 172.21.2.117:0/1188234403 wait complete. 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.236 7fee52424700 1 Processor -- start 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.236 7fee52424700 1 -- start start 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee4c0c3bb0 0x7fee4c0c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee4c107a30 0x7fee4c0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c190b80 con 0x7fee4c107630 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0c4020 con 0x7fee4c0c3bb0 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee52424700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c0afbc0 con 0x7fee4c107a30 2021-02-10T11:39:56.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47406/0 (socket says 172.21.2.117:47406) 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 -- 172.21.2.117:0/737796883 learned_addr learned my addr 172.21.2.117:0/737796883 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 -- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee4c107a30 msgr2=0x7fee4c0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee4c107a30 0x7fee4c0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 -- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee4c0c3bb0 msgr2=0x7fee4c0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee4c0c3bb0 0x7fee4c0c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.237 7fee4b7fe700 1 -- 172.21.2.117:0/737796883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c0afde0 con 0x7fee4c107630 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee4b7fe700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fee4000b5e0 tx=0x7fee4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4000bd70 con 0x7fee4c107630 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee40002030 con 0x7fee4c107630 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee52424700 1 -- 172.21.2.117:0/737796883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c0afde0 con 0x7fee4c107630 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee52424700 1 -- 172.21.2.117:0/737796883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1a7260 con 0x7fee4c107630 2021-02-10T11:39:56.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee40002550 con 0x7fee4c107630 2021-02-10T11:39:56.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee40000c90 con 0x7fee4c107630 2021-02-10T11:39:56.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.238 7fee48ff9700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 0x7fee2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:56.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.239 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fee40032840 con 0x7fee4c107630 2021-02-10T11:39:56.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.239 7fee4bfff700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 0x7fee2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:56.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.239 7fee4bfff700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 0x7fee2c01abc0 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:56.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.270 7fee52424700 1 -- 172.21.2.117:0/737796883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c1051e0 con 0x7fee4c107630 2021-02-10T11:39:56.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.272 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee400009b0 con 0x7fee4c107630 2021-02-10T11:39:56.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.398 7fee52424700 1 -- 172.21.2.117:0/737796883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee4c077a10 con 0x7fee4c107630 2021-02-10T11:39:56.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.756 7fee48ff9700 1 -- 172.21.2.117:0/737796883 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v33) v1 ==== 185+0+0 (secure 0 0 0) 0x7fee40000c90 con 0x7fee4c107630 2021-02-10T11:39:56.744 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 -- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 msgr2=0x7fee2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 0x7fee2c01abc0 crc :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 -- 172.21.2.117:0/737796883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 msgr2=0x7fee4c190490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fee4000b5e0 tx=0x7fee4000b980).stop 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 -- 172.21.2.117:0/737796883 shutdown_connections 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee2c018730 0x7fee2c01abc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee4c107630 0x7fee4c190490 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee4c107a30 0x7fee4c0af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 --2- 172.21.2.117:0/737796883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee4c0c3bb0 0x7fee4c0c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.764 7fee52424700 1 -- 172.21.2.117:0/737796883 >> 172.21.2.117:0/737796883 conn(0x7fee4c0c0480 msgr2=0x7fee4c0bf360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.765 7fee52424700 1 -- 172.21.2.117:0/737796883 shutdown_connections 2021-02-10T11:39:56.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:56.765 7fee52424700 1 -- 172.21.2.117:0/737796883 wait complete. 2021-02-10T11:39:56.755 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:39:57.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.185 7f51693b9700 1 Processor -- start 2021-02-10T11:39:57.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.185 7f51693b9700 1 -- start start 2021-02-10T11:39:57.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f51693b9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f516407da20 0x7f51640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f51693b9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5164077a10 con 0x7f51640e5200 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f51693b9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51641b0920 con 0x7f51640e4e00 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f51693b9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640777d0 con 0x7f516407da20 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f516387e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f51640e5200 0x7f51640d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47662/0 (socket says 172.21.2.117:47662) 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f516387e700 1 -- 172.21.2.117:0/4263861086 learned_addr learned my addr 172.21.2.117:0/4263861086 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136016782 0 0) 0x7f515c001500 con 0x7f51640e5200 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.186 7f516207b700 1 -- 172.21.2.117:0/4263861086 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514c002fe0 con 0x7f51640e5200 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2435354718 0 0) 0x7f515c001460 con 0x7f51640e5200 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 >> v2:172.21.2.125:3300/0 conn(0x7f516407da20 msgr2=0x7f51640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 --2- 172.21.2.117:0/4263861086 >> v2:172.21.2.125:3300/0 conn(0x7f516407da20 0x7f51640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 >> v1:172.21.2.125:6789/0 conn(0x7f51640e4e00 legacy=0x7f51641b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640d1d50 con 0x7f51640e5200 2021-02-10T11:39:57.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f515c001980 con 0x7f51640e5200 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 >> v1:172.21.2.117:6789/0 conn(0x7f51640e5200 legacy=0x7f51640d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f514c003ac0 0x7f514c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 0x7f514c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f514c009590 0x7f514c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c00c300 con 0x7f514c006770 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c00c520 con 0x7f514c009590 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c00c740 con 0x7f514c003ac0 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 0x7f514c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516307d700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f514c003ac0 0x7f514c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516387e700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f514c009590 0x7f514c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 -- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f514c003ac0 msgr2=0x7f514c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f514c003ac0 0x7f514c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 -- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f514c009590 msgr2=0x7f514c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f514c009590 0x7f514c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 -- 172.21.2.117:0/4263861086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c00c9e0 con 0x7f514c006770 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516387e700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f514c009590 0x7f514c00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516387e700 1 -- 172.21.2.117:0/4263861086 reap_dead start 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516287c700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 0x7f514c008c90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f5154006ab0 tx=0x7f515404fbc0).ready entity=mon.0 client_cookie=48ff1aa7a462bc64 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5154006240 con 0x7f514c006770 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51540054a0 con 0x7f514c006770 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f516207b700 1 -- 172.21.2.117:0/4263861086 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51540058f0 con 0x7f514c006770 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f51693b9700 1 -- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 msgr2=0x7f514c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.187 7f51693b9700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 0x7f514c008c90 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f5154006ab0 tx=0x7f515404fbc0).stop 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.188 7f51693b9700 1 -- 172.21.2.117:0/4263861086 shutdown_connections 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.188 7f51693b9700 1 --2- 172.21.2.117:0/4263861086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f514c006770 0x7f514c008c90 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.188 7f51693b9700 1 -- 172.21.2.117:0/4263861086 >> 172.21.2.117:0/4263861086 conn(0x7f51640ce480 msgr2=0x7f51640e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.188 7f51693b9700 1 -- 172.21.2.117:0/4263861086 shutdown_connections 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.188 7f51693b9700 1 -- 172.21.2.117:0/4263861086 wait complete. 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 Processor -- start 2021-02-10T11:39:57.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 -- start start 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51640c3960 0x7f516408a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51641a00d0 0x7f51640c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516408a3b0 con 0x7f516419fcd0 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516408a010 con 0x7f51640c3960 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f51693b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640c3ee0 con 0x7f51641a00d0 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f516287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f516287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47422/0 (socket says 172.21.2.117:47422) 2021-02-10T11:39:57.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.203 7f516287c700 1 -- 172.21.2.117:0/3515541767 learned_addr learned my addr 172.21.2.117:0/3515541767 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 -- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51641a00d0 msgr2=0x7f51640c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51641a00d0 0x7f51640c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 -- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51640c3960 msgr2=0x7f516408a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51640c3960 0x7f516408a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 -- 172.21.2.117:0/3515541767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516408b0c0 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f516287c700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f5154002500 tx=0x7f51540028a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5154000e00 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51540080c0 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f51693b9700 1 -- 172.21.2.117:0/3515541767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516408b0c0 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51540502a0 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.204 7f51693b9700 1 -- 172.21.2.117:0/3515541767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f516408b650 con 0x7f516419fcd0 2021-02-10T11:39:57.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.205 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f515404f670 con 0x7f516419fcd0 2021-02-10T11:39:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.205 7f514bfff700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 0x7f514401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.205 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f5154076620 con 0x7f516419fcd0 2021-02-10T11:39:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.205 7f516307d700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 0x7f514401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:57.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.206 7f516307d700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 0x7f514401abd0 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.236 7f51693b9700 1 -- 172.21.2.117:0/3515541767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51641051c0 con 0x7f516419fcd0 2021-02-10T11:39:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.238 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5154050560 con 0x7f516419fcd0 2021-02-10T11:39:57.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.364 7f51693b9700 1 -- 172.21.2.117:0/3515541767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5164077a10 con 0x7f516419fcd0 2021-02-10T11:39:57.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.766 7f514bfff700 1 -- 172.21.2.117:0/3515541767 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v34) v1 ==== 173+0+0 (secure 0 0 0) 0x7f515404f670 con 0x7f516419fcd0 2021-02-10T11:39:57.762 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:39:57.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 -- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 msgr2=0x7f514401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 0x7f514401abd0 crc :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 -- 172.21.2.117:0/3515541767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 msgr2=0x7f51640c2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f5154002500 tx=0x7f51540028a0).stop 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 -- 172.21.2.117:0/3515541767 shutdown_connections 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51440186d0 0x7f514401abd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f516419fcd0 0x7f51640c2c90 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51641a00d0 0x7f51640c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 --2- 172.21.2.117:0/3515541767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51640c3960 0x7f516408a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.783 7f51693b9700 1 -- 172.21.2.117:0/3515541767 >> 172.21.2.117:0/3515541767 conn(0x7f51640c0870 msgr2=0x7f51640b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.784 7f51693b9700 1 -- 172.21.2.117:0/3515541767 shutdown_connections 2021-02-10T11:39:57.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:57.784 7f51693b9700 1 -- 172.21.2.117:0/3515541767 wait complete. 2021-02-10T11:39:57.773 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:39:57.773 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:39:57.774 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:39:57.774 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:57.774 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:39:57.774 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:39:57.774 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:39:57.774 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:39:57.779 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:39:57.779 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:39:57.780 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:39:57.780 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:39:57.822 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:39:57.822 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:57.823 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:39:57.823 DEBUG:teuthology.parallel:result is None 2021-02-10T11:39:57.823 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:39:58.228 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:39:58.234 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.258 7ff05a94c700 1 Processor -- start 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.258 7ff05a94c700 1 -- start start 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff05a94c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff054079000 0x7ff0540e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff05a94c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff0541b00d0 0x7ff0541b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff05a94c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff054077a70 con 0x7ff05407da20 2021-02-10T11:39:58.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff05a94c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0540e53a0 con 0x7ff0541b00d0 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff05a94c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff054079400 con 0x7ff054079000 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff053fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff05407da20 0x7ff0540771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47694/0 (socket says 172.21.2.117:47694) 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff053fff700 1 -- 172.21.2.117:0/2005446871 learned_addr learned my addr 172.21.2.117:0/2005446871 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102405923 0 0) 0x7ff04c000d30 con 0x7ff05407da20 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff03c002fe0 con 0x7ff05407da20 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff058f69700 1 --2- 172.21.2.117:0/2005446871 >> v2:172.21.2.125:3300/0 conn(0x7ff0541b00d0 0x7ff0541b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2977120535 0 0) 0x7ff04c001390 con 0x7ff05407da20 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff054079000 msgr2=0x7ff0540e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff054079000 0x7ff0540e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff054079000 0x7ff0540e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 >> v2:172.21.2.125:3300/0 conn(0x7ff0541b00d0 msgr2=0x7ff0541b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 --2- 172.21.2.117:0/2005446871 >> v2:172.21.2.125:3300/0 conn(0x7ff0541b00d0 0x7ff0541b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05407de20 con 0x7ff05407da20 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff04c001910 con 0x7ff05407da20 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 >> v1:172.21.2.117:6789/0 conn(0x7ff05407da20 legacy=0x7ff0540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff03c003af0 0x7ff03c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff03c006840 0x7ff03c008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff03c009640 0x7ff03c00bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c050390 con 0x7ff03c006840 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c050570 con 0x7ff03c003af0 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c050790 con 0x7ff03c009640 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.259 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff054079000 0x7ff0540e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff03c006840 0x7ff03c008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff058f69700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff03c009640 0x7ff03c00bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff053fff700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff03c003af0 0x7ff03c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 -- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff03c009640 msgr2=0x7ff03c00bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff03c009640 0x7ff03c00bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 -- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff03c003af0 msgr2=0x7ff03c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff03c003af0 0x7ff03c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 -- 172.21.2.117:0/2005446871 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c050a30 con 0x7ff03c006840 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff053fff700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff03c003af0 0x7ff03c005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff0537fe700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff03c006840 0x7ff03c008d40 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7ff048002e00 tx=0x7ff048009ab0).ready entity=mon.0 client_cookie=7a175ba53ea7f3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff048000c90 con 0x7ff03c006840 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04800c0c0 con 0x7ff03c006840 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff052ffd700 1 -- 172.21.2.117:0/2005446871 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04800c5e0 con 0x7ff03c006840 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 -- 172.21.2.117:0/2005446871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff03c006840 msgr2=0x7ff03c008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 --2- 172.21.2.117:0/2005446871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff03c006840 0x7ff03c008d40 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7ff048002e00 tx=0x7ff048009ab0).stop 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff058f69700 1 -- 172.21.2.117:0/2005446871 reap_dead start 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 -- 172.21.2.117:0/2005446871 shutdown_connections 2021-02-10T11:39:58.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 -- 172.21.2.117:0/2005446871 >> 172.21.2.117:0/2005446871 conn(0x7ff0540ce430 msgr2=0x7ff0540e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 -- 172.21.2.117:0/2005446871 shutdown_connections 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.260 7ff05a94c700 1 -- 172.21.2.117:0/2005446871 wait complete. 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.275 7ff05a94c700 1 Processor -- start 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 -- start start 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0540b4930 0x7ff0540a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0540a2fe0 0x7ff0541996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0541080f0 con 0x7ff054198c90 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff054199170 con 0x7ff0540a2fe0 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff05a94c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff054107c90 con 0x7ff0540b4930 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff0537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff0537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47454/0 (socket says 172.21.2.117:47454) 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff0537fe700 1 -- 172.21.2.117:0/3282432473 learned_addr learned my addr 172.21.2.117:0/3282432473 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:58.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff058f69700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0540a2fe0 0x7ff0541996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.276 7ff053fff700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0540b4930 0x7ff0540a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 -- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0540b4930 msgr2=0x7ff0540a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0540b4930 0x7ff0540a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 -- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0540a2fe0 msgr2=0x7ff0541996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0540a2fe0 0x7ff0541996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 -- 172.21.2.117:0/3282432473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff054199ce0 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff0537fe700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff048009660 tx=0x7ff04800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04800bd70 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff048002030 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff054199ce0 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05419a270 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff048002550 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.277 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff048000c90 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.278 7ff050ff9700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 0x7ff03401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.278 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5186+0+0 (secure 0 0 0) 0x7ff0480326a0 con 0x7ff054198c90 2021-02-10T11:39:58.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.278 7ff053fff700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 0x7ff03401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:58.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.278 7ff053fff700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 0x7ff03401abd0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:58.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.307 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff054077a70 con 0x7ff054198c90 2021-02-10T11:39:58.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.310 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff048000940 con 0x7ff054198c90 2021-02-10T11:39:58.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.449 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff054199170 con 0x7ff054198c90 2021-02-10T11:39:58.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.769 7ff050ff9700 1 -- 172.21.2.117:0/3282432473 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v35) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff048000c90 con 0x7ff054198c90 2021-02-10T11:39:58.796 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 msgr2=0x7ff03401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 0x7ff03401abd0 crc :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 msgr2=0x7ff0540a29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7ff048009660 tx=0x7ff04800b980).stop 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 shutdown_connections 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff0340186d0 0x7ff03401abd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff054198c90 0x7ff0540a29c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0540a2fe0 0x7ff0541996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 --2- 172.21.2.117:0/3282432473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0540b4930 0x7ff0540a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:58.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.817 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 >> 172.21.2.117:0/3282432473 conn(0x7ff05419f5a0 msgr2=0x7ff0541abde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:58.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.818 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 shutdown_connections 2021-02-10T11:39:58.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:58.818 7ff05a94c700 1 -- 172.21.2.117:0/3282432473 wait complete. 2021-02-10T11:39:58.806 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:39:58.807 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:39:59.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.235 7fc2694f2700 1 Processor -- start 2021-02-10T11:39:59.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.235 7fc2694f2700 1 -- start start 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc26407da20 0x7fc2640771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2640e4d90 0x7fc2641b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 0x7fc264077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc264077a70 con 0x7fc26407da20 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640777d0 con 0x7fc2640e5190 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2694f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26407de20 con 0x7fc2640e4d90 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc26407da20 0x7fc2640771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc26407da20 0x7fc2640771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47460/0 (socket says 172.21.2.117:47460) 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 -- 172.21.2.117:0/2025317043 learned_addr learned my addr 172.21.2.117:0/2025317043 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 -- 172.21.2.117:0/2025317043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2640e4d90 msgr2=0x7fc2641b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 --2- 172.21.2.117:0/2025317043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2640e4d90 0x7fc2641b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 -- 172.21.2.117:0/2025317043 >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 msgr2=0x7fc264077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:59.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc263ac0700 1 --2- 172.21.2.117:0/2025317043 >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 0x7fc264077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 --2- 172.21.2.117:0/2025317043 >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 0x7fc264077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc2632bf700 1 -- 172.21.2.117:0/2025317043 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2641b0700 con 0x7fc26407da20 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.236 7fc263ac0700 1 --2- 172.21.2.117:0/2025317043 >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 0x7fc264077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2632bf700 1 --2- 172.21.2.117:0/2025317043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26407da20 0x7fc2640771b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fc254006700 tx=0x7fc254009b00).ready entity=mon.0 client_cookie=8d36f09a6f02f88e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2622bd700 1 -- 172.21.2.117:0/2025317043 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc25400c0c0 con 0x7fc26407da20 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2622bd700 1 -- 172.21.2.117:0/2025317043 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25400c400 con 0x7fc26407da20 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2622bd700 1 -- 172.21.2.117:0/2025317043 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc25400c920 con 0x7fc26407da20 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2694f2700 1 -- 172.21.2.117:0/2025317043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26407da20 msgr2=0x7fc2640771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.237 7fc2694f2700 1 --2- 172.21.2.117:0/2025317043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26407da20 0x7fc2640771b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fc254006700 tx=0x7fc254009b00).stop 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 -- 172.21.2.117:0/2025317043 shutdown_connections 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 --2- 172.21.2.117:0/2025317043 >> v2:172.21.2.125:3300/0 conn(0x7fc2640e5190 0x7fc264077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 --2- 172.21.2.117:0/2025317043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2640e4d90 0x7fc2641b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 --2- 172.21.2.117:0/2025317043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26407da20 0x7fc2640771b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 -- 172.21.2.117:0/2025317043 >> 172.21.2.117:0/2025317043 conn(0x7fc2640ce440 msgr2=0x7fc2640e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 -- 172.21.2.117:0/2025317043 shutdown_connections 2021-02-10T11:39:59.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.238 7fc2694f2700 1 -- 172.21.2.117:0/2025317043 wait complete. 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.252 7fc2694f2700 1 Processor -- start 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.252 7fc2694f2700 1 -- start start 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 0x7fc26418e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc26418ef10 0x7fc26418f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26418d930 con 0x7fc26418d530 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26418e230 con 0x7fc26418e4f0 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2694f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc264192f30 con 0x7fc26418ef10 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47464/0 (socket says 172.21.2.117:47464) 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 -- 172.21.2.117:0/3342109475 learned_addr learned my addr 172.21.2.117:0/3342109475 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc263ac0700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc26418ef10 0x7fc26418f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 -- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc26418ef10 msgr2=0x7fc26418f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc26418ef10 0x7fc26418f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 -- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 msgr2=0x7fc26418e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc262abe700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 0x7fc26418e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 0x7fc26418e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc2632bf700 1 -- 172.21.2.117:0/3342109475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc264193150 con 0x7fc26418d530 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.253 7fc262abe700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 0x7fc26418e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc2632bf700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fc25400b590 tx=0x7fc25400b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc254006390 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc254009100 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2540052e0 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc264193150 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc264193150 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2540052e0 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 0x7fc24401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.254 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc254032210 con 0x7fc26418d530 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.255 7fc262abe700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 0x7fc24401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.255 7fc262abe700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 0x7fc24401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.285 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26418d930 con 0x7fc26418d530 2021-02-10T11:39:59.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.288 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc254006300 con 0x7fc26418d530 2021-02-10T11:39:59.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.416 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc26418d930 con 0x7fc26418d530 2021-02-10T11:39:59.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.417 7fc24bfff700 1 -- 172.21.2.117:0/3342109475 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fc25400bd20 con 0x7fc26418d530 2021-02-10T11:39:59.398 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:39:59.398 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:39:59.398 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:39:59.398 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.399 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.400 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:39:59.401 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.402 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.403 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.404 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.405 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.405 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:39:59.405 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:39:59.405 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.405 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:39:59.406 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:39:59.407 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.408 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:39:59.409 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.410 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:39:59.411 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:39:59.412 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 msgr2=0x7fc24401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 0x7fc24401abc0 crc :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 msgr2=0x7fc26418dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fc25400b590 tx=0x7fc25400b930).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 shutdown_connections 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc26418ef10 0x7fc26418f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc26418e4f0 0x7fc26418e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2440186c0 0x7fc24401abc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 --2- 172.21.2.117:0/3342109475 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc26418d530 0x7fc26418dc10 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 >> 172.21.2.117:0/3342109475 conn(0x7fc2641ace70 msgr2=0x7fc2641af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 shutdown_connections 2021-02-10T11:39:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.419 7fc2694f2700 1 -- 172.21.2.117:0/3342109475 wait complete. 2021-02-10T11:39:59.415 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.845 7f29b5024700 1 Processor -- start 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.845 7f29b5024700 1 -- start start 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29b5024700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29b007da20 0x7f29b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29b5024700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b00e4d90 0x7f29b01b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29b5024700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0077a70 con 0x7f29b00e5190 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29b5024700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b00777d0 con 0x7f29b007da20 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29b5024700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b007de20 con 0x7f29b00e4d90 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29b007da20 0x7f29b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f29b007da20 0x7f29b00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47472/0 (socket says 172.21.2.117:47472) 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 -- 172.21.2.117:0/2953948711 learned_addr learned my addr 172.21.2.117:0/2953948711 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:59.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 -- 172.21.2.117:0/2953948711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b00e4d90 msgr2=0x7f29b01b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 --2- 172.21.2.117:0/2953948711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b00e4d90 0x7f29b01b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.846 7f29aed9d700 1 -- 172.21.2.117:0/2953948711 >> v1:172.21.2.125:6789/0 conn(0x7f29b00e5190 legacy=0x7f29b0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29aed9d700 1 -- 172.21.2.117:0/2953948711 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b01b0700 con 0x7f29b007da20 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29aed9d700 1 --2- 172.21.2.117:0/2953948711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b007da20 0x7f29b00771b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f29a0006700 tx=0x7f29a0009b80).ready entity=mon.0 client_cookie=c32ed0186007c7a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29add9b700 1 -- 172.21.2.117:0/2953948711 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a000c0c0 con 0x7f29b007da20 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29add9b700 1 -- 172.21.2.117:0/2953948711 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a000c440 con 0x7f29b007da20 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29add9b700 1 -- 172.21.2.117:0/2953948711 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a000c960 con 0x7f29b007da20 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29b5024700 1 -- 172.21.2.117:0/2953948711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b007da20 msgr2=0x7f29b00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.847 7f29b5024700 1 --2- 172.21.2.117:0/2953948711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b007da20 0x7f29b00771b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f29a0006700 tx=0x7f29a0009b80).stop 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 -- 172.21.2.117:0/2953948711 shutdown_connections 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 --2- 172.21.2.117:0/2953948711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b00e4d90 0x7f29b01b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 --2- 172.21.2.117:0/2953948711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b007da20 0x7f29b00771b0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 -- 172.21.2.117:0/2953948711 >> 172.21.2.117:0/2953948711 conn(0x7f29b00ce440 msgr2=0x7f29b00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:39:59.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 -- 172.21.2.117:0/2953948711 shutdown_connections 2021-02-10T11:39:59.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.848 7f29b5024700 1 -- 172.21.2.117:0/2953948711 wait complete. 2021-02-10T11:39:59.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.862 7f29b5024700 1 Processor -- start 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.862 7f29b5024700 1 -- start start 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 0x7f29b01a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b01a8c60 0x7f29b01a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b019d3a0 con 0x7f29b019d920 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b01af340 con 0x7f29b010a210 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29b5024700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b019aea0 con 0x7f29b01a8c60 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47478/0 (socket says 172.21.2.117:47478) 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 -- 172.21.2.117:0/549631539 learned_addr learned my addr 172.21.2.117:0/549631539 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29af59e700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b01a8c60 0x7f29b01a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29ae59c700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 0x7f29b01a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 -- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b01a8c60 msgr2=0x7f29b01a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b01a8c60 0x7f29b01a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 -- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 msgr2=0x7f29b01a8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 0x7f29b01a8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29aed9d700 1 -- 172.21.2.117:0/549631539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b010a610 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.863 7f29ae59c700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 0x7f29b01a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f29aed9d700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f29a000b5e0 tx=0x7f29a000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a000bd70 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0002030 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29a00024d0 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f29b5024700 1 -- 172.21.2.117:0/549631539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b010a610 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.864 7f29b5024700 1 -- 172.21.2.117:0/549631539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b010a610 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.865 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f29a0000c90 con 0x7f29b019d920 2021-02-10T11:39:59.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.865 7f299f7fe700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:39:59.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.865 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f29a0032430 con 0x7f29b019d920 2021-02-10T11:39:59.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.865 7f29ae59c700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:39:59.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.865 7f29ae59c700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:39:59.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.896 7f29b5024700 1 -- 172.21.2.117:0/549631539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0108ba0 con 0x7f29b019d920 2021-02-10T11:39:59.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:39:59.898 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29a0030dc0 con 0x7f29b019d920 2021-02-10T11:40:00.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.022 7f29b5024700 1 -- 172.21.2.117:0/549631539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29b0077a70 con 0x7f29b019d920 2021-02-10T11:40:00.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.023 7f299f7fe700 1 -- 172.21.2.117:0/549631539 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+11377 (secure 0 0 0) 0x7f29a0000c90 con 0x7f29b019d920 2021-02-10T11:40:00.004 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:00.005 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":35,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:39:58.765964","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.024 7f29b5024700 1 -- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 msgr2=0x7f299401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.024 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 crc :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.024 7f29b5024700 1 -- 172.21.2.117:0/549631539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 msgr2=0x7f29b0109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.024 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f29a000b5e0 tx=0x7f29a000b980).stop 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 -- 172.21.2.117:0/549631539 shutdown_connections 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29b01a8c60 0x7f29b01a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29b010a210 0x7f29b01a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29940186c0 0x7f299401abc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 --2- 172.21.2.117:0/549631539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29b019d920 0x7f29b0109bf0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 -- 172.21.2.117:0/549631539 >> 172.21.2.117:0/549631539 conn(0x7f29b019a700 msgr2=0x7f29b0199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 -- 172.21.2.117:0/549631539 shutdown_connections 2021-02-10T11:40:00.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.025 7f29b5024700 1 -- 172.21.2.117:0/549631539 wait complete. 2021-02-10T11:40:00.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:40:00.015 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:00.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 Processor -- start 2021-02-10T11:40:00.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 -- start start 2021-02-10T11:40:00.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 0x7fc5100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc5100e51a0 0x7fc5100d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc510077a70 con 0x7fc5100e4da0 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5100e5c20 con 0x7fc5100e51a0 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.446 7fc5173b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100777d0 con 0x7fc51007da20 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5159d0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc5100e51a0 0x7fc5100d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 0x7fc5100771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5149ce700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fc5100e4da0 0x7fc5101b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40660/0 (socket says 172.21.2.117:40660) 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5149ce700 1 -- 172.21.2.117:0/26758078 learned_addr learned my addr 172.21.2.117:0/26758078 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 -- 172.21.2.117:0/26758078 >> v2:172.21.2.125:3300/0 conn(0x7fc5100e51a0 msgr2=0x7fc5100d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 --2- 172.21.2.117:0/26758078 >> v2:172.21.2.125:3300/0 conn(0x7fc5100e51a0 0x7fc5100d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 -- 172.21.2.117:0/26758078 >> v1:172.21.2.125:6789/0 conn(0x7fc5100e4da0 legacy=0x7fc5101b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc507fff700 1 -- 172.21.2.117:0/26758078 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270430508 0 0) 0x7fc5000010c0 con 0x7fc5100e4da0 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 -- 172.21.2.117:0/26758078 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5101b08e0 con 0x7fc51007da20 2021-02-10T11:40:00.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.447 7fc5151cf700 1 --2- 172.21.2.117:0/26758078 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 0x7fc5100771b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc50c006650 tx=0x7fc50c009a80).ready entity=mon.2 client_cookie=cb541ab4cffd76ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc507fff700 1 -- 172.21.2.117:0/26758078 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50c0009b0 con 0x7fc51007da20 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc507fff700 1 -- 172.21.2.117:0/26758078 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c00c0c0 con 0x7fc51007da20 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc507fff700 1 -- 172.21.2.117:0/26758078 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50c00c520 con 0x7fc51007da20 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 -- 172.21.2.117:0/26758078 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 msgr2=0x7fc5100771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 --2- 172.21.2.117:0/26758078 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 0x7fc5100771b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fc50c006650 tx=0x7fc50c009a80).stop 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 -- 172.21.2.117:0/26758078 shutdown_connections 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 --2- 172.21.2.117:0/26758078 >> v2:172.21.2.125:3300/0 conn(0x7fc5100e51a0 0x7fc5100d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 --2- 172.21.2.117:0/26758078 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc51007da20 0x7fc5100771b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 -- 172.21.2.117:0/26758078 >> 172.21.2.117:0/26758078 conn(0x7fc5100ce450 msgr2=0x7fc5100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.448 7fc5173b3700 1 -- 172.21.2.117:0/26758078 shutdown_connections 2021-02-10T11:40:00.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.449 7fc5173b3700 1 -- 172.21.2.117:0/26758078 wait complete. 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 Processor -- start 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 -- start start 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5101079b0 0x7fc51009f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc510107db0 0x7fc51009f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51009ebc0 con 0x7fc51009e670 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5100a2300 con 0x7fc5101079b0 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.463 7fc5173b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc510107510 con 0x7fc510107db0 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47488/0 (socket says 172.21.2.117:47488) 2021-02-10T11:40:00.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 -- 172.21.2.117:0/3371625188 learned_addr learned my addr 172.21.2.117:0/3371625188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 -- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc510107db0 msgr2=0x7fc51009f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc510107db0 0x7fc51009f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 -- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5101079b0 msgr2=0x7fc51009f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5101079b0 0x7fc51009f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 -- 172.21.2.117:0/3371625188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51009fd80 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5151cf700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fc50c009630 tx=0x7fc50c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50c0093a0 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c00bd70 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51009fd80 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc50c005560 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.464 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5100a0310 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.465 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc50c0052e0 con 0x7fc51009e670 2021-02-10T11:40:00.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.465 7fc505ffb700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 0x7fc4f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:00.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.465 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc50c0323a0 con 0x7fc51009e670 2021-02-10T11:40:00.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.465 7fc5149ce700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 0x7fc4f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:00.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.466 7fc5149ce700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 0x7fc4f001abc0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:00.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.496 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510199d30 con 0x7fc51009e670 2021-02-10T11:40:00.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.499 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc50c02dee0 con 0x7fc51009e670 2021-02-10T11:40:00.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.628 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc510199d30 con 0x7fc51009e670 2021-02-10T11:40:00.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.628 7fc505ffb700 1 -- 172.21.2.117:0/3371625188 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v35) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc50c02dee0 con 0x7fc51009e670 2021-02-10T11:40:00.610 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 msgr2=0x7fc4f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 0x7fc4f001abc0 crc :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 msgr2=0x7fc5100a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fc50c009630 tx=0x7fc50c00b980).stop 2021-02-10T11:40:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 shutdown_connections 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc510107db0 0x7fc51009f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5101079b0 0x7fc51009f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc4f00186c0 0x7fc4f001abc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 --2- 172.21.2.117:0/3371625188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc51009e670 0x7fc5100a1ce0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.630 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 >> 172.21.2.117:0/3371625188 conn(0x7fc51008a180 msgr2=0x7fc5100b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.631 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 shutdown_connections 2021-02-10T11:40:00.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:00.631 7fc5173b3700 1 -- 172.21.2.117:0/3371625188 wait complete. 2021-02-10T11:40:00.620 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:40:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.047 7fa5751b1700 1 Processor -- start 2021-02-10T11:40:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.047 7fa5751b1700 1 -- start start 2021-02-10T11:40:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa5751b1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 0x7fa5700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa5751b1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5700e4d90 0x7fa5701b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa5751b1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa570077a70 con 0x7fa5700e5190 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa5751b1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5700e5c10 con 0x7fa5700e4d90 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa5751b1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700777d0 con 0x7fa57007da20 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa56f59e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa5700e5190 0x7fa570077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47746/0 (socket says 172.21.2.117:47746) 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa56f59e700 1 -- 172.21.2.117:0/2716110608 learned_addr learned my addr 172.21.2.117:0/2716110608 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.048 7fa56e59c700 1 --2- 172.21.2.117:0/2716110608 >> v2:172.21.2.117:3300/0 conn(0x7fa5700e4d90 0x7fa5701b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56dd9b700 1 -- 172.21.2.117:0/2716110608 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553555914 0 0) 0x7fa568001500 con 0x7fa5700e5190 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56ed9d700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 0x7fa5700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56dd9b700 1 -- 172.21.2.117:0/2716110608 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558002fe0 con 0x7fa5700e5190 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56e59c700 1 -- 172.21.2.117:0/2716110608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 msgr2=0x7fa5700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56e59c700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 0x7fa5700771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56e59c700 1 -- 172.21.2.117:0/2716110608 >> v1:172.21.2.117:6789/0 conn(0x7fa5700e5190 legacy=0x7fa570077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56e59c700 1 -- 172.21.2.117:0/2716110608 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57007de20 con 0x7fa5700e4d90 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56ed9d700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 0x7fa5700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56e59c700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5700e4d90 0x7fa5701b00e0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fa564006650 tx=0x7fa564009a80).ready entity=mon.0 client_cookie=859692f82abdb842 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56dd9b700 1 -- 172.21.2.117:0/2716110608 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5640009b0 con 0x7fa5700e4d90 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56dd9b700 1 -- 172.21.2.117:0/2716110608 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56400c0c0 con 0x7fa5700e4d90 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa56dd9b700 1 -- 172.21.2.117:0/2716110608 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa56400c5e0 con 0x7fa5700e4d90 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa5751b1700 1 -- 172.21.2.117:0/2716110608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5700e4d90 msgr2=0x7fa5701b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.049 7fa5751b1700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5700e4d90 0x7fa5701b00e0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7fa564006650 tx=0x7fa564009a80).stop 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 -- 172.21.2.117:0/2716110608 shutdown_connections 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5700e4d90 0x7fa5701b00e0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 --2- 172.21.2.117:0/2716110608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57007da20 0x7fa5700771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 -- 172.21.2.117:0/2716110608 >> 172.21.2.117:0/2716110608 conn(0x7fa5700ce440 msgr2=0x7fa5700e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 -- 172.21.2.117:0/2716110608 shutdown_connections 2021-02-10T11:40:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.050 7fa5751b1700 1 -- 172.21.2.117:0/2716110608 wait complete. 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 Processor -- start 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 -- start start 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 0x7fa5700b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57008c080 0x7fa57008c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57009d8a0 con 0x7fa57008caa0 2021-02-10T11:40:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700b8bc0 con 0x7fa57009dd40 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa5751b1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700b8de0 con 0x7fa57008c080 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa56f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa56f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47504/0 (socket says 172.21.2.117:47504) 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.065 7fa56f59e700 1 -- 172.21.2.117:0/3739000457 learned_addr learned my addr 172.21.2.117:0/3739000457 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56ed9d700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 0x7fa5700b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 -- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57008c080 msgr2=0x7fa57008c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57008c080 0x7fa57008c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 -- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 msgr2=0x7fa5700b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 0x7fa5700b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 -- 172.21.2.117:0/3739000457 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57008cea0 con 0x7fa57008caa0 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56ed9d700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 0x7fa5700b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:01.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa56f59e700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fa568001c10 tx=0x7fa56800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa568007ce0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5680070d0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57008cea0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57008cea0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.066 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa568004930 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.067 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5680072b0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.067 7fa55f7fe700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 0x7fa55401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.067 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fa5680329a0 con 0x7fa57008caa0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.067 7fa56ed9d700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 0x7fa55401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.068 7fa56ed9d700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 0x7fa55401ac10 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.098 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa570077a70 con 0x7fa57008caa0 2021-02-10T11:40:01.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.100 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5680072e0 con 0x7fa57008caa0 2021-02-10T11:40:01.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.225 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa570077a70 con 0x7fa57008caa0 2021-02-10T11:40:01.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.225 7fa55f7fe700 1 -- 172.21.2.117:0/3739000457 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+64 (secure 0 0 0) 0x7fa5680070b0 con 0x7fa57008caa0 2021-02-10T11:40:01.206 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:01.207 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:40:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 msgr2=0x7fa55401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 0x7fa55401ac10 crc :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 msgr2=0x7fa57008f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7fa568001c10 tx=0x7fa56800aca0).stop 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 shutdown_connections 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa554018730 0x7fa55401ac10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa57008caa0 0x7fa57008f5c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa57008c080 0x7fa57008c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 --2- 172.21.2.117:0/3739000457 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa57009dd40 0x7fa5700b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.227 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 >> 172.21.2.117:0/3739000457 conn(0x7fa57019f4c0 msgr2=0x7fa570094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.228 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 shutdown_connections 2021-02-10T11:40:01.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.228 7fa5751b1700 1 -- 172.21.2.117:0/3739000457 wait complete. 2021-02-10T11:40:01.216 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:40:01.216 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 Processor -- start 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 -- start start 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 0x7efde00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 0x7efde00d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde0077a70 con 0x7efde00e4da0 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00e5c20 con 0x7efde00e51a0 2021-02-10T11:40:01.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.651 7efde8497700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00777d0 con 0x7efde007da20 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.652 7efde6ab4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 0x7efde00d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.652 7efde62b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 0x7efde00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.652 7efde5ab2700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7efde00e4da0 0x7efde01b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40684/0 (socket says 172.21.2.117:40684) 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.652 7efde5ab2700 1 -- 172.21.2.117:0/2985437737 learned_addr learned my addr 172.21.2.117:0/2985437737 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde62b3700 1 -- 172.21.2.117:0/2985437737 >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 msgr2=0x7efde00d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde62b3700 1 --2- 172.21.2.117:0/2985437737 >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 0x7efde00d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde62b3700 1 -- 172.21.2.117:0/2985437737 >> v1:172.21.2.125:6789/0 conn(0x7efde00e4da0 legacy=0x7efde01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde62b3700 1 -- 172.21.2.117:0/2985437737 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde01b08e0 con 0x7efde007da20 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde6ab4700 1 --2- 172.21.2.117:0/2985437737 >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 0x7efde00d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.653 7efde62b3700 1 --2- 172.21.2.117:0/2985437737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 0x7efde00771b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7efddc006650 tx=0x7efddc009a80).ready entity=mon.2 client_cookie=966469e35b386c12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.654 7efde52b1700 1 -- 172.21.2.117:0/2985437737 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efddc0009b0 con 0x7efde007da20 2021-02-10T11:40:01.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.654 7efde52b1700 1 -- 172.21.2.117:0/2985437737 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc00c0c0 con 0x7efde007da20 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.654 7efde52b1700 1 -- 172.21.2.117:0/2985437737 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efddc00c5e0 con 0x7efde007da20 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.654 7efde8497700 1 -- 172.21.2.117:0/2985437737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 msgr2=0x7efde00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.654 7efde8497700 1 --2- 172.21.2.117:0/2985437737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 0x7efde00771b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7efddc006650 tx=0x7efddc009a80).stop 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 -- 172.21.2.117:0/2985437737 shutdown_connections 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 --2- 172.21.2.117:0/2985437737 >> v2:172.21.2.125:3300/0 conn(0x7efde00e51a0 0x7efde00d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 --2- 172.21.2.117:0/2985437737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde007da20 0x7efde00771b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 -- 172.21.2.117:0/2985437737 >> 172.21.2.117:0/2985437737 conn(0x7efde00ce450 msgr2=0x7efde00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 -- 172.21.2.117:0/2985437737 shutdown_connections 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.655 7efde8497700 1 -- 172.21.2.117:0/2985437737 wait complete. 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 Processor -- start 2021-02-10T11:40:01.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 -- start start 2021-02-10T11:40:01.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde00a41c0 0x7efde00fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 0x7efde00c2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b8010 con 0x7efde00b7390 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b8550 con 0x7efde00c2340 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.669 7efde8497700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00b8770 con 0x7efde00a41c0 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47516/0 (socket says 172.21.2.117:47516) 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 -- 172.21.2.117:0/2440990086 learned_addr learned my addr 172.21.2.117:0/2440990086 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde5ab2700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 0x7efde00c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde62b3700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde00a41c0 0x7efde00fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 -- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde00a41c0 msgr2=0x7efde00fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde00a41c0 0x7efde00fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 -- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 msgr2=0x7efde00c2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 0x7efde00c2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:01.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 -- 172.21.2.117:0/2440990086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00b8990 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde5ab2700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 0x7efde00c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.670 7efde6ab4700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7efdd800b970 tx=0x7efdd800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd8000f90 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd8006790 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdd80015f0 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efde8497700 1 -- 172.21.2.117:0/2440990086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde00b8990 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efde8497700 1 -- 172.21.2.117:0/2440990086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde00b8f20 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efdd80011f0 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 0x7efdc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.671 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5100+0+0 (secure 0 0 0) 0x7efdd80327d0 con 0x7efde00b7390 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.672 7efde62b3700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 0x7efdc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:01.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.672 7efde62b3700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 0x7efdc801abc0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:01.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.701 7efde8497700 1 -- 172.21.2.117:0/2440990086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde01b4f00 con 0x7efde00b7390 2021-02-10T11:40:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.703 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efdd8000f90 con 0x7efde00b7390 2021-02-10T11:40:01.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:01.830 7efde8497700 1 -- 172.21.2.117:0/2440990086 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7efde0077a70 con 0x7efde00b7390 2021-02-10T11:40:02.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.261 7efdd2ffd700 1 -- 172.21.2.117:0/2440990086 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7efdd80011f0 con 0x7efde00b7390 2021-02-10T11:40:02.277 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.298 7efde8497700 1 -- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 msgr2=0x7efdc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.298 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 0x7efdc801abc0 crc :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.298 7efde8497700 1 -- 172.21.2.117:0/2440990086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 msgr2=0x7efde00b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.298 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7efdd800b970 tx=0x7efdd800bd10).stop 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 -- 172.21.2.117:0/2440990086 shutdown_connections 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdc80186c0 0x7efdc801abc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efde00b7390 0x7efde00b7790 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efde00c2340 0x7efde00c2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 --2- 172.21.2.117:0/2440990086 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efde00a41c0 0x7efde00fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 -- 172.21.2.117:0/2440990086 >> 172.21.2.117:0/2440990086 conn(0x7efde008bbc0 msgr2=0x7efde00c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:02.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 -- 172.21.2.117:0/2440990086 shutdown_connections 2021-02-10T11:40:02.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.299 7efde8497700 1 -- 172.21.2.117:0/2440990086 wait complete. 2021-02-10T11:40:02.288 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:40:02.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 Processor -- start 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 -- start start 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66a8079000 0x7f66a80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f66a81b00f0 0x7f66a81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8077a70 con 0x7f66a807da20 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a80e5c20 con 0x7f66a8079000 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.727 7f66acce5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a80e53c0 con 0x7f66a81b00f0 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66a8079000 0x7f66a80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f66a8079000 0x7f66a80e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47520/0 (socket says 172.21.2.117:47520) 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a6ae8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f66a807da20 0x7f66a80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47770/0 (socket says 172.21.2.117:47770) 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 -- 172.21.2.117:0/834097046 learned_addr learned my addr 172.21.2.117:0/834097046 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:02.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 -- 172.21.2.117:0/834097046 >> v2:172.21.2.125:3300/0 conn(0x7f66a81b00f0 msgr2=0x7f66a81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 --2- 172.21.2.117:0/834097046 >> v2:172.21.2.125:3300/0 conn(0x7f66a81b00f0 0x7f66a81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 -- 172.21.2.117:0/834097046 >> v1:172.21.2.117:6789/0 conn(0x7f66a807da20 legacy=0x7f66a80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a5ae6700 1 -- 172.21.2.117:0/834097046 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972445005 0 0) 0x7f6690000ce0 con 0x7f66a807da20 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 -- 172.21.2.117:0/834097046 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8079400 con 0x7f66a8079000 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.728 7f66a62e7700 1 --2- 172.21.2.117:0/834097046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a8079000 0x7f66a80e4da0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7f6698006700 tx=0x7f6698009b00).ready entity=mon.0 client_cookie=ca0bcb457070b254 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66a5ae6700 1 -- 172.21.2.117:0/834097046 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f669800c0c0 con 0x7f66a8079000 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66a5ae6700 1 -- 172.21.2.117:0/834097046 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669800c400 con 0x7f66a8079000 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66a5ae6700 1 -- 172.21.2.117:0/834097046 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f669800c920 con 0x7f66a8079000 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 -- 172.21.2.117:0/834097046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a8079000 msgr2=0x7f66a80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 --2- 172.21.2.117:0/834097046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a8079000 0x7f66a80e4da0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7f6698006700 tx=0x7f6698009b00).stop 2021-02-10T11:40:02.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 -- 172.21.2.117:0/834097046 shutdown_connections 2021-02-10T11:40:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 --2- 172.21.2.117:0/834097046 >> v2:172.21.2.125:3300/0 conn(0x7f66a81b00f0 0x7f66a81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 --2- 172.21.2.117:0/834097046 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a8079000 0x7f66a80e4da0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.729 7f66acce5700 1 -- 172.21.2.117:0/834097046 >> 172.21.2.117:0/834097046 conn(0x7f66a80ce450 msgr2=0x7f66a80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.730 7f66acce5700 1 -- 172.21.2.117:0/834097046 shutdown_connections 2021-02-10T11:40:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.730 7f66acce5700 1 -- 172.21.2.117:0/834097046 wait complete. 2021-02-10T11:40:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.744 7f66acce5700 1 Processor -- start 2021-02-10T11:40:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.744 7f66acce5700 1 -- start start 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a81a7ca0 0x7f66a818de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a818e480 0x7f66a818e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80b8960 con 0x7f66a80a40a0 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a818eea0 con 0x7f66a81a7ca0 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66acce5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a818f0c0 con 0x7f66a818e480 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a6ae8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a6ae8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47526/0 (socket says 172.21.2.117:47526) 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a6ae8700 1 -- 172.21.2.117:0/1826811984 learned_addr learned my addr 172.21.2.117:0/1826811984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a72e9700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a818e480 0x7f66a818e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a62e7700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a81a7ca0 0x7f66a818de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a6ae8700 1 -- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a818e480 msgr2=0x7f66a818e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.745 7f66a6ae8700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a818e480 0x7f66a818e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66a6ae8700 1 -- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a81a7ca0 msgr2=0x7f66a818de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66a6ae8700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a81a7ca0 0x7f66a818de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66a6ae8700 1 -- 172.21.2.117:0/1826811984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a80b6fc0 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66a6ae8700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7f66900014d0 tx=0x7f669000a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6690000ef0 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669000ac90 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6690004620 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66acce5700 1 -- 172.21.2.117:0/1826811984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a80b6fc0 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.746 7f66acce5700 1 -- 172.21.2.117:0/1826811984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a80b7550 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.747 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66900043a0 con 0x7f66a80a40a0 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.747 7f669f7fe700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 0x7f668c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.747 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f6690032510 con 0x7f66a80a40a0 2021-02-10T11:40:02.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.747 7f66a62e7700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 0x7f668c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:02.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.748 7f66a62e7700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 0x7f668c01abc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:02.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.777 7f66acce5700 1 -- 172.21.2.117:0/1826811984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a8103370 con 0x7f66a80a40a0 2021-02-10T11:40:02.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.780 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6690000ef0 con 0x7f66a80a40a0 2021-02-10T11:40:02.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:02.910 7f66acce5700 1 -- 172.21.2.117:0/1826811984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f66a8077a70 con 0x7f66a80a40a0 2021-02-10T11:40:03.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.267 7f669f7fe700 1 -- 172.21.2.117:0/1826811984 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v37) v1 ==== 125+0+0 (secure 0 0 0) 0x7f669001c050 con 0x7f66a80a40a0 2021-02-10T11:40:03.256 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 -- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 msgr2=0x7f668c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 0x7f668c01abc0 crc :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 -- 172.21.2.117:0/1826811984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 msgr2=0x7f66a81a7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7f66900014d0 tx=0x7f669000a8a0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 -- 172.21.2.117:0/1826811984 shutdown_connections 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66a818e480 0x7f66a818e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66a81a7ca0 0x7f66a818de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f668c0186c0 0x7f668c01abc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 --2- 172.21.2.117:0/1826811984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66a80a40a0 0x7f66a81a7680 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 -- 172.21.2.117:0/1826811984 >> 172.21.2.117:0/1826811984 conn(0x7f66a808a180 msgr2=0x7f66a8096ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.277 7f66acce5700 1 -- 172.21.2.117:0/1826811984 shutdown_connections 2021-02-10T11:40:03.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.278 7f66acce5700 1 -- 172.21.2.117:0/1826811984 wait complete. 2021-02-10T11:40:03.267 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:40:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.699 7f6aef151700 1 Processor -- start 2021-02-10T11:40:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.699 7f6aef151700 1 -- start start 2021-02-10T11:40:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6aef151700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6ae807da20 0x7f6ae80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6aef151700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae8077a10 con 0x7f6ae80e4e00 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6aef151700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae81b0920 con 0x7f6ae80e5200 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6aef151700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae80777d0 con 0x7f6ae807da20 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6ae7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6ae80e4e00 0x7f6ae81b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47786/0 (socket says 172.21.2.117:47786) 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6ae7fff700 1 -- 172.21.2.117:0/1793375315 learned_addr learned my addr 172.21.2.117:0/1793375315 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.700 7f6aecf6d700 1 --2- 172.21.2.117:0/1793375315 >> v2:172.21.2.117:3300/0 conn(0x7f6ae807da20 0x7f6ae80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6ae77fe700 1 -- 172.21.2.117:0/1793375315 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521996431 0 0) 0x7f6ae0001400 con 0x7f6ae80e5200 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aecf6d700 1 -- 172.21.2.117:0/1793375315 >> v1:172.21.2.125:6789/0 conn(0x7f6ae80e5200 legacy=0x7f6ae80d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aecf6d700 1 -- 172.21.2.117:0/1793375315 >> v1:172.21.2.117:6789/0 conn(0x7f6ae80e4e00 legacy=0x7f6ae81b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aecf6d700 1 -- 172.21.2.117:0/1793375315 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80d1d50 con 0x7f6ae807da20 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aecf6d700 1 --2- 172.21.2.117:0/1793375315 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae807da20 0x7f6ae80771b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f6ad0006650 tx=0x7f6ad0009a80).ready entity=mon.0 client_cookie=a34b1ae3a27ae4f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:03.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6ae77fe700 1 -- 172.21.2.117:0/1793375315 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad00009b0 con 0x7f6ae807da20 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6ae77fe700 1 -- 172.21.2.117:0/1793375315 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad000c0c0 con 0x7f6ae807da20 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6ae77fe700 1 -- 172.21.2.117:0/1793375315 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad000c520 con 0x7f6ae807da20 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aef151700 1 -- 172.21.2.117:0/1793375315 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae807da20 msgr2=0x7f6ae80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.701 7f6aef151700 1 --2- 172.21.2.117:0/1793375315 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae807da20 0x7f6ae80771b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f6ad0006650 tx=0x7f6ad0009a80).stop 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.702 7f6aef151700 1 -- 172.21.2.117:0/1793375315 shutdown_connections 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.702 7f6aef151700 1 --2- 172.21.2.117:0/1793375315 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae807da20 0x7f6ae80771b0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.702 7f6aef151700 1 -- 172.21.2.117:0/1793375315 >> 172.21.2.117:0/1793375315 conn(0x7f6ae80ce480 msgr2=0x7f6ae80e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.702 7f6aef151700 1 -- 172.21.2.117:0/1793375315 shutdown_connections 2021-02-10T11:40:03.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.702 7f6aef151700 1 -- 172.21.2.117:0/1793375315 wait complete. 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 Processor -- start 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 -- start start 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ae81a36c0 0x7f6ae80b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ae81a3ac0 0x7f6ae80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b43e0 con 0x7f6ae80b3f60 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b53e0 con 0x7f6ae81a3ac0 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.717 7f6aef151700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b5600 con 0x7f6ae81a36c0 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47542/0 (socket says 172.21.2.117:47542) 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 -- 172.21.2.117:0/1947222870 learned_addr learned my addr 172.21.2.117:0/1947222870 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:03.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6ae7fff700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ae81a36c0 0x7f6ae80b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:03.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aed76e700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ae81a3ac0 0x7f6ae80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:03.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 -- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ae81a36c0 msgr2=0x7f6ae80b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ae81a36c0 0x7f6ae80b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 -- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ae81a3ac0 msgr2=0x7f6ae80b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ae81a3ac0 0x7f6ae80b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 -- 172.21.2.117:0/1947222870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80b5820 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aecf6d700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f6ad0009630 tx=0x7f6ad000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad00093a0 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad000bd70 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad00054d0 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aef151700 1 -- 172.21.2.117:0/1947222870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae80b5820 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.718 7f6aef151700 1 -- 172.21.2.117:0/1947222870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae8190df0 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.719 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ad000a000 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.719 7f6ae57fa700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 0x7f6acc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.719 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f6ad0032620 con 0x7f6ae80b3f60 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.719 7f6ae7fff700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 0x7f6acc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:03.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.720 7f6ae7fff700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 0x7f6acc01abc0 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:03.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.750 7f6aef151700 1 -- 172.21.2.117:0/1947222870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae8101030 con 0x7f6ae80b3f60 2021-02-10T11:40:03.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.752 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ad0009340 con 0x7f6ae80b3f60 2021-02-10T11:40:03.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.880 7f6aef151700 1 -- 172.21.2.117:0/1947222870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f6ae8190db0 con 0x7f6ae80b3f60 2021-02-10T11:40:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.883 7f6ae57fa700 1 -- 172.21.2.117:0/1947222870 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 6 and data pool 7 v17) v1 ==== 187+0+0 (secure 0 0 0) 0x7f6ad000a000 con 0x7f6ae80b3f60 2021-02-10T11:40:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 6 and data pool 7 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 msgr2=0x7f6acc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 0x7f6acc01abc0 crc :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 msgr2=0x7f6ae81a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f6ad0009630 tx=0x7f6ad000b980).stop 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 shutdown_connections 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ae81a3ac0 0x7f6ae80b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ae81a36c0 0x7f6ae80b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6acc0186c0 0x7f6acc01abc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 --2- 172.21.2.117:0/1947222870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ae80b3f60 0x7f6ae81a30a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:03.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 >> 172.21.2.117:0/1947222870 conn(0x7f6ae80a19c0 msgr2=0x7f6ae8096870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:03.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 shutdown_connections 2021-02-10T11:40:03.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:03.889 7f6aef151700 1 -- 172.21.2.117:0/1947222870 wait complete. 2021-02-10T11:40:03.878 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:04.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.304 7f2383eae700 1 Processor -- start 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.304 7f2383eae700 1 -- start start 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2383eae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f237c07da20 0x7f237c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2383eae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c0e4d90 0x7f237c1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2383eae700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c077a70 con 0x7f237c0e5190 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2383eae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0777d0 con 0x7f237c07da20 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2383eae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c07de20 con 0x7f237c0e4d90 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f237c07da20 0x7f237c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f237c07da20 0x7f237c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47548/0 (socket says 172.21.2.117:47548) 2021-02-10T11:40:04.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 -- 172.21.2.117:0/167655825 learned_addr learned my addr 172.21.2.117:0/167655825 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 -- 172.21.2.117:0/167655825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c0e4d90 msgr2=0x7f237c1b00e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 --2- 172.21.2.117:0/167655825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c0e4d90 0x7f237c1b00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 -- 172.21.2.117:0/167655825 >> v1:172.21.2.125:6789/0 conn(0x7f237c0e5190 legacy=0x7f237c077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.305 7f2381cca700 1 -- 172.21.2.117:0/167655825 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c1b0700 con 0x7f237c07da20 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2381cca700 1 --2- 172.21.2.117:0/167655825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c07da20 0x7f237c0771b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f2370006700 tx=0x7f2370009b00).ready entity=mon.0 client_cookie=50519350f5faf1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2380cc8700 1 -- 172.21.2.117:0/167655825 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237000c0c0 con 0x7f237c07da20 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2380cc8700 1 -- 172.21.2.117:0/167655825 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237000c400 con 0x7f237c07da20 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2380cc8700 1 -- 172.21.2.117:0/167655825 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237000c920 con 0x7f237c07da20 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 -- 172.21.2.117:0/167655825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c07da20 msgr2=0x7f237c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 --2- 172.21.2.117:0/167655825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c07da20 0x7f237c0771b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f2370006700 tx=0x7f2370009b00).stop 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 -- 172.21.2.117:0/167655825 shutdown_connections 2021-02-10T11:40:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 --2- 172.21.2.117:0/167655825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c0e4d90 0x7f237c1b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 --2- 172.21.2.117:0/167655825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c07da20 0x7f237c0771b0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.306 7f2383eae700 1 -- 172.21.2.117:0/167655825 >> 172.21.2.117:0/167655825 conn(0x7f237c0ce440 msgr2=0x7f237c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.307 7f2383eae700 1 -- 172.21.2.117:0/167655825 shutdown_connections 2021-02-10T11:40:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.307 7f2383eae700 1 -- 172.21.2.117:0/167655825 wait complete. 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 Processor -- start 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 -- start start 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c109bf0 0x7f237c109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f237c09da70 0x7f237c09de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c10a610 con 0x7f237c19d3b0 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c09e490 con 0x7f237c09da70 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.321 7f2383eae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c09e6b0 con 0x7f237c109bf0 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47554/0 (socket says 172.21.2.117:47554) 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 -- 172.21.2.117:0/3566928144 learned_addr learned my addr 172.21.2.117:0/3566928144 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 -- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c109bf0 msgr2=0x7f237c109ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:04.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c109bf0 0x7f237c109ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 -- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f237c09da70 msgr2=0x7f237c09de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f237c09da70 0x7f237c09de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 -- 172.21.2.117:0/3566928144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c09e8d0 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.322 7f2381cca700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f237000b590 tx=0x7f237000b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2370006390 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2370000c90 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23700022b0 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f2383eae700 1 -- 172.21.2.117:0/3566928144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c09e8d0 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f2383eae700 1 -- 172.21.2.117:0/3566928144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c09e8d0 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23700093e0 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 0x7f236401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.323 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2370032710 con 0x7f237c19d3b0 2021-02-10T11:40:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.324 7f23814c9700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 0x7f236401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.325 7f23814c9700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 0x7f236401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.355 7f2383eae700 1 -- 172.21.2.117:0/3566928144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c108ba0 con 0x7f237c19d3b0 2021-02-10T11:40:04.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.358 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f23700090c0 con 0x7f237c19d3b0 2021-02-10T11:40:04.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.481 7f2383eae700 1 -- 172.21.2.117:0/3566928144 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f237c077a70 con 0x7f237c19d3b0 2021-02-10T11:40:04.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.482 7f236e7fc700 1 -- 172.21.2.117:0/3566928144 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f23700093e0 con 0x7f237c19d3b0 2021-02-10T11:40:04.464 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:04.465 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":38,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:03.880946","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:01.847956","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:02.928003","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 msgr2=0x7f236401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 0x7f236401abc0 crc :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 msgr2=0x7f237c19d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7f237000b590 tx=0x7f237000b930).stop 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 shutdown_connections 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f237c09da70 0x7f237c09de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f237c109bf0 0x7f237c109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23640186c0 0x7f236401abc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 --2- 172.21.2.117:0/3566928144 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f237c19d3b0 0x7f237c19d7b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 >> 172.21.2.117:0/3566928144 conn(0x7f237c19a700 msgr2=0x7f237c199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 shutdown_connections 2021-02-10T11:40:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.484 7f2383eae700 1 -- 172.21.2.117:0/3566928144 wait complete. 2021-02-10T11:40:04.475 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:04.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 Processor -- start 2021-02-10T11:40:04.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 -- start start 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feefc079000 0x7feefc0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc1b00e0 0x7feefc1b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feefc077a70 con 0x7feefc07da20 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feefc0e53b0 con 0x7feefc079000 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7fef01808700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc079400 con 0x7feefc1b00e0 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.901 7feefad73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feefc079000 0x7feefc0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feefc079000 0x7feefc0e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47560/0 (socket says 172.21.2.117:47560) 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 -- 172.21.2.117:0/64750989 learned_addr learned my addr 172.21.2.117:0/64750989 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefbd75700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc1b00e0 0x7feefc1b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 -- 172.21.2.117:0/64750989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc1b00e0 msgr2=0x7feefc1b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc1b00e0 0x7feefc1b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 -- 172.21.2.117:0/64750989 >> v1:172.21.2.125:6789/0 conn(0x7feefc07da20 legacy=0x7feefc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 -- 172.21.2.117:0/64750989 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc07de20 con 0x7feefc079000 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefad73700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc079000 0x7feefc0e4d90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7feee4006700 tx=0x7feee4009b80).ready entity=mon.0 client_cookie=77346203434de208 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefa572700 1 -- 172.21.2.117:0/64750989 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feee400c0c0 con 0x7feefc079000 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefa572700 1 -- 172.21.2.117:0/64750989 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee400c400 con 0x7feefc079000 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7feefa572700 1 -- 172.21.2.117:0/64750989 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feee400c8e0 con 0x7feefc079000 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 -- 172.21.2.117:0/64750989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc079000 msgr2=0x7feefc0e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc079000 0x7feefc0e4d90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7feee4006700 tx=0x7feee4009b80).stop 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 -- 172.21.2.117:0/64750989 shutdown_connections 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc1b00e0 0x7feefc1b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 --2- 172.21.2.117:0/64750989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc079000 0x7feefc0e4d90 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.902 7fef01808700 1 -- 172.21.2.117:0/64750989 >> 172.21.2.117:0/64750989 conn(0x7feefc0ce440 msgr2=0x7feefc0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.903 7fef01808700 1 -- 172.21.2.117:0/64750989 shutdown_connections 2021-02-10T11:40:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.903 7fef01808700 1 -- 172.21.2.117:0/64750989 wait complete. 2021-02-10T11:40:04.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 Processor -- start 2021-02-10T11:40:04.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 -- start start 2021-02-10T11:40:04.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 0x7feefc18e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 0x7feefc18f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc18d930 con 0x7feefc18d530 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc18e230 con 0x7feefc18e4f0 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7fef01808700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc192f30 con 0x7feefc18ef10 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7feefb574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7feefb574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47570/0 (socket says 172.21.2.117:47570) 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.917 7feefb574700 1 -- 172.21.2.117:0/847357126 learned_addr learned my addr 172.21.2.117:0/847357126 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 0x7feefc18f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefad73700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 0x7feefc18e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 -- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 msgr2=0x7feefc18e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 0x7feefc18e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 -- 172.21.2.117:0/847357126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 msgr2=0x7feefc18dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:04.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 -- 172.21.2.117:0/847357126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc193150 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefb574700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefad73700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 0x7feefc18e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feefbd75700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 0x7feefc18f310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7feef4009fc0 tx=0x7feef400bc90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef4000f90 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef4003030 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feef40014a0 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.918 7fef01808700 1 -- 172.21.2.117:0/847357126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc193150 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.919 7fef01808700 1 -- 172.21.2.117:0/847357126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc193150 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.919 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feef4006020 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.919 7feef3fff700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 0x7feee001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.919 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feef4032a70 con 0x7feefc18ef10 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.919 7feefb574700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 0x7feee001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:04.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.920 7feefb574700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 0x7feee001abd0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:04.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.951 7fef01808700 1 -- 172.21.2.117:0/847357126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feefc18d930 con 0x7feefc18ef10 2021-02-10T11:40:04.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:04.955 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feef4006ac0 con 0x7feefc18ef10 2021-02-10T11:40:05.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.082 7fef01808700 1 -- 172.21.2.117:0/847357126 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feefc18d930 con 0x7feefc18ef10 2021-02-10T11:40:05.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.083 7feef3fff700 1 -- 172.21.2.117:0/847357126 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+14226 (secure 0 0 0) 0x7feef4006000 con 0x7feefc18ef10 2021-02-10T11:40:05.067 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:05.067 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":39,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:04.882873","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:01.847956","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:02.928003","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:05.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 -- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 msgr2=0x7feee001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 0x7feee001abd0 crc :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 -- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 msgr2=0x7feefc18f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 0x7feefc18f310 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7feef4009fc0 tx=0x7feef400bc90).stop 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 -- 172.21.2.117:0/847357126 shutdown_connections 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.086 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feefc18ef10 0x7feefc18f310 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feefc18e4f0 0x7feefc18e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feee00186d0 0x7feee001abd0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 --2- 172.21.2.117:0/847357126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feefc18d530 0x7feefc18dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 -- 172.21.2.117:0/847357126 >> 172.21.2.117:0/847357126 conn(0x7feefc1ace70 msgr2=0x7feefc1af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 -- 172.21.2.117:0/847357126 shutdown_connections 2021-02-10T11:40:05.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.087 7fef01808700 1 -- 172.21.2.117:0/847357126 wait complete. 2021-02-10T11:40:05.078 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:40:05.317 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:05.320 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.508 7f699e3f9700 1 Processor -- start 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.508 7f699e3f9700 1 -- start start 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699e3f9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6998079000 0x7f69980e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699e3f9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69980d1d20 con 0x7f69981b00f0 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699e3f9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6998077a70 con 0x7f699807da20 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699e3f9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69980e53c0 con 0x7f6998079000 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699ca16700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f69981b00f0 0x7f69980d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47826/0 (socket says 172.21.2.117:47826) 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f699ca16700 1 -- 172.21.2.117:0/407415151 learned_addr learned my addr 172.21.2.117:0/407415151 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:05.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f69977fe700 1 --2- 172.21.2.117:0/407415151 >> v2:172.21.2.117:3300/0 conn(0x7f6998079000 0x7f69980e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f69977fe700 1 -- 172.21.2.117:0/407415151 >> v1:172.21.2.125:6789/0 conn(0x7f699807da20 legacy=0x7f69980771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f69977fe700 1 -- 172.21.2.117:0/407415151 >> v1:172.21.2.117:6789/0 conn(0x7f69981b00f0 legacy=0x7f69980d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f69977fe700 1 -- 172.21.2.117:0/407415151 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6998079400 con 0x7f6998079000 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.509 7f6996ffd700 1 -- 172.21.2.117:0/407415151 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717275458 0 0) 0x7f6990001500 con 0x7f69981b00f0 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f69977fe700 1 --2- 172.21.2.117:0/407415151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6998079000 0x7f69980e4da0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f698c006650 tx=0x7f698c009a00).ready entity=mon.0 client_cookie=2490dc7b32b3555c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f6996ffd700 1 -- 172.21.2.117:0/407415151 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698c009df0 con 0x7f6998079000 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f6996ffd700 1 -- 172.21.2.117:0/407415151 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c0009b0 con 0x7f6998079000 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f6996ffd700 1 -- 172.21.2.117:0/407415151 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698c00c2b0 con 0x7f6998079000 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 -- 172.21.2.117:0/407415151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6998079000 msgr2=0x7f69980e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 --2- 172.21.2.117:0/407415151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6998079000 0x7f69980e4da0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f698c006650 tx=0x7f698c009a00).stop 2021-02-10T11:40:05.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 -- 172.21.2.117:0/407415151 shutdown_connections 2021-02-10T11:40:05.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 --2- 172.21.2.117:0/407415151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6998079000 0x7f69980e4da0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 -- 172.21.2.117:0/407415151 >> 172.21.2.117:0/407415151 conn(0x7f69980ce450 msgr2=0x7f69980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:05.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 -- 172.21.2.117:0/407415151 shutdown_connections 2021-02-10T11:40:05.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.510 7f699e3f9700 1 -- 172.21.2.117:0/407415151 wait complete. 2021-02-10T11:40:05.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.525 7f699e3f9700 1 Processor -- start 2021-02-10T11:40:05.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.525 7f699e3f9700 1 -- start start 2021-02-10T11:40:05.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.525 7f699e3f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:05.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.525 7f699e3f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f699809cb80 0x7f69980c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.525 7f699e3f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69980b4250 0x7f69980c0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f699e3f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69980bff40 con 0x7f699809c780 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f699e3f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69980bfb60 con 0x7f69980b4250 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f699e3f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69980bfd40 con 0x7f699809cb80 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47584/0 (socket says 172.21.2.117:47584) 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 -- 172.21.2.117:0/1846875900 learned_addr learned my addr 172.21.2.117:0/1846875900 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 -- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f699809cb80 msgr2=0x7f69980c04c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f699809cb80 0x7f69980c04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 -- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69980b4250 msgr2=0x7f69980c0ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69980b4250 0x7f69980c0ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 -- 172.21.2.117:0/1846875900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980c1100 con 0x7f699809c780 2021-02-10T11:40:05.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6997fff700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f698800b5b0 tx=0x7f698800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698800bd40 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6988006240 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.526 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69980c1100 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.527 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69980c1100 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.527 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6988005690 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.527 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69880054a0 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.527 7f6994ff9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 0x7f697801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.527 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6988032950 con 0x7f699809c780 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.528 7f69977fe700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 0x7f697801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:05.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.528 7f69977fe700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 0x7f697801abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:05.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.558 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69980ff170 con 0x7f699809c780 2021-02-10T11:40:05.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.561 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69880009b0 con 0x7f699809c780 2021-02-10T11:40:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.700 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f6998077a70 con 0x7f699809c780 2021-02-10T11:40:05.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.889 7f6994ff9700 1 -- 172.21.2.117:0/1846875900 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v20) v1 ==== 118+0+0 (secure 0 0 0) 0x7f69880054a0 con 0x7f699809c780 2021-02-10T11:40:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.917 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 msgr2=0x7f697801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.917 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 0x7f697801abc0 crc :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.917 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 msgr2=0x7f69980b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.917 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f698800b5b0 tx=0x7f698800b950).stop 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 shutdown_connections 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69980b4250 0x7f69980c0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f699809cb80 0x7f69980c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69780186c0 0x7f697801abc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 --2- 172.21.2.117:0/1846875900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f699809c780 0x7f69980b3c30 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 >> 172.21.2.117:0/1846875900 conn(0x7f699809d3b0 msgr2=0x7f69980b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 shutdown_connections 2021-02-10T11:40:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:05.918 7f699e3f9700 1 -- 172.21.2.117:0/1846875900 wait complete. 2021-02-10T11:40:05.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.336 7fdaf2447700 1 Processor -- start 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.336 7fdaf2447700 1 -- start start 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf2447700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdaec079000 0x7fdaec0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf2447700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0d1d10 con 0x7fdaec1b00e0 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf2447700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec077a70 con 0x7fdaec07da20 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf2447700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec0e53b0 con 0x7fdaec079000 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf0a64700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdaec1b00e0 0x7fdaec0d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47842/0 (socket says 172.21.2.117:47842) 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaf0a64700 1 -- 172.21.2.117:0/2151431914 learned_addr learned my addr 172.21.2.117:0/2151431914 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:06.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeb7fe700 1 --2- 172.21.2.117:0/2151431914 >> v2:172.21.2.117:3300/0 conn(0x7fdaec079000 0x7fdaec0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeaffd700 1 -- 172.21.2.117:0/2151431914 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552862732 0 0) 0x7fdae4001180 con 0x7fdaec1b00e0 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeaffd700 1 -- 172.21.2.117:0/2151431914 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad8002fe0 con 0x7fdaec1b00e0 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeb7fe700 1 -- 172.21.2.117:0/2151431914 >> v1:172.21.2.125:6789/0 conn(0x7fdaec07da20 legacy=0x7fdaec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeb7fe700 1 -- 172.21.2.117:0/2151431914 >> v1:172.21.2.117:6789/0 conn(0x7fdaec1b00e0 legacy=0x7fdaec0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeb7fe700 1 -- 172.21.2.117:0/2151431914 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec079400 con 0x7fdaec079000 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.337 7fdaeb7fe700 1 --2- 172.21.2.117:0/2151431914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec079000 0x7fdaec0e4d90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fdad40069d0 tx=0x7fdad4009e40).ready entity=mon.0 client_cookie=34bb1e4dfe9965ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaeaffd700 1 -- 172.21.2.117:0/2151431914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad400b510 con 0x7fdaec079000 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaeaffd700 1 -- 172.21.2.117:0/2151431914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad400b890 con 0x7fdaec079000 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaeaffd700 1 -- 172.21.2.117:0/2151431914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdad400bdb0 con 0x7fdaec079000 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 -- 172.21.2.117:0/2151431914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec079000 msgr2=0x7fdaec0e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 --2- 172.21.2.117:0/2151431914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec079000 0x7fdaec0e4d90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0x7fdad40069d0 tx=0x7fdad4009e40).stop 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 -- 172.21.2.117:0/2151431914 shutdown_connections 2021-02-10T11:40:06.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 --2- 172.21.2.117:0/2151431914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec079000 0x7fdaec0e4d90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 -- 172.21.2.117:0/2151431914 >> 172.21.2.117:0/2151431914 conn(0x7fdaec0ce440 msgr2=0x7fdaec0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 -- 172.21.2.117:0/2151431914 shutdown_connections 2021-02-10T11:40:06.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.338 7fdaf2447700 1 -- 172.21.2.117:0/2151431914 wait complete. 2021-02-10T11:40:06.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.353 7fdaf2447700 1 Processor -- start 2021-02-10T11:40:06.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.353 7fdaf2447700 1 -- start start 2021-02-10T11:40:06.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdaec10a210 0x7fdaec1a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdaec1a8c60 0x7fdaec1a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec19d3a0 con 0x7fdaec19d920 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec1af340 con 0x7fdaec10a210 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf2447700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec19aea0 con 0x7fdaec1a8c60 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47596/0 (socket says 172.21.2.117:47596) 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaeb7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdaec10a210 0x7fdaec1a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 -- 172.21.2.117:0/53719486 learned_addr learned my addr 172.21.2.117:0/53719486 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaf0a64700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdaec1a8c60 0x7fdaec1a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 -- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdaec1a8c60 msgr2=0x7fdaec1a9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdaec1a8c60 0x7fdaec1a9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 -- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdaec10a210 msgr2=0x7fdaec1a8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdaec10a210 0x7fdaec1a8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.354 7fdaebfff700 1 -- 172.21.2.117:0/53719486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec10a610 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdaebfff700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fdadc00b5b0 tx=0x7fdadc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdadc00bd40 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc006240 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdadc005720 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdaf2447700 1 -- 172.21.2.117:0/53719486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec10a610 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdaf2447700 1 -- 172.21.2.117:0/53719486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec1a9970 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.355 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdadc0054a0 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.356 7fdae8ff9700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 0x7fdad001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.356 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdadc032590 con 0x7fdaec19d920 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.356 7fdaeb7fe700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 0x7fdad001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.356 7fdaeb7fe700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 0x7fdad001abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.387 7fdaf2447700 1 -- 172.21.2.117:0/53719486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaec108ba0 con 0x7fdaec19d920 2021-02-10T11:40:06.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.389 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdadc000940 con 0x7fdaec19d920 2021-02-10T11:40:06.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.519 7fdaf2447700 1 -- 172.21.2.117:0/53719486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaec077a70 con 0x7fdaec19d920 2021-02-10T11:40:06.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.519 7fdae8ff9700 1 -- 172.21.2.117:0/53719486 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2096 (secure 0 0 0) 0x7fdadc00bd40 con 0x7fdaec19d920 2021-02-10T11:40:06.502 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:06.503 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:05.887046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7233":{"gid":7233,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/107410323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":107410323},{"type":"v1","addr":"172.21.2.125:6835","nonce":107410323}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:06.503 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 20 2021-02-10T11:40:06.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 -- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 msgr2=0x7fdad001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 0x7fdad001abc0 crc :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 -- 172.21.2.117:0/53719486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 msgr2=0x7fdaec109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7fdadc00b5b0 tx=0x7fdadc00b950).stop 2021-02-10T11:40:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 -- 172.21.2.117:0/53719486 shutdown_connections 2021-02-10T11:40:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdaec1a8c60 0x7fdaec1a9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdaec10a210 0x7fdaec1a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdad00186c0 0x7fdad001abc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 --2- 172.21.2.117:0/53719486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdaec19d920 0x7fdaec109bf0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.521 7fdaf2447700 1 -- 172.21.2.117:0/53719486 >> 172.21.2.117:0/53719486 conn(0x7fdaec19a700 msgr2=0x7fdaec199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.522 7fdaf2447700 1 -- 172.21.2.117:0/53719486 shutdown_connections 2021-02-10T11:40:06.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.522 7fdaf2447700 1 -- 172.21.2.117:0/53719486 wait complete. 2021-02-10T11:40:06.513 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 Processor -- start 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 -- start start 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b5407da20 0x7f5b540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b540d1d10 con 0x7f5b540e5190 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b54077a70 con 0x7f5b540e4d90 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b5afd7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b540777d0 con 0x7f5b5407da20 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b595f4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5b540e5190 0x7f5b540d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47852/0 (socket says 172.21.2.117:47852) 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.942 7f5b595f4700 1 -- 172.21.2.117:0/1670671119 learned_addr learned my addr 172.21.2.117:0/1670671119 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:06.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142715675 0 0) 0x7f5b4c001180 con 0x7f5b540e5190 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b40002fe0 con 0x7f5b540e5190 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b58df3700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b5407da20 0x7f5b540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2937286502 0 0) 0x7f5b4c001180 con 0x7f5b540e5190 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b5407da20 msgr2=0x7f5b540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b5407da20 0x7f5b540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 >> v1:172.21.2.125:6789/0 conn(0x7f5b540e4d90 legacy=0x7f5b541b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b541b08d0 con 0x7f5b540e5190 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5b4c001680 con 0x7f5b540e5190 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 >> v1:172.21.2.117:6789/0 conn(0x7f5b540e5190 legacy=0x7f5b540d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b40003b10 0x7f5b40005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b40006750 0x7f5b40008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b40009510 0x7f5b4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4000c2c0 con 0x7f5b40006750 2021-02-10T11:40:06.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4000c4e0 con 0x7f5b40009510 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4000c700 con 0x7f5b40003b10 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.943 7f5b53fff700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b40006750 0x7f5b40008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b58df3700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b40003b10 0x7f5b40005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b595f4700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b40009510 0x7f5b4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 -- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b40003b10 msgr2=0x7f5b40005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b40003b10 0x7f5b40005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 -- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b40009510 msgr2=0x7f5b4000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b40009510 0x7f5b4000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 -- 172.21.2.117:0/1670671119 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b40050970 con 0x7f5b40006750 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b595f4700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b40009510 0x7f5b4000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b53fff700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b40006750 0x7f5b40008c70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f5b44006e40 tx=0x7f5b44009db0).ready entity=mon.0 client_cookie=9099d18202ae7572 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b4400b580 con 0x7f5b40006750 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4400b840 con 0x7f5b40006750 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b537fe700 1 -- 172.21.2.117:0/1670671119 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b4400bcd0 con 0x7f5b40006750 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b5afd7700 1 -- 172.21.2.117:0/1670671119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b40006750 msgr2=0x7f5b40008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b5afd7700 1 --2- 172.21.2.117:0/1670671119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b40006750 0x7f5b40008c70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7f5b44006e40 tx=0x7f5b44009db0).stop 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.944 7f5b595f4700 1 -- 172.21.2.117:0/1670671119 reap_dead start 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.945 7f5b5afd7700 1 -- 172.21.2.117:0/1670671119 shutdown_connections 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.945 7f5b5afd7700 1 -- 172.21.2.117:0/1670671119 >> 172.21.2.117:0/1670671119 conn(0x7f5b540ce440 msgr2=0x7f5b540e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:06.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.945 7f5b5afd7700 1 -- 172.21.2.117:0/1670671119 shutdown_connections 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.945 7f5b5afd7700 1 -- 172.21.2.117:0/1670671119 wait complete. 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.959 7f5b5afd7700 1 Processor -- start 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.959 7f5b5afd7700 1 -- start start 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b540a3b30 0x7f5b541ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b541aebb0 0x7f5b540bb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b540a4060 con 0x7f5b541aefb0 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b540bc140 con 0x7f5b541aebb0 2021-02-10T11:40:06.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b540bc360 con 0x7f5b540a3b30 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47614/0 (socket says 172.21.2.117:47614) 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 -- 172.21.2.117:0/1905066536 learned_addr learned my addr 172.21.2.117:0/1905066536 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 -- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b540a3b30 msgr2=0x7f5b541ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b540a3b30 0x7f5b541ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 -- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b541aebb0 msgr2=0x7f5b540bb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b541aebb0 0x7f5b540bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 -- 172.21.2.117:0/1905066536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54188cf0 con 0x7f5b541aefb0 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b595f4700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f5b4c001940 tx=0x7f5b4c00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b4c00ace0 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c0043a0 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b4c0048c0 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.960 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b54188cf0 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.961 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b54188cf0 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.961 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b4c004580 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.961 7f5b517fa700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 0x7f5b3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.961 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5b4c032560 con 0x7f5b541aefb0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.962 7f5b58df3700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 0x7f5b3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:06.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.962 7f5b58df3700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 0x7f5b3801abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:06.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.992 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b540a4060 con 0x7f5b541aefb0 2021-02-10T11:40:06.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:06.994 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b4c030d40 con 0x7f5b541aefb0 2021-02-10T11:40:07.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.119 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b540a4060 con 0x7f5b541aefb0 2021-02-10T11:40:07.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.119 7f5b517fa700 1 -- 172.21.2.117:0/1905066536 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f5b4c004560 con 0x7f5b541aefb0 2021-02-10T11:40:07.102 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:07.102 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":39,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:04.882873","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:01.847956","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:02.928003","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:07.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 msgr2=0x7f5b3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 0x7f5b3801abc0 crc :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 msgr2=0x7f5b540bbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f5b4c001940 tx=0x7f5b4c00a8f0).stop 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 shutdown_connections 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b380186c0 0x7f5b3801abc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b541aefb0 0x7f5b540bbb20 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b541aebb0 0x7f5b540bb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 --2- 172.21.2.117:0/1905066536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b540a3b30 0x7f5b541ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 >> 172.21.2.117:0/1905066536 conn(0x7f5b5419f510 msgr2=0x7f5b541a6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 shutdown_connections 2021-02-10T11:40:07.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.121 7f5b5afd7700 1 -- 172.21.2.117:0/1905066536 wait complete. 2021-02-10T11:40:07.112 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 Processor -- start 2021-02-10T11:40:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 -- start start 2021-02-10T11:40:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660079000 0x7f26600e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f26601b00d0 0x7f26601b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2660077a70 con 0x7f266007da20 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26600e53a0 con 0x7f26601b00d0 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f26665e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2660079400 con 0x7f2660079000 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f265ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f266007da20 0x7f26600771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47872/0 (socket says 172.21.2.117:47872) 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.553 7f265ffff700 1 -- 172.21.2.117:0/8915268 learned_addr learned my addr 172.21.2.117:0/8915268 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f265f7fe700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660079000 0x7f26600e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 --2- 172.21.2.117:0/8915268 >> v2:172.21.2.125:3300/0 conn(0x7f26601b00d0 0x7f26601b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f265effd700 1 -- 172.21.2.117:0/8915268 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625460000 0 0) 0x7f2650001180 con 0x7f266007da20 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f265effd700 1 -- 172.21.2.117:0/8915268 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2648002fe0 con 0x7f266007da20 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 -- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660079000 msgr2=0x7f26600e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660079000 0x7f26600e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 -- 172.21.2.117:0/8915268 >> v1:172.21.2.117:6789/0 conn(0x7f266007da20 legacy=0x7f26600771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 -- 172.21.2.117:0/8915268 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266007de20 con 0x7f26601b00d0 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.554 7f2664c06700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26601b00d0 0x7f26601b04d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2658006e50 tx=0x7f2658009df0).ready entity=mon.1 client_cookie=e969fb91b461ee1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.555 7f265effd700 1 -- 172.21.2.117:0/8915268 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f265800b510 con 0x7f26601b00d0 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.555 7f265effd700 1 -- 172.21.2.117:0/8915268 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2658006790 con 0x7f26601b00d0 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.555 7f265effd700 1 -- 172.21.2.117:0/8915268 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f265800b750 con 0x7f26601b00d0 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.555 7f26665e9700 1 -- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26601b00d0 msgr2=0x7f26601b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.555 7f26665e9700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26601b00d0 0x7f26601b04d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2658006e50 tx=0x7f2658009df0).stop 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 -- 172.21.2.117:0/8915268 shutdown_connections 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26601b00d0 0x7f26601b04d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 --2- 172.21.2.117:0/8915268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660079000 0x7f26600e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 -- 172.21.2.117:0/8915268 >> 172.21.2.117:0/8915268 conn(0x7f26600ce430 msgr2=0x7f26600e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 -- 172.21.2.117:0/8915268 shutdown_connections 2021-02-10T11:40:07.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.556 7f26665e9700 1 -- 172.21.2.117:0/8915268 wait complete. 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.570 7f26665e9700 1 Processor -- start 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.570 7f26665e9700 1 -- start start 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660095ab0 0x7f26600b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26600b76f0 0x7f26600b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26600961a0 con 0x7f2660095090 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26600a1090 con 0x7f26600b76f0 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f26665e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26600a0c10 con 0x7f2660095ab0 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47626/0 (socket says 172.21.2.117:47626) 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 -- 172.21.2.117:0/320291110 learned_addr learned my addr 172.21.2.117:0/320291110 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 -- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660095ab0 msgr2=0x7f26600b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660095ab0 0x7f26600b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 -- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26600b76f0 msgr2=0x7f26600b7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26600b76f0 0x7f26600b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.571 7f265ffff700 1 -- 172.21.2.117:0/320291110 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26600966a0 con 0x7f2660095090 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265ffff700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f2650007680 tx=0x7f265000a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2650001ad0 con 0x7f2660095090 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2650007030 con 0x7f2660095090 2021-02-10T11:40:07.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f26665e9700 1 -- 172.21.2.117:0/320291110 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26600966a0 con 0x7f2660095090 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2650004620 con 0x7f2660095090 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f26665e9700 1 -- 172.21.2.117:0/320291110 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660096c30 con 0x7f2660095090 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26500043a0 con 0x7f2660095090 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 0x7f264001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.572 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f26500327a0 con 0x7f2660095090 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.573 7f265f7fe700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 0x7f264001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:07.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.573 7f265f7fe700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 0x7f264001abd0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:07.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.602 7f26665e9700 1 -- 172.21.2.117:0/320291110 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660103280 con 0x7f2660095090 2021-02-10T11:40:07.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.605 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2650004780 con 0x7f2660095090 2021-02-10T11:40:07.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.735 7f26665e9700 1 -- 172.21.2.117:0/320291110 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2660103280 con 0x7f2660095090 2021-02-10T11:40:07.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.739 7f265cff9700 1 -- 172.21.2.117:0/320291110 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f26500043a0 con 0x7f2660095090 2021-02-10T11:40:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:40:07.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.744 7f26665e9700 1 -- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 msgr2=0x7f264001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.744 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 0x7f264001abd0 crc :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.744 7f26665e9700 1 -- 172.21.2.117:0/320291110 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 msgr2=0x7f2660095490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.744 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f2650007680 tx=0x7f265000a920).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 -- 172.21.2.117:0/320291110 shutdown_connections 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26600b76f0 0x7f26600b7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2660095ab0 0x7f26600b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26400186d0 0x7f264001abd0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 --2- 172.21.2.117:0/320291110 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2660095090 0x7f2660095490 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:07.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 -- 172.21.2.117:0/320291110 >> 172.21.2.117:0/320291110 conn(0x7f266019f4b0 msgr2=0x7f2660094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:07.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 -- 172.21.2.117:0/320291110 shutdown_connections 2021-02-10T11:40:07.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:07.745 7f26665e9700 1 -- 172.21.2.117:0/320291110 wait complete. 2021-02-10T11:40:07.735 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:08.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.175 7f7db3d53700 1 Processor -- start 2021-02-10T11:40:08.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.175 7f7db3d53700 1 -- start start 2021-02-10T11:40:08.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db3d53700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7dac0d1530 0x7f7dac0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db3d53700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac077a70 con 0x7f7dac07da20 2021-02-10T11:40:08.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db3d53700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac079400 con 0x7f7dac079000 2021-02-10T11:40:08.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db3d53700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dac07de20 con 0x7f7dac0d1530 2021-02-10T11:40:08.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db1b6f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7dac07da20 0x7f7dac0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47884/0 (socket says 172.21.2.117:47884) 2021-02-10T11:40:08.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db1b6f700 1 -- 172.21.2.117:0/3849420195 learned_addr learned my addr 172.21.2.117:0/3849420195 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:08.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346555982 0 0) 0x7f7da00010b0 con 0x7f7dac07da20 2021-02-10T11:40:08.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98002fe0 con 0x7f7dac07da20 2021-02-10T11:40:08.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.176 7f7db2370700 1 --2- 172.21.2.117:0/3849420195 >> v2:172.21.2.125:3300/0 conn(0x7f7dac0d1530 0x7f7dac0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2122167673 0 0) 0x7f7da0001010 con 0x7f7dac07da20 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 >> v2:172.21.2.125:3300/0 conn(0x7f7dac0d1530 msgr2=0x7f7dac0d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 --2- 172.21.2.117:0/3849420195 >> v2:172.21.2.125:3300/0 conn(0x7f7dac0d1530 0x7f7dac0d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 >> v1:172.21.2.125:6789/0 conn(0x7f7dac079000 legacy=0x7f7dac0e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac1b00f0 con 0x7f7dac07da20 2021-02-10T11:40:08.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7da0001530 con 0x7f7dac07da20 2021-02-10T11:40:08.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 >> v1:172.21.2.117:6789/0 conn(0x7f7dac07da20 legacy=0x7f7dac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 0x7f7d98005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d98006790 0x7f7d98008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d98009590 0x7f7d9800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800c300 con 0x7f7d98003ac0 2021-02-10T11:40:08.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800c520 con 0x7f7d98009590 2021-02-10T11:40:08.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800c740 con 0x7f7d98006790 2021-02-10T11:40:08.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 0x7f7d98005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db2370700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d98009590 0x7f7d9800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db136e700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d98006790 0x7f7d98008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 -- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d98006790 msgr2=0x7f7d98008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d98006790 0x7f7d98008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 -- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d98009590 msgr2=0x7f7d9800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d98009590 0x7f7d9800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 -- 172.21.2.117:0/3849420195 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980509b0 con 0x7f7d98003ac0 2021-02-10T11:40:08.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db2370700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d98009590 0x7f7d9800bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db2370700 1 -- 172.21.2.117:0/3849420195 reap_dead start 2021-02-10T11:40:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db136e700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d98006790 0x7f7d98008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.177 7f7db1b6f700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 0x7f7d98005ef0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f7da00017e0 tx=0x7f7da0007d40).ready entity=mon.0 client_cookie=d20367957300d9a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7da000a520 con 0x7f7d98003ac0 2021-02-10T11:40:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da000a760 con 0x7f7d98003ac0 2021-02-10T11:40:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db0b6d700 1 -- 172.21.2.117:0/3849420195 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7da000ac80 con 0x7f7d98003ac0 2021-02-10T11:40:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db3d53700 1 -- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 msgr2=0x7f7d98005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db3d53700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 0x7f7d98005ef0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f7da00017e0 tx=0x7f7da0007d40).stop 2021-02-10T11:40:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db3d53700 1 -- 172.21.2.117:0/3849420195 shutdown_connections 2021-02-10T11:40:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db3d53700 1 --2- 172.21.2.117:0/3849420195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d98003ac0 0x7f7d98005ef0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.178 7f7db3d53700 1 -- 172.21.2.117:0/3849420195 >> 172.21.2.117:0/3849420195 conn(0x7f7dac0ce450 msgr2=0x7f7dac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.179 7f7db3d53700 1 -- 172.21.2.117:0/3849420195 shutdown_connections 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.179 7f7db3d53700 1 -- 172.21.2.117:0/3849420195 wait complete. 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.193 7f7db3d53700 1 Processor -- start 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.193 7f7db3d53700 1 -- start start 2021-02-10T11:40:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7dac09d8a0 0x7f7dac187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7dac0a2930 0x7f7dac0a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0a23b0 con 0x7f7dac0b8650 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac107400 con 0x7f7dac09d8a0 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db3d53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0a3350 con 0x7f7dac0a2930 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47648/0 (socket says 172.21.2.117:47648) 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 -- 172.21.2.117:0/3438317520 learned_addr learned my addr 172.21.2.117:0/3438317520 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db2370700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7dac0a2930 0x7f7dac0a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db1b6f700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7dac09d8a0 0x7f7dac187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 -- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7dac0a2930 msgr2=0x7f7dac0a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7dac0a2930 0x7f7dac0a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 -- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7dac09d8a0 msgr2=0x7f7dac187bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7dac09d8a0 0x7f7dac187bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.194 7f7db136e700 1 -- 172.21.2.117:0/3438317520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0a1520 con 0x7f7dac0b8650 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7db136e700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f7da8006f20 tx=0x7f7da800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7da800bd80 con 0x7f7dac0b8650 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8006690 con 0x7f7dac0b8650 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7da8002220 con 0x7f7dac0b8650 2021-02-10T11:40:08.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac0a1520 con 0x7f7dac0b8650 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac0a1520 con 0x7f7dac0b8650 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7da80054b0 con 0x7f7dac0b8650 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.195 7f7d9e7fc700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 0x7f7d9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.196 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7da80328f0 con 0x7f7dac0b8650 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.196 7f7db1b6f700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 0x7f7d9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.196 7f7db1b6f700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 0x7f7d9401abc0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.225 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac077a70 con 0x7f7dac0b8650 2021-02-10T11:40:08.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.228 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7da8006690 con 0x7f7dac0b8650 2021-02-10T11:40:08.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.361 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7dac077a70 con 0x7f7dac0b8650 2021-02-10T11:40:08.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.365 7f7d9e7fc700 1 -- 172.21.2.117:0/3438317520 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7da80054b0 con 0x7f7dac0b8650 2021-02-10T11:40:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:40:08.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 msgr2=0x7f7d9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 0x7f7d9401abc0 crc :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 msgr2=0x7f7dac0b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f7da8006f20 tx=0x7f7da800b990).stop 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 shutdown_connections 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d94018730 0x7f7d9401abc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7dac0b8650 0x7f7dac0b8a50 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7dac0a2930 0x7f7dac0a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 --2- 172.21.2.117:0/3438317520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7dac09d8a0 0x7f7dac187bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 >> 172.21.2.117:0/3438317520 conn(0x7f7dac19f530 msgr2=0x7f7dac103560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.370 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 shutdown_connections 2021-02-10T11:40:08.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.371 7f7db3d53700 1 -- 172.21.2.117:0/3438317520 wait complete. 2021-02-10T11:40:08.361 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 Processor -- start 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 -- start start 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc00d1530 0x7f5dc00d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0077a70 con 0x7f5dc0079000 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0079400 con 0x7f5dc007da20 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc8333700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc007de20 con 0x7f5dc00d1530 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc594e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5dc0079000 0x7f5dc00e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47904/0 (socket says 172.21.2.117:47904) 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.800 7f5dc594e700 1 -- 172.21.2.117:0/582881658 learned_addr learned my addr 172.21.2.117:0/582881658 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc6950700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc00d1530 0x7f5dc00d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503038871 0 0) 0x7f5db0001180 con 0x7f5dc0079000 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac002fe0 con 0x7f5dc0079000 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1994566468 0 0) 0x7f5db0001180 con 0x7f5dc0079000 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc00d1530 msgr2=0x7f5dc00d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc00d1530 0x7f5dc00d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 >> v1:172.21.2.125:6789/0 conn(0x7f5dc007da20 legacy=0x7f5dc00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc01b00f0 con 0x7f5dc0079000 2021-02-10T11:40:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5db0001680 con 0x7f5dc0079000 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 >> v1:172.21.2.117:6789/0 conn(0x7f5dc0079000 legacy=0x7f5dc00e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dac003b10 0x7f5dac005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dac006770 0x7f5dac008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dac009510 0x7f5dac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00c2c0 con 0x7f5dac006770 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00c4e0 con 0x7f5dac009510 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc514d700 1 -- 172.21.2.117:0/582881658 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac00c700 con 0x7f5dac003b10 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.801 7f5dc594e700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dac006770 0x7f5dac008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc614f700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dac003b10 0x7f5dac005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc6950700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dac009510 0x7f5dac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 -- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dac003b10 msgr2=0x7f5dac005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dac003b10 0x7f5dac005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 -- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dac009510 msgr2=0x7f5dac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dac009510 0x7f5dac00ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 -- 172.21.2.117:0/582881658 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac050970 con 0x7f5dac006770 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc594e700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dac006770 0x7f5dac008c90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f5db0000b60 tx=0x7f5db0007da0).ready entity=mon.0 client_cookie=d20e31a5fb85b3f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db00070c0 con 0x7f5dac006770 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db00043a0 con 0x7f5dac006770 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc514d700 1 -- 172.21.2.117:0/582881658 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5db0004830 con 0x7f5dac006770 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc8333700 1 -- 172.21.2.117:0/582881658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dac006770 msgr2=0x7f5dac008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc8333700 1 --2- 172.21.2.117:0/582881658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dac006770 0x7f5dac008c90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0x7f5db0000b60 tx=0x7f5db0007da0).stop 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.802 7f5dc6950700 1 -- 172.21.2.117:0/582881658 reap_dead start 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.803 7f5dc8333700 1 -- 172.21.2.117:0/582881658 shutdown_connections 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.803 7f5dc8333700 1 -- 172.21.2.117:0/582881658 >> 172.21.2.117:0/582881658 conn(0x7f5dc00ce450 msgr2=0x7f5dc00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.803 7f5dc8333700 1 -- 172.21.2.117:0/582881658 shutdown_connections 2021-02-10T11:40:08.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.803 7f5dc8333700 1 -- 172.21.2.117:0/582881658 wait complete. 2021-02-10T11:40:08.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.817 7f5dc8333700 1 Processor -- start 2021-02-10T11:40:08.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.817 7f5dc8333700 1 -- start start 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 0x7f5dc009f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dc0191410 0x7f5dc018d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0190650 con 0x7f5dc018cea0 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0107430 con 0x7f5dc0191410 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc8333700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc009fb30 con 0x7f5dc018d2a0 2021-02-10T11:40:08.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47666/0 (socket says 172.21.2.117:47666) 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 -- 172.21.2.117:0/1829717797 learned_addr learned my addr 172.21.2.117:0/1829717797 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc594e700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 0x7f5dc009f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc6950700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dc0191410 0x7f5dc018d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 -- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 msgr2=0x7f5dc009f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 0x7f5dc009f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 -- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dc0191410 msgr2=0x7f5dc018d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dc0191410 0x7f5dc018d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc614f700 1 -- 172.21.2.117:0/1829717797 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc009fd50 con 0x7f5dc018cea0 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.818 7f5dc594e700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 0x7f5dc009f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:08.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5dc614f700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f5dbc006f10 tx=0x7f5dbc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dbc00bd80 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc006030 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dbc0052e0 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc009fd50 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc018e0d0 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5dbc006210 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.819 7f5db6ffd700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 0x7f5da801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.820 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5dbc0328c0 con 0x7f5dc018cea0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.820 7f5dc594e700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 0x7f5da801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:08.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.820 7f5dc594e700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 0x7f5da801abc0 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:08.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.850 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0107400 con 0x7f5dc018cea0 2021-02-10T11:40:08.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.852 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5dbc000960 con 0x7f5dc018cea0 2021-02-10T11:40:08.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.983 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5dc0107400 con 0x7f5dc018cea0 2021-02-10T11:40:08.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.987 7f5db6ffd700 1 -- 172.21.2.117:0/1829717797 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5dbc006030 con 0x7f5dc018cea0 2021-02-10T11:40:08.973 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:40:08.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.992 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 msgr2=0x7f5da801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.992 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 0x7f5da801abc0 crc :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.992 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 msgr2=0x7f5dc0190df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.992 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0x7f5dbc006f10 tx=0x7f5dbc00b990).stop 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 shutdown_connections 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5dc0191410 0x7f5dc018d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5dc018d2a0 0x7f5dc009f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5da80186c0 0x7f5da801abc0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 --2- 172.21.2.117:0/1829717797 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5dc018cea0 0x7f5dc0190df0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:08.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 >> 172.21.2.117:0/1829717797 conn(0x7f5dc019f530 msgr2=0x7f5dc0103560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:08.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 shutdown_connections 2021-02-10T11:40:08.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:08.993 7f5dc8333700 1 -- 172.21.2.117:0/1829717797 wait complete. 2021-02-10T11:40:08.983 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 Processor -- start 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 -- start start 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15b007da20 0x7f15b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00e4da0 0x7f15b01b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f15b00e51a0 0x7f15b0077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b0077a70 con 0x7f15b007da20 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b00777d0 con 0x7f15b00e51a0 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b62f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b007de20 con 0x7f15b00e4da0 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15affff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15b007da20 0x7f15b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f15b007da20 0x7f15b00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47674/0 (socket says 172.21.2.117:47674) 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.423 7f15b490f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f15b00e51a0 0x7f15b0077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 -- 172.21.2.117:0/3792876410 learned_addr learned my addr 172.21.2.117:0/3792876410 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15af7fe700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00e4da0 0x7f15b01b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 -- 172.21.2.117:0/3792876410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00e4da0 msgr2=0x7f15b01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00e4da0 0x7f15b01b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 -- 172.21.2.117:0/3792876410 >> v2:172.21.2.125:3300/0 conn(0x7f15b00e51a0 msgr2=0x7f15b0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 --2- 172.21.2.117:0/3792876410 >> v2:172.21.2.125:3300/0 conn(0x7f15b00e51a0 0x7f15b0077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 -- 172.21.2.117:0/3792876410 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b01b0710 con 0x7f15b007da20 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15affff700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b007da20 0x7f15b00771b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f15a0006700 tx=0x7f15a0009b80).ready entity=mon.0 client_cookie=c96309d5e4acb8ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15aeffd700 1 -- 172.21.2.117:0/3792876410 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a000c0c0 con 0x7f15b007da20 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15aeffd700 1 -- 172.21.2.117:0/3792876410 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a000c340 con 0x7f15b007da20 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15aeffd700 1 -- 172.21.2.117:0/3792876410 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a000c820 con 0x7f15b007da20 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15b62f2700 1 -- 172.21.2.117:0/3792876410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b007da20 msgr2=0x7f15b00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.424 7f15b62f2700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b007da20 0x7f15b00771b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0x7f15a0006700 tx=0x7f15a0009b80).stop 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 -- 172.21.2.117:0/3792876410 shutdown_connections 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 --2- 172.21.2.117:0/3792876410 >> v2:172.21.2.125:3300/0 conn(0x7f15b00e51a0 0x7f15b0077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00e4da0 0x7f15b01b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 --2- 172.21.2.117:0/3792876410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b007da20 0x7f15b00771b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 -- 172.21.2.117:0/3792876410 >> 172.21.2.117:0/3792876410 conn(0x7f15b00ce450 msgr2=0x7f15b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 -- 172.21.2.117:0/3792876410 shutdown_connections 2021-02-10T11:40:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.425 7f15b62f2700 1 -- 172.21.2.117:0/3792876410 wait complete. 2021-02-10T11:40:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 Processor -- start 2021-02-10T11:40:09.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 -- start start 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00a0c70 0x7f15b00b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15b0099d60 0x7f15b009a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00a1070 con 0x7f15b009a780 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b00b4880 con 0x7f15b0099d60 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b62f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b009b9f0 con 0x7f15b00a0c70 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b490f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.440 7f15b490f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47680/0 (socket says 172.21.2.117:47680) 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 -- 172.21.2.117:0/527242537 learned_addr learned my addr 172.21.2.117:0/527242537 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15af7fe700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15b0099d60 0x7f15b009a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15affff700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00a0c70 0x7f15b00b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 -- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00a0c70 msgr2=0x7f15b00b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00a0c70 0x7f15b00b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 -- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15b0099d60 msgr2=0x7f15b009a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15b0099d60 0x7f15b009a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 -- 172.21.2.117:0/527242537 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b009bc10 con 0x7f15b009a780 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b490f700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f15a800b5e0 tx=0x7f15a800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a8006370 con 0x7f15b009a780 2021-02-10T11:40:09.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a800bd70 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b62f2700 1 -- 172.21.2.117:0/527242537 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b009bc10 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15b62f2700 1 -- 172.21.2.117:0/527242537 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b009c1a0 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.441 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15a80024d0 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.442 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f15a8005330 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.442 7f15acff9700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 0x7f159001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.442 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f15a80326d0 con 0x7f15b009a780 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.442 7f15affff700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 0x7f159001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:09.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.443 7f15affff700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 0x7f159001abc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:09.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.472 7f15b62f2700 1 -- 172.21.2.117:0/527242537 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b01032a0 con 0x7f15b009a780 2021-02-10T11:40:09.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.475 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15a8000bc0 con 0x7f15b009a780 2021-02-10T11:40:09.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.604 7f15b62f2700 1 -- 172.21.2.117:0/527242537 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f15b0077a70 con 0x7f15b009a780 2021-02-10T11:40:09.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.608 7f15acff9700 1 -- 172.21.2.117:0/527242537 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7f15a8005690 con 0x7f15b009a780 2021-02-10T11:40:09.593 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 -- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 msgr2=0x7f159001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 0x7f159001abc0 crc :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 -- 172.21.2.117:0/527242537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 msgr2=0x7f15b009b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f15a800b5e0 tx=0x7f15a800b980).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 -- 172.21.2.117:0/527242537 shutdown_connections 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15900186c0 0x7f159001abc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15b009a780 0x7f15b009b3d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15b0099d60 0x7f15b009a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 --2- 172.21.2.117:0/527242537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15b00a0c70 0x7f15b00b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.613 7f15b62f2700 1 -- 172.21.2.117:0/527242537 >> 172.21.2.117:0/527242537 conn(0x7f15b019f4f0 msgr2=0x7f15b0094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.614 7f15b62f2700 1 -- 172.21.2.117:0/527242537 shutdown_connections 2021-02-10T11:40:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:09.614 7f15b62f2700 1 -- 172.21.2.117:0/527242537 wait complete. 2021-02-10T11:40:09.604 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:10.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 Processor -- start 2021-02-10T11:40:10.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 -- start start 2021-02-10T11:40:10.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0c00079000 0x7f0c000e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000d1d10 con 0x7f0c0007da20 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00077a70 con 0x7f0c001b00e0 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.032 7f0c07265700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000e53b0 con 0x7f0c00079000 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.033 7f0c05081700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0c0007da20 0x7f0c000771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47934/0 (socket says 172.21.2.117:47934) 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.033 7f0c05081700 1 -- 172.21.2.117:0/1877348093 learned_addr learned my addr 172.21.2.117:0/1877348093 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.033 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302780100 0 0) 0x7f0bf4001180 con 0x7f0c0007da20 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.033 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bec002fe0 con 0x7f0c0007da20 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.033 7f0c04880700 1 --2- 172.21.2.117:0/1877348093 >> v2:172.21.2.125:3300/0 conn(0x7f0c00079000 0x7f0c000e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624358038 0 0) 0x7f0bf8000f90 con 0x7f0c001b00e0 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c04880700 1 -- 172.21.2.117:0/1877348093 >> v1:172.21.2.125:6789/0 conn(0x7f0c001b00e0 legacy=0x7f0c000d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c04880700 1 -- 172.21.2.117:0/1877348093 >> v1:172.21.2.117:6789/0 conn(0x7f0c0007da20 legacy=0x7f0c000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c04880700 1 -- 172.21.2.117:0/1877348093 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00079400 con 0x7f0c00079000 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c04880700 1 --2- 172.21.2.117:0/1877348093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c00079000 0x7f0c000e4d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0bfc006650 tx=0x7f0bfc009a80).ready entity=mon.1 client_cookie=c86e1becc82175f1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc0009b0 con 0x7f0c00079000 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00c0c0 con 0x7f0c00079000 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0bf3fff700 1 -- 172.21.2.117:0/1877348093 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc00c5e0 con 0x7f0c00079000 2021-02-10T11:40:10.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c07265700 1 -- 172.21.2.117:0/1877348093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c00079000 msgr2=0x7f0c000e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.034 7f0c07265700 1 --2- 172.21.2.117:0/1877348093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c00079000 0x7f0c000e4d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0bfc006650 tx=0x7f0bfc009a80).stop 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.035 7f0c07265700 1 -- 172.21.2.117:0/1877348093 shutdown_connections 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.035 7f0c07265700 1 --2- 172.21.2.117:0/1877348093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c00079000 0x7f0c000e4d90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.035 7f0c07265700 1 -- 172.21.2.117:0/1877348093 >> 172.21.2.117:0/1877348093 conn(0x7f0c000ce440 msgr2=0x7f0c000e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.035 7f0c07265700 1 -- 172.21.2.117:0/1877348093 shutdown_connections 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.035 7f0c07265700 1 -- 172.21.2.117:0/1877348093 wait complete. 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 Processor -- start 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 -- start start 2021-02-10T11:40:10.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 0x7f0c001a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 0x7f0c0018b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0018aac0 con 0x7f0c001a8a90 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0018a610 con 0x7f0c0018a210 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.050 7f0c07265700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0018a7f0 con 0x7f0c0008f410 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47692/0 (socket says 172.21.2.117:47692) 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 -- 172.21.2.117:0/2517464082 learned_addr learned my addr 172.21.2.117:0/2517464082 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c05081700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 0x7f0c001a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c05882700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 0x7f0c0018b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 -- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 msgr2=0x7f0c001a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 0x7f0c001a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 -- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 msgr2=0x7f0c0018b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 0x7f0c0018b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 -- 172.21.2.117:0/2517464082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0018b660 con 0x7f0c001a8a90 2021-02-10T11:40:10.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c05081700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 0x7f0c001a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c05882700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 0x7f0c0018b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c04880700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc00b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc0093a0 con 0x7f0c001a8a90 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc0067f0 con 0x7f0c001a8a90 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c07265700 1 -- 172.21.2.117:0/2517464082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0018b660 con 0x7f0c001a8a90 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bfc0022b0 con 0x7f0c001a8a90 2021-02-10T11:40:10.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.051 7f0c07265700 1 -- 172.21.2.117:0/2517464082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0018bbf0 con 0x7f0c001a8a90 2021-02-10T11:40:10.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.052 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bfc0052e0 con 0x7f0c001a8a90 2021-02-10T11:40:10.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.052 7f0bf1ffb700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 0x7f0be001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:10.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.052 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0bfc0326c0 con 0x7f0c001a8a90 2021-02-10T11:40:10.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.052 7f0c05081700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 0x7f0be001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:10.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.053 7f0c05081700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 0x7f0be001abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:10.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.082 7f0c07265700 1 -- 172.21.2.117:0/2517464082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00103360 con 0x7f0c001a8a90 2021-02-10T11:40:10.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.085 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bfc0093a0 con 0x7f0c001a8a90 2021-02-10T11:40:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.215 7f0c07265700 1 -- 172.21.2.117:0/2517464082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c00077a70 con 0x7f0c001a8a90 2021-02-10T11:40:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.216 7f0bf1ffb700 1 -- 172.21.2.117:0/2517464082 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2096 (secure 0 0 0) 0x7f0bfc00bda0 con 0x7f0c001a8a90 2021-02-10T11:40:10.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:10.199 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:05.887046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7233":{"gid":7233,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/107410323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":107410323},{"type":"v1","addr":"172.21.2.125:6835","nonce":107410323}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 20 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.217 7f0c07265700 1 -- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 msgr2=0x7f0be001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.217 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 0x7f0be001abc0 crc :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.217 7f0c07265700 1 -- 172.21.2.117:0/2517464082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 msgr2=0x7f0c00189bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.217 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc00b9b0).stop 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 -- 172.21.2.117:0/2517464082 shutdown_connections 2021-02-10T11:40:10.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0be00186c0 0x7f0be001abc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c001a8a90 0x7f0c00189bf0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c0018a210 0x7f0c0018b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 --2- 172.21.2.117:0/2517464082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c0008f410 0x7f0c001a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 -- 172.21.2.117:0/2517464082 >> 172.21.2.117:0/2517464082 conn(0x7f0c0019f550 msgr2=0x7f0c0018bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 -- 172.21.2.117:0/2517464082 shutdown_connections 2021-02-10T11:40:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:10.218 7f0c07265700 1 -- 172.21.2.117:0/2517464082 wait complete. 2021-02-10T11:40:10.410 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:03.881036', 'modified': '2021-02-10 11:40:05.887046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7233}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7233': {'gid': 7233, 'name': 'b', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/107410323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 107410323}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 107410323}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:10.410 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:10.411 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:10.810 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:40:10.810 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7233, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9473976899999998, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 20, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 39, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.581403223000001 2021-02-10T11:40:10.811 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:40:10.814 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7233, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9473976899999998, "mdsmap_epoch": 20, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 12.581403223000001 } 2021-02-10T11:40:10.815 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 Processor -- start 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 -- start start 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40079000 0x7fcb400e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb401b00e0 0x7fcb401b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb40077a70 con 0x7fcb4007da20 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb400e53b0 con 0x7fcb401b00e0 2021-02-10T11:40:11.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.243 7fcb45eff700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40079400 con 0x7fcb40079000 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcb4007da20 0x7fcb400771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47948/0 (socket says 172.21.2.117:47948) 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3f7fe700 1 -- 172.21.2.117:0/211529079 learned_addr learned my addr 172.21.2.117:0/211529079 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 --2- 172.21.2.117:0/211529079 >> v2:172.21.2.117:3300/0 conn(0x7fcb401b00e0 0x7fcb401b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3effd700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40079000 0x7fcb400e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3e7fc700 1 -- 172.21.2.117:0/211529079 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457923401 0 0) 0x7fcb34001180 con 0x7fcb4007da20 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3e7fc700 1 -- 172.21.2.117:0/211529079 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb28002fe0 con 0x7fcb4007da20 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 -- 172.21.2.117:0/211529079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40079000 msgr2=0x7fcb400e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40079000 0x7fcb400e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 -- 172.21.2.117:0/211529079 >> v1:172.21.2.117:6789/0 conn(0x7fcb4007da20 legacy=0x7fcb400771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 -- 172.21.2.117:0/211529079 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4007de20 con 0x7fcb401b00e0 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3ffff700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401b00e0 0x7fcb401b04e0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fcb38006e30 tx=0x7fcb38009e60).ready entity=mon.0 client_cookie=718d039dfb9ea264 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3e7fc700 1 -- 172.21.2.117:0/211529079 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb38006a50 con 0x7fcb401b00e0 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3e7fc700 1 -- 172.21.2.117:0/211529079 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3800b300 con 0x7fcb401b00e0 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.244 7fcb3e7fc700 1 -- 172.21.2.117:0/211529079 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3800b7e0 con 0x7fcb401b00e0 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 -- 172.21.2.117:0/211529079 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401b00e0 msgr2=0x7fcb401b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401b00e0 0x7fcb401b04e0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fcb38006e30 tx=0x7fcb38009e60).stop 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 -- 172.21.2.117:0/211529079 shutdown_connections 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401b00e0 0x7fcb401b04e0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 --2- 172.21.2.117:0/211529079 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40079000 0x7fcb400e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 -- 172.21.2.117:0/211529079 >> 172.21.2.117:0/211529079 conn(0x7fcb400ce440 msgr2=0x7fcb400e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 -- 172.21.2.117:0/211529079 shutdown_connections 2021-02-10T11:40:11.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.245 7fcb45eff700 1 -- 172.21.2.117:0/211529079 wait complete. 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 Processor -- start 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 -- start start 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40192ba0 0x7fcb40192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cbd0 0x7fcb4018cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40193730 con 0x7fcb401067a0 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4018d630 con 0x7fcb4018cbd0 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.260 7fcb45eff700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4018d850 con 0x7fcb40192ba0 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47708/0 (socket says 172.21.2.117:47708) 2021-02-10T11:40:11.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 -- 172.21.2.117:0/2915814011 learned_addr learned my addr 172.21.2.117:0/2915814011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 -- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40192ba0 msgr2=0x7fcb40192fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40192ba0 0x7fcb40192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 -- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cbd0 msgr2=0x7fcb4018cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cbd0 0x7fcb4018cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 -- 172.21.2.117:0/2915814011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4018dc30 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb3effd700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fcb30006f60 tx=0x7fcb3000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3000bd60 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb30000c90 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb30002220 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4018dc30 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.261 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4018e1c0 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.262 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb300052e0 con 0x7fcb401067a0 2021-02-10T11:40:11.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.262 7fcb27fff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 0x7fcb2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.262 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb30032560 con 0x7fcb401067a0 2021-02-10T11:40:11.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.263 7fcb3f7fe700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 0x7fcb2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.263 7fcb3f7fe700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 0x7fcb2001abc0 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.293 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb40077a70 con 0x7fcb401067a0 2021-02-10T11:40:11.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.296 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb30000940 con 0x7fcb401067a0 2021-02-10T11:40:11.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.420 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb4018d5f0 con 0x7fcb401067a0 2021-02-10T11:40:11.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.421 7fcb27fff700 1 -- 172.21.2.117:0/2915814011 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+14226 (secure 0 0 0) 0x7fcb300052e0 con 0x7fcb401067a0 2021-02-10T11:40:11.403 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:11.403 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":39,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:04.882873","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:01.847956","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:02.928003","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:11.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 msgr2=0x7fcb2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 0x7fcb2001abc0 crc :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 msgr2=0x7fcb40106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fcb30006f60 tx=0x7fcb3000b970).stop 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 shutdown_connections 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb200186c0 0x7fcb2001abc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb401067a0 0x7fcb40106ba0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb4018cbd0 0x7fcb4018cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 --2- 172.21.2.117:0/2915814011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb40192ba0 0x7fcb40192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 >> 172.21.2.117:0/2915814011 conn(0x7fcb4019f510 msgr2=0x7fcb401abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:11.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.422 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 shutdown_connections 2021-02-10T11:40:11.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.423 7fcb45eff700 1 -- 172.21.2.117:0/2915814011 wait complete. 2021-02-10T11:40:11.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:01.847956', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:02.928003', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:11.413 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:11.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.846 7f661e95c700 1 Processor -- start 2021-02-10T11:40:11.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.846 7f661e95c700 1 -- start start 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f661e95c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661807da20 0x7f66180771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f661e95c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66180d1d20 con 0x7f66180e4da0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f661e95c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6618077a70 con 0x7f66180e51a0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f661e95c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180777d0 con 0x7f661807da20 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f66177fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f66180e4da0 0x7f66181b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47962/0 (socket says 172.21.2.117:47962) 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f66177fe700 1 -- 172.21.2.117:0/1965463325 learned_addr learned my addr 172.21.2.117:0/1965463325 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050234731 0 0) 0x7f66000010b0 con 0x7f66180e4da0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.847 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6604002fe0 con 0x7f66180e4da0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6617fff700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661807da20 0x7f66180771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3976640122 0 0) 0x7f66000010b0 con 0x7f66180e4da0 2021-02-10T11:40:11.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661807da20 msgr2=0x7f66180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661807da20 0x7f66180771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 >> v1:172.21.2.125:6789/0 conn(0x7f66180e51a0 legacy=0x7f66180d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66181b08e0 con 0x7f66180e4da0 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f66000015b0 con 0x7f66180e4da0 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 >> v1:172.21.2.117:6789/0 conn(0x7f66180e4da0 legacy=0x7f66181b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6604003af0 0x7f6604005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6604006800 0x7f6604008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66040095a0 0x7f660400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660400c370 con 0x7f6604006800 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660400c590 con 0x7f66040095a0 2021-02-10T11:40:11.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660400c7b0 con 0x7f6604003af0 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6617fff700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661807da20 0x7f66180771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6604006800 0x7f6604008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f661cf79700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66040095a0 0x7f660400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f6617fff700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6604003af0 0x7f6604005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 -- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6604003af0 msgr2=0x7f6604005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6604003af0 0x7f6604005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 -- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66040095a0 msgr2=0x7f660400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66040095a0 0x7f660400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f66177fe700 1 -- 172.21.2.117:0/1965463325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660400ca50 con 0x7f6604006800 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.848 7f661cf79700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66040095a0 0x7f660400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f66177fe700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6604006800 0x7f6604008d00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f66000017d0 tx=0x7f6600007d30).ready entity=mon.0 client_cookie=75e487d585934563 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660000a310 con 0x7f6604006800 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660000a650 con 0x7f6604006800 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f6616ffd700 1 -- 172.21.2.117:0/1965463325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660000ab30 con 0x7f6604006800 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 -- 172.21.2.117:0/1965463325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6604006800 msgr2=0x7f6604008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 --2- 172.21.2.117:0/1965463325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6604006800 0x7f6604008d00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f66000017d0 tx=0x7f6600007d30).stop 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661cf79700 1 -- 172.21.2.117:0/1965463325 reap_dead start 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 -- 172.21.2.117:0/1965463325 shutdown_connections 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 -- 172.21.2.117:0/1965463325 >> 172.21.2.117:0/1965463325 conn(0x7f66180ce450 msgr2=0x7f66180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 -- 172.21.2.117:0/1965463325 shutdown_connections 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.849 7f661e95c700 1 -- 172.21.2.117:0/1965463325 wait complete. 2021-02-10T11:40:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 Processor -- start 2021-02-10T11:40:11.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 -- start start 2021-02-10T11:40:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66180a46c0 0x7f66180a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66180a50e0 0x7f66180a6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66180c3040 con 0x7f66181a46b0 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618190a20 con 0x7f66180a46c0 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.864 7f661e95c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66180a54e0 con 0x7f66180a50e0 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47722/0 (socket says 172.21.2.117:47722) 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 -- 172.21.2.117:0/220339502 learned_addr learned my addr 172.21.2.117:0/220339502 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 -- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66180a50e0 msgr2=0x7f66180a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66180a50e0 0x7f66180a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 -- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66180a46c0 msgr2=0x7f66180a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66180a46c0 0x7f66180a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 -- 172.21.2.117:0/220339502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66180a5700 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6617fff700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f660804e920 tx=0x7f660804ecc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6608006a50 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6608006d90 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f661e95c700 1 -- 172.21.2.117:0/220339502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66180a5700 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66080097a0 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.865 7f661e95c700 1 -- 172.21.2.117:0/220339502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66180a5700 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.866 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6608009310 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.866 7f6614ff9700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 0x7f65fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.866 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6608076ae0 con 0x7f66181a46b0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.866 7f66177fe700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 0x7f65fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:11.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.867 7f66177fe700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 0x7f65fc01ac10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:11.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.896 7f661e95c700 1 -- 172.21.2.117:0/220339502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6618077a70 con 0x7f66181a46b0 2021-02-10T11:40:11.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:11.899 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6608006d70 con 0x7f66181a46b0 2021-02-10T11:40:12.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.026 7f661e95c700 1 -- 172.21.2.117:0/220339502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f6618077a70 con 0x7f66181a46b0 2021-02-10T11:40:12.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.026 7f6614ff9700 1 -- 172.21.2.117:0/220339502 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v39) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6608009310 con 0x7f66181a46b0 2021-02-10T11:40:12.008 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 msgr2=0x7f65fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 0x7f65fc01ac10 crc :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 msgr2=0x7f6618190400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f660804e920 tx=0x7f660804ecc0).stop 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 shutdown_connections 2021-02-10T11:40:12.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66180a50e0 0x7f66180a6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f66180a46c0 0x7f66180a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65fc018730 0x7f65fc01ac10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 --2- 172.21.2.117:0/220339502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66181a46b0 0x7f6618190400 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 >> 172.21.2.117:0/220339502 conn(0x7f66180b5140 msgr2=0x7f66180a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 shutdown_connections 2021-02-10T11:40:12.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.028 7f661e95c700 1 -- 172.21.2.117:0/220339502 wait complete. 2021-02-10T11:40:12.018 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:12.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 Processor -- start 2021-02-10T11:40:12.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 -- start start 2021-02-10T11:40:12.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e600d1530 0x7f6e600d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e60077a70 con 0x7f6e60079000 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e60079400 con 0x7f6e6007da20 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.450 7f6e650e3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6007de20 con 0x7f6e600d1530 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6e60079000 0x7f6e600e4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:47982/0 (socket says 172.21.2.117:47982) 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5e59c700 1 -- 172.21.2.117:0/1871513933 learned_addr learned my addr 172.21.2.117:0/1871513933 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5f59e700 1 --2- 172.21.2.117:0/1871513933 >> v2:172.21.2.125:3300/0 conn(0x7f6e600d1530 0x7f6e600d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5dd9b700 1 -- 172.21.2.117:0/1871513933 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351679180 0 0) 0x7f6e480010b0 con 0x7f6e60079000 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5dd9b700 1 -- 172.21.2.117:0/1871513933 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c002fe0 con 0x7f6e60079000 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5f59e700 1 -- 172.21.2.117:0/1871513933 >> v1:172.21.2.125:6789/0 conn(0x7f6e6007da20 legacy=0x7f6e600771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5f59e700 1 -- 172.21.2.117:0/1871513933 >> v1:172.21.2.117:6789/0 conn(0x7f6e60079000 legacy=0x7f6e600e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.451 7f6e5f59e700 1 -- 172.21.2.117:0/1871513933 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e601b00f0 con 0x7f6e600d1530 2021-02-10T11:40:12.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e5f59e700 1 --2- 172.21.2.117:0/1871513933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e600d1530 0x7f6e600d1930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6e58006e50 tx=0x7f6e58009e70).ready entity=mon.1 client_cookie=758e633a57f15f3d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e5dd9b700 1 -- 172.21.2.117:0/1871513933 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e58006a20 con 0x7f6e600d1530 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e5dd9b700 1 -- 172.21.2.117:0/1871513933 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58006790 con 0x7f6e600d1530 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e5dd9b700 1 -- 172.21.2.117:0/1871513933 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5800b700 con 0x7f6e600d1530 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e650e3700 1 -- 172.21.2.117:0/1871513933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e600d1530 msgr2=0x7f6e600d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e650e3700 1 --2- 172.21.2.117:0/1871513933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e600d1530 0x7f6e600d1930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6e58006e50 tx=0x7f6e58009e70).stop 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e650e3700 1 -- 172.21.2.117:0/1871513933 shutdown_connections 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.452 7f6e650e3700 1 --2- 172.21.2.117:0/1871513933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e600d1530 0x7f6e600d1930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.453 7f6e650e3700 1 -- 172.21.2.117:0/1871513933 >> 172.21.2.117:0/1871513933 conn(0x7f6e600ce450 msgr2=0x7f6e600e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.453 7f6e650e3700 1 -- 172.21.2.117:0/1871513933 shutdown_connections 2021-02-10T11:40:12.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.453 7f6e650e3700 1 -- 172.21.2.117:0/1871513933 wait complete. 2021-02-10T11:40:12.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.467 7f6e650e3700 1 Processor -- start 2021-02-10T11:40:12.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.467 7f6e650e3700 1 -- start start 2021-02-10T11:40:12.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 0x7f6e60094b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e60094080 0x7f6e60095120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60095740 con 0x7f6e60093c80 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60095960 con 0x7f6e601a7bf0 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e650e3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60095b80 con 0x7f6e60094080 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47736/0 (socket says 172.21.2.117:47736) 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 -- 172.21.2.117:0/1773280723 learned_addr learned my addr 172.21.2.117:0/1773280723 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5e59c700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 0x7f6e60094b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5f59e700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e60094080 0x7f6e60095120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 -- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e60094080 msgr2=0x7f6e60095120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e60094080 0x7f6e60095120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 -- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 msgr2=0x7f6e60094b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 0x7f6e60094b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 -- 172.21.2.117:0/1773280723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60095da0 con 0x7f6e60093c80 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5e59c700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 0x7f6e60094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.468 7f6e5ed9d700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f6e5000b5b0 tx=0x7f6e5000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e5000bd40 con 0x7f6e60093c80 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e50006020 con 0x7f6e60093c80 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e500054a0 con 0x7f6e60093c80 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e60095da0 con 0x7f6e60093c80 2021-02-10T11:40:12.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e60095da0 con 0x7f6e60093c80 2021-02-10T11:40:12.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.469 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e50006200 con 0x7f6e60093c80 2021-02-10T11:40:12.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.470 7f6e577fe700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 0x7f6e4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:12.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.470 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e500326c0 con 0x7f6e60093c80 2021-02-10T11:40:12.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.470 7f6e5e59c700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 0x7f6e4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:12.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.470 7f6e5e59c700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 0x7f6e4401abc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:12.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.500 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e60103310 con 0x7f6e60093c80 2021-02-10T11:40:12.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.503 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e50030d20 con 0x7f6e60093c80 2021-02-10T11:40:12.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.630 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f6e60103310 con 0x7f6e60093c80 2021-02-10T11:40:12.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.630 7f6e577fe700 1 -- 172.21.2.117:0/1773280723 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v39) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6e5000bd40 con 0x7f6e60093c80 2021-02-10T11:40:12.612 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:12.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 msgr2=0x7f6e4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 0x7f6e4401abc0 crc :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 msgr2=0x7f6e601a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f6e5000b5b0 tx=0x7f6e5000b950).stop 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 shutdown_connections 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e60094080 0x7f6e60095120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e601a7bf0 0x7f6e60094b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e440186c0 0x7f6e4401abc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 --2- 172.21.2.117:0/1773280723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e60093c80 0x7f6e601a75d0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:12.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 >> 172.21.2.117:0/1773280723 conn(0x7f6e600a1880 msgr2=0x7f6e600b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:12.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 shutdown_connections 2021-02-10T11:40:12.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:12.632 7f6e650e3700 1 -- 172.21.2.117:0/1773280723 wait complete. 2021-02-10T11:40:12.622 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 Processor -- start 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 -- start start 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fba041b00f0 0x7fba041b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba04077a70 con 0x7fba0407da20 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba040e53c0 con 0x7fba041b00f0 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.054 7fba0a620700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba04079400 con 0x7fba04079000 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba037fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba08c3d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fba041b00f0 0x7fba041b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba08c3d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fba041b00f0 0x7fba041b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:52128/0 (socket says 172.21.2.117:52128) 2021-02-10T11:40:13.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba037fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:45424/0 (socket says 172.21.2.117:45424) 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba08c3d700 1 -- 172.21.2.117:0/2276457792 learned_addr learned my addr 172.21.2.117:0/2276457792 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.055 7fba03fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fba0407da20 0x7fba040771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40920/0 (socket says 172.21.2.117:40920) 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba08c3d700 1 -- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 msgr2=0x7fba040e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba08c3d700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba08c3d700 1 -- 172.21.2.117:0/2276457792 >> v1:172.21.2.125:6789/0 conn(0x7fba0407da20 legacy=0x7fba040771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba08c3d700 1 -- 172.21.2.117:0/2276457792 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0407de20 con 0x7fba041b00f0 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba02ffd700 1 -- 172.21.2.117:0/2276457792 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996529966 0 0) 0x7fb9f40010b0 con 0x7fba0407da20 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba037fe700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba08c3d700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041b00f0 0x7fba041b04f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb9fc006aa0 tx=0x7fb9fc009f20).ready entity=mon.1 client_cookie=1bcdae0c49851349 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba02ffd700 1 -- 172.21.2.117:0/2276457792 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9fc000f90 con 0x7fba041b00f0 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba02ffd700 1 -- 172.21.2.117:0/2276457792 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc003030 con 0x7fba041b00f0 2021-02-10T11:40:13.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba02ffd700 1 -- 172.21.2.117:0/2276457792 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9fc00b7b0 con 0x7fba041b00f0 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba0a620700 1 -- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041b00f0 msgr2=0x7fba041b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.056 7fba0a620700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041b00f0 0x7fba041b04f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fb9fc006aa0 tx=0x7fb9fc009f20).stop 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 -- 172.21.2.117:0/2276457792 shutdown_connections 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041b00f0 0x7fba041b04f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 --2- 172.21.2.117:0/2276457792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04079000 0x7fba040e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 -- 172.21.2.117:0/2276457792 >> 172.21.2.117:0/2276457792 conn(0x7fba040ce450 msgr2=0x7fba040e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 -- 172.21.2.117:0/2276457792 shutdown_connections 2021-02-10T11:40:13.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.057 7fba0a620700 1 -- 172.21.2.117:0/2276457792 wait complete. 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 Processor -- start 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 -- start start 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041a0250 0x7fba04106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04090420 0x7fba04090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba041067f0 con 0x7fba04187e20 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba04107350 con 0x7fba041a0250 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba0a620700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba041042d0 con 0x7fba04090420 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba037fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba037fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47748/0 (socket says 172.21.2.117:47748) 2021-02-10T11:40:13.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.072 7fba037fe700 1 -- 172.21.2.117:0/114834763 learned_addr learned my addr 172.21.2.117:0/114834763 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 -- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04090420 msgr2=0x7fba04090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04090420 0x7fba04090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 -- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041a0250 msgr2=0x7fba04106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041a0250 0x7fba04106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 -- 172.21.2.117:0/114834763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba04090e40 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba037fe700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7fb9f8006f40 tx=0x7fb9f800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9f800bd80 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8000c90 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9f8002220 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba0a620700 1 -- 172.21.2.117:0/114834763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba04090e40 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.073 7fba0a620700 1 -- 172.21.2.117:0/114834763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba04090e40 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.074 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9f8006b00 con 0x7fba04187e20 2021-02-10T11:40:13.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.074 7fba00ff9700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 0x7fb9e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.074 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb9f8032920 con 0x7fba04187e20 2021-02-10T11:40:13.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.074 7fba03fff700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 0x7fb9e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.075 7fba03fff700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 0x7fb9e401abc0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.104 7fba0a620700 1 -- 172.21.2.117:0/114834763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba04103280 con 0x7fba04187e20 2021-02-10T11:40:13.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.107 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9f8000c90 con 0x7fba04187e20 2021-02-10T11:40:13.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.233 7fba0a620700 1 -- 172.21.2.117:0/114834763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fba04103280 con 0x7fba04187e20 2021-02-10T11:40:13.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.234 7fba00ff9700 1 -- 172.21.2.117:0/114834763 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v39) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb9f800bd80 con 0x7fba04187e20 2021-02-10T11:40:13.215 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.235 7fba0a620700 1 -- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 msgr2=0x7fb9e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.235 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 0x7fb9e401abc0 crc :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.235 7fba0a620700 1 -- 172.21.2.117:0/114834763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 msgr2=0x7fba04188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.235 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7fb9f8006f40 tx=0x7fb9f800b990).stop 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 -- 172.21.2.117:0/114834763 shutdown_connections 2021-02-10T11:40:13.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9e4018730 0x7fb9e401abc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba04187e20 0x7fba04188220 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba04090420 0x7fba04090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 --2- 172.21.2.117:0/114834763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba041a0250 0x7fba04106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 -- 172.21.2.117:0/114834763 >> 172.21.2.117:0/114834763 conn(0x7fba040a1850 msgr2=0x7fba040c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 -- 172.21.2.117:0/114834763 shutdown_connections 2021-02-10T11:40:13.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.236 7fba0a620700 1 -- 172.21.2.117:0/114834763 wait complete. 2021-02-10T11:40:13.226 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:13.226 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.653 7f4ce8b19700 1 Processor -- start 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.653 7f4ce8b19700 1 -- start start 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce8b19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce407da20 0x7f4ce40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce8b19700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4ce40e4d90 0x7f4ce41b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce8b19700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce4077a70 con 0x7f4ce40e5190 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce8b19700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce40e5c10 con 0x7f4ce40e4d90 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce8b19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40777d0 con 0x7f4ce407da20 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce285e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce407da20 0x7f4ce40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce205d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4ce40e4d90 0x7f4ce41b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce305f700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f4ce40e5190 0x7f4ce4077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40934/0 (socket says 172.21.2.117:40934) 2021-02-10T11:40:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.654 7f4ce305f700 1 -- 172.21.2.117:0/2352618063 learned_addr learned my addr 172.21.2.117:0/2352618063 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce185c700 1 -- 172.21.2.117:0/2352618063 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311908518 0 0) 0x7f4cdc001440 con 0x7f4ce40e5190 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce185c700 1 -- 172.21.2.117:0/2352618063 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd0002fe0 con 0x7f4ce40e5190 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce205d700 1 -- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce407da20 msgr2=0x7f4ce40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce205d700 1 --2- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce407da20 0x7f4ce40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce205d700 1 -- 172.21.2.117:0/2352618063 >> v1:172.21.2.125:6789/0 conn(0x7f4ce40e5190 legacy=0x7f4ce4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce205d700 1 -- 172.21.2.117:0/2352618063 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce407de20 con 0x7f4ce40e4d90 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce205d700 1 --2- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce40e4d90 0x7f4ce41b00e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f4cd4006650 tx=0x7f4cd4009a80).ready entity=mon.1 client_cookie=10cfe625bede4be2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce185c700 1 -- 172.21.2.117:0/2352618063 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd400c0c0 con 0x7f4ce40e4d90 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce185c700 1 -- 172.21.2.117:0/2352618063 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd400c340 con 0x7f4ce40e4d90 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce185c700 1 -- 172.21.2.117:0/2352618063 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd400c860 con 0x7f4ce40e4d90 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce8b19700 1 -- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce40e4d90 msgr2=0x7f4ce41b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.655 7f4ce8b19700 1 --2- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce40e4d90 0x7f4ce41b00e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f4cd4006650 tx=0x7f4cd4009a80).stop 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 -- 172.21.2.117:0/2352618063 shutdown_connections 2021-02-10T11:40:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 --2- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce40e4d90 0x7f4ce41b00e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 --2- 172.21.2.117:0/2352618063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce407da20 0x7f4ce40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 -- 172.21.2.117:0/2352618063 >> 172.21.2.117:0/2352618063 conn(0x7f4ce40ce440 msgr2=0x7f4ce40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 -- 172.21.2.117:0/2352618063 shutdown_connections 2021-02-10T11:40:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.656 7f4ce8b19700 1 -- 172.21.2.117:0/2352618063 wait complete. 2021-02-10T11:40:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.670 7f4ce8b19700 1 Processor -- start 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.670 7f4ce8b19700 1 -- start start 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce409dd40 0x7f4ce40b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce408c080 0x7f4ce408c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce409d8a0 con 0x7f4ce408caa0 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b8bc0 con 0x7f4ce408c080 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce8b19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b8de0 con 0x7f4ce409dd40 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47762/0 (socket says 172.21.2.117:47762) 2021-02-10T11:40:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce305f700 1 -- 172.21.2.117:0/1968159041 learned_addr learned my addr 172.21.2.117:0/1968159041 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce205d700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce408c080 0x7f4ce408c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.671 7f4ce285e700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce409dd40 0x7f4ce40b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 -- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce409dd40 msgr2=0x7f4ce40b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce409dd40 0x7f4ce40b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 -- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce408c080 msgr2=0x7f4ce408c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce408c080 0x7f4ce408c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 -- 172.21.2.117:0/1968159041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce408cea0 con 0x7f4ce408caa0 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce305f700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f4cdc00a940 tx=0x7f4cdc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cdc0048b0 con 0x7f4ce408caa0 2021-02-10T11:40:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc004af0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cdc0070d0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce408cea0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.672 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce408fed0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.673 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4cdc004cd0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.673 7f4cdb7fe700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 0x7f4cc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.673 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4cdc0328f0 con 0x7f4ce408caa0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.673 7f4ce285e700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 0x7f4cc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:13.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.674 7f4ce285e700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 0x7f4cc801abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:13.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.704 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce4077a70 con 0x7f4ce408caa0 2021-02-10T11:40:13.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.706 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4cdc000e20 con 0x7f4ce408caa0 2021-02-10T11:40:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.836 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ce4077a70 con 0x7f4ce408caa0 2021-02-10T11:40:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.836 7f4cdb7fe700 1 -- 172.21.2.117:0/1968159041 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2096 (secure 0 0 0) 0x7f4cdc004cb0 con 0x7f4ce408caa0 2021-02-10T11:40:13.818 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:13.818 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:05.887046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7233":{"gid":7233,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/107410323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":107410323},{"type":"v1","addr":"172.21.2.125:6835","nonce":107410323}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 20 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 msgr2=0x7f4cc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 0x7f4cc801abc0 crc :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 msgr2=0x7f4ce408f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7f4cdc00a940 tx=0x7f4cdc00ace0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 shutdown_connections 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc80186c0 0x7f4cc801abc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce408caa0 0x7f4ce408f5c0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce408c080 0x7f4ce408c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 --2- 172.21.2.117:0/1968159041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce409dd40 0x7f4ce40b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 >> 172.21.2.117:0/1968159041 conn(0x7f4ce419f4c0 msgr2=0x7f4ce4094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 shutdown_connections 2021-02-10T11:40:13.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:13.838 7f4ce8b19700 1 -- 172.21.2.117:0/1968159041 wait complete. 2021-02-10T11:40:14.029 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:03.881036', 'modified': '2021-02-10 11:40:05.887046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7233}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7233': {'gid': 7233, 'name': 'b', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/107410323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 107410323}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 107410323}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:14.030 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:14.030 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:14.428 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:40:14.429 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:40:14.429 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:40:14.429 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7233, 2021-02-10T11:40:14.429 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:40:14.429 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.565378199, 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 20, 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 39, 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.199385943999999 2021-02-10T11:40:14.430 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:40:14.433 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7233, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.565378199, "mdsmap_epoch": 20, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 16.199385943999999 } 2021-02-10T11:40:14.433 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:14.433 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:40:14.433 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:40:14.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:14.434 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:14.771 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:40:15.114 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:15.115 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:15.456 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:15.456 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:15.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:15.458 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:40:15.792 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:40:15.793 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:15.795 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:40:16.139 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:16.139 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:16.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[121196]: starting ceph client 2021-02-10T11:40:16.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:40:16.223 7f43e08c2f80 -1 init, newargv = 0x559db2263ae0 newargc=9 2021-02-10T11:40:16.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[121196]: starting fuse 2021-02-10T11:40:16.482 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:16.483 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:16.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:16.484 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:40:16.819 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:40:16.819 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:40:16.819 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:40:16.820 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:40:16.821 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:40:17.182 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.121196.asok 2021-02-10T11:40:17.184 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.121196.asok 2021-02-10T11:40:17.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:17.185 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.121196.asok status 2021-02-10T11:40:17.572 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:40:17.573 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "121196", 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:17.574 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 7386, 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 7386 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:17.575 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2129096392 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:17.576 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2129096392 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.7386 172.21.2.117:0/2129096392", 2021-02-10T11:40:17.577 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2129096392", 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 20, 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 39, 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:40:17.578 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:40:17.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:17.580 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:17.580 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:17.915 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:40:17.916 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:17.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:17.916 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:18.261 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 Processor -- start 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 -- start start 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 0x7ffae80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 0x7ffae81b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffae80e5190 0x7ffae8077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae8077a70 con 0x7ffae80e5190 2021-02-10T11:40:18.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae80777d0 con 0x7ffae807da20 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffaedb27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae807de20 con 0x7ffae80e4d90 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffae7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffae80e5190 0x7ffae8077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffae7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffae80e5190 0x7ffae8077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47782/0 (socket says 172.21.2.117:47782) 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.691 7ffae7fff700 1 -- 172.21.2.117:0/2601424824 learned_addr learned my addr 172.21.2.117:0/2601424824 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae6ffd700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 0x7ffae81b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 -- 172.21.2.117:0/2601424824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 msgr2=0x7ffae81b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 0x7ffae81b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae77fe700 1 --2- 172.21.2.117:0/2601424824 >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 0x7ffae80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 -- 172.21.2.117:0/2601424824 >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 msgr2=0x7ffae80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 --2- 172.21.2.117:0/2601424824 >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 0x7ffae80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 -- 172.21.2.117:0/2601424824 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae81b0700 con 0x7ffae80e5190 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae6ffd700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 0x7ffae81b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:18.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae77fe700 1 --2- 172.21.2.117:0/2601424824 >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 0x7ffae80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae7fff700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae80e5190 0x7ffae8077ff0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7ffae0006f00 tx=0x7ffae0009f10).ready entity=mon.0 client_cookie=85e6767da342cdab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae67fc700 1 -- 172.21.2.117:0/2601424824 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae000b3b0 con 0x7ffae80e5190 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae67fc700 1 -- 172.21.2.117:0/2601424824 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0000d30 con 0x7ffae80e5190 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffae67fc700 1 -- 172.21.2.117:0/2601424824 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffae000b820 con 0x7ffae80e5190 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffaedb27700 1 -- 172.21.2.117:0/2601424824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae80e5190 msgr2=0x7ffae8077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.692 7ffaedb27700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae80e5190 0x7ffae8077ff0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7ffae0006f00 tx=0x7ffae0009f10).stop 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 -- 172.21.2.117:0/2601424824 shutdown_connections 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae80e5190 0x7ffae8077ff0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 --2- 172.21.2.117:0/2601424824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae80e4d90 0x7ffae81b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 --2- 172.21.2.117:0/2601424824 >> v2:172.21.2.125:3300/0 conn(0x7ffae807da20 0x7ffae80771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 -- 172.21.2.117:0/2601424824 >> 172.21.2.117:0/2601424824 conn(0x7ffae80ce440 msgr2=0x7ffae80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 -- 172.21.2.117:0/2601424824 shutdown_connections 2021-02-10T11:40:18.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.693 7ffaedb27700 1 -- 172.21.2.117:0/2601424824 wait complete. 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 Processor -- start 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 -- start start 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae8192ba0 0x7ffae8192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffae818cbd0 0x7ffae818cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8193730 con 0x7ffae81067a0 2021-02-10T11:40:18.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae818d630 con 0x7ffae818cbd0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffaedb27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae818d850 con 0x7ffae8192ba0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffae6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffae6ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47788/0 (socket says 172.21.2.117:47788) 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.734 7ffae6ffd700 1 -- 172.21.2.117:0/461692788 learned_addr learned my addr 172.21.2.117:0/461692788 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 -- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae8192ba0 msgr2=0x7ffae8192fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae8192ba0 0x7ffae8192fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 -- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffae818cbd0 msgr2=0x7ffae818cfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffae818cbd0 0x7ffae818cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 -- 172.21.2.117:0/461692788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae818dc30 con 0x7ffae81067a0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffae6ffd700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7ffadc006f20 tx=0x7ffadc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffadc00bd80 con 0x7ffae81067a0 2021-02-10T11:40:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc000c90 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffadc002220 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffaedb27700 1 -- 172.21.2.117:0/461692788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae818dc30 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.735 7ffaedb27700 1 -- 172.21.2.117:0/461692788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae818dc30 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.736 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffadc00a000 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.736 7ffacffff700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 0x7ffac801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.736 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffadc0328e0 con 0x7ffae81067a0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.736 7ffae77fe700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 0x7ffac801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.736 7ffae77fe700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 0x7ffac801ac10 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:18.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.741 7ffaedb27700 1 -- 172.21.2.117:0/461692788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae8077a70 con 0x7ffae81067a0 2021-02-10T11:40:18.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.743 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffadc006bb0 con 0x7ffae81067a0 2021-02-10T11:40:18.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.866 7ffaedb27700 1 -- 172.21.2.117:0/461692788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffae8077a70 con 0x7ffae81067a0 2021-02-10T11:40:18.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.867 7ffacffff700 1 -- 172.21.2.117:0/461692788 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+14226 (secure 0 0 0) 0x7ffadc000c90 con 0x7ffae81067a0 2021-02-10T11:40:18.850 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:18.850 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":39,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:04.882873","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:01.847956","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:02.928003","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.868 7ffaedb27700 1 -- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 msgr2=0x7ffac801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.868 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 0x7ffac801ac10 crc :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.868 7ffaedb27700 1 -- 172.21.2.117:0/461692788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 msgr2=0x7ffae8106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.868 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7ffadc006f20 tx=0x7ffadc00b990).stop 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 -- 172.21.2.117:0/461692788 shutdown_connections 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffac8018730 0x7ffac801ac10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffae81067a0 0x7ffae8106ba0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffae818cbd0 0x7ffae818cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 --2- 172.21.2.117:0/461692788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffae8192ba0 0x7ffae8192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 -- 172.21.2.117:0/461692788 >> 172.21.2.117:0/461692788 conn(0x7ffae819f510 msgr2=0x7ffae81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:18.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 -- 172.21.2.117:0/461692788 shutdown_connections 2021-02-10T11:40:18.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:18.869 7ffaedb27700 1 -- 172.21.2.117:0/461692788 wait complete. 2021-02-10T11:40:18.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:01.847956', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:02.928003', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:40:18.860 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.292 7f3d4ef76700 1 Processor -- start 2021-02-10T11:40:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.292 7f3d4ef76700 1 -- start start 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4ef76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d48079000 0x7f3d480e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4ef76700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3d481b00d0 0x7f3d481b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4ef76700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d48077a70 con 0x7f3d4807da20 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4ef76700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d480e53a0 con 0x7f3d481b00d0 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4ef76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48079400 con 0x7f3d48079000 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d47fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d48079000 0x7f3d480e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4cd92700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f3d4807da20 0x7f3d480771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:40972/0 (socket says 172.21.2.117:40972) 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4cd92700 1 -- 172.21.2.117:0/1229799857 learned_addr learned my addr 172.21.2.117:0/1229799857 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4d593700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3d481b00d0 0x7f3d481b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4d593700 1 -- 172.21.2.117:0/1229799857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d48079000 msgr2=0x7f3d480e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4d593700 1 --2- 172.21.2.117:0/1229799857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d48079000 0x7f3d480e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4d593700 1 -- 172.21.2.117:0/1229799857 >> v1:172.21.2.125:6789/0 conn(0x7f3d4807da20 legacy=0x7f3d480771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d477fe700 1 -- 172.21.2.117:0/1229799857 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715599195 0 0) 0x7f3d300010b0 con 0x7f3d4807da20 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.293 7f3d4d593700 1 -- 172.21.2.117:0/1229799857 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4807de20 con 0x7f3d481b00d0 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4d593700 1 --2- 172.21.2.117:0/1229799857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d481b00d0 0x7f3d481b04d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f3d40006e30 tx=0x7f3d40009de0).ready entity=mon.0 client_cookie=2bbbe9b5cceae42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d477fe700 1 -- 172.21.2.117:0/1229799857 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d4000b300 con 0x7f3d481b00d0 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d477fe700 1 -- 172.21.2.117:0/1229799857 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4000b580 con 0x7f3d481b00d0 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d477fe700 1 -- 172.21.2.117:0/1229799857 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d4000baa0 con 0x7f3d481b00d0 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 -- 172.21.2.117:0/1229799857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d481b00d0 msgr2=0x7f3d481b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 --2- 172.21.2.117:0/1229799857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d481b00d0 0x7f3d481b04d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f3d40006e30 tx=0x7f3d40009de0).stop 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 -- 172.21.2.117:0/1229799857 shutdown_connections 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 --2- 172.21.2.117:0/1229799857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d481b00d0 0x7f3d481b04d0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 --2- 172.21.2.117:0/1229799857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d48079000 0x7f3d480e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 -- 172.21.2.117:0/1229799857 >> 172.21.2.117:0/1229799857 conn(0x7f3d480ce430 msgr2=0x7f3d480e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 -- 172.21.2.117:0/1229799857 shutdown_connections 2021-02-10T11:40:19.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.294 7f3d4ef76700 1 -- 172.21.2.117:0/1229799857 wait complete. 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 Processor -- start 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 -- start start 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d480a1310 0x7f3d4818a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d480b82f0 0x7f3d48104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d480ba970 con 0x7f3d480b78d0 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48104b90 con 0x7f3d480a1310 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.309 7f3d4ef76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48104d70 con 0x7f3d480b82f0 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47804/0 (socket says 172.21.2.117:47804) 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 -- 172.21.2.117:0/4116886838 learned_addr learned my addr 172.21.2.117:0/4116886838 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 -- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d480b82f0 msgr2=0x7f3d48104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d480b82f0 0x7f3d48104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 -- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d480a1310 msgr2=0x7f3d4818a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d480a1310 0x7f3d4818a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 -- 172.21.2.117:0/4116886838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48104f90 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d47fff700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f3d38006f60 tx=0x7f3d3800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d3800bd60 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d38000c90 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3d380022b0 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.310 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d48104f90 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.311 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d48104f90 con 0x7f3d480b78d0 2021-02-10T11:40:19.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.311 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3d38006b00 con 0x7f3d480b78d0 2021-02-10T11:40:19.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.311 7f3d457fa700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 0x7f3d2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.311 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3d38032680 con 0x7f3d480b78d0 2021-02-10T11:40:19.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.312 7f3d4cd92700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 0x7f3d2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.312 7f3d4cd92700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 0x7f3d2c01abc0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.342 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d48077a70 con 0x7f3d480b78d0 2021-02-10T11:40:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.345 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3d38030ce0 con 0x7f3d480b78d0 2021-02-10T11:40:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.472 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3d48077a70 con 0x7f3d480b78d0 2021-02-10T11:40:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.472 7f3d457fa700 1 -- 172.21.2.117:0/4116886838 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v39) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3d38006b00 con 0x7f3d480b78d0 2021-02-10T11:40:19.455 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:19.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 msgr2=0x7f3d2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 0x7f3d2c01abc0 crc :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 msgr2=0x7f3d480b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f3d38006f60 tx=0x7f3d3800b970).stop 2021-02-10T11:40:19.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 shutdown_connections 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3d2c0186c0 0x7f3d2c01abc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3d480b78d0 0x7f3d480b7cd0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3d480b82f0 0x7f3d48104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 --2- 172.21.2.117:0/4116886838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3d480a1310 0x7f3d4818a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 >> 172.21.2.117:0/4116886838 conn(0x7f3d480bacb0 msgr2=0x7f3d4808e590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 shutdown_connections 2021-02-10T11:40:19.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.474 7f3d4ef76700 1 -- 172.21.2.117:0/4116886838 wait complete. 2021-02-10T11:40:19.465 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.896 7fe105f35700 1 Processor -- start 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.896 7fe105f35700 1 -- start start 2021-02-10T11:40:19.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe105f35700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe10007da20 0x7fe1000771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe105f35700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1000e51a0 0x7fe100079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe105f35700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe100077a70 con 0x7fe1000e4da0 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe105f35700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000e5c20 con 0x7fe1000e51a0 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe105f35700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1000777d0 con 0x7fe10007da20 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1000e51a0 0x7fe100079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1000e51a0 0x7fe100079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47808/0 (socket says 172.21.2.117:47808) 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 -- 172.21.2.117:0/3784241856 learned_addr learned my addr 172.21.2.117:0/3784241856 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 -- 172.21.2.117:0/3784241856 >> v2:172.21.2.125:3300/0 conn(0x7fe10007da20 msgr2=0x7fe1000771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 --2- 172.21.2.117:0/3784241856 >> v2:172.21.2.125:3300/0 conn(0x7fe10007da20 0x7fe1000771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 -- 172.21.2.117:0/3784241856 >> v1:172.21.2.117:6789/0 conn(0x7fe1000e4da0 legacy=0x7fe1001b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 -- 172.21.2.117:0/3784241856 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1001b08e0 con 0x7fe1000e51a0 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.897 7fe0fffff700 1 --2- 172.21.2.117:0/3784241856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1000e51a0 0x7fe100079000 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7fe0f8006f00 tx=0x7fe0f8009f10).ready entity=mon.0 client_cookie=f6a6f5b83c043ff9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe0fe7fc700 1 -- 172.21.2.117:0/3784241856 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0f800b3b0 con 0x7fe1000e51a0 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe0fe7fc700 1 -- 172.21.2.117:0/3784241856 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f800b630 con 0x7fe1000e51a0 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe0fe7fc700 1 -- 172.21.2.117:0/3784241856 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0f800bb10 con 0x7fe1000e51a0 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 -- 172.21.2.117:0/3784241856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1000e51a0 msgr2=0x7fe100079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 --2- 172.21.2.117:0/3784241856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1000e51a0 0x7fe100079000 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7fe0f8006f00 tx=0x7fe0f8009f10).stop 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 -- 172.21.2.117:0/3784241856 shutdown_connections 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 --2- 172.21.2.117:0/3784241856 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1000e51a0 0x7fe100079000 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 --2- 172.21.2.117:0/3784241856 >> v2:172.21.2.125:3300/0 conn(0x7fe10007da20 0x7fe1000771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 -- 172.21.2.117:0/3784241856 >> 172.21.2.117:0/3784241856 conn(0x7fe1000ce450 msgr2=0x7fe1000e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:19.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 -- 172.21.2.117:0/3784241856 shutdown_connections 2021-02-10T11:40:19.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.898 7fe105f35700 1 -- 172.21.2.117:0/3784241856 wait complete. 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 Processor -- start 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 -- start start 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10008a7a0 0x7fe1000a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1000a7be0 0x7fe1000a7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10008cfd0 con 0x7fe1001a47d0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10008d550 con 0x7fe10008a7a0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe105f35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10008d730 con 0x7fe1000a7be0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe0ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe0ff7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47812/0 (socket says 172.21.2.117:47812) 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.913 7fe0ff7fe700 1 -- 172.21.2.117:0/3291514129 learned_addr learned my addr 172.21.2.117:0/3291514129 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 -- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1000a7be0 msgr2=0x7fe1000a7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1000a7be0 0x7fe1000a7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 -- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10008a7a0 msgr2=0x7fe1000a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10008a7a0 0x7fe1000a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 -- 172.21.2.117:0/3291514129 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10008d950 con 0x7fe1001a47d0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0ff7fe700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7fe0e8006f00 tx=0x7fe0e800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e8006b90 con 0x7fe1001a47d0 2021-02-10T11:40:19.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e800bd30 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0e80020d0 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe105f35700 1 -- 172.21.2.117:0/3291514129 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10008d950 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.914 7fe105f35700 1 -- 172.21.2.117:0/3291514129 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10008dee0 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.915 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0e8007020 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.915 7fe0f7fff700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 0x7fe0e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.915 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe0e80324f0 con 0x7fe1001a47d0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.915 7fe0feffd700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 0x7fe0e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:19.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.916 7fe0feffd700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 0x7fe0e401abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:19.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.947 7fe105f35700 1 -- 172.21.2.117:0/3291514129 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1001032d0 con 0x7fe1001a47d0 2021-02-10T11:40:19.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:19.949 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe0e8007ef0 con 0x7fe1001a47d0 2021-02-10T11:40:20.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.074 7fe105f35700 1 -- 172.21.2.117:0/3291514129 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe100077a70 con 0x7fe1001a47d0 2021-02-10T11:40:20.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.075 7fe0f7fff700 1 -- 172.21.2.117:0/3291514129 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v39) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe0e8000940 con 0x7fe1001a47d0 2021-02-10T11:40:20.058 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:20.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.076 7fe105f35700 1 -- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 msgr2=0x7fe0e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.076 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 0x7fe0e401abc0 crc :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.076 7fe105f35700 1 -- 172.21.2.117:0/3291514129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 msgr2=0x7fe10008a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.076 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7fe0e8006f00 tx=0x7fe0e800b940).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 -- 172.21.2.117:0/3291514129 shutdown_connections 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1000a7be0 0x7fe1000a7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe10008a7a0 0x7fe1000a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0e40186c0 0x7fe0e401abc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 --2- 172.21.2.117:0/3291514129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1001a47d0 0x7fe10008a180 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 -- 172.21.2.117:0/3291514129 >> 172.21.2.117:0/3291514129 conn(0x7fe1000a9210 msgr2=0x7fe1000a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 -- 172.21.2.117:0/3291514129 shutdown_connections 2021-02-10T11:40:20.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.077 7fe105f35700 1 -- 172.21.2.117:0/3291514129 wait complete. 2021-02-10T11:40:20.068 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 Processor -- start 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 -- start start 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 0x7f56200d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620077a70 con 0x7f562007da20 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620079400 con 0x7f5620079000 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f5624cd4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562007de20 con 0x7f56200d1530 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f561eae8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f562007da20 0x7f56200771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48070/0 (socket says 172.21.2.117:48070) 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.498 7f561eae8700 1 -- 172.21.2.117:0/439067517 learned_addr learned my addr 172.21.2.117:0/439067517 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561f2e9700 1 --2- 172.21.2.117:0/439067517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 0x7f56200d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561dae6700 1 -- 172.21.2.117:0/439067517 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503525275 0 0) 0x7f5610001180 con 0x7f562007da20 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561dae6700 1 -- 172.21.2.117:0/439067517 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f560c002fe0 con 0x7f562007da20 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561f2e9700 1 -- 172.21.2.117:0/439067517 >> v1:172.21.2.125:6789/0 conn(0x7f5620079000 legacy=0x7f56200e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561f2e9700 1 -- 172.21.2.117:0/439067517 >> v1:172.21.2.117:6789/0 conn(0x7f562007da20 legacy=0x7f56200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561f2e9700 1 -- 172.21.2.117:0/439067517 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56201b00f0 con 0x7f56200d1530 2021-02-10T11:40:20.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561f2e9700 1 --2- 172.21.2.117:0/439067517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 0x7f56200d1930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5618006e50 tx=0x7f5618009e70).ready entity=mon.2 client_cookie=46e2673158413678 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561dae6700 1 -- 172.21.2.117:0/439067517 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561800b510 con 0x7f56200d1530 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561dae6700 1 -- 172.21.2.117:0/439067517 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618006790 con 0x7f56200d1530 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f561dae6700 1 -- 172.21.2.117:0/439067517 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f561800b940 con 0x7f56200d1530 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f5624cd4700 1 -- 172.21.2.117:0/439067517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 msgr2=0x7f56200d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.499 7f5624cd4700 1 --2- 172.21.2.117:0/439067517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 0x7f56200d1930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f5618006e50 tx=0x7f5618009e70).stop 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.500 7f5624cd4700 1 -- 172.21.2.117:0/439067517 shutdown_connections 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.500 7f5624cd4700 1 --2- 172.21.2.117:0/439067517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56200d1530 0x7f56200d1930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.500 7f5624cd4700 1 -- 172.21.2.117:0/439067517 >> 172.21.2.117:0/439067517 conn(0x7f56200ce450 msgr2=0x7f56200e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.500 7f5624cd4700 1 -- 172.21.2.117:0/439067517 shutdown_connections 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.500 7f5624cd4700 1 -- 172.21.2.117:0/439067517 wait complete. 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.514 7f5624cd4700 1 Processor -- start 2021-02-10T11:40:20.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.514 7f5624cd4700 1 -- start start 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5620094d90 0x7f5620095190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56201a7290 0x7f56201a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200a9af0 con 0x7f5620095a60 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200957b0 con 0x7f5620094d90 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f5624cd4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56201a7cb0 con 0x7f56201a7290 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47826/0 (socket says 172.21.2.117:47826) 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 -- 172.21.2.117:0/872116340 learned_addr learned my addr 172.21.2.117:0/872116340 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561e2e7700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5620094d90 0x7f5620095190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561f2e9700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56201a7290 0x7f56201a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 -- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56201a7290 msgr2=0x7f56201a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56201a7290 0x7f56201a7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 -- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5620094d90 msgr2=0x7f5620095190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5620094d90 0x7f5620095190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 -- 172.21.2.117:0/872116340 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56201a7ed0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.515 7f561eae8700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f561000a5d0 tx=0x7f561000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5610000ce0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56100043a0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f5624cd4700 1 -- 172.21.2.117:0/872116340 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56201a7ed0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56100048c0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f5624cd4700 1 -- 172.21.2.117:0/872116340 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56201a7ed0 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.516 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5610000c90 con 0x7f5620095a60 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.517 7f56177fe700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 0x7f560401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:20.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.517 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f56100328d0 con 0x7f5620095a60 2021-02-10T11:40:20.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.517 7f561e2e7700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 0x7f560401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:20.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.517 7f561e2e7700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 0x7f560401abd0 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:20.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.548 7f5624cd4700 1 -- 172.21.2.117:0/872116340 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56201032d0 con 0x7f5620095a60 2021-02-10T11:40:20.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.551 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56100043a0 con 0x7f5620095a60 2021-02-10T11:40:20.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.679 7f5624cd4700 1 -- 172.21.2.117:0/872116340 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f5620077a70 con 0x7f5620095a60 2021-02-10T11:40:20.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.679 7f56177fe700 1 -- 172.21.2.117:0/872116340 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v39) v1 ==== 100+0+10 (secure 0 0 0) 0x7f561000ad60 con 0x7f5620095a60 2021-02-10T11:40:20.662 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 msgr2=0x7f560401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 0x7f560401abd0 crc :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 msgr2=0x7f5620190560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f561000a5d0 tx=0x7f561000a970).stop 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 shutdown_connections 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56201a7290 0x7f56201a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5620094d90 0x7f5620095190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56040186d0 0x7f560401abd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 --2- 172.21.2.117:0/872116340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5620095a60 0x7f5620190560 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:20.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 >> 172.21.2.117:0/872116340 conn(0x7f56200b56c0 msgr2=0x7f56200a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:20.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 shutdown_connections 2021-02-10T11:40:20.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:20.681 7f5624cd4700 1 -- 172.21.2.117:0/872116340 wait complete. 2021-02-10T11:40:20.672 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:40:20.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 Processor -- start 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 -- start start 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580d0ac0 0x7f18581b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1858109ab0 con 0x7f18581b3470 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18580d0ec0 con 0x7f18580e51a0 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f18607d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18580d10e0 con 0x7f18580d0ac0 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f185ddee700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f18581b3470 0x7f18581085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48084/0 (socket says 172.21.2.117:48084) 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.099 7f185ddee700 1 -- 172.21.2.117:0/918706843 learned_addr learned my addr 172.21.2.117:0/918706843 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31244661 0 0) 0x7f18540009b0 con 0x7f18581b3470 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1844002fe0 con 0x7f18581b3470 2021-02-10T11:40:21.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2988220381 0 0) 0x7f1854001010 con 0x7f18581b3470 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580d0ac0 msgr2=0x7f18581b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580d0ac0 0x7f18581b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 >> v1:172.21.2.125:6789/0 conn(0x7f18580e51a0 legacy=0x7f1858108de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580fde80 con 0x7f18581b3470 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1854001530 con 0x7f18581b3470 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 >> v1:172.21.2.117:6789/0 conn(0x7f18581b3470 legacy=0x7f18581085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1844003a60 0x7f1844005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1844006770 0x7f1844008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1844009590 0x7f184400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184400c2d0 con 0x7f1844006770 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184400c4f0 con 0x7f1844003a60 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185d5ed700 1 -- 172.21.2.117:0/918706843 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184400c710 con 0x7f1844009590 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1844006770 0x7f1844008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 -- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1844009590 msgr2=0x7f184400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1844009590 0x7f184400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 -- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1844003a60 msgr2=0x7f1844005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1844003a60 0x7f1844005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 -- 172.21.2.117:0/918706843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184400c9b0 con 0x7f1844006770 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.100 7f185ddee700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1844006770 0x7f1844008c90 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f18540017d0 tx=0x7f1854007db0).ready entity=mon.0 client_cookie=95b9b54ccf773462 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1854001010 con 0x7f1844006770 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185400a0c0 con 0x7f1844006770 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f185d5ed700 1 -- 172.21.2.117:0/918706843 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f185400a550 con 0x7f1844006770 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 -- 172.21.2.117:0/918706843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1844006770 msgr2=0x7f1844008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 --2- 172.21.2.117:0/918706843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1844006770 0x7f1844008c90 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0x7f18540017d0 tx=0x7f1854007db0).stop 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f185edf0700 1 -- 172.21.2.117:0/918706843 reap_dead start 2021-02-10T11:40:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 -- 172.21.2.117:0/918706843 shutdown_connections 2021-02-10T11:40:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 -- 172.21.2.117:0/918706843 >> 172.21.2.117:0/918706843 conn(0x7f18580ce410 msgr2=0x7f18580d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 -- 172.21.2.117:0/918706843 shutdown_connections 2021-02-10T11:40:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.101 7f18607d3700 1 -- 172.21.2.117:0/918706843 wait complete. 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 Processor -- start 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 -- start start 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580f1ce0 0x7f185819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18580f20e0 0x7f18580aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18581a0250 con 0x7f18580f1880 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18580f25e0 con 0x7f18580f20e0 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.116 7f18607d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18580f2800 con 0x7f18580f1ce0 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47846/0 (socket says 172.21.2.117:47846) 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 -- 172.21.2.117:0/3284194624 learned_addr learned my addr 172.21.2.117:0/3284194624 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185edf0700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18580f20e0 0x7f18580aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185ddee700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580f1ce0 0x7f185819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 -- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580f1ce0 msgr2=0x7f185819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580f1ce0 0x7f185819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 -- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18580f20e0 msgr2=0x7f18580aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18580f20e0 0x7f18580aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 -- 172.21.2.117:0/3284194624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580ab320 con 0x7f18580f1880 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f185e5ef700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f184c006f00 tx=0x7f184c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f184c00bd30 con 0x7f18580f1880 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.117 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c000c90 con 0x7f18580f1880 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f18607d3700 1 -- 172.21.2.117:0/3284194624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18580ab320 con 0x7f18580f1880 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f184c0022a0 con 0x7f18580f1880 2021-02-10T11:40:21.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f18607d3700 1 -- 172.21.2.117:0/3284194624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18580ab8b0 con 0x7f18580f1880 2021-02-10T11:40:21.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f184c005310 con 0x7f18580f1880 2021-02-10T11:40:21.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f184affd700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 0x7f184001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:21.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f184c0328f0 con 0x7f18580f1880 2021-02-10T11:40:21.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.118 7f185ddee700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 0x7f184001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:21.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.119 7f185ddee700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 0x7f184001abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:21.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.149 7f18607d3700 1 -- 172.21.2.117:0/3284194624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1858109ab0 con 0x7f18580f1880 2021-02-10T11:40:21.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.152 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f184c006b90 con 0x7f18580f1880 2021-02-10T11:40:21.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.280 7f18607d3700 1 -- 172.21.2.117:0/3284194624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1858109ab0 con 0x7f18580f1880 2021-02-10T11:40:21.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.280 7f184affd700 1 -- 172.21.2.117:0/3284194624 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2096 (secure 0 0 0) 0x7f184c00bd30 con 0x7f18580f1880 2021-02-10T11:40:21.264 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:21.264 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:05.887046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7233":{"gid":7233,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/107410323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":107410323},{"type":"v1","addr":"172.21.2.125:6835","nonce":107410323}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:21.264 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 20 2021-02-10T11:40:21.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 -- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 msgr2=0x7f184001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 0x7f184001abc0 crc :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 -- 172.21.2.117:0/3284194624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 msgr2=0x7f18580a9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:21.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7f184c006f00 tx=0x7f184c00b940).stop 2021-02-10T11:40:21.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 -- 172.21.2.117:0/3284194624 shutdown_connections 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18580f20e0 0x7f18580aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18580f1ce0 0x7f185819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18400186c0 0x7f184001abc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 --2- 172.21.2.117:0/3284194624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18580f1880 0x7f18580a9f60 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 -- 172.21.2.117:0/3284194624 >> 172.21.2.117:0/3284194624 conn(0x7f18580bb7a0 msgr2=0x7f18580c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.282 7f18607d3700 1 -- 172.21.2.117:0/3284194624 shutdown_connections 2021-02-10T11:40:21.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:21.283 7f18607d3700 1 -- 172.21.2.117:0/3284194624 wait complete. 2021-02-10T11:40:21.475 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:03.881036', 'modified': '2021-02-10 11:40:05.887046', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7233}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7233': {'gid': 7233, 'name': 'b', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/107410323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 107410323}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 107410323}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:40:21.475 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:40:21.475 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:40:21.873 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:40:21.873 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:40:21.873 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:40:21.873 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7233, 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.010284148, 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 20, 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 39, 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 39, 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.644289623999999 2021-02-10T11:40:21.874 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:40:21.877 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7233, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.010284148, "mdsmap_epoch": 20, "osdmap_epoch": 39, "osdmap_epoch_barrier": 39, "uptime": 23.644289623999999 } 2021-02-10T11:40:21.877 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:40:21.878 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:40:21.878 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:40:21.878 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:21.878 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:22.214 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:40:22.559 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:22.559 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:22.902 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:22.902 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:22.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:22.903 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:40:23.238 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:40:23.238 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:40:23.238 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:23.240 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:40:23.584 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:40:23.584 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:40:23.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[121838]: starting ceph client 2021-02-10T11:40:23.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:40:23.666 7f963aa3ff80 -1 init, newargv = 0x558e04af7b00 newargc=9 2021-02-10T11:40:23.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[121838]: starting fuse 2021-02-10T11:40:23.928 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:40:23.929 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:40:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:40:23.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:23.930 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:40:24.263 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:40:24.263 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:40:24.264 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:40:24.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:40:24.264 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:40:24.265 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:40:24.266 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:40:24.266 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:40:24.266 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:40:24.626 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.121838.asok 2021-02-10T11:40:24.628 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.121838.asok 2021-02-10T11:40:24.628 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:24.629 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.121838.asok status 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:40:25.018 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "121838", 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 7428, 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 7428 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:40:25.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 947587380 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 947587380 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.7428 172.21.2.117:0/947587380", 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/947587380", 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 20, 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 39, 2021-02-10T11:40:25.020 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:40:25.021 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:40:25.021 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:40:25.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:25.022 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:25.022 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:25.357 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:40:25.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:25.358 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:25.358 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:25.703 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:40:26.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.135 7fd9259ee700 1 Processor -- start 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.135 7fd9259ee700 1 -- start start 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.135 7fd9259ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd92007da20 0x7fd9200771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.135 7fd9259ee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9200e4e00 0x7fd9201b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd9200e5200 0x7fd920077f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200e5c60 con 0x7fd9200e4e00 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200777d0 con 0x7fd9200e5200 2021-02-10T11:40:26.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd920077a10 con 0x7fd92007da20 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9200e4e00 0x7fd9201b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9200e4e00 0x7fd9201b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47868/0 (socket says 172.21.2.117:47868) 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 -- 172.21.2.117:0/2803655605 learned_addr learned my addr 172.21.2.117:0/2803655605 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91f7fe700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd92007da20 0x7fd9200771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91ffff700 1 --2- 172.21.2.117:0/2803655605 >> v2:172.21.2.125:3300/0 conn(0x7fd9200e5200 0x7fd920077f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 -- 172.21.2.117:0/2803655605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd92007da20 msgr2=0x7fd9200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd92007da20 0x7fd9200771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 -- 172.21.2.117:0/2803655605 >> v2:172.21.2.125:3300/0 conn(0x7fd9200e5200 msgr2=0x7fd920077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 --2- 172.21.2.117:0/2803655605 >> v2:172.21.2.125:3300/0 conn(0x7fd9200e5200 0x7fd920077f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 -- 172.21.2.117:0/2803655605 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92007de20 con 0x7fd9200e4e00 2021-02-10T11:40:26.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91effd700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9200e4e00 0x7fd9201b0130 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fd910006700 tx=0x7fd910009b00).ready entity=mon.0 client_cookie=e3ad82d19a1cc0bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91e7fc700 1 -- 172.21.2.117:0/2803655605 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91000c0c0 con 0x7fd9200e4e00 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91e7fc700 1 -- 172.21.2.117:0/2803655605 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91000c340 con 0x7fd9200e4e00 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd91e7fc700 1 -- 172.21.2.117:0/2803655605 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91000c820 con 0x7fd9200e4e00 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 -- 172.21.2.117:0/2803655605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9200e4e00 msgr2=0x7fd9201b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.136 7fd9259ee700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9200e4e00 0x7fd9201b0130 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fd910006700 tx=0x7fd910009b00).stop 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 -- 172.21.2.117:0/2803655605 shutdown_connections 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 --2- 172.21.2.117:0/2803655605 >> v2:172.21.2.125:3300/0 conn(0x7fd9200e5200 0x7fd920077f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9200e4e00 0x7fd9201b0130 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 --2- 172.21.2.117:0/2803655605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd92007da20 0x7fd9200771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 -- 172.21.2.117:0/2803655605 >> 172.21.2.117:0/2803655605 conn(0x7fd9200ce480 msgr2=0x7fd9200e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 -- 172.21.2.117:0/2803655605 shutdown_connections 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.137 7fd9259ee700 1 -- 172.21.2.117:0/2803655605 wait complete. 2021-02-10T11:40:26.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.151 7fd9259ee700 1 Processor -- start 2021-02-10T11:40:26.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 -- start start 2021-02-10T11:40:26.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9200b8540 0x7fd92008f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd920090760 0x7fd92008d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200b8940 con 0x7fd92008fd40 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92008dfa0 con 0x7fd920090760 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd9259ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92008e1c0 con 0x7fd9200b8540 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd91effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd91effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47874/0 (socket says 172.21.2.117:47874) 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd91effd700 1 -- 172.21.2.117:0/491183564 learned_addr learned my addr 172.21.2.117:0/491183564 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:26.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd91effd700 1 -- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9200b8540 msgr2=0x7fd92008f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.152 7fd91effd700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9200b8540 0x7fd92008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd91effd700 1 -- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd920090760 msgr2=0x7fd92008d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd91effd700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd920090760 0x7fd92008d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd91effd700 1 -- 172.21.2.117:0/491183564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92008e3e0 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd91effd700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7fd910006f20 tx=0x7fd91000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd91000bcd0 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910000c90 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9100022b0 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd9259ee700 1 -- 172.21.2.117:0/491183564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92008e3e0 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.153 7fd9259ee700 1 -- 172.21.2.117:0/491183564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92008e3e0 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.154 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd910002020 con 0x7fd92008fd40 2021-02-10T11:40:26.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.154 7fd907fff700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 0x7fd90001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.154 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd910032900 con 0x7fd92008fd40 2021-02-10T11:40:26.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.154 7fd91f7fe700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 0x7fd90001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.155 7fd91f7fe700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 0x7fd90001abd0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.185 7fd9259ee700 1 -- 172.21.2.117:0/491183564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd920077a10 con 0x7fd92008fd40 2021-02-10T11:40:26.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.188 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd910006470 con 0x7fd92008fd40 2021-02-10T11:40:26.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.321 7fd9259ee700 1 -- 172.21.2.117:0/491183564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd920077a10 con 0x7fd92008fd40 2021-02-10T11:40:26.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.376 7fd907fff700 1 -- 172.21.2.117:0/491183564 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd910002020 con 0x7fd92008fd40 2021-02-10T11:40:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 msgr2=0x7fd90001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 0x7fd90001abd0 crc :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 msgr2=0x7fd920090140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7fd910006f20 tx=0x7fd91000b8e0).stop 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 shutdown_connections 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9000186d0 0x7fd90001abd0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd92008fd40 0x7fd920090140 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd920090760 0x7fd92008d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 --2- 172.21.2.117:0/491183564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9200b8540 0x7fd92008f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 >> 172.21.2.117:0/491183564 conn(0x7fd9200c0420 msgr2=0x7fd920097c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 shutdown_connections 2021-02-10T11:40:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.388 7fd9259ee700 1 -- 172.21.2.117:0/491183564 wait complete. 2021-02-10T11:40:26.379 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:40:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 Processor -- start 2021-02-10T11:40:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 -- start start 2021-02-10T11:40:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 0x7f37180e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37180d1d00 con 0x7f371807da20 2021-02-10T11:40:26.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3718077a70 con 0x7f37181b00d0 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.806 7f371dd35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180e5410 con 0x7f3718079000 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3717fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f37181b00d0 0x7f37180d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:41062/0 (socket says 172.21.2.117:41062) 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37177fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f371807da20 0x7f37180771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48132/0 (socket says 172.21.2.117:48132) 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3717fff700 1 -- 172.21.2.117:0/83033565 learned_addr learned my addr 172.21.2.117:0/83033565 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3716ffd700 1 --2- 172.21.2.117:0/83033565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 0x7f37180e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499958453 0 0) 0x7f3700001100 con 0x7f371807da20 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3704002fe0 con 0x7f371807da20 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088568396 0 0) 0x7f3710001510 con 0x7f37181b00d0 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3704003210 con 0x7f37181b00d0 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3716ffd700 1 -- 172.21.2.117:0/83033565 >> v1:172.21.2.125:6789/0 conn(0x7f37181b00d0 legacy=0x7f37180d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3716ffd700 1 -- 172.21.2.117:0/83033565 >> v1:172.21.2.117:6789/0 conn(0x7f371807da20 legacy=0x7f37180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3716ffd700 1 -- 172.21.2.117:0/83033565 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3718079400 con 0x7f3718079000 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f3716ffd700 1 --2- 172.21.2.117:0/83033565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 0x7f37180e4d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3708006650 tx=0x7f3708009a80).ready entity=mon.2 client_cookie=dd6a7a35fd546937 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37080009b0 con 0x7f3718079000 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370800c0c0 con 0x7f3718079000 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f37167fc700 1 -- 172.21.2.117:0/83033565 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f370800c520 con 0x7f3718079000 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f371dd35700 1 -- 172.21.2.117:0/83033565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 msgr2=0x7f37180e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.807 7f371dd35700 1 --2- 172.21.2.117:0/83033565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 0x7f37180e4d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3708006650 tx=0x7f3708009a80).stop 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.808 7f371dd35700 1 -- 172.21.2.117:0/83033565 shutdown_connections 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.808 7f371dd35700 1 --2- 172.21.2.117:0/83033565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3718079000 0x7f37180e4d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.808 7f371dd35700 1 -- 172.21.2.117:0/83033565 >> 172.21.2.117:0/83033565 conn(0x7f37180ce430 msgr2=0x7f37180e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.808 7f371dd35700 1 -- 172.21.2.117:0/83033565 shutdown_connections 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.808 7f371dd35700 1 -- 172.21.2.117:0/83033565 wait complete. 2021-02-10T11:40:26.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 Processor -- start 2021-02-10T11:40:26.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 -- start start 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37180a2120 0x7f37180a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180a2f40 0x7f371818f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180ae2e0 con 0x7f37180a2b40 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3718192b30 con 0x7f37180a2120 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f371dd35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371818fbd0 con 0x7f37180a2f40 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f3716ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f3716ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47890/0 (socket says 172.21.2.117:47890) 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.823 7f3716ffd700 1 -- 172.21.2.117:0/121016554 learned_addr learned my addr 172.21.2.117:0/121016554 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 -- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180a2f40 msgr2=0x7f371818f5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:26.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180a2f40 0x7f371818f5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 -- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37180a2120 msgr2=0x7f37180a2520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37180a2120 0x7f37180a2520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 -- 172.21.2.117:0/121016554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371818fdb0 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f3716ffd700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f3708009630 tx=0x7f370800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37080093a0 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3708002030 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3708002550 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f371dd35700 1 -- 172.21.2.117:0/121016554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371818fdb0 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.824 7f371dd35700 1 -- 172.21.2.117:0/121016554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3718190340 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.825 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3708000c90 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.825 7f370ffff700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 0x7f36fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.825 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f37080326f0 con 0x7f37180a2b40 2021-02-10T11:40:26.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.825 7f37177fe700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 0x7f36fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:26.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.826 7f37177fe700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 0x7f36fc01abc0 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:26.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.856 7f371dd35700 1 -- 172.21.2.117:0/121016554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3718190340 con 0x7f37180a2b40 2021-02-10T11:40:26.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.859 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3708021f00 con 0x7f37180a2b40 2021-02-10T11:40:26.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.993 7f371dd35700 1 -- 172.21.2.117:0/121016554 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f3718192b30 con 0x7f36fc0186c0 2021-02-10T11:40:26.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.994 7f370ffff700 1 -- 172.21.2.117:0/121016554 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f37000076f0 con 0x7f36fc0186c0 2021-02-10T11:40:26.979 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:40:26.979 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:26.979 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:40:26.979 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:26.979 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:40:26.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.998 7f371dd35700 1 -- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 msgr2=0x7f36fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.998 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 0x7f36fc01abc0 crc :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.998 7f371dd35700 1 -- 172.21.2.117:0/121016554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 msgr2=0x7f371818ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.998 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7f3708009630 tx=0x7f370800b970).stop 2021-02-10T11:40:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 -- 172.21.2.117:0/121016554 shutdown_connections 2021-02-10T11:40:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36fc0186c0 0x7f36fc01abc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37180a2b40 0x7f371818ef90 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180a2f40 0x7f371818f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 --2- 172.21.2.117:0/121016554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37180a2120 0x7f37180a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 -- 172.21.2.117:0/121016554 >> 172.21.2.117:0/121016554 conn(0x7f37180977a0 msgr2=0x7f3718099fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 -- 172.21.2.117:0/121016554 shutdown_connections 2021-02-10T11:40:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:26.999 7f371dd35700 1 -- 172.21.2.117:0/121016554 wait complete. 2021-02-10T11:40:26.989 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.421 7fad9e59a700 1 Processor -- start 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.421 7fad9e59a700 1 -- start start 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9e59a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 0x7fad980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9e59a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad980d1d20 con 0x7fad980e4da0 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9e59a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad98077a70 con 0x7fad980e51a0 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9e59a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980777d0 con 0x7fad9807da20 2021-02-10T11:40:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad97fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 0x7fad980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9cbb7700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fad980e51a0 0x7fad980d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:41076/0 (socket says 172.21.2.117:41076) 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad9cbb7700 1 -- 172.21.2.117:0/2478150456 learned_addr learned my addr 172.21.2.117:0/2478150456 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679780916 0 0) 0x7fad8c000f30 con 0x7fad980e4da0 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad80002fe0 con 0x7fad980e4da0 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808753819 0 0) 0x7fad90001510 con 0x7fad980e51a0 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad97fff700 1 -- 172.21.2.117:0/2478150456 >> v1:172.21.2.125:6789/0 conn(0x7fad980e51a0 legacy=0x7fad980d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad97fff700 1 -- 172.21.2.117:0/2478150456 >> v1:172.21.2.117:6789/0 conn(0x7fad980e4da0 legacy=0x7fad981b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.422 7fad97fff700 1 -- 172.21.2.117:0/2478150456 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad981b08e0 con 0x7fad9807da20 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad97fff700 1 --2- 172.21.2.117:0/2478150456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 0x7fad980771b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fad88006650 tx=0x7fad88009a80).ready entity=mon.2 client_cookie=a79a2409d302005a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad8800c0c0 con 0x7fad9807da20 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8800c340 con 0x7fad9807da20 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad96ffd700 1 -- 172.21.2.117:0/2478150456 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad8800c860 con 0x7fad9807da20 2021-02-10T11:40:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad9e59a700 1 -- 172.21.2.117:0/2478150456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 msgr2=0x7fad980771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad9e59a700 1 --2- 172.21.2.117:0/2478150456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 0x7fad980771b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fad88006650 tx=0x7fad88009a80).stop 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad9e59a700 1 -- 172.21.2.117:0/2478150456 shutdown_connections 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad9e59a700 1 --2- 172.21.2.117:0/2478150456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad9807da20 0x7fad980771b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.423 7fad9e59a700 1 -- 172.21.2.117:0/2478150456 >> 172.21.2.117:0/2478150456 conn(0x7fad980ce450 msgr2=0x7fad980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.424 7fad9e59a700 1 -- 172.21.2.117:0/2478150456 shutdown_connections 2021-02-10T11:40:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.424 7fad9e59a700 1 -- 172.21.2.117:0/2478150456 wait complete. 2021-02-10T11:40:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.438 7fad9e59a700 1 Processor -- start 2021-02-10T11:40:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.438 7fad9e59a700 1 -- start start 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 0x7fad980b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad98090de0 0x7fad980911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98090600 con 0x7fad980c33d0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98090860 con 0x7fad98090de0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9e59a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980b3c70 con 0x7fad980b3ff0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad9cbb7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad98090de0 0x7fad980911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47904/0 (socket says 172.21.2.117:47904) 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad977fe700 1 -- 172.21.2.117:0/520631368 learned_addr learned my addr 172.21.2.117:0/520631368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 0x7fad980b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 -- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad98090de0 msgr2=0x7fad980911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad98090de0 0x7fad980911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 -- 172.21.2.117:0/520631368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 msgr2=0x7fad980b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.439 7fad97fff700 1 -- 172.21.2.117:0/520631368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98091800 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad977fe700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad97fff700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 0x7fad980b3030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fad8800b530 tx=0x7fad8800b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad880093a0 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8800bcc0 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad9e59a700 1 -- 172.21.2.117:0/520631368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98091800 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad880022b0 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad9e59a700 1 -- 172.21.2.117:0/520631368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad98091d90 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.440 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad88000c90 con 0x7fad980b3ff0 2021-02-10T11:40:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.441 7fad94ff9700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 0x7fad7801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.441 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fad88032720 con 0x7fad980b3ff0 2021-02-10T11:40:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.441 7fad977fe700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 0x7fad7801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.441 7fad977fe700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 0x7fad7801ac10 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:27.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.472 7fad9e59a700 1 -- 172.21.2.117:0/520631368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98103370 con 0x7fad980b3ff0 2021-02-10T11:40:27.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.476 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad88009280 con 0x7fad980b3ff0 2021-02-10T11:40:27.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.626 7fad9e59a700 1 -- 172.21.2.117:0/520631368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fad98077a70 con 0x7fad980b3ff0 2021-02-10T11:40:27.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.628 7fad94ff9700 1 -- 172.21.2.117:0/520631368 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad88000c90 con 0x7fad980b3ff0 2021-02-10T11:40:27.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.630 7fad9e59a700 1 -- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 msgr2=0x7fad7801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.630 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 0x7fad7801ac10 crc :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.630 7fad9e59a700 1 -- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 msgr2=0x7fad980b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.630 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 0x7fad980b3030 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fad8800b530 tx=0x7fad8800b8d0).stop 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 -- 172.21.2.117:0/520631368 shutdown_connections 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad78018730 0x7fad7801ac10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad980c33d0 0x7fad980b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad98090de0 0x7fad980911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 --2- 172.21.2.117:0/520631368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad980b3ff0 0x7fad980b3030 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:27.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 -- 172.21.2.117:0/520631368 >> 172.21.2.117:0/520631368 conn(0x7fad980a1420 msgr2=0x7fad9809efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:27.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 -- 172.21.2.117:0/520631368 shutdown_connections 2021-02-10T11:40:27.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:27.631 7fad9e59a700 1 -- 172.21.2.117:0/520631368 wait complete. 2021-02-10T11:40:27.621 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:40:28.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.061 7ffbcb43d700 1 Processor -- start 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.061 7ffbcb43d700 1 -- start start 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbcb43d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffbc407da20 0x7ffbc40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbcb43d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 0x7ffbc4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbcb43d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc4077a70 con 0x7ffbc40e4da0 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbcb43d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc40777d0 con 0x7ffbc407da20 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbcb43d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41b08e0 con 0x7ffbc40e51a0 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc8a58700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ffbc40e4da0 0x7ffbc41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48160/0 (socket says 172.21.2.117:48160) 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc8a58700 1 -- 172.21.2.117:0/3139162462 learned_addr learned my addr 172.21.2.117:0/3139162462 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc9a5a700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 0x7ffbc4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc9259700 1 --2- 172.21.2.117:0/3139162462 >> v2:172.21.2.125:3300/0 conn(0x7ffbc407da20 0x7ffbc40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbbbfff700 1 -- 172.21.2.117:0/3139162462 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869985221 0 0) 0x7ffbb40010b0 con 0x7ffbc40e4da0 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbbbfff700 1 -- 172.21.2.117:0/3139162462 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb0002fe0 con 0x7ffbc40e4da0 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc9259700 1 -- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 msgr2=0x7ffbc4079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.062 7ffbc9259700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 0x7ffbc4079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbc9259700 1 -- 172.21.2.117:0/3139162462 >> v1:172.21.2.117:6789/0 conn(0x7ffbc40e4da0 legacy=0x7ffbc41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbc9259700 1 -- 172.21.2.117:0/3139162462 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc407de20 con 0x7ffbc407da20 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbc9a5a700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 0x7ffbc4079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbc9259700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc407da20 0x7ffbc40771b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7ffbbc0069d0 tx=0x7ffbbc009e40).ready entity=mon.1 client_cookie=b74612365595f070 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbbbfff700 1 -- 172.21.2.117:0/3139162462 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc0066c0 con 0x7ffbc407da20 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbbbfff700 1 -- 172.21.2.117:0/3139162462 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc00b510 con 0x7ffbc407da20 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbbbfff700 1 -- 172.21.2.117:0/3139162462 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc00ba30 con 0x7ffbc407da20 2021-02-10T11:40:28.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbcb43d700 1 -- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc407da20 msgr2=0x7ffbc40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.063 7ffbcb43d700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc407da20 0x7ffbc40771b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7ffbbc0069d0 tx=0x7ffbbc009e40).stop 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 -- 172.21.2.117:0/3139162462 shutdown_connections 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc40e51a0 0x7ffbc4079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 --2- 172.21.2.117:0/3139162462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc407da20 0x7ffbc40771b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 -- 172.21.2.117:0/3139162462 >> 172.21.2.117:0/3139162462 conn(0x7ffbc40ce450 msgr2=0x7ffbc40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 -- 172.21.2.117:0/3139162462 shutdown_connections 2021-02-10T11:40:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.064 7ffbcb43d700 1 -- 172.21.2.117:0/3139162462 wait complete. 2021-02-10T11:40:28.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 Processor -- start 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 -- start start 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc41adf40 0x7ffbc41ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc41ae960 0x7ffbc4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41ad5a0 con 0x7ffbc41adb20 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41ad7c0 con 0x7ffbc41ae960 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbcb43d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4198160 con 0x7ffbc41adf40 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbc9259700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbc9259700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47914/0 (socket says 172.21.2.117:47914) 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.079 7ffbc9259700 1 -- 172.21.2.117:0/3881112906 learned_addr learned my addr 172.21.2.117:0/3881112906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 -- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc41adf40 msgr2=0x7ffbc41ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc41adf40 0x7ffbc41ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 -- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc41ae960 msgr2=0x7ffbc4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc41ae960 0x7ffbc4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 -- 172.21.2.117:0/3881112906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc4198340 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbc9259700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7ffbbc0069d0 tx=0x7ffbbc00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc006240 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc00be60 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffbbc005660 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc4198340 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.080 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41988d0 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.081 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffbbc00a000 con 0x7ffbc41adb20 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.081 7ffbb9ffb700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 0x7ffba401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.081 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffbbc032a40 con 0x7ffbc41adb20 2021-02-10T11:40:28.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.081 7ffbc8a58700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 0x7ffba401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.082 7ffbc8a58700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 0x7ffba401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.111 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4103270 con 0x7ffbc41adb20 2021-02-10T11:40:28.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.114 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffbbc02d8b0 con 0x7ffbc41adb20 2021-02-10T11:40:28.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.244 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7ffbc4103270 con 0x7ffbc41adb20 2021-02-10T11:40:28.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.248 7ffbb9ffb700 1 -- 172.21.2.117:0/3881112906 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v19) v1 ==== 158+0+64 (secure 0 0 0) 0x7ffbbc02d8b0 con 0x7ffbc41adb20 2021-02-10T11:40:28.234 INFO:teuthology.orchestra.run.gibba017.stdout:[client.guest1] 2021-02-10T11:40:28.234 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQAsxiNgc2ekDhAAdHs9WUR/dyefJ4iFa0LIeg== 2021-02-10T11:40:28.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 msgr2=0x7ffba401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 0x7ffba401abc0 crc :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 msgr2=0x7ffbc41acf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7ffbbc0069d0 tx=0x7ffbbc00bd20).stop 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 shutdown_connections 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbc41ae960 0x7ffbc4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbc41adf40 0x7ffbc41ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffba40186c0 0x7ffba401abc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 --2- 172.21.2.117:0/3881112906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbc41adb20 0x7ffbc41acf80 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 >> 172.21.2.117:0/3881112906 conn(0x7ffbc4104e80 msgr2=0x7ffbc41af500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 shutdown_connections 2021-02-10T11:40:28.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.253 7ffbcb43d700 1 -- 172.21.2.117:0/3881112906 wait complete. 2021-02-10T11:40:28.244 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000025849 2021-02-10T11:40:28.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.675 7f42cf1de700 1 Processor -- start 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.675 7f42cf1de700 1 -- start start 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cf1de700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42c8079000 0x7f42c80e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cf1de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 0x7f42c81b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cf1de700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c8077a70 con 0x7f42c807da20 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cf1de700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c80e53c0 con 0x7f42c8079000 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cf1de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8079400 con 0x7f42c81b00f0 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42ccffa700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f42c807da20 0x7f42c80771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48170/0 (socket says 172.21.2.117:48170) 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42ccffa700 1 -- 172.21.2.117:0/4149823005 learned_addr learned my addr 172.21.2.117:0/4149823005 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:28.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42cd7fb700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 0x7f42c81b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42c7fff700 1 --2- 172.21.2.117:0/4149823005 >> v2:172.21.2.125:3300/0 conn(0x7f42c8079000 0x7f42c80e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42c77fe700 1 -- 172.21.2.117:0/4149823005 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154142453 0 0) 0x7f42b0000ce0 con 0x7f42c807da20 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.676 7f42c77fe700 1 -- 172.21.2.117:0/4149823005 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b4002fe0 con 0x7f42c807da20 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c7fff700 1 -- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 msgr2=0x7f42c81b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c7fff700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 0x7f42c81b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c7fff700 1 -- 172.21.2.117:0/4149823005 >> v1:172.21.2.117:6789/0 conn(0x7f42c807da20 legacy=0x7f42c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c7fff700 1 -- 172.21.2.117:0/4149823005 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c807de20 con 0x7f42c8079000 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42cd7fb700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 0x7f42c81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c7fff700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c8079000 0x7f42c80e4da0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f42b8006650 tx=0x7f42b8009a80).ready entity=mon.1 client_cookie=4a6f2dfb68ac5026 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c77fe700 1 -- 172.21.2.117:0/4149823005 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42b80009b0 con 0x7f42c8079000 2021-02-10T11:40:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c77fe700 1 -- 172.21.2.117:0/4149823005 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b800c0c0 con 0x7f42c8079000 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42c77fe700 1 -- 172.21.2.117:0/4149823005 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42b800c5e0 con 0x7f42c8079000 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42cf1de700 1 -- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c8079000 msgr2=0x7f42c80e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.677 7f42cf1de700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c8079000 0x7f42c80e4da0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f42b8006650 tx=0x7f42b8009a80).stop 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 -- 172.21.2.117:0/4149823005 shutdown_connections 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c81b00f0 0x7f42c81b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 --2- 172.21.2.117:0/4149823005 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c8079000 0x7f42c80e4da0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 -- 172.21.2.117:0/4149823005 >> 172.21.2.117:0/4149823005 conn(0x7f42c80ce450 msgr2=0x7f42c80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 -- 172.21.2.117:0/4149823005 shutdown_connections 2021-02-10T11:40:28.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.678 7f42cf1de700 1 -- 172.21.2.117:0/4149823005 wait complete. 2021-02-10T11:40:28.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 Processor -- start 2021-02-10T11:40:28.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 -- start start 2021-02-10T11:40:28.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c80b7fc0 0x7f42c81042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c81048f0 0x7f42c8104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81054f0 con 0x7f42c8090710 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8195a30 con 0x7f42c81048f0 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cf1de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8195fb0 con 0x7f42c80b7fc0 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cd7fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cd7fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47928/0 (socket says 172.21.2.117:47928) 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.693 7f42cd7fb700 1 -- 172.21.2.117:0/3078906231 learned_addr learned my addr 172.21.2.117:0/3078906231 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42ccffa700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c80b7fc0 0x7f42c81042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42c7fff700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c81048f0 0x7f42c8104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 -- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c80b7fc0 msgr2=0x7f42c81042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c80b7fc0 0x7f42c81042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 -- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c81048f0 msgr2=0x7f42c8104cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c81048f0 0x7f42c8104cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 -- 172.21.2.117:0/3078906231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c80b7b60 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cd7fb700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f42c000b970 tx=0x7f42c000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42c0000f90 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c0006790 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c80b7b60 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42c00015f0 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.694 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c8091420 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.695 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42c0000df0 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.695 7f42c57fa700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 0x7f42ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.695 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f42c0032c50 con 0x7f42c8090710 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.695 7f42ccffa700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 0x7f42ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.696 7f42ccffa700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 0x7f42ac01abc0 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:28.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.725 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c8103280 con 0x7f42c8090710 2021-02-10T11:40:28.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.728 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42c0000df0 con 0x7f42c8090710 2021-02-10T11:40:28.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.855 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000025849"}) v1 -- 0x7f42c8091420 con 0x7f42ac0186c0 2021-02-10T11:40:28.842 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:40:28.854 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:28.842 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:40:28.854 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:28.843 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:40:28.854 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:28.843 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:40:28.854 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:28.843 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:40:28.854 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:40:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.874 7f42c57fa700 1 -- 172.21.2.117:0/3078906231 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f42b0006a10 con 0x7f42ac0186c0 2021-02-10T11:40:28.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 msgr2=0x7f42ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 0x7f42ac01abc0 crc :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 msgr2=0x7f42c8090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:28.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7f42c000b970 tx=0x7f42c000bd10).stop 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 shutdown_connections 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42ac0186c0 0x7f42ac01abc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42c8090710 0x7f42c8090b10 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42c81048f0 0x7f42c8104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 --2- 172.21.2.117:0/3078906231 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42c80b7fc0 0x7f42c81042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 >> 172.21.2.117:0/3078906231 conn(0x7f42c80a1850 msgr2=0x7f42c80c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 shutdown_connections 2021-02-10T11:40:28.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:28.889 7f42cf1de700 1 -- 172.21.2.117:0/3078906231 wait complete. 2021-02-10T11:40:28.880 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000418555 --group_name group_0000000000025849 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 Processor -- start 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 -- start start 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bac07da20 0x7f4bac0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4bac0e4e00 0x7f4bac1b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac077a10 con 0x7f4bac0e5200 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac0777d0 con 0x7f4bac07da20 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.311 7f4bb2e1a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac07de20 con 0x7f4bac0e4e00 2021-02-10T11:40:29.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bac07da20 0x7f4bac0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bac07da20 0x7f4bac0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47936/0 (socket says 172.21.2.117:47936) 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 -- 172.21.2.117:0/834344545 learned_addr learned my addr 172.21.2.117:0/834344545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 -- 172.21.2.117:0/834344545 >> v2:172.21.2.125:3300/0 conn(0x7f4bac0e4e00 msgr2=0x7f4bac1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 --2- 172.21.2.117:0/834344545 >> v2:172.21.2.125:3300/0 conn(0x7f4bac0e4e00 0x7f4bac1b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 -- 172.21.2.117:0/834344545 >> v1:172.21.2.125:6789/0 conn(0x7f4bac0e5200 legacy=0x7f4bac077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 -- 172.21.2.117:0/834344545 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac1b0750 con 0x7f4bac07da20 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb0c36700 1 --2- 172.21.2.117:0/834344545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac07da20 0x7f4bac0771b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f4b94006700 tx=0x7f4b94009b80).ready entity=mon.0 client_cookie=cdccd610cec3f781 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bab7fe700 1 -- 172.21.2.117:0/834344545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9400c0c0 con 0x7f4bac07da20 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bab7fe700 1 -- 172.21.2.117:0/834344545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9400c340 con 0x7f4bac07da20 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bab7fe700 1 -- 172.21.2.117:0/834344545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9400c820 con 0x7f4bac07da20 2021-02-10T11:40:29.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb2e1a700 1 -- 172.21.2.117:0/834344545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac07da20 msgr2=0x7f4bac0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.312 7f4bb2e1a700 1 --2- 172.21.2.117:0/834344545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac07da20 0x7f4bac0771b0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f4b94006700 tx=0x7f4b94009b80).stop 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 -- 172.21.2.117:0/834344545 shutdown_connections 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 --2- 172.21.2.117:0/834344545 >> v2:172.21.2.125:3300/0 conn(0x7f4bac0e4e00 0x7f4bac1b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 --2- 172.21.2.117:0/834344545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac07da20 0x7f4bac0771b0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 -- 172.21.2.117:0/834344545 >> 172.21.2.117:0/834344545 conn(0x7f4bac0ce480 msgr2=0x7f4bac0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 -- 172.21.2.117:0/834344545 shutdown_connections 2021-02-10T11:40:29.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.313 7f4bb2e1a700 1 -- 172.21.2.117:0/834344545 wait complete. 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 Processor -- start 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 -- start start 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bac1906a0 0x7f4bac107c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bac0c3250 0x7f4bac1a7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0c37e0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac190aa0 con 0x7f4bac1906a0 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.328 7f4bb2e1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0c3d60 con 0x7f4bac0c3250 2021-02-10T11:40:29.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47940/0 (socket says 172.21.2.117:47940) 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 -- 172.21.2.117:0/3209753979 learned_addr learned my addr 172.21.2.117:0/3209753979 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 -- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bac0c3250 msgr2=0x7f4bac1a7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bac0c3250 0x7f4bac1a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 -- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bac1906a0 msgr2=0x7f4bac107c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bac1906a0 0x7f4bac107c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 -- 172.21.2.117:0/3209753979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac1a7ea0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4babfff700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f4b9c00b530 tx=0x7f4b9c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9c00bcc0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c000c90 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b9c0022a0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac1a7ea0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.329 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac1a7ea0 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.330 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b9c005310 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.330 7f4ba97fa700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 0x7f4b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.330 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b9c032740 con 0x7f4bac0c2e50 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.330 7f4bb0c36700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 0x7f4b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.331 7f4bb0c36700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 0x7f4b9001abc0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.361 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac105260 con 0x7f4bac0c2e50 2021-02-10T11:40:29.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.363 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b9c006b90 con 0x7f4bac0c2e50 2021-02-10T11:40:29.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.489 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000418555", "target": ["mgr", ""], "group_name": "group_0000000000025849"}) v1 -- 0x7f4bac105260 con 0x7f4b900186c0 2021-02-10T11:40:29.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.515 7f4ba97fa700 1 -- 172.21.2.117:0/3209753979 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4b94000990 con 0x7f4b900186c0 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.522 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 msgr2=0x7f4b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.522 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 0x7f4b9001abc0 crc :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.522 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 msgr2=0x7f4bac1a7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.522 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7f4b9c00b530 tx=0x7f4b9c00b8d0).stop 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 shutdown_connections 2021-02-10T11:40:29.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b900186c0 0x7f4b9001abc0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bac0c2e50 0x7f4bac1a7260 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bac0c3250 0x7f4bac1a7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 --2- 172.21.2.117:0/3209753979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bac1906a0 0x7f4bac107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 >> 172.21.2.117:0/3209753979 conn(0x7f4bac0c0930 msgr2=0x7f4bac0900c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 shutdown_connections 2021-02-10T11:40:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.523 7f4bb2e1a700 1 -- 172.21.2.117:0/3209753979 wait complete. 2021-02-10T11:40:29.513 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000418555 guest1 --group_name group_0000000000025849 --tenant_id tenant1 --allow-existing-id 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.951 7fe8bbcb7700 1 Processor -- start 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 -- start start 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe8b40e4e30 0x7fe8b41b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 0x7fe8b4084c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b40858f0 con 0x7fe8b41b0800 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b4084800 con 0x7fe8b40e4e30 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8bbcb7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40849f0 con 0x7fe8b40e5230 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.952 7fe8ba2d4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe8b41b0800 0x7fe8b4085e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48198/0 (socket says 172.21.2.117:48198) 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.953 7fe8ba2d4700 1 -- 172.21.2.117:0/3154312981 learned_addr learned my addr 172.21.2.117:0/3154312981 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:29.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.953 7fe8b92d2700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 0x7fe8b4084c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.953 7fe8b9ad3700 1 --2- 172.21.2.117:0/3154312981 >> v2:172.21.2.125:3300/0 conn(0x7fe8b40e4e30 0x7fe8b41b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.953 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763113555 0 0) 0x7fe8ac000f90 con 0x7fe8b41b0800 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.953 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0002fe0 con 0x7fe8b41b0800 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b9ad3700 1 -- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 msgr2=0x7fe8b4084c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b9ad3700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 0x7fe8b4084c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b9ad3700 1 -- 172.21.2.117:0/3154312981 >> v1:172.21.2.117:6789/0 conn(0x7fe8b41b0800 legacy=0x7fe8b4085e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4272784125 0 0) 0x7fe8ac001500 con 0x7fe8b41b0800 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b9ad3700 1 -- 172.21.2.117:0/3154312981 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4085240 con 0x7fe8b40e4e30 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b92d2700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 0x7fe8b4084c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.954 7fe8b9ad3700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40e4e30 0x7fe8b41b01e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fe8b0006650 tx=0x7fe8b0009a80).ready entity=mon.1 client_cookie=64924a2f6e416fa3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b00009b0 con 0x7fe8b40e4e30 2021-02-10T11:40:29.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b000c0c0 con 0x7fe8b40e4e30 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8b8ad1700 1 -- 172.21.2.117:0/3154312981 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b000c5e0 con 0x7fe8b40e4e30 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 -- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40e4e30 msgr2=0x7fe8b41b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40e4e30 0x7fe8b41b01e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fe8b0006650 tx=0x7fe8b0009a80).stop 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 -- 172.21.2.117:0/3154312981 shutdown_connections 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b40e5230 0x7fe8b4084c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 --2- 172.21.2.117:0/3154312981 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40e4e30 0x7fe8b41b01e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.955 7fe8bbcb7700 1 -- 172.21.2.117:0/3154312981 >> 172.21.2.117:0/3154312981 conn(0x7fe8b40ce4d0 msgr2=0x7fe8b40e4300 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.956 7fe8bbcb7700 1 -- 172.21.2.117:0/3154312981 shutdown_connections 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.956 7fe8bbcb7700 1 -- 172.21.2.117:0/3154312981 wait complete. 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.970 7fe8bbcb7700 1 Processor -- start 2021-02-10T11:40:29.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.970 7fe8bbcb7700 1 -- start start 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 0x7fe8b40908a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b4090ec0 0x7fe8b408e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b408fd00 con 0x7fe8b40b1f80 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40912c0 con 0x7fe8b40b2380 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8bbcb7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40914a0 con 0x7fe8b4090ec0 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47950/0 (socket says 172.21.2.117:47950) 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 -- 172.21.2.117:0/1720714399 learned_addr learned my addr 172.21.2.117:0/1720714399 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:29.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8ba2d4700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b4090ec0 0x7fe8b408e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 -- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b4090ec0 msgr2=0x7fe8b408e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b4090ec0 0x7fe8b408e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 -- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 msgr2=0x7fe8b40908a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b92d2700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 0x7fe8b40908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 0x7fe8b40908a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 -- 172.21.2.117:0/1720714399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b408ec70 con 0x7fe8b40b1f80 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b92d2700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 0x7fe8b40908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.971 7fe8b9ad3700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7fe8b0009630 tx=0x7fe8b000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b00093a0 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b000bcf0 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b408ec70 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8b0005560 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b408ec70 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.972 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8b00054f0 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.973 7fe8aa7fc700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 0x7fe89c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.973 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe8b00327b0 con 0x7fe8b40b1f80 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.973 7fe8b92d2700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 0x7fe89c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:29.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:29.973 7fe8b92d2700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 0x7fe89c01ac10 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:29.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.002 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b40858f0 con 0x7fe8b40b1f80 2021-02-10T11:40:29.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.005 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8b00052e0 con 0x7fe8b40b1f80 2021-02-10T11:40:30.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.130 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"allow_existing_id": true, "vol_name": "cephfs", "sub_name": "subvolume_0000000000418555", "target": ["mgr", ""], "tenant_id": "tenant1", "group_name": "group_0000000000025849", "prefix": "fs subvolume authorize", "auth_id": "guest1"}) v1 -- 0x7fe8b40858f0 con 0x7fe89c018730 2021-02-10T11:40:30.120 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:40:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.165 7fe8aa7fc700 1 -- 172.21.2.117:0/1720714399 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fe8a4006de0 con 0x7fe89c018730 2021-02-10T11:40:30.176 INFO:teuthology.orchestra.run.gibba017.stdout:AQAsxiNgc2ekDhAAdHs9WUR/dyefJ4iFa0LIeg== 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 msgr2=0x7fe89c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 0x7fe89c01ac10 crc :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 msgr2=0x7fe8b4090280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7fe8b0009630 tx=0x7fe8b000b900).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 shutdown_connections 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8b4090ec0 0x7fe8b408e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8b40b2380 0x7fe8b40908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe89c018730 0x7fe89c01ac10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 --2- 172.21.2.117:0/1720714399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8b40b1f80 0x7fe8b4090280 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.195 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 >> 172.21.2.117:0/1720714399 conn(0x7fe8b40abb40 msgr2=0x7fe8b40c3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.196 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 shutdown_connections 2021-02-10T11:40:30.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.196 7fe8bbcb7700 1 -- 172.21.2.117:0/1720714399 wait complete. 2021-02-10T11:40:30.186 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000418555 guest1 --group_name group_0000000000025849 2021-02-10T11:40:30.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 Processor -- start 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 -- start start 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0600d1870 0x7fd0600d7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd0600d7e80 0x7fd0600d8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0600f6160 con 0x7fd0600d9180 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0600d1c70 con 0x7fd0600d1870 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.619 7fd065a1d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0600d8280 con 0x7fd0600d7e80 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05ffff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fd0600d9180 0x7fd0600d9580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:41134/0 (socket says 172.21.2.117:41134) 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05ffff700 1 -- 172.21.2.117:0/3244238855 learned_addr learned my addr 172.21.2.117:0/3244238855 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 --2- 172.21.2.117:0/3244238855 >> v2:172.21.2.125:3300/0 conn(0x7fd0600d7e80 0x7fd0600d8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05f7fe700 1 --2- 172.21.2.117:0/3244238855 >> v2:172.21.2.117:3300/0 conn(0x7fd0600d1870 0x7fd0600d7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 -- 172.21.2.117:0/3244238855 >> v2:172.21.2.117:3300/0 conn(0x7fd0600d1870 msgr2=0x7fd0600d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 --2- 172.21.2.117:0/3244238855 >> v2:172.21.2.117:3300/0 conn(0x7fd0600d1870 0x7fd0600d7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 -- 172.21.2.117:0/3244238855 >> v1:172.21.2.125:6789/0 conn(0x7fd0600d9180 legacy=0x7fd0600d9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 -- 172.21.2.117:0/3244238855 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0601014a0 con 0x7fd0600d7e80 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05effd700 1 --2- 172.21.2.117:0/3244238855 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0600d7e80 0x7fd0600d8b60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fd048006650 tx=0x7fd048009a80).ready entity=mon.1 client_cookie=592eb664aed7ca46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05e7fc700 1 -- 172.21.2.117:0/3244238855 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0480009b0 con 0x7fd0600d7e80 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05e7fc700 1 -- 172.21.2.117:0/3244238855 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04800c0c0 con 0x7fd0600d7e80 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd05e7fc700 1 -- 172.21.2.117:0/3244238855 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd04800c5e0 con 0x7fd0600d7e80 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd065a1d700 1 -- 172.21.2.117:0/3244238855 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0600d7e80 msgr2=0x7fd0600d8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.620 7fd065a1d700 1 --2- 172.21.2.117:0/3244238855 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0600d7e80 0x7fd0600d8b60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fd048006650 tx=0x7fd048009a80).stop 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 -- 172.21.2.117:0/3244238855 shutdown_connections 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 --2- 172.21.2.117:0/3244238855 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0600d7e80 0x7fd0600d8b60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 --2- 172.21.2.117:0/3244238855 >> v2:172.21.2.117:3300/0 conn(0x7fd0600d1870 0x7fd0600d7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 -- 172.21.2.117:0/3244238855 >> 172.21.2.117:0/3244238855 conn(0x7fd0600cecf0 msgr2=0x7fd0600cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 -- 172.21.2.117:0/3244238855 shutdown_connections 2021-02-10T11:40:30.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.621 7fd065a1d700 1 -- 172.21.2.117:0/3244238855 wait complete. 2021-02-10T11:40:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 Processor -- start 2021-02-10T11:40:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 -- start start 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0600aecc0 0x7fd060100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 0x7fd060083950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060084310 con 0x7fd060100720 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060083f70 con 0x7fd060100b20 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd065a1d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd060084890 con 0x7fd0600aecc0 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd05effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd05effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47964/0 (socket says 172.21.2.117:47964) 2021-02-10T11:40:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.636 7fd05effd700 1 -- 172.21.2.117:0/468225659 learned_addr learned my addr 172.21.2.117:0/468225659 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05ffff700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 0x7fd060083950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05f7fe700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0600aecc0 0x7fd060100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 -- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0600aecc0 msgr2=0x7fd060100100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0600aecc0 0x7fd060100100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 -- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 msgr2=0x7fd060083950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 0x7fd060083950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 -- 172.21.2.117:0/468225659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd060084ab0 con 0x7fd060100720 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05ffff700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 0x7fd060083950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd05effd700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7fd04800b5e0 tx=0x7fd04800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd048000940 con 0x7fd060100720 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04800bd70 con 0x7fd060100720 2021-02-10T11:40:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd048002300 con 0x7fd060100720 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.637 7fd065a1d700 1 -- 172.21.2.117:0/468225659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd060084ab0 con 0x7fd060100720 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.638 7fd065a1d700 1 -- 172.21.2.117:0/468225659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd060084ab0 con 0x7fd060100720 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.638 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd048000c90 con 0x7fd060100720 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.638 7fd057fff700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 0x7fd04401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.638 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd048032800 con 0x7fd060100720 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.639 7fd05f7fe700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 0x7fd04401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.639 7fd05f7fe700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 0x7fd04401abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:30.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.669 7fd065a1d700 1 -- 172.21.2.117:0/468225659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0600fc8d0 con 0x7fd060100720 2021-02-10T11:40:30.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.672 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd048009290 con 0x7fd060100720 2021-02-10T11:40:30.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.797 7fd065a1d700 1 -- 172.21.2.117:0/468225659 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000418555", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000025849"}) v1 -- 0x7fd0600fc8d0 con 0x7fd044018730 2021-02-10T11:40:30.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.833 7fd057fff700 1 -- 172.21.2.117:0/468225659 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd050006e70 con 0x7fd044018730 2021-02-10T11:40:30.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 msgr2=0x7fd04401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 0x7fd04401abc0 crc :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 msgr2=0x7fd060083330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:30.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7fd04800b5e0 tx=0x7fd04800b980).stop 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 shutdown_connections 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd044018730 0x7fd04401abc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd060100720 0x7fd060083330 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd060100b20 0x7fd060083950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 --2- 172.21.2.117:0/468225659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0600aecc0 0x7fd060100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 >> 172.21.2.117:0/468225659 conn(0x7fd060082eb0 msgr2=0x7fd0601b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:30.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 shutdown_connections 2021-02-10T11:40:30.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:30.863 7fd065a1d700 1 -- 172.21.2.117:0/468225659 wait complete. 2021-02-10T11:40:30.853 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 Processor -- start 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 -- start start 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa1280d1510 0x7fa1280d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa128077a70 con 0x7fa12807da20 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa128079400 con 0x7fa128079000 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12e44a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12807de20 con 0x7fa1280d1510 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12ca67700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa1280d1510 0x7fa1280d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12ca67700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa1280d1510 0x7fa1280d1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47972/0 (socket says 172.21.2.117:47972) 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.281 7fa12ca67700 1 -- 172.21.2.117:0/703478906 learned_addr learned my addr 172.21.2.117:0/703478906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:31.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12ca67700 1 -- 172.21.2.117:0/703478906 >> v1:172.21.2.125:6789/0 conn(0x7fa128079000 legacy=0x7fa1280e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12ca67700 1 -- 172.21.2.117:0/703478906 >> v1:172.21.2.117:6789/0 conn(0x7fa12807da20 legacy=0x7fa1280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12ca67700 1 -- 172.21.2.117:0/703478906 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1281b00d0 con 0x7fa1280d1510 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12ca67700 1 --2- 172.21.2.117:0/703478906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280d1510 0x7fa1280d1910 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fa120006f00 tx=0x7fa120009f10).ready entity=mon.0 client_cookie=3c6e25ea553eb692 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa126ffd700 1 -- 172.21.2.117:0/703478906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa12000b3b0 con 0x7fa1280d1510 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa126ffd700 1 -- 172.21.2.117:0/703478906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12000b630 con 0x7fa1280d1510 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa126ffd700 1 -- 172.21.2.117:0/703478906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa12000bb50 con 0x7fa1280d1510 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12e44a700 1 -- 172.21.2.117:0/703478906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280d1510 msgr2=0x7fa1280d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.282 7fa12e44a700 1 --2- 172.21.2.117:0/703478906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280d1510 0x7fa1280d1910 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7fa120006f00 tx=0x7fa120009f10).stop 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.283 7fa12e44a700 1 -- 172.21.2.117:0/703478906 shutdown_connections 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.283 7fa12e44a700 1 --2- 172.21.2.117:0/703478906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280d1510 0x7fa1280d1910 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.283 7fa12e44a700 1 -- 172.21.2.117:0/703478906 >> 172.21.2.117:0/703478906 conn(0x7fa1280ce430 msgr2=0x7fa1280e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.283 7fa12e44a700 1 -- 172.21.2.117:0/703478906 shutdown_connections 2021-02-10T11:40:31.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.283 7fa12e44a700 1 -- 172.21.2.117:0/703478906 wait complete. 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 Processor -- start 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 -- start start 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa128107380 0x7fa1280b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1280be660 0x7fa1280b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280b2a60 con 0x7fa1280b2610 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128107780 con 0x7fa1280be660 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.298 7fa12e44a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1281079a0 con 0x7fa128107380 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47980/0 (socket says 172.21.2.117:47980) 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 -- 172.21.2.117:0/4014691211 learned_addr learned my addr 172.21.2.117:0/4014691211 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa127fff700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa128107380 0x7fa1280b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa12ca67700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1280be660 0x7fa1280b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 -- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa128107380 msgr2=0x7fa1280b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa128107380 0x7fa1280b1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 -- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1280be660 msgr2=0x7fa1280b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1280be660 0x7fa1280b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 -- 172.21.2.117:0/4014691211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280b3c20 con 0x7fa1280b2610 2021-02-10T11:40:31.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.299 7fa1277fe700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7fa11c00b5b0 tx=0x7fa11c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa11c00bd40 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c000ce0 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa11c006210 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1280b3c20 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1280b41b0 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa11c006020 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 0x7fa10801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.300 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa11c032710 con 0x7fa1280b2610 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.301 7fa127fff700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 0x7fa10801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.301 7fa127fff700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 0x7fa10801abc0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.331 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa128106850 con 0x7fa1280b2610 2021-02-10T11:40:31.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.333 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa11c000940 con 0x7fa1280b2610 2021-02-10T11:40:31.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.461 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7fa1280ce430 con 0x7fa1280b2610 2021-02-10T11:40:31.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.465 7fa124ff9700 1 -- 172.21.2.117:0/4014691211 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v22) v1 ==== 89+0+0 (secure 0 0 0) 0x7fa11c006020 con 0x7fa1280b2610 2021-02-10T11:40:31.451 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:40:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 msgr2=0x7fa10801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 0x7fa10801abc0 crc :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 msgr2=0x7fa1280b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7fa11c00b5b0 tx=0x7fa11c00b950).stop 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 shutdown_connections 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1080186c0 0x7fa10801abc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1280b2610 0x7fa1280b2fe0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1280be660 0x7fa1280b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 --2- 172.21.2.117:0/4014691211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa128107380 0x7fa1280b1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 >> 172.21.2.117:0/4014691211 conn(0x7fa12809fd80 msgr2=0x7fa12809ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 shutdown_connections 2021-02-10T11:40:31.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.470 7fa12e44a700 1 -- 172.21.2.117:0/4014691211 wait complete. 2021-02-10T11:40:31.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000418555 --group_name group_0000000000025849 2021-02-10T11:40:31.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.892 7f02b7d9e700 1 Processor -- start 2021-02-10T11:40:31.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.892 7f02b7d9e700 1 -- start start 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b7d9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02b007da20 0x7f02b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b7d9e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f02b00e4e00 0x7f02b01b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b7d9e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b0077a10 con 0x7f02b00e5200 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b7d9e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b00777d0 con 0x7f02b007da20 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b7d9e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b007de20 con 0x7f02b00e4e00 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02b007da20 0x7f02b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f02b007da20 0x7f02b00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47984/0 (socket says 172.21.2.117:47984) 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b63bb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f02b00e5200 0x7f02b0077f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48236/0 (socket says 172.21.2.117:48236) 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b63bb700 1 -- 172.21.2.117:0/391356348 learned_addr learned my addr 172.21.2.117:0/391356348 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b4bb8700 1 -- 172.21.2.117:0/391356348 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842124105 0 0) 0x7f02a8000f90 con 0x7f02b00e5200 2021-02-10T11:40:31.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b4bb8700 1 -- 172.21.2.117:0/391356348 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029c002fe0 con 0x7f02b00e5200 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 -- 172.21.2.117:0/391356348 >> v2:172.21.2.125:3300/0 conn(0x7f02b00e4e00 msgr2=0x7f02b01b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 --2- 172.21.2.117:0/391356348 >> v2:172.21.2.125:3300/0 conn(0x7f02b00e4e00 0x7f02b01b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 -- 172.21.2.117:0/391356348 >> v1:172.21.2.117:6789/0 conn(0x7f02b00e5200 legacy=0x7f02b0077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 -- 172.21.2.117:0/391356348 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b01b0750 con 0x7f02b007da20 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.893 7f02b5bba700 1 --2- 172.21.2.117:0/391356348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b007da20 0x7f02b00771b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f02a4006650 tx=0x7f02a4009a80).ready entity=mon.0 client_cookie=57d5d8abd1b2066a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b4bb8700 1 -- 172.21.2.117:0/391356348 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a40009b0 con 0x7f02b007da20 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b4bb8700 1 -- 172.21.2.117:0/391356348 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400c0c0 con 0x7f02b007da20 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b4bb8700 1 -- 172.21.2.117:0/391356348 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a400c5e0 con 0x7f02b007da20 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 -- 172.21.2.117:0/391356348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b007da20 msgr2=0x7f02b00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 --2- 172.21.2.117:0/391356348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b007da20 0x7f02b00771b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f02a4006650 tx=0x7f02a4009a80).stop 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 -- 172.21.2.117:0/391356348 shutdown_connections 2021-02-10T11:40:31.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 --2- 172.21.2.117:0/391356348 >> v2:172.21.2.125:3300/0 conn(0x7f02b00e4e00 0x7f02b01b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 --2- 172.21.2.117:0/391356348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b007da20 0x7f02b00771b0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 -- 172.21.2.117:0/391356348 >> 172.21.2.117:0/391356348 conn(0x7f02b00ce480 msgr2=0x7f02b00e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.894 7f02b7d9e700 1 -- 172.21.2.117:0/391356348 shutdown_connections 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.895 7f02b7d9e700 1 -- 172.21.2.117:0/391356348 wait complete. 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 Processor -- start 2021-02-10T11:40:31.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 -- start start 2021-02-10T11:40:31.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02b0104dc0 0x7f02b01051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02b01057e0 0x7f02b0105be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00be6c0 con 0x7f02b009c210 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0104840 con 0x7f02b0104dc0 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.909 7f02b7d9e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01044f0 con 0x7f02b01057e0 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:47990/0 (socket says 172.21.2.117:47990) 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 -- 172.21.2.117:0/3582016108 learned_addr learned my addr 172.21.2.117:0/3582016108 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b5bba700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02b0104dc0 0x7f02b01051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b53b9700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02b01057e0 0x7f02b0105be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 -- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02b01057e0 msgr2=0x7f02b0105be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02b01057e0 0x7f02b0105be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 -- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02b0104dc0 msgr2=0x7f02b01051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02b0104dc0 0x7f02b01051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:31.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 -- 172.21.2.117:0/3582016108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b009cc30 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b63bb700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f02a8009980 tx=0x7f02a800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a8006030 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80063b0 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a8004720 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b009cc30 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.910 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b009cc30 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.911 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02a8000e40 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.911 7f02a27fc700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 0x7f029801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.911 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f02a8032ac0 con 0x7f02b009c210 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.911 7f02b5bba700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 0x7f029801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:31.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.912 7f02b5bba700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 0x7f029801abc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:31.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.942 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b01b70b0 con 0x7f02b009c210 2021-02-10T11:40:31.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:31.945 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02a800be10 con 0x7f02b009c210 2021-02-10T11:40:32.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.072 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000418555", "target": ["mgr", ""], "group_name": "group_0000000000025849"}) v1 -- 0x7f02b0077a10 con 0x7f02980186c0 2021-02-10T11:40:32.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.082 7f02a27fc700 1 -- 172.21.2.117:0/3582016108 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f02a4000c90 con 0x7f02980186c0 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 msgr2=0x7f029801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 0x7f029801abc0 crc :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 msgr2=0x7f02b009c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f02a8009980 tx=0x7f02a800bcd0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 shutdown_connections 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02980186c0 0x7f029801abc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02b009c210 0x7f02b009c610 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02b01057e0 0x7f02b0105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 --2- 172.21.2.117:0/3582016108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02b0104dc0 0x7f02b01051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 >> 172.21.2.117:0/3582016108 conn(0x7f02b00a6770 msgr2=0x7f02b01b92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 shutdown_connections 2021-02-10T11:40:32.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.090 7f02b7d9e700 1 -- 172.21.2.117:0/3582016108 wait complete. 2021-02-10T11:40:32.082 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000025849 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 Processor -- start 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 -- start start 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7198079000 0x7f71980e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71980d1d20 con 0x7f719807da20 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7198077a70 con 0x7f71981b00f0 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.513 7f719f265700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71980e5430 con 0x7f7198079000 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f719d081700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f719807da20 0x7f71980771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48248/0 (socket says 172.21.2.117:48248) 2021-02-10T11:40:32.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f719d081700 1 -- 172.21.2.117:0/2338974644 learned_addr learned my addr 172.21.2.117:0/2338974644 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f719c880700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7198079000 0x7f71980e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583753844 0 0) 0x7f718c0010b0 con 0x7f719807da20 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7184002fe0 con 0x7f719807da20 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416487429 0 0) 0x7f7190001400 con 0x7f71981b00f0 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7184003210 con 0x7f71981b00f0 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 130182032 0 0) 0x7f718c0010b0 con 0x7f719807da20 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7198079000 msgr2=0x7f71980e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7198079000 0x7f71980e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 >> v1:172.21.2.125:6789/0 conn(0x7f71981b00f0 legacy=0x7f71980d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7198079400 con 0x7f719807da20 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f718c0015b0 con 0x7f719807da20 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 -- 172.21.2.117:0/2338974644 >> v1:172.21.2.117:6789/0 conn(0x7f719807da20 legacy=0x7f71980771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71840038e0 0x7f7184003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71840067e0 0x7f7184008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.514 7f718bfff700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71840094f0 0x7f718400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840502d0 con 0x7f71840094f0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71840504b0 con 0x7f71840038e0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71840506d0 con 0x7f71840067e0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71840094f0 0x7f718400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d081700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71840038e0 0x7f7184003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719c880700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71840067e0 0x7f7184008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 -- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71840067e0 msgr2=0x7f7184008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71840067e0 0x7f7184008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 -- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71840038e0 msgr2=0x7f7184003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71840038e0 0x7f7184003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 -- 172.21.2.117:0/2338974644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184050970 con 0x7f71840094f0 2021-02-10T11:40:32.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71840094f0 0x7f718400ba10 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f71900013c0 tx=0x7f7190009e20).ready entity=mon.0 client_cookie=71555c0570c52696 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7190001400 con 0x7f71840094f0 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7190006a60 con 0x7f71840094f0 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f718bfff700 1 -- 172.21.2.117:0/2338974644 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71900019c0 con 0x7f71840094f0 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719f265700 1 -- 172.21.2.117:0/2338974644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71840094f0 msgr2=0x7f718400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719f265700 1 --2- 172.21.2.117:0/2338974644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71840094f0 0x7f718400ba10 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f71900013c0 tx=0x7f7190009e20).stop 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.515 7f719d882700 1 -- 172.21.2.117:0/2338974644 reap_dead start 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.516 7f719f265700 1 -- 172.21.2.117:0/2338974644 shutdown_connections 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.516 7f719f265700 1 -- 172.21.2.117:0/2338974644 >> 172.21.2.117:0/2338974644 conn(0x7f71980ce450 msgr2=0x7f71980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.516 7f719f265700 1 -- 172.21.2.117:0/2338974644 shutdown_connections 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.516 7f719f265700 1 -- 172.21.2.117:0/2338974644 wait complete. 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.530 7f719f265700 1 Processor -- start 2021-02-10T11:40:32.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 -- start start 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71980a47d0 0x7f71980a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71980a51f0 0x7f71980a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198189970 con 0x7f71980b77d0 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7198190aa0 con 0x7f71980a47d0 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719f265700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71980b81f0 con 0x7f71980a51f0 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719d882700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719d882700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48014/0 (socket says 172.21.2.117:48014) 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719d882700 1 -- 172.21.2.117:0/469906317 learned_addr learned my addr 172.21.2.117:0/469906317 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719d081700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71980a47d0 0x7f71980a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.531 7f719c880700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71980a51f0 0x7f71980a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 -- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71980a51f0 msgr2=0x7f71980a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71980a51f0 0x7f71980a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 -- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71980a47d0 msgr2=0x7f71980a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71980a47d0 0x7f71980a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 -- 172.21.2.117:0/469906317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980b8410 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719d882700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f719000b910 tx=0x7f719000bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71900030c0 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7190001180 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719f265700 1 -- 172.21.2.117:0/469906317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71980b8410 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f719f265700 1 -- 172.21.2.117:0/469906317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71980b89a0 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.532 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71900016a0 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.533 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f71900058c0 con 0x7f71980b77d0 2021-02-10T11:40:32.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.533 7f7189ffb700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 0x7f717801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:32.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.533 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7190032a30 con 0x7f71980b77d0 2021-02-10T11:40:32.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.533 7f719d081700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 0x7f717801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:32.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.534 7f719d081700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 0x7f717801abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:32.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.563 7f719f265700 1 -- 172.21.2.117:0/469906317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7198077a70 con 0x7f71980b77d0 2021-02-10T11:40:32.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.566 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7190006a60 con 0x7f71980b77d0 2021-02-10T11:40:32.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.692 7f719f265700 1 -- 172.21.2.117:0/469906317 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000025849"}) v1 -- 0x7f7198189970 con 0x7f71780186c0 2021-02-10T11:40:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.696 7f7189ffb700 1 -- 172.21.2.117:0/469906317 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f718c0010b0 con 0x7f71780186c0 2021-02-10T11:40:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 -- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 msgr2=0x7f717801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 0x7f717801abc0 crc :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 -- 172.21.2.117:0/469906317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 msgr2=0x7f71980b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f719000b910 tx=0x7f719000bcb0).stop 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 -- 172.21.2.117:0/469906317 shutdown_connections 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71780186c0 0x7f717801abc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71980b77d0 0x7f71980b7bd0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71980a51f0 0x7f71980a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 --2- 172.21.2.117:0/469906317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71980a47d0 0x7f71980a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:32.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.701 7f719f265700 1 -- 172.21.2.117:0/469906317 >> 172.21.2.117:0/469906317 conn(0x7f71980b5140 msgr2=0x7f71980a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:32.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.702 7f719f265700 1 -- 172.21.2.117:0/469906317 shutdown_connections 2021-02-10T11:40:32.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:32.702 7f719f265700 1 -- 172.21.2.117:0/469906317 wait complete. 2021-02-10T11:40:32.693 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:32.693 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:33.037 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:33.381 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:33.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:33.381 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:33.381 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:33.717 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:40:33.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:33.718 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:40:33.718 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:33.718 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:34.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[121196]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:34.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:34.072 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:34.072 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:34.406 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:40:34.407 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:34.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:34.407 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:34.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:34.742 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:34.742 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:35.078 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:40:35.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:35.078 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:40:35.078 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:40:35.078 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:35.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[121838]: fuse finished with error 0 and tester_r 0 2021-02-10T11:40:35.431 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:35.431 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:35.431 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:35.714 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:35.718 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:40:35.764 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:40:35.764 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:40:35.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:35.764 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:36.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:36.051 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:36.051 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:36.385 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:36.385 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:36.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:36.386 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:36.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:36.720 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:36.720 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:37.055 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:37.056 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:37.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:40:37.056 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:37.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:40:37.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.820 7f7a6d9c1700 1 Processor -- start 2021-02-10T11:40:37.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.820 7f7a6d9c1700 1 -- start start 2021-02-10T11:40:37.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a6d9c1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a68079000 0x7f7a680e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a6d9c1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a681b00d0 0x7f7a681b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a6d9c1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a68077a70 con 0x7f7a6807da20 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a6d9c1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680e5c00 con 0x7f7a68079000 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a6d9c1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680e53a0 con 0x7f7a681b00d0 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a67574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7a6807da20 0x7f7a680771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48268/0 (socket says 172.21.2.117:48268) 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a66d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a68079000 0x7f7a680e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a67574700 1 -- 172.21.2.117:0/2484195621 learned_addr learned my addr 172.21.2.117:0/2484195621 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a66d73700 1 -- 172.21.2.117:0/2484195621 >> v2:172.21.2.125:3300/0 conn(0x7f7a681b00d0 msgr2=0x7f7a681b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a66d73700 1 --2- 172.21.2.117:0/2484195621 >> v2:172.21.2.125:3300/0 conn(0x7f7a681b00d0 0x7f7a681b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a66d73700 1 -- 172.21.2.117:0/2484195621 >> v1:172.21.2.117:6789/0 conn(0x7f7a6807da20 legacy=0x7f7a680771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.821 7f7a66d73700 1 -- 172.21.2.117:0/2484195621 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68079400 con 0x7f7a68079000 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a66d73700 1 --2- 172.21.2.117:0/2484195621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a68079000 0x7f7a680e4d80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f7a50006650 tx=0x7f7a50009a00).ready entity=mon.0 client_cookie=f6cd19b2cb73d609 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a66572700 1 -- 172.21.2.117:0/2484195621 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a50009df0 con 0x7f7a68079000 2021-02-10T11:40:37.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a66572700 1 -- 172.21.2.117:0/2484195621 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a500009b0 con 0x7f7a68079000 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a66572700 1 -- 172.21.2.117:0/2484195621 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a5000c2b0 con 0x7f7a68079000 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a6d9c1700 1 -- 172.21.2.117:0/2484195621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a68079000 msgr2=0x7f7a680e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.822 7f7a6d9c1700 1 --2- 172.21.2.117:0/2484195621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a68079000 0x7f7a680e4d80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f7a50006650 tx=0x7f7a50009a00).stop 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 -- 172.21.2.117:0/2484195621 shutdown_connections 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 --2- 172.21.2.117:0/2484195621 >> v2:172.21.2.125:3300/0 conn(0x7f7a681b00d0 0x7f7a681b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 --2- 172.21.2.117:0/2484195621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a68079000 0x7f7a680e4d80 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 -- 172.21.2.117:0/2484195621 >> 172.21.2.117:0/2484195621 conn(0x7f7a680ce430 msgr2=0x7f7a680e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:37.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 -- 172.21.2.117:0/2484195621 shutdown_connections 2021-02-10T11:40:37.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.823 7f7a6d9c1700 1 -- 172.21.2.117:0/2484195621 wait complete. 2021-02-10T11:40:37.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.837 7f7a6d9c1700 1 Processor -- start 2021-02-10T11:40:37.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.837 7f7a6d9c1700 1 -- start start 2021-02-10T11:40:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a6809a450 0x7f7a680a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 0x7f7a6818d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680a3e20 con 0x7f7a6809a050 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680a4000 con 0x7f7a6818d370 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a6d9c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6818dd90 con 0x7f7a6809a450 2021-02-10T11:40:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48024/0 (socket says 172.21.2.117:48024) 2021-02-10T11:40:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67574700 1 -- 172.21.2.117:0/669925558 learned_addr learned my addr 172.21.2.117:0/669925558 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67d75700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 0x7f7a6818d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67574700 1 -- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a6809a450 msgr2=0x7f7a680a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.838 7f7a67574700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a6809a450 0x7f7a680a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a67574700 1 -- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 msgr2=0x7f7a6818d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a67574700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 0x7f7a6818d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a67574700 1 -- 172.21.2.117:0/669925558 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6818df70 con 0x7f7a6809a050 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a67d75700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 0x7f7a6818d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a67574700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f7a5800a520 tx=0x7f7a5800a8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a580070d0 con 0x7f7a6809a050 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5800acb0 con 0x7f7a6809a050 2021-02-10T11:40:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a58004620 con 0x7f7a6809a050 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6818df70 con 0x7f7a6809a050 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.839 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6818e500 con 0x7f7a6809a050 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.840 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a58008000 con 0x7f7a6809a050 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.840 7f7a5ffff700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 0x7f7a4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.840 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7a580326b0 con 0x7f7a6809a050 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.840 7f7a66d73700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 0x7f7a4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.841 7f7a66d73700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 0x7f7a4c01abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:37.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.871 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a68103370 con 0x7f7a6809a050 2021-02-10T11:40:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:37.873 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a58000ef0 con 0x7f7a6809a050 2021-02-10T11:40:38.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.021 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7a68077a70 con 0x7f7a6809a050 2021-02-10T11:40:38.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.024 7f7a5ffff700 1 -- 172.21.2.117:0/669925558 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7a58008000 con 0x7f7a6809a050 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 msgr2=0x7f7a4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 0x7f7a4c01abc0 crc :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 msgr2=0x7f7a680a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f7a5800a520 tx=0x7f7a5800a8c0).stop 2021-02-10T11:40:38.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 shutdown_connections 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a6818d370 0x7f7a6818d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a6809a450 0x7f7a680a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a4c0186c0 0x7f7a4c01abc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 --2- 172.21.2.117:0/669925558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a6809a050 0x7f7a680a31e0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.029 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 >> 172.21.2.117:0/669925558 conn(0x7f7a6819f610 msgr2=0x7f7a6809af00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.030 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 shutdown_connections 2021-02-10T11:40:38.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.030 7f7a6d9c1700 1 -- 172.21.2.117:0/669925558 wait complete. 2021-02-10T11:40:38.022 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option' 2021-02-10T11:40:38.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 Processor -- start 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 -- start start 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0f3807da20 0x7f0f380771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f380e5190 0x7f0f38079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f38077a70 con 0x7f0f380e4d90 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.448 7f0f3e0ca700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f380777d0 con 0x7f0f3807da20 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f3e0ca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f381b08d0 con 0x7f0f380e5190 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f36ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0f380e4d90 0x7f0f381b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48282/0 (socket says 172.21.2.117:48282) 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f36ffd700 1 -- 172.21.2.117:0/4277376157 learned_addr learned my addr 172.21.2.117:0/4277376157 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546862224 0 0) 0x7f0f200010b0 con 0x7f0f380e4d90 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f24002fe0 con 0x7f0f380e4d90 2021-02-10T11:40:38.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f377fe700 1 --2- 172.21.2.117:0/4277376157 >> v2:172.21.2.125:3300/0 conn(0x7f0f3807da20 0x7f0f380771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f37fff700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f380e5190 0x7f0f38079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 68016041 0 0) 0x7f0f200010b0 con 0x7f0f380e4d90 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f380e5190 msgr2=0x7f0f38079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f380e5190 0x7f0f38079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 >> v2:172.21.2.125:3300/0 conn(0x7f0f3807da20 msgr2=0x7f0f380771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 --2- 172.21.2.117:0/4277376157 >> v2:172.21.2.125:3300/0 conn(0x7f0f3807da20 0x7f0f380771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3807de20 con 0x7f0f380e4d90 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0f200015b0 con 0x7f0f380e4d90 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.449 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 >> v1:172.21.2.117:6789/0 conn(0x7f0f380e4d90 legacy=0x7f0f381b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f24003a60 0x7f0f24005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f24006790 0x7f0f24008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f24009500 0x7f0f2400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240502e0 con 0x7f0f24009500 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24050490 con 0x7f0f24003a60 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f240506b0 con 0x7f0f24006790 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f24009500 0x7f0f2400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f36ffd700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f24006790 0x7f0f24008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f377fe700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f24003a60 0x7f0f24005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 -- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f24006790 msgr2=0x7f0f24008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f24006790 0x7f0f24008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 -- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f24003a60 msgr2=0x7f0f24005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f24003a60 0x7f0f24005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 -- 172.21.2.117:0/4277376157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24050950 con 0x7f0f24009500 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f36ffd700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f24006790 0x7f0f24008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f37fff700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f24009500 0x7f0f2400ba20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f0f30006e50 tx=0x7f0f30009e70).ready entity=mon.0 client_cookie=8e76099395079ccf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.451 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f30006a20 con 0x7f0f24009500 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3000b300 con 0x7f0f24009500 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f367fc700 1 -- 172.21.2.117:0/4277376157 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f3000b760 con 0x7f0f24009500 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 -- 172.21.2.117:0/4277376157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f24009500 msgr2=0x7f0f2400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 --2- 172.21.2.117:0/4277376157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f24009500 0x7f0f2400ba20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f0f30006e50 tx=0x7f0f30009e70).stop 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f37fff700 1 -- 172.21.2.117:0/4277376157 reap_dead start 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 -- 172.21.2.117:0/4277376157 shutdown_connections 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 -- 172.21.2.117:0/4277376157 >> 172.21.2.117:0/4277376157 conn(0x7f0f380ce440 msgr2=0x7f0f380e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 -- 172.21.2.117:0/4277376157 shutdown_connections 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.452 7f0f3e0ca700 1 -- 172.21.2.117:0/4277376157 wait complete. 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 Processor -- start 2021-02-10T11:40:38.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 -- start start 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f381a9e00 0x7f0f381aa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f3819ea10 0x7f0f3819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f381aee80 con 0x7f0f380b4510 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3819f430 con 0x7f0f3819ea10 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.467 7f0f3e0ca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3819f650 con 0x7f0f381a9e00 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48048/0 (socket says 172.21.2.117:48048) 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 -- 172.21.2.117:0/1986976858 learned_addr learned my addr 172.21.2.117:0/1986976858 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 -- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f381a9e00 msgr2=0x7f0f381aa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f381a9e00 0x7f0f381aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 -- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f3819ea10 msgr2=0x7f0f3819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f3819ea10 0x7f0f3819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 -- 172.21.2.117:0/1986976858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f381aae00 con 0x7f0f380b4510 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f36ffd700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f0f2000c750 tx=0x7f0f20007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f20004350 con 0x7f0f380b4510 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f200045d0 con 0x7f0f380b4510 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f20004af0 con 0x7f0f380b4510 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f381aae00 con 0x7f0f380b4510 2021-02-10T11:40:38.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.468 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f381044e0 con 0x7f0f380b4510 2021-02-10T11:40:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.469 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f20004140 con 0x7f0f380b4510 2021-02-10T11:40:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.469 7f0f2ffff700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 0x7f0f1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.469 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0f200348b0 con 0x7f0f380b4510 2021-02-10T11:40:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.469 7f0f377fe700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 0x7f0f1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:38.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.470 7f0f377fe700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 0x7f0f1c01abc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:38.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.499 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f381032e0 con 0x7f0f380b4510 2021-02-10T11:40:38.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.502 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f200018f0 con 0x7f0f380b4510 2021-02-10T11:40:38.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:38.634 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]} v 0) v1 -- 0x7f0f38077a70 con 0x7f0f380b4510 2021-02-10T11:40:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.128 7f0f2ffff700 1 -- 172.21.2.117:0/1986976858 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_allow_existing_id_option"]}]=0 v244) v1 ==== 156+0+0 (secure 0 0 0) 0x7f0f20004140 con 0x7f0f380b4510 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 msgr2=0x7f0f1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 0x7f0f1c01abc0 crc :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 msgr2=0x7f0f380b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0x7f0f2000c750 tx=0x7f0f20007b10).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 shutdown_connections 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f1c0186c0 0x7f0f1c01abc0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f380b4510 0x7f0f380b4910 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f3819ea10 0x7f0f3819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 --2- 172.21.2.117:0/1986976858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f381a9e00 0x7f0f381aa200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 >> 172.21.2.117:0/1986976858 conn(0x7f0f3819c210 msgr2=0x7f0f381a9500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 shutdown_connections 2021-02-10T11:40:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.153 7f0f3e0ca700 1 -- 172.21.2.117:0/1986976858 wait complete. 2021-02-10T11:40:39.144 INFO:tasks.cephfs_test_runner:test_authorize_allow_existing_id_option (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:40:39.144 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:40:39.144 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 Processor -- start 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 -- start start 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc07da20 0x7f24fc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24fc0e5190 0x7f24fc0d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc077a70 con 0x7f24fc0e4d90 2021-02-10T11:40:39.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0e5c10 con 0x7f24fc0e5190 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2502b39700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0777d0 con 0x7f24fc07da20 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f2501156700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24fc0e5190 0x7f24fc0d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f24fbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f24fc0e4d90 0x7f24fc1b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48304/0 (socket says 172.21.2.117:48304) 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.572 7f24fbfff700 1 -- 172.21.2.117:0/2011242059 learned_addr learned my addr 172.21.2.117:0/2011242059 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2501156700 1 -- 172.21.2.117:0/2011242059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc07da20 msgr2=0x7f24fc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2501156700 1 --2- 172.21.2.117:0/2011242059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc07da20 0x7f24fc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2501156700 1 -- 172.21.2.117:0/2011242059 >> v1:172.21.2.117:6789/0 conn(0x7f24fc0e4d90 legacy=0x7f24fc1b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2501156700 1 -- 172.21.2.117:0/2011242059 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc1b08d0 con 0x7f24fc0e5190 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2501156700 1 --2- 172.21.2.117:0/2011242059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc0e5190 0x7f24fc0d0510 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f24f4006ab0 tx=0x7f24f4009b40).ready entity=mon.0 client_cookie=3cc1b1882dd6ec94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f24fb7fe700 1 -- 172.21.2.117:0/2011242059 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f40022e0 con 0x7f24fc0e5190 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f24fb7fe700 1 -- 172.21.2.117:0/2011242059 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f4002560 con 0x7f24fc0e5190 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f24fb7fe700 1 -- 172.21.2.117:0/2011242059 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f40029f0 con 0x7f24fc0e5190 2021-02-10T11:40:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2502b39700 1 -- 172.21.2.117:0/2011242059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc0e5190 msgr2=0x7f24fc0d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.573 7f2502b39700 1 --2- 172.21.2.117:0/2011242059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc0e5190 0x7f24fc0d0510 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f24f4006ab0 tx=0x7f24f4009b40).stop 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 -- 172.21.2.117:0/2011242059 shutdown_connections 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 --2- 172.21.2.117:0/2011242059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc0e5190 0x7f24fc0d0510 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 --2- 172.21.2.117:0/2011242059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc07da20 0x7f24fc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 -- 172.21.2.117:0/2011242059 >> 172.21.2.117:0/2011242059 conn(0x7f24fc0ce440 msgr2=0x7f24fc0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 -- 172.21.2.117:0/2011242059 shutdown_connections 2021-02-10T11:40:39.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.574 7f2502b39700 1 -- 172.21.2.117:0/2011242059 wait complete. 2021-02-10T11:40:39.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 Processor -- start 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 -- start start 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc10c5a0 0x7f24fc096130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24fc1ab450 0x7f24fc1ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc096930 con 0x7f24fc1abe70 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc10cce0 con 0x7f24fc1ab450 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2502b39700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc1ac270 con 0x7f24fc10c5a0 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2501156700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2501156700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48058/0 (socket says 172.21.2.117:48058) 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.589 7f2501156700 1 -- 172.21.2.117:0/3608102048 learned_addr learned my addr 172.21.2.117:0/3608102048 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 -- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc10c5a0 msgr2=0x7f24fc096130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc10c5a0 0x7f24fc096130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 -- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24fc1ab450 msgr2=0x7f24fc1ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24fc1ab450 0x7f24fc1ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 -- 172.21.2.117:0/3608102048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc10c9a0 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2501156700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f24f400b5e0 tx=0x7f24f400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f4009070 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f4002280 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f4002760 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2502b39700 1 -- 172.21.2.117:0/3608102048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc10c9a0 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.590 7f2502b39700 1 -- 172.21.2.117:0/3608102048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc0977c0 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.591 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24f4002460 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.591 7f24f97fa700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 0x7f24e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.591 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f24f40325b0 con 0x7f24fc1abe70 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.591 7f2500955700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 0x7f24e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:39.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.592 7f2500955700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 0x7f24e001abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:39.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.622 7f2502b39700 1 -- 172.21.2.117:0/3608102048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc108bd0 con 0x7f24fc1abe70 2021-02-10T11:40:39.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.625 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24f40093e0 con 0x7f24fc1abe70 2021-02-10T11:40:39.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:39.754 7f2502b39700 1 -- 172.21.2.117:0/3608102048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7f24fc077a70 con 0x7f24fc1abe70 2021-02-10T11:40:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.132 7f24f97fa700 1 -- 172.21.2.117:0/3608102048 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v245) v1 ==== 169+0+0 (secure 0 0 0) 0x7f24f4002460 con 0x7f24fc1abe70 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 msgr2=0x7f24e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 0x7f24e001abc0 crc :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 msgr2=0x7f24fc096eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f24f400b5e0 tx=0x7f24f400b980).stop 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 shutdown_connections 2021-02-10T11:40:40.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24e00186c0 0x7f24e001abc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24fc1abe70 0x7f24fc096eb0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24fc1ab450 0x7f24fc1ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 --2- 172.21.2.117:0/3608102048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24fc10c5a0 0x7f24fc096130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 >> 172.21.2.117:0/3608102048 conn(0x7f24fc10d7d0 msgr2=0x7f24fc0997f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 shutdown_connections 2021-02-10T11:40:40.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.174 7f2502b39700 1 -- 172.21.2.117:0/3608102048 wait complete. 2021-02-10T11:40:40.165 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.596 7fd8bb857700 1 Processor -- start 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.596 7fd8bb857700 1 -- start start 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.596 7fd8bb857700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8b407da20 0x7fd8b40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8bb857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40e4da0 0x7fd8b41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8bb857700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd8b40e51a0 0x7fd8b4077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8bb857700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b4077a70 con 0x7fd8b407da20 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8bb857700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40777d0 con 0x7fd8b40e51a0 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8bb857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b407de20 con 0x7fd8b40e4da0 2021-02-10T11:40:40.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8b407da20 0x7fd8b40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8b407da20 0x7fd8b40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48064/0 (socket says 172.21.2.117:48064) 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 -- 172.21.2.117:0/4031959470 learned_addr learned my addr 172.21.2.117:0/4031959470 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 -- 172.21.2.117:0/4031959470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40e4da0 msgr2=0x7fd8b41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 --2- 172.21.2.117:0/4031959470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40e4da0 0x7fd8b41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 -- 172.21.2.117:0/4031959470 >> v2:172.21.2.125:3300/0 conn(0x7fd8b40e51a0 msgr2=0x7fd8b4077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 --2- 172.21.2.117:0/4031959470 >> v2:172.21.2.125:3300/0 conn(0x7fd8b40e51a0 0x7fd8b4077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 -- 172.21.2.117:0/4031959470 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b41b0710 con 0x7fd8b407da20 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8b9673700 1 --2- 172.21.2.117:0/4031959470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b407da20 0x7fd8b40771b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fd8a8006700 tx=0x7fd8a8009b80).ready entity=mon.0 client_cookie=e9a2e35fe3739b8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.597 7fd8a7fff700 1 -- 172.21.2.117:0/4031959470 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8a800c0c0 con 0x7fd8b407da20 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8a7fff700 1 -- 172.21.2.117:0/4031959470 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a800c340 con 0x7fd8b407da20 2021-02-10T11:40:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8a7fff700 1 -- 172.21.2.117:0/4031959470 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8a800c860 con 0x7fd8b407da20 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 -- 172.21.2.117:0/4031959470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b407da20 msgr2=0x7fd8b40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 --2- 172.21.2.117:0/4031959470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b407da20 0x7fd8b40771b0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7fd8a8006700 tx=0x7fd8a8009b80).stop 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 -- 172.21.2.117:0/4031959470 shutdown_connections 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 --2- 172.21.2.117:0/4031959470 >> v2:172.21.2.125:3300/0 conn(0x7fd8b40e51a0 0x7fd8b4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 --2- 172.21.2.117:0/4031959470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40e4da0 0x7fd8b41b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 --2- 172.21.2.117:0/4031959470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b407da20 0x7fd8b40771b0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.598 7fd8bb857700 1 -- 172.21.2.117:0/4031959470 >> 172.21.2.117:0/4031959470 conn(0x7fd8b40ce450 msgr2=0x7fd8b40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.599 7fd8bb857700 1 -- 172.21.2.117:0/4031959470 shutdown_connections 2021-02-10T11:40:40.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.599 7fd8bb857700 1 -- 172.21.2.117:0/4031959470 wait complete. 2021-02-10T11:40:40.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.613 7fd8bb857700 1 Processor -- start 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.613 7fd8bb857700 1 -- start start 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40b3ff0 0x7fd8b40b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8b4090de0 0x7fd8b40911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4090600 con 0x7fd8b40c33d0 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4090860 con 0x7fd8b4090de0 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8bb857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40b3c70 con 0x7fd8b40b3ff0 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48070/0 (socket says 172.21.2.117:48070) 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 -- 172.21.2.117:0/728859273 learned_addr learned my addr 172.21.2.117:0/728859273 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 -- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40b3ff0 msgr2=0x7fd8b40b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40b3ff0 0x7fd8b40b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 -- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8b4090de0 msgr2=0x7fd8b40911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8b4090de0 0x7fd8b40911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 -- 172.21.2.117:0/728859273 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4091800 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.614 7fd8b8e72700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7fd8b000b530 tx=0x7fd8b000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8b000bcc0 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b00009b0 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8b0002300 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8bb857700 1 -- 172.21.2.117:0/728859273 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b4091800 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8bb857700 1 -- 172.21.2.117:0/728859273 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b4091800 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8b0002d20 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 0x7fd89401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.615 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd8b00326c0 con 0x7fd8b40c33d0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.616 7fd8b9673700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 0x7fd89401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:40.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.617 7fd8b9673700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 0x7fd89401abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:40.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.646 7fd8bb857700 1 -- 172.21.2.117:0/728859273 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b4103370 con 0x7fd8b40c33d0 2021-02-10T11:40:40.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.649 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8b0000c90 con 0x7fd8b40c33d0 2021-02-10T11:40:40.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.798 7fd8bb857700 1 -- 172.21.2.117:0/728859273 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd8b4077a70 con 0x7fd8b40c33d0 2021-02-10T11:40:40.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.802 7fd8a5ffb700 1 -- 172.21.2.117:0/728859273 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8b0002d20 con 0x7fd8b40c33d0 2021-02-10T11:40:40.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 msgr2=0x7fd89401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 0x7fd89401abc0 crc :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 msgr2=0x7fd8b40b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:40.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7fd8b000b530 tx=0x7fd8b000b8d0).stop 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 shutdown_connections 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8940186c0 0x7fd89401abc0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8b40c33d0 0x7fd8b40b3650 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8b4090de0 0x7fd8b40911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 --2- 172.21.2.117:0/728859273 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8b40b3ff0 0x7fd8b40b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 >> 172.21.2.117:0/728859273 conn(0x7fd8b40a1420 msgr2=0x7fd8b409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 shutdown_connections 2021-02-10T11:40:40.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:40.807 7fd8bb857700 1 -- 172.21.2.117:0/728859273 wait complete. 2021-02-10T11:40:40.798 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:40:40.799 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:41.144 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:40:41.488 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:40:41.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:41.488 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:41.488 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:40:41.823 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:40:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:41.823 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:40:41.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:41.823 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:41.823 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:40:42.158 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:40:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:42.159 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:40:42.159 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:42.159 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:42.159 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:40:42.493 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:40:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:42.494 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:40:42.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:40:42.494 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:40:42.494 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:40:42.827 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:40:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:40:42.828 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:40:42.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:40:43.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.254 7fc59156b700 1 Processor -- start 2021-02-10T11:40:43.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.254 7fc59156b700 1 -- start start 2021-02-10T11:40:43.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc59156b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc58c0d1520 0x7fc58c0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc59156b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c077a70 con 0x7fc58c07da20 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc59156b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c079400 con 0x7fc58c079000 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc59156b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c07de20 con 0x7fc58c0d1520 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58b2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc58c07da20 0x7fc58c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48326/0 (socket says 172.21.2.117:48326) 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58b2bf700 1 -- 172.21.2.117:0/4034740612 learned_addr learned my addr 172.21.2.117:0/4034740612 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58bac0700 1 --2- 172.21.2.117:0/4034740612 >> v2:172.21.2.117:3300/0 conn(0x7fc58c0d1520 0x7fc58c0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58bac0700 1 -- 172.21.2.117:0/4034740612 >> v1:172.21.2.125:6789/0 conn(0x7fc58c079000 legacy=0x7fc58c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58bac0700 1 -- 172.21.2.117:0/4034740612 >> v1:172.21.2.117:6789/0 conn(0x7fc58c07da20 legacy=0x7fc58c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58a2bd700 1 -- 172.21.2.117:0/4034740612 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556022208 0 0) 0x7fc57c0010b0 con 0x7fc58c07da20 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.255 7fc58bac0700 1 -- 172.21.2.117:0/4034740612 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c1b00e0 con 0x7fc58c0d1520 2021-02-10T11:40:43.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc58bac0700 1 --2- 172.21.2.117:0/4034740612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c0d1520 0x7fc58c0d1920 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fc584006e30 tx=0x7fc584009de0).ready entity=mon.0 client_cookie=bf977643e34fcd5f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc58a2bd700 1 -- 172.21.2.117:0/4034740612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc58400b300 con 0x7fc58c0d1520 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc58a2bd700 1 -- 172.21.2.117:0/4034740612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400b580 con 0x7fc58c0d1520 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc58a2bd700 1 -- 172.21.2.117:0/4034740612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc58400baa0 con 0x7fc58c0d1520 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 -- 172.21.2.117:0/4034740612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c0d1520 msgr2=0x7fc58c0d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 --2- 172.21.2.117:0/4034740612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c0d1520 0x7fc58c0d1920 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0x7fc584006e30 tx=0x7fc584009de0).stop 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 -- 172.21.2.117:0/4034740612 shutdown_connections 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 --2- 172.21.2.117:0/4034740612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c0d1520 0x7fc58c0d1920 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 -- 172.21.2.117:0/4034740612 >> 172.21.2.117:0/4034740612 conn(0x7fc58c0ce440 msgr2=0x7fc58c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 -- 172.21.2.117:0/4034740612 shutdown_connections 2021-02-10T11:40:43.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.256 7fc59156b700 1 -- 172.21.2.117:0/4034740612 wait complete. 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 Processor -- start 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 -- start start 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc58c0b0750 0x7fc58c08c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc58c1ad6f0 0x7fc58c1ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1a7d80 con 0x7fc58c1accd0 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c08d290 con 0x7fc58c0b0750 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc59156b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c08ce80 con 0x7fc58c1ad6f0 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc58aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc58aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48086/0 (socket says 172.21.2.117:48086) 2021-02-10T11:40:43.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.271 7fc58aabe700 1 -- 172.21.2.117:0/3303724017 learned_addr learned my addr 172.21.2.117:0/3303724017 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 -- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc58c1ad6f0 msgr2=0x7fc58c1ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc58c1ad6f0 0x7fc58c1ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 -- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc58c0b0750 msgr2=0x7fc58c08c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc58c0b0750 0x7fc58c08c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 -- 172.21.2.117:0/3303724017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c1adaf0 con 0x7fc58c1accd0 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc58aabe700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fc574006ed0 tx=0x7fc57400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc574006020 con 0x7fc58c1accd0 2021-02-10T11:40:43.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5740009b0 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc59156b700 1 -- 172.21.2.117:0/3303724017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c1adaf0 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5740054b0 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.272 7fc59156b700 1 -- 172.21.2.117:0/3303724017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c1abda0 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.273 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc574006020 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.273 7fc583fff700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 0x7fc57001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.273 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc574032570 con 0x7fc58c1accd0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.273 7fc58b2bf700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 0x7fc57001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.274 7fc58b2bf700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 0x7fc57001abc0 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.305 7fc59156b700 1 -- 172.21.2.117:0/3303724017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c104a20 con 0x7fc58c1accd0 2021-02-10T11:40:43.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.307 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc574030d50 con 0x7fc58c1accd0 2021-02-10T11:40:43.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.431 7fc59156b700 1 -- 172.21.2.117:0/3303724017 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc58c077a70 con 0x7fc58c1accd0 2021-02-10T11:40:43.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.432 7fc583fff700 1 -- 172.21.2.117:0/3303724017 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v39) v1 ==== 81+0+30752 (secure 0 0 0) 0x7fc574006020 con 0x7fc58c1accd0 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 39, 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:40:04.882873", 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:40:43.416 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:40:43.417 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 7, 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:40:43.418 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:40:43.419 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:40:43.420 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:43.420 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:40:43.421 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:40:43.421 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:40:43.421 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:43.421 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:43.422 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:40:43.423 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:40:43.424 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:40:43.425 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:40:43.426 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 6, 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:40:01.847956", 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:40:43.427 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:40:43.428 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "38", 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:40:43.429 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:40:43.430 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:40:43.431 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:40:43.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 7, 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:40:02.928003", 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:40:43.433 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:40:43.434 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "39", 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:40:43.435 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:40:43.436 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:40:43.437 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:40:43.438 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.439 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.440 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.441 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.444 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.445 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 36, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.446 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.447 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:40:43.448 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.449 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.449 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.449 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.449 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:40:43.449 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.450 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.450 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.450 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.450 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.451 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:40:43.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.452 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.453 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.453 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.453 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.453 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.453 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.454 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.454 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:40:43.454 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.454 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.454 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:40:43.455 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.456 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.457 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.458 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.459 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.460 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.461 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.462 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:40:43.463 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.464 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.465 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.466 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.466 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.466 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:40:43.466 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.466 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.467 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.468 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:40:43.469 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.470 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.471 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.472 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.473 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.474 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.475 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:40:43.476 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.477 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.478 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.479 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.480 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.481 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.482 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.483 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:40:43.484 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:40:43.485 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.486 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.487 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:40:43.488 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:40:43.489 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 37, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:40:43.490 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.491 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:40:43.492 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.493 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.494 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:40:43.495 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.496 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.497 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.498 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:40:43.499 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.500 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:40:43.501 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:40:43.502 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:40:43.503 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:40:43.504 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:40:43.504 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:40:43.504 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:40:43.504 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.433 7fc59156b700 1 -- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 msgr2=0x7fc57001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.433 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 0x7fc57001abc0 crc :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.433 7fc59156b700 1 -- 172.21.2.117:0/3303724017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 msgr2=0x7fc58c1ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.433 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7fc574006ed0 tx=0x7fc57400b960).stop 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 -- 172.21.2.117:0/3303724017 shutdown_connections 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5700186c0 0x7fc57001abc0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc58c1accd0 0x7fc58c1ad0d0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc58c1ad6f0 0x7fc58c1ab490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 --2- 172.21.2.117:0/3303724017 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc58c0b0750 0x7fc58c08c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 -- 172.21.2.117:0/3303724017 >> 172.21.2.117:0/3303724017 conn(0x7fc58c0970a0 msgr2=0x7fc58c10d060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 -- 172.21.2.117:0/3303724017 shutdown_connections 2021-02-10T11:40:43.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.434 7fc59156b700 1 -- 172.21.2.117:0/3303724017 wait complete. 2021-02-10T11:40:43.507 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.933 7f0678b53700 1 Processor -- start 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 -- start start 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f067407da20 0x7f06740771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06740e4d90 0x7f06741b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 0x7f0674077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06740e5c10 con 0x7f06740e4d90 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0674077a70 con 0x7f06740e5190 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f0678b53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06740777d0 con 0x7f067407da20 2021-02-10T11:40:43.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f067205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06740e4d90 0x7f06741b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f067205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06740e4d90 0x7f06741b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48090/0 (socket says 172.21.2.117:48090) 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.934 7f067205d700 1 -- 172.21.2.117:0/3008248603 learned_addr learned my addr 172.21.2.117:0/3008248603 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067305f700 1 --2- 172.21.2.117:0/3008248603 >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 0x7f0674077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067285e700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f067407da20 0x7f06740771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 -- 172.21.2.117:0/3008248603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f067407da20 msgr2=0x7f06740771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f067407da20 0x7f06740771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 -- 172.21.2.117:0/3008248603 >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 msgr2=0x7f0674077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 --2- 172.21.2.117:0/3008248603 >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 0x7f0674077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 -- 172.21.2.117:0/3008248603 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067407de20 con 0x7f06740e4d90 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067305f700 1 --2- 172.21.2.117:0/3008248603 >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 0x7f0674077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.935 7f067205d700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06740e4d90 0x7f06741b00e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f0668006700 tx=0x7f0668009b80).ready entity=mon.0 client_cookie=6d6c8c512d2fbae6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f067185c700 1 -- 172.21.2.117:0/3008248603 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06680009b0 con 0x7f06740e4d90 2021-02-10T11:40:43.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f067185c700 1 -- 172.21.2.117:0/3008248603 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066800c0c0 con 0x7f06740e4d90 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f067185c700 1 -- 172.21.2.117:0/3008248603 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f066800c530 con 0x7f06740e4d90 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 -- 172.21.2.117:0/3008248603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06740e4d90 msgr2=0x7f06741b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06740e4d90 0x7f06741b00e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f0668006700 tx=0x7f0668009b80).stop 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 -- 172.21.2.117:0/3008248603 shutdown_connections 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 --2- 172.21.2.117:0/3008248603 >> v2:172.21.2.125:3300/0 conn(0x7f06740e5190 0x7f0674077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06740e4d90 0x7f06741b00e0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 --2- 172.21.2.117:0/3008248603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f067407da20 0x7f06740771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.936 7f0678b53700 1 -- 172.21.2.117:0/3008248603 >> 172.21.2.117:0/3008248603 conn(0x7f06740ce440 msgr2=0x7f06740e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.937 7f0678b53700 1 -- 172.21.2.117:0/3008248603 shutdown_connections 2021-02-10T11:40:43.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.937 7f0678b53700 1 -- 172.21.2.117:0/3008248603 wait complete. 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 Processor -- start 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 -- start start 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06741a3b40 0x7f06741a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06741a4560 0x7f0674187fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0674189550 con 0x7f0674106950 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.951 7f0678b53700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06740ad1c0 con 0x7f06741a3b40 2021-02-10T11:40:43.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f0678b53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0674188630 con 0x7f06741a4560 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48096/0 (socket says 172.21.2.117:48096) 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 -- 172.21.2.117:0/3603088519 learned_addr learned my addr 172.21.2.117:0/3603088519 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 -- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06741a4560 msgr2=0x7f0674187fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06741a4560 0x7f0674187fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 -- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06741a3b40 msgr2=0x7f06741a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06741a3b40 0x7f06741a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 -- 172.21.2.117:0/3603088519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674188850 con 0x7f0674106950 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.952 7f067285e700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f0664006f60 tx=0x7f066400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f066400bd60 con 0x7f0674106950 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0664000c90 con 0x7f0674106950 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0664002220 con 0x7f0674106950 2021-02-10T11:40:43.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f0678b53700 1 -- 172.21.2.117:0/3603088519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0674188850 con 0x7f0674106950 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f0678b53700 1 -- 172.21.2.117:0/3603088519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0674188850 con 0x7f0674106950 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06640052e0 con 0x7f0674106950 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.953 7f06637fe700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 0x7f065801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.954 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f06640326b0 con 0x7f0674106950 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.954 7f067205d700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 0x7f065801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:43.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.954 7f067205d700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 0x7f065801abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:43.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.984 7f0678b53700 1 -- 172.21.2.117:0/3603088519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06740ad190 con 0x7f0674106950 2021-02-10T11:40:43.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:43.986 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0664006b00 con 0x7f0674106950 2021-02-10T11:40:44.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.111 7f0678b53700 1 -- 172.21.2.117:0/3603088519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06740ad190 con 0x7f0674106950 2021-02-10T11:40:44.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.112 7f06637fe700 1 -- 172.21.2.117:0/3603088519 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2096 (secure 0 0 0) 0x7f066400bd60 con 0x7f0674106950 2021-02-10T11:40:44.096 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:44.096 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:05.887046","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7233":{"gid":7233,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/107410323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":107410323},{"type":"v1","addr":"172.21.2.125:6835","nonce":107410323}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:44.096 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 21 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 msgr2=0x7f065801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 0x7f065801abc0 crc :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 msgr2=0x7f0674106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0x7f0664006f60 tx=0x7f066400b970).stop 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 shutdown_connections 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06741a4560 0x7f0674187fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06741a3b40 0x7f06741a3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:44.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06580186c0 0x7f065801abc0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 --2- 172.21.2.117:0/3603088519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0674106950 0x7f0674106d50 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 >> 172.21.2.117:0/3603088519 conn(0x7f067419f4c0 msgr2=0x7f0674091f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:44.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 shutdown_connections 2021-02-10T11:40:44.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.114 7f0678b53700 1 -- 172.21.2.117:0/3603088519 wait complete. 2021-02-10T11:40:44.106 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:40:44.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.532 7fcc6aa30700 1 Processor -- start 2021-02-10T11:40:44.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.532 7fcc6aa30700 1 -- start start 2021-02-10T11:40:44.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6aa30700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc64079000 0x7fcc640e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6aa30700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcc641b00d0 0x7fcc641b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6aa30700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc64077a70 con 0x7fcc6407da20 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6aa30700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640e53a0 con 0x7fcc641b00d0 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6aa30700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64079400 con 0x7fcc64079000 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6904d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcc641b00d0 0x7fcc641b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6904d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcc641b00d0 0x7fcc641b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48104/0 (socket says 172.21.2.117:48104) 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc6904d700 1 -- 172.21.2.117:0/1217577637 learned_addr learned my addr 172.21.2.117:0/1217577637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.533 7fcc63fff700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc64079000 0x7fcc640e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6904d700 1 -- 172.21.2.117:0/1217577637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc64079000 msgr2=0x7fcc640e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:44.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6904d700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc64079000 0x7fcc640e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6904d700 1 -- 172.21.2.117:0/1217577637 >> v1:172.21.2.117:6789/0 conn(0x7fcc6407da20 legacy=0x7fcc640771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6904d700 1 -- 172.21.2.117:0/1217577637 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6407de20 con 0x7fcc641b00d0 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6904d700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc641b00d0 0x7fcc641b04d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fcc5c006b80 tx=0x7fcc5c009b90).ready entity=mon.0 client_cookie=138c128fd3fd6354 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc637fe700 1 -- 172.21.2.117:0/1217577637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5c0022e0 con 0x7fcc641b00d0 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc637fe700 1 -- 172.21.2.117:0/1217577637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c002560 con 0x7fcc641b00d0 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc637fe700 1 -- 172.21.2.117:0/1217577637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc5c002a80 con 0x7fcc641b00d0 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6aa30700 1 -- 172.21.2.117:0/1217577637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc641b00d0 msgr2=0x7fcc641b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.534 7fcc6aa30700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc641b00d0 0x7fcc641b04d0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0x7fcc5c006b80 tx=0x7fcc5c009b90).stop 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 -- 172.21.2.117:0/1217577637 shutdown_connections 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc641b00d0 0x7fcc641b04d0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 --2- 172.21.2.117:0/1217577637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc64079000 0x7fcc640e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 -- 172.21.2.117:0/1217577637 >> 172.21.2.117:0/1217577637 conn(0x7fcc640ce430 msgr2=0x7fcc640e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:44.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 -- 172.21.2.117:0/1217577637 shutdown_connections 2021-02-10T11:40:44.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.535 7fcc6aa30700 1 -- 172.21.2.117:0/1217577637 wait complete. 2021-02-10T11:40:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 Processor -- start 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 -- start start 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 0x7fcc641b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 0x7fcc640b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc641b7d70 con 0x7fcc6408a180 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640b1c90 con 0x7fcc641b82f0 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.550 7fcc6aa30700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640b1eb0 con 0x7fcc641b8d10 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48108/0 (socket says 172.21.2.117:48108) 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 -- 172.21.2.117:0/267099080 learned_addr learned my addr 172.21.2.117:0/267099080 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc63fff700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 0x7fcc641b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6904d700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 0x7fcc640b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 -- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 msgr2=0x7fcc640b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 0x7fcc640b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 -- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 msgr2=0x7fcc641b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 0x7fcc641b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 -- 172.21.2.117:0/267099080 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640b20d0 con 0x7fcc6408a180 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc63fff700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 0x7fcc641b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6904d700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 0x7fcc640b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.551 7fcc6884c700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fcc4c006ed0 tx=0x7fcc4c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4c006240 con 0x7fcc6408a180 2021-02-10T11:40:44.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c0054b0 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc4c0059d0 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc640b20d0 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc64196d10 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc4c006030 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.552 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcc4c0326e0 con 0x7fcc6408a180 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.553 7fcc63fff700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:44.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.553 7fcc63fff700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 0x7fcc4801abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:44.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.583 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc641b4ee0 con 0x7fcc6408a180 2021-02-10T11:40:44.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.585 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc4c030d40 con 0x7fcc6408a180 2021-02-10T11:40:44.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:44.713 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc64077a70 con 0x7fcc6408a180 2021-02-10T11:40:45.106 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:40:45.123 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:40:45.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.129 7fcc617fa700 1 -- 172.21.2.117:0/267099080 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc4c006030 con 0x7fcc6408a180 2021-02-10T11:40:45.162 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:40:45.163 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:40:45.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 msgr2=0x7fcc4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 0x7fcc4801abc0 crc :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 msgr2=0x7fcc64196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fcc4c006ed0 tx=0x7fcc4c00b960).stop 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 shutdown_connections 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc641b8d10 0x7fcc640b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc641b82f0 0x7fcc641b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc480186c0 0x7fcc4801abc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 --2- 172.21.2.117:0/267099080 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc6408a180 0x7fcc64196480 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 >> 172.21.2.117:0/267099080 conn(0x7fcc64198370 msgr2=0x7fcc641b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 shutdown_connections 2021-02-10T11:40:45.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.181 7fcc6aa30700 1 -- 172.21.2.117:0/267099080 wait complete. 2021-02-10T11:40:45.172 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:45.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.598 7f8cf8727700 1 Processor -- start 2021-02-10T11:40:45.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.598 7f8cf8727700 1 -- start start 2021-02-10T11:40:45.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf8727700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8cf00d0ac0 0x7f8cf01b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf8727700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00e51a0 0x7f8cf01b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf8727700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf0109ab0 con 0x7f8cf01b3470 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf8727700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf00d0ec0 con 0x7f8cf00d0ac0 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf8727700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00d10b0 con 0x7f8cf00e51a0 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf5d42700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8cf01b3470 0x7f8cf01085f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48366/0 (socket says 172.21.2.117:48366) 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf5d42700 1 -- 172.21.2.117:0/2839647812 learned_addr learned my addr 172.21.2.117:0/2839647812 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.599 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428238452 0 0) 0x7f8ce40010b0 con 0x7f8cf01b3470 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cdc002fe0 con 0x7f8cf01b3470 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf6543700 1 --2- 172.21.2.117:0/2839647812 >> v2:172.21.2.125:3300/0 conn(0x7f8cf00d0ac0 0x7f8cf01b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf6d44700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00e51a0 0x7f8cf01b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2183595785 0 0) 0x7f8ce40010b0 con 0x7f8cf01b3470 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00e51a0 msgr2=0x7f8cf01b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00e51a0 0x7f8cf01b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 >> v2:172.21.2.125:3300/0 conn(0x7f8cf00d0ac0 msgr2=0x7f8cf01b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 --2- 172.21.2.117:0/2839647812 >> v2:172.21.2.125:3300/0 conn(0x7f8cf00d0ac0 0x7f8cf01b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf010b040 con 0x7f8cf01b3470 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8ce40015b0 con 0x7f8cf01b3470 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 >> v1:172.21.2.117:6789/0 conn(0x7f8cf01b3470 legacy=0x7f8cf01085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cdc003ac0 0x7f8cdc005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cdc04a780 0x7f8cdc04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cdc04d520 0x7f8cdc04fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0502f0 con 0x7f8cdc04a780 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc050510 con 0x7f8cdc04d520 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc050730 con 0x7f8cdc003ac0 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5d42700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cdc04a780 0x7f8cdc04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf6543700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cdc003ac0 0x7f8cdc005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf6d44700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cdc04d520 0x7f8cdc04fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5d42700 1 -- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cdc003ac0 msgr2=0x7f8cdc005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5d42700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cdc003ac0 0x7f8cdc005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5d42700 1 -- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cdc04d520 msgr2=0x7f8cdc04fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.600 7f8cf5d42700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cdc04d520 0x7f8cdc04fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf5d42700 1 -- 172.21.2.117:0/2839647812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc0509d0 con 0x7f8cdc04a780 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf5d42700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cdc04a780 0x7f8cdc04cc80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f8ce4001310 tx=0x7f8ce4007cc0).ready entity=mon.0 client_cookie=ecbe5b53d7c62742 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce400a310 con 0x7f8cdc04a780 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce400a610 con 0x7f8cdc04a780 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf5541700 1 -- 172.21.2.117:0/2839647812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce400aaf0 con 0x7f8cdc04a780 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf8727700 1 -- 172.21.2.117:0/2839647812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cdc04a780 msgr2=0x7f8cdc04cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf8727700 1 --2- 172.21.2.117:0/2839647812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cdc04a780 0x7f8cdc04cc80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f8ce4001310 tx=0x7f8ce4007cc0).stop 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.601 7f8cf6d44700 1 -- 172.21.2.117:0/2839647812 reap_dead start 2021-02-10T11:40:45.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.602 7f8cf8727700 1 -- 172.21.2.117:0/2839647812 shutdown_connections 2021-02-10T11:40:45.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.602 7f8cf8727700 1 -- 172.21.2.117:0/2839647812 >> 172.21.2.117:0/2839647812 conn(0x7f8cf00ce410 msgr2=0x7f8cf00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:45.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.602 7f8cf8727700 1 -- 172.21.2.117:0/2839647812 shutdown_connections 2021-02-10T11:40:45.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.602 7f8cf8727700 1 -- 172.21.2.117:0/2839647812 wait complete. 2021-02-10T11:40:45.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.616 7f8cf8727700 1 Processor -- start 2021-02-10T11:40:45.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.616 7f8cf8727700 1 -- start start 2021-02-10T11:40:45.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00f1880 0x7f8cf00a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cf00f20e0 0x7f8cf00aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf01a0250 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00f25e0 con 0x7f8cf00f20e0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00f2800 con 0x7f8cf00f1880 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48126/0 (socket says 172.21.2.117:48126) 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 -- 172.21.2.117:0/2900451383 learned_addr learned my addr 172.21.2.117:0/2900451383 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 -- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00f1880 msgr2=0x7f8cf00a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00f1880 0x7f8cf00a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 -- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cf00f20e0 msgr2=0x7f8cf00aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cf00f20e0 0x7f8cf00aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 -- 172.21.2.117:0/2900451383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00ab320 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf5d42700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f8ce4001270 tx=0x7f8ce400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce40009b0 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce400ad20 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf00ab320 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.617 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce40045d0 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.618 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf00ab320 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.618 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ce4004140 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.618 7f8ce2ffd700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 0x7f8cd801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.618 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f8ce40327e0 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.618 7f8cf6543700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 0x7f8cd801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:45.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.619 7f8cf6543700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 0x7f8cd801ac10 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:45.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.649 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf0109ab0 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.651 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ce4000940 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.777 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cf0109ab0 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.778 7f8ce2ffd700 1 -- 172.21.2.117:0/2900451383 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+1754 (secure 0 0 0) 0x7f8ce400ad20 con 0x7f8cf00f1ce0 2021-02-10T11:40:45.761 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:45.761 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1697702191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1697702191},{"type":"v1","addr":"172.21.2.117:6833","nonce":1697702191}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":22,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:03.881036","modified":"2021-02-10 11:40:45.124126","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2021-02-10T11:40:45.762 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 22 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 msgr2=0x7f8cd801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 0x7f8cd801ac10 crc :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 msgr2=0x7f8cf019fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f8ce4001270 tx=0x7f8ce400a930).stop 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 shutdown_connections 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd8018730 0x7f8cd801ac10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cf00f1ce0 0x7f8cf019fc30 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:45.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cf00f20e0 0x7f8cf00aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:45.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 --2- 172.21.2.117:0/2900451383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cf00f1880 0x7f8cf00a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:45.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 >> 172.21.2.117:0/2900451383 conn(0x7f8cf00bb7a0 msgr2=0x7f8cf00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:45.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 shutdown_connections 2021-02-10T11:40:45.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:45.780 7f8cf8727700 1 -- 172.21.2.117:0/2900451383 wait complete. 2021-02-10T11:40:45.772 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:40:46.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.202 7f8559a32700 1 Processor -- start 2021-02-10T11:40:46.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 -- start start 2021-02-10T11:40:46.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8554079000 0x7f85540e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85541b00d0 0x7f85541b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8554077a70 con 0x7f855407da20 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540e53a0 con 0x7f85541b00d0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8559a32700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8554079400 con 0x7f8554079000 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8553fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85541b00d0 0x7f85541b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8553fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f85541b00d0 0x7f85541b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48136/0 (socket says 172.21.2.117:48136) 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8553fff700 1 -- 172.21.2.117:0/3376622851 learned_addr learned my addr 172.21.2.117:0/3376622851 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.203 7f8552ffd700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8554079000 0x7f85540e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8553fff700 1 -- 172.21.2.117:0/3376622851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8554079000 msgr2=0x7f85540e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8553fff700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8554079000 0x7f85540e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8553fff700 1 -- 172.21.2.117:0/3376622851 >> v1:172.21.2.125:6789/0 conn(0x7f855407da20 legacy=0x7f85540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8553fff700 1 -- 172.21.2.117:0/3376622851 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855407de20 con 0x7f85541b00d0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8553fff700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85541b00d0 0x7f85541b04d0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f854c006f00 tx=0x7f854c009f10).ready entity=mon.0 client_cookie=9e6d17be1eb1c69e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f85527fc700 1 -- 172.21.2.117:0/3376622851 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f854c00b3b0 con 0x7f85541b00d0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f85527fc700 1 -- 172.21.2.117:0/3376622851 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c00b730 con 0x7f85541b00d0 2021-02-10T11:40:46.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f85527fc700 1 -- 172.21.2.117:0/3376622851 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f854c00bb90 con 0x7f85541b00d0 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8559a32700 1 -- 172.21.2.117:0/3376622851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85541b00d0 msgr2=0x7f85541b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.204 7f8559a32700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85541b00d0 0x7f85541b04d0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f854c006f00 tx=0x7f854c009f10).stop 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 -- 172.21.2.117:0/3376622851 shutdown_connections 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85541b00d0 0x7f85541b04d0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 --2- 172.21.2.117:0/3376622851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8554079000 0x7f85540e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 -- 172.21.2.117:0/3376622851 >> 172.21.2.117:0/3376622851 conn(0x7f85540ce430 msgr2=0x7f85540e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 -- 172.21.2.117:0/3376622851 shutdown_connections 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.205 7f8559a32700 1 -- 172.21.2.117:0/3376622851 wait complete. 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 Processor -- start 2021-02-10T11:40:46.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 -- start start 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8554090d90 0x7f85541a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85541a85d0 0x7f85541a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554091190 con 0x7f85540aa7b0 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.219 7f8559a32700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554091380 con 0x7f8554090d90 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f8559a32700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85541a89d0 con 0x7f85541a85d0 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48140/0 (socket says 172.21.2.117:48140) 2021-02-10T11:40:46.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 -- 172.21.2.117:0/1099019829 learned_addr learned my addr 172.21.2.117:0/1099019829 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 -- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85541a85d0 msgr2=0x7f85541a9450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85541a85d0 0x7f85541a9450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 -- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8554090d90 msgr2=0x7f85541a7fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8554090d90 0x7f85541a7fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 -- 172.21.2.117:0/1099019829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85541a8bf0 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f85537fe700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f8544006f60 tx=0x7f854400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f854400bd60 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8544006030 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f85440054f0 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f8559a32700 1 -- 172.21.2.117:0/1099019829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85541a8bf0 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.220 7f8559a32700 1 -- 172.21.2.117:0/1099019829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85541a8bf0 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.221 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8544006210 con 0x7f85540aa7b0 2021-02-10T11:40:46.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.221 7f854bfff700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 0x7f853801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:46.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.221 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f8544032950 con 0x7f85540aa7b0 2021-02-10T11:40:46.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.222 7f8552ffd700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 0x7f853801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:46.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.222 7f8552ffd700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 0x7f853801ac10 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.253 7f8559a32700 1 -- 172.21.2.117:0/1099019829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85541b4f80 con 0x7f85540aa7b0 2021-02-10T11:40:46.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.256 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8544006bb0 con 0x7f85540aa7b0 2021-02-10T11:40:46.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:46.383 7f8559a32700 1 -- 172.21.2.117:0/1099019829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f85541b4f80 con 0x7f85540aa7b0 2021-02-10T11:40:47.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.134 7f854bfff700 1 -- 172.21.2.117:0/1099019829 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8544006030 con 0x7f85540aa7b0 2021-02-10T11:40:47.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 msgr2=0x7f853801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 0x7f853801ac10 crc :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 msgr2=0x7f8554090770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f8544006f60 tx=0x7f854400b970).stop 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 shutdown_connections 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f85541a85d0 0x7f85541a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8554090d90 0x7f85541a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8538018730 0x7f853801ac10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 --2- 172.21.2.117:0/1099019829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f85540aa7b0 0x7f8554090770 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 >> 172.21.2.117:0/1099019829 conn(0x7f8554199470 msgr2=0x7f8554098e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 shutdown_connections 2021-02-10T11:40:47.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.153 7f8559a32700 1 -- 172.21.2.117:0/1099019829 wait complete. 2021-02-10T11:40:47.145 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.575 7fb53df59700 1 Processor -- start 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.575 7fb53df59700 1 -- start start 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb53df59700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb53807da20 0x7fb5380771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb53df59700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538077a10 con 0x7fb5380e5200 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb53df59700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5381b0920 con 0x7fb5380e4e00 2021-02-10T11:40:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb53df59700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380777d0 con 0x7fb53807da20 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5377fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb53807da20 0x7fb5380771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb537fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb5380e5200 0x7fb5380d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48396/0 (socket says 172.21.2.117:48396) 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb537fff700 1 -- 172.21.2.117:0/2901752550 learned_addr learned my addr 172.21.2.117:0/2901752550 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb536ffd700 1 --1- 172.21.2.117:0/2901752550 >> v1:172.21.2.125:6789/0 conn(0x7fb5380e4e00 0x7fb5381b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:41322/0 (socket says 172.21.2.117:41322) 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5367fc700 1 -- 172.21.2.117:0/2901752550 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951762814 0 0) 0x7fb530001500 con 0x7fb5380e5200 2021-02-10T11:40:47.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5367fc700 1 -- 172.21.2.117:0/2901752550 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524002fe0 con 0x7fb5380e5200 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5377fe700 1 -- 172.21.2.117:0/2901752550 >> v1:172.21.2.125:6789/0 conn(0x7fb5380e4e00 legacy=0x7fb5381b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5377fe700 1 -- 172.21.2.117:0/2901752550 >> v1:172.21.2.117:6789/0 conn(0x7fb5380e5200 legacy=0x7fb5380d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.576 7fb5377fe700 1 -- 172.21.2.117:0/2901752550 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380d1d50 con 0x7fb53807da20 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb5377fe700 1 --2- 172.21.2.117:0/2901752550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb53807da20 0x7fb5380771b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fb520006650 tx=0x7fb520009a80).ready entity=mon.0 client_cookie=6fcdca0c9fe2880a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb5367fc700 1 -- 172.21.2.117:0/2901752550 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5200009b0 con 0x7fb53807da20 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb5367fc700 1 -- 172.21.2.117:0/2901752550 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52000c0c0 con 0x7fb53807da20 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb5367fc700 1 -- 172.21.2.117:0/2901752550 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb52000c5e0 con 0x7fb53807da20 2021-02-10T11:40:47.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb53df59700 1 -- 172.21.2.117:0/2901752550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb53807da20 msgr2=0x7fb5380771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.577 7fb53df59700 1 --2- 172.21.2.117:0/2901752550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb53807da20 0x7fb5380771b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7fb520006650 tx=0x7fb520009a80).stop 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.578 7fb53df59700 1 -- 172.21.2.117:0/2901752550 shutdown_connections 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.578 7fb53df59700 1 --2- 172.21.2.117:0/2901752550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb53807da20 0x7fb5380771b0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.578 7fb53df59700 1 -- 172.21.2.117:0/2901752550 >> 172.21.2.117:0/2901752550 conn(0x7fb5380ce480 msgr2=0x7fb5380e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.578 7fb53df59700 1 -- 172.21.2.117:0/2901752550 shutdown_connections 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.578 7fb53df59700 1 -- 172.21.2.117:0/2901752550 wait complete. 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.592 7fb53df59700 1 Processor -- start 2021-02-10T11:40:47.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 -- start start 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5380a48d0 0x7fb5380be660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5380bf6a0 0x7fb5380bc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380be0e0 con 0x7fb5380bec80 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380a4cd0 con 0x7fb5380bf6a0 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb53df59700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380bcfd0 con 0x7fb5380a48d0 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb536ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb536ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48150/0 (socket says 172.21.2.117:48150) 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb536ffd700 1 -- 172.21.2.117:0/2445832336 learned_addr learned my addr 172.21.2.117:0/2445832336 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb5377fe700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5380a48d0 0x7fb5380be660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.593 7fb537fff700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5380bf6a0 0x7fb5380bc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 -- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5380a48d0 msgr2=0x7fb5380be660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5380a48d0 0x7fb5380be660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 -- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5380bf6a0 msgr2=0x7fb5380bc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5380bf6a0 0x7fb5380bc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 -- 172.21.2.117:0/2445832336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380bd1f0 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb536ffd700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7fb528001be0 tx=0x7fb52800a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb52800acf0 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528004140 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb53df59700 1 -- 172.21.2.117:0/2445832336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5380bd1f0 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5280045e0 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.594 7fb53df59700 1 -- 172.21.2.117:0/2445832336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380bd1f0 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.595 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb528004320 con 0x7fb5380bec80 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.595 7fb52ffff700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 0x7fb51c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:47.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.595 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fb528032950 con 0x7fb5380bec80 2021-02-10T11:40:47.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.595 7fb5377fe700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 0x7fb51c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:47.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.595 7fb5377fe700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 0x7fb51c01ac10 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:47.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.625 7fb53df59700 1 -- 172.21.2.117:0/2445832336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538101060 con 0x7fb5380bec80 2021-02-10T11:40:47.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.628 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb528000d80 con 0x7fb5380bec80 2021-02-10T11:40:47.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:47.759 7fb53df59700 1 -- 172.21.2.117:0/2445832336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb538077a10 con 0x7fb5380bec80 2021-02-10T11:40:48.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.137 7fb52ffff700 1 -- 172.21.2.117:0/2445832336 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v41) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb528004320 con 0x7fb5380bec80 2021-02-10T11:40:48.160 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:40:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 msgr2=0x7fb51c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 0x7fb51c01ac10 crc :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 msgr2=0x7fb5380bf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7fb528001be0 tx=0x7fb52800a900).stop 2021-02-10T11:40:48.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 shutdown_connections 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb51c018730 0x7fb51c01ac10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5380bec80 0x7fb5380bf080 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5380bf6a0 0x7fb5380bc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 --2- 172.21.2.117:0/2445832336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5380a48d0 0x7fb5380be660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 >> 172.21.2.117:0/2445832336 conn(0x7fb5380a1540 msgr2=0x7fb5380ab4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 shutdown_connections 2021-02-10T11:40:48.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.178 7fb53df59700 1 -- 172.21.2.117:0/2445832336 wait complete. 2021-02-10T11:40:48.170 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:40:48.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 Processor -- start 2021-02-10T11:40:48.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 -- start start 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0ba0078fa0 0x7f0ba00e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0077a10 con 0x7f0ba01b0130 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00e5420 con 0x7f0ba007da20 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0ba4b27700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ba00d1d50 con 0x7f0ba0078fa0 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0b9f05f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0ba01b0130 0x7f0ba00d0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48412/0 (socket says 172.21.2.117:48412) 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.601 7f0b9f05f700 1 -- 172.21.2.117:0/3033596055 learned_addr learned my addr 172.21.2.117:0/3033596055 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838579642 0 0) 0x7f0b98001500 con 0x7f0ba01b0130 2021-02-10T11:40:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e05d700 1 --2- 172.21.2.117:0/3033596055 >> v2:172.21.2.125:3300/0 conn(0x7f0ba0078fa0 0x7f0ba00e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c002fe0 con 0x7f0ba01b0130 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4169935316 0 0) 0x7f0b98001460 con 0x7f0ba01b0130 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 >> v2:172.21.2.125:3300/0 conn(0x7f0ba0078fa0 msgr2=0x7f0ba00e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 --2- 172.21.2.117:0/3033596055 >> v2:172.21.2.125:3300/0 conn(0x7f0ba0078fa0 0x7f0ba00e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 >> v1:172.21.2.125:6789/0 conn(0x7f0ba007da20 legacy=0x7f0ba00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00793a0 con 0x7f0ba01b0130 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0b98001980 con 0x7f0ba01b0130 2021-02-10T11:40:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 >> v1:172.21.2.117:6789/0 conn(0x7f0ba01b0130 legacy=0x7f0ba00d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 0x7f0b8c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b8c006780 0x7f0b8c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b8c0095a0 0x7f0b8c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c00c310 con 0x7f0b8c003a70 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c00c530 con 0x7f0b8c0095a0 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c00c750 con 0x7f0b8c006780 2021-02-10T11:40:48.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 0x7f0b8c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 -- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b8c006780 msgr2=0x7f0b8c008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b8c006780 0x7f0b8c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 -- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b8c0095a0 msgr2=0x7f0b8c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b8c0095a0 0x7f0b8c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9f05f700 1 -- 172.21.2.117:0/3033596055 reap_dead start 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.602 7f0b9e85e700 1 -- 172.21.2.117:0/3033596055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0509c0 con 0x7f0b8c003a70 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0b9e85e700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 0x7f0b8c005f00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f0b88006ab0 tx=0x7f0b88009a90).ready entity=mon.0 client_cookie=c145fb7fadf04818 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b880066a0 con 0x7f0b8c003a70 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b880022e0 con 0x7f0b8c003a70 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0b9d85c700 1 -- 172.21.2.117:0/3033596055 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b88002800 con 0x7f0b8c003a70 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0ba4b27700 1 -- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 msgr2=0x7f0b8c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.603 7f0ba4b27700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 0x7f0b8c005f00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f0b88006ab0 tx=0x7f0b88009a90).stop 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.604 7f0ba4b27700 1 -- 172.21.2.117:0/3033596055 shutdown_connections 2021-02-10T11:40:48.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.604 7f0ba4b27700 1 --2- 172.21.2.117:0/3033596055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b8c003a70 0x7f0b8c005f00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.604 7f0ba4b27700 1 -- 172.21.2.117:0/3033596055 >> 172.21.2.117:0/3033596055 conn(0x7f0ba00ce480 msgr2=0x7f0ba00e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.604 7f0ba4b27700 1 -- 172.21.2.117:0/3033596055 shutdown_connections 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.604 7f0ba4b27700 1 -- 172.21.2.117:0/3033596055 wait complete. 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.618 7f0ba4b27700 1 Processor -- start 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.618 7f0ba4b27700 1 -- start start 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ba01a4650 0x7f0ba009a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 0x7f0ba00ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00ab690 con 0x7f0ba00a0410 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba009a8e0 con 0x7f0ba01a4650 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0ba4b27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba00a0e30 con 0x7f0ba00aac70 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48174/0 (socket says 172.21.2.117:48174) 2021-02-10T11:40:48.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 -- 172.21.2.117:0/2129881843 learned_addr learned my addr 172.21.2.117:0/2129881843 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:48.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9e85e700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ba01a4650 0x7f0ba009a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9e05d700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 0x7f0ba00ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 -- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 msgr2=0x7f0ba00ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 0x7f0ba00ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 -- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ba01a4650 msgr2=0x7f0ba009a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ba01a4650 0x7f0ba009a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9f05f700 1 -- 172.21.2.117:0/2129881843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba00a97e0 con 0x7f0ba00a0410 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.619 7f0b9e05d700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 0x7f0ba00ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0b9f05f700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f0b9800a8d0 tx=0x7f0b9800ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b980070d0 con 0x7f0ba00a0410 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b98007410 con 0x7f0ba00a0410 2021-02-10T11:40:48.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba00a97e0 con 0x7f0ba00a0410 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba00a9d70 con 0x7f0ba00a0410 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b98004bb0 con 0x7f0ba00a0410 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.620 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b98004720 con 0x7f0ba00a0410 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.621 7f0b977fe700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 0x7f0b8401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.621 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f0b98032b70 con 0x7f0ba00a0410 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.621 7f0b9e85e700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 0x7f0b8401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:48.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.621 7f0b9e85e700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 0x7f0b8401ac10 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:48.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.651 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba01051c0 con 0x7f0ba00a0410 2021-02-10T11:40:48.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.653 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b980073f0 con 0x7f0ba00a0410 2021-02-10T11:40:48.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:48.782 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0ba0077a10 con 0x7f0ba00a0410 2021-02-10T11:40:49.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.143 7f0b977fe700 1 -- 172.21.2.117:0/2129881843 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v42) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0b98004720 con 0x7f0ba00a0410 2021-02-10T11:40:49.134 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.151 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 msgr2=0x7f0b8401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.151 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 0x7f0b8401ac10 crc :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.151 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 msgr2=0x7f0ba00a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.151 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f0b9800a8d0 tx=0x7f0b9800ac70).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 shutdown_connections 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b84018730 0x7f0b8401ac10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ba00a0410 0x7f0ba00a0810 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ba00aac70 0x7f0ba00ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 --2- 172.21.2.117:0/2129881843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ba01a4650 0x7f0ba009a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 >> 172.21.2.117:0/2129881843 conn(0x7f0ba009f4c0 msgr2=0x7f0ba0090810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 shutdown_connections 2021-02-10T11:40:49.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.152 7f0ba4b27700 1 -- 172.21.2.117:0/2129881843 wait complete. 2021-02-10T11:40:49.144 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:40:49.145 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:40:49.145 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:49.145 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:40:49.145 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:40:49.145 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:40:49.145 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:40:49.151 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:40:49.151 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:40:49.152 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:40:49.152 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:40:49.193 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:40:49.193 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:49.194 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:40:49.194 DEBUG:teuthology.parallel:result is None 2021-02-10T11:40:49.194 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:40:49.598 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:40:49.601 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 Processor -- start 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 -- start start 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c8807da20 0x7f4c880771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4c880e5180 0x7f4c88079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c88077a70 con 0x7f4c880e4d80 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880e5c00 con 0x7f4c8807da20 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c8cec2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880777d0 con 0x7f4c880e5180 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c86b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c8807da20 0x7f4c880771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c86b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c8807da20 0x7f4c880771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48190/0 (socket says 172.21.2.117:48190) 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c86312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4c880e4d80 0x7f4c881b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48438/0 (socket says 172.21.2.117:48438) 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c86b13700 1 -- 172.21.2.117:0/3384984236 learned_addr learned my addr 172.21.2.117:0/3384984236 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.623 7f4c87314700 1 --2- 172.21.2.117:0/3384984236 >> v2:172.21.2.125:3300/0 conn(0x7f4c880e5180 0x7f4c88079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c86b13700 1 -- 172.21.2.117:0/3384984236 >> v2:172.21.2.125:3300/0 conn(0x7f4c880e5180 msgr2=0x7f4c88079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c86b13700 1 --2- 172.21.2.117:0/3384984236 >> v2:172.21.2.125:3300/0 conn(0x7f4c880e5180 0x7f4c88079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c86b13700 1 -- 172.21.2.117:0/3384984236 >> v1:172.21.2.117:6789/0 conn(0x7f4c880e4d80 legacy=0x7f4c881b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c86b13700 1 -- 172.21.2.117:0/3384984236 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c881b08c0 con 0x7f4c8807da20 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c86b13700 1 --2- 172.21.2.117:0/3384984236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c8807da20 0x7f4c880771b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f4c7c006700 tx=0x7f4c7c009b00).ready entity=mon.0 client_cookie=492f2d444bce3352 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c85b11700 1 -- 172.21.2.117:0/3384984236 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c7c00c0c0 con 0x7f4c8807da20 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c85b11700 1 -- 172.21.2.117:0/3384984236 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c00c340 con 0x7f4c8807da20 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c85b11700 1 -- 172.21.2.117:0/3384984236 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c7c00c820 con 0x7f4c8807da20 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 -- 172.21.2.117:0/3384984236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c8807da20 msgr2=0x7f4c880771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 --2- 172.21.2.117:0/3384984236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c8807da20 0x7f4c880771b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f4c7c006700 tx=0x7f4c7c009b00).stop 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 -- 172.21.2.117:0/3384984236 shutdown_connections 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 --2- 172.21.2.117:0/3384984236 >> v2:172.21.2.125:3300/0 conn(0x7f4c880e5180 0x7f4c88079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 --2- 172.21.2.117:0/3384984236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c8807da20 0x7f4c880771b0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.624 7f4c8cec2700 1 -- 172.21.2.117:0/3384984236 >> 172.21.2.117:0/3384984236 conn(0x7f4c880ce430 msgr2=0x7f4c880e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.625 7f4c8cec2700 1 -- 172.21.2.117:0/3384984236 shutdown_connections 2021-02-10T11:40:49.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.625 7f4c8cec2700 1 -- 172.21.2.117:0/3384984236 wait complete. 2021-02-10T11:40:49.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 Processor -- start 2021-02-10T11:40:49.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 -- start start 2021-02-10T11:40:49.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c881b7240 0x7f4c881b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c881b7c60 0x7f4c88196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c881964d0 con 0x7f4c881b66c0 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88197320 con 0x7f4c881b7c60 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.639 7f4c8cec2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c88197540 con 0x7f4c881b7240 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48194/0 (socket says 172.21.2.117:48194) 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 -- 172.21.2.117:0/2303021 learned_addr learned my addr 172.21.2.117:0/2303021 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86312700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c881b7240 0x7f4c881b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c87314700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c881b7c60 0x7f4c88196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 -- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c881b7240 msgr2=0x7f4c881b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c881b7240 0x7f4c881b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 -- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c881b7c60 msgr2=0x7f4c88196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c881b7c60 0x7f4c88196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 -- 172.21.2.117:0/2303021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c88197760 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c86b13700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f4c7c006f20 tx=0x7f4c7c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c7c000c90 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c009100 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c88197760 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.640 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c88197760 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.641 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c7c002220 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.641 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c7c007000 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.641 7f4c777fe700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 0x7f4c6c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.641 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f4c7c032490 con 0x7f4c881b66c0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.641 7f4c86312700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 0x7f4c6c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:49.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.642 7f4c86312700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 0x7f4c6c01ac10 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:49.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.672 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c881964d0 con 0x7f4c881b66c0 2021-02-10T11:40:49.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.674 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c7c000c90 con 0x7f4c881b66c0 2021-02-10T11:40:49.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:49.813 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f4c881964d0 con 0x7f4c881b66c0 2021-02-10T11:40:50.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.146 7f4c777fe700 1 -- 172.21.2.117:0/2303021 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4c7c007f00 con 0x7f4c881b66c0 2021-02-10T11:40:50.163 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:40:50.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 msgr2=0x7f4c6c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 0x7f4c6c01ac10 crc :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 msgr2=0x7f4c881b6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f4c7c006f20 tx=0x7f4c7c00b990).stop 2021-02-10T11:40:50.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 shutdown_connections 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c881b7c60 0x7f4c88196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c881b7240 0x7f4c881b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c6c018730 0x7f4c6c01ac10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 --2- 172.21.2.117:0/2303021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c881b66c0 0x7f4c881b6ac0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 >> 172.21.2.117:0/2303021 conn(0x7f4c881a9360 msgr2=0x7f4c8809a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 shutdown_connections 2021-02-10T11:40:50.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.181 7f4c8cec2700 1 -- 172.21.2.117:0/2303021 wait complete. 2021-02-10T11:40:50.174 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:40:50.174 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.600 7f0a79d7c700 1 Processor -- start 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.600 7f0a79d7c700 1 -- start start 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0a7407da20 0x7f0a740771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0a740e4d90 0x7f0a741b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a74077a70 con 0x7f0a740e5190 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a740777d0 con 0x7f0a740e4d90 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7407de20 con 0x7f0a7407da20 2021-02-10T11:40:50.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0a740e4d90 0x7f0a741b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0a740e4d90 0x7f0a741b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48202/0 (socket says 172.21.2.117:48202) 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 -- 172.21.2.117:0/3316179985 learned_addr learned my addr 172.21.2.117:0/3316179985 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a737fe700 1 --2- 172.21.2.117:0/3316179985 >> v2:172.21.2.125:3300/0 conn(0x7f0a7407da20 0x7f0a740771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 -- 172.21.2.117:0/3316179985 >> v2:172.21.2.125:3300/0 conn(0x7f0a7407da20 msgr2=0x7f0a740771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 --2- 172.21.2.117:0/3316179985 >> v2:172.21.2.125:3300/0 conn(0x7f0a7407da20 0x7f0a740771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 -- 172.21.2.117:0/3316179985 >> v1:172.21.2.125:6789/0 conn(0x7f0a740e5190 legacy=0x7f0a74077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 -- 172.21.2.117:0/3316179985 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a741b0700 con 0x7f0a740e4d90 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a72ffd700 1 --2- 172.21.2.117:0/3316179985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a740e4d90 0x7f0a741b00e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f0a68006700 tx=0x7f0a68009b80).ready entity=mon.0 client_cookie=c471932e67286f49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a727fc700 1 -- 172.21.2.117:0/3316179985 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6800c0c0 con 0x7f0a740e4d90 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a727fc700 1 -- 172.21.2.117:0/3316179985 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800c340 con 0x7f0a740e4d90 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a727fc700 1 -- 172.21.2.117:0/3316179985 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6800c820 con 0x7f0a740e4d90 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 -- 172.21.2.117:0/3316179985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a740e4d90 msgr2=0x7f0a741b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.601 7f0a79d7c700 1 --2- 172.21.2.117:0/3316179985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a740e4d90 0x7f0a741b00e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7f0a68006700 tx=0x7f0a68009b80).stop 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 -- 172.21.2.117:0/3316179985 shutdown_connections 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 --2- 172.21.2.117:0/3316179985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a740e4d90 0x7f0a741b00e0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 --2- 172.21.2.117:0/3316179985 >> v2:172.21.2.125:3300/0 conn(0x7f0a7407da20 0x7f0a740771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 -- 172.21.2.117:0/3316179985 >> 172.21.2.117:0/3316179985 conn(0x7f0a740ce440 msgr2=0x7f0a740e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 -- 172.21.2.117:0/3316179985 shutdown_connections 2021-02-10T11:40:50.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.602 7f0a79d7c700 1 -- 172.21.2.117:0/3316179985 wait complete. 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 Processor -- start 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 -- start start 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7408c080 0x7f0a7408c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7408caa0 0x7f0a7408f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7409d8a0 con 0x7f0a7409dd40 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740b8bc0 con 0x7f0a7408c080 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.617 7f0a79d7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740b8de0 con 0x7f0a7408caa0 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48204/0 (socket says 172.21.2.117:48204) 2021-02-10T11:40:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 -- 172.21.2.117:0/2801272785 learned_addr learned my addr 172.21.2.117:0/2801272785 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:50.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 -- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7408caa0 msgr2=0x7f0a7408f5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:50.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7408caa0 0x7f0a7408f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 -- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7408c080 msgr2=0x7f0a7408c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7408c080 0x7f0a7408c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 -- 172.21.2.117:0/2801272785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7408cea0 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a737fe700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f0a64006f00 tx=0x7f0a6400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a6400bd30 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a64002030 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a64002590 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7408cea0 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.618 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7408fed0 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.619 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0a64000c90 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.619 7f0a5bfff700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 0x7f0a5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.619 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f0a640323f0 con 0x7f0a7409dd40 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.620 7f0a72ffd700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 0x7f0a5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:50.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.620 7f0a72ffd700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 0x7f0a5401abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:50.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.650 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a74077a70 con 0x7f0a7409dd40 2021-02-10T11:40:50.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.653 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0a640009b0 con 0x7f0a7409dd40 2021-02-10T11:40:50.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.785 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0a74077a70 con 0x7f0a7409dd40 2021-02-10T11:40:50.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.785 7f0a5bfff700 1 -- 172.21.2.117:0/2801272785 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f0a64000c90 con 0x7f0a7409dd40 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.770 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.771 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.772 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.773 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.774 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.775 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:40:50.776 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:40:50.777 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.777 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.777 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:40:50.778 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:40:50.779 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:40:50.780 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.781 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:40:50.782 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:40:50.783 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:40:50.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:40:50.786 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:40:50.786 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.786 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:50.786 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:40:50.786 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:40:50.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 msgr2=0x7f0a5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 0x7f0a5401abc0 crc :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 msgr2=0x7f0a740b85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f0a64006f00 tx=0x7f0a6400b940).stop 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 shutdown_connections 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7408caa0 0x7f0a7408f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7408c080 0x7f0a7408c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a540186c0 0x7f0a5401abc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 --2- 172.21.2.117:0/2801272785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a7409dd40 0x7f0a740b85a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 >> 172.21.2.117:0/2801272785 conn(0x7f0a7419f4c0 msgr2=0x7f0a74094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 shutdown_connections 2021-02-10T11:40:50.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:50.787 7f0a79d7c700 1 -- 172.21.2.117:0/2801272785 wait complete. 2021-02-10T11:40:50.789 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:51.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.215 7f241b521700 1 Processor -- start 2021-02-10T11:40:51.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.215 7f241b521700 1 -- start start 2021-02-10T11:40:51.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241b521700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f241407da20 0x7f24140771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241b521700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 0x7f2414079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241b521700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2414077a70 con 0x7f24140e4d90 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241b521700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140777d0 con 0x7f241407da20 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241b521700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24141b08d0 con 0x7f24140e5190 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f2418b3c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f24140e4d90 0x7f24141b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48462/0 (socket says 172.21.2.117:48462) 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f2418b3c700 1 -- 172.21.2.117:0/2191064971 learned_addr learned my addr 172.21.2.117:0/2191064971 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241933d700 1 --2- 172.21.2.117:0/2191064971 >> v2:172.21.2.125:3300/0 conn(0x7f241407da20 0x7f24140771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f2419b3e700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 0x7f2414079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241933d700 1 -- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 msgr2=0x7f2414079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241933d700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 0x7f2414079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241933d700 1 -- 172.21.2.117:0/2191064971 >> v1:172.21.2.117:6789/0 conn(0x7f24140e4d90 legacy=0x7f24141b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f2407fff700 1 -- 172.21.2.117:0/2191064971 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615539383 0 0) 0x7f24100010b0 con 0x7f24140e4d90 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f241933d700 1 -- 172.21.2.117:0/2191064971 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241407de20 con 0x7f241407da20 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.216 7f2419b3e700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 0x7f2414079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f241933d700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241407da20 0x7f24140771b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2408006650 tx=0x7f2408009a80).ready entity=mon.1 client_cookie=d4d3fb1cfec9d82e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f2407fff700 1 -- 172.21.2.117:0/2191064971 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f240800c0c0 con 0x7f241407da20 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f2407fff700 1 -- 172.21.2.117:0/2191064971 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240800c340 con 0x7f241407da20 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f2407fff700 1 -- 172.21.2.117:0/2191064971 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f240800c860 con 0x7f241407da20 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f241b521700 1 -- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241407da20 msgr2=0x7f24140771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.217 7f241b521700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241407da20 0x7f24140771b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2408006650 tx=0x7f2408009a80).stop 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 -- 172.21.2.117:0/2191064971 shutdown_connections 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24140e5190 0x7f2414079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 --2- 172.21.2.117:0/2191064971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241407da20 0x7f24140771b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 -- 172.21.2.117:0/2191064971 >> 172.21.2.117:0/2191064971 conn(0x7f24140ce440 msgr2=0x7f24140e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 -- 172.21.2.117:0/2191064971 shutdown_connections 2021-02-10T11:40:51.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.218 7f241b521700 1 -- 172.21.2.117:0/2191064971 wait complete. 2021-02-10T11:40:51.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.232 7f241b521700 1 Processor -- start 2021-02-10T11:40:51.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.232 7f241b521700 1 -- start start 2021-02-10T11:40:51.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 0x7f241419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 0x7f241408d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241408c080 con 0x7f241419dfe0 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241408c2a0 con 0x7f241408d410 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241b521700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241408c4c0 con 0x7f241408cd50 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48218/0 (socket says 172.21.2.117:48218) 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 -- 172.21.2.117:0/2257107523 learned_addr learned my addr 172.21.2.117:0/2257107523 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2419b3e700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 0x7f241408d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241933d700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 0x7f241419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 -- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 msgr2=0x7f241419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 0x7f241419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 -- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 msgr2=0x7f241408d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 0x7f241408d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 -- 172.21.2.117:0/2257107523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241408c6e0 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f241933d700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 0x7f241419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2419b3e700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 0x7f241408d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.233 7f2418b3c700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f2410001c60 tx=0x7f241000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241000acc0 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2410004140 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2410004660 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f241b521700 1 -- 172.21.2.117:0/2257107523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241408c6e0 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f241b521700 1 -- 172.21.2.117:0/2257107523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241408c6e0 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24100010b0 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 0x7f23f401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.234 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2410032590 con 0x7f241419dfe0 2021-02-10T11:40:51.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.235 7f241933d700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 0x7f23f401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.235 7f241933d700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 0x7f23f401abd0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.266 7f241b521700 1 -- 172.21.2.117:0/2257107523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2414077a70 con 0x7f241419dfe0 2021-02-10T11:40:51.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.269 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24100009b0 con 0x7f241419dfe0 2021-02-10T11:40:51.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.391 7f241b521700 1 -- 172.21.2.117:0/2257107523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2414077a70 con 0x7f241419dfe0 2021-02-10T11:40:51.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.392 7f2405ffb700 1 -- 172.21.2.117:0/2257107523 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+11377 (secure 0 0 0) 0x7f24100010b0 con 0x7f241419dfe0 2021-02-10T11:40:51.376 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:51.376 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":43,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:50.143332","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:51.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.393 7f241b521700 1 -- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 msgr2=0x7f23f401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.393 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 0x7f23f401abd0 crc :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.393 7f241b521700 1 -- 172.21.2.117:0/2257107523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 msgr2=0x7f241419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.393 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f2410001c60 tx=0x7f241000a8d0).stop 2021-02-10T11:40:51.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 -- 172.21.2.117:0/2257107523 shutdown_connections 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23f40186d0 0x7f23f401abd0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f241419dfe0 0x7f241419e3e0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f241408d410 0x7f241408d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 --2- 172.21.2.117:0/2257107523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f241408cd50 0x7f241419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 -- 172.21.2.117:0/2257107523 >> 172.21.2.117:0/2257107523 conn(0x7f241419f4c0 msgr2=0x7f2414094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 -- 172.21.2.117:0/2257107523 shutdown_connections 2021-02-10T11:40:51.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.394 7f241b521700 1 -- 172.21.2.117:0/2257107523 wait complete. 2021-02-10T11:40:51.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:40:51.387 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.816 7f4749030700 1 Processor -- start 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.816 7f4749030700 1 -- start start 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.816 7f4749030700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f474407da20 0x7f47440771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4749030700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47440e51a0 0x7f4744079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4749030700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4744077a70 con 0x7f47440e4da0 2021-02-10T11:40:51.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4749030700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47440777d0 con 0x7f474407da20 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4749030700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47441b08e0 con 0x7f47440e51a0 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f474407da20 0x7f47440771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f474407da20 0x7f47440771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48224/0 (socket says 172.21.2.117:48224) 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f474259c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f47440e4da0 0x7f47441b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48474/0 (socket says 172.21.2.117:48474) 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 -- 172.21.2.117:0/851924789 learned_addr learned my addr 172.21.2.117:0/851924789 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4741d9b700 1 -- 172.21.2.117:0/851924789 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572358865 0 0) 0x7f4734000ce0 con 0x7f47440e4da0 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4741d9b700 1 -- 172.21.2.117:0/851924789 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4730002fe0 con 0x7f47440e4da0 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f474359e700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47440e51a0 0x7f4744079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 -- 172.21.2.117:0/851924789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47440e51a0 msgr2=0x7f4744079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47440e51a0 0x7f4744079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 -- 172.21.2.117:0/851924789 >> v1:172.21.2.117:6789/0 conn(0x7f47440e4da0 legacy=0x7f47441b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.817 7f4742d9d700 1 -- 172.21.2.117:0/851924789 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474407de20 con 0x7f474407da20 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4742d9d700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f474407da20 0x7f47440771b0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f472c006700 tx=0x7f472c009b80).ready entity=mon.0 client_cookie=1eb18476b6199089 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4741d9b700 1 -- 172.21.2.117:0/851924789 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f472c00c0c0 con 0x7f474407da20 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4741d9b700 1 -- 172.21.2.117:0/851924789 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c00c340 con 0x7f474407da20 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4741d9b700 1 -- 172.21.2.117:0/851924789 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f472c00c860 con 0x7f474407da20 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4749030700 1 -- 172.21.2.117:0/851924789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f474407da20 msgr2=0x7f47440771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.818 7f4749030700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f474407da20 0x7f47440771b0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f472c006700 tx=0x7f472c009b80).stop 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 -- 172.21.2.117:0/851924789 shutdown_connections 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47440e51a0 0x7f4744079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 --2- 172.21.2.117:0/851924789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f474407da20 0x7f47440771b0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 -- 172.21.2.117:0/851924789 >> 172.21.2.117:0/851924789 conn(0x7f47440ce450 msgr2=0x7f47440e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 -- 172.21.2.117:0/851924789 shutdown_connections 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.819 7f4749030700 1 -- 172.21.2.117:0/851924789 wait complete. 2021-02-10T11:40:51.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 Processor -- start 2021-02-10T11:40:51.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 -- start start 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47441b5130 0x7f47441b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47441b5f50 0x7f47440b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440b8da0 con 0x7f47441b5b50 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47440a4140 con 0x7f47441b5130 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f4749030700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47440a4320 con 0x7f47441b5f50 2021-02-10T11:40:51.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f474259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f474259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48232/0 (socket says 172.21.2.117:48232) 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.834 7f474259c700 1 -- 172.21.2.117:0/3984465884 learned_addr learned my addr 172.21.2.117:0/3984465884 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 -- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47441b5f50 msgr2=0x7f47440b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47441b5f50 0x7f47440b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 -- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47441b5130 msgr2=0x7f47441b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47441b5130 0x7f47441b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 -- 172.21.2.117:0/3984465884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47440b62b0 con 0x7f47441b5b50 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f474259c700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f473400b5c0 tx=0x7f473400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4734006020 con 0x7f47441b5b50 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734006260 con 0x7f47441b5b50 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4734004620 con 0x7f47441b5b50 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f4749030700 1 -- 172.21.2.117:0/3984465884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47440b62b0 con 0x7f47441b5b50 2021-02-10T11:40:51.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.835 7f4749030700 1 -- 172.21.2.117:0/3984465884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47440b62b0 con 0x7f47441b5b50 2021-02-10T11:40:51.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.836 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47340043a0 con 0x7f47441b5b50 2021-02-10T11:40:51.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.836 7f473b7fe700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 0x7f472801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:51.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.836 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f47340323d0 con 0x7f47441b5b50 2021-02-10T11:40:51.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.836 7f4742d9d700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 0x7f472801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:51.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.837 7f4742d9d700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 0x7f472801abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:51.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.867 7f4749030700 1 -- 172.21.2.117:0/3984465884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47440a4320 con 0x7f47441b5b50 2021-02-10T11:40:51.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.869 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4734006b80 con 0x7f47441b5b50 2021-02-10T11:40:51.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.996 7f4749030700 1 -- 172.21.2.117:0/3984465884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f47440a4320 con 0x7f47441b5b50 2021-02-10T11:40:51.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.996 7f473b7fe700 1 -- 172.21.2.117:0/3984465884 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7f47340043a0 con 0x7f47441b5b50 2021-02-10T11:40:51.981 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.998 7f4749030700 1 -- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 msgr2=0x7f472801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.998 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 0x7f472801abc0 crc :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.998 7f4749030700 1 -- 172.21.2.117:0/3984465884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 msgr2=0x7f47440b5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.998 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7f473400b5c0 tx=0x7f473400b960).stop 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 -- 172.21.2.117:0/3984465884 shutdown_connections 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47280186c0 0x7f472801abc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47441b5b50 0x7f47440b5670 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47441b5f50 0x7f47440b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:51.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 --2- 172.21.2.117:0/3984465884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47441b5130 0x7f47441b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:51.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 -- 172.21.2.117:0/3984465884 >> 172.21.2.117:0/3984465884 conn(0x7f474408bbc0 msgr2=0x7f47440c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:51.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 -- 172.21.2.117:0/3984465884 shutdown_connections 2021-02-10T11:40:51.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:51.999 7f4749030700 1 -- 172.21.2.117:0/3984465884 wait complete. 2021-02-10T11:40:51.992 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:40:52.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 Processor -- start 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 -- start start 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c07da20 0x7f2f0c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2f0c0e4d90 0x7f2f0c1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f0c0e5190 0x7f2f0c077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c0e5c10 con 0x7f2f0c0e5190 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0c077a70 con 0x7f2f0c0e4d90 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.416 7f2f1467c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c0777d0 con 0x7f2f0c07da20 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f0c0e5190 0x7f2f0c077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f0c0e5190 0x7f2f0c077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48238/0 (socket says 172.21.2.117:48238) 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 -- 172.21.2.117:0/2452323994 learned_addr learned my addr 172.21.2.117:0/2452323994 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 -- 172.21.2.117:0/2452323994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c07da20 msgr2=0x7f2f0c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:52.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 --2- 172.21.2.117:0/2452323994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c07da20 0x7f2f0c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 -- 172.21.2.117:0/2452323994 >> v2:172.21.2.125:3300/0 conn(0x7f2f0c0e4d90 msgr2=0x7f2f0c1b00e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 --2- 172.21.2.117:0/2452323994 >> v2:172.21.2.125:3300/0 conn(0x7f2f0c0e4d90 0x7f2f0c1b00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 -- 172.21.2.117:0/2452323994 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c07de20 con 0x7f2f0c0e5190 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f12c99700 1 --2- 172.21.2.117:0/2452323994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0e5190 0x7f2f0c077ff0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f2f00006b80 tx=0x7f2f00009b90).ready entity=mon.0 client_cookie=fe27243c7edf45d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f11496700 1 -- 172.21.2.117:0/2452323994 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f000022e0 con 0x7f2f0c0e5190 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f11496700 1 -- 172.21.2.117:0/2452323994 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f00002560 con 0x7f2f0c0e5190 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f11496700 1 -- 172.21.2.117:0/2452323994 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f00002a40 con 0x7f2f0c0e5190 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f1467c700 1 -- 172.21.2.117:0/2452323994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0e5190 msgr2=0x7f2f0c077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.417 7f2f1467c700 1 --2- 172.21.2.117:0/2452323994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0e5190 0x7f2f0c077ff0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f2f00006b80 tx=0x7f2f00009b90).stop 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 -- 172.21.2.117:0/2452323994 shutdown_connections 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 --2- 172.21.2.117:0/2452323994 >> v2:172.21.2.125:3300/0 conn(0x7f2f0c0e4d90 0x7f2f0c1b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 --2- 172.21.2.117:0/2452323994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0e5190 0x7f2f0c077ff0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:52.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 --2- 172.21.2.117:0/2452323994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c07da20 0x7f2f0c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 -- 172.21.2.117:0/2452323994 >> 172.21.2.117:0/2452323994 conn(0x7f2f0c0ce440 msgr2=0x7f2f0c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:52.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 -- 172.21.2.117:0/2452323994 shutdown_connections 2021-02-10T11:40:52.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.418 7f2f1467c700 1 -- 172.21.2.117:0/2452323994 wait complete. 2021-02-10T11:40:52.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.432 7f2f1467c700 1 Processor -- start 2021-02-10T11:40:52.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 -- start start 2021-02-10T11:40:52.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f0c197780 0x7f2f0c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c1981a0 0x7f2f0c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c198ef0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c198bc0 con 0x7f2f0c197780 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f1467c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c1b92e0 con 0x7f2f0c1981a0 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f12498700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f12498700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48242/0 (socket says 172.21.2.117:48242) 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f12498700 1 -- 172.21.2.117:0/647084482 learned_addr learned my addr 172.21.2.117:0/647084482 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f12498700 1 -- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c1981a0 msgr2=0x7f2f0c1985a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.433 7f2f12498700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c1981a0 0x7f2f0c1985a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f12498700 1 -- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f0c197780 msgr2=0x7f2f0c197b80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f12498700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f0c197780 0x7f2f0c197b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f12498700 1 -- 172.21.2.117:0/647084482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c1993f0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f12498700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f2f0800b5c0 tx=0x7f2f0800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f0800bd50 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08006030 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f080054d0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f1467c700 1 -- 172.21.2.117:0/647084482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c1993f0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.434 7f2f1467c700 1 -- 172.21.2.117:0/647084482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c1993f0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.435 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f08006210 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.435 7f2efeffd700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 0x7f2ef401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.435 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2f08032380 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.435 7f2f11c97700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 0x7f2ef401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:52.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.436 7f2f11c97700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 0x7f2ef401abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:52.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.466 7f2f1467c700 1 -- 172.21.2.117:0/647084482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0c198bc0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.469 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f08006b80 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.591 7f2f1467c700 1 -- 172.21.2.117:0/647084482 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2f0c1b92e0 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.591 7f2efeffd700 1 -- 172.21.2.117:0/647084482 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+64 (secure 0 0 0) 0x7f2f0800bd50 con 0x7f2f0c0b1d00 2021-02-10T11:40:52.576 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:40:52.577 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:40:52.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.593 7f2f1467c700 1 -- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 msgr2=0x7f2ef401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:52.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.593 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 0x7f2ef401abc0 crc :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:52.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.593 7f2f1467c700 1 -- 172.21.2.117:0/647084482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 msgr2=0x7f2f0c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:52.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.593 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f2f0800b5c0 tx=0x7f2f0800b960).stop 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 -- 172.21.2.117:0/647084482 shutdown_connections 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f0c1981a0 0x7f2f0c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f0c197780 0x7f2f0c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ef40186c0 0x7f2ef401abc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 --2- 172.21.2.117:0/647084482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f0c0b1d00 0x7f2f0c0b2100 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 -- 172.21.2.117:0/647084482 >> 172.21.2.117:0/647084482 conn(0x7f2f0c18cbd0 msgr2=0x7f2f0c18bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 -- 172.21.2.117:0/647084482 shutdown_connections 2021-02-10T11:40:52.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:52.594 7f2f1467c700 1 -- 172.21.2.117:0/647084482 wait complete. 2021-02-10T11:40:52.587 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:40:52.587 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 Processor -- start 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 -- start start 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe9a407da20 0x7fe9a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9a40e4da0 0x7fe9a41b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40e51a0 0x7fe9a4077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a4077a70 con 0x7fe9a40e4da0 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a40777d0 con 0x7fe9a407da20 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a8af4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a407de20 con 0x7fe9a40e51a0 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9a40e4da0 0x7fe9a41b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:53.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe9a40e4da0 0x7fe9a41b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48250/0 (socket says 172.21.2.117:48250) 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.016 7fe9a205d700 1 -- 172.21.2.117:0/1036222135 learned_addr learned my addr 172.21.2.117:0/1036222135 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 -- 172.21.2.117:0/1036222135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40e51a0 msgr2=0x7fe9a4077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 --2- 172.21.2.117:0/1036222135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40e51a0 0x7fe9a4077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 -- 172.21.2.117:0/1036222135 >> v2:172.21.2.125:3300/0 conn(0x7fe9a407da20 msgr2=0x7fe9a40771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 --2- 172.21.2.117:0/1036222135 >> v2:172.21.2.125:3300/0 conn(0x7fe9a407da20 0x7fe9a40771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 -- 172.21.2.117:0/1036222135 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a41b0710 con 0x7fe9a40e4da0 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a205d700 1 --2- 172.21.2.117:0/1036222135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40e4da0 0x7fe9a41b00f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7fe98c006700 tx=0x7fe98c009b80).ready entity=mon.0 client_cookie=f17b38d15bb56ae5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a185c700 1 -- 172.21.2.117:0/1036222135 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe98c00c0c0 con 0x7fe9a40e4da0 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a185c700 1 -- 172.21.2.117:0/1036222135 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c00c340 con 0x7fe9a40e4da0 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a185c700 1 -- 172.21.2.117:0/1036222135 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe98c00c820 con 0x7fe9a40e4da0 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a8af4700 1 -- 172.21.2.117:0/1036222135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40e4da0 msgr2=0x7fe9a41b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.018 7fe9a8af4700 1 --2- 172.21.2.117:0/1036222135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40e4da0 0x7fe9a41b00f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0x7fe98c006700 tx=0x7fe98c009b80).stop 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 -- 172.21.2.117:0/1036222135 shutdown_connections 2021-02-10T11:40:53.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 --2- 172.21.2.117:0/1036222135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40e51a0 0x7fe9a4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 --2- 172.21.2.117:0/1036222135 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40e4da0 0x7fe9a41b00f0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 --2- 172.21.2.117:0/1036222135 >> v2:172.21.2.125:3300/0 conn(0x7fe9a407da20 0x7fe9a40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 -- 172.21.2.117:0/1036222135 >> 172.21.2.117:0/1036222135 conn(0x7fe9a40ce450 msgr2=0x7fe9a40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 -- 172.21.2.117:0/1036222135 shutdown_connections 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.019 7fe9a8af4700 1 -- 172.21.2.117:0/1036222135 wait complete. 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.033 7fe9a8af4700 1 Processor -- start 2021-02-10T11:40:53.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.033 7fe9a8af4700 1 -- start start 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 0x7fe9a40b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 0x7fe9a40b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9a4090de0 0x7fe9a40911e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4090600 con 0x7fe9a40c33d0 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a4090860 con 0x7fe9a4090de0 2021-02-10T11:40:53.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a8af4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40b3c70 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 0x7fe9a40b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a305f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9a4090de0 0x7fe9a40911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 0x7fe9a40b3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48260/0 (socket says 172.21.2.117:48260) 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a205d700 1 -- 172.21.2.117:0/3444189488 learned_addr learned my addr 172.21.2.117:0/3444189488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 0x7fe9a40b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 -- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9a4090de0 msgr2=0x7fe9a40911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9a4090de0 0x7fe9a40911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 -- 172.21.2.117:0/3444189488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 msgr2=0x7fe9a40b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 0x7fe9a40b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.034 7fe9a285e700 1 -- 172.21.2.117:0/3444189488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4091800 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe9a285e700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 0x7fe9a40b3030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe994006e90 tx=0x7fe99400b860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9940009b0 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe994006240 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a4091800 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a4091800 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9940053d0 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9940053d0 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 0x7fe98801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.035 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fe9940324f0 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.036 7fe9a205d700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 0x7fe98801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:53.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.036 7fe9a205d700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 0x7fe98801ac10 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.066 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a4103370 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.069 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe994006030 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.204 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fe9a4077a70 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.625 7fe99b7fe700 1 -- 172.21.2.117:0/3444189488 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v44) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe9940053d0 con 0x7fe9a40b3ff0 2021-02-10T11:40:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:40:53.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 msgr2=0x7fe98801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:53.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 0x7fe98801ac10 crc :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 msgr2=0x7fe9a40b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 0x7fe9a40b3030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe994006e90 tx=0x7fe99400b860).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 shutdown_connections 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.673 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe988018730 0x7fe98801ac10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9a40c33d0 0x7fe9a40b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9a4090de0 0x7fe9a40911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 --2- 172.21.2.117:0/3444189488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9a40b3ff0 0x7fe9a40b3030 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 >> 172.21.2.117:0/3444189488 conn(0x7fe9a40a1420 msgr2=0x7fe9a409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 shutdown_connections 2021-02-10T11:40:53.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:53.674 7fe9a8af4700 1 -- 172.21.2.117:0/3444189488 wait complete. 2021-02-10T11:40:53.665 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:40:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.095 7f1f2c91a700 1 Processor -- start 2021-02-10T11:40:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.095 7f1f2c91a700 1 -- start start 2021-02-10T11:40:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2c91a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f28079000 0x7f1f280e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2c91a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f280d1d20 con 0x7f1f281b00f0 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2c91a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f28077a70 con 0x7f1f2807da20 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2c91a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280e5430 con 0x7f1f28079000 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f26e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1f281b00f0 0x7f1f280d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48516/0 (socket says 172.21.2.117:48516) 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f26e1d700 1 -- 172.21.2.117:0/204574636 learned_addr learned my addr 172.21.2.117:0/204574636 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776491780 0 0) 0x7f1f20001500 con 0x7f1f281b00f0 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f14002fe0 con 0x7f1f281b00f0 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f28079000 0x7f1f280e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1683512571 0 0) 0x7f1f20001460 con 0x7f1f281b00f0 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f28079000 msgr2=0x7f1f280e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f28079000 0x7f1f280e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 >> v1:172.21.2.125:6789/0 conn(0x7f1f2807da20 legacy=0x7f1f280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28079400 con 0x7f1f281b00f0 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.096 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1f20001980 con 0x7f1f281b00f0 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f28079000 0x7f1f280e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 >> v1:172.21.2.117:6789/0 conn(0x7f1f281b00f0 legacy=0x7f1f280d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f14003a60 0x7f1f14005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f14006770 0x7f1f14008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f14009590 0x7f1f1400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1400c2d0 con 0x7f1f14006770 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1400c4f0 con 0x7f1f14003a60 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1400c710 con 0x7f1f14009590 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f14006770 0x7f1f14008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f26e1d700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f14009590 0x7f1f1400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2661c700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f14003a60 0x7f1f14005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 -- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f14009590 msgr2=0x7f1f1400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f14009590 0x7f1f1400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 -- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f14003a60 msgr2=0x7f1f14005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f14003a60 0x7f1f14005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 -- 172.21.2.117:0/204574636 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f140509c0 con 0x7f1f14006770 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f25e1b700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f14006770 0x7f1f14008c90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f1f10006a90 tx=0x7f1f10009a90).ready entity=mon.0 client_cookie=699eeea25c0fd70f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1000c0c0 con 0x7f1f14006770 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000c3c0 con 0x7f1f14006770 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2561a700 1 -- 172.21.2.117:0/204574636 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f1000c8e0 con 0x7f1f14006770 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2c91a700 1 -- 172.21.2.117:0/204574636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f14006770 msgr2=0x7f1f14008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f2c91a700 1 --2- 172.21.2.117:0/204574636 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f14006770 0x7f1f14008c90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7f1f10006a90 tx=0x7f1f10009a90).stop 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.097 7f1f26e1d700 1 -- 172.21.2.117:0/204574636 reap_dead start 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.098 7f1f2c91a700 1 -- 172.21.2.117:0/204574636 shutdown_connections 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.098 7f1f2c91a700 1 -- 172.21.2.117:0/204574636 >> 172.21.2.117:0/204574636 conn(0x7f1f280ce450 msgr2=0x7f1f280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.098 7f1f2c91a700 1 -- 172.21.2.117:0/204574636 shutdown_connections 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.098 7f1f2c91a700 1 -- 172.21.2.117:0/204574636 wait complete. 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 Processor -- start 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 -- start start 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f280a47d0 0x7f1f280a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f280a51f0 0x7f1f280a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28189970 con 0x7f1f280b77d0 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28190aa0 con 0x7f1f280a51f0 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f2c91a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280b81f0 con 0x7f1f280a47d0 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f26e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.113 7f1f26e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48276/0 (socket says 172.21.2.117:48276) 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 -- 172.21.2.117:0/459732683 learned_addr learned my addr 172.21.2.117:0/459732683 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:54.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 -- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f280a47d0 msgr2=0x7f1f280a4bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:54.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f280a47d0 0x7f1f280a4bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:54.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 -- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f280a51f0 msgr2=0x7f1f280a55f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:40:54.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f280a51f0 0x7f1f280a55f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:54.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 -- 172.21.2.117:0/459732683 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f280b8410 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f26e1d700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f1f2000a870 tx=0x7f1f2000ac10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f20007030 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20007370 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f200049a0 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f280b8410 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.114 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f280b89a0 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.115 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f20004720 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.115 7f1f1effd700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 0x7f1f0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.115 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f1f20032870 con 0x7f1f280b77d0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.115 7f1f2661c700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 0x7f1f0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:54.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.116 7f1f2661c700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 0x7f1f0c01abc0 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:54.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.145 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f28077a70 con 0x7f1f280b77d0 2021-02-10T11:40:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.148 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f20004930 con 0x7f1f280b77d0 2021-02-10T11:40:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.276 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f1f28077a70 con 0x7f1f280b77d0 2021-02-10T11:40:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.631 7f1f1effd700 1 -- 172.21.2.117:0/459732683 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v45) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1f20004720 con 0x7f1f280b77d0 2021-02-10T11:40:54.625 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:40:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 msgr2=0x7f1f0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:54.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 0x7f1f0c01abc0 crc :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 msgr2=0x7f1f280b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f1f2000a870 tx=0x7f1f2000ac10).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 shutdown_connections 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1f0c0186c0 0x7f1f0c01abc0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f280b77d0 0x7f1f280b7bd0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f280a51f0 0x7f1f280a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 --2- 172.21.2.117:0/459732683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f280a47d0 0x7f1f280a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:54.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 >> 172.21.2.117:0/459732683 conn(0x7f1f280b5140 msgr2=0x7f1f280a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 shutdown_connections 2021-02-10T11:40:54.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:54.644 7f1f2c91a700 1 -- 172.21.2.117:0/459732683 wait complete. 2021-02-10T11:40:54.636 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.072 7f0e51949700 1 Processor -- start 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.072 7f0e51949700 1 -- start start 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e51949700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e4c07da20 0x7f0e4c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e51949700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0e4c0e4e00 0x7f0e4c1b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e51949700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4c077a10 con 0x7f0e4c0e5200 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e51949700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c0777d0 con 0x7f0e4c07da20 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e51949700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e4c07de20 con 0x7f0e4c0e4e00 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4b574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e4c07da20 0x7f0e4c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4b574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e4c07da20 0x7f0e4c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48284/0 (socket says 172.21.2.117:48284) 2021-02-10T11:40:55.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4bd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0e4c0e5200 0x7f0e4c077f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48534/0 (socket says 172.21.2.117:48534) 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4b574700 1 -- 172.21.2.117:0/3477125188 learned_addr learned my addr 172.21.2.117:0/3477125188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4a572700 1 -- 172.21.2.117:0/3477125188 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909145049 0 0) 0x7f0e44001480 con 0x7f0e4c0e5200 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.073 7f0e4a572700 1 -- 172.21.2.117:0/3477125188 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e34002fe0 con 0x7f0e4c0e5200 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4b574700 1 -- 172.21.2.117:0/3477125188 >> v2:172.21.2.125:3300/0 conn(0x7f0e4c0e4e00 msgr2=0x7f0e4c1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4b574700 1 --2- 172.21.2.117:0/3477125188 >> v2:172.21.2.125:3300/0 conn(0x7f0e4c0e4e00 0x7f0e4c1b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4b574700 1 -- 172.21.2.117:0/3477125188 >> v1:172.21.2.117:6789/0 conn(0x7f0e4c0e5200 legacy=0x7f0e4c077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4b574700 1 -- 172.21.2.117:0/3477125188 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c1b0750 con 0x7f0e4c07da20 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4b574700 1 --2- 172.21.2.117:0/3477125188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c07da20 0x7f0e4c0771b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f0e3c006700 tx=0x7f0e3c009b80).ready entity=mon.0 client_cookie=aa352ed36a587f6b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4a572700 1 -- 172.21.2.117:0/3477125188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e3c00c0c0 con 0x7f0e4c07da20 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4a572700 1 -- 172.21.2.117:0/3477125188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c00c3c0 con 0x7f0e4c07da20 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e4a572700 1 -- 172.21.2.117:0/3477125188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e3c00c8e0 con 0x7f0e4c07da20 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e51949700 1 -- 172.21.2.117:0/3477125188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c07da20 msgr2=0x7f0e4c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.074 7f0e51949700 1 --2- 172.21.2.117:0/3477125188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c07da20 0x7f0e4c0771b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f0e3c006700 tx=0x7f0e3c009b80).stop 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 -- 172.21.2.117:0/3477125188 shutdown_connections 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 --2- 172.21.2.117:0/3477125188 >> v2:172.21.2.125:3300/0 conn(0x7f0e4c0e4e00 0x7f0e4c1b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 --2- 172.21.2.117:0/3477125188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c07da20 0x7f0e4c0771b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 -- 172.21.2.117:0/3477125188 >> 172.21.2.117:0/3477125188 conn(0x7f0e4c0ce480 msgr2=0x7f0e4c0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 -- 172.21.2.117:0/3477125188 shutdown_connections 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.075 7f0e51949700 1 -- 172.21.2.117:0/3477125188 wait complete. 2021-02-10T11:40:55.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.089 7f0e51949700 1 Processor -- start 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.089 7f0e51949700 1 -- start start 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e4c104dc0 0x7f0e4c1051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 0x7f0e4c105be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c0be6c0 con 0x7f0e4c09c210 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c104840 con 0x7f0e4c1057e0 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e51949700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4c1044f0 con 0x7f0e4c104dc0 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48290/0 (socket says 172.21.2.117:48290) 2021-02-10T11:40:55.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 -- 172.21.2.117:0/3558650237 learned_addr learned my addr 172.21.2.117:0/3558650237 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 -- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e4c104dc0 msgr2=0x7f0e4c1051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e4c104dc0 0x7f0e4c1051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 -- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 msgr2=0x7f0e4c105be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4ad73700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 0x7f0e4c105be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.090 7f0e4bd75700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 0x7f0e4c105be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e4bd75700 1 -- 172.21.2.117:0/3558650237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c09cc30 con 0x7f0e4c09c210 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e4ad73700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 0x7f0e4c105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e4bd75700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f0e44009fc0 tx=0x7f0e4400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e44006030 con 0x7f0e4c09c210 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e440062b0 con 0x7f0e4c09c210 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e44004720 con 0x7f0e4c09c210 2021-02-10T11:40:55.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e51949700 1 -- 172.21.2.117:0/3558650237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4c09cc30 con 0x7f0e4c09c210 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.091 7f0e51949700 1 -- 172.21.2.117:0/3558650237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4c09cc30 con 0x7f0e4c09c210 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.092 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e44006490 con 0x7f0e4c09c210 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.092 7f0e33fff700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 0x7f0e2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.092 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f0e440329e0 con 0x7f0e4c09c210 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.092 7f0e4b574700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 0x7f0e2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.093 7f0e4b574700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 0x7f0e2c01abc0 crc :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.123 7f0e51949700 1 -- 172.21.2.117:0/3558650237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4c1b70a0 con 0x7f0e4c09c210 2021-02-10T11:40:55.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.125 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e44025ed0 con 0x7f0e4c09c210 2021-02-10T11:40:55.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.252 7f0e51949700 1 -- 172.21.2.117:0/3558650237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f0e4c077a10 con 0x7f0e4c09c210 2021-02-10T11:40:55.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.255 7f0e33fff700 1 -- 172.21.2.117:0/3558650237 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 8 and data pool 9 v25) v1 ==== 187+0+0 (secure 0 0 0) 0x7f0e44025ed0 con 0x7f0e4c09c210 2021-02-10T11:40:55.244 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 8 and data pool 9 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 -- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 msgr2=0x7f0e2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 0x7f0e2c01abc0 crc :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 -- 172.21.2.117:0/3558650237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 msgr2=0x7f0e4c09c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7f0e44009fc0 tx=0x7f0e4400bc40).stop 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 -- 172.21.2.117:0/3558650237 shutdown_connections 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e2c0186c0 0x7f0e2c01abc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e4c09c210 0x7f0e4c09c610 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e4c1057e0 0x7f0e4c105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 --2- 172.21.2.117:0/3558650237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e4c104dc0 0x7f0e4c1051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:55.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 -- 172.21.2.117:0/3558650237 >> 172.21.2.117:0/3558650237 conn(0x7f0e4c0a6770 msgr2=0x7f0e4c1b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:55.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.261 7f0e51949700 1 -- 172.21.2.117:0/3558650237 shutdown_connections 2021-02-10T11:40:55.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.262 7f0e51949700 1 -- 172.21.2.117:0/3558650237 wait complete. 2021-02-10T11:40:55.254 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:55.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.680 7f499c0df700 1 Processor -- start 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.680 7f499c0df700 1 -- start start 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499c0df700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 0x7f49940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499c0df700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49940e5190 0x7f4994079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499c0df700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4994077a70 con 0x7f49940e4d90 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499c0df700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49940e5c10 con 0x7f49940e5190 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499c0df700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49940777d0 con 0x7f499407da20 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49940e5190 0x7f4994079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49940e5190 0x7f4994079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48306/0 (socket says 172.21.2.117:48306) 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 -- 172.21.2.117:0/2407609712 learned_addr learned my addr 172.21.2.117:0/2407609712 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f4999efb700 1 --2- 172.21.2.117:0/2407609712 >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 0x7f49940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 -- 172.21.2.117:0/2407609712 >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 msgr2=0x7f49940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 --2- 172.21.2.117:0/2407609712 >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 0x7f49940771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 -- 172.21.2.117:0/2407609712 >> v1:172.21.2.117:6789/0 conn(0x7f49940e4d90 legacy=0x7f49941b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f499a6fc700 1 -- 172.21.2.117:0/2407609712 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49941b08d0 con 0x7f49940e5190 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.681 7f4999efb700 1 --2- 172.21.2.117:0/2407609712 >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 0x7f49940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f499a6fc700 1 --2- 172.21.2.117:0/2407609712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49940e5190 0x7f4994079000 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f498c006f00 tx=0x7f498c009f10).ready entity=mon.0 client_cookie=9b53683a14747b48 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f4998ef9700 1 -- 172.21.2.117:0/2407609712 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c00b3b0 con 0x7f49940e5190 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f4998ef9700 1 -- 172.21.2.117:0/2407609712 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00b630 con 0x7f49940e5190 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f4998ef9700 1 -- 172.21.2.117:0/2407609712 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f498c00bb50 con 0x7f49940e5190 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f499c0df700 1 -- 172.21.2.117:0/2407609712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49940e5190 msgr2=0x7f4994079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.682 7f499c0df700 1 --2- 172.21.2.117:0/2407609712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49940e5190 0x7f4994079000 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7f498c006f00 tx=0x7f498c009f10).stop 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 -- 172.21.2.117:0/2407609712 shutdown_connections 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 --2- 172.21.2.117:0/2407609712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49940e5190 0x7f4994079000 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 --2- 172.21.2.117:0/2407609712 >> v2:172.21.2.125:3300/0 conn(0x7f499407da20 0x7f49940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 -- 172.21.2.117:0/2407609712 >> 172.21.2.117:0/2407609712 conn(0x7f49940ce440 msgr2=0x7f49940e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:55.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 -- 172.21.2.117:0/2407609712 shutdown_connections 2021-02-10T11:40:55.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.683 7f499c0df700 1 -- 172.21.2.117:0/2407609712 wait complete. 2021-02-10T11:40:55.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.697 7f499c0df700 1 Processor -- start 2021-02-10T11:40:55.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.697 7f499c0df700 1 -- start start 2021-02-10T11:40:55.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4994095590 0x7f4994104d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 0x7f49940b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49941af170 con 0x7f49941aebe0 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4994105950 con 0x7f49940b1420 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499c0df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4994105b70 con 0x7f4994095590 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499a6fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 0x7f49940b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499a6fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 0x7f49940b1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:52696/0 (socket says 172.21.2.117:52696) 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48310/0 (socket says 172.21.2.117:48310) 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f499a6fc700 1 -- 172.21.2.117:0/811359480 learned_addr learned my addr 172.21.2.117:0/811359480 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f4999efb700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4994095590 0x7f4994104d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 -- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4994095590 msgr2=0x7f4994104d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4994095590 0x7f4994104d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 -- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 msgr2=0x7f49940b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 0x7f49940b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 -- 172.21.2.117:0/811359480 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940b1e40 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.698 7f49996fa700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f498800b490 tx=0x7f498800b830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49880009b0 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800bc20 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f499c0df700 1 -- 172.21.2.117:0/811359480 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49940b1e40 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4988001690 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f499c0df700 1 -- 172.21.2.117:0/811359480 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49940b1e40 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4988001200 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.699 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4988032860 con 0x7f49941aebe0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.700 7f4999efb700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:55.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.700 7f4999efb700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 0x7f497c01ac10 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:55.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.730 7f499c0df700 1 -- 172.21.2.117:0/811359480 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49941b4f30 con 0x7f49941aebe0 2021-02-10T11:40:55.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.733 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4988000ea0 con 0x7f49941aebe0 2021-02-10T11:40:55.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.858 7f499c0df700 1 -- 172.21.2.117:0/811359480 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49941b4f30 con 0x7f49941aebe0 2021-02-10T11:40:55.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.859 7f49867fc700 1 -- 172.21.2.117:0/811359480 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f4988001ee0 con 0x7f49941aebe0 2021-02-10T11:40:55.843 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:55.843 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":46,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:55.253142","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:53.223172","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:54.293749","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.860 7f499c0df700 1 -- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 msgr2=0x7f497c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.860 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 0x7f497c01ac10 crc :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 -- 172.21.2.117:0/811359480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 msgr2=0x7f4994105330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f498800b490 tx=0x7f498800b830).stop 2021-02-10T11:40:55.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 -- 172.21.2.117:0/811359480 shutdown_connections 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f497c018730 0x7f497c01ac10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49941aebe0 0x7f4994105330 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49940b1420 0x7f49940b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 --2- 172.21.2.117:0/811359480 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4994095590 0x7f4994104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 -- 172.21.2.117:0/811359480 >> 172.21.2.117:0/811359480 conn(0x7f49941ace70 msgr2=0x7f4994198a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 -- 172.21.2.117:0/811359480 shutdown_connections 2021-02-10T11:40:55.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:55.861 7f499c0df700 1 -- 172.21.2.117:0/811359480 wait complete. 2021-02-10T11:40:55.853 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:56.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.280 7f04be3ed700 1 Processor -- start 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.280 7f04be3ed700 1 -- start start 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f04b8079000 0x7f04b80e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b81b00e0 0x7f04b81b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b8077a70 con 0x7f04b807da20 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b80e53b0 con 0x7f04b8079000 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b8079400 con 0x7f04b81b00e0 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f04b8079000 0x7f04b80e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f04b8079000 0x7f04b80e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48318/0 (socket says 172.21.2.117:48318) 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 -- 172.21.2.117:0/155773138 learned_addr learned my addr 172.21.2.117:0/155773138 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04bca0a700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b81b00e0 0x7f04b81b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 -- 172.21.2.117:0/155773138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b81b00e0 msgr2=0x7f04b81b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b81b00e0 0x7f04b81b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 -- 172.21.2.117:0/155773138 >> v1:172.21.2.125:6789/0 conn(0x7f04b807da20 legacy=0x7f04b80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 -- 172.21.2.117:0/155773138 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b807de20 con 0x7f04b8079000 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b77fe700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8079000 0x7f04b80e4d90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f04ac006700 tx=0x7f04ac009b80).ready entity=mon.0 client_cookie=7345711edaf74e72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b6ffd700 1 -- 172.21.2.117:0/155773138 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ac00c0c0 con 0x7f04b8079000 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b6ffd700 1 -- 172.21.2.117:0/155773138 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ac00c440 con 0x7f04b8079000 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04b6ffd700 1 -- 172.21.2.117:0/155773138 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ac00c920 con 0x7f04b8079000 2021-02-10T11:40:56.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 -- 172.21.2.117:0/155773138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8079000 msgr2=0x7f04b80e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.281 7f04be3ed700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8079000 0x7f04b80e4d90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f04ac006700 tx=0x7f04ac009b80).stop 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 -- 172.21.2.117:0/155773138 shutdown_connections 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b81b00e0 0x7f04b81b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 --2- 172.21.2.117:0/155773138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8079000 0x7f04b80e4d90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 -- 172.21.2.117:0/155773138 >> 172.21.2.117:0/155773138 conn(0x7f04b80ce440 msgr2=0x7f04b80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:56.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 -- 172.21.2.117:0/155773138 shutdown_connections 2021-02-10T11:40:56.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.282 7f04be3ed700 1 -- 172.21.2.117:0/155773138 wait complete. 2021-02-10T11:40:56.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.296 7f04be3ed700 1 Processor -- start 2021-02-10T11:40:56.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.296 7f04be3ed700 1 -- start start 2021-02-10T11:40:56.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04b81067a0 0x7f04b8106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b818cbd0 0x7f04b818cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b8193730 con 0x7f04b8192ba0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b818d630 con 0x7f04b81067a0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04be3ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b818d850 con 0x7f04b818cbd0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48324/0 (socket says 172.21.2.117:48324) 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 -- 172.21.2.117:0/3852030633 learned_addr learned my addr 172.21.2.117:0/3852030633 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b77fe700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04b81067a0 0x7f04b8106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04bca0a700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b818cbd0 0x7f04b818cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 -- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b818cbd0 msgr2=0x7f04b818cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b818cbd0 0x7f04b818cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 -- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04b81067a0 msgr2=0x7f04b8106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04b81067a0 0x7f04b8106ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.297 7f04b7fff700 1 -- 172.21.2.117:0/3852030633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b818dc30 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b7fff700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f04a800b5b0 tx=0x7f04a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04a800bd40 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8006240 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04a8005690 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b818dc30 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b818dc30 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04a80054a0 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 0x7f049801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.298 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f04a80327a0 con 0x7f04b8192ba0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.299 7f04b77fe700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 0x7f049801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.299 7f04b77fe700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 0x7f049801ac10 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.330 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b8077a70 con 0x7f04b8192ba0 2021-02-10T11:40:56.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.332 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04a8000c60 con 0x7f04b8192ba0 2021-02-10T11:40:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.460 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04b8077a70 con 0x7f04b8192ba0 2021-02-10T11:40:56.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.461 7f04b4ff9700 1 -- 172.21.2.117:0/3852030633 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f04a80054a0 con 0x7f04b8192ba0 2021-02-10T11:40:56.445 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:56.445 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":47,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:56.254993","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:53.223172","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:54.293749","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.462 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 msgr2=0x7f049801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.462 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 0x7f049801ac10 crc :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.462 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 msgr2=0x7f04b8192fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.462 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0x7f04a800b5b0 tx=0x7f04a800b950).stop 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 shutdown_connections 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04b818cbd0 0x7f04b818cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04b81067a0 0x7f04b8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0498018730 0x7f049801ac10 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 --2- 172.21.2.117:0/3852030633 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04b8192ba0 0x7f04b8192fa0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 >> 172.21.2.117:0/3852030633 conn(0x7f04b819f510 msgr2=0x7f04b81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 shutdown_connections 2021-02-10T11:40:56.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.463 7f04be3ed700 1 -- 172.21.2.117:0/3852030633 wait complete. 2021-02-10T11:40:56.456 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 Processor -- start 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 -- start start 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e6807da20 0x7f7e680771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e680d1d20 con 0x7f7e680e51a0 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e68077a70 con 0x7f7e680e4da0 2021-02-10T11:40:56.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.885 7f7e704e7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e680777d0 con 0x7f7e6807da20 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e6807da20 0x7f7e680771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e6807da20 0x7f7e680771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48332/0 (socket says 172.21.2.117:48332) 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 -- 172.21.2.117:0/2262979087 learned_addr learned my addr 172.21.2.117:0/2262979087 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6eb04700 1 --1- 172.21.2.117:0/2262979087 >> v1:172.21.2.117:6789/0 conn(0x7f7e680e51a0 0x7f7e680d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48582/0 (socket says 172.21.2.117:48582) 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 -- 172.21.2.117:0/2262979087 >> v1:172.21.2.125:6789/0 conn(0x7f7e680e4da0 legacy=0x7f7e681b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 -- 172.21.2.117:0/2262979087 >> v1:172.21.2.117:6789/0 conn(0x7f7e680e51a0 legacy=0x7f7e680d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 -- 172.21.2.117:0/2262979087 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e681b08e0 con 0x7f7e6807da20 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6e303700 1 --2- 172.21.2.117:0/2262979087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e6807da20 0x7f7e680771b0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f7e5c006700 tx=0x7f7e5c009b80).ready entity=mon.0 client_cookie=c99dace77711d0b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.886 7f7e6d301700 1 -- 172.21.2.117:0/2262979087 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e5c00c0c0 con 0x7f7e6807da20 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e6d301700 1 -- 172.21.2.117:0/2262979087 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00c340 con 0x7f7e6807da20 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e6d301700 1 -- 172.21.2.117:0/2262979087 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e5c00c860 con 0x7f7e6807da20 2021-02-10T11:40:56.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 -- 172.21.2.117:0/2262979087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e6807da20 msgr2=0x7f7e680771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 --2- 172.21.2.117:0/2262979087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e6807da20 0x7f7e680771b0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f7e5c006700 tx=0x7f7e5c009b80).stop 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 -- 172.21.2.117:0/2262979087 shutdown_connections 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 --2- 172.21.2.117:0/2262979087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e6807da20 0x7f7e680771b0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 -- 172.21.2.117:0/2262979087 >> 172.21.2.117:0/2262979087 conn(0x7f7e680ce450 msgr2=0x7f7e680e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 -- 172.21.2.117:0/2262979087 shutdown_connections 2021-02-10T11:40:56.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.887 7f7e704e7700 1 -- 172.21.2.117:0/2262979087 wait complete. 2021-02-10T11:40:56.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 Processor -- start 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 -- start start 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e680933f0 0x7f7e680937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e68093e10 0x7f7e68106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68107a70 con 0x7f7e68107050 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680c2c60 con 0x7f7e68093e10 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e704e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680c2e80 con 0x7f7e680933f0 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e6eb04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e6eb04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48336/0 (socket says 172.21.2.117:48336) 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e6eb04700 1 -- 172.21.2.117:0/1855468562 learned_addr learned my addr 172.21.2.117:0/1855468562 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e6e303700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e680933f0 0x7f7e680937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.903 7f7e6db02700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e68093e10 0x7f7e68106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 -- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e680933f0 msgr2=0x7f7e680937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e680933f0 0x7f7e680937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 -- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e68093e10 msgr2=0x7f7e68106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e68093e10 0x7f7e68106a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 -- 172.21.2.117:0/1855468562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680c30a0 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e6eb04700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f7e6000a8a0 tx=0x7f7e6000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e60007b10 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60007020 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e680c30a0 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e600074b0 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.904 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e680c30a0 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.905 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e60007200 con 0x7f7e68107050 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.905 7f7e5affd700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 0x7f7e5001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:56.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.905 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7e60032b20 con 0x7f7e68107050 2021-02-10T11:40:56.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.905 7f7e6e303700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 0x7f7e5001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:56.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.905 7f7e6e303700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 0x7f7e5001ac10 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:56.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.935 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e68107e30 con 0x7f7e68107050 2021-02-10T11:40:56.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:56.938 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e60007230 con 0x7f7e68107050 2021-02-10T11:40:57.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.074 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f7e68107e30 con 0x7f7e68107050 2021-02-10T11:40:57.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.262 7f7e5affd700 1 -- 172.21.2.117:0/1855468562 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7e60007020 con 0x7f7e68107050 2021-02-10T11:40:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 msgr2=0x7f7e5001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 0x7f7e5001ac10 crc :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 msgr2=0x7f7e68107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7f7e6000a8a0 tx=0x7f7e6000ac40).stop 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 shutdown_connections 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e50018730 0x7f7e5001ac10 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e68107050 0x7f7e68107450 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e68093e10 0x7f7e68106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 --2- 172.21.2.117:0/1855468562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e680933f0 0x7f7e680937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.292 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 >> 172.21.2.117:0/1855468562 conn(0x7f7e681116a0 msgr2=0x7f7e680af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.293 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 shutdown_connections 2021-02-10T11:40:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.293 7f7e704e7700 1 -- 172.21.2.117:0/1855468562 wait complete. 2021-02-10T11:40:57.284 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:40:57.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.710 7f78a1fb7700 1 Processor -- start 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.710 7f78a1fb7700 1 -- start start 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f78a1fb7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f789c07da20 0x7f789c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f78a1fb7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c0e5190 0x7f789c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f78a1fb7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c077a70 con 0x7f789c0e4d90 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f78a1fb7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c0777d0 con 0x7f789c07da20 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f78a1fb7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1b08d0 con 0x7f789c0e5190 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f789b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f789c07da20 0x7f789c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f789b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f789c07da20 0x7f789c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48344/0 (socket says 172.21.2.117:48344) 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f789b7fe700 1 -- 172.21.2.117:0/1215759845 learned_addr learned my addr 172.21.2.117:0/1215759845 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f789b7fe700 1 -- 172.21.2.117:0/1215759845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c0e5190 msgr2=0x7f789c079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:57.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.711 7f789b7fe700 1 --2- 172.21.2.117:0/1215759845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c0e5190 0x7f789c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789b7fe700 1 -- 172.21.2.117:0/1215759845 >> v1:172.21.2.125:6789/0 conn(0x7f789c0e4d90 legacy=0x7f789c1b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789b7fe700 1 -- 172.21.2.117:0/1215759845 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c07de20 con 0x7f789c07da20 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789b7fe700 1 --2- 172.21.2.117:0/1215759845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c07da20 0x7f789c0771b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f7884006700 tx=0x7f7884009b80).ready entity=mon.0 client_cookie=2967346b47a895b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789a7fc700 1 -- 172.21.2.117:0/1215759845 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788400c0c0 con 0x7f789c07da20 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789a7fc700 1 -- 172.21.2.117:0/1215759845 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788400c340 con 0x7f789c07da20 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f789a7fc700 1 -- 172.21.2.117:0/1215759845 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788400c860 con 0x7f789c07da20 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f78a1fb7700 1 -- 172.21.2.117:0/1215759845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c07da20 msgr2=0x7f789c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.712 7f78a1fb7700 1 --2- 172.21.2.117:0/1215759845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c07da20 0x7f789c0771b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7f7884006700 tx=0x7f7884009b80).stop 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 -- 172.21.2.117:0/1215759845 shutdown_connections 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 --2- 172.21.2.117:0/1215759845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c0e5190 0x7f789c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 --2- 172.21.2.117:0/1215759845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c07da20 0x7f789c0771b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 -- 172.21.2.117:0/1215759845 >> 172.21.2.117:0/1215759845 conn(0x7f789c0ce440 msgr2=0x7f789c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 -- 172.21.2.117:0/1215759845 shutdown_connections 2021-02-10T11:40:57.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.713 7f78a1fb7700 1 -- 172.21.2.117:0/1215759845 wait complete. 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.727 7f78a1fb7700 1 Processor -- start 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.727 7f78a1fb7700 1 -- start start 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c18ee60 0x7f789c18f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f789c18f880 0x7f789c18fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1af0c0 con 0x7f789c18cff0 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c18da10 con 0x7f789c18f880 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f78a1fb7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c18dc30 con 0x7f789c18ee60 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48352/0 (socket says 172.21.2.117:48352) 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 -- 172.21.2.117:0/1797348123 learned_addr learned my addr 172.21.2.117:0/1797348123 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 -- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c18ee60 msgr2=0x7f789c18f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c18ee60 0x7f789c18f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 -- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f789c18f880 msgr2=0x7f789c18fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f789c18f880 0x7f789c18fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 -- 172.21.2.117:0/1797348123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c18de50 con 0x7f789c18cff0 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.728 7f789bfff700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f789400b8c0 tx=0x7f789400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7894005af0 con 0x7f789c18cff0 2021-02-10T11:40:57.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7894005d30 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c18de50 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78940062b0 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c18de50 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7894006030 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.729 7f7893fff700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 0x7f788001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.730 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7894032cd0 con 0x7f789c18cff0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.730 7f789b7fe700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 0x7f788001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:57.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.730 7f789b7fe700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 0x7f788001abc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:57.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.761 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c18e930 con 0x7f789c18cff0 2021-02-10T11:40:57.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.763 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7894005a40 con 0x7f789c18cff0 2021-02-10T11:40:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.892 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f789c18e930 con 0x7f789c18cff0 2021-02-10T11:40:57.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.892 7f7893fff700 1 -- 172.21.2.117:0/1797348123 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f7894006030 con 0x7f789c18cff0 2021-02-10T11:40:57.877 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 28 2021-02-10T11:40:57.877 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:57.877 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:40:57.258727","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5657":{"gid":5657,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/58270881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":58270881},{"type":"v1","addr":"172.21.2.117:6833","nonce":58270881}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 msgr2=0x7f788001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 0x7f788001abc0 crc :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 msgr2=0x7f789c18d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f789400b8c0 tx=0x7f789400bc60).stop 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 shutdown_connections 2021-02-10T11:40:57.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7880018730 0x7f788001abc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f789c18cff0 0x7f789c18d3f0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f789c18f880 0x7f789c18fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 --2- 172.21.2.117:0/1797348123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789c18ee60 0x7f789c18f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 >> 172.21.2.117:0/1797348123 conn(0x7f789c18cba0 msgr2=0x7f789c1a6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 shutdown_connections 2021-02-10T11:40:57.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:57.894 7f78a1fb7700 1 -- 172.21.2.117:0/1797348123 wait complete. 2021-02-10T11:40:57.887 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 Processor -- start 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 -- start start 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd16007da20 0x7fd1600771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600d1d10 con 0x7fd1600e4d90 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160077a70 con 0x7fd1600e5190 2021-02-10T11:40:58.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd16580c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600777d0 con 0x7fd16007da20 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd15ed73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd1600e4d90 0x7fd1601b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48606/0 (socket says 172.21.2.117:48606) 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.313 7fd15ed73700 1 -- 172.21.2.117:0/2392904691 learned_addr learned my addr 172.21.2.117:0/2392904691 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113090105 0 0) 0x7fd1480010b0 con 0x7fd1600e4d90 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15f574700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd16007da20 0x7fd1600771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c002fe0 con 0x7fd1600e4d90 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1107952539 0 0) 0x7fd1480010b0 con 0x7fd1600e4d90 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd16007da20 msgr2=0x7fd1600771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd16007da20 0x7fd1600771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 >> v1:172.21.2.125:6789/0 conn(0x7fd1600e5190 legacy=0x7fd1600d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1601b08d0 con 0x7fd1600e4d90 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd1480015b0 con 0x7fd1600e4d90 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 -- 172.21.2.117:0/2392904691 >> v1:172.21.2.117:6789/0 conn(0x7fd1600e4d90 legacy=0x7fd1601b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd14c003b10 0x7fd14c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14c006800 0x7fd14c008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.314 7fd15e572700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd14c0095a0 0x7fd14c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00c370 con 0x7fd14c0095a0 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00c590 con 0x7fd14c006800 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c00c7b0 con 0x7fd14c003b10 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd14c0095a0 0x7fd14c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15f574700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd16007da20 0x7fd1600771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15ed73700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14c006800 0x7fd14c008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15f574700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd14c003b10 0x7fd14c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 -- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd14c003b10 msgr2=0x7fd14c005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd14c003b10 0x7fd14c005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 -- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14c006800 msgr2=0x7fd14c008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14c006800 0x7fd14c008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 -- 172.21.2.117:0/2392904691 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c050a20 con 0x7fd14c0095a0 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15f574700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd14c003b10 0x7fd14c005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd14c0095a0 0x7fd14c00bac0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fd158006e30 tx=0x7fd158009e60).ready entity=mon.0 client_cookie=3c11fb0010a49090 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800b300 con 0x7fd14c0095a0 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15800b5c0 con 0x7fd14c0095a0 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15e572700 1 -- 172.21.2.117:0/2392904691 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800ba10 con 0x7fd14c0095a0 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd16580c700 1 -- 172.21.2.117:0/2392904691 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd14c0095a0 msgr2=0x7fd14c00bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd16580c700 1 --2- 172.21.2.117:0/2392904691 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd14c0095a0 0x7fd14c00bac0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7fd158006e30 tx=0x7fd158009e60).stop 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.315 7fd15fd75700 1 -- 172.21.2.117:0/2392904691 reap_dead start 2021-02-10T11:40:58.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.316 7fd16580c700 1 -- 172.21.2.117:0/2392904691 shutdown_connections 2021-02-10T11:40:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.316 7fd16580c700 1 -- 172.21.2.117:0/2392904691 >> 172.21.2.117:0/2392904691 conn(0x7fd1600ce440 msgr2=0x7fd1600e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.316 7fd16580c700 1 -- 172.21.2.117:0/2392904691 shutdown_connections 2021-02-10T11:40:58.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.316 7fd16580c700 1 -- 172.21.2.117:0/2392904691 wait complete. 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 Processor -- start 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 -- start start 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd160192ba0 0x7fd160192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1601067a0 0x7fd160106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160193730 con 0x7fd16018cbd0 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16018d630 con 0x7fd160192ba0 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd16580c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16018d850 con 0x7fd1601067a0 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd15fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd15fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48370/0 (socket says 172.21.2.117:48370) 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.331 7fd15fd75700 1 -- 172.21.2.117:0/959038108 learned_addr learned my addr 172.21.2.117:0/959038108 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 -- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1601067a0 msgr2=0x7fd160106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1601067a0 0x7fd160106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 -- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd160192ba0 msgr2=0x7fd160192fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd160192ba0 0x7fd160192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 -- 172.21.2.117:0/959038108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16018dc30 con 0x7fd16018cbd0 2021-02-10T11:40:58.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd15fd75700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7fd158009f80 tx=0x7fd15800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800c2f0 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15800c4f0 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd15800c900 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd16580c700 1 -- 172.21.2.117:0/959038108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16018dc30 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.332 7fd16580c700 1 -- 172.21.2.117:0/959038108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16018e1c0 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.333 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd158005820 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.333 7fd157fff700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 0x7fd14401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.333 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd158032a90 con 0x7fd16018cbd0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.333 7fd15f574700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 0x7fd14401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.334 7fd15f574700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 0x7fd14401abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.363 7fd16580c700 1 -- 172.21.2.117:0/959038108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd160077a70 con 0x7fd16018cbd0 2021-02-10T11:40:58.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.366 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd158009730 con 0x7fd16018cbd0 2021-02-10T11:40:58.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.490 7fd16580c700 1 -- 172.21.2.117:0/959038108 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd160077a70 con 0x7fd16018cbd0 2021-02-10T11:40:58.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.490 7fd157fff700 1 -- 172.21.2.117:0/959038108 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14226 (secure 0 0 0) 0x7fd158005820 con 0x7fd16018cbd0 2021-02-10T11:40:58.474 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:40:58.475 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":47,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:56.254993","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:53.223172","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:54.293749","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:40:58.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 msgr2=0x7fd14401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 0x7fd14401abc0 crc :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 msgr2=0x7fd16018cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7fd158009f80 tx=0x7fd15800bd40).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 shutdown_connections 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1440186c0 0x7fd14401abc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd16018cbd0 0x7fd16018cfd0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1601067a0 0x7fd160106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 --2- 172.21.2.117:0/959038108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd160192ba0 0x7fd160192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:58.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 >> 172.21.2.117:0/959038108 conn(0x7fd16019f510 msgr2=0x7fd1601abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 shutdown_connections 2021-02-10T11:40:58.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.492 7fd16580c700 1 -- 172.21.2.117:0/959038108 wait complete. 2021-02-10T11:40:58.484 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:58.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 Processor -- start 2021-02-10T11:40:58.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 -- start start 2021-02-10T11:40:58.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 0x7f18700d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870077a70 con 0x7f1870079000 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1870079400 con 0x7f187007da20 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f1874ea7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187007de20 con 0x7f18700d1510 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f186e312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1870079000 0x7f18700e4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48626/0 (socket says 172.21.2.117:48626) 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.922 7f186e312700 1 -- 172.21.2.117:0/1873293941 learned_addr learned my addr 172.21.2.117:0/1873293941 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186f314700 1 --2- 172.21.2.117:0/1873293941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 0x7f18700d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186db11700 1 -- 172.21.2.117:0/1873293941 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001546522 0 0) 0x7f18580009b0 con 0x7f1870079000 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186db11700 1 -- 172.21.2.117:0/1873293941 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c002fe0 con 0x7f1870079000 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186db11700 1 -- 172.21.2.117:0/1873293941 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345035591 0 0) 0x7f1860000c60 con 0x7f187007da20 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186db11700 1 -- 172.21.2.117:0/1873293941 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c0042c0 con 0x7f187007da20 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186f314700 1 -- 172.21.2.117:0/1873293941 >> v1:172.21.2.125:6789/0 conn(0x7f187007da20 legacy=0x7f18700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186f314700 1 -- 172.21.2.117:0/1873293941 >> v1:172.21.2.117:6789/0 conn(0x7f1870079000 legacy=0x7f18700e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186f314700 1 -- 172.21.2.117:0/1873293941 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18701b00d0 con 0x7f18700d1510 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.923 7f186f314700 1 --2- 172.21.2.117:0/1873293941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 0x7f18700d1910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1868006e50 tx=0x7f1868009df0).ready entity=mon.2 client_cookie=cd0c7534a597217e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.924 7f186db11700 1 -- 172.21.2.117:0/1873293941 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186800b510 con 0x7f18700d1510 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.924 7f186db11700 1 -- 172.21.2.117:0/1873293941 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1868003030 con 0x7f18700d1510 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.924 7f186db11700 1 -- 172.21.2.117:0/1873293941 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186800b7d0 con 0x7f18700d1510 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.924 7f1874ea7700 1 -- 172.21.2.117:0/1873293941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 msgr2=0x7f18700d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.924 7f1874ea7700 1 --2- 172.21.2.117:0/1873293941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 0x7f18700d1910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1868006e50 tx=0x7f1868009df0).stop 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.925 7f1874ea7700 1 -- 172.21.2.117:0/1873293941 shutdown_connections 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.925 7f1874ea7700 1 --2- 172.21.2.117:0/1873293941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18700d1510 0x7f18700d1910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.925 7f1874ea7700 1 -- 172.21.2.117:0/1873293941 >> 172.21.2.117:0/1873293941 conn(0x7f18700ce430 msgr2=0x7f18700e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.925 7f1874ea7700 1 -- 172.21.2.117:0/1873293941 shutdown_connections 2021-02-10T11:40:58.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.925 7f1874ea7700 1 -- 172.21.2.117:0/1873293941 wait complete. 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 Processor -- start 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 -- start start 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 0x7f18700b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1870187bd0 0x7f1870188740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870189cc0 con 0x7f18701b5770 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1870099c30 con 0x7f18700972c0 2021-02-10T11:40:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.939 7f1874ea7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187018c1b0 con 0x7f1870187bd0 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48382/0 (socket says 172.21.2.117:48382) 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 -- 172.21.2.117:0/1119330188 learned_addr learned my addr 172.21.2.117:0/1119330188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186eb13700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 0x7f18700b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186f314700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1870187bd0 0x7f1870188740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 -- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1870187bd0 msgr2=0x7f1870188740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1870187bd0 0x7f1870188740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 -- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 msgr2=0x7f18700b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 0x7f18700b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 -- 172.21.2.117:0/1119330188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18701881c0 con 0x7f18701b5770 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186eb13700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 0x7f18700b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f186e312700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f1858001c60 tx=0x7f185800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18580009b0 con 0x7f18701b5770 2021-02-10T11:40:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185800acc0 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18580045d0 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18701881c0 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.940 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187019edf0 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.941 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1858007f30 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.941 7f18677fe700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 0x7f185401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.941 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1858032560 con 0x7f18701b5770 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.941 7f186eb13700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 0x7f185401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.942 7f186eb13700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 0x7f185401abc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:58.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.972 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1870099c30 con 0x7f18701b5770 2021-02-10T11:40:58.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:58.975 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1858000e60 con 0x7f18701b5770 2021-02-10T11:40:59.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.102 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1870099c30 con 0x7f18701b5770 2021-02-10T11:40:59.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.106 7f18677fe700 1 -- 172.21.2.117:0/1119330188 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f185801f490 con 0x7f18701b5770 2021-02-10T11:40:59.093 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:40:59.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 msgr2=0x7f185401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 0x7f185401abc0 crc :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 msgr2=0x7f18700b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0x7f1858001c60 tx=0x7f185800a8d0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 shutdown_connections 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18540186c0 0x7f185401abc0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18701b5770 0x7f18700b1ba0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1870187bd0 0x7f1870188740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 --2- 172.21.2.117:0/1119330188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18700972c0 0x7f18700b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 >> 172.21.2.117:0/1119330188 conn(0x7f187008d980 msgr2=0x7f1870193d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 shutdown_connections 2021-02-10T11:40:59.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.111 7f1874ea7700 1 -- 172.21.2.117:0/1119330188 wait complete. 2021-02-10T11:40:59.103 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:40:59.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.531 7fbbfb8dd700 1 Processor -- start 2021-02-10T11:40:59.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.531 7fbbfb8dd700 1 -- start start 2021-02-10T11:40:59.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbfb8dd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbbf407da20 0x7fbbf40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbfb8dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbbf40e51a0 0x7fbbf4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbfb8dd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4077a70 con 0x7fbbf40e4da0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbfb8dd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf40e5c20 con 0x7fbbf40e51a0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbfb8dd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf40777d0 con 0x7fbbf407da20 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbbf40e51a0 0x7fbbf4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbbf40e51a0 0x7fbbf4079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48388/0 (socket says 172.21.2.117:48388) 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 -- 172.21.2.117:0/2635118008 learned_addr learned my addr 172.21.2.117:0/2635118008 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf8ef8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbbf40e4da0 0x7fbbf41b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48638/0 (socket says 172.21.2.117:48638) 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf96f9700 1 --2- 172.21.2.117:0/2635118008 >> v2:172.21.2.125:3300/0 conn(0x7fbbf407da20 0x7fbbf40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 -- 172.21.2.117:0/2635118008 >> v2:172.21.2.125:3300/0 conn(0x7fbbf407da20 msgr2=0x7fbbf40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 --2- 172.21.2.117:0/2635118008 >> v2:172.21.2.125:3300/0 conn(0x7fbbf407da20 0x7fbbf40771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 -- 172.21.2.117:0/2635118008 >> v1:172.21.2.117:6789/0 conn(0x7fbbf40e4da0 legacy=0x7fbbf41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 -- 172.21.2.117:0/2635118008 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf41b08e0 con 0x7fbbf40e51a0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.532 7fbbf9efa700 1 --2- 172.21.2.117:0/2635118008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf40e51a0 0x7fbbf4079000 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fbbec006f00 tx=0x7fbbec009e90).ready entity=mon.0 client_cookie=98c201896ee0367b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbe7fff700 1 -- 172.21.2.117:0/2635118008 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbec00b300 con 0x7fbbf40e51a0 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbe7fff700 1 -- 172.21.2.117:0/2635118008 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec00b640 con 0x7fbbf40e51a0 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbe7fff700 1 -- 172.21.2.117:0/2635118008 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbec00bb20 con 0x7fbbf40e51a0 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 -- 172.21.2.117:0/2635118008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf40e51a0 msgr2=0x7fbbf4079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 --2- 172.21.2.117:0/2635118008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf40e51a0 0x7fbbf4079000 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fbbec006f00 tx=0x7fbbec009e90).stop 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 -- 172.21.2.117:0/2635118008 shutdown_connections 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 --2- 172.21.2.117:0/2635118008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf40e51a0 0x7fbbf4079000 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 --2- 172.21.2.117:0/2635118008 >> v2:172.21.2.125:3300/0 conn(0x7fbbf407da20 0x7fbbf40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 -- 172.21.2.117:0/2635118008 >> 172.21.2.117:0/2635118008 conn(0x7fbbf40ce450 msgr2=0x7fbbf40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 -- 172.21.2.117:0/2635118008 shutdown_connections 2021-02-10T11:40:59.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.533 7fbbfb8dd700 1 -- 172.21.2.117:0/2635118008 wait complete. 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 Processor -- start 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 -- start start 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbbf4199db0 0x7fbbf419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbbf41a5a20 0x7fbbf41a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf40ab820 con 0x7fbbf4199390 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4092080 con 0x7fbbf4199db0 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbfb8dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4198530 con 0x7fbbf41a5a20 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbf96f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbf96f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48394/0 (socket says 172.21.2.117:48394) 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.548 7fbbf96f9700 1 -- 172.21.2.117:0/407932093 learned_addr learned my addr 172.21.2.117:0/407932093 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 -- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbbf41a5a20 msgr2=0x7fbbf41a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbbf41a5a20 0x7fbbf41a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 -- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbbf4199db0 msgr2=0x7fbbf419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbbf4199db0 0x7fbbf419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 -- 172.21.2.117:0/407932093 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf41a6440 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbf96f9700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fbbe8006f00 tx=0x7fbbe800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe800bd30 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe8002030 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf41a6440 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe8002590 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.549 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf41a6440 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.550 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbe8000c90 con 0x7fbbf4199390 2021-02-10T11:40:59.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.550 7fbbe5ffb700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 0x7fbbd401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:40:59.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.550 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbbe80327b0 con 0x7fbbf4199390 2021-02-10T11:40:59.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.550 7fbbf8ef8700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 0x7fbbd401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:40:59.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.551 7fbbf8ef8700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 0x7fbbd401ac10 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:40:59.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.581 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf4092010 con 0x7fbbf4199390 2021-02-10T11:40:59.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.584 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbe801f520 con 0x7fbbf4199390 2021-02-10T11:40:59.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.713 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbbf4092010 con 0x7fbbf4199390 2021-02-10T11:40:59.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.717 7fbbe5ffb700 1 -- 172.21.2.117:0/407932093 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbbe8000c90 con 0x7fbbf4199390 2021-02-10T11:40:59.704 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:40:59.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 msgr2=0x7fbbd401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 0x7fbbd401ac10 crc :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 msgr2=0x7fbbf4199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:40:59.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7fbbe8006f00 tx=0x7fbbe800b940).stop 2021-02-10T11:40:59.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 shutdown_connections 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbbf41a5a20 0x7fbbf41a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbbf4199db0 0x7fbbf419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbd4018730 0x7fbbd401ac10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 --2- 172.21.2.117:0/407932093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbbf4199390 0x7fbbf4199790 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.722 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 >> 172.21.2.117:0/407932093 conn(0x7fbbf4105650 msgr2=0x7fbbf4092460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.723 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 shutdown_connections 2021-02-10T11:40:59.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:40:59.723 7fbbfb8dd700 1 -- 172.21.2.117:0/407932093 wait complete. 2021-02-10T11:40:59.715 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.152 7f9398c97700 1 Processor -- start 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.152 7f9398c97700 1 -- start start 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9394079000 0x7f93940e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93940d1d20 con 0x7f939407da20 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9394077a70 con 0x7f93941b00f0 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940e53c0 con 0x7f9394079000 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9394079000 0x7f93940e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9394079000 0x7f93940e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48402/0 (socket says 172.21.2.117:48402) 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 -- 172.21.2.117:0/1647918747 learned_addr learned my addr 172.21.2.117:0/1647918747 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 -- 172.21.2.117:0/1647918747 >> v1:172.21.2.125:6789/0 conn(0x7f93941b00f0 legacy=0x7f93940d0520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 -- 172.21.2.117:0/1647918747 >> v1:172.21.2.117:6789/0 conn(0x7f939407da20 legacy=0x7f93940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 -- 172.21.2.117:0/1647918747 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394079400 con 0x7f9394079000 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939205d700 1 --2- 172.21.2.117:0/1647918747 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9394079000 0x7f93940e4da0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f937c006700 tx=0x7f937c009b80).ready entity=mon.0 client_cookie=fb73e634fe54509 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939185c700 1 -- 172.21.2.117:0/1647918747 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f937c00c0c0 con 0x7f9394079000 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939185c700 1 -- 172.21.2.117:0/1647918747 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937c00c340 con 0x7f9394079000 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f939185c700 1 -- 172.21.2.117:0/1647918747 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f937c00c820 con 0x7f9394079000 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 -- 172.21.2.117:0/1647918747 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9394079000 msgr2=0x7f93940e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.153 7f9398c97700 1 --2- 172.21.2.117:0/1647918747 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9394079000 0x7f93940e4da0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f937c006700 tx=0x7f937c009b80).stop 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.154 7f9398c97700 1 -- 172.21.2.117:0/1647918747 shutdown_connections 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.154 7f9398c97700 1 --2- 172.21.2.117:0/1647918747 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9394079000 0x7f93940e4da0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.154 7f9398c97700 1 -- 172.21.2.117:0/1647918747 >> 172.21.2.117:0/1647918747 conn(0x7f93940ce450 msgr2=0x7f93940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.154 7f9398c97700 1 -- 172.21.2.117:0/1647918747 shutdown_connections 2021-02-10T11:41:00.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.154 7f9398c97700 1 -- 172.21.2.117:0/1647918747 wait complete. 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.168 7f9398c97700 1 Processor -- start 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.168 7f9398c97700 1 -- start start 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 0x7f93941c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 0x7f939410def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93940b6f10 con 0x7f93940b5e10 2021-02-10T11:41:00.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939410e550 con 0x7f93940aba60 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f9398c97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93941c6720 con 0x7f939410daf0 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48406/0 (socket says 172.21.2.117:48406) 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 -- 172.21.2.117:0/1001826682 learned_addr learned my addr 172.21.2.117:0/1001826682 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 -- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 msgr2=0x7f939410def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939205d700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 0x7f939410def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939285e700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 0x7f93941c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 0x7f939410def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 -- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 msgr2=0x7f93941c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 0x7f93941c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939305f700 1 -- 172.21.2.117:0/1001826682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93940b6830 con 0x7f93940b5e10 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939205d700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 0x7f939410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.169 7f939285e700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 0x7f93941c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f939305f700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f938c009940 tx=0x7f938c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f938c003030 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c001270 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f938c001750 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f9398c97700 1 -- 172.21.2.117:0/1001826682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93940b6830 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.170 7f9398c97700 1 -- 172.21.2.117:0/1001826682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93940b6830 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.171 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f938c001450 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.171 7f938b7fe700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 0x7f937801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.171 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f938c032cc0 con 0x7f93940b5e10 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.171 7f939285e700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 0x7f937801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.172 7f939285e700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 0x7f937801abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.203 7f9398c97700 1 -- 172.21.2.117:0/1001826682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93940ce450 con 0x7f93940b5e10 2021-02-10T11:41:00.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.205 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f938c00be30 con 0x7f93940b5e10 2021-02-10T11:41:00.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.336 7f9398c97700 1 -- 172.21.2.117:0/1001826682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f93940ce450 con 0x7f93940b5e10 2021-02-10T11:41:00.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.339 7f938b7fe700 1 -- 172.21.2.117:0/1001826682 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f938c001490 con 0x7f93940b5e10 2021-02-10T11:41:00.327 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 msgr2=0x7f937801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 0x7f937801abc0 crc :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 msgr2=0x7f93940b6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7f938c009940 tx=0x7f938c00bcf0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 shutdown_connections 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93780186c0 0x7f937801abc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93940b5e10 0x7f93940b6210 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f939410daf0 0x7f939410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 --2- 172.21.2.117:0/1001826682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93940aba60 0x7f93941c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 >> 172.21.2.117:0/1001826682 conn(0x7f93940a01b0 msgr2=0x7f939409b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 shutdown_connections 2021-02-10T11:41:00.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.345 7f9398c97700 1 -- 172.21.2.117:0/1001826682 wait complete. 2021-02-10T11:41:00.337 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:00.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 Processor -- start 2021-02-10T11:41:00.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 -- start start 2021-02-10T11:41:00.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f327007da20 0x7f32700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32700d1d20 con 0x7f32700e51a0 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3270077a70 con 0x7f32700e4da0 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.767 7f32781b3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32700777d0 con 0x7f327007da20 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f327007da20 0x7f32700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f327007da20 0x7f32700771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48414/0 (socket says 172.21.2.117:48414) 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 -- 172.21.2.117:0/592098290 learned_addr learned my addr 172.21.2.117:0/592098290 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f32767d0700 1 --1- 172.21.2.117:0/592098290 >> v1:172.21.2.117:6789/0 conn(0x7f32700e51a0 0x7f32700d0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48664/0 (socket says 172.21.2.117:48664) 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 -- 172.21.2.117:0/592098290 >> v1:172.21.2.125:6789/0 conn(0x7f32700e4da0 legacy=0x7f32701b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 -- 172.21.2.117:0/592098290 >> v1:172.21.2.117:6789/0 conn(0x7f32700e51a0 legacy=0x7f32700d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 -- 172.21.2.117:0/592098290 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32701b08e0 con 0x7f327007da20 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3275fcf700 1 --2- 172.21.2.117:0/592098290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f327007da20 0x7f32700771b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f326c006700 tx=0x7f326c009b00).ready entity=mon.0 client_cookie=fc3c725557fea215 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3274fcd700 1 -- 172.21.2.117:0/592098290 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326c00c0c0 con 0x7f327007da20 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3274fcd700 1 -- 172.21.2.117:0/592098290 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c00c340 con 0x7f327007da20 2021-02-10T11:41:00.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f3274fcd700 1 -- 172.21.2.117:0/592098290 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f326c00c7a0 con 0x7f327007da20 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f32781b3700 1 -- 172.21.2.117:0/592098290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f327007da20 msgr2=0x7f32700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.768 7f32781b3700 1 --2- 172.21.2.117:0/592098290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f327007da20 0x7f32700771b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f326c006700 tx=0x7f326c009b00).stop 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.769 7f32781b3700 1 -- 172.21.2.117:0/592098290 shutdown_connections 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.769 7f32781b3700 1 --2- 172.21.2.117:0/592098290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f327007da20 0x7f32700771b0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.769 7f32781b3700 1 -- 172.21.2.117:0/592098290 >> 172.21.2.117:0/592098290 conn(0x7f32700ce450 msgr2=0x7f32700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.769 7f32781b3700 1 -- 172.21.2.117:0/592098290 shutdown_connections 2021-02-10T11:41:00.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.769 7f32781b3700 1 -- 172.21.2.117:0/592098290 wait complete. 2021-02-10T11:41:00.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 Processor -- start 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 -- start start 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32700ab1d0 0x7f32700ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32700abbf0 0x7f32700a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32700ac070 con 0x7f32700a4650 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32700c0960 con 0x7f32700ab1d0 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32781b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32701c22a0 con 0x7f32700abbf0 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.784 7f32767d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48422/0 (socket says 172.21.2.117:48422) 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 -- 172.21.2.117:0/3824728724 learned_addr learned my addr 172.21.2.117:0/3824728724 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 -- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32700abbf0 msgr2=0x7f32700a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32700abbf0 0x7f32700a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:00.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 -- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32700ab1d0 msgr2=0x7f32700ab5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32700ab1d0 0x7f32700ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 -- 172.21.2.117:0/3824728724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32700a5070 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32767d0700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f3268001b70 tx=0x7f326800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3268004930 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3268004c70 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32680072a0 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.785 7f32781b3700 1 -- 172.21.2.117:0/3824728724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32700a5070 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.786 7f32781b3700 1 -- 172.21.2.117:0/3824728724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32700a5070 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.786 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3268004930 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.786 7f32627fc700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 0x7f325801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.786 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3268032ca0 con 0x7f32700a4650 2021-02-10T11:41:00.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.787 7f3275fcf700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 0x7f325801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:00.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.787 7f3275fcf700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 0x7f325801ac10 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:00.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.817 7f32781b3700 1 -- 172.21.2.117:0/3824728724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32700ce450 con 0x7f32700a4650 2021-02-10T11:41:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.820 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3268004720 con 0x7f32700a4650 2021-02-10T11:41:00.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.950 7f32781b3700 1 -- 172.21.2.117:0/3824728724 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f32700ce450 con 0x7f32700a4650 2021-02-10T11:41:00.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.953 7f32627fc700 1 -- 172.21.2.117:0/3824728724 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3268004af0 con 0x7f32700a4650 2021-02-10T11:41:00.941 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:41:00.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 msgr2=0x7f325801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 0x7f325801ac10 crc :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 msgr2=0x7f32700a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f3268001b70 tx=0x7f326800ac50).stop 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 shutdown_connections 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3258018730 0x7f325801ac10 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32700a4650 0x7f32700a4a50 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32700abbf0 0x7f32700a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 --2- 172.21.2.117:0/3824728724 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32700ab1d0 0x7f32700ab5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:00.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 >> 172.21.2.117:0/3824728724 conn(0x7f32700ac370 msgr2=0x7f3270106960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:00.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 shutdown_connections 2021-02-10T11:41:00.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:00.959 7f32781b3700 1 -- 172.21.2.117:0/3824728724 wait complete. 2021-02-10T11:41:00.951 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:01.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.375 7fbace40d700 1 Processor -- start 2021-02-10T11:41:01.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.375 7fbace40d700 1 -- start start 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.376 7fbace40d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.376 7fbace40d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 0x7fbac80d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.376 7fbace40d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac8077a70 con 0x7fbac80e4d90 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.376 7fbace40d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80e5c10 con 0x7fbac80e5190 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.376 7fbace40d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80777d0 con 0x7fbac807da20 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbacca2a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 0x7fbac80d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbacca2a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 0x7fbac80d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:52810/0 (socket says 172.21.2.117:52810) 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46106/0 (socket says 172.21.2.117:46106) 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbacca2a700 1 -- 172.21.2.117:0/1528066140 learned_addr learned my addr 172.21.2.117:0/1528066140 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 -- 172.21.2.117:0/1528066140 >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 msgr2=0x7fbac80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 --2- 172.21.2.117:0/1528066140 >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 0x7fbac80d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 -- 172.21.2.117:0/1528066140 >> v1:172.21.2.125:6789/0 conn(0x7fbac80e4d90 legacy=0x7fbac81b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 -- 172.21.2.117:0/1528066140 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac81b08d0 con 0x7fbac807da20 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.377 7fbac7fff700 1 --2- 172.21.2.117:0/1528066140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fbab8006650 tx=0x7fbab8009a80).ready entity=mon.2 client_cookie=c2a96a6ff6736c8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbac6ffd700 1 -- 172.21.2.117:0/1528066140 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbab80009b0 con 0x7fbac807da20 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbac6ffd700 1 -- 172.21.2.117:0/1528066140 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab800c0c0 con 0x7fbac807da20 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbac6ffd700 1 -- 172.21.2.117:0/1528066140 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbab800c5e0 con 0x7fbac807da20 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 -- 172.21.2.117:0/1528066140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 msgr2=0x7fbac80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 --2- 172.21.2.117:0/1528066140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fbab8006650 tx=0x7fbab8009a80).stop 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 -- 172.21.2.117:0/1528066140 shutdown_connections 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 --2- 172.21.2.117:0/1528066140 >> v2:172.21.2.125:3300/0 conn(0x7fbac80e5190 0x7fbac80d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 --2- 172.21.2.117:0/1528066140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac807da20 0x7fbac80771b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.378 7fbace40d700 1 -- 172.21.2.117:0/1528066140 >> 172.21.2.117:0/1528066140 conn(0x7fbac80ce440 msgr2=0x7fbac80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.379 7fbace40d700 1 -- 172.21.2.117:0/1528066140 shutdown_connections 2021-02-10T11:41:01.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.379 7fbace40d700 1 -- 172.21.2.117:0/1528066140 wait complete. 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.393 7fbace40d700 1 Processor -- start 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.393 7fbace40d700 1 -- start start 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 0x7fbac8192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac81067a0 0x7fbac8106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 0x7fbac818cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac8193730 con 0x7fbac8192ba0 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac818d630 con 0x7fbac818cbd0 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbace40d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac818d850 con 0x7fbac81067a0 2021-02-10T11:41:01.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbacca2a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 0x7fbac818cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 0x7fbac8192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbacca2a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 0x7fbac818cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:52814/0 (socket says 172.21.2.117:52814) 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbacca2a700 1 -- 172.21.2.117:0/3160121497 learned_addr learned my addr 172.21.2.117:0/3160121497 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac77fe700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac81067a0 0x7fbac8106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 -- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac81067a0 msgr2=0x7fbac8106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac81067a0 0x7fbac8106ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 -- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 msgr2=0x7fbac818cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 0x7fbac818cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 -- 172.21.2.117:0/3160121497 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac818dc30 con 0x7fbac8192ba0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac7fff700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 0x7fbac8192fa0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7fbab800b530 tx=0x7fbab800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.394 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbab8000c90 con 0x7fbac8192ba0 2021-02-10T11:41:01.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8002030 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbab8002480 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbace40d700 1 -- 172.21.2.117:0/3160121497 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac818dc30 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbace40d700 1 -- 172.21.2.117:0/3160121497 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac818dc30 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbab8002210 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbac4ff9700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 0x7fbaa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.395 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbab80324a0 con 0x7fbac8192ba0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.396 7fbac77fe700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 0x7fbaa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.396 7fbac77fe700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 0x7fbaa801abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:01.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.426 7fbace40d700 1 -- 172.21.2.117:0/3160121497 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac8077a70 con 0x7fbac8192ba0 2021-02-10T11:41:01.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.428 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbab8000960 con 0x7fbac8192ba0 2021-02-10T11:41:01.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.555 7fbace40d700 1 -- 172.21.2.117:0/3160121497 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac8077a70 con 0x7fbac8192ba0 2021-02-10T11:41:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.556 7fbac4ff9700 1 -- 172.21.2.117:0/3160121497 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2093 (secure 0 0 0) 0x7fbab80021f0 con 0x7fbac8192ba0 2021-02-10T11:41:01.541 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 28 2021-02-10T11:41:01.542 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:01.542 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:40:57.258727","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5657":{"gid":5657,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/58270881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":58270881},{"type":"v1","addr":"172.21.2.117:6833","nonce":58270881}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 -- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 msgr2=0x7fbaa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 0x7fbaa801abc0 crc :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 -- 172.21.2.117:0/3160121497 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 msgr2=0x7fbac8192fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 0x7fbac8192fa0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0x7fbab800b530 tx=0x7fbab800b8d0).stop 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 -- 172.21.2.117:0/3160121497 shutdown_connections 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbac818cbd0 0x7fbac818cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbac81067a0 0x7fbac8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbaa80186c0 0x7fbaa801abc0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 --2- 172.21.2.117:0/3160121497 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbac8192ba0 0x7fbac8192fa0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 -- 172.21.2.117:0/3160121497 >> 172.21.2.117:0/3160121497 conn(0x7fbac819f510 msgr2=0x7fbac81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.558 7fbace40d700 1 -- 172.21.2.117:0/3160121497 shutdown_connections 2021-02-10T11:41:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:01.559 7fbace40d700 1 -- 172.21.2.117:0/3160121497 wait complete. 2021-02-10T11:41:01.753 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:55.253229', 'modified': '2021-02-10 11:40:57.258727', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5657}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5657': {'gid': 5657, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/58270881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 58270881}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 58270881}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:01.753 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:01.754 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5657, 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:41:02.152 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 6.9133256860000003, 2021-02-10T11:41:02.153 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 28, 2021-02-10T11:41:02.153 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 47, 2021-02-10T11:41:02.153 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 47, 2021-02-10T11:41:02.153 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 12.549223026 2021-02-10T11:41:02.153 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:02.156 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5657, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9133256860000003, "mdsmap_epoch": 28, "osdmap_epoch": 47, "osdmap_epoch_barrier": 47, "uptime": 12.549223026 } 2021-02-10T11:41:02.157 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:02.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.581 7f398a4b4700 1 Processor -- start 2021-02-10T11:41:02.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.581 7f398a4b4700 1 -- start start 2021-02-10T11:41:02.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f398a4b4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f398a4b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f398a4b4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3984077a70 con 0x7f39840e4d90 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f398a4b4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840777d0 con 0x7f398407da20 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f398a4b4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39841b08d0 con 0x7f39840e5190 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f3988ad1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f3983fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f3988ad1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46120/0 (socket says 172.21.2.117:46120) 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f3983fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:52822/0 (socket says 172.21.2.117:52822) 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f3988ad1700 1 -- 172.21.2.117:0/3117417038 learned_addr learned my addr 172.21.2.117:0/3117417038 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.582 7f39837fe700 1 --1- 172.21.2.117:0/3117417038 >> v1:172.21.2.125:6789/0 conn(0x7f39840e4d90 0x7f39841b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:41618/0 (socket says 172.21.2.117:41618) 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3988ad1700 1 -- 172.21.2.117:0/3117417038 >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 msgr2=0x7f39840771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3988ad1700 1 --2- 172.21.2.117:0/3117417038 >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3982ffd700 1 -- 172.21.2.117:0/3117417038 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512503039 0 0) 0x7f39780009b0 con 0x7f39840e4d90 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3988ad1700 1 -- 172.21.2.117:0/3117417038 >> v1:172.21.2.125:6789/0 conn(0x7f39840e4d90 legacy=0x7f39841b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3988ad1700 1 -- 172.21.2.117:0/3117417038 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398407de20 con 0x7f39840e5190 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3983fff700 1 --2- 172.21.2.117:0/3117417038 >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3988ad1700 1 --2- 172.21.2.117:0/3117417038 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f397c006aa0 tx=0x7f397c009f20).ready entity=mon.2 client_cookie=ffd031fa7bc57a72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3982ffd700 1 -- 172.21.2.117:0/3117417038 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397c000f90 con 0x7f39840e5190 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3982ffd700 1 -- 172.21.2.117:0/3117417038 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397c003030 con 0x7f39840e5190 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f3982ffd700 1 -- 172.21.2.117:0/3117417038 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397c00b7b0 con 0x7f39840e5190 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f398a4b4700 1 -- 172.21.2.117:0/3117417038 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 msgr2=0x7f3984079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.583 7f398a4b4700 1 --2- 172.21.2.117:0/3117417038 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f397c006aa0 tx=0x7f397c009f20).stop 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 -- 172.21.2.117:0/3117417038 shutdown_connections 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 --2- 172.21.2.117:0/3117417038 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39840e5190 0x7f3984079000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 --2- 172.21.2.117:0/3117417038 >> v2:172.21.2.125:3300/0 conn(0x7f398407da20 0x7f39840771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 -- 172.21.2.117:0/3117417038 >> 172.21.2.117:0/3117417038 conn(0x7f39840ce440 msgr2=0x7f39840e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 -- 172.21.2.117:0/3117417038 shutdown_connections 2021-02-10T11:41:02.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.584 7f398a4b4700 1 -- 172.21.2.117:0/3117417038 wait complete. 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.598 7f398a4b4700 1 Processor -- start 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 -- start start 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f398409dd40 0x7f39840b85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 0x7f398408f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398409d8a0 con 0x7f398408c080 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b8bc0 con 0x7f398409dd40 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f398a4b4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b8de0 con 0x7f398408caa0 2021-02-10T11:41:02.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f39837fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f39837fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48444/0 (socket says 172.21.2.117:48444) 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f39837fe700 1 -- 172.21.2.117:0/2148093401 learned_addr learned my addr 172.21.2.117:0/2148093401 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f3983fff700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f398409dd40 0x7f39840b85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.599 7f3988ad1700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 0x7f398408f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 -- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 msgr2=0x7f398408f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 0x7f398408f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 -- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f398409dd40 msgr2=0x7f39840b85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f398409dd40 0x7f39840b85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 -- 172.21.2.117:0/2148093401 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398408cea0 con 0x7f398408c080 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f3988ad1700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 0x7f398408f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f39837fe700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f3978001c60 tx=0x7f397800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f397800acc0 con 0x7f398408c080 2021-02-10T11:41:02.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978004140 con 0x7f398408c080 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39780045d0 con 0x7f398408c080 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398408cea0 con 0x7f398408c080 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.600 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398408cea0 con 0x7f398408c080 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.601 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3978000dd0 con 0x7f398408c080 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.601 7f3980ff9700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 0x7f396401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:02.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.601 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f39780328d0 con 0x7f398408c080 2021-02-10T11:41:02.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.601 7f3983fff700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 0x7f396401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:02.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.602 7f3983fff700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 0x7f396401abc0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:02.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.631 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3984077a70 con 0x7f398408c080 2021-02-10T11:41:02.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.634 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39780009b0 con 0x7f398408c080 2021-02-10T11:41:02.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.759 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3984077a70 con 0x7f398408c080 2021-02-10T11:41:02.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.760 7f3980ff9700 1 -- 172.21.2.117:0/2148093401 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14226 (secure 0 0 0) 0x7f3978000dd0 con 0x7f398408c080 2021-02-10T11:41:02.745 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:02.745 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":47,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:56.254993","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:53.223172","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:54.293749","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.761 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 msgr2=0x7f396401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.761 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 0x7f396401abc0 crc :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.761 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 msgr2=0x7f398408c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.761 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7f3978001c60 tx=0x7f397800a8d0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 shutdown_connections 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39640186c0 0x7f396401abc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f398408c080 0x7f398408c480 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f398408caa0 0x7f398408f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 --2- 172.21.2.117:0/2148093401 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f398409dd40 0x7f39840b85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 >> 172.21.2.117:0/2148093401 conn(0x7f398419f4c0 msgr2=0x7f3984094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:02.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 shutdown_connections 2021-02-10T11:41:02.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:02.762 7f398a4b4700 1 -- 172.21.2.117:0/2148093401 wait complete. 2021-02-10T11:41:02.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:53.223172', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:54.293749', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:41:02.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:03.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 Processor -- start 2021-02-10T11:41:03.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 -- start start 2021-02-10T11:41:03.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7aa4079000 0x7f7aa40e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7aa41b00f0 0x7f7aa41b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa4077a70 con 0x7f7aa407da20 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40e5c20 con 0x7f7aa4079000 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.183 7f7aa9656700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa40e53c0 con 0x7f7aa41b00f0 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7aa4079000 0x7f7aa40e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7aa4079000 0x7f7aa40e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48452/0 (socket says 172.21.2.117:48452) 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa32bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7aa407da20 0x7f7aa40771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48702/0 (socket says 172.21.2.117:48702) 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 -- 172.21.2.117:0/4197175377 learned_addr learned my addr 172.21.2.117:0/4197175377 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 -- 172.21.2.117:0/4197175377 >> v2:172.21.2.125:3300/0 conn(0x7f7aa41b00f0 msgr2=0x7f7aa41b04f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 --2- 172.21.2.117:0/4197175377 >> v2:172.21.2.125:3300/0 conn(0x7f7aa41b00f0 0x7f7aa41b04f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 -- 172.21.2.117:0/4197175377 >> v1:172.21.2.117:6789/0 conn(0x7f7aa407da20 legacy=0x7f7aa40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 -- 172.21.2.117:0/4197175377 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa4079400 con 0x7f7aa4079000 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa2abe700 1 --2- 172.21.2.117:0/4197175377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa4079000 0x7f7aa40e4da0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f7a94006700 tx=0x7f7a94009b80).ready entity=mon.0 client_cookie=679c7c9e07029232 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa22bd700 1 -- 172.21.2.117:0/4197175377 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a9400c0c0 con 0x7f7aa4079000 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa22bd700 1 -- 172.21.2.117:0/4197175377 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9400c440 con 0x7f7aa4079000 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa22bd700 1 -- 172.21.2.117:0/4197175377 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a9400c920 con 0x7f7aa4079000 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa9656700 1 -- 172.21.2.117:0/4197175377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa4079000 msgr2=0x7f7aa40e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.184 7f7aa9656700 1 --2- 172.21.2.117:0/4197175377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa4079000 0x7f7aa40e4da0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f7a94006700 tx=0x7f7a94009b80).stop 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 -- 172.21.2.117:0/4197175377 shutdown_connections 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 --2- 172.21.2.117:0/4197175377 >> v2:172.21.2.125:3300/0 conn(0x7f7aa41b00f0 0x7f7aa41b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 --2- 172.21.2.117:0/4197175377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa4079000 0x7f7aa40e4da0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 -- 172.21.2.117:0/4197175377 >> 172.21.2.117:0/4197175377 conn(0x7f7aa40ce450 msgr2=0x7f7aa40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:03.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 -- 172.21.2.117:0/4197175377 shutdown_connections 2021-02-10T11:41:03.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.185 7f7aa9656700 1 -- 172.21.2.117:0/4197175377 wait complete. 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 Processor -- start 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 -- start start 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7aa4093960 0x7f7aa4093d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7aa40ac730 0x7f7aa40acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40adb40 con 0x7f7aa40ad150 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40ad550 con 0x7f7aa40ac730 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa9656700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40ad730 con 0x7f7aa4093960 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa3ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa3ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48458/0 (socket says 172.21.2.117:48458) 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.200 7f7aa3ac0700 1 -- 172.21.2.117:0/2152428181 learned_addr learned my addr 172.21.2.117:0/2152428181 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 -- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7aa4093960 msgr2=0x7f7aa4093d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7aa4093960 0x7f7aa4093d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 -- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7aa40ac730 msgr2=0x7f7aa40acb30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7aa40ac730 0x7f7aa40acb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 -- 172.21.2.117:0/2152428181 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40ae6e0 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa3ac0700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f7a9c009960 tx=0x7f7a9c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a9c003030 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c001310 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a9c001830 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40ae6e0 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.201 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa40aec70 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.202 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a9c0014f0 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.202 7f7a9bfff700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 0x7f7a8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.202 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7a9c032a90 con 0x7f7aa40ad150 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.203 7f7aa32bf700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 0x7f7a8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.203 7f7aa32bf700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 0x7f7a8801abc0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.233 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa4103370 con 0x7f7aa40ad150 2021-02-10T11:41:03.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.236 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a9c021d80 con 0x7f7aa40ad150 2021-02-10T11:41:03.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.364 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f7aa4077a70 con 0x7f7aa40ad150 2021-02-10T11:41:03.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.365 7f7a9bfff700 1 -- 172.21.2.117:0/2152428181 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v47) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7a9c021d80 con 0x7f7aa40ad150 2021-02-10T11:41:03.350 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:03.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 msgr2=0x7f7a8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 0x7f7a8801abc0 crc :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 msgr2=0x7f7aa40ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f7a9c009960 tx=0x7f7a9c00bcf0).stop 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 shutdown_connections 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a880186c0 0x7f7a8801abc0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7aa40ad150 0x7f7aa40ae0c0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7aa40ac730 0x7f7aa40acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 --2- 172.21.2.117:0/2152428181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7aa4093960 0x7f7aa4093d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 >> 172.21.2.117:0/2152428181 conn(0x7f7aa40bb5d0 msgr2=0x7f7aa4190400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:03.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 shutdown_connections 2021-02-10T11:41:03.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.367 7f7aa9656700 1 -- 172.21.2.117:0/2152428181 wait complete. 2021-02-10T11:41:03.360 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:41:03.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.788 7fb190aea700 1 Processor -- start 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb190aea700 1 -- start start 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb190aea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb18c07da20 0x7fb18c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb190aea700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c0d1d20 con 0x7fb18c0e51a0 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb190aea700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c077a70 con 0x7fb18c0e4da0 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb190aea700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18c0777d0 con 0x7fb18c07da20 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb18a85e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb18c07da20 0x7fb18c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb18a85e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb18c07da20 0x7fb18c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48464/0 (socket says 172.21.2.117:48464) 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.789 7fb18a85e700 1 -- 172.21.2.117:0/2566321159 learned_addr learned my addr 172.21.2.117:0/2566321159 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18a85e700 1 -- 172.21.2.117:0/2566321159 >> v1:172.21.2.125:6789/0 conn(0x7fb18c0e4da0 legacy=0x7fb18c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18a85e700 1 -- 172.21.2.117:0/2566321159 >> v1:172.21.2.117:6789/0 conn(0x7fb18c0e51a0 legacy=0x7fb18c0d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18a85e700 1 -- 172.21.2.117:0/2566321159 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c1b08e0 con 0x7fb18c07da20 2021-02-10T11:41:03.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18a85e700 1 --2- 172.21.2.117:0/2566321159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c07da20 0x7fb18c0771b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7fb174006700 tx=0x7fb174009b00).ready entity=mon.0 client_cookie=a4539b29c893cc3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18985c700 1 -- 172.21.2.117:0/2566321159 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb17400c0c0 con 0x7fb18c07da20 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18985c700 1 -- 172.21.2.117:0/2566321159 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17400c340 con 0x7fb18c07da20 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb18985c700 1 -- 172.21.2.117:0/2566321159 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb17400c820 con 0x7fb18c07da20 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb190aea700 1 -- 172.21.2.117:0/2566321159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c07da20 msgr2=0x7fb18c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.790 7fb190aea700 1 --2- 172.21.2.117:0/2566321159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c07da20 0x7fb18c0771b0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7fb174006700 tx=0x7fb174009b00).stop 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.791 7fb190aea700 1 -- 172.21.2.117:0/2566321159 shutdown_connections 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.791 7fb190aea700 1 --2- 172.21.2.117:0/2566321159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c07da20 0x7fb18c0771b0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.791 7fb190aea700 1 -- 172.21.2.117:0/2566321159 >> 172.21.2.117:0/2566321159 conn(0x7fb18c0ce450 msgr2=0x7fb18c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.791 7fb190aea700 1 -- 172.21.2.117:0/2566321159 shutdown_connections 2021-02-10T11:41:03.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.791 7fb190aea700 1 -- 172.21.2.117:0/2566321159 wait complete. 2021-02-10T11:41:03.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 Processor -- start 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 -- start start 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb18c0933f0 0x7fb18c0937f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb18c093e10 0x7fb18c106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c107a70 con 0x7fb18c107050 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c0c2c60 con 0x7fb18c0933f0 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.806 7fb190aea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c0c2e80 con 0x7fb18c093e10 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48472/0 (socket says 172.21.2.117:48472) 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 -- 172.21.2.117:0/3814090060 learned_addr learned my addr 172.21.2.117:0/3814090060 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 -- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb18c093e10 msgr2=0x7fb18c106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb18c093e10 0x7fb18c106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 -- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb18c0933f0 msgr2=0x7fb18c0937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb18c0933f0 0x7fb18c0937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 -- 172.21.2.117:0/3814090060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0c30a0 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb18b05f700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7fb184009980 tx=0x7fb18400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb184003020 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb184005a80 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb184006030 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb190aea700 1 -- 172.21.2.117:0/3814090060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c0c30a0 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.807 7fb190aea700 1 -- 172.21.2.117:0/3814090060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c0c30a0 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.808 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb184006030 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.808 7fb1837fe700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 0x7fb17001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.808 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb184032a70 con 0x7fb18c107050 2021-02-10T11:41:03.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.808 7fb18a85e700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 0x7fb17001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:03.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.809 7fb18a85e700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 0x7fb17001abc0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:03.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.839 7fb190aea700 1 -- 172.21.2.117:0/3814090060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c107e30 con 0x7fb18c107050 2021-02-10T11:41:03.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.841 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb184005870 con 0x7fb18c107050 2021-02-10T11:41:03.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.970 7fb190aea700 1 -- 172.21.2.117:0/3814090060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb18c107e30 con 0x7fb18c107050 2021-02-10T11:41:03.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.970 7fb1837fe700 1 -- 172.21.2.117:0/3814090060 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v47) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb184006030 con 0x7fb18c107050 2021-02-10T11:41:03.956 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 msgr2=0x7fb17001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 0x7fb17001abc0 crc :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 msgr2=0x7fb18c107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7fb184009980 tx=0x7fb18400bcd0).stop 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 shutdown_connections 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1700186c0 0x7fb17001abc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb18c107050 0x7fb18c107450 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:03.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb18c093e10 0x7fb18c106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 --2- 172.21.2.117:0/3814090060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb18c0933f0 0x7fb18c0937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:03.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 >> 172.21.2.117:0/3814090060 conn(0x7fb18c1116a0 msgr2=0x7fb18c0af170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:03.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 shutdown_connections 2021-02-10T11:41:03.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:03.972 7fb190aea700 1 -- 172.21.2.117:0/3814090060 wait complete. 2021-02-10T11:41:03.965 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 Processor -- start 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 -- start start 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f4007da20 0x7f4f400771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4f400e4da0 0x7f4f401b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f40077a70 con 0x7f4f400e51a0 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f400e5c20 con 0x7f4f400e4da0 2021-02-10T11:41:04.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f46c8e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400777d0 con 0x7f4f4007da20 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.394 7f4f3ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4f400e4da0 0x7f4f401b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4f400e4da0 0x7f4f401b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48482/0 (socket says 172.21.2.117:48482) 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 -- 172.21.2.117:0/1624986295 learned_addr learned my addr 172.21.2.117:0/1624986295 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 -- 172.21.2.117:0/1624986295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f4007da20 msgr2=0x7f4f400771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 --2- 172.21.2.117:0/1624986295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f4007da20 0x7f4f400771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 -- 172.21.2.117:0/1624986295 >> v1:172.21.2.125:6789/0 conn(0x7f4f400e51a0 legacy=0x7f4f40077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 -- 172.21.2.117:0/1624986295 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4007de20 con 0x7f4f400e4da0 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3ffff700 1 --2- 172.21.2.117:0/1624986295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400e4da0 0x7f4f401b00f0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f4f30006700 tx=0x7f4f30009b80).ready entity=mon.0 client_cookie=8b3b92663b24ddd8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3f7fe700 1 -- 172.21.2.117:0/1624986295 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f3000c0c0 con 0x7f4f400e4da0 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3f7fe700 1 -- 172.21.2.117:0/1624986295 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3000c340 con 0x7f4f400e4da0 2021-02-10T11:41:04.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f3f7fe700 1 -- 172.21.2.117:0/1624986295 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f3000c860 con 0x7f4f400e4da0 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f46c8e700 1 -- 172.21.2.117:0/1624986295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400e4da0 msgr2=0x7f4f401b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.395 7f4f46c8e700 1 --2- 172.21.2.117:0/1624986295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400e4da0 0x7f4f401b00f0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f4f30006700 tx=0x7f4f30009b80).stop 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 -- 172.21.2.117:0/1624986295 shutdown_connections 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 --2- 172.21.2.117:0/1624986295 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400e4da0 0x7f4f401b00f0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 --2- 172.21.2.117:0/1624986295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f4007da20 0x7f4f400771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 -- 172.21.2.117:0/1624986295 >> 172.21.2.117:0/1624986295 conn(0x7f4f400ce450 msgr2=0x7f4f400e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 -- 172.21.2.117:0/1624986295 shutdown_connections 2021-02-10T11:41:04.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.396 7f4f46c8e700 1 -- 172.21.2.117:0/1624986295 wait complete. 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.410 7f4f46c8e700 1 Processor -- start 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.410 7f4f46c8e700 1 -- start start 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f4018a0e0 0x7f4f4018a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f400b79c0 0x7f4f400b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f40189be0 con 0x7f4f400a9890 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400b83e0 con 0x7f4f4018a0e0 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f46c8e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f400b8600 con 0x7f4f400b79c0 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48486/0 (socket says 172.21.2.117:48486) 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 -- 172.21.2.117:0/1469586997 learned_addr learned my addr 172.21.2.117:0/1469586997 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 -- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f400b79c0 msgr2=0x7f4f400b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f400b79c0 0x7f4f400b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 -- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f4018a0e0 msgr2=0x7f4f4018a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f4018a0e0 0x7f4f4018a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 -- 172.21.2.117:0/1469586997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f400b8820 con 0x7f4f400a9890 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.411 7f4f44aaa700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f4f2800b5c0 tx=0x7f4f2800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f2800bd50 con 0x7f4f400a9890 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f28006030 con 0x7f4f400a9890 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f280054d0 con 0x7f4f400a9890 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f400b8820 con 0x7f4f400a9890 2021-02-10T11:41:04.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f400b8db0 con 0x7f4f400a9890 2021-02-10T11:41:04.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f28006210 con 0x7f4f400a9890 2021-02-10T11:41:04.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 0x7f4f2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.412 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4f280326e0 con 0x7f4f400a9890 2021-02-10T11:41:04.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.413 7f4f3ffff700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 0x7f4f2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.413 7f4f3ffff700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 0x7f4f2401abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:04.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.444 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f401032d0 con 0x7f4f400a9890 2021-02-10T11:41:04.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.446 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f28000940 con 0x7f4f400a9890 2021-02-10T11:41:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.572 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4f40077a70 con 0x7f4f400a9890 2021-02-10T11:41:04.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.573 7f4f3d7fa700 1 -- 172.21.2.117:0/1469586997 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v47) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4f2800bd50 con 0x7f4f400a9890 2021-02-10T11:41:04.558 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:04.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.574 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 msgr2=0x7f4f2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.574 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 0x7f4f2401abc0 crc :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.574 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 msgr2=0x7f4f400a9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.574 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f4f2800b5c0 tx=0x7f4f2800b960).stop 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 shutdown_connections 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f400b79c0 0x7f4f400b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f4018a0e0 0x7f4f4018a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f240186c0 0x7f4f2401abc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 --2- 172.21.2.117:0/1469586997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f400a9890 0x7f4f400a9c90 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 >> 172.21.2.117:0/1469586997 conn(0x7f4f400b56c0 msgr2=0x7f4f400a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:04.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 shutdown_connections 2021-02-10T11:41:04.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.575 7f4f46c8e700 1 -- 172.21.2.117:0/1469586997 wait complete. 2021-02-10T11:41:04.568 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:41:04.568 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.992 7f64a8f5c700 1 Processor -- start 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.992 7f64a8f5c700 1 -- start start 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a407da20 0x7f64a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a40e4d90 0x7f64a41b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f64a40e5190 0x7f64a4077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a4077a70 con 0x7f64a407da20 2021-02-10T11:41:04.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a40777d0 con 0x7f64a40e5190 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a8f5c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a407de20 con 0x7f64a40e4d90 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a407da20 0x7f64a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f64a407da20 0x7f64a40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48492/0 (socket says 172.21.2.117:48492) 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 -- 172.21.2.117:0/1736092769 learned_addr learned my addr 172.21.2.117:0/1736092769 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 -- 172.21.2.117:0/1736092769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a40e4d90 msgr2=0x7f64a41b00e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 --2- 172.21.2.117:0/1736092769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a40e4d90 0x7f64a41b00e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 -- 172.21.2.117:0/1736092769 >> v2:172.21.2.125:3300/0 conn(0x7f64a40e5190 msgr2=0x7f64a4077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 --2- 172.21.2.117:0/1736092769 >> v2:172.21.2.125:3300/0 conn(0x7f64a40e5190 0x7f64a4077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.993 7f64a2b13700 1 -- 172.21.2.117:0/1736092769 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a41b0700 con 0x7f64a407da20 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a2b13700 1 --2- 172.21.2.117:0/1736092769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a407da20 0x7f64a40771b0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f648c006700 tx=0x7f648c009b80).ready entity=mon.0 client_cookie=468cb00662bf42c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a1b11700 1 -- 172.21.2.117:0/1736092769 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f648c00c0c0 con 0x7f64a407da20 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a1b11700 1 -- 172.21.2.117:0/1736092769 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c00c340 con 0x7f64a407da20 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a1b11700 1 -- 172.21.2.117:0/1736092769 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f648c00c860 con 0x7f64a407da20 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a8f5c700 1 -- 172.21.2.117:0/1736092769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a407da20 msgr2=0x7f64a40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.994 7f64a8f5c700 1 --2- 172.21.2.117:0/1736092769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a407da20 0x7f64a40771b0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0x7f648c006700 tx=0x7f648c009b80).stop 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 -- 172.21.2.117:0/1736092769 shutdown_connections 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 --2- 172.21.2.117:0/1736092769 >> v2:172.21.2.125:3300/0 conn(0x7f64a40e5190 0x7f64a4077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 --2- 172.21.2.117:0/1736092769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a40e4d90 0x7f64a41b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 --2- 172.21.2.117:0/1736092769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a407da20 0x7f64a40771b0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 -- 172.21.2.117:0/1736092769 >> 172.21.2.117:0/1736092769 conn(0x7f64a40ce440 msgr2=0x7f64a40e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 -- 172.21.2.117:0/1736092769 shutdown_connections 2021-02-10T11:41:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:04.995 7f64a8f5c700 1 -- 172.21.2.117:0/1736092769 wait complete. 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.009 7f64a8f5c700 1 Processor -- start 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.009 7f64a8f5c700 1 -- start start 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a41a5640 0x7f64a41a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a41a1fd0 0x7f64a419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a41a23d0 con 0x7f64a419c030 2021-02-10T11:41:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a41a5a40 con 0x7f64a41a1fd0 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a8f5c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a408a1b0 con 0x7f64a41a5640 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48502/0 (socket says 172.21.2.117:48502) 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 -- 172.21.2.117:0/1651530572 learned_addr learned my addr 172.21.2.117:0/1651530572 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a2312700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a41a1fd0 0x7f64a419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a2b13700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a41a5640 0x7f64a41a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 -- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a41a5640 msgr2=0x7f64a41a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a41a5640 0x7f64a41a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 -- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a41a1fd0 msgr2=0x7f64a419ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a41a1fd0 0x7f64a419ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 -- 172.21.2.117:0/1651530572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a419ca50 con 0x7f64a419c030 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f64a3314700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f649c009960 tx=0x7f649c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:04.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f649c000d10 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.010 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c003030 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f649c001590 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a419ca50 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a419ca50 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f649c000d10 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f649b7fe700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 0x7f648801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f649c0328f0 con 0x7f64a419c030 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.011 7f64a2b13700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 0x7f648801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.012 7f64a2b13700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 0x7f648801abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:05.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.043 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a41a23d0 con 0x7f64a419c030 2021-02-10T11:41:05.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.045 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f649c00be30 con 0x7f64a419c030 2021-02-10T11:41:05.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.172 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a41a23d0 con 0x7f64a419c030 2021-02-10T11:41:05.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.173 7f649b7fe700 1 -- 172.21.2.117:0/1651530572 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f649c000d10 con 0x7f64a419c030 2021-02-10T11:41:05.158 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 28 2021-02-10T11:41:05.158 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:05.158 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:40:57.258727","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5657":{"gid":5657,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/58270881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":58270881},{"type":"v1","addr":"172.21.2.117:6833","nonce":58270881}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:41:05.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 msgr2=0x7f648801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 0x7f648801abc0 crc :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 msgr2=0x7f64a419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f649c009960 tx=0x7f649c00bcf0).stop 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 shutdown_connections 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f64880186c0 0x7f648801abc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f64a419c030 0x7f64a419c430 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f64a41a1fd0 0x7f64a419ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 --2- 172.21.2.117:0/1651530572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f64a41a5640 0x7f64a41a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 >> 172.21.2.117:0/1651530572 conn(0x7f64a4191110 msgr2=0x7f64a419aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 shutdown_connections 2021-02-10T11:41:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:05.175 7f64a8f5c700 1 -- 172.21.2.117:0/1651530572 wait complete. 2021-02-10T11:41:05.370 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:55.253229', 'modified': '2021-02-10 11:40:57.258727', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5657}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5657': {'gid': 5657, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/58270881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 58270881}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 58270881}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:05.370 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:05.371 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:41:05.767 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5657, 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 10.528553764, 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 28, 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 47, 2021-02-10T11:41:05.768 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 47, 2021-02-10T11:41:05.769 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 16.164452313999998 2021-02-10T11:41:05.769 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:05.772 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5657, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.528553764, "mdsmap_epoch": 28, "osdmap_epoch": 47, "osdmap_epoch_barrier": 47, "uptime": 16.164452313999998 } 2021-02-10T11:41:05.772 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:41:05.772 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:41:05.772 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:41:05.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:05.772 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:06.106 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:06.112 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:06.152 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:06.452 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:06.452 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:06.796 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:06.796 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:06.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:06.797 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:07.131 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:41:07.131 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:07.134 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:07.477 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:07.478 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:07.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[125263]: starting ceph client 2021-02-10T11:41:07.543 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:41:07.558 7f72979d7f80 -1 init, newargv = 0x5598faf46f10 newargc=9 2021-02-10T11:41:07.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[125263]: starting fuse 2021-02-10T11:41:07.820 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:07.820 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:07.821 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:07.821 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:08.156 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:41:08.156 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:41:08.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:08.156 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:41:08.157 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:41:08.158 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:41:08.518 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.125263.asok 2021-02-10T11:41:08.521 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.125263.asok 2021-02-10T11:41:08.521 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:08.521 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.125263.asok status 2021-02-10T11:41:08.910 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:08.910 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:41:08.910 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:41:08.910 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "125263", 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:41:08.911 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 7773, 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 7773 2021-02-10T11:41:08.912 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4086418349 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:08.913 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4086418349 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.7773 172.21.2.117:0/4086418349", 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/4086418349", 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 28, 2021-02-10T11:41:08.914 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 47, 2021-02-10T11:41:08.915 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:41:08.915 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:41:08.915 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:08.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:08.916 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:08.916 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:09.251 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:41:09.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:09.251 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:09.251 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:09.596 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:10.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.020 7fbf9cf69700 1 Processor -- start 2021-02-10T11:41:10.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.020 7fbf9cf69700 1 -- start start 2021-02-10T11:41:10.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf9807da20 0x7fbf980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf980e4d90 0x7fbf981b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980e5190 0x7fbf98077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf98077a70 con 0x7fbf980e4d90 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf980777d0 con 0x7fbf9807da20 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf9cf69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9807de20 con 0x7fbf980e5190 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf980e4d90 0x7fbf981b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf980e4d90 0x7fbf981b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48522/0 (socket says 172.21.2.117:48522) 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 -- 172.21.2.117:0/396624187 learned_addr learned my addr 172.21.2.117:0/396624187 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96b13700 1 --2- 172.21.2.117:0/396624187 >> v2:172.21.2.125:3300/0 conn(0x7fbf9807da20 0x7fbf980771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 -- 172.21.2.117:0/396624187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980e5190 msgr2=0x7fbf98077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 --2- 172.21.2.117:0/396624187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980e5190 0x7fbf98077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 -- 172.21.2.117:0/396624187 >> v2:172.21.2.125:3300/0 conn(0x7fbf9807da20 msgr2=0x7fbf980771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 --2- 172.21.2.117:0/396624187 >> v2:172.21.2.125:3300/0 conn(0x7fbf9807da20 0x7fbf980771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 -- 172.21.2.117:0/396624187 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf981b0700 con 0x7fbf980e4d90 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf96312700 1 --2- 172.21.2.117:0/396624187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980e4d90 0x7fbf981b00e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7fbf88006700 tx=0x7fbf88009b80).ready entity=mon.0 client_cookie=dfb93e7376849ddc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.021 7fbf95b11700 1 -- 172.21.2.117:0/396624187 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf8800c0c0 con 0x7fbf980e4d90 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf95b11700 1 -- 172.21.2.117:0/396624187 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8800c340 con 0x7fbf980e4d90 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf95b11700 1 -- 172.21.2.117:0/396624187 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf8800c820 con 0x7fbf980e4d90 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 -- 172.21.2.117:0/396624187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980e4d90 msgr2=0x7fbf981b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 --2- 172.21.2.117:0/396624187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980e4d90 0x7fbf981b00e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7fbf88006700 tx=0x7fbf88009b80).stop 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 -- 172.21.2.117:0/396624187 shutdown_connections 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 --2- 172.21.2.117:0/396624187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980e5190 0x7fbf98077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 --2- 172.21.2.117:0/396624187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980e4d90 0x7fbf981b00e0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 --2- 172.21.2.117:0/396624187 >> v2:172.21.2.125:3300/0 conn(0x7fbf9807da20 0x7fbf980771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.022 7fbf9cf69700 1 -- 172.21.2.117:0/396624187 >> 172.21.2.117:0/396624187 conn(0x7fbf980ce440 msgr2=0x7fbf980e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.023 7fbf9cf69700 1 -- 172.21.2.117:0/396624187 shutdown_connections 2021-02-10T11:41:10.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.023 7fbf9cf69700 1 -- 172.21.2.117:0/396624187 wait complete. 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.037 7fbf9cf69700 1 Processor -- start 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.037 7fbf9cf69700 1 -- start start 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980937e0 0x7fbf98092fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf9818f250 0x7fbf980b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf98187f30 con 0x7fbf980b1bd0 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf98187be0 con 0x7fbf9818f250 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf9cf69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9818f650 con 0x7fbf980937e0 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48530/0 (socket says 172.21.2.117:48530) 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 -- 172.21.2.117:0/1783381015 learned_addr learned my addr 172.21.2.117:0/1783381015 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf96312700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf9818f250 0x7fbf980b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf96b13700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980937e0 0x7fbf98092fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 -- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980937e0 msgr2=0x7fbf98092fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980937e0 0x7fbf98092fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 -- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf9818f250 msgr2=0x7fbf980b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf9818f250 0x7fbf980b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.038 7fbf97314700 1 -- 172.21.2.117:0/1783381015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980b1fd0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf97314700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7fbf90009980 tx=0x7fbf9000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf90005af0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf90005df0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf980b1fd0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf98188ad0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.039 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf900062b0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.040 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf90005af0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.040 7fbf8f7fe700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 0x7fbf7c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.040 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf90032cc0 con 0x7fbf980b1bd0 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.040 7fbf96b13700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 0x7fbf7c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.040 7fbf96b13700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 0x7fbf7c01ac10 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.070 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf981b4f30 con 0x7fbf980b1bd0 2021-02-10T11:41:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.073 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf90006570 con 0x7fbf980b1bd0 2021-02-10T11:41:10.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.198 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf98077a70 con 0x7fbf980b1bd0 2021-02-10T11:41:10.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.199 7fbf8f7fe700 1 -- 172.21.2.117:0/1783381015 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14226 (secure 0 0 0) 0x7fbf90006f20 con 0x7fbf980b1bd0 2021-02-10T11:41:10.183 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:10.184 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":47,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:40:56.254993","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:40:53.223172","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":9,"pool_name":"cephfs_data","create_time":"2021-02-10 11:40:54.293749","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:10.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 msgr2=0x7fbf7c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 0x7fbf7c01ac10 crc :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 msgr2=0x7fbf981884b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7fbf90009980 tx=0x7fbf9000bcd0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 shutdown_connections 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf7c018730 0x7fbf7c01ac10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf980b1bd0 0x7fbf981884b0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf9818f250 0x7fbf980b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 --2- 172.21.2.117:0/1783381015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf980937e0 0x7fbf98092fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 >> 172.21.2.117:0/1783381015 conn(0x7fbf981ace70 msgr2=0x7fbf981af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 shutdown_connections 2021-02-10T11:41:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.200 7fbf9cf69700 1 -- 172.21.2.117:0/1783381015 wait complete. 2021-02-10T11:41:10.194 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:40:53.223172', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:40:54.293749', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:41:10.194 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 Processor -- start 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 -- start start 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f841807da20 0x7f84180771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f84180e51a0 0x7f84180d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8418077a70 con 0x7f84180e4da0 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84180e5c20 con 0x7f84180e51a0 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.624 7f841d10c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84180777d0 con 0x7f841807da20 2021-02-10T11:41:10.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f841659c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84180e4da0 0x7f84181b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48784/0 (socket says 172.21.2.117:48784) 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f841659c700 1 -- 172.21.2.117:0/631304428 learned_addr learned my addr 172.21.2.117:0/631304428 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f841759e700 1 --2- 172.21.2.117:0/631304428 >> v2:172.21.2.125:3300/0 conn(0x7f84180e51a0 0x7f84180d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f841807da20 0x7f84180771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741738568 0 0) 0x7f8408001180 con 0x7f84180e4da0 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8404002fe0 con 0x7f84180e4da0 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2232113433 0 0) 0x7f8408001180 con 0x7f84180e4da0 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f841807da20 msgr2=0x7f84180771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f841807da20 0x7f84180771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 >> v2:172.21.2.125:3300/0 conn(0x7f84180e51a0 msgr2=0x7f84180d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 --2- 172.21.2.117:0/631304428 >> v2:172.21.2.125:3300/0 conn(0x7f84180e51a0 0x7f84180d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.625 7f8415d9b700 1 -- 172.21.2.117:0/631304428 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84181b08e0 con 0x7f84180e4da0 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8408001680 con 0x7f84180e4da0 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f841759e700 1 --2- 172.21.2.117:0/631304428 >> v2:172.21.2.125:3300/0 conn(0x7f84180e51a0 0x7f84180d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f841807da20 0x7f84180771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 -- 172.21.2.117:0/631304428 >> v1:172.21.2.117:6789/0 conn(0x7f84180e4da0 legacy=0x7f84181b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8404047aa0 0x7f8404049f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f840404a810 0x7f840404cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f840404d5b0 0x7f840404faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 -- 172.21.2.117:0/631304428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84040502d0 con 0x7f840404a810 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 -- 172.21.2.117:0/631304428 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84040504f0 con 0x7f8404047aa0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8415d9b700 1 -- 172.21.2.117:0/631304428 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8404050710 con 0x7f840404d5b0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f841659c700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f840404a810 0x7f840404cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8404047aa0 0x7f8404049f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f841759e700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f840404d5b0 0x7f840404faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 -- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f840404d5b0 msgr2=0x7f840404faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f840404d5b0 0x7f840404faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 -- 172.21.2.117:0/631304428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f840404a810 msgr2=0x7f840404cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f840404a810 0x7f840404cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.626 7f8416d9d700 1 -- 172.21.2.117:0/631304428 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040509b0 con 0x7f8404047aa0 2021-02-10T11:41:10.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f8416d9d700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8404047aa0 0x7f8404049f70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8400006f80 tx=0x7f8400009e20).ready entity=mon.1 client_cookie=1e3b2f9f867ac344 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8400006240 con 0x7f8404047aa0 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400006540 con 0x7f8404047aa0 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f8415d9b700 1 -- 172.21.2.117:0/631304428 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84000058e0 con 0x7f8404047aa0 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f841d10c700 1 -- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8404047aa0 msgr2=0x7f8404049f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f841d10c700 1 --2- 172.21.2.117:0/631304428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8404047aa0 0x7f8404049f70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8400006f80 tx=0x7f8400009e20).stop 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.627 7f841759e700 1 -- 172.21.2.117:0/631304428 reap_dead start 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.628 7f841d10c700 1 -- 172.21.2.117:0/631304428 shutdown_connections 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.628 7f841d10c700 1 -- 172.21.2.117:0/631304428 >> 172.21.2.117:0/631304428 conn(0x7f84180ce450 msgr2=0x7f84180e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.628 7f841d10c700 1 -- 172.21.2.117:0/631304428 shutdown_connections 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.628 7f841d10c700 1 -- 172.21.2.117:0/631304428 wait complete. 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.642 7f841d10c700 1 Processor -- start 2021-02-10T11:41:10.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.642 7f841d10c700 1 -- start start 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84180a7c20 0x7f8418093200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8418092650 0x7f8418093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418092a50 con 0x7f84180a7820 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418092c80 con 0x7f8418092650 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841d10c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84180a8340 con 0x7f84180a7c20 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48544/0 (socket says 172.21.2.117:48544) 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 -- 172.21.2.117:0/2455898055 learned_addr learned my addr 172.21.2.117:0/2455898055 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:10.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841759e700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8418092650 0x7f8418093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f841659c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84180a7c20 0x7f8418093200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 -- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84180a7c20 msgr2=0x7f8418093200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84180a7c20 0x7f8418093200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 -- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8418092650 msgr2=0x7f8418093820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8418092650 0x7f8418093820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.643 7f8416d9d700 1 -- 172.21.2.117:0/2455898055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84180a8520 con 0x7f84180a7820 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f8416d9d700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f840000b960 tx=0x7f840000bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f840000be40 con 0x7f84180a7820 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400000cc0 con 0x7f84180a7820 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f840000c2f0 con 0x7f84180a7820 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f841d10c700 1 -- 172.21.2.117:0/2455898055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84180a8520 con 0x7f84180a7820 2021-02-10T11:41:10.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.644 7f841d10c700 1 -- 172.21.2.117:0/2455898055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84180a8520 con 0x7f84180a7820 2021-02-10T11:41:10.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.645 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8400009730 con 0x7f84180a7820 2021-02-10T11:41:10.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.645 7f840f7fe700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 0x7f83fc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:10.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.645 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8400032d00 con 0x7f84180a7820 2021-02-10T11:41:10.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.645 7f841659c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 0x7f83fc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:10.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.646 7f841659c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 0x7f83fc01abd0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:10.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.674 7f841d10c700 1 -- 172.21.2.117:0/2455898055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8418092a50 con 0x7f84180a7820 2021-02-10T11:41:10.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.677 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8400001010 con 0x7f84180a7820 2021-02-10T11:41:10.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.806 7f841d10c700 1 -- 172.21.2.117:0/2455898055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f8418092a50 con 0x7f84180a7820 2021-02-10T11:41:10.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.806 7f840f7fe700 1 -- 172.21.2.117:0/2455898055 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v47) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8400009730 con 0x7f84180a7820 2021-02-10T11:41:10.791 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:10.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 msgr2=0x7f83fc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 0x7f83fc01abd0 crc :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 msgr2=0x7f8418092030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f840000b960 tx=0x7f840000bd00).stop 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 shutdown_connections 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8418092650 0x7f8418093820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84180a7c20 0x7f8418093200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83fc0186d0 0x7f83fc01abd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 --2- 172.21.2.117:0/2455898055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84180a7820 0x7f8418092030 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 >> 172.21.2.117:0/2455898055 conn(0x7f84180a9210 msgr2=0x7f84180a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 shutdown_connections 2021-02-10T11:41:10.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:10.808 7f841d10c700 1 -- 172.21.2.117:0/2455898055 wait complete. 2021-02-10T11:41:10.801 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:41:11.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 Processor -- start 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 -- start start 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f009807da20 0x7f00980771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f00980e4da0 0x7f00981b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f00980e51a0 0x7f0098077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00980e5c20 con 0x7f00980e4da0 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0098077a70 con 0x7f00980e51a0 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.229 7f009e4b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00980777d0 con 0x7f009807da20 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f00980e4da0 0x7f00981b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f00980e4da0 0x7f00981b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48552/0 (socket says 172.21.2.117:48552) 2021-02-10T11:41:11.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 -- 172.21.2.117:0/1479333357 learned_addr learned my addr 172.21.2.117:0/1479333357 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 -- 172.21.2.117:0/1479333357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f009807da20 msgr2=0x7f00980771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 --2- 172.21.2.117:0/1479333357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f009807da20 0x7f00980771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 -- 172.21.2.117:0/1479333357 >> v2:172.21.2.125:3300/0 conn(0x7f00980e51a0 msgr2=0x7f0098077ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 --2- 172.21.2.117:0/1479333357 >> v2:172.21.2.125:3300/0 conn(0x7f00980e51a0 0x7f0098077ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 -- 172.21.2.117:0/1479333357 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009807de20 con 0x7f00980e4da0 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f00977fe700 1 --2- 172.21.2.117:0/1479333357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00980e4da0 0x7f00981b00f0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f008c006700 tx=0x7f008c009b80).ready entity=mon.0 client_cookie=54e7a08c921dabcd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f0096ffd700 1 -- 172.21.2.117:0/1479333357 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f008c00c0c0 con 0x7f00980e4da0 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f0096ffd700 1 -- 172.21.2.117:0/1479333357 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c00c340 con 0x7f00980e4da0 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f0096ffd700 1 -- 172.21.2.117:0/1479333357 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f008c00c860 con 0x7f00980e4da0 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f009e4b3700 1 -- 172.21.2.117:0/1479333357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00980e4da0 msgr2=0x7f00981b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.230 7f009e4b3700 1 --2- 172.21.2.117:0/1479333357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00980e4da0 0x7f00981b00f0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f008c006700 tx=0x7f008c009b80).stop 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 -- 172.21.2.117:0/1479333357 shutdown_connections 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 --2- 172.21.2.117:0/1479333357 >> v2:172.21.2.125:3300/0 conn(0x7f00980e51a0 0x7f0098077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 --2- 172.21.2.117:0/1479333357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00980e4da0 0x7f00981b00f0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 --2- 172.21.2.117:0/1479333357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f009807da20 0x7f00980771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 -- 172.21.2.117:0/1479333357 >> 172.21.2.117:0/1479333357 conn(0x7f00980ce450 msgr2=0x7f00980e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:11.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 -- 172.21.2.117:0/1479333357 shutdown_connections 2021-02-10T11:41:11.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.231 7f009e4b3700 1 -- 172.21.2.117:0/1479333357 wait complete. 2021-02-10T11:41:11.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 Processor -- start 2021-02-10T11:41:11.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 -- start start 2021-02-10T11:41:11.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f009809bbe0 0x7f0098092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0098093370 0x7f00980b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980927d0 con 0x7f009809bfe0 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00980b08d0 con 0x7f009809bbe0 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f009e4b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0098190400 con 0x7f0098093370 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f00977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f00977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48556/0 (socket says 172.21.2.117:48556) 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.246 7f00977fe700 1 -- 172.21.2.117:0/1252045567 learned_addr learned my addr 172.21.2.117:0/1252045567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 -- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0098093370 msgr2=0x7f00980b02b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0098093370 0x7f00980b02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 -- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f009809bbe0 msgr2=0x7f0098092d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f009809bbe0 0x7f0098092d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 -- 172.21.2.117:0/1252045567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00981908a0 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f00977fe700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f008c00b5e0 tx=0x7f008c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f008c00bd70 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c002030 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00981908a0 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f008c0024d0 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.247 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00981abb00 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.248 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f008c000c90 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.248 7f0094ff9700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 0x7f007801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.248 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f008c0327e0 con 0x7f009809bfe0 2021-02-10T11:41:11.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.249 7f0097fff700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 0x7f007801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.249 7f0097fff700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 0x7f007801ac10 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.279 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0098103370 con 0x7f009809bfe0 2021-02-10T11:41:11.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.282 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f008c000940 con 0x7f009809bfe0 2021-02-10T11:41:11.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.410 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0098077a70 con 0x7f009809bfe0 2021-02-10T11:41:11.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.411 7f0094ff9700 1 -- 172.21.2.117:0/1252045567 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v47) v1 ==== 104+0+10 (secure 0 0 0) 0x7f008c000c90 con 0x7f009809bfe0 2021-02-10T11:41:11.396 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 msgr2=0x7f007801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 0x7f007801ac10 crc :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 msgr2=0x7f00980afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f008c00b5e0 tx=0x7f008c00b980).stop 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 shutdown_connections 2021-02-10T11:41:11.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0078018730 0x7f007801ac10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f009809bfe0 0x7f00980afc90 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0098093370 0x7f00980b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 --2- 172.21.2.117:0/1252045567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f009809bbe0 0x7f0098092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 >> 172.21.2.117:0/1252045567 conn(0x7f00980a1420 msgr2=0x7f009809efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 shutdown_connections 2021-02-10T11:41:11.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.413 7f009e4b3700 1 -- 172.21.2.117:0/1252045567 wait complete. 2021-02-10T11:41:11.406 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.834 7f96c164d700 1 Processor -- start 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.834 7f96c164d700 1 -- start start 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96c164d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f96bc07da20 0x7f96bc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96c164d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 0x7f96bc079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96c164d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc077a70 con 0x7f96bc0e4da0 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96c164d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc0e5c20 con 0x7f96bc07da20 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96c164d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96bc0777d0 con 0x7f96bc0e51a0 2021-02-10T11:41:11.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96bb2bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f96bc07da20 0x7f96bc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96baabe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f96bc0e4da0 0x7f96bc1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48814/0 (socket says 172.21.2.117:48814) 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.835 7f96baabe700 1 -- 172.21.2.117:0/674791376 learned_addr learned my addr 172.21.2.117:0/674791376 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bbac0700 1 --2- 172.21.2.117:0/674791376 >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 0x7f96bc079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bb2bf700 1 -- 172.21.2.117:0/674791376 >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 msgr2=0x7f96bc079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bb2bf700 1 --2- 172.21.2.117:0/674791376 >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 0x7f96bc079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bb2bf700 1 -- 172.21.2.117:0/674791376 >> v1:172.21.2.117:6789/0 conn(0x7f96bc0e4da0 legacy=0x7f96bc1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96ba2bd700 1 -- 172.21.2.117:0/674791376 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748351128 0 0) 0x7f96a40010b0 con 0x7f96bc0e4da0 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bb2bf700 1 -- 172.21.2.117:0/674791376 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc1b08e0 con 0x7f96bc07da20 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bbac0700 1 --2- 172.21.2.117:0/674791376 >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 0x7f96bc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96bb2bf700 1 --2- 172.21.2.117:0/674791376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc07da20 0x7f96bc0771b0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f96ac006650 tx=0x7f96ac009a80).ready entity=mon.0 client_cookie=c29a064291bd2ed3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.836 7f96ba2bd700 1 -- 172.21.2.117:0/674791376 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96ac0009b0 con 0x7f96bc07da20 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96ba2bd700 1 -- 172.21.2.117:0/674791376 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac00c0c0 con 0x7f96bc07da20 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96ba2bd700 1 -- 172.21.2.117:0/674791376 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96ac00c5e0 con 0x7f96bc07da20 2021-02-10T11:41:11.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 -- 172.21.2.117:0/674791376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc07da20 msgr2=0x7f96bc0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 --2- 172.21.2.117:0/674791376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc07da20 0x7f96bc0771b0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f96ac006650 tx=0x7f96ac009a80).stop 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 -- 172.21.2.117:0/674791376 shutdown_connections 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 --2- 172.21.2.117:0/674791376 >> v2:172.21.2.125:3300/0 conn(0x7f96bc0e51a0 0x7f96bc079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 --2- 172.21.2.117:0/674791376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc07da20 0x7f96bc0771b0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 -- 172.21.2.117:0/674791376 >> 172.21.2.117:0/674791376 conn(0x7f96bc0ce450 msgr2=0x7f96bc0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 -- 172.21.2.117:0/674791376 shutdown_connections 2021-02-10T11:41:11.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.837 7f96c164d700 1 -- 172.21.2.117:0/674791376 wait complete. 2021-02-10T11:41:11.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.852 7f96c164d700 1 Processor -- start 2021-02-10T11:41:11.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.852 7f96c164d700 1 -- start start 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.852 7f96c164d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96bc09e670 0x7f96bc0a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.852 7f96c164d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96bc1079b0 0x7f96bc09f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.852 7f96c164d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96c164d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc09ebc0 con 0x7f96bc107db0 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96c164d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc0a2300 con 0x7f96bc09e670 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96c164d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96bc107510 con 0x7f96bc1079b0 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48572/0 (socket says 172.21.2.117:48572) 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 -- 172.21.2.117:0/3784585660 learned_addr learned my addr 172.21.2.117:0/3784585660 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 -- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96bc1079b0 msgr2=0x7f96bc09f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96bc1079b0 0x7f96bc09f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 -- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96bc09e670 msgr2=0x7f96bc0a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96bc09e670 0x7f96bc0a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:11.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 -- 172.21.2.117:0/3784585660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc09fd80 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96bbac0700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f96b400bb90 tx=0x7f96b400de70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b400b0d0 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b4006790 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b4009650 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96c164d700 1 -- 172.21.2.117:0/3784585660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc09fd80 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.853 7f96c164d700 1 -- 172.21.2.117:0/3784585660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc0a0310 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.854 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96b40011f0 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.854 7f96b3fff700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.854 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f96b4034af0 con 0x7f96bc107db0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.854 7f96bb2bf700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:11.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.855 7f96bb2bf700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:11.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.885 7f96c164d700 1 -- 172.21.2.117:0/3784585660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96bc199d30 con 0x7f96bc107db0 2021-02-10T11:41:11.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:11.888 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96b4000f40 con 0x7f96bc107db0 2021-02-10T11:41:12.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.016 7f96c164d700 1 -- 172.21.2.117:0/3784585660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f96bc199d30 con 0x7f96bc107db0 2021-02-10T11:41:12.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.016 7f96b3fff700 1 -- 172.21.2.117:0/3784585660 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v47) v1 ==== 100+0+10 (secure 0 0 0) 0x7f96b40011f0 con 0x7f96bc107db0 2021-02-10T11:41:12.001 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 -- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 msgr2=0x7f96a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 crc :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 -- 172.21.2.117:0/3784585660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 msgr2=0x7f96bc09f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0x7f96b400bb90 tx=0x7f96b400de70).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 -- 172.21.2.117:0/3784585660 shutdown_connections 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96bc107db0 0x7f96bc09f760 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96bc1079b0 0x7f96bc09f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:12.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 --2- 172.21.2.117:0/3784585660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96bc09e670 0x7f96bc0a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 -- 172.21.2.117:0/3784585660 >> 172.21.2.117:0/3784585660 conn(0x7f96bc08a180 msgr2=0x7f96bc0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.018 7f96c164d700 1 -- 172.21.2.117:0/3784585660 shutdown_connections 2021-02-10T11:41:12.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.019 7f96c164d700 1 -- 172.21.2.117:0/3784585660 wait complete. 2021-02-10T11:41:12.011 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:41:12.012 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:12.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 Processor -- start 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 -- start start 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff72407da20 0x7ff7240771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff7240e4d90 0x7ff7241b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff724077a70 con 0x7ff7240e5190 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7240777d0 con 0x7ff7240e4d90 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.437 7ff72a4d1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72407de20 con 0x7ff72407da20 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff728aee700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff7240e5190 0x7ff724077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48826/0 (socket says 172.21.2.117:48826) 2021-02-10T11:41:12.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff728aee700 1 -- 172.21.2.117:0/3666468020 learned_addr learned my addr 172.21.2.117:0/3666468020 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 --2- 172.21.2.117:0/3666468020 >> v2:172.21.2.117:3300/0 conn(0x7ff7240e4d90 0x7ff7241b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff723fff700 1 --2- 172.21.2.117:0/3666468020 >> v2:172.21.2.125:3300/0 conn(0x7ff72407da20 0x7ff7240771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 -- 172.21.2.117:0/3666468020 >> v2:172.21.2.125:3300/0 conn(0x7ff72407da20 msgr2=0x7ff7240771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 --2- 172.21.2.117:0/3666468020 >> v2:172.21.2.125:3300/0 conn(0x7ff72407da20 0x7ff7240771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 -- 172.21.2.117:0/3666468020 >> v1:172.21.2.117:6789/0 conn(0x7ff7240e5190 legacy=0x7ff724077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 -- 172.21.2.117:0/3666468020 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7241b0700 con 0x7ff7240e4d90 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff7237fe700 1 --2- 172.21.2.117:0/3666468020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7240e4d90 0x7ff7241b00e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7ff710006650 tx=0x7ff710009a80).ready entity=mon.0 client_cookie=623bb6a9f3bd3680 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.438 7ff722ffd700 1 -- 172.21.2.117:0/3666468020 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7100009b0 con 0x7ff7240e4d90 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff722ffd700 1 -- 172.21.2.117:0/3666468020 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71000c0c0 con 0x7ff7240e4d90 2021-02-10T11:41:12.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff722ffd700 1 -- 172.21.2.117:0/3666468020 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71000c510 con 0x7ff7240e4d90 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 -- 172.21.2.117:0/3666468020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7240e4d90 msgr2=0x7ff7241b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 --2- 172.21.2.117:0/3666468020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7240e4d90 0x7ff7241b00e0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0x7ff710006650 tx=0x7ff710009a80).stop 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 -- 172.21.2.117:0/3666468020 shutdown_connections 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 --2- 172.21.2.117:0/3666468020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7240e4d90 0x7ff7241b00e0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 --2- 172.21.2.117:0/3666468020 >> v2:172.21.2.125:3300/0 conn(0x7ff72407da20 0x7ff7240771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 -- 172.21.2.117:0/3666468020 >> 172.21.2.117:0/3666468020 conn(0x7ff7240ce440 msgr2=0x7ff7240e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 -- 172.21.2.117:0/3666468020 shutdown_connections 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.439 7ff72a4d1700 1 -- 172.21.2.117:0/3666468020 wait complete. 2021-02-10T11:41:12.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 Processor -- start 2021-02-10T11:41:12.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 -- start start 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7241aeb30 0x7ff72419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7241a04d0 0x7ff72419b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72419efb0 con 0x7ff72419fab0 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72419bd60 con 0x7ff7241a04d0 2021-02-10T11:41:12.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.454 7ff72a4d1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72419bf80 con 0x7ff7241aeb30 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48584/0 (socket says 172.21.2.117:48584) 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 -- 172.21.2.117:0/180297246 learned_addr learned my addr 172.21.2.117:0/180297246 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff728aee700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7241a04d0 0x7ff72419b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff723fff700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7241aeb30 0x7ff72419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 -- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7241aeb30 msgr2=0x7ff72419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7241aeb30 0x7ff72419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 -- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7241a04d0 msgr2=0x7ff72419b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7241a04d0 0x7ff72419b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 -- 172.21.2.117:0/180297246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72419c1a0 con 0x7ff72419fab0 2021-02-10T11:41:12.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff7237fe700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7ff710009630 tx=0x7ff71000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7100093a0 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71000bd70 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72419c1a0 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.455 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7100054d0 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.456 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72419c730 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.456 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7100052e0 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.456 7ff720ff9700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 0x7ff70001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.456 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff710032560 con 0x7ff72419fab0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.457 7ff723fff700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 0x7ff70001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:12.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.457 7ff723fff700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 0x7ff70001abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:12.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.487 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7241b4f30 con 0x7ff72419fab0 2021-02-10T11:41:12.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.490 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff710009340 con 0x7ff72419fab0 2021-02-10T11:41:12.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.617 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff724077a70 con 0x7ff72419fab0 2021-02-10T11:41:12.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.617 7ff720ff9700 1 -- 172.21.2.117:0/180297246 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+2093 (secure 0 0 0) 0x7ff7100052e0 con 0x7ff72419fab0 2021-02-10T11:41:12.602 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:12.602 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":28,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:40:57.258727","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5657":{"gid":5657,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/58270881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":58270881},{"type":"v1","addr":"172.21.2.117:6833","nonce":58270881}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:41:12.603 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 28 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 msgr2=0x7ff70001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 0x7ff70001abc0 crc :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 msgr2=0x7ff72419feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0x7ff710009630 tx=0x7ff71000b980).stop 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 shutdown_connections 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7000186c0 0x7ff70001abc0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72419fab0 0x7ff72419feb0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7241a04d0 0x7ff72419b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 --2- 172.21.2.117:0/180297246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7241aeb30 0x7ff72419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:12.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 >> 172.21.2.117:0/180297246 conn(0x7ff724093470 msgr2=0x7ff7241a8130 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:12.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.619 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 shutdown_connections 2021-02-10T11:41:12.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:12.620 7ff72a4d1700 1 -- 172.21.2.117:0/180297246 wait complete. 2021-02-10T11:41:12.814 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:40:55.253229', 'modified': '2021-02-10 11:40:57.258727', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5657}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5657': {'gid': 5657, 'name': 'a', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/58270881', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 58270881}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 58270881}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:12.814 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:12.814 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:41:13.213 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:13.213 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5657, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 17.974085035000002, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 28, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 47, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 47, 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 23.609982540000001 2021-02-10T11:41:13.214 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:13.218 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5657, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.974085035000002, "mdsmap_epoch": 28, "osdmap_epoch": 47, "osdmap_epoch_barrier": 47, "uptime": 23.609982540000001 } 2021-02-10T11:41:13.218 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:41:13.218 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:41:13.218 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:41:13.218 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:13.218 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:13.554 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:13.898 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:13.899 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:14.241 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:14.242 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:14.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:14.243 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:14.578 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:41:14.578 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:41:14.578 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:14.581 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:14.925 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:14.925 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:14.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:41:15.004 7efdf622ef80 -1 init, newargv = 0x5609b66bcee0 newargc=9 2021-02-10T11:41:14.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[125936]: starting ceph client 2021-02-10T11:41:14.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[125936]: starting fuse 2021-02-10T11:41:15.267 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:15.267 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:15.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:15.268 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:15.603 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:41:15.603 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:41:15.603 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:41:15.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:15.603 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:41:15.603 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:41:15.603 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:41:15.604 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:41:15.605 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:41:15.966 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.125936.asok 2021-02-10T11:41:15.969 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.125936.asok 2021-02-10T11:41:15.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:15.970 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.125936.asok status 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:41:16.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "125936", 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 7815, 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 7815 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:41:16.360 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1399954731 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1399954731 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.7815 172.21.2.117:0/1399954731", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1399954731", 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 28, 2021-02-10T11:41:16.361 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 47, 2021-02-10T11:41:16.362 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:41:16.362 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:41:16.362 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:16.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:16.363 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:16.364 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:16.697 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:41:16.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:16.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:16.698 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:17.042 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:41:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.471 7faee35f4700 1 Processor -- start 2021-02-10T11:41:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.471 7faee35f4700 1 -- start start 2021-02-10T11:41:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee35f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc07da20 0x7faedc0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee35f4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faedc0e5200 0x7faedc0d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee35f4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc077a10 con 0x7faedc0e4e00 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee35f4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0e5c60 con 0x7faedc0e5200 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee35f4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0777d0 con 0x7faedc07da20 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee0c0f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faedc0e4e00 0x7faedc1b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48852/0 (socket says 172.21.2.117:48852) 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee0c0f700 1 -- 172.21.2.117:0/1363482369 learned_addr learned my addr 172.21.2.117:0/1363482369 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee1c11700 1 --2- 172.21.2.117:0/1363482369 >> v2:172.21.2.125:3300/0 conn(0x7faedc0e5200 0x7faedc0d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.472 7faee1410700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc07da20 0x7faedc0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faecffff700 1 -- 172.21.2.117:0/1363482369 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855630165 0 0) 0x7faed00010b0 con 0x7faedc0e4e00 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faecffff700 1 -- 172.21.2.117:0/1363482369 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec8002fe0 con 0x7faedc0e4e00 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee1c11700 1 -- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc07da20 msgr2=0x7faedc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee1c11700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc07da20 0x7faedc0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee1c11700 1 -- 172.21.2.117:0/1363482369 >> v1:172.21.2.117:6789/0 conn(0x7faedc0e4e00 legacy=0x7faedc1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee1c11700 1 -- 172.21.2.117:0/1363482369 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc1b0920 con 0x7faedc0e5200 2021-02-10T11:41:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee1c11700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc0e5200 0x7faedc0d0550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7faed4006e50 tx=0x7faed4009e70).ready entity=mon.1 client_cookie=53719fc143998ea0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faecffff700 1 -- 172.21.2.117:0/1363482369 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed400b510 con 0x7faedc0e5200 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faecffff700 1 -- 172.21.2.117:0/1363482369 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4006790 con 0x7faedc0e5200 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faecffff700 1 -- 172.21.2.117:0/1363482369 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed400b940 con 0x7faedc0e5200 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee35f4700 1 -- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc0e5200 msgr2=0x7faedc0d0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.473 7faee35f4700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc0e5200 0x7faedc0d0550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7faed4006e50 tx=0x7faed4009e70).stop 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 -- 172.21.2.117:0/1363482369 shutdown_connections 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc0e5200 0x7faedc0d0550 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 --2- 172.21.2.117:0/1363482369 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc07da20 0x7faedc0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 -- 172.21.2.117:0/1363482369 >> 172.21.2.117:0/1363482369 conn(0x7faedc0ce480 msgr2=0x7faedc0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 -- 172.21.2.117:0/1363482369 shutdown_connections 2021-02-10T11:41:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.474 7faee35f4700 1 -- 172.21.2.117:0/1363482369 wait complete. 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 Processor -- start 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 -- start start 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc190a20 0x7faedc0ba260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc1a4640 0x7faedc08e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc090700 con 0x7faedc08d980 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc08dd80 con 0x7faedc1a4640 2021-02-10T11:41:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee35f4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc08dfa0 con 0x7faedc190a20 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee0c0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee0c0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48608/0 (socket says 172.21.2.117:48608) 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee0c0f700 1 -- 172.21.2.117:0/385342240 learned_addr learned my addr 172.21.2.117:0/385342240 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.489 7faee1c11700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc1a4640 0x7faedc08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee1410700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc190a20 0x7faedc0ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 -- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc190a20 msgr2=0x7faedc0ba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc190a20 0x7faedc0ba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 -- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc1a4640 msgr2=0x7faedc08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc1a4640 0x7faedc08e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 -- 172.21.2.117:0/385342240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc08ed40 con 0x7faedc08d980 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee0c0f700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7faed0001c60 tx=0x7faed000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed00010b0 con 0x7faedc08d980 2021-02-10T11:41:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0004140 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee35f4700 1 -- 172.21.2.117:0/385342240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc08ed40 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faee35f4700 1 -- 172.21.2.117:0/385342240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc08ed40 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.490 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed0004680 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.491 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faed0004320 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.491 7faecdffb700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 0x7faebc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.491 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faed00326a0 con 0x7faedc08d980 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.491 7faee1410700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 0x7faebc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:17.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.492 7faee1410700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 0x7faebc01abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.522 7faee35f4700 1 -- 172.21.2.117:0/385342240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc077a10 con 0x7faedc08d980 2021-02-10T11:41:17.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.524 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faed0000980 con 0x7faedc08d980 2021-02-10T11:41:17.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:17.660 7faee35f4700 1 -- 172.21.2.117:0/385342240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7faedc077a10 con 0x7faedc08d980 2021-02-10T11:41:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.131 7faecdffb700 1 -- 172.21.2.117:0/385342240 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 136+0+0 (secure 0 0 0) 0x7faed0004320 con 0x7faedc08d980 2021-02-10T11:41:18.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.179 7faee35f4700 1 -- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 msgr2=0x7faebc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.179 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 0x7faebc01abc0 crc :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.179 7faee35f4700 1 -- 172.21.2.117:0/385342240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 msgr2=0x7faedc1a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.179 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7faed0001c60 tx=0x7faed000a8d0).stop 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 -- 172.21.2.117:0/385342240 shutdown_connections 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faebc0186c0 0x7faebc01abc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faedc08d980 0x7faedc1a4020 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faedc1a4640 0x7faedc08e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 --2- 172.21.2.117:0/385342240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faedc190a20 0x7faedc0ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 -- 172.21.2.117:0/385342240 >> 172.21.2.117:0/385342240 conn(0x7faedc0c0420 msgr2=0x7faedc097c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 -- 172.21.2.117:0/385342240 shutdown_connections 2021-02-10T11:41:18.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.180 7faee35f4700 1 -- 172.21.2.117:0/385342240 wait complete. 2021-02-10T11:41:18.174 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:41:18.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.599 7f197585e700 1 Processor -- start 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.599 7f197585e700 1 -- start start 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f197585e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1970079000 0x7f19700e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f197585e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19700d1d00 con 0x7f19701b00d0 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f197585e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1970077a70 con 0x7f197007da20 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f197585e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19700e53a0 con 0x7f1970079000 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196fd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f19701b00d0 0x7f19700d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48866/0 (socket says 172.21.2.117:48866) 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196fd75700 1 -- 172.21.2.117:0/2981870400 learned_addr learned my addr 172.21.2.117:0/2981870400 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196ed73700 1 --2- 172.21.2.117:0/2981870400 >> v2:172.21.2.117:3300/0 conn(0x7f1970079000 0x7f19700e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196e572700 1 -- 172.21.2.117:0/2981870400 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239519173 0 0) 0x7f1968001500 con 0x7f19701b00d0 2021-02-10T11:41:18.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196ed73700 1 -- 172.21.2.117:0/2981870400 >> v1:172.21.2.125:6789/0 conn(0x7f197007da20 legacy=0x7f19700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196ed73700 1 -- 172.21.2.117:0/2981870400 >> v1:172.21.2.117:6789/0 conn(0x7f19701b00d0 legacy=0x7f19700d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.600 7f196ed73700 1 -- 172.21.2.117:0/2981870400 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970079400 con 0x7f1970079000 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f196ed73700 1 --2- 172.21.2.117:0/2981870400 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1970079000 0x7f19700e4d80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f1958006650 tx=0x7f1958009a80).ready entity=mon.0 client_cookie=fb29866159c89073 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f196e572700 1 -- 172.21.2.117:0/2981870400 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195800c0c0 con 0x7f1970079000 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f196e572700 1 -- 172.21.2.117:0/2981870400 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195800c0c0 con 0x7f1970079000 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f196e572700 1 -- 172.21.2.117:0/2981870400 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f195800c400 con 0x7f1970079000 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 -- 172.21.2.117:0/2981870400 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1970079000 msgr2=0x7f19700e4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 --2- 172.21.2.117:0/2981870400 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1970079000 0x7f19700e4d80 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7f1958006650 tx=0x7f1958009a80).stop 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 -- 172.21.2.117:0/2981870400 shutdown_connections 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 --2- 172.21.2.117:0/2981870400 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1970079000 0x7f19700e4d80 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 -- 172.21.2.117:0/2981870400 >> 172.21.2.117:0/2981870400 conn(0x7f19700ce430 msgr2=0x7f19700e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:18.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 -- 172.21.2.117:0/2981870400 shutdown_connections 2021-02-10T11:41:18.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.601 7f197585e700 1 -- 172.21.2.117:0/2981870400 wait complete. 2021-02-10T11:41:18.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.616 7f197585e700 1 Processor -- start 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.616 7f197585e700 1 -- start start 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1970101080 0x7f19700b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 0x7f1970106170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970105bf0 con 0x7f19701023a0 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1970102800 con 0x7f19700b7bc0 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f197585e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197009a820 con 0x7f1970101080 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48620/0 (socket says 172.21.2.117:48620) 2021-02-10T11:41:18.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 -- 172.21.2.117:0/2247546410 learned_addr learned my addr 172.21.2.117:0/2247546410 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 -- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1970101080 msgr2=0x7f19700b75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1970101080 0x7f19700b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 -- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 msgr2=0x7f1970106170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196fd75700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 0x7f1970106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 0x7f1970106170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 -- 172.21.2.117:0/2247546410 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700b7fc0 con 0x7f19701023a0 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196fd75700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 0x7f1970106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.617 7f196f574700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f196000b530 tx=0x7f196000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196000bcc0 con 0x7f19701023a0 2021-02-10T11:41:18.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960006020 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f197585e700 1 -- 172.21.2.117:0/2247546410 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19700b7fc0 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f197585e700 1 -- 172.21.2.117:0/2247546410 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1970106790 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19600054a0 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1960006260 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 0x7f195401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.618 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1960032690 con 0x7f19701023a0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.619 7f196ed73700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 0x7f195401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:18.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.619 7f196ed73700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 0x7f195401abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:18.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.650 7f197585e700 1 -- 172.21.2.117:0/2247546410 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19700ff150 con 0x7f19701023a0 2021-02-10T11:41:18.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.652 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1960000dd0 con 0x7f19701023a0 2021-02-10T11:41:18.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.785 7f197585e700 1 -- 172.21.2.117:0/2247546410 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f1970106790 con 0x7f19540186c0 2021-02-10T11:41:18.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.787 7f1967fff700 1 -- 172.21.2.117:0/2247546410 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f1958006e50 con 0x7f19540186c0 2021-02-10T11:41:18.774 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:41:18.774 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:18.774 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:41:18.774 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:18.774 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:41:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.789 7f197585e700 1 -- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 msgr2=0x7f195401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.789 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 0x7f195401abc0 crc :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.789 7f197585e700 1 -- 172.21.2.117:0/2247546410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 msgr2=0x7f1970105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.789 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7f196000b530 tx=0x7f196000b8d0).stop 2021-02-10T11:41:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 -- 172.21.2.117:0/2247546410 shutdown_connections 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19700b7bc0 0x7f1970106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1970101080 0x7f19700b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19540186c0 0x7f195401abc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 --2- 172.21.2.117:0/2247546410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19701023a0 0x7f1970105390 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 -- 172.21.2.117:0/2247546410 >> 172.21.2.117:0/2247546410 conn(0x7f197009ac10 msgr2=0x7f1970194fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 -- 172.21.2.117:0/2247546410 shutdown_connections 2021-02-10T11:41:18.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:18.790 7f197585e700 1 -- 172.21.2.117:0/2247546410 wait complete. 2021-02-10T11:41:18.784 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 Processor -- start 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 -- start start 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 0x7f9bac1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac077a70 con 0x7f9bac07da20 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac0e53c0 con 0x7f9bac1b00f0 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.212 7f9bb2cf2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac079400 con 0x7f9bac079000 2021-02-10T11:41:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.213 7f9babfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.213 7f9bb130f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 0x7f9bac1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.213 7f9babfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46308/0 (socket says 172.21.2.117:46308) 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.213 7f9bb130f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 0x7f9bac1b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:53014/0 (socket says 172.21.2.117:53014) 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.213 7f9babfff700 1 -- 172.21.2.117:0/1742282304 learned_addr learned my addr 172.21.2.117:0/1742282304 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9babfff700 1 -- 172.21.2.117:0/1742282304 >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 msgr2=0x7f9bac1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9babfff700 1 --2- 172.21.2.117:0/1742282304 >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 0x7f9bac1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9babfff700 1 -- 172.21.2.117:0/1742282304 >> v1:172.21.2.125:6789/0 conn(0x7f9bac07da20 legacy=0x7f9bac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9babfff700 1 -- 172.21.2.117:0/1742282304 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac07de20 con 0x7f9bac079000 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9babfff700 1 --2- 172.21.2.117:0/1742282304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9b9c006700 tx=0x7f9b9c009b80).ready entity=mon.2 client_cookie=2e9805afc2db2652 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9bab7fe700 1 -- 172.21.2.117:0/1742282304 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b9c00c0c0 con 0x7f9bac079000 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9bab7fe700 1 -- 172.21.2.117:0/1742282304 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c00c440 con 0x7f9bac079000 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9bab7fe700 1 -- 172.21.2.117:0/1742282304 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b9c00c960 con 0x7f9bac079000 2021-02-10T11:41:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9bb2cf2700 1 -- 172.21.2.117:0/1742282304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 msgr2=0x7f9bac0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.214 7f9bb2cf2700 1 --2- 172.21.2.117:0/1742282304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9b9c006700 tx=0x7f9b9c009b80).stop 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 -- 172.21.2.117:0/1742282304 shutdown_connections 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 --2- 172.21.2.117:0/1742282304 >> v2:172.21.2.125:3300/0 conn(0x7f9bac1b00f0 0x7f9bac1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 --2- 172.21.2.117:0/1742282304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac079000 0x7f9bac0e4da0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 -- 172.21.2.117:0/1742282304 >> 172.21.2.117:0/1742282304 conn(0x7f9bac0ce450 msgr2=0x7f9bac0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 -- 172.21.2.117:0/1742282304 shutdown_connections 2021-02-10T11:41:19.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.215 7f9bb2cf2700 1 -- 172.21.2.117:0/1742282304 wait complete. 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.229 7f9bb2cf2700 1 Processor -- start 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.229 7f9bb2cf2700 1 -- start start 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac19a4a0 0x7f9bac0b9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bac0b9e60 0x7f9bac0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0a4220 con 0x7f9bac0a3800 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0a4410 con 0x7f9bac0b9e60 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb2cf2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0ba910 con 0x7f9bac19a4a0 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb130f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb130f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48634/0 (socket says 172.21.2.117:48634) 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb130f700 1 -- 172.21.2.117:0/1938491889 learned_addr learned my addr 172.21.2.117:0/1938491889 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9babfff700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bac0b9e60 0x7f9bac0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.230 7f9bb0b0e700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac19a4a0 0x7f9bac0b9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 -- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac19a4a0 msgr2=0x7f9bac0b9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac19a4a0 0x7f9bac0b9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 -- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bac0b9e60 msgr2=0x7f9bac0a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bac0b9e60 0x7f9bac0a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 -- 172.21.2.117:0/1938491889 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac0bab30 con 0x7f9bac0a3800 2021-02-10T11:41:19.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb130f700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f9ba400b970 tx=0x7f9ba400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ba4000f90 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba4006aa0 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ba4001680 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac0bab30 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.231 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac0bb0c0 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.232 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ba40011f0 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.232 7f9ba97fa700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 0x7f9b9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.232 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9ba40328c0 con 0x7f9bac0a3800 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.232 7f9bb0b0e700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 0x7f9b9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.233 7f9bb0b0e700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 0x7f9b9001abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.262 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac0ba3e0 con 0x7f9bac0a3800 2021-02-10T11:41:19.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.265 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ba4027ba0 con 0x7f9bac0a3800 2021-02-10T11:41:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.412 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9bac0ba3e0 con 0x7f9bac0a3800 2021-02-10T11:41:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.412 7f9ba97fa700 1 -- 172.21.2.117:0/1938491889 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9ba4027ba0 con 0x7f9bac0a3800 2021-02-10T11:41:19.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 msgr2=0x7f9b9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 0x7f9b9001abc0 crc :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 msgr2=0x7f9bac0a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f9ba400b970 tx=0x7f9ba400bd10).stop 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 shutdown_connections 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b900186c0 0x7f9b9001abc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bac0a3800 0x7f9bac0a3c00 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bac0b9e60 0x7f9bac0a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 --2- 172.21.2.117:0/1938491889 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bac19a4a0 0x7f9bac0b9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 >> 172.21.2.117:0/1938491889 conn(0x7f9bac107b60 msgr2=0x7f9bac0c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 shutdown_connections 2021-02-10T11:41:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.414 7f9bb2cf2700 1 -- 172.21.2.117:0/1938491889 wait complete. 2021-02-10T11:41:19.408 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get-or-create client.guest1 mds 'allow *' osd 'allow rw' mon 'allow *' 2021-02-10T11:41:19.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.844 7f55ac936700 1 Processor -- start 2021-02-10T11:41:19.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.844 7f55ac936700 1 -- start start 2021-02-10T11:41:19.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55ac936700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a807da20 0x7f55a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55ac936700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f55a80e51a0 0x7f55a80d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55ac936700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a8077a70 con 0x7f55a80e4da0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55ac936700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a80e5c20 con 0x7f55a80e51a0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55ac936700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a80777d0 con 0x7f55a807da20 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a5e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f55a80e4da0 0x7f55a81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48890/0 (socket says 172.21.2.117:48890) 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a5e1b700 1 -- 172.21.2.117:0/2659726294 learned_addr learned my addr 172.21.2.117:0/2659726294 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086140751 0 0) 0x7f559c0009b0 con 0x7f55a80e4da0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a561a700 1 -- 172.21.2.117:0/2659726294 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5590002fe0 con 0x7f55a80e4da0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a6e1d700 1 --2- 172.21.2.117:0/2659726294 >> v2:172.21.2.125:3300/0 conn(0x7f55a80e51a0 0x7f55a80d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.845 7f55a661c700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a807da20 0x7f55a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4217051631 0 0) 0x7f559c0010b0 con 0x7f55a80e4da0 2021-02-10T11:41:19.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a807da20 msgr2=0x7f55a80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a807da20 0x7f55a80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 >> v2:172.21.2.125:3300/0 conn(0x7f55a80e51a0 msgr2=0x7f55a80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 --2- 172.21.2.117:0/2659726294 >> v2:172.21.2.125:3300/0 conn(0x7f55a80e51a0 0x7f55a80d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a81b08e0 con 0x7f55a80e4da0 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f559c0015b0 con 0x7f55a80e4da0 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 >> v1:172.21.2.117:6789/0 conn(0x7f55a80e4da0 legacy=0x7f55a81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5590003af0 0x7f5590005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f559004a770 0x7f559004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f559004d510 0x7f559004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900502e0 con 0x7f559004a770 2021-02-10T11:41:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590050500 con 0x7f559004d510 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a561a700 1 -- 172.21.2.117:0/2659726294 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5590050720 con 0x7f5590003af0 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a661c700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5590003af0 0x7f5590005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a5e1b700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f559004a770 0x7f559004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a6e1d700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f559004d510 0x7f559004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a5e1b700 1 -- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5590003af0 msgr2=0x7f5590005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.846 7f55a5e1b700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5590003af0 0x7f5590005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a5e1b700 1 -- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f559004d510 msgr2=0x7f559004fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a5e1b700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f559004d510 0x7f559004fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a5e1b700 1 -- 172.21.2.117:0/2659726294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900509c0 con 0x7f559004a770 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a5e1b700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f559004a770 0x7f559004cc70 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f559c0017d0 tx=0x7f559c007d30).ready entity=mon.0 client_cookie=8827456973e16433 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f559c0013a0 con 0x7f559004a770 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c00a310 con 0x7f559004a770 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a561a700 1 -- 172.21.2.117:0/2659726294 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f559c00a760 con 0x7f559004a770 2021-02-10T11:41:19.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55ac936700 1 -- 172.21.2.117:0/2659726294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f559004a770 msgr2=0x7f559004cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55ac936700 1 --2- 172.21.2.117:0/2659726294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f559004a770 0x7f559004cc70 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7f559c0017d0 tx=0x7f559c007d30).stop 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.847 7f55a6e1d700 1 -- 172.21.2.117:0/2659726294 reap_dead start 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.848 7f55ac936700 1 -- 172.21.2.117:0/2659726294 shutdown_connections 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.848 7f55ac936700 1 -- 172.21.2.117:0/2659726294 >> 172.21.2.117:0/2659726294 conn(0x7f55a80ce450 msgr2=0x7f55a80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.848 7f55ac936700 1 -- 172.21.2.117:0/2659726294 shutdown_connections 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.848 7f55ac936700 1 -- 172.21.2.117:0/2659726294 wait complete. 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 Processor -- start 2021-02-10T11:41:19.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 -- start start 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 0x7f55a80b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 0x7f55a80a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a80a23b0 con 0x7f55a809d8a0 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a8107400 con 0x7f55a80b8650 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55ac936700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a80a3350 con 0x7f55a80a2930 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55a661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55a661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48654/0 (socket says 172.21.2.117:48654) 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.863 7f55a661c700 1 -- 172.21.2.117:0/3806239645 learned_addr learned my addr 172.21.2.117:0/3806239645 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 -- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 msgr2=0x7f55a80a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a6e1d700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 0x7f55a80a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 0x7f55a80a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a5e1b700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 0x7f55a80b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 -- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 msgr2=0x7f55a80b8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 0x7f55a80b8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 -- 172.21.2.117:0/3806239645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a80a1520 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a6e1d700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 0x7f55a80a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a5e1b700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 0x7f55a80b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55a661c700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f5598006f60 tx=0x7f559800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f559800bd60 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5598000c90 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5598002220 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55ac936700 1 -- 172.21.2.117:0/3806239645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a80a1520 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.864 7f55ac936700 1 -- 172.21.2.117:0/3806239645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a80a1ab0 con 0x7f55a809d8a0 2021-02-10T11:41:19.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.865 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f559800a000 con 0x7f55a809d8a0 2021-02-10T11:41:19.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.865 7f5596ffd700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 0x7f558c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:19.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.865 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f55980326e0 con 0x7f55a809d8a0 2021-02-10T11:41:19.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.865 7f55a5e1b700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 0x7f558c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:19.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.866 7f55a5e1b700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 0x7f558c01abc0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:19.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.894 7f55ac936700 1 -- 172.21.2.117:0/3806239645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8077a70 con 0x7f55a809d8a0 2021-02-10T11:41:19.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:19.897 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5598018ff0 con 0x7f55a809d8a0 2021-02-10T11:41:20.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.027 7f55ac936700 1 -- 172.21.2.117:0/3806239645 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]} v 0) v1 -- 0x7f55a8077a70 con 0x7f55a809d8a0 2021-02-10T11:41:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.030 7f5596ffd700 1 -- 172.21.2.117:0/3806239645 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.guest1", "caps": ["mds", "allow *", "osd", "allow rw", "mon", "allow *"]}]=0 v27) v1 ==== 158+0+64 (secure 0 0 0) 0x7f5598018ff0 con 0x7f55a809d8a0 2021-02-10T11:41:20.020 INFO:teuthology.orchestra.run.gibba017.stdout:[client.guest1] 2021-02-10T11:41:20.020 INFO:teuthology.orchestra.run.gibba017.stdout: key = AQBgxiNg7mu9ARAAmNcmCEih/S9qrYk7tRu8mA== 2021-02-10T11:41:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 msgr2=0x7f558c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 0x7f558c01abc0 crc :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 msgr2=0x7f55a8187bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f5598006f60 tx=0x7f559800b970).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 shutdown_connections 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f55a80a2930 0x7f55a80a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f55a80b8650 0x7f55a80b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f558c0186c0 0x7f558c01abc0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 --2- 172.21.2.117:0/3806239645 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f55a809d8a0 0x7f55a8187bf0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 >> 172.21.2.117:0/3806239645 conn(0x7f55a819f530 msgr2=0x7f55a8103560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 shutdown_connections 2021-02-10T11:41:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.036 7f55ac936700 1 -- 172.21.2.117:0/3806239645 wait complete. 2021-02-10T11:41:20.030 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000957772 2021-02-10T11:41:20.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.458 7f22aab9b700 1 Processor -- start 2021-02-10T11:41:20.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.458 7f22aab9b700 1 -- start start 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22a407da20 0x7f22a40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a40e5180 0x7f22a4079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a4077a70 con 0x7f22a40e4d80 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a40777d0 con 0x7f22a407da20 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a41b08c0 con 0x7f22a40e5180 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22a407da20 0x7f22a40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22a407da20 0x7f22a40771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48660/0 (socket says 172.21.2.117:48660) 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 -- 172.21.2.117:0/3723332866 learned_addr learned my addr 172.21.2.117:0/3723332866 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 -- 172.21.2.117:0/3723332866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a40e5180 msgr2=0x7f22a4079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 --2- 172.21.2.117:0/3723332866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a40e5180 0x7f22a4079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 -- 172.21.2.117:0/3723332866 >> v1:172.21.2.125:6789/0 conn(0x7f22a40e4d80 legacy=0x7f22a41b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 -- 172.21.2.117:0/3723332866 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a407de20 con 0x7f22a407da20 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a89b7700 1 --2- 172.21.2.117:0/3723332866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a407da20 0x7f22a40771b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f228c006700 tx=0x7f228c009b80).ready entity=mon.0 client_cookie=3fced5ddc93b30ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a37fe700 1 -- 172.21.2.117:0/3723332866 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c00c0c0 con 0x7f22a407da20 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a37fe700 1 -- 172.21.2.117:0/3723332866 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c00c340 con 0x7f22a407da20 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22a37fe700 1 -- 172.21.2.117:0/3723332866 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f228c00c860 con 0x7f22a407da20 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 -- 172.21.2.117:0/3723332866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a407da20 msgr2=0x7f22a40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.459 7f22aab9b700 1 --2- 172.21.2.117:0/3723332866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a407da20 0x7f22a40771b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f228c006700 tx=0x7f228c009b80).stop 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 -- 172.21.2.117:0/3723332866 shutdown_connections 2021-02-10T11:41:20.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 --2- 172.21.2.117:0/3723332866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a40e5180 0x7f22a4079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 --2- 172.21.2.117:0/3723332866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a407da20 0x7f22a40771b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 -- 172.21.2.117:0/3723332866 >> 172.21.2.117:0/3723332866 conn(0x7f22a40ce430 msgr2=0x7f22a40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 -- 172.21.2.117:0/3723332866 shutdown_connections 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.460 7f22aab9b700 1 -- 172.21.2.117:0/3723332866 wait complete. 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.474 7f22aab9b700 1 Processor -- start 2021-02-10T11:41:20.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 -- start start 2021-02-10T11:41:20.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22a40a39b0 0x7f22a40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a409c0c0 0x7f22a409c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40a77c0 con 0x7f22a40a3db0 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40a5360 con 0x7f22a40a39b0 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22aab9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40a5580 con 0x7f22a409c0c0 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48666/0 (socket says 172.21.2.117:48666) 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 -- 172.21.2.117:0/1163137142 learned_addr learned my addr 172.21.2.117:0/1163137142 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 -- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a409c0c0 msgr2=0x7f22a409c4c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a409c0c0 0x7f22a409c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 -- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22a40a39b0 msgr2=0x7f22a40a4720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:20.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22a40a39b0 0x7f22a40a4720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.475 7f22a3fff700 1 -- 172.21.2.117:0/1163137142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a409cae0 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a3fff700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f2294009940 tx=0x7f229400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2294000fe0 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229400c2f0 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a409cae0 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a409cae0 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f229400c810 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2294001230 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 0x7f228801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.476 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2294032a90 con 0x7f22a40a3db0 2021-02-10T11:41:20.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.478 7f22a89b7700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 0x7f228801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:20.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.478 7f22a89b7700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 0x7f228801abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:20.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.508 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a40a5360 con 0x7f22a40a3db0 2021-02-10T11:41:20.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.511 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f229400be30 con 0x7f22a40a3db0 2021-02-10T11:41:20.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.637 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000957772"}) v1 -- 0x7f22a40a77c0 con 0x7f22880186c0 2021-02-10T11:41:20.628 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:20.641 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:20.628 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:20.641 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:20.628 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:20.641 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:20.628 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:20.641 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:20.628 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:20.641 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:41:20.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.657 7f22a17fa700 1 -- 172.21.2.117:0/1163137142 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f228c0065e0 con 0x7f22880186c0 2021-02-10T11:41:20.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.670 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 msgr2=0x7f228801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.670 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 0x7f228801abc0 crc :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.670 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 msgr2=0x7f22a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:20.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.670 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f2294009940 tx=0x7f229400bcf0).stop 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 shutdown_connections 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22880186c0 0x7f228801abc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22a40a3db0 0x7f22a40a4d40 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22a409c0c0 0x7f22a409c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 --2- 172.21.2.117:0/1163137142 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22a40a39b0 0x7f22a40a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:20.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 >> 172.21.2.117:0/1163137142 conn(0x7f22a40bb240 msgr2=0x7f22a40a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:20.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 shutdown_connections 2021-02-10T11:41:20.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:20.671 7f22aab9b700 1 -- 172.21.2.117:0/1163137142 wait complete. 2021-02-10T11:41:20.665 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000732851 --group_name group_0000000000957772 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 Processor -- start 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 -- start start 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f308c07da20 0x7f308c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f308c0e4e00 0x7f308c1b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c077a10 con 0x7f308c0e5200 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c0777d0 con 0x7f308c07da20 2021-02-10T11:41:21.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f30920ed700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c07de20 con 0x7f308c0e4e00 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f308b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f308c07da20 0x7f308c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f308b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f308c07da20 0x7f308c0771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48672/0 (socket says 172.21.2.117:48672) 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.095 7f308b7fe700 1 -- 172.21.2.117:0/2956532892 learned_addr learned my addr 172.21.2.117:0/2956532892 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308b7fe700 1 -- 172.21.2.117:0/2956532892 >> v2:172.21.2.125:3300/0 conn(0x7f308c0e4e00 msgr2=0x7f308c1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308b7fe700 1 --2- 172.21.2.117:0/2956532892 >> v2:172.21.2.125:3300/0 conn(0x7f308c0e4e00 0x7f308c1b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308b7fe700 1 -- 172.21.2.117:0/2956532892 >> v1:172.21.2.125:6789/0 conn(0x7f308c0e5200 legacy=0x7f308c077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308b7fe700 1 -- 172.21.2.117:0/2956532892 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c1b0750 con 0x7f308c07da20 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308b7fe700 1 --2- 172.21.2.117:0/2956532892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c07da20 0x7f308c0771b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f307c006700 tx=0x7f307c009b80).ready entity=mon.0 client_cookie=723508d126d93cce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308a7fc700 1 -- 172.21.2.117:0/2956532892 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307c00c0c0 con 0x7f308c07da20 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308a7fc700 1 -- 172.21.2.117:0/2956532892 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c00c400 con 0x7f308c07da20 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f308a7fc700 1 -- 172.21.2.117:0/2956532892 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f307c00c920 con 0x7f308c07da20 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f30920ed700 1 -- 172.21.2.117:0/2956532892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c07da20 msgr2=0x7f308c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.096 7f30920ed700 1 --2- 172.21.2.117:0/2956532892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c07da20 0x7f308c0771b0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f307c006700 tx=0x7f307c009b80).stop 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 -- 172.21.2.117:0/2956532892 shutdown_connections 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 --2- 172.21.2.117:0/2956532892 >> v2:172.21.2.125:3300/0 conn(0x7f308c0e4e00 0x7f308c1b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 --2- 172.21.2.117:0/2956532892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c07da20 0x7f308c0771b0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 -- 172.21.2.117:0/2956532892 >> 172.21.2.117:0/2956532892 conn(0x7f308c0ce480 msgr2=0x7f308c0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 -- 172.21.2.117:0/2956532892 shutdown_connections 2021-02-10T11:41:21.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.097 7f30920ed700 1 -- 172.21.2.117:0/2956532892 wait complete. 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.111 7f30920ed700 1 Processor -- start 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.111 7f30920ed700 1 -- start start 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f308c106c50 0x7f308c0b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f308c1a7c80 0x7f308c0ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c093270 con 0x7f308c1a7260 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c1a8080 con 0x7f308c1a7c80 2021-02-10T11:41:21.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f30920ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0c3e50 con 0x7f308c106c50 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48682/0 (socket says 172.21.2.117:48682) 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 -- 172.21.2.117:0/800172837 learned_addr learned my addr 172.21.2.117:0/800172837 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 -- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f308c106c50 msgr2=0x7f308c0b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f308c106c50 0x7f308c0b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 -- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f308c1a7c80 msgr2=0x7f308c0ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f308c1a7c80 0x7f308c0ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 -- 172.21.2.117:0/800172837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0aee60 con 0x7f308c1a7260 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.112 7f308affd700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f3080006f00 tx=0x7f308000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3080006b90 con 0x7f308c1a7260 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3080006b90 con 0x7f308c1a7260 2021-02-10T11:41:21.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f308000bd30 con 0x7f308c1a7260 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f30920ed700 1 -- 172.21.2.117:0/800172837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c0aee60 con 0x7f308c1a7260 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f30920ed700 1 -- 172.21.2.117:0/800172837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c0af3f0 con 0x7f308c1a7260 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3080000c90 con 0x7f308c1a7260 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 0x7f306c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.113 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3080032920 con 0x7f308c1a7260 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.114 7f308b7fe700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 0x7f306c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.114 7f308b7fe700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 0x7f306c01abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.145 7f30920ed700 1 -- 172.21.2.117:0/800172837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c0c3e20 con 0x7f308c1a7260 2021-02-10T11:41:21.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.147 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3080006b90 con 0x7f308c1a7260 2021-02-10T11:41:21.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.273 7f30920ed700 1 -- 172.21.2.117:0/800172837 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000732851", "target": ["mgr", ""], "group_name": "group_0000000000957772"}) v1 -- 0x7f308c1a8080 con 0x7f306c0186c0 2021-02-10T11:41:21.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.297 7f3073fff700 1 -- 172.21.2.117:0/800172837 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f307c006390 con 0x7f306c0186c0 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 msgr2=0x7f306c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 0x7f306c01abc0 crc :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 msgr2=0x7f308c1a7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 rx=0x7f3080006f00 tx=0x7f308000b940).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 shutdown_connections 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f306c0186c0 0x7f306c01abc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f308c1a7260 0x7f308c1a7660 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f308c1a7c80 0x7f308c0ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 --2- 172.21.2.117:0/800172837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f308c106c50 0x7f308c0b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:21.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 >> 172.21.2.117:0/800172837 conn(0x7f308c0c0930 msgr2=0x7f308c09e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 shutdown_connections 2021-02-10T11:41:21.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.306 7f30920ed700 1 -- 172.21.2.117:0/800172837 wait complete. 2021-02-10T11:41:21.299 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000732851 guest1 --group_name group_0000000000957772 --tenant_id tenant1 2021-02-10T11:41:21.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 Processor -- start 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 -- start start 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f47a8079000 0x7f47a80e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a80d1d10 con 0x7f47a81b00e0 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a8077a70 con 0x7f47a807da20 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.737 7f47af935700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80e53b0 con 0x7f47a8079000 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47adf52700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f47a81b00e0 0x7f47a80d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48940/0 (socket says 172.21.2.117:48940) 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47adf52700 1 -- 172.21.2.117:0/3347278095 learned_addr learned my addr 172.21.2.117:0/3347278095 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47acf50700 1 --2- 172.21.2.117:0/3347278095 >> v2:172.21.2.125:3300/0 conn(0x7f47a8079000 0x7f47a80e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f479bfff700 1 -- 172.21.2.117:0/3347278095 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987041502 0 0) 0x7f47a0001180 con 0x7f47a81b00e0 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f479bfff700 1 -- 172.21.2.117:0/3347278095 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4794002fe0 con 0x7f47a81b00e0 2021-02-10T11:41:21.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47acf50700 1 -- 172.21.2.117:0/3347278095 >> v1:172.21.2.125:6789/0 conn(0x7f47a807da20 legacy=0x7f47a80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47acf50700 1 -- 172.21.2.117:0/3347278095 >> v1:172.21.2.117:6789/0 conn(0x7f47a81b00e0 legacy=0x7f47a80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47acf50700 1 -- 172.21.2.117:0/3347278095 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8079400 con 0x7f47a8079000 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.738 7f47acf50700 1 --2- 172.21.2.117:0/3347278095 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a8079000 0x7f47a80e4d90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f47a4006650 tx=0x7f47a4009a00).ready entity=mon.1 client_cookie=8dfc2e099676886e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f479bfff700 1 -- 172.21.2.117:0/3347278095 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4009df0 con 0x7f47a8079000 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f479bfff700 1 -- 172.21.2.117:0/3347278095 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a40009b0 con 0x7f47a8079000 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f479bfff700 1 -- 172.21.2.117:0/3347278095 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4009df0 con 0x7f47a8079000 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f47af935700 1 -- 172.21.2.117:0/3347278095 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a8079000 msgr2=0x7f47a80e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f47af935700 1 --2- 172.21.2.117:0/3347278095 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a8079000 0x7f47a80e4d90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f47a4006650 tx=0x7f47a4009a00).stop 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f47af935700 1 -- 172.21.2.117:0/3347278095 shutdown_connections 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f47af935700 1 --2- 172.21.2.117:0/3347278095 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a8079000 0x7f47a80e4d90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.739 7f47af935700 1 -- 172.21.2.117:0/3347278095 >> 172.21.2.117:0/3347278095 conn(0x7f47a80ce440 msgr2=0x7f47a80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.740 7f47af935700 1 -- 172.21.2.117:0/3347278095 shutdown_connections 2021-02-10T11:41:21.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.740 7f47af935700 1 -- 172.21.2.117:0/3347278095 wait complete. 2021-02-10T11:41:21.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 Processor -- start 2021-02-10T11:41:21.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 -- start start 2021-02-10T11:41:21.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 0x7f47a81b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 0x7f47a819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a818d1f0 con 0x7f47a81b32a0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a819a920 con 0x7f47a819aec0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.754 7f47af935700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a819ab40 con 0x7f47a819b2c0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48694/0 (socket says 172.21.2.117:48694) 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 -- 172.21.2.117:0/3015087370 learned_addr learned my addr 172.21.2.117:0/3015087370 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47adf52700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 0x7f47a819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 -- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 msgr2=0x7f47a819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 0x7f47a819a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 -- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 msgr2=0x7f47a81b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47ad751700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 0x7f47a81b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 0x7f47a81b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 -- 172.21.2.117:0/3015087370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a818d6f0 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47adf52700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 0x7f47a819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47ad751700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 0x7f47a81b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f47acf50700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f47a40096a0 tx=0x7f47a400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.755 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4000c90 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a4002030 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f47af935700 1 -- 172.21.2.117:0/3015087370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a818d6f0 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f47af935700 1 -- 172.21.2.117:0/3015087370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a818dc80 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47a4002550 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47a4002210 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f4799ffb700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.756 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f47a40326b0 con 0x7f47a81b32a0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.757 7f47ad751700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:21.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.757 7f47ad751700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 0x7f478801abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:21.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.787 7f47af935700 1 -- 172.21.2.117:0/3015087370 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8077a70 con 0x7f47a81b32a0 2021-02-10T11:41:21.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.789 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47a40093a0 con 0x7f47a81b32a0 2021-02-10T11:41:21.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.915 7f47af935700 1 -- 172.21.2.117:0/3015087370 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000732851", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000957772"}) v1 -- 0x7f47a819a920 con 0x7f47880186c0 2021-02-10T11:41:21.906 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:41:21.919 7f59c98e1700 -1 mgr.server reply reply (1) Operation not permitted auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:41:21.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.921 7f4799ffb700 1 -- 172.21.2.117:0/3015087370 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -1 auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify) v1 ==== 83+0+0 (crc 0 0 0) 0x7f479c006c50 con 0x7f47880186c0 2021-02-10T11:41:21.907 INFO:teuthology.orchestra.run.gibba017.stderr:Error EPERM: auth ID: guest1 exists and not created by mgr plugin. Not allowed to modify 2021-02-10T11:41:21.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 msgr2=0x7f478801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 0x7f478801abc0 crc :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 msgr2=0x7f47a8199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:21.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f47a40096a0 tx=0x7f47a400b990).stop 2021-02-10T11:41:21.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 shutdown_connections 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47880186c0 0x7f478801abc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a81b32a0 0x7f47a8199ce0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a819b2c0 0x7f47a819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 --2- 172.21.2.117:0/3015087370 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a819aec0 0x7f47a81b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 >> 172.21.2.117:0/3015087370 conn(0x7f47a81975a0 msgr2=0x7f47a81a8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 shutdown_connections 2021-02-10T11:41:21.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:21.924 7f47af935700 1 -- 172.21.2.117:0/3015087370 wait complete. 2021-02-10T11:41:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:21.917 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:41:22.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 Processor -- start 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 -- start start 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fba4807da20 0x7fba480771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480d1d00 con 0x7fba480e4d80 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba48077a70 con 0x7fba480e5180 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba4de7e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba480777d0 con 0x7fba4807da20 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba46ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fba480e4d80 0x7fba481b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48950/0 (socket says 172.21.2.117:48950) 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.342 7fba46ffd700 1 -- 172.21.2.117:0/2089399994 learned_addr learned my addr 172.21.2.117:0/2089399994 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba477fe700 1 --2- 172.21.2.117:0/2089399994 >> v2:172.21.2.125:3300/0 conn(0x7fba4807da20 0x7fba480771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba467fc700 1 -- 172.21.2.117:0/2089399994 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200630189 0 0) 0x7fba3c0010b0 con 0x7fba480e4d80 2021-02-10T11:41:22.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba467fc700 1 -- 172.21.2.117:0/2089399994 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba30002fe0 con 0x7fba480e4d80 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba477fe700 1 -- 172.21.2.117:0/2089399994 >> v1:172.21.2.125:6789/0 conn(0x7fba480e5180 legacy=0x7fba480d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba477fe700 1 -- 172.21.2.117:0/2089399994 >> v1:172.21.2.117:6789/0 conn(0x7fba480e4d80 legacy=0x7fba481b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba477fe700 1 -- 172.21.2.117:0/2089399994 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba481b08c0 con 0x7fba4807da20 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.343 7fba477fe700 1 --2- 172.21.2.117:0/2089399994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba4807da20 0x7fba480771b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fba38006650 tx=0x7fba38009a80).ready entity=mon.1 client_cookie=5034d7f7b9cb6ce9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba467fc700 1 -- 172.21.2.117:0/2089399994 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba3800c0c0 con 0x7fba4807da20 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba467fc700 1 -- 172.21.2.117:0/2089399994 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3800c340 con 0x7fba4807da20 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba467fc700 1 -- 172.21.2.117:0/2089399994 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba3800c860 con 0x7fba4807da20 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 -- 172.21.2.117:0/2089399994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba4807da20 msgr2=0x7fba480771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 --2- 172.21.2.117:0/2089399994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba4807da20 0x7fba480771b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fba38006650 tx=0x7fba38009a80).stop 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 -- 172.21.2.117:0/2089399994 shutdown_connections 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 --2- 172.21.2.117:0/2089399994 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba4807da20 0x7fba480771b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 -- 172.21.2.117:0/2089399994 >> 172.21.2.117:0/2089399994 conn(0x7fba480ce430 msgr2=0x7fba480e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:22.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 -- 172.21.2.117:0/2089399994 shutdown_connections 2021-02-10T11:41:22.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.344 7fba4de7e700 1 -- 172.21.2.117:0/2089399994 wait complete. 2021-02-10T11:41:22.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 Processor -- start 2021-02-10T11:41:22.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 -- start start 2021-02-10T11:41:22.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba481023a0 0x7fba48105390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba48101080 0x7fba480b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba48105bf0 con 0x7fba480b7bc0 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba48102800 con 0x7fba48101080 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.359 7fba4de7e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4809a820 con 0x7fba481023a0 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48708/0 (socket says 172.21.2.117:48708) 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 -- 172.21.2.117:0/797646814 learned_addr learned my addr 172.21.2.117:0/797646814 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 -- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba481023a0 msgr2=0x7fba48105390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba481023a0 0x7fba48105390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 -- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba48101080 msgr2=0x7fba480b75a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba48101080 0x7fba480b75a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 -- 172.21.2.117:0/797646814 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480b7fc0 con 0x7fba480b7bc0 2021-02-10T11:41:22.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba47fff700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fba40009940 tx=0x7fba4000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba40000ce0 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4000c2f0 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba4de7e700 1 -- 172.21.2.117:0/797646814 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba480b7fc0 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba4de7e700 1 -- 172.21.2.117:0/797646814 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba480b7fc0 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.360 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba4000c810 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.361 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba40001220 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.361 7fba2ffff700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 0x7fba2801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.361 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fba40032c90 con 0x7fba480b7bc0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.361 7fba477fe700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 0x7fba2801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.362 7fba477fe700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 0x7fba2801ac10 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.392 7fba4de7e700 1 -- 172.21.2.117:0/797646814 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba481027b0 con 0x7fba480b7bc0 2021-02-10T11:41:22.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.394 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba4000be30 con 0x7fba480b7bc0 2021-02-10T11:41:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.525 7fba4de7e700 1 -- 172.21.2.117:0/797646814 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7fba481027b0 con 0x7fba480b7bc0 2021-02-10T11:41:22.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.528 7fba2ffff700 1 -- 172.21.2.117:0/797646814 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v28) v1 ==== 89+0+0 (secure 0 0 0) 0x7fba40001220 con 0x7fba480b7bc0 2021-02-10T11:41:22.517 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 msgr2=0x7fba2801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 0x7fba2801ac10 crc :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 msgr2=0x7fba48106170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fba40009940 tx=0x7fba4000bcf0).stop 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 shutdown_connections 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba28018730 0x7fba2801ac10 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba480b7bc0 0x7fba48106170 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba48101080 0x7fba480b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 --2- 172.21.2.117:0/797646814 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba481023a0 0x7fba48105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:22.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 >> 172.21.2.117:0/797646814 conn(0x7fba4809ac10 msgr2=0x7fba48194fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:22.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 shutdown_connections 2021-02-10T11:41:22.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.534 7fba4de7e700 1 -- 172.21.2.117:0/797646814 wait complete. 2021-02-10T11:41:22.527 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000732851 --group_name group_0000000000957772 2021-02-10T11:41:22.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.956 7fc6209c4700 1 Processor -- start 2021-02-10T11:41:22.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.956 7fc6209c4700 1 -- start start 2021-02-10T11:41:22.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc6209c4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc61c078fa0 0x7fc61c0e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc6209c4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc61c1b0130 0x7fc61c1b0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc6209c4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61c077a10 con 0x7fc61c07da20 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc6209c4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c0e5c60 con 0x7fc61c1b0130 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc6209c4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61c0e5420 con 0x7fc61c078fa0 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc61a61c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc61c07da20 0x7fc61c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48962/0 (socket says 172.21.2.117:48962) 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc61a61c700 1 -- 172.21.2.117:0/3025110971 learned_addr learned my addr 172.21.2.117:0/3025110971 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc61ae1d700 1 --2- 172.21.2.117:0/3025110971 >> v2:172.21.2.117:3300/0 conn(0x7fc61c1b0130 0x7fc61c1b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.957 7fc619e1b700 1 --2- 172.21.2.117:0/3025110971 >> v2:172.21.2.125:3300/0 conn(0x7fc61c078fa0 0x7fc61c0e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61ae1d700 1 -- 172.21.2.117:0/3025110971 >> v2:172.21.2.125:3300/0 conn(0x7fc61c078fa0 msgr2=0x7fc61c0e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61ae1d700 1 --2- 172.21.2.117:0/3025110971 >> v2:172.21.2.125:3300/0 conn(0x7fc61c078fa0 0x7fc61c0e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61ae1d700 1 -- 172.21.2.117:0/3025110971 >> v1:172.21.2.117:6789/0 conn(0x7fc61c07da20 legacy=0x7fc61c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61ae1d700 1 -- 172.21.2.117:0/3025110971 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c0793a0 con 0x7fc61c1b0130 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61ae1d700 1 --2- 172.21.2.117:0/3025110971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c1b0130 0x7fc61c1b0530 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7fc614006e30 tx=0x7fc614009e60).ready entity=mon.0 client_cookie=90a2f3a0b1c7edcc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61961a700 1 -- 172.21.2.117:0/3025110971 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc61400b300 con 0x7fc61c1b0130 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61961a700 1 -- 172.21.2.117:0/3025110971 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614000d30 con 0x7fc61c1b0130 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc61961a700 1 -- 172.21.2.117:0/3025110971 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc61400b800 con 0x7fc61c1b0130 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc6209c4700 1 -- 172.21.2.117:0/3025110971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c1b0130 msgr2=0x7fc61c1b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.958 7fc6209c4700 1 --2- 172.21.2.117:0/3025110971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c1b0130 0x7fc61c1b0530 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7fc614006e30 tx=0x7fc614009e60).stop 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 -- 172.21.2.117:0/3025110971 shutdown_connections 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 --2- 172.21.2.117:0/3025110971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c1b0130 0x7fc61c1b0530 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 --2- 172.21.2.117:0/3025110971 >> v2:172.21.2.125:3300/0 conn(0x7fc61c078fa0 0x7fc61c0e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 -- 172.21.2.117:0/3025110971 >> 172.21.2.117:0/3025110971 conn(0x7fc61c0ce480 msgr2=0x7fc61c0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 -- 172.21.2.117:0/3025110971 shutdown_connections 2021-02-10T11:41:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.959 7fc6209c4700 1 -- 172.21.2.117:0/3025110971 wait complete. 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.973 7fc6209c4700 1 Processor -- start 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.973 7fc6209c4700 1 -- start start 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c0ff340 0x7fc61c090ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc61c0b61e0 0x7fc61c18b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c18aef0 con 0x7fc61c0ff340 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c197010 con 0x7fc61c0b61e0 2021-02-10T11:41:22.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc6209c4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c18a020 con 0x7fc61c0b65e0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46402/0 (socket says 172.21.2.117:46402) 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 -- 172.21.2.117:0/3546287608 learned_addr learned my addr 172.21.2.117:0/3546287608 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc619e1b700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc61c0b61e0 0x7fc61c18b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61a61c700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c0ff340 0x7fc61c090ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 -- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc61c0b61e0 msgr2=0x7fc61c18b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc61c0b61e0 0x7fc61c18b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 -- 172.21.2.117:0/3546287608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c0ff340 msgr2=0x7fc61c090ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.974 7fc61ae1d700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c0ff340 0x7fc61c090ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc61ae1d700 1 -- 172.21.2.117:0/3546287608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61c18a240 con 0x7fc61c0b65e0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc61ae1d700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc61400b960 tx=0x7fc61400bd00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc614009750 con 0x7fc61c0b65e0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc614003030 con 0x7fc61c0b65e0 2021-02-10T11:41:22.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c18a240 con 0x7fc61c0b65e0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6140014f0 con 0x7fc61c0b65e0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.975 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c18a240 con 0x7fc61c0b65e0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.976 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc614001270 con 0x7fc61c0b65e0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.976 7fc612ffd700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 0x7fc60001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.976 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc614032a30 con 0x7fc61c0b65e0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.976 7fc61a61c700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 0x7fc60001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:22.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:22.977 7fc61a61c700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 0x7fc60001abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:22.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.007 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61c1b7000 con 0x7fc61c0b65e0 2021-02-10T11:41:22.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.010 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc614009380 con 0x7fc61c0b65e0 2021-02-10T11:41:23.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.140 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000732851", "target": ["mgr", ""], "group_name": "group_0000000000957772"}) v1 -- 0x7fc61c18aef0 con 0x7fc6000186c0 2021-02-10T11:41:23.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.150 7fc612ffd700 1 -- 172.21.2.117:0/3546287608 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc604001a10 con 0x7fc6000186c0 2021-02-10T11:41:23.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 msgr2=0x7fc60001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 0x7fc60001abc0 crc :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 msgr2=0x7fc61c18ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fc61400b960 tx=0x7fc61400bd00).stop 2021-02-10T11:41:23.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 shutdown_connections 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc61c0b65e0 0x7fc61c18ba90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc61c0b61e0 0x7fc61c18b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6000186c0 0x7fc60001abc0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 --2- 172.21.2.117:0/3546287608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc61c0ff340 0x7fc61c090ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 >> 172.21.2.117:0/3546287608 conn(0x7fc61c0ba5e0 msgr2=0x7fc61c09ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 shutdown_connections 2021-02-10T11:41:23.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.157 7fc6209c4700 1 -- 172.21.2.117:0/3546287608 wait complete. 2021-02-10T11:41:23.151 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000957772 2021-02-10T11:41:23.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 Processor -- start 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 -- start start 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe770079000 0x7fe7700e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe7701b00f0 0x7fe7701b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe770077a70 con 0x7fe77007da20 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7700e53c0 con 0x7fe7701b00f0 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.579 7fe777c72700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770079400 con 0x7fe770079000 2021-02-10T11:41:23.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe775a8e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe77007da20 0x7fe7700771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48976/0 (socket says 172.21.2.117:48976) 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe775a8e700 1 -- 172.21.2.117:0/1143601147 learned_addr learned my addr 172.21.2.117:0/1143601147 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514209588 0 0) 0x7fe76c0010b0 con 0x7fe77007da20 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe75c002fe0 con 0x7fe77007da20 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe77628f700 1 --2- 172.21.2.117:0/1143601147 >> v2:172.21.2.125:3300/0 conn(0x7fe7701b00f0 0x7fe7701b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe77528d700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe770079000 0x7fe7700e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 601975542 0 0) 0x7fe76c0010b0 con 0x7fe77007da20 2021-02-10T11:41:23.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe770079000 msgr2=0x7fe7700e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:23.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe770079000 0x7fe7700e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 >> v2:172.21.2.125:3300/0 conn(0x7fe7701b00f0 msgr2=0x7fe7701b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:23.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.580 7fe774a8c700 1 --2- 172.21.2.117:0/1143601147 >> v2:172.21.2.125:3300/0 conn(0x7fe7701b00f0 0x7fe7701b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77007de20 con 0x7fe77007da20 2021-02-10T11:41:23.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe76c0015b0 con 0x7fe77007da20 2021-02-10T11:41:23.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 >> v1:172.21.2.117:6789/0 conn(0x7fe77007da20 legacy=0x7fe7700771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe75c003af0 0x7fe75c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe75c04a810 0x7fe75c04cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe75c04d5b0 0x7fe75c04fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c050380 con 0x7fe75c04a810 2021-02-10T11:41:23.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0505a0 con 0x7fe75c04d5b0 2021-02-10T11:41:23.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0507c0 con 0x7fe75c003af0 2021-02-10T11:41:23.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77628f700 1 --2- 172.21.2.117:0/1143601147 >> v2:172.21.2.125:3300/0 conn(0x7fe7701b00f0 0x7fe7701b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:23.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe75c04a810 0x7fe75c04cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe775a8e700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe75c003af0 0x7fe75c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77628f700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe75c04d5b0 0x7fe75c04fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 -- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe75c003af0 msgr2=0x7fe75c005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe75c003af0 0x7fe75c005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 -- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe75c04d5b0 msgr2=0x7fe75c04fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:23.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe75c04d5b0 0x7fe75c04fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 -- 172.21.2.117:0/1143601147 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c050a60 con 0x7fe75c04a810 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe775a8e700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe75c003af0 0x7fe75c005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77528d700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe75c04a810 0x7fe75c04cd10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7fe764009310 tx=0x7fe764009a90).ready entity=mon.0 client_cookie=42fb2e99e575502c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe76400c0c0 con 0x7fe75c04a810 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe764006400 con 0x7fe75c04a810 2021-02-10T11:41:23.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe774a8c700 1 -- 172.21.2.117:0/1143601147 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe76400c400 con 0x7fe75c04a810 2021-02-10T11:41:23.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe777c72700 1 -- 172.21.2.117:0/1143601147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe75c04a810 msgr2=0x7fe75c04cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe777c72700 1 --2- 172.21.2.117:0/1143601147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe75c04a810 0x7fe75c04cd10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7fe764009310 tx=0x7fe764009a90).stop 2021-02-10T11:41:23.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.581 7fe77628f700 1 -- 172.21.2.117:0/1143601147 reap_dead start 2021-02-10T11:41:23.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.582 7fe777c72700 1 -- 172.21.2.117:0/1143601147 shutdown_connections 2021-02-10T11:41:23.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.582 7fe777c72700 1 -- 172.21.2.117:0/1143601147 >> 172.21.2.117:0/1143601147 conn(0x7fe7700ce450 msgr2=0x7fe7700e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.582 7fe777c72700 1 -- 172.21.2.117:0/1143601147 shutdown_connections 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.582 7fe777c72700 1 -- 172.21.2.117:0/1143601147 wait complete. 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 Processor -- start 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 -- start start 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7700a51f0 0x7fe7700a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7700b77d0 0x7fe7700b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770189970 con 0x7fe7700a47d0 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770190aa0 con 0x7fe7700b77d0 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe777c72700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700b81f0 con 0x7fe7700a51f0 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.597 7fe775a8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48738/0 (socket says 172.21.2.117:48738) 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 -- 172.21.2.117:0/1852427882 learned_addr learned my addr 172.21.2.117:0/1852427882 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 -- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7700a51f0 msgr2=0x7fe7700a55f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7700a51f0 0x7fe7700a55f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 -- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7700b77d0 msgr2=0x7fe7700b7bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7700b77d0 0x7fe7700b7bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 -- 172.21.2.117:0/1852427882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700b8410 con 0x7fe7700a47d0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe775a8e700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7fe76c007650 tx=0x7fe76c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe76c00acf0 con 0x7fe7700a47d0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76c004350 con 0x7fe7700a47d0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe76c004870 con 0x7fe7700a47d0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe777c72700 1 -- 172.21.2.117:0/1852427882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7700b8410 con 0x7fe7700a47d0 2021-02-10T11:41:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.598 7fe777c72700 1 -- 172.21.2.117:0/1852427882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7700b8410 con 0x7fe7700a47d0 2021-02-10T11:41:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.599 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe76c004530 con 0x7fe7700a47d0 2021-02-10T11:41:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.599 7fe7627fc700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 0x7fe75801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.599 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe76c032730 con 0x7fe7700a47d0 2021-02-10T11:41:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.599 7fe77528d700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 0x7fe75801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.600 7fe77528d700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 0x7fe75801ac10 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:23.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.629 7fe777c72700 1 -- 172.21.2.117:0/1852427882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7700b9000 con 0x7fe7700a47d0 2021-02-10T11:41:23.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.632 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe76c004140 con 0x7fe7700a47d0 2021-02-10T11:41:23.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.761 7fe777c72700 1 -- 172.21.2.117:0/1852427882 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000957772"}) v1 -- 0x7fe7700b9000 con 0x7fe758018730 2021-02-10T11:41:23.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.766 7fe7627fc700 1 -- 172.21.2.117:0/1852427882 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe7640093e0 con 0x7fe758018730 2021-02-10T11:41:23.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 msgr2=0x7fe75801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 0x7fe75801ac10 crc :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 msgr2=0x7fe7700a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7fe76c007650 tx=0x7fe76c00a900).stop 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 shutdown_connections 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7700b77d0 0x7fe7700b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7700a51f0 0x7fe7700a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe758018730 0x7fe75801ac10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 --2- 172.21.2.117:0/1852427882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7700a47d0 0x7fe7700a4bd0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 >> 172.21.2.117:0/1852427882 conn(0x7fe7700b5140 msgr2=0x7fe7700a2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 shutdown_connections 2021-02-10T11:41:23.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:23.770 7fe777c72700 1 -- 172.21.2.117:0/1852427882 wait complete. 2021-02-10T11:41:23.763 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:23.764 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:24.109 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:24.455 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:24.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:24.456 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:24.456 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:24.791 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:41:24.791 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:24.791 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:41:24.791 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:24.792 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:25.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[125263]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:25.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:25.147 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:25.147 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:25.486 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:41:25.487 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:25.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:25.487 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:25.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:25.832 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:25.832 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:26.168 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:41:26.168 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:26.168 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:41:26.168 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:41:26.169 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:26.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[125936]: fuse finished with error 0 and tester_r 0 2021-02-10T11:41:26.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:26.519 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:26.519 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:26.854 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:41:26.854 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:41:26.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:26.855 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:27.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:27.189 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:27.189 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:27.524 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:27.524 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:27.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:27.525 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:27.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:27.859 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:27.859 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:28.193 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:28.194 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:28.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:41:28.194 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:28.528 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.957 7f4a5c446700 1 Processor -- start 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.957 7f4a5c446700 1 -- start start 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5c446700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4a5407da20 0x7f4a540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5c446700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 0x7f4a541b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5c446700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a54077a70 con 0x7f4a540e5180 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5c446700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a540777d0 con 0x7f4a5407da20 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5c446700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5407de20 con 0x7f4a540e4d80 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5aa63700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4a540e5180 0x7f4a54077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:48996/0 (socket says 172.21.2.117:48996) 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5aa63700 1 -- 172.21.2.117:0/2493342504 learned_addr learned my addr 172.21.2.117:0/2493342504 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a59a61700 1 --2- 172.21.2.117:0/2493342504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 0x7f4a541b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a5a262700 1 --2- 172.21.2.117:0/2493342504 >> v2:172.21.2.125:3300/0 conn(0x7f4a5407da20 0x7f4a540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.958 7f4a59260700 1 -- 172.21.2.117:0/2493342504 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952413325 0 0) 0x7f4a4c001500 con 0x7f4a540e5180 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59260700 1 -- 172.21.2.117:0/2493342504 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a40002fe0 con 0x7f4a540e5180 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59a61700 1 -- 172.21.2.117:0/2493342504 >> v2:172.21.2.125:3300/0 conn(0x7f4a5407da20 msgr2=0x7f4a540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59a61700 1 --2- 172.21.2.117:0/2493342504 >> v2:172.21.2.125:3300/0 conn(0x7f4a5407da20 0x7f4a540771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59a61700 1 -- 172.21.2.117:0/2493342504 >> v1:172.21.2.117:6789/0 conn(0x7f4a540e5180 legacy=0x7f4a54077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59a61700 1 -- 172.21.2.117:0/2493342504 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a541b06f0 con 0x7f4a540e4d80 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59a61700 1 --2- 172.21.2.117:0/2493342504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 0x7f4a541b00d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f4a44006650 tx=0x7f4a44009a80).ready entity=mon.2 client_cookie=862deac5486b011f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59260700 1 -- 172.21.2.117:0/2493342504 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4400c0c0 con 0x7f4a540e4d80 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59260700 1 -- 172.21.2.117:0/2493342504 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4400c300 con 0x7f4a540e4d80 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a59260700 1 -- 172.21.2.117:0/2493342504 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4400c820 con 0x7f4a540e4d80 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a5c446700 1 -- 172.21.2.117:0/2493342504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 msgr2=0x7f4a541b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.959 7f4a5c446700 1 --2- 172.21.2.117:0/2493342504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 0x7f4a541b00d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f4a44006650 tx=0x7f4a44009a80).stop 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 -- 172.21.2.117:0/2493342504 shutdown_connections 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 --2- 172.21.2.117:0/2493342504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a540e4d80 0x7f4a541b00d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 --2- 172.21.2.117:0/2493342504 >> v2:172.21.2.125:3300/0 conn(0x7f4a5407da20 0x7f4a540771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 -- 172.21.2.117:0/2493342504 >> 172.21.2.117:0/2493342504 conn(0x7f4a540ce430 msgr2=0x7f4a540e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 -- 172.21.2.117:0/2493342504 shutdown_connections 2021-02-10T11:41:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.960 7f4a5c446700 1 -- 172.21.2.117:0/2493342504 wait complete. 2021-02-10T11:41:28.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 Processor -- start 2021-02-10T11:41:28.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 -- start start 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a5408a180 0x7f4a5408b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a5408a580 0x7f4a5408b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5408ab90 con 0x7f4a5408bd50 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540a3180 con 0x7f4a5408a180 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5c446700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5408c150 con 0x7f4a5408a580 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5aa63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5aa63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48754/0 (socket says 172.21.2.117:48754) 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.975 7f4a5aa63700 1 -- 172.21.2.117:0/1740911213 learned_addr learned my addr 172.21.2.117:0/1740911213 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 -- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a5408a580 msgr2=0x7f4a5408b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a5408a580 0x7f4a5408b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 -- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a5408a180 msgr2=0x7f4a5408b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a5408a180 0x7f4a5408b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 -- 172.21.2.117:0/1740911213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5408c370 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5aa63700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f4a4c00a8e0 tx=0x7f4a4c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4c0070d0 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c007410 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5408c370 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a4c004bb0 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.976 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a540a5a00 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.977 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a4c00adc0 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.977 7f4a4affd700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 0x7f4a3c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.977 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a4c032c90 con 0x7f4a5408bd50 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.977 7f4a5a262700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 0x7f4a3c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:28.978 7f4a5a262700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 0x7f4a3c01abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:28.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.008 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54103300 con 0x7f4a5408bd50 2021-02-10T11:41:28.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.010 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a4c000ee0 con 0x7f4a5408bd50 2021-02-10T11:41:29.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.159 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4a54077a70 con 0x7f4a5408bd50 2021-02-10T11:41:29.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.162 7f4a4affd700 1 -- 172.21.2.117:0/1740911213 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4a4c00adc0 con 0x7f4a5408bd50 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.167 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 msgr2=0x7f4a3c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.167 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 0x7f4a3c01abd0 crc :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.167 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 msgr2=0x7f4a540a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0x7f4a4c00a8e0 tx=0x7f4a4c00ac80).stop 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 shutdown_connections 2021-02-10T11:41:29.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a3c0186d0 0x7f4a3c01abd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a5408bd50 0x7f4a540a50f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a5408a580 0x7f4a5408b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 --2- 172.21.2.117:0/1740911213 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a5408a180 0x7f4a5408b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 >> 172.21.2.117:0/1740911213 conn(0x7f4a5419f550 msgr2=0x7f4a54103530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 shutdown_connections 2021-02-10T11:41:29.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.168 7f4a5c446700 1 -- 172.21.2.117:0/1740911213 wait complete. 2021-02-10T11:41:29.163 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes' 2021-02-10T11:41:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 Processor -- start 2021-02-10T11:41:29.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 -- start start 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efda007da20 0x7efda00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efda00e5190 0x7efda0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda0077a70 con 0x7efda00e4d90 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00e5c10 con 0x7efda007da20 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.586 7efda4f10700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00777d0 con 0x7efda00e5190 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9e312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efda00e4d90 0x7efda01b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49010/0 (socket says 172.21.2.117:49010) 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9e312700 1 -- 172.21.2.117:0/2144793898 learned_addr learned my addr 172.21.2.117:0/2144793898 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 --2- 172.21.2.117:0/2144793898 >> v2:172.21.2.117:3300/0 conn(0x7efda007da20 0x7efda00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9db11700 1 -- 172.21.2.117:0/2144793898 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844188657 0 0) 0x7efd900009b0 con 0x7efda00e4d90 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9db11700 1 -- 172.21.2.117:0/2144793898 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c002fe0 con 0x7efda00e4d90 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 -- 172.21.2.117:0/2144793898 >> v2:172.21.2.125:3300/0 conn(0x7efda00e5190 msgr2=0x7efda0079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:29.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 --2- 172.21.2.117:0/2144793898 >> v2:172.21.2.125:3300/0 conn(0x7efda00e5190 0x7efda0079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 -- 172.21.2.117:0/2144793898 >> v1:172.21.2.117:6789/0 conn(0x7efda00e4d90 legacy=0x7efda01b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 -- 172.21.2.117:0/2144793898 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda01b08d0 con 0x7efda007da20 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9eb13700 1 --2- 172.21.2.117:0/2144793898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda007da20 0x7efda00771b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7efd88006650 tx=0x7efd88009a80).ready entity=mon.0 client_cookie=6cc8a3179d75e31f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9db11700 1 -- 172.21.2.117:0/2144793898 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd880009b0 con 0x7efda007da20 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9db11700 1 -- 172.21.2.117:0/2144793898 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8800c0c0 con 0x7efda007da20 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efd9db11700 1 -- 172.21.2.117:0/2144793898 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd8800c5e0 con 0x7efda007da20 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efda4f10700 1 -- 172.21.2.117:0/2144793898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda007da20 msgr2=0x7efda00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.587 7efda4f10700 1 --2- 172.21.2.117:0/2144793898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda007da20 0x7efda00771b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7efd88006650 tx=0x7efd88009a80).stop 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 -- 172.21.2.117:0/2144793898 shutdown_connections 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 --2- 172.21.2.117:0/2144793898 >> v2:172.21.2.125:3300/0 conn(0x7efda00e5190 0x7efda0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 --2- 172.21.2.117:0/2144793898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda007da20 0x7efda00771b0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 -- 172.21.2.117:0/2144793898 >> 172.21.2.117:0/2144793898 conn(0x7efda00ce440 msgr2=0x7efda00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:29.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 -- 172.21.2.117:0/2144793898 shutdown_connections 2021-02-10T11:41:29.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.588 7efda4f10700 1 -- 172.21.2.117:0/2144793898 wait complete. 2021-02-10T11:41:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.602 7efda4f10700 1 Processor -- start 2021-02-10T11:41:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.602 7efda4f10700 1 -- start start 2021-02-10T11:41:29.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efda019dfe0 0x7efda019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efda008d410 0x7efda008d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda008c080 con 0x7efda008cd50 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda008c2a0 con 0x7efda019dfe0 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efda4f10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda008c4c0 con 0x7efda008d410 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efd9eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efd9eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48764/0 (socket says 172.21.2.117:48764) 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efd9eb13700 1 -- 172.21.2.117:0/1684009454 learned_addr learned my addr 172.21.2.117:0/1684009454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efd9e312700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efda019dfe0 0x7efda019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.603 7efd9f314700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efda008d410 0x7efda008d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 -- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efda008d410 msgr2=0x7efda008d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efda008d410 0x7efda008d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 -- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efda019dfe0 msgr2=0x7efda019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efda019dfe0 0x7efda019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 -- 172.21.2.117:0/1684009454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda008c6e0 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd9eb13700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7efd88009630 tx=0x7efd8800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd880093a0 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8800bd70 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd88005560 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efda4f10700 1 -- 172.21.2.117:0/1684009454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda008c6e0 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.604 7efda4f10700 1 -- 172.21.2.117:0/1684009454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda00b70e0 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.605 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd8800a000 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.605 7efd977fe700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 0x7efd8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.605 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efd88032730 con 0x7efda008cd50 2021-02-10T11:41:29.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.605 7efd9e312700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 0x7efd8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:29.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.605 7efd9e312700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 0x7efd8401abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:29.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.636 7efda4f10700 1 -- 172.21.2.117:0/1684009454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda0077a70 con 0x7efda008cd50 2021-02-10T11:41:29.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.639 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd88014560 con 0x7efda008cd50 2021-02-10T11:41:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:29.769 7efda4f10700 1 -- 172.21.2.117:0/1684009454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]} v 0) v1 -- 0x7efda0077a70 con 0x7efda008cd50 2021-02-10T11:41:30.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.132 7efd977fe700 1 -- 172.21.2.117:0/1684009454 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_auth_id_not_created_by_mgr_volumes"]}]=0 v279) v1 ==== 166+0+0 (secure 0 0 0) 0x7efd88014560 con 0x7efda008cd50 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 -- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 msgr2=0x7efd8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 0x7efd8401abc0 crc :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 -- 172.21.2.117:0/1684009454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 msgr2=0x7efda019d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7efd88009630 tx=0x7efd8800b980).stop 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 -- 172.21.2.117:0/1684009454 shutdown_connections 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efda008d410 0x7efda008d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efda019dfe0 0x7efda019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efd840186c0 0x7efd8401abc0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 --2- 172.21.2.117:0/1684009454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efda008cd50 0x7efda019d330 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.137 7efda4f10700 1 -- 172.21.2.117:0/1684009454 >> 172.21.2.117:0/1684009454 conn(0x7efda019f4c0 msgr2=0x7efda0094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:30.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.138 7efda4f10700 1 -- 172.21.2.117:0/1684009454 shutdown_connections 2021-02-10T11:41:30.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.138 7efda4f10700 1 -- 172.21.2.117:0/1684009454 wait complete. 2021-02-10T11:41:30.132 INFO:tasks.cephfs_test_runner:test_authorize_auth_id_not_created_by_mgr_volumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:41:30.132 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:41:30.133 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:41:30.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.556 7fb802a5e700 1 Processor -- start 2021-02-10T11:41:30.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.556 7fb802a5e700 1 -- start start 2021-02-10T11:41:30.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb802a5e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 0x7fb7fc0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:30.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb802a5e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc077a70 con 0x7fb7fc07da20 2021-02-10T11:41:30.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb802a5e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc079400 con 0x7fb7fc079000 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb802a5e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc07de20 con 0x7fb7fc0d1520 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80087a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb7fc07da20 0x7fb7fc0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49022/0 (socket says 172.21.2.117:49022) 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80087a700 1 -- 172.21.2.117:0/1231989377 learned_addr learned my addr 172.21.2.117:0/1231989377 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80107b700 1 --2- 172.21.2.117:0/1231989377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 0x7fb7fc0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80107b700 1 -- 172.21.2.117:0/1231989377 >> v1:172.21.2.125:6789/0 conn(0x7fb7fc079000 legacy=0x7fb7fc0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80107b700 1 -- 172.21.2.117:0/1231989377 >> v1:172.21.2.117:6789/0 conn(0x7fb7fc07da20 legacy=0x7fb7fc0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb7fb7fe700 1 -- 172.21.2.117:0/1231989377 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712077607 0 0) 0x7fb7e40009b0 con 0x7fb7fc07da20 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80107b700 1 -- 172.21.2.117:0/1231989377 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc1b00e0 con 0x7fb7fc0d1520 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.557 7fb80107b700 1 --2- 172.21.2.117:0/1231989377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 0x7fb7fc0d1920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb7f4006e50 tx=0x7fb7f4009df0).ready entity=mon.2 client_cookie=40dfe18cd1236141 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb7fb7fe700 1 -- 172.21.2.117:0/1231989377 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7f400b510 con 0x7fb7fc0d1520 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb7fb7fe700 1 -- 172.21.2.117:0/1231989377 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f4003030 con 0x7fb7fc0d1520 2021-02-10T11:41:30.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb7fb7fe700 1 -- 172.21.2.117:0/1231989377 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7f400b7d0 con 0x7fb7fc0d1520 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 -- 172.21.2.117:0/1231989377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 msgr2=0x7fb7fc0d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 --2- 172.21.2.117:0/1231989377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 0x7fb7fc0d1920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb7f4006e50 tx=0x7fb7f4009df0).stop 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 -- 172.21.2.117:0/1231989377 shutdown_connections 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 --2- 172.21.2.117:0/1231989377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc0d1520 0x7fb7fc0d1920 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 -- 172.21.2.117:0/1231989377 >> 172.21.2.117:0/1231989377 conn(0x7fb7fc0ce440 msgr2=0x7fb7fc0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 -- 172.21.2.117:0/1231989377 shutdown_connections 2021-02-10T11:41:30.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.558 7fb802a5e700 1 -- 172.21.2.117:0/1231989377 wait complete. 2021-02-10T11:41:30.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.573 7fb802a5e700 1 Processor -- start 2021-02-10T11:41:30.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.573 7fb802a5e700 1 -- start start 2021-02-10T11:41:30.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 0x7fb7fc1a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc195300 0x7fb7fc08f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc08feb0 con 0x7fb7fc0b4c00 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0900d0 con 0x7fb7fc195700 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb802a5e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc08a7d0 con 0x7fb7fc195300 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80087a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 0x7fb7fc1a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:53160/0 (socket says 172.21.2.117:53160) 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80087a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 0x7fb7fc1a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48780/0 (socket says 172.21.2.117:48780) 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb7fbfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc195300 0x7fb7fc08f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 -- 172.21.2.117:0/4041837583 learned_addr learned my addr 172.21.2.117:0/4041837583 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:30.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 -- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc195300 msgr2=0x7fb7fc08f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc195300 0x7fb7fc08f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 -- 172.21.2.117:0/4041837583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 msgr2=0x7fb7fc1a2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 0x7fb7fc1a2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 -- 172.21.2.117:0/4041837583 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc08a9f0 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.574 7fb80107b700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb7f4006fa0 tx=0x7fb7f400bd90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7f40093d0 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f40082e0 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7f4008800 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc08a9f0 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc08a9f0 con 0x7fb7fc195700 2021-02-10T11:41:30.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.575 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb7f40084c0 con 0x7fb7fc195700 2021-02-10T11:41:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.576 7fb7f97fa700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 0x7fb7e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.576 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb7f4032a70 con 0x7fb7fc195700 2021-02-10T11:41:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.576 7fb80087a700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 0x7fb7e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:30.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.576 7fb80087a700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 0x7fb7e001abc0 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:30.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.606 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc08af80 con 0x7fb7fc195700 2021-02-10T11:41:30.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.609 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb7f401cc20 con 0x7fb7fc195700 2021-02-10T11:41:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:30.744 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7fb7fc08af80 con 0x7fb7fc195700 2021-02-10T11:41:31.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.137 7fb7f97fa700 1 -- 172.21.2.117:0/4041837583 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v280) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb7f401cc20 con 0x7fb7fc195700 2021-02-10T11:41:31.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 msgr2=0x7fb7e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 0x7fb7e001abc0 crc :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 msgr2=0x7fb7fc08a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb7f4006fa0 tx=0x7fb7f400bd90).stop 2021-02-10T11:41:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 shutdown_connections 2021-02-10T11:41:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7fc195700 0x7fb7fc08a1b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7fc195300 0x7fb7fc08f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7e00186c0 0x7fb7e001abc0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 --2- 172.21.2.117:0/4041837583 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7fc0b4c00 0x7fb7fc1a2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 >> 172.21.2.117:0/4041837583 conn(0x7fb7fc19b3e0 msgr2=0x7fb7fc09a710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 shutdown_connections 2021-02-10T11:41:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.162 7fb802a5e700 1 -- 172.21.2.117:0/4041837583 wait complete. 2021-02-10T11:41:31.157 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:41:31.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 Processor -- start 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 -- start start 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf90079000 0x7fcf900e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf901b00f0 0x7fcf901b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf90077a70 con 0x7fcf9007da20 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900e5c20 con 0x7fcf90079000 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf96dc9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900e53c0 con 0x7fcf901b00f0 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf8ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf90079000 0x7fcf900e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf8ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf90079000 0x7fcf900e4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48784/0 (socket says 172.21.2.117:48784) 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.586 7fcf8ffff700 1 -- 172.21.2.117:0/2361723188 learned_addr learned my addr 172.21.2.117:0/2361723188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:31.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf953e6700 1 --2- 172.21.2.117:0/2361723188 >> v2:172.21.2.125:3300/0 conn(0x7fcf901b00f0 0x7fcf901b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8ffff700 1 -- 172.21.2.117:0/2361723188 >> v2:172.21.2.125:3300/0 conn(0x7fcf901b00f0 msgr2=0x7fcf901b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8ffff700 1 --2- 172.21.2.117:0/2361723188 >> v2:172.21.2.125:3300/0 conn(0x7fcf901b00f0 0x7fcf901b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8ffff700 1 -- 172.21.2.117:0/2361723188 >> v1:172.21.2.117:6789/0 conn(0x7fcf9007da20 legacy=0x7fcf900771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8ffff700 1 -- 172.21.2.117:0/2361723188 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90079400 con 0x7fcf90079000 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8ffff700 1 --2- 172.21.2.117:0/2361723188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf90079000 0x7fcf900e4da0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7fcf80006700 tx=0x7fcf80009b80).ready entity=mon.0 client_cookie=ccc93956aa85390d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8f7fe700 1 -- 172.21.2.117:0/2361723188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8000c0c0 con 0x7fcf90079000 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8f7fe700 1 -- 172.21.2.117:0/2361723188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8000c440 con 0x7fcf90079000 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf8f7fe700 1 -- 172.21.2.117:0/2361723188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf8000c920 con 0x7fcf90079000 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf96dc9700 1 -- 172.21.2.117:0/2361723188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf90079000 msgr2=0x7fcf900e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.587 7fcf96dc9700 1 --2- 172.21.2.117:0/2361723188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf90079000 0x7fcf900e4da0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7fcf80006700 tx=0x7fcf80009b80).stop 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 -- 172.21.2.117:0/2361723188 shutdown_connections 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 --2- 172.21.2.117:0/2361723188 >> v2:172.21.2.125:3300/0 conn(0x7fcf901b00f0 0x7fcf901b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 --2- 172.21.2.117:0/2361723188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf90079000 0x7fcf900e4da0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 -- 172.21.2.117:0/2361723188 >> 172.21.2.117:0/2361723188 conn(0x7fcf900ce450 msgr2=0x7fcf900e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 -- 172.21.2.117:0/2361723188 shutdown_connections 2021-02-10T11:41:31.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.588 7fcf96dc9700 1 -- 172.21.2.117:0/2361723188 wait complete. 2021-02-10T11:41:31.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 Processor -- start 2021-02-10T11:41:31.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 -- start start 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf9019db70 0x7fcf900a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 0x7fcf900a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900c1360 con 0x7fcf900a1090 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900a2b00 con 0x7fcf900a20e0 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf96dc9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900a2d20 con 0x7fcf9019db70 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf94be5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf94be5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48790/0 (socket says 172.21.2.117:48790) 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf94be5700 1 -- 172.21.2.117:0/198776440 learned_addr learned my addr 172.21.2.117:0/198776440 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf953e6700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 0x7fcf900a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.603 7fcf8ffff700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf9019db70 0x7fcf900a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 -- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf9019db70 msgr2=0x7fcf900a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf9019db70 0x7fcf900a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 -- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 msgr2=0x7fcf900a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 0x7fcf900a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 -- 172.21.2.117:0/198776440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90106040 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf953e6700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 0x7fcf900a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf94be5700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7fcf78006f00 tx=0x7fcf7800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf78007070 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78007330 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf78007810 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf90106040 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.604 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf901065d0 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.605 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf78002020 con 0x7fcf900a1090 2021-02-10T11:41:31.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.605 7fcf8d7fa700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 0x7fcf7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:31.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.605 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcf78032940 con 0x7fcf900a1090 2021-02-10T11:41:31.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.605 7fcf8ffff700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 0x7fcf7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:31.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.605 7fcf8ffff700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 0x7fcf7401abc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:31.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.636 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf901b4f00 con 0x7fcf900a1090 2021-02-10T11:41:31.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.639 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf78007590 con 0x7fcf900a1090 2021-02-10T11:41:31.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.786 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcf90077a70 con 0x7fcf900a1090 2021-02-10T11:41:31.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.789 7fcf8d7fa700 1 -- 172.21.2.117:0/198776440 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf78002020 con 0x7fcf900a1090 2021-02-10T11:41:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.794 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 msgr2=0x7fcf7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.794 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 0x7fcf7401abc0 crc :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.794 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 msgr2=0x7fcf9019d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.794 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7fcf78006f00 tx=0x7fcf7800b940).stop 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 shutdown_connections 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf900a20e0 0x7fcf900a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf9019db70 0x7fcf900a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf740186c0 0x7fcf7401abc0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 --2- 172.21.2.117:0/198776440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf900a1090 0x7fcf9019d550 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 >> 172.21.2.117:0/198776440 conn(0x7fcf900ab050 msgr2=0x7fcf90107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 shutdown_connections 2021-02-10T11:41:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:31.795 7fcf96dc9700 1 -- 172.21.2.117:0/198776440 wait complete. 2021-02-10T11:41:31.789 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:41:31.790 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:32.135 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:41:32.480 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:41:32.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:32.480 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:32.481 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:32.815 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:41:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:32.815 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:41:32.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:32.816 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:32.816 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:41:33.150 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:41:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:33.151 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:41:33.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:33.151 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:33.151 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:41:33.485 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:41:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:33.485 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:41:33.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:33.486 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:41:33.486 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:41:33.819 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:41:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:41:33.819 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:41:33.820 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:41:34.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 Processor -- start 2021-02-10T11:41:34.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 -- start start 2021-02-10T11:41:34.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a2807da20 0x7f4a280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4a280e4d90 0x7f4a281b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a280e5190 0x7f4a28077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a280e5c10 con 0x7f4a280e5190 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a28077a70 con 0x7f4a280e4d90 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.244 7f4a2c95d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a280777d0 con 0x7f4a2807da20 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a280e5190 0x7f4a28077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a280e5190 0x7f4a28077ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48798/0 (socket says 172.21.2.117:48798) 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 -- 172.21.2.117:0/2634484874 learned_addr learned my addr 172.21.2.117:0/2634484874 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 -- 172.21.2.117:0/2634484874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a2807da20 msgr2=0x7f4a280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 --2- 172.21.2.117:0/2634484874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a2807da20 0x7f4a280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 -- 172.21.2.117:0/2634484874 >> v2:172.21.2.125:3300/0 conn(0x7f4a280e4d90 msgr2=0x7f4a281b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 --2- 172.21.2.117:0/2634484874 >> v2:172.21.2.125:3300/0 conn(0x7f4a280e4d90 0x7f4a281b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 -- 172.21.2.117:0/2634484874 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2807de20 con 0x7f4a280e5190 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a26e1d700 1 --2- 172.21.2.117:0/2634484874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a280e5190 0x7f4a28077ff0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f4a20006b80 tx=0x7f4a20009b90).ready entity=mon.0 client_cookie=26f9b4250b5eb3a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a2561a700 1 -- 172.21.2.117:0/2634484874 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a200022e0 con 0x7f4a280e5190 2021-02-10T11:41:34.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a2561a700 1 -- 172.21.2.117:0/2634484874 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20002560 con 0x7f4a280e5190 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a2561a700 1 -- 172.21.2.117:0/2634484874 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a20002a80 con 0x7f4a280e5190 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a2c95d700 1 -- 172.21.2.117:0/2634484874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a280e5190 msgr2=0x7f4a28077ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.245 7f4a2c95d700 1 --2- 172.21.2.117:0/2634484874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a280e5190 0x7f4a28077ff0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f4a20006b80 tx=0x7f4a20009b90).stop 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 -- 172.21.2.117:0/2634484874 shutdown_connections 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 --2- 172.21.2.117:0/2634484874 >> v2:172.21.2.125:3300/0 conn(0x7f4a280e4d90 0x7f4a281b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 --2- 172.21.2.117:0/2634484874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a280e5190 0x7f4a28077ff0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 --2- 172.21.2.117:0/2634484874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a2807da20 0x7f4a280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 -- 172.21.2.117:0/2634484874 >> 172.21.2.117:0/2634484874 conn(0x7f4a280ce440 msgr2=0x7f4a280e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 -- 172.21.2.117:0/2634484874 shutdown_connections 2021-02-10T11:41:34.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.246 7f4a2c95d700 1 -- 172.21.2.117:0/2634484874 wait complete. 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 Processor -- start 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 -- start start 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a281a5640 0x7f4a281a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 0x7f4a2819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a280ab7d0 con 0x7f4a281a27c0 2021-02-10T11:41:34.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281a5a40 con 0x7f4a2819b930 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2c95d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a28199d40 con 0x7f4a281a5640 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.261 7f4a2661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48802/0 (socket says 172.21.2.117:48802) 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 -- 172.21.2.117:0/1080355180 learned_addr learned my addr 172.21.2.117:0/1080355180 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 -- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a281a5640 msgr2=0x7f4a281a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a25e1b700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a281a5640 0x7f4a281a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a281a5640 0x7f4a281a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 -- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 msgr2=0x7f4a2819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a26e1d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 0x7f4a2819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 0x7f4a2819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 -- 172.21.2.117:0/1080355180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2819c350 con 0x7f4a281a27c0 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a26e1d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 0x7f4a2819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:34.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2661c700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f4a1800b530 tx=0x7f4a1800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a18006350 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1800bcc0 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2819c350 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2819c8e0 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.262 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4a180020d0 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.263 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4a18007000 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.263 7f4a16ffd700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 0x7f4a0c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.263 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4a18032900 con 0x7f4a281a27c0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.263 7f4a25e1b700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 0x7f4a0c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.264 7f4a25e1b700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 0x7f4a0c01ac10 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.294 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a28108c10 con 0x7f4a281a27c0 2021-02-10T11:41:34.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.297 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4a18000960 con 0x7f4a281a27c0 2021-02-10T11:41:34.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.421 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4a28108c10 con 0x7f4a281a27c0 2021-02-10T11:41:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.421 7f4a16ffd700 1 -- 172.21.2.117:0/1080355180 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v47) v1 ==== 81+0+30752 (secure 0 0 0) 0x7f4a18007000 con 0x7f4a281a27c0 2021-02-10T11:41:34.408 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 47, 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:40:56.254993", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:41:34.409 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 9, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:41:34.410 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:41:34.411 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:41:34.412 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:34.413 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:34.414 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:41:34.415 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 8, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:40:53.223172", 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:41:34.416 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:41:34.417 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "46", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:41:34.418 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:34.419 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:34.420 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:41:34.421 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 9, 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:40:54.293749", 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:41:34.422 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:41:34.423 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "47", 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:41:34.424 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:41:34.425 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:41:34.426 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:41:34.427 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.428 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:41:34.429 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.430 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.431 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.432 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.433 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:41:34.434 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.435 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.436 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.437 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.438 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.439 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.440 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.441 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 29, 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.442 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.443 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.444 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.445 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:41:34.446 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.447 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.448 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:41:34.449 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.450 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.452 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.453 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 44, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.454 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.455 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.456 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.457 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:41:34.458 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.459 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.460 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.461 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.462 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.463 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:41:34.464 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:41:34.465 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 44, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.467 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.468 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.469 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.470 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:41:34.471 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:41:34.472 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.473 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.474 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.475 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:41:34.476 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:41:34.477 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.478 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:41:34.479 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.480 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.481 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.482 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:41:34.483 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:41:34.484 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:41:34.485 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:41:34.486 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:41:34.486 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:41:34.486 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:34.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 msgr2=0x7f4a0c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 0x7f4a0c01ac10 crc :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 msgr2=0x7f4a2819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f4a1800b530 tx=0x7f4a1800b8d0).stop 2021-02-10T11:41:34.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 shutdown_connections 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a2819b930 0x7f4a2819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a281a5640 0x7f4a281a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4a0c018730 0x7f4a0c01ac10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 --2- 172.21.2.117:0/1080355180 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a281a27c0 0x7f4a2819a1e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 >> 172.21.2.117:0/1080355180 conn(0x7f4a28191110 msgr2=0x7f4a2819aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 shutdown_connections 2021-02-10T11:41:34.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.423 7f4a2c95d700 1 -- 172.21.2.117:0/1080355180 wait complete. 2021-02-10T11:41:34.489 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:34.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.913 7f1651bee700 1 Processor -- start 2021-02-10T11:41:34.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.913 7f1651bee700 1 -- start start 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f1651bee700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f164c079000 0x7f164c0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f1651bee700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164c0d1d10 con 0x7f164c07da20 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f1651bee700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f164c077a70 con 0x7f164c1b00e0 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f1651bee700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164c0e53b0 con 0x7f164c079000 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f164c07da20 0x7f164c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49060/0 (socket says 172.21.2.117:49060) 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164b7fe700 1 -- 172.21.2.117:0/1086412236 learned_addr learned my addr 172.21.2.117:0/1086412236 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113411980 0 0) 0x7f16340009b0 con 0x7f164c07da20 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1638002fe0 con 0x7f164c07da20 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3306142741 0 0) 0x7f1634001010 con 0x7f164c07da20 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 >> v2:172.21.2.125:3300/0 conn(0x7f164c079000 msgr2=0x7f164c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 --2- 172.21.2.117:0/1086412236 >> v2:172.21.2.125:3300/0 conn(0x7f164c079000 0x7f164c0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 >> v1:172.21.2.125:6789/0 conn(0x7f164c1b00e0 legacy=0x7f164c0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c079400 con 0x7f164c07da20 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1634001530 con 0x7f164c07da20 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 >> v1:172.21.2.117:6789/0 conn(0x7f164c07da20 legacy=0x7f164c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.914 7f164a7fc700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1638003ac0 0x7f1638005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 0x7f1638008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1638009590 0x7f163800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163800c300 con 0x7f1638006790 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163800c520 con 0x7f1638003ac0 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163800c740 con 0x7f1638009590 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 0x7f1638008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164b7fe700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1638003ac0 0x7f1638005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164bfff700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1638009590 0x7f163800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 -- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1638009590 msgr2=0x7f163800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1638009590 0x7f163800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 -- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1638003ac0 msgr2=0x7f1638005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1638003ac0 0x7f1638005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 -- 172.21.2.117:0/1086412236 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163800c9e0 con 0x7f1638006790 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164bfff700 1 -- 172.21.2.117:0/1086412236 reap_dead start 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164affd700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 0x7f1638008c90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f163c006bf0 tx=0x7f163c04fcd0).ready entity=mon.0 client_cookie=b6292851378a241a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f163c0009b0 con 0x7f1638006790 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c0067e0 con 0x7f1638006790 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f164a7fc700 1 -- 172.21.2.117:0/1086412236 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f163c002220 con 0x7f1638006790 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f1651bee700 1 -- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 msgr2=0x7f1638008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.915 7f1651bee700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 0x7f1638008c90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f163c006bf0 tx=0x7f163c04fcd0).stop 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.916 7f1651bee700 1 -- 172.21.2.117:0/1086412236 shutdown_connections 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.916 7f1651bee700 1 --2- 172.21.2.117:0/1086412236 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1638006790 0x7f1638008c90 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.916 7f1651bee700 1 -- 172.21.2.117:0/1086412236 >> 172.21.2.117:0/1086412236 conn(0x7f164c0ce440 msgr2=0x7f164c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.916 7f1651bee700 1 -- 172.21.2.117:0/1086412236 shutdown_connections 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.916 7f1651bee700 1 -- 172.21.2.117:0/1086412236 wait complete. 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 Processor -- start 2021-02-10T11:41:34.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 -- start start 2021-02-10T11:41:34.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f164c192ba0 0x7f164c192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164c1067a0 0x7f164c106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c193730 con 0x7f164c18cbd0 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c18d630 con 0x7f164c192ba0 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f1651bee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f164c18d850 con 0x7f164c1067a0 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f164bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f164bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48822/0 (socket says 172.21.2.117:48822) 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.931 7f164bfff700 1 -- 172.21.2.117:0/3906183809 learned_addr learned my addr 172.21.2.117:0/3906183809 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 -- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164c1067a0 msgr2=0x7f164c106ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164c1067a0 0x7f164c106ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 -- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f164c192ba0 msgr2=0x7f164c192fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f164c192ba0 0x7f164c192fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 -- 172.21.2.117:0/3906183809 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c18dc30 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f164bfff700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f1644009980 tx=0x7f164400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f164400c2f0 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164400c530 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f1651bee700 1 -- 172.21.2.117:0/3906183809 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c18dc30 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f164400ca50 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.932 7f1651bee700 1 -- 172.21.2.117:0/3906183809 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c18e1c0 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.933 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16440011f0 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.933 7f1643fff700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 0x7f163001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.933 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1644032b10 con 0x7f164c18cbd0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.933 7f164b7fe700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 0x7f163001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:34.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.934 7f164b7fe700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 0x7f163001abc0 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:34.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.963 7f1651bee700 1 -- 172.21.2.117:0/3906183809 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c077a70 con 0x7f164c18cbd0 2021-02-10T11:41:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:34.966 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1644000f40 con 0x7f164c18cbd0 2021-02-10T11:41:35.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.093 7f1651bee700 1 -- 172.21.2.117:0/3906183809 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f164c18d5f0 con 0x7f164c18cbd0 2021-02-10T11:41:35.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.094 7f1643fff700 1 -- 172.21.2.117:0/3906183809 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2093 (secure 0 0 0) 0x7f164401f880 con 0x7f164c18cbd0 2021-02-10T11:41:35.080 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:35.080 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:40:57.258727","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5657},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5657":{"gid":5657,"name":"a","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/58270881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":58270881},{"type":"v1","addr":"172.21.2.117:6833","nonce":58270881}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:41:35.080 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 29 2021-02-10T11:41:35.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 msgr2=0x7f163001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 0x7f163001abc0 crc :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 msgr2=0x7f164c18cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f1644009980 tx=0x7f164400bcd0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 shutdown_connections 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16300186c0 0x7f163001abc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f164c18cbd0 0x7f164c18cfd0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f164c1067a0 0x7f164c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 --2- 172.21.2.117:0/3906183809 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f164c192ba0 0x7f164c192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 >> 172.21.2.117:0/3906183809 conn(0x7f164c19f510 msgr2=0x7f164c1abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 shutdown_connections 2021-02-10T11:41:35.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.096 7f1651bee700 1 -- 172.21.2.117:0/3906183809 wait complete. 2021-02-10T11:41:35.090 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.515 7f693931a700 1 Processor -- start 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.515 7f693931a700 1 -- start start 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693931a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6934109600 0x7f69341b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693931a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69340e4190 0x7f69340e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693931a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6934109ab0 con 0x7f69340d0ac0 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693931a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69341b3c60 con 0x7f69340e4190 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693931a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69340d0ec0 con 0x7f6934109600 2021-02-10T11:41:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69340e4190 0x7f69340e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693307d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f69340d0ac0 0x7f69341b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49080/0 (socket says 172.21.2.117:49080) 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69340e4190 0x7f69340e4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48834/0 (socket says 172.21.2.117:48834) 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693307d700 1 -- 172.21.2.117:0/793448126 learned_addr learned my addr 172.21.2.117:0/793448126 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 -- 172.21.2.117:0/793448126 >> v2:172.21.2.125:3300/0 conn(0x7f6934109600 msgr2=0x7f69341b3640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 --2- 172.21.2.117:0/793448126 >> v2:172.21.2.125:3300/0 conn(0x7f6934109600 0x7f69341b3640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 -- 172.21.2.117:0/793448126 >> v1:172.21.2.117:6789/0 conn(0x7f69340d0ac0 legacy=0x7f69341b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 -- 172.21.2.117:0/793448126 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693410b040 con 0x7f69340e4190 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.516 7f693387e700 1 --2- 172.21.2.117:0/793448126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69340e4190 0x7f69340e4590 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f692c006e30 tx=0x7f692c009de0).ready entity=mon.0 client_cookie=832eef90702a41ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693207b700 1 -- 172.21.2.117:0/793448126 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f692c00b300 con 0x7f69340e4190 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693207b700 1 -- 172.21.2.117:0/793448126 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c00b680 con 0x7f69340e4190 2021-02-10T11:41:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693207b700 1 -- 172.21.2.117:0/793448126 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f692c00bb60 con 0x7f69340e4190 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 -- 172.21.2.117:0/793448126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69340e4190 msgr2=0x7f69340e4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 --2- 172.21.2.117:0/793448126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69340e4190 0x7f69340e4590 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f692c006e30 tx=0x7f692c009de0).stop 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 -- 172.21.2.117:0/793448126 shutdown_connections 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 --2- 172.21.2.117:0/793448126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69340e4190 0x7f69340e4590 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 --2- 172.21.2.117:0/793448126 >> v2:172.21.2.125:3300/0 conn(0x7f6934109600 0x7f69341b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 -- 172.21.2.117:0/793448126 >> 172.21.2.117:0/793448126 conn(0x7f69340ce410 msgr2=0x7f69340d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 -- 172.21.2.117:0/793448126 shutdown_connections 2021-02-10T11:41:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.517 7f693931a700 1 -- 172.21.2.117:0/793448126 wait complete. 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.531 7f693931a700 1 Processor -- start 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 -- start start 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f693408b0b0 0x7f693408b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6934092100 0x7f6934092500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693419e9c0 con 0x7f693408a690 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693419e4d0 con 0x7f693408b0b0 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693931a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693419e6b0 con 0x7f6934092100 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48836/0 (socket says 172.21.2.117:48836) 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 -- 172.21.2.117:0/2438832765 learned_addr learned my addr 172.21.2.117:0/2438832765 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:35.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 -- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6934092100 msgr2=0x7f6934092500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6934092100 0x7f6934092500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 -- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f693408b0b0 msgr2=0x7f693408b4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f693408b0b0 0x7f693408b4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.532 7f693307d700 1 -- 172.21.2.117:0/2438832765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934092b20 con 0x7f693408a690 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f693307d700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f691c001c60 tx=0x7f691c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691c00acc0 con 0x7f693408a690 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c004140 con 0x7f693408a690 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691c004620 con 0x7f693408a690 2021-02-10T11:41:35.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f693931a700 1 -- 172.21.2.117:0/2438832765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934092b20 con 0x7f693408a690 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f693931a700 1 -- 172.21.2.117:0/2438832765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6934092b20 con 0x7f693408a690 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.533 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f691c000ce0 con 0x7f693408a690 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.534 7f692bfff700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 0x7f691801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.534 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f691c032550 con 0x7f693408a690 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.534 7f693287c700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 0x7f691801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:35.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.535 7f693287c700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 0x7f691801abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.566 7f693931a700 1 -- 172.21.2.117:0/2438832765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693419e6b0 con 0x7f693408a690 2021-02-10T11:41:35.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.569 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f691c0009b0 con 0x7f693408a690 2021-02-10T11:41:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:35.696 7f693931a700 1 -- 172.21.2.117:0/2438832765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f693419e6b0 con 0x7f693408a690 2021-02-10T11:41:36.466 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:41:36.480 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:41:36.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.485 7f692bfff700 1 -- 172.21.2.117:0/2438832765 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7f691c000ce0 con 0x7f693408a690 2021-02-10T11:41:36.500 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:41:36.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 -- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 msgr2=0x7f691801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 0x7f691801abc0 crc :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 -- 172.21.2.117:0/2438832765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 msgr2=0x7f693408aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7f691c001c60 tx=0x7f691c00a8d0).stop 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 -- 172.21.2.117:0/2438832765 shutdown_connections 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6934092100 0x7f6934092500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f693408b0b0 0x7f693408b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69180186c0 0x7f691801abc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 --2- 172.21.2.117:0/2438832765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f693408a690 0x7f693408aa90 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.516 7f693931a700 1 -- 172.21.2.117:0/2438832765 >> 172.21.2.117:0/2438832765 conn(0x7f693419f5a0 msgr2=0x7f69341a6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:36.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.517 7f693931a700 1 -- 172.21.2.117:0/2438832765 shutdown_connections 2021-02-10T11:41:36.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.517 7f693931a700 1 -- 172.21.2.117:0/2438832765 wait complete. 2021-02-10T11:41:36.512 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:36.524 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:41:36.529 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:36.532 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:41:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.941 7f1423ca1700 1 Processor -- start 2021-02-10T11:41:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 -- start start 2021-02-10T11:41:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f141c07da20 0x7f141c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f141c0e4d90 0x7f141c1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141c077a70 con 0x7f141c0e5190 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141c0777d0 con 0x7f141c07da20 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1423ca1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f141c07de20 con 0x7f141c0e4d90 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f14222be700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f141c0e5190 0x7f141c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49106/0 (socket says 172.21.2.117:49106) 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f1421abd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f141c07da20 0x7f141c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.942 7f14222be700 1 -- 172.21.2.117:0/1345960899 learned_addr learned my addr 172.21.2.117:0/1345960899 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f14212bc700 1 --2- 172.21.2.117:0/1345960899 >> v2:172.21.2.125:3300/0 conn(0x7f141c0e4d90 0x7f141c1b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1420abb700 1 -- 172.21.2.117:0/1345960899 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343158350 0 0) 0x7f1414001500 con 0x7f141c0e5190 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1420abb700 1 -- 172.21.2.117:0/1345960899 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408002fe0 con 0x7f141c0e5190 2021-02-10T11:41:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1421abd700 1 -- 172.21.2.117:0/1345960899 >> v2:172.21.2.125:3300/0 conn(0x7f141c0e4d90 msgr2=0x7f141c1b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1421abd700 1 --2- 172.21.2.117:0/1345960899 >> v2:172.21.2.125:3300/0 conn(0x7f141c0e4d90 0x7f141c1b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1421abd700 1 -- 172.21.2.117:0/1345960899 >> v1:172.21.2.117:6789/0 conn(0x7f141c0e5190 legacy=0x7f141c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1421abd700 1 -- 172.21.2.117:0/1345960899 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c1b0700 con 0x7f141c07da20 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1421abd700 1 --2- 172.21.2.117:0/1345960899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c07da20 0x7f141c0771b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7f1418006650 tx=0x7f1418009a80).ready entity=mon.0 client_cookie=4261cc8bbbf1cd83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1420abb700 1 -- 172.21.2.117:0/1345960899 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14180009b0 con 0x7f141c07da20 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1420abb700 1 -- 172.21.2.117:0/1345960899 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141800c0c0 con 0x7f141c07da20 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.943 7f1420abb700 1 -- 172.21.2.117:0/1345960899 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141800c5e0 con 0x7f141c07da20 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 -- 172.21.2.117:0/1345960899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c07da20 msgr2=0x7f141c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 --2- 172.21.2.117:0/1345960899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c07da20 0x7f141c0771b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7f1418006650 tx=0x7f1418009a80).stop 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 -- 172.21.2.117:0/1345960899 shutdown_connections 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 --2- 172.21.2.117:0/1345960899 >> v2:172.21.2.125:3300/0 conn(0x7f141c0e4d90 0x7f141c1b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 --2- 172.21.2.117:0/1345960899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c07da20 0x7f141c0771b0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:36.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 -- 172.21.2.117:0/1345960899 >> 172.21.2.117:0/1345960899 conn(0x7f141c0ce440 msgr2=0x7f141c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:36.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 -- 172.21.2.117:0/1345960899 shutdown_connections 2021-02-10T11:41:36.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.944 7f1423ca1700 1 -- 172.21.2.117:0/1345960899 wait complete. 2021-02-10T11:41:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 Processor -- start 2021-02-10T11:41:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 -- start start 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f141c1067a0 0x7f141c106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f141c18cbd0 0x7f141c18cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c193730 con 0x7f141c192ba0 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c18d630 con 0x7f141c18cbd0 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.959 7f1423ca1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141c18d850 con 0x7f141c1067a0 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48860/0 (socket says 172.21.2.117:48860) 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 -- 172.21.2.117:0/894924606 learned_addr learned my addr 172.21.2.117:0/894924606 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 -- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f141c1067a0 msgr2=0x7f141c106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f141c1067a0 0x7f141c106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 -- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f141c18cbd0 msgr2=0x7f141c18cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f141c18cbd0 0x7f141c18cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 -- 172.21.2.117:0/894924606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c18dc30 con 0x7f141c192ba0 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1421abd700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f1418009630 tx=0x7f141800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14180093a0 con 0x7f141c192ba0 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141800bd70 con 0x7f141c192ba0 2021-02-10T11:41:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1418005560 con 0x7f141c192ba0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.960 7f1423ca1700 1 -- 172.21.2.117:0/894924606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c18dc30 con 0x7f141c192ba0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.961 7f1423ca1700 1 -- 172.21.2.117:0/894924606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c18dc30 con 0x7f141c192ba0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.961 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14180054f0 con 0x7f141c192ba0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.961 7f14127fc700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 0x7f140401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.961 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f1418032710 con 0x7f141c192ba0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.962 7f14212bc700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 0x7f140401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.962 7f14212bc700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 0x7f140401abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.992 7f1423ca1700 1 -- 172.21.2.117:0/894924606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141c077a70 con 0x7f141c192ba0 2021-02-10T11:41:36.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:36.994 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1418026c40 con 0x7f141c192ba0 2021-02-10T11:41:37.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.122 7f1423ca1700 1 -- 172.21.2.117:0/894924606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f141c077a70 con 0x7f141c192ba0 2021-02-10T11:41:37.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.123 7f14127fc700 1 -- 172.21.2.117:0/894924606 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1754 (secure 0 0 0) 0x7f1418026c40 con 0x7f141c192ba0 2021-02-10T11:41:37.109 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:37.109 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/2657194791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2657194791},{"type":"v1","addr":"172.21.2.125:6835","nonce":2657194791}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:40:55.253229","modified":"2021-02-10 11:41:36.480830","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2021-02-10T11:41:37.109 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 30 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 msgr2=0x7f140401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 0x7f140401abc0 crc :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 msgr2=0x7f141c192fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7f1418009630 tx=0x7f141800b980).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 shutdown_connections 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f141c18cbd0 0x7f141c18cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f141c1067a0 0x7f141c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14040186c0 0x7f140401abc0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 --2- 172.21.2.117:0/894924606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f141c192ba0 0x7f141c192fa0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 >> 172.21.2.117:0/894924606 conn(0x7f141c19f510 msgr2=0x7f141c1abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:37.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 shutdown_connections 2021-02-10T11:41:37.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.125 7f1423ca1700 1 -- 172.21.2.117:0/894924606 wait complete. 2021-02-10T11:41:37.119 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 Processor -- start 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 -- start start 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a140d0ac0 0x7f7a141b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a141b3470 0x7f7a141b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a14109ab0 con 0x7f7a141085f0 2021-02-10T11:41:37.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140d0ec0 con 0x7f7a141b3470 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.549 7f7a1aed6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140d10b0 con 0x7f7a140d0ac0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a140d0ac0 0x7f7a141b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a194f3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7a141085f0 0x7f7a141089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49120/0 (socket says 172.21.2.117:49120) 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a194f3700 1 -- 172.21.2.117:0/3480529603 learned_addr learned my addr 172.21.2.117:0/3480529603 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a13fff700 1 --2- 172.21.2.117:0/3480529603 >> v2:172.21.2.117:3300/0 conn(0x7f7a141b3470 0x7f7a141b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 -- 172.21.2.117:0/3480529603 >> v2:172.21.2.117:3300/0 conn(0x7f7a141b3470 msgr2=0x7f7a141b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 --2- 172.21.2.117:0/3480529603 >> v2:172.21.2.117:3300/0 conn(0x7f7a141b3470 0x7f7a141b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 -- 172.21.2.117:0/3480529603 >> v1:172.21.2.117:6789/0 conn(0x7f7a141085f0 legacy=0x7f7a141089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 -- 172.21.2.117:0/3480529603 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140e51a0 con 0x7f7a140d0ac0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a18cf2700 1 --2- 172.21.2.117:0/3480529603 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a140d0ac0 0x7f7a141b2e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7a04006650 tx=0x7f7a04009a80).ready entity=mon.1 client_cookie=bc357ed2c75f703d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a137fe700 1 -- 172.21.2.117:0/3480529603 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a0400c0c0 con 0x7f7a140d0ac0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a137fe700 1 -- 172.21.2.117:0/3480529603 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0400c340 con 0x7f7a140d0ac0 2021-02-10T11:41:37.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a137fe700 1 -- 172.21.2.117:0/3480529603 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a0400c860 con 0x7f7a140d0ac0 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a1aed6700 1 -- 172.21.2.117:0/3480529603 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a140d0ac0 msgr2=0x7f7a141b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.550 7f7a1aed6700 1 --2- 172.21.2.117:0/3480529603 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a140d0ac0 0x7f7a141b2e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7a04006650 tx=0x7f7a04009a80).stop 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 -- 172.21.2.117:0/3480529603 shutdown_connections 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 --2- 172.21.2.117:0/3480529603 >> v2:172.21.2.117:3300/0 conn(0x7f7a141b3470 0x7f7a141b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 --2- 172.21.2.117:0/3480529603 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a140d0ac0 0x7f7a141b2e50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 -- 172.21.2.117:0/3480529603 >> 172.21.2.117:0/3480529603 conn(0x7f7a140ce410 msgr2=0x7f7a140d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 -- 172.21.2.117:0/3480529603 shutdown_connections 2021-02-10T11:41:37.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.551 7f7a1aed6700 1 -- 172.21.2.117:0/3480529603 wait complete. 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 Processor -- start 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 -- start start 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 0x7f7a141ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a141ac350 0x7f7a141ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140a8150 con 0x7f7a141ae380 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140a8300 con 0x7f7a141aeda0 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a1aed6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141acd70 con 0x7f7a141ac350 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a13fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a18cf2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 0x7f7a141ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a13fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:53260/0 (socket says 172.21.2.117:53260) 2021-02-10T11:41:37.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a13fff700 1 -- 172.21.2.117:0/3758281626 learned_addr learned my addr 172.21.2.117:0/3758281626 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.566 7f7a18cf2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 0x7f7a141ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48874/0 (socket says 172.21.2.117:48874) 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a194f3700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a141ac350 0x7f7a141ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 -- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a141ac350 msgr2=0x7f7a141ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a141ac350 0x7f7a141ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 -- 172.21.2.117:0/3758281626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 msgr2=0x7f7a141ae780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 0x7f7a141ae780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 -- 172.21.2.117:0/3758281626 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a141acf90 con 0x7f7a141aeda0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a13fff700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7a0000b560 tx=0x7f7a0000b900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a0000bcf0 con 0x7f7a141aeda0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00000c90 con 0x7f7a141aeda0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a141acf90 con 0x7f7a141aeda0 2021-02-10T11:41:37.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a141ad230 con 0x7f7a141aeda0 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.567 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a000022b0 con 0x7f7a141aeda0 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.568 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a00006b00 con 0x7f7a141aeda0 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.568 7f7a117fa700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 0x7f79f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.568 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f7a00032780 con 0x7f7a141aeda0 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.568 7f7a18cf2700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 0x7f79f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:37.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.568 7f7a18cf2700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 0x7f79f001abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:37.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.600 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a140eda30 con 0x7f7a141aeda0 2021-02-10T11:41:37.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.604 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a0002e3d0 con 0x7f7a141aeda0 2021-02-10T11:41:37.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:37.732 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7a14109ab0 con 0x7f7a141aeda0 2021-02-10T11:41:38.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.491 7f7a117fa700 1 -- 172.21.2.117:0/3758281626 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7a0002e3d0 con 0x7f7a141aeda0 2021-02-10T11:41:38.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.504 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 msgr2=0x7f79f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.504 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 0x7f79f001abc0 crc :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.504 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 msgr2=0x7f7a141af1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.504 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7a0000b560 tx=0x7f7a0000b900).stop 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 shutdown_connections 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a141ac350 0x7f7a141ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a141aeda0 0x7f7a141af1a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79f00186c0 0x7f79f001abc0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 --2- 172.21.2.117:0/3758281626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a141ae380 0x7f7a141ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 >> 172.21.2.117:0/3758281626 conn(0x7f7a1419c280 msgr2=0x7f7a140edc60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 shutdown_connections 2021-02-10T11:41:38.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.505 7f7a1aed6700 1 -- 172.21.2.117:0/3758281626 wait complete. 2021-02-10T11:41:38.498 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:41:38.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 Processor -- start 2021-02-10T11:41:38.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 -- start start 2021-02-10T11:41:38.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 0x7f4f540e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4f541b0120 0x7f4f541b0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f54077a10 con 0x7f4f5407da20 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540e5410 con 0x7f4f541b0120 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.927 7f4f5ad9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540793a0 con 0x7f4f54078fa0 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f593bc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4f541b0120 0x7f4f541b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f58bbb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4f5407da20 0x7f4f540771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49132/0 (socket says 172.21.2.117:49132) 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f58bbb700 1 -- 172.21.2.117:0/2701784359 learned_addr learned my addr 172.21.2.117:0/2701784359 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f537fe700 1 -- 172.21.2.117:0/2701784359 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359843179 0 0) 0x7f4f3c001180 con 0x7f4f5407da20 2021-02-10T11:41:38.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f537fe700 1 -- 172.21.2.117:0/2701784359 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f40002fe0 con 0x7f4f5407da20 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f53fff700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 0x7f4f540e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f593bc700 1 -- 172.21.2.117:0/2701784359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 msgr2=0x7f4f540e4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f593bc700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 0x7f4f540e4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f593bc700 1 -- 172.21.2.117:0/2701784359 >> v1:172.21.2.117:6789/0 conn(0x7f4f5407da20 legacy=0x7f4f540771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f593bc700 1 -- 172.21.2.117:0/2701784359 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5407de20 con 0x7f4f541b0120 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.928 7f4f53fff700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 0x7f4f540e4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f593bc700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f541b0120 0x7f4f541b0520 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7f4f4c006e30 tx=0x7f4f4c009e60).ready entity=mon.0 client_cookie=3f084e86f8968bfa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f537fe700 1 -- 172.21.2.117:0/2701784359 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f4c006a50 con 0x7f4f541b0120 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f537fe700 1 -- 172.21.2.117:0/2701784359 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c00b300 con 0x7f4f541b0120 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f537fe700 1 -- 172.21.2.117:0/2701784359 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f4c00b820 con 0x7f4f541b0120 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f5ad9f700 1 -- 172.21.2.117:0/2701784359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f541b0120 msgr2=0x7f4f541b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.929 7f4f5ad9f700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f541b0120 0x7f4f541b0520 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0x7f4f4c006e30 tx=0x7f4f4c009e60).stop 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 -- 172.21.2.117:0/2701784359 shutdown_connections 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f541b0120 0x7f4f541b0520 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 --2- 172.21.2.117:0/2701784359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f54078fa0 0x7f4f540e4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 -- 172.21.2.117:0/2701784359 >> 172.21.2.117:0/2701784359 conn(0x7f4f540ce470 msgr2=0x7f4f540e42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 -- 172.21.2.117:0/2701784359 shutdown_connections 2021-02-10T11:41:38.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.930 7f4f5ad9f700 1 -- 172.21.2.117:0/2701784359 wait complete. 2021-02-10T11:41:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 Processor -- start 2021-02-10T11:41:38.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 -- start start 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 0x7f4f540bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 0x7f4f5419d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540c09e0 con 0x7f4f540c0370 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540b1350 con 0x7f4f5419d390 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f5ad9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540b0f20 con 0x7f4f541b8bb0 2021-02-10T11:41:38.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f53fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f53fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48888/0 (socket says 172.21.2.117:48888) 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.945 7f4f53fff700 1 -- 172.21.2.117:0/3805422163 learned_addr learned my addr 172.21.2.117:0/3805422163 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f593bc700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 0x7f4f5419d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f58bbb700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 0x7f4f540bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 -- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 msgr2=0x7f4f540bfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 0x7f4f540bfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 -- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 msgr2=0x7f4f5419d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 0x7f4f5419d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 -- 172.21.2.117:0/3805422163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f540c0f60 con 0x7f4f540c0370 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f58bbb700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 0x7f4f540bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f593bc700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 0x7f4f5419d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f53fff700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f4f44006f20 tx=0x7f4f4400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.946 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f44006bb0 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4400bd80 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4f440022b0 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f540c0f60 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f540c14f0 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4f44000c90 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f517fa700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 0x7f4f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.947 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f4f44032730 con 0x7f4f540c0370 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.948 7f4f58bbb700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 0x7f4f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.948 7f4f58bbb700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 0x7f4f3801abc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:38.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.978 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f541b7090 con 0x7f4f540c0370 2021-02-10T11:41:38.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:38.980 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4f44000940 con 0x7f4f540c0370 2021-02-10T11:41:39.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.105 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4f54077a10 con 0x7f4f540c0370 2021-02-10T11:41:39.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.498 7f4f517fa700 1 -- 172.21.2.117:0/3805422163 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4f44000c90 con 0x7f4f540c0370 2021-02-10T11:41:39.509 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:41:39.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 msgr2=0x7f4f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 0x7f4f3801abc0 crc :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 msgr2=0x7f4f5419cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f4f44006f20 tx=0x7f4f4400b990).stop 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 shutdown_connections 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4f380186c0 0x7f4f3801abc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4f540c0370 0x7f4f5419cd70 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4f5419d390 0x7f4f5419d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 --2- 172.21.2.117:0/3805422163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4f541b8bb0 0x7f4f540bfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 >> 172.21.2.117:0/3805422163 conn(0x7f4f540b9560 msgr2=0x7f4f540a53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 shutdown_connections 2021-02-10T11:41:39.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.525 7f4f5ad9f700 1 -- 172.21.2.117:0/3805422163 wait complete. 2021-02-10T11:41:39.519 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:41:39.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 Processor -- start 2021-02-10T11:41:39.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 -- start start 2021-02-10T11:41:39.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 0x7f3a941b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3a940e5200 0x7f3a94077f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a940777d0 con 0x7f3a9407da20 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a94077a10 con 0x7f3a940e5200 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.948 7f3a9c073700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9407de20 con 0x7f3a940e4e00 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a99e8f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a99e8f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48896/0 (socket says 172.21.2.117:48896) 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a99e8f700 1 -- 172.21.2.117:0/2771115288 learned_addr learned my addr 172.21.2.117:0/2771115288 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 --2- 172.21.2.117:0/2771115288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 0x7f3a941b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9a690700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.125:3300/0 conn(0x7f3a940e5200 0x7f3a94077f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 -- 172.21.2.117:0/2771115288 >> v2:172.21.2.125:3300/0 conn(0x7f3a940e5200 msgr2=0x7f3a94077f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.125:3300/0 conn(0x7f3a940e5200 0x7f3a94077f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 -- 172.21.2.117:0/2771115288 >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 msgr2=0x7f3a940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:39.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 -- 172.21.2.117:0/2771115288 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a941b0750 con 0x7f3a940e4e00 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a99e8f700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.949 7f3a9968e700 1 --2- 172.21.2.117:0/2771115288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 0x7f3a941b0130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3a90006650 tx=0x7f3a90009a80).ready entity=mon.2 client_cookie=6ecccd44d43f5ed8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a98e8d700 1 -- 172.21.2.117:0/2771115288 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a900009b0 con 0x7f3a940e4e00 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a98e8d700 1 -- 172.21.2.117:0/2771115288 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9000c0c0 con 0x7f3a940e4e00 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a98e8d700 1 -- 172.21.2.117:0/2771115288 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9000c510 con 0x7f3a940e4e00 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 -- 172.21.2.117:0/2771115288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 msgr2=0x7f3a941b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 --2- 172.21.2.117:0/2771115288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 0x7f3a941b0130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3a90006650 tx=0x7f3a90009a80).stop 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 -- 172.21.2.117:0/2771115288 shutdown_connections 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.125:3300/0 conn(0x7f3a940e5200 0x7f3a94077f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 --2- 172.21.2.117:0/2771115288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a940e4e00 0x7f3a941b0130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 --2- 172.21.2.117:0/2771115288 >> v2:172.21.2.117:3300/0 conn(0x7f3a9407da20 0x7f3a940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 -- 172.21.2.117:0/2771115288 >> 172.21.2.117:0/2771115288 conn(0x7f3a940ce480 msgr2=0x7f3a940e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:39.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 -- 172.21.2.117:0/2771115288 shutdown_connections 2021-02-10T11:41:39.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.950 7f3a9c073700 1 -- 172.21.2.117:0/2771115288 wait complete. 2021-02-10T11:41:39.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 Processor -- start 2021-02-10T11:41:39.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 -- start start 2021-02-10T11:41:39.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a94190a20 0x7f3a940ba260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9408d980 0x7f3a941a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94090700 con 0x7f3a941a4640 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9408dd80 con 0x7f3a9408d980 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9c073700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9408dfa0 con 0x7f3a94190a20 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9a690700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9a690700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48902/0 (socket says 172.21.2.117:48902) 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.965 7f3a9a690700 1 -- 172.21.2.117:0/1636396128 learned_addr learned my addr 172.21.2.117:0/1636396128 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a99e8f700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a94190a20 0x7f3a940ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9968e700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9408d980 0x7f3a941a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 -- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a94190a20 msgr2=0x7f3a940ba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a94190a20 0x7f3a940ba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 -- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9408d980 msgr2=0x7f3a941a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9408d980 0x7f3a941a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 -- 172.21.2.117:0/1636396128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9408ed40 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9a690700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f3a8c00b970 tx=0x7f3a8c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a8c000f90 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c006aa0 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a8c001680 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9408ed40 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.966 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9408ed40 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.967 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a8c000df0 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.967 7f3a867fc700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 0x7f3a7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.967 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f3a8c032b30 con 0x7f3a941a4640 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.967 7f3a99e8f700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 0x7f3a7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:39.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.967 7f3a99e8f700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 0x7f3a7c01abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:39.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:39.998 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a94077a10 con 0x7f3a941a4640 2021-02-10T11:41:39.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.000 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a8c0066f0 con 0x7f3a941a4640 2021-02-10T11:41:40.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.128 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3a94077a10 con 0x7f3a941a4640 2021-02-10T11:41:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.501 7f3a867fc700 1 -- 172.21.2.117:0/1636396128 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3a8c000df0 con 0x7f3a941a4640 2021-02-10T11:41:40.531 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.547 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 msgr2=0x7f3a7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.547 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 0x7f3a7c01abc0 crc :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.547 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 msgr2=0x7f3a9408e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.547 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f3a8c00b970 tx=0x7f3a8c00bd10).stop 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.547 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 shutdown_connections 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c018730 0x7f3a7c01abc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a941a4640 0x7f3a9408e720 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9408d980 0x7f3a941a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 --2- 172.21.2.117:0/1636396128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a94190a20 0x7f3a940ba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:40.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 >> 172.21.2.117:0/1636396128 conn(0x7f3a940c0420 msgr2=0x7f3a94097c00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:40.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 shutdown_connections 2021-02-10T11:41:40.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:40.548 7f3a9c073700 1 -- 172.21.2.117:0/1636396128 wait complete. 2021-02-10T11:41:40.542 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:41:40.542 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:41:40.542 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:41:40.542 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:41:40.542 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:41:40.542 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:41:40.543 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:41:40.543 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:41:40.548 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:41:40.548 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:41:40.549 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:41:40.549 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:41:40.591 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:41:40.591 DEBUG:teuthology.parallel:result is None 2021-02-10T11:41:40.591 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:41:40.592 DEBUG:teuthology.parallel:result is None 2021-02-10T11:41:40.592 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:41:40.997 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:41:41.000 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 Processor -- start 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 -- start start 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc5b407da20 0x7fc5b40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b40d1d00 con 0x7fc5b40e5180 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b4077a70 con 0x7fc5b40e4d80 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.020 7fc5b919c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b40777d0 con 0x7fc5b407da20 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b359e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc5b40e5180 0x7fc5b40d0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49170/0 (socket says 172.21.2.117:49170) 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b359e700 1 -- 172.21.2.117:0/4129810877 learned_addr learned my addr 172.21.2.117:0/4129810877 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b2d9d700 1 --2- 172.21.2.117:0/4129810877 >> v2:172.21.2.125:3300/0 conn(0x7fc5b407da20 0x7fc5b40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986669092 0 0) 0x7fc5ac001400 con 0x7fc5b40e4d80 2021-02-10T11:41:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59c002fe0 con 0x7fc5b40e4d80 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717130869 0 0) 0x7fc5a4000c10 con 0x7fc5b40e5180 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59c004370 con 0x7fc5b40e5180 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b2d9d700 1 -- 172.21.2.117:0/4129810877 >> v1:172.21.2.125:6789/0 conn(0x7fc5b40e4d80 legacy=0x7fc5b41b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b2d9d700 1 -- 172.21.2.117:0/4129810877 >> v1:172.21.2.117:6789/0 conn(0x7fc5b40e5180 legacy=0x7fc5b40d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b2d9d700 1 -- 172.21.2.117:0/4129810877 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b41b08c0 con 0x7fc5b407da20 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.021 7fc5b2d9d700 1 --2- 172.21.2.117:0/4129810877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b407da20 0x7fc5b40771b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc5a8006650 tx=0x7fc5a8009a00).ready entity=mon.1 client_cookie=c8056bf217d1679d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.022 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5a8009df0 con 0x7fc5b407da20 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.022 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a80009b0 con 0x7fc5b407da20 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.022 7fc5b1d9b700 1 -- 172.21.2.117:0/4129810877 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5a800c2b0 con 0x7fc5b407da20 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.022 7fc5b919c700 1 -- 172.21.2.117:0/4129810877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b407da20 msgr2=0x7fc5b40771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.022 7fc5b919c700 1 --2- 172.21.2.117:0/4129810877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b407da20 0x7fc5b40771b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc5a8006650 tx=0x7fc5a8009a00).stop 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.023 7fc5b919c700 1 -- 172.21.2.117:0/4129810877 shutdown_connections 2021-02-10T11:41:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.023 7fc5b919c700 1 --2- 172.21.2.117:0/4129810877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b407da20 0x7fc5b40771b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.023 7fc5b919c700 1 -- 172.21.2.117:0/4129810877 >> 172.21.2.117:0/4129810877 conn(0x7fc5b40ce430 msgr2=0x7fc5b40e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:41.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.023 7fc5b919c700 1 -- 172.21.2.117:0/4129810877 shutdown_connections 2021-02-10T11:41:41.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.023 7fc5b919c700 1 -- 172.21.2.117:0/4129810877 wait complete. 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.037 7fc5b919c700 1 Processor -- start 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.037 7fc5b919c700 1 -- start start 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5b4095360 0x7fc5b40935b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b4093bd0 0x7fc5b41a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b41a76c0 con 0x7fc5b40aa760 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b4093fd0 con 0x7fc5b4093bd0 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b919c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b41a81e0 con 0x7fc5b4095360 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48924/0 (socket says 172.21.2.117:48924) 2021-02-10T11:41:41.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 -- 172.21.2.117:0/1592709660 learned_addr learned my addr 172.21.2.117:0/1592709660 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b259c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5b4095360 0x7fc5b40935b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b359e700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b4093bd0 0x7fc5b41a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 -- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5b4095360 msgr2=0x7fc5b40935b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5b4095360 0x7fc5b40935b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 -- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b4093bd0 msgr2=0x7fc5b41a7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b4093bd0 0x7fc5b41a7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 -- 172.21.2.117:0/1592709660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b41a8400 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.038 7fc5b2d9d700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc5a80096a0 tx=0x7fc5a800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5a80093a0 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a80067f0 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc5a80022b0 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b41a8400 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b41a8400 con 0x7fc5b40aa760 2021-02-10T11:41:41.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc5a80052e0 con 0x7fc5b40aa760 2021-02-10T11:41:41.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 0x7fc59801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.039 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fc5a8032380 con 0x7fc5b40aa760 2021-02-10T11:41:41.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.040 7fc5b259c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 0x7fc59801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.041 7fc5b259c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 0x7fc59801abc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:41.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.069 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b41b4ee0 con 0x7fc5b40aa760 2021-02-10T11:41:41.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.071 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc5a802be60 con 0x7fc5b40aa760 2021-02-10T11:41:41.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.210 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fc5b4077a70 con 0x7fc5b40aa760 2021-02-10T11:41:41.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.550 7fc5a37fe700 1 -- 172.21.2.117:0/1592709660 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v51) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc5a802be60 con 0x7fc5b40aa760 2021-02-10T11:41:41.563 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:41:41.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 msgr2=0x7fc59801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 0x7fc59801abc0 crc :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 msgr2=0x7fc5b41b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7fc5a80096a0 tx=0x7fc5a800b9b0).stop 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 shutdown_connections 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5b4093bd0 0x7fc5b41a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5b4095360 0x7fc5b40935b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5980186c0 0x7fc59801abc0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 --2- 172.21.2.117:0/1592709660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5b40aa760 0x7fc5b41b8b00 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:41.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 >> 172.21.2.117:0/1592709660 conn(0x7fc5b41a95f0 msgr2=0x7fc5b4094350 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:41.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 shutdown_connections 2021-02-10T11:41:41.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.579 7fc5b919c700 1 -- 172.21.2.117:0/1592709660 wait complete. 2021-02-10T11:41:41.573 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:41:41.573 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.996 7f9ff46e2700 1 Processor -- start 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.996 7f9ff46e2700 1 -- start start 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9fec079000 0x7f9fec0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fec1b00e0 0x7f9fec1b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec077a70 con 0x7f9fec07da20 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec0e5c10 con 0x7f9fec1b00e0 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec0e53b0 con 0x7f9fec079000 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fec1b00e0 0x7f9fec1b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fec1b00e0 0x7f9fec1b04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48932/0 (socket says 172.21.2.117:48932) 2021-02-10T11:41:41.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 -- 172.21.2.117:0/4118071185 learned_addr learned my addr 172.21.2.117:0/4118071185 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff1cfd700 1 --2- 172.21.2.117:0/4118071185 >> v2:172.21.2.125:3300/0 conn(0x7f9fec079000 0x7f9fec0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 -- 172.21.2.117:0/4118071185 >> v2:172.21.2.125:3300/0 conn(0x7f9fec079000 msgr2=0x7f9fec0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 --2- 172.21.2.117:0/4118071185 >> v2:172.21.2.125:3300/0 conn(0x7f9fec079000 0x7f9fec0e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 -- 172.21.2.117:0/4118071185 >> v1:172.21.2.125:6789/0 conn(0x7f9fec07da20 legacy=0x7f9fec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 -- 172.21.2.117:0/4118071185 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec079400 con 0x7f9fec1b00e0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff2cff700 1 --2- 172.21.2.117:0/4118071185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1b00e0 0x7f9fec1b04e0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f9fe4006f00 tx=0x7f9fe4009f10).ready entity=mon.0 client_cookie=7e8647f549a5377e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff14fc700 1 -- 172.21.2.117:0/4118071185 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe400b3b0 con 0x7f9fec1b00e0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff14fc700 1 -- 172.21.2.117:0/4118071185 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe400b730 con 0x7f9fec1b00e0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff14fc700 1 -- 172.21.2.117:0/4118071185 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe400bc10 con 0x7f9fec1b00e0 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 -- 172.21.2.117:0/4118071185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1b00e0 msgr2=0x7f9fec1b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.997 7f9ff46e2700 1 --2- 172.21.2.117:0/4118071185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1b00e0 0x7f9fec1b04e0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f9fe4006f00 tx=0x7f9fe4009f10).stop 2021-02-10T11:41:42.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 -- 172.21.2.117:0/4118071185 shutdown_connections 2021-02-10T11:41:42.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 --2- 172.21.2.117:0/4118071185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1b00e0 0x7f9fec1b04e0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 --2- 172.21.2.117:0/4118071185 >> v2:172.21.2.125:3300/0 conn(0x7f9fec079000 0x7f9fec0e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 -- 172.21.2.117:0/4118071185 >> 172.21.2.117:0/4118071185 conn(0x7f9fec0ce440 msgr2=0x7f9fec0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 -- 172.21.2.117:0/4118071185 shutdown_connections 2021-02-10T11:41:42.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:41.998 7f9ff46e2700 1 -- 172.21.2.117:0/4118071185 wait complete. 2021-02-10T11:41:42.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.013 7f9ff46e2700 1 Processor -- start 2021-02-10T11:41:42.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.013 7f9ff46e2700 1 -- start start 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.013 7f9ff46e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fec1a26a0 0x7f9fec1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fec09d3c0 0x7f9fec09d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec19a6e0 con 0x7f9fec1a5590 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec1a2150 con 0x7f9fec09d3c0 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec1a50f0 con 0x7f9fec1a26a0 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48938/0 (socket says 172.21.2.117:48938) 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 -- 172.21.2.117:0/1114897729 learned_addr learned my addr 172.21.2.117:0/1114897729 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 -- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fec1a26a0 msgr2=0x7f9fec1a19c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fec1a26a0 0x7f9fec1a19c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 -- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fec09d3c0 msgr2=0x7f9fec09d7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fec09d3c0 0x7f9fec09d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 -- 172.21.2.117:0/1114897729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec1a5990 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff1cfd700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f9fe8006f00 tx=0x7f9fe800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe800bd30 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe8002030 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec1a5990 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.014 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec09e0d0 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.015 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe8002550 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.015 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fe8000c90 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.015 7f9fdeffd700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 0x7f9fd401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.015 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f9fe8032600 con 0x7f9fec1a5590 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.016 7f9ff24fe700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 0x7f9fd401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.016 7f9ff24fe700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 0x7f9fd401ac10 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.047 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec077a70 con 0x7f9fec1a5590 2021-02-10T11:41:42.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.049 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fe8000940 con 0x7f9fec1a5590 2021-02-10T11:41:42.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.179 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9fec1a2150 con 0x7f9fec1a5590 2021-02-10T11:41:42.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.180 7f9fdeffd700 1 -- 172.21.2.117:0/1114897729 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f9fe8000c90 con 0x7f9fec1a5590 2021-02-10T11:41:42.166 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.167 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.168 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:41:42.169 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:41:42.170 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.171 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:41:42.172 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.173 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.174 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:41:42.175 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:41:42.176 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.177 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.178 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:41:42.179 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:41:42.180 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:41:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 msgr2=0x7f9fd401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 0x7f9fd401ac10 crc :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 msgr2=0x7f9fec19a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f9fe8006f00 tx=0x7f9fe800b940).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 shutdown_connections 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd4018730 0x7f9fd401ac10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fec1a5590 0x7f9fec19a040 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fec09d3c0 0x7f9fec09d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 --2- 172.21.2.117:0/1114897729 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fec1a26a0 0x7f9fec1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 >> 172.21.2.117:0/1114897729 conn(0x7f9fec094bf0 msgr2=0x7f9fec19aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 shutdown_connections 2021-02-10T11:41:42.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.182 7f9ff46e2700 1 -- 172.21.2.117:0/1114897729 wait complete. 2021-02-10T11:41:42.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.606 7f43a41e2700 1 Processor -- start 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a41e2700 1 -- start start 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a41e2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f439c0d1520 0x7f439c0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a41e2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c077a70 con 0x7f439c079000 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a41e2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c079400 con 0x7f439c07da20 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a41e2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439c07de20 con 0x7f439c0d1520 2021-02-10T11:41:42.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a27ff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f439c0d1520 0x7f439c0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a17fd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f439c079000 0x7f439c0e4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49192/0 (socket says 172.21.2.117:49192) 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.607 7f43a17fd700 1 -- 172.21.2.117:0/810884618 learned_addr learned my addr 172.21.2.117:0/810884618 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a27ff700 1 -- 172.21.2.117:0/810884618 >> v1:172.21.2.125:6789/0 conn(0x7f439c07da20 legacy=0x7f439c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a27ff700 1 -- 172.21.2.117:0/810884618 >> v1:172.21.2.117:6789/0 conn(0x7f439c079000 legacy=0x7f439c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a27ff700 1 -- 172.21.2.117:0/810884618 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c1b00e0 con 0x7f439c0d1520 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a27ff700 1 --2- 172.21.2.117:0/810884618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0d1520 0x7f439c0d1920 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f4394006e30 tx=0x7f4394009e60).ready entity=mon.0 client_cookie=ebed6cffb16c047e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a0ffc700 1 -- 172.21.2.117:0/810884618 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4394005820 con 0x7f439c0d1520 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a0ffc700 1 -- 172.21.2.117:0/810884618 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394005a60 con 0x7f439c0d1520 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a0ffc700 1 -- 172.21.2.117:0/810884618 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4394006030 con 0x7f439c0d1520 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a41e2700 1 -- 172.21.2.117:0/810884618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0d1520 msgr2=0x7f439c0d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.608 7f43a41e2700 1 --2- 172.21.2.117:0/810884618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0d1520 0x7f439c0d1920 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f4394006e30 tx=0x7f4394009e60).stop 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.609 7f43a41e2700 1 -- 172.21.2.117:0/810884618 shutdown_connections 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.609 7f43a41e2700 1 --2- 172.21.2.117:0/810884618 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0d1520 0x7f439c0d1920 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.609 7f43a41e2700 1 -- 172.21.2.117:0/810884618 >> 172.21.2.117:0/810884618 conn(0x7f439c0ce440 msgr2=0x7f439c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.609 7f43a41e2700 1 -- 172.21.2.117:0/810884618 shutdown_connections 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.609 7f43a41e2700 1 -- 172.21.2.117:0/810884618 wait complete. 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 Processor -- start 2021-02-10T11:41:42.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 -- start start 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 0x7f439c1c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c0c0610 0x7f439c0a9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0a93d0 con 0x7f439c0a9e20 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0a95f0 con 0x7f439c10c430 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.624 7f43a41e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c1c6680 con 0x7f439c0c0610 2021-02-10T11:41:42.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48950/0 (socket says 172.21.2.117:48950) 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 -- 172.21.2.117:0/134242405 learned_addr learned my addr 172.21.2.117:0/134242405 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a17fd700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c0c0610 0x7f439c0a9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a1ffe700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 0x7f439c1c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 -- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c0c0610 msgr2=0x7f439c0a9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c0c0610 0x7f439c0a9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 -- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 msgr2=0x7f439c1c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 0x7f439c1c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 -- 172.21.2.117:0/134242405 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c0aa220 con 0x7f439c0a9e20 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a1ffe700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 0x7f439c1c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:42.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a27ff700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f439400b910 tx=0x7f439400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4394000fa0 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439400bdf0 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a41e2700 1 -- 172.21.2.117:0/134242405 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c0aa220 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4394005a50 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.625 7f43a41e2700 1 -- 172.21.2.117:0/134242405 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c0aa220 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.626 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4394005820 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.626 7f438e7fc700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 0x7f438401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.626 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4394032740 con 0x7f439c0a9e20 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.627 7f43a1ffe700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 0x7f438401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:42.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.627 7f43a1ffe700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 0x7f438401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:42.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.657 7f43a41e2700 1 -- 172.21.2.117:0/134242405 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c0aa4c0 con 0x7f439c0a9e20 2021-02-10T11:41:42.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.659 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f439402cfa0 con 0x7f439c0a9e20 2021-02-10T11:41:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.783 7f43a41e2700 1 -- 172.21.2.117:0/134242405 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f439c0aa4c0 con 0x7f439c0a9e20 2021-02-10T11:41:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.783 7f438e7fc700 1 -- 172.21.2.117:0/134242405 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+11377 (secure 0 0 0) 0x7f439402cfa0 con 0x7f439c0a9e20 2021-02-10T11:41:42.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:42.771 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":51,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:41.545050","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:42.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 -- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 msgr2=0x7f438401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 0x7f438401abc0 crc :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 -- 172.21.2.117:0/134242405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 msgr2=0x7f439c0b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:42.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f439400b910 tx=0x7f439400bcb0).stop 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 -- 172.21.2.117:0/134242405 shutdown_connections 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43840186c0 0x7f438401abc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c0a9e20 0x7f439c0b13c0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c0c0610 0x7f439c0a9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 --2- 172.21.2.117:0/134242405 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c10c430 0x7f439c1c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.785 7f43a41e2700 1 -- 172.21.2.117:0/134242405 >> 172.21.2.117:0/134242405 conn(0x7f439c09f900 msgr2=0x7f439c10cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.786 7f43a41e2700 1 -- 172.21.2.117:0/134242405 shutdown_connections 2021-02-10T11:41:42.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:42.786 7f43a41e2700 1 -- 172.21.2.117:0/134242405 wait complete. 2021-02-10T11:41:42.781 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:41:42.781 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:43.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 Processor -- start 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 -- start start 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c407da20 0x7f48c40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48c40e51a0 0x7f48c40d0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c4077a70 con 0x7f48c40e4da0 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c40e5c20 con 0x7f48c40e51a0 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c9fb2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c40777d0 con 0x7f48c407da20 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48c40e51a0 0x7f48c40d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48c40e51a0 0x7f48c40d0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48954/0 (socket says 172.21.2.117:48954) 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.208 7f48c3fff700 1 -- 172.21.2.117:0/1690587706 learned_addr learned my addr 172.21.2.117:0/1690587706 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c3fff700 1 -- 172.21.2.117:0/1690587706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c407da20 msgr2=0x7f48c40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:43.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c3fff700 1 --2- 172.21.2.117:0/1690587706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c407da20 0x7f48c40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c3fff700 1 -- 172.21.2.117:0/1690587706 >> v1:172.21.2.125:6789/0 conn(0x7f48c40e4da0 legacy=0x7f48c41b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c3fff700 1 -- 172.21.2.117:0/1690587706 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c41b08e0 con 0x7f48c40e51a0 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c3fff700 1 --2- 172.21.2.117:0/1690587706 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c40e51a0 0x7f48c40d0520 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f48bc006b80 tx=0x7f48bc009b90).ready entity=mon.0 client_cookie=548fa937751fade1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c27fc700 1 -- 172.21.2.117:0/1690587706 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48bc0022e0 con 0x7f48c40e51a0 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c27fc700 1 -- 172.21.2.117:0/1690587706 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc0009b0 con 0x7f48c40e51a0 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c27fc700 1 -- 172.21.2.117:0/1690587706 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48bc0028a0 con 0x7f48c40e51a0 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c9fb2700 1 -- 172.21.2.117:0/1690587706 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c40e51a0 msgr2=0x7f48c40d0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.209 7f48c9fb2700 1 --2- 172.21.2.117:0/1690587706 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c40e51a0 0x7f48c40d0520 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0x7f48bc006b80 tx=0x7f48bc009b90).stop 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 -- 172.21.2.117:0/1690587706 shutdown_connections 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 --2- 172.21.2.117:0/1690587706 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c40e51a0 0x7f48c40d0520 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 --2- 172.21.2.117:0/1690587706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c407da20 0x7f48c40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 -- 172.21.2.117:0/1690587706 >> 172.21.2.117:0/1690587706 conn(0x7f48c40ce450 msgr2=0x7f48c40e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:43.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 -- 172.21.2.117:0/1690587706 shutdown_connections 2021-02-10T11:41:43.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.210 7f48c9fb2700 1 -- 172.21.2.117:0/1690587706 wait complete. 2021-02-10T11:41:43.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 Processor -- start 2021-02-10T11:41:43.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 -- start start 2021-02-10T11:41:43.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c408a180 0x7f48c408a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48c41a7240 0x7f48c41a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c4108020 con 0x7f48c4107c20 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c41a7c60 con 0x7f48c41a7240 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.225 7f48c9fb2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c41a7e80 con 0x7f48c408a180 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48958/0 (socket says 172.21.2.117:48958) 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 -- 172.21.2.117:0/3983280 learned_addr learned my addr 172.21.2.117:0/3983280 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 -- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c408a180 msgr2=0x7f48c408a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c408a180 0x7f48c408a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 -- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48c41a7240 msgr2=0x7f48c41a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48c41a7240 0x7f48c41a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 -- 172.21.2.117:0/3983280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c408b250 con 0x7f48c4107c20 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.226 7f48c37fe700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f48b4006f00 tx=0x7f48b400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b400bd30 con 0x7f48c4107c20 2021-02-10T11:41:43.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b4000c90 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48b40022a0 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c408b250 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c408b7e0 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48b4005310 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 0x7f489c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.227 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f48b4032340 con 0x7f48c4107c20 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.228 7f48c2ffd700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 0x7f489c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.229 7f48c2ffd700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 0x7f489c01abc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.258 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c4077a70 con 0x7f48c4107c20 2021-02-10T11:41:43.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.261 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f48b4025f50 con 0x7f48c4107c20 2021-02-10T11:41:43.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.387 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f48c4077a70 con 0x7f48c4107c20 2021-02-10T11:41:43.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.387 7f48bbfff700 1 -- 172.21.2.117:0/3983280 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v51) v1 ==== 92+0+10 (secure 0 0 0) 0x7f48b4025f50 con 0x7f48c4107c20 2021-02-10T11:41:43.375 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.389 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 msgr2=0x7f489c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.389 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 0x7f489c01abc0 crc :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.389 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 msgr2=0x7f48c4096fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.389 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7f48b4006f00 tx=0x7f48b400b940).stop 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 shutdown_connections 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48c41a7240 0x7f48c41a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48c408a180 0x7f48c408a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:43.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f489c0186c0 0x7f489c01abc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 --2- 172.21.2.117:0/3983280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48c4107c20 0x7f48c4096fc0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 >> 172.21.2.117:0/3983280 conn(0x7f48c40a1880 msgr2=0x7f48c40b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:43.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 shutdown_connections 2021-02-10T11:41:43.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.390 7f48c9fb2700 1 -- 172.21.2.117:0/3983280 wait complete. 2021-02-10T11:41:43.387 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:41:43.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.809 7f1754ed7700 1 Processor -- start 2021-02-10T11:41:43.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.809 7f1754ed7700 1 -- start start 2021-02-10T11:41:43.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f1754ed7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17500d0ac0 0x7f17501b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f1754ed7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 0x7f17501b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f1754ed7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1750109ab0 con 0x7f17501085f0 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f1754ed7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500d0ec0 con 0x7f17500d0ac0 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f1754ed7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500d10b0 con 0x7f17501b3470 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174e312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 0x7f17501b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174eb13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17500d0ac0 0x7f17501b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174f314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17501085f0 0x7f17501089f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49220/0 (socket says 172.21.2.117:49220) 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174f314700 1 -- 172.21.2.117:0/4265947177 learned_addr learned my addr 172.21.2.117:0/4265947177 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:43.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174e312700 1 -- 172.21.2.117:0/4265947177 >> v2:172.21.2.117:3300/0 conn(0x7f17500d0ac0 msgr2=0x7f17501b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174e312700 1 --2- 172.21.2.117:0/4265947177 >> v2:172.21.2.117:3300/0 conn(0x7f17500d0ac0 0x7f17501b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174e312700 1 -- 172.21.2.117:0/4265947177 >> v1:172.21.2.117:6789/0 conn(0x7f17501085f0 legacy=0x7f17501089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174db11700 1 -- 172.21.2.117:0/4265947177 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032569131 0 0) 0x7f1748001500 con 0x7f17501085f0 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.810 7f174e312700 1 -- 172.21.2.117:0/4265947177 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500e51a0 con 0x7f17501b3470 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f174e312700 1 --2- 172.21.2.117:0/4265947177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 0x7f17501b3870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f1740006650 tx=0x7f1740009a80).ready entity=mon.2 client_cookie=377f1d9316e181b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f174db11700 1 -- 172.21.2.117:0/4265947177 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174000c0c0 con 0x7f17501b3470 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f174db11700 1 -- 172.21.2.117:0/4265947177 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f174000c0c0 con 0x7f17501b3470 2021-02-10T11:41:43.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f174db11700 1 -- 172.21.2.117:0/4265947177 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f174000c340 con 0x7f17501b3470 2021-02-10T11:41:43.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f1754ed7700 1 -- 172.21.2.117:0/4265947177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 msgr2=0x7f17501b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.811 7f1754ed7700 1 --2- 172.21.2.117:0/4265947177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 0x7f17501b3870 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f1740006650 tx=0x7f1740009a80).stop 2021-02-10T11:41:43.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 -- 172.21.2.117:0/4265947177 shutdown_connections 2021-02-10T11:41:43.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 --2- 172.21.2.117:0/4265947177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17501b3470 0x7f17501b3870 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 --2- 172.21.2.117:0/4265947177 >> v2:172.21.2.117:3300/0 conn(0x7f17500d0ac0 0x7f17501b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 -- 172.21.2.117:0/4265947177 >> 172.21.2.117:0/4265947177 conn(0x7f17500ce410 msgr2=0x7f17500d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:43.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 -- 172.21.2.117:0/4265947177 shutdown_connections 2021-02-10T11:41:43.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.812 7f1754ed7700 1 -- 172.21.2.117:0/4265947177 wait complete. 2021-02-10T11:41:43.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 Processor -- start 2021-02-10T11:41:43.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 -- start start 2021-02-10T11:41:43.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 0x7f17500a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 0x7f175019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 0x7f17500aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17501a0250 con 0x7f17500f20e0 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500f25e0 con 0x7f17500f1ce0 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f1754ed7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500f2800 con 0x7f17500f1880 2021-02-10T11:41:43.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f174f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 0x7f17500aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f174eb13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 0x7f17500a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f174f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 0x7f17500aad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48974/0 (socket says 172.21.2.117:48974) 2021-02-10T11:41:43.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f174f314700 1 -- 172.21.2.117:0/261254598 learned_addr learned my addr 172.21.2.117:0/261254598 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:43.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.827 7f174e312700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 0x7f175019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 -- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 msgr2=0x7f175019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 0x7f175019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 -- 172.21.2.117:0/261254598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 msgr2=0x7f17500aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 0x7f17500aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 -- 172.21.2.117:0/261254598 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500ab320 con 0x7f17500f1880 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174e312700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 0x7f175019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:43.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f174eb13700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 0x7f17500a9f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1738009540 tx=0x7f173800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f173800bcd0 con 0x7f17500f1880 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1738006740 con 0x7f17500f1880 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f1754ed7700 1 -- 172.21.2.117:0/261254598 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17500ab320 con 0x7f17500f1880 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f1754ed7700 1 -- 172.21.2.117:0/261254598 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17500ab320 con 0x7f17500f1880 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.828 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1738002030 con 0x7f17500f1880 2021-02-10T11:41:43.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.829 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17380052e0 con 0x7f17500f1880 2021-02-10T11:41:43.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.829 7f17477fe700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 0x7f173401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:43.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.829 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f17380324f0 con 0x7f17500f1880 2021-02-10T11:41:43.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.829 7f174e312700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 0x7f173401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:43.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.829 7f174e312700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 0x7f173401ac10 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:43.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.859 7f1754ed7700 1 -- 172.21.2.117:0/261254598 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1750109ab0 con 0x7f17500f1880 2021-02-10T11:41:43.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.862 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1738006ae0 con 0x7f17500f1880 2021-02-10T11:41:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.985 7f1754ed7700 1 -- 172.21.2.117:0/261254598 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1750109ab0 con 0x7f17500f1880 2021-02-10T11:41:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.986 7f17477fe700 1 -- 172.21.2.117:0/261254598 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) v1 ==== 84+0+64 (secure 0 0 0) 0x7f17380063c0 con 0x7f17500f1880 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:41:43.974 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:41:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 msgr2=0x7f173401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 0x7f173401ac10 crc :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 msgr2=0x7f17500a9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 0x7f17500a9f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f1738009540 tx=0x7f173800b8e0).stop 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 shutdown_connections 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1734018730 0x7f173401ac10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17500f20e0 0x7f17500aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17500f1ce0 0x7f175019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 --2- 172.21.2.117:0/261254598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17500f1880 0x7f17500a9f60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 >> 172.21.2.117:0/261254598 conn(0x7f17500bb7a0 msgr2=0x7f17500c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 shutdown_connections 2021-02-10T11:41:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:43.988 7f1754ed7700 1 -- 172.21.2.117:0/261254598 wait complete. 2021-02-10T11:41:43.984 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:41:43.984 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:41:44.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 Processor -- start 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 -- start start 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcfd007da20 0x7fcfd00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd00e51a0 0x7fcfd0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd0077a70 con 0x7fcfd00e4da0 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd00777d0 con 0x7fcfd007da20 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd876e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01b08e0 con 0x7fcfd00e51a0 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd658a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcfd007da20 0x7fcfd00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:44.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd658a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcfd007da20 0x7fcfd00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48980/0 (socket says 172.21.2.117:48980) 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.412 7fcfd658a700 1 -- 172.21.2.117:0/3956167604 learned_addr learned my addr 172.21.2.117:0/3956167604 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd658a700 1 -- 172.21.2.117:0/3956167604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd00e51a0 msgr2=0x7fcfd0079000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd658a700 1 --2- 172.21.2.117:0/3956167604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd00e51a0 0x7fcfd0079000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd658a700 1 -- 172.21.2.117:0/3956167604 >> v1:172.21.2.125:6789/0 conn(0x7fcfd00e4da0 legacy=0x7fcfd01b00f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd658a700 1 -- 172.21.2.117:0/3956167604 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd007de20 con 0x7fcfd007da20 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd658a700 1 --2- 172.21.2.117:0/3956167604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd007da20 0x7fcfd00771b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fcfcc006700 tx=0x7fcfcc009b80).ready entity=mon.0 client_cookie=6364f754e5c16a1f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd5588700 1 -- 172.21.2.117:0/3956167604 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfcc00c0c0 con 0x7fcfd007da20 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd5588700 1 -- 172.21.2.117:0/3956167604 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc00c340 con 0x7fcfd007da20 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd5588700 1 -- 172.21.2.117:0/3956167604 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfcc00c860 con 0x7fcfd007da20 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd876e700 1 -- 172.21.2.117:0/3956167604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd007da20 msgr2=0x7fcfd00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.413 7fcfd876e700 1 --2- 172.21.2.117:0/3956167604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd007da20 0x7fcfd00771b0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7fcfcc006700 tx=0x7fcfcc009b80).stop 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 -- 172.21.2.117:0/3956167604 shutdown_connections 2021-02-10T11:41:44.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 --2- 172.21.2.117:0/3956167604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd00e51a0 0x7fcfd0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 --2- 172.21.2.117:0/3956167604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd007da20 0x7fcfd00771b0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 -- 172.21.2.117:0/3956167604 >> 172.21.2.117:0/3956167604 conn(0x7fcfd00ce450 msgr2=0x7fcfd00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 -- 172.21.2.117:0/3956167604 shutdown_connections 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.414 7fcfd876e700 1 -- 172.21.2.117:0/3956167604 wait complete. 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.428 7fcfd876e700 1 Processor -- start 2021-02-10T11:41:44.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.428 7fcfd876e700 1 -- start start 2021-02-10T11:41:44.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfd008a180 0x7fcfd008a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd01a7240 0x7fcfd01a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd0108020 con 0x7fcfd0107c20 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01a7c60 con 0x7fcfd008a180 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd876e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd01a7e80 con 0x7fcfd01a7240 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:44.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48988/0 (socket says 172.21.2.117:48988) 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 -- 172.21.2.117:0/2218629888 learned_addr learned my addr 172.21.2.117:0/2218629888 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 -- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd01a7240 msgr2=0x7fcfd01a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd01a7240 0x7fcfd01a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 -- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfd008a180 msgr2=0x7fcfd008a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfd008a180 0x7fcfd008a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.429 7fcfd658a700 1 -- 172.21.2.117:0/2218629888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd008b250 con 0x7fcfd0107c20 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfd658a700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fcfcc00b580 tx=0x7fcfcc00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfcc00bd10 con 0x7fcfd0107c20 2021-02-10T11:41:44.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfcc002030 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfcc002480 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd008b250 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd008b7e0 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.430 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcfcc000c90 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.431 7fcfc2ffd700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 0x7fcfb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.431 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fcfcc032240 con 0x7fcfd0107c20 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.431 7fcfd5d89700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 0x7fcfb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:44.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.431 7fcfd5d89700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 0x7fcfb801abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:44.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.462 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd0077a70 con 0x7fcfd0107c20 2021-02-10T11:41:44.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.465 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcfcc0009b0 con 0x7fcfd0107c20 2021-02-10T11:41:44.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:44.592 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcfd0077a70 con 0x7fcfd0107c20 2021-02-10T11:41:45.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.019 7fcfc2ffd700 1 -- 172.21.2.117:0/2218629888 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v52) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcfcc000c90 con 0x7fcfd0107c20 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.060 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 msgr2=0x7fcfb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.060 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 0x7fcfb801abc0 crc :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.060 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 msgr2=0x7fcfd0096fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.060 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7fcfcc00b580 tx=0x7fcfcc00b920).stop 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 shutdown_connections 2021-02-10T11:41:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfd01a7240 0x7fcfd01a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfd008a180 0x7fcfd008a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfb80186c0 0x7fcfb801abc0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 --2- 172.21.2.117:0/2218629888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfd0107c20 0x7fcfd0096fc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 >> 172.21.2.117:0/2218629888 conn(0x7fcfd00a1880 msgr2=0x7fcfd00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 shutdown_connections 2021-02-10T11:41:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.061 7fcfd876e700 1 -- 172.21.2.117:0/2218629888 wait complete. 2021-02-10T11:41:45.056 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.483 7f26300b9700 1 Processor -- start 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.483 7f26300b9700 1 -- start start 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f26300b9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f26300b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f26280e4da0 0x7f26281b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f26300b9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2628077a70 con 0x7f26280e51a0 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f26300b9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26280777d0 con 0x7f26280e4da0 2021-02-10T11:41:45.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f26300b9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262807de20 con 0x7f262807da20 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f26280e4da0 0x7f26281b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262e6d6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f26280e51a0 0x7f2628077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49244/0 (socket says 172.21.2.117:49244) 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262e6d6700 1 -- 172.21.2.117:0/2627807597 learned_addr learned my addr 172.21.2.117:0/2627807597 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 -- 172.21.2.117:0/2627807597 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 msgr2=0x7f26280771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 --2- 172.21.2.117:0/2627807597 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 -- 172.21.2.117:0/2627807597 >> v1:172.21.2.117:6789/0 conn(0x7f26280e51a0 legacy=0x7f2628077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 -- 172.21.2.117:0/2627807597 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26281b0710 con 0x7f26280e4da0 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.484 7f262d6d4700 1 --2- 172.21.2.117:0/2627807597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26280e4da0 0x7f26281b00f0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f261c006650 tx=0x7f261c009a80).ready entity=mon.0 client_cookie=155c90e48ae7271d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f262ced3700 1 -- 172.21.2.117:0/2627807597 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c00c0c0 con 0x7f26280e4da0 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f262ced3700 1 -- 172.21.2.117:0/2627807597 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c00c340 con 0x7f26280e4da0 2021-02-10T11:41:45.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f262ced3700 1 -- 172.21.2.117:0/2627807597 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c00c860 con 0x7f26280e4da0 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 -- 172.21.2.117:0/2627807597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26280e4da0 msgr2=0x7f26281b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 --2- 172.21.2.117:0/2627807597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26280e4da0 0x7f26281b00f0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7f261c006650 tx=0x7f261c009a80).stop 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 -- 172.21.2.117:0/2627807597 shutdown_connections 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 --2- 172.21.2.117:0/2627807597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26280e4da0 0x7f26281b00f0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 --2- 172.21.2.117:0/2627807597 >> v2:172.21.2.125:3300/0 conn(0x7f262807da20 0x7f26280771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.485 7f26300b9700 1 -- 172.21.2.117:0/2627807597 >> 172.21.2.117:0/2627807597 conn(0x7f26280ce450 msgr2=0x7f26280e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.486 7f26300b9700 1 -- 172.21.2.117:0/2627807597 shutdown_connections 2021-02-10T11:41:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.486 7f26300b9700 1 -- 172.21.2.117:0/2627807597 wait complete. 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.500 7f26300b9700 1 Processor -- start 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.500 7f26300b9700 1 -- start start 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2628107c20 0x7f2628096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26281a7240 0x7f26281a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628108020 con 0x7f262808a180 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26281a7c60 con 0x7f26281a7240 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f26300b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26281a7e80 con 0x7f2628107c20 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:48998/0 (socket says 172.21.2.117:48998) 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 -- 172.21.2.117:0/3144150346 learned_addr learned my addr 172.21.2.117:0/3144150346 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262e6d6700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26281a7240 0x7f26281a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:45.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262ded5700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2628107c20 0x7f2628096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 -- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2628107c20 msgr2=0x7f2628096fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2628107c20 0x7f2628096fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 -- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26281a7240 msgr2=0x7f26281a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26281a7240 0x7f26281a7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.501 7f262d6d4700 1 -- 172.21.2.117:0/3144150346 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262808b250 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f262d6d4700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f261c00b5e0 tx=0x7f261c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c000940 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c00bd70 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f261c002270 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f26300b9700 1 -- 172.21.2.117:0/3144150346 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262808b250 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.502 7f26300b9700 1 -- 172.21.2.117:0/3144150346 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262808b250 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.503 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f261c000c90 con 0x7f262808a180 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.503 7f261a7fc700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 0x7f261001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.503 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f261c032780 con 0x7f262808a180 2021-02-10T11:41:45.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.503 7f262ded5700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 0x7f261001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:45.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.503 7f262ded5700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 0x7f261001ac10 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:45.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.533 7f26300b9700 1 -- 172.21.2.117:0/3144150346 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628077a70 con 0x7f262808a180 2021-02-10T11:41:45.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.536 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f261c0093e0 con 0x7f262808a180 2021-02-10T11:41:45.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:45.665 7f26300b9700 1 -- 172.21.2.117:0/3144150346 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f2628077a70 con 0x7f262808a180 2021-02-10T11:41:46.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.029 7f261a7fc700 1 -- 172.21.2.117:0/3144150346 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v54) v1 ==== 125+0+0 (secure 0 0 0) 0x7f261c000c90 con 0x7f262808a180 2021-02-10T11:41:46.018 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.032 7f26300b9700 1 -- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 msgr2=0x7f261001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.032 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 0x7f261001ac10 crc :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.032 7f26300b9700 1 -- 172.21.2.117:0/3144150346 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 msgr2=0x7f262808a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.032 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f261c00b5e0 tx=0x7f261c00b980).stop 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 -- 172.21.2.117:0/3144150346 shutdown_connections 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2610018730 0x7f261001ac10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f262808a180 0x7f262808a580 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26281a7240 0x7f26281a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 --2- 172.21.2.117:0/3144150346 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2628107c20 0x7f2628096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 -- 172.21.2.117:0/3144150346 >> 172.21.2.117:0/3144150346 conn(0x7f26280a1880 msgr2=0x7f26280b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 -- 172.21.2.117:0/3144150346 shutdown_connections 2021-02-10T11:41:46.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.033 7f26300b9700 1 -- 172.21.2.117:0/3144150346 wait complete. 2021-02-10T11:41:46.028 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.454 7fb1fa562700 1 Processor -- start 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.454 7fb1fa562700 1 -- start start 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f407da20 0x7fb1f40771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb1f40e4e00 0x7fb1f41b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1f40e5200 0x7fb1f4077f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40e5c60 con 0x7fb1f40e5200 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40777d0 con 0x7fb1f40e4e00 2021-02-10T11:41:46.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1fa562700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4077a10 con 0x7fb1f407da20 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1f40e5200 0x7fb1f4077f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb1f40e5200 0x7fb1f4077f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49008/0 (socket says 172.21.2.117:49008) 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 -- 172.21.2.117:0/2289019389 learned_addr learned my addr 172.21.2.117:0/2289019389 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 -- 172.21.2.117:0/2289019389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f407da20 msgr2=0x7fb1f40771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 --2- 172.21.2.117:0/2289019389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f407da20 0x7fb1f40771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 -- 172.21.2.117:0/2289019389 >> v2:172.21.2.125:3300/0 conn(0x7fb1f40e4e00 msgr2=0x7fb1f41b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 --2- 172.21.2.117:0/2289019389 >> v2:172.21.2.125:3300/0 conn(0x7fb1f40e4e00 0x7fb1f41b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 -- 172.21.2.117:0/2289019389 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f407de20 con 0x7fb1f40e5200 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.455 7fb1f8b7f700 1 --2- 172.21.2.117:0/2289019389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40e5200 0x7fb1f4077f90 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fb1ec006f00 tx=0x7fb1ec009f10).ready entity=mon.0 client_cookie=1b88c5a772ae36d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1f2ffd700 1 -- 172.21.2.117:0/2289019389 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec00b3b0 con 0x7fb1f40e5200 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1f2ffd700 1 -- 172.21.2.117:0/2289019389 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00b730 con 0x7fb1f40e5200 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1f2ffd700 1 -- 172.21.2.117:0/2289019389 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec00bc50 con 0x7fb1f40e5200 2021-02-10T11:41:46.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 -- 172.21.2.117:0/2289019389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40e5200 msgr2=0x7fb1f4077f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 --2- 172.21.2.117:0/2289019389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40e5200 0x7fb1f4077f90 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7fb1ec006f00 tx=0x7fb1ec009f10).stop 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 -- 172.21.2.117:0/2289019389 shutdown_connections 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 --2- 172.21.2.117:0/2289019389 >> v2:172.21.2.125:3300/0 conn(0x7fb1f40e4e00 0x7fb1f41b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 --2- 172.21.2.117:0/2289019389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40e5200 0x7fb1f4077f90 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 --2- 172.21.2.117:0/2289019389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f407da20 0x7fb1f40771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 -- 172.21.2.117:0/2289019389 >> 172.21.2.117:0/2289019389 conn(0x7fb1f40ce480 msgr2=0x7fb1f40e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 -- 172.21.2.117:0/2289019389 shutdown_connections 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.456 7fb1fa562700 1 -- 172.21.2.117:0/2289019389 wait complete. 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 Processor -- start 2021-02-10T11:41:46.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 -- start start 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 0x7fb1f40b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f40c0e50 0x7fb1f41a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4107050 con 0x7fb1f40c1250 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41a37d0 con 0x7fb1f4106c50 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1fa562700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41a39f0 con 0x7fb1f40c0e50 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1f8b7f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1f8b7f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49016/0 (socket says 172.21.2.117:49016) 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1f8b7f700 1 -- 172.21.2.117:0/815979106 learned_addr learned my addr 172.21.2.117:0/815979106 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:46.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.471 7fb1f37fe700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f40c0e50 0x7fb1f41a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f3fff700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 0x7fb1f40b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 -- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f40c0e50 msgr2=0x7fb1f41a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f40c0e50 0x7fb1f41a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 -- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 msgr2=0x7fb1f40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 0x7fb1f40b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 -- 172.21.2.117:0/815979106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40c1c40 con 0x7fb1f40c1250 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f3fff700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 0x7fb1f40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f8b7f700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fb1ec00b930 tx=0x7fb1ec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec009450 con 0x7fb1f40c1250 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec0068e0 con 0x7fb1f40c1250 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1ec00c380 con 0x7fb1f40c1250 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1fa562700 1 -- 172.21.2.117:0/815979106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f40c1c40 con 0x7fb1f40c1250 2021-02-10T11:41:46.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.472 7fb1fa562700 1 -- 172.21.2.117:0/815979106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f40c21d0 con 0x7fb1f40c1250 2021-02-10T11:41:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.473 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb1ec001270 con 0x7fb1f40c1250 2021-02-10T11:41:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.473 7fb1f0ff9700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 0x7fb1d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.473 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fb1ec0329f0 con 0x7fb1f40c1250 2021-02-10T11:41:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.473 7fb1f3fff700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 0x7fb1d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:46.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.474 7fb1f3fff700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 0x7fb1d401abc0 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.505 7fb1fa562700 1 -- 172.21.2.117:0/815979106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f41051c0 con 0x7fb1f40c1250 2021-02-10T11:41:46.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.507 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb1ec026080 con 0x7fb1f40c1250 2021-02-10T11:41:46.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.634 7fb1fa562700 1 -- 172.21.2.117:0/815979106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fb1f4077a10 con 0x7fb1f40c1250 2021-02-10T11:41:46.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.638 7fb1f0ff9700 1 -- 172.21.2.117:0/815979106 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 10 and data pool 11 v34) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb1ec026080 con 0x7fb1f40c1250 2021-02-10T11:41:46.630 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 10 and data pool 11 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 -- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 msgr2=0x7fb1d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 0x7fb1d401abc0 crc :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 -- 172.21.2.117:0/815979106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 msgr2=0x7fb1f41a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fb1ec00b930 tx=0x7fb1ec00bcd0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 -- 172.21.2.117:0/815979106 shutdown_connections 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1d40186c0 0x7fb1d401abc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1f40c1250 0x7fb1f41a31b0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1f40c0e50 0x7fb1f41a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 --2- 172.21.2.117:0/815979106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1f4106c50 0x7fb1f40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:46.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.643 7fb1fa562700 1 -- 172.21.2.117:0/815979106 >> 172.21.2.117:0/815979106 conn(0x7fb1f40c0520 msgr2=0x7fb1f40ad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.644 7fb1fa562700 1 -- 172.21.2.117:0/815979106 shutdown_connections 2021-02-10T11:41:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:46.644 7fb1fa562700 1 -- 172.21.2.117:0/815979106 wait complete. 2021-02-10T11:41:46.639 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:47.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.060 7f4c3aaea700 1 Processor -- start 2021-02-10T11:41:47.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.060 7f4c3aaea700 1 -- start start 2021-02-10T11:41:47.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c3aaea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c3407da20 0x7f4c340771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c3aaea700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4c340e5190 0x7f4c34079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c3aaea700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c34077a70 con 0x7f4c340e4d90 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c3aaea700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340e5c10 con 0x7f4c3407da20 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c3aaea700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340777d0 con 0x7f4c340e5190 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c33fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4c340e4d90 0x7f4c341b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49274/0 (socket says 172.21.2.117:49274) 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c33fff700 1 -- 172.21.2.117:0/2371320436 learned_addr learned my addr 172.21.2.117:0/2371320436 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c38906700 1 --2- 172.21.2.117:0/2371320436 >> v2:172.21.2.117:3300/0 conn(0x7f4c3407da20 0x7f4c340771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c38906700 1 -- 172.21.2.117:0/2371320436 >> v2:172.21.2.125:3300/0 conn(0x7f4c340e5190 msgr2=0x7f4c34079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c38906700 1 --2- 172.21.2.117:0/2371320436 >> v2:172.21.2.125:3300/0 conn(0x7f4c340e5190 0x7f4c34079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c38906700 1 -- 172.21.2.117:0/2371320436 >> v1:172.21.2.117:6789/0 conn(0x7f4c340e4d90 legacy=0x7f4c341b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.061 7f4c337fe700 1 -- 172.21.2.117:0/2371320436 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341644338 0 0) 0x7f4c240010b0 con 0x7f4c340e4d90 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c38906700 1 -- 172.21.2.117:0/2371320436 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c341b08d0 con 0x7f4c3407da20 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c38906700 1 --2- 172.21.2.117:0/2371320436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3407da20 0x7f4c340771b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f4c1c006650 tx=0x7f4c1c009a00).ready entity=mon.0 client_cookie=af6ee83276e3b73d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c337fe700 1 -- 172.21.2.117:0/2371320436 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c1c0009b0 con 0x7f4c3407da20 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c337fe700 1 -- 172.21.2.117:0/2371320436 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c00c0c0 con 0x7f4c3407da20 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c337fe700 1 -- 172.21.2.117:0/2371320436 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c1c00c5a0 con 0x7f4c3407da20 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c3aaea700 1 -- 172.21.2.117:0/2371320436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3407da20 msgr2=0x7f4c340771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c3aaea700 1 --2- 172.21.2.117:0/2371320436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3407da20 0x7f4c340771b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7f4c1c006650 tx=0x7f4c1c009a00).stop 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.062 7f4c3aaea700 1 -- 172.21.2.117:0/2371320436 shutdown_connections 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.063 7f4c3aaea700 1 --2- 172.21.2.117:0/2371320436 >> v2:172.21.2.125:3300/0 conn(0x7f4c340e5190 0x7f4c34079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.063 7f4c3aaea700 1 --2- 172.21.2.117:0/2371320436 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3407da20 0x7f4c340771b0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.063 7f4c3aaea700 1 -- 172.21.2.117:0/2371320436 >> 172.21.2.117:0/2371320436 conn(0x7f4c340ce440 msgr2=0x7f4c340e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.063 7f4c3aaea700 1 -- 172.21.2.117:0/2371320436 shutdown_connections 2021-02-10T11:41:47.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.063 7f4c3aaea700 1 -- 172.21.2.117:0/2371320436 wait complete. 2021-02-10T11:41:47.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.077 7f4c3aaea700 1 Processor -- start 2021-02-10T11:41:47.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.077 7f4c3aaea700 1 -- start start 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 0x7f4c341b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c341b8490 0x7f4c341b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3419e9c0 con 0x7f4c340aa770 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341b49c0 con 0x7f4c341b8490 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c3aaea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341b4be0 con 0x7f4c341b8090 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c33fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 0x7f4c341b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46708/0 (socket says 172.21.2.117:46708) 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 -- 172.21.2.117:0/1058649659 learned_addr learned my addr 172.21.2.117:0/1058649659 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c39107700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c341b8490 0x7f4c341b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 -- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c341b8490 msgr2=0x7f4c341b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c341b8490 0x7f4c341b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 -- 172.21.2.117:0/1058649659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 msgr2=0x7f4c341b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.078 7f4c38906700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 0x7f4c341b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c38906700 1 -- 172.21.2.117:0/1058649659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3419e470 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c33fff700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 0x7f4c341b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c38906700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f4c1c00b600 tx=0x7f4c1c00b9a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c1c0009b0 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c006390 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3419e470 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c1c0009b0 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3419e470 con 0x7f4c341b8090 2021-02-10T11:41:47.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.079 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c1c000c90 con 0x7f4c341b8090 2021-02-10T11:41:47.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.080 7f4c317fa700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 0x7f4c1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.080 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4c1c032550 con 0x7f4c341b8090 2021-02-10T11:41:47.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.080 7f4c33fff700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 0x7f4c1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.080 7f4c33fff700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 0x7f4c1801abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.110 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c34077a70 con 0x7f4c341b8090 2021-02-10T11:41:47.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.114 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c1c0009b0 con 0x7f4c341b8090 2021-02-10T11:41:47.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.241 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c34077a70 con 0x7f4c341b8090 2021-02-10T11:41:47.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.243 7f4c317fa700 1 -- 172.21.2.117:0/1058649659 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f4c1c000c90 con 0x7f4c341b8090 2021-02-10T11:41:47.232 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:47.232 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":55,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:46.636090","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:47.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 msgr2=0x7f4c1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 0x7f4c1801abc0 crc :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 msgr2=0x7f4c340aa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f4c1c00b600 tx=0x7f4c1c00b9a0).stop 2021-02-10T11:41:47.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 shutdown_connections 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c180186c0 0x7f4c1801abc0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c340aa770 0x7f4c341b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c341b8490 0x7f4c341b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 --2- 172.21.2.117:0/1058649659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c341b8090 0x7f4c340aa150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 >> 172.21.2.117:0/1058649659 conn(0x7f4c3419f550 msgr2=0x7f4c3418bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 shutdown_connections 2021-02-10T11:41:47.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.246 7f4c3aaea700 1 -- 172.21.2.117:0/1058649659 wait complete. 2021-02-10T11:41:47.242 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:47.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.666 7fbeb3f64700 1 Processor -- start 2021-02-10T11:41:47.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb3f64700 1 -- start start 2021-02-10T11:41:47.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb3f64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 0x7fbeac0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb3f64700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac0d1d10 con 0x7fbeac07da20 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb3f64700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeac077a70 con 0x7fbeac1b00e0 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb3f64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac0e5420 con 0x7fbeac079000 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb1d80700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbeac07da20 0x7fbeac0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49284/0 (socket says 172.21.2.117:49284) 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.667 7fbeb1d80700 1 -- 172.21.2.117:0/1527978683 learned_addr learned my addr 172.21.2.117:0/1527978683 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb157f700 1 --2- 172.21.2.117:0/1527978683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 0x7fbeac0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761054159 0 0) 0x7fbea0001060 con 0x7fbeac07da20 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe94002fe0 con 0x7fbeac07da20 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179120155 0 0) 0x7fbea40009b0 con 0x7fbeac1b00e0 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe94003210 con 0x7fbeac1b00e0 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb157f700 1 -- 172.21.2.117:0/1527978683 >> v1:172.21.2.125:6789/0 conn(0x7fbeac1b00e0 legacy=0x7fbeac0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb157f700 1 -- 172.21.2.117:0/1527978683 >> v1:172.21.2.117:6789/0 conn(0x7fbeac07da20 legacy=0x7fbeac0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb157f700 1 -- 172.21.2.117:0/1527978683 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac079400 con 0x7fbeac079000 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.668 7fbeb157f700 1 --2- 172.21.2.117:0/1527978683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 0x7fbeac0e4d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbe9c006650 tx=0x7fbe9c009a80).ready entity=mon.2 client_cookie=2546bf0dfa3890f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe9c00c0c0 con 0x7fbeac079000 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9c00c340 con 0x7fbeac079000 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb0d7e700 1 -- 172.21.2.117:0/1527978683 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe9c00c860 con 0x7fbeac079000 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb3f64700 1 -- 172.21.2.117:0/1527978683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 msgr2=0x7fbeac0e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb3f64700 1 --2- 172.21.2.117:0/1527978683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 0x7fbeac0e4d90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbe9c006650 tx=0x7fbe9c009a80).stop 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb3f64700 1 -- 172.21.2.117:0/1527978683 shutdown_connections 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb3f64700 1 --2- 172.21.2.117:0/1527978683 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac079000 0x7fbeac0e4d90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.669 7fbeb3f64700 1 -- 172.21.2.117:0/1527978683 >> 172.21.2.117:0/1527978683 conn(0x7fbeac0ce440 msgr2=0x7fbeac0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.670 7fbeb3f64700 1 -- 172.21.2.117:0/1527978683 shutdown_connections 2021-02-10T11:41:47.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.670 7fbeb3f64700 1 -- 172.21.2.117:0/1527978683 wait complete. 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.684 7fbeb3f64700 1 Processor -- start 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.684 7fbeb3f64700 1 -- start start 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbeac18d530 0x7fbeac18dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac18ef10 0x7fbeac18f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac18d930 con 0x7fbeac18e4f0 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac18e230 con 0x7fbeac18d530 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb3f64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeac192f30 con 0x7fbeac18ef10 2021-02-10T11:41:47.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49040/0 (socket says 172.21.2.117:49040) 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 -- 172.21.2.117:0/661109998 learned_addr learned my addr 172.21.2.117:0/661109998 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb2581700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac18ef10 0x7fbeac18f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb1d80700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbeac18d530 0x7fbeac18dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 -- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac18ef10 msgr2=0x7fbeac18f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac18ef10 0x7fbeac18f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 -- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbeac18d530 msgr2=0x7fbeac18dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbeac18d530 0x7fbeac18dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.685 7fbeb157f700 1 -- 172.21.2.117:0/661109998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac193150 con 0x7fbeac18e4f0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeb157f700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fbe9c009630 tx=0x7fbe9c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe9c0093a0 con 0x7fbeac18e4f0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9c00bd70 con 0x7fbeac18e4f0 2021-02-10T11:41:47.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe9c005560 con 0x7fbeac18e4f0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeac193150 con 0x7fbeac18e4f0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeac193150 con 0x7fbeac18e4f0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe9c00a000 con 0x7fbeac18e4f0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 0x7fbe8c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.686 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbe9c032770 con 0x7fbeac18e4f0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.687 7fbeb1d80700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 0x7fbe8c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:47.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.687 7fbeb1d80700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 0x7fbe8c01ac10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:47.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.716 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeac1b4f30 con 0x7fbeac18e4f0 2021-02-10T11:41:47.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.719 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe9c01f4e0 con 0x7fbeac18e4f0 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.844 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbeac1b4f30 con 0x7fbeac18e4f0 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.845 7fbeaa7fc700 1 -- 172.21.2.117:0/661109998 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7fbe9c00a000 con 0x7fbeac18e4f0 2021-02-10T11:41:47.832 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:47.833 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 msgr2=0x7fbe8c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 0x7fbe8c01ac10 crc :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 msgr2=0x7fbeac18e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fbe9c009630 tx=0x7fbe9c00b980).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 shutdown_connections 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe8c018730 0x7fbe8c01ac10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbeac18e4f0 0x7fbeac18e8f0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbeac18ef10 0x7fbeac18f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 --2- 172.21.2.117:0/661109998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbeac18d530 0x7fbeac18dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:47.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 >> 172.21.2.117:0/661109998 conn(0x7fbeac1ace70 msgr2=0x7fbeac1af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:47.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 shutdown_connections 2021-02-10T11:41:47.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:47.847 7fbeb3f64700 1 -- 172.21.2.117:0/661109998 wait complete. 2021-02-10T11:41:47.843 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:41:48.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 Processor -- start 2021-02-10T11:41:48.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 -- start start 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb21007da20 0x7fb2100771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 0x7fb2101b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb210077a70 con 0x7fb2100e51a0 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2100777d0 con 0x7fb21007da20 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb217615700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21007de20 con 0x7fb2100e4da0 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.270 7fb215431700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb21007da20 0x7fb2100771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb21007da20 0x7fb2100771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49046/0 (socket says 172.21.2.117:49046) 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb214c30700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 0x7fb2101b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 -- 172.21.2.117:0/728069995 learned_addr learned my addr 172.21.2.117:0/728069995 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 -- 172.21.2.117:0/728069995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 msgr2=0x7fb2101b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 0x7fb2101b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 -- 172.21.2.117:0/728069995 >> v1:172.21.2.117:6789/0 conn(0x7fb2100e51a0 legacy=0x7fb210077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 -- 172.21.2.117:0/728069995 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2101b0710 con 0x7fb21007da20 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb214c30700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 0x7fb2101b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb215431700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb21007da20 0x7fb2100771b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7fb20c006700 tx=0x7fb20c009b80).ready entity=mon.0 client_cookie=2e2dc63145a70fd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb207fff700 1 -- 172.21.2.117:0/728069995 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb20c00c0c0 con 0x7fb21007da20 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb207fff700 1 -- 172.21.2.117:0/728069995 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c00c440 con 0x7fb21007da20 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb207fff700 1 -- 172.21.2.117:0/728069995 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb20c00c8a0 con 0x7fb21007da20 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb217615700 1 -- 172.21.2.117:0/728069995 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb21007da20 msgr2=0x7fb2100771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.271 7fb217615700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb21007da20 0x7fb2100771b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7fb20c006700 tx=0x7fb20c009b80).stop 2021-02-10T11:41:48.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 -- 172.21.2.117:0/728069995 shutdown_connections 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2100e4da0 0x7fb2101b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 --2- 172.21.2.117:0/728069995 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb21007da20 0x7fb2100771b0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 -- 172.21.2.117:0/728069995 >> 172.21.2.117:0/728069995 conn(0x7fb2100ce450 msgr2=0x7fb2100e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 -- 172.21.2.117:0/728069995 shutdown_connections 2021-02-10T11:41:48.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.272 7fb217615700 1 -- 172.21.2.117:0/728069995 wait complete. 2021-02-10T11:41:48.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.286 7fb217615700 1 Processor -- start 2021-02-10T11:41:48.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 -- start start 2021-02-10T11:41:48.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 0x7fb210188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 0x7fb210090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101067f0 con 0x7fb210187e20 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb210107350 con 0x7fb210090420 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb217615700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2101042d0 con 0x7fb2101a0250 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb215431700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb215431700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46732/0 (socket says 172.21.2.117:46732) 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb215431700 1 -- 172.21.2.117:0/1377182532 learned_addr learned my addr 172.21.2.117:0/1377182532 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb215c32700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 0x7fb210090820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.287 7fb214c30700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 0x7fb210188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 -- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 msgr2=0x7fb210106cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 -- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 msgr2=0x7fb210090820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 0x7fb210090820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 -- 172.21.2.117:0/1377182532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210090e40 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb215431700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb215c32700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 0x7fb210090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb214c30700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 0x7fb210188220 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fb200009540 tx=0x7fb20000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb20000bcd0 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb200002030 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2000024d0 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb217615700 1 -- 172.21.2.117:0/1377182532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb210090e40 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.288 7fb217615700 1 -- 172.21.2.117:0/1377182532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb210090e40 con 0x7fb210187e20 2021-02-10T11:41:48.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.289 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb200000c90 con 0x7fb210187e20 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.289 7fb205ffb700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 0x7fb1f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.289 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb200032870 con 0x7fb210187e20 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.289 7fb215431700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 0x7fb1f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:48.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.290 7fb215431700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 0x7fb1f001abc0 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:48.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.320 7fb217615700 1 -- 172.21.2.117:0/1377182532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb210103280 con 0x7fb210187e20 2021-02-10T11:41:48.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.322 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb20001f460 con 0x7fb210187e20 2021-02-10T11:41:48.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.457 7fb217615700 1 -- 172.21.2.117:0/1377182532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fb210103280 con 0x7fb210187e20 2021-02-10T11:41:48.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.647 7fb205ffb700 1 -- 172.21.2.117:0/1377182532 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb200000c90 con 0x7fb210187e20 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 msgr2=0x7fb1f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 0x7fb1f001abc0 crc :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 msgr2=0x7fb210188220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 0x7fb210188220 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7fb200009540 tx=0x7fb20000b8e0).stop 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 shutdown_connections 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1f00186c0 0x7fb1f001abc0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb210187e20 0x7fb210188220 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb210090420 0x7fb210090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 --2- 172.21.2.117:0/1377182532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb2101a0250 0x7fb210106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:48.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 >> 172.21.2.117:0/1377182532 conn(0x7fb2100a1850 msgr2=0x7fb2100c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:48.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 shutdown_connections 2021-02-10T11:41:48.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:48.673 7fb217615700 1 -- 172.21.2.117:0/1377182532 wait complete. 2021-02-10T11:41:48.669 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:49.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.091 7f33a6529700 1 Processor -- start 2021-02-10T11:41:49.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.091 7f33a6529700 1 -- start start 2021-02-10T11:41:49.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33a007da20 0x7f33a00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f33a00e4d90 0x7f33a01b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a00e5190 0x7f33a0077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a0077a70 con 0x7f33a007da20 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a00777d0 con 0x7f33a00e4d90 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f33a6529700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a007de20 con 0x7f33a00e5190 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33a007da20 0x7f33a00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33a007da20 0x7f33a00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49060/0 (socket says 172.21.2.117:49060) 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 -- 172.21.2.117:0/1060346335 learned_addr learned my addr 172.21.2.117:0/1060346335 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 -- 172.21.2.117:0/1060346335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a00e5190 msgr2=0x7f33a0077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 --2- 172.21.2.117:0/1060346335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a00e5190 0x7f33a0077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 -- 172.21.2.117:0/1060346335 >> v2:172.21.2.125:3300/0 conn(0x7f33a00e4d90 msgr2=0x7f33a01b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 --2- 172.21.2.117:0/1060346335 >> v2:172.21.2.125:3300/0 conn(0x7f33a00e4d90 0x7f33a01b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 -- 172.21.2.117:0/1060346335 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a01b0700 con 0x7f33a007da20 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.092 7f339ffff700 1 --2- 172.21.2.117:0/1060346335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a007da20 0x7f33a00771b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f3390006700 tx=0x7f3390009b80).ready entity=mon.0 client_cookie=69cc3d50186b5de4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f339effd700 1 -- 172.21.2.117:0/1060346335 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3390006340 con 0x7f33a007da20 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f339effd700 1 -- 172.21.2.117:0/1060346335 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000c0c0 con 0x7f33a007da20 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f339effd700 1 -- 172.21.2.117:0/1060346335 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f339000c5e0 con 0x7f33a007da20 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 -- 172.21.2.117:0/1060346335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a007da20 msgr2=0x7f33a00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 --2- 172.21.2.117:0/1060346335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a007da20 0x7f33a00771b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f3390006700 tx=0x7f3390009b80).stop 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 -- 172.21.2.117:0/1060346335 shutdown_connections 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 --2- 172.21.2.117:0/1060346335 >> v2:172.21.2.125:3300/0 conn(0x7f33a00e4d90 0x7f33a01b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 --2- 172.21.2.117:0/1060346335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a00e5190 0x7f33a0077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 --2- 172.21.2.117:0/1060346335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a007da20 0x7f33a00771b0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 -- 172.21.2.117:0/1060346335 >> 172.21.2.117:0/1060346335 conn(0x7f33a00ce440 msgr2=0x7f33a00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 -- 172.21.2.117:0/1060346335 shutdown_connections 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.093 7f33a6529700 1 -- 172.21.2.117:0/1060346335 wait complete. 2021-02-10T11:41:49.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.108 7f33a6529700 1 Processor -- start 2021-02-10T11:41:49.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.108 7f33a6529700 1 -- start start 2021-02-10T11:41:49.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.108 7f33a6529700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a01a27c0 0x7f33a019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.108 7f33a6529700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33a01a5640 0x7f33a01a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a6529700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a6529700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a00ab7d0 con 0x7f33a019b930 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a6529700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01a5a40 con 0x7f33a01a5640 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a6529700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0199d40 con 0x7f33a01a27c0 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49070/0 (socket says 172.21.2.117:49070) 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 -- 172.21.2.117:0/1302577594 learned_addr learned my addr 172.21.2.117:0/1302577594 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:49.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 -- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a01a27c0 msgr2=0x7f33a019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a01a27c0 0x7f33a019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 -- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33a01a5640 msgr2=0x7f33a01a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33a01a5640 0x7f33a01a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 -- 172.21.2.117:0/1302577594 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a019c350 con 0x7f33a019b930 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a4b46700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7f3398009980 tx=0x7f339800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3398005870 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3398005b30 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3398006030 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.109 7f33a6529700 1 -- 172.21.2.117:0/1302577594 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a019c350 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.110 7f33a6529700 1 -- 172.21.2.117:0/1302577594 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a019c350 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.110 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3398005d10 con 0x7f33a019b930 2021-02-10T11:41:49.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.110 7f339cff9700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 0x7f338001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.110 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3398032b70 con 0x7f33a019b930 2021-02-10T11:41:49.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.111 7f339ffff700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 0x7f338001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.111 7f339ffff700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 0x7f338001abc0 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.141 7f33a6529700 1 -- 172.21.2.117:0/1302577594 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a0108c10 con 0x7f33a019b930 2021-02-10T11:41:49.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.144 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3398006f20 con 0x7f33a019b930 2021-02-10T11:41:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.288 7f33a6529700 1 -- 172.21.2.117:0/1302577594 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33a0077a70 con 0x7f33a019b930 2021-02-10T11:41:49.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.288 7f339cff9700 1 -- 172.21.2.117:0/1302577594 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f3398006f20 con 0x7f33a019b930 2021-02-10T11:41:49.276 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 36 2021-02-10T11:41:49.276 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:49.276 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:41:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 msgr2=0x7f338001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 0x7f338001abc0 crc :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 msgr2=0x7f33a019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7f3398009980 tx=0x7f339800bcd0).stop 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 shutdown_connections 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33800186c0 0x7f338001abc0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33a019b930 0x7f33a019bd30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33a01a5640 0x7f33a01a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 --2- 172.21.2.117:0/1302577594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33a01a27c0 0x7f33a019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 >> 172.21.2.117:0/1302577594 conn(0x7f33a0191110 msgr2=0x7f33a019aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 shutdown_connections 2021-02-10T11:41:49.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.291 7f33a6529700 1 -- 172.21.2.117:0/1302577594 wait complete. 2021-02-10T11:41:49.286 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:49.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 Processor -- start 2021-02-10T11:41:49.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 -- start start 2021-02-10T11:41:49.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b2007da20 0x7f5b200771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5b200e5190 0x7f5b200d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b20077a70 con 0x7f5b200e4d90 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200e5c10 con 0x7f5b200e5190 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b28797700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200777d0 con 0x7f5b2007da20 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.712 7f5b25db2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5b200e4d90 0x7f5b201b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49324/0 (socket says 172.21.2.117:49324) 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b25db2700 1 -- 172.21.2.117:0/1947230233 learned_addr learned my addr 172.21.2.117:0/1947230233 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656307002 0 0) 0x7f5b1c0010b0 con 0x7f5b200e4d90 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c002fe0 con 0x7f5b200e4d90 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b265b3700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b2007da20 0x7f5b200771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b26db4700 1 --2- 172.21.2.117:0/1947230233 >> v2:172.21.2.125:3300/0 conn(0x7f5b200e5190 0x7f5b200d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1220339372 0 0) 0x7f5b1c0010b0 con 0x7f5b200e4d90 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b2007da20 msgr2=0x7f5b200771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b2007da20 0x7f5b200771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 >> v2:172.21.2.125:3300/0 conn(0x7f5b200e5190 msgr2=0x7f5b200d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 --2- 172.21.2.117:0/1947230233 >> v2:172.21.2.125:3300/0 conn(0x7f5b200e5190 0x7f5b200d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b201b08d0 con 0x7f5b200e4d90 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5b1c0015b0 con 0x7f5b200e4d90 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 >> v1:172.21.2.117:6789/0 conn(0x7f5b200e4d90 legacy=0x7f5b201b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.713 7f5b255b1700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b0c003ac0 0x7f5b0c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b0c04a7a0 0x7f5b0c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b0c04d510 0x7f5b0c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c0502e0 con 0x7f5b0c003ac0 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c050500 con 0x7f5b0c04d510 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c050720 con 0x7f5b0c04a7a0 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b0c003ac0 0x7f5b0c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b26db4700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b0c04d510 0x7f5b0c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b25db2700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b0c04a7a0 0x7f5b0c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 -- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b0c04a7a0 msgr2=0x7f5b0c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b0c04a7a0 0x7f5b0c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 -- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b0c04d510 msgr2=0x7f5b0c04fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b0c04d510 0x7f5b0c04fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 -- 172.21.2.117:0/1947230233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c0509c0 con 0x7f5b0c003ac0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b265b3700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b0c003ac0 0x7f5b0c005ef0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f5b14006aa0 tx=0x7f5b1400cb50).ready entity=mon.0 client_cookie=41c872ca96b0181c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b140090d0 con 0x7f5b0c003ac0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14009410 con 0x7f5b0c003ac0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b255b1700 1 -- 172.21.2.117:0/1947230233 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b14009930 con 0x7f5b0c003ac0 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b28797700 1 -- 172.21.2.117:0/1947230233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b0c003ac0 msgr2=0x7f5b0c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b28797700 1 --2- 172.21.2.117:0/1947230233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b0c003ac0 0x7f5b0c005ef0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f5b14006aa0 tx=0x7f5b1400cb50).stop 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.714 7f5b26db4700 1 -- 172.21.2.117:0/1947230233 reap_dead start 2021-02-10T11:41:49.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.715 7f5b28797700 1 -- 172.21.2.117:0/1947230233 shutdown_connections 2021-02-10T11:41:49.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.715 7f5b28797700 1 -- 172.21.2.117:0/1947230233 >> 172.21.2.117:0/1947230233 conn(0x7f5b200ce440 msgr2=0x7f5b200e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.715 7f5b28797700 1 -- 172.21.2.117:0/1947230233 shutdown_connections 2021-02-10T11:41:49.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.715 7f5b28797700 1 -- 172.21.2.117:0/1947230233 wait complete. 2021-02-10T11:41:49.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 Processor -- start 2021-02-10T11:41:49.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 -- start start 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b20192ba0 0x7f5b20192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b201067a0 0x7f5b20106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20193730 con 0x7f5b2018cbd0 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2018d630 con 0x7f5b20192ba0 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b28797700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2018d850 con 0x7f5b201067a0 2021-02-10T11:41:49.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b26db4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b26db4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49084/0 (socket says 172.21.2.117:49084) 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.730 7f5b26db4700 1 -- 172.21.2.117:0/606054418 learned_addr learned my addr 172.21.2.117:0/606054418 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 -- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b201067a0 msgr2=0x7f5b20106ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b201067a0 0x7f5b20106ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 -- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b20192ba0 msgr2=0x7f5b20192fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b20192ba0 0x7f5b20192fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 -- 172.21.2.117:0/606054418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2018dc30 con 0x7f5b2018cbd0 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b26db4700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f5b180099e0 tx=0x7f5b1800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b18001400 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800be60 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b28797700 1 -- 172.21.2.117:0/606054418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2018dc30 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b28797700 1 -- 172.21.2.117:0/606054418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2018e1c0 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.731 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b18001680 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.732 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b18001b50 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.732 7f5b12ffd700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 0x7f5b0801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.732 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5b18032ad0 con 0x7f5b2018cbd0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.732 7f5b265b3700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 0x7f5b0801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:49.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.733 7f5b265b3700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 0x7f5b0801ac10 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:49.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.762 7f5b28797700 1 -- 172.21.2.117:0/606054418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b20077a70 con 0x7f5b2018cbd0 2021-02-10T11:41:49.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.765 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b18032ce0 con 0x7f5b2018cbd0 2021-02-10T11:41:49.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.890 7f5b28797700 1 -- 172.21.2.117:0/606054418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b2018d5f0 con 0x7f5b2018cbd0 2021-02-10T11:41:49.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.890 7f5b12ffd700 1 -- 172.21.2.117:0/606054418 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f5b18001b50 con 0x7f5b2018cbd0 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:49.878 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 -- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 msgr2=0x7f5b0801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 0x7f5b0801ac10 crc :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 -- 172.21.2.117:0/606054418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 msgr2=0x7f5b2018cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:49.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f5b180099e0 tx=0x7f5b1800bd20).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 -- 172.21.2.117:0/606054418 shutdown_connections 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b08018730 0x7f5b0801ac10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b2018cbd0 0x7f5b2018cfd0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b201067a0 0x7f5b20106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.892 7f5b28797700 1 --2- 172.21.2.117:0/606054418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b20192ba0 0x7f5b20192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:49.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.893 7f5b28797700 1 -- 172.21.2.117:0/606054418 >> 172.21.2.117:0/606054418 conn(0x7f5b2019f510 msgr2=0x7f5b201abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.893 7f5b28797700 1 -- 172.21.2.117:0/606054418 shutdown_connections 2021-02-10T11:41:49.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:49.893 7f5b28797700 1 -- 172.21.2.117:0/606054418 wait complete. 2021-02-10T11:41:49.889 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.324 7f781bf97700 1 Processor -- start 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.324 7f781bf97700 1 -- start start 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f781bf97700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78140d1530 0x7f78140d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f781bf97700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7814077a70 con 0x7f781407da20 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f781bf97700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7814079400 con 0x7f7814079000 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f781bf97700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781407de20 con 0x7f78140d1530 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f781a5b4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78140d1530 0x7f78140d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f7819db3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f781407da20 0x7f78140771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49342/0 (socket says 172.21.2.117:49342) 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.325 7f7819db3700 1 -- 172.21.2.117:0/1310556121 learned_addr learned my addr 172.21.2.117:0/1310556121 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781a5b4700 1 -- 172.21.2.117:0/1310556121 >> v1:172.21.2.125:6789/0 conn(0x7f7814079000 legacy=0x7f78140e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781a5b4700 1 -- 172.21.2.117:0/1310556121 >> v1:172.21.2.117:6789/0 conn(0x7f781407da20 legacy=0x7f78140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781a5b4700 1 -- 172.21.2.117:0/1310556121 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78141b00f0 con 0x7f78140d1530 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781a5b4700 1 --2- 172.21.2.117:0/1310556121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140d1530 0x7f78140d1930 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f780c006e30 tx=0x7f780c009e60).ready entity=mon.0 client_cookie=91b9b289a60c5944 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f7818db1700 1 -- 172.21.2.117:0/1310556121 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c006a50 con 0x7f78140d1530 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f7818db1700 1 -- 172.21.2.117:0/1310556121 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c00b300 con 0x7f78140d1530 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f7818db1700 1 -- 172.21.2.117:0/1310556121 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c00b760 con 0x7f78140d1530 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781bf97700 1 -- 172.21.2.117:0/1310556121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140d1530 msgr2=0x7f78140d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.326 7f781bf97700 1 --2- 172.21.2.117:0/1310556121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140d1530 0x7f78140d1930 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7f780c006e30 tx=0x7f780c009e60).stop 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.327 7f781bf97700 1 -- 172.21.2.117:0/1310556121 shutdown_connections 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.327 7f781bf97700 1 --2- 172.21.2.117:0/1310556121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140d1530 0x7f78140d1930 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.327 7f781bf97700 1 -- 172.21.2.117:0/1310556121 >> 172.21.2.117:0/1310556121 conn(0x7f78140ce450 msgr2=0x7f78140e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.327 7f781bf97700 1 -- 172.21.2.117:0/1310556121 shutdown_connections 2021-02-10T11:41:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.327 7f781bf97700 1 -- 172.21.2.117:0/1310556121 wait complete. 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 Processor -- start 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 -- start start 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140c2e30 0x7f78140c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7814099110 0x7f7814099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140c3850 con 0x7f7814090dd0 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78140c3a70 con 0x7f7814099110 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f781bf97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78140911d0 con 0x7f78140c2e30 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f7819db3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f7819db3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49098/0 (socket says 172.21.2.117:49098) 2021-02-10T11:41:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.342 7f7819db3700 1 -- 172.21.2.117:0/1288570185 learned_addr learned my addr 172.21.2.117:0/1288570185 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 -- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140c2e30 msgr2=0x7f78140c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140c2e30 0x7f78140c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 -- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7814099110 msgr2=0x7f7814099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7814099110 0x7f7814099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 -- 172.21.2.117:0/1288570185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78141c6640 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f7819db3700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7f7810001400 tx=0x7f781000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f781000acf0 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78100009b0 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f781bf97700 1 -- 172.21.2.117:0/1288570185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78141c6640 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7810004590 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.343 7f781bf97700 1 -- 172.21.2.117:0/1288570185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78141c6640 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.344 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78100043a0 con 0x7f7814090dd0 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.344 7f780a7fc700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 0x7f77fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.344 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78100327a0 con 0x7f7814090dd0 2021-02-10T11:41:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.344 7f78195b2700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 0x7f77fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.345 7f78195b2700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 0x7f77fc01ac10 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.375 7f781bf97700 1 -- 172.21.2.117:0/1288570185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7814103630 con 0x7f7814090dd0 2021-02-10T11:41:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.378 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f781000acf0 con 0x7f7814090dd0 2021-02-10T11:41:50.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.507 7f781bf97700 1 -- 172.21.2.117:0/1288570185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f78140ce450 con 0x7f7814090dd0 2021-02-10T11:41:50.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.511 7f780a7fc700 1 -- 172.21.2.117:0/1288570185 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f78100043a0 con 0x7f7814090dd0 2021-02-10T11:41:50.501 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:41:50.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 msgr2=0x7f77fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 0x7f77fc01ac10 crc :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 msgr2=0x7f7814098180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7f7810001400 tx=0x7f781000a900).stop 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 shutdown_connections 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7814099110 0x7f7814099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140c2e30 0x7f78140c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77fc018730 0x7f77fc01ac10 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 --2- 172.21.2.117:0/1288570185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7814090dd0 0x7f7814098180 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 >> 172.21.2.117:0/1288570185 conn(0x7f78140a01b0 msgr2=0x7f7814091650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 shutdown_connections 2021-02-10T11:41:50.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.516 7f781bf97700 1 -- 172.21.2.117:0/1288570185 wait complete. 2021-02-10T11:41:50.511 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 Processor -- start 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 -- start start 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 0x7fa3080771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa3080e5180 0x7fa3080d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308077a70 con 0x7fa3080e4d80 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080e5c00 con 0x7fa3080e5180 2021-02-10T11:41:50.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.947 7fa30cf9d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080777d0 con 0x7fa30807da20 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.948 7fa30659c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fa3080e4d80 0x7fa3081b00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42282/0 (socket says 172.21.2.117:42282) 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.948 7fa30659c700 1 -- 172.21.2.117:0/3053486643 learned_addr learned my addr 172.21.2.117:0/3053486643 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.948 7fa306d9d700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 0x7fa3080771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.948 7fa30759e700 1 --2- 172.21.2.117:0/3053486643 >> v2:172.21.2.125:3300/0 conn(0x7fa3080e5180 0x7fa3080d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30759e700 1 -- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 msgr2=0x7fa3080771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30759e700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 0x7fa3080771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa305d9b700 1 -- 172.21.2.117:0/3053486643 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341331006 0 0) 0x7fa2fc000c60 con 0x7fa3080e4d80 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30759e700 1 -- 172.21.2.117:0/3053486643 >> v1:172.21.2.125:6789/0 conn(0x7fa3080e4d80 legacy=0x7fa3081b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30759e700 1 -- 172.21.2.117:0/3053486643 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3081b08c0 con 0x7fa3080e5180 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa306d9d700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 0x7fa3080771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30759e700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080e5180 0x7fa3080d0500 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa300006e50 tx=0x7fa300009e70).ready entity=mon.1 client_cookie=57135e4d14dc092c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa305d9b700 1 -- 172.21.2.117:0/3053486643 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa300006aa0 con 0x7fa3080e5180 2021-02-10T11:41:50.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa305d9b700 1 -- 172.21.2.117:0/3053486643 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa300006790 con 0x7fa3080e5180 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa305d9b700 1 -- 172.21.2.117:0/3053486643 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa300006aa0 con 0x7fa3080e5180 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30cf9d700 1 -- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080e5180 msgr2=0x7fa3080d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.949 7fa30cf9d700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080e5180 0x7fa3080d0500 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa300006e50 tx=0x7fa300009e70).stop 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 -- 172.21.2.117:0/3053486643 shutdown_connections 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080e5180 0x7fa3080d0500 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 --2- 172.21.2.117:0/3053486643 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30807da20 0x7fa3080771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 -- 172.21.2.117:0/3053486643 >> 172.21.2.117:0/3053486643 conn(0x7fa3080ce430 msgr2=0x7fa3080e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 -- 172.21.2.117:0/3053486643 shutdown_connections 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.950 7fa30cf9d700 1 -- 172.21.2.117:0/3053486643 wait complete. 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.965 7fa30cf9d700 1 Processor -- start 2021-02-10T11:41:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 -- start start 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 0x7fa3080a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 0x7fa3080bb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080bbae0 0x7fa3080bbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30809e7a0 con 0x7fa3080a2c60 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30809e3c0 con 0x7fa3080bbae0 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30cf9d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3080a3060 con 0x7fa30809de80 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 0x7fa3080bb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa306d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 0x7fa3080a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa30759e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080bbae0 0x7fa3080bbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa306d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 0x7fa3080a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46792/0 (socket says 172.21.2.117:46792) 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.966 7fa306d9d700 1 -- 172.21.2.117:0/886928239 learned_addr learned my addr 172.21.2.117:0/886928239 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 -- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 msgr2=0x7fa3080a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 0x7fa3080a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 -- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080bbae0 msgr2=0x7fa3080bbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080bbae0 0x7fa3080bbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 -- 172.21.2.117:0/886928239 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3081880d0 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30659c700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 0x7fa3080bb4c0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fa2fc00a4c0 tx=0x7fa2fc00a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2fc000c60 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00ac50 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3081880d0 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2fc004510 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3081880d0 con 0x7fa3080a2c60 2021-02-10T11:41:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.967 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2fc000c60 con 0x7fa3080a2c60 2021-02-10T11:41:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.968 7fa2f77fe700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 0x7fa2ec01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.968 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2fc0326b0 con 0x7fa3080a2c60 2021-02-10T11:41:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.968 7fa306d9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 0x7fa2ec01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.968 7fa306d9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 0x7fa2ec01ac10 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:50.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.997 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30809e390 con 0x7fa3080a2c60 2021-02-10T11:41:50.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:50.999 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2fc007710 con 0x7fa3080a2c60 2021-02-10T11:41:51.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.130 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa30809e390 con 0x7fa3080a2c60 2021-02-10T11:41:51.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.133 7fa2f77fe700 1 -- 172.21.2.117:0/886928239 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa2fc01f420 con 0x7fa3080a2c60 2021-02-10T11:41:51.119 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 msgr2=0x7fa2ec01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 0x7fa2ec01ac10 crc :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 msgr2=0x7fa3080bb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 0x7fa3080bb4c0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fa2fc00a4c0 tx=0x7fa2fc00a860).stop 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 shutdown_connections 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.134 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2ec018730 0x7fa2ec01ac10 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3080a2c60 0x7fa3080bb4c0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3080bbae0 0x7fa3080bbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 --2- 172.21.2.117:0/886928239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa30809de80 0x7fa3080a2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 >> 172.21.2.117:0/886928239 conn(0x7fa30819f4b0 msgr2=0x7fa308094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 shutdown_connections 2021-02-10T11:41:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.135 7fa30cf9d700 1 -- 172.21.2.117:0/886928239 wait complete. 2021-02-10T11:41:51.129 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.563 7fd32152e700 1 Processor -- start 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.563 7fd32152e700 1 -- start start 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd32152e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd31c079000 0x7fd31c0e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd32152e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c1b00f0 0x7fd31c1b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd32152e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c077a70 con 0x7fd31c07da20 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd32152e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c0e53c0 con 0x7fd31c079000 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd32152e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c079400 con 0x7fd31c1b00f0 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd31aabe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd31c079000 0x7fd31c0e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd31bac0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c1b00f0 0x7fd31c1b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd31b2bf700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fd31c07da20 0x7fd31c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42294/0 (socket says 172.21.2.117:42294) 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.564 7fd31b2bf700 1 -- 172.21.2.117:0/795809793 learned_addr learned my addr 172.21.2.117:0/795809793 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31aabe700 1 -- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c1b00f0 msgr2=0x7fd31c1b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31aabe700 1 --2- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c1b00f0 0x7fd31c1b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31aabe700 1 -- 172.21.2.117:0/795809793 >> v1:172.21.2.125:6789/0 conn(0x7fd31c07da20 legacy=0x7fd31c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31aabe700 1 -- 172.21.2.117:0/795809793 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c07de20 con 0x7fd31c079000 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31aabe700 1 --2- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c079000 0x7fd31c0e4da0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fd30c006650 tx=0x7fd30c009a80).ready entity=mon.1 client_cookie=ddf46fbabdd87137 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31a2bd700 1 -- 172.21.2.117:0/795809793 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd30c00c0c0 con 0x7fd31c079000 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31a2bd700 1 -- 172.21.2.117:0/795809793 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00c340 con 0x7fd31c079000 2021-02-10T11:41:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd31a2bd700 1 -- 172.21.2.117:0/795809793 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd30c00c860 con 0x7fd31c079000 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd32152e700 1 -- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c079000 msgr2=0x7fd31c0e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.565 7fd32152e700 1 --2- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c079000 0x7fd31c0e4da0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fd30c006650 tx=0x7fd30c009a80).stop 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 -- 172.21.2.117:0/795809793 shutdown_connections 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 --2- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c1b00f0 0x7fd31c1b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 --2- 172.21.2.117:0/795809793 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c079000 0x7fd31c0e4da0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 -- 172.21.2.117:0/795809793 >> 172.21.2.117:0/795809793 conn(0x7fd31c0ce450 msgr2=0x7fd31c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 -- 172.21.2.117:0/795809793 shutdown_connections 2021-02-10T11:41:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.566 7fd32152e700 1 -- 172.21.2.117:0/795809793 wait complete. 2021-02-10T11:41:51.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 Processor -- start 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 -- start start 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 0x7fd31c19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c19c090 0x7fd31c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c19cca0 con 0x7fd31c1b7d30 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c19d840 con 0x7fd31c1b8880 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd32152e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c195510 con 0x7fd31c19c090 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd31aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd31aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49124/0 (socket says 172.21.2.117:49124) 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.581 7fd31aabe700 1 -- 172.21.2.117:0/1147866647 learned_addr learned my addr 172.21.2.117:0/1147866647 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:51.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31bac0700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c19c090 0x7fd31c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31b2bf700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 0x7fd31c19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 -- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c19c090 msgr2=0x7fd31c19c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c19c090 0x7fd31c19c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 -- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 msgr2=0x7fd31c19d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 0x7fd31c19d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 -- 172.21.2.117:0/1147866647 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c195730 con 0x7fd31c1b7d30 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31b2bf700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 0x7fd31c19d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd31aabe700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd30c009630 tx=0x7fd30c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd30c0093a0 con 0x7fd31c1b7d30 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00bd70 con 0x7fd31c1b7d30 2021-02-10T11:41:51.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd32152e700 1 -- 172.21.2.117:0/1147866647 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c195730 con 0x7fd31c1b7d30 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd32152e700 1 -- 172.21.2.117:0/1147866647 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c195730 con 0x7fd31c1b7d30 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.582 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd30c005560 con 0x7fd31c1b7d30 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.583 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd30c0052e0 con 0x7fd31c1b7d30 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.583 7fd313fff700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 0x7fd30001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.583 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd30c032710 con 0x7fd31c1b7d30 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.583 7fd31b2bf700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 0x7fd30001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:51.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.584 7fd31b2bf700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 0x7fd30001abc0 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:51.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.614 7fd32152e700 1 -- 172.21.2.117:0/1147866647 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c1b4f20 con 0x7fd31c1b7d30 2021-02-10T11:41:51.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.616 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd30c030d70 con 0x7fd31c1b7d30 2021-02-10T11:41:51.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.746 7fd32152e700 1 -- 172.21.2.117:0/1147866647 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd31c077a70 con 0x7fd31c1b7d30 2021-02-10T11:41:51.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.749 7fd313fff700 1 -- 172.21.2.117:0/1147866647 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd30c0052e0 con 0x7fd31c1b7d30 2021-02-10T11:41:51.736 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:41:51.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 -- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 msgr2=0x7fd30001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 0x7fd30001abc0 crc :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 -- 172.21.2.117:0/1147866647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 msgr2=0x7fd31c1b8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd30c009630 tx=0x7fd30c00b980).stop 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 -- 172.21.2.117:0/1147866647 shutdown_connections 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3000186c0 0x7fd30001abc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd31c1b7d30 0x7fd31c1b8130 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd31c19c090 0x7fd31c19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 --2- 172.21.2.117:0/1147866647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd31c1b8880 0x7fd31c19d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:51.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.751 7fd32152e700 1 -- 172.21.2.117:0/1147866647 >> 172.21.2.117:0/1147866647 conn(0x7fd31c1993a0 msgr2=0x7fd31c099190 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:51.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.752 7fd32152e700 1 -- 172.21.2.117:0/1147866647 shutdown_connections 2021-02-10T11:41:51.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:51.752 7fd32152e700 1 -- 172.21.2.117:0/1147866647 wait complete. 2021-02-10T11:41:51.747 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:41:52.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.182 7f47adb29700 1 Processor -- start 2021-02-10T11:41:52.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.182 7f47adb29700 1 -- start start 2021-02-10T11:41:52.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47adb29700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47a807da20 0x7f47a80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47adb29700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a80d1d20 con 0x7f47a80e4da0 2021-02-10T11:41:52.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47adb29700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a8077a70 con 0x7f47a80e51a0 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47adb29700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80777d0 con 0x7f47a807da20 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a6ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f47a80e4da0 0x7f47a81b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49382/0 (socket says 172.21.2.117:49382) 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a6ffd700 1 -- 172.21.2.117:0/15561301 learned_addr learned my addr 172.21.2.117:0/15561301 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a77fe700 1 --2- 172.21.2.117:0/15561301 >> v2:172.21.2.117:3300/0 conn(0x7f47a807da20 0x7f47a80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a77fe700 1 -- 172.21.2.117:0/15561301 >> v1:172.21.2.125:6789/0 conn(0x7f47a80e51a0 legacy=0x7f47a80d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a77fe700 1 -- 172.21.2.117:0/15561301 >> v1:172.21.2.117:6789/0 conn(0x7f47a80e4da0 legacy=0x7f47a81b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.183 7f47a77fe700 1 -- 172.21.2.117:0/15561301 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a81b08e0 con 0x7f47a807da20 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47a77fe700 1 --2- 172.21.2.117:0/15561301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a807da20 0x7f47a80771b0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f4798006650 tx=0x7f4798009a80).ready entity=mon.0 client_cookie=991848d3ae4bf1dd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47a67fc700 1 -- 172.21.2.117:0/15561301 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479800c0c0 con 0x7f47a807da20 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47a67fc700 1 -- 172.21.2.117:0/15561301 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800c340 con 0x7f47a807da20 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47a67fc700 1 -- 172.21.2.117:0/15561301 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479800c860 con 0x7f47a807da20 2021-02-10T11:41:52.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47adb29700 1 -- 172.21.2.117:0/15561301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a807da20 msgr2=0x7f47a80771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.184 7f47adb29700 1 --2- 172.21.2.117:0/15561301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a807da20 0x7f47a80771b0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f4798006650 tx=0x7f4798009a80).stop 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.185 7f47adb29700 1 -- 172.21.2.117:0/15561301 shutdown_connections 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.185 7f47adb29700 1 --2- 172.21.2.117:0/15561301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a807da20 0x7f47a80771b0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.185 7f47adb29700 1 -- 172.21.2.117:0/15561301 >> 172.21.2.117:0/15561301 conn(0x7f47a80ce450 msgr2=0x7f47a80e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.185 7f47adb29700 1 -- 172.21.2.117:0/15561301 shutdown_connections 2021-02-10T11:41:52.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.185 7f47adb29700 1 -- 172.21.2.117:0/15561301 wait complete. 2021-02-10T11:41:52.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.199 7f47adb29700 1 Processor -- start 2021-02-10T11:41:52.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.199 7f47adb29700 1 -- start start 2021-02-10T11:41:52.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a80c2e30 0x7f47a80c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a8099110 0x7f47a8099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80c3850 con 0x7f47a8090dd0 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80c3a70 con 0x7f47a80c2e30 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47adb29700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80911d0 con 0x7f47a8099110 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49138/0 (socket says 172.21.2.117:49138) 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 -- 172.21.2.117:0/215335854 learned_addr learned my addr 172.21.2.117:0/215335854 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 -- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a8099110 msgr2=0x7f47a8099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a8099110 0x7f47a8099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 -- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a80c2e30 msgr2=0x7f47a80c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a80c2e30 0x7f47a80c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 -- 172.21.2.117:0/215335854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a81c6640 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.200 7f47a77fe700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f4798009630 tx=0x7f479800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47980093a0 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800bd70 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47980054d0 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f47adb29700 1 -- 172.21.2.117:0/215335854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a81c6640 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f47adb29700 1 -- 172.21.2.117:0/215335854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a8099e20 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47980054f0 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.201 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4798032550 con 0x7f47a8090dd0 2021-02-10T11:41:52.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.202 7f47a6ffd700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.202 7f47a6ffd700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.232 7f47adb29700 1 -- 172.21.2.117:0/215335854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8103630 con 0x7f47a8090dd0 2021-02-10T11:41:52.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.235 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47980052e0 con 0x7f47a8090dd0 2021-02-10T11:41:52.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.363 7f47adb29700 1 -- 172.21.2.117:0/215335854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f47a80ce450 con 0x7f47a8090dd0 2021-02-10T11:41:52.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.366 7f479ffff700 1 -- 172.21.2.117:0/215335854 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f47980054f0 con 0x7f47a8090dd0 2021-02-10T11:41:52.356 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:41:52.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.371 7f47adb29700 1 -- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 msgr2=0x7f478c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.371 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 crc :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.371 7f47adb29700 1 -- 172.21.2.117:0/215335854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 msgr2=0x7f47a8098180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.371 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f4798009630 tx=0x7f479800b980).stop 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 -- 172.21.2.117:0/215335854 shutdown_connections 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47a8099110 0x7f47a8099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47a80c2e30 0x7f47a80c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f478c0186c0 0x7f478c01abc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 --2- 172.21.2.117:0/215335854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47a8090dd0 0x7f47a8098180 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 -- 172.21.2.117:0/215335854 >> 172.21.2.117:0/215335854 conn(0x7f47a80a01b0 msgr2=0x7f47a8091650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 -- 172.21.2.117:0/215335854 shutdown_connections 2021-02-10T11:41:52.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.372 7f47adb29700 1 -- 172.21.2.117:0/215335854 wait complete. 2021-02-10T11:41:52.367 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 Processor -- start 2021-02-10T11:41:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 -- start start 2021-02-10T11:41:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 0x7f96c00d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c0077a70 con 0x7f96c007da20 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c0079400 con 0x7f96c0079000 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c8567700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c007de20 con 0x7f96c00d1520 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.791 7f96c6383700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f96c007da20 0x7f96c00771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49396/0 (socket says 172.21.2.117:49396) 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6383700 1 -- 172.21.2.117:0/4218550464 learned_addr learned my addr 172.21.2.117:0/4218550464 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6b84700 1 --2- 172.21.2.117:0/4218550464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 0x7f96c00d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849887736 0 0) 0x7f96bc0010b0 con 0x7f96c007da20 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac002fe0 con 0x7f96c007da20 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663261007 0 0) 0x7f96b0000f30 con 0x7f96c0079000 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6b84700 1 -- 172.21.2.117:0/4218550464 >> v1:172.21.2.125:6789/0 conn(0x7f96c0079000 legacy=0x7f96c00e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6b84700 1 -- 172.21.2.117:0/4218550464 >> v1:172.21.2.117:6789/0 conn(0x7f96c007da20 legacy=0x7f96c00771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6b84700 1 -- 172.21.2.117:0/4218550464 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c01b00e0 con 0x7f96c00d1520 2021-02-10T11:41:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c6b84700 1 --2- 172.21.2.117:0/4218550464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 0x7f96c00d1920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f96b8006e50 tx=0x7f96b8009e70).ready entity=mon.2 client_cookie=58dd2cdddde5f3ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b800b510 con 0x7f96c00d1520 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8006790 con 0x7f96c00d1520 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c5381700 1 -- 172.21.2.117:0/4218550464 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b800b940 con 0x7f96c00d1520 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c8567700 1 -- 172.21.2.117:0/4218550464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 msgr2=0x7f96c00d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.792 7f96c8567700 1 --2- 172.21.2.117:0/4218550464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 0x7f96c00d1920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f96b8006e50 tx=0x7f96b8009e70).stop 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.793 7f96c8567700 1 -- 172.21.2.117:0/4218550464 shutdown_connections 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.793 7f96c8567700 1 --2- 172.21.2.117:0/4218550464 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d1520 0x7f96c00d1920 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.793 7f96c8567700 1 -- 172.21.2.117:0/4218550464 >> 172.21.2.117:0/4218550464 conn(0x7f96c00ce440 msgr2=0x7f96c00e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.793 7f96c8567700 1 -- 172.21.2.117:0/4218550464 shutdown_connections 2021-02-10T11:41:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.793 7f96c8567700 1 -- 172.21.2.117:0/4218550464 wait complete. 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 Processor -- start 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 -- start start 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c01a1fd0 0x7f96c019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01a23d0 con 0x7f96c019c030 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01a5a40 con 0x7f96c01a1fd0 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.808 7f96c8567700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c008a1b0 con 0x7f96c01a5640 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6b84700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c5b82700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c01a1fd0 0x7f96c019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6b84700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49150/0 (socket says 172.21.2.117:49150) 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6b84700 1 -- 172.21.2.117:0/504805493 learned_addr learned my addr 172.21.2.117:0/504805493 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:46830/0 (socket says 172.21.2.117:46830) 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 -- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c01a1fd0 msgr2=0x7f96c019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c01a1fd0 0x7f96c019ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 -- 172.21.2.117:0/504805493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 msgr2=0x7f96c019c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 -- 172.21.2.117:0/504805493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c019ca50 con 0x7f96c01a5640 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6b84700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.809 7f96c6383700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f96bc00a4c0 tx=0x7f96bc00a860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96bc0010b0 con 0x7f96c01a5640 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc000e20 con 0x7f96c01a5640 2021-02-10T11:41:52.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96bc004540 con 0x7f96c01a5640 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96c8567700 1 -- 172.21.2.117:0/504805493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c019ca50 con 0x7f96c01a5640 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96c8567700 1 -- 172.21.2.117:0/504805493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c00b1730 con 0x7f96c01a5640 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96bc001090 con 0x7f96c01a5640 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 0x7f96a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.810 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f96bc0325d0 con 0x7f96c01a5640 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.811 7f96c5b82700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 0x7f96a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:52.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.811 7f96c5b82700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 0x7f96a801abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:52.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.841 7f96c8567700 1 -- 172.21.2.117:0/504805493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c0108c10 con 0x7f96c01a5640 2021-02-10T11:41:52.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.845 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96bc001040 con 0x7f96c01a5640 2021-02-10T11:41:52.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.974 7f96c8567700 1 -- 172.21.2.117:0/504805493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c0077a70 con 0x7f96c01a5640 2021-02-10T11:41:52.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.974 7f96b6ffd700 1 -- 172.21.2.117:0/504805493 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f96bc007ef0 con 0x7f96c01a5640 2021-02-10T11:41:52.962 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 36 2021-02-10T11:41:52.962 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:52.962 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:41:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.976 7f96c8567700 1 -- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 msgr2=0x7f96a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.976 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 0x7f96a801abc0 crc :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.976 7f96c8567700 1 -- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 msgr2=0x7f96c01a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:52.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.976 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f96bc00a4c0 tx=0x7f96bc00a860).stop 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 -- 172.21.2.117:0/504805493 shutdown_connections 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a80186c0 0x7f96a801abc0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c019c030 0x7f96c019c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c01a1fd0 0x7f96c019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 --2- 172.21.2.117:0/504805493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c01a5640 0x7f96c01a19b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 -- 172.21.2.117:0/504805493 >> 172.21.2.117:0/504805493 conn(0x7f96c0191110 msgr2=0x7f96c019aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 -- 172.21.2.117:0/504805493 shutdown_connections 2021-02-10T11:41:52.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:52.977 7f96c8567700 1 -- 172.21.2.117:0/504805493 wait complete. 2021-02-10T11:41:53.173 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:46.636181', 'modified': '2021-02-10 11:41:48.646579', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5786}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5786': {'gid': 5786, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3915339620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3915339620}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3915339620}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:53.173 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:53.174 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 5786, 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.947068958, 2021-02-10T11:41:53.572 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 36, 2021-02-10T11:41:53.573 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 56, 2021-02-10T11:41:53.573 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 56, 2021-02-10T11:41:53.573 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.573460796999999 2021-02-10T11:41:53.573 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:41:53.576 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5786, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.947068958, "mdsmap_epoch": 36, "osdmap_epoch": 56, "osdmap_epoch_barrier": 56, "uptime": 12.573460796999999 } 2021-02-10T11:41:53.576 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:53.999 7f8c75969700 1 Processor -- start 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:53.999 7f8c75969700 1 -- start start 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:53.999 7f8c75969700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c7007da20 0x7f8c700771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c75969700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c700d1d10 con 0x7f8c700e4d90 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c75969700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c70077a70 con 0x7f8c700e5190 2021-02-10T11:41:54.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c75969700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c700777d0 con 0x7f8c7007da20 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6f574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8c7007da20 0x7f8c700771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6ed73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8c700e4d90 0x7f8c701b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49406/0 (socket says 172.21.2.117:49406) 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6ed73700 1 -- 172.21.2.117:0/3588707668 learned_addr learned my addr 172.21.2.117:0/3588707668 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6f574700 1 -- 172.21.2.117:0/3588707668 >> v1:172.21.2.125:6789/0 conn(0x7f8c700e5190 legacy=0x7f8c700d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6f574700 1 -- 172.21.2.117:0/3588707668 >> v1:172.21.2.117:6789/0 conn(0x7f8c700e4d90 legacy=0x7f8c701b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6f574700 1 -- 172.21.2.117:0/3588707668 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c701b08d0 con 0x7f8c7007da20 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.000 7f8c6f574700 1 --2- 172.21.2.117:0/3588707668 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7007da20 0x7f8c700771b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f8c58006650 tx=0x7f8c58009a00).ready entity=mon.0 client_cookie=3ff41a1665e9d2e7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c6e572700 1 -- 172.21.2.117:0/3588707668 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c580009b0 con 0x7f8c7007da20 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c6e572700 1 -- 172.21.2.117:0/3588707668 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5800c0c0 con 0x7f8c7007da20 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c6e572700 1 -- 172.21.2.117:0/3588707668 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c5800c520 con 0x7f8c7007da20 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 -- 172.21.2.117:0/3588707668 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7007da20 msgr2=0x7f8c700771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 --2- 172.21.2.117:0/3588707668 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7007da20 0x7f8c700771b0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f8c58006650 tx=0x7f8c58009a00).stop 2021-02-10T11:41:54.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 -- 172.21.2.117:0/3588707668 shutdown_connections 2021-02-10T11:41:54.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 --2- 172.21.2.117:0/3588707668 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7007da20 0x7f8c700771b0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 -- 172.21.2.117:0/3588707668 >> 172.21.2.117:0/3588707668 conn(0x7f8c700ce440 msgr2=0x7f8c700e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 -- 172.21.2.117:0/3588707668 shutdown_connections 2021-02-10T11:41:54.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.001 7f8c75969700 1 -- 172.21.2.117:0/3588707668 wait complete. 2021-02-10T11:41:54.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 Processor -- start 2021-02-10T11:41:54.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 -- start start 2021-02-10T11:41:54.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 0x7f8c70092e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c7010bbe0 0x7f8c7010bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7010d040 con 0x7f8c7010c600 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c70093450 con 0x7f8c7010bbe0 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c75969700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7010ca00 con 0x7f8c70091630 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c6fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.016 7f8c6fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49162/0 (socket says 172.21.2.117:49162) 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 -- 172.21.2.117:0/2500641258 learned_addr learned my addr 172.21.2.117:0/2500641258 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6f574700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 0x7f8c70092e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 -- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 msgr2=0x7f8c70092e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 0x7f8c70092e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 -- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c7010bbe0 msgr2=0x7f8c7010bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c7010bbe0 0x7f8c7010bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 -- 172.21.2.117:0/2500641258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7010cc20 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6f574700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 0x7f8c70092e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c6fd75700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f8c68009980 tx=0x7f8c6800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c68005af0 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c68005d70 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8c68006220 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c75969700 1 -- 172.21.2.117:0/2500641258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7010cc20 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.017 7f8c75969700 1 -- 172.21.2.117:0/2500641258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7010cc20 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.018 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8c68005870 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.018 7f8c67fff700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 0x7f8c5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.018 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8c68032a40 con 0x7f8c7010c600 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.018 7f8c6f574700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 0x7f8c5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.019 7f8c6f574700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 0x7f8c5401abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.049 7f8c75969700 1 -- 172.21.2.117:0/2500641258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c701035a0 con 0x7f8c7010c600 2021-02-10T11:41:54.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.052 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8c6800be10 con 0x7f8c7010c600 2021-02-10T11:41:54.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.177 7f8c75969700 1 -- 172.21.2.117:0/2500641258 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c700ce440 con 0x7f8c7010c600 2021-02-10T11:41:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.178 7f8c67fff700 1 -- 172.21.2.117:0/2500641258 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f8c68005870 con 0x7f8c7010c600 2021-02-10T11:41:54.165 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:54.165 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:41:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 -- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 msgr2=0x7f8c5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 0x7f8c5401abc0 crc :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 -- 172.21.2.117:0/2500641258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 msgr2=0x7f8c7008c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f8c68009980 tx=0x7f8c6800bcd0).stop 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 -- 172.21.2.117:0/2500641258 shutdown_connections 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c540186c0 0x7f8c5401abc0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8c7010c600 0x7f8c7008c230 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8c7010bbe0 0x7f8c7010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 --2- 172.21.2.117:0/2500641258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8c70091630 0x7f8c70092e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.179 7f8c75969700 1 -- 172.21.2.117:0/2500641258 >> 172.21.2.117:0/2500641258 conn(0x7f8c7009f900 msgr2=0x7f8c701037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.180 7f8c75969700 1 -- 172.21.2.117:0/2500641258 shutdown_connections 2021-02-10T11:41:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.180 7f8c75969700 1 -- 172.21.2.117:0/2500641258 wait complete. 2021-02-10T11:41:54.175 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:44.610269', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:45.683464', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:41:54.175 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.603 7f4b10d12700 1 Processor -- start 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 -- start start 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b0c07da20 0x7f4b0c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 0x7f4b0c079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c077a70 con 0x7f4b0c0e4da0 2021-02-10T11:41:54.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c0777d0 con 0x7f4b0c07da20 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.604 7f4b10d12700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c1b08e0 con 0x7f4b0c0e51a0 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.605 7f4b0b314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 0x7f4b0c079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.605 7f4b0ab13700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b0c07da20 0x7f4b0c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.605 7f4b0a312700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f4b0c0e4da0 0x7f4b0c1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42348/0 (socket says 172.21.2.117:42348) 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.605 7f4b0a312700 1 -- 172.21.2.117:0/2117702150 learned_addr learned my addr 172.21.2.117:0/2117702150 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0ab13700 1 -- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 msgr2=0x7f4b0c079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0ab13700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 0x7f4b0c079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0ab13700 1 -- 172.21.2.117:0/2117702150 >> v1:172.21.2.125:6789/0 conn(0x7f4b0c0e4da0 legacy=0x7f4b0c1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b09b11700 1 -- 172.21.2.117:0/2117702150 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023828159 0 0) 0x7f4afc0010b0 con 0x7f4b0c0e4da0 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0ab13700 1 -- 172.21.2.117:0/2117702150 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c07de20 con 0x7f4b0c07da20 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0b314700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 0x7f4b0c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.606 7f4b0ab13700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c07da20 0x7f4b0c0771b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4af4006650 tx=0x7f4af4009a80).ready entity=mon.1 client_cookie=fe8efa733f815723 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b09b11700 1 -- 172.21.2.117:0/2117702150 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4af400c0c0 con 0x7f4b0c07da20 2021-02-10T11:41:54.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b09b11700 1 -- 172.21.2.117:0/2117702150 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af400c340 con 0x7f4b0c07da20 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b09b11700 1 -- 172.21.2.117:0/2117702150 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4af400c860 con 0x7f4b0c07da20 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 -- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c07da20 msgr2=0x7f4b0c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c07da20 0x7f4b0c0771b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4af4006650 tx=0x7f4af4009a80).stop 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 -- 172.21.2.117:0/2117702150 shutdown_connections 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c0e51a0 0x7f4b0c079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 --2- 172.21.2.117:0/2117702150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c07da20 0x7f4b0c0771b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.607 7f4b10d12700 1 -- 172.21.2.117:0/2117702150 >> 172.21.2.117:0/2117702150 conn(0x7f4b0c0ce450 msgr2=0x7f4b0c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.608 7f4b10d12700 1 -- 172.21.2.117:0/2117702150 shutdown_connections 2021-02-10T11:41:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.608 7f4b10d12700 1 -- 172.21.2.117:0/2117702150 wait complete. 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 Processor -- start 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 -- start start 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c0b7fc0 0x7f4b0c1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c090710 0x7f4b0c090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c1054f0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c195a30 con 0x7f4b0c0b7fc0 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.622 7f4b10d12700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c195fb0 con 0x7f4b0c090710 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49178/0 (socket says 172.21.2.117:49178) 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 -- 172.21.2.117:0/1414268030 learned_addr learned my addr 172.21.2.117:0/1414268030 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 -- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c090710 msgr2=0x7f4b0c090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c090710 0x7f4b0c090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 -- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c0b7fc0 msgr2=0x7f4b0c1042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c0b7fc0 0x7f4b0c1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 -- 172.21.2.117:0/1414268030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c0b7b60 con 0x7f4b0c1048f0 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b0a312700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f4afc001c10 tx=0x7f4afc00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4afc0009b0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc0012c0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4afc004350 con 0x7f4b0c1048f0 2021-02-10T11:41:54.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c0b7b60 con 0x7f4b0c1048f0 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.623 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c0b7b60 con 0x7f4b0c1048f0 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.624 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4afc004140 con 0x7f4b0c1048f0 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.624 7f4b037fe700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 0x7f4af001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.624 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4afc0328d0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.624 7f4b0ab13700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 0x7f4af001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:54.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.625 7f4b0ab13700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 0x7f4af001abd0 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:54.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.654 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c103280 con 0x7f4b0c1048f0 2021-02-10T11:41:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.656 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4afc000dd0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.784 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4b0c103280 con 0x7f4b0c1048f0 2021-02-10T11:41:54.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.784 7f4b037fe700 1 -- 172.21.2.117:0/1414268030 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4afc00ace0 con 0x7f4b0c1048f0 2021-02-10T11:41:54.772 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:54.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 msgr2=0x7f4af001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 0x7f4af001abd0 crc :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 msgr2=0x7f4b0c104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:54.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f4afc001c10 tx=0x7f4afc00a8f0).stop 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 shutdown_connections 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4af00186d0 0x7f4af001abd0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b0c1048f0 0x7f4b0c104cf0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b0c090710 0x7f4b0c090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 --2- 172.21.2.117:0/1414268030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b0c0b7fc0 0x7f4b0c1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 >> 172.21.2.117:0/1414268030 conn(0x7f4b0c0a1850 msgr2=0x7f4b0c0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 shutdown_connections 2021-02-10T11:41:54.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:54.787 7f4b10d12700 1 -- 172.21.2.117:0/1414268030 wait complete. 2021-02-10T11:41:54.782 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:41:55.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.208 7f68b6f8c700 1 Processor -- start 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.208 7f68b6f8c700 1 -- start start 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b6f8c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68b007da20 0x7f68b00771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b6f8c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 0x7f68b0079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b6f8c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b0077a70 con 0x7f68b00e4da0 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b6f8c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b00777d0 con 0x7f68b007da20 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b6f8c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b01b08e0 con 0x7f68b00e51a0 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68b007da20 0x7f68b00771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68b007da20 0x7f68b00771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49186/0 (socket says 172.21.2.117:49186) 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 -- 172.21.2.117:0/2224620579 learned_addr learned my addr 172.21.2.117:0/2224620579 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:55.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b55a9700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 0x7f68b0079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 -- 172.21.2.117:0/2224620579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 msgr2=0x7f68b0079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 0x7f68b0079000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.209 7f68b4da8700 1 -- 172.21.2.117:0/2224620579 >> v1:172.21.2.125:6789/0 conn(0x7f68b00e4da0 legacy=0x7f68b01b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68b4da8700 1 -- 172.21.2.117:0/2224620579 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b007de20 con 0x7f68b007da20 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68b55a9700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 0x7f68b0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68b4da8700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b007da20 0x7f68b00771b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f68a8006a80 tx=0x7f68a8009f00).ready entity=mon.0 client_cookie=1c41e9e0e5bc1cad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68af7fe700 1 -- 172.21.2.117:0/2224620579 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a800b5c0 con 0x7f68b007da20 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68af7fe700 1 -- 172.21.2.117:0/2224620579 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a800b8c0 con 0x7f68b007da20 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68af7fe700 1 -- 172.21.2.117:0/2224620579 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a800bda0 con 0x7f68b007da20 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68b6f8c700 1 -- 172.21.2.117:0/2224620579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b007da20 msgr2=0x7f68b00771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.210 7f68b6f8c700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b007da20 0x7f68b00771b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f68a8006a80 tx=0x7f68a8009f00).stop 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 -- 172.21.2.117:0/2224620579 shutdown_connections 2021-02-10T11:41:55.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b00e51a0 0x7f68b0079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 --2- 172.21.2.117:0/2224620579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b007da20 0x7f68b00771b0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 -- 172.21.2.117:0/2224620579 >> 172.21.2.117:0/2224620579 conn(0x7f68b00ce450 msgr2=0x7f68b00e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 -- 172.21.2.117:0/2224620579 shutdown_connections 2021-02-10T11:41:55.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.211 7f68b6f8c700 1 -- 172.21.2.117:0/2224620579 wait complete. 2021-02-10T11:41:55.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.225 7f68b6f8c700 1 Processor -- start 2021-02-10T11:41:55.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.225 7f68b6f8c700 1 -- start start 2021-02-10T11:41:55.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68b01b5130 0x7f68b01b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b01b5b50 0x7f68b00b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00b8da0 con 0x7f68b01b5f50 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a4140 con 0x7f68b01b5130 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b6f8c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a4320 con 0x7f68b01b5b50 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49190/0 (socket says 172.21.2.117:49190) 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 -- 172.21.2.117:0/2244845895 learned_addr learned my addr 172.21.2.117:0/2244845895 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 -- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b01b5b50 msgr2=0x7f68b00b5670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b01b5b50 0x7f68b00b5670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 -- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68b01b5130 msgr2=0x7f68b01b5530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68b01b5130 0x7f68b01b5530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 -- 172.21.2.117:0/2244845895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b00b62b0 con 0x7f68b01b5f50 2021-02-10T11:41:55.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.226 7f68b55a9700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f68a0006f10 tx=0x7f68a000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.227 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a000bd80 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.227 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0006410 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.227 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68a0002220 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.227 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b00b62b0 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.227 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b00b6840 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.228 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68a000a030 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.228 7f68ad7fa700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 0x7f689001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.228 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f68a00326d0 con 0x7f68b01b5f50 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.228 7f68b4da8700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 0x7f689001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.229 7f68b4da8700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 0x7f689001abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.258 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b00a4320 con 0x7f68b01b5f50 2021-02-10T11:41:55.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.261 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f68a0000bc0 con 0x7f68b01b5f50 2021-02-10T11:41:55.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.388 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f68b00a4320 con 0x7f68b01b5f50 2021-02-10T11:41:55.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.388 7f68ad7fa700 1 -- 172.21.2.117:0/2244845895 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v56) v1 ==== 104+0+10 (secure 0 0 0) 0x7f68a000a030 con 0x7f68b01b5f50 2021-02-10T11:41:55.375 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:55.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 msgr2=0x7f689001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 0x7f689001abc0 crc :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 msgr2=0x7f68b00b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f68a0006f10 tx=0x7f68a000b990).stop 2021-02-10T11:41:55.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 shutdown_connections 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68900186c0 0x7f689001abc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68b01b5f50 0x7f68b00b5c90 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68b01b5b50 0x7f68b00b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 --2- 172.21.2.117:0/2244845895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68b01b5130 0x7f68b01b5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.390 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 >> 172.21.2.117:0/2244845895 conn(0x7f68b008bbc0 msgr2=0x7f68b00c3500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.391 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 shutdown_connections 2021-02-10T11:41:55.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.391 7f68b6f8c700 1 -- 172.21.2.117:0/2244845895 wait complete. 2021-02-10T11:41:55.386 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:41:55.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.813 7f90f1cb4700 1 Processor -- start 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.813 7f90f1cb4700 1 -- start start 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec07da20 0x7f90ec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90ec0e4da0 0x7f90ec1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f90ec0e51a0 0x7f90ec077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ec0e5c20 con 0x7f90ec0e4da0 2021-02-10T11:41:55.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90ec077a70 con 0x7f90ec0e51a0 2021-02-10T11:41:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90f1cb4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec0777d0 con 0x7f90ec07da20 2021-02-10T11:41:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90ec0e4da0 0x7f90ec1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90ec0e4da0 0x7f90ec1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49198/0 (socket says 172.21.2.117:49198) 2021-02-10T11:41:55.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 -- 172.21.2.117:0/1059026064 learned_addr learned my addr 172.21.2.117:0/1059026064 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eb7fe700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec07da20 0x7f90ec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90ebfff700 1 --2- 172.21.2.117:0/1059026064 >> v2:172.21.2.125:3300/0 conn(0x7f90ec0e51a0 0x7f90ec077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 -- 172.21.2.117:0/1059026064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec07da20 msgr2=0x7f90ec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec07da20 0x7f90ec0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 -- 172.21.2.117:0/1059026064 >> v2:172.21.2.125:3300/0 conn(0x7f90ec0e51a0 msgr2=0x7f90ec077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 --2- 172.21.2.117:0/1059026064 >> v2:172.21.2.125:3300/0 conn(0x7f90ec0e51a0 0x7f90ec077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 -- 172.21.2.117:0/1059026064 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec07de20 con 0x7f90ec0e4da0 2021-02-10T11:41:55.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.814 7f90eaffd700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec0e4da0 0x7f90ec1b00f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f90e0006700 tx=0x7f90e0009b80).ready entity=mon.0 client_cookie=b7b6b59b99c1db15 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90ea7fc700 1 -- 172.21.2.117:0/1059026064 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e000c0c0 con 0x7f90ec0e4da0 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90ea7fc700 1 -- 172.21.2.117:0/1059026064 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e000c400 con 0x7f90ec0e4da0 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90ea7fc700 1 -- 172.21.2.117:0/1059026064 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e000c8e0 con 0x7f90ec0e4da0 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 -- 172.21.2.117:0/1059026064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec0e4da0 msgr2=0x7f90ec1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec0e4da0 0x7f90ec1b00f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f90e0006700 tx=0x7f90e0009b80).stop 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 -- 172.21.2.117:0/1059026064 shutdown_connections 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 --2- 172.21.2.117:0/1059026064 >> v2:172.21.2.125:3300/0 conn(0x7f90ec0e51a0 0x7f90ec077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec0e4da0 0x7f90ec1b00f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 --2- 172.21.2.117:0/1059026064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec07da20 0x7f90ec0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 -- 172.21.2.117:0/1059026064 >> 172.21.2.117:0/1059026064 conn(0x7f90ec0ce450 msgr2=0x7f90ec0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 -- 172.21.2.117:0/1059026064 shutdown_connections 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.815 7f90f1cb4700 1 -- 172.21.2.117:0/1059026064 wait complete. 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 Processor -- start 2021-02-10T11:41:55.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 -- start start 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec0b7fc0 0x7f90ec1042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90ec090710 0x7f90ec090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec1054f0 con 0x7f90ec1048f0 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec195a30 con 0x7f90ec090710 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.830 7f90f1cb4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec195fb0 con 0x7f90ec0b7fc0 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49200/0 (socket says 172.21.2.117:49200) 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 -- 172.21.2.117:0/1649323173 learned_addr learned my addr 172.21.2.117:0/1649323173 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 -- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec0b7fc0 msgr2=0x7f90ec1042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec0b7fc0 0x7f90ec1042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 -- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90ec090710 msgr2=0x7f90ec090b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90ec090710 0x7f90ec090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 -- 172.21.2.117:0/1649323173 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0b7b60 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90eaffd700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f90e0006420 tx=0x7f90e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e00090d0 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e000bd50 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec0b7b60 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90e0005560 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.831 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec0b7b60 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.832 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f90e0009440 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.832 7f90d3fff700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 0x7f90cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.832 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f90e0032920 con 0x7f90ec1048f0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.832 7f90eb7fe700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 0x7f90cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:55.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.832 7f90eb7fe700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 0x7f90cc01abc0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:55.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.863 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec103280 con 0x7f90ec1048f0 2021-02-10T11:41:55.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.866 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90e0009030 con 0x7f90ec1048f0 2021-02-10T11:41:55.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.994 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f90ec103280 con 0x7f90ec1048f0 2021-02-10T11:41:55.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.994 7f90d3fff700 1 -- 172.21.2.117:0/1649323173 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v56) v1 ==== 100+0+10 (secure 0 0 0) 0x7f90e000bd50 con 0x7f90ec1048f0 2021-02-10T11:41:55.981 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 msgr2=0x7f90cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 0x7f90cc01abc0 crc :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 msgr2=0x7f90ec104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f90e0006420 tx=0x7f90e000b960).stop 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 shutdown_connections 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f90cc018730 0x7f90cc01abc0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90ec1048f0 0x7f90ec104cf0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90ec090710 0x7f90ec090b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 --2- 172.21.2.117:0/1649323173 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90ec0b7fc0 0x7f90ec1042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:55.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 >> 172.21.2.117:0/1649323173 conn(0x7f90ec0a1850 msgr2=0x7f90ec0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:55.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 shutdown_connections 2021-02-10T11:41:55.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:55.996 7f90f1cb4700 1 -- 172.21.2.117:0/1649323173 wait complete. 2021-02-10T11:41:55.992 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:41:55.992 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:41:56.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.415 7f49cd32d700 1 Processor -- start 2021-02-10T11:41:56.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.415 7f49cd32d700 1 -- start start 2021-02-10T11:41:56.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49cd32d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c807da20 0x7f49c80771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49cd32d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80d1d10 con 0x7f49c80e5190 2021-02-10T11:41:56.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49cd32d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c8077a70 con 0x7f49c80e4d90 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49cd32d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80777d0 con 0x7f49c807da20 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49c787e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f49c80e5190 0x7f49c80d0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49458/0 (socket says 172.21.2.117:49458) 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49c787e700 1 -- 172.21.2.117:0/3279028396 learned_addr learned my addr 172.21.2.117:0/3279028396 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426081320 0 0) 0x7f49c0001500 con 0x7f49c80e5190 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49c607b700 1 -- 172.21.2.117:0/3279028396 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b4002fe0 con 0x7f49c80e5190 2021-02-10T11:41:56.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.416 7f49c707d700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c807da20 0x7f49c80771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 515611879 0 0) 0x7f49c0001500 con 0x7f49c80e5190 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c807da20 msgr2=0x7f49c80771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c807da20 0x7f49c80771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 >> v1:172.21.2.125:6789/0 conn(0x7f49c80e4d90 legacy=0x7f49c81b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c81b08d0 con 0x7f49c80e5190 2021-02-10T11:41:56.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f49c0001a00 con 0x7f49c80e5190 2021-02-10T11:41:56.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 >> v1:172.21.2.117:6789/0 conn(0x7f49c80e5190 legacy=0x7f49c80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49b4003a90 0x7f49b4005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49b4006780 0x7f49b4008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49b4009500 0x7f49b400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400c2d0 con 0x7f49b4009500 2021-02-10T11:41:56.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400c4f0 con 0x7f49b4006780 2021-02-10T11:41:56.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c607b700 1 -- 172.21.2.117:0/3279028396 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400c710 con 0x7f49b4003a90 2021-02-10T11:41:56.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49b4009500 0x7f49b400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c707d700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49b4003a90 0x7f49b4005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c687c700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49b4006780 0x7f49b4008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 -- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49b4003a90 msgr2=0x7f49b4005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49b4003a90 0x7f49b4005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 -- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49b4006780 msgr2=0x7f49b4008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49b4006780 0x7f49b4008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.417 7f49c787e700 1 -- 172.21.2.117:0/3279028396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b4050980 con 0x7f49b4009500 2021-02-10T11:41:56.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49c787e700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49b4009500 0x7f49b400ba20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f49c0001860 tx=0x7f49c000a310).ready entity=mon.0 client_cookie=c844c2fc93aaf067 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c00070c0 con 0x7f49b4009500 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c00073c0 con 0x7f49b4009500 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49c607b700 1 -- 172.21.2.117:0/3279028396 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c00049a0 con 0x7f49b4009500 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 -- 172.21.2.117:0/3279028396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49b4009500 msgr2=0x7f49b400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 --2- 172.21.2.117:0/3279028396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49b4009500 0x7f49b400ba20 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f49c0001860 tx=0x7f49c000a310).stop 2021-02-10T11:41:56.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49c787e700 1 -- 172.21.2.117:0/3279028396 reap_dead start 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 -- 172.21.2.117:0/3279028396 shutdown_connections 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 -- 172.21.2.117:0/3279028396 >> 172.21.2.117:0/3279028396 conn(0x7f49c80ce440 msgr2=0x7f49c80e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 -- 172.21.2.117:0/3279028396 shutdown_connections 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.418 7f49cd32d700 1 -- 172.21.2.117:0/3279028396 wait complete. 2021-02-10T11:41:56.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 Processor -- start 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 -- start start 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 0x7f49c8192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c81067a0 0x7f49c8106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8193730 con 0x7f49c818cbd0 2021-02-10T11:41:56.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c818d630 con 0x7f49c81067a0 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49cd32d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c818d850 con 0x7f49c8192ba0 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.433 7f49c787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49220/0 (socket says 172.21.2.117:49220) 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 -- 172.21.2.117:0/3588098426 learned_addr learned my addr 172.21.2.117:0/3588098426 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c707d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 0x7f49c8192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 -- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 msgr2=0x7f49c8192fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 0x7f49c8192fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 -- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c81067a0 msgr2=0x7f49c8106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c81067a0 0x7f49c8106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 -- 172.21.2.117:0/3588098426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c818dc30 con 0x7f49c818cbd0 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c707d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 0x7f49c8192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:41:56.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49c787e700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f49c00019a0 tx=0x7f49c00078d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c00062e0 con 0x7f49c818cbd0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0006560 con 0x7f49c818cbd0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c818dc30 con 0x7f49c818cbd0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c818e1c0 con 0x7f49c818cbd0 2021-02-10T11:41:56.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.434 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49c0006a80 con 0x7f49c818cbd0 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.435 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49c0006740 con 0x7f49c818cbd0 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.435 7f49bffff700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 0x7f49ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.435 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f49c0032a00 con 0x7f49c818cbd0 2021-02-10T11:41:56.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.436 7f49c707d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 0x7f49ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:41:56.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.437 7f49c707d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 0x7f49ac01abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:41:56.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.466 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c8077a70 con 0x7f49c818cbd0 2021-02-10T11:41:56.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.469 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49c001f830 con 0x7f49c818cbd0 2021-02-10T11:41:56.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.595 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49c818d5f0 con 0x7f49c818cbd0 2021-02-10T11:41:56.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.596 7f49bffff700 1 -- 172.21.2.117:0/3588098426 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f49c0006740 con 0x7f49c818cbd0 2021-02-10T11:41:56.583 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 36 2021-02-10T11:41:56.583 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:41:56.583 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:41:56.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 msgr2=0x7f49ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 0x7f49ac01abc0 crc :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 msgr2=0x7f49c818cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:41:56.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f49c00019a0 tx=0x7f49c00078d0).stop 2021-02-10T11:41:56.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 shutdown_connections 2021-02-10T11:41:56.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ac0186c0 0x7f49ac01abc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c818cbd0 0x7f49c818cfd0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c81067a0 0x7f49c8106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 --2- 172.21.2.117:0/3588098426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c8192ba0 0x7f49c8192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.598 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 >> 172.21.2.117:0/3588098426 conn(0x7f49c819f510 msgr2=0x7f49c81abfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.599 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 shutdown_connections 2021-02-10T11:41:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:41:56.599 7f49cd32d700 1 -- 172.21.2.117:0/3588098426 wait complete. 2021-02-10T11:41:56.796 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:46.636181', 'modified': '2021-02-10 11:41:48.646579', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5786}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5786': {'gid': 5786, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3915339620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3915339620}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3915339620}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:41:56.796 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:41:56.797 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:41:57.194 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 5786, 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.569670516, 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 36, 2021-02-10T11:41:57.195 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 56, 2021-02-10T11:41:57.196 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 56, 2021-02-10T11:41:57.196 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.19606185 2021-02-10T11:41:57.196 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:41:57.199 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5786, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.569670516, "mdsmap_epoch": 36, "osdmap_epoch": 56, "osdmap_epoch_barrier": 56, "uptime": 16.19606185 } 2021-02-10T11:41:57.199 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:41:57.199 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:41:57.199 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:41:57.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:57.200 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:57.535 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:57.880 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:57.880 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:58.222 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:58.222 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:58.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:58.223 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:58.557 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:41:58.557 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:41:58.560 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:41:58.903 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:41:58.904 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:41:58.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[129222]: starting ceph client 2021-02-10T11:41:58.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:41:58.982 7fbb7f6faf80 -1 init, newargv = 0x5623c47960c0 newargc=9 2021-02-10T11:41:58.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[129222]: starting fuse 2021-02-10T11:41:59.246 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:41:59.247 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:41:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:41:59.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:59.248 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:41:59.583 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:41:59.584 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:41:59.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:59.585 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:41:59.585 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:41:59.585 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:41:59.585 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:41:59.585 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:59.586 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:41:59.587 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:41:59.587 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:41:59.587 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:41:59.587 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:59.587 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:41:59.588 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:41:59.589 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:41:59.949 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.129222.asok 2021-02-10T11:41:59.952 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.129222.asok 2021-02-10T11:41:59.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:41:59.952 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.129222.asok status 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "129222", 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:00.343 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 5861, 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 5861 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1203801861 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:00.344 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1203801861 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.5861 172.21.2.117:0/1203801861", 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1203801861", 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 36, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 56, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:42:00.345 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:00.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:00.347 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:00.347 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:00.683 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:00.683 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:00.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:00.683 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:01.029 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:01.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.451 7f5633af9700 1 Processor -- start 2021-02-10T11:42:01.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.451 7f5633af9700 1 -- start start 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5633af9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 0x7f562c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5633af9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f562c0e5190 0x7f562c0d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5633af9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f562c077a70 con 0x7f562c0e4d90 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5633af9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562c0e5c10 con 0x7f562c0e5190 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5633af9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c0777d0 con 0x7f562c07da20 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5632116700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f562c0e5190 0x7f562c0d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5631114700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f562c0e4d90 0x7f562c1b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42416/0 (socket says 172.21.2.117:42416) 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5631915700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 0x7f562c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.452 7f5631114700 1 -- 172.21.2.117:0/2226982291 learned_addr learned my addr 172.21.2.117:0/2226982291 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:01.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5631915700 1 -- 172.21.2.117:0/2226982291 >> v2:172.21.2.125:3300/0 conn(0x7f562c0e5190 msgr2=0x7f562c0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:01.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5631915700 1 --2- 172.21.2.117:0/2226982291 >> v2:172.21.2.125:3300/0 conn(0x7f562c0e5190 0x7f562c0d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5631915700 1 -- 172.21.2.117:0/2226982291 >> v1:172.21.2.125:6789/0 conn(0x7f562c0e4d90 legacy=0x7f562c1b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5631915700 1 -- 172.21.2.117:0/2226982291 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c1b08d0 con 0x7f562c07da20 2021-02-10T11:42:01.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5631915700 1 --2- 172.21.2.117:0/2226982291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 0x7f562c0771b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f5620006650 tx=0x7f5620009a80).ready entity=mon.2 client_cookie=ab4501ef51f298f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5630913700 1 -- 172.21.2.117:0/2226982291 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562000c0c0 con 0x7f562c07da20 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5630913700 1 -- 172.21.2.117:0/2226982291 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562000c340 con 0x7f562c07da20 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5630913700 1 -- 172.21.2.117:0/2226982291 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f562000c860 con 0x7f562c07da20 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5633af9700 1 -- 172.21.2.117:0/2226982291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 msgr2=0x7f562c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.453 7f5633af9700 1 --2- 172.21.2.117:0/2226982291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 0x7f562c0771b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f5620006650 tx=0x7f5620009a80).stop 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 -- 172.21.2.117:0/2226982291 shutdown_connections 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 --2- 172.21.2.117:0/2226982291 >> v2:172.21.2.125:3300/0 conn(0x7f562c0e5190 0x7f562c0d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 --2- 172.21.2.117:0/2226982291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c07da20 0x7f562c0771b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 -- 172.21.2.117:0/2226982291 >> 172.21.2.117:0/2226982291 conn(0x7f562c0ce440 msgr2=0x7f562c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 -- 172.21.2.117:0/2226982291 shutdown_connections 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.454 7f5633af9700 1 -- 172.21.2.117:0/2226982291 wait complete. 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 Processor -- start 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 -- start start 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f562c0937e0 0x7f562c092fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c18f250 0x7f562c0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c187f30 con 0x7f562c0b1bd0 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c187be0 con 0x7f562c0937e0 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5633af9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562c18f650 con 0x7f562c18f250 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5632116700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5632116700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49246/0 (socket says 172.21.2.117:49246) 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.469 7f5632116700 1 -- 172.21.2.117:0/3202471409 learned_addr learned my addr 172.21.2.117:0/3202471409 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 -- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c18f250 msgr2=0x7f562c0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c18f250 0x7f562c0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 -- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f562c0937e0 msgr2=0x7f562c092fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f562c0937e0 0x7f562c092fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 -- 172.21.2.117:0/3202471409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c0b1fd0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5632116700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f562400b970 tx=0x7f562400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5624000f90 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5624003030 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5624001400 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5633af9700 1 -- 172.21.2.117:0/3202471409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c0b1fd0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.470 7f5633af9700 1 -- 172.21.2.117:0/3202471409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c0b1fd0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.471 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5624000df0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.471 7f561e7fc700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 0x7f561401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.471 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5624032a30 con 0x7f562c0b1bd0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.472 7f5631915700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 0x7f561401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:01.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.472 7f5631915700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 0x7f561401abc0 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:01.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.502 7f5633af9700 1 -- 172.21.2.117:0/3202471409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c1b4f30 con 0x7f562c0b1bd0 2021-02-10T11:42:01.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.504 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56240066f0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.628 7f5633af9700 1 -- 172.21.2.117:0/3202471409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f562c1b4f30 con 0x7f562c0b1bd0 2021-02-10T11:42:01.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.629 7f561e7fc700 1 -- 172.21.2.117:0/3202471409 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f5624000df0 con 0x7f562c0b1bd0 2021-02-10T11:42:01.617 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:01.617 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:01.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 msgr2=0x7f561401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 0x7f561401abc0 crc :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 msgr2=0x7f562c1884b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:01.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f562400b970 tx=0x7f562400bd10).stop 2021-02-10T11:42:01.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 shutdown_connections 2021-02-10T11:42:01.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56140186c0 0x7f561401abc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f562c0b1bd0 0x7f562c1884b0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f562c18f250 0x7f562c0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 --2- 172.21.2.117:0/3202471409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f562c0937e0 0x7f562c092fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 >> 172.21.2.117:0/3202471409 conn(0x7f562c1ace70 msgr2=0x7f562c1af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 shutdown_connections 2021-02-10T11:42:01.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:01.631 7f5633af9700 1 -- 172.21.2.117:0/3202471409 wait complete. 2021-02-10T11:42:01.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:44.610269', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:45.683464', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:42:01.628 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:02.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 Processor -- start 2021-02-10T11:42:02.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 -- start start 2021-02-10T11:42:02.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23407da20 0x7fe2340771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2340e4da0 0x7fe2341b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe2340e51a0 0x7fe234077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2340e5c20 con 0x7fe2340e4da0 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe234077a70 con 0x7fe2340e51a0 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.055 7fe238d5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340777d0 con 0x7fe23407da20 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2340e4da0 0x7fe2341b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe2340e4da0 0x7fe2341b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49252/0 (socket says 172.21.2.117:49252) 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 -- 172.21.2.117:0/3949620282 learned_addr learned my addr 172.21.2.117:0/3949620282 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:02.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 -- 172.21.2.117:0/3949620282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23407da20 msgr2=0x7fe2340771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 --2- 172.21.2.117:0/3949620282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23407da20 0x7fe2340771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 -- 172.21.2.117:0/3949620282 >> v2:172.21.2.125:3300/0 conn(0x7fe2340e51a0 msgr2=0x7fe234077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 --2- 172.21.2.117:0/3949620282 >> v2:172.21.2.125:3300/0 conn(0x7fe2340e51a0 0x7fe234077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 -- 172.21.2.117:0/3949620282 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23407de20 con 0x7fe2340e4da0 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe232312700 1 --2- 172.21.2.117:0/3949620282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2340e4da0 0x7fe2341b00f0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fe21c006700 tx=0x7fe21c009b80).ready entity=mon.0 client_cookie=ac232e2742cc0a08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe231b11700 1 -- 172.21.2.117:0/3949620282 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c00c0c0 con 0x7fe2340e4da0 2021-02-10T11:42:02.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe231b11700 1 -- 172.21.2.117:0/3949620282 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c00c340 con 0x7fe2340e4da0 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.056 7fe231b11700 1 -- 172.21.2.117:0/3949620282 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c00c7a0 con 0x7fe2340e4da0 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 -- 172.21.2.117:0/3949620282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2340e4da0 msgr2=0x7fe2341b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 --2- 172.21.2.117:0/3949620282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2340e4da0 0x7fe2341b00f0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fe21c006700 tx=0x7fe21c009b80).stop 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 -- 172.21.2.117:0/3949620282 shutdown_connections 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 --2- 172.21.2.117:0/3949620282 >> v2:172.21.2.125:3300/0 conn(0x7fe2340e51a0 0x7fe234077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 --2- 172.21.2.117:0/3949620282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2340e4da0 0x7fe2341b00f0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 --2- 172.21.2.117:0/3949620282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23407da20 0x7fe2340771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 -- 172.21.2.117:0/3949620282 >> 172.21.2.117:0/3949620282 conn(0x7fe2340ce450 msgr2=0x7fe2340e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 -- 172.21.2.117:0/3949620282 shutdown_connections 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.057 7fe238d5a700 1 -- 172.21.2.117:0/3949620282 wait complete. 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 Processor -- start 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 -- start start 2021-02-10T11:42:02.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe23409bbe0 0x7fe234092d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23409bfe0 0x7fe2340afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340927d0 con 0x7fe234093370 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340b08d0 con 0x7fe23409bbe0 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe238d5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe234190400 con 0x7fe23409bfe0 2021-02-10T11:42:02.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe233314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe233314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49258/0 (socket says 172.21.2.117:49258) 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.072 7fe233314700 1 -- 172.21.2.117:0/1586651311 learned_addr learned my addr 172.21.2.117:0/1586651311 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 -- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23409bfe0 msgr2=0x7fe2340afc90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23409bfe0 0x7fe2340afc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 -- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe23409bbe0 msgr2=0x7fe234092d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe23409bbe0 0x7fe234092d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 -- 172.21.2.117:0/1586651311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2341908a0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe233314700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7fe22c009980 tx=0x7fe22c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe22c005af0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c005d30 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2341908a0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2341908a0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.073 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe22c0062b0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.074 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe22c006030 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.074 7fe22b7fe700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 0x7fe21801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.074 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe22c032ca0 con 0x7fe234093370 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.074 7fe232b13700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 0x7fe21801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.075 7fe232b13700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 0x7fe21801ac10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.105 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe234103370 con 0x7fe234093370 2021-02-10T11:42:02.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.108 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe22c005a60 con 0x7fe234093370 2021-02-10T11:42:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.235 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe234077a70 con 0x7fe234093370 2021-02-10T11:42:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.235 7fe22b7fe700 1 -- 172.21.2.117:0/1586651311 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe22c006030 con 0x7fe234093370 2021-02-10T11:42:02.224 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 msgr2=0x7fe21801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 0x7fe21801ac10 crc :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 msgr2=0x7fe2340b02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7fe22c009980 tx=0x7fe22c00bcd0).stop 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 shutdown_connections 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe218018730 0x7fe21801ac10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe234093370 0x7fe2340b02b0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe23409bfe0 0x7fe2340afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 --2- 172.21.2.117:0/1586651311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe23409bbe0 0x7fe234092d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 >> 172.21.2.117:0/1586651311 conn(0x7fe2340a1420 msgr2=0x7fe23409efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 shutdown_connections 2021-02-10T11:42:02.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.237 7fe238d5a700 1 -- 172.21.2.117:0/1586651311 wait complete. 2021-02-10T11:42:02.234 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:42:02.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 Processor -- start 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 -- start start 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c07da20 0x7fea2c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea2c0e4da0 0x7fea2c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c077a70 con 0x7fea2c0e51a0 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea2c0e5c20 con 0x7fea2c0e4da0 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.667 7fea30970700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c0777d0 con 0x7fea2c07da20 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea2c0e4da0 0x7fea2c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea2ae1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fea2c0e51a0 0x7fea2c077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49514/0 (socket says 172.21.2.117:49514) 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea2c0e4da0 0x7fea2c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49264/0 (socket says 172.21.2.117:49264) 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea2ae1d700 1 -- 172.21.2.117:0/1235844852 learned_addr learned my addr 172.21.2.117:0/1235844852 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea2a61c700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c07da20 0x7fea2c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 -- 172.21.2.117:0/1235844852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c07da20 msgr2=0x7fea2c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:02.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c07da20 0x7fea2c0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 -- 172.21.2.117:0/1235844852 >> v1:172.21.2.117:6789/0 conn(0x7fea2c0e51a0 legacy=0x7fea2c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 -- 172.21.2.117:0/1235844852 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c07de20 con 0x7fea2c0e4da0 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea29e1b700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c0e4da0 0x7fea2c1b00f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7fea14006650 tx=0x7fea14009a80).ready entity=mon.0 client_cookie=d4a3d5384d7af670 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.668 7fea2961a700 1 -- 172.21.2.117:0/1235844852 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea140009b0 con 0x7fea2c0e4da0 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea2961a700 1 -- 172.21.2.117:0/1235844852 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1400c0c0 con 0x7fea2c0e4da0 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea2961a700 1 -- 172.21.2.117:0/1235844852 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea1400c510 con 0x7fea2c0e4da0 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 -- 172.21.2.117:0/1235844852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c0e4da0 msgr2=0x7fea2c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c0e4da0 0x7fea2c1b00f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7fea14006650 tx=0x7fea14009a80).stop 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 -- 172.21.2.117:0/1235844852 shutdown_connections 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c0e4da0 0x7fea2c1b00f0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 --2- 172.21.2.117:0/1235844852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c07da20 0x7fea2c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 -- 172.21.2.117:0/1235844852 >> 172.21.2.117:0/1235844852 conn(0x7fea2c0ce450 msgr2=0x7fea2c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:02.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 -- 172.21.2.117:0/1235844852 shutdown_connections 2021-02-10T11:42:02.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.669 7fea30970700 1 -- 172.21.2.117:0/1235844852 wait complete. 2021-02-10T11:42:02.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 Processor -- start 2021-02-10T11:42:02.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 -- start start 2021-02-10T11:42:02.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c08e9d0 0x7fea2c107a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea2c08f750 0x7fea2c08fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1080a0 con 0x7fea2c1b6100 2021-02-10T11:42:02.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c1b6c30 con 0x7fea2c08f750 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea30970700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c08edd0 con 0x7fea2c08e9d0 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea29e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea29e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49270/0 (socket says 172.21.2.117:49270) 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.684 7fea29e1b700 1 -- 172.21.2.117:0/125940 learned_addr learned my addr 172.21.2.117:0/125940 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 -- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c08e9d0 msgr2=0x7fea2c107a80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c08e9d0 0x7fea2c107a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 -- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea2c08f750 msgr2=0x7fea2c08fb50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea2c08f750 0x7fea2c08fb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 -- 172.21.2.117:0/125940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c090170 con 0x7fea2c1b6100 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea29e1b700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7fea14009630 tx=0x7fea1400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea140093a0 con 0x7fea2c1b6100 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1400bd70 con 0x7fea2c1b6100 2021-02-10T11:42:02.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea14005560 con 0x7fea2c1b6100 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea30970700 1 -- 172.21.2.117:0/125940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c090170 con 0x7fea2c1b6100 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.685 7fea30970700 1 -- 172.21.2.117:0/125940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c090700 con 0x7fea2c1b6100 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.686 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea14008240 con 0x7fea2c1b6100 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.686 7fea22ffd700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 0x7fea1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.686 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fea140326e0 con 0x7fea2c1b6100 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.686 7fea2a61c700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 0x7fea1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:02.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.687 7fea2a61c700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 0x7fea1001abc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:02.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.717 7fea30970700 1 -- 172.21.2.117:0/125940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c08f220 con 0x7fea2c1b6100 2021-02-10T11:42:02.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.720 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea14008030 con 0x7fea2c1b6100 2021-02-10T11:42:02.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.849 7fea30970700 1 -- 172.21.2.117:0/125940 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fea2c08f220 con 0x7fea2c1b6100 2021-02-10T11:42:02.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.850 7fea22ffd700 1 -- 172.21.2.117:0/125940 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v56) v1 ==== 104+0+10 (secure 0 0 0) 0x7fea14008220 con 0x7fea2c1b6100 2021-02-10T11:42:02.838 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 msgr2=0x7fea1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 0x7fea1001abc0 crc :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 msgr2=0x7fea2c1b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7fea14009630 tx=0x7fea1400b980).stop 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 shutdown_connections 2021-02-10T11:42:02.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea100186c0 0x7fea1001abc0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea2c1b6100 0x7fea2c1b6500 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea2c08f750 0x7fea2c08fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 --2- 172.21.2.117:0/125940 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea2c08e9d0 0x7fea2c107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 >> 172.21.2.117:0/125940 conn(0x7fea2c0ffe30 msgr2=0x7fea2c0c33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 shutdown_connections 2021-02-10T11:42:02.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:02.852 7fea30970700 1 -- 172.21.2.117:0/125940 wait complete. 2021-02-10T11:42:02.848 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 Processor -- start 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 -- start start 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c07da20 0x7f922c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f922c0e4da0 0x7f922c1b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c077a70 con 0x7f922c0e51a0 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c0e5c20 con 0x7f922c0e4da0 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.273 7f9233fc1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0777d0 con 0x7f922c07da20 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f922c0e4da0 0x7f922c1b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f922c0e4da0 0x7f922c1b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49276/0 (socket says 172.21.2.117:49276) 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 -- 172.21.2.117:0/3301377963 learned_addr learned my addr 172.21.2.117:0/3301377963 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 -- 172.21.2.117:0/3301377963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c07da20 msgr2=0x7f922c0771b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 --2- 172.21.2.117:0/3301377963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c07da20 0x7f922c0771b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 -- 172.21.2.117:0/3301377963 >> v1:172.21.2.125:6789/0 conn(0x7f922c0e51a0 legacy=0x7f922c077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 -- 172.21.2.117:0/3301377963 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c07de20 con 0x7f922c0e4da0 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f92315dc700 1 --2- 172.21.2.117:0/3301377963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c0e4da0 0x7f922c1b00f0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f9228006700 tx=0x7f9228009b00).ready entity=mon.0 client_cookie=7c6643b84a38c4ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f9230ddb700 1 -- 172.21.2.117:0/3301377963 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f922800c0c0 con 0x7f922c0e4da0 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f9230ddb700 1 -- 172.21.2.117:0/3301377963 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922800c340 con 0x7f922c0e4da0 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f9230ddb700 1 -- 172.21.2.117:0/3301377963 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f922800c820 con 0x7f922c0e4da0 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f9233fc1700 1 -- 172.21.2.117:0/3301377963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c0e4da0 msgr2=0x7f922c1b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.274 7f9233fc1700 1 --2- 172.21.2.117:0/3301377963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c0e4da0 0x7f922c1b00f0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f9228006700 tx=0x7f9228009b00).stop 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 -- 172.21.2.117:0/3301377963 shutdown_connections 2021-02-10T11:42:03.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 --2- 172.21.2.117:0/3301377963 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c0e4da0 0x7f922c1b00f0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 --2- 172.21.2.117:0/3301377963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c07da20 0x7f922c0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 -- 172.21.2.117:0/3301377963 >> 172.21.2.117:0/3301377963 conn(0x7f922c0ce450 msgr2=0x7f922c0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 -- 172.21.2.117:0/3301377963 shutdown_connections 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.275 7f9233fc1700 1 -- 172.21.2.117:0/3301377963 wait complete. 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 Processor -- start 2021-02-10T11:42:03.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 -- start start 2021-02-10T11:42:03.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f922c095810 0x7f922c0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c0b2af0 0x7f922c0b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c095c10 con 0x7f922c096490 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0b3520 con 0x7f922c095810 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.290 7f9233fc1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0b3a00 con 0x7f922c0b2af0 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49284/0 (socket says 172.21.2.117:49284) 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 -- 172.21.2.117:0/2905402834 learned_addr learned my addr 172.21.2.117:0/2905402834 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 -- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c0b2af0 msgr2=0x7f922c0b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c0b2af0 0x7f922c0b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 -- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f922c095810 msgr2=0x7f922c0b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f922c095810 0x7f922c0b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 -- 172.21.2.117:0/2905402834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0b3c20 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f92325de700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f9224009940 tx=0x7f922400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9224000d30 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400c2f0 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f922400c750 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c0b3c20 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.291 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c0971a0 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.292 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9224001220 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.292 7f921e7fc700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 0x7f921401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.292 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f92240328e0 con 0x7f922c096490 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.292 7f9231ddd700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 0x7f921401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.293 7f9231ddd700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 0x7f921401abc0 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.323 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c1032a0 con 0x7f922c096490 2021-02-10T11:42:03.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.326 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f922400be30 con 0x7f922c096490 2021-02-10T11:42:03.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.454 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f922c077a70 con 0x7f922c096490 2021-02-10T11:42:03.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.454 7f921e7fc700 1 -- 172.21.2.117:0/2905402834 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v56) v1 ==== 100+0+10 (secure 0 0 0) 0x7f922401f880 con 0x7f922c096490 2021-02-10T11:42:03.443 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:03.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 msgr2=0x7f921401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 0x7f921401abc0 crc :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 msgr2=0x7f922c096890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f9224009940 tx=0x7f922400bcf0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 shutdown_connections 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92140186c0 0x7f921401abc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f922c096490 0x7f922c096890 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f922c0b2af0 0x7f922c0b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 --2- 172.21.2.117:0/2905402834 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f922c095810 0x7f922c0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.456 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 >> 172.21.2.117:0/2905402834 conn(0x7f922c0bc400 msgr2=0x7f922c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.457 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 shutdown_connections 2021-02-10T11:42:03.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.457 7f9233fc1700 1 -- 172.21.2.117:0/2905402834 wait complete. 2021-02-10T11:42:03.453 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:42:03.453 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.875 7f06381fa700 1 Processor -- start 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.875 7f06381fa700 1 -- start start 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 0x7f06300771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06300e4d90 0x7f06301b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300e5190 0x7f0630077ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0630077a70 con 0x7f06300e4d90 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06300777d0 con 0x7f063007da20 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f06381fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063007de20 con 0x7f06300e5190 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06300e4d90 0x7f06301b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f06300e4d90 0x7f06301b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49290/0 (socket says 172.21.2.117:49290) 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 -- 172.21.2.117:0/2850063664 learned_addr learned my addr 172.21.2.117:0/2850063664 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0636817700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300e5190 0x7f0630077ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0636016700 1 --2- 172.21.2.117:0/2850063664 >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 0x7f06300771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 -- 172.21.2.117:0/2850063664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300e5190 msgr2=0x7f0630077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300e5190 0x7f0630077ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 -- 172.21.2.117:0/2850063664 >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 msgr2=0x7f06300771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 --2- 172.21.2.117:0/2850063664 >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 0x7f06300771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0635815700 1 -- 172.21.2.117:0/2850063664 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06301b0700 con 0x7f06300e4d90 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.876 7f0636016700 1 --2- 172.21.2.117:0/2850063664 >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 0x7f06300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f0635815700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06300e4d90 0x7f06301b00e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f0624006700 tx=0x7f0624009b80).ready entity=mon.0 client_cookie=131e8813a675a566 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f0635014700 1 -- 172.21.2.117:0/2850063664 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062400c0c0 con 0x7f06300e4d90 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f0635014700 1 -- 172.21.2.117:0/2850063664 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062400c3c0 con 0x7f06300e4d90 2021-02-10T11:42:03.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f0635014700 1 -- 172.21.2.117:0/2850063664 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062400c810 con 0x7f06300e4d90 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f06381fa700 1 -- 172.21.2.117:0/2850063664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06300e4d90 msgr2=0x7f06301b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.877 7f06381fa700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06300e4d90 0x7f06301b00e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0x7f0624006700 tx=0x7f0624009b80).stop 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 -- 172.21.2.117:0/2850063664 shutdown_connections 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06300e5190 0x7f0630077ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 --2- 172.21.2.117:0/2850063664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06300e4d90 0x7f06301b00e0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 --2- 172.21.2.117:0/2850063664 >> v2:172.21.2.125:3300/0 conn(0x7f063007da20 0x7f06300771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 -- 172.21.2.117:0/2850063664 >> 172.21.2.117:0/2850063664 conn(0x7f06300ce440 msgr2=0x7f06300e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 -- 172.21.2.117:0/2850063664 shutdown_connections 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.878 7f06381fa700 1 -- 172.21.2.117:0/2850063664 wait complete. 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.892 7f06381fa700 1 Processor -- start 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.892 7f06381fa700 1 -- start start 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f063008cd50 0x7f063019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f063008d410 0x7f063008d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063008c080 con 0x7f063019dfe0 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063008c2a0 con 0x7f063008cd50 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f06381fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063008c4c0 con 0x7f063008d410 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49296/0 (socket says 172.21.2.117:49296) 2021-02-10T11:42:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 -- 172.21.2.117:0/2197717099 learned_addr learned my addr 172.21.2.117:0/2197717099 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0636016700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f063008cd50 0x7f063019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0636817700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f063008d410 0x7f063008d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 -- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f063008d410 msgr2=0x7f063008d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f063008d410 0x7f063008d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 -- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f063008cd50 msgr2=0x7f063019d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f063008cd50 0x7f063019d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 -- 172.21.2.117:0/2197717099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063008c6e0 con 0x7f063019dfe0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.893 7f0635815700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f062400b5e0 tx=0x7f062400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f062400bd70 con 0x7f063019dfe0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624002030 con 0x7f063019dfe0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f06381fa700 1 -- 172.21.2.117:0/2197717099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063008c6e0 con 0x7f063019dfe0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06240024c0 con 0x7f063019dfe0 2021-02-10T11:42:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f06381fa700 1 -- 172.21.2.117:0/2197717099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063008c6e0 con 0x7f063019dfe0 2021-02-10T11:42:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0624002210 con 0x7f063019dfe0 2021-02-10T11:42:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.894 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0624032730 con 0x7f063019dfe0 2021-02-10T11:42:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.895 7f0636016700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.895 7f0636016700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 0x7f061801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:03.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.926 7f06381fa700 1 -- 172.21.2.117:0/2197717099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0630077a70 con 0x7f063019dfe0 2021-02-10T11:42:03.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:03.929 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f062401f4a0 con 0x7f063019dfe0 2021-02-10T11:42:04.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.055 7f06381fa700 1 -- 172.21.2.117:0/2197717099 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0630077a70 con 0x7f063019dfe0 2021-02-10T11:42:04.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.055 7f0622ffd700 1 -- 172.21.2.117:0/2197717099 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f062400bd70 con 0x7f063019dfe0 2021-02-10T11:42:04.044 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 36 2021-02-10T11:42:04.044 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:04.044 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:42:04.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.057 7f06381fa700 1 -- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 msgr2=0x7f061801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.057 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 0x7f061801abc0 crc :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.057 7f06381fa700 1 -- 172.21.2.117:0/2197717099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 msgr2=0x7f063019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:04.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.057 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f062400b5e0 tx=0x7f062400b980).stop 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 -- 172.21.2.117:0/2197717099 shutdown_connections 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06180186c0 0x7f061801abc0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f063019dfe0 0x7f063019e3e0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f063008d410 0x7f063008d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 --2- 172.21.2.117:0/2197717099 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f063008cd50 0x7f063019d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 -- 172.21.2.117:0/2197717099 >> 172.21.2.117:0/2197717099 conn(0x7f063019f4c0 msgr2=0x7f0630094480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:04.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 -- 172.21.2.117:0/2197717099 shutdown_connections 2021-02-10T11:42:04.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:04.058 7f06381fa700 1 -- 172.21.2.117:0/2197717099 wait complete. 2021-02-10T11:42:04.256 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:46.636181', 'modified': '2021-02-10 11:41:48.646579', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5786}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5786': {'gid': 5786, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3915339620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3915339620}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3915339620}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:04.256 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:04.256 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:04.655 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 5786, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.030538904, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 36, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 56, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 56, 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.656929877 2021-02-10T11:42:04.656 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:42:04.660 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5786, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.030538904, "mdsmap_epoch": 36, "osdmap_epoch": 56, "osdmap_epoch_barrier": 56, "uptime": 23.656929877 } 2021-02-10T11:42:04.660 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:42:04.661 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:42:04.661 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:42:04.661 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:04.661 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:04.996 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:05.340 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:05.340 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:05.683 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:05.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:05.684 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:06.017 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:06.017 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:42:06.017 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:06.020 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:06.363 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:06.364 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:06.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[129862]: starting ceph client2021-02-10 11:42:06.443 7f4c3e387f80 -1 init, newargv = 0x55f07d47df10 newargc=9 2021-02-10T11:42:06.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr: 2021-02-10T11:42:06.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[129862]: starting fuse 2021-02-10T11:42:06.706 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:06.706 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:06.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:06.707 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:06.931 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:06.934 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:07.042 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:42:07.042 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:42:07.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:42:07.043 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:42:07.044 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:42:07.410 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.129862.asok 2021-02-10T11:42:07.413 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.129862.asok 2021-02-10T11:42:07.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:07.413 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.129862.asok status 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:42:07.803 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "129862", 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8178, 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8178 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:07.804 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2172889998 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2172889998 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8178 172.21.2.117:0/2172889998", 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2172889998", 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 36, 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 56, 2021-02-10T11:42:07.805 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:07.806 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:42:07.806 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:07.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:07.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:07.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:08.143 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:08.144 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:08.144 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:08.144 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:08.491 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:42:08.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.918 7f9e33544700 1 Processor -- start 2021-02-10T11:42:08.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.918 7f9e33544700 1 -- start start 2021-02-10T11:42:08.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.919 7f9e33544700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 0x7f9e2c0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.919 7f9e33544700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 0x7f9e2c0d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.919 7f9e33544700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c077a10 con 0x7f9e2c0e4e00 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.919 7f9e33544700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2c0e5c60 con 0x7f9e2c0e5200 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.919 7f9e33544700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0777d0 con 0x7f9e2c07da20 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31360700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 0x7f9e2c0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e30b5f700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f9e2c0e4e00 0x7f9e2c1b0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42492/0 (socket says 172.21.2.117:42492) 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e30b5f700 1 -- 172.21.2.117:0/2226356244 learned_addr learned my addr 172.21.2.117:0/2226356244 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31b61700 1 --2- 172.21.2.117:0/2226356244 >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 0x7f9e2c0d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31360700 1 -- 172.21.2.117:0/2226356244 >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 msgr2=0x7f9e2c0d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31360700 1 --2- 172.21.2.117:0/2226356244 >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 0x7f9e2c0d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31360700 1 -- 172.21.2.117:0/2226356244 >> v1:172.21.2.125:6789/0 conn(0x7f9e2c0e4e00 legacy=0x7f9e2c1b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e1ffff700 1 -- 172.21.2.117:0/2226356244 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818171105 0 0) 0x7f9e280010b0 con 0x7f9e2c0e4e00 2021-02-10T11:42:08.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31360700 1 -- 172.21.2.117:0/2226356244 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c1b0920 con 0x7f9e2c07da20 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.920 7f9e31b61700 1 --2- 172.21.2.117:0/2226356244 >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 0x7f9e2c0d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e31360700 1 --2- 172.21.2.117:0/2226356244 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 0x7f9e2c0771b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9e20006650 tx=0x7f9e20009a80).ready entity=mon.2 client_cookie=f0752871f593cfbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e1ffff700 1 -- 172.21.2.117:0/2226356244 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e200009b0 con 0x7f9e2c07da20 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e1ffff700 1 -- 172.21.2.117:0/2226356244 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2000c0c0 con 0x7f9e2c07da20 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e1ffff700 1 -- 172.21.2.117:0/2226356244 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e2000c5e0 con 0x7f9e2c07da20 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e33544700 1 -- 172.21.2.117:0/2226356244 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 msgr2=0x7f9e2c0771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.921 7f9e33544700 1 --2- 172.21.2.117:0/2226356244 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 0x7f9e2c0771b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9e20006650 tx=0x7f9e20009a80).stop 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 -- 172.21.2.117:0/2226356244 shutdown_connections 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 --2- 172.21.2.117:0/2226356244 >> v2:172.21.2.125:3300/0 conn(0x7f9e2c0e5200 0x7f9e2c0d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 --2- 172.21.2.117:0/2226356244 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c07da20 0x7f9e2c0771b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 -- 172.21.2.117:0/2226356244 >> 172.21.2.117:0/2226356244 conn(0x7f9e2c0ce480 msgr2=0x7f9e2c0e42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 -- 172.21.2.117:0/2226356244 shutdown_connections 2021-02-10T11:42:08.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.922 7f9e33544700 1 -- 172.21.2.117:0/2226356244 wait complete. 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.936 7f9e33544700 1 Processor -- start 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.936 7f9e33544700 1 -- start start 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c107cf0 0x7f9e2c093710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e2c094750 0x7f9e2c091830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0a2b20 con 0x7f9e2c093d30 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0a1bf0 con 0x7f9e2c094750 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e33544700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c093200 con 0x7f9e2c107cf0 2021-02-10T11:42:08.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49324/0 (socket says 172.21.2.117:49324) 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 -- 172.21.2.117:0/1196864301 learned_addr learned my addr 172.21.2.117:0/1196864301 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 -- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c107cf0 msgr2=0x7f9e2c093710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c107cf0 0x7f9e2c093710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 -- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e2c094750 msgr2=0x7f9e2c091830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e2c094750 0x7f9e2c091830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 -- 172.21.2.117:0/1196864301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c091e80 con 0x7f9e2c093d30 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.937 7f9e30b5f700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f9e28001c10 tx=0x7f9e2800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e280009b0 con 0x7f9e2c093d30 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280012c0 con 0x7f9e2c093d30 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e33544700 1 -- 172.21.2.117:0/1196864301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c091e80 con 0x7f9e2c093d30 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e28004350 con 0x7f9e2c093d30 2021-02-10T11:42:08.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e33544700 1 -- 172.21.2.117:0/1196864301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c092410 con 0x7f9e2c093d30 2021-02-10T11:42:08.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e28004140 con 0x7f9e2c093d30 2021-02-10T11:42:08.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 0x7f9e0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:08.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.938 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e28032560 con 0x7f9e2c093d30 2021-02-10T11:42:08.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.939 7f9e31360700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 0x7f9e0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:08.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.939 7f9e31360700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 0x7f9e0c01abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:08.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.969 7f9e33544700 1 -- 172.21.2.117:0/1196864301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c0a2ad0 con 0x7f9e2c093d30 2021-02-10T11:42:08.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:08.971 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e280009b0 con 0x7f9e2c093d30 2021-02-10T11:42:09.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.106 7f9e33544700 1 -- 172.21.2.117:0/1196864301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9e2c0a2ad0 con 0x7f9e2c093d30 2021-02-10T11:42:09.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.160 7f9e1dffb700 1 -- 172.21.2.117:0/1196864301 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9e28004140 con 0x7f9e2c093d30 2021-02-10T11:42:09.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.171 7f9e33544700 1 -- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 msgr2=0x7f9e0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.171 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 0x7f9e0c01abc0 crc :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.171 7f9e33544700 1 -- 172.21.2.117:0/1196864301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 msgr2=0x7f9e2c094130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.171 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f9e28001c10 tx=0x7f9e2800a8f0).stop 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 -- 172.21.2.117:0/1196864301 shutdown_connections 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e0c0186c0 0x7f9e0c01abc0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e2c093d30 0x7f9e2c094130 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e2c094750 0x7f9e2c091830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 --2- 172.21.2.117:0/1196864301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e2c107cf0 0x7f9e2c093710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 -- 172.21.2.117:0/1196864301 >> 172.21.2.117:0/1196864301 conn(0x7f9e2c0951f0 msgr2=0x7f9e2c090870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 -- 172.21.2.117:0/1196864301 shutdown_connections 2021-02-10T11:42:09.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.172 7f9e33544700 1 -- 172.21.2.117:0/1196864301 wait complete. 2021-02-10T11:42:09.168 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:42:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 Processor -- start 2021-02-10T11:42:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 -- start start 2021-02-10T11:42:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49c80d0ac0 0x7f49c81b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c81b3470 0x7f49c81b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 0x7f49c81089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c8109ab0 con 0x7f49c80d0ac0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80d0ec0 con 0x7f49c81085f0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49ce4fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80d10b0 con 0x7f49c81b3470 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49c7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49c80d0ac0 0x7f49c81b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49c7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f49c80d0ac0 0x7f49c81b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49328/0 (socket says 172.21.2.117:49328) 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.590 7f49c7fff700 1 -- 172.21.2.117:0/4045000290 learned_addr learned my addr 172.21.2.117:0/4045000290 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49ccb17700 1 --2- 172.21.2.117:0/4045000290 >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 0x7f49c81089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c77fe700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c81b3470 0x7f49c81b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 -- 172.21.2.117:0/4045000290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c81b3470 msgr2=0x7f49c81b3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c81b3470 0x7f49c81b3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 -- 172.21.2.117:0/4045000290 >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 msgr2=0x7f49c81089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 --2- 172.21.2.117:0/4045000290 >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 0x7f49c81089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 -- 172.21.2.117:0/4045000290 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8109010 con 0x7f49c80d0ac0 2021-02-10T11:42:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49ccb17700 1 --2- 172.21.2.117:0/4045000290 >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 0x7f49c81089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c7fff700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80d0ac0 0x7f49c81b2e50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f49b8006700 tx=0x7f49b8009b80).ready entity=mon.0 client_cookie=f52c61fc94ff69f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c6ffd700 1 -- 172.21.2.117:0/4045000290 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b80009b0 con 0x7f49c80d0ac0 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c6ffd700 1 -- 172.21.2.117:0/4045000290 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b800c0c0 con 0x7f49c80d0ac0 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49c6ffd700 1 -- 172.21.2.117:0/4045000290 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49b800c560 con 0x7f49c80d0ac0 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49ce4fa700 1 -- 172.21.2.117:0/4045000290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80d0ac0 msgr2=0x7f49c81b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.591 7f49ce4fa700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80d0ac0 0x7f49c81b2e50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f49b8006700 tx=0x7f49b8009b80).stop 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 -- 172.21.2.117:0/4045000290 shutdown_connections 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 --2- 172.21.2.117:0/4045000290 >> v2:172.21.2.125:3300/0 conn(0x7f49c81085f0 0x7f49c81089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c81b3470 0x7f49c81b3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 --2- 172.21.2.117:0/4045000290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80d0ac0 0x7f49c81b2e50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 -- 172.21.2.117:0/4045000290 >> 172.21.2.117:0/4045000290 conn(0x7f49c80ce410 msgr2=0x7f49c80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 -- 172.21.2.117:0/4045000290 shutdown_connections 2021-02-10T11:42:09.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.592 7f49ce4fa700 1 -- 172.21.2.117:0/4045000290 wait complete. 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.606 7f49ce4fa700 1 Processor -- start 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.606 7f49ce4fa700 1 -- start start 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c80bebb0 0x7f49c80befb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c80a0ad0 0x7f49c80c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80bf6e0 con 0x7f49c80a06d0 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80a7070 con 0x7f49c80bebb0 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49ce4fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80a6bd0 con 0x7f49c80a0ad0 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49336/0 (socket says 172.21.2.117:49336) 2021-02-10T11:42:09.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 -- 172.21.2.117:0/2814164690 learned_addr learned my addr 172.21.2.117:0/2814164690 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c7fff700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c80bebb0 0x7f49c80befb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 -- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c80a0ad0 msgr2=0x7f49c80c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c80a0ad0 0x7f49c80c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 -- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c80bebb0 msgr2=0x7f49c80befb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c80bebb0 0x7f49c80befb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.607 7f49c77fe700 1 -- 172.21.2.117:0/2814164690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80c0820 con 0x7f49c80a06d0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c77fe700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f49bc006f00 tx=0x7f49bc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49bc00bd30 con 0x7f49c80a06d0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc002030 con 0x7f49c80a06d0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49bc002490 con 0x7f49c80a06d0 2021-02-10T11:42:09.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c80c0820 con 0x7f49c80a06d0 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c80c0db0 con 0x7f49c80a06d0 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49bc000c90 con 0x7f49c80a06d0 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 0x7f49a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.608 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f49bc032710 con 0x7f49c80a06d0 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.609 7f49c7fff700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 0x7f49a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:09.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.610 7f49c7fff700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 0x7f49a801abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:09.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.640 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c80eda10 con 0x7f49c80a06d0 2021-02-10T11:42:09.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.643 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49bc02cf40 con 0x7f49c80a06d0 2021-02-10T11:42:09.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.776 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f49c80c0db0 con 0x7f49a80186c0 2021-02-10T11:42:09.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.778 7f49c4ff9700 1 -- 172.21.2.117:0/2814164690 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f49b8009050 con 0x7f49a80186c0 2021-02-10T11:42:09.767 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:42:09.767 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:09.767 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:42:09.767 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:09.767 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:42:09.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 msgr2=0x7f49a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 0x7f49a801abc0 crc :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 msgr2=0x7f49c80bfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:09.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f49bc006f00 tx=0x7f49bc00b940).stop 2021-02-10T11:42:09.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 shutdown_connections 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49a80186c0 0x7f49a801abc0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f49c80a06d0 0x7f49c80bfbe0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f49c80a0ad0 0x7f49c80c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 --2- 172.21.2.117:0/2814164690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f49c80bebb0 0x7f49c80befb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 >> 172.21.2.117:0/2814164690 conn(0x7f49c80aa2c0 msgr2=0x7f49c80b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 shutdown_connections 2021-02-10T11:42:09.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:09.781 7f49ce4fa700 1 -- 172.21.2.117:0/2814164690 wait complete. 2021-02-10T11:42:09.776 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 Processor -- start 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 -- start start 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0fec07da20 0x7f0fec0771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 0x7f0fec079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec077a70 con 0x7f0fec0e4da0 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fec0777d0 con 0x7f0fec07da20 2021-02-10T11:42:10.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff47ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1b08e0 con 0x7f0fec0e51a0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff1e07700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0fec0e4da0 0x7f0fec1b00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49592/0 (socket says 172.21.2.117:49592) 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.204 7f0ff1e07700 1 -- 172.21.2.117:0/4008343897 learned_addr learned my addr 172.21.2.117:0/4008343897 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2608700 1 --2- 172.21.2.117:0/4008343897 >> v2:172.21.2.125:3300/0 conn(0x7f0fec07da20 0x7f0fec0771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 --2- 172.21.2.117:0/4008343897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 0x7f0fec079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff1606700 1 -- 172.21.2.117:0/4008343897 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066717619 0 0) 0x7f0fe80009b0 con 0x7f0fec0e4da0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff1606700 1 -- 172.21.2.117:0/4008343897 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8002fe0 con 0x7f0fec0e4da0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 -- 172.21.2.117:0/4008343897 >> v2:172.21.2.125:3300/0 conn(0x7f0fec07da20 msgr2=0x7f0fec0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 --2- 172.21.2.117:0/4008343897 >> v2:172.21.2.125:3300/0 conn(0x7f0fec07da20 0x7f0fec0771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 -- 172.21.2.117:0/4008343897 >> v1:172.21.2.117:6789/0 conn(0x7f0fec0e4da0 legacy=0x7f0fec1b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 -- 172.21.2.117:0/4008343897 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec07de20 con 0x7f0fec0e51a0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff2e09700 1 --2- 172.21.2.117:0/4008343897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 0x7f0fec079000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0fe4006e50 tx=0x7f0fe4009e70).ready entity=mon.2 client_cookie=272adb12407e34ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff1606700 1 -- 172.21.2.117:0/4008343897 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe400b510 con 0x7f0fec0e51a0 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff1606700 1 -- 172.21.2.117:0/4008343897 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4006790 con 0x7f0fec0e51a0 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.205 7f0ff1606700 1 -- 172.21.2.117:0/4008343897 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe400b940 con 0x7f0fec0e51a0 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 -- 172.21.2.117:0/4008343897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 msgr2=0x7f0fec079000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 --2- 172.21.2.117:0/4008343897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 0x7f0fec079000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0fe4006e50 tx=0x7f0fe4009e70).stop 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 -- 172.21.2.117:0/4008343897 shutdown_connections 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 --2- 172.21.2.117:0/4008343897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec0e51a0 0x7f0fec079000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 --2- 172.21.2.117:0/4008343897 >> v2:172.21.2.125:3300/0 conn(0x7f0fec07da20 0x7f0fec0771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.206 7f0ff47ec700 1 -- 172.21.2.117:0/4008343897 >> 172.21.2.117:0/4008343897 conn(0x7f0fec0ce450 msgr2=0x7f0fec0e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:10.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.207 7f0ff47ec700 1 -- 172.21.2.117:0/4008343897 shutdown_connections 2021-02-10T11:42:10.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.207 7f0ff47ec700 1 -- 172.21.2.117:0/4008343897 wait complete. 2021-02-10T11:42:10.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.221 7f0ff47ec700 1 Processor -- start 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.221 7f0ff47ec700 1 -- start start 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec187e20 0x7f0fec188220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fec090420 0x7f0fec090820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1067f0 con 0x7f0fec1a0250 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec107350 con 0x7f0fec090420 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fec1042d0 con 0x7f0fec187e20 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49348/0 (socket says 172.21.2.117:49348) 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 -- 172.21.2.117:0/3160973148 learned_addr learned my addr 172.21.2.117:0/3160973148 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 -- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec187e20 msgr2=0x7f0fec188220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec187e20 0x7f0fec188220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 -- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fec090420 msgr2=0x7f0fec090820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fec090420 0x7f0fec090820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 -- 172.21.2.117:0/3160973148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec090e40 con 0x7f0fec1a0250 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff2608700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f0fe0006f60 tx=0x7f0fe000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe0006340 con 0x7f0fec1a0250 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe000bdf0 con 0x7f0fec1a0250 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec090e40 con 0x7f0fec1a0250 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe0002030 con 0x7f0fec1a0250 2021-02-10T11:42:10.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.222 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec0913d0 con 0x7f0fec1a0250 2021-02-10T11:42:10.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.223 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fe0000c90 con 0x7f0fec1a0250 2021-02-10T11:42:10.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.223 7f0fdeffd700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 0x7f0fd401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:10.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.223 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0fe0032970 con 0x7f0fec1a0250 2021-02-10T11:42:10.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.223 7f0ff1e07700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 0x7f0fd401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:10.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.224 7f0ff1e07700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 0x7f0fd401ac10 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:10.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.254 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec103280 con 0x7f0fec1a0250 2021-02-10T11:42:10.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.257 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fe0000960 con 0x7f0fec1a0250 2021-02-10T11:42:10.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.407 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0fec077a70 con 0x7f0fec1a0250 2021-02-10T11:42:10.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.408 7f0fdeffd700 1 -- 172.21.2.117:0/3160973148 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0fe0000c90 con 0x7f0fec1a0250 2021-02-10T11:42:10.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 msgr2=0x7f0fd401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 0x7f0fd401ac10 crc :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 msgr2=0x7f0fec106cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:10.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f0fe0006f60 tx=0x7f0fe000b970).stop 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 shutdown_connections 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fec090420 0x7f0fec090820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fec187e20 0x7f0fec188220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd4018730 0x7f0fd401ac10 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 --2- 172.21.2.117:0/3160973148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fec1a0250 0x7f0fec106cf0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 >> 172.21.2.117:0/3160973148 conn(0x7f0fec0a1850 msgr2=0x7f0fec0c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 shutdown_connections 2021-02-10T11:42:10.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:10.410 7f0ff47ec700 1 -- 172.21.2.117:0/3160973148 wait complete. 2021-02-10T11:42:10.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:10.407 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:10.407 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:10.742 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:10.742 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:10.742 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:42:10.743 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:10.743 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:11.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[129862]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:11.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:11.096 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:11.097 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:11.430 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:42:11.430 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:11.431 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:42:11.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:11.431 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:11.766 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p ./volumes/group_0000000000003266/subvolume_0000000000601930 2021-02-10T11:42:13.775 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.layout.pool . 2021-02-10T11:42:14.121 INFO:teuthology.orchestra.run.gibba017.stdout:cephfs_data 2021-02-10T11:42:14.121 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo setfattr -n ceph.dir.layout.pool -v cephfs_data ./volumes/group_0000000000003266/subvolume_0000000000601930 2021-02-10T11:42:14.486 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000601930 alice --group_name group_0000000000003266 --tenant_id tenant_id 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.920 7fcf836ab700 1 Processor -- start 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.920 7fcf836ab700 1 -- start start 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf836ab700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 0x7fcf7c0e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf836ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 0x7fcf7c1b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf836ab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c077a70 con 0x7fcf7c07da20 2021-02-10T11:42:14.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf836ab700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c0e53b0 con 0x7fcf7c079000 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf836ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c079400 con 0x7fcf7c1b00e0 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf81cc8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 0x7fcf7c1b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf814c7700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcf7c07da20 0x7fcf7c0771b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42530/0 (socket says 172.21.2.117:42530) 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf814c7700 1 -- 172.21.2.117:0/2879158972 learned_addr learned my addr 172.21.2.117:0/2879158972 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.921 7fcf80cc6700 1 --2- 172.21.2.117:0/2879158972 >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 0x7fcf7c0e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf81cc8700 1 -- 172.21.2.117:0/2879158972 >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 msgr2=0x7fcf7c0e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf81cc8700 1 --2- 172.21.2.117:0/2879158972 >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 0x7fcf7c0e4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf81cc8700 1 -- 172.21.2.117:0/2879158972 >> v1:172.21.2.125:6789/0 conn(0x7fcf7c07da20 legacy=0x7fcf7c0771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf81cc8700 1 -- 172.21.2.117:0/2879158972 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c07de20 con 0x7fcf7c1b00e0 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf6ffff700 1 -- 172.21.2.117:0/2879158972 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847243442 0 0) 0x7fcf700010b0 con 0x7fcf7c07da20 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf80cc6700 1 --2- 172.21.2.117:0/2879158972 >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 0x7fcf7c0e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.922 7fcf81cc8700 1 --2- 172.21.2.117:0/2879158972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 0x7fcf7c1b04e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcf74006e50 tx=0x7fcf74009e70).ready entity=mon.2 client_cookie=88819a11474f937a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf6ffff700 1 -- 172.21.2.117:0/2879158972 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf74000f90 con 0x7fcf7c1b00e0 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf6ffff700 1 -- 172.21.2.117:0/2879158972 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf740066f0 con 0x7fcf7c1b00e0 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf6ffff700 1 -- 172.21.2.117:0/2879158972 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf7400b700 con 0x7fcf7c1b00e0 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 -- 172.21.2.117:0/2879158972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 msgr2=0x7fcf7c1b04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 --2- 172.21.2.117:0/2879158972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 0x7fcf7c1b04e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcf74006e50 tx=0x7fcf74009e70).stop 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 -- 172.21.2.117:0/2879158972 shutdown_connections 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 --2- 172.21.2.117:0/2879158972 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1b00e0 0x7fcf7c1b04e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 --2- 172.21.2.117:0/2879158972 >> v2:172.21.2.125:3300/0 conn(0x7fcf7c079000 0x7fcf7c0e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.923 7fcf836ab700 1 -- 172.21.2.117:0/2879158972 >> 172.21.2.117:0/2879158972 conn(0x7fcf7c0ce440 msgr2=0x7fcf7c0e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.924 7fcf836ab700 1 -- 172.21.2.117:0/2879158972 shutdown_connections 2021-02-10T11:42:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.924 7fcf836ab700 1 -- 172.21.2.117:0/2879158972 wait complete. 2021-02-10T11:42:14.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 Processor -- start 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 -- start start 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf7c18d2f0 0x7fcf7c106e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1aa5b0 0x7fcf7c1aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c198e00 con 0x7fcf7c18cef0 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c1062e0 con 0x7fcf7c18d2f0 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.938 7fcf836ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c1aafd0 con 0x7fcf7c1aa5b0 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49362/0 (socket says 172.21.2.117:49362) 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 -- 172.21.2.117:0/2384364280 learned_addr learned my addr 172.21.2.117:0/2384364280 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 -- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1aa5b0 msgr2=0x7fcf7c1aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1aa5b0 0x7fcf7c1aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 -- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf7c18d2f0 msgr2=0x7fcf7c106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf7c18d2f0 0x7fcf7c106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 -- 172.21.2.117:0/2384364280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c1ab1b0 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf814c7700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7fcf70001c60 tx=0x7fcf7000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf7000acc0 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70004140 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf70004660 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c1ab1b0 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.939 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c1ab1b0 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.940 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf70001090 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.940 7fcf6dffb700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 0x7fcf5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.940 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcf700326b0 con 0x7fcf7c18cef0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.940 7fcf80cc6700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 0x7fcf5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:14.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.941 7fcf80cc6700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 0x7fcf5c01abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:14.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.970 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c1abd20 con 0x7fcf7c18cef0 2021-02-10T11:42:14.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:14.973 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf70000980 con 0x7fcf7c18cef0 2021-02-10T11:42:15.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.097 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000601930", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000003266"}) v1 -- 0x7fcf7c1062e0 con 0x7fcf5c0186c0 2021-02-10T11:42:15.089 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:42:15.101 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:15.089 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:42:15.101 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:15.089 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:42:15.101 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:15.089 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:42:15.101 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:15.090 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:42:15.101 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:42:15.114 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:42:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.157 7fcf6dffb700 1 -- 172.21.2.117:0/2384364280 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fcf78006420 con 0x7fcf5c0186c0 2021-02-10T11:42:15.149 INFO:teuthology.orchestra.run.gibba017.stdout:AQCXxiNggbRSCBAA2kNC8AICLGeWqAj2g/KXLw== 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.162 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 msgr2=0x7fcf5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.162 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 0x7fcf5c01abc0 crc :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.162 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 msgr2=0x7fcf7c106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.162 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7fcf70001c60 tx=0x7fcf7000a8d0).stop 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.162 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 shutdown_connections 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf7c1aa5b0 0x7fcf7c1aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf7c18d2f0 0x7fcf7c106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf5c0186c0 0x7fcf5c01abc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 --2- 172.21.2.117:0/2384364280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf7c18cef0 0x7fcf7c106860 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 >> 172.21.2.117:0/2384364280 conn(0x7fcf7c09cb70 msgr2=0x7fcf7c1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 shutdown_connections 2021-02-10T11:42:15.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.163 7fcf836ab700 1 -- 172.21.2.117:0/2384364280 wait complete. 2021-02-10T11:42:15.159 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:42:15.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 Processor -- start 2021-02-10T11:42:15.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 -- start start 2021-02-10T11:42:15.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6714079000 0x7f67140e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67141b00e0 0x7f67141b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6714077a70 con 0x7f671407da20 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67140e53b0 con 0x7f6714079000 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f671aa4f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6714079400 con 0x7f67141b00e0 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f6713fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6714079000 0x7f67140e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f6713fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6714079000 0x7f67140e4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49366/0 (socket says 172.21.2.117:49366) 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.581 7f6713fff700 1 -- 172.21.2.117:0/370733898 learned_addr learned my addr 172.21.2.117:0/370733898 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f6713fff700 1 -- 172.21.2.117:0/370733898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67141b00e0 msgr2=0x7f67141b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f6713fff700 1 --2- 172.21.2.117:0/370733898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67141b00e0 0x7f67141b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f6713fff700 1 -- 172.21.2.117:0/370733898 >> v1:172.21.2.125:6789/0 conn(0x7f671407da20 legacy=0x7f67140771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f6713fff700 1 -- 172.21.2.117:0/370733898 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671407de20 con 0x7f6714079000 2021-02-10T11:42:15.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f6713fff700 1 --2- 172.21.2.117:0/370733898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6714079000 0x7f67140e4d90 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f6704006a80 tx=0x7f6704009f00).ready entity=mon.0 client_cookie=d72b00b933d08f52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f67137fe700 1 -- 172.21.2.117:0/370733898 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670400b5c0 con 0x7f6714079000 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f67137fe700 1 -- 172.21.2.117:0/370733898 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400b8c0 con 0x7f6714079000 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f67137fe700 1 -- 172.21.2.117:0/370733898 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670400bd20 con 0x7f6714079000 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f671aa4f700 1 -- 172.21.2.117:0/370733898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6714079000 msgr2=0x7f67140e4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f671aa4f700 1 --2- 172.21.2.117:0/370733898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6714079000 0x7f67140e4d90 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0x7f6704006a80 tx=0x7f6704009f00).stop 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.582 7f671aa4f700 1 -- 172.21.2.117:0/370733898 shutdown_connections 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.583 7f671aa4f700 1 --2- 172.21.2.117:0/370733898 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67141b00e0 0x7f67141b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.583 7f671aa4f700 1 --2- 172.21.2.117:0/370733898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6714079000 0x7f67140e4d90 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.583 7f671aa4f700 1 -- 172.21.2.117:0/370733898 >> 172.21.2.117:0/370733898 conn(0x7f67140ce440 msgr2=0x7f67140e4280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.583 7f671aa4f700 1 -- 172.21.2.117:0/370733898 shutdown_connections 2021-02-10T11:42:15.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.583 7f671aa4f700 1 -- 172.21.2.117:0/370733898 wait complete. 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.597 7f671aa4f700 1 Processor -- start 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.597 7f671aa4f700 1 -- start start 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f671419d920 0x7f6714109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f671410a210 0x7f67141a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671419d3a0 con 0x7f67141a8c60 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67141af340 con 0x7f671410a210 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671aa4f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671419aea0 con 0x7f671419d920 2021-02-10T11:42:15.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49376/0 (socket says 172.21.2.117:49376) 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 -- 172.21.2.117:0/1961250971 learned_addr learned my addr 172.21.2.117:0/1961250971 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 -- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f671419d920 msgr2=0x7f6714109bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f671419d920 0x7f6714109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 -- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f671410a210 msgr2=0x7f67141a8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f671410a210 0x7f67141a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 -- 172.21.2.117:0/1961250971 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671410a610 con 0x7f67141a8c60 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.598 7f671906c700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f670c006f10 tx=0x7f670c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670c006240 con 0x7f67141a8c60 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c000c10 con 0x7f67141a8c60 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f670c0056e0 con 0x7f67141a8c60 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671410a610 con 0x7f67141a8c60 2021-02-10T11:42:15.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671410a610 con 0x7f67141a8c60 2021-02-10T11:42:15.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f670c006030 con 0x7f67141a8c60 2021-02-10T11:42:15.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 0x7f66f801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:15.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f670c0327a0 con 0x7f67141a8c60 2021-02-10T11:42:15.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.599 7f671886b700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 0x7f66f801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:15.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.600 7f671886b700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 0x7f66f801ac10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:15.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.631 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6714108ba0 con 0x7f67141a8c60 2021-02-10T11:42:15.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.634 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f670c000c10 con 0x7f67141a8c60 2021-02-10T11:42:15.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.763 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6714077a70 con 0x7f67141a8c60 2021-02-10T11:42:15.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.763 7f67117fa700 1 -- 172.21.2.117:0/1961250971 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) v1 ==== 80+0+4855 (secure 0 0 0) 0x7f670c006030 con 0x7f67141a8c60 2021-02-10T11:42:15.752 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:15.752 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.753 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.754 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.755 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.756 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.757 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:42:15.758 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.759 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.760 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:15.761 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.762 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:15.763 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.alice", 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCXxiNggbRSCBAA2kNC8AICLGeWqAj2g/KXLw==", 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow rw path=/volumes/group_0000000000003266/subvolume_0000000000601930", 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.764 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:15.765 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:15.766 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.766 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:15.766 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:15.766 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.765 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 msgr2=0x7f66f801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.765 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 0x7f66f801ac10 crc :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.765 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 msgr2=0x7f67141a9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.765 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f670c006f10 tx=0x7f670c00b990).stop 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 shutdown_connections 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66f8018730 0x7f66f801ac10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67141a8c60 0x7f67141a9060 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f671410a210 0x7f67141a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 --2- 172.21.2.117:0/1961250971 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f671419d920 0x7f6714109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:15.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 >> 172.21.2.117:0/1961250971 conn(0x7f671419a700 msgr2=0x7f6714199c20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:15.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 shutdown_connections 2021-02-10T11:42:15.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:15.766 7f671aa4f700 1 -- 172.21.2.117:0/1961250971 wait complete. 2021-02-10T11:42:15.768 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:42:15.768 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:42:16.113 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 Processor -- start 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 -- start start 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7feb9407da20 0x7feb940771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feb940e4da0 0x7feb941b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94077a70 con 0x7feb940e51a0 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb940777d0 con 0x7feb940e4da0 2021-02-10T11:42:16.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9c774700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9407de20 con 0x7feb9407da20 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9ad91700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7feb940e51a0 0x7feb94077ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49630/0 (socket says 172.21.2.117:49630) 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb9ad91700 1 -- 172.21.2.117:0/849692282 learned_addr learned my addr 172.21.2.117:0/849692282 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.539 7feb99d8f700 1 --2- 172.21.2.117:0/849692282 >> v2:172.21.2.117:3300/0 conn(0x7feb940e4da0 0x7feb941b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb9958e700 1 -- 172.21.2.117:0/849692282 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383959646 0 0) 0x7feb8c001500 con 0x7feb940e51a0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb99d8f700 1 -- 172.21.2.117:0/849692282 >> v2:172.21.2.125:3300/0 conn(0x7feb9407da20 msgr2=0x7feb940771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb99d8f700 1 --2- 172.21.2.117:0/849692282 >> v2:172.21.2.125:3300/0 conn(0x7feb9407da20 0x7feb940771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb99d8f700 1 -- 172.21.2.117:0/849692282 >> v1:172.21.2.117:6789/0 conn(0x7feb940e51a0 legacy=0x7feb94077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb99d8f700 1 -- 172.21.2.117:0/849692282 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb941b0710 con 0x7feb940e4da0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb99d8f700 1 --2- 172.21.2.117:0/849692282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940e4da0 0x7feb941b00f0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7feb84006650 tx=0x7feb84009a80).ready entity=mon.0 client_cookie=9a3d1b205a8de2a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb9958e700 1 -- 172.21.2.117:0/849692282 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb840009b0 con 0x7feb940e4da0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.540 7feb9958e700 1 -- 172.21.2.117:0/849692282 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8400c0c0 con 0x7feb940e4da0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9958e700 1 -- 172.21.2.117:0/849692282 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb8400c5e0 con 0x7feb940e4da0 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 -- 172.21.2.117:0/849692282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940e4da0 msgr2=0x7feb941b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 --2- 172.21.2.117:0/849692282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940e4da0 0x7feb941b00f0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0x7feb84006650 tx=0x7feb84009a80).stop 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 -- 172.21.2.117:0/849692282 shutdown_connections 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 --2- 172.21.2.117:0/849692282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940e4da0 0x7feb941b00f0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 --2- 172.21.2.117:0/849692282 >> v2:172.21.2.125:3300/0 conn(0x7feb9407da20 0x7feb940771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 -- 172.21.2.117:0/849692282 >> 172.21.2.117:0/849692282 conn(0x7feb940ce450 msgr2=0x7feb940e4290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 -- 172.21.2.117:0/849692282 shutdown_connections 2021-02-10T11:42:16.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.541 7feb9c774700 1 -- 172.21.2.117:0/849692282 wait complete. 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 Processor -- start 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 -- start start 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb941035a0 0x7feb940a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb940b3fc0 0x7feb940b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9409f710 con 0x7feb940b5b00 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb940a08d0 con 0x7feb940b3fc0 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9c774700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb940a0ab0 con 0x7feb941035a0 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9a590700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9a590700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49386/0 (socket says 172.21.2.117:49386) 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.556 7feb9a590700 1 -- 172.21.2.117:0/1994515221 learned_addr learned my addr 172.21.2.117:0/1994515221 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 -- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb941035a0 msgr2=0x7feb940a02b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:16.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb941035a0 0x7feb940a02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 -- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb940b3fc0 msgr2=0x7feb940b43c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb940b3fc0 0x7feb940b43c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 -- 172.21.2.117:0/1994515221 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb940b49e0 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9a590700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7feb90006f00 tx=0x7feb9000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb9000bd30 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb90000c90 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb900022a0 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9c774700 1 -- 172.21.2.117:0/1994515221 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb940b49e0 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.557 7feb9c774700 1 -- 172.21.2.117:0/1994515221 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb940b4f70 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.558 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb90005310 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.558 7feb8affd700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 0x7feb7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.558 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feb900326d0 con 0x7feb940b5b00 2021-02-10T11:42:16.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.559 7feb99d8f700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 0x7feb7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:16.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.560 7feb99d8f700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 0x7feb7c01abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:16.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.589 7feb9c774700 1 -- 172.21.2.117:0/1994515221 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94103370 con 0x7feb940b5b00 2021-02-10T11:42:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.592 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb90000960 con 0x7feb940b5b00 2021-02-10T11:42:16.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.737 7feb9c774700 1 -- 172.21.2.117:0/1994515221 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7feb94077a70 con 0x7feb940b5b00 2021-02-10T11:42:16.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.741 7feb8affd700 1 -- 172.21.2.117:0/1994515221 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v9)=0 v9) v1 ==== 120+0+0 (secure 0 0 0) 0x7feb90005310 con 0x7feb940b5b00 2021-02-10T11:42:16.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 msgr2=0x7feb7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 0x7feb7c01abc0 crc :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 msgr2=0x7feb9409fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7feb90006f00 tx=0x7feb9000b940).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 shutdown_connections 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb940b3fc0 0x7feb940b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb941035a0 0x7feb940a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb7c0186c0 0x7feb7c01abc0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 --2- 172.21.2.117:0/1994515221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb940b5b00 0x7feb9409fc90 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 >> 172.21.2.117:0/1994515221 conn(0x7feb940bb0d0 msgr2=0x7feb94190400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 shutdown_connections 2021-02-10T11:42:16.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:16.746 7feb9c774700 1 -- 172.21.2.117:0/1994515221 wait complete. 2021-02-10T11:42:16.743 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.167 7f57606df700 1 Processor -- start 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 -- start start 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f575807da20 0x7f57580771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57580e5180 0x7f5758079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5758077a70 con 0x7f57580e4d80 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57580777d0 con 0x7f575807da20 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f57606df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57581b08c0 con 0x7f57580e5180 2021-02-10T11:42:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f575e4fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f575807da20 0x7f57580771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f575e4fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f575807da20 0x7f57580771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49394/0 (socket says 172.21.2.117:49394) 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.168 7f575e4fb700 1 -- 172.21.2.117:0/3949643025 learned_addr learned my addr 172.21.2.117:0/3949643025 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575e4fb700 1 -- 172.21.2.117:0/3949643025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57580e5180 msgr2=0x7f5758079000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575e4fb700 1 --2- 172.21.2.117:0/3949643025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57580e5180 0x7f5758079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575e4fb700 1 -- 172.21.2.117:0/3949643025 >> v1:172.21.2.125:6789/0 conn(0x7f57580e4d80 legacy=0x7f57581b00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575e4fb700 1 -- 172.21.2.117:0/3949643025 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575807de20 con 0x7f575807da20 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575e4fb700 1 --2- 172.21.2.117:0/3949643025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f575807da20 0x7f57580771b0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f574c006700 tx=0x7f574c009b80).ready entity=mon.0 client_cookie=12687526c62526cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575d4f9700 1 -- 172.21.2.117:0/3949643025 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c00c0c0 con 0x7f575807da20 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575d4f9700 1 -- 172.21.2.117:0/3949643025 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c00c340 con 0x7f575807da20 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f575d4f9700 1 -- 172.21.2.117:0/3949643025 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c00c860 con 0x7f575807da20 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f57606df700 1 -- 172.21.2.117:0/3949643025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f575807da20 msgr2=0x7f57580771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.169 7f57606df700 1 --2- 172.21.2.117:0/3949643025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f575807da20 0x7f57580771b0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7f574c006700 tx=0x7f574c009b80).stop 2021-02-10T11:42:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 -- 172.21.2.117:0/3949643025 shutdown_connections 2021-02-10T11:42:17.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 --2- 172.21.2.117:0/3949643025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57580e5180 0x7f5758079000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 --2- 172.21.2.117:0/3949643025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f575807da20 0x7f57580771b0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 -- 172.21.2.117:0/3949643025 >> 172.21.2.117:0/3949643025 conn(0x7f57580ce430 msgr2=0x7f57580e4270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:17.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 -- 172.21.2.117:0/3949643025 shutdown_connections 2021-02-10T11:42:17.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.170 7f57606df700 1 -- 172.21.2.117:0/3949643025 wait complete. 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 Processor -- start 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 -- start start 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57580a74a0 0x7f57580a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57581a4180 0x7f575809bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57580a3e20 con 0x7f57580a3ff0 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57580a6f20 con 0x7f57580a74a0 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f57606df700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57581909c0 con 0x7f57581a4180 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f575e4fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f575e4fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49398/0 (socket says 172.21.2.117:49398) 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.185 7f575e4fb700 1 -- 172.21.2.117:0/3088230506 learned_addr learned my addr 172.21.2.117:0/3088230506 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 -- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57581a4180 msgr2=0x7f575809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57581a4180 0x7f575809bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 -- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57580a74a0 msgr2=0x7f57580a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57580a74a0 0x7f57580a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 -- 172.21.2.117:0/3088230506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57581a4580 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f575e4fb700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f574c00b650 tx=0x7f574c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c00bde0 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c006520 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f574c0022b0 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f57606df700 1 -- 172.21.2.117:0/3088230506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57581a4580 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.186 7f57606df700 1 -- 172.21.2.117:0/3088230506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57581a4580 con 0x7f57580a3ff0 2021-02-10T11:42:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.187 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f574c000c90 con 0x7f57580a3ff0 2021-02-10T11:42:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.187 7f574affd700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 0x7f574001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.187 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f574c032810 con 0x7f57580a3ff0 2021-02-10T11:42:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.187 7f575dcfa700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 0x7f574001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.188 7f575dcfa700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 0x7f574001abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.218 7f57606df700 1 -- 172.21.2.117:0/3088230506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57581909c0 con 0x7f57580a3ff0 2021-02-10T11:42:17.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.220 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f574c002000 con 0x7f57580a3ff0 2021-02-10T11:42:17.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.367 7f57606df700 1 -- 172.21.2.117:0/3088230506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7f57581909c0 con 0x7f57580a3ff0 2021-02-10T11:42:17.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.371 7f574affd700 1 -- 172.21.2.117:0/3088230506 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f574c000c90 con 0x7f57580a3ff0 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 -- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 msgr2=0x7f574001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 0x7f574001abc0 crc :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 -- 172.21.2.117:0/3088230506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 msgr2=0x7f57580a6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f574c00b650 tx=0x7f574c00b9f0).stop 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 -- 172.21.2.117:0/3088230506 shutdown_connections 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57581a4180 0x7f575809bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57580a74a0 0x7f57580a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:17.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57400186c0 0x7f574001abc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 --2- 172.21.2.117:0/3088230506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57580a3ff0 0x7f57580a6730 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:17.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.376 7f57606df700 1 -- 172.21.2.117:0/3088230506 >> 172.21.2.117:0/3088230506 conn(0x7f57580bb240 msgr2=0x7f57580a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:17.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.377 7f57606df700 1 -- 172.21.2.117:0/3088230506 shutdown_connections 2021-02-10T11:42:17.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:17.377 7f57606df700 1 -- 172.21.2.117:0/3088230506 wait complete. 2021-02-10T11:42:17.374 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:42:17.377 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout:[global] 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout: chdir = "" 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout: auth supported = cephx 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.737 INFO:teuthology.orchestra.run.gibba017.stdout: filestore xattr use omap = true 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: mon clock drift allowed = 1.000 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: osd crush chooseleaf type = 0 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: auth debug = true 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on old message = true 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on bug = true 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: mon pg warn max object skew = 0 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.738 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default size = 2 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow primary affinity = true 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow pg remap = true 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on no sortbitwise = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on too few osds = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.739 INFO:teuthology.orchestra.run.gibba017.stdout: osd default data pool replay window = 5 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: mon allow pool delete = true 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: mon cluster log file level = debug 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: debug asserts on shutdown = true 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: mon health detail to clog = false 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout:[osd] 2021-02-10T11:42:17.740 INFO:teuthology.orchestra.run.gibba017.stdout: osd journal size = 100 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub load threshold = 5.0 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub max interval = 600 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd recover clone overlap = true 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd recovery max chunk = 1048576 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug shutdown = true 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug op order = true 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug verify stray on activate = true 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd open classes on start = true 2021-02-10T11:42:17.741 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug pg log writeout = true 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: osd map max advance = 10 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: journal zero on create = true 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: filestore apply finisher threads = 3 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: bdev debug aio = true 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug misdirected ops = true 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore block size = 96636764160 2021-02-10T11:42:17.742 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore fsck on mount = True 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluefs = 20 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluestore = 20 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug filestore = 20 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug journal = 20 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 20 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug osd = 25 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: debug rocksdb = 10 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd full ratio = 0.9 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: osd objectstore = bluestore 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:42:17.743 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr] 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug mgr = 20 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 10 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout:[mon] 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug paxos = 20 2021-02-10T11:42:17.744 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon data avail warn = 5 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon mgr mkfs grace = 240 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd reporter subtree level = osd 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd prime pg temp = true 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: mon op complaint time = 120 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout:[client] 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: rgw cache enabled = true 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable ops log = true 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable usage log = true 2021-02-10T11:42:17.745 INFO:teuthology.orchestra.run.gibba017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout: client mount timeout = 600 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 20 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = "" 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.a] 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.b] 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.c] 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:42:17.746 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout:[mds] 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: debug mds = 20 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal frag = True 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal fragment size max = 10000 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal merge size = 5 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split bits = 3 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split size = 100 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug frag = True 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug scatterstat = True 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds op complaint time = 180 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: mds verify scatter = True 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:42:17.747 INFO:teuthology.orchestra.run.gibba017.stdout:[client.alice] 2021-02-10T11:42:17.748 INFO:teuthology.orchestra.run.gibba017.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:42:17.748 INFO:teuthology.orchestra.run.gibba025.stdout:[global] 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: chdir = "" 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: auth supported = cephx 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: filestore xattr use omap = true 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: mon clock drift allowed = 1.000 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: osd crush chooseleaf type = 0 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: auth debug = true 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on old message = true 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on bug = true 2021-02-10T11:42:17.749 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon pg warn max object skew = 0 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default size = 2 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow primary affinity = true 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow pg remap = true 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on no sortbitwise = false 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on too few osds = false 2021-02-10T11:42:17.750 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: osd default data pool replay window = 5 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: mon allow pool delete = true 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: mon cluster log file level = debug 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: debug asserts on shutdown = true 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: mon health detail to clog = false 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:42:17.751 INFO:teuthology.orchestra.run.gibba025.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout:[osd] 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd journal size = 100 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub load threshold = 5.0 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub max interval = 600 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd recover clone overlap = true 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd recovery max chunk = 1048576 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug shutdown = true 2021-02-10T11:42:17.752 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug op order = true 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug verify stray on activate = true 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: osd open classes on start = true 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug pg log writeout = true 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: osd map max advance = 10 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: journal zero on create = true 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: filestore apply finisher threads = 3 2021-02-10T11:42:17.753 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: bdev debug aio = true 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug misdirected ops = true 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore block size = 96636764160 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore fsck on mount = True 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluefs = 20 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluestore = 20 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug filestore = 20 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug journal = 20 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 20 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug osd = 25 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: debug rocksdb = 10 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd full ratio = 0.9 2021-02-10T11:42:17.754 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: osd objectstore = bluestore 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr] 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: debug mgr = 20 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 10 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.755 INFO:teuthology.orchestra.run.gibba025.stdout:[mon] 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: debug paxos = 20 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon data avail warn = 5 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon mgr mkfs grace = 240 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd reporter subtree level = osd 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd prime pg temp = true 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:17.756 INFO:teuthology.orchestra.run.gibba025.stdout: mon op complaint time = 120 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout:[client] 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: rgw cache enabled = true 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable ops log = true 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable usage log = true 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: client mount timeout = 600 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 20 2021-02-10T11:42:17.757 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = "" 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.a] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.b] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.c] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout:[mds] 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout: debug mds = 20 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:17.758 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal frag = True 2021-02-10T11:42:17.759 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal fragment size max = 10000 2021-02-10T11:42:17.759 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal merge size = 5 2021-02-10T11:42:17.759 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split bits = 3 2021-02-10T11:42:17.759 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split size = 100 2021-02-10T11:42:17.759 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug frag = True 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug scatterstat = True 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout: mds op complaint time = 180 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout: mds verify scatter = True 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout:[client.alice] 2021-02-10T11:42:17.760 INFO:teuthology.orchestra.run.gibba025.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:42:17.762 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:18.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.183 7fabc7174700 1 Processor -- start 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.183 7fabc7174700 1 -- start start 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc7174700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabc00e4f50 0x7fabc01b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc7174700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fabc01b3d20 0x7fabc00f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc7174700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00f53c0 con 0x7fabc00e5350 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc7174700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00f6950 con 0x7fabc00e4f50 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc7174700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00f6b70 con 0x7fabc01b3d20 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc5791700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fabc00e5350 0x7fabc00f4330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49658/0 (socket says 172.21.2.117:49658) 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc5791700 1 -- 172.21.2.117:0/3119695754 learned_addr learned my addr 172.21.2.117:0/3119695754 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc4f90700 1 --2- 172.21.2.117:0/3119695754 >> v2:172.21.2.117:3300/0 conn(0x7fabc00e4f50 0x7fabc01b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc4f90700 1 -- 172.21.2.117:0/3119695754 >> v2:172.21.2.125:3300/0 conn(0x7fabc01b3d20 msgr2=0x7fabc00f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc4f90700 1 --2- 172.21.2.117:0/3119695754 >> v2:172.21.2.125:3300/0 conn(0x7fabc01b3d20 0x7fabc00f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc4f90700 1 -- 172.21.2.117:0/3119695754 >> v1:172.21.2.117:6789/0 conn(0x7fabc00e5350 legacy=0x7fabc00f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.184 7fabc4f90700 1 -- 172.21.2.117:0/3119695754 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00f6d90 con 0x7fabc00e4f50 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabc4f90700 1 --2- 172.21.2.117:0/3119695754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc00e4f50 0x7fabc01b3700 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7faba8006650 tx=0x7faba8009a00).ready entity=mon.0 client_cookie=4ed5d3a3351ced94 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabbf7fe700 1 -- 172.21.2.117:0/3119695754 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba8009df0 con 0x7fabc00e4f50 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabbf7fe700 1 -- 172.21.2.117:0/3119695754 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba80009b0 con 0x7fabc00e4f50 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabbf7fe700 1 -- 172.21.2.117:0/3119695754 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba800c2b0 con 0x7fabc00e4f50 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabc7174700 1 -- 172.21.2.117:0/3119695754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc00e4f50 msgr2=0x7fabc01b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.185 7fabc7174700 1 --2- 172.21.2.117:0/3119695754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc00e4f50 0x7fabc01b3700 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7faba8006650 tx=0x7faba8009a00).stop 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 -- 172.21.2.117:0/3119695754 shutdown_connections 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 --2- 172.21.2.117:0/3119695754 >> v2:172.21.2.125:3300/0 conn(0x7fabc01b3d20 0x7fabc00f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 --2- 172.21.2.117:0/3119695754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc00e4f50 0x7fabc01b3700 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 -- 172.21.2.117:0/3119695754 >> 172.21.2.117:0/3119695754 conn(0x7fabc00ce5b0 msgr2=0x7fabc00e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 -- 172.21.2.117:0/3119695754 shutdown_connections 2021-02-10T11:42:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.186 7fabc7174700 1 -- 172.21.2.117:0/3119695754 wait complete. 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.200 7fabc7174700 1 Processor -- start 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.200 7fabc7174700 1 -- start start 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc019fd30 0x7fabc009c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc009c9d0 0x7fabc018dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc018d620 con 0x7fabc018d220 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc009cdd0 con 0x7fabc009c9d0 2021-02-10T11:42:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc7174700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc009cff0 con 0x7fabc019fd30 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49410/0 (socket says 172.21.2.117:49410) 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 -- 172.21.2.117:0/2329657162 learned_addr learned my addr 172.21.2.117:0/2329657162 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 -- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc019fd30 msgr2=0x7fabc009c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc019fd30 0x7fabc009c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 -- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc009c9d0 msgr2=0x7fabc018dec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc009c9d0 0x7fabc018dec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 -- 172.21.2.117:0/2329657162 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc018e4e0 con 0x7fabc018d220 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.201 7fabc4f90700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7faba80096a0 tx=0x7faba800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba80093a0 con 0x7fabc018d220 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba80067f0 con 0x7fabc018d220 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabc7174700 1 -- 172.21.2.117:0/2329657162 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc018e4e0 con 0x7fabc018d220 2021-02-10T11:42:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabc7174700 1 -- 172.21.2.117:0/2329657162 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc018e4e0 con 0x7fabc018d220 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faba8002220 con 0x7fabc018d220 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faba80052e0 con 0x7fabc018d220 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 0x7faba401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.202 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faba8032930 con 0x7fabc018d220 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.203 7fabbffff700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 0x7faba401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.203 7fabbffff700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 0x7faba401ac10 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.233 7fabc7174700 1 -- 172.21.2.117:0/2329657162 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc018d620 con 0x7fabc018d220 2021-02-10T11:42:18.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.236 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faba8000c90 con 0x7fabc018d220 2021-02-10T11:42:18.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.359 7fabc7174700 1 -- 172.21.2.117:0/2329657162 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabc018d620 con 0x7fabc018d220 2021-02-10T11:42:18.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.360 7fabbd7fa700 1 -- 172.21.2.117:0/2329657162 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7faba80052e0 con 0x7fabc018d220 2021-02-10T11:42:18.349 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:18.349 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:18.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.361 7fabc7174700 1 -- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 msgr2=0x7faba401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.361 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 0x7faba401ac10 crc :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.361 7fabc7174700 1 -- 172.21.2.117:0/2329657162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 msgr2=0x7fabc019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.361 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7faba80096a0 tx=0x7faba800b9b0).stop 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 -- 172.21.2.117:0/2329657162 shutdown_connections 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc009c9d0 0x7fabc018dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc019fd30 0x7fabc009c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba4018730 0x7faba401ac10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 --2- 172.21.2.117:0/2329657162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc018d220 0x7fabc019f710 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 -- 172.21.2.117:0/2329657162 >> 172.21.2.117:0/2329657162 conn(0x7fabc01075a0 msgr2=0x7fabc008a880 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 -- 172.21.2.117:0/2329657162 shutdown_connections 2021-02-10T11:42:18.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.362 7fabc7174700 1 -- 172.21.2.117:0/2329657162 wait complete. 2021-02-10T11:42:18.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:44.610269', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:45.683464', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:42:18.359 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.784 7f2846212700 1 Processor -- start 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.784 7f2846212700 1 -- start start 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f2846212700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f28400e5360 0x7f28400f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f2846212700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28400f5950 0x7f28400f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f2846212700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400f53d0 con 0x7f28400e4f60 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f2846212700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28401b3f00 con 0x7f28400f5950 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f2846212700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28400d16a0 con 0x7f28400e5360 2021-02-10T11:42:18.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28400f5950 0x7f28400f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28400f5950 0x7f28400f5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49418/0 (socket says 172.21.2.117:49418) 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 -- 172.21.2.117:0/3549923720 learned_addr learned my addr 172.21.2.117:0/3549923720 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 -- 172.21.2.117:0/3549923720 >> v2:172.21.2.125:3300/0 conn(0x7f28400e5360 msgr2=0x7f28400f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 --2- 172.21.2.117:0/3549923720 >> v2:172.21.2.125:3300/0 conn(0x7f28400e5360 0x7f28400f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 -- 172.21.2.117:0/3549923720 >> v1:172.21.2.125:6789/0 conn(0x7f28400e4f60 legacy=0x7f28401b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.785 7f284482f700 1 -- 172.21.2.117:0/3549923720 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28400d18c0 con 0x7f28400f5950 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f284482f700 1 --2- 172.21.2.117:0/3549923720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28400f5950 0x7f28400f5d50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f2838006f00 tx=0x7f2838009f10).ready entity=mon.0 client_cookie=a9e11a3f4b828ea7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f283effd700 1 -- 172.21.2.117:0/3549923720 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283800b3b0 con 0x7f28400f5950 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f283effd700 1 -- 172.21.2.117:0/3549923720 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283800b730 con 0x7f28400f5950 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f283effd700 1 -- 172.21.2.117:0/3549923720 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283800bc50 con 0x7f28400f5950 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 -- 172.21.2.117:0/3549923720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28400f5950 msgr2=0x7f28400f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 --2- 172.21.2.117:0/3549923720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28400f5950 0x7f28400f5d50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f2838006f00 tx=0x7f2838009f10).stop 2021-02-10T11:42:18.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 -- 172.21.2.117:0/3549923720 shutdown_connections 2021-02-10T11:42:18.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 --2- 172.21.2.117:0/3549923720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28400f5950 0x7f28400f5d50 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 --2- 172.21.2.117:0/3549923720 >> v2:172.21.2.125:3300/0 conn(0x7f28400e5360 0x7f28400f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 -- 172.21.2.117:0/3549923720 >> 172.21.2.117:0/3549923720 conn(0x7f28400ce5c0 msgr2=0x7f28400e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 -- 172.21.2.117:0/3549923720 shutdown_connections 2021-02-10T11:42:18.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.786 7f2846212700 1 -- 172.21.2.117:0/3549923720 wait complete. 2021-02-10T11:42:18.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 Processor -- start 2021-02-10T11:42:18.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 -- start start 2021-02-10T11:42:18.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f284008a330 0x7f284008a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f284008b150 0x7f28400c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284008b6e0 con 0x7f284008ad50 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28400bff10 con 0x7f284008b150 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f2846212700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284019bdc0 con 0x7f284008a330 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f283f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f283f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49426/0 (socket says 172.21.2.117:49426) 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.801 7f283f7fe700 1 -- 172.21.2.117:0/2520175876 learned_addr learned my addr 172.21.2.117:0/2520175876 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 -- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f284008a330 msgr2=0x7f284008a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f284008a330 0x7f284008a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 -- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f284008b150 msgr2=0x7f28400c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f284008b150 0x7f28400c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 -- 172.21.2.117:0/2520175876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284008d7f0 con 0x7f284008ad50 2021-02-10T11:42:18.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283f7fe700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7f2834006f00 tx=0x7f283400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2834006350 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2834006b90 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2834000c90 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f2846212700 1 -- 172.21.2.117:0/2520175876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284008d7f0 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.802 7f2846212700 1 -- 172.21.2.117:0/2520175876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28400c2ed0 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.803 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2834007000 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.803 7f283cff9700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 0x7f282001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.803 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28340326e0 con 0x7f284008ad50 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.803 7f283ffff700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 0x7f282001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:18.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.804 7f283ffff700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 0x7f282001abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:18.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.835 7f2846212700 1 -- 172.21.2.117:0/2520175876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28400bff10 con 0x7f284008ad50 2021-02-10T11:42:18.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.838 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2834025fc0 con 0x7f284008ad50 2021-02-10T11:42:18.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.966 7f2846212700 1 -- 172.21.2.117:0/2520175876 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f28400bff10 con 0x7f284008ad50 2021-02-10T11:42:18.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.966 7f283cff9700 1 -- 172.21.2.117:0/2520175876 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2834025fc0 con 0x7f284008ad50 2021-02-10T11:42:18.956 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 -- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 msgr2=0x7f282001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 0x7f282001abc0 crc :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 -- 172.21.2.117:0/2520175876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 msgr2=0x7f28400c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7f2834006f00 tx=0x7f283400b940).stop 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 -- 172.21.2.117:0/2520175876 shutdown_connections 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28200186c0 0x7f282001abc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f284008ad50 0x7f28400c1fa0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:18.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f284008b150 0x7f28400c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.968 7f2846212700 1 --2- 172.21.2.117:0/2520175876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f284008a330 0x7f284008a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:18.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.969 7f2846212700 1 -- 172.21.2.117:0/2520175876 >> 172.21.2.117:0/2520175876 conn(0x7f2840103170 msgr2=0x7f2840107170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:18.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.969 7f2846212700 1 -- 172.21.2.117:0/2520175876 shutdown_connections 2021-02-10T11:42:18.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:18.969 7f2846212700 1 -- 172.21.2.117:0/2520175876 wait complete. 2021-02-10T11:42:18.966 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:42:19.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.391 7f26011a4700 1 Processor -- start 2021-02-10T11:42:19.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.391 7f26011a4700 1 -- start start 2021-02-10T11:42:19.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25fc0e4f60 0x7f25fc1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f25fc1b3d30 0x7f25fc0f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc0e5360 0x7f25fc0f4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc0f53d0 con 0x7f25fc0e4f60 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc0f4960 con 0x7f25fc1b3d30 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f26011a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0f4b80 con 0x7f25fc0e5360 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25fc0e4f60 0x7f25fc1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f25fc0e4f60 0x7f25fc1b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49430/0 (socket says 172.21.2.117:49430) 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 -- 172.21.2.117:0/1852122011 learned_addr learned my addr 172.21.2.117:0/1852122011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fa59c700 1 --2- 172.21.2.117:0/1852122011 >> v2:172.21.2.125:3300/0 conn(0x7f25fc1b3d30 0x7f25fc0f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fb59e700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc0e5360 0x7f25fc0f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 -- 172.21.2.117:0/1852122011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc0e5360 msgr2=0x7f25fc0f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc0e5360 0x7f25fc0f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 -- 172.21.2.117:0/1852122011 >> v2:172.21.2.125:3300/0 conn(0x7f25fc1b3d30 msgr2=0x7f25fc0f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 --2- 172.21.2.117:0/1852122011 >> v2:172.21.2.125:3300/0 conn(0x7f25fc1b3d30 0x7f25fc0f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 -- 172.21.2.117:0/1852122011 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0f4da0 con 0x7f25fc0e4f60 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.392 7f25fad9d700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc0e4f60 0x7f25fc1b3710 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f25e4006700 tx=0x7f25e4009b80).ready entity=mon.0 client_cookie=9170bee4bc48bded server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f25f9d9b700 1 -- 172.21.2.117:0/1852122011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25e400c0c0 con 0x7f25fc0e4f60 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f25f9d9b700 1 -- 172.21.2.117:0/1852122011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e400c0c0 con 0x7f25fc0e4f60 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f25f9d9b700 1 -- 172.21.2.117:0/1852122011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25e400c340 con 0x7f25fc0e4f60 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 -- 172.21.2.117:0/1852122011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc0e4f60 msgr2=0x7f25fc1b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc0e4f60 0x7f25fc1b3710 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7f25e4006700 tx=0x7f25e4009b80).stop 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 -- 172.21.2.117:0/1852122011 shutdown_connections 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 --2- 172.21.2.117:0/1852122011 >> v2:172.21.2.125:3300/0 conn(0x7f25fc1b3d30 0x7f25fc0f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc0e5360 0x7f25fc0f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 --2- 172.21.2.117:0/1852122011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc0e4f60 0x7f25fc1b3710 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 -- 172.21.2.117:0/1852122011 >> 172.21.2.117:0/1852122011 conn(0x7f25fc0ce5c0 msgr2=0x7f25fc0e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 -- 172.21.2.117:0/1852122011 shutdown_connections 2021-02-10T11:42:19.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.393 7f26011a4700 1 -- 172.21.2.117:0/1852122011 wait complete. 2021-02-10T11:42:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 Processor -- start 2021-02-10T11:42:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 -- start start 2021-02-10T11:42:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 0x7f25fc1a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 0x7f25fc106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25fc0941d0 0x7f25fc1070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc096010 con 0x7f25fc095b60 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc107700 con 0x7f25fc0941d0 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.408 7f26011a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0b1340 con 0x7f25fc1a3430 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fa59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 0x7f25fc106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fa59c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 0x7f25fc106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47116/0 (socket says 172.21.2.117:47116) 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fa59c700 1 -- 172.21.2.117:0/1949119638 learned_addr learned my addr 172.21.2.117:0/1949119638 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 0x7f25fc1a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fb59e700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25fc0941d0 0x7f25fc1070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 -- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 msgr2=0x7f25fc106ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 0x7f25fc106ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 -- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25fc0941d0 msgr2=0x7f25fc1070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25fc0941d0 0x7f25fc1070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 -- 172.21.2.117:0/1949119638 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc096590 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25fad9d700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 0x7f25fc1a2e10 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f25e40096d0 tx=0x7f25e400b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25e400bcb0 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e4002030 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25e4002480 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f26011a4700 1 -- 172.21.2.117:0/1949119638 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc096590 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.409 7f26011a4700 1 -- 172.21.2.117:0/1949119638 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc096590 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.410 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f25e4002210 con 0x7f25fc095b60 2021-02-10T11:42:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.410 7f25f37fe700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 0x7f25e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:19.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.410 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f25e4032850 con 0x7f25fc095b60 2021-02-10T11:42:19.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.410 7f25fa59c700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 0x7f25e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:19.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.410 7f25fa59c700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 0x7f25e001abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:19.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.442 7f26011a4700 1 -- 172.21.2.117:0/1949119638 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc102340 con 0x7f25fc095b60 2021-02-10T11:42:19.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.444 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f25e400bcb0 con 0x7f25fc095b60 2021-02-10T11:42:19.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.571 7f26011a4700 1 -- 172.21.2.117:0/1949119638 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f25fc0f53d0 con 0x7f25fc095b60 2021-02-10T11:42:19.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.572 7f25f37fe700 1 -- 172.21.2.117:0/1949119638 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v56) v1 ==== 104+0+10 (secure 0 0 0) 0x7f25e4002030 con 0x7f25fc095b60 2021-02-10T11:42:19.561 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:19.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.573 7f26011a4700 1 -- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 msgr2=0x7f25e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.573 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 0x7f25e001abc0 crc :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.573 7f26011a4700 1 -- 172.21.2.117:0/1949119638 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 msgr2=0x7f25fc1a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:19.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 0x7f25fc1a2e10 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f25e40096d0 tx=0x7f25e400b8c0).stop 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 -- 172.21.2.117:0/1949119638 shutdown_connections 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f25fc0941d0 0x7f25fc1070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f25fc1a3430 0x7f25fc106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f25e00186c0 0x7f25e001abc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 --2- 172.21.2.117:0/1949119638 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f25fc095b60 0x7f25fc1a2e10 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 -- 172.21.2.117:0/1949119638 >> 172.21.2.117:0/1949119638 conn(0x7f25fc09ad60 msgr2=0x7f25fc102570 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 -- 172.21.2.117:0/1949119638 shutdown_connections 2021-02-10T11:42:19.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.574 7f26011a4700 1 -- 172.21.2.117:0/1949119638 wait complete. 2021-02-10T11:42:19.571 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:42:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 Processor -- start 2021-02-10T11:42:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 -- start start 2021-02-10T11:42:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 0x7f69241b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69241b3d30 0x7f69240f5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69240f53d0 con 0x7f69240e5360 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69240e5d90 con 0x7f69241b3d30 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.996 7f692b146700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69240f4d30 con 0x7f69240e4f60 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69241b3d30 0x7f69240f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69241b3d30 0x7f69240f5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49444/0 (socket says 172.21.2.117:49444) 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 -- 172.21.2.117:0/3569291016 learned_addr learned my addr 172.21.2.117:0/3569291016 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6928f62700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 0x7f69241b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 -- 172.21.2.117:0/3569291016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 msgr2=0x7f69241b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 0x7f69241b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 -- 172.21.2.117:0/3569291016 >> v1:172.21.2.125:6789/0 conn(0x7f69240e5360 legacy=0x7f69240f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 -- 172.21.2.117:0/3569291016 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69240f4f20 con 0x7f69241b3d30 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6928f62700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 0x7f69241b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f6923fff700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69241b3d30 0x7f69240f5950 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f6914006700 tx=0x7f6914009b80).ready entity=mon.0 client_cookie=4027902aca36b430 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f69237fe700 1 -- 172.21.2.117:0/3569291016 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691400c0c0 con 0x7f69241b3d30 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f69237fe700 1 -- 172.21.2.117:0/3569291016 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691400c400 con 0x7f69241b3d30 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f69237fe700 1 -- 172.21.2.117:0/3569291016 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691400c8e0 con 0x7f69241b3d30 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f692b146700 1 -- 172.21.2.117:0/3569291016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69241b3d30 msgr2=0x7f69240f5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.997 7f692b146700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69241b3d30 0x7f69240f5950 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f6914006700 tx=0x7f6914009b80).stop 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 -- 172.21.2.117:0/3569291016 shutdown_connections 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69241b3d30 0x7f69240f5950 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 --2- 172.21.2.117:0/3569291016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69240e4f60 0x7f69241b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 -- 172.21.2.117:0/3569291016 >> 172.21.2.117:0/3569291016 conn(0x7f69240ce5c0 msgr2=0x7f69240e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 -- 172.21.2.117:0/3569291016 shutdown_connections 2021-02-10T11:42:20.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:19.998 7f692b146700 1 -- 172.21.2.117:0/3569291016 wait complete. 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 Processor -- start 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 -- start start 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69241046b0 0x7f69240c3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6924096ed0 0x7f69240972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6924104c20 con 0x7f69240978f0 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69240c3f40 con 0x7f69241046b0 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f692b146700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69240b08e0 con 0x7f6924096ed0 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.013 7f6929763700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49454/0 (socket says 172.21.2.117:49454) 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 -- 172.21.2.117:0/3524679637 learned_addr learned my addr 172.21.2.117:0/3524679637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:20.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 -- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6924096ed0 msgr2=0x7f69240972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6924096ed0 0x7f69240972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 -- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69241046b0 msgr2=0x7f69240c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69241046b0 0x7f69240c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 -- 172.21.2.117:0/3524679637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924097f20 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f6929763700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f691c009940 tx=0x7f691c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691c000f90 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c000d30 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f692b146700 1 -- 172.21.2.117:0/3524679637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6924097f20 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f692b146700 1 -- 172.21.2.117:0/3524679637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69240b1770 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.014 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f691c00c570 con 0x7f69240978f0 2021-02-10T11:42:20.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.015 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f691c001220 con 0x7f69240978f0 2021-02-10T11:42:20.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.015 7f69217fa700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 0x7f690801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.015 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f691c032c80 con 0x7f69240978f0 2021-02-10T11:42:20.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.015 7f6928f62700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 0x7f690801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.016 7f6928f62700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 0x7f690801abc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.046 7f692b146700 1 -- 172.21.2.117:0/3524679637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692407dea0 con 0x7f69240978f0 2021-02-10T11:42:20.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.049 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f691c000f40 con 0x7f69240978f0 2021-02-10T11:42:20.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.175 7f692b146700 1 -- 172.21.2.117:0/3524679637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f692407dea0 con 0x7f69240978f0 2021-02-10T11:42:20.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.175 7f69217fa700 1 -- 172.21.2.117:0/3524679637 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v56) v1 ==== 100+0+10 (secure 0 0 0) 0x7f691c000f40 con 0x7f69240978f0 2021-02-10T11:42:20.165 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:20.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 -- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 msgr2=0x7f690801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 0x7f690801abc0 crc :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 -- 172.21.2.117:0/3524679637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 msgr2=0x7f69240b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f691c009940 tx=0x7f691c00bcf0).stop 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 -- 172.21.2.117:0/3524679637 shutdown_connections 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69080186c0 0x7f690801abc0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69240978f0 0x7f69240b0e60 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6924096ed0 0x7f69240972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:20.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 --2- 172.21.2.117:0/3524679637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69241046b0 0x7f69240c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:20.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.177 7f692b146700 1 -- 172.21.2.117:0/3524679637 >> 172.21.2.117:0/3524679637 conn(0x7f69240b3780 msgr2=0x7f692407e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.178 7f692b146700 1 -- 172.21.2.117:0/3524679637 shutdown_connections 2021-02-10T11:42:20.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.178 7f692b146700 1 -- 172.21.2.117:0/3524679637 wait complete. 2021-02-10T11:42:20.175 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:42:20.175 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:20.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 Processor -- start 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 -- start start 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60640e5350 0x7f60640f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f60640f5940 0x7f60640f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60640f53c0 con 0x7f60640e4f50 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60641b3f60 con 0x7f60640f5940 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.598 7f606c2ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60640d1690 con 0x7f60640e5350 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.599 7f606a108700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f60640e4f50 0x7f60641b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42634/0 (socket says 172.21.2.117:42634) 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.599 7f606a909700 1 --2- 172.21.2.117:0/4132325251 >> v2:172.21.2.125:3300/0 conn(0x7f60640f5940 0x7f60640f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.599 7f6069907700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60640e5350 0x7f60640f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.599 7f606a108700 1 -- 172.21.2.117:0/4132325251 learned_addr learned my addr 172.21.2.117:0/4132325251 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:20.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606a909700 1 -- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60640e5350 msgr2=0x7f60640f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606a909700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60640e5350 0x7f60640f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606a909700 1 -- 172.21.2.117:0/4132325251 >> v1:172.21.2.125:6789/0 conn(0x7f60640e4f50 legacy=0x7f60641b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606a909700 1 -- 172.21.2.117:0/4132325251 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640d18b0 con 0x7f60640f5940 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606a909700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60640f5940 0x7f60640f5d40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f605c006e50 tx=0x7f605c009e70).ready entity=mon.1 client_cookie=3ca263731fd59b62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f6069106700 1 -- 172.21.2.117:0/4132325251 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f605c00b510 con 0x7f60640f5940 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f6069106700 1 -- 172.21.2.117:0/4132325251 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c006790 con 0x7f60640f5940 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f6069106700 1 -- 172.21.2.117:0/4132325251 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f605c00b940 con 0x7f60640f5940 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606c2ec700 1 -- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60640f5940 msgr2=0x7f60640f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.600 7f606c2ec700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60640f5940 0x7f60640f5d40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f605c006e50 tx=0x7f605c009e70).stop 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 -- 172.21.2.117:0/4132325251 shutdown_connections 2021-02-10T11:42:20.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60640f5940 0x7f60640f5d40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 --2- 172.21.2.117:0/4132325251 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60640e5350 0x7f60640f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 -- 172.21.2.117:0/4132325251 >> 172.21.2.117:0/4132325251 conn(0x7f60640ce5b0 msgr2=0x7f60640e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 -- 172.21.2.117:0/4132325251 shutdown_connections 2021-02-10T11:42:20.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.602 7f606c2ec700 1 -- 172.21.2.117:0/4132325251 wait complete. 2021-02-10T11:42:20.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.615 7f606c2ec700 1 Processor -- start 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.615 7f606c2ec700 1 -- start start 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6064105ef0 0x7f606419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f606419feb0 0x7f60641ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641062f0 con 0x7f606419fab0 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60641aed10 con 0x7f6064105ef0 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606c2ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60641aef30 con 0x7f606419feb0 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49466/0 (socket says 172.21.2.117:49466) 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 -- 172.21.2.117:0/3595113690 learned_addr learned my addr 172.21.2.117:0/3595113690 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606a108700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6064105ef0 0x7f606419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f606a909700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f606419feb0 0x7f60641ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 -- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f606419feb0 msgr2=0x7f60641ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f606419feb0 0x7f60641ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 -- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6064105ef0 msgr2=0x7f606419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6064105ef0 0x7f606419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.616 7f6069907700 1 -- 172.21.2.117:0/3595113690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60641acc20 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f6069907700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f6060006f60 tx=0x7f606000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f606000bd60 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6060000c90 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60641acc20 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60641ad1b0 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60600022b0 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6060006b00 con 0x7f606419fab0 2021-02-10T11:42:20.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 0x7f604c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:20.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6060032900 con 0x7f606419fab0 2021-02-10T11:42:20.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f606a108700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 0x7f604c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:20.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.618 7f606a108700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 0x7f604c01ac10 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:20.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.648 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60641022b0 con 0x7f606419fab0 2021-02-10T11:42:20.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.652 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6060000940 con 0x7f606419fab0 2021-02-10T11:42:20.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.780 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60640f53c0 con 0x7f606419fab0 2021-02-10T11:42:20.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.780 7f605affd700 1 -- 172.21.2.117:0/3595113690 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f606000bd60 con 0x7f606419fab0 2021-02-10T11:42:20.769 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:20.769 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:42:20.769 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 37 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.782 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 msgr2=0x7f604c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.782 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 0x7f604c01ac10 crc :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.782 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 msgr2=0x7f60641ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.782 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f6060006f60 tx=0x7f606000b970).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 shutdown_connections 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f604c018730 0x7f604c01ac10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f606419fab0 0x7f60641ae0d0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f606419feb0 0x7f60641ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 --2- 172.21.2.117:0/3595113690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6064105ef0 0x7f606419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 >> 172.21.2.117:0/3595113690 conn(0x7f606409b640 msgr2=0x7f6064106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 shutdown_connections 2021-02-10T11:42:20.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:20.783 7f606c2ec700 1 -- 172.21.2.117:0/3595113690 wait complete. 2021-02-10T11:42:20.980 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:46.636181', 'modified': '2021-02-10 11:41:48.646579', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5786}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5786': {'gid': 5786, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3915339620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3915339620}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3915339620}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:20.980 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:20.981 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:21.377 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 5786, 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 34.752604843, 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 36, 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 56, 2021-02-10T11:42:21.378 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 56, 2021-02-10T11:42:21.379 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 40.378997615000003 2021-02-10T11:42:21.379 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:42:21.381 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5786, "want_state": "up:active", "state": "up:active", "rank_uptime": 34.752604843, "mdsmap_epoch": 36, "osdmap_epoch": 56, "osdmap_epoch_barrier": 56, "uptime": 40.378997615000003 } 2021-02-10T11:42:21.382 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:42:21.382 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:42:21.382 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:42:21.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:21.382 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:21.719 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:22.063 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:22.063 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:22.406 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:22.406 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:22.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:22.407 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:22.740 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:22.741 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:42:22.741 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000003266/subvolume_0000000000601930 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:22.744 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:23.087 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:23.087 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:23.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[131184]: starting ceph client 2021-02-10T11:42:23.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:2021-02-10 11:42:23.165 7fee2aa19f80 -1 init, newargv = 0x5636544d3220 newargc=9 2021-02-10T11:42:23.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[131184]: starting fuse 2021-02-10T11:42:23.432 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:23.432 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:23.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:23.433 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:23.767 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:23.767 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:42:23.767 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:42:23.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:23.767 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:42:23.767 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:42:23.767 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:42:23.768 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:42:23.769 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:42:23.769 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:42:23.769 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:23.769 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.alice")) 2021-02-10T11:42:23.769 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:42:24.130 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.alice.131184.asok 2021-02-10T11:42:24.133 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.131184.asok 2021-02-10T11:42:24.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:24.134 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.131184.asok status 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "alice", 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:42:24.524 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "131184", 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/volumes/group_0000000000003266/subvolume_0000000000601930" 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8259, 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8259 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:24.525 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2991164084 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2991164084 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:24.526 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8259 172.21.2.117:0/2991164084", 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2991164084", 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 36, 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 56, 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:42:24.527 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:24.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:24.528 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:24.529 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:24.863 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:24.863 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:24.864 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:42:25.240 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:42:25.240 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:42:25.241 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0267983 s, 39.1 MB/s 2021-02-10T11:42:25.242 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000601930 alice --group_name group_0000000000003266 --tenant_id tenant_id --access_level r 2021-02-10T11:42:25.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.676 7f7d4ebcc700 1 Processor -- start 2021-02-10T11:42:25.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.676 7f7d4ebcc700 1 -- start start 2021-02-10T11:42:25.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d480e4fd0 0x7f7d481b3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d481b3da0 0x7f7d480f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7d480e53d0 0x7f7d480f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480f5440 con 0x7f7d480e4fd0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480f49d0 con 0x7f7d480e53d0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4ebcc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480f4bc0 con 0x7f7d481b3da0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4c9e8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d480e4fd0 0x7f7d481b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4c9e8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d480e4fd0 0x7f7d481b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49488/0 (socket says 172.21.2.117:49488) 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.677 7f7d4c9e8700 1 -- 172.21.2.117:0/957133091 learned_addr learned my addr 172.21.2.117:0/957133091 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4d1e9700 1 --2- 172.21.2.117:0/957133091 >> v2:172.21.2.125:3300/0 conn(0x7f7d480e53d0 0x7f7d480f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d47fff700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d481b3da0 0x7f7d480f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 -- 172.21.2.117:0/957133091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d481b3da0 msgr2=0x7f7d480f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d481b3da0 0x7f7d480f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 -- 172.21.2.117:0/957133091 >> v2:172.21.2.125:3300/0 conn(0x7f7d480e53d0 msgr2=0x7f7d480f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 --2- 172.21.2.117:0/957133091 >> v2:172.21.2.125:3300/0 conn(0x7f7d480e53d0 0x7f7d480f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 -- 172.21.2.117:0/957133091 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480f4de0 con 0x7f7d480e4fd0 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4c9e8700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480e4fd0 0x7f7d481b3780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f7d38006700 tx=0x7f7d38009b80).ready entity=mon.0 client_cookie=664b52da1d2d3230 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d477fe700 1 -- 172.21.2.117:0/957133091 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3800c0c0 con 0x7f7d480e4fd0 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d477fe700 1 -- 172.21.2.117:0/957133091 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3800c440 con 0x7f7d480e4fd0 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d477fe700 1 -- 172.21.2.117:0/957133091 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3800c8a0 con 0x7f7d480e4fd0 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4ebcc700 1 -- 172.21.2.117:0/957133091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480e4fd0 msgr2=0x7f7d481b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.678 7f7d4ebcc700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480e4fd0 0x7f7d481b3780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7f7d38006700 tx=0x7f7d38009b80).stop 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 -- 172.21.2.117:0/957133091 shutdown_connections 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 --2- 172.21.2.117:0/957133091 >> v2:172.21.2.125:3300/0 conn(0x7f7d480e53d0 0x7f7d480f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d481b3da0 0x7f7d480f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 --2- 172.21.2.117:0/957133091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480e4fd0 0x7f7d481b3780 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 -- 172.21.2.117:0/957133091 >> 172.21.2.117:0/957133091 conn(0x7f7d480ce630 msgr2=0x7f7d480e4470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 -- 172.21.2.117:0/957133091 shutdown_connections 2021-02-10T11:42:25.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.679 7f7d4ebcc700 1 -- 172.21.2.117:0/957133091 wait complete. 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 Processor -- start 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 -- start start 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d481a1a30 0x7f7d481a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d480b81b0 0x7f7d480b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d481a2450 con 0x7f7d480b8bd0 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4818e140 con 0x7f7d481a1a30 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4ebcc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48097260 con 0x7f7d480b81b0 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4d1e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4d1e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49498/0 (socket says 172.21.2.117:49498) 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.694 7f7d4d1e9700 1 -- 172.21.2.117:0/3006837870 learned_addr learned my addr 172.21.2.117:0/3006837870 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 -- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d480b81b0 msgr2=0x7f7d480b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d480b81b0 0x7f7d480b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 -- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d481a1a30 msgr2=0x7f7d481a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d481a1a30 0x7f7d481a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 -- 172.21.2.117:0/3006837870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4818d4a0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4d1e9700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f7d400099a0 tx=0x7f7d4000bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d40006030 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d40000fc0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4818d4a0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4818da30 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.695 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d400056b0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.696 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d400069c0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.696 7f7d457fa700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 0x7f7d2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:25.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.696 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7d400329f0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.696 7f7d4c9e8700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 0x7f7d2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:25.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.697 7f7d4c9e8700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 0x7f7d2c01abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.727 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d481024a0 con 0x7f7d480b8bd0 2021-02-10T11:42:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.729 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d4002e540 con 0x7f7d480b8bd0 2021-02-10T11:42:25.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.855 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000601930", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000003266", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7f7d4818da30 con 0x7f7d2c0186c0 2021-02-10T11:42:25.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.899 7f7d457fa700 1 -- 172.21.2.117:0/3006837870 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f7d38000970 con 0x7f7d2c0186c0 2021-02-10T11:42:25.907 INFO:teuthology.orchestra.run.gibba017.stdout:AQCXxiNggbRSCBAA2kNC8AICLGeWqAj2g/KXLw== 2021-02-10T11:42:25.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 msgr2=0x7f7d2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 0x7f7d2c01abc0 crc :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 msgr2=0x7f7d4818ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:25.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f7d400099a0 tx=0x7f7d4000bc50).stop 2021-02-10T11:42:25.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 shutdown_connections 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d2c0186c0 0x7f7d2c01abc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.920 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d480b8bd0 0x7f7d4818ce80 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.921 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d480b81b0 0x7f7d480b85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.921 7f7d4ebcc700 1 --2- 172.21.2.117:0/3006837870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d481a1a30 0x7f7d481a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.921 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 >> 172.21.2.117:0/3006837870 conn(0x7f7d4819f150 msgr2=0x7f7d48097f20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.921 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 shutdown_connections 2021-02-10T11:42:25.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:25.921 7f7d4ebcc700 1 -- 172.21.2.117:0/3006837870 wait complete. 2021-02-10T11:42:25.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:25.917 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:25.917 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:26.251 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:26.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:26.252 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:42:26.252 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:26.252 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:26.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[131184]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:26.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:26.606 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:26.606 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:26.941 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:42:26.941 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:26.941 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:42:33.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:33.048 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:33.382 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:42:33.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 Processor -- start 2021-02-10T11:42:33.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 -- start start 2021-02-10T11:42:33.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b00e4f40 0x7f43b01b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b00f53b0 con 0x7f43b01b3d10 2021-02-10T11:42:33.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b00f6910 con 0x7f43b00e5340 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.804 7f43b5efd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00f6b30 con 0x7f43b00e4f40 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.805 7f43aeffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f43b01b3d10 0x7f43b00f5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49752/0 (socket says 172.21.2.117:49752) 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.805 7f43aeffd700 1 -- 172.21.2.117:0/513652004 learned_addr learned my addr 172.21.2.117:0/513652004 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.805 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291454953 0 0) 0x7f43a00010b0 con 0x7f43b01b3d10 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.805 7f43af7fe700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b00e4f40 0x7f43b01b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.805 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f439c002fe0 con 0x7f43b01b3d10 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2181438771 0 0) 0x7f43a00010b0 con 0x7f43b01b3d10 2021-02-10T11:42:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b00e4f40 msgr2=0x7f43b01b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b00e4f40 0x7f43b01b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 >> v1:172.21.2.125:6789/0 conn(0x7f43b00e5340 legacy=0x7f43b00f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b00f4d10 con 0x7f43b01b3d10 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f43a00015b0 con 0x7f43b01b3d10 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 >> v1:172.21.2.117:6789/0 conn(0x7f43b01b3d10 legacy=0x7f43b00f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c003aa0 0x7f439c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c006790 0x7f439c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c009500 0x7f439c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c00c2d0 con 0x7f439c003aa0 2021-02-10T11:42:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c00c4f0 con 0x7f439c009500 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c00c710 con 0x7f439c006790 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c003aa0 0x7f439c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43aeffd700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c006790 0x7f439c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43affff700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c009500 0x7f439c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 -- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c006790 msgr2=0x7f439c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c006790 0x7f439c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 -- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c009500 msgr2=0x7f439c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f439c009500 0x7f439c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43af7fe700 1 -- 172.21.2.117:0/513652004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c050980 con 0x7f439c003aa0 2021-02-10T11:42:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.806 7f43aeffd700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f439c006790 0x7f439c008c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43af7fe700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c003aa0 0x7f439c005ef0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f43980067d0 tx=0x7f4398009aa0).ready entity=mon.0 client_cookie=445d29c76f53cc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f439800c0c0 con 0x7f439c003aa0 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43980063a0 con 0x7f439c003aa0 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43ae7fc700 1 -- 172.21.2.117:0/513652004 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f439800c5c0 con 0x7f439c003aa0 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 -- 172.21.2.117:0/513652004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c003aa0 msgr2=0x7f439c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 --2- 172.21.2.117:0/513652004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f439c003aa0 0x7f439c005ef0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f43980067d0 tx=0x7f4398009aa0).stop 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43affff700 1 -- 172.21.2.117:0/513652004 reap_dead start 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 -- 172.21.2.117:0/513652004 shutdown_connections 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 -- 172.21.2.117:0/513652004 >> 172.21.2.117:0/513652004 conn(0x7f43b00ce5a0 msgr2=0x7f43b00e43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 -- 172.21.2.117:0/513652004 shutdown_connections 2021-02-10T11:42:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.807 7f43b5efd700 1 -- 172.21.2.117:0/513652004 wait complete. 2021-02-10T11:42:33.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 Processor -- start 2021-02-10T11:42:33.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 -- start start 2021-02-10T11:42:33.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 0x7f43b0190d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 0x7f43b0188ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0188950 con 0x7f43b01894f0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0199ca0 con 0x7f43b01881a0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.822 7f43b5efd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b01885a0 con 0x7f43b0187da0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49514/0 (socket says 172.21.2.117:49514) 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 -- 172.21.2.117:0/1070452548 learned_addr learned my addr 172.21.2.117:0/1070452548 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43aeffd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 0x7f43b0188ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43af7fe700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 0x7f43b0190d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 -- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 msgr2=0x7f43b0190d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 0x7f43b0190d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 -- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 msgr2=0x7f43b0188ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 0x7f43b0188ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 -- 172.21.2.117:0/1070452548 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b019a160 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43aeffd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 0x7f43b0188ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43af7fe700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 0x7f43b0190d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.823 7f43affff700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f43a800b970 tx=0x7f43a800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.824 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a8006bc0 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.824 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a80058c0 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.824 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a8005de0 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.824 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b019a160 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.824 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b019a160 con 0x7f43b01894f0 2021-02-10T11:42:33.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.825 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43a8005cb0 con 0x7f43b01894f0 2021-02-10T11:42:33.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.825 7f43a7fff700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 0x7f439401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:33.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.825 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f43a8032a10 con 0x7f43b01894f0 2021-02-10T11:42:33.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.825 7f43af7fe700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 0x7f439401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:33.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.826 7f43af7fe700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 0x7f439401abc0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:33.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.854 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b01022f0 con 0x7f43b01894f0 2021-02-10T11:42:33.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.857 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43a8026350 con 0x7f43b01894f0 2021-02-10T11:42:33.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.980 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43b00f53b0 con 0x7f43b01894f0 2021-02-10T11:42:33.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.981 7f43a7fff700 1 -- 172.21.2.117:0/1070452548 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f43a8026350 con 0x7f43b01894f0 2021-02-10T11:42:33.970 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:33.970 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":56,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:41:47.638825","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:41:44.610269","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":11,"pool_name":"cephfs_data","create_time":"2021-02-10 11:41:45.683464","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:42:33.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 msgr2=0x7f439401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 0x7f439401abc0 crc :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 msgr2=0x7f43b01898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:33.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f43a800b970 tx=0x7f43a800bd10).stop 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 shutdown_connections 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43940186c0 0x7f439401abc0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b01894f0 0x7f43b01898f0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b01881a0 0x7f43b0188ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 --2- 172.21.2.117:0/1070452548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b0187da0 0x7f43b0190d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.982 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 >> 172.21.2.117:0/1070452548 conn(0x7f43b019f040 msgr2=0x7f43b01ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.983 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 shutdown_connections 2021-02-10T11:42:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:33.983 7f43b5efd700 1 -- 172.21.2.117:0/1070452548 wait complete. 2021-02-10T11:42:33.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:41:44.610269', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:41:45.683464', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:42:33.980 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 Processor -- start 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 -- start start 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c80e4f60 0x7fd5c81b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd5c80e5360 0x7fd5c80f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80f53d0 con 0x7fd5c81b3d30 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c80e5d90 con 0x7fd5c80e5360 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.405 7fd5d074e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80d16a0 con 0x7fd5c80e4f60 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd5c80e5360 0x7fd5c80f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5cdd69700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd5c81b3d30 0x7fd5c80f5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49770/0 (socket says 172.21.2.117:49770) 2021-02-10T11:42:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5cdd69700 1 -- 172.21.2.117:0/1074776922 learned_addr learned my addr 172.21.2.117:0/1074776922 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 -- 172.21.2.117:0/1074776922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c80e4f60 msgr2=0x7fd5c81b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 --2- 172.21.2.117:0/1074776922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c80e4f60 0x7fd5c81b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 -- 172.21.2.117:0/1074776922 >> v1:172.21.2.117:6789/0 conn(0x7fd5c81b3d30 legacy=0x7fd5c80f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 -- 172.21.2.117:0/1074776922 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80d1890 con 0x7fd5c80e5360 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.406 7fd5ced6b700 1 --2- 172.21.2.117:0/1074776922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c80e5360 0x7fd5c80f4d30 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd5c0006e30 tx=0x7fd5c0009e60).ready entity=mon.0 client_cookie=9228209cc84be44 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5cd568700 1 -- 172.21.2.117:0/1074776922 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5c0006a50 con 0x7fd5c80e5360 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5cd568700 1 -- 172.21.2.117:0/1074776922 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c000b300 con 0x7fd5c80e5360 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5cd568700 1 -- 172.21.2.117:0/1074776922 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5c000b820 con 0x7fd5c80e5360 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 -- 172.21.2.117:0/1074776922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c80e5360 msgr2=0x7fd5c80f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 --2- 172.21.2.117:0/1074776922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c80e5360 0x7fd5c80f4d30 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fd5c0006e30 tx=0x7fd5c0009e60).stop 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 -- 172.21.2.117:0/1074776922 shutdown_connections 2021-02-10T11:42:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 --2- 172.21.2.117:0/1074776922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c80e5360 0x7fd5c80f4d30 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 --2- 172.21.2.117:0/1074776922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c80e4f60 0x7fd5c81b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 -- 172.21.2.117:0/1074776922 >> 172.21.2.117:0/1074776922 conn(0x7fd5c80ce5c0 msgr2=0x7fd5c80e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 -- 172.21.2.117:0/1074776922 shutdown_connections 2021-02-10T11:42:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.407 7fd5d074e700 1 -- 172.21.2.117:0/1074776922 wait complete. 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 Processor -- start 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 -- start start 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5c808ad50 0x7fd5c80c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c808b150 0x7fd5c80c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c808b6e0 con 0x7fd5c808a330 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80bff10 con 0x7fd5c808ad50 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.422 7fd5d074e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c819bdc0 con 0x7fd5c808b150 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5cdd69700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5c808ad50 0x7fd5c80c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49524/0 (socket says 172.21.2.117:49524) 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 -- 172.21.2.117:0/3559603368 learned_addr learned my addr 172.21.2.117:0/3559603368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ced6b700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c808b150 0x7fd5c80c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 -- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c808b150 msgr2=0x7fd5c80c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c808b150 0x7fd5c80c25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 -- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5c808ad50 msgr2=0x7fd5c80c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5c808ad50 0x7fd5c80c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 -- 172.21.2.117:0/3559603368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c808d7f0 con 0x7fd5c808a330 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.423 7fd5ce56a700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd5c400b530 tx=0x7fd5c400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5c4006350 con 0x7fd5c808a330 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c400bcc0 con 0x7fd5c808a330 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5c40020d0 con 0x7fd5c808a330 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c808d7f0 con 0x7fd5c808a330 2021-02-10T11:42:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c80c2ed0 con 0x7fd5c808a330 2021-02-10T11:42:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5c4007000 con 0x7fd5c808a330 2021-02-10T11:42:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 0x7fd5b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.424 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd5c4032690 con 0x7fd5c808a330 2021-02-10T11:42:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.425 7fd5cdd69700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 0x7fd5b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.425 7fd5cdd69700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 0x7fd5b001abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:34.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.455 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c80bff10 con 0x7fd5c808a330 2021-02-10T11:42:34.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.458 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5c4000960 con 0x7fd5c808a330 2021-02-10T11:42:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.583 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd5c80bff10 con 0x7fd5c808a330 2021-02-10T11:42:34.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.583 7fd5baffd700 1 -- 172.21.2.117:0/3559603368 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v56) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd5c4007000 con 0x7fd5c808a330 2021-02-10T11:42:34.573 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.585 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 msgr2=0x7fd5b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.585 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 0x7fd5b001abc0 crc :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.585 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 msgr2=0x7fd5c808a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.585 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fd5c400b530 tx=0x7fd5c400b8d0).stop 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 shutdown_connections 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5c808b150 0x7fd5c80c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5c808ad50 0x7fd5c80c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5b00186c0 0x7fd5b001abc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 --2- 172.21.2.117:0/3559603368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5c808a330 0x7fd5c808a730 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 >> 172.21.2.117:0/3559603368 conn(0x7fd5c8103170 msgr2=0x7fd5c8107170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 shutdown_connections 2021-02-10T11:42:34.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:34.586 7fd5d074e700 1 -- 172.21.2.117:0/3559603368 wait complete. 2021-02-10T11:42:34.583 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:42:35.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.007 7fc558758700 1 Processor -- start 2021-02-10T11:42:35.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.007 7fc558758700 1 -- start start 2021-02-10T11:42:35.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc558758700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc5500e5360 0x7fc5500f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc558758700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5500f53d0 con 0x7fc5500f5950 2021-02-10T11:42:35.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc558758700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5501b3f00 con 0x7fc5500e4f60 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc558758700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5500d16a0 con 0x7fc5500e5360 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc556d75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc5500f5950 0x7fc5500f5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49784/0 (socket says 172.21.2.117:49784) 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc556d75700 1 -- 172.21.2.117:0/1088288335 learned_addr learned my addr 172.21.2.117:0/1088288335 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc555d73700 1 --2- 172.21.2.117:0/1088288335 >> v2:172.21.2.117:3300/0 conn(0x7fc5500e5360 0x7fc5500f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc555d73700 1 -- 172.21.2.117:0/1088288335 >> v1:172.21.2.125:6789/0 conn(0x7fc5500e4f60 legacy=0x7fc5501b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc555d73700 1 -- 172.21.2.117:0/1088288335 >> v1:172.21.2.117:6789/0 conn(0x7fc5500f5950 legacy=0x7fc5500f5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc555d73700 1 -- 172.21.2.117:0/1088288335 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5500d1880 con 0x7fc5500e5360 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.008 7fc555572700 1 -- 172.21.2.117:0/1088288335 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084303378 0 0) 0x7fc548001500 con 0x7fc5500f5950 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc555d73700 1 --2- 172.21.2.117:0/1088288335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5500e5360 0x7fc5500f4d30 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fc54c006650 tx=0x7fc54c009a00).ready entity=mon.0 client_cookie=b9cf73475df7b4f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc555572700 1 -- 172.21.2.117:0/1088288335 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc54c0009b0 con 0x7fc5500e5360 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc555572700 1 -- 172.21.2.117:0/1088288335 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c00c0c0 con 0x7fc5500e5360 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc555572700 1 -- 172.21.2.117:0/1088288335 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc54c00c5a0 con 0x7fc5500e5360 2021-02-10T11:42:35.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc558758700 1 -- 172.21.2.117:0/1088288335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5500e5360 msgr2=0x7fc5500f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.009 7fc558758700 1 --2- 172.21.2.117:0/1088288335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5500e5360 0x7fc5500f4d30 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7fc54c006650 tx=0x7fc54c009a00).stop 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.010 7fc558758700 1 -- 172.21.2.117:0/1088288335 shutdown_connections 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.010 7fc558758700 1 --2- 172.21.2.117:0/1088288335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5500e5360 0x7fc5500f4d30 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.010 7fc558758700 1 -- 172.21.2.117:0/1088288335 >> 172.21.2.117:0/1088288335 conn(0x7fc5500ce5c0 msgr2=0x7fc5500e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.010 7fc558758700 1 -- 172.21.2.117:0/1088288335 shutdown_connections 2021-02-10T11:42:35.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.010 7fc558758700 1 -- 172.21.2.117:0/1088288335 wait complete. 2021-02-10T11:42:35.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.024 7fc558758700 1 Processor -- start 2021-02-10T11:42:35.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.024 7fc558758700 1 -- start start 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc55008a330 0x7fc55008a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc55008b150 0x7fc5500c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55008b6e0 con 0x7fc55008ad50 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5500bff10 con 0x7fc55008a330 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc558758700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55019bdc0 con 0x7fc55008b150 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49540/0 (socket says 172.21.2.117:49540) 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 -- 172.21.2.117:0/1448888813 learned_addr learned my addr 172.21.2.117:0/1448888813 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 -- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc55008b150 msgr2=0x7fc5500c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc55008b150 0x7fc5500c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc556574700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc55008a330 0x7fc55008a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 -- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc55008a330 msgr2=0x7fc55008a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc55008a330 0x7fc55008a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 -- 172.21.2.117:0/1448888813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55008d7f0 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.025 7fc555d73700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fc54c00b640 tx=0x7fc54c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc54c00bdd0 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c000c90 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc54c0022b0 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc558758700 1 -- 172.21.2.117:0/1448888813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55008d7f0 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc558758700 1 -- 172.21.2.117:0/1448888813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55008d7f0 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc54c006a10 con 0x7fc55008ad50 2021-02-10T11:42:35.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 0x7fc53801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.026 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc54c032980 con 0x7fc55008ad50 2021-02-10T11:42:35.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.027 7fc556574700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 0x7fc53801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.027 7fc556574700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 0x7fc53801abc0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.057 7fc558758700 1 -- 172.21.2.117:0/1448888813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5500f53d0 con 0x7fc55008ad50 2021-02-10T11:42:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.060 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc54c01f570 con 0x7fc55008ad50 2021-02-10T11:42:35.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.188 7fc558758700 1 -- 172.21.2.117:0/1448888813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fc5500f53d0 con 0x7fc55008ad50 2021-02-10T11:42:35.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.188 7fc542ffd700 1 -- 172.21.2.117:0/1448888813 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v56) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc54c00bdd0 con 0x7fc55008ad50 2021-02-10T11:42:35.178 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:35.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.190 7fc558758700 1 -- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 msgr2=0x7fc53801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.190 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 0x7fc53801abc0 crc :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.190 7fc558758700 1 -- 172.21.2.117:0/1448888813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 msgr2=0x7fc5500c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.190 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fc54c00b640 tx=0x7fc54c00b9e0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 -- 172.21.2.117:0/1448888813 shutdown_connections 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5380186c0 0x7fc53801abc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc55008ad50 0x7fc5500c1fa0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc55008b150 0x7fc5500c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 --2- 172.21.2.117:0/1448888813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc55008a330 0x7fc55008a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 -- 172.21.2.117:0/1448888813 >> 172.21.2.117:0/1448888813 conn(0x7fc550103170 msgr2=0x7fc550107170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 -- 172.21.2.117:0/1448888813 shutdown_connections 2021-02-10T11:42:35.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.191 7fc558758700 1 -- 172.21.2.117:0/1448888813 wait complete. 2021-02-10T11:42:35.188 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:42:35.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.612 7fe80365c700 1 Processor -- start 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 -- start start 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe7fc0e4f60 0x7fe7fc1b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0e5360 0x7fe7fc0f6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0f53d0 con 0x7fe7fc1b3d30 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc0d16a0 con 0x7fe7fc0e4f60 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe80365c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0d1890 con 0x7fe7fc0e5360 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe800c77700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe7fc1b3d30 0x7fe7fc0f3d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49794/0 (socket says 172.21.2.117:49794) 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe801478700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe7fc0e4f60 0x7fe7fc1b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.613 7fe800c77700 1 -- 172.21.2.117:0/2491145899 learned_addr learned my addr 172.21.2.117:0/2491145899 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe7effff700 1 -- 172.21.2.117:0/2491145899 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71637871 0 0) 0x7fe7f00010b0 con 0x7fe7fc1b3d30 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe7effff700 1 -- 172.21.2.117:0/2491145899 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e8002fe0 con 0x7fe7fc1b3d30 2021-02-10T11:42:35.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe801478700 1 -- 172.21.2.117:0/2491145899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0e5360 msgr2=0x7fe7fc0f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe801478700 1 --2- 172.21.2.117:0/2491145899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0e5360 0x7fe7fc0f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe801478700 1 -- 172.21.2.117:0/2491145899 >> v1:172.21.2.117:6789/0 conn(0x7fe7fc1b3d30 legacy=0x7fe7fc0f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe801478700 1 -- 172.21.2.117:0/2491145899 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0d1ab0 con 0x7fe7fc0e4f60 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe801478700 1 --2- 172.21.2.117:0/2491145899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0e4f60 0x7fe7fc1b3710 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fe7f8006650 tx=0x7fe7f8009a80).ready entity=mon.0 client_cookie=b48d5d29078baf3e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe7effff700 1 -- 172.21.2.117:0/2491145899 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f80009b0 con 0x7fe7fc0e4f60 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe7effff700 1 -- 172.21.2.117:0/2491145899 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f800c0c0 con 0x7fe7fc0e4f60 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe7effff700 1 -- 172.21.2.117:0/2491145899 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f800c520 con 0x7fe7fc0e4f60 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe80365c700 1 -- 172.21.2.117:0/2491145899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0e4f60 msgr2=0x7fe7fc1b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.614 7fe80365c700 1 --2- 172.21.2.117:0/2491145899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0e4f60 0x7fe7fc1b3710 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0x7fe7f8006650 tx=0x7fe7f8009a80).stop 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 -- 172.21.2.117:0/2491145899 shutdown_connections 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 --2- 172.21.2.117:0/2491145899 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0e5360 0x7fe7fc0f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 --2- 172.21.2.117:0/2491145899 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0e4f60 0x7fe7fc1b3710 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 -- 172.21.2.117:0/2491145899 >> 172.21.2.117:0/2491145899 conn(0x7fe7fc0ce5c0 msgr2=0x7fe7fc0e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 -- 172.21.2.117:0/2491145899 shutdown_connections 2021-02-10T11:42:35.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.615 7fe80365c700 1 -- 172.21.2.117:0/2491145899 wait complete. 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 Processor -- start 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 -- start start 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0ace50 0x7fe7fc1a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7fc102540 0x7fe7fc102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0b6120 con 0x7fe7fc0ad410 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1a7940 con 0x7fe7fc102540 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe80365c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1a7b60 con 0x7fe7fc0ace50 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe801478700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe801478700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49548/0 (socket says 172.21.2.117:49548) 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.630 7fe801478700 1 -- 172.21.2.117:0/820155663 learned_addr learned my addr 172.21.2.117:0/820155663 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 -- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0ace50 msgr2=0x7fe7fc1a7320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0ace50 0x7fe7fc1a7320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 -- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7fc102540 msgr2=0x7fe7fc102940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7fc102540 0x7fe7fc102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 -- 172.21.2.117:0/820155663 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc102f60 con 0x7fe7fc0ad410 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe801478700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fe7f8009630 tx=0x7fe7f800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f80093a0 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f800bcf0 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe7f8005560 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.631 7fe80365c700 1 -- 172.21.2.117:0/820155663 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc102f60 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.632 7fe80365c700 1 -- 172.21.2.117:0/820155663 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc102f60 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.632 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe7f8008240 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.632 7fe7edffb700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 0x7fe7dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.632 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe7f80326c0 con 0x7fe7fc0ad410 2021-02-10T11:42:35.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.633 7fe800c77700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 0x7fe7dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:35.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.633 7fe800c77700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 0x7fe7dc01abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:35.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.663 7fe80365c700 1 -- 172.21.2.117:0/820155663 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc102310 con 0x7fe7fc0ad410 2021-02-10T11:42:35.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.666 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe7f8008010 con 0x7fe7fc0ad410 2021-02-10T11:42:35.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.793 7fe80365c700 1 -- 172.21.2.117:0/820155663 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fe7fc102310 con 0x7fe7fc0ad410 2021-02-10T11:42:35.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.794 7fe7edffb700 1 -- 172.21.2.117:0/820155663 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v56) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe7f8008220 con 0x7fe7fc0ad410 2021-02-10T11:42:35.783 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.795 7fe80365c700 1 -- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 msgr2=0x7fe7dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.795 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 0x7fe7dc01abc0 crc :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.795 7fe80365c700 1 -- 172.21.2.117:0/820155663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 msgr2=0x7fe7fc1a6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.795 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fe7f8009630 tx=0x7fe7f800b900).stop 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 -- 172.21.2.117:0/820155663 shutdown_connections 2021-02-10T11:42:35.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe7fc102540 0x7fe7fc102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe7fc0ace50 0x7fe7fc1a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe7dc0186c0 0x7fe7dc01abc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 --2- 172.21.2.117:0/820155663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe7fc0ad410 0x7fe7fc1a6d00 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 -- 172.21.2.117:0/820155663 >> 172.21.2.117:0/820155663 conn(0x7fe7fc0bd540 msgr2=0x7fe7fc0a7820 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 -- 172.21.2.117:0/820155663 shutdown_connections 2021-02-10T11:42:35.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:35.796 7fe80365c700 1 -- 172.21.2.117:0/820155663 wait complete. 2021-02-10T11:42:35.794 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:42:35.794 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:36.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 Processor -- start 2021-02-10T11:42:36.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 -- start start 2021-02-10T11:42:36.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fec540e4f50 0x7fec541b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec541b3d20 0x7fec540f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540f53c0 con 0x7fec540e5350 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec540f6950 con 0x7fec541b3d20 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5c39e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec540f6b70 con 0x7fec540e4f50 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5a9bb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fec540e5350 0x7fec540f4330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49806/0 (socket says 172.21.2.117:49806) 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec5a9bb700 1 -- 172.21.2.117:0/3884265061 learned_addr learned my addr 172.21.2.117:0/3884265061 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.215 7fec599b9700 1 --2- 172.21.2.117:0/3884265061 >> v2:172.21.2.117:3300/0 conn(0x7fec541b3d20 0x7fec540f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec5a1ba700 1 --2- 172.21.2.117:0/3884265061 >> v2:172.21.2.125:3300/0 conn(0x7fec540e4f50 0x7fec541b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec599b9700 1 -- 172.21.2.117:0/3884265061 >> v2:172.21.2.125:3300/0 conn(0x7fec540e4f50 msgr2=0x7fec541b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec599b9700 1 --2- 172.21.2.117:0/3884265061 >> v2:172.21.2.125:3300/0 conn(0x7fec540e4f50 0x7fec541b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec599b9700 1 -- 172.21.2.117:0/3884265061 >> v1:172.21.2.117:6789/0 conn(0x7fec540e5350 legacy=0x7fec540f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec591b8700 1 -- 172.21.2.117:0/3884265061 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092308276 0 0) 0x7fec4c001500 con 0x7fec540e5350 2021-02-10T11:42:36.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec599b9700 1 -- 172.21.2.117:0/3884265061 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540f6d90 con 0x7fec541b3d20 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.216 7fec599b9700 1 --2- 172.21.2.117:0/3884265061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec541b3d20 0x7fec540f3d10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7fec48006650 tx=0x7fec48009a00).ready entity=mon.0 client_cookie=3d4bd2f984e16acb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.217 7fec591b8700 1 -- 172.21.2.117:0/3884265061 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec48009df0 con 0x7fec541b3d20 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.217 7fec591b8700 1 -- 172.21.2.117:0/3884265061 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec480009b0 con 0x7fec541b3d20 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.217 7fec591b8700 1 -- 172.21.2.117:0/3884265061 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec4800c2b0 con 0x7fec541b3d20 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.217 7fec5c39e700 1 -- 172.21.2.117:0/3884265061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec541b3d20 msgr2=0x7fec540f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.217 7fec5c39e700 1 --2- 172.21.2.117:0/3884265061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec541b3d20 0x7fec540f3d10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0x7fec48006650 tx=0x7fec48009a00).stop 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 -- 172.21.2.117:0/3884265061 shutdown_connections 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 --2- 172.21.2.117:0/3884265061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec541b3d20 0x7fec540f3d10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 --2- 172.21.2.117:0/3884265061 >> v2:172.21.2.125:3300/0 conn(0x7fec540e4f50 0x7fec541b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 -- 172.21.2.117:0/3884265061 >> 172.21.2.117:0/3884265061 conn(0x7fec540ce5b0 msgr2=0x7fec540e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 -- 172.21.2.117:0/3884265061 shutdown_connections 2021-02-10T11:42:36.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.218 7fec5c39e700 1 -- 172.21.2.117:0/3884265061 wait complete. 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.232 7fec5c39e700 1 Processor -- start 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.232 7fec5c39e700 1 -- start start 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec54187db0 0x7fec540981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec540a8bd0 0x7fec540a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541881b0 con 0x7fec54196800 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541883d0 con 0x7fec54187db0 2021-02-10T11:42:36.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec5c39e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec540a95f0 con 0x7fec540a8bd0 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49560/0 (socket says 172.21.2.117:49560) 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 -- 172.21.2.117:0/1901369368 learned_addr learned my addr 172.21.2.117:0/1901369368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 -- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec540a8bd0 msgr2=0x7fec540a8fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec540a8bd0 0x7fec540a8fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 -- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec54187db0 msgr2=0x7fec540981d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec54187db0 0x7fec540981d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 -- 172.21.2.117:0/1901369368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540a9810 con 0x7fec54196800 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.233 7fec599b9700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7fec480096a0 tx=0x7fec4800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.234 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec480093a0 con 0x7fec54196800 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.234 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec48002030 con 0x7fec54196800 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.234 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec48002550 con 0x7fec54196800 2021-02-10T11:42:36.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.234 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec540a9810 con 0x7fec54196800 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.234 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec540a9810 con 0x7fec54196800 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.235 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec48000c90 con 0x7fec54196800 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.235 7fec46ffd700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.235 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fec480328e0 con 0x7fec54196800 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.235 7fec5a1ba700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:36.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.236 7fec5a1ba700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:36.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.265 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5407de90 con 0x7fec54196800 2021-02-10T11:42:36.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.268 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec4801f4d0 con 0x7fec54196800 2021-02-10T11:42:36.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.395 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec540f53c0 con 0x7fec54196800 2021-02-10T11:42:36.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.396 7fec46ffd700 1 -- 172.21.2.117:0/1901369368 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fec4800bda0 con 0x7fec54196800 2021-02-10T11:42:36.386 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:36.386 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:42:36.386 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 37 2021-02-10T11:42:36.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 msgr2=0x7fec3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 crc :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 msgr2=0x7fec54196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7fec480096a0 tx=0x7fec4800b9b0).stop 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 shutdown_connections 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec3c0186c0 0x7fec3c01abc0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec54196800 0x7fec54196c00 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec540a8bd0 0x7fec540a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 --2- 172.21.2.117:0/1901369368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec54187db0 0x7fec540981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 >> 172.21.2.117:0/1901369368 conn(0x7fec541935b0 msgr2=0x7fec5407e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 shutdown_connections 2021-02-10T11:42:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:36.398 7fec5c39e700 1 -- 172.21.2.117:0/1901369368 wait complete. 2021-02-10T11:42:36.597 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:41:46.636181', 'modified': '2021-02-10 11:41:48.646579', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5786}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5786': {'gid': 5786, 'name': 'b', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3915339620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3915339620}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3915339620}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:42:36.598 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:42:36.598 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 5786, 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 50.371120099999999, 2021-02-10T11:42:36.996 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 36, 2021-02-10T11:42:36.997 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 56, 2021-02-10T11:42:36.997 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 56, 2021-02-10T11:42:36.997 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 55.997513026999997 2021-02-10T11:42:36.997 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:42:37.000 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 5786, "want_state": "up:active", "state": "up:active", "rank_uptime": 50.371120099999999, "mdsmap_epoch": 36, "osdmap_epoch": 56, "osdmap_epoch_barrier": 56, "uptime": 55.997513026999997 } 2021-02-10T11:42:37.000 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:42:37.000 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:42:37.000 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:42:37.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:37.001 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:37.326 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:37.328 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:42:37.335 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:37.672 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:37.672 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:38.015 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:38.015 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:38.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:38.016 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:38.351 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:38.351 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:42:38.351 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000003266/subvolume_0000000000601930 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:38.354 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:42:38.698 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:42:38.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:42:38.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[132015]: starting ceph client 2021-02-10T11:42:38.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:2021-02-10 11:42:38.775 7fc119b9af80 -1 init, newargv = 0x55ae9d38a210 newargc=9 2021-02-10T11:42:38.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[132015]: starting fuse 2021-02-10T11:42:39.040 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:42:39.041 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:42:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:42:39.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:39.042 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:42:39.375 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:42:39.375 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:42:39.375 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:42:39.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:42:39.376 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:42:39.377 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:42:39.378 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:42:39.378 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.alice")) 2021-02-10T11:42:39.378 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:42:39.738 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.alice.132015.asok 2021-02-10T11:42:39.741 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.132015.asok 2021-02-10T11:42:39.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:39.741 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.132015.asok status 2021-02-10T11:42:40.130 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:40.130 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:42:40.130 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:42:40.130 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "alice", 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "132015", 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/volumes/group_0000000000003266/subvolume_0000000000601930" 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8313, 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:42:40.131 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8313 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1472980329 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1472980329 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:40.132 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8313 172.21.2.117:0/1472980329", 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1472980329", 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 36, 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 56, 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:42:40.133 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:40.134 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:40.479 INFO:teuthology.orchestra.run.gibba017.stdout:data.bin 2021-02-10T11:42:40.481 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:40.481 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:40.481 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:40.815 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:40.815 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:40.815 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:42:41.162 INFO:teuthology.orchestra.run.gibba017.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:42:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:41.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:41.163 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:41.163 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:41.497 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:41.497 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:41.497 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:42:41.497 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:41.497 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:41.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[132015]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:41.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:41.849 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:41.849 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:42.183 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:42:42.184 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:42.184 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:42:42.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:42.184 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:42.519 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000601930 alice --group_name group_0000000000003266 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.944 7f8309fcb700 1 Processor -- start 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.944 7f8309fcb700 1 -- start start 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f83040e4fb0 0x7f83041b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f83041b3d80 0x7f83040f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 0x7f83040f4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040f5420 con 0x7f83040e4fb0 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83040f49b0 con 0x7f83040e53b0 2021-02-10T11:42:42.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8309fcb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040f4ba0 con 0x7f83041b3d80 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f83037fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f83040e4fb0 0x7f83041b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f83037fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f83040e4fb0 0x7f83041b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49580/0 (socket says 172.21.2.117:49580) 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f83037fe700 1 -- 172.21.2.117:0/1328904682 learned_addr learned my addr 172.21.2.117:0/1328904682 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8302ffd700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f83041b3d80 0x7f83040f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f8303fff700 1 --2- 172.21.2.117:0/1328904682 >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 0x7f83040f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f83037fe700 1 -- 172.21.2.117:0/1328904682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f83041b3d80 msgr2=0x7f83040f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.945 7f83037fe700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f83041b3d80 0x7f83040f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83037fe700 1 -- 172.21.2.117:0/1328904682 >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 msgr2=0x7f83040f4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83037fe700 1 --2- 172.21.2.117:0/1328904682 >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 0x7f83040f4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83037fe700 1 -- 172.21.2.117:0/1328904682 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040f4dc0 con 0x7f83040e4fb0 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f8303fff700 1 --2- 172.21.2.117:0/1328904682 >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 0x7f83040f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83037fe700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040e4fb0 0x7f83041b3760 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f82f4006700 tx=0x7f82f4009b00).ready entity=mon.0 client_cookie=dbd78d62d9abf99c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83027fc700 1 -- 172.21.2.117:0/1328904682 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f400c0c0 con 0x7f83040e4fb0 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83027fc700 1 -- 172.21.2.117:0/1328904682 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f400c400 con 0x7f83040e4fb0 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f83027fc700 1 -- 172.21.2.117:0/1328904682 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f400c920 con 0x7f83040e4fb0 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f8309fcb700 1 -- 172.21.2.117:0/1328904682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040e4fb0 msgr2=0x7f83041b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.946 7f8309fcb700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040e4fb0 0x7f83041b3760 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f82f4006700 tx=0x7f82f4009b00).stop 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 -- 172.21.2.117:0/1328904682 shutdown_connections 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 --2- 172.21.2.117:0/1328904682 >> v2:172.21.2.125:3300/0 conn(0x7f83040e53b0 0x7f83040f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f83041b3d80 0x7f83040f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 --2- 172.21.2.117:0/1328904682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040e4fb0 0x7f83041b3760 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 -- 172.21.2.117:0/1328904682 >> 172.21.2.117:0/1328904682 conn(0x7f83040ce610 msgr2=0x7f83040e4450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 -- 172.21.2.117:0/1328904682 shutdown_connections 2021-02-10T11:42:42.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.947 7f8309fcb700 1 -- 172.21.2.117:0/1328904682 wait complete. 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 Processor -- start 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 -- start start 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8304193720 0x7f8304193b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8304194140 0x7f8304194540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83041931a0 con 0x7f83040b4030 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8304192dc0 con 0x7f8304194140 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.962 7f8309fcb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8304192fa0 con 0x7f8304193720 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49584/0 (socket says 172.21.2.117:49584) 2021-02-10T11:42:42.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 -- 172.21.2.117:0/1258917163 learned_addr learned my addr 172.21.2.117:0/1258917163 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 -- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8304193720 msgr2=0x7f8304193b20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8304193720 0x7f8304193b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 -- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8304194140 msgr2=0x7f8304194540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8304194140 0x7f8304194540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 -- 172.21.2.117:0/1258917163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040fff70 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f83037fe700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f82f400b660 tx=0x7f82f400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f4009440 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f4000c90 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83040fff70 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304100500 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.963 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82f4002030 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.964 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82f40093e0 con 0x7f83040b4030 2021-02-10T11:42:42.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.964 7f82ebfff700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 0x7f82e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.964 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f82f40326e0 con 0x7f83040b4030 2021-02-10T11:42:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.964 7f8302ffd700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 0x7f82e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.965 7f8302ffd700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 0x7f82e401abc0 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:42.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.995 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8304080750 con 0x7f83040b4030 2021-02-10T11:42:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:42.997 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82f4030d40 con 0x7f83040b4030 2021-02-10T11:42:43.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.123 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000601930", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000003266"}) v1 -- 0x7f8304100500 con 0x7f82e40186c0 2021-02-10T11:42:43.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.160 7f82ebfff700 1 -- 172.21.2.117:0/1258917163 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f82f8006450 con 0x7f82e40186c0 2021-02-10T11:42:43.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 msgr2=0x7f82e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 0x7f82e401abc0 crc :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 msgr2=0x7f83040b4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f82f400b660 tx=0x7f82f400ba00).stop 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 shutdown_connections 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8304194140 0x7f8304194540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8304193720 0x7f8304193b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82e40186c0 0x7f82e401abc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 --2- 172.21.2.117:0/1258917163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f83040b4030 0x7f83040b4430 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.188 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 >> 172.21.2.117:0/1258917163 conn(0x7f83041aec10 msgr2=0x7f8304107050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.189 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 shutdown_connections 2021-02-10T11:42:43.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.189 7f8309fcb700 1 -- 172.21.2.117:0/1258917163 wait complete. 2021-02-10T11:42:43.185 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:42:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 Processor -- start 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 -- start start 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c580f4d10 0x7f4c580d1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c580f5110 con 0x7f4c580e4f40 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c581b3ee0 con 0x7f4c580e5340 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.606 7f4c5e3ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c580d1ca0 con 0x7f4c580f4d10 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.607 7f4c57fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4c580e4f40 0x7f4c581b36f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49842/0 (socket says 172.21.2.117:49842) 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.607 7f4c57fff700 1 -- 172.21.2.117:0/1338596900 learned_addr learned my addr 172.21.2.117:0/1338596900 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.607 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422273431 0 0) 0x7f4c48001180 con 0x7f4c580e4f40 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.607 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c002fe0 con 0x7f4c580e4f40 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.607 7f4c5ca09700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c580f4d10 0x7f4c580d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4136460552 0 0) 0x7f4c48001180 con 0x7f4c580e4f40 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c580f4d10 msgr2=0x7f4c580d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c580f4d10 0x7f4c580d1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 >> v1:172.21.2.125:6789/0 conn(0x7f4c580e5340 legacy=0x7f4c580f5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c580d1e50 con 0x7f4c580e4f40 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4c48001680 con 0x7f4c580e4f40 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 >> v1:172.21.2.117:6789/0 conn(0x7f4c580e4f40 legacy=0x7f4c581b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c3c003af0 0x7f4c3c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3c006780 0x7f4c3c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c3c009500 0x7f4c3c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c00c2d0 con 0x7f4c3c006780 2021-02-10T11:42:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c00c4f0 con 0x7f4c3c009500 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c00c710 con 0x7f4c3c003af0 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c57fff700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c3c003af0 0x7f4c3c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3c006780 0x7f4c3c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c5ca09700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c3c009500 0x7f4c3c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 -- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c3c003af0 msgr2=0x7f4c3c005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c3c003af0 0x7f4c3c005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 -- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c3c009500 msgr2=0x7f4c3c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c3c009500 0x7f4c3c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 -- 172.21.2.117:0/1338596900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c00c9b0 con 0x7f4c3c006780 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c577fe700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3c006780 0x7f4c3c008c60 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f4c4404aac0 tx=0x7f4c4404daf0).ready entity=mon.0 client_cookie=d882a92ef25b6329 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c4404a6e0 con 0x7f4c3c006780 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c440022e0 con 0x7f4c3c006780 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.608 7f4c56ffd700 1 -- 172.21.2.117:0/1338596900 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c44000960 con 0x7f4c3c006780 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 -- 172.21.2.117:0/1338596900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3c006780 msgr2=0x7f4c3c008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 --2- 172.21.2.117:0/1338596900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c3c006780 0x7f4c3c008c60 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7f4c4404aac0 tx=0x7f4c4404daf0).stop 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5ca09700 1 -- 172.21.2.117:0/1338596900 reap_dead start 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 -- 172.21.2.117:0/1338596900 shutdown_connections 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 -- 172.21.2.117:0/1338596900 >> 172.21.2.117:0/1338596900 conn(0x7f4c580ce5a0 msgr2=0x7f4c580e43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 -- 172.21.2.117:0/1338596900 shutdown_connections 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.609 7f4c5e3ec700 1 -- 172.21.2.117:0/1338596900 wait complete. 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 Processor -- start 2021-02-10T11:42:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 -- start start 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c581881a0 0x7f4c58188ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 0x7f4c581898f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58188950 con 0x7f4c58187da0 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58199ca0 con 0x7f4c581894f0 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.624 7f4c5e3ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581885a0 con 0x7f4c581881a0 2021-02-10T11:42:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49604/0 (socket says 172.21.2.117:49604) 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 -- 172.21.2.117:0/2915680331 learned_addr learned my addr 172.21.2.117:0/2915680331 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c5ca09700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 0x7f4c581898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c577fe700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c581881a0 0x7f4c58188ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 -- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c581881a0 msgr2=0x7f4c58188ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c581881a0 0x7f4c58188ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 -- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 msgr2=0x7f4c581898f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 0x7f4c581898f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 -- 172.21.2.117:0/2915680331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5819a160 con 0x7f4c58187da0 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c5ca09700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 0x7f4c581898f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:43.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c57fff700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4c48009610 tx=0x7f4c480099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c480070c0 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.625 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c480043a0 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4c48004880 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5819a160 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5818a200 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4c48008000 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c54ff9700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 0x7f4c3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.626 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4c48032780 con 0x7f4c58187da0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.627 7f4c577fe700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 0x7f4c3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:43.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.627 7f4c577fe700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 0x7f4c3401abc0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:43.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.656 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c58199ca0 con 0x7f4c58187da0 2021-02-10T11:42:43.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.659 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4c480043a0 con 0x7f4c58187da0 2021-02-10T11:42:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.788 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4c58199ca0 con 0x7f4c58187da0 2021-02-10T11:42:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.789 7f4c54ff9700 1 -- 172.21.2.117:0/2915680331 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f4c48008000 con 0x7f4c58187da0 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:42:43.778 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.779 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.780 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.781 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.782 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.783 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.784 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.785 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.786 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.787 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.788 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.789 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:43.790 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 msgr2=0x7f4c3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 0x7f4c3401abc0 crc :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 msgr2=0x7f4c58190d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f4c48009610 tx=0x7f4c480099b0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 shutdown_connections 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c581894f0 0x7f4c581898f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c581881a0 0x7f4c58188ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4c340186c0 0x7f4c3401abc0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 --2- 172.21.2.117:0/2915680331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c58187da0 0x7f4c58190d20 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:43.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 >> 172.21.2.117:0/2915680331 conn(0x7f4c5819f040 msgr2=0x7f4c581ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:43.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 shutdown_connections 2021-02-10T11:42:43.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:43.791 7f4c5e3ec700 1 -- 172.21.2.117:0/2915680331 wait complete. 2021-02-10T11:42:43.793 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000601930 --group_name group_0000000000003266 2021-02-10T11:42:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.220 7fba45d66700 1 Processor -- start 2021-02-10T11:42:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.220 7fba45d66700 1 -- start start 2021-02-10T11:42:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.220 7fba45d66700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba400e4f60 0x7fba401b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba45d66700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba400e5360 0x7fba400f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba45d66700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400f53d0 con 0x7fba401b3d30 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba45d66700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400e5d90 con 0x7fba400e5360 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba45d66700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba400d16a0 con 0x7fba400e4f60 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba400e5360 0x7fba400f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fba400e5360 0x7fba400f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49608/0 (socket says 172.21.2.117:49608) 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 -- 172.21.2.117:0/7542195 learned_addr learned my addr 172.21.2.117:0/7542195 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3f7fe700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba400e4f60 0x7fba401b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 -- 172.21.2.117:0/7542195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba400e4f60 msgr2=0x7fba401b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba400e4f60 0x7fba401b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 -- 172.21.2.117:0/7542195 >> v1:172.21.2.125:6789/0 conn(0x7fba401b3d30 legacy=0x7fba400f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 -- 172.21.2.117:0/7542195 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400d1890 con 0x7fba400e5360 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.221 7fba3ffff700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba400e5360 0x7fba400f4d30 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7fba38006f00 tx=0x7fba38009f10).ready entity=mon.0 client_cookie=52a1eabb3ded280f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba3e7fc700 1 -- 172.21.2.117:0/7542195 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba3800b3b0 con 0x7fba400e5360 2021-02-10T11:42:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba3e7fc700 1 -- 172.21.2.117:0/7542195 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3800b730 con 0x7fba400e5360 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba3e7fc700 1 -- 172.21.2.117:0/7542195 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba3800bc50 con 0x7fba400e5360 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 -- 172.21.2.117:0/7542195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba400e5360 msgr2=0x7fba400f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba400e5360 0x7fba400f4d30 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7fba38006f00 tx=0x7fba38009f10).stop 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 -- 172.21.2.117:0/7542195 shutdown_connections 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba400e5360 0x7fba400f4d30 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 --2- 172.21.2.117:0/7542195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba400e4f60 0x7fba401b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 -- 172.21.2.117:0/7542195 >> 172.21.2.117:0/7542195 conn(0x7fba400ce5c0 msgr2=0x7fba400e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 -- 172.21.2.117:0/7542195 shutdown_connections 2021-02-10T11:42:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.222 7fba45d66700 1 -- 172.21.2.117:0/7542195 wait complete. 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 Processor -- start 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 -- start start 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba401abef0 0x7fba4019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba4019cb50 0x7fba401b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4019cf50 con 0x7fba401b2860 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401a3010 con 0x7fba401abef0 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.237 7fba45d66700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba401ac2f0 con 0x7fba4019cb50 2021-02-10T11:42:44.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49612/0 (socket says 172.21.2.117:49612) 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 -- 172.21.2.117:0/320835854 learned_addr learned my addr 172.21.2.117:0/320835854 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 -- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba4019cb50 msgr2=0x7fba401b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba4019cb50 0x7fba401b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 -- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba401abef0 msgr2=0x7fba4019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba401abef0 0x7fba4019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 -- 172.21.2.117:0/320835854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba401b2c60 con 0x7fba401b2860 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba3ffff700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7fba3800b9b0 tx=0x7fba3800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba38005820 con 0x7fba401b2860 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba38005a60 con 0x7fba401b2860 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba45d66700 1 -- 172.21.2.117:0/320835854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba401b2c60 con 0x7fba401b2860 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba38006030 con 0x7fba401b2860 2021-02-10T11:42:44.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.238 7fba45d66700 1 -- 172.21.2.117:0/320835854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401b2c60 con 0x7fba401b2860 2021-02-10T11:42:44.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.239 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba38009450 con 0x7fba401b2860 2021-02-10T11:42:44.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.239 7fba37fff700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 0x7fba2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.239 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fba38032c90 con 0x7fba401b2860 2021-02-10T11:42:44.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.239 7fba3f7fe700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 0x7fba2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.240 7fba3f7fe700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 0x7fba2401abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.269 7fba45d66700 1 -- 172.21.2.117:0/320835854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba401b2f00 con 0x7fba401b2860 2021-02-10T11:42:44.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.272 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba38000f90 con 0x7fba401b2860 2021-02-10T11:42:44.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.399 7fba45d66700 1 -- 172.21.2.117:0/320835854 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000601930", "target": ["mgr", ""], "group_name": "group_0000000000003266"}) v1 -- 0x7fba401b2f00 con 0x7fba24018730 2021-02-10T11:42:44.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.411 7fba37fff700 1 -- 172.21.2.117:0/320835854 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fba300009b0 con 0x7fba24018730 2021-02-10T11:42:44.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 -- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 msgr2=0x7fba2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 0x7fba2401abc0 crc :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 -- 172.21.2.117:0/320835854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 msgr2=0x7fba4018a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7fba3800b9b0 tx=0x7fba3800bd50).stop 2021-02-10T11:42:44.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 -- 172.21.2.117:0/320835854 shutdown_connections 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba24018730 0x7fba2401abc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fba401b2860 0x7fba4018a680 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fba4019cb50 0x7fba401b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 --2- 172.21.2.117:0/320835854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fba401abef0 0x7fba4019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.416 7fba45d66700 1 -- 172.21.2.117:0/320835854 >> 172.21.2.117:0/320835854 conn(0x7fba4019be20 msgr2=0x7fba40102600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.417 7fba45d66700 1 -- 172.21.2.117:0/320835854 shutdown_connections 2021-02-10T11:42:44.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.417 7fba45d66700 1 -- 172.21.2.117:0/320835854 wait complete. 2021-02-10T11:42:44.414 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000003266 2021-02-10T11:42:44.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 Processor -- start 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 -- start start 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01380e4f70 0x7f01381b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 0x7f01380f6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380f53e0 con 0x7f01381b3d40 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01380d16b0 con 0x7f01380e4f70 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.839 7f013d22e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380d18a0 con 0x7f01380e5370 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f013659c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f01381b3d40 0x7f01380f3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42796/0 (socket says 172.21.2.117:42796) 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f013659c700 1 -- 172.21.2.117:0/59516767 learned_addr learned my addr 172.21.2.117:0/59516767 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0136d9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01380e4f70 0x7f01381b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f013759e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 0x7f01380f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0136d9d700 1 -- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 msgr2=0x7f01380f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:44.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0136d9d700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 0x7f01380f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0136d9d700 1 -- 172.21.2.117:0/59516767 >> v1:172.21.2.125:6789/0 conn(0x7f01381b3d40 legacy=0x7f01380f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0135d9b700 1 -- 172.21.2.117:0/59516767 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424700846 0 0) 0x7f012c0010b0 con 0x7f01381b3d40 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f0136d9d700 1 -- 172.21.2.117:0/59516767 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380d1ac0 con 0x7f01380e4f70 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.840 7f013759e700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 0x7f01380f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f0136d9d700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01380e4f70 0x7f01381b3720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f0128006650 tx=0x7f0128009a80).ready entity=mon.1 client_cookie=e7ba428dcbe9653b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f0135d9b700 1 -- 172.21.2.117:0/59516767 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01280009b0 con 0x7f01380e4f70 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f0135d9b700 1 -- 172.21.2.117:0/59516767 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012800c0c0 con 0x7f01380e4f70 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f0135d9b700 1 -- 172.21.2.117:0/59516767 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012800c520 con 0x7f01380e4f70 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 -- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01380e4f70 msgr2=0x7f01381b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01380e4f70 0x7f01381b3720 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f0128006650 tx=0x7f0128009a80).stop 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 -- 172.21.2.117:0/59516767 shutdown_connections 2021-02-10T11:42:44.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01380e5370 0x7f01380f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 --2- 172.21.2.117:0/59516767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01380e4f70 0x7f01381b3720 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.841 7f013d22e700 1 -- 172.21.2.117:0/59516767 >> 172.21.2.117:0/59516767 conn(0x7f01380ce5d0 msgr2=0x7f01380e4410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:44.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.842 7f013d22e700 1 -- 172.21.2.117:0/59516767 shutdown_connections 2021-02-10T11:42:44.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.842 7f013d22e700 1 -- 172.21.2.117:0/59516767 wait complete. 2021-02-10T11:42:44.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.856 7f013d22e700 1 Processor -- start 2021-02-10T11:42:44.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.856 7f013d22e700 1 -- start start 2021-02-10T11:42:44.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01380a9b80 0x7f01380a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f013807edf0 0x7f013807f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013808b6b0 con 0x7f01380a9b80 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013808b330 con 0x7f013808af30 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013d22e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013807f810 con 0x7f013807edf0 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01380a9b80 0x7f01380a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54008/0 (socket says 172.21.2.117:54008) 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 -- 172.21.2.117:0/789743044 learned_addr learned my addr 172.21.2.117:0/789743044 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f013759e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f013807edf0 0x7f013807f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 -- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f013807edf0 msgr2=0x7f013807f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f013807edf0 0x7f013807f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 -- 172.21.2.117:0/789743044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01380a9b80 msgr2=0x7f01380a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01380a9b80 0x7f01380a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 -- 172.21.2.117:0/789743044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013807fa30 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.857 7f0136d9d700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f0128009630 tx=0x7f012800b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f012800bd60 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0128006740 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0128002300 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f013d22e700 1 -- 172.21.2.117:0/789743044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013807fa30 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f013d22e700 1 -- 172.21.2.117:0/789743044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0138106ba0 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.858 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0128000c90 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.859 7f01277fe700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.859 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f01280326e0 con 0x7f013808af30 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.859 7f013659c700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:44.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.859 7f013659c700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:44.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.889 7f013d22e700 1 -- 172.21.2.117:0/789743044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01380f53e0 con 0x7f013808af30 2021-02-10T11:42:44.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:44.893 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0128030d40 con 0x7f013808af30 2021-02-10T11:42:45.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.022 7f013d22e700 1 -- 172.21.2.117:0/789743044 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000003266"}) v1 -- 0x7f0138106ba0 con 0x7f011c0186c0 2021-02-10T11:42:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.026 7f01277fe700 1 -- 172.21.2.117:0/789743044 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f012c0009b0 con 0x7f011c0186c0 2021-02-10T11:42:45.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 msgr2=0x7f011c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 0x7f011c01abc0 crc :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 msgr2=0x7f01380bd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f0128009630 tx=0x7f012800b970).stop 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 shutdown_connections 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f011c0186c0 0x7f011c01abc0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01380a9b80 0x7f01380a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f013807edf0 0x7f013807f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 --2- 172.21.2.117:0/789743044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f013808af30 0x7f01380bd330 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:45.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 >> 172.21.2.117:0/789743044 conn(0x7f013808e730 msgr2=0x7f013807ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:45.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 shutdown_connections 2021-02-10T11:42:45.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:45.031 7f013d22e700 1 -- 172.21.2.117:0/789743044 wait complete. 2021-02-10T11:42:45.028 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:42:45.028 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:45.372 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:45.715 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:42:45.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:45.716 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:45.716 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:46.050 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:42:46.050 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:46.050 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:42:46.050 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:42:46.050 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:46.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[129222]: fuse finished with error 0 and tester_r 0 2021-02-10T11:42:46.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:46.403 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:46.403 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:46.736 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:42:46.737 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:42:46.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:46.737 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:47.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:47.071 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:47.071 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:47.405 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:42:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:47.405 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:47.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:47.406 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:47.740 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:47.740 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:47.740 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:48.073 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:42:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:48.073 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:42:48.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:48.074 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:48.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:48.409 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:48.409 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:48.744 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:48.744 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:48.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:42:48.745 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:49.078 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:42:49.081 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout:[global] 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout: chdir = "" 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout: auth supported = cephx 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.439 INFO:teuthology.orchestra.run.gibba017.stdout: filestore xattr use omap = true 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: mon clock drift allowed = 1.000 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: osd crush chooseleaf type = 0 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: auth debug = true 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on old message = true 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on bug = true 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: mon pg warn max object skew = 0 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default size = 2 2021-02-10T11:42:49.440 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow primary affinity = true 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow pg remap = true 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on no sortbitwise = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on too few osds = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: osd default data pool replay window = 5 2021-02-10T11:42:49.441 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: mon allow pool delete = true 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: mon cluster log file level = debug 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: debug asserts on shutdown = true 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: mon health detail to clog = false 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout:[osd] 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: osd journal size = 100 2021-02-10T11:42:49.442 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub load threshold = 5.0 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub max interval = 600 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd recover clone overlap = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd recovery max chunk = 1048576 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug shutdown = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug op order = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug verify stray on activate = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd open classes on start = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug pg log writeout = true 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.443 INFO:teuthology.orchestra.run.gibba017.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: osd map max advance = 10 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: journal zero on create = true 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: filestore apply finisher threads = 3 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: bdev debug aio = true 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug misdirected ops = true 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore block size = 96636764160 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore fsck on mount = True 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluefs = 20 2021-02-10T11:42:49.444 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluestore = 20 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug filestore = 20 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug journal = 20 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 20 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug osd = 25 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug rocksdb = 10 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd full ratio = 0.9 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: osd objectstore = bluestore 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr] 2021-02-10T11:42:49.445 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug mgr = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 10 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout:[mon] 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug paxos = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: mon data avail warn = 5 2021-02-10T11:42:49.446 INFO:teuthology.orchestra.run.gibba017.stdout: mon mgr mkfs grace = 240 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd reporter subtree level = osd 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd prime pg temp = true 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: mon op complaint time = 120 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout:[client] 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: rgw cache enabled = true 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable ops log = true 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable usage log = true 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:42:49.447 INFO:teuthology.orchestra.run.gibba017.stdout: client mount timeout = 600 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 20 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = "" 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.a] 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.b] 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.c] 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:42:49.448 INFO:teuthology.orchestra.run.gibba017.stdout:[mds] 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: debug mds = 20 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal frag = True 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal fragment size max = 10000 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal merge size = 5 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split bits = 3 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split size = 100 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug frag = True 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug scatterstat = True 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds op complaint time = 180 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: mds verify scatter = True 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:42:49.449 INFO:teuthology.orchestra.run.gibba017.stdout:[client.alice] 2021-02-10T11:42:49.450 INFO:teuthology.orchestra.run.gibba025.stdout:[global] 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: chdir = "" 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: auth supported = cephx 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: filestore xattr use omap = true 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: mon clock drift allowed = 1.000 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: osd crush chooseleaf type = 0 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: auth debug = true 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on old message = true 2021-02-10T11:42:49.451 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on bug = true 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon pg warn max object skew = 0 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default size = 2 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow primary affinity = true 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow pg remap = true 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on no sortbitwise = false 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:42:49.452 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on too few osds = false 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: osd default data pool replay window = 5 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: mon allow pool delete = true 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: mon cluster log file level = debug 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: debug asserts on shutdown = true 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: mon health detail to clog = false 2021-02-10T11:42:49.453 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout:[osd] 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: osd journal size = 100 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub load threshold = 5.0 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub max interval = 600 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.454 INFO:teuthology.orchestra.run.gibba025.stdout: osd recover clone overlap = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd recovery max chunk = 1048576 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug shutdown = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug op order = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug verify stray on activate = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd open classes on start = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug pg log writeout = true 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: osd map max advance = 10 2021-02-10T11:42:49.455 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: journal zero on create = true 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: filestore apply finisher threads = 3 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: bdev debug aio = true 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug misdirected ops = true 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore block size = 96636764160 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore fsck on mount = True 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluefs = 20 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluestore = 20 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: debug filestore = 20 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: debug journal = 20 2021-02-10T11:42:49.456 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 20 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: debug osd = 25 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: debug rocksdb = 10 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd full ratio = 0.9 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: osd objectstore = bluestore 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr] 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: debug mgr = 20 2021-02-10T11:42:49.457 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 10 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout:[mon] 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug paxos = 20 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: mon data avail warn = 5 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: mon mgr mkfs grace = 240 2021-02-10T11:42:49.458 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd reporter subtree level = osd 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd prime pg temp = true 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: mon op complaint time = 120 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout:[client] 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: rgw cache enabled = true 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable ops log = true 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable usage log = true 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: client mount timeout = 600 2021-02-10T11:42:49.459 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 20 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = "" 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.a] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.b] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.c] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout:[mds] 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: debug mds = 20 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal frag = True 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal fragment size max = 10000 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal merge size = 5 2021-02-10T11:42:49.460 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split bits = 3 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split size = 100 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug frag = True 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug scatterstat = True 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: mds op complaint time = 180 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: mds verify scatter = True 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:42:49.461 INFO:teuthology.orchestra.run.gibba025.stdout:[client.alice] 2021-02-10T11:42:49.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:42:49.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.886 7f2de0ae2700 1 Processor -- start 2021-02-10T11:42:49.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.886 7f2de0ae2700 1 -- start start 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2de0ae2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2ddc0e5240 0x7f2ddc086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2de0ae2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ddc084c60 0x7f2ddc085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2de0ae2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc085930 con 0x7f2ddc0e4e40 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2de0ae2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0e5cc0 con 0x7f2ddc084c60 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2de0ae2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc0856b0 con 0x7f2ddc0e5240 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ddc084c60 0x7f2ddc085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2dda85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2ddc0e4e40 0x7f2ddc1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49882/0 (socket says 172.21.2.117:49882) 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ddc084c60 0x7f2ddc085060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49636/0 (socket says 172.21.2.117:49636) 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2dda85e700 1 -- 172.21.2.117:0/2048988403 learned_addr learned my addr 172.21.2.117:0/2048988403 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 -- 172.21.2.117:0/2048988403 >> v2:172.21.2.125:3300/0 conn(0x7f2ddc0e5240 msgr2=0x7f2ddc086ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 --2- 172.21.2.117:0/2048988403 >> v2:172.21.2.125:3300/0 conn(0x7f2ddc0e5240 0x7f2ddc086ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 -- 172.21.2.117:0/2048988403 >> v1:172.21.2.117:6789/0 conn(0x7f2ddc0e4e40 legacy=0x7f2ddc1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:49.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.887 7f2ddb05f700 1 -- 172.21.2.117:0/2048988403 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc1b09e0 con 0x7f2ddc084c60 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2ddb05f700 1 --2- 172.21.2.117:0/2048988403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc084c60 0x7f2ddc085060 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f2dd4006e30 tx=0x7f2dd4009e60).ready entity=mon.0 client_cookie=dc35f6ddc4f0688b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2dd985c700 1 -- 172.21.2.117:0/2048988403 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dd4006a50 con 0x7f2ddc084c60 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2dd985c700 1 -- 172.21.2.117:0/2048988403 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd400b300 con 0x7f2ddc084c60 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2dd985c700 1 -- 172.21.2.117:0/2048988403 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dd400b760 con 0x7f2ddc084c60 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2de0ae2700 1 -- 172.21.2.117:0/2048988403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc084c60 msgr2=0x7f2ddc085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2de0ae2700 1 --2- 172.21.2.117:0/2048988403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc084c60 0x7f2ddc085060 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f2dd4006e30 tx=0x7f2dd4009e60).stop 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2de0ae2700 1 -- 172.21.2.117:0/2048988403 shutdown_connections 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2de0ae2700 1 --2- 172.21.2.117:0/2048988403 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc084c60 0x7f2ddc085060 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.888 7f2de0ae2700 1 --2- 172.21.2.117:0/2048988403 >> v2:172.21.2.125:3300/0 conn(0x7f2ddc0e5240 0x7f2ddc086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.889 7f2de0ae2700 1 -- 172.21.2.117:0/2048988403 >> 172.21.2.117:0/2048988403 conn(0x7f2ddc0ce4e0 msgr2=0x7f2ddc0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.889 7f2de0ae2700 1 -- 172.21.2.117:0/2048988403 shutdown_connections 2021-02-10T11:42:49.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.889 7f2de0ae2700 1 -- 172.21.2.117:0/2048988403 wait complete. 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 Processor -- start 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 -- start start 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ddc0b1cf0 0x7f2ddc0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ddc189e50 0x7f2ddc18a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc1898d0 con 0x7f2ddc18a870 2021-02-10T11:42:49.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc093fa0 con 0x7f2ddc0b1cf0 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.903 7f2de0ae2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc187cf0 con 0x7f2ddc189e50 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49638/0 (socket says 172.21.2.117:49638) 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 -- 172.21.2.117:0/3215086130 learned_addr learned my addr 172.21.2.117:0/3215086130 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 -- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ddc189e50 msgr2=0x7f2ddc18a250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ddc189e50 0x7f2ddc18a250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 -- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ddc0b1cf0 msgr2=0x7f2ddc0b20f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ddc0b1cf0 0x7f2ddc0b20f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 -- 172.21.2.117:0/3215086130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc187f10 con 0x7f2ddc18a870 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2ddb05f700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f2dd400b970 tx=0x7f2dd400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dd4009380 con 0x7f2ddc18a870 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd4000f60 con 0x7f2ddc18a870 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc187f10 con 0x7f2ddc18a870 2021-02-10T11:42:49.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc1884a0 con 0x7f2ddc18a870 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.904 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2dd4005aa0 con 0x7f2ddc18a870 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.905 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2dd4005820 con 0x7f2ddc18a870 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.905 7f2dcb7fe700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 0x7f2dc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.905 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2dd4032cc0 con 0x7f2ddc18a870 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.905 7f2dda85e700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 0x7f2dc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:49.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.906 7f2dda85e700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 0x7f2dc001abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:49.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.936 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc085930 con 0x7f2ddc18a870 2021-02-10T11:42:49.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:49.939 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2dd4005d60 con 0x7f2ddc18a870 2021-02-10T11:42:50.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.087 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2ddc085930 con 0x7f2ddc18a870 2021-02-10T11:42:50.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.091 7f2dcb7fe700 1 -- 172.21.2.117:0/3215086130 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2dd4005820 con 0x7f2ddc18a870 2021-02-10T11:42:50.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 msgr2=0x7f2dc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 0x7f2dc001abc0 crc :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 msgr2=0x7f2ddc18ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f2dd400b970 tx=0x7f2dd400bd10).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 shutdown_connections 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2dc00186c0 0x7f2dc001abc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ddc18a870 0x7f2ddc18ac70 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ddc189e50 0x7f2ddc18a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 --2- 172.21.2.117:0/3215086130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ddc0b1cf0 0x7f2ddc0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 >> 172.21.2.117:0/3215086130 conn(0x7f2ddc19f660 msgr2=0x7f2ddc18c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 shutdown_connections 2021-02-10T11:42:50.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.096 7f2de0ae2700 1 -- 172.21.2.117:0/3215086130 wait complete. 2021-02-10T11:42:50.094 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:42:50.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.519 7fd7a39ef700 1 Processor -- start 2021-02-10T11:42:50.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.519 7fd7a39ef700 1 -- start start 2021-02-10T11:42:50.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a39ef700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd79c0e5240 0x7fd79c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a39ef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c085930 con 0x7fd79c0e4e40 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a39ef700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c1b09e0 con 0x7fd79c084c60 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a39ef700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79c0e5cc0 con 0x7fd79c0e5240 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a180b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd79c0e4e40 0x7fd79c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49896/0 (socket says 172.21.2.117:49896) 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a180b700 1 -- 172.21.2.117:0/3646042112 learned_addr learned my addr 172.21.2.117:0/3646042112 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a100a700 1 --2- 172.21.2.117:0/3646042112 >> v2:172.21.2.125:3300/0 conn(0x7fd79c0e5240 0x7fd79c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868482135 0 0) 0x7fd794001060 con 0x7fd79c0e4e40 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.520 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788002fe0 con 0x7fd79c0e4e40 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352032971 0 0) 0x7fd798000c10 con 0x7fd79c084c60 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003210 con 0x7fd79c084c60 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a100a700 1 -- 172.21.2.117:0/3646042112 >> v1:172.21.2.125:6789/0 conn(0x7fd79c084c60 legacy=0x7fd79c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a100a700 1 -- 172.21.2.117:0/3646042112 >> v1:172.21.2.117:6789/0 conn(0x7fd79c0e4e40 legacy=0x7fd79c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a100a700 1 -- 172.21.2.117:0/3646042112 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c084840 con 0x7fd79c0e5240 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a100a700 1 --2- 172.21.2.117:0/3646042112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c0e5240 0x7fd79c086ec0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd78c006650 tx=0x7fd78c009a80).ready entity=mon.1 client_cookie=9d80589e3475266b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd78c0009b0 con 0x7fd79c0e5240 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c00c0c0 con 0x7fd79c0e5240 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a0809700 1 -- 172.21.2.117:0/3646042112 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd78c0009b0 con 0x7fd79c0e5240 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a39ef700 1 -- 172.21.2.117:0/3646042112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c0e5240 msgr2=0x7fd79c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.521 7fd7a39ef700 1 --2- 172.21.2.117:0/3646042112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c0e5240 0x7fd79c086ec0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd78c006650 tx=0x7fd78c009a80).stop 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.522 7fd7a39ef700 1 -- 172.21.2.117:0/3646042112 shutdown_connections 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.522 7fd7a39ef700 1 --2- 172.21.2.117:0/3646042112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c0e5240 0x7fd79c086ec0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.522 7fd7a39ef700 1 -- 172.21.2.117:0/3646042112 >> 172.21.2.117:0/3646042112 conn(0x7fd79c0ce4e0 msgr2=0x7fd79c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.522 7fd7a39ef700 1 -- 172.21.2.117:0/3646042112 shutdown_connections 2021-02-10T11:42:50.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.522 7fd7a39ef700 1 -- 172.21.2.117:0/3646042112 wait complete. 2021-02-10T11:42:50.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 Processor -- start 2021-02-10T11:42:50.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 -- start start 2021-02-10T11:42:50.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 0x7fd79c08b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd79c18bed0 0x7fd79c18c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c08c710 con 0x7fd79c18b0c0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c08be10 con 0x7fd79c18b4c0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a39ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79c08c220 con 0x7fd79c18bed0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.537 7fd7a180b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a100a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 0x7fd79c08b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49656/0 (socket says 172.21.2.117:49656) 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a200c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd79c18bed0 0x7fd79c18c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 -- 172.21.2.117:0/370744516 learned_addr learned my addr 172.21.2.117:0/370744516 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 -- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd79c18bed0 msgr2=0x7fd79c18c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd79c18bed0 0x7fd79c18c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 -- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 msgr2=0x7fd79c08b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 0x7fd79c08b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 -- 172.21.2.117:0/370744516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c18c8f0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a100a700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 0x7fd79c08b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a180b700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fd794009960 tx=0x7fd79400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7940062b0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7940065f0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.538 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c18c8f0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.539 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd7940049a0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.539 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c18c8f0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.539 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd7940062b0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.539 7fd7927fc700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 0x7fd78401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.539 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd794032b10 con 0x7fd79c18b0c0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.540 7fd7a100a700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 0x7fd78401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:50.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.540 7fd7a100a700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 0x7fd78401ac10 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.570 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c10c2a0 con 0x7fd79c18b0c0 2021-02-10T11:42:50.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.573 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd79400be30 con 0x7fd79c18b0c0 2021-02-10T11:42:50.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.723 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7fd79c085930 con 0x7fd79c18b0c0 2021-02-10T11:42:50.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.726 7fd7927fc700 1 -- 172.21.2.117:0/370744516 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v12) v1 ==== 106+0+0 (secure 0 0 0) 0x7fd794006220 con 0x7fd79c18b0c0 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 msgr2=0x7fd78401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 0x7fd78401ac10 crc :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 msgr2=0x7fd79c08b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7fd794009960 tx=0x7fd79400bcf0).stop 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 shutdown_connections 2021-02-10T11:42:50.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd79c18bed0 0x7fd79c18c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd79c18b4c0 0x7fd79c08b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd784018730 0x7fd78401ac10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 --2- 172.21.2.117:0/370744516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd79c18b0c0 0x7fd79c08b1d0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 >> 172.21.2.117:0/370744516 conn(0x7fd79c1a79d0 msgr2=0x7fd79c0924d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 shutdown_connections 2021-02-10T11:42:50.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:50.732 7fd7a39ef700 1 -- 172.21.2.117:0/370744516 wait complete. 2021-02-10T11:42:50.730 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.154 7efcde204700 1 Processor -- start 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.154 7efcde204700 1 -- start start 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcde204700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 0x7efcd80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcde204700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcde204700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd80e77f0 con 0x7efcd80e6b90 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcde204700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd80e5ce0 con 0x7efcd81b07f0 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcde204700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd80e54b0 con 0x7efcd80e6170 2021-02-10T11:42:51.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcd77fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcd7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 0x7efcd80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcd77fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47340/0 (socket says 172.21.2.117:47340) 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcd7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 0x7efcd80e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49660/0 (socket says 172.21.2.117:49660) 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcdc821700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efcd80e6b90 0x7efcd80e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49910/0 (socket says 172.21.2.117:49910) 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.155 7efcd77fe700 1 -- 172.21.2.117:0/2210746263 learned_addr learned my addr 172.21.2.117:0/2210746263 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd77fe700 1 -- 172.21.2.117:0/2210746263 >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 msgr2=0x7efcd80e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd77fe700 1 --2- 172.21.2.117:0/2210746263 >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 0x7efcd80e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd77fe700 1 -- 172.21.2.117:0/2210746263 >> v1:172.21.2.117:6789/0 conn(0x7efcd80e6b90 legacy=0x7efcd80e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd77fe700 1 -- 172.21.2.117:0/2210746263 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd80e8d80 con 0x7efcd80e6170 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd77fe700 1 --2- 172.21.2.117:0/2210746263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7efcc4006700 tx=0x7efcc4009b80).ready entity=mon.2 client_cookie=db7e0ed0f408ad0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd6ffd700 1 -- 172.21.2.117:0/2210746263 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc400c0c0 con 0x7efcd80e6170 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd6ffd700 1 -- 172.21.2.117:0/2210746263 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc400c340 con 0x7efcd80e6170 2021-02-10T11:42:51.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcd6ffd700 1 -- 172.21.2.117:0/2210746263 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcc400c860 con 0x7efcd80e6170 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcde204700 1 -- 172.21.2.117:0/2210746263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 msgr2=0x7efcd80e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.156 7efcde204700 1 --2- 172.21.2.117:0/2210746263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7efcc4006700 tx=0x7efcc4009b80).stop 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 -- 172.21.2.117:0/2210746263 shutdown_connections 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 --2- 172.21.2.117:0/2210746263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80e6170 0x7efcd80e6570 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 --2- 172.21.2.117:0/2210746263 >> v2:172.21.2.117:3300/0 conn(0x7efcd81b07f0 0x7efcd80e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 -- 172.21.2.117:0/2210746263 >> 172.21.2.117:0/2210746263 conn(0x7efcd80ce500 msgr2=0x7efcd80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 -- 172.21.2.117:0/2210746263 shutdown_connections 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.157 7efcde204700 1 -- 172.21.2.117:0/2210746263 wait complete. 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.171 7efcde204700 1 Processor -- start 2021-02-10T11:42:51.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.171 7efcde204700 1 -- start start 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 0x7efcd8093de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcd8094400 0x7efcd81941a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8195010 con 0x7efcd8095710 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8195590 con 0x7efcd8094400 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcde204700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd8195770 con 0x7efcd80b83e0 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49664/0 (socket says 172.21.2.117:49664) 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 -- 172.21.2.117:0/24697860 learned_addr learned my addr 172.21.2.117:0/24697860 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcd7fff700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 0x7efcd8093de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcd77fe700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcd8094400 0x7efcd81941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 -- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 msgr2=0x7efcd8093de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 0x7efcd8093de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 -- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcd8094400 msgr2=0x7efcd81941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcd8094400 0x7efcd81941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 -- 172.21.2.117:0/24697860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd8195990 con 0x7efcd8095710 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcd7fff700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 0x7efcd8093de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.172 7efcdc821700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7efcd0007f90 tx=0x7efcd000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd0000ec0 con 0x7efcd8095710 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0007020 con 0x7efcd8095710 2021-02-10T11:42:51.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcde204700 1 -- 172.21.2.117:0/24697860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd8195990 con 0x7efcd8095710 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcde204700 1 -- 172.21.2.117:0/24697860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd8195f20 con 0x7efcd8095710 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd0007470 con 0x7efcd8095710 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcd0007200 con 0x7efcd8095710 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 0x7efcb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.173 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efcd0032ae0 con 0x7efcd8095710 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.174 7efcd7fff700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 0x7efcb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.174 7efcd7fff700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 0x7efcb401ac10 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.204 7efcde204700 1 -- 172.21.2.117:0/24697860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd80e77f0 con 0x7efcd8095710 2021-02-10T11:42:51.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.207 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcd0001270 con 0x7efcd8095710 2021-02-10T11:42:51.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.357 7efcde204700 1 -- 172.21.2.117:0/24697860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7efcd80e77f0 con 0x7efcd8095710 2021-02-10T11:42:51.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.361 7efcd4ff9700 1 -- 172.21.2.117:0/24697860 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7efcd0007020 con 0x7efcd8095710 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 msgr2=0x7efcb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 0x7efcb401ac10 crc :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 msgr2=0x7efcd81947c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7efcd0007f90 tx=0x7efcd000ac50).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 shutdown_connections 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcb4018730 0x7efcb401ac10 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcd8095710 0x7efcd81947c0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcd8094400 0x7efcd81941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 --2- 172.21.2.117:0/24697860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcd80b83e0 0x7efcd8093de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 >> 172.21.2.117:0/24697860 conn(0x7efcd808cd80 msgr2=0x7efcd810c800 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 shutdown_connections 2021-02-10T11:42:51.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.366 7efcde204700 1 -- 172.21.2.117:0/24697860 wait complete. 2021-02-10T11:42:51.365 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume' 2021-02-10T11:42:51.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.785 7f71ad848700 1 Processor -- start 2021-02-10T11:42:51.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.785 7f71ad848700 1 -- start start 2021-02-10T11:42:51.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71ad848700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 0x7f71a80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71ad848700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 0x7f71a80eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71ad848700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a80ede60 con 0x7f71a80ee3e0 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71ad848700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a80eba90 con 0x7f71a80d18e0 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71ad848700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a80d1ce0 con 0x7f71a80eb000 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a6d73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 0x7f71a80eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a7d75700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f71a80ee3e0 0x7f71a80ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42848/0 (socket says 172.21.2.117:42848) 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a7d75700 1 -- 172.21.2.117:0/2893243051 learned_addr learned my addr 172.21.2.117:0/2893243051 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a7574700 1 --2- 172.21.2.117:0/2893243051 >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 0x7f71a80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a6d73700 1 -- 172.21.2.117:0/2893243051 >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 msgr2=0x7f71a80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.786 7f71a6d73700 1 --2- 172.21.2.117:0/2893243051 >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 0x7f71a80ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6d73700 1 -- 172.21.2.117:0/2893243051 >> v1:172.21.2.125:6789/0 conn(0x7f71a80ee3e0 legacy=0x7f71a80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6d73700 1 -- 172.21.2.117:0/2893243051 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a80eefd0 con 0x7f71a80eb000 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a7574700 1 --2- 172.21.2.117:0/2893243051 >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 0x7f71a80ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6d73700 1 --2- 172.21.2.117:0/2893243051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 0x7f71a80eb400 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f7190006650 tx=0x7f7190009a80).ready entity=mon.2 client_cookie=735513cddfbbe218 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6572700 1 -- 172.21.2.117:0/2893243051 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71900009b0 con 0x7f71a80eb000 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6572700 1 -- 172.21.2.117:0/2893243051 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719000c0c0 con 0x7f71a80eb000 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71a6572700 1 -- 172.21.2.117:0/2893243051 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719000c5e0 con 0x7f71a80eb000 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71ad848700 1 -- 172.21.2.117:0/2893243051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 msgr2=0x7f71a80eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.787 7f71ad848700 1 --2- 172.21.2.117:0/2893243051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 0x7f71a80eb400 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f7190006650 tx=0x7f7190009a80).stop 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 -- 172.21.2.117:0/2893243051 shutdown_connections 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 --2- 172.21.2.117:0/2893243051 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a80eb000 0x7f71a80eb400 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 --2- 172.21.2.117:0/2893243051 >> v2:172.21.2.125:3300/0 conn(0x7f71a80d18e0 0x7f71a80ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 -- 172.21.2.117:0/2893243051 >> 172.21.2.117:0/2893243051 conn(0x7f71a80ced60 msgr2=0x7f71a80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 -- 172.21.2.117:0/2893243051 shutdown_connections 2021-02-10T11:42:51.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.788 7f71ad848700 1 -- 172.21.2.117:0/2893243051 wait complete. 2021-02-10T11:42:51.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 Processor -- start 2021-02-10T11:42:51.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 -- start start 2021-02-10T11:42:51.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a8095850 0x7f71a8095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71a8096270 0x7f71a8196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a80967f0 con 0x7f71a8091b00 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a81972b0 con 0x7f71a8096270 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71ad848700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a8197490 con 0x7f71a8095850 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71a7574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71a7574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49676/0 (socket says 172.21.2.117:49676) 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71a7574700 1 -- 172.21.2.117:0/3639025662 learned_addr learned my addr 172.21.2.117:0/3639025662 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.803 7f71a7d75700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71a8096270 0x7f71a8196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a6d73700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a8095850 0x7f71a8095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 -- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a8095850 msgr2=0x7f71a8095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a8095850 0x7f71a8095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 -- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71a8096270 msgr2=0x7f71a8196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71a8096270 0x7f71a8196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 -- 172.21.2.117:0/3639025662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a81976b0 con 0x7f71a8091b00 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71a7574700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f7198006f40 tx=0x7f719800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719800bd80 con 0x7f71a8091b00 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198000c90 con 0x7f71a8091b00 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7198002220 con 0x7f71a8091b00 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71ad848700 1 -- 172.21.2.117:0/3639025662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a81976b0 con 0x7f71a8091b00 2021-02-10T11:42:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.804 7f71ad848700 1 -- 172.21.2.117:0/3639025662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a81976b0 con 0x7f71a8091b00 2021-02-10T11:42:51.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.805 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f719800a000 con 0x7f71a8091b00 2021-02-10T11:42:51.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.805 7f719ffff700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 0x7f718c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:51.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.805 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7198032550 con 0x7f71a8091b00 2021-02-10T11:42:51.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.805 7f71a6d73700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 0x7f718c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:51.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.806 7f71a6d73700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 0x7f718c01abc0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:51.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.836 7f71ad848700 1 -- 172.21.2.117:0/3639025662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a8087290 con 0x7f71a8091b00 2021-02-10T11:42:51.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.838 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7198006b00 con 0x7f71a8091b00 2021-02-10T11:42:51.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:51.969 7f71ad848700 1 -- 172.21.2.117:0/3639025662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]} v 0) v1 -- 0x7f71a80ede60 con 0x7f71a8091b00 2021-02-10T11:42:52.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.152 7f719ffff700 1 -- 172.21.2.117:0/3639025662 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_legacy_subvolume"]}]=0 v330) v1 ==== 160+0+0 (secure 0 0 0) 0x7f719800a000 con 0x7f71a8091b00 2021-02-10T11:42:52.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.185 7f71ad848700 1 -- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 msgr2=0x7f718c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.185 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 0x7f718c01abc0 crc :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.185 7f71ad848700 1 -- 172.21.2.117:0/3639025662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 msgr2=0x7f71a8091f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.185 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f7198006f40 tx=0x7f719800b990).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 -- 172.21.2.117:0/3639025662 shutdown_connections 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71a8096270 0x7f71a8196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71a8095850 0x7f71a8095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f718c0186c0 0x7f718c01abc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 --2- 172.21.2.117:0/3639025662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71a8091b00 0x7f71a8091f00 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 -- 172.21.2.117:0/3639025662 >> 172.21.2.117:0/3639025662 conn(0x7f71a81a3620 msgr2=0x7f71a8092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:52.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 -- 172.21.2.117:0/3639025662 shutdown_connections 2021-02-10T11:42:52.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.186 7f71ad848700 1 -- 172.21.2.117:0/3639025662 wait complete. 2021-02-10T11:42:52.184 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_legacy_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:42:52.184 INFO:tasks.cephfs_test_runner:Starting test: test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:42:52.184 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.603 7f26e1f1f700 1 Processor -- start 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26e1f1f700 1 -- start start 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26e1f1f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f26dc0ef3f0 0x7f26dc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26e1f1f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0ede60 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26e1f1f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0d1ce0 con 0x7f26dc0d18e0 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26e1f1f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26dc0ef7f0 con 0x7f26dc0ef3f0 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26dbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f26dc0eb7f0 0x7f26dc1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49938/0 (socket says 172.21.2.117:49938) 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.604 7f26dbfff700 1 -- 172.21.2.117:0/1263365808 learned_addr learned my addr 172.21.2.117:0/1263365808 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:52.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26daffd700 1 --2- 172.21.2.117:0/1263365808 >> v2:172.21.2.125:3300/0 conn(0x7f26dc0ef3f0 0x7f26dc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666533268 0 0) 0x7f26d4001500 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8002fe0 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3666227147 0 0) 0x7f26d4001460 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 >> v2:172.21.2.125:3300/0 conn(0x7f26dc0ef3f0 msgr2=0x7f26dc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 --2- 172.21.2.117:0/1263365808 >> v2:172.21.2.125:3300/0 conn(0x7f26dc0ef3f0 0x7f26dc0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 >> v1:172.21.2.125:6789/0 conn(0x7f26dc0d18e0 legacy=0x7f26dc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc1bb180 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f26d4001980 con 0x7f26dc0eb7f0 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 >> v1:172.21.2.117:6789/0 conn(0x7f26dc0eb7f0 legacy=0x7f26dc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26c8003a40 0x7f26c8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26c8006780 0x7f26c8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 0x7f26c800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c800c2f0 con 0x7f26c80095a0 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c800c510 con 0x7f26c8006780 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8050740 con 0x7f26c8003a40 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 0x7f26c800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26db7fe700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26c8003a40 0x7f26c8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26daffd700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26c8006780 0x7f26c8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 -- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26c8003a40 msgr2=0x7f26c8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26c8003a40 0x7f26c8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 -- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26c8006780 msgr2=0x7f26c8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26c8006780 0x7f26c8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 -- 172.21.2.117:0/1263365808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c80509a0 con 0x7f26c80095a0 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26daffd700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26c8006780 0x7f26c8008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26db7fe700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26c8003a40 0x7f26c8005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 -- 172.21.2.117:0/1263365808 reap_dead start 2021-02-10T11:42:52.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.605 7f26dbfff700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 0x7f26c800baa0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f26d4001130 tx=0x7f26d400a310).ready entity=mon.0 client_cookie=aee9df99bd08ef7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26d40070c0 con 0x7f26c80095a0 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d4007340 con 0x7f26c80095a0 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26da7fc700 1 -- 172.21.2.117:0/1263365808 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26d40049a0 con 0x7f26c80095a0 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 -- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 msgr2=0x7f26c800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 0x7f26c800baa0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f26d4001130 tx=0x7f26d400a310).stop 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 -- 172.21.2.117:0/1263365808 shutdown_connections 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 --2- 172.21.2.117:0/1263365808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26c80095a0 0x7f26c800baa0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 -- 172.21.2.117:0/1263365808 >> 172.21.2.117:0/1263365808 conn(0x7f26dc0ced60 msgr2=0x7f26dc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 -- 172.21.2.117:0/1263365808 shutdown_connections 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.606 7f26e1f1f700 1 -- 172.21.2.117:0/1263365808 wait complete. 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 Processor -- start 2021-02-10T11:42:52.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 -- start start 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26dc0b5da0 0x7f26dc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 0x7f26dc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0b61a0 con 0x7f26dc0b59a0 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc08c8b0 con 0x7f26dc1a3480 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.621 7f26e1f1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc0ba1c0 con 0x7f26dc0b5da0 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49698/0 (socket says 172.21.2.117:49698) 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 -- 172.21.2.117:0/1796124746 learned_addr learned my addr 172.21.2.117:0/1796124746 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26daffd700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26dc0b5da0 0x7f26dc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26dbfff700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 0x7f26dc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 -- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26dc0b5da0 msgr2=0x7f26dc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26dc0b5da0 0x7f26dc1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 -- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 msgr2=0x7f26dc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 0x7f26dc0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 -- 172.21.2.117:0/1796124746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc1a3880 con 0x7f26dc0b59a0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26dbfff700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 0x7f26dc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26db7fe700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7f26cc006f10 tx=0x7f26cc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.622 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26cc00bd80 con 0x7f26dc0b59a0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26cc000ca0 con 0x7f26dc0b59a0 2021-02-10T11:42:52.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26cc002220 con 0x7f26dc0b59a0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc1a3880 con 0x7f26dc0b59a0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc0b2bb0 con 0x7f26dc0b59a0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26cc0052e0 con 0x7f26dc0b59a0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26d3fff700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 0x7f26c001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.623 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f26cc032940 con 0x7f26dc0b59a0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.624 7f26daffd700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 0x7f26c001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:52.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.624 7f26daffd700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 0x7f26c001abd0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:52.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.654 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc0ede60 con 0x7f26dc0b59a0 2021-02-10T11:42:52.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.656 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26cc000960 con 0x7f26dc0b59a0 2021-02-10T11:42:52.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:52.788 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7f26dc0ede60 con 0x7f26dc0b59a0 2021-02-10T11:42:53.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.157 7f26d3fff700 1 -- 172.21.2.117:0/1796124746 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v331) v1 ==== 156+0+0 (secure 0 0 0) 0x7f26cc000ca0 con 0x7f26dc0b59a0 2021-02-10T11:42:53.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 msgr2=0x7f26c001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 0x7f26c001abd0 crc :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 msgr2=0x7f26dc08c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7f26cc006f10 tx=0x7f26cc00b990).stop 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 shutdown_connections 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26dc1a3480 0x7f26dc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26dc0b5da0 0x7f26dc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26c00186d0 0x7f26c001abd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 --2- 172.21.2.117:0/1796124746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26dc0b59a0 0x7f26dc08c290 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.207 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 >> 172.21.2.117:0/1796124746 conn(0x7f26dc0bb5e0 msgr2=0x7f26dc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.208 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 shutdown_connections 2021-02-10T11:42:53.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.208 7f26e1f1f700 1 -- 172.21.2.117:0/1796124746 wait complete. 2021-02-10T11:42:53.206 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.629 7f24c6e7b700 1 Processor -- start 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.629 7f24c6e7b700 1 -- start start 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.629 7f24c6e7b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24c00d18f0 0x7f24c00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.629 7f24c6e7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 0x7f24c00eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.630 7f24c6e7b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c00eba60 con 0x7f24c00ee3e0 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.630 7f24c6e7b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c00d1cf0 con 0x7f24c00d18f0 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.630 7f24c6e7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c00eefd0 con 0x7f24c00eb040 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 0x7f24c00eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24c5498700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f24c00ee3e0 0x7f24c00ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42884/0 (socket says 172.21.2.117:42884) 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24c5498700 1 -- 172.21.2.117:0/3385389479 learned_addr learned my addr 172.21.2.117:0/3385389479 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:53.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24c4c97700 1 --2- 172.21.2.117:0/3385389479 >> v2:172.21.2.125:3300/0 conn(0x7f24c00d18f0 0x7f24c00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 -- 172.21.2.117:0/3385389479 >> v2:172.21.2.125:3300/0 conn(0x7f24c00d18f0 msgr2=0x7f24c00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 --2- 172.21.2.117:0/3385389479 >> v2:172.21.2.125:3300/0 conn(0x7f24c00d18f0 0x7f24c00eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 -- 172.21.2.117:0/3385389479 >> v1:172.21.2.125:6789/0 conn(0x7f24c00ee3e0 legacy=0x7f24c00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bf7fe700 1 -- 172.21.2.117:0/3385389479 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427554634 0 0) 0x7f24b8001190 con 0x7f24c00ee3e0 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 -- 172.21.2.117:0/3385389479 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c00ef1f0 con 0x7f24c00eb040 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.631 7f24bffff700 1 --2- 172.21.2.117:0/3385389479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 0x7f24c00eb440 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f24b00069d0 tx=0x7f24b0009e40).ready entity=mon.2 client_cookie=834bd075bf98b4af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.632 7f24bf7fe700 1 -- 172.21.2.117:0/3385389479 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24b000b510 con 0x7f24c00eb040 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.632 7f24bf7fe700 1 -- 172.21.2.117:0/3385389479 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b000b890 con 0x7f24c00eb040 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.632 7f24bf7fe700 1 -- 172.21.2.117:0/3385389479 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24b000bdb0 con 0x7f24c00eb040 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.632 7f24c6e7b700 1 -- 172.21.2.117:0/3385389479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 msgr2=0x7f24c00eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.632 7f24c6e7b700 1 --2- 172.21.2.117:0/3385389479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 0x7f24c00eb440 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f24b00069d0 tx=0x7f24b0009e40).stop 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 -- 172.21.2.117:0/3385389479 shutdown_connections 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 --2- 172.21.2.117:0/3385389479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c00eb040 0x7f24c00eb440 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 --2- 172.21.2.117:0/3385389479 >> v2:172.21.2.125:3300/0 conn(0x7f24c00d18f0 0x7f24c00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 -- 172.21.2.117:0/3385389479 >> 172.21.2.117:0/3385389479 conn(0x7f24c00cf060 msgr2=0x7f24c00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 -- 172.21.2.117:0/3385389479 shutdown_connections 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.633 7f24c6e7b700 1 -- 172.21.2.117:0/3385389479 wait complete. 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 Processor -- start 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 -- start start 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 0x7f24c00a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24c00a13a0 0x7f24c00a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c007cca0 con 0x7f24c007c7c0 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c018fb70 con 0x7f24c00a13a0 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c6e7b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c00dc6d0 con 0x7f24c01a3930 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c4c97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c4c97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49712/0 (socket says 172.21.2.117:49712) 2021-02-10T11:42:53.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.648 7f24c4c97700 1 -- 172.21.2.117:0/2079208292 learned_addr learned my addr 172.21.2.117:0/2079208292 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24bffff700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 0x7f24c00a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c5498700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24c00a13a0 0x7f24c00a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 -- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 msgr2=0x7f24c00a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 0x7f24c00a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 -- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24c00a13a0 msgr2=0x7f24c00a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24c00a13a0 0x7f24c00a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 -- 172.21.2.117:0/2079208292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c007d220 con 0x7f24c007c7c0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24bffff700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 0x7f24c00a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c4c97700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f24a8006f60 tx=0x7f24a800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a8006bb0 con 0x7f24c007c7c0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a8006340 con 0x7f24c007c7c0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c007d220 con 0x7f24c007c7c0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c007d7b0 con 0x7f24c007c7c0 2021-02-10T11:42:53.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.649 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24a8002030 con 0x7f24c007c7c0 2021-02-10T11:42:53.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.650 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24a8006b50 con 0x7f24c007c7c0 2021-02-10T11:42:53.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.650 7f24bd7fa700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 0x7f24a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:53.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.650 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f24a80326b0 con 0x7f24c007c7c0 2021-02-10T11:42:53.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.650 7f24bffff700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 0x7f24a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:53.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.651 7f24bffff700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 0x7f24a401abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:53.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.679 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c00daff0 con 0x7f24c007c7c0 2021-02-10T11:42:53.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.682 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24a801da10 con 0x7f24c007c7c0 2021-02-10T11:42:53.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.830 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f24c00daff0 con 0x7f24c007c7c0 2021-02-10T11:42:53.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.833 7f24bd7fa700 1 -- 172.21.2.117:0/2079208292 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f24a801da10 con 0x7f24c007c7c0 2021-02-10T11:42:53.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.838 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 msgr2=0x7f24a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.838 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 0x7f24a401abc0 crc :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.838 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 msgr2=0x7f24c01a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:53.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.838 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f24a8006f60 tx=0x7f24a800b970).stop 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 shutdown_connections 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24c00a13a0 0x7f24c00a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24c01a3930 0x7f24c00a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a40186c0 0x7f24a401abc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 --2- 172.21.2.117:0/2079208292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24c007c7c0 0x7f24c01a3310 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:53.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 >> 172.21.2.117:0/2079208292 conn(0x7f24c0087090 msgr2=0x7f24c008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:53.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 shutdown_connections 2021-02-10T11:42:53.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:53.839 7f24c6e7b700 1 -- 172.21.2.117:0/2079208292 wait complete. 2021-02-10T11:42:53.837 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:42:53.838 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:54.182 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:42:54.526 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:42:54.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:54.527 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:54.527 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:42:54.864 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:42:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:54.864 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:42:54.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:54.865 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:54.865 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:42:55.200 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:42:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:55.201 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:42:55.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:55.201 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:55.201 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:42:55.536 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:42:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:55.537 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:42:55.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:42:55.537 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:42:55.537 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:42:55.871 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:42:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:42:55.872 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:42:55.872 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:42:56.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.293 7f2ce30d7700 1 Processor -- start 2021-02-10T11:42:56.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.293 7f2ce30d7700 1 -- start start 2021-02-10T11:42:56.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2ce30d7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2cdc0d18e0 0x7f2cdc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2ce30d7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0ede60 con 0x7f2cdc0eb000 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2ce30d7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0d1ce0 con 0x7f2cdc0ef3f0 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2ce30d7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc0ef7f0 con 0x7f2cdc0d18e0 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2cdbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2cdc0eb000 0x7f2cdc0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:49968/0 (socket says 172.21.2.117:49968) 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2cdbfff700 1 -- 172.21.2.117:0/1014546191 learned_addr learned my addr 172.21.2.117:0/1014546191 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875861421 0 0) 0x7f2ccc001060 con 0x7f2cdc0eb000 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.294 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc8002fe0 con 0x7f2cdc0eb000 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 --2- 172.21.2.117:0/1014546191 >> v2:172.21.2.125:3300/0 conn(0x7f2cdc0d18e0 0x7f2cdc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3700339876 0 0) 0x7f2ccc001460 con 0x7f2cdc0eb000 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 >> v2:172.21.2.125:3300/0 conn(0x7f2cdc0d18e0 msgr2=0x7f2cdc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 --2- 172.21.2.117:0/1014546191 >> v2:172.21.2.125:3300/0 conn(0x7f2cdc0d18e0 0x7f2cdc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 >> v1:172.21.2.125:6789/0 conn(0x7f2cdc0ef3f0 legacy=0x7f2cdc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc1bb180 con 0x7f2cdc0eb000 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2ccc0019e0 con 0x7f2cdc0eb000 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 >> v1:172.21.2.117:6789/0 conn(0x7f2cdc0eb000 legacy=0x7f2cdc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 0x7f2cc8006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc80067a0 0x7f2cc8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc80095a0 0x7f2cc800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800c2f0 con 0x7f2cc8003b10 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800c510 con 0x7f2cc80095a0 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800c730 con 0x7f2cc80067a0 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 0x7f2cc8006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce16f4700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc80095a0 0x7f2cc800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2cdbfff700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc80067a0 0x7f2cc8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 -- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc80067a0 msgr2=0x7f2cc8008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:56.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc80067a0 0x7f2cc8008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 -- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc80095a0 msgr2=0x7f2cc800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc80095a0 0x7f2cc800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce16f4700 1 -- 172.21.2.117:0/1014546191 reap_dead start 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.295 7f2ce0ef3700 1 -- 172.21.2.117:0/1014546191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc80509a0 con 0x7f2cc8003b10 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce0ef3700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 0x7f2cc8006010 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f2cc400c470 tx=0x7f2cc4006c20).ready entity=mon.0 client_cookie=ed81f8e3a9f22205 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc40090d0 con 0x7f2cc8003b10 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc4009390 con 0x7f2cc8003b10 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2cdb7fe700 1 -- 172.21.2.117:0/1014546191 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc4009870 con 0x7f2cc8003b10 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 -- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 msgr2=0x7f2cc8006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 0x7f2cc8006010 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f2cc400c470 tx=0x7f2cc4006c20).stop 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 -- 172.21.2.117:0/1014546191 shutdown_connections 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 --2- 172.21.2.117:0/1014546191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc8003b10 0x7f2cc8006010 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 -- 172.21.2.117:0/1014546191 >> 172.21.2.117:0/1014546191 conn(0x7f2cdc0ced60 msgr2=0x7f2cdc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.296 7f2ce30d7700 1 -- 172.21.2.117:0/1014546191 shutdown_connections 2021-02-10T11:42:56.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.297 7f2ce30d7700 1 -- 172.21.2.117:0/1014546191 wait complete. 2021-02-10T11:42:56.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.311 7f2ce30d7700 1 Processor -- start 2021-02-10T11:42:56.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.311 7f2ce30d7700 1 -- start start 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cdc0a3210 0x7f2cdc0979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cdc0a3610 0x7f2cdc098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc097460 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc08bde0 con 0x7f2cdc0a3210 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce30d7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc08bfc0 con 0x7f2cdc0a3610 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49732/0 (socket says 172.21.2.117:49732) 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 -- 172.21.2.117:0/4181940211 learned_addr learned my addr 172.21.2.117:0/4181940211 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 -- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cdc0a3610 msgr2=0x7f2cdc098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cdc0a3610 0x7f2cdc098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 -- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cdc0a3210 msgr2=0x7f2cdc0979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cdc0a3210 0x7f2cdc0979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 -- 172.21.2.117:0/4181940211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc08c1e0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2ce16f4700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f2cd400b610 tx=0x7f2cd400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.312 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd400bda0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd4000c90 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cd40022b0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc08c1e0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc08c1e0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cd4002020 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2cd97fa700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 0x7f2cc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.313 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2cd40326b0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.314 7f2ce0ef3700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 0x7f2cc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.314 7f2ce0ef3700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 0x7f2cc001abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.343 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc0ede60 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.346 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cd4006bd0 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.470 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2cdc0ede60 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.471 7f2cd97fa700 1 -- 172.21.2.117:0/4181940211 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v56) v1 ==== 81+0+30755 (secure 0 0 0) 0x7f2cd4030d10 con 0x7f2cdc08b3c0 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 56, 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:41:47.638825", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:42:56.461 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 11, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:42:56.462 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:42:56.463 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:42:56.464 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:42:56.465 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:42:56.466 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:42:56.467 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 10, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:41:44.610269", 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:42:56.468 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:42:56.469 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "55", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:56.470 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:42:56.471 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:42:56.472 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 11, 2021-02-10T11:42:56.473 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:41:45.683464", 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:42:56.474 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "56", 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:42:56.475 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:42:56.476 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:42:56.477 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:42:56.478 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.479 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.480 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.481 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:42:56.482 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.483 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:42:56.484 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.485 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.486 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.487 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.488 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.489 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:42:56.490 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.491 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:42:56.492 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.493 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.494 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.495 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.496 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.497 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.498 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.499 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.500 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:42:56.501 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.502 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 52, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.503 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.504 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.505 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.506 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.507 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.508 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.509 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.510 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.511 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.512 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.513 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.514 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:42:56.515 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:42:56.517 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.519 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:42:56.520 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.521 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.522 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.523 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.524 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:42:56.525 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.526 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.527 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.528 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:42:56.529 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:42:56.530 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:42:56.531 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:42:56.532 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.472 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 msgr2=0x7f2cc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 0x7f2cc001abc0 crc :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 msgr2=0x7f2cdc08b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f2cd400b610 tx=0x7f2cd400b9b0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 shutdown_connections 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cc00186c0 0x7f2cc001abc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cdc08b3c0 0x7f2cdc08b7c0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cdc0a3610 0x7f2cdc098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 --2- 172.21.2.117:0/4181940211 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cdc0a3210 0x7f2cdc0979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 >> 172.21.2.117:0/4181940211 conn(0x7f2cdc087440 msgr2=0x7f2cdc098790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 shutdown_connections 2021-02-10T11:42:56.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.473 7f2ce30d7700 1 -- 172.21.2.117:0/4181940211 wait complete. 2021-02-10T11:42:56.535 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.956 7f78b1a9a700 1 Processor -- start 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.956 7f78b1a9a700 1 -- start start 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78b1a9a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78b1a9a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac0eb000 0x7f78ac0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78b1a9a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac0ede60 con 0x7f78ac0ee3e0 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78b1a9a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac0eba90 con 0x7f78ac0d18e0 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78b1a9a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0d1ce0 con 0x7f78ac0eb000 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78ab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78ab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49736/0 (socket says 172.21.2.117:49736) 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.957 7f78ab7fe700 1 -- 172.21.2.117:0/2916008074 learned_addr learned my addr 172.21.2.117:0/2916008074 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:56.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78ab7fe700 1 -- 172.21.2.117:0/2916008074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac0eb000 msgr2=0x7f78ac0eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78ab7fe700 1 --2- 172.21.2.117:0/2916008074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac0eb000 0x7f78ac0eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78ab7fe700 1 -- 172.21.2.117:0/2916008074 >> v1:172.21.2.125:6789/0 conn(0x7f78ac0ee3e0 legacy=0x7f78ac0ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78ab7fe700 1 -- 172.21.2.117:0/2916008074 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0eefd0 con 0x7f78ac0d18e0 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78ab7fe700 1 --2- 172.21.2.117:0/2916008074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f7894006700 tx=0x7f7894009b80).ready entity=mon.0 client_cookie=e4710e3fe2fde34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78aa7fc700 1 -- 172.21.2.117:0/2916008074 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789400c0c0 con 0x7f78ac0d18e0 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78aa7fc700 1 -- 172.21.2.117:0/2916008074 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789400c340 con 0x7f78ac0d18e0 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78aa7fc700 1 -- 172.21.2.117:0/2916008074 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789400c860 con 0x7f78ac0d18e0 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78b1a9a700 1 -- 172.21.2.117:0/2916008074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0d18e0 msgr2=0x7f78ac0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.958 7f78b1a9a700 1 --2- 172.21.2.117:0/2916008074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f7894006700 tx=0x7f7894009b80).stop 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 -- 172.21.2.117:0/2916008074 shutdown_connections 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 --2- 172.21.2.117:0/2916008074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac0eb000 0x7f78ac0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 --2- 172.21.2.117:0/2916008074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0d18e0 0x7f78ac0ea9e0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 -- 172.21.2.117:0/2916008074 >> 172.21.2.117:0/2916008074 conn(0x7f78ac0ced60 msgr2=0x7f78ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:56.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 -- 172.21.2.117:0/2916008074 shutdown_connections 2021-02-10T11:42:56.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.959 7f78b1a9a700 1 -- 172.21.2.117:0/2916008074 wait complete. 2021-02-10T11:42:56.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.973 7f78b1a9a700 1 Processor -- start 2021-02-10T11:42:56.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.973 7f78b1a9a700 1 -- start start 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac196390 0x7f78ac0e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78ac0e9960 0x7f78ac0e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac196810 con 0x7f78ac0e9560 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac098ad0 con 0x7f78ac0e9960 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78b1a9a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac0e8a40 con 0x7f78ac196390 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49744/0 (socket says 172.21.2.117:49744) 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 -- 172.21.2.117:0/1550558509 learned_addr learned my addr 172.21.2.117:0/1550558509 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 -- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac196390 msgr2=0x7f78ac0e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78ab7fe700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac196390 0x7f78ac0e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac196390 0x7f78ac0e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 -- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78ac0e9960 msgr2=0x7f78ac0e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78ac0e9960 0x7f78ac0e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 -- 172.21.2.117:0/1550558509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0e8c60 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.974 7f78aaffd700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7f789c00b560 tx=0x7f789c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789c00bcf0 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789c006030 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f789c0054d0 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac0e8c60 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac19cd70 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f789c006210 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 0x7f789001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.975 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f789c032750 con 0x7f78ac0e9560 2021-02-10T11:42:56.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.976 7f78ab7fe700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 0x7f789001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:56.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:56.976 7f78ab7fe700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 0x7f789001abc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:56.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.006 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac098ad0 con 0x7f78ac0e9560 2021-02-10T11:42:56.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.009 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f789c0009b0 con 0x7f78ac0e9560 2021-02-10T11:42:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.136 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78ac098ad0 con 0x7f78ac0e9560 2021-02-10T11:42:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.136 7f78a3fff700 1 -- 172.21.2.117:0/1550558509 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f789c006030 con 0x7f78ac0e9560 2021-02-10T11:42:57.126 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:57.126 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":37,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:41:48.646579","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5786},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5786":{"gid":5786,"name":"b","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3915339620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3915339620},{"type":"v1","addr":"172.21.2.125:6835","nonce":3915339620}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:42:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 37 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.138 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 msgr2=0x7f789001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.138 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 0x7f789001abc0 crc :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.138 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 msgr2=0x7f78ac0e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.138 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7f789c00b560 tx=0x7f789c00b900).stop 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 shutdown_connections 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7890018730 0x7f789001abc0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78ac0e9560 0x7f78ac0e7e00 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78ac0e9960 0x7f78ac0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:57.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 --2- 172.21.2.117:0/1550558509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78ac196390 0x7f78ac0e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 >> 172.21.2.117:0/1550558509 conn(0x7f78ac19f610 msgr2=0x7f78ac1a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:57.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 shutdown_connections 2021-02-10T11:42:57.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.139 7f78b1a9a700 1 -- 172.21.2.117:0/1550558509 wait complete. 2021-02-10T11:42:57.136 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:42:57.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.558 7f0bc83a9700 1 Processor -- start 2021-02-10T11:42:57.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.558 7f0bc83a9700 1 -- start start 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc83a9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0bc00e7180 0x7f0bc00e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc83a9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00e7580 con 0x7f0bc01b07f0 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc83a9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc00e77f0 con 0x7f0bc00e8390 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc83a9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00e5ce0 con 0x7f0bc00e7180 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc61c5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0bc01b07f0 0x7f0bc00e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50000/0 (socket says 172.21.2.117:50000) 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc61c5700 1 -- 172.21.2.117:0/3394518588 learned_addr learned my addr 172.21.2.117:0/3394518588 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc59c4700 1 --2- 172.21.2.117:0/3394518588 >> v2:172.21.2.125:3300/0 conn(0x7f0bc00e7180 0x7f0bc00e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc51c3700 1 -- 172.21.2.117:0/3394518588 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217696581 0 0) 0x7f0bbc000ce0 con 0x7f0bc01b07f0 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.559 7f0bc51c3700 1 -- 172.21.2.117:0/3394518588 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bac002fe0 con 0x7f0bc01b07f0 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc59c4700 1 -- 172.21.2.117:0/3394518588 >> v1:172.21.2.125:6789/0 conn(0x7f0bc00e8390 legacy=0x7f0bc00e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc59c4700 1 -- 172.21.2.117:0/3394518588 >> v1:172.21.2.117:6789/0 conn(0x7f0bc01b07f0 legacy=0x7f0bc00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc59c4700 1 -- 172.21.2.117:0/3394518588 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00e8f80 con 0x7f0bc00e7180 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc59c4700 1 --2- 172.21.2.117:0/3394518588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00e7180 0x7f0bc00e7d70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0bb0006650 tx=0x7f0bb0009a80).ready entity=mon.1 client_cookie=94b9b84106fae9e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc51c3700 1 -- 172.21.2.117:0/3394518588 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb000c0c0 con 0x7f0bc00e7180 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc51c3700 1 -- 172.21.2.117:0/3394518588 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000c340 con 0x7f0bc00e7180 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc83a9700 1 -- 172.21.2.117:0/3394518588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00e7180 msgr2=0x7f0bc00e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.560 7f0bc83a9700 1 --2- 172.21.2.117:0/3394518588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00e7180 0x7f0bc00e7d70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0bb0006650 tx=0x7f0bb0009a80).stop 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.561 7f0bc83a9700 1 -- 172.21.2.117:0/3394518588 shutdown_connections 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.561 7f0bc83a9700 1 --2- 172.21.2.117:0/3394518588 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00e7180 0x7f0bc00e7d70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.561 7f0bc83a9700 1 -- 172.21.2.117:0/3394518588 >> 172.21.2.117:0/3394518588 conn(0x7f0bc00ce500 msgr2=0x7f0bc00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.561 7f0bc83a9700 1 -- 172.21.2.117:0/3394518588 shutdown_connections 2021-02-10T11:42:57.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.561 7f0bc83a9700 1 -- 172.21.2.117:0/3394518588 wait complete. 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 Processor -- start 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 -- start start 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc01c5370 0x7f0bc01c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc008d020 0x7f0bc008d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc008c100 con 0x7f0bc008c600 2021-02-10T11:42:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0085d40 con 0x7f0bc01c5370 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc83a9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00c26b0 con 0x7f0bc008d020 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc59c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc59c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49756/0 (socket says 172.21.2.117:49756) 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.576 7f0bc59c4700 1 -- 172.21.2.117:0/4019141506 learned_addr learned my addr 172.21.2.117:0/4019141506 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc61c5700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc01c5370 0x7f0bc01c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc69c6700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc008d020 0x7f0bc008d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 -- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc008d020 msgr2=0x7f0bc008d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc008d020 0x7f0bc008d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 -- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc01c5370 msgr2=0x7f0bc01c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc01c5370 0x7f0bc01c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 -- 172.21.2.117:0/4019141506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00c28d0 con 0x7f0bc008c600 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc59c4700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f0bb0009630 tx=0x7f0bb000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb00093a0 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000bd70 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc00c28d0 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc00c2e60 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.577 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb0005560 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.578 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bb00052e0 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.578 7f0bb6ffd700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 0x7f0ba801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.578 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0bb0032980 con 0x7f0bc008c600 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.578 7f0bc61c5700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 0x7f0ba801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:57.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.579 7f0bc61c5700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 0x7f0ba801ac10 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:57.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.608 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0085d40 con 0x7f0bc008c600 2021-02-10T11:42:57.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.611 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bb001f570 con 0x7f0bc008c600 2021-02-10T11:42:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:57.737 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0bc0085d40 con 0x7f0bc008c600 2021-02-10T11:42:58.519 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:42:58.530 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:42:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.535 7f0bb6ffd700 1 -- 172.21.2.117:0/4019141506 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0bb00052e0 con 0x7f0bc008c600 2021-02-10T11:42:58.546 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:42:58.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.558 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 msgr2=0x7f0ba801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.558 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 0x7f0ba801ac10 crc :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.558 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 msgr2=0x7f0bc008ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.558 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f0bb0009630 tx=0x7f0bb000b980).stop 2021-02-10T11:42:58.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 shutdown_connections 2021-02-10T11:42:58.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba8018730 0x7f0ba801ac10 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc008c600 0x7f0bc008ca00 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc008d020 0x7f0bc008d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 --2- 172.21.2.117:0/4019141506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc01c5370 0x7f0bc01c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 >> 172.21.2.117:0/4019141506 conn(0x7f0bc007cb20 msgr2=0x7f0bc007dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:58.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 shutdown_connections 2021-02-10T11:42:58.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.559 7f0bc83a9700 1 -- 172.21.2.117:0/4019141506 wait complete. 2021-02-10T11:42:58.556 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:42:58.575 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.977 7f3730b21700 1 Processor -- start 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.977 7f3730b21700 1 -- start start 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f3730b21700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f372c0e5240 0x7f372c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f3730b21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c084c60 0x7f372c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f3730b21700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372c085930 con 0x7f372c0e4e40 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f3730b21700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372c085680 con 0x7f372c0e5240 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f3730b21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c1b09e0 con 0x7f372c084c60 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372a85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f372c0e4e40 0x7f372c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50014/0 (socket says 172.21.2.117:50014) 2021-02-10T11:42:58.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372a85e700 1 -- 172.21.2.117:0/1566064660 learned_addr learned my addr 172.21.2.117:0/1566064660 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621892673 0 0) 0x7f3714001180 con 0x7f372c0e4e40 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372985c700 1 -- 172.21.2.117:0/1566064660 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718002fe0 con 0x7f372c0e4e40 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372b05f700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c084c60 0x7f372c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.978 7f372a05d700 1 --2- 172.21.2.117:0/1566064660 >> v2:172.21.2.125:3300/0 conn(0x7f372c0e5240 0x7f372c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 495143665 0 0) 0x7f3714001180 con 0x7f372c0e4e40 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c084c60 msgr2=0x7f372c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c084c60 0x7f372c085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 >> v2:172.21.2.125:3300/0 conn(0x7f372c0e5240 msgr2=0x7f372c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 --2- 172.21.2.117:0/1566064660 >> v2:172.21.2.125:3300/0 conn(0x7f372c0e5240 0x7f372c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c084840 con 0x7f372c0e4e40 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3714001680 con 0x7f372c0e4e40 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 >> v1:172.21.2.117:6789/0 conn(0x7f372c0e4e40 legacy=0x7f372c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3718003a40 0x7f3718005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3718006780 0x7f3718008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180095a0 0x7f371800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180502d0 con 0x7f3718006780 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37180504b0 con 0x7f3718003a40 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37180506d0 con 0x7f37180095a0 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3718006780 0x7f3718008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372b05f700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180095a0 0x7f371800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a85e700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3718003a40 0x7f3718005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 -- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180095a0 msgr2=0x7f371800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37180095a0 0x7f371800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 -- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3718003a40 msgr2=0x7f3718005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3718003a40 0x7f3718005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:58.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 -- 172.21.2.117:0/1566064660 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3718050970 con 0x7f3718006780 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372a05d700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3718006780 0x7f3718008ca0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f371c006360 tx=0x7f371c009bb0).ready entity=mon.0 client_cookie=83e72f596e108d91 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c006560 con 0x7f3718006780 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c00c0c0 con 0x7f3718006780 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372985c700 1 -- 172.21.2.117:0/1566064660 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f371c00c550 con 0x7f3718006780 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f3730b21700 1 -- 172.21.2.117:0/1566064660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3718006780 msgr2=0x7f3718008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f3730b21700 1 --2- 172.21.2.117:0/1566064660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3718006780 0x7f3718008ca0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7f371c006360 tx=0x7f371c009bb0).stop 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.979 7f372b05f700 1 -- 172.21.2.117:0/1566064660 reap_dead start 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.980 7f3730b21700 1 -- 172.21.2.117:0/1566064660 shutdown_connections 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.980 7f3730b21700 1 -- 172.21.2.117:0/1566064660 >> 172.21.2.117:0/1566064660 conn(0x7f372c0ce4e0 msgr2=0x7f372c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.980 7f3730b21700 1 -- 172.21.2.117:0/1566064660 shutdown_connections 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.980 7f3730b21700 1 -- 172.21.2.117:0/1566064660 wait complete. 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 Processor -- start 2021-02-10T11:42:58.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 -- start start 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f372c10fe50 0x7f372c1106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c110d00 0x7f372c1a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c110250 con 0x7f372c1a3240 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0a5030 con 0x7f372c10fe50 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f3730b21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0a5250 con 0x7f372c110d00 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f372b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f372b05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49780/0 (socket says 172.21.2.117:49780) 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.995 7f372b05f700 1 -- 172.21.2.117:0/2554121916 learned_addr learned my addr 172.21.2.117:0/2554121916 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 -- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c110d00 msgr2=0x7f372c1a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c110d00 0x7f372c1a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 -- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f372c10fe50 msgr2=0x7f372c1106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f372c10fe50 0x7f372c1106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 -- 172.21.2.117:0/2554121916 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372c08ee40 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f372b05f700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f37240099e0 tx=0x7f372400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3724006030 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3724006330 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f3730b21700 1 -- 172.21.2.117:0/2554121916 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c08ee40 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f3730b21700 1 -- 172.21.2.117:0/2554121916 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c08f3d0 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.996 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37240056b0 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.997 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3724006030 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.997 7f37237fe700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 0x7f371001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.997 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f3724032ce0 con 0x7f372c1a3240 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.997 7f372a85e700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 0x7f371001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:58.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:58.997 7f372a85e700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 0x7f371001abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.028 7f3730b21700 1 -- 172.21.2.117:0/2554121916 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c085930 con 0x7f372c1a3240 2021-02-10T11:42:59.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.030 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3724006bc0 con 0x7f372c1a3240 2021-02-10T11:42:59.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.158 7f3730b21700 1 -- 172.21.2.117:0/2554121916 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f372c085930 con 0x7f372c1a3240 2021-02-10T11:42:59.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.159 7f37237fe700 1 -- 172.21.2.117:0/2554121916 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f3724006030 con 0x7f372c1a3240 2021-02-10T11:42:59.149 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:42:59.149 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":38,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7983,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3366963911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3366963911},{"type":"v1","addr":"172.21.2.117:6833","nonce":3366963911}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:41:46.636181","modified":"2021-02-10 11:42:58.530427","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2021-02-10T11:42:59.149 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 38 2021-02-10T11:42:59.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 msgr2=0x7f371001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 0x7f371001abc0 crc :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 msgr2=0x7f372c1a3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7f37240099e0 tx=0x7f372400bcf0).stop 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 shutdown_connections 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3710018730 0x7f371001abc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f372c1a3240 0x7f372c1a3640 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f372c110d00 0x7f372c1a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 --2- 172.21.2.117:0/2554121916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f372c10fe50 0x7f372c1106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 >> 172.21.2.117:0/2554121916 conn(0x7f372c0ace80 msgr2=0x7f372c0c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 shutdown_connections 2021-02-10T11:42:59.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.161 7f3730b21700 1 -- 172.21.2.117:0/2554121916 wait complete. 2021-02-10T11:42:59.159 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:42:59.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 Processor -- start 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 -- start start 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe3600e7180 0x7fe3600e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600e7580 con 0x7fe3601b07f0 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600e77f0 con 0x7fe3600e8390 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.586 7fe366821700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3600e5ce0 con 0x7fe3600e7180 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe3600e7180 0x7fe3600e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe3601b07f0 0x7fe3600e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50034/0 (socket says 172.21.2.117:50034) 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35ffff700 1 -- 172.21.2.117:0/2463331042 learned_addr learned my addr 172.21.2.117:0/2463331042 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35f7fe700 1 -- 172.21.2.117:0/2463331042 >> v1:172.21.2.125:6789/0 conn(0x7fe3600e8390 legacy=0x7fe3600e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35f7fe700 1 -- 172.21.2.117:0/2463331042 >> v1:172.21.2.117:6789/0 conn(0x7fe3601b07f0 legacy=0x7fe3600e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35f7fe700 1 -- 172.21.2.117:0/2463331042 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3600e8f80 con 0x7fe3600e7180 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35f7fe700 1 --2- 172.21.2.117:0/2463331042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3600e7180 0x7fe3600e7d70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7fe354006650 tx=0x7fe354009a80).ready entity=mon.0 client_cookie=def2d3fc34aef1ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35effd700 1 -- 172.21.2.117:0/2463331042 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35400c0c0 con 0x7fe3600e7180 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35effd700 1 -- 172.21.2.117:0/2463331042 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400c340 con 0x7fe3600e7180 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe35effd700 1 -- 172.21.2.117:0/2463331042 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35400c860 con 0x7fe3600e7180 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe366821700 1 -- 172.21.2.117:0/2463331042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3600e7180 msgr2=0x7fe3600e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.587 7fe366821700 1 --2- 172.21.2.117:0/2463331042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3600e7180 0x7fe3600e7d70 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7fe354006650 tx=0x7fe354009a80).stop 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.588 7fe366821700 1 -- 172.21.2.117:0/2463331042 shutdown_connections 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.588 7fe366821700 1 --2- 172.21.2.117:0/2463331042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe3600e7180 0x7fe3600e7d70 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.588 7fe366821700 1 -- 172.21.2.117:0/2463331042 >> 172.21.2.117:0/2463331042 conn(0x7fe3600ce500 msgr2=0x7fe3600e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.588 7fe366821700 1 -- 172.21.2.117:0/2463331042 shutdown_connections 2021-02-10T11:42:59.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.588 7fe366821700 1 -- 172.21.2.117:0/2463331042 wait complete. 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 Processor -- start 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 -- start start 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 0x7fe36008a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3600aa8e0 0x7fe36008a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360085e80 con 0x7fe36008ae60 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36008b360 con 0x7fe3600aa4e0 2021-02-10T11:42:59.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe366821700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36008bf00 con 0x7fe3600aa8e0 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe364e3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe364e3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49792/0 (socket says 172.21.2.117:49792) 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.603 7fe364e3e700 1 -- 172.21.2.117:0/278754114 learned_addr learned my addr 172.21.2.117:0/278754114 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35f7fe700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3600aa8e0 0x7fe36008a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35ffff700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 0x7fe36008a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 -- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3600aa8e0 msgr2=0x7fe36008a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3600aa8e0 0x7fe36008a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 -- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 msgr2=0x7fe36008a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 0x7fe36008a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 -- 172.21.2.117:0/278754114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36008c0e0 con 0x7fe36008ae60 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35ffff700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 0x7fe36008a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe364e3e700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7fe358009940 tx=0x7fe35800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe358000d30 con 0x7fe36008ae60 2021-02-10T11:42:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35800c2f0 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35800c7d0 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.604 7fe366821700 1 -- 172.21.2.117:0/278754114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36008c0e0 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.605 7fe366821700 1 -- 172.21.2.117:0/278754114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36008c0e0 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.605 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe358001220 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.605 7fe35cff9700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 0x7fe34001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.605 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fe358000cc0 con 0x7fe36008ae60 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.605 7fe35ffff700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 0x7fe34001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:42:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.606 7fe35ffff700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 0x7fe34001ac10 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:42:59.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.636 7fe366821700 1 -- 172.21.2.117:0/278754114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36007c0a0 con 0x7fe36008ae60 2021-02-10T11:42:59.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.638 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe358000cc0 con 0x7fe36008ae60 2021-02-10T11:42:59.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:42:59.764 7fe366821700 1 -- 172.21.2.117:0/278754114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe3600e91f0 con 0x7fe36008ae60 2021-02-10T11:43:00.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.543 7fe35cff9700 1 -- 172.21.2.117:0/278754114 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe358001220 con 0x7fe36008ae60 2021-02-10T11:43:00.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 -- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 msgr2=0x7fe34001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 0x7fe34001ac10 crc :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 -- 172.21.2.117:0/278754114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 msgr2=0x7fe36008b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0x7fe358009940 tx=0x7fe35800bcf0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 -- 172.21.2.117:0/278754114 shutdown_connections 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe340018730 0x7fe34001ac10 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36008ae60 0x7fe36008b8e0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe3600aa8e0 0x7fe36008a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 --2- 172.21.2.117:0/278754114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe3600aa4e0 0x7fe36008a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:00.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 -- 172.21.2.117:0/278754114 >> 172.21.2.117:0/278754114 conn(0x7fe36007cb20 msgr2=0x7fe36007dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:00.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.586 7fe366821700 1 -- 172.21.2.117:0/278754114 shutdown_connections 2021-02-10T11:43:00.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:00.587 7fe366821700 1 -- 172.21.2.117:0/278754114 wait complete. 2021-02-10T11:43:00.584 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:43:01.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.009 7fd127d0a700 1 Processor -- start 2021-02-10T11:43:01.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.009 7fd127d0a700 1 -- start start 2021-02-10T11:43:01.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.010 7fd127d0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 0x7fd120086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.010 7fd127d0a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd120084c60 0x7fd120085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.010 7fd127d0a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd120085930 con 0x7fd1200e4e40 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.010 7fd127d0a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd120085680 con 0x7fd120084c60 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.010 7fd127d0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201b09e0 con 0x7fd1200e5240 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd126327700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd120084c60 0x7fd120085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd125325700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 0x7fd120086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd125b26700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fd1200e4e40 0x7fd1201b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:42972/0 (socket says 172.21.2.117:42972) 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd125b26700 1 -- 172.21.2.117:0/1038234075 learned_addr learned my addr 172.21.2.117:0/1038234075 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd126327700 1 -- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 msgr2=0x7fd120086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd126327700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 0x7fd120086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd126327700 1 -- 172.21.2.117:0/1038234075 >> v1:172.21.2.125:6789/0 conn(0x7fd1200e4e40 legacy=0x7fd1201b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd126327700 1 -- 172.21.2.117:0/1038234075 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120084840 con 0x7fd120084c60 2021-02-10T11:43:01.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd124b24700 1 -- 172.21.2.117:0/1038234075 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184470608 0 0) 0x7fd114001190 con 0x7fd1200e4e40 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.011 7fd125325700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 0x7fd120086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd126327700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd120084c60 0x7fd120085060 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fd118006e50 tx=0x7fd118009e70).ready entity=mon.1 client_cookie=2cce86df22b7cab8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd124b24700 1 -- 172.21.2.117:0/1038234075 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd11800b510 con 0x7fd120084c60 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd124b24700 1 -- 172.21.2.117:0/1038234075 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd118006790 con 0x7fd120084c60 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd124b24700 1 -- 172.21.2.117:0/1038234075 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd11800b940 con 0x7fd120084c60 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd127d0a700 1 -- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd120084c60 msgr2=0x7fd120085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.012 7fd127d0a700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd120084c60 0x7fd120085060 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fd118006e50 tx=0x7fd118009e70).stop 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 -- 172.21.2.117:0/1038234075 shutdown_connections 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd120084c60 0x7fd120085060 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 --2- 172.21.2.117:0/1038234075 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200e5240 0x7fd120086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 -- 172.21.2.117:0/1038234075 >> 172.21.2.117:0/1038234075 conn(0x7fd1200ce4e0 msgr2=0x7fd1200e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 -- 172.21.2.117:0/1038234075 shutdown_connections 2021-02-10T11:43:01.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.013 7fd127d0a700 1 -- 172.21.2.117:0/1038234075 wait complete. 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.027 7fd127d0a700 1 Processor -- start 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 -- start start 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200abe50 0x7fd12008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 0x7fd12008aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201a7a80 con 0x7fd1201a7060 2021-02-10T11:43:01.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201a7ca0 con 0x7fd12008a8a0 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd127d0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201a7ec0 con 0x7fd1200abe50 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd126327700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd126327700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49804/0 (socket says 172.21.2.117:49804) 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd126327700 1 -- 172.21.2.117:0/3049764932 learned_addr learned my addr 172.21.2.117:0/3049764932 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd125325700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 0x7fd12008aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.028 7fd125b26700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200abe50 0x7fd12008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 -- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200abe50 msgr2=0x7fd12008a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200abe50 0x7fd12008a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 -- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 msgr2=0x7fd12008aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 0x7fd12008aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 -- 172.21.2.117:0/3049764932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1200b7030 con 0x7fd1201a7060 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd125325700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 0x7fd12008aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd126327700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7fd118006c50 tx=0x7fd11800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1180093d0 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11800be60 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd11800c4e0 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.029 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1200b7030 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.030 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1200b7030 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.030 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1180093d0 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.030 7fd1127fc700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 0x7fd10801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.030 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fd118032ab0 con 0x7fd1201a7060 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.031 7fd125b26700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 0x7fd10801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.031 7fd125b26700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 0x7fd10801abc0 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.060 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12010c2b0 con 0x7fd1201a7060 2021-02-10T11:43:01.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.062 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd118020ed0 con 0x7fd1201a7060 2021-02-10T11:43:01.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.188 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd12010c2b0 con 0x7fd1201a7060 2021-02-10T11:43:01.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.549 7fd1127fc700 1 -- 172.21.2.117:0/3049764932 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd1180093d0 con 0x7fd1201a7060 2021-02-10T11:43:01.544 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:43:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 msgr2=0x7fd10801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 0x7fd10801abc0 crc :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 msgr2=0x7fd1201a7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7fd118006c50 tx=0x7fd11800bd20).stop 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 shutdown_connections 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1080186c0 0x7fd10801abc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1201a7060 0x7fd1201a7460 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd12008a8a0 0x7fd12008aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 --2- 172.21.2.117:0/3049764932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1200abe50 0x7fd12008a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 >> 172.21.2.117:0/3049764932 conn(0x7fd1200acec0 msgr2=0x7fd1200b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 shutdown_connections 2021-02-10T11:43:01.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.557 7fd127d0a700 1 -- 172.21.2.117:0/3049764932 wait complete. 2021-02-10T11:43:01.555 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 Processor -- start 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 -- start start 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f53200e5240 0x7f5320086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5320085930 con 0x7f5320084c60 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53201b09e0 con 0x7f53200e4e40 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f5327d42700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53200e5cc0 con 0x7f53200e5240 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f532635f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5320084c60 0x7f5320085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50062/0 (socket says 172.21.2.117:50062) 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.979 7f532635f700 1 -- 172.21.2.117:0/3150696247 learned_addr learned my addr 172.21.2.117:0/3150696247 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646530225 0 0) 0x7f5318000f90 con 0x7f5320084c60 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f530c002fe0 con 0x7f5320084c60 2021-02-10T11:43:01.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f532535d700 1 --2- 172.21.2.117:0/3150696247 >> v2:172.21.2.125:3300/0 conn(0x7f53200e5240 0x7f5320086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3371216148 0 0) 0x7f5318001270 con 0x7f5320084c60 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 >> v2:172.21.2.125:3300/0 conn(0x7f53200e5240 msgr2=0x7f5320086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 --2- 172.21.2.117:0/3150696247 >> v2:172.21.2.125:3300/0 conn(0x7f53200e5240 0x7f5320086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 >> v1:172.21.2.125:6789/0 conn(0x7f53200e4e40 legacy=0x7f53201b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320084840 con 0x7f5320084c60 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f53180019c0 con 0x7f5320084c60 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 >> v1:172.21.2.117:6789/0 conn(0x7f5320084c60 legacy=0x7f5320085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 0x7f530c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f530c006770 0x7f530c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f530c009590 0x7f530c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c00c300 con 0x7f530c003a60 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c00c520 con 0x7f530c009590 2021-02-10T11:43:01.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530c00c740 con 0x7f530c006770 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f5325b5e700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 0x7f530c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f532635f700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f530c009590 0x7f530c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.980 7f532535d700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f530c006770 0x7f530c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 -- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f530c006770 msgr2=0x7f530c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f530c006770 0x7f530c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 -- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f530c009590 msgr2=0x7f530c00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f530c009590 0x7f530c00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 -- 172.21.2.117:0/3150696247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c00c9e0 con 0x7f530c003a60 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f532635f700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f530c009590 0x7f530c00bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f532635f700 1 -- 172.21.2.117:0/3150696247 reap_dead start 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5325b5e700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 0x7f530c005ed0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f531c006ad0 tx=0x7f531c04db50).ready entity=mon.0 client_cookie=fcca8d44b8ddcc9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f531c0009b0 con 0x7f530c003a60 2021-02-10T11:43:01.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c002280 con 0x7f530c003a60 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5324b5c700 1 -- 172.21.2.117:0/3150696247 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f531c0027a0 con 0x7f530c003a60 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5327d42700 1 -- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 msgr2=0x7f530c005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.981 7f5327d42700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 0x7f530c005ed0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f531c006ad0 tx=0x7f531c04db50).stop 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.982 7f5327d42700 1 -- 172.21.2.117:0/3150696247 shutdown_connections 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.982 7f5327d42700 1 --2- 172.21.2.117:0/3150696247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f530c003a60 0x7f530c005ed0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.982 7f5327d42700 1 -- 172.21.2.117:0/3150696247 >> 172.21.2.117:0/3150696247 conn(0x7f53200ce4e0 msgr2=0x7f53200e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.982 7f5327d42700 1 -- 172.21.2.117:0/3150696247 shutdown_connections 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.982 7f5327d42700 1 -- 172.21.2.117:0/3150696247 wait complete. 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.996 7f5327d42700 1 Processor -- start 2021-02-10T11:43:01.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.996 7f5327d42700 1 -- start start 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f53200a0aa0 0x7f532019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f532010ffa0 0x7f53201103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53201109c0 con 0x7f5320090ae0 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53200a0ea0 con 0x7f532010ffa0 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f5327d42700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53201a0360 con 0x7f53200a0aa0 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49824/0 (socket says 172.21.2.117:49824) 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 -- 172.21.2.117:0/3310256822 learned_addr learned my addr 172.21.2.117:0/3310256822 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 -- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f53200a0aa0 msgr2=0x7f532019fd40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f53200a0aa0 0x7f532019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 -- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f532010ffa0 msgr2=0x7f53201103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f532010ffa0 0x7f53201103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.997 7f532635f700 1 -- 172.21.2.117:0/3310256822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5320110d00 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f532635f700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f5318001500 tx=0x7f531800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5318004930 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5318000fc0 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5318007020 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f5327d42700 1 -- 172.21.2.117:0/3310256822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5320110d00 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.998 7f5327d42700 1 -- 172.21.2.117:0/3310256822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5320110d00 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.999 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f531800add0 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.999 7f53127fc700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 0x7f530801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.999 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f53180328b0 con 0x7f5320090ae0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:01.999 7f5325b5e700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 0x7f530801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.000 7f5325b5e700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 0x7f530801abc0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:02.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.029 7f5327d42700 1 -- 172.21.2.117:0/3310256822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5320091e50 con 0x7f5320090ae0 2021-02-10T11:43:02.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.032 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5318004720 con 0x7f5320090ae0 2021-02-10T11:43:02.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.158 7f5327d42700 1 -- 172.21.2.117:0/3310256822 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5320091e50 con 0x7f5320090ae0 2021-02-10T11:43:02.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.554 7f53127fc700 1 -- 172.21.2.117:0/3310256822 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 173+0+0 (secure 0 0 0) 0x7f531801f830 con 0x7f5320090ae0 2021-02-10T11:43:02.565 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.577 7f5327d42700 1 -- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 msgr2=0x7f530801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.577 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 0x7f530801abc0 crc :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.577 7f5327d42700 1 -- 172.21.2.117:0/3310256822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 msgr2=0x7f5320090ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.577 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f5318001500 tx=0x7f531800ac90).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 -- 172.21.2.117:0/3310256822 shutdown_connections 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53080186c0 0x7f530801abc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5320090ae0 0x7f5320090ee0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f532010ffa0 0x7f53201103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 --2- 172.21.2.117:0/3310256822 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f53200a0aa0 0x7f532019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:02.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 -- 172.21.2.117:0/3310256822 >> 172.21.2.117:0/3310256822 conn(0x7f53200a5e90 msgr2=0x7f53200b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:02.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 -- 172.21.2.117:0/3310256822 shutdown_connections 2021-02-10T11:43:02.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:02.578 7f5327d42700 1 -- 172.21.2.117:0/3310256822 wait complete. 2021-02-10T11:43:02.576 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:43:02.576 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:43:02.576 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:43:02.576 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:02.576 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:43:02.576 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:43:02.576 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:43:02.577 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:43:02.582 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:43:02.582 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:43:02.583 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:43:02.583 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:43:02.624 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:43:02.625 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:02.625 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:43:02.625 DEBUG:teuthology.parallel:result is None 2021-02-10T11:43:02.626 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:43:03.029 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:43:03.031 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.044 7f39b5627700 1 Processor -- start 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.044 7f39b5627700 1 -- start start 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39b5627700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f39b01b07f0 0x7f39b00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39b5627700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f39b00e8d80 0x7f39b00e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39b5627700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00e77f0 con 0x7f39b00e6170 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39b5627700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00e5ce0 con 0x7f39b01b07f0 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39b5627700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b00e54b0 con 0x7f39b00e8d80 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f39b01b07f0 0x7f39b00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f39b01b07f0 0x7f39b00e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49842/0 (socket says 172.21.2.117:49842) 2021-02-10T11:43:03.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 -- 172.21.2.117:0/3915118457 learned_addr learned my addr 172.21.2.117:0/3915118457 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39afac0700 1 --2- 172.21.2.117:0/3915118457 >> v2:172.21.2.125:3300/0 conn(0x7f39b00e8d80 0x7f39b00e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 -- 172.21.2.117:0/3915118457 >> v2:172.21.2.125:3300/0 conn(0x7f39b00e8d80 msgr2=0x7f39b00e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 --2- 172.21.2.117:0/3915118457 >> v2:172.21.2.125:3300/0 conn(0x7f39b00e8d80 0x7f39b00e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 -- 172.21.2.117:0/3915118457 >> v1:172.21.2.125:6789/0 conn(0x7f39b00e6170 legacy=0x7f39b00e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 -- 172.21.2.117:0/3915118457 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00e9180 con 0x7f39b01b07f0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39af2bf700 1 --2- 172.21.2.117:0/3915118457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b01b07f0 0x7f39b00e4e90 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f39a0006700 tx=0x7f39a0009b00).ready entity=mon.0 client_cookie=1c2867eb50dbeb1c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39ae2bd700 1 -- 172.21.2.117:0/3915118457 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39a000c0c0 con 0x7f39b01b07f0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39ae2bd700 1 -- 172.21.2.117:0/3915118457 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000c400 con 0x7f39b01b07f0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.045 7f39ae2bd700 1 -- 172.21.2.117:0/3915118457 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39a000c8e0 con 0x7f39b01b07f0 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 -- 172.21.2.117:0/3915118457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b01b07f0 msgr2=0x7f39b00e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 --2- 172.21.2.117:0/3915118457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b01b07f0 0x7f39b00e4e90 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0x7f39a0006700 tx=0x7f39a0009b00).stop 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 -- 172.21.2.117:0/3915118457 shutdown_connections 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 --2- 172.21.2.117:0/3915118457 >> v2:172.21.2.125:3300/0 conn(0x7f39b00e8d80 0x7f39b00e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 --2- 172.21.2.117:0/3915118457 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b01b07f0 0x7f39b00e4e90 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 -- 172.21.2.117:0/3915118457 >> 172.21.2.117:0/3915118457 conn(0x7f39b00ce500 msgr2=0x7f39b00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:03.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 -- 172.21.2.117:0/3915118457 shutdown_connections 2021-02-10T11:43:03.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.046 7f39b5627700 1 -- 172.21.2.117:0/3915118457 wait complete. 2021-02-10T11:43:03.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.061 7f39b5627700 1 Processor -- start 2021-02-10T11:43:03.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.061 7f39b5627700 1 -- start start 2021-02-10T11:43:03.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.061 7f39b5627700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39b01a4140 0x7f39b00a7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.061 7f39b5627700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39b00a7d20 0x7f39b00a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39b5627700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39b5627700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0196080 con 0x7f39b00a8740 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39b5627700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00ba580 con 0x7f39b01a4140 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39b5627700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b00ba7a0 con 0x7f39b00a7d20 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49848/0 (socket says 172.21.2.117:49848) 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 -- 172.21.2.117:0/1963881947 learned_addr learned my addr 172.21.2.117:0/1963881947 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39aeabe700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39b00a7d20 0x7f39b00a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39af2bf700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39b01a4140 0x7f39b00a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 -- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39b00a7d20 msgr2=0x7f39b00a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39b00a7d20 0x7f39b00a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 -- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39b01a4140 msgr2=0x7f39b00a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39b01a4140 0x7f39b00a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 -- 172.21.2.117:0/1963881947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00ba9c0 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39afac0700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f39a800b5e0 tx=0x7f39a800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.062 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39a800bd70 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8002030 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39b5627700 1 -- 172.21.2.117:0/1963881947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b00ba9c0 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f39a8002550 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39b5627700 1 -- 172.21.2.117:0/1963881947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b00baf50 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f39a8002210 con 0x7f39b00a8740 2021-02-10T11:43:03.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39a7fff700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 0x7f398801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:03.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f39a80322a0 con 0x7f39b00a8740 2021-02-10T11:43:03.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.063 7f39af2bf700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 0x7f398801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:03.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.064 7f39af2bf700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 0x7f398801abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:03.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.094 7f39b5627700 1 -- 172.21.2.117:0/1963881947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b010c290 con 0x7f39b00a8740 2021-02-10T11:43:03.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.096 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f39a802df00 con 0x7f39b00a8740 2021-02-10T11:43:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.235 7f39b5627700 1 -- 172.21.2.117:0/1963881947 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f39b00e77f0 con 0x7f39b00a8740 2021-02-10T11:43:03.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.560 7f39a7fff700 1 -- 172.21.2.117:0/1963881947 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v60) v1 ==== 97+0+0 (secure 0 0 0) 0x7f39a802df00 con 0x7f39b00a8740 2021-02-10T11:43:03.592 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:43:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 -- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 msgr2=0x7f398801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 0x7f398801abc0 crc :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 -- 172.21.2.117:0/1963881947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 msgr2=0x7f39b00b9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:03.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f39a800b5e0 tx=0x7f39a800b980).stop 2021-02-10T11:43:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 -- 172.21.2.117:0/1963881947 shutdown_connections 2021-02-10T11:43:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f39880186c0 0x7f398801abc0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f39b00a8740 0x7f39b00b9f60 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f39b00a7d20 0x7f39b00a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 --2- 172.21.2.117:0/1963881947 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f39b01a4140 0x7f39b00a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.604 7f39b5627700 1 -- 172.21.2.117:0/1963881947 >> 172.21.2.117:0/1963881947 conn(0x7f39b00b9400 msgr2=0x7f39b00aee20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.605 7f39b5627700 1 -- 172.21.2.117:0/1963881947 shutdown_connections 2021-02-10T11:43:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:03.605 7f39b5627700 1 -- 172.21.2.117:0/1963881947 wait complete. 2021-02-10T11:43:03.602 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:43:03.603 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.026 7fa631412700 1 Processor -- start 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa631412700 1 -- start start 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa631412700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c1b6960 0x7fa62c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa631412700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c0ede60 con 0x7fa62c0d18e0 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa631412700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c0eb9c0 con 0x7fa62c0ef3f0 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa631412700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0d1ce0 con 0x7fa62c1b6960 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa62b07d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa62c0d18e0 0x7fa62c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50104/0 (socket says 172.21.2.117:50104) 2021-02-10T11:43:04.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.027 7fa62b07d700 1 -- 172.21.2.117:0/580617414 learned_addr learned my addr 172.21.2.117:0/580617414 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031891548 0 0) 0x7fa614001180 con 0x7fa62c0d18e0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa618002fe0 con 0x7fa62c0d18e0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62b87e700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c1b6960 0x7fa62c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1478248209 0 0) 0x7fa614001180 con 0x7fa62c0d18e0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c1b6960 msgr2=0x7fa62c1b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c1b6960 0x7fa62c1b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 >> v1:172.21.2.125:6789/0 conn(0x7fa62c0ef3f0 legacy=0x7fa62c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c1bb3b0 con 0x7fa62c0d18e0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa614001680 con 0x7fa62c0d18e0 2021-02-10T11:43:04.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 >> v1:172.21.2.117:6789/0 conn(0x7fa62c0d18e0 legacy=0x7fa62c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa618003aa0 0x7fa618005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa618006750 0x7fa618008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6180094f0 0x7fa61800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61800c2c0 con 0x7fa6180094f0 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61800c4e0 con 0x7fa618006750 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.028 7fa62a07b700 1 -- 172.21.2.117:0/580617414 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61800c700 con 0x7fa618003aa0 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6180094f0 0x7fa61800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62a87c700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa618006750 0x7fa618008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b07d700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa618003aa0 0x7fa618005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 -- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa618003aa0 msgr2=0x7fa618005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa618003aa0 0x7fa618005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 -- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa618006750 msgr2=0x7fa618008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa618006750 0x7fa618008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 -- 172.21.2.117:0/580617414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618050940 con 0x7fa6180094f0 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b07d700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa618003aa0 0x7fa618005eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62a87c700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa618006750 0x7fa618008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6180094f0 0x7fa61800ba10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fa624009b10 tx=0x7fa624009eb0).ready entity=mon.0 client_cookie=6b3798ee7155f6aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62400b510 con 0x7fa6180094f0 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa624003030 con 0x7fa6180094f0 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62a07b700 1 -- 172.21.2.117:0/580617414 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62400b790 con 0x7fa6180094f0 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa631412700 1 -- 172.21.2.117:0/580617414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6180094f0 msgr2=0x7fa61800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa631412700 1 --2- 172.21.2.117:0/580617414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6180094f0 0x7fa61800ba10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7fa624009b10 tx=0x7fa624009eb0).stop 2021-02-10T11:43:04.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.029 7fa62b87e700 1 -- 172.21.2.117:0/580617414 reap_dead start 2021-02-10T11:43:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.030 7fa631412700 1 -- 172.21.2.117:0/580617414 shutdown_connections 2021-02-10T11:43:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.030 7fa631412700 1 -- 172.21.2.117:0/580617414 >> 172.21.2.117:0/580617414 conn(0x7fa62c0ced60 msgr2=0x7fa62c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.030 7fa631412700 1 -- 172.21.2.117:0/580617414 shutdown_connections 2021-02-10T11:43:04.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.030 7fa631412700 1 -- 172.21.2.117:0/580617414 wait complete. 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 Processor -- start 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 -- start start 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 0x7fa62c08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c0b5da0 0x7fa62c1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0b61a0 con 0x7fa62c1a3480 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c08c8b0 con 0x7fa62c0b59a0 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa631412700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0ba1c0 con 0x7fa62c0b5da0 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.045 7fa62b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49870/0 (socket says 172.21.2.117:49870) 2021-02-10T11:43:04.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 -- 172.21.2.117:0/1811267521 learned_addr learned my addr 172.21.2.117:0/1811267521 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 -- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c0b5da0 msgr2=0x7fa62c1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c0b5da0 0x7fa62c1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 -- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 msgr2=0x7fa62c08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b07d700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 0x7fa62c08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 0x7fa62c08c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 -- 172.21.2.117:0/1811267521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c1a3880 con 0x7fa62c1a3480 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b07d700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 0x7fa62c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa62b87e700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fa6240099f0 tx=0x7fa62400bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa62400bdc0 con 0x7fa62c1a3480 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa624007070 con 0x7fa62c1a3480 2021-02-10T11:43:04.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa624007590 con 0x7fa62c1a3480 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa631412700 1 -- 172.21.2.117:0/1811267521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c1a3880 con 0x7fa62c1a3480 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.046 7fa631412700 1 -- 172.21.2.117:0/1811267521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c1a3880 con 0x7fa62c1a3480 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.047 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa6240056b0 con 0x7fa62c1a3480 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.047 7fa623fff700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 0x7fa61001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.047 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fa624032990 con 0x7fa62c1a3480 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.047 7fa62b07d700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 0x7fa61001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.047 7fa62b07d700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 0x7fa61001abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.077 7fa631412700 1 -- 172.21.2.117:0/1811267521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c0ede60 con 0x7fa62c1a3480 2021-02-10T11:43:04.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.080 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa624007070 con 0x7fa62c1a3480 2021-02-10T11:43:04.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.212 7fa631412700 1 -- 172.21.2.117:0/1811267521 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa62c0ede60 con 0x7fa62c1a3480 2021-02-10T11:43:04.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.212 7fa623fff700 1 -- 172.21.2.117:0/1811267521 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fa6240056b0 con 0x7fa62c1a3480 2021-02-10T11:43:04.202 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:04.202 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:04.202 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:43:04.202 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.203 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.204 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.205 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.206 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.207 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.208 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:43:04.209 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:43:04.210 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:43:04.211 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.211 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.211 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.211 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.211 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:43:04.212 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:04.213 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.214 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.215 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.216 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:43:04.217 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:04.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 -- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 msgr2=0x7fa61001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 0x7fa61001abc0 crc :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 -- 172.21.2.117:0/1811267521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 msgr2=0x7fa62c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7fa6240099f0 tx=0x7fa62400bc80).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 -- 172.21.2.117:0/1811267521 shutdown_connections 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6100186c0 0x7fa61001abc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa62c1a3480 0x7fa62c0b2590 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa62c0b5da0 0x7fa62c1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 --2- 172.21.2.117:0/1811267521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa62c0b59a0 0x7fa62c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 -- 172.21.2.117:0/1811267521 >> 172.21.2.117:0/1811267521 conn(0x7fa62c0bb5e0 msgr2=0x7fa62c090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.214 7fa631412700 1 -- 172.21.2.117:0/1811267521 shutdown_connections 2021-02-10T11:43:04.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.215 7fa631412700 1 -- 172.21.2.117:0/1811267521 wait complete. 2021-02-10T11:43:04.220 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.639 7f314ab79700 1 Processor -- start 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.639 7f314ab79700 1 -- start start 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.640 7f314ab79700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 0x7f31440eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.640 7f314ab79700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f31440eb7f0 0x7f31441b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.640 7f314ab79700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31440ede60 con 0x7f31440d18e0 2021-02-10T11:43:04.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.640 7f314ab79700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31440ef7f0 con 0x7f31440eb7f0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.640 7f314ab79700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31440d1ce0 con 0x7f31440ef3f0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3143fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 0x7f31440eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3143fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 0x7f31440eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47554/0 (socket says 172.21.2.117:47554) 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3143fff700 1 -- 172.21.2.117:0/913332996 learned_addr learned my addr 172.21.2.117:0/913332996 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 --2- 172.21.2.117:0/913332996 >> v2:172.21.2.125:3300/0 conn(0x7f31440eb7f0 0x7f31441b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 -- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 msgr2=0x7f31440eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 --2- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 0x7f31440eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 -- 172.21.2.117:0/913332996 >> v1:172.21.2.125:6789/0 conn(0x7f31440d18e0 legacy=0x7f31440ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f31437fe700 1 -- 172.21.2.117:0/913332996 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432493492 0 0) 0x7f312c000c60 con 0x7f31440d18e0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 -- 172.21.2.117:0/913332996 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31441b8f90 con 0x7f31440eb7f0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.641 7f3149196700 1 --2- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31440eb7f0 0x7f31441b7960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f313c006ad0 tx=0x7f313c009ad0).ready entity=mon.1 client_cookie=d86444ad5c24111b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f31437fe700 1 -- 172.21.2.117:0/913332996 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f313c0022e0 con 0x7f31440eb7f0 2021-02-10T11:43:04.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f31437fe700 1 -- 172.21.2.117:0/913332996 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c006410 con 0x7f31440eb7f0 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f31437fe700 1 -- 172.21.2.117:0/913332996 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f313c002620 con 0x7f31440eb7f0 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 -- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31440eb7f0 msgr2=0x7f31441b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 --2- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31440eb7f0 0x7f31441b7960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f313c006ad0 tx=0x7f313c009ad0).stop 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 -- 172.21.2.117:0/913332996 shutdown_connections 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 --2- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31440eb7f0 0x7f31441b7960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 --2- 172.21.2.117:0/913332996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31440ef3f0 0x7f31440eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 -- 172.21.2.117:0/913332996 >> 172.21.2.117:0/913332996 conn(0x7f31440ced60 msgr2=0x7f31440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 -- 172.21.2.117:0/913332996 shutdown_connections 2021-02-10T11:43:04.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.642 7f314ab79700 1 -- 172.21.2.117:0/913332996 wait complete. 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.657 7f314ab79700 1 Processor -- start 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.657 7f314ab79700 1 -- start start 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3144091d40 0x7f31440e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 0x7f31441b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 0x7f3144198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144197cd0 con 0x7f3144091d40 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144092140 con 0x7f3144198250 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f314ab79700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31441b2ed0 con 0x7f31441b24b0 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3149196700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 0x7f3144198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3149196700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 0x7f3144198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54266/0 (socket says 172.21.2.117:54266) 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3149196700 1 -- 172.21.2.117:0/2534681524 learned_addr learned my addr 172.21.2.117:0/2534681524 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3148995700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3144091d40 0x7f31440e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 0x7f31441b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 -- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 msgr2=0x7f3144198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 0x7f3144198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 -- 172.21.2.117:0/2534681524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3144091d40 msgr2=0x7f31440e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3144091d40 0x7f31440e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 -- 172.21.2.117:0/2534681524 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3144198c70 con 0x7f31441b24b0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.658 7f3143fff700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 0x7f31441b28b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f313400b8a0 tx=0x7f313400bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3134006740 con 0x7f31441b24b0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313400c2f0 con 0x7f31441b24b0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f314ab79700 1 -- 172.21.2.117:0/2534681524 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144198c70 con 0x7f31441b24b0 2021-02-10T11:43:04.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f313400c780 con 0x7f31441b24b0 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f314ab79700 1 -- 172.21.2.117:0/2534681524 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3144198c70 con 0x7f31441b24b0 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3134001010 con 0x7f31441b24b0 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 0x7f312801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f3134032860 con 0x7f31441b24b0 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.659 7f3148995700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 0x7f312801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:04.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.660 7f3148995700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 0x7f312801abd0 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:04.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.690 7f314ab79700 1 -- 172.21.2.117:0/2534681524 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3144087290 con 0x7f31441b24b0 2021-02-10T11:43:04.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.693 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f313400bd80 con 0x7f31441b24b0 2021-02-10T11:43:04.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.823 7f314ab79700 1 -- 172.21.2.117:0/2534681524 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31440ede60 con 0x7f31441b24b0 2021-02-10T11:43:04.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.824 7f31417fa700 1 -- 172.21.2.117:0/2534681524 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+11378 (secure 0 0 0) 0x7f313401f760 con 0x7f31441b24b0 2021-02-10T11:43:04.815 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:04.815 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":60,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:03.553935","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:04.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 msgr2=0x7f312801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 0x7f312801abd0 crc :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 msgr2=0x7f31441b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:04.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 0x7f31441b28b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f313400b8a0 tx=0x7f313400bc40).stop 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 shutdown_connections 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3144198250 0x7f3144198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31441b24b0 0x7f31441b28b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31280186d0 0x7f312801abd0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 --2- 172.21.2.117:0/2534681524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3144091d40 0x7f31440e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 >> 172.21.2.117:0/2534681524 conn(0x7f31441a3620 msgr2=0x7f31440dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 shutdown_connections 2021-02-10T11:43:04.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:04.826 7f314ab79700 1 -- 172.21.2.117:0/2534681524 wait complete. 2021-02-10T11:43:04.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:43:04.825 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 Processor -- start 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 -- start start 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f09a80d18f0 0x7f09a80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80eb040 0x7f09a80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a80ede60 con 0x7f09a80ee3e0 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a80ebad0 con 0x7f09a80eb040 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09afc08700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a80d1cf0 con 0x7f09a80d18f0 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09ad223700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80eb040 0x7f09a80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09ad223700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09a80eb040 0x7f09a80eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49886/0 (socket says 172.21.2.117:49886) 2021-02-10T11:43:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09ad223700 1 -- 172.21.2.117:0/110077374 learned_addr learned my addr 172.21.2.117:0/110077374 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09ae225700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09a80ee3e0 0x7f09a80ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50136/0 (socket says 172.21.2.117:50136) 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.249 7f09ada24700 1 --2- 172.21.2.117:0/110077374 >> v2:172.21.2.125:3300/0 conn(0x7f09a80d18f0 0x7f09a80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09ad223700 1 -- 172.21.2.117:0/110077374 >> v2:172.21.2.125:3300/0 conn(0x7f09a80d18f0 msgr2=0x7f09a80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09ad223700 1 --2- 172.21.2.117:0/110077374 >> v2:172.21.2.125:3300/0 conn(0x7f09a80d18f0 0x7f09a80eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09ad223700 1 -- 172.21.2.117:0/110077374 >> v1:172.21.2.117:6789/0 conn(0x7f09a80ee3e0 legacy=0x7f09a80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09ad223700 1 -- 172.21.2.117:0/110077374 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80eefd0 con 0x7f09a80eb040 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09ad223700 1 --2- 172.21.2.117:0/110077374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80eb040 0x7f09a80eb440 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f0998006700 tx=0x7f0998009b80).ready entity=mon.0 client_cookie=bf6ff98e2314e040 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09aca22700 1 -- 172.21.2.117:0/110077374 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099800c0c0 con 0x7f09a80eb040 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09aca22700 1 -- 172.21.2.117:0/110077374 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099800c340 con 0x7f09a80eb040 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09aca22700 1 -- 172.21.2.117:0/110077374 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099800c820 con 0x7f09a80eb040 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09afc08700 1 -- 172.21.2.117:0/110077374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80eb040 msgr2=0x7f09a80eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.250 7f09afc08700 1 --2- 172.21.2.117:0/110077374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80eb040 0x7f09a80eb440 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f0998006700 tx=0x7f0998009b80).stop 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 -- 172.21.2.117:0/110077374 shutdown_connections 2021-02-10T11:43:05.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 --2- 172.21.2.117:0/110077374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a80eb040 0x7f09a80eb440 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 --2- 172.21.2.117:0/110077374 >> v2:172.21.2.125:3300/0 conn(0x7f09a80d18f0 0x7f09a80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 -- 172.21.2.117:0/110077374 >> 172.21.2.117:0/110077374 conn(0x7f09a80cf060 msgr2=0x7f09a80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 -- 172.21.2.117:0/110077374 shutdown_connections 2021-02-10T11:43:05.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.251 7f09afc08700 1 -- 172.21.2.117:0/110077374 wait complete. 2021-02-10T11:43:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 Processor -- start 2021-02-10T11:43:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 -- start start 2021-02-10T11:43:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a80a8eb0 0x7f09a80a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a80a98f0 0x7f09a807ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80a9cf0 con 0x7f09a809f5f0 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a809f9f0 con 0x7f09a80a8eb0 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09afc08700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a809fc10 con 0x7f09a80a98f0 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09ad223700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09ad223700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49892/0 (socket says 172.21.2.117:49892) 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.266 7f09ad223700 1 -- 172.21.2.117:0/2440491762 learned_addr learned my addr 172.21.2.117:0/2440491762 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 -- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a80a98f0 msgr2=0x7f09a807ecf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a80a98f0 0x7f09a807ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 -- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a80a8eb0 msgr2=0x7f09a80a8600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a80a8eb0 0x7f09a80a8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 -- 172.21.2.117:0/2440491762 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a807f310 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09ad223700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f099800b5e0 tx=0x7f099800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f099800bd70 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998002030 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0998002550 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09afc08700 1 -- 172.21.2.117:0/2440491762 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a807f310 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.267 7f09afc08700 1 -- 172.21.2.117:0/2440491762 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a807f310 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.268 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0998000c90 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.268 7f099e7fc700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.268 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f09980325f0 con 0x7f09a809f5f0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.268 7f09ada24700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.269 7f09ada24700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.299 7f09afc08700 1 -- 172.21.2.117:0/2440491762 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a809fc10 con 0x7f09a809f5f0 2021-02-10T11:43:05.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.302 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f099801f510 con 0x7f09a809f5f0 2021-02-10T11:43:05.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.428 7f09afc08700 1 -- 172.21.2.117:0/2440491762 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f09a809fc10 con 0x7f09a809f5f0 2021-02-10T11:43:05.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.429 7f099e7fc700 1 -- 172.21.2.117:0/2440491762 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v60) v1 ==== 92+0+10 (secure 0 0 0) 0x7f099800bd70 con 0x7f09a809f5f0 2021-02-10T11:43:05.420 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 msgr2=0x7f099001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 crc :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 msgr2=0x7f09a80a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0x7f099800b5e0 tx=0x7f099800b980).stop 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 shutdown_connections 2021-02-10T11:43:05.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09900186c0 0x7f099001abc0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09a809f5f0 0x7f09a80a92d0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09a80a98f0 0x7f09a807ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 --2- 172.21.2.117:0/2440491762 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09a80a8eb0 0x7f09a80a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 >> 172.21.2.117:0/2440491762 conn(0x7f09a80a71e0 msgr2=0x7f09a80a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 shutdown_connections 2021-02-10T11:43:05.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.431 7f09afc08700 1 -- 172.21.2.117:0/2440491762 wait complete. 2021-02-10T11:43:05.430 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:43:05.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.849 7efe15ace700 1 Processor -- start 2021-02-10T11:43:05.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.849 7efe15ace700 1 -- start start 2021-02-10T11:43:05.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe15ace700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 0x7efe100eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe15ace700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efe100eb7f0 0x7efe101b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe15ace700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe100ede60 con 0x7efe100d18e0 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe15ace700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe100ef7f0 con 0x7efe100eb7f0 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe15ace700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe100d1ce0 con 0x7efe100ef3f0 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efe100eb7f0 0x7efe101b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efe100d18e0 0x7efe100ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50148/0 (socket says 172.21.2.117:50148) 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0f7fe700 1 -- 172.21.2.117:0/2207072707 learned_addr learned my addr 172.21.2.117:0/2207072707 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 --2- 172.21.2.117:0/2207072707 >> v2:172.21.2.117:3300/0 conn(0x7efe100eb7f0 0x7efe101b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49900/0 (socket says 172.21.2.117:49900) 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0effd700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 0x7efe100eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 -- 172.21.2.117:0/2207072707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 msgr2=0x7efe100eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 0x7efe100eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 -- 172.21.2.117:0/2207072707 >> v1:172.21.2.117:6789/0 conn(0x7efe100d18e0 legacy=0x7efe100ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0ffff700 1 -- 172.21.2.117:0/2207072707 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe101b8f90 con 0x7efe100eb7f0 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.850 7efe0effd700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 0x7efe100eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe0ffff700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe100eb7f0 0x7efe101b7960 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7efe08006e30 tx=0x7efe08009e60).ready entity=mon.0 client_cookie=92eb05f43bfe386c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe0e7fc700 1 -- 172.21.2.117:0/2207072707 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe08003030 con 0x7efe100eb7f0 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe0e7fc700 1 -- 172.21.2.117:0/2207072707 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0800b300 con 0x7efe100eb7f0 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe0e7fc700 1 -- 172.21.2.117:0/2207072707 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe0800b820 con 0x7efe100eb7f0 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 -- 172.21.2.117:0/2207072707 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe100eb7f0 msgr2=0x7efe101b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe100eb7f0 0x7efe101b7960 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7efe08006e30 tx=0x7efe08009e60).stop 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 -- 172.21.2.117:0/2207072707 shutdown_connections 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe100eb7f0 0x7efe101b7960 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 --2- 172.21.2.117:0/2207072707 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe100ef3f0 0x7efe100eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.851 7efe15ace700 1 -- 172.21.2.117:0/2207072707 >> 172.21.2.117:0/2207072707 conn(0x7efe100ced60 msgr2=0x7efe100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.852 7efe15ace700 1 -- 172.21.2.117:0/2207072707 shutdown_connections 2021-02-10T11:43:05.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.852 7efe15ace700 1 -- 172.21.2.117:0/2207072707 wait complete. 2021-02-10T11:43:05.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.866 7efe15ace700 1 Processor -- start 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.866 7efe15ace700 1 -- start start 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe10091d40 0x7efe100e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe101b24b0 0x7efe101b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe10197cd0 con 0x7efe10198250 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe10092140 con 0x7efe101b24b0 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe15ace700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe101b2ed0 con 0x7efe10091d40 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49906/0 (socket says 172.21.2.117:49906) 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 -- 172.21.2.117:0/1318089391 learned_addr learned my addr 172.21.2.117:0/1318089391 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 -- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe10091d40 msgr2=0x7efe100e71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe10091d40 0x7efe100e71e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 -- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe101b24b0 msgr2=0x7efe101b28b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe101b24b0 0x7efe101b28b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.867 7efe0ffff700 1 -- 172.21.2.117:0/1318089391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe10198c70 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efe0ffff700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7efe080068b0 tx=0x7efe0800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe080096c0 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0800be60 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efe08001700 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efe15ace700 1 -- 172.21.2.117:0/1318089391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe10198c70 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.868 7efe15ace700 1 -- 172.21.2.117:0/1318089391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe10196c90 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.869 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efe080096c0 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.869 7efdf7fff700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 0x7efdf001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.869 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5100+0+0 (secure 0 0 0) 0x7efe080325f0 con 0x7efe10198250 2021-02-10T11:43:05.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.869 7efe0f7fe700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 0x7efdf001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:05.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.870 7efe0f7fe700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 0x7efdf001abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:05.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.900 7efe15ace700 1 -- 172.21.2.117:0/1318089391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe10087290 con 0x7efe10198250 2021-02-10T11:43:05.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:05.902 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efe08003000 con 0x7efe10198250 2021-02-10T11:43:06.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.026 7efe15ace700 1 -- 172.21.2.117:0/1318089391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efe100ede60 con 0x7efe10198250 2021-02-10T11:43:06.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.026 7efdf7fff700 1 -- 172.21.2.117:0/1318089391 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v60) v1 ==== 84+0+64 (secure 0 0 0) 0x7efe080096c0 con 0x7efe10198250 2021-02-10T11:43:06.017 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:06.017 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:43:06.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:06.018 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:43:06.018 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:43:06.018 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:06.018 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 -- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 msgr2=0x7efdf001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 0x7efdf001abc0 crc :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 -- 172.21.2.117:0/1318089391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 msgr2=0x7efe10198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7efe080068b0 tx=0x7efe0800bd20).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 -- 172.21.2.117:0/1318089391 shutdown_connections 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdf00186c0 0x7efdf001abc0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efe10198250 0x7efe10198650 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efe101b24b0 0x7efe101b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 --2- 172.21.2.117:0/1318089391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efe10091d40 0x7efe100e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.028 7efe15ace700 1 -- 172.21.2.117:0/1318089391 >> 172.21.2.117:0/1318089391 conn(0x7efe101a3620 msgr2=0x7efe100dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:06.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.029 7efe15ace700 1 -- 172.21.2.117:0/1318089391 shutdown_connections 2021-02-10T11:43:06.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.029 7efe15ace700 1 -- 172.21.2.117:0/1318089391 wait complete. 2021-02-10T11:43:06.027 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:43:06.028 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.451 7fc242fee700 1 Processor -- start 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.451 7fc242fee700 1 -- start start 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc242fee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c0d18f0 0x7fc23c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc242fee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc242fee700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c0ede60 con 0x7fc23c0eb040 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc242fee700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c088020 con 0x7fc23c0ef3f0 2021-02-10T11:43:06.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc242fee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0d1d30 con 0x7fc23c0d18f0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49912/0 (socket says 172.21.2.117:49912) 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 -- 172.21.2.117:0/2115417111 learned_addr learned my addr 172.21.2.117:0/2115417111 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 -- 172.21.2.117:0/2115417111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c0d18f0 msgr2=0x7fc23c0eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 --2- 172.21.2.117:0/2115417111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c0d18f0 0x7fc23c0eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 -- 172.21.2.117:0/2115417111 >> v1:172.21.2.125:6789/0 conn(0x7fc23c0eb040 legacy=0x7fc23c0eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 -- 172.21.2.117:0/2115417111 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c1b8fd0 con 0x7fc23c0ef3f0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc24160b700 1 --2- 172.21.2.117:0/2115417111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fc234006f00 tx=0x7fc234009f10).ready entity=mon.0 client_cookie=f511d8d9f9240a88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.452 7fc23b7fe700 1 -- 172.21.2.117:0/2115417111 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc23400b3b0 con 0x7fc23c0ef3f0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc23b7fe700 1 -- 172.21.2.117:0/2115417111 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc234000d30 con 0x7fc23c0ef3f0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc23b7fe700 1 -- 172.21.2.117:0/2115417111 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc23400b820 con 0x7fc23c0ef3f0 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 -- 172.21.2.117:0/2115417111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c0ef3f0 msgr2=0x7fc23c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 --2- 172.21.2.117:0/2115417111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fc234006f00 tx=0x7fc234009f10).stop 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 -- 172.21.2.117:0/2115417111 shutdown_connections 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 --2- 172.21.2.117:0/2115417111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c0ef3f0 0x7fc23c1b79a0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 --2- 172.21.2.117:0/2115417111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c0d18f0 0x7fc23c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 -- 172.21.2.117:0/2115417111 >> 172.21.2.117:0/2115417111 conn(0x7fc23c0cf060 msgr2=0x7fc23c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 -- 172.21.2.117:0/2115417111 shutdown_connections 2021-02-10T11:43:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.453 7fc242fee700 1 -- 172.21.2.117:0/2115417111 wait complete. 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 Processor -- start 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 -- start start 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 0x7fc23c1a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 0x7fc23c08cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c094280 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c093fe0 con 0x7fc23c08c850 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.468 7fc242fee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0ace00 con 0x7fc23c093be0 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49916/0 (socket says 172.21.2.117:49916) 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 -- 172.21.2.117:0/3054543862 learned_addr learned my addr 172.21.2.117:0/3054543862 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 -- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 msgr2=0x7fc23c1a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc240e0a700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 0x7fc23c1a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 0x7fc23c1a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 -- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 msgr2=0x7fc23c08cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc24160b700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 0x7fc23c08cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 0x7fc23c08cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 -- 172.21.2.117:0/3054543862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c094800 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc240e0a700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 0x7fc23c1a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc24160b700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 0x7fc23c08cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.469 7fc23bfff700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fc22c00b5b0 tx=0x7fc22c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c00bd40 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c000c60 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc22c006210 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc242fee700 1 -- 172.21.2.117:0/3054543862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c094800 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc242fee700 1 -- 172.21.2.117:0/3054543862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c094d90 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc22c006020 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 0x7fc22001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.470 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc22c0323a0 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.471 7fc240e0a700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 0x7fc22001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:06.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.471 7fc240e0a700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 0x7fc22001abc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:06.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.501 7fc242fee700 1 -- 172.21.2.117:0/3054543862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c0ede60 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.504 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc22c0009b0 con 0x7fc23c1a7ed0 2021-02-10T11:43:06.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:06.630 7fc242fee700 1 -- 172.21.2.117:0/3054543862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fc23c0ede60 con 0x7fc23c1a7ed0 2021-02-10T11:43:07.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.050 7fc2397fa700 1 -- 172.21.2.117:0/3054543862 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v61) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc22c006020 con 0x7fc23c1a7ed0 2021-02-10T11:43:07.085 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:43:07.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.096 7fc242fee700 1 -- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 msgr2=0x7fc22001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.096 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 0x7fc22001abc0 crc :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.096 7fc242fee700 1 -- 172.21.2.117:0/3054543862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 msgr2=0x7fc23c08c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.096 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7fc22c00b5b0 tx=0x7fc22c00b950).stop 2021-02-10T11:43:07.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 -- 172.21.2.117:0/3054543862 shutdown_connections 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2200186c0 0x7fc22001abc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc23c1a7ed0 0x7fc23c08c230 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc23c08c850 0x7fc23c08cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 --2- 172.21.2.117:0/3054543862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc23c093be0 0x7fc23c1a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 -- 172.21.2.117:0/3054543862 >> 172.21.2.117:0/3054543862 conn(0x7fc23c092d00 msgr2=0x7fc23c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 -- 172.21.2.117:0/3054543862 shutdown_connections 2021-02-10T11:43:07.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.097 7fc242fee700 1 -- 172.21.2.117:0/3054543862 wait complete. 2021-02-10T11:43:07.096 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:43:07.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.519 7f37362c6700 1 Processor -- start 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.519 7f37362c6700 1 -- start start 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37362c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f37300d18f0 0x7f37300eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37362c6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37300eb040 0x7f37300eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37362c6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300eba60 con 0x7f37300ee3e0 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37362c6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37300d1cf0 con 0x7f37300d18f0 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37362c6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37300eefd0 con 0x7f37300eb040 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f372ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f37300d18f0 0x7f37300eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37348e3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f37300ee3e0 0x7f37300ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50174/0 (socket says 172.21.2.117:50174) 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f37348e3700 1 -- 172.21.2.117:0/4119427015 learned_addr learned my addr 172.21.2.117:0/4119427015 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.520 7f372f7fe700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37300eb040 0x7f37300eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372ffff700 1 -- 172.21.2.117:0/4119427015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37300eb040 msgr2=0x7f37300eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372ffff700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37300eb040 0x7f37300eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372ffff700 1 -- 172.21.2.117:0/4119427015 >> v1:172.21.2.117:6789/0 conn(0x7f37300ee3e0 legacy=0x7f37300ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372ffff700 1 -- 172.21.2.117:0/4119427015 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37300ef1f0 con 0x7f37300d18f0 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372ffff700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37300d18f0 0x7f37300eaa20 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f3720006650 tx=0x7f3720009a80).ready entity=mon.0 client_cookie=214b90d04af2f193 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372effd700 1 -- 172.21.2.117:0/4119427015 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37200009b0 con 0x7f37300d18f0 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372effd700 1 -- 172.21.2.117:0/4119427015 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372000c0c0 con 0x7f37300d18f0 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f372effd700 1 -- 172.21.2.117:0/4119427015 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f372000c5e0 con 0x7f37300d18f0 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f37362c6700 1 -- 172.21.2.117:0/4119427015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37300d18f0 msgr2=0x7f37300eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.521 7f37362c6700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37300d18f0 0x7f37300eaa20 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f3720006650 tx=0x7f3720009a80).stop 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 -- 172.21.2.117:0/4119427015 shutdown_connections 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37300eb040 0x7f37300eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 --2- 172.21.2.117:0/4119427015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37300d18f0 0x7f37300eaa20 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 -- 172.21.2.117:0/4119427015 >> 172.21.2.117:0/4119427015 conn(0x7f37300cf060 msgr2=0x7f37300cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 -- 172.21.2.117:0/4119427015 shutdown_connections 2021-02-10T11:43:07.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.522 7f37362c6700 1 -- 172.21.2.117:0/4119427015 wait complete. 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 Processor -- start 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 -- start start 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37301a4180 0x7f37300dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37301a4580 0x7f37300dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730085e50 con 0x7f3730083990 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730097270 con 0x7f37301a4580 2021-02-10T11:43:07.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f37362c6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3730097450 con 0x7f37301a4180 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f372ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f372ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49926/0 (socket says 172.21.2.117:49926) 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.537 7f372ffff700 1 -- 172.21.2.117:0/3042529775 learned_addr learned my addr 172.21.2.117:0/3042529775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 -- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37301a4180 msgr2=0x7f37300dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37301a4180 0x7f37300dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 -- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37301a4580 msgr2=0x7f37300dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37301a4580 0x7f37300dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 -- 172.21.2.117:0/3042529775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373009ed70 con 0x7f3730083990 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372ffff700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f3720009630 tx=0x7f372000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37200093a0 con 0x7f3730083990 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372000bd70 con 0x7f3730083990 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3720005560 con 0x7f3730083990 2021-02-10T11:43:07.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f37362c6700 1 -- 172.21.2.117:0/3042529775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373009ed70 con 0x7f3730083990 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.538 7f37362c6700 1 -- 172.21.2.117:0/3042529775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373009f2c0 con 0x7f3730083990 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.539 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3720007020 con 0x7f3730083990 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.539 7f372cff9700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 0x7f371001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.539 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f3720032500 con 0x7f3730083990 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.540 7f372f7fe700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 0x7f371001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:07.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.541 7f372f7fe700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 0x7f371001abc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:07.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.569 7f37362c6700 1 -- 172.21.2.117:0/3042529775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373009fb50 con 0x7f3730083990 2021-02-10T11:43:07.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.572 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3720009340 con 0x7f3730083990 2021-02-10T11:43:07.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:07.700 7f37362c6700 1 -- 172.21.2.117:0/3042529775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f373009fb50 con 0x7f3730083990 2021-02-10T11:43:07.700 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:07.703 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:08.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.057 7f372cff9700 1 -- 172.21.2.117:0/3042529775 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v62) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3720007020 con 0x7f3730083990 2021-02-10T11:43:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 -- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 msgr2=0x7f371001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 0x7f371001abc0 crc :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 -- 172.21.2.117:0/3042529775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 msgr2=0x7f37300863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f3720009630 tx=0x7f372000b980).stop 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 -- 172.21.2.117:0/3042529775 shutdown_connections 2021-02-10T11:43:08.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37301a4580 0x7f37300dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37301a4180 0x7f37300dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37100186c0 0x7f371001abc0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 --2- 172.21.2.117:0/3042529775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3730083990 0x7f37300863d0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.067 7f37362c6700 1 -- 172.21.2.117:0/3042529775 >> 172.21.2.117:0/3042529775 conn(0x7f37300b6540 msgr2=0x7f373008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.068 7f37362c6700 1 -- 172.21.2.117:0/3042529775 shutdown_connections 2021-02-10T11:43:08.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.068 7f37362c6700 1 -- 172.21.2.117:0/3042529775 wait complete. 2021-02-10T11:43:08.066 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.488 7febcfd72700 1 Processor -- start 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.488 7febcfd72700 1 -- start start 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcfd72700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 0x7febc81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcfd72700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7febc80e5240 0x7febc8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcfd72700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc8085930 con 0x7febc81b0810 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcfd72700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc8087430 con 0x7febc80e5240 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcfd72700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc80e5cc0 con 0x7febc80e4e40 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcd38d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7febc80e5240 0x7febc8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcdb8e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 0x7febc81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcd38d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7febc80e5240 0x7febc8084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54320/0 (socket says 172.21.2.117:54320) 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcdb8e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 0x7febc81b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47616/0 (socket says 172.21.2.117:47616) 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.489 7febcd38d700 1 -- 172.21.2.117:0/876663567 learned_addr learned my addr 172.21.2.117:0/876663567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febcd38d700 1 -- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 msgr2=0x7febc81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febcd38d700 1 --2- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 0x7febc81b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febcd38d700 1 -- 172.21.2.117:0/876663567 >> v1:172.21.2.125:6789/0 conn(0x7febc81b0810 legacy=0x7febc8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febccb8c700 1 -- 172.21.2.117:0/876663567 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62073816 0 0) 0x7febc0000f90 con 0x7febc81b0810 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febcd38d700 1 -- 172.21.2.117:0/876663567 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8084840 con 0x7febc80e5240 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.490 7febcd38d700 1 --2- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80e5240 0x7febc8084c60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7febbc006700 tx=0x7febbc009b00).ready entity=mon.1 client_cookie=33505c46747377d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febccb8c700 1 -- 172.21.2.117:0/876663567 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00c0c0 con 0x7febc80e5240 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febccb8c700 1 -- 172.21.2.117:0/876663567 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc00c340 con 0x7febc80e5240 2021-02-10T11:43:08.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febccb8c700 1 -- 172.21.2.117:0/876663567 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febbc00c820 con 0x7febc80e5240 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 -- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80e5240 msgr2=0x7febc8084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 --2- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80e5240 0x7febc8084c60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7febbc006700 tx=0x7febbc009b00).stop 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 -- 172.21.2.117:0/876663567 shutdown_connections 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 --2- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80e5240 0x7febc8084c60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 --2- 172.21.2.117:0/876663567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc80e4e40 0x7febc81b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 -- 172.21.2.117:0/876663567 >> 172.21.2.117:0/876663567 conn(0x7febc80ce4e0 msgr2=0x7febc80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.491 7febcfd72700 1 -- 172.21.2.117:0/876663567 shutdown_connections 2021-02-10T11:43:08.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.492 7febcfd72700 1 -- 172.21.2.117:0/876663567 wait complete. 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 Processor -- start 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 -- start start 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 0x7febc81915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 0x7febc8096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc8195290 con 0x7febc8194e10 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc80b1460 con 0x7febc80973f0 2021-02-10T11:43:08.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febcfd72700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc80b1680 con 0x7febc8191bf0 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcdb8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 0x7febc81915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.506 7febce38f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcd38d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 0x7febc8096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcd38d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 0x7febc8096dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47620/0 (socket says 172.21.2.117:47620) 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcdb8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 0x7febc81915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49942/0 (socket says 172.21.2.117:49942) 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcd38d700 1 -- 172.21.2.117:0/904536410 learned_addr learned my addr 172.21.2.117:0/904536410 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54328/0 (socket says 172.21.2.117:54328) 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 -- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 msgr2=0x7febc8096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 0x7febc8096dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 -- 172.21.2.117:0/904536410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 msgr2=0x7febc81915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 0x7febc81915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 -- 172.21.2.117:0/904536410 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80b18a0 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febce38f700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7febc000b8d0 tx=0x7febc000bc70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc0001150 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc0001390 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc00018b0 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.507 7febcfd72700 1 -- 172.21.2.117:0/904536410 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc80b18a0 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.508 7febcfd72700 1 -- 172.21.2.117:0/904536410 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc80b18a0 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.508 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febc0001040 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.508 7febba7fc700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 0x7febb001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.508 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5804+0+0 (secure 0 0 0) 0x7febc0032ad0 con 0x7febc80973f0 2021-02-10T11:43:08.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.508 7febcdb8e700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 0x7febb001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:08.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.509 7febcdb8e700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 0x7febb001ac10 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:08.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.539 7febcfd72700 1 -- 172.21.2.117:0/904536410 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc8085930 con 0x7febc80973f0 2021-02-10T11:43:08.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.542 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febc0001370 con 0x7febc80973f0 2021-02-10T11:43:08.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.672 7febcfd72700 1 -- 172.21.2.117:0/904536410 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7febc8085930 con 0x7febc80973f0 2021-02-10T11:43:08.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.678 7febba7fc700 1 -- 172.21.2.117:0/904536410 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 12 and data pool 13 v41) v1 ==== 189+0+0 (secure 0 0 0) 0x7febc0001040 con 0x7febc80973f0 2021-02-10T11:43:08.670 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 12 and data pool 13 2021-02-10T11:43:08.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 msgr2=0x7febb001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 0x7febb001ac10 crc :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 msgr2=0x7febc80977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:08.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7febc000b8d0 tx=0x7febc000bc70).stop 2021-02-10T11:43:08.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 shutdown_connections 2021-02-10T11:43:08.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febc80973f0 0x7febc80977f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febc8191bf0 0x7febc8096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febb0018730 0x7febb001ac10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 --2- 172.21.2.117:0/904536410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febc8194e10 0x7febc81915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:08.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 >> 172.21.2.117:0/904536410 conn(0x7febc818cc70 msgr2=0x7febc8093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:08.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 shutdown_connections 2021-02-10T11:43:08.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:08.681 7febcfd72700 1 -- 172.21.2.117:0/904536410 wait complete. 2021-02-10T11:43:08.680 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:09.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 Processor -- start 2021-02-10T11:43:09.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 -- start start 2021-02-10T11:43:09.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7b9c085450 0x7f7b9c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c085930 con 0x7f7b9c0e4e40 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9c1b09e0 con 0x7f7b9c0e5240 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.099 7f7ba383d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b9c0e5cc0 con 0x7f7b9c085450 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1659700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7b9c0e4e40 0x7f7b9c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50206/0 (socket says 172.21.2.117:50206) 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1659700 1 -- 172.21.2.117:0/3757613315 learned_addr learned my addr 172.21.2.117:0/3757613315 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1e5a700 1 --2- 172.21.2.117:0/3757613315 >> v2:172.21.2.125:3300/0 conn(0x7f7b9c085450 0x7f7b9c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195159415 0 0) 0x7f7b900010b0 con 0x7f7b9c0e4e40 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88002fe0 con 0x7f7b9c0e4e40 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418005738 0 0) 0x7f7b98000ff0 con 0x7f7b9c0e5240 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88004370 con 0x7f7b9c0e5240 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.1 v1:172.21.2.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 681962564 0 0) 0x7f7b98000fd0 con 0x7f7b9c0e5240 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1e5a700 1 -- 172.21.2.117:0/3757613315 >> v1:172.21.2.125:6789/0 conn(0x7f7b9c0e5240 legacy=0x7f7b9c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1e5a700 1 -- 172.21.2.117:0/3757613315 >> v1:172.21.2.117:6789/0 conn(0x7f7b9c0e4e40 legacy=0x7f7b9c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1e5a700 1 -- 172.21.2.117:0/3757613315 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c084840 con 0x7f7b9c085450 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.100 7f7ba1e5a700 1 --2- 172.21.2.117:0/3757613315 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c085450 0x7f7b9c086ec0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f7b94006e50 tx=0x7f7b94009e70).ready entity=mon.1 client_cookie=ccff2c36d5664faa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b94006a20 con 0x7f7b9c085450 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94006790 con 0x7f7b9c085450 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7b8ffff700 1 -- 172.21.2.117:0/3757613315 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b94000f90 con 0x7f7b9c085450 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7ba383d700 1 -- 172.21.2.117:0/3757613315 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c085450 msgr2=0x7f7b9c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7ba383d700 1 --2- 172.21.2.117:0/3757613315 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c085450 0x7f7b9c086ec0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f7b94006e50 tx=0x7f7b94009e70).stop 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7ba383d700 1 -- 172.21.2.117:0/3757613315 shutdown_connections 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7ba383d700 1 --2- 172.21.2.117:0/3757613315 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c085450 0x7f7b9c086ec0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.101 7f7ba383d700 1 -- 172.21.2.117:0/3757613315 >> 172.21.2.117:0/3757613315 conn(0x7f7b9c0ce4e0 msgr2=0x7f7b9c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.102 7f7ba383d700 1 -- 172.21.2.117:0/3757613315 shutdown_connections 2021-02-10T11:43:09.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.102 7f7ba383d700 1 -- 172.21.2.117:0/3757613315 wait complete. 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.116 7f7ba383d700 1 Processor -- start 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.116 7f7ba383d700 1 -- start start 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b9c10db80 0x7f7b9c190500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c1a7290 0x7f7b9c1a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c08e090 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c10df80 con 0x7f7b9c1a7290 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba383d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c190b20 con 0x7f7b9c10db80 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49964/0 (socket says 172.21.2.117:49964) 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 -- 172.21.2.117:0/3215709652 learned_addr learned my addr 172.21.2.117:0/3215709652 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 -- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b9c10db80 msgr2=0x7f7b9c190500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b9c10db80 0x7f7b9c190500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 -- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c1a7290 msgr2=0x7f7b9c1a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c1a7290 0x7f7b9c1a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 -- 172.21.2.117:0/3215709652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c08cfc0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.117 7f7ba1e5a700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f7b94006e60 tx=0x7f7b9400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b94000df0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b940091c0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c08cfc0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b940013d0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c08cfc0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b940056b0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 0x7f7b7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7b94032890 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.118 7f7ba1659700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 0x7f7b7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.119 7f7ba1659700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 0x7f7b7c01abc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.149 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9c085930 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.152 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b9400be30 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.276 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b9c085930 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.277 7f7b8dffb700 1 -- 172.21.2.117:0/3215709652 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f7b940056b0 con 0x7f7b9c1a7cb0 2021-02-10T11:43:09.267 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:09.267 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":63,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:08.674656","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:09.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 msgr2=0x7f7b7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 0x7f7b7c01abc0 crc :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 msgr2=0x7f7b9c08e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0x7f7b94006e60 tx=0x7f7b9400bcf0).stop 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 shutdown_connections 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b7c0186c0 0x7f7b7c01abc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b9c1a7cb0 0x7f7b9c08e610 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b9c1a7290 0x7f7b9c1a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 --2- 172.21.2.117:0/3215709652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b9c10db80 0x7f7b9c190500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:09.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 >> 172.21.2.117:0/3215709652 conn(0x7f7b9c09e890 msgr2=0x7f7b9c08d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 shutdown_connections 2021-02-10T11:43:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.278 7f7ba383d700 1 -- 172.21.2.117:0/3215709652 wait complete. 2021-02-10T11:43:09.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 Processor -- start 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 -- start start 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 0x7fbde80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80ede60 con 0x7fbde80ef3f0 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80d1ce0 con 0x7fbde80eb000 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdf0168700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8088020 con 0x7fbde80d18e0 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdee785700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbde80ef3f0 0x7fbde81b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50224/0 (socket says 172.21.2.117:50224) 2021-02-10T11:43:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.698 7fbdee785700 1 -- 172.21.2.117:0/2383293588 learned_addr learned my addr 172.21.2.117:0/2383293588 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdedf84700 1 --2- 172.21.2.117:0/2383293588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 0x7fbde80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168393858 0 0) 0x7fbdd8000ff0 con 0x7fbde80eb000 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd4002fe0 con 0x7fbde80eb000 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805296217 0 0) 0x7fbde4001180 con 0x7fbde80ef3f0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd40042c0 con 0x7fbde80ef3f0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdedf84700 1 -- 172.21.2.117:0/2383293588 >> v1:172.21.2.125:6789/0 conn(0x7fbde80eb000 legacy=0x7fbde80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdedf84700 1 -- 172.21.2.117:0/2383293588 >> v1:172.21.2.117:6789/0 conn(0x7fbde80ef3f0 legacy=0x7fbde81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdedf84700 1 -- 172.21.2.117:0/2383293588 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde81bb180 con 0x7fbde80d18e0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdedf84700 1 --2- 172.21.2.117:0/2383293588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 0x7fbde80ea9e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbde00069d0 tx=0x7fbde0009dc0).ready entity=mon.2 client_cookie=337c289feaae565c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbde000b510 con 0x7fbde80d18e0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000b790 con 0x7fbde80d18e0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdecf82700 1 -- 172.21.2.117:0/2383293588 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbde000bcb0 con 0x7fbde80d18e0 2021-02-10T11:43:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdf0168700 1 -- 172.21.2.117:0/2383293588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 msgr2=0x7fbde80ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.699 7fbdf0168700 1 --2- 172.21.2.117:0/2383293588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 0x7fbde80ea9e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbde00069d0 tx=0x7fbde0009dc0).stop 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.700 7fbdf0168700 1 -- 172.21.2.117:0/2383293588 shutdown_connections 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.700 7fbdf0168700 1 --2- 172.21.2.117:0/2383293588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80d18e0 0x7fbde80ea9e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.700 7fbdf0168700 1 -- 172.21.2.117:0/2383293588 >> 172.21.2.117:0/2383293588 conn(0x7fbde80ced60 msgr2=0x7fbde80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.700 7fbdf0168700 1 -- 172.21.2.117:0/2383293588 shutdown_connections 2021-02-10T11:43:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.700 7fbdf0168700 1 -- 172.21.2.117:0/2383293588 wait complete. 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 Processor -- start 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 -- start start 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80afa20 0x7fbde808dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 0x7fbde81b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde808d730 con 0x7fbde8097c10 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8098630 con 0x7fbde81b0380 2021-02-10T11:43:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbdf0168700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81b0da0 con 0x7fbde80afa20 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbded783700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbded783700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49978/0 (socket says 172.21.2.117:49978) 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.715 7fbded783700 1 -- 172.21.2.117:0/478018918 learned_addr learned my addr 172.21.2.117:0/478018918 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdedf84700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80afa20 0x7fbde808dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdee785700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 0x7fbde81b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 -- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80afa20 msgr2=0x7fbde808dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80afa20 0x7fbde808dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 -- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 msgr2=0x7fbde81b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 0x7fbde81b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 -- 172.21.2.117:0/478018918 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde808a5a0 con 0x7fbde8097c10 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdee785700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 0x7fbde81b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbded783700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fbdd8006ed0 tx=0x7fbdd800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd800bd50 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8006240 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdd8005730 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdf0168700 1 -- 172.21.2.117:0/478018918 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde808a5a0 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.716 7fbdf0168700 1 -- 172.21.2.117:0/478018918 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde808a5a0 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.717 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbdd80054b0 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.717 7fbdde7fc700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 0x7fbdd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.717 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbdd80326b0 con 0x7fbde8097c10 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.718 7fbdedf84700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 0x7fbdd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.718 7fbdedf84700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 0x7fbdd001abc0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:09.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.748 7fbdf0168700 1 -- 172.21.2.117:0/478018918 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8087290 con 0x7fbde8097c10 2021-02-10T11:43:09.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.751 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbdd8006030 con 0x7fbde8097c10 2021-02-10T11:43:09.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.875 7fbdf0168700 1 -- 172.21.2.117:0/478018918 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbde8087290 con 0x7fbde8097c10 2021-02-10T11:43:09.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.876 7fbdde7fc700 1 -- 172.21.2.117:0/478018918 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7fbdd80054b0 con 0x7fbde8097c10 2021-02-10T11:43:09.867 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:09.867 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 msgr2=0x7fbdd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 0x7fbdd001abc0 crc :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 msgr2=0x7fbde8098010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7fbdd8006ed0 tx=0x7fbdd800b960).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 shutdown_connections 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbdd00186c0 0x7fbdd001abc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbde8097c10 0x7fbde8098010 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbde81b0380 0x7fbde81b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 --2- 172.21.2.117:0/478018918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbde80afa20 0x7fbde808dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 >> 172.21.2.117:0/478018918 conn(0x7fbde81a3620 msgr2=0x7fbde80e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 shutdown_connections 2021-02-10T11:43:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:09.878 7fbdf0168700 1 -- 172.21.2.117:0/478018918 wait complete. 2021-02-10T11:43:09.877 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.300 7f2793a64700 1 Processor -- start 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.300 7f2793a64700 1 -- start start 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2793a64700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f278c0e7180 0x7f278c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2793a64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c0e8390 0x7f278c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2793a64700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0e7580 con 0x7f278c1b07f0 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2793a64700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278c0e5ce0 con 0x7f278c0e7180 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2793a64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c0e77f0 con 0x7f278c0e8390 2021-02-10T11:43:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f279107f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f278c0e7180 0x7f278c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f279107f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f278c0e7180 0x7f278c0e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54368/0 (socket says 172.21.2.117:54368) 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f2792081700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c0e8390 0x7f278c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.301 7f279107f700 1 -- 172.21.2.117:0/555912595 learned_addr learned my addr 172.21.2.117:0/555912595 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279107f700 1 -- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c0e8390 msgr2=0x7f278c0e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279107f700 1 --2- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c0e8390 0x7f278c0e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279107f700 1 -- 172.21.2.117:0/555912595 >> v1:172.21.2.125:6789/0 conn(0x7f278c1b07f0 legacy=0x7f278c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279107f700 1 -- 172.21.2.117:0/555912595 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0e8db0 con 0x7f278c0e7180 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279107f700 1 --2- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c0e7180 0x7f278c0e7d70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f2788006700 tx=0x7f2788009b80).ready entity=mon.1 client_cookie=cefc93b3149aaf8c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279087e700 1 -- 172.21.2.117:0/555912595 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2788006340 con 0x7f278c0e7180 2021-02-10T11:43:10.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279087e700 1 -- 172.21.2.117:0/555912595 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278800c0c0 con 0x7f278c0e7180 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f279087e700 1 -- 172.21.2.117:0/555912595 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f278800c5e0 con 0x7f278c0e7180 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f2793a64700 1 -- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c0e7180 msgr2=0x7f278c0e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.302 7f2793a64700 1 --2- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c0e7180 0x7f278c0e7d70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f2788006700 tx=0x7f2788009b80).stop 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 -- 172.21.2.117:0/555912595 shutdown_connections 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 --2- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c0e8390 0x7f278c0e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 --2- 172.21.2.117:0/555912595 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c0e7180 0x7f278c0e7d70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 -- 172.21.2.117:0/555912595 >> 172.21.2.117:0/555912595 conn(0x7f278c0ce500 msgr2=0x7f278c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 -- 172.21.2.117:0/555912595 shutdown_connections 2021-02-10T11:43:10.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.303 7f2793a64700 1 -- 172.21.2.117:0/555912595 wait complete. 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 Processor -- start 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 -- start start 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c1a4170 0x7f278c0a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c18b9b0 0x7f278c08caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c08c2b0 con 0x7f278c0a7e50 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c0a8250 con 0x7f278c18b9b0 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f2793a64700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c18bdb0 con 0x7f278c1a4170 2021-02-10T11:43:10.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f279107f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f279107f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49992/0 (socket says 172.21.2.117:49992) 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.318 7f279107f700 1 -- 172.21.2.117:0/1887160327 learned_addr learned my addr 172.21.2.117:0/1887160327 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 -- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c1a4170 msgr2=0x7f278c0a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c1a4170 0x7f278c0a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 -- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c18b9b0 msgr2=0x7f278c08caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c18b9b0 0x7f278c08caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 -- 172.21.2.117:0/1887160327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c08d0c0 con 0x7f278c0a7e50 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f279107f700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f278800b580 tx=0x7f278800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2788002030 con 0x7f278c0a7e50 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2788002370 con 0x7f278c0a7e50 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f2793a64700 1 -- 172.21.2.117:0/1887160327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c08d0c0 con 0x7f278c0a7e50 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2788002890 con 0x7f278c0a7e50 2021-02-10T11:43:10.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.319 7f2793a64700 1 -- 172.21.2.117:0/1887160327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c08d0c0 con 0x7f278c0a7e50 2021-02-10T11:43:10.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.320 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2788000c90 con 0x7f278c0a7e50 2021-02-10T11:43:10.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.320 7f277e7fc700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 0x7f277401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:10.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.320 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f27880326d0 con 0x7f278c0a7e50 2021-02-10T11:43:10.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.320 7f2791880700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 0x7f277401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:10.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.321 7f2791880700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 0x7f277401abc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:10.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.351 7f2793a64700 1 -- 172.21.2.117:0/1887160327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c10c290 con 0x7f278c0a7e50 2021-02-10T11:43:10.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.353 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f278802ef00 con 0x7f278c0a7e50 2021-02-10T11:43:10.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.488 7f2793a64700 1 -- 172.21.2.117:0/1887160327 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f278c0e77f0 con 0x7f278c0a7e50 2021-02-10T11:43:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.684 7f277e7fc700 1 -- 172.21.2.117:0/1887160327 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2788002450 con 0x7f278c0a7e50 2021-02-10T11:43:10.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.705 7f2793a64700 1 -- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 msgr2=0x7f277401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.705 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 0x7f277401abc0 crc :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.705 7f2793a64700 1 -- 172.21.2.117:0/1887160327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 msgr2=0x7f278c18b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.705 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f278800b580 tx=0x7f278800b920).stop 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 -- 172.21.2.117:0/1887160327 shutdown_connections 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f27740186c0 0x7f277401abc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f278c0a7e50 0x7f278c18b390 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f278c18b9b0 0x7f278c08caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 --2- 172.21.2.117:0/1887160327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f278c1a4170 0x7f278c0a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 -- 172.21.2.117:0/1887160327 >> 172.21.2.117:0/1887160327 conn(0x7f278c18c1e0 msgr2=0x7f278c090e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 -- 172.21.2.117:0/1887160327 shutdown_connections 2021-02-10T11:43:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:10.706 7f2793a64700 1 -- 172.21.2.117:0/1887160327 wait complete. 2021-02-10T11:43:10.704 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:11.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.125 7f4b94033700 1 Processor -- start 2021-02-10T11:43:11.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.125 7f4b94033700 1 -- start start 2021-02-10T11:43:11.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b94033700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c0d18e0 0x7f4b8c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b94033700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b8c0eb000 0x7f4b8c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b94033700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0ede60 con 0x7f4b8c0ee3e0 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b94033700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c0eba20 con 0x7f4b8c0eb000 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b94033700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0d1ce0 con 0x7f4b8c0d18e0 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b8c0eb000 0x7f4b8c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b8c0eb000 0x7f4b8c0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:49998/0 (socket says 172.21.2.117:49998) 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 -- 172.21.2.117:0/3325649556 learned_addr learned my addr 172.21.2.117:0/3325649556 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 -- 172.21.2.117:0/3325649556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c0d18e0 msgr2=0x7f4b8c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 --2- 172.21.2.117:0/3325649556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c0d18e0 0x7f4b8c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 -- 172.21.2.117:0/3325649556 >> v1:172.21.2.117:6789/0 conn(0x7f4b8c0ee3e0 legacy=0x7f4b8c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 -- 172.21.2.117:0/3325649556 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0eefd0 con 0x7f4b8c0eb000 2021-02-10T11:43:11.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b9164e700 1 --2- 172.21.2.117:0/3325649556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0eb000 0x7f4b8c0eb400 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f4b7c006700 tx=0x7f4b7c009b80).ready entity=mon.0 client_cookie=e34f682c57465c04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b90e4d700 1 -- 172.21.2.117:0/3325649556 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b7c00c0c0 con 0x7f4b8c0eb000 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.126 7f4b90e4d700 1 -- 172.21.2.117:0/3325649556 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c00c440 con 0x7f4b8c0eb000 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b90e4d700 1 -- 172.21.2.117:0/3325649556 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b7c00c920 con 0x7f4b8c0eb000 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 -- 172.21.2.117:0/3325649556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0eb000 msgr2=0x7f4b8c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 --2- 172.21.2.117:0/3325649556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0eb000 0x7f4b8c0eb400 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f4b7c006700 tx=0x7f4b7c009b80).stop 2021-02-10T11:43:11.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 -- 172.21.2.117:0/3325649556 shutdown_connections 2021-02-10T11:43:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 --2- 172.21.2.117:0/3325649556 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0eb000 0x7f4b8c0eb400 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 --2- 172.21.2.117:0/3325649556 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c0d18e0 0x7f4b8c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 -- 172.21.2.117:0/3325649556 >> 172.21.2.117:0/3325649556 conn(0x7f4b8c0ced60 msgr2=0x7f4b8c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 -- 172.21.2.117:0/3325649556 shutdown_connections 2021-02-10T11:43:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.127 7f4b94033700 1 -- 172.21.2.117:0/3325649556 wait complete. 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.142 7f4b94033700 1 Processor -- start 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.142 7f4b94033700 1 -- start start 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c1aa550 0x7f4b8c1a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b8c1a3bf0 0x7f4b8c1a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1a9340 con 0x7f4b8c0df870 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1aa990 con 0x7f4b8c1a3bf0 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b94033700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1a9e60 con 0x7f4b8c1aa550 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50004/0 (socket says 172.21.2.117:50004) 2021-02-10T11:43:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 -- 172.21.2.117:0/4203679178 learned_addr learned my addr 172.21.2.117:0/4203679178 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b91e4f700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c1aa550 0x7f4b8c1a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b9164e700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b8c1a3bf0 0x7f4b8c1a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 -- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c1aa550 msgr2=0x7f4b8c1a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c1aa550 0x7f4b8c1a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 -- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b8c1a3bf0 msgr2=0x7f4b8c1a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b8c1a3bf0 0x7f4b8c1a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.143 7f4b92650700 1 -- 172.21.2.117:0/4203679178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0e0290 con 0x7f4b8c0df870 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b92650700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f4b84009980 tx=0x7f4b8400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b84005870 con 0x7f4b8c0df870 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b84005a70 con 0x7f4b8c0df870 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b94033700 1 -- 172.21.2.117:0/4203679178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c0e0290 con 0x7f4b8c0df870 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b84006030 con 0x7f4b8c0df870 2021-02-10T11:43:11.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.144 7f4b94033700 1 -- 172.21.2.117:0/4203679178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c0e0290 con 0x7f4b8c0df870 2021-02-10T11:43:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.145 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b84005c50 con 0x7f4b8c0df870 2021-02-10T11:43:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.145 7f4b827fc700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 0x7f4b7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.145 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b84032cc0 con 0x7f4b8c0df870 2021-02-10T11:43:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.145 7f4b91e4f700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 0x7f4b7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.145 7f4b91e4f700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 0x7f4b7401abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.176 7f4b94033700 1 -- 172.21.2.117:0/4203679178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c087290 con 0x7f4b8c0df870 2021-02-10T11:43:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.179 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b8400be10 con 0x7f4b8c0df870 2021-02-10T11:43:11.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.307 7f4b94033700 1 -- 172.21.2.117:0/4203679178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b8c1aa950 con 0x7f4b8c0df870 2021-02-10T11:43:11.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.307 7f4b827fc700 1 -- 172.21.2.117:0/4203679178 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f4b84005a70 con 0x7f4b8c0df870 2021-02-10T11:43:11.298 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:11.299 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:11.299 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 44 2021-02-10T11:43:11.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 msgr2=0x7f4b7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 0x7f4b7401abc0 crc :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 msgr2=0x7f4b8c0dfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f4b84009980 tx=0x7f4b8400bcd0).stop 2021-02-10T11:43:11.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 shutdown_connections 2021-02-10T11:43:11.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b740186c0 0x7f4b7401abc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b8c0df870 0x7f4b8c0dfc70 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b8c1a3bf0 0x7f4b8c1a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 --2- 172.21.2.117:0/4203679178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b8c1aa550 0x7f4b8c1a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 >> 172.21.2.117:0/4203679178 conn(0x7f4b8c1a3620 msgr2=0x7f4b8c0e0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 shutdown_connections 2021-02-10T11:43:11.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.309 7f4b94033700 1 -- 172.21.2.117:0/4203679178 wait complete. 2021-02-10T11:43:11.308 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:11.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.729 7f7056283700 1 Processor -- start 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.729 7f7056283700 1 -- start start 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f7056283700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f70500d18e0 0x7f70500ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f7056283700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70500ef3f0 0x7f70501b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f7056283700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70500ede60 con 0x7f70500eb000 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f7056283700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500ef7f0 con 0x7f70500ef3f0 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f7056283700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500d1ce0 con 0x7f70500d18e0 2021-02-10T11:43:11.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70500ef3f0 0x7f70501b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70500ef3f0 0x7f70501b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50012/0 (socket says 172.21.2.117:50012) 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 -- 172.21.2.117:0/3751525416 learned_addr learned my addr 172.21.2.117:0/3751525416 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 -- 172.21.2.117:0/3751525416 >> v2:172.21.2.125:3300/0 conn(0x7f70500d18e0 msgr2=0x7f70500ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 --2- 172.21.2.117:0/3751525416 >> v2:172.21.2.125:3300/0 conn(0x7f70500d18e0 0x7f70500ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 -- 172.21.2.117:0/3751525416 >> v1:172.21.2.117:6789/0 conn(0x7f70500eb000 legacy=0x7f70500eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 -- 172.21.2.117:0/3751525416 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70501b8f90 con 0x7f70500ef3f0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.730 7f70548a0700 1 --2- 172.21.2.117:0/3751525416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500ef3f0 0x7f70501b7960 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f7048006f00 tx=0x7f7048009f10).ready entity=mon.0 client_cookie=dc204d25a8d43145 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f704f7fe700 1 -- 172.21.2.117:0/3751525416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f704800b3b0 con 0x7f70500ef3f0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f704f7fe700 1 -- 172.21.2.117:0/3751525416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704800b730 con 0x7f70500ef3f0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f704f7fe700 1 -- 172.21.2.117:0/3751525416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f704800bc10 con 0x7f70500ef3f0 2021-02-10T11:43:11.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 -- 172.21.2.117:0/3751525416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500ef3f0 msgr2=0x7f70501b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 --2- 172.21.2.117:0/3751525416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500ef3f0 0x7f70501b7960 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f7048006f00 tx=0x7f7048009f10).stop 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 -- 172.21.2.117:0/3751525416 shutdown_connections 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 --2- 172.21.2.117:0/3751525416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500ef3f0 0x7f70501b7960 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 --2- 172.21.2.117:0/3751525416 >> v2:172.21.2.125:3300/0 conn(0x7f70500d18e0 0x7f70500ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 -- 172.21.2.117:0/3751525416 >> 172.21.2.117:0/3751525416 conn(0x7f70500ced60 msgr2=0x7f70500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 -- 172.21.2.117:0/3751525416 shutdown_connections 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.731 7f7056283700 1 -- 172.21.2.117:0/3751525416 wait complete. 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 Processor -- start 2021-02-10T11:43:11.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 -- start start 2021-02-10T11:43:11.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 0x7f705019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f705019dd40 0x7f705019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70500b1f30 con 0x7f70500984c0 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019d830 con 0x7f705019dd40 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.746 7f7056283700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70501a55e0 con 0x7f70500980c0 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50020/0 (socket says 172.21.2.117:50020) 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 -- 172.21.2.117:0/3558626881 learned_addr learned my addr 172.21.2.117:0/3558626881 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f704ffff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 0x7f705019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f70548a0700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f705019dd40 0x7f705019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 -- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 msgr2=0x7f705019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 0x7f705019ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 -- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f705019dd40 msgr2=0x7f705019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f705019dd40 0x7f705019e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 -- 172.21.2.117:0/3558626881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705019e760 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f704ffff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 0x7f705019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7047fff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f7040006ed0 tx=0x7f704000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7040006020 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400009b0 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70400054b0 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.747 7f7056283700 1 -- 172.21.2.117:0/3558626881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705019e760 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.748 7f7056283700 1 -- 172.21.2.117:0/3558626881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705019e760 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.748 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7040006230 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.748 7f704d7fa700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 0x7f703401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.748 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7040032780 con 0x7f70500984c0 2021-02-10T11:43:11.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.749 7f704ffff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 0x7f703401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:11.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.749 7f704ffff700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 0x7f703401abc0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:11.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.779 7f7056283700 1 -- 172.21.2.117:0/3558626881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7050087210 con 0x7f70500984c0 2021-02-10T11:43:11.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.782 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7040006020 con 0x7f70500984c0 2021-02-10T11:43:11.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.906 7f7056283700 1 -- 172.21.2.117:0/3558626881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7050087210 con 0x7f70500984c0 2021-02-10T11:43:11.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.907 7f704d7fa700 1 -- 172.21.2.117:0/3558626881 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f704000bd50 con 0x7f70500984c0 2021-02-10T11:43:11.897 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:11.898 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 msgr2=0x7f703401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 0x7f703401abc0 crc :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 msgr2=0x7f705019d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0x7f7040006ed0 tx=0x7f704000b960).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 shutdown_connections 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7034018730 0x7f703401abc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70500984c0 0x7f705019d0a0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f705019dd40 0x7f705019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 --2- 172.21.2.117:0/3558626881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70500980c0 0x7f705019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 >> 172.21.2.117:0/3558626881 conn(0x7f705019f610 msgr2=0x7f70501a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:11.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 shutdown_connections 2021-02-10T11:43:11.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:11.908 7f7056283700 1 -- 172.21.2.117:0/3558626881 wait complete. 2021-02-10T11:43:11.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:12.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 Processor -- start 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 -- start start 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f824c07ab40 0x7f824c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 0x7f824c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c0e5770 con 0x7f824c07af40 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824c0d18c0 con 0x7f824c07ab40 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.331 7f825309f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824c0d1c60 con 0x7f824c079b10 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f824c07ab40 0x7f824c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f82516bc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f824c07af40 0x7f824c07b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50280/0 (socket says 172.21.2.117:50280) 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f824c07ab40 0x7f824c0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50028/0 (socket says 172.21.2.117:50028) 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f82516bc700 1 -- 172.21.2.117:0/932977452 learned_addr learned my addr 172.21.2.117:0/932977452 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:12.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f824bfff700 1 --2- 172.21.2.117:0/932977452 >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 0x7f824c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 -- 172.21.2.117:0/932977452 >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 msgr2=0x7f824c079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 --2- 172.21.2.117:0/932977452 >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 0x7f824c079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 -- 172.21.2.117:0/932977452 >> v1:172.21.2.117:6789/0 conn(0x7f824c07af40 legacy=0x7f824c07b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 -- 172.21.2.117:0/932977452 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c07bb30 con 0x7f824c07ab40 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f824bfff700 1 --2- 172.21.2.117:0/932977452 >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 0x7f824c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.332 7f8250ebb700 1 --2- 172.21.2.117:0/932977452 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c07ab40 0x7f824c0794f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f8234006650 tx=0x7f8234009a80).ready entity=mon.0 client_cookie=86ca35836fd81851 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f824b7fe700 1 -- 172.21.2.117:0/932977452 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823400c0c0 con 0x7f824c07ab40 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f824b7fe700 1 -- 172.21.2.117:0/932977452 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823400c3c0 con 0x7f824c07ab40 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f824b7fe700 1 -- 172.21.2.117:0/932977452 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823400c820 con 0x7f824c07ab40 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 -- 172.21.2.117:0/932977452 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c07ab40 msgr2=0x7f824c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 --2- 172.21.2.117:0/932977452 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c07ab40 0x7f824c0794f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7f8234006650 tx=0x7f8234009a80).stop 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 -- 172.21.2.117:0/932977452 shutdown_connections 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 --2- 172.21.2.117:0/932977452 >> v2:172.21.2.125:3300/0 conn(0x7f824c079b10 0x7f824c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 --2- 172.21.2.117:0/932977452 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c07ab40 0x7f824c0794f0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 -- 172.21.2.117:0/932977452 >> 172.21.2.117:0/932977452 conn(0x7f824c0d0910 msgr2=0x7f824c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 -- 172.21.2.117:0/932977452 shutdown_connections 2021-02-10T11:43:12.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.333 7f825309f700 1 -- 172.21.2.117:0/932977452 wait complete. 2021-02-10T11:43:12.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.348 7f825309f700 1 Processor -- start 2021-02-10T11:43:12.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.348 7f825309f700 1 -- start start 2021-02-10T11:43:12.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f824c099960 0x7f824c100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f824c09a4d0 0x7f824c09a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c100b30 con 0x7f824c100db0 2021-02-10T11:43:12.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c099f50 con 0x7f824c099960 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f825309f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824c09aef0 con 0x7f824c09a4d0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50032/0 (socket says 172.21.2.117:50032) 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 -- 172.21.2.117:0/1900697652 learned_addr learned my addr 172.21.2.117:0/1900697652 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 -- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f824c09a4d0 msgr2=0x7f824c09a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f824c09a4d0 0x7f824c09a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 -- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f824c099960 msgr2=0x7f824c100510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f824c099960 0x7f824c100510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 -- 172.21.2.117:0/1900697652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c09b0d0 con 0x7f824c100db0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f8250ebb700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f8234006690 tx=0x7f823400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.349 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f823400bd50 con 0x7f824c100db0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82340093a0 con 0x7f824c100db0 2021-02-10T11:43:12.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8234005560 con 0x7f824c100db0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f825309f700 1 -- 172.21.2.117:0/1900697652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824c09b0d0 con 0x7f824c100db0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f825309f700 1 -- 172.21.2.117:0/1900697652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c09b660 con 0x7f824c100db0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f82340054f0 con 0x7f824c100db0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f82497fa700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 0x7f823001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.350 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8234032940 con 0x7f824c100db0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.351 7f824bfff700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 0x7f823001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.352 7f824bfff700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 0x7f823001abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.382 7f825309f700 1 -- 172.21.2.117:0/1900697652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824c0daa50 con 0x7f824c100db0 2021-02-10T11:43:12.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.384 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f82340052e0 con 0x7f824c100db0 2021-02-10T11:43:12.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.513 7f825309f700 1 -- 172.21.2.117:0/1900697652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f824c07a5c0 con 0x7f824c100db0 2021-02-10T11:43:12.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.517 7f82497fa700 1 -- 172.21.2.117:0/1900697652 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f82340093a0 con 0x7f824c100db0 2021-02-10T11:43:12.511 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:43:12.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.522 7f825309f700 1 -- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 msgr2=0x7f823001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.522 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 0x7f823001abc0 crc :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.522 7f825309f700 1 -- 172.21.2.117:0/1900697652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 msgr2=0x7f824c0ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.522 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7f8234006690 tx=0x7f823400b960).stop 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 -- 172.21.2.117:0/1900697652 shutdown_connections 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f824c09a4d0 0x7f824c09a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f824c099960 0x7f824c100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82300186c0 0x7f823001abc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 --2- 172.21.2.117:0/1900697652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f824c100db0 0x7f824c0ffdf0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 -- 172.21.2.117:0/1900697652 >> 172.21.2.117:0/1900697652 conn(0x7f824c08f530 msgr2=0x7f824c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 -- 172.21.2.117:0/1900697652 shutdown_connections 2021-02-10T11:43:12.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.523 7f825309f700 1 -- 172.21.2.117:0/1900697652 wait complete. 2021-02-10T11:43:12.522 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:12.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 Processor -- start 2021-02-10T11:43:12.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 -- start start 2021-02-10T11:43:12.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb9f8079f10 0x7fb9f807b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f807a5c0 con 0x7fb9f8079b10 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f80e5770 con 0x7fb9f8079f10 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.954 7fb9fe21b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f807a310 con 0x7fb9f807ab40 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.955 7fb9f7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.955 7fb9fc838700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb9f8079f10 0x7fb9f807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.955 7fb9f7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47720/0 (socket says 172.21.2.117:47720) 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.955 7fb9f7fff700 1 -- 172.21.2.117:0/1640200537 learned_addr learned my addr 172.21.2.117:0/1640200537 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fc838700 1 -- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 msgr2=0x7fb9f80794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fc838700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fc838700 1 -- 172.21.2.117:0/1640200537 >> v1:172.21.2.125:6789/0 conn(0x7fb9f8079b10 legacy=0x7fb9f807af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fc838700 1 -- 172.21.2.117:0/1640200537 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80d18c0 con 0x7fb9f8079f10 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9f7fff700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fc838700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f8079f10 0x7fb9f807b730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fb9f0006ad0 tx=0x7fb9f0009ad0).ready entity=mon.1 client_cookie=b59b0de256efb272 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9f6ffd700 1 -- 172.21.2.117:0/1640200537 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9f00066a0 con 0x7fb9f8079f10 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9f6ffd700 1 -- 172.21.2.117:0/1640200537 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0006410 con 0x7fb9f8079f10 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9f6ffd700 1 -- 172.21.2.117:0/1640200537 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9f00024d0 con 0x7fb9f8079f10 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fe21b700 1 -- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f8079f10 msgr2=0x7fb9f807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.956 7fb9fe21b700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f8079f10 0x7fb9f807b730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fb9f0006ad0 tx=0x7fb9f0009ad0).stop 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 -- 172.21.2.117:0/1640200537 shutdown_connections 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f8079f10 0x7fb9f807b730 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 --2- 172.21.2.117:0/1640200537 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f807ab40 0x7fb9f80794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 -- 172.21.2.117:0/1640200537 >> 172.21.2.117:0/1640200537 conn(0x7fb9f80d0910 msgr2=0x7fb9f80d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 -- 172.21.2.117:0/1640200537 shutdown_connections 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.958 7fb9fe21b700 1 -- 172.21.2.117:0/1640200537 wait complete. 2021-02-10T11:43:12.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 Processor -- start 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 -- start start 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f81abc20 0x7fb9f80b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f80b3130 0x7fb9f80b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f81ac020 con 0x7fb9f8100770 2021-02-10T11:43:12.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f81ac260 con 0x7fb9f80b3130 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9fe21b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f8100c80 con 0x7fb9f81abc20 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9f7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9f7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50044/0 (socket says 172.21.2.117:50044) 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9f7fff700 1 -- 172.21.2.117:0/4107476886 learned_addr learned my addr 172.21.2.117:0/4107476886 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.972 7fb9f77fe700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f81abc20 0x7fb9f80b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9fc838700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f80b3130 0x7fb9f80b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 -- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f81abc20 msgr2=0x7fb9f80b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f81abc20 0x7fb9f80b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 -- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f80b3130 msgr2=0x7fb9f80b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:12.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f80b3130 0x7fb9f80b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 -- 172.21.2.117:0/4107476886 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f81ac7e0 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f7fff700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fb9e800b970 tx=0x7fb9e800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e8000d30 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e8006240 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f81ac7e0 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb9e8005660 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.973 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f81ac7e0 con 0x7fb9f8100770 2021-02-10T11:43:12.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.974 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb9e8006630 con 0x7fb9f8100770 2021-02-10T11:43:12.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.974 7fb9f4ff9700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 0x7fb9d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:12.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.974 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb9e8032b00 con 0x7fb9f8100770 2021-02-10T11:43:12.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.974 7fb9f77fe700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 0x7fb9d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:12.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:12.975 7fb9f77fe700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 0x7fb9d801ac10 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:12.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.004 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f80a1cf0 con 0x7fb9f8100770 2021-02-10T11:43:12.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.007 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb9e8006240 con 0x7fb9f8100770 2021-02-10T11:43:13.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.136 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb9f80a1cf0 con 0x7fb9f8100770 2021-02-10T11:43:13.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.139 7fb9f4ff9700 1 -- 172.21.2.117:0/4107476886 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb9e8006450 con 0x7fb9f8100770 2021-02-10T11:43:13.133 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:43:13.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 msgr2=0x7fb9d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 0x7fb9d801ac10 crc :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 msgr2=0x7fb9f81ab600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fb9e800b970 tx=0x7fb9e800bd10).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 shutdown_connections 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9f80b3130 0x7fb9f80b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9f81abc20 0x7fb9f80b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9d8018730 0x7fb9d801ac10 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 --2- 172.21.2.117:0/4107476886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9f8100770 0x7fb9f81ab600 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 >> 172.21.2.117:0/4107476886 conn(0x7fb9f81a3700 msgr2=0x7fb9f80dac80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 shutdown_connections 2021-02-10T11:43:13.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.145 7fb9fe21b700 1 -- 172.21.2.117:0/4107476886 wait complete. 2021-02-10T11:43:13.144 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:13.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 Processor -- start 2021-02-10T11:43:13.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 -- start start 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdb3c079b10 0x7fdb3c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c07a5c0 con 0x7fdb3c07af40 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0e5770 con 0x7fdb3c079b10 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.575 7fdb444f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0d18c0 con 0x7fdb3c07ab40 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47732/0 (socket says 172.21.2.117:47732) 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 -- 172.21.2.117:0/1554972775 learned_addr learned my addr 172.21.2.117:0/1554972775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb41b12700 1 --2- 172.21.2.117:0/1554972775 >> v2:172.21.2.125:3300/0 conn(0x7fdb3c079b10 0x7fdb3c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42b14700 1 --1- 172.21.2.117:0/1554972775 >> v1:172.21.2.125:6789/0 conn(0x7fdb3c07af40 0x7fdb3c07b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:43228/0 (socket says 172.21.2.117:43228) 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 -- 172.21.2.117:0/1554972775 >> v2:172.21.2.125:3300/0 conn(0x7fdb3c079b10 msgr2=0x7fdb3c079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 --2- 172.21.2.117:0/1554972775 >> v2:172.21.2.125:3300/0 conn(0x7fdb3c079b10 0x7fdb3c079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 -- 172.21.2.117:0/1554972775 >> v1:172.21.2.125:6789/0 conn(0x7fdb3c07af40 legacy=0x7fdb3c07b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 -- 172.21.2.117:0/1554972775 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0d1cb0 con 0x7fdb3c07ab40 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb42313700 1 --2- 172.21.2.117:0/1554972775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fdb38006700 tx=0x7fdb38009b80).ready entity=mon.2 client_cookie=a2b63230c45c079c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.576 7fdb41311700 1 -- 172.21.2.117:0/1554972775 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb38000c90 con 0x7fdb3c07ab40 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb41311700 1 -- 172.21.2.117:0/1554972775 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38000c90 con 0x7fdb3c07ab40 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb41311700 1 -- 172.21.2.117:0/1554972775 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3800c0c0 con 0x7fdb3c07ab40 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 -- 172.21.2.117:0/1554972775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 msgr2=0x7fdb3c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 --2- 172.21.2.117:0/1554972775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fdb38006700 tx=0x7fdb38009b80).stop 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 -- 172.21.2.117:0/1554972775 shutdown_connections 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 --2- 172.21.2.117:0/1554972775 >> v2:172.21.2.125:3300/0 conn(0x7fdb3c079b10 0x7fdb3c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 --2- 172.21.2.117:0/1554972775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c07ab40 0x7fdb3c0794f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 -- 172.21.2.117:0/1554972775 >> 172.21.2.117:0/1554972775 conn(0x7fdb3c0d0910 msgr2=0x7fdb3c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 -- 172.21.2.117:0/1554972775 shutdown_connections 2021-02-10T11:43:13.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.577 7fdb444f7700 1 -- 172.21.2.117:0/1554972775 wait complete. 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.593 7fdb444f7700 1 Processor -- start 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.593 7fdb444f7700 1 -- start start 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 0x7fdb3c090f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c100970 0x7fdb3c091520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c0be390 con 0x7fdb3c090390 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c090980 con 0x7fdb3c100570 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c100e70 con 0x7fdb3c100970 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50054/0 (socket says 172.21.2.117:50054) 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 -- 172.21.2.117:0/267894243 learned_addr learned my addr 172.21.2.117:0/267894243 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb41b12700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 0x7fdb3c090f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42b14700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c100970 0x7fdb3c091520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 -- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c100970 msgr2=0x7fdb3c091520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c100970 0x7fdb3c091520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 -- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 msgr2=0x7fdb3c090f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 0x7fdb3c090f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 -- 172.21.2.117:0/267894243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c101050 con 0x7fdb3c090390 2021-02-10T11:43:13.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb41b12700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 0x7fdb3c090f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb42313700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fdb38006f00 tx=0x7fdb3800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb3800bd80 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38006030 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb38005560 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 -- 172.21.2.117:0/267894243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c101050 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.594 7fdb444f7700 1 -- 172.21.2.117:0/267894243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c091ea0 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.595 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb380052e0 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.595 7fdb2effd700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 0x7fdb2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.595 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdb38032720 con 0x7fdb3c090390 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.595 7fdb41b12700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 0x7fdb2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:13.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.596 7fdb41b12700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 0x7fdb2401abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:13.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.626 7fdb444f7700 1 -- 172.21.2.117:0/267894243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c0be390 con 0x7fdb3c090390 2021-02-10T11:43:13.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.628 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb3802d270 con 0x7fdb3c090390 2021-02-10T11:43:13.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.757 7fdb444f7700 1 -- 172.21.2.117:0/267894243 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdb3c0be390 con 0x7fdb3c090390 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.760 7fdb2effd700 1 -- 172.21.2.117:0/267894243 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdb380052e0 con 0x7fdb3c090390 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 msgr2=0x7fdb2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 0x7fdb2401abc0 crc :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 msgr2=0x7fdb3c0ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fdb38006f00 tx=0x7fdb3800b990).stop 2021-02-10T11:43:14.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 shutdown_connections 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb3c100970 0x7fdb3c091520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb3c100570 0x7fdb3c090f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb240186c0 0x7fdb2401abc0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 --2- 172.21.2.117:0/267894243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb3c090390 0x7fdb3c0ffe20 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 >> 172.21.2.117:0/267894243 conn(0x7fdb3c08f530 msgr2=0x7fdb3c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 shutdown_connections 2021-02-10T11:43:14.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:13.762 7fdb444f7700 1 -- 172.21.2.117:0/267894243 wait complete. 2021-02-10T11:43:14.863 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.287 7faed321e700 1 Processor -- start 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.287 7faed321e700 1 -- start start 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed321e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faecc0d1900 0x7faecc0eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed321e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faecc0ef400 0x7faecc1b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed321e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0ef800 con 0x7faecc0eb050 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed321e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0ede70 con 0x7faecc0d1900 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed321e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0d1d00 con 0x7faecc0ef400 2021-02-10T11:43:15.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed103a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faecc0d1900 0x7faecc0eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed0839700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faecc0eb050 0x7faecc0eb450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50312/0 (socket says 172.21.2.117:50312) 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed0839700 1 -- 172.21.2.117:0/179538577 learned_addr learned my addr 172.21.2.117:0/179538577 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed103a700 1 -- 172.21.2.117:0/179538577 >> v2:172.21.2.125:3300/0 conn(0x7faecc0ef400 msgr2=0x7faecc1b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed103a700 1 --2- 172.21.2.117:0/179538577 >> v2:172.21.2.125:3300/0 conn(0x7faecc0ef400 0x7faecc1b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed103a700 1 -- 172.21.2.117:0/179538577 >> v1:172.21.2.117:6789/0 conn(0x7faecc0eb050 legacy=0x7faecc0eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.288 7faed103a700 1 -- 172.21.2.117:0/179538577 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc1b8fe0 con 0x7faecc0d1900 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faed103a700 1 --2- 172.21.2.117:0/179538577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc0d1900 0x7faecc0eaa30 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7faec0006650 tx=0x7faec0009a80).ready entity=mon.0 client_cookie=65f4feae5726591f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faebffff700 1 -- 172.21.2.117:0/179538577 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faec00009b0 con 0x7faecc0d1900 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faebffff700 1 -- 172.21.2.117:0/179538577 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec000c0c0 con 0x7faecc0d1900 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faebffff700 1 -- 172.21.2.117:0/179538577 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faec000c5e0 con 0x7faecc0d1900 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faed321e700 1 -- 172.21.2.117:0/179538577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc0d1900 msgr2=0x7faecc0eaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.289 7faed321e700 1 --2- 172.21.2.117:0/179538577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc0d1900 0x7faecc0eaa30 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7faec0006650 tx=0x7faec0009a80).stop 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 -- 172.21.2.117:0/179538577 shutdown_connections 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 --2- 172.21.2.117:0/179538577 >> v2:172.21.2.125:3300/0 conn(0x7faecc0ef400 0x7faecc1b79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 --2- 172.21.2.117:0/179538577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc0d1900 0x7faecc0eaa30 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 -- 172.21.2.117:0/179538577 >> 172.21.2.117:0/179538577 conn(0x7faecc0cf070 msgr2=0x7faecc0cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 -- 172.21.2.117:0/179538577 shutdown_connections 2021-02-10T11:43:15.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.290 7faed321e700 1 -- 172.21.2.117:0/179538577 wait complete. 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.304 7faed321e700 1 Processor -- start 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 -- start start 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faecc096250 0x7faecc08bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 0x7faecc08a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc08bac0 con 0x7faecc096650 2021-02-10T11:43:15.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc08d040 con 0x7faecc08cc00 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed321e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc08d260 con 0x7faecc096250 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed0839700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed0839700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50066/0 (socket says 172.21.2.117:50066) 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.305 7faed0839700 1 -- 172.21.2.117:0/3789169533 learned_addr learned my addr 172.21.2.117:0/3789169533 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed103a700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faecc096250 0x7faecc08bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 -- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faecc096250 msgr2=0x7faecc08bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faecc096250 0x7faecc08bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed183b700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 0x7faecc08a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 -- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 msgr2=0x7faecc08a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 0x7faecc08a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 -- 172.21.2.117:0/3789169533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc08a8d0 con 0x7faecc096650 2021-02-10T11:43:15.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed183b700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 0x7faecc08a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed0839700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7faec800a4c0 tx=0x7faec800a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faec800ac50 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec8004140 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faec8004590 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed321e700 1 -- 172.21.2.117:0/3789169533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc08a8d0 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.306 7faed321e700 1 -- 172.21.2.117:0/3789169533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc08a8d0 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.307 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faec8000ce0 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.307 7faebdffb700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 0x7faeac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.307 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faec8032760 con 0x7faecc096650 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.308 7faed103a700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 0x7faeac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.308 7faed103a700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 0x7faeac01abc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.338 7faed321e700 1 -- 172.21.2.117:0/3789169533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc0ede70 con 0x7faecc096650 2021-02-10T11:43:15.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.341 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faec80009b0 con 0x7faecc096650 2021-02-10T11:43:15.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.471 7faed321e700 1 -- 172.21.2.117:0/3789169533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faecc08d000 con 0x7faecc096650 2021-02-10T11:43:15.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.476 7faebdffb700 1 -- 172.21.2.117:0/3789169533 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7faec801f4d0 con 0x7faecc096650 2021-02-10T11:43:15.468 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:43:15.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.480 7faed321e700 1 -- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 msgr2=0x7faeac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.480 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 0x7faeac01abc0 crc :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.480 7faed321e700 1 -- 172.21.2.117:0/3789169533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 msgr2=0x7faecc08c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.480 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7faec800a4c0 tx=0x7faec800a860).stop 2021-02-10T11:43:15.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 -- 172.21.2.117:0/3789169533 shutdown_connections 2021-02-10T11:43:15.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faeac0186c0 0x7faeac01abc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faecc096650 0x7faecc08c5e0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faecc08cc00 0x7faecc08a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 --2- 172.21.2.117:0/3789169533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faecc096250 0x7faecc08bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 -- 172.21.2.117:0/3789169533 >> 172.21.2.117:0/3789169533 conn(0x7faecc084ac0 msgr2=0x7faecc19ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:15.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 -- 172.21.2.117:0/3789169533 shutdown_connections 2021-02-10T11:43:15.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.481 7faed321e700 1 -- 172.21.2.117:0/3789169533 wait complete. 2021-02-10T11:43:15.479 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.900 7f28cefa9700 1 Processor -- start 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.900 7f28cefa9700 1 -- start start 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28cefa9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f28c80e5240 0x7f28c8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28cefa9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c8085930 con 0x7f28c80e4e40 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28cefa9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c81b09e0 con 0x7f28c8084c60 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28cefa9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c80e5cc0 con 0x7f28c80e5240 2021-02-10T11:43:15.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28ccdc5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f28c80e4e40 0x7f28c81b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50324/0 (socket says 172.21.2.117:50324) 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28ccdc5700 1 -- 172.21.2.117:0/1160783175 learned_addr learned my addr 172.21.2.117:0/1160783175 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.901 7f28c7fff700 1 --2- 172.21.2.117:0/1160783175 >> v2:172.21.2.125:3300/0 conn(0x7f28c80e5240 0x7f28c8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c77fe700 1 -- 172.21.2.117:0/1160783175 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57124031 0 0) 0x7f28b0000ce0 con 0x7f28c80e4e40 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c77fe700 1 -- 172.21.2.117:0/1160783175 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b4002fe0 con 0x7f28c80e4e40 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c7fff700 1 -- 172.21.2.117:0/1160783175 >> v1:172.21.2.125:6789/0 conn(0x7f28c8084c60 legacy=0x7f28c8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c7fff700 1 -- 172.21.2.117:0/1160783175 >> v1:172.21.2.117:6789/0 conn(0x7f28c80e4e40 legacy=0x7f28c81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c7fff700 1 -- 172.21.2.117:0/1160783175 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8084840 con 0x7f28c80e5240 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c7fff700 1 --2- 172.21.2.117:0/1160783175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80e5240 0x7f28c8086ec0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f28b80069d0 tx=0x7f28b8009e40).ready entity=mon.1 client_cookie=2ce013c822e5c124 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c77fe700 1 -- 172.21.2.117:0/1160783175 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b800b510 con 0x7f28c80e5240 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c77fe700 1 -- 172.21.2.117:0/1160783175 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b800b790 con 0x7f28c80e5240 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28c77fe700 1 -- 172.21.2.117:0/1160783175 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b8000d30 con 0x7f28c80e5240 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28cefa9700 1 -- 172.21.2.117:0/1160783175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80e5240 msgr2=0x7f28c8086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.902 7f28cefa9700 1 --2- 172.21.2.117:0/1160783175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80e5240 0x7f28c8086ec0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f28b80069d0 tx=0x7f28b8009e40).stop 2021-02-10T11:43:15.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.903 7f28cefa9700 1 -- 172.21.2.117:0/1160783175 shutdown_connections 2021-02-10T11:43:15.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.903 7f28cefa9700 1 --2- 172.21.2.117:0/1160783175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80e5240 0x7f28c8086ec0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.903 7f28cefa9700 1 -- 172.21.2.117:0/1160783175 >> 172.21.2.117:0/1160783175 conn(0x7f28c80ce4e0 msgr2=0x7f28c80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:15.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.903 7f28cefa9700 1 -- 172.21.2.117:0/1160783175 shutdown_connections 2021-02-10T11:43:15.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.903 7f28cefa9700 1 -- 172.21.2.117:0/1160783175 wait complete. 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 Processor -- start 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 -- start start 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b0700 0x7f28c818c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 0x7f28c8095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8097a10 con 0x7f28c8190f20 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c818c720 con 0x7f28c80985b0 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28cefa9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80989b0 con 0x7f28c80b0700 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28c7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28c7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50080/0 (socket says 172.21.2.117:50080) 2021-02-10T11:43:15.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.918 7f28c7fff700 1 -- 172.21.2.117:0/882222170 learned_addr learned my addr 172.21.2.117:0/882222170 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28ccdc5700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b0700 0x7f28c818c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28cd5c6700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 0x7f28c8095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 -- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b0700 msgr2=0x7f28c818c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b0700 0x7f28c818c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 -- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 msgr2=0x7f28c8095fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 0x7f28c8095fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 -- 172.21.2.117:0/882222170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80965f0 con 0x7f28c8190f20 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28cd5c6700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 0x7f28c8095fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c7fff700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f28b8009ec0 tx=0x7f28b800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b8006240 con 0x7f28c8190f20 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b8006540 con 0x7f28c8190f20 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28b80058e0 con 0x7f28c8190f20 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28cefa9700 1 -- 172.21.2.117:0/882222170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c80965f0 con 0x7f28c8190f20 2021-02-10T11:43:15.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.919 7f28cefa9700 1 -- 172.21.2.117:0/882222170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c8096b80 con 0x7f28c8190f20 2021-02-10T11:43:15.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.920 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28b8005660 con 0x7f28c8190f20 2021-02-10T11:43:15.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.920 7f28c57fa700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 0x7f28ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:15.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.920 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28b8032a40 con 0x7f28c8190f20 2021-02-10T11:43:15.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.920 7f28ccdc5700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 0x7f28ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:15.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.921 7f28ccdc5700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 0x7f28ac01abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:15.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.950 7f28cefa9700 1 -- 172.21.2.117:0/882222170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c810c250 con 0x7f28c8190f20 2021-02-10T11:43:15.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:15.953 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28b8029fc0 con 0x7f28c8190f20 2021-02-10T11:43:16.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.081 7f28cefa9700 1 -- 172.21.2.117:0/882222170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28c8085930 con 0x7f28c8190f20 2021-02-10T11:43:16.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.081 7f28c57fa700 1 -- 172.21.2.117:0/882222170 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f28b8029fc0 con 0x7f28c8190f20 2021-02-10T11:43:16.071 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 44 2021-02-10T11:43:16.072 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:16.072 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:16.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 msgr2=0x7f28ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 0x7f28ac01abc0 crc :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 msgr2=0x7f28c8097f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:16.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f28b8009ec0 tx=0x7f28b800bd40).stop 2021-02-10T11:43:16.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 shutdown_connections 2021-02-10T11:43:16.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28ac0186c0 0x7f28ac01abc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28c8190f20 0x7f28c8097f90 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28c80985b0 0x7f28c8095fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 --2- 172.21.2.117:0/882222170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28c80b0700 0x7f28c818c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 >> 172.21.2.117:0/882222170 conn(0x7f28c8199160 msgr2=0x7f28c808d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 shutdown_connections 2021-02-10T11:43:16.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:16.083 7f28cefa9700 1 -- 172.21.2.117:0/882222170 wait complete. 2021-02-10T11:43:16.283 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.674746', 'modified': '2021-02-10 11:43:10.684137', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/2942282551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 2942282551}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 2942282551}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:16.283 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:16.284 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 6670, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 8.0162323759999996, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 44, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 64, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:43:16.683 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 13.649864383000001 2021-02-10T11:43:16.684 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:16.687 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6670, "want_state": "up:active", "state": "up:active", "rank_uptime": 8.0162323759999996, "mdsmap_epoch": 44, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 13.649864383000001 } 2021-02-10T11:43:16.687 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:17.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 Processor -- start 2021-02-10T11:43:17.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 -- start start 2021-02-10T11:43:17.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5afc0ef3f0 0x7f5afc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0ede60 con 0x7f5afc0d18e0 2021-02-10T11:43:17.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0d1ce0 con 0x7f5afc0eb7f0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.109 7f5b04006700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0ef7f0 con 0x7f5afc0ef3f0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b01e22700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5afc0d18e0 0x7f5afc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50338/0 (socket says 172.21.2.117:50338) 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b01e22700 1 -- 172.21.2.117:0/1971915509 learned_addr learned my addr 172.21.2.117:0/1971915509 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788347466 0 0) 0x7f5af8001180 con 0x7f5afc0d18e0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8002fe0 con 0x7f5afc0d18e0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b01621700 1 --2- 172.21.2.117:0/1971915509 >> v2:172.21.2.125:3300/0 conn(0x7f5afc0ef3f0 0x7f5afc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1485559685 0 0) 0x7f5af8001180 con 0x7f5afc0d18e0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 >> v2:172.21.2.125:3300/0 conn(0x7f5afc0ef3f0 msgr2=0x7f5afc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 --2- 172.21.2.117:0/1971915509 >> v2:172.21.2.125:3300/0 conn(0x7f5afc0ef3f0 0x7f5afc0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 >> v1:172.21.2.125:6789/0 conn(0x7f5afc0eb7f0 legacy=0x7f5afc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc1bb180 con 0x7f5afc0d18e0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.110 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5af8001680 con 0x7f5afc0d18e0 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 >> v1:172.21.2.117:6789/0 conn(0x7f5afc0d18e0 legacy=0x7f5afc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ae8003a40 0x7f5ae8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 0x7f5ae8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ae8009580 0x7f5ae800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae800c2f0 con 0x7f5ae8009580 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae800c510 con 0x7f5ae8006780 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8050740 con 0x7f5ae8003a40 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01e22700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ae8003a40 0x7f5ae8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 0x7f5ae8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b02623700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ae8009580 0x7f5ae800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 -- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ae8003a40 msgr2=0x7f5ae8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ae8003a40 0x7f5ae8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 -- 172.21.2.117:0/1971915509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ae8009580 msgr2=0x7f5ae800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ae8009580 0x7f5ae800baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 -- 172.21.2.117:0/1971915509 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80509a0 con 0x7f5ae8006780 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b02623700 1 -- 172.21.2.117:0/1971915509 reap_dead start 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b01621700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 0x7f5ae8008c80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5af000c8f0 tx=0x7f5af000cc90).ready entity=mon.1 client_cookie=c50be6e68c9c0cbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af0009310 con 0x7f5ae8006780 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0009690 con 0x7f5ae8006780 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b00e20700 1 -- 172.21.2.117:0/1971915509 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af0009b70 con 0x7f5ae8006780 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b04006700 1 -- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 msgr2=0x7f5ae8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.111 7f5b04006700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 0x7f5ae8008c80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5af000c8f0 tx=0x7f5af000cc90).stop 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.112 7f5b04006700 1 -- 172.21.2.117:0/1971915509 shutdown_connections 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.112 7f5b04006700 1 --2- 172.21.2.117:0/1971915509 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ae8006780 0x7f5ae8008c80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.112 7f5b04006700 1 -- 172.21.2.117:0/1971915509 >> 172.21.2.117:0/1971915509 conn(0x7f5afc0ced60 msgr2=0x7f5afc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.112 7f5b04006700 1 -- 172.21.2.117:0/1971915509 shutdown_connections 2021-02-10T11:43:17.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.112 7f5b04006700 1 -- 172.21.2.117:0/1971915509 wait complete. 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 Processor -- start 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 -- start start 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5afc0b59a0 0x7f5afc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5afc0b5da0 0x7f5afc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0b61a0 con 0x7f5afc1a3480 2021-02-10T11:43:17.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc08c8b0 con 0x7f5afc0b59a0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b04006700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0ba1c0 con 0x7f5afc0b5da0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b02623700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b02623700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50102/0 (socket says 172.21.2.117:50102) 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b02623700 1 -- 172.21.2.117:0/2689748990 learned_addr learned my addr 172.21.2.117:0/2689748990 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.127 7f5b01621700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5afc0b5da0 0x7f5afc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b01e22700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5afc0b59a0 0x7f5afc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 -- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5afc0b5da0 msgr2=0x7f5afc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5afc0b5da0 0x7f5afc1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 -- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5afc0b59a0 msgr2=0x7f5afc08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5afc0b59a0 0x7f5afc08c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 -- 172.21.2.117:0/2689748990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc1a3880 con 0x7f5afc1a3480 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b02623700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f5af400a930 tx=0x7f5af400acd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af4000ce0 con 0x7f5afc1a3480 2021-02-10T11:43:17.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af400c300 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af400c7a0 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b04006700 1 -- 172.21.2.117:0/2689748990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc1a3880 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.128 7f5b04006700 1 -- 172.21.2.117:0/2689748990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1a3880 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.129 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5af4005a40 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.129 7f5aee7fc700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 0x7f5ae401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.129 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5af4032b70 con 0x7f5afc1a3480 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.129 7f5b01e22700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 0x7f5ae401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.130 7f5b01e22700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 0x7f5ae401abd0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.159 7f5b04006700 1 -- 172.21.2.117:0/2689748990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0ede60 con 0x7f5afc1a3480 2021-02-10T11:43:17.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.162 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5af4005830 con 0x7f5afc1a3480 2021-02-10T11:43:17.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.285 7f5b04006700 1 -- 172.21.2.117:0/2689748990 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5afc0ede60 con 0x7f5afc1a3480 2021-02-10T11:43:17.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.286 7f5aee7fc700 1 -- 172.21.2.117:0/2689748990 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f5af4005a40 con 0x7f5afc1a3480 2021-02-10T11:43:17.276 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:17.276 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 msgr2=0x7f5ae401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 0x7f5ae401abd0 crc :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 msgr2=0x7f5afc0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f5af400a930 tx=0x7f5af400acd0).stop 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 shutdown_connections 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ae40186d0 0x7f5ae401abd0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5afc1a3480 0x7f5afc0b2590 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5afc0b5da0 0x7f5afc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 --2- 172.21.2.117:0/2689748990 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5afc0b59a0 0x7f5afc08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 >> 172.21.2.117:0/2689748990 conn(0x7f5afc0bb5e0 msgr2=0x7f5afc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 shutdown_connections 2021-02-10T11:43:17.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.288 7f5b04006700 1 -- 172.21.2.117:0/2689748990 wait complete. 2021-02-10T11:43:17.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.649220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.719391', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:17.287 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.711 7f2084c8d700 1 Processor -- start 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.711 7f2084c8d700 1 -- start start 2021-02-10T11:43:17.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f2084c8d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800ef3f0 0x7f20800eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f2084c8d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20800eb830 0x7f20801b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f2084c8d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20800ef7f0 con 0x7f20800d18f0 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f2084c8d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20800eded0 con 0x7f20800eb830 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f2084c8d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20800d1cf0 con 0x7f20800ef3f0 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f207e85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f20800d18f0 0x7f20800eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50358/0 (socket says 172.21.2.117:50358) 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f207e85e700 1 -- 172.21.2.117:0/154175577 learned_addr learned my addr 172.21.2.117:0/154175577 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f207f05f700 1 --2- 172.21.2.117:0/154175577 >> v2:172.21.2.125:3300/0 conn(0x7f20800eb830 0x7f20801b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.712 7f207e05d700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800ef3f0 0x7f20800eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.713 7f207f05f700 1 -- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800ef3f0 msgr2=0x7f20800eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.713 7f207f05f700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800ef3f0 0x7f20800eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.713 7f207f05f700 1 -- 172.21.2.117:0/154175577 >> v1:172.21.2.117:6789/0 conn(0x7f20800d18f0 legacy=0x7f20800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.713 7f207f05f700 1 -- 172.21.2.117:0/154175577 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20801b8fd0 con 0x7f20800eb830 2021-02-10T11:43:17.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.713 7f207f05f700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20800eb830 0x7f20801b79a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f2078006e50 tx=0x7f2078009e70).ready entity=mon.1 client_cookie=fc83aaa15e63cc56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.714 7f207d85c700 1 -- 172.21.2.117:0/154175577 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f207800b510 con 0x7f20800eb830 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.714 7f207d85c700 1 -- 172.21.2.117:0/154175577 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2078006790 con 0x7f20800eb830 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.714 7f207d85c700 1 -- 172.21.2.117:0/154175577 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f207800b940 con 0x7f20800eb830 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.714 7f2084c8d700 1 -- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20800eb830 msgr2=0x7f20801b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.714 7f2084c8d700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20800eb830 0x7f20801b79a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f2078006e50 tx=0x7f2078009e70).stop 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 -- 172.21.2.117:0/154175577 shutdown_connections 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20800eb830 0x7f20801b79a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 --2- 172.21.2.117:0/154175577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800ef3f0 0x7f20800eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 -- 172.21.2.117:0/154175577 >> 172.21.2.117:0/154175577 conn(0x7f20800cf060 msgr2=0x7f20800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 -- 172.21.2.117:0/154175577 shutdown_connections 2021-02-10T11:43:17.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.715 7f2084c8d700 1 -- 172.21.2.117:0/154175577 wait complete. 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 Processor -- start 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 -- start start 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 0x7f208008c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800a9f90 0x7f20800aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20800aae30 con 0x7f208008cf40 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20800aa9f0 con 0x7f20801a48b0 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.730 7f2084c8d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20800ab330 con 0x7f20800a9f90 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50116/0 (socket says 172.21.2.117:50116) 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 -- 172.21.2.117:0/125031 learned_addr learned my addr 172.21.2.117:0/125031 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:17.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207f05f700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800a9f90 0x7f20800aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e85e700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 0x7f208008c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 -- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800a9f90 msgr2=0x7f20800aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800a9f90 0x7f20800aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 -- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 msgr2=0x7f208008c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 0x7f208008c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 -- 172.21.2.117:0/125031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20800ab550 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e85e700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 0x7f208008c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f207e05d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f2070006f60 tx=0x7f207000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2070006c10 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20700009b0 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f2084c8d700 1 -- 172.21.2.117:0/125031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20800ab550 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.731 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2070002030 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.732 7f2084c8d700 1 -- 172.21.2.117:0/125031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20800abae0 con 0x7f208008cf40 2021-02-10T11:43:17.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.732 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2070000c90 con 0x7f208008cf40 2021-02-10T11:43:17.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.732 7f20777fe700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 0x7f206401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:17.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.732 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f20700327a0 con 0x7f208008cf40 2021-02-10T11:43:17.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.733 7f207e85e700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 0x7f206401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:17.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.734 7f207e85e700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 0x7f206401ac10 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:17.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.761 7f2084c8d700 1 -- 172.21.2.117:0/125031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20800ede60 con 0x7f208008cf40 2021-02-10T11:43:17.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.764 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2070006c10 con 0x7f208008cf40 2021-02-10T11:43:17.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.890 7f2084c8d700 1 -- 172.21.2.117:0/125031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f20800aa9b0 con 0x7f208008cf40 2021-02-10T11:43:17.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.891 7f20777fe700 1 -- 172.21.2.117:0/125031 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2070000940 con 0x7f208008cf40 2021-02-10T11:43:17.881 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 msgr2=0x7f206401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 0x7f206401ac10 crc :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 msgr2=0x7f20800a9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f2070006f60 tx=0x7f207000b970).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 shutdown_connections 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2064018730 0x7f206401ac10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f208008cf40 0x7f20800a9970 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20800a9f90 0x7f20800aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 --2- 172.21.2.117:0/125031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20801a48b0 0x7f208008c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:17.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 >> 172.21.2.117:0/125031 conn(0x7f2080092d00 msgr2=0x7f208008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:17.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 shutdown_connections 2021-02-10T11:43:17.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:17.893 7f2084c8d700 1 -- 172.21.2.117:0/125031 wait complete. 2021-02-10T11:43:17.891 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.315 7f7478690700 1 Processor -- start 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.315 7f7478690700 1 -- start start 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7478690700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f74700d15e0 0x7f74700e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7478690700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700e7580 con 0x7f74700e7180 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7478690700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700e77f0 con 0x7f74701b07f0 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7478690700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74700e5ce0 con 0x7f74700d15e0 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7476cad700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f74700d15e0 0x7f74700e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7475cab700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f74700e7180 0x7f74700e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50370/0 (socket says 172.21.2.117:50370) 2021-02-10T11:43:18.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7475cab700 1 -- 172.21.2.117:0/1196463782 learned_addr learned my addr 172.21.2.117:0/1196463782 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7476cad700 1 -- 172.21.2.117:0/1196463782 >> v1:172.21.2.125:6789/0 conn(0x7f74701b07f0 legacy=0x7f74700e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7476cad700 1 -- 172.21.2.117:0/1196463782 >> v1:172.21.2.117:6789/0 conn(0x7f74700e7180 legacy=0x7f74700e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.316 7f7476cad700 1 -- 172.21.2.117:0/1196463782 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700d19e0 con 0x7f74700d15e0 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f7476cad700 1 --2- 172.21.2.117:0/1196463782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74700d15e0 0x7f74700e8560 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f7468006e30 tx=0x7f7468009de0).ready entity=mon.0 client_cookie=dccebf2d9f9dd0d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f74754aa700 1 -- 172.21.2.117:0/1196463782 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746800b300 con 0x7f74700d15e0 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f74754aa700 1 -- 172.21.2.117:0/1196463782 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746800b680 con 0x7f74700d15e0 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f74754aa700 1 -- 172.21.2.117:0/1196463782 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746800bba0 con 0x7f74700d15e0 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f7478690700 1 -- 172.21.2.117:0/1196463782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74700d15e0 msgr2=0x7f74700e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.317 7f7478690700 1 --2- 172.21.2.117:0/1196463782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74700d15e0 0x7f74700e8560 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f7468006e30 tx=0x7f7468009de0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.318 7f7478690700 1 -- 172.21.2.117:0/1196463782 shutdown_connections 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.318 7f7478690700 1 --2- 172.21.2.117:0/1196463782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74700d15e0 0x7f74700e8560 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.318 7f7478690700 1 -- 172.21.2.117:0/1196463782 >> 172.21.2.117:0/1196463782 conn(0x7f74700ce500 msgr2=0x7f74700e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:18.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.318 7f7478690700 1 -- 172.21.2.117:0/1196463782 shutdown_connections 2021-02-10T11:43:18.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.318 7f7478690700 1 -- 172.21.2.117:0/1196463782 wait complete. 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.332 7f7478690700 1 Processor -- start 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.332 7f7478690700 1 -- start start 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 0x7f74700a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74701b51a0 0x7f74701b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700b7480 con 0x7f7470084dc0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74701c6780 con 0x7f74700a23d0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7478690700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700849b0 con 0x7f74701b51a0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50126/0 (socket says 172.21.2.117:50126) 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 -- 172.21.2.117:0/2632776376 learned_addr learned my addr 172.21.2.117:0/2632776376 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7476cad700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74701b51a0 0x7f74701b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f7475cab700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 0x7f74700a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 -- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74701b51a0 msgr2=0x7f74701b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74701b51a0 0x7f74701b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 -- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 msgr2=0x7f74700a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 0x7f74700a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.333 7f74764ac700 1 -- 172.21.2.117:0/2632776376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74701b5bc0 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7475cab700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 0x7f74700a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f74764ac700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f746c00b5b0 tx=0x7f746c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746c00bd40 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c006020 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7478690700 1 -- 172.21.2.117:0/2632776376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74701b5bc0 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7478690700 1 -- 172.21.2.117:0/2632776376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74701b6150 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.334 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f746c0054a0 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.335 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f746c006200 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.335 7f7462ffd700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 0x7f745801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.335 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f746c032940 con 0x7f7470084dc0 2021-02-10T11:43:18.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.335 7f7475cab700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 0x7f745801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.336 7f7475cab700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 0x7f745801abc0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.365 7f7478690700 1 -- 172.21.2.117:0/2632776376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747007c020 con 0x7f7470084dc0 2021-02-10T11:43:18.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.368 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f746c000c10 con 0x7f7470084dc0 2021-02-10T11:43:18.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.494 7f7478690700 1 -- 172.21.2.117:0/2632776376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f74700d1c50 con 0x7f7470084dc0 2021-02-10T11:43:18.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.495 7f7462ffd700 1 -- 172.21.2.117:0/2632776376 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f746c00bd40 con 0x7f7470084dc0 2021-02-10T11:43:18.485 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:18.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 msgr2=0x7f745801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 0x7f745801abc0 crc :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 msgr2=0x7f74700b6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f746c00b5b0 tx=0x7f746c00b950).stop 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 shutdown_connections 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74701b51a0 0x7f74701b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74700a23d0 0x7f74700a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7458018730 0x7f745801abc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 --2- 172.21.2.117:0/2632776376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7470084dc0 0x7f74700b6e60 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 >> 172.21.2.117:0/2632776376 conn(0x7f74700b7910 msgr2=0x7f74700bd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 shutdown_connections 2021-02-10T11:43:18.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.497 7f7478690700 1 -- 172.21.2.117:0/2632776376 wait complete. 2021-02-10T11:43:18.495 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:18.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 Processor -- start 2021-02-10T11:43:18.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 -- start start 2021-02-10T11:43:18.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2cc0d18f0 0x7fd2cc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc0eb040 0x7fd2cc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2cc0eba60 con 0x7fd2cc0ee3e0 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2cc0d1cf0 con 0x7fd2cc0d18f0 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.919 7fd2d1c74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc0eefd0 con 0x7fd2cc0eb040 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2cc0d18f0 0x7fd2cc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd2cc0ee3e0 0x7fd2cc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50384/0 (socket says 172.21.2.117:50384) 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cbfff700 1 -- 172.21.2.117:0/1954721548 learned_addr learned my addr 172.21.2.117:0/1954721548 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 -- 172.21.2.117:0/1954721548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc0eb040 msgr2=0x7fd2cc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 --2- 172.21.2.117:0/1954721548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc0eb040 0x7fd2cc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 -- 172.21.2.117:0/1954721548 >> v1:172.21.2.117:6789/0 conn(0x7fd2cc0ee3e0 legacy=0x7fd2cc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 -- 172.21.2.117:0/1954721548 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc0ef1f0 con 0x7fd2cc0d18f0 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.920 7fd2cb7fe700 1 --2- 172.21.2.117:0/1954721548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc0d18f0 0x7fd2cc0eaa20 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7fd2bc006650 tx=0x7fd2bc009a80).ready entity=mon.0 client_cookie=aa7d0fe7474f1a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2ca7fc700 1 -- 172.21.2.117:0/1954721548 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2bc0009b0 con 0x7fd2cc0d18f0 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2ca7fc700 1 -- 172.21.2.117:0/1954721548 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2bc00c0c0 con 0x7fd2cc0d18f0 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2ca7fc700 1 -- 172.21.2.117:0/1954721548 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2bc00c5e0 con 0x7fd2cc0d18f0 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 -- 172.21.2.117:0/1954721548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc0d18f0 msgr2=0x7fd2cc0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 --2- 172.21.2.117:0/1954721548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc0d18f0 0x7fd2cc0eaa20 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0x7fd2bc006650 tx=0x7fd2bc009a80).stop 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 -- 172.21.2.117:0/1954721548 shutdown_connections 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 --2- 172.21.2.117:0/1954721548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc0eb040 0x7fd2cc0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 --2- 172.21.2.117:0/1954721548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc0d18f0 0x7fd2cc0eaa20 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.921 7fd2d1c74700 1 -- 172.21.2.117:0/1954721548 >> 172.21.2.117:0/1954721548 conn(0x7fd2cc0cf060 msgr2=0x7fd2cc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.922 7fd2d1c74700 1 -- 172.21.2.117:0/1954721548 shutdown_connections 2021-02-10T11:43:18.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.922 7fd2d1c74700 1 -- 172.21.2.117:0/1954721548 wait complete. 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 Processor -- start 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 -- start start 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 0x7fd2cc0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 0x7fd2cc07ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc07f890 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc07f310 con 0x7fd2cc0a67e0 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.936 7fd2d1c74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc1a3870 con 0x7fd2cc07e7d0 2021-02-10T11:43:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50138/0 (socket says 172.21.2.117:50138) 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 -- 172.21.2.117:0/3268600868 learned_addr learned my addr 172.21.2.117:0/3268600868 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 -- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 msgr2=0x7fd2cc07ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2caffd700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 0x7fd2cc07ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cb7fe700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 0x7fd2cc0a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 0x7fd2cc07ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 -- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 msgr2=0x7fd2cc0a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 0x7fd2cc0a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 -- 172.21.2.117:0/3268600868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc1a3a90 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2caffd700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 0x7fd2cc07ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cb7fe700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 0x7fd2cc0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.937 7fd2cbfff700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7fd2c400a940 tx=0x7fd2c400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c4001270 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c40070b0 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2c4007480 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc1a3a90 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc1a3a90 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2c4007290 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.938 7fd2b3fff700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 0x7fd2ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.939 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd2c4032a00 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.939 7fd2cb7fe700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 0x7fd2ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:18.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.940 7fd2cb7fe700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 0x7fd2ac01abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:18.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.969 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2cc0ede60 con 0x7fd2cc1a2e50 2021-02-10T11:43:18.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:18.972 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2c4007cb0 con 0x7fd2cc1a2e50 2021-02-10T11:43:19.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.099 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd2cc0ede60 con 0x7fd2cc1a2e50 2021-02-10T11:43:19.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.099 7fd2b3fff700 1 -- 172.21.2.117:0/3268600868 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd2c4007290 con 0x7fd2cc1a2e50 2021-02-10T11:43:19.090 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:19.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.101 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 msgr2=0x7fd2ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.101 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 0x7fd2ac01abc0 crc :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.101 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 msgr2=0x7fd2cc1a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.101 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7fd2c400a940 tx=0x7fd2c400ace0).stop 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 shutdown_connections 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ac0186c0 0x7fd2ac01abc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2cc1a2e50 0x7fd2cc1a3250 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2cc07e7d0 0x7fd2cc07ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 --2- 172.21.2.117:0/3268600868 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2cc0a67e0 0x7fd2cc0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 >> 172.21.2.117:0/3268600868 conn(0x7fd2cc09e660 msgr2=0x7fd2cc0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 shutdown_connections 2021-02-10T11:43:19.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.102 7fd2d1c74700 1 -- 172.21.2.117:0/3268600868 wait complete. 2021-02-10T11:43:19.100 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:19.101 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:19.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.519 7fc03cf87700 1 Processor -- start 2021-02-10T11:43:19.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.519 7fc03cf87700 1 -- start start 2021-02-10T11:43:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03cf87700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc0381b6960 0x7fc0381b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03cf87700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0380ede60 con 0x7fc0380ef3f0 2021-02-10T11:43:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03cf87700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0380eb9c0 con 0x7fc0380d18e0 2021-02-10T11:43:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03cf87700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0380ef7f0 con 0x7fc0381b6960 2021-02-10T11:43:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03759e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc0381b6960 0x7fc0381b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03659c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc0380ef3f0 0x7fc0380eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50396/0 (socket says 172.21.2.117:50396) 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03659c700 1 -- 172.21.2.117:0/464943591 learned_addr learned my addr 172.21.2.117:0/464943591 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03759e700 1 -- 172.21.2.117:0/464943591 >> v1:172.21.2.125:6789/0 conn(0x7fc0380d18e0 legacy=0x7fc0380ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03759e700 1 -- 172.21.2.117:0/464943591 >> v1:172.21.2.117:6789/0 conn(0x7fc0380ef3f0 legacy=0x7fc0380eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.520 7fc03759e700 1 -- 172.21.2.117:0/464943591 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0380d1d20 con 0x7fc0381b6960 2021-02-10T11:43:19.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc03759e700 1 --2- 172.21.2.117:0/464943591 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0381b6960 0x7fc0381b8d70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fc030006e50 tx=0x7fc030009df0).ready entity=mon.1 client_cookie=f73c52ebd5e28bf5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc035d9b700 1 -- 172.21.2.117:0/464943591 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc03000b510 con 0x7fc0381b6960 2021-02-10T11:43:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc035d9b700 1 -- 172.21.2.117:0/464943591 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc030006790 con 0x7fc0381b6960 2021-02-10T11:43:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc035d9b700 1 -- 172.21.2.117:0/464943591 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc03000b750 con 0x7fc0381b6960 2021-02-10T11:43:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc03cf87700 1 -- 172.21.2.117:0/464943591 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0381b6960 msgr2=0x7fc0381b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.521 7fc03cf87700 1 --2- 172.21.2.117:0/464943591 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0381b6960 0x7fc0381b8d70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fc030006e50 tx=0x7fc030009df0).stop 2021-02-10T11:43:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.522 7fc03cf87700 1 -- 172.21.2.117:0/464943591 shutdown_connections 2021-02-10T11:43:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.522 7fc03cf87700 1 --2- 172.21.2.117:0/464943591 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0381b6960 0x7fc0381b8d70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.522 7fc03cf87700 1 -- 172.21.2.117:0/464943591 >> 172.21.2.117:0/464943591 conn(0x7fc0380ced60 msgr2=0x7fc0380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.522 7fc03cf87700 1 -- 172.21.2.117:0/464943591 shutdown_connections 2021-02-10T11:43:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.522 7fc03cf87700 1 -- 172.21.2.117:0/464943591 wait complete. 2021-02-10T11:43:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.536 7fc03cf87700 1 Processor -- start 2021-02-10T11:43:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.536 7fc03cf87700 1 -- start start 2021-02-10T11:43:19.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0380f2b10 0x7fc0380b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0380b27a0 0x7fc038097b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0380b2ba0 con 0x7fc038098170 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0380f2f10 con 0x7fc0380f2b10 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03cf87700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc038098570 con 0x7fc0380b27a0 2021-02-10T11:43:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50150/0 (socket says 172.21.2.117:50150) 2021-02-10T11:43:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 -- 172.21.2.117:0/219138123 learned_addr learned my addr 172.21.2.117:0/219138123 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03659c700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0380b27a0 0x7fc038097b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc036d9d700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0380f2b10 0x7fc0380b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 -- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0380b27a0 msgr2=0x7fc038097b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0380b27a0 0x7fc038097b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 -- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0380f2b10 msgr2=0x7fc0380b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0380f2b10 0x7fc0380b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 -- 172.21.2.117:0/219138123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc038098790 con 0x7fc038098170 2021-02-10T11:43:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.537 7fc03759e700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fc030006fa0 tx=0x7fc03000bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.538 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc0300093d0 con 0x7fc038098170 2021-02-10T11:43:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.538 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0300082e0 con 0x7fc038098170 2021-02-10T11:43:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.538 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc030008800 con 0x7fc038098170 2021-02-10T11:43:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.538 7fc03cf87700 1 -- 172.21.2.117:0/219138123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc038098790 con 0x7fc038098170 2021-02-10T11:43:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.538 7fc03cf87700 1 -- 172.21.2.117:0/219138123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc038098790 con 0x7fc038098170 2021-02-10T11:43:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.539 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc0300084c0 con 0x7fc038098170 2021-02-10T11:43:19.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.539 7fc02f7fe700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:19.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.539 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc030032cc0 con 0x7fc038098170 2021-02-10T11:43:19.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.539 7fc036d9d700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:19.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.539 7fc036d9d700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 0x7fc01c01abc0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:19.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.570 7fc03cf87700 1 -- 172.21.2.117:0/219138123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc038086140 con 0x7fc038098170 2021-02-10T11:43:19.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.572 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc030008030 con 0x7fc038098170 2021-02-10T11:43:19.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.700 7fc03cf87700 1 -- 172.21.2.117:0/219138123 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0380d18e0 con 0x7fc038098170 2021-02-10T11:43:19.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.701 7fc02f7fe700 1 -- 172.21.2.117:0/219138123 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fc0300082e0 con 0x7fc038098170 2021-02-10T11:43:19.691 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:19.691 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:19.691 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 44 2021-02-10T11:43:19.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 msgr2=0x7fc01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 0x7fc01c01abc0 crc :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 msgr2=0x7fc0381c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fc030006fa0 tx=0x7fc03000bd90).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 shutdown_connections 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc01c0186c0 0x7fc01c01abc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc038098170 0x7fc0381c4e50 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0380b27a0 0x7fc038097b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 --2- 172.21.2.117:0/219138123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0380f2b10 0x7fc0380b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:19.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 >> 172.21.2.117:0/219138123 conn(0x7fc0380f3230 msgr2=0x7fc0380b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:19.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 shutdown_connections 2021-02-10T11:43:19.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:19.703 7fc03cf87700 1 -- 172.21.2.117:0/219138123 wait complete. 2021-02-10T11:43:19.904 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.674746', 'modified': '2021-02-10 11:43:10.684137', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/2942282551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 2942282551}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 2942282551}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:19.904 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:19.904 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:43:20.304 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 6670, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 11.637762198000001, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 44, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 64, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 17.271395562999999 2021-02-10T11:43:20.305 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:20.309 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6670, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.637762198000001, "mdsmap_epoch": 44, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 17.271395562999999 } 2021-02-10T11:43:20.309 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:20.309 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:43:20.309 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:43:20.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:20.310 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:20.644 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:20.988 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:20.988 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:21.331 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:21.331 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:21.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:21.332 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:21.667 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:43:21.667 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:21.670 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:22.017 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:22.017 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:22.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[135245]: starting ceph client 2021-02-10T11:43:22.083 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:43:22.093 7f9cb6800f80 -1 init, newargv = 0x56360f8a1f30 newargc=9 2021-02-10T11:43:22.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[135245]: starting fuse 2021-02-10T11:43:22.358 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:22.359 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:22.359 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:22.360 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:22.693 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:43:22.693 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:43:22.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:22.694 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:43:22.695 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:43:23.058 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.135245.asok 2021-02-10T11:43:23.060 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.135245.asok 2021-02-10T11:43:23.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:23.061 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.135245.asok status 2021-02-10T11:43:23.449 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:23.449 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "135245", 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8592, 2021-02-10T11:43:23.450 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8592 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3415505532 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:23.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3415505532 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8592 172.21.2.117:0/3415505532", 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3415505532", 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 44, 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 64, 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:43:23.452 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:23.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:23.454 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:23.454 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:23.789 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:43:23.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:43:23.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:23.789 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:43:24.135 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 Processor -- start 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 -- start start 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340ef3f0 0x7f8d340eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8d340eb7f0 0x7f8d341b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340ede60 con 0x7f8d340d18e0 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d340ef7f0 con 0x7f8d340eb7f0 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d3b0e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340d1ce0 con 0x7f8d340ef3f0 2021-02-10T11:43:24.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d38f05700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8d340d18e0 0x7f8d340ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50422/0 (socket says 172.21.2.117:50422) 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.557 7f8d38f05700 1 -- 172.21.2.117:0/1874093781 learned_addr learned my addr 172.21.2.117:0/1874093781 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536804107 0 0) 0x7f8d1c001180 con 0x7f8d340d18e0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d20002fe0 con 0x7f8d340d18e0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d39706700 1 --2- 172.21.2.117:0/1874093781 >> v2:172.21.2.125:3300/0 conn(0x7f8d340eb7f0 0x7f8d341b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d33fff700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340ef3f0 0x7f8d340eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1104201866 0 0) 0x7f8d1c001180 con 0x7f8d340d18e0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340ef3f0 msgr2=0x7f8d340eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340ef3f0 0x7f8d340eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 >> v2:172.21.2.125:3300/0 conn(0x7f8d340eb7f0 msgr2=0x7f8d341b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 --2- 172.21.2.117:0/1874093781 >> v2:172.21.2.125:3300/0 conn(0x7f8d340eb7f0 0x7f8d341b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d341b8f90 con 0x7f8d340d18e0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8d1c001680 con 0x7f8d340d18e0 2021-02-10T11:43:24.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 >> v1:172.21.2.117:6789/0 conn(0x7f8d340d18e0 legacy=0x7f8d340ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d20003b10 0x7f8d20006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d2004a830 0x7f8d2004cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2004d5a0 0x7f8d2004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d200502e0 con 0x7f8d2004a830 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20050500 con 0x7f8d20003b10 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20050720 con 0x7f8d2004d5a0 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.558 7f8d33fff700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d2004a830 0x7f8d2004cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d39706700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2004d5a0 0x7f8d2004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d38f05700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d20003b10 0x7f8d20006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 -- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2004d5a0 msgr2=0x7f8d2004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2004d5a0 0x7f8d2004fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 -- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d20003b10 msgr2=0x7f8d20006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:24.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d20003b10 0x7f8d20006030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 -- 172.21.2.117:0/1874093781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200509c0 con 0x7f8d2004a830 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d33fff700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d2004a830 0x7f8d2004cca0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f8d24009310 tx=0x7f8d24009a90).ready entity=mon.0 client_cookie=9a59dbe9198477de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2400c0c0 con 0x7f8d2004a830 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d24006400 con 0x7f8d2004a830 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d337fe700 1 -- 172.21.2.117:0/1874093781 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2400c340 con 0x7f8d2004a830 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d3b0e9700 1 -- 172.21.2.117:0/1874093781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d2004a830 msgr2=0x7f8d2004cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d3b0e9700 1 --2- 172.21.2.117:0/1874093781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d2004a830 0x7f8d2004cca0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f8d24009310 tx=0x7f8d24009a90).stop 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.559 7f8d39706700 1 -- 172.21.2.117:0/1874093781 reap_dead start 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.560 7f8d3b0e9700 1 -- 172.21.2.117:0/1874093781 shutdown_connections 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.560 7f8d3b0e9700 1 -- 172.21.2.117:0/1874093781 >> 172.21.2.117:0/1874093781 conn(0x7f8d340ced60 msgr2=0x7f8d340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.560 7f8d3b0e9700 1 -- 172.21.2.117:0/1874093781 shutdown_connections 2021-02-10T11:43:24.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.560 7f8d3b0e9700 1 -- 172.21.2.117:0/1874093781 wait complete. 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.574 7f8d3b0e9700 1 Processor -- start 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.574 7f8d3b0e9700 1 -- start start 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d340c10e0 0x7f8d340a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340a9530 0x7f8d340a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340e12b0 con 0x7f8d340e9690 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340a9f50 con 0x7f8d340c10e0 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d3b0e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d340e9a90 con 0x7f8d340a9530 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50184/0 (socket says 172.21.2.117:50184) 2021-02-10T11:43:24.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 -- 172.21.2.117:0/4138148998 learned_addr learned my addr 172.21.2.117:0/4138148998 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:24.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 -- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340a9530 msgr2=0x7f8d340a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340a9530 0x7f8d340a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 -- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d340c10e0 msgr2=0x7f8d340a8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d340c10e0 0x7f8d340a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 -- 172.21.2.117:0/4138148998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d341a4750 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.575 7f8d38f05700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f8d1c009610 tx=0x7f8d1c0099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c0070c0 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c0043a0 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c004880 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d341a4750 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d340e00a0 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d1c008000 con 0x7f8d340e9690 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 0x7f8d1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:24.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.576 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d1c032690 con 0x7f8d340e9690 2021-02-10T11:43:24.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.577 7f8d33fff700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 0x7f8d1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:24.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.577 7f8d33fff700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 0x7f8d1801abc0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:24.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.607 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d34087210 con 0x7f8d340e9690 2021-02-10T11:43:24.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.610 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d1c009da0 con 0x7f8d340e9690 2021-02-10T11:43:24.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.734 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d340ede60 con 0x7f8d340e9690 2021-02-10T11:43:24.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.734 7f8d317fa700 1 -- 172.21.2.117:0/4138148998 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f8d1c008000 con 0x7f8d340e9690 2021-02-10T11:43:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:24.726 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:24.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 msgr2=0x7f8d1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 0x7f8d1801abc0 crc :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 msgr2=0x7f8d340c0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:24.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f8d1c009610 tx=0x7f8d1c0099b0).stop 2021-02-10T11:43:24.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 shutdown_connections 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d340a9530 0x7f8d340a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d340c10e0 0x7f8d340a8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d180186c0 0x7f8d1801abc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 --2- 172.21.2.117:0/4138148998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d340e9690 0x7f8d340c0ac0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 >> 172.21.2.117:0/4138148998 conn(0x7f8d34098d40 msgr2=0x7f8d340ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 shutdown_connections 2021-02-10T11:43:24.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:24.736 7f8d3b0e9700 1 -- 172.21.2.117:0/4138148998 wait complete. 2021-02-10T11:43:24.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.649220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.719391', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:24.736 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 Processor -- start 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 -- start start 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f40ef3f0 0x7fd6f40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6f40eb830 0x7fd6f41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40ef7f0 con 0x7fd6f40d18f0 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f40eded0 con 0x7fd6f40eb830 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6fa586700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40d1cf0 con 0x7fd6f40ef3f0 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6f8ba3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6f40eb830 0x7fd6f41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6f3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd6f40d18f0 0x7fd6f40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50442/0 (socket says 172.21.2.117:50442) 2021-02-10T11:43:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.162 7fd6f3fff700 1 -- 172.21.2.117:0/965685390 learned_addr learned my addr 172.21.2.117:0/965685390 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f37fe700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f40ef3f0 0x7fd6f40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f8ba3700 1 -- 172.21.2.117:0/965685390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f40ef3f0 msgr2=0x7fd6f40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f8ba3700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f40ef3f0 0x7fd6f40eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f8ba3700 1 -- 172.21.2.117:0/965685390 >> v1:172.21.2.117:6789/0 conn(0x7fd6f40d18f0 legacy=0x7fd6f40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f8ba3700 1 -- 172.21.2.117:0/965685390 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f41b8fd0 con 0x7fd6f40eb830 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f8ba3700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40eb830 0x7fd6f41b79a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd6e4006ab0 tx=0x7fd6e4009b40).ready entity=mon.0 client_cookie=ba886dbf3b9b8462 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f2ffd700 1 -- 172.21.2.117:0/965685390 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6e40022e0 con 0x7fd6f40eb830 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f2ffd700 1 -- 172.21.2.117:0/965685390 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4002660 con 0x7fd6f40eb830 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6f2ffd700 1 -- 172.21.2.117:0/965685390 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6e4002b40 con 0x7fd6f40eb830 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6fa586700 1 -- 172.21.2.117:0/965685390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40eb830 msgr2=0x7fd6f41b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.163 7fd6fa586700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40eb830 0x7fd6f41b79a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd6e4006ab0 tx=0x7fd6e4009b40).stop 2021-02-10T11:43:25.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 -- 172.21.2.117:0/965685390 shutdown_connections 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40eb830 0x7fd6f41b79a0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 --2- 172.21.2.117:0/965685390 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f40ef3f0 0x7fd6f40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 -- 172.21.2.117:0/965685390 >> 172.21.2.117:0/965685390 conn(0x7fd6f40cf060 msgr2=0x7fd6f40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 -- 172.21.2.117:0/965685390 shutdown_connections 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.164 7fd6fa586700 1 -- 172.21.2.117:0/965685390 wait complete. 2021-02-10T11:43:25.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 Processor -- start 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 -- start start 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 0x7fd6f40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6f40a2040 0x7fd6f40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40a67e0 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40a7120 con 0x7fd6f40a2040 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.179 7fd6fa586700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40a2a60 con 0x7fd6f4085ec0 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 0x7fd6f40dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47882/0 (socket says 172.21.2.117:47882) 2021-02-10T11:43:25.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f37fe700 1 -- 172.21.2.117:0/3637693626 learned_addr learned my addr 172.21.2.117:0/3637693626 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f8ba3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6f40a2040 0x7fd6f40a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 -- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 msgr2=0x7fd6f40862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 -- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6f40a2040 msgr2=0x7fd6f40a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6f40a2040 0x7fd6f40a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 -- 172.21.2.117:0/3637693626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f41a2dc0 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f37fe700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f3fff700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 0x7fd6f40dc420 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fd6ec00a870 tx=0x7fd6ec00ac10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6ec001270 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec007160 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6ec004720 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.180 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f41a2dc0 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.181 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f41a3350 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.181 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd6ec007340 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.181 7fd6f0ff9700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 0x7fd6d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.181 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd6ec032980 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.182 7fd6f37fe700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 0x7fd6d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.182 7fd6f37fe700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 0x7fd6d401abc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.212 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f40daff0 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.215 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd6ec000d30 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.342 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fd6f40ede60 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.342 7fd6f0ff9700 1 -- 172.21.2.117:0/3637693626 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd6ec007340 con 0x7fd6f40a6ce0 2021-02-10T11:43:25.334 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:25.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 msgr2=0x7fd6d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 0x7fd6d401abc0 crc :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 msgr2=0x7fd6f40dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 0x7fd6f40dc420 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7fd6ec00a870 tx=0x7fd6ec00ac10).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 shutdown_connections 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd6f40a2040 0x7fd6f40a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd6f4085ec0 0x7fd6f40862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6d40186c0 0x7fd6d401abc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.344 7fd6fa586700 1 --2- 172.21.2.117:0/3637693626 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6f40a6ce0 0x7fd6f40dc420 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.345 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 >> 172.21.2.117:0/3637693626 conn(0x7fd6f40b6540 msgr2=0x7fd6f408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.345 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 shutdown_connections 2021-02-10T11:43:25.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.345 7fd6fa586700 1 -- 172.21.2.117:0/3637693626 wait complete. 2021-02-10T11:43:25.344 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:25.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 Processor -- start 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 -- start start 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 0x7f2a480eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480ede60 con 0x7f2a480eb040 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480d1cf0 con 0x7f2a480ef3f0 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.766 7f2a4cd88700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a48088020 con 0x7f2a480d18f0 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2a480eb040 0x7f2a480eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50458/0 (socket says 172.21.2.117:50458) 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46312700 1 -- 172.21.2.117:0/3382161589 learned_addr learned my addr 172.21.2.117:0/3382161589 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46b13700 1 --2- 172.21.2.117:0/3382161589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 0x7f2a480eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677245129 0 0) 0x7f2a380010b0 con 0x7f2a480eb040 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a34002fe0 con 0x7f2a480eb040 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250039650 0 0) 0x7f2a40000f90 con 0x7f2a480ef3f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a34004370 con 0x7f2a480ef3f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46b13700 1 -- 172.21.2.117:0/3382161589 >> v1:172.21.2.125:6789/0 conn(0x7f2a480ef3f0 legacy=0x7f2a481b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46b13700 1 -- 172.21.2.117:0/3382161589 >> v1:172.21.2.117:6789/0 conn(0x7f2a480eb040 legacy=0x7f2a480eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46b13700 1 -- 172.21.2.117:0/3382161589 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a481bb200 con 0x7f2a480d18f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a46b13700 1 --2- 172.21.2.117:0/3382161589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 0x7f2a480eaa20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a30006650 tx=0x7f2a30009a80).ready entity=mon.2 client_cookie=6b0277a1b1efef31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a300009b0 con 0x7f2a480d18f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30006740 con 0x7f2a480d18f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.767 7f2a45b11700 1 -- 172.21.2.117:0/3382161589 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a3000c0c0 con 0x7f2a480d18f0 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 -- 172.21.2.117:0/3382161589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 msgr2=0x7f2a480eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 --2- 172.21.2.117:0/3382161589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 0x7f2a480eaa20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2a30006650 tx=0x7f2a30009a80).stop 2021-02-10T11:43:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 -- 172.21.2.117:0/3382161589 shutdown_connections 2021-02-10T11:43:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 --2- 172.21.2.117:0/3382161589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a480d18f0 0x7f2a480eaa20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 -- 172.21.2.117:0/3382161589 >> 172.21.2.117:0/3382161589 conn(0x7f2a480cf060 msgr2=0x7f2a480cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 -- 172.21.2.117:0/3382161589 shutdown_connections 2021-02-10T11:43:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.768 7f2a4cd88700 1 -- 172.21.2.117:0/3382161589 wait complete. 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 Processor -- start 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 -- start start 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a481a3930 0x7f2a480a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a480a13a0 0x7f2a480a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4807cca0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4818fb70 con 0x7f2a480a13a0 2021-02-10T11:43:25.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a4cd88700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a480dc6d0 con 0x7f2a481a3930 2021-02-10T11:43:25.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a46b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a46b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50212/0 (socket says 172.21.2.117:50212) 2021-02-10T11:43:25.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.783 7f2a46b13700 1 -- 172.21.2.117:0/224941798 learned_addr learned my addr 172.21.2.117:0/224941798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46312700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a481a3930 0x7f2a480a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a47314700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a480a13a0 0x7f2a480a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 -- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a481a3930 msgr2=0x7f2a480a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a481a3930 0x7f2a480a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 -- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a480a13a0 msgr2=0x7f2a480a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a480a13a0 0x7f2a480a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 -- 172.21.2.117:0/224941798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4807d220 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a46b13700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f2a30009650 tx=0x7f2a3000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a300093a0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3000bde0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4807d220 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a300052e0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.784 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4807d220 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.785 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a30006030 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.785 7f2a3f7fe700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 0x7f2a2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.785 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2a300326d0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.785 7f2a46312700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 0x7f2a2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.786 7f2a46312700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 0x7f2a2c01abc0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:25.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.816 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4818fb00 con 0x7f2a4807c7c0 2021-02-10T11:43:25.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.819 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a300009c0 con 0x7f2a4807c7c0 2021-02-10T11:43:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.949 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2a4818fb00 con 0x7f2a4807c7c0 2021-02-10T11:43:25.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.949 7f2a3f7fe700 1 -- 172.21.2.117:0/224941798 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2a30006030 con 0x7f2a4807c7c0 2021-02-10T11:43:25.941 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.951 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 msgr2=0x7f2a2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.951 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 0x7f2a2c01abc0 crc :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.951 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 msgr2=0x7f2a481a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.951 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f2a30009650 tx=0x7f2a3000b960).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 shutdown_connections 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a480a13a0 0x7f2a480a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a481a3930 0x7f2a480a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a2c0186c0 0x7f2a2c01abc0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 --2- 172.21.2.117:0/224941798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a4807c7c0 0x7f2a481a3310 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 >> 172.21.2.117:0/224941798 conn(0x7f2a48087090 msgr2=0x7f2a4808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 shutdown_connections 2021-02-10T11:43:25.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:25.952 7f2a4cd88700 1 -- 172.21.2.117:0/224941798 wait complete. 2021-02-10T11:43:25.951 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:26.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 Processor -- start 2021-02-10T11:43:26.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 -- start start 2021-02-10T11:43:26.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef40ef7f0 con 0x7f8ef40d18f0 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef40eded0 con 0x7f8ef40ef3f0 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.373 7f8efb8c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40d1cf0 con 0x7f8ef40eb830 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.374 7f8ef9ee6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.374 7f8ef8ee4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.374 7f8ef9ee6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:47904/0 (socket says 172.21.2.117:47904) 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.374 7f8ef8ee4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54604/0 (socket says 172.21.2.117:54604) 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.374 7f8ef9ee6700 1 -- 172.21.2.117:0/384884236 learned_addr learned my addr 172.21.2.117:0/384884236 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef9ee6700 1 -- 172.21.2.117:0/384884236 >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 msgr2=0x7f8ef40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef9ee6700 1 --2- 172.21.2.117:0/384884236 >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef9ee6700 1 -- 172.21.2.117:0/384884236 >> v1:172.21.2.125:6789/0 conn(0x7f8ef40d18f0 legacy=0x7f8ef40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef9ee6700 1 -- 172.21.2.117:0/384884236 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef41b8fd0 con 0x7f8ef40eb830 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef8ee4700 1 --2- 172.21.2.117:0/384884236 >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ef9ee6700 1 --2- 172.21.2.117:0/384884236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8eec006aa0 tx=0x7f8eec009f20).ready entity=mon.2 client_cookie=65e9ff0bb3284c9b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ee7fff700 1 -- 172.21.2.117:0/384884236 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eec000f90 con 0x7f8ef40eb830 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ee7fff700 1 -- 172.21.2.117:0/384884236 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec003030 con 0x7f8ef40eb830 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.375 7f8ee7fff700 1 -- 172.21.2.117:0/384884236 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eec00b7b0 con 0x7f8ef40eb830 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 -- 172.21.2.117:0/384884236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 msgr2=0x7f8ef41b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 --2- 172.21.2.117:0/384884236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8eec006aa0 tx=0x7f8eec009f20).stop 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 -- 172.21.2.117:0/384884236 shutdown_connections 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 --2- 172.21.2.117:0/384884236 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef40eb830 0x7f8ef41b79a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 --2- 172.21.2.117:0/384884236 >> v2:172.21.2.125:3300/0 conn(0x7f8ef40ef3f0 0x7f8ef40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 -- 172.21.2.117:0/384884236 >> 172.21.2.117:0/384884236 conn(0x7f8ef40cf060 msgr2=0x7f8ef40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 -- 172.21.2.117:0/384884236 shutdown_connections 2021-02-10T11:43:26.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.376 7f8efb8c9700 1 -- 172.21.2.117:0/384884236 wait complete. 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 Processor -- start 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 -- start start 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef409ed70 0x7f8ef409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ef409f790 0x7f8ef40b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef409fb90 con 0x7f8ef41a4240 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40dc640 con 0x7f8ef409f790 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8efb8c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4097270 con 0x7f8ef409ed70 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8ef96e5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8ef96e5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50228/0 (socket says 172.21.2.117:50228) 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.391 7f8ef96e5700 1 -- 172.21.2.117:0/3424197014 learned_addr learned my addr 172.21.2.117:0/3424197014 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:26.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 -- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef409ed70 msgr2=0x7f8ef409f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef409ed70 0x7f8ef409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 -- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ef409f790 msgr2=0x7f8ef40b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ef409f790 0x7f8ef40b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 -- 172.21.2.117:0/3424197014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef40b8780 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ef96e5700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f8ef0006ed0 tx=0x7f8ef000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef0006240 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef00009b0 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ef0005730 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef40b8780 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.392 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef40b8d10 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.393 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ef0006030 con 0x7f8ef41a4240 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.393 7f8ee5ffb700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 0x7f8ed401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.393 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ef0032670 con 0x7f8ef41a4240 2021-02-10T11:43:26.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.393 7f8ef8ee4700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 0x7f8ed401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.394 7f8ef8ee4700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 0x7f8ed401abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:26.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.423 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef40daff0 con 0x7f8ef41a4240 2021-02-10T11:43:26.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.426 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ef0020f40 con 0x7f8ef41a4240 2021-02-10T11:43:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.552 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8ef40ede60 con 0x7f8ef41a4240 2021-02-10T11:43:26.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.552 7f8ee5ffb700 1 -- 172.21.2.117:0/3424197014 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8ef0020f40 con 0x7f8ef41a4240 2021-02-10T11:43:26.544 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 msgr2=0x7f8ed401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 0x7f8ed401abc0 crc :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 msgr2=0x7f8ef41a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f8ef0006ed0 tx=0x7f8ef000b960).stop 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 shutdown_connections 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ef409f790 0x7f8ef40b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ef409ed70 0x7f8ef409f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ed40186c0 0x7f8ed401abc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 --2- 172.21.2.117:0/3424197014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ef41a4240 0x7f8ef41a4640 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.554 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 >> 172.21.2.117:0/3424197014 conn(0x7f8ef4087090 msgr2=0x7f8ef408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.555 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 shutdown_connections 2021-02-10T11:43:26.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.555 7f8efb8c9700 1 -- 172.21.2.117:0/3424197014 wait complete. 2021-02-10T11:43:26.554 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:26.554 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:26.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.973 7f540b2b6700 1 Processor -- start 2021-02-10T11:43:26.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.973 7f540b2b6700 1 -- start start 2021-02-10T11:43:26.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f54040e4e40 0x7f54041b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54040d1db0 con 0x7f54040e5240 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5404085930 con 0x7f5404085450 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5404084840 con 0x7f54040e4e40 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54090d2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f54040e4e40 0x7f54041b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54088d1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f54040e5240 0x7f5404084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50484/0 (socket says 172.21.2.117:50484) 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54088d1700 1 -- 172.21.2.117:0/1963245410 learned_addr learned my addr 172.21.2.117:0/1963245410 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f53f7fff700 1 -- 172.21.2.117:0/1963245410 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677400865 0 0) 0x7f54000010b0 con 0x7f54040e5240 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f53f7fff700 1 -- 172.21.2.117:0/1963245410 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0002fe0 con 0x7f54040e5240 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54090d2700 1 -- 172.21.2.117:0/1963245410 >> v1:172.21.2.125:6789/0 conn(0x7f5404085450 legacy=0x7f54040d05b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54090d2700 1 -- 172.21.2.117:0/1963245410 >> v1:172.21.2.117:6789/0 conn(0x7f54040e5240 legacy=0x7f5404084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54090d2700 1 -- 172.21.2.117:0/1963245410 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54041b0810 con 0x7f54040e4e40 2021-02-10T11:43:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f54090d2700 1 --2- 172.21.2.117:0/1963245410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54040e4e40 0x7f54041b01f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f53f8006650 tx=0x7f53f8009a80).ready entity=mon.0 client_cookie=13fd61f0705cb72d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f53f7fff700 1 -- 172.21.2.117:0/1963245410 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f800c0c0 con 0x7f54040e4e40 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f53f7fff700 1 -- 172.21.2.117:0/1963245410 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f800c340 con 0x7f54040e4e40 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f53f7fff700 1 -- 172.21.2.117:0/1963245410 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f53f800c7a0 con 0x7f54040e4e40 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 -- 172.21.2.117:0/1963245410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54040e4e40 msgr2=0x7f54041b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.974 7f540b2b6700 1 --2- 172.21.2.117:0/1963245410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54040e4e40 0x7f54041b01f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0x7f53f8006650 tx=0x7f53f8009a80).stop 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.975 7f540b2b6700 1 -- 172.21.2.117:0/1963245410 shutdown_connections 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.975 7f540b2b6700 1 --2- 172.21.2.117:0/1963245410 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54040e4e40 0x7f54041b01f0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.975 7f540b2b6700 1 -- 172.21.2.117:0/1963245410 >> 172.21.2.117:0/1963245410 conn(0x7f54040ce4e0 msgr2=0x7f54040e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.975 7f540b2b6700 1 -- 172.21.2.117:0/1963245410 shutdown_connections 2021-02-10T11:43:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.975 7f540b2b6700 1 -- 172.21.2.117:0/1963245410 wait complete. 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 Processor -- start 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 -- start start 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f540409ef40 0x7f540409f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5404191890 0x7f5404191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540409e9c0 con 0x7f540409f960 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54040a0380 con 0x7f540409ef40 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f540b2b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54041922b0 con 0x7f5404191890 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f54088d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f54088d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50240/0 (socket says 172.21.2.117:50240) 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.990 7f54088d1700 1 -- 172.21.2.117:0/3850316843 learned_addr learned my addr 172.21.2.117:0/3850316843 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:26.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54090d2700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f540409ef40 0x7f540409f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 -- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5404191890 msgr2=0x7f5404191c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5404191890 0x7f5404191c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 -- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f540409ef40 msgr2=0x7f540409f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f540409ef40 0x7f540409f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 -- 172.21.2.117:0/3850316843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54041924d0 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f54088d1700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f5400001c60 tx=0x7f540000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f540000acc0 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5400004350 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5400004830 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54041924d0 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.991 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5404192a60 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.992 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5400004530 con 0x7f540409f960 2021-02-10T11:43:26.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.992 7f53f5ffb700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 0x7f53e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:26.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.992 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f54000326a0 con 0x7f540409f960 2021-02-10T11:43:26.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.992 7f54090d2700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 0x7f53e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:26.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:26.993 7f54090d2700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 0x7f53e401abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:27.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.023 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54040a0380 con 0x7f540409f960 2021-02-10T11:43:27.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.026 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5400004140 con 0x7f540409f960 2021-02-10T11:43:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.155 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54040a0380 con 0x7f540409f960 2021-02-10T11:43:27.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.155 7f53f5ffb700 1 -- 172.21.2.117:0/3850316843 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f5400004510 con 0x7f540409f960 2021-02-10T11:43:27.147 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:27.147 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":44,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:27.147 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 44 2021-02-10T11:43:27.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 msgr2=0x7f53e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:27.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 0x7f53e401abc0 crc :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:27.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 msgr2=0x7f540409fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:27.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f5400001c60 tx=0x7f540000a8d0).stop 2021-02-10T11:43:27.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 shutdown_connections 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53e40186c0 0x7f53e401abc0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f540409f960 0x7f540409fd60 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5404191890 0x7f5404191c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 --2- 172.21.2.117:0/3850316843 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f540409ef40 0x7f540409f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 >> 172.21.2.117:0/3850316843 conn(0x7f540418cc40 msgr2=0x7f540418c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 shutdown_connections 2021-02-10T11:43:27.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:27.157 7f540b2b6700 1 -- 172.21.2.117:0/3850316843 wait complete. 2021-02-10T11:43:27.358 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.674746', 'modified': '2021-02-10 11:43:10.684137', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/2942282551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 2942282551}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 2942282551}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:27.358 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:27.358 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:43:27.758 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:27.758 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:43:27.758 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 6670, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 19.091062085000001, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 44, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 64, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 24.724697193000001 2021-02-10T11:43:27.759 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:27.762 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6670, "want_state": "up:active", "state": "up:active", "rank_uptime": 19.091062085000001, "mdsmap_epoch": 44, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 24.724697193000001 } 2021-02-10T11:43:27.762 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:27.762 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:43:27.763 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:43:27.763 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:27.763 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:28.098 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:28.443 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:28.444 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:28.786 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:28.786 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:28.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:28.787 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:29.120 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:43:29.121 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:43:29.121 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:29.123 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:29.467 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:29.467 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:29.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[135919]: starting ceph client2021-02-10 11:43:29.542 7fc4db040f80 -1 init, newargv = 0x55d92c355bb0 newargc=9 2021-02-10T11:43:29.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr: 2021-02-10T11:43:29.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[135919]: starting fuse 2021-02-10T11:43:29.811 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:29.811 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:29.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:29.812 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:30.147 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:43:30.148 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:43:30.148 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:43:30.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:30.148 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:43:30.148 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:43:30.148 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:43:30.148 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:43:30.148 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:43:30.149 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:43:30.150 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:30.150 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:43:30.150 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:43:30.511 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.135919.asok 2021-02-10T11:43:30.514 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.135919.asok 2021-02-10T11:43:30.514 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:30.514 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.135919.asok status 2021-02-10T11:43:30.905 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:30.905 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:43:30.905 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:43:30.905 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:43:30.905 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "135919", 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8628, 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:43:30.906 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8628 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 714879663 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 714879663 2021-02-10T11:43:30.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8628 172.21.2.117:0/714879663", 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/714879663", 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 44, 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 64, 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:43:30.908 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:30.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:30.910 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:30.910 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:31.245 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:43:31.245 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:31.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:31.245 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:31.591 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:43:32.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.014 7fe6b2fd7700 1 Processor -- start 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.014 7fe6b2fd7700 1 -- start start 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b2fd7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6ac0e5240 0x7fe6ac086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b2fd7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe6ac084c60 0x7fe6ac085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b2fd7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac085930 con 0x7fe6ac0e4e40 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b2fd7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac0e5cc0 con 0x7fe6ac0e5240 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b2fd7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac0856b0 con 0x7fe6ac084c60 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6abfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6ac0e5240 0x7fe6ac086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6abfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6ac0e5240 0x7fe6ac086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50264/0 (socket says 172.21.2.117:50264) 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6abfff700 1 -- 172.21.2.117:0/1790903363 learned_addr learned my addr 172.21.2.117:0/1790903363 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.015 7fe6b15f4700 1 --2- 172.21.2.117:0/1790903363 >> v2:172.21.2.125:3300/0 conn(0x7fe6ac084c60 0x7fe6ac085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6abfff700 1 -- 172.21.2.117:0/1790903363 >> v2:172.21.2.125:3300/0 conn(0x7fe6ac084c60 msgr2=0x7fe6ac085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6abfff700 1 --2- 172.21.2.117:0/1790903363 >> v2:172.21.2.125:3300/0 conn(0x7fe6ac084c60 0x7fe6ac085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6abfff700 1 -- 172.21.2.117:0/1790903363 >> v1:172.21.2.117:6789/0 conn(0x7fe6ac0e4e40 legacy=0x7fe6ac1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6abfff700 1 -- 172.21.2.117:0/1790903363 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac1b09e0 con 0x7fe6ac0e5240 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6abfff700 1 --2- 172.21.2.117:0/1790903363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0e5240 0x7fe6ac086ec0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fe69c006a80 tx=0x7fe69c009f00).ready entity=mon.0 client_cookie=f7a088973a50d26f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6ab7fe700 1 -- 172.21.2.117:0/1790903363 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe69c00b5c0 con 0x7fe6ac0e5240 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6ab7fe700 1 -- 172.21.2.117:0/1790903363 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00b8c0 con 0x7fe6ac0e5240 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6ab7fe700 1 -- 172.21.2.117:0/1790903363 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe69c00bd20 con 0x7fe6ac0e5240 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6b2fd7700 1 -- 172.21.2.117:0/1790903363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0e5240 msgr2=0x7fe6ac086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.016 7fe6b2fd7700 1 --2- 172.21.2.117:0/1790903363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0e5240 0x7fe6ac086ec0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fe69c006a80 tx=0x7fe69c009f00).stop 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 -- 172.21.2.117:0/1790903363 shutdown_connections 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 --2- 172.21.2.117:0/1790903363 >> v2:172.21.2.125:3300/0 conn(0x7fe6ac084c60 0x7fe6ac085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 --2- 172.21.2.117:0/1790903363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0e5240 0x7fe6ac086ec0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 -- 172.21.2.117:0/1790903363 >> 172.21.2.117:0/1790903363 conn(0x7fe6ac0ce4e0 msgr2=0x7fe6ac0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 -- 172.21.2.117:0/1790903363 shutdown_connections 2021-02-10T11:43:32.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.017 7fe6b2fd7700 1 -- 172.21.2.117:0/1790903363 wait complete. 2021-02-10T11:43:32.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 Processor -- start 2021-02-10T11:43:32.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 -- start start 2021-02-10T11:43:32.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6ac0c2000 0x7fe6ac191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6ac0b6c00 0x7fe6ac0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac191ab0 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac1916a0 con 0x7fe6ac0b6c00 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6b2fd7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ac191880 con 0x7fe6ac0c2000 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6abfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6abfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50270/0 (socket says 172.21.2.117:50270) 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.032 7fe6abfff700 1 -- 172.21.2.117:0/88710492 learned_addr learned my addr 172.21.2.117:0/88710492 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6b15f4700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6ac0b6c00 0x7fe6ac0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6b0df3700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6ac0c2000 0x7fe6ac191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 -- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6ac0c2000 msgr2=0x7fe6ac191080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6ac0c2000 0x7fe6ac191080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 -- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6ac0b6c00 msgr2=0x7fe6ac0b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6ac0b6c00 0x7fe6ac0b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 -- 172.21.2.117:0/88710492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac0b7620 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6abfff700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fe69c006a90 tx=0x7fe69c00bc90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe69c006710 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00c2f0 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe69c00c7d0 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac0b7620 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.033 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac0b7620 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.034 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe69c001010 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.034 7fe6a97fa700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 0x7fe69001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.034 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe69c032c60 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.035 7fe6b0df3700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 0x7fe69001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:32.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.035 7fe6b0df3700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 0x7fe69001abd0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:32.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.065 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ac191880 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.068 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe69c00bdd0 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.205 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6ac191880 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.660 7fe6a97fa700 1 -- 172.21.2.117:0/88710492 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v45) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe69c001010 con 0x7fe6ac0c14d0 2021-02-10T11:43:32.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.673 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 msgr2=0x7fe69001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.673 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 0x7fe69001abd0 crc :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.673 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 msgr2=0x7fe6ac0c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:32.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.673 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fe69c006a90 tx=0x7fe69c00bc90).stop 2021-02-10T11:43:32.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 shutdown_connections 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6900186d0 0x7fe69001abd0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6ac0c14d0 0x7fe6ac0c18d0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6ac0b6c00 0x7fe6ac0b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 --2- 172.21.2.117:0/88710492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6ac0c2000 0x7fe6ac191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 >> 172.21.2.117:0/88710492 conn(0x7fe6ac0acec0 msgr2=0x7fe6ac0b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 shutdown_connections 2021-02-10T11:43:32.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:32.674 7fe6b2fd7700 1 -- 172.21.2.117:0/88710492 wait complete. 2021-02-10T11:43:32.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:43:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 Processor -- start 2021-02-10T11:43:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 -- start start 2021-02-10T11:43:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 0x7f42400e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f42400e7d70 0x7f42400e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42400e77f0 con 0x7f42400e8790 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42400e92f0 con 0x7f42400e7d70 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.092 7f4246fe7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42400e5ce0 con 0x7f42401b07f0 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f42400e7d70 0x7f42400e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f42400e7d70 0x7f42400e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50276/0 (socket says 172.21.2.117:50276) 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 -- 172.21.2.117:0/2369765590 learned_addr learned my addr 172.21.2.117:0/2369765590 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f4244e03700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 0x7f42400e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 -- 172.21.2.117:0/2369765590 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 msgr2=0x7f42400e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 0x7f42400e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 -- 172.21.2.117:0/2369765590 >> v1:172.21.2.125:6789/0 conn(0x7f42400e8790 legacy=0x7f42400e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 -- 172.21.2.117:0/2369765590 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42400e54b0 con 0x7f42400e7d70 2021-02-10T11:43:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f4244e03700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 0x7f42400e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423ffff700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42400e7d70 0x7f42400e8170 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f4230006700 tx=0x7f4230009b80).ready entity=mon.0 client_cookie=42e87f4fd05bc6b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423f7fe700 1 -- 172.21.2.117:0/2369765590 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000c0c0 con 0x7f42400e7d70 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423f7fe700 1 -- 172.21.2.117:0/2369765590 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000c440 con 0x7f42400e7d70 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f423f7fe700 1 -- 172.21.2.117:0/2369765590 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000c920 con 0x7f42400e7d70 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f4246fe7700 1 -- 172.21.2.117:0/2369765590 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42400e7d70 msgr2=0x7f42400e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.093 7f4246fe7700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42400e7d70 0x7f42400e8170 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7f4230006700 tx=0x7f4230009b80).stop 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 -- 172.21.2.117:0/2369765590 shutdown_connections 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42400e7d70 0x7f42400e8170 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 --2- 172.21.2.117:0/2369765590 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42401b07f0 0x7f42400e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 -- 172.21.2.117:0/2369765590 >> 172.21.2.117:0/2369765590 conn(0x7f42400ce500 msgr2=0x7f42400e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 -- 172.21.2.117:0/2369765590 shutdown_connections 2021-02-10T11:43:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.094 7f4246fe7700 1 -- 172.21.2.117:0/2369765590 wait complete. 2021-02-10T11:43:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.108 7f4246fe7700 1 Processor -- start 2021-02-10T11:43:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.108 7f4246fe7700 1 -- start start 2021-02-10T11:43:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42400a70c0 0x7f42400a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 0x7f424019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400a7760 con 0x7f4240110bd0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424019de00 con 0x7f424019d3e0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4246fe7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424019dfe0 con 0x7f42400a70c0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50284/0 (socket says 172.21.2.117:50284) 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 -- 172.21.2.117:0/1915205479 learned_addr learned my addr 172.21.2.117:0/1915205479 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 -- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42400a70c0 msgr2=0x7f42400a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42400a70c0 0x7f42400a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4245604700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 0x7f424019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 -- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 msgr2=0x7f424019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 0x7f424019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f423ffff700 1 -- 172.21.2.117:0/1915205479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424019e200 con 0x7f4240110bd0 2021-02-10T11:43:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.109 7f4245604700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 0x7f424019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423ffff700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f423000b580 tx=0x7f423000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f423000bd10 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4230002030 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4230002400 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424019e200 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42400a7fd0 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4230000c90 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.110 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4230032750 con 0x7f4240110bd0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.111 7f4244e03700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.111 7f4244e03700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 0x7f422401abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.142 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42401b8070 con 0x7f4240110bd0 2021-02-10T11:43:33.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.145 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42300009b0 con 0x7f4240110bd0 2021-02-10T11:43:33.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.277 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f424019e200 con 0x7f42240186c0 2021-02-10T11:43:33.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.278 7f423d7fa700 1 -- 172.21.2.117:0/1915205479 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f42280077a0 con 0x7f42240186c0 2021-02-10T11:43:33.270 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:43:33.270 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:33.270 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:43:33.270 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:33.270 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 msgr2=0x7f422401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 0x7f422401abc0 crc :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 msgr2=0x7f4240110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f423000b580 tx=0x7f423000b920).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 shutdown_connections 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42240186c0 0x7f422401abc0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4240110bd0 0x7f4240110fd0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f424019d3e0 0x7f424019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 --2- 172.21.2.117:0/1915205479 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42400a70c0 0x7f42400a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.281 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 >> 172.21.2.117:0/1915205479 conn(0x7f42400bff40 msgr2=0x7f4240098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.282 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 shutdown_connections 2021-02-10T11:43:33.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.282 7f4246fe7700 1 -- 172.21.2.117:0/1915205479 wait complete. 2021-02-10T11:43:33.280 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:43:33.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.704 7fdb912e0700 1 Processor -- start 2021-02-10T11:43:33.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.704 7fdb912e0700 1 -- start start 2021-02-10T11:43:33.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb912e0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdb8c0ef3f0 0x7fdb8c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb912e0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb912e0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c0ef7f0 con 0x7fdb8c0d18f0 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb912e0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0ede60 con 0x7fdb8c0eb830 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb912e0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0d1cf0 con 0x7fdb8c0ef3f0 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50294/0 (socket says 172.21.2.117:50294) 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 -- 172.21.2.117:0/559972966 learned_addr learned my addr 172.21.2.117:0/559972966 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 -- 172.21.2.117:0/559972966 >> v2:172.21.2.125:3300/0 conn(0x7fdb8c0ef3f0 msgr2=0x7fdb8c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 --2- 172.21.2.117:0/559972966 >> v2:172.21.2.125:3300/0 conn(0x7fdb8c0ef3f0 0x7fdb8c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 -- 172.21.2.117:0/559972966 >> v1:172.21.2.117:6789/0 conn(0x7fdb8c0d18f0 legacy=0x7fdb8c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.705 7fdb8b87e700 1 -- 172.21.2.117:0/559972966 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c1b8fd0 con 0x7fdb8c0eb830 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb8b87e700 1 --2- 172.21.2.117:0/559972966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdb84006f00 tx=0x7fdb84009e90).ready entity=mon.0 client_cookie=1771398eb72f2a09 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb8a07b700 1 -- 172.21.2.117:0/559972966 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb8400b300 con 0x7fdb8c0eb830 2021-02-10T11:43:33.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb8a07b700 1 -- 172.21.2.117:0/559972966 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8400b640 con 0x7fdb8c0eb830 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb8a07b700 1 -- 172.21.2.117:0/559972966 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb8400ba90 con 0x7fdb8c0eb830 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb912e0700 1 -- 172.21.2.117:0/559972966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c0eb830 msgr2=0x7fdb8c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.706 7fdb912e0700 1 --2- 172.21.2.117:0/559972966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7fdb84006f00 tx=0x7fdb84009e90).stop 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 -- 172.21.2.117:0/559972966 shutdown_connections 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 --2- 172.21.2.117:0/559972966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c0eb830 0x7fdb8c1b79a0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 --2- 172.21.2.117:0/559972966 >> v2:172.21.2.125:3300/0 conn(0x7fdb8c0ef3f0 0x7fdb8c0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 -- 172.21.2.117:0/559972966 >> 172.21.2.117:0/559972966 conn(0x7fdb8c0cf060 msgr2=0x7fdb8c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 -- 172.21.2.117:0/559972966 shutdown_connections 2021-02-10T11:43:33.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.707 7fdb912e0700 1 -- 172.21.2.117:0/559972966 wait complete. 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.721 7fdb912e0700 1 Processor -- start 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.721 7fdb912e0700 1 -- start start 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb8c1a3930 0x7fdb8c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb8c0a13a0 0x7fdb8c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c07cca0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c18fb70 con 0x7fdb8c1a3930 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb912e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0dc6d0 con 0x7fdb8c0a13a0 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50296/0 (socket says 172.21.2.117:50296) 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 -- 172.21.2.117:0/1849641535 learned_addr learned my addr 172.21.2.117:0/1849641535 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 -- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb8c0a13a0 msgr2=0x7fdb8c0a17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb8c0a13a0 0x7fdb8c0a17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 -- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb8c1a3930 msgr2=0x7fdb8c0a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb8c1a3930 0x7fdb8c0a0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 -- 172.21.2.117:0/1849641535 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c07d220 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.722 7fdb8b07d700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fdb74006ed0 tx=0x7fdb7400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb74006240 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb740054b0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c07d220 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c07d7b0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb74005990 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb74006030 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 0x7fdb7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:33.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.723 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdb74032900 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.724 7fdb8a87c700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 0x7fdb7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:33.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.724 7fdb8a87c700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 0x7fdb7001abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:33.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.755 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb64002fd0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.757 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb740054b0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.904 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdb64002ef0 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.904 7fdb83fff700 1 -- 172.21.2.117:0/1849641535 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdb74006030 con 0x7fdb8c07c7c0 2021-02-10T11:43:33.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 msgr2=0x7fdb7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 0x7fdb7001abc0 crc :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 msgr2=0x7fdb8c1a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:33.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7fdb74006ed0 tx=0x7fdb7400b960).stop 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 shutdown_connections 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb8c0a13a0 0x7fdb8c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb8c1a3930 0x7fdb8c0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb700186c0 0x7fdb7001abc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 --2- 172.21.2.117:0/1849641535 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb8c07c7c0 0x7fdb8c1a3310 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 >> 172.21.2.117:0/1849641535 conn(0x7fdb8c087090 msgr2=0x7fdb8c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.906 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 shutdown_connections 2021-02-10T11:43:33.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:33.907 7fdb912e0700 1 -- 172.21.2.117:0/1849641535 wait complete. 2021-02-10T11:43:33.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:33.905 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:33.905 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:34.242 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:43:34.243 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:34.243 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:43:34.243 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:43:34.243 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:34.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[135919]: fuse finished with error 0 and tester_r 0 2021-02-10T11:43:34.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:34.600 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:34.600 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:34.935 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:43:34.935 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:43:34.935 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:43:34.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:34.935 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:35.270 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000285035 2021-02-10T11:43:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 Processor -- start 2021-02-10T11:43:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 -- start start 2021-02-10T11:43:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f541c1b69a0 0x7f541c1b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0ede60 con 0x7f541c0ef3f0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c0eba00 con 0x7f541c0d18f0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f5422686700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c0d1cf0 con 0x7f541c1b69a0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.692 7f541b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f541c0ef3f0 0x7f541c0eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50554/0 (socket says 172.21.2.117:50554) 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541b7fe700 1 -- 172.21.2.117:0/455921486 learned_addr learned my addr 172.21.2.117:0/455921486 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214365677 0 0) 0x7f5410001180 con 0x7f541c0ef3f0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404002fe0 con 0x7f541c0ef3f0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f5420ca3700 1 --2- 172.21.2.117:0/455921486 >> v2:172.21.2.125:3300/0 conn(0x7f541c1b69a0 0x7f541c1b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3753331986 0 0) 0x7f5410001180 con 0x7f541c0ef3f0 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 >> v2:172.21.2.125:3300/0 conn(0x7f541c1b69a0 msgr2=0x7f541c1b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 --2- 172.21.2.117:0/455921486 >> v2:172.21.2.125:3300/0 conn(0x7f541c1b69a0 0x7f541c1b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 >> v1:172.21.2.125:6789/0 conn(0x7f541c0d18f0 legacy=0x7f541c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c1bb3f0 con 0x7f541c0ef3f0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f5410001680 con 0x7f541c0ef3f0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 >> v1:172.21.2.117:6789/0 conn(0x7f541c0ef3f0 legacy=0x7f541c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 0x7f5404005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54040067a0 0x7f5404008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54040095a0 0x7f540400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540400c310 con 0x7f5404003a70 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.693 7f541affd700 1 -- 172.21.2.117:0/455921486 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540400c530 con 0x7f54040067a0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541affd700 1 -- 172.21.2.117:0/455921486 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5404050760 con 0x7f54040095a0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 0x7f5404005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541b7fe700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54040067a0 0x7f5404008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f5420ca3700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54040095a0 0x7f540400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 -- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54040095a0 msgr2=0x7f540400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54040095a0 0x7f540400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 -- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54040067a0 msgr2=0x7f5404008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54040067a0 0x7f5404008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 -- 172.21.2.117:0/455921486 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404050990 con 0x7f5404003a70 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f5420ca3700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54040095a0 0x7f540400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f5420ca3700 1 -- 172.21.2.117:0/455921486 reap_dead start 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541b7fe700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54040067a0 0x7f5404008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.694 7f541bfff700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 0x7f5404005f00 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f540c006ab0 tx=0x7f540c009b40).ready entity=mon.0 client_cookie=24fb089325f45cb9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f540c00b2e0 con 0x7f5404003a70 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540c000c60 con 0x7f5404003a70 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f541affd700 1 -- 172.21.2.117:0/455921486 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f540c00b750 con 0x7f5404003a70 2021-02-10T11:43:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 -- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 msgr2=0x7f5404005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 0x7f5404005f00 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7f540c006ab0 tx=0x7f540c009b40).stop 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 -- 172.21.2.117:0/455921486 shutdown_connections 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 --2- 172.21.2.117:0/455921486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5404003a70 0x7f5404005f00 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 -- 172.21.2.117:0/455921486 >> 172.21.2.117:0/455921486 conn(0x7f541c0cf060 msgr2=0x7f541c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 -- 172.21.2.117:0/455921486 shutdown_connections 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.695 7f5422686700 1 -- 172.21.2.117:0/455921486 wait complete. 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.710 7f5422686700 1 Processor -- start 2021-02-10T11:43:35.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.710 7f5422686700 1 -- start start 2021-02-10T11:43:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 0x7f541c0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f541c0a2040 0x7f541c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0a67e0 con 0x7f541c0a6ce0 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0a7120 con 0x7f541c085ec0 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5422686700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0a2a60 con 0x7f541c0a2040 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50312/0 (socket says 172.21.2.117:50312) 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541bfff700 1 -- 172.21.2.117:0/808457942 learned_addr learned my addr 172.21.2.117:0/808457942 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 0x7f541c0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f5420ca3700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f541c0a2040 0x7f541c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 -- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f541c0a2040 msgr2=0x7f541c0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f541c0a2040 0x7f541c0a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 -- 172.21.2.117:0/808457942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 msgr2=0x7f541c0dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541b7fe700 1 -- 172.21.2.117:0/808457942 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c1a2dc0 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.711 7f541bfff700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f541b7fe700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 0x7f541c0862c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f541000b500 tx=0x7f541000b8a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541000bc90 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410007c50 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541000a350 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5422686700 1 -- 172.21.2.117:0/808457942 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c1a2dc0 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5422686700 1 -- 172.21.2.117:0/808457942 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c1a2dc0 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f541000a560 con 0x7f541c085ec0 2021-02-10T11:43:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 0x7f53fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.712 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f54100326d0 con 0x7f541c085ec0 2021-02-10T11:43:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.713 7f541bfff700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 0x7f53fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.713 7f541bfff700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 0x7f53fc01ac10 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:35.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.742 7f5422686700 1 -- 172.21.2.117:0/808457942 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541c0daff0 con 0x7f541c085ec0 2021-02-10T11:43:35.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.747 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5410007c30 con 0x7f541c085ec0 2021-02-10T11:43:35.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.876 7f5422686700 1 -- 172.21.2.117:0/808457942 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000285035"}) v1 -- 0x7f541c0a6760 con 0x7f53fc018730 2021-02-10T11:43:35.870 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:43:35.883 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:35.871 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:43:35.883 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:35.871 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:43:35.883 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:35.871 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:43:35.883 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:35.871 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:43:35.883 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:43:35.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.895 7f5418ff9700 1 -- 172.21.2.117:0/808457942 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f540c000960 con 0x7f53fc018730 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 msgr2=0x7f53fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 0x7f53fc01ac10 crc :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 msgr2=0x7f541c0862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 0x7f541c0862c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f541000b500 tx=0x7f541000b8a0).stop 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 shutdown_connections 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f541c0a2040 0x7f541c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f541c085ec0 0x7f541c0862c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f53fc018730 0x7f53fc01ac10 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 --2- 172.21.2.117:0/808457942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f541c0a6ce0 0x7f541c0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:35.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 >> 172.21.2.117:0/808457942 conn(0x7f541c0b6540 msgr2=0x7f541c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:35.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 shutdown_connections 2021-02-10T11:43:35.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:35.910 7f5422686700 1 -- 172.21.2.117:0/808457942 wait complete. 2021-02-10T11:43:35.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000934866 --group_name group_0000000000285035 2021-02-10T11:43:36.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 Processor -- start 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 -- start start 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ea8085450 0x7f5ea8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8085930 con 0x7f5ea80e5240 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea81b09e0 con 0x7f5ea80e4e40 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.333 7f5ead08e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80e5cc0 con 0x7f5ea8085450 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea659c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5ea80e5240 0x7f5ea8084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50570/0 (socket says 172.21.2.117:50570) 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea759e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ea8085450 0x7f5ea8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea659c700 1 -- 172.21.2.117:0/3006180157 learned_addr learned my addr 172.21.2.117:0/3006180157 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea5d9b700 1 -- 172.21.2.117:0/3006180157 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161188068 0 0) 0x7f5e9c000ce0 con 0x7f5ea80e5240 2021-02-10T11:43:36.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea5d9b700 1 -- 172.21.2.117:0/3006180157 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e90002fe0 con 0x7f5ea80e5240 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea759e700 1 -- 172.21.2.117:0/3006180157 >> v1:172.21.2.125:6789/0 conn(0x7f5ea80e4e40 legacy=0x7f5ea81b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea759e700 1 -- 172.21.2.117:0/3006180157 >> v1:172.21.2.117:6789/0 conn(0x7f5ea80e5240 legacy=0x7f5ea8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea759e700 1 -- 172.21.2.117:0/3006180157 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8084840 con 0x7f5ea8085450 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea759e700 1 --2- 172.21.2.117:0/3006180157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea8085450 0x7f5ea8086ec0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f5ea0006e30 tx=0x7f5ea0009e60).ready entity=mon.0 client_cookie=9c584ef161eed239 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea5d9b700 1 -- 172.21.2.117:0/3006180157 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea0006a50 con 0x7f5ea8085450 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea5d9b700 1 -- 172.21.2.117:0/3006180157 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea000b300 con 0x7f5ea8085450 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ea5d9b700 1 -- 172.21.2.117:0/3006180157 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ea000b7e0 con 0x7f5ea8085450 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ead08e700 1 -- 172.21.2.117:0/3006180157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea8085450 msgr2=0x7f5ea8086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.334 7f5ead08e700 1 --2- 172.21.2.117:0/3006180157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea8085450 0x7f5ea8086ec0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f5ea0006e30 tx=0x7f5ea0009e60).stop 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.335 7f5ead08e700 1 -- 172.21.2.117:0/3006180157 shutdown_connections 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.335 7f5ead08e700 1 --2- 172.21.2.117:0/3006180157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea8085450 0x7f5ea8086ec0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.335 7f5ead08e700 1 -- 172.21.2.117:0/3006180157 >> 172.21.2.117:0/3006180157 conn(0x7f5ea80ce4e0 msgr2=0x7f5ea80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:36.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.335 7f5ead08e700 1 -- 172.21.2.117:0/3006180157 shutdown_connections 2021-02-10T11:43:36.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.335 7f5ead08e700 1 -- 172.21.2.117:0/3006180157 wait complete. 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 Processor -- start 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 -- start start 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea8099bf0 0x7f5ea81ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea80b30d0 0x7f5ea80b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81ae8d0 con 0x7f5ea818e5b0 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80b3af0 con 0x7f5ea80b30d0 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ead08e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80b3d10 con 0x7f5ea8099bf0 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ea659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ea659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50324/0 (socket says 172.21.2.117:50324) 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.350 7f5ea659c700 1 -- 172.21.2.117:0/4186368670 learned_addr learned my addr 172.21.2.117:0/4186368670 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 -- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea8099bf0 msgr2=0x7f5ea81ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea8099bf0 0x7f5ea81ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 -- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea80b30d0 msgr2=0x7f5ea80b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea80b30d0 0x7f5ea80b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 -- 172.21.2.117:0/4186368670 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80b3f30 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ea659c700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f5e9c001490 tx=0x7f5e9c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e9c00ad40 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c000ce0 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e9c004620 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea80b3f30 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.351 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea81a19b0 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.352 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e9c0043a0 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.352 7f5e977fe700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 0x7f5e8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.352 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e9c0326d0 con 0x7f5ea818e5b0 2021-02-10T11:43:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.352 7f5ea6d9d700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 0x7f5e8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.353 7f5ea6d9d700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 0x7f5e8c01abc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:36.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.383 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea818e080 con 0x7f5ea818e5b0 2021-02-10T11:43:36.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.386 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e9c000940 con 0x7f5ea818e5b0 2021-02-10T11:43:36.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.511 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "group_name": "group_0000000000285035"}) v1 -- 0x7f5ea81ae8d0 con 0x7f5e8c0186c0 2021-02-10T11:43:36.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.536 7f5e977fe700 1 -- 172.21.2.117:0/4186368670 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5e98006a10 con 0x7f5e8c0186c0 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.544 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 msgr2=0x7f5e8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.544 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 0x7f5e8c01abc0 crc :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.544 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 msgr2=0x7f5ea818e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.544 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f5e9c001490 tx=0x7f5e9c00a950).stop 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 shutdown_connections 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e8c0186c0 0x7f5e8c01abc0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea818e5b0 0x7f5ea818e9b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea80b30d0 0x7f5ea80b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 --2- 172.21.2.117:0/4186368670 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea8099bf0 0x7f5ea81ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 >> 172.21.2.117:0/4186368670 conn(0x7f5ea81ad6b0 msgr2=0x7f5ea8190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:36.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 shutdown_connections 2021-02-10T11:43:36.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.545 7f5ead08e700 1 -- 172.21.2.117:0/4186368670 wait complete. 2021-02-10T11:43:36.543 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000934866 --group_name group_0000000000285035 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 Processor -- start 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 -- start start 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43b40e5240 0x7f43b4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4084c60 0x7f43b4085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b4085930 con 0x7f43b40e4e40 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b4085680 con 0x7f43b40e5240 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.967 7f43b9c0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41b09e0 con 0x7f43b4084c60 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43b40e5240 0x7f43b4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43b40e5240 0x7f43b4086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50332/0 (socket says 172.21.2.117:50332) 2021-02-10T11:43:36.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 -- 172.21.2.117:0/1742147542 learned_addr learned my addr 172.21.2.117:0/1742147542 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 -- 172.21.2.117:0/1742147542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4084c60 msgr2=0x7f43b4085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 --2- 172.21.2.117:0/1742147542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4084c60 0x7f43b4085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 -- 172.21.2.117:0/1742147542 >> v1:172.21.2.125:6789/0 conn(0x7f43b40e4e40 legacy=0x7f43b41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 -- 172.21.2.117:0/1742147542 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b4084840 con 0x7f43b40e5240 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b2ffd700 1 --2- 172.21.2.117:0/1742147542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40e5240 0x7f43b4086ec0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f43a4006700 tx=0x7f43a4009b80).ready entity=mon.0 client_cookie=77487b7912245e93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b27fc700 1 -- 172.21.2.117:0/1742147542 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a400c0c0 con 0x7f43b40e5240 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b27fc700 1 -- 172.21.2.117:0/1742147542 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a400c400 con 0x7f43b40e5240 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b27fc700 1 -- 172.21.2.117:0/1742147542 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a400c920 con 0x7f43b40e5240 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b9c0a700 1 -- 172.21.2.117:0/1742147542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40e5240 msgr2=0x7f43b4086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.968 7f43b9c0a700 1 --2- 172.21.2.117:0/1742147542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40e5240 0x7f43b4086ec0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f43a4006700 tx=0x7f43a4009b80).stop 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 -- 172.21.2.117:0/1742147542 shutdown_connections 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 --2- 172.21.2.117:0/1742147542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4084c60 0x7f43b4085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 --2- 172.21.2.117:0/1742147542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40e5240 0x7f43b4086ec0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 -- 172.21.2.117:0/1742147542 >> 172.21.2.117:0/1742147542 conn(0x7f43b40ce4e0 msgr2=0x7f43b40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:36.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 -- 172.21.2.117:0/1742147542 shutdown_connections 2021-02-10T11:43:36.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.969 7f43b9c0a700 1 -- 172.21.2.117:0/1742147542 wait complete. 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 Processor -- start 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 -- start start 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4093070 0x7f43b409fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 0x7f43b409e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b409f540 con 0x7f43b40a00e0 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4093470 con 0x7f43b41ae8d0 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b9c0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b409ecd0 con 0x7f43b4093070 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50342/0 (socket says 172.21.2.117:50342) 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.984 7f43b2ffd700 1 -- 172.21.2.117:0/538080488 learned_addr learned my addr 172.21.2.117:0/538080488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b3fff700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 0x7f43b409e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 -- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4093070 msgr2=0x7f43b409fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4093070 0x7f43b409fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 -- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 msgr2=0x7f43b409e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 0x7f43b409e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 -- 172.21.2.117:0/538080488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b409eef0 con 0x7f43b40a00e0 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b3fff700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 0x7f43b409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:36.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b2ffd700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f43a40066a0 tx=0x7f43a400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a400bd80 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a40052e0 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b409eef0 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b41b20a0 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.985 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f43a40057c0 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.986 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43a4006030 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.986 7f43abfff700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 0x7f439801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.986 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f43a4032790 con 0x7f43b40a00e0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.986 7f43b37fe700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 0x7f439801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:36.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:36.987 7f43b37fe700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 0x7f439801abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:37.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.017 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b410c2f0 con 0x7f43b40a00e0 2021-02-10T11:43:37.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.020 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f43a402dfa0 con 0x7f43b40a00e0 2021-02-10T11:43:37.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.147 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "group_name": "group_0000000000285035"}) v1 -- 0x7f43b41b20a0 con 0x7f43980186c0 2021-02-10T11:43:37.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.151 7f43abfff700 1 -- 172.21.2.117:0/538080488 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7f439c006a10 con 0x7f43980186c0 2021-02-10T11:43:37.144 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63 2021-02-10T11:43:37.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 msgr2=0x7f439801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 0x7f439801abc0 crc :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 msgr2=0x7f43b41ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f43a40066a0 tx=0x7f43a400b990).stop 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 shutdown_connections 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43980186c0 0x7f439801abc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43b40a00e0 0x7f43b41ae2b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43b41ae8d0 0x7f43b409e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 --2- 172.21.2.117:0/538080488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43b4093070 0x7f43b409fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 >> 172.21.2.117:0/538080488 conn(0x7f43b41ad6b0 msgr2=0x7f43b4190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:37.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 shutdown_connections 2021-02-10T11:43:37.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.156 7f43b9c0a700 1 -- 172.21.2.117:0/538080488 wait complete. 2021-02-10T11:43:37.155 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000934866 alice --group_name group_0000000000285035 --tenant_id tenant_id 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 Processor -- start 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 -- start start 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 0x7f8d400eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400ede60 con 0x7f8d400ef3f0 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400d1cf0 con 0x7f8d400eb040 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d47c47700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40088020 con 0x7f8d400d18f0 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.586 7f8d46264700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8d400ef3f0 0x7f8d401b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50596/0 (socket says 172.21.2.117:50596) 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d46264700 1 -- 172.21.2.117:0/4108451426 learned_addr learned my addr 172.21.2.117:0/4108451426 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d45a63700 1 --2- 172.21.2.117:0/4108451426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 0x7f8d400eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:37.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235008211 0 0) 0x7f8d38001500 con 0x7f8d400ef3f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c002fe0 con 0x7f8d400ef3f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085316480 0 0) 0x7f8d3c000ff0 con 0x7f8d400eb040 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c003210 con 0x7f8d400eb040 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d45a63700 1 -- 172.21.2.117:0/4108451426 >> v1:172.21.2.125:6789/0 conn(0x7f8d400eb040 legacy=0x7f8d400eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d45a63700 1 -- 172.21.2.117:0/4108451426 >> v1:172.21.2.117:6789/0 conn(0x7f8d400ef3f0 legacy=0x7f8d401b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d45a63700 1 -- 172.21.2.117:0/4108451426 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d401bb200 con 0x7f8d400d18f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.587 7f8d45a63700 1 --2- 172.21.2.117:0/4108451426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 0x7f8d400eaa20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8d34006650 tx=0x7f8d34009a00).ready entity=mon.2 client_cookie=9ad0715cea45c63a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d34009df0 con 0x7f8d400d18f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d340009b0 con 0x7f8d400d18f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d44a61700 1 -- 172.21.2.117:0/4108451426 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3400c2b0 con 0x7f8d400d18f0 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 -- 172.21.2.117:0/4108451426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 msgr2=0x7f8d400eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 --2- 172.21.2.117:0/4108451426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 0x7f8d400eaa20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8d34006650 tx=0x7f8d34009a00).stop 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 -- 172.21.2.117:0/4108451426 shutdown_connections 2021-02-10T11:43:37.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 --2- 172.21.2.117:0/4108451426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d400d18f0 0x7f8d400eaa20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 -- 172.21.2.117:0/4108451426 >> 172.21.2.117:0/4108451426 conn(0x7f8d400cf060 msgr2=0x7f8d400cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.588 7f8d47c47700 1 -- 172.21.2.117:0/4108451426 shutdown_connections 2021-02-10T11:43:37.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.589 7f8d47c47700 1 -- 172.21.2.117:0/4108451426 wait complete. 2021-02-10T11:43:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 Processor -- start 2021-02-10T11:43:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 -- start start 2021-02-10T11:43:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 0x7f8d40190bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 0x7f8d40191c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40192860 con 0x7f8d401b34a0 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4007c7c0 con 0x7f8d40191750 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.603 7f8d47c47700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4007c9e0 con 0x7f8d401911f0 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50350/0 (socket says 172.21.2.117:50350) 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 -- 172.21.2.117:0/3098716744 learned_addr learned my addr 172.21.2.117:0/3098716744 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d45a63700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 0x7f8d40190bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 -- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 msgr2=0x7f8d40191c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d45262700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 0x7f8d40191c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 0x7f8d40191c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 -- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 msgr2=0x7f8d40190bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 0x7f8d40190bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 -- 172.21.2.117:0/3098716744 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4007cc00 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d45262700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 0x7f8d40191c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d45a63700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 0x7f8d40190bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d46264700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f8d38001c10 tx=0x7f8d3800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d38007ce0 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d380070d0 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d38004930 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.604 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4007cc00 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.605 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4007cc00 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.605 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d380072b0 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.605 7f8d327fc700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.605 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d38032910 con 0x7f8d401b34a0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.605 7f8d45a63700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.606 7f8d45a63700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:37.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.636 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d400ede60 con 0x7f8d401b34a0 2021-02-10T11:43:37.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.639 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d38007ce0 con 0x7f8d401b34a0 2021-02-10T11:43:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.762 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "tenant_id": "tenant_id", "auth_id": "alice", "group_name": "group_0000000000285035"}) v1 -- 0x7f8d400ede60 con 0x7f8d280186c0 2021-02-10T11:43:37.758 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID alice with tenant tenant_id 2021-02-10T11:43:37.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.796 7f8d327fc700 1 -- 172.21.2.117:0/3098716744 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f8d34006390 con 0x7f8d280186c0 2021-02-10T11:43:37.815 INFO:teuthology.orchestra.run.gibba017.stdout:AQDpxiNgHXprLhAAX0TixACiHQrH7yECOphUbQ== 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 msgr2=0x7f8d2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 crc :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 msgr2=0x7f8d40192240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7f8d38001c10 tx=0x7f8d3800aca0).stop 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 shutdown_connections 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401b34a0 0x7f8d40192240 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401911f0 0x7f8d40191c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.827 7f8d47c47700 1 --2- 172.21.2.117:0/3098716744 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d40191750 0x7f8d40190bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:37.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.828 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 >> 172.21.2.117:0/3098716744 conn(0x7f8d4019c3f0 msgr2=0x7f8d401b8640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:37.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.828 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 shutdown_connections 2021-02-10T11:43:37.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:37.828 7f8d47c47700 1 -- 172.21.2.117:0/3098716744 wait complete. 2021-02-10T11:43:37.826 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:43:38.069 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:38.072 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:43:38.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.246 7f0a9640d700 1 Processor -- start 2021-02-10T11:43:38.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 -- start start 2021-02-10T11:43:38.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900d18e0 0x7f0a900ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0a900ef3f0 0x7f0a901b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a900ede60 con 0x7f0a900eb000 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a90088020 con 0x7f0a900ef3f0 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a9640d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900d1ce0 con 0x7f0a900d18e0 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a8f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0a900eb000 0x7f0a900eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50608/0 (socket says 172.21.2.117:50608) 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.247 7f0a8f7fe700 1 -- 172.21.2.117:0/1353192226 learned_addr learned my addr 172.21.2.117:0/1353192226 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a94a2a700 1 --2- 172.21.2.117:0/1353192226 >> v2:172.21.2.125:3300/0 conn(0x7f0a900ef3f0 0x7f0a901b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900d18e0 0x7f0a900ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720093556 0 0) 0x7f0a840010b0 con 0x7f0a900eb000 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a78002fe0 con 0x7f0a900eb000 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3901065676 0 0) 0x7f0a840010b0 con 0x7f0a900eb000 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900d18e0 msgr2=0x7f0a900ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900d18e0 0x7f0a900ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 >> v2:172.21.2.125:3300/0 conn(0x7f0a900ef3f0 msgr2=0x7f0a901b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 --2- 172.21.2.117:0/1353192226 >> v2:172.21.2.125:3300/0 conn(0x7f0a900ef3f0 0x7f0a901b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a901b8f90 con 0x7f0a900eb000 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0a840015b0 con 0x7f0a900eb000 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 >> v1:172.21.2.117:6789/0 conn(0x7f0a900eb000 legacy=0x7f0a900eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a78003ae0 0x7f0a78005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7804a840 0x7f0a7804cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7804d5b0 0x7f0a7804fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78050380 con 0x7f0a78003ae0 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780505a0 con 0x7f0a7804d5b0 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780507c0 con 0x7f0a7804a840 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a94a2a700 1 --2- 172.21.2.117:0/1353192226 >> v2:172.21.2.125:3300/0 conn(0x7f0a900ef3f0 0x7f0a901b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a78003ae0 0x7f0a78005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8f7fe700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7804a840 0x7f0a7804cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a94a2a700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7804d5b0 0x7f0a7804fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 -- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7804a840 msgr2=0x7f0a7804cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7804a840 0x7f0a7804cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 -- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7804d5b0 msgr2=0x7f0a7804fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a7804d5b0 0x7f0a7804fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.248 7f0a8ffff700 1 -- 172.21.2.117:0/1353192226 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78050a60 con 0x7f0a78003ae0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a8f7fe700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a7804a840 0x7f0a7804cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a8ffff700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a78003ae0 0x7f0a78005f90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f0a80006ac0 tx=0x7f0a80009ab0).ready entity=mon.0 client_cookie=ab159f9904ee29e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a8000c0c0 con 0x7f0a78003ae0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000c380 con 0x7f0a78003ae0 2021-02-10T11:43:38.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a8effd700 1 -- 172.21.2.117:0/1353192226 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a8000c8a0 con 0x7f0a78003ae0 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a9640d700 1 -- 172.21.2.117:0/1353192226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a78003ae0 msgr2=0x7f0a78005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a9640d700 1 --2- 172.21.2.117:0/1353192226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a78003ae0 0x7f0a78005f90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f0a80006ac0 tx=0x7f0a80009ab0).stop 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a94a2a700 1 -- 172.21.2.117:0/1353192226 reap_dead start 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a9640d700 1 -- 172.21.2.117:0/1353192226 shutdown_connections 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.249 7f0a9640d700 1 -- 172.21.2.117:0/1353192226 >> 172.21.2.117:0/1353192226 conn(0x7f0a900ced60 msgr2=0x7f0a900cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.250 7f0a9640d700 1 -- 172.21.2.117:0/1353192226 shutdown_connections 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.250 7f0a9640d700 1 -- 172.21.2.117:0/1353192226 wait complete. 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.264 7f0a9640d700 1 Processor -- start 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.264 7f0a9640d700 1 -- start start 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a900bfa80 0x7f0a901a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900a1d70 0x7f0a901a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a901a3c30 con 0x7f0a900bf680 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900a0680 con 0x7f0a900bfa80 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a9640d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a900a08a0 con 0x7f0a900a1d70 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50370/0 (socket says 172.21.2.117:50370) 2021-02-10T11:43:38.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 -- 172.21.2.117:0/778443499 learned_addr learned my addr 172.21.2.117:0/778443499 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:38.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8f7fe700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a900bfa80 0x7f0a901a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 -- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900a1d70 msgr2=0x7f0a901a3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900a1d70 0x7f0a901a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 -- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a900bfa80 msgr2=0x7f0a901a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a900bfa80 0x7f0a901a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 -- 172.21.2.117:0/778443499 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900a0ac0 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.265 7f0a8ffff700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f0a80006ac0 tx=0x7f0a8000b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a80009360 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80002030 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0a80002510 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a9640d700 1 -- 172.21.2.117:0/778443499 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a900a0ac0 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a9640d700 1 -- 172.21.2.117:0/778443499 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a900a0ac0 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0a80000c90 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 0x7f0a7001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.266 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0a800327f0 con 0x7f0a900bf680 2021-02-10T11:43:38.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.267 7f0a8f7fe700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 0x7f0a7001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:38.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.267 7f0a8f7fe700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 0x7f0a7001ac10 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:38.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.297 7f0a9640d700 1 -- 172.21.2.117:0/778443499 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a90087280 con 0x7f0a900bf680 2021-02-10T11:43:38.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.299 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0a80009000 con 0x7f0a900bf680 2021-02-10T11:43:38.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.434 7f0a9640d700 1 -- 172.21.2.117:0/778443499 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0a900ede60 con 0x7f0a900bf680 2021-02-10T11:43:38.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.434 7f0a8cff9700 1 -- 172.21.2.117:0/778443499 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) v1 ==== 80+0+4892 (secure 0 0 0) 0x7f0a80000c90 con 0x7f0a900bf680 2021-02-10T11:43:38.425 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:38.425 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.426 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:43:38.427 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.428 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.429 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:43:38.430 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.431 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:43:38.432 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:38.433 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.434 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.435 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.436 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.alice", 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQDpxiNgHXprLhAAX0TixACiHQrH7yECOphUbQ==", 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow rw path=/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63", 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.437 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:43:38.438 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:43:38.439 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:43:38.439 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.439 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:38.439 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:43:38.439 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 msgr2=0x7f0a7001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 0x7f0a7001ac10 crc :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 msgr2=0x7f0a900a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f0a80006ac0 tx=0x7f0a8000b9c0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 shutdown_connections 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0a900a1d70 0x7f0a901a3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0a900bfa80 0x7f0a901a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0a70018730 0x7f0a7001ac10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 --2- 172.21.2.117:0/778443499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0a900bf680 0x7f0a900a1750 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:38.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 >> 172.21.2.117:0/778443499 conn(0x7f0a900874b0 msgr2=0x7f0a90090b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:38.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 shutdown_connections 2021-02-10T11:43:38.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:38.437 7f0a9640d700 1 -- 172.21.2.117:0/778443499 wait complete. 2021-02-10T11:43:38.442 DEBUG:teuthology.orchestra.run.gibba017:> set -ex 2021-02-10T11:43:38.442 DEBUG:teuthology.orchestra.run.gibba017:> sudo dd of=/etc/ceph/ceph.client.alice.keyring 2021-02-10T11:43:38.787 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_client 20 2021-02-10T11:43:39.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 Processor -- start 2021-02-10T11:43:39.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 -- start start 2021-02-10T11:43:39.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc0d18f0 0x7f9bbc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9bbc0ef3f0 0x7f9bbc1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc0ede60 con 0x7f9bbc0eb040 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.211 7f9bc1e86700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bbc088020 con 0x7f9bbc0ef3f0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bc1e86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0d1d30 con 0x7f9bbc0d18f0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbaffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9bbc0eb040 0x7f9bbc0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50626/0 (socket says 172.21.2.117:50626) 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbaffd700 1 -- 172.21.2.117:0/2060494878 learned_addr learned my addr 172.21.2.117:0/2060494878 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 --2- 172.21.2.117:0/2060494878 >> v2:172.21.2.117:3300/0 conn(0x7f9bbc0ef3f0 0x7f9bbc1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbb7fe700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc0d18f0 0x7f9bbc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 -- 172.21.2.117:0/2060494878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc0d18f0 msgr2=0x7f9bbc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc0d18f0 0x7f9bbc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 -- 172.21.2.117:0/2060494878 >> v1:172.21.2.117:6789/0 conn(0x7f9bbc0eb040 legacy=0x7f9bbc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 -- 172.21.2.117:0/2060494878 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc1b8fd0 con 0x7f9bbc0ef3f0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.212 7f9bbbfff700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0ef3f0 0x7f9bbc1b79a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f9bb4006e30 tx=0x7f9bb4009e60).ready entity=mon.0 client_cookie=4e90b8a64453a92b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bba7fc700 1 -- 172.21.2.117:0/2060494878 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bb4006a50 con 0x7f9bbc0ef3f0 2021-02-10T11:43:39.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bba7fc700 1 -- 172.21.2.117:0/2060494878 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb400b300 con 0x7f9bbc0ef3f0 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bba7fc700 1 -- 172.21.2.117:0/2060494878 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bb400b7e0 con 0x7f9bbc0ef3f0 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 -- 172.21.2.117:0/2060494878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0ef3f0 msgr2=0x7f9bbc1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0ef3f0 0x7f9bbc1b79a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f9bb4006e30 tx=0x7f9bb4009e60).stop 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 -- 172.21.2.117:0/2060494878 shutdown_connections 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0ef3f0 0x7f9bbc1b79a0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 --2- 172.21.2.117:0/2060494878 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc0d18f0 0x7f9bbc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 -- 172.21.2.117:0/2060494878 >> 172.21.2.117:0/2060494878 conn(0x7f9bbc0cf060 msgr2=0x7f9bbc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.213 7f9bc1e86700 1 -- 172.21.2.117:0/2060494878 shutdown_connections 2021-02-10T11:43:39.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.214 7f9bc1e86700 1 -- 172.21.2.117:0/2060494878 wait complete. 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.228 7f9bc1e86700 1 Processor -- start 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.228 7f9bc1e86700 1 -- start start 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc07c7c0 0x7f9bbc1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bbc1a3930 0x7f9bbc0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc07cca0 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc18fb70 con 0x7f9bbc1a3930 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bc1e86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0dc6d0 con 0x7f9bbc07c7c0 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50384/0 (socket says 172.21.2.117:50384) 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 -- 172.21.2.117:0/3947774714 learned_addr learned my addr 172.21.2.117:0/3947774714 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:39.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 -- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc07c7c0 msgr2=0x7f9bbc1a3310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc07c7c0 0x7f9bbc1a3310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 -- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bbc1a3930 msgr2=0x7f9bbc0a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bbc1a3930 0x7f9bbc0a0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 -- 172.21.2.117:0/3947774714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc07d220 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bbbfff700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7f9bb4006ab0 tx=0x7f9bb400bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bb4009070 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb4001270 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.229 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9bb4001790 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.230 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc07d220 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.230 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bbc07d220 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.230 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9bb4001450 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.230 7f9bb3fff700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 0x7f9ba001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.230 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9bb4032c90 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.231 7f9bbb7fe700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 0x7f9ba001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.231 7f9bbb7fe700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 0x7f9ba001abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.261 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bbc0daff0 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.264 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9bb4003070 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.410 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) v1 -- 0x7f9bbc0daff0 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.414 7f9bb3fff700 1 -- 172.21.2.117:0/3947774714 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v15)=0 v15) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9bb4001270 con 0x7f9bbc0a13a0 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 msgr2=0x7f9ba001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 0x7f9ba001abc0 crc :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 msgr2=0x7f9bbc0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7f9bb4006ab0 tx=0x7f9bb400bd60).stop 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 shutdown_connections 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ba0018730 0x7f9ba001abc0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bbc0a13a0 0x7f9bbc0a17a0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bbc1a3930 0x7f9bbc0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 --2- 172.21.2.117:0/3947774714 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bbc07c7c0 0x7f9bbc1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 >> 172.21.2.117:0/3947774714 conn(0x7f9bbc087090 msgr2=0x7f9bbc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.419 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 shutdown_connections 2021-02-10T11:43:39.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.420 7f9bc1e86700 1 -- 172.21.2.117:0/3947774714 wait complete. 2021-02-10T11:43:39.418 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set client.alice debug_objecter 20 2021-02-10T11:43:39.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 Processor -- start 2021-02-10T11:43:39.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 -- start start 2021-02-10T11:43:39.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94b40d18f0 0x7f94b40eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 0x7f94b41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b40ef7f0 con 0x7f94b40eb040 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b40ede60 con 0x7f94b40d18f0 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94ba5d3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b40d1cf0 con 0x7f94b40ef3f0 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.841 7f94b3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94b40d18f0 0x7f94b40eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94b40d18f0 0x7f94b40eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50388/0 (socket says 172.21.2.117:50388) 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 -- 172.21.2.117:0/2161454942 learned_addr learned my addr 172.21.2.117:0/2161454942 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 -- 172.21.2.117:0/2161454942 >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 msgr2=0x7f94b41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b8bf0700 1 --2- 172.21.2.117:0/2161454942 >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 0x7f94b41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 --2- 172.21.2.117:0/2161454942 >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 0x7f94b41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 -- 172.21.2.117:0/2161454942 >> v1:172.21.2.125:6789/0 conn(0x7f94b40eb040 legacy=0x7f94b40eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 -- 172.21.2.117:0/2161454942 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b41b8fd0 con 0x7f94b40d18f0 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b8bf0700 1 --2- 172.21.2.117:0/2161454942 >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 0x7f94b41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:39.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.842 7f94b3fff700 1 --2- 172.21.2.117:0/2161454942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40d18f0 0x7f94b40eaa20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f94a4006700 tx=0x7f94a4009b80).ready entity=mon.0 client_cookie=e94fc573c31cef57 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94b2ffd700 1 -- 172.21.2.117:0/2161454942 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a4000c90 con 0x7f94b40d18f0 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94b2ffd700 1 -- 172.21.2.117:0/2161454942 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a400c0c0 con 0x7f94b40d18f0 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94b2ffd700 1 -- 172.21.2.117:0/2161454942 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a400c5a0 con 0x7f94b40d18f0 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 -- 172.21.2.117:0/2161454942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40d18f0 msgr2=0x7f94b40eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 --2- 172.21.2.117:0/2161454942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40d18f0 0x7f94b40eaa20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f94a4006700 tx=0x7f94a4009b80).stop 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 -- 172.21.2.117:0/2161454942 shutdown_connections 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 --2- 172.21.2.117:0/2161454942 >> v2:172.21.2.125:3300/0 conn(0x7f94b40ef3f0 0x7f94b41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 --2- 172.21.2.117:0/2161454942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40d18f0 0x7f94b40eaa20 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 -- 172.21.2.117:0/2161454942 >> 172.21.2.117:0/2161454942 conn(0x7f94b40cf060 msgr2=0x7f94b40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 -- 172.21.2.117:0/2161454942 shutdown_connections 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.843 7f94ba5d3700 1 -- 172.21.2.117:0/2161454942 wait complete. 2021-02-10T11:43:39.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 Processor -- start 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 -- start start 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94b40a1930 0x7f94b408a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94b40b5d20 0x7f94b40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40ace70 con 0x7f94b40ac450 2021-02-10T11:43:39.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40ad090 con 0x7f94b40a1930 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.858 7f94ba5d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40b6740 con 0x7f94b40b5d20 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50394/0 (socket says 172.21.2.117:50394) 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 -- 172.21.2.117:0/1225611026 learned_addr learned my addr 172.21.2.117:0/1225611026 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 -- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94b40b5d20 msgr2=0x7f94b40b6120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94b40b5d20 0x7f94b40b6120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 -- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94b40a1930 msgr2=0x7f94b408a220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94b40a1930 0x7f94b408a220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 -- 172.21.2.117:0/1225611026 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b40b6960 con 0x7f94b40ac450 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b37fe700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f94a800b5b0 tx=0x7f94a800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a800bd40 con 0x7f94b40ac450 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8000e20 con 0x7f94b40ac450 2021-02-10T11:43:39.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f94a80054a0 con 0x7f94b40ac450 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b40b6960 con 0x7f94b40ac450 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.859 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b40b49a0 con 0x7f94b40ac450 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.860 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f94a8006030 con 0x7f94b40ac450 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.860 7f94b0ff9700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 0x7f949401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.860 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f94a8032900 con 0x7f94b40ac450 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.860 7f94b3fff700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 0x7f949401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:39.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.861 7f94b3fff700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 0x7f949401abd0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.891 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b40f67e0 con 0x7f94b40ac450 2021-02-10T11:43:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:39.894 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f94a800bd40 con 0x7f94b40ac450 2021-02-10T11:43:40.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.042 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=debug_objecter}] v 0) v1 -- 0x7f94b40f67e0 con 0x7f94b40ac450 2021-02-10T11:43:40.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.045 7f94b0ff9700 1 -- 172.21.2.117:0/1225611026 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_objecter}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f94a8006030 con 0x7f94b40ac450 2021-02-10T11:43:40.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 msgr2=0x7f949401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 0x7f949401abd0 crc :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 msgr2=0x7f94b40ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f94a800b5b0 tx=0x7f94a800b950).stop 2021-02-10T11:43:40.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 shutdown_connections 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f94940186d0 0x7f949401abd0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94b40ac450 0x7f94b40ac850 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94b40b5d20 0x7f94b40b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 --2- 172.21.2.117:0/1225611026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94b40a1930 0x7f94b408a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 >> 172.21.2.117:0/1225611026 conn(0x7f94b40b6c70 msgr2=0x7f94b40970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 shutdown_connections 2021-02-10T11:43:40.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.051 7f94ba5d3700 1 -- 172.21.2.117:0/1225611026 wait complete. 2021-02-10T11:43:40.050 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:43:40.053 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:43:40.413 INFO:teuthology.orchestra.run.gibba017.stdout:[global] 2021-02-10T11:43:40.413 INFO:teuthology.orchestra.run.gibba017.stdout: chdir = "" 2021-02-10T11:43:40.413 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: auth supported = cephx 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: filestore xattr use omap = true 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: mon clock drift allowed = 1.000 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: osd crush chooseleaf type = 0 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: auth debug = true 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on old message = true 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on bug = true 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.414 INFO:teuthology.orchestra.run.gibba017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon pg warn max object skew = 0 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default size = 2 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow primary affinity = true 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow pg remap = true 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on no sortbitwise = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on too few osds = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:43:40.415 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: osd default data pool replay window = 5 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: mon allow pool delete = true 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: mon cluster log file level = debug 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: debug asserts on shutdown = true 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: mon health detail to clog = false 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:43:40.416 INFO:teuthology.orchestra.run.gibba017.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout:[osd] 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd journal size = 100 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub load threshold = 5.0 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub max interval = 600 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd recover clone overlap = true 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd recovery max chunk = 1048576 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug shutdown = true 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug op order = true 2021-02-10T11:43:40.417 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug verify stray on activate = true 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: osd open classes on start = true 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug pg log writeout = true 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: osd map max advance = 10 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: journal zero on create = true 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: filestore apply finisher threads = 3 2021-02-10T11:43:40.418 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: bdev debug aio = true 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug misdirected ops = true 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore block size = 96636764160 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore fsck on mount = True 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluefs = 20 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluestore = 20 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug filestore = 20 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug journal = 20 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 20 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug osd = 25 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: debug rocksdb = 10 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd full ratio = 0.9 2021-02-10T11:43:40.419 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: osd objectstore = bluestore 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr] 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: debug mgr = 20 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 10 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.420 INFO:teuthology.orchestra.run.gibba017.stdout:[mon] 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: debug paxos = 20 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon data avail warn = 5 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon mgr mkfs grace = 240 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd reporter subtree level = osd 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd prime pg temp = true 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: mon op complaint time = 120 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout:[client] 2021-02-10T11:43:40.421 INFO:teuthology.orchestra.run.gibba017.stdout: rgw cache enabled = true 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable ops log = true 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable usage log = true 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: client mount timeout = 600 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 20 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = "" 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.a] 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.b] 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.c] 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:43:40.422 INFO:teuthology.orchestra.run.gibba017.stdout:[mds] 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: debug mds = 20 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal frag = True 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal fragment size max = 10000 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal merge size = 5 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split bits = 3 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split size = 100 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug frag = True 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug scatterstat = True 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds op complaint time = 180 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: mds verify scatter = True 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout:[client.alice] 2021-02-10T11:43:40.423 INFO:teuthology.orchestra.run.gibba017.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:43:40.424 INFO:teuthology.orchestra.run.gibba025.stdout:[global] 2021-02-10T11:43:40.424 INFO:teuthology.orchestra.run.gibba025.stdout: chdir = "" 2021-02-10T11:43:40.424 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: auth supported = cephx 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: filestore xattr use omap = true 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: mon clock drift allowed = 1.000 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: osd crush chooseleaf type = 0 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: auth debug = true 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on old message = true 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on bug = true 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:43:40.425 INFO:teuthology.orchestra.run.gibba025.stdout: mon pg warn max object skew = 0 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default size = 2 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow primary affinity = true 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow pg remap = true 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on no sortbitwise = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on too few osds = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:43:40.426 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: osd default data pool replay window = 5 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: mon allow pool delete = true 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: mon cluster log file level = debug 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: debug asserts on shutdown = true 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: mon health detail to clog = false 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:43:40.427 INFO:teuthology.orchestra.run.gibba025.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout:[osd] 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd journal size = 100 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub load threshold = 5.0 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub max interval = 600 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd recover clone overlap = true 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd recovery max chunk = 1048576 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug shutdown = true 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug op order = true 2021-02-10T11:43:40.428 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug verify stray on activate = true 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: osd open classes on start = true 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug pg log writeout = true 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: osd map max advance = 10 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: journal zero on create = true 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: filestore apply finisher threads = 3 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: bdev debug aio = true 2021-02-10T11:43:40.429 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug misdirected ops = true 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore block size = 96636764160 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore fsck on mount = True 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluefs = 20 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluestore = 20 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug filestore = 20 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug journal = 20 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 20 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug osd = 25 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: debug rocksdb = 10 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd full ratio = 0.9 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:43:40.430 INFO:teuthology.orchestra.run.gibba025.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: osd objectstore = bluestore 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr] 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug mgr = 20 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 10 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout:[mon] 2021-02-10T11:43:40.431 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: debug paxos = 20 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon data avail warn = 5 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon mgr mkfs grace = 240 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd reporter subtree level = osd 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd prime pg temp = true 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: mon op complaint time = 120 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout:[client] 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: rgw cache enabled = true 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable ops log = true 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable usage log = true 2021-02-10T11:43:40.432 INFO:teuthology.orchestra.run.gibba025.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: client mount timeout = 600 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 20 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = "" 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.a] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.b] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.c] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout:[mds] 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: debug mds = 20 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:43:40.433 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal frag = True 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal fragment size max = 10000 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal merge size = 5 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split bits = 3 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split size = 100 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug frag = True 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug scatterstat = True 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds op complaint time = 180 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: mds verify scatter = True 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout:[client.alice] 2021-02-10T11:43:40.434 INFO:teuthology.orchestra.run.gibba025.stdout: keyring = /etc/ceph/ceph.client.alice.keyring 2021-02-10T11:43:40.435 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 Processor -- start 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 -- start start 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 0x7efee81b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efee81b3d20 0x7efee80f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee80e5350 0x7efee80f4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80f53c0 con 0x7efee81b3d20 2021-02-10T11:43:40.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80f4950 con 0x7efee80e4f50 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.856 7efeeeffb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee80f4b70 con 0x7efee80e5350 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efee81b3d20 0x7efee80f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efee81b3d20 0x7efee80f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50400/0 (socket says 172.21.2.117:50400) 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 -- 172.21.2.117:0/4244718739 learned_addr learned my addr 172.21.2.117:0/4244718739 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efeece17700 1 --2- 172.21.2.117:0/4244718739 >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 0x7efee81b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efeed618700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee80e5350 0x7efee80f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 -- 172.21.2.117:0/4244718739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee80e5350 msgr2=0x7efee80f4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee80e5350 0x7efee80f4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 -- 172.21.2.117:0/4244718739 >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 msgr2=0x7efee81b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 --2- 172.21.2.117:0/4244718739 >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 0x7efee81b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 -- 172.21.2.117:0/4244718739 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80f4d90 con 0x7efee81b3d20 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efeece17700 1 --2- 172.21.2.117:0/4244718739 >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 0x7efee81b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee7fff700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee81b3d20 0x7efee80f3d10 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7efed8006700 tx=0x7efed8009b80).ready entity=mon.0 client_cookie=11de4307e68ccc4b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee77fe700 1 -- 172.21.2.117:0/4244718739 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed8006340 con 0x7efee81b3d20 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee77fe700 1 -- 172.21.2.117:0/4244718739 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed800c0c0 con 0x7efee81b3d20 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.857 7efee77fe700 1 -- 172.21.2.117:0/4244718739 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed800c5a0 con 0x7efee81b3d20 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 -- 172.21.2.117:0/4244718739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee81b3d20 msgr2=0x7efee80f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee81b3d20 0x7efee80f3d10 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7efed8006700 tx=0x7efed8009b80).stop 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 -- 172.21.2.117:0/4244718739 shutdown_connections 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee80e5350 0x7efee80f4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 --2- 172.21.2.117:0/4244718739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee81b3d20 0x7efee80f3d10 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 --2- 172.21.2.117:0/4244718739 >> v2:172.21.2.125:3300/0 conn(0x7efee80e4f50 0x7efee81b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 -- 172.21.2.117:0/4244718739 >> 172.21.2.117:0/4244718739 conn(0x7efee80ce5b0 msgr2=0x7efee80e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:40.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 -- 172.21.2.117:0/4244718739 shutdown_connections 2021-02-10T11:43:40.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.858 7efeeeffb700 1 -- 172.21.2.117:0/4244718739 wait complete. 2021-02-10T11:43:40.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 Processor -- start 2021-02-10T11:43:40.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 -- start start 2021-02-10T11:43:40.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efee8105ef0 0x7efee819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee819feb0 0x7efee81ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee81062f0 con 0x7efee819fab0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee81aed10 con 0x7efee8105ef0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeeeffb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee81aef30 con 0x7efee819feb0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efee7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efee7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50410/0 (socket says 172.21.2.117:50410) 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeed618700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee819feb0 0x7efee81ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efee7fff700 1 -- 172.21.2.117:0/2309211341 learned_addr learned my addr 172.21.2.117:0/2309211341 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.873 7efeece17700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efee8105ef0 0x7efee819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 -- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee819feb0 msgr2=0x7efee81ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee819feb0 0x7efee81ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 -- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efee8105ef0 msgr2=0x7efee819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efee8105ef0 0x7efee819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 -- 172.21.2.117:0/2309211341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee81acc20 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee7fff700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7efed8009070 tx=0x7efed8002620).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed800c0d0 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed8000940 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efed800c5c0 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee81acc20 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.874 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee81ad1b0 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.875 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efed80097f0 con 0x7efee819fab0 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.875 7efee57fa700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 0x7efecc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.875 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efed8032550 con 0x7efee819fab0 2021-02-10T11:43:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.875 7efeece17700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 0x7efecc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.876 7efeece17700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 0x7efecc01abc0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:40.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.906 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee81022b0 con 0x7efee819fab0 2021-02-10T11:43:40.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:40.908 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efed8025fb0 con 0x7efee819fab0 2021-02-10T11:43:41.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.030 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efee81022b0 con 0x7efee819fab0 2021-02-10T11:43:41.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.031 7efee57fa700 1 -- 172.21.2.117:0/2309211341 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7efed8025fb0 con 0x7efee819fab0 2021-02-10T11:43:41.022 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:41.022 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.032 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 msgr2=0x7efecc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.032 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 0x7efecc01abc0 crc :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.032 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 msgr2=0x7efee81ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.032 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7efed8009070 tx=0x7efed8002620).stop 2021-02-10T11:43:41.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 shutdown_connections 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efecc0186c0 0x7efecc01abc0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efee819fab0 0x7efee81ae0d0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efee819feb0 0x7efee81ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 --2- 172.21.2.117:0/2309211341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efee8105ef0 0x7efee819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 >> 172.21.2.117:0/2309211341 conn(0x7efee809b640 msgr2=0x7efee8106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 shutdown_connections 2021-02-10T11:43:41.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.033 7efeeeffb700 1 -- 172.21.2.117:0/2309211341 wait complete. 2021-02-10T11:43:41.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.649220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.719391', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:41.033 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:41.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 Processor -- start 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 -- start start 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 0x7f00d41b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40f53d0 con 0x7f00d40e5360 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d40f6930 con 0x7f00d41b3d30 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.458 7f00dadd9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40f6b50 con 0x7f00d40e4f60 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d3fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f00d41b3d30 0x7f00d40f5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:43590/0 (socket says 172.21.2.117:43590) 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d93f6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f00d40e5360 0x7f00d40f6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50668/0 (socket says 172.21.2.117:50668) 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d3fff700 1 -- 172.21.2.117:0/3865587717 learned_addr learned my addr 172.21.2.117:0/3865587717 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d8bf5700 1 --2- 172.21.2.117:0/3865587717 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 0x7f00d41b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951183890 0 0) 0x7f00cc000c10 con 0x7f00d40e5360 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c0002fe0 con 0x7f00d40e5360 2021-02-10T11:43:41.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732015306 0 0) 0x7f00c4001440 con 0x7f00d41b3d30 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c0003210 con 0x7f00d41b3d30 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d8bf5700 1 -- 172.21.2.117:0/3865587717 >> v1:172.21.2.125:6789/0 conn(0x7f00d41b3d30 legacy=0x7f00d40f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d8bf5700 1 -- 172.21.2.117:0/3865587717 >> v1:172.21.2.117:6789/0 conn(0x7f00d40e5360 legacy=0x7f00d40f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d8bf5700 1 -- 172.21.2.117:0/3865587717 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d40f4d30 con 0x7f00d40e4f60 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.459 7f00d8bf5700 1 --2- 172.21.2.117:0/3865587717 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 0x7f00d41b3710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f00bc006650 tx=0x7f00bc009a80).ready entity=mon.2 client_cookie=d7d29d57a968f14f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.460 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00bc00c0c0 con 0x7f00d40e4f60 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.460 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc00c340 con 0x7f00d40e4f60 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.460 7f00d37fe700 1 -- 172.21.2.117:0/3865587717 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00bc00c860 con 0x7f00d40e4f60 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.460 7f00dadd9700 1 -- 172.21.2.117:0/3865587717 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 msgr2=0x7f00d41b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.460 7f00dadd9700 1 --2- 172.21.2.117:0/3865587717 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 0x7f00d41b3710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f00bc006650 tx=0x7f00bc009a80).stop 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.461 7f00dadd9700 1 -- 172.21.2.117:0/3865587717 shutdown_connections 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.461 7f00dadd9700 1 --2- 172.21.2.117:0/3865587717 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40e4f60 0x7f00d41b3710 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.461 7f00dadd9700 1 -- 172.21.2.117:0/3865587717 >> 172.21.2.117:0/3865587717 conn(0x7f00d40ce5c0 msgr2=0x7f00d40e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.461 7f00dadd9700 1 -- 172.21.2.117:0/3865587717 shutdown_connections 2021-02-10T11:43:41.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.461 7f00dadd9700 1 -- 172.21.2.117:0/3865587717 wait complete. 2021-02-10T11:43:41.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.475 7f00dadd9700 1 Processor -- start 2021-02-10T11:43:41.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.475 7f00dadd9700 1 -- start start 2021-02-10T11:43:41.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00d419cde0 0x7f00d419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40c2b50 0x7f00d40c2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40c35f0 con 0x7f00d40a4d70 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40a44c0 con 0x7f00d419cde0 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00dadd9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d40c38b0 con 0x7f00d40c2b50 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00d93f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00d93f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50424/0 (socket says 172.21.2.117:50424) 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00d93f6700 1 -- 172.21.2.117:0/140207875 learned_addr learned my addr 172.21.2.117:0/140207875 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00d8bf5700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00d419cde0 0x7f00d419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.476 7f00d3fff700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40c2b50 0x7f00d40c2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 -- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40c2b50 msgr2=0x7f00d40c2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40c2b50 0x7f00d40c2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 -- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00d419cde0 msgr2=0x7f00d419d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00d419cde0 0x7f00d419d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 -- 172.21.2.117:0/140207875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d40c3da0 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d93f6700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7f00cc00b560 tx=0x7f00cc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc00bcf0 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc006030 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00cc004590 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00dadd9700 1 -- 172.21.2.117:0/140207875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d40c3da0 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.477 7f00dadd9700 1 -- 172.21.2.117:0/140207875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d40a5a80 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.478 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f00cc006210 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.478 7f00d17fa700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 0x7f00b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.478 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f00cc0327f0 con 0x7f00d40a4d70 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.478 7f00d8bf5700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 0x7f00b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:41.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.479 7f00d8bf5700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 0x7f00b801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:41.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.508 7f00dadd9700 1 -- 172.21.2.117:0/140207875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d40a5a80 con 0x7f00d40a4d70 2021-02-10T11:43:41.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.510 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f00cc0043a0 con 0x7f00d40a4d70 2021-02-10T11:43:41.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.639 7f00dadd9700 1 -- 172.21.2.117:0/140207875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f00d40a5a80 con 0x7f00d40a4d70 2021-02-10T11:43:41.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.640 7f00d17fa700 1 -- 172.21.2.117:0/140207875 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7f00cc006030 con 0x7f00d40a4d70 2021-02-10T11:43:41.631 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:41.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.641 7f00dadd9700 1 -- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 msgr2=0x7f00b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.641 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 0x7f00b801abc0 crc :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.641 7f00dadd9700 1 -- 172.21.2.117:0/140207875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 msgr2=0x7f00d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.641 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7f00cc00b560 tx=0x7f00cc00b900).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 -- 172.21.2.117:0/140207875 shutdown_connections 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f00b80186c0 0x7f00b801abc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f00d40a4d70 0x7f00d40a5170 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f00d40c2b50 0x7f00d40c2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 --2- 172.21.2.117:0/140207875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f00d419cde0 0x7f00d419d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 -- 172.21.2.117:0/140207875 >> 172.21.2.117:0/140207875 conn(0x7f00d40a1e80 msgr2=0x7f00d40b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 -- 172.21.2.117:0/140207875 shutdown_connections 2021-02-10T11:43:41.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:41.642 7f00dadd9700 1 -- 172.21.2.117:0/140207875 wait complete. 2021-02-10T11:43:41.641 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:42.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.064 7f78d0490700 1 Processor -- start 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.064 7f78d0490700 1 -- start start 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78d0490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 0x7f78c80d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78d0490700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c80f5130 con 0x7f78c80e5360 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78d0490700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c81b3f00 con 0x7f78c80e4f60 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78d0490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80d1cc0 con 0x7f78c80f4d30 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cdaab700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f78c80e5360 0x7f78c80f5950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50680/0 (socket says 172.21.2.117:50680) 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cdaab700 1 -- 172.21.2.117:0/3646536965 learned_addr learned my addr 172.21.2.117:0/3646536965 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78ceaad700 1 --2- 172.21.2.117:0/3646536965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 0x7f78c80d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375783121 0 0) 0x7f78c40009b0 con 0x7f78c80e5360 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b4002fe0 con 0x7f78c80e5360 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825839423 0 0) 0x7f78bc001080 con 0x7f78c80e4f60 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b4004370 con 0x7f78c80e4f60 2021-02-10T11:43:42.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78ceaad700 1 -- 172.21.2.117:0/3646536965 >> v1:172.21.2.125:6789/0 conn(0x7f78c80e4f60 legacy=0x7f78c81b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78ceaad700 1 -- 172.21.2.117:0/3646536965 >> v1:172.21.2.117:6789/0 conn(0x7f78c80e5360 legacy=0x7f78c80f5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.065 7f78ceaad700 1 -- 172.21.2.117:0/3646536965 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c80d1e70 con 0x7f78c80f4d30 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78ceaad700 1 --2- 172.21.2.117:0/3646536965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 0x7f78c80d16a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f78c0006e50 tx=0x7f78c0009e70).ready entity=mon.2 client_cookie=8d186289437af2df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c000b510 con 0x7f78c80f4d30 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0006790 con 0x7f78c80f4d30 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78cd2aa700 1 -- 172.21.2.117:0/3646536965 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c0000f90 con 0x7f78c80f4d30 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78d0490700 1 -- 172.21.2.117:0/3646536965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 msgr2=0x7f78c80d16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.066 7f78d0490700 1 --2- 172.21.2.117:0/3646536965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 0x7f78c80d16a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f78c0006e50 tx=0x7f78c0009e70).stop 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.067 7f78d0490700 1 -- 172.21.2.117:0/3646536965 shutdown_connections 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.067 7f78d0490700 1 --2- 172.21.2.117:0/3646536965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80f4d30 0x7f78c80d16a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.067 7f78d0490700 1 -- 172.21.2.117:0/3646536965 >> 172.21.2.117:0/3646536965 conn(0x7f78c80ce5c0 msgr2=0x7f78c80e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.067 7f78d0490700 1 -- 172.21.2.117:0/3646536965 shutdown_connections 2021-02-10T11:43:42.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.067 7f78d0490700 1 -- 172.21.2.117:0/3646536965 wait complete. 2021-02-10T11:43:42.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.081 7f78d0490700 1 Processor -- start 2021-02-10T11:43:42.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.081 7f78d0490700 1 -- start start 2021-02-10T11:43:42.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 0x7f78c80a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78c81a03e0 0x7f78c81a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80a0700 con 0x7f78c80a24b0 2021-02-10T11:43:42.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c809e600 con 0x7f78c81a03e0 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78d0490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c80b1180 con 0x7f78c80a0c00 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50436/0 (socket says 172.21.2.117:50436) 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 -- 172.21.2.117:0/526645565 learned_addr learned my addr 172.21.2.117:0/526645565 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78ce2ac700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 0x7f78c80a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78ceaad700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78c81a03e0 0x7f78c81a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 -- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 msgr2=0x7f78c80a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 0x7f78c80a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 -- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78c81a03e0 msgr2=0x7f78c81a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78c81a03e0 0x7f78c81a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 -- 172.21.2.117:0/526645565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c80b13a0 con 0x7f78c80a24b0 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78ce2ac700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 0x7f78c80a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:42.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78cdaab700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f78c4001c60 tx=0x7f78c400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.082 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c400acc0 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c4004140 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78c4004660 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78d0490700 1 -- 172.21.2.117:0/526645565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c80b13a0 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78d0490700 1 -- 172.21.2.117:0/526645565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c80b13a0 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78c4004320 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78baffd700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 0x7f78b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78c4032540 con 0x7f78c80a24b0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.083 7f78ce2ac700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 0x7f78b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.084 7f78ce2ac700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 0x7f78b001abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.115 7f78d0490700 1 -- 172.21.2.117:0/526645565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c807dea0 con 0x7f78c80a24b0 2021-02-10T11:43:42.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.117 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78c4000e20 con 0x7f78c80a24b0 2021-02-10T11:43:42.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.245 7f78d0490700 1 -- 172.21.2.117:0/526645565 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f78c807dea0 con 0x7f78c80a24b0 2021-02-10T11:43:42.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.245 7f78baffd700 1 -- 172.21.2.117:0/526645565 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f78c401f2b0 con 0x7f78c80a24b0 2021-02-10T11:43:42.236 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:42.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 -- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 msgr2=0x7f78b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 0x7f78b001abc0 crc :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 -- 172.21.2.117:0/526645565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 msgr2=0x7f78c80a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f78c4001c60 tx=0x7f78c400a8d0).stop 2021-02-10T11:43:42.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 -- 172.21.2.117:0/526645565 shutdown_connections 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78b00186c0 0x7f78b001abc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78c80a24b0 0x7f78c80a28b0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78c81a03e0 0x7f78c81a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 --2- 172.21.2.117:0/526645565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78c80a0c00 0x7f78c80a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.247 7f78d0490700 1 -- 172.21.2.117:0/526645565 >> 172.21.2.117:0/526645565 conn(0x7f78c80bfac0 msgr2=0x7f78c80800e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.248 7f78d0490700 1 -- 172.21.2.117:0/526645565 shutdown_connections 2021-02-10T11:43:42.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.248 7f78d0490700 1 -- 172.21.2.117:0/526645565 wait complete. 2021-02-10T11:43:42.246 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.669 7faab9e57700 1 Processor -- start 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.669 7faab9e57700 1 -- start start 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab9e57700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faab40e4f60 0x7faab41b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab9e57700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 0x7faab40f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab9e57700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40f53d0 con 0x7faab40e5360 2021-02-10T11:43:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab9e57700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40f6960 con 0x7faab40e4f60 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab9e57700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40f6b80 con 0x7faab41b3d30 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faab40e4f60 0x7faab41b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faab40e4f60 0x7faab41b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50442/0 (socket says 172.21.2.117:50442) 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 -- 172.21.2.117:0/4261733637 learned_addr learned my addr 172.21.2.117:0/4261733637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab2ffd700 1 --2- 172.21.2.117:0/4261733637 >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 0x7faab40f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 -- 172.21.2.117:0/4261733637 >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 msgr2=0x7faab40f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 --2- 172.21.2.117:0/4261733637 >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 0x7faab40f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 -- 172.21.2.117:0/4261733637 >> v1:172.21.2.125:6789/0 conn(0x7faab40e5360 legacy=0x7faab40f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 -- 172.21.2.117:0/4261733637 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40f6da0 con 0x7faab40e4f60 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab2ffd700 1 --2- 172.21.2.117:0/4261733637 >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 0x7faab40f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.670 7faab37fe700 1 --2- 172.21.2.117:0/4261733637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab40e4f60 0x7faab41b3710 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7faaa4006700 tx=0x7faaa4009b80).ready entity=mon.0 client_cookie=1f711e93f958f4af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab27fc700 1 -- 172.21.2.117:0/4261733637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa400c0c0 con 0x7faab40e4f60 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab27fc700 1 -- 172.21.2.117:0/4261733637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400c340 con 0x7faab40e4f60 2021-02-10T11:43:42.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab27fc700 1 -- 172.21.2.117:0/4261733637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faaa400c860 con 0x7faab40e4f60 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 -- 172.21.2.117:0/4261733637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab40e4f60 msgr2=0x7faab41b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 --2- 172.21.2.117:0/4261733637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab40e4f60 0x7faab41b3710 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7faaa4006700 tx=0x7faaa4009b80).stop 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 -- 172.21.2.117:0/4261733637 shutdown_connections 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 --2- 172.21.2.117:0/4261733637 >> v2:172.21.2.125:3300/0 conn(0x7faab41b3d30 0x7faab40f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 --2- 172.21.2.117:0/4261733637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab40e4f60 0x7faab41b3710 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 -- 172.21.2.117:0/4261733637 >> 172.21.2.117:0/4261733637 conn(0x7faab40ce5c0 msgr2=0x7faab40e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 -- 172.21.2.117:0/4261733637 shutdown_connections 2021-02-10T11:43:42.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.671 7faab9e57700 1 -- 172.21.2.117:0/4261733637 wait complete. 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.686 7faab9e57700 1 Processor -- start 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.686 7faab9e57700 1 -- start start 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.686 7faab9e57700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faab4093ae0 0x7faab40940a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.686 7faab9e57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.686 7faab9e57700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faab4096350 0x7faab4096750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab9e57700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4095dd0 con 0x7faab4095610 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab9e57700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4094840 con 0x7faab4093ae0 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab9e57700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab4095a10 con 0x7faab4096350 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50446/0 (socket says 172.21.2.117:50446) 2021-02-10T11:43:42.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 -- 172.21.2.117:0/4020893605 learned_addr learned my addr 172.21.2.117:0/4020893605 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 -- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faab4096350 msgr2=0x7faab4096750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faab4096350 0x7faab4096750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 -- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faab4093ae0 msgr2=0x7faab40940a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faab4093ae0 0x7faab40940a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 -- 172.21.2.117:0/4020893605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4199020 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab2ffd700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7faa9c006f20 tx=0x7faa9c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa9c00bd80 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa9c000c90 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab9e57700 1 -- 172.21.2.117:0/4020893605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab4199020 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa9c0022b0 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.687 7faab9e57700 1 -- 172.21.2.117:0/4020893605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4097060 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.688 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa9c00a000 con 0x7faab4095610 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.688 7faaabfff700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 0x7faa9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:42.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.688 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faa9c032730 con 0x7faab4095610 2021-02-10T11:43:42.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.688 7faab37fe700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 0x7faa9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:42.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.689 7faab37fe700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 0x7faa9801abc0 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:42.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.719 7faab9e57700 1 -- 172.21.2.117:0/4020893605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab407dea0 con 0x7faab4095610 2021-02-10T11:43:42.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.722 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa9c0009b0 con 0x7faab4095610 2021-02-10T11:43:42.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.850 7faab9e57700 1 -- 172.21.2.117:0/4020893605 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7faab407dea0 con 0x7faab4095610 2021-02-10T11:43:42.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.850 7faaabfff700 1 -- 172.21.2.117:0/4020893605 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7faa9c00bd80 con 0x7faab4095610 2021-02-10T11:43:42.841 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 -- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 msgr2=0x7faa9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 0x7faa9801abc0 crc :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 -- 172.21.2.117:0/4020893605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 msgr2=0x7faab4198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 rx=0x7faa9c006f20 tx=0x7faa9c00b990).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 -- 172.21.2.117:0/4020893605 shutdown_connections 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa980186c0 0x7faa9801abc0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faab4095610 0x7faab4198a00 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faab4096350 0x7faab4096750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 --2- 172.21.2.117:0/4020893605 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faab4093ae0 0x7faab40940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:42.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.852 7faab9e57700 1 -- 172.21.2.117:0/4020893605 >> 172.21.2.117:0/4020893605 conn(0x7faab40ba5f0 msgr2=0x7faab40bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:42.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.853 7faab9e57700 1 -- 172.21.2.117:0/4020893605 shutdown_connections 2021-02-10T11:43:42.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:42.853 7faab9e57700 1 -- 172.21.2.117:0/4020893605 wait complete. 2021-02-10T11:43:42.851 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:42.851 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.270 7fb5a096c700 1 Processor -- start 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.270 7fb5a096c700 1 -- start start 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb5a096c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb59c0e5350 0x7fb59c0f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb5a096c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 0x7fb59c0f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb5a096c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c1b3ef0 con 0x7fb59c0e4f50 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb5a096c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59c0f5440 con 0x7fb59c0e5350 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb5a096c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0d1690 con 0x7fb59c0f5940 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb59a61c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb59c0e4f50 0x7fb59c1b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50704/0 (socket says 172.21.2.117:50704) 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb59a61c700 1 -- 172.21.2.117:0/2694303963 learned_addr learned my addr 172.21.2.117:0/2694303963 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb599e1b700 1 --2- 172.21.2.117:0/2694303963 >> v2:172.21.2.125:3300/0 conn(0x7fb59c0e5350 0x7fb59c0f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:43.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb59ae1d700 1 --2- 172.21.2.117:0/2694303963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 0x7fb59c0f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb59961a700 1 -- 172.21.2.117:0/2694303963 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288155983 0 0) 0x7fb5840009b0 con 0x7fb59c0e4f50 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.271 7fb59961a700 1 -- 172.21.2.117:0/2694303963 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb588002fe0 con 0x7fb59c0e4f50 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59ae1d700 1 -- 172.21.2.117:0/2694303963 >> v2:172.21.2.125:3300/0 conn(0x7fb59c0e5350 msgr2=0x7fb59c0f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59ae1d700 1 --2- 172.21.2.117:0/2694303963 >> v2:172.21.2.125:3300/0 conn(0x7fb59c0e5350 0x7fb59c0f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59ae1d700 1 -- 172.21.2.117:0/2694303963 >> v1:172.21.2.117:6789/0 conn(0x7fb59c0e4f50 legacy=0x7fb59c1b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59ae1d700 1 -- 172.21.2.117:0/2694303963 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c0d1880 con 0x7fb59c0f5940 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59ae1d700 1 --2- 172.21.2.117:0/2694303963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 0x7fb59c0f5d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb594006e50 tx=0x7fb594009e70).ready entity=mon.2 client_cookie=4f5d9286d308c808 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59961a700 1 -- 172.21.2.117:0/2694303963 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb594006a20 con 0x7fb59c0f5940 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59961a700 1 -- 172.21.2.117:0/2694303963 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb594006790 con 0x7fb59c0f5940 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb59961a700 1 -- 172.21.2.117:0/2694303963 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb59400b790 con 0x7fb59c0f5940 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb5a096c700 1 -- 172.21.2.117:0/2694303963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 msgr2=0x7fb59c0f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.272 7fb5a096c700 1 --2- 172.21.2.117:0/2694303963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 0x7fb59c0f5d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb594006e50 tx=0x7fb594009e70).stop 2021-02-10T11:43:43.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 -- 172.21.2.117:0/2694303963 shutdown_connections 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 --2- 172.21.2.117:0/2694303963 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c0f5940 0x7fb59c0f5d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 --2- 172.21.2.117:0/2694303963 >> v2:172.21.2.125:3300/0 conn(0x7fb59c0e5350 0x7fb59c0f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 -- 172.21.2.117:0/2694303963 >> 172.21.2.117:0/2694303963 conn(0x7fb59c0ce5b0 msgr2=0x7fb59c0e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 -- 172.21.2.117:0/2694303963 shutdown_connections 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.273 7fb5a096c700 1 -- 172.21.2.117:0/2694303963 wait complete. 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.288 7fb5a096c700 1 Processor -- start 2021-02-10T11:43:43.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.288 7fb5a096c700 1 -- start start 2021-02-10T11:43:43.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb59c105ef0 0x7fb59c19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c19feb0 0x7fb59c1ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c1062f0 con 0x7fb59c19fab0 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c1aed10 con 0x7fb59c105ef0 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb5a096c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c1aef30 con 0x7fb59c19feb0 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50464/0 (socket says 172.21.2.117:50464) 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 -- 172.21.2.117:0/1690400176 learned_addr learned my addr 172.21.2.117:0/1690400176 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 -- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c19feb0 msgr2=0x7fb59c1ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c19feb0 0x7fb59c1ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 -- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb59c105ef0 msgr2=0x7fb59c19f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb59c105ef0 0x7fb59c19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:43.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 -- 172.21.2.117:0/1690400176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c1acc20 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.289 7fb599e1b700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fb58c00b560 tx=0x7fb58c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb58c00bcf0 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c000c90 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c1acc20 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c1ad1b0 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb58c0022b0 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb58c006b00 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 0x7fb58001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.290 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb58c032910 con 0x7fb59c19fab0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.291 7fb59a61c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 0x7fb58001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:43.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.291 7fb59a61c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 0x7fb58001abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:43.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.320 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c1022b0 con 0x7fb59c19fab0 2021-02-10T11:43:43.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.323 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb58c0009b0 con 0x7fb59c19fab0 2021-02-10T11:43:43.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.450 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb59c1022b0 con 0x7fb59c19fab0 2021-02-10T11:43:43.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.450 7fb592ffd700 1 -- 172.21.2.117:0/1690400176 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb58c006b00 con 0x7fb59c19fab0 2021-02-10T11:43:43.442 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:43.442 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":45,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:43.442 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 45 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 msgr2=0x7fb58001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 0x7fb58001abc0 crc :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 msgr2=0x7fb59c1ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7fb58c00b560 tx=0x7fb58c00b900).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 shutdown_connections 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb580018730 0x7fb58001abc0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb59c19fab0 0x7fb59c1ae0d0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb59c19feb0 0x7fb59c1ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 --2- 172.21.2.117:0/1690400176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb59c105ef0 0x7fb59c19f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:43.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 >> 172.21.2.117:0/1690400176 conn(0x7fb59c09b640 msgr2=0x7fb59c106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:43.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 shutdown_connections 2021-02-10T11:43:43.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:43.452 7fb5a096c700 1 -- 172.21.2.117:0/1690400176 wait complete. 2021-02-10T11:43:43.655 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.674746', 'modified': '2021-02-10 11:43:10.684137', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/2942282551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 2942282551}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 2942282551}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:43.655 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:43.655 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:43:44.054 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:44.054 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:43:44.054 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 6670, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 35.386807937, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 44, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 64, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 41.020443000999997 2021-02-10T11:43:44.055 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:44.059 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6670, "want_state": "up:active", "state": "up:active", "rank_uptime": 35.386807937, "mdsmap_epoch": 44, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 41.020443000999997 } 2021-02-10T11:43:44.059 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:44.059 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:43:44.059 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:43:44.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:44.059 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:44.394 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:44.737 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:44.738 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:45.096 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:45.097 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:45.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:45.098 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:45.434 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:43:45.435 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:43:45.435 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:45.439 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:43:45.781 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:43:45.781 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:43:45.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[137360]: starting ceph client 2021-02-10T11:43:45.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:2021-02-10 11:43:45.856 7fe76f6cdf80 -1 init, newargv = 0x55ffcf64ee40 newargc=9 2021-02-10T11:43:45.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[137360]: starting fuse 2021-02-10T11:43:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:43:46.124 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:43:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:43:46.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:46.125 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:43:46.461 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:43:46.461 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:43:46.461 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:43:46.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:43:46.462 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.alice")) 2021-02-10T11:43:46.463 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:43:46.826 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.alice.137360.asok 2021-02-10T11:43:46.829 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.137360.asok 2021-02-10T11:43:46.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:46.829 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.137360.asok status 2021-02-10T11:43:47.218 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:47.218 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:43:47.218 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "alice", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "137360", 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63" 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8739, 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:43:47.219 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8739 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 691144372 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:43:47.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 691144372 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8739 172.21.2.117:0/691144372", 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/691144372", 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 44, 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 64, 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:43:47.221 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:47.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:47.223 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:47.223 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:47.559 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:43:47.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:47.559 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=data.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:43:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:43:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:43:47.937 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0277685 s, 37.8 MB/s 2021-02-10T11:43:47.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000934866 alice --group_name group_0000000000285035 --tenant_id tenant_id --access_level r 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 Processor -- start 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 -- start start 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdc2c0e53d0 0x7fdc2c0f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c0f59c0 0x7fdc2c0f5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c1b3f70 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c0f54c0 con 0x7fdc2c0e53d0 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.367 7fdc3112d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0d1710 con 0x7fdc2c0f59c0 2021-02-10T11:43:48.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc2ad9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdc2c0e4fd0 0x7fdc2c1b3780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50732/0 (socket says 172.21.2.117:50732) 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc2ad9d700 1 -- 172.21.2.117:0/3037240780 learned_addr learned my addr 172.21.2.117:0/3037240780 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612129277 0 0) 0x7fdc14001180 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc18002fe0 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc2a59c700 1 --2- 172.21.2.117:0/3037240780 >> v2:172.21.2.125:3300/0 conn(0x7fdc2c0e53d0 0x7fdc2c0f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc2b59e700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c0f59c0 0x7fdc2c0f5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3404869699 0 0) 0x7fdc14001180 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c0f59c0 msgr2=0x7fdc2c0f5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c0f59c0 0x7fdc2c0f5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 >> v2:172.21.2.125:3300/0 conn(0x7fdc2c0e53d0 msgr2=0x7fdc2c0f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 --2- 172.21.2.117:0/3037240780 >> v2:172.21.2.125:3300/0 conn(0x7fdc2c0e53d0 0x7fdc2c0f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0d1900 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fdc14001680 con 0x7fdc2c0e4fd0 2021-02-10T11:43:48.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 >> v1:172.21.2.117:6789/0 conn(0x7fdc2c0e4fd0 legacy=0x7fdc2c1b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc18003b10 0x7fdc18005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc180067b0 0x7fdc18008cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc180095b0 0x7fdc1800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1800c2d0 con 0x7fdc180067b0 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180504c0 con 0x7fdc18003b10 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180506e0 con 0x7fdc180095b0 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.368 7fdc2a59c700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc180067b0 0x7fdc18008cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2b59e700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc180095b0 0x7fdc1800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2ad9d700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc18003b10 0x7fdc18005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 -- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc180095b0 msgr2=0x7fdc1800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc180095b0 0x7fdc1800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 -- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc18003b10 msgr2=0x7fdc18005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:48.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc18003b10 0x7fdc18005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 -- 172.21.2.117:0/3037240780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18050980 con 0x7fdc180067b0 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2a59c700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc180067b0 0x7fdc18008cd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7fdc1c006360 tx=0x7fdc1c009c30).ready entity=mon.0 client_cookie=45e257fb06742dd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc1c0009b0 con 0x7fdc180067b0 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c000d20 con 0x7fdc180067b0 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc29d9b700 1 -- 172.21.2.117:0/3037240780 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc1c002220 con 0x7fdc180067b0 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc3112d700 1 -- 172.21.2.117:0/3037240780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc180067b0 msgr2=0x7fdc18008cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc3112d700 1 --2- 172.21.2.117:0/3037240780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc180067b0 0x7fdc18008cd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7fdc1c006360 tx=0x7fdc1c009c30).stop 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.369 7fdc2b59e700 1 -- 172.21.2.117:0/3037240780 reap_dead start 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.370 7fdc3112d700 1 -- 172.21.2.117:0/3037240780 shutdown_connections 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.370 7fdc3112d700 1 -- 172.21.2.117:0/3037240780 >> 172.21.2.117:0/3037240780 conn(0x7fdc2c0ce630 msgr2=0x7fdc2c0e4470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.370 7fdc3112d700 1 -- 172.21.2.117:0/3037240780 shutdown_connections 2021-02-10T11:43:48.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.370 7fdc3112d700 1 -- 172.21.2.117:0/3037240780 wait complete. 2021-02-10T11:43:48.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.384 7fdc3112d700 1 Processor -- start 2021-02-10T11:43:48.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.384 7fdc3112d700 1 -- start start 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c1038a0 0x7fdc2c103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc2c102970 0x7fdc2c102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c092ec0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c104370 con 0x7fdc2c102970 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc3112d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c093440 con 0x7fdc2c1038a0 2021-02-10T11:43:48.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50496/0 (socket says 172.21.2.117:50496) 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 -- 172.21.2.117:0/238266446 learned_addr learned my addr 172.21.2.117:0/238266446 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2a59c700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c1038a0 0x7fdc2c103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2b59e700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc2c102970 0x7fdc2c102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 -- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c1038a0 msgr2=0x7fdc2c103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c1038a0 0x7fdc2c103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 -- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc2c102970 msgr2=0x7fdc2c102d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc2c102970 0x7fdc2c102d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.385 7fdc2ad9d700 1 -- 172.21.2.117:0/238266446 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c093660 con 0x7fdc2c1034a0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc2ad9d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7fdc1400b530 tx=0x7fdc1400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc14006020 con 0x7fdc2c1034a0 2021-02-10T11:43:48.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc140063a0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc14004590 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc3112d700 1 -- 172.21.2.117:0/238266446 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c093660 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.386 7fdc3112d700 1 -- 172.21.2.117:0/238266446 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c093bf0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.387 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc14009fb0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.387 7fdc237fe700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 0x7fdc1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.387 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdc14032670 con 0x7fdc2c1034a0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.387 7fdc2a59c700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 0x7fdc1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:48.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.387 7fdc2a59c700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 0x7fdc1001abc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:48.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.417 7fdc3112d700 1 -- 172.21.2.117:0/238266446 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c093ea0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.420 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc140043a0 con 0x7fdc2c1034a0 2021-02-10T11:43:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.547 7fdc3112d700 1 -- 172.21.2.117:0/238266446 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "tenant_id": "tenant_id", "group_name": "group_0000000000285035", "access_level": "r", "prefix": "fs subvolume authorize", "auth_id": "alice"}) v1 -- 0x7fdc2c093bf0 con 0x7fdc100186c0 2021-02-10T11:43:48.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.590 7fdc237fe700 1 -- 172.21.2.117:0/238266446 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fdc1c006c80 con 0x7fdc100186c0 2021-02-10T11:43:48.602 INFO:teuthology.orchestra.run.gibba017.stdout:AQDpxiNgHXprLhAAX0TixACiHQrH7yECOphUbQ== 2021-02-10T11:43:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 -- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 msgr2=0x7fdc1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 0x7fdc1001abc0 crc :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 -- 172.21.2.117:0/238266446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 msgr2=0x7fdc2c0927c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7fdc1400b530 tx=0x7fdc1400b8d0).stop 2021-02-10T11:43:48.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 -- 172.21.2.117:0/238266446 shutdown_connections 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc2c102970 0x7fdc2c102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc2c1038a0 0x7fdc2c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc100186c0 0x7fdc1001abc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 --2- 172.21.2.117:0/238266446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc2c1034a0 0x7fdc2c0927c0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 -- 172.21.2.117:0/238266446 >> 172.21.2.117:0/238266446 conn(0x7fdc2c1b2470 msgr2=0x7fdc2c19df00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.612 7fdc3112d700 1 -- 172.21.2.117:0/238266446 shutdown_connections 2021-02-10T11:43:48.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:48.613 7fdc3112d700 1 -- 172.21.2.117:0/238266446 wait complete. 2021-02-10T11:43:48.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:48.613 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:48.613 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:48.950 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:43:48.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:43:48.951 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:43:48.951 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:43:48.951 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:49.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[137360]: fuse finished with error 0 and tester_r 0 2021-02-10T11:43:49.304 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:49.304 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:43:49.304 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:49.640 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:43:49.640 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:43:49.640 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:43:55.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:43:55.742 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:43:56.078 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.497 7fa03da3f700 1 Processor -- start 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.497 7fa03da3f700 1 -- start start 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa03da3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 0x7fa0381b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa03da3f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0380f53c0 con 0x7fa0381b3d20 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa03da3f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0380f6920 con 0x7fa0380e5350 2021-02-10T11:43:56.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa03da3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0380f6b40 con 0x7fa0380e4f50 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa036ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa0381b3d20 0x7fa0380f5940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50752/0 (socket says 172.21.2.117:50752) 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa036ffd700 1 -- 172.21.2.117:0/4146481807 learned_addr learned my addr 172.21.2.117:0/4146481807 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.498 7fa0377fe700 1 --2- 172.21.2.117:0/4146481807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 0x7fa0381b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049317061 0 0) 0x7fa0200009b0 con 0x7fa0381b3d20 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024002fe0 con 0x7fa0381b3d20 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498846976 0 0) 0x7fa030001400 con 0x7fa0380e5350 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024003210 con 0x7fa0380e5350 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0377fe700 1 -- 172.21.2.117:0/4146481807 >> v1:172.21.2.125:6789/0 conn(0x7fa0380e5350 legacy=0x7fa0380f6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0377fe700 1 -- 172.21.2.117:0/4146481807 >> v1:172.21.2.117:6789/0 conn(0x7fa0381b3d20 legacy=0x7fa0380f5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4191075596 0 0) 0x7fa020001010 con 0x7fa0381b3d20 2021-02-10T11:43:56.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0377fe700 1 -- 172.21.2.117:0/4146481807 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0380f4d20 con 0x7fa0380e4f50 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0377fe700 1 --2- 172.21.2.117:0/4146481807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 0x7fa0381b3700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa028006650 tx=0x7fa028009a80).ready entity=mon.2 client_cookie=cd8bf6c609c00a90 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa02800c0c0 con 0x7fa0380e4f50 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02800c340 con 0x7fa0380e4f50 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa0367fc700 1 -- 172.21.2.117:0/4146481807 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa02800c820 con 0x7fa0380e4f50 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa03da3f700 1 -- 172.21.2.117:0/4146481807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 msgr2=0x7fa0381b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.499 7fa03da3f700 1 --2- 172.21.2.117:0/4146481807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 0x7fa0381b3700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa028006650 tx=0x7fa028009a80).stop 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.500 7fa03da3f700 1 -- 172.21.2.117:0/4146481807 shutdown_connections 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.500 7fa03da3f700 1 --2- 172.21.2.117:0/4146481807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0380e4f50 0x7fa0381b3700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.500 7fa03da3f700 1 -- 172.21.2.117:0/4146481807 >> 172.21.2.117:0/4146481807 conn(0x7fa0380ce5b0 msgr2=0x7fa0380e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.500 7fa03da3f700 1 -- 172.21.2.117:0/4146481807 shutdown_connections 2021-02-10T11:43:56.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.500 7fa03da3f700 1 -- 172.21.2.117:0/4146481807 wait complete. 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.514 7fa03da3f700 1 Processor -- start 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 -- start start 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 0x7fa03819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 0x7fa0381ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 0x7fa0381ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381062f0 con 0x7fa03819feb0 2021-02-10T11:43:56.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381aed10 con 0x7fa038105ef0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa03da3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381aef30 con 0x7fa03819fab0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa037fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 0x7fa0381ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa037fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 0x7fa0381ae6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50510/0 (socket says 172.21.2.117:50510) 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa037fff700 1 -- 172.21.2.117:0/3622407032 learned_addr learned my addr 172.21.2.117:0/3622407032 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa036ffd700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 0x7fa0381ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.515 7fa0377fe700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 0x7fa03819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 -- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 msgr2=0x7fa03819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 0x7fa03819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 -- 172.21.2.117:0/3622407032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 msgr2=0x7fa0381ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 0x7fa0381ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 -- 172.21.2.117:0/3622407032 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0381acc20 con 0x7fa03819fab0 2021-02-10T11:43:56.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa0377fe700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 0x7fa03819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa036ffd700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 0x7fa0381ae0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa020006e80 tx=0x7fa02000b870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa02000bc60 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0200051e0 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0200056c0 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0381acc20 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.516 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381acc20 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.517 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa020006030 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.517 7fa02ffff700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 0x7fa01c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.517 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa020032430 con 0x7fa03819fab0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.518 7fa0377fe700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 0x7fa01c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:56.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.518 7fa0377fe700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 0x7fa01c01abc0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:56.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.548 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0381022b0 con 0x7fa03819fab0 2021-02-10T11:43:56.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.552 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa020030c10 con 0x7fa03819fab0 2021-02-10T11:43:56.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.677 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0381022b0 con 0x7fa03819fab0 2021-02-10T11:43:56.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.678 7fa02ffff700 1 -- 172.21.2.117:0/3622407032 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+14229 (secure 0 0 0) 0x7fa020006030 con 0x7fa03819fab0 2021-02-10T11:43:56.669 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:56.669 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":64,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:43:09.677005","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:43:06.649220","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":13,"pool_name":"cephfs_data","create_time":"2021-02-10 11:43:07.719391","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 msgr2=0x7fa01c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 0x7fa01c01abc0 crc :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 msgr2=0x7fa0381ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 0x7fa0381ae0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa020006e80 tx=0x7fa02000b870).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 shutdown_connections 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa01c0186c0 0x7fa01c01abc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa03819feb0 0x7fa0381ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa03819fab0 0x7fa0381ae0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.679 7fa03da3f700 1 --2- 172.21.2.117:0/3622407032 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa038105ef0 0x7fa03819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:56.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.680 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 >> 172.21.2.117:0/3622407032 conn(0x7fa03809b640 msgr2=0x7fa038106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:56.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.680 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 shutdown_connections 2021-02-10T11:43:56.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:56.680 7fa03da3f700 1 -- 172.21.2.117:0/3622407032 wait complete. 2021-02-10T11:43:56.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:43:06.649220', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:43:07.719391', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:43:56.680 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:43:57.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.103 7ff1cd0f9700 1 Processor -- start 2021-02-10T11:43:57.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.103 7ff1cd0f9700 1 -- start start 2021-02-10T11:43:57.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1cd0f9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff1c80e4f60 0x7ff1c81b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1cd0f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 0x7ff1c80f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1cd0f9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c80f53d0 con 0x7ff1c80e5360 2021-02-10T11:43:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1cd0f9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c80f6960 con 0x7ff1c80e4f60 2021-02-10T11:43:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1cd0f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c80f6b50 con 0x7ff1c81b3d30 2021-02-10T11:43:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1c6d9d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff1c80e4f60 0x7ff1c81b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1c759e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff1c80e5360 0x7ff1c80f4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50768/0 (socket says 172.21.2.117:50768) 2021-02-10T11:43:57.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1c759e700 1 -- 172.21.2.117:0/1436997664 learned_addr learned my addr 172.21.2.117:0/1436997664 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:57.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1c6d9d700 1 --2- 172.21.2.117:0/1436997664 >> v2:172.21.2.125:3300/0 conn(0x7ff1c80e4f60 0x7ff1c81b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54898/0 (socket says 172.21.2.117:54898) 2021-02-10T11:43:57.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.104 7ff1c659c700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 0x7ff1c80f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c5d9b700 1 -- 172.21.2.117:0/1436997664 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675350761 0 0) 0x7ff1c0001500 con 0x7ff1c80e5360 2021-02-10T11:43:57.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c5d9b700 1 -- 172.21.2.117:0/1436997664 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b4002fe0 con 0x7ff1c80e5360 2021-02-10T11:43:57.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c6d9d700 1 -- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 msgr2=0x7ff1c80f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:57.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c6d9d700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 0x7ff1c80f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c6d9d700 1 -- 172.21.2.117:0/1436997664 >> v1:172.21.2.117:6789/0 conn(0x7ff1c80e5360 legacy=0x7ff1c80f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c6d9d700 1 -- 172.21.2.117:0/1436997664 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80f6d40 con 0x7ff1c80e4f60 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c659c700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 0x7ff1c80f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c6d9d700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c80e4f60 0x7ff1c81b3710 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff1b0006650 tx=0x7ff1b0009a80).ready entity=mon.1 client_cookie=f48a76cf0ad611cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c5d9b700 1 -- 172.21.2.117:0/1436997664 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b00009b0 con 0x7ff1c80e4f60 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c5d9b700 1 -- 172.21.2.117:0/1436997664 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b000c0c0 con 0x7ff1c80e4f60 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1c5d9b700 1 -- 172.21.2.117:0/1436997664 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1b000c510 con 0x7ff1c80e4f60 2021-02-10T11:43:57.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1cd0f9700 1 -- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c80e4f60 msgr2=0x7ff1c81b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.105 7ff1cd0f9700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c80e4f60 0x7ff1c81b3710 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff1b0006650 tx=0x7ff1b0009a80).stop 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 -- 172.21.2.117:0/1436997664 shutdown_connections 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c81b3d30 0x7ff1c80f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 --2- 172.21.2.117:0/1436997664 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c80e4f60 0x7ff1c81b3710 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 -- 172.21.2.117:0/1436997664 >> 172.21.2.117:0/1436997664 conn(0x7ff1c80ce5c0 msgr2=0x7ff1c80e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 -- 172.21.2.117:0/1436997664 shutdown_connections 2021-02-10T11:43:57.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.106 7ff1cd0f9700 1 -- 172.21.2.117:0/1436997664 wait complete. 2021-02-10T11:43:57.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.120 7ff1cd0f9700 1 Processor -- start 2021-02-10T11:43:57.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.120 7ff1cd0f9700 1 -- start start 2021-02-10T11:43:57.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 0x7ff1c8092b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c8093160 0x7ff1c8102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 0x7ff1c8102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8191f90 con 0x7ff1c8102b60 2021-02-10T11:43:57.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c80b9aa0 con 0x7ff1c8093160 2021-02-10T11:43:57.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1cd0f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c81a4190 con 0x7ff1c80b9470 2021-02-10T11:43:57.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1c659c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c8093160 0x7ff1c8102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1c6d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 0x7ff1c8092b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1c6d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 0x7ff1c8092b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:48200/0 (socket says 172.21.2.117:48200) 2021-02-10T11:43:57.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1c759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 0x7ff1c8102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.121 7ff1c6d9d700 1 -- 172.21.2.117:0/286789849 learned_addr learned my addr 172.21.2.117:0/286789849 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:57.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 -- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 msgr2=0x7ff1c8092b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 0x7ff1c8092b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 -- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c8093160 msgr2=0x7ff1c8102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c8093160 0x7ff1c8102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 -- 172.21.2.117:0/286789849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c80b9fb0 con 0x7ff1c8102b60 2021-02-10T11:43:57.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1c759e700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 0x7ff1c8102f60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7ff1c0007f90 tx=0x7ff1c000abb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1c0007800 con 0x7ff1c8102b60 2021-02-10T11:43:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c80b9fb0 con 0x7ff1c8102b60 2021-02-10T11:43:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c80ba540 con 0x7ff1c8102b60 2021-02-10T11:43:57.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c0000f90 con 0x7ff1c8102b60 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.122 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1c0007030 con 0x7ff1c8102b60 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.123 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1c0007440 con 0x7ff1c8102b60 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.123 7ff1bf7fe700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 0x7ff1ac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.123 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff1c0032a10 con 0x7ff1c8102b60 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.123 7ff1c6d9d700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 0x7ff1ac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.123 7ff1c6d9d700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 0x7ff1ac01ac10 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.154 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c8102310 con 0x7ff1c8102b60 2021-02-10T11:43:57.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.156 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1c0007230 con 0x7ff1c8102b60 2021-02-10T11:43:57.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.283 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff1c80f53d0 con 0x7ff1c8102b60 2021-02-10T11:43:57.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.283 7ff1bf7fe700 1 -- 172.21.2.117:0/286789849 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v64) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff1c0007440 con 0x7ff1c8102b60 2021-02-10T11:43:57.275 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 msgr2=0x7ff1ac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 0x7ff1ac01ac10 crc :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 msgr2=0x7ff1c8102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 0x7ff1c8102f60 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7ff1c0007f90 tx=0x7ff1c000abb0).stop 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 shutdown_connections 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1ac018730 0x7ff1ac01ac10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c8102b60 0x7ff1c8102f60 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c8093160 0x7ff1c8102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 --2- 172.21.2.117:0/286789849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c80b9470 0x7ff1c8092b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 >> 172.21.2.117:0/286789849 conn(0x7ff1c8096b20 msgr2=0x7ff1c80c3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.285 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 shutdown_connections 2021-02-10T11:43:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.286 7ff1cd0f9700 1 -- 172.21.2.117:0/286789849 wait complete. 2021-02-10T11:43:57.285 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.709 7f7197ba4700 1 Processor -- start 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.709 7f7197ba4700 1 -- start start 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f7197ba4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900e5360 0x7f71900f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f7197ba4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71900f5950 0x7f71900f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f7197ba4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900f53d0 con 0x7f71900e4f60 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f7197ba4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71901b3f70 con 0x7f71900f5950 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f7197ba4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71900d16a0 con 0x7f71900e5360 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71900f5950 0x7f71900f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71900f5950 0x7f71900f5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50528/0 (socket says 172.21.2.117:50528) 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 -- 172.21.2.117:0/4252372000 learned_addr learned my addr 172.21.2.117:0/4252372000 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 -- 172.21.2.117:0/4252372000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900e5360 msgr2=0x7f71900f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 --2- 172.21.2.117:0/4252372000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900e5360 0x7f71900f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 -- 172.21.2.117:0/4252372000 >> v1:172.21.2.125:6789/0 conn(0x7f71900e4f60 legacy=0x7f71901b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 -- 172.21.2.117:0/4252372000 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900d18c0 con 0x7f71900f5950 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.710 7f71961c1700 1 --2- 172.21.2.117:0/4252372000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900f5950 0x7f71900f5d50 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f7188006f00 tx=0x7f7188009f10).ready entity=mon.0 client_cookie=184801abddada5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.711 7f71949be700 1 -- 172.21.2.117:0/4252372000 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718800b3b0 con 0x7f71900f5950 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.711 7f71949be700 1 -- 172.21.2.117:0/4252372000 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718800b630 con 0x7f71900f5950 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.711 7f71949be700 1 -- 172.21.2.117:0/4252372000 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718800bb50 con 0x7f71900f5950 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.711 7f7197ba4700 1 -- 172.21.2.117:0/4252372000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900f5950 msgr2=0x7f71900f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.711 7f7197ba4700 1 --2- 172.21.2.117:0/4252372000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900f5950 0x7f71900f5d50 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f7188006f00 tx=0x7f7188009f10).stop 2021-02-10T11:43:57.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 -- 172.21.2.117:0/4252372000 shutdown_connections 2021-02-10T11:43:57.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 --2- 172.21.2.117:0/4252372000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71900f5950 0x7f71900f5d50 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 --2- 172.21.2.117:0/4252372000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71900e5360 0x7f71900f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:43:57.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 -- 172.21.2.117:0/4252372000 >> 172.21.2.117:0/4252372000 conn(0x7f71900ce5c0 msgr2=0x7f71900e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 -- 172.21.2.117:0/4252372000 shutdown_connections 2021-02-10T11:43:57.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.712 7f7197ba4700 1 -- 172.21.2.117:0/4252372000 wait complete. 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.726 7f7197ba4700 1 Processor -- start 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.726 7f7197ba4700 1 -- start start 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 0x7f719008a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 0x7f71900c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719008b6e0 con 0x7f719008ad50 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71900bff10 con 0x7f719008a330 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f7197ba4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719019bdc0 con 0x7f719008b150 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71951bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71951bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50536/0 (socket says 172.21.2.117:50536) 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71951bf700 1 -- 172.21.2.117:0/1758913666 learned_addr learned my addr 172.21.2.117:0/1758913666 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71959c0700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 0x7f719008a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71961c1700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 0x7f71900c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.727 7f71951bf700 1 -- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 msgr2=0x7f71900c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71951bf700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 0x7f71900c25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71951bf700 1 -- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 msgr2=0x7f719008a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71951bf700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 0x7f719008a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71951bf700 1 -- 172.21.2.117:0/1758913666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719008d7f0 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71961c1700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 0x7f71900c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71959c0700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 0x7f719008a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71951bf700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f718c006f00 tx=0x7f718c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718c006350 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c00bdc0 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718c002020 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719008d7f0 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.728 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719008d7f0 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.729 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f718c007020 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.729 7f71827fc700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 0x7f717801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.729 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f718c032770 con 0x7f719008ad50 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.729 7f71959c0700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 0x7f717801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.730 7f71959c0700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 0x7f717801ac10 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.760 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71900f53d0 con 0x7f719008ad50 2021-02-10T11:43:57.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.762 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f718c006b90 con 0x7f719008ad50 2021-02-10T11:43:57.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.890 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f71900f53d0 con 0x7f719008ad50 2021-02-10T11:43:57.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.891 7f71827fc700 1 -- 172.21.2.117:0/1758913666 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v64) v1 ==== 104+0+10 (secure 0 0 0) 0x7f718c00bd30 con 0x7f719008ad50 2021-02-10T11:43:57.882 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:57.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 msgr2=0x7f717801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 0x7f717801ac10 crc :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 msgr2=0x7f71900c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7f718c006f00 tx=0x7f718c00b940).stop 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 shutdown_connections 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7178018730 0x7f717801ac10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f719008ad50 0x7f71900c1fa0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f719008b150 0x7f71900c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 --2- 172.21.2.117:0/1758913666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f719008a330 0x7f719008a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 >> 172.21.2.117:0/1758913666 conn(0x7f7190103170 msgr2=0x7f7190107170 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:57.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 shutdown_connections 2021-02-10T11:43:57.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:57.893 7f7197ba4700 1 -- 172.21.2.117:0/1758913666 wait complete. 2021-02-10T11:43:57.892 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:43:58.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.315 7fa5f36ab700 1 Processor -- start 2021-02-10T11:43:58.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.315 7fa5f36ab700 1 -- start start 2021-02-10T11:43:58.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.315 7fa5f36ab700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5ec0e5360 0x7fa5ec0f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f36ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 0x7fa5ec0f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f36ab700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec1b3f00 con 0x7fa5ec0e4f60 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f36ab700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec0f5450 con 0x7fa5ec0e5360 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f36ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0d16a0 con 0x7fa5ec0f5950 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f14c7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa5ec0e4f60 0x7fa5ec1b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50790/0 (socket says 172.21.2.117:50790) 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f14c7700 1 -- 172.21.2.117:0/4113104307 learned_addr learned my addr 172.21.2.117:0/4113104307 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f1cc8700 1 --2- 172.21.2.117:0/4113104307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 0x7fa5ec0f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5f0cc6700 1 --2- 172.21.2.117:0/4113104307 >> v2:172.21.2.125:3300/0 conn(0x7fa5ec0e5360 0x7fa5ec0f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5dffff700 1 -- 172.21.2.117:0/4113104307 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375221001 0 0) 0x7fa5e8001180 con 0x7fa5ec0e4f60 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.316 7fa5dffff700 1 -- 172.21.2.117:0/4113104307 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d8002fe0 con 0x7fa5ec0e4f60 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f1cc8700 1 -- 172.21.2.117:0/4113104307 >> v2:172.21.2.125:3300/0 conn(0x7fa5ec0e5360 msgr2=0x7fa5ec0f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f1cc8700 1 --2- 172.21.2.117:0/4113104307 >> v2:172.21.2.125:3300/0 conn(0x7fa5ec0e5360 0x7fa5ec0f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f1cc8700 1 -- 172.21.2.117:0/4113104307 >> v1:172.21.2.117:6789/0 conn(0x7fa5ec0e4f60 legacy=0x7fa5ec1b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f1cc8700 1 -- 172.21.2.117:0/4113104307 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0d1890 con 0x7fa5ec0f5950 2021-02-10T11:43:58.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f1cc8700 1 --2- 172.21.2.117:0/4113104307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 0x7fa5ec0f5d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa5e4006e50 tx=0x7fa5e4009e70).ready entity=mon.2 client_cookie=8e2da43a743ecc7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5dffff700 1 -- 172.21.2.117:0/4113104307 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e4006a20 con 0x7fa5ec0f5950 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5dffff700 1 -- 172.21.2.117:0/4113104307 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4006790 con 0x7fa5ec0f5950 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5dffff700 1 -- 172.21.2.117:0/4113104307 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e400b700 con 0x7fa5ec0f5950 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f36ab700 1 -- 172.21.2.117:0/4113104307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 msgr2=0x7fa5ec0f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.317 7fa5f36ab700 1 --2- 172.21.2.117:0/4113104307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 0x7fa5ec0f5d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa5e4006e50 tx=0x7fa5e4009e70).stop 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 -- 172.21.2.117:0/4113104307 shutdown_connections 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 --2- 172.21.2.117:0/4113104307 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0f5950 0x7fa5ec0f5d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 --2- 172.21.2.117:0/4113104307 >> v2:172.21.2.125:3300/0 conn(0x7fa5ec0e5360 0x7fa5ec0f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 -- 172.21.2.117:0/4113104307 >> 172.21.2.117:0/4113104307 conn(0x7fa5ec0ce5c0 msgr2=0x7fa5ec0e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 -- 172.21.2.117:0/4113104307 shutdown_connections 2021-02-10T11:43:58.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.318 7fa5f36ab700 1 -- 172.21.2.117:0/4113104307 wait complete. 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 Processor -- start 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 -- start start 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0b0d80 0x7fa5ec0aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5ec0aa9e0 0x7fa5ec0aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0b1180 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0b1f80 con 0x7fa5ec0aa9e0 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f36ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec0b21a0 con 0x7fa5ec0b0d80 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f14c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f14c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50548/0 (socket says 172.21.2.117:50548) 2021-02-10T11:43:58.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.333 7fa5f14c7700 1 -- 172.21.2.117:0/1719823699 learned_addr learned my addr 172.21.2.117:0/1719823699 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f0cc6700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0b0d80 0x7fa5ec0aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f1cc8700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5ec0aa9e0 0x7fa5ec0aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 -- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0b0d80 msgr2=0x7fa5ec0aa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0b0d80 0x7fa5ec0aa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 -- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5ec0aa9e0 msgr2=0x7fa5ec0aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5ec0aa9e0 0x7fa5ec0aade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 -- 172.21.2.117:0/1719823699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0ab400 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f14c7700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7fa5e800a5d0 tx=0x7fa5e800a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e8000ce0 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e80043a0 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec0ab400 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e80048c0 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.334 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec0ab400 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.335 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5e8000c90 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.335 7fa5ddffb700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 0x7fa5cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.335 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa5e8032750 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.336 7fa5f0cc6700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 0x7fa5cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.336 7fa5f0cc6700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 0x7fa5cc01abc0 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.366 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec0b0230 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.368 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5e801f4c0 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.493 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa5ec0b0230 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.493 7fa5ddffb700 1 -- 172.21.2.117:0/1719823699 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v64) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa5e8000c90 con 0x7fa5ec0b1b80 2021-02-10T11:43:58.485 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:43:58.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.495 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 msgr2=0x7fa5cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.495 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 0x7fa5cc01abc0 crc :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.495 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 msgr2=0x7fa5ec0b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.495 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7fa5e800a5d0 tx=0x7fa5e800a970).stop 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 shutdown_connections 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5ec0aa9e0 0x7fa5ec0aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5ec0b0d80 0x7fa5ec0aa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5cc018700 0x7fa5cc01abc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 --2- 172.21.2.117:0/1719823699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5ec0b1b80 0x7fa5ec0b0760 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 >> 172.21.2.117:0/1719823699 conn(0x7fa5ec18c240 msgr2=0x7fa5ec09e370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 shutdown_connections 2021-02-10T11:43:58.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.496 7fa5f36ab700 1 -- 172.21.2.117:0/1719823699 wait complete. 2021-02-10T11:43:58.495 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:43:58.495 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:43:58.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 Processor -- start 2021-02-10T11:43:58.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 -- start start 2021-02-10T11:43:58.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c0e5350 0x7fdb2c0f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c0f53c0 con 0x7fdb2c0f5940 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c1b3f60 con 0x7fdb2c0e4f50 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.916 7fdb33647700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0d1700 con 0x7fdb2c0e5350 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb31c64700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdb2c0f5940 0x7fdb2c0f5d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50804/0 (socket says 172.21.2.117:50804) 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb31c64700 1 -- 172.21.2.117:0/3312593210 learned_addr learned my addr 172.21.2.117:0/3312593210 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785872055 0 0) 0x7fdb24001500 con 0x7fdb2c0f5940 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb18002fe0 con 0x7fdb2c0f5940 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb30c62700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c0e5350 0x7fdb2c0f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2897620451 0 0) 0x7fdb24001270 con 0x7fdb2c0f5940 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c0e5350 msgr2=0x7fdb2c0f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c0e5350 0x7fdb2c0f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 >> v1:172.21.2.125:6789/0 conn(0x7fdb2c0e4f50 legacy=0x7fdb2c1b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c0d1920 con 0x7fdb2c0f5940 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fdb24001b50 con 0x7fdb2c0f5940 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 >> v1:172.21.2.117:6789/0 conn(0x7fdb2c0f5940 legacy=0x7fdb2c0f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb18003a60 0x7fdb18005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb18006790 0x7fdb18008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb18009500 0x7fdb1800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1800c2d0 con 0x7fdb18009500 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1800c4f0 con 0x7fdb18003a60 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.917 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1800c710 con 0x7fdb18006790 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb18009500 0x7fdb1800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31463700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb18003a60 0x7fdb18005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb30c62700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb18006790 0x7fdb18008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 -- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb18006790 msgr2=0x7fdb18008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb18006790 0x7fdb18008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 -- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb18003a60 msgr2=0x7fdb18005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb18003a60 0x7fdb18005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 -- 172.21.2.117:0/3312593210 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1800c9b0 con 0x7fdb18009500 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb18009500 0x7fdb1800ba20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fdb24002240 tx=0x7fdb2404e320).ready entity=mon.0 client_cookie=5d864e28c2d2f962 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb24001e80 con 0x7fdb18009500 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb240070d0 con 0x7fdb18009500 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb1ffff700 1 -- 172.21.2.117:0/3312593210 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb24007520 con 0x7fdb18009500 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb33647700 1 -- 172.21.2.117:0/3312593210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb18009500 msgr2=0x7fdb1800ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb33647700 1 --2- 172.21.2.117:0/3312593210 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb18009500 0x7fdb1800ba20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fdb24002240 tx=0x7fdb2404e320).stop 2021-02-10T11:43:58.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.918 7fdb31c64700 1 -- 172.21.2.117:0/3312593210 reap_dead start 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.919 7fdb33647700 1 -- 172.21.2.117:0/3312593210 shutdown_connections 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.919 7fdb33647700 1 -- 172.21.2.117:0/3312593210 >> 172.21.2.117:0/3312593210 conn(0x7fdb2c0ce5b0 msgr2=0x7fdb2c0e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.919 7fdb33647700 1 -- 172.21.2.117:0/3312593210 shutdown_connections 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.919 7fdb33647700 1 -- 172.21.2.117:0/3312593210 wait complete. 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.934 7fdb33647700 1 Processor -- start 2021-02-10T11:43:58.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.934 7fdb33647700 1 -- start start 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 0x7fdb2c188ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 0x7fdb2c189900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c188960 con 0x7fdb2c187db0 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c199cb0 con 0x7fdb2c189500 2021-02-10T11:43:58.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb33647700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c1885b0 con 0x7fdb2c1881b0 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb31463700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb31463700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50564/0 (socket says 172.21.2.117:50564) 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb31463700 1 -- 172.21.2.117:0/974725884 learned_addr learned my addr 172.21.2.117:0/974725884 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb31c64700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 0x7fdb2c189900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.935 7fdb30c62700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 0x7fdb2c188ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 -- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 msgr2=0x7fdb2c188ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 0x7fdb2c188ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 -- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 msgr2=0x7fdb2c189900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 0x7fdb2c189900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 -- 172.21.2.117:0/974725884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c19a170 con 0x7fdb2c187db0 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb30c62700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 0x7fdb2c188ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31c64700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 0x7fdb2c189900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:43:58.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb31463700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fdb2000b5e0 tx=0x7fdb2000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb20002030 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb200023b0 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb33647700 1 -- 172.21.2.117:0/974725884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c19a170 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb33647700 1 -- 172.21.2.117:0/974725884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c19a170 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.936 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdb200028d0 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.937 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdb20002590 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.937 7fdb1dffb700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 0x7fdb0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.937 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdb200326b0 con 0x7fdb2c187db0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.937 7fdb30c62700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 0x7fdb0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:43:58.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.937 7fdb30c62700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 0x7fdb0c01abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:43:58.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.966 7fdb33647700 1 -- 172.21.2.117:0/974725884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c102300 con 0x7fdb2c187db0 2021-02-10T11:43:58.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:58.969 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdb20030630 con 0x7fdb2c187db0 2021-02-10T11:43:59.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.098 7fdb33647700 1 -- 172.21.2.117:0/974725884 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb2c0f53c0 con 0x7fdb2c187db0 2021-02-10T11:43:59.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.098 7fdb1dffb700 1 -- 172.21.2.117:0/974725884 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fdb20030630 con 0x7fdb2c187db0 2021-02-10T11:43:59.090 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:43:59.090 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":45,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:43:59.090 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 45 2021-02-10T11:43:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 -- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 msgr2=0x7fdb0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 0x7fdb0c01abc0 crc :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 -- 172.21.2.117:0/974725884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 msgr2=0x7fdb2c190d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:43:59.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7fdb2000b5e0 tx=0x7fdb2000b980).stop 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 -- 172.21.2.117:0/974725884 shutdown_connections 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdb2c189500 0x7fdb2c189900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdb2c1881b0 0x7fdb2c188ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdb0c0186c0 0x7fdb0c01abc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 --2- 172.21.2.117:0/974725884 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdb2c187db0 0x7fdb2c190d30 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 -- 172.21.2.117:0/974725884 >> 172.21.2.117:0/974725884 conn(0x7fdb2c19f050 msgr2=0x7fdb2c1ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:43:59.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.100 7fdb33647700 1 -- 172.21.2.117:0/974725884 shutdown_connections 2021-02-10T11:43:59.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:43:59.101 7fdb33647700 1 -- 172.21.2.117:0/974725884 wait complete. 2021-02-10T11:43:59.301 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:43:08.674746', 'modified': '2021-02-10 11:43:10.684137', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6670': {'gid': 6670, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.117:6833/2942282551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.117:6832', 'nonce': 2942282551}, {'type': 'v1', 'addr': '172.21.2.117:6833', 'nonce': 2942282551}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:43:59.301 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:43:59.302 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2021-02-10T11:43:59.702 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:43:59.702 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:43:59.702 INFO:teuthology.orchestra.run.gibba017.stdout: "whoami": 0, 2021-02-10T11:43:59.702 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 6670, 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "want_state": "up:active", 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "up:active", 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "rank_uptime": 51.034356707000001, 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "mdsmap_epoch": 44, 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch": 64, 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "osdmap_epoch_barrier": 64, 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout: "uptime": 56.667990224 2021-02-10T11:43:59.703 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:43:59.706 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6670, "want_state": "up:active", "state": "up:active", "rank_uptime": 51.034356707000001, "mdsmap_epoch": 44, "osdmap_epoch": 64, "osdmap_epoch_barrier": 64, "uptime": 56.667990224 } 2021-02-10T11:43:59.707 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:43:59.707 INFO:tasks.cephfs.fuse_mount:Client client.alice config is {} 2021-02-10T11:43:59.707 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.alice at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:43:59.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:43:59.707 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:00.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:00.385 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:00.386 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:00.729 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:00.730 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:00.730 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:01.066 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:44:01.067 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:44:01.067 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --client_mountpoint=/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63 --name client.alice /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:01.069 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:01.412 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:01.412 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:01.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[138198]: starting ceph client 2021-02-10T11:44:01.479 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:2021-02-10 11:44:01.488 7f649fe52f80 -1 init, newargv = 0x55bf3e793240 newargc=9 2021-02-10T11:44:01.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[138198]: starting fuse 2021-02-10T11:44:01.755 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:01.756 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:01.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:01.757 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:02.091 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:44:02.091 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:44:02.092 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:44:02.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:02.092 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:44:02.092 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:44:02.092 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:44:02.092 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:44:02.092 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.alice.*.asok" 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:02.093 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:44:02.094 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:02.094 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:02.094 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.alice")) 2021-02-10T11:44:02.094 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:44:02.456 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.alice.138198.asok 2021-02-10T11:44:02.459 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.alice.138198.asok 2021-02-10T11:44:02.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:02.459 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.alice.138198.asok status 2021-02-10T11:44:02.848 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:02.848 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "alice", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "138198", 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/volumes/group_0000000000285035/subvolume_0000000000934866/febcf813-d515-440f-ad98-e79829b6fb63" 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 8781, 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:44:02.849 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 8781 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2856821612 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:02.850 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2856821612 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.8781 172.21.2.117:0/2856821612", 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2856821612", 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 44, 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 64, 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:44:02.851 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:02.853 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:03.198 INFO:teuthology.orchestra.run.gibba017.stdout:data.bin 2021-02-10T11:44:03.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:03.199 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:03.200 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:03.535 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:03.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:03.535 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo dd if=/dev/urandom of=rogue.bin bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:44:03.881 INFO:teuthology.orchestra.run.gibba017.stderr:dd: failed to open ‘rogue.bin’: Permission denied 2021-02-10T11:44:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:03.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:03.883 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:03.883 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:04.217 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:04.218 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:04.218 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:44:04.218 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:44:04.218 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:04.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.alice.gibba017.stderr:ceph-fuse[138198]: fuse finished with error 0 and tester_r 0 2021-02-10T11:44:04.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:04.575 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:04.575 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:04.910 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:44:04.911 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:44:04.911 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:44:04.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:04.911 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:05.245 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000934866 alice --group_name group_0000000000285035 2021-02-10T11:44:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.668 7efdd3c83700 1 Processor -- start 2021-02-10T11:44:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.668 7efdd3c83700 1 -- start start 2021-02-10T11:44:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efdcc0e4fb0 0x7efdcc1b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1b3d80 0x7efdcc0f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdcc0e53b0 0x7efdcc0f4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0f5420 con 0x7efdcc0e53b0 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0f49b0 con 0x7efdcc0e4fb0 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd3c83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0f4ba0 con 0x7efdcc1b3d80 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdcc0e53b0 0x7efdcc0f4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdcc0e53b0 0x7efdcc0f4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50586/0 (socket says 172.21.2.117:50586) 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 -- 172.21.2.117:0/2846263784 learned_addr learned my addr 172.21.2.117:0/2846263784 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd1a9f700 1 --2- 172.21.2.117:0/2846263784 >> v2:172.21.2.125:3300/0 conn(0x7efdcc0e4fb0 0x7efdcc1b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd129e700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1b3d80 0x7efdcc0f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 -- 172.21.2.117:0/2846263784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1b3d80 msgr2=0x7efdcc0f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1b3d80 0x7efdcc0f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 -- 172.21.2.117:0/2846263784 >> v2:172.21.2.125:3300/0 conn(0x7efdcc0e4fb0 msgr2=0x7efdcc1b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 --2- 172.21.2.117:0/2846263784 >> v2:172.21.2.125:3300/0 conn(0x7efdcc0e4fb0 0x7efdcc1b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 -- 172.21.2.117:0/2846263784 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0f4dc0 con 0x7efdcc0e53b0 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.669 7efdd22a0700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc0e53b0 0x7efdcc0f4390 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7efdc4006f00 tx=0x7efdc4009f10).ready entity=mon.0 client_cookie=b8fb5f3d9a68cb36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd0a9d700 1 -- 172.21.2.117:0/2846263784 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc400b3b0 con 0x7efdcc0e53b0 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd0a9d700 1 -- 172.21.2.117:0/2846263784 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc400b630 con 0x7efdcc0e53b0 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd0a9d700 1 -- 172.21.2.117:0/2846263784 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc400bb50 con 0x7efdcc0e53b0 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 -- 172.21.2.117:0/2846263784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc0e53b0 msgr2=0x7efdcc0f4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc0e53b0 0x7efdcc0f4390 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7efdc4006f00 tx=0x7efdc4009f10).stop 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 -- 172.21.2.117:0/2846263784 shutdown_connections 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc0e53b0 0x7efdcc0f4390 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 --2- 172.21.2.117:0/2846263784 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1b3d80 0x7efdcc0f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 --2- 172.21.2.117:0/2846263784 >> v2:172.21.2.125:3300/0 conn(0x7efdcc0e4fb0 0x7efdcc1b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.670 7efdd3c83700 1 -- 172.21.2.117:0/2846263784 >> 172.21.2.117:0/2846263784 conn(0x7efdcc0ce610 msgr2=0x7efdcc0e4450 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.671 7efdd3c83700 1 -- 172.21.2.117:0/2846263784 shutdown_connections 2021-02-10T11:44:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.671 7efdd3c83700 1 -- 172.21.2.117:0/2846263784 wait complete. 2021-02-10T11:44:05.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.685 7efdd3c83700 1 Processor -- start 2021-02-10T11:44:05.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.685 7efdd3c83700 1 -- start start 2021-02-10T11:44:05.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc187e10 0x7efdcc0fe720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1013c0 0x7efdcc0afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc188210 con 0x7efdcc187e10 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b02a0 con 0x7efdcc100fc0 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd3c83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b04c0 con 0x7efdcc1013c0 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:54972/0 (socket says 172.21.2.117:54972) 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 -- 172.21.2.117:0/1480144108 learned_addr learned my addr 172.21.2.117:0/1480144108 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd1a9f700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc187e10 0x7efdcc0fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd22a0700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1013c0 0x7efdcc0afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 -- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1013c0 msgr2=0x7efdcc0afc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1013c0 0x7efdcc0afc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 -- 172.21.2.117:0/1480144108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc187e10 msgr2=0x7efdcc0fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc187e10 0x7efdcc0fe720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 -- 172.21.2.117:0/1480144108 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc188b20 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.686 7efdd129e700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7efdc8006f00 tx=0x7efdc800b940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc800bd30 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8000c90 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdc80022b0 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc188b20 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc188b20 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efdc8002020 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 0x7efdb401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.687 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efdc8032b00 con 0x7efdcc100fc0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.688 7efdd1a9f700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 0x7efdb401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:05.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.688 7efdd1a9f700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 0x7efdb401ac10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:05.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.718 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc080750 con 0x7efdcc100fc0 2021-02-10T11:44:05.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.722 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efdc8000990 con 0x7efdcc100fc0 2021-02-10T11:44:05.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.850 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000285035"}) v1 -- 0x7efdcc0f5420 con 0x7efdb4018730 2021-02-10T11:44:05.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.887 7efdbe7fc700 1 -- 172.21.2.117:0/1480144108 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efdc00009b0 con 0x7efdb4018730 2021-02-10T11:44:05.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 msgr2=0x7efdb401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 0x7efdb401ac10 crc :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 msgr2=0x7efdcc0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:05.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7efdc8006f00 tx=0x7efdc800b940).stop 2021-02-10T11:44:05.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 shutdown_connections 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdcc1013c0 0x7efdcc0afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdcc100fc0 0x7efdcc0af660 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efdb4018730 0x7efdb401ac10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 --2- 172.21.2.117:0/1480144108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdcc187e10 0x7efdcc0fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 >> 172.21.2.117:0/1480144108 conn(0x7efdcc1aec10 msgr2=0x7efdcc107050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 shutdown_connections 2021-02-10T11:44:05.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:05.915 7efdd3c83700 1 -- 172.21.2.117:0/1480144108 wait complete. 2021-02-10T11:44:05.915 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 Processor -- start 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 -- start start 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac0e5350 0x7fccac0f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccac0f5940 0x7fccac0f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0f53c0 con 0x7fccac0e4f50 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccac1b3f60 con 0x7fccac0f5940 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.333 7fccb1791700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0d1690 con 0x7fccac0e5350 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccab574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fccac0e4f50 0x7fccac1b3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50846/0 (socket says 172.21.2.117:50846) 2021-02-10T11:44:06.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccab574700 1 -- 172.21.2.117:0/3956627445 learned_addr learned my addr 172.21.2.117:0/3956627445 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 --2- 172.21.2.117:0/3956627445 >> v2:172.21.2.117:3300/0 conn(0x7fccac0f5940 0x7fccac0f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccaad73700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac0e5350 0x7fccac0f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 -- 172.21.2.117:0/3956627445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac0e5350 msgr2=0x7fccac0f4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac0e5350 0x7fccac0f4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccaa572700 1 -- 172.21.2.117:0/3956627445 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058252384 0 0) 0x7fcc94001430 con 0x7fccac0e4f50 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 -- 172.21.2.117:0/3956627445 >> v1:172.21.2.117:6789/0 conn(0x7fccac0e4f50 legacy=0x7fccac1b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 -- 172.21.2.117:0/3956627445 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0d18b0 con 0x7fccac0f5940 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.334 7fccabd75700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0f5940 0x7fccac0f5d40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fcca4006ab0 tx=0x7fcca4009b40).ready entity=mon.0 client_cookie=4e1440046e9e257d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccaa572700 1 -- 172.21.2.117:0/3956627445 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca40022e0 con 0x7fccac0f5940 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccaa572700 1 -- 172.21.2.117:0/3956627445 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca4002560 con 0x7fccac0f5940 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccaa572700 1 -- 172.21.2.117:0/3956627445 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcca40029f0 con 0x7fccac0f5940 2021-02-10T11:44:06.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 -- 172.21.2.117:0/3956627445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0f5940 msgr2=0x7fccac0f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0f5940 0x7fccac0f5d40 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7fcca4006ab0 tx=0x7fcca4009b40).stop 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 -- 172.21.2.117:0/3956627445 shutdown_connections 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0f5940 0x7fccac0f5d40 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 --2- 172.21.2.117:0/3956627445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac0e5350 0x7fccac0f4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 -- 172.21.2.117:0/3956627445 >> 172.21.2.117:0/3956627445 conn(0x7fccac0ce5b0 msgr2=0x7fccac0e43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 -- 172.21.2.117:0/3956627445 shutdown_connections 2021-02-10T11:44:06.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.335 7fccb1791700 1 -- 172.21.2.117:0/3956627445 wait complete. 2021-02-10T11:44:06.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 Processor -- start 2021-02-10T11:44:06.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 -- start start 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccac192730 0x7fccac192b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac08c480 0x7fccac08c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac193150 con 0x7fccac0b4230 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac08cea0 con 0x7fccac192730 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.350 7fccb1791700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac08d0c0 con 0x7fccac08c480 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50602/0 (socket says 172.21.2.117:50602) 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 -- 172.21.2.117:0/682195949 learned_addr learned my addr 172.21.2.117:0/682195949 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 -- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac08c480 msgr2=0x7fccac08c880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:06.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac08c480 0x7fccac08c880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 -- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccac192730 msgr2=0x7fccac192b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccac192730 0x7fccac192b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 -- 172.21.2.117:0/682195949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac08d2e0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fccab574700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7fcc9400a8f0 tx=0x7fcc9400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc94000d30 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc940070d0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.351 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc940046d0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.352 7fccb1791700 1 -- 172.21.2.117:0/682195949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac08d2e0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.352 7fccb1791700 1 -- 172.21.2.117:0/682195949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac193d60 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.352 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc940044c0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.352 7fcca3fff700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 0x7fcc9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.352 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcc940328a0 con 0x7fccac0b4230 2021-02-10T11:44:06.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.353 7fccaad73700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 0x7fcc9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.353 7fccaad73700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 0x7fcc9001abc0 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.383 7fccb1791700 1 -- 172.21.2.117:0/682195949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac0f53c0 con 0x7fccac0b4230 2021-02-10T11:44:06.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.386 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc940072b0 con 0x7fccac0b4230 2021-02-10T11:44:06.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.516 7fccb1791700 1 -- 172.21.2.117:0/682195949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fccac0f53c0 con 0x7fccac0b4230 2021-02-10T11:44:06.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.517 7fcca3fff700 1 -- 172.21.2.117:0/682195949 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fcc940044c0 con 0x7fccac0b4230 2021-02-10T11:44:06.509 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.510 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.511 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.512 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.513 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:44:06.514 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.515 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.516 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.517 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.518 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:06.519 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.520 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:06.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:06.522 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:06.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 msgr2=0x7fcc9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 0x7fcc9001abc0 crc :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 msgr2=0x7fccac0b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7fcc9400a8f0 tx=0x7fcc9400ac90).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 shutdown_connections 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccac08c480 0x7fccac08c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccac192730 0x7fccac192b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc900186c0 0x7fcc9001abc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 --2- 172.21.2.117:0/682195949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccac0b4230 0x7fccac0b4630 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 >> 172.21.2.117:0/682195949 conn(0x7fccac103f10 msgr2=0x7fccac190bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 shutdown_connections 2021-02-10T11:44:06.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.519 7fccb1791700 1 -- 172.21.2.117:0/682195949 wait complete. 2021-02-10T11:44:06.525 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000934866 --group_name group_0000000000285035 2021-02-10T11:44:06.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.948 7effd96ae700 1 Processor -- start 2021-02-10T11:44:06.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.948 7effd96ae700 1 -- start start 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd96ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd40e4f70 0x7effd41b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd96ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effd41b3d40 0x7effd40f5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd96ae700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd40f53e0 con 0x7effd40e5370 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd96ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd40e5da0 con 0x7effd41b3d40 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd96ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40f4d40 con 0x7effd40e4f70 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effd41b3d40 0x7effd40f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effd41b3d40 0x7effd40f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50610/0 (socket says 172.21.2.117:50610) 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 -- 172.21.2.117:0/427490575 learned_addr learned my addr 172.21.2.117:0/427490575 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 -- 172.21.2.117:0/427490575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd40e4f70 msgr2=0x7effd41b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:06.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 --2- 172.21.2.117:0/427490575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd40e4f70 0x7effd41b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 -- 172.21.2.117:0/427490575 >> v1:172.21.2.125:6789/0 conn(0x7effd40e5370 legacy=0x7effd40f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 -- 172.21.2.117:0/427490575 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40f4f30 con 0x7effd41b3d40 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.949 7effd2abe700 1 --2- 172.21.2.117:0/427490575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd41b3d40 0x7effd40f5960 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7effbc006700 tx=0x7effbc009b80).ready entity=mon.0 client_cookie=5ac2a92458d3f5bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd22bd700 1 -- 172.21.2.117:0/427490575 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effbc00c0c0 con 0x7effd41b3d40 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd22bd700 1 -- 172.21.2.117:0/427490575 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc00c340 con 0x7effd41b3d40 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd22bd700 1 -- 172.21.2.117:0/427490575 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effbc00c860 con 0x7effd41b3d40 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 -- 172.21.2.117:0/427490575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd41b3d40 msgr2=0x7effd40f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 --2- 172.21.2.117:0/427490575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd41b3d40 0x7effd40f5960 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7effbc006700 tx=0x7effbc009b80).stop 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 -- 172.21.2.117:0/427490575 shutdown_connections 2021-02-10T11:44:06.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 --2- 172.21.2.117:0/427490575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd41b3d40 0x7effd40f5960 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:06.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 --2- 172.21.2.117:0/427490575 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd40e4f70 0x7effd41b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 -- 172.21.2.117:0/427490575 >> 172.21.2.117:0/427490575 conn(0x7effd40ce5d0 msgr2=0x7effd40e4410 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:06.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 -- 172.21.2.117:0/427490575 shutdown_connections 2021-02-10T11:44:06.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.950 7effd96ae700 1 -- 172.21.2.117:0/427490575 wait complete. 2021-02-10T11:44:06.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 Processor -- start 2021-02-10T11:44:06.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 -- start start 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effd419bc40 0x7effd418f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd418fea0 0x7effd418e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd408a2d0 con 0x7effd418ee10 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd41902a0 con 0x7effd419bc40 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd96ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd408ab20 con 0x7effd418fea0 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd3ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd3ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50620/0 (socket says 172.21.2.117:50620) 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.965 7effd3ac0700 1 -- 172.21.2.117:0/1842040866 learned_addr learned my addr 172.21.2.117:0/1842040866 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 -- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd418fea0 msgr2=0x7effd418e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd418fea0 0x7effd418e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 -- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effd419bc40 msgr2=0x7effd418f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:06.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effd419bc40 0x7effd418f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 -- 172.21.2.117:0/1842040866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd408ad40 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd3ac0700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7effcc009940 tx=0x7effcc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effcc000f90 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effcc000d30 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effcc00c570 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd96ae700 1 -- 172.21.2.117:0/1842040866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd408ad40 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.966 7effd96ae700 1 -- 172.21.2.117:0/1842040866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd408b2d0 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.967 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7effcc001220 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.967 7effcbfff700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 0x7effb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.967 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7effcc032a40 con 0x7effd418ee10 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.967 7effd32bf700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 0x7effb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:06.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.968 7effd32bf700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 0x7effb801abc0 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:06.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:06.999 7effd96ae700 1 -- 172.21.2.117:0/1842040866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd40f53e0 con 0x7effd418ee10 2021-02-10T11:44:06.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.001 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7effcc000f40 con 0x7effd418ee10 2021-02-10T11:44:07.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.126 7effd96ae700 1 -- 172.21.2.117:0/1842040866 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000934866", "target": ["mgr", ""], "group_name": "group_0000000000285035"}) v1 -- 0x7effd408b2d0 con 0x7effb80186c0 2021-02-10T11:44:07.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.135 7effcbfff700 1 -- 172.21.2.117:0/1842040866 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7effc40009b0 con 0x7effb80186c0 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.143 7effd96ae700 1 -- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 msgr2=0x7effb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.143 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 0x7effb801abc0 crc :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.143 7effd96ae700 1 -- 172.21.2.117:0/1842040866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 msgr2=0x7effd418f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.143 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7effcc009940 tx=0x7effcc00bcf0).stop 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 -- 172.21.2.117:0/1842040866 shutdown_connections 2021-02-10T11:44:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7effb80186c0 0x7effb801abc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effd418ee10 0x7effd418f210 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effd418fea0 0x7effd418e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 --2- 172.21.2.117:0/1842040866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effd419bc40 0x7effd418f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 -- 172.21.2.117:0/1842040866 >> 172.21.2.117:0/1842040866 conn(0x7effd41ad5a0 msgr2=0x7effd4106340 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 -- 172.21.2.117:0/1842040866 shutdown_connections 2021-02-10T11:44:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.144 7effd96ae700 1 -- 172.21.2.117:0/1842040866 wait complete. 2021-02-10T11:44:07.145 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000285035 2021-02-10T11:44:07.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.566 7f311d96a700 1 Processor -- start 2021-02-10T11:44:07.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f311d96a700 1 -- start start 2021-02-10T11:44:07.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f311d96a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 0x7f31180f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:07.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f311d96a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31180f53d0 con 0x7f31180f5950 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f311d96a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31181b3f70 con 0x7f31180e4f60 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f311d96a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31180d1710 con 0x7f31180e5360 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f3117d75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f31180f5950 0x7f31180f5d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50878/0 (socket says 172.21.2.117:50878) 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f3117d75700 1 -- 172.21.2.117:0/1417301733 learned_addr learned my addr 172.21.2.117:0/1417301733 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.567 7f3116d73700 1 --2- 172.21.2.117:0/1417301733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 0x7f31180f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794383814 0 0) 0x7f3110000f90 con 0x7f31180f5950 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104002fe0 con 0x7f31180f5950 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510131047 0 0) 0x7f3108000c60 con 0x7f31180e4f60 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104003210 con 0x7f31180e4f60 2021-02-10T11:44:07.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116d73700 1 -- 172.21.2.117:0/1417301733 >> v1:172.21.2.125:6789/0 conn(0x7f31180e4f60 legacy=0x7f31181b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116d73700 1 -- 172.21.2.117:0/1417301733 >> v1:172.21.2.117:6789/0 conn(0x7f31180f5950 legacy=0x7f31180f5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116d73700 1 -- 172.21.2.117:0/1417301733 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180d1930 con 0x7f31180e5360 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116d73700 1 --2- 172.21.2.117:0/1417301733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 0x7f31180f4d30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3100006650 tx=0x7f3100009a00).ready entity=mon.2 client_cookie=fe10cb1a88cd5d8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31000009b0 con 0x7f31180e5360 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310000c0c0 con 0x7f31180e5360 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f3116572700 1 -- 172.21.2.117:0/1417301733 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f310000c5a0 con 0x7f31180e5360 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f311d96a700 1 -- 172.21.2.117:0/1417301733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 msgr2=0x7f31180f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.568 7f311d96a700 1 --2- 172.21.2.117:0/1417301733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 0x7f31180f4d30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3100006650 tx=0x7f3100009a00).stop 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.569 7f311d96a700 1 -- 172.21.2.117:0/1417301733 shutdown_connections 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.569 7f311d96a700 1 --2- 172.21.2.117:0/1417301733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31180e5360 0x7f31180f4d30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.569 7f311d96a700 1 -- 172.21.2.117:0/1417301733 >> 172.21.2.117:0/1417301733 conn(0x7f31180ce5c0 msgr2=0x7f31180e4400 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:07.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.569 7f311d96a700 1 -- 172.21.2.117:0/1417301733 shutdown_connections 2021-02-10T11:44:07.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.569 7f311d96a700 1 -- 172.21.2.117:0/1417301733 wait complete. 2021-02-10T11:44:07.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 Processor -- start 2021-02-10T11:44:07.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 -- start start 2021-02-10T11:44:07.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f311819cde0 0x7f311819d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:07.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 0x7f31180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180c35f0 con 0x7f31180c2b50 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31180a44c0 con 0x7f31180a4d70 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f311d96a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31180c38b0 con 0x7f311819cde0 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f3116d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f3116d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50630/0 (socket says 172.21.2.117:50630) 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.584 7f3116d73700 1 -- 172.21.2.117:0/1727340064 learned_addr learned my addr 172.21.2.117:0/1727340064 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 -- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f311819cde0 msgr2=0x7f311819d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f311819cde0 0x7f311819d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 -- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 msgr2=0x7f31180a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3117d75700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 0x7f31180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 0x7f31180a5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 -- 172.21.2.117:0/1727340064 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180c3da0 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3117d75700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 0x7f31180a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f3116d73700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f310000b640 tx=0x7f310000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f310000bdd0 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100002030 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31000024d0 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f311d96a700 1 -- 172.21.2.117:0/1727340064 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31180c3da0 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.585 7f311d96a700 1 -- 172.21.2.117:0/1727340064 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31180a5a80 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.586 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3100006a10 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.586 7f310ffff700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 0x7f30fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.586 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f31000327c0 con 0x7f31180c2b50 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.586 7f3117574700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 0x7f30fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.586 7f3117574700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 0x7f30fc01abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:07.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.617 7f311d96a700 1 -- 172.21.2.117:0/1727340064 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31180f53d0 con 0x7f31180c2b50 2021-02-10T11:44:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.619 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f310001f530 con 0x7f31180c2b50 2021-02-10T11:44:07.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.750 7f311d96a700 1 -- 172.21.2.117:0/1727340064 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000285035"}) v1 -- 0x7f31180a5a80 con 0x7f30fc0186c0 2021-02-10T11:44:07.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.754 7f310ffff700 1 -- 172.21.2.117:0/1727340064 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3108006e50 con 0x7f30fc0186c0 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.758 7f311d96a700 1 -- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 msgr2=0x7f30fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.758 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 0x7f30fc01abc0 crc :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.758 7f311d96a700 1 -- 172.21.2.117:0/1727340064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 msgr2=0x7f31180c2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.758 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 rx=0x7f310000b640 tx=0x7f310000b9e0).stop 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 -- 172.21.2.117:0/1727340064 shutdown_connections 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30fc0186c0 0x7f30fc01abc0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31180c2b50 0x7f31180c2f50 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31180a4d70 0x7f31180a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:07.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 --2- 172.21.2.117:0/1727340064 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f311819cde0 0x7f311819d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:07.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 -- 172.21.2.117:0/1727340064 >> 172.21.2.117:0/1727340064 conn(0x7f31180a1e80 msgr2=0x7f31180b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:07.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 -- 172.21.2.117:0/1727340064 shutdown_connections 2021-02-10T11:44:07.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:07.759 7f311d96a700 1 -- 172.21.2.117:0/1727340064 wait complete. 2021-02-10T11:44:07.759 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:44:07.759 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:08.103 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:08.443 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:08.447 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:08.448 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:44:08.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:08.448 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:08.449 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:08.779 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:08.779 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:08.779 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:44:08.780 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:44:08.780 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:09.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[135245]: fuse finished with error 0 and tester_r 0 2021-02-10T11:44:09.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:09.138 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:09.138 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:09.473 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:44:09.473 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:44:09.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:09.474 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:09.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:09.809 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:09.809 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:10.143 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:44:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:10.143 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:10.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:10.143 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:10.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:10.479 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:10.479 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:10.814 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:44:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:10.814 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:44:10.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:10.815 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:11.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:11.149 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:11.149 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:11.483 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:44:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:11.484 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:44:11.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:44:11.484 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:11.818 DEBUG:teuthology.orchestra.run.gibba017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:44:11.821 DEBUG:teuthology.orchestra.run.gibba025:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout:[global] 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout: chdir = "" 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout: auth supported = cephx 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.183 INFO:teuthology.orchestra.run.gibba025.stdout: filestore xattr use omap = true 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: mon clock drift allowed = 1.000 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: osd crush chooseleaf type = 0 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: auth debug = true 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on old message = true 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: ms die on bug = true 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: mon pg warn max object skew = 0 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.184 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default size = 2 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow primary affinity = true 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd allow pg remap = true 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on no sortbitwise = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon warn on too few osds = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.185 INFO:teuthology.orchestra.run.gibba025.stdout: osd default data pool replay window = 5 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: mon allow pool delete = true 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: mon cluster log file level = debug 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: debug asserts on shutdown = true 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: mon health detail to clog = false 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:44:12.186 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout:[osd] 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd journal size = 100 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub load threshold = 5.0 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd scrub max interval = 600 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd recover clone overlap = true 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd recovery max chunk = 1048576 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug shutdown = true 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug op order = true 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug verify stray on activate = true 2021-02-10T11:44:12.187 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: osd open classes on start = true 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug pg log writeout = true 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: osd map max advance = 10 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: journal zero on create = true 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: filestore apply finisher threads = 3 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.188 INFO:teuthology.orchestra.run.gibba025.stdout: bdev debug aio = true 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: osd debug misdirected ops = true 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore block size = 96636764160 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: bluestore fsck on mount = True 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluefs = 20 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug bluestore = 20 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug filestore = 20 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug journal = 20 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 20 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug osd = 25 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: debug rocksdb = 10 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd full ratio = 0.9 2021-02-10T11:44:12.189 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: osd objectstore = bluestore 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout:[mgr] 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: debug mgr = 20 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:44:12.190 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 10 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout:[mon] 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: debug mon = 20 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: debug paxos = 20 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: debug auth = 20 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon data avail warn = 5 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon mgr mkfs grace = 240 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd reporter subtree level = osd 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon osd prime pg temp = true 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:44:12.191 INFO:teuthology.orchestra.run.gibba025.stdout: mon op complaint time = 120 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout:[client] 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: rgw cache enabled = true 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable ops log = true 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: rgw enable usage log = true 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: client mount timeout = 600 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: debug client = 20 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout: pid file = "" 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.a] 2021-02-10T11:44:12.192 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.b] 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout:[mon.c] 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.a] 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout:[mds.b] 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout:[mds] 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: debug mds = 20 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: debug ms = 1 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal frag = True 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal fragment size max = 10000 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal merge size = 5 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split bits = 3 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds bal split size = 100 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug frag = True 2021-02-10T11:44:12.193 INFO:teuthology.orchestra.run.gibba025.stdout: mds debug scatterstat = True 2021-02-10T11:44:12.194 INFO:teuthology.orchestra.run.gibba025.stdout: mds op complaint time = 180 2021-02-10T11:44:12.194 INFO:teuthology.orchestra.run.gibba025.stdout: mds verify scatter = True 2021-02-10T11:44:12.194 INFO:teuthology.orchestra.run.gibba025.stdout: osd op complaint time = 180 2021-02-10T11:44:12.194 INFO:teuthology.orchestra.run.gibba025.stdout:[client.alice] 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout:[global] 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout: chdir = "" 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout: auth supported = cephx 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.195 INFO:teuthology.orchestra.run.gibba017.stdout: filestore xattr use omap = true 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: mon clock drift allowed = 1.000 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: osd crush chooseleaf type = 0 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: auth debug = true 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on old message = true 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: ms die on bug = true 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: mon pg warn max object skew = 0 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.196 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default size = 2 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow primary affinity = true 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd allow pg remap = true 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on legacy crush tunables = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on crush straw calc version zero = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on no sortbitwise = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on osd down out interval zero = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon warn on too few osds = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-10T11:44:12.197 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: osd default data pool replay window = 5 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: mon allow pool delete = true 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: mon cluster log file level = debug 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: debug asserts on shutdown = true 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: mon health detail to clog = false 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: # we see this fail in qa on *nautilus*; bump up retries 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: mon_client_directed_command_retry = 4 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: mon host = "172.21.2.117,172.21.2.125,[v2:172.21.2.125:3301,v1:172.21.2.125:6790]" 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: fsid = 722ef60b-feda-4d6a-867f-a5d7f666055b 2021-02-10T11:44:12.198 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout:[osd] 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd journal size = 100 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub load threshold = 5.0 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd scrub max interval = 600 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd recover clone overlap = true 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd recovery max chunk = 1048576 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug shutdown = true 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug op order = true 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug verify stray on activate = true 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.199 INFO:teuthology.orchestra.run.gibba017.stdout: osd open classes on start = true 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug pg log writeout = true 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: osd deep scrub update digest min age = 30 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: osd map max advance = 10 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: journal zero on create = true 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: filestore ondisk finisher threads = 3 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: filestore apply finisher threads = 3 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: bdev debug aio = true 2021-02-10T11:44:12.200 INFO:teuthology.orchestra.run.gibba017.stdout: osd debug misdirected ops = true 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore block size = 96636764160 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: bluestore fsck on mount = True 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluefs = 20 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug bluestore = 20 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug filestore = 20 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug journal = 20 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 20 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug osd = 25 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: debug rocksdb = 10 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd backfillfull_ratio = 0.85 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd full ratio = 0.9 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd nearfull ratio = 0.8 2021-02-10T11:44:12.201 INFO:teuthology.orchestra.run.gibba017.stdout: osd failsafe full ratio = 0.95 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: osd objectstore = bluestore 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout:[mgr] 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: debug mgr = 20 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 10 2021-02-10T11:44:12.202 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout:[mon] 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: debug mon = 20 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: debug paxos = 20 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: debug auth = 20 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon data avail warn = 5 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon mgr mkfs grace = 240 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min pgs per osd = 4 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd reporter subtree level = osd 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon osd prime pg temp = true 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon reweight min bytes per osd = 10 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: mon op complaint time = 120 2021-02-10T11:44:12.203 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout:[client] 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: rgw cache enabled = true 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable ops log = true 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: rgw enable usage log = true 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: client mount timeout = 600 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: debug client = 20 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout: pid file = "" 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.a] 2021-02-10T11:44:12.204 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.b] 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout:[mon.c] 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.a] 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout:[mds.b] 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout:[mds] 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout: debug mds = 20 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout: debug ms = 1 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal frag = True 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal fragment size max = 10000 2021-02-10T11:44:12.205 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal merge size = 5 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split bits = 3 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds bal split size = 100 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug frag = True 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds debug scatterstat = True 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds op complaint time = 180 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: mds verify scatter = True 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout: osd op complaint time = 180 2021-02-10T11:44:12.206 INFO:teuthology.orchestra.run.gibba017.stdout:[client.alice] 2021-02-10T11:44:12.207 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 Processor -- start 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 -- start start 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41b07f0 0x7f1ac40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1ac40e7d70 0x7f1ac40e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40e77f0 con 0x7f1ac40e8790 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40e92f0 con 0x7f1ac40e7d70 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.629 7f1ac92f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40e5ce0 con 0x7f1ac41b07f0 2021-02-10T11:44:12.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1ac40e7d70 0x7f1ac40e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1ac40e7d70 0x7f1ac40e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50640/0 (socket says 172.21.2.117:50640) 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 -- 172.21.2.117:0/106795047 learned_addr learned my addr 172.21.2.117:0/106795047 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac307d700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41b07f0 0x7f1ac40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 -- 172.21.2.117:0/106795047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41b07f0 msgr2=0x7f1ac40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41b07f0 0x7f1ac40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 -- 172.21.2.117:0/106795047 >> v1:172.21.2.125:6789/0 conn(0x7f1ac40e8790 legacy=0x7f1ac40e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 -- 172.21.2.117:0/106795047 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40e54b0 con 0x7f1ac40e7d70 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac287c700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac40e7d70 0x7f1ac40e8170 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f1ab8006700 tx=0x7f1ab8009b80).ready entity=mon.0 client_cookie=9b4117ecb0119c4e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.630 7f1ac207b700 1 -- 172.21.2.117:0/106795047 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab800c0c0 con 0x7f1ac40e7d70 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac207b700 1 -- 172.21.2.117:0/106795047 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab800c440 con 0x7f1ac40e7d70 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac207b700 1 -- 172.21.2.117:0/106795047 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab800c960 con 0x7f1ac40e7d70 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 -- 172.21.2.117:0/106795047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac40e7d70 msgr2=0x7f1ac40e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac40e7d70 0x7f1ac40e8170 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f1ab8006700 tx=0x7f1ab8009b80).stop 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 -- 172.21.2.117:0/106795047 shutdown_connections 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac40e7d70 0x7f1ac40e8170 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 --2- 172.21.2.117:0/106795047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41b07f0 0x7f1ac40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 -- 172.21.2.117:0/106795047 >> 172.21.2.117:0/106795047 conn(0x7f1ac40ce500 msgr2=0x7f1ac40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 -- 172.21.2.117:0/106795047 shutdown_connections 2021-02-10T11:44:12.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.631 7f1ac92f9700 1 -- 172.21.2.117:0/106795047 wait complete. 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 Processor -- start 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 -- start start 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 0x7f1ac40adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41aded0 0x7f1ac41ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4093f70 con 0x7f1ac41ad4b0 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4197350 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.646 7f1ac92f9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4197570 con 0x7f1ac41aded0 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50646/0 (socket says 172.21.2.117:50646) 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac287c700 1 -- 172.21.2.117:0/2307255425 learned_addr learned my addr 172.21.2.117:0/2307255425 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 0x7f1ac40adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac387e700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41aded0 0x7f1ac41ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 -- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41aded0 msgr2=0x7f1ac41ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41aded0 0x7f1ac41ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 -- 172.21.2.117:0/2307255425 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 msgr2=0x7f1ac41ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 -- 172.21.2.117:0/2307255425 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4197790 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac287c700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac307d700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 0x7f1ac40adad0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f1ab4009540 tx=0x7f1ab400b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab40063c0 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab400bd60 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac4197790 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.647 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ab4002030 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.648 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac4197790 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.648 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ab4000c90 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.648 7f1aabfff700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 0x7f1aa401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.648 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1ab40326d0 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.648 7f1ac287c700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 0x7f1aa401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:12.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.649 7f1ac287c700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 0x7f1aa401ac10 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:12.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.680 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac40e77f0 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.683 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ab4006ae0 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.836 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1ac40e77f0 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.841 7f1aabfff700 1 -- 172.21.2.117:0/2307255425 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1ab4000c90 con 0x7f1ac40ad6d0 2021-02-10T11:44:12.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.844 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 msgr2=0x7f1aa401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.844 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 0x7f1aa401ac10 crc :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.844 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 msgr2=0x7f1ac40adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:12.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.844 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 0x7f1ac40adad0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f1ab4009540 tx=0x7f1ab400b8e0).stop 2021-02-10T11:44:12.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 shutdown_connections 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1aa4018730 0x7f1aa401ac10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ac41ad4b0 0x7f1ac41ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ac41aded0 0x7f1ac41ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 --2- 172.21.2.117:0/2307255425 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ac40ad6d0 0x7f1ac40adad0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 >> 172.21.2.117:0/2307255425 conn(0x7f1ac41998b0 msgr2=0x7f1ac41a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 shutdown_connections 2021-02-10T11:44:12.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:12.845 7f1ac92f9700 1 -- 172.21.2.117:0/2307255425 wait complete. 2021-02-10T11:44:12.846 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_objecter 2021-02-10T11:44:13.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.267 7fb1208b6700 1 Processor -- start 2021-02-10T11:44:13.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.267 7fb1208b6700 1 -- start start 2021-02-10T11:44:13.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb1208b6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 0x7fb11c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb1208b6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb11c0e5240 0x7fb11c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb1208b6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c085930 con 0x7fb11c1b0810 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb1208b6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c0e5cc0 con 0x7fb11c0e5240 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb1208b6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c084840 con 0x7fb11c0e4e40 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb11ae1d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fb11c1b0810 0x7fb11c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:43828/0 (socket says 172.21.2.117:43828) 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb11ae1d700 1 -- 172.21.2.117:0/146473781 learned_addr learned my addr 172.21.2.117:0/146473781 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 --2- 172.21.2.117:0/146473781 >> v2:172.21.2.117:3300/0 conn(0x7fb11c0e5240 0x7fb11c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb11a61c700 1 --2- 172.21.2.117:0/146473781 >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 0x7fb11c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 -- 172.21.2.117:0/146473781 >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 msgr2=0x7fb11c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 --2- 172.21.2.117:0/146473781 >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 0x7fb11c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 -- 172.21.2.117:0/146473781 >> v1:172.21.2.125:6789/0 conn(0x7fb11c1b0810 legacy=0x7fb11c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 -- 172.21.2.117:0/146473781 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c085280 con 0x7fb11c0e5240 2021-02-10T11:44:13.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb11a61c700 1 --2- 172.21.2.117:0/146473781 >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 0x7fb11c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.268 7fb119e1b700 1 --2- 172.21.2.117:0/146473781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0e5240 0x7fb11c084c60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7fb104006650 tx=0x7fb104009a80).ready entity=mon.0 client_cookie=d19376490a2af390 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb11961a700 1 -- 172.21.2.117:0/146473781 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1040009b0 con 0x7fb11c0e5240 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb11961a700 1 -- 172.21.2.117:0/146473781 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400c0c0 con 0x7fb11c0e5240 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb11961a700 1 -- 172.21.2.117:0/146473781 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb10400c5e0 con 0x7fb11c0e5240 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 -- 172.21.2.117:0/146473781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0e5240 msgr2=0x7fb11c084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 --2- 172.21.2.117:0/146473781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0e5240 0x7fb11c084c60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7fb104006650 tx=0x7fb104009a80).stop 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 -- 172.21.2.117:0/146473781 shutdown_connections 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 --2- 172.21.2.117:0/146473781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0e5240 0x7fb11c084c60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 --2- 172.21.2.117:0/146473781 >> v2:172.21.2.125:3300/0 conn(0x7fb11c0e4e40 0x7fb11c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.269 7fb1208b6700 1 -- 172.21.2.117:0/146473781 >> 172.21.2.117:0/146473781 conn(0x7fb11c0ce4e0 msgr2=0x7fb11c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.270 7fb1208b6700 1 -- 172.21.2.117:0/146473781 shutdown_connections 2021-02-10T11:44:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.270 7fb1208b6700 1 -- 172.21.2.117:0/146473781 wait complete. 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.284 7fb1208b6700 1 Processor -- start 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.284 7fb1208b6700 1 -- start start 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 0x7fb11c1a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0baab0 con 0x7fb11c0b9e40 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0bb030 con 0x7fb11c1a2dd0 2021-02-10T11:44:13.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb1208b6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c1a37f0 con 0x7fb11c0b79b0 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb119e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11ae1d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 0x7fb11c1a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:48336/0 (socket says 172.21.2.117:48336) 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb119e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50660/0 (socket says 172.21.2.117:50660) 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 -- 172.21.2.117:0/4279610422 learned_addr learned my addr 172.21.2.117:0/4279610422 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 -- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 msgr2=0x7fb11c1a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 0x7fb11c1a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 -- 172.21.2.117:0/4279610422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 msgr2=0x7fb11c0ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11a61c700 1 -- 172.21.2.117:0/4279610422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c1a3a10 con 0x7fb11c0b79b0 2021-02-10T11:44:13.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.285 7fb11ae1d700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 0x7fb11c1a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb119e1b700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb11a61c700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb10c006f40 tx=0x7fb10c00b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb10c00bd80 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c000c90 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb10c0022b0 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c1a3a10 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.286 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c1a3a10 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.287 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb10c00a000 con 0x7fb11c0b79b0 2021-02-10T11:44:13.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.287 7fb112ffd700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 0x7fb10001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.287 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb10c0326c0 con 0x7fb11c0b79b0 2021-02-10T11:44:13.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.287 7fb119e1b700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 0x7fb10001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.288 7fb119e1b700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 0x7fb10001abc0 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.317 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c10c290 con 0x7fb11c0b79b0 2021-02-10T11:44:13.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.321 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb10c0304b0 con 0x7fb11c0b79b0 2021-02-10T11:44:13.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.473 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"} v 0) v1 -- 0x7fb11c085930 con 0x7fb11c0b79b0 2021-02-10T11:44:13.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.479 7fb112ffd700 1 -- 172.21.2.117:0/4279610422 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_objecter"}]=0 v18) v1 ==== 106+0+0 (secure 0 0 0) 0x7fb10c0304b0 con 0x7fb11c0b79b0 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 msgr2=0x7fb10001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 0x7fb10001abc0 crc :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 msgr2=0x7fb11c0b7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb10c006f40 tx=0x7fb10c00b990).stop 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 shutdown_connections 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1000186c0 0x7fb10001abc0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb11c0b9e40 0x7fb11c0ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb11c1a2dd0 0x7fb11c1a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 --2- 172.21.2.117:0/4279610422 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb11c0b79b0 0x7fb11c0b7160 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.482 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 >> 172.21.2.117:0/4279610422 conn(0x7fb11c0ace70 msgr2=0x7fb11c0b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.483 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 shutdown_connections 2021-02-10T11:44:13.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.483 7fb1208b6700 1 -- 172.21.2.117:0/4279610422 wait complete. 2021-02-10T11:44:13.483 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm client.alice debug_client 2021-02-10T11:44:13.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.906 7f21a1a9b700 1 Processor -- start 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.906 7f21a1a9b700 1 -- start start 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f21a1a9b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f219c0d15e0 0x7f219c0e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f21a1a9b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0e7580 con 0x7f219c1b07f0 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f21a1a9b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0e77f0 con 0x7f219c0e7180 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f21a1a9b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c0e5ce0 con 0x7f219c0d15e0 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f219b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f219c1b07f0 0x7f219c0e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50914/0 (socket says 172.21.2.117:50914) 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f219b7fe700 1 -- 172.21.2.117:0/2895324293 learned_addr learned my addr 172.21.2.117:0/2895324293 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.907 7f219bfff700 1 --2- 172.21.2.117:0/2895324293 >> v2:172.21.2.125:3300/0 conn(0x7f219c0d15e0 0x7f219c0e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219a7fc700 1 -- 172.21.2.117:0/2895324293 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293765412 0 0) 0x7f2184001180 con 0x7f219c1b07f0 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219a7fc700 1 -- 172.21.2.117:0/2895324293 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188002fe0 con 0x7f219c1b07f0 2021-02-10T11:44:13.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219bfff700 1 -- 172.21.2.117:0/2895324293 >> v1:172.21.2.125:6789/0 conn(0x7f219c0e7180 legacy=0x7f219c0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219bfff700 1 -- 172.21.2.117:0/2895324293 >> v1:172.21.2.117:6789/0 conn(0x7f219c1b07f0 legacy=0x7f219c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219bfff700 1 -- 172.21.2.117:0/2895324293 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c0d3150 con 0x7f219c0d15e0 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219bfff700 1 --2- 172.21.2.117:0/2895324293 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c0d15e0 0x7f219c0e8560 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f218c006ad0 tx=0x7f218c009ad0).ready entity=mon.1 client_cookie=95a65e90647dac51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219a7fc700 1 -- 172.21.2.117:0/2895324293 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f218c0022e0 con 0x7f219c0d15e0 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219a7fc700 1 -- 172.21.2.117:0/2895324293 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c006410 con 0x7f219c0d15e0 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.908 7f219a7fc700 1 -- 172.21.2.117:0/2895324293 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f218c002560 con 0x7f219c0d15e0 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 -- 172.21.2.117:0/2895324293 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c0d15e0 msgr2=0x7f219c0e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 --2- 172.21.2.117:0/2895324293 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c0d15e0 0x7f219c0e8560 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f218c006ad0 tx=0x7f218c009ad0).stop 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 -- 172.21.2.117:0/2895324293 shutdown_connections 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 --2- 172.21.2.117:0/2895324293 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c0d15e0 0x7f219c0e8560 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 -- 172.21.2.117:0/2895324293 >> 172.21.2.117:0/2895324293 conn(0x7f219c0ce500 msgr2=0x7f219c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 -- 172.21.2.117:0/2895324293 shutdown_connections 2021-02-10T11:44:13.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.909 7f21a1a9b700 1 -- 172.21.2.117:0/2895324293 wait complete. 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 Processor -- start 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 -- start start 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f219c0a6300 0x7f219c0a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c1a5af0 0x7f219c1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0a5d80 con 0x7f219c1a56f0 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c1a5240 con 0x7f219c1a5af0 2021-02-10T11:44:13.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f21a1a9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c1a5420 con 0x7f219c0a6300 2021-02-10T11:44:13.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f219affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f219affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50672/0 (socket says 172.21.2.117:50672) 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.924 7f219affd700 1 -- 172.21.2.117:0/1632433624 learned_addr learned my addr 172.21.2.117:0/1632433624 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219bfff700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c1a5af0 0x7f219c1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219b7fe700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f219c0a6300 0x7f219c0a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 -- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f219c0a6300 msgr2=0x7f219c0a6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f219c0a6300 0x7f219c0a6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 -- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c1a5af0 msgr2=0x7f219c1a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c1a5af0 0x7f219c1a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 -- 172.21.2.117:0/1632433624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c0b77c0 con 0x7f219c1a56f0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f219affd700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f2194009940 tx=0x7f219400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2194000ce0 con 0x7f219c1a56f0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219400c2f0 con 0x7f219c1a56f0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.925 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219400c810 con 0x7f219c1a56f0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.926 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c0b77c0 con 0x7f219c1a56f0 2021-02-10T11:44:13.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.926 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c0b7d50 con 0x7f219c1a56f0 2021-02-10T11:44:13.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.926 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2194001230 con 0x7f219c1a56f0 2021-02-10T11:44:13.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.926 7f2193fff700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 0x7f218001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:13.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.926 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2194032ac0 con 0x7f219c1a56f0 2021-02-10T11:44:13.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.927 7f219b7fe700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 0x7f218001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:13.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.927 7f219b7fe700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 0x7f218001abc0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:13.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.956 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c0e77f0 con 0x7f219c1a56f0 2021-02-10T11:44:13.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:13.958 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f219402cfc0 con 0x7f219c1a56f0 2021-02-10T11:44:14.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.109 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "client.alice", "name": "debug_client"} v 0) v1 -- 0x7f219c0e77f0 con 0x7f219c1a56f0 2021-02-10T11:44:14.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.113 7f2193fff700 1 -- 172.21.2.117:0/1632433624 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "client.alice", "name": "debug_client"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f219402cfc0 con 0x7f219c1a56f0 2021-02-10T11:44:14.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.117 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 msgr2=0x7f218001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.117 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 0x7f218001abc0 crc :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.117 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 msgr2=0x7f219c1a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.117 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7f2194009940 tx=0x7f219400bcf0).stop 2021-02-10T11:44:14.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 shutdown_connections 2021-02-10T11:44:14.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21800186c0 0x7f218001abc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f219c1a56f0 0x7f219c1a4600 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f219c1a5af0 0x7f219c1a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 --2- 172.21.2.117:0/1632433624 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f219c0a6300 0x7f219c0a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 >> 172.21.2.117:0/1632433624 conn(0x7f219c097700 msgr2=0x7f219c096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 shutdown_connections 2021-02-10T11:44:14.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.118 7f21a1a9b700 1 -- 172.21.2.117:0/1632433624 wait complete. 2021-02-10T11:44:14.118 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume' 2021-02-10T11:44:14.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.535 7fb5e7388700 1 Processor -- start 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.535 7fb5e7388700 1 -- start start 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e7388700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb5e00d18e0 0x7fb5e00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e7388700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb5e00eb000 0x7fb5e00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e7388700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00ede60 con 0x7fb5e00ee3e0 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e7388700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00eba20 con 0x7fb5e00eb000 2021-02-10T11:44:14.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e7388700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00d1ce0 con 0x7fb5e00d18e0 2021-02-10T11:44:14.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb5e00eb000 0x7fb5e00eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:14.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb5e00eb000 0x7fb5e00eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50676/0 (socket says 172.21.2.117:50676) 2021-02-10T11:44:14.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 -- 172.21.2.117:0/1191750 learned_addr learned my addr 172.21.2.117:0/1191750 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:14.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 -- 172.21.2.117:0/1191750 >> v2:172.21.2.125:3300/0 conn(0x7fb5e00d18e0 msgr2=0x7fb5e00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:14.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 --2- 172.21.2.117:0/1191750 >> v2:172.21.2.125:3300/0 conn(0x7fb5e00d18e0 0x7fb5e00ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 -- 172.21.2.117:0/1191750 >> v1:172.21.2.125:6789/0 conn(0x7fb5e00ee3e0 legacy=0x7fb5e00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.536 7fb5e49a3700 1 -- 172.21.2.117:0/1191750 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00eefd0 con 0x7fb5e00eb000 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5e49a3700 1 --2- 172.21.2.117:0/1191750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e00eb000 0x7fb5e00eb400 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fb5dc006700 tx=0x7fb5dc009b80).ready entity=mon.0 client_cookie=3c88b982f6746a42 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5d3fff700 1 -- 172.21.2.117:0/1191750 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5dc00c0c0 con 0x7fb5e00eb000 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5d3fff700 1 -- 172.21.2.117:0/1191750 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc00c440 con 0x7fb5e00eb000 2021-02-10T11:44:14.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5d3fff700 1 -- 172.21.2.117:0/1191750 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5dc00c920 con 0x7fb5e00eb000 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5e7388700 1 -- 172.21.2.117:0/1191750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e00eb000 msgr2=0x7fb5e00eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.537 7fb5e7388700 1 --2- 172.21.2.117:0/1191750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e00eb000 0x7fb5e00eb400 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fb5dc006700 tx=0x7fb5dc009b80).stop 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 -- 172.21.2.117:0/1191750 shutdown_connections 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 --2- 172.21.2.117:0/1191750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e00eb000 0x7fb5e00eb400 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 --2- 172.21.2.117:0/1191750 >> v2:172.21.2.125:3300/0 conn(0x7fb5e00d18e0 0x7fb5e00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:14.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 -- 172.21.2.117:0/1191750 >> 172.21.2.117:0/1191750 conn(0x7fb5e00ced60 msgr2=0x7fb5e00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:14.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 -- 172.21.2.117:0/1191750 shutdown_connections 2021-02-10T11:44:14.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.538 7fb5e7388700 1 -- 172.21.2.117:0/1191750 wait complete. 2021-02-10T11:44:14.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.552 7fb5e7388700 1 Processor -- start 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.552 7fb5e7388700 1 -- start start 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 0x7fb5e00ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 0x7fb5e00df020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e008d0e0 con 0x7fb5e008c6c0 2021-02-10T11:44:14.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e008d2d0 con 0x7fb5e00de260 2021-02-10T11:44:14.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e7388700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00de660 con 0x7fb5e00dd840 2021-02-10T11:44:14.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e51a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:14.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e51a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50680/0 (socket says 172.21.2.117:50680) 2021-02-10T11:44:14.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e51a4700 1 -- 172.21.2.117:0/137978880 learned_addr learned my addr 172.21.2.117:0/137978880 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:14.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e59a5700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 0x7fb5e00df020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.553 7fb5e49a3700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 0x7fb5e00ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 -- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 msgr2=0x7fb5e00ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 0x7fb5e00ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 -- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 msgr2=0x7fb5e00df020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 0x7fb5e00df020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 -- 172.21.2.117:0/137978880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00de840 con 0x7fb5e008c6c0 2021-02-10T11:44:14.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e49a3700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 0x7fb5e00ddc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:14.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e59a5700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 0x7fb5e00df020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:14.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e51a4700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7fb5d4009980 tx=0x7fb5d400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:14.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d400c2f0 con 0x7fb5e008c6c0 2021-02-10T11:44:14.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d400c570 con 0x7fb5e008c6c0 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e7388700 1 -- 172.21.2.117:0/137978880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e00de840 con 0x7fb5e008c6c0 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5e7388700 1 -- 172.21.2.117:0/137978880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e00df9a0 con 0x7fb5e008c6c0 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.554 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb5d400ca90 con 0x7fb5e008c6c0 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.555 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb5d4001010 con 0x7fb5e008c6c0 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.555 7fb5d1ffb700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 0x7fb5c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:14.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.555 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb5d4032a10 con 0x7fb5e008c6c0 2021-02-10T11:44:14.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.555 7fb5e49a3700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 0x7fb5c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:14.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.556 7fb5e49a3700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 0x7fb5c001abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:14.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.586 7fb5e7388700 1 -- 172.21.2.117:0/137978880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0087280 con 0x7fb5e008c6c0 2021-02-10T11:44:14.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.589 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb5d402b480 con 0x7fb5e008c6c0 2021-02-10T11:44:14.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.718 7fb5e7388700 1 -- 172.21.2.117:0/137978880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]} v 0) v1 -- 0x7fb5e0087280 con 0x7fb5e008c6c0 2021-02-10T11:44:14.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.896 7fb5d1ffb700 1 -- 172.21.2.117:0/137978880 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_authorize_deauthorize_subvolume"]}]=0 v384) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb5d402b480 con 0x7fb5e008c6c0 2021-02-10T11:44:14.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 -- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 msgr2=0x7fb5c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 0x7fb5c001abc0 crc :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 -- 172.21.2.117:0/137978880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 msgr2=0x7fb5e008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:14.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7fb5d4009980 tx=0x7fb5d400bcd0).stop 2021-02-10T11:44:14.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 -- 172.21.2.117:0/137978880 shutdown_connections 2021-02-10T11:44:14.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb5e00de260 0x7fb5e00df020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb5e00dd840 0x7fb5e00ddc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb5c00186c0 0x7fb5c001abc0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 --2- 172.21.2.117:0/137978880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb5e008c6c0 0x7fb5e008cac0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.935 7fb5e7388700 1 -- 172.21.2.117:0/137978880 >> 172.21.2.117:0/137978880 conn(0x7fb5e00874b0 msgr2=0x7fb5e01a6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.936 7fb5e7388700 1 -- 172.21.2.117:0/137978880 shutdown_connections 2021-02-10T11:44:14.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:14.936 7fb5e7388700 1 -- 172.21.2.117:0/137978880 wait complete. 2021-02-10T11:44:14.936 INFO:tasks.cephfs_test_runner:test_authorize_deauthorize_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:44:14.937 INFO:tasks.cephfs_test_runner:Starting test: test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:44:14.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.357 7f11dbf77700 1 Processor -- start 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.357 7f11dbf77700 1 -- start start 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40d18e0 0x7f11d40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11d40eb000 0x7f11d40eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f11d40ee3e0 0x7f11d40ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d40ede60 con 0x7f11d40eb000 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d40eba20 con 0x7f11d40ee3e0 2021-02-10T11:44:15.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40d1ce0 con 0x7f11d40d18e0 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11d40eb000 0x7f11d40eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11d40eb000 0x7f11d40eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50688/0 (socket says 172.21.2.117:50688) 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 -- 172.21.2.117:0/653452126 learned_addr learned my addr 172.21.2.117:0/653452126 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 -- 172.21.2.117:0/653452126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40d18e0 msgr2=0x7f11d40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 --2- 172.21.2.117:0/653452126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40d18e0 0x7f11d40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 -- 172.21.2.117:0/653452126 >> v2:172.21.2.125:3300/0 conn(0x7f11d40ee3e0 msgr2=0x7f11d40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 --2- 172.21.2.117:0/653452126 >> v2:172.21.2.125:3300/0 conn(0x7f11d40ee3e0 0x7f11d40ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 -- 172.21.2.117:0/653452126 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40eee00 con 0x7f11d40eb000 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d9592700 1 --2- 172.21.2.117:0/653452126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40eb000 0x7f11d40eb400 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f11c8006700 tx=0x7f11c8009b80).ready entity=mon.0 client_cookie=147bd4d749b62375 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d8d91700 1 -- 172.21.2.117:0/653452126 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c800c0c0 con 0x7f11d40eb000 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d8d91700 1 -- 172.21.2.117:0/653452126 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c800c440 con 0x7f11d40eb000 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11d8d91700 1 -- 172.21.2.117:0/653452126 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c800c8a0 con 0x7f11d40eb000 2021-02-10T11:44:15.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 -- 172.21.2.117:0/653452126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40eb000 msgr2=0x7f11d40eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.358 7f11dbf77700 1 --2- 172.21.2.117:0/653452126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40eb000 0x7f11d40eb400 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 rx=0x7f11c8006700 tx=0x7f11c8009b80).stop 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 -- 172.21.2.117:0/653452126 shutdown_connections 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 --2- 172.21.2.117:0/653452126 >> v2:172.21.2.125:3300/0 conn(0x7f11d40ee3e0 0x7f11d40ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 --2- 172.21.2.117:0/653452126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40eb000 0x7f11d40eb400 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 --2- 172.21.2.117:0/653452126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40d18e0 0x7f11d40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 -- 172.21.2.117:0/653452126 >> 172.21.2.117:0/653452126 conn(0x7f11d40ced60 msgr2=0x7f11d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 -- 172.21.2.117:0/653452126 shutdown_connections 2021-02-10T11:44:15.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.359 7f11dbf77700 1 -- 172.21.2.117:0/653452126 wait complete. 2021-02-10T11:44:15.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 Processor -- start 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 -- start start 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11d40ded50 0x7f11d40e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40e0cc0 0x7f11d40e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40e24f0 con 0x7f11d40e1f10 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40e9020 con 0x7f11d40ded50 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11dbf77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40e95a0 con 0x7f11d40e0cc0 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.374 7f11da594700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50694/0 (socket says 172.21.2.117:50694) 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 -- 172.21.2.117:0/4162775690 learned_addr learned my addr 172.21.2.117:0/4162775690 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 -- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40e0cc0 msgr2=0x7f11d40e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:15.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40e0cc0 0x7f11d40e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 -- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11d40ded50 msgr2=0x7f11d40e12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11d40ded50 0x7f11d40e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 -- 172.21.2.117:0/4162775690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40dfb30 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11da594700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f11cc009980 tx=0x7f11cc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11cc005af0 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc005d70 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11cc0062b0 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d40dfb30 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.375 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40dfb30 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.376 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11cc006030 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.376 7f11c67fc700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 0x7f11bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.376 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f11cc032a50 con 0x7f11d40e1f10 2021-02-10T11:44:15.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.376 7f11d9d93700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 0x7f11bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:15.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.377 7f11d9d93700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 0x7f11bc01abc0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:15.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.407 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4087210 con 0x7f11d40e1f10 2021-02-10T11:44:15.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.410 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11cc005af0 con 0x7f11d40e1f10 2021-02-10T11:44:15.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.541 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f11d4087210 con 0x7f11d40e1f10 2021-02-10T11:44:15.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.904 7f11c67fc700 1 -- 172.21.2.117:0/4162775690 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v385) v1 ==== 145+0+0 (secure 0 0 0) 0x7f11cc006030 con 0x7f11d40e1f10 2021-02-10T11:44:15.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 msgr2=0x7f11bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:15.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 0x7f11bc01abc0 crc :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 msgr2=0x7f11d40df510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f11cc009980 tx=0x7f11cc00bcd0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 shutdown_connections 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11bc0186c0 0x7f11bc01abc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11d40e1f10 0x7f11d40df510 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11d40e0cc0 0x7f11d40e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 --2- 172.21.2.117:0/4162775690 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11d40ded50 0x7f11d40e12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 >> 172.21.2.117:0/4162775690 conn(0x7f11d419f610 msgr2=0x7f11d4199b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 shutdown_connections 2021-02-10T11:44:15.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:15.910 7f11dbf77700 1 -- 172.21.2.117:0/4162775690 wait complete. 2021-02-10T11:44:15.943 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:44:16.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 Processor -- start 2021-02-10T11:44:16.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 -- start start 2021-02-10T11:44:16.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5400d18f0 0x7fa5400eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5400ef3f0 0x7fa5401b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400ef7f0 con 0x7fa5400eb040 2021-02-10T11:44:16.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400eded0 con 0x7fa5400d18f0 2021-02-10T11:44:16.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.387 7fa546bc5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400d1cf0 con 0x7fa5400ef3f0 2021-02-10T11:44:16.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5400d18f0 0x7fa5400eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:16.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa53ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa5400eb040 0x7fa5400eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50948/0 (socket says 172.21.2.117:50948) 2021-02-10T11:44:16.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa53ffff700 1 -- 172.21.2.117:0/2318967974 learned_addr learned my addr 172.21.2.117:0/2318967974 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 -- 172.21.2.117:0/2318967974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5400ef3f0 msgr2=0x7fa5401b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 --2- 172.21.2.117:0/2318967974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5400ef3f0 0x7fa5401b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 -- 172.21.2.117:0/2318967974 >> v1:172.21.2.117:6789/0 conn(0x7fa5400eb040 legacy=0x7fa5400eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 -- 172.21.2.117:0/2318967974 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5401b8fd0 con 0x7fa5400d18f0 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa5449e1700 1 --2- 172.21.2.117:0/2318967974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5400d18f0 0x7fa5400eaa20 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7fa528006650 tx=0x7fa528009a80).ready entity=mon.0 client_cookie=3e2d6565b4034490 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa53f7fe700 1 -- 172.21.2.117:0/2318967974 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa52800c0c0 con 0x7fa5400d18f0 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa53f7fe700 1 -- 172.21.2.117:0/2318967974 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52800c440 con 0x7fa5400d18f0 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa53f7fe700 1 -- 172.21.2.117:0/2318967974 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa52800c960 con 0x7fa5400d18f0 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa546bc5700 1 -- 172.21.2.117:0/2318967974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5400d18f0 msgr2=0x7fa5400eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:16.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.388 7fa546bc5700 1 --2- 172.21.2.117:0/2318967974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5400d18f0 0x7fa5400eaa20 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7fa528006650 tx=0x7fa528009a80).stop 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 -- 172.21.2.117:0/2318967974 shutdown_connections 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 --2- 172.21.2.117:0/2318967974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5400ef3f0 0x7fa5401b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 --2- 172.21.2.117:0/2318967974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5400d18f0 0x7fa5400eaa20 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 -- 172.21.2.117:0/2318967974 >> 172.21.2.117:0/2318967974 conn(0x7fa5400cf060 msgr2=0x7fa5400cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 -- 172.21.2.117:0/2318967974 shutdown_connections 2021-02-10T11:44:16.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.389 7fa546bc5700 1 -- 172.21.2.117:0/2318967974 wait complete. 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.404 7fa546bc5700 1 Processor -- start 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.404 7fa546bc5700 1 -- start start 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.404 7fa546bc5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa540093be0 0x7fa5401a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.404 7fa546bc5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5401a7ed0 0x7fa54008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.404 7fa546bc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa546bc5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa540094280 con 0x7fa54008c850 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa546bc5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa540093fe0 con 0x7fa540093be0 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa546bc5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400ace00 con 0x7fa5401a7ed0 2021-02-10T11:44:16.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50706/0 (socket says 172.21.2.117:50706) 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 -- 172.21.2.117:0/1857565167 learned_addr learned my addr 172.21.2.117:0/1857565167 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 -- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5401a7ed0 msgr2=0x7fa54008c230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5401a7ed0 0x7fa54008c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 -- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa540093be0 msgr2=0x7fa5401a78b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa540093be0 0x7fa5401a78b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 -- 172.21.2.117:0/1857565167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540094800 con 0x7fa54008c850 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa5451e2700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7fa53800b8c0 tx=0x7fa53800bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa538005870 con 0x7fa54008c850 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa538005b30 con 0x7fa54008c850 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa538006030 con 0x7fa54008c850 2021-02-10T11:44:16.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.405 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa540094800 con 0x7fa54008c850 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.406 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa540094800 con 0x7fa54008c850 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.406 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa538005d10 con 0x7fa54008c850 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.406 7fa53d7fa700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 0x7fa52401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.406 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa538032aa0 con 0x7fa54008c850 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.407 7fa5449e1700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 0x7fa52401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:16.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.408 7fa5449e1700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 0x7fa52401abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:16.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.437 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5400ede60 con 0x7fa54008c850 2021-02-10T11:44:16.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.440 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa538005870 con 0x7fa54008c850 2021-02-10T11:44:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.588 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa5400ede60 con 0x7fa54008c850 2021-02-10T11:44:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.591 7fa53d7fa700 1 -- 172.21.2.117:0/1857565167 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa538005d10 con 0x7fa54008c850 2021-02-10T11:44:16.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.596 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 msgr2=0x7fa52401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:16.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.596 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 0x7fa52401abc0 crc :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:16.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.596 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 msgr2=0x7fa54008cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:16.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7fa53800b8c0 tx=0x7fa53800bc60).stop 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 shutdown_connections 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5240186c0 0x7fa52401abc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa54008c850 0x7fa54008cc50 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5401a7ed0 0x7fa54008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 --2- 172.21.2.117:0/1857565167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa540093be0 0x7fa5401a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 >> 172.21.2.117:0/1857565167 conn(0x7fa540092d00 msgr2=0x7fa54008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 shutdown_connections 2021-02-10T11:44:16.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:16.597 7fa546bc5700 1 -- 172.21.2.117:0/1857565167 wait complete. 2021-02-10T11:44:16.597 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:44:16.597 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:16.941 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:44:17.284 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:44:17.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.285 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:17.285 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:17.620 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:44:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:17.621 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:17.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.621 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:17.621 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:17.955 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:44:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:17.956 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:17.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:17.956 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:17.956 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:44:18.292 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:44:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:18.292 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:44:18.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:18.292 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:18.293 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:44:18.626 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:44:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:44:18.626 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:44:18.627 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 Processor -- start 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 -- start start 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f885c0ef3f0 0x7f885c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c0eb7f0 0x7f885c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c0ede60 con 0x7f885c0d18e0 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c0ef7f0 con 0x7f885c0ef3f0 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.046 7f8861388700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0d1ce0 con 0x7f885c0eb7f0 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f885c0ef3f0 0x7f885c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f885c0ef3f0 0x7f885c0eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50712/0 (socket says 172.21.2.117:50712) 2021-02-10T11:44:19.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 -- 172.21.2.117:0/2205874609 learned_addr learned my addr 172.21.2.117:0/2205874609 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885b07d700 1 --1- 172.21.2.117:0/2205874609 >> v1:172.21.2.117:6789/0 conn(0x7f885c0d18e0 0x7f885c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50962/0 (socket says 172.21.2.117:50962) 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885b87e700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c0eb7f0 0x7f885c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 -- 172.21.2.117:0/2205874609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c0eb7f0 msgr2=0x7f885c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c0eb7f0 0x7f885c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 -- 172.21.2.117:0/2205874609 >> v1:172.21.2.117:6789/0 conn(0x7f885c0d18e0 legacy=0x7f885c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 -- 172.21.2.117:0/2205874609 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c1b8f90 con 0x7f885c0ef3f0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a87c700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c0ef3f0 0x7f885c0eb1d0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f8844006700 tx=0x7f8844009b80).ready entity=mon.0 client_cookie=9566d307cdb87d1b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.047 7f885a07b700 1 -- 172.21.2.117:0/2205874609 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f884400c0c0 con 0x7f885c0ef3f0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f885a07b700 1 -- 172.21.2.117:0/2205874609 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884400c340 con 0x7f885c0ef3f0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f885a07b700 1 -- 172.21.2.117:0/2205874609 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f884400c820 con 0x7f885c0ef3f0 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 -- 172.21.2.117:0/2205874609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c0ef3f0 msgr2=0x7f885c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c0ef3f0 0x7f885c0eb1d0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f8844006700 tx=0x7f8844009b80).stop 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 -- 172.21.2.117:0/2205874609 shutdown_connections 2021-02-10T11:44:19.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c0eb7f0 0x7f885c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 --2- 172.21.2.117:0/2205874609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c0ef3f0 0x7f885c0eb1d0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 -- 172.21.2.117:0/2205874609 >> 172.21.2.117:0/2205874609 conn(0x7f885c0ced60 msgr2=0x7f885c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 -- 172.21.2.117:0/2205874609 shutdown_connections 2021-02-10T11:44:19.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.048 7f8861388700 1 -- 172.21.2.117:0/2205874609 wait complete. 2021-02-10T11:44:19.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 Processor -- start 2021-02-10T11:44:19.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 -- start start 2021-02-10T11:44:19.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f885c1b24b0 0x7f885c1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c198250 0x7f885c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c197cd0 con 0x7f885c091d40 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c092140 con 0x7f885c1b24b0 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f8861388700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c1b2ed0 con 0x7f885c198250 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50718/0 (socket says 172.21.2.117:50718) 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 -- 172.21.2.117:0/261254303 learned_addr learned my addr 172.21.2.117:0/261254303 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 -- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c198250 msgr2=0x7f885c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c198250 0x7f885c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 -- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f885c1b24b0 msgr2=0x7f885c1b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f885c1b24b0 0x7f885c1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.063 7f885b07d700 1 -- 172.21.2.117:0/261254303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c198c70 con 0x7f885c091d40 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f885b07d700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f884c001c60 tx=0x7f884c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f884c00acc0 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c004140 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f884c0045a0 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8861388700 1 -- 172.21.2.117:0/261254303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c198c70 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8861388700 1 -- 172.21.2.117:0/261254303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c198c70 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f884c000d90 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.064 7f8853fff700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 0x7f884001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.065 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f884c032780 con 0x7f885c091d40 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.065 7f885a87c700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 0x7f884001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.065 7f885a87c700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 0x7f884001abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.096 7f8861388700 1 -- 172.21.2.117:0/261254303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c087290 con 0x7f885c091d40 2021-02-10T11:44:19.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.099 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f884c0009b0 con 0x7f885c091d40 2021-02-10T11:44:19.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.224 7f8861388700 1 -- 172.21.2.117:0/261254303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f885c087290 con 0x7f885c091d40 2021-02-10T11:44:19.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.225 7f8853fff700 1 -- 172.21.2.117:0/261254303 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v64) v1 ==== 81+0+30755 (secure 0 0 0) 0x7f884c000d90 con 0x7f885c091d40 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 64, 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:43:09.677005", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:44:19.217 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 13, 2021-02-10T11:44:19.218 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:44:19.219 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:44:19.220 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:44:19.221 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:19.222 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:44:19.223 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 12, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:43:06.649220", 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:44:19.224 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:44:19.225 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "63", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:44:19.226 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:19.227 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:44:19.228 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.229 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 13, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:43:07.719391", 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:44:19.230 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "64", 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:44:19.231 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:44:19.232 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:44:19.233 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:44:19.234 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.235 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.236 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.237 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.238 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.239 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:44:19.240 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.241 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 61, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.242 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.243 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.244 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.245 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.246 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.247 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 54, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.248 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.249 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:44:19.250 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:44:19.251 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.252 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.253 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 45, 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:44:19.254 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.255 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.256 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.257 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.258 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:44:19.260 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.261 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:44:19.262 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:44:19.263 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:44:19.264 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.265 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.266 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.267 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.268 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.269 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.270 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:44:19.271 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.272 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.273 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.274 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.275 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:44:19.276 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:44:19.277 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.278 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.279 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.280 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:44:19.281 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:44:19.282 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.283 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.284 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.285 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.286 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:44:19.287 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:44:19.288 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:44:19.289 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:44:19.290 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 -- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 msgr2=0x7f884001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 0x7f884001abc0 crc :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 -- 172.21.2.117:0/261254303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 msgr2=0x7f885c0e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f884c001c60 tx=0x7f884c00a8d0).stop 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 -- 172.21.2.117:0/261254303 shutdown_connections 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f885c198250 0x7f885c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f885c1b24b0 0x7f885c1b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8840018730 0x7f884001abc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 --2- 172.21.2.117:0/261254303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f885c091d40 0x7f885c0e71e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.226 7f8861388700 1 -- 172.21.2.117:0/261254303 >> 172.21.2.117:0/261254303 conn(0x7f885c1a3620 msgr2=0x7f885c0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.227 7f8861388700 1 -- 172.21.2.117:0/261254303 shutdown_connections 2021-02-10T11:44:19.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.227 7f8861388700 1 -- 172.21.2.117:0/261254303 wait complete. 2021-02-10T11:44:19.293 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:19.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.711 7f768b447700 1 Processor -- start 2021-02-10T11:44:19.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.711 7f768b447700 1 -- start start 2021-02-10T11:44:19.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f768b447700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840d18e0 0x7f76840ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f768b447700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840ede60 con 0x7f76840eb000 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f768b447700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840d1ce0 con 0x7f76840ef3f0 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f768b447700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7684088020 con 0x7f76840d18e0 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f7688a62700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f76840eb000 0x7f76840eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50976/0 (socket says 172.21.2.117:50976) 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f7688a62700 1 -- 172.21.2.117:0/179092613 learned_addr learned my addr 172.21.2.117:0/179092613 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676279213 0 0) 0x7f76780010b0 con 0x7f76840eb000 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f7677fff700 1 -- 172.21.2.117:0/179092613 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670002fe0 con 0x7f76840eb000 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.712 7f7689263700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840d18e0 0x7f76840ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3688876610 0 0) 0x7f7678001010 con 0x7f76840eb000 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840d18e0 msgr2=0x7f76840ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840d18e0 0x7f76840ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 >> v1:172.21.2.125:6789/0 conn(0x7f76840ef3f0 legacy=0x7f76841b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76841bb180 con 0x7f76840eb000 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7678001530 con 0x7f76840eb000 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 >> v1:172.21.2.117:6789/0 conn(0x7f76840eb000 legacy=0x7f76840eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7670003b10 0x7f7670005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7670006770 0x7f7670008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76700094f0 0x7f767000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c2c0 con 0x7f76700094f0 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c4e0 con 0x7f7670006770 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7677fff700 1 -- 172.21.2.117:0/179092613 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767000c700 con 0x7f7670003b10 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7688a62700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7670006770 0x7f7670008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689263700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7670003b10 0x7f7670005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76700094f0 0x7f767000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 -- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7670003b10 msgr2=0x7f7670005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7670003b10 0x7f7670005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 -- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7670006770 msgr2=0x7f7670008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7670006770 0x7f7670008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689a64700 1 -- 172.21.2.117:0/179092613 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670050970 con 0x7f76700094f0 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.713 7f7689263700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7670003b10 0x7f7670005f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f7689a64700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76700094f0 0x7f767000ba10 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f767c006e40 tx=0x7f767c009e30).ready entity=mon.0 client_cookie=bd72f7b518004a7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767c00b300 con 0x7f76700094f0 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c00b580 con 0x7f76700094f0 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f7677fff700 1 -- 172.21.2.117:0/179092613 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767c00baa0 con 0x7f76700094f0 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f768b447700 1 -- 172.21.2.117:0/179092613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76700094f0 msgr2=0x7f767000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f768b447700 1 --2- 172.21.2.117:0/179092613 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76700094f0 0x7f767000ba10 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7f767c006e40 tx=0x7f767c009e30).stop 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f7689a64700 1 -- 172.21.2.117:0/179092613 reap_dead start 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f768b447700 1 -- 172.21.2.117:0/179092613 shutdown_connections 2021-02-10T11:44:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.714 7f768b447700 1 -- 172.21.2.117:0/179092613 >> 172.21.2.117:0/179092613 conn(0x7f76840ced60 msgr2=0x7f76840cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.715 7f768b447700 1 -- 172.21.2.117:0/179092613 shutdown_connections 2021-02-10T11:44:19.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.715 7f768b447700 1 -- 172.21.2.117:0/179092613 wait complete. 2021-02-10T11:44:19.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.729 7f768b447700 1 Processor -- start 2021-02-10T11:44:19.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.729 7f768b447700 1 -- start start 2021-02-10T11:44:19.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76841a33d0 0x7f76841a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76840b8d10 0x7f76840b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c0670 con 0x7f76841a33d0 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840deb20 con 0x7f76840b8d10 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f768b447700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840ded40 con 0x7f76840b82f0 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f7688a62700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f7688a62700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:48414/0 (socket says 172.21.2.117:48414) 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f7688a62700 1 -- 172.21.2.117:0/1606677465 learned_addr learned my addr 172.21.2.117:0/1606677465 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f7689263700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76841a33d0 0x7f76841a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.730 7f7689a64700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76840b8d10 0x7f76840b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 -- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76840b8d10 msgr2=0x7f76840b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76840b8d10 0x7f76840b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 -- 172.21.2.117:0/1606677465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76841a33d0 msgr2=0x7f76841a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76841a33d0 0x7f76841a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 -- 172.21.2.117:0/1606677465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840def60 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7688a62700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f767800a580 tx=0x7f767800a920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800ad10 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7678004140 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7678004660 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f768b447700 1 -- 172.21.2.117:0/1606677465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76840def60 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.731 7f768b447700 1 -- 172.21.2.117:0/1606677465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76840df4f0 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.732 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7678009030 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.732 7f7675ffb700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 0x7f766401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.732 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f76780326a0 con 0x7f76840b82f0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.732 7f7689263700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 0x7f766401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:19.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.732 7f7689263700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 0x7f766401abc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:19.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.762 7f768b447700 1 -- 172.21.2.117:0/1606677465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7684087290 con 0x7f76840b82f0 2021-02-10T11:44:19.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.766 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7678000dd0 con 0x7f76840b82f0 2021-02-10T11:44:19.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.894 7f768b447700 1 -- 172.21.2.117:0/1606677465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76840ede60 con 0x7f76840b82f0 2021-02-10T11:44:19.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.895 7f7675ffb700 1 -- 172.21.2.117:0/1606677465 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f7678009030 con 0x7f76840b82f0 2021-02-10T11:44:19.888 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 45 2021-02-10T11:44:19.888 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:19.888 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":45,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:43:10.684137","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6670":{"gid":6670,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.2.117:6833/2942282551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2942282551},{"type":"v1","addr":"172.21.2.117:6833","nonce":2942282551}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:44:19.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 -- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 msgr2=0x7f766401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 0x7f766401abc0 crc :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 -- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 msgr2=0x7f76840b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:19.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f767800a580 tx=0x7f767800a920).stop 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 -- 172.21.2.117:0/1606677465 shutdown_connections 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76840b8d10 0x7f76840b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76840b82f0 0x7f76840b86f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.898 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76640186c0 0x7f766401abc0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.899 7f768b447700 1 --2- 172.21.2.117:0/1606677465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76841a33d0 0x7f76841a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.899 7f768b447700 1 -- 172.21.2.117:0/1606677465 >> 172.21.2.117:0/1606677465 conn(0x7f76840874c0 msgr2=0x7f7684096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.899 7f768b447700 1 -- 172.21.2.117:0/1606677465 shutdown_connections 2021-02-10T11:44:19.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:19.899 7f768b447700 1 -- 172.21.2.117:0/1606677465 wait complete. 2021-02-10T11:44:19.899 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:44:20.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 Processor -- start 2021-02-10T11:44:20.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 -- start start 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faa981b07f0 0x7faa980e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa980e8d80 0x7faa980e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa980e77f0 con 0x7faa980e6170 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa980e54b0 con 0x7faa981b07f0 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.317 7faa9c809700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa980e9180 con 0x7faa980e8d80 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa95e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faa980e6170 0x7faa980e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:50992/0 (socket says 172.21.2.117:50992) 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa95e1b700 1 -- 172.21.2.117:0/3941308925 learned_addr learned my addr 172.21.2.117:0/3941308925 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 --2- 172.21.2.117:0/3941308925 >> v2:172.21.2.117:3300/0 conn(0x7faa981b07f0 0x7faa980e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9561a700 1 -- 172.21.2.117:0/3941308925 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839216588 0 0) 0x7faa880009b0 con 0x7faa980e6170 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9561a700 1 -- 172.21.2.117:0/3941308925 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa84002fe0 con 0x7faa980e6170 2021-02-10T11:44:20.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 -- 172.21.2.117:0/3941308925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa980e8d80 msgr2=0x7faa980e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 --2- 172.21.2.117:0/3941308925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa980e8d80 0x7faa980e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 -- 172.21.2.117:0/3941308925 >> v1:172.21.2.117:6789/0 conn(0x7faa980e6170 legacy=0x7faa980e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 -- 172.21.2.117:0/3941308925 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa980e7380 con 0x7faa981b07f0 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.318 7faa9661c700 1 --2- 172.21.2.117:0/3941308925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981b07f0 0x7faa980e4e90 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7faa80006650 tx=0x7faa80009a80).ready entity=mon.0 client_cookie=b082c3bdeff9912b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9561a700 1 -- 172.21.2.117:0/3941308925 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa8000c0c0 con 0x7faa981b07f0 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9561a700 1 -- 172.21.2.117:0/3941308925 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8000c340 con 0x7faa981b07f0 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9561a700 1 -- 172.21.2.117:0/3941308925 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa8000c860 con 0x7faa981b07f0 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 -- 172.21.2.117:0/3941308925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981b07f0 msgr2=0x7faa980e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 --2- 172.21.2.117:0/3941308925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981b07f0 0x7faa980e4e90 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7faa80006650 tx=0x7faa80009a80).stop 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 -- 172.21.2.117:0/3941308925 shutdown_connections 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 --2- 172.21.2.117:0/3941308925 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa980e8d80 0x7faa980e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 --2- 172.21.2.117:0/3941308925 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981b07f0 0x7faa980e4e90 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.319 7faa9c809700 1 -- 172.21.2.117:0/3941308925 >> 172.21.2.117:0/3941308925 conn(0x7faa980ce500 msgr2=0x7faa980e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:20.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.320 7faa9c809700 1 -- 172.21.2.117:0/3941308925 shutdown_connections 2021-02-10T11:44:20.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.320 7faa9c809700 1 -- 172.21.2.117:0/3941308925 wait complete. 2021-02-10T11:44:20.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.334 7faa9c809700 1 Processor -- start 2021-02-10T11:44:20.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.334 7faa9c809700 1 -- start start 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa980b55b0 0x7faa9819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa9819da30 0x7faa980b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa980b5a30 con 0x7faa981882f0 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa981886f0 con 0x7faa980b55b0 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9c809700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa98188b30 con 0x7faa9819da30 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50744/0 (socket says 172.21.2.117:50744) 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 -- 172.21.2.117:0/3984758555 learned_addr learned my addr 172.21.2.117:0/3984758555 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 -- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa9819da30 msgr2=0x7faa980b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa9819da30 0x7faa980b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 -- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa980b55b0 msgr2=0x7faa9819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa980b55b0 0x7faa9819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.335 7faa9661c700 1 -- 172.21.2.117:0/3984758555 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9819de30 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa9661c700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7faa80009630 tx=0x7faa8000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa800093a0 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8000bd70 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa800054d0 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa9c809700 1 -- 172.21.2.117:0/3984758555 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9819de30 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.336 7faa9c809700 1 -- 172.21.2.117:0/3984758555 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9819de30 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.337 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa8000a000 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.337 7faa8effd700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 0x7faa7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.337 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faa800328c0 con 0x7faa981882f0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.337 7faa95e1b700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 0x7faa7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:20.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.338 7faa95e1b700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 0x7faa7c01abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:20.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.368 7faa9c809700 1 -- 172.21.2.117:0/3984758555 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9807e800 con 0x7faa981882f0 2021-02-10T11:44:20.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.370 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa8001f4b0 con 0x7faa981882f0 2021-02-10T11:44:20.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:20.496 7faa9c809700 1 -- 172.21.2.117:0/3984758555 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faa9807e800 con 0x7faa981882f0 2021-02-10T11:44:21.157 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:44:21.166 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:44:21.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.172 7faa8effd700 1 -- 172.21.2.117:0/3984758555 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) v1 ==== 161+0+0 (secure 0 0 0) 0x7faa8001f6c0 con 0x7faa981882f0 2021-02-10T11:44:21.206 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 msgr2=0x7faa7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 0x7faa7c01abc0 crc :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 msgr2=0x7faa980b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7faa80009630 tx=0x7faa8000b980).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 shutdown_connections 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa9819da30 0x7faa980b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa980b55b0 0x7faa9819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa7c0186c0 0x7faa7c01abc0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 --2- 172.21.2.117:0/3984758555 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa981882f0 0x7faa980b4f90 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 >> 172.21.2.117:0/3984758555 conn(0x7faa980b7c90 msgr2=0x7faa980c0020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 shutdown_connections 2021-02-10T11:44:21.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.217 7faa9c809700 1 -- 172.21.2.117:0/3984758555 wait complete. 2021-02-10T11:44:21.217 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:44:21.217 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:21.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.636 7f4b73c8b700 1 Processor -- start 2021-02-10T11:44:21.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.636 7f4b73c8b700 1 -- start start 2021-02-10T11:44:21.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b73c8b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4b6c0ef3f0 0x7f4b6c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b73c8b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0ede60 con 0x7f4b6c0d18e0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b73c8b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0d1ce0 con 0x7f4b6c0eb7f0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b73c8b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c0ef7f0 con 0x7f4b6c0ef3f0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b71aa7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4b6c0d18e0 0x7f4b6c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51014/0 (socket says 172.21.2.117:51014) 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b71aa7700 1 -- 172.21.2.117:0/3586752768 learned_addr learned my addr 172.21.2.117:0/3586752768 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b712a6700 1 --2- 172.21.2.117:0/3586752768 >> v2:172.21.2.125:3300/0 conn(0x7f4b6c0ef3f0 0x7f4b6c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15596128 0 0) 0x7f4b68001180 con 0x7f4b6c0d18e0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b58002fe0 con 0x7f4b6c0d18e0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639586412 0 0) 0x7f4b64001400 con 0x7f4b6c0eb7f0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.637 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b58004370 con 0x7f4b6c0eb7f0 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b712a6700 1 -- 172.21.2.117:0/3586752768 >> v1:172.21.2.125:6789/0 conn(0x7f4b6c0eb7f0 legacy=0x7f4b6c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b712a6700 1 -- 172.21.2.117:0/3586752768 >> v1:172.21.2.117:6789/0 conn(0x7f4b6c0d18e0 legacy=0x7f4b6c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b712a6700 1 -- 172.21.2.117:0/3586752768 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c1bb180 con 0x7f4b6c0ef3f0 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b712a6700 1 --2- 172.21.2.117:0/3586752768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ef3f0 0x7f4b6c0eb1d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f4b5c006650 tx=0x7f4b5c009a80).ready entity=mon.1 client_cookie=573fc2e3d92a6200 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b5c0009b0 con 0x7f4b6c0ef3f0 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c006740 con 0x7f4b6c0ef3f0 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b70aa5700 1 -- 172.21.2.117:0/3586752768 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b5c00c0c0 con 0x7f4b6c0ef3f0 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b73c8b700 1 -- 172.21.2.117:0/3586752768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ef3f0 msgr2=0x7f4b6c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.638 7f4b73c8b700 1 --2- 172.21.2.117:0/3586752768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ef3f0 0x7f4b6c0eb1d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f4b5c006650 tx=0x7f4b5c009a80).stop 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.639 7f4b73c8b700 1 -- 172.21.2.117:0/3586752768 shutdown_connections 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.639 7f4b73c8b700 1 --2- 172.21.2.117:0/3586752768 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ef3f0 0x7f4b6c0eb1d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.639 7f4b73c8b700 1 -- 172.21.2.117:0/3586752768 >> 172.21.2.117:0/3586752768 conn(0x7f4b6c0ced60 msgr2=0x7f4b6c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.639 7f4b73c8b700 1 -- 172.21.2.117:0/3586752768 shutdown_connections 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.639 7f4b73c8b700 1 -- 172.21.2.117:0/3586752768 wait complete. 2021-02-10T11:44:21.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 Processor -- start 2021-02-10T11:44:21.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 -- start start 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ae5b0 0x7f4b6c0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b6c0e6190 0x7f4b6c0e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c08a5a0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c08a780 con 0x7f4b6c0ae5b0 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b73c8b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c08a9a0 con 0x7f4b6c0e6190 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b722a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b722a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50774/0 (socket says 172.21.2.117:50774) 2021-02-10T11:44:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.654 7f4b722a8700 1 -- 172.21.2.117:0/1833359813 learned_addr learned my addr 172.21.2.117:0/1833359813 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 -- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b6c0e6190 msgr2=0x7f4b6c0e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b6c0e6190 0x7f4b6c0e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 -- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ae5b0 msgr2=0x7f4b6c0ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ae5b0 0x7f4b6c0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 -- 172.21.2.117:0/1833359813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c08abc0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b722a8700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f4b64009940 tx=0x7f4b6400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b64000f40 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6400c2f0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b6400c750 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c08abc0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.655 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c08abc0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.656 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b64003080 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.656 7f4b627fc700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:21.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.656 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f4b64032bb0 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.656 7f4b71aa7700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:21.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.656 7f4b71aa7700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:21.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.687 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c0e5c50 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.689 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b6400be30 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.818 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b6c0e5c50 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.819 7f4b627fc700 1 -- 172.21.2.117:0/1833359813 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f4b64003060 con 0x7f4b6c0e6bb0 2021-02-10T11:44:21.811 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:21.811 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":46,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8436,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.125:6835/3920367910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3920367910},{"type":"v1","addr":"172.21.2.125:6835","nonce":3920367910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:43:08.674746","modified":"2021-02-10 11:44:21.166489","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2021-02-10T11:44:21.811 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 46 2021-02-10T11:44:21.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 msgr2=0x7f4b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 crc :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 msgr2=0x7f4b6c0e6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f4b64009940 tx=0x7f4b6400bcf0).stop 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 shutdown_connections 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b54018730 0x7f4b5401abc0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b6c0e6bb0 0x7f4b6c0e6fb0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:21.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b6c0e6190 0x7f4b6c0e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 --2- 172.21.2.117:0/1833359813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b6c0ae5b0 0x7f4b6c0ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:21.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 >> 172.21.2.117:0/1833359813 conn(0x7f4b6c1a3620 msgr2=0x7f4b6c0e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:21.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 shutdown_connections 2021-02-10T11:44:21.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:21.821 7f4b73c8b700 1 -- 172.21.2.117:0/1833359813 wait complete. 2021-02-10T11:44:21.821 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.245 7fbaad7f1700 1 Processor -- start 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.245 7fbaad7f1700 1 -- start start 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaad7f1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbaa80e7180 0x7fbaa80e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaad7f1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa80e7580 con 0x7fbaa81b07f0 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaad7f1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa80e77f0 con 0x7fbaa80e8390 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaad7f1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa80e5ce0 con 0x7fbaa80e7180 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa7574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbaa81b07f0 0x7fbaa80e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51028/0 (socket says 172.21.2.117:51028) 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa7574700 1 -- 172.21.2.117:0/18208295 learned_addr learned my addr 172.21.2.117:0/18208295 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa6d73700 1 --2- 172.21.2.117:0/18208295 >> v2:172.21.2.125:3300/0 conn(0x7fbaa80e7180 0x7fbaa80e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:22.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa6572700 1 -- 172.21.2.117:0/18208295 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076747830 0 0) 0x7fba90000ff0 con 0x7fbaa81b07f0 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa6d73700 1 -- 172.21.2.117:0/18208295 >> v1:172.21.2.125:6789/0 conn(0x7fbaa80e8390 legacy=0x7fbaa80e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa6d73700 1 -- 172.21.2.117:0/18208295 >> v1:172.21.2.117:6789/0 conn(0x7fbaa81b07f0 legacy=0x7fbaa80e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.246 7fbaa6d73700 1 -- 172.21.2.117:0/18208295 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa80e8f80 con 0x7fbaa80e7180 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaa6d73700 1 --2- 172.21.2.117:0/18208295 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa80e7180 0x7fbaa80e7d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fba98006650 tx=0x7fba98009a80).ready entity=mon.1 client_cookie=d90560effc2c4479 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaa6572700 1 -- 172.21.2.117:0/18208295 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9800c0c0 con 0x7fbaa80e7180 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaa6572700 1 -- 172.21.2.117:0/18208295 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9800c340 con 0x7fbaa80e7180 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaa6572700 1 -- 172.21.2.117:0/18208295 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba9800c860 con 0x7fbaa80e7180 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaad7f1700 1 -- 172.21.2.117:0/18208295 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa80e7180 msgr2=0x7fbaa80e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.247 7fbaad7f1700 1 --2- 172.21.2.117:0/18208295 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa80e7180 0x7fbaa80e7d70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fba98006650 tx=0x7fba98009a80).stop 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.248 7fbaad7f1700 1 -- 172.21.2.117:0/18208295 shutdown_connections 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.248 7fbaad7f1700 1 --2- 172.21.2.117:0/18208295 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa80e7180 0x7fbaa80e7d70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.248 7fbaad7f1700 1 -- 172.21.2.117:0/18208295 >> 172.21.2.117:0/18208295 conn(0x7fbaa80ce500 msgr2=0x7fbaa80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.248 7fbaad7f1700 1 -- 172.21.2.117:0/18208295 shutdown_connections 2021-02-10T11:44:22.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.248 7fbaad7f1700 1 -- 172.21.2.117:0/18208295 wait complete. 2021-02-10T11:44:22.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.262 7fbaad7f1700 1 Processor -- start 2021-02-10T11:44:22.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 -- start start 2021-02-10T11:44:22.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 0x7fbaa81ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:22.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:22.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa81ad310 0x7fbaa8108f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81abd50 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81095a0 con 0x7fbaa81ab920 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaad7f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81097c0 con 0x7fbaa81ad310 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaa6d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaa6d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50786/0 (socket says 172.21.2.117:50786) 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaa6d73700 1 -- 172.21.2.117:0/1835007362 learned_addr learned my addr 172.21.2.117:0/1835007362 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaa7574700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 0x7fbaa81ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.263 7fbaa7d75700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa81ad310 0x7fbaa8108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 -- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa81ad310 msgr2=0x7fbaa8108f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa81ad310 0x7fbaa8108f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 -- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 msgr2=0x7fbaa81ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 0x7fbaa81ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:22.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 -- 172.21.2.117:0/1835007362 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa81099e0 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa7574700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 0x7fbaa81ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaa6d73700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7fba9800b5c0 tx=0x7fba9800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba98006740 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba980009b0 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa81099e0 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fba98002030 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.264 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa81099e0 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.265 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fba98000c90 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.265 7fba9ffff700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 0x7fba8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.265 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fba98032970 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.265 7fbaa7574700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 0x7fba8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:22.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.265 7fbaa7574700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 0x7fba8c01abc0 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:22.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.296 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa8108a60 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.298 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fba980064d0 con 0x7fbaa81ac8f0 2021-02-10T11:44:22.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:22.426 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbaa80e77f0 con 0x7fbaa81ac8f0 2021-02-10T11:44:23.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.178 7fba9ffff700 1 -- 172.21.2.117:0/1835007362 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7fba98000c90 con 0x7fbaa81ac8f0 2021-02-10T11:44:23.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.198 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 msgr2=0x7fba8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:23.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.198 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 0x7fba8c01abc0 crc :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.198 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 msgr2=0x7fbaa81accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:23.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.198 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7fba9800b5c0 tx=0x7fba9800b960).stop 2021-02-10T11:44:23.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 shutdown_connections 2021-02-10T11:44:23.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fba8c0186c0 0x7fba8c01abc0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbaa81ac8f0 0x7fbaa81accf0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbaa81ad310 0x7fbaa8108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 --2- 172.21.2.117:0/1835007362 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbaa81ab920 0x7fbaa81ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 >> 172.21.2.117:0/1835007362 conn(0x7fbaa8097700 msgr2=0x7fbaa8096df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 shutdown_connections 2021-02-10T11:44:23.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.199 7fbaad7f1700 1 -- 172.21.2.117:0/1835007362 wait complete. 2021-02-10T11:44:23.199 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:44:23.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.621 7fabce873700 1 Processor -- start 2021-02-10T11:44:23.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 -- start start 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabc80e4e40 0x7fabc81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fabc8085450 0x7fabc8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc8085930 con 0x7fabc80e5240 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc80e5cc0 con 0x7fabc80e4e40 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabce873700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc8084840 con 0x7fabc8085450 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabc7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabc80e4e40 0x7fabc81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabc7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fabc80e4e40 0x7fabc81b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50790/0 (socket says 172.21.2.117:50790) 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.622 7fabc7fff700 1 -- 172.21.2.117:0/2418516037 learned_addr learned my addr 172.21.2.117:0/2418516037 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc7fff700 1 -- 172.21.2.117:0/2418516037 >> v2:172.21.2.125:3300/0 conn(0x7fabc8085450 msgr2=0x7fabc8086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:23.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc7fff700 1 --2- 172.21.2.117:0/2418516037 >> v2:172.21.2.125:3300/0 conn(0x7fabc8085450 0x7fabc8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc7fff700 1 -- 172.21.2.117:0/2418516037 >> v1:172.21.2.125:6789/0 conn(0x7fabc80e5240 legacy=0x7fabc8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc7fff700 1 -- 172.21.2.117:0/2418516037 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc81b0810 con 0x7fabc80e4e40 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc7fff700 1 --2- 172.21.2.117:0/2418516037 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80e4e40 0x7fabc81b01f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7fabb8006700 tx=0x7fabb8009b80).ready entity=mon.0 client_cookie=50b9aee35a2c51aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc6ffd700 1 -- 172.21.2.117:0/2418516037 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb800c0c0 con 0x7fabc80e4e40 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc6ffd700 1 -- 172.21.2.117:0/2418516037 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb800c440 con 0x7fabc80e4e40 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabc6ffd700 1 -- 172.21.2.117:0/2418516037 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb800c960 con 0x7fabc80e4e40 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabce873700 1 -- 172.21.2.117:0/2418516037 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80e4e40 msgr2=0x7fabc81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.623 7fabce873700 1 --2- 172.21.2.117:0/2418516037 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80e4e40 0x7fabc81b01f0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 rx=0x7fabb8006700 tx=0x7fabb8009b80).stop 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 -- 172.21.2.117:0/2418516037 shutdown_connections 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 --2- 172.21.2.117:0/2418516037 >> v2:172.21.2.125:3300/0 conn(0x7fabc8085450 0x7fabc8086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 --2- 172.21.2.117:0/2418516037 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc80e4e40 0x7fabc81b01f0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 -- 172.21.2.117:0/2418516037 >> 172.21.2.117:0/2418516037 conn(0x7fabc80ce4e0 msgr2=0x7fabc80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 -- 172.21.2.117:0/2418516037 shutdown_connections 2021-02-10T11:44:23.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.624 7fabce873700 1 -- 172.21.2.117:0/2418516037 wait complete. 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.638 7fabce873700 1 Processor -- start 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.638 7fabce873700 1 -- start start 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80a00e0 0x7fabc81ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc81ae8d0 0x7fabc809e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc809f540 con 0x7fabc8093070 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc8093470 con 0x7fabc80a00e0 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabce873700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc809ecd0 con 0x7fabc81ae8d0 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabc7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabc7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50794/0 (socket says 172.21.2.117:50794) 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabc7fff700 1 -- 172.21.2.117:0/3994382416 learned_addr learned my addr 172.21.2.117:0/3994382416 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:23.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabcce90700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc81ae8d0 0x7fabc809e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.639 7fabc77fe700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80a00e0 0x7fabc81ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 -- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc81ae8d0 msgr2=0x7fabc809e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc81ae8d0 0x7fabc809e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 -- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80a00e0 msgr2=0x7fabc81ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80a00e0 0x7fabc81ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 -- 172.21.2.117:0/3994382416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc809eef0 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc7fff700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7fabb800b580 tx=0x7fabb800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb800bd10 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb8002030 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabce873700 1 -- 172.21.2.117:0/3994382416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc809eef0 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fabb8002480 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.640 7fabce873700 1 -- 172.21.2.117:0/3994382416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc81b20a0 con 0x7fabc8093070 2021-02-10T11:44:23.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.641 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fabb8000c90 con 0x7fabc8093070 2021-02-10T11:44:23.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.641 7fabc4ff9700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:23.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.641 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fabb8032740 con 0x7fabc8093070 2021-02-10T11:44:23.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.641 7fabc77fe700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:23.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.641 7fabc77fe700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:23.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.672 7fabce873700 1 -- 172.21.2.117:0/3994382416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc810c2f0 con 0x7fabc8093070 2021-02-10T11:44:23.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.675 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fabb8030dc0 con 0x7fabc8093070 2021-02-10T11:44:23.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:23.799 7fabce873700 1 -- 172.21.2.117:0/3994382416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fabc8085930 con 0x7fabc8093070 2021-02-10T11:44:24.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.182 7fabc4ff9700 1 -- 172.21.2.117:0/3994382416 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v66) v1 ==== 185+0+0 (secure 0 0 0) 0x7fabb801f720 con 0x7fabc8093070 2021-02-10T11:44:24.209 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:44:24.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 msgr2=0x7faba801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 crc :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 msgr2=0x7fabc809fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7fabb800b580 tx=0x7fabb800b920).stop 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 shutdown_connections 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fabc81ae8d0 0x7fabc809e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fabc80a00e0 0x7fabc81ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faba80186c0 0x7faba801abc0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 --2- 172.21.2.117:0/3994382416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fabc8093070 0x7fabc809fac0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 >> 172.21.2.117:0/3994382416 conn(0x7fabc81ad6b0 msgr2=0x7fabc8190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 shutdown_connections 2021-02-10T11:44:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.219 7fabce873700 1 -- 172.21.2.117:0/3994382416 wait complete. 2021-02-10T11:44:24.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:44:24.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.640 7fd86cd72700 1 Processor -- start 2021-02-10T11:44:24.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd86cd72700 1 -- start start 2021-02-10T11:44:24.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd86cd72700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd8680d15d0 0x7fd8680e8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:24.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd86cd72700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8680e7570 con 0x7fd8681b07e0 2021-02-10T11:44:24.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd86cd72700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8680e77e0 con 0x7fd8680e7170 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd86cd72700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8680e5cd0 con 0x7fd8680d15d0 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd866312700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fd8680e7170 0x7fd8680e7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:43978/0 (socket says 172.21.2.117:43978) 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd866b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd8681b07e0 0x7fd8680e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51052/0 (socket says 172.21.2.117:51052) 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd866312700 1 -- 172.21.2.117:0/2304471824 learned_addr learned my addr 172.21.2.117:0/2304471824 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.641 7fd867314700 1 --2- 172.21.2.117:0/2304471824 >> v2:172.21.2.125:3300/0 conn(0x7fd8680d15d0 0x7fd8680e8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:24.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd867314700 1 -- 172.21.2.117:0/2304471824 >> v1:172.21.2.125:6789/0 conn(0x7fd8680e7170 legacy=0x7fd8680e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd867314700 1 -- 172.21.2.117:0/2304471824 >> v1:172.21.2.117:6789/0 conn(0x7fd8681b07e0 legacy=0x7fd8680e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd867314700 1 -- 172.21.2.117:0/2304471824 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680d19d0 con 0x7fd8680d15d0 2021-02-10T11:44:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd865b11700 1 -- 172.21.2.117:0/2304471824 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977704233 0 0) 0x7fd858000ce0 con 0x7fd8681b07e0 2021-02-10T11:44:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd867314700 1 --2- 172.21.2.117:0/2304471824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8680d15d0 0x7fd8680e8550 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd860006e50 tx=0x7fd860009df0).ready entity=mon.1 client_cookie=86c899cf1cb3b8a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd865b11700 1 -- 172.21.2.117:0/2304471824 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86000b510 con 0x7fd8680d15d0 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd865b11700 1 -- 172.21.2.117:0/2304471824 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860003030 con 0x7fd8680d15d0 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd865b11700 1 -- 172.21.2.117:0/2304471824 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86000b7d0 con 0x7fd8680d15d0 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd86cd72700 1 -- 172.21.2.117:0/2304471824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8680d15d0 msgr2=0x7fd8680e8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.642 7fd86cd72700 1 --2- 172.21.2.117:0/2304471824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8680d15d0 0x7fd8680e8550 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd860006e50 tx=0x7fd860009df0).stop 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.643 7fd86cd72700 1 -- 172.21.2.117:0/2304471824 shutdown_connections 2021-02-10T11:44:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.643 7fd86cd72700 1 --2- 172.21.2.117:0/2304471824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8680d15d0 0x7fd8680e8550 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.643 7fd86cd72700 1 -- 172.21.2.117:0/2304471824 >> 172.21.2.117:0/2304471824 conn(0x7fd8680ce4f0 msgr2=0x7fd8680e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.643 7fd86cd72700 1 -- 172.21.2.117:0/2304471824 shutdown_connections 2021-02-10T11:44:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.643 7fd86cd72700 1 -- 172.21.2.117:0/2304471824 wait complete. 2021-02-10T11:44:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 Processor -- start 2021-02-10T11:44:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 -- start start 2021-02-10T11:44:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 0x7fd86818fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd868097e30 0x7fd86810d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86810d160 con 0x7fd868098230 2021-02-10T11:44:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd868190190 con 0x7fd86810cc20 2021-02-10T11:44:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd86cd72700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd868097970 con 0x7fd868097e30 2021-02-10T11:44:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd867314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd867314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50806/0 (socket says 172.21.2.117:50806) 2021-02-10T11:44:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.658 7fd867314700 1 -- 172.21.2.117:0/1661419303 learned_addr learned my addr 172.21.2.117:0/1661419303 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd866b13700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 0x7fd86818fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd866312700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd868097e30 0x7fd86810d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 -- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd868097e30 msgr2=0x7fd86810d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd868097e30 0x7fd86810d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 -- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 msgr2=0x7fd86818fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 0x7fd86818fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 -- 172.21.2.117:0/1661419303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86810e320 con 0x7fd868098230 2021-02-10T11:44:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd866b13700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 0x7fd86818fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd867314700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7fd860006fa0 tx=0x7fd86000bd90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8600093d0 con 0x7fd868098230 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600082e0 con 0x7fd868098230 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86810e320 con 0x7fd868098230 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86810e320 con 0x7fd868098230 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.659 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd860008800 con 0x7fd868098230 2021-02-10T11:44:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.660 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8600084c0 con 0x7fd868098230 2021-02-10T11:44:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.660 7fd8577fe700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 0x7fd84c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.660 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5568+0+0 (secure 0 0 0) 0x7fd8600329e0 con 0x7fd868098230 2021-02-10T11:44:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.660 7fd866b13700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 0x7fd84c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.661 7fd866b13700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 0x7fd84c01ac10 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:24.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.691 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8681089f0 con 0x7fd868098230 2021-02-10T11:44:24.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.694 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8600080d0 con 0x7fd868098230 2021-02-10T11:44:24.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:24.817 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd8681089f0 con 0x7fd868098230 2021-02-10T11:44:25.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.187 7fd8577fe700 1 -- 172.21.2.117:0/1661419303 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v67) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd8600082a0 con 0x7fd868098230 2021-02-10T11:44:25.224 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:44:25.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 msgr2=0x7fd84c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 0x7fd84c01ac10 crc :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 msgr2=0x7fd86810dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7fd860006fa0 tx=0x7fd86000bd90).stop 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 shutdown_connections 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.234 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd84c018730 0x7fd84c01ac10 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd868098230 0x7fd86810dd00 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd868097e30 0x7fd86810d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 --2- 172.21.2.117:0/1661419303 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd86810cc20 0x7fd86818fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 >> 172.21.2.117:0/1661419303 conn(0x7fd86819b800 msgr2=0x7fd8680987c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 shutdown_connections 2021-02-10T11:44:25.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.235 7fd86cd72700 1 -- 172.21.2.117:0/1661419303 wait complete. 2021-02-10T11:44:25.235 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:44:25.235 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:44:25.235 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:44:25.235 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:44:25.235 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:44:25.236 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:44:25.236 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:44:25.236 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:44:25.241 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:44:25.241 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:44:25.242 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:44:25.242 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:44:25.283 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:44:25.284 DEBUG:teuthology.parallel:result is None 2021-02-10T11:44:25.284 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:44:25.284 DEBUG:teuthology.parallel:result is None 2021-02-10T11:44:25.285 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:44:25.688 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:44:25.692 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:44:25.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.706 7f7766ff9700 1 Processor -- start 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.706 7f7766ff9700 1 -- start start 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7766ff9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601b07f0 0x7f77600e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7766ff9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77600e7d70 0x7f77600e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7766ff9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77600e77f0 con 0x7f77600e8790 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7766ff9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77600e92f0 con 0x7f77600e7d70 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7766ff9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77600e54b0 con 0x7f77601b07f0 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7765616700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f77600e8790 0x7f77600e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51076/0 (socket says 172.21.2.117:51076) 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7765616700 1 -- 172.21.2.117:0/3303220527 learned_addr learned my addr 172.21.2.117:0/3303220527 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7757fff700 1 --2- 172.21.2.117:0/3303220527 >> v2:172.21.2.117:3300/0 conn(0x7f77600e7d70 0x7f77600e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876798034 0 0) 0x7f7758000d30 con 0x7f77600e8790 2021-02-10T11:44:25.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7748002fe0 con 0x7f77600e8790 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2168415429 0 0) 0x7f7758001270 con 0x7f77600e8790 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601b07f0 msgr2=0x7f77600e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77601b07f0 0x7f77600e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 >> v2:172.21.2.117:3300/0 conn(0x7f77600e7d70 msgr2=0x7f77600e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 --2- 172.21.2.117:0/3303220527 >> v2:172.21.2.117:3300/0 conn(0x7f77600e7d70 0x7f77600e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600e5ce0 con 0x7f77600e8790 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7758001980 con 0x7f77600e8790 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f7757fff700 1 --2- 172.21.2.117:0/3303220527 >> v2:172.21.2.117:3300/0 conn(0x7f77600e7d70 0x7f77600e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 -- 172.21.2.117:0/3303220527 >> v1:172.21.2.117:6789/0 conn(0x7f77600e8790 legacy=0x7f77600e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7748003a40 0x7f7748005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.707 7f77577fe700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7748006750 0x7f7748008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77480094f0 0x7f774800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77480502d0 con 0x7f7748006750 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77480504b0 con 0x7f77480094f0 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77480506d0 con 0x7f7748003a40 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7757fff700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7748006750 0x7f7748008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7748003a40 0x7f7748005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7765616700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77480094f0 0x7f774800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 -- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77480094f0 msgr2=0x7f774800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77480094f0 0x7f774800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 -- 172.21.2.117:0/3303220527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7748006750 msgr2=0x7f7748008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7748006750 0x7f7748008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 -- 172.21.2.117:0/3303220527 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7748050970 con 0x7f7748003a40 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7757fff700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7748006750 0x7f7748008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7764e15700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7748003a40 0x7f7748005eb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7750006650 tx=0x7f7750009a00).ready entity=mon.2 client_cookie=d159617ac0bb85c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77500009b0 con 0x7f7748003a40 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750006340 con 0x7f7748003a40 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f77577fe700 1 -- 172.21.2.117:0/3303220527 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775000c2b0 con 0x7f7748003a40 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7766ff9700 1 -- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7748003a40 msgr2=0x7f7748005eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7766ff9700 1 --2- 172.21.2.117:0/3303220527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7748003a40 0x7f7748005eb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7750006650 tx=0x7f7750009a00).stop 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.708 7f7765616700 1 -- 172.21.2.117:0/3303220527 reap_dead start 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.709 7f7766ff9700 1 -- 172.21.2.117:0/3303220527 shutdown_connections 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.709 7f7766ff9700 1 -- 172.21.2.117:0/3303220527 >> 172.21.2.117:0/3303220527 conn(0x7f77600ce500 msgr2=0x7f77600e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.709 7f7766ff9700 1 -- 172.21.2.117:0/3303220527 shutdown_connections 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.709 7f7766ff9700 1 -- 172.21.2.117:0/3303220527 wait complete. 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 Processor -- start 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 -- start start 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f776019fe70 0x7f77600b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7760093a10 0x7f7760093e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600b3830 con 0x7f77600b3090 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600b3490 con 0x7f776019fe70 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.724 7f7766ff9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77600b3db0 con 0x7f7760093a10 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50836/0 (socket says 172.21.2.117:50836) 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 -- 172.21.2.117:0/870347970 learned_addr learned my addr 172.21.2.117:0/870347970 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 -- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7760093a10 msgr2=0x7f7760093e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7760093a10 0x7f7760093e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 -- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f776019fe70 msgr2=0x7f77600b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f776019fe70 0x7f77600b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:25.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 -- 172.21.2.117:0/870347970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600b3fd0 con 0x7f77600b3090 2021-02-10T11:44:25.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7757fff700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f775c00b5e0 tx=0x7f775c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:25.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775c00bd70 con 0x7f77600b3090 2021-02-10T11:44:25.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c006740 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775c0022b0 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7766ff9700 1 -- 172.21.2.117:0/870347970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77600b3fd0 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.725 7f7766ff9700 1 -- 172.21.2.117:0/870347970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77600b3fd0 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.726 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f775c00a000 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.726 7f77557fa700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 0x7f774001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.726 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f775c032260 con 0x7f77600b3090 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.727 7f7764e15700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 0x7f774001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:25.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.728 7f7764e15700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 0x7f774001abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:25.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.756 7f7766ff9700 1 -- 172.21.2.117:0/870347970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776010c1a0 con 0x7f77600b3090 2021-02-10T11:44:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.759 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f775c026030 con 0x7f77600b3090 2021-02-10T11:44:25.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:25.898 7f7766ff9700 1 -- 172.21.2.117:0/870347970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f77600e77f0 con 0x7f77600b3090 2021-02-10T11:44:26.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.234 7f77557fa700 1 -- 172.21.2.117:0/870347970 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v68) v1 ==== 97+0+0 (secure 0 0 0) 0x7f775c026030 con 0x7f77600b3090 2021-02-10T11:44:26.256 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:44:26.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 msgr2=0x7f774001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 0x7f774001abc0 crc :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 msgr2=0x7f77600933f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7f775c00b5e0 tx=0x7f775c00b980).stop 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 shutdown_connections 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77400186c0 0x7f774001abc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77600b3090 0x7f77600933f0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7760093a10 0x7f7760093e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 --2- 172.21.2.117:0/870347970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f776019fe70 0x7f77600b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 >> 172.21.2.117:0/870347970 conn(0x7f77600a7260 msgr2=0x7f77600bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 shutdown_connections 2021-02-10T11:44:26.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.266 7f7766ff9700 1 -- 172.21.2.117:0/870347970 wait complete. 2021-02-10T11:44:26.266 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:44:26.266 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 Processor -- start 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 -- start start 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd15c0e4e40 0x7fd15c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c085450 0x7fd15c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c085930 con 0x7fd15c0e5240 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0e5cc0 con 0x7fd15c0e4e40 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.704 7fd1619a9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c084840 con 0x7fd15c085450 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd15c0e4e40 0x7fd15c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd15c0e4e40 0x7fd15c1b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50844/0 (socket says 172.21.2.117:50844) 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 -- 172.21.2.117:0/1698133071 learned_addr learned my addr 172.21.2.117:0/1698133071 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 -- 172.21.2.117:0/1698133071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c085450 msgr2=0x7fd15c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 --2- 172.21.2.117:0/1698133071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c085450 0x7fd15c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 -- 172.21.2.117:0/1698133071 >> v1:172.21.2.125:6789/0 conn(0x7fd15c0e5240 legacy=0x7fd15c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 -- 172.21.2.117:0/1698133071 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c1b0810 con 0x7fd15c0e4e40 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.705 7fd15b574700 1 --2- 172.21.2.117:0/1698133071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c0e4e40 0x7fd15c1b01f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7fd144006700 tx=0x7fd144009b80).ready entity=mon.0 client_cookie=fa862c93939c303f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd15a572700 1 -- 172.21.2.117:0/1698133071 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14400c0c0 con 0x7fd15c0e4e40 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd15a572700 1 -- 172.21.2.117:0/1698133071 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14400c400 con 0x7fd15c0e4e40 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd15a572700 1 -- 172.21.2.117:0/1698133071 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14400c920 con 0x7fd15c0e4e40 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 -- 172.21.2.117:0/1698133071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c0e4e40 msgr2=0x7fd15c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 --2- 172.21.2.117:0/1698133071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c0e4e40 0x7fd15c1b01f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 rx=0x7fd144006700 tx=0x7fd144009b80).stop 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 -- 172.21.2.117:0/1698133071 shutdown_connections 2021-02-10T11:44:26.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 --2- 172.21.2.117:0/1698133071 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c085450 0x7fd15c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 --2- 172.21.2.117:0/1698133071 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c0e4e40 0x7fd15c1b01f0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 -- 172.21.2.117:0/1698133071 >> 172.21.2.117:0/1698133071 conn(0x7fd15c0ce4e0 msgr2=0x7fd15c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 -- 172.21.2.117:0/1698133071 shutdown_connections 2021-02-10T11:44:26.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.706 7fd1619a9700 1 -- 172.21.2.117:0/1698133071 wait complete. 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 Processor -- start 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 -- start start 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c1893b0 0x7fd15c10c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd15c1897b0 0x7fd15c187fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1b2610 con 0x7fd15c10caa0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1b2830 con 0x7fd15c1897b0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd1619a9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0986a0 con 0x7fd15c1893b0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd15ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd15ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50850/0 (socket says 172.21.2.117:50850) 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.721 7fd15ad73700 1 -- 172.21.2.117:0/4147516111 learned_addr learned my addr 172.21.2.117:0/4147516111 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 -- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c1893b0 msgr2=0x7fd15c10c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c1893b0 0x7fd15c10c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 -- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd15c1897b0 msgr2=0x7fd15c187fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd15c1897b0 0x7fd15c187fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 -- 172.21.2.117:0/4147516111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0988c0 con 0x7fd15c10caa0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd15ad73700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fd14c00b5b0 tx=0x7fd14c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14c00bd40 con 0x7fd15c10caa0 2021-02-10T11:44:26.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c000e20 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd14c0054a0 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c0988c0 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.722 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c1888f0 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.723 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd14c006030 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.723 7fd153fff700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.723 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fd14c032390 con 0x7fd15c10caa0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.724 7fd15b574700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:26.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.724 7fd15b574700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 0x7fd14001abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:26.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.755 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c10c250 con 0x7fd15c10caa0 2021-02-10T11:44:26.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.757 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd14c0009b0 con 0x7fd15c10caa0 2021-02-10T11:44:26.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.886 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd15c085930 con 0x7fd15c10caa0 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.887 7fd153fff700 1 -- 172.21.2.117:0/4147516111 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fd14c00bd40 con 0x7fd15c10caa0 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.880 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.881 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.882 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:44:26.883 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.884 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:44:26.885 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.886 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:44:26.887 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.888 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:44:26.889 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:44:26.890 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:44:26.891 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:44:26.892 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:26.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 msgr2=0x7fd14001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 0x7fd14001abc0 crc :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 msgr2=0x7fd15c098010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7fd14c00b5b0 tx=0x7fd14c00b950).stop 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 shutdown_connections 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1400186c0 0x7fd14001abc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd15c10caa0 0x7fd15c098010 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd15c1897b0 0x7fd15c187fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 --2- 172.21.2.117:0/4147516111 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd15c1893b0 0x7fd15c10c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 >> 172.21.2.117:0/4147516111 conn(0x7fd15c199160 msgr2=0x7fd15c08d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 shutdown_connections 2021-02-10T11:44:26.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:26.889 7fd1619a9700 1 -- 172.21.2.117:0/4147516111 wait complete. 2021-02-10T11:44:26.895 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:27.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.311 7f367373c700 1 Processor -- start 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.311 7f367373c700 1 -- start start 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f367373c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f366c0ef3f0 0x7f366c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f367373c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c0ede60 con 0x7f366c0d18e0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f367373c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f366c0d1ce0 con 0x7f366c0eb7f0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f367373c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366c0ef7f0 con 0x7f366c0ef3f0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f3671558700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f366c0d18e0 0x7f366c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51104/0 (socket says 172.21.2.117:51104) 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f3671558700 1 -- 172.21.2.117:0/508744979 learned_addr learned my addr 172.21.2.117:0/508744979 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.312 7f3670d57700 1 --2- 172.21.2.117:0/508744979 >> v2:172.21.2.125:3300/0 conn(0x7f366c0ef3f0 0x7f366c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927862278 0 0) 0x7f36600010b0 con 0x7f366c0d18e0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3658002fe0 con 0x7f366c0d18e0 2021-02-10T11:44:27.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472245631 0 0) 0x7f3664000f90 con 0x7f366c0eb7f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36580042c0 con 0x7f366c0eb7f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f3670d57700 1 -- 172.21.2.117:0/508744979 >> v1:172.21.2.125:6789/0 conn(0x7f366c0eb7f0 legacy=0x7f366c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f3670d57700 1 -- 172.21.2.117:0/508744979 >> v1:172.21.2.117:6789/0 conn(0x7f366c0d18e0 legacy=0x7f366c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f3670d57700 1 -- 172.21.2.117:0/508744979 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c1bb180 con 0x7f366c0ef3f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f3670d57700 1 --2- 172.21.2.117:0/508744979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0ef3f0 0x7f366c0eb1d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3668006650 tx=0x7f3668009a80).ready entity=mon.1 client_cookie=9f9153867dc26799 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f366800c0c0 con 0x7f366c0ef3f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800c340 con 0x7f366c0ef3f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f365ffff700 1 -- 172.21.2.117:0/508744979 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f366800c860 con 0x7f366c0ef3f0 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f367373c700 1 -- 172.21.2.117:0/508744979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0ef3f0 msgr2=0x7f366c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.313 7f367373c700 1 --2- 172.21.2.117:0/508744979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0ef3f0 0x7f366c0eb1d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f3668006650 tx=0x7f3668009a80).stop 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.314 7f367373c700 1 -- 172.21.2.117:0/508744979 shutdown_connections 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.314 7f367373c700 1 --2- 172.21.2.117:0/508744979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0ef3f0 0x7f366c0eb1d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.314 7f367373c700 1 -- 172.21.2.117:0/508744979 >> 172.21.2.117:0/508744979 conn(0x7f366c0ced60 msgr2=0x7f366c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:27.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.314 7f367373c700 1 -- 172.21.2.117:0/508744979 shutdown_connections 2021-02-10T11:44:27.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.314 7f367373c700 1 -- 172.21.2.117:0/508744979 wait complete. 2021-02-10T11:44:27.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.328 7f367373c700 1 Processor -- start 2021-02-10T11:44:27.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.328 7f367373c700 1 -- start start 2021-02-10T11:44:27.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f366c0ae5b0 0x7f366c0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 0x7f366c0e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c08a5a0 con 0x7f366c0e6190 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c08a780 con 0x7f366c0e6bb0 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f367373c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366c08a9a0 con 0x7f366c0ae5b0 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50860/0 (socket says 172.21.2.117:50860) 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 -- 172.21.2.117:0/2708384419 learned_addr learned my addr 172.21.2.117:0/2708384419 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3671d59700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 0x7f366c0e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3671558700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f366c0ae5b0 0x7f366c0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 -- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f366c0ae5b0 msgr2=0x7f366c0ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f366c0ae5b0 0x7f366c0ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 -- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 msgr2=0x7f366c0e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 0x7f366c0e6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3670d57700 1 -- 172.21.2.117:0/2708384419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c08abc0 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.329 7f3671d59700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 0x7f366c0e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f3670d57700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f3668009630 tx=0x7f366800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36680093a0 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800bd70 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f367373c700 1 -- 172.21.2.117:0/2708384419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c08abc0 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f367373c700 1 -- 172.21.2.117:0/2708384419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c08b150 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3668005560 con 0x7f366c0e6190 2021-02-10T11:44:27.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.330 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f366800a000 con 0x7f366c0e6190 2021-02-10T11:44:27.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.331 7f365dffb700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 0x7f364c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.331 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f36680323a0 con 0x7f366c0e6190 2021-02-10T11:44:27.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.331 7f3671558700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 0x7f364c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.331 7f3671558700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 0x7f364c01abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.362 7f367373c700 1 -- 172.21.2.117:0/2708384419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366c0e5c50 con 0x7f366c0e6190 2021-02-10T11:44:27.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.364 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3668009340 con 0x7f366c0e6190 2021-02-10T11:44:27.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.487 7f367373c700 1 -- 172.21.2.117:0/2708384419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f366c0e5c50 con 0x7f366c0e6190 2021-02-10T11:44:27.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.488 7f365dffb700 1 -- 172.21.2.117:0/2708384419 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+11378 (secure 0 0 0) 0x7f366800a000 con 0x7f366c0e6190 2021-02-10T11:44:27.481 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:27.481 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":68,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:26.231579","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:27.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 -- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 msgr2=0x7f364c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 0x7f364c01abc0 crc :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 -- 172.21.2.117:0/2708384419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 msgr2=0x7f366c0e6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 rx=0x7f3668009630 tx=0x7f366800b980).stop 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 -- 172.21.2.117:0/2708384419 shutdown_connections 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f364c0186c0 0x7f364c01abc0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f366c0e6190 0x7f366c0e6590 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f366c0e6bb0 0x7f366c0e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 --2- 172.21.2.117:0/2708384419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f366c0ae5b0 0x7f366c0ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.489 7f367373c700 1 -- 172.21.2.117:0/2708384419 >> 172.21.2.117:0/2708384419 conn(0x7f366c1a3620 msgr2=0x7f366c0e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.490 7f367373c700 1 -- 172.21.2.117:0/2708384419 shutdown_connections 2021-02-10T11:44:27.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.490 7f367373c700 1 -- 172.21.2.117:0/2708384419 wait complete. 2021-02-10T11:44:27.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:44:27.491 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.912 7f4b4f9dd700 1 Processor -- start 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.912 7f4b4f9dd700 1 -- start start 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4f9dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480e7180 0x7f4b480e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4f9dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b480e8390 0x7f4b480e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4f9dd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480e7580 con 0x7f4b481b07f0 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4f9dd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b480e5ce0 con 0x7f4b480e8390 2021-02-10T11:44:27.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4f9dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480e77f0 con 0x7f4b480e7180 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4dffa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b480e8390 0x7f4b480e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4d7f9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4b481b07f0 0x7f4b480e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51118/0 (socket says 172.21.2.117:51118) 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.913 7f4b4d7f9700 1 -- 172.21.2.117:0/4076633766 learned_addr learned my addr 172.21.2.117:0/4076633766 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4dffa700 1 -- 172.21.2.117:0/4076633766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480e7180 msgr2=0x7f4b480e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4dffa700 1 --2- 172.21.2.117:0/4076633766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480e7180 0x7f4b480e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4dffa700 1 -- 172.21.2.117:0/4076633766 >> v1:172.21.2.117:6789/0 conn(0x7f4b481b07f0 legacy=0x7f4b480e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4dffa700 1 -- 172.21.2.117:0/4076633766 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480e8db0 con 0x7f4b480e8390 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4dffa700 1 --2- 172.21.2.117:0/4076633766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b480e8390 0x7f4b480e8790 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7f4b40006e30 tx=0x7f4b40009e60).ready entity=mon.0 client_cookie=edbb01076998e8a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b3bfff700 1 -- 172.21.2.117:0/4076633766 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b40006a50 con 0x7f4b480e8390 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b3bfff700 1 -- 172.21.2.117:0/4076633766 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4000b300 con 0x7f4b480e8390 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b3bfff700 1 -- 172.21.2.117:0/4076633766 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b4000b760 con 0x7f4b480e8390 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4f9dd700 1 -- 172.21.2.117:0/4076633766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b480e8390 msgr2=0x7f4b480e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.914 7f4b4f9dd700 1 --2- 172.21.2.117:0/4076633766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b480e8390 0x7f4b480e8790 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 rx=0x7f4b40006e30 tx=0x7f4b40009e60).stop 2021-02-10T11:44:27.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 -- 172.21.2.117:0/4076633766 shutdown_connections 2021-02-10T11:44:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 --2- 172.21.2.117:0/4076633766 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b480e8390 0x7f4b480e8790 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 --2- 172.21.2.117:0/4076633766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480e7180 0x7f4b480e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 -- 172.21.2.117:0/4076633766 >> 172.21.2.117:0/4076633766 conn(0x7f4b480ce500 msgr2=0x7f4b480e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 -- 172.21.2.117:0/4076633766 shutdown_connections 2021-02-10T11:44:27.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.915 7f4b4f9dd700 1 -- 172.21.2.117:0/4076633766 wait complete. 2021-02-10T11:44:27.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.929 7f4b4f9dd700 1 Processor -- start 2021-02-10T11:44:27.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 -- start start 2021-02-10T11:44:27.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480b99e0 0x7f4b480b9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b480ba400 0x7f4b48109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480a6ad0 con 0x7f4b481a3a40 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48109b60 con 0x7f4b480ba400 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4f9dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48109d80 con 0x7f4b480b99e0 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.930 7f4b4d7f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50874/0 (socket says 172.21.2.117:50874) 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 -- 172.21.2.117:0/2019374310 learned_addr learned my addr 172.21.2.117:0/2019374310 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4dffa700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b480ba400 0x7f4b48109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4cff8700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480b99e0 0x7f4b480b9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 -- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480b99e0 msgr2=0x7f4b480b9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480b99e0 0x7f4b480b9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 -- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b480ba400 msgr2=0x7f4b48109540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b480ba400 0x7f4b48109540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 -- 172.21.2.117:0/2019374310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48109fa0 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b4d7f9700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f4b3c001400 tx=0x7f4b3c00a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.931 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b3c00acf0 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c0009b0 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b3c004620 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b48109fa0 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4810ac10 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b3c0043a0 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b39ffb700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 0x7f4b2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.932 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4b3c032230 con 0x7f4b481a3a40 2021-02-10T11:44:27.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.933 7f4b4cff8700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 0x7f4b2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:27.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.933 7f4b4cff8700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 0x7f4b2801abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:27.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.963 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b481089c0 con 0x7f4b481a3a40 2021-02-10T11:44:27.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:27.966 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b3c014e20 con 0x7f4b481a3a40 2021-02-10T11:44:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.092 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4b480e77f0 con 0x7f4b481a3a40 2021-02-10T11:44:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.092 7f4b39ffb700 1 -- 172.21.2.117:0/2019374310 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v68) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4b3c014e20 con 0x7f4b481a3a40 2021-02-10T11:44:28.085 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 msgr2=0x7f4b2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 0x7f4b2801abc0 crc :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 msgr2=0x7f4b481a3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 rx=0x7f4b3c001400 tx=0x7f4b3c00a900).stop 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 shutdown_connections 2021-02-10T11:44:28.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b480ba400 0x7f4b48109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b480b99e0 0x7f4b480b9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b280186c0 0x7f4b2801abc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 --2- 172.21.2.117:0/2019374310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b481a3a40 0x7f4b481a3e40 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.094 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 >> 172.21.2.117:0/2019374310 conn(0x7f4b48108bf0 msgr2=0x7f4b4809ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.095 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 shutdown_connections 2021-02-10T11:44:28.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.095 7f4b4f9dd700 1 -- 172.21.2.117:0/2019374310 wait complete. 2021-02-10T11:44:28.096 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.513 7fc36a0ae700 1 Processor -- start 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.513 7fc36a0ae700 1 -- start start 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640d18e0 0x7fc3640ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3640eb000 0x7fc3640eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc3640ee3e0 0x7fc3640ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3640ede60 con 0x7fc3640eb000 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3640eba20 con 0x7fc3640ee3e0 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc36a0ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640d1ce0 con 0x7fc3640d18e0 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc362ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3640eb000 0x7fc3640eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc362ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3640eb000 0x7fc3640eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50882/0 (socket says 172.21.2.117:50882) 2021-02-10T11:44:28.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.514 7fc362ffd700 1 -- 172.21.2.117:0/2332741824 learned_addr learned my addr 172.21.2.117:0/2332741824 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 -- 172.21.2.117:0/2332741824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640d18e0 msgr2=0x7fc3640ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 --2- 172.21.2.117:0/2332741824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640d18e0 0x7fc3640ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 -- 172.21.2.117:0/2332741824 >> v2:172.21.2.125:3300/0 conn(0x7fc3640ee3e0 msgr2=0x7fc3640ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 --2- 172.21.2.117:0/2332741824 >> v2:172.21.2.125:3300/0 conn(0x7fc3640ee3e0 0x7fc3640ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 -- 172.21.2.117:0/2332741824 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640eee00 con 0x7fc3640eb000 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc362ffd700 1 --2- 172.21.2.117:0/2332741824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3640eb000 0x7fc3640eb400 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7fc358006700 tx=0x7fc358009b00).ready entity=mon.0 client_cookie=5e591c8cdf5fcf22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc3627fc700 1 -- 172.21.2.117:0/2332741824 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc35800c0c0 con 0x7fc3640eb000 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc3627fc700 1 -- 172.21.2.117:0/2332741824 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35800c400 con 0x7fc3640eb000 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc3627fc700 1 -- 172.21.2.117:0/2332741824 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc35800c860 con 0x7fc3640eb000 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc36a0ae700 1 -- 172.21.2.117:0/2332741824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3640eb000 msgr2=0x7fc3640eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.515 7fc36a0ae700 1 --2- 172.21.2.117:0/2332741824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3640eb000 0x7fc3640eb400 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 rx=0x7fc358006700 tx=0x7fc358009b00).stop 2021-02-10T11:44:28.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 -- 172.21.2.117:0/2332741824 shutdown_connections 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 --2- 172.21.2.117:0/2332741824 >> v2:172.21.2.125:3300/0 conn(0x7fc3640ee3e0 0x7fc3640ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 --2- 172.21.2.117:0/2332741824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3640eb000 0x7fc3640eb400 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 --2- 172.21.2.117:0/2332741824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640d18e0 0x7fc3640ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 -- 172.21.2.117:0/2332741824 >> 172.21.2.117:0/2332741824 conn(0x7fc3640ced60 msgr2=0x7fc3640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 -- 172.21.2.117:0/2332741824 shutdown_connections 2021-02-10T11:44:28.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.516 7fc36a0ae700 1 -- 172.21.2.117:0/2332741824 wait complete. 2021-02-10T11:44:28.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.530 7fc36a0ae700 1 Processor -- start 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 -- start start 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640e9130 0x7fc3640a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3640a89d0 0x7fc3641a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3640e95b0 con 0x7fc3641a6590 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc364192f60 con 0x7fc3640a89d0 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc36a0ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc364193180 con 0x7fc3640e9130 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc363fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc363fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50886/0 (socket says 172.21.2.117:50886) 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.531 7fc363fff700 1 -- 172.21.2.117:0/2645940896 learned_addr learned my addr 172.21.2.117:0/2645940896 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc362ffd700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3640a89d0 0x7fc3641a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc3637fe700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640e9130 0x7fc3640a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 -- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640e9130 msgr2=0x7fc3640a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640e9130 0x7fc3640a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 -- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3640a89d0 msgr2=0x7fc3641a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3640a89d0 0x7fc3641a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 -- 172.21.2.117:0/2645940896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3641933a0 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc363fff700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7fc35c009980 tx=0x7fc35c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc35c005870 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c005b30 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc35c006030 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3641933a0 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.532 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3641933a0 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.533 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc35c005d10 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.533 7fc34bfff700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 0x7fc34401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.533 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc35c032840 con 0x7fc3641a6590 2021-02-10T11:44:28.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.533 7fc3637fe700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 0x7fc34401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:28.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.534 7fc3637fe700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 0x7fc34401abd0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:28.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.564 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc364087210 con 0x7fc3641a6590 2021-02-10T11:44:28.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.566 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc35c00be10 con 0x7fc3641a6590 2021-02-10T11:44:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.689 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc364087210 con 0x7fc3641a6590 2021-02-10T11:44:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.689 7fc34bfff700 1 -- 172.21.2.117:0/2645940896 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) v1 ==== 84+0+64 (secure 0 0 0) 0x7fc35c006f20 con 0x7fc3641a6590 2021-02-10T11:44:28.683 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:28.683 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:44:28.683 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:28.683 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:44:28.683 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 msgr2=0x7fc34401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 0x7fc34401abd0 crc :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 msgr2=0x7fc364192940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 rx=0x7fc35c009980 tx=0x7fc35c00bcd0).stop 2021-02-10T11:44:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 shutdown_connections 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3440186d0 0x7fc34401abd0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3641a6590 0x7fc364192940 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3640a89d0 0x7fc3641a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 --2- 172.21.2.117:0/2645940896 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3640e9130 0x7fc3640a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 >> 172.21.2.117:0/2645940896 conn(0x7fc364087490 msgr2=0x7fc364195020 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 shutdown_connections 2021-02-10T11:44:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:28.691 7fc36a0ae700 1 -- 172.21.2.117:0/2645940896 wait complete. 2021-02-10T11:44:28.693 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:44:28.693 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.115 7f137c1ed700 1 Processor -- start 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.115 7f137c1ed700 1 -- start start 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137c1ed700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13740ef3f0 0x7f13740eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137c1ed700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740ef7f0 con 0x7f13740d18f0 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137c1ed700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740ede60 con 0x7f13740eb830 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137c1ed700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740d1da0 con 0x7f13740ef3f0 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f1379808700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13740ef3f0 0x7f13740eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137a009700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f13740d18f0 0x7f13740eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51142/0 (socket says 172.21.2.117:51142) 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f137a009700 1 -- 172.21.2.117:0/3033200845 learned_addr learned my addr 172.21.2.117:0/3033200845 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:29.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f1379808700 1 -- 172.21.2.117:0/3033200845 >> v1:172.21.2.125:6789/0 conn(0x7f13740eb830 legacy=0x7f13741b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.116 7f1379808700 1 -- 172.21.2.117:0/3033200845 >> v1:172.21.2.117:6789/0 conn(0x7f13740d18f0 legacy=0x7f13740eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f1379808700 1 -- 172.21.2.117:0/3033200845 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13741bb1c0 con 0x7f13740ef3f0 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f1379808700 1 --2- 172.21.2.117:0/3033200845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740ef3f0 0x7f13740eb210 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f1364006650 tx=0x7f1364009a80).ready entity=mon.0 client_cookie=624dd9e7c104207f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f1379007700 1 -- 172.21.2.117:0/3033200845 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13640009b0 con 0x7f13740ef3f0 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f1379007700 1 -- 172.21.2.117:0/3033200845 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136400c0c0 con 0x7f13740ef3f0 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f1379007700 1 -- 172.21.2.117:0/3033200845 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f136400c5e0 con 0x7f13740ef3f0 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f137c1ed700 1 -- 172.21.2.117:0/3033200845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740ef3f0 msgr2=0x7f13740eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.117 7f137c1ed700 1 --2- 172.21.2.117:0/3033200845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740ef3f0 0x7f13740eb210 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 rx=0x7f1364006650 tx=0x7f1364009a80).stop 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.118 7f137c1ed700 1 -- 172.21.2.117:0/3033200845 shutdown_connections 2021-02-10T11:44:29.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.118 7f137c1ed700 1 --2- 172.21.2.117:0/3033200845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740ef3f0 0x7f13740eb210 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.118 7f137c1ed700 1 -- 172.21.2.117:0/3033200845 >> 172.21.2.117:0/3033200845 conn(0x7f13740cf060 msgr2=0x7f13740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:29.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.118 7f137c1ed700 1 -- 172.21.2.117:0/3033200845 shutdown_connections 2021-02-10T11:44:29.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.118 7f137c1ed700 1 -- 172.21.2.117:0/3033200845 wait complete. 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.132 7f137c1ed700 1 Processor -- start 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 -- start start 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 0x7f13740e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 0x7f13741c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:29.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13740df000 con 0x7f13740a66b0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137409f8d0 con 0x7f13740e96c0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f137c1ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13741c5a30 con 0x7f13740a6ab0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f1379808700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f1379808700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50902/0 (socket says 172.21.2.117:50902) 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.133 7f1379808700 1 -- 172.21.2.117:0/233228718 learned_addr learned my addr 172.21.2.117:0/233228718 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137a80a700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 0x7f13741c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137a009700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 0x7f13740e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 -- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 msgr2=0x7f13741c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 0x7f13741c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 -- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 msgr2=0x7f13740e8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 0x7f13740e8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 -- 172.21.2.117:0/233228718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13741c5c10 con 0x7f13740a66b0 2021-02-10T11:44:29.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137a80a700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 0x7f13741c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137a009700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 0x7f13740e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f1379808700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f1364009630 tx=0x7f136400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13640093a0 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136400bd70 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1364005560 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137c1ed700 1 -- 172.21.2.117:0/233228718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13741c5c10 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.134 7f137c1ed700 1 -- 172.21.2.117:0/233228718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741c5c10 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.135 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f136400a000 con 0x7f13740a66b0 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.135 7f136affd700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:29.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.135 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f1364032370 con 0x7f13740a66b0 2021-02-10T11:44:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.135 7f137a009700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.136 7f137a009700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 0x7f135c01abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:29.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.166 7f137c1ed700 1 -- 172.21.2.117:0/233228718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1374087390 con 0x7f13740a66b0 2021-02-10T11:44:29.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.168 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1364030d00 con 0x7f13740a66b0 2021-02-10T11:44:29.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.299 7f137c1ed700 1 -- 172.21.2.117:0/233228718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f13740df000 con 0x7f13740a66b0 2021-02-10T11:44:29.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.703 7f136affd700 1 -- 172.21.2.117:0/233228718 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v69) v1 ==== 133+0+0 (secure 0 0 0) 0x7f136400a000 con 0x7f13740a66b0 2021-02-10T11:44:29.707 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:44:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 msgr2=0x7f135c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 0x7f135c01abc0 crc :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 msgr2=0x7f13741c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 rx=0x7f1364009630 tx=0x7f136400b980).stop 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 shutdown_connections 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f135c0186c0 0x7f135c01abc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13740a66b0 0x7f13741c4df0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13740a6ab0 0x7f13741c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 --2- 172.21.2.117:0/233228718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13740e96c0 0x7f13740e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 >> 172.21.2.117:0/233228718 conn(0x7f13740e71d0 msgr2=0x7f13740b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 shutdown_connections 2021-02-10T11:44:29.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:29.716 7f137c1ed700 1 -- 172.21.2.117:0/233228718 wait complete. 2021-02-10T11:44:29.717 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:44:30.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 Processor -- start 2021-02-10T11:44:30.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 -- start start 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a80d18f0 0x7f32a80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32a80eb040 0x7f32a80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f32a80ee3e0 0x7f32a80ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80ede60 con 0x7f32a80eb040 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a80eba60 con 0x7f32a80ee3e0 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.137 7f32b06b1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80d1cf0 con 0x7f32a80d18f0 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32a80eb040 0x7f32a80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32a80eb040 0x7f32a80eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50906/0 (socket says 172.21.2.117:50906) 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 -- 172.21.2.117:0/1921878592 learned_addr learned my addr 172.21.2.117:0/1921878592 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 -- 172.21.2.117:0/1921878592 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a80d18f0 msgr2=0x7f32a80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 --2- 172.21.2.117:0/1921878592 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a80d18f0 0x7f32a80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 -- 172.21.2.117:0/1921878592 >> v2:172.21.2.125:3300/0 conn(0x7f32a80ee3e0 msgr2=0x7f32a80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:30.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 --2- 172.21.2.117:0/1921878592 >> v2:172.21.2.125:3300/0 conn(0x7f32a80ee3e0 0x7f32a80ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 -- 172.21.2.117:0/1921878592 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80eee00 con 0x7f32a80eb040 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32adccc700 1 --2- 172.21.2.117:0/1921878592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a80eb040 0x7f32a80eb440 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f32a4006700 tx=0x7f32a4009b80).ready entity=mon.0 client_cookie=56a7c9ede8d68848 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32ad4cb700 1 -- 172.21.2.117:0/1921878592 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32a400c0c0 con 0x7f32a80eb040 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32ad4cb700 1 -- 172.21.2.117:0/1921878592 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a400c340 con 0x7f32a80eb040 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.138 7f32ad4cb700 1 -- 172.21.2.117:0/1921878592 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32a400c860 con 0x7f32a80eb040 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 -- 172.21.2.117:0/1921878592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a80eb040 msgr2=0x7f32a80eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 --2- 172.21.2.117:0/1921878592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a80eb040 0x7f32a80eb440 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 rx=0x7f32a4006700 tx=0x7f32a4009b80).stop 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 -- 172.21.2.117:0/1921878592 shutdown_connections 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 --2- 172.21.2.117:0/1921878592 >> v2:172.21.2.125:3300/0 conn(0x7f32a80ee3e0 0x7f32a80ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 --2- 172.21.2.117:0/1921878592 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a80eb040 0x7f32a80eb440 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 --2- 172.21.2.117:0/1921878592 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a80d18f0 0x7f32a80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 -- 172.21.2.117:0/1921878592 >> 172.21.2.117:0/1921878592 conn(0x7f32a80cf060 msgr2=0x7f32a80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 -- 172.21.2.117:0/1921878592 shutdown_connections 2021-02-10T11:44:30.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.139 7f32b06b1700 1 -- 172.21.2.117:0/1921878592 wait complete. 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 Processor -- start 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 -- start start 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a807c7c0 0x7f32a81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32a80a13a0 0x7f32a80a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a807cca0 con 0x7f32a81a3930 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a818fb70 con 0x7f32a80a13a0 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.154 7f32b06b1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a80dc6d0 con 0x7f32a807c7c0 2021-02-10T11:44:30.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50912/0 (socket says 172.21.2.117:50912) 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 -- 172.21.2.117:0/845715520 learned_addr learned my addr 172.21.2.117:0/845715520 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 -- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a807c7c0 msgr2=0x7f32a81a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a807c7c0 0x7f32a81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 -- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32a80a13a0 msgr2=0x7f32a80a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32a80a13a0 0x7f32a80a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 -- 172.21.2.117:0/845715520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a807d220 con 0x7f32a81a3930 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.155 7f32adccc700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f32a400b650 tx=0x7f32a400b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32a400bde0 con 0x7f32a81a3930 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a4006460 con 0x7f32a81a3930 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32a4002220 con 0x7f32a81a3930 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f32b06b1700 1 -- 172.21.2.117:0/845715520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a807d220 con 0x7f32a81a3930 2021-02-10T11:44:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f32b06b1700 1 -- 172.21.2.117:0/845715520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a807d220 con 0x7f32a81a3930 2021-02-10T11:44:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32a4000c90 con 0x7f32a81a3930 2021-02-10T11:44:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 0x7f329001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.156 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f32a4032530 con 0x7f32a81a3930 2021-02-10T11:44:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.157 7f32ae4cd700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 0x7f329001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.158 7f32ae4cd700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 0x7f329001abc0 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:30.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.187 7f32b06b1700 1 -- 172.21.2.117:0/845715520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a80daff0 con 0x7f32a81a3930 2021-02-10T11:44:30.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.190 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32a4030d60 con 0x7f32a81a3930 2021-02-10T11:44:30.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.318 7f32b06b1700 1 -- 172.21.2.117:0/845715520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f32a80daff0 con 0x7f32a81a3930 2021-02-10T11:44:30.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.711 7f329affd700 1 -- 172.21.2.117:0/845715520 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v71) v1 ==== 125+0+0 (secure 0 0 0) 0x7f32a4000c90 con 0x7f32a81a3930 2021-02-10T11:44:30.727 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 msgr2=0x7f329001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 0x7f329001abc0 crc :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 msgr2=0x7f32a80a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 rx=0x7f32a400b650 tx=0x7f32a400b9f0).stop 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 shutdown_connections 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32900186c0 0x7f329001abc0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32a81a3930 0x7f32a80a0d80 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32a80a13a0 0x7f32a80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 --2- 172.21.2.117:0/845715520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32a807c7c0 0x7f32a81a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 >> 172.21.2.117:0/845715520 conn(0x7f32a8087090 msgr2=0x7f32a808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 shutdown_connections 2021-02-10T11:44:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:30.736 7f32b06b1700 1 -- 172.21.2.117:0/845715520 wait complete. 2021-02-10T11:44:30.737 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.158 7f2cc7802700 1 Processor -- start 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc7802700 1 -- start start 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc7802700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2cc00e5240 0x7f2cc0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc7802700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc0085930 con 0x7f2cc00e4e40 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc7802700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc01b09e0 con 0x7f2cc0084c60 2021-02-10T11:44:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc7802700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00e5cc0 con 0x7f2cc00e5240 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc561e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2cc00e4e40 0x7f2cc01b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51172/0 (socket says 172.21.2.117:51172) 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc561e700 1 -- 172.21.2.117:0/3182180977 learned_addr learned my addr 172.21.2.117:0/3182180977 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.159 7f2cc4e1d700 1 --2- 172.21.2.117:0/3182180977 >> v2:172.21.2.125:3300/0 conn(0x7f2cc00e5240 0x7f2cc0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180152753 0 0) 0x7f2cbc0009b0 con 0x7f2cc00e4e40 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac002fe0 con 0x7f2cc00e4e40 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874345868 0 0) 0x7f2cb8001400 con 0x7f2cc0084c60 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac003210 con 0x7f2cc0084c60 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc4e1d700 1 -- 172.21.2.117:0/3182180977 >> v1:172.21.2.125:6789/0 conn(0x7f2cc0084c60 legacy=0x7f2cc0085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc4e1d700 1 -- 172.21.2.117:0/3182180977 >> v1:172.21.2.117:6789/0 conn(0x7f2cc00e4e40 legacy=0x7f2cc01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc4e1d700 1 -- 172.21.2.117:0/3182180977 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0084840 con 0x7f2cc00e5240 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc4e1d700 1 --2- 172.21.2.117:0/3182180977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00e5240 0x7f2cc0086ec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f2cb0006650 tx=0x7f2cb0009a80).ready entity=mon.1 client_cookie=ef76139ea42f8f14 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb000c0c0 con 0x7f2cc00e5240 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb000c340 con 0x7f2cc00e5240 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cb7fff700 1 -- 172.21.2.117:0/3182180977 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb000c860 con 0x7f2cc00e5240 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc7802700 1 -- 172.21.2.117:0/3182180977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00e5240 msgr2=0x7f2cc0086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.160 7f2cc7802700 1 --2- 172.21.2.117:0/3182180977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00e5240 0x7f2cc0086ec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f2cb0006650 tx=0x7f2cb0009a80).stop 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.161 7f2cc7802700 1 -- 172.21.2.117:0/3182180977 shutdown_connections 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.161 7f2cc7802700 1 --2- 172.21.2.117:0/3182180977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00e5240 0x7f2cc0086ec0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.161 7f2cc7802700 1 -- 172.21.2.117:0/3182180977 >> 172.21.2.117:0/3182180977 conn(0x7f2cc00ce4e0 msgr2=0x7f2cc00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.161 7f2cc7802700 1 -- 172.21.2.117:0/3182180977 shutdown_connections 2021-02-10T11:44:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.161 7f2cc7802700 1 -- 172.21.2.117:0/3182180977 wait complete. 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 Processor -- start 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 -- start start 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00973b0 0x7f2cc0097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc00ad1b0 0x7f2cc00ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc018d0c0 con 0x7f2cc00adbd0 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc009ae70 con 0x7f2cc00973b0 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc7802700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc009ba10 con 0x7f2cc00ad1b0 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc5e1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc5e1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50932/0 (socket says 172.21.2.117:50932) 2021-02-10T11:44:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.176 7f2cc5e1f700 1 -- 172.21.2.117:0/685703265 learned_addr learned my addr 172.21.2.117:0/685703265 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 -- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc00ad1b0 msgr2=0x7f2cc00ad5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc00ad1b0 0x7f2cc00ad5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 -- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00973b0 msgr2=0x7f2cc0097980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00973b0 0x7f2cc0097980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 -- 172.21.2.117:0/685703265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc009bbf0 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc5e1f700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f2cb800b8e0 tx=0x7f2cb800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb800c2f0 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb800c570 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb800ca50 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc7802700 1 -- 172.21.2.117:0/685703265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc009bbf0 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.177 7f2cc7802700 1 -- 172.21.2.117:0/685703265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc009c180 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.178 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cb800c750 con 0x7f2cc00adbd0 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.178 7f2cb5ffb700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 0x7f2ca001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.178 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f2cb8032a30 con 0x7f2cc00adbd0 2021-02-10T11:44:31.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.178 7f2cc561e700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 0x7f2ca001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.179 7f2cc561e700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 0x7f2ca001abc0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.209 7f2cc7802700 1 -- 172.21.2.117:0/685703265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0085930 con 0x7f2cc00adbd0 2021-02-10T11:44:31.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.211 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cb800cd10 con 0x7f2cc00adbd0 2021-02-10T11:44:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.338 7f2cc7802700 1 -- 172.21.2.117:0/685703265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f2cc0085930 con 0x7f2cc00adbd0 2021-02-10T11:44:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.342 7f2cb5ffb700 1 -- 172.21.2.117:0/685703265 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 14 and data pool 15 v50) v1 ==== 189+0+0 (secure 0 0 0) 0x7f2cb800c630 con 0x7f2cc00adbd0 2021-02-10T11:44:31.338 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 14 and data pool 15 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.347 7f2cc7802700 1 -- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 msgr2=0x7f2ca001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.347 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 0x7f2ca001abc0 crc :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.347 7f2cc7802700 1 -- 172.21.2.117:0/685703265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 msgr2=0x7f2cc009b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.347 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 rx=0x7f2cb800b8e0 tx=0x7f2cb800bc80).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 -- 172.21.2.117:0/685703265 shutdown_connections 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ca00186c0 0x7f2ca001abc0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2cc00adbd0 0x7f2cc009b3f0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2cc00ad1b0 0x7f2cc00ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 --2- 172.21.2.117:0/685703265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2cc00973b0 0x7f2cc0097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 -- 172.21.2.117:0/685703265 >> 172.21.2.117:0/685703265 conn(0x7f2cc018cc70 msgr2=0x7f2cc0093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 -- 172.21.2.117:0/685703265 shutdown_connections 2021-02-10T11:44:31.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.348 7f2cc7802700 1 -- 172.21.2.117:0/685703265 wait complete. 2021-02-10T11:44:31.349 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:31.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.766 7f2e7cdf2700 1 Processor -- start 2021-02-10T11:44:31.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.766 7f2e7cdf2700 1 -- start start 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e7cdf2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e781b6960 0x7f2e781b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e7cdf2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780ede60 con 0x7f2e780d18e0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e7cdf2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780eb9c0 con 0x7f2e780ef3f0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e7cdf2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780d1ce0 con 0x7f2e781b6960 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e76b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2e780d18e0 0x7f2e780ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51186/0 (socket says 172.21.2.117:51186) 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e76b13700 1 -- 172.21.2.117:0/3341622491 learned_addr learned my addr 172.21.2.117:0/3341622491 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252415469 0 0) 0x7f2e60001180 con 0x7f2e780d18e0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e64002fe0 con 0x7f2e780d18e0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e77314700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e781b6960 0x7f2e781b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2786421493 0 0) 0x7f2e60001180 con 0x7f2e780d18e0 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.767 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e781b6960 msgr2=0x7f2e781b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e781b6960 0x7f2e781b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 >> v1:172.21.2.125:6789/0 conn(0x7f2e780ef3f0 legacy=0x7f2e780eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e781bb3b0 con 0x7f2e780d18e0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2e60001680 con 0x7f2e780d18e0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 >> v1:172.21.2.117:6789/0 conn(0x7f2e780d18e0 legacy=0x7f2e780ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e64003b90 0x7f2e64006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e640067b0 0x7f2e64008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e64009540 0x7f2e6400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6400c2d0 con 0x7f2e64009540 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6400c4f0 con 0x7f2e64003b90 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6400c710 con 0x7f2e640067b0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e64009540 0x7f2e6400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e76312700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e640067b0 0x7f2e64008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e76b13700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e64003b90 0x7f2e64006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 -- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e640067b0 msgr2=0x7f2e64008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e640067b0 0x7f2e64008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 -- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e64003b90 msgr2=0x7f2e64006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e64003b90 0x7f2e64006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 -- 172.21.2.117:0/3341622491 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e64050980 con 0x7f2e64009540 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e64009540 0x7f2e6400ba20 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7f2e70009b10 tx=0x7f2e70009eb0).ready entity=mon.0 client_cookie=13e3b728c68d42e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e7000b510 con 0x7f2e64009540 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70003030 con 0x7f2e64009540 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e75b11700 1 -- 172.21.2.117:0/3341622491 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e7000b790 con 0x7f2e64009540 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e7cdf2700 1 -- 172.21.2.117:0/3341622491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e64009540 msgr2=0x7f2e6400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e7cdf2700 1 --2- 172.21.2.117:0/3341622491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e64009540 0x7f2e6400ba20 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 rx=0x7f2e70009b10 tx=0x7f2e70009eb0).stop 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.768 7f2e77314700 1 -- 172.21.2.117:0/3341622491 reap_dead start 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.769 7f2e7cdf2700 1 -- 172.21.2.117:0/3341622491 shutdown_connections 2021-02-10T11:44:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.769 7f2e7cdf2700 1 -- 172.21.2.117:0/3341622491 >> 172.21.2.117:0/3341622491 conn(0x7f2e780ced60 msgr2=0x7f2e780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.769 7f2e7cdf2700 1 -- 172.21.2.117:0/3341622491 shutdown_connections 2021-02-10T11:44:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.769 7f2e7cdf2700 1 -- 172.21.2.117:0/3341622491 wait complete. 2021-02-10T11:44:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 Processor -- start 2021-02-10T11:44:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 -- start start 2021-02-10T11:44:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 0x7f2e7808c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 0x7f2e781a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780b61a0 con 0x7f2e781a3480 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7808c8b0 con 0x7f2e780b5da0 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e7cdf2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780ba1c0 con 0x7f2e780b59a0 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e77314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e77314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50948/0 (socket says 172.21.2.117:50948) 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e77314700 1 -- 172.21.2.117:0/1501968443 learned_addr learned my addr 172.21.2.117:0/1501968443 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.784 7f2e76312700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 0x7f2e781a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e76b13700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 0x7f2e7808c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 -- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 msgr2=0x7f2e7808c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 0x7f2e7808c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 -- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 msgr2=0x7f2e781a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 0x7f2e781a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 -- 172.21.2.117:0/1501968443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e781a3880 con 0x7f2e781a3480 2021-02-10T11:44:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e76312700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 0x7f2e781a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e76b13700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 0x7f2e7808c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e77314700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f2e7000b940 tx=0x7f2e7000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e70000df0 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e700011a0 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e70001600 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e781a3880 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.785 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e781a3880 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.786 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e70000df0 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.786 7f2e6f7fe700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 0x7f2e5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.786 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2e70032c90 con 0x7f2e781a3480 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.786 7f2e76b13700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 0x7f2e5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.787 7f2e76b13700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 0x7f2e5c01abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.816 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e780ede60 con 0x7f2e781a3480 2021-02-10T11:44:31.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.819 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e700011a0 con 0x7f2e781a3480 2021-02-10T11:44:31.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.943 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e780ede60 con 0x7f2e781a3480 2021-02-10T11:44:31.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.944 7f2e6f7fe700 1 -- 172.21.2.117:0/1501968443 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f2e70000df0 con 0x7f2e781a3480 2021-02-10T11:44:31.937 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:31.937 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":72,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:31.340659","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:29.317633","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:30.338240","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 msgr2=0x7f2e5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 0x7f2e5c01abc0 crc :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 msgr2=0x7f2e780b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f2e7000b940 tx=0x7f2e7000bce0).stop 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 shutdown_connections 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e5c0186c0 0x7f2e5c01abc0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e781a3480 0x7f2e780b2590 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e780b5da0 0x7f2e781a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 --2- 172.21.2.117:0/1501968443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e780b59a0 0x7f2e7808c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:31.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 >> 172.21.2.117:0/1501968443 conn(0x7f2e780bb5e0 msgr2=0x7f2e78090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:31.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 shutdown_connections 2021-02-10T11:44:31.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:31.945 7f2e7cdf2700 1 -- 172.21.2.117:0/1501968443 wait complete. 2021-02-10T11:44:31.947 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.364 7f28fb334700 1 Processor -- start 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.364 7f28fb334700 1 -- start start 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28fb334700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28f40d18e0 0x7f28f40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28fb334700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40ef3f0 0x7f28f41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28fb334700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f40ede60 con 0x7f28f40eb000 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28fb334700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f40ef7f0 con 0x7f28f40d18e0 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28fb334700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40d1ce0 con 0x7f28f40ef3f0 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9150700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28f40d18e0 0x7f28f40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f894f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f28f40eb000 0x7f28f40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51206/0 (socket says 172.21.2.117:51206) 2021-02-10T11:44:32.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f894f700 1 -- 172.21.2.117:0/1288414517 learned_addr learned my addr 172.21.2.117:0/1288414517 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9951700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40ef3f0 0x7f28f41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28ebfff700 1 -- 172.21.2.117:0/1288414517 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662214607 0 0) 0x7f28e40010b0 con 0x7f28f40eb000 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28ebfff700 1 -- 172.21.2.117:0/1288414517 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e0002fe0 con 0x7f28f40eb000 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9150700 1 -- 172.21.2.117:0/1288414517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40ef3f0 msgr2=0x7f28f41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9150700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40ef3f0 0x7f28f41b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9150700 1 -- 172.21.2.117:0/1288414517 >> v1:172.21.2.117:6789/0 conn(0x7f28f40eb000 legacy=0x7f28f40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.365 7f28f9150700 1 -- 172.21.2.117:0/1288414517 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f41b8f90 con 0x7f28f40d18e0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28f9150700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40d18e0 0x7f28f40ea9e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f28f0006650 tx=0x7f28f0009a80).ready entity=mon.0 client_cookie=63386c2a0daa7029 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28ebfff700 1 -- 172.21.2.117:0/1288414517 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f000c0c0 con 0x7f28f40d18e0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28ebfff700 1 -- 172.21.2.117:0/1288414517 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f000c340 con 0x7f28f40d18e0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28ebfff700 1 -- 172.21.2.117:0/1288414517 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28f000c820 con 0x7f28f40d18e0 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28fb334700 1 -- 172.21.2.117:0/1288414517 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40d18e0 msgr2=0x7f28f40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.366 7f28fb334700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40d18e0 0x7f28f40ea9e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 rx=0x7f28f0006650 tx=0x7f28f0009a80).stop 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 -- 172.21.2.117:0/1288414517 shutdown_connections 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40ef3f0 0x7f28f41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 --2- 172.21.2.117:0/1288414517 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40d18e0 0x7f28f40ea9e0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 -- 172.21.2.117:0/1288414517 >> 172.21.2.117:0/1288414517 conn(0x7f28f40ced60 msgr2=0x7f28f40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 -- 172.21.2.117:0/1288414517 shutdown_connections 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.367 7f28fb334700 1 -- 172.21.2.117:0/1288414517 wait complete. 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.381 7f28fb334700 1 Processor -- start 2021-02-10T11:44:32.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.381 7f28fb334700 1 -- start start 2021-02-10T11:44:32.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28f41a33d0 0x7f28f41a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 0x7f28f40b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40c0670 con 0x7f28f40b82f0 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40deb20 con 0x7f28f41a33d0 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28fb334700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f40ded40 con 0x7f28f40b8d10 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f894f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f894f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50962/0 (socket says 172.21.2.117:50962) 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f894f700 1 -- 172.21.2.117:0/1561934602 learned_addr learned my addr 172.21.2.117:0/1561934602 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 0x7f28f40b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9150700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28f41a33d0 0x7f28f41a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 -- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28f41a33d0 msgr2=0x7f28f41a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28f41a33d0 0x7f28f41a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 -- 172.21.2.117:0/1561934602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 msgr2=0x7f28f40b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 -- 172.21.2.117:0/1561934602 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40def60 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f894f700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.382 7f28f9951700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 0x7f28f40b9110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f28ec00b8a0 tx=0x7f28ec00bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec000f90 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec00bd80 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28ec001470 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28fb334700 1 -- 172.21.2.117:0/1561934602 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40def60 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28fb334700 1 -- 172.21.2.117:0/1561934602 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f40def60 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28ec0056b0 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 0x7f28d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28ec0327e0 con 0x7f28f40b8d10 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.383 7f28f9150700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 0x7f28d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.384 7f28f9150700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 0x7f28d401abc0 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.415 7f28fb334700 1 -- 172.21.2.117:0/1561934602 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f40c0670 con 0x7f28f40b8d10 2021-02-10T11:44:32.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.418 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28ec006710 con 0x7f28f40b8d10 2021-02-10T11:44:32.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.546 7f28fb334700 1 -- 172.21.2.117:0/1561934602 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28f40c0670 con 0x7f28f40b8d10 2021-02-10T11:44:32.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.548 7f28e9ffb700 1 -- 172.21.2.117:0/1561934602 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f28ec000df0 con 0x7f28f40b8d10 2021-02-10T11:44:32.542 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:32.543 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":73,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:32.342643","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:29.317633","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:30.338240","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 msgr2=0x7f28d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 0x7f28d401abc0 crc :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 msgr2=0x7f28f40b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 0x7f28f40b9110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f28ec00b8a0 tx=0x7f28ec00bc40).stop 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 shutdown_connections 2021-02-10T11:44:32.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28d40186c0 0x7f28d401abc0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28f40b82f0 0x7f28f40b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28f40b8d10 0x7f28f40b9110 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 --2- 172.21.2.117:0/1561934602 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28f41a33d0 0x7f28f41a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 >> 172.21.2.117:0/1561934602 conn(0x7f28f40874c0 msgr2=0x7f28f4096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 shutdown_connections 2021-02-10T11:44:32.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.551 7f28fb334700 1 -- 172.21.2.117:0/1561934602 wait complete. 2021-02-10T11:44:32.553 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:44:32.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 Processor -- start 2021-02-10T11:44:32.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 -- start start 2021-02-10T11:44:32.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff7040ef3f0 0x7ff7040eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040eb830 0x7ff7041b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040ef7f0 con 0x7ff7040d18f0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7040eded0 con 0x7ff7040ef3f0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.973 7ff70c6ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040d1cf0 con 0x7ff7040eb830 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff7040ef3f0 0x7ff7040eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff70a4ca700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff7040d18f0 0x7ff7040eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51218/0 (socket says 172.21.2.117:51218) 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff70a4ca700 1 -- 172.21.2.117:0/135084469 learned_addr learned my addr 172.21.2.117:0/135084469 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff70accb700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040eb830 0x7ff7041b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 -- 172.21.2.117:0/135084469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040eb830 msgr2=0x7ff7041b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040eb830 0x7ff7041b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 -- 172.21.2.117:0/135084469 >> v1:172.21.2.117:6789/0 conn(0x7ff7040d18f0 legacy=0x7ff7040eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 -- 172.21.2.117:0/135084469 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7041b8fd0 con 0x7ff7040ef3f0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff709cc9700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7040ef3f0 0x7ff7040eb210 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7ff700006650 tx=0x7ff700009a80).ready entity=mon.0 client_cookie=d6008db3a4af3c63 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff7094c8700 1 -- 172.21.2.117:0/135084469 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7000009b0 con 0x7ff7040ef3f0 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff7094c8700 1 -- 172.21.2.117:0/135084469 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000c0c0 con 0x7ff7040ef3f0 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff7094c8700 1 -- 172.21.2.117:0/135084469 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff70000c510 con 0x7ff7040ef3f0 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff70c6ae700 1 -- 172.21.2.117:0/135084469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7040ef3f0 msgr2=0x7ff7040eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.974 7ff70c6ae700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7040ef3f0 0x7ff7040eb210 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 rx=0x7ff700006650 tx=0x7ff700009a80).stop 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 -- 172.21.2.117:0/135084469 shutdown_connections 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040eb830 0x7ff7041b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 --2- 172.21.2.117:0/135084469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7040ef3f0 0x7ff7040eb210 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 -- 172.21.2.117:0/135084469 >> 172.21.2.117:0/135084469 conn(0x7ff7040cf060 msgr2=0x7ff7040cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 -- 172.21.2.117:0/135084469 shutdown_connections 2021-02-10T11:44:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.975 7ff70c6ae700 1 -- 172.21.2.117:0/135084469 wait complete. 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 Processor -- start 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 -- start start 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040a6ce0 0x7ff7040dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7040a2040 0x7ff7040a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040a67e0 con 0x7ff704085ec0 2021-02-10T11:44:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040a7120 con 0x7ff7040a2040 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70c6ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7040a2a60 con 0x7ff7040a6ce0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff709cc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff709cc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50976/0 (socket says 172.21.2.117:50976) 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff709cc9700 1 -- 172.21.2.117:0/2075939505 learned_addr learned my addr 172.21.2.117:0/2075939505 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70a4ca700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040a6ce0 0x7ff7040dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.990 7ff70accb700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7040a2040 0x7ff7040a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 -- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040a6ce0 msgr2=0x7ff7040dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040a6ce0 0x7ff7040dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 -- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7040a2040 msgr2=0x7ff7040a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7040a2040 0x7ff7040a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 -- 172.21.2.117:0/2075939505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7041a2dc0 con 0x7ff704085ec0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff709cc9700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7ff700009650 tx=0x7ff70000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7000093a0 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000bd50 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7041a2dc0 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7000052e0 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.991 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7041a2dc0 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.992 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff700006030 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.992 7ff6f6ffd700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 0x7ff6ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.992 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff700032740 con 0x7ff704085ec0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.992 7ff70a4ca700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 0x7ff6ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:32.992 7ff70a4ca700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 0x7ff6ec01abc0 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:33.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.023 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7040daff0 con 0x7ff704085ec0 2021-02-10T11:44:33.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.025 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7000009c0 con 0x7ff704085ec0 2021-02-10T11:44:33.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.162 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7ff7040a70e0 con 0x7ff704085ec0 2021-02-10T11:44:33.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.349 7ff6f6ffd700 1 -- 172.21.2.117:0/2075939505 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v53) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff700006030 con 0x7ff704085ec0 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 msgr2=0x7ff6ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 0x7ff6ec01abc0 crc :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 msgr2=0x7ff7040862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 rx=0x7ff700009650 tx=0x7ff70000b960).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 shutdown_connections 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6ec0186c0 0x7ff6ec01abc0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff704085ec0 0x7ff7040862c0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7040a2040 0x7ff7040a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 --2- 172.21.2.117:0/2075939505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7040a6ce0 0x7ff7040dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 >> 172.21.2.117:0/2075939505 conn(0x7ff7040b6540 msgr2=0x7ff70408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 shutdown_connections 2021-02-10T11:44:33.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.379 7ff70c6ae700 1 -- 172.21.2.117:0/2075939505 wait complete. 2021-02-10T11:44:33.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:33.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 Processor -- start 2021-02-10T11:44:33.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 -- start start 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 0x7ff89c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89c0ede60 con 0x7ff89c0eb000 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.798 7ff8a2f78700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89c0ef7f0 con 0x7ff89c0d18e0 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a2f78700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c0d1ce0 con 0x7ff89c0ef3f0 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50982/0 (socket says 172.21.2.117:50982) 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 -- 172.21.2.117:0/1467391918 learned_addr learned my addr 172.21.2.117:0/1467391918 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a1595700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 0x7ff89c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 -- 172.21.2.117:0/1467391918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 msgr2=0x7ff89c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 0x7ff89c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 -- 172.21.2.117:0/1467391918 >> v1:172.21.2.125:6789/0 conn(0x7ff89c0eb000 legacy=0x7ff89c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 -- 172.21.2.117:0/1467391918 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c1b8f90 con 0x7ff89c0d18e0 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a1595700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 0x7ff89c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.799 7ff8a0d94700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7ff884006700 tx=0x7ff884009b80).ready entity=mon.0 client_cookie=ad5eee2fb155e741 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff89b7fe700 1 -- 172.21.2.117:0/1467391918 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff88400c0c0 con 0x7ff89c0d18e0 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff89b7fe700 1 -- 172.21.2.117:0/1467391918 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88400c400 con 0x7ff89c0d18e0 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff89b7fe700 1 -- 172.21.2.117:0/1467391918 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff88400c920 con 0x7ff89c0d18e0 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 -- 172.21.2.117:0/1467391918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c0d18e0 msgr2=0x7ff89c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 rx=0x7ff884006700 tx=0x7ff884009b80).stop 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 -- 172.21.2.117:0/1467391918 shutdown_connections 2021-02-10T11:44:33.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c0ef3f0 0x7ff89c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 --2- 172.21.2.117:0/1467391918 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c0d18e0 0x7ff89c0ea9e0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 -- 172.21.2.117:0/1467391918 >> 172.21.2.117:0/1467391918 conn(0x7ff89c0ced60 msgr2=0x7ff89c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:33.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 -- 172.21.2.117:0/1467391918 shutdown_connections 2021-02-10T11:44:33.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.800 7ff8a2f78700 1 -- 172.21.2.117:0/1467391918 wait complete. 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.815 7ff8a2f78700 1 Processor -- start 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.815 7ff8a2f78700 1 -- start start 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 0x7ff89c0b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 0x7ff89c0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c0e80e0 con 0x7ff89c1b0130 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c0e7e20 con 0x7ff89c0b43f0 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a2f78700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c096a70 con 0x7ff89c1afd30 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:50990/0 (socket says 172.21.2.117:50990) 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 -- 172.21.2.117:0/3549141824 learned_addr learned my addr 172.21.2.117:0/3549141824 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:33.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a0d94700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 0x7ff89c0b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 -- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 msgr2=0x7ff89c0b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff89bfff700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 0x7ff89c0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 0x7ff89c0b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 -- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 msgr2=0x7ff89c0e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 0x7ff89c0e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a1595700 1 -- 172.21.2.117:0/3549141824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c0e8660 con 0x7ff89c1b0130 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff8a0d94700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 0x7ff89c0b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.816 7ff89bfff700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 0x7ff89c0e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8a1595700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7ff894006f10 tx=0x7ff89400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff894000c10 con 0x7ff89c1b0130 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8940063b0 con 0x7ff89c1b0130 2021-02-10T11:44:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff894002280 con 0x7ff89c1b0130 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c0e8660 con 0x7ff89c1b0130 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.817 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c0e8bf0 con 0x7ff89c1b0130 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.818 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff894002070 con 0x7ff89c1b0130 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.818 7ff8997fa700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 0x7ff88001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.818 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8940326c0 con 0x7ff89c1b0130 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.819 7ff8a0d94700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 0x7ff88001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:33.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.819 7ff8a0d94700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 0x7ff88001abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:33.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.848 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89c087290 con 0x7ff89c1b0130 2021-02-10T11:44:33.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.851 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff894000c10 con 0x7ff89c1b0130 2021-02-10T11:44:33.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.978 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff89c0ede60 con 0x7ff89c1b0130 2021-02-10T11:44:33.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.979 7ff8997fa700 1 -- 172.21.2.117:0/3549141824 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2095 (secure 0 0 0) 0x7ff89400bd80 con 0x7ff89c1b0130 2021-02-10T11:44:33.972 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:33.972 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:44:33.346284","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/319870470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":319870470},{"type":"v1","addr":"172.21.2.125:6835","nonce":319870470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:44:33.973 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 53 2021-02-10T11:44:33.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 msgr2=0x7ff88001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 0x7ff88001abc0 crc :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 msgr2=0x7ff89c0e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 rx=0x7ff894006f10 tx=0x7ff89400b990).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 shutdown_connections 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8800186c0 0x7ff88001abc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89c1b0130 0x7ff89c0e7800 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89c0b43f0 0x7ff89c0e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 --2- 172.21.2.117:0/3549141824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff89c1afd30 0x7ff89c0b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:33.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 >> 172.21.2.117:0/3549141824 conn(0x7ff89c1a3620 msgr2=0x7ff89c0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:33.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 shutdown_connections 2021-02-10T11:44:33.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:33.981 7ff8a2f78700 1 -- 172.21.2.117:0/3549141824 wait complete. 2021-02-10T11:44:33.982 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:34.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 Processor -- start 2021-02-10T11:44:34.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 -- start start 2021-02-10T11:44:34.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57480d18e0 0x7f57480ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f57480eb000 0x7f57480eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480ede60 con 0x7f57480ee3e0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57480eba20 con 0x7f57480eb000 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574d187700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480d1ce0 con 0x7f57480d18e0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574759e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f57480ee3e0 0x7f57480ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51250/0 (socket says 172.21.2.117:51250) 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.401 7f574759e700 1 -- 172.21.2.117:0/1205180862 learned_addr learned my addr 172.21.2.117:0/1205180862 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 --2- 172.21.2.117:0/1205180862 >> v2:172.21.2.125:3300/0 conn(0x7f57480eb000 0x7f57480eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f5746d9d700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57480d18e0 0x7f57480ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f5745d9b700 1 -- 172.21.2.117:0/1205180862 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544387509 0 0) 0x7f5740001500 con 0x7f57480ee3e0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f5745d9b700 1 -- 172.21.2.117:0/1205180862 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5730002fe0 con 0x7f57480ee3e0 2021-02-10T11:44:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 -- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57480d18e0 msgr2=0x7f57480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57480d18e0 0x7f57480ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 -- 172.21.2.117:0/1205180862 >> v1:172.21.2.117:6789/0 conn(0x7f57480ee3e0 legacy=0x7f57480ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 -- 172.21.2.117:0/1205180862 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480eefd0 con 0x7f57480eb000 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.402 7f574659c700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480eb000 0x7f57480eb400 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f573c006650 tx=0x7f573c009a80).ready entity=mon.1 client_cookie=ba1bbbfb7d38d54a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f5745d9b700 1 -- 172.21.2.117:0/1205180862 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f573c00c0c0 con 0x7f57480eb000 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f5745d9b700 1 -- 172.21.2.117:0/1205180862 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c00c340 con 0x7f57480eb000 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f5745d9b700 1 -- 172.21.2.117:0/1205180862 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f573c00c860 con 0x7f57480eb000 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 -- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480eb000 msgr2=0x7f57480eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480eb000 0x7f57480eb400 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f573c006650 tx=0x7f573c009a80).stop 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 -- 172.21.2.117:0/1205180862 shutdown_connections 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480eb000 0x7f57480eb400 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 --2- 172.21.2.117:0/1205180862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57480d18e0 0x7f57480ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 -- 172.21.2.117:0/1205180862 >> 172.21.2.117:0/1205180862 conn(0x7f57480ced60 msgr2=0x7f57480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 -- 172.21.2.117:0/1205180862 shutdown_connections 2021-02-10T11:44:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.403 7f574d187700 1 -- 172.21.2.117:0/1205180862 wait complete. 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 Processor -- start 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 -- start start 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57481a33d0 0x7f57481a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480b82f0 0x7f57480b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480c0670 con 0x7f57480b8d10 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480deb20 con 0x7f57480b82f0 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.418 7f574d187700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480ded40 con 0x7f57481a33d0 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51002/0 (socket says 172.21.2.117:51002) 2021-02-10T11:44:34.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 -- 172.21.2.117:0/50653201 learned_addr learned my addr 172.21.2.117:0/50653201 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f5746d9d700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57481a33d0 0x7f57481a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574659c700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480b82f0 0x7f57480b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 -- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57481a33d0 msgr2=0x7f57481a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57481a33d0 0x7f57481a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 -- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480b82f0 msgr2=0x7f57480b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480b82f0 0x7f57480b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 -- 172.21.2.117:0/50653201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480def60 con 0x7f57480b8d10 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574759e700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f574000a8f0 tx=0x7f574000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5740007d10 con 0x7f57480b8d10 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5740007160 con 0x7f57480b8d10 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5740004720 con 0x7f57480b8d10 2021-02-10T11:44:34.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574d187700 1 -- 172.21.2.117:0/50653201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480def60 con 0x7f57480b8d10 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.419 7f574d187700 1 -- 172.21.2.117:0/50653201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480df4f0 con 0x7f57480b8d10 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.420 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5740007340 con 0x7f57480b8d10 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.420 7f57377fe700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 0x7f572c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.420 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f57400329e0 con 0x7f57480b8d10 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.420 7f5746d9d700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 0x7f572c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:34.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.421 7f5746d9d700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 0x7f572c01abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:34.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.451 7f574d187700 1 -- 172.21.2.117:0/50653201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5748087290 con 0x7f57480b8d10 2021-02-10T11:44:34.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.453 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5740026370 con 0x7f57480b8d10 2021-02-10T11:44:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.578 7f574d187700 1 -- 172.21.2.117:0/50653201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57480ede60 con 0x7f57480b8d10 2021-02-10T11:44:34.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.579 7f57377fe700 1 -- 172.21.2.117:0/50653201 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f5740026370 con 0x7f57480b8d10 2021-02-10T11:44:34.572 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:34.573 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":73,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:32.342643","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:29.317633","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:30.338240","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 msgr2=0x7f572c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 0x7f572c01abc0 crc :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 msgr2=0x7f57480b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 rx=0x7f574000a8f0 tx=0x7f574000ac90).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 shutdown_connections 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f572c0186c0 0x7f572c01abc0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57480b8d10 0x7f57480b9110 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f57480b82f0 0x7f57480b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 --2- 172.21.2.117:0/50653201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57481a33d0 0x7f57481a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:34.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 >> 172.21.2.117:0/50653201 conn(0x7f57480874c0 msgr2=0x7f5748096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 shutdown_connections 2021-02-10T11:44:34.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:34.581 7f574d187700 1 -- 172.21.2.117:0/50653201 wait complete. 2021-02-10T11:44:34.583 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:35.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 Processor -- start 2021-02-10T11:44:35.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 -- start start 2021-02-10T11:44:35.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc35407ab40 0x7fc3540794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc354079f10 0x7fc35407b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3540e5770 con 0x7fc354079b10 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35407a310 con 0x7fc35407ab40 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc35b4ad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540d18c0 con 0x7fc354079f10 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.012 7fc3592c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc35407ab40 0x7fc3540794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc35407ab40 0x7fc3540794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51010/0 (socket says 172.21.2.117:51010) 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 -- 172.21.2.117:0/1774627567 learned_addr learned my addr 172.21.2.117:0/1774627567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc359aca700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc354079f10 0x7fc35407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 -- 172.21.2.117:0/1774627567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc354079f10 msgr2=0x7fc35407b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc354079f10 0x7fc35407b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 -- 172.21.2.117:0/1774627567 >> v1:172.21.2.125:6789/0 conn(0x7fc354079b10 legacy=0x7fc35407af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 -- 172.21.2.117:0/1774627567 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3540d1c60 con 0x7fc35407ab40 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc3592c9700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc35407ab40 0x7fc3540794f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7fc344006700 tx=0x7fc344009b80).ready entity=mon.0 client_cookie=5bc93099fc083f43 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc34bfff700 1 -- 172.21.2.117:0/1774627567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc34400c0c0 con 0x7fc35407ab40 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc34bfff700 1 -- 172.21.2.117:0/1774627567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34400c440 con 0x7fc35407ab40 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc34bfff700 1 -- 172.21.2.117:0/1774627567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc34400c960 con 0x7fc35407ab40 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc35b4ad700 1 -- 172.21.2.117:0/1774627567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc35407ab40 msgr2=0x7fc3540794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.013 7fc35b4ad700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc35407ab40 0x7fc3540794f0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7fc344006700 tx=0x7fc344009b80).stop 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 -- 172.21.2.117:0/1774627567 shutdown_connections 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc354079f10 0x7fc35407b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 --2- 172.21.2.117:0/1774627567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc35407ab40 0x7fc3540794f0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 -- 172.21.2.117:0/1774627567 >> 172.21.2.117:0/1774627567 conn(0x7fc3540d0910 msgr2=0x7fc3540d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 -- 172.21.2.117:0/1774627567 shutdown_connections 2021-02-10T11:44:35.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.014 7fc35b4ad700 1 -- 172.21.2.117:0/1774627567 wait complete. 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 Processor -- start 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 -- start start 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3540dc6e0 0x7fc354191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3540fe660 0x7fc3540fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540ff300 con 0x7fc3540ff800 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540a8b30 con 0x7fc3540dc6e0 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc35b4ad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540dc230 con 0x7fc3540fe660 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc359aca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc359aca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51018/0 (socket says 172.21.2.117:51018) 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.029 7fc359aca700 1 -- 172.21.2.117:0/4210675150 learned_addr learned my addr 172.21.2.117:0/4210675150 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 -- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3540fe660 msgr2=0x7fc3540fea60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3540fe660 0x7fc3540fea60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 -- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3540dc6e0 msgr2=0x7fc354191c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3540dc6e0 0x7fc354191c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 -- 172.21.2.117:0/4210675150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354100220 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc359aca700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7fc34c0099a0 tx=0x7fc34c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc34c0066f0 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c003030 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc354100220 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3541007b0 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.030 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc34c0013e0 con 0x7fc3540ff800 2021-02-10T11:44:35.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.031 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc34c0011f0 con 0x7fc3540ff800 2021-02-10T11:44:35.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.031 7fc349ffb700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 0x7fc33401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.031 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc34c032cb0 con 0x7fc3540ff800 2021-02-10T11:44:35.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.031 7fc3592c9700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 0x7fc33401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.032 7fc3592c9700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 0x7fc33401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.062 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3540da9a0 con 0x7fc3540ff800 2021-02-10T11:44:35.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.064 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc34c000f40 con 0x7fc3540ff800 2021-02-10T11:44:35.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.194 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc3540e5770 con 0x7fc3540ff800 2021-02-10T11:44:35.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.197 7fc349ffb700 1 -- 172.21.2.117:0/4210675150 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v43) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc34c01f8a0 con 0x7fc3540ff800 2021-02-10T11:44:35.194 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:44:35.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.202 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 msgr2=0x7fc33401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.202 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 0x7fc33401abc0 crc :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 msgr2=0x7fc3540ffc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 rx=0x7fc34c0099a0 tx=0x7fc34c00bcf0).stop 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 shutdown_connections 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc334018730 0x7fc33401abc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3540ff800 0x7fc3540ffc00 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3540fe660 0x7fc3540fea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 --2- 172.21.2.117:0/4210675150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3540dc6e0 0x7fc354191c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 >> 172.21.2.117:0/4210675150 conn(0x7fc35419f690 msgr2=0x7fc354096490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:35.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 shutdown_connections 2021-02-10T11:44:35.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.203 7fc35b4ad700 1 -- 172.21.2.117:0/4210675150 wait complete. 2021-02-10T11:44:35.204 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:35.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.625 7f4608db9700 1 Processor -- start 2021-02-10T11:44:35.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.625 7f4608db9700 1 -- start start 2021-02-10T11:44:35.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.625 7f4608db9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4604079ce0 0x7f46041b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4608db9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460407a5c0 con 0x7f460407ab40 2021-02-10T11:44:35.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4608db9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46040d18c0 con 0x7f460407a0e0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4608db9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460407bf50 con 0x7f4604079ce0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4604079ce0 0x7f46041b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f460407ab40 0x7f46040794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51272/0 (socket says 172.21.2.117:51272) 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602b13700 1 -- 172.21.2.117:0/859620426 learned_addr learned my addr 172.21.2.117:0/859620426 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602312700 1 -- 172.21.2.117:0/859620426 >> v1:172.21.2.125:6789/0 conn(0x7f460407a0e0 legacy=0x7f46041b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602312700 1 -- 172.21.2.117:0/859620426 >> v1:172.21.2.117:6789/0 conn(0x7f460407ab40 legacy=0x7f46040794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602312700 1 -- 172.21.2.117:0/859620426 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46040d1c60 con 0x7f4604079ce0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4602312700 1 --2- 172.21.2.117:0/859620426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4604079ce0 0x7f46041b59d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7f45f8006650 tx=0x7f45f8009a80).ready entity=mon.0 client_cookie=1a58355540ca87e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4601b11700 1 -- 172.21.2.117:0/859620426 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f800c0c0 con 0x7f4604079ce0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4601b11700 1 -- 172.21.2.117:0/859620426 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f800c340 con 0x7f4604079ce0 2021-02-10T11:44:35.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.626 7f4601b11700 1 -- 172.21.2.117:0/859620426 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45f800c860 con 0x7f4604079ce0 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 -- 172.21.2.117:0/859620426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4604079ce0 msgr2=0x7f46041b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 --2- 172.21.2.117:0/859620426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4604079ce0 0x7f46041b59d0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 rx=0x7f45f8006650 tx=0x7f45f8009a80).stop 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 -- 172.21.2.117:0/859620426 shutdown_connections 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 --2- 172.21.2.117:0/859620426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4604079ce0 0x7f46041b59d0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 -- 172.21.2.117:0/859620426 >> 172.21.2.117:0/859620426 conn(0x7f46040d0910 msgr2=0x7f46040d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 -- 172.21.2.117:0/859620426 shutdown_connections 2021-02-10T11:44:35.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.627 7f4608db9700 1 -- 172.21.2.117:0/859620426 wait complete. 2021-02-10T11:44:35.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 Processor -- start 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 -- start start 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46040b8c10 0x7f46041c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46041c9e60 0x7f46040c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46040b90e0 con 0x7f46040c2da0 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4604100c20 con 0x7f46040b8c10 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4608db9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46040c31a0 con 0x7f46041c9e60 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4603314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4603314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51030/0 (socket says 172.21.2.117:51030) 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.642 7f4603314700 1 -- 172.21.2.117:0/3096566360 learned_addr learned my addr 172.21.2.117:0/3096566360 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 -- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46041c9e60 msgr2=0x7f46040c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46041c9e60 0x7f46040c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 -- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46040b8c10 msgr2=0x7f46041c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46040b8c10 0x7f46041c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 -- 172.21.2.117:0/3096566360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46040c33c0 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4603314700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7f45fc009940 tx=0x7f45fc00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45fc000f90 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc00c2f0 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4608db9700 1 -- 172.21.2.117:0/3096566360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46040c33c0 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f45fc00c7d0 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.643 7f4608db9700 1 -- 172.21.2.117:0/3096566360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46040c0fd0 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.644 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45fc001220 con 0x7f46040c2da0 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.644 7f45f37fe700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 0x7f45e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:35.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.644 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f45fc032890 con 0x7f46040c2da0 2021-02-10T11:44:35.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.644 7f4602b13700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 0x7f45e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:35.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.644 7f4602b13700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 0x7f45e801abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:35.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.675 7f4608db9700 1 -- 172.21.2.117:0/3096566360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46040d0910 con 0x7f46040c2da0 2021-02-10T11:44:35.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.678 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f45fc02cf70 con 0x7f46040c2da0 2021-02-10T11:44:35.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.807 7f4608db9700 1 -- 172.21.2.117:0/3096566360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f46040d0910 con 0x7f46040c2da0 2021-02-10T11:44:35.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.810 7f45f37fe700 1 -- 172.21.2.117:0/3096566360 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v44) v1 ==== 184+0+0 (secure 0 0 0) 0x7f45fc02cf70 con 0x7f46040c2da0 2021-02-10T11:44:35.807 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:44:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 msgr2=0x7f45e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 0x7f45e801abc0 crc :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 msgr2=0x7f46040c06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 rx=0x7f45fc009940 tx=0x7f45fc00bcf0).stop 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 shutdown_connections 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f45e80186c0 0x7f45e801abc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46040c2da0 0x7f46040c06c0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46041c9e60 0x7f46040c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 --2- 172.21.2.117:0/3096566360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46040b8c10 0x7f46041c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 >> 172.21.2.117:0/3096566360 conn(0x7f460409e020 msgr2=0x7f460409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 shutdown_connections 2021-02-10T11:44:35.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:35.816 7f4608db9700 1 -- 172.21.2.117:0/3096566360 wait complete. 2021-02-10T11:44:35.818 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 Processor -- start 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 -- start start 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faee0079ce0 0x7faee01b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee007a5c0 con 0x7faee007ab40 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00d18c0 con 0x7faee007a0e0 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.249 7faee69b1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee007bf50 con 0x7faee0079ce0 2021-02-10T11:44:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faee007ab40 0x7faee00794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51286/0 (socket says 172.21.2.117:51286) 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedffff700 1 -- 172.21.2.117:0/1409244139 learned_addr learned my addr 172.21.2.117:0/1409244139 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedf7fe700 1 --2- 172.21.2.117:0/1409244139 >> v2:172.21.2.125:3300/0 conn(0x7faee0079ce0 0x7faee01b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedeffd700 1 -- 172.21.2.117:0/1409244139 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870736025 0 0) 0x7faed00009b0 con 0x7faee007ab40 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedeffd700 1 -- 172.21.2.117:0/1409244139 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec8002fe0 con 0x7faee007ab40 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedf7fe700 1 -- 172.21.2.117:0/1409244139 >> v1:172.21.2.125:6789/0 conn(0x7faee007a0e0 legacy=0x7faee01b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedf7fe700 1 -- 172.21.2.117:0/1409244139 >> v1:172.21.2.117:6789/0 conn(0x7faee007ab40 legacy=0x7faee00794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedf7fe700 1 -- 172.21.2.117:0/1409244139 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee00d1c60 con 0x7faee0079ce0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.250 7faedf7fe700 1 --2- 172.21.2.117:0/1409244139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee0079ce0 0x7faee01b59d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7faed4006650 tx=0x7faed4009a00).ready entity=mon.1 client_cookie=cf161bacac90aa45 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faedeffd700 1 -- 172.21.2.117:0/1409244139 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed4009df0 con 0x7faee0079ce0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faedeffd700 1 -- 172.21.2.117:0/1409244139 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40009b0 con 0x7faee0079ce0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faedeffd700 1 -- 172.21.2.117:0/1409244139 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed400c2b0 con 0x7faee0079ce0 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 -- 172.21.2.117:0/1409244139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee0079ce0 msgr2=0x7faee01b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 --2- 172.21.2.117:0/1409244139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee0079ce0 0x7faee01b59d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7faed4006650 tx=0x7faed4009a00).stop 2021-02-10T11:44:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 -- 172.21.2.117:0/1409244139 shutdown_connections 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 --2- 172.21.2.117:0/1409244139 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee0079ce0 0x7faee01b59d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 -- 172.21.2.117:0/1409244139 >> 172.21.2.117:0/1409244139 conn(0x7faee00d0910 msgr2=0x7faee00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 -- 172.21.2.117:0/1409244139 shutdown_connections 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.251 7faee69b1700 1 -- 172.21.2.117:0/1409244139 wait complete. 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 Processor -- start 2021-02-10T11:44:36.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 -- start start 2021-02-10T11:44:36.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faee0199620 0x7faee008abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee008b210 0x7faee008b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee008a6f0 con 0x7faee00f8de0 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee00f91e0 con 0x7faee008b210 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.266 7faee69b1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee00f9520 con 0x7faee0199620 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51044/0 (socket says 172.21.2.117:51044) 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 -- 172.21.2.117:0/2554102260 learned_addr learned my addr 172.21.2.117:0/2554102260 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 -- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faee0199620 msgr2=0x7faee008abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faee0199620 0x7faee008abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 -- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee008b210 msgr2=0x7faee008b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee008b210 0x7faee008b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 -- 172.21.2.117:0/2554102260 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0199a20 con 0x7faee00f8de0 2021-02-10T11:44:36.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedffff700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7faed000a4c0 tx=0x7faed000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.267 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed000ac50 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0004140 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faed0004660 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faee69b1700 1 -- 172.21.2.117:0/2554102260 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee0199a20 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faee69b1700 1 -- 172.21.2.117:0/2554102260 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee008bf20 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faed0004320 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faedcff9700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.268 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faed0032670 con 0x7faee00f8de0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.269 7faedf7fe700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.269 7faedf7fe700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 0x7faec001abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.299 7faee69b1700 1 -- 172.21.2.117:0/2554102260 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee00f91e0 con 0x7faee00f8de0 2021-02-10T11:44:36.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.301 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faed0000940 con 0x7faee00f8de0 2021-02-10T11:44:36.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.432 7faee69b1700 1 -- 172.21.2.117:0/2554102260 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faee00f91e0 con 0x7faee00f8de0 2021-02-10T11:44:36.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.435 7faedcff9700 1 -- 172.21.2.117:0/2554102260 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7faed00046b0 con 0x7faee00f8de0 2021-02-10T11:44:36.432 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:44:36.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.440 7faee69b1700 1 -- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 msgr2=0x7faec001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.440 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 0x7faec001abc0 crc :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.440 7faee69b1700 1 -- 172.21.2.117:0/2554102260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 msgr2=0x7faee0199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.440 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 rx=0x7faed000a4c0 tx=0x7faed000a860).stop 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 -- 172.21.2.117:0/2554102260 shutdown_connections 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faee008b210 0x7faee008b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faee0199620 0x7faee008abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faec00186c0 0x7faec001abc0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 --2- 172.21.2.117:0/2554102260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faee00f8de0 0x7faee0199000 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 -- 172.21.2.117:0/2554102260 >> 172.21.2.117:0/2554102260 conn(0x7faee01a03b0 msgr2=0x7faee01ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:36.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 -- 172.21.2.117:0/2554102260 shutdown_connections 2021-02-10T11:44:36.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.441 7faee69b1700 1 -- 172.21.2.117:0/2554102260 wait complete. 2021-02-10T11:44:36.442 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.871 7f2335a85700 1 Processor -- start 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.871 7f2335a85700 1 -- start start 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2330079ce0 0x7f23301b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233007a5c0 con 0x7f233007ab40 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23300d18c0 con 0x7f233007a0e0 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233007bf50 con 0x7f2330079ce0 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2330079ce0 0x7f23301b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2330079ce0 0x7f23301b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51048/0 (socket says 172.21.2.117:51048) 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 -- 172.21.2.117:0/2516320760 learned_addr learned my addr 172.21.2.117:0/2516320760 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f233007ab40 0x7f23300794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51298/0 (socket says 172.21.2.117:51298) 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 -- 172.21.2.117:0/2516320760 >> v1:172.21.2.125:6789/0 conn(0x7f233007a0e0 legacy=0x7f23301b8000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 -- 172.21.2.117:0/2516320760 >> v1:172.21.2.117:6789/0 conn(0x7f233007ab40 legacy=0x7f23300794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 -- 172.21.2.117:0/2516320760 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23300d1c60 con 0x7f2330079ce0 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232effd700 1 --2- 172.21.2.117:0/2516320760 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2330079ce0 0x7f23301b59d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7f2318006700 tx=0x7f2318009b80).ready entity=mon.0 client_cookie=486fe886345719a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232e7fc700 1 -- 172.21.2.117:0/2516320760 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f231800c0c0 con 0x7f2330079ce0 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232e7fc700 1 -- 172.21.2.117:0/2516320760 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231800c3c0 con 0x7f2330079ce0 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f232e7fc700 1 -- 172.21.2.117:0/2516320760 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f231800c8e0 con 0x7f2330079ce0 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 -- 172.21.2.117:0/2516320760 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2330079ce0 msgr2=0x7f23301b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.872 7f2335a85700 1 --2- 172.21.2.117:0/2516320760 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2330079ce0 0x7f23301b59d0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 rx=0x7f2318006700 tx=0x7f2318009b80).stop 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.873 7f2335a85700 1 -- 172.21.2.117:0/2516320760 shutdown_connections 2021-02-10T11:44:36.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.873 7f2335a85700 1 --2- 172.21.2.117:0/2516320760 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2330079ce0 0x7f23301b59d0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.873 7f2335a85700 1 -- 172.21.2.117:0/2516320760 >> 172.21.2.117:0/2516320760 conn(0x7f23300d0910 msgr2=0x7f23300d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:36.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.873 7f2335a85700 1 -- 172.21.2.117:0/2516320760 shutdown_connections 2021-02-10T11:44:36.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.873 7f2335a85700 1 -- 172.21.2.117:0/2516320760 wait complete. 2021-02-10T11:44:36.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.887 7f2335a85700 1 Processor -- start 2021-02-10T11:44:36.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 -- start start 2021-02-10T11:44:36.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 0x7f233009a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 0x7f233009ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23300c3910 con 0x7f23300feaa0 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2330098c30 con 0x7f23300ff4c0 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f2335a85700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233009b3b0 con 0x7f233009a890 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f232f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f232f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51058/0 (socket says 172.21.2.117:51058) 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f232f7fe700 1 -- 172.21.2.117:0/2827406235 learned_addr learned my addr 172.21.2.117:0/2827406235 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f232effd700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 0x7f233009a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.888 7f232ffff700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 0x7f233009ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 -- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 msgr2=0x7f233009ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 0x7f233009ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 -- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 msgr2=0x7f233009a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 0x7f233009a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 -- 172.21.2.117:0/2827406235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233009b5d0 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232ffff700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 0x7f233009ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232effd700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 0x7f233009a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f232f7fe700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7f2320001c60 tx=0x7f232000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f232000acc0 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320004140 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2320004590 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f2335a85700 1 -- 172.21.2.117:0/2827406235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233009b5d0 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.889 7f2335a85700 1 -- 172.21.2.117:0/2827406235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233009bb60 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.890 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2320004320 con 0x7f23300feaa0 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.890 7f2327fff700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 0x7f231401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:36.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.890 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2320032690 con 0x7f23300feaa0 2021-02-10T11:44:36.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.891 7f232effd700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 0x7f231401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:36.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.891 7f232effd700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 0x7f231401abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:36.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.921 7f2335a85700 1 -- 172.21.2.117:0/2827406235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23300a7740 con 0x7f23300feaa0 2021-02-10T11:44:36.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:36.924 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2320000dd0 con 0x7f23300feaa0 2021-02-10T11:44:37.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.052 7f2335a85700 1 -- 172.21.2.117:0/2827406235 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f23300a7740 con 0x7f23300feaa0 2021-02-10T11:44:37.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.056 7f2327fff700 1 -- 172.21.2.117:0/2827406235 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7f232001f670 con 0x7f23300feaa0 2021-02-10T11:44:37.052 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 msgr2=0x7f231401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 0x7f231401abc0 crc :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 msgr2=0x7f23300feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 rx=0x7f2320001c60 tx=0x7f232000a8d0).stop 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 shutdown_connections 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f233009a890 0x7f233009ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23300ff4c0 0x7f233009a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f23140186c0 0x7f231401abc0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 --2- 172.21.2.117:0/2827406235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23300feaa0 0x7f23300feea0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 >> 172.21.2.117:0/2827406235 conn(0x7f233009e020 msgr2=0x7f23300fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:37.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 shutdown_connections 2021-02-10T11:44:37.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.061 7f2335a85700 1 -- 172.21.2.117:0/2827406235 wait complete. 2021-02-10T11:44:37.062 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:37.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 Processor -- start 2021-02-10T11:44:37.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 -- start start 2021-02-10T11:44:37.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7500d18e0 0x7fa7500ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa7500eb000 0x7fa7500eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7500ede60 con 0x7fa7500ee3e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7500eba20 con 0x7fa7500d18e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa756bb6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7500d1ce0 con 0x7fa7500eb000 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa7549d2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7500d18e0 0x7fa7500ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa7549d2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7500d18e0 0x7fa7500ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51062/0 (socket says 172.21.2.117:51062) 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.481 7fa7549d2700 1 -- 172.21.2.117:0/2385448664 learned_addr learned my addr 172.21.2.117:0/2385448664 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa7549d2700 1 -- 172.21.2.117:0/2385448664 >> v2:172.21.2.125:3300/0 conn(0x7fa7500eb000 msgr2=0x7fa7500eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa7549d2700 1 --2- 172.21.2.117:0/2385448664 >> v2:172.21.2.125:3300/0 conn(0x7fa7500eb000 0x7fa7500eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa7549d2700 1 -- 172.21.2.117:0/2385448664 >> v1:172.21.2.125:6789/0 conn(0x7fa7500ee3e0 legacy=0x7fa7500ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa7549d2700 1 -- 172.21.2.117:0/2385448664 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7500eefd0 con 0x7fa7500d18e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa7549d2700 1 --2- 172.21.2.117:0/2385448664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500d18e0 0x7fa7500ea9e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7fa738006700 tx=0x7fa738009b80).ready entity=mon.0 client_cookie=34581ff5b9d4d81a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa74f7fe700 1 -- 172.21.2.117:0/2385448664 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73800c0c0 con 0x7fa7500d18e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa74f7fe700 1 -- 172.21.2.117:0/2385448664 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73800c440 con 0x7fa7500d18e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa74f7fe700 1 -- 172.21.2.117:0/2385448664 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73800c960 con 0x7fa7500d18e0 2021-02-10T11:44:37.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa756bb6700 1 -- 172.21.2.117:0/2385448664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500d18e0 msgr2=0x7fa7500ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.482 7fa756bb6700 1 --2- 172.21.2.117:0/2385448664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500d18e0 0x7fa7500ea9e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 rx=0x7fa738006700 tx=0x7fa738009b80).stop 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 -- 172.21.2.117:0/2385448664 shutdown_connections 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 --2- 172.21.2.117:0/2385448664 >> v2:172.21.2.125:3300/0 conn(0x7fa7500eb000 0x7fa7500eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 --2- 172.21.2.117:0/2385448664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500d18e0 0x7fa7500ea9e0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 -- 172.21.2.117:0/2385448664 >> 172.21.2.117:0/2385448664 conn(0x7fa7500ced60 msgr2=0x7fa7500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 -- 172.21.2.117:0/2385448664 shutdown_connections 2021-02-10T11:44:37.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.483 7fa756bb6700 1 -- 172.21.2.117:0/2385448664 wait complete. 2021-02-10T11:44:37.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 Processor -- start 2021-02-10T11:44:37.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 -- start start 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7500984c0 0x7fa75019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa75019dd40 0x7fa75019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7500b1f30 con 0x7fa7500980c0 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75019d830 con 0x7fa75019dd40 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa756bb6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7501a55e0 con 0x7fa7500984c0 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa7549d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa7549d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51070/0 (socket says 172.21.2.117:51070) 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.498 7fa7549d2700 1 -- 172.21.2.117:0/235209960 learned_addr learned my addr 172.21.2.117:0/235209960 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 -- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7500984c0 msgr2=0x7fa75019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7500984c0 0x7fa75019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 -- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa75019dd40 msgr2=0x7fa75019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa75019dd40 0x7fa75019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 -- 172.21.2.117:0/235209960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75019e760 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa7549d2700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fa73800b5e0 tx=0x7fa73800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa73800bd70 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa738002030 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa756bb6700 1 -- 172.21.2.117:0/235209960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75019e760 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa756bb6700 1 -- 172.21.2.117:0/235209960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75019ecf0 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.499 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa738002550 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.500 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa738002210 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.500 7fa74d7fa700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.500 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa738032850 con 0x7fa7500980c0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.500 7fa74ffff700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:37.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.500 7fa74ffff700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:37.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.531 7fa756bb6700 1 -- 172.21.2.117:0/235209960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa750087210 con 0x7fa7500980c0 2021-02-10T11:44:37.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.534 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa7380009b0 con 0x7fa7500980c0 2021-02-10T11:44:37.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.660 7fa756bb6700 1 -- 172.21.2.117:0/235209960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7500ede60 con 0x7fa7500980c0 2021-02-10T11:44:37.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.661 7fa74d7fa700 1 -- 172.21.2.117:0/235209960 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2095 (secure 0 0 0) 0x7fa738002030 con 0x7fa7500980c0 2021-02-10T11:44:37.654 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 53 2021-02-10T11:44:37.655 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:37.655 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:44:33.346284","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/319870470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":319870470},{"type":"v1","addr":"172.21.2.125:6835","nonce":319870470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 msgr2=0x7fa73401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 crc :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 msgr2=0x7fa75019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 rx=0x7fa73800b5e0 tx=0x7fa73800b980).stop 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 shutdown_connections 2021-02-10T11:44:37.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa75019dd40 0x7fa75019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7500984c0 0x7fa75019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7340186c0 0x7fa73401abc0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 --2- 172.21.2.117:0/235209960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7500980c0 0x7fa75019ca80 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 >> 172.21.2.117:0/235209960 conn(0x7fa75019f610 msgr2=0x7fa7501a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 shutdown_connections 2021-02-10T11:44:37.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:37.664 7fa756bb6700 1 -- 172.21.2.117:0/235209960 wait complete. 2021-02-10T11:44:37.866 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:31.340745', 'modified': '2021-02-10 11:44:33.346284', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8910': {'gid': 8910, 'name': 'b', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/319870470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 319870470}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 319870470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:37.866 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:37.866 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:38.265 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8910, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9318988189999997, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 53, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 73, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.576017554 2021-02-10T11:44:38.266 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:44:38.270 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8910, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9318988189999997, "mdsmap_epoch": 53, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 12.576017554 } 2021-02-10T11:44:38.271 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.686 7f3f5ffc3700 1 Processor -- start 2021-02-10T11:44:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 -- start start 2021-02-10T11:44:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3f580d18e0 0x7f3f580ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580eb000 0x7f3f580eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f580ede60 con 0x7f3f580ee3e0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580eba90 con 0x7f3f580d18e0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5ffc3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580d1ce0 con 0x7f3f580eb000 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5e5e0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3f580ee3e0 0x7f3f580ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51326/0 (socket says 172.21.2.117:51326) 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.687 7f3f5e5e0700 1 -- 172.21.2.117:0/3696304016 learned_addr learned my addr 172.21.2.117:0/3696304016 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5d5de700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580eb000 0x7f3f580eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> v2:172.21.2.125:3300/0 conn(0x7f3f580d18e0 0x7f3f580ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034359217 0 0) 0x7f3f50001500 con 0x7f3f580ee3e0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f44002fe0 con 0x7f3f580ee3e0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 677578672 0 0) 0x7f3f50001500 con 0x7f3f580ee3e0 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580eb000 msgr2=0x7f3f580eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580eb000 0x7f3f580eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 >> v2:172.21.2.125:3300/0 conn(0x7f3f580d18e0 msgr2=0x7f3f580ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 --2- 172.21.2.117:0/3696304016 >> v2:172.21.2.125:3300/0 conn(0x7f3f580d18e0 0x7f3f580ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580eefd0 con 0x7f3f580ee3e0 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3f50001a00 con 0x7f3f580ee3e0 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 >> v1:172.21.2.117:6789/0 conn(0x7f3f580ee3e0 legacy=0x7f3f580ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f44003af0 0x7f3f44005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f4404a830 0x7f3f4404cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f4404d650 0x7f3f4404fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f44050390 con 0x7f3f44003af0 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440505b0 con 0x7f3f4404a830 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440507d0 con 0x7f3f4404d650 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> v2:172.21.2.125:3300/0 conn(0x7f3f580d18e0 0x7f3f580ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5d5de700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f4404a830 0x7f3f4404cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.688 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f44003af0 0x7f3f44005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5e5e0700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f4404d650 0x7f3f4404fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 -- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f4404d650 msgr2=0x7f3f4404fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f4404d650 0x7f3f4404fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 -- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f4404a830 msgr2=0x7f3f4404cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f4404a830 0x7f3f4404cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 -- 172.21.2.117:0/3696304016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44050a70 con 0x7f3f44003af0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5dddf700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f44003af0 0x7f3f44005f80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f3f54000b80 tx=0x7f3f54009e80).ready entity=mon.0 client_cookie=c2fc7ed511e70111 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f5400c5f0 con 0x7f3f44003af0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5400c7f0 con 0x7f3f44003af0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5cddd700 1 -- 172.21.2.117:0/3696304016 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f5400cd10 con 0x7f3f44003af0 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5ffc3700 1 -- 172.21.2.117:0/3696304016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f44003af0 msgr2=0x7f3f44005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5ffc3700 1 --2- 172.21.2.117:0/3696304016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f44003af0 0x7f3f44005f80 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f3f54000b80 tx=0x7f3f54009e80).stop 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5e5e0700 1 -- 172.21.2.117:0/3696304016 reap_dead start 2021-02-10T11:44:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5ffc3700 1 -- 172.21.2.117:0/3696304016 shutdown_connections 2021-02-10T11:44:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.689 7f3f5ffc3700 1 -- 172.21.2.117:0/3696304016 >> 172.21.2.117:0/3696304016 conn(0x7f3f580ced60 msgr2=0x7f3f580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.690 7f3f5ffc3700 1 -- 172.21.2.117:0/3696304016 shutdown_connections 2021-02-10T11:44:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.690 7f3f5ffc3700 1 -- 172.21.2.117:0/3696304016 wait complete. 2021-02-10T11:44:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.704 7f3f5ffc3700 1 Processor -- start 2021-02-10T11:44:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.704 7f3f5ffc3700 1 -- start start 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f580bf680 0x7f3f580a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580bfa80 0x7f3f581a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f581a3c30 con 0x7f3f580a1d70 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580a0680 con 0x7f3f580bf680 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5ffc3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580a08a0 con 0x7f3f580bfa80 2021-02-10T11:44:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51090/0 (socket says 172.21.2.117:51090) 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 -- 172.21.2.117:0/1875207187 learned_addr learned my addr 172.21.2.117:0/1875207187 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 -- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580bfa80 msgr2=0x7f3f581a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580bfa80 0x7f3f581a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 -- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f580bf680 msgr2=0x7f3f580a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f580bf680 0x7f3f580a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.705 7f3f5e5e0700 1 -- 172.21.2.117:0/1875207187 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f580a0ac0 con 0x7f3f580a1d70 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f5e5e0700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7f3f5000b940 tx=0x7f3f5000bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f50007bd0 con 0x7f3f580a1d70 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f50006880 con 0x7f3f580a1d70 2021-02-10T11:44:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f50006300 con 0x7f3f580a1d70 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f580a0ac0 con 0x7f3f580a1d70 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.706 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f580a0ac0 con 0x7f3f580a1d70 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.707 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f50006020 con 0x7f3f580a1d70 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.707 7f3f4a7fc700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 0x7f3f4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.707 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f50032a30 con 0x7f3f580a1d70 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.708 7f3f5dddf700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 0x7f3f4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.709 7f3f5dddf700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 0x7f3f4001abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:38.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.737 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f58087280 con 0x7f3f580a1d70 2021-02-10T11:44:38.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.740 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f5000be20 con 0x7f3f580a1d70 2021-02-10T11:44:38.841 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:38.844 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:44:38.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.864 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f580ede60 con 0x7f3f580a1d70 2021-02-10T11:44:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.865 7f3f4a7fc700 1 -- 172.21.2.117:0/1875207187 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f3f50006000 con 0x7f3f580a1d70 2021-02-10T11:44:38.857 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:38.857 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":73,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:32.342643","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:29.317633","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:30.338240","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:38.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 msgr2=0x7f3f4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 0x7f3f4001abc0 crc :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 msgr2=0x7f3f581a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:38.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 rx=0x7f3f5000b940 tx=0x7f3f5000bce0).stop 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 shutdown_connections 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f400186c0 0x7f3f4001abc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f580a1d70 0x7f3f581a3480 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f580bfa80 0x7f3f581a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 --2- 172.21.2.117:0/1875207187 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f580bf680 0x7f3f580a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 >> 172.21.2.117:0/1875207187 conn(0x7f3f580874b0 msgr2=0x7f3f58090b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 shutdown_connections 2021-02-10T11:44:38.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:38.867 7f3f5ffc3700 1 -- 172.21.2.117:0/1875207187 wait complete. 2021-02-10T11:44:38.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:44:29.317633', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:44:30.338240', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:38.868 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:39.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 Processor -- start 2021-02-10T11:44:39.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 -- start start 2021-02-10T11:44:39.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00e7180 0x7f58c00e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58c00e8390 0x7f58c00e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c00e7580 con 0x7f58c01b07f0 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c00e5ce0 con 0x7f58c00e8390 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c69a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00e77f0 con 0x7f58c00e7180 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c4fc4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58c00e8390 0x7f58c00e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c4fc4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58c00e8390 0x7f58c00e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51096/0 (socket says 172.21.2.117:51096) 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58bffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f58c01b07f0 0x7f58c00e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51344/0 (socket says 172.21.2.117:51344) 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.267 7f58c4fc4700 1 -- 172.21.2.117:0/103546723 learned_addr learned my addr 172.21.2.117:0/103546723 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58beffd700 1 -- 172.21.2.117:0/103546723 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468118292 0 0) 0x7f58b0000ce0 con 0x7f58c01b07f0 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58beffd700 1 -- 172.21.2.117:0/103546723 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a8002fe0 con 0x7f58c01b07f0 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58bf7fe700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00e7180 0x7f58c00e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c4fc4700 1 -- 172.21.2.117:0/103546723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00e7180 msgr2=0x7f58c00e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c4fc4700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00e7180 0x7f58c00e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c4fc4700 1 -- 172.21.2.117:0/103546723 >> v1:172.21.2.117:6789/0 conn(0x7f58c01b07f0 legacy=0x7f58c00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c4fc4700 1 -- 172.21.2.117:0/103546723 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00e8db0 con 0x7f58c00e8390 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c4fc4700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00e8390 0x7f58c00e8790 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f58b8006f00 tx=0x7f58b8009f10).ready entity=mon.0 client_cookie=8059a875fb2121a9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58beffd700 1 -- 172.21.2.117:0/103546723 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b800b3b0 con 0x7f58c00e8390 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58beffd700 1 -- 172.21.2.117:0/103546723 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800b730 con 0x7f58c00e8390 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58beffd700 1 -- 172.21.2.117:0/103546723 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b800bc50 con 0x7f58c00e8390 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c69a7700 1 -- 172.21.2.117:0/103546723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00e8390 msgr2=0x7f58c00e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.268 7f58c69a7700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00e8390 0x7f58c00e8790 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 rx=0x7f58b8006f00 tx=0x7f58b8009f10).stop 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 -- 172.21.2.117:0/103546723 shutdown_connections 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00e8390 0x7f58c00e8790 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 --2- 172.21.2.117:0/103546723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00e7180 0x7f58c00e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 -- 172.21.2.117:0/103546723 >> 172.21.2.117:0/103546723 conn(0x7f58c00ce500 msgr2=0x7f58c00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 -- 172.21.2.117:0/103546723 shutdown_connections 2021-02-10T11:44:39.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.269 7f58c69a7700 1 -- 172.21.2.117:0/103546723 wait complete. 2021-02-10T11:44:39.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.283 7f58c69a7700 1 Processor -- start 2021-02-10T11:44:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 -- start start 2021-02-10T11:44:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00b8ac0 0x7f58c019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c019df40 0x7f58c00bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00bb430 con 0x7f58c00bbfd0 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00bc9f0 con 0x7f58c019df40 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c69a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00bcc10 con 0x7f58c00b8ac0 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c4fc4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c4fc4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51102/0 (socket says 172.21.2.117:51102) 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58c4fc4700 1 -- 172.21.2.117:0/3185672191 learned_addr learned my addr 172.21.2.117:0/3185672191 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58bffff700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00b8ac0 0x7f58c019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.284 7f58bf7fe700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c019df40 0x7f58c00bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 -- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00b8ac0 msgr2=0x7f58c019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00b8ac0 0x7f58c019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 -- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c019df40 msgr2=0x7f58c00bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c019df40 0x7f58c00bb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 -- 172.21.2.117:0/3185672191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00b6f70 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c4fc4700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f58b8009390 tx=0x7f58b800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b8003030 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b8006c80 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58b800c570 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c00b6f70 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.285 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c00b7500 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.286 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58b8001270 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.286 7f58bcff9700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 0x7f58a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.286 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f58b8032a10 con 0x7f58c00bbfd0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.286 7f58bffff700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 0x7f58a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.286 7f58bffff700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 0x7f58a001abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.317 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c007e800 con 0x7f58c00bbfd0 2021-02-10T11:44:39.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.320 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58b8009450 con 0x7f58c00bbfd0 2021-02-10T11:44:39.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.447 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f58c00e77f0 con 0x7f58c00bbfd0 2021-02-10T11:44:39.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.447 7f58bcff9700 1 -- 172.21.2.117:0/3185672191 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v73) v1 ==== 92+0+10 (secure 0 0 0) 0x7f58b8001270 con 0x7f58c00bbfd0 2021-02-10T11:44:39.440 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:39.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 msgr2=0x7f58a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 0x7f58a001abc0 crc :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 msgr2=0x7f58c00bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 rx=0x7f58b8009390 tx=0x7f58b800bd40).stop 2021-02-10T11:44:39.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 shutdown_connections 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58a00186c0 0x7f58a001abc0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58c00bbfd0 0x7f58c00bc3d0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58c019df40 0x7f58c00bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 --2- 172.21.2.117:0/3185672191 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58c00b8ac0 0x7f58c019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.449 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 >> 172.21.2.117:0/3185672191 conn(0x7f58c0110940 msgr2=0x7f58c007fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.450 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 shutdown_connections 2021-02-10T11:44:39.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.450 7f58c69a7700 1 -- 172.21.2.117:0/3185672191 wait complete. 2021-02-10T11:44:39.450 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:39.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.871 7fc33c52d700 1 Processor -- start 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.871 7fc33c52d700 1 -- start start 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33c52d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3340d18f0 0x7fc3340eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33c52d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3340eb040 0x7fc3340eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33c52d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3340ede60 con 0x7fc3340ee3e0 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33c52d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3340eba60 con 0x7fc3340eb040 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33c52d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340d1cf0 con 0x7fc3340d18f0 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3340eb040 0x7fc3340eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33ab4a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc3340ee3e0 0x7fc3340ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51358/0 (socket says 172.21.2.117:51358) 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc3340eb040 0x7fc3340eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51108/0 (socket says 172.21.2.117:51108) 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc33ab4a700 1 -- 172.21.2.117:0/861635266 learned_addr learned my addr 172.21.2.117:0/861635266 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:39.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 -- 172.21.2.117:0/861635266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3340d18f0 msgr2=0x7fc3340eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 --2- 172.21.2.117:0/861635266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3340d18f0 0x7fc3340eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 -- 172.21.2.117:0/861635266 >> v1:172.21.2.117:6789/0 conn(0x7fc3340ee3e0 legacy=0x7fc3340ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 -- 172.21.2.117:0/861635266 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340eefd0 con 0x7fc3340eb040 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.872 7fc339b48700 1 --2- 172.21.2.117:0/861635266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3340eb040 0x7fc3340eb440 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7fc328006650 tx=0x7fc328009a80).ready entity=mon.0 client_cookie=86e6c58315c976c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc339347700 1 -- 172.21.2.117:0/861635266 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3280009b0 con 0x7fc3340eb040 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc339347700 1 -- 172.21.2.117:0/861635266 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32800c0c0 con 0x7fc3340eb040 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc339347700 1 -- 172.21.2.117:0/861635266 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc32800c5e0 con 0x7fc3340eb040 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 -- 172.21.2.117:0/861635266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3340eb040 msgr2=0x7fc3340eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 --2- 172.21.2.117:0/861635266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3340eb040 0x7fc3340eb440 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 rx=0x7fc328006650 tx=0x7fc328009a80).stop 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 -- 172.21.2.117:0/861635266 shutdown_connections 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 --2- 172.21.2.117:0/861635266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3340eb040 0x7fc3340eb440 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 --2- 172.21.2.117:0/861635266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3340d18f0 0x7fc3340eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.873 7fc33c52d700 1 -- 172.21.2.117:0/861635266 >> 172.21.2.117:0/861635266 conn(0x7fc3340cf060 msgr2=0x7fc3340cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:39.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.874 7fc33c52d700 1 -- 172.21.2.117:0/861635266 shutdown_connections 2021-02-10T11:44:39.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.874 7fc33c52d700 1 -- 172.21.2.117:0/861635266 wait complete. 2021-02-10T11:44:39.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.888 7fc33c52d700 1 Processor -- start 2021-02-10T11:44:39.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.888 7fc33c52d700 1 -- start start 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3341a4240 0x7fc3341a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc33409f790 0x7fc3340b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33409fb90 con 0x7fc33409ed70 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340dc640 con 0x7fc33409f790 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc334097270 con 0x7fc3341a4240 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51116/0 (socket says 172.21.2.117:51116) 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 -- 172.21.2.117:0/2581401292 learned_addr learned my addr 172.21.2.117:0/2581401292 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 -- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3341a4240 msgr2=0x7fc3341a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3341a4240 0x7fc3341a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 -- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc33409f790 msgr2=0x7fc3340b8160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:39.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc33409f790 0x7fc3340b8160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 -- 172.21.2.117:0/2581401292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340b8780 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc339b48700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 rx=0x7fc328009630 tx=0x7fc32800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3280093a0 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32800bd70 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.889 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3340b8780 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.890 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3340b8780 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.890 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc328005560 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.890 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3280052e0 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.890 7fc326ffd700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 0x7fc31c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.890 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc328032980 con 0x7fc33409ed70 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.891 7fc33a349700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 0x7fc31c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:39.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.891 7fc33a349700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 0x7fc31c01ac10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:39.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.922 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33409fb90 con 0x7fc33409ed70 2021-02-10T11:44:39.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:39.924 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc32801f570 con 0x7fc33409ed70 2021-02-10T11:44:40.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.053 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fc33409fb90 con 0x7fc33409ed70 2021-02-10T11:44:40.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.054 7fc326ffd700 1 -- 172.21.2.117:0/2581401292 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v73) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc32800bd70 con 0x7fc33409ed70 2021-02-10T11:44:40.046 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 msgr2=0x7fc31c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 0x7fc31c01ac10 crc :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 msgr2=0x7fc33409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 rx=0x7fc328009630 tx=0x7fc32800b980).stop 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 shutdown_connections 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc31c018730 0x7fc31c01ac10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc33409ed70 0x7fc33409f170 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc33409f790 0x7fc3340b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 --2- 172.21.2.117:0/2581401292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3341a4240 0x7fc3341a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 >> 172.21.2.117:0/2581401292 conn(0x7fc334087090 msgr2=0x7fc33408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 shutdown_connections 2021-02-10T11:44:40.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.056 7fc33c52d700 1 -- 172.21.2.117:0/2581401292 wait complete. 2021-02-10T11:44:40.056 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:44:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 Processor -- start 2021-02-10T11:44:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 -- start start 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800d18f0 0x7f87800eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87800eb040 0x7f87800eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87800ee3e0 0x7f87800ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87800ede60 con 0x7f87800eb040 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87800eba60 con 0x7f87800ee3e0 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8788634700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87800d1cf0 con 0x7f87800d18f0 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8785c4f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87800eb040 0x7f87800eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8785c4f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87800eb040 0x7f87800eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51120/0 (socket says 172.21.2.117:51120) 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.479 7f8785c4f700 1 -- 172.21.2.117:0/2642386044 learned_addr learned my addr 172.21.2.117:0/2642386044 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 -- 172.21.2.117:0/2642386044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800d18f0 msgr2=0x7f87800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 --2- 172.21.2.117:0/2642386044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800d18f0 0x7f87800eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 -- 172.21.2.117:0/2642386044 >> v2:172.21.2.125:3300/0 conn(0x7f87800ee3e0 msgr2=0x7f87800ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 --2- 172.21.2.117:0/2642386044 >> v2:172.21.2.125:3300/0 conn(0x7f87800ee3e0 0x7f87800ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 -- 172.21.2.117:0/2642386044 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800eee00 con 0x7f87800eb040 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8785c4f700 1 --2- 172.21.2.117:0/2642386044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87800eb040 0x7f87800eb440 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f877c006700 tx=0x7f877c009b80).ready entity=mon.0 client_cookie=bfd69eb6c74457de server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f878544e700 1 -- 172.21.2.117:0/2642386044 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c00c0c0 con 0x7f87800eb040 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f878544e700 1 -- 172.21.2.117:0/2642386044 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c00c340 con 0x7f87800eb040 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f878544e700 1 -- 172.21.2.117:0/2642386044 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f877c00c860 con 0x7f87800eb040 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8788634700 1 -- 172.21.2.117:0/2642386044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87800eb040 msgr2=0x7f87800eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.480 7f8788634700 1 --2- 172.21.2.117:0/2642386044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87800eb040 0x7f87800eb440 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 rx=0x7f877c006700 tx=0x7f877c009b80).stop 2021-02-10T11:44:40.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 -- 172.21.2.117:0/2642386044 shutdown_connections 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 --2- 172.21.2.117:0/2642386044 >> v2:172.21.2.125:3300/0 conn(0x7f87800ee3e0 0x7f87800ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 --2- 172.21.2.117:0/2642386044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87800eb040 0x7f87800eb440 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 --2- 172.21.2.117:0/2642386044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800d18f0 0x7f87800eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 -- 172.21.2.117:0/2642386044 >> 172.21.2.117:0/2642386044 conn(0x7f87800cf060 msgr2=0x7f87800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 -- 172.21.2.117:0/2642386044 shutdown_connections 2021-02-10T11:44:40.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.481 7f8788634700 1 -- 172.21.2.117:0/2642386044 wait complete. 2021-02-10T11:44:40.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 Processor -- start 2021-02-10T11:44:40.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 -- start start 2021-02-10T11:44:40.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87801a3930 0x7f87800a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 0x7f87800a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878007cca0 con 0x7f878007c7c0 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878018fb70 con 0x7f87801a3930 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8788634700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87800dc6d0 con 0x7f87800a13a0 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8786450700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8786450700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51122/0 (socket says 172.21.2.117:51122) 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.496 7f8786450700 1 -- 172.21.2.117:0/1779449420 learned_addr learned my addr 172.21.2.117:0/1779449420 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8785c4f700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87801a3930 0x7f87800a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786c51700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 0x7f87800a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 -- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 msgr2=0x7f87800a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 0x7f87800a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 -- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87801a3930 msgr2=0x7f87800a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87801a3930 0x7f87800a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 -- 172.21.2.117:0/1779449420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878007d220 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786c51700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 0x7f87800a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8786450700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 rx=0x7f8774006f00 tx=0x7f877400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8774006350 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877400bdc0 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8774002020 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8788634700 1 -- 172.21.2.117:0/1779449420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878007d220 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.497 7f8788634700 1 -- 172.21.2.117:0/1779449420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878007d7b0 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.498 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8774007020 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.498 7f8772ffd700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 0x7f876801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.498 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8774032660 con 0x7f878007c7c0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.499 7f8785c4f700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 0x7f876801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:40.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.499 7f8785c4f700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 0x7f876801abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:40.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.529 7f8788634700 1 -- 172.21.2.117:0/1779449420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87800daff0 con 0x7f878007c7c0 2021-02-10T11:44:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.531 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8774000960 con 0x7f878007c7c0 2021-02-10T11:44:40.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.658 7f8788634700 1 -- 172.21.2.117:0/1779449420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f87800daff0 con 0x7f878007c7c0 2021-02-10T11:44:40.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.658 7f8772ffd700 1 -- 172.21.2.117:0/1779449420 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v73) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8774007000 con 0x7f878007c7c0 2021-02-10T11:44:40.651 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:40.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.660 7f8788634700 1 -- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 msgr2=0x7f876801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.660 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 0x7f876801abc0 crc :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.660 7f8788634700 1 -- 172.21.2.117:0/1779449420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 msgr2=0x7f87801a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:40.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.660 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 rx=0x7f8774006f00 tx=0x7f877400b940).stop 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 -- 172.21.2.117:0/1779449420 shutdown_connections 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87800a13a0 0x7f87800a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87801a3930 0x7f87800a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87680186c0 0x7f876801abc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 --2- 172.21.2.117:0/1779449420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f878007c7c0 0x7f87801a3310 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 -- 172.21.2.117:0/1779449420 >> 172.21.2.117:0/1779449420 conn(0x7f8780087090 msgr2=0x7f878008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 -- 172.21.2.117:0/1779449420 shutdown_connections 2021-02-10T11:44:40.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:40.661 7f8788634700 1 -- 172.21.2.117:0/1779449420 wait complete. 2021-02-10T11:44:40.661 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:44:40.661 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:41.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 Processor -- start 2021-02-10T11:44:41.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 -- start start 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e00e5240 0x7f12e0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e0085930 con 0x7f12e0084c60 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e01b09e0 con 0x7f12e00e4e40 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12e596c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e00e5cc0 con 0x7f12e00e5240 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12dfd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f12e0084c60 0x7f12e0085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51384/0 (socket says 172.21.2.117:51384) 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.081 7f12dfd75700 1 -- 172.21.2.117:0/3291033323 learned_addr learned my addr 172.21.2.117:0/3291033323 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78392013 0 0) 0x7f12d8001500 con 0x7f12e0084c60 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12cc002fe0 con 0x7f12e0084c60 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e00e5240 0x7f12e0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2755647336 0 0) 0x7f12d8001460 con 0x7f12e0084c60 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e00e5240 msgr2=0x7f12e0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e00e5240 0x7f12e0086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 >> v1:172.21.2.125:6789/0 conn(0x7f12e00e4e40 legacy=0x7f12e01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0084840 con 0x7f12e0084c60 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f12d8001980 con 0x7f12e0084c60 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 >> v1:172.21.2.117:6789/0 conn(0x7f12e0084c60 legacy=0x7f12e0085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12cc003af0 0x7f12cc005f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12cc0067e0 0x7f12cc008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12cc0095a0 0x7f12cc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc00c370 con 0x7f12cc0067e0 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc00c590 con 0x7f12cc0095a0 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12de572700 1 -- 172.21.2.117:0/3291033323 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12cc00c7b0 con 0x7f12cc003af0 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e00e5240 0x7f12e0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12cc0067e0 0x7f12cc008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12df574700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12cc003af0 0x7f12cc005f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.082 7f12dfd75700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12cc0095a0 0x7f12cc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 -- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12cc003af0 msgr2=0x7f12cc005f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12cc003af0 0x7f12cc005f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 -- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12cc0095a0 msgr2=0x7f12cc00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12cc0095a0 0x7f12cc00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 -- 172.21.2.117:0/3291033323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc00ca50 con 0x7f12cc0067e0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12df574700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12cc003af0 0x7f12cc005f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12ded73700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12cc0067e0 0x7f12cc008d00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f12d0002e00 tx=0x7f12d004dac0).ready entity=mon.0 client_cookie=8014035a438254ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12d0000c90 con 0x7f12cc0067e0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d0002030 con 0x7f12cc0067e0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12de572700 1 -- 172.21.2.117:0/3291033323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12d00065d0 con 0x7f12cc0067e0 2021-02-10T11:44:41.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 -- 172.21.2.117:0/3291033323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12cc0067e0 msgr2=0x7f12cc008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 --2- 172.21.2.117:0/3291033323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12cc0067e0 0x7f12cc008d00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 rx=0x7f12d0002e00 tx=0x7f12d004dac0).stop 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12dfd75700 1 -- 172.21.2.117:0/3291033323 reap_dead start 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 -- 172.21.2.117:0/3291033323 shutdown_connections 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 -- 172.21.2.117:0/3291033323 >> 172.21.2.117:0/3291033323 conn(0x7f12e00ce4e0 msgr2=0x7f12e00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 -- 172.21.2.117:0/3291033323 shutdown_connections 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.083 7f12e596c700 1 -- 172.21.2.117:0/3291033323 wait complete. 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 Processor -- start 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 -- start start 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e010fe50 0x7f12e01106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12e0110d00 0x7f12e01a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0110250 con 0x7f12e01a3240 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e00a5030 con 0x7f12e0110d00 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12e596c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e00a5250 con 0x7f12e010fe50 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12dfd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12dfd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51142/0 (socket says 172.21.2.117:51142) 2021-02-10T11:44:41.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.098 7f12dfd75700 1 -- 172.21.2.117:0/767907015 learned_addr learned my addr 172.21.2.117:0/767907015 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:41.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 -- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e010fe50 msgr2=0x7f12e01106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e010fe50 0x7f12e01106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 -- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12e0110d00 msgr2=0x7f12e01a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12e0110d00 0x7f12e01a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 -- 172.21.2.117:0/767907015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e008ee40 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12dfd75700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 rx=0x7f12d800a8f0 tx=0x7f12d800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12d80070d0 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d80073d0 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12e596c700 1 -- 172.21.2.117:0/767907015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e008ee40 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12d80049a0 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.099 7f12e596c700 1 -- 172.21.2.117:0/767907015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e008ee40 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.100 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f12d8004720 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.100 7f12d7fff700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 0x7f12c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.100 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f12d8032a10 con 0x7f12e01a3240 2021-02-10T11:44:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.100 7f12df574700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 0x7f12c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:41.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.101 7f12df574700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 0x7f12c401abc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:41.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.131 7f12e596c700 1 -- 172.21.2.117:0/767907015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0085930 con 0x7f12e01a3240 2021-02-10T11:44:41.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.134 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12d8026f60 con 0x7f12e01a3240 2021-02-10T11:44:41.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.261 7f12e596c700 1 -- 172.21.2.117:0/767907015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e0085930 con 0x7f12e01a3240 2021-02-10T11:44:41.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.262 7f12d7fff700 1 -- 172.21.2.117:0/767907015 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2095 (secure 0 0 0) 0x7f12d8026f60 con 0x7f12e01a3240 2021-02-10T11:44:41.254 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:41.254 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:44:33.346284","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/319870470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":319870470},{"type":"v1","addr":"172.21.2.125:6835","nonce":319870470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:44:41.254 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 53 2021-02-10T11:44:41.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 -- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 msgr2=0x7f12c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 0x7f12c401abc0 crc :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 -- 172.21.2.117:0/767907015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 msgr2=0x7f12e01a3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 rx=0x7f12d800a8f0 tx=0x7f12d800ac90).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 -- 172.21.2.117:0/767907015 shutdown_connections 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12c40186c0 0x7f12c401abc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12e01a3240 0x7f12e01a3640 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12e0110d00 0x7f12e01a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 --2- 172.21.2.117:0/767907015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12e010fe50 0x7f12e01106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.263 7f12e596c700 1 -- 172.21.2.117:0/767907015 >> 172.21.2.117:0/767907015 conn(0x7f12e00ace80 msgr2=0x7f12e00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.264 7f12e596c700 1 -- 172.21.2.117:0/767907015 shutdown_connections 2021-02-10T11:44:41.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:41.264 7f12e596c700 1 -- 172.21.2.117:0/767907015 wait complete. 2021-02-10T11:44:41.466 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:31.340745', 'modified': '2021-02-10 11:44:33.346284', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8910': {'gid': 8910, 'name': 'b', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/319870470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 319870470}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 319870470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:41.466 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:41.466 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:41.865 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8910, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.531938359, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 53, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 73, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.176059961 2021-02-10T11:44:41.866 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:44:41.870 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8910, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.531938359, "mdsmap_epoch": 53, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 16.176059961 } 2021-02-10T11:44:41.870 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:44:41.870 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:44:41.870 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:44:41.870 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:41.871 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:42.205 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:42.548 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:42.548 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:42.892 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:42.892 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:42.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:42.893 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:43.227 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:44:43.227 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:43.229 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:43.573 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:43.574 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:43.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[141399]: starting ceph client 2021-02-10T11:44:43.641 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:44:43.649 7f9a8cfeff80 -1 init, newargv = 0x55df8e5aef90 newargc=9 2021-02-10T11:44:43.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[141399]: starting fuse 2021-02-10T11:44:43.917 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:43.917 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:43.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:43.918 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:44.253 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:44:44.254 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:44:44.254 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:44:44.254 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:44.255 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:44.256 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:44:44.256 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:44.256 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:44.256 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:44:44.256 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:44:44.618 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.141399.asok 2021-02-10T11:44:44.620 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.141399.asok 2021-02-10T11:44:44.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:44.621 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.141399.asok status 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "141399", 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9084, 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9084 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 622631750 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 622631750 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9084 172.21.2.117:0/622631750", 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/622631750", 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 53, 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 73, 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:44:45.012 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:45.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:45.014 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:45.014 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:45.350 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:45.350 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:45.350 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:45.351 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:45.697 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:44:46.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.113 7f16fd7cd700 1 Processor -- start 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.113 7f16fd7cd700 1 -- start start 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16fd7cd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16f80ef3f0 0x7f16f80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16fd7cd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f16f80eb7f0 0x7f16f81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16fd7cd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80ede60 con 0x7f16f80d18e0 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16fd7cd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80ef7f0 con 0x7f16f80ef3f0 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16fd7cd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80d1ce0 con 0x7f16f80eb7f0 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16f6d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16f80ef3f0 0x7f16f80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16f6d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f16f80ef3f0 0x7f16f80eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51170/0 (socket says 172.21.2.117:51170) 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.114 7f16f6d73700 1 -- 172.21.2.117:0/82964405 learned_addr learned my addr 172.21.2.117:0/82964405 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6d73700 1 -- 172.21.2.117:0/82964405 >> v2:172.21.2.125:3300/0 conn(0x7f16f80eb7f0 msgr2=0x7f16f81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6d73700 1 --2- 172.21.2.117:0/82964405 >> v2:172.21.2.125:3300/0 conn(0x7f16f80eb7f0 0x7f16f81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:46.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6d73700 1 -- 172.21.2.117:0/82964405 >> v1:172.21.2.125:6789/0 conn(0x7f16f80d18e0 legacy=0x7f16f80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6d73700 1 -- 172.21.2.117:0/82964405 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f81b8f90 con 0x7f16f80ef3f0 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6d73700 1 --2- 172.21.2.117:0/82964405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80ef3f0 0x7f16f80eb1d0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 rx=0x7f16e8006700 tx=0x7f16e8009b80).ready entity=mon.0 client_cookie=5147b43baff5e5ab server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6572700 1 -- 172.21.2.117:0/82964405 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e800c0c0 con 0x7f16f80ef3f0 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6572700 1 -- 172.21.2.117:0/82964405 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e800c400 con 0x7f16f80ef3f0 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16f6572700 1 -- 172.21.2.117:0/82964405 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e800c920 con 0x7f16f80ef3f0 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16fd7cd700 1 -- 172.21.2.117:0/82964405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80ef3f0 msgr2=0x7f16f80eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.115 7f16fd7cd700 1 --2- 172.21.2.117:0/82964405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80ef3f0 0x7f16f80eb1d0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 rx=0x7f16e8006700 tx=0x7f16e8009b80).stop 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 -- 172.21.2.117:0/82964405 shutdown_connections 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 --2- 172.21.2.117:0/82964405 >> v2:172.21.2.125:3300/0 conn(0x7f16f80eb7f0 0x7f16f81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 --2- 172.21.2.117:0/82964405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80ef3f0 0x7f16f80eb1d0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 -- 172.21.2.117:0/82964405 >> 172.21.2.117:0/82964405 conn(0x7f16f80ced60 msgr2=0x7f16f80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 -- 172.21.2.117:0/82964405 shutdown_connections 2021-02-10T11:44:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.116 7f16fd7cd700 1 -- 172.21.2.117:0/82964405 wait complete. 2021-02-10T11:44:46.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.130 7f16fd7cd700 1 Processor -- start 2021-02-10T11:44:46.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 -- start start 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 0x7f16f80e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16f80e9960 0x7f16f80e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8196810 con 0x7f16f80e9560 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8098ad0 con 0x7f16f8196390 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16fd7cd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80e8a40 con 0x7f16f80e9960 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16f6d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16f6d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51178/0 (socket says 172.21.2.117:51178) 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.131 7f16f6d73700 1 -- 172.21.2.117:0/842392808 learned_addr learned my addr 172.21.2.117:0/842392808 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f7d75700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16f80e9960 0x7f16f80e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f7574700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 0x7f16f80e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 -- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16f80e9960 msgr2=0x7f16f80e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16f80e9960 0x7f16f80e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 -- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 msgr2=0x7f16f80e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 0x7f16f80e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 -- 172.21.2.117:0/842392808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80e8c60 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f7574700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 0x7f16f80e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16f6d73700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f16e8006420 tx=0x7f16e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e80090d0 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e800bd50 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16e8005560 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f80e8c60 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.132 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f80e8c60 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.133 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16e8009440 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.133 7f16effff700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 0x7f16dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.133 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f16e8032930 con 0x7f16f80e9560 2021-02-10T11:44:46.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.133 7f16f7574700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 0x7f16dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.134 7f16f7574700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 0x7f16dc01abc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.164 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8087210 con 0x7f16f80e9560 2021-02-10T11:44:46.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.167 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16e8009030 con 0x7f16f80e9560 2021-02-10T11:44:46.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.291 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16f80ede60 con 0x7f16f80e9560 2021-02-10T11:44:46.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.292 7f16effff700 1 -- 172.21.2.117:0/842392808 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f16e8009440 con 0x7f16f80e9560 2021-02-10T11:44:46.285 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:46.285 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":73,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:44:32.342643","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:44:29.317633","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":15,"pool_name":"cephfs_data","create_time":"2021-02-10 11:44:30.338240","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:44:46.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.293 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 msgr2=0x7f16dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.293 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 0x7f16dc01abc0 crc :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.293 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 msgr2=0x7f16f80e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.293 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 rx=0x7f16e8006420 tx=0x7f16e800b960).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 shutdown_connections 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16dc0186c0 0x7f16dc01abc0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16f80e9560 0x7f16f80e7e00 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16f80e9960 0x7f16f80e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 --2- 172.21.2.117:0/842392808 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16f8196390 0x7f16f80e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 >> 172.21.2.117:0/842392808 conn(0x7f16f819f610 msgr2=0x7f16f81a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 shutdown_connections 2021-02-10T11:44:46.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.294 7f16fd7cd700 1 -- 172.21.2.117:0/842392808 wait complete. 2021-02-10T11:44:46.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:44:29.317633', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:44:30.338240', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:44:46.295 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.716 7f75aba2e700 1 Processor -- start 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.716 7f75aba2e700 1 -- start start 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.716 7f75aba2e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75a40ef3f0 0x7f75a40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75aba2e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f75a40eb830 0x7f75a41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75aba2e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40ef7f0 con 0x7f75a40d18f0 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75aba2e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a40ede60 con 0x7f75a40ef3f0 2021-02-10T11:44:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75aba2e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a40d1cf0 con 0x7f75a40eb830 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75a40ef3f0 0x7f75a40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75a40ef3f0 0x7f75a40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51184/0 (socket says 172.21.2.117:51184) 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 -- 172.21.2.117:0/717502372 learned_addr learned my addr 172.21.2.117:0/717502372 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a984a700 1 --1- 172.21.2.117:0/717502372 >> v1:172.21.2.117:6789/0 conn(0x7f75a40d18f0 0x7f75a40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51434/0 (socket says 172.21.2.117:51434) 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75aa04b700 1 --2- 172.21.2.117:0/717502372 >> v2:172.21.2.125:3300/0 conn(0x7f75a40eb830 0x7f75a41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 -- 172.21.2.117:0/717502372 >> v2:172.21.2.125:3300/0 conn(0x7f75a40eb830 msgr2=0x7f75a41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 --2- 172.21.2.117:0/717502372 >> v2:172.21.2.125:3300/0 conn(0x7f75a40eb830 0x7f75a41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 -- 172.21.2.117:0/717502372 >> v1:172.21.2.117:6789/0 conn(0x7f75a40d18f0 legacy=0x7f75a40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 -- 172.21.2.117:0/717502372 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a41b8fd0 con 0x7f75a40ef3f0 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.717 7f75a9049700 1 --2- 172.21.2.117:0/717502372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a40ef3f0 0x7f75a40eb210 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 rx=0x7f759c006a80 tx=0x7f759c009f00).ready entity=mon.0 client_cookie=5ad782da8a430670 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75a8848700 1 -- 172.21.2.117:0/717502372 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f759c00b5c0 con 0x7f75a40ef3f0 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75a8848700 1 -- 172.21.2.117:0/717502372 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c00b8c0 con 0x7f75a40ef3f0 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75a8848700 1 -- 172.21.2.117:0/717502372 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f759c00bda0 con 0x7f75a40ef3f0 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 -- 172.21.2.117:0/717502372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a40ef3f0 msgr2=0x7f75a40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 --2- 172.21.2.117:0/717502372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a40ef3f0 0x7f75a40eb210 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 rx=0x7f759c006a80 tx=0x7f759c009f00).stop 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 -- 172.21.2.117:0/717502372 shutdown_connections 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 --2- 172.21.2.117:0/717502372 >> v2:172.21.2.125:3300/0 conn(0x7f75a40eb830 0x7f75a41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 --2- 172.21.2.117:0/717502372 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a40ef3f0 0x7f75a40eb210 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.718 7f75aba2e700 1 -- 172.21.2.117:0/717502372 >> 172.21.2.117:0/717502372 conn(0x7f75a40cf060 msgr2=0x7f75a40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.719 7f75aba2e700 1 -- 172.21.2.117:0/717502372 shutdown_connections 2021-02-10T11:44:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.719 7f75aba2e700 1 -- 172.21.2.117:0/717502372 wait complete. 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.733 7f75aba2e700 1 Processor -- start 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.733 7f75aba2e700 1 -- start start 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 0x7f75a40dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75a41a4580 0x7f75a40dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4085e50 con 0x7f75a4083990 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4097270 con 0x7f75a41a4580 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aba2e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4097450 con 0x7f75a41a4180 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51192/0 (socket says 172.21.2.117:51192) 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 -- 172.21.2.117:0/3086210298 learned_addr learned my addr 172.21.2.117:0/3086210298 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a9049700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 0x7f75a40dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75aa04b700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75a41a4580 0x7f75a40dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 -- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 msgr2=0x7f75a40dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 0x7f75a40dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 -- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75a41a4580 msgr2=0x7f75a40dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75a41a4580 0x7f75a40dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a984a700 1 -- 172.21.2.117:0/3086210298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a409ed70 con 0x7f75a4083990 2021-02-10T11:44:46.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.734 7f75a9049700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 0x7f75a40dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f75a984a700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f759400a590 tx=0x7f759400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7594000ef0 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759400ad20 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7594004590 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a409ed70 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.735 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a409ed70 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.736 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f75940043a0 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.736 7f759a7fc700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 0x7f758c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.736 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f75940326d0 con 0x7f75a4083990 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.736 7f75a9049700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 0x7f758c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.737 7f75a9049700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 0x7f758c01abc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.767 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a4097450 con 0x7f75a4083990 2021-02-10T11:44:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.769 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7594030d30 con 0x7f75a4083990 2021-02-10T11:44:46.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.897 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f75a4097450 con 0x7f75a4083990 2021-02-10T11:44:46.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.898 7f759a7fc700 1 -- 172.21.2.117:0/3086210298 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v73) v1 ==== 92+0+10 (secure 0 0 0) 0x7f75940043a0 con 0x7f75a4083990 2021-02-10T11:44:46.891 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 msgr2=0x7f758c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 0x7f758c01abc0 crc :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 msgr2=0x7f75a40863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f759400a590 tx=0x7f759400a930).stop 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 shutdown_connections 2021-02-10T11:44:46.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75a41a4580 0x7f75a40dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75a41a4180 0x7f75a40dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f758c0186c0 0x7f758c01abc0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 --2- 172.21.2.117:0/3086210298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75a4083990 0x7f75a40863d0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 >> 172.21.2.117:0/3086210298 conn(0x7f75a40b6540 msgr2=0x7f75a408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 shutdown_connections 2021-02-10T11:44:46.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:46.900 7f75aba2e700 1 -- 172.21.2.117:0/3086210298 wait complete. 2021-02-10T11:44:46.900 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:44:47.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.321 7fdaa389c700 1 Processor -- start 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa389c700 1 -- start start 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa389c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fda9c0ef3f0 0x7fda9c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa389c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0ef7f0 con 0x7fda9c0eb830 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa389c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0ede60 con 0x7fda9c0d18f0 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa389c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c0d1da0 con 0x7fda9c0ef3f0 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa1eb9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fda9c0eb830 0x7fda9c1b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51448/0 (socket says 172.21.2.117:51448) 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.322 7fdaa1eb9700 1 -- 172.21.2.117:0/1187107200 learned_addr learned my addr 172.21.2.117:0/1187107200 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403163794 0 0) 0x7fda94001500 con 0x7fda9c0eb830 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda88002fe0 con 0x7fda9c0eb830 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa0eb7700 1 --2- 172.21.2.117:0/1187107200 >> v2:172.21.2.125:3300/0 conn(0x7fda9c0ef3f0 0x7fda9c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2183058566 0 0) 0x7fda94001460 con 0x7fda9c0eb830 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 >> v2:172.21.2.125:3300/0 conn(0x7fda9c0ef3f0 msgr2=0x7fda9c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 --2- 172.21.2.117:0/1187107200 >> v2:172.21.2.125:3300/0 conn(0x7fda9c0ef3f0 0x7fda9c0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 >> v1:172.21.2.125:6789/0 conn(0x7fda9c0d18f0 legacy=0x7fda9c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c1bb1c0 con 0x7fda9c0eb830 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fda94001980 con 0x7fda9c0eb830 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 >> v1:172.21.2.117:6789/0 conn(0x7fda9c0eb830 legacy=0x7fda9c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda88003a60 0x7fda88005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda88006790 0x7fda88008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 0x7fda8800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8800c300 con 0x7fda88009590 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8800c520 con 0x7fda88003a60 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8800c740 con 0x7fda88006790 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 0x7fda8800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa16b8700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda88003a60 0x7fda88005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa0eb7700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda88006790 0x7fda88008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 -- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda88006790 msgr2=0x7fda88008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda88006790 0x7fda88008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 -- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda88003a60 msgr2=0x7fda88005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda88003a60 0x7fda88005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa1eb9700 1 -- 172.21.2.117:0/1187107200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8800c9e0 con 0x7fda88009590 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.323 7fdaa16b8700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda88003a60 0x7fda88005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa1eb9700 1 -- 172.21.2.117:0/1187107200 reap_dead start 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa1eb9700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 0x7fda8800bab0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7fda94001ae0 tx=0x7fda9404e310).ready entity=mon.0 client_cookie=ad57cb9d88953306 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda94001460 con 0x7fda88009590 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9404e700 con 0x7fda88009590 2021-02-10T11:44:47.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fda8ffff700 1 -- 172.21.2.117:0/1187107200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda9404ec20 con 0x7fda88009590 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 -- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 msgr2=0x7fda8800bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 0x7fda8800bab0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 rx=0x7fda94001ae0 tx=0x7fda9404e310).stop 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 -- 172.21.2.117:0/1187107200 shutdown_connections 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 --2- 172.21.2.117:0/1187107200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda88009590 0x7fda8800bab0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 -- 172.21.2.117:0/1187107200 >> 172.21.2.117:0/1187107200 conn(0x7fda9c0cf060 msgr2=0x7fda9c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 -- 172.21.2.117:0/1187107200 shutdown_connections 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.324 7fdaa389c700 1 -- 172.21.2.117:0/1187107200 wait complete. 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 Processor -- start 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 -- start start 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda9c08b7f0 0x7fda9c08bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda9c08c210 0x7fda9c08c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c08cc30 con 0x7fda9c0b7a40 2021-02-10T11:44:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0b8460 con 0x7fda9c08b7f0 2021-02-10T11:44:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.339 7fdaa389c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c0b8680 con 0x7fda9c08c210 2021-02-10T11:44:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51208/0 (socket says 172.21.2.117:51208) 2021-02-10T11:44:47.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 -- 172.21.2.117:0/3426644825 learned_addr learned my addr 172.21.2.117:0/3426644825 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 -- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda9c08c210 msgr2=0x7fda9c08c610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda9c08c210 0x7fda9c08c610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 -- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda9c08b7f0 msgr2=0x7fda9c08bbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda9c08b7f0 0x7fda9c08bbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 -- 172.21.2.117:0/3426644825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c0b88a0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa1eb9700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7fda9404e8f0 tx=0x7fda9404ec90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda9404bd20 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda94006030 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9c0b88a0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c0b88a0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.340 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda94004720 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.341 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda94006210 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.341 7fda8dffb700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 0x7fda7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.341 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fda94076ca0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.341 7fdaa16b8700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 0x7fda7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.342 7fdaa16b8700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 0x7fda7c01abc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.372 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c08cc30 con 0x7fda9c0b7a40 2021-02-10T11:44:47.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.375 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda9404edd0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.501 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fda9c08cc30 con 0x7fda9c0b7a40 2021-02-10T11:44:47.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.501 7fda8dffb700 1 -- 172.21.2.117:0/3426644825 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v73) v1 ==== 104+0+10 (secure 0 0 0) 0x7fda94001ba0 con 0x7fda9c0b7a40 2021-02-10T11:44:47.494 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 msgr2=0x7fda7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 0x7fda7c01abc0 crc :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 msgr2=0x7fda9c0b7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 rx=0x7fda9404e8f0 tx=0x7fda9404ec90).stop 2021-02-10T11:44:47.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 shutdown_connections 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda7c018700 0x7fda7c01abc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda9c0b7a40 0x7fda9c0b7e40 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda9c08c210 0x7fda9c08c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 --2- 172.21.2.117:0/3426644825 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda9c08b7f0 0x7fda9c08bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.503 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 >> 172.21.2.117:0/3426644825 conn(0x7fda9c08ede0 msgr2=0x7fda9c0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.504 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 shutdown_connections 2021-02-10T11:44:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.504 7fdaa389c700 1 -- 172.21.2.117:0/3426644825 wait complete. 2021-02-10T11:44:47.504 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:44:47.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.925 7f096bc18700 1 Processor -- start 2021-02-10T11:44:47.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 -- start start 2021-02-10T11:44:47.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f09640ef3f0 0x7f09640eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09640eb830 0x7f09641b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640ef7f0 con 0x7f09640d18f0 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09640ede60 con 0x7f09640eb830 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096bc18700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09640d1cf0 con 0x7f09640ef3f0 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096a235700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09640eb830 0x7f09641b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096a235700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09640eb830 0x7f09641b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51218/0 (socket says 172.21.2.117:51218) 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.926 7f096a235700 1 -- 172.21.2.117:0/679371459 learned_addr learned my addr 172.21.2.117:0/679371459 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096a235700 1 -- 172.21.2.117:0/679371459 >> v2:172.21.2.125:3300/0 conn(0x7f09640ef3f0 msgr2=0x7f09640eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096a235700 1 --2- 172.21.2.117:0/679371459 >> v2:172.21.2.125:3300/0 conn(0x7f09640ef3f0 0x7f09640eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096a235700 1 -- 172.21.2.117:0/679371459 >> v1:172.21.2.125:6789/0 conn(0x7f09640d18f0 legacy=0x7f09640eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:47.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096a235700 1 -- 172.21.2.117:0/679371459 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09641b8fd0 con 0x7f09640eb830 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096a235700 1 --2- 172.21.2.117:0/679371459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640eb830 0x7f09641b79a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f095c006f00 tx=0x7f095c009f10).ready entity=mon.0 client_cookie=59f4e07b4fa97912 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f0968a32700 1 -- 172.21.2.117:0/679371459 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f095c00b3b0 con 0x7f09640eb830 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f0968a32700 1 -- 172.21.2.117:0/679371459 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c00b630 con 0x7f09640eb830 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f0968a32700 1 -- 172.21.2.117:0/679371459 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f095c00bb50 con 0x7f09640eb830 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096bc18700 1 -- 172.21.2.117:0/679371459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640eb830 msgr2=0x7f09641b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.927 7f096bc18700 1 --2- 172.21.2.117:0/679371459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640eb830 0x7f09641b79a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 rx=0x7f095c006f00 tx=0x7f095c009f10).stop 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 -- 172.21.2.117:0/679371459 shutdown_connections 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 --2- 172.21.2.117:0/679371459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640eb830 0x7f09641b79a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 --2- 172.21.2.117:0/679371459 >> v2:172.21.2.125:3300/0 conn(0x7f09640ef3f0 0x7f09640eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 -- 172.21.2.117:0/679371459 >> 172.21.2.117:0/679371459 conn(0x7f09640cf060 msgr2=0x7f09640cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 -- 172.21.2.117:0/679371459 shutdown_connections 2021-02-10T11:44:47.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.928 7f096bc18700 1 -- 172.21.2.117:0/679371459 wait complete. 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.942 7f096bc18700 1 Processor -- start 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 -- start start 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09640b4c10 0x7f09640aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09640b5010 0x7f09640ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09640b4320 con 0x7f09640b3e10 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096408a220 con 0x7f09640b5010 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f096bc18700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09640abbc0 con 0x7f09640b4c10 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51220/0 (socket says 172.21.2.117:51220) 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 -- 172.21.2.117:0/740743612 learned_addr learned my addr 172.21.2.117:0/740743612 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 -- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09640b4c10 msgr2=0x7f09640aaf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09640b4c10 0x7f09640aaf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 -- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09640b5010 msgr2=0x7f09640ab5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09640b5010 0x7f09640ab5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 -- 172.21.2.117:0/740743612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640abde0 con 0x7f09640b3e10 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.943 7f0969a34700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f096000b5b0 tx=0x7f096000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f096000bd40 con 0x7f09640b3e10 2021-02-10T11:44:47.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960006240 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0960005720 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f096bc18700 1 -- 172.21.2.117:0/740743612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09640abde0 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f096bc18700 1 -- 172.21.2.117:0/740743612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09640abde0 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.944 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09600054a0 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.945 7f095a7fc700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 0x7f094c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.945 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f09600326d0 con 0x7f09640b3e10 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.946 7f0969233700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 0x7f094c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:47.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.946 7f0969233700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 0x7f094c01abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:47.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.976 7f096bc18700 1 -- 172.21.2.117:0/740743612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096408a220 con 0x7f09640b3e10 2021-02-10T11:44:47.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:47.979 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09600224e0 con 0x7f09640b3e10 2021-02-10T11:44:48.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.105 7f096bc18700 1 -- 172.21.2.117:0/740743612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f096408a220 con 0x7f09640b3e10 2021-02-10T11:44:48.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.105 7f095a7fc700 1 -- 172.21.2.117:0/740743612 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v73) v1 ==== 100+0+10 (secure 0 0 0) 0x7f09600224e0 con 0x7f09640b3e10 2021-02-10T11:44:48.098 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 -- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 msgr2=0x7f094c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 0x7f094c01abc0 crc :-1 s=READY pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 -- 172.21.2.117:0/740743612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 msgr2=0x7f09640b45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 rx=0x7f096000b5b0 tx=0x7f096000b950).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 -- 172.21.2.117:0/740743612 shutdown_connections 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09640b5010 0x7f09640ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09640b4c10 0x7f09640aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f094c0186c0 0x7f094c01abc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 --2- 172.21.2.117:0/740743612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09640b3e10 0x7f09640b45f0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 -- 172.21.2.117:0/740743612 >> 172.21.2.117:0/740743612 conn(0x7f09640b6c70 msgr2=0x7f09640970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.107 7f096bc18700 1 -- 172.21.2.117:0/740743612 shutdown_connections 2021-02-10T11:44:48.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.108 7f096bc18700 1 -- 172.21.2.117:0/740743612 wait complete. 2021-02-10T11:44:48.108 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:44:48.108 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:44:48.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.524 7efcf2ad5700 1 Processor -- start 2021-02-10T11:44:48.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.524 7efcf2ad5700 1 -- start start 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcf2ad5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec0d18e0 0x7efcec0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcf2ad5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcec0eb000 0x7efcec0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcf2ad5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec0ede60 con 0x7efcec0ee3e0 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcf2ad5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0eba20 con 0x7efcec0eb000 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcf2ad5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0d1ce0 con 0x7efcec0d18e0 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcec0eb000 0x7efcec0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efcec0eb000 0x7efcec0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51228/0 (socket says 172.21.2.117:51228) 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 -- 172.21.2.117:0/3989725263 learned_addr learned my addr 172.21.2.117:0/3989725263 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 -- 172.21.2.117:0/3989725263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec0d18e0 msgr2=0x7efcec0ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 --2- 172.21.2.117:0/3989725263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec0d18e0 0x7efcec0ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 -- 172.21.2.117:0/3989725263 >> v1:172.21.2.125:6789/0 conn(0x7efcec0ee3e0 legacy=0x7efcec0ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:48.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.525 7efcebfff700 1 -- 172.21.2.117:0/3989725263 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0eefd0 con 0x7efcec0eb000 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcebfff700 1 --2- 172.21.2.117:0/3989725263 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0eb000 0x7efcec0eb400 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7efcdc006a80 tx=0x7efcdc009f00).ready entity=mon.0 client_cookie=921168c4a1a737ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efceb7fe700 1 -- 172.21.2.117:0/3989725263 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc00b5c0 con 0x7efcec0eb000 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efceb7fe700 1 -- 172.21.2.117:0/3989725263 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00b5c0 con 0x7efcec0eb000 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efceb7fe700 1 -- 172.21.2.117:0/3989725263 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcdc00b8c0 con 0x7efcec0eb000 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 -- 172.21.2.117:0/3989725263 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0eb000 msgr2=0x7efcec0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 --2- 172.21.2.117:0/3989725263 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0eb000 0x7efcec0eb400 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 rx=0x7efcdc006a80 tx=0x7efcdc009f00).stop 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 -- 172.21.2.117:0/3989725263 shutdown_connections 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 --2- 172.21.2.117:0/3989725263 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0eb000 0x7efcec0eb400 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 --2- 172.21.2.117:0/3989725263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec0d18e0 0x7efcec0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 -- 172.21.2.117:0/3989725263 >> 172.21.2.117:0/3989725263 conn(0x7efcec0ced60 msgr2=0x7efcec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 -- 172.21.2.117:0/3989725263 shutdown_connections 2021-02-10T11:44:48.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.526 7efcf2ad5700 1 -- 172.21.2.117:0/3989725263 wait complete. 2021-02-10T11:44:48.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 Processor -- start 2021-02-10T11:44:48.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 -- start start 2021-02-10T11:44:48.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec08c770 0x7efcec08cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcec0e7970 0x7efcec0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec08e2c0 con 0x7efcec0e7d70 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0b51f0 con 0x7efcec0e7970 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf2ad5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec08d440 con 0x7efcec08c770 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf10f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf10f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51232/0 (socket says 172.21.2.117:51232) 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.541 7efcf10f2700 1 -- 172.21.2.117:0/3029115297 learned_addr learned my addr 172.21.2.117:0/3029115297 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf08f1700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec08c770 0x7efcec08cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcebfff700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcec0e7970 0x7efcec0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 -- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec08c770 msgr2=0x7efcec08cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec08c770 0x7efcec08cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 -- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcec0e7970 msgr2=0x7efcec0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcec0e7970 0x7efcec0b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 -- 172.21.2.117:0/3029115297 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0e8e40 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf10f2700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7efce400b5b0 tx=0x7efce400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efce400bd40 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce4006240 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec0e8e40 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec0e93d0 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.542 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efce4005720 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.543 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efce40054a0 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.543 7efce97fa700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 0x7efcd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.543 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efce4032970 con 0x7efcec0e7d70 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.543 7efcf08f1700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 0x7efcd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:48.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.544 7efcf08f1700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 0x7efcd001abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:48.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.575 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec087290 con 0x7efcec0e7d70 2021-02-10T11:44:48.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.577 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efce4000bc0 con 0x7efcec0e7d70 2021-02-10T11:44:48.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.703 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcec0ede60 con 0x7efcec0e7d70 2021-02-10T11:44:48.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.703 7efce97fa700 1 -- 172.21.2.117:0/3029115297 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+2095 (secure 0 0 0) 0x7efce4006280 con 0x7efcec0e7d70 2021-02-10T11:44:48.696 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:44:48.696 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:44:33.346284","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/319870470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":319870470},{"type":"v1","addr":"172.21.2.125:6835","nonce":319870470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:44:48.696 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 53 2021-02-10T11:44:48.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 msgr2=0x7efcd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 0x7efcd001abc0 crc :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 msgr2=0x7efcec0e8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:48.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 rx=0x7efce400b5b0 tx=0x7efce400b950).stop 2021-02-10T11:44:48.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 shutdown_connections 2021-02-10T11:44:48.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcd00186c0 0x7efcd001abc0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcec0e7d70 0x7efcec0e8820 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcec0e7970 0x7efcec0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 --2- 172.21.2.117:0/3029115297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcec08c770 0x7efcec08cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 >> 172.21.2.117:0/3029115297 conn(0x7efcec1a3620 msgr2=0x7efcec0a0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 shutdown_connections 2021-02-10T11:44:48.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:48.705 7efcf2ad5700 1 -- 172.21.2.117:0/3029115297 wait complete. 2021-02-10T11:44:48.908 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:44:31.340745', 'modified': '2021-02-10 11:44:33.346284', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8910': {'gid': 8910, 'name': 'b', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/319870470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 319870470}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 319870470}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:44:48.908 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:44:48.909 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8910, 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:44:49.310 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 17.976384161999999, 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 53, 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 73, 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 73, 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.620505747999999 2021-02-10T11:44:49.311 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:44:49.314 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8910, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.976384161999999, "mdsmap_epoch": 53, "osdmap_epoch": 73, "osdmap_epoch_barrier": 73, "uptime": 23.620505747999999 } 2021-02-10T11:44:49.314 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:44:49.315 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:44:49.315 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:44:49.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:49.315 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:49.650 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:49.992 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:49.993 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:50.335 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:50.335 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:50.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:50.336 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:50.670 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:44:50.671 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:44:50.671 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:50.674 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:44:51.017 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:44:51.018 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:44:51.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[142039]: starting ceph client 2021-02-10T11:44:51.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:44:51.090 7fbafc542f80 -1 init, newargv = 0x5614019b8f30 newargc=9 2021-02-10T11:44:51.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[142039]: starting fuse 2021-02-10T11:44:51.360 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:44:51.360 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:44:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:44:51.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:51.362 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:44:51.697 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:44:51.697 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:44:51.697 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:44:51.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:44:51.698 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:44:51.699 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:44:52.059 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.142039.asok 2021-02-10T11:44:52.062 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.142039.asok 2021-02-10T11:44:52.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:52.062 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.142039.asok status 2021-02-10T11:44:52.449 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "142039", 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:52.450 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9126, 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9126 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2006118375 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:52.451 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2006118375 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9126 172.21.2.117:0/2006118375", 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2006118375", 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 53, 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 73, 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:44:52.452 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:52.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:52.455 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:52.455 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:52.789 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:52.789 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:44:52.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:52.790 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:44:53.136 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 Processor -- start 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 -- start start 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f73fc0e4e40 0x7f73fc1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc0e5240 0x7f73fc084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f73fc1b0810 0x7f73fc085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc0e5cc0 con 0x7f73fc1b0810 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73fc085930 con 0x7f73fc0e4e40 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f74029a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc084840 con 0x7f73fc0e5240 2021-02-10T11:44:53.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f7400fc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f73fc1b0810 0x7f73fc085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f7400fc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f73fc1b0810 0x7f73fc085280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51252/0 (socket says 172.21.2.117:51252) 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.557 7f7400fc2700 1 -- 172.21.2.117:0/2957043912 learned_addr learned my addr 172.21.2.117:0/2957043912 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 -- 172.21.2.117:0/2957043912 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc0e5240 msgr2=0x7f73fc084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 --2- 172.21.2.117:0/2957043912 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc0e5240 0x7f73fc084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 -- 172.21.2.117:0/2957043912 >> v2:172.21.2.125:3300/0 conn(0x7f73fc0e4e40 msgr2=0x7f73fc1b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 --2- 172.21.2.117:0/2957043912 >> v2:172.21.2.125:3300/0 conn(0x7f73fc0e4e40 0x7f73fc1b01f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 -- 172.21.2.117:0/2957043912 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc085eb0 con 0x7f73fc1b0810 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f7400fc2700 1 --2- 172.21.2.117:0/2957043912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc1b0810 0x7f73fc085280 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f73f4006f00 tx=0x7f73f4009f10).ready entity=mon.0 client_cookie=990809ced95a2b5a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f73fb7fe700 1 -- 172.21.2.117:0/2957043912 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f400b3b0 con 0x7f73fc1b0810 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f73fb7fe700 1 -- 172.21.2.117:0/2957043912 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f400b630 con 0x7f73fc1b0810 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f73fb7fe700 1 -- 172.21.2.117:0/2957043912 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f400bb50 con 0x7f73fc1b0810 2021-02-10T11:44:53.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f74029a5700 1 -- 172.21.2.117:0/2957043912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc1b0810 msgr2=0x7f73fc085280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.558 7f74029a5700 1 --2- 172.21.2.117:0/2957043912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc1b0810 0x7f73fc085280 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 rx=0x7f73f4006f00 tx=0x7f73f4009f10).stop 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 -- 172.21.2.117:0/2957043912 shutdown_connections 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 --2- 172.21.2.117:0/2957043912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc1b0810 0x7f73fc085280 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 --2- 172.21.2.117:0/2957043912 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc0e5240 0x7f73fc084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 --2- 172.21.2.117:0/2957043912 >> v2:172.21.2.125:3300/0 conn(0x7f73fc0e4e40 0x7f73fc1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 -- 172.21.2.117:0/2957043912 >> 172.21.2.117:0/2957043912 conn(0x7f73fc0ce4e0 msgr2=0x7f73fc0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 -- 172.21.2.117:0/2957043912 shutdown_connections 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.559 7f74029a5700 1 -- 172.21.2.117:0/2957043912 wait complete. 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.573 7f74029a5700 1 Processor -- start 2021-02-10T11:44:53.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 -- start start 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc194e10 0x7f73fc1915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73fc191bf0 0x7f73fc096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc195290 con 0x7f73fc0973f0 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b1460 con 0x7f73fc191bf0 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f74029a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b1680 con 0x7f73fc194e10 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f7400fc2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f7400fc2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51258/0 (socket says 172.21.2.117:51258) 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.574 7f7400fc2700 1 -- 172.21.2.117:0/2873310681 learned_addr learned my addr 172.21.2.117:0/2873310681 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 -- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc194e10 msgr2=0x7f73fc1915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc194e10 0x7f73fc1915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 -- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73fc191bf0 msgr2=0x7f73fc096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73fc191bf0 0x7f73fc096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 -- 172.21.2.117:0/2873310681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0b18a0 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f7400fc2700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7f73f4009390 tx=0x7f73f400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f4009450 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4006960 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f74029a5700 1 -- 172.21.2.117:0/2873310681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0b18a0 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f74029a5700 1 -- 172.21.2.117:0/2873310681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0b1e30 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.575 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73f400c4e0 con 0x7f73fc0973f0 2021-02-10T11:44:53.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.576 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73f4001270 con 0x7f73fc0973f0 2021-02-10T11:44:53.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.576 7f73f97fa700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 0x7f73e001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:53.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.576 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f73f4032b10 con 0x7f73fc0973f0 2021-02-10T11:44:53.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.576 7f73fbfff700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 0x7f73e001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:53.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.577 7f73fbfff700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 0x7f73e001abd0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:53.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.607 7f74029a5700 1 -- 172.21.2.117:0/2873310681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc085930 con 0x7f73fc0973f0 2021-02-10T11:44:53.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.609 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73f4002f70 con 0x7f73fc0973f0 2021-02-10T11:44:53.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:53.744 7f74029a5700 1 -- 172.21.2.117:0/2873310681 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f73fc085930 con 0x7f73fc0973f0 2021-02-10T11:44:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.176 7f73f97fa700 1 -- 172.21.2.117:0/2873310681 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v54) v1 ==== 136+0+0 (secure 0 0 0) 0x7f73f4001270 con 0x7f73fc0973f0 2021-02-10T11:44:54.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 -- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 msgr2=0x7f73e001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 0x7f73e001abd0 crc :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 -- 172.21.2.117:0/2873310681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 msgr2=0x7f73fc0977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 rx=0x7f73f4009390 tx=0x7f73f400bd40).stop 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 -- 172.21.2.117:0/2873310681 shutdown_connections 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73e00186d0 0x7f73e001abd0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73fc0973f0 0x7f73fc0977f0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73fc191bf0 0x7f73fc096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 --2- 172.21.2.117:0/2873310681 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73fc194e10 0x7f73fc1915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.212 7f74029a5700 1 -- 172.21.2.117:0/2873310681 >> 172.21.2.117:0/2873310681 conn(0x7f73fc18cc70 msgr2=0x7f73fc093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.213 7f74029a5700 1 -- 172.21.2.117:0/2873310681 shutdown_connections 2021-02-10T11:44:54.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.213 7f74029a5700 1 -- 172.21.2.117:0/2873310681 wait complete. 2021-02-10T11:44:54.214 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:44:54.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.631 7f9e0bde9700 1 Processor -- start 2021-02-10T11:44:54.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.631 7f9e0bde9700 1 -- start start 2021-02-10T11:44:54.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e041b07f0 0x7f9e040e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e040e7d70 0x7f9e040e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9e040e8790 0x7f9e040e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e040e92f0 con 0x7f9e040e7d70 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e040e5ce0 con 0x7f9e040e8790 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0bde9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040e77f0 con 0x7f9e041b07f0 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e09404700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e040e7d70 0x7f9e040e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e09404700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9e040e7d70 0x7f9e040e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51264/0 (socket says 172.21.2.117:51264) 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e09404700 1 -- 172.21.2.117:0/1757224225 learned_addr learned my addr 172.21.2.117:0/1757224225 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:54.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e0a406700 1 --2- 172.21.2.117:0/1757224225 >> v2:172.21.2.125:3300/0 conn(0x7f9e040e8790 0x7f9e040e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.632 7f9e09c05700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e041b07f0 0x7f9e040e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 -- 172.21.2.117:0/1757224225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e041b07f0 msgr2=0x7f9e040e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e041b07f0 0x7f9e040e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 -- 172.21.2.117:0/1757224225 >> v2:172.21.2.125:3300/0 conn(0x7f9e040e8790 msgr2=0x7f9e040e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 --2- 172.21.2.117:0/1757224225 >> v2:172.21.2.125:3300/0 conn(0x7f9e040e8790 0x7f9e040e8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 -- 172.21.2.117:0/1757224225 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040e54b0 con 0x7f9e040e7d70 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e09404700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e040e7d70 0x7f9e040e8170 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f9df4006700 tx=0x7f9df4009b80).ready entity=mon.0 client_cookie=fee10cb85693ee69 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e08c03700 1 -- 172.21.2.117:0/1757224225 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df400c0c0 con 0x7f9e040e7d70 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e08c03700 1 -- 172.21.2.117:0/1757224225 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df400c440 con 0x7f9e040e7d70 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e08c03700 1 -- 172.21.2.117:0/1757224225 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9df400c8a0 con 0x7f9e040e7d70 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e0bde9700 1 -- 172.21.2.117:0/1757224225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e040e7d70 msgr2=0x7f9e040e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.633 7f9e0bde9700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e040e7d70 0x7f9e040e8170 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f9df4006700 tx=0x7f9df4009b80).stop 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.634 7f9e0bde9700 1 -- 172.21.2.117:0/1757224225 shutdown_connections 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.634 7f9e0bde9700 1 --2- 172.21.2.117:0/1757224225 >> v2:172.21.2.125:3300/0 conn(0x7f9e040e8790 0x7f9e040e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.634 7f9e0bde9700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e040e7d70 0x7f9e040e8170 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.634 7f9e0bde9700 1 --2- 172.21.2.117:0/1757224225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e041b07f0 0x7f9e040e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.634 7f9e0bde9700 1 -- 172.21.2.117:0/1757224225 >> 172.21.2.117:0/1757224225 conn(0x7f9e040ce500 msgr2=0x7f9e040e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.635 7f9e0bde9700 1 -- 172.21.2.117:0/1757224225 shutdown_connections 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.635 7f9e0bde9700 1 -- 172.21.2.117:0/1757224225 wait complete. 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 Processor -- start 2021-02-10T11:44:54.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 -- start start 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e040935b0 0x7f9e040939b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 0x7f9e041a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e040a4f50 con 0x7f9e0418f200 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e04094060 con 0x7f9e041a6d60 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.649 7f9e0bde9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e041a7780 con 0x7f9e040935b0 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51270/0 (socket says 172.21.2.117:51270) 2021-02-10T11:44:54.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 -- 172.21.2.117:0/2456698402 learned_addr learned my addr 172.21.2.117:0/2456698402 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e0a406700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 0x7f9e041a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09404700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e040935b0 0x7f9e040939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 -- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e040935b0 msgr2=0x7f9e040939b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e040935b0 0x7f9e040939b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 -- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 msgr2=0x7f9e041a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 0x7f9e041a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 -- 172.21.2.117:0/2456698402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e041a7960 con 0x7f9e0418f200 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e0a406700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 0x7f9e041a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e09c05700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f9e00006f60 tx=0x7f9e0000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:54.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e000009b0 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0000bdf0 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9e00002030 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.650 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e041a7960 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.651 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e041a7960 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.651 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9e00006b00 con 0x7f9e0418f200 2021-02-10T11:44:54.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.651 7f9dfa7fc700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.651 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9e00032510 con 0x7f9e0418f200 2021-02-10T11:44:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.651 7f9e09404700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.652 7f9e09404700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 0x7f9dec01abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:54.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.682 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0410c290 con 0x7f9e0418f200 2021-02-10T11:44:54.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.685 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9e00006340 con 0x7f9e0418f200 2021-02-10T11:44:54.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.818 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f9e040e77f0 con 0x7f9dec0186c0 2021-02-10T11:44:54.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.820 7f9dfa7fc700 1 -- 172.21.2.117:0/2456698402 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9df4006430 con 0x7f9dec0186c0 2021-02-10T11:44:54.814 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:44:54.815 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:44:54.815 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:44:54.815 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:54.815 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 msgr2=0x7f9dec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 0x7f9dec01abc0 crc :-1 s=READY pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 msgr2=0x7f9e040bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 rx=0x7f9e00006f60 tx=0x7f9e0000b970).stop 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 shutdown_connections 2021-02-10T11:44:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9e041a6d60 0x7f9e041a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9e040935b0 0x7f9e040939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9dec0186c0 0x7f9dec01abc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 --2- 172.21.2.117:0/2456698402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9e0418f200 0x7f9e040bb860 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 >> 172.21.2.117:0/2456698402 conn(0x7f9e040a7260 msgr2=0x7f9e040bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 shutdown_connections 2021-02-10T11:44:54.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:54.823 7f9e0bde9700 1 -- 172.21.2.117:0/2456698402 wait complete. 2021-02-10T11:44:54.825 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.247 7fad4447f700 1 Processor -- start 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.247 7fad4447f700 1 -- start start 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad3c0e7180 0x7fad3c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c0e7580 con 0x7fad3c0e8390 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c0e77f0 con 0x7fad3c1b07f0 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3c0e5ce0 con 0x7fad3c0e7180 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad3c0e7180 0x7fad3c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad3c0e7180 0x7fad3c0e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51278/0 (socket says 172.21.2.117:51278) 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad42a9c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fad3c0e8390 0x7fad3c0e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51532/0 (socket says 172.21.2.117:51532) 2021-02-10T11:44:55.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 -- 172.21.2.117:0/270908712 learned_addr learned my addr 172.21.2.117:0/270908712 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 -- 172.21.2.117:0/270908712 >> v1:172.21.2.125:6789/0 conn(0x7fad3c1b07f0 legacy=0x7fad3c0e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 -- 172.21.2.117:0/270908712 >> v1:172.21.2.117:6789/0 conn(0x7fad3c0e8390 legacy=0x7fad3c0e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 -- 172.21.2.117:0/270908712 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c0e8f80 con 0x7fad3c0e7180 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41a9a700 1 --2- 172.21.2.117:0/270908712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0e7180 0x7fad3c0e7d70 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7fad30006700 tx=0x7fad30009b80).ready entity=mon.0 client_cookie=12cc4b91f9b9e323 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41299700 1 -- 172.21.2.117:0/270908712 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad3000c0c0 con 0x7fad3c0e7180 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41299700 1 -- 172.21.2.117:0/270908712 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3000c340 con 0x7fad3c0e7180 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad41299700 1 -- 172.21.2.117:0/270908712 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad3000c820 con 0x7fad3c0e7180 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 -- 172.21.2.117:0/270908712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0e7180 msgr2=0x7fad3c0e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.248 7fad4447f700 1 --2- 172.21.2.117:0/270908712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0e7180 0x7fad3c0e7d70 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 rx=0x7fad30006700 tx=0x7fad30009b80).stop 2021-02-10T11:44:55.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.249 7fad4447f700 1 -- 172.21.2.117:0/270908712 shutdown_connections 2021-02-10T11:44:55.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.249 7fad4447f700 1 --2- 172.21.2.117:0/270908712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0e7180 0x7fad3c0e7d70 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.249 7fad4447f700 1 -- 172.21.2.117:0/270908712 >> 172.21.2.117:0/270908712 conn(0x7fad3c0ce500 msgr2=0x7fad3c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:55.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.249 7fad4447f700 1 -- 172.21.2.117:0/270908712 shutdown_connections 2021-02-10T11:44:55.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.249 7fad4447f700 1 -- 172.21.2.117:0/270908712 wait complete. 2021-02-10T11:44:55.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 Processor -- start 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 -- start start 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad3c08ba00 0x7fad3c0a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad3c0c3b20 0x7fad3c1c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0aa260 con 0x7fad3c0a9d50 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c08bf10 con 0x7fad3c08ba00 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad4447f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c1c9bf0 con 0x7fad3c0c3b20 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad41a9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad41a9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51286/0 (socket says 172.21.2.117:51286) 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.264 7fad41a9a700 1 -- 172.21.2.117:0/618955992 learned_addr learned my addr 172.21.2.117:0/618955992 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 -- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad3c0c3b20 msgr2=0x7fad3c1c95d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:55.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad3c0c3b20 0x7fad3c1c95d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 -- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad3c08ba00 msgr2=0x7fad3c0a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad3c08ba00 0x7fad3c0a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 -- 172.21.2.117:0/618955992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c1c9e10 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad41a9a700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7fad3000b580 tx=0x7fad3000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad3000bd10 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30000c90 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad300022b0 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad4447f700 1 -- 172.21.2.117:0/618955992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c1c9e10 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.265 7fad4447f700 1 -- 172.21.2.117:0/618955992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c1c9e10 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.266 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad30009f90 con 0x7fad3c0a9d50 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.266 7fad2effd700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 0x7fad2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.266 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fad30032540 con 0x7fad3c0a9d50 2021-02-10T11:44:55.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.266 7fad4229b700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 0x7fad2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.267 7fad4229b700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 0x7fad2401abc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.298 7fad4447f700 1 -- 172.21.2.117:0/618955992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c07bfc0 con 0x7fad3c0a9d50 2021-02-10T11:44:55.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.300 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad30025880 con 0x7fad3c0a9d50 2021-02-10T11:44:55.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.450 7fad4447f700 1 -- 172.21.2.117:0/618955992 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fad3c0e91f0 con 0x7fad3c0a9d50 2021-02-10T11:44:55.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.451 7fad2effd700 1 -- 172.21.2.117:0/618955992 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad30025880 con 0x7fad3c0a9d50 2021-02-10T11:44:55.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 msgr2=0x7fad2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 0x7fad2401abc0 crc :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 msgr2=0x7fad3c1c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 rx=0x7fad3000b580 tx=0x7fad3000b920).stop 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 shutdown_connections 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad240186c0 0x7fad2401abc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad3c0a9d50 0x7fad3c1c8fb0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad3c0c3b20 0x7fad3c1c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 --2- 172.21.2.117:0/618955992 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad3c08ba00 0x7fad3c0a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 >> 172.21.2.117:0/618955992 conn(0x7fad3c07e630 msgr2=0x7fad3c07d620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 shutdown_connections 2021-02-10T11:44:55.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.453 7fad4447f700 1 -- 172.21.2.117:0/618955992 wait complete. 2021-02-10T11:44:55.455 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000071660 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 Processor -- start 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 -- start start 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 0x7febdc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7febdc0eb040 0x7febdc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0ede60 con 0x7febdc0ee3e0 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febdc0eba60 con 0x7febdc0eb040 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febe0d1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc0d1cf0 con 0x7febdc0d18f0 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febdb314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7febdc0ee3e0 0x7febdc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51546/0 (socket says 172.21.2.117:51546) 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.875 7febdb314700 1 -- 172.21.2.117:0/3575351669 learned_addr learned my addr 172.21.2.117:0/3575351669 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:55.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.876 7febdab13700 1 --2- 172.21.2.117:0/3575351669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 0x7febdc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.876 7febda312700 1 --2- 172.21.2.117:0/3575351669 >> v2:172.21.2.125:3300/0 conn(0x7febdc0eb040 0x7febdc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.876 7febd9b11700 1 -- 172.21.2.117:0/3575351669 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199145903 0 0) 0x7febd4001500 con 0x7febdc0ee3e0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.876 7febd9b11700 1 -- 172.21.2.117:0/3575351669 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8002fe0 con 0x7febdc0ee3e0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febdab13700 1 -- 172.21.2.117:0/3575351669 >> v2:172.21.2.125:3300/0 conn(0x7febdc0eb040 msgr2=0x7febdc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febdab13700 1 --2- 172.21.2.117:0/3575351669 >> v2:172.21.2.125:3300/0 conn(0x7febdc0eb040 0x7febdc0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febdab13700 1 -- 172.21.2.117:0/3575351669 >> v1:172.21.2.117:6789/0 conn(0x7febdc0ee3e0 legacy=0x7febdc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febdab13700 1 -- 172.21.2.117:0/3575351669 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0eefd0 con 0x7febdc0d18f0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febdab13700 1 --2- 172.21.2.117:0/3575351669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 0x7febdc0eaa20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7febc4006650 tx=0x7febc4009a80).ready entity=mon.2 client_cookie=d87fa04ee3f03266 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febd9b11700 1 -- 172.21.2.117:0/3575351669 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc400c0c0 con 0x7febdc0d18f0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febd9b11700 1 -- 172.21.2.117:0/3575351669 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc400c340 con 0x7febdc0d18f0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febd9b11700 1 -- 172.21.2.117:0/3575351669 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febc400c860 con 0x7febdc0d18f0 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febe0d1a700 1 -- 172.21.2.117:0/3575351669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 msgr2=0x7febdc0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.877 7febe0d1a700 1 --2- 172.21.2.117:0/3575351669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 0x7febdc0eaa20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7febc4006650 tx=0x7febc4009a80).stop 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 -- 172.21.2.117:0/3575351669 shutdown_connections 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 --2- 172.21.2.117:0/3575351669 >> v2:172.21.2.125:3300/0 conn(0x7febdc0eb040 0x7febdc0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 --2- 172.21.2.117:0/3575351669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc0d18f0 0x7febdc0eaa20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 -- 172.21.2.117:0/3575351669 >> 172.21.2.117:0/3575351669 conn(0x7febdc0cf060 msgr2=0x7febdc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 -- 172.21.2.117:0/3575351669 shutdown_connections 2021-02-10T11:44:55.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.878 7febe0d1a700 1 -- 172.21.2.117:0/3575351669 wait complete. 2021-02-10T11:44:55.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 Processor -- start 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 -- start start 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc083990 0x7febdc0863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febdc1a4580 0x7febdc0dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc085e50 con 0x7febdc1a4180 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc097270 con 0x7febdc1a4580 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.893 7febe0d1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc097450 con 0x7febdc083990 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51302/0 (socket says 172.21.2.117:51302) 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 -- 172.21.2.117:0/2594118515 learned_addr learned my addr 172.21.2.117:0/2594118515 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:55.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febdb314700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febdc1a4580 0x7febdc0dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febdab13700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc083990 0x7febdc0863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 -- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc083990 msgr2=0x7febdc0863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc083990 0x7febdc0863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 -- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febdc1a4580 msgr2=0x7febdc0dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febdc1a4580 0x7febdc0dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 -- 172.21.2.117:0/2594118515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc09ed70 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febda312700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7febcc006f60 tx=0x7febcc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.894 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febcc00bd60 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febcc000c90 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7febcc0022b0 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc09ed70 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc09f2c0 con 0x7febdc1a4180 2021-02-10T11:44:55.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7febcc006b00 con 0x7febdc1a4180 2021-02-10T11:44:55.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.895 7febd37fe700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 0x7febc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:55.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.896 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7febcc0326d0 con 0x7febdc1a4180 2021-02-10T11:44:55.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.896 7febdab13700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 0x7febc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:55.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.896 7febdab13700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 0x7febc001abc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:55.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.925 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc09fb50 con 0x7febdc1a4180 2021-02-10T11:44:55.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:55.928 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7febcc000960 con 0x7febdc1a4180 2021-02-10T11:44:56.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.053 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""]}) v1 -- 0x7febdc09f2c0 con 0x7febc00186c0 2021-02-10T11:44:56.050 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:44:56.063 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:56.051 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:44:56.063 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:56.051 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:44:56.063 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:56.051 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:44:56.063 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:56.051 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:44:56.063 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:44:56.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.087 7febd37fe700 1 -- 172.21.2.117:0/2594118515 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7febc4000940 con 0x7febc00186c0 2021-02-10T11:44:56.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.119 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 msgr2=0x7febc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.119 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 0x7febc001abc0 crc :-1 s=READY pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.119 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 msgr2=0x7febdc0dbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.119 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 rx=0x7febcc006f60 tx=0x7febcc00b970).stop 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 shutdown_connections 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7febc00186c0 0x7febc001abc0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7febdc1a4180 0x7febdc0dbe40 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7febdc1a4580 0x7febdc0dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 --2- 172.21.2.117:0/2594118515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7febdc083990 0x7febdc0863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 >> 172.21.2.117:0/2594118515 conn(0x7febdc0b6540 msgr2=0x7febdc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 shutdown_connections 2021-02-10T11:44:56.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.120 7febe0d1a700 1 -- 172.21.2.117:0/2594118515 wait complete. 2021-02-10T11:44:56.121 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000071660 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.542 7eff17ebf700 1 Processor -- start 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.542 7eff17ebf700 1 -- start start 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff17ebf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff100d18f0 0x7eff100eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff17ebf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff100ef3f0 0x7eff101b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff17ebf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100ede60 con 0x7eff100eb040 2021-02-10T11:44:56.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff17ebf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff10088020 con 0x7eff100ef3f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff17ebf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff100d1d30 con 0x7eff100d18f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff164dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff100ef3f0 0x7eff101b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff154da700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7eff100eb040 0x7eff100eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51556/0 (socket says 172.21.2.117:51556) 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff154da700 1 -- 172.21.2.117:0/1186028129 learned_addr learned my addr 172.21.2.117:0/1186028129 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff164dc700 1 -- 172.21.2.117:0/1186028129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff100d18f0 msgr2=0x7eff100eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff164dc700 1 --2- 172.21.2.117:0/1186028129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff100d18f0 0x7eff100eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff164dc700 1 -- 172.21.2.117:0/1186028129 >> v1:172.21.2.117:6789/0 conn(0x7eff100eb040 legacy=0x7eff100eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.543 7eff164dc700 1 -- 172.21.2.117:0/1186028129 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff101b8fd0 con 0x7eff100ef3f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff164dc700 1 --2- 172.21.2.117:0/1186028129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100ef3f0 0x7eff101b79a0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7eff08006e30 tx=0x7eff08009e60).ready entity=mon.0 client_cookie=ad9bde4271cd05fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff14cd9700 1 -- 172.21.2.117:0/1186028129 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff08006a50 con 0x7eff100ef3f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff14cd9700 1 -- 172.21.2.117:0/1186028129 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0800b300 con 0x7eff100ef3f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff14cd9700 1 -- 172.21.2.117:0/1186028129 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff0800b7e0 con 0x7eff100ef3f0 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 -- 172.21.2.117:0/1186028129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100ef3f0 msgr2=0x7eff101b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 --2- 172.21.2.117:0/1186028129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100ef3f0 0x7eff101b79a0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 rx=0x7eff08006e30 tx=0x7eff08009e60).stop 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 -- 172.21.2.117:0/1186028129 shutdown_connections 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 --2- 172.21.2.117:0/1186028129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100ef3f0 0x7eff101b79a0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 --2- 172.21.2.117:0/1186028129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff100d18f0 0x7eff100eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.544 7eff17ebf700 1 -- 172.21.2.117:0/1186028129 >> 172.21.2.117:0/1186028129 conn(0x7eff100cf060 msgr2=0x7eff100cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.545 7eff17ebf700 1 -- 172.21.2.117:0/1186028129 shutdown_connections 2021-02-10T11:44:56.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.545 7eff17ebf700 1 -- 172.21.2.117:0/1186028129 wait complete. 2021-02-10T11:44:56.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.559 7eff17ebf700 1 Processor -- start 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.559 7eff17ebf700 1 -- start start 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff1007c7c0 0x7eff101a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff101a3930 0x7eff100a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1007cca0 con 0x7eff100a13a0 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1018fb70 con 0x7eff101a3930 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff17ebf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff100dc6d0 con 0x7eff1007c7c0 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51312/0 (socket says 172.21.2.117:51312) 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 -- 172.21.2.117:0/2172792826 learned_addr learned my addr 172.21.2.117:0/2172792826 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 -- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff1007c7c0 msgr2=0x7eff101a3310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:56.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff1007c7c0 0x7eff101a3310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 -- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff101a3930 msgr2=0x7eff100a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff101a3930 0x7eff100a0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 -- 172.21.2.117:0/2172792826 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1007d220 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff164dc700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7eff0800b970 tx=0x7eff0800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.560 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff08009380 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff08005820 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff08005d40 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1007d220 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1007d220 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff08005a00 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff027fc700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 0x7efef801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.561 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7eff08032c80 con 0x7eff100a13a0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.562 7eff15cdb700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 0x7efef801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:56.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.562 7eff15cdb700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 0x7efef801ac10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:56.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.593 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff100daff0 con 0x7eff100a13a0 2021-02-10T11:44:56.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.595 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff08009380 con 0x7eff100a13a0 2021-02-10T11:44:56.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.725 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""]}) v1 -- 0x7eff100daff0 con 0x7efef8018730 2021-02-10T11:44:56.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.729 7eff027fc700 1 -- 172.21.2.117:0/2172792826 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7eff0c006970 con 0x7efef8018730 2021-02-10T11:44:56.726 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000071660/f031223c-e3e5-429e-81cf-b94f70fa5e21 2021-02-10T11:44:56.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 msgr2=0x7efef801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 0x7efef801ac10 crc :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 msgr2=0x7eff100a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:56.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 rx=0x7eff0800b970 tx=0x7eff0800bd10).stop 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 shutdown_connections 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efef8018730 0x7efef801ac10 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff100a13a0 0x7eff100a17a0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff101a3930 0x7eff100a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 --2- 172.21.2.117:0/2172792826 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff1007c7c0 0x7eff101a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 >> 172.21.2.117:0/2172792826 conn(0x7eff10087090 msgr2=0x7eff1008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 shutdown_connections 2021-02-10T11:44:56.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:56.735 7eff17ebf700 1 -- 172.21.2.117:0/2172792826 wait complete. 2021-02-10T11:44:56.736 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000071660 with 1 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000071660/f031223c-e3e5-429e-81cf-b94f70fa5e21 2021-02-10T11:44:56.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:44:56.737 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:44:56.737 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:44:57.071 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:44:57.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:44:57.072 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000071660/f031223c-e3e5-429e-81cf-b94f70fa5e21/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:44:57.459 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:44:57.460 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:44:57.460 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252473 s, 41.5 MB/s 2021-02-10T11:44:57.463 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000071660 snapshot_0000000000938350 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.889 7f97e448d700 1 Processor -- start 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.889 7f97e448d700 1 -- start start 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e448d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97dc0e4e40 0x7f97dc1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e448d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc0e5240 0x7f97dc084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e448d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97dc085930 con 0x7f97dc1b0810 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e448d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97dc084840 con 0x7f97dc0e4e40 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e448d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc084a30 con 0x7f97dc0e5240 2021-02-10T11:44:57.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97dc0e4e40 0x7f97dc1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97dc0e4e40 0x7f97dc1b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51320/0 (socket says 172.21.2.117:51320) 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 -- 172.21.2.117:0/2907717959 learned_addr learned my addr 172.21.2.117:0/2907717959 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 -- 172.21.2.117:0/2907717959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc0e5240 msgr2=0x7f97dc084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 --2- 172.21.2.117:0/2907717959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc0e5240 0x7f97dc084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 -- 172.21.2.117:0/2907717959 >> v1:172.21.2.125:6789/0 conn(0x7f97dc1b0810 legacy=0x7f97dc085eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.890 7f97e22a9700 1 -- 172.21.2.117:0/2907717959 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc085280 con 0x7f97dc0e4e40 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e22a9700 1 --2- 172.21.2.117:0/2907717959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc0e4e40 0x7f97dc1b01f0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f97cc006700 tx=0x7f97cc009b80).ready entity=mon.0 client_cookie=1537a8561d3b0246 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e12a7700 1 -- 172.21.2.117:0/2907717959 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97cc00c0c0 con 0x7f97dc0e4e40 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e12a7700 1 -- 172.21.2.117:0/2907717959 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc00c340 con 0x7f97dc0e4e40 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e12a7700 1 -- 172.21.2.117:0/2907717959 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97cc00c860 con 0x7f97dc0e4e40 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e448d700 1 -- 172.21.2.117:0/2907717959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc0e4e40 msgr2=0x7f97dc1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:57.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.891 7f97e448d700 1 --2- 172.21.2.117:0/2907717959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc0e4e40 0x7f97dc1b01f0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 rx=0x7f97cc006700 tx=0x7f97cc009b80).stop 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 -- 172.21.2.117:0/2907717959 shutdown_connections 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 --2- 172.21.2.117:0/2907717959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc0e5240 0x7f97dc084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 --2- 172.21.2.117:0/2907717959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc0e4e40 0x7f97dc1b01f0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 -- 172.21.2.117:0/2907717959 >> 172.21.2.117:0/2907717959 conn(0x7f97dc0ce4e0 msgr2=0x7f97dc0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 -- 172.21.2.117:0/2907717959 shutdown_connections 2021-02-10T11:44:57.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.892 7f97e448d700 1 -- 172.21.2.117:0/2907717959 wait complete. 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 Processor -- start 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 -- start start 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f97dc1b1570 0x7f97dc1b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc1b1f90 0x7f97dc1b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1110e0 con 0x7f97dc094390 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1b0ff0 con 0x7f97dc1b1570 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.906 7f97e448d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc094db0 con 0x7f97dc1b1f90 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51330/0 (socket says 172.21.2.117:51330) 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 -- 172.21.2.117:0/419222208 learned_addr learned my addr 172.21.2.117:0/419222208 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:57.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 -- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc1b1f90 msgr2=0x7f97dc1b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc1b1f90 0x7f97dc1b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 -- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f97dc1b1570 msgr2=0x7f97dc1b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f97dc1b1570 0x7f97dc1b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 -- 172.21.2.117:0/419222208 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc094f90 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e2aaa700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f97d4009940 tx=0x7f97d400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d4003030 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d4001270 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e448d700 1 -- 172.21.2.117:0/419222208 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc094f90 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f97d4001790 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.907 7f97e448d700 1 -- 172.21.2.117:0/419222208 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc095520 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.908 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f97d4001450 con 0x7f97dc094390 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.908 7f97d2ffd700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 0x7f97c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:57.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.908 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f97d4032b70 con 0x7f97dc094390 2021-02-10T11:44:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.908 7f97e22a9700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 0x7f97c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.909 7f97e22a9700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 0x7f97c401abc0 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.939 7f97e448d700 1 -- 172.21.2.117:0/419222208 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc10c2f0 con 0x7f97dc094390 2021-02-10T11:44:57.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:57.942 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f97d4000f40 con 0x7f97dc094390 2021-02-10T11:44:58.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.070 7f97e448d700 1 -- 172.21.2.117:0/419222208 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""], "snap_name": "snapshot_0000000000938350"}) v1 -- 0x7f97dc1b0ff0 con 0x7f97c40186c0 2021-02-10T11:44:58.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.081 7f97d2ffd700 1 -- 172.21.2.117:0/419222208 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f97cc006390 con 0x7f97c40186c0 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 -- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 msgr2=0x7f97c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 0x7f97c401abc0 crc :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 -- 172.21.2.117:0/419222208 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 msgr2=0x7f97dc094790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 rx=0x7f97d4009940 tx=0x7f97d400bcf0).stop 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 -- 172.21.2.117:0/419222208 shutdown_connections 2021-02-10T11:44:58.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.086 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97c40186c0 0x7f97c401abc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97dc094390 0x7f97dc094790 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97dc1b1f90 0x7f97dc1b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 --2- 172.21.2.117:0/419222208 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f97dc1b1570 0x7f97dc1b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 -- 172.21.2.117:0/419222208 >> 172.21.2.117:0/419222208 conn(0x7f97dc1b3150 msgr2=0x7f97dc1ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 -- 172.21.2.117:0/419222208 shutdown_connections 2021-02-10T11:44:58.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.087 7f97e448d700 1 -- 172.21.2.117:0/419222208 wait complete. 2021-02-10T11:44:58.088 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000071660 snapshot_0000000000938350 clone_0000000000862156 2021-02-10T11:44:58.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.508 7f4ae2220700 1 Processor -- start 2021-02-10T11:44:58.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.508 7f4ae2220700 1 -- start start 2021-02-10T11:44:58.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae2220700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae2220700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae2220700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc0ede60 con 0x7f4adc0d18e0 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae2220700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc0ef7f0 con 0x7f4adc0eb7f0 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae2220700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0d1ce0 con 0x7f4adc0ef3f0 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae083d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4adb7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4adbfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:44510/0 (socket says 172.21.2.117:44510) 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4adbfff700 1 -- 172.21.2.117:0/3318219529 learned_addr learned my addr 172.21.2.117:0/3318219529 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4ae083d700 1 --2- 172.21.2.117:0/3318219529 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51334/0 (socket says 172.21.2.117:51334) 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4adb7fe700 1 --2- 172.21.2.117:0/3318219529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:49014/0 (socket says 172.21.2.117:49014) 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.509 7f4adb7fe700 1 -- 172.21.2.117:0/3318219529 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 msgr2=0x7f4adc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:58.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adb7fe700 1 --2- 172.21.2.117:0/3318219529 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adb7fe700 1 -- 172.21.2.117:0/3318219529 >> v1:172.21.2.125:6789/0 conn(0x7f4adc0d18e0 legacy=0x7f4adc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adb7fe700 1 -- 172.21.2.117:0/3318219529 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc1b8f90 con 0x7f4adc0ef3f0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adaffd700 1 -- 172.21.2.117:0/3318219529 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045987402 0 0) 0x7f4acc001190 con 0x7f4adc0d18e0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4ae083d700 1 --2- 172.21.2.117:0/3318219529 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adb7fe700 1 --2- 172.21.2.117:0/3318219529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f4ad0006650 tx=0x7f4ad0009a80).ready entity=mon.2 client_cookie=89044581cacfb405 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adaffd700 1 -- 172.21.2.117:0/3318219529 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad00009b0 con 0x7f4adc0ef3f0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adaffd700 1 -- 172.21.2.117:0/3318219529 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad000c0c0 con 0x7f4adc0ef3f0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4ae2220700 1 -- 172.21.2.117:0/3318219529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 msgr2=0x7f4adc0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4ae2220700 1 --2- 172.21.2.117:0/3318219529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f4ad0006650 tx=0x7f4ad0009a80).stop 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.510 7f4adaffd700 1 -- 172.21.2.117:0/3318219529 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad000c510 con 0x7f4adc0ef3f0 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 -- 172.21.2.117:0/3318219529 shutdown_connections 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 --2- 172.21.2.117:0/3318219529 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0eb7f0 0x7f4adc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 --2- 172.21.2.117:0/3318219529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc0eb1d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 -- 172.21.2.117:0/3318219529 >> 172.21.2.117:0/3318219529 conn(0x7f4adc0ced60 msgr2=0x7f4adc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:58.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 -- 172.21.2.117:0/3318219529 shutdown_connections 2021-02-10T11:44:58.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.511 7f4ae2220700 1 -- 172.21.2.117:0/3318219529 wait complete. 2021-02-10T11:44:58.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.525 7f4ae2220700 1 Processor -- start 2021-02-10T11:44:58.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.525 7f4ae2220700 1 -- start start 2021-02-10T11:44:58.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc0901d0 0x7f4adc090e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc091450 0x7f4adc091850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0905d0 con 0x7f4adc08f380 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc08f780 con 0x7f4adc0901d0 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae2220700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc1a5f10 con 0x7f4adc091450 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51342/0 (socket says 172.21.2.117:51342) 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adbfff700 1 -- 172.21.2.117:0/2434597944 learned_addr learned my addr 172.21.2.117:0/2434597944 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adb7fe700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc0901d0 0x7f4adc090e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4ae083d700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc091450 0x7f4adc091850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adbfff700 1 -- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc091450 msgr2=0x7f4adc091850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.526 7f4adbfff700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc091450 0x7f4adc091850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4adbfff700 1 -- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc0901d0 msgr2=0x7f4adc090e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4adbfff700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc0901d0 0x7f4adc090e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4adbfff700 1 -- 172.21.2.117:0/2434597944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc1a83c0 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4adbfff700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f4acc00a590 tx=0x7f4acc00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4acc0070d0 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc000c10 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4acc004530 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc1a83c0 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.527 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc092160 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.528 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4acc007fe0 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.528 7f4ad8ff9700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 0x7f4abc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.528 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4acc0326b0 con 0x7f4adc08f380 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.529 7f4adb7fe700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 0x7f4abc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:58.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.529 7f4adb7fe700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 0x7f4abc01abc0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:58.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.559 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc07f0a0 con 0x7f4adc08f380 2021-02-10T11:44:58.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.561 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4acc030d10 con 0x7f4adc08f380 2021-02-10T11:44:58.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.689 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""], "snap_name": "snapshot_0000000000938350", "target_sub_name": "clone_0000000000862156"}) v1 -- 0x7f4adc0ede60 con 0x7f4abc0186c0 2021-02-10T11:44:58.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.732 7f4ad8ff9700 1 -- 172.21.2.117:0/2434597944 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4ad0009180 con 0x7f4abc0186c0 2021-02-10T11:44:58.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 msgr2=0x7f4abc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 0x7f4abc01abc0 crc :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 msgr2=0x7f4adc08fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 rx=0x7f4acc00a590 tx=0x7f4acc00a930).stop 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 shutdown_connections 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc091450 0x7f4adc091850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc0901d0 0x7f4adc090e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4abc0186c0 0x7f4abc01abc0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 --2- 172.21.2.117:0/2434597944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc08f380 0x7f4adc08fbb0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 >> 172.21.2.117:0/2434597944 conn(0x7f4adc197410 msgr2=0x7f4adc07f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:58.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.755 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 shutdown_connections 2021-02-10T11:44:58.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:58.756 7f4ae2220700 1 -- 172.21.2.117:0/2434597944 wait complete. 2021-02-10T11:44:58.757 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs clone_0000000000862156 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 Processor -- start 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 -- start start 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff9e80d18f0 0x7ff9e80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 0x7ff9e80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80edee0 con 0x7ff9e80ee3e0 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e80eba60 con 0x7ff9e80d18f0 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9f0526700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80d1cf0 con 0x7ff9e80eb040 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9eeb43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9eeb43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51348/0 (socket says 172.21.2.117:51348) 2021-02-10T11:44:59.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.178 7ff9eeb43700 1 -- 172.21.2.117:0/2807455019 learned_addr learned my addr 172.21.2.117:0/2807455019 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9ee342700 1 --2- 172.21.2.117:0/2807455019 >> v2:172.21.2.125:3300/0 conn(0x7ff9e80d18f0 0x7ff9e80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9edb41700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 0x7ff9e80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 -- 172.21.2.117:0/2807455019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 msgr2=0x7ff9e80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 0x7ff9e80eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 -- 172.21.2.117:0/2807455019 >> v2:172.21.2.125:3300/0 conn(0x7ff9e80d18f0 msgr2=0x7ff9e80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 --2- 172.21.2.117:0/2807455019 >> v2:172.21.2.125:3300/0 conn(0x7ff9e80d18f0 0x7ff9e80eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 -- 172.21.2.117:0/2807455019 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80eee00 con 0x7ff9e80ee3e0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9edb41700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 0x7ff9e80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9eeb43700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7ff9e0006f00 tx=0x7ff9e0009f10).ready entity=mon.0 client_cookie=e40b9fdcde2dc74a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9ed340700 1 -- 172.21.2.117:0/2807455019 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e000b3b0 con 0x7ff9e80ee3e0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9ed340700 1 -- 172.21.2.117:0/2807455019 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0000d30 con 0x7ff9e80ee3e0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9ed340700 1 -- 172.21.2.117:0/2807455019 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e000b7e0 con 0x7ff9e80ee3e0 2021-02-10T11:44:59.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9f0526700 1 -- 172.21.2.117:0/2807455019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e80ee3e0 msgr2=0x7ff9e80ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.179 7ff9f0526700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 rx=0x7ff9e0006f00 tx=0x7ff9e0009f10).stop 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 -- 172.21.2.117:0/2807455019 shutdown_connections 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e80ee3e0 0x7ff9e80ee7e0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 --2- 172.21.2.117:0/2807455019 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e80eb040 0x7ff9e80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 --2- 172.21.2.117:0/2807455019 >> v2:172.21.2.125:3300/0 conn(0x7ff9e80d18f0 0x7ff9e80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 -- 172.21.2.117:0/2807455019 >> 172.21.2.117:0/2807455019 conn(0x7ff9e80cf060 msgr2=0x7ff9e80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 -- 172.21.2.117:0/2807455019 shutdown_connections 2021-02-10T11:44:59.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.180 7ff9f0526700 1 -- 172.21.2.117:0/2807455019 wait complete. 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 Processor -- start 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 -- start start 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9e81a4180 0x7ff9e80dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e81a4580 0x7ff9e80dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8085e50 con 0x7ff9e8083990 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8097270 con 0x7ff9e81a4180 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9f0526700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8097450 con 0x7ff9e81a4580 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9ee342700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9ee342700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51354/0 (socket says 172.21.2.117:51354) 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.195 7ff9ee342700 1 -- 172.21.2.117:0/3438300895 learned_addr learned my addr 172.21.2.117:0/3438300895 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9edb41700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9e81a4180 0x7ff9e80dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9eeb43700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e81a4580 0x7ff9e80dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 -- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e81a4580 msgr2=0x7ff9e80dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e81a4580 0x7ff9e80dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 -- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9e81a4180 msgr2=0x7ff9e80dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9e81a4180 0x7ff9e80dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 -- 172.21.2.117:0/3438300895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e809ed70 con 0x7ff9e8083990 2021-02-10T11:44:59.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9ee342700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7ff9e4006f00 tx=0x7ff9e400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e400bd30 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4002030 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9e4002490 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e809ed70 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.196 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e809ed70 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.197 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9e4000c90 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.197 7ff9deffd700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 0x7ff9d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.197 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff9e40327d0 con 0x7ff9e8083990 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.197 7ff9edb41700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 0x7ff9d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.198 7ff9edb41700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 0x7ff9d001abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.228 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e8097450 con 0x7ff9e8083990 2021-02-10T11:44:59.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.230 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9e401f540 con 0x7ff9e8083990 2021-02-10T11:44:59.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.364 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "clone_0000000000862156", "target": ["mgr", ""]}) v1 -- 0x7ff9e8097450 con 0x7ff9d00186c0 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.367 7ff9deffd700 1 -- 172.21.2.117:0/3438300895 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+46 (crc 0 0 0) 0x7ff9d8006540 con 0x7ff9d00186c0 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stdout: "status": { 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "complete" 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:44:59.360 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:44:59.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 msgr2=0x7ff9d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 0x7ff9d001abc0 crc :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 msgr2=0x7ff9e80863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 rx=0x7ff9e4006f00 tx=0x7ff9e400b940).stop 2021-02-10T11:44:59.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 shutdown_connections 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9e81a4580 0x7ff9e80dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9e81a4180 0x7ff9e80dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d00186c0 0x7ff9d001abc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.368 7ff9f0526700 1 --2- 172.21.2.117:0/3438300895 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9e8083990 0x7ff9e80863d0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.369 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 >> 172.21.2.117:0/3438300895 conn(0x7ff9e80b6540 msgr2=0x7ff9e808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.369 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 shutdown_connections 2021-02-10T11:44:59.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.369 7ff9f0526700 1 -- 172.21.2.117:0/3438300895 wait complete. 2021-02-10T11:44:59.370 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000071660 snapshot_0000000000938350 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.793 7fd8a17b9700 1 Processor -- start 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.793 7fd8a17b9700 1 -- start start 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd8a17b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd89c0e5240 0x7fd89c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd8a17b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c084c60 0x7fd89c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd8a17b9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89c085930 con 0x7fd89c0e4e40 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd8a17b9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89c085680 con 0x7fd89c0e5240 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd8a17b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c1b09e0 con 0x7fd89c084c60 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89b574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd89c0e4e40 0x7fd89c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51612/0 (socket says 172.21.2.117:51612) 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89b574700 1 -- 172.21.2.117:0/1961057648 learned_addr learned my addr 172.21.2.117:0/1961057648 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:44:59.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 --2- 172.21.2.117:0/1961057648 >> v2:172.21.2.117:3300/0 conn(0x7fd89c0e5240 0x7fd89c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89a572700 1 -- 172.21.2.117:0/1961057648 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472567013 0 0) 0x7fd8840009b0 con 0x7fd89c0e4e40 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89a572700 1 -- 172.21.2.117:0/1961057648 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd888002fe0 con 0x7fd89c0e4e40 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89bd75700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c084c60 0x7fd89c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 -- 172.21.2.117:0/1961057648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c084c60 msgr2=0x7fd89c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c084c60 0x7fd89c085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 -- 172.21.2.117:0/1961057648 >> v1:172.21.2.117:6789/0 conn(0x7fd89c0e4e40 legacy=0x7fd89c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 -- 172.21.2.117:0/1961057648 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c084840 con 0x7fd89c0e5240 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.794 7fd89ad73700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0e5240 0x7fd89c086ec0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 rx=0x7fd88c006650 tx=0x7fd88c009a00).ready entity=mon.0 client_cookie=d9b59e8c5ea529eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd89a572700 1 -- 172.21.2.117:0/1961057648 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd88c009df0 con 0x7fd89c0e5240 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd89a572700 1 -- 172.21.2.117:0/1961057648 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c0009b0 con 0x7fd89c0e5240 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd89a572700 1 -- 172.21.2.117:0/1961057648 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd88c00c2b0 con 0x7fd89c0e5240 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 -- 172.21.2.117:0/1961057648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0e5240 msgr2=0x7fd89c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0e5240 0x7fd89c086ec0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 rx=0x7fd88c006650 tx=0x7fd88c009a00).stop 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 -- 172.21.2.117:0/1961057648 shutdown_connections 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c084c60 0x7fd89c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 --2- 172.21.2.117:0/1961057648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0e5240 0x7fd89c086ec0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 -- 172.21.2.117:0/1961057648 >> 172.21.2.117:0/1961057648 conn(0x7fd89c0ce4e0 msgr2=0x7fd89c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 -- 172.21.2.117:0/1961057648 shutdown_connections 2021-02-10T11:44:59.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.795 7fd8a17b9700 1 -- 172.21.2.117:0/1961057648 wait complete. 2021-02-10T11:44:59.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 Processor -- start 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 -- start start 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 0x7fd89c097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c0adbd0 0x7fd89c09b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c18d0c0 con 0x7fd89c0ad1b0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c09ae70 con 0x7fd89c0973b0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd8a17b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c09ba10 con 0x7fd89c0adbd0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd89ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd89bd75700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c0adbd0 0x7fd89c09b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd89ad73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51372/0 (socket says 172.21.2.117:51372) 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd89ad73700 1 -- 172.21.2.117:0/3491796023 learned_addr learned my addr 172.21.2.117:0/3491796023 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.810 7fd89b574700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 0x7fd89c097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 -- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c0adbd0 msgr2=0x7fd89c09b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c0adbd0 0x7fd89c09b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 -- 172.21.2.117:0/3491796023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 msgr2=0x7fd89c0ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 -- 172.21.2.117:0/3491796023 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c09bbf0 con 0x7fd89c0973b0 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89ad73700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd89b574700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 0x7fd89c097980 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fd88400a4a0 tx=0x7fd88400a840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd884000d50 con 0x7fd89c0973b0 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88400ac30 con 0x7fd89c0973b0 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd884004350 con 0x7fd89c0973b0 2021-02-10T11:44:59.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c09bbf0 con 0x7fd89c0973b0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.811 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c09c180 con 0x7fd89c0973b0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.812 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd884000d30 con 0x7fd89c0973b0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.812 7fd893fff700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 0x7fd88001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.812 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd884032480 con 0x7fd89c0973b0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.812 7fd89ad73700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 0x7fd88001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:44:59.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.813 7fd89ad73700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 0x7fd88001abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:44:59.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.844 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c085930 con 0x7fd89c0973b0 2021-02-10T11:44:59.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.847 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd884000940 con 0x7fd89c0973b0 2021-02-10T11:44:59.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.977 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""], "snap_name": "snapshot_0000000000938350"}) v1 -- 0x7fd89c09c180 con 0x7fd8800186c0 2021-02-10T11:44:59.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.988 7fd893fff700 1 -- 172.21.2.117:0/3491796023 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fd88c009620 con 0x7fd8800186c0 2021-02-10T11:44:59.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 msgr2=0x7fd88001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 0x7fd88001abc0 crc :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 msgr2=0x7fd89c097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 0x7fd89c097980 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fd88400a4a0 tx=0x7fd88400a840).stop 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 shutdown_connections 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8800186c0 0x7fd88001abc0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd89c0ad1b0 0x7fd89c0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd89c0adbd0 0x7fd89c09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 --2- 172.21.2.117:0/3491796023 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd89c0973b0 0x7fd89c097980 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 >> 172.21.2.117:0/3491796023 conn(0x7fd89c18cc70 msgr2=0x7fd89c093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:44:59.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 shutdown_connections 2021-02-10T11:44:59.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:44:59.995 7fd8a17b9700 1 -- 172.21.2.117:0/3491796023 wait complete. 2021-02-10T11:44:59.997 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume info cephfs clone_0000000000862156 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.417 7f1db2269700 1 Processor -- start 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.417 7f1db2269700 1 -- start start 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db2269700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1dac0d18f0 0x7f1dac0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db2269700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1dac0eb040 0x7f1dac0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db2269700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0ede60 con 0x7f1dac0ee3e0 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db2269700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0ebad0 con 0x7f1dac0eb040 2021-02-10T11:45:00.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db2269700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0d1cf0 con 0x7f1dac0d18f0 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1dab7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1dac0eb040 0x7f1dac0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db0886700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1dac0ee3e0 0x7f1dac0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51626/0 (socket says 172.21.2.117:51626) 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.418 7f1db0886700 1 -- 172.21.2.117:0/3250267182 learned_addr learned my addr 172.21.2.117:0/3250267182 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dabfff700 1 --2- 172.21.2.117:0/3250267182 >> v2:172.21.2.125:3300/0 conn(0x7f1dac0d18f0 0x7f1dac0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dab7fe700 1 -- 172.21.2.117:0/3250267182 >> v2:172.21.2.125:3300/0 conn(0x7f1dac0d18f0 msgr2=0x7f1dac0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dab7fe700 1 --2- 172.21.2.117:0/3250267182 >> v2:172.21.2.125:3300/0 conn(0x7f1dac0d18f0 0x7f1dac0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dab7fe700 1 -- 172.21.2.117:0/3250267182 >> v1:172.21.2.117:6789/0 conn(0x7f1dac0ee3e0 legacy=0x7f1dac0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dab7fe700 1 -- 172.21.2.117:0/3250267182 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac0eefd0 con 0x7f1dac0eb040 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1dab7fe700 1 --2- 172.21.2.117:0/3250267182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac0eb040 0x7f1dac0eb440 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f1da0006650 tx=0x7f1da0009a80).ready entity=mon.0 client_cookie=c3063e6b91c2661c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1daaffd700 1 -- 172.21.2.117:0/3250267182 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da00009b0 con 0x7f1dac0eb040 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1daaffd700 1 -- 172.21.2.117:0/3250267182 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da000c0c0 con 0x7f1dac0eb040 2021-02-10T11:45:00.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1daaffd700 1 -- 172.21.2.117:0/3250267182 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da000c5e0 con 0x7f1dac0eb040 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1db2269700 1 -- 172.21.2.117:0/3250267182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac0eb040 msgr2=0x7f1dac0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.419 7f1db2269700 1 --2- 172.21.2.117:0/3250267182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac0eb040 0x7f1dac0eb440 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 rx=0x7f1da0006650 tx=0x7f1da0009a80).stop 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 -- 172.21.2.117:0/3250267182 shutdown_connections 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 --2- 172.21.2.117:0/3250267182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac0eb040 0x7f1dac0eb440 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 --2- 172.21.2.117:0/3250267182 >> v2:172.21.2.125:3300/0 conn(0x7f1dac0d18f0 0x7f1dac0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 -- 172.21.2.117:0/3250267182 >> 172.21.2.117:0/3250267182 conn(0x7f1dac0cf060 msgr2=0x7f1dac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 -- 172.21.2.117:0/3250267182 shutdown_connections 2021-02-10T11:45:00.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.420 7f1db2269700 1 -- 172.21.2.117:0/3250267182 wait complete. 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.435 7f1db2269700 1 Processor -- start 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.435 7f1db2269700 1 -- start start 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 0x7f1dac1a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 0x7f1dac097310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0a95d0 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0a1a50 con 0x7f1dac1a79d0 2021-02-10T11:45:00.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db2269700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac0a1660 con 0x7f1dac0a1260 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51380/0 (socket says 172.21.2.117:51380) 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 -- 172.21.2.117:0/4216356858 learned_addr learned my addr 172.21.2.117:0/4216356858 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 -- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 msgr2=0x7f1dac1a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1dabfff700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 0x7f1dac1a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1dab7fe700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 0x7f1dac097310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 0x7f1dac1a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 -- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 msgr2=0x7f1dac097310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 0x7f1dac097310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1db0886700 1 -- 172.21.2.117:0/4216356858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac097f50 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1dabfff700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 0x7f1dac1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.436 7f1dab7fe700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 0x7f1dac097310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:00.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1db0886700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7f1da400a940 tx=0x7f1da400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da4001270 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da40070b0 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1da4007540 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1db2269700 1 -- 172.21.2.117:0/4216356858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac097f50 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.437 7f1db2269700 1 -- 172.21.2.117:0/4216356858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac097f50 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.438 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1da4007290 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.438 7f1da8ff9700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 0x7f1d8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.438 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1da4032a50 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.438 7f1dabfff700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 0x7f1d8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:00.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.439 7f1dabfff700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 0x7f1d8c01abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:00.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.468 7f1db2269700 1 -- 172.21.2.117:0/4216356858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac0ede60 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.471 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1da4007cb0 con 0x7f1dac1a7dd0 2021-02-10T11:45:00.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.600 7f1db2269700 1 -- 172.21.2.117:0/4216356858 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume info", "vol_name": "cephfs", "sub_name": "clone_0000000000862156", "target": ["mgr", ""]}) v1 -- 0x7f1dac0ede60 con 0x7f1d8c0186c0 2021-02-10T11:45:00.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.606 7f1da8ff9700 1 -- 172.21.2.117:0/4216356858 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+724 (crc 0 0 0) 0x7f1d9c006ae0 con 0x7f1d8c0186c0 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "atime": "2021-02-10 11:44:56", 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_pcent": "undefined", 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_quota": "infinite", 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "bytes_used": 0, 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "created_at": "2021-02-10 11:44:58", 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "ctime": "2021-02-10 11:44:58", 2021-02-10T11:45:00.601 INFO:teuthology.orchestra.run.gibba017.stdout: "data_pool": "cephfs_data", 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "features": [ 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snapshot-clone", 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snapshot-autoprotect", 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snapshot-retention" 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "gid": 0, 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "mode": 16877, 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "mon_addrs": [ 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "172.21.2.117:6789", 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "172.21.2.125:6789", 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: "172.21.2.125:6790" 2021-02-10T11:45:00.602 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "mtime": "2021-02-10 11:44:57", 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "path": "/volumes/_nogroup/clone_0000000000862156/67c9ac7d-6342-456b-a050-6ee947fef71d", 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_namespace": "", 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "state": "complete", 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "clone", 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout: "uid": 0 2021-02-10T11:45:00.603 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 -- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 msgr2=0x7f1d8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 0x7f1d8c01abc0 crc :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 -- 172.21.2.117:0/4216356858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 msgr2=0x7f1dac097930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 rx=0x7f1da400a940 tx=0x7f1da400ace0).stop 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 -- 172.21.2.117:0/4216356858 shutdown_connections 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1d8c0186c0 0x7f1d8c01abc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1dac1a7dd0 0x7f1dac097930 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1dac1a79d0 0x7f1dac097310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 --2- 172.21.2.117:0/4216356858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1dac0a1260 0x7f1dac1a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:00.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.609 7f1db2269700 1 -- 172.21.2.117:0/4216356858 >> 172.21.2.117:0/4216356858 conn(0x7f1dac08ab30 msgr2=0x7f1dac0935b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:00.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.610 7f1db2269700 1 -- 172.21.2.117:0/4216356858 shutdown_connections 2021-02-10T11:45:00.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:00.610 7f1db2269700 1 -- 172.21.2.117:0/4216356858 wait complete. 2021-02-10T11:45:00.612 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000071660 2021-02-10T11:45:01.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.032 7fa5f9201700 1 Processor -- start 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.032 7fa5f9201700 1 -- start start 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5f41b07f0 0x7fa5f40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f40e6170 0x7fa5f40e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 0x7fa5f40e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40e5ce0 con 0x7fa5f41b07f0 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40e75b0 con 0x7fa5f40e6b90 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f9201700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40e77f0 con 0x7fa5f40e6170 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5f41b07f0 0x7fa5f40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5f41b07f0 0x7fa5f40e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51388/0 (socket says 172.21.2.117:51388) 2021-02-10T11:45:01.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 -- 172.21.2.117:0/67434773 learned_addr learned my addr 172.21.2.117:0/67434773 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f259c700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f40e6170 0x7fa5f40e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f359e700 1 --2- 172.21.2.117:0/67434773 >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 0x7fa5f40e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 -- 172.21.2.117:0/67434773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f40e6170 msgr2=0x7fa5f40e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f40e6170 0x7fa5f40e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 -- 172.21.2.117:0/67434773 >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 msgr2=0x7fa5f40e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 --2- 172.21.2.117:0/67434773 >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 0x7fa5f40e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f2d9d700 1 -- 172.21.2.117:0/67434773 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40e54b0 con 0x7fa5f41b07f0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.033 7fa5f359e700 1 --2- 172.21.2.117:0/67434773 >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 0x7fa5f40e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f2d9d700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f41b07f0 0x7fa5f40e4e90 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fa5e4006700 tx=0x7fa5e4009b80).ready entity=mon.0 client_cookie=e251cfc4c8fb5695 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f1d9b700 1 -- 172.21.2.117:0/67434773 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e4006340 con 0x7fa5f41b07f0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f1d9b700 1 -- 172.21.2.117:0/67434773 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e400c0c0 con 0x7fa5f41b07f0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f1d9b700 1 -- 172.21.2.117:0/67434773 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5e400c5e0 con 0x7fa5f41b07f0 2021-02-10T11:45:01.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f9201700 1 -- 172.21.2.117:0/67434773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f41b07f0 msgr2=0x7fa5f40e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.034 7fa5f9201700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f41b07f0 0x7fa5f40e4e90 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 rx=0x7fa5e4006700 tx=0x7fa5e4009b80).stop 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 -- 172.21.2.117:0/67434773 shutdown_connections 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 --2- 172.21.2.117:0/67434773 >> v2:172.21.2.125:3300/0 conn(0x7fa5f40e6b90 0x7fa5f40e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f40e6170 0x7fa5f40e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 --2- 172.21.2.117:0/67434773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f41b07f0 0x7fa5f40e4e90 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 -- 172.21.2.117:0/67434773 >> 172.21.2.117:0/67434773 conn(0x7fa5f40ce500 msgr2=0x7fa5f40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 -- 172.21.2.117:0/67434773 shutdown_connections 2021-02-10T11:45:01.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.035 7fa5f9201700 1 -- 172.21.2.117:0/67434773 wait complete. 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 Processor -- start 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 -- start start 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 0x7fa5f40ba290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 0x7fa5f4093f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f408f810 con 0x7fa5f40bad40 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40935b0 con 0x7fa5f408f410 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f9201700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40ba8b0 con 0x7fa5f4093b30 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51394/0 (socket says 172.21.2.117:51394) 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.050 7fa5f359e700 1 -- 172.21.2.117:0/3487961426 learned_addr learned my addr 172.21.2.117:0/3487961426 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 -- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 msgr2=0x7fa5f4093f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f259c700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 0x7fa5f4093f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f2d9d700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 0x7fa5f40ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 0x7fa5f4093f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 -- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 msgr2=0x7fa5f40ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 0x7fa5f40ba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 -- 172.21.2.117:0/3487961426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f41a6bc0 con 0x7fa5f40bad40 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f2d9d700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 0x7fa5f40ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f259c700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 0x7fa5f4093f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f359e700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fa5ec009980 tx=0x7fa5ec00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ec006020 con 0x7fa5f40bad40 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0062a0 con 0x7fa5f40bad40 2021-02-10T11:45:01.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f41a6bc0 con 0x7fa5f40bad40 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5ec006700 con 0x7fa5f40bad40 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.051 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f41a7150 con 0x7fa5f40bad40 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.052 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa5ec006480 con 0x7fa5f40bad40 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.052 7fa5e37fe700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 0x7fa5d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.052 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa5ec032a50 con 0x7fa5f40bad40 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.053 7fa5f2d9d700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 0x7fa5d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.053 7fa5f2d9d700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 0x7fa5d801abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.083 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f410c290 con 0x7fa5f40bad40 2021-02-10T11:45:01.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.085 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa5ec000f90 con 0x7fa5f40bad40 2021-02-10T11:45:01.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.211 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000071660", "target": ["mgr", ""]}) v1 -- 0x7fa5f41a7150 con 0x7fa5d80186c0 2021-02-10T11:45:01.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.224 7fa5e37fe700 1 -- 172.21.2.117:0/3487961426 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa5e40009b0 con 0x7fa5d80186c0 2021-02-10T11:45:01.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 msgr2=0x7fa5d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 0x7fa5d801abc0 crc :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 msgr2=0x7fa5f40bb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 rx=0x7fa5ec009980 tx=0x7fa5ec00bcd0).stop 2021-02-10T11:45:01.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 shutdown_connections 2021-02-10T11:45:01.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5d80186c0 0x7fa5d801abc0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5f40bad40 0x7fa5f40bb140 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5f4093b30 0x7fa5f4093f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 --2- 172.21.2.117:0/3487961426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5f408f410 0x7fa5f40ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 >> 172.21.2.117:0/3487961426 conn(0x7fa5f408fb00 msgr2=0x7fa5f408fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 shutdown_connections 2021-02-10T11:45:01.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.229 7fa5f9201700 1 -- 172.21.2.117:0/3487961426 wait complete. 2021-02-10T11:45:01.231 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs clone_0000000000862156 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.653 7f38e8c20700 1 Processor -- start 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.653 7f38e8c20700 1 -- start start 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e8c20700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38e40d18f0 0x7f38e40eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e8c20700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e40ef3f0 0x7f38e41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e8c20700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e40ef7f0 con 0x7f38e40eb040 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e8c20700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e40eded0 con 0x7f38e40d18f0 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e8c20700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40d1cf0 con 0x7f38e40ef3f0 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e285e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38e40d18f0 0x7f38e40eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e205d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f38e40eb040 0x7f38e40eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51654/0 (socket says 172.21.2.117:51654) 2021-02-10T11:45:01.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e205d700 1 -- 172.21.2.117:0/4265541405 learned_addr learned my addr 172.21.2.117:0/4265541405 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.654 7f38e305f700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e40ef3f0 0x7f38e41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.655 7f38e285e700 1 -- 172.21.2.117:0/4265541405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e40ef3f0 msgr2=0x7f38e41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.655 7f38e285e700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e40ef3f0 0x7f38e41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.655 7f38e285e700 1 -- 172.21.2.117:0/4265541405 >> v1:172.21.2.117:6789/0 conn(0x7f38e40eb040 legacy=0x7f38e40eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.655 7f38e285e700 1 -- 172.21.2.117:0/4265541405 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e41b8fd0 con 0x7f38e40d18f0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.655 7f38e285e700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e40d18f0 0x7f38e40eaa20 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f38dc0069d0 tx=0x7f38dc009e40).ready entity=mon.0 client_cookie=eb7bf86101f332b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.656 7f38e185c700 1 -- 172.21.2.117:0/4265541405 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc00b510 con 0x7f38e40d18f0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.656 7f38e185c700 1 -- 172.21.2.117:0/4265541405 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc00b890 con 0x7f38e40d18f0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.656 7f38e185c700 1 -- 172.21.2.117:0/4265541405 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc00bcf0 con 0x7f38e40d18f0 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.656 7f38e8c20700 1 -- 172.21.2.117:0/4265541405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e40d18f0 msgr2=0x7f38e40eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.656 7f38e8c20700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e40d18f0 0x7f38e40eaa20 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 rx=0x7f38dc0069d0 tx=0x7f38dc009e40).stop 2021-02-10T11:45:01.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 -- 172.21.2.117:0/4265541405 shutdown_connections 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e40ef3f0 0x7f38e41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 --2- 172.21.2.117:0/4265541405 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e40d18f0 0x7f38e40eaa20 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 -- 172.21.2.117:0/4265541405 >> 172.21.2.117:0/4265541405 conn(0x7f38e40cf060 msgr2=0x7f38e40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 -- 172.21.2.117:0/4265541405 shutdown_connections 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.657 7f38e8c20700 1 -- 172.21.2.117:0/4265541405 wait complete. 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.671 7f38e8c20700 1 Processor -- start 2021-02-10T11:45:01.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.671 7f38e8c20700 1 -- start start 2021-02-10T11:45:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e409ed70 0x7f38e409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e409f790 0x7f38e40b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e409fb90 con 0x7f38e41a4240 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40dc640 con 0x7f38e409ed70 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e8c20700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e4097270 con 0x7f38e409f790 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51406/0 (socket says 172.21.2.117:51406) 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e285e700 1 -- 172.21.2.117:0/702656408 learned_addr learned my addr 172.21.2.117:0/702656408 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e305f700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e409f790 0x7f38e40b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.672 7f38e205d700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e409ed70 0x7f38e409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 -- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e409f790 msgr2=0x7f38e40b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e409f790 0x7f38e40b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 -- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e409ed70 msgr2=0x7f38e409f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e409ed70 0x7f38e409f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 -- 172.21.2.117:0/702656408 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e40b8780 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e285e700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 rx=0x7f38dc009f40 tx=0x7f38dc00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc006240 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0064c0 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38dc0058e0 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e8c20700 1 -- 172.21.2.117:0/702656408 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e40b8780 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38e8c20700 1 -- 172.21.2.117:0/702656408 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e40b8d10 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.673 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38dc005660 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.674 7f38d37fe700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 0x7f38c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.674 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f38dc0329f0 con 0x7f38e41a4240 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.674 7f38e205d700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 0x7f38c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:01.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.674 7f38e205d700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 0x7f38c801abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.704 7f38e8c20700 1 -- 172.21.2.117:0/702656408 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e40daff0 con 0x7f38e41a4240 2021-02-10T11:45:01.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.707 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38dc000d30 con 0x7f38e41a4240 2021-02-10T11:45:01.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.832 7f38e8c20700 1 -- 172.21.2.117:0/702656408 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "clone_0000000000862156", "target": ["mgr", ""]}) v1 -- 0x7f38e40b8d10 con 0x7f38c80186c0 2021-02-10T11:45:01.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.841 7f38d37fe700 1 -- 172.21.2.117:0/702656408 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f38d80014a0 con 0x7f38c80186c0 2021-02-10T11:45:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 msgr2=0x7f38c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 0x7f38c801abc0 crc :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 msgr2=0x7f38e41a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 rx=0x7f38dc009f40 tx=0x7f38dc00bd50).stop 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 shutdown_connections 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38e409f790 0x7f38e40b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38e409ed70 0x7f38e409f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38c80186c0 0x7f38c801abc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 --2- 172.21.2.117:0/702656408 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38e41a4240 0x7f38e41a4640 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 >> 172.21.2.117:0/702656408 conn(0x7f38e4087090 msgr2=0x7f38e408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 shutdown_connections 2021-02-10T11:45:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:01.850 7f38e8c20700 1 -- 172.21.2.117:0/702656408 wait complete. 2021-02-10T11:45:01.851 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:45:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:45:02.203 INFO:teuthology.orchestra.run.gibba017.stdout:0 2021-02-10T11:45:02.204 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:45:02.204 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:45:02.204 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:02.549 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:02.893 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:45:02.893 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:02.893 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:02.893 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:03.228 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:03.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:03.229 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:45:03.229 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:03.229 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:03.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[141399]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:03.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:03.585 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:03.585 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:03.920 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:45:03.921 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:03.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:03.921 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:04.258 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:04.258 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:04.258 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:04.593 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:04.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:04.594 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:45:04.594 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:04.594 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:04.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[142039]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:04.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:04.943 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:04.943 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:05.278 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:45:05.278 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:05.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:05.279 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:05.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:05.615 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:05.615 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:05.949 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:45:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:05.949 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:45:05.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:05.950 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:06.173 INFO:tasks.ceph.osd.6.gibba025.stderr:2021-02-10 11:45:06.186 7f21f2a9f700 -1 osd.6 pg_epoch: 74 pg[15.5( v 73'8 (0'0,73'8] local-lis/les=71/72 n=2 ec=71/71 lis/c 71/71 les/c/f 72/72/0 71/71/71) [6,3] r=0 lpr=71 crt=73'8 lcod 73'7 mlcod 73'7 active+clean+snaptrim trimq=[2~2]] removing snap head 2021-02-10T11:45:06.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:06.284 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:06.284 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:06.617 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:45:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:06.617 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:45:06.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:06.618 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:06.952 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:45:07.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 Processor -- start 2021-02-10T11:45:07.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 -- start start 2021-02-10T11:45:07.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0df01b07f0 0x7f0df00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df00e6170 0x7f0df00e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 0x7f0df00e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df00e5ce0 con 0x7f0df01b07f0 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df00e75b0 con 0x7f0df00e6b90 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.376 7f0df70b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00e77f0 con 0x7f0df00e6170 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0df01b07f0 0x7f0df00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0df01b07f0 0x7f0df00e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51416/0 (socket says 172.21.2.117:51416) 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 -- 172.21.2.117:0/949041567 learned_addr learned my addr 172.21.2.117:0/949041567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 -- 172.21.2.117:0/949041567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df00e6170 msgr2=0x7f0df00e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df56d5700 1 --2- 172.21.2.117:0/949041567 >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 0x7f0df00e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 --2- 172.21.2.117:0/949041567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df00e6170 0x7f0df00e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 -- 172.21.2.117:0/949041567 >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 msgr2=0x7f0df00e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:07.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 --2- 172.21.2.117:0/949041567 >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 0x7f0df00e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 -- 172.21.2.117:0/949041567 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00e54b0 con 0x7f0df01b07f0 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df56d5700 1 --2- 172.21.2.117:0/949041567 >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 0x7f0df00e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.377 7f0df4ed4700 1 --2- 172.21.2.117:0/949041567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df01b07f0 0x7f0df00e4e90 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f0dd8006700 tx=0x7f0dd8009b80).ready entity=mon.0 client_cookie=e8bf81424683e3f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0def7fe700 1 -- 172.21.2.117:0/949041567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dd800c0c0 con 0x7f0df01b07f0 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0def7fe700 1 -- 172.21.2.117:0/949041567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd800c440 con 0x7f0df01b07f0 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0def7fe700 1 -- 172.21.2.117:0/949041567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dd800c8a0 con 0x7f0df01b07f0 2021-02-10T11:45:07.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 -- 172.21.2.117:0/949041567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df01b07f0 msgr2=0x7f0df00e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 --2- 172.21.2.117:0/949041567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df01b07f0 0x7f0df00e4e90 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 rx=0x7f0dd8006700 tx=0x7f0dd8009b80).stop 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 -- 172.21.2.117:0/949041567 shutdown_connections 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 --2- 172.21.2.117:0/949041567 >> v2:172.21.2.125:3300/0 conn(0x7f0df00e6b90 0x7f0df00e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 --2- 172.21.2.117:0/949041567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df00e6170 0x7f0df00e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 --2- 172.21.2.117:0/949041567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df01b07f0 0x7f0df00e4e90 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 -- 172.21.2.117:0/949041567 >> 172.21.2.117:0/949041567 conn(0x7f0df00ce500 msgr2=0x7f0df00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 -- 172.21.2.117:0/949041567 shutdown_connections 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.378 7f0df70b8700 1 -- 172.21.2.117:0/949041567 wait complete. 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 Processor -- start 2021-02-10T11:45:07.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 -- start start 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0df0109100 0x7f0df00aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df01079a0 0x7f0df01a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.393 7f0df70b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df019fb10 con 0x7f0df00aa790 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0df70b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df009c600 con 0x7f0df0109100 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0df70b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df0109500 con 0x7f0df01079a0 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51422/0 (socket says 172.21.2.117:51422) 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 -- 172.21.2.117:0/2966028781 learned_addr learned my addr 172.21.2.117:0/2966028781 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:07.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 -- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df01079a0 msgr2=0x7f0df01a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df01079a0 0x7f0df01a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 -- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0df0109100 msgr2=0x7f0df00aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0df0109100 0x7f0df00aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 -- 172.21.2.117:0/2966028781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df01a0630 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0deffff700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f0de0006f00 tx=0x7f0de000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de0006350 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de000bdc0 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0de0002020 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df01a0630 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.394 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df01a0bc0 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.395 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0de0007020 con 0x7f0df00aa790 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.395 7f0ded7fa700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 0x7f0dd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:07.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.395 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f0de0032590 con 0x7f0df00aa790 2021-02-10T11:45:07.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.396 7f0df4ed4700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 0x7f0dd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:07.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.396 7f0df4ed4700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 0x7f0dd401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:07.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.426 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df007e840 con 0x7f0df00aa790 2021-02-10T11:45:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.429 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0de0000960 con 0x7f0df00aa790 2021-02-10T11:45:07.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.579 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0df00e77f0 con 0x7f0df00aa790 2021-02-10T11:45:07.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.582 7f0ded7fa700 1 -- 172.21.2.117:0/2966028781 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0de0007020 con 0x7f0df00aa790 2021-02-10T11:45:07.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 msgr2=0x7f0dd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 0x7f0dd401abc0 crc :-1 s=READY pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 msgr2=0x7f0df0107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f0de0006f00 tx=0x7f0de000b940).stop 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 shutdown_connections 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0dd40186c0 0x7f0dd401abc0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0df00aa790 0x7f0df0107380 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0df01079a0 0x7f0df01a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 --2- 172.21.2.117:0/2966028781 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0df0109100 0x7f0df00aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 >> 172.21.2.117:0/2966028781 conn(0x7f0df00914c0 msgr2=0x7f0df0110fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 shutdown_connections 2021-02-10T11:45:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:07.588 7f0df70b8700 1 -- 172.21.2.117:0/2966028781 wait complete. 2021-02-10T11:45:07.589 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info' 2021-02-10T11:45:08.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.008 7f84dc982700 1 Processor -- start 2021-02-10T11:45:08.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.008 7f84dc982700 1 -- start start 2021-02-10T11:45:08.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84dc982700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84d80d18e0 0x7f84d80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84dc982700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80ef3f0 0x7f84d81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84dc982700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d80ede60 con 0x7f84d80eb000 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84dc982700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d80ef7f0 con 0x7f84d80d18e0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84dc982700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80d1ce0 con 0x7f84d80ef3f0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84d661c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84d80d18e0 0x7f84d80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84d661c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84d80d18e0 0x7f84d80ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51428/0 (socket says 172.21.2.117:51428) 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.009 7f84d661c700 1 -- 172.21.2.117:0/1859743495 learned_addr learned my addr 172.21.2.117:0/1859743495 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d6e1d700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80ef3f0 0x7f84d81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d661c700 1 -- 172.21.2.117:0/1859743495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80ef3f0 msgr2=0x7f84d81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d661c700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80ef3f0 0x7f84d81b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d661c700 1 -- 172.21.2.117:0/1859743495 >> v1:172.21.2.125:6789/0 conn(0x7f84d80eb000 legacy=0x7f84d80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d661c700 1 -- 172.21.2.117:0/1859743495 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d81b8f90 con 0x7f84d80d18e0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d661c700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d80d18e0 0x7f84d80ea9e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f84c8006700 tx=0x7f84c8009b80).ready entity=mon.0 client_cookie=b42ac00374a01182 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:08.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d561a700 1 -- 172.21.2.117:0/1859743495 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84c800c0c0 con 0x7f84d80d18e0 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d561a700 1 -- 172.21.2.117:0/1859743495 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c800c440 con 0x7f84d80d18e0 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84d561a700 1 -- 172.21.2.117:0/1859743495 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84c800c960 con 0x7f84d80d18e0 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84dc982700 1 -- 172.21.2.117:0/1859743495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d80d18e0 msgr2=0x7f84d80ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.010 7f84dc982700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d80d18e0 0x7f84d80ea9e0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 rx=0x7f84c8006700 tx=0x7f84c8009b80).stop 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 -- 172.21.2.117:0/1859743495 shutdown_connections 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80ef3f0 0x7f84d81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 --2- 172.21.2.117:0/1859743495 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d80d18e0 0x7f84d80ea9e0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 -- 172.21.2.117:0/1859743495 >> 172.21.2.117:0/1859743495 conn(0x7f84d80ced60 msgr2=0x7f84d80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 -- 172.21.2.117:0/1859743495 shutdown_connections 2021-02-10T11:45:08.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.011 7f84dc982700 1 -- 172.21.2.117:0/1859743495 wait complete. 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.026 7f84dc982700 1 Processor -- start 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.026 7f84dc982700 1 -- start start 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84d808bf90 0x7f84d81a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80a7e40 0x7f84d80a8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d819ce20 con 0x7f84d81a8c60 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81a86e0 con 0x7f84d808bf90 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84dc982700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8197870 con 0x7f84d80a7e40 2021-02-10T11:45:08.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51430/0 (socket says 172.21.2.117:51430) 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 -- 172.21.2.117:0/394146883 learned_addr learned my addr 172.21.2.117:0/394146883 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 -- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80a7e40 msgr2=0x7f84d80a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80a7e40 0x7f84d80a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 -- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84d808bf90 msgr2=0x7f84d81a7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84d808bf90 0x7f84d81a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 -- 172.21.2.117:0/394146883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d819bd40 con 0x7f84d81a8c60 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.027 7f84d661c700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f84c800b5e0 tx=0x7f84c800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84c800bd70 con 0x7f84d81a8c60 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c8002030 con 0x7f84d81a8c60 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84c8002550 con 0x7f84d81a8c60 2021-02-10T11:45:08.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84dc982700 1 -- 172.21.2.117:0/394146883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d819bd40 con 0x7f84d81a8c60 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84dc982700 1 -- 172.21.2.117:0/394146883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d819bd40 con 0x7f84d81a8c60 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84c8000c90 con 0x7f84d81a8c60 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.028 7f84c6ffd700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 0x7f84bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.029 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f84c8032760 con 0x7f84d81a8c60 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.029 7f84d5e1b700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 0x7f84bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:08.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.029 7f84d5e1b700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 0x7f84bc01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:08.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.059 7f84dc982700 1 -- 172.21.2.117:0/394146883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d8087210 con 0x7f84d81a8c60 2021-02-10T11:45:08.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.061 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84c8030d70 con 0x7f84d81a8c60 2021-02-10T11:45:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:08.194 7f84dc982700 1 -- 172.21.2.117:0/394146883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]} v 0) v1 -- 0x7f84d80ede60 con 0x7f84d81a8c60 2021-02-10T11:45:09.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.190 7f84c6ffd700 1 -- 172.21.2.117:0/394146883 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_clone_subvolume_info"]}]=0 v419) v1 ==== 142+0+0 (secure 0 0 0) 0x7f84c8000c90 con 0x7f84d81a8c60 2021-02-10T11:45:09.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 msgr2=0x7f84bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 0x7f84bc01abc0 crc :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 msgr2=0x7f84d8095ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 rx=0x7f84c800b5e0 tx=0x7f84c800b980).stop 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 shutdown_connections 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84d80a7e40 0x7f84d80a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84d808bf90 0x7f84d81a7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84bc0186c0 0x7f84bc01abc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 --2- 172.21.2.117:0/394146883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84d81a8c60 0x7f84d8095ee0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 >> 172.21.2.117:0/394146883 conn(0x7f84d819f610 msgr2=0x7f84d80e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:09.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 shutdown_connections 2021-02-10T11:45:09.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.217 7f84dc982700 1 -- 172.21.2.117:0/394146883 wait complete. 2021-02-10T11:45:09.218 INFO:tasks.cephfs_test_runner:test_clone_subvolume_info (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:45:09.218 INFO:tasks.cephfs_test_runner:Starting test: test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:45:09.218 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:45:09.236 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:09.263 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:09.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.637 7f974e386700 1 Processor -- start 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.637 7f974e386700 1 -- start start 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f974e386700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97480e4e40 0x7f97481b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f974e386700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 0x7f9748084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f974e386700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9748085930 con 0x7f97481b0810 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f974e386700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9748084840 con 0x7f97480e4e40 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f974e386700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9748084a30 con 0x7f97480e5240 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f9747fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97480e4e40 0x7f97481b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f9747fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97480e4e40 0x7f97481b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51438/0 (socket says 172.21.2.117:51438) 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f9747fff700 1 -- 172.21.2.117:0/1072023366 learned_addr learned my addr 172.21.2.117:0/1072023366 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.638 7f97477fe700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 0x7f9748084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:09.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9747fff700 1 -- 172.21.2.117:0/1072023366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 msgr2=0x7f9748084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9747fff700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 0x7f9748084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9747fff700 1 -- 172.21.2.117:0/1072023366 >> v1:172.21.2.125:6789/0 conn(0x7f97481b0810 legacy=0x7f9748085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9747fff700 1 -- 172.21.2.117:0/1072023366 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748085280 con 0x7f97480e4e40 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f97477fe700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 0x7f9748084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9747fff700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97480e4e40 0x7f97481b01f0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f9738006700 tx=0x7f9738009b80).ready entity=mon.0 client_cookie=26be35ea0892feae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9746ffd700 1 -- 172.21.2.117:0/1072023366 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f973800c0c0 con 0x7f97480e4e40 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9746ffd700 1 -- 172.21.2.117:0/1072023366 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800c400 con 0x7f97480e4e40 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f9746ffd700 1 -- 172.21.2.117:0/1072023366 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f973800c920 con 0x7f97480e4e40 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f974e386700 1 -- 172.21.2.117:0/1072023366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97480e4e40 msgr2=0x7f97481b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.639 7f974e386700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97480e4e40 0x7f97481b01f0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 rx=0x7f9738006700 tx=0x7f9738009b80).stop 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 -- 172.21.2.117:0/1072023366 shutdown_connections 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97480e5240 0x7f9748084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 --2- 172.21.2.117:0/1072023366 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97480e4e40 0x7f97481b01f0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:09.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 -- 172.21.2.117:0/1072023366 >> 172.21.2.117:0/1072023366 conn(0x7f97480ce4e0 msgr2=0x7f97480e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:09.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 -- 172.21.2.117:0/1072023366 shutdown_connections 2021-02-10T11:45:09.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.640 7f974e386700 1 -- 172.21.2.117:0/1072023366 wait complete. 2021-02-10T11:45:09.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.654 7f974e386700 1 Processor -- start 2021-02-10T11:45:09.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 -- start start 2021-02-10T11:45:09.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f974819e4e0 0x7f97481b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f974818d0b0 0x7f974818d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97481b8cc0 con 0x7f97481b4c30 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974819e8e0 con 0x7f974819e4e0 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974e386700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97481b5650 con 0x7f974818d0b0 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51442/0 (socket says 172.21.2.117:51442) 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 -- 172.21.2.117:0/2767160504 learned_addr learned my addr 172.21.2.117:0/2767160504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 -- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f974818d0b0 msgr2=0x7f974818d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f974818d0b0 0x7f974818d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 -- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f974819e4e0 msgr2=0x7f97481b86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f974819e4e0 0x7f97481b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:09.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.655 7f974c9a3700 1 -- 172.21.2.117:0/2767160504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97481b5870 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f974c9a3700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f974000b8e0 tx=0x7f974000bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9740000ce0 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740000f90 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f974000c570 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f974e386700 1 -- 172.21.2.117:0/2767160504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97481b5870 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f974e386700 1 -- 172.21.2.117:0/2767160504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97481b5870 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9740001220 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.656 7f9744ff9700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 0x7f972801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.657 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f9740032ad0 con 0x7f97481b4c30 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.657 7f9747fff700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 0x7f972801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:09.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.657 7f9747fff700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 0x7f972801abc0 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:09.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.688 7f974e386700 1 -- 172.21.2.117:0/2767160504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97481b8170 con 0x7f97481b4c30 2021-02-10T11:45:09.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.690 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f974000bdc0 con 0x7f97481b4c30 2021-02-10T11:45:09.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:09.824 7f974e386700 1 -- 172.21.2.117:0/2767160504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7f97481b8170 con 0x7f97481b4c30 2021-02-10T11:45:10.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.194 7f9744ff9700 1 -- 172.21.2.117:0/2767160504 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v420) v1 ==== 146+0+0 (secure 0 0 0) 0x7f9740001220 con 0x7f97481b4c30 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 msgr2=0x7f972801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 0x7f972801abc0 crc :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 msgr2=0x7f97481b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 rx=0x7f974000b8e0 tx=0x7f974000bc80).stop 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 shutdown_connections 2021-02-10T11:45:10.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f97280186c0 0x7f972801abc0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97481b4c30 0x7f97481b5030 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f974818d0b0 0x7f974818d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 --2- 172.21.2.117:0/2767160504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f974819e4e0 0x7f97481b86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 >> 172.21.2.117:0/2767160504 conn(0x7f97481a60a0 msgr2=0x7f974810c490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 shutdown_connections 2021-02-10T11:45:10.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.243 7f974e386700 1 -- 172.21.2.117:0/2767160504 wait complete. 2021-02-10T11:45:10.244 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 Processor -- start 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 -- start start 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12340ef3f0 0x7f12340eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340ef7f0 con 0x7f12340eb830 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340d1cf0 con 0x7f12340d18f0 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f123a74e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340eded0 con 0x7f12340ef3f0 2021-02-10T11:45:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f1238d6b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f12340eb830 0x7f12341b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51702/0 (socket says 172.21.2.117:51702) 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.665 7f1238d6b700 1 -- 172.21.2.117:0/947762593 learned_addr learned my addr 172.21.2.117:0/947762593 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903983555 0 0) 0x7f122c001500 con 0x7f12340eb830 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f121c002fe0 con 0x7f12340eb830 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3713418347 0 0) 0x7f122c001460 con 0x7f12340eb830 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12340ef3f0 msgr2=0x7f12340eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12340ef3f0 0x7f12340eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 >> v1:172.21.2.125:6789/0 conn(0x7f12340d18f0 legacy=0x7f12340eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12341bb1c0 con 0x7f12340eb830 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f122c001980 con 0x7f12340eb830 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 >> v1:172.21.2.117:6789/0 conn(0x7f12340eb830 legacy=0x7f12341b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f121c003b90 0x7f121c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f121c0067a0 0x7f121c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f121c009520 0x7f121c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c00c2d0 con 0x7f121c0067a0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c00c4f0 con 0x7f121c009520 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1232ffd700 1 -- 172.21.2.117:0/947762593 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121c00c710 con 0x7f121c003b90 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f12337fe700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f121c0067a0 0x7f121c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1238d6b700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f121c009520 0x7f121c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.666 7f1233fff700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f121c003b90 0x7f121c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 -- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f121c003b90 msgr2=0x7f121c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f121c003b90 0x7f121c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 -- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f121c009520 msgr2=0x7f121c00ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f121c009520 0x7f121c00ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 -- 172.21.2.117:0/947762593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c00c9b0 con 0x7f121c0067a0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f12337fe700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f121c0067a0 0x7f121c008c80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f1228006ab0 tx=0x7f122804fb40).ready entity=mon.0 client_cookie=1f667e5e864b8653 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1228002030 con 0x7f121c0067a0 2021-02-10T11:45:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12280063f0 con 0x7f121c0067a0 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f1232ffd700 1 -- 172.21.2.117:0/947762593 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1228002330 con 0x7f121c0067a0 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f123a74e700 1 -- 172.21.2.117:0/947762593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f121c0067a0 msgr2=0x7f121c008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f123a74e700 1 --2- 172.21.2.117:0/947762593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f121c0067a0 0x7f121c008c80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 rx=0x7f1228006ab0 tx=0x7f122804fb40).stop 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.667 7f1238d6b700 1 -- 172.21.2.117:0/947762593 reap_dead start 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.668 7f123a74e700 1 -- 172.21.2.117:0/947762593 shutdown_connections 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.668 7f123a74e700 1 -- 172.21.2.117:0/947762593 >> 172.21.2.117:0/947762593 conn(0x7f12340cf060 msgr2=0x7f12340cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.668 7f123a74e700 1 -- 172.21.2.117:0/947762593 shutdown_connections 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.668 7f123a74e700 1 -- 172.21.2.117:0/947762593 wait complete. 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.682 7f123a74e700 1 Processor -- start 2021-02-10T11:45:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.682 7f123a74e700 1 -- start start 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f123408a220 0x7f123407c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 0x7f123409dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123409e2a0 con 0x7f123408ace0 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12341a7e50 con 0x7f123408a220 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f123a74e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12341a8070 con 0x7f123408a8e0 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51466/0 (socket says 172.21.2.117:51466) 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 -- 172.21.2.117:0/1915795769 learned_addr learned my addr 172.21.2.117:0/1915795769 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f12337fe700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 0x7f123409dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1233fff700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f123408a220 0x7f123407c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 -- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 msgr2=0x7f123409dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 0x7f123409dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 -- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f123408a220 msgr2=0x7f123407c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f123408a220 0x7f123407c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f1238d6b700 1 -- 172.21.2.117:0/1915795769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123407d3d0 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.683 7f12337fe700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 0x7f123409dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f1238d6b700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f122c00a8f0 tx=0x7f122c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f122c004930 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c004bf0 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f123a74e700 1 -- 172.21.2.117:0/1915795769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123407d3d0 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f123a74e700 1 -- 172.21.2.117:0/1915795769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123407d960 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.684 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f122c007210 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.685 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f122c004720 con 0x7f123408ace0 2021-02-10T11:45:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.685 7f1230ff9700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 0x7f121401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.685 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f122c032ce0 con 0x7f123408ace0 2021-02-10T11:45:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.685 7f1233fff700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 0x7f121401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.685 7f1233fff700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 0x7f121401abd0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:10.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.715 7f123a74e700 1 -- 172.21.2.117:0/1915795769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123407cea0 con 0x7f123408ace0 2021-02-10T11:45:10.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.718 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f122c001500 con 0x7f123408ace0 2021-02-10T11:45:10.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.866 7f123a74e700 1 -- 172.21.2.117:0/1915795769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f123407cea0 con 0x7f123408ace0 2021-02-10T11:45:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.870 7f1230ff9700 1 -- 172.21.2.117:0/1915795769 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f122c004720 con 0x7f123408ace0 2021-02-10T11:45:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 msgr2=0x7f121401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 0x7f121401abd0 crc :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 msgr2=0x7f12341a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 rx=0x7f122c00a8f0 tx=0x7f122c00ac90).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 shutdown_connections 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12140186d0 0x7f121401abd0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f123408ace0 0x7f12341a7830 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f123408a8e0 0x7f123409dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 --2- 172.21.2.117:0/1915795769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f123408a220 0x7f123407c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 >> 172.21.2.117:0/1915795769 conn(0x7f12340a71e0 msgr2=0x7f12340a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 shutdown_connections 2021-02-10T11:45:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:10.875 7f123a74e700 1 -- 172.21.2.117:0/1915795769 wait complete. 2021-02-10T11:45:10.876 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:45:10.876 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:11.221 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:45:11.565 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:45:11.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:11.566 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:11.566 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:11.901 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:45:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:11.902 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:11.902 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:11.902 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:11.902 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:12.237 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:45:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:12.237 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:12.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:12.238 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:12.238 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:45:12.572 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:45:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:12.573 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:45:12.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:12.573 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:12.573 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:45:12.907 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:45:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:45:12.907 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:45:12.907 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.325 7f2127e27700 1 Processor -- start 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.325 7f2127e27700 1 -- start start 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2127e27700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f21200ef3f0 0x7f21200eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2127e27700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21200eb7f0 0x7f21201b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2127e27700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21200ede60 con 0x7f21200d18e0 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2127e27700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21200ef7f0 con 0x7f21200eb7f0 2021-02-10T11:45:13.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2127e27700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21200d1ce0 con 0x7f21200ef3f0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2125c43700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f21200d18e0 0x7f21200ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51720/0 (socket says 172.21.2.117:51720) 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2125c43700 1 -- 172.21.2.117:0/2802592675 learned_addr learned my addr 172.21.2.117:0/2802592675 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.326 7f2126444700 1 --2- 172.21.2.117:0/2802592675 >> v2:172.21.2.117:3300/0 conn(0x7f21200eb7f0 0x7f21201b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2126444700 1 -- 172.21.2.117:0/2802592675 >> v2:172.21.2.125:3300/0 conn(0x7f21200ef3f0 msgr2=0x7f21200eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2126444700 1 --2- 172.21.2.117:0/2802592675 >> v2:172.21.2.125:3300/0 conn(0x7f21200ef3f0 0x7f21200eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2126444700 1 -- 172.21.2.117:0/2802592675 >> v1:172.21.2.117:6789/0 conn(0x7f21200d18e0 legacy=0x7f21200ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2124c41700 1 -- 172.21.2.117:0/2802592675 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530978090 0 0) 0x7f211c001180 con 0x7f21200d18e0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2126444700 1 -- 172.21.2.117:0/2802592675 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21201b8f90 con 0x7f21200eb7f0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2126444700 1 --2- 172.21.2.117:0/2802592675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200eb7f0 0x7f21201b7960 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7f2118006e30 tx=0x7f2118009de0).ready entity=mon.0 client_cookie=61ba505696f928fe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2124c41700 1 -- 172.21.2.117:0/2802592675 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211800b300 con 0x7f21200eb7f0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2124c41700 1 -- 172.21.2.117:0/2802592675 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211800b680 con 0x7f21200eb7f0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2124c41700 1 -- 172.21.2.117:0/2802592675 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211800bae0 con 0x7f21200eb7f0 2021-02-10T11:45:13.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2127e27700 1 -- 172.21.2.117:0/2802592675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200eb7f0 msgr2=0x7f21201b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.327 7f2127e27700 1 --2- 172.21.2.117:0/2802592675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200eb7f0 0x7f21201b7960 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 rx=0x7f2118006e30 tx=0x7f2118009de0).stop 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 -- 172.21.2.117:0/2802592675 shutdown_connections 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 --2- 172.21.2.117:0/2802592675 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200eb7f0 0x7f21201b7960 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 --2- 172.21.2.117:0/2802592675 >> v2:172.21.2.125:3300/0 conn(0x7f21200ef3f0 0x7f21200eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 -- 172.21.2.117:0/2802592675 >> 172.21.2.117:0/2802592675 conn(0x7f21200ced60 msgr2=0x7f21200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 -- 172.21.2.117:0/2802592675 shutdown_connections 2021-02-10T11:45:13.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.328 7f2127e27700 1 -- 172.21.2.117:0/2802592675 wait complete. 2021-02-10T11:45:13.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.342 7f2127e27700 1 Processor -- start 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 -- start start 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f212008a8a0 0x7f2120095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21200960c0 0x7f21200e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21200964c0 con 0x7f21200ac7b0 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21200e7bd0 con 0x7f21200960c0 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2127e27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21200e7df0 con 0x7f212008a8a0 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51474/0 (socket says 172.21.2.117:51474) 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 -- 172.21.2.117:0/1853614470 learned_addr learned my addr 172.21.2.117:0/1853614470 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 -- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f212008a8a0 msgr2=0x7f2120095aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f212008a8a0 0x7f2120095aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 -- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21200960c0 msgr2=0x7f21200e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21200960c0 0x7f21200e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.343 7f2125c43700 1 -- 172.21.2.117:0/1853614470 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21200e8010 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f2125c43700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7f211c00a570 tx=0x7f211c00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211c00ad00 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211c001180 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f211c004620 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f2127e27700 1 -- 172.21.2.117:0/1853614470 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21200e8010 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f2127e27700 1 -- 172.21.2.117:0/1853614470 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21200e8010 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.344 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f211c0043a0 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.345 7f21167fc700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 0x7f210801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.345 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5964+0+0 (secure 0 0 0) 0x7f211c032900 con 0x7f21200ac7b0 2021-02-10T11:45:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.345 7f2125442700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 0x7f210801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:13.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.345 7f2125442700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 0x7f210801ac10 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:13.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.376 7f2127e27700 1 -- 172.21.2.117:0/1853614470 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2120087210 con 0x7f21200ac7b0 2021-02-10T11:45:13.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.379 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f211c000c90 con 0x7f21200ac7b0 2021-02-10T11:45:13.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.503 7f2127e27700 1 -- 172.21.2.117:0/1853614470 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2120087210 con 0x7f21200ac7b0 2021-02-10T11:45:13.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.503 7f21167fc700 1 -- 172.21.2.117:0/1853614470 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v75) v1 ==== 81+0+30966 (secure 0 0 0) 0x7f211c001160 con 0x7f21200ac7b0 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 75, 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:45:12.461417", 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:45:13.497 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:45:13.498 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 15, 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:45:13.499 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:45:13.500 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:45:13.501 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:45:13.502 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:45:13.503 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.504 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 14, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:44:29.317633", 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:45:13.505 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:45:13.506 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "72", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:13.507 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:13.508 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:45:13.509 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.510 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 15, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:44:30.338240", 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 8193, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:45:13.511 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:45:13.512 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "74", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 3, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 74, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:45:13.513 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:45:13.514 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:45:13.515 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:45:13.516 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.517 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.518 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.519 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.520 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.521 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.522 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.523 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.524 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.525 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:45:13.526 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.527 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.528 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.529 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:45:13.530 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.531 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.532 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.533 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.534 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.535 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 69, 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.536 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.537 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.538 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.539 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.540 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:45:13.541 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:45:13.542 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.543 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.544 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.545 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.546 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.547 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 69, 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:45:13.548 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.549 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.550 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.551 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.552 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.553 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.554 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.555 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:45:13.556 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.557 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:45:13.558 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:45:13.559 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 62, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:45:13.560 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.561 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.562 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:45:13.563 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:13.564 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.565 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.566 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.567 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.568 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.569 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:45:13.570 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:45:13.571 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:45:13.572 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [ 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 15, 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "snaps": [ 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:13.573 INFO:teuthology.orchestra.run.gibba017.stdout: "begin": 2, 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: "length": 2 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:45:13.574 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:45:13.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 -- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 msgr2=0x7f210801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 0x7f210801ac10 crc :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 -- 172.21.2.117:0/1853614470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 msgr2=0x7f212008a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 rx=0x7f211c00a570 tx=0x7f211c00a910).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 -- 172.21.2.117:0/1853614470 shutdown_connections 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21200960c0 0x7f21200e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f212008a8a0 0x7f2120095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2108018730 0x7f210801ac10 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 --2- 172.21.2.117:0/1853614470 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21200ac7b0 0x7f212008a280 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 -- 172.21.2.117:0/1853614470 >> 172.21.2.117:0/1853614470 conn(0x7f212019f610 msgr2=0x7f21200e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.505 7f2127e27700 1 -- 172.21.2.117:0/1853614470 shutdown_connections 2021-02-10T11:45:13.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.506 7f2127e27700 1 -- 172.21.2.117:0/1853614470 wait complete. 2021-02-10T11:45:13.577 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.995 7fe5a4536700 1 Processor -- start 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.995 7fe5a4536700 1 -- start start 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c0e4e40 0x7fe59c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe59c0e5240 0x7fe59c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe59c1b0810 0x7fe59c085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c087430 con 0x7fe59c1b0810 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59c0e5cc0 con 0x7fe59c0e5240 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a4536700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c085930 con 0x7fe59c0e4e40 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe59c1b0810 0x7fe59c085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe59c1b0810 0x7fe59c085280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51482/0 (socket says 172.21.2.117:51482) 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 -- 172.21.2.117:0/3814060967 learned_addr learned my addr 172.21.2.117:0/3814060967 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a1b51700 1 --2- 172.21.2.117:0/3814060967 >> v2:172.21.2.125:3300/0 conn(0x7fe59c0e5240 0x7fe59c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2352700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c0e4e40 0x7fe59c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 -- 172.21.2.117:0/3814060967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c0e4e40 msgr2=0x7fe59c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c0e4e40 0x7fe59c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 -- 172.21.2.117:0/3814060967 >> v2:172.21.2.125:3300/0 conn(0x7fe59c0e5240 msgr2=0x7fe59c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 --2- 172.21.2.117:0/3814060967 >> v2:172.21.2.125:3300/0 conn(0x7fe59c0e5240 0x7fe59c084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 -- 172.21.2.117:0/3814060967 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c084840 con 0x7fe59c1b0810 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.996 7fe5a2b53700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1b0810 0x7fe59c085280 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7fe594006f00 tx=0x7fe594009e90).ready entity=mon.0 client_cookie=41fea250e0ab670f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a1350700 1 -- 172.21.2.117:0/3814060967 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe59400b300 con 0x7fe59c1b0810 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a1350700 1 -- 172.21.2.117:0/3814060967 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59400b580 con 0x7fe59c1b0810 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a1350700 1 -- 172.21.2.117:0/3814060967 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe59400b9d0 con 0x7fe59c1b0810 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 -- 172.21.2.117:0/3814060967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1b0810 msgr2=0x7fe59c085280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1b0810 0x7fe59c085280 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 rx=0x7fe594006f00 tx=0x7fe594009e90).stop 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 -- 172.21.2.117:0/3814060967 shutdown_connections 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 --2- 172.21.2.117:0/3814060967 >> v2:172.21.2.125:3300/0 conn(0x7fe59c0e5240 0x7fe59c084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1b0810 0x7fe59c085280 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 --2- 172.21.2.117:0/3814060967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c0e4e40 0x7fe59c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.997 7fe5a4536700 1 -- 172.21.2.117:0/3814060967 >> 172.21.2.117:0/3814060967 conn(0x7fe59c0ce4e0 msgr2=0x7fe59c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.998 7fe5a4536700 1 -- 172.21.2.117:0/3814060967 shutdown_connections 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:13.998 7fe5a4536700 1 -- 172.21.2.117:0/3814060967 wait complete. 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.012 7fe5a4536700 1 Processor -- start 2021-02-10T11:45:14.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.012 7fe5a4536700 1 -- start start 2021-02-10T11:45:14.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c1105f0 0x7fe59c0aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe59c1109f0 0x7fe59c0aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0952a0 con 0x7fe59c1a4830 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0ab460 con 0x7fe59c1109f0 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a4536700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c0ab640 con 0x7fe59c1105f0 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51486/0 (socket says 172.21.2.117:51486) 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 -- 172.21.2.117:0/1137901011 learned_addr learned my addr 172.21.2.117:0/1137901011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 -- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c1105f0 msgr2=0x7fe59c0aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c1105f0 0x7fe59c0aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 -- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe59c1109f0 msgr2=0x7fe59c0aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe59c1109f0 0x7fe59c0aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 -- 172.21.2.117:0/1137901011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c0ab860 con 0x7fe59c1a4830 2021-02-10T11:45:14.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.013 7fe5a2352700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7fe5980067e0 tx=0x7fe59800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe598006900 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59800be10 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59c0ab860 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c095c20 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe598002030 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe598000c90 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 0x7fe58401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.014 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fe598032970 con 0x7fe59c1a4830 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.015 7fe5a1b51700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 0x7fe58401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.015 7fe5a1b51700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 0x7fe58401ac10 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.046 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c085930 con 0x7fe59c1a4830 2021-02-10T11:45:14.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.048 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe598006900 con 0x7fe59c1a4830 2021-02-10T11:45:14.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.176 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe59c085930 con 0x7fe59c1a4830 2021-02-10T11:45:14.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.177 7fe58effd700 1 -- 172.21.2.117:0/1137901011 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+2095 (secure 0 0 0) 0x7fe598000c90 con 0x7fe59c1a4830 2021-02-10T11:45:14.170 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:14.170 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":54,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:44:33.346284","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8910":{"gid":8910,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/319870470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":319870470},{"type":"v1","addr":"172.21.2.125:6835","nonce":319870470}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:45:14.170 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 54 2021-02-10T11:45:14.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 msgr2=0x7fe58401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 0x7fe58401ac10 crc :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 msgr2=0x7fe59c0abcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 rx=0x7fe5980067e0 tx=0x7fe59800b990).stop 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 shutdown_connections 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe59c1109f0 0x7fe59c0aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe59c1105f0 0x7fe59c0aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe584018730 0x7fe58401ac10 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 --2- 172.21.2.117:0/1137901011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe59c1a4830 0x7fe59c0abcf0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 >> 172.21.2.117:0/1137901011 conn(0x7fe59c0b5bd0 msgr2=0x7fe59c0c39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:14.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 shutdown_connections 2021-02-10T11:45:14.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.179 7fe5a4536700 1 -- 172.21.2.117:0/1137901011 wait complete. 2021-02-10T11:45:14.180 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 Processor -- start 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 -- start start 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b280d15e0 0x7f4b280e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b280e7580 con 0x7f4b281b07f0 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b280e77f0 con 0x7f4b280e7180 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.598 7f4b2e424700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b280e5ce0 con 0x7f4b280d15e0 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b27fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4b281b07f0 0x7f4b280e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51744/0 (socket says 172.21.2.117:51744) 2021-02-10T11:45:14.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b27fff700 1 -- 172.21.2.117:0/2269852166 learned_addr learned my addr 172.21.2.117:0/2269852166 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2ca41700 1 --2- 172.21.2.117:0/2269852166 >> v2:172.21.2.117:3300/0 conn(0x7f4b280d15e0 0x7f4b280e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2ca41700 1 -- 172.21.2.117:0/2269852166 >> v1:172.21.2.125:6789/0 conn(0x7f4b280e7180 legacy=0x7f4b280e7d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2ca41700 1 -- 172.21.2.117:0/2269852166 >> v1:172.21.2.117:6789/0 conn(0x7f4b281b07f0 legacy=0x7f4b280e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2ca41700 1 -- 172.21.2.117:0/2269852166 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280d19e0 con 0x7f4b280d15e0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2ca41700 1 --2- 172.21.2.117:0/2269852166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280d15e0 0x7f4b280e8560 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f4b20006e30 tx=0x7f4b20009e60).ready entity=mon.0 client_cookie=61321891bb0f3a80 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b26ffd700 1 -- 172.21.2.117:0/2269852166 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b20006a50 con 0x7f4b280d15e0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b26ffd700 1 -- 172.21.2.117:0/2269852166 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2000b300 con 0x7f4b280d15e0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b26ffd700 1 -- 172.21.2.117:0/2269852166 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b2000b820 con 0x7f4b280d15e0 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2e424700 1 -- 172.21.2.117:0/2269852166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280d15e0 msgr2=0x7f4b280e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.599 7f4b2e424700 1 --2- 172.21.2.117:0/2269852166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280d15e0 0x7f4b280e8560 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 rx=0x7f4b20006e30 tx=0x7f4b20009e60).stop 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.600 7f4b2e424700 1 -- 172.21.2.117:0/2269852166 shutdown_connections 2021-02-10T11:45:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.600 7f4b2e424700 1 --2- 172.21.2.117:0/2269852166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280d15e0 0x7f4b280e8560 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:14.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.600 7f4b2e424700 1 -- 172.21.2.117:0/2269852166 >> 172.21.2.117:0/2269852166 conn(0x7f4b280ce500 msgr2=0x7f4b280e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:14.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.600 7f4b2e424700 1 -- 172.21.2.117:0/2269852166 shutdown_connections 2021-02-10T11:45:14.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.600 7f4b2e424700 1 -- 172.21.2.117:0/2269852166 wait complete. 2021-02-10T11:45:14.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 Processor -- start 2021-02-10T11:45:14.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 -- start start 2021-02-10T11:45:14.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281c6850 0x7f4b280a6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a76c0 0x7f4b28097fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280a6040 con 0x7f4b280a6ca0 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280985c0 con 0x7f4b280a76c0 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b2e424700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b280987a0 con 0x7f4b281c6850 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51502/0 (socket says 172.21.2.117:51502) 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.615 7f4b277fe700 1 -- 172.21.2.117:0/4109994869 learned_addr learned my addr 172.21.2.117:0/4109994869 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 -- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281c6850 msgr2=0x7f4b280a6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281c6850 0x7f4b280a6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 -- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a76c0 msgr2=0x7f4b28097fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a76c0 0x7f4b28097fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 -- 172.21.2.117:0/4109994869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b280989c0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b277fe700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7f4b1c00b530 tx=0x7f4b1c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b1c00bcc0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c006020 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b1c0054a0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b280989c0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.616 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b280989c0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.617 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b1c006200 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.617 7f4b24ff9700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.617 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5964+0+0 (secure 0 0 0) 0x7f4b1c0329c0 con 0x7f4b280a6ca0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.617 7f4b27fff700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:14.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.618 7f4b27fff700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 0x7f4b0801abc0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:14.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.648 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2807c100 con 0x7f4b280a6ca0 2021-02-10T11:45:14.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.651 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b1c002040 con 0x7f4b280a6ca0 2021-02-10T11:45:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:14.778 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4b2807c100 con 0x7f4b280a6ca0 2021-02-10T11:45:15.465 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:45:15.471 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:45:15.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.478 7f4b24ff9700 1 -- 172.21.2.117:0/4109994869 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4b1c0060e0 con 0x7f4b280a6ca0 2021-02-10T11:45:15.489 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:45:15.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 msgr2=0x7f4b0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 0x7f4b0801abc0 crc :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 msgr2=0x7f4b280a70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 rx=0x7f4b1c00b530 tx=0x7f4b1c00b8d0).stop 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 shutdown_connections 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b080186c0 0x7f4b0801abc0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b280a6ca0 0x7f4b280a70a0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b280a76c0 0x7f4b28097fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 --2- 172.21.2.117:0/4109994869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b281c6850 0x7f4b280a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 >> 172.21.2.117:0/4109994869 conn(0x7f4b28086120 msgr2=0x7f4b2807c640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.497 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 shutdown_connections 2021-02-10T11:45:15.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.498 7f4b2e424700 1 -- 172.21.2.117:0/4109994869 wait complete. 2021-02-10T11:45:15.499 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:15.522 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:45:15.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.916 7f8f4b5a7700 1 Processor -- start 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 -- start start 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440d18e0 0x7f8f440ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f440eb000 0x7f8f440eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8f440ee3e0 0x7f8f440ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f440ede60 con 0x7f8f440eb000 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f440eba20 con 0x7f8f440ee3e0 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f4b5a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440d1ce0 con 0x7f8f440d18e0 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f48bc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f440eb000 0x7f8f440eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f48bc2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8f440eb000 0x7f8f440eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51506/0 (socket says 172.21.2.117:51506) 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.917 7f8f48bc2700 1 -- 172.21.2.117:0/633111449 learned_addr learned my addr 172.21.2.117:0/633111449 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:15.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 -- 172.21.2.117:0/633111449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440d18e0 msgr2=0x7f8f440ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 --2- 172.21.2.117:0/633111449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440d18e0 0x7f8f440ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 -- 172.21.2.117:0/633111449 >> v2:172.21.2.125:3300/0 conn(0x7f8f440ee3e0 msgr2=0x7f8f440ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 --2- 172.21.2.117:0/633111449 >> v2:172.21.2.125:3300/0 conn(0x7f8f440ee3e0 0x7f8f440ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 -- 172.21.2.117:0/633111449 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440eee00 con 0x7f8f440eb000 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f48bc2700 1 --2- 172.21.2.117:0/633111449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440eb000 0x7f8f440eb400 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f8f40006a80 tx=0x7f8f40009e80).ready entity=mon.0 client_cookie=5f42b6daacddde9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f37fff700 1 -- 172.21.2.117:0/633111449 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f4000b510 con 0x7f8f440eb000 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f37fff700 1 -- 172.21.2.117:0/633111449 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000b7d0 con 0x7f8f440eb000 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f37fff700 1 -- 172.21.2.117:0/633111449 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f4000bc20 con 0x7f8f440eb000 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f4b5a7700 1 -- 172.21.2.117:0/633111449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440eb000 msgr2=0x7f8f440eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.918 7f8f4b5a7700 1 --2- 172.21.2.117:0/633111449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440eb000 0x7f8f440eb400 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 rx=0x7f8f40006a80 tx=0x7f8f40009e80).stop 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 -- 172.21.2.117:0/633111449 shutdown_connections 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 --2- 172.21.2.117:0/633111449 >> v2:172.21.2.125:3300/0 conn(0x7f8f440ee3e0 0x7f8f440ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 --2- 172.21.2.117:0/633111449 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440eb000 0x7f8f440eb400 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:15.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 --2- 172.21.2.117:0/633111449 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440d18e0 0x7f8f440ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 -- 172.21.2.117:0/633111449 >> 172.21.2.117:0/633111449 conn(0x7f8f440ced60 msgr2=0x7f8f440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:15.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 -- 172.21.2.117:0/633111449 shutdown_connections 2021-02-10T11:45:15.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.919 7f8f4b5a7700 1 -- 172.21.2.117:0/633111449 wait complete. 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.933 7f8f4b5a7700 1 Processor -- start 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.933 7f8f4b5a7700 1 -- start start 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f440e0cc0 0x7f8f440e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440e1f10 0x7f8f440df510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440e24f0 con 0x7f8f440ded50 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440e9020 con 0x7f8f440e0cc0 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f4b5a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440e95a0 con 0x7f8f440e1f10 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51514/0 (socket says 172.21.2.117:51514) 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 -- 172.21.2.117:0/1875563723 learned_addr learned my addr 172.21.2.117:0/1875563723 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 -- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440e1f10 msgr2=0x7f8f440df510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440e1f10 0x7f8f440df510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 -- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f440e0cc0 msgr2=0x7f8f440e18f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f440e0cc0 0x7f8f440e18f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 -- 172.21.2.117:0/1875563723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440dfb30 con 0x7f8f440ded50 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.934 7f8f493c3700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7f8f3800b5c0 tx=0x7f8f3800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f3800bd50 con 0x7f8f440ded50 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38006030 con 0x7f8f440ded50 2021-02-10T11:45:15.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8f380054d0 con 0x7f8f440ded50 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f440dfb30 con 0x7f8f440ded50 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f440dfb30 con 0x7f8f440ded50 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8f38006210 con 0x7f8f440ded50 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.935 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6022+0+0 (secure 0 0 0) 0x7f8f38032710 con 0x7f8f440ded50 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.936 7f8f48bc2700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:15.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.937 7f8f48bc2700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 0x7f8f2401abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:15.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.967 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f44087210 con 0x7f8f440ded50 2021-02-10T11:45:15.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:15.970 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8f380300a0 con 0x7f8f440ded50 2021-02-10T11:45:16.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.096 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f44087210 con 0x7f8f440ded50 2021-02-10T11:45:16.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.096 7f8f35ffb700 1 -- 172.21.2.117:0/1875563723 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+1753 (secure 0 0 0) 0x7f8f380300a0 con 0x7f8f440ded50 2021-02-10T11:45:16.091 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 55 2021-02-10T11:45:16.091 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:16.091 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8913,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/773459821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":773459821},{"type":"v1","addr":"172.21.2.117:6833","nonce":773459821}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":55,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:44:31.340745","modified":"2021-02-10 11:45:15.472693","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2021-02-10T11:45:16.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 msgr2=0x7f8f2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 0x7f8f2401abc0 crc :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 msgr2=0x7f8f440e12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 rx=0x7f8f3800b5c0 tx=0x7f8f3800b960).stop 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 shutdown_connections 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8f440e1f10 0x7f8f440df510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8f440e0cc0 0x7f8f440e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8f240186c0 0x7f8f2401abc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 --2- 172.21.2.117:0/1875563723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8f440ded50 0x7f8f440e12d0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.098 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 >> 172.21.2.117:0/1875563723 conn(0x7f8f4419f610 msgr2=0x7f8f44199b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.099 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 shutdown_connections 2021-02-10T11:45:16.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.099 7f8f4b5a7700 1 -- 172.21.2.117:0/1875563723 wait complete. 2021-02-10T11:45:16.101 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.522 7ff0908f7700 1 Processor -- start 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.522 7ff0908f7700 1 -- start start 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff0908f7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff08c1b07f0 0x7ff08c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff0908f7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0e77f0 con 0x7ff08c0e6170 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff0908f7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0e7550 con 0x7ff08c0e8d80 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff0908f7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0e54b0 con 0x7ff08c1b07f0 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff08a61c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff08c1b07f0 0x7ff08c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff089e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff08c0e6170 0x7ff08c0e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51768/0 (socket says 172.21.2.117:51768) 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff089e1b700 1 -- 172.21.2.117:0/116987240 learned_addr learned my addr 172.21.2.117:0/116987240 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff08a61c700 1 -- 172.21.2.117:0/116987240 >> v1:172.21.2.125:6789/0 conn(0x7ff08c0e8d80 legacy=0x7ff08c0e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff08a61c700 1 -- 172.21.2.117:0/116987240 >> v1:172.21.2.117:6789/0 conn(0x7ff08c0e6170 legacy=0x7ff08c0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.523 7ff08a61c700 1 -- 172.21.2.117:0/116987240 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0e9180 con 0x7ff08c1b07f0 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff08a61c700 1 --2- 172.21.2.117:0/116987240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c1b07f0 0x7ff08c0e4e90 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ff07c006650 tx=0x7ff07c009a80).ready entity=mon.0 client_cookie=34c2de17f7fb345b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff08961a700 1 -- 172.21.2.117:0/116987240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c0009b0 con 0x7ff08c1b07f0 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff08961a700 1 -- 172.21.2.117:0/116987240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c00c0c0 con 0x7ff08c1b07f0 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff08961a700 1 -- 172.21.2.117:0/116987240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c00c5e0 con 0x7ff08c1b07f0 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff0908f7700 1 -- 172.21.2.117:0/116987240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c1b07f0 msgr2=0x7ff08c0e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.524 7ff0908f7700 1 --2- 172.21.2.117:0/116987240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c1b07f0 0x7ff08c0e4e90 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7ff07c006650 tx=0x7ff07c009a80).stop 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.525 7ff0908f7700 1 -- 172.21.2.117:0/116987240 shutdown_connections 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.525 7ff0908f7700 1 --2- 172.21.2.117:0/116987240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c1b07f0 0x7ff08c0e4e90 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.525 7ff0908f7700 1 -- 172.21.2.117:0/116987240 >> 172.21.2.117:0/116987240 conn(0x7ff08c0ce500 msgr2=0x7ff08c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.525 7ff0908f7700 1 -- 172.21.2.117:0/116987240 shutdown_connections 2021-02-10T11:45:16.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.525 7ff0908f7700 1 -- 172.21.2.117:0/116987240 wait complete. 2021-02-10T11:45:16.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.539 7ff0908f7700 1 Processor -- start 2021-02-10T11:45:16.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.539 7ff0908f7700 1 -- start start 2021-02-10T11:45:16.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff08c080290 0x7ff08c080690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c08cf20 con 0x7ff08c08c230 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c08cc50 con 0x7ff08c084f70 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff0908f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0bc130 con 0x7ff08c080290 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff089e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff089e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51524/0 (socket says 172.21.2.117:51524) 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff089e1b700 1 -- 172.21.2.117:0/2995484290 learned_addr learned my addr 172.21.2.117:0/2995484290 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:55906/0 (socket says 172.21.2.117:55906) 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08ae1d700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff08c080290 0x7ff08c080690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 -- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff08c080290 msgr2=0x7ff08c080690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff08c080290 0x7ff08c080690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 -- 172.21.2.117:0/2995484290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 msgr2=0x7ff08c08c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff08a61c700 1 -- 172.21.2.117:0/2995484290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c1c6830 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.540 7ff089e1b700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff08a61c700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7ff07c00b4b0 tx=0x7ff07c00b850).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c009280 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c00bc40 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff07c0022b0 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c1c6830 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c1c6830 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff07c009280 con 0x7ff08c084f70 2021-02-10T11:45:16.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 0x7ff07001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.541 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6022+0+0 (secure 0 0 0) 0x7ff07c0328e0 con 0x7ff08c084f70 2021-02-10T11:45:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.542 7ff089e1b700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 0x7ff07001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:16.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.542 7ff089e1b700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 0x7ff07001ac10 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:16.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.572 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c07c060 con 0x7ff08c084f70 2021-02-10T11:45:16.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.576 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff07c01f4a0 con 0x7ff08c084f70 2021-02-10T11:45:16.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:16.707 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff08c07c060 con 0x7ff08c084f70 2021-02-10T11:45:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.481 7ff07affd700 1 -- 172.21.2.117:0/2995484290 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff07c009280 con 0x7ff08c084f70 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 msgr2=0x7ff07001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 0x7ff07001ac10 crc :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 msgr2=0x7ff08c1c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7ff07c00b4b0 tx=0x7ff07c00b850).stop 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 shutdown_connections 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff070018730 0x7ff07001ac10 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff08c08c230 0x7ff08c08c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff08c080290 0x7ff08c080690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 --2- 172.21.2.117:0/2995484290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff08c084f70 0x7ff08c1c6210 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 >> 172.21.2.117:0/2995484290 conn(0x7ff08c09b070 msgr2=0x7ff08c09a750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:17.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 shutdown_connections 2021-02-10T11:45:17.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.528 7ff0908f7700 1 -- 172.21.2.117:0/2995484290 wait complete. 2021-02-10T11:45:17.530 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:45:17.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.951 7fd691e5e700 1 Processor -- start 2021-02-10T11:45:17.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.951 7fd691e5e700 1 -- start start 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd691e5e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd68c0e4e40 0x7fd68c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd691e5e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c0e5240 0x7fd68c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd691e5e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c085930 con 0x7fd68c1b0810 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd691e5e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68c084840 con 0x7fd68c0e4e40 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd691e5e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c084a30 con 0x7fd68c0e5240 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd68c0e4e40 0x7fd68c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd68c1b0810 0x7fd68c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51780/0 (socket says 172.21.2.117:51780) 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68bfff700 1 -- 172.21.2.117:0/3441349383 learned_addr learned my addr 172.21.2.117:0/3441349383 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68b7fe700 1 -- 172.21.2.117:0/3441349383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c0e5240 msgr2=0x7fd68c084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68b7fe700 1 --2- 172.21.2.117:0/3441349383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c0e5240 0x7fd68c084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68b7fe700 1 -- 172.21.2.117:0/3441349383 >> v1:172.21.2.117:6789/0 conn(0x7fd68c1b0810 legacy=0x7fd68c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:17.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.952 7fd68b7fe700 1 -- 172.21.2.117:0/3441349383 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c085280 con 0x7fd68c0e4e40 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd68b7fe700 1 --2- 172.21.2.117:0/3441349383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c0e4e40 0x7fd68c1b01f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7fd674006650 tx=0x7fd674009a00).ready entity=mon.0 client_cookie=c12a2dd56ba89532 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd68a7fc700 1 -- 172.21.2.117:0/3441349383 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd6740009b0 con 0x7fd68c0e4e40 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd68a7fc700 1 -- 172.21.2.117:0/3441349383 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67400c0c0 con 0x7fd68c0e4e40 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd68a7fc700 1 -- 172.21.2.117:0/3441349383 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd67400c5a0 con 0x7fd68c0e4e40 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 -- 172.21.2.117:0/3441349383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c0e4e40 msgr2=0x7fd68c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 --2- 172.21.2.117:0/3441349383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c0e4e40 0x7fd68c1b01f0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 rx=0x7fd674006650 tx=0x7fd674009a00).stop 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 -- 172.21.2.117:0/3441349383 shutdown_connections 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 --2- 172.21.2.117:0/3441349383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c0e5240 0x7fd68c084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 --2- 172.21.2.117:0/3441349383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c0e4e40 0x7fd68c1b01f0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.953 7fd691e5e700 1 -- 172.21.2.117:0/3441349383 >> 172.21.2.117:0/3441349383 conn(0x7fd68c0ce4e0 msgr2=0x7fd68c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.954 7fd691e5e700 1 -- 172.21.2.117:0/3441349383 shutdown_connections 2021-02-10T11:45:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.954 7fd691e5e700 1 -- 172.21.2.117:0/3441349383 wait complete. 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 Processor -- start 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 -- start start 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c1b1f90 0x7fd68c1b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd68c094390 0x7fd68c094790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1110e0 con 0x7fd68c1b1570 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1b0ff0 con 0x7fd68c094390 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.968 7fd691e5e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c094db0 con 0x7fd68c1b1f90 2021-02-10T11:45:17.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51536/0 (socket says 172.21.2.117:51536) 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 -- 172.21.2.117:0/654923490 learned_addr learned my addr 172.21.2.117:0/654923490 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 -- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c1b1f90 msgr2=0x7fd68c1b2390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c1b1f90 0x7fd68c1b2390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 -- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd68c094390 msgr2=0x7fd68c094790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd68c094390 0x7fd68c094790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 -- 172.21.2.117:0/654923490 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c094f90 con 0x7fd68c1b1570 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd68b7fe700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7fd67400b640 tx=0x7fd67400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd67400bdd0 con 0x7fd68c1b1570 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd674002030 con 0x7fd68c1b1570 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd691e5e700 1 -- 172.21.2.117:0/654923490 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c094f90 con 0x7fd68c1b1570 2021-02-10T11:45:17.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd691e5e700 1 -- 172.21.2.117:0/654923490 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c095520 con 0x7fd68c1b1570 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.969 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd674002550 con 0x7fd68c1b1570 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.970 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd674006a10 con 0x7fd68c1b1570 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.970 7fd683fff700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 0x7fd67001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.970 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fd674032850 con 0x7fd68c1b1570 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.970 7fd68affd700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 0x7fd67001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:17.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:17.971 7fd68affd700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 0x7fd67001ac10 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:17.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.002 7fd691e5e700 1 -- 172.21.2.117:0/654923490 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c10c2f0 con 0x7fd68c1b1570 2021-02-10T11:45:17.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.004 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd6740009b0 con 0x7fd68c1b1570 2021-02-10T11:45:18.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.130 7fd691e5e700 1 -- 172.21.2.117:0/654923490 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd68c085930 con 0x7fd68c1b1570 2021-02-10T11:45:18.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.489 7fd683fff700 1 -- 172.21.2.117:0/654923490 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v77) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd674006a10 con 0x7fd68c1b1570 2021-02-10T11:45:18.489 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:45:18.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 msgr2=0x7fd67001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 0x7fd67001ac10 crc :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 msgr2=0x7fd68c1b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 rx=0x7fd67400b640 tx=0x7fd67400b9e0).stop 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 shutdown_connections 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd68c094390 0x7fd68c094790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd68c1b1f90 0x7fd68c1b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd670018730 0x7fd67001ac10 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 --2- 172.21.2.117:0/654923490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd68c1b1570 0x7fd68c1b1970 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 >> 172.21.2.117:0/654923490 conn(0x7fd68c1b3150 msgr2=0x7fd68c1ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 shutdown_connections 2021-02-10T11:45:18.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.497 7fd691e5e700 1 -- 172.21.2.117:0/654923490 wait complete. 2021-02-10T11:45:18.499 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 Processor -- start 2021-02-10T11:45:18.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 -- start start 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb00e7170 0x7fafb00e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fafb00e8380 0x7fafb00e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb00e7570 con 0x7fafb01b07e0 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00e5cd0 con 0x7fafb00e7170 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.919 7fafb5b0c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb00e77e0 con 0x7fafb00e8380 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb00e7170 0x7fafb00e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fafb00e7170 0x7fafb00e7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51542/0 (socket says 172.21.2.117:51542) 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaf7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fafb01b07e0 0x7fafb00e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51792/0 (socket says 172.21.2.117:51792) 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 -- 172.21.2.117:0/3465381746 learned_addr learned my addr 172.21.2.117:0/3465381746 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaffff700 1 --2- 172.21.2.117:0/3465381746 >> v2:172.21.2.125:3300/0 conn(0x7fafb00e8380 0x7fafb00e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 -- 172.21.2.117:0/3465381746 >> v2:172.21.2.125:3300/0 conn(0x7fafb00e8380 msgr2=0x7fafb00e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 --2- 172.21.2.117:0/3465381746 >> v2:172.21.2.125:3300/0 conn(0x7fafb00e8380 0x7fafb00e8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 -- 172.21.2.117:0/3465381746 >> v1:172.21.2.117:6789/0 conn(0x7fafb01b07e0 legacy=0x7fafb00e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 -- 172.21.2.117:0/3465381746 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00e8da0 con 0x7fafb00e7170 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.920 7fafaeffd700 1 --2- 172.21.2.117:0/3465381746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00e7170 0x7fafb00e7d60 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7fafa4006700 tx=0x7fafa4009b80).ready entity=mon.0 client_cookie=6952faeee8668a5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.921 7fafae7fc700 1 -- 172.21.2.117:0/3465381746 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa400c0c0 con 0x7fafb00e7170 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.921 7fafae7fc700 1 -- 172.21.2.117:0/3465381746 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa400c440 con 0x7fafb00e7170 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.921 7fafae7fc700 1 -- 172.21.2.117:0/3465381746 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa400c960 con 0x7fafb00e7170 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.921 7fafb5b0c700 1 -- 172.21.2.117:0/3465381746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00e7170 msgr2=0x7fafb00e7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.921 7fafb5b0c700 1 --2- 172.21.2.117:0/3465381746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00e7170 0x7fafb00e7d60 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 rx=0x7fafa4006700 tx=0x7fafa4009b80).stop 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 -- 172.21.2.117:0/3465381746 shutdown_connections 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 --2- 172.21.2.117:0/3465381746 >> v2:172.21.2.125:3300/0 conn(0x7fafb00e8380 0x7fafb00e8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 --2- 172.21.2.117:0/3465381746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00e7170 0x7fafb00e7d60 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 -- 172.21.2.117:0/3465381746 >> 172.21.2.117:0/3465381746 conn(0x7fafb00ce4f0 msgr2=0x7fafb00e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 -- 172.21.2.117:0/3465381746 shutdown_connections 2021-02-10T11:45:18.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.922 7fafb5b0c700 1 -- 172.21.2.117:0/3465381746 wait complete. 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.936 7fafb5b0c700 1 Processor -- start 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.936 7fafb5b0c700 1 -- start start 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb01aa0c0 0x7fafb01a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb00a8350 0x7fafb01aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0106ed0 con 0x7fafb00a7f50 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb01ab1d0 con 0x7fafb00a8350 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafb5b0c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb01ab3b0 con 0x7fafb01aa0c0 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51546/0 (socket says 172.21.2.117:51546) 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 -- 172.21.2.117:0/2489070088 learned_addr learned my addr 172.21.2.117:0/2489070088 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaf7fe700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb01aa0c0 0x7fafb01a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaffff700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb00a8350 0x7fafb01aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 -- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb01aa0c0 msgr2=0x7fafb01a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb01aa0c0 0x7fafb01a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 -- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb00a8350 msgr2=0x7fafb01aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb00a8350 0x7fafb01aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.937 7fafaeffd700 1 -- 172.21.2.117:0/2489070088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0107450 con 0x7fafb00a7f50 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7fafaeffd700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7fafa400b580 tx=0x7fafa400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa400bd10 con 0x7fafb00a7f50 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4002030 con 0x7fafb00a7f50 2021-02-10T11:45:18.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fafa4002480 con 0x7fafb00a7f50 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb0107450 con 0x7fafb00a7f50 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb0107450 con 0x7fafb00a7f50 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fafa4000c90 con 0x7fafb00a7f50 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.938 7faf97fff700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 0x7faf9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.939 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5584+0+0 (secure 0 0 0) 0x7fafa4032630 con 0x7fafb00a7f50 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.939 7fafaf7fe700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 0x7faf9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:18.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.939 7fafaf7fe700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 0x7faf9001abc0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:18.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.969 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb007e840 con 0x7fafb00a7f50 2021-02-10T11:45:18.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:18.972 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fafa4000960 con 0x7fafb00a7f50 2021-02-10T11:45:19.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.097 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fafb00e77e0 con 0x7fafb00a7f50 2021-02-10T11:45:19.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.494 7faf97fff700 1 -- 172.21.2.117:0/2489070088 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v78) v1 ==== 173+0+0 (secure 0 0 0) 0x7fafa4000c90 con 0x7fafb00a7f50 2021-02-10T11:45:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:45:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.515 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 msgr2=0x7faf9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.515 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 0x7faf9001abc0 crc :-1 s=READY pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.515 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 msgr2=0x7fafb01aa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.515 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 rx=0x7fafa400b580 tx=0x7fafa400b920).stop 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 shutdown_connections 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faf90018730 0x7faf9001abc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fafb00a7f50 0x7fafb01aa590 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fafb00a8350 0x7fafb01aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 --2- 172.21.2.117:0/2489070088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fafb01aa0c0 0x7fafb01a9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 >> 172.21.2.117:0/2489070088 conn(0x7fafb00809f0 msgr2=0x7fafb019a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 shutdown_connections 2021-02-10T11:45:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.516 7fafb5b0c700 1 -- 172.21.2.117:0/2489070088 wait complete. 2021-02-10T11:45:19.518 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:45:19.519 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:45:19.519 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:45:19.519 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:45:19.519 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:45:19.519 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:45:19.519 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:45:19.519 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:45:19.525 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:45:19.525 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:45:19.526 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:45:19.526 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:45:19.568 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:45:19.569 DEBUG:teuthology.parallel:result is None 2021-02-10T11:45:19.570 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:45:19.570 DEBUG:teuthology.parallel:result is None 2021-02-10T11:45:19.571 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:45:19.973 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:45:19.975 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 Processor -- start 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 -- start start 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03a80e7180 0x7f03a80e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a80e8390 0x7f03a80e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a80e7580 con 0x7f03a81b07f0 2021-02-10T11:45:19.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a80e5ce0 con 0x7f03a80e7180 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03aecc7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80e77f0 con 0x7f03a80e8390 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03a80e7180 0x7f03a80e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03a80e7180 0x7f03a80e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51572/0 (socket says 172.21.2.117:51572) 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 -- 172.21.2.117:0/404824939 learned_addr learned my addr 172.21.2.117:0/404824939 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03ad2e4700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a80e8390 0x7f03a80e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 -- 172.21.2.117:0/404824939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a80e8390 msgr2=0x7f03a80e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a80e8390 0x7f03a80e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 -- 172.21.2.117:0/404824939 >> v1:172.21.2.125:6789/0 conn(0x7f03a81b07f0 legacy=0x7f03a80e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.985 7f03a7fff700 1 -- 172.21.2.117:0/404824939 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a80e8db0 con 0x7f03a80e7180 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03a7fff700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80e7180 0x7f03a80e7d70 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7f0398006a80 tx=0x7f0398009e80).ready entity=mon.0 client_cookie=40c2518627e46bc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03a77fe700 1 -- 172.21.2.117:0/404824939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039800b510 con 0x7f03a80e7180 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03a77fe700 1 -- 172.21.2.117:0/404824939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039800b750 con 0x7f03a80e7180 2021-02-10T11:45:19.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03a77fe700 1 -- 172.21.2.117:0/404824939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039800bc70 con 0x7f03a80e7180 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 -- 172.21.2.117:0/404824939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80e7180 msgr2=0x7f03a80e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80e7180 0x7f03a80e7d70 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 rx=0x7f0398006a80 tx=0x7f0398009e80).stop 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 -- 172.21.2.117:0/404824939 shutdown_connections 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a80e8390 0x7f03a80e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 --2- 172.21.2.117:0/404824939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80e7180 0x7f03a80e7d70 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 -- 172.21.2.117:0/404824939 >> 172.21.2.117:0/404824939 conn(0x7f03a80ce500 msgr2=0x7f03a80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 -- 172.21.2.117:0/404824939 shutdown_connections 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:19.986 7f03aecc7700 1 -- 172.21.2.117:0/404824939 wait complete. 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.001 7f03aecc7700 1 Processor -- start 2021-02-10T11:45:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.001 7f03aecc7700 1 -- start start 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 0x7f03a8199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 0x7f03a8199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a808cca0 con 0x7f03a80af5f0 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8199c90 con 0x7f03a808c780 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03aecc7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8199e70 con 0x7f03a808c380 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03acae3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03acae3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51578/0 (socket says 172.21.2.117:51578) 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03acae3700 1 -- 172.21.2.117:0/3136627151 learned_addr learned my addr 172.21.2.117:0/3136627151 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:19.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 0x7f03a8199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03ad2e4700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 0x7f03a8199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 -- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 msgr2=0x7f03a8199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 0x7f03a8199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 -- 172.21.2.117:0/3136627151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 msgr2=0x7f03a808bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03a7fff700 1 -- 172.21.2.117:0/3136627151 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a809d790 con 0x7f03a808c380 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03acae3700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.002 7f03ad2e4700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 0x7f03a8199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a7fff700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 0x7f03a8199050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f039800b8c0 tx=0x7f039800bc60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039800c2f0 con 0x7f03a808c380 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039800c630 con 0x7f03a808c380 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f039800cb10 con 0x7f03a808c380 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a809d790 con 0x7f03a808c380 2021-02-10T11:45:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a809d790 con 0x7f03a808c380 2021-02-10T11:45:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0398001010 con 0x7f03a808c380 2021-02-10T11:45:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 0x7f038c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.003 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f0398032520 con 0x7f03a808c380 2021-02-10T11:45:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.004 7f03acae3700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 0x7f038c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.004 7f03acae3700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 0x7f038c01abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.034 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a807e800 con 0x7f03a808c380 2021-02-10T11:45:20.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.037 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f039800c2f0 con 0x7f03a808c380 2021-02-10T11:45:20.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.178 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f03a807e800 con 0x7f03a808c380 2021-02-10T11:45:20.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.497 7f03a57fa700 1 -- 172.21.2.117:0/3136627151 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0398001010 con 0x7f03a808c380 2021-02-10T11:45:20.539 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 msgr2=0x7f038c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 0x7f038c01abc0 crc :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 msgr2=0x7f03a8199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 0x7f03a8199050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f039800b8c0 tx=0x7f039800bc60).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 shutdown_connections 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03a808c780 0x7f03a8199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03a808c380 0x7f03a8199050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f038c0186c0 0x7f038c01abc0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 --2- 172.21.2.117:0/3136627151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03a80af5f0 0x7f03a808bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 >> 172.21.2.117:0/3136627151 conn(0x7f03a80953a0 msgr2=0x7f03a809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 shutdown_connections 2021-02-10T11:45:20.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.547 7f03aecc7700 1 -- 172.21.2.117:0/3136627151 wait complete. 2021-02-10T11:45:20.549 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:45:20.549 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:45:20.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 Processor -- start 2021-02-10T11:45:20.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 -- start start 2021-02-10T11:45:20.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0d18e0 0x7f187c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f187c0eb000 0x7f187c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c0ede60 con 0x7f187c0ee3e0 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187c0eba20 con 0x7f187c0eb000 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.966 7f1880f3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0d1ce0 con 0x7f187c0d18e0 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187b314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f187c0ee3e0 0x7f187c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51838/0 (socket says 172.21.2.117:51838) 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187b314700 1 -- 172.21.2.117:0/1403849876 learned_addr learned my addr 172.21.2.117:0/1403849876 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187ab13700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0d18e0 0x7f187c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 --2- 172.21.2.117:0/1403849876 >> v2:172.21.2.125:3300/0 conn(0x7f187c0eb000 0x7f187c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f1879b11700 1 -- 172.21.2.117:0/1403849876 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440137216 0 0) 0x7f1874001500 con 0x7f187c0ee3e0 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 -- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0d18e0 msgr2=0x7f187c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0d18e0 0x7f187c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 -- 172.21.2.117:0/1403849876 >> v1:172.21.2.117:6789/0 conn(0x7f187c0ee3e0 legacy=0x7f187c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 -- 172.21.2.117:0/1403849876 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c0eefd0 con 0x7f187c0eb000 2021-02-10T11:45:20.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.967 7f187a312700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0eb000 0x7f187c0eb400 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f1864006650 tx=0x7f1864009a80).ready entity=mon.1 client_cookie=c34d0c7a61ab6215 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1879b11700 1 -- 172.21.2.117:0/1403849876 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18640009b0 con 0x7f187c0eb000 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1879b11700 1 -- 172.21.2.117:0/1403849876 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186400c0c0 con 0x7f187c0eb000 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1879b11700 1 -- 172.21.2.117:0/1403849876 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186400c5e0 con 0x7f187c0eb000 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 -- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0eb000 msgr2=0x7f187c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0eb000 0x7f187c0eb400 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f1864006650 tx=0x7f1864009a80).stop 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 -- 172.21.2.117:0/1403849876 shutdown_connections 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0eb000 0x7f187c0eb400 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 --2- 172.21.2.117:0/1403849876 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0d18e0 0x7f187c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.968 7f1880f3f700 1 -- 172.21.2.117:0/1403849876 >> 172.21.2.117:0/1403849876 conn(0x7f187c0ced60 msgr2=0x7f187c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.969 7f1880f3f700 1 -- 172.21.2.117:0/1403849876 shutdown_connections 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.969 7f1880f3f700 1 -- 172.21.2.117:0/1403849876 wait complete. 2021-02-10T11:45:20.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.983 7f1880f3f700 1 Processor -- start 2021-02-10T11:45:20.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.983 7f1880f3f700 1 -- start start 2021-02-10T11:45:20.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0ab280 0x7f187c0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0a6fe0 0x7f187c0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0a7d10 con 0x7f187c1a4290 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187c0a7a00 con 0x7f187c0a6fe0 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f1880f3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187c096370 con 0x7f187c0ab280 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51592/0 (socket says 172.21.2.117:51592) 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 -- 172.21.2.117:0/3295007885 learned_addr learned my addr 172.21.2.117:0/3295007885 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187b314700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0a6fe0 0x7f187c0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187a312700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0ab280 0x7f187c0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 -- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0ab280 msgr2=0x7f187c0ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0ab280 0x7f187c0ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 -- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0a6fe0 msgr2=0x7f187c0a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.984 7f187ab13700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0a6fe0 0x7f187c0a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f187ab13700 1 -- 172.21.2.117:0/3295007885 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c0a8290 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f187ab13700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7f186c006f60 tx=0x7f186c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c00bd60 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c000c90 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f186c0022b0 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c0a8290 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.985 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187c0a8290 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.986 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f186c006b00 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.986 7f18737fe700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 0x7f186001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.986 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f186c032380 con 0x7f187c1a4290 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.986 7f187a312700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 0x7f186001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:20.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:20.987 7f187a312700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 0x7f186001abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.016 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187c087290 con 0x7f187c1a4290 2021-02-10T11:45:21.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.019 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f186c015240 con 0x7f187c1a4290 2021-02-10T11:45:21.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.152 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f187c087290 con 0x7f187c1a4290 2021-02-10T11:45:21.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.152 7f18737fe700 1 -- 172.21.2.117:0/3295007885 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f186c015240 con 0x7f187c1a4290 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.147 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.148 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.149 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.150 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:45:21.151 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:45:21.153 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.154 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.155 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.156 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:45:21.157 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:45:21.158 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.159 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:21.159 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:45:21.159 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.154 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 msgr2=0x7f186001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.154 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 0x7f186001abc0 crc :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.154 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 msgr2=0x7f187c1a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.154 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 rx=0x7f186c006f60 tx=0x7f186c00b970).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 shutdown_connections 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f187c0a6fe0 0x7f187c0a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f187c0ab280 0x7f187c0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18600186c0 0x7f186001abc0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 --2- 172.21.2.117:0/3295007885 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f187c1a4290 0x7f187c1a4690 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 >> 172.21.2.117:0/3295007885 conn(0x7f187c0874c0 msgr2=0x7f187c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:21.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 shutdown_connections 2021-02-10T11:45:21.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.155 7f1880f3f700 1 -- 172.21.2.117:0/3295007885 wait complete. 2021-02-10T11:45:21.161 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.579 7f89c84c1700 1 Processor -- start 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.579 7f89c84c1700 1 -- start start 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c84c1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f89c01b6960 0x7f89c01b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c84c1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c00ede60 con 0x7f89c00d18e0 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c84c1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c00eb9c0 con 0x7f89c00ef3f0 2021-02-10T11:45:21.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c84c1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c00ef7f0 con 0x7f89c01b6960 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c62dd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f89c00d18e0 0x7f89c00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51850/0 (socket says 172.21.2.117:51850) 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c62dd700 1 -- 172.21.2.117:0/113448720 learned_addr learned my addr 172.21.2.117:0/113448720 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c6ade700 1 --2- 172.21.2.117:0/113448720 >> v2:172.21.2.117:3300/0 conn(0x7f89c01b6960 0x7f89c01b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c52db700 1 -- 172.21.2.117:0/113448720 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330205537 0 0) 0x7f89bc0009b0 con 0x7f89c00d18e0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c52db700 1 -- 172.21.2.117:0/113448720 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ac002fe0 con 0x7f89c00d18e0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c52db700 1 -- 172.21.2.117:0/113448720 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366087007 0 0) 0x7f89b4000c60 con 0x7f89c00ef3f0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c52db700 1 -- 172.21.2.117:0/113448720 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ac004370 con 0x7f89c00ef3f0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.580 7f89c6ade700 1 -- 172.21.2.117:0/113448720 >> v1:172.21.2.125:6789/0 conn(0x7f89c00ef3f0 legacy=0x7f89c00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c6ade700 1 -- 172.21.2.117:0/113448720 >> v1:172.21.2.117:6789/0 conn(0x7f89c00d18e0 legacy=0x7f89c00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c6ade700 1 -- 172.21.2.117:0/113448720 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c00d1d20 con 0x7f89c01b6960 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c6ade700 1 --2- 172.21.2.117:0/113448720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c01b6960 0x7f89c01b8d70 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f89b8006e30 tx=0x7f89b8009e60).ready entity=mon.0 client_cookie=e469ccd7aa6b3939 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c52db700 1 -- 172.21.2.117:0/113448720 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b8006a50 con 0x7f89c01b6960 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c52db700 1 -- 172.21.2.117:0/113448720 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b800b300 con 0x7f89c01b6960 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c52db700 1 -- 172.21.2.117:0/113448720 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89b800b7a0 con 0x7f89c01b6960 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c84c1700 1 -- 172.21.2.117:0/113448720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c01b6960 msgr2=0x7f89c01b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.581 7f89c84c1700 1 --2- 172.21.2.117:0/113448720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c01b6960 0x7f89c01b8d70 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 rx=0x7f89b8006e30 tx=0x7f89b8009e60).stop 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.582 7f89c84c1700 1 -- 172.21.2.117:0/113448720 shutdown_connections 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.582 7f89c84c1700 1 --2- 172.21.2.117:0/113448720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c01b6960 0x7f89c01b8d70 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.582 7f89c84c1700 1 -- 172.21.2.117:0/113448720 >> 172.21.2.117:0/113448720 conn(0x7f89c00ced60 msgr2=0x7f89c00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.582 7f89c84c1700 1 -- 172.21.2.117:0/113448720 shutdown_connections 2021-02-10T11:45:21.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.582 7f89c84c1700 1 -- 172.21.2.117:0/113448720 wait complete. 2021-02-10T11:45:21.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 Processor -- start 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 -- start start 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 0x7f89c008ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 0x7f89c00ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 0x7f89c008a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c008b270 con 0x7f89c00ae480 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c008aca0 con 0x7f89c00979b0 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c84c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c008ae80 con 0x7f89c008a280 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c62dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 0x7f89c008ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c5adc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 0x7f89c00ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c6ade700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 0x7f89c008a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c5adc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 0x7f89c00ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51608/0 (socket says 172.21.2.117:51608) 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.597 7f89c5adc700 1 -- 172.21.2.117:0/4021396756 learned_addr learned my addr 172.21.2.117:0/4021396756 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 -- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 msgr2=0x7f89c008a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 0x7f89c008a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 -- 172.21.2.117:0/4021396756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 msgr2=0x7f89c00ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 0x7f89c00ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 -- 172.21.2.117:0/4021396756 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c008b7f0 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c6ade700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 0x7f89c008a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c62dd700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 0x7f89c008ddd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f89bc001c60 tx=0x7f89bc00a870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89bc000cc0 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc0009b0 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c008b7f0 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89bc004350 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.598 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c008bd80 con 0x7f89c00979b0 2021-02-10T11:45:21.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.599 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89bc004140 con 0x7f89c00979b0 2021-02-10T11:45:21.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.599 7f89b2ffd700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 0x7f89a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:21.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.599 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f89bc032370 con 0x7f89c00979b0 2021-02-10T11:45:21.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.599 7f89c5adc700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 0x7f89a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:21.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.599 7f89c5adc700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 0x7f89a801abd0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:21.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.629 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c008e440 con 0x7f89c00979b0 2021-02-10T11:45:21.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.634 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89bc000960 con 0x7f89c00979b0 2021-02-10T11:45:21.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.760 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89c008e440 con 0x7f89c00979b0 2021-02-10T11:45:21.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.762 7f89b2ffd700 1 -- 172.21.2.117:0/4021396756 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+11378 (secure 0 0 0) 0x7f89bc004140 con 0x7f89c00979b0 2021-02-10T11:45:21.757 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:21.757 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":79,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:20.494559","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:21.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 msgr2=0x7f89a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 0x7f89a801abd0 crc :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 msgr2=0x7f89c008ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 0x7f89c008ddd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f89bc001c60 tx=0x7f89bc00a870).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 shutdown_connections 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89a80186d0 0x7f89a801abd0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89c00ae480 0x7f89c00ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89c008a280 0x7f89c008a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 --2- 172.21.2.117:0/4021396756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89c00979b0 0x7f89c008ddd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.764 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 >> 172.21.2.117:0/4021396756 conn(0x7f89c01a3620 msgr2=0x7f89c01a93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.765 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 shutdown_connections 2021-02-10T11:45:21.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:21.765 7f89c84c1700 1 -- 172.21.2.117:0/4021396756 wait complete. 2021-02-10T11:45:21.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:45:21.768 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.186 7f687443a700 1 Processor -- start 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.186 7f687443a700 1 -- start start 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f687443a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c0d18f0 0x7f686c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f687443a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f686c0ef3f0 0x7f686c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f687443a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0ede60 con 0x7f686c0eb040 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f687443a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686c088020 con 0x7f686c0ef3f0 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f687443a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c0d1d30 con 0x7f686c0d18f0 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6871a55700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f686c0eb040 0x7f686c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51862/0 (socket says 172.21.2.117:51862) 2021-02-10T11:45:22.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6871a55700 1 -- 172.21.2.117:0/3841905335 learned_addr learned my addr 172.21.2.117:0/3841905335 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816236302 0 0) 0x7f68600010b0 con 0x7f686c0eb040 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6872256700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c0d18f0 0x7f686c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6871254700 1 -- 172.21.2.117:0/3841905335 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6858002fe0 con 0x7f686c0eb040 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.187 7f6872a57700 1 --2- 172.21.2.117:0/3841905335 >> v2:172.21.2.125:3300/0 conn(0x7f686c0ef3f0 0x7f686c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4195461898 0 0) 0x7f68600010b0 con 0x7f686c0eb040 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c0d18f0 msgr2=0x7f686c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c0d18f0 0x7f686c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 >> v2:172.21.2.125:3300/0 conn(0x7f686c0ef3f0 msgr2=0x7f686c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 --2- 172.21.2.117:0/3841905335 >> v2:172.21.2.125:3300/0 conn(0x7f686c0ef3f0 0x7f686c1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c1b8fd0 con 0x7f686c0eb040 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f68600015b0 con 0x7f686c0eb040 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 >> v1:172.21.2.117:6789/0 conn(0x7f686c0eb040 legacy=0x7f686c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6858003b10 0x7f6858005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f685804a790 0x7f685804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f685804d510 0x7f685804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580502e0 con 0x7f685804d510 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6858050500 con 0x7f685804a790 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871254700 1 -- 172.21.2.117:0/3841905335 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6858050720 con 0x7f6858003b10 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f685804d510 0x7f685804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872256700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6858003b10 0x7f6858005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6871a55700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f685804a790 0x7f685804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 -- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6858003b10 msgr2=0x7f6858005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6858003b10 0x7f6858005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 -- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f685804a790 msgr2=0x7f685804cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f685804a790 0x7f685804cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.188 7f6872a57700 1 -- 172.21.2.117:0/3841905335 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580509c0 con 0x7f685804d510 2021-02-10T11:45:22.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f6872a57700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f685804d510 0x7f685804fa30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7f6864006a40 tx=0x7f686400c0c0).ready entity=mon.0 client_cookie=f992f452d784eb21 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686400c4b0 con 0x7f685804d510 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686400c770 con 0x7f685804d510 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f6871254700 1 -- 172.21.2.117:0/3841905335 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686400cc90 con 0x7f685804d510 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f687443a700 1 -- 172.21.2.117:0/3841905335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f685804d510 msgr2=0x7f685804fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f687443a700 1 --2- 172.21.2.117:0/3841905335 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f685804d510 0x7f685804fa30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 rx=0x7f6864006a40 tx=0x7f686400c0c0).stop 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.189 7f6872a57700 1 -- 172.21.2.117:0/3841905335 reap_dead start 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.190 7f687443a700 1 -- 172.21.2.117:0/3841905335 shutdown_connections 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.190 7f687443a700 1 -- 172.21.2.117:0/3841905335 >> 172.21.2.117:0/3841905335 conn(0x7f686c0cf060 msgr2=0x7f686c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.190 7f687443a700 1 -- 172.21.2.117:0/3841905335 shutdown_connections 2021-02-10T11:45:22.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.190 7f687443a700 1 -- 172.21.2.117:0/3841905335 wait complete. 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.204 7f687443a700 1 Processor -- start 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.204 7f687443a700 1 -- start start 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f686c0a8eb0 0x7f686c0a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 0x7f686c0a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c0a9cf0 con 0x7f686c0a98f0 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c09f9f0 con 0x7f686c0a8eb0 2021-02-10T11:45:22.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f687443a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c09fc10 con 0x7f686c09f5f0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f6872a57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f6872a57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51626/0 (socket says 172.21.2.117:51626) 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.205 7f6872a57700 1 -- 172.21.2.117:0/2494113780 learned_addr learned my addr 172.21.2.117:0/2494113780 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872256700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f686c0a8eb0 0x7f686c0a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6871a55700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 0x7f686c0a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 -- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 msgr2=0x7f686c0a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 0x7f686c0a92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 -- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f686c0a8eb0 msgr2=0x7f686c0a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f686c0a8eb0 0x7f686c0a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 -- 172.21.2.117:0/2494113780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c07f310 con 0x7f686c0a98f0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6871a55700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 0x7f686c0a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f6872a57700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f6864006d20 tx=0x7f686400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6864009500 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686400c2f0 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f687443a700 1 -- 172.21.2.117:0/2494113780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c07f310 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f687443a700 1 -- 172.21.2.117:0/2494113780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c1a2df0 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.206 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f686400c830 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.207 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f686400c4d0 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.207 7f685effd700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 0x7f685401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.207 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f6864032920 con 0x7f686c0a98f0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.207 7f6872256700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 0x7f685401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.208 7f6872256700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 0x7f685401ac10 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.236 7f687443a700 1 -- 172.21.2.117:0/2494113780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c0ede60 con 0x7f686c0a98f0 2021-02-10T11:45:22.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.239 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f686400c500 con 0x7f686c0a98f0 2021-02-10T11:45:22.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.366 7f687443a700 1 -- 172.21.2.117:0/2494113780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f686c0ede60 con 0x7f686c0a98f0 2021-02-10T11:45:22.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.367 7f685effd700 1 -- 172.21.2.117:0/2494113780 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v79) v1 ==== 92+0+10 (secure 0 0 0) 0x7f686401f840 con 0x7f686c0a98f0 2021-02-10T11:45:22.361 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:22.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.368 7f687443a700 1 -- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 msgr2=0x7f685401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.368 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 0x7f685401ac10 crc :-1 s=READY pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.368 7f687443a700 1 -- 172.21.2.117:0/2494113780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 msgr2=0x7f686c07ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.368 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 rx=0x7f6864006d20 tx=0x7f686400bcc0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 -- 172.21.2.117:0/2494113780 shutdown_connections 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6854018730 0x7f685401ac10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686c0a98f0 0x7f686c07ecf0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686c09f5f0 0x7f686c0a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 --2- 172.21.2.117:0/2494113780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f686c0a8eb0 0x7f686c0a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 -- 172.21.2.117:0/2494113780 >> 172.21.2.117:0/2494113780 conn(0x7f686c0a71e0 msgr2=0x7f686c0a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 -- 172.21.2.117:0/2494113780 shutdown_connections 2021-02-10T11:45:22.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.369 7f687443a700 1 -- 172.21.2.117:0/2494113780 wait complete. 2021-02-10T11:45:22.371 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:45:22.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.787 7f31d7dca700 1 Processor -- start 2021-02-10T11:45:22.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.787 7f31d7dca700 1 -- start start 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d7dca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f31d00d18e0 0x7f31d00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d7dca700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f31d00eb000 0x7f31d00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d7dca700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00ede60 con 0x7f31d00ee3e0 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d7dca700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d00eba20 con 0x7f31d00d18e0 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d7dca700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d00d1ce0 con 0x7f31d00eb000 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f31d00d18e0 0x7f31d00ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f31d00d18e0 0x7f31d00ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51630/0 (socket says 172.21.2.117:51630) 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 -- 172.21.2.117:0/671932250 learned_addr learned my addr 172.21.2.117:0/671932250 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 -- 172.21.2.117:0/671932250 >> v2:172.21.2.125:3300/0 conn(0x7f31d00eb000 msgr2=0x7f31d00eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 --2- 172.21.2.117:0/671932250 >> v2:172.21.2.125:3300/0 conn(0x7f31d00eb000 0x7f31d00eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 -- 172.21.2.117:0/671932250 >> v1:172.21.2.125:6789/0 conn(0x7f31d00ee3e0 legacy=0x7f31d00ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:22.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 -- 172.21.2.117:0/671932250 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00eefd0 con 0x7f31d00d18e0 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.788 7f31d5be6700 1 --2- 172.21.2.117:0/671932250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00d18e0 0x7f31d00ea9e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7f31cc006700 tx=0x7f31cc009b80).ready entity=mon.0 client_cookie=f25a598f49f61cb6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d4be4700 1 -- 172.21.2.117:0/671932250 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc00c0c0 con 0x7f31d00d18e0 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d4be4700 1 -- 172.21.2.117:0/671932250 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00c400 con 0x7f31d00d18e0 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d4be4700 1 -- 172.21.2.117:0/671932250 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31cc00c920 con 0x7f31d00d18e0 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 -- 172.21.2.117:0/671932250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00d18e0 msgr2=0x7f31d00ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 --2- 172.21.2.117:0/671932250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00d18e0 0x7f31d00ea9e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 rx=0x7f31cc006700 tx=0x7f31cc009b80).stop 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 -- 172.21.2.117:0/671932250 shutdown_connections 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 --2- 172.21.2.117:0/671932250 >> v2:172.21.2.125:3300/0 conn(0x7f31d00eb000 0x7f31d00eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 --2- 172.21.2.117:0/671932250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00d18e0 0x7f31d00ea9e0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.789 7f31d7dca700 1 -- 172.21.2.117:0/671932250 >> 172.21.2.117:0/671932250 conn(0x7f31d00ced60 msgr2=0x7f31d00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.790 7f31d7dca700 1 -- 172.21.2.117:0/671932250 shutdown_connections 2021-02-10T11:45:22.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.790 7f31d7dca700 1 -- 172.21.2.117:0/671932250 wait complete. 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 Processor -- start 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 -- start start 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31d00980c0 0x7f31d019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31d019dd40 0x7f31d019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d00b1f30 con 0x7f31d00984c0 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d019d830 con 0x7f31d019dd40 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.804 7f31d7dca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d01a55e0 con 0x7f31d00980c0 2021-02-10T11:45:22.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51636/0 (socket says 172.21.2.117:51636) 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 -- 172.21.2.117:0/1023183640 learned_addr learned my addr 172.21.2.117:0/1023183640 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 -- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31d00980c0 msgr2=0x7f31d019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31d00980c0 0x7f31d019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 -- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31d019dd40 msgr2=0x7f31d019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31d019dd40 0x7f31d019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 -- 172.21.2.117:0/1023183640 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d019e760 con 0x7f31d00984c0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d53e5700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7f31c400b560 tx=0x7f31c400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c4000c40 con 0x7f31d00984c0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c400bcf0 con 0x7f31d00984c0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d019e760 con 0x7f31d00984c0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.805 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31c40054d0 con 0x7f31d00984c0 2021-02-10T11:45:22.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.806 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d019e760 con 0x7f31d00984c0 2021-02-10T11:45:22.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.806 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31c4006030 con 0x7f31d00984c0 2021-02-10T11:45:22.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.806 7f31c27fc700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 0x7f31b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:22.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.806 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f31c401f550 con 0x7f31d00984c0 2021-02-10T11:45:22.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.806 7f31d5be6700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 0x7f31b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:22.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.807 7f31d5be6700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 0x7f31b801ac10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:22.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.838 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d0087210 con 0x7f31d00984c0 2021-02-10T11:45:22.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.840 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31c4006030 con 0x7f31d00984c0 2021-02-10T11:45:22.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.963 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f31d0087210 con 0x7f31d00984c0 2021-02-10T11:45:22.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.963 7f31c27fc700 1 -- 172.21.2.117:0/1023183640 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+64 (secure 0 0 0) 0x7f31c4000960 con 0x7f31d00984c0 2021-02-10T11:45:22.957 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:22.957 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:45:22.957 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:22.957 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:45:22.958 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:45:22.958 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:22.958 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 msgr2=0x7f31b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 0x7f31b801ac10 crc :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 msgr2=0x7f31d019d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 rx=0x7f31c400b560 tx=0x7f31c400b900).stop 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 shutdown_connections 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31b8018730 0x7f31b801ac10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31d00984c0 0x7f31d019d0a0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:22.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31d019dd40 0x7f31d019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 --2- 172.21.2.117:0/1023183640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31d00980c0 0x7f31d019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 >> 172.21.2.117:0/1023183640 conn(0x7f31d019f610 msgr2=0x7f31d01a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 shutdown_connections 2021-02-10T11:45:22.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:22.965 7f31d7dca700 1 -- 172.21.2.117:0/1023183640 wait complete. 2021-02-10T11:45:22.967 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:45:22.967 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:45:23.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.387 7f0f5eb9c700 1 Processor -- start 2021-02-10T11:45:23.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.387 7f0f5eb9c700 1 -- start start 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.388 7f0f5eb9c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0f580d18f0 0x7f0f580eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.388 7f0f5eb9c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.388 7f0f5eb9c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f580eba60 con 0x7f0f580ee3e0 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.388 7f0f5eb9c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580d1cf0 con 0x7f0f580d18f0 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.388 7f0f5eb9c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580eefd0 con 0x7f0f580eb040 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:49324/0 (socket says 172.21.2.117:49324) 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 -- 172.21.2.117:0/2277466562 learned_addr learned my addr 172.21.2.117:0/2277466562 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f5c9b8700 1 --2- 172.21.2.117:0/2277466562 >> v2:172.21.2.125:3300/0 conn(0x7f0f580d18f0 0x7f0f580eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 -- 172.21.2.117:0/2277466562 >> v2:172.21.2.125:3300/0 conn(0x7f0f580d18f0 msgr2=0x7f0f580eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 --2- 172.21.2.117:0/2277466562 >> v2:172.21.2.125:3300/0 conn(0x7f0f580d18f0 0x7f0f580eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 -- 172.21.2.117:0/2277466562 >> v1:172.21.2.125:6789/0 conn(0x7f0f580ee3e0 legacy=0x7f0f580ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.389 7f0f57fff700 1 -- 172.21.2.117:0/2277466562 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580ef1f0 con 0x7f0f580eb040 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f57fff700 1 --2- 172.21.2.117:0/2277466562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0f48006700 tx=0x7f0f48009b80).ready entity=mon.2 client_cookie=12c197b4fc97ce36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f577fe700 1 -- 172.21.2.117:0/2277466562 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4800c0c0 con 0x7f0f580eb040 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f577fe700 1 -- 172.21.2.117:0/2277466562 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4800c440 con 0x7f0f580eb040 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f577fe700 1 -- 172.21.2.117:0/2277466562 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4800c960 con 0x7f0f580eb040 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f5eb9c700 1 -- 172.21.2.117:0/2277466562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 msgr2=0x7f0f580eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.390 7f0f5eb9c700 1 --2- 172.21.2.117:0/2277466562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f0f48006700 tx=0x7f0f48009b80).stop 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 -- 172.21.2.117:0/2277466562 shutdown_connections 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 --2- 172.21.2.117:0/2277466562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580eb040 0x7f0f580eb440 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 --2- 172.21.2.117:0/2277466562 >> v2:172.21.2.125:3300/0 conn(0x7f0f580d18f0 0x7f0f580eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 -- 172.21.2.117:0/2277466562 >> 172.21.2.117:0/2277466562 conn(0x7f0f580cf060 msgr2=0x7f0f580cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 -- 172.21.2.117:0/2277466562 shutdown_connections 2021-02-10T11:45:23.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.391 7f0f5eb9c700 1 -- 172.21.2.117:0/2277466562 wait complete. 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 Processor -- start 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 -- start start 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f5807c7c0 0x7f0f581a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580a13a0 0x7f0f580a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5807cca0 con 0x7f0f581a3930 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5818fb70 con 0x7f0f5807c7c0 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.406 7f0f5eb9c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580dc6d0 con 0x7f0f580a13a0 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51652/0 (socket says 172.21.2.117:51652) 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 -- 172.21.2.117:0/3878269904 learned_addr learned my addr 172.21.2.117:0/3878269904 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 -- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580a13a0 msgr2=0x7f0f580a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580a13a0 0x7f0f580a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 -- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f5807c7c0 msgr2=0x7f0f581a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f5807c7c0 0x7f0f581a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 -- 172.21.2.117:0/3878269904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5807d220 con 0x7f0f581a3930 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.407 7f0f57fff700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f0f4800b5e0 tx=0x7f0f4800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f4800bd70 con 0x7f0f581a3930 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f48002030 con 0x7f0f581a3930 2021-02-10T11:45:23.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0f48002550 con 0x7f0f581a3930 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5807d220 con 0x7f0f581a3930 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5807d220 con 0x7f0f581a3930 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0f48002210 con 0x7f0f581a3930 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 0x7f0f3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.408 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f0f48032340 con 0x7f0f581a3930 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.409 7f0f5c9b8700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 0x7f0f3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:23.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.409 7f0f5c9b8700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 0x7f0f3c01abc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:23.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.437 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f580daff0 con 0x7f0f581a3930 2021-02-10T11:45:23.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.440 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0f48030cd0 con 0x7f0f581a3930 2021-02-10T11:45:23.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.570 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0f580daff0 con 0x7f0f581a3930 2021-02-10T11:45:23.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:23.990 7f0f557fa700 1 -- 172.21.2.117:0/3878269904 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v80) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0f48002210 con 0x7f0f581a3930 2021-02-10T11:45:24.030 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:45:24.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.038 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 msgr2=0x7f0f3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.038 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 0x7f0f3c01abc0 crc :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.038 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 msgr2=0x7f0f580a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.038 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 rx=0x7f0f4800b5e0 tx=0x7f0f4800b980).stop 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 shutdown_connections 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0f3c0186c0 0x7f0f3c01abc0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0f581a3930 0x7f0f580a0d80 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0f580a13a0 0x7f0f580a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 --2- 172.21.2.117:0/3878269904 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0f5807c7c0 0x7f0f581a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 >> 172.21.2.117:0/3878269904 conn(0x7f0f58087090 msgr2=0x7f0f5808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 shutdown_connections 2021-02-10T11:45:24.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.039 7f0f5eb9c700 1 -- 172.21.2.117:0/3878269904 wait complete. 2021-02-10T11:45:24.041 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.460 7fc786ebe700 1 Processor -- start 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.460 7fc786ebe700 1 -- start start 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc786ebe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc7800ef3f0 0x7fc7800eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc786ebe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7800eb830 0x7fc7801b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc786ebe700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7800ef7f0 con 0x7fc7800d18f0 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc786ebe700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7800eded0 con 0x7fc7800ef3f0 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc786ebe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7800d1cf0 con 0x7fc7800eb830 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc7800ef3f0 0x7fc7800eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc784cda700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc7800d18f0 0x7fc7800eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51910/0 (socket says 172.21.2.117:51910) 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc784cda700 1 -- 172.21.2.117:0/4018585233 learned_addr learned my addr 172.21.2.117:0/4018585233 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:24.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77ffff700 1 -- 172.21.2.117:0/4018585233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7800eb830 msgr2=0x7fc7801b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77ffff700 1 --2- 172.21.2.117:0/4018585233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7800eb830 0x7fc7801b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77ffff700 1 -- 172.21.2.117:0/4018585233 >> v1:172.21.2.117:6789/0 conn(0x7fc7800d18f0 legacy=0x7fc7800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77f7fe700 1 -- 172.21.2.117:0/4018585233 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237363695 0 0) 0x7fc7680009b0 con 0x7fc7800d18f0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.461 7fc77ffff700 1 -- 172.21.2.117:0/4018585233 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7801b8fd0 con 0x7fc7800ef3f0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc77ffff700 1 --2- 172.21.2.117:0/4018585233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7800ef3f0 0x7fc7800eb210 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7fc770006650 tx=0x7fc770009a00).ready entity=mon.0 client_cookie=b6ad3880f1ff20dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc77f7fe700 1 -- 172.21.2.117:0/4018585233 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7700009b0 con 0x7fc7800ef3f0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc77f7fe700 1 -- 172.21.2.117:0/4018585233 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77000c0c0 con 0x7fc7800ef3f0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc77f7fe700 1 -- 172.21.2.117:0/4018585233 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc77000c5a0 con 0x7fc7800ef3f0 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc786ebe700 1 -- 172.21.2.117:0/4018585233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7800ef3f0 msgr2=0x7fc7800eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc786ebe700 1 --2- 172.21.2.117:0/4018585233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7800ef3f0 0x7fc7800eb210 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7fc770006650 tx=0x7fc770009a00).stop 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc786ebe700 1 -- 172.21.2.117:0/4018585233 shutdown_connections 2021-02-10T11:45:24.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.462 7fc786ebe700 1 --2- 172.21.2.117:0/4018585233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7800eb830 0x7fc7801b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.463 7fc786ebe700 1 --2- 172.21.2.117:0/4018585233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7800ef3f0 0x7fc7800eb210 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:24.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.463 7fc786ebe700 1 -- 172.21.2.117:0/4018585233 >> 172.21.2.117:0/4018585233 conn(0x7fc7800cf060 msgr2=0x7fc7800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:24.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.463 7fc786ebe700 1 -- 172.21.2.117:0/4018585233 shutdown_connections 2021-02-10T11:45:24.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.463 7fc786ebe700 1 -- 172.21.2.117:0/4018585233 wait complete. 2021-02-10T11:45:24.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.477 7fc786ebe700 1 Processor -- start 2021-02-10T11:45:24.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.477 7fc786ebe700 1 -- start start 2021-02-10T11:45:24.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a4240 0x7fc7801a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009f790 0x7fc7800b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78009fb90 con 0x7fc78009ed70 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7800dc640 con 0x7fc7801a4240 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc786ebe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc780097270 con 0x7fc78009f790 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51666/0 (socket says 172.21.2.117:51666) 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 -- 172.21.2.117:0/1416227928 learned_addr learned my addr 172.21.2.117:0/1416227928 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 -- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009f790 msgr2=0x7fc7800b8160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009f790 0x7fc7800b8160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 -- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a4240 msgr2=0x7fc7801a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a4240 0x7fc7801a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 -- 172.21.2.117:0/1416227928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7800b8780 con 0x7fc78009ed70 2021-02-10T11:45:24.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.478 7fc77ffff700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fc77000b640 tx=0x7fc77000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc77000bdd0 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc770002030 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc770002550 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7800b8780 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7800b8d10 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.479 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc770006a10 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.480 7fc77d7fa700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.480 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5454+0+0 (secure 0 0 0) 0x7fc770032530 con 0x7fc78009ed70 2021-02-10T11:45:24.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.480 7fc784cda700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:24.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.481 7fc784cda700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.511 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7800daff0 con 0x7fc78009ed70 2021-02-10T11:45:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.513 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc770000940 con 0x7fc78009ed70 2021-02-10T11:45:24.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.641 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fc7800ede60 con 0x7fc78009ed70 2021-02-10T11:45:24.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:24.994 7fc77d7fa700 1 -- 172.21.2.117:0/1416227928 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v81) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc770006a10 con 0x7fc78009ed70 2021-02-10T11:45:25.000 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:45:25.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 msgr2=0x7fc76401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 crc :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 msgr2=0x7fc78009f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 rx=0x7fc77000b640 tx=0x7fc77000b9e0).stop 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 shutdown_connections 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7640186c0 0x7fc76401abc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc78009ed70 0x7fc78009f170 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc78009f790 0x7fc7800b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 --2- 172.21.2.117:0/1416227928 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7801a4240 0x7fc7801a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 >> 172.21.2.117:0/1416227928 conn(0x7fc780087090 msgr2=0x7fc78008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 shutdown_connections 2021-02-10T11:45:25.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.008 7fc786ebe700 1 -- 172.21.2.117:0/1416227928 wait complete. 2021-02-10T11:45:25.011 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:45:25.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 Processor -- start 2021-02-10T11:45:25.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 -- start start 2021-02-10T11:45:25.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7feb900e4e40 0x7feb901b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feb90085450 0x7feb90086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90085930 con 0x7feb900e5240 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb900e5cc0 con 0x7feb90085450 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.430 7feb9820a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb90084840 con 0x7feb900e4e40 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feb90085450 0x7feb90086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feb90085450 0x7feb90086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51676/0 (socket says 172.21.2.117:51676) 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 -- 172.21.2.117:0/2722820764 learned_addr learned my addr 172.21.2.117:0/2722820764 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 -- 172.21.2.117:0/2722820764 >> v2:172.21.2.125:3300/0 conn(0x7feb900e4e40 msgr2=0x7feb901b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 --2- 172.21.2.117:0/2722820764 >> v2:172.21.2.125:3300/0 conn(0x7feb900e4e40 0x7feb901b01f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 -- 172.21.2.117:0/2722820764 >> v1:172.21.2.125:6789/0 conn(0x7feb900e5240 legacy=0x7feb90084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 -- 172.21.2.117:0/2722820764 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb901b0810 con 0x7feb90085450 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb96827700 1 --2- 172.21.2.117:0/2722820764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90085450 0x7feb90086ec0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7feb88006f00 tx=0x7feb88009f10).ready entity=mon.0 client_cookie=ae05da91cc8ca3e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb95024700 1 -- 172.21.2.117:0/2722820764 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb8800b3b0 con 0x7feb90085450 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb95024700 1 -- 172.21.2.117:0/2722820764 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8800b730 con 0x7feb90085450 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb95024700 1 -- 172.21.2.117:0/2722820764 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb8800bc10 con 0x7feb90085450 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb9820a700 1 -- 172.21.2.117:0/2722820764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90085450 msgr2=0x7feb90086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.431 7feb9820a700 1 --2- 172.21.2.117:0/2722820764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90085450 0x7feb90086ec0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 rx=0x7feb88006f00 tx=0x7feb88009f10).stop 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 -- 172.21.2.117:0/2722820764 shutdown_connections 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 --2- 172.21.2.117:0/2722820764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90085450 0x7feb90086ec0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 --2- 172.21.2.117:0/2722820764 >> v2:172.21.2.125:3300/0 conn(0x7feb900e4e40 0x7feb901b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 -- 172.21.2.117:0/2722820764 >> 172.21.2.117:0/2722820764 conn(0x7feb900ce4e0 msgr2=0x7feb900e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 -- 172.21.2.117:0/2722820764 shutdown_connections 2021-02-10T11:45:25.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.432 7feb9820a700 1 -- 172.21.2.117:0/2722820764 wait complete. 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.446 7feb9820a700 1 Processor -- start 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 -- start start 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb900a00e0 0x7feb901ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb901ae8d0 0x7feb9009e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9009f540 con 0x7feb90093070 2021-02-10T11:45:25.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90093470 con 0x7feb901ae8d0 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb9820a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9009ecd0 con 0x7feb900a00e0 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51680/0 (socket says 172.21.2.117:51680) 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 -- 172.21.2.117:0/1168944873 learned_addr learned my addr 172.21.2.117:0/1168944873 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 -- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb900a00e0 msgr2=0x7feb901ae2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb900a00e0 0x7feb901ae2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 -- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb901ae8d0 msgr2=0x7feb9009e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb901ae8d0 0x7feb9009e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.447 7feb96026700 1 -- 172.21.2.117:0/1168944873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9009eef0 con 0x7feb90093070 2021-02-10T11:45:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb96026700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7feb8c00b5c0 tx=0x7feb8c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb8c0009b0 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c00bd50 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb8c0062d0 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb9820a700 1 -- 172.21.2.117:0/1168944873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9009eef0 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.448 7feb9820a700 1 -- 172.21.2.117:0/1168944873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb901b20a0 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.449 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb8c006030 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.449 7feb86ffd700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 0x7feb7801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.449 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5804+0+0 (secure 0 0 0) 0x7feb8c032650 con 0x7feb90093070 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.449 7feb95825700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 0x7feb7801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.450 7feb95825700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 0x7feb7801abc0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:25.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.481 7feb9820a700 1 -- 172.21.2.117:0/1168944873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9010c2f0 con 0x7feb90093070 2021-02-10T11:45:25.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.483 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb8c0009b0 con 0x7feb90093070 2021-02-10T11:45:25.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.611 7feb9820a700 1 -- 172.21.2.117:0/1168944873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7feb90085930 con 0x7feb90093070 2021-02-10T11:45:25.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.614 7feb86ffd700 1 -- 172.21.2.117:0/1168944873 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 16 and data pool 17 v58) v1 ==== 189+0+0 (secure 0 0 0) 0x7feb8c006030 con 0x7feb90093070 2021-02-10T11:45:25.612 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 16 and data pool 17 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 msgr2=0x7feb7801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 0x7feb7801abc0 crc :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 msgr2=0x7feb9009fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 rx=0x7feb8c00b5c0 tx=0x7feb8c00b960).stop 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 shutdown_connections 2021-02-10T11:45:25.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb901ae8d0 0x7feb9009e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb900a00e0 0x7feb901ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb780186c0 0x7feb7801abc0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 --2- 172.21.2.117:0/1168944873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb90093070 0x7feb9009fac0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 >> 172.21.2.117:0/1168944873 conn(0x7feb901ad6b0 msgr2=0x7feb90190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 shutdown_connections 2021-02-10T11:45:25.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:25.620 7feb9820a700 1 -- 172.21.2.117:0/1168944873 wait complete. 2021-02-10T11:45:25.623 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:26.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 Processor -- start 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 -- start start 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f51cc0ef3f0 0x7f51cc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0ede60 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0d1ce0 con 0x7f51cc0d18e0 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.039 7f51d3dd2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51cc0ef7f0 con 0x7f51cc0ef3f0 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d23ef700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f51cc0eb7f0 0x7f51cc1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51938/0 (socket says 172.21.2.117:51938) 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d23ef700 1 -- 172.21.2.117:0/447125934 learned_addr learned my addr 172.21.2.117:0/447125934 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662856480 0 0) 0x7f51c4001500 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b8002fe0 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2545137511 0 0) 0x7f51c4001460 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 >> v2:172.21.2.125:3300/0 conn(0x7f51cc0ef3f0 msgr2=0x7f51cc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 --2- 172.21.2.117:0/447125934 >> v2:172.21.2.125:3300/0 conn(0x7f51cc0ef3f0 0x7f51cc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 >> v1:172.21.2.125:6789/0 conn(0x7f51cc0d18e0 legacy=0x7f51cc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc1bb180 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f51c4001980 con 0x7f51cc0eb7f0 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 >> v1:172.21.2.117:6789/0 conn(0x7f51cc0eb7f0 legacy=0x7f51cc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51b8003a40 0x7f51b8005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 0x7f51b8008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51b8009590 0x7f51b800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c300 con 0x7f51b8009590 2021-02-10T11:45:26.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c520 con 0x7f51b8003a40 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.040 7f51d0bec700 1 -- 172.21.2.117:0/447125934 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800c740 con 0x7f51b8006770 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 0x7f51b8008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d23ef700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51b8009590 0x7f51b800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d1bee700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51b8003a40 0x7f51b8005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 -- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51b8003a40 msgr2=0x7f51b8005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51b8003a40 0x7f51b8005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 -- 172.21.2.117:0/447125934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51b8009590 msgr2=0x7f51b800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51b8009590 0x7f51b800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 -- 172.21.2.117:0/447125934 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b800c9e0 con 0x7f51b8006770 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d1bee700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51b8003a40 0x7f51b8005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:26.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d23ef700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51b8009590 0x7f51b800bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d23ef700 1 -- 172.21.2.117:0/447125934 reap_dead start 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d13ed700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 0x7f51b8008c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f51bc006c10 tx=0x7f51bc009c50).ready entity=mon.2 client_cookie=35de4ebf548f7c69 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51bc0009b0 con 0x7f51b8006770 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc006850 con 0x7f51b8006770 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d0bec700 1 -- 172.21.2.117:0/447125934 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51bc002030 con 0x7f51b8006770 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d3dd2700 1 -- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 msgr2=0x7f51b8008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.041 7f51d3dd2700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 0x7f51b8008c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f51bc006c10 tx=0x7f51bc009c50).stop 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.042 7f51d3dd2700 1 -- 172.21.2.117:0/447125934 shutdown_connections 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.042 7f51d3dd2700 1 --2- 172.21.2.117:0/447125934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51b8006770 0x7f51b8008c90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.042 7f51d3dd2700 1 -- 172.21.2.117:0/447125934 >> 172.21.2.117:0/447125934 conn(0x7f51cc0ced60 msgr2=0x7f51cc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.042 7f51d3dd2700 1 -- 172.21.2.117:0/447125934 shutdown_connections 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.042 7f51d3dd2700 1 -- 172.21.2.117:0/447125934 wait complete. 2021-02-10T11:45:26.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 Processor -- start 2021-02-10T11:45:26.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 -- start start 2021-02-10T11:45:26.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 0x7f51cc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 0x7f51cc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0b61a0 con 0x7f51cc0b59a0 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc08c8b0 con 0x7f51cc1a3480 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d3dd2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0ba1c0 con 0x7f51cc0b5da0 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d1bee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d1bee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51696/0 (socket says 172.21.2.117:51696) 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.057 7f51d1bee700 1 -- 172.21.2.117:0/3612724546 learned_addr learned my addr 172.21.2.117:0/3612724546 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d13ed700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 0x7f51cc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 -- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 msgr2=0x7f51cc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 0x7f51cc1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 -- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 msgr2=0x7f51cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d23ef700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 0x7f51cc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 0x7f51cc0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 -- 172.21.2.117:0/3612724546 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc1a3880 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d13ed700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 0x7f51cc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d23ef700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 0x7f51cc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d1bee700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f51c8006b40 tx=0x7f51c800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51c8006c20 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8000c60 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51c8002350 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc1a3880 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.058 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc1a3880 con 0x7f51cc0b59a0 2021-02-10T11:45:26.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.059 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51c8007030 con 0x7f51cc0b59a0 2021-02-10T11:45:26.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.059 7f51c27fc700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 0x7f51b401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.059 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f51c80327e0 con 0x7f51cc0b59a0 2021-02-10T11:45:26.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.059 7f51d13ed700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 0x7f51b401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.060 7f51d13ed700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 0x7f51b401ac10 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.089 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc0ede60 con 0x7f51cc0b59a0 2021-02-10T11:45:26.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.092 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51c80067d0 con 0x7f51cc0b59a0 2021-02-10T11:45:26.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.218 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51cc0ede60 con 0x7f51cc0b59a0 2021-02-10T11:45:26.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.218 7f51c27fc700 1 -- 172.21.2.117:0/3612724546 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f51c8000c10 con 0x7f51cc0b59a0 2021-02-10T11:45:26.213 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:26.213 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":82,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:25.613173","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:23.589151","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:24.659131","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:26.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 msgr2=0x7f51b401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 0x7f51b401ac10 crc :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 msgr2=0x7f51cc08c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 rx=0x7f51c8006b40 tx=0x7f51c800b990).stop 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 shutdown_connections 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51cc1a3480 0x7f51cc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51cc0b5da0 0x7f51cc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b4018730 0x7f51b401ac10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 --2- 172.21.2.117:0/3612724546 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51cc0b59a0 0x7f51cc08c290 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 >> 172.21.2.117:0/3612724546 conn(0x7f51cc0bb5e0 msgr2=0x7f51cc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 shutdown_connections 2021-02-10T11:45:26.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.220 7f51d3dd2700 1 -- 172.21.2.117:0/3612724546 wait complete. 2021-02-10T11:45:26.223 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:26.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 Processor -- start 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 -- start start 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 0x7f20f80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 0x7f20f81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f80ede60 con 0x7f20f80d18e0 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f80ef7f0 con 0x7f20f80eb7f0 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.641 7f210072f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80d1ce0 con 0x7f20f80ef3f0 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fed4c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 0x7f20f81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 0x7f20f80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fed4c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 0x7f20f81b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:56092/0 (socket says 172.21.2.117:56092) 2021-02-10T11:45:26.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fed4c700 1 -- 172.21.2.117:0/3077679853 learned_addr learned my addr 172.21.2.117:0/3077679853 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fe54b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f20f80d18e0 0x7f20f80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:44880/0 (socket says 172.21.2.117:44880) 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 -- 172.21.2.117:0/3077679853 >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 msgr2=0x7f20f81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 --2- 172.21.2.117:0/3077679853 >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 0x7f20f81b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 -- 172.21.2.117:0/3077679853 >> v1:172.21.2.125:6789/0 conn(0x7f20f80d18e0 legacy=0x7f20f80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 -- 172.21.2.117:0/3077679853 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f81b8f90 con 0x7f20f80ef3f0 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fdd4a700 1 --2- 172.21.2.117:0/3077679853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 0x7f20f80eb1d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f20ec006650 tx=0x7f20ec009a80).ready entity=mon.2 client_cookie=2547fe239ddc4a4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.642 7f20fd549700 1 -- 172.21.2.117:0/3077679853 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20ec00c0c0 con 0x7f20f80ef3f0 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f20fd549700 1 -- 172.21.2.117:0/3077679853 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20ec00c340 con 0x7f20f80ef3f0 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f20fd549700 1 -- 172.21.2.117:0/3077679853 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20ec00c860 con 0x7f20f80ef3f0 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 -- 172.21.2.117:0/3077679853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 msgr2=0x7f20f80eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 --2- 172.21.2.117:0/3077679853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 0x7f20f80eb1d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f20ec006650 tx=0x7f20ec009a80).stop 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 -- 172.21.2.117:0/3077679853 shutdown_connections 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 --2- 172.21.2.117:0/3077679853 >> v2:172.21.2.125:3300/0 conn(0x7f20f80eb7f0 0x7f20f81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 --2- 172.21.2.117:0/3077679853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f80ef3f0 0x7f20f80eb1d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.643 7f210072f700 1 -- 172.21.2.117:0/3077679853 >> 172.21.2.117:0/3077679853 conn(0x7f20f80ced60 msgr2=0x7f20f80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.644 7f210072f700 1 -- 172.21.2.117:0/3077679853 shutdown_connections 2021-02-10T11:45:26.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.644 7f210072f700 1 -- 172.21.2.117:0/3077679853 wait complete. 2021-02-10T11:45:26.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.658 7f210072f700 1 Processor -- start 2021-02-10T11:45:26.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.658 7f210072f700 1 -- start start 2021-02-10T11:45:26.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f81a33d0 0x7f20f81a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 0x7f20f80b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80c0670 con 0x7f20f80b8d10 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80deb20 con 0x7f20f80b82f0 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ded40 con 0x7f20f81a33d0 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51712/0 (socket says 172.21.2.117:51712) 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 -- 172.21.2.117:0/4068160653 learned_addr learned my addr 172.21.2.117:0/4068160653 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fdd4a700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 0x7f20f80b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fe54b700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f81a33d0 0x7f20f81a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 -- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f81a33d0 msgr2=0x7f20f81a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f81a33d0 0x7f20f81a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 -- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 msgr2=0x7f20f80b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 0x7f20f80b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 -- 172.21.2.117:0/4068160653 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f80def60 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fdd4a700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 0x7f20f80b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20fed4c700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f20f0009940 tx=0x7f20f000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20f0006950 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f0003030 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f210072f700 1 -- 172.21.2.117:0/4068160653 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f80def60 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.659 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f20f0001470 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.660 7f210072f700 1 -- 172.21.2.117:0/4068160653 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f80df4f0 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.660 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f20f00011f0 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.660 7f20eaffd700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 0x7f20e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.660 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f20f00328e0 con 0x7f20f80b8d10 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.661 7f20fe54b700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 0x7f20e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:26.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.661 7f20fe54b700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 0x7f20e001abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:26.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.691 7f210072f700 1 -- 172.21.2.117:0/4068160653 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f8087290 con 0x7f20f80b8d10 2021-02-10T11:45:26.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.694 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f20f0025ee0 con 0x7f20f80b8d10 2021-02-10T11:45:26.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.820 7f210072f700 1 -- 172.21.2.117:0/4068160653 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f80ede60 con 0x7f20f80b8d10 2021-02-10T11:45:26.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.821 7f20eaffd700 1 -- 172.21.2.117:0/4068160653 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f20f0025ee0 con 0x7f20f80b8d10 2021-02-10T11:45:26.815 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:26.815 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":83,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:26.614628","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:23.589151","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:24.659131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:26.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.822 7f210072f700 1 -- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 msgr2=0x7f20e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.822 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 0x7f20e001abc0 crc :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.822 7f210072f700 1 -- 172.21.2.117:0/4068160653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 msgr2=0x7f20f80b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:26.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.822 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 rx=0x7f20f0009940 tx=0x7f20f000bcf0).stop 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 -- 172.21.2.117:0/4068160653 shutdown_connections 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20e00186c0 0x7f20e001abc0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20f80b8d10 0x7f20f80b9110 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20f80b82f0 0x7f20f80b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 --2- 172.21.2.117:0/4068160653 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20f81a33d0 0x7f20f81a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 -- 172.21.2.117:0/4068160653 >> 172.21.2.117:0/4068160653 conn(0x7f20f80874c0 msgr2=0x7f20f8096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 -- 172.21.2.117:0/4068160653 shutdown_connections 2021-02-10T11:45:26.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:26.823 7f210072f700 1 -- 172.21.2.117:0/4068160653 wait complete. 2021-02-10T11:45:26.825 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.244 7f9db53de700 1 Processor -- start 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.244 7f9db53de700 1 -- start start 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9db00d18f0 0x7f9db00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 0x7f9db00eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 0x7f9db00ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db00edee0 con 0x7f9db00d18f0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db00eba60 con 0x7f9db00ee3e0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9db53de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00d1cf0 con 0x7f9db00eb040 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9daf07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9db00d18f0 0x7f9db00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9daf07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9db00d18f0 0x7f9db00eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51716/0 (socket says 172.21.2.117:51716) 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9daf07d700 1 -- 172.21.2.117:0/1482765314 learned_addr learned my addr 172.21.2.117:0/1482765314 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.245 7f9dae87c700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 0x7f9db00eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 -- 172.21.2.117:0/1482765314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 msgr2=0x7f9db00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 0x7f9db00eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf87e700 1 --2- 172.21.2.117:0/1482765314 >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 0x7f9db00ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 -- 172.21.2.117:0/1482765314 >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 msgr2=0x7f9db00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 --2- 172.21.2.117:0/1482765314 >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 0x7f9db00ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 -- 172.21.2.117:0/1482765314 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db00eee00 con 0x7f9db00d18f0 2021-02-10T11:45:27.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9dae87c700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 0x7f9db00eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf87e700 1 --2- 172.21.2.117:0/1482765314 >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 0x7f9db00ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9daf07d700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db00d18f0 0x7f9db00eaa20 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7f9d98006700 tx=0x7f9d98009b80).ready entity=mon.0 client_cookie=d5a6f3f3311b8a88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9dae07b700 1 -- 172.21.2.117:0/1482765314 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d9800c0c0 con 0x7f9db00d18f0 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9dae07b700 1 -- 172.21.2.117:0/1482765314 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9800c340 con 0x7f9db00d18f0 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9dae07b700 1 -- 172.21.2.117:0/1482765314 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d9800c860 con 0x7f9db00d18f0 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9db53de700 1 -- 172.21.2.117:0/1482765314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db00d18f0 msgr2=0x7f9db00eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.246 7f9db53de700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db00d18f0 0x7f9db00eaa20 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 rx=0x7f9d98006700 tx=0x7f9d98009b80).stop 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 -- 172.21.2.117:0/1482765314 shutdown_connections 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 --2- 172.21.2.117:0/1482765314 >> v2:172.21.2.125:3300/0 conn(0x7f9db00ee3e0 0x7f9db00ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00eb040 0x7f9db00eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 --2- 172.21.2.117:0/1482765314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db00d18f0 0x7f9db00eaa20 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 -- 172.21.2.117:0/1482765314 >> 172.21.2.117:0/1482765314 conn(0x7f9db00cf060 msgr2=0x7f9db00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:27.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 -- 172.21.2.117:0/1482765314 shutdown_connections 2021-02-10T11:45:27.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.247 7f9db53de700 1 -- 172.21.2.117:0/1482765314 wait complete. 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.261 7f9db53de700 1 Processor -- start 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 -- start start 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9db00a8eb0 0x7f9db00a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00a98f0 0x7f9db007ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db00a9cf0 con 0x7f9db009f5f0 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db009f9f0 con 0x7f9db00a8eb0 2021-02-10T11:45:27.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9db53de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db009fc10 con 0x7f9db00a98f0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51720/0 (socket says 172.21.2.117:51720) 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 -- 172.21.2.117:0/3570912803 learned_addr learned my addr 172.21.2.117:0/3570912803 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 -- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00a98f0 msgr2=0x7f9db007ecf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00a98f0 0x7f9db007ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 -- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9db00a8eb0 msgr2=0x7f9db00a8600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9db00a8eb0 0x7f9db00a8600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.262 7f9dae87c700 1 -- 172.21.2.117:0/3570912803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db007f310 con 0x7f9db009f5f0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9dae87c700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 rx=0x7f9da0006f20 tx=0x7f9da000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da0006bb0 con 0x7f9db009f5f0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da000bd80 con 0x7f9db009f5f0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9da0002220 con 0x7f9db009f5f0 2021-02-10T11:45:27.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9db53de700 1 -- 172.21.2.117:0/3570912803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db007f310 con 0x7f9db009f5f0 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9db53de700 1 -- 172.21.2.117:0/3570912803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db007f310 con 0x7f9db009f5f0 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9da000a000 con 0x7f9db009f5f0 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 0x7f9d9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.263 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9da00326c0 con 0x7f9db009f5f0 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.264 7f9daf07d700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 0x7f9d9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:27.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.265 7f9daf07d700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 0x7f9d9401abc0 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:27.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.295 7f9db53de700 1 -- 172.21.2.117:0/3570912803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db009fc10 con 0x7f9db009f5f0 2021-02-10T11:45:27.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.298 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9da0025fa0 con 0x7f9db009f5f0 2021-02-10T11:45:27.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.435 7f9db53de700 1 -- 172.21.2.117:0/3570912803 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f9db009fc10 con 0x7f9db009f5f0 2021-02-10T11:45:27.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.625 7f9da7fff700 1 -- 172.21.2.117:0/3570912803 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9da0025fa0 con 0x7f9db009f5f0 2021-02-10T11:45:27.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 -- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 msgr2=0x7f9d9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 0x7f9d9401abc0 crc :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 -- 172.21.2.117:0/3570912803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 msgr2=0x7f9db00a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:27.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 rx=0x7f9da0006f20 tx=0x7f9da000b990).stop 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 -- 172.21.2.117:0/3570912803 shutdown_connections 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d940186c0 0x7f9d9401abc0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9db009f5f0 0x7f9db00a92d0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9db00a98f0 0x7f9db007ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 --2- 172.21.2.117:0/3570912803 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9db00a8eb0 0x7f9db00a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.653 7f9db53de700 1 -- 172.21.2.117:0/3570912803 >> 172.21.2.117:0/3570912803 conn(0x7f9db00a71e0 msgr2=0x7f9db00a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.654 7f9db53de700 1 -- 172.21.2.117:0/3570912803 shutdown_connections 2021-02-10T11:45:27.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:27.654 7f9db53de700 1 -- 172.21.2.117:0/3570912803 wait complete. 2021-02-10T11:45:27.656 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 Processor -- start 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 -- start start 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa4940d18e0 0x7fa4940ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4940eb000 0x7fa4940eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4940ee3e0 0x7fa4940ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940edee0 con 0x7fa4940eb000 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940eba20 con 0x7fa4940d18e0 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.073 7fa49b785700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940d1ce0 con 0x7fa4940ee3e0 2021-02-10T11:45:28.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4940eb000 0x7fa4940eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4940eb000 0x7fa4940eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51726/0 (socket says 172.21.2.117:51726) 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 -- 172.21.2.117:0/2234333358 learned_addr learned my addr 172.21.2.117:0/2234333358 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 -- 172.21.2.117:0/2234333358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4940ee3e0 msgr2=0x7fa4940ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 --2- 172.21.2.117:0/2234333358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4940ee3e0 0x7fa4940ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 -- 172.21.2.117:0/2234333358 >> v2:172.21.2.125:3300/0 conn(0x7fa4940d18e0 msgr2=0x7fa4940ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 --2- 172.21.2.117:0/2234333358 >> v2:172.21.2.125:3300/0 conn(0x7fa4940d18e0 0x7fa4940ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 -- 172.21.2.117:0/2234333358 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940eee00 con 0x7fa4940eb000 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.074 7fa498da0700 1 --2- 172.21.2.117:0/2234333358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940eb000 0x7fa4940eb400 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7fa488006a80 tx=0x7fa488009f00).ready entity=mon.0 client_cookie=18bd450a010a6deb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa487fff700 1 -- 172.21.2.117:0/2234333358 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa48800b5c0 con 0x7fa4940eb000 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa487fff700 1 -- 172.21.2.117:0/2234333358 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48800b800 con 0x7fa4940eb000 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa487fff700 1 -- 172.21.2.117:0/2234333358 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa48800bd20 con 0x7fa4940eb000 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 -- 172.21.2.117:0/2234333358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940eb000 msgr2=0x7fa4940eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 --2- 172.21.2.117:0/2234333358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940eb000 0x7fa4940eb400 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 rx=0x7fa488006a80 tx=0x7fa488009f00).stop 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 -- 172.21.2.117:0/2234333358 shutdown_connections 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 --2- 172.21.2.117:0/2234333358 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4940ee3e0 0x7fa4940ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 --2- 172.21.2.117:0/2234333358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940eb000 0x7fa4940eb400 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 --2- 172.21.2.117:0/2234333358 >> v2:172.21.2.125:3300/0 conn(0x7fa4940d18e0 0x7fa4940ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.075 7fa49b785700 1 -- 172.21.2.117:0/2234333358 >> 172.21.2.117:0/2234333358 conn(0x7fa4940ced60 msgr2=0x7fa4940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.076 7fa49b785700 1 -- 172.21.2.117:0/2234333358 shutdown_connections 2021-02-10T11:45:28.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.076 7fa49b785700 1 -- 172.21.2.117:0/2234333358 wait complete. 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.090 7fa49b785700 1 Processor -- start 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.090 7fa49b785700 1 -- start start 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa49419c7c0 0x7fa494092f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa49408a280 0x7fa49408a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940a9aa0 con 0x7fa4940e8f90 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49419cbc0 con 0x7fa49408a280 2021-02-10T11:45:28.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa49b785700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49408aca0 con 0x7fa49419c7c0 2021-02-10T11:45:28.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51734/0 (socket says 172.21.2.117:51734) 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 -- 172.21.2.117:0/2902302413 learned_addr learned my addr 172.21.2.117:0/2902302413 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 -- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa49419c7c0 msgr2=0x7fa494092f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa49419c7c0 0x7fa494092f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 -- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa49408a280 msgr2=0x7fa49408a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa49408a280 0x7fa49408a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.091 7fa499da2700 1 -- 172.21.2.117:0/2902302413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940e99b0 con 0x7fa4940e8f90 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa499da2700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7fa48c006f10 tx=0x7fa48c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa48c001060 con 0x7fa4940e8f90 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c00bd80 con 0x7fa4940e8f90 2021-02-10T11:45:28.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa48c0062b0 con 0x7fa4940e8f90 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa49b785700 1 -- 172.21.2.117:0/2902302413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4940e99b0 con 0x7fa4940e8f90 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.092 7fa49b785700 1 -- 172.21.2.117:0/2902302413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa49419b330 con 0x7fa4940e8f90 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.093 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa48c006240 con 0x7fa4940e8f90 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.093 7fa485ffb700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 0x7fa47401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.093 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa48c0326d0 con 0x7fa4940e8f90 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.093 7fa4995a1700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 0x7fa47401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.094 7fa4995a1700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 0x7fa47401abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.124 7fa49b785700 1 -- 172.21.2.117:0/2902302413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4940e8a60 con 0x7fa4940e8f90 2021-02-10T11:45:28.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.126 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa48c006030 con 0x7fa4940e8f90 2021-02-10T11:45:28.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.254 7fa49b785700 1 -- 172.21.2.117:0/2902302413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4940e8a60 con 0x7fa4940e8f90 2021-02-10T11:45:28.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.254 7fa485ffb700 1 -- 172.21.2.117:0/2902302413 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa48c006240 con 0x7fa4940e8f90 2021-02-10T11:45:28.248 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 61 2021-02-10T11:45:28.248 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:28.248 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:28.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 -- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 msgr2=0x7fa47401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 0x7fa47401abc0 crc :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 -- 172.21.2.117:0/2902302413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 msgr2=0x7fa4940e9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7fa48c006f10 tx=0x7fa48c00b990).stop 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 -- 172.21.2.117:0/2902302413 shutdown_connections 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa4740186c0 0x7fa47401abc0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4940e8f90 0x7fa4940e9390 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa49408a280 0x7fa49408a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 --2- 172.21.2.117:0/2902302413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa49419c7c0 0x7fa494092f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 -- 172.21.2.117:0/2902302413 >> 172.21.2.117:0/2902302413 conn(0x7fa49419f610 msgr2=0x7fa494098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.256 7fa49b785700 1 -- 172.21.2.117:0/2902302413 shutdown_connections 2021-02-10T11:45:28.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.257 7fa49b785700 1 -- 172.21.2.117:0/2902302413 wait complete. 2021-02-10T11:45:28.258 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.676 7f9f4792a700 1 Processor -- start 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.676 7f9f4792a700 1 -- start start 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f4792a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f401b6960 0x7f9f401b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f4792a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f400ede60 con 0x7f9f400d18e0 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f4792a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f400eb9c0 con 0x7f9f400ef3f0 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f4792a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f400ef7f0 con 0x7f9f401b6960 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f45f47700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f401b6960 0x7f9f401b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f45f47700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9f401b6960 0x7f9f401b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51740/0 (socket says 172.21.2.117:51740) 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f45f47700 1 -- 172.21.2.117:0/1075201014 learned_addr learned my addr 172.21.2.117:0/1075201014 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.677 7f9f45746700 1 --1- 172.21.2.117:0/1075201014 >> v1:172.21.2.117:6789/0 conn(0x7f9f400d18e0 0x7f9f400ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:51988/0 (socket says 172.21.2.117:51988) 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f45f47700 1 -- 172.21.2.117:0/1075201014 >> v1:172.21.2.125:6789/0 conn(0x7f9f400ef3f0 legacy=0x7f9f400eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f45f47700 1 -- 172.21.2.117:0/1075201014 >> v1:172.21.2.117:6789/0 conn(0x7f9f400d18e0 legacy=0x7f9f400ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f45f47700 1 -- 172.21.2.117:0/1075201014 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400d1d20 con 0x7f9f401b6960 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f45f47700 1 --2- 172.21.2.117:0/1075201014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f401b6960 0x7f9f401b8d70 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f9f38006f00 tx=0x7f9f38009e90).ready entity=mon.0 client_cookie=3c143851b6aafea1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f37fff700 1 -- 172.21.2.117:0/1075201014 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f3800b300 con 0x7f9f401b6960 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f37fff700 1 -- 172.21.2.117:0/1075201014 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3800b580 con 0x7f9f401b6960 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f37fff700 1 -- 172.21.2.117:0/1075201014 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f3800b9d0 con 0x7f9f401b6960 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f4792a700 1 -- 172.21.2.117:0/1075201014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f401b6960 msgr2=0x7f9f401b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.678 7f9f4792a700 1 --2- 172.21.2.117:0/1075201014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f401b6960 0x7f9f401b8d70 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 rx=0x7f9f38006f00 tx=0x7f9f38009e90).stop 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.679 7f9f4792a700 1 -- 172.21.2.117:0/1075201014 shutdown_connections 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.679 7f9f4792a700 1 --2- 172.21.2.117:0/1075201014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f401b6960 0x7f9f401b8d70 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.679 7f9f4792a700 1 -- 172.21.2.117:0/1075201014 >> 172.21.2.117:0/1075201014 conn(0x7f9f400ced60 msgr2=0x7f9f400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.679 7f9f4792a700 1 -- 172.21.2.117:0/1075201014 shutdown_connections 2021-02-10T11:45:28.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.679 7f9f4792a700 1 -- 172.21.2.117:0/1075201014 wait complete. 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.693 7f9f4792a700 1 Processor -- start 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.693 7f9f4792a700 1 -- start start 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f400a3200 0x7f9f400f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f40099780 0x7f9f401c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40099200 con 0x7f9f40099780 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400f2530 con 0x7f9f401c68d0 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f4792a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400f2710 con 0x7f9f400a3200 2021-02-10T11:45:28.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f44f45700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f40099780 0x7f9f401c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:56126/0 (socket says 172.21.2.117:56126) 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 -- 172.21.2.117:0/3076918349 learned_addr learned my addr 172.21.2.117:0/3076918349 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45746700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f400a3200 0x7f9f400f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 -- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f400a3200 msgr2=0x7f9f400f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f400a3200 0x7f9f400f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 -- 172.21.2.117:0/3076918349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f40099780 msgr2=0x7f9f401c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f40099780 0x7f9f401c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 -- 172.21.2.117:0/3076918349 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f40099b80 con 0x7f9f401c68d0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.694 7f9f45f47700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f9f38009a20 tx=0x7f9f3800bd10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f38000f90 con 0x7f9f401c68d0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f38006d90 con 0x7f9f401c68d0 2021-02-10T11:45:28.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9f38005a10 con 0x7f9f401c68d0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f40099b80 con 0x7f9f401c68d0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f40099b80 con 0x7f9f401c68d0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9f38005a30 con 0x7f9f401c68d0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 0x7f9f2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.695 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9f38032870 con 0x7f9f401c68d0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.696 7f9f45746700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 0x7f9f2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:28.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.696 7f9f45746700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 0x7f9f2001abc0 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:28.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.727 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f40086140 con 0x7f9f401c68d0 2021-02-10T11:45:28.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.730 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9f38005820 con 0x7f9f401c68d0 2021-02-10T11:45:28.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.853 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f40086140 con 0x7f9f401c68d0 2021-02-10T11:45:28.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.854 7f9f35ffb700 1 -- 172.21.2.117:0/3076918349 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f9f38005a30 con 0x7f9f401c68d0 2021-02-10T11:45:28.849 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:28.849 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":83,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:26.614628","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:23.589151","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:24.659131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:28.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 msgr2=0x7f9f2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 0x7f9f2001abc0 crc :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 msgr2=0x7f9f400edd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f9f38009a20 tx=0x7f9f3800bd10).stop 2021-02-10T11:45:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 shutdown_connections 2021-02-10T11:45:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f200186c0 0x7f9f2001abc0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9f40099780 0x7f9f401c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9f401c68d0 0x7f9f400edd00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 --2- 172.21.2.117:0/3076918349 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9f400a3200 0x7f9f400f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 >> 172.21.2.117:0/3076918349 conn(0x7f9f400f3230 msgr2=0x7f9f400b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 shutdown_connections 2021-02-10T11:45:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:28.858 7f9f4792a700 1 -- 172.21.2.117:0/3076918349 wait complete. 2021-02-10T11:45:28.860 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.289 7f4b75f9e700 1 Processor -- start 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.289 7f4b75f9e700 1 -- start start 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b75f9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b7007ab40 0x7f4b700794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b75f9e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700e5770 con 0x7f4b70079f10 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b75f9e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7007a310 con 0x7f4b70079b10 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b75f9e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700d18c0 con 0x7f4b7007ab40 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4b70079f10 0x7f4b7007b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52004/0 (socket says 172.21.2.117:52004) 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4b7007ab40 0x7f4b700794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6ffff700 1 -- 172.21.2.117:0/3837501944 learned_addr learned my addr 172.21.2.117:0/3837501944 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6f7fe700 1 -- 172.21.2.117:0/3837501944 >> v1:172.21.2.125:6789/0 conn(0x7f4b70079b10 legacy=0x7f4b7007af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6f7fe700 1 -- 172.21.2.117:0/3837501944 >> v1:172.21.2.117:6789/0 conn(0x7f4b70079f10 legacy=0x7f4b7007b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.290 7f4b6f7fe700 1 -- 172.21.2.117:0/3837501944 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7007bf20 con 0x7f4b7007ab40 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b6f7fe700 1 --2- 172.21.2.117:0/3837501944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007ab40 0x7f4b700794f0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f4b58006650 tx=0x7f4b58009a80).ready entity=mon.0 client_cookie=acd1810fa99cfb70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b6e7fc700 1 -- 172.21.2.117:0/3837501944 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b5800c0c0 con 0x7f4b7007ab40 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b6e7fc700 1 -- 172.21.2.117:0/3837501944 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5800c340 con 0x7f4b7007ab40 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b6e7fc700 1 -- 172.21.2.117:0/3837501944 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b5800c860 con 0x7f4b7007ab40 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b75f9e700 1 -- 172.21.2.117:0/3837501944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007ab40 msgr2=0x7f4b700794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.291 7f4b75f9e700 1 --2- 172.21.2.117:0/3837501944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007ab40 0x7f4b700794f0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 rx=0x7f4b58006650 tx=0x7f4b58009a80).stop 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.292 7f4b75f9e700 1 -- 172.21.2.117:0/3837501944 shutdown_connections 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.292 7f4b75f9e700 1 --2- 172.21.2.117:0/3837501944 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b7007ab40 0x7f4b700794f0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.292 7f4b75f9e700 1 -- 172.21.2.117:0/3837501944 >> 172.21.2.117:0/3837501944 conn(0x7f4b700d0910 msgr2=0x7f4b700d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:29.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.292 7f4b75f9e700 1 -- 172.21.2.117:0/3837501944 shutdown_connections 2021-02-10T11:45:29.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.292 7f4b75f9e700 1 -- 172.21.2.117:0/3837501944 wait complete. 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.306 7f4b75f9e700 1 Processor -- start 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.306 7f4b75f9e700 1 -- start start 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7009bf80 0x7f4b701c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b700ff0c0 0x7f4b700ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c3910 con 0x7f4b701c6940 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70098c30 con 0x7f4b700ff0c0 2021-02-10T11:45:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b75f9e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70091e90 con 0x7f4b7009bf80 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51760/0 (socket says 172.21.2.117:51760) 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 -- 172.21.2.117:0/1568739387 learned_addr learned my addr 172.21.2.117:0/1568739387 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 -- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7009bf80 msgr2=0x7f4b701c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7009bf80 0x7f4b701c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 -- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b700ff0c0 msgr2=0x7f4b700ff4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b700ff0c0 0x7f4b700ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.307 7f4b6effd700 1 -- 172.21.2.117:0/1568739387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700ba880 con 0x7f4b701c6940 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b6effd700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f4b6000b5b0 tx=0x7f4b6000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b6000bd40 con 0x7f4b701c6940 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60006240 con 0x7f4b701c6940 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4b60005720 con 0x7f4b701c6940 2021-02-10T11:45:29.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b700ba880 con 0x7f4b701c6940 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.308 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b700bae10 con 0x7f4b701c6940 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.309 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4b600054a0 con 0x7f4b701c6940 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.309 7f4b67fff700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 0x7f4b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.309 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4b60032930 con 0x7f4b701c6940 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.309 7f4b6f7fe700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 0x7f4b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.310 7f4b6f7fe700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 0x7f4b5401abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.339 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b700d1c60 con 0x7f4b701c6940 2021-02-10T11:45:29.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.342 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4b60000c60 con 0x7f4b701c6940 2021-02-10T11:45:29.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.471 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4b700d1c60 con 0x7f4b701c6940 2021-02-10T11:45:29.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.475 7f4b67fff700 1 -- 172.21.2.117:0/1568739387 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4b600054a0 con 0x7f4b701c6940 2021-02-10T11:45:29.472 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:45:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 msgr2=0x7f4b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 0x7f4b5401abc0 crc :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 msgr2=0x7f4b700feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 rx=0x7f4b6000b5b0 tx=0x7f4b6000b950).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 shutdown_connections 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4b540186c0 0x7f4b5401abc0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4b701c6940 0x7f4b700feaa0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4b700ff0c0 0x7f4b700ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 --2- 172.21.2.117:0/1568739387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4b7009bf80 0x7f4b701c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 >> 172.21.2.117:0/1568739387 conn(0x7f4b7009e020 msgr2=0x7f4b700fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 shutdown_connections 2021-02-10T11:45:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.481 7f4b75f9e700 1 -- 172.21.2.117:0/1568739387 wait complete. 2021-02-10T11:45:29.482 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:29.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.912 7f1747938700 1 Processor -- start 2021-02-10T11:45:29.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 -- start start 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f174007ab40 0x7f17400794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1740079b10 0x7f1740079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f174007af40 0x7f174007b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400e5770 con 0x7f174007ab40 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17400d18c0 con 0x7f1740079b10 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1747938700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17400d1c60 con 0x7f174007af40 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1745754700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f174007ab40 0x7f17400794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1745754700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f174007ab40 0x7f17400794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51764/0 (socket says 172.21.2.117:51764) 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.913 7f1745754700 1 -- 172.21.2.117:0/1003379371 learned_addr learned my addr 172.21.2.117:0/1003379371 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 -- 172.21.2.117:0/1003379371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f174007af40 msgr2=0x7f174007b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 --2- 172.21.2.117:0/1003379371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f174007af40 0x7f174007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 -- 172.21.2.117:0/1003379371 >> v2:172.21.2.125:3300/0 conn(0x7f1740079b10 msgr2=0x7f1740079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 --2- 172.21.2.117:0/1003379371 >> v2:172.21.2.125:3300/0 conn(0x7f1740079b10 0x7f1740079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 -- 172.21.2.117:0/1003379371 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174007b960 con 0x7f174007ab40 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1745754700 1 --2- 172.21.2.117:0/1003379371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f174007ab40 0x7f17400794f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f173c006700 tx=0x7f173c009b80).ready entity=mon.0 client_cookie=b1112d227912764a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1737fff700 1 -- 172.21.2.117:0/1003379371 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f173c00c0c0 con 0x7f174007ab40 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1737fff700 1 -- 172.21.2.117:0/1003379371 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c00c340 con 0x7f174007ab40 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1737fff700 1 -- 172.21.2.117:0/1003379371 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f173c00c860 con 0x7f174007ab40 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1747938700 1 -- 172.21.2.117:0/1003379371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f174007ab40 msgr2=0x7f17400794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.914 7f1747938700 1 --2- 172.21.2.117:0/1003379371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f174007ab40 0x7f17400794f0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 rx=0x7f173c006700 tx=0x7f173c009b80).stop 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 -- 172.21.2.117:0/1003379371 shutdown_connections 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 --2- 172.21.2.117:0/1003379371 >> v2:172.21.2.125:3300/0 conn(0x7f1740079b10 0x7f1740079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 --2- 172.21.2.117:0/1003379371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f174007af40 0x7f174007b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 --2- 172.21.2.117:0/1003379371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f174007ab40 0x7f17400794f0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 -- 172.21.2.117:0/1003379371 >> 172.21.2.117:0/1003379371 conn(0x7f17400d0910 msgr2=0x7f17400d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 -- 172.21.2.117:0/1003379371 shutdown_connections 2021-02-10T11:45:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.915 7f1747938700 1 -- 172.21.2.117:0/1003379371 wait complete. 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 Processor -- start 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 -- start start 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17400942a0 0x7f1740092ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17401a6da0 0x7f1740094e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401a71a0 con 0x7f17401a6380 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1740094890 con 0x7f17400942a0 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1747938700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f174019f070 con 0x7f17401a6da0 2021-02-10T11:45:29.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1744f53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.930 7f1744f53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51768/0 (socket says 172.21.2.117:51768) 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 -- 172.21.2.117:0/231620897 learned_addr learned my addr 172.21.2.117:0/231620897 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1745f55700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17401a6da0 0x7f1740094e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1745754700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17400942a0 0x7f1740092ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 -- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17401a6da0 msgr2=0x7f1740094e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17401a6da0 0x7f1740094e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 -- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17400942a0 msgr2=0x7f1740092ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17400942a0 0x7f1740092ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 -- 172.21.2.117:0/231620897 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1740095430 con 0x7f17401a6380 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1744f53700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f173000b530 tx=0x7f173000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1730006350 con 0x7f17401a6380 2021-02-10T11:45:29.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1730000c90 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1747938700 1 -- 172.21.2.117:0/231620897 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1740095430 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1730002020 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.931 7f1747938700 1 -- 172.21.2.117:0/231620897 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17400959c0 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.932 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1730005310 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.932 7f1735ffb700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 0x7f172001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.932 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f17300326e0 con 0x7f17401a6380 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.932 7f1745754700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 0x7f172001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:29.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.933 7f1745754700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 0x7f172001abc0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:29.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.963 7f1747938700 1 -- 172.21.2.117:0/231620897 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17401a71a0 con 0x7f17401a6380 2021-02-10T11:45:29.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:29.966 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1730000960 con 0x7f17401a6380 2021-02-10T11:45:30.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.095 7f1747938700 1 -- 172.21.2.117:0/231620897 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f17401a71a0 con 0x7f17401a6380 2021-02-10T11:45:30.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.099 7f1735ffb700 1 -- 172.21.2.117:0/231620897 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f17300056f0 con 0x7f17401a6380 2021-02-10T11:45:30.095 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:45:30.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 msgr2=0x7f172001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 0x7f172001abc0 crc :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 msgr2=0x7f17401a6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 rx=0x7f173000b530 tx=0x7f173000b8d0).stop 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 shutdown_connections 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17200186c0 0x7f172001abc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17401a6380 0x7f17401a6780 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17401a6da0 0x7f1740094e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 --2- 172.21.2.117:0/231620897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17400942a0 0x7f1740092ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 >> 172.21.2.117:0/231620897 conn(0x7f174019f690 msgr2=0x7f1740198b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 shutdown_connections 2021-02-10T11:45:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.105 7f1747938700 1 -- 172.21.2.117:0/231620897 wait complete. 2021-02-10T11:45:30.106 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 Processor -- start 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 -- start start 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa97c07a0e0 0x7fa97c1b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c0e5770 con 0x7fa97c079ce0 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c0d18c0 con 0x7fa97c07ab40 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa982fbf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c07a640 con 0x7fa97c07a0e0 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa9815dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa97c07a0e0 0x7fa97c1b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa97bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa97c079ce0 0x7fa97c1b59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52028/0 (socket says 172.21.2.117:52028) 2021-02-10T11:45:30.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.536 7fa97bfff700 1 -- 172.21.2.117:0/2705075800 learned_addr learned my addr 172.21.2.117:0/2705075800 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa9815dc700 1 -- 172.21.2.117:0/2705075800 >> v1:172.21.2.125:6789/0 conn(0x7fa97c07ab40 legacy=0x7fa97c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa9815dc700 1 -- 172.21.2.117:0/2705075800 >> v1:172.21.2.117:6789/0 conn(0x7fa97c079ce0 legacy=0x7fa97c1b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa9815dc700 1 -- 172.21.2.117:0/2705075800 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c07bf50 con 0x7fa97c07a0e0 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa9815dc700 1 --2- 172.21.2.117:0/2705075800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c07a0e0 0x7fa97c1b91e0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7fa96c006ab0 tx=0x7fa96c009b40).ready entity=mon.0 client_cookie=11e81873f19ac18d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa97b7fe700 1 -- 172.21.2.117:0/2705075800 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c0022e0 con 0x7fa97c07a0e0 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa97b7fe700 1 -- 172.21.2.117:0/2705075800 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c0025e0 con 0x7fa97c07a0e0 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa97b7fe700 1 -- 172.21.2.117:0/2705075800 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c002b00 con 0x7fa97c07a0e0 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa982fbf700 1 -- 172.21.2.117:0/2705075800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c07a0e0 msgr2=0x7fa97c1b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.537 7fa982fbf700 1 --2- 172.21.2.117:0/2705075800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c07a0e0 0x7fa97c1b91e0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 rx=0x7fa96c006ab0 tx=0x7fa96c009b40).stop 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.538 7fa982fbf700 1 -- 172.21.2.117:0/2705075800 shutdown_connections 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.538 7fa982fbf700 1 --2- 172.21.2.117:0/2705075800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c07a0e0 0x7fa97c1b91e0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.538 7fa982fbf700 1 -- 172.21.2.117:0/2705075800 >> 172.21.2.117:0/2705075800 conn(0x7fa97c0d0910 msgr2=0x7fa97c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.538 7fa982fbf700 1 -- 172.21.2.117:0/2705075800 shutdown_connections 2021-02-10T11:45:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.538 7fa982fbf700 1 -- 172.21.2.117:0/2705075800 wait complete. 2021-02-10T11:45:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.553 7fa982fbf700 1 Processor -- start 2021-02-10T11:45:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.553 7fa982fbf700 1 -- start start 2021-02-10T11:45:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.553 7fa982fbf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa97c100010 0x7fa97c1b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.553 7fa982fbf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa97c100820 0x7fa97c1c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.553 7fa982fbf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa982fbf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c1b4130 con 0x7fa97c1c5460 2021-02-10T11:45:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa982fbf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c100410 con 0x7fa97c100820 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa982fbf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c100c20 con 0x7fa97c100010 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51784/0 (socket says 172.21.2.117:51784) 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 -- 172.21.2.117:0/234781637 learned_addr learned my addr 172.21.2.117:0/234781637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 -- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa97c100010 msgr2=0x7fa97c1b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa97c100010 0x7fa97c1b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 -- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa97c100820 msgr2=0x7fa97c1c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa97c100820 0x7fa97c1c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 -- 172.21.2.117:0/234781637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c1c5860 con 0x7fa97c1c5460 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.554 7fa9815dc700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7fa96c00b5e0 tx=0x7fa96c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c006b40 con 0x7fa97c1c5460 2021-02-10T11:45:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c009070 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa96c002470 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa982fbf700 1 -- 172.21.2.117:0/234781637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c1c5860 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa982fbf700 1 -- 172.21.2.117:0/234781637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c1b4c50 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa96c006a20 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 0x7fa95801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.555 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa96c0326d0 con 0x7fa97c1c5460 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.556 7fa980ddb700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 0x7fa95801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:30.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.556 7fa980ddb700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 0x7fa95801abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:30.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.586 7fa982fbf700 1 -- 172.21.2.117:0/234781637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c0d1c60 con 0x7fa97c1c5460 2021-02-10T11:45:30.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.589 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa96c0093e0 con 0x7fa97c1c5460 2021-02-10T11:45:30.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.719 7fa982fbf700 1 -- 172.21.2.117:0/234781637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa97c1b40b0 con 0x7fa97c1c5460 2021-02-10T11:45:30.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.722 7fa9797fa700 1 -- 172.21.2.117:0/234781637 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa96c01f2c0 con 0x7fa97c1c5460 2021-02-10T11:45:30.715 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:45:30.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.723 7fa982fbf700 1 -- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 msgr2=0x7fa95801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.723 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 0x7fa95801abc0 crc :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.723 7fa982fbf700 1 -- 172.21.2.117:0/234781637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 msgr2=0x7fa97c1b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:30.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.723 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 rx=0x7fa96c00b5e0 tx=0x7fa96c00b980).stop 2021-02-10T11:45:30.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 -- 172.21.2.117:0/234781637 shutdown_connections 2021-02-10T11:45:30.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9580186c0 0x7fa95801abc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa97c1c5460 0x7fa97c1b4630 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:30.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa97c100820 0x7fa97c1c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:30.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 --2- 172.21.2.117:0/234781637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa97c100010 0x7fa97c1b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:30.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 -- 172.21.2.117:0/234781637 >> 172.21.2.117:0/234781637 conn(0x7fa97c09e020 msgr2=0x7fa97c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:30.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 -- 172.21.2.117:0/234781637 shutdown_connections 2021-02-10T11:45:30.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:30.724 7fa982fbf700 1 -- 172.21.2.117:0/234781637 wait complete. 2021-02-10T11:45:30.726 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.154 7f3b6161e700 1 Processor -- start 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.154 7f3b6161e700 1 -- start start 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b6161e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b5c079ce0 0x7f3b5c1b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b6161e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c07a5c0 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b6161e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0d18c0 con 0x7f3b5c07ab40 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b6161e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c07bf50 con 0x7f3b5c079ce0 2021-02-10T11:45:31.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5bac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3b5c07a0e0 0x7f3b5c1b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52044/0 (socket says 172.21.2.117:52044) 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5bac0700 1 -- 172.21.2.117:0/1138692407 learned_addr learned my addr 172.21.2.117:0/1138692407 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338535621 0 0) 0x7f3b54001500 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b48002fe0 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5aabe700 1 --2- 172.21.2.117:0/1138692407 >> v2:172.21.2.125:3300/0 conn(0x7f3b5c079ce0 0x7f3b5c1b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3398892619 0 0) 0x7f3b54001500 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 >> v2:172.21.2.125:3300/0 conn(0x7f3b5c079ce0 msgr2=0x7f3b5c1b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 --2- 172.21.2.117:0/1138692407 >> v2:172.21.2.125:3300/0 conn(0x7f3b5c079ce0 0x7f3b5c1b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 >> v1:172.21.2.125:6789/0 conn(0x7f3b5c07ab40 legacy=0x7f3b5c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c0d1c60 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3b54001a00 con 0x7f3b5c07a0e0 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 >> v1:172.21.2.117:6789/0 conn(0x7f3b5c07a0e0 legacy=0x7f3b5c1b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 0x7f3b48005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b480067a0 0x7f3b48008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b480095a0 0x7f3b4800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4800c310 con 0x7f3b48003a70 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.155 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4800c530 con 0x7f3b480095a0 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b48050760 con 0x7f3b480067a0 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 0x7f3b48005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5bac0700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b480095a0 0x7f3b4800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5aabe700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b480067a0 0x7f3b48008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 -- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b480067a0 msgr2=0x7f3b48008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b480067a0 0x7f3b48008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 -- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b480095a0 msgr2=0x7f3b4800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b480095a0 0x7f3b4800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 -- 172.21.2.117:0/1138692407 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480509c0 con 0x7f3b48003a70 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5bac0700 1 -- 172.21.2.117:0/1138692407 reap_dead start 2021-02-10T11:45:31.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.156 7f3b5b2bf700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 0x7f3b48005f00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7f3b4c006ac0 tx=0x7f3b4c009af0).ready entity=mon.0 client_cookie=526b03fa15f96f67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4c007280 con 0x7f3b48003a70 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c007600 con 0x7f3b48003a70 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b5a2bd700 1 -- 172.21.2.117:0/1138692407 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4c007b20 con 0x7f3b48003a70 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 -- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 msgr2=0x7f3b48005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 0x7f3b48005f00 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 rx=0x7f3b4c006ac0 tx=0x7f3b4c009af0).stop 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 -- 172.21.2.117:0/1138692407 shutdown_connections 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 --2- 172.21.2.117:0/1138692407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b48003a70 0x7f3b48005f00 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 -- 172.21.2.117:0/1138692407 >> 172.21.2.117:0/1138692407 conn(0x7f3b5c0d0910 msgr2=0x7f3b5c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 -- 172.21.2.117:0/1138692407 shutdown_connections 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.157 7f3b6161e700 1 -- 172.21.2.117:0/1138692407 wait complete. 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 Processor -- start 2021-02-10T11:45:31.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 -- start start 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b5c1a3ae0 0x7f3b5c08bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b5c08c3e0 0x7f3b5c100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0b2770 con 0x7f3b5c0b2190 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0dce90 con 0x7f3b5c1a3ae0 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.172 7f3b6161e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0ffdd0 con 0x7f3b5c08c3e0 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51802/0 (socket says 172.21.2.117:51802) 2021-02-10T11:45:31.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 -- 172.21.2.117:0/1633406917 learned_addr learned my addr 172.21.2.117:0/1633406917 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 -- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b5c08c3e0 msgr2=0x7f3b5c100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b5c08c3e0 0x7f3b5c100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 -- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b5c1a3ae0 msgr2=0x7f3b5c08bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b5c1a3ae0 0x7f3b5c08bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 -- 172.21.2.117:0/1633406917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c100940 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b5b2bf700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f3b4c006af0 tx=0x7f3b4c00c920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4c00cd10 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00a0d0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.173 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c100940 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.174 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4c00a520 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.174 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c100ed0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.174 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b4c002050 con 0x7f3b5c0b2190 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.174 7f3b53fff700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 0x7f3b4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.174 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b4c0326c0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.175 7f3b5aabe700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 0x7f3b4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.176 7f3b5aabe700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 0x7f3b4001abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.204 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c0dce90 con 0x7f3b5c0b2190 2021-02-10T11:45:31.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.207 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b4c007de0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.338 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3b5c07a5c0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.341 7f3b53fff700 1 -- 172.21.2.117:0/1633406917 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v50) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3b4c01f2e0 con 0x7f3b5c0b2190 2021-02-10T11:45:31.338 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:45:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.346 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 msgr2=0x7f3b4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.346 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 0x7f3b4001abc0 crc :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.346 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 msgr2=0x7f3b5c1a34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.346 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 rx=0x7f3b4c006af0 tx=0x7f3b4c00c920).stop 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 shutdown_connections 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b5c08c3e0 0x7f3b5c100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b5c1a3ae0 0x7f3b5c08bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b400186c0 0x7f3b4001abc0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 --2- 172.21.2.117:0/1633406917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5c0b2190 0x7f3b5c1a34c0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 >> 172.21.2.117:0/1633406917 conn(0x7f3b5c09f900 msgr2=0x7f3b5c0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 shutdown_connections 2021-02-10T11:45:31.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.347 7f3b6161e700 1 -- 172.21.2.117:0/1633406917 wait complete. 2021-02-10T11:45:31.348 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.766 7f7912cb3700 1 Processor -- start 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.766 7f7912cb3700 1 -- start start 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f7912cb3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1b6960 0x7f790c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f7912cb3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0ede60 con 0x7f790c0ef3f0 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f7912cb3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0eb9c0 con 0x7f790c0d18e0 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f7912cb3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0d1ce0 con 0x7f790c1b6960 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f790c0ef3f0 0x7f790c0eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52062/0 (socket says 172.21.2.117:52062) 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790bfff700 1 -- 172.21.2.117:0/1931602396 learned_addr learned my addr 172.21.2.117:0/1931602396 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f79112d0700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1b6960 0x7f790c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408487347 0 0) 0x7f78fc000d30 con 0x7f790c0ef3f0 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f8002fe0 con 0x7f790c0ef3f0 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 407653143 0 0) 0x7f78fc001390 con 0x7f790c0ef3f0 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1b6960 msgr2=0x7f790c1b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1b6960 0x7f790c1b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 >> v1:172.21.2.125:6789/0 conn(0x7f790c0d18e0 legacy=0x7f790c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c1bb3b0 con 0x7f790c0ef3f0 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f78fc001910 con 0x7f790c0ef3f0 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 >> v1:172.21.2.117:6789/0 conn(0x7f790c0ef3f0 legacy=0x7f790c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78f8003ae0 0x7f78f8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78f8006830 0x7f78f8008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78f80095a0 0x7f78f800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f800c370 con 0x7f78f8003ae0 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.767 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f800c590 con 0x7f78f80095a0 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80507c0 con 0x7f78f8006830 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78f8003ae0 0x7f78f8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f79112d0700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1b6960 0x7f790c1b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f790bfff700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78f8006830 0x7f78f8008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 -- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78f8006830 msgr2=0x7f78f8008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78f8006830 0x7f78f8008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 -- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78f80095a0 msgr2=0x7f78f800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78f80095a0 0x7f78f800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 -- 172.21.2.117:0/1931602396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8050a20 con 0x7f78f8003ae0 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7910acf700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78f8003ae0 0x7f78f8005f90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7f78f4006ad0 tx=0x7f78f4009b40).ready entity=mon.0 client_cookie=c3f6c5dcec9860bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78f400c0c0 con 0x7f78f8003ae0 2021-02-10T11:45:31.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f400c380 con 0x7f78f8003ae0 2021-02-10T11:45:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f790b7fe700 1 -- 172.21.2.117:0/1931602396 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78f400c860 con 0x7f78f8003ae0 2021-02-10T11:45:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7912cb3700 1 -- 172.21.2.117:0/1931602396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78f8003ae0 msgr2=0x7f78f8005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7912cb3700 1 --2- 172.21.2.117:0/1931602396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78f8003ae0 0x7f78f8005f90 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 rx=0x7f78f4006ad0 tx=0x7f78f4009b40).stop 2021-02-10T11:45:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f79112d0700 1 -- 172.21.2.117:0/1931602396 reap_dead start 2021-02-10T11:45:31.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7912cb3700 1 -- 172.21.2.117:0/1931602396 shutdown_connections 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.768 7f7912cb3700 1 -- 172.21.2.117:0/1931602396 >> 172.21.2.117:0/1931602396 conn(0x7f790c0ced60 msgr2=0x7f790c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.769 7f7912cb3700 1 -- 172.21.2.117:0/1931602396 shutdown_connections 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.769 7f7912cb3700 1 -- 172.21.2.117:0/1931602396 wait complete. 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.783 7f7912cb3700 1 Processor -- start 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.783 7f7912cb3700 1 -- start start 2021-02-10T11:45:31.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 0x7f790c08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1a3480 0x7f790c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0b61a0 con 0x7f790c0b5da0 2021-02-10T11:45:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c08c8b0 con 0x7f790c0b59a0 2021-02-10T11:45:31.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7912cb3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0ba1c0 con 0x7f790c1a3480 2021-02-10T11:45:31.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51822/0 (socket says 172.21.2.117:51822) 2021-02-10T11:45:31.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 -- 172.21.2.117:0/962498586 learned_addr learned my addr 172.21.2.117:0/962498586 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:31.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f79112d0700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1a3480 0x7f790c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7910acf700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 0x7f790c08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 -- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1a3480 msgr2=0x7f790c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1a3480 0x7f790c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 -- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 msgr2=0x7f790c08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 0x7f790c08c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 -- 172.21.2.117:0/962498586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c1a3880 con 0x7f790c0b5da0 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f7910acf700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 0x7f790c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f790bfff700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7f78fc001620 tx=0x7f78fc00ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.784 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78fc0018b0 con 0x7f790c0b5da0 2021-02-10T11:45:31.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc0070d0 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f7912cb3700 1 -- 172.21.2.117:0/962498586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c1a3880 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78fc0048c0 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f7912cb3700 1 -- 172.21.2.117:0/962498586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c1a3880 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78fc001010 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f79097fa700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 0x7f78f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78fc032b50 con 0x7f790c0b5da0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.785 7f7910acf700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 0x7f78f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:31.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.786 7f7910acf700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 0x7f78f001abc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.816 7f7912cb3700 1 -- 172.21.2.117:0/962498586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c0ede60 con 0x7f790c0b5da0 2021-02-10T11:45:31.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.819 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78fc0070b0 con 0x7f790c0b5da0 2021-02-10T11:45:31.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.947 7f7912cb3700 1 -- 172.21.2.117:0/962498586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f790c0ede60 con 0x7f790c0b5da0 2021-02-10T11:45:31.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.948 7f79097fa700 1 -- 172.21.2.117:0/962498586 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f78fc001010 con 0x7f790c0b5da0 2021-02-10T11:45:31.941 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:31.941 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:31.941 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 61 2021-02-10T11:45:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.949 7f7912cb3700 1 -- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 msgr2=0x7f78f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.949 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 0x7f78f001abc0 crc :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.949 7f7912cb3700 1 -- 172.21.2.117:0/962498586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 msgr2=0x7f790c1a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:31.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.949 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 rx=0x7f78fc001620 tx=0x7f78fc00ac60).stop 2021-02-10T11:45:31.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 -- 172.21.2.117:0/962498586 shutdown_connections 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78f0018730 0x7f78f001abc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f790c0b5da0 0x7f790c1a2e60 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f790c1a3480 0x7f790c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 --2- 172.21.2.117:0/962498586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f790c0b59a0 0x7f790c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 -- 172.21.2.117:0/962498586 >> 172.21.2.117:0/962498586 conn(0x7f790c0bb5e0 msgr2=0x7f790c090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 -- 172.21.2.117:0/962498586 shutdown_connections 2021-02-10T11:45:31.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:31.950 7f7912cb3700 1 -- 172.21.2.117:0/962498586 wait complete. 2021-02-10T11:45:32.153 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:25.613265', 'modified': '2021-02-10 11:45:27.622343', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7144}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7144': {'gid': 7144, 'name': 'b', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1925085474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1925085474}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1925085474}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:32.153 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:32.153 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7144, 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9437528720000001, 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 61, 2021-02-10T11:45:32.552 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 83, 2021-02-10T11:45:32.553 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:45:32.553 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.577264763000001 2021-02-10T11:45:32.553 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:45:32.556 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7144, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9437528720000001, "mdsmap_epoch": 61, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 12.577264763000001 } 2021-02-10T11:45:32.556 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:32.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 Processor -- start 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 -- start start 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef740d18e0 0x7fef740ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740eb000 0x7fef740eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740ede60 con 0x7fef740ee3e0 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740eba90 con 0x7fef740d18e0 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.974 7fef78970700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740d1ce0 con 0x7fef740eb000 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef740d18e0 0x7fef740ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef740d18e0 0x7fef740ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51830/0 (socket says 172.21.2.117:51830) 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 -- 172.21.2.117:0/3039562102 learned_addr learned my addr 172.21.2.117:0/3039562102 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef71e1b700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740eb000 0x7fef740eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 -- 172.21.2.117:0/3039562102 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740eb000 msgr2=0x7fef740eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:32.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740eb000 0x7fef740eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 -- 172.21.2.117:0/3039562102 >> v1:172.21.2.125:6789/0 conn(0x7fef740ee3e0 legacy=0x7fef740ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 -- 172.21.2.117:0/3039562102 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740eefd0 con 0x7fef740d18e0 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7261c700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef740d18e0 0x7fef740ea9e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7fef64006700 tx=0x7fef64009b00).ready entity=mon.0 client_cookie=39ffa1339362b22d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7161a700 1 -- 172.21.2.117:0/3039562102 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef6400c0c0 con 0x7fef740d18e0 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7161a700 1 -- 172.21.2.117:0/3039562102 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6400c400 con 0x7fef740d18e0 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef7161a700 1 -- 172.21.2.117:0/3039562102 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef6400c920 con 0x7fef740d18e0 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef78970700 1 -- 172.21.2.117:0/3039562102 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef740d18e0 msgr2=0x7fef740ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.975 7fef78970700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef740d18e0 0x7fef740ea9e0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 rx=0x7fef64006700 tx=0x7fef64009b00).stop 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 -- 172.21.2.117:0/3039562102 shutdown_connections 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740eb000 0x7fef740eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 --2- 172.21.2.117:0/3039562102 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef740d18e0 0x7fef740ea9e0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 -- 172.21.2.117:0/3039562102 >> 172.21.2.117:0/3039562102 conn(0x7fef740ced60 msgr2=0x7fef740cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:32.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 -- 172.21.2.117:0/3039562102 shutdown_connections 2021-02-10T11:45:32.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.976 7fef78970700 1 -- 172.21.2.117:0/3039562102 wait complete. 2021-02-10T11:45:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.991 7fef78970700 1 Processor -- start 2021-02-10T11:45:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.991 7fef78970700 1 -- start start 2021-02-10T11:45:32.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740b82f0 0x7fef740b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef740b8d10 0x7fef740b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740c0670 con 0x7fef741a33d0 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740deb20 con 0x7fef740b8d10 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef78970700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740ded40 con 0x7fef740b82f0 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51836/0 (socket says 172.21.2.117:51836) 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 -- 172.21.2.117:0/2939217648 learned_addr learned my addr 172.21.2.117:0/2939217648 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 -- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740b82f0 msgr2=0x7fef740b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740b82f0 0x7fef740b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 -- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef740b8d10 msgr2=0x7fef740b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef740b8d10 0x7fef740b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:32.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 -- 172.21.2.117:0/2939217648 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740def60 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.992 7fef7261c700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7fef6400b660 tx=0x7fef6400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef64009440 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef64000c90 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef64002030 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef78970700 1 -- 172.21.2.117:0/2939217648 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef740def60 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef78970700 1 -- 172.21.2.117:0/2939217648 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef740def60 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef640093e0 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 0x7fef5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.993 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef64032710 con 0x7fef741a33d0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.994 7fef71e1b700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 0x7fef5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:32.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:32.995 7fef71e1b700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 0x7fef5801abc0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.024 7fef78970700 1 -- 172.21.2.117:0/2939217648 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef740c0670 con 0x7fef741a33d0 2021-02-10T11:45:33.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.027 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef640090c0 con 0x7fef741a33d0 2021-02-10T11:45:33.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.149 7fef78970700 1 -- 172.21.2.117:0/2939217648 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef740c0670 con 0x7fef741a33d0 2021-02-10T11:45:33.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.150 7fef6affd700 1 -- 172.21.2.117:0/2939217648 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+14229 (secure 0 0 0) 0x7fef640093e0 con 0x7fef741a33d0 2021-02-10T11:45:33.143 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:33.144 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":83,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:26.614628","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:23.589151","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:24.659131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.151 7fef78970700 1 -- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 msgr2=0x7fef5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 0x7fef5801abc0 crc :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 -- 172.21.2.117:0/2939217648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 msgr2=0x7fef741a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7fef6400b660 tx=0x7fef6400ba00).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 -- 172.21.2.117:0/2939217648 shutdown_connections 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef740b8d10 0x7fef740b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef740b82f0 0x7fef740b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef580186c0 0x7fef5801abc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 --2- 172.21.2.117:0/2939217648 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef741a33d0 0x7fef741a37d0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 -- 172.21.2.117:0/2939217648 >> 172.21.2.117:0/2939217648 conn(0x7fef740874c0 msgr2=0x7fef74096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 -- 172.21.2.117:0/2939217648 shutdown_connections 2021-02-10T11:45:33.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.152 7fef78970700 1 -- 172.21.2.117:0/2939217648 wait complete. 2021-02-10T11:45:33.154 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:45:23.589151', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:45:24.659131', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:45:33.154 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.577 7f51ff0b9700 1 Processor -- start 2021-02-10T11:45:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.577 7f51ff0b9700 1 -- start start 2021-02-10T11:45:33.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51ff0b9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f51f80d18f0 0x7f51f80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51ff0b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51f80ef3f0 0x7f51f81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51ff0b9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f80ef7f0 con 0x7f51f80eb040 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51ff0b9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f80ede60 con 0x7f51f80ef3f0 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51ff0b9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f80d1cf0 con 0x7f51f80d18f0 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51fd6d6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51f80ef3f0 0x7f51f81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51f7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f51f80eb040 0x7f51f80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52094/0 (socket says 172.21.2.117:52094) 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.578 7f51f7fff700 1 -- 172.21.2.117:0/4232236536 learned_addr learned my addr 172.21.2.117:0/4232236536 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51fd6d6700 1 -- 172.21.2.117:0/4232236536 >> v2:172.21.2.125:3300/0 conn(0x7f51f80d18f0 msgr2=0x7f51f80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51fd6d6700 1 --2- 172.21.2.117:0/4232236536 >> v2:172.21.2.125:3300/0 conn(0x7f51f80d18f0 0x7f51f80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51fd6d6700 1 -- 172.21.2.117:0/4232236536 >> v1:172.21.2.117:6789/0 conn(0x7f51f80eb040 legacy=0x7f51f80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51f77fe700 1 -- 172.21.2.117:0/4232236536 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515357998 0 0) 0x7f51e8001180 con 0x7f51f80eb040 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51fd6d6700 1 -- 172.21.2.117:0/4232236536 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f81b8fd0 con 0x7f51f80ef3f0 2021-02-10T11:45:33.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51fd6d6700 1 --2- 172.21.2.117:0/4232236536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f80ef3f0 0x7f51f81b79a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7f51f0006e30 tx=0x7f51f0009de0).ready entity=mon.0 client_cookie=3e1c9f4919c96ac2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51f77fe700 1 -- 172.21.2.117:0/4232236536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f000b300 con 0x7f51f80ef3f0 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51f77fe700 1 -- 172.21.2.117:0/4232236536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f000b680 con 0x7f51f80ef3f0 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51f77fe700 1 -- 172.21.2.117:0/4232236536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51f000bba0 con 0x7f51f80ef3f0 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51ff0b9700 1 -- 172.21.2.117:0/4232236536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f80ef3f0 msgr2=0x7f51f81b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.579 7f51ff0b9700 1 --2- 172.21.2.117:0/4232236536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f80ef3f0 0x7f51f81b79a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 rx=0x7f51f0006e30 tx=0x7f51f0009de0).stop 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 -- 172.21.2.117:0/4232236536 shutdown_connections 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 --2- 172.21.2.117:0/4232236536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f80ef3f0 0x7f51f81b79a0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 --2- 172.21.2.117:0/4232236536 >> v2:172.21.2.125:3300/0 conn(0x7f51f80d18f0 0x7f51f80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 -- 172.21.2.117:0/4232236536 >> 172.21.2.117:0/4232236536 conn(0x7f51f80cf060 msgr2=0x7f51f80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 -- 172.21.2.117:0/4232236536 shutdown_connections 2021-02-10T11:45:33.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.580 7f51ff0b9700 1 -- 172.21.2.117:0/4232236536 wait complete. 2021-02-10T11:45:33.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 Processor -- start 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 -- start start 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51f81a7e20 0x7f51f80abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51f80ac320 0x7f51f80ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80ab780 con 0x7f51f8083720 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80acd40 con 0x7f51f81a7e20 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51ff0b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f80acf20 con 0x7f51f80ac320 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51fced5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51fced5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51850/0 (socket says 172.21.2.117:51850) 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.595 7f51fced5700 1 -- 172.21.2.117:0/1353339914 learned_addr learned my addr 172.21.2.117:0/1353339914 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 -- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51f80ac320 msgr2=0x7f51f80ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51f80ac320 0x7f51f80ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 -- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51f81a7e20 msgr2=0x7f51f80abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51f81a7e20 0x7f51f80abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 -- 172.21.2.117:0/1353339914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f80aa140 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51fced5700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f51e0006f00 tx=0x7f51e000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51e000bd30 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e0000c90 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51e00022a0 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f80aa140 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.596 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f80aa140 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.597 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51e0005310 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.597 7f51f57fa700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 0x7f51dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.597 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f51e00326b0 con 0x7f51f8083720 2021-02-10T11:45:33.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.598 7f51f7fff700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 0x7f51dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:33.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.598 7f51f7fff700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 0x7f51dc01abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:33.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.628 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f80f67e0 con 0x7f51f8083720 2021-02-10T11:45:33.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.630 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51e0006b90 con 0x7f51f8083720 2021-02-10T11:45:33.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.756 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f51f80f67e0 con 0x7f51f8083720 2021-02-10T11:45:33.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.757 7f51f57fa700 1 -- 172.21.2.117:0/1353339914 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v83) v1 ==== 92+0+10 (secure 0 0 0) 0x7f51e000bd30 con 0x7f51f8083720 2021-02-10T11:45:33.750 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.758 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 msgr2=0x7f51dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.758 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 0x7f51dc01abc0 crc :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.758 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 msgr2=0x7f51f81a7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.758 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 rx=0x7f51e0006f00 tx=0x7f51e000b940).stop 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 shutdown_connections 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51f80ac320 0x7f51f80ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51f81a7e20 0x7f51f80abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51dc0186c0 0x7f51dc01abc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 --2- 172.21.2.117:0/1353339914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51f8083720 0x7f51f81a7800 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:33.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 >> 172.21.2.117:0/1353339914 conn(0x7f51f8090920 msgr2=0x7f51f80a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:33.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 shutdown_connections 2021-02-10T11:45:33.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:33.759 7f51ff0b9700 1 -- 172.21.2.117:0/1353339914 wait complete. 2021-02-10T11:45:33.760 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:45:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.180 7f24bfc3f700 1 Processor -- start 2021-02-10T11:45:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.180 7f24bfc3f700 1 -- start start 2021-02-10T11:45:34.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bfc3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b80d18f0 0x7f24b80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bfc3f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b80ede60 con 0x7f24b80eb040 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bfc3f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b80d1cf0 con 0x7f24b80ef3f0 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bfc3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8088020 con 0x7f24b80d18f0 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bd25a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f24b80eb040 0x7f24b80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52108/0 (socket says 172.21.2.117:52108) 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bd25a700 1 -- 172.21.2.117:0/1734595635 learned_addr learned my addr 172.21.2.117:0/1734595635 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.181 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851063307 0 0) 0x7f24b40010b0 con 0x7f24b80eb040 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a4002fe0 con 0x7f24b80eb040 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bda5b700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b80d18f0 0x7f24b80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3038147652 0 0) 0x7f24b40010b0 con 0x7f24b80eb040 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b80d18f0 msgr2=0x7f24b80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b80d18f0 0x7f24b80eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 >> v1:172.21.2.125:6789/0 conn(0x7f24b80ef3f0 legacy=0x7f24b81b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b81bb200 con 0x7f24b80eb040 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f24b40015b0 con 0x7f24b80eb040 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 >> v1:172.21.2.117:6789/0 conn(0x7f24b80eb040 legacy=0x7f24b80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24a4003a70 0x7f24a4005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24a4006760 0x7f24a4008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24a4009500 0x7f24a400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a400c2d0 con 0x7f24a4006760 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a400c4f0 con 0x7f24a4009500 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bca59700 1 -- 172.21.2.117:0/1734595635 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a400c710 con 0x7f24a4003a70 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bda5b700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24a4003a70 0x7f24a4005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24bd25a700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24a4006760 0x7f24a4008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.182 7f24be25c700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24a4009500 0x7f24a400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 -- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24a4003a70 msgr2=0x7f24a4005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24a4003a70 0x7f24a4005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 -- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24a4009500 msgr2=0x7f24a400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24a4009500 0x7f24a400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 -- 172.21.2.117:0/1734595635 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4050950 con 0x7f24a4006760 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bd25a700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24a4006760 0x7f24a4008c60 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f24b4007560 tx=0x7f24b4007c90).ready entity=mon.0 client_cookie=2620eb4addebcd71 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24b40009b0 con 0x7f24a4006760 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b400a310 con 0x7f24a4006760 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bca59700 1 -- 172.21.2.117:0/1734595635 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24b400a760 con 0x7f24a4006760 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bfc3f700 1 -- 172.21.2.117:0/1734595635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24a4006760 msgr2=0x7f24a4008c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24bfc3f700 1 --2- 172.21.2.117:0/1734595635 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24a4006760 0x7f24a4008c60 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 rx=0x7f24b4007560 tx=0x7f24b4007c90).stop 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.183 7f24be25c700 1 -- 172.21.2.117:0/1734595635 reap_dead start 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.184 7f24bfc3f700 1 -- 172.21.2.117:0/1734595635 shutdown_connections 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.184 7f24bfc3f700 1 -- 172.21.2.117:0/1734595635 >> 172.21.2.117:0/1734595635 conn(0x7f24b80cf060 msgr2=0x7f24b80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.184 7f24bfc3f700 1 -- 172.21.2.117:0/1734595635 shutdown_connections 2021-02-10T11:45:34.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.184 7f24bfc3f700 1 -- 172.21.2.117:0/1734595635 wait complete. 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 Processor -- start 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 -- start start 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b8085ec0 0x7f24b80862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24b80a2040 0x7f24b80a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80a67e0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80a7120 con 0x7f24b80a2040 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bfc3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b80a2a60 con 0x7f24b8085ec0 2021-02-10T11:45:34.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bda5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bda5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51868/0 (socket says 172.21.2.117:51868) 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.199 7f24bda5b700 1 -- 172.21.2.117:0/2375589444 learned_addr learned my addr 172.21.2.117:0/2375589444 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 -- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b8085ec0 msgr2=0x7f24b80862c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b8085ec0 0x7f24b80862c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 -- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24b80a2040 msgr2=0x7f24b80a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24b80a2040 0x7f24b80a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 -- 172.21.2.117:0/2375589444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b81a2dc0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bda5b700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7f24ac006f60 tx=0x7f24ac00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ac006bb0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac00bd60 con 0x7f24b80a6ce0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ac0022b0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b81a2dc0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.200 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b81a2dc0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.201 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24ac000c90 con 0x7f24b80a6ce0 2021-02-10T11:45:34.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.201 7f24aa7fc700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 0x7f24a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.201 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f24ac032750 con 0x7f24b80a6ce0 2021-02-10T11:45:34.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.201 7f24bd25a700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 0x7f24a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.202 7f24bd25a700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 0x7f24a001abc0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.231 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b80daff0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.233 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24ac030db0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.361 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f24b80daff0 con 0x7f24b80a6ce0 2021-02-10T11:45:34.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.362 7f24aa7fc700 1 -- 172.21.2.117:0/2375589444 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v83) v1 ==== 104+0+10 (secure 0 0 0) 0x7f24ac000c90 con 0x7f24b80a6ce0 2021-02-10T11:45:34.355 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:34.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 msgr2=0x7f24a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 0x7f24a001abc0 crc :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 msgr2=0x7f24b80dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 rx=0x7f24ac006f60 tx=0x7f24ac00b970).stop 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 shutdown_connections 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24b80a2040 0x7f24b80a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24b8085ec0 0x7f24b80862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24a00186c0 0x7f24a001abc0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 --2- 172.21.2.117:0/2375589444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24b80a6ce0 0x7f24b80dc420 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.364 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 >> 172.21.2.117:0/2375589444 conn(0x7f24b80b6540 msgr2=0x7f24b808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.365 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 shutdown_connections 2021-02-10T11:45:34.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.365 7f24bfc3f700 1 -- 172.21.2.117:0/2375589444 wait complete. 2021-02-10T11:45:34.365 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.788 7fd87f286700 1 Processor -- start 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.788 7fd87f286700 1 -- start start 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87f286700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 0x7fd8780eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87f286700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8780ef3f0 0x7fd8781b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87f286700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8780ede60 con 0x7fd8780eb040 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87f286700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd878088020 con 0x7fd8780ef3f0 2021-02-10T11:45:34.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87f286700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780d1d30 con 0x7fd8780d18f0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8780ef3f0 0x7fd8781b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd8780ef3f0 0x7fd8781b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51874/0 (socket says 172.21.2.117:51874) 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 -- 172.21.2.117:0/2087943776 learned_addr learned my addr 172.21.2.117:0/2087943776 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d0a2700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 0x7fd8780eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 -- 172.21.2.117:0/2087943776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 msgr2=0x7fd8780eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 0x7fd8780eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 -- 172.21.2.117:0/2087943776 >> v1:172.21.2.117:6789/0 conn(0x7fd8780eb040 legacy=0x7fd8780eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 -- 172.21.2.117:0/2087943776 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8781b8fd0 con 0x7fd8780ef3f0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d0a2700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 0x7fd8780eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.789 7fd87d8a3700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780ef3f0 0x7fd8781b79a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7fd870006f00 tx=0x7fd870009f10).ready entity=mon.0 client_cookie=15e5a8dad9cb386b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd86bfff700 1 -- 172.21.2.117:0/2087943776 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd87000b3b0 con 0x7fd8780ef3f0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd86bfff700 1 -- 172.21.2.117:0/2087943776 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87000b3b0 con 0x7fd8780ef3f0 2021-02-10T11:45:34.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd86bfff700 1 -- 172.21.2.117:0/2087943776 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd87000b730 con 0x7fd8780ef3f0 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 -- 172.21.2.117:0/2087943776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780ef3f0 msgr2=0x7fd8781b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780ef3f0 0x7fd8781b79a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 rx=0x7fd870006f00 tx=0x7fd870009f10).stop 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 -- 172.21.2.117:0/2087943776 shutdown_connections 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780ef3f0 0x7fd8781b79a0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 --2- 172.21.2.117:0/2087943776 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780d18f0 0x7fd8780eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.790 7fd87f286700 1 -- 172.21.2.117:0/2087943776 >> 172.21.2.117:0/2087943776 conn(0x7fd8780cf060 msgr2=0x7fd8780cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.791 7fd87f286700 1 -- 172.21.2.117:0/2087943776 shutdown_connections 2021-02-10T11:45:34.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.791 7fd87f286700 1 -- 172.21.2.117:0/2087943776 wait complete. 2021-02-10T11:45:34.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.805 7fd87f286700 1 Processor -- start 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.805 7fd87f286700 1 -- start start 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8780b4c10 0x7fd8780aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780b5010 0x7fd8780ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780b4320 con 0x7fd8780b3e10 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87808a220 con 0x7fd8780b4c10 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780abbc0 con 0x7fd8780b5010 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51878/0 (socket says 172.21.2.117:51878) 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 -- 172.21.2.117:0/596862471 learned_addr learned my addr 172.21.2.117:0/596862471 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 -- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780b5010 msgr2=0x7fd8780ab5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:45:34.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780b5010 0x7fd8780ab5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 -- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8780b4c10 msgr2=0x7fd8780aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8780b4c10 0x7fd8780aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 -- 172.21.2.117:0/596862471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780abde0 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87d0a2700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7fd86c00b580 tx=0x7fd86c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86c00bd10 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c000c90 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd86c0022b0 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.806 7fd87f286700 1 -- 172.21.2.117:0/596862471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8780abde0 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.807 7fd87f286700 1 -- 172.21.2.117:0/596862471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8780abde0 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.807 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd86c00a000 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.807 7fd869ffb700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.807 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd86c032530 con 0x7fd8780b3e10 2021-02-10T11:45:34.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.807 7fd87c8a1700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:34.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.808 7fd87c8a1700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 0x7fd85801abc0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:34.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.839 7fd87f286700 1 -- 172.21.2.117:0/596862471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87808a220 con 0x7fd8780b3e10 2021-02-10T11:45:34.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.841 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd86c006bb0 con 0x7fd8780b3e10 2021-02-10T11:45:34.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.968 7fd87f286700 1 -- 172.21.2.117:0/596862471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fd87808a220 con 0x7fd8780b3e10 2021-02-10T11:45:34.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.969 7fd869ffb700 1 -- 172.21.2.117:0/596862471 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v83) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd86c00bd10 con 0x7fd8780b3e10 2021-02-10T11:45:34.962 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:34.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.970 7fd87f286700 1 -- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 msgr2=0x7fd85801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.970 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 0x7fd85801abc0 crc :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.970 7fd87f286700 1 -- 172.21.2.117:0/596862471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 msgr2=0x7fd8780b45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.970 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 rx=0x7fd86c00b580 tx=0x7fd86c00b920).stop 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 -- 172.21.2.117:0/596862471 shutdown_connections 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd8780b5010 0x7fd8780ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd8780b4c10 0x7fd8780aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8580186c0 0x7fd85801abc0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 --2- 172.21.2.117:0/596862471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd8780b3e10 0x7fd8780b45f0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:34.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 -- 172.21.2.117:0/596862471 >> 172.21.2.117:0/596862471 conn(0x7fd8780b6c70 msgr2=0x7fd8780970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:34.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 -- 172.21.2.117:0/596862471 shutdown_connections 2021-02-10T11:45:34.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:34.971 7fd87f286700 1 -- 172.21.2.117:0/596862471 wait complete. 2021-02-10T11:45:34.972 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:45:34.973 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:35.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 Processor -- start 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 -- start start 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09381b6960 0x7f09381b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380ede60 con 0x7f09380ef3f0 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09380eb9c0 con 0x7f09380d18e0 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.391 7f093e267700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09380d1ce0 con 0x7f09381b6960 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f09377fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09380ef3f0 0x7f09380eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52140/0 (socket says 172.21.2.117:52140) 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f09377fe700 1 -- 172.21.2.117:0/2650110914 learned_addr learned my addr 172.21.2.117:0/2650110914 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f093c884700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09381b6960 0x7f09381b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347502655 0 0) 0x7f092c0009b0 con 0x7f09380ef3f0 2021-02-10T11:45:35.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920002fe0 con 0x7f09380ef3f0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068678743 0 0) 0x7f0928000c60 con 0x7f09380d18e0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0920003210 con 0x7f09380d18e0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 555913949 0 0) 0x7f092c0010b0 con 0x7f09380ef3f0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09381b6960 msgr2=0x7f09381b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09381b6960 0x7f09381b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 >> v1:172.21.2.125:6789/0 conn(0x7f09380d18e0 legacy=0x7f09380ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09381bb3b0 con 0x7f09380ef3f0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f092c0015b0 con 0x7f09380ef3f0 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 >> v1:172.21.2.117:6789/0 conn(0x7f09380ef3f0 legacy=0x7f09380eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09200037e0 0x7f0920003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0920006860 0x7f0920008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09200095a0 0x7f092000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0920050380 con 0x7f0920006860 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0920050560 con 0x7f09200095a0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0920050780 con 0x7f09200037e0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f09377fe700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0920006860 0x7f0920008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.392 7f093c884700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09381b6960 0x7f09381b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 -- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09200037e0 msgr2=0x7f0920003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f0937fff700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09200037e0 0x7f0920003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09200037e0 0x7f0920003c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f093c884700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09200095a0 0x7f092000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 -- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09200095a0 msgr2=0x7f092000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09200095a0 0x7f092000bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 -- 172.21.2.117:0/2650110914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920050a20 con 0x7f0920006860 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f09377fe700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0920006860 0x7f0920008d00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f092c001270 tx=0x7f092c007cf0).ready entity=mon.0 client_cookie=e7acd63869999c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f092c0009b0 con 0x7f0920006860 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c00a520 con 0x7f0920006860 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f0936ffd700 1 -- 172.21.2.117:0/2650110914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f092c00a9b0 con 0x7f0920006860 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f093e267700 1 -- 172.21.2.117:0/2650110914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0920006860 msgr2=0x7f0920008d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f093e267700 1 --2- 172.21.2.117:0/2650110914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0920006860 0x7f0920008d00 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 rx=0x7f092c001270 tx=0x7f092c007cf0).stop 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.393 7f093c884700 1 -- 172.21.2.117:0/2650110914 reap_dead start 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.394 7f093e267700 1 -- 172.21.2.117:0/2650110914 shutdown_connections 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.394 7f093e267700 1 -- 172.21.2.117:0/2650110914 >> 172.21.2.117:0/2650110914 conn(0x7f09380ced60 msgr2=0x7f09380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.394 7f093e267700 1 -- 172.21.2.117:0/2650110914 shutdown_connections 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.394 7f093e267700 1 -- 172.21.2.117:0/2650110914 wait complete. 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 Processor -- start 2021-02-10T11:45:35.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 -- start start 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 0x7f093808c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09380b5da0 0x7f09381a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 0x7f09380b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380b61a0 con 0x7f09381a3480 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093808c8b0 con 0x7f09380b59a0 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093e267700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09380ba1c0 con 0x7f09380b5da0 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093c884700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 0x7f09380b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093c884700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 0x7f09380b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51900/0 (socket says 172.21.2.117:51900) 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f093c884700 1 -- 172.21.2.117:0/3203856752 learned_addr learned my addr 172.21.2.117:0/3203856752 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:35.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f0937fff700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 0x7f093808c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.409 7f09377fe700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09380b5da0 0x7f09381a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 -- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09380b5da0 msgr2=0x7f09381a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09380b5da0 0x7f09381a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 -- 172.21.2.117:0/3203856752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 msgr2=0x7f09380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 0x7f09380b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 -- 172.21.2.117:0/3203856752 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09381a3880 con 0x7f09380b59a0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0937fff700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 0x7f093808c290 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f0928009540 tx=0x7f092800b890).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f092800bc80 con 0x7f09380b59a0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0928000c10 con 0x7f09380b59a0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f093e267700 1 -- 172.21.2.117:0/3203856752 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381a3880 con 0x7f09380b59a0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0928006550 con 0x7f09380b59a0 2021-02-10T11:45:35.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.410 7f093e267700 1 -- 172.21.2.117:0/3203856752 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09381a3880 con 0x7f09380b59a0 2021-02-10T11:45:35.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.411 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0928006030 con 0x7f09380b59a0 2021-02-10T11:45:35.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.411 7f0934ff9700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 0x7f091801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:35.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.411 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0928032470 con 0x7f09380b59a0 2021-02-10T11:45:35.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.411 7f09377fe700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 0x7f091801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:35.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.412 7f09377fe700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 0x7f091801abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:35.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.441 7f093e267700 1 -- 172.21.2.117:0/3203856752 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09380ede60 con 0x7f09380b59a0 2021-02-10T11:45:35.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.444 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0928025ed0 con 0x7f09380b59a0 2021-02-10T11:45:35.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.574 7f093e267700 1 -- 172.21.2.117:0/3203856752 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09380ede60 con 0x7f09380b59a0 2021-02-10T11:45:35.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.575 7f0934ff9700 1 -- 172.21.2.117:0/3203856752 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f0928025ed0 con 0x7f09380b59a0 2021-02-10T11:45:35.568 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:35.568 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:35.568 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 61 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 -- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 msgr2=0x7f091801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 0x7f091801abc0 crc :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 -- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 msgr2=0x7f093808c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 0x7f093808c290 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f0928009540 tx=0x7f092800b890).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 -- 172.21.2.117:0/3203856752 shutdown_connections 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09180186c0 0x7f091801abc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09381a3480 0x7f09380b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09380b5da0 0x7f09381a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 --2- 172.21.2.117:0/3203856752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09380b59a0 0x7f093808c290 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.576 7f093e267700 1 -- 172.21.2.117:0/3203856752 >> 172.21.2.117:0/3203856752 conn(0x7f09380bb5e0 msgr2=0x7f0938090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:35.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.577 7f093e267700 1 -- 172.21.2.117:0/3203856752 shutdown_connections 2021-02-10T11:45:35.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:35.577 7f093e267700 1 -- 172.21.2.117:0/3203856752 wait complete. 2021-02-10T11:45:35.780 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:25.613265', 'modified': '2021-02-10 11:45:27.622343', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7144}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7144': {'gid': 7144, 'name': 'b', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1925085474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1925085474}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1925085474}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:35.780 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:35.780 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:36.176 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:45:36.176 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:45:36.176 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7144, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.568257651, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 61, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 83, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.201768991000002 2021-02-10T11:45:36.177 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:45:36.180 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7144, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.568257651, "mdsmap_epoch": 61, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 16.201768991000002 } 2021-02-10T11:45:36.180 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:45:36.181 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:45:36.181 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:45:36.181 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:36.181 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:36.517 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:45:36.860 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:36.860 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:37.204 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:37.204 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:37.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:37.205 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:45:37.539 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:45:37.540 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:37.542 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:45:37.887 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:37.887 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:37.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[145505]: starting ceph client 2021-02-10T11:45:37.952 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:45:37.959 7f54d55e0f80 -1 init, newargv = 0x56177f250f60 newargc=9 2021-02-10T11:45:37.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[145505]: starting fuse 2021-02-10T11:45:38.229 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:38.229 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:38.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:38.230 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:45:38.566 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:45:38.566 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:45:38.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:45:38.567 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:45:38.568 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:45:38.941 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.145505.asok 2021-02-10T11:45:38.944 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.145505.asok 2021-02-10T11:45:38.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:38.944 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.145505.asok status 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:45:39.333 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "145505", 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9471, 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9471 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:39.334 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4192698106 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4192698106 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9471 172.21.2.117:0/4192698106", 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/4192698106", 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 61, 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 83, 2021-02-10T11:45:39.335 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:45:39.336 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:45:39.336 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:45:39.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:39.337 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:39.337 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:39.630 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:39.634 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:45:39.673 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:39.673 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:39.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:39.673 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:39.978 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:45:40.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.394 7f6a9b5bb700 1 Processor -- start 2021-02-10T11:45:40.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.394 7f6a9b5bb700 1 -- start start 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940d18e0 0x7f6a940ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6a940ef3f0 0x7f6a941b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940ede60 con 0x7f6a940eb000 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a94088020 con 0x7f6a940ef3f0 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940d1ce0 con 0x7f6a940d18e0 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6a940ef3f0 0x7f6a941b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a98bd6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6a940eb000 0x7f6a940eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52176/0 (socket says 172.21.2.117:52176) 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a98bd6700 1 -- 172.21.2.117:0/3753803792 learned_addr learned my addr 172.21.2.117:0/3753803792 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a993d7700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940d18e0 0x7f6a940ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 -- 172.21.2.117:0/3753803792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940d18e0 msgr2=0x7f6a940ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:40.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940d18e0 0x7f6a940ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 -- 172.21.2.117:0/3753803792 >> v1:172.21.2.117:6789/0 conn(0x7f6a940eb000 legacy=0x7f6a940eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 -- 172.21.2.117:0/3753803792 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a941b8f90 con 0x7f6a940ef3f0 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a99bd8700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ef3f0 0x7f6a941b7960 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f6a8c006e30 tx=0x7f6a8c009e60).ready entity=mon.0 client_cookie=acf24ae37a5eceda server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a87fff700 1 -- 172.21.2.117:0/3753803792 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a8c006a50 con 0x7f6a940ef3f0 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a87fff700 1 -- 172.21.2.117:0/3753803792 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c00b300 con 0x7f6a940ef3f0 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a87fff700 1 -- 172.21.2.117:0/3753803792 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a8c00b820 con 0x7f6a940ef3f0 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 -- 172.21.2.117:0/3753803792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ef3f0 msgr2=0x7f6a941b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.395 7f6a9b5bb700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ef3f0 0x7f6a941b7960 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 rx=0x7f6a8c006e30 tx=0x7f6a8c009e60).stop 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 -- 172.21.2.117:0/3753803792 shutdown_connections 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ef3f0 0x7f6a941b7960 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 --2- 172.21.2.117:0/3753803792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940d18e0 0x7f6a940ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 -- 172.21.2.117:0/3753803792 >> 172.21.2.117:0/3753803792 conn(0x7f6a940ced60 msgr2=0x7f6a940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 -- 172.21.2.117:0/3753803792 shutdown_connections 2021-02-10T11:45:40.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.396 7f6a9b5bb700 1 -- 172.21.2.117:0/3753803792 wait complete. 2021-02-10T11:45:40.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 Processor -- start 2021-02-10T11:45:40.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 -- start start 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 0x7f6a94095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940960c0 0x7f6a940e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940964c0 con 0x7f6a940ac7b0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940e7bd0 con 0x7f6a9408a8a0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a9b5bb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a940e7df0 con 0x7f6a940960c0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a993d7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a993d7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51932/0 (socket says 172.21.2.117:51932) 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.411 7f6a993d7700 1 -- 172.21.2.117:0/1131486145 learned_addr learned my addr 172.21.2.117:0/1131486145 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a98bd6700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 0x7f6a94095aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a99bd8700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940960c0 0x7f6a940e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 -- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940960c0 msgr2=0x7f6a940e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940960c0 0x7f6a940e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 -- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 msgr2=0x7f6a94095aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 0x7f6a94095aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 -- 172.21.2.117:0/1131486145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a940e8010 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a98bd6700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 0x7f6a94095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a993d7700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f6a88006f00 tx=0x7f6a8800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a8800bd30 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88002030 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a940e8010 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.412 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a88002510 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.413 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a940e8010 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.413 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a88000c90 con 0x7f6a940ac7b0 2021-02-10T11:45:40.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.413 7f6a85ffb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 0x7f6a7401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:40.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.413 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6a88032780 con 0x7f6a940ac7b0 2021-02-10T11:45:40.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.414 7f6a98bd6700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 0x7f6a7401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:40.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.414 7f6a98bd6700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 0x7f6a7401ac10 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:40.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.445 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a94087210 con 0x7f6a940ac7b0 2021-02-10T11:45:40.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.447 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a8801f4f0 con 0x7f6a940ac7b0 2021-02-10T11:45:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.570 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a94087210 con 0x7f6a940ac7b0 2021-02-10T11:45:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.571 7f6a85ffb700 1 -- 172.21.2.117:0/1131486145 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f6a88000c90 con 0x7f6a940ac7b0 2021-02-10T11:45:40.565 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:40.565 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":83,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:45:26.614628","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:45:23.589151","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":17,"pool_name":"cephfs_data","create_time":"2021-02-10 11:45:24.659131","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:45:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 msgr2=0x7f6a7401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 0x7f6a7401ac10 crc :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 msgr2=0x7f6a9408a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f6a88006f00 tx=0x7f6a8800b940).stop 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 shutdown_connections 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a940960c0 0x7f6a940e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a9408a8a0 0x7f6a94095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a74018730 0x7f6a7401ac10 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 --2- 172.21.2.117:0/1131486145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a940ac7b0 0x7f6a9408a280 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 >> 172.21.2.117:0/1131486145 conn(0x7f6a9419f610 msgr2=0x7f6a940e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 shutdown_connections 2021-02-10T11:45:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.573 7f6a9b5bb700 1 -- 172.21.2.117:0/1131486145 wait complete. 2021-02-10T11:45:40.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:45:23.589151', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:45:24.659131', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:45:40.575 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:45:41.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.997 7f12b338c700 1 Processor -- start 2021-02-10T11:45:41.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.997 7f12b338c700 1 -- start start 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.998 7f12b338c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f12ac0d18f0 0x7f12ac0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.998 7f12b338c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac0ef3f0 0x7f12ac1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.998 7f12b338c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ac0ef7f0 con 0x7f12ac0eb040 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.998 7f12b338c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac0eded0 con 0x7f12ac0d18f0 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.998 7f12b338c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0d1cf0 con 0x7f12ac0ef3f0 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b19a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac0ef3f0 0x7f12ac1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b11a8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f12ac0d18f0 0x7f12ac0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b09a7700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f12ac0eb040 0x7f12ac0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45116/0 (socket says 172.21.2.117:45116) 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b09a7700 1 -- 172.21.2.117:0/458069096 learned_addr learned my addr 172.21.2.117:0/458069096 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b11a8700 1 -- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac0ef3f0 msgr2=0x7f12ac1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b11a8700 1 --2- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac0ef3f0 0x7f12ac1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b11a8700 1 -- 172.21.2.117:0/458069096 >> v1:172.21.2.125:6789/0 conn(0x7f12ac0eb040 legacy=0x7f12ac0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:40.999 7f12b11a8700 1 -- 172.21.2.117:0/458069096 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac1b8fd0 con 0x7f12ac0d18f0 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.000 7f12b11a8700 1 --2- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac0d18f0 0x7f12ac0eaa20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f12a0006650 tx=0x7f12a0009a80).ready entity=mon.1 client_cookie=22d9a0be702a57f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.000 7f129ffff700 1 -- 172.21.2.117:0/458069096 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a000c0c0 con 0x7f12ac0d18f0 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.000 7f129ffff700 1 -- 172.21.2.117:0/458069096 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a000c340 con 0x7f12ac0d18f0 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.000 7f129ffff700 1 -- 172.21.2.117:0/458069096 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a000c860 con 0x7f12ac0d18f0 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 -- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac0d18f0 msgr2=0x7f12ac0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 --2- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac0d18f0 0x7f12ac0eaa20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f12a0006650 tx=0x7f12a0009a80).stop 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 -- 172.21.2.117:0/458069096 shutdown_connections 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 --2- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac0ef3f0 0x7f12ac1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 --2- 172.21.2.117:0/458069096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac0d18f0 0x7f12ac0eaa20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 -- 172.21.2.117:0/458069096 >> 172.21.2.117:0/458069096 conn(0x7f12ac0cf060 msgr2=0x7f12ac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 -- 172.21.2.117:0/458069096 shutdown_connections 2021-02-10T11:45:41.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.001 7f12b338c700 1 -- 172.21.2.117:0/458069096 wait complete. 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 Processor -- start 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 -- start start 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac07c7c0 0x7f12ac1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 0x7f12ac0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac07cca0 con 0x7f12ac0a13a0 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac18fb70 con 0x7f12ac07c7c0 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b338c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0dc6d0 con 0x7f12ac1a3930 2021-02-10T11:45:41.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b19a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b19a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51944/0 (socket says 172.21.2.117:51944) 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.016 7f12b19a9700 1 -- 172.21.2.117:0/198945031 learned_addr learned my addr 172.21.2.117:0/198945031 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b09a7700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 0x7f12ac0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 -- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 msgr2=0x7f12ac0a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 0x7f12ac0a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 -- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac07c7c0 msgr2=0x7f12ac1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac07c7c0 0x7f12ac1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 -- 172.21.2.117:0/198945031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac07d220 con 0x7f12ac0a13a0 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b09a7700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 0x7f12ac0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b19a9700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7f12a400b970 tx=0x7f12a400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a4000f90 con 0x7f12ac0a13a0 2021-02-10T11:45:41.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4003030 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12a4001400 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b338c700 1 -- 172.21.2.117:0/198945031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac07d220 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.017 7f12b338c700 1 -- 172.21.2.117:0/198945031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac07d7b0 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.018 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f12a4006aa0 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.018 7f129dffb700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 0x7f128c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.018 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f12a4032a10 con 0x7f12ac0a13a0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.018 7f12b11a8700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 0x7f128c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.018 7f12b11a8700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 0x7f128c01abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.048 7f12b338c700 1 -- 172.21.2.117:0/198945031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac18fb20 con 0x7f12ac0a13a0 2021-02-10T11:45:41.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.050 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f12a4000ec0 con 0x7f12ac0a13a0 2021-02-10T11:45:41.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.179 7f12b338c700 1 -- 172.21.2.117:0/198945031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f12ac18fb20 con 0x7f12ac0a13a0 2021-02-10T11:45:41.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.179 7f129dffb700 1 -- 172.21.2.117:0/198945031 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v83) v1 ==== 92+0+10 (secure 0 0 0) 0x7f12a4006aa0 con 0x7f12ac0a13a0 2021-02-10T11:45:41.173 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:41.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 -- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 msgr2=0x7f128c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 0x7f128c01abc0 crc :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 -- 172.21.2.117:0/198945031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 msgr2=0x7f12ac0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 rx=0x7f12a400b970 tx=0x7f12a400bd10).stop 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 -- 172.21.2.117:0/198945031 shutdown_connections 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f128c0186c0 0x7f128c01abc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12ac0a13a0 0x7f12ac0a17a0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12ac1a3930 0x7f12ac0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 --2- 172.21.2.117:0/198945031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12ac07c7c0 0x7f12ac1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.181 7f12b338c700 1 -- 172.21.2.117:0/198945031 >> 172.21.2.117:0/198945031 conn(0x7f12ac087090 msgr2=0x7f12ac08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:41.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.182 7f12b338c700 1 -- 172.21.2.117:0/198945031 shutdown_connections 2021-02-10T11:45:41.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.182 7f12b338c700 1 -- 172.21.2.117:0/198945031 wait complete. 2021-02-10T11:45:41.183 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:45:41.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 Processor -- start 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 -- start start 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 0x7f95940eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95940eb830 0x7f95941b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95940ef7f0 con 0x7f95940d18f0 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940ede60 con 0x7f95940eb830 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f959b9b4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940d1cf0 con 0x7f95940ef3f0 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.604 7f9599fd1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95940eb830 0x7f95941b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95940eb830 0x7f95941b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51954/0 (socket says 172.21.2.117:51954) 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 -- 172.21.2.117:0/3729474101 learned_addr learned my addr 172.21.2.117:0/3729474101 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9598fcf700 1 --2- 172.21.2.117:0/3729474101 >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 0x7f95940eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 -- 172.21.2.117:0/3729474101 >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 msgr2=0x7f95940eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:41.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 --2- 172.21.2.117:0/3729474101 >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 0x7f95940eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 -- 172.21.2.117:0/3729474101 >> v1:172.21.2.125:6789/0 conn(0x7f95940d18f0 legacy=0x7f95940eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 -- 172.21.2.117:0/3729474101 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95941b8fd0 con 0x7f95940eb830 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9598fcf700 1 --2- 172.21.2.117:0/3729474101 >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 0x7f95940eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9599fd1700 1 --2- 172.21.2.117:0/3729474101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940eb830 0x7f95941b79a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f958c006f00 tx=0x7f958c009f10).ready entity=mon.0 client_cookie=6c6017617fbf5e18 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9587fff700 1 -- 172.21.2.117:0/3729474101 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f958c00b3b0 con 0x7f95940eb830 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9587fff700 1 -- 172.21.2.117:0/3729474101 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c00b670 con 0x7f95940eb830 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f9587fff700 1 -- 172.21.2.117:0/3729474101 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f958c00bb90 con 0x7f95940eb830 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f959b9b4700 1 -- 172.21.2.117:0/3729474101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940eb830 msgr2=0x7f95941b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.605 7f959b9b4700 1 --2- 172.21.2.117:0/3729474101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940eb830 0x7f95941b79a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 rx=0x7f958c006f00 tx=0x7f958c009f10).stop 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 -- 172.21.2.117:0/3729474101 shutdown_connections 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 --2- 172.21.2.117:0/3729474101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940eb830 0x7f95941b79a0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 --2- 172.21.2.117:0/3729474101 >> v2:172.21.2.125:3300/0 conn(0x7f95940ef3f0 0x7f95940eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 -- 172.21.2.117:0/3729474101 >> 172.21.2.117:0/3729474101 conn(0x7f95940cf060 msgr2=0x7f95940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:41.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 -- 172.21.2.117:0/3729474101 shutdown_connections 2021-02-10T11:45:41.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.606 7f959b9b4700 1 -- 172.21.2.117:0/3729474101 wait complete. 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 Processor -- start 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 -- start start 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 0x7f95941a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 0x7f95940a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940a1820 con 0x7f95940a65a0 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940a69a0 con 0x7f95941a4860 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.621 7f959b9b4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959409dc80 con 0x7f959409e160 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51956/0 (socket says 172.21.2.117:51956) 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 -- 172.21.2.117:0/3086623198 learned_addr learned my addr 172.21.2.117:0/3086623198 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9598fcf700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 0x7f95940a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f95997d0700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 0x7f95941a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 -- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 msgr2=0x7f95941a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 0x7f95941a4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 -- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 msgr2=0x7f95940a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 0x7f95940a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 -- 172.21.2.117:0/3086623198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940a23c0 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f95997d0700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 0x7f95941a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9598fcf700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 0x7f95940a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.622 7f9599fd1700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f958c009390 tx=0x7f958c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f958c003030 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c0068e0 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95940a23c0 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95940a2950 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f958c00c570 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f958c001270 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 0x7f957401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.623 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f958c032c50 con 0x7f95940a65a0 2021-02-10T11:45:41.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.624 7f95997d0700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 0x7f957401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:41.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.625 7f95997d0700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 0x7f957401abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:41.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.654 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95940ede60 con 0x7f95940a65a0 2021-02-10T11:45:41.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.657 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f958c009450 con 0x7f95940a65a0 2021-02-10T11:45:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.782 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f95940ede60 con 0x7f95940a65a0 2021-02-10T11:45:41.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.782 7f9585ffb700 1 -- 172.21.2.117:0/3086623198 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v83) v1 ==== 104+0+10 (secure 0 0 0) 0x7f958c001270 con 0x7f95940a65a0 2021-02-10T11:45:41.777 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.784 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 msgr2=0x7f957401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.784 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 0x7f957401abc0 crc :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.784 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 msgr2=0x7f95940a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.784 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 rx=0x7f958c009390 tx=0x7f958c00bd40).stop 2021-02-10T11:45:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 shutdown_connections 2021-02-10T11:45:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95740186c0 0x7f957401abc0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95940a65a0 0x7f95940a1da0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95941a4860 0x7f95940a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 --2- 172.21.2.117:0/3086623198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f959409e160 0x7f95941a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 >> 172.21.2.117:0/3086623198 conn(0x7f959409e660 msgr2=0x7f95940bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 shutdown_connections 2021-02-10T11:45:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:41.785 7f959b9b4700 1 -- 172.21.2.117:0/3086623198 wait complete. 2021-02-10T11:45:41.787 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:45:42.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.206 7f79e4826700 1 Processor -- start 2021-02-10T11:45:42.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.206 7f79e4826700 1 -- start start 2021-02-10T11:45:42.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f79e00ef3f0 0x7f79e00eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e00eb830 0x7f79e01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e00ef7f0 con 0x7f79e00d18f0 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e00ede60 con 0x7f79e00eb830 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e00d1cf0 con 0x7f79e00ef3f0 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e00eb830 0x7f79e01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f79e00eb830 0x7f79e01b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51968/0 (socket says 172.21.2.117:51968) 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 -- 172.21.2.117:0/1868817201 learned_addr learned my addr 172.21.2.117:0/1868817201 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 -- 172.21.2.117:0/1868817201 >> v2:172.21.2.125:3300/0 conn(0x7f79e00ef3f0 msgr2=0x7f79e00eb210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 --2- 172.21.2.117:0/1868817201 >> v2:172.21.2.125:3300/0 conn(0x7f79e00ef3f0 0x7f79e00eb210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 -- 172.21.2.117:0/1868817201 >> v1:172.21.2.125:6789/0 conn(0x7f79e00d18f0 legacy=0x7f79e00eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 -- 172.21.2.117:0/1868817201 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e01b8fd0 con 0x7f79e00eb830 2021-02-10T11:45:42.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dee1d700 1 --2- 172.21.2.117:0/1868817201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00eb830 0x7f79e01b79a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f79d8006f00 tx=0x7f79d8009f10).ready entity=mon.0 client_cookie=316eebd40cbe5d49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dd61a700 1 -- 172.21.2.117:0/1868817201 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d800b3b0 con 0x7f79e00eb830 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dd61a700 1 -- 172.21.2.117:0/1868817201 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d800b630 con 0x7f79e00eb830 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79dd61a700 1 -- 172.21.2.117:0/1868817201 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d800bb50 con 0x7f79e00eb830 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 -- 172.21.2.117:0/1868817201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00eb830 msgr2=0x7f79e01b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.207 7f79e4826700 1 --2- 172.21.2.117:0/1868817201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00eb830 0x7f79e01b79a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 rx=0x7f79d8006f00 tx=0x7f79d8009f10).stop 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 -- 172.21.2.117:0/1868817201 shutdown_connections 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 --2- 172.21.2.117:0/1868817201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00eb830 0x7f79e01b79a0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 --2- 172.21.2.117:0/1868817201 >> v2:172.21.2.125:3300/0 conn(0x7f79e00ef3f0 0x7f79e00eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 -- 172.21.2.117:0/1868817201 >> 172.21.2.117:0/1868817201 conn(0x7f79e00cf060 msgr2=0x7f79e00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:42.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 -- 172.21.2.117:0/1868817201 shutdown_connections 2021-02-10T11:45:42.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.208 7f79e4826700 1 -- 172.21.2.117:0/1868817201 wait complete. 2021-02-10T11:45:42.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 Processor -- start 2021-02-10T11:45:42.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 -- start start 2021-02-10T11:45:42.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e00a1930 0x7f79e008a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e00b5d20 0x7f79e00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00ace70 con 0x7f79e00ac450 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00ad090 con 0x7f79e00b5d20 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79e4826700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00b6740 con 0x7f79e00a1930 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79dde1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79dde1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51972/0 (socket says 172.21.2.117:51972) 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79dde1b700 1 -- 172.21.2.117:0/2313785203 learned_addr learned my addr 172.21.2.117:0/2313785203 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.223 7f79de61c700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e00a1930 0x7f79e008a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dee1d700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e00b5d20 0x7f79e00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 -- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e00a1930 msgr2=0x7f79e008a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e00a1930 0x7f79e008a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 -- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e00b5d20 msgr2=0x7f79e00b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e00b5d20 0x7f79e00b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 -- 172.21.2.117:0/2313785203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e00b6960 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79dde1b700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f79d000b560 tx=0x7f79d000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d000bcf0 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0006030 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79d00054d0 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79e4826700 1 -- 172.21.2.117:0/2313785203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e00b6960 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.224 7f79e4826700 1 -- 172.21.2.117:0/2313785203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e00b6960 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.225 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79d0006210 con 0x7f79e00ac450 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.225 7f79d6ffd700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 0x7f79c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.225 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f79d0032700 con 0x7f79e00ac450 2021-02-10T11:45:42.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.225 7f79de61c700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 0x7f79c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.226 7f79de61c700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 0x7f79c401abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.256 7f79e4826700 1 -- 172.21.2.117:0/2313785203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e00f67e0 con 0x7f79e00ac450 2021-02-10T11:45:42.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.259 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f79d0006b80 con 0x7f79e00ac450 2021-02-10T11:45:42.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.385 7f79e4826700 1 -- 172.21.2.117:0/2313785203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f79e00f67e0 con 0x7f79e00ac450 2021-02-10T11:45:42.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.385 7f79d6ffd700 1 -- 172.21.2.117:0/2313785203 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v83) v1 ==== 100+0+10 (secure 0 0 0) 0x7f79d0006b80 con 0x7f79e00ac450 2021-02-10T11:45:42.380 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:45:42.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.387 7f79e4826700 1 -- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 msgr2=0x7f79c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.387 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 0x7f79c401abc0 crc :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.387 7f79e4826700 1 -- 172.21.2.117:0/2313785203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 msgr2=0x7f79e00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.387 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 rx=0x7f79d000b560 tx=0x7f79d000b900).stop 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 -- 172.21.2.117:0/2313785203 shutdown_connections 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79c40186c0 0x7f79c401abc0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f79e00ac450 0x7f79e00ac850 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f79e00b5d20 0x7f79e00b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 --2- 172.21.2.117:0/2313785203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f79e00a1930 0x7f79e008a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 -- 172.21.2.117:0/2313785203 >> 172.21.2.117:0/2313785203 conn(0x7f79e00b6c70 msgr2=0x7f79e00970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:42.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 -- 172.21.2.117:0/2313785203 shutdown_connections 2021-02-10T11:45:42.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.388 7f79e4826700 1 -- 172.21.2.117:0/2313785203 wait complete. 2021-02-10T11:45:42.390 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:45:42.390 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:42.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 Processor -- start 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 -- start start 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fea940e4e40 0x7fea941b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea94085450 0x7fea94086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea94085930 con 0x7fea940e5240 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940e5cc0 con 0x7fea94085450 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea999da700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea94084840 con 0x7fea940e4e40 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea93d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea94085450 0x7fea94086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea93d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fea94085450 0x7fea94086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51978/0 (socket says 172.21.2.117:51978) 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.806 7fea93d75700 1 -- 172.21.2.117:0/1368589467 learned_addr learned my addr 172.21.2.117:0/1368589467 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:42.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93574700 1 --2- 172.21.2.117:0/1368589467 >> v2:172.21.2.125:3300/0 conn(0x7fea940e4e40 0x7fea941b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93d75700 1 -- 172.21.2.117:0/1368589467 >> v2:172.21.2.125:3300/0 conn(0x7fea940e4e40 msgr2=0x7fea941b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93d75700 1 --2- 172.21.2.117:0/1368589467 >> v2:172.21.2.125:3300/0 conn(0x7fea940e4e40 0x7fea941b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93d75700 1 -- 172.21.2.117:0/1368589467 >> v1:172.21.2.125:6789/0 conn(0x7fea940e5240 legacy=0x7fea94084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93d75700 1 -- 172.21.2.117:0/1368589467 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea941b0810 con 0x7fea94085450 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea93d75700 1 --2- 172.21.2.117:0/1368589467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea94085450 0x7fea94086ec0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7fea8c006f00 tx=0x7fea8c009f10).ready entity=mon.0 client_cookie=4349b1c529467d90 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea92572700 1 -- 172.21.2.117:0/1368589467 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8c00b3b0 con 0x7fea94085450 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea92572700 1 -- 172.21.2.117:0/1368589467 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c00b630 con 0x7fea94085450 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea92572700 1 -- 172.21.2.117:0/1368589467 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8c00bb50 con 0x7fea94085450 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea999da700 1 -- 172.21.2.117:0/1368589467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea94085450 msgr2=0x7fea94086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.807 7fea999da700 1 --2- 172.21.2.117:0/1368589467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea94085450 0x7fea94086ec0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 rx=0x7fea8c006f00 tx=0x7fea8c009f10).stop 2021-02-10T11:45:42.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 -- 172.21.2.117:0/1368589467 shutdown_connections 2021-02-10T11:45:42.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 --2- 172.21.2.117:0/1368589467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea94085450 0x7fea94086ec0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 --2- 172.21.2.117:0/1368589467 >> v2:172.21.2.125:3300/0 conn(0x7fea940e4e40 0x7fea941b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 -- 172.21.2.117:0/1368589467 >> 172.21.2.117:0/1368589467 conn(0x7fea940ce4e0 msgr2=0x7fea940e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:42.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 -- 172.21.2.117:0/1368589467 shutdown_connections 2021-02-10T11:45:42.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.808 7fea999da700 1 -- 172.21.2.117:0/1368589467 wait complete. 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 Processor -- start 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 -- start start 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea940b63a0 0x7fea941a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 0x7fea941a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea941a3be0 con 0x7fea9409b510 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea941a3e00 con 0x7fea94110bd0 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea999da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9409bf30 con 0x7fea940b63a0 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea93d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea92d73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 0x7fea941a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea93d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:51984/0 (socket says 172.21.2.117:51984) 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.823 7fea93d75700 1 -- 172.21.2.117:0/2930919830 learned_addr learned my addr 172.21.2.117:0/2930919830 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea93574700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea940b63a0 0x7fea941a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 -- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea940b63a0 msgr2=0x7fea941a2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea940b63a0 0x7fea941a2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 -- 172.21.2.117:0/2930919830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 msgr2=0x7fea9409b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 -- 172.21.2.117:0/2930919830 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9409c150 con 0x7fea94110bd0 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea93d75700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea92d73700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 0x7fea941a35c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fea88006e80 tx=0x7fea8800b8b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8800bca0 con 0x7fea94110bd0 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88006030 con 0x7fea94110bd0 2021-02-10T11:45:42.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fea8800bca0 con 0x7fea94110bd0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea999da700 1 -- 172.21.2.117:0/2930919830 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9409c150 con 0x7fea94110bd0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.824 7fea999da700 1 -- 172.21.2.117:0/2930919830 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9409c150 con 0x7fea94110bd0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.825 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fea88006210 con 0x7fea94110bd0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.825 7fea7bfff700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 0x7fea7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.825 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fea880326a0 con 0x7fea94110bd0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.825 7fea93574700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 0x7fea7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:42.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.826 7fea93574700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 0x7fea7401abc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:42.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.856 7fea999da700 1 -- 172.21.2.117:0/2930919830 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea94085930 con 0x7fea94110bd0 2021-02-10T11:45:42.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.859 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fea88006b50 con 0x7fea94110bd0 2021-02-10T11:45:42.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.986 7fea999da700 1 -- 172.21.2.117:0/2930919830 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea94085930 con 0x7fea94110bd0 2021-02-10T11:45:42.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.988 7fea7bfff700 1 -- 172.21.2.117:0/2930919830 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fea88006030 con 0x7fea94110bd0 2021-02-10T11:45:42.984 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 61 2021-02-10T11:45:42.984 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:42.984 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":61,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 msgr2=0x7fea7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 0x7fea7401abc0 crc :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 msgr2=0x7fea941a35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 0x7fea941a35c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fea88006e80 tx=0x7fea8800b8b0).stop 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 shutdown_connections 2021-02-10T11:45:42.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea74018730 0x7fea7401abc0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fea9409b510 0x7fea9409b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fea94110bd0 0x7fea941a35c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 --2- 172.21.2.117:0/2930919830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fea940b63a0 0x7fea941a2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 >> 172.21.2.117:0/2930919830 conn(0x7fea940b5bd0 msgr2=0x7fea940c39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 shutdown_connections 2021-02-10T11:45:42.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:42.991 7fea999da700 1 -- 172.21.2.117:0/2930919830 wait complete. 2021-02-10T11:45:43.195 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:45:25.613265', 'modified': '2021-02-10 11:45:27.622343', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7144}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7144': {'gid': 7144, 'name': 'b', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1925085474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1925085474}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1925085474}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:45:43.195 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:45:43.195 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7144, 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 17.984655666999998, 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 61, 2021-02-10T11:45:43.593 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 83, 2021-02-10T11:45:43.594 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 83, 2021-02-10T11:45:43.594 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.618168812 2021-02-10T11:45:43.594 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:45:43.597 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7144, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.984655666999998, "mdsmap_epoch": 61, "osdmap_epoch": 83, "osdmap_epoch_barrier": 83, "uptime": 23.618168812 } 2021-02-10T11:45:43.597 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:45:43.597 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:45:43.597 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:45:43.597 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:43.597 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:43.933 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:45:44.278 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:44.278 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:44.622 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:44.622 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:44.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:44.624 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:45:44.958 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:45:44.959 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:45:44.959 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:44.961 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:45:45.306 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:45:45.306 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:45:45.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[146172]: starting ceph client 2021-02-10T11:45:45.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:45:45.379 7f37bcee5f80 -1 init, newargv = 0x5566d4254f30 newargc=9 2021-02-10T11:45:45.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[146172]: starting fuse 2021-02-10T11:45:45.649 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:45:45.650 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:45:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:45:45.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:45.651 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:45:45.986 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:45:45.987 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:45:45.987 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:45:45.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:45.987 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:45:45.987 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:45:45.987 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:45:45.987 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:45:45.988 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:45:45.989 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:45:45.989 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:45:45.989 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:45:45.989 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:45:46.350 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.146172.asok 2021-02-10T11:45:46.353 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.146172.asok 2021-02-10T11:45:46.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:46.353 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.146172.asok status 2021-02-10T11:45:46.742 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:45:46.742 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:45:46.742 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "146172", 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9510, 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:45:46.743 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9510 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 156703483 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 156703483 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:45:46.744 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9510 172.21.2.117:0/156703483", 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/156703483", 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 61, 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 83, 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:45:46.745 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:45:46.746 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:46.747 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:46.747 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:47.083 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:47.083 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:47.084 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:47.084 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:47.430 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:45:47.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.851 7fa36c67c700 1 Processor -- start 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.851 7fa36c67c700 1 -- start start 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36c67c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3640e4e40 0x7fa3641b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36c67c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa364085450 0x7fa3640d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36c67c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa364085930 con 0x7fa3640e5240 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36c67c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3640e5cc0 con 0x7fa364085450 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36c67c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa364084840 con 0x7fa3640e4e40 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa364085450 0x7fa3640d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa364085450 0x7fa3640d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52002/0 (socket says 172.21.2.117:52002) 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 -- 172.21.2.117:0/3556595100 learned_addr learned my addr 172.21.2.117:0/3556595100 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36a498700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3640e4e40 0x7fa3641b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 -- 172.21.2.117:0/3556595100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3640e4e40 msgr2=0x7fa3641b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:47.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3640e4e40 0x7fa3641b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 -- 172.21.2.117:0/3556595100 >> v1:172.21.2.125:6789/0 conn(0x7fa3640e5240 legacy=0x7fa364084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.852 7fa36ac99700 1 -- 172.21.2.117:0/3556595100 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3640d1be0 con 0x7fa364085450 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36ac99700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa364085450 0x7fa3640d05b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7fa35c006f00 tx=0x7fa35c009f10).ready entity=mon.0 client_cookie=ac9abac2124e0051 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa369496700 1 -- 172.21.2.117:0/3556595100 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c00b3b0 con 0x7fa364085450 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa369496700 1 -- 172.21.2.117:0/3556595100 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c00b730 con 0x7fa364085450 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa369496700 1 -- 172.21.2.117:0/3556595100 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c00bb90 con 0x7fa364085450 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 -- 172.21.2.117:0/3556595100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa364085450 msgr2=0x7fa3640d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa364085450 0x7fa3640d05b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 rx=0x7fa35c006f00 tx=0x7fa35c009f10).stop 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 -- 172.21.2.117:0/3556595100 shutdown_connections 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa364085450 0x7fa3640d05b0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 --2- 172.21.2.117:0/3556595100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3640e4e40 0x7fa3641b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 -- 172.21.2.117:0/3556595100 >> 172.21.2.117:0/3556595100 conn(0x7fa3640ce4e0 msgr2=0x7fa3640e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 -- 172.21.2.117:0/3556595100 shutdown_connections 2021-02-10T11:45:47.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.853 7fa36c67c700 1 -- 172.21.2.117:0/3556595100 wait complete. 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 Processor -- start 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 -- start start 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa364194e10 0x7fa3641915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa364191bf0 0x7fa364096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa364195290 con 0x7fa3640973f0 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640b1460 con 0x7fa364194e10 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36c67c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640b1680 con 0x7fa364191bf0 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36ac99700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36ac99700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52010/0 (socket says 172.21.2.117:52010) 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36ac99700 1 -- 172.21.2.117:0/1769643153 learned_addr learned my addr 172.21.2.117:0/1769643153 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa369c97700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa364191bf0 0x7fa364096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.868 7fa36a498700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa364194e10 0x7fa3641915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 -- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa364191bf0 msgr2=0x7fa364096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa364191bf0 0x7fa364096dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 -- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa364194e10 msgr2=0x7fa3641915d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa364194e10 0x7fa3641915d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 -- 172.21.2.117:0/1769643153 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3640b18a0 con 0x7fa3640973f0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36ac99700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7fa35c00b8e0 tx=0x7fa35c00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c003030 con 0x7fa3640973f0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c0068e0 con 0x7fa3640973f0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3640b18a0 con 0x7fa3640973f0 2021-02-10T11:45:47.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3640b1e30 con 0x7fa3640973f0 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.869 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa35c00c4e0 con 0x7fa3640973f0 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.870 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa35c001270 con 0x7fa3640973f0 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.870 7fa356ffd700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 0x7fa34c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.870 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa35c032c70 con 0x7fa3640973f0 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.870 7fa36a498700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 0x7fa34c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:47.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.871 7fa36a498700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 0x7fa34c01ac10 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:47.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.902 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa364085930 con 0x7fa3640973f0 2021-02-10T11:45:47.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:47.904 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa35c009430 con 0x7fa3640973f0 2021-02-10T11:45:48.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.040 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa364085930 con 0x7fa3640973f0 2021-02-10T11:45:48.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.189 7fa356ffd700 1 -- 172.21.2.117:0/1769643153 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v62) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa35c001270 con 0x7fa3640973f0 2021-02-10T11:45:48.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 msgr2=0x7fa34c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 0x7fa34c01ac10 crc :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 msgr2=0x7fa3640977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 rx=0x7fa35c00b8e0 tx=0x7fa35c00bc80).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 shutdown_connections 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa34c018730 0x7fa34c01ac10 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3640973f0 0x7fa3640977f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa364191bf0 0x7fa364096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 --2- 172.21.2.117:0/1769643153 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa364194e10 0x7fa3641915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 >> 172.21.2.117:0/1769643153 conn(0x7fa36418cc70 msgr2=0x7fa364093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 shutdown_connections 2021-02-10T11:45:48.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.207 7fa36c67c700 1 -- 172.21.2.117:0/1769643153 wait complete. 2021-02-10T11:45:48.209 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:45:48.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 Processor -- start 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 -- start start 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1b07f0 0x7f138c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f138c0e7d70 0x7f138c0e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0e77f0 con 0x7f138c0e8790 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f138c0e92f0 con 0x7f138c0e7d70 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.626 7f1391ece700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0e5ce0 con 0x7f138c1b07f0 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138bfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f138c0e8790 0x7f138c0e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45192/0 (socket says 172.21.2.117:45192) 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f138c0e7d70 0x7f138c0e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1b07f0 0x7f138c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138bfff700 1 -- 172.21.2.117:0/2669303426 learned_addr learned my addr 172.21.2.117:0/2669303426 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138a7fc700 1 -- 172.21.2.117:0/2669303426 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296528651 0 0) 0x7f1384001510 con 0x7f138c0e8790 2021-02-10T11:45:48.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138a7fc700 1 -- 172.21.2.117:0/2669303426 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1374002fe0 con 0x7f138c0e8790 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 -- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1b07f0 msgr2=0x7f138c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 --2- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1b07f0 0x7f138c0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 -- 172.21.2.117:0/2669303426 >> v1:172.21.2.125:6789/0 conn(0x7f138c0e8790 legacy=0x7f138c0e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 -- 172.21.2.117:0/2669303426 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0e54b0 con 0x7f138c0e7d70 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.628 7f138affd700 1 --2- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0e7d70 0x7f138c0e8170 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f1380006650 tx=0x7f1380009a80).ready entity=mon.1 client_cookie=525fbe05cdeb5ad4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f138a7fc700 1 -- 172.21.2.117:0/2669303426 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f138000c0c0 con 0x7f138c0e7d70 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f138a7fc700 1 -- 172.21.2.117:0/2669303426 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138000c340 con 0x7f138c0e7d70 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f138a7fc700 1 -- 172.21.2.117:0/2669303426 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f138000c860 con 0x7f138c0e7d70 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 -- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0e7d70 msgr2=0x7f138c0e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 --2- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0e7d70 0x7f138c0e8170 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f1380006650 tx=0x7f1380009a80).stop 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 -- 172.21.2.117:0/2669303426 shutdown_connections 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 --2- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0e7d70 0x7f138c0e8170 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 --2- 172.21.2.117:0/2669303426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1b07f0 0x7f138c0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 -- 172.21.2.117:0/2669303426 >> 172.21.2.117:0/2669303426 conn(0x7f138c0ce500 msgr2=0x7f138c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:48.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 -- 172.21.2.117:0/2669303426 shutdown_connections 2021-02-10T11:45:48.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.629 7f1391ece700 1 -- 172.21.2.117:0/2669303426 wait complete. 2021-02-10T11:45:48.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 Processor -- start 2021-02-10T11:45:48.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 -- start start 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0935b0 0x7f138c0939b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1a6d60 0x7f138c1a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0a4f50 con 0x7f138c18f200 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c094060 con 0x7f138c0935b0 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f1391ece700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1a7780 con 0x7f138c1a6d60 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f138b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f138b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52020/0 (socket says 172.21.2.117:52020) 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.644 7f138b7fe700 1 -- 172.21.2.117:0/320728571 learned_addr learned my addr 172.21.2.117:0/320728571 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 -- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1a6d60 msgr2=0x7f138c1a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1a6d60 0x7f138c1a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 -- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0935b0 msgr2=0x7f138c0939b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0935b0 0x7f138c0939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 -- 172.21.2.117:0/320728571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c1a7960 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f138b7fe700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7f137c006f60 tx=0x7f137c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f137c0009b0 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137c006bb0 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f1391ece700 1 -- 172.21.2.117:0/320728571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c1a7960 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f137c002030 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.645 7f1391ece700 1 -- 172.21.2.117:0/320728571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1a7960 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.646 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f137c000c90 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.646 7f1373fff700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 0x7f136c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.646 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f137c032980 con 0x7f138c18f200 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.646 7f138affd700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 0x7f136c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:48.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.647 7f138affd700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 0x7f136c01ac10 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:48.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.677 7f1391ece700 1 -- 172.21.2.117:0/320728571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c10c290 con 0x7f138c18f200 2021-02-10T11:45:48.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.679 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f137c006340 con 0x7f138c18f200 2021-02-10T11:45:48.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.813 7f1391ece700 1 -- 172.21.2.117:0/320728571 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f138c0e77f0 con 0x7f136c018730 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.814 7f1373fff700 1 -- 172.21.2.117:0/320728571 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f1380006040 con 0x7f136c018730 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:45:48.808 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 msgr2=0x7f136c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 0x7f136c01ac10 crc :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 msgr2=0x7f138c0bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 rx=0x7f137c006f60 tx=0x7f137c00b970).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 shutdown_connections 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f138c1a6d60 0x7f138c1a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f138c0935b0 0x7f138c0939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f136c018730 0x7f136c01ac10 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 --2- 172.21.2.117:0/320728571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f138c18f200 0x7f138c0bb860 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 >> 172.21.2.117:0/320728571 conn(0x7f138c0a7260 msgr2=0x7f138c0bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 shutdown_connections 2021-02-10T11:45:48.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:48.815 7f1391ece700 1 -- 172.21.2.117:0/320728571 wait complete. 2021-02-10T11:45:48.818 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:45:49.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 Processor -- start 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 -- start start 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcbd80d18f0 0x7fcbd80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd80ef7f0 con 0x7fcbd80eb040 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd80ede60 con 0x7fcbd80ef3f0 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.239 7fcbe0435700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd80d1da0 con 0x7fcbd80d18f0 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbde251700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcbd80d18f0 0x7fcbd80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbdda50700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcbd80eb040 0x7fcbd80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52278/0 (socket says 172.21.2.117:52278) 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbdda50700 1 -- 172.21.2.117:0/1455514367 learned_addr learned my addr 172.21.2.117:0/1455514367 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbde251700 1 -- 172.21.2.117:0/1455514367 >> v1:172.21.2.125:6789/0 conn(0x7fcbd80ef3f0 legacy=0x7fcbd81b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbde251700 1 -- 172.21.2.117:0/1455514367 >> v1:172.21.2.117:6789/0 conn(0x7fcbd80eb040 legacy=0x7fcbd80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbde251700 1 -- 172.21.2.117:0/1455514367 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd81bb1c0 con 0x7fcbd80d18f0 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbde251700 1 --2- 172.21.2.117:0/1455514367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80d18f0 0x7fcbd80eaa20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fcbd4006650 tx=0x7fcbd4009a80).ready entity=mon.0 client_cookie=a63daf4d1e8bcaf8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.240 7fcbdd24f700 1 -- 172.21.2.117:0/1455514367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd400c0c0 con 0x7fcbd80d18f0 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbdd24f700 1 -- 172.21.2.117:0/1455514367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd400c340 con 0x7fcbd80d18f0 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbdd24f700 1 -- 172.21.2.117:0/1455514367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd400c860 con 0x7fcbd80d18f0 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 -- 172.21.2.117:0/1455514367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80d18f0 msgr2=0x7fcbd80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 --2- 172.21.2.117:0/1455514367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80d18f0 0x7fcbd80eaa20 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7fcbd4006650 tx=0x7fcbd4009a80).stop 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 -- 172.21.2.117:0/1455514367 shutdown_connections 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 --2- 172.21.2.117:0/1455514367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80d18f0 0x7fcbd80eaa20 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 -- 172.21.2.117:0/1455514367 >> 172.21.2.117:0/1455514367 conn(0x7fcbd80cf060 msgr2=0x7fcbd80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 -- 172.21.2.117:0/1455514367 shutdown_connections 2021-02-10T11:45:49.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.241 7fcbe0435700 1 -- 172.21.2.117:0/1455514367 wait complete. 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 Processor -- start 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 -- start start 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd80e96c0 0x7fcbd80e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd80a66b0 0x7fcbd81c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80df000 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd809f8d0 con 0x7fcbd80e96c0 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbe0435700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd81c5a30 con 0x7fcbd80a66b0 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbdea52700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbdea52700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52032/0 (socket says 172.21.2.117:52032) 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.256 7fcbdea52700 1 -- 172.21.2.117:0/2413146228 learned_addr learned my addr 172.21.2.117:0/2413146228 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:49.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 -- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd80a66b0 msgr2=0x7fcbd81c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd80a66b0 0x7fcbd81c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 -- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd80e96c0 msgr2=0x7fcbd80e8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd80e96c0 0x7fcbd80e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 -- 172.21.2.117:0/2413146228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd81c5c10 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbdea52700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 rx=0x7fcbd0009940 tx=0x7fcbd000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd0000f90 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd000c2f0 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd000c810 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd81c5c10 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.257 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd81c5c10 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.258 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbd0001220 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.258 7fcbceffd700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 0x7fcbc001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:49.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.258 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcbd0032cd0 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.258 7fcbde251700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 0x7fcbc001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:49.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.259 7fcbde251700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 0x7fcbc001abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:49.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.289 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd8087390 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.292 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbd000be30 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.442 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcbd80df000 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.443 7fcbceffd700 1 -- 172.21.2.117:0/2413146228 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcbd0001220 con 0x7fcbd80a6ab0 2021-02-10T11:45:49.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 msgr2=0x7fcbc001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 0x7fcbc001abc0 crc :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 msgr2=0x7fcbd81c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 rx=0x7fcbd0009940 tx=0x7fcbd000bcf0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 shutdown_connections 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbc0018730 0x7fcbc001abc0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd80a6ab0 0x7fcbd81c5410 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd80a66b0 0x7fcbd81c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 --2- 172.21.2.117:0/2413146228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd80e96c0 0x7fcbd80e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 >> 172.21.2.117:0/2413146228 conn(0x7fcbd80e71d0 msgr2=0x7fcbd80b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 shutdown_connections 2021-02-10T11:45:49.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:49.445 7fcbe0435700 1 -- 172.21.2.117:0/2413146228 wait complete. 2021-02-10T11:45:49.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:49.448 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:49.448 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:49.784 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:49.784 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:45:49.784 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:45:49.784 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:49.785 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:50.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[145505]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:50.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:50.137 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:50.138 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:50.472 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:45:50.472 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:50.473 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:45:50.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:50.473 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:45:50.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:50.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:50.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:51.143 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:45:51.143 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:45:51.143 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:45:51.144 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:45:51.144 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:51.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[146172]: fuse finished with error 0 and tester_r 0 2021-02-10T11:45:51.496 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:45:51.497 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:45:51.497 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:51.831 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:45:51.832 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:45:51.832 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:45:51.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:45:51.832 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:45:52.168 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.583 7f71721ef700 1 Processor -- start 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.583 7f71721ef700 1 -- start start 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f71721ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f71721ef700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 0x7f716c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f71721ef700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c0ede60 con 0x7f716c0ee3e0 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f71721ef700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c0eba20 con 0x7f716c0eb000 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f71721ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d1ce0 con 0x7f716c0d18e0 2021-02-10T11:45:52.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f716bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f716b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 0x7f716c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f716bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:49720/0 (socket says 172.21.2.117:49720) 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f716b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 0x7f716c0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52040/0 (socket says 172.21.2.117:52040) 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.584 7f716bfff700 1 -- 172.21.2.117:0/3590042832 learned_addr learned my addr 172.21.2.117:0/3590042832 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716bfff700 1 -- 172.21.2.117:0/3590042832 >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 msgr2=0x7f716c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716bfff700 1 --2- 172.21.2.117:0/3590042832 >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 0x7f716c0eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716bfff700 1 -- 172.21.2.117:0/3590042832 >> v1:172.21.2.125:6789/0 conn(0x7f716c0ee3e0 legacy=0x7f716c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716bfff700 1 -- 172.21.2.117:0/3590042832 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0eefd0 con 0x7f716c0d18e0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716bfff700 1 --2- 172.21.2.117:0/3590042832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f715c006700 tx=0x7f715c009b80).ready entity=mon.2 client_cookie=d6b3dc9899e33a4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716affd700 1 -- 172.21.2.117:0/3590042832 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f715c00c0c0 con 0x7f716c0d18e0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716affd700 1 -- 172.21.2.117:0/3590042832 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c00c440 con 0x7f716c0d18e0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f716affd700 1 -- 172.21.2.117:0/3590042832 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f715c00c960 con 0x7f716c0d18e0 2021-02-10T11:45:52.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f71721ef700 1 -- 172.21.2.117:0/3590042832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 msgr2=0x7f716c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.585 7f71721ef700 1 --2- 172.21.2.117:0/3590042832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f715c006700 tx=0x7f715c009b80).stop 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 -- 172.21.2.117:0/3590042832 shutdown_connections 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 --2- 172.21.2.117:0/3590042832 >> v2:172.21.2.117:3300/0 conn(0x7f716c0eb000 0x7f716c0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 --2- 172.21.2.117:0/3590042832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c0d18e0 0x7f716c0ea9e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 -- 172.21.2.117:0/3590042832 >> 172.21.2.117:0/3590042832 conn(0x7f716c0ced60 msgr2=0x7f716c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 -- 172.21.2.117:0/3590042832 shutdown_connections 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.586 7f71721ef700 1 -- 172.21.2.117:0/3590042832 wait complete. 2021-02-10T11:45:52.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 Processor -- start 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 -- start start 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c1a33d0 0x7f716c1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f716c0b8d10 0x7f716c0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0c0670 con 0x7f716c0b82f0 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0deb20 con 0x7f716c0b8d10 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f71721ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0ded40 con 0x7f716c1a33d0 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f716b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f716b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52044/0 (socket says 172.21.2.117:52044) 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.601 7f716b7fe700 1 -- 172.21.2.117:0/3222328376 learned_addr learned my addr 172.21.2.117:0/3222328376 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 -- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c1a33d0 msgr2=0x7f716c1a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c1a33d0 0x7f716c1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 -- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f716c0b8d10 msgr2=0x7f716c0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f716c0b8d10 0x7f716c0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 -- 172.21.2.117:0/3222328376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0def60 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f716b7fe700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f7160006f60 tx=0x7f716000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7160006340 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716000bd60 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71600022b0 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f71721ef700 1 -- 172.21.2.117:0/3222328376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c0def60 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.602 7f71721ef700 1 -- 172.21.2.117:0/3222328376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c0df4f0 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.603 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7160006b50 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.603 7f7168ff9700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 0x7f714c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.603 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7160032690 con 0x7f716c0b82f0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.603 7f716bfff700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 0x7f714c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:52.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.603 7f716bfff700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 0x7f714c01abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:52.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.634 7f71721ef700 1 -- 172.21.2.117:0/3222328376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c087290 con 0x7f716c0b82f0 2021-02-10T11:45:52.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.637 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7160000960 con 0x7f716c0b82f0 2021-02-10T11:45:52.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.765 7f71721ef700 1 -- 172.21.2.117:0/3222328376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f716c0ede60 con 0x7f716c0b82f0 2021-02-10T11:45:52.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.766 7f7168ff9700 1 -- 172.21.2.117:0/3222328376 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f7160006b50 con 0x7f716c0b82f0 2021-02-10T11:45:52.761 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:52.761 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:52.761 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:45:52.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 msgr2=0x7f714c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 0x7f714c01abc0 crc :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 msgr2=0x7f716c0b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:52.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 rx=0x7f7160006f60 tx=0x7f716000b970).stop 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 shutdown_connections 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f714c0186c0 0x7f714c01abc0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f716c0b82f0 0x7f716c0b86f0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f716c0b8d10 0x7f716c0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 --2- 172.21.2.117:0/3222328376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f716c1a33d0 0x7f716c1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 >> 172.21.2.117:0/3222328376 conn(0x7f716c0874c0 msgr2=0x7f716c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 shutdown_connections 2021-02-10T11:45:52.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:52.768 7f71721ef700 1 -- 172.21.2.117:0/3222328376 wait complete. 2021-02-10T11:45:52.771 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:45:53.168 INFO:teuthology.orchestra.run.gibba025.stdout:[] 2021-02-10T11:45:53.172 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:45:53.172 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000217470 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.594 7f6e35f03700 1 Processor -- start 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.594 7f6e35f03700 1 -- start start 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e35f03700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e301b69a0 0x7f6e301b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e35f03700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300ede60 con 0x7f6e300ef3f0 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e35f03700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e300eba00 con 0x7f6e300d18f0 2021-02-10T11:45:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e35f03700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e300d1cf0 con 0x7f6e301b69a0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e2effd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6e300ef3f0 0x7f6e300eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52302/0 (socket says 172.21.2.117:52302) 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e2effd700 1 -- 172.21.2.117:0/3364713696 learned_addr learned my addr 172.21.2.117:0/3364713696 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890283612 0 0) 0x7f6e200010b0 con 0x7f6e300ef3f0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.595 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c002fe0 con 0x7f6e300ef3f0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2ffff700 1 --2- 172.21.2.117:0/3364713696 >> v2:172.21.2.125:3300/0 conn(0x7f6e301b69a0 0x7f6e301b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1443505109 0 0) 0x7f6e200010b0 con 0x7f6e300ef3f0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 >> v2:172.21.2.125:3300/0 conn(0x7f6e301b69a0 msgr2=0x7f6e301b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 --2- 172.21.2.117:0/3364713696 >> v2:172.21.2.125:3300/0 conn(0x7f6e301b69a0 0x7f6e301b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 >> v1:172.21.2.125:6789/0 conn(0x7f6e300d18f0 legacy=0x7f6e300eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e301bb3f0 con 0x7f6e300ef3f0 2021-02-10T11:45:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6e200015b0 con 0x7f6e300ef3f0 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 >> v1:172.21.2.117:6789/0 conn(0x7f6e300ef3f0 legacy=0x7f6e300eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 0x7f6e1c005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e1c006780 0x7f6e1c008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e1c009580 0x7f6e1c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c00c2f0 con 0x7f6e1c003af0 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c00c510 con 0x7f6e1c006780 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c00c730 con 0x7f6e1c009580 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 0x7f6e1c005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2ffff700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e1c009580 0x7f6e1c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2effd700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e1c006780 0x7f6e1c008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 -- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e1c009580 msgr2=0x7f6e1c00baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e1c009580 0x7f6e1c00baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 -- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e1c006780 msgr2=0x7f6e1c008c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e1c006780 0x7f6e1c008c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 -- 172.21.2.117:0/3364713696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0509a0 con 0x7f6e1c003af0 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2ffff700 1 -- 172.21.2.117:0/3364713696 reap_dead start 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.596 7f6e2f7fe700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 0x7f6e1c005f00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7f6e18006ab0 tx=0x7f6e18009ac0).ready entity=mon.0 client_cookie=561d4116dcc4f494 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e180066a0 con 0x7f6e1c003af0 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e180022e0 con 0x7f6e1c003af0 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e2e7fc700 1 -- 172.21.2.117:0/3364713696 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e180027c0 con 0x7f6e1c003af0 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 -- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 msgr2=0x7f6e1c005f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 0x7f6e1c005f00 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 rx=0x7f6e18006ab0 tx=0x7f6e18009ac0).stop 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 -- 172.21.2.117:0/3364713696 shutdown_connections 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 --2- 172.21.2.117:0/3364713696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e1c003af0 0x7f6e1c005f00 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 -- 172.21.2.117:0/3364713696 >> 172.21.2.117:0/3364713696 conn(0x7f6e300cf060 msgr2=0x7f6e300cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 -- 172.21.2.117:0/3364713696 shutdown_connections 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.597 7f6e35f03700 1 -- 172.21.2.117:0/3364713696 wait complete. 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 Processor -- start 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 -- start start 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e300a6ce0 0x7f6e300dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 0x7f6e300862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300a67e0 con 0x7f6e300a2040 2021-02-10T11:45:53.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300a7120 con 0x7f6e30085ec0 2021-02-10T11:45:53.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.612 7f6e35f03700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e300a2a60 con 0x7f6e300a6ce0 2021-02-10T11:45:53.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52068/0 (socket says 172.21.2.117:52068) 2021-02-10T11:45:53.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 -- 172.21.2.117:0/580756746 learned_addr learned my addr 172.21.2.117:0/580756746 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:53.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 -- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e300a6ce0 msgr2=0x7f6e300dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e300a6ce0 0x7f6e300dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 -- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 msgr2=0x7f6e300862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2effd700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 0x7f6e300862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 0x7f6e300862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 -- 172.21.2.117:0/580756746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e301a2dc0 con 0x7f6e300a2040 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2effd700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 0x7f6e300862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:53.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e2ffff700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7f6e280099e0 tx=0x7f6e2800bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e28001400 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2800be60 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e280016c0 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.613 7f6e35f03700 1 -- 172.21.2.117:0/580756746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e301a2dc0 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.614 7f6e35f03700 1 -- 172.21.2.117:0/580756746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e301a2dc0 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.614 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e280011f0 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.614 7f6e27fff700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 0x7f6e1401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.614 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e28032c90 con 0x7f6e300a2040 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.615 7f6e2f7fe700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 0x7f6e1401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:53.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.615 7f6e2f7fe700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 0x7f6e1401abc0 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:53.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.644 7f6e35f03700 1 -- 172.21.2.117:0/580756746 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e300daff0 con 0x7f6e300a2040 2021-02-10T11:45:53.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.647 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e28006af0 con 0x7f6e300a2040 2021-02-10T11:45:53.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.770 7f6e35f03700 1 -- 172.21.2.117:0/580756746 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000217470", "target": ["mgr", ""]}) v1 -- 0x7f6e300daff0 con 0x7f6e140186c0 2021-02-10T11:45:53.768 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:45:53.781 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:53.769 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:45:53.781 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:53.769 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:45:53.781 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:53.769 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:45:53.781 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:53.769 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:45:53.781 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:45:53.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.806 7f6e27fff700 1 -- 172.21.2.117:0/580756746 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6e180093f0 con 0x7f6e140186c0 2021-02-10T11:45:53.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 msgr2=0x7f6e1401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 0x7f6e1401abc0 crc :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 msgr2=0x7f6e300a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7f6e280099e0 tx=0x7f6e2800bd20).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 shutdown_connections 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e140186c0 0x7f6e1401abc0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e300a2040 0x7f6e300a2440 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e30085ec0 0x7f6e300862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 --2- 172.21.2.117:0/580756746 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e300a6ce0 0x7f6e300dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 >> 172.21.2.117:0/580756746 conn(0x7f6e300b6540 msgr2=0x7f6e3008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:53.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 shutdown_connections 2021-02-10T11:45:53.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:53.836 7f6e35f03700 1 -- 172.21.2.117:0/580756746 wait complete. 2021-02-10T11:45:53.839 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.255 7ff265c53700 1 Processor -- start 2021-02-10T11:45:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.255 7ff265c53700 1 -- start start 2021-02-10T11:45:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600d18e0 0x7ff2600ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff2600eb000 0x7ff2600eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff2600ee3e0 0x7ff2600ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2600ede60 con 0x7ff2600ee3e0 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2600eba20 con 0x7ff2600eb000 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff265c53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600d1ce0 con 0x7ff2600d18e0 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff2600ee3e0 0x7ff2600ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff2600ee3e0 0x7ff2600ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52076/0 (socket says 172.21.2.117:52076) 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 -- 172.21.2.117:0/116556570 learned_addr learned my addr 172.21.2.117:0/116556570 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25f7fe700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600d18e0 0x7ff2600ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25effd700 1 --2- 172.21.2.117:0/116556570 >> v2:172.21.2.125:3300/0 conn(0x7ff2600eb000 0x7ff2600eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 -- 172.21.2.117:0/116556570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600d18e0 msgr2=0x7ff2600ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:54.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600d18e0 0x7ff2600ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 -- 172.21.2.117:0/116556570 >> v2:172.21.2.125:3300/0 conn(0x7ff2600eb000 msgr2=0x7ff2600eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 --2- 172.21.2.117:0/116556570 >> v2:172.21.2.125:3300/0 conn(0x7ff2600eb000 0x7ff2600eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 -- 172.21.2.117:0/116556570 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2600eee00 con 0x7ff2600ee3e0 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.256 7ff25ffff700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2600ee3e0 0x7ff2600ee7e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7ff258006f00 tx=0x7ff258009f10).ready entity=mon.0 client_cookie=2ccc9829a504bbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff25e7fc700 1 -- 172.21.2.117:0/116556570 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff25800b3b0 con 0x7ff2600ee3e0 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff25e7fc700 1 -- 172.21.2.117:0/116556570 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25800b730 con 0x7ff2600ee3e0 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff25e7fc700 1 -- 172.21.2.117:0/116556570 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff25800bc50 con 0x7ff2600ee3e0 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 -- 172.21.2.117:0/116556570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2600ee3e0 msgr2=0x7ff2600ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2600ee3e0 0x7ff2600ee7e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 rx=0x7ff258006f00 tx=0x7ff258009f10).stop 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 -- 172.21.2.117:0/116556570 shutdown_connections 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 --2- 172.21.2.117:0/116556570 >> v2:172.21.2.125:3300/0 conn(0x7ff2600eb000 0x7ff2600eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2600ee3e0 0x7ff2600ee7e0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 --2- 172.21.2.117:0/116556570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600d18e0 0x7ff2600ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 -- 172.21.2.117:0/116556570 >> 172.21.2.117:0/116556570 conn(0x7ff2600ced60 msgr2=0x7ff2600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:54.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 -- 172.21.2.117:0/116556570 shutdown_connections 2021-02-10T11:45:54.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.257 7ff265c53700 1 -- 172.21.2.117:0/116556570 wait complete. 2021-02-10T11:45:54.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 Processor -- start 2021-02-10T11:45:54.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 -- start start 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600b82f0 0x7ff2600b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2600b8d10 0x7ff2600b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600c0670 con 0x7ff2601a33d0 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600deb20 con 0x7ff2600b8d10 2021-02-10T11:45:54.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff265c53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2600ded40 con 0x7ff2600b82f0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff25f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff25f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52080/0 (socket says 172.21.2.117:52080) 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.272 7ff25f7fe700 1 -- 172.21.2.117:0/4022650534 learned_addr learned my addr 172.21.2.117:0/4022650534 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25effd700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600b82f0 0x7ff2600b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25ffff700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2600b8d10 0x7ff2600b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 -- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600b82f0 msgr2=0x7ff2600b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600b82f0 0x7ff2600b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 -- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2600b8d10 msgr2=0x7ff2600b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2600b8d10 0x7ff2600b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 -- 172.21.2.117:0/4022650534 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2600def60 con 0x7ff2601a33d0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff25f7fe700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7ff248006f60 tx=0x7ff24800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff24800bd60 con 0x7ff2601a33d0 2021-02-10T11:45:54.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff248000c90 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2480022b0 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff265c53700 1 -- 172.21.2.117:0/4022650534 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2600def60 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.273 7ff265c53700 1 -- 172.21.2.117:0/4022650534 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2600df4f0 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.274 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff248006b00 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.274 7ff257fff700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 0x7ff24401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.274 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff248032920 con 0x7ff2601a33d0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.274 7ff25effd700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 0x7ff24401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:54.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.275 7ff25effd700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 0x7ff24401abd0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:54.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.305 7ff265c53700 1 -- 172.21.2.117:0/4022650534 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff260087290 con 0x7ff2601a33d0 2021-02-10T11:45:54.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.308 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff248000940 con 0x7ff2601a33d0 2021-02-10T11:45:54.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.434 7ff265c53700 1 -- 172.21.2.117:0/4022650534 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2600ede60 con 0x7ff2601a33d0 2021-02-10T11:45:54.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.435 7ff257fff700 1 -- 172.21.2.117:0/4022650534 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7ff248000940 con 0x7ff2601a33d0 2021-02-10T11:45:54.431 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:54.431 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:54.431 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 msgr2=0x7ff24401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 0x7ff24401abd0 crc :-1 s=READY pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 msgr2=0x7ff2601a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 rx=0x7ff248006f60 tx=0x7ff24800b970).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 shutdown_connections 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff2600b8d10 0x7ff2600b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2600b82f0 0x7ff2600b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff2440186d0 0x7ff24401abd0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 --2- 172.21.2.117:0/4022650534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2601a33d0 0x7ff2601a37d0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:54.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 >> 172.21.2.117:0/4022650534 conn(0x7ff2600874c0 msgr2=0x7ff260096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:54.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 shutdown_connections 2021-02-10T11:45:54.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:54.437 7ff265c53700 1 -- 172.21.2.117:0/4022650534 wait complete. 2021-02-10T11:45:54.440 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:45:54.838 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:45:54.838 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.839 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 13, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 1.0679870309999999, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 1, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 0, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:45:54.840 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 19.951239466076988, 2021-02-10T11:45:54.841 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [], 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:45:54.842 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [ 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627781, 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 1 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:45:54.843 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:54.844 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:54.844 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:45:54.844 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 1.0679870309999999, "requests_in_flight": 1, "completed_requests": 0, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.951239466076988, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [ { "start": 1099511627781, "length": 1 } ], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:45:54.845 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.262 7f1f0b74c700 1 Processor -- start 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 -- start start 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f040d18e0 0x7f1f040ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1f040ef3f0 0x7f1f041b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f040ede60 con 0x7f1f040eb000 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040ef7f0 con 0x7f1f040d18e0 2021-02-10T11:45:55.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f0b74c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040d1ce0 con 0x7f1f040ef3f0 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f09568700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f040d18e0 0x7f1f040ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f08d67700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1f040eb000 0x7f1f040eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52340/0 (socket says 172.21.2.117:52340) 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f08d67700 1 -- 172.21.2.117:0/3218994095 learned_addr learned my addr 172.21.2.117:0/3218994095 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.263 7f1f09d69700 1 --2- 172.21.2.117:0/3218994095 >> v2:172.21.2.125:3300/0 conn(0x7f1f040ef3f0 0x7f1f041b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f09568700 1 -- 172.21.2.117:0/3218994095 >> v2:172.21.2.125:3300/0 conn(0x7f1f040ef3f0 msgr2=0x7f1f041b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f09568700 1 --2- 172.21.2.117:0/3218994095 >> v2:172.21.2.125:3300/0 conn(0x7f1f040ef3f0 0x7f1f041b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f09568700 1 -- 172.21.2.117:0/3218994095 >> v1:172.21.2.117:6789/0 conn(0x7f1f040eb000 legacy=0x7f1f040eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f09568700 1 -- 172.21.2.117:0/3218994095 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f041b8f90 con 0x7f1f040d18e0 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f09568700 1 --2- 172.21.2.117:0/3218994095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040d18e0 0x7f1f040ea9e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7f1f00006650 tx=0x7f1f00009a80).ready entity=mon.0 client_cookie=d45f8b46c7e701db server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1efbfff700 1 -- 172.21.2.117:0/3218994095 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0000c0c0 con 0x7f1f040d18e0 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1efbfff700 1 -- 172.21.2.117:0/3218994095 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0000c340 con 0x7f1f040d18e0 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1efbfff700 1 -- 172.21.2.117:0/3218994095 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0000c860 con 0x7f1f040d18e0 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 -- 172.21.2.117:0/3218994095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040d18e0 msgr2=0x7f1f040ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 --2- 172.21.2.117:0/3218994095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040d18e0 0x7f1f040ea9e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 rx=0x7f1f00006650 tx=0x7f1f00009a80).stop 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 -- 172.21.2.117:0/3218994095 shutdown_connections 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 --2- 172.21.2.117:0/3218994095 >> v2:172.21.2.125:3300/0 conn(0x7f1f040ef3f0 0x7f1f041b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 --2- 172.21.2.117:0/3218994095 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040d18e0 0x7f1f040ea9e0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.264 7f1f0b74c700 1 -- 172.21.2.117:0/3218994095 >> 172.21.2.117:0/3218994095 conn(0x7f1f040ced60 msgr2=0x7f1f040cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.265 7f1f0b74c700 1 -- 172.21.2.117:0/3218994095 shutdown_connections 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.265 7f1f0b74c700 1 -- 172.21.2.117:0/3218994095 wait complete. 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.279 7f1f0b74c700 1 Processor -- start 2021-02-10T11:45:55.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.279 7f1f0b74c700 1 -- start start 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 0x7f1f040c2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 0x7f1f041a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040c28b0 con 0x7f1f040a8d10 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f041a7bd0 con 0x7f1f040b4360 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f041a7db0 con 0x7f1f041a71b0 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52092/0 (socket says 172.21.2.117:52092) 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 -- 172.21.2.117:0/2644490179 learned_addr learned my addr 172.21.2.117:0/2644490179 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:45:55.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f08d67700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 0x7f1f040c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09d69700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 0x7f1f041a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 -- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 msgr2=0x7f1f041a75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 0x7f1f041a75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 -- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 msgr2=0x7f1f040c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 0x7f1f040c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 -- 172.21.2.117:0/2644490179 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f041a7fd0 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f08d67700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 0x7f1f040c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09d69700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 0x7f1f041a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f09568700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f1f00009630 tx=0x7f1f0000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f000093a0 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0000bd70 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f00005560 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f041a7fd0 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.280 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f041a7fd0 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.281 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f00000c90 con 0x7f1f040a8d10 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.281 7f1ef9ffb700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 0x7f1ee401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:45:55.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.281 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1f00032520 con 0x7f1f040a8d10 2021-02-10T11:45:55.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.282 7f1f08d67700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 0x7f1ee401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:45:55.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.282 7f1f08d67700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 0x7f1ee401abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:45:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.313 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04087290 con 0x7f1f040a8d10 2021-02-10T11:45:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.315 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f00023c90 con 0x7f1f040a8d10 2021-02-10T11:45:55.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.444 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f040ede60 con 0x7f1f040a8d10 2021-02-10T11:45:55.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.444 7f1ef9ffb700 1 -- 172.21.2.117:0/2644490179 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1f00023c90 con 0x7f1f040a8d10 2021-02-10T11:45:55.438 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:45:55.439 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:45:55.439 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:45:55.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.446 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 msgr2=0x7f1ee401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.446 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 0x7f1ee401abc0 crc :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.446 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 msgr2=0x7f1f040a9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.446 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 rx=0x7f1f00009630 tx=0x7f1f0000b980).stop 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 shutdown_connections 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f041a71b0 0x7f1f041a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f040b4360 0x7f1f040c2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ee40186c0 0x7f1ee401abc0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 --2- 172.21.2.117:0/2644490179 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f040a8d10 0x7f1f040a9160 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:45:55.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 >> 172.21.2.117:0/2644490179 conn(0x7f1f0409fb80 msgr2=0x7f1f0408d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:45:55.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 shutdown_connections 2021-02-10T11:45:55.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:45:55.447 7f1f0b74c700 1 -- 172.21.2.117:0/2644490179 wait complete. 2021-02-10T11:45:55.449 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:45:55.848 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:45:55.849 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 13, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 2.0789747529999998, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:45:55.850 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:55.851 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 19.904690199972109, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:45:55.852 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:45:55.853 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:45:55.854 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:45:55.855 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 13, "uptime": 2.0789747529999998, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.904690199972109, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:45:55.855 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:00.856 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 Processor -- start 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 -- start start 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f12640ef3f0 0x7f12640eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f12640eb7f0 0x7f12641b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640ede60 con 0x7f12640d18e0 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.275 7f126a75e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640ef7f0 con 0x7f12640ef3f0 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f126a75e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640d1ce0 con 0x7f12640eb7f0 2021-02-10T11:46:01.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f12640ef3f0 0x7f12640eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1263fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f12640d18e0 0x7f12640ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52350/0 (socket says 172.21.2.117:52350) 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1263fff700 1 -- 172.21.2.117:0/3527477281 learned_addr learned my addr 172.21.2.117:0/3527477281 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1268d7b700 1 --2- 172.21.2.117:0/3527477281 >> v2:172.21.2.125:3300/0 conn(0x7f12640eb7f0 0x7f12641b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 -- 172.21.2.117:0/3527477281 >> v2:172.21.2.125:3300/0 conn(0x7f12640eb7f0 msgr2=0x7f12641b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 --2- 172.21.2.117:0/3527477281 >> v2:172.21.2.125:3300/0 conn(0x7f12640eb7f0 0x7f12641b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 -- 172.21.2.117:0/3527477281 >> v1:172.21.2.117:6789/0 conn(0x7f12640d18e0 legacy=0x7f12640ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 -- 172.21.2.117:0/3527477281 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12641b8f90 con 0x7f12640ef3f0 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f12637fe700 1 --2- 172.21.2.117:0/3527477281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640ef3f0 0x7f12640eb1d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7f1258006650 tx=0x7f1258009a00).ready entity=mon.0 client_cookie=c1a6a7ea421c9ec4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:01.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1262ffd700 1 -- 172.21.2.117:0/3527477281 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12580009b0 con 0x7f12640ef3f0 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1262ffd700 1 -- 172.21.2.117:0/3527477281 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800c0c0 con 0x7f12640ef3f0 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f1262ffd700 1 -- 172.21.2.117:0/3527477281 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125800c510 con 0x7f12640ef3f0 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f126a75e700 1 -- 172.21.2.117:0/3527477281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640ef3f0 msgr2=0x7f12640eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.276 7f126a75e700 1 --2- 172.21.2.117:0/3527477281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640ef3f0 0x7f12640eb1d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 rx=0x7f1258006650 tx=0x7f1258009a00).stop 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 -- 172.21.2.117:0/3527477281 shutdown_connections 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 --2- 172.21.2.117:0/3527477281 >> v2:172.21.2.125:3300/0 conn(0x7f12640eb7f0 0x7f12641b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 --2- 172.21.2.117:0/3527477281 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640ef3f0 0x7f12640eb1d0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 -- 172.21.2.117:0/3527477281 >> 172.21.2.117:0/3527477281 conn(0x7f12640ced60 msgr2=0x7f12640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:01.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 -- 172.21.2.117:0/3527477281 shutdown_connections 2021-02-10T11:46:01.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.277 7f126a75e700 1 -- 172.21.2.117:0/3527477281 wait complete. 2021-02-10T11:46:01.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 Processor -- start 2021-02-10T11:46:01.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 -- start start 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12640959c0 0x7f12640c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12640c2910 0x7f12640c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1264095e70 con 0x7f12640963f0 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12640e86d0 con 0x7f12640959c0 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f126a75e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1264096e10 con 0x7f12640c2910 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f1268d7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f1268d7b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52108/0 (socket says 172.21.2.117:52108) 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f1268d7b700 1 -- 172.21.2.117:0/860759096 learned_addr learned my addr 172.21.2.117:0/860759096 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.292 7f1263fff700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12640959c0 0x7f12640c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f12637fe700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12640c2910 0x7f12640c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 -- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12640c2910 msgr2=0x7f12640c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12640c2910 0x7f12640c2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 -- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12640959c0 msgr2=0x7f12640c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12640959c0 0x7f12640c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 -- 172.21.2.117:0/860759096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264097030 con 0x7f12640963f0 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1268d7b700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f125c006f40 tx=0x7f125c00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:01.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125c0011f0 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c003030 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f125c001730 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f126a75e700 1 -- 172.21.2.117:0/860759096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1264097030 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.293 7f126a75e700 1 -- 172.21.2.117:0/860759096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1264097030 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.294 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f125c0056b0 con 0x7f12640963f0 2021-02-10T11:46:01.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.294 7f1260ff9700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 0x7f124401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:01.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.294 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f125c032b70 con 0x7f12640963f0 2021-02-10T11:46:01.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.294 7f1263fff700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 0x7f124401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:01.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.295 7f1263fff700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 0x7f124401abd0 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:01.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.325 7f126a75e700 1 -- 172.21.2.117:0/860759096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264087290 con 0x7f12640963f0 2021-02-10T11:46:01.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.328 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f125c000f40 con 0x7f12640963f0 2021-02-10T11:46:01.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.455 7f126a75e700 1 -- 172.21.2.117:0/860759096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12640ede60 con 0x7f12640963f0 2021-02-10T11:46:01.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.456 7f1260ff9700 1 -- 172.21.2.117:0/860759096 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f125c0056b0 con 0x7f12640963f0 2021-02-10T11:46:01.451 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:01.451 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:01.451 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:01.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 msgr2=0x7f124401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 0x7f124401abd0 crc :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 msgr2=0x7f12640967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 rx=0x7f125c006f40 tx=0x7f125c00bd40).stop 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 shutdown_connections 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f12440186d0 0x7f124401abd0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f12640963f0 0x7f12640967f0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f12640c2910 0x7f12640c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 --2- 172.21.2.117:0/860759096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f12640959c0 0x7f12640c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:01.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 >> 172.21.2.117:0/860759096 conn(0x7f12640e8c80 msgr2=0x7f1264092070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:01.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 shutdown_connections 2021-02-10T11:46:01.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:01.458 7f126a75e700 1 -- 172.21.2.117:0/860759096 wait complete. 2021-02-10T11:46:01.460 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:01.857 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:01.857 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:01.857 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:01.857 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:01.858 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:01.858 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:01.858 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:01.858 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.858 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.859 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 12, 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 8.0869017920000008, 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:01.860 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.861 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:01.862 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.863 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 20.625488879878933, 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:01.864 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:01.865 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:01.866 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:01.867 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:01.868 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:01.868 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:01.868 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:01.868 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:01.868 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:01.869 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:01.869 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:01.869 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:01.869 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:01.870 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 12, "uptime": 8.0869017920000008, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.625488879878933, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:01.870 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:06.871 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:07.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.289 7f8832ced700 1 Processor -- start 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.289 7f8832ced700 1 -- start start 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8832ced700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c0d18e0 0x7f882c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8832ced700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f882c0ef3f0 0x7f882c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8832ced700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0ede60 con 0x7f882c0eb000 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8832ced700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882c088020 con 0x7f882c0ef3f0 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8832ced700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0d1ce0 con 0x7f882c0d18e0 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f882bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f882c0eb000 0x7f882c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52364/0 (socket says 172.21.2.117:52364) 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f882bfff700 1 -- 172.21.2.117:0/1485816426 learned_addr learned my addr 172.21.2.117:0/1485816426 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f883130a700 1 --2- 172.21.2.117:0/1485816426 >> v2:172.21.2.125:3300/0 conn(0x7f882c0ef3f0 0x7f882c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f8830b09700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c0d18e0 0x7f882c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f882b7fe700 1 -- 172.21.2.117:0/1485816426 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808116375 0 0) 0x7f881c000ce0 con 0x7f882c0eb000 2021-02-10T11:46:07.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f882b7fe700 1 -- 172.21.2.117:0/1485816426 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8818002fe0 con 0x7f882c0eb000 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f883130a700 1 -- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c0d18e0 msgr2=0x7f882c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f883130a700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c0d18e0 0x7f882c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f883130a700 1 -- 172.21.2.117:0/1485816426 >> v1:172.21.2.117:6789/0 conn(0x7f882c0eb000 legacy=0x7f882c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.290 7f883130a700 1 -- 172.21.2.117:0/1485816426 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c1b8f90 con 0x7f882c0ef3f0 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f883130a700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0ef3f0 0x7f882c1b7960 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f8824006e50 tx=0x7f8824009e70).ready entity=mon.1 client_cookie=fe3d7a9273823cc5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f882b7fe700 1 -- 172.21.2.117:0/1485816426 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f882400b510 con 0x7f882c0ef3f0 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f882b7fe700 1 -- 172.21.2.117:0/1485816426 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8824006790 con 0x7f882c0ef3f0 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f882b7fe700 1 -- 172.21.2.117:0/1485816426 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f882400b940 con 0x7f882c0ef3f0 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 -- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0ef3f0 msgr2=0x7f882c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0ef3f0 0x7f882c1b7960 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f8824006e50 tx=0x7f8824009e70).stop 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 -- 172.21.2.117:0/1485816426 shutdown_connections 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0ef3f0 0x7f882c1b7960 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 --2- 172.21.2.117:0/1485816426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c0d18e0 0x7f882c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 -- 172.21.2.117:0/1485816426 >> 172.21.2.117:0/1485816426 conn(0x7f882c0ced60 msgr2=0x7f882c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 -- 172.21.2.117:0/1485816426 shutdown_connections 2021-02-10T11:46:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.291 7f8832ced700 1 -- 172.21.2.117:0/1485816426 wait complete. 2021-02-10T11:46:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.306 7f8832ced700 1 Processor -- start 2021-02-10T11:46:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.306 7f8832ced700 1 -- start start 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.306 7f8832ced700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c09e380 0x7f882c0bb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.306 7f8832ced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.306 7f8832ced700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 0x7f882c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f8832ced700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0a5a70 con 0x7f882c096b40 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f8832ced700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0975a0 con 0x7f882c0a5f70 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f8832ced700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0a6990 con 0x7f882c09e380 2021-02-10T11:46:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f883130a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 0x7f882c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52122/0 (socket says 172.21.2.117:52122) 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 -- 172.21.2.117:0/4249320824 learned_addr learned my addr 172.21.2.117:0/4249320824 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f8830b09700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c09e380 0x7f882c0bb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 -- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c09e380 msgr2=0x7f882c0bb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c09e380 0x7f882c0bb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 -- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 msgr2=0x7f882c0a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 0x7f882c0a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 -- 172.21.2.117:0/4249320824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c09e780 con 0x7f882c096b40 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f883130a700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 0x7f882c0a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.307 7f882bfff700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f881c00b5c0 tx=0x7f881c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f881c00bd50 con 0x7f882c096b40 2021-02-10T11:46:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c006280 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f881c004620 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f8832ced700 1 -- 172.21.2.117:0/4249320824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c09e780 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f8832ced700 1 -- 172.21.2.117:0/4249320824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c0a4ac0 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f881c0045b0 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.308 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f881c032690 con 0x7f882c096b40 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.309 7f8830b09700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.310 7f8830b09700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:07.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.339 7f8832ced700 1 -- 172.21.2.117:0/4249320824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c087290 con 0x7f882c096b40 2021-02-10T11:46:07.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.342 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f881c0043a0 con 0x7f882c096b40 2021-02-10T11:46:07.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.468 7f8832ced700 1 -- 172.21.2.117:0/4249320824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f882c0ede60 con 0x7f882c096b40 2021-02-10T11:46:07.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.469 7f88297fa700 1 -- 172.21.2.117:0/4249320824 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f881c00bd50 con 0x7f882c096b40 2021-02-10T11:46:07.463 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:07.463 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:07.463 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:07.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 msgr2=0x7f881001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 crc :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 msgr2=0x7f882c096f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:07.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 rx=0x7f881c00b5c0 tx=0x7f881c00b960).stop 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 shutdown_connections 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f882c096b40 0x7f882c096f40 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f882c0a5f70 0x7f882c0a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 --2- 172.21.2.117:0/4249320824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f882c09e380 0x7f882c0bb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 >> 172.21.2.117:0/4249320824 conn(0x7f882c0aa2d0 msgr2=0x7f882c0c1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 shutdown_connections 2021-02-10T11:46:07.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:07.471 7f8832ced700 1 -- 172.21.2.117:0/4249320824 wait complete. 2021-02-10T11:46:07.473 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:07.870 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:07.870 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:07.870 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:07.871 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 11, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 14.09982877, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:07.872 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 20.340924134597643, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.873 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:07.874 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:07.875 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:07.876 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:07.876 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 14.09982877, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.340924134597643, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:07.877 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:10.208 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:10.212 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:12.878 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:13.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.295 7f47941e7700 1 Processor -- start 2021-02-10T11:46:13.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.295 7f47941e7700 1 -- start start 2021-02-10T11:46:13.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f47941e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f478c0d18e0 0x7f478c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f47941e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c0eb000 0x7f478c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f47941e7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478c0ede60 con 0x7f478c0ee3e0 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f47941e7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478c0eba90 con 0x7f478c0d18e0 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f47941e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c0d1ce0 con 0x7f478c0eb000 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f4792003700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f478c0d18e0 0x7f478c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f4792804700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f478c0ee3e0 0x7f478c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52376/0 (socket says 172.21.2.117:52376) 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.296 7f4792804700 1 -- 172.21.2.117:0/556104200 learned_addr learned my addr 172.21.2.117:0/556104200 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4792003700 1 -- 172.21.2.117:0/556104200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c0eb000 msgr2=0x7f478c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4792003700 1 --2- 172.21.2.117:0/556104200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c0eb000 0x7f478c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4792003700 1 -- 172.21.2.117:0/556104200 >> v1:172.21.2.117:6789/0 conn(0x7f478c0ee3e0 legacy=0x7f478c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4792003700 1 -- 172.21.2.117:0/556104200 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c0eefd0 con 0x7f478c0d18e0 2021-02-10T11:46:13.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4791001700 1 -- 172.21.2.117:0/556104200 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597781457 0 0) 0x7f4784001480 con 0x7f478c0ee3e0 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4792003700 1 --2- 172.21.2.117:0/556104200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c0d18e0 0x7f478c0ea9e0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f4780006650 tx=0x7f4780009a80).ready entity=mon.0 client_cookie=37843568b38ffb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4791001700 1 -- 172.21.2.117:0/556104200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f478000c0c0 con 0x7f478c0d18e0 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4791001700 1 -- 172.21.2.117:0/556104200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478000c340 con 0x7f478c0d18e0 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f4791001700 1 -- 172.21.2.117:0/556104200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f478000c860 con 0x7f478c0d18e0 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f47941e7700 1 -- 172.21.2.117:0/556104200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c0d18e0 msgr2=0x7f478c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.297 7f47941e7700 1 --2- 172.21.2.117:0/556104200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c0d18e0 0x7f478c0ea9e0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 rx=0x7f4780006650 tx=0x7f4780009a80).stop 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 -- 172.21.2.117:0/556104200 shutdown_connections 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 --2- 172.21.2.117:0/556104200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c0eb000 0x7f478c0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 --2- 172.21.2.117:0/556104200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c0d18e0 0x7f478c0ea9e0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 -- 172.21.2.117:0/556104200 >> 172.21.2.117:0/556104200 conn(0x7f478c0ced60 msgr2=0x7f478c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 -- 172.21.2.117:0/556104200 shutdown_connections 2021-02-10T11:46:13.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.298 7f47941e7700 1 -- 172.21.2.117:0/556104200 wait complete. 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 Processor -- start 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 -- start start 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f478c0a7350 0x7f478c0a7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c19fce0 0x7f478c1a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c191a90 con 0x7f478c190ff0 2021-02-10T11:46:13.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1931f0 con 0x7f478c0a7350 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.313 7f47941e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f478c1a0700 con 0x7f478c19fce0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52134/0 (socket says 172.21.2.117:52134) 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 -- 172.21.2.117:0/1511400186 learned_addr learned my addr 172.21.2.117:0/1511400186 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 -- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c19fce0 msgr2=0x7f478c1a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c19fce0 0x7f478c1a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 -- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f478c0a7350 msgr2=0x7f478c0a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f478c0a7350 0x7f478c0a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 -- 172.21.2.117:0/1511400186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478c1a0920 con 0x7f478c190ff0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f4791802700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f4788006f00 tx=0x7f478800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4788006350 con 0x7f478c190ff0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788006b90 con 0x7f478c190ff0 2021-02-10T11:46:13.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f47941e7700 1 -- 172.21.2.117:0/1511400186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c1a0920 con 0x7f478c190ff0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f47941e7700 1 -- 172.21.2.117:0/1511400186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478c191cc0 con 0x7f478c190ff0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.314 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4788000c90 con 0x7f478c190ff0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.315 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4788007000 con 0x7f478c190ff0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.315 7f477effd700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 0x7f477401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.315 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f47880328d0 con 0x7f478c190ff0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.315 7f4792003700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 0x7f477401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:13.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.316 7f4792003700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 0x7f477401abc0 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:13.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.346 7f47941e7700 1 -- 172.21.2.117:0/1511400186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478c087210 con 0x7f478c190ff0 2021-02-10T11:46:13.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.349 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47880009b0 con 0x7f478c190ff0 2021-02-10T11:46:13.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.475 7f47941e7700 1 -- 172.21.2.117:0/1511400186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f478c0ede60 con 0x7f478c190ff0 2021-02-10T11:46:13.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.475 7f477effd700 1 -- 172.21.2.117:0/1511400186 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f4788007000 con 0x7f478c190ff0 2021-02-10T11:46:13.470 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:13.470 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:13.470 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:13.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 msgr2=0x7f477401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 0x7f477401abc0 crc :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 msgr2=0x7f478c1913f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 rx=0x7f4788006f00 tx=0x7f478800b940).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 shutdown_connections 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4774018730 0x7f477401abc0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f478c190ff0 0x7f478c1913f0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f478c19fce0 0x7f478c1a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 --2- 172.21.2.117:0/1511400186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f478c0a7350 0x7f478c0a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 >> 172.21.2.117:0/1511400186 conn(0x7f478c087490 msgr2=0x7f478c0e9380 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 shutdown_connections 2021-02-10T11:46:13.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:13.477 7f47941e7700 1 -- 172.21.2.117:0/1511400186 wait complete. 2021-02-10T11:46:13.481 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:13.877 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:13.877 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:13.877 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:13.877 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:13.878 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 11, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 20.106755821, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:13.879 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 20.060563551551702, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.880 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:13.881 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:13.882 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:13.883 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:13.883 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:13.883 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 11, "uptime": 20.106755821, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 20.060563551551702, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:13.884 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:18.885 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:19.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 Processor -- start 2021-02-10T11:46:19.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 -- start start 2021-02-10T11:46:19.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa95c0ef3f0 0x7fa95c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0ede60 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0d1ce0 con 0x7fa95c0d18e0 2021-02-10T11:46:19.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa961860700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0ef7f0 con 0x7fa95c0ef3f0 2021-02-10T11:46:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa95bd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa95c0eb7f0 0x7fa95c1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52392/0 (socket says 172.21.2.117:52392) 2021-02-10T11:46:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.303 7fa95bd75700 1 -- 172.21.2.117:0/2472562444 learned_addr learned my addr 172.21.2.117:0/2472562444 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530253955 0 0) 0x7fa954000f90 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa944002fe0 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 --2- 172.21.2.117:0/2472562444 >> v2:172.21.2.125:3300/0 conn(0x7fa95c0ef3f0 0x7fa95c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1549320154 0 0) 0x7fa954001460 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 >> v2:172.21.2.125:3300/0 conn(0x7fa95c0ef3f0 msgr2=0x7fa95c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 --2- 172.21.2.117:0/2472562444 >> v2:172.21.2.125:3300/0 conn(0x7fa95c0ef3f0 0x7fa95c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 >> v1:172.21.2.125:6789/0 conn(0x7fa95c0d18e0 legacy=0x7fa95c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c1bb180 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa9540019e0 con 0x7fa95c0eb7f0 2021-02-10T11:46:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 >> v1:172.21.2.117:6789/0 conn(0x7fa95c0eb7f0 legacy=0x7fa95c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa944003a70 0x7fa944005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 0x7fa944008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9440095a0 0x7fa94400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94400c310 con 0x7fa944006780 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94400c530 con 0x7fa944003a70 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95a572700 1 -- 172.21.2.117:0/2472562444 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94400c750 con 0x7fa9440095a0 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 0x7fa944008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95b574700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa944003a70 0x7fa944005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95bd75700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9440095a0 0x7fa94400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 -- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9440095a0 msgr2=0x7fa94400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:19.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9440095a0 0x7fa94400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 -- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa944003a70 msgr2=0x7fa944005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:19.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95ad73700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa944003a70 0x7fa944005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.304 7fa95bd75700 1 -- 172.21.2.117:0/2472562444 reap_dead start 2021-02-10T11:46:19.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa95ad73700 1 -- 172.21.2.117:0/2472562444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94400c9f0 con 0x7fa944006780 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa95ad73700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 0x7fa944008ca0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7fa94c006360 tx=0x7fa94c006820).ready entity=mon.0 client_cookie=7a503a5e69110086 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa94c006560 con 0x7fa944006780 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c000c90 con 0x7fa944006780 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa95a572700 1 -- 172.21.2.117:0/2472562444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa94c050340 con 0x7fa944006780 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 -- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 msgr2=0x7fa944008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 0x7fa944008ca0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7fa94c006360 tx=0x7fa94c006820).stop 2021-02-10T11:46:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 -- 172.21.2.117:0/2472562444 shutdown_connections 2021-02-10T11:46:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 --2- 172.21.2.117:0/2472562444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa944006780 0x7fa944008ca0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 -- 172.21.2.117:0/2472562444 >> 172.21.2.117:0/2472562444 conn(0x7fa95c0ced60 msgr2=0x7fa95c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 -- 172.21.2.117:0/2472562444 shutdown_connections 2021-02-10T11:46:19.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.305 7fa961860700 1 -- 172.21.2.117:0/2472562444 wait complete. 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 Processor -- start 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 -- start start 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 0x7fa95c08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa95c0b5da0 0x7fa95c1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b61a0 con 0x7fa95c1a3480 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c08c8b0 con 0x7fa95c0b5da0 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa961860700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ba1c0 con 0x7fa95c0b59a0 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.320 7fa95bd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52154/0 (socket says 172.21.2.117:52154) 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 -- 172.21.2.117:0/3414296247 learned_addr learned my addr 172.21.2.117:0/3414296247 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:19.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95b574700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 0x7fa95c08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95ad73700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa95c0b5da0 0x7fa95c1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 -- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 msgr2=0x7fa95c08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 0x7fa95c08c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 -- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa95c0b5da0 msgr2=0x7fa95c1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa95c0b5da0 0x7fa95c1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 -- 172.21.2.117:0/3414296247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c1a3880 con 0x7fa95c1a3480 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95b574700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 0x7fa95c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa95bd75700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7fa95400a940 tx=0x7fa95400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa954001d90 con 0x7fa95c1a3480 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa954004930 con 0x7fa95c1a3480 2021-02-10T11:46:19.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa961860700 1 -- 172.21.2.117:0/3414296247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c1a3880 con 0x7fa95c1a3480 2021-02-10T11:46:19.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa961860700 1 -- 172.21.2.117:0/3414296247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0b2bb0 con 0x7fa95c1a3480 2021-02-10T11:46:19.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.321 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa954004dd0 con 0x7fa95c1a3480 2021-02-10T11:46:19.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.322 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa95400ae20 con 0x7fa95c1a3480 2021-02-10T11:46:19.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.322 7fa943fff700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 0x7fa93c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:19.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.322 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa954032b80 con 0x7fa95c1a3480 2021-02-10T11:46:19.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.322 7fa95b574700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 0x7fa93c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:19.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.323 7fa95b574700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 0x7fa93c01abd0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:19.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.353 7fa961860700 1 -- 172.21.2.117:0/3414296247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c0ede60 con 0x7fa95c1a3480 2021-02-10T11:46:19.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.356 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa954001d90 con 0x7fa95c1a3480 2021-02-10T11:46:19.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.482 7fa961860700 1 -- 172.21.2.117:0/3414296247 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa95c0ede60 con 0x7fa95c1a3480 2021-02-10T11:46:19.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.482 7fa943fff700 1 -- 172.21.2.117:0/3414296247 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa95400ae20 con 0x7fa95c1a3480 2021-02-10T11:46:19.478 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:19.478 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:19.478 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 -- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 msgr2=0x7fa93c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 0x7fa93c01abd0 crc :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 -- 172.21.2.117:0/3414296247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 msgr2=0x7fa95c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 rx=0x7fa95400a940 tx=0x7fa95400ace0).stop 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 -- 172.21.2.117:0/3414296247 shutdown_connections 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa93c0186d0 0x7fa93c01abd0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa95c1a3480 0x7fa95c0b2590 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa95c0b5da0 0x7fa95c1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 --2- 172.21.2.117:0/3414296247 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa95c0b59a0 0x7fa95c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:19.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.484 7fa961860700 1 -- 172.21.2.117:0/3414296247 >> 172.21.2.117:0/3414296247 conn(0x7fa95c0bb5e0 msgr2=0x7fa95c090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:19.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.485 7fa961860700 1 -- 172.21.2.117:0/3414296247 shutdown_connections 2021-02-10T11:46:19.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:19.485 7fa961860700 1 -- 172.21.2.117:0/3414296247 wait complete. 2021-02-10T11:46:19.488 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:19.884 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:20.485 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:20.485 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:20.485 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:20.485 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:20.485 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:20.486 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:20.487 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:20.487 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.487 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:20.487 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 1, 2021-02-10T11:46:20.487 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 10, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 26.113682871999998, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:20.488 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:20.489 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:20.490 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.490 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:20.490 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:20.490 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:20.490 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:20.491 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 19.784067200829021, 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:20.492 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:20.493 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:20.496 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:20.496 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:20.496 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:20.496 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:20.497 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:20.497 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:20.497 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:20.497 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:20.497 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:20.498 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:20.499 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:20.499 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:20.499 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:20.499 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:20.501 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 1, "num_caps": 7, "request_load_avg": 10, "uptime": 26.113682871999998, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.784067200829021, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:20.501 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:25.502 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 Processor -- start 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 -- start start 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb80e4e40 0x7effb81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effb8085450 0x7effb80d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb8085930 con 0x7effb80e5240 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb80e5cc0 con 0x7effb8085450 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.920 7effbe6f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb8084840 con 0x7effb80e4e40 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effb8085450 0x7effb80d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7effb8085450 0x7effb80d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52158/0 (socket says 172.21.2.117:52158) 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 -- 172.21.2.117:0/3653299270 learned_addr learned my addr 172.21.2.117:0/3653299270 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effb7fff700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb80e4e40 0x7effb81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 -- 172.21.2.117:0/3653299270 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb80e4e40 msgr2=0x7effb81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb80e4e40 0x7effb81b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 -- 172.21.2.117:0/3653299270 >> v1:172.21.2.125:6789/0 conn(0x7effb80e5240 legacy=0x7effb8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 -- 172.21.2.117:0/3653299270 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80d1be0 con 0x7effb8085450 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.921 7effbcd0e700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb8085450 0x7effb80d05b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 rx=0x7effb0006f00 tx=0x7effb0009f10).ready entity=mon.0 client_cookie=931d66df12263be8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.922 7effb6ffd700 1 -- 172.21.2.117:0/3653299270 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effb000b3b0 con 0x7effb8085450 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.922 7effb6ffd700 1 -- 172.21.2.117:0/3653299270 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb000b730 con 0x7effb8085450 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.922 7effb6ffd700 1 -- 172.21.2.117:0/3653299270 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effb000bc50 con 0x7effb8085450 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.922 7effbe6f1700 1 -- 172.21.2.117:0/3653299270 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb8085450 msgr2=0x7effb80d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.922 7effbe6f1700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb8085450 0x7effb80d05b0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 rx=0x7effb0006f00 tx=0x7effb0009f10).stop 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 -- 172.21.2.117:0/3653299270 shutdown_connections 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb8085450 0x7effb80d05b0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 --2- 172.21.2.117:0/3653299270 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb80e4e40 0x7effb81b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 -- 172.21.2.117:0/3653299270 >> 172.21.2.117:0/3653299270 conn(0x7effb80ce4e0 msgr2=0x7effb80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 -- 172.21.2.117:0/3653299270 shutdown_connections 2021-02-10T11:46:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.923 7effbe6f1700 1 -- 172.21.2.117:0/3653299270 wait complete. 2021-02-10T11:46:25.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.937 7effbe6f1700 1 Processor -- start 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 -- start start 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effb80a6430 0x7effb81ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb81ac750 0x7effb81acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb81abd30 con 0x7effb81ab930 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb81abf10 con 0x7effb80a6430 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effbe6f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb818e5a0 con 0x7effb81ac750 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effb7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effb7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52164/0 (socket says 172.21.2.117:52164) 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.938 7effb7fff700 1 -- 172.21.2.117:0/3589671696 learned_addr learned my addr 172.21.2.117:0/3589671696 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effbcd0e700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb81ac750 0x7effb81acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb77fe700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effb80a6430 0x7effb81ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 -- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb81ac750 msgr2=0x7effb81acb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb81ac750 0x7effb81acb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 -- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effb80a6430 msgr2=0x7effb81ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effb80a6430 0x7effb81ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 -- 172.21.2.117:0/3589671696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb818e7c0 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb7fff700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7effa8006f00 tx=0x7effa800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa800bd30 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa8002030 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa8002510 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb818e7c0 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.939 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb818ed50 con 0x7effb81ab930 2021-02-10T11:46:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.940 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7effa8000c90 con 0x7effb81ab930 2021-02-10T11:46:25.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.940 7effb4ff9700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 0x7eff9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:25.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.940 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7effa80326c0 con 0x7effb81ab930 2021-02-10T11:46:25.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.941 7effb77fe700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 0x7eff9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:25.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.942 7effb77fe700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 0x7eff9801abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:25.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.971 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb810c280 con 0x7effb81ab930 2021-02-10T11:46:25.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:25.974 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7effa8000940 con 0x7effb81ab930 2021-02-10T11:46:26.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.101 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb8085930 con 0x7effb81ab930 2021-02-10T11:46:26.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.102 7effb4ff9700 1 -- 172.21.2.117:0/3589671696 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7effa8000c90 con 0x7effb81ab930 2021-02-10T11:46:26.096 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:26.096 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:26.096 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.103 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 msgr2=0x7eff9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.103 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 0x7eff9801abc0 crc :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.103 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 msgr2=0x7effb80a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.103 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 rx=0x7effa8006f00 tx=0x7effa800b940).stop 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 shutdown_connections 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effb81ac750 0x7effb81acb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effb80a6430 0x7effb81ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff980186c0 0x7eff9801abc0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 --2- 172.21.2.117:0/3589671696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effb81ab930 0x7effb80a5e10 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 >> 172.21.2.117:0/3589671696 conn(0x7effb810de90 msgr2=0x7effb819e360 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 shutdown_connections 2021-02-10T11:46:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:26.104 7effbe6f1700 1 -- 172.21.2.117:0/3589671696 wait complete. 2021-02-10T11:46:26.106 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:26.504 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:26.504 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:26.504 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.505 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:26.506 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:26.506 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:26.506 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:26.506 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:26.506 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 9, 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 32.734602465999998, 2021-02-10T11:46:26.507 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:26.508 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:26.509 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.510 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 19.483722130866553, 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:26.511 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:26.512 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:26.513 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:26.514 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:26.515 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:26.515 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:26.515 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:26.515 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:26.516 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 9, "uptime": 32.734602465999998, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.483722130866553, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:26.517 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:31.518 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:31.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.936 7fcb8378a700 1 Processor -- start 2021-02-10T11:46:31.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.936 7fcb8378a700 1 -- start start 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb8378a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 0x7fcb7c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb8378a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb7c0eb000 0x7fcb7c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb8378a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0ede60 con 0x7fcb7c0ee3e0 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb8378a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c0eba20 con 0x7fcb7c0eb000 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb8378a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0d1ce0 con 0x7fcb7c0d18e0 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb80da5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb7c0eb000 0x7fcb7c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb80da5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb7c0eb000 0x7fcb7c0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52172/0 (socket says 172.21.2.117:52172) 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb80da5700 1 -- 172.21.2.117:0/800160949 learned_addr learned my addr 172.21.2.117:0/800160949 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.937 7fcb815a6700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 0x7fcb7c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb80da5700 1 -- 172.21.2.117:0/800160949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 msgr2=0x7fcb7c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:31.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb80da5700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 0x7fcb7c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb80da5700 1 -- 172.21.2.117:0/800160949 >> v1:172.21.2.125:6789/0 conn(0x7fcb7c0ee3e0 legacy=0x7fcb7c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb80da5700 1 -- 172.21.2.117:0/800160949 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0eefd0 con 0x7fcb7c0eb000 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb815a6700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 0x7fcb7c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb80da5700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c0eb000 0x7fcb7c0eb400 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7fcb70006700 tx=0x7fcb70009b80).ready entity=mon.0 client_cookie=80a999427a756298 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb6ffff700 1 -- 172.21.2.117:0/800160949 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb7000c0c0 con 0x7fcb7c0eb000 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb6ffff700 1 -- 172.21.2.117:0/800160949 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7000c440 con 0x7fcb7c0eb000 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb6ffff700 1 -- 172.21.2.117:0/800160949 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb7000c960 con 0x7fcb7c0eb000 2021-02-10T11:46:31.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb8378a700 1 -- 172.21.2.117:0/800160949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c0eb000 msgr2=0x7fcb7c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.938 7fcb8378a700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c0eb000 0x7fcb7c0eb400 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 rx=0x7fcb70006700 tx=0x7fcb70009b80).stop 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 -- 172.21.2.117:0/800160949 shutdown_connections 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c0eb000 0x7fcb7c0eb400 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 --2- 172.21.2.117:0/800160949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c0d18e0 0x7fcb7c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 -- 172.21.2.117:0/800160949 >> 172.21.2.117:0/800160949 conn(0x7fcb7c0ced60 msgr2=0x7fcb7c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 -- 172.21.2.117:0/800160949 shutdown_connections 2021-02-10T11:46:31.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.939 7fcb8378a700 1 -- 172.21.2.117:0/800160949 wait complete. 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 Processor -- start 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 -- start start 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c091b00 0x7fcb7c091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 0x7fcb7c196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0967f0 con 0x7fcb7c095850 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c1972b0 con 0x7fcb7c096270 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.954 7fcb8378a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c197490 con 0x7fcb7c091b00 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52176/0 (socket says 172.21.2.117:52176) 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 -- 172.21.2.117:0/1472159991 learned_addr learned my addr 172.21.2.117:0/1472159991 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb815a6700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c091b00 0x7fcb7c091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb81da7700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 0x7fcb7c196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 -- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c091b00 msgr2=0x7fcb7c091f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c091b00 0x7fcb7c091f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 -- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 msgr2=0x7fcb7c196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 0x7fcb7c196c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 -- 172.21.2.117:0/1472159991 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c1976b0 con 0x7fcb7c095850 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb81da7700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 0x7fcb7c196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb80da5700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fcb7000b650 tx=0x7fcb7000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb7000bde0 con 0x7fcb7c095850 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb70002030 con 0x7fcb7c095850 2021-02-10T11:46:31.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c1976b0 con 0x7fcb7c095850 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c197c40 con 0x7fcb7c095850 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.955 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb70002550 con 0x7fcb7c095850 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.956 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb70000c90 con 0x7fcb7c095850 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.956 7fcb6dffb700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 0x7fcb5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.956 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb70032990 con 0x7fcb7c095850 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.956 7fcb815a6700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 0x7fcb5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:31.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.957 7fcb815a6700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 0x7fcb5c01abc0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:31.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.987 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c087290 con 0x7fcb7c095850 2021-02-10T11:46:31.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:31.989 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb7001f580 con 0x7fcb7c095850 2021-02-10T11:46:32.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.117 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7c0ede60 con 0x7fcb7c095850 2021-02-10T11:46:32.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.118 7fcb6dffb700 1 -- 172.21.2.117:0/1472159991 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fcb7000bde0 con 0x7fcb7c095850 2021-02-10T11:46:32.112 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:32.112 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:32.113 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:32.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 msgr2=0x7fcb5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:32.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 0x7fcb5c01abc0 crc :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 msgr2=0x7fcb7c095c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 rx=0x7fcb7000b650 tx=0x7fcb7000b9f0).stop 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 shutdown_connections 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb5c0186c0 0x7fcb5c01abc0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb7c095850 0x7fcb7c095c50 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:32.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb7c096270 0x7fcb7c196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:32.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 --2- 172.21.2.117:0/1472159991 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb7c091b00 0x7fcb7c091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:32.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 >> 172.21.2.117:0/1472159991 conn(0x7fcb7c1a3620 msgr2=0x7fcb7c092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:32.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 shutdown_connections 2021-02-10T11:46:32.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:32.120 7fcb8378a700 1 -- 172.21.2.117:0/1472159991 wait complete. 2021-02-10T11:46:32.123 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:32.521 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:32.522 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 8, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 38.751319422999998, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:32.523 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:32.524 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 19.214697406731485, 2021-02-10T11:46:32.525 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:32.526 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:32.527 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:32.528 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:32.528 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:32.528 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:32.528 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:32.529 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:32.529 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:32.529 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:32.529 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:32.529 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:32.530 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 8, "uptime": 38.751319422999998, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 19.214697406731485, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:32.530 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:37.532 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 Processor -- start 2021-02-10T11:46:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 -- start start 2021-02-10T11:46:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7400ef3f0 0x7fc7400eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc7400eb7f0 0x7fc7401b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7400ede60 con 0x7fc7400d18e0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7400ef7f0 con 0x7fc7400eb7f0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc74861f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400d1ce0 con 0x7fc7400ef3f0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc746c3c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc7400eb7f0 0x7fc7401b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc746c3c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc7400eb7f0 0x7fc7401b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52186/0 (socket says 172.21.2.117:52186) 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc746c3c700 1 -- 172.21.2.117:0/14658902 learned_addr learned my addr 172.21.2.117:0/14658902 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.949 7fc745c3a700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7400ef3f0 0x7fc7400eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc746c3c700 1 -- 172.21.2.117:0/14658902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7400ef3f0 msgr2=0x7fc7400eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc746c3c700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7400ef3f0 0x7fc7400eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc746c3c700 1 -- 172.21.2.117:0/14658902 >> v1:172.21.2.125:6789/0 conn(0x7fc7400d18e0 legacy=0x7fc7400ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc746c3c700 1 -- 172.21.2.117:0/14658902 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7401b8f90 con 0x7fc7400eb7f0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc746c3c700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7400eb7f0 0x7fc7401b7960 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fc738006f00 tx=0x7fc738009f10).ready entity=mon.0 client_cookie=12886b6faa697164 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc745439700 1 -- 172.21.2.117:0/14658902 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc73800b3b0 con 0x7fc7400eb7f0 2021-02-10T11:46:37.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc745439700 1 -- 172.21.2.117:0/14658902 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73800b630 con 0x7fc7400eb7f0 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc745439700 1 -- 172.21.2.117:0/14658902 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc73800bb50 con 0x7fc7400eb7f0 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc74861f700 1 -- 172.21.2.117:0/14658902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7400eb7f0 msgr2=0x7fc7401b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.950 7fc74861f700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7400eb7f0 0x7fc7401b7960 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 rx=0x7fc738006f00 tx=0x7fc738009f10).stop 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 -- 172.21.2.117:0/14658902 shutdown_connections 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc7400eb7f0 0x7fc7401b7960 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 --2- 172.21.2.117:0/14658902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7400ef3f0 0x7fc7400eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 -- 172.21.2.117:0/14658902 >> 172.21.2.117:0/14658902 conn(0x7fc7400ced60 msgr2=0x7fc7400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 -- 172.21.2.117:0/14658902 shutdown_connections 2021-02-10T11:46:37.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.951 7fc74861f700 1 -- 172.21.2.117:0/14658902 wait complete. 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 Processor -- start 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 -- start start 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc740091d40 0x7fc7400e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7401b24b0 0x7fc7401b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740197cd0 con 0x7fc740198250 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc740092140 con 0x7fc740091d40 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc74861f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7401b2ed0 con 0x7fc7401b24b0 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52188/0 (socket says 172.21.2.117:52188) 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 -- 172.21.2.117:0/3940854733 learned_addr learned my addr 172.21.2.117:0/3940854733 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:37.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 -- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7401b24b0 msgr2=0x7fc7401b28b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7401b24b0 0x7fc7401b28b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 -- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc740091d40 msgr2=0x7fc7400e71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc740091d40 0x7fc7400e71e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 -- 172.21.2.117:0/3940854733 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740198c70 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.966 7fc746c3c700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7fc738009390 tx=0x7fc73800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc738003030 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7380068e0 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc738001670 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc74861f700 1 -- 172.21.2.117:0/3940854733 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc740198c70 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc74861f700 1 -- 172.21.2.117:0/3940854733 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc740198c70 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc738001270 con 0x7fc740198250 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 0x7fc72801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:37.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.967 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc738032aa0 con 0x7fc740198250 2021-02-10T11:46:37.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.968 7fc74643b700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 0x7fc72801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:37.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.969 7fc74643b700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 0x7fc72801abc0 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:37.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:37.999 7fc74861f700 1 -- 172.21.2.117:0/3940854733 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc740087290 con 0x7fc740198250 2021-02-10T11:46:37.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.001 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc738000cc0 con 0x7fc740198250 2021-02-10T11:46:38.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.129 7fc74861f700 1 -- 172.21.2.117:0/3940854733 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7400ede60 con 0x7fc740198250 2021-02-10T11:46:38.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.129 7fc732ffd700 1 -- 172.21.2.117:0/3940854733 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fc738001270 con 0x7fc740198250 2021-02-10T11:46:38.124 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:38.124 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:38.124 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:38.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 -- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 msgr2=0x7fc72801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:38.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 0x7fc72801abc0 crc :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:38.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 -- 172.21.2.117:0/3940854733 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 msgr2=0x7fc740198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:38.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 rx=0x7fc738009390 tx=0x7fc73800bd40).stop 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 -- 172.21.2.117:0/3940854733 shutdown_connections 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7280186c0 0x7fc72801abc0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc740198250 0x7fc740198650 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc7401b24b0 0x7fc7401b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 --2- 172.21.2.117:0/3940854733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc740091d40 0x7fc7400e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 -- 172.21.2.117:0/3940854733 >> 172.21.2.117:0/3940854733 conn(0x7fc7401a3620 msgr2=0x7fc7400dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.131 7fc74861f700 1 -- 172.21.2.117:0/3940854733 shutdown_connections 2021-02-10T11:46:38.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:38.132 7fc74861f700 1 -- 172.21.2.117:0/3940854733 wait complete. 2021-02-10T11:46:38.134 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:38.531 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:38.532 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 8, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 44.760903648000003, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:38.533 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.534 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 18.949787136928077, 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:38.535 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:38.536 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:38.537 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:38.538 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 8, "uptime": 44.760903648000003, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.949787136928077, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:38.539 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:40.610 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:40.616 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:46:43.540 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:43.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.955 7f999d011700 1 Processor -- start 2021-02-10T11:46:43.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.955 7f999d011700 1 -- start start 2021-02-10T11:46:43.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999d011700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f99980d18e0 0x7f99980ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999d011700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 0x7f99980eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999d011700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980ede60 con 0x7f99980ee3e0 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999d011700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980eba90 con 0x7f99980d18e0 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999d011700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99980d1ce0 con 0x7f99980eb000 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999759e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f99980ee3e0 0x7f99980ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52450/0 (socket says 172.21.2.117:52450) 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999759e700 1 -- 172.21.2.117:0/2376345003 learned_addr learned my addr 172.21.2.117:0/2376345003 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:43.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f999659c700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 0x7f99980eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f9996d9d700 1 --2- 172.21.2.117:0/2376345003 >> v2:172.21.2.125:3300/0 conn(0x7f99980d18e0 0x7f99980ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.956 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359815839 0 0) 0x7f9990001500 con 0x7f99980ee3e0 2021-02-10T11:46:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9984002fe0 con 0x7f99980ee3e0 2021-02-10T11:46:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3616663247 0 0) 0x7f9990001270 con 0x7f99980ee3e0 2021-02-10T11:46:43.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9996d9d700 1 -- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 msgr2=0x7f99980eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9996d9d700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 0x7f99980eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9996d9d700 1 -- 172.21.2.117:0/2376345003 >> v1:172.21.2.117:6789/0 conn(0x7f99980ee3e0 legacy=0x7f99980ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9996d9d700 1 -- 172.21.2.117:0/2376345003 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980eefd0 con 0x7f99980d18e0 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f999659c700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 0x7f99980eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9996d9d700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99980d18e0 0x7f99980ea9e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f9988006650 tx=0x7f9988009a80).ready entity=mon.1 client_cookie=9662783a5bff1636 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:43.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99880009b0 con 0x7f99980d18e0 2021-02-10T11:46:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998800c0c0 con 0x7f99980d18e0 2021-02-10T11:46:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f9995d9b700 1 -- 172.21.2.117:0/2376345003 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f998800c5e0 con 0x7f99980d18e0 2021-02-10T11:46:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f999d011700 1 -- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99980d18e0 msgr2=0x7f99980ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.957 7f999d011700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99980d18e0 0x7f99980ea9e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f9988006650 tx=0x7f9988009a80).stop 2021-02-10T11:46:43.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 -- 172.21.2.117:0/2376345003 shutdown_connections 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980eb000 0x7f99980eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 --2- 172.21.2.117:0/2376345003 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99980d18e0 0x7f99980ea9e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 -- 172.21.2.117:0/2376345003 >> 172.21.2.117:0/2376345003 conn(0x7f99980ced60 msgr2=0x7f99980cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 -- 172.21.2.117:0/2376345003 shutdown_connections 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.958 7f999d011700 1 -- 172.21.2.117:0/2376345003 wait complete. 2021-02-10T11:46:43.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 Processor -- start 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 -- start start 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99981a4290 0x7f99981a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980a6fe0 0x7f99980a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99980a7d10 con 0x7f99980ab280 2021-02-10T11:46:43.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99980a7a00 con 0x7f99981a4290 2021-02-10T11:46:43.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999d011700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9998096370 con 0x7f99980a6fe0 2021-02-10T11:46:43.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52204/0 (socket says 172.21.2.117:52204) 2021-02-10T11:46:43.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.973 7f999659c700 1 -- 172.21.2.117:0/224928025 learned_addr learned my addr 172.21.2.117:0/224928025 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:43.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 -- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980a6fe0 msgr2=0x7f99980a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980a6fe0 0x7f99980a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:43.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 -- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99981a4290 msgr2=0x7f99981a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:43.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99981a4290 0x7f99981a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:43.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 -- 172.21.2.117:0/224928025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980a8290 con 0x7f99980ab280 2021-02-10T11:46:43.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999659c700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f9980006f40 tx=0x7f998000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9980006bb0 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998000bd80 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999d011700 1 -- 172.21.2.117:0/224928025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99980a8290 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9980002220 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.974 7f999d011700 1 -- 172.21.2.117:0/224928025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99980a8290 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.975 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9980006b50 con 0x7f99980ab280 2021-02-10T11:46:43.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.975 7f998f7fe700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 0x7f997c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:43.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.975 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9980032930 con 0x7f99980ab280 2021-02-10T11:46:43.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.975 7f9996d9d700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 0x7f997c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:43.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:43.976 7f9996d9d700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 0x7f997c01abc0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.006 7f999d011700 1 -- 172.21.2.117:0/224928025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9998087290 con 0x7f99980ab280 2021-02-10T11:46:44.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.008 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9980000c90 con 0x7f99980ab280 2021-02-10T11:46:44.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.135 7f999d011700 1 -- 172.21.2.117:0/224928025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99980ede60 con 0x7f99980ab280 2021-02-10T11:46:44.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.135 7f998f7fe700 1 -- 172.21.2.117:0/224928025 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f998000bd80 con 0x7f99980ab280 2021-02-10T11:46:44.131 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:44.131 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:44.131 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 -- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 msgr2=0x7f997c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 0x7f997c01abc0 crc :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 -- 172.21.2.117:0/224928025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 msgr2=0x7f99980ab680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 rx=0x7f9980006f40 tx=0x7f998000b990).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 -- 172.21.2.117:0/224928025 shutdown_connections 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f997c0186c0 0x7f997c01abc0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99980ab280 0x7f99980ab680 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.137 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99980a6fe0 0x7f99980a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.138 7f999d011700 1 --2- 172.21.2.117:0/224928025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99981a4290 0x7f99981a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.138 7f999d011700 1 -- 172.21.2.117:0/224928025 >> 172.21.2.117:0/224928025 conn(0x7f99980874c0 msgr2=0x7f9998096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.138 7f999d011700 1 -- 172.21.2.117:0/224928025 shutdown_connections 2021-02-10T11:46:44.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:44.138 7f999d011700 1 -- 172.21.2.117:0/224928025 wait complete. 2021-02-10T11:46:44.141 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:44.538 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 7, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 50.767495142000001, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:44.539 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:44.540 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 18.688615195606197, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:44.541 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:44.542 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:44.543 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:44.544 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 50.767495142000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.688615195606197, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:44.544 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:49.546 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.962 7fb88d5ec700 1 Processor -- start 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.962 7fb88d5ec700 1 -- start start 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb88d5ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880d18e0 0x7fb8880ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb88d5ec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8880ef3f0 0x7fb8881b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb88d5ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880ede60 con 0x7fb8880eb000 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb88d5ec700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb888088020 con 0x7fb8880ef3f0 2021-02-10T11:46:49.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb88d5ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880d1ce0 con 0x7fb8880d18e0 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8880ef3f0 0x7fb8881b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8880ef3f0 0x7fb8881b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52212/0 (socket says 172.21.2.117:52212) 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 -- 172.21.2.117:0/2863045798 learned_addr learned my addr 172.21.2.117:0/2863045798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb886abe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fb8880eb000 0x7fb8880eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45386/0 (socket says 172.21.2.117:45386) 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb8872bf700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880d18e0 0x7fb8880ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 -- 172.21.2.117:0/2863045798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880d18e0 msgr2=0x7fb8880ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880d18e0 0x7fb8880ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 -- 172.21.2.117:0/2863045798 >> v1:172.21.2.125:6789/0 conn(0x7fb8880eb000 legacy=0x7fb8880eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.963 7fb887ac0700 1 -- 172.21.2.117:0/2863045798 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8881b8f90 con 0x7fb8880ef3f0 2021-02-10T11:46:49.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb887ac0700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8880ef3f0 0x7fb8881b7960 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 rx=0x7fb878006b80 tx=0x7fb878009b90).ready entity=mon.0 client_cookie=c91dcf9be3de35e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb8862bd700 1 -- 172.21.2.117:0/2863045798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8780022e0 con 0x7fb8880ef3f0 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb8862bd700 1 -- 172.21.2.117:0/2863045798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878002620 con 0x7fb8880ef3f0 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb8862bd700 1 -- 172.21.2.117:0/2863045798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb878002b00 con 0x7fb8880ef3f0 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 -- 172.21.2.117:0/2863045798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8880ef3f0 msgr2=0x7fb8881b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8880ef3f0 0x7fb8881b7960 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 rx=0x7fb878006b80 tx=0x7fb878009b90).stop 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 -- 172.21.2.117:0/2863045798 shutdown_connections 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8880ef3f0 0x7fb8881b7960 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 --2- 172.21.2.117:0/2863045798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880d18e0 0x7fb8880ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:49.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 -- 172.21.2.117:0/2863045798 >> 172.21.2.117:0/2863045798 conn(0x7fb8880ced60 msgr2=0x7fb8880cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:49.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 -- 172.21.2.117:0/2863045798 shutdown_connections 2021-02-10T11:46:49.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.964 7fb88d5ec700 1 -- 172.21.2.117:0/2863045798 wait complete. 2021-02-10T11:46:49.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 Processor -- start 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 -- start start 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8881afd30 0x7fb8880b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880b43f0 0x7fb8880e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880e80e0 con 0x7fb8881b0130 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880e7e20 con 0x7fb8881afd30 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.979 7fb88d5ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb888096a70 con 0x7fb8880b43f0 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52218/0 (socket says 172.21.2.117:52218) 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 -- 172.21.2.117:0/1266289836 learned_addr learned my addr 172.21.2.117:0/1266289836 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 -- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880b43f0 msgr2=0x7fb8880e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880b43f0 0x7fb8880e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 -- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8881afd30 msgr2=0x7fb8880b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8881afd30 0x7fb8880b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 -- 172.21.2.117:0/1266289836 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8880e8660 con 0x7fb8881b0130 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb887ac0700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fb878006f10 tx=0x7fb87800c930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb878000b90 con 0x7fb8881b0130 2021-02-10T11:46:49.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87800cd20 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8880e8660 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8880e8660 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.980 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8780022b0 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.981 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb878009690 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.981 7fb86ffff700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 0x7fb86801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.981 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb8780326e0 con 0x7fb8881b0130 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.982 7fb8872bf700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 0x7fb86801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:49.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:49.982 7fb8872bf700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 0x7fb86801abc0 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:50.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.013 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8880e7e20 con 0x7fb8881b0130 2021-02-10T11:46:50.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.015 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb878009310 con 0x7fb8881b0130 2021-02-10T11:46:50.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.143 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8880e7e20 con 0x7fb8881b0130 2021-02-10T11:46:50.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.143 7fb86ffff700 1 -- 172.21.2.117:0/1266289836 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb878009690 con 0x7fb8881b0130 2021-02-10T11:46:50.139 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:50.139 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:50.139 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:50.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 msgr2=0x7fb86801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:50.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 0x7fb86801abc0 crc :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 msgr2=0x7fb8880e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 rx=0x7fb878006f10 tx=0x7fb87800c930).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 shutdown_connections 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.145 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8680186c0 0x7fb86801abc0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8881b0130 0x7fb8880e7800 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8880b43f0 0x7fb8880e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 --2- 172.21.2.117:0/1266289836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8881afd30 0x7fb8880b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:46:50.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 >> 172.21.2.117:0/1266289836 conn(0x7fb8881a3620 msgr2=0x7fb8880dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:50.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 shutdown_connections 2021-02-10T11:46:50.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:50.146 7fb88d5ec700 1 -- 172.21.2.117:0/1266289836 wait complete. 2021-02-10T11:46:50.149 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:50.545 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:50.546 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 7, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 56.774093526000001, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:50.547 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 18.431042514920019, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:50.548 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:50.549 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:50.550 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:50.551 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:50.551 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:50.551 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:50.552 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 7, "uptime": 56.774093526000001, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.431042514920019, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:50.552 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:46:55.553 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:46:55.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 Processor -- start 2021-02-10T11:46:55.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 -- start start 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1800d18e0 0x7fb1800ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800ede60 con 0x7fb1800ef3f0 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800d1ce0 con 0x7fb1800eb000 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.972 7fb184f76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb180088020 con 0x7fb1800d18e0 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17f314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb1800ef3f0 0x7fb1801b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52474/0 (socket says 172.21.2.117:52474) 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17f314700 1 -- 172.21.2.117:0/2524603512 learned_addr learned my addr 172.21.2.117:0/2524603512 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:46:55.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605711694 0 0) 0x7fb178001500 con 0x7fb1800ef3f0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb168002fe0 con 0x7fb1800ef3f0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1800d18e0 0x7fb1800ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2727758246 0 0) 0x7fb178001460 con 0x7fb1800ef3f0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1800d18e0 msgr2=0x7fb1800ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1800d18e0 0x7fb1800ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 >> v1:172.21.2.125:6789/0 conn(0x7fb1800eb000 legacy=0x7fb1800eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1801bb180 con 0x7fb1800ef3f0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fb178001980 con 0x7fb1800ef3f0 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 >> v1:172.21.2.117:6789/0 conn(0x7fb1800ef3f0 legacy=0x7fb1801b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb168003ae0 0x7fb168005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb168006810 0x7fb168008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb168009630 0x7fb16800bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16800c370 con 0x7fb168003ae0 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16800c590 con 0x7fb168006810 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17db11700 1 -- 172.21.2.117:0/2524603512 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16800c7b0 con 0x7fb168009630 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.973 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1800d18e0 0x7fb1800ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb168003ae0 0x7fb168005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17e312700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb168006810 0x7fb168008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17f314700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb168009630 0x7fb16800bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 -- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb168009630 msgr2=0x7fb16800bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb168009630 0x7fb16800bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 -- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb168006810 msgr2=0x7fb168008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb168006810 0x7fb168008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:55.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 -- 172.21.2.117:0/2524603512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16800ca50 con 0x7fb168003ae0 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17eb13700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb168003ae0 0x7fb168005f90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7fb1700064c0 tx=0x7fb17004dac0).ready entity=mon.0 client_cookie=90396add924a9c02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1700500c0 con 0x7fb168003ae0 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170050340 con 0x7fb168003ae0 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17db11700 1 -- 172.21.2.117:0/2524603512 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1700507b0 con 0x7fb168003ae0 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb184f76700 1 -- 172.21.2.117:0/2524603512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb168003ae0 msgr2=0x7fb168005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb184f76700 1 --2- 172.21.2.117:0/2524603512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb168003ae0 0x7fb168005f90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7fb1700064c0 tx=0x7fb17004dac0).stop 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.974 7fb17f314700 1 -- 172.21.2.117:0/2524603512 reap_dead start 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.975 7fb184f76700 1 -- 172.21.2.117:0/2524603512 shutdown_connections 2021-02-10T11:46:55.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.975 7fb184f76700 1 -- 172.21.2.117:0/2524603512 >> 172.21.2.117:0/2524603512 conn(0x7fb1800ced60 msgr2=0x7fb1800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.975 7fb184f76700 1 -- 172.21.2.117:0/2524603512 shutdown_connections 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.975 7fb184f76700 1 -- 172.21.2.117:0/2524603512 wait complete. 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 Processor -- start 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 -- start start 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 0x7fb1801a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1800b82f0 0x7fb1800b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800c0670 con 0x7fb1800b8d10 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800deb20 con 0x7fb1800b82f0 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb184f76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800ded40 con 0x7fb1801a33d0 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.990 7fb17f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52236/0 (socket says 172.21.2.117:52236) 2021-02-10T11:46:55.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 -- 172.21.2.117:0/1215133230 learned_addr learned my addr 172.21.2.117:0/1215133230 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17e312700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1800b82f0 0x7fb1800b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17eb13700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 0x7fb1801a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 -- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 msgr2=0x7fb1801a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 0x7fb1801a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 -- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1800b82f0 msgr2=0x7fb1800b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1800b82f0 0x7fb1800b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 -- 172.21.2.117:0/1215133230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800def60 con 0x7fb1800b8d10 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17eb13700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 0x7fb1801a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb17f314700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 rx=0x7fb17800a8f0 tx=0x7fb17800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb178001e70 con 0x7fb1800b8d10 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb178004930 con 0x7fb1800b8d10 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb178004dd0 con 0x7fb1800b8d10 2021-02-10T11:46:55.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb184f76700 1 -- 172.21.2.117:0/1215133230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1800def60 con 0x7fb1800b8d10 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.991 7fb184f76700 1 -- 172.21.2.117:0/1215133230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1800df4f0 con 0x7fb1800b8d10 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.992 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb178001d00 con 0x7fb1800b8d10 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.992 7fb16f7fe700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 0x7fb16401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.992 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb178032a10 con 0x7fb1800b8d10 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.992 7fb17eb13700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 0x7fb16401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:46:55.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:55.993 7fb17eb13700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 0x7fb16401abc0 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:46:56.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.022 7fb184f76700 1 -- 172.21.2.117:0/1215133230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb180087290 con 0x7fb1800b8d10 2021-02-10T11:46:56.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.025 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb178026650 con 0x7fb1800b8d10 2021-02-10T11:46:56.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.151 7fb184f76700 1 -- 172.21.2.117:0/1215133230 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1800ede60 con 0x7fb1800b8d10 2021-02-10T11:46:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.151 7fb16f7fe700 1 -- 172.21.2.117:0/1215133230 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb178026650 con 0x7fb1800b8d10 2021-02-10T11:46:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:46:56.146 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:46:56.147 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 msgr2=0x7fb16401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 0x7fb16401abc0 crc :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 msgr2=0x7fb1800b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 rx=0x7fb17800a8f0 tx=0x7fb17800ac90).stop 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 shutdown_connections 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1640186c0 0x7fb16401abc0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:56.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb1800b8d10 0x7fb1800b9110 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:56.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb1800b82f0 0x7fb1800b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:56.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 --2- 172.21.2.117:0/1215133230 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb1801a33d0 0x7fb1801a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:46:56.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 >> 172.21.2.117:0/1215133230 conn(0x7fb1800874c0 msgr2=0x7fb180096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:46:56.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 shutdown_connections 2021-02-10T11:46:56.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:46:56.153 7fb184f76700 1 -- 172.21.2.117:0/1215133230 wait complete. 2021-02-10T11:46:56.157 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:46:56.554 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:46:56.555 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 6, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 62.782698516000004, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:56.556 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.557 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 18.17693551326829, 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:46:56.558 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:46:56.559 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:46:56.560 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:46:56.561 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 6, "uptime": 62.782698516000004, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 18.17693551326829, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:46:56.561 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:47:01.562 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.979 7fa416f67700 1 Processor -- start 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.979 7fa416f67700 1 -- start start 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa416f67700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100d18e0 0x7fa4100ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa416f67700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4100eb000 0x7fa4100eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa416f67700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4100ede60 con 0x7fa4100ee3e0 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa416f67700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4100eba20 con 0x7fa4100eb000 2021-02-10T11:47:01.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa416f67700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4100d1ce0 con 0x7fa4100d18e0 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4100eb000 0x7fa4100eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa4100eb000 0x7fa4100eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52240/0 (socket says 172.21.2.117:52240) 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 -- 172.21.2.117:0/3108524514 learned_addr learned my addr 172.21.2.117:0/3108524514 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 -- 172.21.2.117:0/3108524514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100d18e0 msgr2=0x7fa4100ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 --2- 172.21.2.117:0/3108524514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100d18e0 0x7fa4100ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 -- 172.21.2.117:0/3108524514 >> v1:172.21.2.125:6789/0 conn(0x7fa4100ee3e0 legacy=0x7fa4100ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 -- 172.21.2.117:0/3108524514 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100eefd0 con 0x7fa4100eb000 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.980 7fa40ffff700 1 --2- 172.21.2.117:0/3108524514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4100eb000 0x7fa4100eb400 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7fa400006a80 tx=0x7fa400009f00).ready entity=mon.0 client_cookie=309768ba865c6a8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa40f7fe700 1 -- 172.21.2.117:0/3108524514 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40000b5c0 con 0x7fa4100eb000 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa40f7fe700 1 -- 172.21.2.117:0/3108524514 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40000b8c0 con 0x7fa4100eb000 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa40f7fe700 1 -- 172.21.2.117:0/3108524514 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa40000bde0 con 0x7fa4100eb000 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 -- 172.21.2.117:0/3108524514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4100eb000 msgr2=0x7fa4100eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:01.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 --2- 172.21.2.117:0/3108524514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4100eb000 0x7fa4100eb400 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 rx=0x7fa400006a80 tx=0x7fa400009f00).stop 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 -- 172.21.2.117:0/3108524514 shutdown_connections 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 --2- 172.21.2.117:0/3108524514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4100eb000 0x7fa4100eb400 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 --2- 172.21.2.117:0/3108524514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100d18e0 0x7fa4100ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.981 7fa416f67700 1 -- 172.21.2.117:0/3108524514 >> 172.21.2.117:0/3108524514 conn(0x7fa4100ced60 msgr2=0x7fa4100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.982 7fa416f67700 1 -- 172.21.2.117:0/3108524514 shutdown_connections 2021-02-10T11:47:01.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.982 7fa416f67700 1 -- 172.21.2.117:0/3108524514 wait complete. 2021-02-10T11:47:01.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.996 7fa416f67700 1 Processor -- start 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.996 7fa416f67700 1 -- start start 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 0x7fa4101a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100df870 0x7fa4100dfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101a9340 con 0x7fa4101aa550 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101aa990 con 0x7fa4101a3bf0 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa416f67700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4101a9e60 con 0x7fa4100df870 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52246/0 (socket says 172.21.2.117:52246) 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 -- 172.21.2.117:0/1852084227 learned_addr learned my addr 172.21.2.117:0/1852084227 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa415584700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100df870 0x7fa4100dfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:01.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa40ffff700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 0x7fa4101a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 -- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100df870 msgr2=0x7fa4100dfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100df870 0x7fa4100dfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 -- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 msgr2=0x7fa4101a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 0x7fa4101a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa414d83700 1 -- 172.21.2.117:0/1852084227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100e0290 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.997 7fa40ffff700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 0x7fa4101a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa414d83700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7fa3f8006f00 tx=0x7fa3f800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3f8006350 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f800be00 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa416f67700 1 -- 172.21.2.117:0/1852084227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4100e0290 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa416f67700 1 -- 172.21.2.117:0/1852084227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4100e0820 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.998 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3f8002020 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.999 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3f8005310 con 0x7fa4101aa550 2021-02-10T11:47:01.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.999 7fa40d7fa700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 0x7fa3f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:01.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.999 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa3f80328f0 con 0x7fa4101aa550 2021-02-10T11:47:01.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.999 7fa40ffff700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 0x7fa3f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:01.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:01.999 7fa40ffff700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 0x7fa3f401abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:02.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.030 7fa416f67700 1 -- 172.21.2.117:0/1852084227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa410087290 con 0x7fa4101aa550 2021-02-10T11:47:02.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.032 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3f8000960 con 0x7fa4101aa550 2021-02-10T11:47:02.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.159 7fa416f67700 1 -- 172.21.2.117:0/1852084227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa410087290 con 0x7fa4101aa550 2021-02-10T11:47:02.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.159 7fa40d7fa700 1 -- 172.21.2.117:0/1852084227 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa3f800be00 con 0x7fa4101aa550 2021-02-10T11:47:02.154 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:02.154 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:02.154 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:47:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 msgr2=0x7fa3f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 0x7fa3f401abc0 crc :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 msgr2=0x7fa4101a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:02.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 rx=0x7fa3f8006f00 tx=0x7fa3f800b940).stop 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 shutdown_connections 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa4100df870 0x7fa4100dfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa4101a3bf0 0x7fa4101a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3f40186c0 0x7fa3f401abc0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 --2- 172.21.2.117:0/1852084227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa4101aa550 0x7fa4101a9840 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 >> 172.21.2.117:0/1852084227 conn(0x7fa4101a3620 msgr2=0x7fa4100e0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 shutdown_connections 2021-02-10T11:47:02.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:02.161 7fa416f67700 1 -- 172.21.2.117:0/1852084227 wait complete. 2021-02-10T11:47:02.164 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:47:02.560 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:47:02.561 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 6, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 68.789310215, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:47:02.562 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:47:02.563 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 17.926414419842111, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:47:02.564 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:47:02.565 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:02.566 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:47:02.567 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 6, "uptime": 68.789310215, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.926414419842111, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:47:02.567 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:47:07.568 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.984 7f7cd595b700 1 Processor -- start 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.984 7f7cd595b700 1 -- start start 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cd595b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 0x7f7cd00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cd595b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7cd00eb000 0x7f7cd00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cd595b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd00ede60 con 0x7f7cd00ee3e0 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cd595b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd00eba20 con 0x7f7cd00eb000 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cd595b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00d1ce0 con 0x7f7cd00d18e0 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cced73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7cd00eb000 0x7f7cd00eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7ccfd75700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f7cd00ee3e0 0x7f7cd00ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45430/0 (socket says 172.21.2.117:45430) 2021-02-10T11:47:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7cced73700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7cd00eb000 0x7f7cd00eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:56640/0 (socket says 172.21.2.117:56640) 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7ccf574700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 0x7f7cd00ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.985 7f7ccfd75700 1 -- 172.21.2.117:0/1489316715 learned_addr learned my addr 172.21.2.117:0/1489316715 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cced73700 1 -- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 msgr2=0x7f7cd00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cced73700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 0x7f7cd00ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cced73700 1 -- 172.21.2.117:0/1489316715 >> v1:172.21.2.125:6789/0 conn(0x7f7cd00ee3e0 legacy=0x7f7cd00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cced73700 1 -- 172.21.2.117:0/1489316715 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00d34b0 con 0x7f7cd00eb000 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7ccf574700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 0x7f7cd00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cced73700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd00eb000 0x7f7cd00eb400 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f7cc0006650 tx=0x7f7cc0009a80).ready entity=mon.1 client_cookie=774664b67fe442d8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cce572700 1 -- 172.21.2.117:0/1489316715 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc00009b0 con 0x7f7cd00eb000 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cce572700 1 -- 172.21.2.117:0/1489316715 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc000c0c0 con 0x7f7cd00eb000 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cce572700 1 -- 172.21.2.117:0/1489316715 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc000c5e0 con 0x7f7cd00eb000 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cd595b700 1 -- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd00eb000 msgr2=0x7f7cd00eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.986 7f7cd595b700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd00eb000 0x7f7cd00eb400 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f7cc0006650 tx=0x7f7cc0009a80).stop 2021-02-10T11:47:07.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 -- 172.21.2.117:0/1489316715 shutdown_connections 2021-02-10T11:47:07.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd00eb000 0x7f7cd00eb400 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 --2- 172.21.2.117:0/1489316715 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd00d18e0 0x7f7cd00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:07.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 -- 172.21.2.117:0/1489316715 >> 172.21.2.117:0/1489316715 conn(0x7f7cd00ced60 msgr2=0x7f7cd00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:07.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 -- 172.21.2.117:0/1489316715 shutdown_connections 2021-02-10T11:47:07.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:07.987 7f7cd595b700 1 -- 172.21.2.117:0/1489316715 wait complete. 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 Processor -- start 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 -- start start 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd0091b00 0x7f7cd0091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd0095850 0x7f7cd0095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00967f0 con 0x7f7cd0096270 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01972b0 con 0x7f7cd0091b00 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.002 7f7cd595b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd0197490 con 0x7f7cd0095850 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52262/0 (socket says 172.21.2.117:52262) 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 -- 172.21.2.117:0/3719377811 learned_addr learned my addr 172.21.2.117:0/3719377811 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7cced73700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd0095850 0x7f7cd0095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccf574700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd0091b00 0x7f7cd0091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 -- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd0095850 msgr2=0x7f7cd0095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd0095850 0x7f7cd0095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 -- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd0091b00 msgr2=0x7f7cd0091f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd0091b00 0x7f7cd0091f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 -- 172.21.2.117:0/3719377811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd01976b0 con 0x7f7cd0096270 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7ccfd75700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f7cc800a8f0 tx=0x7f7cc800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc80070d0 con 0x7f7cd0096270 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc80078e0 con 0x7f7cd0096270 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd01976b0 con 0x7f7cd0096270 2021-02-10T11:47:08.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.003 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cc8007580 con 0x7f7cd0096270 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.004 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd0197c40 con 0x7f7cd0096270 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.004 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7cc8007350 con 0x7f7cd0096270 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.004 7f7cc7fff700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 0x7f7cb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.004 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7cc8032930 con 0x7f7cd0096270 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.004 7f7ccf574700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 0x7f7cb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:08.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.005 7f7ccf574700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 0x7f7cb401abc0 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:08.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.034 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd0087290 con 0x7f7cd0096270 2021-02-10T11:47:08.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.037 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7cc80048b0 con 0x7f7cd0096270 2021-02-10T11:47:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.164 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cd00ede60 con 0x7f7cd0096270 2021-02-10T11:47:08.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.164 7f7cc7fff700 1 -- 172.21.2.117:0/3719377811 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f7cc8007330 con 0x7f7cd0096270 2021-02-10T11:47:08.159 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:47:08.159 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:08.159 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 msgr2=0x7f7cb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 0x7f7cb401abc0 crc :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 msgr2=0x7f7cd0196c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:08.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 rx=0x7f7cc800a8f0 tx=0x7f7cc800ac90).stop 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 shutdown_connections 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb40186c0 0x7f7cb401abc0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd0096270 0x7f7cd0196c90 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd0095850 0x7f7cd0095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 --2- 172.21.2.117:0/3719377811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd0091b00 0x7f7cd0091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.166 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 >> 172.21.2.117:0/3719377811 conn(0x7f7cd01a3620 msgr2=0x7f7cd0092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.167 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 shutdown_connections 2021-02-10T11:47:08.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:08.167 7f7cd595b700 1 -- 172.21.2.117:0/3719377811 wait complete. 2021-02-10T11:47:08.169 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:47:08.565 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:47:08.565 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:08.566 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 5, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 74.792928525999997, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.567 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 17.679468372875714, 2021-02-10T11:47:08.568 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:08.569 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:47:08.570 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:47:08.571 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:47:08.571 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:47:08.571 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:08.571 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:08.571 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:47:08.572 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 5, "uptime": 74.792928525999997, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.679468372875714, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:47:08.572 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:47:11.045 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:11.048 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:13.573 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:14.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.988 7fe6c8ee9700 1 Processor -- start 2021-02-10T11:47:14.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.988 7fe6c8ee9700 1 -- start start 2021-02-10T11:47:14.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c8ee9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c8ee9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c40eb7f0 0x7fe6c41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c8ee9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c40ede60 con 0x7fe6c40d18e0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c8ee9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c40ef7f0 con 0x7fe6c40ef3f0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c8ee9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c40d1ce0 con 0x7fe6c40eb7f0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52266/0 (socket says 172.21.2.117:52266) 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 -- 172.21.2.117:0/701088935 learned_addr learned my addr 172.21.2.117:0/701088935 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c3314700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c40eb7f0 0x7fe6c41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 -- 172.21.2.117:0/701088935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c40eb7f0 msgr2=0x7fe6c41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c40eb7f0 0x7fe6c41b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 -- 172.21.2.117:0/701088935 >> v1:172.21.2.125:6789/0 conn(0x7fe6c40d18e0 legacy=0x7fe6c40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.989 7fe6c2312700 1 -- 172.21.2.117:0/701088935 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c41b8f90 con 0x7fe6c40ef3f0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c2312700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7fe6b4006700 tx=0x7fe6b4009b80).ready entity=mon.0 client_cookie=57fe76a2a64514fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c1b11700 1 -- 172.21.2.117:0/701088935 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6b400c0c0 con 0x7fe6c40ef3f0 2021-02-10T11:47:14.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c1b11700 1 -- 172.21.2.117:0/701088935 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b400c440 con 0x7fe6c40ef3f0 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c1b11700 1 -- 172.21.2.117:0/701088935 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6b400c960 con 0x7fe6c40ef3f0 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c8ee9700 1 -- 172.21.2.117:0/701088935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c40ef3f0 msgr2=0x7fe6c40eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.990 7fe6c8ee9700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 rx=0x7fe6b4006700 tx=0x7fe6b4009b80).stop 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 -- 172.21.2.117:0/701088935 shutdown_connections 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c40eb7f0 0x7fe6c41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 --2- 172.21.2.117:0/701088935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c40ef3f0 0x7fe6c40eb1d0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 -- 172.21.2.117:0/701088935 >> 172.21.2.117:0/701088935 conn(0x7fe6c40ced60 msgr2=0x7fe6c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 -- 172.21.2.117:0/701088935 shutdown_connections 2021-02-10T11:47:14.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:13.991 7fe6c8ee9700 1 -- 172.21.2.117:0/701088935 wait complete. 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.005 7fe6c8ee9700 1 Processor -- start 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.005 7fe6c8ee9700 1 -- start start 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 0x7fe6c409b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c409b6e0 0x7fe6c40b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c409a420 con 0x7fe6c408a280 2021-02-10T11:47:14.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c409bae0 con 0x7fe6c409acc0 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c8ee9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c409bd00 con 0x7fe6c409b6e0 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52276/0 (socket says 172.21.2.117:52276) 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 -- 172.21.2.117:0/84982798 learned_addr learned my addr 172.21.2.117:0/84982798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 -- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c409b6e0 msgr2=0x7fe6c40b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c409b6e0 0x7fe6c40b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2b13700 1 -- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 msgr2=0x7fe6c409b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.006 7fe6c2312700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 0x7fe6c409b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c2b13700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 0x7fe6c409b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c2b13700 1 -- 172.21.2.117:0/84982798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c40b3f00 con 0x7fe6c408a280 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c2312700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 0x7fe6c409b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c2b13700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7fe6ac00b530 tx=0x7fe6ac00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6ac00bcc0 con 0x7fe6c408a280 2021-02-10T11:47:14.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac006020 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c40b3f00 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c40b4490 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.007 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe6ac0054a0 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.008 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6ac006260 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.008 7fe6bb7fe700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 0x7fe6a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.008 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe6ac032960 con 0x7fe6c408a280 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.008 7fe6c2312700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 0x7fe6a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:14.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.009 7fe6c2312700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 0x7fe6a801abd0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:14.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.039 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c4087290 con 0x7fe6c408a280 2021-02-10T11:47:14.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.041 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe6ac006000 con 0x7fe6c408a280 2021-02-10T11:47:14.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.169 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6c40ede60 con 0x7fe6c408a280 2021-02-10T11:47:14.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.169 7fe6bb7fe700 1 -- 172.21.2.117:0/84982798 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fe6ac000940 con 0x7fe6c408a280 2021-02-10T11:47:14.165 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:14.165 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:14.165 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:47:14.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 msgr2=0x7fe6a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 0x7fe6a801abd0 crc :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 msgr2=0x7fe6c4099e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:14.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 rx=0x7fe6ac00b530 tx=0x7fe6ac00b8d0).stop 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 shutdown_connections 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6c409b6e0 0x7fe6c40b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6c409acc0 0x7fe6c409b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6a80186d0 0x7fe6a801abd0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 --2- 172.21.2.117:0/84982798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe6c408a280 0x7fe6c4099e00 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 >> 172.21.2.117:0/84982798 conn(0x7fe6c40874c0 msgr2=0x7fe6c4091940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 shutdown_connections 2021-02-10T11:47:14.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:14.171 7fe6c8ee9700 1 -- 172.21.2.117:0/84982798 wait complete. 2021-02-10T11:47:14.175 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout:[ 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6443, 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "entity": { 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "name": { 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "client", 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "num": 6443 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": { 2021-02-10T11:47:14.574 INFO:teuthology.orchestra.run.gibba025.stdout: "type": "any", 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "addr": "172.21.2.125:0", 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "nonce": 1629621976 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "open", 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "num_leases": 0, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "num_caps": 7, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "request_load_avg": 5, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 80.802552735000006, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "requests_in_flight": 0, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": 1, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "reconnecting": false, 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps": { 2021-02-10T11:47:14.575 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "release_caps": { 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 60 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle": { 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 2.5 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "recall_caps_throttle2o": { 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:14.576 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 0.5 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "session_cache_liveness": { 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 17.435682192009736, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 300 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "cap_acquisition": { 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "value": 0, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "halflife": 10 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: }, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "inst": "client.6443 172.21.2.125:0/1629621976", 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "completed_requests": [ 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "tid": 14, 2021-02-10T11:47:14.577 INFO:teuthology.orchestra.run.gibba025.stdout: "created_ino": "0x10000000005" 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "prealloc_inos": [ 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: { 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "start": 1099511627782, 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "length": 995 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: ], 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "used_inos": [], 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "client_metadata": { 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "features": "0x0000000000001fff", 2021-02-10T11:47:14.578 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: "entity_id": "x", 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: "hostname": "gibba025", 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: "pid": "34993", 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: "root": "/" 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout: } 2021-02-10T11:47:14.579 INFO:teuthology.orchestra.run.gibba025.stdout:] 2021-02-10T11:47:14.580 INFO:tasks.cephfs.filesystem:_json_asok output: [ { "id": 6443, "entity": { "name": { "type": "client", "num": 6443 }, "addr": { "type": "any", "addr": "172.21.2.125:0", "nonce": 1629621976 } }, "state": "open", "num_leases": 0, "num_caps": 7, "request_load_avg": 5, "uptime": 80.802552735000006, "requests_in_flight": 0, "completed_requests": 1, "reconnecting": false, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 0, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 2.5 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 17.435682192009736, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 10 }, "inst": "client.6443 172.21.2.125:0/1629621976", "completed_requests": [ { "tid": 14, "created_ino": "0x10000000005" } ], "prealloc_inos": [ { "start": 1099511627782, "length": 995 } ], "used_inos": [], "client_metadata": { "features": "0x0000000000001fff", "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", "entity_id": "x", "hostname": "gibba025", "pid": "34993", "root": "/" } } ] 2021-02-10T11:47:14.580 DEBUG:tasks.ceph_test_case:wait_until_true: waiting... 2021-02-10T11:47:19.581 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:20.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 Processor -- start 2021-02-10T11:47:20.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 -- start start 2021-02-10T11:47:20.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1ecc0d18e0 0x7f1ecc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc0eb000 0x7f1ecc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ecc0ede60 con 0x7f1ecc0ee3e0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ecc0eba90 con 0x7f1ecc0d18e0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.997 7f1ed136a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc0d1ce0 con 0x7f1ecc0eb000 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb87e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1ecc0ee3e0 0x7f1ecc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52530/0 (socket says 172.21.2.117:52530) 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb87e700 1 -- 172.21.2.117:0/3459711881 learned_addr learned my addr 172.21.2.117:0/3459711881 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 --2- 172.21.2.117:0/3459711881 >> v2:172.21.2.117:3300/0 conn(0x7f1ecc0d18e0 0x7f1ecc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca87c700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc0eb000 0x7f1ecc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca07b700 1 -- 172.21.2.117:0/3459711881 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581361576 0 0) 0x7f1ec4001500 con 0x7f1ecc0ee3e0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca07b700 1 -- 172.21.2.117:0/3459711881 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb8002fe0 con 0x7f1ecc0ee3e0 2021-02-10T11:47:20.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 -- 172.21.2.117:0/3459711881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc0eb000 msgr2=0x7f1ecc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc0eb000 0x7f1ecc0eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 -- 172.21.2.117:0/3459711881 >> v1:172.21.2.117:6789/0 conn(0x7f1ecc0ee3e0 legacy=0x7f1ecc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 -- 172.21.2.117:0/3459711881 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc0eefd0 con 0x7f1ecc0d18e0 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ecb07d700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0d18e0 0x7f1ecc0ea9e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f1ebc006650 tx=0x7f1ebc009a80).ready entity=mon.0 client_cookie=5f74d8f5e2ce019f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca07b700 1 -- 172.21.2.117:0/3459711881 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ebc00c0c0 con 0x7f1ecc0d18e0 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca07b700 1 -- 172.21.2.117:0/3459711881 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc00c340 con 0x7f1ecc0d18e0 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1eca07b700 1 -- 172.21.2.117:0/3459711881 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ebc00c860 con 0x7f1ecc0d18e0 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ed136a700 1 -- 172.21.2.117:0/3459711881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0d18e0 msgr2=0x7f1ecc0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.998 7f1ed136a700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0d18e0 0x7f1ecc0ea9e0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 rx=0x7f1ebc006650 tx=0x7f1ebc009a80).stop 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 -- 172.21.2.117:0/3459711881 shutdown_connections 2021-02-10T11:47:20.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc0eb000 0x7f1ecc0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 --2- 172.21.2.117:0/3459711881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0d18e0 0x7f1ecc0ea9e0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 -- 172.21.2.117:0/3459711881 >> 172.21.2.117:0/3459711881 conn(0x7f1ecc0ced60 msgr2=0x7f1ecc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 -- 172.21.2.117:0/3459711881 shutdown_connections 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:19.999 7f1ed136a700 1 -- 172.21.2.117:0/3459711881 wait complete. 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 Processor -- start 2021-02-10T11:47:20.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 -- start start 2021-02-10T11:47:20.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 0x7f1ecc0ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 0x7f1ecc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc0c10f0 con 0x7f1ecc0ae470 2021-02-10T11:47:20.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc0ae9a0 con 0x7f1ecc0addc0 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ed136a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ecc0aebc0 con 0x7f1ecc1a0240 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ecb87e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ecb07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 0x7f1ecc0ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ecb87e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:49962/0 (socket says 172.21.2.117:49962) 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1ecb87e700 1 -- 172.21.2.117:0/2171873754 learned_addr learned my addr 172.21.2.117:0/2171873754 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.014 7f1eca87c700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 0x7f1ecc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 -- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 msgr2=0x7f1ecc0c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 -- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 msgr2=0x7f1ecc19fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 0x7f1ecc19fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 -- 172.21.2.117:0/2171873754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc1a0640 con 0x7f1ecc0ae470 2021-02-10T11:47:20.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb87e700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1eca87c700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 0x7f1ecc19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ecb07d700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 0x7f1ecc0ad7a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 rx=0x7f1ebc00b530 tx=0x7f1ebc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ebc000c90 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc002030 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ebc002480 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc1a0640 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.015 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ecc1a0640 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.016 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ebc002210 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.016 7f1ec3fff700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 0x7f1eb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.016 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1ebc0326b0 con 0x7f1ecc0ae470 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.016 7f1eca87c700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 0x7f1eb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:20.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.016 7f1eca87c700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 0x7f1eb001abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:20.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.047 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ecc087290 con 0x7f1ecc0ae470 2021-02-10T11:47:20.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.050 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ebc000960 con 0x7f1ecc0ae470 2021-02-10T11:47:20.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.178 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ecc087290 con 0x7f1ecc0ae470 2021-02-10T11:47:20.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.178 7f1ec3fff700 1 -- 172.21.2.117:0/2171873754 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1ebc002210 con 0x7f1ecc0ae470 2021-02-10T11:47:20.174 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:20.174 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:20.174 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:47:20.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 msgr2=0x7f1eb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 0x7f1eb001abc0 crc :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 msgr2=0x7f1ecc0ad7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 0x7f1ecc0ad7a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 rx=0x7f1ebc00b530 tx=0x7f1ebc00b8d0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 shutdown_connections 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ecc1a0240 0x7f1ecc0c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ecc0addc0 0x7f1ecc19fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1eb00186c0 0x7f1eb001abc0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 --2- 172.21.2.117:0/2171873754 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ecc0ae470 0x7f1ecc0ad7a0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 >> 172.21.2.117:0/2171873754 conn(0x7f1ecc09eb70 msgr2=0x7f1ecc0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 shutdown_connections 2021-02-10T11:47:20.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:20.180 7f1ed136a700 1 -- 172.21.2.117:0/2171873754 wait complete. 2021-02-10T11:47:20.185 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok session ls 2021-02-10T11:47:20.581 INFO:teuthology.orchestra.run.gibba025.stdout:[] 2021-02-10T11:47:20.586 INFO:tasks.cephfs.filesystem:_json_asok output: [] 2021-02-10T11:47:20.586 DEBUG:tasks.ceph_test_case:wait_until_true: success in 70s 2021-02-10T11:47:20.586 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:47:20.586 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:20.932 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:21.276 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:47:21.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:21.277 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:21.277 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:21.611 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:47:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:21.611 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:21.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:21.611 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:21.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:21.947 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:21.947 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:22.282 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:47:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:22.282 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:22.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:22.283 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:22.618 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:22.618 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:22.618 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:22.954 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:47:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:22.954 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:47:22.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:22.954 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:23.289 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:23.290 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:23.290 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:23.624 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:47:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:23.625 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:47:23.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:47:23.625 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:23.959 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 Processor -- start 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 -- start start 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51e00e7180 0x7f51e00e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00e7580 con 0x7f51e00e8390 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e00e77f0 con 0x7f51e01b07f0 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.379 7f51e6be1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e00e5ce0 con 0x7f51e00e7180 2021-02-10T11:47:24.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51e00e7180 0x7f51e00e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51e00e7180 0x7f51e00e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52290/0 (socket says 172.21.2.117:52290) 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51e51fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f51e00e8390 0x7f51e00e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52542/0 (socket says 172.21.2.117:52542) 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 -- 172.21.2.117:0/1269807053 learned_addr learned my addr 172.21.2.117:0/1269807053 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 -- 172.21.2.117:0/1269807053 >> v1:172.21.2.125:6789/0 conn(0x7f51e01b07f0 legacy=0x7f51e00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 -- 172.21.2.117:0/1269807053 >> v1:172.21.2.117:6789/0 conn(0x7f51e00e8390 legacy=0x7f51e00e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 -- 172.21.2.117:0/1269807053 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00e8f80 con 0x7f51e00e7180 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51dffff700 1 --2- 172.21.2.117:0/1269807053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e00e7180 0x7f51e00e7d70 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f51d0006700 tx=0x7f51d0009b80).ready entity=mon.0 client_cookie=b9d47f2df484191a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51df7fe700 1 -- 172.21.2.117:0/1269807053 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51d0006340 con 0x7f51e00e7180 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51df7fe700 1 -- 172.21.2.117:0/1269807053 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d000c0c0 con 0x7f51e00e7180 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51df7fe700 1 -- 172.21.2.117:0/1269807053 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51d000c5a0 con 0x7f51e00e7180 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51e6be1700 1 -- 172.21.2.117:0/1269807053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e00e7180 msgr2=0x7f51e00e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.380 7f51e6be1700 1 --2- 172.21.2.117:0/1269807053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e00e7180 0x7f51e00e7d70 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 rx=0x7f51d0006700 tx=0x7f51d0009b80).stop 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.381 7f51e6be1700 1 -- 172.21.2.117:0/1269807053 shutdown_connections 2021-02-10T11:47:24.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.381 7f51e6be1700 1 --2- 172.21.2.117:0/1269807053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e00e7180 0x7f51e00e7d70 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.381 7f51e6be1700 1 -- 172.21.2.117:0/1269807053 >> 172.21.2.117:0/1269807053 conn(0x7f51e00ce500 msgr2=0x7f51e00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.381 7f51e6be1700 1 -- 172.21.2.117:0/1269807053 shutdown_connections 2021-02-10T11:47:24.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.381 7f51e6be1700 1 -- 172.21.2.117:0/1269807053 wait complete. 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 Processor -- start 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 -- start start 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51e008c600 0x7f51e008ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51e008d020 0x7f51e008d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e008c100 con 0x7f51e01c5370 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e0085d40 con 0x7f51e008c600 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e6be1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00c26b0 con 0x7f51e008d020 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.396 7f51e49fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52298/0 (socket says 172.21.2.117:52298) 2021-02-10T11:47:24.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 -- 172.21.2.117:0/3927996973 learned_addr learned my addr 172.21.2.117:0/3927996973 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e51fe700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51e008d020 0x7f51e008d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51dffff700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51e008c600 0x7f51e008ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 -- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51e008d020 msgr2=0x7f51e008d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51e008d020 0x7f51e008d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 -- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51e008c600 msgr2=0x7f51e008ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51e008c600 0x7f51e008ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 -- 172.21.2.117:0/3927996973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00c28d0 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e49fd700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 rx=0x7f51c800b5b0 tx=0x7f51c800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51c800bd40 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8006240 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51c8005690 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e00c28d0 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.397 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e00c28d0 con 0x7f51e01c5370 2021-02-10T11:47:24.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.398 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51c80054a0 con 0x7f51e01c5370 2021-02-10T11:47:24.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.398 7f51dd7fa700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 0x7f51c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:24.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.398 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f51c80326e0 con 0x7f51e01c5370 2021-02-10T11:47:24.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.398 7f51dffff700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 0x7f51c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:24.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.399 7f51dffff700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 0x7f51c401abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:24.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.429 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e0085d40 con 0x7f51e01c5370 2021-02-10T11:47:24.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.431 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51c8000c60 con 0x7f51e01c5370 2021-02-10T11:47:24.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.579 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f51e0085d40 con 0x7f51e01c5370 2021-02-10T11:47:24.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.583 7f51dd7fa700 1 -- 172.21.2.117:0/3927996973 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f51c80054a0 con 0x7f51e01c5370 2021-02-10T11:47:24.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.587 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 msgr2=0x7f51c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.587 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 0x7f51c401abc0 crc :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.587 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 msgr2=0x7f51e01c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:24.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.587 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 rx=0x7f51c800b5b0 tx=0x7f51c800b950).stop 2021-02-10T11:47:24.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 shutdown_connections 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51e008d020 0x7f51e008d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51e008c600 0x7f51e008ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51c40186c0 0x7f51c401abc0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 --2- 172.21.2.117:0/3927996973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51e01c5370 0x7f51e01c5770 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 >> 172.21.2.117:0/3927996973 conn(0x7f51e007cb20 msgr2=0x7f51e007dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 shutdown_connections 2021-02-10T11:47:24.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:24.588 7f51e6be1700 1 -- 172.21.2.117:0/3927996973 wait complete. 2021-02-10T11:47:24.592 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration' 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 Processor -- start 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 -- start start 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c0ef3f0 0x7f598c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f598c0eb7f0 0x7f598c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0ede60 con 0x7f598c0d18e0 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0ef7f0 con 0x7f598c0eb7f0 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5993441700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0d1ce0 con 0x7f598c0ef3f0 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.007 7f5991a5e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f598c0eb7f0 0x7f598c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:25.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f598c0eb7f0 0x7f598c1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52306/0 (socket says 172.21.2.117:52306) 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 -- 172.21.2.117:0/3913380020 learned_addr learned my addr 172.21.2.117:0/3913380020 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 -- 172.21.2.117:0/3913380020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c0ef3f0 msgr2=0x7f598c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 --2- 172.21.2.117:0/3913380020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c0ef3f0 0x7f598c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 -- 172.21.2.117:0/3913380020 >> v1:172.21.2.125:6789/0 conn(0x7f598c0d18e0 legacy=0x7f598c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 -- 172.21.2.117:0/3913380020 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c1b8f90 con 0x7f598c0eb7f0 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5991a5e700 1 --2- 172.21.2.117:0/3913380020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0eb7f0 0x7f598c1b7960 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f5984006f00 tx=0x7f5984009f10).ready entity=mon.0 client_cookie=62da6423546088c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f597ffff700 1 -- 172.21.2.117:0/3913380020 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5984003030 con 0x7f598c0eb7f0 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f597ffff700 1 -- 172.21.2.117:0/3913380020 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598400b3b0 con 0x7f598c0eb7f0 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f597ffff700 1 -- 172.21.2.117:0/3913380020 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f598400b8d0 con 0x7f598c0eb7f0 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5993441700 1 -- 172.21.2.117:0/3913380020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0eb7f0 msgr2=0x7f598c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.008 7f5993441700 1 --2- 172.21.2.117:0/3913380020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0eb7f0 0x7f598c1b7960 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f5984006f00 tx=0x7f5984009f10).stop 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 -- 172.21.2.117:0/3913380020 shutdown_connections 2021-02-10T11:47:25.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 --2- 172.21.2.117:0/3913380020 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0eb7f0 0x7f598c1b7960 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 --2- 172.21.2.117:0/3913380020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c0ef3f0 0x7f598c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:25.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 -- 172.21.2.117:0/3913380020 >> 172.21.2.117:0/3913380020 conn(0x7f598c0ced60 msgr2=0x7f598c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:25.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 -- 172.21.2.117:0/3913380020 shutdown_connections 2021-02-10T11:47:25.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.009 7f5993441700 1 -- 172.21.2.117:0/3913380020 wait complete. 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 Processor -- start 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 -- start start 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 0x7f598c0e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 0x7f598c0e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c196810 con 0x7f598c0e9560 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c098ad0 con 0x7f598c0e9960 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5993441700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0e8a40 con 0x7f598c196390 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.024 7f5990a5c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:25.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52308/0 (socket says 172.21.2.117:52308) 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 -- 172.21.2.117:0/2621906288 learned_addr learned my addr 172.21.2.117:0/2621906288 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 -- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 msgr2=0x7f598c0e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f599125d700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 0x7f598c0e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 0x7f598c0e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 -- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 msgr2=0x7f598c0e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5991a5e700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 0x7f598c0e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 0x7f598c0e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 -- 172.21.2.117:0/2621906288 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0e8c60 con 0x7f598c0e9560 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f599125d700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 0x7f598c0e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5991a5e700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 0x7f598c0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.025 7f5990a5c700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f5988006f00 tx=0x7f598800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f598800bd30 con 0x7f598c0e9560 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5988000c90 con 0x7f598c0e9560 2021-02-10T11:47:25.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5988002210 con 0x7f598c0e9560 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f5993441700 1 -- 172.21.2.117:0/2621906288 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0e8c60 con 0x7f598c0e9560 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f5993441700 1 -- 172.21.2.117:0/2621906288 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c19cd70 con 0x7f598c0e9560 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.026 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5988005310 con 0x7f598c0e9560 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.027 7f597dffb700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 0x7f596c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.027 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f59880326b0 con 0x7f598c0e9560 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.027 7f599125d700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 0x7f596c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:25.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.027 7f599125d700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 0x7f596c01abc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:25.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.057 7f5993441700 1 -- 172.21.2.117:0/2621906288 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c098ad0 con 0x7f598c0e9560 2021-02-10T11:47:25.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.060 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f59880009b0 con 0x7f598c0e9560 2021-02-10T11:47:25.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.189 7f5993441700 1 -- 172.21.2.117:0/2621906288 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]} v 0) v1 -- 0x7f598c098ad0 con 0x7f598c0e9560 2021-02-10T11:47:25.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.638 7f597dffb700 1 -- 172.21.2.117:0/2621906288 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_connection_expiration"]}]=0 v479) v1 ==== 143+0+0 (secure 0 0 0) 0x7f5988005310 con 0x7f598c0e9560 2021-02-10T11:47:25.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 msgr2=0x7f596c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:25.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 0x7f596c01abc0 crc :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 msgr2=0x7f598c0e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 rx=0x7f5988006f00 tx=0x7f598800b940).stop 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 shutdown_connections 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f596c0186c0 0x7f596c01abc0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f598c0e9560 0x7f598c0e7e00 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f598c0e9960 0x7f598c0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 --2- 172.21.2.117:0/2621906288 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f598c196390 0x7f598c0e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:25.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 >> 172.21.2.117:0/2621906288 conn(0x7f598c19f610 msgr2=0x7f598c1a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:25.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 shutdown_connections 2021-02-10T11:47:25.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:25.657 7f5993441700 1 -- 172.21.2.117:0/2621906288 wait complete. 2021-02-10T11:47:25.661 INFO:tasks.cephfs_test_runner:test_connection_expiration (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:47:25.661 INFO:tasks.cephfs_test_runner:Starting test: test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:47:25.661 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 Processor -- start 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 -- start start 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80e80ef3f0 0x7f80e80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80eb7f0 0x7f80e81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e80ede60 con 0x7f80e80d18e0 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e80ef7f0 con 0x7f80e80ef3f0 2021-02-10T11:47:26.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.076 7f80f008d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80d1ce0 con 0x7f80e80eb7f0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80e80ef3f0 0x7f80e80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80e80ef3f0 0x7f80e80eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52316/0 (socket says 172.21.2.117:52316) 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 -- 172.21.2.117:0/2902282438 learned_addr learned my addr 172.21.2.117:0/2902282438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 -- 172.21.2.117:0/2902282438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80eb7f0 msgr2=0x7f80e81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 --2- 172.21.2.117:0/2902282438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80eb7f0 0x7f80e81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 -- 172.21.2.117:0/2902282438 >> v1:172.21.2.125:6789/0 conn(0x7f80e80d18e0 legacy=0x7f80e80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 -- 172.21.2.117:0/2902282438 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e81b8f90 con 0x7f80e80ef3f0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ed6a8700 1 --2- 172.21.2.117:0/2902282438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80ef3f0 0x7f80e80eb1d0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f80dc006700 tx=0x7f80dc009b80).ready entity=mon.0 client_cookie=2b9a94ca34d3106a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ecea7700 1 -- 172.21.2.117:0/2902282438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc00c0c0 con 0x7f80e80ef3f0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ecea7700 1 -- 172.21.2.117:0/2902282438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc00c3c0 con 0x7f80e80ef3f0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80ecea7700 1 -- 172.21.2.117:0/2902282438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc00c8e0 con 0x7f80e80ef3f0 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80f008d700 1 -- 172.21.2.117:0/2902282438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80ef3f0 msgr2=0x7f80e80eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.077 7f80f008d700 1 --2- 172.21.2.117:0/2902282438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80ef3f0 0x7f80e80eb1d0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 rx=0x7f80dc006700 tx=0x7f80dc009b80).stop 2021-02-10T11:47:26.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 -- 172.21.2.117:0/2902282438 shutdown_connections 2021-02-10T11:47:26.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 --2- 172.21.2.117:0/2902282438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80eb7f0 0x7f80e81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 --2- 172.21.2.117:0/2902282438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80ef3f0 0x7f80e80eb1d0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:26.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 -- 172.21.2.117:0/2902282438 >> 172.21.2.117:0/2902282438 conn(0x7f80e80ced60 msgr2=0x7f80e80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:26.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 -- 172.21.2.117:0/2902282438 shutdown_connections 2021-02-10T11:47:26.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.078 7f80f008d700 1 -- 172.21.2.117:0/2902282438 wait complete. 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 Processor -- start 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 -- start start 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80980c0 0x7f80e819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80e819dd40 0x7f80e819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80b1f30 con 0x7f80e80984c0 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e819d830 con 0x7f80e819dd40 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.093 7f80f008d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e81a55e0 con 0x7f80e80980c0 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52322/0 (socket says 172.21.2.117:52322) 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 -- 172.21.2.117:0/187909192 learned_addr learned my addr 172.21.2.117:0/187909192 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:26.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 -- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80980c0 msgr2=0x7f80e819ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80980c0 0x7f80e819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 -- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80e819dd40 msgr2=0x7f80e819e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80e819dd40 0x7f80e819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 -- 172.21.2.117:0/187909192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e819e760 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80ed6a8700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f80dc00b650 tx=0x7f80dc00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc00bde0 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc006480 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80dc002220 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80f008d700 1 -- 172.21.2.117:0/187909192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e819e760 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.094 7f80f008d700 1 -- 172.21.2.117:0/187909192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e819e760 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.095 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80dc000c90 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.095 7f80da7fc700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 0x7f80d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.095 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f80dc032710 con 0x7f80e80984c0 2021-02-10T11:47:26.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.095 7f80edea9700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 0x7f80d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:26.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.096 7f80edea9700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 0x7f80d001abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:26.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.127 7f80f008d700 1 -- 172.21.2.117:0/187909192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e8087210 con 0x7f80e80984c0 2021-02-10T11:47:26.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.130 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80dc0261f0 con 0x7f80e80984c0 2021-02-10T11:47:26.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.264 7f80f008d700 1 -- 172.21.2.117:0/187909192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7f80e80ede60 con 0x7f80e80984c0 2021-02-10T11:47:26.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.642 7f80da7fc700 1 -- 172.21.2.117:0/187909192 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v480) v1 ==== 169+0+0 (secure 0 0 0) 0x7f80dc0261f0 con 0x7f80e80984c0 2021-02-10T11:47:26.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.683 7f80f008d700 1 -- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 msgr2=0x7f80d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:26.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.683 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 0x7f80d001abc0 crc :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:26.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.683 7f80f008d700 1 -- 172.21.2.117:0/187909192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 msgr2=0x7f80e819d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:26.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.683 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 rx=0x7f80dc00b650 tx=0x7f80dc00b9f0).stop 2021-02-10T11:47:26.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 -- 172.21.2.117:0/187909192 shutdown_connections 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80d00186c0 0x7f80d001abc0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80e80984c0 0x7f80e819d0a0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80e819dd40 0x7f80e819e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 --2- 172.21.2.117:0/187909192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80e80980c0 0x7f80e819ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 -- 172.21.2.117:0/187909192 >> 172.21.2.117:0/187909192 conn(0x7f80e819f610 msgr2=0x7f80e81a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 -- 172.21.2.117:0/187909192 shutdown_connections 2021-02-10T11:47:26.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:26.684 7f80f008d700 1 -- 172.21.2.117:0/187909192 wait complete. 2021-02-10T11:47:26.688 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:47:27.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.107 7fc6af862700 1 Processor -- start 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.107 7fc6af862700 1 -- start start 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.108 7fc6af862700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 0x7fc6a80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.108 7fc6af862700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc6a80e7180 0x7fc6a80e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.108 7fc6af862700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a80e7580 con 0x7fc6a80e7d70 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.108 7fc6af862700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a80e92f0 con 0x7fc6a80e7180 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.108 7fc6af862700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80e5ce0 con 0x7fc6a81b07f0 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.109 7fc6ad67e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 0x7fc6a80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.109 7fc6ace7d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fc6a80e7d70 0x7fc6a80e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45506/0 (socket says 172.21.2.117:45506) 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.109 7fc6ace7d700 1 -- 172.21.2.117:0/1490065825 learned_addr learned my addr 172.21.2.117:0/1490065825 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:27.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.109 7fc6ade7f700 1 --2- 172.21.2.117:0/1490065825 >> v2:172.21.2.125:3300/0 conn(0x7fc6a80e7180 0x7fc6a80e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6ad67e700 1 -- 172.21.2.117:0/1490065825 >> v2:172.21.2.125:3300/0 conn(0x7fc6a80e7180 msgr2=0x7fc6a80e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6ad67e700 1 --2- 172.21.2.117:0/1490065825 >> v2:172.21.2.125:3300/0 conn(0x7fc6a80e7180 0x7fc6a80e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6ad67e700 1 -- 172.21.2.117:0/1490065825 >> v1:172.21.2.125:6789/0 conn(0x7fc6a80e7d70 legacy=0x7fc6a80e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6ad67e700 1 -- 172.21.2.117:0/1490065825 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a80e77f0 con 0x7fc6a81b07f0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6ad67e700 1 --2- 172.21.2.117:0/1490065825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 0x7fc6a80e4e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc6a4006650 tx=0x7fc6a4009a80).ready entity=mon.2 client_cookie=b7eef4dc72626719 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc69ffff700 1 -- 172.21.2.117:0/1490065825 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6a400c0c0 con 0x7fc6a81b07f0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc69ffff700 1 -- 172.21.2.117:0/1490065825 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400c340 con 0x7fc6a81b07f0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc69ffff700 1 -- 172.21.2.117:0/1490065825 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6a400c860 con 0x7fc6a81b07f0 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6af862700 1 -- 172.21.2.117:0/1490065825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 msgr2=0x7fc6a80e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.110 7fc6af862700 1 --2- 172.21.2.117:0/1490065825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 0x7fc6a80e4e90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fc6a4006650 tx=0x7fc6a4009a80).stop 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 -- 172.21.2.117:0/1490065825 shutdown_connections 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 --2- 172.21.2.117:0/1490065825 >> v2:172.21.2.125:3300/0 conn(0x7fc6a80e7180 0x7fc6a80e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 --2- 172.21.2.117:0/1490065825 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a81b07f0 0x7fc6a80e4e90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 -- 172.21.2.117:0/1490065825 >> 172.21.2.117:0/1490065825 conn(0x7fc6a80ce500 msgr2=0x7fc6a80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:27.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 -- 172.21.2.117:0/1490065825 shutdown_connections 2021-02-10T11:47:27.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.111 7fc6af862700 1 -- 172.21.2.117:0/1490065825 wait complete. 2021-02-10T11:47:27.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.125 7fc6af862700 1 Processor -- start 2021-02-10T11:47:27.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.125 7fc6af862700 1 -- start start 2021-02-10T11:47:27.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a80a7e50 0x7fc6a818b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6a818b9b0 0x7fc6a808caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a808c2b0 con 0x7fc6a81a4170 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80a8250 con 0x7fc6a818b9b0 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6af862700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a818bdb0 con 0x7fc6a80a7e50 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52336/0 (socket says 172.21.2.117:52336) 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 -- 172.21.2.117:0/552549837 learned_addr learned my addr 172.21.2.117:0/552549837 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 -- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a80a7e50 msgr2=0x7fc6a818b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a80a7e50 0x7fc6a818b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 -- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6a818b9b0 msgr2=0x7fc6a808caa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6a818b9b0 0x7fc6a808caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 -- 172.21.2.117:0/552549837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a808d0c0 con 0x7fc6a81a4170 2021-02-10T11:47:27.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.126 7fc6ad67e700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7fc6a4009630 tx=0x7fc6a400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6a40093a0 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400bcf0 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc6af862700 1 -- 172.21.2.117:0/552549837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a808d0c0 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6a4005560 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc6af862700 1 -- 172.21.2.117:0/552549837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a808d650 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc6a40052e0 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 0x7fc68801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc6a40326b0 con 0x7fc6a81a4170 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc6ace7d700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 0x7fc68801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:27.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.128 7fc6ace7d700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 0x7fc68801abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:27.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.158 7fc6af862700 1 -- 172.21.2.117:0/552549837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a810c290 con 0x7fc6a81a4170 2021-02-10T11:47:27.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.161 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc6a4009340 con 0x7fc6a81a4170 2021-02-10T11:47:27.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.309 7fc6af862700 1 -- 172.21.2.117:0/552549837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc6a80e77f0 con 0x7fc6a81a4170 2021-02-10T11:47:27.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.315 7fc69dffb700 1 -- 172.21.2.117:0/552549837 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc6a40052e0 con 0x7fc6a81a4170 2021-02-10T11:47:27.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.319 7fc6af862700 1 -- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 msgr2=0x7fc68801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:27.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.319 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 0x7fc68801abc0 crc :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.319 7fc6af862700 1 -- 172.21.2.117:0/552549837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 msgr2=0x7fc6a80a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.319 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 rx=0x7fc6a4009630 tx=0x7fc6a400b900).stop 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 -- 172.21.2.117:0/552549837 shutdown_connections 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6a818b9b0 0x7fc6a808caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6a80a7e50 0x7fc6a818b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6880186c0 0x7fc68801abc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 --2- 172.21.2.117:0/552549837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6a81a4170 0x7fc6a80a7830 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 -- 172.21.2.117:0/552549837 >> 172.21.2.117:0/552549837 conn(0x7fc6a818c1e0 msgr2=0x7fc6a8090e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 -- 172.21.2.117:0/552549837 shutdown_connections 2021-02-10T11:47:27.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:27.320 7fc6af862700 1 -- 172.21.2.117:0/552549837 wait complete. 2021-02-10T11:47:27.323 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:47:27.323 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:27.668 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:47:28.013 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:47:28.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:28.013 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:28.013 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:28.349 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:47:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:28.349 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:28.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:28.350 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:28.350 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:47:28.685 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:47:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:28.685 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:47:28.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:28.686 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:28.686 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:47:29.021 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:47:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:29.021 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:47:29.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:29.021 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:29.022 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:47:29.355 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:47:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:47:29.356 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:47:29.356 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:47:29.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.772 7f2859bfd700 1 Processor -- start 2021-02-10T11:47:29.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.772 7f2859bfd700 1 -- start start 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f2859bfd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28540d18e0 0x7f28540ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f2859bfd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28540ef3f0 0x7f28541b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f2859bfd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28540ede60 con 0x7f28540eb000 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f2859bfd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28540ef7f0 con 0x7f28540d18e0 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f2859bfd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540d1ce0 con 0x7f28540ef3f0 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28540d18e0 0x7f28540ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28540d18e0 0x7f28540ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52344/0 (socket says 172.21.2.117:52344) 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 -- 172.21.2.117:0/854831239 learned_addr learned my addr 172.21.2.117:0/854831239 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 -- 172.21.2.117:0/854831239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28540ef3f0 msgr2=0x7f28541b7960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 --2- 172.21.2.117:0/854831239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28540ef3f0 0x7f28541b7960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 -- 172.21.2.117:0/854831239 >> v1:172.21.2.125:6789/0 conn(0x7f28540eb000 legacy=0x7f28540eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 -- 172.21.2.117:0/854831239 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28541b8f90 con 0x7f28540d18e0 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.773 7f28537fe700 1 --2- 172.21.2.117:0/854831239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28540d18e0 0x7f28540ea9e0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f283c006700 tx=0x7f283c009b00).ready entity=mon.0 client_cookie=3e8ca9d78f9363e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f28527fc700 1 -- 172.21.2.117:0/854831239 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283c00c0c0 con 0x7f28540d18e0 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f28527fc700 1 -- 172.21.2.117:0/854831239 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c00c400 con 0x7f28540d18e0 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f28527fc700 1 -- 172.21.2.117:0/854831239 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f283c00c920 con 0x7f28540d18e0 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 -- 172.21.2.117:0/854831239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28540d18e0 msgr2=0x7f28540ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 --2- 172.21.2.117:0/854831239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28540d18e0 0x7f28540ea9e0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 rx=0x7f283c006700 tx=0x7f283c009b00).stop 2021-02-10T11:47:29.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 -- 172.21.2.117:0/854831239 shutdown_connections 2021-02-10T11:47:29.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 --2- 172.21.2.117:0/854831239 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28540ef3f0 0x7f28541b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 --2- 172.21.2.117:0/854831239 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28540d18e0 0x7f28540ea9e0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:29.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.774 7f2859bfd700 1 -- 172.21.2.117:0/854831239 >> 172.21.2.117:0/854831239 conn(0x7f28540ced60 msgr2=0x7f28540cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:29.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.775 7f2859bfd700 1 -- 172.21.2.117:0/854831239 shutdown_connections 2021-02-10T11:47:29.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.775 7f2859bfd700 1 -- 172.21.2.117:0/854831239 wait complete. 2021-02-10T11:47:29.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.789 7f2859bfd700 1 Processor -- start 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.789 7f2859bfd700 1 -- start start 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28541af6a0 0x7f28540e81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f285409dd60 0x7f285409e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285409d7e0 con 0x7f285409e780 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28540afab0 con 0x7f285409dd60 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2859bfd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28540afcd0 con 0x7f28541af6a0 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52352/0 (socket says 172.21.2.117:52352) 2021-02-10T11:47:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 -- 172.21.2.117:0/501099248 learned_addr learned my addr 172.21.2.117:0/501099248 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 -- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28541af6a0 msgr2=0x7f28540e81d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28541af6a0 0x7f28540e81d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 -- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f285409dd60 msgr2=0x7f285409e160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f285409dd60 0x7f285409e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 -- 172.21.2.117:0/501099248 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540afef0 con 0x7f285409e780 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.790 7f2853fff700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7f284c009960 tx=0x7f284c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c003030 con 0x7f285409e780 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c001310 con 0x7f285409e780 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f284c001830 con 0x7f285409e780 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f2859bfd700 1 -- 172.21.2.117:0/501099248 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28540afef0 con 0x7f285409e780 2021-02-10T11:47:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f2859bfd700 1 -- 172.21.2.117:0/501099248 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28540e8a60 con 0x7f285409e780 2021-02-10T11:47:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f284c0014f0 con 0x7f285409e780 2021-02-10T11:47:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.791 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f284c032aa0 con 0x7f285409e780 2021-02-10T11:47:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.792 7f28537fe700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.793 7f28537fe700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 0x7f283801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:29.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.822 7f2859bfd700 1 -- 172.21.2.117:0/501099248 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2854087210 con 0x7f285409e780 2021-02-10T11:47:29.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.825 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f284c001ee0 con 0x7f285409e780 2021-02-10T11:47:29.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.950 7f2859bfd700 1 -- 172.21.2.117:0/501099248 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f28540ede60 con 0x7f285409e780 2021-02-10T11:47:29.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.950 7f284bfff700 1 -- 172.21.2.117:0/501099248 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v83) v1 ==== 81+0+30755 (secure 0 0 0) 0x7f284c0013d0 con 0x7f285409e780 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 83, 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:45:26.614628", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:47:29.946 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:47:29.947 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 17, 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:47:29.948 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:47:29.949 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:29.950 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:47:29.951 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:47:29.952 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:47:29.953 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.954 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 16, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:45:23.589151", 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:47:29.955 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "82", 2021-02-10T11:47:29.956 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:47:29.957 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:47:29.958 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:47:29.959 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 17, 2021-02-10T11:47:29.960 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:45:24.659131", 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:47:29.961 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "83", 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:47:29.962 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:47:29.963 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:47:29.964 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:47:29.965 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:47:29.966 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 81, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.967 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.968 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.969 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.970 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:47:29.971 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.972 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 71, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.973 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.974 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.975 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.976 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.977 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:47:29.978 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.979 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 80, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.980 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.981 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.982 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.983 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.984 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.985 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 80, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.986 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.987 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.988 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.989 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:47:29.990 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.991 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 81, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.992 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.993 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.994 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:29.996 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:29.997 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 80, 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:47:29.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:47:29.999 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:47:30.000 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:47:30.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 81, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:30.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:30.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.007 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:47:30.008 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 81, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:47:30.011 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:47:30.012 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:47:30.013 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.015 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.016 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.017 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:47:30.018 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:47:30.019 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:47:30.020 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:47:30.021 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:47:30.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 -- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 msgr2=0x7f283801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 0x7f283801abc0 crc :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 -- 172.21.2.117:0/501099248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 msgr2=0x7f28540af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 rx=0x7f284c009960 tx=0x7f284c00bcf0).stop 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 -- 172.21.2.117:0/501099248 shutdown_connections 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28380186c0 0x7f283801abc0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f285409e780 0x7f28540af490 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f285409dd60 0x7f285409e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 --2- 172.21.2.117:0/501099248 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28541af6a0 0x7f28540e81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.952 7f2859bfd700 1 -- 172.21.2.117:0/501099248 >> 172.21.2.117:0/501099248 conn(0x7f285419f610 msgr2=0x7f28541a6370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.953 7f2859bfd700 1 -- 172.21.2.117:0/501099248 shutdown_connections 2021-02-10T11:47:30.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:29.953 7f2859bfd700 1 -- 172.21.2.117:0/501099248 wait complete. 2021-02-10T11:47:30.025 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.441 7f9ff9add700 1 Processor -- start 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.441 7f9ff9add700 1 -- start start 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff9add700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9ff40ef3f0 0x7f9ff40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff9add700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40ede60 con 0x7f9ff40eb7f0 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff9add700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40d1ce0 con 0x7f9ff40d18e0 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff9add700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40ef7f0 con 0x7f9ff40ef3f0 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9ff40eb7f0 0x7f9ff41b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52610/0 (socket says 172.21.2.117:52610) 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff3fff700 1 -- 172.21.2.117:0/3384711136 learned_addr learned my addr 172.21.2.117:0/3384711136 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff2ffd700 1 --2- 172.21.2.117:0/3384711136 >> v2:172.21.2.117:3300/0 conn(0x7f9ff40ef3f0 0x7f9ff40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff2ffd700 1 -- 172.21.2.117:0/3384711136 >> v1:172.21.2.125:6789/0 conn(0x7f9ff40d18e0 legacy=0x7f9ff40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff2ffd700 1 -- 172.21.2.117:0/3384711136 >> v1:172.21.2.117:6789/0 conn(0x7f9ff40eb7f0 legacy=0x7f9ff41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.442 7f9ff2ffd700 1 -- 172.21.2.117:0/3384711136 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff41bb180 con 0x7f9ff40ef3f0 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff2ffd700 1 --2- 172.21.2.117:0/3384711136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40ef3f0 0x7f9ff40eb1d0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f9fdc006650 tx=0x7f9fdc009a80).ready entity=mon.0 client_cookie=1dfc0be824f3cb1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff27fc700 1 -- 172.21.2.117:0/3384711136 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fdc00c0c0 con 0x7f9ff40ef3f0 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff27fc700 1 -- 172.21.2.117:0/3384711136 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc00c440 con 0x7f9ff40ef3f0 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff27fc700 1 -- 172.21.2.117:0/3384711136 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fdc00c920 con 0x7f9ff40ef3f0 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff9add700 1 -- 172.21.2.117:0/3384711136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40ef3f0 msgr2=0x7f9ff40eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.443 7f9ff9add700 1 --2- 172.21.2.117:0/3384711136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40ef3f0 0x7f9ff40eb1d0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 rx=0x7f9fdc006650 tx=0x7f9fdc009a80).stop 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.444 7f9ff9add700 1 -- 172.21.2.117:0/3384711136 shutdown_connections 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.444 7f9ff9add700 1 --2- 172.21.2.117:0/3384711136 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40ef3f0 0x7f9ff40eb1d0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.444 7f9ff9add700 1 -- 172.21.2.117:0/3384711136 >> 172.21.2.117:0/3384711136 conn(0x7f9ff40ced60 msgr2=0x7f9ff40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.444 7f9ff9add700 1 -- 172.21.2.117:0/3384711136 shutdown_connections 2021-02-10T11:47:30.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.444 7f9ff9add700 1 -- 172.21.2.117:0/3384711136 wait complete. 2021-02-10T11:47:30.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.458 7f9ff9add700 1 Processor -- start 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 -- start start 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff4099780 0x7f9ff41c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff41c68d0 0x7f9ff40edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4099200 con 0x7f9ff40a3200 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40f2530 con 0x7f9ff4099780 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff9add700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40f2710 con 0x7f9ff41c68d0 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52362/0 (socket says 172.21.2.117:52362) 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.459 7f9ff37fe700 1 -- 172.21.2.117:0/2405862644 learned_addr learned my addr 172.21.2.117:0/2405862644 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 -- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff41c68d0 msgr2=0x7f9ff40edd00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff41c68d0 0x7f9ff40edd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 -- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff4099780 msgr2=0x7f9ff41c62b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:30.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff4099780 0x7f9ff41c62b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 -- 172.21.2.117:0/2405862644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4099b80 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff37fe700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f9fe400b5b0 tx=0x7f9fe400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe400bd40 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe40009b0 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff4099b80 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff40ee610 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.460 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe4006210 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.461 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fe40054a0 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.461 7f9febfff700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.461 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9fe4032690 con 0x7f9ff40a3200 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.461 7f9ff2ffd700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:30.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.462 7f9ff2ffd700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:30.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.492 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff4086140 con 0x7f9ff40a3200 2021-02-10T11:47:30.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.495 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fe4025f70 con 0x7f9ff40a3200 2021-02-10T11:47:30.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.624 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ff4086140 con 0x7f9ff40a3200 2021-02-10T11:47:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.624 7f9febfff700 1 -- 172.21.2.117:0/2405862644 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f9fe4025f70 con 0x7f9ff40a3200 2021-02-10T11:47:30.619 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 62 2021-02-10T11:47:30.619 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:30.620 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":62,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:45:27.622343","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7144},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7144":{"gid":7144,"name":"b","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1925085474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1925085474},{"type":"v1","addr":"172.21.2.125:6835","nonce":1925085474}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:30.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 msgr2=0x7f9fd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 crc :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 msgr2=0x7f9ff40f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 rx=0x7f9fe400b5b0 tx=0x7f9fe400b950).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 shutdown_connections 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff41c68d0 0x7f9ff40edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff4099780 0x7f9ff41c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 --2- 172.21.2.117:0/2405862644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff40a3200 0x7f9ff40f29d0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:30.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.626 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 >> 172.21.2.117:0/2405862644 conn(0x7f9ff40f3230 msgr2=0x7f9ff40b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.627 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 shutdown_connections 2021-02-10T11:47:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:30.627 7f9ff9add700 1 -- 172.21.2.117:0/2405862644 wait complete. 2021-02-10T11:47:30.630 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.044 7f344a580700 1 Processor -- start 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.044 7f344a580700 1 -- start start 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f344a580700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34441b07f0 0x7f34440e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f344a580700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34440e7d70 0x7f34440e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f344a580700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440e77f0 con 0x7f34440e8790 2021-02-10T11:47:31.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f344a580700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34440e92f0 con 0x7f34440e7d70 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f344a580700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34440e5ce0 con 0x7f34441b07f0 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f3448b9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f34440e8790 0x7f34440e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52622/0 (socket says 172.21.2.117:52622) 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f3448b9d700 1 -- 172.21.2.117:0/456993958 learned_addr learned my addr 172.21.2.117:0/456993958 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f34437fe700 1 --2- 172.21.2.117:0/456993958 >> v2:172.21.2.117:3300/0 conn(0x7f34440e7d70 0x7f34440e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f3442ffd700 1 -- 172.21.2.117:0/456993958 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149811378 0 0) 0x7f343c001500 con 0x7f34440e8790 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f3442ffd700 1 -- 172.21.2.117:0/456993958 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f342c002fe0 con 0x7f34440e8790 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f34437fe700 1 -- 172.21.2.117:0/456993958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34441b07f0 msgr2=0x7f34440e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.045 7f34437fe700 1 --2- 172.21.2.117:0/456993958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34441b07f0 0x7f34440e4e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f34437fe700 1 -- 172.21.2.117:0/456993958 >> v1:172.21.2.117:6789/0 conn(0x7f34440e8790 legacy=0x7f34440e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f34437fe700 1 -- 172.21.2.117:0/456993958 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34440e54b0 con 0x7f34440e7d70 2021-02-10T11:47:31.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f34437fe700 1 --2- 172.21.2.117:0/456993958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440e7d70 0x7f34440e8170 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f3438006650 tx=0x7f3438009a80).ready entity=mon.0 client_cookie=62cc1d6f0211fcac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f3442ffd700 1 -- 172.21.2.117:0/456993958 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f343800c0c0 con 0x7f34440e7d70 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f3442ffd700 1 -- 172.21.2.117:0/456993958 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343800c340 con 0x7f34440e7d70 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f3442ffd700 1 -- 172.21.2.117:0/456993958 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f343800c860 con 0x7f34440e7d70 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f344a580700 1 -- 172.21.2.117:0/456993958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440e7d70 msgr2=0x7f34440e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.046 7f344a580700 1 --2- 172.21.2.117:0/456993958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440e7d70 0x7f34440e8170 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 rx=0x7f3438006650 tx=0x7f3438009a80).stop 2021-02-10T11:47:31.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 -- 172.21.2.117:0/456993958 shutdown_connections 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 --2- 172.21.2.117:0/456993958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440e7d70 0x7f34440e8170 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 --2- 172.21.2.117:0/456993958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34441b07f0 0x7f34440e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 -- 172.21.2.117:0/456993958 >> 172.21.2.117:0/456993958 conn(0x7f34440ce500 msgr2=0x7f34440e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 -- 172.21.2.117:0/456993958 shutdown_connections 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.047 7f344a580700 1 -- 172.21.2.117:0/456993958 wait complete. 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.061 7f344a580700 1 Processor -- start 2021-02-10T11:47:31.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.061 7f344a580700 1 -- start start 2021-02-10T11:47:31.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34440b0a50 0x7f34441a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34440b15a0 0x7f3444093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440b19a0 con 0x7f34440a3a60 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440930b0 con 0x7f34440b15a0 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f344a580700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34441a3450 con 0x7f34440b0a50 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52376/0 (socket says 172.21.2.117:52376) 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 -- 172.21.2.117:0/3358911267 learned_addr learned my addr 172.21.2.117:0/3358911267 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 -- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34440b0a50 msgr2=0x7f34441a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34440b0a50 0x7f34441a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 -- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34440b15a0 msgr2=0x7f3444093630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34440b15a0 0x7f3444093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 -- 172.21.2.117:0/3358911267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34441a3630 con 0x7f34440a3a60 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.062 7f3443fff700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f343400b5c0 tx=0x7f343400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3434000c40 con 0x7f34440a3a60 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400bd50 con 0x7f34440a3a60 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f344a580700 1 -- 172.21.2.117:0/3358911267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34441a3630 con 0x7f34440a3a60 2021-02-10T11:47:31.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f344a580700 1 -- 172.21.2.117:0/3358911267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3444093fb0 con 0x7f34440a3a60 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34340054d0 con 0x7f34440a3a60 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3434006030 con 0x7f34440a3a60 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 0x7f342401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.063 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f34340326e0 con 0x7f34440a3a60 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.064 7f34437fe700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 0x7f342401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:31.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.064 7f34437fe700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 0x7f342401abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:31.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.094 7f344a580700 1 -- 172.21.2.117:0/3358911267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34440b19a0 con 0x7f34440a3a60 2021-02-10T11:47:31.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.097 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3434000940 con 0x7f34440a3a60 2021-02-10T11:47:31.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:31.225 7f344a580700 1 -- 172.21.2.117:0/3358911267 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f34440b19a0 con 0x7f34440a3a60 2021-02-10T11:47:32.215 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:47:32.221 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:47:32.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.226 7f3440ff9700 1 -- 172.21.2.117:0/3358911267 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3434006030 con 0x7f34440a3a60 2021-02-10T11:47:32.240 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:47:32.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 -- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 msgr2=0x7f342401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 0x7f342401abc0 crc :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 -- 172.21.2.117:0/3358911267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 msgr2=0x7f34441a2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 rx=0x7f343400b5c0 tx=0x7f343400b960).stop 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 -- 172.21.2.117:0/3358911267 shutdown_connections 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34440b15a0 0x7f3444093630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34440b0a50 0x7f34441a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34240186c0 0x7f342401abc0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 --2- 172.21.2.117:0/3358911267 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34440a3a60 0x7f34441a2810 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.247 7f344a580700 1 -- 172.21.2.117:0/3358911267 >> 172.21.2.117:0/3358911267 conn(0x7f34440a7260 msgr2=0x7f34440bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.248 7f344a580700 1 -- 172.21.2.117:0/3358911267 shutdown_connections 2021-02-10T11:47:32.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.248 7f344a580700 1 -- 172.21.2.117:0/3358911267 wait complete. 2021-02-10T11:47:32.251 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:32.270 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 Processor -- start 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 -- start start 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5d3c0ef3f0 0x7f5d3c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0ede60 con 0x7f5d3c0d18e0 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c0ef7f0 con 0x7f5d3c0eb7f0 2021-02-10T11:47:32.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.667 7f5d429b8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c0d1ce0 con 0x7f5d3c0ef3f0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52388/0 (socket says 172.21.2.117:52388) 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 -- 172.21.2.117:0/3948865338 learned_addr learned my addr 172.21.2.117:0/3948865338 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d3b7fe700 1 --2- 172.21.2.117:0/3948865338 >> v2:172.21.2.125:3300/0 conn(0x7f5d3c0ef3f0 0x7f5d3c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 -- 172.21.2.117:0/3948865338 >> v2:172.21.2.125:3300/0 conn(0x7f5d3c0ef3f0 msgr2=0x7f5d3c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 --2- 172.21.2.117:0/3948865338 >> v2:172.21.2.125:3300/0 conn(0x7f5d3c0ef3f0 0x7f5d3c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 -- 172.21.2.117:0/3948865338 >> v1:172.21.2.125:6789/0 conn(0x7f5d3c0d18e0 legacy=0x7f5d3c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 -- 172.21.2.117:0/3948865338 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c1b8f90 con 0x7f5d3c0eb7f0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.668 7f5d40fd5700 1 --2- 172.21.2.117:0/3948865338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f5d34006f00 tx=0x7f5d34009f10).ready entity=mon.0 client_cookie=d8716f079c9e5d2c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d3affd700 1 -- 172.21.2.117:0/3948865338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3400b3b0 con 0x7f5d3c0eb7f0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d3affd700 1 -- 172.21.2.117:0/3948865338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3400b630 con 0x7f5d3c0eb7f0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d3affd700 1 -- 172.21.2.117:0/3948865338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d3400ba90 con 0x7f5d3c0eb7f0 2021-02-10T11:47:32.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 -- 172.21.2.117:0/3948865338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c0eb7f0 msgr2=0x7f5d3c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 --2- 172.21.2.117:0/3948865338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 rx=0x7f5d34006f00 tx=0x7f5d34009f10).stop 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 -- 172.21.2.117:0/3948865338 shutdown_connections 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 --2- 172.21.2.117:0/3948865338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c0eb7f0 0x7f5d3c1b7960 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 --2- 172.21.2.117:0/3948865338 >> v2:172.21.2.125:3300/0 conn(0x7f5d3c0ef3f0 0x7f5d3c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 -- 172.21.2.117:0/3948865338 >> 172.21.2.117:0/3948865338 conn(0x7f5d3c0ced60 msgr2=0x7f5d3c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 -- 172.21.2.117:0/3948865338 shutdown_connections 2021-02-10T11:47:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.669 7f5d429b8700 1 -- 172.21.2.117:0/3948865338 wait complete. 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 Processor -- start 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 -- start start 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 0x7f5d3c1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 0x7f5d3c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c197cd0 con 0x7f5d3c091d40 2021-02-10T11:47:32.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c092140 con 0x7f5d3c198250 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.684 7f5d429b8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c1b2ed0 con 0x7f5d3c1b24b0 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52390/0 (socket says 172.21.2.117:52390) 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 -- 172.21.2.117:0/3774144148 learned_addr learned my addr 172.21.2.117:0/3774144148 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d40fd5700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 0x7f5d3c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3b7fe700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 0x7f5d3c1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 -- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 msgr2=0x7f5d3c1b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 0x7f5d3c1b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 -- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 msgr2=0x7f5d3c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 0x7f5d3c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 -- 172.21.2.117:0/3774144148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c198c70 con 0x7f5d3c091d40 2021-02-10T11:47:32.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3b7fe700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 0x7f5d3c1b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d40fd5700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 0x7f5d3c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d3bfff700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f5d2c00b5b0 tx=0x7f5d2c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.685 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d2c00bd40 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c006020 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c198c70 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d2c0054a0 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c196c90 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d2c006200 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d38ff9700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 0x7f5d1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.686 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f5d2c032770 con 0x7f5d3c091d40 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.687 7f5d3b7fe700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 0x7f5d1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:32.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.687 7f5d3b7fe700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 0x7f5d1801abc0 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:32.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.717 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c087290 con 0x7f5d3c091d40 2021-02-10T11:47:32.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.720 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d2c000940 con 0x7f5d3c091d40 2021-02-10T11:47:32.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.848 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d3c087290 con 0x7f5d3c091d40 2021-02-10T11:47:32.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.848 7f5d38ff9700 1 -- 172.21.2.117:0/3774144148 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f5d2c006200 con 0x7f5d3c091d40 2021-02-10T11:47:32.843 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 63 2021-02-10T11:47:32.843 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:32.843 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9312,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1764682874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1764682874},{"type":"v1","addr":"172.21.2.117:6833","nonce":1764682874}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":63,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:45:25.613265","modified":"2021-02-10 11:47:32.221693","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2021-02-10T11:47:32.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 msgr2=0x7f5d1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 0x7f5d1801abc0 crc :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 msgr2=0x7f5d3c0e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 rx=0x7f5d2c00b5b0 tx=0x7f5d2c00b950).stop 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 shutdown_connections 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d3c198250 0x7f5d3c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d3c1b24b0 0x7f5d3c1b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d180186c0 0x7f5d1801abc0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 --2- 172.21.2.117:0/3774144148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d3c091d40 0x7f5d3c0e71e0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 >> 172.21.2.117:0/3774144148 conn(0x7f5d3c1a3620 msgr2=0x7f5d3c0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:32.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 shutdown_connections 2021-02-10T11:47:32.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:32.850 7f5d429b8700 1 -- 172.21.2.117:0/3774144148 wait complete. 2021-02-10T11:47:32.853 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:47:33.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 Processor -- start 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 -- start start 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9af01b07f0 0x7f9af00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00e6170 0x7f9af00e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00e77f0 con 0x7f9af00e6b90 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af00e5ce0 con 0x7f9af01b07f0 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.274 7f9af6566700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00e54b0 con 0x7f9af00e6170 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aef7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00e6170 0x7f9af00e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9af01b07f0 0x7f9af00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9af4b83700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f9af00e6b90 0x7f9af00e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45574/0 (socket says 172.21.2.117:45574) 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9af4b83700 1 -- 172.21.2.117:0/3252977959 learned_addr learned my addr 172.21.2.117:0/3252977959 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 -- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00e6170 msgr2=0x7f9af00e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:33.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 --2- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00e6170 0x7f9af00e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 -- 172.21.2.117:0/3252977959 >> v1:172.21.2.125:6789/0 conn(0x7f9af00e6b90 legacy=0x7f9af00e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 -- 172.21.2.117:0/3252977959 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00e8d80 con 0x7f9af01b07f0 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeffff700 1 --2- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af01b07f0 0x7f9af00e4e90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f9ae0006650 tx=0x7f9ae0009a80).ready entity=mon.1 client_cookie=7b0e421eddf6c06e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeeffd700 1 -- 172.21.2.117:0/3252977959 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae000c0c0 con 0x7f9af01b07f0 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeeffd700 1 -- 172.21.2.117:0/3252977959 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae000c340 con 0x7f9af01b07f0 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9aeeffd700 1 -- 172.21.2.117:0/3252977959 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae000c790 con 0x7f9af01b07f0 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9af6566700 1 -- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af01b07f0 msgr2=0x7f9af00e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.275 7f9af6566700 1 --2- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af01b07f0 0x7f9af00e4e90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f9ae0006650 tx=0x7f9ae0009a80).stop 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 -- 172.21.2.117:0/3252977959 shutdown_connections 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 --2- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00e6170 0x7f9af00e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 --2- 172.21.2.117:0/3252977959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af01b07f0 0x7f9af00e4e90 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:33.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 -- 172.21.2.117:0/3252977959 >> 172.21.2.117:0/3252977959 conn(0x7f9af00ce500 msgr2=0x7f9af00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:33.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 -- 172.21.2.117:0/3252977959 shutdown_connections 2021-02-10T11:47:33.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.276 7f9af6566700 1 -- 172.21.2.117:0/3252977959 wait complete. 2021-02-10T11:47:33.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.290 7f9af6566700 1 Processor -- start 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 -- start start 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00966e0 0x7f9af009a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 0x7f9af0187d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af009b1a0 con 0x7f9af0110ae0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af0096ae0 con 0x7f9af0110ee0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af6566700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af009bd40 con 0x7f9af00966e0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9aef7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9aef7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52402/0 (socket says 172.21.2.117:52402) 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9aef7fe700 1 -- 172.21.2.117:0/1895419596 learned_addr learned my addr 172.21.2.117:0/1895419596 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9aeffff700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00966e0 0x7f9af009a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.291 7f9af4b83700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 0x7f9af0187d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 -- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00966e0 msgr2=0x7f9af009a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00966e0 0x7f9af009a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 -- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 msgr2=0x7f9af0187d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 0x7f9af0187d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 -- 172.21.2.117:0/1895419596 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af01a28c0 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9af4b83700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 0x7f9af0187d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aef7fe700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7f9ae4006f60 tx=0x7f9ae400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae40009b0 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae400bdf0 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9af6566700 1 -- 172.21.2.117:0/1895419596 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af01a28c0 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ae4002030 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.292 7f9af6566700 1 -- 172.21.2.117:0/1895419596 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af0188620 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.293 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ae4006b00 con 0x7f9af0110ae0 2021-02-10T11:47:33.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.293 7f9aecff9700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 0x7f9ad001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:33.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.293 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f9ae4032960 con 0x7f9af0110ae0 2021-02-10T11:47:33.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.293 7f9aeffff700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 0x7f9ad001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:33.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.293 7f9aeffff700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 0x7f9ad001abd0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:33.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.324 7f9af6566700 1 -- 172.21.2.117:0/1895419596 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af010c2c0 con 0x7f9af0110ae0 2021-02-10T11:47:33.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.327 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ae40009b0 con 0x7f9af0110ae0 2021-02-10T11:47:33.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:33.454 7f9af6566700 1 -- 172.21.2.117:0/1895419596 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9af00e77f0 con 0x7f9af0110ae0 2021-02-10T11:47:34.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.235 7f9aecff9700 1 -- 172.21.2.117:0/1895419596 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9ae4006b00 con 0x7f9af0110ae0 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 msgr2=0x7f9ad001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 0x7f9ad001abd0 crc :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 msgr2=0x7f9af009b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7f9ae4006f60 tx=0x7f9ae400b970).stop 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 shutdown_connections 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ad00186d0 0x7f9ad001abd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9af0110ae0 0x7f9af009b720 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9af0110ee0 0x7f9af0187d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 --2- 172.21.2.117:0/1895419596 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9af00966e0 0x7f9af009a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 >> 172.21.2.117:0/1895419596 conn(0x7f9af0097720 msgr2=0x7f9af01994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:34.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 shutdown_connections 2021-02-10T11:47:34.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.276 7f9af6566700 1 -- 172.21.2.117:0/1895419596 wait complete. 2021-02-10T11:47:34.279 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:47:34.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.700 7f1ca4b22700 1 Processor -- start 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.700 7f1ca4b22700 1 -- start start 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1ca4b22700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 0x7f1ca0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1ca4b22700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 0x7f1ca0085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1ca4b22700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca0085930 con 0x7f1ca00e4e40 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1ca4b22700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca0085680 con 0x7f1ca00e5240 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1ca4b22700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca01b09e0 con 0x7f1ca0084c60 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9e85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1ca00e4e40 0x7f1ca01b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52660/0 (socket says 172.21.2.117:52660) 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9e85e700 1 -- 172.21.2.117:0/1448048167 learned_addr learned my addr 172.21.2.117:0/1448048167 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9f05f700 1 --2- 172.21.2.117:0/1448048167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 0x7f1ca0085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9e05d700 1 --2- 172.21.2.117:0/1448048167 >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 0x7f1ca0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9d85c700 1 -- 172.21.2.117:0/1448048167 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042789530 0 0) 0x7f1c98001500 con 0x7f1ca00e4e40 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9f05f700 1 -- 172.21.2.117:0/1448048167 >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 msgr2=0x7f1ca0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.701 7f1c9f05f700 1 --2- 172.21.2.117:0/1448048167 >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 0x7f1ca0086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9f05f700 1 -- 172.21.2.117:0/1448048167 >> v1:172.21.2.117:6789/0 conn(0x7f1ca00e4e40 legacy=0x7f1ca01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9f05f700 1 -- 172.21.2.117:0/1448048167 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0084840 con 0x7f1ca0084c60 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9e05d700 1 --2- 172.21.2.117:0/1448048167 >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 0x7f1ca0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9f05f700 1 --2- 172.21.2.117:0/1448048167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 0x7f1ca0085060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1c90006ad0 tx=0x7f1c90009ad0).ready entity=mon.2 client_cookie=1dffa679fe16edb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9d85c700 1 -- 172.21.2.117:0/1448048167 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c900066a0 con 0x7f1ca0084c60 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9d85c700 1 -- 172.21.2.117:0/1448048167 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c90006410 con 0x7f1ca0084c60 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1c9d85c700 1 -- 172.21.2.117:0/1448048167 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c90000c10 con 0x7f1ca0084c60 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1ca4b22700 1 -- 172.21.2.117:0/1448048167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 msgr2=0x7f1ca0085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.702 7f1ca4b22700 1 --2- 172.21.2.117:0/1448048167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 0x7f1ca0085060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1c90006ad0 tx=0x7f1c90009ad0).stop 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 -- 172.21.2.117:0/1448048167 shutdown_connections 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 --2- 172.21.2.117:0/1448048167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca0084c60 0x7f1ca0085060 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 --2- 172.21.2.117:0/1448048167 >> v2:172.21.2.125:3300/0 conn(0x7f1ca00e5240 0x7f1ca0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 -- 172.21.2.117:0/1448048167 >> 172.21.2.117:0/1448048167 conn(0x7f1ca00ce4e0 msgr2=0x7f1ca00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:34.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 -- 172.21.2.117:0/1448048167 shutdown_connections 2021-02-10T11:47:34.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.703 7f1ca4b22700 1 -- 172.21.2.117:0/1448048167 wait complete. 2021-02-10T11:47:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.717 7f1ca4b22700 1 Processor -- start 2021-02-10T11:47:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.717 7f1ca4b22700 1 -- start start 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 0x7f1ca0097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ca00adbd0 0x7f1ca009b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca018d0c0 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca009ae70 con 0x7f1ca00adbd0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1ca4b22700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca009ba10 con 0x7f1ca00973b0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52420/0 (socket says 172.21.2.117:52420) 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 -- 172.21.2.117:0/3105008923 learned_addr learned my addr 172.21.2.117:0/3105008923 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9f05f700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ca00adbd0 0x7f1ca009b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e85e700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 0x7f1ca0097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 -- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 msgr2=0x7f1ca0097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 0x7f1ca0097980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.718 7f1c9e05d700 1 -- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ca00adbd0 msgr2=0x7f1ca009b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c9e05d700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ca00adbd0 0x7f1ca009b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c9e05d700 1 -- 172.21.2.117:0/3105008923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca009bbf0 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c9e85e700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 0x7f1ca0097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c9e05d700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7f1c94006f40 tx=0x7f1c9400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c9400bd80 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94006b00 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1c940022b0 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca009bbf0 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.719 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca009c180 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.720 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1c94000c90 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.720 7f1c8f7fe700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 0x7f1c8401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:34.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.720 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f1c94032960 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.720 7f1c9e85e700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 0x7f1c8401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:34.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.720 7f1c9e85e700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 0x7f1c8401abd0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:34.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.750 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca0085930 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.753 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1c94000960 con 0x7f1ca00ad1b0 2021-02-10T11:47:34.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:34.878 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1ca0085930 con 0x7f1ca00ad1b0 2021-02-10T11:47:35.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.239 7f1c8f7fe700 1 -- 172.21.2.117:0/3105008923 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 185+0+0 (secure 0 0 0) 0x7f1c94000c90 con 0x7f1ca00ad1b0 2021-02-10T11:47:35.238 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 msgr2=0x7f1c8401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 0x7f1c8401abd0 crc :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 msgr2=0x7f1ca00ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 rx=0x7f1c94006f40 tx=0x7f1c9400b990).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 shutdown_connections 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1c840186d0 0x7f1c8401abd0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1ca00ad1b0 0x7f1ca00ad5b0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1ca00adbd0 0x7f1ca009b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 --2- 172.21.2.117:0/3105008923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1ca00973b0 0x7f1ca0097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 >> 172.21.2.117:0/3105008923 conn(0x7f1ca018cc70 msgr2=0x7f1ca0093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:35.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 shutdown_connections 2021-02-10T11:47:35.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.245 7f1ca4b22700 1 -- 172.21.2.117:0/3105008923 wait complete. 2021-02-10T11:47:35.248 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:47:35.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.669 7f42ab74f700 1 Processor -- start 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.669 7f42ab74f700 1 -- start start 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42ab74f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 0x7f42a41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42ab74f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42a4085450 0x7f42a40d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42ab74f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a4085930 con 0x7f42a40e5240 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42ab74f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42a40e5cc0 con 0x7f42a4085450 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42ab74f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4084840 con 0x7f42a40e4e40 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42a9d6c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42a4085450 0x7f42a40d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42a9d6c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42a4085450 0x7f42a40d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:56808/0 (socket says 172.21.2.117:56808) 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42a9d6c700 1 -- 172.21.2.117:0/3670505483 learned_addr learned my addr 172.21.2.117:0/3670505483 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42a8d6a700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f42a40e5240 0x7f42a4084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45600/0 (socket says 172.21.2.117:45600) 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.670 7f42a956b700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 0x7f42a41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a9d6c700 1 -- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 msgr2=0x7f42a41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a9d6c700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 0x7f42a41b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a9d6c700 1 -- 172.21.2.117:0/3670505483 >> v1:172.21.2.125:6789/0 conn(0x7f42a40e5240 legacy=0x7f42a4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a9d6c700 1 -- 172.21.2.117:0/3670505483 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40d1be0 con 0x7f42a4085450 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a956b700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 0x7f42a41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f42a9d6c700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a4085450 0x7f42a40d05b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f429c006aa0 tx=0x7f429c009f20).ready entity=mon.1 client_cookie=30bd8dc7165885d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f4297fff700 1 -- 172.21.2.117:0/3670505483 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f429c00b5c0 con 0x7f42a4085450 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f4297fff700 1 -- 172.21.2.117:0/3670505483 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c003030 con 0x7f42a4085450 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.671 7f4297fff700 1 -- 172.21.2.117:0/3670505483 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f429c00b9f0 con 0x7f42a4085450 2021-02-10T11:47:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 -- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a4085450 msgr2=0x7f42a40d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a4085450 0x7f42a40d05b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f429c006aa0 tx=0x7f429c009f20).stop 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 -- 172.21.2.117:0/3670505483 shutdown_connections 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a4085450 0x7f42a40d05b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 --2- 172.21.2.117:0/3670505483 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40e4e40 0x7f42a41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 -- 172.21.2.117:0/3670505483 >> 172.21.2.117:0/3670505483 conn(0x7f42a40ce4e0 msgr2=0x7f42a40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 -- 172.21.2.117:0/3670505483 shutdown_connections 2021-02-10T11:47:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.672 7f42ab74f700 1 -- 172.21.2.117:0/3670505483 wait complete. 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.686 7f42ab74f700 1 Processor -- start 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.686 7f42ab74f700 1 -- start start 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40973b0 0x7f42a4097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a40ad1b0 0x7f42a40ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a418d0c0 con 0x7f42a40adbd0 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a409ae70 con 0x7f42a40ad1b0 2021-02-10T11:47:35.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42ab74f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a409ba10 con 0x7f42a40973b0 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42a9d6c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42a9d6c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52428/0 (socket says 172.21.2.117:52428) 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42a9d6c700 1 -- 172.21.2.117:0/2267760952 learned_addr learned my addr 172.21.2.117:0/2267760952 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42a8d6a700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a40ad1b0 0x7f42a40ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.687 7f42a956b700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40973b0 0x7f42a4097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 -- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40973b0 msgr2=0x7f42a4097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40973b0 0x7f42a4097980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 -- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a40ad1b0 msgr2=0x7f42a40ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a40ad1b0 0x7f42a40ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 -- 172.21.2.117:0/2267760952 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a409bbf0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42a9d6c700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f429c00b970 tx=0x7f429c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f429c009450 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429c00c2f0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f429c00c810 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a409bbf0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.688 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a409bbf0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.689 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f429c009fb0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.689 7f4295ffb700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 0x7f428401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.689 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f429c0328c0 con 0x7f42a40adbd0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.689 7f42a956b700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 0x7f428401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.690 7f42a956b700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 0x7f428401abc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:35.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.720 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4085930 con 0x7f42a40adbd0 2021-02-10T11:47:35.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.722 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f429c01cd50 con 0x7f42a40adbd0 2021-02-10T11:47:35.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:35.847 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f42a4085930 con 0x7f42a40adbd0 2021-02-10T11:47:36.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.246 7f4295ffb700 1 -- 172.21.2.117:0/2267760952 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 173+0+0 (secure 0 0 0) 0x7f429c01cd50 con 0x7f42a40adbd0 2021-02-10T11:47:36.257 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:47:36.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 msgr2=0x7f428401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 0x7f428401abc0 crc :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 msgr2=0x7f42a409b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 rx=0x7f429c00b970 tx=0x7f429c00bd10).stop 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 shutdown_connections 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42840186c0 0x7f428401abc0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42a40adbd0 0x7f42a409b3f0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42a40ad1b0 0x7f42a40ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 --2- 172.21.2.117:0/2267760952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42a40973b0 0x7f42a4097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.264 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 >> 172.21.2.117:0/2267760952 conn(0x7f42a418cc70 msgr2=0x7f42a4093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.265 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 shutdown_connections 2021-02-10T11:47:36.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.265 7f42ab74f700 1 -- 172.21.2.117:0/2267760952 wait complete. 2021-02-10T11:47:36.268 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:47:36.268 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:47:36.268 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:47:36.268 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:47:36.268 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:47:36.268 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:47:36.269 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:47:36.269 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:47:36.274 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:47:36.274 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:47:36.275 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:47:36.275 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:47:36.317 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:47:36.317 DEBUG:teuthology.parallel:result is None 2021-02-10T11:47:36.317 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:47:36.318 DEBUG:teuthology.parallel:result is None 2021-02-10T11:47:36.318 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:47:36.722 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:47:36.724 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:47:36.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.732 7f16a8a90700 1 Processor -- start 2021-02-10T11:47:36.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.732 7f16a8a90700 1 -- start start 2021-02-10T11:47:36.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a8a90700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 0x7f16a40e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a8a90700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f16a40e8390 0x7f16a40e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a8a90700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a40e7580 con 0x7f16a41b07f0 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a8a90700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a40e5ce0 con 0x7f16a40e8390 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a8a90700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40e77f0 con 0x7f16a40e7180 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a285e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f16a41b07f0 0x7f16a40e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45622/0 (socket says 172.21.2.117:45622) 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a285e700 1 -- 172.21.2.117:0/2637059559 learned_addr learned my addr 172.21.2.117:0/2637059559 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 --2- 172.21.2.117:0/2637059559 >> v2:172.21.2.125:3300/0 conn(0x7f16a40e8390 0x7f16a40e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f169a05d700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 0x7f16a40e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a205d700 1 -- 172.21.2.117:0/2637059559 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482308597 0 0) 0x7f1694001510 con 0x7f16a41b07f0 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a205d700 1 -- 172.21.2.117:0/2637059559 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1690002fe0 con 0x7f16a41b07f0 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 -- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 msgr2=0x7f16a40e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 0x7f16a40e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 -- 172.21.2.117:0/2637059559 >> v1:172.21.2.125:6789/0 conn(0x7f16a41b07f0 legacy=0x7f16a40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 -- 172.21.2.117:0/2637059559 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a40e8db0 con 0x7f16a40e8390 2021-02-10T11:47:36.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f169a05d700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 0x7f16a40e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a305f700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a40e8390 0x7f16a40e8790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f169c006ad0 tx=0x7f169c009ad0).ready entity=mon.1 client_cookie=c35294b9b7fd42f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a205d700 1 -- 172.21.2.117:0/2637059559 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f169c0022e0 con 0x7f16a40e8390 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a205d700 1 -- 172.21.2.117:0/2637059559 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c006410 con 0x7f16a40e8390 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 -- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a40e8390 msgr2=0x7f16a40e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a40e8390 0x7f16a40e8790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f169c006ad0 tx=0x7f169c009ad0).stop 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.733 7f16a205d700 1 -- 172.21.2.117:0/2637059559 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f169c002620 con 0x7f16a40e8390 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 -- 172.21.2.117:0/2637059559 shutdown_connections 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a40e8390 0x7f16a40e8790 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 --2- 172.21.2.117:0/2637059559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a40e7180 0x7f16a40e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 -- 172.21.2.117:0/2637059559 >> 172.21.2.117:0/2637059559 conn(0x7f16a40ce500 msgr2=0x7f16a40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 -- 172.21.2.117:0/2637059559 shutdown_connections 2021-02-10T11:47:36.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.734 7f16a8a90700 1 -- 172.21.2.117:0/2637059559 wait complete. 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 Processor -- start 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 -- start start 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a4094c60 0x7f16a40b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a419dd10 0x7f16a4093af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a419e110 con 0x7f16a419d2f0 2021-02-10T11:47:36.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40ae1b0 con 0x7f16a419dd10 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.749 7f16a8a90700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a4095060 con 0x7f16a4094c60 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52456/0 (socket says 172.21.2.117:52456) 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 -- 172.21.2.117:0/937269207 learned_addr learned my addr 172.21.2.117:0/937269207 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 -- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a4094c60 msgr2=0x7f16a40b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a4094c60 0x7f16a40b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 -- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a419dd10 msgr2=0x7f16a4093af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a419dd10 0x7f16a4093af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 -- 172.21.2.117:0/937269207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4094110 con 0x7f16a419d2f0 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169a05d700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f168c006f40 tx=0x7f168c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f168c00bd80 con 0x7f16a419d2f0 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c000c90 con 0x7f16a419d2f0 2021-02-10T11:47:36.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f168c0022b0 con 0x7f16a419d2f0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f16a8a90700 1 -- 172.21.2.117:0/937269207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a4094110 con 0x7f16a419d2f0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.750 7f16a8a90700 1 -- 172.21.2.117:0/937269207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a4094110 con 0x7f16a419d2f0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.751 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f168c006b00 con 0x7f16a419d2f0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.751 7f169bfff700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 0x7f168801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.751 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f168c0324d0 con 0x7f16a419d2f0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.751 7f16a285e700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 0x7f168801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:36.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.751 7f16a285e700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 0x7f168801abd0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:36.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.781 7f16a8a90700 1 -- 172.21.2.117:0/937269207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a40ae1b0 con 0x7f16a419d2f0 2021-02-10T11:47:36.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.784 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f168c006b00 con 0x7f16a419d2f0 2021-02-10T11:47:36.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:36.922 7f16a8a90700 1 -- 172.21.2.117:0/937269207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f16a40ae1b0 con 0x7f16a419d2f0 2021-02-10T11:47:37.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.250 7f169bfff700 1 -- 172.21.2.117:0/937269207 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7f168c000c90 con 0x7f16a419d2f0 2021-02-10T11:47:37.280 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.287 7f16a8a90700 1 -- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 msgr2=0x7f168801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.287 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 0x7f168801abd0 crc :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.287 7f16a8a90700 1 -- 172.21.2.117:0/937269207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 msgr2=0x7f16a419d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.287 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 rx=0x7f168c006f40 tx=0x7f168c00b990).stop 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 -- 172.21.2.117:0/937269207 shutdown_connections 2021-02-10T11:47:37.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f16880186d0 0x7f168801abd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16a419d2f0 0x7f16a419d6f0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16a419dd10 0x7f16a4093af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 --2- 172.21.2.117:0/937269207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16a4094c60 0x7f16a40b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 -- 172.21.2.117:0/937269207 >> 172.21.2.117:0/937269207 conn(0x7f16a4096150 msgr2=0x7f16a407ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 -- 172.21.2.117:0/937269207 shutdown_connections 2021-02-10T11:47:37.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.288 7f16a8a90700 1 -- 172.21.2.117:0/937269207 wait complete. 2021-02-10T11:47:37.291 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:47:37.291 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.708 7f409326b700 1 Processor -- start 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.708 7f409326b700 1 -- start start 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f409326b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f408c0e5240 0x7f408c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f409326b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c085930 con 0x7f408c0e4e40 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f409326b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c1b09e0 con 0x7f408c084c60 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f409326b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c0e5cc0 con 0x7f408c0e5240 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4090886700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f408c0e5240 0x7f408c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4091087700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f408c0e4e40 0x7f408c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52710/0 (socket says 172.21.2.117:52710) 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4091087700 1 -- 172.21.2.117:0/165691358 learned_addr learned my addr 172.21.2.117:0/165691358 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:37.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4090886700 1 -- 172.21.2.117:0/165691358 >> v1:172.21.2.125:6789/0 conn(0x7f408c084c60 legacy=0x7f408c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f407ffff700 1 -- 172.21.2.117:0/165691358 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276549890 0 0) 0x7f4084001400 con 0x7f408c084c60 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4090886700 1 -- 172.21.2.117:0/165691358 >> v1:172.21.2.117:6789/0 conn(0x7f408c0e4e40 legacy=0x7f408c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.709 7f4090886700 1 -- 172.21.2.117:0/165691358 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c084840 con 0x7f408c0e5240 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f4090886700 1 --2- 172.21.2.117:0/165691358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c0e5240 0x7f408c086ec0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f4088006650 tx=0x7f4088009a00).ready entity=mon.0 client_cookie=b3ba2a79b78f7ad0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f407ffff700 1 -- 172.21.2.117:0/165691358 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40880009b0 con 0x7f408c0e5240 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f407ffff700 1 -- 172.21.2.117:0/165691358 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408800c0c0 con 0x7f408c0e5240 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f407ffff700 1 -- 172.21.2.117:0/165691358 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f408800c5a0 con 0x7f408c0e5240 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f409326b700 1 -- 172.21.2.117:0/165691358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c0e5240 msgr2=0x7f408c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.710 7f409326b700 1 --2- 172.21.2.117:0/165691358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c0e5240 0x7f408c086ec0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 rx=0x7f4088006650 tx=0x7f4088009a00).stop 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.711 7f409326b700 1 -- 172.21.2.117:0/165691358 shutdown_connections 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.711 7f409326b700 1 --2- 172.21.2.117:0/165691358 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c0e5240 0x7f408c086ec0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.711 7f409326b700 1 -- 172.21.2.117:0/165691358 >> 172.21.2.117:0/165691358 conn(0x7f408c0ce4e0 msgr2=0x7f408c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.711 7f409326b700 1 -- 172.21.2.117:0/165691358 shutdown_connections 2021-02-10T11:47:37.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.711 7f409326b700 1 -- 172.21.2.117:0/165691358 wait complete. 2021-02-10T11:47:37.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.725 7f409326b700 1 Processor -- start 2021-02-10T11:47:37.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 -- start start 2021-02-10T11:47:37.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 0x7f408c10d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 0x7f408c10ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1aa870 con 0x7f408c10e8f0 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1aaae0 con 0x7f408c1acf90 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f409326b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10f310 con 0x7f408c1aa3a0 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4090886700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4091888700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 0x7f408c10ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4090886700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:56854/0 (socket says 172.21.2.117:56854) 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4090886700 1 -- 172.21.2.117:0/790377256 learned_addr learned my addr 172.21.2.117:0/790377256 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4091888700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 0x7f408c10ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52466/0 (socket says 172.21.2.117:52466) 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.726 7f4091087700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 0x7f408c10d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 -- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 msgr2=0x7f408c10dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 -- 172.21.2.117:0/790377256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 msgr2=0x7f408c10ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 0x7f408c10ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 -- 172.21.2.117:0/790377256 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c10f530 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4090886700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f4091087700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 0x7f408c10d6e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4080001430 tx=0x7f408000a8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f408000ac90 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080000d30 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f409326b700 1 -- 172.21.2.117:0/790377256 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c10f530 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40800070d0 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.727 7f409326b700 1 -- 172.21.2.117:0/790377256 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c10f530 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.728 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4080008000 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.728 7f407dffb700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 0x7f406c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.728 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4080032510 con 0x7f408c1aa3a0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.728 7f4090886700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 0x7f406c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:37.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.729 7f4090886700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 0x7f406c01abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:37.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.758 7f409326b700 1 -- 172.21.2.117:0/790377256 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c10c260 con 0x7f408c1aa3a0 2021-02-10T11:47:37.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.762 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4080007480 con 0x7f408c1aa3a0 2021-02-10T11:47:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.896 7f409326b700 1 -- 172.21.2.117:0/790377256 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f408c085930 con 0x7f408c1aa3a0 2021-02-10T11:47:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.897 7f407dffb700 1 -- 172.21.2.117:0/790377256 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f4080000d30 con 0x7f408c1aa3a0 2021-02-10T11:47:37.894 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:37.894 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:47:37.894 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:47:37.895 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.896 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.897 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.898 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.899 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.900 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.901 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.902 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.903 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:47:37.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:37.905 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:47:37.906 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:47:37.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 msgr2=0x7f406c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 0x7f406c01abc0 crc :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 msgr2=0x7f408c10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 0x7f408c10d6e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f4080001430 tx=0x7f408000a8a0).stop 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 shutdown_connections 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f406c018730 0x7f406c01abc0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f408c10e8f0 0x7f408c10ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f408c1acf90 0x7f408c10dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 --2- 172.21.2.117:0/790377256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408c1aa3a0 0x7f408c10d6e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 >> 172.21.2.117:0/790377256 conn(0x7f408c1b2c50 msgr2=0x7f408c1a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 shutdown_connections 2021-02-10T11:47:37.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:37.902 7f409326b700 1 -- 172.21.2.117:0/790377256 wait complete. 2021-02-10T11:47:37.909 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:38.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 Processor -- start 2021-02-10T11:47:38.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 -- start start 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc0ef3f0 0x7f7cbc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0ede60 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0d1ce0 con 0x7f7cbc0d18e0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc47ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0ef7f0 con 0x7f7cbc0ef3f0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc2e0c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7cbc0eb7f0 0x7f7cbc1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52728/0 (socket says 172.21.2.117:52728) 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.324 7f7cc2e0c700 1 -- 172.21.2.117:0/1871357656 learned_addr learned my addr 172.21.2.117:0/1871357656 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708823509 0 0) 0x7f7cb4001500 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca8002fe0 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1e0a700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc0ef3f0 0x7f7cbc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 498028535 0 0) 0x7f7cb4001270 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc0ef3f0 msgr2=0x7f7cbc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc0ef3f0 0x7f7cbc0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 >> v1:172.21.2.125:6789/0 conn(0x7f7cbc0d18e0 legacy=0x7f7cbc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc1bb180 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7cb4001b50 con 0x7f7cbc0eb7f0 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 >> v1:172.21.2.117:6789/0 conn(0x7f7cbc0eb7f0 legacy=0x7f7cbc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ca8003b90 0x7f7ca8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ca8006790 0x7f7ca8008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ca80095b0 0x7f7ca800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca800c2d0 con 0x7f7ca8006790 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca800c4f0 con 0x7f7ca8003b90 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.325 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca800c710 con 0x7f7ca80095b0 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc260b700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ca8003b90 0x7f7ca8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc1e0a700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ca8006790 0x7f7ca8008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ca80095b0 0x7f7ca800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 -- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ca8003b90 msgr2=0x7f7ca8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7ca8003b90 0x7f7ca8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 -- 172.21.2.117:0/1871357656 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ca8006790 msgr2=0x7f7ca8008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7ca8006790 0x7f7ca8008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 -- 172.21.2.117:0/1871357656 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca800c9b0 con 0x7f7ca80095b0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ca80095b0 0x7f7ca800ba80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f7cb4007c90 tx=0x7f7cb400a310).ready entity=mon.2 client_cookie=8221c93ce7c0721 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cb4001e80 con 0x7f7ca80095b0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb40070c0 con 0x7f7ca80095b0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc1609700 1 -- 172.21.2.117:0/1871357656 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cb40075e0 con 0x7f7ca80095b0 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc47ef700 1 -- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ca80095b0 msgr2=0x7f7ca800ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc47ef700 1 --2- 172.21.2.117:0/1871357656 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7ca80095b0 0x7f7ca800ba80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f7cb4007c90 tx=0x7f7cb400a310).stop 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.326 7f7cc2e0c700 1 -- 172.21.2.117:0/1871357656 reap_dead start 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.327 7f7cc47ef700 1 -- 172.21.2.117:0/1871357656 shutdown_connections 2021-02-10T11:47:38.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.327 7f7cc47ef700 1 -- 172.21.2.117:0/1871357656 >> 172.21.2.117:0/1871357656 conn(0x7f7cbc0ced60 msgr2=0x7f7cbc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.327 7f7cc47ef700 1 -- 172.21.2.117:0/1871357656 shutdown_connections 2021-02-10T11:47:38.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.327 7f7cc47ef700 1 -- 172.21.2.117:0/1871357656 wait complete. 2021-02-10T11:47:38.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 Processor -- start 2021-02-10T11:47:38.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 -- start start 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cbc0b2160 0x7f7cbc0e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc08c600 0x7f7cbc08a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc08b660 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc08ca00 con 0x7f7cbc0b2160 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.342 7f7cc47ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0e9a80 con 0x7f7cbc08c600 2021-02-10T11:47:38.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52488/0 (socket says 172.21.2.117:52488) 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 -- 172.21.2.117:0/2775329735 learned_addr learned my addr 172.21.2.117:0/2775329735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 -- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc08c600 msgr2=0x7f7cbc08a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc08c600 0x7f7cbc08a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 -- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cbc0b2160 msgr2=0x7f7cbc0e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cbc0b2160 0x7f7cbc0e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 -- 172.21.2.117:0/2775329735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc08a8a0 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc1e0a700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f7cb000b590 tx=0x7f7cb000b930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cb000bd20 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb0006710 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cb00022b0 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc08a8a0 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.343 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc08ae30 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.344 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7cb0006710 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.344 7f7caeffd700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 0x7f7ca401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.344 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f7cb0032320 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.345 7f7cc260b700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 0x7f7ca401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.345 7f7cc260b700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 0x7f7ca401abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.374 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc08b410 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.377 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7cb0000c90 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.502 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cbc08b410 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.503 7f7caeffd700 1 -- 172.21.2.117:0/2775329735 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+11378 (secure 0 0 0) 0x7f7cb0006710 con 0x7f7cbc08bbe0 2021-02-10T11:47:38.498 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:38.498 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":87,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:37.245459","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 msgr2=0x7f7ca401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 0x7f7ca401abc0 crc :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 msgr2=0x7f7cbc08bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 rx=0x7f7cb000b590 tx=0x7f7cb000b930).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 shutdown_connections 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ca40186c0 0x7f7ca401abc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cbc08bbe0 0x7f7cbc08bfe0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cbc08c600 0x7f7cbc08a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 --2- 172.21.2.117:0/2775329735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cbc0b2160 0x7f7cbc0e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 >> 172.21.2.117:0/2775329735 conn(0x7f7cbc0bb5e0 msgr2=0x7f7cbc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 shutdown_connections 2021-02-10T11:47:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.505 7f7cc47ef700 1 -- 172.21.2.117:0/2775329735 wait complete. 2021-02-10T11:47:38.508 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:47:38.508 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.926 7fce08e77700 1 Processor -- start 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.926 7fce08e77700 1 -- start start 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce08e77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 0x7fce040eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce08e77700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce040ef3f0 0x7fce041b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce08e77700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce040ede60 con 0x7fce040eb040 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce08e77700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce04088020 con 0x7fce040ef3f0 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce08e77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce040d1d30 con 0x7fce040d18f0 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce040ef3f0 0x7fce041b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce040ef3f0 0x7fce041b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52496/0 (socket says 172.21.2.117:52496) 2021-02-10T11:47:38.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 -- 172.21.2.117:0/2093812694 learned_addr learned my addr 172.21.2.117:0/2093812694 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 -- 172.21.2.117:0/2093812694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 msgr2=0x7fce040eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce02b13700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 0x7fce040eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 0x7fce040eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 -- 172.21.2.117:0/2093812694 >> v1:172.21.2.125:6789/0 conn(0x7fce040eb040 legacy=0x7fce040eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce03314700 1 -- 172.21.2.117:0/2093812694 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce041b8fd0 con 0x7fce040ef3f0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.927 7fce02b13700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 0x7fce040eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce03314700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040ef3f0 0x7fce041b79a0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7fcdfc006f00 tx=0x7fcdfc009f10).ready entity=mon.0 client_cookie=e4cbb14a5e9dda36 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce01b11700 1 -- 172.21.2.117:0/2093812694 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdfc00b3b0 con 0x7fce040ef3f0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce01b11700 1 -- 172.21.2.117:0/2093812694 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc00b730 con 0x7fce040ef3f0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce01b11700 1 -- 172.21.2.117:0/2093812694 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdfc00bc50 con 0x7fce040ef3f0 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce08e77700 1 -- 172.21.2.117:0/2093812694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040ef3f0 msgr2=0x7fce041b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.928 7fce08e77700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040ef3f0 0x7fce041b79a0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 rx=0x7fcdfc006f00 tx=0x7fcdfc009f10).stop 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 -- 172.21.2.117:0/2093812694 shutdown_connections 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040ef3f0 0x7fce041b79a0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 --2- 172.21.2.117:0/2093812694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040d18f0 0x7fce040eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 -- 172.21.2.117:0/2093812694 >> 172.21.2.117:0/2093812694 conn(0x7fce040cf060 msgr2=0x7fce040cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 -- 172.21.2.117:0/2093812694 shutdown_connections 2021-02-10T11:47:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.929 7fce08e77700 1 -- 172.21.2.117:0/2093812694 wait complete. 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.943 7fce08e77700 1 Processor -- start 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 -- start start 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce040ac450 0x7fce040ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040b5d20 0x7fce040b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce040ace70 con 0x7fce040a1930 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce040ad090 con 0x7fce040ac450 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce08e77700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce040b6740 con 0x7fce040b5d20 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52498/0 (socket says 172.21.2.117:52498) 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 -- 172.21.2.117:0/1620727120 learned_addr learned my addr 172.21.2.117:0/1620727120 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 -- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040b5d20 msgr2=0x7fce040b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040b5d20 0x7fce040b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 -- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce040ac450 msgr2=0x7fce040ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce040ac450 0x7fce040ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.944 7fce02b13700 1 -- 172.21.2.117:0/1620727120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce040b6960 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fce02b13700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 rx=0x7fcdf4006f20 tx=0x7fcdf400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdf40009b0 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf400bd80 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdf4002220 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fce08e77700 1 -- 172.21.2.117:0/1620727120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce040b6960 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.945 7fce08e77700 1 -- 172.21.2.117:0/1620727120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce040b6960 con 0x7fce040a1930 2021-02-10T11:47:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.946 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcdf4000c90 con 0x7fce040a1930 2021-02-10T11:47:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.946 7fcdfb7fe700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 0x7fcde801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.946 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fcdf4032600 con 0x7fce040a1930 2021-02-10T11:47:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.946 7fce02312700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 0x7fcde801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.947 7fce02312700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 0x7fcde801abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.977 7fce08e77700 1 -- 172.21.2.117:0/1620727120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040f67e0 con 0x7fce040a1930 2021-02-10T11:47:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:38.979 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcdf4006bb0 con 0x7fce040a1930 2021-02-10T11:47:39.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.106 7fce08e77700 1 -- 172.21.2.117:0/1620727120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fce040f67e0 con 0x7fce040a1930 2021-02-10T11:47:39.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.106 7fcdfb7fe700 1 -- 172.21.2.117:0/1620727120 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v87) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcdf400bd80 con 0x7fce040a1930 2021-02-10T11:47:39.102 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:39.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.108 7fce08e77700 1 -- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 msgr2=0x7fcde801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.108 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 0x7fcde801abc0 crc :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.108 7fce08e77700 1 -- 172.21.2.117:0/1620727120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 msgr2=0x7fce0408a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.108 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 rx=0x7fcdf4006f20 tx=0x7fcdf400b990).stop 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 -- 172.21.2.117:0/1620727120 shutdown_connections 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce040b5d20 0x7fce040b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce040ac450 0x7fce040ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcde8018730 0x7fcde801abc0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 --2- 172.21.2.117:0/1620727120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce040a1930 0x7fce0408a220 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 -- 172.21.2.117:0/1620727120 >> 172.21.2.117:0/1620727120 conn(0x7fce040b6c70 msgr2=0x7fce040970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 -- 172.21.2.117:0/1620727120 shutdown_connections 2021-02-10T11:47:39.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.109 7fce08e77700 1 -- 172.21.2.117:0/1620727120 wait complete. 2021-02-10T11:47:39.112 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.527 7fd1779db700 1 Processor -- start 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.527 7fd1779db700 1 -- start start 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1779db700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd1701b6960 0x7fd1701b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1779db700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1700ede60 con 0x7fd1700d18e0 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1779db700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1700eb9c0 con 0x7fd1700ef3f0 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1779db700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1700ef7f0 con 0x7fd1701b6960 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1757f7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd1700d18e0 0x7fd1700ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52756/0 (socket says 172.21.2.117:52756) 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.528 7fd1757f7700 1 -- 172.21.2.117:0/914032788 learned_addr learned my addr 172.21.2.117:0/914032788 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:39.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd175ff8700 1 --2- 172.21.2.117:0/914032788 >> v2:172.21.2.125:3300/0 conn(0x7fd1701b6960 0x7fd1701b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd167fff700 1 -- 172.21.2.117:0/914032788 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335728582 0 0) 0x7fd16c001180 con 0x7fd1700d18e0 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd167fff700 1 -- 172.21.2.117:0/914032788 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c002fe0 con 0x7fd1700d18e0 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd175ff8700 1 -- 172.21.2.117:0/914032788 >> v1:172.21.2.125:6789/0 conn(0x7fd1700ef3f0 legacy=0x7fd1700eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd175ff8700 1 -- 172.21.2.117:0/914032788 >> v1:172.21.2.117:6789/0 conn(0x7fd1700d18e0 legacy=0x7fd1700ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd175ff8700 1 -- 172.21.2.117:0/914032788 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700d1d20 con 0x7fd1701b6960 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.529 7fd175ff8700 1 --2- 172.21.2.117:0/914032788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1701b6960 0x7fd1701b8d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fd168006e50 tx=0x7fd168009e70).ready entity=mon.1 client_cookie=a98825392c69a890 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd167fff700 1 -- 172.21.2.117:0/914032788 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd168006a20 con 0x7fd1701b6960 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd167fff700 1 -- 172.21.2.117:0/914032788 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd168006790 con 0x7fd1701b6960 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd167fff700 1 -- 172.21.2.117:0/914032788 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16800b700 con 0x7fd1701b6960 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd1779db700 1 -- 172.21.2.117:0/914032788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1701b6960 msgr2=0x7fd1701b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd1779db700 1 --2- 172.21.2.117:0/914032788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1701b6960 0x7fd1701b8d70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fd168006e50 tx=0x7fd168009e70).stop 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd1779db700 1 -- 172.21.2.117:0/914032788 shutdown_connections 2021-02-10T11:47:39.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd1779db700 1 --2- 172.21.2.117:0/914032788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1701b6960 0x7fd1701b8d70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.530 7fd1779db700 1 -- 172.21.2.117:0/914032788 >> 172.21.2.117:0/914032788 conn(0x7fd1700ced60 msgr2=0x7fd1700cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.531 7fd1779db700 1 -- 172.21.2.117:0/914032788 shutdown_connections 2021-02-10T11:47:39.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.531 7fd1779db700 1 -- 172.21.2.117:0/914032788 wait complete. 2021-02-10T11:47:39.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.545 7fd1779db700 1 Processor -- start 2021-02-10T11:47:39.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.545 7fd1779db700 1 -- start start 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd170196390 0x7fd1700e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1700e9560 0x7fd1700e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd170196810 con 0x7fd1700e9960 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd170098ad0 con 0x7fd1700e9560 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd1779db700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1700e8a40 con 0x7fd170196390 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd175ff8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd175ff8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52512/0 (socket says 172.21.2.117:52512) 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.546 7fd175ff8700 1 -- 172.21.2.117:0/3784626089 learned_addr learned my addr 172.21.2.117:0/3784626089 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 -- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd170196390 msgr2=0x7fd1700e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd170196390 0x7fd1700e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 -- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1700e9560 msgr2=0x7fd1700e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1700e9560 0x7fd1700e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 -- 172.21.2.117:0/3784626089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1700e8c60 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd175ff8700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7fd1680099f0 tx=0x7fd16800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16800c2f0 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1680093b0 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd16800c760 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd1779db700 1 -- 172.21.2.117:0/3784626089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700e8c60 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.547 7fd1779db700 1 -- 172.21.2.117:0/3784626089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17019cd70 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.548 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1680080d0 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.548 7fd165ffb700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 0x7fd15001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.548 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fd168032770 con 0x7fd1700e9960 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.548 7fd1757f7700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 0x7fd15001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:39.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.549 7fd1757f7700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 0x7fd15001abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:39.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.578 7fd1779db700 1 -- 172.21.2.117:0/3784626089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1700e8c60 con 0x7fd1700e9960 2021-02-10T11:47:39.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.580 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd168008400 con 0x7fd1700e9960 2021-02-10T11:47:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.704 7fd1779db700 1 -- 172.21.2.117:0/3784626089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd1700e8c60 con 0x7fd1700e9960 2021-02-10T11:47:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.705 7fd165ffb700 1 -- 172.21.2.117:0/3784626089 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v87) v1 ==== 84+0+64 (secure 0 0 0) 0x7fd168008110 con 0x7fd1700e9960 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:39.700 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:47:39.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.706 7fd1779db700 1 -- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 msgr2=0x7fd15001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.706 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 0x7fd15001abc0 crc :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.706 7fd1779db700 1 -- 172.21.2.117:0/3784626089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 msgr2=0x7fd1700e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:39.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.706 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 rx=0x7fd1680099f0 tx=0x7fd16800bd10).stop 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 -- 172.21.2.117:0/3784626089 shutdown_connections 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1500186c0 0x7fd15001abc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1700e9960 0x7fd1700e8420 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1700e9560 0x7fd1700e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 --2- 172.21.2.117:0/3784626089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd170196390 0x7fd1700e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 -- 172.21.2.117:0/3784626089 >> 172.21.2.117:0/3784626089 conn(0x7fd17019f610 msgr2=0x7fd1701a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 -- 172.21.2.117:0/3784626089 shutdown_connections 2021-02-10T11:47:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:39.707 7fd1779db700 1 -- 172.21.2.117:0/3784626089 wait complete. 2021-02-10T11:47:39.711 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:47:39.711 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:47:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 Processor -- start 2021-02-10T11:47:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 -- start start 2021-02-10T11:47:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6701b69a0 0x7fc6701b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6700ede60 con 0x7fc6700d18f0 2021-02-10T11:47:40.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6700eba00 con 0x7fc6700ef3f0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.131 7fc674b18700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6700d1cf0 con 0x7fc6701b69a0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66f05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc6701b69a0 0x7fc6701b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66e85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc6700d18f0 0x7fc6700eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52768/0 (socket says 172.21.2.117:52768) 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66e85e700 1 -- 172.21.2.117:0/1815422101 learned_addr learned my addr 172.21.2.117:0/1815422101 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66e05d700 1 --1- 172.21.2.117:0/1815422101 >> v1:172.21.2.125:6789/0 conn(0x7fc6700ef3f0 0x7fc6700eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45694/0 (socket says 172.21.2.117:45694) 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66f05f700 1 -- 172.21.2.117:0/1815422101 >> v1:172.21.2.125:6789/0 conn(0x7fc6700ef3f0 legacy=0x7fc6700eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66d85c700 1 -- 172.21.2.117:0/1815422101 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401886828 0 0) 0x7fc660000c60 con 0x7fc6700ef3f0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66f05f700 1 -- 172.21.2.117:0/1815422101 >> v1:172.21.2.117:6789/0 conn(0x7fc6700d18f0 legacy=0x7fc6700eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66f05f700 1 -- 172.21.2.117:0/1815422101 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6701bb3f0 con 0x7fc6701b69a0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.132 7fc66f05f700 1 --2- 172.21.2.117:0/1815422101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6701b69a0 0x7fc6701b8db0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fc668006e30 tx=0x7fc668009e60).ready entity=mon.0 client_cookie=3be51c5f0dbf9132 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc66d85c700 1 -- 172.21.2.117:0/1815422101 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc668006a50 con 0x7fc6701b69a0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc66d85c700 1 -- 172.21.2.117:0/1815422101 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66800b300 con 0x7fc6701b69a0 2021-02-10T11:47:40.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc66d85c700 1 -- 172.21.2.117:0/1815422101 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc66800b760 con 0x7fc6701b69a0 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 -- 172.21.2.117:0/1815422101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6701b69a0 msgr2=0x7fc6701b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 --2- 172.21.2.117:0/1815422101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6701b69a0 0x7fc6701b8db0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 rx=0x7fc668006e30 tx=0x7fc668009e60).stop 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 -- 172.21.2.117:0/1815422101 shutdown_connections 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 --2- 172.21.2.117:0/1815422101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6701b69a0 0x7fc6701b8db0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 -- 172.21.2.117:0/1815422101 >> 172.21.2.117:0/1815422101 conn(0x7fc6700cf060 msgr2=0x7fc6700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 -- 172.21.2.117:0/1815422101 shutdown_connections 2021-02-10T11:47:40.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.133 7fc674b18700 1 -- 172.21.2.117:0/1815422101 wait complete. 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 Processor -- start 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 -- start start 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6700b3e10 0x7fc6700b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6700b4c10 0x7fc6700aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700b4320 con 0x7fc6700b5010 2021-02-10T11:47:40.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67008a220 con 0x7fc6700b3e10 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc674b18700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700abbc0 con 0x7fc6700b4c10 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc66f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc66f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52526/0 (socket says 172.21.2.117:52526) 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.148 7fc66f05f700 1 -- 172.21.2.117:0/2747218254 learned_addr learned my addr 172.21.2.117:0/2747218254 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66e05d700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6700b4c10 0x7fc6700aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66e85e700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6700b3e10 0x7fc6700b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 -- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6700b4c10 msgr2=0x7fc6700aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6700b4c10 0x7fc6700aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 -- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6700b3e10 msgr2=0x7fc6700b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6700b3e10 0x7fc6700b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 -- 172.21.2.117:0/2747218254 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700abde0 con 0x7fc6700b5010 2021-02-10T11:47:40.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc66f05f700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fc668001070 tx=0x7fc66800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc668001270 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668006df0 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc674b18700 1 -- 172.21.2.117:0/2747218254 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6700abde0 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc6680017b0 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.149 7fc674b18700 1 -- 172.21.2.117:0/2747218254 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6700abde0 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.150 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc668001270 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.150 7fc65f7fe700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 0x7fc65401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.150 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc668032770 con 0x7fc6700b5010 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.151 7fc66e85e700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 0x7fc65401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:40.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.151 7fc66e85e700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 0x7fc65401abc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:40.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.181 7fc674b18700 1 -- 172.21.2.117:0/2747218254 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6700f67e0 con 0x7fc6700b5010 2021-02-10T11:47:40.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.184 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc668020f20 con 0x7fc6700b5010 2021-02-10T11:47:40.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.311 7fc674b18700 1 -- 172.21.2.117:0/2747218254 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fc6700f67e0 con 0x7fc6700b5010 2021-02-10T11:47:40.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.738 7fc65f7fe700 1 -- 172.21.2.117:0/2747218254 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v88) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc668020f20 con 0x7fc6700b5010 2021-02-10T11:47:40.772 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:47:40.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.779 7fc674b18700 1 -- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 msgr2=0x7fc65401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.779 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 0x7fc65401abc0 crc :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.779 7fc674b18700 1 -- 172.21.2.117:0/2747218254 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 msgr2=0x7fc6700ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 rx=0x7fc668001070 tx=0x7fc66800bd40).stop 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 -- 172.21.2.117:0/2747218254 shutdown_connections 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc6540186c0 0x7fc65401abc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc6700b5010 0x7fc6700ab5a0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc6700b4c10 0x7fc6700aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 --2- 172.21.2.117:0/2747218254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc6700b3e10 0x7fc6700b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:40.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 -- 172.21.2.117:0/2747218254 >> 172.21.2.117:0/2747218254 conn(0x7fc6700b6c70 msgr2=0x7fc6700970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:40.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 -- 172.21.2.117:0/2747218254 shutdown_connections 2021-02-10T11:47:40.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:40.780 7fc674b18700 1 -- 172.21.2.117:0/2747218254 wait complete. 2021-02-10T11:47:40.783 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.202 7f46b2178700 1 Processor -- start 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 -- start start 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f46ac0ef3f0 0x7f46ac0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 0x7f46ac1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0ef7f0 con 0x7f46ac0d18f0 2021-02-10T11:47:41.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ac0eded0 con 0x7f46ac0ef3f0 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46b2178700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac0d1cf0 con 0x7f46ac0eb830 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46ab7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f46ac0d18f0 0x7f46ac0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52782/0 (socket says 172.21.2.117:52782) 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.203 7f46ab7fe700 1 -- 172.21.2.117:0/1668541852 learned_addr learned my addr 172.21.2.117:0/1668541852 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 --2- 172.21.2.117:0/1668541852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 0x7f46ac1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aaffd700 1 --2- 172.21.2.117:0/1668541852 >> v2:172.21.2.125:3300/0 conn(0x7f46ac0ef3f0 0x7f46ac0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aa7fc700 1 -- 172.21.2.117:0/1668541852 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920428286 0 0) 0x7f469c000ce0 con 0x7f46ac0d18f0 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aa7fc700 1 -- 172.21.2.117:0/1668541852 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4698002fe0 con 0x7f46ac0d18f0 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 -- 172.21.2.117:0/1668541852 >> v2:172.21.2.125:3300/0 conn(0x7f46ac0ef3f0 msgr2=0x7f46ac0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 --2- 172.21.2.117:0/1668541852 >> v2:172.21.2.125:3300/0 conn(0x7f46ac0ef3f0 0x7f46ac0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 -- 172.21.2.117:0/1668541852 >> v1:172.21.2.117:6789/0 conn(0x7f46ac0d18f0 legacy=0x7f46ac0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 -- 172.21.2.117:0/1668541852 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac1b8fd0 con 0x7f46ac0eb830 2021-02-10T11:47:41.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46abfff700 1 --2- 172.21.2.117:0/1668541852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 0x7f46ac1b79a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f46a4006ad0 tx=0x7f46a4009ad0).ready entity=mon.2 client_cookie=f8247943e49caa53 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aa7fc700 1 -- 172.21.2.117:0/1668541852 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46a40022e0 con 0x7f46ac0eb830 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aa7fc700 1 -- 172.21.2.117:0/1668541852 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a4006410 con 0x7f46ac0eb830 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.204 7f46aa7fc700 1 -- 172.21.2.117:0/1668541852 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46a4002560 con 0x7f46ac0eb830 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 -- 172.21.2.117:0/1668541852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 msgr2=0x7f46ac1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 --2- 172.21.2.117:0/1668541852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 0x7f46ac1b79a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f46a4006ad0 tx=0x7f46a4009ad0).stop 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 -- 172.21.2.117:0/1668541852 shutdown_connections 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 --2- 172.21.2.117:0/1668541852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac0eb830 0x7f46ac1b79a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 --2- 172.21.2.117:0/1668541852 >> v2:172.21.2.125:3300/0 conn(0x7f46ac0ef3f0 0x7f46ac0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 -- 172.21.2.117:0/1668541852 >> 172.21.2.117:0/1668541852 conn(0x7f46ac0cf060 msgr2=0x7f46ac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 -- 172.21.2.117:0/1668541852 shutdown_connections 2021-02-10T11:47:41.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.205 7f46b2178700 1 -- 172.21.2.117:0/1668541852 wait complete. 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 Processor -- start 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 -- start start 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 0x7f46ac0dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 0x7f46ac0dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac085e50 con 0x7f46ac083990 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac097270 con 0x7f46ac1a4180 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.220 7f46b2178700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac097450 con 0x7f46ac1a4580 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52540/0 (socket says 172.21.2.117:52540) 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 -- 172.21.2.117:0/2502133718 learned_addr learned my addr 172.21.2.117:0/2502133718 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 -- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 msgr2=0x7f46ac0dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46abfff700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 0x7f46ac0dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46aaffd700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 0x7f46ac0dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 0x7f46ac0dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 -- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 msgr2=0x7f46ac0dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 0x7f46ac0dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 -- 172.21.2.117:0/2502133718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac09ed70 con 0x7f46ac083990 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46aaffd700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 0x7f46ac0dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46abfff700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 0x7f46ac0dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.221 7f46ab7fe700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7f469c00b5c0 tx=0x7f469c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.222 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f469c006030 con 0x7f46ac083990 2021-02-10T11:47:41.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.222 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469c0043a0 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.222 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f469c004880 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.222 7f46b2178700 1 -- 172.21.2.117:0/2502133718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac09ed70 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.222 7f46b2178700 1 -- 172.21.2.117:0/2502133718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac09f2c0 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.223 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f469c004810 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.223 7f46a3fff700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 0x7f469001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.223 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f469c032530 con 0x7f46ac083990 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.223 7f46aaffd700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 0x7f469001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:41.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.224 7f46aaffd700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 0x7f469001abc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:41.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.253 7f46b2178700 1 -- 172.21.2.117:0/2502133718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac09fb50 con 0x7f46ac083990 2021-02-10T11:47:41.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.256 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f469c01c330 con 0x7f46ac083990 2021-02-10T11:47:41.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.385 7f46b2178700 1 -- 172.21.2.117:0/2502133718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f46ac09fb50 con 0x7f46ac083990 2021-02-10T11:47:41.456 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:41.459 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:47:41.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.743 7f46a3fff700 1 -- 172.21.2.117:0/2502133718 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v89) v1 ==== 125+0+0 (secure 0 0 0) 0x7f469c01c330 con 0x7f46ac083990 2021-02-10T11:47:41.745 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 msgr2=0x7f469001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 0x7f469001abc0 crc :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 msgr2=0x7f46ac0863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 rx=0x7f469c00b5c0 tx=0x7f469c00b960).stop 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 shutdown_connections 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46ac1a4580 0x7f46ac0dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46ac1a4180 0x7f46ac0dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46900186c0 0x7f469001abc0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 --2- 172.21.2.117:0/2502133718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46ac083990 0x7f46ac0863d0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 >> 172.21.2.117:0/2502133718 conn(0x7f46ac0b6540 msgr2=0x7f46ac08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 shutdown_connections 2021-02-10T11:47:41.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:41.752 7f46b2178700 1 -- 172.21.2.117:0/2502133718 wait complete. 2021-02-10T11:47:41.755 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:47:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 Processor -- start 2021-02-10T11:47:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 -- start start 2021-02-10T11:47:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd6780e4e40 0x7fd6781b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6780e5240 0x7fd678084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd678085930 con 0x7fd6781b0810 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780e5cc0 con 0x7fd6780e5240 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.175 7fd67de5f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd678084840 con 0x7fd6780e4e40 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6780e5240 0x7fd678084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd6780e5240 0x7fd678084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52548/0 (socket says 172.21.2.117:52548) 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 -- 172.21.2.117:0/2826528271 learned_addr learned my addr 172.21.2.117:0/2826528271 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 -- 172.21.2.117:0/2826528271 >> v2:172.21.2.125:3300/0 conn(0x7fd6780e4e40 msgr2=0x7fd6781b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 --2- 172.21.2.117:0/2826528271 >> v2:172.21.2.125:3300/0 conn(0x7fd6780e4e40 0x7fd6781b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 -- 172.21.2.117:0/2826528271 >> v1:172.21.2.117:6789/0 conn(0x7fd6781b0810 legacy=0x7fd678085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 -- 172.21.2.117:0/2826528271 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678085280 con 0x7fd6780e5240 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd676ffd700 1 --2- 172.21.2.117:0/2826528271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780e5240 0x7fd678084c60 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7fd668006700 tx=0x7fd668009b80).ready entity=mon.0 client_cookie=478889b18beef1c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd6767fc700 1 -- 172.21.2.117:0/2826528271 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd66800c0c0 con 0x7fd6780e5240 2021-02-10T11:47:42.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd6767fc700 1 -- 172.21.2.117:0/2826528271 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66800c400 con 0x7fd6780e5240 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd6767fc700 1 -- 172.21.2.117:0/2826528271 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd66800c860 con 0x7fd6780e5240 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd67de5f700 1 -- 172.21.2.117:0/2826528271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780e5240 msgr2=0x7fd678084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.176 7fd67de5f700 1 --2- 172.21.2.117:0/2826528271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780e5240 0x7fd678084c60 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 rx=0x7fd668006700 tx=0x7fd668009b80).stop 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 -- 172.21.2.117:0/2826528271 shutdown_connections 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 --2- 172.21.2.117:0/2826528271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780e5240 0x7fd678084c60 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 --2- 172.21.2.117:0/2826528271 >> v2:172.21.2.125:3300/0 conn(0x7fd6780e4e40 0x7fd6781b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 -- 172.21.2.117:0/2826528271 >> 172.21.2.117:0/2826528271 conn(0x7fd6780ce4e0 msgr2=0x7fd6780e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 -- 172.21.2.117:0/2826528271 shutdown_connections 2021-02-10T11:47:42.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.177 7fd67de5f700 1 -- 172.21.2.117:0/2826528271 wait complete. 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 Processor -- start 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 -- start start 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd678099bf0 0x7fd6781ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd67818e5b0 0x7fd67818e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6781ae8d0 con 0x7fd6780b30d0 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780b3af0 con 0x7fd67818e5b0 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.192 7fd67de5f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780b3d10 con 0x7fd678099bf0 2021-02-10T11:47:42.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52556/0 (socket says 172.21.2.117:52556) 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 -- 172.21.2.117:0/743613399 learned_addr learned my addr 172.21.2.117:0/743613399 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 -- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd678099bf0 msgr2=0x7fd6781ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd678099bf0 0x7fd6781ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 -- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd67818e5b0 msgr2=0x7fd67818e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd67818e5b0 0x7fd67818e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 -- 172.21.2.117:0/743613399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780b3f30 con 0x7fd6780b30d0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd677fff700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7fd670009980 tx=0x7fd67000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd670005870 con 0x7fd6780b30d0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd670005a70 con 0x7fd6780b30d0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd67de5f700 1 -- 172.21.2.117:0/743613399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6780b3f30 con 0x7fd6780b30d0 2021-02-10T11:47:42.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd67de5f700 1 -- 172.21.2.117:0/743613399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6781a19b0 con 0x7fd6780b30d0 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.193 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd670006030 con 0x7fd6780b30d0 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.194 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd670005c50 con 0x7fd6780b30d0 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.194 7fd66ffff700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 0x7fd65c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.194 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fd670032c90 con 0x7fd6780b30d0 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.194 7fd6777fe700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 0x7fd65c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.195 7fd6777fe700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 0x7fd65c01abc0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.225 7fd67de5f700 1 -- 172.21.2.117:0/743613399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67818e080 con 0x7fd6780b30d0 2021-02-10T11:47:42.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.228 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd670005c80 con 0x7fd6780b30d0 2021-02-10T11:47:42.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.356 7fd67de5f700 1 -- 172.21.2.117:0/743613399 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fd67818e080 con 0x7fd6780b30d0 2021-02-10T11:47:42.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.360 7fd66ffff700 1 -- 172.21.2.117:0/743613399 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 18 and data pool 19 v66) v1 ==== 189+0+0 (secure 0 0 0) 0x7fd6700062f0 con 0x7fd6780b30d0 2021-02-10T11:47:42.358 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 18 and data pool 19 2021-02-10T11:47:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.365 7fd67de5f700 1 -- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 msgr2=0x7fd65c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.365 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 0x7fd65c01abc0 crc :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.365 7fd67de5f700 1 -- 172.21.2.117:0/743613399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 msgr2=0x7fd6780b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.365 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 rx=0x7fd670009980 tx=0x7fd67000bcd0).stop 2021-02-10T11:47:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 -- 172.21.2.117:0/743613399 shutdown_connections 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd65c0186c0 0x7fd65c01abc0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd6780b30d0 0x7fd6780b34d0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd67818e5b0 0x7fd67818e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 --2- 172.21.2.117:0/743613399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd678099bf0 0x7fd6781ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 -- 172.21.2.117:0/743613399 >> 172.21.2.117:0/743613399 conn(0x7fd6781ad6b0 msgr2=0x7fd678190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 -- 172.21.2.117:0/743613399 shutdown_connections 2021-02-10T11:47:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.366 7fd67de5f700 1 -- 172.21.2.117:0/743613399 wait complete. 2021-02-10T11:47:42.369 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:42.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.784 7f8a22f1a700 1 Processor -- start 2021-02-10T11:47:42.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.784 7f8a22f1a700 1 -- start start 2021-02-10T11:47:42.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a22f1a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8a1c1b6960 0x7f8a1c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a22f1a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0ede60 con 0x7f8a1c0ef3f0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a22f1a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0eb9c0 con 0x7f8a1c0d18e0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a22f1a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1c0ef7f0 con 0x7f8a1c1b6960 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a1bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8a1c0ef3f0 0x7f8a1c0eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52814/0 (socket says 172.21.2.117:52814) 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.785 7f8a1bfff700 1 -- 172.21.2.117:0/926793949 learned_addr learned my addr 172.21.2.117:0/926793949 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a21537700 1 --2- 172.21.2.117:0/926793949 >> v2:172.21.2.125:3300/0 conn(0x7f8a1c1b6960 0x7f8a1c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213864980 0 0) 0x7f8a0c001430 con 0x7f8a1c0ef3f0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a08002fe0 con 0x7f8a1c0ef3f0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343919704 0 0) 0x7f8a04001080 con 0x7f8a1c0d18e0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a08003210 con 0x7f8a1c0d18e0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3543344660 0 0) 0x7f8a0c001430 con 0x7f8a1c0ef3f0 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a21537700 1 -- 172.21.2.117:0/926793949 >> v1:172.21.2.125:6789/0 conn(0x7f8a1c0d18e0 legacy=0x7f8a1c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a21537700 1 -- 172.21.2.117:0/926793949 >> v1:172.21.2.117:6789/0 conn(0x7f8a1c0ef3f0 legacy=0x7f8a1c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a21537700 1 -- 172.21.2.117:0/926793949 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0d1d20 con 0x7f8a1c1b6960 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.786 7f8a21537700 1 --2- 172.21.2.117:0/926793949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c1b6960 0x7f8a1c1b8d70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f8a14006ad0 tx=0x7f8a14009ad0).ready entity=mon.1 client_cookie=580f857565c91564 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a140022e0 con 0x7f8a1c1b6960 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14006410 con 0x7f8a1c1b6960 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a1b7fe700 1 -- 172.21.2.117:0/926793949 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a14002620 con 0x7f8a1c1b6960 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 -- 172.21.2.117:0/926793949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c1b6960 msgr2=0x7f8a1c1b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 --2- 172.21.2.117:0/926793949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c1b6960 0x7f8a1c1b8d70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f8a14006ad0 tx=0x7f8a14009ad0).stop 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 -- 172.21.2.117:0/926793949 shutdown_connections 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 --2- 172.21.2.117:0/926793949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c1b6960 0x7f8a1c1b8d70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 -- 172.21.2.117:0/926793949 >> 172.21.2.117:0/926793949 conn(0x7f8a1c0ced60 msgr2=0x7f8a1c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 -- 172.21.2.117:0/926793949 shutdown_connections 2021-02-10T11:47:42.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.787 7f8a22f1a700 1 -- 172.21.2.117:0/926793949 wait complete. 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 Processor -- start 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 -- start start 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8a1c0979b0 0x7f8a1c08ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c0ae480 0x7f8a1c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c08b270 con 0x7f8a1c08a280 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c08aca0 con 0x7f8a1c0ae480 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.802 7f8a22f1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c08ae80 con 0x7f8a1c0979b0 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52568/0 (socket says 172.21.2.117:52568) 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 -- 172.21.2.117:0/1951891409 learned_addr learned my addr 172.21.2.117:0/1951891409 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 -- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8a1c0979b0 msgr2=0x7f8a1c08ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8a1c0979b0 0x7f8a1c08ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 -- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c0ae480 msgr2=0x7f8a1c0ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c0ae480 0x7f8a1c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 -- 172.21.2.117:0/1951891409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c08b7f0 con 0x7f8a1c08a280 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a21537700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 rx=0x7f8a1400b610 tx=0x7f8a1400b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.803 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a14000da0 con 0x7f8a1c08a280 2021-02-10T11:47:42.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14000b00 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8a14002220 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c08b7f0 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c08b7f0 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8a14002020 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a197fa700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.804 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8a140326d0 con 0x7f8a1c08a280 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.805 7f8a20d36700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:42.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.806 7f8a20d36700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 0x7f8a0001abc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:42.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.835 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c087290 con 0x7f8a1c08a280 2021-02-10T11:47:42.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.837 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8a14000da0 con 0x7f8a1c08a280 2021-02-10T11:47:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.961 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a1c087290 con 0x7f8a1c08a280 2021-02-10T11:47:42.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.962 7f8a197fa700 1 -- 172.21.2.117:0/1951891409 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+14242 (secure 0 0 0) 0x7f8a14030d30 con 0x7f8a1c08a280 2021-02-10T11:47:42.957 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:42.957 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":90,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:42.357410","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:40.329394","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:41.402719","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.963 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 msgr2=0x7f8a0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.963 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 0x7f8a0001abc0 crc :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.963 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 msgr2=0x7f8a1c08a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:42.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.963 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 rx=0x7f8a1400b610 tx=0x7f8a1400b9b0).stop 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 shutdown_connections 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a000186c0 0x7f8a0001abc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8a1c08a280 0x7f8a1c08a680 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8a1c0ae480 0x7f8a1c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 --2- 172.21.2.117:0/1951891409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8a1c0979b0 0x7f8a1c08ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 >> 172.21.2.117:0/1951891409 conn(0x7f8a1c1a3620 msgr2=0x7f8a1c1a93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 shutdown_connections 2021-02-10T11:47:42.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:42.964 7f8a22f1a700 1 -- 172.21.2.117:0/1951891409 wait complete. 2021-02-10T11:47:42.967 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:43.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 Processor -- start 2021-02-10T11:47:43.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 -- start start 2021-02-10T11:47:43.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f140c0d18e0 0x7f140c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f140c0ef3f0 0x7f140c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140c0ede60 con 0x7f140c0eb000 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140c0ef7f0 con 0x7f140c0ef3f0 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.382 7f1411231700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140c0d1ce0 con 0x7f140c0d18e0 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f140c0ef3f0 0x7f140c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140a59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f140c0eb000 0x7f140c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52826/0 (socket says 172.21.2.117:52826) 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140a59c700 1 -- 172.21.2.117:0/1547007593 learned_addr learned my addr 172.21.2.117:0/1547007593 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 -- 172.21.2.117:0/1547007593 >> v2:172.21.2.125:3300/0 conn(0x7f140c0d18e0 msgr2=0x7f140c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 --2- 172.21.2.117:0/1547007593 >> v2:172.21.2.125:3300/0 conn(0x7f140c0d18e0 0x7f140c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 -- 172.21.2.117:0/1547007593 >> v1:172.21.2.117:6789/0 conn(0x7f140c0eb000 legacy=0x7f140c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 -- 172.21.2.117:0/1547007593 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c1b8f90 con 0x7f140c0ef3f0 2021-02-10T11:47:43.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.383 7f140b59e700 1 --2- 172.21.2.117:0/1547007593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0ef3f0 0x7f140c1b7960 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f1404006e30 tx=0x7f1404009e60).ready entity=mon.0 client_cookie=5eaf7bd2ab182bb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1409d9b700 1 -- 172.21.2.117:0/1547007593 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1404006a50 con 0x7f140c0ef3f0 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1409d9b700 1 -- 172.21.2.117:0/1547007593 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1404005820 con 0x7f140c0ef3f0 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1409d9b700 1 -- 172.21.2.117:0/1547007593 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1404005cc0 con 0x7f140c0ef3f0 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 -- 172.21.2.117:0/1547007593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0ef3f0 msgr2=0x7f140c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 --2- 172.21.2.117:0/1547007593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0ef3f0 0x7f140c1b7960 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 rx=0x7f1404006e30 tx=0x7f1404009e60).stop 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 -- 172.21.2.117:0/1547007593 shutdown_connections 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 --2- 172.21.2.117:0/1547007593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0ef3f0 0x7f140c1b7960 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 --2- 172.21.2.117:0/1547007593 >> v2:172.21.2.125:3300/0 conn(0x7f140c0d18e0 0x7f140c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 -- 172.21.2.117:0/1547007593 >> 172.21.2.117:0/1547007593 conn(0x7f140c0ced60 msgr2=0x7f140c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 -- 172.21.2.117:0/1547007593 shutdown_connections 2021-02-10T11:47:43.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.384 7f1411231700 1 -- 172.21.2.117:0/1547007593 wait complete. 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 Processor -- start 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 -- start start 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f140c196390 0x7f140c0e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f140c0e9960 0x7f140c0e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c196810 con 0x7f140c0e9560 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c098ad0 con 0x7f140c196390 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.399 7f1411231700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140c0e8a40 con 0x7f140c0e9960 2021-02-10T11:47:43.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52584/0 (socket says 172.21.2.117:52584) 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 -- 172.21.2.117:0/506198813 learned_addr learned my addr 172.21.2.117:0/506198813 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 -- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f140c0e9960 msgr2=0x7f140c0e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f140c0e9960 0x7f140c0e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 -- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f140c196390 msgr2=0x7f140c0e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140ad9d700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f140c196390 0x7f140c0e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f140c196390 0x7f140c0e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 -- 172.21.2.117:0/506198813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c0e8c60 con 0x7f140c0e9560 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f140a59c700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7f13fc00a570 tx=0x7f13fc00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13fc00ad00 con 0x7f140c0e9560 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc001180 con 0x7f140c0e9560 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13fc004620 con 0x7f140c0e9560 2021-02-10T11:47:43.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f1411231700 1 -- 172.21.2.117:0/506198813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c0e8c60 con 0x7f140c0e9560 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.400 7f1411231700 1 -- 172.21.2.117:0/506198813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c0e8c60 con 0x7f140c0e9560 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.401 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13fc0043a0 con 0x7f140c0e9560 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.401 7f14037fe700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 0x7f13f001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.401 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f13fc032750 con 0x7f140c0e9560 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.401 7f140ad9d700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 0x7f13f001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.402 7f140ad9d700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 0x7f13f001abd0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.432 7f1411231700 1 -- 172.21.2.117:0/506198813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c087210 con 0x7f140c0e9560 2021-02-10T11:47:43.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.434 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13fc000c90 con 0x7f140c0e9560 2021-02-10T11:47:43.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.559 7f1411231700 1 -- 172.21.2.117:0/506198813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f140c0ede60 con 0x7f140c0e9560 2021-02-10T11:47:43.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.560 7f14037fe700 1 -- 172.21.2.117:0/506198813 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f13fc001160 con 0x7f140c0e9560 2021-02-10T11:47:43.555 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:43.555 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":91,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:43.359349","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:40.329394","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:41.402719","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:43.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 msgr2=0x7f13f001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 0x7f13f001abd0 crc :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 msgr2=0x7f140c0e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 rx=0x7f13fc00a570 tx=0x7f13fc00a910).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 shutdown_connections 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13f00186d0 0x7f13f001abd0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f140c0e9560 0x7f140c0e7e00 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f140c0e9960 0x7f140c0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 --2- 172.21.2.117:0/506198813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f140c196390 0x7f140c0e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 >> 172.21.2.117:0/506198813 conn(0x7f140c19f610 msgr2=0x7f140c1a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 shutdown_connections 2021-02-10T11:47:43.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.562 7f1411231700 1 -- 172.21.2.117:0/506198813 wait complete. 2021-02-10T11:47:43.565 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 Processor -- start 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 -- start start 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0d18f0 0x7f76cc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f76cc0eb040 0x7f76cc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f76cc0ee3e0 0x7f76cc0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0ede60 con 0x7f76cc0eb040 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0eba60 con 0x7f76cc0ee3e0 2021-02-10T11:47:43.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.986 7f76d0d5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0d1cf0 con 0x7f76cc0d18f0 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f76cc0eb040 0x7f76cc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f76cc0eb040 0x7f76cc0eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52592/0 (socket says 172.21.2.117:52592) 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 -- 172.21.2.117:0/4284252473 learned_addr learned my addr 172.21.2.117:0/4284252473 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 -- 172.21.2.117:0/4284252473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0d18f0 msgr2=0x7f76cc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 --2- 172.21.2.117:0/4284252473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0d18f0 0x7f76cc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 -- 172.21.2.117:0/4284252473 >> v2:172.21.2.125:3300/0 conn(0x7f76cc0ee3e0 msgr2=0x7f76cc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 --2- 172.21.2.117:0/4284252473 >> v2:172.21.2.125:3300/0 conn(0x7f76cc0ee3e0 0x7f76cc0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 -- 172.21.2.117:0/4284252473 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0eee00 con 0x7f76cc0eb040 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76ca312700 1 --2- 172.21.2.117:0/4284252473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc0eb040 0x7f76cc0eb440 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7f76c0006700 tx=0x7f76c0009b00).ready entity=mon.0 client_cookie=550c1cd74843faa1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76c9b11700 1 -- 172.21.2.117:0/4284252473 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76c000c0c0 con 0x7f76cc0eb040 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76c9b11700 1 -- 172.21.2.117:0/4284252473 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c000c400 con 0x7f76cc0eb040 2021-02-10T11:47:43.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76c9b11700 1 -- 172.21.2.117:0/4284252473 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76c000c920 con 0x7f76cc0eb040 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76d0d5b700 1 -- 172.21.2.117:0/4284252473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc0eb040 msgr2=0x7f76cc0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.987 7f76d0d5b700 1 --2- 172.21.2.117:0/4284252473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc0eb040 0x7f76cc0eb440 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 rx=0x7f76c0006700 tx=0x7f76c0009b00).stop 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 -- 172.21.2.117:0/4284252473 shutdown_connections 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 --2- 172.21.2.117:0/4284252473 >> v2:172.21.2.125:3300/0 conn(0x7f76cc0ee3e0 0x7f76cc0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 --2- 172.21.2.117:0/4284252473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc0eb040 0x7f76cc0eb440 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 --2- 172.21.2.117:0/4284252473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0d18f0 0x7f76cc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 -- 172.21.2.117:0/4284252473 >> 172.21.2.117:0/4284252473 conn(0x7f76cc0cf060 msgr2=0x7f76cc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 -- 172.21.2.117:0/4284252473 shutdown_connections 2021-02-10T11:47:43.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:43.988 7f76d0d5b700 1 -- 172.21.2.117:0/4284252473 wait complete. 2021-02-10T11:47:44.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 Processor -- start 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 -- start start 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76cc1a3930 0x7f76cc0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0a13a0 0x7f76cc0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc07cca0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc18fb70 con 0x7f76cc1a3930 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76d0d5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0dc6d0 con 0x7f76cc0a13a0 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52598/0 (socket says 172.21.2.117:52598) 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 -- 172.21.2.117:0/2760365088 learned_addr learned my addr 172.21.2.117:0/2760365088 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 -- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0a13a0 msgr2=0x7f76cc0a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0a13a0 0x7f76cc0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 -- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76cc1a3930 msgr2=0x7f76cc0a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76cc1a3930 0x7f76cc0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.003 7f76cab13700 1 -- 172.21.2.117:0/2760365088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc07d220 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76cab13700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f76bc00b530 tx=0x7f76bc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76bc00bcc0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc000c90 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f76bc0022a0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc07d220 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc07d7b0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f76bc005310 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.004 7f76bb7fe700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 0x7f76b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.005 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f76bc032550 con 0x7f76cc07c7c0 2021-02-10T11:47:44.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.005 7f76ca312700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 0x7f76b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.006 7f76ca312700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 0x7f76b001abc0 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.036 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc0daff0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.039 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f76bc000960 con 0x7f76cc07c7c0 2021-02-10T11:47:44.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.172 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f76cc0daff0 con 0x7f76cc07c7c0 2021-02-10T11:47:44.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.489 7f76bb7fe700 1 -- 172.21.2.117:0/2760365088 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v68) v1 ==== 118+0+0 (secure 0 0 0) 0x7f76bc005310 con 0x7f76cc07c7c0 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 msgr2=0x7f76b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 0x7f76b001abc0 crc :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 msgr2=0x7f76cc1a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f76bc00b530 tx=0x7f76bc00b8d0).stop 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 shutdown_connections 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f76cc0a13a0 0x7f76cc0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f76cc1a3930 0x7f76cc0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76b00186c0 0x7f76b001abc0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 --2- 172.21.2.117:0/2760365088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f76cc07c7c0 0x7f76cc1a3310 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 >> 172.21.2.117:0/2760365088 conn(0x7f76cc087090 msgr2=0x7f76cc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 shutdown_connections 2021-02-10T11:47:44.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.541 7f76d0d5b700 1 -- 172.21.2.117:0/2760365088 wait complete. 2021-02-10T11:47:44.544 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:44.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.959 7fde65bc5700 1 Processor -- start 2021-02-10T11:47:44.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 -- start start 2021-02-10T11:47:44.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fde600d18e0 0x7fde600ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fde600ef3f0 0x7fde601b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600ede60 con 0x7fde600eb000 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde600ef7f0 con 0x7fde600d18e0 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde65bc5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde600d1ce0 con 0x7fde600ef3f0 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde5effd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fde600eb000 0x7fde600eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52854/0 (socket says 172.21.2.117:52854) 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde5effd700 1 -- 172.21.2.117:0/186816849 learned_addr learned my addr 172.21.2.117:0/186816849 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde5f7fe700 1 --2- 172.21.2.117:0/186816849 >> v2:172.21.2.117:3300/0 conn(0x7fde600d18e0 0x7fde600ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.960 7fde5e7fc700 1 -- 172.21.2.117:0/186816849 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271150703 0 0) 0x7fde480010b0 con 0x7fde600eb000 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5e7fc700 1 -- 172.21.2.117:0/186816849 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c002fe0 con 0x7fde600eb000 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5f7fe700 1 -- 172.21.2.117:0/186816849 >> v2:172.21.2.125:3300/0 conn(0x7fde600ef3f0 msgr2=0x7fde601b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5f7fe700 1 --2- 172.21.2.117:0/186816849 >> v2:172.21.2.125:3300/0 conn(0x7fde600ef3f0 0x7fde601b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5f7fe700 1 -- 172.21.2.117:0/186816849 >> v1:172.21.2.117:6789/0 conn(0x7fde600eb000 legacy=0x7fde600eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5f7fe700 1 -- 172.21.2.117:0/186816849 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde601b8f90 con 0x7fde600d18e0 2021-02-10T11:47:44.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5f7fe700 1 --2- 172.21.2.117:0/186816849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600d18e0 0x7fde600ea9e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7fde50006650 tx=0x7fde50009a80).ready entity=mon.0 client_cookie=8441b6635b55b4d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5e7fc700 1 -- 172.21.2.117:0/186816849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5000c0c0 con 0x7fde600d18e0 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5e7fc700 1 -- 172.21.2.117:0/186816849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5000c340 con 0x7fde600d18e0 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde5e7fc700 1 -- 172.21.2.117:0/186816849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde5000c860 con 0x7fde600d18e0 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde65bc5700 1 -- 172.21.2.117:0/186816849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600d18e0 msgr2=0x7fde600ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.961 7fde65bc5700 1 --2- 172.21.2.117:0/186816849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600d18e0 0x7fde600ea9e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7fde50006650 tx=0x7fde50009a80).stop 2021-02-10T11:47:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 -- 172.21.2.117:0/186816849 shutdown_connections 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 --2- 172.21.2.117:0/186816849 >> v2:172.21.2.125:3300/0 conn(0x7fde600ef3f0 0x7fde601b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 --2- 172.21.2.117:0/186816849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600d18e0 0x7fde600ea9e0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 -- 172.21.2.117:0/186816849 >> 172.21.2.117:0/186816849 conn(0x7fde600ced60 msgr2=0x7fde600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 -- 172.21.2.117:0/186816849 shutdown_connections 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.962 7fde65bc5700 1 -- 172.21.2.117:0/186816849 wait complete. 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 Processor -- start 2021-02-10T11:47:44.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 -- start start 2021-02-10T11:47:44.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde600b7310 0x7fde600aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 0x7fde600abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600ab8c0 con 0x7fde600b77e0 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde600ac460 con 0x7fde600ab2c0 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde65bc5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde600ac680 con 0x7fde600b7310 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde5f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde5f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52608/0 (socket says 172.21.2.117:52608) 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.977 7fde5f7fe700 1 -- 172.21.2.117:0/3344140326 learned_addr learned my addr 172.21.2.117:0/3344140326 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5ffff700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 0x7fde600abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5effd700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde600b7310 0x7fde600aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 -- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde600b7310 msgr2=0x7fde600aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde600b7310 0x7fde600aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 -- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 msgr2=0x7fde600abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 0x7fde600abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 -- 172.21.2.117:0/3344140326 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde600ac8a0 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5ffff700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 0x7fde600abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde5f7fe700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7fde50009630 tx=0x7fde5000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde500093a0 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5000bcf0 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fde50005560 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde600ac8a0 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.978 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde600ac8a0 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.979 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fde50005700 con 0x7fde600b77e0 2021-02-10T11:47:44.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.979 7fde57fff700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:44.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.979 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fde50032660 con 0x7fde600b77e0 2021-02-10T11:47:44.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.979 7fde5effd700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:44.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:44.980 7fde5effd700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:45.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.009 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60087290 con 0x7fde600b77e0 2021-02-10T11:47:45.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.012 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fde500054f0 con 0x7fde600b77e0 2021-02-10T11:47:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.139 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde600ede60 con 0x7fde600b77e0 2021-02-10T11:47:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.140 7fde57fff700 1 -- 172.21.2.117:0/3344140326 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fde50005700 con 0x7fde600b77e0 2021-02-10T11:47:45.136 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 68 2021-02-10T11:47:45.136 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:45.136 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":68,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:47:44.486277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6539":{"gid":6539,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/4168849820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":4168849820},{"type":"v1","addr":"172.21.2.125:6835","nonce":4168849820}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:47:45.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.141 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 msgr2=0x7fde4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.141 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 0x7fde4401abc0 crc :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.141 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 msgr2=0x7fde600b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.141 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 rx=0x7fde50009630 tx=0x7fde5000b900).stop 2021-02-10T11:47:45.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 shutdown_connections 2021-02-10T11:47:45.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fde600ab2c0 0x7fde600abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fde600b7310 0x7fde600aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fde440186c0 0x7fde4401abc0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 --2- 172.21.2.117:0/3344140326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fde600b77e0 0x7fde600b6cf0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 >> 172.21.2.117:0/3344140326 conn(0x7fde600874c0 msgr2=0x7fde600af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:45.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 shutdown_connections 2021-02-10T11:47:45.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.142 7fde65bc5700 1 -- 172.21.2.117:0/3344140326 wait complete. 2021-02-10T11:47:45.146 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.562 7f6a90be0700 1 Processor -- start 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.562 7f6a90be0700 1 -- start start 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a90be0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6a8c0d18e0 0x7f6a8c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a90be0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8c0ede60 con 0x7f6a8c0eb000 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a90be0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8c0d1ce0 con 0x7f6a8c0ef3f0 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a90be0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8c0ef7f0 con 0x7f6a8c0d18e0 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a8a05d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6a8c0eb000 0x7f6a8c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52870/0 (socket says 172.21.2.117:52870) 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a8a05d700 1 -- 172.21.2.117:0/1979860954 learned_addr learned my addr 172.21.2.117:0/1979860954 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:45.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a8a85e700 1 --2- 172.21.2.117:0/1979860954 >> v2:172.21.2.125:3300/0 conn(0x7f6a8c0d18e0 0x7f6a8c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54856562 0 0) 0x7f6a740009b0 con 0x7f6a8c0eb000 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.563 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78002fe0 con 0x7f6a8c0eb000 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 213658897 0 0) 0x7f6a74001010 con 0x7f6a8c0eb000 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8a85e700 1 -- 172.21.2.117:0/1979860954 >> v1:172.21.2.125:6789/0 conn(0x7f6a8c0ef3f0 legacy=0x7f6a8c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8a85e700 1 -- 172.21.2.117:0/1979860954 >> v1:172.21.2.117:6789/0 conn(0x7f6a8c0eb000 legacy=0x7f6a8c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8a85e700 1 -- 172.21.2.117:0/1979860954 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c1bb180 con 0x7f6a8c0d18e0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8a85e700 1 --2- 172.21.2.117:0/1979860954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c0d18e0 0x7f6a8c0ea9e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f6a7c006650 tx=0x7f6a7c009a80).ready entity=mon.1 client_cookie=7e9066687a31a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a7c00c0c0 con 0x7f6a8c0d18e0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c00c340 con 0x7f6a8c0d18e0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a8985c700 1 -- 172.21.2.117:0/1979860954 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a7c00c860 con 0x7f6a8c0d18e0 2021-02-10T11:47:45.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a90be0700 1 -- 172.21.2.117:0/1979860954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c0d18e0 msgr2=0x7f6a8c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.564 7f6a90be0700 1 --2- 172.21.2.117:0/1979860954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c0d18e0 0x7f6a8c0ea9e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f6a7c006650 tx=0x7f6a7c009a80).stop 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.565 7f6a90be0700 1 -- 172.21.2.117:0/1979860954 shutdown_connections 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.565 7f6a90be0700 1 --2- 172.21.2.117:0/1979860954 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c0d18e0 0x7f6a8c0ea9e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.565 7f6a90be0700 1 -- 172.21.2.117:0/1979860954 >> 172.21.2.117:0/1979860954 conn(0x7f6a8c0ced60 msgr2=0x7f6a8c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.565 7f6a90be0700 1 -- 172.21.2.117:0/1979860954 shutdown_connections 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.565 7f6a90be0700 1 -- 172.21.2.117:0/1979860954 wait complete. 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 Processor -- start 2021-02-10T11:47:45.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 -- start start 2021-02-10T11:47:45.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c093d00 0x7f6a8c093180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a8c0947f0 0x7f6a8c091060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c091680 con 0x7f6a8c0937a0 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0918a0 con 0x7f6a8c093d00 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a90be0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c091ac0 con 0x7f6a8c0947f0 2021-02-10T11:47:45.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a8a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a8a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52622/0 (socket says 172.21.2.117:52622) 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.580 7f6a8a05d700 1 -- 172.21.2.117:0/3778064996 learned_addr learned my addr 172.21.2.117:0/3778064996 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 -- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a8c0947f0 msgr2=0x7f6a8c091060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a8c0947f0 0x7f6a8c091060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 -- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c093d00 msgr2=0x7f6a8c093180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c093d00 0x7f6a8c093180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 -- 172.21.2.117:0/3778064996 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c091ce0 con 0x7f6a8c0937a0 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a8a05d700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f6a74001c60 tx=0x7f6a7400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a7400acc0 con 0x7f6a8c0937a0 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a74004140 con 0x7f6a8c0937a0 2021-02-10T11:47:45.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a74004660 con 0x7f6a8c0937a0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c091ce0 con 0x7f6a8c0937a0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.581 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c091ce0 con 0x7f6a8c0937a0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.582 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a74004320 con 0x7f6a8c0937a0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.582 7f6a837fe700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 0x7f6a7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.582 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6a74032690 con 0x7f6a8c0937a0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.582 7f6a8a85e700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 0x7f6a7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:45.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.583 7f6a8a85e700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 0x7f6a7001abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.613 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c087210 con 0x7f6a8c0937a0 2021-02-10T11:47:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.615 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a74000e20 con 0x7f6a8c0937a0 2021-02-10T11:47:45.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.739 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a8c0ede60 con 0x7f6a8c0937a0 2021-02-10T11:47:45.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.739 7f6a837fe700 1 -- 172.21.2.117:0/3778064996 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f6a74004320 con 0x7f6a8c0937a0 2021-02-10T11:47:45.735 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:45.736 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":91,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:43.359349","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:40.329394","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:41.402719","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:45.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 msgr2=0x7f6a7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 0x7f6a7001abc0 crc :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 msgr2=0x7f6a8c0941d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:45.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 rx=0x7f6a74001c60 tx=0x7f6a7400a8d0).stop 2021-02-10T11:47:45.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 shutdown_connections 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a700186c0 0x7f6a7001abc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a8c0937a0 0x7f6a8c0941d0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a8c0947f0 0x7f6a8c091060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 --2- 172.21.2.117:0/3778064996 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a8c093d00 0x7f6a8c093180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.741 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 >> 172.21.2.117:0/3778064996 conn(0x7f6a8c19f610 msgr2=0x7f6a8c08e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.742 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 shutdown_connections 2021-02-10T11:47:45.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:45.742 7f6a90be0700 1 -- 172.21.2.117:0/3778064996 wait complete. 2021-02-10T11:47:45.746 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:46.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.173 7faa60432700 1 Processor -- start 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.173 7faa60432700 1 -- start start 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa60432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5807ab40 0x7faa580794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa60432700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faa58079f10 0x7faa5807b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa60432700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5807a5c0 con 0x7faa58079b10 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa60432700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa580e5770 con 0x7faa58079f10 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa60432700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5807a310 con 0x7faa5807ab40 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5da4d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7faa58079b10 0x7faa5807af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52880/0 (socket says 172.21.2.117:52880) 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5da4d700 1 -- 172.21.2.117:0/3968361617 learned_addr learned my addr 172.21.2.117:0/3968361617 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632443542 0 0) 0x7faa540009b0 con 0x7faa58079b10 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa44002fe0 con 0x7faa58079b10 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5ea4f700 1 --2- 172.21.2.117:0/3968361617 >> v2:172.21.2.125:3300/0 conn(0x7faa58079f10 0x7faa5807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5e24e700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5807ab40 0x7faa580794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3717339546 0 0) 0x7faa54001010 con 0x7faa58079b10 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.174 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5807ab40 msgr2=0x7faa580794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5807ab40 0x7faa580794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 >> v2:172.21.2.125:3300/0 conn(0x7faa58079f10 msgr2=0x7faa5807b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 --2- 172.21.2.117:0/3968361617 >> v2:172.21.2.125:3300/0 conn(0x7faa58079f10 0x7faa5807b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa580d18c0 con 0x7faa58079b10 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7faa54001590 con 0x7faa58079b10 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 >> v1:172.21.2.117:6789/0 conn(0x7faa58079b10 legacy=0x7faa5807af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa44003b90 0x7faa44006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa4404a870 0x7faa4404cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa4404d650 0x7faa4404fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa44050390 con 0x7faa44003b90 2021-02-10T11:47:46.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa440505b0 con 0x7faa4404a870 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440507d0 con 0x7faa4404d650 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5ea4f700 1 --2- 172.21.2.117:0/3968361617 >> v2:172.21.2.125:3300/0 conn(0x7faa58079f10 0x7faa5807b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa44003b90 0x7faa44006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5da4d700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa4404a870 0x7faa4404cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 -- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa4404d650 msgr2=0x7faa4404fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa4404d650 0x7faa4404fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 -- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa4404a870 msgr2=0x7faa4404cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa4404a870 0x7faa4404cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 -- 172.21.2.117:0/3968361617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44050a70 con 0x7faa44003b90 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5e24e700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa44003b90 0x7faa44006040 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7faa4c009310 tx=0x7faa4c009a10).ready entity=mon.0 client_cookie=9d0ec39bf4600d7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa4c0009b0 con 0x7faa44003b90 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c006400 con 0x7faa44003b90 2021-02-10T11:47:46.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5d24c700 1 -- 172.21.2.117:0/3968361617 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa4c00c2b0 con 0x7faa44003b90 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa60432700 1 -- 172.21.2.117:0/3968361617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa44003b90 msgr2=0x7faa44006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa60432700 1 --2- 172.21.2.117:0/3968361617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa44003b90 0x7faa44006040 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7faa4c009310 tx=0x7faa4c009a10).stop 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.175 7faa5ea4f700 1 -- 172.21.2.117:0/3968361617 reap_dead start 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.176 7faa60432700 1 -- 172.21.2.117:0/3968361617 shutdown_connections 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.176 7faa60432700 1 -- 172.21.2.117:0/3968361617 >> 172.21.2.117:0/3968361617 conn(0x7faa580d0910 msgr2=0x7faa580d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.176 7faa60432700 1 -- 172.21.2.117:0/3968361617 shutdown_connections 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.176 7faa60432700 1 -- 172.21.2.117:0/3968361617 wait complete. 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 Processor -- start 2021-02-10T11:47:46.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 -- start start 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa5808c2c0 0x7faa581a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5808c6c0 0x7faa581a3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa580fe9c0 con 0x7faa580dc6f0 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa581a3a70 con 0x7faa5808c2c0 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa60432700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa581a3c90 con 0x7faa5808c6c0 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa5e24e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa5e24e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52640/0 (socket says 172.21.2.117:52640) 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.191 7faa5e24e700 1 -- 172.21.2.117:0/3577828274 learned_addr learned my addr 172.21.2.117:0/3577828274 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 -- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5808c6c0 msgr2=0x7faa581a3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5808c6c0 0x7faa581a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 -- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa5808c2c0 msgr2=0x7faa581a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa5808c2c0 0x7faa581a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 -- 172.21.2.117:0/3577828274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa580ff1e0 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa5e24e700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7faa4c00b690 tx=0x7faa4c00ba30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa4c000960 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c00be20 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa60432700 1 -- 172.21.2.117:0/3577828274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa580ff1e0 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa4c002030 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.192 7faa60432700 1 -- 172.21.2.117:0/3577828274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa580ff770 con 0x7faa580dc6f0 2021-02-10T11:47:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.193 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa4c002020 con 0x7faa580dc6f0 2021-02-10T11:47:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.193 7faa4affd700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 0x7faa4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.193 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faa4c032550 con 0x7faa580dc6f0 2021-02-10T11:47:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.193 7faa5da4d700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 0x7faa4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.193 7faa5da4d700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 0x7faa4001abc0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.223 7faa60432700 1 -- 172.21.2.117:0/3577828274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5807a5c0 con 0x7faa580dc6f0 2021-02-10T11:47:46.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.226 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa4c0093e0 con 0x7faa580dc6f0 2021-02-10T11:47:46.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.353 7faa60432700 1 -- 172.21.2.117:0/3577828274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7faa5807a5c0 con 0x7faa580dc6f0 2021-02-10T11:47:46.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.358 7faa4affd700 1 -- 172.21.2.117:0/3577828274 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v51) v1 ==== 184+0+0 (secure 0 0 0) 0x7faa4c001ee0 con 0x7faa580dc6f0 2021-02-10T11:47:46.356 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:47:46.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.362 7faa60432700 1 -- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 msgr2=0x7faa4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.362 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 0x7faa4001abc0 crc :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.362 7faa60432700 1 -- 172.21.2.117:0/3577828274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 msgr2=0x7faa58093cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.362 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7faa4c00b690 tx=0x7faa4c00ba30).stop 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 -- 172.21.2.117:0/3577828274 shutdown_connections 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa5808c6c0 0x7faa581a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa5808c2c0 0x7faa581a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa400186c0 0x7faa4001abc0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 --2- 172.21.2.117:0/3577828274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa580dc6f0 0x7faa58093cc0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 -- 172.21.2.117:0/3577828274 >> 172.21.2.117:0/3577828274 conn(0x7faa580dccc0 msgr2=0x7faa580dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 -- 172.21.2.117:0/3577828274 shutdown_connections 2021-02-10T11:47:46.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.363 7faa60432700 1 -- 172.21.2.117:0/3577828274 wait complete. 2021-02-10T11:47:46.367 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 Processor -- start 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 -- start start 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbd88079ce0 0x7fbd881b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8807a5c0 con 0x7fbd8807a0e0 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd880d18c0 con 0x7fbd8807ab40 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.793 7fbd8e24f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8807bf50 con 0x7fbd88079ce0 2021-02-10T11:47:46.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbd88079ce0 0x7fbd881b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbd88079ce0 0x7fbd881b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52646/0 (socket says 172.21.2.117:52646) 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 -- 172.21.2.117:0/1849888803 learned_addr learned my addr 172.21.2.117:0/1849888803 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 -- 172.21.2.117:0/1849888803 >> v1:172.21.2.125:6789/0 conn(0x7fbd8807ab40 legacy=0x7fbd880794f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 -- 172.21.2.117:0/1849888803 >> v1:172.21.2.117:6789/0 conn(0x7fbd8807a0e0 legacy=0x7fbd881b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 -- 172.21.2.117:0/1849888803 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880d1c60 con 0x7fbd88079ce0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd877fe700 1 --2- 172.21.2.117:0/1849888803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd88079ce0 0x7fbd881b59d0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7fbd7c006700 tx=0x7fbd7c009b80).ready entity=mon.0 client_cookie=25c77def011c5e23 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd86ffd700 1 -- 172.21.2.117:0/1849888803 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7c00c0c0 con 0x7fbd88079ce0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd86ffd700 1 -- 172.21.2.117:0/1849888803 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c00c340 con 0x7fbd88079ce0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd86ffd700 1 -- 172.21.2.117:0/1849888803 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7c00c7a0 con 0x7fbd88079ce0 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd8e24f700 1 -- 172.21.2.117:0/1849888803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd88079ce0 msgr2=0x7fbd881b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.794 7fbd8e24f700 1 --2- 172.21.2.117:0/1849888803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd88079ce0 0x7fbd881b59d0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 rx=0x7fbd7c006700 tx=0x7fbd7c009b80).stop 2021-02-10T11:47:46.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.795 7fbd8e24f700 1 -- 172.21.2.117:0/1849888803 shutdown_connections 2021-02-10T11:47:46.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.795 7fbd8e24f700 1 --2- 172.21.2.117:0/1849888803 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd88079ce0 0x7fbd881b59d0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.795 7fbd8e24f700 1 -- 172.21.2.117:0/1849888803 >> 172.21.2.117:0/1849888803 conn(0x7fbd880d0910 msgr2=0x7fbd880d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.795 7fbd8e24f700 1 -- 172.21.2.117:0/1849888803 shutdown_connections 2021-02-10T11:47:46.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.795 7fbd8e24f700 1 -- 172.21.2.117:0/1849888803 wait complete. 2021-02-10T11:47:46.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.809 7fbd8e24f700 1 Processor -- start 2021-02-10T11:47:46.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 -- start start 2021-02-10T11:47:46.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd881c68f0 0x7fbd88099710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd88099d30 0x7fbd881000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8809a130 con 0x7fbd880b0bb0 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd881006f0 con 0x7fbd88099d30 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8e24f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88100910 con 0x7fbd881c68f0 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd87fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd87fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52656/0 (socket says 172.21.2.117:52656) 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd87fff700 1 -- 172.21.2.117:0/3086631593 learned_addr learned my addr 172.21.2.117:0/3086631593 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd877fe700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd881c68f0 0x7fbd88099710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.810 7fbd8c86c700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd88099d30 0x7fbd881000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 -- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd881c68f0 msgr2=0x7fbd88099710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd881c68f0 0x7fbd88099710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 -- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd88099d30 msgr2=0x7fbd881000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd88099d30 0x7fbd881000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 -- 172.21.2.117:0/3086631593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88100b30 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd87fff700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7fbd7800b5b0 tx=0x7fbd7800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7800bd40 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd78006240 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd78005720 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88100b30 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88100b30 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.811 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd780054a0 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.812 7fbd84ff9700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 0x7fbd6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.812 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbd780327c0 con 0x7fbd880b0bb0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.812 7fbd877fe700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 0x7fbd6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:46.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.812 7fbd877fe700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 0x7fbd6801abc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:46.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.843 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8807d4f0 con 0x7fbd880b0bb0 2021-02-10T11:47:46.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.846 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd78006030 con 0x7fbd880b0bb0 2021-02-10T11:47:46.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.976 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbd880d0910 con 0x7fbd880b0bb0 2021-02-10T11:47:46.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.980 7fbd84ff9700 1 -- 172.21.2.117:0/3086631593 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v52) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbd780054a0 con 0x7fbd880b0bb0 2021-02-10T11:47:46.978 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:47:46.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 msgr2=0x7fbd6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 0x7fbd6801abc0 crc :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 msgr2=0x7fbd881c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:46.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 rx=0x7fbd7800b5b0 tx=0x7fbd7800b950).stop 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 shutdown_connections 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd88099d30 0x7fbd881000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd881c68f0 0x7fbd88099710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd68018730 0x7fbd6801abc0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 --2- 172.21.2.117:0/3086631593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd880b0bb0 0x7fbd881c62d0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:46.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 >> 172.21.2.117:0/3086631593 conn(0x7fbd8809e020 msgr2=0x7fbd880a7140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:46.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 shutdown_connections 2021-02-10T11:47:46.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:46.985 7fbd8e24f700 1 -- 172.21.2.117:0/3086631593 wait complete. 2021-02-10T11:47:46.989 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:47.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.407 7ff8b41c1700 1 Processor -- start 2021-02-10T11:47:47.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.407 7ff8b41c1700 1 -- start start 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b41c1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8ac0d1900 0x7ff8ac0eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b41c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac0ef400 0x7ff8ac1b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b41c1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac0ef800 con 0x7ff8ac0eb050 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b41c1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac0edee0 con 0x7ff8ac0d1900 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b41c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0d1d00 con 0x7ff8ac0ef400 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b17dc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8ac0eb050 0x7ff8ac0eb450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52910/0 (socket says 172.21.2.117:52910) 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b17dc700 1 -- 172.21.2.117:0/63399854 learned_addr learned my addr 172.21.2.117:0/63399854 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109610813 0 0) 0x7ff8a80010b0 con 0x7ff8ac0eb050 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff898002fe0 con 0x7ff8ac0eb050 2021-02-10T11:47:47.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b1fdd700 1 --2- 172.21.2.117:0/63399854 >> v2:172.21.2.125:3300/0 conn(0x7ff8ac0d1900 0x7ff8ac0eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.408 7ff8b27de700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac0ef400 0x7ff8ac1b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1642265853 0 0) 0x7ff8a80010b0 con 0x7ff8ac0eb050 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac0ef400 msgr2=0x7ff8ac1b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac0ef400 0x7ff8ac1b79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 >> v2:172.21.2.125:3300/0 conn(0x7ff8ac0d1900 msgr2=0x7ff8ac0eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 --2- 172.21.2.117:0/63399854 >> v2:172.21.2.125:3300/0 conn(0x7ff8ac0d1900 0x7ff8ac0eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac1b8fe0 con 0x7ff8ac0eb050 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff8a80015b0 con 0x7ff8ac0eb050 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 >> v1:172.21.2.117:6789/0 conn(0x7ff8ac0eb050 legacy=0x7ff8ac0eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff898003b10 0x7ff898005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89804a780 0x7ff89804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89804d500 0x7ff89804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980502d0 con 0x7ff89804d500 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8980504f0 con 0x7ff89804a780 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff898050710 con 0x7ff898003b10 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b27de700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89804d500 0x7ff89804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b1fdd700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff898003b10 0x7ff898005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.409 7ff8b17dc700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89804a780 0x7ff89804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 -- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff898003b10 msgr2=0x7ff898005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff898003b10 0x7ff898005f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 -- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89804a780 msgr2=0x7ff89804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff89804a780 0x7ff89804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 -- 172.21.2.117:0/63399854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980509b0 con 0x7ff89804d500 2021-02-10T11:47:47.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89804d500 0x7ff89804fa20 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7ff8a4009b10 tx=0x7ff8a4009eb0).ready entity=mon.0 client_cookie=4e643bf7f3aa94b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a400b510 con 0x7ff89804d500 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a4003030 con 0x7ff89804d500 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b0fdb700 1 -- 172.21.2.117:0/63399854 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a400b790 con 0x7ff89804d500 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b41c1700 1 -- 172.21.2.117:0/63399854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89804d500 msgr2=0x7ff89804fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b41c1700 1 --2- 172.21.2.117:0/63399854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff89804d500 0x7ff89804fa20 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7ff8a4009b10 tx=0x7ff8a4009eb0).stop 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.410 7ff8b27de700 1 -- 172.21.2.117:0/63399854 reap_dead start 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.411 7ff8b41c1700 1 -- 172.21.2.117:0/63399854 shutdown_connections 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.411 7ff8b41c1700 1 -- 172.21.2.117:0/63399854 >> 172.21.2.117:0/63399854 conn(0x7ff8ac0cf070 msgr2=0x7ff8ac0cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.411 7ff8b41c1700 1 -- 172.21.2.117:0/63399854 shutdown_connections 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.411 7ff8b41c1700 1 -- 172.21.2.117:0/63399854 wait complete. 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 Processor -- start 2021-02-10T11:47:47.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 -- start start 2021-02-10T11:47:47.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 0x7ff8ac0a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac1a2e50 0x7ff8ac1a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0a35c0 con 0x7ff8ac08c800 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a3870 con 0x7ff8ac08c400 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b41c1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1a3a90 con 0x7ff8ac1a2e50 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b17dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b17dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52672/0 (socket says 172.21.2.117:52672) 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.426 7ff8b17dc700 1 -- 172.21.2.117:0/758454177 learned_addr learned my addr 172.21.2.117:0/758454177 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b27de700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac1a2e50 0x7ff8ac1a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 -- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac1a2e50 msgr2=0x7ff8ac1a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac1a2e50 0x7ff8ac1a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 -- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 msgr2=0x7ff8ac0a2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b1fdd700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 0x7ff8ac0a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 0x7ff8ac0a2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 -- 172.21.2.117:0/758454177 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac0b25f0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b1fdd700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 0x7ff8ac0a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b17dc700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7ff8a800a4d0 tx=0x7ff8a800a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a80009b0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a800ac60 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8a80045d0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.427 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac0b25f0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.428 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac0b25f0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.428 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8a8004350 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.428 7ff89e7fc700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.428 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8a80326b0 con 0x7ff8ac08c800 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.429 7ff8b1fdd700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:47.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.429 7ff8b1fdd700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 0x7ff89401abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:47.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.458 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac0ede70 con 0x7ff8ac08c800 2021-02-10T11:47:47.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.460 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8a8004140 con 0x7ff8ac08c800 2021-02-10T11:47:47.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.591 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff8ac0ede70 con 0x7ff8ac08c800 2021-02-10T11:47:47.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.594 7ff89e7fc700 1 -- 172.21.2.117:0/758454177 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v53) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff8a8004350 con 0x7ff8ac08c800 2021-02-10T11:47:47.593 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:47:47.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 msgr2=0x7ff89401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 0x7ff89401abc0 crc :-1 s=READY pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 msgr2=0x7ff8ac0a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:47.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 rx=0x7ff8a800a4d0 tx=0x7ff8a800a870).stop 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 shutdown_connections 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8940186c0 0x7ff89401abc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8ac08c800 0x7ff8ac0a2f20 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8ac1a2e50 0x7ff8ac1a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 --2- 172.21.2.117:0/758454177 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8ac08c400 0x7ff8ac0a2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.599 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 >> 172.21.2.117:0/758454177 conn(0x7ff8ac0bb610 msgr2=0x7ff8ac083ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.600 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 shutdown_connections 2021-02-10T11:47:47.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:47.600 7ff8b41c1700 1 -- 172.21.2.117:0/758454177 wait complete. 2021-02-10T11:47:47.604 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 Processor -- start 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 -- start start 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d0079ce0 0x7f21d01b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f21d007a0e0 0x7f21d01b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d00e5770 con 0x7f21d007ab40 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d00d18c0 con 0x7f21d007a0e0 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21d6482700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d007bf50 con 0x7f21d0079ce0 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21cffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f21d007ab40 0x7f21d00794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52928/0 (socket says 172.21.2.117:52928) 2021-02-10T11:47:48.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.023 7f21cffff700 1 -- 172.21.2.117:0/1265422850 learned_addr learned my addr 172.21.2.117:0/1265422850 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.024 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662002950 0 0) 0x7f21c0001180 con 0x7f21d007ab40 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.024 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b8002fe0 con 0x7f21d007ab40 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.024 7f21d4a9f700 1 --2- 172.21.2.117:0/1265422850 >> v2:172.21.2.125:3300/0 conn(0x7f21d007a0e0 0x7f21d01b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.024 7f21cf7fe700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d0079ce0 0x7f21d01b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.024 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1277870847 0 0) 0x7f21c0001180 con 0x7f21d007ab40 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d0079ce0 msgr2=0x7f21d01b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d0079ce0 0x7f21d01b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 >> v2:172.21.2.125:3300/0 conn(0x7f21d007a0e0 msgr2=0x7f21d01b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 --2- 172.21.2.117:0/1265422850 >> v2:172.21.2.125:3300/0 conn(0x7f21d007a0e0 0x7f21d01b8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00d1c60 con 0x7f21d007ab40 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f21c0001680 con 0x7f21d007ab40 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 >> v1:172.21.2.117:6789/0 conn(0x7f21d007ab40 legacy=0x7f21d00794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21b8003af0 0x7f21b8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21b804a790 0x7f21b804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21b804d510 0x7f21b804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80502e0 con 0x7f21b804d510 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8050500 con 0x7f21b804a790 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8050720 con 0x7f21b8003af0 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21cffff700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21b8003af0 0x7f21b8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21b804d510 0x7f21b804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21cf7fe700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21b804a790 0x7f21b804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 -- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21b8003af0 msgr2=0x7f21b8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21b8003af0 0x7f21b8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 -- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21b804a790 msgr2=0x7f21b804cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21b804a790 0x7f21b804cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 -- 172.21.2.117:0/1265422850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80509c0 con 0x7f21b804d510 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.025 7f21d4a9f700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21b804d510 0x7f21b804fa30 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f21c8006a40 tx=0x7f21c800c0c0).ready entity=mon.0 client_cookie=98628b65f9d3e556 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c800c4b0 con 0x7f21b804d510 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c800c6f0 con 0x7f21b804d510 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21ceffd700 1 -- 172.21.2.117:0/1265422850 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c800cc10 con 0x7f21b804d510 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 -- 172.21.2.117:0/1265422850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21b804d510 msgr2=0x7f21b804fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 --2- 172.21.2.117:0/1265422850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21b804d510 0x7f21b804fa30 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f21c8006a40 tx=0x7f21c800c0c0).stop 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d4a9f700 1 -- 172.21.2.117:0/1265422850 reap_dead start 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 -- 172.21.2.117:0/1265422850 shutdown_connections 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 -- 172.21.2.117:0/1265422850 >> 172.21.2.117:0/1265422850 conn(0x7f21d00d0910 msgr2=0x7f21d00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 -- 172.21.2.117:0/1265422850 shutdown_connections 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.026 7f21d6482700 1 -- 172.21.2.117:0/1265422850 wait complete. 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 Processor -- start 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 -- start start 2021-02-10T11:47:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d008c2c0 0x7f21d01a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d008c6c0 0x7f21d01a3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d00fe9c0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a3a70 con 0x7f21d008c6c0 2021-02-10T11:47:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.041 7f21d6482700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a3c90 con 0x7f21d008c2c0 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52690/0 (socket says 172.21.2.117:52690) 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 -- 172.21.2.117:0/4194309101 learned_addr learned my addr 172.21.2.117:0/4194309101 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 -- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d008c2c0 msgr2=0x7f21d01a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d008c2c0 0x7f21d01a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:48.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 -- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d008c6c0 msgr2=0x7f21d01a3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d008c6c0 0x7f21d01a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 -- 172.21.2.117:0/4194309101 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00ff1e0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21cffff700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f21c0009610 tx=0x7f21c00099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c00070c0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c00043a0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21d6482700 1 -- 172.21.2.117:0/4194309101 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d00ff1e0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21d6482700 1 -- 172.21.2.117:0/4194309101 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d00ff770 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.042 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21c00047f0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.043 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21c0007a70 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.043 7f21ccff9700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.043 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f21c0032700 con 0x7f21d00dc6f0 2021-02-10T11:47:48.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.043 7f21cf7fe700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.044 7f21cf7fe700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.073 7f21d6482700 1 -- 172.21.2.117:0/4194309101 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d007a5c0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.076 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21c002e540 con 0x7f21d00dc6f0 2021-02-10T11:47:48.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.204 7f21d6482700 1 -- 172.21.2.117:0/4194309101 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f21d007a5c0 con 0x7f21d00dc6f0 2021-02-10T11:47:48.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.208 7f21ccff9700 1 -- 172.21.2.117:0/4194309101 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v54) v1 ==== 184+0+0 (secure 0 0 0) 0x7f21c002e540 con 0x7f21d00dc6f0 2021-02-10T11:47:48.207 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.212 7f21d6482700 1 -- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 msgr2=0x7f21b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.212 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 crc :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.212 7f21d6482700 1 -- 172.21.2.117:0/4194309101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 msgr2=0x7f21d0093cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.212 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 rx=0x7f21c0009610 tx=0x7f21c00099b0).stop 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 -- 172.21.2.117:0/4194309101 shutdown_connections 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21d008c6c0 0x7f21d01a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:48.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21d008c2c0 0x7f21d01a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:48.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21b00186c0 0x7f21b001abc0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 --2- 172.21.2.117:0/4194309101 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21d00dc6f0 0x7f21d0093cc0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 -- 172.21.2.117:0/4194309101 >> 172.21.2.117:0/4194309101 conn(0x7f21d00dccc0 msgr2=0x7f21d00dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 -- 172.21.2.117:0/4194309101 shutdown_connections 2021-02-10T11:47:48.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.213 7f21d6482700 1 -- 172.21.2.117:0/4194309101 wait complete. 2021-02-10T11:47:48.217 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:48.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 Processor -- start 2021-02-10T11:47:48.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 -- start start 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0d340d18e0 0x7f0d340ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340ede60 con 0x7f0d340ef3f0 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340d1ce0 con 0x7f0d340eb000 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.632 7f0d38901700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d340ef7f0 con 0x7f0d340d18e0 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0d340d18e0 0x7f0d340ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d32e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0d340ef3f0 0x7f0d341b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52948/0 (socket says 172.21.2.117:52948) 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d32e1d700 1 -- 172.21.2.117:0/909962534 learned_addr learned my addr 172.21.2.117:0/909962534 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:48.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3261c700 1 -- 172.21.2.117:0/909962534 >> v1:172.21.2.125:6789/0 conn(0x7f0d340eb000 legacy=0x7f0d340eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3261c700 1 -- 172.21.2.117:0/909962534 >> v1:172.21.2.117:6789/0 conn(0x7f0d340ef3f0 legacy=0x7f0d341b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3261c700 1 -- 172.21.2.117:0/909962534 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d341bb180 con 0x7f0d340d18e0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3161a700 1 -- 172.21.2.117:0/909962534 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299632678 0 0) 0x7f0d2c001180 con 0x7f0d340ef3f0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3261c700 1 --2- 172.21.2.117:0/909962534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340d18e0 0x7f0d340ea9e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7f0d1c0069d0 tx=0x7f0d1c009e40).ready entity=mon.0 client_cookie=1ff9fe452b61730b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.633 7f0d3161a700 1 -- 172.21.2.117:0/909962534 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c00b510 con 0x7f0d340d18e0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d3161a700 1 -- 172.21.2.117:0/909962534 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d1c00b790 con 0x7f0d340d18e0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d3161a700 1 -- 172.21.2.117:0/909962534 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d1c00bcb0 con 0x7f0d340d18e0 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 -- 172.21.2.117:0/909962534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340d18e0 msgr2=0x7f0d340ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 --2- 172.21.2.117:0/909962534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340d18e0 0x7f0d340ea9e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 rx=0x7f0d1c0069d0 tx=0x7f0d1c009e40).stop 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 -- 172.21.2.117:0/909962534 shutdown_connections 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 --2- 172.21.2.117:0/909962534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340d18e0 0x7f0d340ea9e0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 -- 172.21.2.117:0/909962534 >> 172.21.2.117:0/909962534 conn(0x7f0d340ced60 msgr2=0x7f0d340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 -- 172.21.2.117:0/909962534 shutdown_connections 2021-02-10T11:47:48.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.634 7f0d38901700 1 -- 172.21.2.117:0/909962534 wait complete. 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 Processor -- start 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 -- start start 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0d34196390 0x7f0d340e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 0x7f0d340e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34196810 con 0x7f0d340e9960 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34098ad0 con 0x7f0d34196390 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.649 7f0d38901700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340e8a40 con 0x7f0d340e9560 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52706/0 (socket says 172.21.2.117:52706) 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 -- 172.21.2.117:0/4150814367 learned_addr learned my addr 172.21.2.117:0/4150814367 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d3261c700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0d34196390 0x7f0d340e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d31e1b700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 0x7f0d340e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 -- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 msgr2=0x7f0d340e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 0x7f0d340e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 -- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0d34196390 msgr2=0x7f0d340e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0d34196390 0x7f0d340e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 -- 172.21.2.117:0/4150814367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d340e8c60 con 0x7f0d340e9960 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d31e1b700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 0x7f0d340e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:47:48.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.650 7f0d32e1d700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f0d2c001ea0 tx=0x7f0d2c00a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d2c00ac90 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c007030 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0d2c004620 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d38901700 1 -- 172.21.2.117:0/4150814367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d340e8c60 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d38901700 1 -- 172.21.2.117:0/4150814367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3419cd70 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.651 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0d2c0043a0 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.652 7f0d2affd700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 0x7f0d1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.652 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0d2c032710 con 0x7f0d340e9960 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.652 7f0d3261c700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 0x7f0d1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:48.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.652 7f0d3261c700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 0x7f0d1801abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:48.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.682 7f0d38901700 1 -- 172.21.2.117:0/4150814367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d34087210 con 0x7f0d340e9960 2021-02-10T11:47:48.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.685 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0d2c000c10 con 0x7f0d340e9960 2021-02-10T11:47:48.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.813 7f0d38901700 1 -- 172.21.2.117:0/4150814367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d340ede60 con 0x7f0d340e9960 2021-02-10T11:47:48.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.814 7f0d2affd700 1 -- 172.21.2.117:0/4150814367 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f0d2c0043a0 con 0x7f0d340e9960 2021-02-10T11:47:48.810 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:48.810 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":68,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:47:44.486277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6539":{"gid":6539,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/4168849820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":4168849820},{"type":"v1","addr":"172.21.2.125:6835","nonce":4168849820}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:47:48.811 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 68 2021-02-10T11:47:48.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 msgr2=0x7f0d1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 0x7f0d1801abc0 crc :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 msgr2=0x7f0d340e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 rx=0x7f0d2c001ea0 tx=0x7f0d2c00a8a0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 shutdown_connections 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0d180186c0 0x7f0d1801abc0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0d340e9960 0x7f0d340e8420 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0d340e9560 0x7f0d340e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 --2- 172.21.2.117:0/4150814367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0d34196390 0x7f0d340e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:48.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 >> 172.21.2.117:0/4150814367 conn(0x7f0d3419f610 msgr2=0x7f0d341a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:48.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 shutdown_connections 2021-02-10T11:47:48.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:48.816 7f0d38901700 1 -- 172.21.2.117:0/4150814367 wait complete. 2021-02-10T11:47:49.023 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:42.357500', 'modified': '2021-02-10 11:47:44.486277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6539}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6539': {'gid': 6539, 'name': 'b', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/4168849820', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 4168849820}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 4168849820}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:47:49.023 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:47:49.023 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:47:49.421 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:47:49.421 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6539, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 7.066874576, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 68, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 91, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 91, 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.697281091000001 2021-02-10T11:47:49.422 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:47:49.425 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6539, "want_state": "up:active", "state": "up:active", "rank_uptime": 7.066874576, "mdsmap_epoch": 68, "osdmap_epoch": 91, "osdmap_epoch_barrier": 91, "uptime": 12.697281091000001 } 2021-02-10T11:47:49.426 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:49.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.840 7f9b3e8d0700 1 Processor -- start 2021-02-10T11:47:49.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.840 7f9b3e8d0700 1 -- start start 2021-02-10T11:47:49.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3e8d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380d18e0 0x7f9b380ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3e8d0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9b380eb000 0x7f9b380eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3e8d0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b380ede60 con 0x7f9b380ee3e0 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3e8d0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b380eba20 con 0x7f9b380eb000 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3e8d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380d1ce0 con 0x7f9b380d18e0 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3ceed700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f9b380ee3e0 0x7f9b380ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45886/0 (socket says 172.21.2.117:45886) 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b3ceed700 1 -- 172.21.2.117:0/1638399711 learned_addr learned my addr 172.21.2.117:0/1638399711 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 --2- 172.21.2.117:0/1638399711 >> v2:172.21.2.125:3300/0 conn(0x7f9b380eb000 0x7f9b380eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b37fff700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380d18e0 0x7f9b380ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 -- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380d18e0 msgr2=0x7f9b380ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380d18e0 0x7f9b380ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 -- 172.21.2.117:0/1638399711 >> v1:172.21.2.125:6789/0 conn(0x7f9b380ee3e0 legacy=0x7f9b380ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 -- 172.21.2.117:0/1638399711 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380eefd0 con 0x7f9b380eb000 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.841 7f9b377fe700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380eb000 0x7f9b380eb400 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f9b2c006650 tx=0x7f9b2c009a80).ready entity=mon.1 client_cookie=89c9d8d51e500573 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b36ffd700 1 -- 172.21.2.117:0/1638399711 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c00c0c0 con 0x7f9b380eb000 2021-02-10T11:47:49.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b36ffd700 1 -- 172.21.2.117:0/1638399711 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c00c340 con 0x7f9b380eb000 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b36ffd700 1 -- 172.21.2.117:0/1638399711 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b2c00c860 con 0x7f9b380eb000 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 -- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380eb000 msgr2=0x7f9b380eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380eb000 0x7f9b380eb400 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f9b2c006650 tx=0x7f9b2c009a80).stop 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 -- 172.21.2.117:0/1638399711 shutdown_connections 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380eb000 0x7f9b380eb400 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 --2- 172.21.2.117:0/1638399711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380d18e0 0x7f9b380ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 -- 172.21.2.117:0/1638399711 >> 172.21.2.117:0/1638399711 conn(0x7f9b380ced60 msgr2=0x7f9b380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 -- 172.21.2.117:0/1638399711 shutdown_connections 2021-02-10T11:47:49.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.842 7f9b3e8d0700 1 -- 172.21.2.117:0/1638399711 wait complete. 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 Processor -- start 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 -- start start 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380b77e0 0x7f9b380b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380b7310 0x7f9b380aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380ab8c0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380ac460 con 0x7f9b380b77e0 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3e8d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b380ac680 con 0x7f9b380b7310 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3ceed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3ceed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52716/0 (socket says 172.21.2.117:52716) 2021-02-10T11:47:49.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.857 7f9b3ceed700 1 -- 172.21.2.117:0/3456939756 learned_addr learned my addr 172.21.2.117:0/3456939756 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 -- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380b7310 msgr2=0x7f9b380aaca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380b7310 0x7f9b380aaca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 -- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380b77e0 msgr2=0x7f9b380b6cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380b77e0 0x7f9b380b6cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 -- 172.21.2.117:0/3456939756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b380ac8a0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3ceed700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7f9b30007fc0 tx=0x7f9b3000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b300070d0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b30007390 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b380ac8a0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9b300046a0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.858 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b380ac8a0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.859 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9b300046a0 con 0x7f9b380ab2c0 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.859 7f9b34ff9700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 0x7f9b1801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:49.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.859 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9b30032b00 con 0x7f9b380ab2c0 2021-02-10T11:47:49.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.859 7f9b37fff700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 0x7f9b1801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:49.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.860 7f9b37fff700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 0x7f9b1801abd0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:49.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.891 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b38087290 con 0x7f9b380ab2c0 2021-02-10T11:47:49.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:49.893 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9b30000cc0 con 0x7f9b380ab2c0 2021-02-10T11:47:50.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.016 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b380ede60 con 0x7f9b380ab2c0 2021-02-10T11:47:50.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.017 7f9b34ff9700 1 -- 172.21.2.117:0/3456939756 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f9b30000cc0 con 0x7f9b380ab2c0 2021-02-10T11:47:50.013 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:50.013 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":91,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:43.359349","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:40.329394","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:41.402719","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:50.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 msgr2=0x7f9b1801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 0x7f9b1801abd0 crc :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 msgr2=0x7f9b380abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 rx=0x7f9b30007fc0 tx=0x7f9b3000ac80).stop 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 shutdown_connections 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9b180186d0 0x7f9b1801abd0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9b380ab2c0 0x7f9b380abe40 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9b380b7310 0x7f9b380aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 --2- 172.21.2.117:0/3456939756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9b380b77e0 0x7f9b380b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 >> 172.21.2.117:0/3456939756 conn(0x7f9b380874c0 msgr2=0x7f9b380af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 shutdown_connections 2021-02-10T11:47:50.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.019 7f9b3e8d0700 1 -- 172.21.2.117:0/3456939756 wait complete. 2021-02-10T11:47:50.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:47:40.329394', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:47:41.402719', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:47:50.024 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.444 7f78ba7eb700 1 Processor -- start 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.444 7f78ba7eb700 1 -- start start 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78ba7eb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f78b40ef3f0 0x7f78b40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78ba7eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40eb830 0x7f78b41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78ba7eb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b40ef7f0 con 0x7f78b40d18f0 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78ba7eb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b40eded0 con 0x7f78b40ef3f0 2021-02-10T11:47:50.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78ba7eb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40d1cf0 con 0x7f78b40eb830 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f78b40d18f0 0x7f78b40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52972/0 (socket says 172.21.2.117:52972) 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b3fff700 1 -- 172.21.2.117:0/4019347434 learned_addr learned my addr 172.21.2.117:0/4019347434 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b37fe700 1 --2- 172.21.2.117:0/4019347434 >> v2:172.21.2.125:3300/0 conn(0x7f78b40ef3f0 0x7f78b40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b8e08700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40eb830 0x7f78b41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b2ffd700 1 -- 172.21.2.117:0/4019347434 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65417782 0 0) 0x7f78a4001180 con 0x7f78b40d18f0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b2ffd700 1 -- 172.21.2.117:0/4019347434 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f789c002fe0 con 0x7f78b40d18f0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b37fe700 1 -- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40eb830 msgr2=0x7f78b41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b37fe700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40eb830 0x7f78b41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b37fe700 1 -- 172.21.2.117:0/4019347434 >> v1:172.21.2.117:6789/0 conn(0x7f78b40d18f0 legacy=0x7f78b40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.445 7f78b37fe700 1 -- 172.21.2.117:0/4019347434 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b41b8fd0 con 0x7f78b40ef3f0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78b37fe700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b40ef3f0 0x7f78b40eb210 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f78a8006650 tx=0x7f78a8009a80).ready entity=mon.1 client_cookie=6615c6d3270b22f3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78b2ffd700 1 -- 172.21.2.117:0/4019347434 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a80009b0 con 0x7f78b40ef3f0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78b2ffd700 1 -- 172.21.2.117:0/4019347434 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a800c0c0 con 0x7f78b40ef3f0 2021-02-10T11:47:50.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78b2ffd700 1 -- 172.21.2.117:0/4019347434 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a80009b0 con 0x7f78b40ef3f0 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 -- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b40ef3f0 msgr2=0x7f78b40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b40ef3f0 0x7f78b40eb210 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f78a8006650 tx=0x7f78a8009a80).stop 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 -- 172.21.2.117:0/4019347434 shutdown_connections 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40eb830 0x7f78b41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 --2- 172.21.2.117:0/4019347434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b40ef3f0 0x7f78b40eb210 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 -- 172.21.2.117:0/4019347434 >> 172.21.2.117:0/4019347434 conn(0x7f78b40cf060 msgr2=0x7f78b40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 -- 172.21.2.117:0/4019347434 shutdown_connections 2021-02-10T11:47:50.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.446 7f78ba7eb700 1 -- 172.21.2.117:0/4019347434 wait complete. 2021-02-10T11:47:50.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 Processor -- start 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 -- start start 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 0x7f78b40b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 0x7f78b419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40b35e0 0x7f78b41a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b3b50 con 0x7f78b40b31e0 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b4094490 con 0x7f78b4094090 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78ba7eb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b41a08a0 con 0x7f78b40b35e0 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78b3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 0x7f78b40b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78b3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 0x7f78b40b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57112/0 (socket says 172.21.2.117:57112) 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78b37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 0x7f78b419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.461 7f78b3fff700 1 -- 172.21.2.117:0/1796386125 learned_addr learned my addr 172.21.2.117:0/1796386125 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:50.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b8e08700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40b35e0 0x7f78b41a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 -- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40b35e0 msgr2=0x7f78b41a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40b35e0 0x7f78b41a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 -- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 msgr2=0x7f78b40b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 0x7f78b40b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 -- 172.21.2.117:0/1796386125 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b407c7c0 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b37fe700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 0x7f78b419fc60 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f78a80066b0 tx=0x7f78a800b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a80067f0 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a8000d60 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b407c7c0 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b407c7c0 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.462 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78a8002220 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.463 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78a8000940 con 0x7f78b40b31e0 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.463 7f78b0ff9700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 0x7f789401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:50.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.463 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78a8032830 con 0x7f78b40b31e0 2021-02-10T11:47:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.463 7f78b3fff700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 0x7f789401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:50.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.463 7f78b3fff700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 0x7f789401abd0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:50.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.494 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b40daff0 con 0x7f78b40b31e0 2021-02-10T11:47:50.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.497 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f78a80067f0 con 0x7f78b40b31e0 2021-02-10T11:47:50.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.623 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f78b40ede60 con 0x7f78b40b31e0 2021-02-10T11:47:50.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.624 7f78b0ff9700 1 -- 172.21.2.117:0/1796386125 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f78a8000940 con 0x7f78b40b31e0 2021-02-10T11:47:50.620 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:50.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 msgr2=0x7f789401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 0x7f789401abd0 crc :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 msgr2=0x7f78b419fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 0x7f78b419fc60 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 rx=0x7f78a80066b0 tx=0x7f78a800b8e0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 shutdown_connections 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78940186d0 0x7f789401abd0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78b40b31e0 0x7f78b419fc60 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78b40b35e0 0x7f78b41a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 --2- 172.21.2.117:0/1796386125 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78b4094090 0x7f78b40b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:50.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 >> 172.21.2.117:0/1796386125 conn(0x7f78b40b8180 msgr2=0x7f78b40b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:50.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 shutdown_connections 2021-02-10T11:47:50.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:50.626 7f78ba7eb700 1 -- 172.21.2.117:0/1796386125 wait complete. 2021-02-10T11:47:50.631 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 Processor -- start 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 -- start start 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f31a40d18f0 0x7f31a40eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 0x7f31a41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a40ef7f0 con 0x7f31a40eb040 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a40eded0 con 0x7f31a40d18f0 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.049 7f31aad9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40d1cf0 con 0x7f31a40ef3f0 2021-02-10T11:47:51.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 0x7f31a41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f31a40eb040 0x7f31a40eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:52986/0 (socket says 172.21.2.117:52986) 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a3fff700 1 -- 172.21.2.117:0/2831406124 learned_addr learned my addr 172.21.2.117:0/2831406124 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a8bbb700 1 --2- 172.21.2.117:0/2831406124 >> v2:172.21.2.125:3300/0 conn(0x7f31a40d18f0 0x7f31a40eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 -- 172.21.2.117:0/2831406124 >> v2:172.21.2.125:3300/0 conn(0x7f31a40d18f0 msgr2=0x7f31a40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 --2- 172.21.2.117:0/2831406124 >> v2:172.21.2.125:3300/0 conn(0x7f31a40d18f0 0x7f31a40eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 -- 172.21.2.117:0/2831406124 >> v1:172.21.2.117:6789/0 conn(0x7f31a40eb040 legacy=0x7f31a40eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a37fe700 1 -- 172.21.2.117:0/2831406124 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733083247 0 0) 0x7f3194001060 con 0x7f31a40eb040 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 -- 172.21.2.117:0/2831406124 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a41b8fd0 con 0x7f31a40ef3f0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.050 7f31a93bc700 1 --2- 172.21.2.117:0/2831406124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 0x7f31a41b79a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f319c006ad0 tx=0x7f319c009a50).ready entity=mon.2 client_cookie=489fcd3c4438586f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31a37fe700 1 -- 172.21.2.117:0/2831406124 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c0066a0 con 0x7f31a40ef3f0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31a37fe700 1 -- 172.21.2.117:0/2831406124 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c006410 con 0x7f31a40ef3f0 2021-02-10T11:47:51.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31a37fe700 1 -- 172.21.2.117:0/2831406124 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c0024d0 con 0x7f31a40ef3f0 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 -- 172.21.2.117:0/2831406124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 msgr2=0x7f31a41b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 --2- 172.21.2.117:0/2831406124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 0x7f31a41b79a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f319c006ad0 tx=0x7f319c009a50).stop 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 -- 172.21.2.117:0/2831406124 shutdown_connections 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 --2- 172.21.2.117:0/2831406124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a40ef3f0 0x7f31a41b79a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 --2- 172.21.2.117:0/2831406124 >> v2:172.21.2.125:3300/0 conn(0x7f31a40d18f0 0x7f31a40eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 -- 172.21.2.117:0/2831406124 >> 172.21.2.117:0/2831406124 conn(0x7f31a40cf060 msgr2=0x7f31a40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 -- 172.21.2.117:0/2831406124 shutdown_connections 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.051 7f31aad9f700 1 -- 172.21.2.117:0/2831406124 wait complete. 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 Processor -- start 2021-02-10T11:47:51.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 -- start start 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a407c7c0 0x7f31a41a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a41a3930 0x7f31a40a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a407cca0 con 0x7f31a40a13a0 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a418fb70 con 0x7f31a41a3930 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.066 7f31aad9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40dc6d0 con 0x7f31a407c7c0 2021-02-10T11:47:51.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52744/0 (socket says 172.21.2.117:52744) 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 -- 172.21.2.117:0/3897550189 learned_addr learned my addr 172.21.2.117:0/3897550189 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a8bbb700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a407c7c0 0x7f31a41a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a3fff700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a41a3930 0x7f31a40a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 -- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a407c7c0 msgr2=0x7f31a41a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a407c7c0 0x7f31a41a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 -- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a41a3930 msgr2=0x7f31a40a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a41a3930 0x7f31a40a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 -- 172.21.2.117:0/3897550189 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a407d220 con 0x7f31a40a13a0 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.067 7f31a93bc700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7f319c0095e0 tx=0x7f319c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c002030 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319c002270 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a407d220 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a407d7b0 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319c002790 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f319c002450 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.068 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f319c032690 con 0x7f31a40a13a0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.069 7f31a8bbb700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.069 7f31a8bbb700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 0x7f318801abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.099 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a418fb20 con 0x7f31a40a13a0 2021-02-10T11:47:51.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.102 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f319c006cc0 con 0x7f31a40a13a0 2021-02-10T11:47:51.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.230 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f31a418fb20 con 0x7f31a40a13a0 2021-02-10T11:47:51.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.231 7f31a17fa700 1 -- 172.21.2.117:0/3897550189 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7f319c00bd80 con 0x7f31a40a13a0 2021-02-10T11:47:51.227 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 msgr2=0x7f318801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 0x7f318801abc0 crc :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 msgr2=0x7f31a40a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7f319c0095e0 tx=0x7f319c00b990).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 shutdown_connections 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31880186c0 0x7f318801abc0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a40a13a0 0x7f31a40a17a0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a41a3930 0x7f31a40a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 --2- 172.21.2.117:0/3897550189 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a407c7c0 0x7f31a41a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 >> 172.21.2.117:0/3897550189 conn(0x7f31a4087090 msgr2=0x7f31a408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:51.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 shutdown_connections 2021-02-10T11:47:51.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.233 7f31aad9f700 1 -- 172.21.2.117:0/3897550189 wait complete. 2021-02-10T11:47:51.237 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:47:51.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.656 7ff1ca0ee700 1 Processor -- start 2021-02-10T11:47:51.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.656 7ff1ca0ee700 1 -- start start 2021-02-10T11:47:51.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.657 7ff1ca0ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 0x7ff1c40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.657 7ff1ca0ee700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 0x7ff1c41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.657 7ff1ca0ee700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40ef7f0 con 0x7ff1c40d18f0 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.657 7ff1ca0ee700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40eded0 con 0x7ff1c40eb830 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.657 7ff1ca0ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40d1cf0 con 0x7ff1c40ef3f0 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c37fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7ff1c40d18f0 0x7ff1c40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45924/0 (socket says 172.21.2.117:45924) 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 0x7ff1c40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c37fe700 1 -- 172.21.2.117:0/852162964 learned_addr learned my addr 172.21.2.117:0/852162964 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c3fff700 1 --2- 172.21.2.117:0/852162964 >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 0x7ff1c41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 -- 172.21.2.117:0/852162964 >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 msgr2=0x7ff1c41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 --2- 172.21.2.117:0/852162964 >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 0x7ff1c41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 -- 172.21.2.117:0/852162964 >> v1:172.21.2.125:6789/0 conn(0x7ff1c40d18f0 legacy=0x7ff1c40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 -- 172.21.2.117:0/852162964 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c41b8fd0 con 0x7ff1c40ef3f0 2021-02-10T11:47:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c3fff700 1 --2- 172.21.2.117:0/852162964 >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 0x7ff1c41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.658 7ff1c2ffd700 1 --2- 172.21.2.117:0/852162964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 0x7ff1c40eb210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff1ac006650 tx=0x7ff1ac009a80).ready entity=mon.2 client_cookie=a5936099c70b2fe6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1c27fc700 1 -- 172.21.2.117:0/852162964 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1ac0009b0 con 0x7ff1c40ef3f0 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1c27fc700 1 -- 172.21.2.117:0/852162964 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac00c0c0 con 0x7ff1c40ef3f0 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1c27fc700 1 -- 172.21.2.117:0/852162964 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1ac00c5e0 con 0x7ff1c40ef3f0 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1ca0ee700 1 -- 172.21.2.117:0/852162964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 msgr2=0x7ff1c40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1ca0ee700 1 --2- 172.21.2.117:0/852162964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 0x7ff1c40eb210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7ff1ac006650 tx=0x7ff1ac009a80).stop 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.659 7ff1ca0ee700 1 -- 172.21.2.117:0/852162964 shutdown_connections 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.660 7ff1ca0ee700 1 --2- 172.21.2.117:0/852162964 >> v2:172.21.2.125:3300/0 conn(0x7ff1c40eb830 0x7ff1c41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.660 7ff1ca0ee700 1 --2- 172.21.2.117:0/852162964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c40ef3f0 0x7ff1c40eb210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.660 7ff1ca0ee700 1 -- 172.21.2.117:0/852162964 >> 172.21.2.117:0/852162964 conn(0x7ff1c40cf060 msgr2=0x7ff1c40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.660 7ff1ca0ee700 1 -- 172.21.2.117:0/852162964 shutdown_connections 2021-02-10T11:47:51.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.660 7ff1ca0ee700 1 -- 172.21.2.117:0/852162964 wait complete. 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 Processor -- start 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 -- start start 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c41a48b0 0x7ff1c408c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c408cf40 0x7ff1c40a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40aae30 con 0x7ff1c40a9f90 2021-02-10T11:47:51.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40aa9f0 con 0x7ff1c41a48b0 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1ca0ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40ab330 con 0x7ff1c408cf40 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1c3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1c3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52754/0 (socket says 172.21.2.117:52754) 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.674 7ff1c3fff700 1 -- 172.21.2.117:0/3049085936 learned_addr learned my addr 172.21.2.117:0/3049085936 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 -- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c408cf40 msgr2=0x7ff1c40a9970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c408cf40 0x7ff1c40a9970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 -- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c41a48b0 msgr2=0x7ff1c408c920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c41a48b0 0x7ff1c408c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 -- 172.21.2.117:0/3049085936 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40ab550 con 0x7ff1c40a9f90 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1c3fff700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 rx=0x7ff1bc00b970 tx=0x7ff1bc00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1bc000f90 con 0x7ff1c40a9f90 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc003030 con 0x7ff1c40a9f90 2021-02-10T11:47:51.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff1bc001400 con 0x7ff1c40a9f90 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c40ab550 con 0x7ff1c40a9f90 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.675 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c40ab550 con 0x7ff1c40a9f90 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.676 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff1bc000df0 con 0x7ff1c40a9f90 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.676 7ff1bbfff700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 0x7ff1a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.676 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff1bc032b00 con 0x7ff1c40a9f90 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.676 7ff1c37fe700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 0x7ff1a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:51.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.676 7ff1c37fe700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 0x7ff1a801abd0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:51.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.707 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c40ede60 con 0x7ff1c40a9f90 2021-02-10T11:47:51.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.709 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff1bc0066f0 con 0x7ff1c40a9f90 2021-02-10T11:47:51.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.836 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ff1c40ede60 con 0x7ff1c40a9f90 2021-02-10T11:47:51.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.836 7ff1bbfff700 1 -- 172.21.2.117:0/3049085936 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff1bc000df0 con 0x7ff1c40a9f90 2021-02-10T11:47:51.833 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:51.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.838 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 msgr2=0x7ff1a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.838 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 0x7ff1a801abd0 crc :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.838 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 msgr2=0x7ff1c40aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.838 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 rx=0x7ff1bc00b970 tx=0x7ff1bc00bd10).stop 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 shutdown_connections 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff1a80186d0 0x7ff1a801abd0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff1c40a9f90 0x7ff1c40aa390 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff1c408cf40 0x7ff1c40a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 --2- 172.21.2.117:0/3049085936 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff1c41a48b0 0x7ff1c408c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 >> 172.21.2.117:0/3049085936 conn(0x7ff1c4092d00 msgr2=0x7ff1c408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 shutdown_connections 2021-02-10T11:47:51.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:51.839 7ff1ca0ee700 1 -- 172.21.2.117:0/3049085936 wait complete. 2021-02-10T11:47:51.843 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:47:51.843 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:52.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 Processor -- start 2021-02-10T11:47:52.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 -- start start 2021-02-10T11:47:52.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8640e4e40 0x7ff8641b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8640e5240 0x7ff864084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff864085930 con 0x7ff8641b0810 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8640e5cc0 con 0x7ff8640e5240 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.257 7ff86a794700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff864084840 con 0x7ff8640e4e40 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8640e5240 0x7ff864084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8640e5240 0x7ff864084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52762/0 (socket says 172.21.2.117:52762) 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 -- 172.21.2.117:0/1336900777 learned_addr learned my addr 172.21.2.117:0/1336900777 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff863fff700 1 --2- 172.21.2.117:0/1336900777 >> v2:172.21.2.125:3300/0 conn(0x7ff8640e4e40 0x7ff8641b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 -- 172.21.2.117:0/1336900777 >> v2:172.21.2.125:3300/0 conn(0x7ff8640e4e40 msgr2=0x7ff8641b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 --2- 172.21.2.117:0/1336900777 >> v2:172.21.2.125:3300/0 conn(0x7ff8640e4e40 0x7ff8641b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 -- 172.21.2.117:0/1336900777 >> v1:172.21.2.125:6789/0 conn(0x7ff8641b0810 legacy=0x7ff864085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 -- 172.21.2.117:0/1336900777 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864085280 con 0x7ff8640e5240 2021-02-10T11:47:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff8637fe700 1 --2- 172.21.2.117:0/1336900777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640e5240 0x7ff864084c60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7ff858006700 tx=0x7ff858009b80).ready entity=mon.0 client_cookie=3496966ec2830254 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff862ffd700 1 -- 172.21.2.117:0/1336900777 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff85800c0c0 con 0x7ff8640e5240 2021-02-10T11:47:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff862ffd700 1 -- 172.21.2.117:0/1336900777 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85800c340 con 0x7ff8640e5240 2021-02-10T11:47:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff862ffd700 1 -- 172.21.2.117:0/1336900777 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff85800c860 con 0x7ff8640e5240 2021-02-10T11:47:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff86a794700 1 -- 172.21.2.117:0/1336900777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640e5240 msgr2=0x7ff864084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.258 7ff86a794700 1 --2- 172.21.2.117:0/1336900777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640e5240 0x7ff864084c60 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7ff858006700 tx=0x7ff858009b80).stop 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 -- 172.21.2.117:0/1336900777 shutdown_connections 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 --2- 172.21.2.117:0/1336900777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640e5240 0x7ff864084c60 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 --2- 172.21.2.117:0/1336900777 >> v2:172.21.2.125:3300/0 conn(0x7ff8640e4e40 0x7ff8641b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 -- 172.21.2.117:0/1336900777 >> 172.21.2.117:0/1336900777 conn(0x7ff8640ce4e0 msgr2=0x7ff8640e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:52.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 -- 172.21.2.117:0/1336900777 shutdown_connections 2021-02-10T11:47:52.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.259 7ff86a794700 1 -- 172.21.2.117:0/1336900777 wait complete. 2021-02-10T11:47:52.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 Processor -- start 2021-02-10T11:47:52.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 -- start start 2021-02-10T11:47:52.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640a2010 0x7ff864092e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8640ae820 0x7ff8640a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640934a0 con 0x7ff8640ade00 2021-02-10T11:47:52.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640a2410 con 0x7ff8640a2010 2021-02-10T11:47:52.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff86a794700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640a2630 con 0x7ff8640ae820 2021-02-10T11:47:52.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff8637fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff8637fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52766/0 (socket says 172.21.2.117:52766) 2021-02-10T11:47:52.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.274 7ff8637fe700 1 -- 172.21.2.117:0/2225258469 learned_addr learned my addr 172.21.2.117:0/2225258469 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:52.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff863fff700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640a2010 0x7ff864092e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff868db1700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8640ae820 0x7ff8640a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 -- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8640ae820 msgr2=0x7ff8640a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8640ae820 0x7ff8640a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 -- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640a2010 msgr2=0x7ff864092e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640a2010 0x7ff864092e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 -- 172.21.2.117:0/2225258469 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640aec20 con 0x7ff8640ade00 2021-02-10T11:47:52.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff8637fe700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7ff85800b5e0 tx=0x7ff85800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff85800bd70 con 0x7ff8640ade00 2021-02-10T11:47:52.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff858002030 con 0x7ff8640ade00 2021-02-10T11:47:52.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff86a794700 1 -- 172.21.2.117:0/2225258469 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8640aec20 con 0x7ff8640ade00 2021-02-10T11:47:52.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff86a794700 1 -- 172.21.2.117:0/2225258469 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8640aec20 con 0x7ff8640ade00 2021-02-10T11:47:52.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.275 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff858002550 con 0x7ff8640ade00 2021-02-10T11:47:52.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.276 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff858002210 con 0x7ff8640ade00 2021-02-10T11:47:52.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.276 7ff860ff9700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:52.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.276 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff858032930 con 0x7ff8640ade00 2021-02-10T11:47:52.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.276 7ff863fff700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:52.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.277 7ff863fff700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:52.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.307 7ff86a794700 1 -- 172.21.2.117:0/2225258469 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff864085930 con 0x7ff8640ade00 2021-02-10T11:47:52.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.310 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff858000c90 con 0x7ff8640ade00 2021-02-10T11:47:52.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.438 7ff86a794700 1 -- 172.21.2.117:0/2225258469 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff864085930 con 0x7ff8640ade00 2021-02-10T11:47:52.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.438 7ff860ff9700 1 -- 172.21.2.117:0/2225258469 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2101 (secure 0 0 0) 0x7ff85800bd70 con 0x7ff8640ade00 2021-02-10T11:47:52.434 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 68 2021-02-10T11:47:52.435 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:52.435 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":68,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:47:44.486277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6539":{"gid":6539,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/4168849820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":4168849820},{"type":"v1","addr":"172.21.2.125:6835","nonce":4168849820}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:47:52.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 -- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 msgr2=0x7ff84401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 crc :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 -- 172.21.2.117:0/2225258469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 msgr2=0x7ff8640ae200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 rx=0x7ff85800b5e0 tx=0x7ff85800b980).stop 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 -- 172.21.2.117:0/2225258469 shutdown_connections 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8640ade00 0x7ff8640ae200 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8640ae820 0x7ff8640a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 --2- 172.21.2.117:0/2225258469 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640a2010 0x7ff864092e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 -- 172.21.2.117:0/2225258469 >> 172.21.2.117:0/2225258469 conn(0x7ff8640b5bd0 msgr2=0x7ff8640c39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.440 7ff86a794700 1 -- 172.21.2.117:0/2225258469 shutdown_connections 2021-02-10T11:47:52.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:52.441 7ff86a794700 1 -- 172.21.2.117:0/2225258469 wait complete. 2021-02-10T11:47:52.646 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:42.357500', 'modified': '2021-02-10 11:47:44.486277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6539}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6539': {'gid': 6539, 'name': 'b', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/4168849820', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 4168849820}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 4168849820}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:47:52.647 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:47:52.647 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:47:53.044 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:47:53.044 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:47:53.044 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6539, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.689446633999999, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 68, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 91, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 91, 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.319854118999999 2021-02-10T11:47:53.045 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:47:53.048 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6539, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.689446633999999, "mdsmap_epoch": 68, "osdmap_epoch": 91, "osdmap_epoch_barrier": 91, "uptime": 16.319854118999999 } 2021-02-10T11:47:53.049 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:47:53.049 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:47:53.049 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:47:53.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:53.049 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:53.385 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:47:53.729 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:53.729 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:54.072 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:54.072 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:54.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:54.073 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:47:54.408 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:47:54.408 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:54.410 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:47:54.754 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:47:54.754 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:47:54.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[150254]: starting ceph client 2021-02-10T11:47:54.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:47:54.824 7ff7d9d00f80 -1 init, newargv = 0x55ee26e99fb0 newargc=9 2021-02-10T11:47:54.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[150254]: starting fuse 2021-02-10T11:47:55.096 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:47:55.096 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:47:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:47:55.097 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:55.098 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:47:55.432 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:47:55.432 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:47:55.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:47:55.433 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:47:55.434 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:47:55.794 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.150254.asok 2021-02-10T11:47:55.797 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.150254.asok 2021-02-10T11:47:55.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:55.797 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.150254.asok status 2021-02-10T11:47:56.188 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:47:56.188 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:47:56.188 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:47:56.188 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:47:56.188 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "150254", 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9882, 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9882 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:56.189 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3337755100 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3337755100 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9882 172.21.2.117:0/3337755100", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3337755100", 2021-02-10T11:47:56.190 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:47:56.191 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 68, 2021-02-10T11:47:56.191 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 91, 2021-02-10T11:47:56.191 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:47:56.191 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:47:56.191 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:47:56.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:56.192 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:47:56.192 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:56.528 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:47:56.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:47:56.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:47:56.529 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:47:56.876 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 Processor -- start 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 -- start start 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640e4e40 0x7f59641b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f59640e5240 0x7f5964084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5964085930 con 0x7f59641b0810 2021-02-10T11:47:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5964087430 con 0x7f59640e5240 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5969871700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59640e5cc0 con 0x7f59640e4e40 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.291 7f5963d75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f59641b0810 0x7f5964085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53036/0 (socket says 172.21.2.117:53036) 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5963d75700 1 -- 172.21.2.117:0/1769937344 learned_addr learned my addr 172.21.2.117:0/1769937344 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 --2- 172.21.2.117:0/1769937344 >> v2:172.21.2.117:3300/0 conn(0x7f59640e5240 0x7f5964084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 -- 172.21.2.117:0/1769937344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640e4e40 msgr2=0x7f59641b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962572700 1 -- 172.21.2.117:0/1769937344 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547962847 0 0) 0x7f595c001500 con 0x7f59641b0810 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 --2- 172.21.2.117:0/1769937344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640e4e40 0x7f59641b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 -- 172.21.2.117:0/1769937344 >> v1:172.21.2.117:6789/0 conn(0x7f59641b0810 legacy=0x7f5964085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 -- 172.21.2.117:0/1769937344 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964084840 con 0x7f59640e5240 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962d73700 1 --2- 172.21.2.117:0/1769937344 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59640e5240 0x7f5964084c60 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f5954006650 tx=0x7f5954009a80).ready entity=mon.0 client_cookie=2087b4755892ad7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962572700 1 -- 172.21.2.117:0/1769937344 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595400c0c0 con 0x7f59640e5240 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962572700 1 -- 172.21.2.117:0/1769937344 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595400c340 con 0x7f59640e5240 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5962572700 1 -- 172.21.2.117:0/1769937344 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595400c860 con 0x7f59640e5240 2021-02-10T11:47:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5969871700 1 -- 172.21.2.117:0/1769937344 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59640e5240 msgr2=0x7f5964084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.292 7f5969871700 1 --2- 172.21.2.117:0/1769937344 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59640e5240 0x7f5964084c60 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 rx=0x7f5954006650 tx=0x7f5954009a80).stop 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 -- 172.21.2.117:0/1769937344 shutdown_connections 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 --2- 172.21.2.117:0/1769937344 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59640e5240 0x7f5964084c60 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 --2- 172.21.2.117:0/1769937344 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640e4e40 0x7f59641b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 -- 172.21.2.117:0/1769937344 >> 172.21.2.117:0/1769937344 conn(0x7f59640ce4e0 msgr2=0x7f59640e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 -- 172.21.2.117:0/1769937344 shutdown_connections 2021-02-10T11:47:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.293 7f5969871700 1 -- 172.21.2.117:0/1769937344 wait complete. 2021-02-10T11:47:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 Processor -- start 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 -- start start 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f596419b620 0x7f59640b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640b20b0 0x7f59641aa070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596419baa0 con 0x7f59641aa690 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596419a2e0 con 0x7f596419b620 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5969871700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5964199e40 con 0x7f59640b20b0 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5963d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5963d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52792/0 (socket says 172.21.2.117:52792) 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.308 7f5963d75700 1 -- 172.21.2.117:0/1777772545 learned_addr learned my addr 172.21.2.117:0/1777772545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 -- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640b20b0 msgr2=0x7f59641aa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640b20b0 0x7f59641aa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 -- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f596419b620 msgr2=0x7f59640b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f596419b620 0x7f59640b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 -- 172.21.2.117:0/1777772545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59641aaa90 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5963d75700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f595c001c10 tx=0x7f595c00ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595c007020 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c0072a0 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f595c004720 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5969871700 1 -- 172.21.2.117:0/1777772545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59641aaa90 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.309 7f5969871700 1 -- 172.21.2.117:0/1777772545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596419cc10 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.310 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f595c007480 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.310 7f595bfff700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.310 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f595c032a20 con 0x7f59641aa690 2021-02-10T11:47:57.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.310 7f5963574700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.311 7f5963574700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 0x7f594801abc0 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.341 7f5969871700 1 -- 172.21.2.117:0/1777772545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596410c280 con 0x7f59641aa690 2021-02-10T11:47:57.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.344 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f595c000f90 con 0x7f59641aa690 2021-02-10T11:47:57.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.467 7f5969871700 1 -- 172.21.2.117:0/1777772545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5964085930 con 0x7f59641aa690 2021-02-10T11:47:57.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.467 7f595bfff700 1 -- 172.21.2.117:0/1777772545 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f595c007460 con 0x7f59641aa690 2021-02-10T11:47:57.464 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:57.464 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":91,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:47:43.359349","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:47:40.329394","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":19,"pool_name":"cephfs_data","create_time":"2021-02-10 11:47:41.402719","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:47:57.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 msgr2=0x7f594801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 0x7f594801abc0 crc :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 msgr2=0x7f596419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 rx=0x7f595c001c10 tx=0x7f595c00ac60).stop 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 shutdown_connections 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59480186c0 0x7f594801abc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59641aa690 0x7f596419c300 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59640b20b0 0x7f59641aa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 --2- 172.21.2.117:0/1777772545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f596419b620 0x7f59640b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 >> 172.21.2.117:0/1777772545 conn(0x7f59641b8ea0 msgr2=0x7f596418e110 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:57.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 shutdown_connections 2021-02-10T11:47:57.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.469 7f5969871700 1 -- 172.21.2.117:0/1777772545 wait complete. 2021-02-10T11:47:57.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:47:40.329394', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:47:41.402719', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:47:57.474 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.892 7f21a5af2700 1 Processor -- start 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.892 7f21a5af2700 1 -- start start 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f21a5af2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f21a00ef3f0 0x7f21a00eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f21a5af2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21a00eb830 0x7f21a01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f21a5af2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a00ef7f0 con 0x7f21a00d18f0 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f21a5af2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00ede60 con 0x7f21a00eb830 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f21a5af2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00d1cf0 con 0x7f21a00ef3f0 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21a00eb830 0x7f21a01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21a00eb830 0x7f21a01b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52800/0 (socket says 172.21.2.117:52800) 2021-02-10T11:47:57.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 -- 172.21.2.117:0/1913141011 learned_addr learned my addr 172.21.2.117:0/1913141011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 -- 172.21.2.117:0/1913141011 >> v2:172.21.2.125:3300/0 conn(0x7f21a00ef3f0 msgr2=0x7f21a00eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 --2- 172.21.2.117:0/1913141011 >> v2:172.21.2.125:3300/0 conn(0x7f21a00ef3f0 0x7f21a00eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 -- 172.21.2.117:0/1913141011 >> v1:172.21.2.125:6789/0 conn(0x7f21a00d18f0 legacy=0x7f21a00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.893 7f219ffff700 1 -- 172.21.2.117:0/1913141011 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a01b8fd0 con 0x7f21a00eb830 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f219ffff700 1 --2- 172.21.2.117:0/1913141011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00eb830 0x7f21a01b79a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f2198006f00 tx=0x7f2198009f10).ready entity=mon.0 client_cookie=1f76c621dfb90dc1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f219e7fc700 1 -- 172.21.2.117:0/1913141011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219800b3b0 con 0x7f21a00eb830 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f219e7fc700 1 -- 172.21.2.117:0/1913141011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219800b630 con 0x7f21a00eb830 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f219e7fc700 1 -- 172.21.2.117:0/1913141011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f219800bb50 con 0x7f21a00eb830 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 -- 172.21.2.117:0/1913141011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00eb830 msgr2=0x7f21a01b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 --2- 172.21.2.117:0/1913141011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00eb830 0x7f21a01b79a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 rx=0x7f2198006f00 tx=0x7f2198009f10).stop 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 -- 172.21.2.117:0/1913141011 shutdown_connections 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 --2- 172.21.2.117:0/1913141011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00eb830 0x7f21a01b79a0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 --2- 172.21.2.117:0/1913141011 >> v2:172.21.2.125:3300/0 conn(0x7f21a00ef3f0 0x7f21a00eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.894 7f21a5af2700 1 -- 172.21.2.117:0/1913141011 >> 172.21.2.117:0/1913141011 conn(0x7f21a00cf060 msgr2=0x7f21a00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:57.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.895 7f21a5af2700 1 -- 172.21.2.117:0/1913141011 shutdown_connections 2021-02-10T11:47:57.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.895 7f21a5af2700 1 -- 172.21.2.117:0/1913141011 wait complete. 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.909 7f21a5af2700 1 Processor -- start 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.909 7f21a5af2700 1 -- start start 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21a007c7c0 0x7f21a01a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21a01a3930 0x7f21a00a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a007cca0 con 0x7f21a00a13a0 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a018fb70 con 0x7f21a007c7c0 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f21a5af2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00dc6d0 con 0x7f21a01a3930 2021-02-10T11:47:57.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52806/0 (socket says 172.21.2.117:52806) 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 -- 172.21.2.117:0/998866350 learned_addr learned my addr 172.21.2.117:0/998866350 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 -- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21a01a3930 msgr2=0x7f21a00a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21a01a3930 0x7f21a00a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 -- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21a007c7c0 msgr2=0x7f21a01a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21a007c7c0 0x7f21a01a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 -- 172.21.2.117:0/998866350 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a007d220 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.910 7f219ffff700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 rx=0x7f2198009390 tx=0x7f219800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2198003030 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21980068e0 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2198001700 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f21a5af2700 1 -- 172.21.2.117:0/998866350 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a007d220 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f21a5af2700 1 -- 172.21.2.117:0/998866350 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a007d220 con 0x7f21a00a13a0 2021-02-10T11:47:57.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.911 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2198001270 con 0x7f21a00a13a0 2021-02-10T11:47:57.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.912 7f2197fff700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 0x7f218401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:57.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.912 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2198032c60 con 0x7f21a00a13a0 2021-02-10T11:47:57.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.912 7f219f7fe700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 0x7f218401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:57.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.913 7f219f7fe700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 0x7f218401abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:57.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.943 7f21a5af2700 1 -- 172.21.2.117:0/998866350 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a00daff0 con 0x7f21a00a13a0 2021-02-10T11:47:57.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:57.946 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2198001f00 con 0x7f21a00a13a0 2021-02-10T11:47:58.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.072 7f21a5af2700 1 -- 172.21.2.117:0/998866350 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f21a00daff0 con 0x7f21a00a13a0 2021-02-10T11:47:58.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.073 7f2197fff700 1 -- 172.21.2.117:0/998866350 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f21980068e0 con 0x7f21a00a13a0 2021-02-10T11:47:58.069 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:58.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.074 7f21a5af2700 1 -- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 msgr2=0x7f218401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.074 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 0x7f218401abc0 crc :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.074 7f21a5af2700 1 -- 172.21.2.117:0/998866350 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 msgr2=0x7f21a00a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.074 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 rx=0x7f2198009390 tx=0x7f219800bd40).stop 2021-02-10T11:47:58.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 -- 172.21.2.117:0/998866350 shutdown_connections 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21840186c0 0x7f218401abc0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21a00a13a0 0x7f21a00a17a0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21a01a3930 0x7f21a00a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 --2- 172.21.2.117:0/998866350 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21a007c7c0 0x7f21a01a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 -- 172.21.2.117:0/998866350 >> 172.21.2.117:0/998866350 conn(0x7f21a0087090 msgr2=0x7f21a008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 -- 172.21.2.117:0/998866350 shutdown_connections 2021-02-10T11:47:58.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.075 7f21a5af2700 1 -- 172.21.2.117:0/998866350 wait complete. 2021-02-10T11:47:58.079 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:47:58.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.497 7fef4460c700 1 Processor -- start 2021-02-10T11:47:58.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.497 7fef4460c700 1 -- start start 2021-02-10T11:47:58.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef4460c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fef3c0d18f0 0x7fef3c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef4460c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0eb040 0x7fef3c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef4460c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c0eba60 con 0x7fef3c0ee3e0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef4460c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c0d1cf0 con 0x7fef3c0d18f0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef4460c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0eefd0 con 0x7fef3c0eb040 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42428700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fef3c0d18f0 0x7fef3c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42c29700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fef3c0ee3e0 0x7fef3c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45986/0 (socket says 172.21.2.117:45986) 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42c29700 1 -- 172.21.2.117:0/1281793216 learned_addr learned my addr 172.21.2.117:0/1281793216 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef41c27700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0eb040 0x7fef3c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42428700 1 -- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0eb040 msgr2=0x7fef3c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42428700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0eb040 0x7fef3c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42428700 1 -- 172.21.2.117:0/1281793216 >> v1:172.21.2.125:6789/0 conn(0x7fef3c0ee3e0 legacy=0x7fef3c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.498 7fef42428700 1 -- 172.21.2.117:0/1281793216 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0ef1f0 con 0x7fef3c0d18f0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef42428700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c0d18f0 0x7fef3c0eaa20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fef30006650 tx=0x7fef30009a00).ready entity=mon.1 client_cookie=e37eff2c3f4d0235 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef41426700 1 -- 172.21.2.117:0/1281793216 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef300009b0 con 0x7fef3c0d18f0 2021-02-10T11:47:58.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef41426700 1 -- 172.21.2.117:0/1281793216 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3000c0c0 con 0x7fef3c0d18f0 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef41426700 1 -- 172.21.2.117:0/1281793216 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef3000c5a0 con 0x7fef3c0d18f0 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef4460c700 1 -- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c0d18f0 msgr2=0x7fef3c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.499 7fef4460c700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c0d18f0 0x7fef3c0eaa20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fef30006650 tx=0x7fef30009a00).stop 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 -- 172.21.2.117:0/1281793216 shutdown_connections 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0eb040 0x7fef3c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 --2- 172.21.2.117:0/1281793216 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c0d18f0 0x7fef3c0eaa20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 -- 172.21.2.117:0/1281793216 >> 172.21.2.117:0/1281793216 conn(0x7fef3c0cf060 msgr2=0x7fef3c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 -- 172.21.2.117:0/1281793216 shutdown_connections 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.500 7fef4460c700 1 -- 172.21.2.117:0/1281793216 wait complete. 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.514 7fef4460c700 1 Processor -- start 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 -- start start 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 0x7fef3c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0a13a0 0x7fef3c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c07cca0 con 0x7fef3c1a3930 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c18fb70 con 0x7fef3c07c7c0 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef4460c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0dc6d0 con 0x7fef3c0a13a0 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef41c27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef41c27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52816/0 (socket says 172.21.2.117:52816) 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef41c27700 1 -- 172.21.2.117:0/2498035836 learned_addr learned my addr 172.21.2.117:0/2498035836 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef42428700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 0x7fef3c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.515 7fef42c29700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0a13a0 0x7fef3c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 -- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0a13a0 msgr2=0x7fef3c0a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0a13a0 0x7fef3c0a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 -- 172.21.2.117:0/2498035836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 msgr2=0x7fef3c0a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 -- 172.21.2.117:0/2498035836 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c07d220 con 0x7fef3c07c7c0 2021-02-10T11:47:58.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef41c27700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef42428700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 0x7fef3c1a3310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fef300096a0 tx=0x7fef3000b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef30006740 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3000bd00 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef300022b0 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef4460c700 1 -- 172.21.2.117:0/2498035836 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c07d220 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.516 7fef4460c700 1 -- 172.21.2.117:0/2498035836 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c07d220 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.517 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef30000c90 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.517 7fef2effd700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 0x7fef2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.517 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef300324b0 con 0x7fef3c07c7c0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.517 7fef41c27700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 0x7fef2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:58.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.517 7fef41c27700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 0x7fef2401abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:58.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.548 7fef4460c700 1 -- 172.21.2.117:0/2498035836 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c0daff0 con 0x7fef3c07c7c0 2021-02-10T11:47:58.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.552 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef30006740 con 0x7fef3c07c7c0 2021-02-10T11:47:58.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.681 7fef4460c700 1 -- 172.21.2.117:0/2498035836 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fef3c0daff0 con 0x7fef3c07c7c0 2021-02-10T11:47:58.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef2effd700 1 -- 172.21.2.117:0/2498035836 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7fef30000c90 con 0x7fef3c07c7c0 2021-02-10T11:47:58.678 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 -- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 msgr2=0x7fef2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 0x7fef2401abc0 crc :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 -- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 msgr2=0x7fef3c1a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 0x7fef3c1a3310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7fef300096a0 tx=0x7fef3000b910).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 -- 172.21.2.117:0/2498035836 shutdown_connections 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef240186c0 0x7fef2401abc0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef3c1a3930 0x7fef3c0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef3c0a13a0 0x7fef3c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.683 7fef4460c700 1 --2- 172.21.2.117:0/2498035836 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef3c07c7c0 0x7fef3c1a3310 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.684 7fef4460c700 1 -- 172.21.2.117:0/2498035836 >> 172.21.2.117:0/2498035836 conn(0x7fef3c087090 msgr2=0x7fef3c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.684 7fef4460c700 1 -- 172.21.2.117:0/2498035836 shutdown_connections 2021-02-10T11:47:58.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:58.684 7fef4460c700 1 -- 172.21.2.117:0/2498035836 wait complete. 2021-02-10T11:47:58.688 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:47:59.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.105 7f876d699700 1 Processor -- start 2021-02-10T11:47:59.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 -- start start 2021-02-10T11:47:59.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 0x7f87680eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87680eb040 0x7f87680eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680ede60 con 0x7f87680ee3e0 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680eba60 con 0x7f87680eb040 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.106 7f876d699700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680d1cf0 con 0x7f87680d18f0 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f87672bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 0x7f87680eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87680eb040 0x7f87680eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8767ac0700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f87680ee3e0 0x7f87680ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:45998/0 (socket says 172.21.2.117:45998) 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8767ac0700 1 -- 172.21.2.117:0/2280507519 learned_addr learned my addr 172.21.2.117:0/2280507519 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 -- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 msgr2=0x7f87680eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 0x7f87680eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 -- 172.21.2.117:0/2280507519 >> v1:172.21.2.125:6789/0 conn(0x7f87680ee3e0 legacy=0x7f87680ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 -- 172.21.2.117:0/2280507519 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87680eefd0 con 0x7f87680eb040 2021-02-10T11:47:59.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f87662bd700 1 -- 172.21.2.117:0/2280507519 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847956772 0 0) 0x7f8760001510 con 0x7f87680ee3e0 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f87672bf700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 0x7f87680eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.108 7f8766abe700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680eb040 0x7f87680eb440 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f8750006650 tx=0x7f8750009a80).ready entity=mon.1 client_cookie=c6222194bd0f494 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f87662bd700 1 -- 172.21.2.117:0/2280507519 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87500009b0 con 0x7f87680eb040 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f87662bd700 1 -- 172.21.2.117:0/2280507519 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875000c0c0 con 0x7f87680eb040 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f87662bd700 1 -- 172.21.2.117:0/2280507519 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f875000c5e0 con 0x7f87680eb040 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 -- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680eb040 msgr2=0x7f87680eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680eb040 0x7f87680eb440 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f8750006650 tx=0x7f8750009a80).stop 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 -- 172.21.2.117:0/2280507519 shutdown_connections 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680eb040 0x7f87680eb440 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 --2- 172.21.2.117:0/2280507519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680d18f0 0x7f87680eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 -- 172.21.2.117:0/2280507519 >> 172.21.2.117:0/2280507519 conn(0x7f87680cf060 msgr2=0x7f87680cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 -- 172.21.2.117:0/2280507519 shutdown_connections 2021-02-10T11:47:59.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.109 7f876d699700 1 -- 172.21.2.117:0/2280507519 wait complete. 2021-02-10T11:47:59.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 Processor -- start 2021-02-10T11:47:59.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 -- start start 2021-02-10T11:47:59.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680a6ce0 0x7f87680dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680a2040 0x7f87680a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680a67e0 con 0x7f8768085ec0 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87680a7120 con 0x7f87680a6ce0 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.124 7f876d699700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87680a2a60 con 0x7f87680a2040 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52826/0 (socket says 172.21.2.117:52826) 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 -- 172.21.2.117:0/2996544767 learned_addr learned my addr 172.21.2.117:0/2996544767 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 -- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680a2040 msgr2=0x7f87680a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680a2040 0x7f87680a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 -- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680a6ce0 msgr2=0x7f87680dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680a6ce0 0x7f87680dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 -- 172.21.2.117:0/2996544767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87681a2dc0 con 0x7f8768085ec0 2021-02-10T11:47:59.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f8766abe700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7f8750009630 tx=0x7f875000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.125 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87500093a0 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875000bd70 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8750005560 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f876d699700 1 -- 172.21.2.117:0/2996544767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87681a2dc0 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f876d699700 1 -- 172.21.2.117:0/2996544767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87681a3350 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f875000a000 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f875ffff700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 0x7f874c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.126 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8750032520 con 0x7f8768085ec0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.127 7f87672bf700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 0x7f874c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.127 7f87672bf700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 0x7f874c01abc0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.156 7f876d699700 1 -- 172.21.2.117:0/2996544767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680a2c80 con 0x7f8768085ec0 2021-02-10T11:47:59.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.158 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8750021b80 con 0x7f8768085ec0 2021-02-10T11:47:59.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.287 7f876d699700 1 -- 172.21.2.117:0/2996544767 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f87680a2c80 con 0x7f8768085ec0 2021-02-10T11:47:59.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.288 7f875ffff700 1 -- 172.21.2.117:0/2996544767 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8750021b80 con 0x7f8768085ec0 2021-02-10T11:47:59.284 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 msgr2=0x7f874c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 0x7f874c01abc0 crc :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 msgr2=0x7f87680862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 rx=0x7f8750009630 tx=0x7f875000b980).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 shutdown_connections 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f874c0186c0 0x7f874c01abc0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8768085ec0 0x7f87680862c0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87680a2040 0x7f87680a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 --2- 172.21.2.117:0/2996544767 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87680a6ce0 0x7f87680dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 >> 172.21.2.117:0/2996544767 conn(0x7f87680b6540 msgr2=0x7f876808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 shutdown_connections 2021-02-10T11:47:59.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.290 7f876d699700 1 -- 172.21.2.117:0/2996544767 wait complete. 2021-02-10T11:47:59.294 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:47:59.294 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 Processor -- start 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 -- start start 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d18e0 0x7f96c00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb000 0x7f96c00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f96c00ee3e0 0x7f96c00ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00ede60 con 0x7f96c00ee3e0 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00eba20 con 0x7f96c00eb000 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.709 7f96c5a17700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00d1ce0 con 0x7f96c00d18e0 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f96c00ee3e0 0x7f96c00ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f96c00ee3e0 0x7f96c00ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52834/0 (socket says 172.21.2.117:52834) 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 -- 172.21.2.117:0/228085550 learned_addr learned my addr 172.21.2.117:0/228085550 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 -- 172.21.2.117:0/228085550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d18e0 msgr2=0x7f96c00ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 --2- 172.21.2.117:0/228085550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d18e0 0x7f96c00ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 -- 172.21.2.117:0/228085550 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb000 msgr2=0x7f96c00eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 --2- 172.21.2.117:0/228085550 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb000 0x7f96c00eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 -- 172.21.2.117:0/228085550 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00eee00 con 0x7f96c00ee3e0 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96bffff700 1 --2- 172.21.2.117:0/228085550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ee3e0 0x7f96c00ee7e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7f96b8006f00 tx=0x7f96b8009f10).ready entity=mon.0 client_cookie=3035c5baf3ba8dce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96be7fc700 1 -- 172.21.2.117:0/228085550 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b800b3b0 con 0x7f96c00ee3e0 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96be7fc700 1 -- 172.21.2.117:0/228085550 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b800b630 con 0x7f96c00ee3e0 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96be7fc700 1 -- 172.21.2.117:0/228085550 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b800bb50 con 0x7f96c00ee3e0 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96c5a17700 1 -- 172.21.2.117:0/228085550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ee3e0 msgr2=0x7f96c00ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.710 7f96c5a17700 1 --2- 172.21.2.117:0/228085550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ee3e0 0x7f96c00ee7e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 rx=0x7f96b8006f00 tx=0x7f96b8009f10).stop 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 -- 172.21.2.117:0/228085550 shutdown_connections 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 --2- 172.21.2.117:0/228085550 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb000 0x7f96c00eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 --2- 172.21.2.117:0/228085550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ee3e0 0x7f96c00ee7e0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 --2- 172.21.2.117:0/228085550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00d18e0 0x7f96c00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 -- 172.21.2.117:0/228085550 >> 172.21.2.117:0/228085550 conn(0x7f96c00ced60 msgr2=0x7f96c00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 -- 172.21.2.117:0/228085550 shutdown_connections 2021-02-10T11:47:59.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.711 7f96c5a17700 1 -- 172.21.2.117:0/228085550 wait complete. 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.725 7f96c5a17700 1 Processor -- start 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 -- start start 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00e0cc0 0x7f96c00e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00e1f10 0x7f96c00df510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00e24f0 con 0x7f96c00ded50 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00e9020 con 0x7f96c00e0cc0 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96c5a17700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00e95a0 con 0x7f96c00e1f10 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96bf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96bf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52840/0 (socket says 172.21.2.117:52840) 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.726 7f96bf7fe700 1 -- 172.21.2.117:0/4165534474 learned_addr learned my addr 172.21.2.117:0/4165534474 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:47:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 -- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00e1f10 msgr2=0x7f96c00df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00e1f10 0x7f96c00df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 -- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00e0cc0 msgr2=0x7f96c00e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00e0cc0 0x7f96c00e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 -- 172.21.2.117:0/4165534474 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00dfb30 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96bf7fe700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f96a800b5c0 tx=0x7f96a800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a800bd50 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8006030 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c00dfb30 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c00e00c0 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.727 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96a80054d0 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.728 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96a8006210 con 0x7f96c00ded50 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.728 7f96b7fff700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 0x7f96a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:47:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.728 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f96a8032720 con 0x7f96c00ded50 2021-02-10T11:47:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.729 7f96beffd700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 0x7f96a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:47:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.730 7f96beffd700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 0x7f96a401abc0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:47:59.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.759 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c0087210 con 0x7f96c00ded50 2021-02-10T11:47:59.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.762 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96a802cf80 con 0x7f96c00ded50 2021-02-10T11:47:59.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.889 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c00ede60 con 0x7f96c00ded50 2021-02-10T11:47:59.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.890 7f96b7fff700 1 -- 172.21.2.117:0/4165534474 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f96a802cf80 con 0x7f96c00ded50 2021-02-10T11:47:59.886 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:47:59.887 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":68,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:47:44.486277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6539":{"gid":6539,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/4168849820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":4168849820},{"type":"v1","addr":"172.21.2.125:6835","nonce":4168849820}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:47:59.887 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 68 2021-02-10T11:47:59.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 msgr2=0x7f96a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 0x7f96a401abc0 crc :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 msgr2=0x7f96c00e12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f96a800b5c0 tx=0x7f96a800b960).stop 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 shutdown_connections 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00e1f10 0x7f96c00df510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00e0cc0 0x7f96c00e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a40186c0 0x7f96a401abc0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 --2- 172.21.2.117:0/4165534474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c00ded50 0x7f96c00e12d0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:47:59.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 >> 172.21.2.117:0/4165534474 conn(0x7f96c019f610 msgr2=0x7f96c0199b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:47:59.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 shutdown_connections 2021-02-10T11:47:59.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:47:59.892 7f96c5a17700 1 -- 172.21.2.117:0/4165534474 wait complete. 2021-02-10T11:48:00.097 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:47:42.357500', 'modified': '2021-02-10 11:47:44.486277', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6539}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6539': {'gid': 6539, 'name': 'b', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/4168849820', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 4168849820}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 4168849820}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:00.097 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:00.098 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6539, 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:48:00.496 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.140885439000002, 2021-02-10T11:48:00.497 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 68, 2021-02-10T11:48:00.497 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 91, 2021-02-10T11:48:00.497 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 91, 2021-02-10T11:48:00.497 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.771293475 2021-02-10T11:48:00.497 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:48:00.500 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6539, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.140885439000002, "mdsmap_epoch": 68, "osdmap_epoch": 91, "osdmap_epoch_barrier": 91, "uptime": 23.771293475 } 2021-02-10T11:48:00.500 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:48:00.500 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:48:00.500 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:48:00.501 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:00.501 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:00.836 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:01.180 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:01.180 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:01.524 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:01.524 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:01.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:01.525 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:01.861 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:48:01.861 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:48:01.862 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:01.864 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:02.208 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:02.208 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:02.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[150896]: starting ceph client 2021-02-10T11:48:02.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:48:02.279 7f113bc68f80 -1 init, newargv = 0x55d68877bf30 newargc=9 2021-02-10T11:48:02.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[150896]: starting fuse 2021-02-10T11:48:02.552 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:02.552 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:02.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:02.553 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:02.889 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:48:02.889 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:48:02.889 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:48:02.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:48:02.890 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:48:02.891 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:48:03.252 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.150896.asok 2021-02-10T11:48:03.255 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.150896.asok 2021-02-10T11:48:03.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:03.255 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.150896.asok status 2021-02-10T11:48:03.646 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:48:03.646 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:48:03.646 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:48:03.646 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:48:03.646 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "150896", 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 9915, 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:48:03.647 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 9915 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1766492164 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:03.648 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1766492164 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.9915 172.21.2.117:0/1766492164", 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1766492164", 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 68, 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 91, 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:48:03.649 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:48:03.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:03.650 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:03.651 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:03.986 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:48:03.987 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:03.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:03.988 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:04.335 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:48:04.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 Processor -- start 2021-02-10T11:48:04.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 -- start start 2021-02-10T11:48:04.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd07c0e5240 0x7fd07c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd07c084c60 0x7fd07c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c085930 con 0x7fd07c0e4e40 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c0e5cc0 con 0x7fd07c084c60 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.755 7fd084028700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c0856b0 con 0x7fd07c0e5240 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd07c084c60 0x7fd07c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd07c084c60 0x7fd07c085060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52862/0 (socket says 172.21.2.117:52862) 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 -- 172.21.2.117:0/243279188 learned_addr learned my addr 172.21.2.117:0/243279188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 -- 172.21.2.117:0/243279188 >> v2:172.21.2.125:3300/0 conn(0x7fd07c0e5240 msgr2=0x7fd07c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 --2- 172.21.2.117:0/243279188 >> v2:172.21.2.125:3300/0 conn(0x7fd07c0e5240 0x7fd07c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 -- 172.21.2.117:0/243279188 >> v1:172.21.2.117:6789/0 conn(0x7fd07c0e4e40 legacy=0x7fd07c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 -- 172.21.2.117:0/243279188 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c1b09e0 con 0x7fd07c084c60 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd082645700 1 --2- 172.21.2.117:0/243279188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c084c60 0x7fd07c085060 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7fd074006f00 tx=0x7fd074009f10).ready entity=mon.0 client_cookie=b7033b7b8e626ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd080e42700 1 -- 172.21.2.117:0/243279188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd07400b3b0 con 0x7fd07c084c60 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd080e42700 1 -- 172.21.2.117:0/243279188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07400b630 con 0x7fd07c084c60 2021-02-10T11:48:04.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd080e42700 1 -- 172.21.2.117:0/243279188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd07400bb50 con 0x7fd07c084c60 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd084028700 1 -- 172.21.2.117:0/243279188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c084c60 msgr2=0x7fd07c085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.756 7fd084028700 1 --2- 172.21.2.117:0/243279188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c084c60 0x7fd07c085060 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7fd074006f00 tx=0x7fd074009f10).stop 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 -- 172.21.2.117:0/243279188 shutdown_connections 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 --2- 172.21.2.117:0/243279188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c084c60 0x7fd07c085060 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 --2- 172.21.2.117:0/243279188 >> v2:172.21.2.125:3300/0 conn(0x7fd07c0e5240 0x7fd07c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 -- 172.21.2.117:0/243279188 >> 172.21.2.117:0/243279188 conn(0x7fd07c0ce4e0 msgr2=0x7fd07c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 -- 172.21.2.117:0/243279188 shutdown_connections 2021-02-10T11:48:04.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.757 7fd084028700 1 -- 172.21.2.117:0/243279188 wait complete. 2021-02-10T11:48:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 Processor -- start 2021-02-10T11:48:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 -- start start 2021-02-10T11:48:04.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd07c099bf0 0x7fd07c1ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd07c18e5b0 0x7fd07c18e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1ae8d0 con 0x7fd07c0b30d0 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0b3af0 con 0x7fd07c18e5b0 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd084028700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0b3d10 con 0x7fd07c099bf0 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd082645700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd082645700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52864/0 (socket says 172.21.2.117:52864) 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.772 7fd082645700 1 -- 172.21.2.117:0/3647402471 learned_addr learned my addr 172.21.2.117:0/3647402471 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 -- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd07c099bf0 msgr2=0x7fd07c1ae2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd07c099bf0 0x7fd07c1ae2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:04.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 -- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd07c18e5b0 msgr2=0x7fd07c18e9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd07c18e5b0 0x7fd07c18e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 -- 172.21.2.117:0/3647402471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0b3f30 con 0x7fd07c0b30d0 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd082645700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7fd07400b9b0 tx=0x7fd07400bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd074003030 con 0x7fd07c0b30d0 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07400c2f0 con 0x7fd07c0b30d0 2021-02-10T11:48:04.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd07400c810 con 0x7fd07c0b30d0 2021-02-10T11:48:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd084028700 1 -- 172.21.2.117:0/3647402471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c0b3f30 con 0x7fd07c0b30d0 2021-02-10T11:48:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.773 7fd084028700 1 -- 172.21.2.117:0/3647402471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c0b3f30 con 0x7fd07c0b30d0 2021-02-10T11:48:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.774 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd074009450 con 0x7fd07c0b30d0 2021-02-10T11:48:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.774 7fd0727fc700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 0x7fd06401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:04.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.774 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd074032a60 con 0x7fd07c0b30d0 2021-02-10T11:48:04.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.774 7fd081e44700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 0x7fd06401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:04.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.775 7fd081e44700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 0x7fd06401abc0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:04.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.805 7fd084028700 1 -- 172.21.2.117:0/3647402471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c18e080 con 0x7fd07c0b30d0 2021-02-10T11:48:04.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.808 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd074030a10 con 0x7fd07c0b30d0 2021-02-10T11:48:04.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.943 7fd084028700 1 -- 172.21.2.117:0/3647402471 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd07c18e080 con 0x7fd07c0b30d0 2021-02-10T11:48:04.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:04.998 7fd0727fc700 1 -- 172.21.2.117:0/3647402471 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd074030a10 con 0x7fd07c0b30d0 2021-02-10T11:48:05.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 -- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 msgr2=0x7fd06401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 0x7fd06401abc0 crc :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 -- 172.21.2.117:0/3647402471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 msgr2=0x7fd07c0b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 rx=0x7fd07400b9b0 tx=0x7fd07400bd50).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 -- 172.21.2.117:0/3647402471 shutdown_connections 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0640186c0 0x7fd06401abc0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd07c0b30d0 0x7fd07c0b34d0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd07c18e5b0 0x7fd07c18e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 --2- 172.21.2.117:0/3647402471 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd07c099bf0 0x7fd07c1ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.010 7fd084028700 1 -- 172.21.2.117:0/3647402471 >> 172.21.2.117:0/3647402471 conn(0x7fd07c1ad6b0 msgr2=0x7fd07c190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.011 7fd084028700 1 -- 172.21.2.117:0/3647402471 shutdown_connections 2021-02-10T11:48:05.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.011 7fd084028700 1 -- 172.21.2.117:0/3647402471 wait complete. 2021-02-10T11:48:05.013 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:48:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 Processor -- start 2021-02-10T11:48:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 -- start start 2021-02-10T11:48:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94741b07f0 0x7f94740e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94740e7d70 0x7f94740e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740e77f0 con 0x7f94740e8790 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94740e92f0 con 0x7f94740e7d70 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f947a07b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740e5ce0 con 0x7f94741b07f0 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f9472ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94740e7d70 0x7f94740e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f9472ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f94740e7d70 0x7f94740e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52872/0 (socket says 172.21.2.117:52872) 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f9472ffd700 1 -- 172.21.2.117:0/3295103639 learned_addr learned my addr 172.21.2.117:0/3295103639 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.429 7f9473fff700 1 --1- 172.21.2.117:0/3295103639 >> v1:172.21.2.117:6789/0 conn(0x7f94740e8790 0x7f94740e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53122/0 (socket says 172.21.2.117:53122) 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f9472ffd700 1 -- 172.21.2.117:0/3295103639 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94741b07f0 msgr2=0x7f94740e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f9472ffd700 1 --2- 172.21.2.117:0/3295103639 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94741b07f0 0x7f94740e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f9472ffd700 1 -- 172.21.2.117:0/3295103639 >> v1:172.21.2.117:6789/0 conn(0x7f94740e8790 legacy=0x7f94740e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f9472ffd700 1 -- 172.21.2.117:0/3295103639 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740e54b0 con 0x7f94740e7d70 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f9472ffd700 1 --2- 172.21.2.117:0/3295103639 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740e7d70 0x7f94740e8170 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f9468006700 tx=0x7f9468009b80).ready entity=mon.0 client_cookie=f5d065ce3259e779 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f94727fc700 1 -- 172.21.2.117:0/3295103639 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f946800c0c0 con 0x7f94740e7d70 2021-02-10T11:48:05.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f94727fc700 1 -- 172.21.2.117:0/3295103639 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946800c340 con 0x7f94740e7d70 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.430 7f94727fc700 1 -- 172.21.2.117:0/3295103639 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f946800c860 con 0x7f94740e7d70 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 -- 172.21.2.117:0/3295103639 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740e7d70 msgr2=0x7f94740e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 --2- 172.21.2.117:0/3295103639 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740e7d70 0x7f94740e8170 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 rx=0x7f9468006700 tx=0x7f9468009b80).stop 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 -- 172.21.2.117:0/3295103639 shutdown_connections 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 --2- 172.21.2.117:0/3295103639 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740e7d70 0x7f94740e8170 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 --2- 172.21.2.117:0/3295103639 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94741b07f0 0x7f94740e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 -- 172.21.2.117:0/3295103639 >> 172.21.2.117:0/3295103639 conn(0x7f94740ce500 msgr2=0x7f94740e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 -- 172.21.2.117:0/3295103639 shutdown_connections 2021-02-10T11:48:05.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.431 7f947a07b700 1 -- 172.21.2.117:0/3295103639 wait complete. 2021-02-10T11:48:05.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 Processor -- start 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 -- start start 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9474110bd0 0x7f9474110fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f947419d3e0 0x7f947419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94740a7760 con 0x7f94740a70c0 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947419de00 con 0x7f9474110bd0 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f947a07b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947419dfe0 con 0x7f947419d3e0 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f94737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f94737fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52876/0 (socket says 172.21.2.117:52876) 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.446 7f94737fe700 1 -- 172.21.2.117:0/3699977531 learned_addr learned my addr 172.21.2.117:0/3699977531 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 -- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f947419d3e0 msgr2=0x7f947419d7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:05.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f947419d3e0 0x7f947419d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 -- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9474110bd0 msgr2=0x7f9474110fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9474110bd0 0x7f9474110fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 -- 172.21.2.117:0/3699977531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947419e200 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f94737fe700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f9464006f00 tx=0x7f946400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9464006b90 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946400bdf0 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f947a07b700 1 -- 172.21.2.117:0/3699977531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947419e200 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9464002020 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.447 7f947a07b700 1 -- 172.21.2.117:0/3699977531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740a7fd0 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.448 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9464005310 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.448 7f945bfff700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 0x7f945401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.448 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9464032740 con 0x7f94740a70c0 2021-02-10T11:48:05.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.448 7f9472ffd700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 0x7f945401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:05.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.448 7f9472ffd700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 0x7f945401ac10 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:05.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.479 7f947a07b700 1 -- 172.21.2.117:0/3699977531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94741b8070 con 0x7f94740a70c0 2021-02-10T11:48:05.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.481 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9464000940 con 0x7f94740a70c0 2021-02-10T11:48:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.615 7f947a07b700 1 -- 172.21.2.117:0/3699977531 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f94740a7fd0 con 0x7f9454018730 2021-02-10T11:48:05.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.616 7f945bfff700 1 -- 172.21.2.117:0/3699977531 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f9468006500 con 0x7f9454018730 2021-02-10T11:48:05.613 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:48:05.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:05.613 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:48:05.613 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:05.613 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:48:05.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.619 7f947a07b700 1 -- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 msgr2=0x7f945401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.619 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 0x7f945401ac10 crc :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.619 7f947a07b700 1 -- 172.21.2.117:0/3699977531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 msgr2=0x7f94740a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.619 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7f9464006f00 tx=0x7f946400b940).stop 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 -- 172.21.2.117:0/3699977531 shutdown_connections 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f947419d3e0 0x7f947419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9474110bd0 0x7f9474110fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9454018730 0x7f945401ac10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 --2- 172.21.2.117:0/3699977531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94740a70c0 0x7f94740a6880 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:05.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 -- 172.21.2.117:0/3699977531 >> 172.21.2.117:0/3699977531 conn(0x7f94740bff40 msgr2=0x7f9474098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:05.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 -- 172.21.2.117:0/3699977531 shutdown_connections 2021-02-10T11:48:05.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:05.620 7f947a07b700 1 -- 172.21.2.117:0/3699977531 wait complete. 2021-02-10T11:48:05.623 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.042 7feda840c700 1 Processor -- start 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.042 7feda840c700 1 -- start start 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda840c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feda00d18f0 0x7feda00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda840c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00ef3f0 0x7feda01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda840c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00ef7f0 con 0x7feda00eb040 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda840c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda00eded0 con 0x7feda00d18f0 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda840c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda00d1cf0 con 0x7feda00ef3f0 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feda00d18f0 0x7feda00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda5a27700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7feda00eb040 0x7feda00eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53134/0 (socket says 172.21.2.117:53134) 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feda00d18f0 0x7feda00eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52884/0 (socket says 172.21.2.117:52884) 2021-02-10T11:48:06.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda5a27700 1 -- 172.21.2.117:0/2883965084 learned_addr learned my addr 172.21.2.117:0/2883965084 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 -- 172.21.2.117:0/2883965084 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00ef3f0 msgr2=0x7feda01b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 --2- 172.21.2.117:0/2883965084 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00ef3f0 0x7feda01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 -- 172.21.2.117:0/2883965084 >> v1:172.21.2.117:6789/0 conn(0x7feda00eb040 legacy=0x7feda00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 -- 172.21.2.117:0/2883965084 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda01b8fd0 con 0x7feda00d18f0 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.043 7feda6228700 1 --2- 172.21.2.117:0/2883965084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00d18f0 0x7feda00eaa20 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fed94006650 tx=0x7fed94009a80).ready entity=mon.0 client_cookie=81c5967b1f20c136 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda5226700 1 -- 172.21.2.117:0/2883965084 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed940009b0 con 0x7feda00d18f0 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda5226700 1 -- 172.21.2.117:0/2883965084 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9400c0c0 con 0x7feda00d18f0 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda5226700 1 -- 172.21.2.117:0/2883965084 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed9400c5e0 con 0x7feda00d18f0 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda840c700 1 -- 172.21.2.117:0/2883965084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00d18f0 msgr2=0x7feda00eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda840c700 1 --2- 172.21.2.117:0/2883965084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00d18f0 0x7feda00eaa20 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7fed94006650 tx=0x7fed94009a80).stop 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda840c700 1 -- 172.21.2.117:0/2883965084 shutdown_connections 2021-02-10T11:48:06.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda840c700 1 --2- 172.21.2.117:0/2883965084 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00ef3f0 0x7feda01b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.044 7feda840c700 1 --2- 172.21.2.117:0/2883965084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00d18f0 0x7feda00eaa20 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.045 7feda840c700 1 -- 172.21.2.117:0/2883965084 >> 172.21.2.117:0/2883965084 conn(0x7feda00cf060 msgr2=0x7feda00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.045 7feda840c700 1 -- 172.21.2.117:0/2883965084 shutdown_connections 2021-02-10T11:48:06.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.045 7feda840c700 1 -- 172.21.2.117:0/2883965084 wait complete. 2021-02-10T11:48:06.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.059 7feda840c700 1 Processor -- start 2021-02-10T11:48:06.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.059 7feda840c700 1 -- start start 2021-02-10T11:48:06.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00b3e10 0x7feda00b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda00b5010 0x7feda00ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda00b4320 con 0x7feda00b4c10 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda008a220 con 0x7feda00b5010 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda840c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda00abbc0 con 0x7feda00b3e10 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52892/0 (socket says 172.21.2.117:52892) 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 -- 172.21.2.117:0/870505935 learned_addr learned my addr 172.21.2.117:0/870505935 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 -- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00b3e10 msgr2=0x7feda00b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00b3e10 0x7feda00b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 -- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda00b5010 msgr2=0x7feda00ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda00b5010 0x7feda00ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 -- 172.21.2.117:0/870505935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00abde0 con 0x7feda00b4c10 2021-02-10T11:48:06.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7feda5a27700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fed9c001c60 tx=0x7fed9c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.060 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed9c00acc0 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed9c004140 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed9c004660 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7feda840c700 1 -- 172.21.2.117:0/870505935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda00abde0 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7feda840c700 1 -- 172.21.2.117:0/870505935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda00ac420 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fed9c004320 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7fed92ffd700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 0x7fed8801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.061 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fed9c0326e0 con 0x7feda00b4c10 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.062 7feda6228700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 0x7fed8801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.062 7feda6228700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 0x7fed8801abc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.093 7feda840c700 1 -- 172.21.2.117:0/870505935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda008a220 con 0x7feda00b4c10 2021-02-10T11:48:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.095 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fed9c000dd0 con 0x7feda00b4c10 2021-02-10T11:48:06.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.245 7feda840c700 1 -- 172.21.2.117:0/870505935 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7feda008a220 con 0x7feda00b4c10 2021-02-10T11:48:06.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.246 7fed92ffd700 1 -- 172.21.2.117:0/870505935 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fed9c004300 con 0x7feda00b4c10 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 msgr2=0x7fed8801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 0x7fed8801abc0 crc :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 msgr2=0x7feda00aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 rx=0x7fed9c001c60 tx=0x7fed9c00a8d0).stop 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 shutdown_connections 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed880186c0 0x7fed8801abc0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda00b4c10 0x7feda00aaf80 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda00b5010 0x7feda00ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 --2- 172.21.2.117:0/870505935 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda00b3e10 0x7feda00b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 >> 172.21.2.117:0/870505935 conn(0x7feda00b6c70 msgr2=0x7feda00970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 shutdown_connections 2021-02-10T11:48:06.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.248 7feda840c700 1 -- 172.21.2.117:0/870505935 wait complete. 2021-02-10T11:48:06.251 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000001012261 2021-02-10T11:48:06.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 Processor -- start 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 -- start start 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0d18f0 0x7f21fc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f21fc0eb040 0x7f21fc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0ede60 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc0eba60 con 0x7f21fc0eb040 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f2201bcd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0d1cf0 con 0x7f21fc0d18f0 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f21fbfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f21fc0ee3e0 0x7f21fc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53146/0 (socket says 172.21.2.117:53146) 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.671 7f21fbfff700 1 -- 172.21.2.117:0/2594092788 learned_addr learned my addr 172.21.2.117:0/2594092788 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435420193 0 0) 0x7f21f4001500 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e8002fe0 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fb7fe700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0d18f0 0x7f21fc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21faffd700 1 --2- 172.21.2.117:0/2594092788 >> v2:172.21.2.125:3300/0 conn(0x7f21fc0eb040 0x7f21fc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2017085661 0 0) 0x7f21f4001500 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0d18f0 msgr2=0x7f21fc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0d18f0 0x7f21fc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 >> v2:172.21.2.125:3300/0 conn(0x7f21fc0eb040 msgr2=0x7f21fc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 --2- 172.21.2.117:0/2594092788 >> v2:172.21.2.125:3300/0 conn(0x7f21fc0eb040 0x7f21fc0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0eefd0 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f21f4001a00 con 0x7f21fc0ee3e0 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 >> v1:172.21.2.117:6789/0 conn(0x7f21fc0ee3e0 legacy=0x7f21fc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21e8003ae0 0x7f21e8005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21e804a7a0 0x7f21e804cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21e804d5a0 0x7f21e804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e80502e0 con 0x7f21e804a7a0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8050500 con 0x7f21e8003ae0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e8050720 con 0x7f21e804d5a0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21faffd700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21e804a7a0 0x7f21e804cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.672 7f21fb7fe700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21e8003ae0 0x7f21e8005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fbfff700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21e804d5a0 0x7f21e804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 -- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21e804d5a0 msgr2=0x7f21e804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21e804d5a0 0x7f21e804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 -- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21e8003ae0 msgr2=0x7f21e8005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21e8003ae0 0x7f21e8005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 -- 172.21.2.117:0/2594092788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e80509c0 con 0x7f21e804a7a0 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fbfff700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21e804d5a0 0x7f21e804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:06.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21faffd700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21e804a7a0 0x7f21e804cca0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f21e400c8f0 tx=0x7f21e400cc90).ready entity=mon.0 client_cookie=cab3d87f9fc933c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e400cdd0 con 0x7f21e804a7a0 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4009310 con 0x7f21e804a7a0 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fa7fc700 1 -- 172.21.2.117:0/2594092788 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21e4009830 con 0x7f21e804a7a0 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f2201bcd700 1 -- 172.21.2.117:0/2594092788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21e804a7a0 msgr2=0x7f21e804cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f2201bcd700 1 --2- 172.21.2.117:0/2594092788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21e804a7a0 0x7f21e804cca0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f21e400c8f0 tx=0x7f21e400cc90).stop 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.673 7f21fbfff700 1 -- 172.21.2.117:0/2594092788 reap_dead start 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.674 7f2201bcd700 1 -- 172.21.2.117:0/2594092788 shutdown_connections 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.674 7f2201bcd700 1 -- 172.21.2.117:0/2594092788 >> 172.21.2.117:0/2594092788 conn(0x7f21fc0cf060 msgr2=0x7f21fc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.674 7f2201bcd700 1 -- 172.21.2.117:0/2594092788 shutdown_connections 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.674 7f2201bcd700 1 -- 172.21.2.117:0/2594092788 wait complete. 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 Processor -- start 2021-02-10T11:48:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 -- start start 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0a6ce0 0x7f21fc0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21fc085ec0 0x7f21fc0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0a67e0 con 0x7f21fc0a2040 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0a7120 con 0x7f21fc085ec0 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f2201bcd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0a2a60 con 0x7f21fc0a6ce0 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f21fbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f21fbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52906/0 (socket says 172.21.2.117:52906) 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.689 7f21fbfff700 1 -- 172.21.2.117:0/122381950 learned_addr learned my addr 172.21.2.117:0/122381950 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 -- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0a6ce0 msgr2=0x7f21fc0dc420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0a6ce0 0x7f21fc0dc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 -- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21fc085ec0 msgr2=0x7f21fc0862c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21fc085ec0 0x7f21fc0862c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 -- 172.21.2.117:0/122381950 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc1a2dc0 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21fbfff700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f21f400b940 tx=0x7f21f400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21f4007a50 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f4006880 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21f40062b0 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f2201bcd700 1 -- 172.21.2.117:0/122381950 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc1a2dc0 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.690 7f2201bcd700 1 -- 172.21.2.117:0/122381950 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc1a3350 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.691 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f21f4004720 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.691 7f21f3fff700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 0x7f21e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.691 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f21f4032a40 con 0x7f21fc0a2040 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.691 7f21fb7fe700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 0x7f21e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:06.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.691 7f21fb7fe700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 0x7f21e001abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:06.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.721 7f2201bcd700 1 -- 172.21.2.117:0/122381950 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc0daff0 con 0x7f21fc0a2040 2021-02-10T11:48:06.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.723 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f21f4026380 con 0x7f21fc0a2040 2021-02-10T11:48:06.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.850 7f2201bcd700 1 -- 172.21.2.117:0/122381950 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000001012261"}) v1 -- 0x7f21fc1a3350 con 0x7f21e00186c0 2021-02-10T11:48:06.849 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:48:06.858 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:06.849 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:48:06.858 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:06.849 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:48:06.858 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:06.849 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:48:06.858 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:06.849 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:48:06.858 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:48:06.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.868 7f21f3fff700 1 -- 172.21.2.117:0/122381950 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f21ec0009b0 con 0x7f21e00186c0 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 -- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 msgr2=0x7f21e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 0x7f21e001abc0 crc :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 -- 172.21.2.117:0/122381950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 msgr2=0x7f21fc0a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 rx=0x7f21f400b940 tx=0x7f21f400bce0).stop 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 -- 172.21.2.117:0/122381950 shutdown_connections 2021-02-10T11:48:06.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f21e00186c0 0x7f21e001abc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21fc0a2040 0x7f21fc0a2440 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21fc085ec0 0x7f21fc0862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 --2- 172.21.2.117:0/122381950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21fc0a6ce0 0x7f21fc0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.883 7f2201bcd700 1 -- 172.21.2.117:0/122381950 >> 172.21.2.117:0/122381950 conn(0x7f21fc0b6540 msgr2=0x7f21fc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.884 7f2201bcd700 1 -- 172.21.2.117:0/122381950 shutdown_connections 2021-02-10T11:48:06.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:06.884 7f2201bcd700 1 -- 172.21.2.117:0/122381950 wait complete. 2021-02-10T11:48:06.887 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000212731 --group_name group_0000000001012261 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 Processor -- start 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 -- start start 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f279c0e5240 0x7f279c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f279c084c60 0x7f279c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c085930 con 0x7f279c0e4e40 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c0e5cc0 con 0x7f279c0e5240 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f27a1e4d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c0856b0 con 0x7f279c084c60 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f279affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f279c0e5240 0x7f279c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f279b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f279c0e4e40 0x7f279c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53162/0 (socket says 172.21.2.117:53162) 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.308 7f279b7fe700 1 -- 172.21.2.117:0/3024110509 learned_addr learned my addr 172.21.2.117:0/3024110509 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279affd700 1 -- 172.21.2.117:0/3024110509 >> v2:172.21.2.125:3300/0 conn(0x7f279c084c60 msgr2=0x7f279c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279affd700 1 --2- 172.21.2.117:0/3024110509 >> v2:172.21.2.125:3300/0 conn(0x7f279c084c60 0x7f279c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279affd700 1 -- 172.21.2.117:0/3024110509 >> v1:172.21.2.117:6789/0 conn(0x7f279c0e4e40 legacy=0x7f279c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279affd700 1 -- 172.21.2.117:0/3024110509 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c1b09e0 con 0x7f279c0e5240 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279affd700 1 --2- 172.21.2.117:0/3024110509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0e5240 0x7f279c086ec0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f2790006650 tx=0x7f2790009a80).ready entity=mon.0 client_cookie=44ea870c6ee7bc67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279a7fc700 1 -- 172.21.2.117:0/3024110509 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279000c0c0 con 0x7f279c0e5240 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279a7fc700 1 -- 172.21.2.117:0/3024110509 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279000c340 con 0x7f279c0e5240 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f279a7fc700 1 -- 172.21.2.117:0/3024110509 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f279000c860 con 0x7f279c0e5240 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f27a1e4d700 1 -- 172.21.2.117:0/3024110509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0e5240 msgr2=0x7f279c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.309 7f27a1e4d700 1 --2- 172.21.2.117:0/3024110509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0e5240 0x7f279c086ec0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 rx=0x7f2790006650 tx=0x7f2790009a80).stop 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 -- 172.21.2.117:0/3024110509 shutdown_connections 2021-02-10T11:48:07.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 --2- 172.21.2.117:0/3024110509 >> v2:172.21.2.125:3300/0 conn(0x7f279c084c60 0x7f279c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 --2- 172.21.2.117:0/3024110509 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0e5240 0x7f279c086ec0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 -- 172.21.2.117:0/3024110509 >> 172.21.2.117:0/3024110509 conn(0x7f279c0ce4e0 msgr2=0x7f279c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 -- 172.21.2.117:0/3024110509 shutdown_connections 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.310 7f27a1e4d700 1 -- 172.21.2.117:0/3024110509 wait complete. 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 Processor -- start 2021-02-10T11:48:07.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 -- start start 2021-02-10T11:48:07.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f279c093070 0x7f279c09fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279c1ae8d0 0x7f279c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c09f540 con 0x7f279c0a00e0 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c093470 con 0x7f279c1ae8d0 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f27a1e4d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c09ecd0 con 0x7f279c093070 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f279affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.325 7f279affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52918/0 (socket says 172.21.2.117:52918) 2021-02-10T11:48:07.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 -- 172.21.2.117:0/3258871606 learned_addr learned my addr 172.21.2.117:0/3258871606 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 -- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f279c093070 msgr2=0x7f279c09fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f279c093070 0x7f279c09fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 -- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279c1ae8d0 msgr2=0x7f279c09e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279c1ae8d0 0x7f279c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 -- 172.21.2.117:0/3258871606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c09eef0 con 0x7f279c0a00e0 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f279affd700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f2790009630 tx=0x7f279000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27900093a0 con 0x7f279c0a00e0 2021-02-10T11:48:07.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279000bd70 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c09eef0 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c1b20a0 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.326 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2790005560 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.327 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f279000a000 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.327 7f2783fff700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 0x7f277c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.327 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2790032920 con 0x7f279c0a00e0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.327 7f279b7fe700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 0x7f277c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.328 7f279b7fe700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 0x7f277c01abd0 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.358 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279c10c2f0 con 0x7f279c0a00e0 2021-02-10T11:48:07.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.361 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2790009340 con 0x7f279c0a00e0 2021-02-10T11:48:07.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.484 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000212731", "target": ["mgr", ""], "group_name": "group_0000000001012261"}) v1 -- 0x7f279c093470 con 0x7f277c0186d0 2021-02-10T11:48:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.511 7f2783fff700 1 -- 172.21.2.117:0/3258871606 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f278c000e20 con 0x7f277c0186d0 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 msgr2=0x7f277c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 0x7f277c01abd0 crc :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 msgr2=0x7f279c1ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7f2790009630 tx=0x7f279000b980).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 shutdown_connections 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c0186d0 0x7f277c01abd0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f279c0a00e0 0x7f279c1ae2b0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279c1ae8d0 0x7f279c09e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 --2- 172.21.2.117:0/3258871606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f279c093070 0x7f279c09fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.517 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 >> 172.21.2.117:0/3258871606 conn(0x7f279c1ad6b0 msgr2=0x7f279c190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.518 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 shutdown_connections 2021-02-10T11:48:07.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.518 7f27a1e4d700 1 -- 172.21.2.117:0/3258871606 wait complete. 2021-02-10T11:48:07.521 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000212731 guest1 --group_name group_0000000001012261 --tenant_id tenant1 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.948 7f88dd4c5700 1 Processor -- start 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.948 7f88dd4c5700 1 -- start start 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88dd4c5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 0x7f88d80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88dd4c5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 0x7f88d81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88dd4c5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d80ede60 con 0x7f88d80eb040 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88dd4c5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d8088020 con 0x7f88d80ef3f0 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88dd4c5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d80d1d30 con 0x7f88d80d18f0 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88d6abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f88d80eb040 0x7f88d80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53176/0 (socket says 172.21.2.117:53176) 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88d6abe700 1 -- 172.21.2.117:0/2964733097 learned_addr learned my addr 172.21.2.117:0/2964733097 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:07.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88d7ac0700 1 --2- 172.21.2.117:0/2964733097 >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 0x7f88d81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88d72bf700 1 --2- 172.21.2.117:0/2964733097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 0x7f88d80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.949 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125892530 0 0) 0x7f88cc0009b0 con 0x7f88d80eb040 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c0002fe0 con 0x7f88d80eb040 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d72bf700 1 -- 172.21.2.117:0/2964733097 >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 msgr2=0x7f88d81b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1175681739 0 0) 0x7f88cc0010b0 con 0x7f88d80eb040 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d72bf700 1 --2- 172.21.2.117:0/2964733097 >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 0x7f88d81b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d72bf700 1 -- 172.21.2.117:0/2964733097 >> v1:172.21.2.117:6789/0 conn(0x7f88d80eb040 legacy=0x7f88d80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d72bf700 1 -- 172.21.2.117:0/2964733097 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d81b8fd0 con 0x7f88d80d18f0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d7ac0700 1 --2- 172.21.2.117:0/2964733097 >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 0x7f88d81b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d72bf700 1 --2- 172.21.2.117:0/2964733097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 0x7f88d80eaa20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f88c8006650 tx=0x7f88c8009a80).ready entity=mon.2 client_cookie=377ff65a060e003e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c80009b0 con 0x7f88d80d18f0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800c0c0 con 0x7f88d80d18f0 2021-02-10T11:48:07.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88d62bd700 1 -- 172.21.2.117:0/2964733097 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c800c490 con 0x7f88d80d18f0 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88dd4c5700 1 -- 172.21.2.117:0/2964733097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 msgr2=0x7f88d80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.950 7f88dd4c5700 1 --2- 172.21.2.117:0/2964733097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 0x7f88d80eaa20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f88c8006650 tx=0x7f88c8009a80).stop 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 -- 172.21.2.117:0/2964733097 shutdown_connections 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 --2- 172.21.2.117:0/2964733097 >> v2:172.21.2.125:3300/0 conn(0x7f88d80ef3f0 0x7f88d81b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 --2- 172.21.2.117:0/2964733097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80d18f0 0x7f88d80eaa20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 -- 172.21.2.117:0/2964733097 >> 172.21.2.117:0/2964733097 conn(0x7f88d80cf060 msgr2=0x7f88d80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 -- 172.21.2.117:0/2964733097 shutdown_connections 2021-02-10T11:48:07.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.951 7f88dd4c5700 1 -- 172.21.2.117:0/2964733097 wait complete. 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.965 7f88dd4c5700 1 Processor -- start 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.965 7f88dd4c5700 1 -- start start 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88d80a9b70 0x7f88d81a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80a9f70 0x7f88d81a7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8097490 con 0x7f88d80bac40 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d81a7ea0 con 0x7f88d80a9b70 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88dd4c5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d81a80c0 con 0x7f88d80a9f70 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d72bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d72bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52932/0 (socket says 172.21.2.117:52932) 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d72bf700 1 -- 172.21.2.117:0/3632954873 learned_addr learned my addr 172.21.2.117:0/3632954873 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d6abe700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88d80a9b70 0x7f88d81a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d7ac0700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80a9f70 0x7f88d81a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d72bf700 1 -- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80a9f70 msgr2=0x7f88d81a7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.966 7f88d72bf700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80a9f70 0x7f88d81a7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88d72bf700 1 -- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88d80a9b70 msgr2=0x7f88d81a7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88d72bf700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88d80a9b70 0x7f88d81a7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88d72bf700 1 -- 172.21.2.117:0/3632954873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d8097a10 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88d72bf700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f88c8009630 tx=0x7f88c800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c80093a0 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800bd70 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d8097a10 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88c8005560 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.967 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d8097a10 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.968 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88c80052e0 con 0x7f88d80bac40 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.968 7f88bffff700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 0x7f88b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:07.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.968 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f88c8032550 con 0x7f88d80bac40 2021-02-10T11:48:07.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.968 7f88d6abe700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 0x7f88b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:07.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.968 7f88d6abe700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 0x7f88b801abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:07.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:07.998 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d80ede60 con 0x7f88d80bac40 2021-02-10T11:48:07.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.001 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88c8025fb0 con 0x7f88d80bac40 2021-02-10T11:48:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.126 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000212731", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000001012261"}) v1 -- 0x7f88d81a7ea0 con 0x7f88b80186c0 2021-02-10T11:48:08.126 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:48:08.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.161 7f88bffff700 1 -- 172.21.2.117:0/3632954873 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f88cc000940 con 0x7f88b80186c0 2021-02-10T11:48:08.185 INFO:teuthology.orchestra.run.gibba017.stdout:AQD4xyNgNEmGCBAAZCKSOn5QYVZ+mlwzwGAfpg== 2021-02-10T11:48:08.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 msgr2=0x7f88b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 0x7f88b801abc0 crc :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 msgr2=0x7f88d80a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 rx=0x7f88c8009630 tx=0x7f88c800b980).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 shutdown_connections 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88d80a9f70 0x7f88d81a7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88d80a9b70 0x7f88d81a7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88b80186c0 0x7f88b801abc0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 --2- 172.21.2.117:0/3632954873 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88d80bac40 0x7f88d80a9550 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 >> 172.21.2.117:0/3632954873 conn(0x7f88d80b19e0 msgr2=0x7f88d80aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 shutdown_connections 2021-02-10T11:48:08.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.192 7f88dd4c5700 1 -- 172.21.2.117:0/3632954873 wait complete. 2021-02-10T11:48:08.195 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000212731 --group_name group_0000000001012261 2021-02-10T11:48:08.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.614 7f6e1b114700 1 Processor -- start 2021-02-10T11:48:08.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.614 7f6e1b114700 1 -- start start 2021-02-10T11:48:08.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e1b114700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e140e7170 0x7f6e140e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e1b114700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e140e7570 con 0x7f6e141b07e0 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e1b114700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e140e77e0 con 0x7f6e140e8380 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e1b114700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e140e5cd0 con 0x7f6e140e7170 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e18f30700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6e141b07e0 0x7f6e140e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53190/0 (socket says 172.21.2.117:53190) 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e18f30700 1 -- 172.21.2.117:0/2779491991 learned_addr learned my addr 172.21.2.117:0/2779491991 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e13fff700 1 --2- 172.21.2.117:0/2779491991 >> v2:172.21.2.125:3300/0 conn(0x7f6e140e7170 0x7f6e140e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.615 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461335224 0 0) 0x7f6dfc001180 con 0x7f6e141b07e0 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e00002fe0 con 0x7f6e141b07e0 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671532758 0 0) 0x7f6e0c001080 con 0x7f6e140e8380 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e000042c0 con 0x7f6e140e8380 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e13fff700 1 -- 172.21.2.117:0/2779491991 >> v1:172.21.2.125:6789/0 conn(0x7f6e140e8380 legacy=0x7f6e140e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e13fff700 1 -- 172.21.2.117:0/2779491991 >> v1:172.21.2.117:6789/0 conn(0x7f6e141b07e0 legacy=0x7f6e140e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e13fff700 1 -- 172.21.2.117:0/2779491991 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e140e8f70 con 0x7f6e140e7170 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e13fff700 1 --2- 172.21.2.117:0/2779491991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140e7170 0x7f6e140e7d60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f6e040069d0 tx=0x7f6e04009e40).ready entity=mon.1 client_cookie=9dd03fe72ee85cd7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0400b510 con 0x7f6e140e7170 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0400b890 con 0x7f6e140e7170 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e137fe700 1 -- 172.21.2.117:0/2779491991 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0400bd70 con 0x7f6e140e7170 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e1b114700 1 -- 172.21.2.117:0/2779491991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140e7170 msgr2=0x7f6e140e7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.616 7f6e1b114700 1 --2- 172.21.2.117:0/2779491991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140e7170 0x7f6e140e7d60 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f6e040069d0 tx=0x7f6e04009e40).stop 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.617 7f6e1b114700 1 -- 172.21.2.117:0/2779491991 shutdown_connections 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.617 7f6e1b114700 1 --2- 172.21.2.117:0/2779491991 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140e7170 0x7f6e140e7d60 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.617 7f6e1b114700 1 -- 172.21.2.117:0/2779491991 >> 172.21.2.117:0/2779491991 conn(0x7f6e140ce4f0 msgr2=0x7f6e140e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.617 7f6e1b114700 1 -- 172.21.2.117:0/2779491991 shutdown_connections 2021-02-10T11:48:08.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.617 7f6e1b114700 1 -- 172.21.2.117:0/2779491991 wait complete. 2021-02-10T11:48:08.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 Processor -- start 2021-02-10T11:48:08.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 -- start start 2021-02-10T11:48:08.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 0x7f6e140a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 0x7f6e1407fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140807e0 con 0x7f6e140803e0 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141a51f0 con 0x7f6e140a09f0 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e1b114700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140aeb60 con 0x7f6e141a4d90 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e19731700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.632 7f6e18f30700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 0x7f6e140a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52948/0 (socket says 172.21.2.117:52948) 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e18f30700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 0x7f6e140a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57330/0 (socket says 172.21.2.117:57330) 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 -- 172.21.2.117:0/1760066617 learned_addr learned my addr 172.21.2.117:0/1760066617 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e13fff700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 0x7f6e1407fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 -- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 msgr2=0x7f6e1407fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 0x7f6e1407fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 -- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 msgr2=0x7f6e140a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 0x7f6e140a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 -- 172.21.2.117:0/1760066617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14195b50 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e13fff700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 0x7f6e1407fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e19731700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f6e0c00b5b0 tx=0x7f6e0c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c000bc0 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c006020 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14195b50 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e0c0054a0 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.633 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14195b50 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.634 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e0c0056b0 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.634 7f6e117fa700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 0x7f6df801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.634 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6e0c032790 con 0x7f6e140803e0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.634 7f6e18f30700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 0x7f6df801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:08.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.635 7f6e18f30700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 0x7f6df801abd0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:08.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.665 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1407e8b0 con 0x7f6e140803e0 2021-02-10T11:48:08.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.667 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e0c0054a0 con 0x7f6e140803e0 2021-02-10T11:48:08.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.796 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000212731", "target": ["mgr", ""], "group_name": "group_0000000001012261"}) v1 -- 0x7f6e1407e8b0 con 0x7f6df80186d0 2021-02-10T11:48:08.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.799 7f6e117fa700 1 -- 172.21.2.117:0/1760066617 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+96 (crc 0 0 0) 0x7f6dfc007770 con 0x7f6df80186d0 2021-02-10T11:48:08.798 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/group_0000000001012261/subvolume_0000000000212731/2238504f-d9a6-4026-82ce-0e4ebc82c731 2021-02-10T11:48:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.805 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 msgr2=0x7f6df801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.805 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 0x7f6df801abd0 crc :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.805 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 msgr2=0x7f6e14195530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:08.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.805 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 rx=0x7f6e0c00b5b0 tx=0x7f6e0c00b950).stop 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 shutdown_connections 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6df80186d0 0x7f6df801abd0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e140803e0 0x7f6e14195530 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e141a4d90 0x7f6e1407fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 --2- 172.21.2.117:0/1760066617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e140a09f0 0x7f6e140a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 >> 172.21.2.117:0/1760066617 conn(0x7f6e141998a0 msgr2=0x7f6e141aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 shutdown_connections 2021-02-10T11:48:08.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:08.806 7f6e1b114700 1 -- 172.21.2.117:0/1760066617 wait complete. 2021-02-10T11:48:08.808 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.guest1 mds 'allow rw path=/volumes/group_0000000001012261, allow rw path=/volumes/group_0000000001012261/subvolume_0000000000212731/2238504f-d9a6-4026-82ce-0e4ebc82c731' osd 'allow rw pool=cephfs_data' mon 'allow r' mgr 'allow *' 2021-02-10T11:48:09.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 Processor -- start 2021-02-10T11:48:09.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 -- start start 2021-02-10T11:48:09.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 0x7f1be410ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be41b0d40 con 0x7f1be410a040 2021-02-10T11:48:09.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be4109ac0 con 0x7f1be40e53a0 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be9f63700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be40e57a0 con 0x7f1be40d1880 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be2ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1be410a040 0x7f1be410a440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53204/0 (socket says 172.21.2.117:53204) 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be2ffd700 1 -- 172.21.2.117:0/3232568146 learned_addr learned my addr 172.21.2.117:0/3232568146 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.238 7f1be3fff700 1 --2- 172.21.2.117:0/3232568146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 0x7f1be410ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756927865 0 0) 0x7f1bdc001060 con 0x7f1be410a040 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bcc002fe0 con 0x7f1be410a040 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be3fff700 1 -- 172.21.2.117:0/3232568146 >> v1:172.21.2.125:6789/0 conn(0x7f1be40e53a0 legacy=0x7f1be41b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 752237431 0 0) 0x7f1bdc001460 con 0x7f1be410a040 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be3fff700 1 -- 172.21.2.117:0/3232568146 >> v1:172.21.2.117:6789/0 conn(0x7f1be410a040 legacy=0x7f1be410a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be3fff700 1 -- 172.21.2.117:0/3232568146 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be40d1c80 con 0x7f1be40d1880 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be3fff700 1 --2- 172.21.2.117:0/3232568146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 0x7f1be410ac30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1bd4006ad0 tx=0x7f1bd4009ad0).ready entity=mon.2 client_cookie=e749a61d8b516cf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bd40066a0 con 0x7f1be40d1880 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd4006410 con 0x7f1be40d1880 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be27fc700 1 -- 172.21.2.117:0/3232568146 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bd40024d0 con 0x7f1be40d1880 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be9f63700 1 -- 172.21.2.117:0/3232568146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 msgr2=0x7f1be410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.239 7f1be9f63700 1 --2- 172.21.2.117:0/3232568146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 0x7f1be410ac30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1bd4006ad0 tx=0x7f1bd4009ad0).stop 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.240 7f1be9f63700 1 -- 172.21.2.117:0/3232568146 shutdown_connections 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.240 7f1be9f63700 1 --2- 172.21.2.117:0/3232568146 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be40d1880 0x7f1be410ac30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.240 7f1be9f63700 1 -- 172.21.2.117:0/3232568146 >> 172.21.2.117:0/3232568146 conn(0x7f1be40ce4b0 msgr2=0x7f1be40e45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.240 7f1be9f63700 1 -- 172.21.2.117:0/3232568146 shutdown_connections 2021-02-10T11:48:09.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.240 7f1be9f63700 1 -- 172.21.2.117:0/3232568146 wait complete. 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 Processor -- start 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 -- start start 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be419e1e0 0x7f1be419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1be41b7830 0x7f1be40adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be41b7e00 con 0x7f1be41b7430 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be41b5040 con 0x7f1be41b7830 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be9f63700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be40ae2e0 con 0x7f1be419e1e0 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52964/0 (socket says 172.21.2.117:52964) 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.255 7f1be2ffd700 1 -- 172.21.2.117:0/2499827016 learned_addr learned my addr 172.21.2.117:0/2499827016 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 -- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be419e1e0 msgr2=0x7f1be419e780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be419e1e0 0x7f1be419e780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 -- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1be41b7830 msgr2=0x7f1be40adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1be41b7830 0x7f1be40adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 -- 172.21.2.117:0/2499827016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be41b8380 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be2ffd700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f1bdc00a920 tx=0x7f1bdc00acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bdc004930 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bdc001eb0 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be41b8380 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be41b8380 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1bdc007020 con 0x7f1be41b7430 2021-02-10T11:48:09.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.256 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1bdc004720 con 0x7f1be41b7430 2021-02-10T11:48:09.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.257 7f1bcbfff700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 0x7f1bc401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.257 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1bdc032ce0 con 0x7f1be41b7430 2021-02-10T11:48:09.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.257 7f1be37fe700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 0x7f1bc401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.257 7f1be37fe700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 0x7f1bc401ac10 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.288 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be41037b0 con 0x7f1be41b7430 2021-02-10T11:48:09.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.290 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1bdc001e90 con 0x7f1be41b7430 2021-02-10T11:48:09.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.419 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000001012261, allow rw path=/volumes/group_0000000001012261/subvolume_0000000000212731/2238504f-d9a6-4026-82ce-0e4ebc82c731", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]} v 0) v1 -- 0x7f1be41037b0 con 0x7f1be41b7430 2021-02-10T11:48:09.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.422 7f1bcbfff700 1 -- 172.21.2.117:0/2499827016 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.guest1", "caps": ["mds", "allow rw path=/volumes/group_0000000001012261, allow rw path=/volumes/group_0000000001012261/subvolume_0000000000212731/2238504f-d9a6-4026-82ce-0e4ebc82c731", "osd", "allow rw pool=cephfs_data", "mon", "allow r", "mgr", "allow *"]}]=0 updated caps for client.guest1 v56) v1 ==== 363+0+0 (secure 0 0 0) 0x7f1bdc004720 con 0x7f1be41b7430 2021-02-10T11:48:09.421 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.guest1 2021-02-10T11:48:09.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 msgr2=0x7f1bc401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 0x7f1bc401ac10 crc :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 msgr2=0x7f1be40ad6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 rx=0x7f1bdc00a920 tx=0x7f1bdc00acc0).stop 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 shutdown_connections 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1bc4018730 0x7f1bc401ac10 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1be41b7430 0x7f1be40ad6a0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1be41b7830 0x7f1be40adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 --2- 172.21.2.117:0/2499827016 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1be419e1e0 0x7f1be419e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 >> 172.21.2.117:0/2499827016 conn(0x7f1be419f8d0 msgr2=0x7f1be419aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 shutdown_connections 2021-02-10T11:48:09.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.428 7f1be9f63700 1 -- 172.21.2.117:0/2499827016 wait complete. 2021-02-10T11:48:09.431 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000212731 guest1 --group_name group_0000000001012261 2021-02-10T11:48:09.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 Processor -- start 2021-02-10T11:48:09.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 -- start start 2021-02-10T11:48:09.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f40a40d18e0 0x7f40a40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40a40eb000 0x7f40a40eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a40ede60 con 0x7f40a40ee3e0 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40eba20 con 0x7f40a40eb000 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.852 7f40aa8eb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40d1ce0 con 0x7f40a40d18e0 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40a40eb000 0x7f40a40eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a8f08700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f40a40ee3e0 0x7f40a40ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53218/0 (socket says 172.21.2.117:53218) 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f40a40eb000 0x7f40a40eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52970/0 (socket says 172.21.2.117:52970) 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a8f08700 1 -- 172.21.2.117:0/3399531563 learned_addr learned my addr 172.21.2.117:0/3399531563 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a37fe700 1 -- 172.21.2.117:0/3399531563 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069611764 0 0) 0x7f409c001500 con 0x7f40a40ee3e0 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a37fe700 1 -- 172.21.2.117:0/3399531563 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4090002fe0 con 0x7f40a40ee3e0 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 -- 172.21.2.117:0/3399531563 >> v2:172.21.2.125:3300/0 conn(0x7f40a40d18e0 msgr2=0x7f40a40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 --2- 172.21.2.117:0/3399531563 >> v2:172.21.2.125:3300/0 conn(0x7f40a40d18e0 0x7f40a40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 -- 172.21.2.117:0/3399531563 >> v1:172.21.2.117:6789/0 conn(0x7f40a40ee3e0 legacy=0x7f40a40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 -- 172.21.2.117:0/3399531563 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40eefd0 con 0x7f40a40eb000 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f409bfff700 1 --2- 172.21.2.117:0/3399531563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a40eb000 0x7f40a40eb400 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f4094006650 tx=0x7f4094009a80).ready entity=mon.0 client_cookie=adf6d1fa16ce5fe6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a37fe700 1 -- 172.21.2.117:0/3399531563 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40940009b0 con 0x7f40a40eb000 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a37fe700 1 -- 172.21.2.117:0/3399531563 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409400c0c0 con 0x7f40a40eb000 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40a37fe700 1 -- 172.21.2.117:0/3399531563 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f409400c520 con 0x7f40a40eb000 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.853 7f40aa8eb700 1 -- 172.21.2.117:0/3399531563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a40eb000 msgr2=0x7f40a40eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 --2- 172.21.2.117:0/3399531563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a40eb000 0x7f40a40eb400 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f4094006650 tx=0x7f4094009a80).stop 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 -- 172.21.2.117:0/3399531563 shutdown_connections 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 --2- 172.21.2.117:0/3399531563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a40eb000 0x7f40a40eb400 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 --2- 172.21.2.117:0/3399531563 >> v2:172.21.2.125:3300/0 conn(0x7f40a40d18e0 0x7f40a40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 -- 172.21.2.117:0/3399531563 >> 172.21.2.117:0/3399531563 conn(0x7f40a40ced60 msgr2=0x7f40a40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:09.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 -- 172.21.2.117:0/3399531563 shutdown_connections 2021-02-10T11:48:09.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.854 7f40aa8eb700 1 -- 172.21.2.117:0/3399531563 wait complete. 2021-02-10T11:48:09.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 Processor -- start 2021-02-10T11:48:09.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 -- start start 2021-02-10T11:48:09.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40a4091130 0x7f40a4091530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40a4092330 0x7f40a4092730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4091b50 con 0x7f40a408ad80 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4091db0 con 0x7f40a4091130 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40aa8eb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a41a3e50 con 0x7f40a4092330 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52972/0 (socket says 172.21.2.117:52972) 2021-02-10T11:48:09.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.869 7f40a3fff700 1 -- 172.21.2.117:0/3519124545 learned_addr learned my addr 172.21.2.117:0/3519124545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 -- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40a4092330 msgr2=0x7f40a4092730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40a4092330 0x7f40a4092730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 -- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40a4091130 msgr2=0x7f40a4091530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40a4091130 0x7f40a4091530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 -- 172.21.2.117:0/3519124545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a419a550 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a3fff700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f408c006f00 tx=0x7f408c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f408c00bd30 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c000c90 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f408c0022a0 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a419a550 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.870 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a419a550 con 0x7f40a408ad80 2021-02-10T11:48:09.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.871 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f408c005310 con 0x7f40a408ad80 2021-02-10T11:48:09.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.871 7f40a17fa700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:09.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.871 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f408c032690 con 0x7f40a408ad80 2021-02-10T11:48:09.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.871 7f409bfff700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:09.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.872 7f409bfff700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 0x7f408801abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:09.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.902 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a40ede60 con 0x7f40a408ad80 2021-02-10T11:48:09.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:09.905 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f408c006b90 con 0x7f40a408ad80 2021-02-10T11:48:10.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.030 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000212731", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000001012261"}) v1 -- 0x7f40a4091b50 con 0x7f40880186c0 2021-02-10T11:48:10.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.066 7f40a17fa700 1 -- 172.21.2.117:0/3519124545 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4094000d10 con 0x7f40880186c0 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.095 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 msgr2=0x7f408801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.095 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 0x7f408801abc0 crc :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.095 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 msgr2=0x7f40a418fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.095 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 rx=0x7f408c006f00 tx=0x7f408c00b940).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 shutdown_connections 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40a4092330 0x7f40a4092730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40a4091130 0x7f40a4091530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40880186c0 0x7f408801abc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 --2- 172.21.2.117:0/3519124545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40a408ad80 0x7f40a418fc00 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 >> 172.21.2.117:0/3519124545 conn(0x7f40a419b1c0 msgr2=0x7f40a40ddfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 shutdown_connections 2021-02-10T11:48:10.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.096 7f40aa8eb700 1 -- 172.21.2.117:0/3519124545 wait complete. 2021-02-10T11:48:10.099 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth get client.guest1 --format=json-pretty 2021-02-10T11:48:10.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 Processor -- start 2021-02-10T11:48:10.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 -- start start 2021-02-10T11:48:10.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7300e4e40 0x7fb7301b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb730085450 0x7fb7300d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb730085930 con 0x7fb7300e5240 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7300e5cc0 con 0x7fb730085450 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb7355bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb730084840 con 0x7fb7300e4e40 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb72fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb730085450 0x7fb7300d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb72fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb730085450 0x7fb7300d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52980/0 (socket says 172.21.2.117:52980) 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.516 7fb72fac0700 1 -- 172.21.2.117:0/2069195246 learned_addr learned my addr 172.21.2.117:0/2069195246 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72fac0700 1 -- 172.21.2.117:0/2069195246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7300e4e40 msgr2=0x7fb7301b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72fac0700 1 --2- 172.21.2.117:0/2069195246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7300e4e40 0x7fb7301b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72fac0700 1 -- 172.21.2.117:0/2069195246 >> v1:172.21.2.125:6789/0 conn(0x7fb7300e5240 legacy=0x7fb730084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72fac0700 1 -- 172.21.2.117:0/2069195246 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7300d1be0 con 0x7fb730085450 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72fac0700 1 --2- 172.21.2.117:0/2069195246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb730085450 0x7fb7300d05b0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7fb728006f00 tx=0x7fb728009f10).ready entity=mon.0 client_cookie=3eb6f4683ff4e356 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72e2bd700 1 -- 172.21.2.117:0/2069195246 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72800b3b0 con 0x7fb730085450 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72e2bd700 1 -- 172.21.2.117:0/2069195246 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72800b730 con 0x7fb730085450 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.517 7fb72e2bd700 1 -- 172.21.2.117:0/2069195246 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72800bc50 con 0x7fb730085450 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 -- 172.21.2.117:0/2069195246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb730085450 msgr2=0x7fb7300d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 --2- 172.21.2.117:0/2069195246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb730085450 0x7fb7300d05b0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7fb728006f00 tx=0x7fb728009f10).stop 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 -- 172.21.2.117:0/2069195246 shutdown_connections 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 --2- 172.21.2.117:0/2069195246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb730085450 0x7fb7300d05b0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 --2- 172.21.2.117:0/2069195246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7300e4e40 0x7fb7301b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 -- 172.21.2.117:0/2069195246 >> 172.21.2.117:0/2069195246 conn(0x7fb7300ce4e0 msgr2=0x7fb7300e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 -- 172.21.2.117:0/2069195246 shutdown_connections 2021-02-10T11:48:10.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.518 7fb7355bf700 1 -- 172.21.2.117:0/2069195246 wait complete. 2021-02-10T11:48:10.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 Processor -- start 2021-02-10T11:48:10.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 -- start start 2021-02-10T11:48:10.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 0x7fb73009e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7301afe20 0x7fb73009e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7301b17d0 con 0x7fb73009f460 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73009ef10 con 0x7fb7301afa20 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.533 7fb7355bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73009f0f0 con 0x7fb7301afe20 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:52984/0 (socket says 172.21.2.117:52984) 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 -- 172.21.2.117:0/58090292 learned_addr learned my addr 172.21.2.117:0/58090292 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 -- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7301afe20 msgr2=0x7fb73009e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7301afe20 0x7fb73009e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 -- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 msgr2=0x7fb73009e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72eabe700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 0x7fb73009e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 0x7fb73009e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 -- 172.21.2.117:0/58090292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7301b03d0 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72eabe700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 0x7fb73009e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.534 7fb72f2bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7fb72000b530 tx=0x7fb72000b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb72000bcc0 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb7355bf700 1 -- 172.21.2.117:0/58090292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7301b03d0 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb7355bf700 1 -- 172.21.2.117:0/58090292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7301b0960 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000bcc0 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb720000c90 con 0x7fb73009f460 2021-02-10T11:48:10.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb720002230 con 0x7fb73009f460 2021-02-10T11:48:10.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 0x7fb71001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:10.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.535 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb720032650 con 0x7fb73009f460 2021-02-10T11:48:10.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.536 7fb72eabe700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 0x7fb71001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:10.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.536 7fb72eabe700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 0x7fb71001abc0 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.566 7fb7355bf700 1 -- 172.21.2.117:0/58090292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb708002fd0 con 0x7fb73009f460 2021-02-10T11:48:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.569 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb720002020 con 0x7fb73009f460 2021-02-10T11:48:10.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.701 7fb7355bf700 1 -- 172.21.2.117:0/58090292 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"} v 0) v1 -- 0x7fb708002ef0 con 0x7fb73009f460 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.702 7fb717fff700 1 -- 172.21.2.117:0/58090292 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.guest1", "format": "json-pretty"}]=0 exported keyring for client.guest1 v57) v1 ==== 142+0+235 (secure 0 0 0) 0x7fb720002230 con 0x7fb73009f460 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.guest1", 2021-02-10T11:48:10.697 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQD4xyNgNEmGCBAAZCKSOn5QYVZ+mlwzwGAfpg==", 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow rw path=/volumes/group_0000000001012261", 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *" 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:48:10.698 INFO:teuthology.orchestra.run.gibba017.stderr:exported keyring for client.guest1 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.703 7fb7355bf700 1 -- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 msgr2=0x7fb71001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.703 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 0x7fb71001abc0 crc :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.703 7fb7355bf700 1 -- 172.21.2.117:0/58090292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 msgr2=0x7fb730194ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.703 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 rx=0x7fb72000b530 tx=0x7fb72000b8d0).stop 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 -- 172.21.2.117:0/58090292 shutdown_connections 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7301afe20 0x7fb73009e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:10.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7301afa20 0x7fb73009e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7100186c0 0x7fb71001abc0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 --2- 172.21.2.117:0/58090292 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb73009f460 0x7fb730194ed0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:10.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 -- 172.21.2.117:0/58090292 >> 172.21.2.117:0/58090292 conn(0x7fb7301a6900 msgr2=0x7fb73018c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:10.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 -- 172.21.2.117:0/58090292 shutdown_connections 2021-02-10T11:48:10.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:10.704 7fb7355bf700 1 -- 172.21.2.117:0/58090292 wait complete. 2021-02-10T11:48:10.708 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.123 7f448f16c700 1 Processor -- start 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.123 7f448f16c700 1 -- start start 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448f16c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f44880e7180 0x7f44880e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448f16c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44880e7580 con 0x7f44881b07f0 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448f16c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44880e77f0 con 0x7f44880e8390 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448f16c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44880e5ce0 con 0x7f44880e7180 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448cf88700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f44881b07f0 0x7f44880e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53242/0 (socket says 172.21.2.117:53242) 2021-02-10T11:48:11.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f448cf88700 1 -- 172.21.2.117:0/3709746123 learned_addr learned my addr 172.21.2.117:0/3709746123 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f4487fff700 1 --2- 172.21.2.117:0/3709746123 >> v2:172.21.2.125:3300/0 conn(0x7f44880e7180 0x7f44880e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f44877fe700 1 -- 172.21.2.117:0/3709746123 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474698931 0 0) 0x7f4470000ce0 con 0x7f44881b07f0 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.124 7f44877fe700 1 -- 172.21.2.117:0/3709746123 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4474002fe0 con 0x7f44881b07f0 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f4487fff700 1 -- 172.21.2.117:0/3709746123 >> v1:172.21.2.125:6789/0 conn(0x7f44880e8390 legacy=0x7f44880e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f4487fff700 1 -- 172.21.2.117:0/3709746123 >> v1:172.21.2.117:6789/0 conn(0x7f44881b07f0 legacy=0x7f44880e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f4487fff700 1 -- 172.21.2.117:0/3709746123 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880e8f80 con 0x7f44880e7180 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f4487fff700 1 --2- 172.21.2.117:0/3709746123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880e7180 0x7f44880e7d70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f44780069d0 tx=0x7f4478009e40).ready entity=mon.1 client_cookie=f75f709058344b26 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f44877fe700 1 -- 172.21.2.117:0/3709746123 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f447800b510 con 0x7f44880e7180 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f44877fe700 1 -- 172.21.2.117:0/3709746123 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447800b790 con 0x7f44880e7180 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f44877fe700 1 -- 172.21.2.117:0/3709746123 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f447800bcb0 con 0x7f44880e7180 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f448f16c700 1 -- 172.21.2.117:0/3709746123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880e7180 msgr2=0x7f44880e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.125 7f448f16c700 1 --2- 172.21.2.117:0/3709746123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880e7180 0x7f44880e7d70 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f44780069d0 tx=0x7f4478009e40).stop 2021-02-10T11:48:11.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.126 7f448f16c700 1 -- 172.21.2.117:0/3709746123 shutdown_connections 2021-02-10T11:48:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.126 7f448f16c700 1 --2- 172.21.2.117:0/3709746123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880e7180 0x7f44880e7d70 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.126 7f448f16c700 1 -- 172.21.2.117:0/3709746123 >> 172.21.2.117:0/3709746123 conn(0x7f44880ce500 msgr2=0x7f44880e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.126 7f448f16c700 1 -- 172.21.2.117:0/3709746123 shutdown_connections 2021-02-10T11:48:11.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.126 7f448f16c700 1 -- 172.21.2.117:0/3709746123 wait complete. 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.140 7f448f16c700 1 Processor -- start 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.140 7f448f16c700 1 -- start start 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4488109830 0x7f44880a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44880a3600 0x7f44880a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488109c30 con 0x7f4488109830 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448810c400 con 0x7f44880a4950 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448f16c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4488109ee0 con 0x7f44880a3600 2021-02-10T11:48:11.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57380/0 (socket says 172.21.2.117:57380) 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 -- 172.21.2.117:0/3143758222 learned_addr learned my addr 172.21.2.117:0/3143758222 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448cf88700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4488109830 0x7f44880a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f4487fff700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44880a3600 0x7f44880a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 -- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44880a3600 msgr2=0x7f44880a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44880a3600 0x7f44880a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 -- 172.21.2.117:0/3143758222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4488109830 msgr2=0x7f44880a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4488109830 0x7f44880a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.141 7f448d789700 1 -- 172.21.2.117:0/3143758222 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880a3a00 con 0x7f44880a4950 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f448d789700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f4480006f10 tx=0x7f448000b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4480006030 con 0x7f44880a4950 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44800062f0 con 0x7f44880a4950 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f448f16c700 1 -- 172.21.2.117:0/3143758222 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44880a3a00 con 0x7f44880a4950 2021-02-10T11:48:11.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f448f16c700 1 -- 172.21.2.117:0/3143758222 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44880a5660 con 0x7f44880a4950 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.142 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44800056e0 con 0x7f44880a4950 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.143 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44800054f0 con 0x7f44880a4950 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.143 7f44857fa700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 0x7f446c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.143 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4480032710 con 0x7f44880a4950 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.143 7f448cf88700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 0x7f446c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.143 7f448cf88700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 0x7f446c01abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.174 7f448f16c700 1 -- 172.21.2.117:0/3143758222 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4488109c30 con 0x7f44880a4950 2021-02-10T11:48:11.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.178 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f448002ad20 con 0x7f44880a4950 2021-02-10T11:48:11.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.310 7f448f16c700 1 -- 172.21.2.117:0/3143758222 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f4488109c30 con 0x7f44880a4950 2021-02-10T11:48:11.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.315 7f44857fa700 1 -- 172.21.2.117:0/3143758222 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 updated v58) v1 ==== 89+0+0 (secure 0 0 0) 0x7f44800061d0 con 0x7f44880a4950 2021-02-10T11:48:11.312 INFO:teuthology.orchestra.run.gibba017.stderr:updated 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 msgr2=0x7f446c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 0x7f446c01abc0 crc :-1 s=READY pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 msgr2=0x7f44880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f4480006f10 tx=0x7f448000b990).stop 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 shutdown_connections 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44880a4950 0x7f44880a4d50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44880a3600 0x7f44880a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f446c0186c0 0x7f446c01abc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 --2- 172.21.2.117:0/3143758222 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4488109830 0x7f44880a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 >> 172.21.2.117:0/3143758222 conn(0x7f4488111150 msgr2=0x7f4488187d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 shutdown_connections 2021-02-10T11:48:11.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.319 7f448f16c700 1 -- 172.21.2.117:0/3143758222 wait complete. 2021-02-10T11:48:11.322 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000212731 --group_name group_0000000001012261 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 Processor -- start 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 -- start start 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 0x7f5ea41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5ea40e5240 0x7f5ea4084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea4085930 con 0x7f5ea41b0810 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea4087430 con 0x7f5ea40e5240 2021-02-10T11:48:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5eab958700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40e5cc0 con 0x7f5ea40e4e40 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5ea9f75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5ea41b0810 0x7f5ea4085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53254/0 (socket says 172.21.2.117:53254) 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.744 7f5ea9f75700 1 -- 172.21.2.117:0/2603732201 learned_addr learned my addr 172.21.2.117:0/2603732201 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 --2- 172.21.2.117:0/2603732201 >> v2:172.21.2.125:3300/0 conn(0x7f5ea40e5240 0x7f5ea4084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea9774700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 0x7f5ea41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5e97fff700 1 -- 172.21.2.117:0/2603732201 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254568674 0 0) 0x7f5e9c000f90 con 0x7f5ea41b0810 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 -- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 msgr2=0x7f5ea41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 0x7f5ea41b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 -- 172.21.2.117:0/2603732201 >> v1:172.21.2.117:6789/0 conn(0x7f5ea41b0810 legacy=0x7f5ea4085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 -- 172.21.2.117:0/2603732201 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4084840 con 0x7f5ea40e5240 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea9774700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 0x7f5ea41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5ea8f73700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40e5240 0x7f5ea4084c60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f5e98006650 tx=0x7f5e98009a80).ready entity=mon.1 client_cookie=bee89a10b6160cee server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5e97fff700 1 -- 172.21.2.117:0/2603732201 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e98006390 con 0x7f5ea40e5240 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5e97fff700 1 -- 172.21.2.117:0/2603732201 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e980009b0 con 0x7f5ea40e5240 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5e97fff700 1 -- 172.21.2.117:0/2603732201 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e98006390 con 0x7f5ea40e5240 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5eab958700 1 -- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40e5240 msgr2=0x7f5ea4084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.745 7f5eab958700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40e5240 0x7f5ea4084c60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f5e98006650 tx=0x7f5e98009a80).stop 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 -- 172.21.2.117:0/2603732201 shutdown_connections 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40e5240 0x7f5ea4084c60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 --2- 172.21.2.117:0/2603732201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40e4e40 0x7f5ea41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 -- 172.21.2.117:0/2603732201 >> 172.21.2.117:0/2603732201 conn(0x7f5ea40ce4e0 msgr2=0x7f5ea40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 -- 172.21.2.117:0/2603732201 shutdown_connections 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.746 7f5eab958700 1 -- 172.21.2.117:0/2603732201 wait complete. 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 Processor -- start 2021-02-10T11:48:11.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 -- start start 2021-02-10T11:48:11.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 0x7f5ea4191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40b6c00 0x7f5ea40b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4191ab0 con 0x7f5ea40c14d0 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea41916a0 con 0x7f5ea40c2000 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.761 7f5eab958700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4191880 con 0x7f5ea40b6c00 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53010/0 (socket says 172.21.2.117:53010) 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 -- 172.21.2.117:0/2962178934 learned_addr learned my addr 172.21.2.117:0/2962178934 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea9774700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 0x7f5ea4191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea9f75700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40b6c00 0x7f5ea40b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 -- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40b6c00 msgr2=0x7f5ea40b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40b6c00 0x7f5ea40b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 -- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 msgr2=0x7f5ea4191080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 0x7f5ea4191080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 -- 172.21.2.117:0/2962178934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea40b7620 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea9774700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 0x7f5ea4191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5ea8f73700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f5e9800b660 tx=0x7f5e9800ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e980009b0 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98006760 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.762 7f5eab958700 1 -- 172.21.2.117:0/2962178934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea40b7620 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.763 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e98002030 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.763 7f5eab958700 1 -- 172.21.2.117:0/2962178934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea40b7bb0 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.763 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e98000c90 con 0x7f5ea40c14d0 2021-02-10T11:48:11.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.763 7f5e95ffb700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 0x7f5e8401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.763 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e98032910 con 0x7f5ea40c14d0 2021-02-10T11:48:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.764 7f5ea9774700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 0x7f5e8401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:11.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.764 7f5ea9774700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 0x7f5e8401ac10 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:11.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.793 7f5eab958700 1 -- 172.21.2.117:0/2962178934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4191880 con 0x7f5ea40c14d0 2021-02-10T11:48:11.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.796 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e98006740 con 0x7f5ea40c14d0 2021-02-10T11:48:11.858 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:11.861 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:11.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.925 7f5eab958700 1 -- 172.21.2.117:0/2962178934 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000212731", "target": ["mgr", ""], "group_name": "group_0000000001012261"}) v1 -- 0x7f5ea40b7bb0 con 0x7f5e84018730 2021-02-10T11:48:11.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.934 7f5e95ffb700 1 -- 172.21.2.117:0/2962178934 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f5ea00009b0 con 0x7f5e84018730 2021-02-10T11:48:11.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.942 7f5eab958700 1 -- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 msgr2=0x7f5e8401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.942 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 0x7f5e8401ac10 crc :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.942 7f5eab958700 1 -- 172.21.2.117:0/2962178934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 msgr2=0x7f5ea40c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:11.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.942 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 rx=0x7f5e9800b660 tx=0x7f5e9800ba00).stop 2021-02-10T11:48:11.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 -- 172.21.2.117:0/2962178934 shutdown_connections 2021-02-10T11:48:11.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e84018730 0x7f5e8401ac10 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ea40c14d0 0x7f5ea40c18d0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ea40b6c00 0x7f5ea40b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 --2- 172.21.2.117:0/2962178934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ea40c2000 0x7f5ea4191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:11.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 -- 172.21.2.117:0/2962178934 >> 172.21.2.117:0/2962178934 conn(0x7f5ea40acec0 msgr2=0x7f5ea40b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:11.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 -- 172.21.2.117:0/2962178934 shutdown_connections 2021-02-10T11:48:11.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:11.943 7f5eab958700 1 -- 172.21.2.117:0/2962178934 wait complete. 2021-02-10T11:48:11.947 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000001012261 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.280 7ff8ccc4c700 1 Processor -- start 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.280 7ff8ccc4c700 1 -- start start 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80d18f0 0x7ff8c80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c80ede60 con 0x7ff8c80eb040 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c8088020 con 0x7ff8c80ef3f0 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c80d1d30 con 0x7ff8c80d18f0 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53018/0 (socket says 172.21.2.117:53018) 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 -- 172.21.2.117:0/3877107565 learned_addr learned my addr 172.21.2.117:0/3877107565 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c685e700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80d18f0 0x7ff8c80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 -- 172.21.2.117:0/3877107565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80d18f0 msgr2=0x7ff8c80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80d18f0 0x7ff8c80eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 -- 172.21.2.117:0/3877107565 >> v1:172.21.2.125:6789/0 conn(0x7ff8c80eb040 legacy=0x7ff8c80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 -- 172.21.2.117:0/3877107565 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c81b8fd0 con 0x7ff8c80ef3f0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c705f700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7ff8c0006f00 tx=0x7ff8c0009f10).ready entity=mon.0 client_cookie=89dbbe18cbcdd588 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c585c700 1 -- 172.21.2.117:0/3877107565 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8c000b3b0 con 0x7ff8c80ef3f0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c585c700 1 -- 172.21.2.117:0/3877107565 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c000b630 con 0x7ff8c80ef3f0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8c585c700 1 -- 172.21.2.117:0/3877107565 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8c000bb10 con 0x7ff8c80ef3f0 2021-02-10T11:48:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 -- 172.21.2.117:0/3877107565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c80ef3f0 msgr2=0x7ff8c81b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.281 7ff8ccc4c700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 rx=0x7ff8c0006f00 tx=0x7ff8c0009f10).stop 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 -- 172.21.2.117:0/3877107565 shutdown_connections 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c80ef3f0 0x7ff8c81b79a0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 --2- 172.21.2.117:0/3877107565 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80d18f0 0x7ff8c80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 -- 172.21.2.117:0/3877107565 >> 172.21.2.117:0/3877107565 conn(0x7ff8c80cf060 msgr2=0x7ff8c80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 -- 172.21.2.117:0/3877107565 shutdown_connections 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.282 7ff8ccc4c700 1 -- 172.21.2.117:0/3877107565 wait complete. 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 Processor -- start 2021-02-10T11:48:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 -- start start 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80a6ce0 0x7ff8c80dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c80a2040 0x7ff8c80a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c80a67e0 con 0x7ff8c8085ec0 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c80a7120 con 0x7ff8c80a2040 2021-02-10T11:48:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8ccc4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c80a2a60 con 0x7ff8c80a6ce0 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8c605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8c605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53024/0 (socket says 172.21.2.117:53024) 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8c605d700 1 -- 172.21.2.117:0/1700341380 learned_addr learned my addr 172.21.2.117:0/1700341380 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.297 7ff8c705f700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c80a2040 0x7ff8c80a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c685e700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80a6ce0 0x7ff8c80dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 -- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80a6ce0 msgr2=0x7ff8c80dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80a6ce0 0x7ff8c80dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 -- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c80a2040 msgr2=0x7ff8c80a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c80a2040 0x7ff8c80a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 -- 172.21.2.117:0/1700341380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c81a2dc0 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8c605d700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7ff8b800b5b0 tx=0x7ff8b800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8b800bd40 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b8000ce0 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8b80054a0 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c81a2dc0 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.298 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c81a3350 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.299 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8b8006030 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.299 7ff8bf7fe700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 0x7ff8ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.299 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8b8032560 con 0x7ff8c8085ec0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.299 7ff8c685e700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 0x7ff8ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.299 7ff8c685e700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 0x7ff8ac01abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:12.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.330 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c80a2c80 con 0x7ff8c8085ec0 2021-02-10T11:48:12.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.333 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8b8000990 con 0x7ff8c8085ec0 2021-02-10T11:48:12.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.463 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000001012261"}) v1 -- 0x7ff8c81a3be0 con 0x7ff8ac0186c0 2021-02-10T11:48:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.466 7ff8bf7fe700 1 -- 172.21.2.117:0/1700341380 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff8b0006a70 con 0x7ff8ac0186c0 2021-02-10T11:48:12.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.471 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 msgr2=0x7ff8ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.471 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 0x7ff8ac01abc0 crc :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.471 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 msgr2=0x7ff8c80862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:12.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.471 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 rx=0x7ff8b800b5b0 tx=0x7ff8b800b950).stop 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 shutdown_connections 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8ac0186c0 0x7ff8ac01abc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c8085ec0 0x7ff8c80862c0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c80a2040 0x7ff8c80a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 --2- 172.21.2.117:0/1700341380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c80a6ce0 0x7ff8c80dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 >> 172.21.2.117:0/1700341380 conn(0x7ff8c80b6540 msgr2=0x7ff8c808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 shutdown_connections 2021-02-10T11:48:12.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:12.472 7ff8ccc4c700 1 -- 172.21.2.117:0/1700341380 wait complete. 2021-02-10T11:48:12.475 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:48:12.475 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:12.821 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:13.165 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:48:13.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:13.166 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:13.166 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:13.501 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:48:13.502 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:13.502 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:48:13.502 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:48:13.502 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:13.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[150254]: fuse finished with error 0 and tester_r 0 2021-02-10T11:48:13.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:13.865 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:13.866 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:14.201 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:48:14.202 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:48:14.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:14.202 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:14.539 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:14.539 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:14.539 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:14.875 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:48:14.875 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:14.875 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:48:14.875 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:48:14.875 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:15.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[150896]: fuse finished with error 0 and tester_r 0 2021-02-10T11:48:15.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:15.227 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:15.227 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:15.562 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:48:15.562 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:48:15.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:15.563 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:15.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:15.898 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:15.898 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:16.232 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:48:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:16.233 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:48:16.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:16.233 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:16.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:16.567 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:16.567 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:16.902 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:48:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:16.903 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:48:16.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:48:16.903 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:17.236 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:48:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.658 7f6cc303c700 1 Processor -- start 2021-02-10T11:48:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 -- start start 2021-02-10T11:48:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 0x7f6cbc0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6cbc0e7d70 0x7f6cbc0e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc0e77f0 con 0x7f6cbc0e8790 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cbc0e92f0 con 0x7f6cbc0e7d70 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.659 7f6cc303c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc0e5ce0 con 0x7f6cbc1b07f0 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.660 7f6cc0e58700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 0x7f6cbc0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.660 7f6cbbfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6cbc0e7d70 0x7f6cbc0e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.660 7f6cc1659700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6cbc0e8790 0x7f6cbc0e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46204/0 (socket says 172.21.2.117:46204) 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.660 7f6cc1659700 1 -- 172.21.2.117:0/1209969352 learned_addr learned my addr 172.21.2.117:0/1209969352 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbb7fe700 1 -- 172.21.2.117:0/1209969352 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436576674 0 0) 0x7f6cb4001510 con 0x7f6cbc0e8790 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbbfff700 1 -- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 msgr2=0x7f6cbc0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbbfff700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 0x7f6cbc0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbbfff700 1 -- 172.21.2.117:0/1209969352 >> v1:172.21.2.125:6789/0 conn(0x7f6cbc0e8790 legacy=0x7f6cbc0e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbbfff700 1 -- 172.21.2.117:0/1209969352 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0e54b0 con 0x7f6cbc0e7d70 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cc0e58700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 0x7f6cbc0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbbfff700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc0e7d70 0x7f6cbc0e8170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f6cac006650 tx=0x7f6cac009a80).ready entity=mon.1 client_cookie=e7381254079a0797 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbb7fe700 1 -- 172.21.2.117:0/1209969352 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cac0009b0 con 0x7f6cbc0e7d70 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbb7fe700 1 -- 172.21.2.117:0/1209969352 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cac00c0c0 con 0x7f6cbc0e7d70 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cbb7fe700 1 -- 172.21.2.117:0/1209969352 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cac00c510 con 0x7f6cbc0e7d70 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cc303c700 1 -- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc0e7d70 msgr2=0x7f6cbc0e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.661 7f6cc303c700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc0e7d70 0x7f6cbc0e8170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f6cac006650 tx=0x7f6cac009a80).stop 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 -- 172.21.2.117:0/1209969352 shutdown_connections 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc0e7d70 0x7f6cbc0e8170 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 --2- 172.21.2.117:0/1209969352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc1b07f0 0x7f6cbc0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 -- 172.21.2.117:0/1209969352 >> 172.21.2.117:0/1209969352 conn(0x7f6cbc0ce500 msgr2=0x7f6cbc0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 -- 172.21.2.117:0/1209969352 shutdown_connections 2021-02-10T11:48:17.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.662 7f6cc303c700 1 -- 172.21.2.117:0/1209969352 wait complete. 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 Processor -- start 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 -- start start 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc096570 0x7f6cbc10cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc10d8a0 0x7f6cbc10dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc10d130 con 0x7f6cbc096170 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc10ee50 con 0x7f6cbc096570 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc303c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc10d390 con 0x7f6cbc10d8a0 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc0e58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc0e58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53034/0 (socket says 172.21.2.117:53034) 2021-02-10T11:48:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.677 7f6cc0e58700 1 -- 172.21.2.117:0/1201927755 learned_addr learned my addr 172.21.2.117:0/1201927755 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 -- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc10d8a0 msgr2=0x7f6cbc10dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc10d8a0 0x7f6cbc10dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 -- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc096570 msgr2=0x7f6cbc10cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc096570 0x7f6cbc10cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 -- 172.21.2.117:0/1201927755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc10e2f0 con 0x7f6cbc096170 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc0e58700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f6ca4006f40 tx=0x7f6ca400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ca4006bb0 con 0x7f6cbc096170 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca400bd80 con 0x7f6cbc096170 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ca4002220 con 0x7f6cbc096170 2021-02-10T11:48:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cbc10e2f0 con 0x7f6cbc096170 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.678 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cbc10e880 con 0x7f6cbc096170 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.679 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ca4006b50 con 0x7f6cbc096170 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.679 7f6cb97fa700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 0x7f6ca001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.679 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ca40326d0 con 0x7f6cbc096170 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.679 7f6cbbfff700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 0x7f6ca001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.680 7f6cbbfff700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 0x7f6ca001abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:17.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.709 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc10c2c0 con 0x7f6cbc096170 2021-02-10T11:48:17.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.711 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ca4000960 con 0x7f6cbc096170 2021-02-10T11:48:17.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.859 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6cbc0e77f0 con 0x7f6cbc096170 2021-02-10T11:48:17.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.863 7f6cb97fa700 1 -- 172.21.2.117:0/1201927755 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ca4006b50 con 0x7f6cbc096170 2021-02-10T11:48:17.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 msgr2=0x7f6ca001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 0x7f6ca001abc0 crc :-1 s=READY pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 msgr2=0x7f6cbc10c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:17.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f6ca4006f40 tx=0x7f6ca400b990).stop 2021-02-10T11:48:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 shutdown_connections 2021-02-10T11:48:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6cbc10d8a0 0x7f6cbc10dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:17.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6cbc096570 0x7f6cbc10cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ca00186c0 0x7f6ca001abc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 --2- 172.21.2.117:0/1201927755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6cbc096170 0x7f6cbc10c4f0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 >> 172.21.2.117:0/1201927755 conn(0x7f6cbc097720 msgr2=0x7f6cbc1994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 shutdown_connections 2021-02-10T11:48:17.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:17.868 7f6cc303c700 1 -- 172.21.2.117:0/1201927755 wait complete. 2021-02-10T11:48:17.871 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update' 2021-02-10T11:48:18.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.287 7f17e3b4d700 1 Processor -- start 2021-02-10T11:48:18.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 -- start start 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0e4e40 0x7f17dc1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17dc0e5240 0x7f17dc084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc085930 con 0x7f17dc1b0810 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc087430 con 0x7f17dc0e5240 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e3b4d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0e5cc0 con 0x7f17dc0e4e40 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e1168700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17dc0e5240 0x7f17dc084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e216a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17dc1b0810 0x7f17dc085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53290/0 (socket says 172.21.2.117:53290) 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.288 7f17e216a700 1 -- 172.21.2.117:0/1112352749 learned_addr learned my addr 172.21.2.117:0/1112352749 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e1168700 1 -- 172.21.2.117:0/1112352749 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0e4e40 msgr2=0x7f17dc1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e1168700 1 --2- 172.21.2.117:0/1112352749 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0e4e40 0x7f17dc1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e1168700 1 -- 172.21.2.117:0/1112352749 >> v1:172.21.2.117:6789/0 conn(0x7f17dc1b0810 legacy=0x7f17dc085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e1168700 1 -- 172.21.2.117:0/1112352749 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc084840 con 0x7f17dc0e5240 2021-02-10T11:48:18.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e1168700 1 --2- 172.21.2.117:0/1112352749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0e5240 0x7f17dc084c60 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f17d8006650 tx=0x7f17d8009a80).ready entity=mon.0 client_cookie=48f961e635304fef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e0967700 1 -- 172.21.2.117:0/1112352749 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800c0c0 con 0x7f17dc0e5240 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e0967700 1 -- 172.21.2.117:0/1112352749 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d800c340 con 0x7f17dc0e5240 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e0967700 1 -- 172.21.2.117:0/1112352749 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d800c860 con 0x7f17dc0e5240 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e3b4d700 1 -- 172.21.2.117:0/1112352749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0e5240 msgr2=0x7f17dc084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.289 7f17e3b4d700 1 --2- 172.21.2.117:0/1112352749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0e5240 0x7f17dc084c60 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 rx=0x7f17d8006650 tx=0x7f17d8009a80).stop 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 -- 172.21.2.117:0/1112352749 shutdown_connections 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 --2- 172.21.2.117:0/1112352749 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0e5240 0x7f17dc084c60 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 --2- 172.21.2.117:0/1112352749 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0e4e40 0x7f17dc1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 -- 172.21.2.117:0/1112352749 >> 172.21.2.117:0/1112352749 conn(0x7f17dc0ce4e0 msgr2=0x7f17dc0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 -- 172.21.2.117:0/1112352749 shutdown_connections 2021-02-10T11:48:18.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.290 7f17e3b4d700 1 -- 172.21.2.117:0/1112352749 wait complete. 2021-02-10T11:48:18.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 Processor -- start 2021-02-10T11:48:18.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 -- start start 2021-02-10T11:48:18.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0b0700 0x7f17dc18c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17dc190f20 0x7f17dc097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc097a10 con 0x7f17dc0985b0 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc18c720 con 0x7f17dc190f20 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e3b4d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0989b0 con 0x7f17dc0b0700 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e216a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e216a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53044/0 (socket says 172.21.2.117:53044) 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.305 7f17e216a700 1 -- 172.21.2.117:0/3551553033 learned_addr learned my addr 172.21.2.117:0/3551553033 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 -- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0b0700 msgr2=0x7f17dc18c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0b0700 0x7f17dc18c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 -- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17dc190f20 msgr2=0x7f17dc097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17dc190f20 0x7f17dc097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 -- 172.21.2.117:0/3551553033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0965f0 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e216a700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f17d400a940 tx=0x7f17d400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d4000d30 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d40070b0 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17d4007540 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc0965f0 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.306 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc0965f0 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.307 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17d4000d30 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.307 7f17ce7fc700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 0x7f17c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.307 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f17d4032b50 con 0x7f17dc0985b0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.307 7f17e1969700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 0x7f17c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:18.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.308 7f17e1969700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 0x7f17c401ac10 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:18.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.338 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc10c250 con 0x7f17dc0985b0 2021-02-10T11:48:18.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.340 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17d4007cb0 con 0x7f17dc0985b0 2021-02-10T11:48:18.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:18.472 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]} v 0) v1 -- 0x7f17dc10c250 con 0x7f17dc0985b0 2021-02-10T11:48:19.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.237 7f17ce7fc700 1 -- 172.21.2.117:0/3551553033 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_deauthorize_auth_id_after_out_of_band_update"]}]=0 v516) v1 ==== 166+0+0 (secure 0 0 0) 0x7f17d40070b0 con 0x7f17dc0985b0 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 msgr2=0x7f17c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 0x7f17c401ac10 crc :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 msgr2=0x7f17dc095fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 rx=0x7f17d400a940 tx=0x7f17d400ace0).stop 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 shutdown_connections 2021-02-10T11:48:19.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17c4018730 0x7f17c401ac10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17dc0985b0 0x7f17dc095fd0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17dc190f20 0x7f17dc097f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 --2- 172.21.2.117:0/3551553033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17dc0b0700 0x7f17dc18c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 >> 172.21.2.117:0/3551553033 conn(0x7f17dc199160 msgr2=0x7f17dc08d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 shutdown_connections 2021-02-10T11:48:19.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.243 7f17e3b4d700 1 -- 172.21.2.117:0/3551553033 wait complete. 2021-02-10T11:48:19.246 INFO:tasks.cephfs_test_runner:test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:48:19.247 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:48:19.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:48:19.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.666 7f5f1720e700 1 Processor -- start 2021-02-10T11:48:19.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.666 7f5f1720e700 1 -- start start 2021-02-10T11:48:19.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1720e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5f100ef3f0 0x7f5f100eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1720e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 0x7f5f101b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1720e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f100ede60 con 0x7f5f100d18e0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1720e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f100ef7f0 con 0x7f5f100ef3f0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1720e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100d1ce0 con 0x7f5f100eb7f0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1502a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5f100d18e0 0x7f5f100ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53302/0 (socket says 172.21.2.117:53302) 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1502a700 1 -- 172.21.2.117:0/488048528 learned_addr learned my addr 172.21.2.117:0/488048528 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f1582b700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 0x7f5f101b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f14829700 1 --2- 172.21.2.117:0/488048528 >> v2:172.21.2.125:3300/0 conn(0x7f5f100ef3f0 0x7f5f100eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f07fff700 1 -- 172.21.2.117:0/488048528 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643347552 0 0) 0x7f5f08001500 con 0x7f5f100d18e0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.667 7f5f07fff700 1 -- 172.21.2.117:0/488048528 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc002fe0 con 0x7f5f100d18e0 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f14829700 1 -- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 msgr2=0x7f5f101b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:19.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f14829700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 0x7f5f101b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f14829700 1 -- 172.21.2.117:0/488048528 >> v1:172.21.2.117:6789/0 conn(0x7f5f100d18e0 legacy=0x7f5f100ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f07fff700 1 -- 172.21.2.117:0/488048528 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2181516315 0 0) 0x7f5f08001460 con 0x7f5f100d18e0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f14829700 1 -- 172.21.2.117:0/488048528 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f101b8f90 con 0x7f5f100ef3f0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f1582b700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 0x7f5f101b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f14829700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100ef3f0 0x7f5f100eb1d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f5f00006650 tx=0x7f5f00009a80).ready entity=mon.1 client_cookie=7977ddb0f3196d01 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f07fff700 1 -- 172.21.2.117:0/488048528 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f0000c0c0 con 0x7f5f100ef3f0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f07fff700 1 -- 172.21.2.117:0/488048528 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0000c340 con 0x7f5f100ef3f0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f07fff700 1 -- 172.21.2.117:0/488048528 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f0000c860 con 0x7f5f100ef3f0 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f1720e700 1 -- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100ef3f0 msgr2=0x7f5f100eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.668 7f5f1720e700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100ef3f0 0x7f5f100eb1d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f5f00006650 tx=0x7f5f00009a80).stop 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 -- 172.21.2.117:0/488048528 shutdown_connections 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100eb7f0 0x7f5f101b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 --2- 172.21.2.117:0/488048528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100ef3f0 0x7f5f100eb1d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 -- 172.21.2.117:0/488048528 >> 172.21.2.117:0/488048528 conn(0x7f5f100ced60 msgr2=0x7f5f100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:19.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 -- 172.21.2.117:0/488048528 shutdown_connections 2021-02-10T11:48:19.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.669 7f5f1720e700 1 -- 172.21.2.117:0/488048528 wait complete. 2021-02-10T11:48:19.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.683 7f5f1720e700 1 Processor -- start 2021-02-10T11:48:19.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 -- start start 2021-02-10T11:48:19.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 0x7f5f101a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100b8d10 0x7f5f100b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100c0670 con 0x7f5f101a33d0 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100deb20 con 0x7f5f100b82f0 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1720e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100ded40 con 0x7f5f100b8d10 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f14829700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f14829700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57446/0 (socket says 172.21.2.117:57446) 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f14829700 1 -- 172.21.2.117:0/1578488640 learned_addr learned my addr 172.21.2.117:0/1578488640 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1502a700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 0x7f5f101a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.684 7f5f1582b700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100b8d10 0x7f5f100b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 -- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100b8d10 msgr2=0x7f5f100b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100b8d10 0x7f5f100b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 -- 172.21.2.117:0/1578488640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 msgr2=0x7f5f101a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 0x7f5f101a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:19.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 -- 172.21.2.117:0/1578488640 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f100def60 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f1502a700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 0x7f5f101a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f14829700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f5f00009630 tx=0x7f5f0000b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f0000bd60 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f00006740 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f00002270 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f100def60 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.685 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f100def60 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.686 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f00000c90 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.686 7f5f05ffb700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 0x7f5ef001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.686 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5f00032690 con 0x7f5f100b82f0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.686 7f5f1502a700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 0x7f5ef001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:19.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.687 7f5f1502a700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 0x7f5ef001abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:19.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.717 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f10087290 con 0x7f5f100b82f0 2021-02-10T11:48:19.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.721 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f00030cf0 con 0x7f5f100b82f0 2021-02-10T11:48:19.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:19.855 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7f5f100ede60 con 0x7f5f100b82f0 2021-02-10T11:48:20.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.243 7f5f05ffb700 1 -- 172.21.2.117:0/1578488640 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v517) v1 ==== 150+0+0 (secure 0 0 0) 0x7f5f0001f490 con 0x7f5f100b82f0 2021-02-10T11:48:20.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 msgr2=0x7f5ef001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 0x7f5ef001abc0 crc :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 msgr2=0x7f5f100b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f5f00009630 tx=0x7f5f0000b970).stop 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 shutdown_connections 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f100b8d10 0x7f5f100b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f100b82f0 0x7f5f100b86f0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ef00186c0 0x7f5ef001abc0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 --2- 172.21.2.117:0/1578488640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f101a33d0 0x7f5f101a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.275 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 >> 172.21.2.117:0/1578488640 conn(0x7f5f100874c0 msgr2=0x7f5f10096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.276 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 shutdown_connections 2021-02-10T11:48:20.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.276 7f5f1720e700 1 -- 172.21.2.117:0/1578488640 wait complete. 2021-02-10T11:48:20.278 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:48:20.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.698 7f8d957ef700 1 Processor -- start 2021-02-10T11:48:20.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d957ef700 1 -- start start 2021-02-10T11:48:20.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d957ef700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8d900d18f0 0x7f8d900eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d957ef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d900ef7f0 con 0x7f8d900eb040 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d957ef700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d900ede60 con 0x7f8d900ef3f0 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d957ef700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900d1da0 con 0x7f8d900d18f0 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d8ed73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8d900eb040 0x7f8d900eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53316/0 (socket says 172.21.2.117:53316) 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.699 7f8d8ed73700 1 -- 172.21.2.117:0/2147898493 learned_addr learned my addr 172.21.2.117:0/2147898493 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126944749 0 0) 0x7f8d780010b0 con 0x7f8d900eb040 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c002fe0 con 0x7f8d900eb040 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> v2:172.21.2.125:3300/0 conn(0x7f8d900d18f0 0x7f8d900eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1390313059 0 0) 0x7f8d780010b0 con 0x7f8d900eb040 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 >> v2:172.21.2.125:3300/0 conn(0x7f8d900d18f0 msgr2=0x7f8d900eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:20.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 --2- 172.21.2.117:0/2147898493 >> v2:172.21.2.125:3300/0 conn(0x7f8d900d18f0 0x7f8d900eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 >> v1:172.21.2.125:6789/0 conn(0x7f8d900ef3f0 legacy=0x7f8d901b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d901bb1c0 con 0x7f8d900eb040 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8d780015b0 con 0x7f8d900eb040 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 >> v1:172.21.2.117:6789/0 conn(0x7f8d900eb040 legacy=0x7f8d900eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 0x7f8d7c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d7c0068b0 0x7f8d7c008dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d7c009640 0x7f8d7c00bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00c3b0 con 0x7f8d7c003b10 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00c5d0 con 0x7f8d7c0068b0 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c00c7f0 con 0x7f8d7c009640 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.700 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> v2:172.21.2.125:3300/0 conn(0x7f8d900d18f0 0x7f8d900eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:20.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8ed73700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d7c0068b0 0x7f8d7c008dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 0x7f8d7c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8fd75700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d7c009640 0x7f8d7c00bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 -- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d7c009640 msgr2=0x7f8d7c00bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d7c009640 0x7f8d7c00bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 -- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d7c0068b0 msgr2=0x7f8d7c008dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d7c0068b0 0x7f8d7c008dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 -- 172.21.2.117:0/2147898493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c050a60 con 0x7f8d7c003b10 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8fd75700 1 -- 172.21.2.117:0/2147898493 reap_dead start 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8ed73700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d7c0068b0 0x7f8d7c008dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8f574700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 0x7f8d7c006010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f8d80000b80 tx=0x7f8d80009e80).ready entity=mon.0 client_cookie=3821e71e93122731 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8000a070 con 0x7f8d7c003b10 2021-02-10T11:48:20.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8000a2f0 con 0x7f8d7c003b10 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d8e572700 1 -- 172.21.2.117:0/2147898493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8000a070 con 0x7f8d7c003b10 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d957ef700 1 -- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 msgr2=0x7f8d7c006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.701 7f8d957ef700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 0x7f8d7c006010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 rx=0x7f8d80000b80 tx=0x7f8d80009e80).stop 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.702 7f8d957ef700 1 -- 172.21.2.117:0/2147898493 shutdown_connections 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.702 7f8d957ef700 1 --2- 172.21.2.117:0/2147898493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d7c003b10 0x7f8d7c006010 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.702 7f8d957ef700 1 -- 172.21.2.117:0/2147898493 >> 172.21.2.117:0/2147898493 conn(0x7f8d900cf060 msgr2=0x7f8d900cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.702 7f8d957ef700 1 -- 172.21.2.117:0/2147898493 shutdown_connections 2021-02-10T11:48:20.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.702 7f8d957ef700 1 -- 172.21.2.117:0/2147898493 wait complete. 2021-02-10T11:48:20.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 Processor -- start 2021-02-10T11:48:20.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 -- start start 2021-02-10T11:48:20.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d90083990 0x7f8d900863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d901a4180 0x7f8d900dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90085e50 con 0x7f8d901a4580 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90097270 con 0x7f8d901a4180 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.717 7f8d957ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90097450 con 0x7f8d90083990 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53082/0 (socket says 172.21.2.117:53082) 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 -- 172.21.2.117:0/1146397165 learned_addr learned my addr 172.21.2.117:0/1146397165 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8f574700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d90083990 0x7f8d900863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8ed73700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d901a4180 0x7f8d900dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 -- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d90083990 msgr2=0x7f8d900863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d90083990 0x7f8d900863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 -- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d901a4180 msgr2=0x7f8d900dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d901a4180 0x7f8d900dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 -- 172.21.2.117:0/1146397165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9009ed70 con 0x7f8d901a4580 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d8fd75700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f8d8800a8f0 tx=0x7f8d8800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8800c300 con 0x7f8d901a4580 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8800c540 con 0x7f8d901a4580 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9009ed70 con 0x7f8d901a4580 2021-02-10T11:48:20.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.718 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9009f2c0 con 0x7f8d901a4580 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.719 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8800ca60 con 0x7f8d901a4580 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.719 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d880069b0 con 0x7f8d901a4580 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.719 7f8d87fff700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 0x7f8d7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.719 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d880329f0 con 0x7f8d901a4580 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.720 7f8d8f574700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 0x7f8d7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:20.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.720 7f8d8f574700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 0x7f8d7401abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:20.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.749 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9009fb50 con 0x7f8d901a4580 2021-02-10T11:48:20.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.752 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d88030970 con 0x7f8d901a4580 2021-02-10T11:48:20.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.900 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8d9009fb50 con 0x7f8d901a4580 2021-02-10T11:48:20.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.904 7f8d87fff700 1 -- 172.21.2.117:0/1146397165 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8d88030970 con 0x7f8d901a4580 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.909 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 msgr2=0x7f8d7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.909 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 0x7f8d7401abc0 crc :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.909 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 msgr2=0x7f8d900dc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.909 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f8d8800a8f0 tx=0x7f8d8800ac90).stop 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 shutdown_connections 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d740186c0 0x7f8d7401abc0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d901a4580 0x7f8d900dc460 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d901a4180 0x7f8d900dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 --2- 172.21.2.117:0/1146397165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d90083990 0x7f8d900863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:20.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 >> 172.21.2.117:0/1146397165 conn(0x7f8d900b6540 msgr2=0x7f8d9008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:20.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 shutdown_connections 2021-02-10T11:48:20.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:20.910 7f8d957ef700 1 -- 172.21.2.117:0/1146397165 wait complete. 2021-02-10T11:48:20.914 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:48:20.914 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:21.259 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:48:21.604 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:48:21.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:21.604 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:21.604 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:21.940 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:48:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:21.941 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:21.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:21.941 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:21.941 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:22.276 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:48:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:22.277 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:22.277 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:22.277 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:22.277 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:48:22.612 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:48:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:22.612 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:48:22.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:22.613 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:22.613 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:48:22.947 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:48:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:48:22.948 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:48:22.948 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:48:23.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 Processor -- start 2021-02-10T11:48:23.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 -- start start 2021-02-10T11:48:23.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc2c40d18e0 0x7fc2c40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc2c40ef3f0 0x7fc2c41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40ede60 con 0x7fc2c40eb000 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c40ef7f0 con 0x7fc2c40d18e0 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.363 7fc2ca913700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c40d1ce0 con 0x7fc2c40ef3f0 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc2c40d18e0 0x7fc2c40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc2c40eb000 0x7fc2c40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53340/0 (socket says 172.21.2.117:53340) 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c37fe700 1 -- 172.21.2.117:0/932706575 learned_addr learned my addr 172.21.2.117:0/932706575 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 -- 172.21.2.117:0/932706575 >> v2:172.21.2.125:3300/0 conn(0x7fc2c40ef3f0 msgr2=0x7fc2c41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 --2- 172.21.2.117:0/932706575 >> v2:172.21.2.125:3300/0 conn(0x7fc2c40ef3f0 0x7fc2c41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 -- 172.21.2.117:0/932706575 >> v1:172.21.2.117:6789/0 conn(0x7fc2c40eb000 legacy=0x7fc2c40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 -- 172.21.2.117:0/932706575 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c41b8f90 con 0x7fc2c40d18e0 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c3fff700 1 --2- 172.21.2.117:0/932706575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40d18e0 0x7fc2c40ea9e0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7fc2b4006650 tx=0x7fc2b4009a80).ready entity=mon.0 client_cookie=b0ccc51d6411d592 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:23.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c2ffd700 1 -- 172.21.2.117:0/932706575 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b400c0c0 con 0x7fc2c40d18e0 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c2ffd700 1 -- 172.21.2.117:0/932706575 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b400c3c0 con 0x7fc2c40d18e0 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2c2ffd700 1 -- 172.21.2.117:0/932706575 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b400c810 con 0x7fc2c40d18e0 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2ca913700 1 -- 172.21.2.117:0/932706575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40d18e0 msgr2=0x7fc2c40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.364 7fc2ca913700 1 --2- 172.21.2.117:0/932706575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40d18e0 0x7fc2c40ea9e0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 rx=0x7fc2b4006650 tx=0x7fc2b4009a80).stop 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 -- 172.21.2.117:0/932706575 shutdown_connections 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 --2- 172.21.2.117:0/932706575 >> v2:172.21.2.125:3300/0 conn(0x7fc2c40ef3f0 0x7fc2c41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 --2- 172.21.2.117:0/932706575 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40d18e0 0x7fc2c40ea9e0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 -- 172.21.2.117:0/932706575 >> 172.21.2.117:0/932706575 conn(0x7fc2c40ced60 msgr2=0x7fc2c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 -- 172.21.2.117:0/932706575 shutdown_connections 2021-02-10T11:48:23.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.365 7fc2ca913700 1 -- 172.21.2.117:0/932706575 wait complete. 2021-02-10T11:48:23.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.379 7fc2ca913700 1 Processor -- start 2021-02-10T11:48:23.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 -- start start 2021-02-10T11:48:23.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2c40e00e0 0x7fc2c40b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4192970 con 0x7fc2c40b8270 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4192550 con 0x7fc2c4096b40 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2ca913700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4192770 con 0x7fc2c40e00e0 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c8f30700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c8f30700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53096/0 (socket says 172.21.2.117:53096) 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57478/0 (socket says 172.21.2.117:57478) 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c8f30700 1 -- 172.21.2.117:0/1727956353 learned_addr learned my addr 172.21.2.117:0/1727956353 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.380 7fc2c37fe700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2c40e00e0 0x7fc2c40b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 -- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2c40e00e0 msgr2=0x7fc2c40b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2c40e00e0 0x7fc2c40b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 -- 172.21.2.117:0/1727956353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 msgr2=0x7fc2c40e0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 -- 172.21.2.117:0/1727956353 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40b8670 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c8f30700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c3fff700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fc2b4006af0 tx=0x7fc2b400c8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b400ccd0 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b40063a0 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c40b8670 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2b4002020 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.381 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c40b8670 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.382 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc2b40052e0 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.382 7fc2c0ff9700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 0x7fc2a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.382 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc2b4032820 con 0x7fc2c4096b40 2021-02-10T11:48:23.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.382 7fc2c37fe700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 0x7fc2a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:23.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.382 7fc2c37fe700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 0x7fc2a401ac10 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:23.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.413 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c4087210 con 0x7fc2c4096b40 2021-02-10T11:48:23.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.416 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc2b4006790 con 0x7fc2c4096b40 2021-02-10T11:48:23.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.543 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc2c40b8670 con 0x7fc2c4096b40 2021-02-10T11:48:23.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.544 7fc2c0ff9700 1 -- 172.21.2.117:0/1727956353 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v91) v1 ==== 81+0+30755 (secure 0 0 0) 0x7fc2b401f410 con 0x7fc2c4096b40 2021-02-10T11:48:23.542 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:23.542 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:48:23.542 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 91, 2021-02-10T11:48:23.542 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:48:23.542 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:47:43.359349", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:48:23.543 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 19, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:48:23.544 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:48:23.545 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:48:23.546 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:23.547 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:23.548 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:23.549 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.550 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 18, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:47:40.329394", 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:48:23.551 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:48:23.552 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "90", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:48:23.553 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:48:23.554 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:48:23.555 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.556 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 19, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:47:41.402719", 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:48:23.557 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.558 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "91", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:48:23.559 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:48:23.560 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:48:23.561 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.562 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 88, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.563 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.564 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.565 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:48:23.566 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.567 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.568 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.569 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.570 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.571 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.572 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.572 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.572 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.572 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.572 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.573 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.574 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.575 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.576 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.577 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.578 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:48:23.579 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.580 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.581 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.582 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:48:23.583 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.584 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:48:23.585 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.586 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:48:23.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.588 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.589 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.590 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.591 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.592 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:48:23.593 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.594 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.595 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.596 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.597 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:48:23.598 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:48:23.599 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.600 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 81, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.601 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.602 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.603 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:48:23.604 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:48:23.605 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:48:23.606 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.607 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.608 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.609 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:48:23.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:48:23.611 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.612 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:48:23.613 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.614 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.615 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:48:23.617 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:48:23.618 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:48:23.619 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:48:23.620 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:48:23.620 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:48:23.620 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 msgr2=0x7fc2a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 0x7fc2a401ac10 crc :-1 s=READY pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 msgr2=0x7fc2c419fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fc2b4006af0 tx=0x7fc2b400c8e0).stop 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 shutdown_connections 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2a4018730 0x7fc2a401ac10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2c40b8270 0x7fc2c40e0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc2c40e00e0 0x7fc2c40b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 --2- 172.21.2.117:0/1727956353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2c4096b40 0x7fc2c419fb40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:23.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.547 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 >> 172.21.2.117:0/1727956353 conn(0x7fc2c41a97a0 msgr2=0x7fc2c40dd890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:23.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.548 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 shutdown_connections 2021-02-10T11:48:23.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:23.548 7fc2ca913700 1 -- 172.21.2.117:0/1727956353 wait complete. 2021-02-10T11:48:23.623 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:24.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.038 7fcc8bce2700 1 Processor -- start 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.038 7fcc8bce2700 1 -- start start 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8bce2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcc840ef3f0 0x7fcc840eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8bce2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840ede60 con 0x7fcc840eb7f0 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8bce2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840d1ce0 con 0x7fcc840d18e0 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8bce2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840ef7f0 con 0x7fcc840ef3f0 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc892fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcc840ef3f0 0x7fcc840eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8a2ff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcc840eb7f0 0x7fcc841b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53354/0 (socket says 172.21.2.117:53354) 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc8a2ff700 1 -- 172.21.2.117:0/3117119061 learned_addr learned my addr 172.21.2.117:0/3117119061 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc892fd700 1 -- 172.21.2.117:0/3117119061 >> v1:172.21.2.125:6789/0 conn(0x7fcc840d18e0 legacy=0x7fcc840ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:24.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc892fd700 1 -- 172.21.2.117:0/3117119061 >> v1:172.21.2.117:6789/0 conn(0x7fcc840eb7f0 legacy=0x7fcc841b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc88afc700 1 -- 172.21.2.117:0/3117119061 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602030077 0 0) 0x7fcc7c001500 con 0x7fcc840eb7f0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc892fd700 1 -- 172.21.2.117:0/3117119061 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc841bb180 con 0x7fcc840ef3f0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.039 7fcc892fd700 1 --2- 172.21.2.117:0/3117119061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc840ef3f0 0x7fcc840eb1d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7fcc78006650 tx=0x7fcc78009a00).ready entity=mon.0 client_cookie=2fe0c306dac5a96f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc88afc700 1 -- 172.21.2.117:0/3117119061 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc78009df0 con 0x7fcc840ef3f0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc88afc700 1 -- 172.21.2.117:0/3117119061 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc780009b0 con 0x7fcc840ef3f0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc88afc700 1 -- 172.21.2.117:0/3117119061 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc7800c2b0 con 0x7fcc840ef3f0 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 -- 172.21.2.117:0/3117119061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc840ef3f0 msgr2=0x7fcc840eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 --2- 172.21.2.117:0/3117119061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc840ef3f0 0x7fcc840eb1d0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7fcc78006650 tx=0x7fcc78009a00).stop 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 -- 172.21.2.117:0/3117119061 shutdown_connections 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 --2- 172.21.2.117:0/3117119061 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc840ef3f0 0x7fcc840eb1d0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 -- 172.21.2.117:0/3117119061 >> 172.21.2.117:0/3117119061 conn(0x7fcc840ced60 msgr2=0x7fcc840cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:24.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 -- 172.21.2.117:0/3117119061 shutdown_connections 2021-02-10T11:48:24.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.040 7fcc8bce2700 1 -- 172.21.2.117:0/3117119061 wait complete. 2021-02-10T11:48:24.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 Processor -- start 2021-02-10T11:48:24.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 -- start start 2021-02-10T11:48:24.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc840eb2f0 0x7fcc840aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc840af2a0 0x7fcc840f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840ae700 con 0x7fcc8408a280 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840eb6f0 con 0x7fcc840af2a0 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.055 7fcc8bce2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840eb8d0 con 0x7fcc840eb2f0 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53110/0 (socket says 172.21.2.117:53110) 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 -- 172.21.2.117:0/1267642539 learned_addr learned my addr 172.21.2.117:0/1267642539 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 -- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc840eb2f0 msgr2=0x7fcc840aec80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc840eb2f0 0x7fcc840aec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 -- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc840af2a0 msgr2=0x7fcc840f4790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc840af2a0 0x7fcc840f4790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 -- 172.21.2.117:0/1267642539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840af6a0 con 0x7fcc8408a280 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc89afe700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7fcc8000b5b0 tx=0x7fcc8000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc8000bd40 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc80006240 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcc80005720 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.056 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc840af6a0 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.057 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc840af6a0 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.057 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcc800054a0 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.057 7fcc767fc700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 0x7fcc6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.057 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcc80032570 con 0x7fcc8408a280 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.058 7fcc892fd700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 0x7fcc6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.058 7fcc892fd700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 0x7fcc6c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.089 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc84086140 con 0x7fcc8408a280 2021-02-10T11:48:24.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.091 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcc8002a4d0 con 0x7fcc8408a280 2021-02-10T11:48:24.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.217 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc840d18e0 con 0x7fcc8408a280 2021-02-10T11:48:24.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.218 7fcc767fc700 1 -- 172.21.2.117:0/1267642539 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fcc8002a4d0 con 0x7fcc8408a280 2021-02-10T11:48:24.214 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:24.214 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":69,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:47:44.486277","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6539":{"gid":6539,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/4168849820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":4168849820},{"type":"v1","addr":"172.21.2.125:6835","nonce":4168849820}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:48:24.215 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 69 2021-02-10T11:48:24.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 msgr2=0x7fcc6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 0x7fcc6c01abc0 crc :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 msgr2=0x7fcc840eacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7fcc8000b5b0 tx=0x7fcc8000b950).stop 2021-02-10T11:48:24.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 shutdown_connections 2021-02-10T11:48:24.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcc840af2a0 0x7fcc840f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcc840eb2f0 0x7fcc840aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcc6c0186c0 0x7fcc6c01abc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 --2- 172.21.2.117:0/1267642539 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcc8408a280 0x7fcc840eacd0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 >> 172.21.2.117:0/1267642539 conn(0x7fcc8409b100 msgr2=0x7fcc840edb40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:24.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 shutdown_connections 2021-02-10T11:48:24.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.220 7fcc8bce2700 1 -- 172.21.2.117:0/1267642539 wait complete. 2021-02-10T11:48:24.225 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:48:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 Processor -- start 2021-02-10T11:48:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 -- start start 2021-02-10T11:48:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 0x7f68640e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f68640e7180 0x7f68640e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640e7580 con 0x7f68640e7d70 2021-02-10T11:48:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640e92f0 con 0x7f68640e7180 2021-02-10T11:48:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.640 7f686ae00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640e5ce0 con 0x7f68641b07f0 2021-02-10T11:48:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f686941d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f68640e7180 0x7f68640e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f6868c1c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 0x7f68640e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f6863fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f68640e7d70 0x7f68640e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46290/0 (socket says 172.21.2.117:46290) 2021-02-10T11:48:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f6863fff700 1 -- 172.21.2.117:0/2830892733 learned_addr learned my addr 172.21.2.117:0/2830892733 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f686941d700 1 -- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 msgr2=0x7f68640e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f686941d700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 0x7f68640e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f686941d700 1 -- 172.21.2.117:0/2830892733 >> v1:172.21.2.125:6789/0 conn(0x7f68640e7d70 legacy=0x7f68640e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f686941d700 1 -- 172.21.2.117:0/2830892733 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68640e77f0 con 0x7f68640e7180 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.642 7f6868c1c700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 0x7f68640e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f686941d700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68640e7180 0x7f68640e8960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f685c006ad0 tx=0x7f685c009ad0).ready entity=mon.1 client_cookie=4b38d0fc515eb099 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f68637fe700 1 -- 172.21.2.117:0/2830892733 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685c0066a0 con 0x7f68640e7180 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f68637fe700 1 -- 172.21.2.117:0/2830892733 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c006410 con 0x7f68640e7180 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f68637fe700 1 -- 172.21.2.117:0/2830892733 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685c0024d0 con 0x7f68640e7180 2021-02-10T11:48:24.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f686ae00700 1 -- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68640e7180 msgr2=0x7f68640e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.643 7f686ae00700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68640e7180 0x7f68640e8960 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f685c006ad0 tx=0x7f685c009ad0).stop 2021-02-10T11:48:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 -- 172.21.2.117:0/2830892733 shutdown_connections 2021-02-10T11:48:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68640e7180 0x7f68640e8960 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 --2- 172.21.2.117:0/2830892733 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68641b07f0 0x7f68640e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:24.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 -- 172.21.2.117:0/2830892733 >> 172.21.2.117:0/2830892733 conn(0x7f68640ce500 msgr2=0x7f68640e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 -- 172.21.2.117:0/2830892733 shutdown_connections 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.644 7f686ae00700 1 -- 172.21.2.117:0/2830892733 wait complete. 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 Processor -- start 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 -- start start 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6864094c60 0x7f68640b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686419d2f0 0x7f686419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686419e110 con 0x7f686419dd10 2021-02-10T11:48:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68640ae1b0 con 0x7f6864094c60 2021-02-10T11:48:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686ae00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6864095060 con 0x7f686419d2f0 2021-02-10T11:48:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686941d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686941d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53122/0 (socket says 172.21.2.117:53122) 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.659 7f686941d700 1 -- 172.21.2.117:0/2280606065 learned_addr learned my addr 172.21.2.117:0/2280606065 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 -- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686419d2f0 msgr2=0x7f686419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686419d2f0 0x7f686419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 -- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6864094c60 msgr2=0x7f68640b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6864094c60 0x7f68640b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 -- 172.21.2.117:0/2280606065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864094110 con 0x7f686419dd10 2021-02-10T11:48:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686941d700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f685c009670 tx=0x7f685c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685c00bd80 con 0x7f686419dd10 2021-02-10T11:48:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c002080 con 0x7f686419dd10 2021-02-10T11:48:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686ae00700 1 -- 172.21.2.117:0/2280606065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864094110 con 0x7f686419dd10 2021-02-10T11:48:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f686ae00700 1 -- 172.21.2.117:0/2280606065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68640946a0 con 0x7f686419dd10 2021-02-10T11:48:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.660 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f685c002560 con 0x7f686419dd10 2021-02-10T11:48:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.661 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f685c002260 con 0x7f686419dd10 2021-02-10T11:48:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.661 7f68617fa700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 0x7f684801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:24.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.661 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f685c0326a0 con 0x7f686419dd10 2021-02-10T11:48:24.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.662 7f6868c1c700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 0x7f684801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:24.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.662 7f6868c1c700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 0x7f684801abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:24.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.691 7f686ae00700 1 -- 172.21.2.117:0/2280606065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864094940 con 0x7f686419dd10 2021-02-10T11:48:24.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.693 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f685c0093e0 con 0x7f686419dd10 2021-02-10T11:48:24.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:24.821 7f686ae00700 1 -- 172.21.2.117:0/2280606065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6864094940 con 0x7f686419dd10 2021-02-10T11:48:25.226 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:48:25.230 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:48:25.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.236 7f68617fa700 1 -- 172.21.2.117:0/2280606065 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v70) v1 ==== 161+0+0 (secure 0 0 0) 0x7f685c0025f0 con 0x7f686419dd10 2021-02-10T11:48:25.233 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:48:25.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.238 7f686ae00700 1 -- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 msgr2=0x7f684801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 0x7f684801abc0 crc :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 -- 172.21.2.117:0/2280606065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 msgr2=0x7f6864093af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f685c009670 tx=0x7f685c00b990).stop 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 -- 172.21.2.117:0/2280606065 shutdown_connections 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68480186c0 0x7f684801abc0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f686419dd10 0x7f6864093af0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f686419d2f0 0x7f686419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 --2- 172.21.2.117:0/2280606065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6864094c60 0x7f68640b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 -- 172.21.2.117:0/2280606065 >> 172.21.2.117:0/2280606065 conn(0x7f6864096150 msgr2=0x7f686407ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 -- 172.21.2.117:0/2280606065 shutdown_connections 2021-02-10T11:48:25.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.239 7f686ae00700 1 -- 172.21.2.117:0/2280606065 wait complete. 2021-02-10T11:48:25.243 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:25.284 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:48:25.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.660 7f5ce98b5700 1 Processor -- start 2021-02-10T11:48:25.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.660 7f5ce98b5700 1 -- start start 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.660 7f5ce98b5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce98b5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5ce40ef3f0 0x7f5ce41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce98b5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40ede60 con 0x7f5ce40eb000 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce98b5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40ef7f0 con 0x7f5ce40d18e0 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce98b5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40d1ce0 con 0x7f5ce40ef3f0 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53128/0 (socket says 172.21.2.117:53128) 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 -- 172.21.2.117:0/623037433 learned_addr learned my addr 172.21.2.117:0/623037433 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 -- 172.21.2.117:0/623037433 >> v2:172.21.2.125:3300/0 conn(0x7f5ce40ef3f0 msgr2=0x7f5ce41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:25.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 --2- 172.21.2.117:0/623037433 >> v2:172.21.2.125:3300/0 conn(0x7f5ce40ef3f0 0x7f5ce41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 -- 172.21.2.117:0/623037433 >> v1:172.21.2.125:6789/0 conn(0x7f5ce40eb000 legacy=0x7f5ce40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 -- 172.21.2.117:0/623037433 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce41b8f90 con 0x7f5ce40d18e0 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.661 7f5ce3574700 1 --2- 172.21.2.117:0/623037433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f5cd4006700 tx=0x7f5cd4009b00).ready entity=mon.0 client_cookie=bf19f33e04fc1cc8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce2572700 1 -- 172.21.2.117:0/623037433 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c0c0 con 0x7f5ce40d18e0 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce2572700 1 -- 172.21.2.117:0/623037433 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd400c340 con 0x7f5ce40d18e0 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce2572700 1 -- 172.21.2.117:0/623037433 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c820 con 0x7f5ce40d18e0 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 -- 172.21.2.117:0/623037433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce40d18e0 msgr2=0x7f5ce40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 --2- 172.21.2.117:0/623037433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 rx=0x7f5cd4006700 tx=0x7f5cd4009b00).stop 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 -- 172.21.2.117:0/623037433 shutdown_connections 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 --2- 172.21.2.117:0/623037433 >> v2:172.21.2.125:3300/0 conn(0x7f5ce40ef3f0 0x7f5ce41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 --2- 172.21.2.117:0/623037433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce40d18e0 0x7f5ce40ea9e0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 -- 172.21.2.117:0/623037433 >> 172.21.2.117:0/623037433 conn(0x7f5ce40ced60 msgr2=0x7f5ce40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 -- 172.21.2.117:0/623037433 shutdown_connections 2021-02-10T11:48:25.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.662 7f5ce98b5700 1 -- 172.21.2.117:0/623037433 wait complete. 2021-02-10T11:48:25.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 Processor -- start 2021-02-10T11:48:25.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 -- start start 2021-02-10T11:48:25.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce419c130 0x7f5ce40b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce40b8700 0x7f5ce408a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce408bbf0 con 0x7f5ce419c530 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4097140 con 0x7f5ce40b8700 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce98b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce419dfa0 con 0x7f5ce419c130 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53136/0 (socket says 172.21.2.117:53136) 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.677 7f5ce2d73700 1 -- 172.21.2.117:0/1697200968 learned_addr learned my addr 172.21.2.117:0/1697200968 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 -- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce419c130 msgr2=0x7f5ce40b7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce419c130 0x7f5ce40b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 -- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce40b8700 msgr2=0x7f5ce408a8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce40b8700 0x7f5ce408a8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 -- 172.21.2.117:0/1697200968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce408aee0 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce2d73700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f5cd800b530 tx=0x7f5cd800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd800bcc0 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd8000e20 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd80062a0 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce408aee0 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.678 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce408b470 con 0x7f5ce419c530 2021-02-10T11:48:25.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.679 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5cd8006020 con 0x7f5ce419c530 2021-02-10T11:48:25.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.679 7f5ccbfff700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 0x7f5cc401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:25.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.679 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f5cd8032730 con 0x7f5ce419c530 2021-02-10T11:48:25.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.680 7f5ce3574700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 0x7f5cc401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:25.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.680 7f5ce3574700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 0x7f5cc401abc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:25.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.710 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4087280 con 0x7f5ce419c530 2021-02-10T11:48:25.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.713 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5cd8000940 con 0x7f5ce419c530 2021-02-10T11:48:25.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.841 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce40ede60 con 0x7f5ce419c530 2021-02-10T11:48:25.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.841 7f5ccbfff700 1 -- 172.21.2.117:0/1697200968 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+1756 (secure 0 0 0) 0x7f5cd802e0e0 con 0x7f5ce419c530 2021-02-10T11:48:25.838 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:25.838 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":70,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9726,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3719676910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3719676910},{"type":"v1","addr":"172.21.2.117:6833","nonce":3719676910}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":65}],"filesystems":[{"mdsmap":{"epoch":70,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:47:42.357500","modified":"2021-02-10 11:48:25.232303","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2021-02-10T11:48:25.838 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 70 2021-02-10T11:48:25.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 msgr2=0x7f5cc401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 0x7f5cc401abc0 crc :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 msgr2=0x7f5ce40b80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7f5cd800b530 tx=0x7f5cd800b8d0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 shutdown_connections 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc40186c0 0x7f5cc401abc0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce419c530 0x7f5ce40b80e0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce40b8700 0x7f5ce408a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 --2- 172.21.2.117:0/1697200968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce419c130 0x7f5ce40b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.843 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 >> 172.21.2.117:0/1697200968 conn(0x7f5ce40874b0 msgr2=0x7f5ce41a6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.844 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 shutdown_connections 2021-02-10T11:48:25.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:25.844 7f5ce98b5700 1 -- 172.21.2.117:0/1697200968 wait complete. 2021-02-10T11:48:25.848 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:48:26.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.266 7fc8c5f02700 1 Processor -- start 2021-02-10T11:48:26.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.266 7fc8c5f02700 1 -- start start 2021-02-10T11:48:26.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8c5f02700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 0x7fc8c00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8c5f02700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8c00e8d80 0x7fc8c00e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8c5f02700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c00e77f0 con 0x7fc8c00e6170 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8c5f02700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c00e5ce0 con 0x7fc8c00e8d80 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8c5f02700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c00e54b0 con 0x7fc8c01b07f0 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8c00e8d80 0x7fc8c00e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8c00e8d80 0x7fc8c00e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53142/0 (socket says 172.21.2.117:53142) 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 -- 172.21.2.117:0/2484559057 learned_addr learned my addr 172.21.2.117:0/2484559057 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:26.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bf7fe700 1 --2- 172.21.2.117:0/2484559057 >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 0x7fc8c00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 -- 172.21.2.117:0/2484559057 >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 msgr2=0x7fc8c00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 --2- 172.21.2.117:0/2484559057 >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 0x7fc8c00e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 -- 172.21.2.117:0/2484559057 >> v1:172.21.2.125:6789/0 conn(0x7fc8c00e6170 legacy=0x7fc8c00e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 -- 172.21.2.117:0/2484559057 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c00e9180 con 0x7fc8c00e8d80 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bf7fe700 1 --2- 172.21.2.117:0/2484559057 >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 0x7fc8c00e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:26.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.267 7fc8bffff700 1 --2- 172.21.2.117:0/2484559057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00e8d80 0x7fc8c00e6d60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fc8b8006f00 tx=0x7fc8b8009f10).ready entity=mon.0 client_cookie=c3e1f52ebfcb045b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8be7fc700 1 -- 172.21.2.117:0/2484559057 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b8003030 con 0x7fc8c00e8d80 2021-02-10T11:48:26.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8be7fc700 1 -- 172.21.2.117:0/2484559057 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b800b3b0 con 0x7fc8c00e8d80 2021-02-10T11:48:26.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8be7fc700 1 -- 172.21.2.117:0/2484559057 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b800b890 con 0x7fc8c00e8d80 2021-02-10T11:48:26.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 -- 172.21.2.117:0/2484559057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00e8d80 msgr2=0x7fc8c00e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 --2- 172.21.2.117:0/2484559057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00e8d80 0x7fc8c00e6d60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fc8b8006f00 tx=0x7fc8b8009f10).stop 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 -- 172.21.2.117:0/2484559057 shutdown_connections 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 --2- 172.21.2.117:0/2484559057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00e8d80 0x7fc8c00e6d60 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 --2- 172.21.2.117:0/2484559057 >> v2:172.21.2.125:3300/0 conn(0x7fc8c01b07f0 0x7fc8c00e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 -- 172.21.2.117:0/2484559057 >> 172.21.2.117:0/2484559057 conn(0x7fc8c00ce500 msgr2=0x7fc8c00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 -- 172.21.2.117:0/2484559057 shutdown_connections 2021-02-10T11:48:26.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.268 7fc8c5f02700 1 -- 172.21.2.117:0/2484559057 wait complete. 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.283 7fc8c5f02700 1 Processor -- start 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.283 7fc8c5f02700 1 -- start start 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.283 7fc8c5f02700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.283 7fc8c5f02700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 0x7fc8c0195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8c5f02700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 0x7fc8c010cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8c5f02700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c0195f60 con 0x7fc8c00aea50 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8c5f02700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c0080650 con 0x7fc8c0195b60 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8c5f02700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c01afbf0 con 0x7fc8c0080250 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53148/0 (socket says 172.21.2.117:53148) 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bf7fe700 1 -- 172.21.2.117:0/999469779 learned_addr learned my addr 172.21.2.117:0/999469779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:26.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 0x7fc8c0195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bffff700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 0x7fc8c010cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 -- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 msgr2=0x7fc8c010cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 0x7fc8c010cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 -- 172.21.2.117:0/999469779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 msgr2=0x7fc8c01aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:26.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 -- 172.21.2.117:0/999469779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c010d100 con 0x7fc8c0080250 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bffff700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 0x7fc8c010cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8bf7fe700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8beffd700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 0x7fc8c0195540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc8b0006e80 tx=0x7fc8b000b8b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b000bca0 con 0x7fc8c0080250 2021-02-10T11:48:26.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b00060c0 con 0x7fc8c0080250 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c010d100 con 0x7fc8c0080250 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.284 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8b00054b0 con 0x7fc8c0080250 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.285 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c010d100 con 0x7fc8c0080250 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.285 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc8b0000d00 con 0x7fc8c0080250 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.285 7fc8b7fff700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 0x7fc8a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:26.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.285 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fc8b0032880 con 0x7fc8c0080250 2021-02-10T11:48:26.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.285 7fc8bf7fe700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 0x7fc8a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:26.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.286 7fc8bf7fe700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 0x7fc8a401abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:26.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.316 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c007e840 con 0x7fc8c0080250 2021-02-10T11:48:26.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.320 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8b0006b50 con 0x7fc8c0080250 2021-02-10T11:48:26.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:26.452 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc8c00e77f0 con 0x7fc8c0080250 2021-02-10T11:48:27.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.240 7fc8b7fff700 1 -- 172.21.2.117:0/999469779 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc8b0000d00 con 0x7fc8c0080250 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 msgr2=0x7fc8a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 0x7fc8a401abc0 crc :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 msgr2=0x7fc8c0195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 0x7fc8c0195540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fc8b0006e80 tx=0x7fc8b000b8b0).stop 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 shutdown_connections 2021-02-10T11:48:27.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8c0195b60 0x7fc8c010cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8c0080250 0x7fc8c0195540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8a40186c0 0x7fc8a401abc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 --2- 172.21.2.117:0/999469779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8c00aea50 0x7fc8c01aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 >> 172.21.2.117:0/999469779 conn(0x7fc8c01998b0 msgr2=0x7fc8c01ab430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 shutdown_connections 2021-02-10T11:48:27.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.273 7fc8c5f02700 1 -- 172.21.2.117:0/999469779 wait complete. 2021-02-10T11:48:27.277 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.695 7f33a4754700 1 Processor -- start 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.695 7f33a4754700 1 -- start start 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a4754700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f339c0e5240 0x7f339c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a4754700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f339c084c60 0x7f339c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a4754700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339c085930 con 0x7f339c0e4e40 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a4754700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0e5cc0 con 0x7f339c0e5240 2021-02-10T11:48:27.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a4754700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0856b0 con 0x7f339c084c60 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a1d6f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f339c0e5240 0x7f339c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a2570700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f339c0e4e40 0x7f339c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53404/0 (socket says 172.21.2.117:53404) 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a2570700 1 -- 172.21.2.117:0/432693886 learned_addr learned my addr 172.21.2.117:0/432693886 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a1d6f700 1 -- 172.21.2.117:0/432693886 >> v2:172.21.2.125:3300/0 conn(0x7f339c084c60 msgr2=0x7f339c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a1d6f700 1 --2- 172.21.2.117:0/432693886 >> v2:172.21.2.125:3300/0 conn(0x7f339c084c60 0x7f339c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a1d6f700 1 -- 172.21.2.117:0/432693886 >> v1:172.21.2.117:6789/0 conn(0x7f339c0e4e40 legacy=0x7f339c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.696 7f33a1d6f700 1 -- 172.21.2.117:0/432693886 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c1b09e0 con 0x7f339c0e5240 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a1d6f700 1 --2- 172.21.2.117:0/432693886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c0e5240 0x7f339c086ec0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f3398006650 tx=0x7f3398009a80).ready entity=mon.0 client_cookie=3c31c018dbaf1345 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a156e700 1 -- 172.21.2.117:0/432693886 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33980009b0 con 0x7f339c0e5240 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a156e700 1 -- 172.21.2.117:0/432693886 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339800c0c0 con 0x7f339c0e5240 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a156e700 1 -- 172.21.2.117:0/432693886 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f339800c5e0 con 0x7f339c0e5240 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 -- 172.21.2.117:0/432693886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c0e5240 msgr2=0x7f339c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 --2- 172.21.2.117:0/432693886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c0e5240 0x7f339c086ec0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f3398006650 tx=0x7f3398009a80).stop 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 -- 172.21.2.117:0/432693886 shutdown_connections 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 --2- 172.21.2.117:0/432693886 >> v2:172.21.2.125:3300/0 conn(0x7f339c084c60 0x7f339c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 --2- 172.21.2.117:0/432693886 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c0e5240 0x7f339c086ec0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.697 7f33a4754700 1 -- 172.21.2.117:0/432693886 >> 172.21.2.117:0/432693886 conn(0x7f339c0ce4e0 msgr2=0x7f339c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.698 7f33a4754700 1 -- 172.21.2.117:0/432693886 shutdown_connections 2021-02-10T11:48:27.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.698 7f33a4754700 1 -- 172.21.2.117:0/432693886 wait complete. 2021-02-10T11:48:27.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.712 7f33a4754700 1 Processor -- start 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.712 7f33a4754700 1 -- start start 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f339c093070 0x7f339c09fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f339c0a00e0 0x7f339c1ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c09f540 con 0x7f339c1ae8d0 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c093470 con 0x7f339c093070 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a4754700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c09ecd0 con 0x7f339c0a00e0 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53164/0 (socket says 172.21.2.117:53164) 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 -- 172.21.2.117:0/2927840559 learned_addr learned my addr 172.21.2.117:0/2927840559 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 -- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f339c0a00e0 msgr2=0x7f339c1ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f339c0a00e0 0x7f339c1ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 -- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f339c093070 msgr2=0x7f339c09fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f339c093070 0x7f339c09fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.713 7f33a2d71700 1 -- 172.21.2.117:0/2927840559 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c09eef0 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f33a2d71700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7f339400b8c0 tx=0x7f339400bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3394005870 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3394005b30 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f33a4754700 1 -- 172.21.2.117:0/2927840559 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c09eef0 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f33a4754700 1 -- 172.21.2.117:0/2927840559 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c1b20a0 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.714 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3394006030 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.715 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3394005d10 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.715 7f338effd700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 0x7f338401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.715 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f3394032d10 con 0x7f339c1ae8d0 2021-02-10T11:48:27.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.716 7f33a2570700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 0x7f338401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:27.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.716 7f33a2570700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 0x7f338401abd0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:27.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.746 7f33a4754700 1 -- 172.21.2.117:0/2927840559 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c10c2f0 con 0x7f339c1ae8d0 2021-02-10T11:48:27.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.749 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f339400bda0 con 0x7f339c1ae8d0 2021-02-10T11:48:27.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:27.874 7f33a4754700 1 -- 172.21.2.117:0/2927840559 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f339c085930 con 0x7f339c1ae8d0 2021-02-10T11:48:28.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.247 7f338effd700 1 -- 172.21.2.117:0/2927840559 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3394006f20 con 0x7f339c1ae8d0 2021-02-10T11:48:28.287 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:48:28.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 -- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 msgr2=0x7f338401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 0x7f338401abd0 crc :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 -- 172.21.2.117:0/2927840559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 msgr2=0x7f339c09e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 rx=0x7f339400b8c0 tx=0x7f339400bc60).stop 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 -- 172.21.2.117:0/2927840559 shutdown_connections 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33840186d0 0x7f338401abd0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f339c1ae8d0 0x7f339c09e6b0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f339c0a00e0 0x7f339c1ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 --2- 172.21.2.117:0/2927840559 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f339c093070 0x7f339c09fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 -- 172.21.2.117:0/2927840559 >> 172.21.2.117:0/2927840559 conn(0x7f339c1ad6b0 msgr2=0x7f339c190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.293 7f33a4754700 1 -- 172.21.2.117:0/2927840559 shutdown_connections 2021-02-10T11:48:28.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.294 7f33a4754700 1 -- 172.21.2.117:0/2927840559 wait complete. 2021-02-10T11:48:28.298 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:48:28.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 Processor -- start 2021-02-10T11:48:28.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 -- start start 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280e4e40 0x7f2a281b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a28085450 0x7f2a280d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a28085930 con 0x7f2a280e5240 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a280e5cc0 con 0x7f2a28085450 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.715 7f2a2cdae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28084840 con 0x7f2a280e4e40 2021-02-10T11:48:28.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a28085450 0x7f2a280d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2a28085450 0x7f2a280d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53172/0 (socket says 172.21.2.117:53172) 2021-02-10T11:48:28.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 -- 172.21.2.117:0/2140669021 learned_addr learned my addr 172.21.2.117:0/2140669021 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:28.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a26312700 1 --1- 172.21.2.117:0/2140669021 >> v1:172.21.2.125:6789/0 conn(0x7f2a280e5240 0x7f2a28084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46346/0 (socket says 172.21.2.117:46346) 2021-02-10T11:48:28.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a26b13700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280e4e40 0x7f2a281b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 -- 172.21.2.117:0/2140669021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280e4e40 msgr2=0x7f2a281b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280e4e40 0x7f2a281b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 -- 172.21.2.117:0/2140669021 >> v1:172.21.2.125:6789/0 conn(0x7f2a280e5240 legacy=0x7f2a28084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 -- 172.21.2.117:0/2140669021 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a280d1be0 con 0x7f2a28085450 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a27314700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a28085450 0x7f2a280d05b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7f2a20006f00 tx=0x7f2a20009f10).ready entity=mon.0 client_cookie=e2a865e330d4573f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a25b11700 1 -- 172.21.2.117:0/2140669021 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2000b3b0 con 0x7f2a28085450 2021-02-10T11:48:28.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a25b11700 1 -- 172.21.2.117:0/2140669021 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2000b630 con 0x7f2a28085450 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a25b11700 1 -- 172.21.2.117:0/2140669021 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2000bb50 con 0x7f2a28085450 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a2cdae700 1 -- 172.21.2.117:0/2140669021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a28085450 msgr2=0x7f2a280d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.716 7f2a2cdae700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a28085450 0x7f2a280d05b0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7f2a20006f00 tx=0x7f2a20009f10).stop 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 -- 172.21.2.117:0/2140669021 shutdown_connections 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a28085450 0x7f2a280d05b0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 --2- 172.21.2.117:0/2140669021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280e4e40 0x7f2a281b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 -- 172.21.2.117:0/2140669021 >> 172.21.2.117:0/2140669021 conn(0x7f2a280ce4e0 msgr2=0x7f2a280e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 -- 172.21.2.117:0/2140669021 shutdown_connections 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.717 7f2a2cdae700 1 -- 172.21.2.117:0/2140669021 wait complete. 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 Processor -- start 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 -- start start 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a280973b0 0x7f2a28097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 0x7f2a280ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 0x7f2a2809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2818d0c0 con 0x7f2a280ad1b0 2021-02-10T11:48:28.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2809ae70 con 0x7f2a280973b0 2021-02-10T11:48:28.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a2cdae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2809ba10 con 0x7f2a280adbd0 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.732 7f2a26312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 0x7f2a280ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 0x7f2a2809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a26b13700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a280973b0 0x7f2a28097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a26312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 0x7f2a280ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53178/0 (socket says 172.21.2.117:53178) 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a26312700 1 -- 172.21.2.117:0/4276824325 learned_addr learned my addr 172.21.2.117:0/4276824325 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 -- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a280973b0 msgr2=0x7f2a28097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a280973b0 0x7f2a28097980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 -- 172.21.2.117:0/4276824325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 msgr2=0x7f2a280ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 0x7f2a280ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 -- 172.21.2.117:0/4276824325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2809bbf0 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a27314700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 0x7f2a2809b3f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f2a20006f30 tx=0x7f2a2000bca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2000bde0 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a20000f90 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2a2000c570 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2809bbf0 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.733 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2809bbf0 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.734 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2a20001270 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.734 7f2a1f7fe700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 0x7f2a0c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.734 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f2a20032910 con 0x7f2a280adbd0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.734 7f2a26b13700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 0x7f2a0c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:28.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.735 7f2a26b13700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 0x7f2a0c01abd0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:28.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.765 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a28085930 con 0x7f2a280adbd0 2021-02-10T11:48:28.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.769 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2a20003070 con 0x7f2a280adbd0 2021-02-10T11:48:28.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:28.898 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2a28085930 con 0x7f2a280adbd0 2021-02-10T11:48:29.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.257 7f2a1f7fe700 1 -- 172.21.2.117:0/4276824325 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2a2001f7e0 con 0x7f2a280adbd0 2021-02-10T11:48:31.965 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:48:31.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.264 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 msgr2=0x7f2a0c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:31.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.264 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 0x7f2a0c01abd0 crc :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:31.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.264 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 msgr2=0x7f2a2809b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:31.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.264 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 0x7f2a2809b3f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f2a20006f30 tx=0x7f2a2000bca0).stop 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 shutdown_connections 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a0c0186d0 0x7f2a0c01abd0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2a280ad1b0 0x7f2a280ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2a280adbd0 0x7f2a2809b3f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 --2- 172.21.2.117:0/4276824325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2a280973b0 0x7f2a28097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 >> 172.21.2.117:0/4276824325 conn(0x7f2a2818cc70 msgr2=0x7f2a28093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 shutdown_connections 2021-02-10T11:48:31.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:29.265 7f2a2cdae700 1 -- 172.21.2.117:0/4276824325 wait complete. 2021-02-10T11:48:31.967 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:48:31.967 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:48:31.967 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:48:31.967 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:31.967 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:48:31.967 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:48:31.967 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:48:31.968 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:48:31.973 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:48:31.974 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:48:31.975 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:48:31.975 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:48:32.016 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:48:32.016 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:32.017 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:48:32.017 DEBUG:teuthology.parallel:result is None 2021-02-10T11:48:32.017 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:48:32.422 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:48:32.425 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:48:32.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.436 7f67dd603700 1 Processor -- start 2021-02-10T11:48:32.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.436 7f67dd603700 1 -- start start 2021-02-10T11:48:32.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.436 7f67dd603700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d81b07f0 0x7f67d80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.436 7f67dd603700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80e7580 con 0x7f67d80e7d70 2021-02-10T11:48:32.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.436 7f67dd603700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80e77f0 con 0x7f67d80e7180 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67dd603700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80e92f0 con 0x7f67d81b07f0 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f67d80e7d70 0x7f67d80e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53448/0 (socket says 172.21.2.117:53448) 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 -- 172.21.2.117:0/56684871 learned_addr learned my addr 172.21.2.117:0/56684871 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191504922 0 0) 0x7f67c00009b0 con 0x7f67d80e7d70 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c4002fe0 con 0x7f67d80e7d70 2021-02-10T11:48:32.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d72bf700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d81b07f0 0x7f67d80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4175673451 0 0) 0x7f67c0001010 con 0x7f67d80e7d70 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d81b07f0 msgr2=0x7f67d80e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d81b07f0 0x7f67d80e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 >> v1:172.21.2.125:6789/0 conn(0x7f67d80e7180 legacy=0x7f67d80e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80e54b0 con 0x7f67d80e7d70 2021-02-10T11:48:32.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f67c0001590 con 0x7f67d80e7d70 2021-02-10T11:48:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 >> v1:172.21.2.117:6789/0 conn(0x7f67d80e7d70 legacy=0x7f67d80e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67c4003a60 0x7f67c4005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67c4006800 0x7f67c4008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67c4009620 0x7f67c400bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c400c2e0 con 0x7f67c4006800 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c400c500 con 0x7f67c4003a60 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4050730 con 0x7f67c4009620 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67c4006800 0x7f67c4008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d72bf700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67c4003a60 0x7f67c4005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 -- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67c4009620 msgr2=0x7f67c400bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67c4009620 0x7f67c400bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 -- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67c4003a60 msgr2=0x7f67c4005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67c4003a60 0x7f67c4005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 -- 172.21.2.117:0/56684871 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c4050990 con 0x7f67c4006800 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67ceabe700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67c4006800 0x7f67c4008d20 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7f67c0000b60 tx=0x7f67c0007d40).ready entity=mon.0 client_cookie=9cc58bc067a0537f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.437 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c0001010 con 0x7f67c4006800 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c00013d0 con 0x7f67c4006800 2021-02-10T11:48:32.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67d6abe700 1 -- 172.21.2.117:0/56684871 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c000a710 con 0x7f67c4006800 2021-02-10T11:48:32.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 -- 172.21.2.117:0/56684871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67c4006800 msgr2=0x7f67c4008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 --2- 172.21.2.117:0/56684871 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67c4006800 0x7f67c4008d20 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 rx=0x7f67c0000b60 tx=0x7f67c0007d40).stop 2021-02-10T11:48:32.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67d7ac0700 1 -- 172.21.2.117:0/56684871 reap_dead start 2021-02-10T11:48:32.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 -- 172.21.2.117:0/56684871 shutdown_connections 2021-02-10T11:48:32.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 -- 172.21.2.117:0/56684871 >> 172.21.2.117:0/56684871 conn(0x7f67d80ce500 msgr2=0x7f67d80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 -- 172.21.2.117:0/56684871 shutdown_connections 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.438 7f67dd603700 1 -- 172.21.2.117:0/56684871 wait complete. 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 Processor -- start 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 -- start start 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67d80b3090 0x7f67d80933f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d8093a10 0x7f67d8093e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b3830 con 0x7f67d819fe70 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b3490 con 0x7f67d80b3090 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.453 7f67dd603700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d80b3db0 con 0x7f67d8093a10 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53206/0 (socket says 172.21.2.117:53206) 2021-02-10T11:48:32.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 -- 172.21.2.117:0/2276763705 learned_addr learned my addr 172.21.2.117:0/2276763705 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:32.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67ceabe700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67d80b3090 0x7f67d80933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d7ac0700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d8093a10 0x7f67d8093e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 -- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d8093a10 msgr2=0x7f67d8093e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d8093a10 0x7f67d8093e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 -- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67d80b3090 msgr2=0x7f67d80933f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67d80b3090 0x7f67d80933f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 -- 172.21.2.117:0/2276763705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80b3fd0 con 0x7f67d819fe70 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d72bf700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f67c800b580 tx=0x7f67c800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c800bd10 con 0x7f67d819fe70 2021-02-10T11:48:32.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c8006740 con 0x7f67d819fe70 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67dd603700 1 -- 172.21.2.117:0/2276763705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d80b3fd0 con 0x7f67d819fe70 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67dd603700 1 -- 172.21.2.117:0/2276763705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d80b3fd0 con 0x7f67d819fe70 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.454 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f67c80022b0 con 0x7f67d819fe70 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.455 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f67c8000c90 con 0x7f67d819fe70 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.455 7f67d4aba700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 0x7f67bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:32.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.455 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f67c8032410 con 0x7f67d819fe70 2021-02-10T11:48:32.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.455 7f67ceabe700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 0x7f67bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:32.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.456 7f67ceabe700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 0x7f67bc01abc0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:32.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.485 7f67dd603700 1 -- 172.21.2.117:0/2276763705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d810c1a0 con 0x7f67d819fe70 2021-02-10T11:48:32.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.488 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f67c80279c0 con 0x7f67d819fe70 2021-02-10T11:48:32.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.629 7f67dd603700 1 -- 172.21.2.117:0/2276763705 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f67d80e77f0 con 0x7f67d819fe70 2021-02-10T11:48:32.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.683 7f67d4aba700 1 -- 172.21.2.117:0/2276763705 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v95) v1 ==== 97+0+0 (secure 0 0 0) 0x7f67c80279c0 con 0x7f67d819fe70 2021-02-10T11:48:32.689 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 msgr2=0x7f67bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 0x7f67bc01abc0 crc :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 msgr2=0x7f67d80b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f67c800b580 tx=0x7f67c800b920).stop 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 shutdown_connections 2021-02-10T11:48:32.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f67d8093a10 0x7f67d8093e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f67d80b3090 0x7f67d80933f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f67bc0186c0 0x7f67bc01abc0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 --2- 172.21.2.117:0/2276763705 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f67d819fe70 0x7f67d80b2a70 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 >> 172.21.2.117:0/2276763705 conn(0x7f67d80a7260 msgr2=0x7f67d80bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:32.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 shutdown_connections 2021-02-10T11:48:32.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:32.695 7f67dd603700 1 -- 172.21.2.117:0/2276763705 wait complete. 2021-02-10T11:48:32.699 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:48:32.700 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:48:33.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.114 7f70afed6700 1 Processor -- start 2021-02-10T11:48:33.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.114 7f70afed6700 1 -- start start 2021-02-10T11:48:33.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70afed6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70a80e5240 0x7f70a8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70afed6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a8084c60 0x7f70a8085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70afed6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8085930 con 0x7f70a80e4e40 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70afed6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a8085680 con 0x7f70a80e5240 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70afed6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a81b09e0 con 0x7f70a8084c60 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70ad4f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70a80e5240 0x7f70a8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70ad4f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70a80e5240 0x7f70a8086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53214/0 (socket says 172.21.2.117:53214) 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70ad4f1700 1 -- 172.21.2.117:0/1323242438 learned_addr learned my addr 172.21.2.117:0/1323242438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.115 7f70ad4f1700 1 -- 172.21.2.117:0/1323242438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a8084c60 msgr2=0x7f70a8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70ad4f1700 1 --2- 172.21.2.117:0/1323242438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a8084c60 0x7f70a8085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70ad4f1700 1 -- 172.21.2.117:0/1323242438 >> v1:172.21.2.125:6789/0 conn(0x7f70a80e4e40 legacy=0x7f70a81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70ad4f1700 1 -- 172.21.2.117:0/1323242438 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a8084840 con 0x7f70a80e5240 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70ad4f1700 1 --2- 172.21.2.117:0/1323242438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a80e5240 0x7f70a8086ec0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f7098006700 tx=0x7f7098009b80).ready entity=mon.0 client_cookie=935b4090ca6d3f9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70accf0700 1 -- 172.21.2.117:0/1323242438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709800c0c0 con 0x7f70a80e5240 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70accf0700 1 -- 172.21.2.117:0/1323242438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709800c340 con 0x7f70a80e5240 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70accf0700 1 -- 172.21.2.117:0/1323242438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709800c860 con 0x7f70a80e5240 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70afed6700 1 -- 172.21.2.117:0/1323242438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a80e5240 msgr2=0x7f70a8086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.116 7f70afed6700 1 --2- 172.21.2.117:0/1323242438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a80e5240 0x7f70a8086ec0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7f7098006700 tx=0x7f7098009b80).stop 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 -- 172.21.2.117:0/1323242438 shutdown_connections 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 --2- 172.21.2.117:0/1323242438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a8084c60 0x7f70a8085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 --2- 172.21.2.117:0/1323242438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a80e5240 0x7f70a8086ec0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 -- 172.21.2.117:0/1323242438 >> 172.21.2.117:0/1323242438 conn(0x7f70a80ce4e0 msgr2=0x7f70a80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 -- 172.21.2.117:0/1323242438 shutdown_connections 2021-02-10T11:48:33.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.117 7f70afed6700 1 -- 172.21.2.117:0/1323242438 wait complete. 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 Processor -- start 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 -- start start 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 0x7f70a818c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a80985b0 0x7f70a8095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8097a10 con 0x7f70a8190f20 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a818c720 con 0x7f70a80b0700 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70afed6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80989b0 con 0x7f70a80985b0 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70ad4f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70ad4f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53224/0 (socket says 172.21.2.117:53224) 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.132 7f70ad4f1700 1 -- 172.21.2.117:0/945569939 learned_addr learned my addr 172.21.2.117:0/945569939 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70adcf2700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 0x7f70a818c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ae4f3700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a80985b0 0x7f70a8095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 -- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a80985b0 msgr2=0x7f70a8095fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a80985b0 0x7f70a8095fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 -- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 msgr2=0x7f70a818c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 0x7f70a818c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 -- 172.21.2.117:0/945569939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80965f0 con 0x7f70a8190f20 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70adcf2700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 0x7f70a818c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:33.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70ad4f1700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f709800b650 tx=0x7f709800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f709800bde0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098006520 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f70980022b0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70afed6700 1 -- 172.21.2.117:0/945569939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a80965f0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.133 7f70afed6700 1 -- 172.21.2.117:0/945569939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a80965f0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.134 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f70980052e0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.134 7f709e7fc700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 0x7f709001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.134 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f70980324e0 con 0x7f70a8190f20 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.134 7f70adcf2700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 0x7f709001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.135 7f70adcf2700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 0x7f709001abc0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.165 7f70afed6700 1 -- 172.21.2.117:0/945569939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a810c250 con 0x7f70a8190f20 2021-02-10T11:48:33.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.167 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f70980064d0 con 0x7f70a8190f20 2021-02-10T11:48:33.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.295 7f70afed6700 1 -- 172.21.2.117:0/945569939 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f70a810c250 con 0x7f70a8190f20 2021-02-10T11:48:33.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.296 7f709e7fc700 1 -- 172.21.2.117:0/945569939 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v58) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f709800bde0 con 0x7f70a8190f20 2021-02-10T11:48:33.292 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.293 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.294 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:48:33.295 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.296 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:48:33.297 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.298 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.299 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.300 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.301 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:48:33.302 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:48:33.303 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:48:33.304 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.305 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:33.306 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:48:33.306 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 msgr2=0x7f709001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 0x7f709001abc0 crc :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 msgr2=0x7f70a8097f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f709800b650 tx=0x7f709800b9f0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 shutdown_connections 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7090018700 0x7f709001abc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70a8190f20 0x7f70a8097f90 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70a80985b0 0x7f70a8095fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 --2- 172.21.2.117:0/945569939 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70a80b0700 0x7f70a818c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 >> 172.21.2.117:0/945569939 conn(0x7f70a8199160 msgr2=0x7f70a808d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 shutdown_connections 2021-02-10T11:48:33.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.298 7f70afed6700 1 -- 172.21.2.117:0/945569939 wait complete. 2021-02-10T11:48:33.308 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:33.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.723 7f8e3143b700 1 Processor -- start 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e3143b700 1 -- start start 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e3143b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 0x7f8e2c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e3143b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c0ede60 con 0x7f8e2c0d18e0 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e3143b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2c0eb9c0 con 0x7f8e2c0ef3f0 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e3143b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c0d1ce0 con 0x7f8e2c1b6960 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e2a87c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f8e2c0ef3f0 0x7f8e2c0eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46404/0 (socket says 172.21.2.117:46404) 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e2a87c700 1 -- 172.21.2.117:0/2848974817 learned_addr learned my addr 172.21.2.117:0/2848974817 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.724 7f8e2b87e700 1 --2- 172.21.2.117:0/2848974817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 0x7f8e2c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783229508 0 0) 0x7f8e20000c60 con 0x7f8e2c0ef3f0 2021-02-10T11:48:33.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e14002fe0 con 0x7f8e2c0ef3f0 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036276542 0 0) 0x7f8e1c000ce0 con 0x7f8e2c0d18e0 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e14004370 con 0x7f8e2c0d18e0 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2b87e700 1 -- 172.21.2.117:0/2848974817 >> v1:172.21.2.125:6789/0 conn(0x7f8e2c0ef3f0 legacy=0x7f8e2c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2b87e700 1 -- 172.21.2.117:0/2848974817 >> v1:172.21.2.117:6789/0 conn(0x7f8e2c0d18e0 legacy=0x7f8e2c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2b87e700 1 -- 172.21.2.117:0/2848974817 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c1bb3b0 con 0x7f8e2c1b6960 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.725 7f8e2b87e700 1 --2- 172.21.2.117:0/2848974817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 0x7f8e2c1b8d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8e24006e50 tx=0x7f8e24009e70).ready entity=mon.2 client_cookie=289cd05bfbf5858 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e24006a20 con 0x7f8e2c1b6960 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e24006790 con 0x7f8e2c1b6960 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e2a07b700 1 -- 172.21.2.117:0/2848974817 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e2400b700 con 0x7f8e2c1b6960 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e3143b700 1 -- 172.21.2.117:0/2848974817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 msgr2=0x7f8e2c1b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e3143b700 1 --2- 172.21.2.117:0/2848974817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 0x7f8e2c1b8d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8e24006e50 tx=0x7f8e24009e70).stop 2021-02-10T11:48:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e3143b700 1 -- 172.21.2.117:0/2848974817 shutdown_connections 2021-02-10T11:48:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e3143b700 1 --2- 172.21.2.117:0/2848974817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c1b6960 0x7f8e2c1b8d70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.726 7f8e3143b700 1 -- 172.21.2.117:0/2848974817 >> 172.21.2.117:0/2848974817 conn(0x7f8e2c0ced60 msgr2=0x7f8e2c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.727 7f8e3143b700 1 -- 172.21.2.117:0/2848974817 shutdown_connections 2021-02-10T11:48:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.727 7f8e3143b700 1 -- 172.21.2.117:0/2848974817 wait complete. 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 Processor -- start 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 -- start start 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2c0ae480 0x7f8e2c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c08a280 0x7f8e2c08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c08b270 con 0x7f8e2c0979b0 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.741 7f8e3143b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c08aca0 con 0x7f8e2c0ae480 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e3143b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c08ae80 con 0x7f8e2c08a280 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53238/0 (socket says 172.21.2.117:53238) 2021-02-10T11:48:33.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 -- 172.21.2.117:0/3081930393 learned_addr learned my addr 172.21.2.117:0/3081930393 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2a87c700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2c0ae480 0x7f8e2c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b87e700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c08a280 0x7f8e2c08a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 -- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c08a280 msgr2=0x7f8e2c08a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c08a280 0x7f8e2c08a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 -- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2c0ae480 msgr2=0x7f8e2c0ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2c0ae480 0x7f8e2c0ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 -- 172.21.2.117:0/3081930393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2c08b7f0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e2b07d700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f8e1c00b5b0 tx=0x7f8e1c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.742 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c006240 con 0x7f8e2c0979b0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c0054a0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c08b7f0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c0058f0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c08bd80 con 0x7f8e2c0979b0 2021-02-10T11:48:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e1c006030 con 0x7f8e2c0979b0 2021-02-10T11:48:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e13fff700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 0x7f8e0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f8e1c0322e0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.743 7f8e2a87c700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 0x7f8e0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:33.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.744 7f8e2a87c700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 0x7f8e0c01abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:33.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.774 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c08e440 con 0x7f8e2c0979b0 2021-02-10T11:48:33.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.777 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e1c030df0 con 0x7f8e2c0979b0 2021-02-10T11:48:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.900 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e2c08e440 con 0x7f8e2c0979b0 2021-02-10T11:48:33.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.901 7f8e13fff700 1 -- 172.21.2.117:0/3081930393 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+11378 (secure 0 0 0) 0x7f8e1c006030 con 0x7f8e2c0979b0 2021-02-10T11:48:33.898 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:33.898 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":95,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:32.681213","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 msgr2=0x7f8e0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 0x7f8e0c01abc0 crc :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 msgr2=0x7f8e2c08ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7f8e1c00b5b0 tx=0x7f8e1c00b950).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 shutdown_connections 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e2c08a280 0x7f8e2c08a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2c0ae480 0x7f8e2c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e0c0186c0 0x7f8e0c01abc0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 --2- 172.21.2.117:0/3081930393 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e2c0979b0 0x7f8e2c08ddd0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:33.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 >> 172.21.2.117:0/3081930393 conn(0x7f8e2c1a3620 msgr2=0x7f8e2c1a93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:33.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 shutdown_connections 2021-02-10T11:48:33.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:33.903 7f8e3143b700 1 -- 172.21.2.117:0/3081930393 wait complete. 2021-02-10T11:48:33.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:48:33.908 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:34.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.326 7fe4295b5700 1 Processor -- start 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.326 7fe4295b5700 1 -- start start 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4295b5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe4240e7180 0x7fe4240e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4295b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 0x7fe4240e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4295b5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4240e7580 con 0x7fe4241b07f0 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4295b5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4240e5ce0 con 0x7fe4240e7180 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4295b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240e77f0 con 0x7fe4240e8390 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4232bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe4241b07f0 0x7fe4240e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53492/0 (socket says 172.21.2.117:53492) 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe422abe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe4240e7180 0x7fe4240e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe423ac0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 0x7fe4240e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.327 7fe4232bf700 1 -- 172.21.2.117:0/832353893 learned_addr learned my addr 172.21.2.117:0/832353893 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:34.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4222bd700 1 -- 172.21.2.117:0/832353893 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501294102 0 0) 0x7fe4140009b0 con 0x7fe4241b07f0 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe423ac0700 1 -- 172.21.2.117:0/832353893 >> v2:172.21.2.125:3300/0 conn(0x7fe4240e7180 msgr2=0x7fe4240e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe423ac0700 1 --2- 172.21.2.117:0/832353893 >> v2:172.21.2.125:3300/0 conn(0x7fe4240e7180 0x7fe4240e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe423ac0700 1 -- 172.21.2.117:0/832353893 >> v1:172.21.2.117:6789/0 conn(0x7fe4241b07f0 legacy=0x7fe4240e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe423ac0700 1 -- 172.21.2.117:0/832353893 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240e8db0 con 0x7fe4240e8390 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe423ac0700 1 --2- 172.21.2.117:0/832353893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 0x7fe4240e8790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe41c006e50 tx=0x7fe41c009e70).ready entity=mon.2 client_cookie=d556a0557f198489 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4222bd700 1 -- 172.21.2.117:0/832353893 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c00b510 con 0x7fe4240e8390 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4222bd700 1 -- 172.21.2.117:0/832353893 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c006790 con 0x7fe4240e8390 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4222bd700 1 -- 172.21.2.117:0/832353893 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c00b510 con 0x7fe4240e8390 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4295b5700 1 -- 172.21.2.117:0/832353893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 msgr2=0x7fe4240e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.328 7fe4295b5700 1 --2- 172.21.2.117:0/832353893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 0x7fe4240e8790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe41c006e50 tx=0x7fe41c009e70).stop 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 -- 172.21.2.117:0/832353893 shutdown_connections 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 --2- 172.21.2.117:0/832353893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4240e8390 0x7fe4240e8790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 --2- 172.21.2.117:0/832353893 >> v2:172.21.2.125:3300/0 conn(0x7fe4240e7180 0x7fe4240e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 -- 172.21.2.117:0/832353893 >> 172.21.2.117:0/832353893 conn(0x7fe4240ce500 msgr2=0x7fe4240e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:34.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 -- 172.21.2.117:0/832353893 shutdown_connections 2021-02-10T11:48:34.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.329 7fe4295b5700 1 -- 172.21.2.117:0/832353893 wait complete. 2021-02-10T11:48:34.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 Processor -- start 2021-02-10T11:48:34.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 -- start start 2021-02-10T11:48:34.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe42408f410 0x7fe4241a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4241a2dd0 0x7fe4240bad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe424095b70 con 0x7fe4241a31d0 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42408f810 con 0x7fe4241a2dd0 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe4295b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4240bb980 con 0x7fe42408f410 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe423ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe423ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53248/0 (socket says 172.21.2.117:53248) 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.344 7fe423ac0700 1 -- 172.21.2.117:0/357503160 learned_addr learned my addr 172.21.2.117:0/357503160 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 -- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe42408f410 msgr2=0x7fe4241a27b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe42408f410 0x7fe4241a27b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 -- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4241a2dd0 msgr2=0x7fe4240bad40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4241a2dd0 0x7fe4240bad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 -- 172.21.2.117:0/357503160 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240bbba0 con 0x7fe4241a31d0 2021-02-10T11:48:34.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe423ac0700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7fe41c006c50 tx=0x7fe41c00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c0093d0 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c00be60 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c00c570 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe4295b5700 1 -- 172.21.2.117:0/357503160 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4240bbba0 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.345 7fe4295b5700 1 -- 172.21.2.117:0/357503160 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4240bbba0 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.346 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe41c002ef0 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.346 7fe40bfff700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.346 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fe41c032790 con 0x7fe4241a31d0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.346 7fe4232bf700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.346 7fe4232bf700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 0x7fe40401abd0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.377 7fe4295b5700 1 -- 172.21.2.117:0/357503160 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424095b70 con 0x7fe4241a31d0 2021-02-10T11:48:34.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.380 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe41c009550 con 0x7fe4241a31d0 2021-02-10T11:48:34.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.506 7fe4295b5700 1 -- 172.21.2.117:0/357503160 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe424095b70 con 0x7fe4241a31d0 2021-02-10T11:48:34.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.506 7fe40bfff700 1 -- 172.21.2.117:0/357503160 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v95) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe41c009550 con 0x7fe4241a31d0 2021-02-10T11:48:34.503 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:34.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 -- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 msgr2=0x7fe40401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 0x7fe40401abd0 crc :-1 s=READY pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 -- 172.21.2.117:0/357503160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 msgr2=0x7fe4240bb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7fe41c006c50 tx=0x7fe41c00bd20).stop 2021-02-10T11:48:34.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 -- 172.21.2.117:0/357503160 shutdown_connections 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4040186d0 0x7fe40401abd0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4241a31d0 0x7fe4240bb360 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4241a2dd0 0x7fe4240bad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 --2- 172.21.2.117:0/357503160 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe42408f410 0x7fe4241a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.508 7fe4295b5700 1 -- 172.21.2.117:0/357503160 >> 172.21.2.117:0/357503160 conn(0x7fe42408fb00 msgr2=0x7fe42408fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.509 7fe4295b5700 1 -- 172.21.2.117:0/357503160 shutdown_connections 2021-02-10T11:48:34.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.509 7fe4295b5700 1 -- 172.21.2.117:0/357503160 wait complete. 2021-02-10T11:48:34.513 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:48:34.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.929 7efd1b296700 1 Processor -- start 2021-02-10T11:48:34.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.929 7efd1b296700 1 -- start start 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd1b296700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efd140ef3f0 0x7efd140eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd1b296700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efd140eb7f0 0x7efd141b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd1b296700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140ede60 con 0x7efd140d18e0 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd1b296700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd140ef7f0 con 0x7efd140eb7f0 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd1b296700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd140d1ce0 con 0x7efd140ef3f0 2021-02-10T11:48:34.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd190b2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efd140d18e0 0x7efd140ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53504/0 (socket says 172.21.2.117:53504) 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd190b2700 1 -- 172.21.2.117:0/1864238576 learned_addr learned my addr 172.21.2.117:0/1864238576 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd198b3700 1 --2- 172.21.2.117:0/1864238576 >> v2:172.21.2.117:3300/0 conn(0x7efd140eb7f0 0x7efd141b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.930 7efd188b1700 1 --2- 172.21.2.117:0/1864238576 >> v2:172.21.2.125:3300/0 conn(0x7efd140ef3f0 0x7efd140eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd198b3700 1 -- 172.21.2.117:0/1864238576 >> v2:172.21.2.125:3300/0 conn(0x7efd140ef3f0 msgr2=0x7efd140eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd198b3700 1 --2- 172.21.2.117:0/1864238576 >> v2:172.21.2.125:3300/0 conn(0x7efd140ef3f0 0x7efd140eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd198b3700 1 -- 172.21.2.117:0/1864238576 >> v1:172.21.2.117:6789/0 conn(0x7efd140d18e0 legacy=0x7efd140ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd198b3700 1 -- 172.21.2.117:0/1864238576 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd141b8f90 con 0x7efd140eb7f0 2021-02-10T11:48:34.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd198b3700 1 --2- 172.21.2.117:0/1864238576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd140eb7f0 0x7efd141b7960 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7efd0c006e30 tx=0x7efd0c009e60).ready entity=mon.0 client_cookie=b869b044e3c30d4c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd0bfff700 1 -- 172.21.2.117:0/1864238576 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd0c003030 con 0x7efd140eb7f0 2021-02-10T11:48:34.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd0bfff700 1 -- 172.21.2.117:0/1864238576 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0c00b300 con 0x7efd140eb7f0 2021-02-10T11:48:34.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd0bfff700 1 -- 172.21.2.117:0/1864238576 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd0c00b820 con 0x7efd140eb7f0 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd1b296700 1 -- 172.21.2.117:0/1864238576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd140eb7f0 msgr2=0x7efd141b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.931 7efd1b296700 1 --2- 172.21.2.117:0/1864238576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd140eb7f0 0x7efd141b7960 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7efd0c006e30 tx=0x7efd0c009e60).stop 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 -- 172.21.2.117:0/1864238576 shutdown_connections 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 --2- 172.21.2.117:0/1864238576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd140eb7f0 0x7efd141b7960 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 --2- 172.21.2.117:0/1864238576 >> v2:172.21.2.125:3300/0 conn(0x7efd140ef3f0 0x7efd140eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:34.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 -- 172.21.2.117:0/1864238576 >> 172.21.2.117:0/1864238576 conn(0x7efd140ced60 msgr2=0x7efd140cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:34.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 -- 172.21.2.117:0/1864238576 shutdown_connections 2021-02-10T11:48:34.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.932 7efd1b296700 1 -- 172.21.2.117:0/1864238576 wait complete. 2021-02-10T11:48:34.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 Processor -- start 2021-02-10T11:48:34.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 -- start start 2021-02-10T11:48:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd141b24b0 0x7efd141b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd14198250 0x7efd14198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd14197cd0 con 0x7efd14091d40 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd14092140 con 0x7efd14198250 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.947 7efd1b296700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd141b2ed0 con 0x7efd141b24b0 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53262/0 (socket says 172.21.2.117:53262) 2021-02-10T11:48:34.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 -- 172.21.2.117:0/2242017488 learned_addr learned my addr 172.21.2.117:0/2242017488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 -- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd141b24b0 msgr2=0x7efd141b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd141b24b0 0x7efd141b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 -- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd14198250 msgr2=0x7efd14198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd14198250 0x7efd14198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:34.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 -- 172.21.2.117:0/2242017488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd14198c70 con 0x7efd14091d40 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd190b2700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7efd10001410 tx=0x7efd1000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd1000ad40 con 0x7efd14091d40 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd100043a0 con 0x7efd14091d40 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd1b296700 1 -- 172.21.2.117:0/2242017488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd14198c70 con 0x7efd14091d40 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd1b296700 1 -- 172.21.2.117:0/2242017488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd14198c70 con 0x7efd14091d40 2021-02-10T11:48:34.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.948 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efd100047b0 con 0x7efd14091d40 2021-02-10T11:48:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.949 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efd10004c80 con 0x7efd14091d40 2021-02-10T11:48:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.949 7efd09ffb700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 0x7efcf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.949 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5100+0+0 (secure 0 0 0) 0x7efd10032400 con 0x7efd14091d40 2021-02-10T11:48:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.949 7efd188b1700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 0x7efcf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:34.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.950 7efd188b1700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 0x7efcf401abc0 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:34.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.980 7efd1b296700 1 -- 172.21.2.117:0/2242017488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd14087290 con 0x7efd14091d40 2021-02-10T11:48:34.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:34.983 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efd10004a70 con 0x7efd14091d40 2021-02-10T11:48:35.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.106 7efd1b296700 1 -- 172.21.2.117:0/2242017488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efd14087290 con 0x7efd14091d40 2021-02-10T11:48:35.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.106 7efd09ffb700 1 -- 172.21.2.117:0/2242017488 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+64 (secure 0 0 0) 0x7efd1000ad40 con 0x7efd14091d40 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:35.103 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:48:35.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 msgr2=0x7efcf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 0x7efcf401abc0 crc :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 msgr2=0x7efd140e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7efd10001410 tx=0x7efd1000a950).stop 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 shutdown_connections 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd14198250 0x7efd14198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd141b24b0 0x7efd141b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcf40186c0 0x7efcf401abc0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 --2- 172.21.2.117:0/2242017488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd14091d40 0x7efd140e71e0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 >> 172.21.2.117:0/2242017488 conn(0x7efd141a3620 msgr2=0x7efd140dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 shutdown_connections 2021-02-10T11:48:35.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.108 7efd1b296700 1 -- 172.21.2.117:0/2242017488 wait complete. 2021-02-10T11:48:35.112 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:48:35.113 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:48:35.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 Processor -- start 2021-02-10T11:48:35.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 -- start start 2021-02-10T11:48:35.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff240ef3f0 0x7eff240eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7eff240eb830 0x7eff241b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff240ef7f0 con 0x7eff240d18f0 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240ede60 con 0x7eff240ef3f0 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff2ac0e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240d1cf0 con 0x7eff240eb830 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.531 7eff23fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff240ef3f0 0x7eff240eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7eff240ef3f0 0x7eff240eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53268/0 (socket says 172.21.2.117:53268) 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 -- 172.21.2.117:0/3163033552 learned_addr learned my addr 172.21.2.117:0/3163033552 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff2922b700 1 --2- 172.21.2.117:0/3163033552 >> v2:172.21.2.125:3300/0 conn(0x7eff240eb830 0x7eff241b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 -- 172.21.2.117:0/3163033552 >> v2:172.21.2.125:3300/0 conn(0x7eff240eb830 msgr2=0x7eff241b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 --2- 172.21.2.117:0/3163033552 >> v2:172.21.2.125:3300/0 conn(0x7eff240eb830 0x7eff241b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 -- 172.21.2.117:0/3163033552 >> v1:172.21.2.125:6789/0 conn(0x7eff240d18f0 legacy=0x7eff240eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 -- 172.21.2.117:0/3163033552 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff241b8fd0 con 0x7eff240ef3f0 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.532 7eff23fff700 1 --2- 172.21.2.117:0/3163033552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff240ef3f0 0x7eff240eb210 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7eff14006a80 tx=0x7eff14009f00).ready entity=mon.0 client_cookie=35fa80e4ca5f2bdd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff237fe700 1 -- 172.21.2.117:0/3163033552 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1400b5c0 con 0x7eff240ef3f0 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff237fe700 1 -- 172.21.2.117:0/3163033552 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1400b8c0 con 0x7eff240ef3f0 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff237fe700 1 -- 172.21.2.117:0/3163033552 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1400bde0 con 0x7eff240ef3f0 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 -- 172.21.2.117:0/3163033552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff240ef3f0 msgr2=0x7eff240eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 --2- 172.21.2.117:0/3163033552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff240ef3f0 0x7eff240eb210 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7eff14006a80 tx=0x7eff14009f00).stop 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 -- 172.21.2.117:0/3163033552 shutdown_connections 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 --2- 172.21.2.117:0/3163033552 >> v2:172.21.2.125:3300/0 conn(0x7eff240eb830 0x7eff241b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 --2- 172.21.2.117:0/3163033552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff240ef3f0 0x7eff240eb210 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 -- 172.21.2.117:0/3163033552 >> 172.21.2.117:0/3163033552 conn(0x7eff240cf060 msgr2=0x7eff240cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 -- 172.21.2.117:0/3163033552 shutdown_connections 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.533 7eff2ac0e700 1 -- 172.21.2.117:0/3163033552 wait complete. 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.548 7eff2ac0e700 1 Processor -- start 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.548 7eff2ac0e700 1 -- start start 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff240a8eb0 0x7eff240a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 0x7eff2407ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240a9cf0 con 0x7eff2409f5f0 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2409f9f0 con 0x7eff240a98f0 2021-02-10T11:48:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2ac0e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2409fc10 con 0x7eff240a8eb0 2021-02-10T11:48:35.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53272/0 (socket says 172.21.2.117:53272) 2021-02-10T11:48:35.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 -- 172.21.2.117:0/1848987375 learned_addr learned my addr 172.21.2.117:0/1848987375 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2922b700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 0x7eff2407ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 -- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff240a8eb0 msgr2=0x7eff240a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff240a8eb0 0x7eff240a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 -- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 msgr2=0x7eff2407ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 0x7eff2407ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 -- 172.21.2.117:0/1848987375 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2407f310 con 0x7eff2409f5f0 2021-02-10T11:48:35.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff2922b700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 0x7eff2407ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.549 7eff23fff700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7eff14009450 tx=0x7eff1400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.550 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1400c2f0 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.550 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1400be60 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.550 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7eff1400c720 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.550 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2407f310 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.550 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff241a2df0 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.551 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7eff14005660 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.551 7eff217fa700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 0x7eff0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.551 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5100+0+0 (secure 0 0 0) 0x7eff14032730 con 0x7eff2409f5f0 2021-02-10T11:48:35.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.551 7eff28a2a700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 0x7eff0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:35.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.553 7eff28a2a700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 0x7eff0801abc0 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:35.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.581 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2409fc10 con 0x7eff2409f5f0 2021-02-10T11:48:35.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.584 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7eff14005960 con 0x7eff2409f5f0 2021-02-10T11:48:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:35.711 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7eff2409fc10 con 0x7eff2409f5f0 2021-02-10T11:48:36.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.435 7eff217fa700 1 -- 172.21.2.117:0/1848987375 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v96) v1 ==== 133+0+0 (secure 0 0 0) 0x7eff14005660 con 0x7eff2409f5f0 2021-02-10T11:48:36.476 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:48:36.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 msgr2=0x7eff0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 0x7eff0801abc0 crc :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 msgr2=0x7eff240a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7eff14009450 tx=0x7eff1400bd20).stop 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 shutdown_connections 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.482 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff080186c0 0x7eff0801abc0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7eff2409f5f0 0x7eff240a92d0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7eff240a98f0 0x7eff2407ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 --2- 172.21.2.117:0/1848987375 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7eff240a8eb0 0x7eff240a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 >> 172.21.2.117:0/1848987375 conn(0x7eff240a71e0 msgr2=0x7eff240a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 shutdown_connections 2021-02-10T11:48:36.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.483 7eff2ac0e700 1 -- 172.21.2.117:0/1848987375 wait complete. 2021-02-10T11:48:36.487 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:48:36.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 Processor -- start 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 -- start start 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9000e7180 0x7fd9000e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9000e8390 0x7fd9000e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9000e7580 con 0x7fd9001b07f0 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9000e5ce0 con 0x7fd9000e8390 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd9057f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000e77f0 con 0x7fd9000e7180 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd8ffd75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9000e8390 0x7fd9000e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd8ffd75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9000e8390 0x7fd9000e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53284/0 (socket says 172.21.2.117:53284) 2021-02-10T11:48:36.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd8ffd75700 1 -- 172.21.2.117:0/1537948082 learned_addr learned my addr 172.21.2.117:0/1537948082 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.905 7fd8fed73700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9000e7180 0x7fd9000e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8ffd75700 1 -- 172.21.2.117:0/1537948082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9000e7180 msgr2=0x7fd9000e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8ffd75700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9000e7180 0x7fd9000e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8ffd75700 1 -- 172.21.2.117:0/1537948082 >> v1:172.21.2.125:6789/0 conn(0x7fd9001b07f0 legacy=0x7fd9000e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8ffd75700 1 -- 172.21.2.117:0/1537948082 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000e8db0 con 0x7fd9000e8390 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8ffd75700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000e8390 0x7fd9000e8790 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7fd8f8006f00 tx=0x7fd8f8009f10).ready entity=mon.0 client_cookie=482f1a6878226c59 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8fe572700 1 -- 172.21.2.117:0/1537948082 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8f800b3b0 con 0x7fd9000e8390 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8fe572700 1 -- 172.21.2.117:0/1537948082 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f800b630 con 0x7fd9000e8390 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd8fe572700 1 -- 172.21.2.117:0/1537948082 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8f800bb50 con 0x7fd9000e8390 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd9057f2700 1 -- 172.21.2.117:0/1537948082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000e8390 msgr2=0x7fd9000e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.906 7fd9057f2700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000e8390 0x7fd9000e8790 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7fd8f8006f00 tx=0x7fd8f8009f10).stop 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 -- 172.21.2.117:0/1537948082 shutdown_connections 2021-02-10T11:48:36.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000e8390 0x7fd9000e8790 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 --2- 172.21.2.117:0/1537948082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9000e7180 0x7fd9000e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 -- 172.21.2.117:0/1537948082 >> 172.21.2.117:0/1537948082 conn(0x7fd9000ce500 msgr2=0x7fd9000e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:36.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 -- 172.21.2.117:0/1537948082 shutdown_connections 2021-02-10T11:48:36.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.907 7fd9057f2700 1 -- 172.21.2.117:0/1537948082 wait complete. 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 Processor -- start 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 -- start start 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd90019a340 0x7fd9000c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90019a740 0x7fd9000c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000a5c70 con 0x7fd9000a7720 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd900095110 con 0x7fd90019a340 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.922 7fd9057f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000c4060 con 0x7fd90019a740 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53288/0 (socket says 172.21.2.117:53288) 2021-02-10T11:48:36.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 -- 172.21.2.117:0/3666151532 learned_addr learned my addr 172.21.2.117:0/3666151532 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8fed73700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd90019a340 0x7fd9000c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ffd75700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90019a740 0x7fd9000c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 -- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90019a740 msgr2=0x7fd9000c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90019a740 0x7fd9000c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 -- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd90019a340 msgr2=0x7fd9000c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd90019a340 0x7fd9000c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 -- 172.21.2.117:0/3666151532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000c3bb0 con 0x7fd9000a7720 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8ff574700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7fd8e8006ed0 tx=0x7fd8e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e8006020 con 0x7fd9000a7720 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e80009b0 con 0x7fd9000a7720 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9000c3bb0 con 0x7fd9000a7720 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8e80054b0 con 0x7fd9000a7720 2021-02-10T11:48:36.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.923 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9000a64e0 con 0x7fd9000a7720 2021-02-10T11:48:36.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.924 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8e8002050 con 0x7fd9000a7720 2021-02-10T11:48:36.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.924 7fd8f7fff700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 0x7fd8e401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:36.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.924 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5454+0+0 (secure 0 0 0) 0x7fd8e8032520 con 0x7fd9000a7720 2021-02-10T11:48:36.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.925 7fd8fed73700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 0x7fd8e401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:36.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.925 7fd8fed73700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 0x7fd8e401abc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:36.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.955 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90007e800 con 0x7fd9000a7720 2021-02-10T11:48:36.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:36.958 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8e8030d50 con 0x7fd9000a7720 2021-02-10T11:48:37.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.084 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fd9000e77f0 con 0x7fd9000a7720 2021-02-10T11:48:37.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.445 7fd8f7fff700 1 -- 172.21.2.117:0/3666151532 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v98) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd8e8002050 con 0x7fd9000a7720 2021-02-10T11:48:37.446 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 msgr2=0x7fd8e401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 0x7fd8e401abc0 crc :-1 s=READY pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 msgr2=0x7fd900094af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7fd8e8006ed0 tx=0x7fd8e800b960).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 shutdown_connections 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90019a740 0x7fd9000c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd90019a340 0x7fd9000c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e40186c0 0x7fd8e401abc0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 --2- 172.21.2.117:0/3666151532 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9000a7720 0x7fd900094af0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.451 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 >> 172.21.2.117:0/3666151532 conn(0x7fd9000a7210 msgr2=0x7fd900095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.452 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 shutdown_connections 2021-02-10T11:48:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.452 7fd9057f2700 1 -- 172.21.2.117:0/3666151532 wait complete. 2021-02-10T11:48:37.456 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:48:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.873 7f60a7797700 1 Processor -- start 2021-02-10T11:48:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 -- start start 2021-02-10T11:48:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a00e4e40 0x7f60a01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60a0085450 0x7f60a00d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a0085930 con 0x7f60a00e5240 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a00e5cc0 con 0x7f60a0085450 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a7797700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0084840 con 0x7f60a00e4e40 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a5db4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60a0085450 0x7f60a00d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a5db4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60a0085450 0x7f60a00d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53294/0 (socket says 172.21.2.117:53294) 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a5db4700 1 -- 172.21.2.117:0/2887391438 learned_addr learned my addr 172.21.2.117:0/2887391438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.874 7f60a55b3700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a00e4e40 0x7f60a01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a5db4700 1 -- 172.21.2.117:0/2887391438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a00e4e40 msgr2=0x7f60a01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a5db4700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a00e4e40 0x7f60a01b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a5db4700 1 -- 172.21.2.117:0/2887391438 >> v1:172.21.2.125:6789/0 conn(0x7f60a00e5240 legacy=0x7f60a0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f6093fff700 1 -- 172.21.2.117:0/2887391438 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934328468 0 0) 0x7f609c000ff0 con 0x7f60a00e5240 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a5db4700 1 -- 172.21.2.117:0/2887391438 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00d1be0 con 0x7f60a0085450 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a5db4700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a0085450 0x7f60a00d05b0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7f6098006f00 tx=0x7f6098009e90).ready entity=mon.0 client_cookie=229af05849c7283b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f6093fff700 1 -- 172.21.2.117:0/2887391438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609800b300 con 0x7f60a0085450 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f6093fff700 1 -- 172.21.2.117:0/2887391438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609800b580 con 0x7f60a0085450 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f6093fff700 1 -- 172.21.2.117:0/2887391438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609800baa0 con 0x7f60a0085450 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a7797700 1 -- 172.21.2.117:0/2887391438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a0085450 msgr2=0x7f60a00d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.875 7f60a7797700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a0085450 0x7f60a00d05b0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7f6098006f00 tx=0x7f6098009e90).stop 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 -- 172.21.2.117:0/2887391438 shutdown_connections 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a0085450 0x7f60a00d05b0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 --2- 172.21.2.117:0/2887391438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a00e4e40 0x7f60a01b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 -- 172.21.2.117:0/2887391438 >> 172.21.2.117:0/2887391438 conn(0x7f60a00ce4e0 msgr2=0x7f60a00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 -- 172.21.2.117:0/2887391438 shutdown_connections 2021-02-10T11:48:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.877 7f60a7797700 1 -- 172.21.2.117:0/2887391438 wait complete. 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.890 7f60a7797700 1 Processor -- start 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.890 7f60a7797700 1 -- start start 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a008a8a0 0x7f60a008aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60a01a7060 0x7f60a01a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a01a7a80 con 0x7f60a00abe50 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a01a7ca0 con 0x7f60a01a7060 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a7797700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a01a7ec0 con 0x7f60a008a8a0 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a55b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a55b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53304/0 (socket says 172.21.2.117:53304) 2021-02-10T11:48:37.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a4db2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a008a8a0 0x7f60a008aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a55b3700 1 -- 172.21.2.117:0/3320770669 learned_addr learned my addr 172.21.2.117:0/3320770669 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.891 7f60a5db4700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60a01a7060 0x7f60a01a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 -- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a008a8a0 msgr2=0x7f60a008aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a008a8a0 0x7f60a008aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 -- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60a01a7060 msgr2=0x7f60a01a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60a01a7060 0x7f60a01a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 -- 172.21.2.117:0/3320770669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a00b7030 con 0x7f60a00abe50 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a55b3700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f6094006f60 tx=0x7f609400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f609400bd60 con 0x7f60a00abe50 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094000c90 con 0x7f60a00abe50 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60940022b0 con 0x7f60a00abe50 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a7797700 1 -- 172.21.2.117:0/3320770669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a00b7030 con 0x7f60a00abe50 2021-02-10T11:48:37.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.892 7f60a7797700 1 -- 172.21.2.117:0/3320770669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a00b7030 con 0x7f60a00abe50 2021-02-10T11:48:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.893 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6094006b00 con 0x7f60a00abe50 2021-02-10T11:48:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.893 7f6091ffb700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.893 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f6094032630 con 0x7f60a00abe50 2021-02-10T11:48:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.893 7f60a4db2700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:37.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.894 7f60a4db2700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 0x7f608001abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:37.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.924 7f60a7797700 1 -- 172.21.2.117:0/3320770669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a010c2b0 con 0x7f60a00abe50 2021-02-10T11:48:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:37.927 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6094030d00 con 0x7f60a00abe50 2021-02-10T11:48:38.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.053 7f60a7797700 1 -- 172.21.2.117:0/3320770669 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f60a010c2b0 con 0x7f60a00abe50 2021-02-10T11:48:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.057 7f6091ffb700 1 -- 172.21.2.117:0/3320770669 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 20 and data pool 21 v74) v1 ==== 189+0+0 (secure 0 0 0) 0x7f6094006b00 con 0x7f60a00abe50 2021-02-10T11:48:38.056 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 20 and data pool 21 2021-02-10T11:48:38.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.062 7f60a7797700 1 -- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 msgr2=0x7f608001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.062 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 0x7f608001abc0 crc :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.062 7f60a7797700 1 -- 172.21.2.117:0/3320770669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 msgr2=0x7f60a008a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.062 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f6094006f60 tx=0x7f609400b970).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 -- 172.21.2.117:0/3320770669 shutdown_connections 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60a01a7060 0x7f60a01a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60a008a8a0 0x7f60a008aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60800186c0 0x7f608001abc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 --2- 172.21.2.117:0/3320770669 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60a00abe50 0x7f60a008a280 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 -- 172.21.2.117:0/3320770669 >> 172.21.2.117:0/3320770669 conn(0x7f60a00acec0 msgr2=0x7f60a00b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 -- 172.21.2.117:0/3320770669 shutdown_connections 2021-02-10T11:48:38.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.063 7f60a7797700 1 -- 172.21.2.117:0/3320770669 wait complete. 2021-02-10T11:48:38.067 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 Processor -- start 2021-02-10T11:48:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 -- start start 2021-02-10T11:48:38.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fca200d18e0 0x7fca200ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200eb000 0x7fca200eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca200ede60 con 0x7fca200ee3e0 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca200eba90 con 0x7fca200d18e0 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca2586c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca200d1ce0 con 0x7fca200eb000 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.483 7fca1f574700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fca200d18e0 0x7fca200ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1fd75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fca200ee3e0 0x7fca200ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53562/0 (socket says 172.21.2.117:53562) 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1fd75700 1 -- 172.21.2.117:0/2980236092 learned_addr learned my addr 172.21.2.117:0/2980236092 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1ed73700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200eb000 0x7fca200eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1f574700 1 -- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200eb000 msgr2=0x7fca200eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1f574700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200eb000 0x7fca200eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1f574700 1 -- 172.21.2.117:0/2980236092 >> v1:172.21.2.117:6789/0 conn(0x7fca200ee3e0 legacy=0x7fca200ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1f574700 1 -- 172.21.2.117:0/2980236092 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200eefd0 con 0x7fca200d18e0 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1f574700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200d18e0 0x7fca200ea9e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fca180069d0 tx=0x7fca18009e40).ready entity=mon.1 client_cookie=b9fcda32a6c1f7b6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1e572700 1 -- 172.21.2.117:0/2980236092 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1800b510 con 0x7fca200d18e0 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1e572700 1 -- 172.21.2.117:0/2980236092 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1800b890 con 0x7fca200d18e0 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca1e572700 1 -- 172.21.2.117:0/2980236092 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca1800bdb0 con 0x7fca200d18e0 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca2586c700 1 -- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200d18e0 msgr2=0x7fca200ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.484 7fca2586c700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200d18e0 0x7fca200ea9e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7fca180069d0 tx=0x7fca18009e40).stop 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 -- 172.21.2.117:0/2980236092 shutdown_connections 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200eb000 0x7fca200eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 --2- 172.21.2.117:0/2980236092 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200d18e0 0x7fca200ea9e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 -- 172.21.2.117:0/2980236092 >> 172.21.2.117:0/2980236092 conn(0x7fca200ced60 msgr2=0x7fca200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 -- 172.21.2.117:0/2980236092 shutdown_connections 2021-02-10T11:48:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.485 7fca2586c700 1 -- 172.21.2.117:0/2980236092 wait complete. 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 Processor -- start 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 -- start start 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200e1840 0x7fca201a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200e7ef0 0x7fca201a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca200e8430 con 0x7fca200e74d0 2021-02-10T11:48:38.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca201a2280 con 0x7fca200e7ef0 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca2586c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca201a9350 con 0x7fca200e1840 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca1ed73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca1ed73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53316/0 (socket says 172.21.2.117:53316) 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca1ed73700 1 -- 172.21.2.117:0/4250503120 learned_addr learned my addr 172.21.2.117:0/4250503120 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca1fd75700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200e7ef0 0x7fca201a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.500 7fca1f574700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200e1840 0x7fca201a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 -- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200e1840 msgr2=0x7fca201a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200e1840 0x7fca201a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 -- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200e7ef0 msgr2=0x7fca201a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200e7ef0 0x7fca201a8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 -- 172.21.2.117:0/4250503120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca201a9570 con 0x7fca200e74d0 2021-02-10T11:48:38.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1ed73700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7fca14006f60 tx=0x7fca1400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca140009b0 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1400bdf0 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca14002030 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca2586c700 1 -- 172.21.2.117:0/4250503120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca201a9570 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca2586c700 1 -- 172.21.2.117:0/4250503120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca201a9b00 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca14006b00 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 0x7fca0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fca140326d0 con 0x7fca200e74d0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.502 7fca1f574700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 0x7fca0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:38.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.504 7fca1f574700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 0x7fca0001abc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.534 7fca2586c700 1 -- 172.21.2.117:0/4250503120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20087290 con 0x7fca200e74d0 2021-02-10T11:48:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.537 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca140009b0 con 0x7fca200e74d0 2021-02-10T11:48:38.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.659 7fca2586c700 1 -- 172.21.2.117:0/4250503120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca200ede60 con 0x7fca200e74d0 2021-02-10T11:48:38.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.660 7fca07fff700 1 -- 172.21.2.117:0/4250503120 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+14242 (secure 0 0 0) 0x7fca14006b00 con 0x7fca200e74d0 2021-02-10T11:48:38.655 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:38.656 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":99,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:38.055017","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:35.729626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:37.102743","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:38.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.661 7fca2586c700 1 -- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 msgr2=0x7fca0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.661 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 0x7fca0001abc0 crc :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.661 7fca2586c700 1 -- 172.21.2.117:0/4250503120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 msgr2=0x7fca200e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.661 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7fca14006f60 tx=0x7fca1400b970).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 -- 172.21.2.117:0/4250503120 shutdown_connections 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca000186c0 0x7fca0001abc0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fca200e74d0 0x7fca200e78d0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fca200e7ef0 0x7fca201a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 --2- 172.21.2.117:0/4250503120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fca200e1840 0x7fca201a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 -- 172.21.2.117:0/4250503120 >> 172.21.2.117:0/4250503120 conn(0x7fca201a7900 msgr2=0x7fca20198120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:38.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 -- 172.21.2.117:0/4250503120 shutdown_connections 2021-02-10T11:48:38.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:38.662 7fca2586c700 1 -- 172.21.2.117:0/4250503120 wait complete. 2021-02-10T11:48:38.666 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:39.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 Processor -- start 2021-02-10T11:48:39.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 -- start start 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 0x7ff3c81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80ede60 con 0x7ff3c80eb000 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c80ef7f0 con 0x7ff3c80d18e0 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.080 7ff3cf5fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80d1ce0 con 0x7ff3c80ef3f0 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3ccc16700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff3c80eb000 0x7ff3c80eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53574/0 (socket says 172.21.2.117:53574) 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3ccc16700 1 -- 172.21.2.117:0/434113377 learned_addr learned my addr 172.21.2.117:0/434113377 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53322/0 (socket says 172.21.2.117:53322) 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 -- 172.21.2.117:0/434113377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 msgr2=0x7ff3c81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cdc18700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 0x7ff3c81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 0x7ff3c81b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 -- 172.21.2.117:0/434113377 >> v1:172.21.2.117:6789/0 conn(0x7ff3c80eb000 legacy=0x7ff3c80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 -- 172.21.2.117:0/434113377 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c81b8f90 con 0x7ff3c80d18e0 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cdc18700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 0x7ff3c81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cd417700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7ff3c00069d0 tx=0x7ff3c0009e40).ready entity=mon.0 client_cookie=8cb35245eef1f61a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3bffff700 1 -- 172.21.2.117:0/434113377 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c000b510 con 0x7ff3c80d18e0 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3bffff700 1 -- 172.21.2.117:0/434113377 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000b890 con 0x7ff3c80d18e0 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3bffff700 1 -- 172.21.2.117:0/434113377 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3c000bdb0 con 0x7ff3c80d18e0 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cf5fb700 1 -- 172.21.2.117:0/434113377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80d18e0 msgr2=0x7ff3c80ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.081 7ff3cf5fb700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7ff3c00069d0 tx=0x7ff3c0009e40).stop 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 -- 172.21.2.117:0/434113377 shutdown_connections 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80ef3f0 0x7ff3c81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 --2- 172.21.2.117:0/434113377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80d18e0 0x7ff3c80ea9e0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 -- 172.21.2.117:0/434113377 >> 172.21.2.117:0/434113377 conn(0x7ff3c80ced60 msgr2=0x7ff3c80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 -- 172.21.2.117:0/434113377 shutdown_connections 2021-02-10T11:48:39.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.082 7ff3cf5fb700 1 -- 172.21.2.117:0/434113377 wait complete. 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 Processor -- start 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 -- start start 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 0x7ff3c80deb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80df120 0x7ff3c80b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c818fd10 con 0x7ff3c80984a0 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c818f950 con 0x7ff3c81a41e0 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3cf5fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80b4260 con 0x7ff3c80df120 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.097 7ff3ccc16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53330/0 (socket says 172.21.2.117:53330) 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 -- 172.21.2.117:0/2833029832 learned_addr learned my addr 172.21.2.117:0/2833029832 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:39.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3cd417700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 0x7ff3c80deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 -- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80df120 msgr2=0x7ff3c80b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80df120 0x7ff3c80b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 -- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 msgr2=0x7ff3c80deb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 0x7ff3c80deb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 -- 172.21.2.117:0/2833029832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8190210 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3cd417700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 0x7ff3c80deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3ccc16700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7ff3b8001c60 tx=0x7ff3b800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3b800acc0 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b8004140 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c8190210 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3b80045a0 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.098 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c8190210 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.099 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3b8000d90 con 0x7ff3c80984a0 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.099 7ff3bdffb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 0x7ff3a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.099 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff3b8032900 con 0x7ff3c80984a0 2021-02-10T11:48:39.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.099 7ff3cd417700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 0x7ff3a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.100 7ff3cd417700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 0x7ff3a801abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.130 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c8087290 con 0x7ff3c80984a0 2021-02-10T11:48:39.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.133 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3b80009b0 con 0x7ff3c80984a0 2021-02-10T11:48:39.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.256 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3c818f910 con 0x7ff3c80984a0 2021-02-10T11:48:39.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.257 7ff3bdffb700 1 -- 172.21.2.117:0/2833029832 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+14231 (secure 0 0 0) 0x7ff3b8000d90 con 0x7ff3c80984a0 2021-02-10T11:48:39.253 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:39.253 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":100,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:39.057620","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:35.729626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:37.102743","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:39.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.258 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 msgr2=0x7ff3a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.258 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 0x7ff3a801abc0 crc :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.258 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 msgr2=0x7ff3c80b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.258 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7ff3b8001c60 tx=0x7ff3b800a8d0).stop 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 shutdown_connections 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3a80186c0 0x7ff3a801abc0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3c80984a0 0x7ff3c80b3620 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3c80df120 0x7ff3c80b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 --2- 172.21.2.117:0/2833029832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3c81a41e0 0x7ff3c80deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 >> 172.21.2.117:0/2833029832 conn(0x7ff3c819aec0 msgr2=0x7ff3c80dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 shutdown_connections 2021-02-10T11:48:39.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.259 7ff3cf5fb700 1 -- 172.21.2.117:0/2833029832 wait complete. 2021-02-10T11:48:39.263 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:48:39.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.685 7f29a4453700 1 Processor -- start 2021-02-10T11:48:39.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.685 7f29a4453700 1 -- start start 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a4453700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f299c0d18f0 0x7f299c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a4453700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c0ef3f0 0x7f299c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a4453700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c0ef7f0 con 0x7f299c0eb040 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a4453700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299c0eded0 con 0x7f299c0d18f0 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a4453700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0d1cf0 con 0x7f299c0ef3f0 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a226f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f299c0d18f0 0x7f299c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a1a6e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f299c0eb040 0x7f299c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53586/0 (socket says 172.21.2.117:53586) 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a1a6e700 1 -- 172.21.2.117:0/3308831221 learned_addr learned my addr 172.21.2.117:0/3308831221 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a226f700 1 -- 172.21.2.117:0/3308831221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c0ef3f0 msgr2=0x7f299c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:39.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a226f700 1 --2- 172.21.2.117:0/3308831221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c0ef3f0 0x7f299c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a226f700 1 -- 172.21.2.117:0/3308831221 >> v1:172.21.2.117:6789/0 conn(0x7f299c0eb040 legacy=0x7f299c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.686 7f29a226f700 1 -- 172.21.2.117:0/3308831221 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c1b8fd0 con 0x7f299c0d18f0 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a226f700 1 --2- 172.21.2.117:0/3308831221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c0d18f0 0x7f299c0eaa20 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f2990006650 tx=0x7f2990009a80).ready entity=mon.0 client_cookie=f37ad6413d293f52 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a126d700 1 -- 172.21.2.117:0/3308831221 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f299000c0c0 con 0x7f299c0d18f0 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a126d700 1 -- 172.21.2.117:0/3308831221 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299000c340 con 0x7f299c0d18f0 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a126d700 1 -- 172.21.2.117:0/3308831221 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f299000c860 con 0x7f299c0d18f0 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a4453700 1 -- 172.21.2.117:0/3308831221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c0d18f0 msgr2=0x7f299c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.687 7f29a4453700 1 --2- 172.21.2.117:0/3308831221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c0d18f0 0x7f299c0eaa20 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7f2990006650 tx=0x7f2990009a80).stop 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 -- 172.21.2.117:0/3308831221 shutdown_connections 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 --2- 172.21.2.117:0/3308831221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c0ef3f0 0x7f299c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 --2- 172.21.2.117:0/3308831221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c0d18f0 0x7f299c0eaa20 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:39.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 -- 172.21.2.117:0/3308831221 >> 172.21.2.117:0/3308831221 conn(0x7f299c0cf060 msgr2=0x7f299c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:39.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 -- 172.21.2.117:0/3308831221 shutdown_connections 2021-02-10T11:48:39.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.688 7f29a4453700 1 -- 172.21.2.117:0/3308831221 wait complete. 2021-02-10T11:48:39.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 Processor -- start 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 -- start start 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f299c1a4240 0x7f299c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c09ed70 0x7f299c09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c09fb90 con 0x7f299c09f790 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0dc640 con 0x7f299c1a4240 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a4453700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c097270 con 0x7f299c09ed70 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a2a70700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a2a70700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53344/0 (socket says 172.21.2.117:53344) 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.703 7f29a2a70700 1 -- 172.21.2.117:0/3887522883 learned_addr learned my addr 172.21.2.117:0/3887522883 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:39.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 -- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c09ed70 msgr2=0x7f299c09f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c09ed70 0x7f299c09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 -- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f299c1a4240 msgr2=0x7f299c1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f299c1a4240 0x7f299c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 -- 172.21.2.117:0/3887522883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0b8780 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a2a70700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f2994009980 tx=0x7f299400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2994003020 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994005a80 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2994006030 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a4453700 1 -- 172.21.2.117:0/3887522883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c0b8780 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.704 7f29a4453700 1 -- 172.21.2.117:0/3887522883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c0b8d10 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.705 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2994006500 con 0x7f299c09f790 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.705 7f298effd700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 0x7f298401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:39.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.705 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2994032a90 con 0x7f299c09f790 2021-02-10T11:48:39.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.705 7f29a226f700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 0x7f298401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:39.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.706 7f29a226f700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 0x7f298401abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:39.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.736 7f29a4453700 1 -- 172.21.2.117:0/3887522883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c0daff0 con 0x7f299c09f790 2021-02-10T11:48:39.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.738 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f29940061f0 con 0x7f299c09f790 2021-02-10T11:48:39.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:39.872 7f29a4453700 1 -- 172.21.2.117:0/3887522883 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f299c0ede60 con 0x7f299c09f790 2021-02-10T11:48:40.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.065 7f298effd700 1 -- 172.21.2.117:0/3887522883 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f29940061f0 con 0x7f299c09f790 2021-02-10T11:48:40.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 -- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 msgr2=0x7f298401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 0x7f298401abc0 crc :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 -- 172.21.2.117:0/3887522883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 msgr2=0x7f299c0b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f2994009980 tx=0x7f299400bcd0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 -- 172.21.2.117:0/3887522883 shutdown_connections 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29840186c0 0x7f298401abc0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f299c09f790 0x7f299c0b8160 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f299c09ed70 0x7f299c09f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 --2- 172.21.2.117:0/3887522883 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f299c1a4240 0x7f299c1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.090 7f29a4453700 1 -- 172.21.2.117:0/3887522883 >> 172.21.2.117:0/3887522883 conn(0x7f299c087090 msgr2=0x7f299c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.091 7f29a4453700 1 -- 172.21.2.117:0/3887522883 shutdown_connections 2021-02-10T11:48:40.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.091 7f29a4453700 1 -- 172.21.2.117:0/3887522883 wait complete. 2021-02-10T11:48:40.093 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.509 7fdda25b0700 1 Processor -- start 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.509 7fdda25b0700 1 -- start start 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda25b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 0x7fdd9c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda25b0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 0x7fdd9c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda25b0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c0ede60 con 0x7fdd9c0eb000 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda25b0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9c088020 con 0x7fdd9c0ef3f0 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda25b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c0d1ce0 con 0x7fdd9c0d18e0 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda0bcd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 0x7fdd9c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda0bcd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 0x7fdd9c1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53352/0 (socket says 172.21.2.117:53352) 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdda0bcd700 1 -- 172.21.2.117:0/2356063609 learned_addr learned my addr 172.21.2.117:0/2356063609 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdd9bfff700 1 --2- 172.21.2.117:0/2356063609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 0x7fdd9c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.510 7fdd9b7fe700 1 --1- 172.21.2.117:0/2356063609 >> v1:172.21.2.117:6789/0 conn(0x7fdd9c0eb000 0x7fdd9c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53598/0 (socket says 172.21.2.117:53598) 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9bfff700 1 -- 172.21.2.117:0/2356063609 >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 msgr2=0x7fdd9c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9bfff700 1 --2- 172.21.2.117:0/2356063609 >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 0x7fdd9c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9bfff700 1 -- 172.21.2.117:0/2356063609 >> v1:172.21.2.117:6789/0 conn(0x7fdd9c0eb000 legacy=0x7fdd9c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9bfff700 1 -- 172.21.2.117:0/2356063609 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c1b8f90 con 0x7fdd9c0d18e0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9affd700 1 -- 172.21.2.117:0/2356063609 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761507493 0 0) 0x7fdd84001100 con 0x7fdd9c0eb000 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9bfff700 1 --2- 172.21.2.117:0/2356063609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 0x7fdd9c0ea9e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdd8c0069d0 tx=0x7fdd8c009e40).ready entity=mon.2 client_cookie=ba8ce8fc87f75b08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9affd700 1 -- 172.21.2.117:0/2356063609 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd8c00b510 con 0x7fdd9c0d18e0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9affd700 1 -- 172.21.2.117:0/2356063609 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c00b890 con 0x7fdd9c0d18e0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdd9affd700 1 -- 172.21.2.117:0/2356063609 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd8c00bd70 con 0x7fdd9c0d18e0 2021-02-10T11:48:40.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdda25b0700 1 -- 172.21.2.117:0/2356063609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 msgr2=0x7fdd9c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.511 7fdda25b0700 1 --2- 172.21.2.117:0/2356063609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 0x7fdd9c0ea9e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdd8c0069d0 tx=0x7fdd8c009e40).stop 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 -- 172.21.2.117:0/2356063609 shutdown_connections 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 --2- 172.21.2.117:0/2356063609 >> v2:172.21.2.117:3300/0 conn(0x7fdd9c0ef3f0 0x7fdd9c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 --2- 172.21.2.117:0/2356063609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c0d18e0 0x7fdd9c0ea9e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 -- 172.21.2.117:0/2356063609 >> 172.21.2.117:0/2356063609 conn(0x7fdd9c0ced60 msgr2=0x7fdd9c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 -- 172.21.2.117:0/2356063609 shutdown_connections 2021-02-10T11:48:40.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.512 7fdda25b0700 1 -- 172.21.2.117:0/2356063609 wait complete. 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.527 7fdda25b0700 1 Processor -- start 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.527 7fdda25b0700 1 -- start start 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c1a4290 0x7fdd9c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd9c0ab280 0x7fdd9c0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c0a7d10 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c0a7a00 con 0x7fdd9c0ab280 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda25b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9c096370 con 0x7fdd9c1a4290 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda0bcd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda0bcd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53358/0 (socket says 172.21.2.117:53358) 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdda0bcd700 1 -- 172.21.2.117:0/217326488 learned_addr learned my addr 172.21.2.117:0/217326488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:40.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdd9b7fe700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd9c0ab280 0x7fdd9c0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.528 7fdd9bfff700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c1a4290 0x7fdd9c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 -- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c1a4290 msgr2=0x7fdd9c1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c1a4290 0x7fdd9c1a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 -- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd9c0ab280 msgr2=0x7fdd9c0ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd9c0ab280 0x7fdd9c0ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 -- 172.21.2.117:0/217326488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c0a8290 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda0bcd700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7fdd94006ed0 tx=0x7fdd9400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd94000c10 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9400bd50 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd94002220 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda25b0700 1 -- 172.21.2.117:0/217326488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c0a8290 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.529 7fdda25b0700 1 -- 172.21.2.117:0/217326488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c0a8820 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.530 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd94000ab0 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.530 7fdd98ff9700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 0x7fdd8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:40.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.530 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdd94032540 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.530 7fdd9bfff700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 0x7fdd8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:40.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.531 7fdd9bfff700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 0x7fdd8001abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:40.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.560 7fdda25b0700 1 -- 172.21.2.117:0/217326488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c087290 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.563 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd94006820 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.691 7fdda25b0700 1 -- 172.21.2.117:0/217326488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd9c0ede60 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.692 7fdd98ff9700 1 -- 172.21.2.117:0/217326488 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fdd94000ab0 con 0x7fdd9c0a6fe0 2021-02-10T11:48:40.687 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:40.688 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":77,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:48:40.061715","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1898190750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1898190750},{"type":"v1","addr":"172.21.2.125:6835","nonce":1898190750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:48:40.688 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 77 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 msgr2=0x7fdd8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 0x7fdd8001abc0 crc :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 msgr2=0x7fdd9c0a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7fdd94006ed0 tx=0x7fdd9400b960).stop 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 shutdown_connections 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd800186c0 0x7fdd8001abc0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd9c0a6fe0 0x7fdd9c0a73e0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd9c0ab280 0x7fdd9c0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 --2- 172.21.2.117:0/217326488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd9c1a4290 0x7fdd9c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:40.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 >> 172.21.2.117:0/217326488 conn(0x7fdd9c0874c0 msgr2=0x7fdd9c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:40.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 shutdown_connections 2021-02-10T11:48:40.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:40.694 7fdda25b0700 1 -- 172.21.2.117:0/217326488 wait complete. 2021-02-10T11:48:40.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:41.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.114 7f4397e15700 1 Processor -- start 2021-02-10T11:48:41.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.114 7f4397e15700 1 -- start start 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4397e15700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900d18e0 0x7f43900ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4397e15700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f43900ef3f0 0x7f43901b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4397e15700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900ede60 con 0x7f43900eb000 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4397e15700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4390088020 con 0x7f43900ef3f0 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4397e15700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43900d1ce0 con 0x7f43900d18e0 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4395430700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f43900eb000 0x7f43900eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53612/0 (socket says 172.21.2.117:53612) 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4395430700 1 -- 172.21.2.117:0/2042851652 learned_addr learned my addr 172.21.2.117:0/2042851652 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4395c31700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900d18e0 0x7f43900ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4396432700 1 --2- 172.21.2.117:0/2042851652 >> v2:172.21.2.125:3300/0 conn(0x7f43900ef3f0 0x7f43901b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.115 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679933089 0 0) 0x7f43800010b0 con 0x7f43900eb000 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c002fe0 con 0x7f43900eb000 2021-02-10T11:48:41.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2874673865 0 0) 0x7f43800010b0 con 0x7f43900eb000 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900d18e0 msgr2=0x7f43900ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900d18e0 0x7f43900ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 >> v2:172.21.2.125:3300/0 conn(0x7f43900ef3f0 msgr2=0x7f43901b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 --2- 172.21.2.117:0/2042851652 >> v2:172.21.2.125:3300/0 conn(0x7f43900ef3f0 0x7f43901b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43901b8f90 con 0x7f43900eb000 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f43800015b0 con 0x7f43900eb000 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 >> v1:172.21.2.117:6789/0 conn(0x7f43900eb000 legacy=0x7f43900eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f437c003a60 0x7f437c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f437c04a7e0 0x7f437c04cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f437c04d510 0x7f437c04fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0502e0 con 0x7f437c04d510 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c050500 con 0x7f437c04a7e0 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c050720 con 0x7f437c003a60 2021-02-10T11:48:41.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.116 7f4396432700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f437c04d510 0x7f437c04fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4395430700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f437c04a7e0 0x7f437c04cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4395c31700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f437c003a60 0x7f437c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 -- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f437c003a60 msgr2=0x7f437c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f437c003a60 0x7f437c005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 -- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f437c04a7e0 msgr2=0x7f437c04cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f437c04a7e0 0x7f437c04cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 -- 172.21.2.117:0/2042851652 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0509c0 con 0x7f437c04d510 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f437c04d510 0x7f437c04fa30 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f4388006a40 tx=0x7f438800c0c0).ready entity=mon.0 client_cookie=6ed923c9a8162eb1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f438800c4b0 con 0x7f437c04d510 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438800c6f0 con 0x7f437c04d510 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4394c2f700 1 -- 172.21.2.117:0/2042851652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f438800cc10 con 0x7f437c04d510 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4397e15700 1 -- 172.21.2.117:0/2042851652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f437c04d510 msgr2=0x7f437c04fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4397e15700 1 --2- 172.21.2.117:0/2042851652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f437c04d510 0x7f437c04fa30 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 rx=0x7f4388006a40 tx=0x7f438800c0c0).stop 2021-02-10T11:48:41.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.117 7f4396432700 1 -- 172.21.2.117:0/2042851652 reap_dead start 2021-02-10T11:48:41.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.118 7f4397e15700 1 -- 172.21.2.117:0/2042851652 shutdown_connections 2021-02-10T11:48:41.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.118 7f4397e15700 1 -- 172.21.2.117:0/2042851652 >> 172.21.2.117:0/2042851652 conn(0x7f43900ced60 msgr2=0x7f43900cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.118 7f4397e15700 1 -- 172.21.2.117:0/2042851652 shutdown_connections 2021-02-10T11:48:41.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.118 7f4397e15700 1 -- 172.21.2.117:0/2042851652 wait complete. 2021-02-10T11:48:41.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 Processor -- start 2021-02-10T11:48:41.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 -- start start 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43901a33d0 0x7f43901a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900b82f0 0x7f43900b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900c0670 con 0x7f43900b8d10 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43900deb20 con 0x7f43901a33d0 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4397e15700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43900ded40 con 0x7f43900b82f0 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4396432700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4396432700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53376/0 (socket says 172.21.2.117:53376) 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.133 7f4396432700 1 -- 172.21.2.117:0/895240063 learned_addr learned my addr 172.21.2.117:0/895240063 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4395430700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900b82f0 0x7f43900b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4395c31700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43901a33d0 0x7f43901a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 -- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900b82f0 msgr2=0x7f43900b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900b82f0 0x7f43900b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 -- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43901a33d0 msgr2=0x7f43901a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43901a33d0 0x7f43901a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 -- 172.21.2.117:0/895240063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900def60 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4396432700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7f4388006d20 tx=0x7f438800bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4388009500 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438800c2f0 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f4397e15700 1 -- 172.21.2.117:0/895240063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43900def60 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.134 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f438800c810 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.135 7f4397e15700 1 -- 172.21.2.117:0/895240063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43900def60 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.135 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4388009500 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.135 7f43867fc700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 0x7f437801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.135 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4388032ca0 con 0x7f43900b8d10 2021-02-10T11:48:41.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.135 7f4395c31700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 0x7f437801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.136 7f4395c31700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 0x7f437801ac10 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.165 7f4397e15700 1 -- 172.21.2.117:0/895240063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4390087290 con 0x7f43900b8d10 2021-02-10T11:48:41.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.167 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f438800be00 con 0x7f43900b8d10 2021-02-10T11:48:41.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.291 7f4397e15700 1 -- 172.21.2.117:0/895240063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43900ede60 con 0x7f43900b8d10 2021-02-10T11:48:41.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.292 7f43867fc700 1 -- 172.21.2.117:0/895240063 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+14231 (secure 0 0 0) 0x7f4388009500 con 0x7f43900b8d10 2021-02-10T11:48:41.287 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:41.287 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":100,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:39.057620","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:35.729626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:37.102743","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:41.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.293 7f4397e15700 1 -- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 msgr2=0x7f437801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.293 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 0x7f437801ac10 crc :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.293 7f4397e15700 1 -- 172.21.2.117:0/895240063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 msgr2=0x7f43900b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.293 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7f4388006d20 tx=0x7f438800bcc0).stop 2021-02-10T11:48:41.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 -- 172.21.2.117:0/895240063 shutdown_connections 2021-02-10T11:48:41.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4378018730 0x7f437801ac10 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43900b8d10 0x7f43900b9110 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43900b82f0 0x7f43900b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 --2- 172.21.2.117:0/895240063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43901a33d0 0x7f43901a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 -- 172.21.2.117:0/895240063 >> 172.21.2.117:0/895240063 conn(0x7f43900874c0 msgr2=0x7f4390096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 -- 172.21.2.117:0/895240063 shutdown_connections 2021-02-10T11:48:41.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.294 7f4397e15700 1 -- 172.21.2.117:0/895240063 wait complete. 2021-02-10T11:48:41.297 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:41.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 Processor -- start 2021-02-10T11:48:41.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 -- start start 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cb40d1900 0x7f8cb40eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40ef800 con 0x7f8cb40eb050 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb40ede70 con 0x7f8cb40ef400 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.726 7f8cbbdb8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb40d1db0 con 0x7f8cb40d1900 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb9bd4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cb40d1900 0x7f8cb40eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb93d3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8cb40eb050 0x7f8cb40eb450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53630/0 (socket says 172.21.2.117:53630) 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb93d3700 1 -- 172.21.2.117:0/3994415948 learned_addr learned my addr 172.21.2.117:0/3994415948 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb9bd4700 1 -- 172.21.2.117:0/3994415948 >> v1:172.21.2.125:6789/0 conn(0x7f8cb40ef400 legacy=0x7f8cb41b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb9bd4700 1 -- 172.21.2.117:0/3994415948 >> v1:172.21.2.117:6789/0 conn(0x7f8cb40eb050 legacy=0x7f8cb40eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.727 7f8cb9bd4700 1 -- 172.21.2.117:0/3994415948 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb41bb1d0 con 0x7f8cb40d1900 2021-02-10T11:48:41.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cb9bd4700 1 --2- 172.21.2.117:0/3994415948 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb40d1900 0x7f8cb40eaa30 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7f8cb0006650 tx=0x7f8cb0009a80).ready entity=mon.0 client_cookie=62fcd2a4fce1007a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cb8bd2700 1 -- 172.21.2.117:0/3994415948 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb000c0c0 con 0x7f8cb40d1900 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cb8bd2700 1 -- 172.21.2.117:0/3994415948 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb000c340 con 0x7f8cb40d1900 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cb8bd2700 1 -- 172.21.2.117:0/3994415948 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cb000c860 con 0x7f8cb40d1900 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cbbdb8700 1 -- 172.21.2.117:0/3994415948 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb40d1900 msgr2=0x7f8cb40eaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.728 7f8cbbdb8700 1 --2- 172.21.2.117:0/3994415948 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb40d1900 0x7f8cb40eaa30 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7f8cb0006650 tx=0x7f8cb0009a80).stop 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.729 7f8cbbdb8700 1 -- 172.21.2.117:0/3994415948 shutdown_connections 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.729 7f8cbbdb8700 1 --2- 172.21.2.117:0/3994415948 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb40d1900 0x7f8cb40eaa30 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.729 7f8cbbdb8700 1 -- 172.21.2.117:0/3994415948 >> 172.21.2.117:0/3994415948 conn(0x7f8cb40cf070 msgr2=0x7f8cb40cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.729 7f8cbbdb8700 1 -- 172.21.2.117:0/3994415948 shutdown_connections 2021-02-10T11:48:41.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.729 7f8cbbdb8700 1 -- 172.21.2.117:0/3994415948 wait complete. 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 Processor -- start 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 -- start start 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cb40bb010 0x7f8cb41c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cb40bcb70 0x7f8cb40bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40bcf70 con 0x7f8cb41c6970 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40bd190 con 0x7f8cb40bcb70 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cbbdb8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb40bd3b0 con 0x7f8cb40bb010 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cb93d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cb93d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53386/0 (socket says 172.21.2.117:53386) 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.744 7f8cb93d3700 1 -- 172.21.2.117:0/172403926 learned_addr learned my addr 172.21.2.117:0/172403926 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 -- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cb40bb010 msgr2=0x7f8cb41c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cb40bb010 0x7f8cb41c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 -- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cb40bcb70 msgr2=0x7f8cb40bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cb40bcb70 0x7f8cb40bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 -- 172.21.2.117:0/172403926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40c02d0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cb93d3700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f8ca4001c10 tx=0x7f8ca400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca40010b0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca400acc0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ca40045d0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb40c02d0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.745 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb40c02d0 con 0x7f8cb41c6970 2021-02-10T11:48:41.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.746 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ca4004140 con 0x7f8cb41c6970 2021-02-10T11:48:41.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.746 7f8caa7fc700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:41.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.746 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ca40326d0 con 0x7f8cb41c6970 2021-02-10T11:48:41.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.746 7f8cb9bd4700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:41.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.747 7f8cb9bd4700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:41.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.776 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb4087350 con 0x7f8cb41c6970 2021-02-10T11:48:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.779 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ca4001050 con 0x7f8cb41c6970 2021-02-10T11:48:41.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.908 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8cb40d1900 con 0x7f8cb41c6970 2021-02-10T11:48:41.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.911 7f8caa7fc700 1 -- 172.21.2.117:0/172403926 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v59) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8ca401f500 con 0x7f8cb41c6970 2021-02-10T11:48:41.910 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.916 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 msgr2=0x7f8c9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.916 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 crc :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.916 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 msgr2=0x7f8cb40bc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.916 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f8ca4001c10 tx=0x7f8ca400a8d0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 shutdown_connections 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8c9c0186c0 0x7f8c9c01abc0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cb41c6970 0x7f8cb40bc550 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cb40bcb70 0x7f8cb40bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 --2- 172.21.2.117:0/172403926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cb40bb010 0x7f8cb41c6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 >> 172.21.2.117:0/172403926 conn(0x7f8cb40e71e0 msgr2=0x7f8cb40df290 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:41.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 shutdown_connections 2021-02-10T11:48:41.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:41.917 7f8cbbdb8700 1 -- 172.21.2.117:0/172403926 wait complete. 2021-02-10T11:48:41.920 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:42.227 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:42.230 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:48:42.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 Processor -- start 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 -- start start 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa29407ab40 0x7fa2940794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 0x7fa294079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29407af40 0x7fa29407b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940e5770 con 0x7fa294079b10 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940d18c0 con 0x7fa29407ab40 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa299eb1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940d1c60 con 0x7fa29407af40 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa2937fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa29407ab40 0x7fa2940794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa292ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 0x7fa294079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa2937fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa29407ab40 0x7fa2940794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57776/0 (socket says 172.21.2.117:57776) 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa2937fe700 1 -- 172.21.2.117:0/2942392263 learned_addr learned my addr 172.21.2.117:0/2942392263 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.348 7fa293fff700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29407af40 0x7fa29407b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 -- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29407af40 msgr2=0x7fa29407b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29407af40 0x7fa29407b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 -- 172.21.2.117:0/2942392263 >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 msgr2=0x7fa294079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 --2- 172.21.2.117:0/2942392263 >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 0x7fa294079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 -- 172.21.2.117:0/2942392263 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29407b960 con 0x7fa29407ab40 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa292ffd700 1 --2- 172.21.2.117:0/2942392263 >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 0x7fa294079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2937fe700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29407ab40 0x7fa2940794f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fa28c006a80 tx=0x7fa28c009f00).ready entity=mon.1 client_cookie=75a3ed73e3e6c072 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2927fc700 1 -- 172.21.2.117:0/2942392263 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28c00b5c0 con 0x7fa29407ab40 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2927fc700 1 -- 172.21.2.117:0/2942392263 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c00b800 con 0x7fa29407ab40 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa2927fc700 1 -- 172.21.2.117:0/2942392263 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28c00bd20 con 0x7fa29407ab40 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa299eb1700 1 -- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29407ab40 msgr2=0x7fa2940794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.349 7fa299eb1700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29407ab40 0x7fa2940794f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fa28c006a80 tx=0x7fa28c009f00).stop 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 -- 172.21.2.117:0/2942392263 shutdown_connections 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29407af40 0x7fa29407b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 --2- 172.21.2.117:0/2942392263 >> v2:172.21.2.117:3300/0 conn(0x7fa294079b10 0x7fa294079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 --2- 172.21.2.117:0/2942392263 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29407ab40 0x7fa2940794f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 -- 172.21.2.117:0/2942392263 >> 172.21.2.117:0/2942392263 conn(0x7fa2940d0910 msgr2=0x7fa2940d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:42.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 -- 172.21.2.117:0/2942392263 shutdown_connections 2021-02-10T11:48:42.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.350 7fa299eb1700 1 -- 172.21.2.117:0/2942392263 wait complete. 2021-02-10T11:48:42.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 Processor -- start 2021-02-10T11:48:42.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 -- start start 2021-02-10T11:48:42.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 0x7fa29408c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940923b0 0x7fa2940913c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29408ce10 con 0x7fa2940923b0 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940927b0 con 0x7fa2940919e0 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa299eb1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940929d0 con 0x7fa29408ca10 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.365 7fa293fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:57784/0 (socket says 172.21.2.117:57784) 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 -- 172.21.2.117:0/2855752610 learned_addr learned my addr 172.21.2.117:0/2855752610 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa292ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940923b0 0x7fa2940913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa2937fe700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 0x7fa29408c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 -- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 msgr2=0x7fa29408c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 0x7fa29408c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 -- 172.21.2.117:0/2855752610 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940923b0 msgr2=0x7fa2940913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940923b0 0x7fa2940913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 -- 172.21.2.117:0/2855752610 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940932f0 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa2937fe700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 0x7fa29408c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa293fff700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fa28400b590 tx=0x7fa28400b930).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28400bd20 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa284002030 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2940932f0 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2941a72a0 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.366 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2840024c0 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.367 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa284006710 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.367 7fa27bfff700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 0x7fa27401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.367 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa284032680 con 0x7fa2940919e0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.367 7fa2937fe700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 0x7fa27401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.368 7fa2937fe700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 0x7fa27401abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:42.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.398 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2940e5770 con 0x7fa2940919e0 2021-02-10T11:48:42.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.402 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa284000bc0 con 0x7fa2940919e0 2021-02-10T11:48:42.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.535 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa2940e5770 con 0x7fa2940919e0 2021-02-10T11:48:42.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.540 7fa27bfff700 1 -- 172.21.2.117:0/2855752610 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v60) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa284002210 con 0x7fa2940919e0 2021-02-10T11:48:42.537 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:48:42.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 msgr2=0x7fa27401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 0x7fa27401abc0 crc :-1 s=READY pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 msgr2=0x7fa294092cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fa28400b590 tx=0x7fa28400b930).stop 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 shutdown_connections 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2740186c0 0x7fa27401abc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940923b0 0x7fa2940913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2940919e0 0x7fa294092cd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 --2- 172.21.2.117:0/2855752610 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa29408ca10 0x7fa29408c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.544 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 >> 172.21.2.117:0/2855752610 conn(0x7fa294098f90 msgr2=0x7fa29408e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.545 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 shutdown_connections 2021-02-10T11:48:42.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.545 7fa299eb1700 1 -- 172.21.2.117:0/2855752610 wait complete. 2021-02-10T11:48:42.548 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:42.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.976 7f4932893700 1 Processor -- start 2021-02-10T11:48:42.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 -- start start 2021-02-10T11:48:42.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f492c07ab40 0x7f492c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c079b10 0x7f492c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0e5770 con 0x7f492c07af40 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c0d18c0 con 0x7f492c07ab40 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4932893700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0d1c60 con 0x7f492c079b10 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4930eb0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f492c07af40 0x7f492c07b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53656/0 (socket says 172.21.2.117:53656) 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.977 7f4930eb0700 1 -- 172.21.2.117:0/4027247414 learned_addr learned my addr 172.21.2.117:0/4027247414 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100136188 0 0) 0x7f4924001500 con 0x7f492c07af40 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4914002fe0 con 0x7f492c07af40 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492bfff700 1 --2- 172.21.2.117:0/4027247414 >> v2:172.21.2.125:3300/0 conn(0x7f492c07ab40 0x7f492c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492b7fe700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c079b10 0x7f492c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3814004290 0 0) 0x7f4924001460 con 0x7f492c07af40 2021-02-10T11:48:42.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c079b10 msgr2=0x7f492c079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c079b10 0x7f492c079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 >> v2:172.21.2.125:3300/0 conn(0x7f492c07ab40 msgr2=0x7f492c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 --2- 172.21.2.117:0/4027247414 >> v2:172.21.2.125:3300/0 conn(0x7f492c07ab40 0x7f492c0794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c07bb30 con 0x7f492c07af40 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4924001980 con 0x7f492c07af40 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 >> v1:172.21.2.117:6789/0 conn(0x7f492c07af40 legacy=0x7f492c07b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4914003a40 0x7f4914005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f491404a790 0x7f491404cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f491404d590 0x7f491404fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140502d0 con 0x7f491404a790 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49140504f0 con 0x7f4914003a40 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492affd700 1 -- 172.21.2.117:0/4027247414 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4914050710 con 0x7f491404d590 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492b7fe700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f491404a790 0x7f491404cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.978 7f492bfff700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4914003a40 0x7f4914005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4930eb0700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f491404d590 0x7f491404fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 -- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f491404d590 msgr2=0x7f491404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f491404d590 0x7f491404fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 -- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4914003a40 msgr2=0x7f4914005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4914003a40 0x7f4914005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 -- 172.21.2.117:0/4027247414 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49140509b0 con 0x7f491404a790 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492b7fe700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f491404a790 0x7f491404cc90 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f4920009310 tx=0x7f4920009a90).ready entity=mon.0 client_cookie=af1518a4e7b84f7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f492000c0c0 con 0x7f491404a790 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920006400 con 0x7f491404a790 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f492affd700 1 -- 172.21.2.117:0/4027247414 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f492000c340 con 0x7f491404a790 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 -- 172.21.2.117:0/4027247414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f491404a790 msgr2=0x7f491404cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 --2- 172.21.2.117:0/4027247414 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f491404a790 0x7f491404cc90 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f4920009310 tx=0x7f4920009a90).stop 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4930eb0700 1 -- 172.21.2.117:0/4027247414 reap_dead start 2021-02-10T11:48:42.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 -- 172.21.2.117:0/4027247414 shutdown_connections 2021-02-10T11:48:42.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 -- 172.21.2.117:0/4027247414 >> 172.21.2.117:0/4027247414 conn(0x7f492c0d0910 msgr2=0x7f492c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:42.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 -- 172.21.2.117:0/4027247414 shutdown_connections 2021-02-10T11:48:42.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.979 7f4932893700 1 -- 172.21.2.117:0/4027247414 wait complete. 2021-02-10T11:48:42.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.994 7f4932893700 1 Processor -- start 2021-02-10T11:48:42.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.994 7f4932893700 1 -- start start 2021-02-10T11:48:42.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 0x7f492c1a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c1a4830 0x7f492c0931a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0f9840 con 0x7f492c0fa000 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0928f0 con 0x7f492c0f9ad0 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4932893700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0b8650 con 0x7f492c1a4830 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53416/0 (socket says 172.21.2.117:53416) 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 -- 172.21.2.117:0/1556709219 learned_addr learned my addr 172.21.2.117:0/1556709219 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492b7fe700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 0x7f492c1a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 -- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c1a4830 msgr2=0x7f492c0931a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c1a4830 0x7f492c0931a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 -- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 msgr2=0x7f492c1a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 0x7f492c1a4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 -- 172.21.2.117:0/1556709219 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0937c0 con 0x7f492c0fa000 2021-02-10T11:48:42.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492b7fe700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 0x7f492c1a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f492bfff700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f491c006f60 tx=0x7f491c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.995 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f491c0063a0 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491c006c40 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f491c0052e0 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4932893700 1 -- 172.21.2.117:0/1556709219 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0937c0 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4932893700 1 -- 172.21.2.117:0/1556709219 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c0937c0 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f491c0052e0 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4928ff9700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.996 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f491c032510 con 0x7f492c0fa000 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.997 7f492b7fe700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:42.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:42.997 7f492b7fe700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 0x7f490c01abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:43.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.027 7f4932893700 1 -- 172.21.2.117:0/1556709219 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c0928f0 con 0x7f492c0fa000 2021-02-10T11:48:43.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.029 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f491c0063a0 con 0x7f492c0fa000 2021-02-10T11:48:43.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.159 7f4932893700 1 -- 172.21.2.117:0/1556709219 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f492c0928f0 con 0x7f492c0fa000 2021-02-10T11:48:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.162 7f4928ff9700 1 -- 172.21.2.117:0/1556709219 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v61) v1 ==== 184+0+0 (secure 0 0 0) 0x7f491c01f4f0 con 0x7f492c0fa000 2021-02-10T11:48:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 msgr2=0x7f490c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 0x7f490c01abc0 crc :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 msgr2=0x7f492c0f91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f491c006f60 tx=0x7f491c00b970).stop 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 shutdown_connections 2021-02-10T11:48:43.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f492c1a4830 0x7f492c0931a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f492c0f9ad0 0x7f492c1a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f490c0186c0 0x7f490c01abc0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 --2- 172.21.2.117:0/1556709219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f492c0fa000 0x7f492c0f91a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 >> 172.21.2.117:0/1556709219 conn(0x7f492c0954b0 msgr2=0x7f492c0faad0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 shutdown_connections 2021-02-10T11:48:43.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.168 7f4932893700 1 -- 172.21.2.117:0/1556709219 wait complete. 2021-02-10T11:48:43.171 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 Processor -- start 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 -- start start 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f93d407ab40 0x7f93d40794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d40e5770 con 0x7f93d4079f10 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d407a310 con 0x7f93d4079b10 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93db387700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d40d18c0 con 0x7f93d407ab40 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93d99a4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f93d4079f10 0x7f93d407b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53674/0 (socket says 172.21.2.117:53674) 2021-02-10T11:48:43.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.600 7f93d99a4700 1 -- 172.21.2.117:0/3058903458 learned_addr learned my addr 172.21.2.117:0/3058903458 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93d91a3700 1 --2- 172.21.2.117:0/3058903458 >> v2:172.21.2.125:3300/0 conn(0x7f93d407ab40 0x7f93d40794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690363789 0 0) 0x7f93cc001500 con 0x7f93d4079f10 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0002fe0 con 0x7f93d4079f10 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487741022 0 0) 0x7f93c8000ff0 con 0x7f93d4079b10 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c0003210 con 0x7f93d4079b10 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93d91a3700 1 -- 172.21.2.117:0/3058903458 >> v1:172.21.2.125:6789/0 conn(0x7f93d4079b10 legacy=0x7f93d407af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3264546626 0 0) 0x7f93cc001500 con 0x7f93d4079f10 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93d91a3700 1 -- 172.21.2.117:0/3058903458 >> v1:172.21.2.117:6789/0 conn(0x7f93d4079f10 legacy=0x7f93d407b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.601 7f93d91a3700 1 -- 172.21.2.117:0/3058903458 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d407bf20 con 0x7f93d407ab40 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93d91a3700 1 --2- 172.21.2.117:0/3058903458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d407ab40 0x7f93d40794f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f93d0006650 tx=0x7f93d0009a00).ready entity=mon.1 client_cookie=b69f44b9ca37d448 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d00009b0 con 0x7f93d407ab40 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d000c0c0 con 0x7f93d407ab40 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93c7fff700 1 -- 172.21.2.117:0/3058903458 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93d000c5a0 con 0x7f93d407ab40 2021-02-10T11:48:43.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93db387700 1 -- 172.21.2.117:0/3058903458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d407ab40 msgr2=0x7f93d40794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.602 7f93db387700 1 --2- 172.21.2.117:0/3058903458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d407ab40 0x7f93d40794f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f93d0006650 tx=0x7f93d0009a00).stop 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.603 7f93db387700 1 -- 172.21.2.117:0/3058903458 shutdown_connections 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.603 7f93db387700 1 --2- 172.21.2.117:0/3058903458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d407ab40 0x7f93d40794f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.603 7f93db387700 1 -- 172.21.2.117:0/3058903458 >> 172.21.2.117:0/3058903458 conn(0x7f93d40d0910 msgr2=0x7f93d40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.603 7f93db387700 1 -- 172.21.2.117:0/3058903458 shutdown_connections 2021-02-10T11:48:43.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.603 7f93db387700 1 -- 172.21.2.117:0/3058903458 wait complete. 2021-02-10T11:48:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.617 7f93db387700 1 Processor -- start 2021-02-10T11:48:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.617 7f93db387700 1 -- start start 2021-02-10T11:48:43.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93d40b02b0 0x7f93d41a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d40b0850 0x7f93d4098890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4098f30 con 0x7f93d41a4930 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4099150 con 0x7f93d40b0850 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93db387700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4099370 con 0x7f93d40b02b0 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53428/0 (socket says 172.21.2.117:53428) 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 -- 172.21.2.117:0/3985528076 learned_addr learned my addr 172.21.2.117:0/3985528076 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 -- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93d40b02b0 msgr2=0x7f93d41a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93d40b02b0 0x7f93d41a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 -- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d40b0850 msgr2=0x7f93d4098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d40b0850 0x7f93d4098890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.618 7f93d89a2700 1 -- 172.21.2.117:0/3985528076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d40b61d0 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93d89a2700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f93c8006f20 tx=0x7f93c800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93c800bd50 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c80067e0 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93c8001260 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93db387700 1 -- 172.21.2.117:0/3985528076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d40b61d0 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93db387700 1 -- 172.21.2.117:0/3985528076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d40b61d0 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f93c800a000 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 0x7f93b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.619 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f93c80328f0 con 0x7f93d41a4930 2021-02-10T11:48:43.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.620 7f93d91a3700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 0x7f93b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:43.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.621 7f93d91a3700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 0x7f93b401abc0 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:43.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.650 7f93db387700 1 -- 172.21.2.117:0/3985528076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4099370 con 0x7f93d41a4930 2021-02-10T11:48:43.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.653 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f93c8006c10 con 0x7f93d41a4930 2021-02-10T11:48:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.781 7f93db387700 1 -- 172.21.2.117:0/3985528076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f93d4099370 con 0x7f93d41a4930 2021-02-10T11:48:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.785 7f93c5ffb700 1 -- 172.21.2.117:0/3985528076 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v62) v1 ==== 184+0+0 (secure 0 0 0) 0x7f93c800a000 con 0x7f93d41a4930 2021-02-10T11:48:43.783 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:48:43.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 -- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 msgr2=0x7f93b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 0x7f93b401abc0 crc :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 -- 172.21.2.117:0/3985528076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 msgr2=0x7f93d4098270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:43.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f93c8006f20 tx=0x7f93c800b960).stop 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 -- 172.21.2.117:0/3985528076 shutdown_connections 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93b40186c0 0x7f93b401abc0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93d41a4930 0x7f93d4098270 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93d40b0850 0x7f93d4098890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 --2- 172.21.2.117:0/3985528076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93d40b02b0 0x7f93d41a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.789 7f93db387700 1 -- 172.21.2.117:0/3985528076 >> 172.21.2.117:0/3985528076 conn(0x7f93d40b0e40 msgr2=0x7f93d40a6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:43.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.790 7f93db387700 1 -- 172.21.2.117:0/3985528076 shutdown_connections 2021-02-10T11:48:43.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:43.790 7f93db387700 1 -- 172.21.2.117:0/3985528076 wait complete. 2021-02-10T11:48:43.793 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:44.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.209 7f99c95c0700 1 Processor -- start 2021-02-10T11:48:44.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.209 7f99c95c0700 1 -- start start 2021-02-10T11:48:44.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c95c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99c40d18e0 0x7f99c40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:44.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c95c0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40ede60 con 0x7f99c40eb000 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c95c0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40d1ce0 con 0x7f99c40ef3f0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c95c0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c40ef7f0 con 0x7f99c40d18e0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c32bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f99c40d18e0 0x7f99c40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c2abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f99c40eb000 0x7f99c40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53686/0 (socket says 172.21.2.117:53686) 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c2abe700 1 -- 172.21.2.117:0/2552738558 learned_addr learned my addr 172.21.2.117:0/2552738558 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c32bf700 1 -- 172.21.2.117:0/2552738558 >> v1:172.21.2.125:6789/0 conn(0x7f99c40ef3f0 legacy=0x7f99c41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c32bf700 1 -- 172.21.2.117:0/2552738558 >> v1:172.21.2.117:6789/0 conn(0x7f99c40eb000 legacy=0x7f99c40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.210 7f99c32bf700 1 -- 172.21.2.117:0/2552738558 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c41bb180 con 0x7f99c40d18e0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c32bf700 1 --2- 172.21.2.117:0/2552738558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40d18e0 0x7f99c40ea9e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7f99b4006650 tx=0x7f99b4009a80).ready entity=mon.0 client_cookie=bea905b29ac90f81 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c22bd700 1 -- 172.21.2.117:0/2552738558 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99b400c0c0 con 0x7f99c40d18e0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c22bd700 1 -- 172.21.2.117:0/2552738558 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b400c340 con 0x7f99c40d18e0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c22bd700 1 -- 172.21.2.117:0/2552738558 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99b400c860 con 0x7f99c40d18e0 2021-02-10T11:48:44.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c95c0700 1 -- 172.21.2.117:0/2552738558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40d18e0 msgr2=0x7f99c40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.211 7f99c95c0700 1 --2- 172.21.2.117:0/2552738558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40d18e0 0x7f99c40ea9e0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7f99b4006650 tx=0x7f99b4009a80).stop 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.212 7f99c95c0700 1 -- 172.21.2.117:0/2552738558 shutdown_connections 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.212 7f99c95c0700 1 --2- 172.21.2.117:0/2552738558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40d18e0 0x7f99c40ea9e0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.212 7f99c95c0700 1 -- 172.21.2.117:0/2552738558 >> 172.21.2.117:0/2552738558 conn(0x7f99c40ced60 msgr2=0x7f99c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.212 7f99c95c0700 1 -- 172.21.2.117:0/2552738558 shutdown_connections 2021-02-10T11:48:44.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.212 7f99c95c0700 1 -- 172.21.2.117:0/2552738558 wait complete. 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.226 7f99c95c0700 1 Processor -- start 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 -- start start 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 0x7f99c40b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99c4098170 0x7f99c41c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40b2ba0 con 0x7f99c40b27a0 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40f2f10 con 0x7f99c4098170 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c95c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c4098570 con 0x7f99c40f2b10 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c2abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c2abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53442/0 (socket says 172.21.2.117:53442) 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.227 7f99c2abe700 1 -- 172.21.2.117:0/2332304409 learned_addr learned my addr 172.21.2.117:0/2332304409 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:44.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c32bf700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 0x7f99c40b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c3ac0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99c4098170 0x7f99c41c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 -- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 msgr2=0x7f99c40b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 0x7f99c40b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 -- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99c4098170 msgr2=0x7f99c41c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99c4098170 0x7f99c41c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 -- 172.21.2.117:0/2332304409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4098790 con 0x7f99c40b27a0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c32bf700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 0x7f99c40b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c2abe700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f99ac00a920 tx=0x7f99ac00acc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99ac0070d0 con 0x7f99c40b27a0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ac007390 con 0x7f99c40b27a0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99ac004740 con 0x7f99c40b27a0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c4098790 con 0x7f99c40b27a0 2021-02-10T11:48:44.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.228 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c41c5760 con 0x7f99c40b27a0 2021-02-10T11:48:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.229 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99ac0044c0 con 0x7f99c40b27a0 2021-02-10T11:48:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.229 7f99bbfff700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 0x7f99a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.229 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f99ac032a70 con 0x7f99c40b27a0 2021-02-10T11:48:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.230 7f99c32bf700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 0x7f99a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:44.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.230 7f99c32bf700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 0x7f99a801abc0 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:44.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.260 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c4086140 con 0x7f99c40b27a0 2021-02-10T11:48:44.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.263 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99ac0011a0 con 0x7f99c40b27a0 2021-02-10T11:48:44.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.389 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99c40d18e0 con 0x7f99c40b27a0 2021-02-10T11:48:44.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.390 7f99bbfff700 1 -- 172.21.2.117:0/2332304409 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f99ac004ee0 con 0x7f99c40b27a0 2021-02-10T11:48:44.385 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 77 2021-02-10T11:48:44.385 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:44.385 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":77,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:48:40.061715","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1898190750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1898190750},{"type":"v1","addr":"172.21.2.125:6835","nonce":1898190750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.391 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 msgr2=0x7f99a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.391 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 0x7f99a801abc0 crc :-1 s=READY pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.391 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 msgr2=0x7f99c4097b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.391 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f99ac00a920 tx=0x7f99ac00acc0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 shutdown_connections 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99a80186c0 0x7f99a801abc0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99c40b27a0 0x7f99c4097b50 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99c4098170 0x7f99c41c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 --2- 172.21.2.117:0/2332304409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99c40f2b10 0x7f99c40b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:44.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 >> 172.21.2.117:0/2332304409 conn(0x7f99c40f3230 msgr2=0x7f99c40b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:44.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 shutdown_connections 2021-02-10T11:48:44.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:44.392 7f99c95c0700 1 -- 172.21.2.117:0/2332304409 wait complete. 2021-02-10T11:48:44.597 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:38.055110', 'modified': '2021-02-10 11:48:40.061715', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'b', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1898190750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1898190750}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1898190750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:44.597 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:44.598 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:44.996 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6686, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9436052589999999, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 77, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 100, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.572190328 2021-02-10T11:48:44.997 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:48:45.000 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6686, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9436052589999999, "mdsmap_epoch": 77, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 12.572190328 } 2021-02-10T11:48:45.001 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:45.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 Processor -- start 2021-02-10T11:48:45.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 -- start start 2021-02-10T11:48:45.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e5c0ef3f0 0x7f5e5c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 0x7f5e5c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0ede60 con 0x7f5e5c0d18e0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c0ef7f0 con 0x7f5e5c0ef3f0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.416 7f5e61609700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0d1ce0 con 0x7f5e5c0eb7f0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5b2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5e5c0d18e0 0x7f5e5c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53698/0 (socket says 172.21.2.117:53698) 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5b2bf700 1 -- 172.21.2.117:0/361981697 learned_addr learned my addr 172.21.2.117:0/361981697 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5aabe700 1 --2- 172.21.2.117:0/361981697 >> v2:172.21.2.125:3300/0 conn(0x7f5e5c0ef3f0 0x7f5e5c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5bac0700 1 --2- 172.21.2.117:0/361981697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 0x7f5e5c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5a2bd700 1 -- 172.21.2.117:0/361981697 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839020351 0 0) 0x7f5e4c000ce0 con 0x7f5e5c0d18e0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.417 7f5e5a2bd700 1 -- 172.21.2.117:0/361981697 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e44002fe0 con 0x7f5e5c0d18e0 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5bac0700 1 -- 172.21.2.117:0/361981697 >> v2:172.21.2.125:3300/0 conn(0x7f5e5c0ef3f0 msgr2=0x7f5e5c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5bac0700 1 --2- 172.21.2.117:0/361981697 >> v2:172.21.2.125:3300/0 conn(0x7f5e5c0ef3f0 0x7f5e5c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5bac0700 1 -- 172.21.2.117:0/361981697 >> v1:172.21.2.117:6789/0 conn(0x7f5e5c0d18e0 legacy=0x7f5e5c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5bac0700 1 -- 172.21.2.117:0/361981697 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c1b8f90 con 0x7f5e5c0eb7f0 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5bac0700 1 --2- 172.21.2.117:0/361981697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 0x7f5e5c1b7960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f5e54006e50 tx=0x7f5e54009e70).ready entity=mon.2 client_cookie=85a69bd163e7520c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5a2bd700 1 -- 172.21.2.117:0/361981697 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e5400b510 con 0x7f5e5c0eb7f0 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5a2bd700 1 -- 172.21.2.117:0/361981697 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54006790 con 0x7f5e5c0eb7f0 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e5a2bd700 1 -- 172.21.2.117:0/361981697 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e5400b940 con 0x7f5e5c0eb7f0 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e61609700 1 -- 172.21.2.117:0/361981697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 msgr2=0x7f5e5c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.418 7f5e61609700 1 --2- 172.21.2.117:0/361981697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 0x7f5e5c1b7960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f5e54006e50 tx=0x7f5e54009e70).stop 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 -- 172.21.2.117:0/361981697 shutdown_connections 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 --2- 172.21.2.117:0/361981697 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c0eb7f0 0x7f5e5c1b7960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 --2- 172.21.2.117:0/361981697 >> v2:172.21.2.125:3300/0 conn(0x7f5e5c0ef3f0 0x7f5e5c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 -- 172.21.2.117:0/361981697 >> 172.21.2.117:0/361981697 conn(0x7f5e5c0ced60 msgr2=0x7f5e5c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 -- 172.21.2.117:0/361981697 shutdown_connections 2021-02-10T11:48:45.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.419 7f5e61609700 1 -- 172.21.2.117:0/361981697 wait complete. 2021-02-10T11:48:45.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.433 7f5e61609700 1 Processor -- start 2021-02-10T11:48:45.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 -- start start 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c091d40 0x7f5e5c0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e5c198250 0x7f5e5c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c197cd0 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c092140 con 0x7f5e5c198250 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e61609700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c1b2ed0 con 0x7f5e5c091d40 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e5aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e5aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53456/0 (socket says 172.21.2.117:53456) 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.434 7f5e5aabe700 1 -- 172.21.2.117:0/4273367892 learned_addr learned my addr 172.21.2.117:0/4273367892 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 -- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c091d40 msgr2=0x7f5e5c0e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c091d40 0x7f5e5c0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:45.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 -- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e5c198250 msgr2=0x7f5e5c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e5c198250 0x7f5e5c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 -- 172.21.2.117:0/4273367892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c198c70 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e5aabe700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f5e50006f60 tx=0x7f5e5000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e5000bd60 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e50000c90 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e61609700 1 -- 172.21.2.117:0/4273367892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c198c70 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e50002220 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.435 7f5e61609700 1 -- 172.21.2.117:0/4273367892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c196c90 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.436 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e500052e0 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.436 7f5e43fff700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 0x7f5e3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.436 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e50032540 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.437 7f5e5b2bf700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 0x7f5e3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:45.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.437 7f5e5b2bf700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 0x7f5e3c01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:45.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.466 7f5e61609700 1 -- 172.21.2.117:0/4273367892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c087290 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.469 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e50023370 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.592 7f5e61609700 1 -- 172.21.2.117:0/4273367892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e5c0ede60 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.592 7f5e43fff700 1 -- 172.21.2.117:0/4273367892 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+14231 (secure 0 0 0) 0x7f5e50023370 con 0x7f5e5c1b24b0 2021-02-10T11:48:45.588 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:45.588 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":100,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:39.057620","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:35.729626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:37.102743","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:45.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 msgr2=0x7f5e3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 0x7f5e3c01abc0 crc :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 msgr2=0x7f5e5c1b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:45.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f5e50006f60 tx=0x7f5e5000b970).stop 2021-02-10T11:48:45.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 shutdown_connections 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e3c0186c0 0x7f5e3c01abc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e5c1b24b0 0x7f5e5c1b28b0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e5c198250 0x7f5e5c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 --2- 172.21.2.117:0/4273367892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e5c091d40 0x7f5e5c0e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 >> 172.21.2.117:0/4273367892 conn(0x7f5e5c1a3620 msgr2=0x7f5e5c0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 shutdown_connections 2021-02-10T11:48:45.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:45.594 7f5e61609700 1 -- 172.21.2.117:0/4273367892 wait complete. 2021-02-10T11:48:45.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:48:35.729626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:48:37.102743', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:48:45.598 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 Processor -- start 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 -- start start 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 0x7f48280e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48280e77f0 con 0x7f48280e8790 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48280e92f0 con 0x7f48280e7d70 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.018 7f482c8bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280e5ce0 con 0x7f48281b07f0 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f4825e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 0x7f48280e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f4826e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f48280e8790 0x7f48280e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53712/0 (socket says 172.21.2.117:53712) 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f4826e1d700 1 -- 172.21.2.117:0/2560174572 learned_addr learned my addr 172.21.2.117:0/2560174572 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51142/0 (socket says 172.21.2.117:51142) 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f4825e1b700 1 --2- 172.21.2.117:0/2560174572 >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 0x7f48280e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53462/0 (socket says 172.21.2.117:53462) 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 -- 172.21.2.117:0/2560174572 >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 msgr2=0x7f48280e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 --2- 172.21.2.117:0/2560174572 >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 0x7f48280e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 -- 172.21.2.117:0/2560174572 >> v1:172.21.2.117:6789/0 conn(0x7f48280e8790 legacy=0x7f48280e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 -- 172.21.2.117:0/2560174572 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280e54b0 con 0x7f48281b07f0 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482661c700 1 --2- 172.21.2.117:0/2560174572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4818006650 tx=0x7f4818009a80).ready entity=mon.2 client_cookie=12356f8b9b86f527 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482561a700 1 -- 172.21.2.117:0/2560174572 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48180009b0 con 0x7f48281b07f0 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482561a700 1 -- 172.21.2.117:0/2560174572 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481800c0c0 con 0x7f48281b07f0 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.019 7f482561a700 1 -- 172.21.2.117:0/2560174572 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481800c5e0 con 0x7f48281b07f0 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 -- 172.21.2.117:0/2560174572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 msgr2=0x7f48280e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 --2- 172.21.2.117:0/2560174572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4818006650 tx=0x7f4818009a80).stop 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 -- 172.21.2.117:0/2560174572 shutdown_connections 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 --2- 172.21.2.117:0/2560174572 >> v2:172.21.2.117:3300/0 conn(0x7f48280e7d70 0x7f48280e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 --2- 172.21.2.117:0/2560174572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281b07f0 0x7f48280e4e90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 -- 172.21.2.117:0/2560174572 >> 172.21.2.117:0/2560174572 conn(0x7f48280ce500 msgr2=0x7f48280e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 -- 172.21.2.117:0/2560174572 shutdown_connections 2021-02-10T11:48:46.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.020 7f482c8bf700 1 -- 172.21.2.117:0/2560174572 wait complete. 2021-02-10T11:48:46.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 Processor -- start 2021-02-10T11:48:46.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 -- start start 2021-02-10T11:48:46.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f482808f410 0x7f48281a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a2dd0 0x7f48280bad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828095b70 con 0x7f48281a31d0 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482808f810 con 0x7f482808f410 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.035 7f482c8bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280bb980 con 0x7f48281a2dd0 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4825e1b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a2dd0 0x7f48280bad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53468/0 (socket says 172.21.2.117:53468) 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 -- 172.21.2.117:0/1756554352 learned_addr learned my addr 172.21.2.117:0/1756554352 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f482661c700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f482808f410 0x7f48281a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 -- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a2dd0 msgr2=0x7f48280bad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a2dd0 0x7f48280bad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 -- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f482808f410 msgr2=0x7f48281a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f482808f410 0x7f48281a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 -- 172.21.2.117:0/1756554352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280bbba0 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4826e1d700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f482000a8a0 tx=0x7f482000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4820001270 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48280bbba0 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48280bbba0 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820000d30 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.036 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f48200070a0 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.037 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f48200070a0 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.037 7f4816ffd700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 0x7f480c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.037 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4820032c90 con 0x7f48281a31d0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.037 7f482661c700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 0x7f480c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.038 7f482661c700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 0x7f480c01ac10 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.068 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4828095b70 con 0x7f48281a31d0 2021-02-10T11:48:46.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.070 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4820000d30 con 0x7f48281a31d0 2021-02-10T11:48:46.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.198 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f4828095b70 con 0x7f48281a31d0 2021-02-10T11:48:46.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.199 7f4816ffd700 1 -- 172.21.2.117:0/1756554352 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4820000d30 con 0x7f48281a31d0 2021-02-10T11:48:46.194 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:46.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 msgr2=0x7f480c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 0x7f480c01ac10 crc :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 msgr2=0x7f48280bb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f482000a8a0 tx=0x7f482000ac40).stop 2021-02-10T11:48:46.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 shutdown_connections 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f480c018730 0x7f480c01ac10 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48281a31d0 0x7f48280bb360 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a2dd0 0x7f48280bad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 --2- 172.21.2.117:0/1756554352 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f482808f410 0x7f48281a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 >> 172.21.2.117:0/1756554352 conn(0x7f482808fb00 msgr2=0x7f482808fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 shutdown_connections 2021-02-10T11:48:46.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.201 7f482c8bf700 1 -- 172.21.2.117:0/1756554352 wait complete. 2021-02-10T11:48:46.205 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:48:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.624 7feb7a09f700 1 Processor -- start 2021-02-10T11:48:46.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.624 7feb7a09f700 1 -- start start 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb7a09f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feb740d18f0 0x7feb740eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb7a09f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb740eb040 0x7feb740eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb7a09f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb740eba60 con 0x7feb740ee3e0 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb7a09f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740d1cf0 con 0x7feb740d18f0 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb7a09f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb740eefd0 con 0x7feb740eb040 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb73fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7feb740ee3e0 0x7feb740ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46650/0 (socket says 172.21.2.117:46650) 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb73fff700 1 -- 172.21.2.117:0/664058800 learned_addr learned my addr 172.21.2.117:0/664058800 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb737fe700 1 --2- 172.21.2.117:0/664058800 >> v2:172.21.2.117:3300/0 conn(0x7feb740d18f0 0x7feb740eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb72ffd700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb740eb040 0x7feb740eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb737fe700 1 -- 172.21.2.117:0/664058800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb740eb040 msgr2=0x7feb740eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb737fe700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb740eb040 0x7feb740eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.625 7feb737fe700 1 -- 172.21.2.117:0/664058800 >> v1:172.21.2.125:6789/0 conn(0x7feb740ee3e0 legacy=0x7feb740ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb737fe700 1 -- 172.21.2.117:0/664058800 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740ef1f0 con 0x7feb740d18f0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb727fc700 1 -- 172.21.2.117:0/664058800 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169000535 0 0) 0x7feb6c000f90 con 0x7feb740ee3e0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb737fe700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb740d18f0 0x7feb740eaa20 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7feb64006650 tx=0x7feb64009a00).ready entity=mon.0 client_cookie=65da878c448a483 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb727fc700 1 -- 172.21.2.117:0/664058800 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb640009b0 con 0x7feb740d18f0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb727fc700 1 -- 172.21.2.117:0/664058800 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6400c0c0 con 0x7feb740d18f0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb727fc700 1 -- 172.21.2.117:0/664058800 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb6400c5a0 con 0x7feb740d18f0 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb7a09f700 1 -- 172.21.2.117:0/664058800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb740d18f0 msgr2=0x7feb740eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.626 7feb7a09f700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb740d18f0 0x7feb740eaa20 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7feb64006650 tx=0x7feb64009a00).stop 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 -- 172.21.2.117:0/664058800 shutdown_connections 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb740eb040 0x7feb740eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 --2- 172.21.2.117:0/664058800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb740d18f0 0x7feb740eaa20 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 -- 172.21.2.117:0/664058800 >> 172.21.2.117:0/664058800 conn(0x7feb740cf060 msgr2=0x7feb740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 -- 172.21.2.117:0/664058800 shutdown_connections 2021-02-10T11:48:46.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.627 7feb7a09f700 1 -- 172.21.2.117:0/664058800 wait complete. 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 Processor -- start 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 -- start start 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb741a4240 0x7feb741a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb7409f790 0x7feb740b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7409fb90 con 0x7feb7409ed70 2021-02-10T11:48:46.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb740dc640 con 0x7feb7409f790 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb7a09f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb74097270 con 0x7feb741a4240 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb72ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb72ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53480/0 (socket says 172.21.2.117:53480) 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb72ffd700 1 -- 172.21.2.117:0/2769621960 learned_addr learned my addr 172.21.2.117:0/2769621960 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb737fe700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb741a4240 0x7feb741a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.642 7feb73fff700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb7409f790 0x7feb740b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 -- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb741a4240 msgr2=0x7feb741a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb741a4240 0x7feb741a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 -- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb7409f790 msgr2=0x7feb740b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb7409f790 0x7feb740b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 -- 172.21.2.117:0/2769621960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740b8780 con 0x7feb7409ed70 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb72ffd700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7feb6000b560 tx=0x7feb6000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb6000bcf0 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb60000c90 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb740b8780 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb600022b0 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.643 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb740b8d10 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.644 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb60006b00 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.644 7feb6bfff700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 0x7feb5001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.644 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feb600327b0 con 0x7feb7409ed70 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.644 7feb737fe700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 0x7feb5001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:46.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.644 7feb737fe700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 0x7feb5001abd0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:46.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.674 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb740daff0 con 0x7feb7409ed70 2021-02-10T11:48:46.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.677 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb60000940 con 0x7feb7409ed70 2021-02-10T11:48:46.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.806 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7feb740ede60 con 0x7feb7409ed70 2021-02-10T11:48:46.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.806 7feb6bfff700 1 -- 172.21.2.117:0/2769621960 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7feb60006b00 con 0x7feb7409ed70 2021-02-10T11:48:46.802 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 msgr2=0x7feb5001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 0x7feb5001abd0 crc :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 msgr2=0x7feb7409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7feb6000b560 tx=0x7feb6000b900).stop 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 shutdown_connections 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb500186d0 0x7feb5001abd0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb7409ed70 0x7feb7409f170 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb7409f790 0x7feb740b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 --2- 172.21.2.117:0/2769621960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb741a4240 0x7feb741a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:46.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 >> 172.21.2.117:0/2769621960 conn(0x7feb74087090 msgr2=0x7feb7408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:46.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 shutdown_connections 2021-02-10T11:48:46.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:46.808 7feb7a09f700 1 -- 172.21.2.117:0/2769621960 wait complete. 2021-02-10T11:48:46.811 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 Processor -- start 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 -- start start 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 0x7f98900e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900e7580 con 0x7f98900e8390 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900e77f0 con 0x7f98901b07f0 2021-02-10T11:48:47.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.232 7f9895701700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98900e8f80 con 0x7f98900e7180 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988fac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f98900e8390 0x7f98900e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53740/0 (socket says 172.21.2.117:53740) 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988fac0700 1 -- 172.21.2.117:0/2055711443 learned_addr learned my addr 172.21.2.117:0/2055711443 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988eabe700 1 --2- 172.21.2.117:0/2055711443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 0x7f98900e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031373677 0 0) 0x7f9888001500 con 0x7f98900e8390 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878002fe0 con 0x7f98900e8390 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417693854 0 0) 0x7f9880001080 con 0x7f98901b07f0 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9878003210 con 0x7f98901b07f0 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988eabe700 1 -- 172.21.2.117:0/2055711443 >> v1:172.21.2.125:6789/0 conn(0x7f98901b07f0 legacy=0x7f98900e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.233 7f988eabe700 1 -- 172.21.2.117:0/2055711443 >> v1:172.21.2.117:6789/0 conn(0x7f98900e8390 legacy=0x7f98900e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f988eabe700 1 -- 172.21.2.117:0/2055711443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900e9170 con 0x7f98900e7180 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f988eabe700 1 --2- 172.21.2.117:0/2055711443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 0x7f98900e7d70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9884006650 tx=0x7f9884009a80).ready entity=mon.2 client_cookie=7fc67b8080943ff4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f988400c0c0 con 0x7f98900e7180 2021-02-10T11:48:47.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988400c340 con 0x7f98900e7180 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f988e2bd700 1 -- 172.21.2.117:0/2055711443 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f988400c860 con 0x7f98900e7180 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f9895701700 1 -- 172.21.2.117:0/2055711443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 msgr2=0x7f98900e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.234 7f9895701700 1 --2- 172.21.2.117:0/2055711443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 0x7f98900e7d70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9884006650 tx=0x7f9884009a80).stop 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.235 7f9895701700 1 -- 172.21.2.117:0/2055711443 shutdown_connections 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.235 7f9895701700 1 --2- 172.21.2.117:0/2055711443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900e7180 0x7f98900e7d70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.235 7f9895701700 1 -- 172.21.2.117:0/2055711443 >> 172.21.2.117:0/2055711443 conn(0x7f98900ce500 msgr2=0x7f98900e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.235 7f9895701700 1 -- 172.21.2.117:0/2055711443 shutdown_connections 2021-02-10T11:48:47.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.235 7f9895701700 1 -- 172.21.2.117:0/2055711443 wait complete. 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.249 7f9895701700 1 Processor -- start 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.249 7f9895701700 1 -- start start 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900b8ac0 0x7f989019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98900bbfd0 0x7f98900bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900bb430 con 0x7f989019df40 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900bc9f0 con 0x7f98900bbfd0 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f9895701700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98900bcc10 con 0x7f98900b8ac0 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53496/0 (socket says 172.21.2.117:53496) 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 -- 172.21.2.117:0/4211678694 learned_addr learned my addr 172.21.2.117:0/4211678694 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 -- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900b8ac0 msgr2=0x7f989019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900b8ac0 0x7f989019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 -- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98900bbfd0 msgr2=0x7f98900bc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98900bbfd0 0x7f98900bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 -- 172.21.2.117:0/4211678694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900b6f70 con 0x7f989019df40 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.250 7f988eabe700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f9884009630 tx=0x7f988400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f988400bd70 con 0x7f989019df40 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9884000960 con 0x7f989019df40 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9895701700 1 -- 172.21.2.117:0/4211678694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98900b6f70 con 0x7f989019df40 2021-02-10T11:48:47.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9895701700 1 -- 172.21.2.117:0/4211678694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98900b7500 con 0x7f989019df40 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98840054d0 con 0x7f989019df40 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f988400a000 con 0x7f989019df40 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 0x7f987001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.251 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9884032530 con 0x7f989019df40 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.252 7f988f2bf700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 0x7f987001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.252 7f988f2bf700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 0x7f987001abc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.282 7f9895701700 1 -- 172.21.2.117:0/4211678694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989007e800 con 0x7f989019df40 2021-02-10T11:48:47.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.285 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9884009340 con 0x7f989019df40 2021-02-10T11:48:47.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.412 7f9895701700 1 -- 172.21.2.117:0/4211678694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f98900e77f0 con 0x7f989019df40 2021-02-10T11:48:47.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.412 7f9877fff700 1 -- 172.21.2.117:0/4211678694 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f988400a000 con 0x7f989019df40 2021-02-10T11:48:47.408 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:47.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 -- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 msgr2=0x7f987001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 0x7f987001abc0 crc :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 -- 172.21.2.117:0/4211678694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 msgr2=0x7f98900bb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f9884009630 tx=0x7f988400b980).stop 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 -- 172.21.2.117:0/4211678694 shutdown_connections 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98700186c0 0x7f987001abc0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f989019df40 0x7f98900bb9b0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98900bbfd0 0x7f98900bc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 --2- 172.21.2.117:0/4211678694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98900b8ac0 0x7f989019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.414 7f9895701700 1 -- 172.21.2.117:0/4211678694 >> 172.21.2.117:0/4211678694 conn(0x7f9890110940 msgr2=0x7f989007fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.415 7f9895701700 1 -- 172.21.2.117:0/4211678694 shutdown_connections 2021-02-10T11:48:47.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.415 7f9895701700 1 -- 172.21.2.117:0/4211678694 wait complete. 2021-02-10T11:48:47.418 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:48:47.418 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:47.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 Processor -- start 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 -- start start 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 0x7f6adc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6adc0eb000 0x7f6adc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc0ede60 con 0x7f6adc0ee3e0 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0eba20 con 0x7f6adc0eb000 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.835 7f6ae3ad6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc0d1ce0 con 0x7f6adc0d18e0 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6adc0eb000 0x7f6adc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae20f3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6adc0ee3e0 0x7f6adc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53750/0 (socket says 172.21.2.117:53750) 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae20f3700 1 -- 172.21.2.117:0/2413935237 learned_addr learned my addr 172.21.2.117:0/2413935237 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 -- 172.21.2.117:0/2413935237 >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 msgr2=0x7f6adc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae18f2700 1 --2- 172.21.2.117:0/2413935237 >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 0x7f6adc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 --2- 172.21.2.117:0/2413935237 >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 0x7f6adc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 -- 172.21.2.117:0/2413935237 >> v1:172.21.2.117:6789/0 conn(0x7f6adc0ee3e0 legacy=0x7f6adc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 -- 172.21.2.117:0/2413935237 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0eefd0 con 0x7f6adc0eb000 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae18f2700 1 --2- 172.21.2.117:0/2413935237 >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 0x7f6adc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae10f1700 1 --2- 172.21.2.117:0/2413935237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0eb000 0x7f6adc0eb400 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f6acc006650 tx=0x7f6acc009a80).ready entity=mon.0 client_cookie=85c06ed22257d371 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae08f0700 1 -- 172.21.2.117:0/2413935237 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc0009b0 con 0x7f6adc0eb000 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae08f0700 1 -- 172.21.2.117:0/2413935237 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc00c0c0 con 0x7f6adc0eb000 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae08f0700 1 -- 172.21.2.117:0/2413935237 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6acc00c550 con 0x7f6adc0eb000 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.836 7f6ae3ad6700 1 -- 172.21.2.117:0/2413935237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0eb000 msgr2=0x7f6adc0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 --2- 172.21.2.117:0/2413935237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0eb000 0x7f6adc0eb400 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7f6acc006650 tx=0x7f6acc009a80).stop 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 -- 172.21.2.117:0/2413935237 shutdown_connections 2021-02-10T11:48:47.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 --2- 172.21.2.117:0/2413935237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0eb000 0x7f6adc0eb400 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 --2- 172.21.2.117:0/2413935237 >> v2:172.21.2.125:3300/0 conn(0x7f6adc0d18e0 0x7f6adc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 -- 172.21.2.117:0/2413935237 >> 172.21.2.117:0/2413935237 conn(0x7f6adc0ced60 msgr2=0x7f6adc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 -- 172.21.2.117:0/2413935237 shutdown_connections 2021-02-10T11:48:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.837 7f6ae3ad6700 1 -- 172.21.2.117:0/2413935237 wait complete. 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 Processor -- start 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 -- start start 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6adc190960 0x7f6adc190d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6adc0b2a40 0x7f6adc0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1b12b0 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0b3460 con 0x7f6adc0b2a40 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.852 7f6ae3ad6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0b3640 con 0x7f6adc190960 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53504/0 (socket says 172.21.2.117:53504) 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 -- 172.21.2.117:0/476938172 learned_addr learned my addr 172.21.2.117:0/476938172 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 -- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6adc190960 msgr2=0x7f6adc190d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6adc190960 0x7f6adc190d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 -- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6adc0b2a40 msgr2=0x7f6adc0b2e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6adc0b2a40 0x7f6adc0b2e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 -- 172.21.2.117:0/476938172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0b3860 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae18f2700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7f6ad8006f20 tx=0x7f6ad800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad800bd80 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad8000c90 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ad80022b0 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.853 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc0b3860 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.854 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc0b3860 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.854 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ad800a000 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.854 7f6ad27fc700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 0x7f6ac401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:47.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.854 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ad8032550 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.855 7f6ae10f1700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 0x7f6ac401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:47.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.855 7f6ae10f1700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 0x7f6ac401abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:47.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.885 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc1915a0 con 0x7f6adc0e6ec0 2021-02-10T11:48:47.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:47.888 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ad8006bb0 con 0x7f6adc0e6ec0 2021-02-10T11:48:48.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.016 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6adc1915a0 con 0x7f6adc0e6ec0 2021-02-10T11:48:48.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.017 7f6ad27fc700 1 -- 172.21.2.117:0/476938172 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f6ad800bd80 con 0x7f6adc0e6ec0 2021-02-10T11:48:48.012 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:48.013 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":77,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:48:40.061715","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1898190750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1898190750},{"type":"v1","addr":"172.21.2.125:6835","nonce":1898190750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:48:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 77 2021-02-10T11:48:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 msgr2=0x7f6ac401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 0x7f6ac401abc0 crc :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 msgr2=0x7f6adc190340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7f6ad8006f20 tx=0x7f6ad800b990).stop 2021-02-10T11:48:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 shutdown_connections 2021-02-10T11:48:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6adc0b2a40 0x7f6adc0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6adc190960 0x7f6adc190d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ac40186c0 0x7f6ac401abc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 --2- 172.21.2.117:0/476938172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6adc0e6ec0 0x7f6adc190340 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 >> 172.21.2.117:0/476938172 conn(0x7f6adc1ac380 msgr2=0x7f6adc192000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 shutdown_connections 2021-02-10T11:48:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:48.019 7f6ae3ad6700 1 -- 172.21.2.117:0/476938172 wait complete. 2021-02-10T11:48:48.223 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:38.055110', 'modified': '2021-02-10 11:48:40.061715', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'b', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1898190750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1898190750}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1898190750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:48.224 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:48.224 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6686, 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.568852898999999, 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 77, 2021-02-10T11:48:48.622 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 100, 2021-02-10T11:48:48.623 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:48:48.623 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.197438002999998 2021-02-10T11:48:48.623 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:48:48.626 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6686, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.568852898999999, "mdsmap_epoch": 77, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 16.197438002999998 } 2021-02-10T11:48:48.626 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:48:48.626 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:48:48.626 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:48:48.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:48.627 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:48.962 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:49.307 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:49.307 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:49.650 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:49.650 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:49.651 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:49.651 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:49.988 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:48:49.989 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:49.991 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:50.337 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:50.337 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:50.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[154361]: starting ceph client 2021-02-10T11:48:50.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:48:50.407 7f9274f23f80 -1 init, newargv = 0x55accecd8f90 newargc=9 2021-02-10T11:48:50.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[154361]: starting fuse 2021-02-10T11:48:50.679 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:50.680 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:50.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:50.681 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:51.014 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:48:51.014 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:48:51.014 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:48:51.015 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:48:51.016 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:48:51.378 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.154361.asok 2021-02-10T11:48:51.380 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.154361.asok 2021-02-10T11:48:51.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.380 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.154361.asok status 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "154361", 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:48:51.772 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10263, 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10263 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1188480996 2021-02-10T11:48:51.773 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1188480996 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10263 172.21.2.117:0/1188480996", 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1188480996", 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 77, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 100, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:48:51.774 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:48:51.775 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:48:51.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:51.776 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:51.776 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:52.111 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:48:52.111 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:48:52.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:52.111 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:48:52.458 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.874 7f782a0f2700 1 Processor -- start 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.874 7f782a0f2700 1 -- start start 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f782a0f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78240ef3f0 0x7f78240eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f782a0f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78240eb7f0 0x7f78241b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f782a0f2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78240ede60 con 0x7f78240d18e0 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f782a0f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78240ef7f0 con 0x7f78240ef3f0 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f782a0f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78240d1ce0 con 0x7f78240eb7f0 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78240ef3f0 0x7f78240eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f78237fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f78240d18e0 0x7f78240ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53774/0 (socket says 172.21.2.117:53774) 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f78237fe700 1 -- 172.21.2.117:0/2925856698 learned_addr learned my addr 172.21.2.117:0/2925856698 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 -- 172.21.2.117:0/2925856698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78240eb7f0 msgr2=0x7f78241b7960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 --2- 172.21.2.117:0/2925856698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78240eb7f0 0x7f78241b7960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 -- 172.21.2.117:0/2925856698 >> v1:172.21.2.117:6789/0 conn(0x7f78240d18e0 legacy=0x7f78240ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 -- 172.21.2.117:0/2925856698 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78241b8f90 con 0x7f78240ef3f0 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.875 7f7822ffd700 1 --2- 172.21.2.117:0/2925856698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240ef3f0 0x7f78240eb1d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7f780c006650 tx=0x7f780c009a80).ready entity=mon.0 client_cookie=8a362ca4ac836e5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f78227fc700 1 -- 172.21.2.117:0/2925856698 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c0009b0 con 0x7f78240ef3f0 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f78227fc700 1 -- 172.21.2.117:0/2925856698 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c00c0c0 con 0x7f78240ef3f0 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f78227fc700 1 -- 172.21.2.117:0/2925856698 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f780c00c5e0 con 0x7f78240ef3f0 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 -- 172.21.2.117:0/2925856698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240ef3f0 msgr2=0x7f78240eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 --2- 172.21.2.117:0/2925856698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240ef3f0 0x7f78240eb1d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7f780c006650 tx=0x7f780c009a80).stop 2021-02-10T11:48:52.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 -- 172.21.2.117:0/2925856698 shutdown_connections 2021-02-10T11:48:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 --2- 172.21.2.117:0/2925856698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78240eb7f0 0x7f78241b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 --2- 172.21.2.117:0/2925856698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240ef3f0 0x7f78240eb1d0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 -- 172.21.2.117:0/2925856698 >> 172.21.2.117:0/2925856698 conn(0x7f78240ced60 msgr2=0x7f78240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 -- 172.21.2.117:0/2925856698 shutdown_connections 2021-02-10T11:48:52.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.876 7f782a0f2700 1 -- 172.21.2.117:0/2925856698 wait complete. 2021-02-10T11:48:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.891 7f782a0f2700 1 Processor -- start 2021-02-10T11:48:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.891 7f782a0f2700 1 -- start start 2021-02-10T11:48:52.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78241b20d0 0x7f78241b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7824091740 0x7f7824091b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78241b1b50 con 0x7f78240b51d0 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782419b770 con 0x7f78241b20d0 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f782a0f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782419b950 con 0x7f7824091740 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53532/0 (socket says 172.21.2.117:53532) 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 -- 172.21.2.117:0/3133285568 learned_addr learned my addr 172.21.2.117:0/3133285568 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 -- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7824091740 msgr2=0x7f7824091b40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7824091740 0x7f7824091b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 -- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78241b20d0 msgr2=0x7f78241b24d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78241b20d0 0x7f78241b24d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:52.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 -- 172.21.2.117:0/3133285568 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824092190 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f78237fe700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f781400a570 tx=0x7f781400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.892 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f781400ad00 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7814000f90 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7814004830 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7824092190 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7824092190 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f78140043a0 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f781bfff700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 0x7f780801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.893 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7814032790 con 0x7f78240b51d0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.894 7f7822ffd700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 0x7f780801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:52.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.894 7f7822ffd700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 0x7f780801abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:52.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.925 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782419b990 con 0x7f78240b51d0 2021-02-10T11:48:52.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:52.928 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7814000f90 con 0x7f78240b51d0 2021-02-10T11:48:53.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.051 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782419b990 con 0x7f78240b51d0 2021-02-10T11:48:53.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.052 7f781bfff700 1 -- 172.21.2.117:0/3133285568 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+14231 (secure 0 0 0) 0x7f78140043a0 con 0x7f78240b51d0 2021-02-10T11:48:53.048 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:53.048 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":100,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:48:39.057620","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:48:35.729626","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":21,"pool_name":"cephfs_data","create_time":"2021-02-10 11:48:37.102743","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:48:53.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 msgr2=0x7f780801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 0x7f780801abc0 crc :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 msgr2=0x7f782419b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f781400a570 tx=0x7f781400a910).stop 2021-02-10T11:48:53.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 shutdown_connections 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7824091740 0x7f7824091b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78241b20d0 0x7f78241b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7808018700 0x7f780801abc0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 --2- 172.21.2.117:0/3133285568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78240b51d0 0x7f782419b150 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 >> 172.21.2.117:0/3133285568 conn(0x7f782419f610 msgr2=0x7f78241ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 shutdown_connections 2021-02-10T11:48:53.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.054 7f782a0f2700 1 -- 172.21.2.117:0/3133285568 wait complete. 2021-02-10T11:48:53.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:48:35.729626', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:48:37.102743', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:48:53.058 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:48:53.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.477 7f57885cf700 1 Processor -- start 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.477 7f57885cf700 1 -- start start 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f57885cf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f57800ef3f0 0x7f57800eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f57885cf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f57800eb830 0x7f57801b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f57885cf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57800ef7f0 con 0x7f57800d18f0 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f57885cf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57800ede60 con 0x7f57800eb830 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f57885cf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57800d1cf0 con 0x7f57800ef3f0 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f57800eb830 0x7f57801b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f57800eb830 0x7f57801b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53542/0 (socket says 172.21.2.117:53542) 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 -- 172.21.2.117:0/3647199906 learned_addr learned my addr 172.21.2.117:0/3647199906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:53.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 -- 172.21.2.117:0/3647199906 >> v2:172.21.2.125:3300/0 conn(0x7f57800ef3f0 msgr2=0x7f57800eb210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 --2- 172.21.2.117:0/3647199906 >> v2:172.21.2.125:3300/0 conn(0x7f57800ef3f0 0x7f57800eb210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 -- 172.21.2.117:0/3647199906 >> v1:172.21.2.125:6789/0 conn(0x7f57800d18f0 legacy=0x7f57800eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.478 7f5786bec700 1 -- 172.21.2.117:0/3647199906 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57801b8fd0 con 0x7f57800eb830 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f5786bec700 1 --2- 172.21.2.117:0/3647199906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57800eb830 0x7f57801b79a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f5778006f00 tx=0x7f5778009f10).ready entity=mon.0 client_cookie=829784c548bf8171 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f57853e9700 1 -- 172.21.2.117:0/3647199906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f577800b3b0 con 0x7f57800eb830 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f57853e9700 1 -- 172.21.2.117:0/3647199906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577800b730 con 0x7f57800eb830 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f57853e9700 1 -- 172.21.2.117:0/3647199906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f577800bc50 con 0x7f57800eb830 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f57885cf700 1 -- 172.21.2.117:0/3647199906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57800eb830 msgr2=0x7f57801b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.479 7f57885cf700 1 --2- 172.21.2.117:0/3647199906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57800eb830 0x7f57801b79a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f5778006f00 tx=0x7f5778009f10).stop 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 -- 172.21.2.117:0/3647199906 shutdown_connections 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 --2- 172.21.2.117:0/3647199906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f57800eb830 0x7f57801b79a0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 --2- 172.21.2.117:0/3647199906 >> v2:172.21.2.125:3300/0 conn(0x7f57800ef3f0 0x7f57800eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:53.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 -- 172.21.2.117:0/3647199906 >> 172.21.2.117:0/3647199906 conn(0x7f57800cf060 msgr2=0x7f57800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 -- 172.21.2.117:0/3647199906 shutdown_connections 2021-02-10T11:48:53.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.480 7f57885cf700 1 -- 172.21.2.117:0/3647199906 wait complete. 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 Processor -- start 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 -- start start 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57801a4240 0x7f57801a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f578009ed70 0x7f578009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578009fb90 con 0x7f578009f790 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57800dc640 con 0x7f578009ed70 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.494 7f57885cf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5780097270 con 0x7f57801a4240 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53548/0 (socket says 172.21.2.117:53548) 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 -- 172.21.2.117:0/2180900726 learned_addr learned my addr 172.21.2.117:0/2180900726 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5785bea700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f578009ed70 0x7f578009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f57863eb700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57801a4240 0x7f57801a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 -- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57801a4240 msgr2=0x7f57801a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57801a4240 0x7f57801a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 -- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f578009ed70 msgr2=0x7f578009f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f578009ed70 0x7f578009f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 -- 172.21.2.117:0/2180900726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57800b8780 con 0x7f578009f790 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5786bec700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f5778009390 tx=0x7f577800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.495 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5778003030 con 0x7f578009f790 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57780068e0 con 0x7f578009f790 2021-02-10T11:48:53.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f577800c570 con 0x7f578009f790 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f57885cf700 1 -- 172.21.2.117:0/2180900726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57800b8780 con 0x7f578009f790 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f57885cf700 1 -- 172.21.2.117:0/2180900726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57800b8780 con 0x7f578009f790 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5778005820 con 0x7f578009f790 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f5772ffd700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 0x7f576801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.496 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5778032cc0 con 0x7f578009f790 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.497 7f57863eb700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 0x7f576801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:53.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.497 7f57863eb700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 0x7f576801abc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:53.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.528 7f57885cf700 1 -- 172.21.2.117:0/2180900726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57800daff0 con 0x7f578009f790 2021-02-10T11:48:53.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.530 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5778006c80 con 0x7f578009f790 2021-02-10T11:48:53.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.657 7f57885cf700 1 -- 172.21.2.117:0/2180900726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f57800ede60 con 0x7f578009f790 2021-02-10T11:48:53.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.657 7f5772ffd700 1 -- 172.21.2.117:0/2180900726 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f57780068e0 con 0x7f578009f790 2021-02-10T11:48:53.653 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 msgr2=0x7f576801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 0x7f576801abc0 crc :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 msgr2=0x7f57800b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f5778009390 tx=0x7f577800bd40).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 shutdown_connections 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f57680186c0 0x7f576801abc0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f578009f790 0x7f57800b8160 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f578009ed70 0x7f578009f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 --2- 172.21.2.117:0/2180900726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f57801a4240 0x7f57801a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:53.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 >> 172.21.2.117:0/2180900726 conn(0x7f5780087090 msgr2=0x7f578008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 shutdown_connections 2021-02-10T11:48:53.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:53.659 7f57885cf700 1 -- 172.21.2.117:0/2180900726 wait complete. 2021-02-10T11:48:53.662 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:48:54.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 Processor -- start 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 -- start start 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc001b07f0 0x7fbc000e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbc000e7d70 0x7fbc000e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbc000e8790 0x7fbc000e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc000e92f0 con 0x7fbc000e7d70 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc000e5ce0 con 0x7fbc000e8790 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.083 7fbc06de0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc000e77f0 con 0x7fbc001b07f0 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbc000e7d70 0x7fbc000e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbc000e7d70 0x7fbc000e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53554/0 (socket says 172.21.2.117:53554) 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 -- 172.21.2.117:0/551358973 learned_addr learned my addr 172.21.2.117:0/551358973 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:54.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 -- 172.21.2.117:0/551358973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc001b07f0 msgr2=0x7fbc000e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 --2- 172.21.2.117:0/551358973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc001b07f0 0x7fbc000e4e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 -- 172.21.2.117:0/551358973 >> v2:172.21.2.125:3300/0 conn(0x7fbc000e8790 msgr2=0x7fbc000e8b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 --2- 172.21.2.117:0/551358973 >> v2:172.21.2.125:3300/0 conn(0x7fbc000e8790 0x7fbc000e8b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 -- 172.21.2.117:0/551358973 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000e54b0 con 0x7fbc000e7d70 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbfffff700 1 --2- 172.21.2.117:0/551358973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000e7d70 0x7fbc000e8170 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7fbbf0006700 tx=0x7fbbf0009b00).ready entity=mon.0 client_cookie=eefc7fdc4a6e9284 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbff7fe700 1 -- 172.21.2.117:0/551358973 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf000c0c0 con 0x7fbc000e7d70 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbff7fe700 1 -- 172.21.2.117:0/551358973 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf000c340 con 0x7fbc000e7d70 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbbff7fe700 1 -- 172.21.2.117:0/551358973 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbf000c820 con 0x7fbc000e7d70 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbc06de0700 1 -- 172.21.2.117:0/551358973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000e7d70 msgr2=0x7fbc000e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.084 7fbc06de0700 1 --2- 172.21.2.117:0/551358973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000e7d70 0x7fbc000e8170 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7fbbf0006700 tx=0x7fbbf0009b00).stop 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 -- 172.21.2.117:0/551358973 shutdown_connections 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 --2- 172.21.2.117:0/551358973 >> v2:172.21.2.125:3300/0 conn(0x7fbc000e8790 0x7fbc000e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 --2- 172.21.2.117:0/551358973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000e7d70 0x7fbc000e8170 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 --2- 172.21.2.117:0/551358973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc001b07f0 0x7fbc000e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 -- 172.21.2.117:0/551358973 >> 172.21.2.117:0/551358973 conn(0x7fbc000ce500 msgr2=0x7fbc000e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 -- 172.21.2.117:0/551358973 shutdown_connections 2021-02-10T11:48:54.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.085 7fbc06de0700 1 -- 172.21.2.117:0/551358973 wait complete. 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 Processor -- start 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 -- start start 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc000b2d90 0x7fbc000b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc000b3e50 0x7fbc000b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc000b38d0 con 0x7fbc000b2370 2021-02-10T11:48:54.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0009ca00 con 0x7fbc000b3e50 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc06de0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0009cbe0 con 0x7fbc000b2d90 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc04bfc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc04bfc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53560/0 (socket says 172.21.2.117:53560) 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.100 7fbc04bfc700 1 -- 172.21.2.117:0/1847508353 learned_addr learned my addr 172.21.2.117:0/1847508353 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 -- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc000b2d90 msgr2=0x7fbc000b3190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc000b2d90 0x7fbc000b3190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 -- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc000b3e50 msgr2=0x7fbc000b4250 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc000b3e50 0x7fbc000b4250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 -- 172.21.2.117:0/1847508353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000b4870 con 0x7fbc000b2370 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc04bfc700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7fbbe800b5c0 tx=0x7fbbe800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe800bd50 con 0x7fbc000b2370 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe8006030 con 0x7fbc000b2370 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbbe80054d0 con 0x7fbc000b2370 2021-02-10T11:48:54.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc000b4870 con 0x7fbc000b2370 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.101 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc000b4dc0 con 0x7fbc000b2370 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.102 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbbe8006210 con 0x7fbc000b2370 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.102 7fbbfd7fa700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 0x7fbbe401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.102 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbbe80326f0 con 0x7fbc000b2370 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.103 7fbbfffff700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 0x7fbbe401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.103 7fbbfffff700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 0x7fbbe401abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.133 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc00193250 con 0x7fbc000b2370 2021-02-10T11:48:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.136 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbbe8008160 con 0x7fbc000b2370 2021-02-10T11:48:54.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.261 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fbc00193250 con 0x7fbc000b2370 2021-02-10T11:48:54.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.262 7fbbfd7fa700 1 -- 172.21.2.117:0/1847508353 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7fbbe8008160 con 0x7fbc000b2370 2021-02-10T11:48:54.258 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 msgr2=0x7fbbe401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 0x7fbbe401abc0 crc :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 msgr2=0x7fbc000b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7fbbe800b5c0 tx=0x7fbbe800b960).stop 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 shutdown_connections 2021-02-10T11:48:54.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbc000b3e50 0x7fbc000b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbc000b2d90 0x7fbc000b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbbe40186c0 0x7fbbe401abc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 --2- 172.21.2.117:0/1847508353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbc000b2370 0x7fbc000b2770 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 >> 172.21.2.117:0/1847508353 conn(0x7fbc000b1f50 msgr2=0x7fbc001049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 shutdown_connections 2021-02-10T11:48:54.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.264 7fbc06de0700 1 -- 172.21.2.117:0/1847508353 wait complete. 2021-02-10T11:48:54.267 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:48:54.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.687 7f01fa77a700 1 Processor -- start 2021-02-10T11:48:54.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.687 7f01fa77a700 1 -- start start 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01fa77a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f01f40ef3f0 0x7f01f40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01fa77a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40eb830 0x7f01f41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01fa77a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f40ef7f0 con 0x7f01f40d18f0 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01fa77a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f40eded0 con 0x7f01f40ef3f0 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01fa77a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40d1cf0 con 0x7f01f40eb830 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f37fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f01f40ef3f0 0x7f01f40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f01f40d18f0 0x7f01f40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53816/0 (socket says 172.21.2.117:53816) 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f3fff700 1 -- 172.21.2.117:0/2183122022 learned_addr learned my addr 172.21.2.117:0/2183122022 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f8d97700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40eb830 0x7f01f41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f37fe700 1 -- 172.21.2.117:0/2183122022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40eb830 msgr2=0x7f01f41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f37fe700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40eb830 0x7f01f41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f37fe700 1 -- 172.21.2.117:0/2183122022 >> v1:172.21.2.117:6789/0 conn(0x7f01f40d18f0 legacy=0x7f01f40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f2ffd700 1 -- 172.21.2.117:0/2183122022 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345784229 0 0) 0x7f01e4000ce0 con 0x7f01f40d18f0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.688 7f01f37fe700 1 -- 172.21.2.117:0/2183122022 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f41b8fd0 con 0x7f01f40ef3f0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01f37fe700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40ef3f0 0x7f01f40eb210 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f01e8006650 tx=0x7f01e8009a00).ready entity=mon.0 client_cookie=dd7567f998286feb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01f2ffd700 1 -- 172.21.2.117:0/2183122022 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01e8009df0 con 0x7f01f40ef3f0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01f2ffd700 1 -- 172.21.2.117:0/2183122022 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e80009b0 con 0x7f01f40ef3f0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01f2ffd700 1 -- 172.21.2.117:0/2183122022 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01e800c2b0 con 0x7f01f40ef3f0 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 -- 172.21.2.117:0/2183122022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40ef3f0 msgr2=0x7f01f40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40ef3f0 0x7f01f40eb210 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f01e8006650 tx=0x7f01e8009a00).stop 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 -- 172.21.2.117:0/2183122022 shutdown_connections 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40eb830 0x7f01f41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 --2- 172.21.2.117:0/2183122022 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40ef3f0 0x7f01f40eb210 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 -- 172.21.2.117:0/2183122022 >> 172.21.2.117:0/2183122022 conn(0x7f01f40cf060 msgr2=0x7f01f40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 -- 172.21.2.117:0/2183122022 shutdown_connections 2021-02-10T11:48:54.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.689 7f01fa77a700 1 -- 172.21.2.117:0/2183122022 wait complete. 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 Processor -- start 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 -- start start 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40a6ce0 0x7f01f40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01f4085ec0 0x7f01f40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40a67e0 con 0x7f01f40a2040 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40a7120 con 0x7f01f4085ec0 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01fa77a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40a2a60 con 0x7f01f40a6ce0 2021-02-10T11:48:54.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01f8d97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01f8d97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53572/0 (socket says 172.21.2.117:53572) 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01f8d97700 1 -- 172.21.2.117:0/1136178716 learned_addr learned my addr 172.21.2.117:0/1136178716 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.704 7f01f37fe700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01f4085ec0 0x7f01f40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f3fff700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40a6ce0 0x7f01f40dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 -- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40a6ce0 msgr2=0x7f01f40dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40a6ce0 0x7f01f40dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 -- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01f4085ec0 msgr2=0x7f01f40862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01f4085ec0 0x7f01f40862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 -- 172.21.2.117:0/1136178716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f41a2dc0 con 0x7f01f40a2040 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f8d97700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7f01ec00b970 tx=0x7f01ec00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ec000f90 con 0x7f01f40a2040 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec006aa0 con 0x7f01f40a2040 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ec001680 con 0x7f01f40a2040 2021-02-10T11:48:54.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f41a2dc0 con 0x7f01f40a2040 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.705 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f41a3350 con 0x7f01f40a2040 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.706 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01ec0011f0 con 0x7f01f40a2040 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.706 7f01f0ff9700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.706 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f01ec032a90 con 0x7f01f40a2040 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.707 7f01f3fff700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:54.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.707 7f01f3fff700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 0x7f01d401abc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:54.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.737 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f40daff0 con 0x7f01f40a2040 2021-02-10T11:48:54.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.740 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01ec000f20 con 0x7f01f40a2040 2021-02-10T11:48:54.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.867 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f01f40ede60 con 0x7f01f40a2040 2021-02-10T11:48:54.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.867 7f01f0ff9700 1 -- 172.21.2.117:0/1136178716 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f01ec0011f0 con 0x7f01f40a2040 2021-02-10T11:48:54.863 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 msgr2=0x7f01d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 0x7f01d401abc0 crc :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 msgr2=0x7f01f40a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7f01ec00b970 tx=0x7f01ec00bd10).stop 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 shutdown_connections 2021-02-10T11:48:54.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01d40186c0 0x7f01d401abc0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01f40a2040 0x7f01f40a2440 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01f4085ec0 0x7f01f40862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 --2- 172.21.2.117:0/1136178716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01f40a6ce0 0x7f01f40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.869 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 >> 172.21.2.117:0/1136178716 conn(0x7f01f40b6540 msgr2=0x7f01f408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.870 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 shutdown_connections 2021-02-10T11:48:54.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:54.870 7f01fa77a700 1 -- 172.21.2.117:0/1136178716 wait complete. 2021-02-10T11:48:54.873 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:48:54.874 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.288 7f7823fc0700 1 Processor -- start 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.288 7f7823fc0700 1 -- start start 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7823fc0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0e4e40 0x7f781c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7823fc0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f781c0e5240 0x7f781c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7823fc0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c085930 con 0x7f781c1b0810 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7823fc0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c087430 con 0x7f781c0e5240 2021-02-10T11:48:55.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7823fc0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0e5cc0 con 0x7f781c0e4e40 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f78225dd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f781c1b0810 0x7f781c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53828/0 (socket says 172.21.2.117:53828) 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f78225dd700 1 -- 172.21.2.117:0/3542105549 learned_addr learned my addr 172.21.2.117:0/3542105549 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967027595 0 0) 0x7f7814001500 con 0x7f781c1b0810 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 -- 172.21.2.117:0/3542105549 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7808002fe0 con 0x7f781c1b0810 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f78215db700 1 --2- 172.21.2.117:0/3542105549 >> v2:172.21.2.125:3300/0 conn(0x7f781c0e5240 0x7f781c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7821ddc700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0e4e40 0x7f781c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3516124270 0 0) 0x7f7814001500 con 0x7f781c1b0810 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 -- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0e4e40 msgr2=0x7f781c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0e4e40 0x7f781c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 -- 172.21.2.117:0/3542105549 >> v2:172.21.2.125:3300/0 conn(0x7f781c0e5240 msgr2=0x7f781c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.289 7f7820dda700 1 --2- 172.21.2.117:0/3542105549 >> v2:172.21.2.125:3300/0 conn(0x7f781c0e5240 0x7f781c084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c084840 con 0x7f781c1b0810 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7814001a00 con 0x7f781c1b0810 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7821ddc700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0e4e40 0x7f781c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 >> v1:172.21.2.117:6789/0 conn(0x7f781c1b0810 legacy=0x7f781c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7808003a70 0x7f7808005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7808006760 0x7f7808008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f780804d510 0x7f780804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78080502e0 con 0x7f780804d510 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7808050500 con 0x7f7808006760 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7808050720 con 0x7f7808003a70 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7821ddc700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7808003a70 0x7f7808005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78215db700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7808006760 0x7f7808008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f780804d510 0x7f780804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 -- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7808003a70 msgr2=0x7f7808005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:55.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7808003a70 0x7f7808005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 -- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7808006760 msgr2=0x7f7808008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7808006760 0x7f7808008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 -- 172.21.2.117:0/3542105549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78080509c0 con 0x7f780804d510 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f780804d510 0x7f780804fa30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f7814007a70 tx=0x7f781400a310).ready entity=mon.0 client_cookie=96c5e693da34c8fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f781400a910 con 0x7f780804d510 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781400ab90 con 0x7f780804d510 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7820dda700 1 -- 172.21.2.117:0/3542105549 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f78140070c0 con 0x7f780804d510 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7823fc0700 1 -- 172.21.2.117:0/3542105549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f780804d510 msgr2=0x7f780804fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f7823fc0700 1 --2- 172.21.2.117:0/3542105549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f780804d510 0x7f780804fa30 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f7814007a70 tx=0x7f781400a310).stop 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.290 7f78225dd700 1 -- 172.21.2.117:0/3542105549 reap_dead start 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.291 7f7823fc0700 1 -- 172.21.2.117:0/3542105549 shutdown_connections 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.291 7f7823fc0700 1 -- 172.21.2.117:0/3542105549 >> 172.21.2.117:0/3542105549 conn(0x7f781c0ce4e0 msgr2=0x7f781c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:55.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.291 7f7823fc0700 1 -- 172.21.2.117:0/3542105549 shutdown_connections 2021-02-10T11:48:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.291 7f7823fc0700 1 -- 172.21.2.117:0/3542105549 wait complete. 2021-02-10T11:48:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 Processor -- start 2021-02-10T11:48:55.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 -- start start 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781c0ab150 0x7f781c0ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0abb70 0x7f781c0ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0ac330 con 0x7f781c0ad550 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0adad0 con 0x7f781c0ab150 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7823fc0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0bf490 con 0x7f781c0abb70 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7821ddc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7821ddc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53588/0 (socket says 172.21.2.117:53588) 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f7821ddc700 1 -- 172.21.2.117:0/1242402541 learned_addr learned my addr 172.21.2.117:0/1242402541 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.306 7f78215db700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781c0ab150 0x7f781c0ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f78225dd700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0abb70 0x7f781c0ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 -- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0abb70 msgr2=0x7f781c0ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0abb70 0x7f781c0ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 -- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781c0ab150 msgr2=0x7f781c0ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781c0ab150 0x7f781c0ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 -- 172.21.2.117:0/1242402541 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c0abf70 con 0x7f781c0ad550 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7821ddc700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7f7810006f30 tx=0x7f781000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f781000bd80 con 0x7f781c0ad550 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7810002030 con 0x7f781c0ad550 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7810002510 con 0x7f781c0ad550 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c0abf70 con 0x7f781c0ad550 2021-02-10T11:48:55.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.307 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c0abf70 con 0x7f781c0ad550 2021-02-10T11:48:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.308 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7810000c90 con 0x7f781c0ad550 2021-02-10T11:48:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.308 7f780e7fc700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 0x7f780401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:48:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.308 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7810032940 con 0x7f781c0ad550 2021-02-10T11:48:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.308 7f78215db700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 0x7f780401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:48:55.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.309 7f78215db700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 0x7f780401abc0 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:48:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.338 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781c085930 con 0x7f781c0ad550 2021-02-10T11:48:55.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.341 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f781001f5b0 con 0x7f781c0ad550 2021-02-10T11:48:55.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.467 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f781c085930 con 0x7f781c0ad550 2021-02-10T11:48:55.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.468 7f780e7fc700 1 -- 172.21.2.117:0/1242402541 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f781000bd80 con 0x7f781c0ad550 2021-02-10T11:48:55.464 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:48:55.464 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":77,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:48:40.061715","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1898190750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1898190750},{"type":"v1","addr":"172.21.2.125:6835","nonce":1898190750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:48:55.464 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 77 2021-02-10T11:48:55.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 msgr2=0x7f780401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 0x7f780401abc0 crc :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 msgr2=0x7f781c0bee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:48:55.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7f7810006f30 tx=0x7f781000b990).stop 2021-02-10T11:48:55.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 shutdown_connections 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781c0abb70 0x7f781c0ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781c0ab150 0x7f781c0ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78040186c0 0x7f780401abc0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 --2- 172.21.2.117:0/1242402541 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f781c0ad550 0x7f781c0bee70 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 >> 172.21.2.117:0/1242402541 conn(0x7f781c0ace80 msgr2=0x7f781c0c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 shutdown_connections 2021-02-10T11:48:55.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:48:55.470 7f7823fc0700 1 -- 172.21.2.117:0/1242402541 wait complete. 2021-02-10T11:48:55.675 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:48:38.055110', 'modified': '2021-02-10 11:48:40.061715', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6686': {'gid': 6686, 'name': 'b', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1898190750', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1898190750}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1898190750}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:48:55.675 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:48:55.675 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:48:56.074 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:48:56.074 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:48:56.074 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 6686, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.021155061000002, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 77, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 100, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 100, 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.649737964 2021-02-10T11:48:56.075 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:48:56.078 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 6686, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.021155061000002, "mdsmap_epoch": 77, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "uptime": 23.649737964 } 2021-02-10T11:48:56.078 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:48:56.079 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:48:56.079 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:48:56.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:56.079 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:56.414 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:56.757 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:56.757 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:57.101 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:57.101 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:57.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:57.102 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:57.438 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:48:57.439 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:48:57.439 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:57.441 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:48:57.785 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:48:57.785 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:48:57.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[155002]: starting ceph client 2021-02-10T11:48:57.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:48:57.856 7f04c6519f80 -1 init, newargv = 0x55a58023cfd0 newargc=9 2021-02-10T11:48:57.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[155002]: starting fuse 2021-02-10T11:48:58.128 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:48:58.128 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:48:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:48:58.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:58.129 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:48:58.463 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:48:58.464 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:48:58.464 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:48:58.464 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:48:58.464 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:48:58.465 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:48:58.466 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:48:58.466 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:48:58.466 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:48:58.825 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.155002.asok 2021-02-10T11:48:58.828 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.155002.asok 2021-02-10T11:48:58.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:58.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.155002.asok status 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "155002", 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:48:59.219 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10308, 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10308 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:59.220 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3283090567 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3283090567 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10308 172.21.2.117:0/3283090567", 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3283090567", 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 77, 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 100, 2021-02-10T11:48:59.221 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:48:59.222 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:48:59.222 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:48:59.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:59.223 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:48:59.223 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:59.559 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:48:59.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:48:59.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:48:59.560 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:48:59.906 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:49:00.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 Processor -- start 2021-02-10T11:49:00.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 -- start start 2021-02-10T11:49:00.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f03e40e7170 0x7f03e40e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40e8380 0x7f03e40e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e40e7570 con 0x7f03e41b07e0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e40e5cd0 con 0x7f03e40e7170 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03ea5f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e40e77e0 con 0x7f03e40e8380 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03e3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f03e41b07e0 0x7f03e40e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53858/0 (socket says 172.21.2.117:53858) 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03e3fff700 1 -- 172.21.2.117:0/3580199753 learned_addr learned my addr 172.21.2.117:0/3580199753 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389439359 0 0) 0x7f03d40010b0 con 0x7f03e41b07e0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.674 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03cc002fe0 con 0x7f03e41b07e0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e37fe700 1 --2- 172.21.2.117:0/3580199753 >> v2:172.21.2.125:3300/0 conn(0x7f03e40e7170 0x7f03e40e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40e8380 0x7f03e40e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 35312363 0 0) 0x7f03d4001010 con 0x7f03e41b07e0 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40e8380 msgr2=0x7f03e40e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:00.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40e8380 0x7f03e40e8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 >> v2:172.21.2.125:3300/0 conn(0x7f03e40e7170 msgr2=0x7f03e40e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 --2- 172.21.2.117:0/3580199753 >> v2:172.21.2.125:3300/0 conn(0x7f03e40e7170 0x7f03e40e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e40e8da0 con 0x7f03e41b07e0 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f03d4001530 con 0x7f03e41b07e0 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 >> v1:172.21.2.117:6789/0 conn(0x7f03e41b07e0 legacy=0x7f03e40e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03cc003a60 0x7f03cc005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03cc006750 0x7f03cc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03cc04d500 0x7f03cc04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0502d0 con 0x7f03cc04d500 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0504f0 con 0x7f03cc006750 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc050710 con 0x7f03cc003a60 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03cc04d500 0x7f03cc04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e3fff700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03cc003a60 0x7f03cc005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e37fe700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03cc006750 0x7f03cc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 -- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03cc003a60 msgr2=0x7f03cc005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03cc003a60 0x7f03cc005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 -- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03cc006750 msgr2=0x7f03cc008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03cc006750 0x7f03cc008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 -- 172.21.2.117:0/3580199753 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0509b0 con 0x7f03cc04d500 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.675 7f03e8c0f700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03cc04d500 0x7f03cc04fa20 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f03dc006e50 tx=0x7f03dc009e70).ready entity=mon.0 client_cookie=77e0df44b9429c28 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03dc006a20 con 0x7f03cc04d500 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc00b300 con 0x7f03cc04d500 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03e2ffd700 1 -- 172.21.2.117:0/3580199753 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03dc00b7a0 con 0x7f03cc04d500 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 -- 172.21.2.117:0/3580199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03cc04d500 msgr2=0x7f03cc04fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:00.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 --2- 172.21.2.117:0/3580199753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03cc04d500 0x7f03cc04fa20 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f03dc006e50 tx=0x7f03dc009e70).stop 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03e8c0f700 1 -- 172.21.2.117:0/3580199753 reap_dead start 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 -- 172.21.2.117:0/3580199753 shutdown_connections 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 -- 172.21.2.117:0/3580199753 >> 172.21.2.117:0/3580199753 conn(0x7f03e40ce4f0 msgr2=0x7f03e40e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 -- 172.21.2.117:0/3580199753 shutdown_connections 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.676 7f03ea5f2700 1 -- 172.21.2.117:0/3580199753 wait complete. 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 Processor -- start 2021-02-10T11:49:00.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 -- start start 2021-02-10T11:49:00.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03e410caa0 0x7f03e40bb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40bb840 0x7f03e419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e419d3f0 con 0x7f03e410cea0 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e41a8380 con 0x7f03e410caa0 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03ea5f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e41a7ed0 con 0x7f03e40bb840 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03e8c0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03e8c0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53618/0 (socket says 172.21.2.117:53618) 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.691 7f03e8c0f700 1 -- 172.21.2.117:0/1189842413 learned_addr learned my addr 172.21.2.117:0/1189842413 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 -- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40bb840 msgr2=0x7f03e419d8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40bb840 0x7f03e419d8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 -- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03e410caa0 msgr2=0x7f03e40bb220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03e410caa0 0x7f03e40bb220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 -- 172.21.2.117:0/1189842413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e419df10 con 0x7f03e410cea0 2021-02-10T11:49:00.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e8c0f700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f03dc0098f0 tx=0x7f03dc009c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03dc00c2f0 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc009dd0 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e419df10 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e419c330 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.692 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03dc00c830 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.693 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03dc0056b0 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.693 7f03e0ff9700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 0x7f03c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.693 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f03dc032a60 con 0x7f03e410cea0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.693 7f03e3fff700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 0x7f03c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:00.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.694 7f03e3fff700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 0x7f03c401abc0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:00.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.724 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e410c180 con 0x7f03e410cea0 2021-02-10T11:49:00.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.726 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03dc0066f0 con 0x7f03e410cea0 2021-02-10T11:49:00.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:00.860 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f03e40e77e0 con 0x7f03e410cea0 2021-02-10T11:49:01.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.672 7f03e0ff9700 1 -- 172.21.2.117:0/1189842413 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v78) v1 ==== 136+0+0 (secure 0 0 0) 0x7f03dc01f880 con 0x7f03e410cea0 2021-02-10T11:49:01.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 msgr2=0x7f03c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 0x7f03c401abc0 crc :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 msgr2=0x7f03e419bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f03dc0098f0 tx=0x7f03dc009c90).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 shutdown_connections 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03c40186c0 0x7f03c401abc0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03e410cea0 0x7f03e419bd10 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03e40bb840 0x7f03e419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.679 7f03ea5f2700 1 --2- 172.21.2.117:0/1189842413 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03e410caa0 0x7f03e40bb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:01.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.680 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 >> 172.21.2.117:0/1189842413 conn(0x7f03e409a7e0 msgr2=0x7f03e41abce0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:01.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.680 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 shutdown_connections 2021-02-10T11:49:01.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:01.680 7f03ea5f2700 1 -- 172.21.2.117:0/1189842413 wait complete. 2021-02-10T11:49:01.684 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.099 7fbf74856700 1 Processor -- start 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.099 7fbf74856700 1 -- start start 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf74856700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf701b07f0 0x7fbf700e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf74856700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf700e7d70 0x7fbf700e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf74856700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700e77f0 con 0x7fbf700e8790 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf74856700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf700e92f0 con 0x7fbf700e7d70 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf74856700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf700e5ce0 con 0x7fbf701b07f0 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf6ee1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbf700e8790 0x7fbf700e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53876/0 (socket says 172.21.2.117:53876) 2021-02-10T11:49:02.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.100 7fbf6ee1d700 1 -- 172.21.2.117:0/1760804551 learned_addr learned my addr 172.21.2.117:0/1760804551 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949167525 0 0) 0x7fbf68000f90 con 0x7fbf700e8790 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c002fe0 con 0x7fbf700e8790 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6de1b700 1 --2- 172.21.2.117:0/1760804551 >> v2:172.21.2.125:3300/0 conn(0x7fbf700e7d70 0x7fbf700e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6e61c700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf701b07f0 0x7fbf700e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2827136302 0 0) 0x7fbf68001500 con 0x7fbf700e8790 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf701b07f0 msgr2=0x7fbf700e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf701b07f0 0x7fbf700e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 >> v2:172.21.2.125:3300/0 conn(0x7fbf700e7d70 msgr2=0x7fbf700e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 --2- 172.21.2.117:0/1760804551 >> v2:172.21.2.125:3300/0 conn(0x7fbf700e7d70 0x7fbf700e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700e54b0 con 0x7fbf700e8790 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fbf68001a00 con 0x7fbf700e8790 2021-02-10T11:49:02.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 >> v1:172.21.2.117:6789/0 conn(0x7fbf700e8790 legacy=0x7fbf700e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c003ac0 0x7fbf5c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c04a7a0 0x7fbf5c04cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c04d5a0 0x7fbf5c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0502e0 con 0x7fbf5c003ac0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c050500 con 0x7fbf5c04a7a0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c050720 con 0x7fbf5c04d5a0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6e61c700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c003ac0 0x7fbf5c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.101 7fbf6ee1d700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c04d5a0 0x7fbf5c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 -- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c04d5a0 msgr2=0x7fbf5c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c04d5a0 0x7fbf5c04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 -- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c04a7a0 msgr2=0x7fbf5c04cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6de1b700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c04a7a0 0x7fbf5c04cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c04a7a0 0x7fbf5c04cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 -- 172.21.2.117:0/1760804551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0509c0 con 0x7fbf5c003ac0 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6de1b700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c04a7a0 0x7fbf5c04cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6e61c700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c003ac0 0x7fbf5c005ef0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7fbf58006ac0 tx=0x7fbf58009a30).ready entity=mon.0 client_cookie=c78ea5f2ae2a6cc8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5800c0c0 con 0x7fbf5c003ac0 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf58009e20 con 0x7fbf5c003ac0 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6d61a700 1 -- 172.21.2.117:0/1760804551 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5800c6c0 con 0x7fbf5c003ac0 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf74856700 1 -- 172.21.2.117:0/1760804551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c003ac0 msgr2=0x7fbf5c005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf74856700 1 --2- 172.21.2.117:0/1760804551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c003ac0 0x7fbf5c005ef0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7fbf58006ac0 tx=0x7fbf58009a30).stop 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.102 7fbf6ee1d700 1 -- 172.21.2.117:0/1760804551 reap_dead start 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.103 7fbf74856700 1 -- 172.21.2.117:0/1760804551 shutdown_connections 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.103 7fbf74856700 1 -- 172.21.2.117:0/1760804551 >> 172.21.2.117:0/1760804551 conn(0x7fbf700ce500 msgr2=0x7fbf700e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.103 7fbf74856700 1 -- 172.21.2.117:0/1760804551 shutdown_connections 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.103 7fbf74856700 1 -- 172.21.2.117:0/1760804551 wait complete. 2021-02-10T11:49:02.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.117 7fbf74856700 1 Processor -- start 2021-02-10T11:49:02.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 -- start start 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf70110080 0x7fbf70194640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf70194c60 0x7fbf70193570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70110610 con 0x7fbf70193b90 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70195060 con 0x7fbf70194c60 2021-02-10T11:49:02.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf74856700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf701a26e0 con 0x7fbf70110080 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf6ee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf6ee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53640/0 (socket says 172.21.2.117:53640) 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.118 7fbf6ee1d700 1 -- 172.21.2.117:0/2722013838 learned_addr learned my addr 172.21.2.117:0/2722013838 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 -- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf70110080 msgr2=0x7fbf70194640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf70110080 0x7fbf70194640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 -- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf70194c60 msgr2=0x7fbf70193570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf70194c60 0x7fbf70193570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 -- 172.21.2.117:0/2722013838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf701a2900 con 0x7fbf70193b90 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf6ee1d700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7fbf6800a8a0 tx=0x7fbf6800ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf68004930 con 0x7fbf70193b90 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf68004b70 con 0x7fbf70193b90 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf68007020 con 0x7fbf70193b90 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf74856700 1 -- 172.21.2.117:0/2722013838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf701a2900 con 0x7fbf70193b90 2021-02-10T11:49:02.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf74856700 1 -- 172.21.2.117:0/2722013838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf701a2900 con 0x7fbf70193b90 2021-02-10T11:49:02.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.119 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf68004720 con 0x7fbf70193b90 2021-02-10T11:49:02.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.120 7fbf66ffd700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 0x7fbf5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.120 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf68032ae0 con 0x7fbf70193b90 2021-02-10T11:49:02.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.120 7fbf6e61c700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 0x7fbf5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.120 7fbf6e61c700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 0x7fbf5401abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.150 7fbf74856700 1 -- 172.21.2.117:0/2722013838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf70195060 con 0x7fbf70193b90 2021-02-10T11:49:02.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.152 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf68004ee0 con 0x7fbf70193b90 2021-02-10T11:49:02.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.286 7fbf74856700 1 -- 172.21.2.117:0/2722013838 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fbf70195060 con 0x7fbf540186c0 2021-02-10T11:49:02.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.287 7fbf66ffd700 1 -- 172.21.2.117:0/2722013838 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fbf58006520 con 0x7fbf540186c0 2021-02-10T11:49:02.285 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:49:02.285 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:02.285 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:49:02.285 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:02.286 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:49:02.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 msgr2=0x7fbf5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 0x7fbf5401abc0 crc :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 msgr2=0x7fbf70193f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 rx=0x7fbf6800a8a0 tx=0x7fbf6800ac40).stop 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 shutdown_connections 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf540186c0 0x7fbf5401abc0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf70193b90 0x7fbf70193f90 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf70194c60 0x7fbf70193570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 --2- 172.21.2.117:0/2722013838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf70110080 0x7fbf70194640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 >> 172.21.2.117:0/2722013838 conn(0x7fbf7008a780 msgr2=0x7fbf700c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 shutdown_connections 2021-02-10T11:49:02.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.291 7fbf74856700 1 -- 172.21.2.117:0/2722013838 wait complete. 2021-02-10T11:49:02.296 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:49:02.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.714 7f34b9bff700 1 Processor -- start 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.714 7f34b9bff700 1 -- start start 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b9bff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 0x7f34b40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b9bff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34b40e6170 0x7f34b40e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b9bff700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b40e77f0 con 0x7f34b40e6b90 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b9bff700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40e5ce0 con 0x7f34b40e6170 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b9bff700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40e54b0 con 0x7f34b41b07f0 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34b40e6170 0x7f34b40e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f34b40e6170 0x7f34b40e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53644/0 (socket says 172.21.2.117:53644) 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b2ffd700 1 -- 172.21.2.117:0/2332477058 learned_addr learned my addr 172.21.2.117:0/2332477058 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f34b40e6b90 0x7f34b40e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53894/0 (socket says 172.21.2.117:53894) 2021-02-10T11:49:02.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.715 7f34b37fe700 1 --2- 172.21.2.117:0/2332477058 >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 0x7f34b40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b2ffd700 1 -- 172.21.2.117:0/2332477058 >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 msgr2=0x7f34b40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b2ffd700 1 --2- 172.21.2.117:0/2332477058 >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 0x7f34b40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b2ffd700 1 -- 172.21.2.117:0/2332477058 >> v1:172.21.2.117:6789/0 conn(0x7f34b40e6b90 legacy=0x7f34b40e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b2ffd700 1 -- 172.21.2.117:0/2332477058 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40e8d80 con 0x7f34b40e6170 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b37fe700 1 --2- 172.21.2.117:0/2332477058 >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 0x7f34b40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b2ffd700 1 --2- 172.21.2.117:0/2332477058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b40e6170 0x7f34b40e6570 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f34a8006700 tx=0x7f34a8009b80).ready entity=mon.0 client_cookie=357cb5ae4705cbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b27fc700 1 -- 172.21.2.117:0/2332477058 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a800c0c0 con 0x7f34b40e6170 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b27fc700 1 -- 172.21.2.117:0/2332477058 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a800c340 con 0x7f34b40e6170 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b27fc700 1 -- 172.21.2.117:0/2332477058 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34a800c860 con 0x7f34b40e6170 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b9bff700 1 -- 172.21.2.117:0/2332477058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b40e6170 msgr2=0x7f34b40e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.716 7f34b9bff700 1 --2- 172.21.2.117:0/2332477058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b40e6170 0x7f34b40e6570 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7f34a8006700 tx=0x7f34a8009b80).stop 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 -- 172.21.2.117:0/2332477058 shutdown_connections 2021-02-10T11:49:02.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 --2- 172.21.2.117:0/2332477058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b40e6170 0x7f34b40e6570 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 --2- 172.21.2.117:0/2332477058 >> v2:172.21.2.125:3300/0 conn(0x7f34b41b07f0 0x7f34b40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 -- 172.21.2.117:0/2332477058 >> 172.21.2.117:0/2332477058 conn(0x7f34b40ce500 msgr2=0x7f34b40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 -- 172.21.2.117:0/2332477058 shutdown_connections 2021-02-10T11:49:02.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.717 7f34b9bff700 1 -- 172.21.2.117:0/2332477058 wait complete. 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 Processor -- start 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 -- start start 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 0x7f34b40a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 0x7f34b40b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40ba4a0 con 0x7f34b4196330 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40bad20 con 0x7f34b4195f30 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b9bff700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40baf40 con 0x7f34b40a4f80 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53652/0 (socket says 172.21.2.117:53652) 2021-02-10T11:49:02.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.732 7f34b3fff700 1 -- 172.21.2.117:0/1360164506 learned_addr learned my addr 172.21.2.117:0/1360164506 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b2ffd700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 0x7f34b40b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b37fe700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 0x7f34b40a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 -- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 msgr2=0x7f34b40a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 0x7f34b40a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 -- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 msgr2=0x7f34b40b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 0x7f34b40b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 -- 172.21.2.117:0/1360164506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40bb160 con 0x7f34b4196330 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b2ffd700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 0x7f34b40b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b37fe700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 0x7f34b40a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f34b3fff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f34ac009960 tx=0x7f34ac00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34ac0060c0 con 0x7f34b4196330 2021-02-10T11:49:02.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.733 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac006340 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f34ac004720 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b40bb160 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b40bb6f0 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f34ac006520 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f349bfff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 0x7f349401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.734 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f34ac032930 con 0x7f34b4196330 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.735 7f34b37fe700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 0x7f349401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:02.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.735 7f34b37fe700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 0x7f349401abc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:02.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.765 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b407e800 con 0x7f34b4196330 2021-02-10T11:49:02.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.767 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f34ac006f00 con 0x7f34b4196330 2021-02-10T11:49:02.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.917 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f34b40e77f0 con 0x7f34b4196330 2021-02-10T11:49:02.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.917 7f349bfff700 1 -- 172.21.2.117:0/1360164506 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f34ac006500 con 0x7f34b4196330 2021-02-10T11:49:02.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 msgr2=0x7f349401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 0x7f349401abc0 crc :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 msgr2=0x7f34b40b9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7f34ac009960 tx=0x7f34ac00bcf0).stop 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 shutdown_connections 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34940186c0 0x7f349401abc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f34b4196330 0x7f34b40b9e80 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f34b4195f30 0x7f34b40b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 --2- 172.21.2.117:0/1360164506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f34b40a4f80 0x7f34b40a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 >> 172.21.2.117:0/1360164506 conn(0x7f34b40a7210 msgr2=0x7f34b4095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 shutdown_connections 2021-02-10T11:49:02.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:02.919 7f34b9bff700 1 -- 172.21.2.117:0/1360164506 wait complete. 2021-02-10T11:49:02.924 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000471908 2021-02-10T11:49:03.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.344 7ffa93d7c700 1 Processor -- start 2021-02-10T11:49:03.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.344 7ffa93d7c700 1 -- start start 2021-02-10T11:49:03.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa93d7c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa93d7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0eb040 0x7ffa8c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa93d7c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8c0eba60 con 0x7ffa8c0ee3e0 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa93d7c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa8c0d1cf0 con 0x7ffa8c0d18f0 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa93d7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c0eefd0 con 0x7ffa8c0eb040 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53656/0 (socket says 172.21.2.117:53656) 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 -- 172.21.2.117:0/538474260 learned_addr learned my addr 172.21.2.117:0/538474260 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91397700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0eb040 0x7ffa8c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 -- 172.21.2.117:0/538474260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0eb040 msgr2=0x7ffa8c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0eb040 0x7ffa8c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 -- 172.21.2.117:0/538474260 >> v1:172.21.2.125:6789/0 conn(0x7ffa8c0ee3e0 legacy=0x7ffa8c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.345 7ffa91b98700 1 -- 172.21.2.117:0/538474260 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c0ef1f0 con 0x7ffa8c0d18f0 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa91b98700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7ffa80006700 tx=0x7ffa80009b80).ready entity=mon.0 client_cookie=c7892581dbd97918 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa90b96700 1 -- 172.21.2.117:0/538474260 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8000c0c0 con 0x7ffa8c0d18f0 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa90b96700 1 -- 172.21.2.117:0/538474260 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8000c440 con 0x7ffa8c0d18f0 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa90b96700 1 -- 172.21.2.117:0/538474260 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8000c960 con 0x7ffa8c0d18f0 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 -- 172.21.2.117:0/538474260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c0d18f0 msgr2=0x7ffa8c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7ffa80006700 tx=0x7ffa80009b80).stop 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 -- 172.21.2.117:0/538474260 shutdown_connections 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0eb040 0x7ffa8c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 --2- 172.21.2.117:0/538474260 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c0d18f0 0x7ffa8c0eaa20 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.346 7ffa93d7c700 1 -- 172.21.2.117:0/538474260 >> 172.21.2.117:0/538474260 conn(0x7ffa8c0cf060 msgr2=0x7ffa8c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.347 7ffa93d7c700 1 -- 172.21.2.117:0/538474260 shutdown_connections 2021-02-10T11:49:03.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.347 7ffa93d7c700 1 -- 172.21.2.117:0/538474260 wait complete. 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.361 7ffa93d7c700 1 Processor -- start 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.361 7ffa93d7c700 1 -- start start 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0a6ce0 0x7ffa8c0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa8c0a2040 0x7ffa8c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c0a67e0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c0a7120 con 0x7ffa8c0a2040 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa93d7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c0a2a60 con 0x7ffa8c0a6ce0 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53664/0 (socket says 172.21.2.117:53664) 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 -- 172.21.2.117:0/4124423150 learned_addr learned my addr 172.21.2.117:0/4124423150 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 -- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0a6ce0 msgr2=0x7ffa8c0dc420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0a6ce0 0x7ffa8c0dc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 -- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa8c0a2040 msgr2=0x7ffa8c0a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa8c0a2040 0x7ffa8c0a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 -- 172.21.2.117:0/4124423150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c1a2dc0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.362 7ffa91397700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7ffa88006f00 tx=0x7ffa8800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa8800bd30 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa88002030 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa88002510 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c1a2dc0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c1a2dc0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa88000c90 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 0x7ffa7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.363 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffa88032500 con 0x7ffa8c085ec0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.364 7ffa91b98700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 0x7ffa7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:03.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.364 7ffa91b98700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 0x7ffa7401abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:03.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.395 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8c0daff0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.398 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa88030ce0 con 0x7ffa8c085ec0 2021-02-10T11:49:03.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.526 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000471908", "target": ["mgr", ""]}) v1 -- 0x7ffa8c0a70e0 con 0x7ffa740186c0 2021-02-10T11:49:03.526 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:03.532 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:03.526 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:03.532 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:03.526 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:03.532 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:03.526 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:03.532 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:03.526 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:03.532 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:03.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.560 7ffa7e7fc700 1 -- 172.21.2.117:0/4124423150 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ffa80006500 con 0x7ffa740186c0 2021-02-10T11:49:03.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 msgr2=0x7ffa7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 0x7ffa7401abc0 crc :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 msgr2=0x7ffa8c0862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7ffa88006f00 tx=0x7ffa8800b940).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 shutdown_connections 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa740186c0 0x7ffa7401abc0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa8c085ec0 0x7ffa8c0862c0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa8c0a2040 0x7ffa8c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 --2- 172.21.2.117:0/4124423150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa8c0a6ce0 0x7ffa8c0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 >> 172.21.2.117:0/4124423150 conn(0x7ffa8c0b6540 msgr2=0x7ffa8c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 shutdown_connections 2021-02-10T11:49:03.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:03.591 7ffa93d7c700 1 -- 172.21.2.117:0/4124423150 wait complete. 2021-02-10T11:49:03.595 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000471908 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.012 7f214cf3a700 1 Processor -- start 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.012 7f214cf3a700 1 -- start start 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f214cf3a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f21480d18f0 0x7f21480eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f214cf3a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21480ef3f0 0x7f21481b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f214cf3a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480ef7f0 con 0x7f21480eb040 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f214cf3a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480ede60 con 0x7f21480ef3f0 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f214cf3a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480d1cf0 con 0x7f21480d18f0 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21480ef3f0 0x7f21481b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2146312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f21480eb040 0x7f21480eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53920/0 (socket says 172.21.2.117:53920) 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f21480ef3f0 0x7f21481b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53670/0 (socket says 172.21.2.117:53670) 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2146312700 1 -- 172.21.2.117:0/1085725033 learned_addr learned my addr 172.21.2.117:0/1085725033 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 -- 172.21.2.117:0/1085725033 >> v2:172.21.2.125:3300/0 conn(0x7f21480d18f0 msgr2=0x7f21480eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 --2- 172.21.2.117:0/1085725033 >> v2:172.21.2.125:3300/0 conn(0x7f21480d18f0 0x7f21480eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 -- 172.21.2.117:0/1085725033 >> v1:172.21.2.117:6789/0 conn(0x7f21480eb040 legacy=0x7f21480eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.013 7f2147314700 1 -- 172.21.2.117:0/1085725033 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21481b8fd0 con 0x7f21480ef3f0 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f2147314700 1 --2- 172.21.2.117:0/1085725033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480ef3f0 0x7f21481b79a0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f2140006e30 tx=0x7f2140009e60).ready entity=mon.0 client_cookie=74c9f6931b903f88 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f2145b11700 1 -- 172.21.2.117:0/1085725033 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2140006a50 con 0x7f21480ef3f0 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f2145b11700 1 -- 172.21.2.117:0/1085725033 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f214000b300 con 0x7f21480ef3f0 2021-02-10T11:49:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f2145b11700 1 -- 172.21.2.117:0/1085725033 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f214000b820 con 0x7f21480ef3f0 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f214cf3a700 1 -- 172.21.2.117:0/1085725033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480ef3f0 msgr2=0x7f21481b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.014 7f214cf3a700 1 --2- 172.21.2.117:0/1085725033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480ef3f0 0x7f21481b79a0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f2140006e30 tx=0x7f2140009e60).stop 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 -- 172.21.2.117:0/1085725033 shutdown_connections 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 --2- 172.21.2.117:0/1085725033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480ef3f0 0x7f21481b79a0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 --2- 172.21.2.117:0/1085725033 >> v2:172.21.2.125:3300/0 conn(0x7f21480d18f0 0x7f21480eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 -- 172.21.2.117:0/1085725033 >> 172.21.2.117:0/1085725033 conn(0x7f21480cf060 msgr2=0x7f21480cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 -- 172.21.2.117:0/1085725033 shutdown_connections 2021-02-10T11:49:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.015 7f214cf3a700 1 -- 172.21.2.117:0/1085725033 wait complete. 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 Processor -- start 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 -- start start 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21480a6090 0x7f21480b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21480a6490 0x7f21480b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480f49d0 con 0x7f21480b5750 2021-02-10T11:49:04.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21480a69f0 con 0x7f21480a6090 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.029 7f214cf3a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21480a8a50 con 0x7f21480a6490 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53676/0 (socket says 172.21.2.117:53676) 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 -- 172.21.2.117:0/3381707328 learned_addr learned my addr 172.21.2.117:0/3381707328 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 -- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21480a6490 msgr2=0x7f21480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21480a6490 0x7f21480b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 -- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21480a6090 msgr2=0x7f21480b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21480a6090 0x7f21480b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 -- 172.21.2.117:0/3381707328 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480a7720 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f2146b13700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7f2138006f00 tx=0x7f213800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f213800bd30 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138000c90 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f21380022a0 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.030 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21480a7720 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.031 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480a7720 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.031 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2138005310 con 0x7f21480b5750 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.031 7f21377fe700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:04.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.031 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2138032670 con 0x7f21480b5750 2021-02-10T11:49:04.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.032 7f2146312700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:04.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.032 7f2146312700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 0x7f212c01abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:04.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.063 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21480f49d0 con 0x7f21480b5750 2021-02-10T11:49:04.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.066 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2138006b90 con 0x7f21480b5750 2021-02-10T11:49:04.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.190 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000471908", "target": ["mgr", ""]}) v1 -- 0x7f21480f49d0 con 0x7f212c0186c0 2021-02-10T11:49:04.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.195 7f21377fe700 1 -- 172.21.2.117:0/3381707328 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f213c0077d0 con 0x7f212c0186c0 2021-02-10T11:49:04.194 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000471908/3956d001-6c6e-4378-b23c-055701fa75bc 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.199 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 msgr2=0x7f212c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.199 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 0x7f212c01abc0 crc :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.199 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 msgr2=0x7f21480f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.199 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7f2138006f00 tx=0x7f213800b940).stop 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 shutdown_connections 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f21480a6490 0x7f21480b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f21480a6090 0x7f21480b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f212c0186c0 0x7f212c01abc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 --2- 172.21.2.117:0/3381707328 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f21480b5750 0x7f21480f43b0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:04.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 >> 172.21.2.117:0/3381707328 conn(0x7f2148085180 msgr2=0x7f21480a6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:04.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 shutdown_connections 2021-02-10T11:49:04.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:04.200 7f214cf3a700 1 -- 172.21.2.117:0/3381707328 wait complete. 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> import stat 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> import json 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> import sys 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> try: 2021-02-10T11:49:04.205 DEBUG:teuthology.orchestra.run.gibba017:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/_nogroup/subvolume_0000000000471908/3956d001-6c6e-4378-b23c-055701fa75bc") 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> except OSError as e: 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> sys.exit(e.errno) 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> print(json.dumps( 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> indent=2)) 2021-02-10T11:49:04.206 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:49:04.622 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_mode": 16877, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_ino": 1099511627779, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_dev": 42, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_nlink": 2, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_uid": 0, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_gid": 0, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_size": 0, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_atime": 1612957743.5439198, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_mtime": 1612957743.5439198, 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout: "st_ctime": 1612957743.5530317 2021-02-10T11:49:04.623 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:04.780 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000471908 2021-02-10T11:49:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 Processor -- start 2021-02-10T11:49:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 -- start start 2021-02-10T11:49:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 0x7efc680eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680ede60 con 0x7efc680eb040 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680d1cf0 con 0x7efc680ef3f0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc6d925700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc68088020 con 0x7efc680d18f0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc66d73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efc680eb040 0x7efc680eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53930/0 (socket says 172.21.2.117:53930) 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.237 7efc66d73700 1 -- 172.21.2.117:0/3911506549 learned_addr learned my addr 172.21.2.117:0/3911506549 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc67574700 1 --2- 172.21.2.117:0/3911506549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 0x7efc680eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc66572700 1 -- 172.21.2.117:0/3911506549 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672564930 0 0) 0x7efc5c0010b0 con 0x7efc680eb040 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc66572700 1 -- 172.21.2.117:0/3911506549 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50002fe0 con 0x7efc680eb040 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc66572700 1 -- 172.21.2.117:0/3911506549 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290891792 0 0) 0x7efc60001400 con 0x7efc680ef3f0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc66572700 1 -- 172.21.2.117:0/3911506549 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50003210 con 0x7efc680ef3f0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc67574700 1 -- 172.21.2.117:0/3911506549 >> v1:172.21.2.125:6789/0 conn(0x7efc680ef3f0 legacy=0x7efc681b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc67574700 1 -- 172.21.2.117:0/3911506549 >> v1:172.21.2.117:6789/0 conn(0x7efc680eb040 legacy=0x7efc680eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc67574700 1 -- 172.21.2.117:0/3911506549 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc681bb200 con 0x7efc680d18f0 2021-02-10T11:49:05.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc67574700 1 --2- 172.21.2.117:0/3911506549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 0x7efc680eaa20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7efc58006650 tx=0x7efc58009a80).ready entity=mon.2 client_cookie=54a344d08369f32f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.238 7efc66572700 1 -- 172.21.2.117:0/3911506549 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc580009b0 con 0x7efc680d18f0 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.239 7efc66572700 1 -- 172.21.2.117:0/3911506549 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc58006740 con 0x7efc680d18f0 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.239 7efc66572700 1 -- 172.21.2.117:0/3911506549 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5800c0c0 con 0x7efc680d18f0 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.239 7efc6d925700 1 -- 172.21.2.117:0/3911506549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 msgr2=0x7efc680eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.239 7efc6d925700 1 --2- 172.21.2.117:0/3911506549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 0x7efc680eaa20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7efc58006650 tx=0x7efc58009a80).stop 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.240 7efc6d925700 1 -- 172.21.2.117:0/3911506549 shutdown_connections 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.240 7efc6d925700 1 --2- 172.21.2.117:0/3911506549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc680d18f0 0x7efc680eaa20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.240 7efc6d925700 1 -- 172.21.2.117:0/3911506549 >> 172.21.2.117:0/3911506549 conn(0x7efc680cf060 msgr2=0x7efc680cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.240 7efc6d925700 1 -- 172.21.2.117:0/3911506549 shutdown_connections 2021-02-10T11:49:05.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.240 7efc6d925700 1 -- 172.21.2.117:0/3911506549 wait complete. 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.254 7efc6d925700 1 Processor -- start 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.254 7efc6d925700 1 -- start start 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 0x7efc681a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 0x7efc680a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6807cca0 con 0x7efc681a3930 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6818fb70 con 0x7efc680a13a0 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc6d925700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc680dc6d0 con 0x7efc6807c7c0 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc66d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 0x7efc681a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc66d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53686/0 (socket says 172.21.2.117:53686) 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc66d73700 1 -- 172.21.2.117:0/2360661716 learned_addr learned my addr 172.21.2.117:0/2360661716 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67d75700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 0x7efc680a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 -- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 msgr2=0x7efc680a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 0x7efc680a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 -- 172.21.2.117:0/2360661716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 msgr2=0x7efc680a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 -- 172.21.2.117:0/2360661716 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6807d220 con 0x7efc6807c7c0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67d75700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 0x7efc680a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc66d73700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.255 7efc67574700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 0x7efc681a3310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7efc5800b530 tx=0x7efc5800b8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc580092d0 con 0x7efc6807c7c0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5800bcc0 con 0x7efc6807c7c0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc580022b0 con 0x7efc6807c7c0 2021-02-10T11:49:05.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc6d925700 1 -- 172.21.2.117:0/2360661716 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6807d220 con 0x7efc6807c7c0 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc6d925700 1 -- 172.21.2.117:0/2360661716 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6807d7b0 con 0x7efc6807c7c0 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc580092d0 con 0x7efc6807c7c0 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 0x7efc4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.256 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efc58032650 con 0x7efc6807c7c0 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.257 7efc66d73700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 0x7efc4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:05.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.257 7efc66d73700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 0x7efc4801abc0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:05.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.287 7efc6d925700 1 -- 172.21.2.117:0/2360661716 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6818fb00 con 0x7efc6807c7c0 2021-02-10T11:49:05.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.291 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc58030cb0 con 0x7efc6807c7c0 2021-02-10T11:49:05.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.419 7efc6d925700 1 -- 172.21.2.117:0/2360661716 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000471908", "target": ["mgr", ""]}) v1 -- 0x7efc6807d7b0 con 0x7efc480186c0 2021-02-10T11:49:05.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.432 7efc4ffff700 1 -- 172.21.2.117:0/2360661716 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efc5c001610 con 0x7efc480186c0 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.435 7efc6d925700 1 -- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 msgr2=0x7efc4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.435 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 0x7efc4801abc0 crc :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.435 7efc6d925700 1 -- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 msgr2=0x7efc681a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.435 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 0x7efc681a3310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7efc5800b530 tx=0x7efc5800b8d0).stop 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 -- 172.21.2.117:0/2360661716 shutdown_connections 2021-02-10T11:49:05.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc480186c0 0x7efc4801abc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc681a3930 0x7efc680a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc680a13a0 0x7efc680a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 --2- 172.21.2.117:0/2360661716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6807c7c0 0x7efc681a3310 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 -- 172.21.2.117:0/2360661716 >> 172.21.2.117:0/2360661716 conn(0x7efc68087090 msgr2=0x7efc6808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 -- 172.21.2.117:0/2360661716 shutdown_connections 2021-02-10T11:49:05.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:05.436 7efc6d925700 1 -- 172.21.2.117:0/2360661716 wait complete. 2021-02-10T11:49:05.440 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:49:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:49:05.791 INFO:teuthology.orchestra.run.gibba017.stdout:0 2021-02-10T11:49:05.791 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:49:05.791 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:49:05.792 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:06.136 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:06.480 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:49:06.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:06.480 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:06.480 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:06.816 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:06.817 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:06.817 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:49:06.817 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:49:06.817 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:07.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[154361]: fuse finished with error 0 and tester_r 0 2021-02-10T11:49:07.170 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:07.171 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:07.171 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:07.505 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:49:07.506 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:49:07.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:07.506 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:07.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:07.843 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:07.843 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:08.179 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:08.179 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:49:08.180 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:49:08.180 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:49:08.180 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:08.531 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[155002]: fuse finished with error 0 and tester_r 0 2021-02-10T11:49:08.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:08.532 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:08.533 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:08.867 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:49:08.867 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:49:08.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:08.868 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:09.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:09.203 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:09.203 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:49:09.538 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:49:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:09.538 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:49:09.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:09.538 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:49:09.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:09.873 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:09.873 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:49:10.207 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:49:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:10.208 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:49:10.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:10.208 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:49:10.541 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 Processor -- start 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 -- start start 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f400d15e0 0x7f6f400e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f400e7580 con 0x7f6f401b07f0 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f400e77f0 con 0x7f6f400e7180 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.961 7f6f4585f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400e5ce0 con 0x7f6f400d15e0 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.962 7f6f3f574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6f401b07f0 0x7f6f400e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53940/0 (socket says 172.21.2.117:53940) 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.962 7f6f3f574700 1 -- 172.21.2.117:0/36882043 learned_addr learned my addr 172.21.2.117:0/36882043 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:10.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.962 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373145657 0 0) 0x7f6f300010b0 con 0x7f6f401b07f0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.962 7f6f3e572700 1 -- 172.21.2.117:0/36882043 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f2c002fe0 con 0x7f6f401b07f0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.962 7f6f3fd75700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f400d15e0 0x7f6f400e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 531797220 0 0) 0x7f6f300010b0 con 0x7f6f401b07f0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f400d15e0 msgr2=0x7f6f400e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f400d15e0 0x7f6f400e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 >> v1:172.21.2.125:6789/0 conn(0x7f6f400e7180 legacy=0x7f6f400e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400d1a10 con 0x7f6f401b07f0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6f300015b0 con 0x7f6f401b07f0 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 >> v1:172.21.2.117:6789/0 conn(0x7f6f401b07f0 legacy=0x7f6f400e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f2c003a60 0x7f6f2c005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f2c006790 0x7f6f2c008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f2c009500 0x7f6f2c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c00c2d0 con 0x7f6f2c009500 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c00c4f0 con 0x7f6f2c003a60 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2c050720 con 0x7f6f2c006790 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3f574700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f2c003a60 0x7f6f2c005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f2c009500 0x7f6f2c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3ed73700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f2c006790 0x7f6f2c008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 -- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f2c006790 msgr2=0x7f6f2c008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f2c006790 0x7f6f2c008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 -- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f2c003a60 msgr2=0x7f6f2c005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f2c003a60 0x7f6f2c005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 -- 172.21.2.117:0/36882043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c050980 con 0x7f6f2c009500 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3fd75700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f2c009500 0x7f6f2c00ba20 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f6f38009b10 tx=0x7f6f38009eb0).ready entity=mon.0 client_cookie=5632a79156654393 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3800b510 con 0x7f6f2c009500 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f38003030 con 0x7f6f2c009500 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f3e572700 1 -- 172.21.2.117:0/36882043 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3800b790 con 0x7f6f2c009500 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f4585f700 1 -- 172.21.2.117:0/36882043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f2c009500 msgr2=0x7f6f2c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.963 7f6f4585f700 1 --2- 172.21.2.117:0/36882043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f2c009500 0x7f6f2c00ba20 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f6f38009b10 tx=0x7f6f38009eb0).stop 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.964 7f6f3fd75700 1 -- 172.21.2.117:0/36882043 reap_dead start 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.964 7f6f4585f700 1 -- 172.21.2.117:0/36882043 shutdown_connections 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.964 7f6f4585f700 1 -- 172.21.2.117:0/36882043 >> 172.21.2.117:0/36882043 conn(0x7f6f400ce500 msgr2=0x7f6f400e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.964 7f6f4585f700 1 -- 172.21.2.117:0/36882043 shutdown_connections 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.964 7f6f4585f700 1 -- 172.21.2.117:0/36882043 wait complete. 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 Processor -- start 2021-02-10T11:49:10.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 -- start start 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f401af980 0x7f6f4009ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f401afd80 0x7f6f4009b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4010e2b0 con 0x7f6f400b3f30 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4009a8f0 con 0x7f6f401afd80 2021-02-10T11:49:10.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f4585f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400992b0 con 0x7f6f401af980 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f3f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f3f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53700/0 (socket says 172.21.2.117:53700) 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.979 7f6f3f574700 1 -- 172.21.2.117:0/919494198 learned_addr learned my addr 172.21.2.117:0/919494198 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 -- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f401af980 msgr2=0x7f6f4009ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f401af980 0x7f6f4009ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 -- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f401afd80 msgr2=0x7f6f4009b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f401afd80 0x7f6f4009b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 -- 172.21.2.117:0/919494198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400994d0 con 0x7f6f400b3f30 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f3f574700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f6f30001660 tx=0x7f6f3000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3000ad20 con 0x7f6f400b3f30 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f300016a0 con 0x7f6f400b3f30 2021-02-10T11:49:10.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f300043c0 con 0x7f6f400b3f30 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f4585f700 1 -- 172.21.2.117:0/919494198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f400994d0 con 0x7f6f400b3f30 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.980 7f6f4585f700 1 -- 172.21.2.117:0/919494198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f400994d0 con 0x7f6f400b3f30 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.981 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f30004140 con 0x7f6f400b3f30 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.981 7f6f37fff700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 0x7f6f2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.981 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6f30032520 con 0x7f6f400b3f30 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.981 7f6f3ed73700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 0x7f6f2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:10.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:10.982 7f6f3ed73700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 0x7f6f2401abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.011 7f6f4585f700 1 -- 172.21.2.117:0/919494198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f400e77f0 con 0x7f6f400b3f30 2021-02-10T11:49:11.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.014 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f30025040 con 0x7f6f400b3f30 2021-02-10T11:49:11.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.164 7f6f4585f700 1 -- 172.21.2.117:0/919494198 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6f400e77f0 con 0x7f6f400b3f30 2021-02-10T11:49:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.168 7f6f37fff700 1 -- 172.21.2.117:0/919494198 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6f30025040 con 0x7f6f400b3f30 2021-02-10T11:49:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.173 7f6f4585f700 1 -- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 msgr2=0x7f6f2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.173 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 0x7f6f2401abc0 crc :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.173 7f6f4585f700 1 -- 172.21.2.117:0/919494198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 msgr2=0x7f6f4010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.173 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f6f30001660 tx=0x7f6f3000a930).stop 2021-02-10T11:49:11.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 -- 172.21.2.117:0/919494198 shutdown_connections 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f401afd80 0x7f6f4009b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f401af980 0x7f6f4009ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f240186c0 0x7f6f2401abc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 --2- 172.21.2.117:0/919494198 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f400b3f30 0x7f6f4010e530 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 -- 172.21.2.117:0/919494198 >> 172.21.2.117:0/919494198 conn(0x7f6f4019f010 msgr2=0x7f6f40191b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 -- 172.21.2.117:0/919494198 shutdown_connections 2021-02-10T11:49:11.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.174 7f6f4585f700 1 -- 172.21.2.117:0/919494198 wait complete. 2021-02-10T11:49:11.179 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume' 2021-02-10T11:49:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.592 7f8467c19700 1 Processor -- start 2021-02-10T11:49:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 -- start start 2021-02-10T11:49:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84600e4e40 0x7f84601b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8460085450 0x7f84600d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8460085930 con 0x7f84600e5240 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84600e5cc0 con 0x7f8460085450 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8467c19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8460084840 con 0x7f84600e4e40 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8466236700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8460085450 0x7f84600d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8466236700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8460085450 0x7f84600d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53708/0 (socket says 172.21.2.117:53708) 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8465234700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84600e5240 0x7f8460084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53958/0 (socket says 172.21.2.117:53958) 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.593 7f8466236700 1 -- 172.21.2.117:0/1406357830 learned_addr learned my addr 172.21.2.117:0/1406357830 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8466236700 1 -- 172.21.2.117:0/1406357830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84600e4e40 msgr2=0x7f84601b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8466236700 1 --2- 172.21.2.117:0/1406357830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84600e4e40 0x7f84601b01f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8466236700 1 -- 172.21.2.117:0/1406357830 >> v1:172.21.2.117:6789/0 conn(0x7f84600e5240 legacy=0x7f8460084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8466236700 1 -- 172.21.2.117:0/1406357830 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600d1be0 con 0x7f8460085450 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8466236700 1 --2- 172.21.2.117:0/1406357830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8460085450 0x7f84600d05b0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f8458006b80 tx=0x7f8458009b10).ready entity=mon.0 client_cookie=ccba673310b4503b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8464a33700 1 -- 172.21.2.117:0/1406357830 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84580022e0 con 0x7f8460085450 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8464a33700 1 -- 172.21.2.117:0/1406357830 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8458002560 con 0x7f8460085450 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8464a33700 1 -- 172.21.2.117:0/1406357830 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8458002a80 con 0x7f8460085450 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8467c19700 1 -- 172.21.2.117:0/1406357830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8460085450 msgr2=0x7f84600d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.594 7f8467c19700 1 --2- 172.21.2.117:0/1406357830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8460085450 0x7f84600d05b0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f8458006b80 tx=0x7f8458009b10).stop 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 -- 172.21.2.117:0/1406357830 shutdown_connections 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 --2- 172.21.2.117:0/1406357830 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8460085450 0x7f84600d05b0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 --2- 172.21.2.117:0/1406357830 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84600e4e40 0x7f84601b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 -- 172.21.2.117:0/1406357830 >> 172.21.2.117:0/1406357830 conn(0x7f84600ce4e0 msgr2=0x7f84600e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 -- 172.21.2.117:0/1406357830 shutdown_connections 2021-02-10T11:49:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.595 7f8467c19700 1 -- 172.21.2.117:0/1406357830 wait complete. 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.609 7f8467c19700 1 Processor -- start 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.609 7f8467c19700 1 -- start start 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f846008b240 0x7f846008b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 0x7f8460197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846008a2e0 con 0x7f846008a820 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846008bc60 con 0x7f8460196f80 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8467c19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84601979a0 con 0x7f846008b240 2021-02-10T11:49:11.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8465a35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8465a35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53712/0 (socket says 172.21.2.117:53712) 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8465a35700 1 -- 172.21.2.117:0/983998197 learned_addr learned my addr 172.21.2.117:0/983998197 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8466236700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 0x7f8460197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.610 7f8465234700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f846008b240 0x7f846008b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 -- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f846008b240 msgr2=0x7f846008b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f846008b240 0x7f846008b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 -- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 msgr2=0x7f8460197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 0x7f8460197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 -- 172.21.2.117:0/983998197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460197bc0 con 0x7f846008a820 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8466236700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 0x7f8460197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:11.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8465a35700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f845000b5c0 tx=0x7f845000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f845000bd50 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8450000cd0 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84500062a0 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8467c19700 1 -- 172.21.2.117:0/983998197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8460197bc0 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f8467c19700 1 -- 172.21.2.117:0/983998197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8460197bc0 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.611 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8450006030 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.612 7f845e7fc700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 0x7f844001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.612 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f84500326a0 con 0x7f846008a820 2021-02-10T11:49:11.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.612 7f8465234700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 0x7f844001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:11.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.613 7f8465234700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 0x7f844001abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:11.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.643 7f8467c19700 1 -- 172.21.2.117:0/983998197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846008a2e0 con 0x7f846008a820 2021-02-10T11:49:11.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.646 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8450006b80 con 0x7f846008a820 2021-02-10T11:49:11.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:11.777 7f8467c19700 1 -- 172.21.2.117:0/983998197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]} v 0) v1 -- 0x7f846008a2e0 con 0x7f846008a820 2021-02-10T11:49:12.627 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:12.630 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:12.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.674 7f845e7fc700 1 -- 172.21.2.117:0/983998197 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume"]}]=0 v553) v1 ==== 147+0+0 (secure 0 0 0) 0x7f845001f4d0 con 0x7f846008a820 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.698 7f8467c19700 1 -- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 msgr2=0x7f844001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.698 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 0x7f844001abc0 crc :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.698 7f8467c19700 1 -- 172.21.2.117:0/983998197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 msgr2=0x7f846008ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.698 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 rx=0x7f845000b5c0 tx=0x7f845000b960).stop 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 -- 172.21.2.117:0/983998197 shutdown_connections 2021-02-10T11:49:12.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8460196f80 0x7f8460197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f846008b240 0x7f846008b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84400186c0 0x7f844001abc0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 --2- 172.21.2.117:0/983998197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f846008a820 0x7f846008ac20 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 -- 172.21.2.117:0/983998197 >> 172.21.2.117:0/983998197 conn(0x7f846019c290 msgr2=0x7f84601b96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 -- 172.21.2.117:0/983998197 shutdown_connections 2021-02-10T11:49:12.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:12.699 7f8467c19700 1 -- 172.21.2.117:0/983998197 wait complete. 2021-02-10T11:49:12.703 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:49:12.703 INFO:tasks.cephfs_test_runner:Starting test: test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:49:12.703 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 Processor -- start 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 -- start start 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0d18e0 0x7f489c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c0ede60 con 0x7f489c0eb000 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489c0d1ce0 con 0x7f489c0ef3f0 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f48a246d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c088020 con 0x7f489c0d18e0 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.158 7f489b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f489c0eb000 0x7f489c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:53974/0 (socket says 172.21.2.117:53974) 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489b7fe700 1 -- 172.21.2.117:0/965748795 learned_addr learned my addr 172.21.2.117:0/965748795 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999845234 0 0) 0x7f48900009b0 con 0x7f489c0eb000 2021-02-10T11:49:13.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4884002fe0 con 0x7f489c0eb000 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489bfff700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0d18e0 0x7f489c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 93245848 0 0) 0x7f4890001010 con 0x7f489c0eb000 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0d18e0 msgr2=0x7f489c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0d18e0 0x7f489c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 >> v1:172.21.2.125:6789/0 conn(0x7f489c0ef3f0 legacy=0x7f489c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c1bb180 con 0x7f489c0eb000 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4890001590 con 0x7f489c0eb000 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 >> v1:172.21.2.117:6789/0 conn(0x7f489c0eb000 legacy=0x7f489c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4884003ac0 0x7f4884005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4884006770 0x7f4884008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4884009500 0x7f488400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488400c2d0 con 0x7f4884003ac0 2021-02-10T11:49:13.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488400c4f0 con 0x7f4884009500 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489affd700 1 -- 172.21.2.117:0/965748795 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488400c710 con 0x7f4884006770 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.159 7f489bfff700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4884003ac0 0x7f4884005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489b7fe700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4884006770 0x7f4884008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f48a0a8a700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4884009500 0x7f488400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 -- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4884006770 msgr2=0x7f4884008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4884006770 0x7f4884008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 -- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4884009500 msgr2=0x7f488400ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4884009500 0x7f488400ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 -- 172.21.2.117:0/965748795 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488400c9b0 con 0x7f4884003ac0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489bfff700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4884003ac0 0x7f4884005ef0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f488c04aaa0 tx=0x7f488c04da20).ready entity=mon.0 client_cookie=4c5dce4513ac43e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c0009b0 con 0x7f4884003ac0 2021-02-10T11:49:13.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c0500c0 con 0x7f4884003ac0 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f489affd700 1 -- 172.21.2.117:0/965748795 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c050550 con 0x7f4884003ac0 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f48a246d700 1 -- 172.21.2.117:0/965748795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4884003ac0 msgr2=0x7f4884005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f48a246d700 1 --2- 172.21.2.117:0/965748795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4884003ac0 0x7f4884005ef0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7f488c04aaa0 tx=0x7f488c04da20).stop 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.160 7f48a0a8a700 1 -- 172.21.2.117:0/965748795 reap_dead start 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.161 7f48a246d700 1 -- 172.21.2.117:0/965748795 shutdown_connections 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.161 7f48a246d700 1 -- 172.21.2.117:0/965748795 >> 172.21.2.117:0/965748795 conn(0x7f489c0ced60 msgr2=0x7f489c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.161 7f48a246d700 1 -- 172.21.2.117:0/965748795 shutdown_connections 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.161 7f48a246d700 1 -- 172.21.2.117:0/965748795 wait complete. 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.175 7f48a246d700 1 Processor -- start 2021-02-10T11:49:13.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.175 7f48a246d700 1 -- start start 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0b82f0 0x7f489c0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c0b8d10 0x7f489c0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0c0670 con 0x7f489c1a33d0 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0deb20 con 0x7f489c0b8d10 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f48a246d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f489c0ded40 con 0x7f489c0b82f0 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53734/0 (socket says 172.21.2.117:53734) 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 -- 172.21.2.117:0/1920343127 learned_addr learned my addr 172.21.2.117:0/1920343127 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 -- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0b82f0 msgr2=0x7f489c0b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0b82f0 0x7f489c0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 -- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c0b8d10 msgr2=0x7f489c0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c0b8d10 0x7f489c0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.176 7f489bfff700 1 -- 172.21.2.117:0/1920343127 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c0def60 con 0x7f489c1a33d0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f489bfff700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f488c04d8a0 tx=0x7f488c04f9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c002030 con 0x7f489c1a33d0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488c002330 con 0x7f489c1a33d0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f488c002850 con 0x7f489c1a33d0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f48a246d700 1 -- 172.21.2.117:0/1920343127 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c0def60 con 0x7f489c1a33d0 2021-02-10T11:49:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.177 7f48a246d700 1 -- 172.21.2.117:0/1920343127 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c0df4f0 con 0x7f489c1a33d0 2021-02-10T11:49:13.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.178 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f488c000c90 con 0x7f489c1a33d0 2021-02-10T11:49:13.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.178 7f4898ff9700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 0x7f487c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:13.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.178 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f488c0766b0 con 0x7f489c1a33d0 2021-02-10T11:49:13.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.178 7f489b7fe700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 0x7f487c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:13.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.179 7f489b7fe700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 0x7f487c01abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.209 7f48a246d700 1 -- 172.21.2.117:0/1920343127 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c0c0670 con 0x7f489c1a33d0 2021-02-10T11:49:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.211 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f488c04d090 con 0x7f489c1a33d0 2021-02-10T11:49:13.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.343 7f48a246d700 1 -- 172.21.2.117:0/1920343127 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7f489c0c0670 con 0x7f489c1a33d0 2021-02-10T11:49:13.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.678 7f4898ff9700 1 -- 172.21.2.117:0/1920343127 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v554) v1 ==== 156+0+0 (secure 0 0 0) 0x7f488c000c90 con 0x7f489c1a33d0 2021-02-10T11:49:13.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 msgr2=0x7f487c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 0x7f487c01abc0 crc :-1 s=READY pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 msgr2=0x7f489c1a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f488c04d8a0 tx=0x7f488c04f9c0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 shutdown_connections 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f489c0b8d10 0x7f489c0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f489c0b82f0 0x7f489c0b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f487c0186c0 0x7f487c01abc0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 --2- 172.21.2.117:0/1920343127 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f489c1a33d0 0x7f489c1a37d0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 >> 172.21.2.117:0/1920343127 conn(0x7f489c0874c0 msgr2=0x7f489c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 shutdown_connections 2021-02-10T11:49:13.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:13.712 7f48a246d700 1 -- 172.21.2.117:0/1920343127 wait complete. 2021-02-10T11:49:13.716 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.135 7fe8789a6700 1 Processor -- start 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.135 7fe8789a6700 1 -- start start 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe8789a6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe8789a6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 0x7fe8740eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe8789a6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740eba60 con 0x7fe8740ee3e0 2021-02-10T11:49:14.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe8789a6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740d1cf0 con 0x7fe8740d18f0 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe8789a6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740eefd0 con 0x7fe8740eb040 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe87261c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe87261c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:58124/0 (socket says 172.21.2.117:58124) 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe87261c700 1 -- 172.21.2.117:0/3324067233 learned_addr learned my addr 172.21.2.117:0/3324067233 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.136 7fe871e1b700 1 --2- 172.21.2.117:0/3324067233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 0x7fe8740eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe871e1b700 1 -- 172.21.2.117:0/3324067233 >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 msgr2=0x7fe8740eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe871e1b700 1 --2- 172.21.2.117:0/3324067233 >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe871e1b700 1 -- 172.21.2.117:0/3324067233 >> v1:172.21.2.125:6789/0 conn(0x7fe8740ee3e0 legacy=0x7fe8740ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe871e1b700 1 -- 172.21.2.117:0/3324067233 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740ef1f0 con 0x7fe8740eb040 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe87261c700 1 --2- 172.21.2.117:0/3324067233 >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe871e1b700 1 --2- 172.21.2.117:0/3324067233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 0x7fe8740eb440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fe868006650 tx=0x7fe868009a80).ready entity=mon.2 client_cookie=3134d28b4102f499 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe87161a700 1 -- 172.21.2.117:0/3324067233 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe86800c0c0 con 0x7fe8740eb040 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe87161a700 1 -- 172.21.2.117:0/3324067233 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86800c340 con 0x7fe8740eb040 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe87161a700 1 -- 172.21.2.117:0/3324067233 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe86800c860 con 0x7fe8740eb040 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe8789a6700 1 -- 172.21.2.117:0/3324067233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 msgr2=0x7fe8740eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.137 7fe8789a6700 1 --2- 172.21.2.117:0/3324067233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 0x7fe8740eb440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fe868006650 tx=0x7fe868009a80).stop 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 -- 172.21.2.117:0/3324067233 shutdown_connections 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 --2- 172.21.2.117:0/3324067233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8740eb040 0x7fe8740eb440 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 --2- 172.21.2.117:0/3324067233 >> v2:172.21.2.125:3300/0 conn(0x7fe8740d18f0 0x7fe8740eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 -- 172.21.2.117:0/3324067233 >> 172.21.2.117:0/3324067233 conn(0x7fe8740cf060 msgr2=0x7fe8740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 -- 172.21.2.117:0/3324067233 shutdown_connections 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.138 7fe8789a6700 1 -- 172.21.2.117:0/3324067233 wait complete. 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 Processor -- start 2021-02-10T11:49:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 -- start start 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe874085ec0 0x7fe8740862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8740a2040 0x7fe8740a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740a67e0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740a7120 con 0x7fe8740a2040 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe8789a6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740a2a60 con 0x7fe874085ec0 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe87261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe87261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53744/0 (socket says 172.21.2.117:53744) 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe87261c700 1 -- 172.21.2.117:0/1871052905 learned_addr learned my addr 172.21.2.117:0/1871052905 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe872e1d700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8740a2040 0x7fe8740a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.153 7fe871e1b700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe874085ec0 0x7fe8740862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 -- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe874085ec0 msgr2=0x7fe8740862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe874085ec0 0x7fe8740862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 -- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8740a2040 msgr2=0x7fe8740a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8740a2040 0x7fe8740a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 -- 172.21.2.117:0/1871052905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8741a2dc0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe87261c700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7fe864006ee0 tx=0x7fe86400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe86400bcd0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8640052e0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8741a2dc0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8741a3350 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.154 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8640057c0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.155 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8640054c0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.155 7fe862ffd700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 0x7fe85801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.155 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe8640328d0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.156 7fe871e1b700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 0x7fe85801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.156 7fe871e1b700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 0x7fe85801ac10 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:14.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.185 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740daff0 con 0x7fe8740a6ce0 2021-02-10T11:49:14.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.188 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe864007f00 con 0x7fe8740a6ce0 2021-02-10T11:49:14.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.337 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe8740ede60 con 0x7fe8740a6ce0 2021-02-10T11:49:14.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.341 7fe862ffd700 1 -- 172.21.2.117:0/1871052905 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe864000990 con 0x7fe8740a6ce0 2021-02-10T11:49:14.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.345 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 msgr2=0x7fe85801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.345 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 0x7fe85801ac10 crc :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.345 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 msgr2=0x7fe8740dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.345 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7fe864006ee0 tx=0x7fe86400b8e0).stop 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 shutdown_connections 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8740a2040 0x7fe8740a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe874085ec0 0x7fe8740862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe858018730 0x7fe85801ac10 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 --2- 172.21.2.117:0/1871052905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8740a6ce0 0x7fe8740dc420 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 >> 172.21.2.117:0/1871052905 conn(0x7fe8740b6540 msgr2=0x7fe87408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 shutdown_connections 2021-02-10T11:49:14.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:14.346 7fe8789a6700 1 -- 172.21.2.117:0/1871052905 wait complete. 2021-02-10T11:49:14.350 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:49:14.350 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:14.695 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:15.039 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:49:15.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:15.039 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:15.039 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:15.375 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:49:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:15.376 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:15.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:15.376 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:15.376 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:15.709 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:49:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:15.710 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:49:15.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:15.710 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:15.710 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:49:16.045 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:49:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:16.045 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:49:16.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:16.046 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:16.046 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:49:16.381 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:49:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:49:16.381 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:49:16.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:49:16.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.795 7f8cf19a3700 1 Processor -- start 2021-02-10T11:49:16.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.795 7f8cf19a3700 1 -- start start 2021-02-10T11:49:16.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cf19a3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8cec0e5240 0x7f8cec086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cf19a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cec084c60 0x7f8cec085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cf19a3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec085930 con 0x7f8cec0e4e40 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cf19a3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0e5cc0 con 0x7f8cec084c60 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cf19a3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0856b0 con 0x7f8cec0e5240 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cec084c60 0x7f8cec085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cec084c60 0x7f8cec085060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53756/0 (socket says 172.21.2.117:53756) 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 -- 172.21.2.117:0/872663849 learned_addr learned my addr 172.21.2.117:0/872663849 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:16.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 -- 172.21.2.117:0/872663849 >> v2:172.21.2.125:3300/0 conn(0x7f8cec0e5240 msgr2=0x7f8cec086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:16.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 --2- 172.21.2.117:0/872663849 >> v2:172.21.2.125:3300/0 conn(0x7f8cec0e5240 0x7f8cec086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:16.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 -- 172.21.2.117:0/872663849 >> v1:172.21.2.117:6789/0 conn(0x7f8cec0e4e40 legacy=0x7f8cec1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:16.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.796 7f8cebd75700 1 -- 172.21.2.117:0/872663849 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec1b09e0 con 0x7f8cec084c60 2021-02-10T11:49:16.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cebd75700 1 --2- 172.21.2.117:0/872663849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec084c60 0x7f8cec085060 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f8ce4006f00 tx=0x7f8ce4009f10).ready entity=mon.0 client_cookie=4e222c5b1294ba0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:16.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cea572700 1 -- 172.21.2.117:0/872663849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce400b3b0 con 0x7f8cec084c60 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cea572700 1 -- 172.21.2.117:0/872663849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce400b630 con 0x7f8cec084c60 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cea572700 1 -- 172.21.2.117:0/872663849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ce400bb50 con 0x7f8cec084c60 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cf19a3700 1 -- 172.21.2.117:0/872663849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec084c60 msgr2=0x7f8cec085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.797 7f8cf19a3700 1 --2- 172.21.2.117:0/872663849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec084c60 0x7f8cec085060 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f8ce4006f00 tx=0x7f8ce4009f10).stop 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 -- 172.21.2.117:0/872663849 shutdown_connections 2021-02-10T11:49:16.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 --2- 172.21.2.117:0/872663849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec084c60 0x7f8cec085060 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:16.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 --2- 172.21.2.117:0/872663849 >> v2:172.21.2.125:3300/0 conn(0x7f8cec0e5240 0x7f8cec086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:16.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 -- 172.21.2.117:0/872663849 >> 172.21.2.117:0/872663849 conn(0x7f8cec0ce4e0 msgr2=0x7f8cec0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:16.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 -- 172.21.2.117:0/872663849 shutdown_connections 2021-02-10T11:49:16.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.798 7f8cf19a3700 1 -- 172.21.2.117:0/872663849 wait complete. 2021-02-10T11:49:16.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.812 7f8cf19a3700 1 Processor -- start 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.812 7f8cf19a3700 1 -- start start 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cec0b0700 0x7f8cec18c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cec0985b0 0x7f8cec095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec097a10 con 0x7f8cec190f20 2021-02-10T11:49:16.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec18c720 con 0x7f8cec0985b0 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cf19a3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0989b0 con 0x7f8cec0b0700 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cead73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cead73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53760/0 (socket says 172.21.2.117:53760) 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cead73700 1 -- 172.21.2.117:0/505838929 learned_addr learned my addr 172.21.2.117:0/505838929 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8cebd75700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cec0985b0 0x7f8cec095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:16.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.813 7f8ceb574700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cec0b0700 0x7f8cec18c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 -- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cec0b0700 msgr2=0x7f8cec18c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cec0b0700 0x7f8cec18c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 -- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cec0985b0 msgr2=0x7f8cec095fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cec0985b0 0x7f8cec095fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 -- 172.21.2.117:0/505838929 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0965f0 con 0x7f8cec190f20 2021-02-10T11:49:16.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cead73700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f8cdc006f00 tx=0x7f8cdc00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cdc006350 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc006b90 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cdc000c90 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec0965f0 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.814 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec0965f0 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.815 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cdc005310 con 0x7f8cec190f20 2021-02-10T11:49:16.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.815 7f8ce3fff700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 0x7f8cd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:16.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.815 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8cdc0324f0 con 0x7f8cec190f20 2021-02-10T11:49:16.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.816 7f8ceb574700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 0x7f8cd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:16.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.816 7f8ceb574700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 0x7f8cd001abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:16.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.846 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec0989b0 con 0x7f8cec190f20 2021-02-10T11:49:16.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.850 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cdc02dfe0 con 0x7f8cec190f20 2021-02-10T11:49:16.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.970 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8cec0989b0 con 0x7f8cec190f20 2021-02-10T11:49:16.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.971 7f8ce3fff700 1 -- 172.21.2.117:0/505838929 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v100) v1 ==== 81+0+30757 (secure 0 0 0) 0x7f8cdc02dfe0 con 0x7f8cec190f20 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 100, 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:48:39.057620", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:49:16.968 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 21, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:49:16.969 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:49:16.970 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:49:16.971 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:49:16.972 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:49:16.973 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:49:16.974 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:49:16.975 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 20, 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:49:16.976 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:48:35.729626", 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:49:16.977 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "99", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:49:16.978 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:49:16.979 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:49:16.980 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:49:16.981 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 21, 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:48:37.102743", 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:49:16.982 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:49:16.983 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "100", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:49:16.984 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:49:16.985 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:49:16.986 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:49:16.987 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 96, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.988 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.989 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:16.990 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.991 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.992 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:16.993 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.994 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.997 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:49:16.998 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:16.999 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.004 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.005 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.006 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:49:17.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.010 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 89, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.011 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.013 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.014 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.015 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:49:17.016 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.017 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:49:17.021 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:17.022 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:49:17.023 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:49:17.024 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.026 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:49:17.027 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:49:17.028 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.030 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.031 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:49:17.033 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.035 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.036 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:49:17.038 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:49:17.039 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:49:17.040 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:49:17.041 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:17.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.972 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 msgr2=0x7f8cd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.972 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 0x7f8cd001abc0 crc :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.972 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 msgr2=0x7f8cec097f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.972 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f8cdc006f00 tx=0x7f8cdc00b940).stop 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 shutdown_connections 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8cd00186c0 0x7f8cd001abc0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cec190f20 0x7f8cec097f90 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cec0985b0 0x7f8cec095fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 --2- 172.21.2.117:0/505838929 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cec0b0700 0x7f8cec18c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 >> 172.21.2.117:0/505838929 conn(0x7f8cec199160 msgr2=0x7f8cec08d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 shutdown_connections 2021-02-10T11:49:17.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:16.973 7f8cf19a3700 1 -- 172.21.2.117:0/505838929 wait complete. 2021-02-10T11:49:17.044 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:17.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 Processor -- start 2021-02-10T11:49:17.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 -- start start 2021-02-10T11:49:17.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78780e5240 0x7f7878086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:17.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7878085930 con 0x7f7878084c60 2021-02-10T11:49:17.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78781b09e0 con 0x7f78780e4e40 2021-02-10T11:49:17.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f7880373700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78780e5cc0 con 0x7f78780e5240 2021-02-10T11:49:17.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f787e990700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7878084c60 0x7f7878085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54020/0 (socket says 172.21.2.117:54020) 2021-02-10T11:49:17.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f787e990700 1 -- 172.21.2.117:0/2735477566 learned_addr learned my addr 172.21.2.117:0/2735477566 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:17.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.459 7f787d98e700 1 --2- 172.21.2.117:0/2735477566 >> v2:172.21.2.117:3300/0 conn(0x7f78780e5240 0x7f7878086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d18d700 1 -- 172.21.2.117:0/2735477566 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504552854 0 0) 0x7f7870001500 con 0x7f7878084c60 2021-02-10T11:49:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d18d700 1 -- 172.21.2.117:0/2735477566 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7864002fe0 con 0x7f7878084c60 2021-02-10T11:49:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d98e700 1 -- 172.21.2.117:0/2735477566 >> v1:172.21.2.125:6789/0 conn(0x7f78780e4e40 legacy=0x7f78781b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d98e700 1 -- 172.21.2.117:0/2735477566 >> v1:172.21.2.117:6789/0 conn(0x7f7878084c60 legacy=0x7f7878085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d98e700 1 -- 172.21.2.117:0/2735477566 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878084840 con 0x7f78780e5240 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d98e700 1 --2- 172.21.2.117:0/2735477566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780e5240 0x7f7878086ec0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f786c006650 tx=0x7f786c009a80).ready entity=mon.0 client_cookie=7f9639f541172c95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d18d700 1 -- 172.21.2.117:0/2735477566 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f786c00c0c0 con 0x7f78780e5240 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d18d700 1 -- 172.21.2.117:0/2735477566 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c00c340 con 0x7f78780e5240 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f787d18d700 1 -- 172.21.2.117:0/2735477566 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f786c00c860 con 0x7f78780e5240 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f7880373700 1 -- 172.21.2.117:0/2735477566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780e5240 msgr2=0x7f7878086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.460 7f7880373700 1 --2- 172.21.2.117:0/2735477566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780e5240 0x7f7878086ec0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f786c006650 tx=0x7f786c009a80).stop 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.461 7f7880373700 1 -- 172.21.2.117:0/2735477566 shutdown_connections 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.461 7f7880373700 1 --2- 172.21.2.117:0/2735477566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780e5240 0x7f7878086ec0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.461 7f7880373700 1 -- 172.21.2.117:0/2735477566 >> 172.21.2.117:0/2735477566 conn(0x7f78780ce4e0 msgr2=0x7f78780e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.461 7f7880373700 1 -- 172.21.2.117:0/2735477566 shutdown_connections 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.461 7f7880373700 1 -- 172.21.2.117:0/2735477566 wait complete. 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 Processor -- start 2021-02-10T11:49:17.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 -- start start 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7878190f20 0x7f7878097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78780985b0 0x7f7878095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7878097a10 con 0x7f78780b0700 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787818c720 con 0x7f7878190f20 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f7880373700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780989b0 con 0x7f78780985b0 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.476 7f787e18f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53774/0 (socket says 172.21.2.117:53774) 2021-02-10T11:49:17.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 -- 172.21.2.117:0/2750143661 learned_addr learned my addr 172.21.2.117:0/2750143661 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 -- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78780985b0 msgr2=0x7f7878095fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78780985b0 0x7f7878095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 -- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7878190f20 msgr2=0x7f7878097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7878190f20 0x7f7878097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:17.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 -- 172.21.2.117:0/2750143661 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78780965f0 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f787e18f700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f787400b5b0 tx=0x7f787400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f787400bd40 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7874000e20 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f7880373700 1 -- 172.21.2.117:0/2750143661 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78780965f0 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7874006210 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.477 7f7880373700 1 -- 172.21.2.117:0/2750143661 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7878096b80 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.478 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7874006020 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.478 7f786affd700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 0x7f786001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.478 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f78740327f0 con 0x7f78780b0700 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.478 7f787d98e700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 0x7f786001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:17.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.479 7f787d98e700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 0x7f786001ac10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.509 7f7880373700 1 -- 172.21.2.117:0/2750143661 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787810c250 con 0x7f78780b0700 2021-02-10T11:49:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.511 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7874000940 con 0x7f78780b0700 2021-02-10T11:49:17.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.640 7f7880373700 1 -- 172.21.2.117:0/2750143661 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7878085930 con 0x7f78780b0700 2021-02-10T11:49:17.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.640 7f786affd700 1 -- 172.21.2.117:0/2750143661 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f787400bd40 con 0x7f78780b0700 2021-02-10T11:49:17.636 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 78 2021-02-10T11:49:17.636 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:17.636 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":78,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:48:40.061715","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6686":{"gid":6686,"name":"b","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1898190750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1898190750},{"type":"v1","addr":"172.21.2.125:6835","nonce":1898190750}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 msgr2=0x7f786001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 0x7f786001ac10 crc :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 msgr2=0x7f787818c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f787400b5b0 tx=0x7f787400b950).stop 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 shutdown_connections 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78780985b0 0x7f7878095fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7878190f20 0x7f7878097f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7860018730 0x7f786001ac10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 --2- 172.21.2.117:0/2750143661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78780b0700 0x7f787818c100 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:17.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 >> 172.21.2.117:0/2750143661 conn(0x7f7878199160 msgr2=0x7f787808d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:17.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 shutdown_connections 2021-02-10T11:49:17.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:17.642 7f7880373700 1 -- 172.21.2.117:0/2750143661 wait complete. 2021-02-10T11:49:17.647 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:49:18.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.063 7fdc7403b700 1 Processor -- start 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.063 7fdc7403b700 1 -- start start 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc7403b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdc6c1b07f0 0x7fdc6c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc7403b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0e77f0 con 0x7fdc6c0e8d80 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc7403b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0e7550 con 0x7fdc6c0e6170 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc7403b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0e54b0 con 0x7fdc6c1b07f0 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc72658700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdc6c0e8d80 0x7fdc6c0e6d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54034/0 (socket says 172.21.2.117:54034) 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc71656700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fdc6c0e6170 0x7fdc6c0e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:46958/0 (socket says 172.21.2.117:46958) 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc72658700 1 -- 172.21.2.117:0/4122021649 learned_addr learned my addr 172.21.2.117:0/4122021649 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.064 7fdc71e57700 1 --2- 172.21.2.117:0/4122021649 >> v2:172.21.2.125:3300/0 conn(0x7fdc6c1b07f0 0x7fdc6c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc70e55700 1 -- 172.21.2.117:0/4122021649 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964518455 0 0) 0x7fdc68001100 con 0x7fdc6c0e8d80 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc70e55700 1 -- 172.21.2.117:0/4122021649 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc58002fe0 con 0x7fdc6c0e8d80 2021-02-10T11:49:18.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc71e57700 1 -- 172.21.2.117:0/4122021649 >> v1:172.21.2.125:6789/0 conn(0x7fdc6c0e6170 legacy=0x7fdc6c0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc71e57700 1 -- 172.21.2.117:0/4122021649 >> v1:172.21.2.117:6789/0 conn(0x7fdc6c0e8d80 legacy=0x7fdc6c0e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc71e57700 1 -- 172.21.2.117:0/4122021649 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0e9180 con 0x7fdc6c1b07f0 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc71e57700 1 --2- 172.21.2.117:0/4122021649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c1b07f0 0x7fdc6c0e4e90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7fdc640069d0 tx=0x7fdc64009e40).ready entity=mon.1 client_cookie=82173eff7ff74557 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc70e55700 1 -- 172.21.2.117:0/4122021649 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc6400b510 con 0x7fdc6c1b07f0 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc70e55700 1 -- 172.21.2.117:0/4122021649 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6400b890 con 0x7fdc6c1b07f0 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc70e55700 1 -- 172.21.2.117:0/4122021649 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc6400bd70 con 0x7fdc6c1b07f0 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc7403b700 1 -- 172.21.2.117:0/4122021649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c1b07f0 msgr2=0x7fdc6c0e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.065 7fdc7403b700 1 --2- 172.21.2.117:0/4122021649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c1b07f0 0x7fdc6c0e4e90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7fdc640069d0 tx=0x7fdc64009e40).stop 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.066 7fdc7403b700 1 -- 172.21.2.117:0/4122021649 shutdown_connections 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.066 7fdc7403b700 1 --2- 172.21.2.117:0/4122021649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c1b07f0 0x7fdc6c0e4e90 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.066 7fdc7403b700 1 -- 172.21.2.117:0/4122021649 >> 172.21.2.117:0/4122021649 conn(0x7fdc6c0ce500 msgr2=0x7fdc6c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.066 7fdc7403b700 1 -- 172.21.2.117:0/4122021649 shutdown_connections 2021-02-10T11:49:18.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.066 7fdc7403b700 1 -- 172.21.2.117:0/4122021649 wait complete. 2021-02-10T11:49:18.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 Processor -- start 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 -- start start 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 0x7fdc6c199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 0x7fdc6c0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c105f50 con 0x7fdc6c199a70 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c08f300 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc7403b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0c3f80 con 0x7fdc6c08ef00 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc71656700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 0x7fdc6c0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc72658700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc72658700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:58170/0 (socket says 172.21.2.117:58170) 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc72658700 1 -- 172.21.2.117:0/209760663 learned_addr learned my addr 172.21.2.117:0/209760663 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:18.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.081 7fdc71656700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 0x7fdc6c0b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53788/0 (socket says 172.21.2.117:53788) 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc71e57700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 0x7fdc6c199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 -- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 msgr2=0x7fdc6c199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 0x7fdc6c199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 -- 172.21.2.117:0/209760663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 msgr2=0x7fdc6c0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 0x7fdc6c0b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 -- 172.21.2.117:0/209760663 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0b2510 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc71e57700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 0x7fdc6c199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc72658700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fdc6800b5c0 tx=0x7fdc6800b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc68006b80 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6800bd50 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc680062c0 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc7403b700 1 -- 172.21.2.117:0/209760663 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c0b2510 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.082 7fdc7403b700 1 -- 172.21.2.117:0/209760663 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c0b2510 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.083 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc68007000 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.083 7fdc627fc700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 0x7fdc5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:18.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.083 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdc680326d0 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.083 7fdc71e57700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 0x7fdc5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:18.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.084 7fdc71e57700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 0x7fdc5401abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.114 7fdc7403b700 1 -- 172.21.2.117:0/209760663 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c1b8070 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.117 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc6802dee0 con 0x7fdc6c0b1af0 2021-02-10T11:49:18.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:18.249 7fdc7403b700 1 -- 172.21.2.117:0/209760663 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdc6c1b8070 con 0x7fdc6c0b1af0 2021-02-10T11:49:19.026 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:49:19.031 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:49:19.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.035 7fdc627fc700 1 -- 172.21.2.117:0/209760663 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdc6802dee0 con 0x7fdc6c0b1af0 2021-02-10T11:49:19.064 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:49:19.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 -- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 msgr2=0x7fdc5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 0x7fdc5401abc0 crc :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 -- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 msgr2=0x7fdc6c0b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fdc6800b5c0 tx=0x7fdc6800b960).stop 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 -- 172.21.2.117:0/209760663 shutdown_connections 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc540186c0 0x7fdc5401abc0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c199a70 0x7fdc6c0b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c0b1af0 0x7fdc6c0b1ef0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 --2- 172.21.2.117:0/209760663 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c08ef00 0x7fdc6c199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 -- 172.21.2.117:0/209760663 >> 172.21.2.117:0/209760663 conn(0x7fdc6c0b97b0 msgr2=0x7fdc6c098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.071 7fdc7403b700 1 -- 172.21.2.117:0/209760663 shutdown_connections 2021-02-10T11:49:19.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.072 7fdc7403b700 1 -- 172.21.2.117:0/209760663 wait complete. 2021-02-10T11:49:19.075 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:19.082 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:49:19.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.490 7fd2228b0700 1 Processor -- start 2021-02-10T11:49:19.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.490 7fd2228b0700 1 -- start start 2021-02-10T11:49:19.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd2228b0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd21c0d18e0 0x7fd21c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd2228b0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c0ede60 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd2228b0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21c0d1ce0 con 0x7fd21c0eb000 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd2228b0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd21c0ef7f0 con 0x7fd21c0d18e0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd220ecd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd21c0ef3f0 0x7fd21c1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54046/0 (socket says 172.21.2.117:54046) 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd220ecd700 1 -- 172.21.2.117:0/1109600694 learned_addr learned my addr 172.21.2.117:0/1109600694 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.491 7fd21bfff700 1 --2- 172.21.2.117:0/1109600694 >> v2:172.21.2.125:3300/0 conn(0x7fd21c0d18e0 0x7fd21c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971425738 0 0) 0x7fd214001500 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd204002fe0 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2178287272 0 0) 0x7fd214001500 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 >> v2:172.21.2.125:3300/0 conn(0x7fd21c0d18e0 msgr2=0x7fd21c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 --2- 172.21.2.117:0/1109600694 >> v2:172.21.2.125:3300/0 conn(0x7fd21c0d18e0 0x7fd21c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 >> v1:172.21.2.125:6789/0 conn(0x7fd21c0eb000 legacy=0x7fd21c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c1bb180 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd214001a00 con 0x7fd21c0ef3f0 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 >> v1:172.21.2.117:6789/0 conn(0x7fd21c0ef3f0 legacy=0x7fd21c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 0x7fd204005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd204006790 0x7fd204008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd204009590 0x7fd20400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20400c300 con 0x7fd204003ac0 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20400c520 con 0x7fd204009590 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21affd700 1 -- 172.21.2.117:0/1109600694 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd204050750 con 0x7fd204006790 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 0x7fd204005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd220ecd700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd204009590 0x7fd20400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21b7fe700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd204006790 0x7fd204008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 -- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd204006790 msgr2=0x7fd204008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd204006790 0x7fd204008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 -- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd204009590 msgr2=0x7fd20400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd204009590 0x7fd20400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21bfff700 1 -- 172.21.2.117:0/1109600694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040509b0 con 0x7fd204003ac0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd21b7fe700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd204006790 0x7fd204008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.492 7fd220ecd700 1 -- 172.21.2.117:0/1109600694 reap_dead start 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd21bfff700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 0x7fd204005f10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7fd20c00c8f0 tx=0x7fd20c00cc90).ready entity=mon.0 client_cookie=56be384c69cd903 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c009310 con 0x7fd204003ac0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c0095d0 con 0x7fd204003ac0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd21affd700 1 -- 172.21.2.117:0/1109600694 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c009a30 con 0x7fd204003ac0 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 -- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 msgr2=0x7fd204005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 0x7fd204005f10 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7fd20c00c8f0 tx=0x7fd20c00cc90).stop 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 -- 172.21.2.117:0/1109600694 shutdown_connections 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 --2- 172.21.2.117:0/1109600694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd204003ac0 0x7fd204005f10 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 -- 172.21.2.117:0/1109600694 >> 172.21.2.117:0/1109600694 conn(0x7fd21c0ced60 msgr2=0x7fd21c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 -- 172.21.2.117:0/1109600694 shutdown_connections 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.493 7fd2228b0700 1 -- 172.21.2.117:0/1109600694 wait complete. 2021-02-10T11:49:19.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.508 7fd2228b0700 1 Processor -- start 2021-02-10T11:49:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.508 7fd2228b0700 1 -- start start 2021-02-10T11:49:19.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd21c099a10 0x7fd21c1a7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd21c1a7c30 0x7fd21c09a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c099e10 con 0x7fd21c1a7830 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c09aee0 con 0x7fd21c1a7c30 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd2228b0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c09b100 con 0x7fd21c099a10 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53806/0 (socket says 172.21.2.117:53806) 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 -- 172.21.2.117:0/2833020481 learned_addr learned my addr 172.21.2.117:0/2833020481 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 -- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd21c099a10 msgr2=0x7fd21c1a7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd21c099a10 0x7fd21c1a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 -- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd21c1a7c30 msgr2=0x7fd21c09a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd21c1a7c30 0x7fd21c09a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 -- 172.21.2.117:0/2833020481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c09b320 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.509 7fd21b7fe700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7fd210006f20 tx=0x7fd21000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd210006870 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21000bd80 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21c09b320 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21c0e50f0 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd210002220 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2100054b0 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 0x7fd1fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.510 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fd210032780 con 0x7fd21c1a7830 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.511 7fd21bfff700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 0x7fd1fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:19.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.511 7fd21bfff700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 0x7fd1fc01abc0 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.541 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21c097a50 con 0x7fd21c1a7830 2021-02-10T11:49:19.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.543 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd210006870 con 0x7fd21c1a7830 2021-02-10T11:49:19.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.670 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd21c097a50 con 0x7fd21c1a7830 2021-02-10T11:49:19.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.670 7fd218ff9700 1 -- 172.21.2.117:0/2833020481 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1760 (secure 0 0 0) 0x7fd21000bd80 con 0x7fd21c1a7830 2021-02-10T11:49:19.666 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 79 2021-02-10T11:49:19.666 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:19.667 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":79,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1585144044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1585144044},{"type":"v1","addr":"172.21.2.117:6833","nonce":1585144044}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":73}],"filesystems":[{"mdsmap":{"epoch":79,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:48:38.055110","modified":"2021-02-10 11:49:19.031694","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 msgr2=0x7fd1fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 0x7fd1fc01abc0 crc :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 msgr2=0x7fd21c09a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7fd210006f20 tx=0x7fd21000b990).stop 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 shutdown_connections 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1fc0186c0 0x7fd1fc01abc0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd21c1a7830 0x7fd21c09a2a0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd21c1a7c30 0x7fd21c09a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 --2- 172.21.2.117:0/2833020481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd21c099a10 0x7fd21c1a7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.672 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 >> 172.21.2.117:0/2833020481 conn(0x7fd21c0e2eb0 msgr2=0x7fd21c0982a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.673 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 shutdown_connections 2021-02-10T11:49:19.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:19.673 7fd2228b0700 1 -- 172.21.2.117:0/2833020481 wait complete. 2021-02-10T11:49:19.676 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:49:20.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.097 7f728bef5700 1 Processor -- start 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.097 7f728bef5700 1 -- start start 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72841b07f0 0x7f72840e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f72840e6170 0x7f72840e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72840e6b90 0x7f72840e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72840e5ce0 con 0x7f72841b07f0 2021-02-10T11:49:20.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72840e75b0 con 0x7f72840e6170 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728bef5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72840e77f0 con 0x7f72840e6b90 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72841b07f0 0x7f72840e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72841b07f0 0x7f72840e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53814/0 (socket says 172.21.2.117:53814) 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 -- 172.21.2.117:0/3421628745 learned_addr learned my addr 172.21.2.117:0/3421628745 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289510700 1 --2- 172.21.2.117:0/3421628745 >> v2:172.21.2.125:3300/0 conn(0x7f72840e6170 0x7f72840e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f728a512700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72840e6b90 0x7f72840e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 -- 172.21.2.117:0/3421628745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72840e6b90 msgr2=0x7f72840e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72840e6b90 0x7f72840e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 -- 172.21.2.117:0/3421628745 >> v2:172.21.2.125:3300/0 conn(0x7f72840e6170 msgr2=0x7f72840e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 --2- 172.21.2.117:0/3421628745 >> v2:172.21.2.125:3300/0 conn(0x7f72840e6170 0x7f72840e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 -- 172.21.2.117:0/3421628745 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72840e54b0 con 0x7f72841b07f0 2021-02-10T11:49:20.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.098 7f7289d11700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72841b07f0 0x7f72840e4e90 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f7274006700 tx=0x7f7274009b80).ready entity=mon.0 client_cookie=40e3bbdf4af8b49e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f7288d0f700 1 -- 172.21.2.117:0/3421628745 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727400c0c0 con 0x7f72841b07f0 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f7288d0f700 1 -- 172.21.2.117:0/3421628745 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727400c340 con 0x7f72841b07f0 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f7288d0f700 1 -- 172.21.2.117:0/3421628745 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727400c860 con 0x7f72841b07f0 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 -- 172.21.2.117:0/3421628745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72841b07f0 msgr2=0x7f72840e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72841b07f0 0x7f72840e4e90 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f7274006700 tx=0x7f7274009b80).stop 2021-02-10T11:49:20.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 -- 172.21.2.117:0/3421628745 shutdown_connections 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 --2- 172.21.2.117:0/3421628745 >> v2:172.21.2.125:3300/0 conn(0x7f72840e6170 0x7f72840e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72840e6b90 0x7f72840e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 --2- 172.21.2.117:0/3421628745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72841b07f0 0x7f72840e4e90 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.099 7f728bef5700 1 -- 172.21.2.117:0/3421628745 >> 172.21.2.117:0/3421628745 conn(0x7f72840ce500 msgr2=0x7f72840e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.100 7f728bef5700 1 -- 172.21.2.117:0/3421628745 shutdown_connections 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.100 7f728bef5700 1 -- 172.21.2.117:0/3421628745 wait complete. 2021-02-10T11:49:20.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.114 7f728bef5700 1 Processor -- start 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.114 7f728bef5700 1 -- start start 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72840966e0 0x7f728409a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 0x7f728409b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 0x7f7284187d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728409b1a0 con 0x7f7284110ae0 2021-02-10T11:49:20.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7284096ae0 con 0x7f72840966e0 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728bef5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728409bd40 con 0x7f7284110ee0 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289d11700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72840966e0 0x7f728409a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728a512700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 0x7f7284187d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 0x7f728409b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728a512700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 0x7f7284187d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51502/0 (socket says 172.21.2.117:51502) 2021-02-10T11:49:20.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f728a512700 1 -- 172.21.2.117:0/2738236854 learned_addr learned my addr 172.21.2.117:0/2738236854 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:20.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 -- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 msgr2=0x7f7284187d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 0x7f7284187d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 -- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72840966e0 msgr2=0x7f728409a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:20.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72840966e0 0x7f728409a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:20.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 -- 172.21.2.117:0/2738236854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72841a28c0 con 0x7f7284110ae0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.115 7f7289510700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 0x7f728409b720 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f727c009920 tx=0x7f727c00bc10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.116 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727c006240 con 0x7f7284110ae0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.116 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c005660 con 0x7f7284110ae0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.116 7f728bef5700 1 -- 172.21.2.117:0/2738236854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72841a28c0 con 0x7f7284110ae0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.116 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f727c005b80 con 0x7f7284110ae0 2021-02-10T11:49:20.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.116 7f728bef5700 1 -- 172.21.2.117:0/2738236854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72841a28c0 con 0x7f7284110ae0 2021-02-10T11:49:20.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.117 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f727c005a50 con 0x7f7284110ae0 2021-02-10T11:49:20.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.117 7f727a7fc700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 0x7f726c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:20.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.117 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f727c032c10 con 0x7f7284110ae0 2021-02-10T11:49:20.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.117 7f7289d11700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 0x7f726c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:20.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.117 7f7289d11700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 0x7f726c01abd0 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:20.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.147 7f728bef5700 1 -- 172.21.2.117:0/2738236854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728410c2c0 con 0x7f7284110ae0 2021-02-10T11:49:20.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.150 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f727c005660 con 0x7f7284110ae0 2021-02-10T11:49:20.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:20.277 7f728bef5700 1 -- 172.21.2.117:0/2738236854 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f72840e77f0 con 0x7f7284110ae0 2021-02-10T11:49:21.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.046 7f727a7fc700 1 -- 172.21.2.117:0/2738236854 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v80) v1 ==== 104+0+0 (secure 0 0 0) 0x7f727c005870 con 0x7f7284110ae0 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.048 7f728bef5700 1 -- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 msgr2=0x7f726c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 0x7f726c01abd0 crc :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 -- 172.21.2.117:0/2738236854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 msgr2=0x7f728409b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 0x7f728409b720 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f727c009920 tx=0x7f727c00bc10).stop 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 -- 172.21.2.117:0/2738236854 shutdown_connections 2021-02-10T11:49:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f726c0186d0 0x7f726c01abd0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7284110ae0 0x7f728409b720 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7284110ee0 0x7f7284187d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 --2- 172.21.2.117:0/2738236854 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72840966e0 0x7f728409a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 -- 172.21.2.117:0/2738236854 >> 172.21.2.117:0/2738236854 conn(0x7f7284097720 msgr2=0x7f72841994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 -- 172.21.2.117:0/2738236854 shutdown_connections 2021-02-10T11:49:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.049 7f728bef5700 1 -- 172.21.2.117:0/2738236854 wait complete. 2021-02-10T11:49:21.053 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:49:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.473 7f6d0f533700 1 Processor -- start 2021-02-10T11:49:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.473 7f6d0f533700 1 -- start start 2021-02-10T11:49:21.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6d08085450 0x7f6d08086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d08085930 con 0x7f6d080e5240 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d081b09e0 con 0x7f6d080e4e40 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080e5cc0 con 0x7f6d08085450 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0db50700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6d08085450 0x7f6d08086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0cb4e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6d080e5240 0x7f6d08084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54078/0 (socket says 172.21.2.117:54078) 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0cb4e700 1 -- 172.21.2.117:0/1577070460 learned_addr learned my addr 172.21.2.117:0/1577070460 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0db50700 1 -- 172.21.2.117:0/1577070460 >> v1:172.21.2.125:6789/0 conn(0x7f6d080e4e40 legacy=0x7f6d081b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0db50700 1 -- 172.21.2.117:0/1577070460 >> v1:172.21.2.117:6789/0 conn(0x7f6d080e5240 legacy=0x7f6d08084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0db50700 1 -- 172.21.2.117:0/1577070460 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d08084840 con 0x7f6d08085450 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0db50700 1 --2- 172.21.2.117:0/1577070460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d08085450 0x7f6d08086ec0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f6d00006e30 tx=0x7f6d00009e60).ready entity=mon.0 client_cookie=6cbb2e8e36efd6d2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6cfbfff700 1 -- 172.21.2.117:0/1577070460 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d00006a50 con 0x7f6d08085450 2021-02-10T11:49:21.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6cfbfff700 1 -- 172.21.2.117:0/1577070460 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0000b300 con 0x7f6d08085450 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6cfbfff700 1 -- 172.21.2.117:0/1577070460 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d0000b820 con 0x7f6d08085450 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 -- 172.21.2.117:0/1577070460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d08085450 msgr2=0x7f6d08086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.474 7f6d0f533700 1 --2- 172.21.2.117:0/1577070460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d08085450 0x7f6d08086ec0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f6d00006e30 tx=0x7f6d00009e60).stop 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.475 7f6d0f533700 1 -- 172.21.2.117:0/1577070460 shutdown_connections 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.475 7f6d0f533700 1 --2- 172.21.2.117:0/1577070460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d08085450 0x7f6d08086ec0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.475 7f6d0f533700 1 -- 172.21.2.117:0/1577070460 >> 172.21.2.117:0/1577070460 conn(0x7f6d080ce4e0 msgr2=0x7f6d080e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.475 7f6d0f533700 1 -- 172.21.2.117:0/1577070460 shutdown_connections 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.475 7f6d0f533700 1 -- 172.21.2.117:0/1577070460 wait complete. 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 Processor -- start 2021-02-10T11:49:21.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 -- start start 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d080ab010 0x7f6d080c0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d080ab410 0x7f6d080c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080ab9a0 con 0x7f6d081076a0 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08107aa0 con 0x7f6d080ab010 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.490 7f6d0f533700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0809f010 con 0x7f6d080ab410 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53836/0 (socket says 172.21.2.117:53836) 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 -- 172.21.2.117:0/3834876002 learned_addr learned my addr 172.21.2.117:0/3834876002 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:21.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0cb4e700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d080ab010 0x7f6d080c0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 -- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d080ab410 msgr2=0x7f6d080c10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d080ab410 0x7f6d080c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 -- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d080ab010 msgr2=0x7f6d080c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d080ab010 0x7f6d080c0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 -- 172.21.2.117:0/3834876002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080c16d0 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6d0d34f700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f6cfc00b5b0 tx=0x7f6cfc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.491 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cfc00bd40 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc006240 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6cfc005720 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d080c16d0 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080c16d0 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6cfc0054a0 con 0x7f6d081076a0 2021-02-10T11:49:21.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6cf9ffb700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 0x7f6ce801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:21.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.492 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f6cfc0329a0 con 0x7f6d081076a0 2021-02-10T11:49:21.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.493 7f6d0cb4e700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 0x7f6ce801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:21.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.493 7f6d0cb4e700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 0x7f6ce801abc0 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:21.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.523 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08106450 con 0x7f6d081076a0 2021-02-10T11:49:21.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.526 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6cfc006030 con 0x7f6d081076a0 2021-02-10T11:49:21.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:21.654 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d080e5eb0 con 0x7f6d081076a0 2021-02-10T11:49:22.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.052 7f6cf9ffb700 1 -- 172.21.2.117:0/3834876002 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 185+0+0 (secure 0 0 0) 0x7f6cfc0054a0 con 0x7f6d081076a0 2021-02-10T11:49:22.065 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:49:22.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 msgr2=0x7f6ce801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:22.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 0x7f6ce801abc0 crc :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 msgr2=0x7f6d08106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:22.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7f6cfc00b5b0 tx=0x7f6cfc00b950).stop 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 shutdown_connections 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d080ab410 0x7f6d080c10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d080ab010 0x7f6d080c0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ce8018730 0x7f6ce801abc0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 --2- 172.21.2.117:0/3834876002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d081076a0 0x7f6d08106f80 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 >> 172.21.2.117:0/3834876002 conn(0x7f6d0809fb10 msgr2=0x7f6d081109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 shutdown_connections 2021-02-10T11:49:22.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.072 7f6d0f533700 1 -- 172.21.2.117:0/3834876002 wait complete. 2021-02-10T11:49:22.076 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:49:22.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.494 7f9fb4b0a700 1 Processor -- start 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.494 7f9fb4b0a700 1 -- start start 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fb4b0a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9fb00e4e40 0x7f9fb01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fb4b0a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fb0085450 0x7f9fb0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fb4b0a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb0085930 con 0x7f9fb00e5240 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fb4b0a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb00e5cc0 con 0x7f9fb0085450 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fb4b0a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb0084840 con 0x7f9fb00e4e40 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fb0085450 0x7f9fb0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9fb0085450 0x7f9fb0086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53842/0 (socket says 172.21.2.117:53842) 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 -- 172.21.2.117:0/911739558 learned_addr learned my addr 172.21.2.117:0/911739558 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9fae85e700 1 --2- 172.21.2.117:0/911739558 >> v2:172.21.2.125:3300/0 conn(0x7f9fb00e4e40 0x7f9fb01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 -- 172.21.2.117:0/911739558 >> v2:172.21.2.125:3300/0 conn(0x7f9fb00e4e40 msgr2=0x7f9fb01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 --2- 172.21.2.117:0/911739558 >> v2:172.21.2.125:3300/0 conn(0x7f9fb00e4e40 0x7f9fb01b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 -- 172.21.2.117:0/911739558 >> v1:172.21.2.125:6789/0 conn(0x7f9fb00e5240 legacy=0x7f9fb0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.495 7f9faf05f700 1 -- 172.21.2.117:0/911739558 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb01b0810 con 0x7f9fb0085450 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9faf05f700 1 --2- 172.21.2.117:0/911739558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0085450 0x7f9fb0086ec0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7f9fa8006f00 tx=0x7f9fa8009e90).ready entity=mon.0 client_cookie=9337d9de22a3f6b9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9fad85c700 1 -- 172.21.2.117:0/911739558 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa800b300 con 0x7f9fb0085450 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9fad85c700 1 -- 172.21.2.117:0/911739558 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800b580 con 0x7f9fb0085450 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9fad85c700 1 -- 172.21.2.117:0/911739558 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa800baa0 con 0x7f9fb0085450 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9fb4b0a700 1 -- 172.21.2.117:0/911739558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0085450 msgr2=0x7f9fb0086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.496 7f9fb4b0a700 1 --2- 172.21.2.117:0/911739558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0085450 0x7f9fb0086ec0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7f9fa8006f00 tx=0x7f9fa8009e90).stop 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 -- 172.21.2.117:0/911739558 shutdown_connections 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 --2- 172.21.2.117:0/911739558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0085450 0x7f9fb0086ec0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 --2- 172.21.2.117:0/911739558 >> v2:172.21.2.125:3300/0 conn(0x7f9fb00e4e40 0x7f9fb01b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 -- 172.21.2.117:0/911739558 >> 172.21.2.117:0/911739558 conn(0x7f9fb00ce4e0 msgr2=0x7f9fb00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:22.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 -- 172.21.2.117:0/911739558 shutdown_connections 2021-02-10T11:49:22.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.497 7f9fb4b0a700 1 -- 172.21.2.117:0/911739558 wait complete. 2021-02-10T11:49:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 Processor -- start 2021-02-10T11:49:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 -- start start 2021-02-10T11:49:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 0x7f9fb0096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 0x7f9fb00977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb0195290 con 0x7f9fb0194e10 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00b1460 con 0x7f9fb00973f0 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fb4b0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb00b1680 con 0x7f9fb0191bf0 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fae85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fae85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53846/0 (socket says 172.21.2.117:53846) 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.512 7f9fae85e700 1 -- 172.21.2.117:0/1460502103 learned_addr learned my addr 172.21.2.117:0/1460502103 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae05d700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 0x7f9fb0096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9faf05f700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 0x7f9fb00977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 -- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 msgr2=0x7f9fb0096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 0x7f9fb0096dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 -- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 msgr2=0x7f9fb00977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:22.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 0x7f9fb00977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 -- 172.21.2.117:0/1460502103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb00b18a0 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae05d700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 0x7f9fb0096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9faf05f700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 0x7f9fb00977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fae85e700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f9fa0006f60 tx=0x7f9fa000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa000bd60 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa0000c90 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fa00022b0 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb00b18a0 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.513 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb00b1e30 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.514 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fa0006b00 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.514 7f9fa77fe700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 0x7f9f9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.514 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f9fa00323e0 con 0x7f9fb0194e10 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.514 7f9fae05d700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 0x7f9f9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:22.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.515 7f9fae05d700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 0x7f9f9401abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:22.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.545 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb0085930 con 0x7f9fb0194e10 2021-02-10T11:49:22.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.547 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fa0000940 con 0x7f9fb0194e10 2021-02-10T11:49:22.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:22.673 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fb0085930 con 0x7f9fb0194e10 2021-02-10T11:49:23.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.054 7f9fa77fe700 1 -- 172.21.2.117:0/1460502103 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9fa0000940 con 0x7f9fb0194e10 2021-02-10T11:49:23.086 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.092 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 msgr2=0x7f9f9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.092 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 0x7f9f9401abc0 crc :-1 s=READY pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 msgr2=0x7f9fb01915d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7f9fa0006f60 tx=0x7f9fa000b970).stop 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 shutdown_connections 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fb00973f0 0x7f9fb00977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fb0191bf0 0x7f9fb0096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9f940186c0 0x7f9f9401abc0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 --2- 172.21.2.117:0/1460502103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fb0194e10 0x7f9fb01915d0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 >> 172.21.2.117:0/1460502103 conn(0x7f9fb018cc70 msgr2=0x7f9fb0093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 shutdown_connections 2021-02-10T11:49:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.093 7f9fb4b0a700 1 -- 172.21.2.117:0/1460502103 wait complete. 2021-02-10T11:49:23.096 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:49:23.096 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:49:23.096 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:49:23.097 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:49:23.097 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:49:23.097 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:49:23.097 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:49:23.097 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:49:23.102 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:49:23.103 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:49:23.103 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:49:23.104 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:49:23.144 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:49:23.145 DEBUG:teuthology.parallel:result is None 2021-02-10T11:49:23.145 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:49:23.145 DEBUG:teuthology.parallel:result is None 2021-02-10T11:49:23.146 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:49:23.549 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:49:23.553 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.563 7fddab9be700 1 Processor -- start 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.563 7fddab9be700 1 -- start start 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fddab9be700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdda40e7180 0x7fdda40e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fddab9be700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 0x7fdda40e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fddab9be700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda40e7580 con 0x7fdda41b07f0 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fddab9be700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda40e5ce0 con 0x7fdda40e7180 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fddab9be700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40e77f0 con 0x7fdda40e8390 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda97da700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdda41b07f0 0x7fdda40e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54116/0 (socket says 172.21.2.117:54116) 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda97da700 1 -- 172.21.2.117:0/3746055243 learned_addr learned my addr 172.21.2.117:0/3746055243 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:23.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda9fdb700 1 --2- 172.21.2.117:0/3746055243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 0x7fdda40e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda8fd9700 1 --2- 172.21.2.117:0/3746055243 >> v2:172.21.2.125:3300/0 conn(0x7fdda40e7180 0x7fdda40e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdd9bfff700 1 -- 172.21.2.117:0/3746055243 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700750640 0 0) 0x7fdd940010b0 con 0x7fdda41b07f0 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdd9bfff700 1 -- 172.21.2.117:0/3746055243 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd90002fe0 con 0x7fdda41b07f0 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda9fdb700 1 -- 172.21.2.117:0/3746055243 >> v2:172.21.2.125:3300/0 conn(0x7fdda40e7180 msgr2=0x7fdda40e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda9fdb700 1 --2- 172.21.2.117:0/3746055243 >> v2:172.21.2.125:3300/0 conn(0x7fdda40e7180 0x7fdda40e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda9fdb700 1 -- 172.21.2.117:0/3746055243 >> v1:172.21.2.117:6789/0 conn(0x7fdda41b07f0 legacy=0x7fdda40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.564 7fdda9fdb700 1 -- 172.21.2.117:0/3746055243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40e8db0 con 0x7fdda40e8390 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fdda9fdb700 1 --2- 172.21.2.117:0/3746055243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 0x7fdda40e8790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdd9c006e50 tx=0x7fdd9c009e70).ready entity=mon.2 client_cookie=9eba560c3be7b379 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fdd9bfff700 1 -- 172.21.2.117:0/3746055243 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd9c0066f0 con 0x7fdda40e8390 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fdd9bfff700 1 -- 172.21.2.117:0/3746055243 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c003030 con 0x7fdda40e8390 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fdd9bfff700 1 -- 172.21.2.117:0/3746055243 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd9c00b790 con 0x7fdda40e8390 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 -- 172.21.2.117:0/3746055243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 msgr2=0x7fdda40e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 --2- 172.21.2.117:0/3746055243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 0x7fdda40e8790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdd9c006e50 tx=0x7fdd9c009e70).stop 2021-02-10T11:49:23.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 -- 172.21.2.117:0/3746055243 shutdown_connections 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 --2- 172.21.2.117:0/3746055243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40e8390 0x7fdda40e8790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 --2- 172.21.2.117:0/3746055243 >> v2:172.21.2.125:3300/0 conn(0x7fdda40e7180 0x7fdda40e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.565 7fddab9be700 1 -- 172.21.2.117:0/3746055243 >> 172.21.2.117:0/3746055243 conn(0x7fdda40ce500 msgr2=0x7fdda40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.566 7fddab9be700 1 -- 172.21.2.117:0/3746055243 shutdown_connections 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.566 7fddab9be700 1 -- 172.21.2.117:0/3746055243 wait complete. 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.580 7fddab9be700 1 Processor -- start 2021-02-10T11:49:23.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.580 7fddab9be700 1 -- start start 2021-02-10T11:49:23.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda40a3a60 0x7fdda41a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40b0a50 0x7fdda41a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40b19a0 con 0x7fdda40b15a0 2021-02-10T11:49:23.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40930b0 con 0x7fdda40a3a60 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fddab9be700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda41a3450 con 0x7fdda40b0a50 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53872/0 (socket says 172.21.2.117:53872) 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 -- 172.21.2.117:0/3919130172 learned_addr learned my addr 172.21.2.117:0/3919130172 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 -- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40b0a50 msgr2=0x7fdda41a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40b0a50 0x7fdda41a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 -- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda40a3a60 msgr2=0x7fdda41a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda40a3a60 0x7fdda41a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 -- 172.21.2.117:0/3919130172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda41a3630 con 0x7fdda40b15a0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.581 7fdda9fdb700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fdd9c006d00 tx=0x7fdd9c00bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd9c00be20 con 0x7fdda40b15a0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c0093d0 con 0x7fdda40b15a0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd9c00c570 con 0x7fdda40b15a0 2021-02-10T11:49:23.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fddab9be700 1 -- 172.21.2.117:0/3919130172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda41a3630 con 0x7fdda40b15a0 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fddab9be700 1 -- 172.21.2.117:0/3919130172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda41a3630 con 0x7fdda40b15a0 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd9c002020 con 0x7fdda40b15a0 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 0x7fdd8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.582 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fdd9c032650 con 0x7fdda40b15a0 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.583 7fdda97da700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 0x7fdd8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:23.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.583 7fdda97da700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 0x7fdd8401abc0 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:23.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.613 7fddab9be700 1 -- 172.21.2.117:0/3919130172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda40b19a0 con 0x7fdda40b15a0 2021-02-10T11:49:23.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.616 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd9c000df0 con 0x7fdda40b15a0 2021-02-10T11:49:23.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:23.755 7fddab9be700 1 -- 172.21.2.117:0/3919130172 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fdda40b19a0 con 0x7fdda40b15a0 2021-02-10T11:49:24.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.098 7fdd99ffb700 1 -- 172.21.2.117:0/3919130172 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdd9c002020 con 0x7fdda40b15a0 2021-02-10T11:49:24.116 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:49:24.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.122 7fddab9be700 1 -- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 msgr2=0x7fdd8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.122 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 0x7fdd8401abc0 crc :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.122 7fddab9be700 1 -- 172.21.2.117:0/3919130172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 msgr2=0x7fdda4093630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.122 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7fdd9c006d00 tx=0x7fdd9c00bce0).stop 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 -- 172.21.2.117:0/3919130172 shutdown_connections 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd840186c0 0x7fdd8401abc0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda40b15a0 0x7fdda4093630 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda40b0a50 0x7fdda41a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 --2- 172.21.2.117:0/3919130172 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda40a3a60 0x7fdda41a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 -- 172.21.2.117:0/3919130172 >> 172.21.2.117:0/3919130172 conn(0x7fdda40a7260 msgr2=0x7fdda40bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 -- 172.21.2.117:0/3919130172 shutdown_connections 2021-02-10T11:49:24.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.123 7fddab9be700 1 -- 172.21.2.117:0/3919130172 wait complete. 2021-02-10T11:49:24.126 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:49:24.127 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:49:24.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 Processor -- start 2021-02-10T11:49:24.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 -- start start 2021-02-10T11:49:24.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a540ede60 con 0x7f7a540eb000 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a540ef7f0 con 0x7f7a540d18e0 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a5ad97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540d1ce0 con 0x7f7a540ef3f0 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a53fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7a540eb000 0x7f7a540eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54132/0 (socket says 172.21.2.117:54132) 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.543 7f7a53fff700 1 -- 172.21.2.117:0/3367659682 learned_addr learned my addr 172.21.2.117:0/3367659682 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538118104 0 0) 0x7f7a440009b0 con 0x7f7a540eb000 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a40002fe0 con 0x7f7a540eb000 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2426916207 0 0) 0x7f7a440010b0 con 0x7f7a540eb000 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 msgr2=0x7f7a541b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a593b4700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a58bb3700 1 --2- 172.21.2.117:0/3367659682 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 msgr2=0x7f7a540ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:24.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 --2- 172.21.2.117:0/3367659682 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a541b8f90 con 0x7f7a540eb000 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7a440015b0 con 0x7f7a540eb000 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 >> v1:172.21.2.117:6789/0 conn(0x7f7a540eb000 legacy=0x7f7a540eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a40003b90 0x7f7a40006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a400068e0 0x7f7a40008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a400096e0 0x7f7a4000bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4000c420 con 0x7f7a400068e0 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4000c640 con 0x7f7a40003b90 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4000c860 con 0x7f7a400096e0 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a58bb3700 1 --2- 172.21.2.117:0/3367659682 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a593b4700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a400068e0 0x7f7a40008de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a58bb3700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a40003b90 0x7f7a40006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 -- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a400096e0 msgr2=0x7f7a4000bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a400096e0 0x7f7a4000bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 -- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a40003b90 msgr2=0x7f7a40006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a40003b90 0x7f7a40006040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.544 7f7a53fff700 1 -- 172.21.2.117:0/3367659682 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4000cb00 con 0x7f7a400068e0 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a53fff700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a400068e0 0x7f7a40008de0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f7a44001370 tx=0x7f7a44007cb0).ready entity=mon.0 client_cookie=1e9521c0a761e5d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a440070c0 con 0x7f7a400068e0 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a440010b0 con 0x7f7a400068e0 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a537fe700 1 -- 172.21.2.117:0/3367659682 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a4400a710 con 0x7f7a400068e0 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 -- 172.21.2.117:0/3367659682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a400068e0 msgr2=0x7f7a40008de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 --2- 172.21.2.117:0/3367659682 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a400068e0 0x7f7a40008de0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7f7a44001370 tx=0x7f7a44007cb0).stop 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a593b4700 1 -- 172.21.2.117:0/3367659682 reap_dead start 2021-02-10T11:49:24.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 -- 172.21.2.117:0/3367659682 shutdown_connections 2021-02-10T11:49:24.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 -- 172.21.2.117:0/3367659682 >> 172.21.2.117:0/3367659682 conn(0x7f7a540ced60 msgr2=0x7f7a540cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:24.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 -- 172.21.2.117:0/3367659682 shutdown_connections 2021-02-10T11:49:24.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.545 7f7a5ad97700 1 -- 172.21.2.117:0/3367659682 wait complete. 2021-02-10T11:49:24.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 Processor -- start 2021-02-10T11:49:24.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 -- start start 2021-02-10T11:49:24.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540b2160 0x7f7a540e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a5408bbe0 0x7f7a5408bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5408b660 con 0x7f7a5408c600 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5408ca00 con 0x7f7a540b2160 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.560 7f7a5ad97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540e9a80 con 0x7f7a5408bbe0 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53894/0 (socket says 172.21.2.117:53894) 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 -- 172.21.2.117:0/2085524418 learned_addr learned my addr 172.21.2.117:0/2085524418 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 -- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a5408bbe0 msgr2=0x7f7a5408bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a5408bbe0 0x7f7a5408bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 -- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540b2160 msgr2=0x7f7a540e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540b2160 0x7f7a540e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 -- 172.21.2.117:0/2085524418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5408a8a0 con 0x7f7a5408c600 2021-02-10T11:49:24.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a593b4700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f7a4c0099a0 tx=0x7f7a4c00bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a4c0062b0 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c0056b0 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a4c005bd0 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5408a8a0 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.561 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5408ae30 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.562 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a4c005b60 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.562 7f7a517fa700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 0x7f7a3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.562 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f7a4c032740 con 0x7f7a5408c600 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.562 7f7a58bb3700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 0x7f7a3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:24.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.563 7f7a58bb3700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 0x7f7a3801abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:24.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.593 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a540ede60 con 0x7f7a5408c600 2021-02-10T11:49:24.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.595 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a4c005950 con 0x7f7a5408c600 2021-02-10T11:49:24.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.726 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7a540ede60 con 0x7f7a5408c600 2021-02-10T11:49:24.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.727 7f7a517fa700 1 -- 172.21.2.117:0/2085524418 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v62) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f7a4c005b60 con 0x7f7a5408c600 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.723 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.724 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.725 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.726 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.727 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.728 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.729 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.730 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.731 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.732 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:49:24.733 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:24.734 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:49:24.735 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:24.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 msgr2=0x7f7a3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 0x7f7a3801abc0 crc :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 msgr2=0x7f7a5408a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f7a4c0099a0 tx=0x7f7a4c00bcc0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 shutdown_connections 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a380186c0 0x7f7a3801abc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a5408c600 0x7f7a5408a280 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a5408bbe0 0x7f7a5408bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 --2- 172.21.2.117:0/2085524418 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540b2160 0x7f7a540e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 >> 172.21.2.117:0/2085524418 conn(0x7f7a540bb5e0 msgr2=0x7f7a54090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 shutdown_connections 2021-02-10T11:49:24.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:24.729 7f7a5ad97700 1 -- 172.21.2.117:0/2085524418 wait complete. 2021-02-10T11:49:24.737 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:25.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 Processor -- start 2021-02-10T11:49:25.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 -- start start 2021-02-10T11:49:25.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48340d18e0 0x7f48340ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f48340ef3f0 0x7f48341b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48340ede60 con 0x7f48340eb000 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48340ef7f0 con 0x7f48340d18e0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f4839ebc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48340d1ce0 con 0x7f48340ef3f0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.153 7f48337fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48340d18e0 0x7f48340ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f4832ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f48340eb000 0x7f48340eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54148/0 (socket says 172.21.2.117:54148) 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f4832ffd700 1 -- 172.21.2.117:0/1244210536 learned_addr learned my addr 172.21.2.117:0/1244210536 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48337fe700 1 -- 172.21.2.117:0/1244210536 >> v2:172.21.2.125:3300/0 conn(0x7f48340ef3f0 msgr2=0x7f48341b7960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48337fe700 1 --2- 172.21.2.117:0/1244210536 >> v2:172.21.2.125:3300/0 conn(0x7f48340ef3f0 0x7f48341b7960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48337fe700 1 -- 172.21.2.117:0/1244210536 >> v1:172.21.2.117:6789/0 conn(0x7f48340eb000 legacy=0x7f48340eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48337fe700 1 -- 172.21.2.117:0/1244210536 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48341b8f90 con 0x7f48340d18e0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48337fe700 1 --2- 172.21.2.117:0/1244210536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340d18e0 0x7f48340ea9e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f481c006650 tx=0x7f481c009a80).ready entity=mon.0 client_cookie=cac70e6b79ee8e10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48327fc700 1 -- 172.21.2.117:0/1244210536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c00c0c0 con 0x7f48340d18e0 2021-02-10T11:49:25.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48327fc700 1 -- 172.21.2.117:0/1244210536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c00c340 con 0x7f48340d18e0 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f48327fc700 1 -- 172.21.2.117:0/1244210536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c00c860 con 0x7f48340d18e0 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f4839ebc700 1 -- 172.21.2.117:0/1244210536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340d18e0 msgr2=0x7f48340ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.154 7f4839ebc700 1 --2- 172.21.2.117:0/1244210536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340d18e0 0x7f48340ea9e0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f481c006650 tx=0x7f481c009a80).stop 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 -- 172.21.2.117:0/1244210536 shutdown_connections 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 --2- 172.21.2.117:0/1244210536 >> v2:172.21.2.125:3300/0 conn(0x7f48340ef3f0 0x7f48341b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 --2- 172.21.2.117:0/1244210536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340d18e0 0x7f48340ea9e0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 -- 172.21.2.117:0/1244210536 >> 172.21.2.117:0/1244210536 conn(0x7f48340ced60 msgr2=0x7f48340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 -- 172.21.2.117:0/1244210536 shutdown_connections 2021-02-10T11:49:25.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.155 7f4839ebc700 1 -- 172.21.2.117:0/1244210536 wait complete. 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.169 7f4839ebc700 1 Processor -- start 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.169 7f4839ebc700 1 -- start start 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 0x7f483419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 0x7f48341afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48341af950 con 0x7f483419b760 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483419bb60 con 0x7f48340b5370 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4839ebc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483419ea20 con 0x7f483419e620 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53902/0 (socket says 172.21.2.117:53902) 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 -- 172.21.2.117:0/3923948467 learned_addr learned my addr 172.21.2.117:0/3923948467 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 -- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 msgr2=0x7f48341afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4833fff700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 0x7f48341afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f4832ffd700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 0x7f483419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 0x7f48341afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.170 7f48337fe700 1 -- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 msgr2=0x7f483419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f48337fe700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 0x7f483419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f48337fe700 1 -- 172.21.2.117:0/3923948467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483419ec40 con 0x7f483419b760 2021-02-10T11:49:25.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f4833fff700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 0x7f48341afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f4832ffd700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 0x7f483419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f48337fe700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f481c009630 tx=0x7f481c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c0093a0 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c00bd70 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c0054d0 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483419ec40 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.171 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483419ec40 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.172 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f481c00a000 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.172 7f482bfff700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 0x7f481801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.172 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f481c0324a0 con 0x7f483419b760 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.172 7f4832ffd700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 0x7f481801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.173 7f4832ffd700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 0x7f481801abd0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.203 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4834087210 con 0x7f483419b760 2021-02-10T11:49:25.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.206 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f481c01f540 con 0x7f483419b760 2021-02-10T11:49:25.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.329 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4834087210 con 0x7f483419b760 2021-02-10T11:49:25.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.329 7f482bfff700 1 -- 172.21.2.117:0/3923948467 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+11379 (secure 0 0 0) 0x7f481c00a000 con 0x7f483419b760 2021-02-10T11:49:25.325 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:25.325 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":104,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:24.093118","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:25.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 msgr2=0x7f481801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 0x7f481801abd0 crc :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 msgr2=0x7f48340b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f481c009630 tx=0x7f481c00b980).stop 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 shutdown_connections 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f483419e620 0x7f48341afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340b5370 0x7f483419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186d0 0x7f481801abd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 --2- 172.21.2.117:0/3923948467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f483419b760 0x7f48340b4d50 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.331 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 >> 172.21.2.117:0/3923948467 conn(0x7f483419f610 msgr2=0x7f48341ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:25.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.332 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 shutdown_connections 2021-02-10T11:49:25.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.332 7f4839ebc700 1 -- 172.21.2.117:0/3923948467 wait complete. 2021-02-10T11:49:25.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:49:25.336 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 Processor -- start 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 -- start start 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8400ef3f0 0x7ff8400eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400ef7f0 con 0x7ff8400d18f0 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8400ede60 con 0x7ff8400eb830 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff8458bf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8400d1da0 con 0x7ff8400ef3f0 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff83f574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8400d18f0 0x7ff8400eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54160/0 (socket says 172.21.2.117:54160) 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.756 7ff83f574700 1 -- 172.21.2.117:0/2425939042 learned_addr learned my addr 172.21.2.117:0/2425939042 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007140684 0 0) 0x7ff828001180 con 0x7ff8400d18f0 2021-02-10T11:49:25.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c002fe0 con 0x7ff8400d18f0 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83ed73700 1 --2- 172.21.2.117:0/2425939042 >> v2:172.21.2.125:3300/0 conn(0x7ff8400ef3f0 0x7ff8400eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1082430257 0 0) 0x7ff828001180 con 0x7ff8400d18f0 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 >> v2:172.21.2.125:3300/0 conn(0x7ff8400ef3f0 msgr2=0x7ff8400eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 --2- 172.21.2.117:0/2425939042 >> v2:172.21.2.125:3300/0 conn(0x7ff8400ef3f0 0x7ff8400eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 >> v1:172.21.2.125:6789/0 conn(0x7ff8400eb830 legacy=0x7ff8401b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8401bb1c0 con 0x7ff8400d18f0 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff828001680 con 0x7ff8400d18f0 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 >> v1:172.21.2.117:6789/0 conn(0x7ff8400d18f0 legacy=0x7ff8400eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff82c003b10 0x7ff82c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 0x7ff82c008d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff82c0095c0 0x7ff82c00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c00c310 con 0x7ff82c006830 2021-02-10T11:49:25.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c00c530 con 0x7ff82c003b10 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83e572700 1 -- 172.21.2.117:0/2425939042 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c00c750 con 0x7ff82c0095c0 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.757 7ff83ed73700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 0x7ff82c008d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83f574700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff82c003b10 0x7ff82c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83fd75700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff82c0095c0 0x7ff82c00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 -- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff82c0095c0 msgr2=0x7ff82c00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff82c0095c0 0x7ff82c00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 -- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff82c003b10 msgr2=0x7ff82c006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff82c003b10 0x7ff82c006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 -- 172.21.2.117:0/2425939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c00c9f0 con 0x7ff82c006830 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83fd75700 1 -- 172.21.2.117:0/2425939042 reap_dead start 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83ed73700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 0x7ff82c008d50 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7ff830006a40 tx=0x7ff830006820).ready entity=mon.0 client_cookie=b9865dfce2f26da0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff830000c90 con 0x7ff82c006830 2021-02-10T11:49:25.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830002030 con 0x7ff82c006830 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff83e572700 1 -- 172.21.2.117:0/2425939042 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff830002510 con 0x7ff82c006830 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff8458bf700 1 -- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 msgr2=0x7ff82c008d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.758 7ff8458bf700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 0x7ff82c008d50 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7ff830006a40 tx=0x7ff830006820).stop 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.759 7ff8458bf700 1 -- 172.21.2.117:0/2425939042 shutdown_connections 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.759 7ff8458bf700 1 --2- 172.21.2.117:0/2425939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff82c006830 0x7ff82c008d50 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.759 7ff8458bf700 1 -- 172.21.2.117:0/2425939042 >> 172.21.2.117:0/2425939042 conn(0x7ff8400cf060 msgr2=0x7ff8400cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.759 7ff8458bf700 1 -- 172.21.2.117:0/2425939042 shutdown_connections 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.759 7ff8458bf700 1 -- 172.21.2.117:0/2425939042 wait complete. 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.773 7ff8458bf700 1 Processor -- start 2021-02-10T11:49:25.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.773 7ff8458bf700 1 -- start start 2021-02-10T11:49:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 0x7ff8400862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8400a2040 0x7ff8400a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400a67e0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400a7120 con 0x7ff8400a2040 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff8458bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8400a2a60 con 0x7ff840085ec0 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff83f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff83f574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53922/0 (socket says 172.21.2.117:53922) 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff83f574700 1 -- 172.21.2.117:0/1932449726 learned_addr learned my addr 172.21.2.117:0/1932449726 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff83ed73700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 0x7ff8400862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.774 7ff83fd75700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8400a2040 0x7ff8400a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 -- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 msgr2=0x7ff8400862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 0x7ff8400862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 -- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8400a2040 msgr2=0x7ff8400a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8400a2040 0x7ff8400a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 -- 172.21.2.117:0/1932449726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8401a2dc0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83ed73700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 0x7ff8400862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff83f574700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7ff828009610 tx=0x7ff8280099b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff828007a70 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8280043a0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8401a2dc0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8401a3350 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.775 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8280047f0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.776 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff828007a70 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.776 7ff837fff700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 0x7ff82401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.776 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5100+0+0 (secure 0 0 0) 0x7ff828032330 con 0x7ff8400a6ce0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.776 7ff83ed73700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 0x7ff82401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:25.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.777 7ff83ed73700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 0x7ff82401abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:25.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.806 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8400daff0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.809 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8280070c0 con 0x7ff8400a6ce0 2021-02-10T11:49:25.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.937 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff8400ede60 con 0x7ff8400a6ce0 2021-02-10T11:49:25.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.937 7ff837fff700 1 -- 172.21.2.117:0/1932449726 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v104) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff828007a70 con 0x7ff8400a6ce0 2021-02-10T11:49:25.933 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 msgr2=0x7ff82401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 0x7ff82401abc0 crc :-1 s=READY pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 msgr2=0x7ff8400dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7ff828009610 tx=0x7ff8280099b0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 shutdown_connections 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.939 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8400a2040 0x7ff8400a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff840085ec0 0x7ff8400862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8240186c0 0x7ff82401abc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 --2- 172.21.2.117:0/1932449726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8400a6ce0 0x7ff8400dc420 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:25.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 >> 172.21.2.117:0/1932449726 conn(0x7ff8400b6540 msgr2=0x7ff84008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 shutdown_connections 2021-02-10T11:49:25.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:25.940 7ff8458bf700 1 -- 172.21.2.117:0/1932449726 wait complete. 2021-02-10T11:49:25.944 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 Processor -- start 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 -- start start 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb7400d18e0 0x7fb7400ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400ede60 con 0x7fb7400ef3f0 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400d1ce0 con 0x7fb7400eb000 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb74849d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7400ef7f0 con 0x7fb7400d18e0 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb7462b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb7400d18e0 0x7fb7400ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb7462b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb7400d18e0 0x7fb7400ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53930/0 (socket says 172.21.2.117:53930) 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb7462b9700 1 -- 172.21.2.117:0/443654561 learned_addr learned my addr 172.21.2.117:0/443654561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.359 7fb746aba700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb7400ef3f0 0x7fb7401b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54184/0 (socket says 172.21.2.117:54184) 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7462b9700 1 -- 172.21.2.117:0/443654561 >> v1:172.21.2.125:6789/0 conn(0x7fb7400eb000 legacy=0x7fb7400eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7462b9700 1 -- 172.21.2.117:0/443654561 >> v1:172.21.2.117:6789/0 conn(0x7fb7400ef3f0 legacy=0x7fb7401b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7462b9700 1 -- 172.21.2.117:0/443654561 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7401bb180 con 0x7fb7400d18e0 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7462b9700 1 --2- 172.21.2.117:0/443654561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7400d18e0 0x7fb7400ea9e0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7fb738006a80 tx=0x7fb738009f00).ready entity=mon.0 client_cookie=927db5a22c4ac8a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7452b7700 1 -- 172.21.2.117:0/443654561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb73800b5c0 con 0x7fb7400d18e0 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7452b7700 1 -- 172.21.2.117:0/443654561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73800b8c0 con 0x7fb7400d18e0 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb7452b7700 1 -- 172.21.2.117:0/443654561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb73800bde0 con 0x7fb7400d18e0 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb74849d700 1 -- 172.21.2.117:0/443654561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7400d18e0 msgr2=0x7fb7400ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.360 7fb74849d700 1 --2- 172.21.2.117:0/443654561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7400d18e0 0x7fb7400ea9e0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7fb738006a80 tx=0x7fb738009f00).stop 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.361 7fb74849d700 1 -- 172.21.2.117:0/443654561 shutdown_connections 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.361 7fb74849d700 1 --2- 172.21.2.117:0/443654561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7400d18e0 0x7fb7400ea9e0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.361 7fb74849d700 1 -- 172.21.2.117:0/443654561 >> 172.21.2.117:0/443654561 conn(0x7fb7400ced60 msgr2=0x7fb7400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:26.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.361 7fb74849d700 1 -- 172.21.2.117:0/443654561 shutdown_connections 2021-02-10T11:49:26.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.361 7fb74849d700 1 -- 172.21.2.117:0/443654561 wait complete. 2021-02-10T11:49:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 Processor -- start 2021-02-10T11:49:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 -- start start 2021-02-10T11:49:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7400ecb00 0x7fb7401b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 0x7fb7401c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74008d0f0 con 0x7fb7401b6030 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7401c9ac0 con 0x7fb7401c90a0 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb74849d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7401c9ce0 con 0x7fb7400ecb00 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb745ab8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb745ab8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53940/0 (socket says 172.21.2.117:53940) 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb745ab8700 1 -- 172.21.2.117:0/1939831922 learned_addr learned my addr 172.21.2.117:0/1939831922 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb7462b9700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7400ecb00 0x7fb7401b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.376 7fb746aba700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 0x7fb7401c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 -- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7400ecb00 msgr2=0x7fb7401b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7400ecb00 0x7fb7401b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 -- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 msgr2=0x7fb7401c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 0x7fb7401c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 -- 172.21.2.117:0/1939831922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7401c9f00 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb746aba700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 0x7fb7401c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb745ab8700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7fb730006ed0 tx=0x7fb73000b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb730006240 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7300009b0 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb74849d700 1 -- 172.21.2.117:0/1939831922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7401c9f00 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb74849d700 1 -- 172.21.2.117:0/1939831922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74008d960 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.377 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb730005730 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.378 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb730005c00 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.378 7fb736ffd700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 0x7fb72801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.378 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fb7300325a0 con 0x7fb7401b6030 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.378 7fb7462b9700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 0x7fb72801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.379 7fb7462b9700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 0x7fb72801ac10 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.409 7fb74849d700 1 -- 172.21.2.117:0/1939831922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb740086140 con 0x7fb7401b6030 2021-02-10T11:49:26.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.411 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb7300059f0 con 0x7fb7401b6030 2021-02-10T11:49:26.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.537 7fb74849d700 1 -- 172.21.2.117:0/1939831922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb7400d18e0 con 0x7fb7401b6030 2021-02-10T11:49:26.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.538 7fb736ffd700 1 -- 172.21.2.117:0/1939831922 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+64 (secure 0 0 0) 0x7fb730005c00 con 0x7fb7401b6030 2021-02-10T11:49:26.533 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:26.534 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:49:26.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 msgr2=0x7fb72801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 0x7fb72801ac10 crc :-1 s=READY pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 msgr2=0x7fb7401b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7fb730006ed0 tx=0x7fb73000b8e0).stop 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 shutdown_connections 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb728018730 0x7fb72801ac10 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7401b6030 0x7fb7401b6430 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7401c90a0 0x7fb7401c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 --2- 172.21.2.117:0/1939831922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7400ecb00 0x7fb7401b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 >> 172.21.2.117:0/1939831922 conn(0x7fb740086d40 msgr2=0x7fb7400f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 shutdown_connections 2021-02-10T11:49:26.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.540 7fb74849d700 1 -- 172.21.2.117:0/1939831922 wait complete. 2021-02-10T11:49:26.544 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:49:26.544 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 Processor -- start 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 -- start start 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 0x7f98e41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40ef7f0 con 0x7f98e40d18f0 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e40eded0 con 0x7f98e40eb830 2021-02-10T11:49:26.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.964 7f98ea85d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40d1cf0 con 0x7f98e40ef3f0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.965 7f98e37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.965 7f98e37fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51628/0 (socket says 172.21.2.117:51628) 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.965 7f98e37fe700 1 -- 172.21.2.117:0/3924554667 learned_addr learned my addr 172.21.2.117:0/3924554667 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.965 7f98e8e7a700 1 --2- 172.21.2.117:0/3924554667 >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 0x7f98e41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e2ffd700 1 -- 172.21.2.117:0/3924554667 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158725749 0 0) 0x7f98d4000c60 con 0x7f98e40d18f0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e37fe700 1 -- 172.21.2.117:0/3924554667 >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 msgr2=0x7f98e41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e37fe700 1 --2- 172.21.2.117:0/3924554667 >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 0x7f98e41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e37fe700 1 -- 172.21.2.117:0/3924554667 >> v1:172.21.2.125:6789/0 conn(0x7f98e40d18f0 legacy=0x7f98e40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e37fe700 1 -- 172.21.2.117:0/3924554667 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40d3170 con 0x7f98e40ef3f0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e8e7a700 1 --2- 172.21.2.117:0/3924554667 >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 0x7f98e41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.966 7f98e37fe700 1 --2- 172.21.2.117:0/3924554667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f98d8006700 tx=0x7f98d8009b80).ready entity=mon.2 client_cookie=d0e225b49aadaa6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98e2ffd700 1 -- 172.21.2.117:0/3924554667 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d8006340 con 0x7f98e40ef3f0 2021-02-10T11:49:26.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98e2ffd700 1 -- 172.21.2.117:0/3924554667 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d8006340 con 0x7f98e40ef3f0 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98e2ffd700 1 -- 172.21.2.117:0/3924554667 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d800c0c0 con 0x7f98e40ef3f0 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 -- 172.21.2.117:0/3924554667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 msgr2=0x7f98e40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 --2- 172.21.2.117:0/3924554667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f98d8006700 tx=0x7f98d8009b80).stop 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 -- 172.21.2.117:0/3924554667 shutdown_connections 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 --2- 172.21.2.117:0/3924554667 >> v2:172.21.2.125:3300/0 conn(0x7f98e40eb830 0x7f98e41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 --2- 172.21.2.117:0/3924554667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40ef3f0 0x7f98e40eb210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 -- 172.21.2.117:0/3924554667 >> 172.21.2.117:0/3924554667 conn(0x7f98e40cf060 msgr2=0x7f98e40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 -- 172.21.2.117:0/3924554667 shutdown_connections 2021-02-10T11:49:26.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.967 7f98ea85d700 1 -- 172.21.2.117:0/3924554667 wait complete. 2021-02-10T11:49:26.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 Processor -- start 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 -- start start 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40a6ce0 0x7f98e40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98e40a2040 0x7f98e40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40a67e0 con 0x7f98e4085ec0 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40a7120 con 0x7f98e40a2040 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.982 7f98ea85d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40a2a60 con 0x7f98e40a6ce0 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53950/0 (socket says 172.21.2.117:53950) 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 -- 172.21.2.117:0/3344168930 learned_addr learned my addr 172.21.2.117:0/3344168930 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:26.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 -- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40a6ce0 msgr2=0x7f98e40dc420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40a6ce0 0x7f98e40dc420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 -- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98e40a2040 msgr2=0x7f98e40a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98e40a2040 0x7f98e40a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 -- 172.21.2.117:0/3344168930 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e41a2dc0 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e37fe700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f98d800b600 tx=0x7f98d800b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d8002030 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d8002330 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e41a2dc0 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.983 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e41a3350 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.984 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98d8002850 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.984 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f98d8000c90 con 0x7f98e4085ec0 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.984 7f98e0ff9700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 0x7f98c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:26.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.984 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f98d8032360 con 0x7f98e4085ec0 2021-02-10T11:49:26.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.984 7f98e3fff700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 0x7f98c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:26.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:26.985 7f98e3fff700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 0x7f98c401abc0 crc :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:27.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.014 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e40daff0 con 0x7f98e4085ec0 2021-02-10T11:49:27.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.017 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f98d80009b0 con 0x7f98e4085ec0 2021-02-10T11:49:27.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.144 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f98e40ede60 con 0x7f98e4085ec0 2021-02-10T11:49:27.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.562 7f98e0ff9700 1 -- 172.21.2.117:0/3344168930 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v105) v1 ==== 133+0+0 (secure 0 0 0) 0x7f98d8000c90 con 0x7f98e4085ec0 2021-02-10T11:49:27.606 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:49:27.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 msgr2=0x7f98c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 0x7f98c401abc0 crc :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 msgr2=0x7f98e40862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f98d800b600 tx=0x7f98d800b9a0).stop 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 shutdown_connections 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98c40186c0 0x7f98c401abc0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98e4085ec0 0x7f98e40862c0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:27.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98e40a2040 0x7f98e40a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 --2- 172.21.2.117:0/3344168930 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98e40a6ce0 0x7f98e40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:27.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.612 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 >> 172.21.2.117:0/3344168930 conn(0x7f98e40b6540 msgr2=0x7f98e408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:27.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.613 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 shutdown_connections 2021-02-10T11:49:27.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:27.613 7f98ea85d700 1 -- 172.21.2.117:0/3344168930 wait complete. 2021-02-10T11:49:27.616 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:49:28.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.036 7f471ce8f700 1 Processor -- start 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.036 7f471ce8f700 1 -- start start 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47180d18f0 0x7f47180eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f47180eb040 0x7f47180eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47180ee3e0 0x7f47180ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180edee0 con 0x7f47180d18f0 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47180ebaa0 con 0x7f47180eb040 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47180d1cf0 con 0x7f47180ee3e0 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47180d18f0 0x7f47180eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47180d18f0 0x7f47180eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53958/0 (socket says 172.21.2.117:53958) 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 -- 172.21.2.117:0/2265800448 learned_addr learned my addr 172.21.2.117:0/2265800448 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:28.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 -- 172.21.2.117:0/2265800448 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47180ee3e0 msgr2=0x7f47180ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 --2- 172.21.2.117:0/2265800448 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47180ee3e0 0x7f47180ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 -- 172.21.2.117:0/2265800448 >> v2:172.21.2.125:3300/0 conn(0x7f47180eb040 msgr2=0x7f47180eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 --2- 172.21.2.117:0/2265800448 >> v2:172.21.2.125:3300/0 conn(0x7f47180eb040 0x7f47180eb440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 -- 172.21.2.117:0/2265800448 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47180eee00 con 0x7f47180d18f0 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4716b13700 1 --2- 172.21.2.117:0/2265800448 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47180d18f0 0x7f47180eaa20 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f4708006700 tx=0x7f4708009b80).ready entity=mon.0 client_cookie=22ef259bf9605a3a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4715b11700 1 -- 172.21.2.117:0/2265800448 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470800c0c0 con 0x7f47180d18f0 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4715b11700 1 -- 172.21.2.117:0/2265800448 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470800c3c0 con 0x7f47180d18f0 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f4715b11700 1 -- 172.21.2.117:0/2265800448 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470800c810 con 0x7f47180d18f0 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 -- 172.21.2.117:0/2265800448 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47180d18f0 msgr2=0x7f47180eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.037 7f471ce8f700 1 --2- 172.21.2.117:0/2265800448 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47180d18f0 0x7f47180eaa20 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f4708006700 tx=0x7f4708009b80).stop 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 -- 172.21.2.117:0/2265800448 shutdown_connections 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 --2- 172.21.2.117:0/2265800448 >> v2:172.21.2.125:3300/0 conn(0x7f47180eb040 0x7f47180eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 --2- 172.21.2.117:0/2265800448 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47180ee3e0 0x7f47180ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 --2- 172.21.2.117:0/2265800448 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47180d18f0 0x7f47180eaa20 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 -- 172.21.2.117:0/2265800448 >> 172.21.2.117:0/2265800448 conn(0x7f47180cf060 msgr2=0x7f47180cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:28.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 -- 172.21.2.117:0/2265800448 shutdown_connections 2021-02-10T11:49:28.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.038 7f471ce8f700 1 -- 172.21.2.117:0/2265800448 wait complete. 2021-02-10T11:49:28.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 Processor -- start 2021-02-10T11:49:28.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 -- start start 2021-02-10T11:49:28.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f471809ed70 0x7f471809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f471809f790 0x7f47180b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471809fb90 con 0x7f47181a4240 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47180dc640 con 0x7f471809f790 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f471ce8f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4718097270 con 0x7f471809ed70 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53966/0 (socket says 172.21.2.117:53966) 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 -- 172.21.2.117:0/2639326319 learned_addr learned my addr 172.21.2.117:0/2639326319 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 -- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f471809ed70 msgr2=0x7f471809f170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:28.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f471809ed70 0x7f471809f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 -- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f471809f790 msgr2=0x7f47180b8160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f471809f790 0x7f47180b8160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.053 7f4716b13700 1 -- 172.21.2.117:0/2639326319 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47180b8780 con 0x7f47181a4240 2021-02-10T11:49:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f4716b13700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f470800b650 tx=0x7f470800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f470800bde0 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4708006390 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47180b8780 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47180b8d10 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47080022b0 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47080052e0 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.054 7f47077fe700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 0x7f46fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.055 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f4708032740 con 0x7f47181a4240 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.055 7f4716312700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 0x7f46fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:28.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.055 7f4716312700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 0x7f46fc01ac10 crc :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:28.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.087 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47180daff0 con 0x7f47181a4240 2021-02-10T11:49:28.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.089 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47080090b0 con 0x7f47181a4240 2021-02-10T11:49:28.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.215 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f47180daff0 con 0x7f47181a4240 2021-02-10T11:49:28.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.570 7f47077fe700 1 -- 172.21.2.117:0/2639326319 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v107) v1 ==== 125+0+0 (secure 0 0 0) 0x7f47080052e0 con 0x7f47181a4240 2021-02-10T11:49:28.575 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 msgr2=0x7f46fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 0x7f46fc01ac10 crc :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 msgr2=0x7f47181a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f470800b650 tx=0x7f470800b9f0).stop 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 shutdown_connections 2021-02-10T11:49:28.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f471809f790 0x7f47180b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f471809ed70 0x7f471809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46fc018730 0x7f46fc01ac10 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 --2- 172.21.2.117:0/2639326319 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47181a4240 0x7f47181a4640 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.582 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 >> 172.21.2.117:0/2639326319 conn(0x7f4718087090 msgr2=0x7f471808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.583 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 shutdown_connections 2021-02-10T11:49:28.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:28.583 7f471ce8f700 1 -- 172.21.2.117:0/2639326319 wait complete. 2021-02-10T11:49:28.586 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:49:29.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.008 7ff80af6e700 1 Processor -- start 2021-02-10T11:49:29.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.008 7ff80af6e700 1 -- start start 2021-02-10T11:49:29.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80af6e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8040e4e40 0x7ff8041b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80af6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff804085450 0x7ff804086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80af6e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804085930 con 0x7ff8040e5240 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80af6e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8040e5cc0 con 0x7ff8040e4e40 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80af6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804084840 con 0x7ff804085450 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff803fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8040e5240 0x7ff804084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54224/0 (socket says 172.21.2.117:54224) 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff803fff700 1 -- 172.21.2.117:0/3631089734 learned_addr learned my addr 172.21.2.117:0/3631089734 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912926175 0 0) 0x7ff7f4000ce0 con 0x7ff8040e5240 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f0002fe0 con 0x7ff8040e5240 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff80958b700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff804085450 0x7ff804086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.009 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> v2:172.21.2.125:3300/0 conn(0x7ff8040e4e40 0x7ff8041b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1701807945 0 0) 0x7ff7f40010e0 con 0x7ff8040e5240 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff804085450 msgr2=0x7ff804086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff804085450 0x7ff804086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 >> v2:172.21.2.125:3300/0 conn(0x7ff8040e4e40 msgr2=0x7ff8041b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 --2- 172.21.2.117:0/3631089734 >> v2:172.21.2.125:3300/0 conn(0x7ff8040e4e40 0x7ff8041b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8041b0810 con 0x7ff8040e5240 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff7f4001660 con 0x7ff8040e5240 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 >> v1:172.21.2.117:6789/0 conn(0x7ff8040e5240 legacy=0x7ff804084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f0003b90 0x7ff7f0006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f0006860 0x7ff7f0008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00095d0 0x7ff7f000bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f0050390 con 0x7ff7f0003b90 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00505b0 con 0x7ff7f00095d0 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00507d0 con 0x7ff7f0006860 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> v2:172.21.2.125:3300/0 conn(0x7ff8040e4e40 0x7ff8041b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f0003b90 0x7ff7f0006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff803fff700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f0006860 0x7ff7f0008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff80958b700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00095d0 0x7ff7f000bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 -- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f0006860 msgr2=0x7ff7f0008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff7f0006860 0x7ff7f0008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 -- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00095d0 msgr2=0x7ff7f000bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff7f00095d0 0x7ff7f000bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 -- 172.21.2.117:0/3631089734 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0050a70 con 0x7ff7f0003b90 2021-02-10T11:49:29.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.010 7ff808d8a700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f0003b90 0x7ff7f0006040 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7ff7ec009880 tx=0x7ff7ec009c20).ready entity=mon.0 client_cookie=db5f4638aa7ce8fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ec000960 con 0x7ff7f0003b90 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec000c90 con 0x7ff7f0003b90 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff8037fe700 1 -- 172.21.2.117:0/3631089734 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ec002030 con 0x7ff7f0003b90 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 -- 172.21.2.117:0/3631089734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f0003b90 msgr2=0x7ff7f0006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 --2- 172.21.2.117:0/3631089734 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff7f0003b90 0x7ff7f0006040 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7ff7ec009880 tx=0x7ff7ec009c20).stop 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80958b700 1 -- 172.21.2.117:0/3631089734 reap_dead start 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 -- 172.21.2.117:0/3631089734 shutdown_connections 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 -- 172.21.2.117:0/3631089734 >> 172.21.2.117:0/3631089734 conn(0x7ff8040ce4e0 msgr2=0x7ff8040e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 -- 172.21.2.117:0/3631089734 shutdown_connections 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.011 7ff80af6e700 1 -- 172.21.2.117:0/3631089734 wait complete. 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 Processor -- start 2021-02-10T11:49:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 -- start start 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8040bfa00 0x7ff8040bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8040c0420 0x7ff8040a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040bee60 con 0x7ff8040be700 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040beb00 con 0x7ff8040bfa00 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80af6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040912f0 con 0x7ff8040c0420 2021-02-10T11:49:29.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff808d8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff808d8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53986/0 (socket says 172.21.2.117:53986) 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff808d8a700 1 -- 172.21.2.117:0/2386566302 learned_addr learned my addr 172.21.2.117:0/2386566302 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.026 7ff80958b700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8040c0420 0x7ff8040a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff803fff700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8040bfa00 0x7ff8040bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 -- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8040c0420 msgr2=0x7ff8040a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8040c0420 0x7ff8040a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 -- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8040bfa00 msgr2=0x7ff8040bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8040bfa00 0x7ff8040bfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 -- 172.21.2.117:0/2386566302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8040a9bd0 con 0x7ff8040be700 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff808d8a700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7ff7ec009770 tx=0x7ff7ec00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ec006930 con 0x7ff8040be700 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ec002030 con 0x7ff8040be700 2021-02-10T11:49:29.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8040a9bd0 con 0x7ff8040be700 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8040a9bd0 con 0x7ff8040be700 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.027 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff7ec002550 con 0x7ff8040be700 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.028 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff7ec000c90 con 0x7ff8040be700 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.028 7ff8017fa700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 0x7ff7e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.028 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ff7ec032510 con 0x7ff8040be700 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.028 7ff803fff700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 0x7ff7e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.029 7ff803fff700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 0x7ff7e801abc0 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.058 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80410c2f0 con 0x7ff8040be700 2021-02-10T11:49:29.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.061 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff7ec030d60 con 0x7ff8040be700 2021-02-10T11:49:29.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.187 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7ff804085930 con 0x7ff8040be700 2021-02-10T11:49:29.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.191 7ff8017fa700 1 -- 172.21.2.117:0/2386566302 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 22 and data pool 23 v83) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff7ec000c90 con 0x7ff8040be700 2021-02-10T11:49:29.190 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 22 and data pool 23 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.197 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 msgr2=0x7ff7e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.197 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 0x7ff7e801abc0 crc :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.197 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 msgr2=0x7ff8040bf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.197 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7ff7ec009770 tx=0x7ff7ec00b990).stop 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 shutdown_connections 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8040c0420 0x7ff8040a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8040bfa00 0x7ff8040bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7e80186c0 0x7ff7e801abc0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 --2- 172.21.2.117:0/2386566302 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8040be700 0x7ff8040bf3e0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 >> 172.21.2.117:0/2386566302 conn(0x7ff8040bd990 msgr2=0x7ff8040c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 shutdown_connections 2021-02-10T11:49:29.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.198 7ff80af6e700 1 -- 172.21.2.117:0/2386566302 wait complete. 2021-02-10T11:49:29.200 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:29.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 Processor -- start 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 -- start start 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88c40e4e40 0x7f88c41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c40e5240 0x7f88c4084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f88c41b0810 0x7f88c4085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40e5cc0 con 0x7f88c40e4e40 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c4085930 con 0x7f88c41b0810 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c92c3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4084840 con 0x7f88c40e5240 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c307d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88c40e4e40 0x7f88c41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c307d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88c40e4e40 0x7f88c41b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:53994/0 (socket says 172.21.2.117:53994) 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.616 7f88c307d700 1 -- 172.21.2.117:0/1364124343 learned_addr learned my addr 172.21.2.117:0/1364124343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:29.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 -- 172.21.2.117:0/1364124343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c40e5240 msgr2=0x7f88c4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 --2- 172.21.2.117:0/1364124343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c40e5240 0x7f88c4084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 -- 172.21.2.117:0/1364124343 >> v2:172.21.2.125:3300/0 conn(0x7f88c41b0810 msgr2=0x7f88c4085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 --2- 172.21.2.117:0/1364124343 >> v2:172.21.2.125:3300/0 conn(0x7f88c41b0810 0x7f88c4085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 -- 172.21.2.117:0/1364124343 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c4085eb0 con 0x7f88c40e4e40 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c307d700 1 --2- 172.21.2.117:0/1364124343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40e4e40 0x7f88c41b01f0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f88b4006700 tx=0x7f88b4009b80).ready entity=mon.0 client_cookie=bf8aa40e5009a95b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c207b700 1 -- 172.21.2.117:0/1364124343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b400c0c0 con 0x7f88c40e4e40 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c207b700 1 -- 172.21.2.117:0/1364124343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b400c340 con 0x7f88c40e4e40 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c207b700 1 -- 172.21.2.117:0/1364124343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b400c860 con 0x7f88c40e4e40 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c92c3700 1 -- 172.21.2.117:0/1364124343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40e4e40 msgr2=0x7f88c41b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.617 7f88c92c3700 1 --2- 172.21.2.117:0/1364124343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40e4e40 0x7f88c41b01f0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f88b4006700 tx=0x7f88b4009b80).stop 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 -- 172.21.2.117:0/1364124343 shutdown_connections 2021-02-10T11:49:29.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 --2- 172.21.2.117:0/1364124343 >> v2:172.21.2.125:3300/0 conn(0x7f88c41b0810 0x7f88c4085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 --2- 172.21.2.117:0/1364124343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c40e5240 0x7f88c4084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:29.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 --2- 172.21.2.117:0/1364124343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40e4e40 0x7f88c41b01f0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 -- 172.21.2.117:0/1364124343 >> 172.21.2.117:0/1364124343 conn(0x7f88c40ce4e0 msgr2=0x7f88c40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 -- 172.21.2.117:0/1364124343 shutdown_connections 2021-02-10T11:49:29.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.618 7f88c92c3700 1 -- 172.21.2.117:0/1364124343 wait complete. 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 Processor -- start 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 -- start start 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88c419a980 0x7f88c40a91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c418a500 0x7f88c418a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41aee30 con 0x7f88c40a97e0 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41af020 con 0x7f88c419a980 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c92c3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4187cf0 con 0x7f88c418a500 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54000/0 (socket says 172.21.2.117:54000) 2021-02-10T11:49:29.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.633 7f88c287c700 1 -- 172.21.2.117:0/275153882 learned_addr learned my addr 172.21.2.117:0/275153882 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c387e700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c418a500 0x7f88c418a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c307d700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88c419a980 0x7f88c40a91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 -- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c418a500 msgr2=0x7f88c418a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c418a500 0x7f88c418a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 -- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88c419a980 msgr2=0x7f88c40a91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88c419a980 0x7f88c40a91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 -- 172.21.2.117:0/275153882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c4187f10 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88c287c700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f88b8006f00 tx=0x7f88b800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.634 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b800bd30 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8000c90 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f88b80022a0 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88c92c3700 1 -- 172.21.2.117:0/275153882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c4187f10 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88c92c3700 1 -- 172.21.2.117:0/275153882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c41884a0 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f88b8005310 con 0x7f88c40a97e0 2021-02-10T11:49:29.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88abfff700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 0x7f88a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:29.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.635 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f88b8032540 con 0x7f88c40a97e0 2021-02-10T11:49:29.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.636 7f88c307d700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 0x7f88a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:29.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.636 7f88c307d700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 0x7f88a401abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:29.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.666 7f88c92c3700 1 -- 172.21.2.117:0/275153882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c41899b0 con 0x7f88c40a97e0 2021-02-10T11:49:29.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.669 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f88b8000960 con 0x7f88c40a97e0 2021-02-10T11:49:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.792 7f88c92c3700 1 -- 172.21.2.117:0/275153882 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c41899b0 con 0x7f88c40a97e0 2021-02-10T11:49:29.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.793 7f88abfff700 1 -- 172.21.2.117:0/275153882 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+14252 (secure 0 0 0) 0x7f88b8019ec0 con 0x7f88c40a97e0 2021-02-10T11:49:29.789 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:29.789 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":108,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:29.188240","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:27.162612","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:28.233491","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 msgr2=0x7f88a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 0x7f88a401abc0 crc :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 msgr2=0x7f88c4189ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f88b8006f00 tx=0x7f88b800b940).stop 2021-02-10T11:49:29.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 shutdown_connections 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88a40186c0 0x7f88a401abc0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88c40a97e0 0x7f88c4189ee0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88c418a500 0x7f88c418a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 --2- 172.21.2.117:0/275153882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88c419a980 0x7f88c40a91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 >> 172.21.2.117:0/275153882 conn(0x7f88c410de90 msgr2=0x7f88c419e470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 shutdown_connections 2021-02-10T11:49:29.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:29.795 7f88c92c3700 1 -- 172.21.2.117:0/275153882 wait complete. 2021-02-10T11:49:29.799 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:30.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.213 7f930ab97700 1 Processor -- start 2021-02-10T11:49:30.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.213 7f930ab97700 1 -- start start 2021-02-10T11:49:30.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f930ab97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 0x7f93040eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f930ab97700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 0x7f93041b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f930ab97700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93040ede60 con 0x7f93040d18e0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f930ab97700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93040ef7f0 con 0x7f93040eb7f0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f930ab97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93040d1ce0 con 0x7f93040ef3f0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f93089b3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f93040d18e0 0x7f93040ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54256/0 (socket says 172.21.2.117:54256) 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f93089b3700 1 -- 172.21.2.117:0/2618387505 learned_addr learned my addr 172.21.2.117:0/2618387505 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f9303fff700 1 --2- 172.21.2.117:0/2618387505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 0x7f93040eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.214 7f93091b4700 1 --2- 172.21.2.117:0/2618387505 >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 0x7f93041b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93037fe700 1 -- 172.21.2.117:0/2618387505 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285827023 0 0) 0x7f92ec000ce0 con 0x7f93040d18e0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93037fe700 1 -- 172.21.2.117:0/2618387505 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0002fe0 con 0x7f93040d18e0 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f9303fff700 1 -- 172.21.2.117:0/2618387505 >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 msgr2=0x7f93041b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:30.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f9303fff700 1 --2- 172.21.2.117:0/2618387505 >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 0x7f93041b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f9303fff700 1 -- 172.21.2.117:0/2618387505 >> v1:172.21.2.117:6789/0 conn(0x7f93040d18e0 legacy=0x7f93040ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f9303fff700 1 -- 172.21.2.117:0/2618387505 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93041b8f90 con 0x7f93040ef3f0 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93091b4700 1 --2- 172.21.2.117:0/2618387505 >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 0x7f93041b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f9303fff700 1 --2- 172.21.2.117:0/2618387505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 0x7f93040eb1d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f92f4006650 tx=0x7f92f4009a80).ready entity=mon.2 client_cookie=d14f6c3fa55d300b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93037fe700 1 -- 172.21.2.117:0/2618387505 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92f400c0c0 con 0x7f93040ef3f0 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93037fe700 1 -- 172.21.2.117:0/2618387505 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f400c3c0 con 0x7f93040ef3f0 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f930ab97700 1 -- 172.21.2.117:0/2618387505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 msgr2=0x7f93040eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f930ab97700 1 --2- 172.21.2.117:0/2618387505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 0x7f93040eb1d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f92f4006650 tx=0x7f92f4009a80).stop 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.215 7f93037fe700 1 -- 172.21.2.117:0/2618387505 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92f400c820 con 0x7f93040ef3f0 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 -- 172.21.2.117:0/2618387505 shutdown_connections 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 --2- 172.21.2.117:0/2618387505 >> v2:172.21.2.117:3300/0 conn(0x7f93040eb7f0 0x7f93041b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 --2- 172.21.2.117:0/2618387505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93040ef3f0 0x7f93040eb1d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 -- 172.21.2.117:0/2618387505 >> 172.21.2.117:0/2618387505 conn(0x7f93040ced60 msgr2=0x7f93040cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:30.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 -- 172.21.2.117:0/2618387505 shutdown_connections 2021-02-10T11:49:30.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.216 7f930ab97700 1 -- 172.21.2.117:0/2618387505 wait complete. 2021-02-10T11:49:30.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.230 7f930ab97700 1 Processor -- start 2021-02-10T11:49:30.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.230 7f930ab97700 1 -- start start 2021-02-10T11:49:30.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93040b43f0 0x7f93040e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93041b0130 0x7f93040e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040e80e0 con 0x7f93041b0130 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93040e7e20 con 0x7f93040b43f0 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f930ab97700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9304096a70 con 0x7f93041afd30 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f93089b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f93091b4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93041b0130 0x7f93040e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f93089b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51694/0 (socket says 172.21.2.117:51694) 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f9303fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93040b43f0 0x7f93040e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.231 7f93089b3700 1 -- 172.21.2.117:0/1794788521 learned_addr learned my addr 172.21.2.117:0/1794788521 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:30.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 -- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93040b43f0 msgr2=0x7f93040e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93040b43f0 0x7f93040e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 -- 172.21.2.117:0/1794788521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93041b0130 msgr2=0x7f93040e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93041b0130 0x7f93040e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 -- 172.21.2.117:0/1794788521 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93040e8660 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93089b3700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f92ec00b5c0 tx=0x7f92ec00b960).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ec006020 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec006260 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f930ab97700 1 -- 172.21.2.117:0/1794788521 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93040e8660 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ec004620 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f930ab97700 1 -- 172.21.2.117:0/1794788521 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93040e8660 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92ec00a000 con 0x7f93041afd30 2021-02-10T11:49:30.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 0x7f92e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.232 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f92ec032760 con 0x7f93041afd30 2021-02-10T11:49:30.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.233 7f9303fff700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 0x7f92e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.233 7f9303fff700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 0x7f92e801abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.263 7f930ab97700 1 -- 172.21.2.117:0/1794788521 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93040e7e20 con 0x7f93041afd30 2021-02-10T11:49:30.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.267 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92ec006b80 con 0x7f93041afd30 2021-02-10T11:49:30.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.396 7f930ab97700 1 -- 172.21.2.117:0/1794788521 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93040e7e20 con 0x7f93041afd30 2021-02-10T11:49:30.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.397 7f93017fa700 1 -- 172.21.2.117:0/1794788521 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f92ec00a000 con 0x7f93041afd30 2021-02-10T11:49:30.395 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:30.395 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":109,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:30.190724","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:27.162612","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:28.233491","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:30.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.400 7f930ab97700 1 -- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 msgr2=0x7f92e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.400 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 0x7f92e801abc0 crc :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.400 7f930ab97700 1 -- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 msgr2=0x7f93040b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.400 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f92ec00b5c0 tx=0x7f92ec00b960).stop 2021-02-10T11:49:30.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 -- 172.21.2.117:0/1794788521 shutdown_connections 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92e8018730 0x7f92e801abc0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f93041b0130 0x7f93040e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f93040b43f0 0x7f93040e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 --2- 172.21.2.117:0/1794788521 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f93041afd30 0x7f93040b3dd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 -- 172.21.2.117:0/1794788521 >> 172.21.2.117:0/1794788521 conn(0x7f93041a3620 msgr2=0x7f93040dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 -- 172.21.2.117:0/1794788521 shutdown_connections 2021-02-10T11:49:30.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.401 7f930ab97700 1 -- 172.21.2.117:0/1794788521 wait complete. 2021-02-10T11:49:30.405 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:49:30.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 Processor -- start 2021-02-10T11:49:30.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 -- start start 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 0x7f02ac0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f02ac0eb040 0x7f02ac0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0ede60 con 0x7f02ac0ee3e0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac0eba60 con 0x7f02ac0eb040 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02b1f82700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0d1cf0 con 0x7f02ac0d18f0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.825 7f02abfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f02ac0ee3e0 0x7f02ac0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54270/0 (socket says 172.21.2.117:54270) 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02abfff700 1 -- 172.21.2.117:0/1480773218 learned_addr learned my addr 172.21.2.117:0/1480773218 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02ab7fe700 1 --2- 172.21.2.117:0/1480773218 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 0x7f02ac0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02aaffd700 1 --2- 172.21.2.117:0/1480773218 >> v2:172.21.2.125:3300/0 conn(0x7f02ac0eb040 0x7f02ac0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02aa7fc700 1 -- 172.21.2.117:0/1480773218 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329071408 0 0) 0x7f02a4001500 con 0x7f02ac0ee3e0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02aa7fc700 1 -- 172.21.2.117:0/1480773218 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294002fe0 con 0x7f02ac0ee3e0 2021-02-10T11:49:30.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02ab7fe700 1 -- 172.21.2.117:0/1480773218 >> v2:172.21.2.125:3300/0 conn(0x7f02ac0eb040 msgr2=0x7f02ac0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02ab7fe700 1 --2- 172.21.2.117:0/1480773218 >> v2:172.21.2.125:3300/0 conn(0x7f02ac0eb040 0x7f02ac0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02ab7fe700 1 -- 172.21.2.117:0/1480773218 >> v1:172.21.2.117:6789/0 conn(0x7f02ac0ee3e0 legacy=0x7f02ac0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.826 7f02ab7fe700 1 -- 172.21.2.117:0/1480773218 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0eefd0 con 0x7f02ac0d18f0 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02ab7fe700 1 --2- 172.21.2.117:0/1480773218 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 0x7f02ac0eaa20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f029c006650 tx=0x7f029c009a80).ready entity=mon.2 client_cookie=794a32a6075f5aa4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02aa7fc700 1 -- 172.21.2.117:0/1480773218 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f029c0009b0 con 0x7f02ac0d18f0 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02aa7fc700 1 -- 172.21.2.117:0/1480773218 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c00c0c0 con 0x7f02ac0d18f0 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02aa7fc700 1 -- 172.21.2.117:0/1480773218 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f029c00c520 con 0x7f02ac0d18f0 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02b1f82700 1 -- 172.21.2.117:0/1480773218 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 msgr2=0x7f02ac0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.827 7f02b1f82700 1 --2- 172.21.2.117:0/1480773218 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 0x7f02ac0eaa20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f029c006650 tx=0x7f029c009a80).stop 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 -- 172.21.2.117:0/1480773218 shutdown_connections 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 --2- 172.21.2.117:0/1480773218 >> v2:172.21.2.125:3300/0 conn(0x7f02ac0eb040 0x7f02ac0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 --2- 172.21.2.117:0/1480773218 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0d18f0 0x7f02ac0eaa20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:30.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 -- 172.21.2.117:0/1480773218 >> 172.21.2.117:0/1480773218 conn(0x7f02ac0cf060 msgr2=0x7f02ac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:30.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 -- 172.21.2.117:0/1480773218 shutdown_connections 2021-02-10T11:49:30.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.828 7f02b1f82700 1 -- 172.21.2.117:0/1480773218 wait complete. 2021-02-10T11:49:30.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.842 7f02b1f82700 1 Processor -- start 2021-02-10T11:49:30.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.842 7f02b1f82700 1 -- start start 2021-02-10T11:49:30.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02ac0a6ce0 0x7f02ac0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0a2040 0x7f02ac0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0a67e0 con 0x7f02ac085ec0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0a7120 con 0x7f02ac0a6ce0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0a2a60 con 0x7f02ac0a2040 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54028/0 (socket says 172.21.2.117:54028) 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 -- 172.21.2.117:0/3713129911 learned_addr learned my addr 172.21.2.117:0/3713129911 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 -- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0a2040 msgr2=0x7f02ac0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0a2040 0x7f02ac0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 -- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02ac0a6ce0 msgr2=0x7f02ac0dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02ac0a6ce0 0x7f02ac0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 -- 172.21.2.117:0/3713129911 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac1a2dc0 con 0x7f02ac085ec0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02aaffd700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f02a0006f60 tx=0x7f02a000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a000bd60 con 0x7f02ac085ec0 2021-02-10T11:49:30.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0000c90 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f02a00022b0 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.843 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac1a2dc0 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.844 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac1a3350 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.844 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02a0006b00 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.844 7f0293fff700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 0x7f028c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.844 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f02a0032680 con 0x7f02ac085ec0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.845 7f02ab7fe700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 0x7f028c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:30.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.845 7f02ab7fe700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 0x7f028c01abc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:30.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.875 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac0daff0 con 0x7f02ac085ec0 2021-02-10T11:49:30.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:30.878 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f02a002e390 con 0x7f02ac085ec0 2021-02-10T11:49:31.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.015 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f02ac0ede60 con 0x7f02ac085ec0 2021-02-10T11:49:31.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.202 7f0293fff700 1 -- 172.21.2.117:0/3713129911 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v86) v1 ==== 118+0+0 (secure 0 0 0) 0x7f02a002e390 con 0x7f02ac085ec0 2021-02-10T11:49:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 msgr2=0x7f028c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 0x7f028c01abc0 crc :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 msgr2=0x7f02ac0862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f02a0006f60 tx=0x7f02a000b970).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 shutdown_connections 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f028c0186c0 0x7f028c01abc0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f02ac085ec0 0x7f02ac0862c0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02ac0a2040 0x7f02ac0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 --2- 172.21.2.117:0/3713129911 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f02ac0a6ce0 0x7f02ac0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 >> 172.21.2.117:0/3713129911 conn(0x7f02ac0b6540 msgr2=0x7f02ac08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 shutdown_connections 2021-02-10T11:49:31.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.234 7f02b1f82700 1 -- 172.21.2.117:0/3713129911 wait complete. 2021-02-10T11:49:31.238 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.654 7f24502cc700 1 Processor -- start 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.654 7f24502cc700 1 -- start start 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f24502cc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f24502cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 0x7f24480eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f24502cc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24480ede60 con 0x7f24480ee3e0 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f24502cc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480eba90 con 0x7f24480d18e0 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f24502cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480d1ce0 con 0x7f24480eb000 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244e0e8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244e0e8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54032/0 (socket says 172.21.2.117:54032) 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244e0e8700 1 -- 172.21.2.117:0/68388667 learned_addr learned my addr 172.21.2.117:0/68388667 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244d8e7700 1 --2- 172.21.2.117:0/68388667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 0x7f24480eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244d8e7700 1 -- 172.21.2.117:0/68388667 >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 msgr2=0x7f24480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244d8e7700 1 --2- 172.21.2.117:0/68388667 >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.655 7f244d8e7700 1 -- 172.21.2.117:0/68388667 >> v1:172.21.2.125:6789/0 conn(0x7f24480ee3e0 legacy=0x7f24480ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244d8e7700 1 -- 172.21.2.117:0/68388667 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480eefd0 con 0x7f24480eb000 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244e0e8700 1 --2- 172.21.2.117:0/68388667 >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244d8e7700 1 --2- 172.21.2.117:0/68388667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 0x7f24480eb400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f2438006650 tx=0x7f2438009a80).ready entity=mon.2 client_cookie=f57e5fcba190a146 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244d0e6700 1 -- 172.21.2.117:0/68388667 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24380009b0 con 0x7f24480eb000 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244d0e6700 1 -- 172.21.2.117:0/68388667 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243800c0c0 con 0x7f24480eb000 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f244d0e6700 1 -- 172.21.2.117:0/68388667 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f243800c510 con 0x7f24480eb000 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f24502cc700 1 -- 172.21.2.117:0/68388667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 msgr2=0x7f24480eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.656 7f24502cc700 1 --2- 172.21.2.117:0/68388667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 0x7f24480eb400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f2438006650 tx=0x7f2438009a80).stop 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 -- 172.21.2.117:0/68388667 shutdown_connections 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 --2- 172.21.2.117:0/68388667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24480eb000 0x7f24480eb400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 --2- 172.21.2.117:0/68388667 >> v2:172.21.2.117:3300/0 conn(0x7f24480d18e0 0x7f24480ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 -- 172.21.2.117:0/68388667 >> 172.21.2.117:0/68388667 conn(0x7f24480ced60 msgr2=0x7f24480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 -- 172.21.2.117:0/68388667 shutdown_connections 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.657 7f24502cc700 1 -- 172.21.2.117:0/68388667 wait complete. 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 Processor -- start 2021-02-10T11:49:31.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 -- start start 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24480b82f0 0x7f24480b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24480b8d10 0x7f24480b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480c0670 con 0x7f24480b8d10 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480deb20 con 0x7f24480b82f0 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f24502cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480ded40 con 0x7f24481a33d0 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f244e0e8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f244e0e8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51716/0 (socket says 172.21.2.117:51716) 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f244e0e8700 1 -- 172.21.2.117:0/2067116094 learned_addr learned my addr 172.21.2.117:0/2067116094 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f244e8e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24480b8d10 0x7f24480b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.672 7f244d8e7700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24480b82f0 0x7f24480b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 -- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24480b82f0 msgr2=0x7f24480b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24480b82f0 0x7f24480b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 -- 172.21.2.117:0/2067116094 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24480b8d10 msgr2=0x7f24480b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24480b8d10 0x7f24480b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 -- 172.21.2.117:0/2067116094 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480def60 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f244e0e8700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f244000b910 tx=0x7f244000bcb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f244000c2f0 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244000c5b0 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f244000ca00 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f24502cc700 1 -- 172.21.2.117:0/2067116094 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24480def60 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f24502cc700 1 -- 172.21.2.117:0/2067116094 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24480df4f0 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.673 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2440009fe0 con 0x7f24481a33d0 2021-02-10T11:49:31.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.674 7f243effd700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 0x7f243001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:31.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.674 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2440032a00 con 0x7f24481a33d0 2021-02-10T11:49:31.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.674 7f244d8e7700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 0x7f243001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:31.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.674 7f244d8e7700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 0x7f243001abc0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:31.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.704 7f24502cc700 1 -- 172.21.2.117:0/2067116094 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2448087290 con 0x7f24481a33d0 2021-02-10T11:49:31.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.708 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2440000cc0 con 0x7f24481a33d0 2021-02-10T11:49:31.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.840 7f24502cc700 1 -- 172.21.2.117:0/2067116094 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24480ede60 con 0x7f24481a33d0 2021-02-10T11:49:31.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.841 7f243effd700 1 -- 172.21.2.117:0/2067116094 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f2440009fe0 con 0x7f24481a33d0 2021-02-10T11:49:31.839 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:31.839 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":86,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:49:31.199085","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1582108166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1582108166},{"type":"v1","addr":"172.21.2.125:6835","nonce":1582108166}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:49:31.839 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 86 2021-02-10T11:49:31.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 msgr2=0x7f243001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 0x7f243001abc0 crc :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 msgr2=0x7f24481a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f244000b910 tx=0x7f244000bcb0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 shutdown_connections 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24300186c0 0x7f243001abc0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24480b8d10 0x7f24480b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24480b82f0 0x7f24480b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 --2- 172.21.2.117:0/2067116094 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24481a33d0 0x7f24481a37d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 >> 172.21.2.117:0/2067116094 conn(0x7f24480874c0 msgr2=0x7f2448096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 shutdown_connections 2021-02-10T11:49:31.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:31.845 7f24502cc700 1 -- 172.21.2.117:0/2067116094 wait complete. 2021-02-10T11:49:31.849 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 Processor -- start 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 -- start start 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 0x7f75601b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600ede60 con 0x7f75600d18e0 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75600eb9c0 con 0x7f75600ef3f0 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.266 7f75670e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75600d1ce0 con 0x7f75601b6960 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f7564efc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f75600d18e0 0x7f75600ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54294/0 (socket says 172.21.2.117:54294) 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f7564efc700 1 -- 172.21.2.117:0/3828616588 learned_addr learned my addr 172.21.2.117:0/3828616588 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:32.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75656fd700 1 --2- 172.21.2.117:0/3828616588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 0x7f75601b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75577fe700 1 -- 172.21.2.117:0/3828616588 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7238945 0 0) 0x7f7550000ce0 con 0x7f75600d18e0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75577fe700 1 -- 172.21.2.117:0/3828616588 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548002fe0 con 0x7f75600d18e0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75577fe700 1 -- 172.21.2.117:0/3828616588 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987946392 0 0) 0x7f755c000ff0 con 0x7f75600ef3f0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75577fe700 1 -- 172.21.2.117:0/3828616588 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548003210 con 0x7f75600ef3f0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75656fd700 1 -- 172.21.2.117:0/3828616588 >> v1:172.21.2.125:6789/0 conn(0x7f75600ef3f0 legacy=0x7f75600eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75656fd700 1 -- 172.21.2.117:0/3828616588 >> v1:172.21.2.117:6789/0 conn(0x7f75600d18e0 legacy=0x7f75600ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75656fd700 1 -- 172.21.2.117:0/3828616588 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75601bb3b0 con 0x7f75601b6960 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.267 7f75656fd700 1 --2- 172.21.2.117:0/3828616588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 0x7f75601b8d70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7558006e50 tx=0x7f7558009e70).ready entity=mon.2 client_cookie=46195a28bc7817b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75577fe700 1 -- 172.21.2.117:0/3828616588 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7558006a20 con 0x7f75601b6960 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75577fe700 1 -- 172.21.2.117:0/3828616588 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558006790 con 0x7f75601b6960 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75577fe700 1 -- 172.21.2.117:0/3828616588 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7558000f90 con 0x7f75601b6960 2021-02-10T11:49:32.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 -- 172.21.2.117:0/3828616588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 msgr2=0x7f75601b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 --2- 172.21.2.117:0/3828616588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 0x7f75601b8d70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7558006e50 tx=0x7f7558009e70).stop 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 -- 172.21.2.117:0/3828616588 shutdown_connections 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 --2- 172.21.2.117:0/3828616588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b6960 0x7f75601b8d70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 -- 172.21.2.117:0/3828616588 >> 172.21.2.117:0/3828616588 conn(0x7f75600ced60 msgr2=0x7f75600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 -- 172.21.2.117:0/3828616588 shutdown_connections 2021-02-10T11:49:32.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.268 7f75670e0700 1 -- 172.21.2.117:0/3828616588 wait complete. 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 Processor -- start 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 -- start start 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 0x7f75601b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7560098ba0 0x7f7560098fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560098000 con 0x7f7560098ba0 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75600911a0 con 0x7f75601b82b0 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75670e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7560097cf0 con 0x7f75601b8cd0 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f7557fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f75656fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7560098ba0 0x7f7560098fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f7557fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51730/0 (socket says 172.21.2.117:51730) 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.283 7f7557fff700 1 -- 172.21.2.117:0/532042368 learned_addr learned my addr 172.21.2.117:0/532042368 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7564efc700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 0x7f75601b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 -- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 msgr2=0x7f75601b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 0x7f75601b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 -- 172.21.2.117:0/532042368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7560098ba0 msgr2=0x7f7560098fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7560098ba0 0x7f7560098fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 -- 172.21.2.117:0/532042368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7560090d40 con 0x7f75601b8cd0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7564efc700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 0x7f75601b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f7557fff700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f755c006f00 tx=0x7f755c00b940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f755c000940 con 0x7f75601b8cd0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755c006340 con 0x7f75601b8cd0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f755c002030 con 0x7f75601b8cd0 2021-02-10T11:49:32.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f75670e0700 1 -- 172.21.2.117:0/532042368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560090d40 con 0x7f75601b8cd0 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.284 7f75670e0700 1 -- 172.21.2.117:0/532042368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75600998b0 con 0x7f75601b8cd0 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.285 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f755c00a030 con 0x7f75601b8cd0 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.285 7f75557fa700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 0x7f754001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.285 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f755c032700 con 0x7f75601b8cd0 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.286 7f7564efc700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 0x7f754001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.286 7f7564efc700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 0x7f754001abc0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.316 7f75670e0700 1 -- 172.21.2.117:0/532042368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560087290 con 0x7f75601b8cd0 2021-02-10T11:49:32.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.321 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f755c000c50 con 0x7f75601b8cd0 2021-02-10T11:49:32.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.446 7f75670e0700 1 -- 172.21.2.117:0/532042368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75600ede60 con 0x7f75601b8cd0 2021-02-10T11:49:32.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.447 7f75557fa700 1 -- 172.21.2.117:0/532042368 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f755c00a030 con 0x7f75601b8cd0 2021-02-10T11:49:32.445 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:32.445 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":109,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:30.190724","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:27.162612","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:28.233491","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.451 7f75670e0700 1 -- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 msgr2=0x7f754001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.451 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 0x7f754001abc0 crc :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.451 7f75670e0700 1 -- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 msgr2=0x7f7560098580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.451 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f755c006f00 tx=0x7f755c00b940).stop 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 -- 172.21.2.117:0/532042368 shutdown_connections 2021-02-10T11:49:32.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f75400186c0 0x7f754001abc0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7560098ba0 0x7f7560098fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75601b8cd0 0x7f7560098580 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 --2- 172.21.2.117:0/532042368 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f75601b82b0 0x7f75601b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 -- 172.21.2.117:0/532042368 >> 172.21.2.117:0/532042368 conn(0x7f75600874c0 msgr2=0x7f75600df850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 -- 172.21.2.117:0/532042368 shutdown_connections 2021-02-10T11:49:32.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.452 7f75670e0700 1 -- 172.21.2.117:0/532042368 wait complete. 2021-02-10T11:49:32.456 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:32.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.884 7f18ab3f8700 1 Processor -- start 2021-02-10T11:49:32.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.884 7f18ab3f8700 1 -- start start 2021-02-10T11:49:32.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18ab3f8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18a4079ce0 0x7f18a41b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18ab3f8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f18a407a0e0 0x7f18a41b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18ab3f8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a40e5770 con 0x7f18a407ab40 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18ab3f8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a407a640 con 0x7f18a4079ce0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18ab3f8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a40d31d0 con 0x7f18a407a0e0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18a4079ce0 0x7f18a41b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18a4079ce0 0x7f18a41b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54060/0 (socket says 172.21.2.117:54060) 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 -- 172.21.2.117:0/1753549250 learned_addr learned my addr 172.21.2.117:0/1753549250 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a9a15700 1 --2- 172.21.2.117:0/1753549250 >> v2:172.21.2.125:3300/0 conn(0x7f18a407a0e0 0x7f18a41b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 -- 172.21.2.117:0/1753549250 >> v2:172.21.2.125:3300/0 conn(0x7f18a407a0e0 msgr2=0x7f18a41b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 --2- 172.21.2.117:0/1753549250 >> v2:172.21.2.125:3300/0 conn(0x7f18a407a0e0 0x7f18a41b8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 -- 172.21.2.117:0/1753549250 >> v1:172.21.2.125:6789/0 conn(0x7f18a407ab40 legacy=0x7f18a40794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 -- 172.21.2.117:0/1753549250 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a40d1900 con 0x7f18a4079ce0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.885 7f18a8a13700 1 --2- 172.21.2.117:0/1753549250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a4079ce0 0x7f18a41b59d0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f1894006700 tx=0x7f1894009b80).ready entity=mon.0 client_cookie=a7ad3a4b9cea3582 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f189bfff700 1 -- 172.21.2.117:0/1753549250 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f189400c0c0 con 0x7f18a4079ce0 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f189bfff700 1 -- 172.21.2.117:0/1753549250 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189400c440 con 0x7f18a4079ce0 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f189bfff700 1 -- 172.21.2.117:0/1753549250 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f189400c8a0 con 0x7f18a4079ce0 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 -- 172.21.2.117:0/1753549250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a4079ce0 msgr2=0x7f18a41b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 --2- 172.21.2.117:0/1753549250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a4079ce0 0x7f18a41b59d0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f1894006700 tx=0x7f1894009b80).stop 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 -- 172.21.2.117:0/1753549250 shutdown_connections 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 --2- 172.21.2.117:0/1753549250 >> v2:172.21.2.125:3300/0 conn(0x7f18a407a0e0 0x7f18a41b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 --2- 172.21.2.117:0/1753549250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a4079ce0 0x7f18a41b59d0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.886 7f18ab3f8700 1 -- 172.21.2.117:0/1753549250 >> 172.21.2.117:0/1753549250 conn(0x7f18a40d0910 msgr2=0x7f18a40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.887 7f18ab3f8700 1 -- 172.21.2.117:0/1753549250 shutdown_connections 2021-02-10T11:49:32.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.887 7f18ab3f8700 1 -- 172.21.2.117:0/1753549250 wait complete. 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.901 7f18ab3f8700 1 Processor -- start 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.901 7f18ab3f8700 1 -- start start 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18a41b2030 0x7f18a40abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 0x7f18a4090310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40f8830 con 0x7f18a41009f0 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40ac1b0 con 0x7f18a41b2030 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18ab3f8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a40ac3d0 con 0x7f18a41b2b50 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54068/0 (socket says 172.21.2.117:54068) 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 -- 172.21.2.117:0/3263956599 learned_addr learned my addr 172.21.2.117:0/3263956599 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a9214700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18a41b2030 0x7f18a40abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a9a15700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 0x7f18a4090310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 -- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 msgr2=0x7f18a4090310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 0x7f18a4090310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 -- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18a41b2030 msgr2=0x7f18a40abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18a41b2030 0x7f18a40abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 -- 172.21.2.117:0/3263956599 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4090930 con 0x7f18a41009f0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a9a15700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 0x7f18a4090310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.902 7f18a8a13700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f189400b5e0 tx=0x7f189400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f189400bd70 con 0x7f18a41009f0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1894002030 con 0x7f18a41009f0 2021-02-10T11:49:32.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a4090930 con 0x7f18a41009f0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a4090ec0 con 0x7f18a41009f0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f18940024d0 con 0x7f18a41009f0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1894000c90 con 0x7f18a41009f0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 0x7f188401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.903 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1894032710 con 0x7f18a41009f0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.904 7f18a9214700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 0x7f188401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:32.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.904 7f18a9214700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 0x7f188401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:32.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.934 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a4100ea0 con 0x7f18a41009f0 2021-02-10T11:49:32.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:32.937 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f18940009b0 con 0x7f18a41009f0 2021-02-10T11:49:33.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.065 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f18a4100ea0 con 0x7f18a41009f0 2021-02-10T11:49:33.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.068 7f1899ffb700 1 -- 172.21.2.117:0/3263956599 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v63) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1894000c90 con 0x7f18a41009f0 2021-02-10T11:49:33.067 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:49:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.073 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 msgr2=0x7f188401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.073 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 0x7f188401abc0 crc :-1 s=READY pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.073 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 msgr2=0x7f18a41b2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.073 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f189400b5e0 tx=0x7f189400b980).stop 2021-02-10T11:49:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 shutdown_connections 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18840186c0 0x7f188401abc0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18a41009f0 0x7f18a41b2530 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18a41b2b50 0x7f18a4090310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 --2- 172.21.2.117:0/3263956599 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f18a41b2030 0x7f18a40abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 >> 172.21.2.117:0/3263956599 conn(0x7f18a408f550 msgr2=0x7f18a41afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 shutdown_connections 2021-02-10T11:49:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.074 7f18ab3f8700 1 -- 172.21.2.117:0/3263956599 wait complete. 2021-02-10T11:49:33.077 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 Processor -- start 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 -- start start 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f36a4079ce0 0x7f36a41b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a40e5770 con 0x7f36a407ab40 2021-02-10T11:49:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a40d18c0 con 0x7f36a4079ce0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.505 7f36a936c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a407bf50 con 0x7f36a407a0e0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a307d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f36a407ab40 0x7f36a40794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47248/0 (socket says 172.21.2.117:47248) 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51756/0 (socket says 172.21.2.117:51756) 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a307d700 1 -- 172.21.2.117:0/880028588 learned_addr learned my addr 172.21.2.117:0/880028588 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a287c700 1 --2- 172.21.2.117:0/880028588 >> v2:172.21.2.125:3300/0 conn(0x7f36a4079ce0 0x7f36a41b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 -- 172.21.2.117:0/880028588 >> v2:172.21.2.125:3300/0 conn(0x7f36a4079ce0 msgr2=0x7f36a41b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 --2- 172.21.2.117:0/880028588 >> v2:172.21.2.125:3300/0 conn(0x7f36a4079ce0 0x7f36a41b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 -- 172.21.2.117:0/880028588 >> v1:172.21.2.125:6789/0 conn(0x7f36a407ab40 legacy=0x7f36a40794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 -- 172.21.2.117:0/880028588 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40d1c60 con 0x7f36a407a0e0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.506 7f36a387e700 1 --2- 172.21.2.117:0/880028588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f369c006e50 tx=0x7f369c009e70).ready entity=mon.2 client_cookie=255689a31372dbb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a207b700 1 -- 172.21.2.117:0/880028588 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369c00b510 con 0x7f36a407a0e0 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a207b700 1 -- 172.21.2.117:0/880028588 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369c006790 con 0x7f36a407a0e0 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a207b700 1 -- 172.21.2.117:0/880028588 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369c00b940 con 0x7f36a407a0e0 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 -- 172.21.2.117:0/880028588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 msgr2=0x7f36a41b8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 --2- 172.21.2.117:0/880028588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f369c006e50 tx=0x7f369c009e70).stop 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 -- 172.21.2.117:0/880028588 shutdown_connections 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 --2- 172.21.2.117:0/880028588 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a407a0e0 0x7f36a41b8000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 --2- 172.21.2.117:0/880028588 >> v2:172.21.2.125:3300/0 conn(0x7f36a4079ce0 0x7f36a41b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 -- 172.21.2.117:0/880028588 >> 172.21.2.117:0/880028588 conn(0x7f36a40d0910 msgr2=0x7f36a40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 -- 172.21.2.117:0/880028588 shutdown_connections 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.507 7f36a936c700 1 -- 172.21.2.117:0/880028588 wait complete. 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 Processor -- start 2021-02-10T11:49:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 -- start start 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 0x7f36a41b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 0x7f36a4090310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40f8830 con 0x7f36a41b2030 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40ac1b0 con 0x7f36a41009f0 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a936c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40ac3d0 con 0x7f36a41b2b50 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54080/0 (socket says 172.21.2.117:54080) 2021-02-10T11:49:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.522 7f36a307d700 1 -- 172.21.2.117:0/45043104 learned_addr learned my addr 172.21.2.117:0/45043104 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a387e700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 0x7f36a4090310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a287c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 0x7f36a41b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 -- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 msgr2=0x7f36a4090310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 0x7f36a4090310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 -- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 msgr2=0x7f36a41b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 0x7f36a41b2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 -- 172.21.2.117:0/45043104 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a4090930 con 0x7f36a41b2030 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a387e700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 0x7f36a4090310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a287c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 0x7f36a41b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a307d700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f3698001c60 tx=0x7f369800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f369800acc0 con 0x7f36a41b2030 2021-02-10T11:49:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3698004140 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a936c700 1 -- 172.21.2.117:0/45043104 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a4090930 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3698004590 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.523 7f36a936c700 1 -- 172.21.2.117:0/45043104 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a4090ec0 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.524 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3698004320 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.524 7f368bfff700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.524 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3698032500 con 0x7f36a41b2030 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.524 7f36a287c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.525 7f36a287c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:33.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.555 7f36a936c700 1 -- 172.21.2.117:0/45043104 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a4100ea0 con 0x7f36a41b2030 2021-02-10T11:49:33.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.558 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3698000dd0 con 0x7f36a41b2030 2021-02-10T11:49:33.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.685 7f36a936c700 1 -- 172.21.2.117:0/45043104 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f36a4100ea0 con 0x7f36a41b2030 2021-02-10T11:49:33.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.689 7f368bfff700 1 -- 172.21.2.117:0/45043104 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v64) v1 ==== 184+0+0 (secure 0 0 0) 0x7f369801f670 con 0x7f36a41b2030 2021-02-10T11:49:33.688 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:49:33.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 -- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 msgr2=0x7f368401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 crc :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 -- 172.21.2.117:0/45043104 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 msgr2=0x7f36a40abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:33.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7f3698001c60 tx=0x7f369800a8d0).stop 2021-02-10T11:49:33.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 -- 172.21.2.117:0/45043104 shutdown_connections 2021-02-10T11:49:33.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36a41b2b50 0x7f36a4090310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36a41009f0 0x7f36a41b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36840186c0 0x7f368401abc0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 --2- 172.21.2.117:0/45043104 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36a41b2030 0x7f36a40abb90 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:33.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.694 7f36a936c700 1 -- 172.21.2.117:0/45043104 >> 172.21.2.117:0/45043104 conn(0x7f36a408f550 msgr2=0x7f36a41afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:33.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.695 7f36a936c700 1 -- 172.21.2.117:0/45043104 shutdown_connections 2021-02-10T11:49:33.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:33.695 7f36a936c700 1 -- 172.21.2.117:0/45043104 wait complete. 2021-02-10T11:49:33.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:34.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.126 7f6d99be8700 1 Processor -- start 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.126 7f6d99be8700 1 -- start start 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d99be8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d9407ab40 0x7f6d940794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d99be8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6d94079f10 0x7f6d9407b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d99be8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9407a5c0 con 0x7f6d94079b10 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d99be8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d940e5770 con 0x7f6d94079f10 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d99be8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9407a310 con 0x7f6d9407ab40 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d92ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6d94079b10 0x7f6d9407af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54336/0 (socket says 172.21.2.117:54336) 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d92ffd700 1 -- 172.21.2.117:0/3953739467 learned_addr learned my addr 172.21.2.117:0/3953739467 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d93fff700 1 --2- 172.21.2.117:0/3953739467 >> v2:172.21.2.125:3300/0 conn(0x7f6d94079f10 0x7f6d9407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d937fe700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d9407ab40 0x7f6d940794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d927fc700 1 -- 172.21.2.117:0/3953739467 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381364700 0 0) 0x7f6d880010b0 con 0x7f6d94079b10 2021-02-10T11:49:34.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.127 7f6d927fc700 1 -- 172.21.2.117:0/3953739467 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c002fe0 con 0x7f6d94079b10 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d93fff700 1 -- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d9407ab40 msgr2=0x7f6d940794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d93fff700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d9407ab40 0x7f6d940794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d93fff700 1 -- 172.21.2.117:0/3953739467 >> v1:172.21.2.117:6789/0 conn(0x7f6d94079b10 legacy=0x7f6d9407af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d93fff700 1 -- 172.21.2.117:0/3953739467 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d940d18c0 con 0x7f6d94079f10 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d93fff700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d94079f10 0x7f6d9407b730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f6d8c006e50 tx=0x7f6d8c009e70).ready entity=mon.1 client_cookie=3affd0c1f5c65de5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d927fc700 1 -- 172.21.2.117:0/3953739467 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d8c006a20 con 0x7f6d94079f10 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d927fc700 1 -- 172.21.2.117:0/3953739467 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c006790 con 0x7f6d94079f10 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d927fc700 1 -- 172.21.2.117:0/3953739467 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d8c00b700 con 0x7f6d94079f10 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d99be8700 1 -- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d94079f10 msgr2=0x7f6d9407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.128 7f6d99be8700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d94079f10 0x7f6d9407b730 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f6d8c006e50 tx=0x7f6d8c009e70).stop 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 -- 172.21.2.117:0/3953739467 shutdown_connections 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d94079f10 0x7f6d9407b730 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 --2- 172.21.2.117:0/3953739467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d9407ab40 0x7f6d940794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 -- 172.21.2.117:0/3953739467 >> 172.21.2.117:0/3953739467 conn(0x7f6d940d0910 msgr2=0x7f6d940d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:34.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 -- 172.21.2.117:0/3953739467 shutdown_connections 2021-02-10T11:49:34.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.129 7f6d99be8700 1 -- 172.21.2.117:0/3953739467 wait complete. 2021-02-10T11:49:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 Processor -- start 2021-02-10T11:49:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 -- start start 2021-02-10T11:49:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 0x7f6d940ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d940db5b0 0x7f6d940db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d941001d0 con 0x7f6d94094ea0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940b8ae0 con 0x7f6d940db5b0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d99be8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940dcdc0 con 0x7f6d940ab1f0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d937fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d92ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 0x7f6d940ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d937fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54088/0 (socket says 172.21.2.117:54088) 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d937fe700 1 -- 172.21.2.117:0/1625165376 learned_addr learned my addr 172.21.2.117:0/1625165376 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.144 7f6d93fff700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d940db5b0 0x7f6d940db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 -- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 msgr2=0x7f6d940ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 0x7f6d940ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 -- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d940db5b0 msgr2=0x7f6d940db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d940db5b0 0x7f6d940db9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 -- 172.21.2.117:0/1625165376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94100d70 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d92ffd700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 0x7f6d940ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d937fe700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f6d84006f60 tx=0x7f6d8400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d8400bd60 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d84000c90 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d94100d70 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d940dc2c0 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.145 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d84002220 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.146 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d840052e0 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.146 7f6d7bfff700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.146 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6d84032670 con 0x7f6d94094ea0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.146 7f6d92ffd700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.147 7f6d92ffd700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.177 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d940daa50 con 0x7f6d94094ea0 2021-02-10T11:49:34.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.179 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d84000940 con 0x7f6d94094ea0 2021-02-10T11:49:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.308 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6d940daa50 con 0x7f6d94094ea0 2021-02-10T11:49:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.311 7f6d7bfff700 1 -- 172.21.2.117:0/1625165376 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v65) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6d8401f4b0 con 0x7f6d94094ea0 2021-02-10T11:49:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:49:34.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 msgr2=0x7f6d7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 crc :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 msgr2=0x7f6d94100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7f6d84006f60 tx=0x7f6d8400b970).stop 2021-02-10T11:49:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 shutdown_connections 2021-02-10T11:49:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d940db5b0 0x7f6d940db9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d940ab1f0 0x7f6d940ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d740186c0 0x7f6d7401abc0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 --2- 172.21.2.117:0/1625165376 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d94094ea0 0x7f6d94100750 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 >> 172.21.2.117:0/1625165376 conn(0x7f6d940bacc0 msgr2=0x7f6d940b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 shutdown_connections 2021-02-10T11:49:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.313 7f6d99be8700 1 -- 172.21.2.117:0/1625165376 wait complete. 2021-02-10T11:49:34.317 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:49:34.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 Processor -- start 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 -- start start 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97080ef400 0x7f97080eb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97080eb840 0x7f97081b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97080ef800 con 0x7f97080d1900 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97080edee0 con 0x7f97080eb840 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970ca69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97080d1d00 con 0x7f97080ef400 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97080eb840 0x7f97081b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970705f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f97080eb840 0x7f97081b79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54100/0 (socket says 172.21.2.117:54100) 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.738 7f970705f700 1 -- 172.21.2.117:0/1742286343 learned_addr learned my addr 172.21.2.117:0/1742286343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970705f700 1 -- 172.21.2.117:0/1742286343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97080ef400 msgr2=0x7f97080eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970705f700 1 --2- 172.21.2.117:0/1742286343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97080ef400 0x7f97080eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970705f700 1 -- 172.21.2.117:0/1742286343 >> v1:172.21.2.125:6789/0 conn(0x7f97080d1900 legacy=0x7f97080eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970705f700 1 -- 172.21.2.117:0/1742286343 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97081b8fe0 con 0x7f97080eb840 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970705f700 1 --2- 172.21.2.117:0/1742286343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97080eb840 0x7f97081b79b0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f9700006f00 tx=0x7f9700009f10).ready entity=mon.0 client_cookie=b3004520617661b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970585c700 1 -- 172.21.2.117:0/1742286343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f970000b3b0 con 0x7f97080eb840 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970585c700 1 -- 172.21.2.117:0/1742286343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000b630 con 0x7f97080eb840 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970585c700 1 -- 172.21.2.117:0/1742286343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f970000bb50 con 0x7f97080eb840 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970ca69700 1 -- 172.21.2.117:0/1742286343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97080eb840 msgr2=0x7f97081b79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.739 7f970ca69700 1 --2- 172.21.2.117:0/1742286343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97080eb840 0x7f97081b79b0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f9700006f00 tx=0x7f9700009f10).stop 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 -- 172.21.2.117:0/1742286343 shutdown_connections 2021-02-10T11:49:34.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 --2- 172.21.2.117:0/1742286343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f97080eb840 0x7f97081b79b0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 --2- 172.21.2.117:0/1742286343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f97080ef400 0x7f97080eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 -- 172.21.2.117:0/1742286343 >> 172.21.2.117:0/1742286343 conn(0x7f97080cf070 msgr2=0x7f97080cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:34.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 -- 172.21.2.117:0/1742286343 shutdown_connections 2021-02-10T11:49:34.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.740 7f970ca69700 1 -- 172.21.2.117:0/1742286343 wait complete. 2021-02-10T11:49:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 Processor -- start 2021-02-10T11:49:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 -- start start 2021-02-10T11:49:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9708096650 0x7f970808c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f970808cc00 0x7f970808a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970808bac0 con 0x7f9708096250 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970808d040 con 0x7f9708096650 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970ca69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970808d260 con 0x7f970808cc00 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54102/0 (socket says 172.21.2.117:54102) 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.755 7f970685e700 1 -- 172.21.2.117:0/4252117833 learned_addr learned my addr 172.21.2.117:0/4252117833 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 -- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f970808cc00 msgr2=0x7f970808a2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f970808cc00 0x7f970808a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 -- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9708096650 msgr2=0x7f970808c5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9708096650 0x7f970808c5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 -- 172.21.2.117:0/4252117833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970808a8d0 con 0x7f9708096250 2021-02-10T11:49:34.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970685e700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f96f000b5b0 tx=0x7f96f000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96f000bd40 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f0006020 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96f00054a0 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970ca69700 1 -- 172.21.2.117:0/4252117833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970808a8d0 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.756 7f970ca69700 1 -- 172.21.2.117:0/4252117833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970808a8d0 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.757 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96f0006200 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.757 7f96ff7fe700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 0x7f96ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.757 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f96f00326a0 con 0x7f9708096250 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.758 7f970605d700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 0x7f96ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:34.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.758 7f970605d700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 0x7f96ec01abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:34.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.788 7f970ca69700 1 -- 172.21.2.117:0/4252117833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97080ede70 con 0x7f9708096250 2021-02-10T11:49:34.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.791 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96f0030d00 con 0x7f9708096250 2021-02-10T11:49:34.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.920 7f970ca69700 1 -- 172.21.2.117:0/4252117833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f970808ba40 con 0x7f9708096250 2021-02-10T11:49:34.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.924 7f96ff7fe700 1 -- 172.21.2.117:0/4252117833 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v66) v1 ==== 184+0+0 (secure 0 0 0) 0x7f96f001f6f0 con 0x7f9708096250 2021-02-10T11:49:34.922 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:49:34.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 -- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 msgr2=0x7f96ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 0x7f96ec01abc0 crc :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 -- 172.21.2.117:0/4252117833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 msgr2=0x7f970808bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:34.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f96f000b5b0 tx=0x7f96f000b950).stop 2021-02-10T11:49:34.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 -- 172.21.2.117:0/4252117833 shutdown_connections 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f970808cc00 0x7f970808a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9708096650 0x7f970808c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96ec0186c0 0x7f96ec01abc0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 --2- 172.21.2.117:0/4252117833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9708096250 0x7f970808bfc0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.928 7f970ca69700 1 -- 172.21.2.117:0/4252117833 >> 172.21.2.117:0/4252117833 conn(0x7f9708084ac0 msgr2=0x7f970819ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.929 7f970ca69700 1 -- 172.21.2.117:0/4252117833 shutdown_connections 2021-02-10T11:49:34.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:34.929 7f970ca69700 1 -- 172.21.2.117:0/4252117833 wait complete. 2021-02-10T11:49:34.933 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:35.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.348 7f3fc445c700 1 Processor -- start 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.348 7f3fc445c700 1 -- start start 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.349 7f3fc445c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3fbc0ef3f0 0x7f3fbc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.349 7f3fc445c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 0x7f3fbc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.349 7f3fc445c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0ede60 con 0x7f3fbc0d18e0 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.349 7f3fc445c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0ef7f0 con 0x7f3fbc0ef3f0 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.349 7f3fc445c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0d1ce0 con 0x7f3fbc0eb7f0 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc2a79700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 0x7f3fbc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc1a77700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3fbc0ef3f0 0x7f3fbc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc2278700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f3fbc0d18e0 0x7f3fbc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47286/0 (socket says 172.21.2.117:47286) 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc2278700 1 -- 172.21.2.117:0/3817605330 learned_addr learned my addr 172.21.2.117:0/3817605330 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:35.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc1a77700 1 -- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 msgr2=0x7f3fbc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc1a77700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 0x7f3fbc1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc1a77700 1 -- 172.21.2.117:0/3817605330 >> v1:172.21.2.125:6789/0 conn(0x7f3fbc0d18e0 legacy=0x7f3fbc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc1a77700 1 -- 172.21.2.117:0/3817605330 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc1b8f90 con 0x7f3fbc0ef3f0 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.350 7f3fc2a79700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 0x7f3fbc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc1a77700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0ef3f0 0x7f3fbc0eb1d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f3fb8006650 tx=0x7f3fb8009a80).ready entity=mon.1 client_cookie=f234bb207f038379 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc1276700 1 -- 172.21.2.117:0/3817605330 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb800c0c0 con 0x7f3fbc0ef3f0 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc1276700 1 -- 172.21.2.117:0/3817605330 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb800c340 con 0x7f3fbc0ef3f0 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc1276700 1 -- 172.21.2.117:0/3817605330 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb800c860 con 0x7f3fbc0ef3f0 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc445c700 1 -- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0ef3f0 msgr2=0x7f3fbc0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.351 7f3fc445c700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0ef3f0 0x7f3fbc0eb1d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f3fb8006650 tx=0x7f3fb8009a80).stop 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 -- 172.21.2.117:0/3817605330 shutdown_connections 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0eb7f0 0x7f3fbc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 --2- 172.21.2.117:0/3817605330 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0ef3f0 0x7f3fbc0eb1d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 -- 172.21.2.117:0/3817605330 >> 172.21.2.117:0/3817605330 conn(0x7f3fbc0ced60 msgr2=0x7f3fbc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:35.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 -- 172.21.2.117:0/3817605330 shutdown_connections 2021-02-10T11:49:35.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.352 7f3fc445c700 1 -- 172.21.2.117:0/3817605330 wait complete. 2021-02-10T11:49:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 Processor -- start 2021-02-10T11:49:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 -- start start 2021-02-10T11:49:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc1b24b0 0x7f3fbc1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc198250 0x7f3fbc198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc197cd0 con 0x7f3fbc091d40 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc092140 con 0x7f3fbc1b24b0 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc445c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1b2ed0 con 0x7f3fbc198250 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc2278700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc2278700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54116/0 (socket says 172.21.2.117:54116) 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.367 7f3fc2278700 1 -- 172.21.2.117:0/3585861389 learned_addr learned my addr 172.21.2.117:0/3585861389 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2a79700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc198250 0x7f3fbc198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 -- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc198250 msgr2=0x7f3fbc198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc198250 0x7f3fbc198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 -- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc1b24b0 msgr2=0x7f3fbc1b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc1b24b0 0x7f3fbc1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 -- 172.21.2.117:0/3585861389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc198c70 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc2278700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f3fb0007fa0 tx=0x7f3fb000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb00070d0 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb000ad40 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fb0004510 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc198c70 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.368 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc196c90 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.369 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fb0004320 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.369 7f3faeffd700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 0x7f3fa401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.369 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3fb00326e0 con 0x7f3fbc091d40 2021-02-10T11:49:35.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.369 7f3fc1a77700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 0x7f3fa401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:35.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.371 7f3fc1a77700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 0x7f3fa401abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:35.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.399 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc087290 con 0x7f3fbc091d40 2021-02-10T11:49:35.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.401 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fb0030d40 con 0x7f3fbc091d40 2021-02-10T11:49:35.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.531 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fbc0ede60 con 0x7f3fbc091d40 2021-02-10T11:49:35.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.531 7f3faeffd700 1 -- 172.21.2.117:0/3585861389 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f3fb0004320 con 0x7f3fbc091d40 2021-02-10T11:49:35.527 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:35.527 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":86,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:49:31.199085","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1582108166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1582108166},{"type":"v1","addr":"172.21.2.125:6835","nonce":1582108166}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:49:35.527 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 86 2021-02-10T11:49:35.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 msgr2=0x7f3fa401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 0x7f3fa401abc0 crc :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 msgr2=0x7f3fbc0e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f3fb0007fa0 tx=0x7f3fb000a950).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 shutdown_connections 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc198250 0x7f3fbc198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc1b24b0 0x7f3fbc1b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa40186c0 0x7f3fa401abc0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 --2- 172.21.2.117:0/3585861389 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc091d40 0x7f3fbc0e71e0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:35.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.533 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 >> 172.21.2.117:0/3585861389 conn(0x7f3fbc1a3620 msgr2=0x7f3fbc0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:35.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.534 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 shutdown_connections 2021-02-10T11:49:35.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:35.534 7f3fc445c700 1 -- 172.21.2.117:0/3585861389 wait complete. 2021-02-10T11:49:35.739 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:29.188327', 'modified': '2021-02-10 11:49:31.199085', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7801': {'gid': 7801, 'name': 'b', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1582108166', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1582108166}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1582108166}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:35.739 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:35.740 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:36.149 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:49:37.035 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:49:37.035 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7801, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9625942849999998, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 86, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 109, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 109, 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.598315898999999 2021-02-10T11:49:37.036 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:49:37.037 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7801, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9625942849999998, "mdsmap_epoch": 86, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "uptime": 12.598315898999999 } 2021-02-10T11:49:37.038 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:37.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.457 7f81affc5700 1 Processor -- start 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.457 7f81affc5700 1 -- start start 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81affc5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f81a80ef3f0 0x7f81a80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81affc5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 0x7f81a81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81affc5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80ede60 con 0x7f81a80d18e0 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81affc5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a80ef7f0 con 0x7f81a80ef3f0 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81affc5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a80d1ce0 con 0x7f81a80eb7f0 2021-02-10T11:49:37.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f81a80ef3f0 0x7f81a80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81adde1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f81a80d18e0 0x7f81a80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54374/0 (socket says 172.21.2.117:54374) 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81adde1700 1 -- 172.21.2.117:0/3918482085 learned_addr learned my addr 172.21.2.117:0/3918482085 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ae5e2700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 0x7f81a81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 -- 172.21.2.117:0/3918482085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 msgr2=0x7f81a81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 0x7f81a81b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 -- 172.21.2.117:0/3918482085 >> v1:172.21.2.117:6789/0 conn(0x7f81a80d18e0 legacy=0x7f81a80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 -- 172.21.2.117:0/3918482085 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a81b8f90 con 0x7f81a80ef3f0 2021-02-10T11:49:37.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ae5e2700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 0x7f81a81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.458 7f81ad5e0700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a80ef3f0 0x7f81a80eb1d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f81a00069d0 tx=0x7f81a0009e40).ready entity=mon.0 client_cookie=73a2bc10b348cec6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81acddf700 1 -- 172.21.2.117:0/3918482085 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a000b510 con 0x7f81a80ef3f0 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81acddf700 1 -- 172.21.2.117:0/3918482085 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a000b890 con 0x7f81a80ef3f0 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81acddf700 1 -- 172.21.2.117:0/3918482085 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a000bdb0 con 0x7f81a80ef3f0 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 -- 172.21.2.117:0/3918482085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a80ef3f0 msgr2=0x7f81a80eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a80ef3f0 0x7f81a80eb1d0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f81a00069d0 tx=0x7f81a0009e40).stop 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 -- 172.21.2.117:0/3918482085 shutdown_connections 2021-02-10T11:49:37.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a80eb7f0 0x7f81a81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 --2- 172.21.2.117:0/3918482085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a80ef3f0 0x7f81a80eb1d0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 -- 172.21.2.117:0/3918482085 >> 172.21.2.117:0/3918482085 conn(0x7f81a80ced60 msgr2=0x7f81a80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 -- 172.21.2.117:0/3918482085 shutdown_connections 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.459 7f81affc5700 1 -- 172.21.2.117:0/3918482085 wait complete. 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 Processor -- start 2021-02-10T11:49:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 -- start start 2021-02-10T11:49:37.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f81a8091d40 0x7f81a80e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 0x7f81a81b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a8197cd0 con 0x7f81a81b24b0 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a8092140 con 0x7f81a8091d40 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81affc5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a81b2ed0 con 0x7f81a8198250 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.474 7f81ae5e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ad5e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 0x7f81a81b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51810/0 (socket says 172.21.2.117:51810) 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 -- 172.21.2.117:0/1892620300 learned_addr learned my addr 172.21.2.117:0/1892620300 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:37.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81adde1700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f81a8091d40 0x7f81a80e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 -- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f81a8091d40 msgr2=0x7f81a80e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f81a8091d40 0x7f81a80e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 -- 172.21.2.117:0/1892620300 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 msgr2=0x7f81a81b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 0x7f81a81b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 -- 172.21.2.117:0/1892620300 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8198c70 con 0x7f81a8198250 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ad5e0700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 0x7f81a81b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81ae5e2700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f81a4006f10 tx=0x7f81a400b990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a4000c10 con 0x7f81a8198250 2021-02-10T11:49:37.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f81affc5700 1 -- 172.21.2.117:0/1892620300 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a8198c70 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4006c20 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.475 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f81a400bd80 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.476 7f81affc5700 1 -- 172.21.2.117:0/1892620300 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8198c70 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.476 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f81a4002070 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.476 7f819e7fc700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 0x7f819001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.476 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f81a4032a90 con 0x7f81a8198250 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.476 7f81adde1700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 0x7f819001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.477 7f81adde1700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 0x7f819001ac10 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:37.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.507 7f81affc5700 1 -- 172.21.2.117:0/1892620300 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a8087290 con 0x7f81a8198250 2021-02-10T11:49:37.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.512 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f81a4006ba0 con 0x7f81a8198250 2021-02-10T11:49:37.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.637 7f81affc5700 1 -- 172.21.2.117:0/1892620300 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81a80ede60 con 0x7f81a8198250 2021-02-10T11:49:37.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.639 7f819e7fc700 1 -- 172.21.2.117:0/1892620300 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f81a4000bc0 con 0x7f81a8198250 2021-02-10T11:49:37.638 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:37.638 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":109,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:30.190724","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:27.162612","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:28.233491","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 -- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 msgr2=0x7f819001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 0x7f819001ac10 crc :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 -- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 msgr2=0x7f81a8198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f81a4006f10 tx=0x7f81a400b990).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 -- 172.21.2.117:0/1892620300 shutdown_connections 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8190018730 0x7f819001ac10 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81a81b24b0 0x7f81a81b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81a8198250 0x7f81a8198650 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 --2- 172.21.2.117:0/1892620300 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f81a8091d40 0x7f81a80e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.642 7f81affc5700 1 -- 172.21.2.117:0/1892620300 >> 172.21.2.117:0/1892620300 conn(0x7f81a81a3620 msgr2=0x7f81a80dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:37.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.643 7f81affc5700 1 -- 172.21.2.117:0/1892620300 shutdown_connections 2021-02-10T11:49:37.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:37.643 7f81affc5700 1 -- 172.21.2.117:0/1892620300 wait complete. 2021-02-10T11:49:37.646 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:49:27.162612', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:49:28.233491', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:49:37.647 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 Processor -- start 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 -- start start 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17840d18f0 0x7f17840eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17840ef3f0 0x7f17841b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17840ede60 con 0x7f17840eb040 2021-02-10T11:49:38.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1784088020 con 0x7f17840ef3f0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1789c95700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17840d1d30 con 0x7f17840d18f0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1783fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17840ef3f0 0x7f17841b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1783fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17840ef3f0 0x7f17841b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54140/0 (socket says 172.21.2.117:54140) 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1783fff700 1 -- 172.21.2.117:0/3033527869 learned_addr learned my addr 172.21.2.117:0/3033527869 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f17837fe700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17840d18f0 0x7f17840eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1783fff700 1 -- 172.21.2.117:0/3033527869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17840d18f0 msgr2=0x7f17840eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.072 7f1783fff700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17840d18f0 0x7f17840eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1783fff700 1 -- 172.21.2.117:0/3033527869 >> v1:172.21.2.125:6789/0 conn(0x7f17840eb040 legacy=0x7f17840eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1783fff700 1 -- 172.21.2.117:0/3033527869 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17841b8fd0 con 0x7f17840ef3f0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1783fff700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17840ef3f0 0x7f17841b79a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f177c006f00 tx=0x7f177c009f10).ready entity=mon.0 client_cookie=f3ddd29600e56da4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1782ffd700 1 -- 172.21.2.117:0/3033527869 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177c00b3b0 con 0x7f17840ef3f0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1782ffd700 1 -- 172.21.2.117:0/3033527869 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c00b630 con 0x7f17840ef3f0 2021-02-10T11:49:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1782ffd700 1 -- 172.21.2.117:0/3033527869 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177c00bb50 con 0x7f17840ef3f0 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 -- 172.21.2.117:0/3033527869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17840ef3f0 msgr2=0x7f17841b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17840ef3f0 0x7f17841b79a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f177c006f00 tx=0x7f177c009f10).stop 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 -- 172.21.2.117:0/3033527869 shutdown_connections 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17840ef3f0 0x7f17841b79a0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 --2- 172.21.2.117:0/3033527869 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17840d18f0 0x7f17840eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.073 7f1789c95700 1 -- 172.21.2.117:0/3033527869 >> 172.21.2.117:0/3033527869 conn(0x7f17840cf060 msgr2=0x7f17840cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.074 7f1789c95700 1 -- 172.21.2.117:0/3033527869 shutdown_connections 2021-02-10T11:49:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.074 7f1789c95700 1 -- 172.21.2.117:0/3033527869 wait complete. 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.088 7f1789c95700 1 Processor -- start 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.088 7f1789c95700 1 -- start start 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17841a4240 0x7f17841a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f178409f790 0x7f17840b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178409fb90 con 0x7f178409ed70 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840dc640 con 0x7f178409f790 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1789c95700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1784097270 con 0x7f17841a4240 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54146/0 (socket says 172.21.2.117:54146) 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 -- 172.21.2.117:0/1141814921 learned_addr learned my addr 172.21.2.117:0/1141814921 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f17837fe700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17841a4240 0x7f17841a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f1783fff700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f178409f790 0x7f17840b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 -- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17841a4240 msgr2=0x7f17841a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17841a4240 0x7f17841a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 -- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f178409f790 msgr2=0x7f17840b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f178409f790 0x7f17840b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.089 7f177affd700 1 -- 172.21.2.117:0/1141814921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17840b8780 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f177affd700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f176c00b5b0 tx=0x7f176c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176c00bd40 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c000e20 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176c0054a0 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f1789c95700 1 -- 172.21.2.117:0/1141814921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17840b8780 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.090 7f1789c95700 1 -- 172.21.2.117:0/1141814921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17840b8d10 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.091 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f176c006030 con 0x7f178409ed70 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.091 7f1780ff9700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 0x7f176801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.091 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f176c0326c0 con 0x7f178409ed70 2021-02-10T11:49:38.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.091 7f17837fe700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 0x7f176801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.091 7f17837fe700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 0x7f176801abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.122 7f1789c95700 1 -- 172.21.2.117:0/1141814921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17840daff0 con 0x7f178409ed70 2021-02-10T11:49:38.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.124 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f176c0009b0 con 0x7f178409ed70 2021-02-10T11:49:38.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.252 7f1789c95700 1 -- 172.21.2.117:0/1141814921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f17840ede60 con 0x7f178409ed70 2021-02-10T11:49:38.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.252 7f1780ff9700 1 -- 172.21.2.117:0/1141814921 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v109) v1 ==== 92+0+10 (secure 0 0 0) 0x7f176c006030 con 0x7f178409ed70 2021-02-10T11:49:38.249 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.254 7f1789c95700 1 -- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 msgr2=0x7f176801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.254 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 0x7f176801abc0 crc :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.254 7f1789c95700 1 -- 172.21.2.117:0/1141814921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 msgr2=0x7f178409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.254 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7f176c00b5b0 tx=0x7f176c00b950).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 -- 172.21.2.117:0/1141814921 shutdown_connections 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17680186c0 0x7f176801abc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f178409ed70 0x7f178409f170 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f178409f790 0x7f17840b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 --2- 172.21.2.117:0/1141814921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17841a4240 0x7f17841a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 -- 172.21.2.117:0/1141814921 >> 172.21.2.117:0/1141814921 conn(0x7f1784087090 msgr2=0x7f178408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:38.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 -- 172.21.2.117:0/1141814921 shutdown_connections 2021-02-10T11:49:38.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.255 7f1789c95700 1 -- 172.21.2.117:0/1141814921 wait complete. 2021-02-10T11:49:38.260 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:49:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 Processor -- start 2021-02-10T11:49:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 -- start start 2021-02-10T11:49:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e980e77f0 con 0x7f7e980e8790 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e980e92f0 con 0x7f7e980e7d70 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.679 7f7e9d289700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980e5ce0 con 0x7f7e981b07f0 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.680 7f7e9687c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.680 7f7e9707d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.680 7f7e9707d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51832/0 (socket says 172.21.2.117:51832) 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.680 7f7e9687c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:58536/0 (socket says 172.21.2.117:58536) 2021-02-10T11:49:38.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.680 7f7e9687c700 1 -- 172.21.2.117:0/949354842 learned_addr learned my addr 172.21.2.117:0/949354842 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9707d700 1 -- 172.21.2.117:0/949354842 >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 msgr2=0x7f7e980e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9707d700 1 --2- 172.21.2.117:0/949354842 >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9707d700 1 -- 172.21.2.117:0/949354842 >> v1:172.21.2.125:6789/0 conn(0x7f7e980e8790 legacy=0x7f7e980e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9607b700 1 -- 172.21.2.117:0/949354842 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511973072 0 0) 0x7f7e90001400 con 0x7f7e980e8790 2021-02-10T11:49:38.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9707d700 1 -- 172.21.2.117:0/949354842 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980e54b0 con 0x7f7e981b07f0 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9687c700 1 --2- 172.21.2.117:0/949354842 >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.681 7f7e9707d700 1 --2- 172.21.2.117:0/949354842 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7e88006650 tx=0x7f7e88009a80).ready entity=mon.2 client_cookie=cdba10452e1f6675 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9607b700 1 -- 172.21.2.117:0/949354842 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e880009b0 con 0x7f7e981b07f0 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9607b700 1 -- 172.21.2.117:0/949354842 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8800c0c0 con 0x7f7e981b07f0 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 -- 172.21.2.117:0/949354842 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 msgr2=0x7f7e980e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 --2- 172.21.2.117:0/949354842 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7e88006650 tx=0x7f7e88009a80).stop 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9607b700 1 -- 172.21.2.117:0/949354842 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e880009b0 con 0x7f7e981b07f0 2021-02-10T11:49:38.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 -- 172.21.2.117:0/949354842 shutdown_connections 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 --2- 172.21.2.117:0/949354842 >> v2:172.21.2.125:3300/0 conn(0x7f7e980e7d70 0x7f7e980e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 --2- 172.21.2.117:0/949354842 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e981b07f0 0x7f7e980e4e90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.682 7f7e9d289700 1 -- 172.21.2.117:0/949354842 >> 172.21.2.117:0/949354842 conn(0x7f7e980ce500 msgr2=0x7f7e980e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.683 7f7e9d289700 1 -- 172.21.2.117:0/949354842 shutdown_connections 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.683 7f7e9d289700 1 -- 172.21.2.117:0/949354842 wait complete. 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.697 7f7e9d289700 1 Processor -- start 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.697 7f7e9d289700 1 -- start start 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e980a7720 0x7f7e98094af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e9819a340 0x7f7e980c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980a5c70 con 0x7f7e9819a740 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e98095110 con 0x7f7e980a7720 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9d289700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980c4060 con 0x7f7e9819a340 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54158/0 (socket says 172.21.2.117:54158) 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 -- 172.21.2.117:0/3625704779 learned_addr learned my addr 172.21.2.117:0/3625704779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:38.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 -- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e9819a340 msgr2=0x7f7e980c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e9819a340 0x7f7e980c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 -- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e980a7720 msgr2=0x7f7e98094af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e980a7720 0x7f7e98094af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 -- 172.21.2.117:0/3625704779 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980c3bb0 con 0x7f7e9819a740 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e9787e700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f7e90009940 tx=0x7f7e9000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.698 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e9000c2f0 con 0x7f7e9819a740 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9000c530 con 0x7f7e9819a740 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e980c3bb0 con 0x7f7e9819a740 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e980a64e0 con 0x7f7e9819a740 2021-02-10T11:49:38.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e9000c990 con 0x7f7e9819a740 2021-02-10T11:49:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e9000c710 con 0x7f7e9819a740 2021-02-10T11:49:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e8ffff700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 0x7f7e7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.699 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7e90032c70 con 0x7f7e9819a740 2021-02-10T11:49:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.700 7f7e9707d700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 0x7f7e7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:38.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.700 7f7e9707d700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 0x7f7e7c01abc0 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:38.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.729 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9807e800 con 0x7f7e9819a740 2021-02-10T11:49:38.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.732 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e9000c740 con 0x7f7e9819a740 2021-02-10T11:49:38.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.858 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7e980e77f0 con 0x7f7e9819a740 2021-02-10T11:49:38.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.858 7f7e8ffff700 1 -- 172.21.2.117:0/3625704779 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v109) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7e9000c530 con 0x7f7e9819a740 2021-02-10T11:49:38.855 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:38.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 msgr2=0x7f7e7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 0x7f7e7c01abc0 crc :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 msgr2=0x7f7e980c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:38.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f7e90009940 tx=0x7f7e9000bcf0).stop 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 shutdown_connections 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e7c0186c0 0x7f7e7c01abc0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e9819a740 0x7f7e980c3470 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e9819a340 0x7f7e980c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 --2- 172.21.2.117:0/3625704779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e980a7720 0x7f7e98094af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 >> 172.21.2.117:0/3625704779 conn(0x7f7e980a7210 msgr2=0x7f7e98095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 shutdown_connections 2021-02-10T11:49:38.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:38.860 7f7e9d289700 1 -- 172.21.2.117:0/3625704779 wait complete. 2021-02-10T11:49:38.865 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.285 7f2f60c00700 1 Processor -- start 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.285 7f2f60c00700 1 -- start start 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c0d18f0 0x7f2f5c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2f5c0eb040 0x7f2f5c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5c0ede60 con 0x7f2f5c0ee3e0 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5c0eba60 con 0x7f2f5c0eb040 2021-02-10T11:49:39.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f60c00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0d1cf0 con 0x7f2f5c0d18f0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54168/0 (socket says 172.21.2.117:54168) 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 -- 172.21.2.117:0/290204503 learned_addr learned my addr 172.21.2.117:0/290204503 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 -- 172.21.2.117:0/290204503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c0d18f0 msgr2=0x7f2f5c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 --2- 172.21.2.117:0/290204503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c0d18f0 0x7f2f5c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 -- 172.21.2.117:0/290204503 >> v2:172.21.2.125:3300/0 conn(0x7f2f5c0eb040 msgr2=0x7f2f5c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 --2- 172.21.2.117:0/290204503 >> v2:172.21.2.125:3300/0 conn(0x7f2f5c0eb040 0x7f2f5c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.286 7f2f5b05f700 1 -- 172.21.2.117:0/290204503 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0eee00 con 0x7f2f5c0ee3e0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f5b05f700 1 --2- 172.21.2.117:0/290204503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f2f54006f00 tx=0x7f2f54009f10).ready entity=mon.0 client_cookie=f31491874a6e3325 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f5985c700 1 -- 172.21.2.117:0/290204503 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f5400b3b0 con 0x7f2f5c0ee3e0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f5985c700 1 -- 172.21.2.117:0/290204503 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5400b730 con 0x7f2f5c0ee3e0 2021-02-10T11:49:39.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f5985c700 1 -- 172.21.2.117:0/290204503 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f5400bc10 con 0x7f2f5c0ee3e0 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 -- 172.21.2.117:0/290204503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c0ee3e0 msgr2=0x7f2f5c0ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 --2- 172.21.2.117:0/290204503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f2f54006f00 tx=0x7f2f54009f10).stop 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 -- 172.21.2.117:0/290204503 shutdown_connections 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 --2- 172.21.2.117:0/290204503 >> v2:172.21.2.125:3300/0 conn(0x7f2f5c0eb040 0x7f2f5c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 --2- 172.21.2.117:0/290204503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c0ee3e0 0x7f2f5c0ee7e0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 --2- 172.21.2.117:0/290204503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c0d18f0 0x7f2f5c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.287 7f2f60c00700 1 -- 172.21.2.117:0/290204503 >> 172.21.2.117:0/290204503 conn(0x7f2f5c0cf060 msgr2=0x7f2f5c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.288 7f2f60c00700 1 -- 172.21.2.117:0/290204503 shutdown_connections 2021-02-10T11:49:39.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.288 7f2f60c00700 1 -- 172.21.2.117:0/290204503 wait complete. 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 Processor -- start 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 -- start start 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c1a4880 0x7f2f5c08c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f5c08ce70 0x7f2f5c0945e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c094060 con 0x7f2f5c093be0 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0b5ee0 con 0x7f2f5c08ce70 2021-02-10T11:49:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f60c00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c087360 con 0x7f2f5c1a4880 2021-02-10T11:49:39.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.302 7f2f5a85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54172/0 (socket says 172.21.2.117:54172) 2021-02-10T11:49:39.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 -- 172.21.2.117:0/3035017115 learned_addr learned my addr 172.21.2.117:0/3035017115 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:39.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 -- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c1a4880 msgr2=0x7f2f5c08c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c1a4880 0x7f2f5c08c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 -- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f5c08ce70 msgr2=0x7f2f5c0945e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f5c08ce70 0x7f2f5c0945e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 -- 172.21.2.117:0/3035017115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c094c00 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f5a85e700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f2f4c006f00 tx=0x7f2f4c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4c006350 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c00be00 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4c002020 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c094c00 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.303 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c095190 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.304 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f4c005310 con 0x7f2f5c093be0 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.304 7f2f537fe700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 0x7f2f4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.304 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2f4c032540 con 0x7f2f5c093be0 2021-02-10T11:49:39.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.304 7f2f5a05d700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 0x7f2f4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.305 7f2f5a05d700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 0x7f2f4001abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.336 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c0ede60 con 0x7f2f5c093be0 2021-02-10T11:49:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.338 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f4c000960 con 0x7f2f5c093be0 2021-02-10T11:49:39.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.465 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2f5c0ede60 con 0x7f2f5c093be0 2021-02-10T11:49:39.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.465 7f2f537fe700 1 -- 172.21.2.117:0/3035017115 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v109) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2f4c005310 con 0x7f2f5c093be0 2021-02-10T11:49:39.462 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 msgr2=0x7f2f4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 0x7f2f4001abc0 crc :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 msgr2=0x7f2f5c08c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7f2f4c006f00 tx=0x7f2f4c00b940).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 shutdown_connections 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f5c08ce70 0x7f2f5c0945e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5c1a4880 0x7f2f5c08c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f400186c0 0x7f2f4001abc0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.467 7f2f60c00700 1 --2- 172.21.2.117:0/3035017115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f5c093be0 0x7f2f5c08c230 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:39.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.468 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 >> 172.21.2.117:0/3035017115 conn(0x7f2f5c0937a0 msgr2=0x7f2f5c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:39.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.468 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 shutdown_connections 2021-02-10T11:49:39.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.468 7f2f60c00700 1 -- 172.21.2.117:0/3035017115 wait complete. 2021-02-10T11:49:39.472 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:49:39.473 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:39.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 Processor -- start 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 -- start start 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc40d18e0 0x7efdc40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdc40ef3f0 0x7efdc41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc40ede60 con 0x7efdc40eb000 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc4088020 con 0x7efdc40ef3f0 2021-02-10T11:49:39.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdcb06a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40d1ce0 con 0x7efdc40d18e0 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.887 7efdc9687700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdc40ef3f0 0x7efdc41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efdc40ef3f0 0x7efdc41b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54182/0 (socket says 172.21.2.117:54182) 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 -- 172.21.2.117:0/3009948961 learned_addr learned my addr 172.21.2.117:0/3009948961 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc3fff700 1 --1- 172.21.2.117:0/3009948961 >> v1:172.21.2.117:6789/0 conn(0x7efdc40eb000 0x7efdc40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54428/0 (socket says 172.21.2.117:54428) 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 -- 172.21.2.117:0/3009948961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc40d18e0 msgr2=0x7efdc40ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 --2- 172.21.2.117:0/3009948961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc40d18e0 0x7efdc40ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 -- 172.21.2.117:0/3009948961 >> v1:172.21.2.117:6789/0 conn(0x7efdc40eb000 legacy=0x7efdc40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 -- 172.21.2.117:0/3009948961 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc41b8f90 con 0x7efdc40ef3f0 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc9687700 1 --2- 172.21.2.117:0/3009948961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc40ef3f0 0x7efdc41b7960 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7efdbc006f00 tx=0x7efdbc009e90).ready entity=mon.0 client_cookie=eb060cba8846e09f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc37fe700 1 -- 172.21.2.117:0/3009948961 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdbc00b300 con 0x7efdc40ef3f0 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc37fe700 1 -- 172.21.2.117:0/3009948961 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdbc00b580 con 0x7efdc40ef3f0 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdc37fe700 1 -- 172.21.2.117:0/3009948961 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdbc00b9d0 con 0x7efdc40ef3f0 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdcb06a700 1 -- 172.21.2.117:0/3009948961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc40ef3f0 msgr2=0x7efdc41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.888 7efdcb06a700 1 --2- 172.21.2.117:0/3009948961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc40ef3f0 0x7efdc41b7960 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7efdbc006f00 tx=0x7efdbc009e90).stop 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 -- 172.21.2.117:0/3009948961 shutdown_connections 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 --2- 172.21.2.117:0/3009948961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc40ef3f0 0x7efdc41b7960 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 --2- 172.21.2.117:0/3009948961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc40d18e0 0x7efdc40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 -- 172.21.2.117:0/3009948961 >> 172.21.2.117:0/3009948961 conn(0x7efdc40ced60 msgr2=0x7efdc40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:39.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 -- 172.21.2.117:0/3009948961 shutdown_connections 2021-02-10T11:49:39.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.889 7efdcb06a700 1 -- 172.21.2.117:0/3009948961 wait complete. 2021-02-10T11:49:39.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.903 7efdcb06a700 1 Processor -- start 2021-02-10T11:49:39.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.903 7efdcb06a700 1 -- start start 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdc4190240 0x7efdc4094300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc4094d20 0x7efdc40af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4095220 con 0x7efdc4094920 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4190680 con 0x7efdc4190240 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdcb06a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc40afe90 con 0x7efdc4094d20 2021-02-10T11:49:39.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54186/0 (socket says 172.21.2.117:54186) 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 -- 172.21.2.117:0/2529248798 learned_addr learned my addr 172.21.2.117:0/2529248798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 -- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc4094d20 msgr2=0x7efdc40af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc4094d20 0x7efdc40af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 -- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdc4190240 msgr2=0x7efdc4094300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdc4190240 0x7efdc4094300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.904 7efdc3fff700 1 -- 172.21.2.117:0/2529248798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc4095720 con 0x7efdc4094920 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc3fff700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7efdb400a590 tx=0x7efdb400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdb4000ef0 con 0x7efdc4094920 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb400ad20 con 0x7efdc4094920 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efdb4004620 con 0x7efdc4094920 2021-02-10T11:49:39.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc4095720 con 0x7efdc4094920 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4095720 con 0x7efdc4094920 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efdb40045b0 con 0x7efdc4094920 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.905 7efdc17fa700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 0x7efda801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.906 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efdb4032500 con 0x7efdc4094920 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.906 7efdc8e86700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 0x7efda801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:39.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.906 7efdc8e86700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 0x7efda801abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:39.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.937 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc4087210 con 0x7efdc4094920 2021-02-10T11:49:39.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:39.939 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efdb40043a0 con 0x7efdc4094920 2021-02-10T11:49:40.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.068 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdc4087210 con 0x7efdc4094920 2021-02-10T11:49:40.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.068 7efdc17fa700 1 -- 172.21.2.117:0/2529248798 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2101 (secure 0 0 0) 0x7efdb40045b0 con 0x7efdc4094920 2021-02-10T11:49:40.066 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:40.066 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":86,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:49:31.199085","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1582108166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1582108166},{"type":"v1","addr":"172.21.2.125:6835","nonce":1582108166}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:49:40.066 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 86 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 msgr2=0x7efda801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 0x7efda801abc0 crc :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 msgr2=0x7efdc40af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7efdb400a590 tx=0x7efdb400a930).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 shutdown_connections 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efda80186c0 0x7efda801abc0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efdc4094920 0x7efdc40af250 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efdc4094d20 0x7efdc40af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 --2- 172.21.2.117:0/2529248798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efdc4190240 0x7efdc4094300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:40.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.070 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 >> 172.21.2.117:0/2529248798 conn(0x7efdc41a0330 msgr2=0x7efdc40dd890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:40.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.071 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 shutdown_connections 2021-02-10T11:49:40.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:40.071 7efdcb06a700 1 -- 172.21.2.117:0/2529248798 wait complete. 2021-02-10T11:49:40.277 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:29.188327', 'modified': '2021-02-10 11:49:31.199085', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7801': {'gid': 7801, 'name': 'b', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1582108166', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1582108166}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1582108166}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:40.278 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:40.278 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7801, 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 11.490144264, 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 86, 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 109, 2021-02-10T11:49:40.677 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 109, 2021-02-10T11:49:40.678 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 17.125853289999998 2021-02-10T11:49:40.678 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:49:40.681 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7801, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.490144264, "mdsmap_epoch": 86, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "uptime": 17.125853289999998 } 2021-02-10T11:49:40.681 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:49:40.681 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:49:40.681 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:49:40.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:40.681 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:41.017 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:49:41.362 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:41.362 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:41.706 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:41.706 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:41.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:41.707 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:49:42.042 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:49:42.042 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:42.045 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:49:42.388 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:42.388 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:42.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[158098]: starting ceph client 2021-02-10T11:49:42.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:49:42.458 7f80f0848f80 -1 init, newargv = 0x55c09bf2ff60 newargc=9 2021-02-10T11:49:42.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[158098]: starting fuse 2021-02-10T11:49:42.731 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:42.731 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:42.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:42.733 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:49:43.041 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:43.045 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:49:43.067 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:49:43.067 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:49:43.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:49:43.068 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:49:43.069 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:49:43.405 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.158098.asok 2021-02-10T11:49:43.408 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.158098.asok 2021-02-10T11:49:43.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:43.408 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.158098.asok status 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "158098", 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:49:43.799 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10596, 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10596 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3954989985 2021-02-10T11:49:43.800 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3954989985 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10596 172.21.2.117:0/3954989985", 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3954989985", 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 86, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 109, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:49:43.801 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:49:43.802 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:43.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:43.803 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:43.803 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:44.138 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:44.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:44.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:44.139 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:44.486 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:49:44.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.901 7f205ed74700 1 Processor -- start 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.901 7f205ed74700 1 -- start start 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205ed74700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20580d18e0 0x7f20580ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205ed74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20580eb000 0x7f20580eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205ed74700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580ede60 con 0x7f20580ee3e0 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205ed74700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20580eba90 con 0x7f20580d18e0 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205ed74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20580d1ce0 con 0x7f20580eb000 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20580d18e0 0x7f20580ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20580d18e0 0x7f20580ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54204/0 (socket says 172.21.2.117:54204) 2021-02-10T11:49:44.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 -- 172.21.2.117:0/698516641 learned_addr learned my addr 172.21.2.117:0/698516641 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 -- 172.21.2.117:0/698516641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20580eb000 msgr2=0x7f20580eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 --2- 172.21.2.117:0/698516641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20580eb000 0x7f20580eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 -- 172.21.2.117:0/698516641 >> v1:172.21.2.125:6789/0 conn(0x7f20580ee3e0 legacy=0x7f20580ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 -- 172.21.2.117:0/698516641 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20580eefd0 con 0x7f20580d18e0 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f205cb90700 1 --2- 172.21.2.117:0/698516641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580d18e0 0x7f20580ea9e0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f2040006700 tx=0x7f2040009b80).ready entity=mon.0 client_cookie=6eb20b739b5523ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f20577fe700 1 -- 172.21.2.117:0/698516641 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f204000c0c0 con 0x7f20580d18e0 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f20577fe700 1 -- 172.21.2.117:0/698516641 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204000c340 con 0x7f20580d18e0 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.902 7f20577fe700 1 -- 172.21.2.117:0/698516641 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f204000c860 con 0x7f20580d18e0 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 -- 172.21.2.117:0/698516641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580d18e0 msgr2=0x7f20580ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 --2- 172.21.2.117:0/698516641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580d18e0 0x7f20580ea9e0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f2040006700 tx=0x7f2040009b80).stop 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 -- 172.21.2.117:0/698516641 shutdown_connections 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 --2- 172.21.2.117:0/698516641 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20580eb000 0x7f20580eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:44.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 --2- 172.21.2.117:0/698516641 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580d18e0 0x7f20580ea9e0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:44.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 -- 172.21.2.117:0/698516641 >> 172.21.2.117:0/698516641 conn(0x7f20580ced60 msgr2=0x7f20580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:44.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 -- 172.21.2.117:0/698516641 shutdown_connections 2021-02-10T11:49:44.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.903 7f205ed74700 1 -- 172.21.2.117:0/698516641 wait complete. 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 Processor -- start 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 -- start start 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f205819b760 0x7f20580b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f205819e620 0x7f20581afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20581af950 con 0x7f20580b5370 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205819bb60 con 0x7f205819e620 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f205ed74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205819ea20 con 0x7f205819b760 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f2057fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f2057fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54210/0 (socket says 172.21.2.117:54210) 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.918 7f2057fff700 1 -- 172.21.2.117:0/680951695 learned_addr learned my addr 172.21.2.117:0/680951695 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 -- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f205819b760 msgr2=0x7f20580b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f205819b760 0x7f20580b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 -- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f205819e620 msgr2=0x7f20581afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f205819e620 0x7f20581afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 -- 172.21.2.117:0/680951695 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205819ec40 con 0x7f20580b5370 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f2057fff700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f2048009960 tx=0x7f204800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:44.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2048000fc0 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20480030c0 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f205ed74700 1 -- 172.21.2.117:0/680951695 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205819ec40 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2048001500 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.919 7f205ed74700 1 -- 172.21.2.117:0/680951695 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205819ec40 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.920 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f204800a030 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.920 7f20557fa700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 0x7f203c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.920 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2048032a60 con 0x7f20580b5370 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.921 7f205cb90700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 0x7f203c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:44.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.921 7f205cb90700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 0x7f203c01abc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:44.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.952 7f205ed74700 1 -- 172.21.2.117:0/680951695 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2058087210 con 0x7f20580b5370 2021-02-10T11:49:44.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:44.954 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f204802df80 con 0x7f20580b5370 2021-02-10T11:49:45.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.075 7f205ed74700 1 -- 172.21.2.117:0/680951695 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20580ede60 con 0x7f20580b5370 2021-02-10T11:49:45.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.076 7f20557fa700 1 -- 172.21.2.117:0/680951695 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+14239 (secure 0 0 0) 0x7f204802df80 con 0x7f20580b5370 2021-02-10T11:49:45.073 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:45.073 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":109,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:49:30.190724","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:49:27.162612","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":23,"pool_name":"cephfs_data","create_time":"2021-02-10 11:49:28.233491","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.077 7f205ed74700 1 -- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 msgr2=0x7f203c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.077 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 0x7f203c01abc0 crc :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.077 7f205ed74700 1 -- 172.21.2.117:0/680951695 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 msgr2=0x7f205819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.077 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f2048009960 tx=0x7f204800bcf0).stop 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 -- 172.21.2.117:0/680951695 shutdown_connections 2021-02-10T11:49:45.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f203c0186c0 0x7f203c01abc0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20580b5370 0x7f205819e000 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f205819e620 0x7f20581afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 --2- 172.21.2.117:0/680951695 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f205819b760 0x7f20580b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 -- 172.21.2.117:0/680951695 >> 172.21.2.117:0/680951695 conn(0x7f205819f610 msgr2=0x7f20581ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 -- 172.21.2.117:0/680951695 shutdown_connections 2021-02-10T11:49:45.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.078 7f205ed74700 1 -- 172.21.2.117:0/680951695 wait complete. 2021-02-10T11:49:45.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:49:27.162612', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:49:28.233491', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:49:45.083 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:49:45.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 Processor -- start 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 -- start start 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f786c0d18f0 0x7f786c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f786c0ef3f0 0x7f786c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0ef7f0 con 0x7f786c0eb040 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c0ede60 con 0x7f786c0d18f0 2021-02-10T11:49:45.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f78727fb700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c0d1cf0 con 0x7f786c0ef3f0 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f786b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f786c0eb040 0x7f786c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54466/0 (socket says 172.21.2.117:54466) 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f786bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f786c0d18f0 0x7f786c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.501 7f786b7fe700 1 -- 172.21.2.117:0/4121584084 learned_addr learned my addr 172.21.2.117:0/4121584084 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786affd700 1 -- 172.21.2.117:0/4121584084 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846055837 0 0) 0x7f78600010b0 con 0x7f786c0eb040 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786affd700 1 -- 172.21.2.117:0/4121584084 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7854002fe0 con 0x7f786c0eb040 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786bfff700 1 -- 172.21.2.117:0/4121584084 >> v2:172.21.2.125:3300/0 conn(0x7f786c0ef3f0 msgr2=0x7f786c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786bfff700 1 --2- 172.21.2.117:0/4121584084 >> v2:172.21.2.125:3300/0 conn(0x7f786c0ef3f0 0x7f786c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786bfff700 1 -- 172.21.2.117:0/4121584084 >> v1:172.21.2.117:6789/0 conn(0x7f786c0eb040 legacy=0x7f786c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786bfff700 1 -- 172.21.2.117:0/4121584084 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c1b8fd0 con 0x7f786c0d18f0 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786bfff700 1 --2- 172.21.2.117:0/4121584084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0d18f0 0x7f786c0eaa20 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f785c006650 tx=0x7f785c009a80).ready entity=mon.0 client_cookie=2a17e144e32e1b6c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:45.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786affd700 1 -- 172.21.2.117:0/4121584084 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c0009b0 con 0x7f786c0d18f0 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786affd700 1 -- 172.21.2.117:0/4121584084 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c00c0c0 con 0x7f786c0d18f0 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f786affd700 1 -- 172.21.2.117:0/4121584084 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c00c5e0 con 0x7f786c0d18f0 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f78727fb700 1 -- 172.21.2.117:0/4121584084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0d18f0 msgr2=0x7f786c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.502 7f78727fb700 1 --2- 172.21.2.117:0/4121584084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0d18f0 0x7f786c0eaa20 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f785c006650 tx=0x7f785c009a80).stop 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 -- 172.21.2.117:0/4121584084 shutdown_connections 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 --2- 172.21.2.117:0/4121584084 >> v2:172.21.2.125:3300/0 conn(0x7f786c0ef3f0 0x7f786c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 --2- 172.21.2.117:0/4121584084 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0d18f0 0x7f786c0eaa20 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 -- 172.21.2.117:0/4121584084 >> 172.21.2.117:0/4121584084 conn(0x7f786c0cf060 msgr2=0x7f786c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 -- 172.21.2.117:0/4121584084 shutdown_connections 2021-02-10T11:49:45.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.503 7f78727fb700 1 -- 172.21.2.117:0/4121584084 wait complete. 2021-02-10T11:49:45.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.517 7f78727fb700 1 Processor -- start 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 -- start start 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f786c0b4c10 0x7f786c0aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f786c0b5010 0x7f786c0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0b4320 con 0x7f786c0b3e10 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c08a220 con 0x7f786c0b4c10 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f78727fb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0abbc0 con 0x7f786c0b5010 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f786bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f786bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54222/0 (socket says 172.21.2.117:54222) 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.518 7f786bfff700 1 -- 172.21.2.117:0/3428123280 learned_addr learned my addr 172.21.2.117:0/3428123280 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 -- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f786c0b5010 msgr2=0x7f786c0ab5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:45.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f786c0b5010 0x7f786c0ab5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 -- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f786c0b4c10 msgr2=0x7f786c0aaf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f786c0b4c10 0x7f786c0aaf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 -- 172.21.2.117:0/3428123280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0abde0 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f786bfff700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f785c009650 tx=0x7f785c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c0093a0 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c00bde0 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f785c0052e0 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f78727fb700 1 -- 172.21.2.117:0/3428123280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c0abde0 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.519 7f78727fb700 1 -- 172.21.2.117:0/3428123280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c0ac420 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.520 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f785c009300 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.520 7f7868ff9700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 0x7f784c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.520 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f785c032500 con 0x7f786c0b3e10 2021-02-10T11:49:45.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.521 7f786b7fe700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 0x7f784c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:45.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.522 7f786b7fe700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 0x7f784c01abc0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:45.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.551 7f78727fb700 1 -- 172.21.2.117:0/3428123280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c08a220 con 0x7f786c0b3e10 2021-02-10T11:49:45.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.554 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f785c0009c0 con 0x7f786c0b3e10 2021-02-10T11:49:45.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.678 7f78727fb700 1 -- 172.21.2.117:0/3428123280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f786c08a220 con 0x7f786c0b3e10 2021-02-10T11:49:45.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.678 7f7868ff9700 1 -- 172.21.2.117:0/3428123280 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v109) v1 ==== 92+0+10 (secure 0 0 0) 0x7f785c009300 con 0x7f786c0b3e10 2021-02-10T11:49:45.676 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:45.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 -- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 msgr2=0x7f784c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 0x7f784c01abc0 crc :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 -- 172.21.2.117:0/3428123280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 msgr2=0x7f786c0b45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f785c009650 tx=0x7f785c00b960).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 -- 172.21.2.117:0/3428123280 shutdown_connections 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f786c0b5010 0x7f786c0ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f786c0b4c10 0x7f786c0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f784c0186c0 0x7f784c01abc0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 --2- 172.21.2.117:0/3428123280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f786c0b3e10 0x7f786c0b45f0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:45.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.680 7f78727fb700 1 -- 172.21.2.117:0/3428123280 >> 172.21.2.117:0/3428123280 conn(0x7f786c0b6c70 msgr2=0x7f786c0970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:45.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.681 7f78727fb700 1 -- 172.21.2.117:0/3428123280 shutdown_connections 2021-02-10T11:49:45.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:45.681 7f78727fb700 1 -- 172.21.2.117:0/3428123280 wait complete. 2021-02-10T11:49:45.685 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.103 7f70406f5700 1 Processor -- start 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.103 7f70406f5700 1 -- start start 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f70380ef3f0 0x7f70380eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70380eb830 0x7f70381b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380ef7f0 con 0x7f70380d18f0 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380ede60 con 0x7f70380eb830 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380d1cf0 con 0x7f70380ef3f0 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70380eb830 0x7f70381b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f70380eb830 0x7f70381b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54234/0 (socket says 172.21.2.117:54234) 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 -- 172.21.2.117:0/54450503 learned_addr learned my addr 172.21.2.117:0/54450503 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 -- 172.21.2.117:0/54450503 >> v2:172.21.2.125:3300/0 conn(0x7f70380ef3f0 msgr2=0x7f70380eb210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 --2- 172.21.2.117:0/54450503 >> v2:172.21.2.125:3300/0 conn(0x7f70380ef3f0 0x7f70380eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 -- 172.21.2.117:0/54450503 >> v1:172.21.2.125:6789/0 conn(0x7f70380d18f0 legacy=0x7f70380eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 -- 172.21.2.117:0/54450503 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70381b8fd0 con 0x7f70380eb830 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703ed12700 1 --2- 172.21.2.117:0/54450503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70380eb830 0x7f70381b79a0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7f7030006f00 tx=0x7f7030009e90).ready entity=mon.0 client_cookie=2dc40c2f8ce88679 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703d50f700 1 -- 172.21.2.117:0/54450503 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f703000b300 con 0x7f70380eb830 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703d50f700 1 -- 172.21.2.117:0/54450503 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000b580 con 0x7f70380eb830 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f703d50f700 1 -- 172.21.2.117:0/54450503 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f703000b9d0 con 0x7f70380eb830 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 -- 172.21.2.117:0/54450503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70380eb830 msgr2=0x7f70381b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.104 7f70406f5700 1 --2- 172.21.2.117:0/54450503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70380eb830 0x7f70381b79a0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7f7030006f00 tx=0x7f7030009e90).stop 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 -- 172.21.2.117:0/54450503 shutdown_connections 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 --2- 172.21.2.117:0/54450503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f70380eb830 0x7f70381b79a0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 --2- 172.21.2.117:0/54450503 >> v2:172.21.2.125:3300/0 conn(0x7f70380ef3f0 0x7f70380eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 -- 172.21.2.117:0/54450503 >> 172.21.2.117:0/54450503 conn(0x7f70380cf060 msgr2=0x7f70380cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:46.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 -- 172.21.2.117:0/54450503 shutdown_connections 2021-02-10T11:49:46.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.105 7f70406f5700 1 -- 172.21.2.117:0/54450503 wait complete. 2021-02-10T11:49:46.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 Processor -- start 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 -- start start 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70381a78b0 0x7f70381a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 0x7f70380916e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380926c0 con 0x7f7038096ab0 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038091d00 con 0x7f70380974d0 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f70406f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038091ee0 con 0x7f70381a78b0 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f703dd10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f703dd10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54236/0 (socket says 172.21.2.117:54236) 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.120 7f703dd10700 1 -- 172.21.2.117:0/2902938290 learned_addr learned my addr 172.21.2.117:0/2902938290 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 -- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70381a78b0 msgr2=0x7f70381a7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70381a78b0 0x7f70381a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 -- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 msgr2=0x7f70380916e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703ed12700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 0x7f70380916e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 0x7f70380916e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 -- 172.21.2.117:0/2902938290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038092100 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703ed12700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 0x7f70380916e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f703dd10700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7f702c00b530 tx=0x7f702c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f702c00bcc0 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c000c90 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f702c0022a0 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.121 7f70406f5700 1 -- 172.21.2.117:0/2902938290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7038092100 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.122 7f70406f5700 1 -- 172.21.2.117:0/2902938290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038092100 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.122 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f702c005310 con 0x7f7038096ab0 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.122 7f702affd700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 0x7f702001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.122 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f702c0328f0 con 0x7f7038096ab0 2021-02-10T11:49:46.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.123 7f703e511700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 0x7f702001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.123 7f703e511700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 0x7f702001ac10 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.154 7f70406f5700 1 -- 172.21.2.117:0/2902938290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70380f2f00 con 0x7f7038096ab0 2021-02-10T11:49:46.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.156 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f702c006b90 con 0x7f7038096ab0 2021-02-10T11:49:46.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.283 7f70406f5700 1 -- 172.21.2.117:0/2902938290 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f70380f67e0 con 0x7f7038096ab0 2021-02-10T11:49:46.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.283 7f702affd700 1 -- 172.21.2.117:0/2902938290 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v109) v1 ==== 104+0+10 (secure 0 0 0) 0x7f702c000c90 con 0x7f7038096ab0 2021-02-10T11:49:46.280 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 -- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 msgr2=0x7f702001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 0x7f702001ac10 crc :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 -- 172.21.2.117:0/2902938290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 msgr2=0x7f7038096eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7f702c00b530 tx=0x7f702c00b8d0).stop 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 -- 172.21.2.117:0/2902938290 shutdown_connections 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7020018730 0x7f702001ac10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7038096ab0 0x7f7038096eb0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f70380974d0 0x7f70380916e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 --2- 172.21.2.117:0/2902938290 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f70381a78b0 0x7f70381a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.285 7f70406f5700 1 -- 172.21.2.117:0/2902938290 >> 172.21.2.117:0/2902938290 conn(0x7f70380c3df0 msgr2=0x7f70380a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:46.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.286 7f70406f5700 1 -- 172.21.2.117:0/2902938290 shutdown_connections 2021-02-10T11:49:46.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.286 7f70406f5700 1 -- 172.21.2.117:0/2902938290 wait complete. 2021-02-10T11:49:46.290 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.707 7f87f61f5700 1 Processor -- start 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.707 7f87f61f5700 1 -- start start 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f87f00d18f0 0x7f87f00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f00eb040 0x7f87f00eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87f00ee3e0 0x7f87f00ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f00edee0 con 0x7f87f00ee3e0 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f00eba60 con 0x7f87f00d18f0 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f61f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00d1cf0 con 0x7f87f00eb040 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87f00ee3e0 0x7f87f00ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f87f00ee3e0 0x7f87f00ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54244/0 (socket says 172.21.2.117:54244) 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 -- 172.21.2.117:0/1879137343 learned_addr learned my addr 172.21.2.117:0/1879137343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 -- 172.21.2.117:0/1879137343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f00eb040 msgr2=0x7f87f00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 --2- 172.21.2.117:0/1879137343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f00eb040 0x7f87f00eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 -- 172.21.2.117:0/1879137343 >> v2:172.21.2.125:3300/0 conn(0x7f87f00d18f0 msgr2=0x7f87f00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 --2- 172.21.2.117:0/1879137343 >> v2:172.21.2.125:3300/0 conn(0x7f87f00d18f0 0x7f87f00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 -- 172.21.2.117:0/1879137343 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00eee00 con 0x7f87f00ee3e0 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.708 7f87f4812700 1 --2- 172.21.2.117:0/1879137343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00ee3e0 0x7f87f00ee7e0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7f87e8006f00 tx=0x7f87e8009f10).ready entity=mon.0 client_cookie=3fc5d03228b8cad0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87eeffd700 1 -- 172.21.2.117:0/1879137343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e800b3b0 con 0x7f87f00ee3e0 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87eeffd700 1 -- 172.21.2.117:0/1879137343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e800b630 con 0x7f87f00ee3e0 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87eeffd700 1 -- 172.21.2.117:0/1879137343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e800bb50 con 0x7f87f00ee3e0 2021-02-10T11:49:46.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 -- 172.21.2.117:0/1879137343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00ee3e0 msgr2=0x7f87f00ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 --2- 172.21.2.117:0/1879137343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00ee3e0 0x7f87f00ee7e0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7f87e8006f00 tx=0x7f87e8009f10).stop 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 -- 172.21.2.117:0/1879137343 shutdown_connections 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 --2- 172.21.2.117:0/1879137343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00ee3e0 0x7f87f00ee7e0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 --2- 172.21.2.117:0/1879137343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f00eb040 0x7f87f00eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 --2- 172.21.2.117:0/1879137343 >> v2:172.21.2.125:3300/0 conn(0x7f87f00d18f0 0x7f87f00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 -- 172.21.2.117:0/1879137343 >> 172.21.2.117:0/1879137343 conn(0x7f87f00cf060 msgr2=0x7f87f00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 -- 172.21.2.117:0/1879137343 shutdown_connections 2021-02-10T11:49:46.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.709 7f87f61f5700 1 -- 172.21.2.117:0/1879137343 wait complete. 2021-02-10T11:49:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 Processor -- start 2021-02-10T11:49:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 -- start start 2021-02-10T11:49:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87f019f870 0x7f87f0096440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f0096a60 0x7f87f0096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0095ec0 con 0x7f87f00bf010 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0097480 con 0x7f87f019f870 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.724 7f87f61f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0097660 con 0x7f87f0096a60 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54248/0 (socket says 172.21.2.117:54248) 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 -- 172.21.2.117:0/2054380149 learned_addr learned my addr 172.21.2.117:0/2054380149 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:46.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 -- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f0096a60 msgr2=0x7f87f0096e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f0096a60 0x7f87f0096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 -- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87f019f870 msgr2=0x7f87f0096440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87f019f870 0x7f87f0096440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 -- 172.21.2.117:0/2054380149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0094450 con 0x7f87f00bf010 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87effff700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f87e0006f00 tx=0x7f87e000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.725 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e000bd30 con 0x7f87f00bf010 2021-02-10T11:49:46.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0000c90 con 0x7f87f00bf010 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f87e00022a0 con 0x7f87f00bf010 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f0094450 con 0x7f87f00bf010 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f0094450 con 0x7f87f00bf010 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f87e0005310 con 0x7f87f00bf010 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87ecff9700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:46.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.726 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f87e0032540 con 0x7f87f00bf010 2021-02-10T11:49:46.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.727 7f87ef7fe700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:46.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.727 7f87ef7fe700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 0x7f87d001abc0 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:46.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.758 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f00ede60 con 0x7f87f00bf010 2021-02-10T11:49:46.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.760 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f87e0006b90 con 0x7f87f00bf010 2021-02-10T11:49:46.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.887 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f87f00ede60 con 0x7f87f00bf010 2021-02-10T11:49:46.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.887 7f87ecff9700 1 -- 172.21.2.117:0/2054380149 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v109) v1 ==== 100+0+10 (secure 0 0 0) 0x7f87e000bd30 con 0x7f87f00bf010 2021-02-10T11:49:46.884 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:49:46.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 msgr2=0x7f87d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 0x7f87d001abc0 crc :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 msgr2=0x7f87f019f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f87e0006f00 tx=0x7f87e000b940).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 shutdown_connections 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f87f0096a60 0x7f87f0096e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f87f019f870 0x7f87f0096440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f87d00186c0 0x7f87d001abc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 --2- 172.21.2.117:0/2054380149 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f87f00bf010 0x7f87f019f250 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:46.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.889 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 >> 172.21.2.117:0/2054380149 conn(0x7f87f00ae940 msgr2=0x7f87f008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.890 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 shutdown_connections 2021-02-10T11:49:46.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:46.890 7f87f61f5700 1 -- 172.21.2.117:0/2054380149 wait complete. 2021-02-10T11:49:46.894 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:49:46.895 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:49:47.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.310 7f5d0f4f3700 1 Processor -- start 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.310 7f5d0f4f3700 1 -- start start 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0f4f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5d081b6960 0x7f5d081b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0f4f3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080ede60 con 0x7f5d080d18e0 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0f4f3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080eb9c0 con 0x7f5d080ef3f0 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0f4f3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d080ef7f0 con 0x7f5d081b6960 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0d30f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5d080d18e0 0x7f5d080ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54506/0 (socket says 172.21.2.117:54506) 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0d30f700 1 -- 172.21.2.117:0/2190408950 learned_addr learned my addr 172.21.2.117:0/2190408950 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5cfbfff700 1 -- 172.21.2.117:0/2190408950 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048013882 0 0) 0x7f5cfc001180 con 0x7f5d080d18e0 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5d0db10700 1 --2- 172.21.2.117:0/2190408950 >> v2:172.21.2.125:3300/0 conn(0x7f5d081b6960 0x7f5d081b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.311 7f5cfbfff700 1 -- 172.21.2.117:0/2190408950 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4002fe0 con 0x7f5d080d18e0 2021-02-10T11:49:47.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0db10700 1 -- 172.21.2.117:0/2190408950 >> v1:172.21.2.125:6789/0 conn(0x7f5d080ef3f0 legacy=0x7f5d080eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0db10700 1 -- 172.21.2.117:0/2190408950 >> v1:172.21.2.117:6789/0 conn(0x7f5d080d18e0 legacy=0x7f5d080ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0db10700 1 -- 172.21.2.117:0/2190408950 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080d1d20 con 0x7f5d081b6960 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0db10700 1 --2- 172.21.2.117:0/2190408950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d081b6960 0x7f5d081b8d70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f5d00006e50 tx=0x7f5d00009e70).ready entity=mon.1 client_cookie=148fde0810349d0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5cfbfff700 1 -- 172.21.2.117:0/2190408950 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0000b510 con 0x7f5d081b6960 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5cfbfff700 1 -- 172.21.2.117:0/2190408950 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00006790 con 0x7f5d081b6960 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5cfbfff700 1 -- 172.21.2.117:0/2190408950 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0000b940 con 0x7f5d081b6960 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0f4f3700 1 -- 172.21.2.117:0/2190408950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d081b6960 msgr2=0x7f5d081b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.312 7f5d0f4f3700 1 --2- 172.21.2.117:0/2190408950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d081b6960 0x7f5d081b8d70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f5d00006e50 tx=0x7f5d00009e70).stop 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.313 7f5d0f4f3700 1 -- 172.21.2.117:0/2190408950 shutdown_connections 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.313 7f5d0f4f3700 1 --2- 172.21.2.117:0/2190408950 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d081b6960 0x7f5d081b8d70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.313 7f5d0f4f3700 1 -- 172.21.2.117:0/2190408950 >> 172.21.2.117:0/2190408950 conn(0x7f5d080ced60 msgr2=0x7f5d080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:47.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.313 7f5d0f4f3700 1 -- 172.21.2.117:0/2190408950 shutdown_connections 2021-02-10T11:49:47.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.313 7f5d0f4f3700 1 -- 172.21.2.117:0/2190408950 wait complete. 2021-02-10T11:49:47.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.327 7f5d0f4f3700 1 Processor -- start 2021-02-10T11:49:47.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.327 7f5d0f4f3700 1 -- start start 2021-02-10T11:49:47.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 0x7f5d080e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d080e9560 0x7f5d080e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d08196810 con 0x7f5d080e9960 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d08098ad0 con 0x7f5d080e9560 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0f4f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d080e8a40 con 0x7f5d08196390 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54262/0 (socket says 172.21.2.117:54262) 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 -- 172.21.2.117:0/3967650426 learned_addr learned my addr 172.21.2.117:0/3967650426 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0d30f700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 0x7f5d080e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 -- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 msgr2=0x7f5d080e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 0x7f5d080e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 -- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d080e9560 msgr2=0x7f5d080e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d080e9560 0x7f5d080e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0db10700 1 -- 172.21.2.117:0/3967650426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d080e8c60 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.328 7f5d0d30f700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 0x7f5d080e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5d0db10700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f5d00006c50 tx=0x7f5d0000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d000093d0 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0000be60 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0000c4e0 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080e8c60 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.329 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d080e8c60 con 0x7f5d080e9960 2021-02-10T11:49:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.330 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d00002ef0 con 0x7f5d080e9960 2021-02-10T11:49:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.330 7f5cf9ffb700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 0x7f5ce801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.330 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5d00032a80 con 0x7f5d080e9960 2021-02-10T11:49:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.330 7f5d0d30f700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 0x7f5ce801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:47.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.331 7f5d0d30f700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 0x7f5ce801abc0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:47.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.361 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08087210 con 0x7f5d080e9960 2021-02-10T11:49:47.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.363 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d00000df0 con 0x7f5d080e9960 2021-02-10T11:49:47.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.489 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d080ede60 con 0x7f5d080e9960 2021-02-10T11:49:47.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.489 7f5cf9ffb700 1 -- 172.21.2.117:0/3967650426 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f5d00002ef0 con 0x7f5d080e9960 2021-02-10T11:49:47.486 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 86 2021-02-10T11:49:47.487 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:49:47.487 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":86,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:49:31.199085","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1582108166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1582108166},{"type":"v1","addr":"172.21.2.125:6835","nonce":1582108166}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:49:47.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.491 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 msgr2=0x7f5ce801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.491 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 0x7f5ce801abc0 crc :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.491 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 msgr2=0x7f5d080e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:47.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.491 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f5d00006c50 tx=0x7f5d0000bd20).stop 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 shutdown_connections 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ce80186c0 0x7f5ce801abc0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d080e9960 0x7f5d080e8420 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d080e9560 0x7f5d080e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 --2- 172.21.2.117:0/3967650426 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d08196390 0x7f5d080e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 >> 172.21.2.117:0/3967650426 conn(0x7f5d0819f610 msgr2=0x7f5d081a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:47.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 shutdown_connections 2021-02-10T11:49:47.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:47.492 7f5d0f4f3700 1 -- 172.21.2.117:0/3967650426 wait complete. 2021-02-10T11:49:47.698 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:49:29.188327', 'modified': '2021-02-10 11:49:31.199085', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7801': {'gid': 7801, 'name': 'b', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1582108166', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1582108166}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1582108166}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:49:47.698 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:49:47.699 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:49:48.097 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:49:48.097 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7801, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.910631456000001, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 86, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 109, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 109, 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 24.546339550999999 2021-02-10T11:49:48.098 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:49:48.101 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7801, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.910631456000001, "mdsmap_epoch": 86, "osdmap_epoch": 109, "osdmap_epoch_barrier": 109, "uptime": 24.546339550999999 } 2021-02-10T11:49:48.101 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:49:48.102 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:49:48.102 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:49:48.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:48.102 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:48.439 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:49:48.783 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:48.783 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:49.126 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:49.126 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:49.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:49.127 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:49:49.462 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:49:49.462 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:49:49.462 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:49.465 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:49:49.809 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:49:49.809 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:49:49.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:49:49.879 7f6860caef80 -1 init, newargv = 0x560e9e796760 newargc=9 2021-02-10T11:49:49.875 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[158765]: starting ceph client 2021-02-10T11:49:49.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[158765]: starting fuse 2021-02-10T11:49:50.151 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:49:50.151 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:49:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:49:50.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:50.152 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:49:50.487 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:49:50.488 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:49:50.488 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:49:50.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:50.488 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:49:50.488 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:49:50.488 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:49:50.488 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:49:50.488 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:49:50.489 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:49:50.490 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:50.490 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:49:50.490 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:49:50.850 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.158765.asok 2021-02-10T11:49:50.853 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.158765.asok 2021-02-10T11:49:50.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:50.854 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.158765.asok status 2021-02-10T11:49:51.245 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:51.245 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:49:51.245 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:49:51.245 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "158765", 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10638, 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:49:51.246 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10638 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1432985189 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1432985189 2021-02-10T11:49:51.247 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10638 172.21.2.117:0/1432985189", 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1432985189", 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 86, 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 109, 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:49:51.248 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:51.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:51.250 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:51.250 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:51.585 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:51.585 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:49:51.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:51.586 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:51.930 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.350 7f7384941700 1 Processor -- start 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 -- start start 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800e5240 0x7f7380086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7380084c60 0x7f7380085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7380085930 con 0x7f73800e4e40 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7380085680 con 0x7f7380084c60 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f7384941700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73801b09e0 con 0x7f73800e5240 2021-02-10T11:49:52.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f737e61c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f73800e4e40 0x7f73801b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54534/0 (socket says 172.21.2.117:54534) 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.351 7f737e61c700 1 -- 172.21.2.117:0/3977013416 learned_addr learned my addr 172.21.2.117:0/3977013416 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348435203 0 0) 0x7f7368000ce0 con 0x7f73800e4e40 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f736c002fe0 con 0x7f73800e4e40 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737de1b700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800e5240 0x7f7380086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 --2- 172.21.2.117:0/3977013416 >> v2:172.21.2.125:3300/0 conn(0x7f7380084c60 0x7f7380085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1819349443 0 0) 0x7f73680010e0 con 0x7f73800e4e40 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800e5240 msgr2=0x7f7380086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800e5240 0x7f7380086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 >> v2:172.21.2.125:3300/0 conn(0x7f7380084c60 msgr2=0x7f7380085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 --2- 172.21.2.117:0/3977013416 >> v2:172.21.2.125:3300/0 conn(0x7f7380084c60 0x7f7380085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380084840 con 0x7f73800e4e40 2021-02-10T11:49:52.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f7368001660 con 0x7f73800e4e40 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 >> v1:172.21.2.117:6789/0 conn(0x7f73800e4e40 legacy=0x7f73801b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c003a60 0x7f736c005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c04a760 0x7f736c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c04d500 0x7f736c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0502d0 con 0x7f736c04d500 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0504f0 con 0x7f736c04a760 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737d61a700 1 -- 172.21.2.117:0/3977013416 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736c050710 con 0x7f736c003a60 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c04d500 0x7f736c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737de1b700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c04a760 0x7f736c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737e61c700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c003a60 0x7f736c005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 -- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c003a60 msgr2=0x7f736c005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736c003a60 0x7f736c005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 -- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c04a760 msgr2=0x7f736c04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c04a760 0x7f736c04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.352 7f737ee1d700 1 -- 172.21.2.117:0/3977013416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c0509b0 con 0x7f736c04d500 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737de1b700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f736c04a760 0x7f736c04cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737ee1d700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c04d500 0x7f736c04fa20 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7f7378006a40 tx=0x7f737800c0c0).ready entity=mon.0 client_cookie=54b587ace0e81c7f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73780066f0 con 0x7f736c04d500 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737800c4b0 con 0x7f736c04d500 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737d61a700 1 -- 172.21.2.117:0/3977013416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f737800c940 con 0x7f736c04d500 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f7384941700 1 -- 172.21.2.117:0/3977013416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c04d500 msgr2=0x7f736c04fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f7384941700 1 --2- 172.21.2.117:0/3977013416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f736c04d500 0x7f736c04fa20 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7f7378006a40 tx=0x7f737800c0c0).stop 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f737ee1d700 1 -- 172.21.2.117:0/3977013416 reap_dead start 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f7384941700 1 -- 172.21.2.117:0/3977013416 shutdown_connections 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.353 7f7384941700 1 -- 172.21.2.117:0/3977013416 >> 172.21.2.117:0/3977013416 conn(0x7f73800ce4e0 msgr2=0x7f73800e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:52.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.354 7f7384941700 1 -- 172.21.2.117:0/3977013416 shutdown_connections 2021-02-10T11:49:52.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.354 7f7384941700 1 -- 172.21.2.117:0/3977013416 wait complete. 2021-02-10T11:49:52.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.368 7f7384941700 1 Processor -- start 2021-02-10T11:49:52.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.368 7f7384941700 1 -- start start 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800be700 0x7f73800bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73800c0420 0x7f73800a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800bee60 con 0x7f73800bfa00 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800beb00 con 0x7f73800c0420 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f7384941700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73800912f0 con 0x7f73800be700 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54300/0 (socket says 172.21.2.117:54300) 2021-02-10T11:49:52.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 -- 172.21.2.117:0/3288476540 learned_addr learned my addr 172.21.2.117:0/3288476540 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 -- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800be700 msgr2=0x7f73800bf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800be700 0x7f73800bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 -- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73800c0420 msgr2=0x7f73800a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73800c0420 0x7f73800a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 -- 172.21.2.117:0/3288476540 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800a9bd0 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.369 7f737de1b700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7f7370006f40 tx=0x7f737000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f737000bd80 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370000c90 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7370002030 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7384941700 1 -- 172.21.2.117:0/3288476540 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73800a9bd0 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7384941700 1 -- 172.21.2.117:0/3288476540 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73800aa160 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7370006b00 con 0x7f73800bfa00 2021-02-10T11:49:52.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 0x7f736401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:52.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.370 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f73700326e0 con 0x7f73800bfa00 2021-02-10T11:49:52.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.371 7f737e61c700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 0x7f736401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:52.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.372 7f737e61c700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 0x7f736401abc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:52.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.401 7f7384941700 1 -- 172.21.2.117:0/3288476540 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738010c2f0 con 0x7f73800bfa00 2021-02-10T11:49:52.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.404 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7370000940 con 0x7f73800bfa00 2021-02-10T11:49:52.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.539 7f7384941700 1 -- 172.21.2.117:0/3288476540 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7380085930 con 0x7f73800bfa00 2021-02-10T11:49:52.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.680 7f7376ffd700 1 -- 172.21.2.117:0/3288476540 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v87) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7370006b00 con 0x7f73800bfa00 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 msgr2=0x7f736401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 0x7f736401abc0 crc :-1 s=READY pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 msgr2=0x7f73800bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7f7370006f40 tx=0x7f737000b990).stop 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 shutdown_connections 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73640186c0 0x7f736401abc0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73800bfa00 0x7f73800bfe00 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:52.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73800c0420 0x7f73800a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:52.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 --2- 172.21.2.117:0/3288476540 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73800be700 0x7f73800bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:52.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 >> 172.21.2.117:0/3288476540 conn(0x7f73800bd990 msgr2=0x7f73800c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:52.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 shutdown_connections 2021-02-10T11:49:52.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:52.707 7f7384941700 1 -- 172.21.2.117:0/3288476540 wait complete. 2021-02-10T11:49:52.711 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:49:53.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.125 7fbf4c616700 1 Processor -- start 2021-02-10T11:49:53.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.125 7fbf4c616700 1 -- start start 2021-02-10T11:49:53.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.126 7fbf4c616700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.126 7fbf4c616700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf440e7d70 0x7fbf440e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.126 7fbf4c616700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf440e77f0 con 0x7fbf440e8790 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.126 7fbf4c616700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440e92f0 con 0x7fbf440e7d70 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.126 7fbf4c616700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440e5ce0 con 0x7fbf441b07f0 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:51984/0 (socket says 172.21.2.117:51984) 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 -- 172.21.2.117:0/1622391996 learned_addr learned my addr 172.21.2.117:0/1622391996 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf49c31700 1 --2- 172.21.2.117:0/1622391996 >> v2:172.21.2.125:3300/0 conn(0x7fbf440e7d70 0x7fbf440e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 -- 172.21.2.117:0/1622391996 >> v2:172.21.2.125:3300/0 conn(0x7fbf440e7d70 msgr2=0x7fbf440e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 --2- 172.21.2.117:0/1622391996 >> v2:172.21.2.125:3300/0 conn(0x7fbf440e7d70 0x7fbf440e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 -- 172.21.2.117:0/1622391996 >> v1:172.21.2.125:6789/0 conn(0x7fbf440e8790 legacy=0x7fbf440e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.127 7fbf4a432700 1 -- 172.21.2.117:0/1622391996 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf440e54b0 con 0x7fbf441b07f0 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf4a432700 1 --2- 172.21.2.117:0/1622391996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbf40006700 tx=0x7fbf40009b80).ready entity=mon.2 client_cookie=b794556ff89bcbdf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf49430700 1 -- 172.21.2.117:0/1622391996 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4000c0c0 con 0x7fbf441b07f0 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf49430700 1 -- 172.21.2.117:0/1622391996 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4000c440 con 0x7fbf441b07f0 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf49430700 1 -- 172.21.2.117:0/1622391996 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4000c960 con 0x7fbf441b07f0 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf4c616700 1 -- 172.21.2.117:0/1622391996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 msgr2=0x7fbf440e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.128 7fbf4c616700 1 --2- 172.21.2.117:0/1622391996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fbf40006700 tx=0x7fbf40009b80).stop 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 -- 172.21.2.117:0/1622391996 shutdown_connections 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 --2- 172.21.2.117:0/1622391996 >> v2:172.21.2.125:3300/0 conn(0x7fbf440e7d70 0x7fbf440e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 --2- 172.21.2.117:0/1622391996 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf441b07f0 0x7fbf440e4e90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 -- 172.21.2.117:0/1622391996 >> 172.21.2.117:0/1622391996 conn(0x7fbf440ce500 msgr2=0x7fbf440e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 -- 172.21.2.117:0/1622391996 shutdown_connections 2021-02-10T11:49:53.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.129 7fbf4c616700 1 -- 172.21.2.117:0/1622391996 wait complete. 2021-02-10T11:49:53.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 Processor -- start 2021-02-10T11:49:53.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 -- start start 2021-02-10T11:49:53.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440a3a60 0x7fbf441a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440b15a0 0x7fbf44093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440b19a0 con 0x7fbf440b0a50 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440930b0 con 0x7fbf440a3a60 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4c616700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf441a3450 con 0x7fbf440b15a0 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf49c31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf49c31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54308/0 (socket says 172.21.2.117:54308) 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf49c31700 1 -- 172.21.2.117:0/709939042 learned_addr learned my addr 172.21.2.117:0/709939042 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4ac33700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440b15a0 0x7fbf44093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.144 7fbf4a432700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440a3a60 0x7fbf441a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 -- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440b15a0 msgr2=0x7fbf44093630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440b15a0 0x7fbf44093630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 -- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440a3a60 msgr2=0x7fbf441a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440a3a60 0x7fbf441a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 -- 172.21.2.117:0/709939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf441a3630 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf49c31700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fbf34006f60 tx=0x7fbf3400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf340009b0 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3400bd60 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf4c616700 1 -- 172.21.2.117:0/709939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf441a3630 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf34002030 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.145 7fbf4c616700 1 -- 172.21.2.117:0/709939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf441a3630 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.146 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf34000c90 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.146 7fbf3affd700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 0x7fbf2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.146 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf34032910 con 0x7fbf440b0a50 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.146 7fbf4a432700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 0x7fbf2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.146 7fbf4a432700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 0x7fbf2c01ac10 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:53.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.176 7fbf4c616700 1 -- 172.21.2.117:0/709939042 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf440b19a0 con 0x7fbf440b0a50 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.178 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf34006b00 con 0x7fbf440b0a50 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.311 7fbf4c616700 1 -- 172.21.2.117:0/709939042 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fbf440930b0 con 0x7fbf2c018730 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.313 7fbf3affd700 1 -- 172.21.2.117:0/709939042 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fbf40000940 con 0x7fbf2c018730 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 msgr2=0x7fbf2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 0x7fbf2c01ac10 crc :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 msgr2=0x7fbf441a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7fbf34006f60 tx=0x7fbf3400b970).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 shutdown_connections 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c018730 0x7fbf2c01ac10 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf440b0a50 0x7fbf441a2e30 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440b15a0 0x7fbf44093630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 --2- 172.21.2.117:0/709939042 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440a3a60 0x7fbf441a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:53.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 >> 172.21.2.117:0/709939042 conn(0x7fbf440a7260 msgr2=0x7fbf440bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 shutdown_connections 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:53.316 7fbf4c616700 1 -- 172.21.2.117:0/709939042 wait complete. 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:49:53.698 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:49:53.698 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 Processor -- start 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 -- start start 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 0x7f484c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f484c0eb040 0x7f484c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f484c0ede60 con 0x7f484c0ee3e0 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c0ebad0 con 0x7f484c0eb040 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.119 7f485333a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f484c0d1cf0 con 0x7f484c0d18f0 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f484c0eb040 0x7f484c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4851957700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f484c0ee3e0 0x7f484c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47492/0 (socket says 172.21.2.117:47492) 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4851957700 1 -- 172.21.2.117:0/461810120 learned_addr learned my addr 172.21.2.117:0/461810120 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:54.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4851156700 1 --2- 172.21.2.117:0/461810120 >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 0x7f484c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 -- 172.21.2.117:0/461810120 >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 msgr2=0x7f484c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 --2- 172.21.2.117:0/461810120 >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 0x7f484c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 -- 172.21.2.117:0/461810120 >> v1:172.21.2.125:6789/0 conn(0x7f484c0ee3e0 legacy=0x7f484c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 -- 172.21.2.117:0/461810120 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0eefd0 con 0x7f484c0eb040 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4851156700 1 --2- 172.21.2.117:0/461810120 >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 0x7f484c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4850955700 1 --2- 172.21.2.117:0/461810120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c0eb040 0x7f484c0eb440 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7f483c006650 tx=0x7f483c009a80).ready entity=mon.0 client_cookie=4c98846dafdba551 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4843fff700 1 -- 172.21.2.117:0/461810120 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c0009b0 con 0x7f484c0eb040 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4843fff700 1 -- 172.21.2.117:0/461810120 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c00c0c0 con 0x7f484c0eb040 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f4843fff700 1 -- 172.21.2.117:0/461810120 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c00c510 con 0x7f484c0eb040 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f485333a700 1 -- 172.21.2.117:0/461810120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c0eb040 msgr2=0x7f484c0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.120 7f485333a700 1 --2- 172.21.2.117:0/461810120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c0eb040 0x7f484c0eb440 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7f483c006650 tx=0x7f483c009a80).stop 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 -- 172.21.2.117:0/461810120 shutdown_connections 2021-02-10T11:49:54.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 --2- 172.21.2.117:0/461810120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c0eb040 0x7f484c0eb440 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 --2- 172.21.2.117:0/461810120 >> v2:172.21.2.125:3300/0 conn(0x7f484c0d18f0 0x7f484c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 -- 172.21.2.117:0/461810120 >> 172.21.2.117:0/461810120 conn(0x7f484c0cf060 msgr2=0x7f484c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:54.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 -- 172.21.2.117:0/461810120 shutdown_connections 2021-02-10T11:49:54.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.121 7f485333a700 1 -- 172.21.2.117:0/461810120 wait complete. 2021-02-10T11:49:54.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.135 7f485333a700 1 Processor -- start 2021-02-10T11:49:54.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.135 7f485333a700 1 -- start start 2021-02-10T11:49:54.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f484c0978f0 0x7f484c19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 0x7f484c0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0981c0 con 0x7f484c1a02a0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c0a0350 con 0x7f484c0978f0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f485333a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f484c097cf0 con 0x7f484c1a06a0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4850955700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4850955700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54322/0 (socket says 172.21.2.117:54322) 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4850955700 1 -- 172.21.2.117:0/9269227 learned_addr learned my addr 172.21.2.117:0/9269227 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4851957700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 0x7f484c0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4851156700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f484c0978f0 0x7f484c19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4850955700 1 -- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 msgr2=0x7f484c0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.136 7f4850955700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 0x7f484c0b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4850955700 1 -- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f484c0978f0 msgr2=0x7f484c19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4850955700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f484c0978f0 0x7f484c19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4850955700 1 -- 172.21.2.117:0/9269227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0b17f0 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4851957700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 0x7f484c0b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4850955700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7f483c009630 tx=0x7f483c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c0093a0 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c00bd70 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f485333a700 1 -- 172.21.2.117:0/9269227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c0b17f0 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f483c005560 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.137 7f485333a700 1 -- 172.21.2.117:0/9269227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c0b17f0 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.138 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f483c005770 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.138 7f4841ffb700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 0x7f482c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.138 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f483c0326b0 con 0x7f484c1a02a0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.138 7f4851156700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 0x7f482c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.138 7f4851156700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 0x7f482c01abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.169 7f485333a700 1 -- 172.21.2.117:0/9269227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c0ede60 con 0x7f484c1a02a0 2021-02-10T11:49:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.172 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f483c02e390 con 0x7f484c1a02a0 2021-02-10T11:49:54.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.321 7f485333a700 1 -- 172.21.2.117:0/9269227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f484c0ede60 con 0x7f484c1a02a0 2021-02-10T11:49:54.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.321 7f4841ffb700 1 -- 172.21.2.117:0/9269227 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f483c02e390 con 0x7f484c1a02a0 2021-02-10T11:49:54.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 msgr2=0x7f482c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 0x7f482c01abc0 crc :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 msgr2=0x7f484c0b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7f483c009630 tx=0x7f483c00b980).stop 2021-02-10T11:49:54.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 shutdown_connections 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f482c0186c0 0x7f482c01abc0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f484c1a02a0 0x7f484c0b0bb0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f484c1a06a0 0x7f484c0b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 --2- 172.21.2.117:0/9269227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f484c0978f0 0x7f484c19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 >> 172.21.2.117:0/9269227 conn(0x7f484c08ab30 msgr2=0x7f484c0935b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 shutdown_connections 2021-02-10T11:49:54.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.323 7f485333a700 1 -- 172.21.2.117:0/9269227 wait complete. 2021-02-10T11:49:54.327 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000294417 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.746 7fddca808700 1 Processor -- start 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.746 7fddca808700 1 -- start start 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc40ef3f0 0x7fddc40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fddc40eb830 0x7fddc41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc40ef7f0 con 0x7fddc40d18f0 2021-02-10T11:49:54.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc40eded0 con 0x7fddc40eb830 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40d1cf0 con 0x7fddc40ef3f0 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fddc40eb830 0x7fddc41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fddc40eb830 0x7fddc41b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54328/0 (socket says 172.21.2.117:54328) 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 -- 172.21.2.117:0/3028154993 learned_addr learned my addr 172.21.2.117:0/3028154993 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 -- 172.21.2.117:0/3028154993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc40ef3f0 msgr2=0x7fddc40eb210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 --2- 172.21.2.117:0/3028154993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc40ef3f0 0x7fddc40eb210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 -- 172.21.2.117:0/3028154993 >> v1:172.21.2.125:6789/0 conn(0x7fddc40d18f0 legacy=0x7fddc40eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 -- 172.21.2.117:0/3028154993 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc41b8fd0 con 0x7fddc40eb830 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc8e25700 1 --2- 172.21.2.117:0/3028154993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc40eb830 0x7fddc41b79a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7fddbc006f00 tx=0x7fddbc009f10).ready entity=mon.0 client_cookie=6f3925ea64edaf9e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc2ffd700 1 -- 172.21.2.117:0/3028154993 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddbc00b3b0 con 0x7fddc40eb830 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc2ffd700 1 -- 172.21.2.117:0/3028154993 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc00b630 con 0x7fddc40eb830 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddc2ffd700 1 -- 172.21.2.117:0/3028154993 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddbc00bb50 con 0x7fddc40eb830 2021-02-10T11:49:54.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 -- 172.21.2.117:0/3028154993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc40eb830 msgr2=0x7fddc41b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.747 7fddca808700 1 --2- 172.21.2.117:0/3028154993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc40eb830 0x7fddc41b79a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7fddbc006f00 tx=0x7fddbc009f10).stop 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 -- 172.21.2.117:0/3028154993 shutdown_connections 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 --2- 172.21.2.117:0/3028154993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc40eb830 0x7fddc41b79a0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 --2- 172.21.2.117:0/3028154993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc40ef3f0 0x7fddc40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 -- 172.21.2.117:0/3028154993 >> 172.21.2.117:0/3028154993 conn(0x7fddc40cf060 msgr2=0x7fddc40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 -- 172.21.2.117:0/3028154993 shutdown_connections 2021-02-10T11:49:54.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.748 7fddca808700 1 -- 172.21.2.117:0/3028154993 wait complete. 2021-02-10T11:49:54.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 Processor -- start 2021-02-10T11:49:54.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 -- start start 2021-02-10T11:49:54.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddc4093be0 0x7fddc41a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc41a7ed0 0x7fddc408c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4094280 con 0x7fddc408c850 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4093fe0 con 0x7fddc4093be0 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddca808700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40ace00 con 0x7fddc41a7ed0 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddc8e25700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddc8e25700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54332/0 (socket says 172.21.2.117:54332) 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddc8e25700 1 -- 172.21.2.117:0/1210727832 learned_addr learned my addr 172.21.2.117:0/1210727832 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddc37fe700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc41a7ed0 0x7fddc408c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.763 7fddc3fff700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddc4093be0 0x7fddc41a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 -- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc41a7ed0 msgr2=0x7fddc408c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc41a7ed0 0x7fddc408c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 -- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddc4093be0 msgr2=0x7fddc41a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddc4093be0 0x7fddc41a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 -- 172.21.2.117:0/1210727832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4094800 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc8e25700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7fddbc009390 tx=0x7fddbc00bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddbc003030 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc0068e0 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddca808700 1 -- 172.21.2.117:0/1210727832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc4094800 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddca808700 1 -- 172.21.2.117:0/1210727832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc4094d90 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.764 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddbc00c570 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.765 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fddbc005820 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.765 7fddc0ff9700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 0x7fdda401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.765 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fddbc032b70 con 0x7fddc408c850 2021-02-10T11:49:54.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.765 7fddc3fff700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 0x7fdda401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:54.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.765 7fddc3fff700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 0x7fdda401ac10 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:54.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.797 7fddca808700 1 -- 172.21.2.117:0/1210727832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc40ede60 con 0x7fddc408c850 2021-02-10T11:49:54.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.799 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fddbc009450 con 0x7fddc408c850 2021-02-10T11:49:54.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.927 7fddca808700 1 -- 172.21.2.117:0/1210727832 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000294417"}) v1 -- 0x7fddc4094d90 con 0x7fdda4018730 2021-02-10T11:49:54.928 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:54.933 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:54.928 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:54.933 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:54.928 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:54.933 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:54.928 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:54.933 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:54.928 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:49:54.933 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:49:54.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.946 7fddc0ff9700 1 -- 172.21.2.117:0/1210727832 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fddb4006b40 con 0x7fdda4018730 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 -- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 msgr2=0x7fdda401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 0x7fdda401ac10 crc :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 -- 172.21.2.117:0/1210727832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 msgr2=0x7fddc408cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7fddbc009390 tx=0x7fddbc00bd40).stop 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 -- 172.21.2.117:0/1210727832 shutdown_connections 2021-02-10T11:49:54.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda4018730 0x7fdda401ac10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddc408c850 0x7fddc408cc50 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddc41a7ed0 0x7fddc408c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 --2- 172.21.2.117:0/1210727832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddc4093be0 0x7fddc41a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.960 7fddca808700 1 -- 172.21.2.117:0/1210727832 >> 172.21.2.117:0/1210727832 conn(0x7fddc4092d00 msgr2=0x7fddc408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.961 7fddca808700 1 -- 172.21.2.117:0/1210727832 shutdown_connections 2021-02-10T11:49:54.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:54.961 7fddca808700 1 -- 172.21.2.117:0/1210727832 wait complete. 2021-02-10T11:49:54.965 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup getpath cephfs group_0000000000294417 2021-02-10T11:49:55.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 Processor -- start 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 -- start start 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0b00d18f0 0x7fa0b00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00eb040 0x7fa0b00eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00eba60 con 0x7fa0b00ee3e0 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b00d1cf0 con 0x7fa0b00d18f0 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.384 7fa0b674b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00eefd0 con 0x7fa0b00eb040 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0b00d18f0 0x7fa0b00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0b00d18f0 0x7fa0b00eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54340/0 (socket says 172.21.2.117:54340) 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 -- 172.21.2.117:0/1866752968 learned_addr learned my addr 172.21.2.117:0/1866752968 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0af7fe700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00eb040 0x7fa0b00eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 -- 172.21.2.117:0/1866752968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00eb040 msgr2=0x7fa0b00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00eb040 0x7fa0b00eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 -- 172.21.2.117:0/1866752968 >> v1:172.21.2.125:6789/0 conn(0x7fa0b00ee3e0 legacy=0x7fa0b00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 -- 172.21.2.117:0/1866752968 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00ef1f0 con 0x7fa0b00d18f0 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0affff700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00d18f0 0x7fa0b00eaa20 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fa0a0006700 tx=0x7fa0a0009b80).ready entity=mon.0 client_cookie=b413b313a257840a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0aeffd700 1 -- 172.21.2.117:0/1866752968 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a000c0c0 con 0x7fa0b00d18f0 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0aeffd700 1 -- 172.21.2.117:0/1866752968 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a000c340 con 0x7fa0b00d18f0 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0aeffd700 1 -- 172.21.2.117:0/1866752968 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a000c820 con 0x7fa0b00d18f0 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0b674b700 1 -- 172.21.2.117:0/1866752968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00d18f0 msgr2=0x7fa0b00eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.385 7fa0b674b700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00d18f0 0x7fa0b00eaa20 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fa0a0006700 tx=0x7fa0a0009b80).stop 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 -- 172.21.2.117:0/1866752968 shutdown_connections 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00eb040 0x7fa0b00eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 --2- 172.21.2.117:0/1866752968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00d18f0 0x7fa0b00eaa20 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 -- 172.21.2.117:0/1866752968 >> 172.21.2.117:0/1866752968 conn(0x7fa0b00cf060 msgr2=0x7fa0b00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:55.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 -- 172.21.2.117:0/1866752968 shutdown_connections 2021-02-10T11:49:55.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.386 7fa0b674b700 1 -- 172.21.2.117:0/1866752968 wait complete. 2021-02-10T11:49:55.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.400 7fa0b674b700 1 Processor -- start 2021-02-10T11:49:55.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.400 7fa0b674b700 1 -- start start 2021-02-10T11:49:55.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 0x7fa0b00dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0b0085ec0 0x7fa0b00862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00a67e0 con 0x7fa0b00a2040 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00a7120 con 0x7fa0b0085ec0 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b674b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00a2a60 con 0x7fa0b00a6ce0 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54344/0 (socket says 172.21.2.117:54344) 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 -- 172.21.2.117:0/618735845 learned_addr learned my addr 172.21.2.117:0/618735845 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0affff700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 0x7fa0b00dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0af7fe700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0b0085ec0 0x7fa0b00862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 -- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 msgr2=0x7fa0b00dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 0x7fa0b00dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 -- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0b0085ec0 msgr2=0x7fa0b00862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0b0085ec0 0x7fa0b00862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 -- 172.21.2.117:0/618735845 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b01a2dc0 con 0x7fa0b00a2040 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0affff700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 0x7fa0b00dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0b4d68700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fa0a8009940 tx=0x7fa0a800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.401 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a8000ce0 con 0x7fa0b00a2040 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a800c2f0 con 0x7fa0b00a2040 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0b674b700 1 -- 172.21.2.117:0/618735845 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b01a2dc0 con 0x7fa0b00a2040 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0b674b700 1 -- 172.21.2.117:0/618735845 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b01a2dc0 con 0x7fa0b00a2040 2021-02-10T11:49:55.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0a800c7d0 con 0x7fa0b00a2040 2021-02-10T11:49:55.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0a8001220 con 0x7fa0b00a2040 2021-02-10T11:49:55.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0acff9700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 0x7fa09001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:55.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa0a8032a80 con 0x7fa0b00a2040 2021-02-10T11:49:55.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.402 7fa0affff700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 0x7fa09001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:55.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.403 7fa0affff700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 0x7fa09001abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:55.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.434 7fa0b674b700 1 -- 172.21.2.117:0/618735845 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b00daff0 con 0x7fa0b00a2040 2021-02-10T11:49:55.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.437 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0a800be30 con 0x7fa0b00a2040 2021-02-10T11:49:55.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.562 7fa0b674b700 1 -- 172.21.2.117:0/618735845 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup getpath", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000294417"}) v1 -- 0x7fa0b00daff0 con 0x7fa0900186c0 2021-02-10T11:49:55.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.564 7fa0acff9700 1 -- 172.21.2.117:0/618735845 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+32 (crc 0 0 0) 0x7fa0a0006e80 con 0x7fa0900186c0 2021-02-10T11:49:55.561 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/group_0000000000294417 2021-02-10T11:49:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 -- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 msgr2=0x7fa09001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 0x7fa09001abc0 crc :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 -- 172.21.2.117:0/618735845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 msgr2=0x7fa0b00a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:55.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fa0a8009940 tx=0x7fa0a800bcf0).stop 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 -- 172.21.2.117:0/618735845 shutdown_connections 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0900186c0 0x7fa09001abc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0b00a2040 0x7fa0b00a2440 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0b0085ec0 0x7fa0b00862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 --2- 172.21.2.117:0/618735845 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0b00a6ce0 0x7fa0b00dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.567 7fa0b674b700 1 -- 172.21.2.117:0/618735845 >> 172.21.2.117:0/618735845 conn(0x7fa0b00b6540 msgr2=0x7fa0b008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.568 7fa0b674b700 1 -- 172.21.2.117:0/618735845 shutdown_connections 2021-02-10T11:49:55.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:55.568 7fa0b674b700 1 -- 172.21.2.117:0/618735845 wait complete. 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> import stat 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> import json 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> import sys 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> try: 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> s = os.stat("/home/ubuntu/cephtest/mnt.0/volumes/group_0000000000294417") 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> except OSError as e: 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> sys.exit(e.errno) 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2021-02-10T11:49:55.572 DEBUG:teuthology.orchestra.run.gibba017:> print(json.dumps( 2021-02-10T11:49:55.573 DEBUG:teuthology.orchestra.run.gibba017:> dict([(a, getattr(s, a)) for a in attrs]), 2021-02-10T11:49:55.573 DEBUG:teuthology.orchestra.run.gibba017:> indent=2)) 2021-02-10T11:49:55.573 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:49:55.964 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:49:55.964 INFO:teuthology.orchestra.run.gibba017.stdout: "st_mode": 16877, 2021-02-10T11:49:55.964 INFO:teuthology.orchestra.run.gibba017.stdout: "st_ino": 1099511627777, 2021-02-10T11:49:55.964 INFO:teuthology.orchestra.run.gibba017.stdout: "st_dev": 42, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_nlink": 2, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_uid": 0, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_gid": 0, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_size": 0, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_atime": 1612957794.9414845, 2021-02-10T11:49:55.965 INFO:teuthology.orchestra.run.gibba017.stdout: "st_mtime": 1612957794.9414845, 2021-02-10T11:49:55.966 INFO:teuthology.orchestra.run.gibba017.stdout: "st_ctime": 1612957794.9470074 2021-02-10T11:49:55.966 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:49:56.147 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000294417 2021-02-10T11:49:56.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 Processor -- start 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 -- start start 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2e680d18f0 0x7f2e680eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2e680ef3f0 0x7f2e681b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e680ef7f0 con 0x7f2e680eb040 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e680ede60 con 0x7f2e680ef3f0 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.606 7f2e6f8ae700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e680d1cf0 con 0x7f2e680d18f0 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2e680ef3f0 0x7f2e681b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6cec9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2e680eb040 0x7f2e680eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54602/0 (socket says 172.21.2.117:54602) 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6cec9700 1 -- 172.21.2.117:0/290668325 learned_addr learned my addr 172.21.2.117:0/290668325 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6d6ca700 1 --2- 172.21.2.117:0/290668325 >> v2:172.21.2.125:3300/0 conn(0x7f2e680d18f0 0x7f2e680eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 -- 172.21.2.117:0/290668325 >> v2:172.21.2.125:3300/0 conn(0x7f2e680d18f0 msgr2=0x7f2e680eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 --2- 172.21.2.117:0/290668325 >> v2:172.21.2.125:3300/0 conn(0x7f2e680d18f0 0x7f2e680eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 -- 172.21.2.117:0/290668325 >> v1:172.21.2.117:6789/0 conn(0x7f2e680eb040 legacy=0x7f2e680eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 -- 172.21.2.117:0/290668325 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e681b8fd0 con 0x7f2e680ef3f0 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e6decb700 1 --2- 172.21.2.117:0/290668325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e680ef3f0 0x7f2e681b79a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f2e60006ab0 tx=0x7f2e60009b40).ready entity=mon.0 client_cookie=efc480dedcc3258a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e5ffff700 1 -- 172.21.2.117:0/290668325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e600022e0 con 0x7f2e680ef3f0 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e5ffff700 1 -- 172.21.2.117:0/290668325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e600024e0 con 0x7f2e680ef3f0 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.607 7f2e5ffff700 1 -- 172.21.2.117:0/290668325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e60002930 con 0x7f2e680ef3f0 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 -- 172.21.2.117:0/290668325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e680ef3f0 msgr2=0x7f2e681b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 --2- 172.21.2.117:0/290668325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e680ef3f0 0x7f2e681b79a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7f2e60006ab0 tx=0x7f2e60009b40).stop 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 -- 172.21.2.117:0/290668325 shutdown_connections 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 --2- 172.21.2.117:0/290668325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e680ef3f0 0x7f2e681b79a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 --2- 172.21.2.117:0/290668325 >> v2:172.21.2.125:3300/0 conn(0x7f2e680d18f0 0x7f2e680eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 -- 172.21.2.117:0/290668325 >> 172.21.2.117:0/290668325 conn(0x7f2e680cf060 msgr2=0x7f2e680cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:56.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 -- 172.21.2.117:0/290668325 shutdown_connections 2021-02-10T11:49:56.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.608 7f2e6f8ae700 1 -- 172.21.2.117:0/290668325 wait complete. 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.623 7f2e6f8ae700 1 Processor -- start 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.623 7f2e6f8ae700 1 -- start start 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e680a6ce0 0x7f2e680dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e680a2040 0x7f2e680a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680a67e0 con 0x7f2e68085ec0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680a7120 con 0x7f2e680a2040 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6f8ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680a2a60 con 0x7f2e680a6ce0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54356/0 (socket says 172.21.2.117:54356) 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 -- 172.21.2.117:0/3452961106 learned_addr learned my addr 172.21.2.117:0/3452961106 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6d6ca700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e680a6ce0 0x7f2e680dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6decb700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e680a2040 0x7f2e680a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 -- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e680a6ce0 msgr2=0x7f2e680dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e680a6ce0 0x7f2e680dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 -- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e680a2040 msgr2=0x7f2e680a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e680a2040 0x7f2e680a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.624 7f2e6cec9700 1 -- 172.21.2.117:0/3452961106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e681a2dc0 con 0x7f2e68085ec0 2021-02-10T11:49:56.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e6cec9700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7f2e58001c60 tx=0x7f2e5800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e5800acc0 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58004140 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2e580045a0 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e681a2dc0 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.625 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e681a3350 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.626 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2e58004320 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.626 7f2e5dffb700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 0x7f2e4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.626 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2e58032780 con 0x7f2e68085ec0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.626 7f2e6d6ca700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 0x7f2e4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:49:56.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.627 7f2e6d6ca700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 0x7f2e4801abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:49:56.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.656 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e680a2c80 con 0x7f2e68085ec0 2021-02-10T11:49:56.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.658 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2e58000dd0 con 0x7f2e68085ec0 2021-02-10T11:49:56.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.783 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000294417"}) v1 -- 0x7f2e681a3be0 con 0x7f2e480186c0 2021-02-10T11:49:56.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.787 7f2e5dffb700 1 -- 172.21.2.117:0/3452961106 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2e64006ce0 con 0x7f2e480186c0 2021-02-10T11:49:56.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.792 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 msgr2=0x7f2e4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.792 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 0x7f2e4801abc0 crc :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.792 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 msgr2=0x7f2e680862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:49:56.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.792 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7f2e58001c60 tx=0x7f2e5800a8d0).stop 2021-02-10T11:49:56.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 shutdown_connections 2021-02-10T11:49:56.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2e480186c0 0x7f2e4801abc0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2e68085ec0 0x7f2e680862c0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2e680a2040 0x7f2e680a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 --2- 172.21.2.117:0/3452961106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2e680a6ce0 0x7f2e680dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:49:56.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 >> 172.21.2.117:0/3452961106 conn(0x7f2e680b6540 msgr2=0x7f2e6808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:49:56.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 shutdown_connections 2021-02-10T11:49:56.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:49:56.793 7f2e6f8ae700 1 -- 172.21.2.117:0/3452961106 wait complete. 2021-02-10T11:49:56.797 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:49:56.797 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:57.144 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:49:57.498 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:49:57.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:57.499 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:57.499 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:57.837 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:57.838 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:49:57.838 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:49:57.838 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:49:57.838 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:58.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[158098]: fuse finished with error 0 and tester_r 0 2021-02-10T11:49:58.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:58.197 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:58.197 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:58.537 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:49:58.537 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:49:58.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:58.537 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:49:58.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:58.880 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:58.880 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:59.217 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:49:59.218 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:49:59.218 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:49:59.218 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:49:59.218 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:59.574 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[158765]: fuse finished with error 0 and tester_r 0 2021-02-10T11:49:59.576 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:49:59.576 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:49:59.576 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:49:59.912 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:49:59.913 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:49:59.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:49:59.913 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:00.247 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:00.247 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:00.248 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:00.584 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:50:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:00.585 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:50:00.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:00.585 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:00.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:00.920 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:00.920 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:01.255 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:50:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:01.256 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:50:01.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:01.256 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:01.555 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 Processor -- start 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 -- start start 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 0x7fe0180e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe0180e8390 0x7fe0180e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180e7580 con 0x7fe0181b07f0 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0180e5ce0 con 0x7fe0180e8390 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.902 7fe01f6d4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180e77f0 con 0x7fe0180e7180 2021-02-10T11:50:01.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.903 7fe01ccef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 0x7fe0180e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.903 7fe01d4f0700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fe0181b07f0 0x7fe0180e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47540/0 (socket says 172.21.2.117:47540) 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.903 7fe01d4f0700 1 -- 172.21.2.117:0/1579799751 learned_addr learned my addr 172.21.2.117:0/1579799751 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.903 7fe01dcf1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe0180e8390 0x7fe0180e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01dcf1700 1 -- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 msgr2=0x7fe0180e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01dcf1700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 0x7fe0180e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01dcf1700 1 -- 172.21.2.117:0/1579799751 >> v1:172.21.2.125:6789/0 conn(0x7fe0181b07f0 legacy=0x7fe0180e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01dcf1700 1 -- 172.21.2.117:0/1579799751 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180e8db0 con 0x7fe0180e8390 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01ccef700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 0x7fe0180e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01dcf1700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0180e8390 0x7fe0180e8790 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe010006e50 tx=0x7fe010009e70).ready entity=mon.1 client_cookie=c9363c7a6aa389cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe00bfff700 1 -- 172.21.2.117:0/1579799751 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01000b510 con 0x7fe0180e8390 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe00bfff700 1 -- 172.21.2.117:0/1579799751 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe010006790 con 0x7fe0180e8390 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe00bfff700 1 -- 172.21.2.117:0/1579799751 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01000b940 con 0x7fe0180e8390 2021-02-10T11:50:01.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01f6d4700 1 -- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0180e8390 msgr2=0x7fe0180e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.904 7fe01f6d4700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0180e8390 0x7fe0180e8790 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7fe010006e50 tx=0x7fe010009e70).stop 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 -- 172.21.2.117:0/1579799751 shutdown_connections 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0180e8390 0x7fe0180e8790 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 --2- 172.21.2.117:0/1579799751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0180e7180 0x7fe0180e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 -- 172.21.2.117:0/1579799751 >> 172.21.2.117:0/1579799751 conn(0x7fe0180ce500 msgr2=0x7fe0180e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 -- 172.21.2.117:0/1579799751 shutdown_connections 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.905 7fe01f6d4700 1 -- 172.21.2.117:0/1579799751 wait complete. 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 Processor -- start 2021-02-10T11:50:01.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 -- start start 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe01809fe90 0x7fe0181ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0181aeed0 0x7fe0181af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180bb5e0 con 0x7fe0180babc0 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180bb7d0 con 0x7fe0181aeed0 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01f6d4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180bb9f0 con 0x7fe01809fe90 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01dcf1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.920 7fe01ccef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0181aeed0 0x7fe0181af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54372/0 (socket says 172.21.2.117:54372) 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 -- 172.21.2.117:0/2892182934 learned_addr learned my addr 172.21.2.117:0/2892182934 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01d4f0700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe01809fe90 0x7fe0181ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 -- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe01809fe90 msgr2=0x7fe0181ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe01809fe90 0x7fe0181ac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 -- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0181aeed0 msgr2=0x7fe0181af2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0181aeed0 0x7fe0181af2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 -- 172.21.2.117:0/2892182934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180a0930 con 0x7fe0180babc0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01dcf1700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fe010006c50 tx=0x7fe01000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe0100093d0 con 0x7fe0180babc0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01000be60 con 0x7fe0180babc0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01000c570 con 0x7fe0180babc0 2021-02-10T11:50:01.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0180a0930 con 0x7fe0180babc0 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.921 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0180a0930 con 0x7fe0180babc0 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.922 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe010002ef0 con 0x7fe0180babc0 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.922 7fe009ffb700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 0x7fdff801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.922 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe010032af0 con 0x7fe0180babc0 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.922 7fe01d4f0700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 0x7fdff801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:01.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.923 7fe01d4f0700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 0x7fdff801abd0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:01.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.952 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0180a0400 con 0x7fe0180babc0 2021-02-10T11:50:01.952 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:01.955 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe010002ef0 con 0x7fe0180babc0 2021-02-10T11:50:02.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.102 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe0180a0400 con 0x7fe0180babc0 2021-02-10T11:50:02.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.106 7fe009ffb700 1 -- 172.21.2.117:0/2892182934 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe010002ef0 con 0x7fe0180babc0 2021-02-10T11:50:02.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 msgr2=0x7fdff801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 0x7fdff801abd0 crc :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 msgr2=0x7fe0180bafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fe010006c50 tx=0x7fe01000bd20).stop 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 shutdown_connections 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdff80186d0 0x7fdff801abd0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0180babc0 0x7fe0180bafc0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0181aeed0 0x7fe0181af2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 --2- 172.21.2.117:0/2892182934 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe01809fe90 0x7fe0181ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 >> 172.21.2.117:0/2892182934 conn(0x7fe018097120 msgr2=0x7fe018096420 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 shutdown_connections 2021-02-10T11:50:02.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.111 7fe01f6d4700 1 -- 172.21.2.117:0/2892182934 wait complete. 2021-02-10T11:50:02.116 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group' 2021-02-10T11:50:02.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 Processor -- start 2021-02-10T11:50:02.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 -- start start 2021-02-10T11:50:02.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff6780d18e0 0x7ff6780ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff6780ef3f0 0x7ff6781b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6780ede60 con 0x7ff6780eb000 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6780ef7f0 con 0x7ff6780ef3f0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.531 7ff68052c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6780d1ce0 con 0x7ff6780d18e0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff6780ef3f0 0x7ff6781b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67db47700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff6780eb000 0x7ff6780eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54626/0 (socket says 172.21.2.117:54626) 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67db47700 1 -- 172.21.2.117:0/1492861519 learned_addr learned my addr 172.21.2.117:0/1492861519 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 -- 172.21.2.117:0/1492861519 >> v2:172.21.2.125:3300/0 conn(0x7ff6780d18e0 msgr2=0x7ff6780ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 --2- 172.21.2.117:0/1492861519 >> v2:172.21.2.125:3300/0 conn(0x7ff6780d18e0 0x7ff6780ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 -- 172.21.2.117:0/1492861519 >> v1:172.21.2.117:6789/0 conn(0x7ff6780eb000 legacy=0x7ff6780eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 -- 172.21.2.117:0/1492861519 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6781b8f90 con 0x7ff6780ef3f0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.532 7ff67eb49700 1 --2- 172.21.2.117:0/1492861519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6780ef3f0 0x7ff6781b7960 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7ff670006e30 tx=0x7ff670009e60).ready entity=mon.0 client_cookie=ad301abb2d9829e4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff67d346700 1 -- 172.21.2.117:0/1492861519 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff670006a50 con 0x7ff6780ef3f0 2021-02-10T11:50:02.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff67d346700 1 -- 172.21.2.117:0/1492861519 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67000b300 con 0x7ff6780ef3f0 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff67d346700 1 -- 172.21.2.117:0/1492861519 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff67000b7e0 con 0x7ff6780ef3f0 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 -- 172.21.2.117:0/1492861519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6780ef3f0 msgr2=0x7ff6781b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 --2- 172.21.2.117:0/1492861519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6780ef3f0 0x7ff6781b7960 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7ff670006e30 tx=0x7ff670009e60).stop 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 -- 172.21.2.117:0/1492861519 shutdown_connections 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 --2- 172.21.2.117:0/1492861519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6780ef3f0 0x7ff6781b7960 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 --2- 172.21.2.117:0/1492861519 >> v2:172.21.2.125:3300/0 conn(0x7ff6780d18e0 0x7ff6780ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.533 7ff68052c700 1 -- 172.21.2.117:0/1492861519 >> 172.21.2.117:0/1492861519 conn(0x7ff6780ced60 msgr2=0x7ff6780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.534 7ff68052c700 1 -- 172.21.2.117:0/1492861519 shutdown_connections 2021-02-10T11:50:02.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.534 7ff68052c700 1 -- 172.21.2.117:0/1492861519 wait complete. 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.548 7ff68052c700 1 Processor -- start 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.548 7ff68052c700 1 -- start start 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 0x7ff6780e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 0x7ff6780e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff678196810 con 0x7ff678196390 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff678098ad0 con 0x7ff6780e9960 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff68052c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6780e8a40 con 0x7ff6780e9560 2021-02-10T11:50:02.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54384/0 (socket says 172.21.2.117:54384) 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 -- 172.21.2.117:0/1171132513 learned_addr learned my addr 172.21.2.117:0/1171132513 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67db47700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 0x7ff6780e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67eb49700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 0x7ff6780e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 -- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 msgr2=0x7ff6780e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 0x7ff6780e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 -- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 msgr2=0x7ff6780e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 0x7ff6780e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.549 7ff67e348700 1 -- 172.21.2.117:0/1171132513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6780e8c60 con 0x7ff678196390 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff67eb49700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 0x7ff6780e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff67db47700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 0x7ff6780e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff67e348700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7ff674006f00 tx=0x7ff67400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:02.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff674006b90 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67400bd30 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6740020d0 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff68052c700 1 -- 172.21.2.117:0/1171132513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6780e8c60 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.550 7ff68052c700 1 -- 172.21.2.117:0/1171132513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67819cd70 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.551 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff674007020 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.551 7ff66affd700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 0x7ff66001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.551 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff6740326f0 con 0x7ff678196390 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.551 7ff67db47700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 0x7ff66001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:02.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.552 7ff67db47700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 0x7ff66001abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:02.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.582 7ff68052c700 1 -- 172.21.2.117:0/1171132513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff678087210 con 0x7ff678196390 2021-02-10T11:50:02.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.584 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff674007ef0 con 0x7ff678196390 2021-02-10T11:50:02.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:02.717 7ff68052c700 1 -- 172.21.2.117:0/1171132513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]} v 0) v1 -- 0x7ff6780ede60 con 0x7ff678196390 2021-02-10T11:50:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.269 7ff66affd700 1 -- 172.21.2.117:0/1171132513 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_default_uid_gid_subvolume_group"]}]=0 v590) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff674007000 con 0x7ff678196390 2021-02-10T11:50:03.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 -- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 msgr2=0x7ff66001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 0x7ff66001abc0 crc :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 -- 172.21.2.117:0/1171132513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 msgr2=0x7ff6780e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7ff674006f00 tx=0x7ff67400b940).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 -- 172.21.2.117:0/1171132513 shutdown_connections 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6780e9960 0x7ff6780e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6780e9560 0x7ff6780e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6600186c0 0x7ff66001abc0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 --2- 172.21.2.117:0/1171132513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff678196390 0x7ff6780e8f40 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 -- 172.21.2.117:0/1171132513 >> 172.21.2.117:0/1171132513 conn(0x7ff67819f610 msgr2=0x7ff6781a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:03.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.287 7ff68052c700 1 -- 172.21.2.117:0/1171132513 shutdown_connections 2021-02-10T11:50:03.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.288 7ff68052c700 1 -- 172.21.2.117:0/1171132513 wait complete. 2021-02-10T11:50:03.293 INFO:tasks.cephfs_test_runner:test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:50:03.293 INFO:tasks.cephfs_test_runner:Starting test: test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:03.294 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 Processor -- start 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 -- start start 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9400d18e0 0x7fa9400ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9400eb000 0x7fa9400eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9400ede60 con 0x7fa9400ee3e0 2021-02-10T11:50:03.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9400eba20 con 0x7fa9400eb000 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.708 7fa9480cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400d1ce0 con 0x7fa9400d18e0 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9400eb000 0x7fa9400eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9466e9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa9400ee3e0 0x7fa9400ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54640/0 (socket says 172.21.2.117:54640) 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9466e9700 1 -- 172.21.2.117:0/1837917199 learned_addr learned my addr 172.21.2.117:0/1837917199 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa945ee8700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9400d18e0 0x7fa9400ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 -- 172.21.2.117:0/1837917199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9400d18e0 msgr2=0x7fa9400ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9400d18e0 0x7fa9400ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 -- 172.21.2.117:0/1837917199 >> v1:172.21.2.117:6789/0 conn(0x7fa9400ee3e0 legacy=0x7fa9400ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 -- 172.21.2.117:0/1837917199 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9400eefd0 con 0x7fa9400eb000 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa9456e7700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400eb000 0x7fa9400eb400 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7fa93c006650 tx=0x7fa93c009a80).ready entity=mon.0 client_cookie=f74994a22257ee5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:03.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa944ee6700 1 -- 172.21.2.117:0/1837917199 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93c00c0c0 con 0x7fa9400eb000 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa944ee6700 1 -- 172.21.2.117:0/1837917199 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c00c340 con 0x7fa9400eb000 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.709 7fa944ee6700 1 -- 172.21.2.117:0/1837917199 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93c00c860 con 0x7fa9400eb000 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 -- 172.21.2.117:0/1837917199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400eb000 msgr2=0x7fa9400eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400eb000 0x7fa9400eb400 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7fa93c006650 tx=0x7fa93c009a80).stop 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 -- 172.21.2.117:0/1837917199 shutdown_connections 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400eb000 0x7fa9400eb400 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 --2- 172.21.2.117:0/1837917199 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9400d18e0 0x7fa9400ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.710 7fa9480cc700 1 -- 172.21.2.117:0/1837917199 >> 172.21.2.117:0/1837917199 conn(0x7fa9400ced60 msgr2=0x7fa9400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.711 7fa9480cc700 1 -- 172.21.2.117:0/1837917199 shutdown_connections 2021-02-10T11:50:03.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.711 7fa9480cc700 1 -- 172.21.2.117:0/1837917199 wait complete. 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.725 7fa9480cc700 1 Processor -- start 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.725 7fa9480cc700 1 -- start start 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9401a4290 0x7fa9401a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9400a6fe0 0x7fa9400a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400a7d10 con 0x7fa9400ab280 2021-02-10T11:50:03.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9400a7a00 con 0x7fa9400a6fe0 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9480cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa940096370 con 0x7fa9401a4290 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54398/0 (socket says 172.21.2.117:54398) 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 -- 172.21.2.117:0/1732970678 learned_addr learned my addr 172.21.2.117:0/1732970678 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 -- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9401a4290 msgr2=0x7fa9401a4690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9401a4290 0x7fa9401a4690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 -- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9400a6fe0 msgr2=0x7fa9400a73e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9400a6fe0 0x7fa9400a73e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 -- 172.21.2.117:0/1732970678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9400a8290 con 0x7fa9400ab280 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.726 7fa9456e7700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7fa93c009630 tx=0x7fa93c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93c0093a0 con 0x7fa9400ab280 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c00bd70 con 0x7fa9400ab280 2021-02-10T11:50:03.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93c005560 con 0x7fa9400ab280 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9400a8290 con 0x7fa9400ab280 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9400a8290 con 0x7fa9400ab280 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa93c000c90 con 0x7fa9400ab280 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.727 7fa9327fc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 0x7fa92801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.728 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa93c0326f0 con 0x7fa9400ab280 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.728 7fa945ee8700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 0x7fa92801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:03.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.728 7fa945ee8700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 0x7fa92801abc0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:03.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.759 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa940087290 con 0x7fa9400ab280 2021-02-10T11:50:03.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.761 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa93c021c40 con 0x7fa9400ab280 2021-02-10T11:50:03.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:03.893 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7fa9400ede60 con 0x7fa9400ab280 2021-02-10T11:50:04.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.272 7fa9327fc700 1 -- 172.21.2.117:0/1732970678 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v591) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa93c021c40 con 0x7fa9400ab280 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 msgr2=0x7fa92801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 0x7fa92801abc0 crc :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 msgr2=0x7fa9400ab680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7fa93c009630 tx=0x7fa93c00b980).stop 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 shutdown_connections 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9280186c0 0x7fa92801abc0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9400ab280 0x7fa9400ab680 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9400a6fe0 0x7fa9400a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 --2- 172.21.2.117:0/1732970678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9401a4290 0x7fa9401a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.313 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 >> 172.21.2.117:0/1732970678 conn(0x7fa9400874c0 msgr2=0x7fa940096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.314 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 shutdown_connections 2021-02-10T11:50:04.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.314 7fa9480cc700 1 -- 172.21.2.117:0/1732970678 wait complete. 2021-02-10T11:50:04.319 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.737 7f2eeb059700 1 Processor -- start 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.737 7f2eeb059700 1 -- start start 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2eeb059700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ee41b69a0 0x7f2ee41b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2eeb059700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40ede60 con 0x7f2ee40ef3f0 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2eeb059700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee40eba00 con 0x7f2ee40d18f0 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2eeb059700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee40d1cf0 con 0x7f2ee41b69a0 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ee41b69a0 0x7f2ee41b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ee41b69a0 0x7f2ee41b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54406/0 (socket says 172.21.2.117:54406) 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 -- 172.21.2.117:0/735994854 learned_addr learned my addr 172.21.2.117:0/735994854 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 -- 172.21.2.117:0/735994854 >> v1:172.21.2.125:6789/0 conn(0x7f2ee40d18f0 legacy=0x7f2ee40eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 -- 172.21.2.117:0/735994854 >> v1:172.21.2.117:6789/0 conn(0x7f2ee40ef3f0 legacy=0x7f2ee40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 -- 172.21.2.117:0/735994854 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee41bb3f0 con 0x7f2ee41b69a0 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.738 7f2ee9676700 1 --2- 172.21.2.117:0/735994854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41b69a0 0x7f2ee41b8db0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f2edc006b80 tx=0x7f2edc009b90).ready entity=mon.0 client_cookie=587cdf217a4a547 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2ee37fe700 1 -- 172.21.2.117:0/735994854 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2edc0022e0 con 0x7f2ee41b69a0 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2ee37fe700 1 -- 172.21.2.117:0/735994854 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc002620 con 0x7f2ee41b69a0 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2ee37fe700 1 -- 172.21.2.117:0/735994854 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2edc002b00 con 0x7f2ee41b69a0 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 -- 172.21.2.117:0/735994854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41b69a0 msgr2=0x7f2ee41b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 --2- 172.21.2.117:0/735994854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41b69a0 0x7f2ee41b8db0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7f2edc006b80 tx=0x7f2edc009b90).stop 2021-02-10T11:50:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 -- 172.21.2.117:0/735994854 shutdown_connections 2021-02-10T11:50:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 --2- 172.21.2.117:0/735994854 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41b69a0 0x7f2ee41b8db0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 -- 172.21.2.117:0/735994854 >> 172.21.2.117:0/735994854 conn(0x7f2ee40cf060 msgr2=0x7f2ee40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 -- 172.21.2.117:0/735994854 shutdown_connections 2021-02-10T11:50:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.739 7f2eeb059700 1 -- 172.21.2.117:0/735994854 wait complete. 2021-02-10T11:50:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 Processor -- start 2021-02-10T11:50:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 -- start start 2021-02-10T11:50:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ee40a2160 0x7f2ee41c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ee41c36c0 0x7f2ee40dead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40c2df0 con 0x7f2ee41c32c0 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40c0980 con 0x7f2ee41c36c0 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.754 7f2eeb059700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40df0f0 con 0x7f2ee40a2160 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54412/0 (socket says 172.21.2.117:54412) 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 -- 172.21.2.117:0/2602926124 learned_addr learned my addr 172.21.2.117:0/2602926124 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 -- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ee40a2160 msgr2=0x7f2ee41c2ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ee40a2160 0x7f2ee41c2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 -- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ee41c36c0 msgr2=0x7f2ee40dead0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ee41c36c0 0x7f2ee40dead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 -- 172.21.2.117:0/2602926124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40df310 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee3fff700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f2ed4009980 tx=0x7f2ed400bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ed4010700 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed4003270 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee40df310 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ed4010c00 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.755 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee40a7a10 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.756 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ed4001270 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.756 7f2ee17fa700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 0x7f2ec801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.756 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2ed4036f40 con 0x7f2ee41c32c0 2021-02-10T11:50:04.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.757 7f2ee8e75700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 0x7f2ec801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:04.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.757 7f2ee8e75700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 0x7f2ec801abc0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:04.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.788 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee4087390 con 0x7f2ee41c32c0 2021-02-10T11:50:04.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.790 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ed4003250 con 0x7f2ee41c32c0 2021-02-10T11:50:04.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.939 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2ee40d18f0 con 0x7f2ee41c32c0 2021-02-10T11:50:04.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.943 7f2ee17fa700 1 -- 172.21.2.117:0/2602926124 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ed4003250 con 0x7f2ee41c32c0 2021-02-10T11:50:04.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 msgr2=0x7f2ec801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 0x7f2ec801abc0 crc :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 msgr2=0x7f2ee40de4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:04.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7f2ed4009980 tx=0x7f2ed400bc50).stop 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 shutdown_connections 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2ec80186c0 0x7f2ec801abc0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ee41c32c0 0x7f2ee40de4b0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ee41c36c0 0x7f2ee40dead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 --2- 172.21.2.117:0/2602926124 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ee40a2160 0x7f2ee41c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:04.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 >> 172.21.2.117:0/2602926124 conn(0x7f2ee40e9120 msgr2=0x7f2ee40c3520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:04.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.948 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 shutdown_connections 2021-02-10T11:50:04.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:04.949 7f2eeb059700 1 -- 172.21.2.117:0/2602926124 wait complete. 2021-02-10T11:50:04.954 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:50:04.954 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:05.300 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:05.646 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:50:05.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:05.647 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:05.647 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:05.983 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:50:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:05.984 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:05.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:05.984 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:05.984 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:06.319 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:50:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:06.320 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:06.320 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:06.320 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:06.320 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:06.655 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:50:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:06.656 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:50:06.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:06.656 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:06.656 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:06.990 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:50:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:06.991 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:50:06.991 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:50:07.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.406 7fc2a1bc2700 1 Processor -- start 2021-02-10T11:50:07.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.406 7fc2a1bc2700 1 -- start start 2021-02-10T11:50:07.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.407 7fc2a1bc2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 0x7fc29c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.407 7fc2a1bc2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 0x7fc29c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.407 7fc2a1bc2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c085930 con 0x7fc29c0e4e40 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.407 7fc2a1bc2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c085680 con 0x7fc29c0e5240 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.407 7fc2a1bc2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1b09e0 con 0x7fc29c084c60 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 0x7fc29c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc292ffd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 0x7fc29c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29b7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fc29c0e4e40 0x7fc29c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47592/0 (socket says 172.21.2.117:47592) 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29b7fe700 1 -- 172.21.2.117:0/2017638986 learned_addr learned my addr 172.21.2.117:0/2017638986 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29bfff700 1 -- 172.21.2.117:0/2017638986 >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 msgr2=0x7fc29c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29bfff700 1 --2- 172.21.2.117:0/2017638986 >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 0x7fc29c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29bfff700 1 -- 172.21.2.117:0/2017638986 >> v1:172.21.2.125:6789/0 conn(0x7fc29c0e4e40 legacy=0x7fc29c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29bfff700 1 -- 172.21.2.117:0/2017638986 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c084840 con 0x7fc29c084c60 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc29affd700 1 -- 172.21.2.117:0/2017638986 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281208186 0 0) 0x7fc28c0010c0 con 0x7fc29c0e4e40 2021-02-10T11:50:07.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.408 7fc292ffd700 1 --2- 172.21.2.117:0/2017638986 >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 0x7fc29c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc29bfff700 1 --2- 172.21.2.117:0/2017638986 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 0x7fc29c085060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fc294006e50 tx=0x7fc294009e70).ready entity=mon.2 client_cookie=9c91240a876836ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc29affd700 1 -- 172.21.2.117:0/2017638986 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc29400b510 con 0x7fc29c084c60 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc29affd700 1 -- 172.21.2.117:0/2017638986 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc294006790 con 0x7fc29c084c60 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc29affd700 1 -- 172.21.2.117:0/2017638986 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc29400b940 con 0x7fc29c084c60 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc2a1bc2700 1 -- 172.21.2.117:0/2017638986 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 msgr2=0x7fc29c085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.409 7fc2a1bc2700 1 --2- 172.21.2.117:0/2017638986 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 0x7fc29c085060 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fc294006e50 tx=0x7fc294009e70).stop 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 -- 172.21.2.117:0/2017638986 shutdown_connections 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 --2- 172.21.2.117:0/2017638986 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c084c60 0x7fc29c085060 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 --2- 172.21.2.117:0/2017638986 >> v2:172.21.2.125:3300/0 conn(0x7fc29c0e5240 0x7fc29c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 -- 172.21.2.117:0/2017638986 >> 172.21.2.117:0/2017638986 conn(0x7fc29c0ce4e0 msgr2=0x7fc29c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 -- 172.21.2.117:0/2017638986 shutdown_connections 2021-02-10T11:50:07.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.410 7fc2a1bc2700 1 -- 172.21.2.117:0/2017638986 wait complete. 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.424 7fc2a1bc2700 1 Processor -- start 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.424 7fc2a1bc2700 1 -- start start 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc29c110bd0 0x7fc29c1a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c09b510 0x7fc29c09b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1a3be0 con 0x7fc29c0b63a0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1a3e00 con 0x7fc29c110bd0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc2a1bc2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c09bf30 con 0x7fc29c09b510 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54422/0 (socket says 172.21.2.117:54422) 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 -- 172.21.2.117:0/1146191294 learned_addr learned my addr 172.21.2.117:0/1146191294 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 -- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c09b510 msgr2=0x7fc29c09b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c09b510 0x7fc29c09b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 -- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc29c110bd0 msgr2=0x7fc29c1a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc29c110bd0 0x7fc29c1a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.425 7fc29b7fe700 1 -- 172.21.2.117:0/1146191294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c09c150 con 0x7fc29c0b63a0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc29b7fe700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7fc28c00a5d0 tx=0x7fc28c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc28c000c10 con 0x7fc29c0b63a0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c0070d0 con 0x7fc29c0b63a0 2021-02-10T11:50:07.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c09c150 con 0x7fc29c0b63a0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c0b6d60 con 0x7fc29c0b63a0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.426 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc28c0047d0 con 0x7fc29c0b63a0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.427 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc28c000c10 con 0x7fc29c0b63a0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.427 7fc298ff9700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 0x7fc28001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.427 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc28c0326e0 con 0x7fc29c0b63a0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.427 7fc292ffd700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 0x7fc28001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:07.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.427 7fc292ffd700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 0x7fc28001abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:07.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.457 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c085930 con 0x7fc29c0b63a0 2021-02-10T11:50:07.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.459 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc28c020d50 con 0x7fc29c0b63a0 2021-02-10T11:50:07.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.584 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc29c085930 con 0x7fc29c0b63a0 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.585 7fc298ff9700 1 -- 172.21.2.117:0/1146191294 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v109) v1 ==== 81+0+30765 (secure 0 0 0) 0x7fc28c020fb0 con 0x7fc29c0b63a0 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 109, 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:50:07.582 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:49:30.190724", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:50:07.583 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 23, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:50:07.584 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:50:07.585 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:50:07.586 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:50:07.587 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:50:07.588 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 22, 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:49:27.162612", 2021-02-10T11:50:07.589 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:50:07.590 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:50:07.591 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "108", 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:50:07.592 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:50:07.593 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.594 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:50:07.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 23, 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:49:28.233491", 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:50:07.596 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:50:07.597 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "109", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:50:07.598 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:50:07.599 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:50:07.600 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.601 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:50:07.602 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.603 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.604 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.605 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:50:07.606 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:50:07.607 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 106, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.608 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.610 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.611 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.612 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:50:07.613 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.614 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.615 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.616 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.617 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.618 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:50:07.619 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.620 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:50:07.621 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.623 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.623 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.623 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.624 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.625 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.626 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:50:07.627 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.628 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:50:07.629 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.630 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.631 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.633 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.635 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.636 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.637 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.638 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.639 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:50:07.640 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 105, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.641 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.642 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:50:07.645 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:50:07.646 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 98, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:50:07.647 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:50:07.648 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:50:07.651 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.652 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.653 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:50:07.654 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.655 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:50:07.656 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:50:07.657 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:50:07.658 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 msgr2=0x7fc28001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 0x7fc28001abc0 crc :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 msgr2=0x7fc29c1a2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7fc28c00a5d0 tx=0x7fc28c00a970).stop 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 shutdown_connections 2021-02-10T11:50:07.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc29c09b510 0x7fc29c09b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc29c110bd0 0x7fc29c1a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2800186c0 0x7fc28001abc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 --2- 172.21.2.117:0/1146191294 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc29c0b63a0 0x7fc29c1a2fa0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 >> 172.21.2.117:0/1146191294 conn(0x7fc29c0b5bd0 msgr2=0x7fc29c0c39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 shutdown_connections 2021-02-10T11:50:07.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:07.587 7fc2a1bc2700 1 -- 172.21.2.117:0/1146191294 wait complete. 2021-02-10T11:50:07.661 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:08.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 Processor -- start 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 -- start start 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4681b6960 0x7fd4681b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680ede60 con 0x7fd4680ef3f0 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4680eb9c0 con 0x7fd4680d18e0 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46fdce700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4680ef7f0 con 0x7fd4681b6960 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46e3eb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4681b6960 0x7fd4681b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.075 7fd46d3e9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd4680ef3f0 0x7fd4680eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54680/0 (socket says 172.21.2.117:54680) 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46d3e9700 1 -- 172.21.2.117:0/3057775789 learned_addr learned my addr 172.21.2.117:0/3057775789 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46e3eb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4681b6960 0x7fd4681b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54434/0 (socket says 172.21.2.117:54434) 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46e3eb700 1 -- 172.21.2.117:0/3057775789 >> v1:172.21.2.125:6789/0 conn(0x7fd4680d18e0 legacy=0x7fd4680ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46e3eb700 1 -- 172.21.2.117:0/3057775789 >> v1:172.21.2.117:6789/0 conn(0x7fd4680ef3f0 legacy=0x7fd4680eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46e3eb700 1 -- 172.21.2.117:0/3057775789 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4680d1d20 con 0x7fd4681b6960 2021-02-10T11:50:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46e3eb700 1 --2- 172.21.2.117:0/3057775789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b6960 0x7fd4681b8d70 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fd460006e30 tx=0x7fd460009e60).ready entity=mon.0 client_cookie=bee689b72721d57 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46cbe8700 1 -- 172.21.2.117:0/3057775789 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd460003030 con 0x7fd4681b6960 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46cbe8700 1 -- 172.21.2.117:0/3057775789 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46000b300 con 0x7fd4681b6960 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46cbe8700 1 -- 172.21.2.117:0/3057775789 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd46000b820 con 0x7fd4681b6960 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46fdce700 1 -- 172.21.2.117:0/3057775789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b6960 msgr2=0x7fd4681b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.076 7fd46fdce700 1 --2- 172.21.2.117:0/3057775789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b6960 0x7fd4681b8d70 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7fd460006e30 tx=0x7fd460009e60).stop 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.077 7fd46fdce700 1 -- 172.21.2.117:0/3057775789 shutdown_connections 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.077 7fd46fdce700 1 --2- 172.21.2.117:0/3057775789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b6960 0x7fd4681b8d70 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.077 7fd46fdce700 1 -- 172.21.2.117:0/3057775789 >> 172.21.2.117:0/3057775789 conn(0x7fd4680ced60 msgr2=0x7fd4680cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.077 7fd46fdce700 1 -- 172.21.2.117:0/3057775789 shutdown_connections 2021-02-10T11:50:08.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.077 7fd46fdce700 1 -- 172.21.2.117:0/3057775789 wait complete. 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.091 7fd46fdce700 1 Processor -- start 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 -- start start 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 0x7fd4680ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4680ee7c0 0x7fd4680eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680efd40 con 0x7fd4681b8410 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4681b8940 con 0x7fd4680f5980 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46fdce700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4680ee270 con 0x7fd4680ee7c0 2021-02-10T11:50:08.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54438/0 (socket says 172.21.2.117:54438) 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 -- 172.21.2.117:0/4109074908 learned_addr learned my addr 172.21.2.117:0/4109074908 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46e3eb700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4680ee7c0 0x7fd4680eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46d3e9700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 0x7fd4680ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 -- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4680ee7c0 msgr2=0x7fd4680eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4680ee7c0 0x7fd4680eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 -- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 msgr2=0x7fd4680ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 0x7fd4680ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46dbea700 1 -- 172.21.2.117:0/4109074908 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4680ef1e0 con 0x7fd4681b8410 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.092 7fd46d3e9700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 0x7fd4680ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd46dbea700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fd46400b5b0 tx=0x7fd46400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd46400bd40 con 0x7fd4681b8410 2021-02-10T11:50:08.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd464006240 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4680ef1e0 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd464005720 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.093 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4680ef1e0 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.094 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4640054a0 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.094 7fd45a7fc700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 0x7fd45001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.094 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd464032940 con 0x7fd4681b8410 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.094 7fd46d3e9700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 0x7fd45001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.094 7fd46d3e9700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 0x7fd45001abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.125 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd468086140 con 0x7fd4681b8410 2021-02-10T11:50:08.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.128 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4640009b0 con 0x7fd4681b8410 2021-02-10T11:50:08.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.254 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4680d18e0 con 0x7fd4681b8410 2021-02-10T11:50:08.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.254 7fd45a7fc700 1 -- 172.21.2.117:0/4109074908 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fd46400bd40 con 0x7fd4681b8410 2021-02-10T11:50:08.251 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 87 2021-02-10T11:50:08.252 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:08.252 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":87,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:49:31.199085","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7801":{"gid":7801,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1582108166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1582108166},{"type":"v1","addr":"172.21.2.125:6835","nonce":1582108166}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:50:08.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.256 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 msgr2=0x7fd45001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.256 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 0x7fd45001abc0 crc :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.256 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 msgr2=0x7fd4680eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.256 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7fd46400b5b0 tx=0x7fd46400b950).stop 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 shutdown_connections 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4680ee7c0 0x7fd4680eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4680f5980 0x7fd4680ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd450018730 0x7fd45001abc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 --2- 172.21.2.117:0/4109074908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4681b8410 0x7fd4680eda80 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 >> 172.21.2.117:0/4109074908 conn(0x7fd468086d40 msgr2=0x7fd46809c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 shutdown_connections 2021-02-10T11:50:08.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.257 7fd46fdce700 1 -- 172.21.2.117:0/4109074908 wait complete. 2021-02-10T11:50:08.262 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 Processor -- start 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 -- start start 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb6540e7180 0x7fb6540e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540e7580 con 0x7fb6540e8390 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540e77f0 con 0x7fb6541b07f0 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb658baf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6540e5ce0 con 0x7fb6540e7180 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb65205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb6540e7180 0x7fb6540e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb65305f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb6540e8390 0x7fb6540e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54698/0 (socket says 172.21.2.117:54698) 2021-02-10T11:50:08.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.677 7fb65305f700 1 -- 172.21.2.117:0/709801191 learned_addr learned my addr 172.21.2.117:0/709801191 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65185c700 1 -- 172.21.2.117:0/709801191 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116915654 0 0) 0x7fb64c001500 con 0x7fb6540e8390 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65205d700 1 -- 172.21.2.117:0/709801191 >> v1:172.21.2.125:6789/0 conn(0x7fb6541b07f0 legacy=0x7fb6540e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65205d700 1 -- 172.21.2.117:0/709801191 >> v1:172.21.2.117:6789/0 conn(0x7fb6540e8390 legacy=0x7fb6540e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65205d700 1 -- 172.21.2.117:0/709801191 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6540e8f80 con 0x7fb6540e7180 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65205d700 1 --2- 172.21.2.117:0/709801191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540e7180 0x7fb6540e7d70 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7fb644006650 tx=0x7fb644009a80).ready entity=mon.0 client_cookie=fe97bc2a3d29fb12 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65185c700 1 -- 172.21.2.117:0/709801191 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb6440009b0 con 0x7fb6540e7180 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65185c700 1 -- 172.21.2.117:0/709801191 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64400c0c0 con 0x7fb6540e7180 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb65185c700 1 -- 172.21.2.117:0/709801191 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb64400c520 con 0x7fb6540e7180 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb658baf700 1 -- 172.21.2.117:0/709801191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540e7180 msgr2=0x7fb6540e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.678 7fb658baf700 1 --2- 172.21.2.117:0/709801191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540e7180 0x7fb6540e7d70 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7fb644006650 tx=0x7fb644009a80).stop 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.679 7fb658baf700 1 -- 172.21.2.117:0/709801191 shutdown_connections 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.679 7fb658baf700 1 --2- 172.21.2.117:0/709801191 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540e7180 0x7fb6540e7d70 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:08.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.679 7fb658baf700 1 -- 172.21.2.117:0/709801191 >> 172.21.2.117:0/709801191 conn(0x7fb6540ce500 msgr2=0x7fb6540e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:08.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.679 7fb658baf700 1 -- 172.21.2.117:0/709801191 shutdown_connections 2021-02-10T11:50:08.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.679 7fb658baf700 1 -- 172.21.2.117:0/709801191 wait complete. 2021-02-10T11:50:08.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 Processor -- start 2021-02-10T11:50:08.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 -- start start 2021-02-10T11:50:08.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb65409a3c0 0x7fb6541a3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb6541a3850 0x7fb6541a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540b4390 con 0x7fb6540b3970 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540b45b0 con 0x7fb6541a3850 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb658baf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540b47d0 con 0x7fb65409a3c0 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb65305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb65305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54452/0 (socket says 172.21.2.117:54452) 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.694 7fb65305f700 1 -- 172.21.2.117:0/1711768096 learned_addr learned my addr 172.21.2.117:0/1711768096 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 -- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb65409a3c0 msgr2=0x7fb6541a3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb65409a3c0 0x7fb6541a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 -- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb6541a3850 msgr2=0x7fb6541a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb6541a3850 0x7fb6541a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 -- 172.21.2.117:0/1711768096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6540b49f0 con 0x7fb6540b3970 2021-02-10T11:50:08.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb65305f700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fb64c001c10 tx=0x7fb64c00ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb64c0070d0 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64c007350 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb64c0049a0 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb658baf700 1 -- 172.21.2.117:0/1711768096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6540b49f0 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.695 7fb658baf700 1 -- 172.21.2.117:0/1711768096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb654109910 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.696 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb64c004720 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.696 7fb64b7fe700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 0x7fb63801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.696 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb64c032a10 con 0x7fb6540b3970 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.696 7fb65285e700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 0x7fb63801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:08.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.697 7fb65285e700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 0x7fb63801abc0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:08.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.727 7fb658baf700 1 -- 172.21.2.117:0/1711768096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb654108ac0 con 0x7fb6540b3970 2021-02-10T11:50:08.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.730 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb64c0149d0 con 0x7fb6540b3970 2021-02-10T11:50:08.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:08.856 7fb658baf700 1 -- 172.21.2.117:0/1711768096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb6540e77f0 con 0x7fb6540b3970 2021-02-10T11:50:09.255 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:50:09.259 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:50:09.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.262 7fb64b7fe700 1 -- 172.21.2.117:0/1711768096 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb64c0149d0 con 0x7fb6540b3970 2021-02-10T11:50:09.268 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 -- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 msgr2=0x7fb63801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 0x7fb63801abc0 crc :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 -- 172.21.2.117:0/1711768096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 msgr2=0x7fb6540b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7fb64c001c10 tx=0x7fb64c00ac60).stop 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 -- 172.21.2.117:0/1711768096 shutdown_connections 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb6380186c0 0x7fb63801abc0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb6540b3970 0x7fb6540b3d70 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb6541a3850 0x7fb6541a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:09.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 --2- 172.21.2.117:0/1711768096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb65409a3c0 0x7fb6541a3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:09.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.273 7fb658baf700 1 -- 172.21.2.117:0/1711768096 >> 172.21.2.117:0/1711768096 conn(0x7fb654111150 msgr2=0x7fb654187d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:09.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.274 7fb658baf700 1 -- 172.21.2.117:0/1711768096 shutdown_connections 2021-02-10T11:50:09.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.274 7fb658baf700 1 -- 172.21.2.117:0/1711768096 wait complete. 2021-02-10T11:50:09.278 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:09.305 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:50:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.692 7f51dadc8700 1 Processor -- start 2021-02-10T11:50:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.692 7f51dadc8700 1 -- start start 2021-02-10T11:50:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51dadc8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51d40d18e0 0x7f51d40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51dadc8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f51d40ef3f0 0x7f51d41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51dadc8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40ede60 con 0x7f51d40eb000 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51dadc8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40ef7f0 con 0x7f51d40d18e0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51dadc8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d40d1ce0 con 0x7f51d40ef3f0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d8be4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f51d40d18e0 0x7f51d40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f51d40eb000 0x7f51d40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54708/0 (socket says 172.21.2.117:54708) 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d3fff700 1 -- 172.21.2.117:0/3711989378 learned_addr learned my addr 172.21.2.117:0/3711989378 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d93e5700 1 --2- 172.21.2.117:0/3711989378 >> v2:172.21.2.125:3300/0 conn(0x7f51d40ef3f0 0x7f51d41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d93e5700 1 -- 172.21.2.117:0/3711989378 >> v2:172.21.2.117:3300/0 conn(0x7f51d40d18e0 msgr2=0x7f51d40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d93e5700 1 --2- 172.21.2.117:0/3711989378 >> v2:172.21.2.117:3300/0 conn(0x7f51d40d18e0 0x7f51d40ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d93e5700 1 -- 172.21.2.117:0/3711989378 >> v1:172.21.2.117:6789/0 conn(0x7f51d40eb000 legacy=0x7f51d40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.693 7f51d93e5700 1 -- 172.21.2.117:0/3711989378 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d41b8f90 con 0x7f51d40ef3f0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51d93e5700 1 --2- 172.21.2.117:0/3711989378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ef3f0 0x7f51d41b7960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f51cc006e50 tx=0x7f51cc009e70).ready entity=mon.1 client_cookie=3a3eba2bef6713c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51d37fe700 1 -- 172.21.2.117:0/3711989378 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51cc006a20 con 0x7f51d40ef3f0 2021-02-10T11:50:09.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51d37fe700 1 -- 172.21.2.117:0/3711989378 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc006790 con 0x7f51d40ef3f0 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51d37fe700 1 -- 172.21.2.117:0/3711989378 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51cc00b700 con 0x7f51d40ef3f0 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 -- 172.21.2.117:0/3711989378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ef3f0 msgr2=0x7f51d41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 --2- 172.21.2.117:0/3711989378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ef3f0 0x7f51d41b7960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f51cc006e50 tx=0x7f51cc009e70).stop 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 -- 172.21.2.117:0/3711989378 shutdown_connections 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 --2- 172.21.2.117:0/3711989378 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ef3f0 0x7f51d41b7960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 --2- 172.21.2.117:0/3711989378 >> v2:172.21.2.117:3300/0 conn(0x7f51d40d18e0 0x7f51d40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 -- 172.21.2.117:0/3711989378 >> 172.21.2.117:0/3711989378 conn(0x7f51d40ced60 msgr2=0x7f51d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 -- 172.21.2.117:0/3711989378 shutdown_connections 2021-02-10T11:50:09.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.694 7f51dadc8700 1 -- 172.21.2.117:0/3711989378 wait complete. 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.709 7f51dadc8700 1 Processor -- start 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.709 7f51dadc8700 1 -- start start 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51d41a4290 0x7f51d41a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 0x7f51d40ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 0x7f51d40a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40a7d10 con 0x7f51d41a4290 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40a7a00 con 0x7f51d40ab280 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51dadc8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d4096370 con 0x7f51d40a6fe0 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 0x7f51d40ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d8be4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51d41a4290 0x7f51d41a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 0x7f51d40ab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:58848/0 (socket says 172.21.2.117:58848) 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d3fff700 1 -- 172.21.2.117:0/191786201 learned_addr learned my addr 172.21.2.117:0/191786201 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 0x7f51d40a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 -- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 msgr2=0x7f51d40ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 0x7f51d40ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 -- 172.21.2.117:0/191786201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51d41a4290 msgr2=0x7f51d41a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51d41a4290 0x7f51d41a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 -- 172.21.2.117:0/191786201 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40a8290 con 0x7f51d40a6fe0 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.710 7f51d93e5700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 0x7f51d40a73e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f51cc00b880 tx=0x7f51cc00bc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51cc000df0 con 0x7f51d40a6fe0 2021-02-10T11:50:09.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc009730 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51dadc8700 1 -- 172.21.2.117:0/191786201 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d40a8290 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51dadc8700 1 -- 172.21.2.117:0/191786201 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d40a8290 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f51cc00c520 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.711 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f51cc00bd60 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.712 7f51d17fa700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 0x7f51b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.712 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f51cc032a00 con 0x7f51d40a6fe0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.712 7f51d8be4700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 0x7f51b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:09.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.712 7f51d8be4700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 0x7f51b801abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:09.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.742 7f51dadc8700 1 -- 172.21.2.117:0/191786201 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d4087290 con 0x7f51d40a6fe0 2021-02-10T11:50:09.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.748 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f51cc030080 con 0x7f51d40a6fe0 2021-02-10T11:50:09.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.876 7f51dadc8700 1 -- 172.21.2.117:0/191786201 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d40ede60 con 0x7f51d40a6fe0 2021-02-10T11:50:09.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.876 7f51d17fa700 1 -- 172.21.2.117:0/191786201 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+1757 (secure 0 0 0) 0x7f51cc030080 con 0x7f51d40a6fe0 2021-02-10T11:50:09.873 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:09.873 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":88,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10449,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/682457859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":682457859},{"type":"v1","addr":"172.21.2.117:6833","nonce":682457859}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:49:29.188327","modified":"2021-02-10 11:50:09.259732","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":11}]} 2021-02-10T11:50:09.873 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 88 2021-02-10T11:50:09.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 msgr2=0x7f51b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 0x7f51b801abc0 crc :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 msgr2=0x7f51d40a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:09.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 0x7f51d40a73e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f51cc00b880 tx=0x7f51cc00bc20).stop 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 shutdown_connections 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f51d40a6fe0 0x7f51d40a73e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f51d40ab280 0x7f51d40ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f51b80186c0 0x7f51b801abc0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 --2- 172.21.2.117:0/191786201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f51d41a4290 0x7f51d41a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:09.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 >> 172.21.2.117:0/191786201 conn(0x7f51d40874c0 msgr2=0x7f51d4096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:09.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 shutdown_connections 2021-02-10T11:50:09.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:09.878 7f51dadc8700 1 -- 172.21.2.117:0/191786201 wait complete. 2021-02-10T11:50:09.883 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.302 7fcf9ca5a700 1 Processor -- start 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.302 7fcf9ca5a700 1 -- start start 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9ca5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980e7180 0x7fcf980e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9ca5a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcf980e8390 0x7fcf980e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9ca5a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980e7580 con 0x7fcf981b07f0 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9ca5a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980e5ce0 con 0x7fcf980e8390 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9ca5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980e77f0 con 0x7fcf980e7180 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9685e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcf981b07f0 0x7fcf980e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54720/0 (socket says 172.21.2.117:54720) 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9685e700 1 -- 172.21.2.117:0/3547756519 learned_addr learned my addr 172.21.2.117:0/3547756519 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:10.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9605d700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980e7180 0x7fcf980e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9705f700 1 --2- 172.21.2.117:0/3547756519 >> v2:172.21.2.125:3300/0 conn(0x7fcf980e8390 0x7fcf980e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9585c700 1 -- 172.21.2.117:0/3547756519 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790091473 0 0) 0x7fcf88000ce0 con 0x7fcf981b07f0 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.303 7fcf9585c700 1 -- 172.21.2.117:0/3547756519 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf84002fe0 con 0x7fcf981b07f0 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9705f700 1 -- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980e7180 msgr2=0x7fcf980e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9705f700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980e7180 0x7fcf980e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9705f700 1 -- 172.21.2.117:0/3547756519 >> v1:172.21.2.117:6789/0 conn(0x7fcf981b07f0 legacy=0x7fcf980e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9705f700 1 -- 172.21.2.117:0/3547756519 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980e8db0 con 0x7fcf980e8390 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9705f700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf980e8390 0x7fcf980e8790 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fcf90006e50 tx=0x7fcf90009e70).ready entity=mon.1 client_cookie=2c8e5c33ba55b263 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9585c700 1 -- 172.21.2.117:0/3547756519 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9000b510 con 0x7fcf980e8390 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9585c700 1 -- 172.21.2.117:0/3547756519 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90006790 con 0x7fcf980e8390 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9585c700 1 -- 172.21.2.117:0/3547756519 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf9000b9d0 con 0x7fcf980e8390 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9ca5a700 1 -- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf980e8390 msgr2=0x7fcf980e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:10.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.304 7fcf9ca5a700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf980e8390 0x7fcf980e8790 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fcf90006e50 tx=0x7fcf90009e70).stop 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 -- 172.21.2.117:0/3547756519 shutdown_connections 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf980e8390 0x7fcf980e8790 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 --2- 172.21.2.117:0/3547756519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980e7180 0x7fcf980e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 -- 172.21.2.117:0/3547756519 >> 172.21.2.117:0/3547756519 conn(0x7fcf980ce500 msgr2=0x7fcf980e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 -- 172.21.2.117:0/3547756519 shutdown_connections 2021-02-10T11:50:10.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.305 7fcf9ca5a700 1 -- 172.21.2.117:0/3547756519 wait complete. 2021-02-10T11:50:10.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.319 7fcf9ca5a700 1 Processor -- start 2021-02-10T11:50:10.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 -- start start 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980ad6d0 0x7fcf980adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf981ad4b0 0x7fcf981ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98093f70 con 0x7fcf981aded0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98197350 con 0x7fcf981ad4b0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9ca5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98197570 con 0x7fcf980ad6d0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54474/0 (socket says 172.21.2.117:54474) 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9705f700 1 -- 172.21.2.117:0/3261682709 learned_addr learned my addr 172.21.2.117:0/3261682709 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9605d700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf981ad4b0 0x7fcf981ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.320 7fcf9685e700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980ad6d0 0x7fcf980adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 -- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980ad6d0 msgr2=0x7fcf980adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:10.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980ad6d0 0x7fcf980adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 -- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf981ad4b0 msgr2=0x7fcf981ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf981ad4b0 0x7fcf981ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 -- 172.21.2.117:0/3261682709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98197790 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9705f700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7fcf90009eb0 tx=0x7fcf9000bd60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf90009650 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90001070 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf90001500 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf98197790 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.321 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf98197d20 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.322 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf900066f0 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.322 7fcf8f7fe700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 0x7fcf7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.322 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fcf90032a90 con 0x7fcf981aded0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.322 7fcf9685e700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 0x7fcf7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:10.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.323 7fcf9685e700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 0x7fcf7c01abc0 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:10.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.353 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf980e77f0 con 0x7fcf981aded0 2021-02-10T11:50:10.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.355 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf90009140 con 0x7fcf981aded0 2021-02-10T11:50:10.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:10.485 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcf980e77f0 con 0x7fcf981aded0 2021-02-10T11:50:11.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.273 7fcf8f7fe700 1 -- 172.21.2.117:0/3261682709 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcf900066f0 con 0x7fcf981aded0 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 msgr2=0x7fcf7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 0x7fcf7c01abc0 crc :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 msgr2=0x7fcf981ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7fcf90009eb0 tx=0x7fcf9000bd60).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 shutdown_connections 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf7c0186c0 0x7fcf7c01abc0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf981aded0 0x7fcf981ae2d0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf981ad4b0 0x7fcf981ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 --2- 172.21.2.117:0/3261682709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf980ad6d0 0x7fcf980adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 >> 172.21.2.117:0/3261682709 conn(0x7fcf981998b0 msgr2=0x7fcf981a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:11.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 shutdown_connections 2021-02-10T11:50:11.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.307 7fcf9ca5a700 1 -- 172.21.2.117:0/3261682709 wait complete. 2021-02-10T11:50:11.312 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:50:11.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.730 7f99d7879700 1 Processor -- start 2021-02-10T11:50:11.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.730 7f99d7879700 1 -- start start 2021-02-10T11:50:11.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d7879700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d0085450 0x7f99d0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d7879700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d0085930 con 0x7f99d00e4e40 2021-02-10T11:50:11.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d7879700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d01b09e0 con 0x7f99d00e5240 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d7879700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00e5cc0 con 0x7f99d0085450 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d5695700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f99d00e4e40 0x7f99d01b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54732/0 (socket says 172.21.2.117:54732) 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.731 7f99d5695700 1 -- 172.21.2.117:0/3279745433 learned_addr learned my addr 172.21.2.117:0/3279745433 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367289150 0 0) 0x7f99cc001180 con 0x7f99d00e4e40 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc002fe0 con 0x7f99d00e4e40 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99d5e96700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d0085450 0x7f99d0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3215151736 0 0) 0x7f99cc001180 con 0x7f99d00e4e40 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d0085450 msgr2=0x7f99d0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d0085450 0x7f99d0086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 >> v1:172.21.2.125:6789/0 conn(0x7f99d00e5240 legacy=0x7f99d0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d0084840 con 0x7f99d00e4e40 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f99cc001680 con 0x7f99d00e4e40 2021-02-10T11:50:11.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 >> v1:172.21.2.117:6789/0 conn(0x7f99d00e4e40 legacy=0x7f99d01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99bc003b40 0x7f99bc006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99bc006840 0x7f99bc008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99bc0095b0 0x7f99bc00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc00c360 con 0x7f99bc0095b0 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc00c580 con 0x7f99bc003b40 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc00c7a0 con 0x7f99bc006840 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99d5695700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99bc003b40 0x7f99bc006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.732 7f99d5e96700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d0085450 0x7f99d0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d4e94700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99bc006840 0x7f99bc008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5e96700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99bc0095b0 0x7f99bc00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 -- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99bc006840 msgr2=0x7f99bc008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99bc006840 0x7f99bc008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 -- 172.21.2.117:0/3279745433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99bc0095b0 msgr2=0x7f99bc00bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:11.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99bc0095b0 0x7f99bc00bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 -- 172.21.2.117:0/3279745433 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99bc050a10 con 0x7f99bc003b40 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d4e94700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99bc006840 0x7f99bc008d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99d5695700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99bc003b40 0x7f99bc006020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f99cc007680 tx=0x7f99cc001c10).ready entity=mon.1 client_cookie=7d024de7d08a0589 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.733 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99cc000ce0 con 0x7f99bc003b40 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc0070c0 con 0x7f99bc003b40 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99c7fff700 1 -- 172.21.2.117:0/3279745433 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99cc00a7a0 con 0x7f99bc003b40 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99d7879700 1 -- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99bc003b40 msgr2=0x7f99bc006020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99d7879700 1 --2- 172.21.2.117:0/3279745433 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99bc003b40 0x7f99bc006020 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f99cc007680 tx=0x7f99cc001c10).stop 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99d5e96700 1 -- 172.21.2.117:0/3279745433 reap_dead start 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99d7879700 1 -- 172.21.2.117:0/3279745433 shutdown_connections 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.734 7f99d7879700 1 -- 172.21.2.117:0/3279745433 >> 172.21.2.117:0/3279745433 conn(0x7f99d00ce4e0 msgr2=0x7f99d00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:11.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.735 7f99d7879700 1 -- 172.21.2.117:0/3279745433 shutdown_connections 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.735 7f99d7879700 1 -- 172.21.2.117:0/3279745433 wait complete. 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.749 7f99d7879700 1 Processor -- start 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.749 7f99d7879700 1 -- start start 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d019bf50 0x7f99d009c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d00a0860 0x7f99d010f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00a0c60 con 0x7f99d019bf50 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d010e930 con 0x7f99d00a0860 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d7879700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d010f8c0 con 0x7f99d010e530 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:52176/0 (socket says 172.21.2.117:52176) 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 -- 172.21.2.117:0/3430030625 learned_addr learned my addr 172.21.2.117:0/3430030625 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:11.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d5695700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d019bf50 0x7f99d009c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d5e96700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d00a0860 0x7f99d010f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 -- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d00a0860 msgr2=0x7f99d010f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d00a0860 0x7f99d010f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 -- 172.21.2.117:0/3430030625 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d019bf50 msgr2=0x7f99d009c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d019bf50 0x7f99d009c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 -- 172.21.2.117:0/3430030625 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d010fae0 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.750 7f99d4e94700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f99c0006b10 tx=0x7f99c000b9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c000bda0 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c0002030 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99d7879700 1 -- 172.21.2.117:0/3430030625 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d010fae0 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99c0002480 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99d7879700 1 -- 172.21.2.117:0/3430030625 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d010fae0 con 0x7f99d010e530 2021-02-10T11:50:11.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f99c0000cb0 con 0x7f99d010e530 2021-02-10T11:50:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 0x7f99b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f99c00327c0 con 0x7f99d010e530 2021-02-10T11:50:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.751 7f99d5695700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 0x7f99b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:11.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.752 7f99d5695700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 0x7f99b001ac10 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.780 7f99d7879700 1 -- 172.21.2.117:0/3430030625 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d010ed70 con 0x7f99d010e530 2021-02-10T11:50:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.783 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f99c001f4d0 con 0x7f99d010e530 2021-02-10T11:50:11.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:11.910 7f99d7879700 1 -- 172.21.2.117:0/3430030625 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f99d010ed70 con 0x7f99d010e530 2021-02-10T11:50:12.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.275 7f99c5ffb700 1 -- 172.21.2.117:0/3430030625 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v111) v1 ==== 185+0+0 (secure 0 0 0) 0x7f99c001f6e0 con 0x7f99d010e530 2021-02-10T11:50:12.274 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:50:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 msgr2=0x7f99b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 0x7f99b001ac10 crc :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 msgr2=0x7f99d00a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f99c0006b10 tx=0x7f99c000b9b0).stop 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 shutdown_connections 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99d00a0860 0x7f99d010f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f99d010e530 0x7f99d00a0240 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99b0018730 0x7f99b001ac10 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 --2- 172.21.2.117:0/3430030625 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f99d019bf50 0x7f99d009c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 >> 172.21.2.117:0/3430030625 conn(0x7f99d009d5b0 msgr2=0x7f99d00b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 shutdown_connections 2021-02-10T11:50:12.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.279 7f99d7879700 1 -- 172.21.2.117:0/3430030625 wait complete. 2021-02-10T11:50:12.284 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.702 7f662123e700 1 Processor -- start 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.702 7f662123e700 1 -- start start 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f662123e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c085450 0x7f661c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f662123e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c085930 con 0x7f661c0e4e40 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f662123e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c1b09e0 con 0x7f661c0e5240 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f662123e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0e5cc0 con 0x7f661c085450 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f661ad9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f661c0e4e40 0x7f661c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54752/0 (socket says 172.21.2.117:54752) 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f661ad9d700 1 -- 172.21.2.117:0/3343366276 learned_addr learned my addr 172.21.2.117:0/3343366276 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053787176 0 0) 0x7f6604001180 con 0x7f661c0e4e40 2021-02-10T11:50:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6608002fe0 con 0x7f661c0e4e40 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.703 7f661b59e700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c085450 0x7f661c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1674870257 0 0) 0x7f6604001180 con 0x7f661c0e4e40 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c085450 msgr2=0x7f661c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c085450 0x7f661c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 >> v1:172.21.2.125:6789/0 conn(0x7f661c0e5240 legacy=0x7f661c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c084840 con 0x7f661c0e4e40 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6604001680 con 0x7f661c0e4e40 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 >> v1:172.21.2.117:6789/0 conn(0x7f661c0e4e40 legacy=0x7f661c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6608003b10 0x7f6608006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66080067a0 0x7f6608008cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66080095c0 0x7f660800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660800c2c0 con 0x7f66080067a0 2021-02-10T11:50:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660800c4e0 con 0x7f6608003b10 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6608050710 con 0x7f66080095c0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66080067a0 0x7f6608008cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661ad9d700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6608003b10 0x7f6608006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661b59e700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66080095c0 0x7f660800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 -- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66080095c0 msgr2=0x7f660800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f66080095c0 0x7f660800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 -- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6608003b10 msgr2=0x7f6608006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6608003b10 0x7f6608006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 -- 172.21.2.117:0/3343366276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6608050970 con 0x7f66080067a0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661ad9d700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6608003b10 0x7f6608006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.704 7f661a59c700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66080067a0 0x7f6608008cc0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7f660c006ac0 tx=0x7f660c009af0).ready entity=mon.0 client_cookie=8f5600d3f14b17da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c007280 con 0x7f66080067a0 2021-02-10T11:50:12.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c007580 con 0x7f66080067a0 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f6619d9b700 1 -- 172.21.2.117:0/3343366276 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c0079e0 con 0x7f66080067a0 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f662123e700 1 -- 172.21.2.117:0/3343366276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66080067a0 msgr2=0x7f6608008cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f662123e700 1 --2- 172.21.2.117:0/3343366276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f66080067a0 0x7f6608008cc0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7f660c006ac0 tx=0x7f660c009af0).stop 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f661b59e700 1 -- 172.21.2.117:0/3343366276 reap_dead start 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f662123e700 1 -- 172.21.2.117:0/3343366276 shutdown_connections 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f662123e700 1 -- 172.21.2.117:0/3343366276 >> 172.21.2.117:0/3343366276 conn(0x7f661c0ce4e0 msgr2=0x7f661c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.705 7f662123e700 1 -- 172.21.2.117:0/3343366276 shutdown_connections 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.706 7f662123e700 1 -- 172.21.2.117:0/3343366276 wait complete. 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.720 7f662123e700 1 Processor -- start 2021-02-10T11:50:12.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.720 7f662123e700 1 -- start start 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c09fa30 0x7f661c09fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c093b50 0x7f661c093f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0a08d0 con 0x7f661c0a0dd0 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0c2890 con 0x7f661c093b50 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f662123e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0c2420 con 0x7f661c09fa30 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54512/0 (socket says 172.21.2.117:54512) 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 -- 172.21.2.117:0/1844492082 learned_addr learned my addr 172.21.2.117:0/1844492082 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661a59c700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c09fa30 0x7f661c09fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661b59e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c093b50 0x7f661c093f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 -- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c09fa30 msgr2=0x7f661c09fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c09fa30 0x7f661c09fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 -- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c093b50 msgr2=0x7f661c093f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c093b50 0x7f661c093f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 -- 172.21.2.117:0/1844492082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0a0450 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.721 7f661ad9d700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f6604009560 tx=0x7f6604009900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f66040070c0 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66040043a0 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6604004880 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f662123e700 1 -- 172.21.2.117:0/1844492082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c0a0450 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f662123e700 1 -- 172.21.2.117:0/1844492082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c094570 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6604008000 con 0x7f661c0a0dd0 2021-02-10T11:50:12.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:12.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.722 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f6604032420 con 0x7f661c0a0dd0 2021-02-10T11:50:12.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.723 7f661a59c700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:12.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.723 7f661a59c700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 0x7f660001abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:12.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.752 7f662123e700 1 -- 172.21.2.117:0/1844492082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c10c2a0 con 0x7f661c0a0dd0 2021-02-10T11:50:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.755 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6604030ce0 con 0x7f661c0a0dd0 2021-02-10T11:50:12.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:12.881 7f662123e700 1 -- 172.21.2.117:0/1844492082 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f661c085930 con 0x7f661c0a0dd0 2021-02-10T11:50:13.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.281 7f66137fe700 1 -- 172.21.2.117:0/1844492082 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v112) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6604008000 con 0x7f661c0a0dd0 2021-02-10T11:50:13.294 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:50:13.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 msgr2=0x7f660001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 0x7f660001abc0 crc :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 msgr2=0x7f661c19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f6604009560 tx=0x7f6604009900).stop 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 shutdown_connections 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c093b50 0x7f661c093f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c09fa30 0x7f661c09fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f66000186c0 0x7f660001abc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 --2- 172.21.2.117:0/1844492082 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0a0dd0 0x7f661c19b990 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 >> 172.21.2.117:0/1844492082 conn(0x7f661c09d5a0 msgr2=0x7f661c0c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:13.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 shutdown_connections 2021-02-10T11:50:13.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.299 7f662123e700 1 -- 172.21.2.117:0/1844492082 wait complete. 2021-02-10T11:50:13.304 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:50:13.305 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:50:13.305 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:50:13.305 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:13.305 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:50:13.305 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:50:13.305 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:50:13.305 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:50:13.311 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:50:13.311 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:50:13.312 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:50:13.312 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:50:13.354 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:50:13.354 DEBUG:teuthology.parallel:result is None 2021-02-10T11:50:13.355 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:50:13.355 DEBUG:teuthology.parallel:result is None 2021-02-10T11:50:13.355 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:50:13.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:13.466 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:13.761 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:50:13.762 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 Processor -- start 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 -- start start 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4cd81b07f0 0x7f4cd80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd80e77f0 con 0x7f4cd80e8d80 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd80e7550 con 0x7f4cd80e6170 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4ce0227700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd80e54b0 con 0x7f4cd81b07f0 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4cde844700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4cd80e8d80 0x7f4cd80e6d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54786/0 (socket says 172.21.2.117:54786) 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4cde844700 1 -- 172.21.2.117:0/2551751872 learned_addr learned my addr 172.21.2.117:0/2551751872 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.772 7f4cde043700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4cd81b07f0 0x7f4cd80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:13.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cdd041700 1 -- 172.21.2.117:0/2551751872 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974308729 0 0) 0x7f4cd0001500 con 0x7f4cd80e8d80 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cdd041700 1 -- 172.21.2.117:0/2551751872 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc4002fe0 con 0x7f4cd80e8d80 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cde043700 1 -- 172.21.2.117:0/2551751872 >> v1:172.21.2.125:6789/0 conn(0x7f4cd80e6170 legacy=0x7f4cd80e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cde043700 1 -- 172.21.2.117:0/2551751872 >> v1:172.21.2.117:6789/0 conn(0x7f4cd80e8d80 legacy=0x7f4cd80e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cde043700 1 -- 172.21.2.117:0/2551751872 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd80e9180 con 0x7f4cd81b07f0 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cde043700 1 --2- 172.21.2.117:0/2551751872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd81b07f0 0x7f4cd80e4e90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f4ccc006650 tx=0x7f4ccc009a00).ready entity=mon.0 client_cookie=fdce8a878e9b79cb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cdd041700 1 -- 172.21.2.117:0/2551751872 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ccc0009b0 con 0x7f4cd81b07f0 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cdd041700 1 -- 172.21.2.117:0/2551751872 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc00c0c0 con 0x7f4cd81b07f0 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4cdd041700 1 -- 172.21.2.117:0/2551751872 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ccc00c5a0 con 0x7f4cd81b07f0 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4ce0227700 1 -- 172.21.2.117:0/2551751872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd81b07f0 msgr2=0x7f4cd80e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.773 7f4ce0227700 1 --2- 172.21.2.117:0/2551751872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd81b07f0 0x7f4cd80e4e90 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f4ccc006650 tx=0x7f4ccc009a00).stop 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.774 7f4ce0227700 1 -- 172.21.2.117:0/2551751872 shutdown_connections 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.774 7f4ce0227700 1 --2- 172.21.2.117:0/2551751872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd81b07f0 0x7f4cd80e4e90 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.774 7f4ce0227700 1 -- 172.21.2.117:0/2551751872 >> 172.21.2.117:0/2551751872 conn(0x7f4cd80ce500 msgr2=0x7f4cd80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:13.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.774 7f4ce0227700 1 -- 172.21.2.117:0/2551751872 shutdown_connections 2021-02-10T11:50:13.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.774 7f4ce0227700 1 -- 172.21.2.117:0/2551751872 wait complete. 2021-02-10T11:50:13.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.789 7f4ce0227700 1 Processor -- start 2021-02-10T11:50:13.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.789 7f4ce0227700 1 -- start start 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 0x7f4cd819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4cd819f6d0 0x7f4cd8108c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80bfa90 con 0x7f4cd8097920 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c11c0 con 0x7f4cd8098340 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4ce0227700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80974f0 con 0x7f4cd819f6d0 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54538/0 (socket says 172.21.2.117:54538) 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 -- 172.21.2.117:0/1773530893 learned_addr learned my addr 172.21.2.117:0/1773530893 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde844700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4cd819f6d0 0x7f4cd8108c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:13.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cdd842700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 0x7f4cd819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 -- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4cd819f6d0 msgr2=0x7f4cd8108c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4cd819f6d0 0x7f4cd8108c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 -- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 msgr2=0x7f4cd819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 0x7f4cd819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 -- 172.21.2.117:0/1773530893 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd819fad0 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cdd842700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 0x7f4cd819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.790 7f4cde043700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f4ccc00b5e0 tx=0x7f4ccc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ccc0090c0 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc0009b0 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd819fad0 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ccc0022a0 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd8109290 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ccc000c90 con 0x7f4cd8097920 2021-02-10T11:50:13.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 0x7f4cc001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:13.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.791 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f4ccc032610 con 0x7f4cd8097920 2021-02-10T11:50:13.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.792 7f4cdd842700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 0x7f4cc001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:13.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.792 7f4cdd842700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 0x7f4cc001ac10 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.821 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd8108a40 con 0x7f4cd8097920 2021-02-10T11:50:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.824 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ccc0090c0 con 0x7f4cd8097920 2021-02-10T11:50:13.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:13.965 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f4cd80e77f0 con 0x7f4cd8097920 2021-02-10T11:50:14.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.285 7f4ccaffd700 1 -- 172.21.2.117:0/1773530893 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v113) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4ccc000c90 con 0x7f4cd8097920 2021-02-10T11:50:14.328 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:50:14.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.332 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 msgr2=0x7f4cc001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.332 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 0x7f4cc001ac10 crc :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.332 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 msgr2=0x7f4cd8097d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.332 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f4ccc00b5e0 tx=0x7f4ccc00b980).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 shutdown_connections 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4cd819f6d0 0x7f4cd8108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4cd8098340 0x7f4cd819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cc0018730 0x7f4cc001ac10 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 --2- 172.21.2.117:0/1773530893 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4cd8097920 0x7f4cd8097d20 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 >> 172.21.2.117:0/1773530893 conn(0x7f4cd80950b0 msgr2=0x7f4cd80a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 shutdown_connections 2021-02-10T11:50:14.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.333 7f4ce0227700 1 -- 172.21.2.117:0/1773530893 wait complete. 2021-02-10T11:50:14.338 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:50:14.339 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:50:14.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.752 7f2332555700 1 Processor -- start 2021-02-10T11:50:14.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.752 7f2332555700 1 -- start start 2021-02-10T11:50:14.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2332555700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 0x7f232c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:14.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2332555700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0ede60 con 0x7f232c0eb7f0 2021-02-10T11:50:14.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2332555700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0d1ce0 con 0x7f232c0d18e0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2332555700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0ef7f0 con 0x7f232c0ef3f0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2330b72700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f232c0eb7f0 0x7f232c1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54798/0 (socket says 172.21.2.117:54798) 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f2330b72700 1 -- 172.21.2.117:0/3204096789 learned_addr learned my addr 172.21.2.117:0/3204096789 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f232b7fe700 1 --2- 172.21.2.117:0/3204096789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 0x7f232c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f232affd700 1 -- 172.21.2.117:0/3204096789 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861948141 0 0) 0x7f2324001500 con 0x7f232c0eb7f0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f232affd700 1 -- 172.21.2.117:0/3204096789 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314002fe0 con 0x7f232c0eb7f0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f232affd700 1 -- 172.21.2.117:0/3204096789 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902364215 0 0) 0x7f231c001080 con 0x7f232c0d18e0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.753 7f232affd700 1 -- 172.21.2.117:0/3204096789 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314003210 con 0x7f232c0d18e0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232b7fe700 1 -- 172.21.2.117:0/3204096789 >> v1:172.21.2.125:6789/0 conn(0x7f232c0d18e0 legacy=0x7f232c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232b7fe700 1 -- 172.21.2.117:0/3204096789 >> v1:172.21.2.117:6789/0 conn(0x7f232c0eb7f0 legacy=0x7f232c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232b7fe700 1 -- 172.21.2.117:0/3204096789 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c1bb180 con 0x7f232c0ef3f0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232b7fe700 1 --2- 172.21.2.117:0/3204096789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 0x7f232c0eb1d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2320006650 tx=0x7f2320009a80).ready entity=mon.2 client_cookie=b703f0baeb560ff9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:14.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232affd700 1 -- 172.21.2.117:0/3204096789 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f232000c0c0 con 0x7f232c0ef3f0 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232affd700 1 -- 172.21.2.117:0/3204096789 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232000c340 con 0x7f232c0ef3f0 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f232affd700 1 -- 172.21.2.117:0/3204096789 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f232000c860 con 0x7f232c0ef3f0 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f2332555700 1 -- 172.21.2.117:0/3204096789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 msgr2=0x7f232c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f2332555700 1 --2- 172.21.2.117:0/3204096789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 0x7f232c0eb1d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2320006650 tx=0x7f2320009a80).stop 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f2332555700 1 -- 172.21.2.117:0/3204096789 shutdown_connections 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.754 7f2332555700 1 --2- 172.21.2.117:0/3204096789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c0ef3f0 0x7f232c0eb1d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.755 7f2332555700 1 -- 172.21.2.117:0/3204096789 >> 172.21.2.117:0/3204096789 conn(0x7f232c0ced60 msgr2=0x7f232c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.755 7f2332555700 1 -- 172.21.2.117:0/3204096789 shutdown_connections 2021-02-10T11:50:14.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.755 7f2332555700 1 -- 172.21.2.117:0/3204096789 wait complete. 2021-02-10T11:50:14.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 Processor -- start 2021-02-10T11:50:14.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 -- start start 2021-02-10T11:50:14.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f232c0afa20 0x7f232c08dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c1b0380 0x7f232c1b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c08d730 con 0x7f232c097c10 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c098630 con 0x7f232c0afa20 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.769 7f2332555700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c1b0da0 con 0x7f232c1b0380 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54552/0 (socket says 172.21.2.117:54552) 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 -- 172.21.2.117:0/2807145829 learned_addr learned my addr 172.21.2.117:0/2807145829 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 -- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c1b0380 msgr2=0x7f232c1b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c1b0380 0x7f232c1b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 -- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f232c0afa20 msgr2=0x7f232c08dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f232c0afa20 0x7f232c08dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 -- 172.21.2.117:0/2807145829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c08a5a0 con 0x7f232c097c10 2021-02-10T11:50:14.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f232b7fe700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f2320009650 tx=0x7f232000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2320009300 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320006030 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f2332555700 1 -- 172.21.2.117:0/2807145829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c08a5a0 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f2332555700 1 -- 172.21.2.117:0/2807145829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c08ab30 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.770 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2320005560 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.771 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2320009300 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.771 7f2328ff9700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 0x7f230c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.771 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2320032360 con 0x7f232c097c10 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.771 7f232bfff700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 0x7f230c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:14.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.772 7f232bfff700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 0x7f230c01abc0 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:14.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.803 7f2332555700 1 -- 172.21.2.117:0/2807145829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f232c087290 con 0x7f232c097c10 2021-02-10T11:50:14.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.805 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2320030cf0 con 0x7f232c097c10 2021-02-10T11:50:14.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.937 7f2332555700 1 -- 172.21.2.117:0/2807145829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f232c0ede60 con 0x7f232c097c10 2021-02-10T11:50:14.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.938 7f2328ff9700 1 -- 172.21.2.117:0/2807145829 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v66) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f232000bd50 con 0x7f232c097c10 2021-02-10T11:50:14.936 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:14.936 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:50:14.936 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:50:14.936 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:50:14.937 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.938 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.939 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.940 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.941 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.942 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.943 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:50:14.944 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:50:14.945 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:50:14.946 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:50:14.947 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:50:14.948 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 msgr2=0x7f230c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 0x7f230c01abc0 crc :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 msgr2=0x7f232c098010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f2320009650 tx=0x7f232000b960).stop 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 shutdown_connections 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f230c0186c0 0x7f230c01abc0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f232c097c10 0x7f232c098010 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f232c1b0380 0x7f232c1b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 --2- 172.21.2.117:0/2807145829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f232c0afa20 0x7f232c08dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 >> 172.21.2.117:0/2807145829 conn(0x7f232c1a3620 msgr2=0x7f232c0e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 shutdown_connections 2021-02-10T11:50:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:14.940 7f2332555700 1 -- 172.21.2.117:0/2807145829 wait complete. 2021-02-10T11:50:14.950 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:15.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.363 7fc3d3f42700 1 Processor -- start 2021-02-10T11:50:15.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 -- start start 2021-02-10T11:50:15.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc0ef3f0 0x7fc3cc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc3cc0eb7f0 0x7fc3cc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0ede60 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3cc0ef7f0 con 0x7fc3cc0eb7f0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d3f42700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc0d1ce0 con 0x7fc3cc0ef3f0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d1d5e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc3cc0d18e0 0x7fc3cc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54810/0 (socket says 172.21.2.117:54810) 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.364 7fc3d1d5e700 1 -- 172.21.2.117:0/2707209259 learned_addr learned my addr 172.21.2.117:0/2707209259 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d255f700 1 --2- 172.21.2.117:0/2707209259 >> v2:172.21.2.125:3300/0 conn(0x7fc3cc0eb7f0 0x7fc3cc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d155d700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc0ef3f0 0x7fc3cc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381110104 0 0) 0x7fc3c00010b0 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b8002fe0 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 915709566 0 0) 0x7fc3c00010b0 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc0ef3f0 msgr2=0x7fc3cc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:15.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc0ef3f0 0x7fc3cc0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 >> v2:172.21.2.125:3300/0 conn(0x7fc3cc0eb7f0 msgr2=0x7fc3cc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 --2- 172.21.2.117:0/2707209259 >> v2:172.21.2.125:3300/0 conn(0x7fc3cc0eb7f0 0x7fc3cc1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc1b8f90 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fc3c00015b0 con 0x7fc3cc0d18e0 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 >> v1:172.21.2.117:6789/0 conn(0x7fc3cc0d18e0 legacy=0x7fc3cc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3b8003b10 0x7fc3b8005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3b804a7a0 0x7fc3b804cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3b804d500 0x7fc3b804fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80502d0 con 0x7fc3b804d500 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80504f0 con 0x7fc3b804a7a0 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b8050710 con 0x7fc3b8003b10 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d255f700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3b804d500 0x7fc3b804fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.365 7fc3d1d5e700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3b8003b10 0x7fc3b8005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d155d700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3b804a7a0 0x7fc3b804cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 -- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3b8003b10 msgr2=0x7fc3b8005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3b8003b10 0x7fc3b8005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 -- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3b804a7a0 msgr2=0x7fc3b804cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3b804a7a0 0x7fc3b804cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 -- 172.21.2.117:0/2707209259 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80509b0 con 0x7fc3b804d500 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3b804d500 0x7fc3b804fa20 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7fc3c4006a40 tx=0x7fc3c4009f40).ready entity=mon.0 client_cookie=5f6fd78f51a42679 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c400c420 con 0x7fc3b804d500 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c400c720 con 0x7fc3b804d500 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d0d5c700 1 -- 172.21.2.117:0/2707209259 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c400cc00 con 0x7fc3b804d500 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d3f42700 1 -- 172.21.2.117:0/2707209259 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3b804d500 msgr2=0x7fc3b804fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d3f42700 1 --2- 172.21.2.117:0/2707209259 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3b804d500 0x7fc3b804fa20 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7fc3c4006a40 tx=0x7fc3c4009f40).stop 2021-02-10T11:50:15.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.366 7fc3d255f700 1 -- 172.21.2.117:0/2707209259 reap_dead start 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.367 7fc3d3f42700 1 -- 172.21.2.117:0/2707209259 shutdown_connections 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.367 7fc3d3f42700 1 -- 172.21.2.117:0/2707209259 >> 172.21.2.117:0/2707209259 conn(0x7fc3cc0ced60 msgr2=0x7fc3cc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.367 7fc3d3f42700 1 -- 172.21.2.117:0/2707209259 shutdown_connections 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.367 7fc3d3f42700 1 -- 172.21.2.117:0/2707209259 wait complete. 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 Processor -- start 2021-02-10T11:50:15.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 -- start start 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 0x7fc3cc0e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc08c600 0x7fc3cc08a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc08b660 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc08ca00 con 0x7fc3cc0b2160 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d3f42700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc0e9a80 con 0x7fc3cc08c600 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d155d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d155d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54574/0 (socket says 172.21.2.117:54574) 2021-02-10T11:50:15.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d155d700 1 -- 172.21.2.117:0/2328760514 learned_addr learned my addr 172.21.2.117:0/2328760514 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d1d5e700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 0x7fc3cc0e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.382 7fc3d255f700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc08c600 0x7fc3cc08a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 -- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc08c600 msgr2=0x7fc3cc08a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc08c600 0x7fc3cc08a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 -- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 msgr2=0x7fc3cc0e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 0x7fc3cc0e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 -- 172.21.2.117:0/2328760514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc08a8a0 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d1d5e700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 0x7fc3cc0e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d155d700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7fc3c8006f20 tx=0x7fc3c800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c800bd80 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c8000c90 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3c80022b0 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc08a8a0 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.383 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc08a8a0 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.384 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3c800a000 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.384 7fc3be7fc700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.384 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fc3c80323c0 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.384 7fc3d1d5e700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.384 7fc3d1d5e700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 0x7fc3b401abc0 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:15.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.414 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc0ede60 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.416 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc3c802df00 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.539 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3cc0ede60 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.539 7fc3be7fc700 1 -- 172.21.2.117:0/2328760514 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+11386 (secure 0 0 0) 0x7fc3c802df00 con 0x7fc3cc08bbe0 2021-02-10T11:50:15.538 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:15.538 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":113,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:14.281105","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:15.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 msgr2=0x7fc3b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 0x7fc3b401abc0 crc :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 msgr2=0x7fc3cc08bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7fc3c8006f20 tx=0x7fc3c800b990).stop 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 shutdown_connections 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3b40186c0 0x7fc3b401abc0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3cc08bbe0 0x7fc3cc08bfe0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3cc08c600 0x7fc3cc08a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 --2- 172.21.2.117:0/2328760514 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc3cc0b2160 0x7fc3cc0e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 >> 172.21.2.117:0/2328760514 conn(0x7fc3cc0bb5e0 msgr2=0x7fc3cc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.541 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 shutdown_connections 2021-02-10T11:50:15.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.542 7fc3d3f42700 1 -- 172.21.2.117:0/2328760514 wait complete. 2021-02-10T11:50:15.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:50:15.548 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.965 7f455dfd6700 1 Processor -- start 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.965 7f455dfd6700 1 -- start start 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f455dfd6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f45580ef3f0 0x7f45580eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f455dfd6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f45580eb830 0x7f45581b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f455dfd6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580ef7f0 con 0x7f45580d18f0 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f455dfd6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45580ede60 con 0x7f45580eb830 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f455dfd6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45580d1cf0 con 0x7f45580ef3f0 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f4557fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f45580eb830 0x7f45581b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f45577fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f45580d18f0 0x7f45580eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54830/0 (socket says 172.21.2.117:54830) 2021-02-10T11:50:15.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f45577fe700 1 -- 172.21.2.117:0/133439480 learned_addr learned my addr 172.21.2.117:0/133439480 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.966 7f4556ffd700 1 --2- 172.21.2.117:0/133439480 >> v2:172.21.2.125:3300/0 conn(0x7f45580ef3f0 0x7f45580eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f4557fff700 1 -- 172.21.2.117:0/133439480 >> v2:172.21.2.125:3300/0 conn(0x7f45580ef3f0 msgr2=0x7f45580eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f4557fff700 1 --2- 172.21.2.117:0/133439480 >> v2:172.21.2.125:3300/0 conn(0x7f45580ef3f0 0x7f45580eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f4557fff700 1 -- 172.21.2.117:0/133439480 >> v1:172.21.2.117:6789/0 conn(0x7f45580d18f0 legacy=0x7f45580eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f45567fc700 1 -- 172.21.2.117:0/133439480 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133228315 0 0) 0x7f4548000ce0 con 0x7f45580d18f0 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f4557fff700 1 -- 172.21.2.117:0/133439480 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45581b8fd0 con 0x7f45580eb830 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f4557fff700 1 --2- 172.21.2.117:0/133439480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45580eb830 0x7f45581b79a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f4550006e30 tx=0x7f4550009de0).ready entity=mon.0 client_cookie=494ba177637e24d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f45567fc700 1 -- 172.21.2.117:0/133439480 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4550003030 con 0x7f45580eb830 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f45567fc700 1 -- 172.21.2.117:0/133439480 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455000b300 con 0x7f45580eb830 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f45567fc700 1 -- 172.21.2.117:0/133439480 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f455000b820 con 0x7f45580eb830 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f455dfd6700 1 -- 172.21.2.117:0/133439480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45580eb830 msgr2=0x7f45581b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.967 7f455dfd6700 1 --2- 172.21.2.117:0/133439480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45580eb830 0x7f45581b79a0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f4550006e30 tx=0x7f4550009de0).stop 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 -- 172.21.2.117:0/133439480 shutdown_connections 2021-02-10T11:50:15.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 --2- 172.21.2.117:0/133439480 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45580eb830 0x7f45581b79a0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 --2- 172.21.2.117:0/133439480 >> v2:172.21.2.125:3300/0 conn(0x7f45580ef3f0 0x7f45580eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 -- 172.21.2.117:0/133439480 >> 172.21.2.117:0/133439480 conn(0x7f45580cf060 msgr2=0x7f45580cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:15.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 -- 172.21.2.117:0/133439480 shutdown_connections 2021-02-10T11:50:15.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.968 7f455dfd6700 1 -- 172.21.2.117:0/133439480 wait complete. 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.982 7f455dfd6700 1 Processor -- start 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.982 7f455dfd6700 1 -- start start 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 0x7f45580a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45580a13a0 0x7f45580a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455807cca0 con 0x7f455807c7c0 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455818fb70 con 0x7f45580a13a0 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f455dfd6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45580dc6d0 con 0x7f45581a3930 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54590/0 (socket says 172.21.2.117:54590) 2021-02-10T11:50:15.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 -- 172.21.2.117:0/3699866233 learned_addr learned my addr 172.21.2.117:0/3699866233 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f4557fff700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45580a13a0 0x7f45580a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f4556ffd700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 0x7f45580a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 -- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 msgr2=0x7f45580a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 0x7f45580a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 -- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45580a13a0 msgr2=0x7f45580a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45580a13a0 0x7f45580a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f45577fe700 1 -- 172.21.2.117:0/3699866233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455807d220 con 0x7f455807c7c0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.983 7f4556ffd700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 0x7f45580a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f45577fe700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f4548001430 tx=0x7f454800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4548000990 con 0x7f455807c7c0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454800ad20 con 0x7f455807c7c0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455807d220 con 0x7f455807c7c0 2021-02-10T11:50:15.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455807d7b0 con 0x7f455807c7c0 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.984 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4548004620 con 0x7f455807c7c0 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.985 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f45480043a0 con 0x7f455807c7c0 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.985 7f454ffff700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.985 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4548032410 con 0x7f455807c7c0 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.985 7f4556ffd700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:15.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:15.985 7f4556ffd700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 0x7f453c01abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.015 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45580daff0 con 0x7f455807c7c0 2021-02-10T11:50:16.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.018 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4548000990 con 0x7f455807c7c0 2021-02-10T11:50:16.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.144 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f45580daff0 con 0x7f455807c7c0 2021-02-10T11:50:16.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.144 7f454ffff700 1 -- 172.21.2.117:0/3699866233 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v113) v1 ==== 92+0+10 (secure 0 0 0) 0x7f45480043a0 con 0x7f455807c7c0 2021-02-10T11:50:16.143 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:16.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 msgr2=0x7f453c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 0x7f453c01abc0 crc :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 msgr2=0x7f45581a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f4548001430 tx=0x7f454800a930).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 shutdown_connections 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45580a13a0 0x7f45580a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45581a3930 0x7f45580a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f453c0186c0 0x7f453c01abc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.146 7f455dfd6700 1 --2- 172.21.2.117:0/3699866233 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f455807c7c0 0x7f45581a3310 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.147 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 >> 172.21.2.117:0/3699866233 conn(0x7f4558087090 msgr2=0x7f455808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.147 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 shutdown_connections 2021-02-10T11:50:16.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.147 7f455dfd6700 1 -- 172.21.2.117:0/3699866233 wait complete. 2021-02-10T11:50:16.153 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.564 7f8d8bba1700 1 Processor -- start 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 -- start start 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d840d18e0 0x7f8d840ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d840eb000 0x7f8d840eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d840ede60 con 0x7f8d840ee3e0 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840eba20 con 0x7f8d840eb000 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8bba1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840d1ce0 con 0x7f8d840d18e0 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8a1be700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8d840ee3e0 0x7f8d840ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54844/0 (socket says 172.21.2.117:54844) 2021-02-10T11:50:16.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d8a1be700 1 -- 172.21.2.117:0/7208969 learned_addr learned my addr 172.21.2.117:0/7208969 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.565 7f8d891bc700 1 --2- 172.21.2.117:0/7208969 >> v2:172.21.2.117:3300/0 conn(0x7f8d840eb000 0x7f8d840eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d889bb700 1 -- 172.21.2.117:0/7208969 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045010566 0 0) 0x7f8d7c001500 con 0x7f8d840ee3e0 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d891bc700 1 -- 172.21.2.117:0/7208969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d840d18e0 msgr2=0x7f8d840ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d891bc700 1 --2- 172.21.2.117:0/7208969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d840d18e0 0x7f8d840ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d891bc700 1 -- 172.21.2.117:0/7208969 >> v1:172.21.2.117:6789/0 conn(0x7f8d840ee3e0 legacy=0x7f8d840ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d891bc700 1 -- 172.21.2.117:0/7208969 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840eefd0 con 0x7f8d840eb000 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d891bc700 1 --2- 172.21.2.117:0/7208969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840eb000 0x7f8d840eb400 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f8d80006650 tx=0x7f8d80009a80).ready entity=mon.0 client_cookie=bf8a5d1a2ba64ad3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d889bb700 1 -- 172.21.2.117:0/7208969 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d800009b0 con 0x7f8d840eb000 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d889bb700 1 -- 172.21.2.117:0/7208969 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8000c0c0 con 0x7f8d840eb000 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d889bb700 1 -- 172.21.2.117:0/7208969 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d8000c510 con 0x7f8d840eb000 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d8bba1700 1 -- 172.21.2.117:0/7208969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840eb000 msgr2=0x7f8d840eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.566 7f8d8bba1700 1 --2- 172.21.2.117:0/7208969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840eb000 0x7f8d840eb400 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f8d80006650 tx=0x7f8d80009a80).stop 2021-02-10T11:50:16.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 -- 172.21.2.117:0/7208969 shutdown_connections 2021-02-10T11:50:16.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 --2- 172.21.2.117:0/7208969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840eb000 0x7f8d840eb400 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 --2- 172.21.2.117:0/7208969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d840d18e0 0x7f8d840ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 -- 172.21.2.117:0/7208969 >> 172.21.2.117:0/7208969 conn(0x7f8d840ced60 msgr2=0x7f8d840cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 -- 172.21.2.117:0/7208969 shutdown_connections 2021-02-10T11:50:16.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.567 7f8d8bba1700 1 -- 172.21.2.117:0/7208969 wait complete. 2021-02-10T11:50:16.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.581 7f8d8bba1700 1 Processor -- start 2021-02-10T11:50:16.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.581 7f8d8bba1700 1 -- start start 2021-02-10T11:50:16.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d8419a910 0x7f8d840e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d841aa7c0 0x7f8d841aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a8b90 con 0x7f8d840e8ff0 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d841aa240 con 0x7f8d841aa7c0 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8bba1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840e9a10 con 0x7f8d8419a910 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54598/0 (socket says 172.21.2.117:54598) 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 -- 172.21.2.117:0/1788270545 learned_addr learned my addr 172.21.2.117:0/1788270545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 -- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d8419a910 msgr2=0x7f8d840e89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d8a1be700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d841aa7c0 0x7f8d841aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d8419a910 0x7f8d840e89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 -- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d841aa7c0 msgr2=0x7f8d841aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d841aa7c0 0x7f8d841aabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.582 7f8d891bc700 1 -- 172.21.2.117:0/1788270545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d841ab1e0 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d891bc700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f8d80009630 tx=0x7f8d8000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d800093a0 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8000bcf0 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d80005560 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d841ab1e0 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.583 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d841ab1e0 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.584 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d80008240 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.584 7f8d767fc700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 0x7f8d6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.584 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f8d80032310 con 0x7f8d840e8ff0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.584 7f8d899bd700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 0x7f8d6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:16.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.585 7f8d899bd700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 0x7f8d6c01abc0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:16.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.615 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d84087210 con 0x7f8d840e8ff0 2021-02-10T11:50:16.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.618 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d80022910 con 0x7f8d840e8ff0 2021-02-10T11:50:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.742 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8d840ede60 con 0x7f8d840e8ff0 2021-02-10T11:50:16.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.742 7f8d767fc700 1 -- 172.21.2.117:0/1788270545 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v113) v1 ==== 84+0+64 (secure 0 0 0) 0x7f8d80022910 con 0x7f8d840e8ff0 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:16.741 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:50:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 msgr2=0x7f8d6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 0x7f8d6c01abc0 crc :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 msgr2=0x7f8d840e93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f8d80009630 tx=0x7f8d8000b900).stop 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 shutdown_connections 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d6c0186c0 0x7f8d6c01abc0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d840e8ff0 0x7f8d840e93f0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d841aa7c0 0x7f8d841aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 --2- 172.21.2.117:0/1788270545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d8419a910 0x7f8d840e89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.744 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 >> 172.21.2.117:0/1788270545 conn(0x7f8d8419f610 msgr2=0x7f8d84099520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.745 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 shutdown_connections 2021-02-10T11:50:16.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:16.745 7f8d8bba1700 1 -- 172.21.2.117:0/1788270545 wait complete. 2021-02-10T11:50:16.750 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:50:16.751 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:50:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 Processor -- start 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 -- start start 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe740ef3f0 0x7fbe740eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbe740eb830 0x7fbe741b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe740ef7f0 con 0x7fbe740d18f0 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe740eded0 con 0x7fbe740eb830 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.168 7fbe7c6c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe740d1cf0 con 0x7fbe740ef3f0 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe7a4e5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbe740d18f0 0x7fbe740eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54856/0 (socket says 172.21.2.117:54856) 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe7a4e5700 1 -- 172.21.2.117:0/2610590852 learned_addr learned my addr 172.21.2.117:0/2610590852 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574524970 0 0) 0x7fbe680010b0 con 0x7fbe740d18f0 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe60002fe0 con 0x7fbe740d18f0 2021-02-10T11:50:17.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe7ace6700 1 --2- 172.21.2.117:0/2610590852 >> v2:172.21.2.125:3300/0 conn(0x7fbe740eb830 0x7fbe741b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe79ce4700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe740ef3f0 0x7fbe740eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2314051450 0 0) 0x7fbe680010b0 con 0x7fbe740d18f0 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe740ef3f0 msgr2=0x7fbe740eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe740ef3f0 0x7fbe740eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 >> v2:172.21.2.125:3300/0 conn(0x7fbe740eb830 msgr2=0x7fbe741b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 --2- 172.21.2.117:0/2610590852 >> v2:172.21.2.125:3300/0 conn(0x7fbe740eb830 0x7fbe741b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe741b8fd0 con 0x7fbe740d18f0 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fbe680015b0 con 0x7fbe740d18f0 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 >> v1:172.21.2.117:6789/0 conn(0x7fbe740d18f0 legacy=0x7fbe740eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.169 7fbe794e3700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe60003b10 0x7fbe60005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe6004a7b0 0x7fbe6004cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe6004d590 0x7fbe6004fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600502d0 con 0x7fbe6004a7b0 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600504f0 con 0x7fbe60003b10 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60050710 con 0x7fbe6004d590 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe60003b10 0x7fbe60005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe79ce4700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe6004a7b0 0x7fbe6004cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7ace6700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe6004d590 0x7fbe6004fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 -- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe6004d590 msgr2=0x7fbe6004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe6004d590 0x7fbe6004fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 -- 172.21.2.117:0/2610590852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe6004a7b0 msgr2=0x7fbe6004cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe6004a7b0 0x7fbe6004cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 -- 172.21.2.117:0/2610590852 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe600509b0 con 0x7fbe60003b10 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe79ce4700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe6004a7b0 0x7fbe6004cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe7a4e5700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe60003b10 0x7fbe60005f20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fbe68000b40 tx=0x7fbe68007d00).ready entity=mon.1 client_cookie=4b59c898399b17d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.170 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe68001010 con 0x7fbe60003b10 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe68001450 con 0x7fbe60003b10 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe794e3700 1 -- 172.21.2.117:0/2610590852 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6800a520 con 0x7fbe60003b10 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 -- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe60003b10 msgr2=0x7fbe60005f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 --2- 172.21.2.117:0/2610590852 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe60003b10 0x7fbe60005f20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fbe68000b40 tx=0x7fbe68007d00).stop 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7ace6700 1 -- 172.21.2.117:0/2610590852 reap_dead start 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 -- 172.21.2.117:0/2610590852 shutdown_connections 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 -- 172.21.2.117:0/2610590852 >> 172.21.2.117:0/2610590852 conn(0x7fbe740cf060 msgr2=0x7fbe740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 -- 172.21.2.117:0/2610590852 shutdown_connections 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.171 7fbe7c6c9700 1 -- 172.21.2.117:0/2610590852 wait complete. 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 Processor -- start 2021-02-10T11:50:17.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 -- start start 2021-02-10T11:50:17.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe7408a220 0x7fbe7407c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe7408a8e0 0x7fbe7409dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7409e2a0 con 0x7fbe7408ace0 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741a7e50 con 0x7fbe7408a8e0 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7c6c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741a8070 con 0x7fbe7408a220 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7ace6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.186 7fbe7ace6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54620/0 (socket says 172.21.2.117:54620) 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 -- 172.21.2.117:0/3340687951 learned_addr learned my addr 172.21.2.117:0/3340687951 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 -- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe7408a220 msgr2=0x7fbe7407c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe7408a220 0x7fbe7407c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 -- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe7408a8e0 msgr2=0x7fbe7409dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe7408a8e0 0x7fbe7409dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:17.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 -- 172.21.2.117:0/3340687951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7407d3d0 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7ace6700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7fbe6c00b910 tx=0x7fbe6c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6c00c2f0 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c006d20 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe6c00c830 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7407d3d0 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.187 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7407d960 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.188 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe6c0011f0 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.188 7fbe66ffd700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 0x7fbe5c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.188 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fbe6c0325d0 con 0x7fbe7408ace0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.188 7fbe7a4e5700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 0x7fbe5c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:17.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.189 7fbe7a4e5700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 0x7fbe5c01abc0 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:17.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.218 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7407cea0 con 0x7fbe7408ace0 2021-02-10T11:50:17.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.221 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe6c02ffa0 con 0x7fbe7408ace0 2021-02-10T11:50:17.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.346 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fbe7407cea0 con 0x7fbe7408ace0 2021-02-10T11:50:17.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.774 7fbe66ffd700 1 -- 172.21.2.117:0/3340687951 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v114) v1 ==== 133+0+0 (secure 0 0 0) 0x7fbe6c02ffa0 con 0x7fbe7408ace0 2021-02-10T11:50:17.810 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 msgr2=0x7fbe5c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 0x7fbe5c01abc0 crc :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 msgr2=0x7fbe741a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 rx=0x7fbe6c00b910 tx=0x7fbe6c00bcb0).stop 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 shutdown_connections 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe5c0186c0 0x7fbe5c01abc0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe7408ace0 0x7fbe741a7830 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe7408a8e0 0x7fbe7409dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:17.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 --2- 172.21.2.117:0/3340687951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe7408a220 0x7fbe7407c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 >> 172.21.2.117:0/3340687951 conn(0x7fbe740a71e0 msgr2=0x7fbe740a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 shutdown_connections 2021-02-10T11:50:17.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:17.815 7fbe7c6c9700 1 -- 172.21.2.117:0/3340687951 wait complete. 2021-02-10T11:50:17.822 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.238 7f95f3a34700 1 Processor -- start 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.238 7f95f3a34700 1 -- start start 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95ec0e7180 0x7f95ec0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f95ec0e8390 0x7f95ec0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ec0e7580 con 0x7f95ec1b07f0 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0e5ce0 con 0x7f95ec0e7180 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0e77f0 con 0x7f95ec0e8390 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95ec0e7180 0x7f95ec0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:18.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f95ec0e7180 0x7f95ec0e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54626/0 (socket says 172.21.2.117:54626) 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 -- 172.21.2.117:0/1446405038 learned_addr learned my addr 172.21.2.117:0/1446405038 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 -- 172.21.2.117:0/1446405038 >> v2:172.21.2.125:3300/0 conn(0x7f95ec0e8390 msgr2=0x7f95ec0e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 --2- 172.21.2.117:0/1446405038 >> v2:172.21.2.125:3300/0 conn(0x7f95ec0e8390 0x7f95ec0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 -- 172.21.2.117:0/1446405038 >> v1:172.21.2.125:6789/0 conn(0x7f95ec1b07f0 legacy=0x7f95ec0e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 -- 172.21.2.117:0/1446405038 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec0e8db0 con 0x7f95ec0e7180 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f104f700 1 --2- 172.21.2.117:0/1446405038 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec0e7180 0x7f95ec0e7d70 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f95e8006700 tx=0x7f95e8009b00).ready entity=mon.0 client_cookie=9ff4551c44309f5b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f084e700 1 -- 172.21.2.117:0/1446405038 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e800c0c0 con 0x7f95ec0e7180 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f084e700 1 -- 172.21.2.117:0/1446405038 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e800c340 con 0x7f95ec0e7180 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f084e700 1 -- 172.21.2.117:0/1446405038 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e800c820 con 0x7f95ec0e7180 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 -- 172.21.2.117:0/1446405038 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec0e7180 msgr2=0x7f95ec0e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.239 7f95f3a34700 1 --2- 172.21.2.117:0/1446405038 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec0e7180 0x7f95ec0e7d70 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 rx=0x7f95e8006700 tx=0x7f95e8009b00).stop 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 -- 172.21.2.117:0/1446405038 shutdown_connections 2021-02-10T11:50:18.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 --2- 172.21.2.117:0/1446405038 >> v2:172.21.2.125:3300/0 conn(0x7f95ec0e8390 0x7f95ec0e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:18.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 --2- 172.21.2.117:0/1446405038 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec0e7180 0x7f95ec0e7d70 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 -- 172.21.2.117:0/1446405038 >> 172.21.2.117:0/1446405038 conn(0x7f95ec0ce500 msgr2=0x7f95ec0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:18.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 -- 172.21.2.117:0/1446405038 shutdown_connections 2021-02-10T11:50:18.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.240 7f95f3a34700 1 -- 172.21.2.117:0/1446405038 wait complete. 2021-02-10T11:50:18.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.254 7f95f3a34700 1 Processor -- start 2021-02-10T11:50:18.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.254 7f95f3a34700 1 -- start start 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95ec0ace40 0x7f95ec0ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f95ec0ad860 0x7f95ec1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0adc60 con 0x7f95ec19c4e0 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0beba0 con 0x7f95ec0ace40 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f3a34700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec0bedc0 con 0x7f95ec0ad860 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54634/0 (socket says 172.21.2.117:54634) 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 -- 172.21.2.117:0/888766255 learned_addr learned my addr 172.21.2.117:0/888766255 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f104f700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95ec0ace40 0x7f95ec0ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f2051700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f95ec0ad860 0x7f95ec1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 -- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f95ec0ad860 msgr2=0x7f95ec1091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:18.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f95ec0ad860 0x7f95ec1091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 -- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95ec0ace40 msgr2=0x7f95ec0ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95ec0ace40 0x7f95ec0ad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 -- 172.21.2.117:0/888766255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec109800 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.255 7f95f1850700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7f95e000b5b0 tx=0x7f95e000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e000bd40 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e0000c60 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f95e00054a0 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95f3a34700 1 -- 172.21.2.117:0/888766255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec109800 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95f3a34700 1 -- 172.21.2.117:0/888766255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec109800 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f95e0006030 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.256 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f95e0032750 con 0x7f95ec19c4e0 2021-02-10T11:50:18.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.257 7f95f104f700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:18.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.257 7f95f104f700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 0x7f95d401abc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:18.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.289 7f95f3a34700 1 -- 172.21.2.117:0/888766255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec0e77f0 con 0x7f95ec19c4e0 2021-02-10T11:50:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.291 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f95e001f510 con 0x7f95ec19c4e0 2021-02-10T11:50:18.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.417 7f95f3a34700 1 -- 172.21.2.117:0/888766255 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f95ec0e77f0 con 0x7f95ec19c4e0 2021-02-10T11:50:18.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.777 7f95de7fc700 1 -- 172.21.2.117:0/888766255 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v115) v1 ==== 125+0+0 (secure 0 0 0) 0x7f95e0006030 con 0x7f95ec19c4e0 2021-02-10T11:50:18.780 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:50:18.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 msgr2=0x7f95d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:18.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 0x7f95d401abc0 crc :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 msgr2=0x7f95ec0bb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7f95e000b5b0 tx=0x7f95e000b950).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 shutdown_connections 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f95ec0ad860 0x7f95ec1091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f95ec0ace40 0x7f95ec0ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f95d40186c0 0x7f95d401abc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 --2- 172.21.2.117:0/888766255 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f95ec19c4e0 0x7f95ec0bb0f0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 >> 172.21.2.117:0/888766255 conn(0x7f95ec0c3f30 msgr2=0x7f95ec111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 shutdown_connections 2021-02-10T11:50:18.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:18.784 7f95f3a34700 1 -- 172.21.2.117:0/888766255 wait complete. 2021-02-10T11:50:18.791 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.208 7f0509a00700 1 Processor -- start 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.208 7f0509a00700 1 -- start start 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0509a00700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 0x7f05040e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0509a00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 0x7f05040e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0509a00700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05040e7570 con 0x7f05041b07e0 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0509a00700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05040e5cd0 con 0x7f05040e7170 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0509a00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05040e77e0 con 0x7f05040e8380 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0502ffd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 0x7f05040e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f05037fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f05041b07e0 0x7f05040e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47818/0 (socket says 172.21.2.117:47818) 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f05037fe700 1 -- 172.21.2.117:0/4033287484 learned_addr learned my addr 172.21.2.117:0/4033287484 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.209 7f0503fff700 1 --2- 172.21.2.117:0/4033287484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 0x7f05040e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0503fff700 1 -- 172.21.2.117:0/4033287484 >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 msgr2=0x7f05040e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0503fff700 1 --2- 172.21.2.117:0/4033287484 >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 0x7f05040e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f05027fc700 1 -- 172.21.2.117:0/4033287484 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320280555 0 0) 0x7f04f4001190 con 0x7f05041b07e0 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0503fff700 1 -- 172.21.2.117:0/4033287484 >> v1:172.21.2.125:6789/0 conn(0x7f05041b07e0 legacy=0x7f05040e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0503fff700 1 -- 172.21.2.117:0/4033287484 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05040e8da0 con 0x7f05040e8380 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0502ffd700 1 --2- 172.21.2.117:0/4033287484 >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 0x7f05040e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0503fff700 1 --2- 172.21.2.117:0/4033287484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 0x7f05040e8780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f04fc006e50 tx=0x7f04fc009e70).ready entity=mon.2 client_cookie=6f71502032c343ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f05027fc700 1 -- 172.21.2.117:0/4033287484 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc00b300 con 0x7f05040e8380 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f05027fc700 1 -- 172.21.2.117:0/4033287484 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc003030 con 0x7f05040e8380 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f05027fc700 1 -- 172.21.2.117:0/4033287484 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04fc00b8c0 con 0x7f05040e8380 2021-02-10T11:50:19.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0509a00700 1 -- 172.21.2.117:0/4033287484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 msgr2=0x7f05040e8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.210 7f0509a00700 1 --2- 172.21.2.117:0/4033287484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 0x7f05040e8780 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f04fc006e50 tx=0x7f04fc009e70).stop 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 -- 172.21.2.117:0/4033287484 shutdown_connections 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 --2- 172.21.2.117:0/4033287484 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040e8380 0x7f05040e8780 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 --2- 172.21.2.117:0/4033287484 >> v2:172.21.2.125:3300/0 conn(0x7f05040e7170 0x7f05040e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 -- 172.21.2.117:0/4033287484 >> 172.21.2.117:0/4033287484 conn(0x7f05040ce4f0 msgr2=0x7f05040e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 -- 172.21.2.117:0/4033287484 shutdown_connections 2021-02-10T11:50:19.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.211 7f0509a00700 1 -- 172.21.2.117:0/4033287484 wait complete. 2021-02-10T11:50:19.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.225 7f0509a00700 1 Processor -- start 2021-02-10T11:50:19.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.225 7f0509a00700 1 -- start start 2021-02-10T11:50:19.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f050410fca0 0x7f05040b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 0x7f050418dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0504110250 con 0x7f050418e310 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05041a67b0 con 0x7f050410fca0 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0509a00700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05041a69d0 con 0x7f05040b4000 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0503fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0502ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 0x7f050418dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0503fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54652/0 (socket says 172.21.2.117:54652) 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f0503fff700 1 -- 172.21.2.117:0/1687387550 learned_addr learned my addr 172.21.2.117:0/1687387550 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.226 7f05037fe700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f050410fca0 0x7f05040b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 -- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f050410fca0 msgr2=0x7f05040b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f050410fca0 0x7f05040b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 -- 172.21.2.117:0/1687387550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 msgr2=0x7f05041a6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 -- 172.21.2.117:0/1687387550 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0504110530 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0503fff700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0502ffd700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 0x7f050418dcf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f04ec009540 tx=0x7f04ec00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec00bcd0 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0063e0 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04ec002030 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0509a00700 1 -- 172.21.2.117:0/1687387550 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0504110530 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.227 7f0509a00700 1 -- 172.21.2.117:0/1687387550 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0504110ac0 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.228 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04ec009fe0 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.228 7f04fbfff700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 0x7f04e801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.228 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f04ec032620 con 0x7f05040b4000 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.228 7f05037fe700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 0x7f04e801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.229 7f05037fe700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 0x7f04e801abd0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.259 7f0509a00700 1 -- 172.21.2.117:0/1687387550 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050410c250 con 0x7f05040b4000 2021-02-10T11:50:19.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.262 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04ec0009b0 con 0x7f05040b4000 2021-02-10T11:50:19.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.392 7f0509a00700 1 -- 172.21.2.117:0/1687387550 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f05040e77e0 con 0x7f05040b4000 2021-02-10T11:50:19.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.397 7f04fbfff700 1 -- 172.21.2.117:0/1687387550 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 24 and data pool 25 v91) v1 ==== 189+0+0 (secure 0 0 0) 0x7f04ec009fe0 con 0x7f05040b4000 2021-02-10T11:50:19.398 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 24 and data pool 25 2021-02-10T11:50:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 -- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 msgr2=0x7f04e801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 0x7f04e801abd0 crc :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 -- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 msgr2=0x7f050418dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 0x7f050418dcf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f04ec009540 tx=0x7f04ec00b8e0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 -- 172.21.2.117:0/1687387550 shutdown_connections 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04e80186d0 0x7f04e801abd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f050418e310 0x7f05041a6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05040b4000 0x7f050418dcf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 --2- 172.21.2.117:0/1687387550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f050410fca0 0x7f05040b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.403 7f0509a00700 1 -- 172.21.2.117:0/1687387550 >> 172.21.2.117:0/1687387550 conn(0x7f050419f020 msgr2=0x7f05041af160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.404 7f0509a00700 1 -- 172.21.2.117:0/1687387550 shutdown_connections 2021-02-10T11:50:19.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.404 7f0509a00700 1 -- 172.21.2.117:0/1687387550 wait complete. 2021-02-10T11:50:19.409 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.822 7f549f9ec700 1 Processor -- start 2021-02-10T11:50:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.822 7f549f9ec700 1 -- start start 2021-02-10T11:50:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549f9ec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f54980ef3f0 0x7f54980eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549f9ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54980eb7f0 0x7f54981b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549f9ec700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980ede60 con 0x7f54980d18e0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549f9ec700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54980ef7f0 con 0x7f54980ef3f0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549f9ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980d1ce0 con 0x7f54980eb7f0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d007700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f54980ef3f0 0x7f54980eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d808700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f54980d18e0 0x7f54980ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54906/0 (socket says 172.21.2.117:54906) 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d808700 1 -- 172.21.2.117:0/4015032159 learned_addr learned my addr 172.21.2.117:0/4015032159 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d007700 1 -- 172.21.2.117:0/4015032159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54980eb7f0 msgr2=0x7f54981b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d007700 1 --2- 172.21.2.117:0/4015032159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54980eb7f0 0x7f54981b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d007700 1 -- 172.21.2.117:0/4015032159 >> v1:172.21.2.117:6789/0 conn(0x7f54980d18e0 legacy=0x7f54980ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.823 7f549d007700 1 -- 172.21.2.117:0/4015032159 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54981b8f90 con 0x7f54980ef3f0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549d007700 1 --2- 172.21.2.117:0/4015032159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980ef3f0 0x7f54980eb1d0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f5488006650 tx=0x7f5488009a80).ready entity=mon.0 client_cookie=7f1c653bc604c1f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549c806700 1 -- 172.21.2.117:0/4015032159 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f548800c0c0 con 0x7f54980ef3f0 2021-02-10T11:50:19.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549c806700 1 -- 172.21.2.117:0/4015032159 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548800c340 con 0x7f54980ef3f0 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549c806700 1 -- 172.21.2.117:0/4015032159 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f548800c860 con 0x7f54980ef3f0 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549f9ec700 1 -- 172.21.2.117:0/4015032159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980ef3f0 msgr2=0x7f54980eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.824 7f549f9ec700 1 --2- 172.21.2.117:0/4015032159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980ef3f0 0x7f54980eb1d0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 rx=0x7f5488006650 tx=0x7f5488009a80).stop 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 -- 172.21.2.117:0/4015032159 shutdown_connections 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 --2- 172.21.2.117:0/4015032159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54980eb7f0 0x7f54981b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 --2- 172.21.2.117:0/4015032159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980ef3f0 0x7f54980eb1d0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 -- 172.21.2.117:0/4015032159 >> 172.21.2.117:0/4015032159 conn(0x7f54980ced60 msgr2=0x7f54980cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 -- 172.21.2.117:0/4015032159 shutdown_connections 2021-02-10T11:50:19.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.825 7f549f9ec700 1 -- 172.21.2.117:0/4015032159 wait complete. 2021-02-10T11:50:19.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 Processor -- start 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 -- start start 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5498196390 0x7f54980e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54980e9960 0x7f54980e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5498196810 con 0x7f54980e9560 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5498098ad0 con 0x7f54980e9960 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549f9ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980e8a40 con 0x7f5498196390 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549d007700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549d007700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54662/0 (socket says 172.21.2.117:54662) 2021-02-10T11:50:19.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.840 7f549d007700 1 -- 172.21.2.117:0/1793959787 learned_addr learned my addr 172.21.2.117:0/1793959787 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 -- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5498196390 msgr2=0x7f54980e8f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5498196390 0x7f54980e8f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 -- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54980e9960 msgr2=0x7f54980e8420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54980e9960 0x7f54980e8420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 -- 172.21.2.117:0/1793959787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980e8c60 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549d007700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f5488009630 tx=0x7f548800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54880093a0 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548800bd70 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54980e8c60 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54880054d0 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.841 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549819cd70 con 0x7f54980e9560 2021-02-10T11:50:19.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.842 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54880054f0 con 0x7f54980e9560 2021-02-10T11:50:19.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.842 7f548e7fc700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 0x7f548001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:19.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.842 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5488032950 con 0x7f54980e9560 2021-02-10T11:50:19.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.842 7f549d808700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 0x7f548001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:19.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.843 7f549d808700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 0x7f548001ac10 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:19.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.872 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5498087210 con 0x7f54980e9560 2021-02-10T11:50:19.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.875 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54880052e0 con 0x7f54980e9560 2021-02-10T11:50:19.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:19.999 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54980ede60 con 0x7f54980e9560 2021-02-10T11:50:19.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.000 7f548e7fc700 1 -- 172.21.2.117:0/1793959787 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f54880054f0 con 0x7f54980e9560 2021-02-10T11:50:19.998 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:19.999 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":116,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:19.395584","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:50:17.365216","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:50:18.436070","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 msgr2=0x7f548001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 0x7f548001ac10 crc :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 msgr2=0x7f54980e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f5488009630 tx=0x7f548800b980).stop 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 shutdown_connections 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5480018730 0x7f548001ac10 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54980e9560 0x7f54980e7e00 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54980e9960 0x7f54980e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 --2- 172.21.2.117:0/1793959787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5498196390 0x7f54980e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 >> 172.21.2.117:0/1793959787 conn(0x7f549819f610 msgr2=0x7f54981a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 shutdown_connections 2021-02-10T11:50:20.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.002 7f549f9ec700 1 -- 172.21.2.117:0/1793959787 wait complete. 2021-02-10T11:50:20.009 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:20.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.421 7fbf4ae1d700 1 Processor -- start 2021-02-10T11:50:20.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.421 7fbf4ae1d700 1 -- start start 2021-02-10T11:50:20.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4ae1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf440d18e0 0x7fbf440ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4ae1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 0x7fbf441b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4ae1d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf440ede60 con 0x7fbf440eb000 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4ae1d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440ef7f0 con 0x7fbf440d18e0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4ae1d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440d1ce0 con 0x7fbf440ef3f0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf48c39700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf440d18e0 0x7fbf440ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4943a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 0x7fbf441b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf43fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbf440eb000 0x7fbf440eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54922/0 (socket says 172.21.2.117:54922) 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf43fff700 1 -- 172.21.2.117:0/3863118243 learned_addr learned my addr 172.21.2.117:0/3863118243 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4943a700 1 -- 172.21.2.117:0/3863118243 >> v2:172.21.2.117:3300/0 conn(0x7fbf440d18e0 msgr2=0x7fbf440ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.422 7fbf4943a700 1 --2- 172.21.2.117:0/3863118243 >> v2:172.21.2.117:3300/0 conn(0x7fbf440d18e0 0x7fbf440ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4943a700 1 -- 172.21.2.117:0/3863118243 >> v1:172.21.2.117:6789/0 conn(0x7fbf440eb000 legacy=0x7fbf440eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4943a700 1 -- 172.21.2.117:0/3863118243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf441b8f90 con 0x7fbf440ef3f0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4943a700 1 --2- 172.21.2.117:0/3863118243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 0x7fbf441b7960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fbf3c006ad0 tx=0x7fbf3c009ad0).ready entity=mon.2 client_cookie=69e80256d644a561 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf437fe700 1 -- 172.21.2.117:0/3863118243 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf3c0022e0 con 0x7fbf440ef3f0 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf437fe700 1 -- 172.21.2.117:0/3863118243 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c006410 con 0x7fbf440ef3f0 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf437fe700 1 -- 172.21.2.117:0/3863118243 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf3c002620 con 0x7fbf440ef3f0 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 -- 172.21.2.117:0/3863118243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 msgr2=0x7fbf441b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 --2- 172.21.2.117:0/3863118243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 0x7fbf441b7960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fbf3c006ad0 tx=0x7fbf3c009ad0).stop 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 -- 172.21.2.117:0/3863118243 shutdown_connections 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 --2- 172.21.2.117:0/3863118243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440ef3f0 0x7fbf441b7960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 --2- 172.21.2.117:0/3863118243 >> v2:172.21.2.117:3300/0 conn(0x7fbf440d18e0 0x7fbf440ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.423 7fbf4ae1d700 1 -- 172.21.2.117:0/3863118243 >> 172.21.2.117:0/3863118243 conn(0x7fbf440ced60 msgr2=0x7fbf440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.424 7fbf4ae1d700 1 -- 172.21.2.117:0/3863118243 shutdown_connections 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.424 7fbf4ae1d700 1 -- 172.21.2.117:0/3863118243 wait complete. 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.438 7fbf4ae1d700 1 Processor -- start 2021-02-10T11:50:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.438 7fbf4ae1d700 1 -- start start 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 0x7fbf440b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440aee60 0x7fbf440b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440b95e0 con 0x7fbf4409b600 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440b9230 con 0x7fbf440aee60 2021-02-10T11:50:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4ae1d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf441a2e60 con 0x7fbf440aea60 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54680/0 (socket says 172.21.2.117:54680) 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 -- 172.21.2.117:0/1362971563 learned_addr learned my addr 172.21.2.117:0/1362971563 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf4943a700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440aee60 0x7fbf440b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf43fff700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 0x7fbf440b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 -- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 msgr2=0x7fbf440b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 0x7fbf440b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 -- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440aee60 msgr2=0x7fbf440b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440aee60 0x7fbf440b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 -- 172.21.2.117:0/1362971563 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf441a3080 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf43fff700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 0x7fbf440b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.439 7fbf48c39700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fbf2c006f60 tx=0x7fbf2c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf2c0009b0 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c00bd60 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf2c002220 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf441a3080 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf441a3080 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf2c006b00 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 0x7fbf2801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.440 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf2c032910 con 0x7fbf4409b600 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.441 7fbf43fff700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 0x7fbf2801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.441 7fbf43fff700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 0x7fbf2801ac10 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:20.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.472 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf441a3c70 con 0x7fbf4409b600 2021-02-10T11:50:20.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.474 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf2c006340 con 0x7fbf4409b600 2021-02-10T11:50:20.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.598 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf441a3c70 con 0x7fbf4409b600 2021-02-10T11:50:20.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.599 7fbf417fa700 1 -- 172.21.2.117:0/1362971563 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fbf2c006b00 con 0x7fbf4409b600 2021-02-10T11:50:20.597 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:20.597 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":117,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:20.397701","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:50:17.365216","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:50:18.436070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:20.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.600 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 msgr2=0x7fbf2801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.600 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 0x7fbf2801ac10 crc :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.600 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 msgr2=0x7fbf440a7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:20.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.600 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fbf2c006f60 tx=0x7fbf2c00b970).stop 2021-02-10T11:50:20.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 shutdown_connections 2021-02-10T11:50:20.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf440aee60 0x7fbf440b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf440aea60 0x7fbf440b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf28018730 0x7fbf2801ac10 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 --2- 172.21.2.117:0/1362971563 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4409b600 0x7fbf440a7360 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:20.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 >> 172.21.2.117:0/1362971563 conn(0x7fbf440925a0 msgr2=0x7fbf440b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:20.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 shutdown_connections 2021-02-10T11:50:20.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:20.601 7fbf4ae1d700 1 -- 172.21.2.117:0/1362971563 wait complete. 2021-02-10T11:50:20.607 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:50:21.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 Processor -- start 2021-02-10T11:50:21.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 -- start start 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e140d18f0 0x7f7e140eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7e140eb040 0x7f7e140eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140ede60 con 0x7f7e140ee3e0 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e140ebad0 con 0x7f7e140d18f0 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.025 7f7e1a7b2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e140d1cf0 con 0x7f7e140eb040 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e140d18f0 0x7f7e140eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e140d18f0 0x7f7e140eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54684/0 (socket says 172.21.2.117:54684) 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 -- 172.21.2.117:0/1941767612 learned_addr learned my addr 172.21.2.117:0/1941767612 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 -- 172.21.2.117:0/1941767612 >> v2:172.21.2.125:3300/0 conn(0x7f7e140eb040 msgr2=0x7f7e140eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 --2- 172.21.2.117:0/1941767612 >> v2:172.21.2.125:3300/0 conn(0x7f7e140eb040 0x7f7e140eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 -- 172.21.2.117:0/1941767612 >> v1:172.21.2.125:6789/0 conn(0x7f7e140ee3e0 legacy=0x7f7e140ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 -- 172.21.2.117:0/1941767612 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140eefd0 con 0x7f7e140d18f0 2021-02-10T11:50:21.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e13fff700 1 --2- 172.21.2.117:0/1941767612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140d18f0 0x7f7e140eaa20 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f7e04006700 tx=0x7f7e04009b80).ready entity=mon.0 client_cookie=f026aa9f3480776e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.026 7f7e12ffd700 1 -- 172.21.2.117:0/1941767612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0400c0c0 con 0x7f7e140d18f0 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e12ffd700 1 -- 172.21.2.117:0/1941767612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0400c340 con 0x7f7e140d18f0 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e12ffd700 1 -- 172.21.2.117:0/1941767612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0400c860 con 0x7f7e140d18f0 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 -- 172.21.2.117:0/1941767612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140d18f0 msgr2=0x7f7e140eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 --2- 172.21.2.117:0/1941767612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140d18f0 0x7f7e140eaa20 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 rx=0x7f7e04006700 tx=0x7f7e04009b80).stop 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 -- 172.21.2.117:0/1941767612 shutdown_connections 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 --2- 172.21.2.117:0/1941767612 >> v2:172.21.2.125:3300/0 conn(0x7f7e140eb040 0x7f7e140eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 --2- 172.21.2.117:0/1941767612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140d18f0 0x7f7e140eaa20 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 -- 172.21.2.117:0/1941767612 >> 172.21.2.117:0/1941767612 conn(0x7f7e140cf060 msgr2=0x7f7e140cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 -- 172.21.2.117:0/1941767612 shutdown_connections 2021-02-10T11:50:21.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.027 7f7e1a7b2700 1 -- 172.21.2.117:0/1941767612 wait complete. 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 Processor -- start 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 -- start start 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e1407e7d0 0x7f7e1407ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e141a2e50 0x7f7e141a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1407f890 con 0x7f7e140a67e0 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1407f310 con 0x7f7e141a2e50 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.042 7f7e1a7b2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e141a3870 con 0x7f7e1407e7d0 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54688/0 (socket says 172.21.2.117:54688) 2021-02-10T11:50:21.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 -- 172.21.2.117:0/4043547564 learned_addr learned my addr 172.21.2.117:0/4043547564 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 -- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e1407e7d0 msgr2=0x7f7e1407ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e1407e7d0 0x7f7e1407ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 -- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e141a2e50 msgr2=0x7f7e141a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e141a2e50 0x7f7e141a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 -- 172.21.2.117:0/4043547564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e141a3a90 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e13fff700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f7e0400b580 tx=0x7f7e0400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e0400bd10 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04002030 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e141a3a90 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1409f8e0 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.043 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e04002550 con 0x7f7e140a67e0 2021-02-10T11:50:21.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.044 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e04000c90 con 0x7f7e140a67e0 2021-02-10T11:50:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.044 7f7e10ff9700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 0x7f7df401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.044 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7e040326a0 con 0x7f7e140a67e0 2021-02-10T11:50:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.045 7f7e137fe700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 0x7f7df401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.047 7f7e137fe700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 0x7f7df401abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.075 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e140ede60 con 0x7f7e140a67e0 2021-02-10T11:50:21.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.078 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e04020f50 con 0x7f7e140a67e0 2021-02-10T11:50:21.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.213 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f7e140ede60 con 0x7f7e140a67e0 2021-02-10T11:50:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.406 7f7e10ff9700 1 -- 172.21.2.117:0/4043547564 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v94) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7e04020f50 con 0x7f7e140a67e0 2021-02-10T11:50:21.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 msgr2=0x7f7df401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 0x7f7df401abc0 crc :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 msgr2=0x7f7e140a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f7e0400b580 tx=0x7f7e0400b920).stop 2021-02-10T11:50:21.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 shutdown_connections 2021-02-10T11:50:21.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e141a2e50 0x7f7e141a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e1407e7d0 0x7f7e1407ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:21.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7df40186c0 0x7f7df401abc0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 --2- 172.21.2.117:0/4043547564 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e140a67e0 0x7f7e140a5f80 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 >> 172.21.2.117:0/4043547564 conn(0x7f7e1409e660 msgr2=0x7f7e140bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:21.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 shutdown_connections 2021-02-10T11:50:21.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.430 7f7e1a7b2700 1 -- 172.21.2.117:0/4043547564 wait complete. 2021-02-10T11:50:21.437 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:21.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 Processor -- start 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 -- start start 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 0x7f120c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0ede60 con 0x7f120c0ef3f0 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0eb9c0 con 0x7f120c0d18e0 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.850 7f1213fe5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0d34b0 con 0x7f120c1b6960 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1211e01700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f120c0d18e0 0x7f120c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47872/0 (socket says 172.21.2.117:47872) 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1211e01700 1 -- 172.21.2.117:0/1932185388 learned_addr learned my addr 172.21.2.117:0/1932185388 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1211600700 1 --1- 172.21.2.117:0/1932185388 >> v1:172.21.2.117:6789/0 conn(0x7f120c0ef3f0 0x7f120c0eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54948/0 (socket says 172.21.2.117:54948) 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1212602700 1 --2- 172.21.2.117:0/1932185388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 0x7f120c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1210dff700 1 -- 172.21.2.117:0/1932185388 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418028433 0 0) 0x7f1204001510 con 0x7f120c0d18e0 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1210dff700 1 -- 172.21.2.117:0/1932185388 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f4002fe0 con 0x7f120c0d18e0 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1212602700 1 -- 172.21.2.117:0/1932185388 >> v1:172.21.2.125:6789/0 conn(0x7f120c0d18e0 legacy=0x7f120c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1212602700 1 -- 172.21.2.117:0/1932185388 >> v1:172.21.2.117:6789/0 conn(0x7f120c0ef3f0 legacy=0x7f120c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1212602700 1 -- 172.21.2.117:0/1932185388 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0d1ce0 con 0x7f120c1b6960 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1212602700 1 --2- 172.21.2.117:0/1932185388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 0x7f120c1b8d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f11fc006ad0 tx=0x7f11fc009a50).ready entity=mon.2 client_cookie=9bf41ca0ff39150d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1210dff700 1 -- 172.21.2.117:0/1932185388 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11fc0066a0 con 0x7f120c1b6960 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.851 7f1210dff700 1 -- 172.21.2.117:0/1932185388 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc006410 con 0x7f120c1b6960 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1210dff700 1 -- 172.21.2.117:0/1932185388 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11fc0024d0 con 0x7f120c1b6960 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 -- 172.21.2.117:0/1932185388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 msgr2=0x7f120c1b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 --2- 172.21.2.117:0/1932185388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 0x7f120c1b8d70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f11fc006ad0 tx=0x7f11fc009a50).stop 2021-02-10T11:50:21.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 -- 172.21.2.117:0/1932185388 shutdown_connections 2021-02-10T11:50:21.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 --2- 172.21.2.117:0/1932185388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c1b6960 0x7f120c1b8d70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 -- 172.21.2.117:0/1932185388 >> 172.21.2.117:0/1932185388 conn(0x7f120c0ced60 msgr2=0x7f120c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:21.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 -- 172.21.2.117:0/1932185388 shutdown_connections 2021-02-10T11:50:21.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.852 7f1213fe5700 1 -- 172.21.2.117:0/1932185388 wait complete. 2021-02-10T11:50:21.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 Processor -- start 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 -- start start 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f120c0e0800 0x7f120c0e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 0x7f120c0b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c196e60 con 0x7f120c197360 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0e14d0 con 0x7f120c0e0800 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.867 7f1213fe5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c196a30 con 0x7f120c196480 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1212602700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1212602700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54704/0 (socket says 172.21.2.117:54704) 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1212602700 1 -- 172.21.2.117:0/3443596640 learned_addr learned my addr 172.21.2.117:0/3443596640 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 0x7f120c0b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211e01700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f120c0e0800 0x7f120c0e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 -- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f120c0e0800 msgr2=0x7f120c0e0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f120c0e0800 0x7f120c0e0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 -- 172.21.2.117:0/3443596640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 msgr2=0x7f120c197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 -- 172.21.2.117:0/3443596640 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0b0d60 con 0x7f120c196480 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1212602700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1211600700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 0x7f120c0b0740 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f120000a4c0 tx=0x7f120000a860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f12000010b0 con 0x7f120c196480 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200000e20 con 0x7f120c196480 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c0b0d60 con 0x7f120c196480 2021-02-10T11:50:21.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.868 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1200004540 con 0x7f120c196480 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.869 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c0b0d60 con 0x7f120c196480 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.869 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1200007ef0 con 0x7f120c196480 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.869 7f120a7fc700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 0x7f11ec01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.869 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1200032680 con 0x7f120c196480 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.870 7f1211e01700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 0x7f11ec01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:21.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.870 7f1211e01700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 0x7f11ec01abd0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:21.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.900 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c1986d0 con 0x7f120c196480 2021-02-10T11:50:21.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:21.906 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1200000e20 con 0x7f120c196480 2021-02-10T11:50:22.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.036 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120c1986d0 con 0x7f120c196480 2021-02-10T11:50:22.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.036 7f120a7fc700 1 -- 172.21.2.117:0/3443596640 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f1200007ef0 con 0x7f120c196480 2021-02-10T11:50:22.035 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 94 2021-02-10T11:50:22.035 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:22.035 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":94,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:50:21.403231","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7010":{"gid":7010,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/143190062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":143190062},{"type":"v1","addr":"172.21.2.125:6835","nonce":143190062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 msgr2=0x7f11ec01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 0x7f11ec01abd0 crc :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 msgr2=0x7f120c0b0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 0x7f120c0b0740 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f120000a4c0 tx=0x7f120000a860).stop 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 shutdown_connections 2021-02-10T11:50:22.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11ec0186d0 0x7f11ec01abd0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f120c197360 0x7f120c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f120c196480 0x7f120c0b0740 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 --2- 172.21.2.117:0/3443596640 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f120c0e0800 0x7f120c0e0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 >> 172.21.2.117:0/3443596640 conn(0x7f120c094230 msgr2=0x7f120c08d000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 shutdown_connections 2021-02-10T11:50:22.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.038 7f1213fe5700 1 -- 172.21.2.117:0/3443596640 wait complete. 2021-02-10T11:50:22.044 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:22.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 Processor -- start 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 -- start start 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0140d18e0 0x7fd0140ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0140ede60 con 0x7fd0140eb000 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0140d1ce0 con 0x7fd0140ef3f0 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd01b05d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0140ef7f0 con 0x7fd0140d18e0 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.460 7fd018e79700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0140d18e0 0x7fd0140ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0140d18e0 0x7fd0140ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54712/0 (socket says 172.21.2.117:54712) 2021-02-10T11:50:22.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 -- 172.21.2.117:0/1344617779 learned_addr learned my addr 172.21.2.117:0/1344617779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd013fff700 1 --1- 172.21.2.117:0/1344617779 >> v1:172.21.2.117:6789/0 conn(0x7fd0140eb000 0x7fd0140eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:54960/0 (socket says 172.21.2.117:54960) 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 -- 172.21.2.117:0/1344617779 >> v1:172.21.2.125:6789/0 conn(0x7fd0140ef3f0 legacy=0x7fd0141b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 -- 172.21.2.117:0/1344617779 >> v1:172.21.2.117:6789/0 conn(0x7fd0140eb000 legacy=0x7fd0140eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 -- 172.21.2.117:0/1344617779 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0141bb180 con 0x7fd0140d18e0 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd018e79700 1 --2- 172.21.2.117:0/1344617779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0140d18e0 0x7fd0140ea9e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7fcffc006700 tx=0x7fcffc009b80).ready entity=mon.0 client_cookie=57e672cc555dfa6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd0137fe700 1 -- 172.21.2.117:0/1344617779 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc00c0c0 con 0x7fd0140d18e0 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd0137fe700 1 -- 172.21.2.117:0/1344617779 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc00c440 con 0x7fd0140d18e0 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd0137fe700 1 -- 172.21.2.117:0/1344617779 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcffc00c960 con 0x7fd0140d18e0 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd01b05d700 1 -- 172.21.2.117:0/1344617779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0140d18e0 msgr2=0x7fd0140ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.461 7fd01b05d700 1 --2- 172.21.2.117:0/1344617779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0140d18e0 0x7fd0140ea9e0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7fcffc006700 tx=0x7fcffc009b80).stop 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.462 7fd01b05d700 1 -- 172.21.2.117:0/1344617779 shutdown_connections 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.462 7fd01b05d700 1 --2- 172.21.2.117:0/1344617779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0140d18e0 0x7fd0140ea9e0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.462 7fd01b05d700 1 -- 172.21.2.117:0/1344617779 >> 172.21.2.117:0/1344617779 conn(0x7fd0140ced60 msgr2=0x7fd0140cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:22.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.462 7fd01b05d700 1 -- 172.21.2.117:0/1344617779 shutdown_connections 2021-02-10T11:50:22.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.462 7fd01b05d700 1 -- 172.21.2.117:0/1344617779 wait complete. 2021-02-10T11:50:22.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 Processor -- start 2021-02-10T11:50:22.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 -- start start 2021-02-10T11:50:22.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 0x7fd0141b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 0x7fd0141b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01408d0f0 con 0x7fd0141c90a0 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0141c9ac0 con 0x7fd0140ecb00 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01b05d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0141c9ce0 con 0x7fd0141b6030 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01967a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01967a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54716/0 (socket says 172.21.2.117:54716) 2021-02-10T11:50:22.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.477 7fd01967a700 1 -- 172.21.2.117:0/1185390515 learned_addr learned my addr 172.21.2.117:0/1185390515 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:22.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd018e79700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 0x7fd0141b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd013fff700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 0x7fd0141b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 -- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 msgr2=0x7fd0141b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 0x7fd0141b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 -- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 msgr2=0x7fd0141b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 0x7fd0141b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 -- 172.21.2.117:0/1185390515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0141c9f00 con 0x7fd0141c90a0 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd013fff700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 0x7fd0141b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd018e79700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 0x7fd0141b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01967a700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7fd00c006ed0 tx=0x7fd00c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00c00bd50 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c0009b0 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd00c0062a0 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0141c9f00 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.478 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01408d960 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.479 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd00c006020 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.479 7fd0117fa700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 0x7fcff801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.479 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd00c0328e0 con 0x7fd0141c90a0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.479 7fd018e79700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 0x7fcff801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:22.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.480 7fd018e79700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 0x7fcff801abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:22.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.510 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014086140 con 0x7fd0141c90a0 2021-02-10T11:50:22.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.513 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd00c000bc0 con 0x7fd0141c90a0 2021-02-10T11:50:22.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.636 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0140d18e0 con 0x7fd0141c90a0 2021-02-10T11:50:22.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.637 7fd0117fa700 1 -- 172.21.2.117:0/1185390515 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fd00c006000 con 0x7fd0141c90a0 2021-02-10T11:50:22.634 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:22.634 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":117,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:20.397701","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:50:17.365216","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:50:18.436070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:22.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.638 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 msgr2=0x7fcff801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.638 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 0x7fcff801abc0 crc :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.638 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 msgr2=0x7fd0141c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:22.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.638 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7fd00c006ed0 tx=0x7fd00c00b960).stop 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 shutdown_connections 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcff80186c0 0x7fcff801abc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0141c90a0 0x7fd0141c94a0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0141b6030 0x7fd0141b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 --2- 172.21.2.117:0/1185390515 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0140ecb00 0x7fd0141b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:22.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 >> 172.21.2.117:0/1185390515 conn(0x7fd014086d40 msgr2=0x7fd0140f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:22.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 shutdown_connections 2021-02-10T11:50:22.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:22.639 7fd01b05d700 1 -- 172.21.2.117:0/1185390515 wait complete. 2021-02-10T11:50:22.644 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:50:23.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.070 7f5fa1b62700 1 Processor -- start 2021-02-10T11:50:23.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.070 7f5fa1b62700 1 -- start start 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5fa1b62700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5fa1b62700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5f9c07a0e0 0x7f5f9c1b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5fa1b62700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0e5770 con 0x7f5f9c07ab40 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5fa1b62700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9c07a640 con 0x7f5f9c079ce0 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5fa1b62700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9c0d1900 con 0x7f5f9c07a0e0 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54724/0 (socket says 172.21.2.117:54724) 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 -- 172.21.2.117:0/1235352981 learned_addr learned my addr 172.21.2.117:0/1235352981 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9bfff700 1 --2- 172.21.2.117:0/1235352981 >> v2:172.21.2.125:3300/0 conn(0x7f5f9c07a0e0 0x7f5f9c1b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 -- 172.21.2.117:0/1235352981 >> v2:172.21.2.125:3300/0 conn(0x7f5f9c07a0e0 msgr2=0x7f5f9c1b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 --2- 172.21.2.117:0/1235352981 >> v2:172.21.2.125:3300/0 conn(0x7f5f9c07a0e0 0x7f5f9c1b8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 -- 172.21.2.117:0/1235352981 >> v1:172.21.2.125:6789/0 conn(0x7f5f9c07ab40 legacy=0x7f5f9c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 -- 172.21.2.117:0/1235352981 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c07bf50 con 0x7f5f9c079ce0 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9affd700 1 --2- 172.21.2.117:0/1235352981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f5f8c006700 tx=0x7f5f8c009b80).ready entity=mon.0 client_cookie=3c263ab3f317d507 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.071 7f5f9a7fc700 1 -- 172.21.2.117:0/1235352981 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c00c0c0 con 0x7f5f9c079ce0 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5f9a7fc700 1 -- 172.21.2.117:0/1235352981 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c00c340 con 0x7f5f9c079ce0 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5f9a7fc700 1 -- 172.21.2.117:0/1235352981 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f8c00c820 con 0x7f5f9c079ce0 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 -- 172.21.2.117:0/1235352981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c079ce0 msgr2=0x7f5f9c1b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 --2- 172.21.2.117:0/1235352981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f5f8c006700 tx=0x7f5f8c009b80).stop 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 -- 172.21.2.117:0/1235352981 shutdown_connections 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 --2- 172.21.2.117:0/1235352981 >> v2:172.21.2.125:3300/0 conn(0x7f5f9c07a0e0 0x7f5f9c1b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 --2- 172.21.2.117:0/1235352981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c079ce0 0x7f5f9c1b59d0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 -- 172.21.2.117:0/1235352981 >> 172.21.2.117:0/1235352981 conn(0x7f5f9c0d0910 msgr2=0x7f5f9c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 -- 172.21.2.117:0/1235352981 shutdown_connections 2021-02-10T11:50:23.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.072 7f5fa1b62700 1 -- 172.21.2.117:0/1235352981 wait complete. 2021-02-10T11:50:23.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 Processor -- start 2021-02-10T11:50:23.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 -- start start 2021-02-10T11:50:23.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 0x7f5f9c100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f9c09a4d0 0x7f5f9c09a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c100b30 con 0x7f5f9c099960 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c099f50 con 0x7f5f9c100db0 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5fa1b62700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c09aef0 con 0x7f5f9c09a4d0 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5f9b7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5f9affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 0x7f5f9c100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5f9b7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:59114/0 (socket says 172.21.2.117:59114) 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5f9b7fe700 1 -- 172.21.2.117:0/1930630604 learned_addr learned my addr 172.21.2.117:0/1930630604 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.087 7f5f9bfff700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f9c09a4d0 0x7f5f9c09a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 -- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f9c09a4d0 msgr2=0x7f5f9c09a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f9c09a4d0 0x7f5f9c09a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 -- 172.21.2.117:0/1930630604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 msgr2=0x7f5f9c100510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 0x7f5f9c100510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 -- 172.21.2.117:0/1930630604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c09b0d0 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9affd700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 0x7f5f9c100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f9b7fe700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f5f84006f00 tx=0x7f5f8400b940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f84000c60 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84002030 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c09b0d0 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f84002550 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.088 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c09b0d0 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.089 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f84000c10 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.089 7f5f93fff700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 0x7f5f8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.089 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5f84032590 con 0x7f5f9c100db0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.089 7f5f9affd700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 0x7f5f8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.090 7f5f9affd700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 0x7f5f8001abc0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.120 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c09b950 con 0x7f5f9c100db0 2021-02-10T11:50:23.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.125 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f84006b90 con 0x7f5f9c100db0 2021-02-10T11:50:23.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.255 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5f9c09b950 con 0x7f5f9c100db0 2021-02-10T11:50:23.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.260 7f5f93fff700 1 -- 172.21.2.117:0/1930630604 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v67) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5f840021f0 con 0x7f5f9c100db0 2021-02-10T11:50:23.259 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:50:23.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 msgr2=0x7f5f8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 0x7f5f8001abc0 crc :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 msgr2=0x7f5f9c0ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f5f84006f00 tx=0x7f5f8400b940).stop 2021-02-10T11:50:23.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 shutdown_connections 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f800186c0 0x7f5f8001abc0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f9c099960 0x7f5f9c100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f9c09a4d0 0x7f5f9c09a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 --2- 172.21.2.117:0/1930630604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f9c100db0 0x7f5f9c0ffdf0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 >> 172.21.2.117:0/1930630604 conn(0x7f5f9c08f530 msgr2=0x7f5f9c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 shutdown_connections 2021-02-10T11:50:23.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.264 7f5fa1b62700 1 -- 172.21.2.117:0/1930630604 wait complete. 2021-02-10T11:50:23.270 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:50:23.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.696 7fe96f77e700 1 Processor -- start 2021-02-10T11:50:23.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.696 7fe96f77e700 1 -- start start 2021-02-10T11:50:23.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96f77e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 0x7fe9680794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96f77e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96f77e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680e5770 con 0x7fe968079b10 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96f77e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96807a310 con 0x7fe96807ab40 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96f77e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680d18c0 con 0x7fe968079f10 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96d59a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 0x7fe9680794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96cd99700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fe968079b10 0x7fe96807af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:47912/0 (socket says 172.21.2.117:47912) 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96cd99700 1 -- 172.21.2.117:0/776587964 learned_addr learned my addr 172.21.2.117:0/776587964 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:52416/0 (socket says 172.21.2.117:52416) 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96d59a700 1 --2- 172.21.2.117:0/776587964 >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 0x7fe9680794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54736/0 (socket says 172.21.2.117:54736) 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 -- 172.21.2.117:0/776587964 >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 msgr2=0x7fe9680794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 --2- 172.21.2.117:0/776587964 >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 0x7fe9680794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 -- 172.21.2.117:0/776587964 >> v1:172.21.2.125:6789/0 conn(0x7fe968079b10 legacy=0x7fe96807af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.697 7fe96dd9b700 1 -- 172.21.2.117:0/776587964 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680d1c60 con 0x7fe968079f10 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe96dd9b700 1 --2- 172.21.2.117:0/776587964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe960006e50 tx=0x7fe960009e70).ready entity=mon.2 client_cookie=3a1a445d3c945a6d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe95ffff700 1 -- 172.21.2.117:0/776587964 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96000b510 con 0x7fe968079f10 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe95ffff700 1 -- 172.21.2.117:0/776587964 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe960006790 con 0x7fe968079f10 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe95ffff700 1 -- 172.21.2.117:0/776587964 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96000b940 con 0x7fe968079f10 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe96f77e700 1 -- 172.21.2.117:0/776587964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 msgr2=0x7fe96807b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.698 7fe96f77e700 1 --2- 172.21.2.117:0/776587964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe960006e50 tx=0x7fe960009e70).stop 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 -- 172.21.2.117:0/776587964 shutdown_connections 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 --2- 172.21.2.117:0/776587964 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe968079f10 0x7fe96807b730 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 --2- 172.21.2.117:0/776587964 >> v2:172.21.2.117:3300/0 conn(0x7fe96807ab40 0x7fe9680794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 -- 172.21.2.117:0/776587964 >> 172.21.2.117:0/776587964 conn(0x7fe9680d0910 msgr2=0x7fe9680d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 -- 172.21.2.117:0/776587964 shutdown_connections 2021-02-10T11:50:23.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.699 7fe96f77e700 1 -- 172.21.2.117:0/776587964 wait complete. 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.713 7fe96f77e700 1 Processor -- start 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 -- start start 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9680dc6e0 0x7fe968191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9680fe660 0x7fe9680fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680ff300 con 0x7fe9680ff800 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680a8b30 con 0x7fe9680dc6e0 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96f77e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680dc230 con 0x7fe9680fe660 2021-02-10T11:50:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96dd9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96dd9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54742/0 (socket says 172.21.2.117:54742) 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.714 7fe96dd9b700 1 -- 172.21.2.117:0/4276107463 learned_addr learned my addr 172.21.2.117:0/4276107463 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 -- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9680fe660 msgr2=0x7fe9680fea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9680fe660 0x7fe9680fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 -- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9680dc6e0 msgr2=0x7fe968191c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9680dc6e0 0x7fe968191c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 -- 172.21.2.117:0/4276107463 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968100220 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96dd9b700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7fe960006c50 tx=0x7fe96000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9600093d0 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96000be60 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96000c570 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe968100220 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.715 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968100220 con 0x7fe9680ff800 2021-02-10T11:50:23.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.716 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe960002ef0 con 0x7fe9680ff800 2021-02-10T11:50:23.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.716 7fe95dffb700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 0x7fe94801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:23.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.716 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe960032cc0 con 0x7fe9680ff800 2021-02-10T11:50:23.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.716 7fe96d59a700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 0x7fe94801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:23.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.716 7fe96d59a700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 0x7fe94801abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:23.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.747 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9680da9a0 con 0x7fe9680ff800 2021-02-10T11:50:23.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.749 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe960000df0 con 0x7fe9680ff800 2021-02-10T11:50:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.879 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe9680da9a0 con 0x7fe9680ff800 2021-02-10T11:50:23.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.883 7fe95dffb700 1 -- 172.21.2.117:0/4276107463 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v68) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe960002ef0 con 0x7fe9680ff800 2021-02-10T11:50:23.883 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:50:23.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.887 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 msgr2=0x7fe94801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.887 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 0x7fe94801abc0 crc :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.887 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 msgr2=0x7fe9680ffc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.887 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7fe960006c50 tx=0x7fe96000bd20).stop 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 shutdown_connections 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe948018730 0x7fe94801abc0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9680ff800 0x7fe9680ffc00 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:23.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9680fe660 0x7fe9680fea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:23.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 --2- 172.21.2.117:0/4276107463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9680dc6e0 0x7fe968191c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:23.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 >> 172.21.2.117:0/4276107463 conn(0x7fe96819f690 msgr2=0x7fe968096490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:23.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 shutdown_connections 2021-02-10T11:50:23.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:23.888 7fe96f77e700 1 -- 172.21.2.117:0/4276107463 wait complete. 2021-02-10T11:50:23.893 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.319 7f2ab05d8700 1 Processor -- start 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 -- start start 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2aa807ab40 0x7f2aa80794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2aa8079b10 0x7f2aa8079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa807af40 0x7f2aa807b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa80e5770 con 0x7f2aa807ab40 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa80d18c0 con 0x7f2aa8079b10 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2ab05d8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80d1c60 con 0x7f2aa807af40 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2aae3f4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2aa807ab40 0x7f2aa80794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.320 7f2aae3f4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2aa807ab40 0x7f2aa80794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54746/0 (socket says 172.21.2.117:54746) 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 -- 172.21.2.117:0/3386125361 learned_addr learned my addr 172.21.2.117:0/3386125361 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 -- 172.21.2.117:0/3386125361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa807af40 msgr2=0x7f2aa807b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 --2- 172.21.2.117:0/3386125361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa807af40 0x7f2aa807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 -- 172.21.2.117:0/3386125361 >> v2:172.21.2.125:3300/0 conn(0x7f2aa8079b10 msgr2=0x7f2aa8079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 --2- 172.21.2.117:0/3386125361 >> v2:172.21.2.125:3300/0 conn(0x7f2aa8079b10 0x7f2aa8079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 -- 172.21.2.117:0/3386125361 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa807b960 con 0x7f2aa807ab40 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aae3f4700 1 --2- 172.21.2.117:0/3386125361 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa807ab40 0x7f2aa80794f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f2aa4006700 tx=0x7f2aa4009b80).ready entity=mon.0 client_cookie=b435782c73f64867 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.321 7f2aad3f2700 1 -- 172.21.2.117:0/3386125361 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2aa400c0c0 con 0x7f2aa807ab40 2021-02-10T11:50:24.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2aad3f2700 1 -- 172.21.2.117:0/3386125361 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa400c340 con 0x7f2aa807ab40 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2aad3f2700 1 -- 172.21.2.117:0/3386125361 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2aa400c860 con 0x7f2aa807ab40 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2ab05d8700 1 -- 172.21.2.117:0/3386125361 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa807ab40 msgr2=0x7f2aa80794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2ab05d8700 1 --2- 172.21.2.117:0/3386125361 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa807ab40 0x7f2aa80794f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f2aa4006700 tx=0x7f2aa4009b80).stop 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2ab05d8700 1 -- 172.21.2.117:0/3386125361 shutdown_connections 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2ab05d8700 1 --2- 172.21.2.117:0/3386125361 >> v2:172.21.2.125:3300/0 conn(0x7f2aa8079b10 0x7f2aa8079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.322 7f2ab05d8700 1 --2- 172.21.2.117:0/3386125361 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa807af40 0x7f2aa807b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.323 7f2ab05d8700 1 --2- 172.21.2.117:0/3386125361 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa807ab40 0x7f2aa80794f0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.323 7f2ab05d8700 1 -- 172.21.2.117:0/3386125361 >> 172.21.2.117:0/3386125361 conn(0x7f2aa80d0910 msgr2=0x7f2aa80d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.323 7f2ab05d8700 1 -- 172.21.2.117:0/3386125361 shutdown_connections 2021-02-10T11:50:24.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.323 7f2ab05d8700 1 -- 172.21.2.117:0/3386125361 wait complete. 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.337 7f2ab05d8700 1 Processor -- start 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.337 7f2ab05d8700 1 -- start start 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa81a2ce0 0x7f2aa80dc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 0x7f2aa809da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa80dbc20 con 0x7f2aa80f73f0 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa81a30e0 con 0x7f2aa809d610 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2ab05d8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa809e030 con 0x7f2aa81a2ce0 2021-02-10T11:50:24.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54750/0 (socket says 172.21.2.117:54750) 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 -- 172.21.2.117:0/267065870 learned_addr learned my addr 172.21.2.117:0/267065870 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aae3f4700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa81a2ce0 0x7f2aa80dc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aadbf3700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 0x7f2aa809da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 -- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa81a2ce0 msgr2=0x7f2aa80dc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa81a2ce0 0x7f2aa80dc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 -- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 msgr2=0x7f2aa809da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 0x7f2aa809da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 -- 172.21.2.117:0/267065870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa80f83b0 con 0x7f2aa80f73f0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aadbf3700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 0x7f2aa809da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.338 7f2aaebf5700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f2aa0009980 tx=0x7f2aa000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2aa0003020 con 0x7f2aa80f73f0 2021-02-10T11:50:24.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0005a80 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa80f83b0 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa80f8100 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2aa0006030 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2aa0005c60 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 0x7f2a9001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.339 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2aa0032ca0 con 0x7f2aa80f73f0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.340 7f2aae3f4700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 0x7f2a9001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.340 7f2aae3f4700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 0x7f2a9001ac10 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.370 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa80da9a0 con 0x7f2aa80f73f0 2021-02-10T11:50:24.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.373 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2aa0005c90 con 0x7f2aa80f73f0 2021-02-10T11:50:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.502 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2aa80e5770 con 0x7f2aa80f73f0 2021-02-10T11:50:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.506 7f2a9effd700 1 -- 172.21.2.117:0/267065870 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v69) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2aa0005a80 con 0x7f2aa80f73f0 2021-02-10T11:50:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:50:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.511 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 msgr2=0x7f2a9001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.511 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 0x7f2a9001ac10 crc :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.511 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 msgr2=0x7f2aa80f77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.511 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7f2aa0009980 tx=0x7f2aa000bcd0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 shutdown_connections 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2a90018730 0x7f2a9001ac10 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2aa80f73f0 0x7f2aa80f77f0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2aa809d610 0x7f2aa809da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 --2- 172.21.2.117:0/267065870 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2aa81a2ce0 0x7f2aa80dc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 >> 172.21.2.117:0/267065870 conn(0x7f2aa819f690 msgr2=0x7f2aa81a0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 shutdown_connections 2021-02-10T11:50:24.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.512 7f2ab05d8700 1 -- 172.21.2.117:0/267065870 wait complete. 2021-02-10T11:50:24.517 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.944 7f11cb334700 1 Processor -- start 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.944 7f11cb334700 1 -- start start 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11cb334700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11c407ab40 0x7f11c40794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11cb334700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4079f10 0x7f11c407b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11cb334700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40e5770 con 0x7f11c4079b10 2021-02-10T11:50:24.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11cb334700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c407a310 con 0x7f11c407ab40 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11cb334700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40d18c0 con 0x7f11c4079f10 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11c407ab40 0x7f11c40794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f11c407ab40 0x7f11c40794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54758/0 (socket says 172.21.2.117:54758) 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 -- 172.21.2.117:0/3427955131 learned_addr learned my addr 172.21.2.117:0/3427955131 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 -- 172.21.2.117:0/3427955131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4079f10 msgr2=0x7f11c407b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 --2- 172.21.2.117:0/3427955131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4079f10 0x7f11c407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 -- 172.21.2.117:0/3427955131 >> v1:172.21.2.125:6789/0 conn(0x7f11c4079b10 legacy=0x7f11c407af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.945 7f11c9150700 1 -- 172.21.2.117:0/3427955131 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40d1c60 con 0x7f11c407ab40 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11c9150700 1 --2- 172.21.2.117:0/3427955131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c407ab40 0x7f11c40794f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f11c0006700 tx=0x7f11c0009b80).ready entity=mon.0 client_cookie=87185f8b05f2154c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11b7fff700 1 -- 172.21.2.117:0/3427955131 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c000c0c0 con 0x7f11c407ab40 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11b7fff700 1 -- 172.21.2.117:0/3427955131 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c000c380 con 0x7f11c407ab40 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11b7fff700 1 -- 172.21.2.117:0/3427955131 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c000c8a0 con 0x7f11c407ab40 2021-02-10T11:50:24.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11cb334700 1 -- 172.21.2.117:0/3427955131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c407ab40 msgr2=0x7f11c40794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.946 7f11cb334700 1 --2- 172.21.2.117:0/3427955131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c407ab40 0x7f11c40794f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f11c0006700 tx=0x7f11c0009b80).stop 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 -- 172.21.2.117:0/3427955131 shutdown_connections 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 --2- 172.21.2.117:0/3427955131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4079f10 0x7f11c407b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 --2- 172.21.2.117:0/3427955131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c407ab40 0x7f11c40794f0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 -- 172.21.2.117:0/3427955131 >> 172.21.2.117:0/3427955131 conn(0x7f11c40d0910 msgr2=0x7f11c40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 -- 172.21.2.117:0/3427955131 shutdown_connections 2021-02-10T11:50:24.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.947 7f11cb334700 1 -- 172.21.2.117:0/3427955131 wait complete. 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.961 7f11cb334700 1 Processor -- start 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.961 7f11cb334700 1 -- start start 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 0x7f11c4092370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4093cd0 0x7f11c419b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4091df0 con 0x7f11c40940d0 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40ff350 con 0x7f11c40dc3e0 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11cb334700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40dc7e0 con 0x7f11c4093cd0 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11c9951700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11c9951700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54768/0 (socket says 172.21.2.117:54768) 2021-02-10T11:50:24.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11c9951700 1 -- 172.21.2.117:0/1922626008 learned_addr learned my addr 172.21.2.117:0/1922626008 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11c9150700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 0x7f11c4092370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.962 7f11c894f700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4093cd0 0x7f11c419b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 -- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4093cd0 msgr2=0x7f11c419b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4093cd0 0x7f11c419b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 -- 172.21.2.117:0/1922626008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 msgr2=0x7f11c419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 -- 172.21.2.117:0/1922626008 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40db270 con 0x7f11c40dc3e0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9951700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11c9150700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 0x7f11c4092370 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f11c00096d0 tx=0x7f11c000b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c000bcb0 con 0x7f11c40dc3e0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c0002030 con 0x7f11c40dc3e0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11cb334700 1 -- 172.21.2.117:0/1922626008 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c40db270 con 0x7f11c40dc3e0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11cb334700 1 -- 172.21.2.117:0/1922626008 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c40db270 con 0x7f11c40dc3e0 2021-02-10T11:50:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.963 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11c0002440 con 0x7f11c40dc3e0 2021-02-10T11:50:24.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.964 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f11c0002210 con 0x7f11c40dc3e0 2021-02-10T11:50:24.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.964 7f11b5ffb700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 0x7f11a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:24.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.964 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f11c0032600 con 0x7f11c40dc3e0 2021-02-10T11:50:24.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.964 7f11c894f700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 0x7f11a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:24.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.965 7f11c894f700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 0x7f11a401abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:24.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.994 7f11cb334700 1 -- 172.21.2.117:0/1922626008 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c40daa00 con 0x7f11c40dc3e0 2021-02-10T11:50:24.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:24.998 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f11c0000c90 con 0x7f11c40dc3e0 2021-02-10T11:50:25.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.130 7f11cb334700 1 -- 172.21.2.117:0/1922626008 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f11c40daa00 con 0x7f11c40dc3e0 2021-02-10T11:50:25.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.134 7f11b5ffb700 1 -- 172.21.2.117:0/1922626008 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v70) v1 ==== 184+0+0 (secure 0 0 0) 0x7f11c001f420 con 0x7f11c40dc3e0 2021-02-10T11:50:25.134 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:50:25.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 -- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 msgr2=0x7f11a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 0x7f11a401abc0 crc :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 -- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 msgr2=0x7f11c4092370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 0x7f11c4092370 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7f11c00096d0 tx=0x7f11c000b8c0).stop 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 -- 172.21.2.117:0/1922626008 shutdown_connections 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f11a40186c0 0x7f11a401abc0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11c40940d0 0x7f11c419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11c4093cd0 0x7f11c419b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 --2- 172.21.2.117:0/1922626008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11c40dc3e0 0x7f11c4092370 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.139 7f11cb334700 1 -- 172.21.2.117:0/1922626008 >> 172.21.2.117:0/1922626008 conn(0x7f11c419f690 msgr2=0x7f11c41919a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.140 7f11cb334700 1 -- 172.21.2.117:0/1922626008 shutdown_connections 2021-02-10T11:50:25.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.140 7f11cb334700 1 -- 172.21.2.117:0/1922626008 wait complete. 2021-02-10T11:50:25.145 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:25.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.557 7fa2b2ad6700 1 Processor -- start 2021-02-10T11:50:25.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.557 7fa2b2ad6700 1 -- start start 2021-02-10T11:50:25.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2b2ad6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 0x7fa2ac0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:25.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2b2ad6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0ede60 con 0x7fa2ac0eb000 2021-02-10T11:50:25.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2b2ad6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2ac0d1ce0 con 0x7fa2ac0ef3f0 2021-02-10T11:50:25.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2b2ad6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac088020 con 0x7fa2ac0d18e0 2021-02-10T11:50:25.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2abfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa2ac0eb000 0x7fa2ac0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55026/0 (socket says 172.21.2.117:55026) 2021-02-10T11:50:25.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2abfff700 1 -- 172.21.2.117:0/1568752949 learned_addr learned my addr 172.21.2.117:0/1568752949 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:25.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.558 7fa2b08f2700 1 --2- 172.21.2.117:0/1568752949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 0x7fa2ac0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:25.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b08f2700 1 -- 172.21.2.117:0/1568752949 >> v1:172.21.2.125:6789/0 conn(0x7fa2ac0ef3f0 legacy=0x7fa2ac1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b08f2700 1 -- 172.21.2.117:0/1568752949 >> v1:172.21.2.117:6789/0 conn(0x7fa2ac0eb000 legacy=0x7fa2ac0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b08f2700 1 -- 172.21.2.117:0/1568752949 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac1bb180 con 0x7fa2ac0d18e0 2021-02-10T11:50:25.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b08f2700 1 --2- 172.21.2.117:0/1568752949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 0x7fa2ac0ea9e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa294006650 tx=0x7fa294009a80).ready entity=mon.2 client_cookie=d99ca7a691161955 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:25.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2ab7fe700 1 -- 172.21.2.117:0/1568752949 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa29400c0c0 con 0x7fa2ac0d18e0 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2ab7fe700 1 -- 172.21.2.117:0/1568752949 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29400c340 con 0x7fa2ac0d18e0 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2ab7fe700 1 -- 172.21.2.117:0/1568752949 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa29400c860 con 0x7fa2ac0d18e0 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b2ad6700 1 -- 172.21.2.117:0/1568752949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 msgr2=0x7fa2ac0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.559 7fa2b2ad6700 1 --2- 172.21.2.117:0/1568752949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 0x7fa2ac0ea9e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fa294006650 tx=0x7fa294009a80).stop 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.560 7fa2b2ad6700 1 -- 172.21.2.117:0/1568752949 shutdown_connections 2021-02-10T11:50:25.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.560 7fa2b2ad6700 1 --2- 172.21.2.117:0/1568752949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0d18e0 0x7fa2ac0ea9e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.560 7fa2b2ad6700 1 -- 172.21.2.117:0/1568752949 >> 172.21.2.117:0/1568752949 conn(0x7fa2ac0ced60 msgr2=0x7fa2ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:25.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.560 7fa2b2ad6700 1 -- 172.21.2.117:0/1568752949 shutdown_connections 2021-02-10T11:50:25.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.560 7fa2b2ad6700 1 -- 172.21.2.117:0/1568752949 wait complete. 2021-02-10T11:50:25.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.574 7fa2b2ad6700 1 Processor -- start 2021-02-10T11:50:25.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 -- start start 2021-02-10T11:50:25.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2ac1b8410 0x7fa2ac0eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:25.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0f5980 0x7fa2ac0ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:25.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:25.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac0efd40 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac1b8940 con 0x7fa2ac1b8410 2021-02-10T11:50:25.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b2ad6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac0ee270 con 0x7fa2ac0f5980 2021-02-10T11:50:25.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b10f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:25.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b10f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54782/0 (socket says 172.21.2.117:54782) 2021-02-10T11:50:25.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b10f3700 1 -- 172.21.2.117:0/2066615612 learned_addr learned my addr 172.21.2.117:0/2066615612 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:25.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2abfff700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0f5980 0x7fa2ac0ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:25.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.575 7fa2b08f2700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2ac1b8410 0x7fa2ac0eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:25.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 -- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0f5980 msgr2=0x7fa2ac0ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0f5980 0x7fa2ac0ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 -- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2ac1b8410 msgr2=0x7fa2ac0eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2ac1b8410 0x7fa2ac0eda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 -- 172.21.2.117:0/2066615612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac0ef1e0 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b10f3700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7fa2a4006ed0 tx=0x7fa2a400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a400bd50 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4000c10 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a4006210 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac0ef1e0 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.576 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac0ef770 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.577 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2a4006020 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.577 7fa2a97fa700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:25.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.577 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2a4032520 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.577 7fa2b08f2700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:25.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.577 7fa2b08f2700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:25.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.608 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac0ef770 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.610 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2a401b3f0 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.738 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2ac0ef770 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.739 7fa2a97fa700 1 -- 172.21.2.117:0/2066615612 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fa2a401b3f0 con 0x7fa2ac0ee7c0 2021-02-10T11:50:25.736 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 94 2021-02-10T11:50:25.737 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:25.737 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":94,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:50:21.403231","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7010":{"gid":7010,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/143190062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":143190062},{"type":"v1","addr":"172.21.2.125:6835","nonce":143190062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 msgr2=0x7fa29001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 crc :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 msgr2=0x7fa2ac0eebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7fa2a4006ed0 tx=0x7fa2a400b960).stop 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 shutdown_connections 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2ac0ee7c0 0x7fa2ac0eebc0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2ac0f5980 0x7fa2ac0ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 --2- 172.21.2.117:0/2066615612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2ac1b8410 0x7fa2ac0eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 >> 172.21.2.117:0/2066615612 conn(0x7fa2ac086d40 msgr2=0x7fa2ac09c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 shutdown_connections 2021-02-10T11:50:25.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:25.741 7fa2b2ad6700 1 -- 172.21.2.117:0/2066615612 wait complete. 2021-02-10T11:50:25.948 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:50:19.395674', 'modified': '2021-02-10 11:50:21.403231', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7010': {'gid': 7010, 'name': 'b', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/143190062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 143190062}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 143190062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:50:25.948 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:50:25.949 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:50:26.349 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:50:26.349 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:50:26.349 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:50:26.349 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7010, 2021-02-10T11:50:26.350 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:50:26.350 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:50:26.350 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9539330039999996, 2021-02-10T11:50:26.350 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 94, 2021-02-10T11:50:26.350 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 117, 2021-02-10T11:50:26.351 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:50:26.351 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.585809190000001 2021-02-10T11:50:26.351 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:50:26.353 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7010, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9539330039999996, "mdsmap_epoch": 94, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 12.585809190000001 } 2021-02-10T11:50:26.354 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 Processor -- start 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 -- start start 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 0x7feb34086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb34085930 con 0x7feb34084c60 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb341b09e0 con 0x7feb340e4e40 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb39314700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340e5cc0 con 0x7feb340e5240 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb3387e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7feb34084c60 0x7feb34085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55036/0 (socket says 172.21.2.117:55036) 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.767 7feb3387e700 1 -- 172.21.2.117:0/4211791414 learned_addr learned my addr 172.21.2.117:0/4211791414 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:26.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3287c700 1 --2- 172.21.2.117:0/4211791414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 0x7feb34086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159334231 0 0) 0x7feb2c001500 con 0x7feb34084c60 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3207b700 1 -- 172.21.2.117:0/4211791414 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20002fe0 con 0x7feb34084c60 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995268549 0 0) 0x7feb24001080 con 0x7feb340e4e40 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3207b700 1 -- 172.21.2.117:0/4211791414 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20003210 con 0x7feb340e4e40 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3287c700 1 -- 172.21.2.117:0/4211791414 >> v1:172.21.2.125:6789/0 conn(0x7feb340e4e40 legacy=0x7feb341b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 267471821 0 0) 0x7feb2c001500 con 0x7feb34084c60 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3287c700 1 -- 172.21.2.117:0/4211791414 >> v1:172.21.2.117:6789/0 conn(0x7feb34084c60 legacy=0x7feb34085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3287c700 1 -- 172.21.2.117:0/4211791414 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34084840 con 0x7feb340e5240 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.768 7feb3287c700 1 --2- 172.21.2.117:0/4211791414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 0x7feb34086ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7feb1c006650 tx=0x7feb1c009a00).ready entity=mon.2 client_cookie=563873e34fa0494d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1c009df0 con 0x7feb340e5240 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c0009b0 con 0x7feb340e5240 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb3207b700 1 -- 172.21.2.117:0/4211791414 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb1c00c2b0 con 0x7feb340e5240 2021-02-10T11:50:26.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 -- 172.21.2.117:0/4211791414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 msgr2=0x7feb34086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 --2- 172.21.2.117:0/4211791414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 0x7feb34086ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7feb1c006650 tx=0x7feb1c009a00).stop 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 -- 172.21.2.117:0/4211791414 shutdown_connections 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 --2- 172.21.2.117:0/4211791414 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb340e5240 0x7feb34086ec0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 -- 172.21.2.117:0/4211791414 >> 172.21.2.117:0/4211791414 conn(0x7feb340ce4e0 msgr2=0x7feb340e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 -- 172.21.2.117:0/4211791414 shutdown_connections 2021-02-10T11:50:26.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.769 7feb39314700 1 -- 172.21.2.117:0/4211791414 wait complete. 2021-02-10T11:50:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.784 7feb39314700 1 Processor -- start 2021-02-10T11:50:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.784 7feb39314700 1 -- start start 2021-02-10T11:50:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.784 7feb39314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.784 7feb39314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb34189d00 0x7feb3418a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.784 7feb39314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb3418a720 0x7feb3418b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb39314700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3418d4b0 con 0x7feb3418d0b0 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb39314700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3418ac30 con 0x7feb3418a720 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb39314700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3419a480 con 0x7feb34189d00 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54792/0 (socket says 172.21.2.117:54792) 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 -- 172.21.2.117:0/2275612721 learned_addr learned my addr 172.21.2.117:0/2275612721 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 -- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb34189d00 msgr2=0x7feb3418a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb34189d00 0x7feb3418a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 -- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb3418a720 msgr2=0x7feb3418b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb3418a720 0x7feb3418b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 -- 172.21.2.117:0/2275612721 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3418b7d0 con 0x7feb3418d0b0 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb3307d700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7feb24006f00 tx=0x7feb2400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb2400bd30 con 0x7feb3418d0b0 2021-02-10T11:50:26.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24000e20 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.785 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feb240054f0 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.786 7feb39314700 1 -- 172.21.2.117:0/2275612721 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3418b7d0 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.786 7feb39314700 1 -- 172.21.2.117:0/2275612721 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3418bd60 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.786 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feb24006030 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.786 7feb2bfff700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 0x7feb1801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.786 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feb240326d0 con 0x7feb3418d0b0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.787 7feb3287c700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 0x7feb1801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:26.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.787 7feb3287c700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 0x7feb1801abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:26.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.817 7feb39314700 1 -- 172.21.2.117:0/2275612721 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3410c260 con 0x7feb3418d0b0 2021-02-10T11:50:26.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.820 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feb24000bc0 con 0x7feb3418d0b0 2021-02-10T11:50:26.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.944 7feb39314700 1 -- 172.21.2.117:0/2275612721 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb34085930 con 0x7feb3418d0b0 2021-02-10T11:50:26.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.944 7feb2bfff700 1 -- 172.21.2.117:0/2275612721 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14240 (secure 0 0 0) 0x7feb24006030 con 0x7feb3418d0b0 2021-02-10T11:50:26.942 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:26.942 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":117,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:20.397701","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:50:17.365216","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:50:18.436070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:26.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 -- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 msgr2=0x7feb1801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 0x7feb1801abc0 crc :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 -- 172.21.2.117:0/2275612721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 msgr2=0x7feb3419e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:26.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7feb24006f00 tx=0x7feb2400b940).stop 2021-02-10T11:50:26.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 -- 172.21.2.117:0/2275612721 shutdown_connections 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feb3418a720 0x7feb3418b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feb34189d00 0x7feb3418a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feb180186c0 0x7feb1801abc0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 --2- 172.21.2.117:0/2275612721 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feb3418d0b0 0x7feb3419e4e0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 -- 172.21.2.117:0/2275612721 >> 172.21.2.117:0/2275612721 conn(0x7feb341a60a0 msgr2=0x7feb3410c490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.946 7feb39314700 1 -- 172.21.2.117:0/2275612721 shutdown_connections 2021-02-10T11:50:26.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:26.947 7feb39314700 1 -- 172.21.2.117:0/2275612721 wait complete. 2021-02-10T11:50:26.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:50:17.365216', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:50:18.436070', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:50:26.952 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.369 7fa2c7263700 1 Processor -- start 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.369 7fa2c7263700 1 -- start start 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.369 7fa2c7263700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c00d18f0 0x7fa2c00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.369 7fa2c7263700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2c00eb040 0x7fa2c00eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.369 7fa2c7263700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c00ede60 con 0x7fa2c00ee3e0 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c7263700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c00eba60 con 0x7fa2c00eb040 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c7263700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00d1cf0 con 0x7fa2c00d18f0 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2c00eb040 0x7fa2c00eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2c00eb040 0x7fa2c00eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54798/0 (socket says 172.21.2.117:54798) 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 -- 172.21.2.117:0/2361815066 learned_addr learned my addr 172.21.2.117:0/2361815066 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:27.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 -- 172.21.2.117:0/2361815066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c00d18f0 msgr2=0x7fa2c00eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 --2- 172.21.2.117:0/2361815066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c00d18f0 0x7fa2c00eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 -- 172.21.2.117:0/2361815066 >> v1:172.21.2.125:6789/0 conn(0x7fa2c00ee3e0 legacy=0x7fa2c00ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 -- 172.21.2.117:0/2361815066 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00eefd0 con 0x7fa2c00eb040 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c487e700 1 --2- 172.21.2.117:0/2361815066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c00eb040 0x7fa2c00eb440 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7fa2b4006700 tx=0x7fa2b4009b00).ready entity=mon.0 client_cookie=2d29347f23c6db55 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2b3fff700 1 -- 172.21.2.117:0/2361815066 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b4000c90 con 0x7fa2c00eb040 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2b3fff700 1 -- 172.21.2.117:0/2361815066 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b400c0c0 con 0x7fa2c00eb040 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2b3fff700 1 -- 172.21.2.117:0/2361815066 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b400c5c0 con 0x7fa2c00eb040 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c7263700 1 -- 172.21.2.117:0/2361815066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c00eb040 msgr2=0x7fa2c00eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.370 7fa2c7263700 1 --2- 172.21.2.117:0/2361815066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c00eb040 0x7fa2c00eb440 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7fa2b4006700 tx=0x7fa2b4009b00).stop 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 -- 172.21.2.117:0/2361815066 shutdown_connections 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 --2- 172.21.2.117:0/2361815066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c00eb040 0x7fa2c00eb440 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 --2- 172.21.2.117:0/2361815066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c00d18f0 0x7fa2c00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 -- 172.21.2.117:0/2361815066 >> 172.21.2.117:0/2361815066 conn(0x7fa2c00cf060 msgr2=0x7fa2c00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 -- 172.21.2.117:0/2361815066 shutdown_connections 2021-02-10T11:50:27.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.371 7fa2c7263700 1 -- 172.21.2.117:0/2361815066 wait complete. 2021-02-10T11:50:27.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.385 7fa2c7263700 1 Processor -- start 2021-02-10T11:50:27.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.385 7fa2c7263700 1 -- start start 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c0093be0 0x7fa2c01a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2c008c850 0x7fa2c008cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0094280 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0093fe0 con 0x7fa2c008c850 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c7263700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00ace00 con 0x7fa2c0093be0 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54806/0 (socket says 172.21.2.117:54806) 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 -- 172.21.2.117:0/2859458577 learned_addr learned my addr 172.21.2.117:0/2859458577 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 -- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c0093be0 msgr2=0x7fa2c01a78b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c0093be0 0x7fa2c01a78b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 -- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2c008c850 msgr2=0x7fa2c008cc50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2c008c850 0x7fa2c008cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 -- 172.21.2.117:0/2859458577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0094800 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.386 7fa2c487e700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fa2b4006f20 tx=0x7fa2b400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b4000c90 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b4002030 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2b4002510 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c0094800 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c0094800 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2b4006340 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 0x7fa2a001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.387 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2b4032950 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.388 7fa2c507f700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 0x7fa2a001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.388 7fa2c507f700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 0x7fa2a001abd0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.419 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c00ede60 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.422 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2b4000c90 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.554 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa2c00ede60 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.554 7fa2b1ffb700 1 -- 172.21.2.117:0/2859458577 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v117) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa2b4000ed0 con 0x7fa2c01a7ed0 2021-02-10T11:50:27.552 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:27.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.556 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 msgr2=0x7fa2a001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.556 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 0x7fa2a001abd0 crc :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.556 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 msgr2=0x7fa2c008c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.556 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7fa2b4006f20 tx=0x7fa2b400b990).stop 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 shutdown_connections 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2a00186d0 0x7fa2a001abd0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2c01a7ed0 0x7fa2c008c230 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2c008c850 0x7fa2c008cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 --2- 172.21.2.117:0/2859458577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2c0093be0 0x7fa2c01a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:27.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 >> 172.21.2.117:0/2859458577 conn(0x7fa2c0092d00 msgr2=0x7fa2c008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 shutdown_connections 2021-02-10T11:50:27.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.557 7fa2c7263700 1 -- 172.21.2.117:0/2859458577 wait complete. 2021-02-10T11:50:27.563 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 Processor -- start 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 -- start start 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f828c0d18f0 0x7f828c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0ef7f0 con 0x7f828c0eb040 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0ede60 con 0x7f828c0ef3f0 2021-02-10T11:50:27.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.981 7f8292723700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c0d1da0 con 0x7f828c0d18f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f828c0d18f0 0x7f828c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f828c0d18f0 0x7f828c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54812/0 (socket says 172.21.2.117:54812) 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 -- 172.21.2.117:0/1438752240 learned_addr learned my addr 172.21.2.117:0/1438752240 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828b7fe700 1 -- 172.21.2.117:0/1438752240 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542021550 0 0) 0x7f8284000d30 con 0x7f828c0ef3f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 -- 172.21.2.117:0/1438752240 >> v1:172.21.2.125:6789/0 conn(0x7f828c0ef3f0 legacy=0x7f828c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 -- 172.21.2.117:0/1438752240 >> v1:172.21.2.117:6789/0 conn(0x7f828c0eb040 legacy=0x7f828c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 -- 172.21.2.117:0/1438752240 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c1bb1c0 con 0x7f828c0d18f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.982 7f828bfff700 1 --2- 172.21.2.117:0/1438752240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0d18f0 0x7f828c0eaa20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f8274006700 tx=0x7f8274009b80).ready entity=mon.0 client_cookie=32d86215ecbe53d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f828b7fe700 1 -- 172.21.2.117:0/1438752240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827400c0c0 con 0x7f828c0d18f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f828b7fe700 1 -- 172.21.2.117:0/1438752240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827400c340 con 0x7f828c0d18f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f828b7fe700 1 -- 172.21.2.117:0/1438752240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827400c860 con 0x7f828c0d18f0 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 -- 172.21.2.117:0/1438752240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0d18f0 msgr2=0x7f828c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:27.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 --2- 172.21.2.117:0/1438752240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0d18f0 0x7f828c0eaa20 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f8274006700 tx=0x7f8274009b80).stop 2021-02-10T11:50:27.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 -- 172.21.2.117:0/1438752240 shutdown_connections 2021-02-10T11:50:27.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 --2- 172.21.2.117:0/1438752240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0d18f0 0x7f828c0eaa20 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:27.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 -- 172.21.2.117:0/1438752240 >> 172.21.2.117:0/1438752240 conn(0x7f828c0cf060 msgr2=0x7f828c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:27.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 -- 172.21.2.117:0/1438752240 shutdown_connections 2021-02-10T11:50:27.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.983 7f8292723700 1 -- 172.21.2.117:0/1438752240 wait complete. 2021-02-10T11:50:27.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.998 7f8292723700 1 Processor -- start 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.998 7f8292723700 1 -- start start 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f828c0ac450 0x7f828c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f828c0b5d20 0x7f828c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0ace70 con 0x7f828c0a1930 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0ad090 con 0x7f828c0b5d20 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8292723700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0b6740 con 0x7f828c0ac450 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54818/0 (socket says 172.21.2.117:54818) 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 -- 172.21.2.117:0/2856999025 learned_addr learned my addr 172.21.2.117:0/2856999025 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8290d40700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f828c0b5d20 0x7f828c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f8283fff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f828c0ac450 0x7f828c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:27.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 -- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f828c0ac450 msgr2=0x7f828c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f828c0ac450 0x7f828c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 -- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f828c0b5d20 msgr2=0x7f828c0b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f828c0b5d20 0x7f828c0b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:27.999 7f828bfff700 1 -- 172.21.2.117:0/2856999025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0b6960 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f828bfff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f827400b5e0 tx=0x7f827400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f827400bd70 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274002030 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f82740024d0 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f8292723700 1 -- 172.21.2.117:0/2856999025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c0b6960 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f8292723700 1 -- 172.21.2.117:0/2856999025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c0b49a0 con 0x7f828c0a1930 2021-02-10T11:50:28.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.000 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8274000c90 con 0x7f828c0a1930 2021-02-10T11:50:28.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.001 7f82897fa700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 0x7f827001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.001 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8274032740 con 0x7f828c0a1930 2021-02-10T11:50:28.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.001 7f8283fff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 0x7f827001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.001 7f8283fff700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 0x7f827001abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.031 7f8292723700 1 -- 172.21.2.117:0/2856999025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c0f67e0 con 0x7f828c0a1930 2021-02-10T11:50:28.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.034 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8274030da0 con 0x7f828c0a1930 2021-02-10T11:50:28.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.162 7f8292723700 1 -- 172.21.2.117:0/2856999025 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f828c0f67e0 con 0x7f828c0a1930 2021-02-10T11:50:28.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.162 7f82897fa700 1 -- 172.21.2.117:0/2856999025 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v117) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8274000c90 con 0x7f828c0a1930 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 -- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 msgr2=0x7f827001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 0x7f827001abc0 crc :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 -- 172.21.2.117:0/2856999025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 msgr2=0x7f828c08a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f827400b5e0 tx=0x7f827400b980).stop 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 -- 172.21.2.117:0/2856999025 shutdown_connections 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f828c0b5d20 0x7f828c0b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f828c0ac450 0x7f828c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f82700186c0 0x7f827001abc0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 --2- 172.21.2.117:0/2856999025 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f828c0a1930 0x7f828c08a220 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 -- 172.21.2.117:0/2856999025 >> 172.21.2.117:0/2856999025 conn(0x7f828c0b6c70 msgr2=0x7f828c0970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:28.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.164 7f8292723700 1 -- 172.21.2.117:0/2856999025 shutdown_connections 2021-02-10T11:50:28.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.165 7f8292723700 1 -- 172.21.2.117:0/2856999025 wait complete. 2021-02-10T11:50:28.209 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.636 7f4e763dc700 1 Processor -- start 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.636 7f4e763dc700 1 -- start start 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 0x7f4e700eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e700eb040 0x7f4e700eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4e700ee3e0 0x7f4e700ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700ede60 con 0x7f4e700eb040 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e700eba60 con 0x7f4e700ee3e0 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e763dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700d1cf0 con 0x7f4e700d18f0 2021-02-10T11:50:28.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e700eb040 0x7f4e700eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4e700eb040 0x7f4e700eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54826/0 (socket says 172.21.2.117:54826) 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 -- 172.21.2.117:0/738715256 learned_addr learned my addr 172.21.2.117:0/738715256 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e749f9700 1 --2- 172.21.2.117:0/738715256 >> v2:172.21.2.125:3300/0 conn(0x7f4e700ee3e0 0x7f4e700ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6ffff700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 0x7f4e700eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 -- 172.21.2.117:0/738715256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 msgr2=0x7f4e700eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 0x7f4e700eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 -- 172.21.2.117:0/738715256 >> v2:172.21.2.125:3300/0 conn(0x7f4e700ee3e0 msgr2=0x7f4e700ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 --2- 172.21.2.117:0/738715256 >> v2:172.21.2.125:3300/0 conn(0x7f4e700ee3e0 0x7f4e700ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 -- 172.21.2.117:0/738715256 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700eee00 con 0x7f4e700eb040 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6ffff700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 0x7f4e700eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.637 7f4e6f7fe700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e700eb040 0x7f4e700eb440 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f4e64006700 tx=0x7f4e64009b80).ready entity=mon.0 client_cookie=983cf957353a417f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e6effd700 1 -- 172.21.2.117:0/738715256 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6400c0c0 con 0x7f4e700eb040 2021-02-10T11:50:28.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e6effd700 1 -- 172.21.2.117:0/738715256 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6400c340 con 0x7f4e700eb040 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e6effd700 1 -- 172.21.2.117:0/738715256 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e6400c860 con 0x7f4e700eb040 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 -- 172.21.2.117:0/738715256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e700eb040 msgr2=0x7f4e700eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e700eb040 0x7f4e700eb440 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f4e64006700 tx=0x7f4e64009b80).stop 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 -- 172.21.2.117:0/738715256 shutdown_connections 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 --2- 172.21.2.117:0/738715256 >> v2:172.21.2.125:3300/0 conn(0x7f4e700ee3e0 0x7f4e700ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e700eb040 0x7f4e700eb440 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 --2- 172.21.2.117:0/738715256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e700d18f0 0x7f4e700eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 -- 172.21.2.117:0/738715256 >> 172.21.2.117:0/738715256 conn(0x7f4e700cf060 msgr2=0x7f4e700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 -- 172.21.2.117:0/738715256 shutdown_connections 2021-02-10T11:50:28.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.638 7f4e763dc700 1 -- 172.21.2.117:0/738715256 wait complete. 2021-02-10T11:50:28.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 Processor -- start 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 -- start start 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 0x7f4e700a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e700a13a0 0x7f4e700a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7007cca0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7018fb70 con 0x7f4e700a13a0 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.653 7f4e763dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700dc6d0 con 0x7f4e701a3930 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54832/0 (socket says 172.21.2.117:54832) 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 -- 172.21.2.117:0/3192523334 learned_addr learned my addr 172.21.2.117:0/3192523334 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e749f9700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e700a13a0 0x7f4e700a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6f7fe700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 0x7f4e700a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 -- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 msgr2=0x7f4e700a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 0x7f4e700a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 -- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e700a13a0 msgr2=0x7f4e700a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e700a13a0 0x7f4e700a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 -- 172.21.2.117:0/3192523334 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7007d220 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6f7fe700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 0x7f4e700a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6ffff700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f4e60006f20 tx=0x7f4e6000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e600009b0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6000bd80 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e600022b0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7007d220 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.654 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7007d220 con 0x7f4e7007c7c0 2021-02-10T11:50:28.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.655 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e6000a000 con 0x7f4e7007c7c0 2021-02-10T11:50:28.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.655 7f4e6cff9700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 0x7f4e5001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:28.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.655 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4e600327b0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.655 7f4e6f7fe700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 0x7f4e5001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:28.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.656 7f4e6f7fe700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 0x7f4e5001abd0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:28.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.687 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e700daff0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.689 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e60000c90 con 0x7f4e7007c7c0 2021-02-10T11:50:28.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.817 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f4e700daff0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.817 7f4e6cff9700 1 -- 172.21.2.117:0/3192523334 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v117) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4e60000ee0 con 0x7f4e7007c7c0 2021-02-10T11:50:28.815 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:28.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 msgr2=0x7f4e5001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 0x7f4e5001abd0 crc :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 msgr2=0x7f4e701a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:28.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f4e60006f20 tx=0x7f4e6000b990).stop 2021-02-10T11:50:28.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 shutdown_connections 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e700a13a0 0x7f4e700a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e701a3930 0x7f4e700a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186d0 0x7f4e5001abd0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 --2- 172.21.2.117:0/3192523334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e7007c7c0 0x7f4e701a3310 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 >> 172.21.2.117:0/3192523334 conn(0x7f4e70087090 msgr2=0x7f4e7008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 shutdown_connections 2021-02-10T11:50:28.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:28.819 7f4e763dc700 1 -- 172.21.2.117:0/3192523334 wait complete. 2021-02-10T11:50:28.825 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:50:28.825 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.243 7fb9a9d04700 1 Processor -- start 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.243 7fb9a9d04700 1 -- start start 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a9d04700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 0x7fb9a40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a9d04700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9a40eb7f0 0x7fb9a41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a9d04700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40ede60 con 0x7fb9a40d18e0 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a9d04700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40ef7f0 con 0x7fb9a40eb7f0 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a9d04700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a40d1ce0 con 0x7fb9a40ef3f0 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9a40eb7f0 0x7fb9a41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9a40eb7f0 0x7fb9a41b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54840/0 (socket says 172.21.2.117:54840) 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 -- 172.21.2.117:0/191983779 learned_addr learned my addr 172.21.2.117:0/191983779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:29.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a2ffd700 1 --2- 172.21.2.117:0/191983779 >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 0x7fb9a40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 -- 172.21.2.117:0/191983779 >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 msgr2=0x7fb9a40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 --2- 172.21.2.117:0/191983779 >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 0x7fb9a40eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 -- 172.21.2.117:0/191983779 >> v1:172.21.2.125:6789/0 conn(0x7fb9a40d18e0 legacy=0x7fb9a40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 -- 172.21.2.117:0/191983779 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a41b8f90 con 0x7fb9a40eb7f0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a2ffd700 1 --2- 172.21.2.117:0/191983779 >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 0x7fb9a40eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.244 7fb9a3fff700 1 --2- 172.21.2.117:0/191983779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40eb7f0 0x7fb9a41b7960 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7fb99c006f00 tx=0x7fb99c009f10).ready entity=mon.0 client_cookie=3ce9ca6f3e2c5483 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a27fc700 1 -- 172.21.2.117:0/191983779 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb99c00b3b0 con 0x7fb9a40eb7f0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a27fc700 1 -- 172.21.2.117:0/191983779 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c00b630 con 0x7fb9a40eb7f0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a27fc700 1 -- 172.21.2.117:0/191983779 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb99c00bb50 con 0x7fb9a40eb7f0 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 -- 172.21.2.117:0/191983779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40eb7f0 msgr2=0x7fb9a41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 --2- 172.21.2.117:0/191983779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40eb7f0 0x7fb9a41b7960 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7fb99c006f00 tx=0x7fb99c009f10).stop 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 -- 172.21.2.117:0/191983779 shutdown_connections 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 --2- 172.21.2.117:0/191983779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40eb7f0 0x7fb9a41b7960 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 --2- 172.21.2.117:0/191983779 >> v2:172.21.2.125:3300/0 conn(0x7fb9a40ef3f0 0x7fb9a40eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 -- 172.21.2.117:0/191983779 >> 172.21.2.117:0/191983779 conn(0x7fb9a40ced60 msgr2=0x7fb9a40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:29.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 -- 172.21.2.117:0/191983779 shutdown_connections 2021-02-10T11:50:29.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.245 7fb9a9d04700 1 -- 172.21.2.117:0/191983779 wait complete. 2021-02-10T11:50:29.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 Processor -- start 2021-02-10T11:50:29.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 -- start start 2021-02-10T11:50:29.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9a41afd30 0x7fb9a40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9a41b0130 0x7fb9a40e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40e80e0 con 0x7fb9a40b43f0 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40e7e20 con 0x7fb9a41afd30 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a9d04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4096a70 con 0x7fb9a41b0130 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54844/0 (socket says 172.21.2.117:54844) 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.260 7fb9a2ffd700 1 -- 172.21.2.117:0/3503537621 learned_addr learned my addr 172.21.2.117:0/3503537621 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 -- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9a41b0130 msgr2=0x7fb9a40e7800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9a41b0130 0x7fb9a40e7800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 -- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9a41afd30 msgr2=0x7fb9a40b3dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9a41afd30 0x7fb9a40b3dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 -- 172.21.2.117:0/3503537621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40e8660 con 0x7fb9a40b43f0 2021-02-10T11:50:29.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a2ffd700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fb98c006f20 tx=0x7fb98c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98c006bb0 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c00bd80 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb98c002220 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a40e8660 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40e8660 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.261 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb98c000c90 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.262 7fb99bfff700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 0x7fb98801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.262 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb98c0327b0 con 0x7fb9a40b43f0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.262 7fb9a37fe700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 0x7fb98801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:29.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.262 7fb9a37fe700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 0x7fb98801abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:29.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.293 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a40e7e20 con 0x7fb9a40b43f0 2021-02-10T11:50:29.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.296 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb98c006bb0 con 0x7fb9a40b43f0 2021-02-10T11:50:29.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.426 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a40e7e20 con 0x7fb9a40b43f0 2021-02-10T11:50:29.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.426 7fb99bfff700 1 -- 172.21.2.117:0/3503537621 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fb98c00bd80 con 0x7fb9a40b43f0 2021-02-10T11:50:29.424 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:29.424 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":94,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:50:21.403231","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7010":{"gid":7010,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/143190062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":143190062},{"type":"v1","addr":"172.21.2.125:6835","nonce":143190062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:50:29.425 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 94 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.428 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 msgr2=0x7fb98801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.428 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 0x7fb98801abc0 crc :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.428 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 msgr2=0x7fb9a40e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.428 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 rx=0x7fb98c006f20 tx=0x7fb98c00b990).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 shutdown_connections 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9880186c0 0x7fb98801abc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9a40b43f0 0x7fb9a40e71e0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9a41b0130 0x7fb9a40e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 --2- 172.21.2.117:0/3503537621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9a41afd30 0x7fb9a40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:29.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 >> 172.21.2.117:0/3503537621 conn(0x7fb9a41a3620 msgr2=0x7fb9a40dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:29.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 shutdown_connections 2021-02-10T11:50:29.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:29.429 7fb9a9d04700 1 -- 172.21.2.117:0/3503537621 wait complete. 2021-02-10T11:50:29.637 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:50:19.395674', 'modified': '2021-02-10 11:50:21.403231', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7010': {'gid': 7010, 'name': 'b', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/143190062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 143190062}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 143190062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:50:29.637 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:50:29.637 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:50:30.034 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:50:30.034 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7010, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.639526306, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 94, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 117, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.271402999999999 2021-02-10T11:50:30.035 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:50:30.038 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7010, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.639526306, "mdsmap_epoch": 94, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 16.271402999999999 } 2021-02-10T11:50:30.038 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:50:30.039 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:50:30.039 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:50:30.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:30.039 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:30.375 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:50:30.720 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:50:30.721 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:50:31.064 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:50:31.065 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:50:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:50:31.065 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:31.066 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:50:31.400 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:50:31.401 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:31.403 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:50:31.747 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:50:31.748 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:50:31.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[161842]: starting ceph client 2021-02-10T11:50:31.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:50:31.815 7fe2134c6f80 -1 init, newargv = 0x55b979214f30 newargc=9 2021-02-10T11:50:31.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[161842]: starting fuse 2021-02-10T11:50:32.090 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:50:32.090 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:50:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:50:32.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:32.091 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:50:32.428 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:50:32.428 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:50:32.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:32.428 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:50:32.428 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:50:32.428 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:50:32.429 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:50:32.430 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:50:32.430 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:50:32.430 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:32.430 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:50:32.430 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:50:32.791 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.161842.asok 2021-02-10T11:50:32.794 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.161842.asok 2021-02-10T11:50:32.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:32.794 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.161842.asok status 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:50:33.185 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "161842", 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10935, 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10935 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:50:33.186 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3528724584 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3528724584 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10935 172.21.2.117:0/3528724584", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3528724584", 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 94, 2021-02-10T11:50:33.187 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 117, 2021-02-10T11:50:33.188 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:50:33.188 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:50:33.188 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:50:33.189 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:33.189 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:33.189 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:33.526 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:50:33.526 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:33.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:33.527 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:33.874 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.287 7f8e27f58700 1 Processor -- start 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.287 7f8e27f58700 1 -- start start 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e27f58700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200ef3f0 0x7f8e200eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e27f58700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8e200eb7f0 0x7f8e201b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e27f58700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e200ede60 con 0x7f8e200d18e0 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e27f58700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e200ef7f0 con 0x7f8e200eb7f0 2021-02-10T11:50:34.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e27f58700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e200d1ce0 con 0x7f8e200ef3f0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e25d74700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8e200d18e0 0x7f8e200ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55114/0 (socket says 172.21.2.117:55114) 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e25d74700 1 -- 172.21.2.117:0/1681897027 learned_addr learned my addr 172.21.2.117:0/1681897027 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596594073 0 0) 0x7f8e1c0010b0 con 0x7f8e200d18e0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.288 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c002fe0 con 0x7f8e200d18e0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e26575700 1 --2- 172.21.2.117:0/1681897027 >> v2:172.21.2.125:3300/0 conn(0x7f8e200eb7f0 0x7f8e201b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25573700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200ef3f0 0x7f8e200eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 792603825 0 0) 0x7f8e1c0010b0 con 0x7f8e200d18e0 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200ef3f0 msgr2=0x7f8e200eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200ef3f0 0x7f8e200eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 >> v2:172.21.2.125:3300/0 conn(0x7f8e200eb7f0 msgr2=0x7f8e201b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 --2- 172.21.2.117:0/1681897027 >> v2:172.21.2.125:3300/0 conn(0x7f8e200eb7f0 0x7f8e201b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e201b8f90 con 0x7f8e200d18e0 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8e1c0015b0 con 0x7f8e200d18e0 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 >> v1:172.21.2.117:6789/0 conn(0x7f8e200d18e0 legacy=0x7f8e200ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e0c003b10 0x7f8e0c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e0c006840 0x7f8e0c008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e0c0095b0 0x7f8e0c00bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c00c380 con 0x7f8e0c003b10 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c050570 con 0x7f8e0c0095b0 2021-02-10T11:50:34.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c050790 con 0x7f8e0c006840 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e26575700 1 --2- 172.21.2.117:0/1681897027 >> v2:172.21.2.125:3300/0 conn(0x7f8e200eb7f0 0x7f8e201b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e0c003b10 0x7f8e0c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25573700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e0c006840 0x7f8e0c008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e26575700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e0c0095b0 0x7f8e0c00bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 -- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e0c006840 msgr2=0x7f8e0c008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e0c006840 0x7f8e0c008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 -- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e0c0095b0 msgr2=0x7f8e0c00bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e0c0095b0 0x7f8e0c00bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25d74700 1 -- 172.21.2.117:0/1681897027 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c050a30 con 0x7f8e0c003b10 2021-02-10T11:50:34.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e26575700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e0c0095b0 0x7f8e0c00bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.289 7f8e25573700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e0c006840 0x7f8e0c008d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e25d74700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e0c003b10 0x7f8e0c005fa0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f8e1c007990 tx=0x7f8e1c007d30).ready entity=mon.0 client_cookie=5c4629c693aa4dc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c0009b0 con 0x7f8e0c003b10 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c00a310 con 0x7f8e0c003b10 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e24d72700 1 -- 172.21.2.117:0/1681897027 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c00a830 con 0x7f8e0c003b10 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 -- 172.21.2.117:0/1681897027 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e0c003b10 msgr2=0x7f8e0c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 --2- 172.21.2.117:0/1681897027 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e0c003b10 0x7f8e0c005fa0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f8e1c007990 tx=0x7f8e1c007d30).stop 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e26575700 1 -- 172.21.2.117:0/1681897027 reap_dead start 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 -- 172.21.2.117:0/1681897027 shutdown_connections 2021-02-10T11:50:34.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 -- 172.21.2.117:0/1681897027 >> 172.21.2.117:0/1681897027 conn(0x7f8e200ced60 msgr2=0x7f8e200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 -- 172.21.2.117:0/1681897027 shutdown_connections 2021-02-10T11:50:34.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.290 7f8e27f58700 1 -- 172.21.2.117:0/1681897027 wait complete. 2021-02-10T11:50:34.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.305 7f8e27f58700 1 Processor -- start 2021-02-10T11:50:34.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.305 7f8e27f58700 1 -- start start 2021-02-10T11:50:34.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200a3610 0x7f8e20098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2008b3c0 0x7f8e2008b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e20097460 con 0x7f8e200a3210 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2008bde0 con 0x7f8e2008b3c0 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e27f58700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2008bfc0 con 0x7f8e200a3610 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54876/0 (socket says 172.21.2.117:54876) 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 -- 172.21.2.117:0/969742567 learned_addr learned my addr 172.21.2.117:0/969742567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 -- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200a3610 msgr2=0x7f8e20098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200a3610 0x7f8e20098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 -- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2008b3c0 msgr2=0x7f8e2008b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2008b3c0 0x7f8e2008b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.306 7f8e25d74700 1 -- 172.21.2.117:0/969742567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2008c1e0 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e25d74700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7f8e1c00a5c0 tx=0x7f8e1c00a960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c00ad50 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c007030 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e1c004330 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e27f58700 1 -- 172.21.2.117:0/969742567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2008c1e0 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e27f58700 1 -- 172.21.2.117:0/969742567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2008c1e0 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e1c004140 con 0x7f8e200a3210 2021-02-10T11:50:34.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 0x7f8e0801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.307 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8e1c032910 con 0x7f8e200a3210 2021-02-10T11:50:34.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.308 7f8e25573700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 0x7f8e0801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.308 7f8e25573700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 0x7f8e0801ac10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.338 7f8e27f58700 1 -- 172.21.2.117:0/969742567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e200ede60 con 0x7f8e200a3210 2021-02-10T11:50:34.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.340 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e1c001060 con 0x7f8e200a3210 2021-02-10T11:50:34.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.466 7f8e27f58700 1 -- 172.21.2.117:0/969742567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e200ede60 con 0x7f8e200a3210 2021-02-10T11:50:34.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.467 7f8e127fc700 1 -- 172.21.2.117:0/969742567 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f8e1c004140 con 0x7f8e200a3210 2021-02-10T11:50:34.466 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:34.466 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":117,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:50:20.397701","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:50:17.365216","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"116","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":25,"pool_name":"cephfs_data","create_time":"2021-02-10 11:50:18.436070","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:50:34.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 msgr2=0x7f8e0801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 0x7f8e0801ac10 crc :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 msgr2=0x7f8e200979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7f8e1c00a5c0 tx=0x7f8e1c00a960).stop 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 shutdown_connections 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e2008b3c0 0x7f8e2008b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e200a3610 0x7f8e20098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e08018730 0x7f8e0801ac10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 --2- 172.21.2.117:0/969742567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e200a3210 0x7f8e200979e0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 >> 172.21.2.117:0/969742567 conn(0x7f8e20087440 msgr2=0x7f8e20098790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 shutdown_connections 2021-02-10T11:50:34.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.469 7f8e27f58700 1 -- 172.21.2.117:0/969742567 wait complete. 2021-02-10T11:50:34.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:50:17.365216', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '116', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:50:18.436070', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:50:34.476 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.892 7f03154e7700 1 Processor -- start 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.892 7f03154e7700 1 -- start start 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f03154e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 0x7f03100eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f03154e7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100ef7f0 con 0x7f03100d18f0 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f03154e7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03100d1cf0 con 0x7f03100eb830 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f03154e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03100eded0 con 0x7f03100ef3f0 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030f2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f03100d18f0 0x7f03100eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55134/0 (socket says 172.21.2.117:55134) 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030f2bf700 1 -- 172.21.2.117:0/3288294135 learned_addr learned my addr 172.21.2.117:0/3288294135 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:34.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030eabe700 1 --2- 172.21.2.117:0/3288294135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 0x7f03100eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400621592 0 0) 0x7f02f80009b0 con 0x7f03100d18f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02fc002fe0 con 0x7f03100d18f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363525370 0 0) 0x7f0308000f90 con 0x7f03100eb830 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.893 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02fc003210 con 0x7f03100eb830 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030eabe700 1 -- 172.21.2.117:0/3288294135 >> v1:172.21.2.125:6789/0 conn(0x7f03100eb830 legacy=0x7f03101b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030eabe700 1 -- 172.21.2.117:0/3288294135 >> v1:172.21.2.117:6789/0 conn(0x7f03100d18f0 legacy=0x7f03100eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 616880640 0 0) 0x7f02f8001010 con 0x7f03100d18f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030eabe700 1 -- 172.21.2.117:0/3288294135 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03101bb1c0 con 0x7f03100ef3f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030eabe700 1 --2- 172.21.2.117:0/3288294135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 0x7f03100eb210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0300006650 tx=0x7f0300009a00).ready entity=mon.2 client_cookie=d1e5410ffc93666a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03000009b0 con 0x7f03100ef3f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030000c0c0 con 0x7f03100ef3f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f030e2bd700 1 -- 172.21.2.117:0/3288294135 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f030000c5a0 con 0x7f03100ef3f0 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f03154e7700 1 -- 172.21.2.117:0/3288294135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 msgr2=0x7f03100eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.894 7f03154e7700 1 --2- 172.21.2.117:0/3288294135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 0x7f03100eb210 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0300006650 tx=0x7f0300009a00).stop 2021-02-10T11:50:34.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.895 7f03154e7700 1 -- 172.21.2.117:0/3288294135 shutdown_connections 2021-02-10T11:50:34.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.895 7f03154e7700 1 --2- 172.21.2.117:0/3288294135 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03100ef3f0 0x7f03100eb210 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:34.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.895 7f03154e7700 1 -- 172.21.2.117:0/3288294135 >> 172.21.2.117:0/3288294135 conn(0x7f03100cf060 msgr2=0x7f03100cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:34.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.895 7f03154e7700 1 -- 172.21.2.117:0/3288294135 shutdown_connections 2021-02-10T11:50:34.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.895 7f03154e7700 1 -- 172.21.2.117:0/3288294135 wait complete. 2021-02-10T11:50:34.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 Processor -- start 2021-02-10T11:50:34.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 -- start start 2021-02-10T11:50:34.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0310093be0 0x7f03101a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f031008c850 0x7f031008cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310094280 con 0x7f03101a7ed0 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310093fe0 con 0x7f0310093be0 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f03154e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03100ace00 con 0x7f031008c850 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f030eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f030eabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54892/0 (socket says 172.21.2.117:54892) 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.910 7f030eabe700 1 -- 172.21.2.117:0/2323356388 learned_addr learned my addr 172.21.2.117:0/2323356388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 -- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f031008c850 msgr2=0x7f031008cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f031008c850 0x7f031008cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 -- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0310093be0 msgr2=0x7f03101a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0310093be0 0x7f03101a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 -- 172.21.2.117:0/2323356388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310094800 con 0x7f03101a7ed0 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f030eabe700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f030000b640 tx=0x7f030000b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f030000bdd0 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300000c90 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03000022b0 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f03154e7700 1 -- 172.21.2.117:0/2323356388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0310094800 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.911 7f03154e7700 1 -- 172.21.2.117:0/2323356388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310094d90 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.912 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0300006a10 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.912 7f0307fff700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 0x7f02f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.912 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f03000325a0 con 0x7f03101a7ed0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.912 7f030f2bf700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 0x7f02f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:34.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.913 7f030f2bf700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 0x7f02f401abc0 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:34.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.943 7f03154e7700 1 -- 172.21.2.117:0/2323356388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03100ede60 con 0x7f03101a7ed0 2021-02-10T11:50:34.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:34.945 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0300000940 con 0x7f03101a7ed0 2021-02-10T11:50:35.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.071 7f03154e7700 1 -- 172.21.2.117:0/2323356388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f03100ede60 con 0x7f03101a7ed0 2021-02-10T11:50:35.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.072 7f0307fff700 1 -- 172.21.2.117:0/2323356388 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v117) v1 ==== 92+0+10 (secure 0 0 0) 0x7f030000bdd0 con 0x7f03101a7ed0 2021-02-10T11:50:35.070 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 -- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 msgr2=0x7f02f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 0x7f02f401abc0 crc :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 -- 172.21.2.117:0/2323356388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 msgr2=0x7f031008c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f030000b640 tx=0x7f030000b9e0).stop 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 -- 172.21.2.117:0/2323356388 shutdown_connections 2021-02-10T11:50:35.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f02f40186c0 0x7f02f401abc0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.073 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03101a7ed0 0x7f031008c230 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.074 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f031008c850 0x7f031008cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.074 7f03154e7700 1 --2- 172.21.2.117:0/2323356388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0310093be0 0x7f03101a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.074 7f03154e7700 1 -- 172.21.2.117:0/2323356388 >> 172.21.2.117:0/2323356388 conn(0x7f0310092d00 msgr2=0x7f031008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.074 7f03154e7700 1 -- 172.21.2.117:0/2323356388 shutdown_connections 2021-02-10T11:50:35.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.074 7f03154e7700 1 -- 172.21.2.117:0/2323356388 wait complete. 2021-02-10T11:50:35.081 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:50:35.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 Processor -- start 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 -- start start 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa3380d15e0 0x7fa3380e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380e7580 con 0x7fa3380e7180 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380e77f0 con 0x7fa3381b07f0 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33fca6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3380e5ce0 con 0x7fa3380d15e0 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.497 7fa33d2c1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa3380e7180 0x7fa3380e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55148/0 (socket says 172.21.2.117:55148) 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33d2c1700 1 -- 172.21.2.117:0/793245283 learned_addr learned my addr 172.21.2.117:0/793245283 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33e2c3700 1 --2- 172.21.2.117:0/793245283 >> v2:172.21.2.125:3300/0 conn(0x7fa3380d15e0 0x7fa3380e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33cac0700 1 -- 172.21.2.117:0/793245283 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254264355 0 0) 0x7fa3280010b0 con 0x7fa3380e7180 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33cac0700 1 -- 172.21.2.117:0/793245283 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324002fe0 con 0x7fa3380e7180 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33cac0700 1 -- 172.21.2.117:0/793245283 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50318686 0 0) 0x7fa334001080 con 0x7fa3381b07f0 2021-02-10T11:50:35.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33cac0700 1 -- 172.21.2.117:0/793245283 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003210 con 0x7fa3381b07f0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33e2c3700 1 -- 172.21.2.117:0/793245283 >> v1:172.21.2.125:6789/0 conn(0x7fa3381b07f0 legacy=0x7fa3380e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33e2c3700 1 -- 172.21.2.117:0/793245283 >> v1:172.21.2.117:6789/0 conn(0x7fa3380e7180 legacy=0x7fa3380e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.498 7fa33e2c3700 1 -- 172.21.2.117:0/793245283 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380d19e0 con 0x7fa3380d15e0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33e2c3700 1 --2- 172.21.2.117:0/793245283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3380d15e0 0x7fa3380e8560 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fa330006e50 tx=0x7fa330009e70).ready entity=mon.1 client_cookie=62b6db649b21860d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33cac0700 1 -- 172.21.2.117:0/793245283 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa330006a20 con 0x7fa3380d15e0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33cac0700 1 -- 172.21.2.117:0/793245283 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330006790 con 0x7fa3380d15e0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33cac0700 1 -- 172.21.2.117:0/793245283 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa33000b700 con 0x7fa3380d15e0 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33fca6700 1 -- 172.21.2.117:0/793245283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3380d15e0 msgr2=0x7fa3380e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.499 7fa33fca6700 1 --2- 172.21.2.117:0/793245283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3380d15e0 0x7fa3380e8560 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fa330006e50 tx=0x7fa330009e70).stop 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.500 7fa33fca6700 1 -- 172.21.2.117:0/793245283 shutdown_connections 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.500 7fa33fca6700 1 --2- 172.21.2.117:0/793245283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3380d15e0 0x7fa3380e8560 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.500 7fa33fca6700 1 -- 172.21.2.117:0/793245283 >> 172.21.2.117:0/793245283 conn(0x7fa3380ce500 msgr2=0x7fa3380e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.500 7fa33fca6700 1 -- 172.21.2.117:0/793245283 shutdown_connections 2021-02-10T11:50:35.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.500 7fa33fca6700 1 -- 172.21.2.117:0/793245283 wait complete. 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.514 7fa33fca6700 1 Processor -- start 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.514 7fa33fca6700 1 -- start start 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3380a4f80 0x7fa3380a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa338195f30 0x7fa3380b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380ba4a0 con 0x7fa338196330 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380bad20 con 0x7fa338195f30 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33fca6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380baf40 con 0x7fa3380a4f80 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33e2c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33e2c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54902/0 (socket says 172.21.2.117:54902) 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33e2c3700 1 -- 172.21.2.117:0/4000501566 learned_addr learned my addr 172.21.2.117:0/4000501566 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.515 7fa33d2c1700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa338195f30 0x7fa3380b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33dac2700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3380a4f80 0x7fa3380a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 -- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3380a4f80 msgr2=0x7fa3380a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3380a4f80 0x7fa3380a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 -- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa338195f30 msgr2=0x7fa3380b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa338195f30 0x7fa3380b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 -- 172.21.2.117:0/4000501566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380bb160 con 0x7fa338196330 2021-02-10T11:50:35.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33e2c3700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7fa330006e60 tx=0x7fa33000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa330000df0 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3300091c0 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3300013d0 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3380bb160 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.516 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3380bb160 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.517 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3300011a0 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.517 7fa32e7fc700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 0x7fa32001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.517 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa330032a10 con 0x7fa338196330 2021-02-10T11:50:35.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.517 7fa33dac2700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 0x7fa32001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:35.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.518 7fa33dac2700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 0x7fa32001abc0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:35.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.547 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33807e800 con 0x7fa338196330 2021-02-10T11:50:35.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.550 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa33000be30 con 0x7fa338196330 2021-02-10T11:50:35.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.676 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa3380e77f0 con 0x7fa338196330 2021-02-10T11:50:35.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.676 7fa32e7fc700 1 -- 172.21.2.117:0/4000501566 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v117) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa33001f880 con 0x7fa338196330 2021-02-10T11:50:35.675 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 msgr2=0x7fa32001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 0x7fa32001abc0 crc :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 msgr2=0x7fa3380b9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7fa330006e60 tx=0x7fa33000bcf0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 shutdown_connections 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3200186c0 0x7fa32001abc0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa338196330 0x7fa3380b9e80 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa338195f30 0x7fa3380b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 --2- 172.21.2.117:0/4000501566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3380a4f80 0x7fa3380a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:35.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 >> 172.21.2.117:0/4000501566 conn(0x7fa3380a7210 msgr2=0x7fa338095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:35.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.678 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 shutdown_connections 2021-02-10T11:50:35.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:35.679 7fa33fca6700 1 -- 172.21.2.117:0/4000501566 wait complete. 2021-02-10T11:50:35.686 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:50:36.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.102 7f8aeca8a700 1 Processor -- start 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 -- start start 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ae80d18f0 0x7f8ae80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8ae80eb040 0x7f8ae80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae80ee3e0 0x7f8ae80ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80edee0 con 0x7f8ae80d18f0 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80ebaa0 con 0x7f8ae80eb040 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8aeca8a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80d1cf0 con 0x7f8ae80ee3e0 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8ae685e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ae80d18f0 0x7f8ae80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8ae685e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ae80d18f0 0x7f8ae80eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54910/0 (socket says 172.21.2.117:54910) 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.103 7f8ae685e700 1 -- 172.21.2.117:0/3867758687 learned_addr learned my addr 172.21.2.117:0/3867758687 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae605d700 1 --2- 172.21.2.117:0/3867758687 >> v2:172.21.2.125:3300/0 conn(0x7f8ae80eb040 0x7f8ae80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae705f700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae80ee3e0 0x7f8ae80ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 -- 172.21.2.117:0/3867758687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae80ee3e0 msgr2=0x7f8ae80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae80ee3e0 0x7f8ae80ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 -- 172.21.2.117:0/3867758687 >> v2:172.21.2.125:3300/0 conn(0x7f8ae80eb040 msgr2=0x7f8ae80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 --2- 172.21.2.117:0/3867758687 >> v2:172.21.2.125:3300/0 conn(0x7f8ae80eb040 0x7f8ae80eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 -- 172.21.2.117:0/3867758687 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae80eee00 con 0x7f8ae80d18f0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae685e700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae80d18f0 0x7f8ae80eaa20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f8ad8006700 tx=0x7f8ad8009b80).ready entity=mon.0 client_cookie=d873ad5e432ae755 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae585c700 1 -- 172.21.2.117:0/3867758687 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ad800c0c0 con 0x7f8ae80d18f0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae585c700 1 -- 172.21.2.117:0/3867758687 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad800c340 con 0x7f8ae80d18f0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8ae585c700 1 -- 172.21.2.117:0/3867758687 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ad800c860 con 0x7f8ae80d18f0 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8aeca8a700 1 -- 172.21.2.117:0/3867758687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae80d18f0 msgr2=0x7f8ae80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.104 7f8aeca8a700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae80d18f0 0x7f8ae80eaa20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f8ad8006700 tx=0x7f8ad8009b80).stop 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 -- 172.21.2.117:0/3867758687 shutdown_connections 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 --2- 172.21.2.117:0/3867758687 >> v2:172.21.2.125:3300/0 conn(0x7f8ae80eb040 0x7f8ae80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae80ee3e0 0x7f8ae80ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 --2- 172.21.2.117:0/3867758687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae80d18f0 0x7f8ae80eaa20 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 -- 172.21.2.117:0/3867758687 >> 172.21.2.117:0/3867758687 conn(0x7f8ae80cf060 msgr2=0x7f8ae80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 -- 172.21.2.117:0/3867758687 shutdown_connections 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.105 7f8aeca8a700 1 -- 172.21.2.117:0/3867758687 wait complete. 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 Processor -- start 2021-02-10T11:50:36.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 -- start start 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 0x7f8ae80863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 0x7f8ae80dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 0x7f8ae80dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8085e50 con 0x7f8ae81a4180 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8097270 con 0x7f8ae81a4580 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.120 7f8aeca8a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae8097450 con 0x7f8ae8083990 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 0x7f8ae80dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae605d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 0x7f8ae80dbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54918/0 (socket says 172.21.2.117:54918) 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae605d700 1 -- 172.21.2.117:0/1709418044 learned_addr learned my addr 172.21.2.117:0/1709418044 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 0x7f8ae80dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae685e700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 0x7f8ae80863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 -- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 msgr2=0x7f8ae80863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 0x7f8ae80863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 -- 172.21.2.117:0/1709418044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 msgr2=0x7f8ae80dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 0x7f8ae80dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 -- 172.21.2.117:0/1709418044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae809ed70 con 0x7f8ae81a4580 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae685e700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 0x7f8ae80863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.121 7f8ae705f700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 0x7f8ae80dc460 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f8ae000b8a0 tx=0x7f8ae000bc40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae0000f90 con 0x7f8ae81a4580 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae000bd80 con 0x7f8ae81a4580 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae809ed70 con 0x7f8ae81a4580 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae809ed70 con 0x7f8ae81a4580 2021-02-10T11:50:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae0001470 con 0x7f8ae81a4580 2021-02-10T11:50:36.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ae00056b0 con 0x7f8ae81a4580 2021-02-10T11:50:36.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 0x7f8acc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.122 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ae0032bd0 con 0x7f8ae81a4580 2021-02-10T11:50:36.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.123 7f8ae685e700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 0x7f8acc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.123 7f8ae685e700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 0x7f8acc01ac10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.153 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae80daff0 con 0x7f8ae81a4580 2021-02-10T11:50:36.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.156 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ae00066f0 con 0x7f8ae81a4580 2021-02-10T11:50:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.286 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8ae809ed70 con 0x7f8ae81a4580 2021-02-10T11:50:36.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.288 7f8adf7fe700 1 -- 172.21.2.117:0/1709418044 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v117) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8ae00056b0 con 0x7f8ae81a4580 2021-02-10T11:50:36.288 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 msgr2=0x7f8acc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 0x7f8acc01ac10 crc :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 msgr2=0x7f8ae80dc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 0x7f8ae80dc460 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f8ae000b8a0 tx=0x7f8ae000bc40).stop 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 shutdown_connections 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8acc018730 0x7f8acc01ac10 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ae81a4180 0x7f8ae80dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ae81a4580 0x7f8ae80dc460 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 --2- 172.21.2.117:0/1709418044 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ae8083990 0x7f8ae80863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 >> 172.21.2.117:0/1709418044 conn(0x7f8ae80b6540 msgr2=0x7f8ae808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 shutdown_connections 2021-02-10T11:50:36.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.291 7f8aeca8a700 1 -- 172.21.2.117:0/1709418044 wait complete. 2021-02-10T11:50:36.298 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:50:36.298 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:50:36.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.712 7f88377da700 1 Processor -- start 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.712 7f88377da700 1 -- start start 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f88377da700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f88300ef3f0 0x7f88300eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f88377da700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88300ede60 con 0x7f88300eb7f0 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f88377da700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88300d1ce0 con 0x7f88300d18e0 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f88377da700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88300ef7f0 con 0x7f88300ef3f0 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8835df7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f88300eb7f0 0x7f88301b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55178/0 (socket says 172.21.2.117:55178) 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8835df7700 1 -- 172.21.2.117:0/2062231367 learned_addr learned my addr 172.21.2.117:0/2062231367 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8834df5700 1 --2- 172.21.2.117:0/2062231367 >> v2:172.21.2.117:3300/0 conn(0x7f88300ef3f0 0x7f88300eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8827fff700 1 -- 172.21.2.117:0/2062231367 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829409154 0 0) 0x7f8828001500 con 0x7f88300eb7f0 2021-02-10T11:50:36.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8827fff700 1 -- 172.21.2.117:0/2062231367 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f881c002fe0 con 0x7f88300eb7f0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8834df5700 1 -- 172.21.2.117:0/2062231367 >> v1:172.21.2.125:6789/0 conn(0x7f88300d18e0 legacy=0x7f88300ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8834df5700 1 -- 172.21.2.117:0/2062231367 >> v1:172.21.2.117:6789/0 conn(0x7f88300eb7f0 legacy=0x7f88301b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8834df5700 1 -- 172.21.2.117:0/2062231367 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88301bb180 con 0x7f88300ef3f0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.713 7f8834df5700 1 --2- 172.21.2.117:0/2062231367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88300ef3f0 0x7f88300eb1d0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f8820006650 tx=0x7f8820009a80).ready entity=mon.0 client_cookie=5fbc1b9eee83c1c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.714 7f8827fff700 1 -- 172.21.2.117:0/2062231367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f882000c0c0 con 0x7f88300ef3f0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.714 7f8827fff700 1 -- 172.21.2.117:0/2062231367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882000c340 con 0x7f88300ef3f0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.714 7f8827fff700 1 -- 172.21.2.117:0/2062231367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f882000c860 con 0x7f88300ef3f0 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.714 7f88377da700 1 -- 172.21.2.117:0/2062231367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88300ef3f0 msgr2=0x7f88300eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.714 7f88377da700 1 --2- 172.21.2.117:0/2062231367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88300ef3f0 0x7f88300eb1d0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f8820006650 tx=0x7f8820009a80).stop 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.715 7f88377da700 1 -- 172.21.2.117:0/2062231367 shutdown_connections 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.715 7f88377da700 1 --2- 172.21.2.117:0/2062231367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f88300ef3f0 0x7f88300eb1d0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.715 7f88377da700 1 -- 172.21.2.117:0/2062231367 >> 172.21.2.117:0/2062231367 conn(0x7f88300ced60 msgr2=0x7f88300cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.715 7f88377da700 1 -- 172.21.2.117:0/2062231367 shutdown_connections 2021-02-10T11:50:36.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.715 7f88377da700 1 -- 172.21.2.117:0/2062231367 wait complete. 2021-02-10T11:50:36.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.729 7f88377da700 1 Processor -- start 2021-02-10T11:50:36.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.729 7f88377da700 1 -- start start 2021-02-10T11:50:36.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 0x7f883019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 0x7f883019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88300b1f30 con 0x7f883019dd40 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883019d830 con 0x7f88300984c0 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88377da700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88301a55e0 con 0x7f88300980c0 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f8835df7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f8835df7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54934/0 (socket says 172.21.2.117:54934) 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f8835df7700 1 -- 172.21.2.117:0/2509676059 learned_addr learned my addr 172.21.2.117:0/2509676059 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f8834df5700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 0x7f883019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f88355f6700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 0x7f883019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.730 7f8835df7700 1 -- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 msgr2=0x7f883019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8835df7700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 0x7f883019ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8835df7700 1 -- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 msgr2=0x7f883019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8835df7700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 0x7f883019d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8835df7700 1 -- 172.21.2.117:0/2509676059 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883019e760 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f88355f6700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 0x7f883019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8834df5700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 0x7f883019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8835df7700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f8828001c10 tx=0x7f882800aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8828007020 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88280072a0 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8828004720 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f88377da700 1 -- 172.21.2.117:0/2509676059 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883019e760 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.731 7f88377da700 1 -- 172.21.2.117:0/2509676059 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883019ecf0 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.732 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8828007480 con 0x7f883019dd40 2021-02-10T11:50:36.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.732 7f8825ffb700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:36.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.732 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8828032a20 con 0x7f883019dd40 2021-02-10T11:50:36.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.732 7f88355f6700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:36.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.733 7f88355f6700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:36.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.762 7f88377da700 1 -- 172.21.2.117:0/2509676059 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8830087210 con 0x7f883019dd40 2021-02-10T11:50:36.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.765 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8828000f40 con 0x7f883019dd40 2021-02-10T11:50:36.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.892 7f88377da700 1 -- 172.21.2.117:0/2509676059 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88300ede60 con 0x7f883019dd40 2021-02-10T11:50:36.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.892 7f8825ffb700 1 -- 172.21.2.117:0/2509676059 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+2101 (secure 0 0 0) 0x7f8828007480 con 0x7f883019dd40 2021-02-10T11:50:36.892 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 94 2021-02-10T11:50:36.892 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:50:36.892 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":94,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:50:21.403231","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7010":{"gid":7010,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/143190062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":143190062},{"type":"v1","addr":"172.21.2.125:6835","nonce":143190062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:50:36.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.894 7f88377da700 1 -- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 msgr2=0x7f881001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.894 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 crc :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.894 7f88377da700 1 -- 172.21.2.117:0/2509676059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 msgr2=0x7f883019e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:36.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.894 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f8828001c10 tx=0x7f882800aca0).stop 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 -- 172.21.2.117:0/2509676059 shutdown_connections 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f88100186c0 0x7f881001abc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f883019dd40 0x7f883019e140 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f88300984c0 0x7f883019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 --2- 172.21.2.117:0/2509676059 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f88300980c0 0x7f883019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 -- 172.21.2.117:0/2509676059 >> 172.21.2.117:0/2509676059 conn(0x7f883019f610 msgr2=0x7f88301a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 -- 172.21.2.117:0/2509676059 shutdown_connections 2021-02-10T11:50:36.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:36.895 7f88377da700 1 -- 172.21.2.117:0/2509676059 wait complete. 2021-02-10T11:50:37.105 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:50:19.395674', 'modified': '2021-02-10 11:50:21.403231', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7010': {'gid': 7010, 'name': 'b', 'rank': 0, 'incarnation': 91, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/143190062', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 143190062}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 143190062}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:50:37.105 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:50:37.105 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7010, 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:50:37.505 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.109958293999998, 2021-02-10T11:50:37.506 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 94, 2021-02-10T11:50:37.506 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 117, 2021-02-10T11:50:37.506 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 117, 2021-02-10T11:50:37.506 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.741834568000002 2021-02-10T11:50:37.506 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:50:37.509 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7010, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.109958293999998, "mdsmap_epoch": 94, "osdmap_epoch": 117, "osdmap_epoch_barrier": 117, "uptime": 23.741834568000002 } 2021-02-10T11:50:37.509 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:50:37.509 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:50:37.509 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:50:37.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:37.510 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:37.846 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:50:38.190 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:50:38.191 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:50:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:50:38.535 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:50:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:50:38.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:38.537 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:50:38.872 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:50:38.873 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:50:38.873 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:38.876 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:50:39.220 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:50:39.220 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:50:39.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:50:39.289 7f0af54f2f80 -1 init, newargv = 0x55e3252dc770 newargc=9 2021-02-10T11:50:39.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[162483]: starting ceph client 2021-02-10T11:50:39.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[162483]: starting fuse 2021-02-10T11:50:39.563 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:50:39.563 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:50:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:50:39.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:39.564 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:50:39.899 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:50:39.899 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:50:39.899 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:50:39.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:39.900 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:50:39.901 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:50:40.262 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.162483.asok 2021-02-10T11:50:40.265 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.162483.asok 2021-02-10T11:50:40.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:40.265 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.162483.asok status 2021-02-10T11:50:40.654 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "162483", 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:40.655 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 10974, 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 10974 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:50:40.656 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2516517867 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2516517867 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.10974 172.21.2.117:0/2516517867", 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2516517867", 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:50:40.657 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 94, 2021-02-10T11:50:40.658 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 117, 2021-02-10T11:50:40.658 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:50:40.658 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:50:40.658 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:50:40.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:40.659 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:40.659 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:40.995 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:50:40.995 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:40.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:40.996 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:41.342 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:50:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.760 7f7d56e03700 1 Processor -- start 2021-02-10T11:50:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 -- start start 2021-02-10T11:50:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d500e5240 0x7f7d50086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7d50084c60 0x7f7d50085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d50085930 con 0x7f7d500e4e40 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500e5cc0 con 0x7f7d500e5240 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d56e03700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500856b0 con 0x7f7d50084c60 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d500e5240 0x7f7d50086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7d500e5240 0x7f7d50086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54950/0 (socket says 172.21.2.117:54950) 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 -- 172.21.2.117:0/1096432412 learned_addr learned my addr 172.21.2.117:0/1096432412 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 -- 172.21.2.117:0/1096432412 >> v2:172.21.2.125:3300/0 conn(0x7f7d50084c60 msgr2=0x7f7d50085060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 --2- 172.21.2.117:0/1096432412 >> v2:172.21.2.125:3300/0 conn(0x7f7d50084c60 0x7f7d50085060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 -- 172.21.2.117:0/1096432412 >> v1:172.21.2.125:6789/0 conn(0x7f7d500e4e40 legacy=0x7f7d501b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.761 7f7d4ffff700 1 -- 172.21.2.117:0/1096432412 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d501b09e0 con 0x7f7d500e5240 2021-02-10T11:50:41.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d4ffff700 1 --2- 172.21.2.117:0/1096432412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d500e5240 0x7f7d50086ec0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7f7d40006700 tx=0x7f7d40009b00).ready entity=mon.0 client_cookie=b1ef3e91dc2f8fe9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d4f7fe700 1 -- 172.21.2.117:0/1096432412 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4000c0c0 con 0x7f7d500e5240 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d4f7fe700 1 -- 172.21.2.117:0/1096432412 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4000c340 con 0x7f7d500e5240 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d4f7fe700 1 -- 172.21.2.117:0/1096432412 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d4000c820 con 0x7f7d500e5240 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 -- 172.21.2.117:0/1096432412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d500e5240 msgr2=0x7f7d50086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 --2- 172.21.2.117:0/1096432412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d500e5240 0x7f7d50086ec0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7f7d40006700 tx=0x7f7d40009b00).stop 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 -- 172.21.2.117:0/1096432412 shutdown_connections 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 --2- 172.21.2.117:0/1096432412 >> v2:172.21.2.125:3300/0 conn(0x7f7d50084c60 0x7f7d50085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 --2- 172.21.2.117:0/1096432412 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d500e5240 0x7f7d50086ec0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 -- 172.21.2.117:0/1096432412 >> 172.21.2.117:0/1096432412 conn(0x7f7d500ce4e0 msgr2=0x7f7d500e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.762 7f7d56e03700 1 -- 172.21.2.117:0/1096432412 shutdown_connections 2021-02-10T11:50:41.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.763 7f7d56e03700 1 -- 172.21.2.117:0/1096432412 wait complete. 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 Processor -- start 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 -- start start 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d5018e5b0 0x7f7d5018e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d500b30d0 0x7f7d500b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501ae8d0 con 0x7f7d50099bf0 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3af0 con 0x7f7d500b30d0 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d56e03700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b3d10 con 0x7f7d5018e5b0 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d54c1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:41.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d54c1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54958/0 (socket says 172.21.2.117:54958) 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.777 7f7d54c1f700 1 -- 172.21.2.117:0/763422282 learned_addr learned my addr 172.21.2.117:0/763422282 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 -- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d5018e5b0 msgr2=0x7f7d5018e9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d5018e5b0 0x7f7d5018e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 -- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d500b30d0 msgr2=0x7f7d500b34d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d500b30d0 0x7f7d500b34d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 -- 172.21.2.117:0/763422282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d500b3f30 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d54c1f700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7f7d3800b5b0 tx=0x7f7d3800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d3800bd40 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38006240 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d56e03700 1 -- 172.21.2.117:0/763422282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d500b3f30 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7d38005720 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.778 7f7d56e03700 1 -- 172.21.2.117:0/763422282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d501a19b0 con 0x7f7d50099bf0 2021-02-10T11:50:41.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.779 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7d380054a0 con 0x7f7d50099bf0 2021-02-10T11:50:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.779 7f7d4d7fa700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 0x7f7d3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.779 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7d38032940 con 0x7f7d50099bf0 2021-02-10T11:50:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.779 7f7d4ffff700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 0x7f7d3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:41.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.780 7f7d4ffff700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 0x7f7d3401abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.811 7f7d56e03700 1 -- 172.21.2.117:0/763422282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5018e080 con 0x7f7d50099bf0 2021-02-10T11:50:41.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.813 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7d38000c60 con 0x7f7d50099bf0 2021-02-10T11:50:41.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:41.949 7f7d56e03700 1 -- 172.21.2.117:0/763422282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7d5018e080 con 0x7f7d50099bf0 2021-02-10T11:50:42.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.678 7f7d4d7fa700 1 -- 172.21.2.117:0/763422282 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v95) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7d380054a0 con 0x7f7d50099bf0 2021-02-10T11:50:42.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 msgr2=0x7f7d3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 0x7f7d3401abc0 crc :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 msgr2=0x7f7d501ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7f7d3800b5b0 tx=0x7f7d3800b950).stop 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 shutdown_connections 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7d500b30d0 0x7f7d500b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7d5018e5b0 0x7f7d5018e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7d34018730 0x7f7d3401abc0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:42.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 --2- 172.21.2.117:0/763422282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7d50099bf0 0x7f7d501ae2b0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:42.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 >> 172.21.2.117:0/763422282 conn(0x7f7d501ad6b0 msgr2=0x7f7d50190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:42.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 shutdown_connections 2021-02-10T11:50:42.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:42.722 7f7d56e03700 1 -- 172.21.2.117:0/763422282 wait complete. 2021-02-10T11:50:42.728 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:50:43.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.142 7fd71623b700 1 Processor -- start 2021-02-10T11:50:43.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.142 7fd71623b700 1 -- start start 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd71623b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd7101b07f0 0x7fd7100e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd71623b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd7100e6170 0x7fd7100e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd71623b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7100e77f0 con 0x7fd7100e6b90 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd71623b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7100e5ce0 con 0x7fd7101b07f0 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd71623b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7100e54b0 con 0x7fd7100e6170 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd7101b07f0 0x7fd7100e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd7101b07f0 0x7fd7100e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54964/0 (socket says 172.21.2.117:54964) 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd714858700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd7100e6b90 0x7fd7100e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55218/0 (socket says 172.21.2.117:55218) 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 -- 172.21.2.117:0/3492982544 learned_addr learned my addr 172.21.2.117:0/3492982544 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 -- 172.21.2.117:0/3492982544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd7100e6170 msgr2=0x7fd7100e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 --2- 172.21.2.117:0/3492982544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd7100e6170 0x7fd7100e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 -- 172.21.2.117:0/3492982544 >> v1:172.21.2.117:6789/0 conn(0x7fd7100e6b90 legacy=0x7fd7100e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 -- 172.21.2.117:0/3492982544 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100e8d80 con 0x7fd7101b07f0 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.143 7fd70ffff700 1 --2- 172.21.2.117:0/3492982544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7101b07f0 0x7fd7100e4e90 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fd700006700 tx=0x7fd700009b00).ready entity=mon.0 client_cookie=451e7a7d937eee46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd70effd700 1 -- 172.21.2.117:0/3492982544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd70000c0c0 con 0x7fd7101b07f0 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd70effd700 1 -- 172.21.2.117:0/3492982544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70000c340 con 0x7fd7101b07f0 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd70effd700 1 -- 172.21.2.117:0/3492982544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd70000c7a0 con 0x7fd7101b07f0 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 -- 172.21.2.117:0/3492982544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7101b07f0 msgr2=0x7fd7100e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 --2- 172.21.2.117:0/3492982544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7101b07f0 0x7fd7100e4e90 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fd700006700 tx=0x7fd700009b00).stop 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 -- 172.21.2.117:0/3492982544 shutdown_connections 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 --2- 172.21.2.117:0/3492982544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd7100e6170 0x7fd7100e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 --2- 172.21.2.117:0/3492982544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7101b07f0 0x7fd7100e4e90 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 -- 172.21.2.117:0/3492982544 >> 172.21.2.117:0/3492982544 conn(0x7fd7100ce500 msgr2=0x7fd7100e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 -- 172.21.2.117:0/3492982544 shutdown_connections 2021-02-10T11:50:43.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.144 7fd71623b700 1 -- 172.21.2.117:0/3492982544 wait complete. 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 Processor -- start 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 -- start start 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd710109830 0x7fd7100a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd7100a3600 0x7fd7100a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd710109c30 con 0x7fd7100a4950 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71010c400 con 0x7fd7100a3600 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.159 7fd71623b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd710109ee0 con 0x7fd710109830 2021-02-10T11:50:43.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54974/0 (socket says 172.21.2.117:54974) 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 -- 172.21.2.117:0/3250179914 learned_addr learned my addr 172.21.2.117:0/3250179914 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 -- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd710109830 msgr2=0x7fd7100a2fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd710109830 0x7fd7100a2fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 -- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd7100a3600 msgr2=0x7fd7100a4330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd7100a3600 0x7fd7100a4330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 -- 172.21.2.117:0/3250179914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100a3a00 con 0x7fd7100a4950 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd714858700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fd708007f90 tx=0x7fd70800ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd708001270 con 0x7fd7100a4950 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd708004930 con 0x7fd7100a4950 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd708007020 con 0x7fd7100a4950 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd71623b700 1 -- 172.21.2.117:0/3250179914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7100a3a00 con 0x7fd7100a4950 2021-02-10T11:50:43.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.160 7fd71623b700 1 -- 172.21.2.117:0/3250179914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7100a5660 con 0x7fd7100a4950 2021-02-10T11:50:43.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.161 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd708007020 con 0x7fd7100a4950 2021-02-10T11:50:43.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.161 7fd70cff9700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 0x7fd6f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.161 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd708032a30 con 0x7fd7100a4950 2021-02-10T11:50:43.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.162 7fd70ffff700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 0x7fd6f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.162 7fd70ffff700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 0x7fd6f001abc0 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.192 7fd71623b700 1 -- 172.21.2.117:0/3250179914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd71010c400 con 0x7fd7100a4950 2021-02-10T11:50:43.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.195 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd708007d00 con 0x7fd7100a4950 2021-02-10T11:50:43.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.328 7fd71623b700 1 -- 172.21.2.117:0/3250179914 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fd710109c30 con 0x7fd6f00186c0 2021-02-10T11:50:43.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.329 7fd70cff9700 1 -- 172.21.2.117:0/3250179914 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fd700009100 con 0x7fd6f00186c0 2021-02-10T11:50:43.329 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:50:43.330 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:50:43.330 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:50:43.330 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:50:43.330 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:50:43.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 -- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 msgr2=0x7fd6f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 0x7fd6f001abc0 crc :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 -- 172.21.2.117:0/3250179914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 msgr2=0x7fd7100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7fd708007f90 tx=0x7fd70800ac50).stop 2021-02-10T11:50:43.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 -- 172.21.2.117:0/3250179914 shutdown_connections 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd6f00186c0 0x7fd6f001abc0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd7100a4950 0x7fd7100a4d50 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd7100a3600 0x7fd7100a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 --2- 172.21.2.117:0/3250179914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd710109830 0x7fd7100a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.333 7fd71623b700 1 -- 172.21.2.117:0/3250179914 >> 172.21.2.117:0/3250179914 conn(0x7fd710111150 msgr2=0x7fd710187d10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.334 7fd71623b700 1 -- 172.21.2.117:0/3250179914 shutdown_connections 2021-02-10T11:50:43.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.334 7fd71623b700 1 -- 172.21.2.117:0/3250179914 wait complete. 2021-02-10T11:50:43.339 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 Processor -- start 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 -- start start 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f60c41b07f0 0x7f60c40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40e77f0 con 0x7f60c40e6170 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40e7550 con 0x7f60c40e8d80 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.758 7f60cb779700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c40e54b0 con 0x7f60c41b07f0 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60c8d94700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f60c40e6170 0x7f60c40e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55228/0 (socket says 172.21.2.117:55228) 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60c8d94700 1 -- 172.21.2.117:0/248294554 learned_addr learned my addr 172.21.2.117:0/248294554 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024920864 0 0) 0x7f60c00010b0 con 0x7f60c40e6170 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60c9595700 1 --2- 172.21.2.117:0/248294554 >> v2:172.21.2.125:3300/0 conn(0x7f60c41b07f0 0x7f60c40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b0002fe0 con 0x7f60c40e6170 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2502639380 0 0) 0x7f60c00010b0 con 0x7f60c40e6170 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 >> v2:172.21.2.125:3300/0 conn(0x7f60c41b07f0 msgr2=0x7f60c40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 --2- 172.21.2.117:0/248294554 >> v2:172.21.2.125:3300/0 conn(0x7f60c41b07f0 0x7f60c40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 >> v1:172.21.2.125:6789/0 conn(0x7f60c40e8d80 legacy=0x7f60c40e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40e9180 con 0x7f60c40e6170 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f60c00015b0 con 0x7f60c40e6170 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 -- 172.21.2.117:0/248294554 >> v1:172.21.2.117:6789/0 conn(0x7f60c40e6170 legacy=0x7f60c40e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 0x7f60b0005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60b0006790 0x7f60b0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.759 7f60b7fff700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60b00095b0 0x7f60b000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b000c300 con 0x7f60b0003a70 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b000c520 con 0x7f60b00095b0 2021-02-10T11:50:43.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0050750 con 0x7f60b0006790 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 0x7f60b0005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9d96700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60b00095b0 0x7f60b000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c8d94700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60b0006790 0x7f60b0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 -- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60b0006790 msgr2=0x7f60b0008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60b0006790 0x7f60b0008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 -- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60b00095b0 msgr2=0x7f60b000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60b00095b0 0x7f60b000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9d96700 1 -- 172.21.2.117:0/248294554 reap_dead start 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 -- 172.21.2.117:0/248294554 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b00509b0 con 0x7f60b0003a70 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c8d94700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60b0006790 0x7f60b0008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60c9595700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 0x7f60b0005f70 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f60b800c470 tx=0x7f60b8006c20).ready entity=mon.0 client_cookie=9f748912b9d6c799 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b80090d0 con 0x7f60b0003a70 2021-02-10T11:50:43.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b8009410 con 0x7f60b0003a70 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60b7fff700 1 -- 172.21.2.117:0/248294554 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b80098f0 con 0x7f60b0003a70 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60cb779700 1 -- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 msgr2=0x7f60b0005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.760 7f60cb779700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 0x7f60b0005f70 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f60b800c470 tx=0x7f60b8006c20).stop 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.761 7f60cb779700 1 -- 172.21.2.117:0/248294554 shutdown_connections 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.761 7f60cb779700 1 --2- 172.21.2.117:0/248294554 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60b0003a70 0x7f60b0005f70 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.761 7f60cb779700 1 -- 172.21.2.117:0/248294554 >> 172.21.2.117:0/248294554 conn(0x7f60c40ce500 msgr2=0x7f60c40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.761 7f60cb779700 1 -- 172.21.2.117:0/248294554 shutdown_connections 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.761 7f60cb779700 1 -- 172.21.2.117:0/248294554 wait complete. 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 Processor -- start 2021-02-10T11:50:43.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 -- start start 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60c4110db0 0x7f60c41111b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60c40c0950 0x7f60c40c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40c4040 con 0x7f60c41a2df0 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40c1370 con 0x7f60c40c0950 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60cb779700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c40c1590 con 0x7f60c4110db0 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60c9595700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60c9595700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:54990/0 (socket says 172.21.2.117:54990) 2021-02-10T11:50:43.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.776 7f60c9595700 1 -- 172.21.2.117:0/4216007406 learned_addr learned my addr 172.21.2.117:0/4216007406 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9d96700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60c40c0950 0x7f60c40c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c8d94700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60c4110db0 0x7f60c41111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 -- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60c4110db0 msgr2=0x7f60c41111b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60c4110db0 0x7f60c41111b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 -- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60c40c0950 msgr2=0x7f60c40c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60c40c0950 0x7f60c40c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 -- 172.21.2.117:0/4216007406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40c17b0 con 0x7f60c41a2df0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60c9595700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7f60b800c160 tx=0x7f60b800b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b800c7c0 con 0x7f60c41a2df0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b800ca40 con 0x7f60c41a2df0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60cb779700 1 -- 172.21.2.117:0/4216007406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c40c17b0 con 0x7f60c41a2df0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60cb779700 1 -- 172.21.2.117:0/4216007406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c40bf0e0 con 0x7f60c41a2df0 2021-02-10T11:50:43.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.777 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60b800b6f0 con 0x7f60c41a2df0 2021-02-10T11:50:43.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.778 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60b8000c90 con 0x7f60c41a2df0 2021-02-10T11:50:43.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.778 7f60b5ffb700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 0x7f60a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:43.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.778 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f60b8032980 con 0x7f60c41a2df0 2021-02-10T11:50:43.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.778 7f60c8d94700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 0x7f60a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:43.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.779 7f60c8d94700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 0x7f60a401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.808 7f60cb779700 1 -- 172.21.2.117:0/4216007406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c410c190 con 0x7f60c41a2df0 2021-02-10T11:50:43.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.811 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f60b800c4d0 con 0x7f60c41a2df0 2021-02-10T11:50:43.869 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:43.872 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:50:43.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.960 7f60cb779700 1 -- 172.21.2.117:0/4216007406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f60c410c190 con 0x7f60c41a2df0 2021-02-10T11:50:43.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.961 7f60b5ffb700 1 -- 172.21.2.117:0/4216007406 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60b8000c90 con 0x7f60c41a2df0 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 msgr2=0x7f60a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 0x7f60a401abc0 crc :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 msgr2=0x7f60c41a31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7f60b800c160 tx=0x7f60b800b300).stop 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 shutdown_connections 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60c40c0950 0x7f60c40c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60c4110db0 0x7f60c41111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60a4018730 0x7f60a401abc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 --2- 172.21.2.117:0/4216007406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60c41a2df0 0x7f60c41a31f0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:43.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 >> 172.21.2.117:0/4216007406 conn(0x7f60c40be140 msgr2=0x7f60c40b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:43.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 shutdown_connections 2021-02-10T11:50:43.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:43.963 7f60cb779700 1 -- 172.21.2.117:0/4216007406 wait complete. 2021-02-10T11:50:43.969 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000519176 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 Processor -- start 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 -- start start 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac0ef3f0 0x7fb7ac0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0ef7f0 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac0d1cf0 con 0x7fb7ac0eb830 2021-02-10T11:50:44.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7b1fe6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac0eded0 con 0x7fb7ac0ef3f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7ab7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb7ac0d18f0 0x7fb7ac0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55248/0 (socket says 172.21.2.117:55248) 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.288 7fb7ab7fe700 1 -- 172.21.2.117:0/1092213203 learned_addr learned my addr 172.21.2.117:0/1092213203 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182591295 0 0) 0x7fb794001180 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aaffd700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac0ef3f0 0x7fb7ac0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb798002fe0 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 102401911 0 0) 0x7fb794001180 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac0ef3f0 msgr2=0x7fb7ac0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac0ef3f0 0x7fb7ac0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 >> v1:172.21.2.125:6789/0 conn(0x7fb7ac0eb830 legacy=0x7fb7ac1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac1bb1c0 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fb794001680 con 0x7fb7ac0d18f0 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 >> v1:172.21.2.117:6789/0 conn(0x7fb7ac0d18f0 legacy=0x7fb7ac0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb798003aa0 0x7fb798005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb798006750 0x7fb798008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb798009510 0x7fb79800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79800c2c0 con 0x7fb798009510 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79800c4e0 con 0x7fb798006750 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79800c700 con 0x7fb798003aa0 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7abfff700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb798009510 0x7fb79800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7ab7fe700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb798003aa0 0x7fb798005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.289 7fb7aaffd700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb798006750 0x7fb798008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 -- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb798003aa0 msgr2=0x7fb798005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb798003aa0 0x7fb798005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 -- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb798006750 msgr2=0x7fb798008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb798006750 0x7fb798008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 -- 172.21.2.117:0/1092213203 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb798050940 con 0x7fb798009510 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7ab7fe700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb798003aa0 0x7fb798005f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb798009510 0x7fb79800ba10 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7fb7a4006e30 tx=0x7fb7a4009e60).ready entity=mon.0 client_cookie=8f8f94b7f89fd2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a400b300 con 0x7fb798009510 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a400b5c0 con 0x7fb798009510 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7aa7fc700 1 -- 172.21.2.117:0/1092213203 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7a400b300 con 0x7fb798009510 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 -- 172.21.2.117:0/1092213203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb798009510 msgr2=0x7fb79800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 --2- 172.21.2.117:0/1092213203 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb798009510 0x7fb79800ba10 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7fb7a4006e30 tx=0x7fb7a4009e60).stop 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7abfff700 1 -- 172.21.2.117:0/1092213203 reap_dead start 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 -- 172.21.2.117:0/1092213203 shutdown_connections 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 -- 172.21.2.117:0/1092213203 >> 172.21.2.117:0/1092213203 conn(0x7fb7ac0cf060 msgr2=0x7fb7ac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 -- 172.21.2.117:0/1092213203 shutdown_connections 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.290 7fb7b1fe6700 1 -- 172.21.2.117:0/1092213203 wait complete. 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.305 7fb7b1fe6700 1 Processor -- start 2021-02-10T11:50:44.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.305 7fb7b1fe6700 1 -- start start 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac08cf40 0x7fb7ac0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7ac0a9f90 0x7fb7ac0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac0aae30 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac0aa9f0 con 0x7fb7ac0a9f90 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac0ab330 con 0x7fb7ac08cf40 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55010/0 (socket says 172.21.2.117:55010) 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 -- 172.21.2.117:0/817026528 learned_addr learned my addr 172.21.2.117:0/817026528 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 -- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac08cf40 msgr2=0x7fb7ac0a9970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:44.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac08cf40 0x7fb7ac0a9970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 -- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7ac0a9f90 msgr2=0x7fb7ac0aa390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7ac0a9f90 0x7fb7ac0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 -- 172.21.2.117:0/817026528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac0ab550 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7ab7fe700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7fb7940095e0 tx=0x7fb794009980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb7940070c0 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7940043a0 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac0ab550 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.306 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb794004770 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.307 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac0ab550 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.307 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb794008000 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.307 7fb7a3fff700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 0x7fb79001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.307 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb7940328d0 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.308 7fb7aaffd700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 0x7fb79001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.308 7fb7aaffd700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 0x7fb79001abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.338 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac0ede60 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.340 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb7940043a0 con 0x7fb7ac1a48b0 2021-02-10T11:50:44.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.468 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000519176"}) v1 -- 0x7fb7ac0ede60 con 0x7fb7900186c0 2021-02-10T11:50:44.471 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:44.475 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:50:44.471 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:44.475 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:50:44.471 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:44.475 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:50:44.471 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:44.475 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:50:44.471 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:44.475 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:50:44.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.485 7fb7a3fff700 1 -- 172.21.2.117:0/817026528 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb79c0068c0 con 0x7fb7900186c0 2021-02-10T11:50:44.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 msgr2=0x7fb79001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 0x7fb79001abc0 crc :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 msgr2=0x7fb7ac08c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7fb7940095e0 tx=0x7fb794009980).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 shutdown_connections 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb7ac0a9f90 0x7fb7ac0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb7ac08cf40 0x7fb7ac0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb7900186c0 0x7fb79001abc0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 --2- 172.21.2.117:0/817026528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb7ac1a48b0 0x7fb7ac08c920 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 >> 172.21.2.117:0/817026528 conn(0x7fb7ac092d00 msgr2=0x7fb7ac08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:44.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 shutdown_connections 2021-02-10T11:50:44.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.502 7fb7b1fe6700 1 -- 172.21.2.117:0/817026528 wait complete. 2021-02-10T11:50:44.508 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000787922 --group_name group_0000000000519176 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 Processor -- start 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 -- start start 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440e7170 0x7fcb440e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb440e8380 0x7fcb440e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb440e7570 con 0x7fcb441b07e0 2021-02-10T11:50:44.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb440e5cd0 con 0x7fcb440e8380 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.926 7fcb49f68700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb440e77e0 con 0x7fcb440e7170 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb440e8380 0x7fcb440e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb440e8380 0x7fcb440e8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55022/0 (socket says 172.21.2.117:55022) 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 -- 172.21.2.117:0/1199603217 learned_addr learned my addr 172.21.2.117:0/1199603217 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 -- 172.21.2.117:0/1199603217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440e7170 msgr2=0x7fcb440e7d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 --2- 172.21.2.117:0/1199603217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440e7170 0x7fcb440e7d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 -- 172.21.2.117:0/1199603217 >> v1:172.21.2.125:6789/0 conn(0x7fcb441b07e0 legacy=0x7fcb440e4e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 -- 172.21.2.117:0/1199603217 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb440e8da0 con 0x7fcb440e8380 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb43fff700 1 --2- 172.21.2.117:0/1199603217 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb440e8380 0x7fcb440e8780 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7fcb3c006f00 tx=0x7fcb3c009f10).ready entity=mon.0 client_cookie=610d9a9e840f791d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb427fc700 1 -- 172.21.2.117:0/1199603217 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c00b3b0 con 0x7fcb440e8380 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb427fc700 1 -- 172.21.2.117:0/1199603217 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00b630 con 0x7fcb440e8380 2021-02-10T11:50:44.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb427fc700 1 -- 172.21.2.117:0/1199603217 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c00bb10 con 0x7fcb440e8380 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb49f68700 1 -- 172.21.2.117:0/1199603217 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb440e8380 msgr2=0x7fcb440e8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.927 7fcb49f68700 1 --2- 172.21.2.117:0/1199603217 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb440e8380 0x7fcb440e8780 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7fcb3c006f00 tx=0x7fcb3c009f10).stop 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 -- 172.21.2.117:0/1199603217 shutdown_connections 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 --2- 172.21.2.117:0/1199603217 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb440e8380 0x7fcb440e8780 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 --2- 172.21.2.117:0/1199603217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440e7170 0x7fcb440e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 -- 172.21.2.117:0/1199603217 >> 172.21.2.117:0/1199603217 conn(0x7fcb440ce4f0 msgr2=0x7fcb440e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 -- 172.21.2.117:0/1199603217 shutdown_connections 2021-02-10T11:50:44.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.928 7fcb49f68700 1 -- 172.21.2.117:0/1199603217 wait complete. 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 Processor -- start 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 -- start start 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 0x7fcb440b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 0x7fcb4418a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4419ec40 con 0x7fcb4418a7f0 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb44187d00 con 0x7fcb440b7630 2021-02-10T11:50:44.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb49f68700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb441a84d0 con 0x7fcb440b8050 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb43fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb43fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55024/0 (socket says 172.21.2.117:55024) 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.943 7fcb43fff700 1 -- 172.21.2.117:0/2191866544 learned_addr learned my addr 172.21.2.117:0/2191866544 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb437fe700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 0x7fcb440b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 -- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 msgr2=0x7fcb4418a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb42ffd700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 0x7fcb4418a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 0x7fcb4418a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 -- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 msgr2=0x7fcb440b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 0x7fcb440b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 -- 172.21.2.117:0/2191866544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb441a86f0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb437fe700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 0x7fcb440b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:44.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb42ffd700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 0x7fcb4418a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb43fff700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7fcb3c00b9b0 tx=0x7fcb3c00bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c003030 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3c00c2f0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb3c00c7d0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb441a86f0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.944 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb441a8c80 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.945 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb3c001270 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.945 7fcb3bfff700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.945 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb3c0328d0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.945 7fcb437fe700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:44.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.946 7fcb437fe700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 0x7fcb2801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:44.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.976 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb44108ab0 con 0x7fcb4418a7f0 2021-02-10T11:50:44.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:44.979 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb3c009450 con 0x7fcb4418a7f0 2021-02-10T11:50:45.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.103 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000787922", "target": ["mgr", ""], "group_name": "group_0000000000519176"}) v1 -- 0x7fcb441a8c80 con 0x7fcb280186c0 2021-02-10T11:50:45.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.129 7fcb3bfff700 1 -- 172.21.2.117:0/2191866544 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fcb34006bc0 con 0x7fcb280186c0 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.136 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 msgr2=0x7fcb2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.136 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 0x7fcb2801abc0 crc :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.136 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 msgr2=0x7fcb4418abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.136 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7fcb3c00b9b0 tx=0x7fcb3c00bd50).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 shutdown_connections 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb280186c0 0x7fcb2801abc0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb4418a7f0 0x7fcb4418abf0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb440b8050 0x7fcb4418a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 --2- 172.21.2.117:0/2191866544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb440b7630 0x7fcb440b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 >> 172.21.2.117:0/2191866544 conn(0x7fcb4419f870 msgr2=0x7fcb44110fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 shutdown_connections 2021-02-10T11:50:45.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.137 7fcb49f68700 1 -- 172.21.2.117:0/2191866544 wait complete. 2021-02-10T11:50:45.143 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:50:45.494 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000519176 2021-02-10T11:50:45.494 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000519176:subvolume_0000000000787922.meta 2021-02-10T11:50:45.495 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000787922 alice --group_name group_0000000000519176 --tenant_id tenant1 2021-02-10T11:50:45.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.918 7f8b76a2a700 1 Processor -- start 2021-02-10T11:50:45.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.918 7f8b76a2a700 1 -- start start 2021-02-10T11:50:45.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b76a2a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 0x7f8b700eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b76a2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b700eb830 0x7f8b701b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b76a2a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b700ef7f0 con 0x7f8b700d18f0 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b76a2a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b700eded0 con 0x7f8b700eb830 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b76a2a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700d1cf0 con 0x7f8b700ef3f0 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b75047700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b700eb830 0x7f8b701b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b74846700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8b700d18f0 0x7f8b700eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55282/0 (socket says 172.21.2.117:55282) 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b74846700 1 -- 172.21.2.117:0/4183660770 learned_addr learned my addr 172.21.2.117:0/4183660770 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b75047700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b700eb830 0x7f8b701b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55032/0 (socket says 172.21.2.117:55032) 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.919 7f8b6ffff700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 0x7f8b700eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b75047700 1 -- 172.21.2.117:0/4183660770 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 msgr2=0x7f8b700eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b75047700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 0x7f8b700eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b75047700 1 -- 172.21.2.117:0/4183660770 >> v1:172.21.2.117:6789/0 conn(0x7f8b700d18f0 legacy=0x7f8b700eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b75047700 1 -- 172.21.2.117:0/4183660770 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b701b8fd0 con 0x7f8b700eb830 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b6ffff700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 0x7f8b700eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b75047700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b700eb830 0x7f8b701b79a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f8b68006ab0 tx=0x7f8b68009b40).ready entity=mon.0 client_cookie=fb99748b4102e70a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b6f7fe700 1 -- 172.21.2.117:0/4183660770 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b680022e0 con 0x7f8b700eb830 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b6f7fe700 1 -- 172.21.2.117:0/4183660770 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68002660 con 0x7f8b700eb830 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b6f7fe700 1 -- 172.21.2.117:0/4183660770 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b68002a70 con 0x7f8b700eb830 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b76a2a700 1 -- 172.21.2.117:0/4183660770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b700eb830 msgr2=0x7f8b701b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.920 7f8b76a2a700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b700eb830 0x7f8b701b79a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f8b68006ab0 tx=0x7f8b68009b40).stop 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 -- 172.21.2.117:0/4183660770 shutdown_connections 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b700eb830 0x7f8b701b79a0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 --2- 172.21.2.117:0/4183660770 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b700ef3f0 0x7f8b700eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 -- 172.21.2.117:0/4183660770 >> 172.21.2.117:0/4183660770 conn(0x7f8b700cf060 msgr2=0x7f8b700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 -- 172.21.2.117:0/4183660770 shutdown_connections 2021-02-10T11:50:45.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.921 7f8b76a2a700 1 -- 172.21.2.117:0/4183660770 wait complete. 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 Processor -- start 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 -- start start 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 0x7f8b701b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 0x7f8b70084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7008c290 con 0x7f8b701b27a0 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70191a60 con 0x7f8b701b23a0 2021-02-10T11:50:45.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.936 7f8b76a2a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70191c80 con 0x7f8b701b1980 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b74846700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 0x7f8b701b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b6ffff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 0x7f8b70084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55038/0 (socket says 172.21.2.117:55038) 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 -- 172.21.2.117:0/1808458432 learned_addr learned my addr 172.21.2.117:0/1808458432 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b6ffff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 0x7f8b70084ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:59418/0 (socket says 172.21.2.117:59418) 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 -- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 msgr2=0x7f8b701b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 0x7f8b701b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 -- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 msgr2=0x7f8b70084ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 0x7f8b70084ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 -- 172.21.2.117:0/1808458432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b700858e0 con 0x7f8b701b27a0 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b74846700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 0x7f8b701b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:45.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b75047700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f8b6800b5e0 tx=0x7f8b6800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b68009070 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68002280 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b700858e0 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b68002720 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.937 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b700858e0 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.938 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b68002460 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.938 7f8b6d7fa700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 0x7f8b5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.938 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b680327b0 con 0x7f8b701b27a0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.938 7f8b74846700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 0x7f8b5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:45.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.939 7f8b74846700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 0x7f8b5401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:45.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.969 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7008c680 con 0x7f8b701b27a0 2021-02-10T11:50:45.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:45.972 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b68002020 con 0x7f8b701b27a0 2021-02-10T11:50:46.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.096 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000787922", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "alice", "group_name": "group_0000000000519176"}) v1 -- 0x7f8b7008c680 con 0x7f8b54018700 2021-02-10T11:50:46.100 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID alice with tenant tenant1 2021-02-10T11:50:46.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.131 7f8b6d7fa700 1 -- 172.21.2.117:0/1808458432 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f8b58000ef0 con 0x7f8b54018700 2021-02-10T11:50:46.158 INFO:teuthology.orchestra.run.gibba017.stdout:AQCWyCNgExXRBhAAzN6GIHaVMjwt+4UFJzL66Q== 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 msgr2=0x7f8b5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 0x7f8b5401abc0 crc :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 msgr2=0x7f8b700852c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f8b6800b5e0 tx=0x7f8b6800b980).stop 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 shutdown_connections 2021-02-10T11:50:46.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b54018700 0x7f8b5401abc0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b701b27a0 0x7f8b700852c0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b701b23a0 0x7f8b70084ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 --2- 172.21.2.117:0/1808458432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b701b1980 0x7f8b701b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.162 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 >> 172.21.2.117:0/1808458432 conn(0x7f8b7019f710 msgr2=0x7f8b700db250 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.163 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 shutdown_connections 2021-02-10T11:50:46.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:46.163 7f8b76a2a700 1 -- 172.21.2.117:0/1808458432 wait complete. 2021-02-10T11:50:46.168 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:50:46.514 INFO:teuthology.orchestra.run.gibba017.stdout:$alice.meta 2021-02-10T11:50:46.514 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000519176 2021-02-10T11:50:46.514 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000519176:subvolume_0000000000787922.meta 2021-02-10T11:50:46.515 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat '/home/ubuntu/cephtest/mnt.1/volumes/$alice.meta' 2021-02-10T11:50:46.878 INFO:teuthology.orchestra.run.gibba017.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000519176/subvolume_0000000000787922": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:50:46.879 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo sudo cat /home/ubuntu/cephtest/mnt.1/volumes/_group_0000000000519176:subvolume_0000000000787922.meta 2021-02-10T11:50:47.238 INFO:teuthology.orchestra.run.gibba017.stdout:{"auths": {"alice": {"access_level": "rw", "dirty": false}}, "version": 5, "compat_version": 1} 2021-02-10T11:50:47.238 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000787922 alice --group_name group_0000000000519176 --tenant_id tenant2 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.664 7fdca0e21700 1 Processor -- start 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.664 7fdca0e21700 1 -- start start 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.665 7fdca0e21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 0x7fdc9c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.665 7fdca0e21700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 0x7fdc9c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.665 7fdca0e21700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c0ef7f0 con 0x7fdc9c0d18f0 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.665 7fdca0e21700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c0eded0 con 0x7fdc9c0eb830 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.665 7fdca0e21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0d1cf0 con 0x7fdc9c0ef3f0 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9ab13700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fdc9c0d18f0 0x7fdc9c0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48222/0 (socket says 172.21.2.117:48222) 2021-02-10T11:50:47.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9a312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 0x7fdc9c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9ab13700 1 -- 172.21.2.117:0/2143791977 learned_addr learned my addr 172.21.2.117:0/2143791977 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9b314700 1 --2- 172.21.2.117:0/2143791977 >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 0x7fdc9c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9a312700 1 -- 172.21.2.117:0/2143791977 >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 msgr2=0x7fdc9c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9a312700 1 --2- 172.21.2.117:0/2143791977 >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 0x7fdc9c1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9a312700 1 -- 172.21.2.117:0/2143791977 >> v1:172.21.2.125:6789/0 conn(0x7fdc9c0d18f0 legacy=0x7fdc9c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9a312700 1 -- 172.21.2.117:0/2143791977 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c1b8fd0 con 0x7fdc9c0ef3f0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc99b11700 1 -- 172.21.2.117:0/2143791977 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079328253 0 0) 0x7fdc8c0010c0 con 0x7fdc9c0d18f0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.666 7fdc9b314700 1 --2- 172.21.2.117:0/2143791977 >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 0x7fdc9c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdc9a312700 1 --2- 172.21.2.117:0/2143791977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 0x7fdc9c0eb210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdc90006650 tx=0x7fdc90009a80).ready entity=mon.2 client_cookie=9ec6730e72401c19 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdc99b11700 1 -- 172.21.2.117:0/2143791977 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc9000c0c0 con 0x7fdc9c0ef3f0 2021-02-10T11:50:47.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdc99b11700 1 -- 172.21.2.117:0/2143791977 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9000c340 con 0x7fdc9c0ef3f0 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdc99b11700 1 -- 172.21.2.117:0/2143791977 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc9000c860 con 0x7fdc9c0ef3f0 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdca0e21700 1 -- 172.21.2.117:0/2143791977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 msgr2=0x7fdc9c0eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.667 7fdca0e21700 1 --2- 172.21.2.117:0/2143791977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 0x7fdc9c0eb210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fdc90006650 tx=0x7fdc90009a80).stop 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 -- 172.21.2.117:0/2143791977 shutdown_connections 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 --2- 172.21.2.117:0/2143791977 >> v2:172.21.2.125:3300/0 conn(0x7fdc9c0eb830 0x7fdc9c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 --2- 172.21.2.117:0/2143791977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0ef3f0 0x7fdc9c0eb210 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 -- 172.21.2.117:0/2143791977 >> 172.21.2.117:0/2143791977 conn(0x7fdc9c0cf060 msgr2=0x7fdc9c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 -- 172.21.2.117:0/2143791977 shutdown_connections 2021-02-10T11:50:47.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.668 7fdca0e21700 1 -- 172.21.2.117:0/2143791977 wait complete. 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 Processor -- start 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 -- start start 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc9c07f6e0 0x7fdc9c0a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0a9550 0x7fdc9c07e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0aa220 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c07edf0 con 0x7fdc9c07f6e0 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.682 7fdca0e21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c1a38c0 con 0x7fdc9c0a9550 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:47.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55052/0 (socket says 172.21.2.117:55052) 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 -- 172.21.2.117:0/3680617957 learned_addr learned my addr 172.21.2.117:0/3680617957 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 -- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0a9550 msgr2=0x7fdc9c07e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0a9550 0x7fdc9c07e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 -- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc9c07f6e0 msgr2=0x7fdc9c0a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc9c07f6e0 0x7fdc9c0a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 -- 172.21.2.117:0/3680617957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c1a3ae0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc9b314700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7fdc9400b910 tx=0x7fdc9400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc94006aa0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.683 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc940011f0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c1a3ae0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c0841e0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc940016d0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc94006aa0 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdc8b7fe700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 0x7fdc8001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:47.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.684 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdc94032b70 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.685 7fdc9ab13700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 0x7fdc8001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:47.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.685 7fdc9ab13700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 0x7fdc8001ac10 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:47.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.714 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c0ede60 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.717 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc94000e20 con 0x7fdc9c1a2ea0 2021-02-10T11:50:47.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.841 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000787922", "target": ["mgr", ""], "tenant_id": "tenant2", "auth_id": "alice", "group_name": "group_0000000000519176"}) v1 -- 0x7fdc9c0841e0 con 0x7fdc80018730 2021-02-10T11:50:47.846 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:50:47.850 7f59c98e1700 -1 mgr.server reply reply (1) Operation not permitted auth ID: alice is already in use 2021-02-10T11:50:47.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.849 7fdc8b7fe700 1 -- 172.21.2.117:0/3680617957 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -1 auth ID: alice is already in use) v1 ==== 40+0+0 (crc 0 0 0) 0x7fdc8c000c30 con 0x7fdc80018730 2021-02-10T11:50:47.847 INFO:teuthology.orchestra.run.gibba017.stderr:Error EPERM: auth ID: alice is already in use 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 msgr2=0x7fdc8001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 0x7fdc8001ac10 crc :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 msgr2=0x7fdc9c1a32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7fdc9400b910 tx=0x7fdc9400bcb0).stop 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 shutdown_connections 2021-02-10T11:50:47.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc80018730 0x7fdc8001ac10 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc9c1a2ea0 0x7fdc9c1a32a0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc9c0a9550 0x7fdc9c07e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 --2- 172.21.2.117:0/3680617957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc9c07f6e0 0x7fdc9c0a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 >> 172.21.2.117:0/3680617957 conn(0x7fdc9c0b19e0 msgr2=0x7fdc9c0aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 shutdown_connections 2021-02-10T11:50:47.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:47.851 7fdca0e21700 1 -- 172.21.2.117:0/3680617957 wait complete. 2021-02-10T11:50:47.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:47.857 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000787922 alice --group_name group_0000000000519176 2021-02-10T11:50:48.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.273 7fb480d07700 1 Processor -- start 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.273 7fb480d07700 1 -- start start 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb480d07700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb480d07700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb47c0ef3f0 0x7fb47c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb480d07700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0ede60 con 0x7fb47c0eb000 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb480d07700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0ef7f0 con 0x7fb47c0d18e0 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb480d07700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0d1ce0 con 0x7fb47c0ef3f0 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55058/0 (socket says 172.21.2.117:55058) 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 -- 172.21.2.117:0/1517374547 learned_addr learned my addr 172.21.2.117:0/1517374547 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47b314700 1 --2- 172.21.2.117:0/1517374547 >> v2:172.21.2.125:3300/0 conn(0x7fb47c0ef3f0 0x7fb47c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 -- 172.21.2.117:0/1517374547 >> v2:172.21.2.125:3300/0 conn(0x7fb47c0ef3f0 msgr2=0x7fb47c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 --2- 172.21.2.117:0/1517374547 >> v2:172.21.2.125:3300/0 conn(0x7fb47c0ef3f0 0x7fb47c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 -- 172.21.2.117:0/1517374547 >> v1:172.21.2.125:6789/0 conn(0x7fb47c0eb000 legacy=0x7fb47c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 -- 172.21.2.117:0/1517374547 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c1b8f90 con 0x7fb47c0d18e0 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.274 7fb47ab13700 1 --2- 172.21.2.117:0/1517374547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fb46c006700 tx=0x7fb46c009b80).ready entity=mon.0 client_cookie=e7acc822e25f82e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb479b11700 1 -- 172.21.2.117:0/1517374547 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb46c00c0c0 con 0x7fb47c0d18e0 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb479b11700 1 -- 172.21.2.117:0/1517374547 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c00c340 con 0x7fb47c0d18e0 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb479b11700 1 -- 172.21.2.117:0/1517374547 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb46c00c860 con 0x7fb47c0d18e0 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 -- 172.21.2.117:0/1517374547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c0d18e0 msgr2=0x7fb47c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 --2- 172.21.2.117:0/1517374547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fb46c006700 tx=0x7fb46c009b80).stop 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 -- 172.21.2.117:0/1517374547 shutdown_connections 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 --2- 172.21.2.117:0/1517374547 >> v2:172.21.2.125:3300/0 conn(0x7fb47c0ef3f0 0x7fb47c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 --2- 172.21.2.117:0/1517374547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c0d18e0 0x7fb47c0ea9e0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 -- 172.21.2.117:0/1517374547 >> 172.21.2.117:0/1517374547 conn(0x7fb47c0ced60 msgr2=0x7fb47c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:48.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.275 7fb480d07700 1 -- 172.21.2.117:0/1517374547 shutdown_connections 2021-02-10T11:50:48.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.276 7fb480d07700 1 -- 172.21.2.117:0/1517374547 wait complete. 2021-02-10T11:50:48.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.290 7fb480d07700 1 Processor -- start 2021-02-10T11:50:48.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.290 7fb480d07700 1 -- start start 2021-02-10T11:50:48.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 0x7fb47c09bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb47c1a2e60 0x7fb47c1a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c09c1f0 con 0x7fb47c1a3880 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b5810 con 0x7fb47c09b630 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb480d07700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b5a30 con 0x7fb47c1a2e60 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55064/0 (socket says 172.21.2.117:55064) 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 -- 172.21.2.117:0/2549579212 learned_addr learned my addr 172.21.2.117:0/2549579212 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47ab13700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 0x7fb47c09bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47a312700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb47c1a2e60 0x7fb47c1a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 -- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb47c1a2e60 msgr2=0x7fb47c1a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb47c1a2e60 0x7fb47c1a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 -- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 msgr2=0x7fb47c09bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 0x7fb47c09bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 -- 172.21.2.117:0/2549579212 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0b5c50 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47ab13700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 0x7fb47c09bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.291 7fb47b314700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fb47400b970 tx=0x7fb47400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb474000f90 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb474006aa0 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb474001680 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb480d07700 1 -- 172.21.2.117:0/2549579212 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0b5c50 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb480d07700 1 -- 172.21.2.117:0/2549579212 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0b61e0 con 0x7fb47c1a3880 2021-02-10T11:50:48.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4740011f0 con 0x7fb47c1a3880 2021-02-10T11:50:48.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.292 7fb4737fe700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:48.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.293 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb474032a40 con 0x7fb47c1a3880 2021-02-10T11:50:48.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.293 7fb47ab13700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:48.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.293 7fb47ab13700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.323 7fb480d07700 1 -- 172.21.2.117:0/2549579212 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0ede60 con 0x7fb47c1a3880 2021-02-10T11:50:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.326 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb47401f010 con 0x7fb47c1a3880 2021-02-10T11:50:48.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.450 7fb480d07700 1 -- 172.21.2.117:0/2549579212 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000787922", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000519176"}) v1 -- 0x7fb47c0b61e0 con 0x7fb4600186c0 2021-02-10T11:50:48.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.487 7fb4737fe700 1 -- 172.21.2.117:0/2549579212 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb46c006e80 con 0x7fb4600186c0 2021-02-10T11:50:48.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 msgr2=0x7fb46001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 0x7fb46001abc0 crc :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 msgr2=0x7fb47c0b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fb47400b970 tx=0x7fb47400bd10).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 shutdown_connections 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4600186c0 0x7fb46001abc0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb47c1a3880 0x7fb47c0b51f0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb47c1a2e60 0x7fb47c1a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 --2- 172.21.2.117:0/2549579212 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb47c09b630 0x7fb47c09bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 >> 172.21.2.117:0/2549579212 conn(0x7fb47c092880 msgr2=0x7fb47c0aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 shutdown_connections 2021-02-10T11:50:48.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:48.516 7fb480d07700 1 -- 172.21.2.117:0/2549579212 wait complete. 2021-02-10T11:50:48.523 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:50:48.870 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000519176 2021-02-10T11:50:48.870 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000519176:subvolume_0000000000787922.meta 2021-02-10T11:50:48.871 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000787922 --group_name group_0000000000519176 2021-02-10T11:50:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 Processor -- start 2021-02-10T11:50:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 -- start start 2021-02-10T11:50:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb8680e4e40 0x7fb8681b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb868085930 con 0x7fb8681b0810 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb868084840 con 0x7fb8680e4e40 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.288 7fb8703e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb868084a30 con 0x7fb8680e5240 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:52748/0 (socket says 172.21.2.117:52748) 2021-02-10T11:50:49.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 -- 172.21.2.117:0/3093877478 learned_addr learned my addr 172.21.2.117:0/3093877478 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86e9fd700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fb8681b0810 0x7fb868085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48244/0 (socket says 172.21.2.117:48244) 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86e1fc700 1 --2- 172.21.2.117:0/3093877478 >> v2:172.21.2.125:3300/0 conn(0x7fb8680e4e40 0x7fb8681b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 -- 172.21.2.117:0/3093877478 >> v2:172.21.2.125:3300/0 conn(0x7fb8680e4e40 msgr2=0x7fb8681b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 --2- 172.21.2.117:0/3093877478 >> v2:172.21.2.125:3300/0 conn(0x7fb8680e4e40 0x7fb8681b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 -- 172.21.2.117:0/3093877478 >> v1:172.21.2.125:6789/0 conn(0x7fb8681b0810 legacy=0x7fb868085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 -- 172.21.2.117:0/3093877478 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868085280 con 0x7fb8680e5240 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d9fb700 1 --2- 172.21.2.117:0/3093877478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fb85c006700 tx=0x7fb85c009b80).ready entity=mon.2 client_cookie=f140b013d6d0eb62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.289 7fb86d1fa700 1 -- 172.21.2.117:0/3093877478 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb85c00c0c0 con 0x7fb8680e5240 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb86d1fa700 1 -- 172.21.2.117:0/3093877478 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c00c380 con 0x7fb8680e5240 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb86d1fa700 1 -- 172.21.2.117:0/3093877478 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb85c00c8a0 con 0x7fb8680e5240 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 -- 172.21.2.117:0/3093877478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 msgr2=0x7fb868084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 --2- 172.21.2.117:0/3093877478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fb85c006700 tx=0x7fb85c009b80).stop 2021-02-10T11:50:49.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 -- 172.21.2.117:0/3093877478 shutdown_connections 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 --2- 172.21.2.117:0/3093877478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680e5240 0x7fb868084c60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 --2- 172.21.2.117:0/3093877478 >> v2:172.21.2.125:3300/0 conn(0x7fb8680e4e40 0x7fb8681b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 -- 172.21.2.117:0/3093877478 >> 172.21.2.117:0/3093877478 conn(0x7fb8680ce4e0 msgr2=0x7fb8680e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 -- 172.21.2.117:0/3093877478 shutdown_connections 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.290 7fb8703e0700 1 -- 172.21.2.117:0/3093877478 wait complete. 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 Processor -- start 2021-02-10T11:50:49.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 -- start start 2021-02-10T11:50:49.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8680ad1b0 0x7fb8680ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680adbd0 0x7fb86809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86818d0c0 con 0x7fb8680973b0 2021-02-10T11:50:49.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86809ae70 con 0x7fb8680ad1b0 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb8703e0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86809ba10 con 0x7fb8680adbd0 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb86e1fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb86e1fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55074/0 (socket says 172.21.2.117:55074) 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb86e1fc700 1 -- 172.21.2.117:0/2978662967 learned_addr learned my addr 172.21.2.117:0/2978662967 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb86e9fd700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680adbd0 0x7fb86809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.305 7fb86d9fb700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8680ad1b0 0x7fb8680ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 -- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680adbd0 msgr2=0x7fb86809b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680adbd0 0x7fb86809b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 -- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8680ad1b0 msgr2=0x7fb8680ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8680ad1b0 0x7fb8680ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 -- 172.21.2.117:0/2978662967 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86809bbf0 con 0x7fb8680973b0 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb86e1fc700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7fb864006f60 tx=0x7fb86400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb86400bd60 con 0x7fb8680973b0 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb864000c90 con 0x7fb8680973b0 2021-02-10T11:50:49.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb864002220 con 0x7fb8680973b0 2021-02-10T11:50:49.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86809bbf0 con 0x7fb8680973b0 2021-02-10T11:50:49.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.306 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86809c180 con 0x7fb8680973b0 2021-02-10T11:50:49.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.307 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb864006b00 con 0x7fb8680973b0 2021-02-10T11:50:49.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.307 7fb85affd700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:49.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.307 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb8640328f0 con 0x7fb8680973b0 2021-02-10T11:50:49.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.307 7fb86d9fb700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:49.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.307 7fb86d9fb700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 0x7fb85001abc0 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:49.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.338 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb868085930 con 0x7fb8680973b0 2021-02-10T11:50:49.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.341 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb864000940 con 0x7fb8680973b0 2021-02-10T11:50:49.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.465 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000787922", "target": ["mgr", ""], "group_name": "group_0000000000519176"}) v1 -- 0x7fb86809c180 con 0x7fb8500186c0 2021-02-10T11:50:49.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.478 7fb85affd700 1 -- 172.21.2.117:0/2978662967 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb85c006430 con 0x7fb8500186c0 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 msgr2=0x7fb85001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 0x7fb85001abc0 crc :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 msgr2=0x7fb868097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7fb864006f60 tx=0x7fb86400b970).stop 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 shutdown_connections 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8680adbd0 0x7fb86809b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8680ad1b0 0x7fb8680ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8500186c0 0x7fb85001abc0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 --2- 172.21.2.117:0/2978662967 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8680973b0 0x7fb868097980 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:49.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 >> 172.21.2.117:0/2978662967 conn(0x7fb86818cc70 msgr2=0x7fb868093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:49.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 shutdown_connections 2021-02-10T11:50:49.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:49.483 7fb8703e0700 1 -- 172.21.2.117:0/2978662967 wait complete. 2021-02-10T11:50:49.490 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:50:49.837 INFO:teuthology.orchestra.run.gibba017.stdout:_deleting 2021-02-10T11:50:49.838 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000519176 2021-02-10T11:50:49.839 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:49.839 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:49.839 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:50.174 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:50:50.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:50.174 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:50:50.174 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:50:50.174 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:50.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[162483]: fuse finished with error 0 and tester_r 0 2021-02-10T11:50:50.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:50.527 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:50.527 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:50.862 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:50:50.863 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:50:50.863 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:50:50.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:50.863 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:51.198 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000519176 2021-02-10T11:50:51.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 Processor -- start 2021-02-10T11:50:51.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 -- start start 2021-02-10T11:50:51.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 0x7f2b140e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 0x7f2b140e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140e77f0 con 0x7f2b140e6170 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b140e54b0 con 0x7f2b141b07f0 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.615 7f2b18ab1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140e9180 con 0x7f2b140e8d80 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1205d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2b140e6170 0x7f2b140e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55330/0 (socket says 172.21.2.117:55330) 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1205d700 1 -- 172.21.2.117:0/3176170295 learned_addr learned my addr 172.21.2.117:0/3176170295 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1285e700 1 --2- 172.21.2.117:0/3176170295 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 0x7f2b140e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1305f700 1 --2- 172.21.2.117:0/3176170295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 0x7f2b140e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1185c700 1 -- 172.21.2.117:0/3176170295 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033238072 0 0) 0x7f2b080009b0 con 0x7f2b140e6170 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.616 7f2b1185c700 1 -- 172.21.2.117:0/3176170295 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc002fe0 con 0x7f2b140e6170 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1305f700 1 -- 172.21.2.117:0/3176170295 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 msgr2=0x7f2b140e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:51.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1305f700 1 --2- 172.21.2.117:0/3176170295 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 0x7f2b140e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1305f700 1 -- 172.21.2.117:0/3176170295 >> v1:172.21.2.117:6789/0 conn(0x7f2b140e6170 legacy=0x7f2b140e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1305f700 1 -- 172.21.2.117:0/3176170295 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140e7380 con 0x7f2b140e8d80 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1285e700 1 --2- 172.21.2.117:0/3176170295 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 0x7f2b140e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1305f700 1 --2- 172.21.2.117:0/3176170295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 0x7f2b140e6d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f2b0c006e50 tx=0x7f2b0c009e70).ready entity=mon.2 client_cookie=247d319e9c955dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1185c700 1 -- 172.21.2.117:0/3176170295 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0c006a20 con 0x7f2b140e8d80 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1185c700 1 -- 172.21.2.117:0/3176170295 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c006790 con 0x7f2b140e8d80 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b18ab1700 1 -- 172.21.2.117:0/3176170295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 msgr2=0x7f2b140e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b18ab1700 1 --2- 172.21.2.117:0/3176170295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 0x7f2b140e6d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f2b0c006e50 tx=0x7f2b0c009e70).stop 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.617 7f2b1185c700 1 -- 172.21.2.117:0/3176170295 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0c000f90 con 0x7f2b140e8d80 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 -- 172.21.2.117:0/3176170295 shutdown_connections 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 --2- 172.21.2.117:0/3176170295 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b140e8d80 0x7f2b140e6d60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 --2- 172.21.2.117:0/3176170295 >> v2:172.21.2.125:3300/0 conn(0x7f2b141b07f0 0x7f2b140e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 -- 172.21.2.117:0/3176170295 >> 172.21.2.117:0/3176170295 conn(0x7f2b140ce500 msgr2=0x7f2b140e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:51.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 -- 172.21.2.117:0/3176170295 shutdown_connections 2021-02-10T11:50:51.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.618 7f2b18ab1700 1 -- 172.21.2.117:0/3176170295 wait complete. 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.632 7f2b18ab1700 1 Processor -- start 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 -- start start 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b140b8ac0 0x7f2b1419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b1419df40 0x7f2b140bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140bb430 con 0x7f2b140bbfd0 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140bc9f0 con 0x7f2b140b8ac0 2021-02-10T11:50:51.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b18ab1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140bcc10 con 0x7f2b1419df40 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b1305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b1305f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55086/0 (socket says 172.21.2.117:55086) 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.633 7f2b1305f700 1 -- 172.21.2.117:0/69803674 learned_addr learned my addr 172.21.2.117:0/69803674 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 -- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b1419df40 msgr2=0x7f2b140bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b1419df40 0x7f2b140bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 -- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b140b8ac0 msgr2=0x7f2b1419d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b140b8ac0 0x7f2b1419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 -- 172.21.2.117:0/69803674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140b6f70 con 0x7f2b140bbfd0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b1305f700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f2b0c00b9e0 tx=0x7f2b0c00bd80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0c009170 con 0x7f2b140bbfd0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c003030 con 0x7f2b140bbfd0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b0c0013b0 con 0x7f2b140bbfd0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b140b6f70 con 0x7f2b140bbfd0 2021-02-10T11:50:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.634 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b140b6f70 con 0x7f2b140bbfd0 2021-02-10T11:50:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.635 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b0c001160 con 0x7f2b140bbfd0 2021-02-10T11:50:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.635 7f2b037fe700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 0x7f2af801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.635 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2b0c032a80 con 0x7f2b140bbfd0 2021-02-10T11:50:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.635 7f2b1285e700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 0x7f2af801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.636 7f2b1285e700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 0x7f2af801abc0 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:51.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.665 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1407e800 con 0x7f2b140bbfd0 2021-02-10T11:50:51.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.668 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b0c02cfb0 con 0x7f2b140bbfd0 2021-02-10T11:50:51.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.793 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000519176"}) v1 -- 0x7f2b140bb430 con 0x7f2af80186c0 2021-02-10T11:50:51.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.796 7f2b037fe700 1 -- 172.21.2.117:0/69803674 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2b04006420 con 0x7f2af80186c0 2021-02-10T11:50:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 msgr2=0x7f2af801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 0x7f2af801abc0 crc :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 msgr2=0x7f2b140bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7f2b0c00b9e0 tx=0x7f2b0c00bd80).stop 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 shutdown_connections 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2af80186c0 0x7f2af801abc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b140bbfd0 0x7f2b140bc3d0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b1419df40 0x7f2b140bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 --2- 172.21.2.117:0/69803674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b140b8ac0 0x7f2b1419d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 >> 172.21.2.117:0/69803674 conn(0x7f2b14110940 msgr2=0x7f2b1407fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 shutdown_connections 2021-02-10T11:50:51.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:51.798 7f2b18ab1700 1 -- 172.21.2.117:0/69803674 wait complete. 2021-02-10T11:50:51.806 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:50:51.806 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:52.153 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:52.499 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:50:52.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:52.499 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:52.499 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:52.834 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:50:52.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:50:52.834 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:50:52.835 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:50:52.835 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:53.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[161842]: fuse finished with error 0 and tester_r 0 2021-02-10T11:50:53.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:53.188 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:53.189 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:53.524 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:50:53.524 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:50:53.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:53.525 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:50:53.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:53.860 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:53.860 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:54.195 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:50:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:54.196 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:50:54.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:54.196 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:50:54.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:54.532 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:54.532 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:54.866 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:50:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:54.866 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:50:54.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:54.867 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:50:55.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:55.201 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:55.202 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:55.536 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:50:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:50:55.536 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:50:55.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:50:55.537 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:50:55.872 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:50:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.288 7f0294e7f700 1 Processor -- start 2021-02-10T11:50:56.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.288 7f0294e7f700 1 -- start start 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.289 7f0294e7f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 0x7f0290086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.289 7f0294e7f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0290084c60 0x7f0290085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.289 7f0294e7f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0290085930 con 0x7f02900e4e40 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.289 7f0294e7f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0290085680 con 0x7f0290084c60 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.289 7f0294e7f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02901b09e0 con 0x7f02900e5240 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028eb13700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f02900e4e40 0x7f02901b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48266/0 (socket says 172.21.2.117:48266) 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028f314700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0290084c60 0x7f0290085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028e312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 0x7f0290086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028eb13700 1 -- 172.21.2.117:0/1101635365 learned_addr learned my addr 172.21.2.117:0/1101635365 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028f314700 1 -- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 msgr2=0x7f0290086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028f314700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 0x7f0290086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028f314700 1 -- 172.21.2.117:0/1101635365 >> v1:172.21.2.125:6789/0 conn(0x7f02900e4e40 legacy=0x7f02901b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028f314700 1 -- 172.21.2.117:0/1101635365 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0290084840 con 0x7f0290084c60 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.290 7f028e312700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 0x7f0290086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f028f314700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0290084c60 0x7f0290085060 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f0288006e50 tx=0x7f0288009e70).ready entity=mon.1 client_cookie=116a4675eac713a0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f028db11700 1 -- 172.21.2.117:0/1101635365 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0288000f90 con 0x7f0290084c60 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f028db11700 1 -- 172.21.2.117:0/1101635365 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02880066f0 con 0x7f0290084c60 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f028db11700 1 -- 172.21.2.117:0/1101635365 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f028800b700 con 0x7f0290084c60 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f0294e7f700 1 -- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0290084c60 msgr2=0x7f0290085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.291 7f0294e7f700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0290084c60 0x7f0290085060 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f0288006e50 tx=0x7f0288009e70).stop 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 -- 172.21.2.117:0/1101635365 shutdown_connections 2021-02-10T11:50:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0290084c60 0x7f0290085060 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 --2- 172.21.2.117:0/1101635365 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f02900e5240 0x7f0290086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 -- 172.21.2.117:0/1101635365 >> 172.21.2.117:0/1101635365 conn(0x7f02900ce4e0 msgr2=0x7f02900e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:56.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 -- 172.21.2.117:0/1101635365 shutdown_connections 2021-02-10T11:50:56.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.293 7f0294e7f700 1 -- 172.21.2.117:0/1101635365 wait complete. 2021-02-10T11:50:56.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.306 7f0294e7f700 1 Processor -- start 2021-02-10T11:50:56.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.306 7f0294e7f700 1 -- start start 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f029019e5b0 0x7f029019fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f029018ecc0 0x7f029018fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029018f660 con 0x7f029018f0c0 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029019e9b0 con 0x7f029018ecc0 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f0294e7f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029019e0c0 con 0x7f029019e5b0 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55096/0 (socket says 172.21.2.117:55096) 2021-02-10T11:50:56.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 -- 172.21.2.117:0/1386097763 learned_addr learned my addr 172.21.2.117:0/1386097763 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 -- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f029019e5b0 msgr2=0x7f029019fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f029019e5b0 0x7f029019fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 -- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f029018ecc0 msgr2=0x7f029018fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f029018ecc0 0x7f029018fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 -- 172.21.2.117:0/1386097763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0290190820 con 0x7f029018f0c0 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.307 7f028f314700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f028800b970 tx=0x7f028800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0288000f90 con 0x7f029018f0c0 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0288003030 con 0x7f029018f0c0 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0288001420 con 0x7f029018f0c0 2021-02-10T11:50:56.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0290190820 con 0x7f029018f0c0 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0290190820 con 0x7f029018f0c0 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f02880011a0 con 0x7f029018f0c0 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 0x7f027401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.308 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0288032d20 con 0x7f029018f0c0 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.309 7f028eb13700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 0x7f027401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.309 7f028eb13700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 0x7f027401abc0 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.339 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029010c290 con 0x7f029018f0c0 2021-02-10T11:50:56.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.341 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0288000f90 con 0x7f029018f0c0 2021-02-10T11:50:56.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.487 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0290085930 con 0x7f029018f0c0 2021-02-10T11:50:56.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.491 7f02877fe700 1 -- 172.21.2.117:0/1386097763 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f02880011a0 con 0x7f029018f0c0 2021-02-10T11:50:56.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 msgr2=0x7f027401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 0x7f027401abc0 crc :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 msgr2=0x7f0290190200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7f028800b970 tx=0x7f028800bd10).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 shutdown_connections 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0274018700 0x7f027401abc0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f029018f0c0 0x7f0290190200 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f029018ecc0 0x7f029018fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 --2- 172.21.2.117:0/1386097763 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f029019e5b0 0x7f029019fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 >> 172.21.2.117:0/1386097763 conn(0x7f02901a5f90 msgr2=0x7f02901a6560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 shutdown_connections 2021-02-10T11:50:56.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.496 7f0294e7f700 1 -- 172.21.2.117:0/1386097763 wait complete. 2021-02-10T11:50:56.504 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes' 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.916 7fc848607700 1 Processor -- start 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.916 7fc848607700 1 -- start start 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc848607700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 0x7fc8400ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc848607700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8400eb000 0x7fc8400eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc848607700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400ede60 con 0x7fc8400ee3e0 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc848607700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8400eba20 con 0x7fc8400eb000 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc848607700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8400d1ce0 con 0x7fc8400d18e0 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc845c22700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8400eb000 0x7fc8400eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc846c24700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc8400ee3e0 0x7fc8400ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55354/0 (socket says 172.21.2.117:55354) 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.917 7fc846c24700 1 -- 172.21.2.117:0/3387705912 learned_addr learned my addr 172.21.2.117:0/3387705912 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845c22700 1 -- 172.21.2.117:0/3387705912 >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 msgr2=0x7fc8400ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc846423700 1 --2- 172.21.2.117:0/3387705912 >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 0x7fc8400ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845c22700 1 --2- 172.21.2.117:0/3387705912 >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 0x7fc8400ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845c22700 1 -- 172.21.2.117:0/3387705912 >> v1:172.21.2.117:6789/0 conn(0x7fc8400ee3e0 legacy=0x7fc8400ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845421700 1 -- 172.21.2.117:0/3387705912 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582234637 0 0) 0x7fc838001500 con 0x7fc8400ee3e0 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845c22700 1 -- 172.21.2.117:0/3387705912 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400eefd0 con 0x7fc8400eb000 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc846423700 1 --2- 172.21.2.117:0/3387705912 >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 0x7fc8400ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845c22700 1 --2- 172.21.2.117:0/3387705912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8400eb000 0x7fc8400eb400 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7fc830006650 tx=0x7fc830009a80).ready entity=mon.0 client_cookie=a040a38ee663e442 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845421700 1 -- 172.21.2.117:0/3387705912 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc83000c0c0 con 0x7fc8400eb000 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845421700 1 -- 172.21.2.117:0/3387705912 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83000c340 con 0x7fc8400eb000 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.918 7fc845421700 1 -- 172.21.2.117:0/3387705912 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc83000c860 con 0x7fc8400eb000 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 -- 172.21.2.117:0/3387705912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8400eb000 msgr2=0x7fc8400eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 --2- 172.21.2.117:0/3387705912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8400eb000 0x7fc8400eb400 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7fc830006650 tx=0x7fc830009a80).stop 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 -- 172.21.2.117:0/3387705912 shutdown_connections 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 --2- 172.21.2.117:0/3387705912 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8400eb000 0x7fc8400eb400 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 --2- 172.21.2.117:0/3387705912 >> v2:172.21.2.125:3300/0 conn(0x7fc8400d18e0 0x7fc8400ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 -- 172.21.2.117:0/3387705912 >> 172.21.2.117:0/3387705912 conn(0x7fc8400ced60 msgr2=0x7fc8400cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 -- 172.21.2.117:0/3387705912 shutdown_connections 2021-02-10T11:50:56.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.919 7fc848607700 1 -- 172.21.2.117:0/3387705912 wait complete. 2021-02-10T11:50:56.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 Processor -- start 2021-02-10T11:50:56.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 -- start start 2021-02-10T11:50:56.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc840091b00 0x7fc840091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc840096270 0x7fc840196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400967f0 con 0x7fc840095850 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401972b0 con 0x7fc840096270 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc848607700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc840197490 con 0x7fc840091b00 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.934 7fc845c22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55112/0 (socket says 172.21.2.117:55112) 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 -- 172.21.2.117:0/1794391769 learned_addr learned my addr 172.21.2.117:0/1794391769 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 -- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc840091b00 msgr2=0x7fc840091f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc840091b00 0x7fc840091f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 -- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc840096270 msgr2=0x7fc840196c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc840096270 0x7fc840196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 -- 172.21.2.117:0/1794391769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8401976b0 con 0x7fc840095850 2021-02-10T11:50:56.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc845c22700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7fc830009630 tx=0x7fc83000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8300093a0 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83000bcf0 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc830005560 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.935 7fc848607700 1 -- 172.21.2.117:0/1794391769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8401976b0 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.936 7fc848607700 1 -- 172.21.2.117:0/1794391769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8401976b0 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.936 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc830008240 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.936 7fc836ffd700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 0x7fc82801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.936 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc8300328d0 con 0x7fc840095850 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.937 7fc846423700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 0x7fc82801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:56.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.937 7fc846423700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 0x7fc82801ac10 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:56.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.967 7fc848607700 1 -- 172.21.2.117:0/1794391769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc840087290 con 0x7fc840095850 2021-02-10T11:50:56.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:56.970 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc830008030 con 0x7fc840095850 2021-02-10T11:50:57.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.101 7fc848607700 1 -- 172.21.2.117:0/1794391769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]} v 0) v1 -- 0x7fc8400ede60 con 0x7fc840095850 2021-02-10T11:50:57.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.547 7fc836ffd700 1 -- 172.21.2.117:0/1794391769 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_multitenant_subvolumes"]}]=0 v628) v1 ==== 144+0+0 (secure 0 0 0) 0x7fc830008240 con 0x7fc840095850 2021-02-10T11:50:57.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 msgr2=0x7fc82801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 0x7fc82801ac10 crc :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 msgr2=0x7fc840095c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7fc830009630 tx=0x7fc83000b900).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 shutdown_connections 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc828018730 0x7fc82801ac10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc840095850 0x7fc840095c50 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc840096270 0x7fc840196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 --2- 172.21.2.117:0/1794391769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc840091b00 0x7fc840091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 >> 172.21.2.117:0/1794391769 conn(0x7fc8401a3620 msgr2=0x7fc840092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 shutdown_connections 2021-02-10T11:50:57.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.570 7fc848607700 1 -- 172.21.2.117:0/1794391769 wait complete. 2021-02-10T11:50:57.576 INFO:tasks.cephfs_test_runner:test_multitenant_subvolumes (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:50:57.577 INFO:tasks.cephfs_test_runner:Starting test: test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:50:57.577 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:50:58.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 Processor -- start 2021-02-10T11:50:58.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 -- start start 2021-02-10T11:50:58.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec0d18e0 0x7fd0ec0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0ec0eb000 0x7fd0ec0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd0ec0ee3e0 0x7fd0ec0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec0ede60 con 0x7fd0ec0eb000 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0ec0eba20 con 0x7fd0ec0ee3e0 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0f2b7a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0d1ce0 con 0x7fd0ec0d18e0 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0ebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0ec0eb000 0x7fd0ec0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0ebfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd0ec0eb000 0x7fd0ec0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55116/0 (socket says 172.21.2.117:55116) 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.991 7fd0ebfff700 1 -- 172.21.2.117:0/515794116 learned_addr learned my addr 172.21.2.117:0/515794116 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 -- 172.21.2.117:0/515794116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec0d18e0 msgr2=0x7fd0ec0ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 --2- 172.21.2.117:0/515794116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec0d18e0 0x7fd0ec0ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 -- 172.21.2.117:0/515794116 >> v2:172.21.2.125:3300/0 conn(0x7fd0ec0ee3e0 msgr2=0x7fd0ec0ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 --2- 172.21.2.117:0/515794116 >> v2:172.21.2.125:3300/0 conn(0x7fd0ec0ee3e0 0x7fd0ec0ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 -- 172.21.2.117:0/515794116 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0eee00 con 0x7fd0ec0eb000 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0ebfff700 1 --2- 172.21.2.117:0/515794116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0eb000 0x7fd0ec0eb400 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7fd0dc006a80 tx=0x7fd0dc009f00).ready entity=mon.0 client_cookie=cb8b1ce3060e34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0eb7fe700 1 -- 172.21.2.117:0/515794116 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0dc000d30 con 0x7fd0ec0eb000 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0eb7fe700 1 -- 172.21.2.117:0/515794116 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc00b5c0 con 0x7fd0ec0eb000 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0eb7fe700 1 -- 172.21.2.117:0/515794116 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0dc00ba50 con 0x7fd0ec0eb000 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0f2b7a700 1 -- 172.21.2.117:0/515794116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0eb000 msgr2=0x7fd0ec0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.992 7fd0f2b7a700 1 --2- 172.21.2.117:0/515794116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0eb000 0x7fd0ec0eb400 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7fd0dc006a80 tx=0x7fd0dc009f00).stop 2021-02-10T11:50:58.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 -- 172.21.2.117:0/515794116 shutdown_connections 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 --2- 172.21.2.117:0/515794116 >> v2:172.21.2.125:3300/0 conn(0x7fd0ec0ee3e0 0x7fd0ec0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 --2- 172.21.2.117:0/515794116 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0eb000 0x7fd0ec0eb400 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 --2- 172.21.2.117:0/515794116 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec0d18e0 0x7fd0ec0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 -- 172.21.2.117:0/515794116 >> 172.21.2.117:0/515794116 conn(0x7fd0ec0ced60 msgr2=0x7fd0ec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 -- 172.21.2.117:0/515794116 shutdown_connections 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:57.993 7fd0f2b7a700 1 -- 172.21.2.117:0/515794116 wait complete. 2021-02-10T11:50:58.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.007 7fd0f2b7a700 1 Processor -- start 2021-02-10T11:50:58.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.007 7fd0f2b7a700 1 -- start start 2021-02-10T11:50:58.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec19c7c0 0x7fd0ec092f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0ec08a280 0x7fd0ec08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0a9aa0 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec19cbc0 con 0x7fd0ec08a280 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f2b7a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec08aca0 con 0x7fd0ec19c7c0 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55126/0 (socket says 172.21.2.117:55126) 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 -- 172.21.2.117:0/4117252476 learned_addr learned my addr 172.21.2.117:0/4117252476 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 -- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec19c7c0 msgr2=0x7fd0ec092f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:58.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec19c7c0 0x7fd0ec092f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 -- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0ec08a280 msgr2=0x7fd0ec08a680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0ec08a280 0x7fd0ec08a680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.008 7fd0f1197700 1 -- 172.21.2.117:0/4117252476 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0e99b0 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0f1197700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fd0e400b5c0 tx=0x7fd0e400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e400bd50 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e4006030 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec0e99b0 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec19b330 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0e40054d0 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.009 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0e4006210 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.010 7fd0e97fa700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 0x7fd0d001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.010 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd0e4032790 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.010 7fd0f0996700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 0x7fd0d001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:58.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.010 7fd0f0996700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 0x7fd0d001abd0 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:58.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.041 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec0e8a60 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.043 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0e4000bc0 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.174 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7fd0ec0e8a60 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.551 7fd0e97fa700 1 -- 172.21.2.117:0/4117252476 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v629) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd0e4006030 con 0x7fd0ec0e8f90 2021-02-10T11:50:58.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.594 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 msgr2=0x7fd0d001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.594 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 0x7fd0d001abd0 crc :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.594 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 msgr2=0x7fd0ec0e9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:58.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.594 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7fd0e400b5c0 tx=0x7fd0e400b960).stop 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 shutdown_connections 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0d00186d0 0x7fd0d001abd0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0ec0e8f90 0x7fd0ec0e9390 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0ec08a280 0x7fd0ec08a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 --2- 172.21.2.117:0/4117252476 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0ec19c7c0 0x7fd0ec092f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 >> 172.21.2.117:0/4117252476 conn(0x7fd0ec19f610 msgr2=0x7fd0ec098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 shutdown_connections 2021-02-10T11:50:58.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:58.595 7fd0f2b7a700 1 -- 172.21.2.117:0/4117252476 wait complete. 2021-02-10T11:50:58.601 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:50:59.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.017 7f77c2b1b700 1 Processor -- start 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.017 7f77c2b1b700 1 -- start start 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c2b1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77bc0d18f0 0x7f77bc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c2b1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f77bc0eb040 0x7f77bc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c2b1b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77bc0ede60 con 0x7f77bc0ee3e0 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c2b1b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77bc0ebad0 con 0x7f77bc0d18f0 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c2b1b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77bc0d1cf0 con 0x7f77bc0eb040 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c0937700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77bc0d18f0 0x7f77bc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77bbfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f77bc0eb040 0x7f77bc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c1138700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f77bc0ee3e0 0x7f77bc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55380/0 (socket says 172.21.2.117:55380) 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.018 7f77c1138700 1 -- 172.21.2.117:0/2092748133 learned_addr learned my addr 172.21.2.117:0/2092748133 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bbfff700 1 -- 172.21.2.117:0/2092748133 >> v2:172.21.2.117:3300/0 conn(0x7f77bc0d18f0 msgr2=0x7f77bc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bbfff700 1 --2- 172.21.2.117:0/2092748133 >> v2:172.21.2.117:3300/0 conn(0x7f77bc0d18f0 0x7f77bc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bbfff700 1 -- 172.21.2.117:0/2092748133 >> v1:172.21.2.117:6789/0 conn(0x7f77bc0ee3e0 legacy=0x7f77bc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bbfff700 1 -- 172.21.2.117:0/2092748133 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc0eefd0 con 0x7f77bc0eb040 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bbfff700 1 --2- 172.21.2.117:0/2092748133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc0eb040 0x7f77bc0eb440 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f77ac0069d0 tx=0x7f77ac009e40).ready entity=mon.1 client_cookie=ac0d3a8f80cd19a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bb7fe700 1 -- 172.21.2.117:0/2092748133 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77ac00b510 con 0x7f77bc0eb040 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bb7fe700 1 -- 172.21.2.117:0/2092748133 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac00b890 con 0x7f77bc0eb040 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77bb7fe700 1 -- 172.21.2.117:0/2092748133 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77ac00bd70 con 0x7f77bc0eb040 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77c2b1b700 1 -- 172.21.2.117:0/2092748133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc0eb040 msgr2=0x7f77bc0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.019 7f77c2b1b700 1 --2- 172.21.2.117:0/2092748133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc0eb040 0x7f77bc0eb440 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f77ac0069d0 tx=0x7f77ac009e40).stop 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 -- 172.21.2.117:0/2092748133 shutdown_connections 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 --2- 172.21.2.117:0/2092748133 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc0eb040 0x7f77bc0eb440 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 --2- 172.21.2.117:0/2092748133 >> v2:172.21.2.117:3300/0 conn(0x7f77bc0d18f0 0x7f77bc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 -- 172.21.2.117:0/2092748133 >> 172.21.2.117:0/2092748133 conn(0x7f77bc0cf060 msgr2=0x7f77bc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 -- 172.21.2.117:0/2092748133 shutdown_connections 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.020 7f77c2b1b700 1 -- 172.21.2.117:0/2092748133 wait complete. 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 Processor -- start 2021-02-10T11:50:59.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 -- start start 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77bc083990 0x7f77bc0863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc1a4180 0x7f77bc0dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc085e50 con 0x7f77bc1a4580 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc097270 con 0x7f77bc1a4180 2021-02-10T11:50:59.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c2b1b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc097450 con 0x7f77bc083990 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c1138700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c1138700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55138/0 (socket says 172.21.2.117:55138) 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c1138700 1 -- 172.21.2.117:0/756666765 learned_addr learned my addr 172.21.2.117:0/756666765 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.035 7f77c0937700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77bc083990 0x7f77bc0863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77bbfff700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc1a4180 0x7f77bc0dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 -- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77bc083990 msgr2=0x7f77bc0863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77bc083990 0x7f77bc0863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 -- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc1a4180 msgr2=0x7f77bc0dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc1a4180 0x7f77bc0dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 -- 172.21.2.117:0/756666765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc09ed70 con 0x7f77bc1a4580 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c1138700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f77b400a570 tx=0x7f77b400a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b40070d0 con 0x7f77bc1a4580 2021-02-10T11:50:59.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b400ad00 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc09ed70 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77bc09f2c0 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.036 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77b4004620 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.037 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77b4000990 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.037 7f77b97fa700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 0x7f77a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.037 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f77b40326f0 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.037 7f77c0937700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 0x7f77a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.037 7f77c0937700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 0x7f77a001abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.067 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc09fb50 con 0x7f77bc1a4580 2021-02-10T11:50:59.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.070 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77b4000ef0 con 0x7f77bc1a4580 2021-02-10T11:50:59.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.217 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f77bc09fb50 con 0x7f77bc1a4580 2021-02-10T11:50:59.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.220 7f77b97fa700 1 -- 172.21.2.117:0/756666765 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f77b4000990 con 0x7f77bc1a4580 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 msgr2=0x7f77a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 0x7f77a001abc0 crc :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 msgr2=0x7f77bc0dc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7f77b400a570 tx=0x7f77b400a910).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 shutdown_connections 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77a00186c0 0x7f77a001abc0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77bc1a4580 0x7f77bc0dc460 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f77bc1a4180 0x7f77bc0dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 --2- 172.21.2.117:0/756666765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77bc083990 0x7f77bc0863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:50:59.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 >> 172.21.2.117:0/756666765 conn(0x7f77bc0b6540 msgr2=0x7f77bc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:50:59.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 shutdown_connections 2021-02-10T11:50:59.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:50:59.221 7f77c2b1b700 1 -- 172.21.2.117:0/756666765 wait complete. 2021-02-10T11:50:59.228 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:50:59.228 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:59.576 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:50:59.920 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:50:59.920 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:50:59.920 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:50:59.921 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:00.256 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:51:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:00.256 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:51:00.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:00.256 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:00.256 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:00.592 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:51:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:00.592 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:51:00.592 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:00.593 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:00.593 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:51:00.927 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:51:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:00.927 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:51:00.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:00.927 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:00.927 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:51:01.262 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:51:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:01.263 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:51:01.263 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:51:01.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.677 7f42b3b55700 1 Processor -- start 2021-02-10T11:51:01.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.677 7f42b3b55700 1 -- start start 2021-02-10T11:51:01.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b3b55700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42ac0d18e0 0x7f42ac0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b3b55700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f42ac0ef3f0 0x7f42ac1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b3b55700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ac0ede60 con 0x7f42ac0eb000 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b3b55700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ac0ef7f0 con 0x7f42ac0ef3f0 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b3b55700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ac0d1ce0 con 0x7f42ac0d18e0 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b1170700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f42ac0eb000 0x7f42ac0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55392/0 (socket says 172.21.2.117:55392) 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b1170700 1 -- 172.21.2.117:0/3568099183 learned_addr learned my addr 172.21.2.117:0/3568099183 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 --2- 172.21.2.117:0/3568099183 >> v2:172.21.2.117:3300/0 conn(0x7f42ac0ef3f0 0x7f42ac1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b1971700 1 --2- 172.21.2.117:0/3568099183 >> v2:172.21.2.125:3300/0 conn(0x7f42ac0d18e0 0x7f42ac0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b096f700 1 -- 172.21.2.117:0/3568099183 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19202996 0 0) 0x7f42a80009b0 con 0x7f42ac0eb000 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b096f700 1 -- 172.21.2.117:0/3568099183 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4298002fe0 con 0x7f42ac0eb000 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 -- 172.21.2.117:0/3568099183 >> v2:172.21.2.125:3300/0 conn(0x7f42ac0d18e0 msgr2=0x7f42ac0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 --2- 172.21.2.117:0/3568099183 >> v2:172.21.2.125:3300/0 conn(0x7f42ac0d18e0 0x7f42ac0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 -- 172.21.2.117:0/3568099183 >> v1:172.21.2.117:6789/0 conn(0x7f42ac0eb000 legacy=0x7f42ac0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 -- 172.21.2.117:0/3568099183 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac1b8f90 con 0x7f42ac0ef3f0 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.678 7f42b2172700 1 --2- 172.21.2.117:0/3568099183 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ef3f0 0x7f42ac1b7960 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f42a4006e30 tx=0x7f42a4009e60).ready entity=mon.0 client_cookie=612a005d14555dcb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b096f700 1 -- 172.21.2.117:0/3568099183 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42a4006a50 con 0x7f42ac0ef3f0 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b096f700 1 -- 172.21.2.117:0/3568099183 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a400b300 con 0x7f42ac0ef3f0 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b096f700 1 -- 172.21.2.117:0/3568099183 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42a400b820 con 0x7f42ac0ef3f0 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 -- 172.21.2.117:0/3568099183 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ef3f0 msgr2=0x7f42ac1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 --2- 172.21.2.117:0/3568099183 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ef3f0 0x7f42ac1b7960 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f42a4006e30 tx=0x7f42a4009e60).stop 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 -- 172.21.2.117:0/3568099183 shutdown_connections 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 --2- 172.21.2.117:0/3568099183 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ef3f0 0x7f42ac1b7960 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 --2- 172.21.2.117:0/3568099183 >> v2:172.21.2.125:3300/0 conn(0x7f42ac0d18e0 0x7f42ac0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.679 7f42b3b55700 1 -- 172.21.2.117:0/3568099183 >> 172.21.2.117:0/3568099183 conn(0x7f42ac0ced60 msgr2=0x7f42ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.680 7f42b3b55700 1 -- 172.21.2.117:0/3568099183 shutdown_connections 2021-02-10T11:51:01.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.680 7f42b3b55700 1 -- 172.21.2.117:0/3568099183 wait complete. 2021-02-10T11:51:01.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 Processor -- start 2021-02-10T11:51:01.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 -- start start 2021-02-10T11:51:01.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42ac1a4290 0x7f42ac1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42ac0a6fe0 0x7f42ac0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac0a7d10 con 0x7f42ac0ab280 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac0a7a00 con 0x7f42ac1a4290 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.694 7f42b3b55700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac096370 con 0x7f42ac0a6fe0 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55148/0 (socket says 172.21.2.117:55148) 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 -- 172.21.2.117:0/3661169001 learned_addr learned my addr 172.21.2.117:0/3661169001 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 -- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42ac0a6fe0 msgr2=0x7f42ac0a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42ac0a6fe0 0x7f42ac0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 -- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42ac1a4290 msgr2=0x7f42ac1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42ac1a4290 0x7f42ac1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:01.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 -- 172.21.2.117:0/3661169001 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac0a8290 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f42b1170700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f42a8001c60 tx=0x7f42a800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.695 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42a800acc0 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8004140 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42a8004660 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ac0a8290 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ac0a8290 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42a8007f30 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f429e7fc700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 0x7f429401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.696 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f42a8032740 con 0x7f42ac0ab280 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.697 7f42b1971700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 0x7f429401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:01.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.697 7f42b1971700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 0x7f429401ac10 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:01.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.727 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac087290 con 0x7f42ac0ab280 2021-02-10T11:51:01.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.730 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42a8000e20 con 0x7f42ac0ab280 2021-02-10T11:51:01.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.853 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f42ac0ede60 con 0x7f42ac0ab280 2021-02-10T11:51:01.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.854 7f429e7fc700 1 -- 172.21.2.117:0/3661169001 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v117) v1 ==== 81+0+30766 (secure 0 0 0) 0x7f42a8004140 con 0x7f42ac0ab280 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 117, 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:51:01.853 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:50:20.397701", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:01.854 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 25, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:51:01.855 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:01.856 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:01.857 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:01.858 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.859 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:51:01.860 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 24, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:50:17.365216", 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:01.861 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:01.862 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "116", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:01.863 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:01.864 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:01.865 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.866 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 25, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:50:18.436070", 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:01.867 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "117", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:01.868 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:01.869 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:01.870 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:51:01.871 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.872 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.873 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:51:01.874 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.875 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.876 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.877 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.878 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 114, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.879 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.880 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.881 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.882 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.883 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.884 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.885 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.886 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.887 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.888 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.889 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.890 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:51:01.891 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.892 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.892 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.892 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.892 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:51:01.892 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.893 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.894 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.895 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:51:01.896 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:51:01.897 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.898 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 107, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.899 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.900 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:51:01.901 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.902 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:51:01.903 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.904 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 114, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:51:01.905 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.906 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.907 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.908 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.909 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:51:01.910 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.911 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.912 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.913 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.914 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:01.915 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:51:01.916 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.917 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.918 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:01.919 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:01.920 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.921 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:51:01.922 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.923 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.924 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.925 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.926 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:51:01.927 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:51:01.928 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:51:01.929 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 msgr2=0x7f429401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 0x7f429401ac10 crc :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 msgr2=0x7f42ac0ab680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7f42a8001c60 tx=0x7f42a800a8d0).stop 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 shutdown_connections 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4294018730 0x7f429401ac10 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42ac0ab280 0x7f42ac0ab680 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:01.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42ac0a6fe0 0x7f42ac0a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:01.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 --2- 172.21.2.117:0/3661169001 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42ac1a4290 0x7f42ac1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:01.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 >> 172.21.2.117:0/3661169001 conn(0x7f42ac0874c0 msgr2=0x7f42ac096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:01.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 shutdown_connections 2021-02-10T11:51:01.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:01.856 7f42b3b55700 1 -- 172.21.2.117:0/3661169001 wait complete. 2021-02-10T11:51:01.931 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 Processor -- start 2021-02-10T11:51:02.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 -- start start 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 0x7fae4c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fae4c085450 0x7fae4c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c085930 con 0x7fae4c0e5240 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0e5cc0 con 0x7fae4c085450 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.345 7fae50d37700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c084840 con 0x7fae4c0e4e40 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4a312700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fae4c0e5240 0x7fae4c084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48332/0 (socket says 172.21.2.117:48332) 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4a312700 1 -- 172.21.2.117:0/2776147808 learned_addr learned my addr 172.21.2.117:0/2776147808 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 --2- 172.21.2.117:0/2776147808 >> v2:172.21.2.117:3300/0 conn(0x7fae4c085450 0x7fae4c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4ab13700 1 --2- 172.21.2.117:0/2776147808 >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 0x7fae4c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 -- 172.21.2.117:0/2776147808 >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 msgr2=0x7fae4c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:02.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 --2- 172.21.2.117:0/2776147808 >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 0x7fae4c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 -- 172.21.2.117:0/2776147808 >> v1:172.21.2.125:6789/0 conn(0x7fae4c0e5240 legacy=0x7fae4c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 -- 172.21.2.117:0/2776147808 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c1b0810 con 0x7fae4c085450 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4ab13700 1 --2- 172.21.2.117:0/2776147808 >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 0x7fae4c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.346 7fae4b314700 1 --2- 172.21.2.117:0/2776147808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c085450 0x7fae4c086ec0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fae44006e30 tx=0x7fae44009e60).ready entity=mon.0 client_cookie=5235df54d4dc687f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae49b11700 1 -- 172.21.2.117:0/2776147808 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4400b300 con 0x7fae4c085450 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae49b11700 1 -- 172.21.2.117:0/2776147808 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4400b580 con 0x7fae4c085450 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae49b11700 1 -- 172.21.2.117:0/2776147808 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae4400ba60 con 0x7fae4c085450 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 -- 172.21.2.117:0/2776147808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c085450 msgr2=0x7fae4c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 --2- 172.21.2.117:0/2776147808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c085450 0x7fae4c086ec0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fae44006e30 tx=0x7fae44009e60).stop 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 -- 172.21.2.117:0/2776147808 shutdown_connections 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 --2- 172.21.2.117:0/2776147808 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c085450 0x7fae4c086ec0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 --2- 172.21.2.117:0/2776147808 >> v2:172.21.2.125:3300/0 conn(0x7fae4c0e4e40 0x7fae4c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.347 7fae50d37700 1 -- 172.21.2.117:0/2776147808 >> 172.21.2.117:0/2776147808 conn(0x7fae4c0ce4e0 msgr2=0x7fae4c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:02.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.348 7fae50d37700 1 -- 172.21.2.117:0/2776147808 shutdown_connections 2021-02-10T11:51:02.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.348 7fae50d37700 1 -- 172.21.2.117:0/2776147808 wait complete. 2021-02-10T11:51:02.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.362 7fae50d37700 1 Processor -- start 2021-02-10T11:51:02.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.362 7fae50d37700 1 -- start start 2021-02-10T11:51:02.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae4c190a00 0x7fae4c190e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 0x7fae4c1aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c190480 con 0x7fae4c08a570 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c18f440 con 0x7fae4c190a00 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae50d37700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c18f620 con 0x7fae4c1aa2c0 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55160/0 (socket says 172.21.2.117:55160) 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 -- 172.21.2.117:0/1932261165 learned_addr learned my addr 172.21.2.117:0/1932261165 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4a312700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae4c190a00 0x7fae4c190e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4b314700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 0x7fae4c1aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 -- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 msgr2=0x7fae4c1aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 0x7fae4c1aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 -- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae4c190a00 msgr2=0x7fae4c190e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae4c190a00 0x7fae4c190e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.363 7fae4ab13700 1 -- 172.21.2.117:0/1932261165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c18f840 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae4b314700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 0x7fae4c1aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae4ab13700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fae34006f40 tx=0x7fae3400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae340009b0 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3400bd80 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae340022b0 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae50d37700 1 -- 172.21.2.117:0/1932261165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c18f840 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae50d37700 1 -- 172.21.2.117:0/1932261165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c18fdd0 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.364 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae34000c90 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.365 7fae437fe700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 0x7fae3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.365 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fae34032580 con 0x7fae4c08a570 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.365 7fae4a312700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 0x7fae3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.365 7fae4a312700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 0x7fae3001abc0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.395 7fae50d37700 1 -- 172.21.2.117:0/1932261165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c10c280 con 0x7fae4c08a570 2021-02-10T11:51:02.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.397 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae34030d60 con 0x7fae4c08a570 2021-02-10T11:51:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.526 7fae50d37700 1 -- 172.21.2.117:0/1932261165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae4c085930 con 0x7fae4c08a570 2021-02-10T11:51:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.526 7fae437fe700 1 -- 172.21.2.117:0/1932261165 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+2101 (secure 0 0 0) 0x7fae34000c90 con 0x7fae4c08a570 2021-02-10T11:51:02.525 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:02.525 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":95,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:50:21.403231","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7010":{"gid":7010,"name":"b","rank":0,"incarnation":91,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/143190062","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":143190062},{"type":"v1","addr":"172.21.2.125:6835","nonce":143190062}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:51:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 95 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 msgr2=0x7fae3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 0x7fae3001abc0 crc :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 msgr2=0x7fae4c08a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fae34006f40 tx=0x7fae3400b990).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 shutdown_connections 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae4c1aa2c0 0x7fae4c1aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae4c190a00 0x7fae4c190e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae300186c0 0x7fae3001abc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 --2- 172.21.2.117:0/1932261165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae4c08a570 0x7fae4c08a970 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 >> 172.21.2.117:0/1932261165 conn(0x7fae4c1a9b30 msgr2=0x7fae4c18c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 shutdown_connections 2021-02-10T11:51:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.528 7fae50d37700 1 -- 172.21.2.117:0/1932261165 wait complete. 2021-02-10T11:51:02.535 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:51:02.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 Processor -- start 2021-02-10T11:51:02.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 -- start start 2021-02-10T11:51:02.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01581b07f0 0x7f01580e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580e7580 con 0x7f01580e7d70 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01580e77f0 con 0x7f01580e7180 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.948 7f015f3ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01580e92f0 con 0x7f01581b07f0 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f015c9c6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f01580e7d70 0x7f01580e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55418/0 (socket says 172.21.2.117:55418) 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f015c9c6700 1 -- 172.21.2.117:0/4250940154 learned_addr learned my addr 172.21.2.117:0/4250940154 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603465535 0 0) 0x7f01540010b0 con 0x7f01580e7d70 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0144002fe0 con 0x7f01580e7d70 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f015d1c7700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01581b07f0 0x7f01580e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 596713871 0 0) 0x7f01540010b0 con 0x7f01580e7d70 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01581b07f0 msgr2=0x7f01580e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01581b07f0 0x7f01580e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 >> v1:172.21.2.125:6789/0 conn(0x7f01580e7180 legacy=0x7f01580e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580e5ce0 con 0x7f01580e7d70 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f01540015b0 con 0x7f01580e7d70 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 -- 172.21.2.117:0/4250940154 >> v1:172.21.2.117:6789/0 conn(0x7f01580e7d70 legacy=0x7f01580e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.949 7f014bfff700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0144003a40 0x7f0144005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0144006800 0x7f0144008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01440095b0 0x7f014400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014400c2f0 con 0x7f0144003a40 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014400c510 con 0x7f0144006800 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014400c730 con 0x7f01440095b0 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0144003a40 0x7f0144005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015c9c6700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0144006800 0x7f0144008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d9c8700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01440095b0 0x7f014400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 -- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01440095b0 msgr2=0x7f014400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01440095b0 0x7f014400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 -- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0144006800 msgr2=0x7f0144008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0144006800 0x7f0144008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 -- 172.21.2.117:0/4250940154 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0144050970 con 0x7f0144003a40 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d9c8700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01440095b0 0x7f014400baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f015d1c7700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0144003a40 0x7f0144005f60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f014c006ac0 tx=0x7f014c009ab0).ready entity=mon.0 client_cookie=fbe5ac4ed794105b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f014c00c0c0 con 0x7f0144003a40 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00c3c0 con 0x7f0144003a40 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.950 7f014bfff700 1 -- 172.21.2.117:0/4250940154 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f014c00c8a0 con 0x7f0144003a40 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 -- 172.21.2.117:0/4250940154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0144003a40 msgr2=0x7f0144005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 --2- 172.21.2.117:0/4250940154 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0144003a40 0x7f0144005f60 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7f014c006ac0 tx=0x7f014c009ab0).stop 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015d9c8700 1 -- 172.21.2.117:0/4250940154 reap_dead start 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 -- 172.21.2.117:0/4250940154 shutdown_connections 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 -- 172.21.2.117:0/4250940154 >> 172.21.2.117:0/4250940154 conn(0x7f01580ce500 msgr2=0x7f01580e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:02.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 -- 172.21.2.117:0/4250940154 shutdown_connections 2021-02-10T11:51:02.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.951 7f015f3ab700 1 -- 172.21.2.117:0/4250940154 wait complete. 2021-02-10T11:51:02.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 Processor -- start 2021-02-10T11:51:02.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 -- start start 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f015819fe70 0x7f01580b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0158093a10 0x7f0158093e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01580b3830 con 0x7f01580b3090 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01580b3490 con 0x7f015819fe70 2021-02-10T11:51:02.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.966 7f015f3ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01580b3db0 con 0x7f0158093a10 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55180/0 (socket says 172.21.2.117:55180) 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 -- 172.21.2.117:0/2970647688 learned_addr learned my addr 172.21.2.117:0/2970647688 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 -- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0158093a10 msgr2=0x7f0158093e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0158093a10 0x7f0158093e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 -- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f015819fe70 msgr2=0x7f01580b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f015819fe70 0x7f01580b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 -- 172.21.2.117:0/2970647688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01580b3fd0 con 0x7f01580b3090 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015c9c6700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f0154001660 tx=0x7f015400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015400ad20 con 0x7f01580b3090 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01540016a0 con 0x7f01580b3090 2021-02-10T11:51:02.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01540043c0 con 0x7f01580b3090 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01580b3fd0 con 0x7f01580b3090 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.967 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01580b3fd0 con 0x7f01580b3090 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.968 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0154004140 con 0x7f01580b3090 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.968 7f0149ffb700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 0x7f013801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.968 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f01540326d0 con 0x7f01580b3090 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.968 7f015d1c7700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 0x7f013801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:02.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.969 7f015d1c7700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 0x7f013801abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:02.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:02.998 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015810c1a0 con 0x7f01580b3090 2021-02-10T11:51:02.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.001 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0154001180 con 0x7f01580b3090 2021-02-10T11:51:03.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.130 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f01580e77f0 con 0x7f01580b3090 2021-02-10T11:51:03.674 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:51:03.676 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:51:03.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.682 7f0149ffb700 1 -- 172.21.2.117:0/2970647688 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0154004140 con 0x7f01580b3090 2021-02-10T11:51:03.694 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 msgr2=0x7f013801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 0x7f013801abc0 crc :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 msgr2=0x7f01580933f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7f0154001660 tx=0x7f015400a930).stop 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 shutdown_connections 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01380186c0 0x7f013801abc0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:03.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01580b3090 0x7f01580933f0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0158093a10 0x7f0158093e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 --2- 172.21.2.117:0/2970647688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f015819fe70 0x7f01580b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 >> 172.21.2.117:0/2970647688 conn(0x7f01580a7260 msgr2=0x7f01580bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.697 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 shutdown_connections 2021-02-10T11:51:03.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:03.698 7f015f3ab700 1 -- 172.21.2.117:0/2970647688 wait complete. 2021-02-10T11:51:03.704 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:03.733 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:51:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 Processor -- start 2021-02-10T11:51:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 -- start start 2021-02-10T11:51:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300ef3f0 0x7fa5300eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5300eb7f0 0x7fa5301b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300ede60 con 0x7fa5300d18e0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5300ef7f0 con 0x7fa5300eb7f0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa534809700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300d1ce0 con 0x7fa5300ef3f0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa52e61c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa5300d18e0 0x7fa5300ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55436/0 (socket says 172.21.2.117:55436) 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa52e61c700 1 -- 172.21.2.117:0/3247734553 learned_addr learned my addr 172.21.2.117:0/3247734553 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa52ee1d700 1 --2- 172.21.2.117:0/3247734553 >> v2:172.21.2.117:3300/0 conn(0x7fa5300eb7f0 0x7fa5301b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.116 7fa52de1b700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300ef3f0 0x7fa5300eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52ee1d700 1 -- 172.21.2.117:0/3247734553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300ef3f0 msgr2=0x7fa5300eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52ee1d700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300ef3f0 0x7fa5300eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52ee1d700 1 -- 172.21.2.117:0/3247734553 >> v1:172.21.2.117:6789/0 conn(0x7fa5300d18e0 legacy=0x7fa5300ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52ee1d700 1 -- 172.21.2.117:0/3247734553 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5301b8f90 con 0x7fa5300eb7f0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52ee1d700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300eb7f0 0x7fa5301b7960 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7fa528006e30 tx=0x7fa528009e60).ready entity=mon.0 client_cookie=49474b092a029d5d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52d61a700 1 -- 172.21.2.117:0/3247734553 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa528006a50 con 0x7fa5300eb7f0 2021-02-10T11:51:04.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52d61a700 1 -- 172.21.2.117:0/3247734553 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52800b300 con 0x7fa5300eb7f0 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa52d61a700 1 -- 172.21.2.117:0/3247734553 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa52800b7e0 con 0x7fa5300eb7f0 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa534809700 1 -- 172.21.2.117:0/3247734553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300eb7f0 msgr2=0x7fa5301b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.117 7fa534809700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300eb7f0 0x7fa5301b7960 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7fa528006e30 tx=0x7fa528009e60).stop 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 -- 172.21.2.117:0/3247734553 shutdown_connections 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300eb7f0 0x7fa5301b7960 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 --2- 172.21.2.117:0/3247734553 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300ef3f0 0x7fa5300eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 -- 172.21.2.117:0/3247734553 >> 172.21.2.117:0/3247734553 conn(0x7fa5300ced60 msgr2=0x7fa5300cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 -- 172.21.2.117:0/3247734553 shutdown_connections 2021-02-10T11:51:04.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.118 7fa534809700 1 -- 172.21.2.117:0/3247734553 wait complete. 2021-02-10T11:51:04.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 Processor -- start 2021-02-10T11:51:04.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 -- start start 2021-02-10T11:51:04.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5301a33d0 0x7fa5301a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 0x7fa5300b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300c0670 con 0x7fa5300b82f0 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300deb20 con 0x7fa5301a33d0 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.133 7fa534809700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5300ded40 con 0x7fa5300b8d10 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55194/0 (socket says 172.21.2.117:55194) 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 -- 172.21.2.117:0/3747429207 learned_addr learned my addr 172.21.2.117:0/3747429207 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52ee1d700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 0x7fa5300b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52e61c700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5301a33d0 0x7fa5301a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 -- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 msgr2=0x7fa5300b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 0x7fa5300b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 -- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5301a33d0 msgr2=0x7fa5301a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5301a33d0 0x7fa5301a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 -- 172.21.2.117:0/3747429207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5300def60 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52ee1d700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 0x7fa5300b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.134 7fa52de1b700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fa524006f60 tx=0x7fa52400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.135 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa52400bd60 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.135 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa524000c90 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.135 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa5240022b0 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.135 7fa534809700 1 -- 172.21.2.117:0/3747429207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5300def60 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.135 7fa534809700 1 -- 172.21.2.117:0/3747429207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5300df4f0 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.136 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa524006b00 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.136 7fa51effd700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 0x7fa51401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.136 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fa524032700 con 0x7fa5300b82f0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.136 7fa52e61c700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 0x7fa51401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.136 7fa52e61c700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 0x7fa51401abc0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.166 7fa534809700 1 -- 172.21.2.117:0/3747429207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa530087290 con 0x7fa5300b82f0 2021-02-10T11:51:04.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.169 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa524000940 con 0x7fa5300b82f0 2021-02-10T11:51:04.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.299 7fa534809700 1 -- 172.21.2.117:0/3747429207 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5300ede60 con 0x7fa5300b82f0 2021-02-10T11:51:04.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.299 7fa51effd700 1 -- 172.21.2.117:0/3747429207 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+1760 (secure 0 0 0) 0x7fa52400bd60 con 0x7fa5300b82f0 2021-02-10T11:51:04.299 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:04.299 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":96,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10770,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1089768522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1089768522},{"type":"v1","addr":"172.21.2.117:6833","nonce":1089768522}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":90}],"filesystems":[{"mdsmap":{"epoch":96,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:50:19.395674","modified":"2021-02-10 11:51:03.676664","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2021-02-10T11:51:04.299 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 96 2021-02-10T11:51:04.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.301 7fa534809700 1 -- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 msgr2=0x7fa51401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.301 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 0x7fa51401abc0 crc :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.301 7fa534809700 1 -- 172.21.2.117:0/3747429207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 msgr2=0x7fa5300b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.301 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7fa524006f60 tx=0x7fa52400b970).stop 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 -- 172.21.2.117:0/3747429207 shutdown_connections 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa5140186c0 0x7fa51401abc0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5300b82f0 0x7fa5300b86f0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5300b8d10 0x7fa5300b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 --2- 172.21.2.117:0/3747429207 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa5301a33d0 0x7fa5301a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 -- 172.21.2.117:0/3747429207 >> 172.21.2.117:0/3747429207 conn(0x7fa5300874c0 msgr2=0x7fa530096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:04.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 -- 172.21.2.117:0/3747429207 shutdown_connections 2021-02-10T11:51:04.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.302 7fa534809700 1 -- 172.21.2.117:0/3747429207 wait complete. 2021-02-10T11:51:04.309 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.727 7f0c52c9a700 1 Processor -- start 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.727 7f0c52c9a700 1 -- start start 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.727 7f0c52c9a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0c4c1b07f0 0x7f0c4c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.727 7f0c52c9a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.727 7f0c52c9a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c0e77f0 con 0x7f0c4c0e6170 2021-02-10T11:51:04.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c52c9a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4c0e5ce0 con 0x7f0c4c0e8d80 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c52c9a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4c0e54b0 con 0x7f0c4c1b07f0 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55200/0 (socket says 172.21.2.117:55200) 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 -- 172.21.2.117:0/17832442 learned_addr learned my addr 172.21.2.117:0/17832442 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c50ab6700 1 --2- 172.21.2.117:0/17832442 >> v2:172.21.2.125:3300/0 conn(0x7f0c4c1b07f0 0x7f0c4c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 -- 172.21.2.117:0/17832442 >> v2:172.21.2.125:3300/0 conn(0x7f0c4c1b07f0 msgr2=0x7f0c4c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:04.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 --2- 172.21.2.117:0/17832442 >> v2:172.21.2.125:3300/0 conn(0x7f0c4c1b07f0 0x7f0c4c0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 -- 172.21.2.117:0/17832442 >> v1:172.21.2.125:6789/0 conn(0x7f0c4c0e6170 legacy=0x7f0c4c0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 -- 172.21.2.117:0/17832442 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c0e9180 con 0x7f0c4c0e8d80 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c512b7700 1 --2- 172.21.2.117:0/17832442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f0c44006f00 tx=0x7f0c44009f10).ready entity=mon.0 client_cookie=e6443f9f7c4e4ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.728 7f0c4b7fe700 1 -- 172.21.2.117:0/17832442 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c4400b3b0 con 0x7f0c4c0e8d80 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c4b7fe700 1 -- 172.21.2.117:0/17832442 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4400b730 con 0x7f0c4c0e8d80 2021-02-10T11:51:04.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c4b7fe700 1 -- 172.21.2.117:0/17832442 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c44000f90 con 0x7f0c4c0e8d80 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 -- 172.21.2.117:0/17832442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0e8d80 msgr2=0x7f0c4c0e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 --2- 172.21.2.117:0/17832442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f0c44006f00 tx=0x7f0c44009f10).stop 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 -- 172.21.2.117:0/17832442 shutdown_connections 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 --2- 172.21.2.117:0/17832442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0e8d80 0x7f0c4c0e6d60 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 --2- 172.21.2.117:0/17832442 >> v2:172.21.2.125:3300/0 conn(0x7f0c4c1b07f0 0x7f0c4c0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 -- 172.21.2.117:0/17832442 >> 172.21.2.117:0/17832442 conn(0x7f0c4c0ce500 msgr2=0x7f0c4c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:04.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 -- 172.21.2.117:0/17832442 shutdown_connections 2021-02-10T11:51:04.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.729 7f0c52c9a700 1 -- 172.21.2.117:0/17832442 wait complete. 2021-02-10T11:51:04.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 Processor -- start 2021-02-10T11:51:04.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 -- start start 2021-02-10T11:51:04.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 0x7f0c4c1ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c4c1aded0 0x7f0c4c1ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c093f70 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c197350 con 0x7f0c4c1ad4b0 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.744 7f0c52c9a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c197570 con 0x7f0c4c1aded0 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55204/0 (socket says 172.21.2.117:55204) 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 -- 172.21.2.117:0/654784159 learned_addr learned my addr 172.21.2.117:0/654784159 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c512b7700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c4c1aded0 0x7f0c4c1ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c4bfff700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 0x7f0c4c1ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 -- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c4c1aded0 msgr2=0x7f0c4c1ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c4c1aded0 0x7f0c4c1ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 -- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 msgr2=0x7f0c4c1ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:04.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 0x7f0c4c1ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 -- 172.21.2.117:0/654784159 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c197790 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c4bfff700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 0x7f0c4c1ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.745 7f0c50ab6700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7f0c3400b530 tx=0x7f0c3400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c3400bcc0 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34002030 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c34002480 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c197790 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c197790 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c34002210 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 0x7f0c3001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.746 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f0c34032970 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.747 7f0c4bfff700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 0x7f0c3001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:04.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.748 7f0c4bfff700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 0x7f0c3001ac10 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:04.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.777 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c0e77f0 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.780 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c34002240 con 0x7f0c4c0ad6d0 2021-02-10T11:51:04.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:04.908 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0c4c0e77f0 con 0x7f0c4c0ad6d0 2021-02-10T11:51:05.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.688 7f0c497fa700 1 -- 172.21.2.117:0/654784159 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v97) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0c34002030 con 0x7f0c4c0ad6d0 2021-02-10T11:51:05.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 msgr2=0x7f0c3001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 0x7f0c3001ac10 crc :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 msgr2=0x7f0c4c0adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7f0c3400b530 tx=0x7f0c3400b8d0).stop 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 shutdown_connections 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c4c1aded0 0x7f0c4c1ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c4c1ad4b0 0x7f0c4c1ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:05.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0c30018730 0x7f0c3001ac10 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:05.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 --2- 172.21.2.117:0/654784159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c4c0ad6d0 0x7f0c4c0adad0 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:05.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 >> 172.21.2.117:0/654784159 conn(0x7f0c4c1998b0 msgr2=0x7f0c4c1a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:05.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 shutdown_connections 2021-02-10T11:51:05.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:05.730 7f0c52c9a700 1 -- 172.21.2.117:0/654784159 wait complete. 2021-02-10T11:51:05.737 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 Processor -- start 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 -- start start 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d240e4e40 0x7f5d241b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240e5240 0x7f5d24084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d24085930 con 0x7f5d241b0810 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d24084840 con 0x7f5d240e4e40 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d2ab44700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24084a30 con 0x7f5d240e5240 2021-02-10T11:51:06.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d29161700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5d241b0810 0x7f5d24085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55462/0 (socket says 172.21.2.117:55462) 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d29161700 1 -- 172.21.2.117:0/2566281659 learned_addr learned my addr 172.21.2.117:0/2566281659 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.153 7f5d28960700 1 --2- 172.21.2.117:0/2566281659 >> v2:172.21.2.117:3300/0 conn(0x7f5d240e4e40 0x7f5d241b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d28960700 1 -- 172.21.2.117:0/2566281659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240e5240 msgr2=0x7f5d24084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d28960700 1 --2- 172.21.2.117:0/2566281659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240e5240 0x7f5d24084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d28960700 1 -- 172.21.2.117:0/2566281659 >> v1:172.21.2.117:6789/0 conn(0x7f5d241b0810 legacy=0x7f5d24085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d237fe700 1 -- 172.21.2.117:0/2566281659 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52082256 0 0) 0x7f5d1c001180 con 0x7f5d241b0810 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d28960700 1 -- 172.21.2.117:0/2566281659 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24085280 con 0x7f5d240e4e40 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d28960700 1 --2- 172.21.2.117:0/2566281659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d240e4e40 0x7f5d241b01f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7f5d0c006650 tx=0x7f5d0c009a00).ready entity=mon.0 client_cookie=b6030390a7b4c128 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d237fe700 1 -- 172.21.2.117:0/2566281659 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0c0009b0 con 0x7f5d240e4e40 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d237fe700 1 -- 172.21.2.117:0/2566281659 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c00c0c0 con 0x7f5d240e4e40 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d237fe700 1 -- 172.21.2.117:0/2566281659 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0c00c5a0 con 0x7f5d240e4e40 2021-02-10T11:51:06.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d2ab44700 1 -- 172.21.2.117:0/2566281659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d240e4e40 msgr2=0x7f5d241b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.154 7f5d2ab44700 1 --2- 172.21.2.117:0/2566281659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d240e4e40 0x7f5d241b01f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7f5d0c006650 tx=0x7f5d0c009a00).stop 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 -- 172.21.2.117:0/2566281659 shutdown_connections 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 --2- 172.21.2.117:0/2566281659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240e5240 0x7f5d24084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 --2- 172.21.2.117:0/2566281659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d240e4e40 0x7f5d241b01f0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 -- 172.21.2.117:0/2566281659 >> 172.21.2.117:0/2566281659 conn(0x7f5d240ce4e0 msgr2=0x7f5d240e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 -- 172.21.2.117:0/2566281659 shutdown_connections 2021-02-10T11:51:06.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.155 7f5d2ab44700 1 -- 172.21.2.117:0/2566281659 wait complete. 2021-02-10T11:51:06.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 Processor -- start 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 -- start start 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d24191bf0 0x7f5d24096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240973f0 0x7f5d240977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d24195290 con 0x7f5d24194e10 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240b1460 con 0x7f5d24191bf0 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d2ab44700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d240b1680 con 0x7f5d240973f0 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d28960700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d28960700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55216/0 (socket says 172.21.2.117:55216) 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.170 7f5d28960700 1 -- 172.21.2.117:0/602774504 learned_addr learned my addr 172.21.2.117:0/602774504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 -- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240973f0 msgr2=0x7f5d240977f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240973f0 0x7f5d240977f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 -- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d24191bf0 msgr2=0x7f5d24096dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d24191bf0 0x7f5d24096dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 -- 172.21.2.117:0/602774504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d240b18a0 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d28960700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7f5d0c00b640 tx=0x7f5d0c00b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0c00bdd0 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c000c90 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d0c0022b0 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d240b18a0 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.171 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d240b18a0 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.172 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d0c006a10 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.172 7f5d217fa700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 0x7f5d0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.172 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f5d0c032a10 con 0x7f5d24194e10 2021-02-10T11:51:06.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.172 7f5d23fff700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 0x7f5d0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:06.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.173 7f5d23fff700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 0x7f5d0801abc0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:06.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.202 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d24085930 con 0x7f5d24194e10 2021-02-10T11:51:06.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.205 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d0c01f5a0 con 0x7f5d24194e10 2021-02-10T11:51:06.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.330 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d24085930 con 0x7f5d24194e10 2021-02-10T11:51:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.689 7f5d217fa700 1 -- 172.21.2.117:0/602774504 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 185+0+0 (secure 0 0 0) 0x7f5d0c006a10 con 0x7f5d24194e10 2021-02-10T11:51:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:51:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 msgr2=0x7f5d0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 0x7f5d0801abc0 crc :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 msgr2=0x7f5d241915d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7f5d0c00b640 tx=0x7f5d0c00b9e0).stop 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 shutdown_connections 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d240973f0 0x7f5d240977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d24191bf0 0x7f5d24096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d080186c0 0x7f5d0801abc0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 --2- 172.21.2.117:0/602774504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d24194e10 0x7f5d241915d0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 >> 172.21.2.117:0/602774504 conn(0x7f5d2418cc70 msgr2=0x7f5d24093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:06.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 shutdown_connections 2021-02-10T11:51:06.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:06.698 7f5d2ab44700 1 -- 172.21.2.117:0/602774504 wait complete. 2021-02-10T11:51:06.706 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.124 7f7691dd8700 1 Processor -- start 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.124 7f7691dd8700 1 -- start start 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f7691dd8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0e7170 0x7f768c0e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f7691dd8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f768c0e8380 0x7f768c0e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f7691dd8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0e7570 con 0x7f768c1b07e0 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f7691dd8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0e5cd0 con 0x7f768c0e8380 2021-02-10T11:51:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f7691dd8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0e77e0 con 0x7f768c0e7170 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f768bfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f768c0e8380 0x7f768c0e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f768affd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0e7170 0x7f768c0e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f768b7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f768c1b07e0 0x7f768c0e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48400/0 (socket says 172.21.2.117:48400) 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.125 7f768b7fe700 1 -- 172.21.2.117:0/2072170407 learned_addr learned my addr 172.21.2.117:0/2072170407 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768bfff700 1 -- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0e7170 msgr2=0x7f768c0e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768bfff700 1 --2- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0e7170 0x7f768c0e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768bfff700 1 -- 172.21.2.117:0/2072170407 >> v1:172.21.2.125:6789/0 conn(0x7f768c1b07e0 legacy=0x7f768c0e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768bfff700 1 -- 172.21.2.117:0/2072170407 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0e8da0 con 0x7f768c0e8380 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768bfff700 1 --2- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0e8380 0x7f768c0e8780 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7684006e50 tx=0x7f7684009df0).ready entity=mon.1 client_cookie=4a281402f1d4a488 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768a7fc700 1 -- 172.21.2.117:0/2072170407 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f768400b510 con 0x7f768c0e8380 2021-02-10T11:51:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768a7fc700 1 -- 172.21.2.117:0/2072170407 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684006790 con 0x7f768c0e8380 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f768a7fc700 1 -- 172.21.2.117:0/2072170407 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f768400b750 con 0x7f768c0e8380 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f7691dd8700 1 -- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0e8380 msgr2=0x7f768c0e8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.126 7f7691dd8700 1 --2- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0e8380 0x7f768c0e8780 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f7684006e50 tx=0x7f7684009df0).stop 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 -- 172.21.2.117:0/2072170407 shutdown_connections 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 --2- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0e8380 0x7f768c0e8780 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 --2- 172.21.2.117:0/2072170407 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0e7170 0x7f768c0e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 -- 172.21.2.117:0/2072170407 >> 172.21.2.117:0/2072170407 conn(0x7f768c0ce4f0 msgr2=0x7f768c0e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 -- 172.21.2.117:0/2072170407 shutdown_connections 2021-02-10T11:51:07.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.127 7f7691dd8700 1 -- 172.21.2.117:0/2072170407 wait complete. 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 Processor -- start 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 -- start start 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0a09f0 0x7f768c0a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0803e0 0x7f768c195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0807e0 con 0x7f768c1a4d90 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1a51f0 con 0x7f768c0a09f0 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f7691dd8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0aeb60 con 0x7f768c0803e0 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f768affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f768affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55234/0 (socket says 172.21.2.117:55234) 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.142 7f768affd700 1 -- 172.21.2.117:0/1578423860 learned_addr learned my addr 172.21.2.117:0/1578423860 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:07.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768bfff700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0803e0 0x7f768c195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768b7fe700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0a09f0 0x7f768c0a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 -- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0803e0 msgr2=0x7f768c195530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0803e0 0x7f768c195530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 -- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0a09f0 msgr2=0x7f768c0a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0a09f0 0x7f768c0a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 -- 172.21.2.117:0/1578423860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c195b50 con 0x7f768c1a4d90 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f768affd700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7f767c006f60 tx=0x7f767c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767c00bd60 con 0x7f768c1a4d90 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c000c90 con 0x7f768c1a4d90 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c195b50 con 0x7f768c1a4d90 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c1960e0 con 0x7f768c1a4d90 2021-02-10T11:51:07.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.143 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767c002220 con 0x7f768c1a4d90 2021-02-10T11:51:07.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.144 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f767c0052e0 con 0x7f768c1a4d90 2021-02-10T11:51:07.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.144 7f7683fff700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 0x7f767001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:07.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.144 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f767c032530 con 0x7f768c1a4d90 2021-02-10T11:51:07.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.144 7f768b7fe700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 0x7f767001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:07.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.145 7f768b7fe700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 0x7f767001abc0 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:07.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.175 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c07e8b0 con 0x7f768c1a4d90 2021-02-10T11:51:07.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.177 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f767c000940 con 0x7f768c1a4d90 2021-02-10T11:51:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.305 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f768c0e77e0 con 0x7f768c1a4d90 2021-02-10T11:51:07.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.694 7f7683fff700 1 -- 172.21.2.117:0/1578423860 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v120) v1 ==== 173+0+0 (secure 0 0 0) 0x7f767c0052e0 con 0x7f768c1a4d90 2021-02-10T11:51:07.722 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:51:07.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 msgr2=0x7f767001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 0x7f767001abc0 crc :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 msgr2=0x7f768c07fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7f767c006f60 tx=0x7f767c00b970).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 shutdown_connections 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f76700186c0 0x7f767001abc0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f768c1a4d90 0x7f768c07fdc0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f768c0803e0 0x7f768c195530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 --2- 172.21.2.117:0/1578423860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f768c0a09f0 0x7f768c0a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.724 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 >> 172.21.2.117:0/1578423860 conn(0x7f768c1998a0 msgr2=0x7f768c1aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.725 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 shutdown_connections 2021-02-10T11:51:07.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:07.725 7f7691dd8700 1 -- 172.21.2.117:0/1578423860 wait complete. 2021-02-10T11:51:07.733 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:51:07.733 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:51:07.733 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:51:07.733 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:07.734 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:51:07.734 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:51:07.734 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:51:07.734 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:07.739 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:51:07.739 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:51:07.740 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:51:07.740 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:51:07.781 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:51:07.782 DEBUG:teuthology.parallel:result is None 2021-02-10T11:51:07.782 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:51:07.783 DEBUG:teuthology.parallel:result is None 2021-02-10T11:51:07.783 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:51:08.187 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:51:08.189 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 Processor -- start 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 -- start start 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde01b07f0 0x7fcde00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcde00e7d70 0x7fcde00e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde00e77f0 con 0x7fcde00e8790 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde00e92f0 con 0x7fcde00e7d70 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde84c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00e5ce0 con 0x7fcde01b07f0 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.192 7fcde5ae4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcde00e7d70 0x7fcde00e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcde00e7d70 0x7fcde00e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55254/0 (socket says 172.21.2.117:55254) 2021-02-10T11:51:08.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 -- 172.21.2.117:0/924328917 learned_addr learned my addr 172.21.2.117:0/924328917 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde62e5700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde01b07f0 0x7fcde00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 -- 172.21.2.117:0/924328917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde01b07f0 msgr2=0x7fcde00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde01b07f0 0x7fcde00e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 -- 172.21.2.117:0/924328917 >> v1:172.21.2.125:6789/0 conn(0x7fcde00e8790 legacy=0x7fcde00e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 -- 172.21.2.117:0/924328917 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00e54b0 con 0x7fcde00e7d70 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde5ae4700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00e7d70 0x7fcde00e8170 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7fcdd0006700 tx=0x7fcdd0009b80).ready entity=mon.0 client_cookie=29c2a47aac7137f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde52e3700 1 -- 172.21.2.117:0/924328917 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdd000c0c0 con 0x7fcde00e7d70 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde52e3700 1 -- 172.21.2.117:0/924328917 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000c440 con 0x7fcde00e7d70 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde52e3700 1 -- 172.21.2.117:0/924328917 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdd000c920 con 0x7fcde00e7d70 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde84c9700 1 -- 172.21.2.117:0/924328917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00e7d70 msgr2=0x7fcde00e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.193 7fcde84c9700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00e7d70 0x7fcde00e8170 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7fcdd0006700 tx=0x7fcdd0009b80).stop 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 -- 172.21.2.117:0/924328917 shutdown_connections 2021-02-10T11:51:08.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00e7d70 0x7fcde00e8170 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 --2- 172.21.2.117:0/924328917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde01b07f0 0x7fcde00e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 -- 172.21.2.117:0/924328917 >> 172.21.2.117:0/924328917 conn(0x7fcde00ce500 msgr2=0x7fcde00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:08.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 -- 172.21.2.117:0/924328917 shutdown_connections 2021-02-10T11:51:08.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.194 7fcde84c9700 1 -- 172.21.2.117:0/924328917 wait complete. 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.208 7fcde84c9700 1 Processor -- start 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 -- start start 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 0x7fcde01a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde00b15a0 0x7fcde0093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00b19a0 con 0x7fcde00b0a50 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00930b0 con 0x7fcde00a3a60 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde84c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01a3450 con 0x7fcde00b15a0 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde5ae4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde5ae4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55258/0 (socket says 172.21.2.117:55258) 2021-02-10T11:51:08.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde5ae4700 1 -- 172.21.2.117:0/4031160014 learned_addr learned my addr 172.21.2.117:0/4031160014 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde6ae6700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde00b15a0 0x7fcde0093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde62e5700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 0x7fcde01a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.209 7fcde5ae4700 1 -- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde00b15a0 msgr2=0x7fcde0093630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde5ae4700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde00b15a0 0x7fcde0093630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde5ae4700 1 -- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 msgr2=0x7fcde01a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde5ae4700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 0x7fcde01a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde5ae4700 1 -- 172.21.2.117:0/4031160014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde01a3630 con 0x7fcde00b0a50 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde62e5700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 0x7fcde01a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde5ae4700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fcdd000b650 tx=0x7fcdd000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdd00009b0 con 0x7fcde00b0a50 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0006570 con 0x7fcde00b0a50 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdd0002030 con 0x7fcde00b0a50 2021-02-10T11:51:08.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde01a3630 con 0x7fcde00b0a50 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0093fb0 con 0x7fcde00b0a50 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcdd0009f90 con 0x7fcde00b0a50 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 0x7fcdc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.210 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fcdd0032400 con 0x7fcde00b0a50 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.211 7fcde62e5700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 0x7fcdc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:08.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.211 7fcde62e5700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 0x7fcdc801abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:08.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.241 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde00b19a0 con 0x7fcde00b0a50 2021-02-10T11:51:08.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.244 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcdd00267c0 con 0x7fcde00b0a50 2021-02-10T11:51:08.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.380 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fcde00b19a0 con 0x7fcde00b0a50 2021-02-10T11:51:08.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.697 7fcdd6ffd700 1 -- 172.21.2.117:0/4031160014 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcdd00267c0 con 0x7fcde00b0a50 2021-02-10T11:51:08.747 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:51:08.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 msgr2=0x7fcdc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 0x7fcdc801abc0 crc :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 msgr2=0x7fcde01a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7fcdd000b650 tx=0x7fcdd000b9f0).stop 2021-02-10T11:51:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 shutdown_connections 2021-02-10T11:51:08.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdc80186c0 0x7fcdc801abc0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcde00b0a50 0x7fcde01a2e30 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcde00b15a0 0x7fcde0093630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 --2- 172.21.2.117:0/4031160014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcde00a3a60 0x7fcde01a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 >> 172.21.2.117:0/4031160014 conn(0x7fcde00a7260 msgr2=0x7fcde00bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 shutdown_connections 2021-02-10T11:51:08.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:08.749 7fcde84c9700 1 -- 172.21.2.117:0/4031160014 wait complete. 2021-02-10T11:51:08.757 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:51:08.757 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:51:09.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 Processor -- start 2021-02-10T11:51:09.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 -- start start 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6fdc0d18e0 0x7f6fdc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0ef3f0 0x7f6fdc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc0ede60 con 0x7f6fdc0eb000 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fdc0ef7f0 con 0x7f6fdc0d18e0 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.170 7f6fe26a1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0d1ce0 con 0x7f6fdc0ef3f0 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdb7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6fdc0eb000 0x7f6fdc0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55514/0 (socket says 172.21.2.117:55514) 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdb7fe700 1 -- 172.21.2.117:0/526493644 learned_addr learned my addr 172.21.2.117:0/526493644 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdbfff700 1 --2- 172.21.2.117:0/526493644 >> v2:172.21.2.125:3300/0 conn(0x7f6fdc0d18e0 0x7f6fdc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064859018 0 0) 0x7f6fd00009b0 con 0x7f6fdc0eb000 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fe0cbe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0ef3f0 0x7f6fdc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4002fe0 con 0x7f6fdc0eb000 2021-02-10T11:51:09.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3126163938 0 0) 0x7f6fd0001010 con 0x7f6fdc0eb000 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0ef3f0 msgr2=0x7f6fdc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0ef3f0 0x7f6fdc1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 >> v2:172.21.2.125:3300/0 conn(0x7f6fdc0d18e0 msgr2=0x7f6fdc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 --2- 172.21.2.117:0/526493644 >> v2:172.21.2.125:3300/0 conn(0x7f6fdc0d18e0 0x7f6fdc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc1b8f90 con 0x7f6fdc0eb000 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6fd0001590 con 0x7f6fdc0eb000 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 >> v1:172.21.2.117:6789/0 conn(0x7f6fdc0eb000 legacy=0x7f6fdc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fc4003a70 0x7f6fc4005f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fc404a7b0 0x7f6fc404ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.171 7f6fdaffd700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fc404d5b0 0x7f6fc404faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc40502f0 con 0x7f6fc404a7b0 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4050510 con 0x7f6fc4003a70 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4050730 con 0x7f6fc404d5b0 2021-02-10T11:51:09.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fc404a7b0 0x7f6fc404ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdbfff700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fc4003a70 0x7f6fc4005f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fe0cbe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fc404d5b0 0x7f6fc404faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 -- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fc404d5b0 msgr2=0x7f6fc404faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fc404d5b0 0x7f6fc404faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 -- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fc4003a70 msgr2=0x7f6fc4005f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fc4003a70 0x7f6fc4005f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 -- 172.21.2.117:0/526493644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40509d0 con 0x7f6fc404a7b0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdb7fe700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fc404a7b0 0x7f6fc404ccb0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f6fd00017d0 tx=0x7f6fd0007cb0).ready entity=mon.0 client_cookie=3694919b4d7564bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd00013a0 con 0x7f6fc404a7b0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000a310 con 0x7f6fc404a7b0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fdaffd700 1 -- 172.21.2.117:0/526493644 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd000a760 con 0x7f6fc404a7b0 2021-02-10T11:51:09.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fe26a1700 1 -- 172.21.2.117:0/526493644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fc404a7b0 msgr2=0x7f6fc404ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fe26a1700 1 --2- 172.21.2.117:0/526493644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fc404a7b0 0x7f6fc404ccb0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f6fd00017d0 tx=0x7f6fd0007cb0).stop 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.172 7f6fe0cbe700 1 -- 172.21.2.117:0/526493644 reap_dead start 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.173 7f6fe26a1700 1 -- 172.21.2.117:0/526493644 shutdown_connections 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.173 7f6fe26a1700 1 -- 172.21.2.117:0/526493644 >> 172.21.2.117:0/526493644 conn(0x7f6fdc0ced60 msgr2=0x7f6fdc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.173 7f6fe26a1700 1 -- 172.21.2.117:0/526493644 shutdown_connections 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.173 7f6fe26a1700 1 -- 172.21.2.117:0/526493644 wait complete. 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 Processor -- start 2021-02-10T11:51:09.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 -- start start 2021-02-10T11:51:09.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0b59a0 0x7f6fdc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fdc0b5da0 0x7f6fdc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0b61a0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc08c8b0 con 0x7f6fdc0b5da0 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe26a1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0ba1c0 con 0x7f6fdc0b59a0 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe0cbe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe0cbe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55276/0 (socket says 172.21.2.117:55276) 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.188 7f6fe0cbe700 1 -- 172.21.2.117:0/3763791088 learned_addr learned my addr 172.21.2.117:0/3763791088 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 -- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0b59a0 msgr2=0x7f6fdc08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0b59a0 0x7f6fdc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 -- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fdc0b5da0 msgr2=0x7f6fdc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fdc0b5da0 0x7f6fdc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 -- 172.21.2.117:0/3763791088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc1a3880 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe0cbe700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7f6fd400b940 tx=0x7f6fd400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd400c2f0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4006b40 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6fd400c7e0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc1a3880 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.189 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc0b2bb0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.190 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6fd400c2f0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.190 7f6fd8ff9700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 0x7f6fbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.190 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f6fd4032630 con 0x7f6fdc1a3480 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.190 7f6fdbfff700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 0x7f6fbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.191 7f6fdbfff700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 0x7f6fbc01abc0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.220 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc0ede60 con 0x7f6fdc1a3480 2021-02-10T11:51:09.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.223 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6fd40011f0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.354 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6fdc0ede60 con 0x7f6fdc1a3480 2021-02-10T11:51:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.354 7f6fd8ff9700 1 -- 172.21.2.117:0/3763791088 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v72) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f6fd400c2f0 con 0x7f6fdc1a3480 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.355 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.356 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:51:09.357 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.359 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:51:09.361 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.362 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.363 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.364 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:09.365 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.366 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:09.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:09.368 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.356 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 msgr2=0x7f6fbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.356 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 0x7f6fbc01abc0 crc :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.356 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 msgr2=0x7f6fdc0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.356 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7f6fd400b940 tx=0x7f6fd400bce0).stop 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 shutdown_connections 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6fbc0186c0 0x7f6fbc01abc0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6fdc1a3480 0x7f6fdc0b2590 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6fdc0b5da0 0x7f6fdc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 --2- 172.21.2.117:0/3763791088 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6fdc0b59a0 0x7f6fdc08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 >> 172.21.2.117:0/3763791088 conn(0x7f6fdc0bb5e0 msgr2=0x7f6fdc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 shutdown_connections 2021-02-10T11:51:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.357 7f6fe26a1700 1 -- 172.21.2.117:0/3763791088 wait complete. 2021-02-10T11:51:09.371 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:09.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.783 7f6e52f25700 1 Processor -- start 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 -- start start 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c0d18e0 0x7f6e4c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e4c0eb000 0x7f6e4c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6e4c0ee3e0 0x7f6e4c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0ede60 con 0x7f6e4c0eb000 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0eba20 con 0x7f6e4c0ee3e0 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e52f25700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0d1ce0 con 0x7f6e4c0d18e0 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e4bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e4c0eb000 0x7f6e4c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e4bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6e4c0eb000 0x7f6e4c0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55284/0 (socket says 172.21.2.117:55284) 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.784 7f6e4bfff700 1 -- 172.21.2.117:0/3741148213 learned_addr learned my addr 172.21.2.117:0/3741148213 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 -- 172.21.2.117:0/3741148213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c0d18e0 msgr2=0x7f6e4c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 --2- 172.21.2.117:0/3741148213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c0d18e0 0x7f6e4c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 -- 172.21.2.117:0/3741148213 >> v2:172.21.2.125:3300/0 conn(0x7f6e4c0ee3e0 msgr2=0x7f6e4c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 --2- 172.21.2.117:0/3741148213 >> v2:172.21.2.125:3300/0 conn(0x7f6e4c0ee3e0 0x7f6e4c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 -- 172.21.2.117:0/3741148213 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0eee00 con 0x7f6e4c0eb000 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4bfff700 1 --2- 172.21.2.117:0/3741148213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0eb000 0x7f6e4c0eb400 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7f6e3c006700 tx=0x7f6e3c009b80).ready entity=mon.0 client_cookie=69f0e9ffe76358ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4b7fe700 1 -- 172.21.2.117:0/3741148213 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c00c0c0 con 0x7f6e4c0eb000 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4b7fe700 1 -- 172.21.2.117:0/3741148213 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c00c400 con 0x7f6e4c0eb000 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e4b7fe700 1 -- 172.21.2.117:0/3741148213 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e3c00c8e0 con 0x7f6e4c0eb000 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 -- 172.21.2.117:0/3741148213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0eb000 msgr2=0x7f6e4c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 --2- 172.21.2.117:0/3741148213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0eb000 0x7f6e4c0eb400 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7f6e3c006700 tx=0x7f6e3c009b80).stop 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 -- 172.21.2.117:0/3741148213 shutdown_connections 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 --2- 172.21.2.117:0/3741148213 >> v2:172.21.2.125:3300/0 conn(0x7f6e4c0ee3e0 0x7f6e4c0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 --2- 172.21.2.117:0/3741148213 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0eb000 0x7f6e4c0eb400 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 --2- 172.21.2.117:0/3741148213 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c0d18e0 0x7f6e4c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:09.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.785 7f6e52f25700 1 -- 172.21.2.117:0/3741148213 >> 172.21.2.117:0/3741148213 conn(0x7f6e4c0ced60 msgr2=0x7f6e4c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:09.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.786 7f6e52f25700 1 -- 172.21.2.117:0/3741148213 shutdown_connections 2021-02-10T11:51:09.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.786 7f6e52f25700 1 -- 172.21.2.117:0/3741148213 wait complete. 2021-02-10T11:51:09.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.800 7f6e52f25700 1 Processor -- start 2021-02-10T11:51:09.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.800 7f6e52f25700 1 -- start start 2021-02-10T11:51:09.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e4c19c7c0 0x7f6e4c092f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c08a280 0x7f6e4c08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0a9aa0 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c19cbc0 con 0x7f6e4c19c7c0 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c08aca0 con 0x7f6e4c08a280 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55286/0 (socket says 172.21.2.117:55286) 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 -- 172.21.2.117:0/2807157506 learned_addr learned my addr 172.21.2.117:0/2807157506 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e4bfff700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c08a280 0x7f6e4c08a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e50d41700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e4c19c7c0 0x7f6e4c092f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 -- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c08a280 msgr2=0x7f6e4c08a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c08a280 0x7f6e4c08a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 -- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e4c19c7c0 msgr2=0x7f6e4c092f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e4c19c7c0 0x7f6e4c092f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 -- 172.21.2.117:0/2807157506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0e99b0 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e51542700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f6e44009980 tx=0x7f6e4400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e44005a60 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e44005ce0 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6e44006220 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.801 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c0e99b0 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.802 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c0e99b0 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.802 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6e44005870 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.802 7f6e497fa700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 0x7f6e3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:09.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.802 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f6e44032950 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.802 7f6e50d41700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 0x7f6e3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:09.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.803 7f6e50d41700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 0x7f6e3001abc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:09.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.834 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c0e8a60 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.836 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6e4400be10 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.958 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e4c0e8a60 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.959 7f6e497fa700 1 -- 172.21.2.117:0/2807157506 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f6e4400be10 con 0x7f6e4c0e8f90 2021-02-10T11:51:09.959 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:09.959 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":121,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:08.694489","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 msgr2=0x7f6e3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 0x7f6e3001abc0 crc :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 msgr2=0x7f6e4c0e9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7f6e44009980 tx=0x7f6e4400bcd0).stop 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 shutdown_connections 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6e300186c0 0x7f6e3001abc0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6e4c0e8f90 0x7f6e4c0e9390 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6e4c08a280 0x7f6e4c08a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 --2- 172.21.2.117:0/2807157506 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6e4c19c7c0 0x7f6e4c092f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.960 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 >> 172.21.2.117:0/2807157506 conn(0x7f6e4c19f610 msgr2=0x7f6e4c098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.961 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 shutdown_connections 2021-02-10T11:51:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:09.961 7f6e52f25700 1 -- 172.21.2.117:0/2807157506 wait complete. 2021-02-10T11:51:09.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:51:09.969 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:51:10.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 Processor -- start 2021-02-10T11:51:10.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 -- start start 2021-02-10T11:51:10.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b480d18f0 0x7f5b480eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5b480eb040 0x7f5b480eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5b480ee3e0 0x7f5b480ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480ede60 con 0x7f5b480eb040 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480eba60 con 0x7f5b480ee3e0 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b4eb4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480d1cf0 con 0x7f5b480d18f0 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5b480eb040 0x7f5b480eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5b480eb040 0x7f5b480eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55294/0 (socket says 172.21.2.117:55294) 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 -- 172.21.2.117:0/4165885658 learned_addr learned my addr 172.21.2.117:0/4165885658 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 -- 172.21.2.117:0/4165885658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b480d18f0 msgr2=0x7f5b480eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 --2- 172.21.2.117:0/4165885658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b480d18f0 0x7f5b480eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 -- 172.21.2.117:0/4165885658 >> v2:172.21.2.125:3300/0 conn(0x7f5b480ee3e0 msgr2=0x7f5b480ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:10.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 --2- 172.21.2.117:0/4165885658 >> v2:172.21.2.125:3300/0 conn(0x7f5b480ee3e0 0x7f5b480ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.384 7f5b47fff700 1 -- 172.21.2.117:0/4165885658 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b480eee00 con 0x7f5b480eb040 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b47fff700 1 --2- 172.21.2.117:0/4165885658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480eb040 0x7f5b480eb440 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f5b38006700 tx=0x7f5b38009b80).ready entity=mon.0 client_cookie=57af7faf271317c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b477fe700 1 -- 172.21.2.117:0/4165885658 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b3800c0c0 con 0x7f5b480eb040 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b477fe700 1 -- 172.21.2.117:0/4165885658 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3800c440 con 0x7f5b480eb040 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b477fe700 1 -- 172.21.2.117:0/4165885658 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b3800c960 con 0x7f5b480eb040 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 -- 172.21.2.117:0/4165885658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480eb040 msgr2=0x7f5b480eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 --2- 172.21.2.117:0/4165885658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480eb040 0x7f5b480eb440 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f5b38006700 tx=0x7f5b38009b80).stop 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 -- 172.21.2.117:0/4165885658 shutdown_connections 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 --2- 172.21.2.117:0/4165885658 >> v2:172.21.2.125:3300/0 conn(0x7f5b480ee3e0 0x7f5b480ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 --2- 172.21.2.117:0/4165885658 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480eb040 0x7f5b480eb440 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:10.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 --2- 172.21.2.117:0/4165885658 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b480d18f0 0x7f5b480eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.385 7f5b4eb4c700 1 -- 172.21.2.117:0/4165885658 >> 172.21.2.117:0/4165885658 conn(0x7f5b480cf060 msgr2=0x7f5b480cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:10.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.386 7f5b4eb4c700 1 -- 172.21.2.117:0/4165885658 shutdown_connections 2021-02-10T11:51:10.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.386 7f5b4eb4c700 1 -- 172.21.2.117:0/4165885658 wait complete. 2021-02-10T11:51:10.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 Processor -- start 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 -- start start 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b4807e7d0 0x7f5b4807ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b481a2e50 0x7f5b481a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4807f890 con 0x7f5b480a67e0 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4807f310 con 0x7f5b481a2e50 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4eb4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b481a3870 con 0x7f5b4807e7d0 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4c968700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:10.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.400 7f5b4c968700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55302/0 (socket says 172.21.2.117:55302) 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 -- 172.21.2.117:0/3331865430 learned_addr learned my addr 172.21.2.117:0/3331865430 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 -- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b4807e7d0 msgr2=0x7f5b4807ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b4807e7d0 0x7f5b4807ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 -- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b481a2e50 msgr2=0x7f5b481a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b481a2e50 0x7f5b481a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 -- 172.21.2.117:0/3331865430 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b481a3a90 con 0x7f5b480a67e0 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4c968700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f5b3000b5c0 tx=0x7f5b3000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:10.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b3000bd50 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b30006030 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b481a3a90 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4809f8e0 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.401 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5b300054d0 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.402 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5b30006210 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.402 7f5b457fa700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 0x7f5b2c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.402 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f5b30032620 con 0x7f5b480a67e0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.402 7f5b47fff700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 0x7f5b2c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:10.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.403 7f5b47fff700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 0x7f5b2c01abd0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:10.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.434 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b480ede60 con 0x7f5b480a67e0 2021-02-10T11:51:10.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.437 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5b30000940 con 0x7f5b480a67e0 2021-02-10T11:51:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.563 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f5b480ede60 con 0x7f5b480a67e0 2021-02-10T11:51:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.563 7f5b457fa700 1 -- 172.21.2.117:0/3331865430 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v121) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5b3000bd50 con 0x7f5b480a67e0 2021-02-10T11:51:10.564 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.565 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 msgr2=0x7f5b2c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:10.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.565 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 0x7f5b2c01abd0 crc :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.565 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 msgr2=0x7f5b480a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.565 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7f5b3000b5c0 tx=0x7f5b3000b960).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 shutdown_connections 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5b481a2e50 0x7f5b481a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5b4807e7d0 0x7f5b4807ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5b2c0186d0 0x7f5b2c01abd0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 --2- 172.21.2.117:0/3331865430 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5b480a67e0 0x7f5b480a5f80 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 >> 172.21.2.117:0/3331865430 conn(0x7f5b4809e660 msgr2=0x7f5b480bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 shutdown_connections 2021-02-10T11:51:10.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.566 7f5b4eb4c700 1 -- 172.21.2.117:0/3331865430 wait complete. 2021-02-10T11:51:10.574 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.984 7fda41306700 1 Processor -- start 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.984 7fda41306700 1 -- start start 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda41306700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 0x7fda3c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda41306700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fda3c0eb000 0x7fda3c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda41306700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0ede60 con 0x7fda3c0ee3e0 2021-02-10T11:51:11.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda41306700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c0eba20 con 0x7fda3c0eb000 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda41306700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0d1ce0 con 0x7fda3c0d18e0 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda3b87e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fda3c0ee3e0 0x7fda3c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48484/0 (socket says 172.21.2.117:48484) 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda3b87e700 1 -- 172.21.2.117:0/3298280374 learned_addr learned my addr 172.21.2.117:0/3298280374 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.985 7fda3a87c700 1 --2- 172.21.2.117:0/3298280374 >> v2:172.21.2.125:3300/0 conn(0x7fda3c0eb000 0x7fda3c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3b07d700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 0x7fda3c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3a87c700 1 -- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 msgr2=0x7fda3c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3a87c700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 0x7fda3c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3a87c700 1 -- 172.21.2.117:0/3298280374 >> v1:172.21.2.125:6789/0 conn(0x7fda3c0ee3e0 legacy=0x7fda3c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3a87c700 1 -- 172.21.2.117:0/3298280374 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0eefd0 con 0x7fda3c0eb000 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3b07d700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 0x7fda3c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.986 7fda3a87c700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0eb000 0x7fda3c0eb400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fda2c006650 tx=0x7fda2c009a80).ready entity=mon.1 client_cookie=e66a6a478f290da7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda3a07b700 1 -- 172.21.2.117:0/3298280374 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2c0009b0 con 0x7fda3c0eb000 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda3a07b700 1 -- 172.21.2.117:0/3298280374 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c00c0c0 con 0x7fda3c0eb000 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda3a07b700 1 -- 172.21.2.117:0/3298280374 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2c00c5e0 con 0x7fda3c0eb000 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 -- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0eb000 msgr2=0x7fda3c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0eb000 0x7fda3c0eb400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fda2c006650 tx=0x7fda2c009a80).stop 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 -- 172.21.2.117:0/3298280374 shutdown_connections 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0eb000 0x7fda3c0eb400 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 --2- 172.21.2.117:0/3298280374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0d18e0 0x7fda3c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 -- 172.21.2.117:0/3298280374 >> 172.21.2.117:0/3298280374 conn(0x7fda3c0ced60 msgr2=0x7fda3c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 -- 172.21.2.117:0/3298280374 shutdown_connections 2021-02-10T11:51:11.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:10.987 7fda41306700 1 -- 172.21.2.117:0/3298280374 wait complete. 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 Processor -- start 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 -- start start 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0e74d0 0x7fda3c0e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 0x7fda3c1a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c0e8430 con 0x7fda3c0e1840 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a2280 con 0x7fda3c0e74d0 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda41306700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1a9350 con 0x7fda3c0e7ef0 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda3b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda3b07d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55312/0 (socket says 172.21.2.117:55312) 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.002 7fda3b07d700 1 -- 172.21.2.117:0/3312451632 learned_addr learned my addr 172.21.2.117:0/3312451632 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b87e700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 0x7fda3c1a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3a87c700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0e74d0 0x7fda3c0e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 -- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 msgr2=0x7fda3c1a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 0x7fda3c1a8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 -- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0e74d0 msgr2=0x7fda3c0e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0e74d0 0x7fda3c0e78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 -- 172.21.2.117:0/3312451632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c1a9570 con 0x7fda3c0e1840 2021-02-10T11:51:11.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b87e700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 0x7fda3c1a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.003 7fda3b07d700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7fda24006f40 tx=0x7fda2400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda2400bd80 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24000c90 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda24002220 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda41306700 1 -- 172.21.2.117:0/3312451632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c1a9570 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda41306700 1 -- 172.21.2.117:0/3312451632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1a9570 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda2400a000 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 0x7fda2001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.004 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fda24032390 con 0x7fda3c0e1840 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.005 7fda3a87c700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 0x7fda2001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.005 7fda3a87c700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 0x7fda2001abc0 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.035 7fda41306700 1 -- 172.21.2.117:0/3312451632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c087290 con 0x7fda3c0e1840 2021-02-10T11:51:11.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.037 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda24006b00 con 0x7fda3c0e1840 2021-02-10T11:51:11.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.159 7fda41306700 1 -- 172.21.2.117:0/3312451632 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fda3c087290 con 0x7fda3c0e1840 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.160 7fda33fff700 1 -- 172.21.2.117:0/3312451632 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+64 (secure 0 0 0) 0x7fda2400bd80 con 0x7fda3c0e1840 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:51:11.160 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:11.161 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:51:11.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.161 7fda41306700 1 -- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 msgr2=0x7fda2001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.161 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 0x7fda2001abc0 crc :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.161 7fda41306700 1 -- 172.21.2.117:0/3312451632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 msgr2=0x7fda3c1a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.161 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7fda24006f40 tx=0x7fda2400b990).stop 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 -- 172.21.2.117:0/3312451632 shutdown_connections 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda3c0e7ef0 0x7fda3c1a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda3c0e74d0 0x7fda3c0e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda200186c0 0x7fda2001abc0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 --2- 172.21.2.117:0/3312451632 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda3c0e1840 0x7fda3c1a1c60 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 -- 172.21.2.117:0/3312451632 >> 172.21.2.117:0/3312451632 conn(0x7fda3c1a7900 msgr2=0x7fda3c198120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 -- 172.21.2.117:0/3312451632 shutdown_connections 2021-02-10T11:51:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.162 7fda41306700 1 -- 172.21.2.117:0/3312451632 wait complete. 2021-02-10T11:51:11.170 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:51:11.170 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.586 7f40223f6700 1 Processor -- start 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.586 7f40223f6700 1 -- start start 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c0d18f0 0x7f401c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f401c0eb040 0x7f401c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f401c0ee3e0 0x7f401c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c0ede60 con 0x7f401c0eb040 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f401c0eba60 con 0x7f401c0ee3e0 2021-02-10T11:51:11.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f40223f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0d1cf0 con 0x7f401c0d18f0 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f401c0eb040 0x7f401c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f401c0eb040 0x7f401c0eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55320/0 (socket says 172.21.2.117:55320) 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 -- 172.21.2.117:0/3519864661 learned_addr learned my addr 172.21.2.117:0/3519864661 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 -- 172.21.2.117:0/3519864661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c0d18f0 msgr2=0x7f401c0eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 --2- 172.21.2.117:0/3519864661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c0d18f0 0x7f401c0eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 -- 172.21.2.117:0/3519864661 >> v2:172.21.2.125:3300/0 conn(0x7f401c0ee3e0 msgr2=0x7f401c0ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 --2- 172.21.2.117:0/3519864661 >> v2:172.21.2.125:3300/0 conn(0x7f401c0ee3e0 0x7f401c0ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 -- 172.21.2.117:0/3519864661 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c0eee00 con 0x7f401c0eb040 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.587 7f401b7fe700 1 --2- 172.21.2.117:0/3519864661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0eb040 0x7f401c0eb440 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f4010006700 tx=0x7f4010009b00).ready entity=mon.0 client_cookie=41a8607324080430 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f401affd700 1 -- 172.21.2.117:0/3519864661 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f401000c0c0 con 0x7f401c0eb040 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f401affd700 1 -- 172.21.2.117:0/3519864661 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401000c340 con 0x7f401c0eb040 2021-02-10T11:51:11.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f401affd700 1 -- 172.21.2.117:0/3519864661 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f401000c820 con 0x7f401c0eb040 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 -- 172.21.2.117:0/3519864661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0eb040 msgr2=0x7f401c0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 --2- 172.21.2.117:0/3519864661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0eb040 0x7f401c0eb440 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f4010006700 tx=0x7f4010009b00).stop 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 -- 172.21.2.117:0/3519864661 shutdown_connections 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 --2- 172.21.2.117:0/3519864661 >> v2:172.21.2.125:3300/0 conn(0x7f401c0ee3e0 0x7f401c0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 --2- 172.21.2.117:0/3519864661 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0eb040 0x7f401c0eb440 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 --2- 172.21.2.117:0/3519864661 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c0d18f0 0x7f401c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 -- 172.21.2.117:0/3519864661 >> 172.21.2.117:0/3519864661 conn(0x7f401c0cf060 msgr2=0x7f401c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 -- 172.21.2.117:0/3519864661 shutdown_connections 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.588 7f40223f6700 1 -- 172.21.2.117:0/3519864661 wait complete. 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 Processor -- start 2021-02-10T11:51:11.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 -- start start 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c07c7c0 0x7f401c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f401c1a3930 0x7f401c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c07cca0 con 0x7f401c0a13a0 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c18fb70 con 0x7f401c1a3930 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.603 7f40223f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f401c0dc6d0 con 0x7f401c07c7c0 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55330/0 (socket says 172.21.2.117:55330) 2021-02-10T11:51:11.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 -- 172.21.2.117:0/2556692113 learned_addr learned my addr 172.21.2.117:0/2556692113 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 -- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c07c7c0 msgr2=0x7f401c1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c07c7c0 0x7f401c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 -- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f401c1a3930 msgr2=0x7f401c0a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f401c1a3930 0x7f401c0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 -- 172.21.2.117:0/2556692113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c07d220 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4020a13700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f4014009960 tx=0x7f401400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4014003030 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4014001310 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f40223f6700 1 -- 172.21.2.117:0/2556692113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c07d220 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f40223f6700 1 -- 172.21.2.117:0/2556692113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c07d7b0 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.604 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4014001830 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.605 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40140014f0 con 0x7f401c0a13a0 2021-02-10T11:51:11.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.605 7f4018ff9700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 0x7f3ffc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:11.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.605 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4014032990 con 0x7f401c0a13a0 2021-02-10T11:51:11.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.606 7f401bfff700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 0x7f3ffc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:11.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.606 7f401bfff700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 0x7f3ffc01ac10 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:11.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.636 7f40223f6700 1 -- 172.21.2.117:0/2556692113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401c18fb20 con 0x7f401c0a13a0 2021-02-10T11:51:11.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.639 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4014001ee0 con 0x7f401c0a13a0 2021-02-10T11:51:11.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:11.769 7f40223f6700 1 -- 172.21.2.117:0/2556692113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f401c18fb20 con 0x7f401c0a13a0 2021-02-10T11:51:12.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.198 7f4018ff9700 1 -- 172.21.2.117:0/2556692113 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v122) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4014000f40 con 0x7f401c0a13a0 2021-02-10T11:51:12.236 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:51:12.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 msgr2=0x7f3ffc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 0x7f3ffc01ac10 crc :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 msgr2=0x7f401c0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f4014009960 tx=0x7f401400bcf0).stop 2021-02-10T11:51:12.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 shutdown_connections 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3ffc018730 0x7f3ffc01ac10 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f401c0a13a0 0x7f401c0a17a0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f401c1a3930 0x7f401c0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 --2- 172.21.2.117:0/2556692113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f401c07c7c0 0x7f401c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 >> 172.21.2.117:0/2556692113 conn(0x7f401c087090 msgr2=0x7f401c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 shutdown_connections 2021-02-10T11:51:12.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.238 7f40223f6700 1 -- 172.21.2.117:0/2556692113 wait complete. 2021-02-10T11:51:12.246 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:51:12.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.660 7fd9dbd68700 1 Processor -- start 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.660 7fd9dbd68700 1 -- start start 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9dbd68700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9d41b07f0 0x7fd9d40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9dbd68700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 0x7fd9d40e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9dbd68700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40e77f0 con 0x7fd9d40e6b90 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9dbd68700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d40e5ce0 con 0x7fd9d41b07f0 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9dbd68700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d40e54b0 con 0x7fd9d40e6170 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9b84700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9d41b07f0 0x7fd9d40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9da385700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd9d40e6b90 0x7fd9d40e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55584/0 (socket says 172.21.2.117:55584) 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9da385700 1 -- 172.21.2.117:0/602087409 learned_addr learned my addr 172.21.2.117:0/602087409 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:12.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869086956 0 0) 0x7fd9cc001500 con 0x7fd9d40e6b90 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0002fe0 con 0x7fd9d40e6b90 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9b84700 1 -- 172.21.2.117:0/602087409 >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 msgr2=0x7fd9d40e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9383700 1 --2- 172.21.2.117:0/602087409 >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 0x7fd9d40e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9b84700 1 --2- 172.21.2.117:0/602087409 >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 0x7fd9d40e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9b84700 1 -- 172.21.2.117:0/602087409 >> v1:172.21.2.117:6789/0 conn(0x7fd9d40e6b90 legacy=0x7fd9d40e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1836266371 0 0) 0x7fd9cc001460 con 0x7fd9d40e6b90 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9b84700 1 -- 172.21.2.117:0/602087409 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40e8d80 con 0x7fd9d41b07f0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.661 7fd9d9383700 1 --2- 172.21.2.117:0/602087409 >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 0x7fd9d40e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9d9b84700 1 --2- 172.21.2.117:0/602087409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d41b07f0 0x7fd9d40e4e90 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7fd9c8006650 tx=0x7fd9c8009a80).ready entity=mon.0 client_cookie=84a13b5e57fe96c2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9c800c0c0 con 0x7fd9d41b07f0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c800c440 con 0x7fd9d41b07f0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9d8b82700 1 -- 172.21.2.117:0/602087409 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9c800c8a0 con 0x7fd9d41b07f0 2021-02-10T11:51:12.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 -- 172.21.2.117:0/602087409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d41b07f0 msgr2=0x7fd9d40e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 --2- 172.21.2.117:0/602087409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d41b07f0 0x7fd9d40e4e90 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7fd9c8006650 tx=0x7fd9c8009a80).stop 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 -- 172.21.2.117:0/602087409 shutdown_connections 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 --2- 172.21.2.117:0/602087409 >> v2:172.21.2.125:3300/0 conn(0x7fd9d40e6170 0x7fd9d40e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 --2- 172.21.2.117:0/602087409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d41b07f0 0x7fd9d40e4e90 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.662 7fd9dbd68700 1 -- 172.21.2.117:0/602087409 >> 172.21.2.117:0/602087409 conn(0x7fd9d40ce500 msgr2=0x7fd9d40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.663 7fd9dbd68700 1 -- 172.21.2.117:0/602087409 shutdown_connections 2021-02-10T11:51:12.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.663 7fd9dbd68700 1 -- 172.21.2.117:0/602087409 wait complete. 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.677 7fd9dbd68700 1 Processor -- start 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.677 7fd9dbd68700 1 -- start start 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9d40b8ac0 0x7fd9d419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9d419df40 0x7fd9d40bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40bb430 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40bc9f0 con 0x7fd9d40b8ac0 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9dbd68700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40bcc10 con 0x7fd9d419df40 2021-02-10T11:51:12.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55338/0 (socket says 172.21.2.117:55338) 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 -- 172.21.2.117:0/190145169 learned_addr learned my addr 172.21.2.117:0/190145169 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 -- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9d419df40 msgr2=0x7fd9d40bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9d419df40 0x7fd9d40bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 -- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9d40b8ac0 msgr2=0x7fd9d419d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9d40b8ac0 0x7fd9d419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 -- 172.21.2.117:0/190145169 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40b6f70 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.678 7fd9da385700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7fd9cc009980 tx=0x7fd9cc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9cc006030 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc006370 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9cc004720 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d40b6f70 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d40b6f70 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd9cc004720 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 0x7fd9bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:12.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.679 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5454+0+0 (secure 0 0 0) 0x7fd9cc032860 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.680 7fd9d9b84700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 0x7fd9bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:12.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.680 7fd9d9b84700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 0x7fd9bc01abc0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:12.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.711 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d407e800 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.714 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd9cc00be10 con 0x7fd9d40bbfd0 2021-02-10T11:51:12.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:12.845 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fd9d40e77f0 con 0x7fd9d40bbfd0 2021-02-10T11:51:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.203 7fd9c67fc700 1 -- 172.21.2.117:0/190145169 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v123) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd9cc004720 con 0x7fd9d40bbfd0 2021-02-10T11:51:13.209 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 msgr2=0x7fd9bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 0x7fd9bc01abc0 crc :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 msgr2=0x7fd9d40bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7fd9cc009980 tx=0x7fd9cc00bcd0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 shutdown_connections 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9bc0186c0 0x7fd9bc01abc0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9d40bbfd0 0x7fd9d40bc3d0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9d419df40 0x7fd9d40bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 --2- 172.21.2.117:0/190145169 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9d40b8ac0 0x7fd9d419d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.210 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 >> 172.21.2.117:0/190145169 conn(0x7fd9d4110940 msgr2=0x7fd9d407fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:13.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.211 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 shutdown_connections 2021-02-10T11:51:13.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.211 7fd9dbd68700 1 -- 172.21.2.117:0/190145169 wait complete. 2021-02-10T11:51:13.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:51:13.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 Processor -- start 2021-02-10T11:51:13.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 -- start start 2021-02-10T11:51:13.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 0x7ff9601b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff9600e5240 0x7ff960084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960085930 con 0x7ff9601b0810 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff960087430 con 0x7ff9600e5240 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff966e5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff960084840 con 0x7ff9600e4e40 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff96547a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff9601b0810 0x7ff960085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55600/0 (socket says 172.21.2.117:55600) 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff96547a700 1 -- 172.21.2.117:0/1882897345 learned_addr learned my addr 172.21.2.117:0/1882897345 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff964c79700 1 --2- 172.21.2.117:0/1882897345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 0x7ff9601b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.634 7ff957fff700 1 --2- 172.21.2.117:0/1882897345 >> v2:172.21.2.125:3300/0 conn(0x7ff9600e5240 0x7ff960084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff9577fe700 1 -- 172.21.2.117:0/1882897345 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58904774 0 0) 0x7ff958001500 con 0x7ff9601b0810 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff9577fe700 1 -- 172.21.2.117:0/1882897345 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff948002fe0 con 0x7ff9601b0810 2021-02-10T11:51:13.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff964c79700 1 -- 172.21.2.117:0/1882897345 >> v2:172.21.2.125:3300/0 conn(0x7ff9600e5240 msgr2=0x7ff960084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff964c79700 1 --2- 172.21.2.117:0/1882897345 >> v2:172.21.2.125:3300/0 conn(0x7ff9600e5240 0x7ff960084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff964c79700 1 -- 172.21.2.117:0/1882897345 >> v1:172.21.2.117:6789/0 conn(0x7ff9601b0810 legacy=0x7ff960085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff964c79700 1 -- 172.21.2.117:0/1882897345 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600e5cc0 con 0x7ff9600e4e40 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff964c79700 1 --2- 172.21.2.117:0/1882897345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 0x7ff9601b01f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff950006650 tx=0x7ff950009a80).ready entity=mon.2 client_cookie=9e2188bcdeacab61 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff9577fe700 1 -- 172.21.2.117:0/1882897345 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9500009b0 con 0x7ff9600e4e40 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff9577fe700 1 -- 172.21.2.117:0/1882897345 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95000c0c0 con 0x7ff9600e4e40 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff9577fe700 1 -- 172.21.2.117:0/1882897345 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff95000c510 con 0x7ff9600e4e40 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff966e5d700 1 -- 172.21.2.117:0/1882897345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 msgr2=0x7ff9601b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.635 7ff966e5d700 1 --2- 172.21.2.117:0/1882897345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 0x7ff9601b01f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff950006650 tx=0x7ff950009a80).stop 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 -- 172.21.2.117:0/1882897345 shutdown_connections 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 --2- 172.21.2.117:0/1882897345 >> v2:172.21.2.125:3300/0 conn(0x7ff9600e5240 0x7ff960084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 --2- 172.21.2.117:0/1882897345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9600e4e40 0x7ff9601b01f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 -- 172.21.2.117:0/1882897345 >> 172.21.2.117:0/1882897345 conn(0x7ff9600ce4e0 msgr2=0x7ff9600e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 -- 172.21.2.117:0/1882897345 shutdown_connections 2021-02-10T11:51:13.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.636 7ff966e5d700 1 -- 172.21.2.117:0/1882897345 wait complete. 2021-02-10T11:51:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 Processor -- start 2021-02-10T11:51:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 -- start start 2021-02-10T11:51:13.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 0x7ff960110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 0x7ff96010f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601101e0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96010dfe0 con 0x7ff9600bb8c0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff966e5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff960110d80 con 0x7ff96010dad0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff964c79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff964c79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55356/0 (socket says 172.21.2.117:55356) 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.651 7ff964c79700 1 -- 172.21.2.117:0/3687742874 learned_addr learned my addr 172.21.2.117:0/3687742874 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 -- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 msgr2=0x7ff960110760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff957fff700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 0x7ff960110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff96547a700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 0x7ff96010f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 0x7ff960110760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 -- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 msgr2=0x7ff96010f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 0x7ff96010f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 -- 172.21.2.117:0/3687742874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960110fa0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff957fff700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 0x7ff960110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff96547a700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 0x7ff96010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff964c79700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7ff950009630 tx=0x7ff95000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9500093a0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95000bd70 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff960110fa0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff950005560 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.652 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601101e0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.653 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff950005740 con 0x7ff9600bb4c0 2021-02-10T11:51:13.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.653 7ff9557fa700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 0x7ff94001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.653 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ff9500328e0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.653 7ff957fff700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 0x7ff94001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:13.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.654 7ff957fff700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 0x7ff94001abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:13.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.684 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9601101e0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.686 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff950005510 con 0x7ff9600bb4c0 2021-02-10T11:51:13.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.816 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7ff9601101e0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.819 7ff9557fa700 1 -- 172.21.2.117:0/3687742874 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 26 and data pool 27 v99) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff950005ad0 con 0x7ff9600bb4c0 2021-02-10T11:51:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 26 and data pool 27 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 msgr2=0x7ff94001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 0x7ff94001abc0 crc :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 msgr2=0x7ff96010d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7ff950009630 tx=0x7ff95000b980).stop 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 shutdown_connections 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9600bb8c0 0x7ff96010f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff96010dad0 0x7ff960110760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9400186c0 0x7ff94001abc0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 --2- 172.21.2.117:0/3687742874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9600bb4c0 0x7ff96010d4b0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:13.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 >> 172.21.2.117:0/3687742874 conn(0x7ff96009e8b0 msgr2=0x7ff96010c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:13.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 shutdown_connections 2021-02-10T11:51:13.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:13.825 7ff966e5d700 1 -- 172.21.2.117:0/3687742874 wait complete. 2021-02-10T11:51:13.833 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:14.256 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:14.258 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:14.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 Processor -- start 2021-02-10T11:51:14.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 -- start start 2021-02-10T11:51:14.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c085930 con 0x7f336c0e4e40 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c085680 con 0x7f336c084c60 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f3370a4f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c1b09e0 con 0x7f336c0e5240 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f336a05d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f336b05f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f336a05d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53044/0 (socket says 172.21.2.117:53044) 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f336b05f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:59750/0 (socket says 172.21.2.117:59750) 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 -- 172.21.2.117:0/1974852736 learned_addr learned my addr 172.21.2.117:0/1974852736 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.245 7f336a85e700 1 --1- 172.21.2.117:0/1974852736 >> v1:172.21.2.125:6789/0 conn(0x7f336c0e4e40 0x7f336c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48538/0 (socket says 172.21.2.117:48538) 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 -- 172.21.2.117:0/1974852736 >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 msgr2=0x7f336c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:14.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 --2- 172.21.2.117:0/1974852736 >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 -- 172.21.2.117:0/1974852736 >> v1:172.21.2.125:6789/0 conn(0x7f336c0e4e40 legacy=0x7f336c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 -- 172.21.2.117:0/1974852736 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c084840 con 0x7f336c0e5240 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336b05f700 1 --2- 172.21.2.117:0/1974852736 >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.246 7f336a05d700 1 --2- 172.21.2.117:0/1974852736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f335c006700 tx=0x7f335c009b80).ready entity=mon.2 client_cookie=dc798b1eed56869a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f336985c700 1 -- 172.21.2.117:0/1974852736 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00c0c0 con 0x7f336c0e5240 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f336985c700 1 -- 172.21.2.117:0/1974852736 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00c340 con 0x7f336c0e5240 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f336985c700 1 -- 172.21.2.117:0/1974852736 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00c860 con 0x7f336c0e5240 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 -- 172.21.2.117:0/1974852736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 msgr2=0x7f336c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 --2- 172.21.2.117:0/1974852736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f335c006700 tx=0x7f335c009b80).stop 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 -- 172.21.2.117:0/1974852736 shutdown_connections 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 --2- 172.21.2.117:0/1974852736 >> v2:172.21.2.125:3300/0 conn(0x7f336c084c60 0x7f336c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 --2- 172.21.2.117:0/1974852736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c0e5240 0x7f336c086ec0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 -- 172.21.2.117:0/1974852736 >> 172.21.2.117:0/1974852736 conn(0x7f336c0ce4e0 msgr2=0x7f336c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:14.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 -- 172.21.2.117:0/1974852736 shutdown_connections 2021-02-10T11:51:14.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.247 7f3370a4f700 1 -- 172.21.2.117:0/1974852736 wait complete. 2021-02-10T11:51:14.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 Processor -- start 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 -- start start 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c10db80 0x7f336c190500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f336c1a7cb0 0x7f336c08e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c08e090 con 0x7f336c1a7290 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c10df80 con 0x7f336c1a7cb0 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f3370a4f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c190b20 con 0x7f336c10db80 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f336a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f336a05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55368/0 (socket says 172.21.2.117:55368) 2021-02-10T11:51:14.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.262 7f336a05d700 1 -- 172.21.2.117:0/2828170851 learned_addr learned my addr 172.21.2.117:0/2828170851 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a85e700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c10db80 0x7f336c190500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336b05f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f336c1a7cb0 0x7f336c08e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 -- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c10db80 msgr2=0x7f336c190500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c10db80 0x7f336c190500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 -- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f336c1a7cb0 msgr2=0x7f336c08e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f336c1a7cb0 0x7f336c08e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 -- 172.21.2.117:0/2828170851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c08cfc0 con 0x7f336c1a7290 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f336a05d700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f335c00b650 tx=0x7f335c00b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c00bde0 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c006390 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f335c002220 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c08cfc0 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.263 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c08cfc0 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.264 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f335c000c90 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.264 7f33637fe700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 0x7f335001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.264 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f335c032940 con 0x7f336c1a7290 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.264 7f336a85e700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 0x7f335001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.264 7f336a85e700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 0x7f335001abc0 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.295 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c085930 con 0x7f336c1a7290 2021-02-10T11:51:14.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.298 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f335c006390 con 0x7f336c1a7290 2021-02-10T11:51:14.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.424 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f336c085930 con 0x7f336c1a7290 2021-02-10T11:51:14.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.425 7f33637fe700 1 -- 172.21.2.117:0/2828170851 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f335c000c90 con 0x7f336c1a7290 2021-02-10T11:51:14.425 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:14.426 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":124,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:13.818047","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:51:11.788181","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:51:12.863682","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 msgr2=0x7f335001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 0x7f335001abc0 crc :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 msgr2=0x7f336c1a7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f335c00b650 tx=0x7f335c00b9f0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 shutdown_connections 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3350018730 0x7f335001abc0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f336c1a7290 0x7f336c1a7690 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f336c1a7cb0 0x7f336c08e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 --2- 172.21.2.117:0/2828170851 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f336c10db80 0x7f336c190500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 >> 172.21.2.117:0/2828170851 conn(0x7f336c09e890 msgr2=0x7f336c08d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:14.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 shutdown_connections 2021-02-10T11:51:14.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.427 7f3370a4f700 1 -- 172.21.2.117:0/2828170851 wait complete. 2021-02-10T11:51:14.436 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:14.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 Processor -- start 2021-02-10T11:51:14.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 -- start start 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 0x7f63800ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f63800eb000 0x7f63800eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63800ee3e0 0x7f63800ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63800ede60 con 0x7f63800ee3e0 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63800eba20 con 0x7f63800eb000 2021-02-10T11:51:14.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f6385105700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63800d1ce0 con 0x7f63800d18e0 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f637f59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63800ee3e0 0x7f63800ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f637f59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63800ee3e0 0x7f63800ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55380/0 (socket says 172.21.2.117:55380) 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.889 7f637f59e700 1 -- 172.21.2.117:0/780760332 learned_addr learned my addr 172.21.2.117:0/780760332 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637ed9d700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 0x7f63800ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637e59c700 1 --2- 172.21.2.117:0/780760332 >> v2:172.21.2.125:3300/0 conn(0x7f63800eb000 0x7f63800eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 -- 172.21.2.117:0/780760332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 msgr2=0x7f63800ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:14.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 0x7f63800ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 -- 172.21.2.117:0/780760332 >> v2:172.21.2.125:3300/0 conn(0x7f63800eb000 msgr2=0x7f63800eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 --2- 172.21.2.117:0/780760332 >> v2:172.21.2.125:3300/0 conn(0x7f63800eb000 0x7f63800eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 -- 172.21.2.117:0/780760332 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800eee00 con 0x7f63800ee3e0 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637ed9d700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 0x7f63800ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637f59e700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800ee3e0 0x7f63800ee7e0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f6378006f00 tx=0x7f6378009f10).ready entity=mon.0 client_cookie=a1c1248c978ae7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637dd9b700 1 -- 172.21.2.117:0/780760332 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637800b3b0 con 0x7f63800ee3e0 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637dd9b700 1 -- 172.21.2.117:0/780760332 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637800b630 con 0x7f63800ee3e0 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f637dd9b700 1 -- 172.21.2.117:0/780760332 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637800bb50 con 0x7f63800ee3e0 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f6385105700 1 -- 172.21.2.117:0/780760332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800ee3e0 msgr2=0x7f63800ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.890 7f6385105700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800ee3e0 0x7f63800ee7e0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7f6378006f00 tx=0x7f6378009f10).stop 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 -- 172.21.2.117:0/780760332 shutdown_connections 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 --2- 172.21.2.117:0/780760332 >> v2:172.21.2.125:3300/0 conn(0x7f63800eb000 0x7f63800eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800ee3e0 0x7f63800ee7e0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 --2- 172.21.2.117:0/780760332 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800d18e0 0x7f63800ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 -- 172.21.2.117:0/780760332 >> 172.21.2.117:0/780760332 conn(0x7f63800ced60 msgr2=0x7f63800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 -- 172.21.2.117:0/780760332 shutdown_connections 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.891 7f6385105700 1 -- 172.21.2.117:0/780760332 wait complete. 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 Processor -- start 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 -- start start 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63800b77e0 0x7f63800b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800ab2c0 0x7f63800abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800ab8c0 con 0x7f63800b7310 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800ac460 con 0x7f63800b77e0 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f6385105700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63800ac680 con 0x7f63800ab2c0 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f637e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f637e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55382/0 (socket says 172.21.2.117:55382) 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f637e59c700 1 -- 172.21.2.117:0/1421436381 learned_addr learned my addr 172.21.2.117:0/1421436381 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.906 7f637e59c700 1 -- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800ab2c0 msgr2=0x7f63800abe40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f637e59c700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800ab2c0 0x7f63800abe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f637e59c700 1 -- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63800b77e0 msgr2=0x7f63800b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f637e59c700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63800b77e0 0x7f63800b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f637e59c700 1 -- 172.21.2.117:0/1421436381 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800ac8a0 con 0x7f63800b7310 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f637e59c700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f6370006f10 tx=0x7f637000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637000bd80 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370000c90 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f6385105700 1 -- 172.21.2.117:0/1421436381 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63800ac8a0 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6370002220 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f6385105700 1 -- 172.21.2.117:0/1421436381 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63800ac8a0 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.907 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63700052e0 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.908 7f63777fe700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 0x7f636401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.908 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6370032660 con 0x7f63800b7310 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.908 7f637ed9d700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 0x7f636401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:14.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.908 7f637ed9d700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 0x7f636401abc0 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:14.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.939 7f6385105700 1 -- 172.21.2.117:0/1421436381 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6380087290 con 0x7f63800b7310 2021-02-10T11:51:14.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:14.942 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6370006ba0 con 0x7f63800b7310 2021-02-10T11:51:15.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.063 7f6385105700 1 -- 172.21.2.117:0/1421436381 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63800ede60 con 0x7f63800b7310 2021-02-10T11:51:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.064 7f63777fe700 1 -- 172.21.2.117:0/1421436381 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f63700052e0 con 0x7f63800b7310 2021-02-10T11:51:15.064 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:15.065 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":125,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:14.819978","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:51:11.788181","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:51:12.863682","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 msgr2=0x7f636401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 0x7f636401abc0 crc :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 msgr2=0x7f63800aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7f6370006f10 tx=0x7f637000b990).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 shutdown_connections 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63640186c0 0x7f636401abc0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63800b7310 0x7f63800aaca0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63800ab2c0 0x7f63800abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 --2- 172.21.2.117:0/1421436381 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63800b77e0 0x7f63800b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:15.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 >> 172.21.2.117:0/1421436381 conn(0x7f63800874c0 msgr2=0x7f63800af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 shutdown_connections 2021-02-10T11:51:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.066 7f6385105700 1 -- 172.21.2.117:0/1421436381 wait complete. 2021-02-10T11:51:15.075 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 Processor -- start 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 -- start start 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef980d18f0 0x7fef980eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef980eb040 0x7fef980eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef980ede60 con 0x7fef980ee3e0 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef980eba60 con 0x7fef980eb040 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.492 7fef9da91700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef980d1cf0 con 0x7fef980d18f0 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef980eb040 0x7fef980eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef980eb040 0x7fef980eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55390/0 (socket says 172.21.2.117:55390) 2021-02-10T11:51:15.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 -- 172.21.2.117:0/1524275018 learned_addr learned my addr 172.21.2.117:0/1524275018 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 -- 172.21.2.117:0/1524275018 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef980d18f0 msgr2=0x7fef980eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 --2- 172.21.2.117:0/1524275018 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef980d18f0 0x7fef980eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 -- 172.21.2.117:0/1524275018 >> v1:172.21.2.125:6789/0 conn(0x7fef980ee3e0 legacy=0x7fef980ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 -- 172.21.2.117:0/1524275018 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980eefd0 con 0x7fef980eb040 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef96ffd700 1 --2- 172.21.2.117:0/1524275018 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef980eb040 0x7fef980eb440 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7fef80006700 tx=0x7fef80009b80).ready entity=mon.0 client_cookie=aa33713685d05907 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef967fc700 1 -- 172.21.2.117:0/1524275018 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef8000c0c0 con 0x7fef980eb040 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.493 7fef967fc700 1 -- 172.21.2.117:0/1524275018 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8000c440 con 0x7fef980eb040 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef967fc700 1 -- 172.21.2.117:0/1524275018 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef8000c960 con 0x7fef980eb040 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 -- 172.21.2.117:0/1524275018 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef980eb040 msgr2=0x7fef980eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 --2- 172.21.2.117:0/1524275018 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef980eb040 0x7fef980eb440 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7fef80006700 tx=0x7fef80009b80).stop 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 -- 172.21.2.117:0/1524275018 shutdown_connections 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 --2- 172.21.2.117:0/1524275018 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef980eb040 0x7fef980eb440 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 --2- 172.21.2.117:0/1524275018 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef980d18f0 0x7fef980eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:15.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 -- 172.21.2.117:0/1524275018 >> 172.21.2.117:0/1524275018 conn(0x7fef980cf060 msgr2=0x7fef980cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:15.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 -- 172.21.2.117:0/1524275018 shutdown_connections 2021-02-10T11:51:15.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.494 7fef9da91700 1 -- 172.21.2.117:0/1524275018 wait complete. 2021-02-10T11:51:15.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 Processor -- start 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 -- start start 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef981a4240 0x7fef981a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef9809ed70 0x7fef9809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9809fb90 con 0x7fef9809f790 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980dc640 con 0x7fef981a4240 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.509 7fef9da91700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef98097270 con 0x7fef9809ed70 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55398/0 (socket says 172.21.2.117:55398) 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 -- 172.21.2.117:0/3389844659 learned_addr learned my addr 172.21.2.117:0/3389844659 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef96ffd700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef9809ed70 0x7fef9809f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:15.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef977fe700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef981a4240 0x7fef981a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 -- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef9809ed70 msgr2=0x7fef9809f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef9809ed70 0x7fef9809f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 -- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef981a4240 msgr2=0x7fef981a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef981a4240 0x7fef981a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 -- 172.21.2.117:0/3389844659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980b8780 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef97fff700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7fef90009940 tx=0x7fef9000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.510 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef90000f90 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9000c2f0 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef9da91700 1 -- 172.21.2.117:0/3389844659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef980b8780 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef9da91700 1 -- 172.21.2.117:0/3389844659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef980b8780 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef9000c750 con 0x7fef9809f790 2021-02-10T11:51:15.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef90001220 con 0x7fef9809f790 2021-02-10T11:51:15.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef8ffff700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 0x7fef7c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:15.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.511 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef90032b20 con 0x7fef9809f790 2021-02-10T11:51:15.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.512 7fef977fe700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 0x7fef7c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:15.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.512 7fef977fe700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 0x7fef7c01ac10 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:15.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.542 7fef9da91700 1 -- 172.21.2.117:0/3389844659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef980daff0 con 0x7fef9809f790 2021-02-10T11:51:15.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.545 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef90000d30 con 0x7fef9809f790 2021-02-10T11:51:15.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.677 7fef9da91700 1 -- 172.21.2.117:0/3389844659 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fef980ede60 con 0x7fef9809f790 2021-02-10T11:51:15.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.828 7fef8ffff700 1 -- 172.21.2.117:0/3389844659 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v102) v1 ==== 118+0+0 (secure 0 0 0) 0x7fef90001220 con 0x7fef9809f790 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.844 7fef9da91700 1 -- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 msgr2=0x7fef7c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.844 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 0x7fef7c01ac10 crc :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.844 7fef9da91700 1 -- 172.21.2.117:0/3389844659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 msgr2=0x7fef980b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.844 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7fef90009940 tx=0x7fef9000bcf0).stop 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 -- 172.21.2.117:0/3389844659 shutdown_connections 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fef7c018730 0x7fef7c01ac10 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef9809f790 0x7fef980b8160 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef9809ed70 0x7fef9809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 --2- 172.21.2.117:0/3389844659 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef981a4240 0x7fef981a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 -- 172.21.2.117:0/3389844659 >> 172.21.2.117:0/3389844659 conn(0x7fef98087090 msgr2=0x7fef9808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 -- 172.21.2.117:0/3389844659 shutdown_connections 2021-02-10T11:51:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:15.845 7fef9da91700 1 -- 172.21.2.117:0/3389844659 wait complete. 2021-02-10T11:51:15.852 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:16.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.265 7f23950dc700 1 Processor -- start 2021-02-10T11:51:16.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.265 7f23950dc700 1 -- start start 2021-02-10T11:51:16.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f23900d18e0 0x7f23900ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900eb000 0x7f23900eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23900ee3e0 0x7f23900ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23900ede60 con 0x7f23900ee3e0 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23900eba20 con 0x7f23900d18e0 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23900d1ce0 con 0x7f23900eb000 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23900ee3e0 0x7f23900ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f23900ee3e0 0x7f23900ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55406/0 (socket says 172.21.2.117:55406) 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 -- 172.21.2.117:0/2074289013 learned_addr learned my addr 172.21.2.117:0/2074289013 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 -- 172.21.2.117:0/2074289013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900eb000 msgr2=0x7f23900eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 --2- 172.21.2.117:0/2074289013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900eb000 0x7f23900eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 -- 172.21.2.117:0/2074289013 >> v2:172.21.2.125:3300/0 conn(0x7f23900d18e0 msgr2=0x7f23900ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 --2- 172.21.2.117:0/2074289013 >> v2:172.21.2.125:3300/0 conn(0x7f23900d18e0 0x7f23900ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 -- 172.21.2.117:0/2074289013 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23900eee00 con 0x7f23900ee3e0 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238f59e700 1 --2- 172.21.2.117:0/2074289013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23900ee3e0 0x7f23900ee7e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f2388006f00 tx=0x7f2388009f10).ready entity=mon.0 client_cookie=f223eab25f7ef99c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238dd9b700 1 -- 172.21.2.117:0/2074289013 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f238800b3b0 con 0x7f23900ee3e0 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238dd9b700 1 -- 172.21.2.117:0/2074289013 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238800b630 con 0x7f23900ee3e0 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f238dd9b700 1 -- 172.21.2.117:0/2074289013 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f238800bb50 con 0x7f23900ee3e0 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 -- 172.21.2.117:0/2074289013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23900ee3e0 msgr2=0x7f23900ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.266 7f23950dc700 1 --2- 172.21.2.117:0/2074289013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23900ee3e0 0x7f23900ee7e0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7f2388006f00 tx=0x7f2388009f10).stop 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 -- 172.21.2.117:0/2074289013 shutdown_connections 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 --2- 172.21.2.117:0/2074289013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f23900ee3e0 0x7f23900ee7e0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 --2- 172.21.2.117:0/2074289013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900eb000 0x7f23900eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 --2- 172.21.2.117:0/2074289013 >> v2:172.21.2.125:3300/0 conn(0x7f23900d18e0 0x7f23900ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 -- 172.21.2.117:0/2074289013 >> 172.21.2.117:0/2074289013 conn(0x7f23900ced60 msgr2=0x7f23900cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 -- 172.21.2.117:0/2074289013 shutdown_connections 2021-02-10T11:51:16.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.267 7f23950dc700 1 -- 172.21.2.117:0/2074289013 wait complete. 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.281 7f23950dc700 1 Processor -- start 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.281 7f23950dc700 1 -- start start 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2390093070 0x7f239019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900b3ab0 0x7f23900b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23900e3700 con 0x7f239019b660 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23900e2e20 con 0x7f2390093070 2021-02-10T11:51:16.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f23950dc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23900e3000 con 0x7f23900b3ab0 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55412/0 (socket says 172.21.2.117:55412) 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 -- 172.21.2.117:0/3183154504 learned_addr learned my addr 172.21.2.117:0/3183154504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 -- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900b3ab0 msgr2=0x7f23900b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900b3ab0 0x7f23900b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 -- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2390093070 msgr2=0x7f239019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2390093070 0x7f239019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.282 7f238e59c700 1 -- 172.21.2.117:0/3183154504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23900b44d0 con 0x7f239019b660 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f238e59c700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f238000b5c0 tx=0x7f238000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2380006b80 con 0x7f239019b660 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238000bd50 con 0x7f239019b660 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23950dc700 1 -- 172.21.2.117:0/3183154504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23900b44d0 con 0x7f239019b660 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23950dc700 1 -- 172.21.2.117:0/3183154504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23900b44d0 con 0x7f239019b660 2021-02-10T11:51:16.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f23800054d0 con 0x7f239019b660 2021-02-10T11:51:16.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2380006240 con 0x7f239019b660 2021-02-10T11:51:16.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 0x7f237401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.283 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2380032930 con 0x7f239019b660 2021-02-10T11:51:16.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.284 7f238ed9d700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 0x7f237401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.284 7f238ed9d700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 0x7f237401ac10 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.315 7f23950dc700 1 -- 172.21.2.117:0/3183154504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2390087210 con 0x7f239019b660 2021-02-10T11:51:16.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.318 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2380006030 con 0x7f239019b660 2021-02-10T11:51:16.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.445 7f23950dc700 1 -- 172.21.2.117:0/3183154504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23900ede60 con 0x7f239019b660 2021-02-10T11:51:16.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.445 7f23877fe700 1 -- 172.21.2.117:0/3183154504 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f238000bd50 con 0x7f239019b660 2021-02-10T11:51:16.445 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:16.445 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":102,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:15.824496","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11124":{"gid":11124,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1123044872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1123044872},{"type":"v1","addr":"172.21.2.125:6835","nonce":1123044872}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:16.445 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 102 2021-02-10T11:51:16.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 msgr2=0x7f237401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 0x7f237401ac10 crc :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 msgr2=0x7f239019ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7f238000b5c0 tx=0x7f238000b960).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 shutdown_connections 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2374018730 0x7f237401ac10 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f239019b660 0x7f239019ba60 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23900b3ab0 0x7f23900b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 --2- 172.21.2.117:0/3183154504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2390093070 0x7f239019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 >> 172.21.2.117:0/3183154504 conn(0x7f239019f610 msgr2=0x7f2390098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 shutdown_connections 2021-02-10T11:51:16.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.447 7f23950dc700 1 -- 172.21.2.117:0/3183154504 wait complete. 2021-02-10T11:51:16.455 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.867 7f983a3c9700 1 Processor -- start 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.867 7f983a3c9700 1 -- start start 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f983a3c9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f98340d18e0 0x7f98340ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f983a3c9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f98340eb000 0x7f98340eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f983a3c9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340ede60 con 0x7f98340ee3e0 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f983a3c9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98340eba20 con 0x7f98340eb000 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f983a3c9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98340d1ce0 con 0x7f98340d18e0 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98337fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f98340eb000 0x7f98340eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98389e6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f98340ee3e0 0x7f98340ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55666/0 (socket says 172.21.2.117:55666) 2021-02-10T11:51:16.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98389e6700 1 -- 172.21.2.117:0/2526877594 learned_addr learned my addr 172.21.2.117:0/2526877594 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f9833fff700 1 --2- 172.21.2.117:0/2526877594 >> v2:172.21.2.125:3300/0 conn(0x7f98340d18e0 0x7f98340ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98337fe700 1 -- 172.21.2.117:0/2526877594 >> v2:172.21.2.125:3300/0 conn(0x7f98340d18e0 msgr2=0x7f98340ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98337fe700 1 --2- 172.21.2.117:0/2526877594 >> v2:172.21.2.125:3300/0 conn(0x7f98340d18e0 0x7f98340ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98337fe700 1 -- 172.21.2.117:0/2526877594 >> v1:172.21.2.117:6789/0 conn(0x7f98340ee3e0 legacy=0x7f98340ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.868 7f98337fe700 1 -- 172.21.2.117:0/2526877594 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340eefd0 con 0x7f98340eb000 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f98337fe700 1 --2- 172.21.2.117:0/2526877594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340eb000 0x7f98340eb400 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f98280069d0 tx=0x7f9828009dc0).ready entity=mon.0 client_cookie=a3c8e737a666f1bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f9832ffd700 1 -- 172.21.2.117:0/2526877594 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f982800b510 con 0x7f98340eb000 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f9832ffd700 1 -- 172.21.2.117:0/2526877594 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982800b890 con 0x7f98340eb000 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f9832ffd700 1 -- 172.21.2.117:0/2526877594 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f982800bd70 con 0x7f98340eb000 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 -- 172.21.2.117:0/2526877594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340eb000 msgr2=0x7f98340eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 --2- 172.21.2.117:0/2526877594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340eb000 0x7f98340eb400 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f98280069d0 tx=0x7f9828009dc0).stop 2021-02-10T11:51:16.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 -- 172.21.2.117:0/2526877594 shutdown_connections 2021-02-10T11:51:16.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 --2- 172.21.2.117:0/2526877594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340eb000 0x7f98340eb400 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 --2- 172.21.2.117:0/2526877594 >> v2:172.21.2.125:3300/0 conn(0x7f98340d18e0 0x7f98340ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 -- 172.21.2.117:0/2526877594 >> 172.21.2.117:0/2526877594 conn(0x7f98340ced60 msgr2=0x7f98340cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:16.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 -- 172.21.2.117:0/2526877594 shutdown_connections 2021-02-10T11:51:16.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.869 7f983a3c9700 1 -- 172.21.2.117:0/2526877594 wait complete. 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.884 7f983a3c9700 1 Processor -- start 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.884 7f983a3c9700 1 -- start start 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.884 7f983a3c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 0x7f98341a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.884 7f983a3c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f983a3c9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98340e7ef0 0x7f98341a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f983a3c9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98340e8430 con 0x7f98340e74d0 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f983a3c9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98341a2280 con 0x7f98340e7ef0 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f983a3c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98341a9350 con 0x7f98340e1840 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f98337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 0x7f98341a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f98337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55420/0 (socket says 172.21.2.117:55420) 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f98337fe700 1 -- 172.21.2.117:0/2684616166 learned_addr learned my addr 172.21.2.117:0/2684616166 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f98389e6700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98340e7ef0 0x7f98341a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 -- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98340e7ef0 msgr2=0x7f98341a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98340e7ef0 0x7f98341a8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 -- 172.21.2.117:0/2684616166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 msgr2=0x7f98340e78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 -- 172.21.2.117:0/2684616166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98341a9570 con 0x7f98340e1840 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f98337fe700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.885 7f9833fff700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 0x7f98341a1c60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9824009540 tx=0x7f982400b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f98240063c0 con 0x7f98340e1840 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982400bd60 con 0x7f98340e1840 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9824002030 con 0x7f98340e1840 2021-02-10T11:51:16.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98341a9570 con 0x7f98340e1840 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98341a9570 con 0x7f98340e1840 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9824000c90 con 0x7f98340e1840 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.886 7f9830ff9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 0x7f981401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.887 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9824032850 con 0x7f98340e1840 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.887 7f98337fe700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 0x7f981401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:16.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.887 7f98337fe700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 0x7f981401abc0 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:16.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.917 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9834087290 con 0x7f98340e1840 2021-02-10T11:51:16.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:16.920 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9824006ae0 con 0x7f98340e1840 2021-02-10T11:51:17.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.050 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98340ede60 con 0x7f98340e1840 2021-02-10T11:51:17.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.050 7f9830ff9700 1 -- 172.21.2.117:0/2684616166 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f9824006740 con 0x7f98340e1840 2021-02-10T11:51:17.050 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:17.050 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":125,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:14.819978","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:51:11.788181","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:51:12.863682","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:17.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 msgr2=0x7f981401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 0x7f981401abc0 crc :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 msgr2=0x7f98341a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 0x7f98341a1c60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9824009540 tx=0x7f982400b8e0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 shutdown_connections 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f98140186c0 0x7f981401abc0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f98340e74d0 0x7f98340e78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f98340e7ef0 0x7f98341a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 --2- 172.21.2.117:0/2684616166 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f98340e1840 0x7f98341a1c60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.052 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 >> 172.21.2.117:0/2684616166 conn(0x7f98341a7900 msgr2=0x7f9834198120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.053 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 shutdown_connections 2021-02-10T11:51:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.053 7f983a3c9700 1 -- 172.21.2.117:0/2684616166 wait complete. 2021-02-10T11:51:17.060 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:51:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.486 7fdecfb85700 1 Processor -- start 2021-02-10T11:51:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.486 7fdecfb85700 1 -- start start 2021-02-10T11:51:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecfb85700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdec8079ce0 0x7fdec81b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecfb85700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec807a5c0 con 0x7fdec807ab40 2021-02-10T11:51:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecfb85700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec80d18c0 con 0x7fdec807a0e0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecfb85700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec807bf50 con 0x7fdec8079ce0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecd9a1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdec807ab40 0x7fdec80794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55678/0 (socket says 172.21.2.117:55678) 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecd1a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdec8079ce0 0x7fdec81b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecd9a1700 1 -- 172.21.2.117:0/1668219711 learned_addr learned my addr 172.21.2.117:0/1668219711 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670536429 0 0) 0x7fdec40009b0 con 0x7fdec807ab40 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4002fe0 con 0x7fdec807ab40 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876402854 0 0) 0x7fdec0000f90 con 0x7fdec807a0e0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4004370 con 0x7fdec807a0e0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.487 7fdecd1a0700 1 -- 172.21.2.117:0/1668219711 >> v1:172.21.2.125:6789/0 conn(0x7fdec807a0e0 legacy=0x7fdec81b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecd1a0700 1 -- 172.21.2.117:0/1668219711 >> v1:172.21.2.117:6789/0 conn(0x7fdec807ab40 legacy=0x7fdec80794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecd1a0700 1 -- 172.21.2.117:0/1668219711 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80d1c60 con 0x7fdec8079ce0 2021-02-10T11:51:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecd1a0700 1 --2- 172.21.2.117:0/1668219711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec8079ce0 0x7fdec81b59d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7fdebc006650 tx=0x7fdebc009a80).ready entity=mon.0 client_cookie=8a69b8df1c2be6fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdebc00c0c0 con 0x7fdec8079ce0 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc00c340 con 0x7fdec8079ce0 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecc99f700 1 -- 172.21.2.117:0/1668219711 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdebc00c7a0 con 0x7fdec8079ce0 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecfb85700 1 -- 172.21.2.117:0/1668219711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec8079ce0 msgr2=0x7fdec81b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecfb85700 1 --2- 172.21.2.117:0/1668219711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec8079ce0 0x7fdec81b59d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7fdebc006650 tx=0x7fdebc009a80).stop 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecfb85700 1 -- 172.21.2.117:0/1668219711 shutdown_connections 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecfb85700 1 --2- 172.21.2.117:0/1668219711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec8079ce0 0x7fdec81b59d0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.488 7fdecfb85700 1 -- 172.21.2.117:0/1668219711 >> 172.21.2.117:0/1668219711 conn(0x7fdec80d0910 msgr2=0x7fdec80d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.489 7fdecfb85700 1 -- 172.21.2.117:0/1668219711 shutdown_connections 2021-02-10T11:51:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.489 7fdecfb85700 1 -- 172.21.2.117:0/1668219711 wait complete. 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.503 7fdecfb85700 1 Processor -- start 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.503 7fdecfb85700 1 -- start start 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdec809a9f0 0x7fdec808d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 0x7fdec808e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 0x7fdec808ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec809adf0 con 0x7fdec808eb70 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec808dc90 con 0x7fdec809a9f0 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecfb85700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec81b2130 con 0x7fdec808d890 2021-02-10T11:51:17.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 0x7fdec808e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdece1a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 0x7fdec808ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdece1a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 0x7fdec808ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55438/0 (socket says 172.21.2.117:55438) 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdece1a2700 1 -- 172.21.2.117:0/3578152276 learned_addr learned my addr 172.21.2.117:0/3578152276 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd9a1700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdec809a9f0 0x7fdec808d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 -- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdec809a9f0 msgr2=0x7fdec808d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdec809a9f0 0x7fdec808d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 -- 172.21.2.117:0/3578152276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 msgr2=0x7fdec808ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 0x7fdec808ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.504 7fdecd1a0700 1 -- 172.21.2.117:0/3578152276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec81afc50 con 0x7fdec808d890 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdecd1a0700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 0x7fdec808e550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdebc009080 tx=0x7fdebc00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdebc0009b0 con 0x7fdec808d890 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc00bcd0 con 0x7fdec808d890 2021-02-10T11:51:17.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec81afc50 con 0x7fdec808d890 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdebc0009b0 con 0x7fdec808d890 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec81afc50 con 0x7fdec808d890 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdebc0052e0 con 0x7fdec808d890 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 0x7fdeb001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.505 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdebc032690 con 0x7fdec808d890 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.506 7fdecd9a1700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 0x7fdeb001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:17.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.506 7fdecd9a1700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 0x7fdeb001ac10 crc :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:17.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.536 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec80daa00 con 0x7fdec808d890 2021-02-10T11:51:17.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.539 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdebc0067a0 con 0x7fdec808d890 2021-02-10T11:51:17.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.669 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdec80f6820 con 0x7fdec808d890 2021-02-10T11:51:17.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.673 7fdeba7fc700 1 -- 172.21.2.117:0/3578152276 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v73) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdebc01f400 con 0x7fdec808d890 2021-02-10T11:51:17.675 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:51:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 msgr2=0x7fdeb001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 0x7fdeb001ac10 crc :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 msgr2=0x7fdec808e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 0x7fdec808e550 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdebc009080 tx=0x7fdebc00b8e0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 shutdown_connections 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdeb0018730 0x7fdeb001ac10 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdec808eb70 0x7fdec808ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdec808d890 0x7fdec808e550 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 --2- 172.21.2.117:0/3578152276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdec809a9f0 0x7fdec808d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.677 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 >> 172.21.2.117:0/3578152276 conn(0x7fdec809b580 msgr2=0x7fdec81b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.678 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 shutdown_connections 2021-02-10T11:51:17.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:17.678 7fdecfb85700 1 -- 172.21.2.117:0/3578152276 wait complete. 2021-02-10T11:51:17.685 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:51:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.099 7f3c0e1d3700 1 Processor -- start 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 -- start start 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c0807ab40 0x7f3c080794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 0x7f3c08079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c080e5770 con 0x7f3c0807af40 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080d18c0 con 0x7f3c0807ab40 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c0e1d3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080d1c60 con 0x7f3c08079b10 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c0807ab40 0x7f3c080794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3c0807ab40 0x7f3c080794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55442/0 (socket says 172.21.2.117:55442) 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.100 7f3c077fe700 1 -- 172.21.2.117:0/267226696 learned_addr learned my addr 172.21.2.117:0/267226696 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c077fe700 1 -- 172.21.2.117:0/267226696 >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 msgr2=0x7f3c08079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c06ffd700 1 --2- 172.21.2.117:0/267226696 >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 0x7f3c08079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c077fe700 1 --2- 172.21.2.117:0/267226696 >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 0x7f3c08079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c077fe700 1 -- 172.21.2.117:0/267226696 >> v1:172.21.2.125:6789/0 conn(0x7f3c0807af40 legacy=0x7f3c0807b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c077fe700 1 -- 172.21.2.117:0/267226696 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0807bb30 con 0x7f3c0807ab40 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c06ffd700 1 --2- 172.21.2.117:0/267226696 >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 0x7f3c08079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c077fe700 1 --2- 172.21.2.117:0/267226696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c0807ab40 0x7f3c080794f0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7f3bf8006700 tx=0x7f3bf8009b80).ready entity=mon.0 client_cookie=82f22aedc5fa4303 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c067fc700 1 -- 172.21.2.117:0/267226696 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf800c0c0 con 0x7f3c0807ab40 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c067fc700 1 -- 172.21.2.117:0/267226696 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800c3c0 con 0x7f3c0807ab40 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c067fc700 1 -- 172.21.2.117:0/267226696 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf800c8a0 con 0x7f3c0807ab40 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c0e1d3700 1 -- 172.21.2.117:0/267226696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c0807ab40 msgr2=0x7f3c080794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.101 7f3c0e1d3700 1 --2- 172.21.2.117:0/267226696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c0807ab40 0x7f3c080794f0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7f3bf8006700 tx=0x7f3bf8009b80).stop 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 -- 172.21.2.117:0/267226696 shutdown_connections 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 --2- 172.21.2.117:0/267226696 >> v2:172.21.2.125:3300/0 conn(0x7f3c08079b10 0x7f3c08079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 --2- 172.21.2.117:0/267226696 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c0807ab40 0x7f3c080794f0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 -- 172.21.2.117:0/267226696 >> 172.21.2.117:0/267226696 conn(0x7f3c080d0910 msgr2=0x7f3c080d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:18.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 -- 172.21.2.117:0/267226696 shutdown_connections 2021-02-10T11:51:18.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.102 7f3c0e1d3700 1 -- 172.21.2.117:0/267226696 wait complete. 2021-02-10T11:51:18.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.116 7f3c0e1d3700 1 Processor -- start 2021-02-10T11:51:18.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 -- start start 2021-02-10T11:51:18.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080af250 0x7f3c08191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 0x7f3c08191f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08194730 con 0x7f3c080ae830 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081b7970 con 0x7f3c08191b60 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c0e1d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081b74d0 con 0x7f3c080af250 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55448/0 (socket says 172.21.2.117:55448) 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.117 7f3c077fe700 1 -- 172.21.2.117:0/422114250 learned_addr learned my addr 172.21.2.117:0/422114250 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c06ffd700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080af250 0x7f3c08191540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c07fff700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 0x7f3c08191f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 -- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080af250 msgr2=0x7f3c08191540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080af250 0x7f3c08191540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 -- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 msgr2=0x7f3c08191f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 0x7f3c08191f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 -- 172.21.2.117:0/422114250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08194cb0 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c07fff700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 0x7f3c08191f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c077fe700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f3bf800b530 tx=0x7f3bf800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf8009410 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf8002030 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf8002490 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08194cb0 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.118 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c08194cb0 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.119 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3bf80093b0 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.119 7f3bfffff700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 0x7f3bec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.119 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3bf8032560 con 0x7f3c080ae830 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.119 7f3c06ffd700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 0x7f3bec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.120 7f3c06ffd700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 0x7f3bec01abc0 crc :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.150 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c081b7920 con 0x7f3c080ae830 2021-02-10T11:51:18.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.153 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3bf8000c90 con 0x7f3c080ae830 2021-02-10T11:51:18.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.280 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3c081b7920 con 0x7f3c080ae830 2021-02-10T11:51:18.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.285 7f3bfffff700 1 -- 172.21.2.117:0/422114250 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v74) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3bf80093b0 con 0x7f3c080ae830 2021-02-10T11:51:18.286 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 msgr2=0x7f3bec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 0x7f3bec01abc0 crc :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 msgr2=0x7f3c080aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f3bf800b530 tx=0x7f3bf800b8d0).stop 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 shutdown_connections 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c08191b60 0x7f3c08191f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080af250 0x7f3c08191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3bec0186c0 0x7f3bec01abc0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 --2- 172.21.2.117:0/422114250 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080ae830 0x7f3c080aec30 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.289 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 >> 172.21.2.117:0/422114250 conn(0x7f3c0819f6d0 msgr2=0x7f3c08095d30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:18.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.290 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 shutdown_connections 2021-02-10T11:51:18.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.290 7f3c0e1d3700 1 -- 172.21.2.117:0/422114250 wait complete. 2021-02-10T11:51:18.297 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:51:18.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.721 7f197bf4c700 1 Processor -- start 2021-02-10T11:51:18.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.721 7f197bf4c700 1 -- start start 2021-02-10T11:51:18.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f197bf4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 0x7f19741b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f197bf4c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19740e5770 con 0x7f197407ab40 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f197bf4c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19740d18c0 con 0x7f197407a0e0 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f197bf4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197407bf50 con 0x7f1974079ce0 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1979d68700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f197407ab40 0x7f19740794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55704/0 (socket says 172.21.2.117:55704) 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1979d68700 1 -- 172.21.2.117:0/2235244415 learned_addr learned my addr 172.21.2.117:0/2235244415 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1979567700 1 --2- 172.21.2.117:0/2235244415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 0x7f19741b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1978d66700 1 -- 172.21.2.117:0/2235244415 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702989790 0 0) 0x7f1970001180 con 0x7f197407ab40 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1978d66700 1 -- 172.21.2.117:0/2235244415 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1960002fe0 con 0x7f197407ab40 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1979567700 1 -- 172.21.2.117:0/2235244415 >> v1:172.21.2.125:6789/0 conn(0x7f197407a0e0 legacy=0x7f19741b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.722 7f1979567700 1 -- 172.21.2.117:0/2235244415 >> v1:172.21.2.117:6789/0 conn(0x7f197407ab40 legacy=0x7f19740794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f1979567700 1 -- 172.21.2.117:0/2235244415 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740d1c60 con 0x7f1974079ce0 2021-02-10T11:51:18.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f1979567700 1 --2- 172.21.2.117:0/2235244415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 0x7f19741b59d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f1964006650 tx=0x7f1964009a00).ready entity=mon.2 client_cookie=3a031c395c5853ed server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f1978d66700 1 -- 172.21.2.117:0/2235244415 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19640009b0 con 0x7f1974079ce0 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f1978d66700 1 -- 172.21.2.117:0/2235244415 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196400c0c0 con 0x7f1974079ce0 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f1978d66700 1 -- 172.21.2.117:0/2235244415 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f196400c5a0 con 0x7f1974079ce0 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f197bf4c700 1 -- 172.21.2.117:0/2235244415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 msgr2=0x7f19741b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.723 7f197bf4c700 1 --2- 172.21.2.117:0/2235244415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 0x7f19741b59d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f1964006650 tx=0x7f1964009a00).stop 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.724 7f197bf4c700 1 -- 172.21.2.117:0/2235244415 shutdown_connections 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.724 7f197bf4c700 1 --2- 172.21.2.117:0/2235244415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974079ce0 0x7f19741b59d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.724 7f197bf4c700 1 -- 172.21.2.117:0/2235244415 >> 172.21.2.117:0/2235244415 conn(0x7f19740d0910 msgr2=0x7f19740d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.724 7f197bf4c700 1 -- 172.21.2.117:0/2235244415 shutdown_connections 2021-02-10T11:51:18.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.724 7f197bf4c700 1 -- 172.21.2.117:0/2235244415 wait complete. 2021-02-10T11:51:18.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 Processor -- start 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 -- start start 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974100b50 0x7f19740fa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19740faa40 0x7f19740fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19740987b0 con 0x7f1974100750 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974100210 con 0x7f19740faa40 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f197bf4c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19740fb600 con 0x7f1974100b50 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f1979d68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f1979d68700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55462/0 (socket says 172.21.2.117:55462) 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.739 7f1979d68700 1 -- 172.21.2.117:0/244614711 learned_addr learned my addr 172.21.2.117:0/244614711 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 -- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974100b50 msgr2=0x7f19740fa420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:18.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974100b50 0x7f19740fa420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 -- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19740faa40 msgr2=0x7f19740fafe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19740faa40 0x7f19740fafe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 -- 172.21.2.117:0/244614711 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740fb820 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f1979d68700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f197000a5d0 tx=0x7f197000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1970000ce0 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700043a0 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f197bf4c700 1 -- 172.21.2.117:0/244614711 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19740fb820 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19700047f0 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.740 7f197bf4c700 1 -- 172.21.2.117:0/244614711 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19740fb820 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.741 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1970000c90 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.741 7f196a7fc700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 0x7f195c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.741 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1970032910 con 0x7f1974100750 2021-02-10T11:51:18.742 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.741 7f1979567700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 0x7f195c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:18.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.742 7f1979567700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 0x7f195c01ac10 crc :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:18.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.771 7f197bf4c700 1 -- 172.21.2.117:0/244614711 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19740e5770 con 0x7f1974100750 2021-02-10T11:51:18.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.774 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19700043a0 con 0x7f1974100750 2021-02-10T11:51:18.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.904 7f197bf4c700 1 -- 172.21.2.117:0/244614711 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f19740e5770 con 0x7f1974100750 2021-02-10T11:51:18.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.907 7f196a7fc700 1 -- 172.21.2.117:0/244614711 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v75) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1970004cc0 con 0x7f1974100750 2021-02-10T11:51:18.906 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 -- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 msgr2=0x7f195c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 0x7f195c01ac10 crc :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 -- 172.21.2.117:0/244614711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 msgr2=0x7f1974098110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f197000a5d0 tx=0x7f197000a970).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 -- 172.21.2.117:0/244614711 shutdown_connections 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19740faa40 0x7f19740fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1974100b50 0x7f19740fa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f195c018730 0x7f195c01ac10 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 --2- 172.21.2.117:0/244614711 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1974100750 0x7f1974098110 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:18.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.908 7f197bf4c700 1 -- 172.21.2.117:0/244614711 >> 172.21.2.117:0/244614711 conn(0x7f197419f690 msgr2=0x7f19741950e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:18.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.909 7f197bf4c700 1 -- 172.21.2.117:0/244614711 shutdown_connections 2021-02-10T11:51:18.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:18.909 7f197bf4c700 1 -- 172.21.2.117:0/244614711 wait complete. 2021-02-10T11:51:18.916 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:51:19.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 Processor -- start 2021-02-10T11:51:19.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 -- start start 2021-02-10T11:51:19.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f91f8079ce0 0x7f91f81b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f807a5c0 con 0x7f91f807a0e0 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f80d18c0 con 0x7f91f807ab40 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.340 7f91fcd0d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f807bf50 con 0x7f91f8079ce0 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f7314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f91f807a0e0 0x7f91f81b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55722/0 (socket says 172.21.2.117:55722) 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f7314700 1 -- 172.21.2.117:0/2306631621 learned_addr learned my addr 172.21.2.117:0/2306631621 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f6b13700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f91f807ab40 0x7f91f80794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48644/0 (socket says 172.21.2.117:48644) 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f6312700 1 --2- 172.21.2.117:0/2306631621 >> v2:172.21.2.125:3300/0 conn(0x7f91f8079ce0 0x7f91f81b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839951707 0 0) 0x7f91f0001500 con 0x7f91f807a0e0 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4002fe0 con 0x7f91f807a0e0 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832036870 0 0) 0x7f91e8001080 con 0x7f91f807ab40 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4003210 con 0x7f91f807ab40 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f6312700 1 -- 172.21.2.117:0/2306631621 >> v1:172.21.2.125:6789/0 conn(0x7f91f807ab40 legacy=0x7f91f80794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f6312700 1 -- 172.21.2.117:0/2306631621 >> v1:172.21.2.117:6789/0 conn(0x7f91f807a0e0 legacy=0x7f91f81b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.341 7f91f6312700 1 -- 172.21.2.117:0/2306631621 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f80d1c60 con 0x7f91f8079ce0 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91f6312700 1 --2- 172.21.2.117:0/2306631621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f8079ce0 0x7f91f81b59d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f91e0006650 tx=0x7f91e0009a80).ready entity=mon.1 client_cookie=c16faadf8ebb1a03 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e000c0c0 con 0x7f91f8079ce0 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e000c340 con 0x7f91f8079ce0 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91f5b11700 1 -- 172.21.2.117:0/2306631621 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e000c860 con 0x7f91f8079ce0 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91fcd0d700 1 -- 172.21.2.117:0/2306631621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f8079ce0 msgr2=0x7f91f81b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.342 7f91fcd0d700 1 --2- 172.21.2.117:0/2306631621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f8079ce0 0x7f91f81b59d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f91e0006650 tx=0x7f91e0009a80).stop 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.343 7f91fcd0d700 1 -- 172.21.2.117:0/2306631621 shutdown_connections 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.343 7f91fcd0d700 1 --2- 172.21.2.117:0/2306631621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f8079ce0 0x7f91f81b59d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.343 7f91fcd0d700 1 -- 172.21.2.117:0/2306631621 >> 172.21.2.117:0/2306631621 conn(0x7f91f80d0910 msgr2=0x7f91f80d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.343 7f91fcd0d700 1 -- 172.21.2.117:0/2306631621 shutdown_connections 2021-02-10T11:51:19.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.343 7f91fcd0d700 1 -- 172.21.2.117:0/2306631621 wait complete. 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.357 7f91fcd0d700 1 Processor -- start 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.357 7f91fcd0d700 1 -- start start 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91f8197d00 0x7f91f8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91f80b2860 0x7f91f808e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f808f710 con 0x7f91f8197d00 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81989d0 con 0x7f91f80b1e40 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91fcd0d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8198720 con 0x7f91f80b2860 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91f8197d00 0x7f91f8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:59858/0 (socket says 172.21.2.117:59858) 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 -- 172.21.2.117:0/3992812552 learned_addr learned my addr 172.21.2.117:0/3992812552 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f7314700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91f80b2860 0x7f91f808e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 -- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91f80b2860 msgr2=0x7f91f808e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91f80b2860 0x7f91f808e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 -- 172.21.2.117:0/3992812552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91f8197d00 msgr2=0x7f91f8198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91f8197d00 0x7f91f8198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.358 7f91f6312700 1 -- 172.21.2.117:0/3992812552 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f808e990 con 0x7f91f80b1e40 2021-02-10T11:51:19.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91f6312700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f91e0009630 tx=0x7f91e000b970).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e0002030 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e0006740 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f808e990 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f808e990 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.359 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91e00025f0 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.360 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91e0000c90 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.360 7f91ef7fe700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 0x7f91dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.360 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f91e00327b0 con 0x7f91f80b1e40 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.360 7f91f6b13700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 0x7f91dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.360 7f91f6b13700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 0x7f91dc01abc0 crc :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:19.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.390 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f8198720 con 0x7f91f80b1e40 2021-02-10T11:51:19.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.393 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91e001f580 con 0x7f91f80b1e40 2021-02-10T11:51:19.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.523 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f91f8198720 con 0x7f91f80b1e40 2021-02-10T11:51:19.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.528 7f91ef7fe700 1 -- 172.21.2.117:0/3992812552 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v76) v1 ==== 184+0+0 (secure 0 0 0) 0x7f91e0000c90 con 0x7f91f80b1e40 2021-02-10T11:51:19.529 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:51:19.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 msgr2=0x7f91dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 0x7f91dc01abc0 crc :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 msgr2=0x7f91f80b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f91e0009630 tx=0x7f91e000b970).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 shutdown_connections 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91f80b2860 0x7f91f808e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91f80b1e40 0x7f91f80b2240 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91dc0186c0 0x7f91dc01abc0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 --2- 172.21.2.117:0/3992812552 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91f8197d00 0x7f91f8198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.531 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 >> 172.21.2.117:0/3992812552 conn(0x7f91f81a4280 msgr2=0x7f91f81afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.532 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 shutdown_connections 2021-02-10T11:51:19.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.532 7f91fcd0d700 1 -- 172.21.2.117:0/3992812552 wait complete. 2021-02-10T11:51:19.539 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.952 7f6886411700 1 Processor -- start 2021-02-10T11:51:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.952 7f6886411700 1 -- start start 2021-02-10T11:51:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6886411700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 0x7f68800eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6886411700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f68800eb7f0 0x7f68801b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6886411700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800ede60 con 0x7f68800d18e0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6886411700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800ef7f0 con 0x7f68800eb7f0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6886411700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68800d1ce0 con 0x7f68800ef3f0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f687ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f68800d18e0 0x7f68800ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55732/0 (socket says 172.21.2.117:55732) 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f687ffff700 1 -- 172.21.2.117:0/2298770314 learned_addr learned my addr 172.21.2.117:0/2298770314 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f687f7fe700 1 --2- 172.21.2.117:0/2298770314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 0x7f68800eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f687effd700 1 -- 172.21.2.117:0/2298770314 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994727304 0 0) 0x7f6870000ce0 con 0x7f68800d18e0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.953 7f6884a2e700 1 --2- 172.21.2.117:0/2298770314 >> v2:172.21.2.125:3300/0 conn(0x7f68800eb7f0 0x7f68801b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687effd700 1 -- 172.21.2.117:0/2298770314 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6868002fe0 con 0x7f68800d18e0 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687f7fe700 1 -- 172.21.2.117:0/2298770314 >> v2:172.21.2.125:3300/0 conn(0x7f68800eb7f0 msgr2=0x7f68801b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687f7fe700 1 --2- 172.21.2.117:0/2298770314 >> v2:172.21.2.125:3300/0 conn(0x7f68800eb7f0 0x7f68801b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687f7fe700 1 -- 172.21.2.117:0/2298770314 >> v1:172.21.2.117:6789/0 conn(0x7f68800d18e0 legacy=0x7f68800ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687effd700 1 -- 172.21.2.117:0/2298770314 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 494098176 0 0) 0x7f68700010e0 con 0x7f68800d18e0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687f7fe700 1 -- 172.21.2.117:0/2298770314 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68801b8f90 con 0x7f68800ef3f0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687f7fe700 1 --2- 172.21.2.117:0/2298770314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 0x7f68800eb1d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6874006650 tx=0x7f6874009a00).ready entity=mon.2 client_cookie=f7b6941908936a74 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687effd700 1 -- 172.21.2.117:0/2298770314 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68740009b0 con 0x7f68800ef3f0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687effd700 1 -- 172.21.2.117:0/2298770314 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400c0c0 con 0x7f68800ef3f0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f687effd700 1 -- 172.21.2.117:0/2298770314 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f687400c5e0 con 0x7f68800ef3f0 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f6886411700 1 -- 172.21.2.117:0/2298770314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 msgr2=0x7f68800eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.954 7f6886411700 1 --2- 172.21.2.117:0/2298770314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 0x7f68800eb1d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6874006650 tx=0x7f6874009a00).stop 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 -- 172.21.2.117:0/2298770314 shutdown_connections 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 --2- 172.21.2.117:0/2298770314 >> v2:172.21.2.125:3300/0 conn(0x7f68800eb7f0 0x7f68801b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 --2- 172.21.2.117:0/2298770314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800ef3f0 0x7f68800eb1d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 -- 172.21.2.117:0/2298770314 >> 172.21.2.117:0/2298770314 conn(0x7f68800ced60 msgr2=0x7f68800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 -- 172.21.2.117:0/2298770314 shutdown_connections 2021-02-10T11:51:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.955 7f6886411700 1 -- 172.21.2.117:0/2298770314 wait complete. 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 Processor -- start 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 -- start start 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800959c0 0x7f68800c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68800963f0 0x7f68800967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6880095e70 con 0x7f68800c2910 2021-02-10T11:51:19.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800e86d0 con 0x7f68800963f0 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f6886411700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6880096e10 con 0x7f68800959c0 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55488/0 (socket says 172.21.2.117:55488) 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 -- 172.21.2.117:0/2590497677 learned_addr learned my addr 172.21.2.117:0/2590497677 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 -- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800959c0 msgr2=0x7f68800c22f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800959c0 0x7f68800c22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 -- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68800963f0 msgr2=0x7f68800967f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68800963f0 0x7f68800967f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:19.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.970 7f687f7fe700 1 -- 172.21.2.117:0/2590497677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6880097030 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f687f7fe700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7f687400b5d0 tx=0x7f687400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f687400bd60 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6874002080 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f6886411700 1 -- 172.21.2.117:0/2590497677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6880097030 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f6886411700 1 -- 172.21.2.117:0/2590497677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6880097030 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.971 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68740025a0 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.972 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6874000c90 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.972 7f687cff9700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.972 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f68740326e0 con 0x7f68800c2910 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.973 7f687ffff700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:19.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:19.973 7f687ffff700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 0x7f686001abc0 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:20.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.003 7f6886411700 1 -- 172.21.2.117:0/2590497677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6880087290 con 0x7f68800c2910 2021-02-10T11:51:20.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.005 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6874030d40 con 0x7f68800c2910 2021-02-10T11:51:20.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.133 7f6886411700 1 -- 172.21.2.117:0/2590497677 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6880087290 con 0x7f68800c2910 2021-02-10T11:51:20.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.133 7f687cff9700 1 -- 172.21.2.117:0/2590497677 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f687400bd60 con 0x7f68800c2910 2021-02-10T11:51:20.133 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 102 2021-02-10T11:51:20.133 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:20.133 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":102,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:15.824496","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11124":{"gid":11124,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1123044872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1123044872},{"type":"v1","addr":"172.21.2.125:6835","nonce":1123044872}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 msgr2=0x7f686001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 0x7f686001abc0 crc :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 msgr2=0x7f68800c2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7f687400b5d0 tx=0x7f687400b970).stop 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 shutdown_connections 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f68600186c0 0x7f686001abc0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:20.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68800c2910 0x7f68800c2d10 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f68800963f0 0x7f68800967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 --2- 172.21.2.117:0/2590497677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68800959c0 0x7f68800c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 >> 172.21.2.117:0/2590497677 conn(0x7f68800e8c80 msgr2=0x7f6880092070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 shutdown_connections 2021-02-10T11:51:20.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:20.135 7f6886411700 1 -- 172.21.2.117:0/2590497677 wait complete. 2021-02-10T11:51:20.344 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:51:13.818137', 'modified': '2021-02-10 11:51:15.824496', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11124}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11124': {'gid': 11124, 'name': 'b', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1123044872', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1123044872}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1123044872}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:51:20.344 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:51:20.345 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:51:20.743 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:51:20.743 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 11124, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9250515070000001, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 102, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 125, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.555096656 2021-02-10T11:51:20.744 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:51:20.747 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 11124, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9250515070000001, "mdsmap_epoch": 102, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 12.555096656 } 2021-02-10T11:51:20.748 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:21.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 Processor -- start 2021-02-10T11:51:21.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 -- start start 2021-02-10T11:51:21.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1d100d18e0 0x7f1d100ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1d100ef3f0 0x7f1d101b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d100ede60 con 0x7f1d100eb000 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100ef7f0 con 0x7f1d100d18e0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d16c82700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100d1ce0 con 0x7f1d100ef3f0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d14a9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1d100d18e0 0x7f1d100ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d14a9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1d100d18e0 0x7f1d100ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55498/0 (socket says 172.21.2.117:55498) 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.161 7f1d14a9e700 1 -- 172.21.2.117:0/1452220516 learned_addr learned my addr 172.21.2.117:0/1452220516 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d14a9e700 1 -- 172.21.2.117:0/1452220516 >> v2:172.21.2.125:3300/0 conn(0x7f1d100ef3f0 msgr2=0x7f1d101b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d14a9e700 1 --2- 172.21.2.117:0/1452220516 >> v2:172.21.2.125:3300/0 conn(0x7f1d100ef3f0 0x7f1d101b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d14a9e700 1 -- 172.21.2.117:0/1452220516 >> v1:172.21.2.125:6789/0 conn(0x7f1d100eb000 legacy=0x7f1d100eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d14a9e700 1 -- 172.21.2.117:0/1452220516 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d101b8f90 con 0x7f1d100d18e0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d14a9e700 1 --2- 172.21.2.117:0/1452220516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100d18e0 0x7f1d100ea9e0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7f1cf8006700 tx=0x7f1cf8009b80).ready entity=mon.0 client_cookie=bc780d9069a7abc8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d0f7fe700 1 -- 172.21.2.117:0/1452220516 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cf800c0c0 con 0x7f1d100d18e0 2021-02-10T11:51:21.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d0f7fe700 1 -- 172.21.2.117:0/1452220516 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf800c340 con 0x7f1d100d18e0 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d0f7fe700 1 -- 172.21.2.117:0/1452220516 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1cf800c860 con 0x7f1d100d18e0 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d16c82700 1 -- 172.21.2.117:0/1452220516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100d18e0 msgr2=0x7f1d100ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.162 7f1d16c82700 1 --2- 172.21.2.117:0/1452220516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100d18e0 0x7f1d100ea9e0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7f1cf8006700 tx=0x7f1cf8009b80).stop 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 -- 172.21.2.117:0/1452220516 shutdown_connections 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 --2- 172.21.2.117:0/1452220516 >> v2:172.21.2.125:3300/0 conn(0x7f1d100ef3f0 0x7f1d101b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 --2- 172.21.2.117:0/1452220516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100d18e0 0x7f1d100ea9e0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 -- 172.21.2.117:0/1452220516 >> 172.21.2.117:0/1452220516 conn(0x7f1d100ced60 msgr2=0x7f1d100cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 -- 172.21.2.117:0/1452220516 shutdown_connections 2021-02-10T11:51:21.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.163 7f1d16c82700 1 -- 172.21.2.117:0/1452220516 wait complete. 2021-02-10T11:51:21.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.177 7f1d16c82700 1 Processor -- start 2021-02-10T11:51:21.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 -- start start 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d10196390 0x7f1d100e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d100e9960 0x7f1d100e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10196810 con 0x7f1d100e9560 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10098ad0 con 0x7f1d10196390 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d16c82700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100e8a40 con 0x7f1d100e9960 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d0ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55502/0 (socket says 172.21.2.117:55502) 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.178 7f1d0ffff700 1 -- 172.21.2.117:0/3438508200 learned_addr learned my addr 172.21.2.117:0/3438508200 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 -- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d100e9960 msgr2=0x7f1d100e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d100e9960 0x7f1d100e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 -- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d10196390 msgr2=0x7f1d100e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d10196390 0x7f1d100e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 -- 172.21.2.117:0/3438508200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100e8c60 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0ffff700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f1d0000b5b0 tx=0x7f1d0000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d0000bd40 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d000009b0 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d100e8c60 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1d000062a0 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.179 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1019cd70 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.180 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1d00006020 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.180 7f1d0d7fa700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 0x7f1cf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.180 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1d00032910 con 0x7f1d100e9560 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.180 7f1d14a9e700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 0x7f1cf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.181 7f1d14a9e700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 0x7f1cf401abc0 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.211 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d10087210 con 0x7f1d100e9560 2021-02-10T11:51:21.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.214 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1d00006230 con 0x7f1d100e9560 2021-02-10T11:51:21.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.336 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d100ede60 con 0x7f1d100e9560 2021-02-10T11:51:21.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.337 7f1d0d7fa700 1 -- 172.21.2.117:0/3438508200 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f1d00006000 con 0x7f1d100e9560 2021-02-10T11:51:21.337 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:21.337 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":125,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:14.819978","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:51:11.788181","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:51:12.863682","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 msgr2=0x7f1cf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 0x7f1cf401abc0 crc :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 msgr2=0x7f1d100e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7f1d0000b5b0 tx=0x7f1d0000b950).stop 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 shutdown_connections 2021-02-10T11:51:21.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1cf4018730 0x7f1cf401abc0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1d100e9560 0x7f1d100e7e00 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1d100e9960 0x7f1d100e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 --2- 172.21.2.117:0/3438508200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1d10196390 0x7f1d100e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 >> 172.21.2.117:0/3438508200 conn(0x7f1d1019f610 msgr2=0x7f1d101a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 shutdown_connections 2021-02-10T11:51:21.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.339 7f1d16c82700 1 -- 172.21.2.117:0/3438508200 wait complete. 2021-02-10T11:51:21.347 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:51:11.788181', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:51:12.863682', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:51:21.347 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.764 7f9449f22700 1 Processor -- start 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.764 7f9449f22700 1 -- start start 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.764 7f9449f22700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f94440ef3f0 0x7f94440eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f9449f22700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440ef7f0 con 0x7f94440d18f0 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f9449f22700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94440ede60 con 0x7f94440eb830 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f9449f22700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94440d1da0 con 0x7f94440ef3f0 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94437fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f94440d18f0 0x7f94440eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55760/0 (socket says 172.21.2.117:55760) 2021-02-10T11:51:21.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94437fe700 1 -- 172.21.2.117:0/854724933 learned_addr learned my addr 172.21.2.117:0/854724933 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690507510 0 0) 0x7f942c001180 con 0x7f94440d18f0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430002fe0 con 0x7f94440d18f0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f9442ffd700 1 --2- 172.21.2.117:0/854724933 >> v2:172.21.2.125:3300/0 conn(0x7f94440ef3f0 0x7f94440eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1444882424 0 0) 0x7f942c001180 con 0x7f94440d18f0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 >> v2:172.21.2.125:3300/0 conn(0x7f94440ef3f0 msgr2=0x7f94440eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 --2- 172.21.2.117:0/854724933 >> v2:172.21.2.125:3300/0 conn(0x7f94440ef3f0 0x7f94440eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 >> v1:172.21.2.125:6789/0 conn(0x7f94440eb830 legacy=0x7f94441b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94441bb1c0 con 0x7f94440d18f0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f942c001680 con 0x7f94440d18f0 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 -- 172.21.2.117:0/854724933 >> v1:172.21.2.117:6789/0 conn(0x7f94440d18f0 legacy=0x7f94440eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.765 7f94427fc700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 0x7f9430005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9430006780 0x7f9430008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94300095a0 0x7f943000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943000c2f0 con 0x7f9430003a40 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943000c510 con 0x7f94300095a0 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9430050740 con 0x7f9430006780 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 0x7f9430005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9442ffd700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9430006780 0x7f9430008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9443fff700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94300095a0 0x7f943000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 -- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9430006780 msgr2=0x7f9430008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9430006780 0x7f9430008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 -- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94300095a0 msgr2=0x7f943000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94300095a0 0x7f943000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 -- 172.21.2.117:0/854724933 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94300509a0 con 0x7f9430003a40 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9442ffd700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9430006780 0x7f9430008ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:21.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94437fe700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 0x7f9430005f60 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7f942c0014e0 tx=0x7f942c007d20).ready entity=mon.0 client_cookie=e16a6d4f1eb6bf31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9443fff700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f94300095a0 0x7f943000baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9443fff700 1 -- 172.21.2.117:0/854724933 reap_dead start 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f942c000c90 con 0x7f9430003a40 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c0070c0 con 0x7f9430003a40 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f94427fc700 1 -- 172.21.2.117:0/854724933 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f942c00a7a0 con 0x7f9430003a40 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9449f22700 1 -- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 msgr2=0x7f9430005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.766 7f9449f22700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 0x7f9430005f60 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7f942c0014e0 tx=0x7f942c007d20).stop 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.767 7f9449f22700 1 -- 172.21.2.117:0/854724933 shutdown_connections 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.767 7f9449f22700 1 --2- 172.21.2.117:0/854724933 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9430003a40 0x7f9430005f60 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.767 7f9449f22700 1 -- 172.21.2.117:0/854724933 >> 172.21.2.117:0/854724933 conn(0x7f94440cf060 msgr2=0x7f94440cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.767 7f9449f22700 1 -- 172.21.2.117:0/854724933 shutdown_connections 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.767 7f9449f22700 1 -- 172.21.2.117:0/854724933 wait complete. 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 Processor -- start 2021-02-10T11:51:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 -- start start 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94440a6ce0 0x7f94440dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9444085ec0 0x7f94440862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440a67e0 con 0x7f94440a2040 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440a7120 con 0x7f9444085ec0 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9449f22700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94440a2a60 con 0x7f94440a6ce0 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9443fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9443fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55522/0 (socket says 172.21.2.117:55522) 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.782 7f9443fff700 1 -- 172.21.2.117:0/114919069 learned_addr learned my addr 172.21.2.117:0/114919069 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 -- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94440a6ce0 msgr2=0x7f94440dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94440a6ce0 0x7f94440dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 -- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9444085ec0 msgr2=0x7f94440862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9444085ec0 0x7f94440862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 -- 172.21.2.117:0/114919069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94441a2dc0 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9443fff700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f943c0099a0 tx=0x7f943c00bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f943c006030 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943c0062b0 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9449f22700 1 -- 172.21.2.117:0/114919069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94441a2dc0 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f9449f22700 1 -- 172.21.2.117:0/114919069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94441a3350 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.783 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f943c0056a0 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.784 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f943c0056a0 con 0x7f94440a2040 2021-02-10T11:51:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.784 7f943bfff700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 0x7f942801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:21.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.784 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f943c032cd0 con 0x7f94440a2040 2021-02-10T11:51:21.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.784 7f94437fe700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 0x7f942801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:21.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.785 7f94437fe700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 0x7f942801abc0 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:21.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.814 7f9449f22700 1 -- 172.21.2.117:0/114919069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94440daff0 con 0x7f94440a2040 2021-02-10T11:51:21.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.817 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f943c000ce0 con 0x7f94440a2040 2021-02-10T11:51:21.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.944 7f9449f22700 1 -- 172.21.2.117:0/114919069 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f94440ede60 con 0x7f94440a2040 2021-02-10T11:51:21.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.944 7f943bfff700 1 -- 172.21.2.117:0/114919069 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v125) v1 ==== 92+0+10 (secure 0 0 0) 0x7f943c0056a0 con 0x7f94440a2040 2021-02-10T11:51:21.944 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:21.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 -- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 msgr2=0x7f942801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 0x7f942801abc0 crc :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 -- 172.21.2.117:0/114919069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 msgr2=0x7f94440a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f943c0099a0 tx=0x7f943c00bcc0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 -- 172.21.2.117:0/114919069 shutdown_connections 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9428018730 0x7f942801abc0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f94440a2040 0x7f94440a2440 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9444085ec0 0x7f94440862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 --2- 172.21.2.117:0/114919069 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94440a6ce0 0x7f94440dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:21.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.946 7f9449f22700 1 -- 172.21.2.117:0/114919069 >> 172.21.2.117:0/114919069 conn(0x7f94440b6540 msgr2=0x7f944408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:21.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.947 7f9449f22700 1 -- 172.21.2.117:0/114919069 shutdown_connections 2021-02-10T11:51:21.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:21.947 7f9449f22700 1 -- 172.21.2.117:0/114919069 wait complete. 2021-02-10T11:51:21.954 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:51:22.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.369 7fa0cc831700 1 Processor -- start 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.369 7fa0cc831700 1 -- start start 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0cc831700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c80d18f0 0x7fa0c80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0cc831700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0c80ef3f0 0x7fa0c81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0cc831700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80ede60 con 0x7fa0c80eb040 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0cc831700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c8088020 con 0x7fa0c80ef3f0 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0cc831700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80d1d30 con 0x7fa0c80d18f0 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0c80ef3f0 0x7fa0c81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c5e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa0c80eb040 0x7fa0c80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55778/0 (socket says 172.21.2.117:55778) 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c5e1b700 1 -- 172.21.2.117:0/54035593 learned_addr learned my addr 172.21.2.117:0/54035593 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 -- 172.21.2.117:0/54035593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c80d18f0 msgr2=0x7fa0c80eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 --2- 172.21.2.117:0/54035593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c80d18f0 0x7fa0c80eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 -- 172.21.2.117:0/54035593 >> v1:172.21.2.117:6789/0 conn(0x7fa0c80eb040 legacy=0x7fa0c80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 -- 172.21.2.117:0/54035593 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c81b8fd0 con 0x7fa0c80ef3f0 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.370 7fa0c6e1d700 1 --2- 172.21.2.117:0/54035593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c80ef3f0 0x7fa0c81b79a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7fa0c0006e30 tx=0x7fa0c0009e60).ready entity=mon.0 client_cookie=97c47959e8e6893c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0c561a700 1 -- 172.21.2.117:0/54035593 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c0003030 con 0x7fa0c80ef3f0 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0c561a700 1 -- 172.21.2.117:0/54035593 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c000b300 con 0x7fa0c80ef3f0 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0c561a700 1 -- 172.21.2.117:0/54035593 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0c000b820 con 0x7fa0c80ef3f0 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 -- 172.21.2.117:0/54035593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c80ef3f0 msgr2=0x7fa0c81b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 --2- 172.21.2.117:0/54035593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c80ef3f0 0x7fa0c81b79a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7fa0c0006e30 tx=0x7fa0c0009e60).stop 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 -- 172.21.2.117:0/54035593 shutdown_connections 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 --2- 172.21.2.117:0/54035593 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c80ef3f0 0x7fa0c81b79a0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 --2- 172.21.2.117:0/54035593 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c80d18f0 0x7fa0c80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 -- 172.21.2.117:0/54035593 >> 172.21.2.117:0/54035593 conn(0x7fa0c80cf060 msgr2=0x7fa0c80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 -- 172.21.2.117:0/54035593 shutdown_connections 2021-02-10T11:51:22.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.371 7fa0cc831700 1 -- 172.21.2.117:0/54035593 wait complete. 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 Processor -- start 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 -- start start 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0c809ed70 0x7fa0c809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c809f790 0x7fa0c80b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c809fb90 con 0x7fa0c81a4240 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80dc640 con 0x7fa0c809ed70 2021-02-10T11:51:22.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0cc831700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8097270 con 0x7fa0c809f790 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0c661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0c661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55534/0 (socket says 172.21.2.117:55534) 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.386 7fa0c661c700 1 -- 172.21.2.117:0/2416204193 learned_addr learned my addr 172.21.2.117:0/2416204193 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 -- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c809f790 msgr2=0x7fa0c80b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c809f790 0x7fa0c80b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 -- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0c809ed70 msgr2=0x7fa0c809f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0c809ed70 0x7fa0c809f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 -- 172.21.2.117:0/2416204193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80b8780 con 0x7fa0c81a4240 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0c661c700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7fa0b000b5c0 tx=0x7fa0b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0b0000c40 con 0x7fa0c81a4240 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b000be20 con 0x7fa0c81a4240 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0b0006220 con 0x7fa0c81a4240 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c80b8780 con 0x7fa0c81a4240 2021-02-10T11:51:22.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.387 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c80b8d10 con 0x7fa0c81a4240 2021-02-10T11:51:22.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.388 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa0b0006030 con 0x7fa0c81a4240 2021-02-10T11:51:22.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.388 7fa0beffd700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 0x7fa0ac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.388 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa0b00329a0 con 0x7fa0c81a4240 2021-02-10T11:51:22.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.388 7fa0c5e1b700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 0x7fa0ac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.389 7fa0c5e1b700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 0x7fa0ac01abd0 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:22.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.419 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c80daff0 con 0x7fa0c81a4240 2021-02-10T11:51:22.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.422 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0b0000960 con 0x7fa0c81a4240 2021-02-10T11:51:22.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.549 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa0c80ede60 con 0x7fa0c81a4240 2021-02-10T11:51:22.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.549 7fa0beffd700 1 -- 172.21.2.117:0/2416204193 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v125) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa0b0006030 con 0x7fa0c81a4240 2021-02-10T11:51:22.549 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:22.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 msgr2=0x7fa0ac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 0x7fa0ac01abd0 crc :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 msgr2=0x7fa0c81a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7fa0b000b5c0 tx=0x7fa0b000b960).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 shutdown_connections 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0c809f790 0x7fa0c80b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0c809ed70 0x7fa0c809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0ac0186d0 0x7fa0ac01abd0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 --2- 172.21.2.117:0/2416204193 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0c81a4240 0x7fa0c81a4640 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.551 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 >> 172.21.2.117:0/2416204193 conn(0x7fa0c8087090 msgr2=0x7fa0c808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.552 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 shutdown_connections 2021-02-10T11:51:22.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.552 7fa0cc831700 1 -- 172.21.2.117:0/2416204193 wait complete. 2021-02-10T11:51:22.559 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:51:22.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.977 7f7194398700 1 Processor -- start 2021-02-10T11:51:22.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.977 7f7194398700 1 -- start start 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.978 7f7194398700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 0x7f718c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.978 7f7194398700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 0x7f718c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.978 7f7194398700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718c0ede60 con 0x7f718c0ee3e0 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.978 7f7194398700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718c0eba60 con 0x7f718c0eb040 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.978 7f7194398700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0d1cf0 con 0x7f718c0d18f0 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.979 7f71929b5700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f718c0ee3e0 0x7f718c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48716/0 (socket says 172.21.2.117:48716) 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.979 7f71929b5700 1 -- 172.21.2.117:0/2118342008 learned_addr learned my addr 172.21.2.117:0/2118342008 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.979 7f71919b3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 0x7f718c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.979 7f71921b4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 0x7f718c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71921b4700 1 -- 172.21.2.117:0/2118342008 >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 msgr2=0x7f718c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71921b4700 1 --2- 172.21.2.117:0/2118342008 >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 0x7f718c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71921b4700 1 -- 172.21.2.117:0/2118342008 >> v1:172.21.2.125:6789/0 conn(0x7f718c0ee3e0 legacy=0x7f718c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71921b4700 1 -- 172.21.2.117:0/2118342008 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c0eefd0 con 0x7f718c0d18f0 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71919b3700 1 --2- 172.21.2.117:0/2118342008 >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 0x7f718c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71921b4700 1 --2- 172.21.2.117:0/2118342008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 0x7f718c0eaa20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f7188006650 tx=0x7f7188009a80).ready entity=mon.2 client_cookie=d3ec29219b53720d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71911b2700 1 -- 172.21.2.117:0/2118342008 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718800c0c0 con 0x7f718c0d18f0 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71911b2700 1 -- 172.21.2.117:0/2118342008 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718800c340 con 0x7f718c0d18f0 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f71911b2700 1 -- 172.21.2.117:0/2118342008 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f718800c860 con 0x7f718c0d18f0 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f7194398700 1 -- 172.21.2.117:0/2118342008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 msgr2=0x7f718c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.980 7f7194398700 1 --2- 172.21.2.117:0/2118342008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 0x7f718c0eaa20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f7188006650 tx=0x7f7188009a80).stop 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 -- 172.21.2.117:0/2118342008 shutdown_connections 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 --2- 172.21.2.117:0/2118342008 >> v2:172.21.2.125:3300/0 conn(0x7f718c0eb040 0x7f718c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 --2- 172.21.2.117:0/2118342008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c0d18f0 0x7f718c0eaa20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 -- 172.21.2.117:0/2118342008 >> 172.21.2.117:0/2118342008 conn(0x7f718c0cf060 msgr2=0x7f718c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:22.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 -- 172.21.2.117:0/2118342008 shutdown_connections 2021-02-10T11:51:22.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.981 7f7194398700 1 -- 172.21.2.117:0/2118342008 wait complete. 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.995 7f7194398700 1 Processor -- start 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 -- start start 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f718c1a4240 0x7f718c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c09f790 0x7f718c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c09fb90 con 0x7f718c09ed70 2021-02-10T11:51:22.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c0dc640 con 0x7f718c1a4240 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f7194398700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f718c097270 con 0x7f718c09f790 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f71919b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f71919b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55546/0 (socket says 172.21.2.117:55546) 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.996 7f71919b3700 1 -- 172.21.2.117:0/4119003415 learned_addr learned my addr 172.21.2.117:0/4119003415 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 -- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c09f790 msgr2=0x7f718c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c09f790 0x7f718c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 -- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f718c1a4240 msgr2=0x7f718c1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f718c1a4240 0x7f718c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 -- 172.21.2.117:0/4119003415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718c0b8780 con 0x7f718c09ed70 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f71919b3700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f7180006f40 tx=0x7f718000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71800009b0 con 0x7f718c09ed70 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718000bd80 con 0x7f718c09ed70 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7180002220 con 0x7f718c09ed70 2021-02-10T11:51:22.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f7194398700 1 -- 172.21.2.117:0/4119003415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c0b8780 con 0x7f718c09ed70 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f7194398700 1 -- 172.21.2.117:0/4119003415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718c0b8d10 con 0x7f718c09ed70 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.997 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7180000c90 con 0x7f718c09ed70 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.998 7f717effd700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 0x7f717401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.998 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7180032720 con 0x7f718c09ed70 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.998 7f71921b4700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 0x7f717401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:22.998 7f71921b4700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 0x7f717401abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:23.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.027 7f7194398700 1 -- 172.21.2.117:0/4119003415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718c0daff0 con 0x7f718c09ed70 2021-02-10T11:51:23.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.030 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7180000940 con 0x7f718c09ed70 2021-02-10T11:51:23.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.157 7f7194398700 1 -- 172.21.2.117:0/4119003415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f718c0ede60 con 0x7f718c09ed70 2021-02-10T11:51:23.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.157 7f717effd700 1 -- 172.21.2.117:0/4119003415 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v125) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7180000c90 con 0x7f718c09ed70 2021-02-10T11:51:23.157 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 -- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 msgr2=0x7f717401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 0x7f717401abc0 crc :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 -- 172.21.2.117:0/4119003415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 msgr2=0x7f718c09f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 rx=0x7f7180006f40 tx=0x7f718000b990).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 -- 172.21.2.117:0/4119003415 shutdown_connections 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71740186c0 0x7f717401abc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f718c09ed70 0x7f718c09f170 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f718c09f790 0x7f718c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 --2- 172.21.2.117:0/4119003415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f718c1a4240 0x7f718c1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 -- 172.21.2.117:0/4119003415 >> 172.21.2.117:0/4119003415 conn(0x7f718c087090 msgr2=0x7f718c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:23.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.159 7f7194398700 1 -- 172.21.2.117:0/4119003415 shutdown_connections 2021-02-10T11:51:23.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.160 7f7194398700 1 -- 172.21.2.117:0/4119003415 wait complete. 2021-02-10T11:51:23.167 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:51:23.167 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 Processor -- start 2021-02-10T11:51:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 -- start start 2021-02-10T11:51:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00ef3f0 0x7f96c00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb7f0 0x7f96c01b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00ede60 con 0x7f96c00d18e0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00ef7f0 con 0x7f96c00eb7f0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.579 7f96c67fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00d1ce0 con 0x7f96c00ef3f0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96bffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f96c00d18e0 0x7f96c00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55802/0 (socket says 172.21.2.117:55802) 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96bffff700 1 -- 172.21.2.117:0/2384702880 learned_addr learned my addr 172.21.2.117:0/2384702880 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119800701 0 0) 0x7f96b00010b0 con 0x7f96c00d18e0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8002fe0 con 0x7f96c00d18e0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96bf7fe700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00ef3f0 0x7f96c00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96c4e19700 1 --2- 172.21.2.117:0/2384702880 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb7f0 0x7f96c01b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3948478487 0 0) 0x7f96b00010b0 con 0x7f96c00d18e0 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00ef3f0 msgr2=0x7f96c00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00ef3f0 0x7f96c00eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb7f0 msgr2=0x7f96c01b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 --2- 172.21.2.117:0/2384702880 >> v2:172.21.2.125:3300/0 conn(0x7f96c00eb7f0 0x7f96c01b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c01b8f90 con 0x7f96c00d18e0 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f96b00015b0 con 0x7f96c00d18e0 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 -- 172.21.2.117:0/2384702880 >> v1:172.21.2.117:6789/0 conn(0x7f96c00d18e0 legacy=0x7f96c00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.580 7f96beffd700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96a8003a70 0x7f96a8005ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96a804a750 0x7f96a804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96a804d510 0x7f96a804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a80502e0 con 0x7f96a804a750 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a8050500 con 0x7f96a804d510 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a8050720 con 0x7f96a8003a70 2021-02-10T11:51:23.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96a804a750 0x7f96a804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96c4e19700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96a804d510 0x7f96a804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bffff700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96a8003a70 0x7f96a8005ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 -- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96a8003a70 msgr2=0x7f96a8005ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96a8003a70 0x7f96a8005ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 -- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96a804d510 msgr2=0x7f96a804fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96a804d510 0x7f96a804fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 -- 172.21.2.117:0/2384702880 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a80509c0 con 0x7f96a804a750 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96bf7fe700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96a804a750 0x7f96a804cc70 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7f96b4009310 tx=0x7f96b4009a90).ready entity=mon.0 client_cookie=718c8159b8cfed16 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b400c0c0 con 0x7f96a804a750 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b4006400 con 0x7f96a804a750 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96beffd700 1 -- 172.21.2.117:0/2384702880 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b400c340 con 0x7f96a804a750 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96c67fc700 1 -- 172.21.2.117:0/2384702880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96a804a750 msgr2=0x7f96a804cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96c67fc700 1 --2- 172.21.2.117:0/2384702880 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96a804a750 0x7f96a804cc70 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 rx=0x7f96b4009310 tx=0x7f96b4009a90).stop 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.581 7f96c4e19700 1 -- 172.21.2.117:0/2384702880 reap_dead start 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.582 7f96c67fc700 1 -- 172.21.2.117:0/2384702880 shutdown_connections 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.582 7f96c67fc700 1 -- 172.21.2.117:0/2384702880 >> 172.21.2.117:0/2384702880 conn(0x7f96c00ced60 msgr2=0x7f96c00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.582 7f96c67fc700 1 -- 172.21.2.117:0/2384702880 shutdown_connections 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.582 7f96c67fc700 1 -- 172.21.2.117:0/2384702880 wait complete. 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 Processor -- start 2021-02-10T11:51:23.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 -- start start 2021-02-10T11:51:23.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00a3610 0x7f96c0098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 0x7f96c008b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0097460 con 0x7f96c008b3c0 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c008bde0 con 0x7f96c00a3610 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.597 7f96c67fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c008bfc0 con 0x7f96c00a3210 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 0x7f96c008b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53244/0 (socket says 172.21.2.117:53244) 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bffff700 1 -- 172.21.2.117:0/962484411 learned_addr learned my addr 172.21.2.117:0/962484411 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bf7fe700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00a3610 0x7f96c0098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 -- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 msgr2=0x7f96c00979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 -- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00a3610 msgr2=0x7f96c0098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00a3610 0x7f96c0098000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 -- 172.21.2.117:0/962484411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c008c1e0 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bffff700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c4e19700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 0x7f96c008b7c0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 rx=0x7f96b8006f00 tx=0x7f96b800bc50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b800bd90 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8003030 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f96b80013e0 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c67fc700 1 -- 172.21.2.117:0/962484411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c008c1e0 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.598 7f96c67fc700 1 -- 172.21.2.117:0/962484411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c008c770 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.599 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f96b8000e80 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.599 7f96bcff9700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.599 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f96b8032930 con 0x7f96c008b3c0 2021-02-10T11:51:23.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.600 7f96bffff700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:23.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.600 7f96bffff700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:23.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.629 7f96c67fc700 1 -- 172.21.2.117:0/962484411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c00ede60 con 0x7f96c008b3c0 2021-02-10T11:51:23.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.631 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f96b80090a0 con 0x7f96c008b3c0 2021-02-10T11:51:23.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.758 7f96c67fc700 1 -- 172.21.2.117:0/962484411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c00ede60 con 0x7f96c008b3c0 2021-02-10T11:51:23.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.758 7f96bcff9700 1 -- 172.21.2.117:0/962484411 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f96b8000e80 con 0x7f96c008b3c0 2021-02-10T11:51:23.759 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 102 2021-02-10T11:51:23.759 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:23.759 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":102,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:15.824496","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11124":{"gid":11124,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1123044872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1123044872},{"type":"v1","addr":"172.21.2.125:6835","nonce":1123044872}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:23.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 -- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 msgr2=0x7f96a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 crc :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 -- 172.21.2.117:0/962484411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 msgr2=0x7f96c008b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:23.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 0x7f96c008b7c0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 rx=0x7f96b8006f00 tx=0x7f96b800bc50).stop 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 -- 172.21.2.117:0/962484411 shutdown_connections 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f96a00186c0 0x7f96a001abc0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f96c008b3c0 0x7f96c008b7c0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f96c00a3610 0x7f96c0098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 --2- 172.21.2.117:0/962484411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f96c00a3210 0x7f96c00979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.760 7f96c67fc700 1 -- 172.21.2.117:0/962484411 >> 172.21.2.117:0/962484411 conn(0x7f96c0087440 msgr2=0x7f96c0098790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.761 7f96c67fc700 1 -- 172.21.2.117:0/962484411 shutdown_connections 2021-02-10T11:51:23.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:23.761 7f96c67fc700 1 -- 172.21.2.117:0/962484411 wait complete. 2021-02-10T11:51:23.970 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:51:13.818137', 'modified': '2021-02-10 11:51:15.824496', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11124}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11124': {'gid': 11124, 'name': 'b', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1123044872', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1123044872}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1123044872}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:51:23.970 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:51:23.970 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:51:24.367 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:51:24.367 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:51:24.367 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:51:24.367 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 11124, 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.54873467, 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 102, 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 125, 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.178779675000001 2021-02-10T11:51:24.368 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:51:24.371 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 11124, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.54873467, "mdsmap_epoch": 102, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 16.178779675000001 } 2021-02-10T11:51:24.371 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:51:24.371 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:51:24.371 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:51:24.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:24.372 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:24.707 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:51:25.052 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:51:25.052 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:51:25.394 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:51:25.394 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:51:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:51:25.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:25.395 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:51:25.729 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:51:25.730 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:25.732 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:51:26.077 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:51:26.077 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:51:26.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[166003]: starting ceph client 2021-02-10T11:51:26.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:51:26.143 7f4734eb0f80 -1 init, newargv = 0x55698cd19ff0 newargc=9 2021-02-10T11:51:26.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[166003]: starting fuse 2021-02-10T11:51:26.419 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:51:26.419 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:51:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:51:26.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:26.420 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:51:26.756 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:51:26.756 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:51:26.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:26.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:51:26.756 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:51:26.756 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:51:26.757 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:51:26.758 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:51:26.758 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:51:26.758 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:26.758 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:51:26.758 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:51:27.122 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.166003.asok 2021-02-10T11:51:27.124 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.166003.asok 2021-02-10T11:51:27.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:27.125 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.166003.asok status 2021-02-10T11:51:27.514 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "166003", 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11274, 2021-02-10T11:51:27.515 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11274 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3011254739 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:27.516 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3011254739 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11274 172.21.2.117:0/3011254739", 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3011254739", 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 102, 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 125, 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:51:27.517 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:27.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:27.519 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:27.519 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:27.855 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:51:27.855 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:51:27.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:27.856 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:28.202 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.613 7faf0b070700 1 Processor -- start 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 -- start start 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faf040d18e0 0x7faf040ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faf040ef3f0 0x7faf041b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf040ede60 con 0x7faf040eb000 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf040ef7f0 con 0x7faf040ef3f0 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0b070700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf040d1ce0 con 0x7faf040d18e0 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0968d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faf040ef3f0 0x7faf041b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0968d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7faf040ef3f0 0x7faf041b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55590/0 (socket says 172.21.2.117:55590) 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.614 7faf0968d700 1 -- 172.21.2.117:0/2174988777 learned_addr learned my addr 172.21.2.117:0/2174988777 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:28.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0968d700 1 -- 172.21.2.117:0/2174988777 >> v2:172.21.2.125:3300/0 conn(0x7faf040d18e0 msgr2=0x7faf040ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0968d700 1 --2- 172.21.2.117:0/2174988777 >> v2:172.21.2.125:3300/0 conn(0x7faf040d18e0 0x7faf040ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0968d700 1 -- 172.21.2.117:0/2174988777 >> v1:172.21.2.125:6789/0 conn(0x7faf040eb000 legacy=0x7faf040eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0968d700 1 -- 172.21.2.117:0/2174988777 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf041b8f90 con 0x7faf040ef3f0 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0968d700 1 --2- 172.21.2.117:0/2174988777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040ef3f0 0x7faf041b7960 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7faefc006f00 tx=0x7faefc009f10).ready entity=mon.0 client_cookie=412d72cd18e3e31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf037fe700 1 -- 172.21.2.117:0/2174988777 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faefc00b3b0 con 0x7faf040ef3f0 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf037fe700 1 -- 172.21.2.117:0/2174988777 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc00b730 con 0x7faf040ef3f0 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf037fe700 1 -- 172.21.2.117:0/2174988777 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faefc00bc10 con 0x7faf040ef3f0 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0b070700 1 -- 172.21.2.117:0/2174988777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040ef3f0 msgr2=0x7faf041b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.615 7faf0b070700 1 --2- 172.21.2.117:0/2174988777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040ef3f0 0x7faf041b7960 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 rx=0x7faefc006f00 tx=0x7faefc009f10).stop 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 -- 172.21.2.117:0/2174988777 shutdown_connections 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 --2- 172.21.2.117:0/2174988777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040ef3f0 0x7faf041b7960 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 --2- 172.21.2.117:0/2174988777 >> v2:172.21.2.125:3300/0 conn(0x7faf040d18e0 0x7faf040ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 -- 172.21.2.117:0/2174988777 >> 172.21.2.117:0/2174988777 conn(0x7faf040ced60 msgr2=0x7faf040cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:28.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 -- 172.21.2.117:0/2174988777 shutdown_connections 2021-02-10T11:51:28.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.616 7faf0b070700 1 -- 172.21.2.117:0/2174988777 wait complete. 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 Processor -- start 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 -- start start 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faf04196390 0x7faf040e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faf040e9960 0x7faf040e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04196810 con 0x7faf040e9560 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04098ad0 con 0x7faf040e9960 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf0b070700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf040e8a40 con 0x7faf04196390 2021-02-10T11:51:28.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf03fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf03fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55596/0 (socket says 172.21.2.117:55596) 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.631 7faf03fff700 1 -- 172.21.2.117:0/2275501716 learned_addr learned my addr 172.21.2.117:0/2275501716 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 -- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faf04196390 msgr2=0x7faf040e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faf04196390 0x7faf040e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 -- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faf040e9960 msgr2=0x7faf040e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faf040e9960 0x7faf040e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 -- 172.21.2.117:0/2275501716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf040e8c60 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf03fff700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7faef400b5b0 tx=0x7faef400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faef400bd40 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef4006240 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faef4005720 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf0b070700 1 -- 172.21.2.117:0/2275501716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf040e8c60 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.632 7faf0b070700 1 -- 172.21.2.117:0/2275501716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0419cd70 con 0x7faf040e9560 2021-02-10T11:51:28.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.633 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faef40054a0 con 0x7faf040e9560 2021-02-10T11:51:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.633 7faf017fa700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 0x7faee801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.633 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7faef40326e0 con 0x7faf040e9560 2021-02-10T11:51:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.633 7faf08e8c700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 0x7faee801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:28.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.634 7faf08e8c700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 0x7faee801abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:28.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.664 7faf0b070700 1 -- 172.21.2.117:0/2275501716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf04087210 con 0x7faf040e9560 2021-02-10T11:51:28.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.667 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faef4000c60 con 0x7faf040e9560 2021-02-10T11:51:28.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.790 7faf0b070700 1 -- 172.21.2.117:0/2275501716 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf040ede60 con 0x7faf040e9560 2021-02-10T11:51:28.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.790 7faf017fa700 1 -- 172.21.2.117:0/2275501716 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+14240 (secure 0 0 0) 0x7faef40054a0 con 0x7faf040e9560 2021-02-10T11:51:28.791 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:28.791 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":125,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:14.819978","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:51:11.788181","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":27,"pool_name":"cephfs_data","create_time":"2021-02-10 11:51:12.863682","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:28.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 msgr2=0x7faee801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:28.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 0x7faee801abc0 crc :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:28.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 msgr2=0x7faf040e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 rx=0x7faef400b5b0 tx=0x7faef400b950).stop 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 shutdown_connections 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faee80186c0 0x7faee801abc0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faf040e9560 0x7faf040e7e00 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faf040e9960 0x7faf040e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 --2- 172.21.2.117:0/2275501716 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faf04196390 0x7faf040e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 >> 172.21.2.117:0/2275501716 conn(0x7faf0419f610 msgr2=0x7faf041a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 shutdown_connections 2021-02-10T11:51:28.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:28.792 7faf0b070700 1 -- 172.21.2.117:0/2275501716 wait complete. 2021-02-10T11:51:28.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:51:11.788181', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:51:12.863682', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:51:28.801 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 Processor -- start 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 -- start start 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcb500d18f0 0x7fcb500eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb500ef3f0 0x7fcb501b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500ef7f0 con 0x7fcb500eb040 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb500ede60 con 0x7fcb500ef3f0 2021-02-10T11:51:29.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.217 7fcb54857700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb500d1cf0 con 0x7fcb500d18f0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb500ef3f0 0x7fcb501b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb500ef3f0 0x7fcb501b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55600/0 (socket says 172.21.2.117:55600) 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 -- 172.21.2.117:0/954296542 learned_addr learned my addr 172.21.2.117:0/954296542 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 -- 172.21.2.117:0/954296542 >> v2:172.21.2.125:3300/0 conn(0x7fcb500d18f0 msgr2=0x7fcb500eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 --2- 172.21.2.117:0/954296542 >> v2:172.21.2.125:3300/0 conn(0x7fcb500d18f0 0x7fcb500eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 -- 172.21.2.117:0/954296542 >> v1:172.21.2.117:6789/0 conn(0x7fcb500eb040 legacy=0x7fcb500eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 -- 172.21.2.117:0/954296542 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb501b8fd0 con 0x7fcb500ef3f0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4ee1d700 1 --2- 172.21.2.117:0/954296542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb500ef3f0 0x7fcb501b79a0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7fcb48006f00 tx=0x7fcb48009e90).ready entity=mon.0 client_cookie=aca3a63da562448d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4d61a700 1 -- 172.21.2.117:0/954296542 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4800b300 con 0x7fcb500ef3f0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4d61a700 1 -- 172.21.2.117:0/954296542 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4800b580 con 0x7fcb500ef3f0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb4d61a700 1 -- 172.21.2.117:0/954296542 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4800b9d0 con 0x7fcb500ef3f0 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb54857700 1 -- 172.21.2.117:0/954296542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb500ef3f0 msgr2=0x7fcb501b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.218 7fcb54857700 1 --2- 172.21.2.117:0/954296542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb500ef3f0 0x7fcb501b79a0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 rx=0x7fcb48006f00 tx=0x7fcb48009e90).stop 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 -- 172.21.2.117:0/954296542 shutdown_connections 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 --2- 172.21.2.117:0/954296542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb500ef3f0 0x7fcb501b79a0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 --2- 172.21.2.117:0/954296542 >> v2:172.21.2.125:3300/0 conn(0x7fcb500d18f0 0x7fcb500eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 -- 172.21.2.117:0/954296542 >> 172.21.2.117:0/954296542 conn(0x7fcb500cf060 msgr2=0x7fcb500cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 -- 172.21.2.117:0/954296542 shutdown_connections 2021-02-10T11:51:29.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.219 7fcb54857700 1 -- 172.21.2.117:0/954296542 wait complete. 2021-02-10T11:51:29.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 Processor -- start 2021-02-10T11:51:29.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 -- start start 2021-02-10T11:51:29.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb501a3930 0x7fcb500a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb500a13a0 0x7fcb500a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5007cca0 con 0x7fcb5007c7c0 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5018fb70 con 0x7fcb500a13a0 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.234 7fcb54857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500dc6d0 con 0x7fcb501a3930 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55610/0 (socket says 172.21.2.117:55610) 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 -- 172.21.2.117:0/3240642050 learned_addr learned my addr 172.21.2.117:0/3240642050 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 -- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb501a3930 msgr2=0x7fcb500a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb501a3930 0x7fcb500a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 -- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb500a13a0 msgr2=0x7fcb500a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb500a13a0 0x7fcb500a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 -- 172.21.2.117:0/3240642050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5007d220 con 0x7fcb5007c7c0 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb4e61c700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fcb40006f00 tx=0x7fcb4000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:29.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb40006b90 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4000bd30 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb400020d0 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb54857700 1 -- 172.21.2.117:0/3240642050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5007d220 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.235 7fcb54857700 1 -- 172.21.2.117:0/3240642050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5007d7b0 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.236 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb40007020 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.236 7fcb3effd700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 0x7fcb3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.236 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb40032690 con 0x7fcb5007c7c0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.237 7fcb4de1b700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 0x7fcb3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:29.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.237 7fcb4de1b700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 0x7fcb3401abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:29.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.267 7fcb54857700 1 -- 172.21.2.117:0/3240642050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb500daff0 con 0x7fcb5007c7c0 2021-02-10T11:51:29.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.270 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb4002b100 con 0x7fcb5007c7c0 2021-02-10T11:51:29.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.397 7fcb54857700 1 -- 172.21.2.117:0/3240642050 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fcb500daff0 con 0x7fcb5007c7c0 2021-02-10T11:51:29.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.397 7fcb3effd700 1 -- 172.21.2.117:0/3240642050 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v125) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcb4002b100 con 0x7fcb5007c7c0 2021-02-10T11:51:29.398 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 -- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 msgr2=0x7fcb3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 0x7fcb3401abc0 crc :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 -- 172.21.2.117:0/3240642050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 msgr2=0x7fcb501a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 rx=0x7fcb40006f00 tx=0x7fcb4000b940).stop 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 -- 172.21.2.117:0/3240642050 shutdown_connections 2021-02-10T11:51:29.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.399 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb500a13a0 0x7fcb500a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb501a3930 0x7fcb500a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb340186c0 0x7fcb3401abc0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 --2- 172.21.2.117:0/3240642050 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb5007c7c0 0x7fcb501a3310 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 -- 172.21.2.117:0/3240642050 >> 172.21.2.117:0/3240642050 conn(0x7fcb50087090 msgr2=0x7fcb5008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 -- 172.21.2.117:0/3240642050 shutdown_connections 2021-02-10T11:51:29.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.400 7fcb54857700 1 -- 172.21.2.117:0/3240642050 wait complete. 2021-02-10T11:51:29.408 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:51:29.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.823 7f1a0578a700 1 Processor -- start 2021-02-10T11:51:30.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.823 7f1a0578a700 1 -- start start 2021-02-10T11:51:30.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f1a0578a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a001b69a0 0x7f1a001b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f1a0578a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a000ede60 con 0x7f1a000ef3f0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f1a0578a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a000eba00 con 0x7f1a000d18f0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f1a0578a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000d1cf0 con 0x7f1a001b69a0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f19fed73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1a000ef3f0 0x7f1a000eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55864/0 (socket says 172.21.2.117:55864) 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f19fed73700 1 -- 172.21.2.117:0/3756474266 learned_addr learned my addr 172.21.2.117:0/3756474266 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f19ffd75700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a001b69a0 0x7f1a001b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032589598 0 0) 0x7f19f4001180 con 0x7f1a000ef3f0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.824 7f19fe572700 1 -- 172.21.2.117:0/3756474266 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e8002fe0 con 0x7f1a000ef3f0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1724176791 0 0) 0x7f19f4001180 con 0x7f1a000ef3f0 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a001b69a0 msgr2=0x7f1a001b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a001b69a0 0x7f1a001b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 >> v1:172.21.2.125:6789/0 conn(0x7f1a000d18f0 legacy=0x7f1a000eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a001bb3f0 con 0x7f1a000ef3f0 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f19f4001680 con 0x7f1a000ef3f0 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 >> v1:172.21.2.117:6789/0 conn(0x7f1a000ef3f0 legacy=0x7f1a000eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19e8003b20 0x7f19e8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19e8006830 0x7f19e8008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19e8009630 0x7f19e800bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e800c370 con 0x7f19e8006830 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e800c590 con 0x7f19e8003b20 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fe572700 1 -- 172.21.2.117:0/3756474266 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e800c7b0 con 0x7f19e8009630 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19ffd75700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a001b69a0 0x7f1a001b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19e8006830 0x7f19e8008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19ff574700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19e8003b20 0x7f19e8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19ffd75700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19e8009630 0x7f19e800bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 -- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19e8009630 msgr2=0x7f19e800bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f19e8009630 0x7f19e800bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 -- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19e8003b20 msgr2=0x7f19e8005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f19e8003b20 0x7f19e8005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.825 7f19fed73700 1 -- 172.21.2.117:0/3756474266 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8050a20 con 0x7f19e8006830 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f19fed73700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19e8006830 0x7f19e8008d30 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f19f40018c0 tx=0x7f19f4007cf0).ready entity=mon.0 client_cookie=4e030dba1ce97c62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f40070c0 con 0x7f19e8006830 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f40043a0 con 0x7f19e8006830 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f19fe572700 1 -- 172.21.2.117:0/3756474266 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f40048c0 con 0x7f19e8006830 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f1a0578a700 1 -- 172.21.2.117:0/3756474266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19e8006830 msgr2=0x7f19e8008d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f1a0578a700 1 --2- 172.21.2.117:0/3756474266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f19e8006830 0x7f19e8008d30 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 rx=0x7f19f40018c0 tx=0x7f19f4007cf0).stop 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f19ffd75700 1 -- 172.21.2.117:0/3756474266 reap_dead start 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f1a0578a700 1 -- 172.21.2.117:0/3756474266 shutdown_connections 2021-02-10T11:51:30.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.826 7f1a0578a700 1 -- 172.21.2.117:0/3756474266 >> 172.21.2.117:0/3756474266 conn(0x7f1a000cf060 msgr2=0x7f1a000cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:30.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.827 7f1a0578a700 1 -- 172.21.2.117:0/3756474266 shutdown_connections 2021-02-10T11:51:30.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.827 7f1a0578a700 1 -- 172.21.2.117:0/3756474266 wait complete. 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.841 7f1a0578a700 1 Processor -- start 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.841 7f1a0578a700 1 -- start start 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a00085ec0 0x7f1a000862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a000a2040 0x7f1a000a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000a67e0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000a7120 con 0x7f1a00085ec0 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a000a2a60 con 0x7f1a000a2040 2021-02-10T11:51:30.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55626/0 (socket says 172.21.2.117:55626) 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 -- 172.21.2.117:0/3679754612 learned_addr learned my addr 172.21.2.117:0/3679754612 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 -- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a000a2040 msgr2=0x7f1a000a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a000a2040 0x7f1a000a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 -- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a00085ec0 msgr2=0x7f1a000862c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a00085ec0 0x7f1a000862c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 -- 172.21.2.117:0/3679754612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a001a2dc0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19ff574700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f19f0005be0 tx=0x7f19f000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f000ad20 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f0000e20 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f19f0002350 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a001a2dc0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.842 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a001a3350 con 0x7f1a000a6ce0 2021-02-10T11:51:30.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.843 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19f0006620 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.843 7f19e7fff700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 0x7f19e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.843 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f19f00326a0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.844 7f19fed73700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 0x7f19e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.844 7f19fed73700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 0x7f19e001abc0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.873 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a000daff0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:29.876 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f19f0026610 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.003 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f1a000daff0 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.003 7f19e7fff700 1 -- 172.21.2.117:0/3679754612 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v125) v1 ==== 104+0+10 (secure 0 0 0) 0x7f19f0026610 con 0x7f1a000a6ce0 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.005 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 msgr2=0x7f19e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.005 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 0x7f19e001abc0 crc :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.005 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 msgr2=0x7f1a000dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.005 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 rx=0x7f19f0005be0 tx=0x7f19f000a930).stop 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 shutdown_connections 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1a000a2040 0x7f1a000a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1a00085ec0 0x7f1a000862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f19e00186c0 0x7f19e001abc0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 --2- 172.21.2.117:0/3679754612 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1a000a6ce0 0x7f1a000dc420 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 >> 172.21.2.117:0/3679754612 conn(0x7f1a000b6540 msgr2=0x7f1a0008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 shutdown_connections 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.006 7f1a0578a700 1 -- 172.21.2.117:0/3679754612 wait complete. 2021-02-10T11:51:30.043 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:30.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:51:30.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.458 7f40882c4700 1 Processor -- start 2021-02-10T11:51:30.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.458 7f40882c4700 1 -- start start 2021-02-10T11:51:30.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40882c4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40800d18f0 0x7f40800eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40882c4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40800ede60 con 0x7f40800ef3f0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40882c4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40800d1cf0 con 0x7f40800eb040 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40882c4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4080088020 con 0x7f40800d18f0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40868e1700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f40800ef3f0 0x7f40801b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55888/0 (socket says 172.21.2.117:55888) 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40868e1700 1 -- 172.21.2.117:0/185849798 learned_addr learned my addr 172.21.2.117:0/185849798 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756691225 0 0) 0x7f4078001500 con 0x7f40800ef3f0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40850de700 1 -- 172.21.2.117:0/185849798 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c002fe0 con 0x7f40800ef3f0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.459 7f40860e0700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40800d18f0 0x7f40800eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4079862938 0 0) 0x7f4078001460 con 0x7f40800ef3f0 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40800d18f0 msgr2=0x7f40800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40800d18f0 0x7f40800eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 >> v1:172.21.2.125:6789/0 conn(0x7f40800eb040 legacy=0x7f40800eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40801bb200 con 0x7f40800ef3f0 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4078001980 con 0x7f40800ef3f0 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 >> v1:172.21.2.117:6789/0 conn(0x7f40800ef3f0 legacy=0x7f40801b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f406c003b90 0x7f406c0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f406c006840 0x7f406c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f406c009550 0x7f406c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c300 con 0x7f406c003b90 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c520 con 0x7f406c009550 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c740 con 0x7f406c006840 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f406c003b90 0x7f406c0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40858df700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f406c006840 0x7f406c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40868e1700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f406c009550 0x7f406c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 -- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f406c006840 msgr2=0x7f406c008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f406c006840 0x7f406c008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 -- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f406c009550 msgr2=0x7f406c00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f406c009550 0x7f406c00ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 -- 172.21.2.117:0/185849798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c050980 con 0x7f406c003b90 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40858df700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f406c006840 0x7f406c008cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40868e1700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f406c009550 0x7f406c00ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40860e0700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f406c003b90 0x7f406c0060b0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f4074006ac0 tx=0x7f4074009ab0).ready entity=mon.0 client_cookie=28eacd4f7aefb4c0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f407400c0c0 con 0x7f406c003b90 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4074006540 con 0x7f406c003b90 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40850de700 1 -- 172.21.2.117:0/185849798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f407400c5f0 con 0x7f406c003b90 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40882c4700 1 -- 172.21.2.117:0/185849798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f406c003b90 msgr2=0x7f406c0060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.460 7f40882c4700 1 --2- 172.21.2.117:0/185849798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f406c003b90 0x7f406c0060b0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 rx=0x7f4074006ac0 tx=0x7f4074009ab0).stop 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.461 7f40868e1700 1 -- 172.21.2.117:0/185849798 reap_dead start 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.461 7f40882c4700 1 -- 172.21.2.117:0/185849798 shutdown_connections 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.461 7f40882c4700 1 -- 172.21.2.117:0/185849798 >> 172.21.2.117:0/185849798 conn(0x7f40800cf060 msgr2=0x7f40800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.461 7f40882c4700 1 -- 172.21.2.117:0/185849798 shutdown_connections 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.461 7f40882c4700 1 -- 172.21.2.117:0/185849798 wait complete. 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 Processor -- start 2021-02-10T11:51:30.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 -- start start 2021-02-10T11:51:30.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40801a48b0 0x7f408008c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408008cf40 0x7f40800a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40800aae30 con 0x7f40800a9f90 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40800aa9f0 con 0x7f40801a48b0 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40882c4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40800ab330 con 0x7f408008cf40 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40868e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40868e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55646/0 (socket says 172.21.2.117:55646) 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.476 7f40868e1700 1 -- 172.21.2.117:0/2545254945 learned_addr learned my addr 172.21.2.117:0/2545254945 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 -- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408008cf40 msgr2=0x7f40800a9970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408008cf40 0x7f40800a9970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 -- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40801a48b0 msgr2=0x7f408008c920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40801a48b0 0x7f408008c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 -- 172.21.2.117:0/2545254945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40800ab550 con 0x7f40800a9f90 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40868e1700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f4078007f60 tx=0x7f407800ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4078000cf0 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4078004930 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40780070d0 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40882c4700 1 -- 172.21.2.117:0/2545254945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40800ab550 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.477 7f40882c4700 1 -- 172.21.2.117:0/2545254945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40800ab550 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.478 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4078004b10 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.478 7f4072ffd700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 0x7f406801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.478 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f40780329e0 con 0x7f40800a9f90 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.478 7f40860e0700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 0x7f406801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:30.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.479 7f40860e0700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 0x7f406801abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:30.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.508 7f40882c4700 1 -- 172.21.2.117:0/2545254945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40800ede60 con 0x7f40800a9f90 2021-02-10T11:51:30.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.511 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4078004720 con 0x7f40800a9f90 2021-02-10T11:51:30.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.639 7f40882c4700 1 -- 172.21.2.117:0/2545254945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f40800ede60 con 0x7f40800a9f90 2021-02-10T11:51:30.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.639 7f4072ffd700 1 -- 172.21.2.117:0/2545254945 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v125) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4078004b10 con 0x7f40800a9f90 2021-02-10T11:51:30.640 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:30.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 msgr2=0x7f406801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 0x7f406801abc0 crc :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 msgr2=0x7f40800aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:30.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 rx=0x7f4078007f60 tx=0x7f407800ac70).stop 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 shutdown_connections 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40680186c0 0x7f406801abc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40800a9f90 0x7f40800aa390 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f408008cf40 0x7f40800a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 --2- 172.21.2.117:0/2545254945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40801a48b0 0x7f408008c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 >> 172.21.2.117:0/2545254945 conn(0x7f4080092d00 msgr2=0x7f408008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:30.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 shutdown_connections 2021-02-10T11:51:30.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:30.641 7f40882c4700 1 -- 172.21.2.117:0/2545254945 wait complete. 2021-02-10T11:51:30.650 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:51:30.650 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.061 7fc04eace700 1 Processor -- start 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.061 7fc04eace700 1 -- start start 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04eace700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc0480d18e0 0x7fc0480ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04eace700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04eace700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0480ede60 con 0x7fc0480eb000 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04eace700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0480ef7f0 con 0x7fc0480d18e0 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04eace700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480d1ce0 con 0x7fc0480ef3f0 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04d0eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:31.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04d0eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53338/0 (socket says 172.21.2.117:53338) 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04d0eb700 1 -- 172.21.2.117:0/3819746529 learned_addr learned my addr 172.21.2.117:0/3819746529 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc047fff700 1 --1- 172.21.2.117:0/3819746529 >> v1:172.21.2.125:6789/0 conn(0x7fc0480eb000 0x7fc0480eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48830/0 (socket says 172.21.2.117:48830) 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.062 7fc04c8ea700 1 --2- 172.21.2.117:0/3819746529 >> v2:172.21.2.125:3300/0 conn(0x7fc0480d18e0 0x7fc0480ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04d0eb700 1 -- 172.21.2.117:0/3819746529 >> v2:172.21.2.125:3300/0 conn(0x7fc0480d18e0 msgr2=0x7fc0480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04d0eb700 1 --2- 172.21.2.117:0/3819746529 >> v2:172.21.2.125:3300/0 conn(0x7fc0480d18e0 0x7fc0480ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04d0eb700 1 -- 172.21.2.117:0/3819746529 >> v1:172.21.2.125:6789/0 conn(0x7fc0480eb000 legacy=0x7fc0480eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04d0eb700 1 -- 172.21.2.117:0/3819746529 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0481b8f90 con 0x7fc0480ef3f0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04d0eb700 1 --2- 172.21.2.117:0/3819746529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc040006aa0 tx=0x7fc040009f20).ready entity=mon.2 client_cookie=624baf78a4ad0f67 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc0477fe700 1 -- 172.21.2.117:0/3819746529 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04000b5c0 con 0x7fc0480ef3f0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc0477fe700 1 -- 172.21.2.117:0/3819746529 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040003030 con 0x7fc0480ef3f0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc0477fe700 1 -- 172.21.2.117:0/3819746529 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc04000ba30 con 0x7fc0480ef3f0 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04eace700 1 -- 172.21.2.117:0/3819746529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 msgr2=0x7fc0481b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.063 7fc04eace700 1 --2- 172.21.2.117:0/3819746529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fc040006aa0 tx=0x7fc040009f20).stop 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 -- 172.21.2.117:0/3819746529 shutdown_connections 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 --2- 172.21.2.117:0/3819746529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480ef3f0 0x7fc0481b7960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 --2- 172.21.2.117:0/3819746529 >> v2:172.21.2.125:3300/0 conn(0x7fc0480d18e0 0x7fc0480ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 -- 172.21.2.117:0/3819746529 >> 172.21.2.117:0/3819746529 conn(0x7fc0480ced60 msgr2=0x7fc0480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 -- 172.21.2.117:0/3819746529 shutdown_connections 2021-02-10T11:51:31.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.064 7fc04eace700 1 -- 172.21.2.117:0/3819746529 wait complete. 2021-02-10T11:51:31.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 Processor -- start 2021-02-10T11:51:31.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 -- start start 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0481a33d0 0x7fc0481a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480b8d10 0x7fc0480b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480c0670 con 0x7fc0480b82f0 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480deb20 con 0x7fc0481a33d0 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.079 7fc04eace700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480ded40 con 0x7fc0480b8d10 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55660/0 (socket says 172.21.2.117:55660) 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 -- 172.21.2.117:0/1462739096 learned_addr learned my addr 172.21.2.117:0/1462739096 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 -- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480b8d10 msgr2=0x7fc0480b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480b8d10 0x7fc0480b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:31.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 -- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0481a33d0 msgr2=0x7fc0481a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0481a33d0 0x7fc0481a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 -- 172.21.2.117:0/1462739096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480def60 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc047fff700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7fc038001450 tx=0x7fc03800a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc03800ad20 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0380075a0 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc04eace700 1 -- 172.21.2.117:0/1462739096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0480def60 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc04eace700 1 -- 172.21.2.117:0/1462739096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0480df4f0 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.080 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc038004320 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.081 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc038006020 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.081 7fc0457fa700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 0x7fc02c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.081 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc0380326b0 con 0x7fc0480b82f0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.082 7fc04c8ea700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 0x7fc02c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:31.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.082 7fc04c8ea700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 0x7fc02c01abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:31.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.111 7fc04eace700 1 -- 172.21.2.117:0/1462739096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc048087290 con 0x7fc0480b82f0 2021-02-10T11:51:31.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.114 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc038000e70 con 0x7fc0480b82f0 2021-02-10T11:51:31.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.241 7fc04eace700 1 -- 172.21.2.117:0/1462739096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0480ede60 con 0x7fc0480b82f0 2021-02-10T11:51:31.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.241 7fc0457fa700 1 -- 172.21.2.117:0/1462739096 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2106 (secure 0 0 0) 0x7fc03800ad20 con 0x7fc0480b82f0 2021-02-10T11:51:31.242 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 102 2021-02-10T11:51:31.242 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:31.242 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":102,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:15.824496","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11124":{"gid":11124,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1123044872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1123044872},{"type":"v1","addr":"172.21.2.125:6835","nonce":1123044872}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:31.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 -- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 msgr2=0x7fc02c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 0x7fc02c01abc0 crc :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 -- 172.21.2.117:0/1462739096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 msgr2=0x7fc0480b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 rx=0x7fc038001450 tx=0x7fc03800a930).stop 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 -- 172.21.2.117:0/1462739096 shutdown_connections 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc02c0186c0 0x7fc02c01abc0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc0480b82f0 0x7fc0480b86f0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc0480b8d10 0x7fc0480b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:31.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 --2- 172.21.2.117:0/1462739096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc0481a33d0 0x7fc0481a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:31.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.243 7fc04eace700 1 -- 172.21.2.117:0/1462739096 >> 172.21.2.117:0/1462739096 conn(0x7fc0480874c0 msgr2=0x7fc048096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:31.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.244 7fc04eace700 1 -- 172.21.2.117:0/1462739096 shutdown_connections 2021-02-10T11:51:31.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:31.244 7fc04eace700 1 -- 172.21.2.117:0/1462739096 wait complete. 2021-02-10T11:51:31.453 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:51:13.818137', 'modified': '2021-02-10 11:51:15.824496', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11124}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11124': {'gid': 11124, 'name': 'b', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1123044872', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1123044872}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1123044872}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:51:31.454 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:51:31.454 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:51:31.851 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 11124, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.032924893000001, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 102, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 125, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 125, 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.662967888000001 2021-02-10T11:51:31.852 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:51:31.856 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 11124, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.032924893000001, "mdsmap_epoch": 102, "osdmap_epoch": 125, "osdmap_epoch_barrier": 125, "uptime": 23.662967888000001 } 2021-02-10T11:51:31.857 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:51:31.857 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:51:31.857 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:51:31.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:31.857 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:32.192 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:51:32.536 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:51:32.537 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:51:32.880 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:51:32.880 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:51:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:51:32.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:32.881 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:51:33.216 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:51:33.217 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:51:33.217 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:33.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:51:33.564 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:51:33.564 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:51:33.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[166643]: starting ceph client 2021-02-10T11:51:33.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:51:33.628 7f599bd17f80 -1 init, newargv = 0x556c90929f90 newargc=9 2021-02-10T11:51:33.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[166643]: starting fuse 2021-02-10T11:51:33.906 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:51:33.906 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:51:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:51:33.907 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:33.907 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:51:34.243 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:51:34.243 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:51:34.243 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:51:34.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:51:34.244 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:51:34.245 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:51:34.606 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.166643.asok 2021-02-10T11:51:34.609 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.166643.asok 2021-02-10T11:51:34.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:34.609 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.166643.asok status 2021-02-10T11:51:34.999 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:34.999 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:51:34.999 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "166643", 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11313, 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:51:35.000 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11313 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2637652788 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2637652788 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:35.001 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11313 172.21.2.117:0/2637652788", 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2637652788", 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 102, 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 125, 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:51:35.002 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:35.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:35.004 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:35.004 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:35.342 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:51:35.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:51:35.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:35.343 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:35.690 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:51:36.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.105 7fda13c5d700 1 Processor -- start 2021-02-10T11:51:36.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 -- start start 2021-02-10T11:51:36.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c0e4e40 0x7fda0c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fda0c0e5240 0x7fda0c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fda0c1b0810 0x7fda0c085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c087430 con 0x7fda0c0e5240 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c0e5cc0 con 0x7fda0c1b0810 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda13c5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c085930 con 0x7fda0c0e4e40 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda11278700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fda0c0e5240 0x7fda0c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda11278700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fda0c0e5240 0x7fda0c084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55680/0 (socket says 172.21.2.117:55680) 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.106 7fda11278700 1 -- 172.21.2.117:0/100880603 learned_addr learned my addr 172.21.2.117:0/100880603 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 -- 172.21.2.117:0/100880603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c0e4e40 msgr2=0x7fda0c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11a79700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c0e4e40 0x7fda0c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c0e4e40 0x7fda0c1b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 -- 172.21.2.117:0/100880603 >> v2:172.21.2.125:3300/0 conn(0x7fda0c1b0810 msgr2=0x7fda0c085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 --2- 172.21.2.117:0/100880603 >> v2:172.21.2.125:3300/0 conn(0x7fda0c1b0810 0x7fda0c085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 -- 172.21.2.117:0/100880603 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c084840 con 0x7fda0c0e5240 2021-02-10T11:51:36.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda11278700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0e5240 0x7fda0c084c60 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7fd9fc006700 tx=0x7fd9fc009b80).ready entity=mon.0 client_cookie=b180d67b3d01f577 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda10a77700 1 -- 172.21.2.117:0/100880603 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9fc00c0c0 con 0x7fda0c0e5240 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda10a77700 1 -- 172.21.2.117:0/100880603 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9fc00c440 con 0x7fda0c0e5240 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda10a77700 1 -- 172.21.2.117:0/100880603 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9fc00c960 con 0x7fda0c0e5240 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda13c5d700 1 -- 172.21.2.117:0/100880603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0e5240 msgr2=0x7fda0c084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.107 7fda13c5d700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0e5240 0x7fda0c084c60 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7fd9fc006700 tx=0x7fd9fc009b80).stop 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 -- 172.21.2.117:0/100880603 shutdown_connections 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 --2- 172.21.2.117:0/100880603 >> v2:172.21.2.125:3300/0 conn(0x7fda0c1b0810 0x7fda0c085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0e5240 0x7fda0c084c60 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 --2- 172.21.2.117:0/100880603 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c0e4e40 0x7fda0c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 -- 172.21.2.117:0/100880603 >> 172.21.2.117:0/100880603 conn(0x7fda0c0ce4e0 msgr2=0x7fda0c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.108 7fda13c5d700 1 -- 172.21.2.117:0/100880603 shutdown_connections 2021-02-10T11:51:36.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.109 7fda13c5d700 1 -- 172.21.2.117:0/100880603 wait complete. 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 Processor -- start 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 -- start start 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 0x7fda0c0a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 0x7fda0c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0bb4e0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0a4cb0 con 0x7fda0c19bdf0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.123 7fda13c5d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0a2ea0 con 0x7fda0c19b9f0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55686/0 (socket says 172.21.2.117:55686) 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 -- 172.21.2.117:0/4245555875 learned_addr learned my addr 172.21.2.117:0/4245555875 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11278700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 0x7fda0c0a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 -- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 msgr2=0x7fda0c0a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 0x7fda0c0a4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 -- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 msgr2=0x7fda0c0a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda1227a700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 0x7fda0c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 0x7fda0c0a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11a79700 1 -- 172.21.2.117:0/4245555875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c0a30c0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda11278700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 0x7fda0c0a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.124 7fda1227a700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 0x7fda0c0a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda11a79700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fda08006f10 tx=0x7fda0800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda0800bd80 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda08000c90 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda080022b0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c0a30c0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.125 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c0a30c0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.126 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda080052e0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.126 7fda027fc700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 0x7fd9f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.126 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fda08032540 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.127 7fda11278700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 0x7fd9f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.127 7fda11278700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 0x7fd9f401abc0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.156 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c10c2b0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.159 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda08006ba0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.294 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fda0c085930 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.348 7fda027fc700 1 -- 172.21.2.117:0/4245555875 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v103) v1 ==== 136+0+0 (secure 0 0 0) 0x7fda080052e0 con 0x7fda0c0bb0e0 2021-02-10T11:51:36.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 msgr2=0x7fd9f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 0x7fd9f401abc0 crc :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 msgr2=0x7fda0c0a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 rx=0x7fda08006f10 tx=0x7fda0800b990).stop 2021-02-10T11:51:36.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 shutdown_connections 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda0c19bdf0 0x7fda0c0a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda0c19b9f0 0x7fda0c0a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9f40186c0 0x7fd9f401abc0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 --2- 172.21.2.117:0/4245555875 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda0c0bb0e0 0x7fda0c0a2630 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 >> 172.21.2.117:0/4245555875 conn(0x7fda0c09ce20 msgr2=0x7fda0c0b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 shutdown_connections 2021-02-10T11:51:36.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.361 7fda13c5d700 1 -- 172.21.2.117:0/4245555875 wait complete. 2021-02-10T11:51:36.369 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.780 7fec704ef700 1 Processor -- start 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.780 7fec704ef700 1 -- start start 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec704ef700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec680e7180 0x7fec680e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec704ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec680e8390 0x7fec680e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec704ef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec680e7580 con 0x7fec681b07f0 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec704ef700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec680e5ce0 con 0x7fec680e7180 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec704ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec680e77f0 con 0x7fec680e8390 2021-02-10T11:51:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6db0a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec680e7180 0x7fec680e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6e30b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fec681b07f0 0x7fec680e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55942/0 (socket says 172.21.2.117:55942) 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6e30b700 1 -- 172.21.2.117:0/871709150 learned_addr learned my addr 172.21.2.117:0/871709150 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6eb0c700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec680e8390 0x7fec680e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6db0a700 1 -- 172.21.2.117:0/871709150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec680e8390 msgr2=0x7fec680e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6db0a700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec680e8390 0x7fec680e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6db0a700 1 -- 172.21.2.117:0/871709150 >> v1:172.21.2.117:6789/0 conn(0x7fec681b07f0 legacy=0x7fec680e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.781 7fec6db0a700 1 -- 172.21.2.117:0/871709150 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680e8db0 con 0x7fec680e7180 2021-02-10T11:51:36.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec6db0a700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680e7180 0x7fec680e7d70 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 rx=0x7fec64006650 tx=0x7fec64009a80).ready entity=mon.0 client_cookie=9268f0e3059c2c57 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec6d309700 1 -- 172.21.2.117:0/871709150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec640009b0 con 0x7fec680e7180 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec6d309700 1 -- 172.21.2.117:0/871709150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6400c0c0 con 0x7fec680e7180 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec6d309700 1 -- 172.21.2.117:0/871709150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec6400c5e0 con 0x7fec680e7180 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec704ef700 1 -- 172.21.2.117:0/871709150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680e7180 msgr2=0x7fec680e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.782 7fec704ef700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680e7180 0x7fec680e7d70 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 rx=0x7fec64006650 tx=0x7fec64009a80).stop 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 -- 172.21.2.117:0/871709150 shutdown_connections 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec680e8390 0x7fec680e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 --2- 172.21.2.117:0/871709150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680e7180 0x7fec680e7d70 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 -- 172.21.2.117:0/871709150 >> 172.21.2.117:0/871709150 conn(0x7fec680ce500 msgr2=0x7fec680e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 -- 172.21.2.117:0/871709150 shutdown_connections 2021-02-10T11:51:36.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.783 7fec704ef700 1 -- 172.21.2.117:0/871709150 wait complete. 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.797 7fec704ef700 1 Processor -- start 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.797 7fec704ef700 1 -- start start 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec6819a820 0x7fec680af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec68094e00 0x7fec6808c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec68106e10 con 0x7fec680af630 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6808bf50 con 0x7fec6819a820 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec704ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec680c4090 con 0x7fec68094e00 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec6db0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec6db0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55700/0 (socket says 172.21.2.117:55700) 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec6db0a700 1 -- 172.21.2.117:0/2841634894 learned_addr learned my addr 172.21.2.117:0/2841634894 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec6eb0c700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec68094e00 0x7fec6808c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.798 7fec6e30b700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec6819a820 0x7fec680af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 -- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec68094e00 msgr2=0x7fec6808c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec68094e00 0x7fec6808c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 -- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec6819a820 msgr2=0x7fec680af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec6819a820 0x7fec680af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 -- 172.21.2.117:0/2841634894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6808caf0 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec6db0a700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7fec64009650 tx=0x7fec6400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec640093a0 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6400bde0 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec640052e0 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec704ef700 1 -- 172.21.2.117:0/2841634894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6808caf0 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.799 7fec704ef700 1 -- 172.21.2.117:0/2841634894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6808d080 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.800 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec64006030 con 0x7fec680af630 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.800 7fec5affd700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 0x7fec5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:36.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.800 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fec64032660 con 0x7fec680af630 2021-02-10T11:51:36.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.800 7fec6e30b700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 0x7fec5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:36.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.801 7fec6e30b700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 0x7fec5001abc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:36.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.830 7fec704ef700 1 -- 172.21.2.117:0/2841634894 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec68106e10 con 0x7fec680af630 2021-02-10T11:51:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.833 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec6402bfe0 con 0x7fec680af630 2021-02-10T11:51:36.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.967 7fec704ef700 1 -- 172.21.2.117:0/2841634894 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7fec6808d080 con 0x7fec500186c0 2021-02-10T11:51:36.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.968 7fec5affd700 1 -- 172.21.2.117:0/2841634894 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7fec5c001410 con 0x7fec500186c0 2021-02-10T11:51:36.969 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:51:36.970 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:36.970 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:51:36.970 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:36.970 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:51:36.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 -- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 msgr2=0x7fec5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 0x7fec5001abc0 crc :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 -- 172.21.2.117:0/2841634894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 msgr2=0x7fec680947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:36.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 rx=0x7fec64009650 tx=0x7fec6400b960).stop 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 -- 172.21.2.117:0/2841634894 shutdown_connections 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec500186c0 0x7fec5001abc0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec680af630 0x7fec680947e0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec68094e00 0x7fec6808c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 --2- 172.21.2.117:0/2841634894 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec6819a820 0x7fec680af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.971 7fec704ef700 1 -- 172.21.2.117:0/2841634894 >> 172.21.2.117:0/2841634894 conn(0x7fec680953a0 msgr2=0x7fec6809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.972 7fec704ef700 1 -- 172.21.2.117:0/2841634894 shutdown_connections 2021-02-10T11:51:36.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:36.972 7fec704ef700 1 -- 172.21.2.117:0/2841634894 wait complete. 2021-02-10T11:51:36.979 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 Processor -- start 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 -- start start 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f570c1b07f0 0x7f570c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c0e77f0 con 0x7f570c0e6b90 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570c0e5ce0 con 0x7f570c1b07f0 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.396 7f5714161700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0e54b0 con 0x7f570c0e6170 2021-02-10T11:51:37.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53386/0 (socket says 172.21.2.117:53386) 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 -- 172.21.2.117:0/431540196 learned_addr learned my addr 172.21.2.117:0/431540196 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f5711f7d700 1 --2- 172.21.2.117:0/431540196 >> v2:172.21.2.125:3300/0 conn(0x7f570c1b07f0 0x7f570c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 -- 172.21.2.117:0/431540196 >> v2:172.21.2.125:3300/0 conn(0x7f570c1b07f0 msgr2=0x7f570c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 --2- 172.21.2.117:0/431540196 >> v2:172.21.2.125:3300/0 conn(0x7f570c1b07f0 0x7f570c0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 -- 172.21.2.117:0/431540196 >> v1:172.21.2.125:6789/0 conn(0x7f570c0e6b90 legacy=0x7f570c0e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 -- 172.21.2.117:0/431540196 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0e8d80 con 0x7f570c0e6170 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f571177c700 1 --2- 172.21.2.117:0/431540196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f5700006700 tx=0x7f5700009b80).ready entity=mon.2 client_cookie=2954849732484657 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f5710f7b700 1 -- 172.21.2.117:0/431540196 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f570000c0c0 con 0x7f570c0e6170 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f5710f7b700 1 -- 172.21.2.117:0/431540196 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570000c340 con 0x7f570c0e6170 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.397 7f5710f7b700 1 -- 172.21.2.117:0/431540196 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f570000c860 con 0x7f570c0e6170 2021-02-10T11:51:37.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 -- 172.21.2.117:0/431540196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 msgr2=0x7f570c0e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 --2- 172.21.2.117:0/431540196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f5700006700 tx=0x7f5700009b80).stop 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 -- 172.21.2.117:0/431540196 shutdown_connections 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 --2- 172.21.2.117:0/431540196 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c0e6170 0x7f570c0e6570 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 --2- 172.21.2.117:0/431540196 >> v2:172.21.2.125:3300/0 conn(0x7f570c1b07f0 0x7f570c0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 -- 172.21.2.117:0/431540196 >> 172.21.2.117:0/431540196 conn(0x7f570c0ce500 msgr2=0x7f570c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 -- 172.21.2.117:0/431540196 shutdown_connections 2021-02-10T11:51:37.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.398 7f5714161700 1 -- 172.21.2.117:0/431540196 wait complete. 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 Processor -- start 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 -- start start 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 0x7f570c094af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 0x7f570c0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0a5c70 con 0x7f570c19a340 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570c095110 con 0x7f570c0a7720 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f5714161700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570c0c4060 con 0x7f570c19a740 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f571177c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f571177c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55714/0 (socket says 172.21.2.117:55714) 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.413 7f571177c700 1 -- 172.21.2.117:0/2277752545 learned_addr learned my addr 172.21.2.117:0/2277752545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:37.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 -- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 msgr2=0x7f570c0c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571277e700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 0x7f570c0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f5711f7d700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 0x7f570c094af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 0x7f570c0c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 -- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 msgr2=0x7f570c094af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 0x7f570c094af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 -- 172.21.2.117:0/2277752545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0c3bb0 con 0x7f570c19a340 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571277e700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 0x7f570c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f5711f7d700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 0x7f570c094af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f571177c700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f570000b650 tx=0x7f570000b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f570000bde0 con 0x7f570c19a340 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5700006520 con 0x7f570c19a340 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f57000022b0 con 0x7f570c19a340 2021-02-10T11:51:37.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f5714161700 1 -- 172.21.2.117:0/2277752545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c0c3bb0 con 0x7f570c19a340 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.414 7f5714161700 1 -- 172.21.2.117:0/2277752545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c0c3bb0 con 0x7f570c19a340 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.415 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f57000052e0 con 0x7f570c19a340 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.415 7f56fe7fc700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 0x7f56f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.415 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5700032720 con 0x7f570c19a340 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.416 7f5711f7d700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 0x7f56f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:37.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.416 7f5711f7d700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 0x7f56f401abc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:37.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.446 7f5714161700 1 -- 172.21.2.117:0/2277752545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c07e800 con 0x7f570c19a340 2021-02-10T11:51:37.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.449 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f570001f580 con 0x7f570c19a340 2021-02-10T11:51:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.598 7f5714161700 1 -- 172.21.2.117:0/2277752545 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f570c0e77f0 con 0x7f570c19a340 2021-02-10T11:51:37.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.598 7f56fe7fc700 1 -- 172.21.2.117:0/2277752545 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f57000052e0 con 0x7f570c19a340 2021-02-10T11:51:37.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 msgr2=0x7f56f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 0x7f56f401abc0 crc :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 msgr2=0x7f570c0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 rx=0x7f570000b650 tx=0x7f570000b9f0).stop 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 shutdown_connections 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56f40186c0 0x7f56f401abc0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f570c19a340 0x7f570c0c2e50 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f570c19a740 0x7f570c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 --2- 172.21.2.117:0/2277752545 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f570c0a7720 0x7f570c094af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 >> 172.21.2.117:0/2277752545 conn(0x7f570c0a7210 msgr2=0x7f570c095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 shutdown_connections 2021-02-10T11:51:37.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:37.601 7f5714161700 1 -- 172.21.2.117:0/2277752545 wait complete. 2021-02-10T11:51:37.609 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000848281 2021-02-10T11:51:38.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 Processor -- start 2021-02-10T11:51:38.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 -- start start 2021-02-10T11:51:38.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efc940d18f0 0x7efc940eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc940ef3f0 0x7efc941b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940ef7f0 con 0x7efc940eb040 2021-02-10T11:51:38.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940eded0 con 0x7efc940d18f0 2021-02-10T11:51:38.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.025 7efc9b63b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc940d1cf0 con 0x7efc940ef3f0 2021-02-10T11:51:38.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efc940d18f0 0x7efc940eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc98c56700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efc940eb040 0x7efc940eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:55968/0 (socket says 172.21.2.117:55968) 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc98c56700 1 -- 172.21.2.117:0/1023007515 learned_addr learned my addr 172.21.2.117:0/1023007515 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 -- 172.21.2.117:0/1023007515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc940ef3f0 msgr2=0x7efc941b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 --2- 172.21.2.117:0/1023007515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc940ef3f0 0x7efc941b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 -- 172.21.2.117:0/1023007515 >> v1:172.21.2.117:6789/0 conn(0x7efc940eb040 legacy=0x7efc940eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 -- 172.21.2.117:0/1023007515 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc941b8fd0 con 0x7efc940d18f0 2021-02-10T11:51:38.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc99457700 1 --2- 172.21.2.117:0/1023007515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc940d18f0 0x7efc940eaa20 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7efc90006650 tx=0x7efc90009a80).ready entity=mon.0 client_cookie=7c7ef003f185351a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.026 7efc8bfff700 1 -- 172.21.2.117:0/1023007515 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc9000c0c0 con 0x7efc940d18f0 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc8bfff700 1 -- 172.21.2.117:0/1023007515 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000c340 con 0x7efc940d18f0 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc8bfff700 1 -- 172.21.2.117:0/1023007515 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc9000c860 con 0x7efc940d18f0 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 -- 172.21.2.117:0/1023007515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc940d18f0 msgr2=0x7efc940eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 --2- 172.21.2.117:0/1023007515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc940d18f0 0x7efc940eaa20 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 rx=0x7efc90006650 tx=0x7efc90009a80).stop 2021-02-10T11:51:38.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 -- 172.21.2.117:0/1023007515 shutdown_connections 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 --2- 172.21.2.117:0/1023007515 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc940ef3f0 0x7efc941b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 --2- 172.21.2.117:0/1023007515 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc940d18f0 0x7efc940eaa20 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.027 7efc9b63b700 1 -- 172.21.2.117:0/1023007515 >> 172.21.2.117:0/1023007515 conn(0x7efc940cf060 msgr2=0x7efc940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.028 7efc9b63b700 1 -- 172.21.2.117:0/1023007515 shutdown_connections 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.028 7efc9b63b700 1 -- 172.21.2.117:0/1023007515 wait complete. 2021-02-10T11:51:38.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.042 7efc9b63b700 1 Processor -- start 2021-02-10T11:51:38.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.042 7efc9b63b700 1 -- start start 2021-02-10T11:51:38.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc9409ed70 0x7efc9409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc9409f790 0x7efc940b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9409fb90 con 0x7efc941a4240 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940dc640 con 0x7efc9409ed70 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc9b63b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc94097270 con 0x7efc9409f790 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55722/0 (socket says 172.21.2.117:55722) 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 -- 172.21.2.117:0/678884325 learned_addr learned my addr 172.21.2.117:0/678884325 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 -- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc9409f790 msgr2=0x7efc940b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc9409f790 0x7efc940b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 -- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc9409ed70 msgr2=0x7efc9409f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc9409ed70 0x7efc9409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 -- 172.21.2.117:0/678884325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940b8780 con 0x7efc941a4240 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.043 7efc99457700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7efc90009630 tx=0x7efc9000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc900093a0 con 0x7efc941a4240 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9000bd70 con 0x7efc941a4240 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc90005560 con 0x7efc941a4240 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc9b63b700 1 -- 172.21.2.117:0/678884325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc940b8780 con 0x7efc941a4240 2021-02-10T11:51:38.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc9b63b700 1 -- 172.21.2.117:0/678884325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc940b8d10 con 0x7efc941a4240 2021-02-10T11:51:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.044 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc9000a000 con 0x7efc941a4240 2021-02-10T11:51:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.045 7efc89ffb700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 0x7efc7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.045 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efc900326a0 con 0x7efc941a4240 2021-02-10T11:51:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.045 7efc98c56700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 0x7efc7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.045 7efc98c56700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 0x7efc7401abc0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.076 7efc9b63b700 1 -- 172.21.2.117:0/678884325 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc940b94e0 con 0x7efc941a4240 2021-02-10T11:51:38.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.078 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc90009340 con 0x7efc941a4240 2021-02-10T11:51:38.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.203 7efc9b63b700 1 -- 172.21.2.117:0/678884325 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000848281", "target": ["mgr", ""]}) v1 -- 0x7efc940b8780 con 0x7efc740186c0 2021-02-10T11:51:38.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.209 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:51:38.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.209 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:51:38.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.209 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:51:38.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.209 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:51:38.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.209 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:51:38.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.236 7efc89ffb700 1 -- 172.21.2.117:0/678884325 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efc84000e20 con 0x7efc740186c0 2021-02-10T11:51:38.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 msgr2=0x7efc7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 0x7efc7401abc0 crc :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 msgr2=0x7efc941a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 rx=0x7efc90009630 tx=0x7efc9000b980).stop 2021-02-10T11:51:38.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 shutdown_connections 2021-02-10T11:51:38.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc9409f790 0x7efc940b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc9409ed70 0x7efc9409f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc740186c0 0x7efc7401abc0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 --2- 172.21.2.117:0/678884325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc941a4240 0x7efc941a4640 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 >> 172.21.2.117:0/678884325 conn(0x7efc94087090 msgr2=0x7efc9408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:38.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 shutdown_connections 2021-02-10T11:51:38.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.268 7efc9b63b700 1 -- 172.21.2.117:0/678884325 wait complete. 2021-02-10T11:51:38.276 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs clone status cephfs subvolume_0000000000848281 2021-02-10T11:51:38.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.691 7f3b0f0fa700 1 Processor -- start 2021-02-10T11:51:38.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.691 7f3b0f0fa700 1 -- start start 2021-02-10T11:51:38.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b080d18f0 0x7f3b080eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b080eb040 0x7f3b080eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b080ee3e0 0x7f3b080ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b080edee0 con 0x7f3b080d18f0 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b080ebaa0 con 0x7f3b080eb040 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0f0fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080d1cf0 con 0x7f3b080ee3e0 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b080d18f0 0x7f3b080eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b080d18f0 0x7f3b080eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55732/0 (socket says 172.21.2.117:55732) 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 -- 172.21.2.117:0/1847108866 learned_addr learned my addr 172.21.2.117:0/1847108866 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 -- 172.21.2.117:0/1847108866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b080ee3e0 msgr2=0x7f3b080ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 --2- 172.21.2.117:0/1847108866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b080ee3e0 0x7f3b080ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 -- 172.21.2.117:0/1847108866 >> v2:172.21.2.125:3300/0 conn(0x7f3b080eb040 msgr2=0x7f3b080eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 --2- 172.21.2.117:0/1847108866 >> v2:172.21.2.125:3300/0 conn(0x7f3b080eb040 0x7f3b080eb440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 -- 172.21.2.117:0/1847108866 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080eee00 con 0x7f3b080d18f0 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.692 7f3b0cf16700 1 --2- 172.21.2.117:0/1847108866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b080d18f0 0x7f3b080eaa20 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f3af0006700 tx=0x7f3af0009b00).ready entity=mon.0 client_cookie=560133cb79db7a24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b077fe700 1 -- 172.21.2.117:0/1847108866 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af000c0c0 con 0x7f3b080d18f0 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b077fe700 1 -- 172.21.2.117:0/1847108866 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af000c400 con 0x7f3b080d18f0 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b077fe700 1 -- 172.21.2.117:0/1847108866 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3af000c920 con 0x7f3b080d18f0 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 -- 172.21.2.117:0/1847108866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b080d18f0 msgr2=0x7f3b080eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 --2- 172.21.2.117:0/1847108866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b080d18f0 0x7f3b080eaa20 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 rx=0x7f3af0006700 tx=0x7f3af0009b00).stop 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 -- 172.21.2.117:0/1847108866 shutdown_connections 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 --2- 172.21.2.117:0/1847108866 >> v2:172.21.2.125:3300/0 conn(0x7f3b080eb040 0x7f3b080eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 --2- 172.21.2.117:0/1847108866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b080ee3e0 0x7f3b080ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 --2- 172.21.2.117:0/1847108866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b080d18f0 0x7f3b080eaa20 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.693 7f3b0f0fa700 1 -- 172.21.2.117:0/1847108866 >> 172.21.2.117:0/1847108866 conn(0x7f3b080cf060 msgr2=0x7f3b080cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.694 7f3b0f0fa700 1 -- 172.21.2.117:0/1847108866 shutdown_connections 2021-02-10T11:51:38.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.694 7f3b0f0fa700 1 -- 172.21.2.117:0/1847108866 wait complete. 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 Processor -- start 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 -- start start 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b081a4240 0x7f3b081a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b0809ed70 0x7f3b0809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0809fb90 con 0x7f3b0809f790 2021-02-10T11:51:38.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080dc640 con 0x7f3b0809ed70 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.708 7f3b0f0fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08097270 con 0x7f3b081a4240 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55740/0 (socket says 172.21.2.117:55740) 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 -- 172.21.2.117:0/573763041 learned_addr learned my addr 172.21.2.117:0/573763041 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 -- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b081a4240 msgr2=0x7f3b081a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b081a4240 0x7f3b081a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 -- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b0809ed70 msgr2=0x7f3b0809f170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b0809ed70 0x7f3b0809f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 -- 172.21.2.117:0/573763041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080b8780 con 0x7f3b0809f790 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0d717700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7f3b0000b5c0 tx=0x7f3b0000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b00000990 con 0x7f3b0809f790 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0000bd50 con 0x7f3b0809f790 2021-02-10T11:51:38.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b080b8780 con 0x7f3b0809f790 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b000054d0 con 0x7f3b0809f790 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.709 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b080b8780 con 0x7f3b0809f790 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.710 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b00006030 con 0x7f3b0809f790 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.710 7f3b057fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 0x7f3aec01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.710 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b00032990 con 0x7f3b0809f790 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.710 7f3b0cf16700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 0x7f3aec01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:38.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.711 7f3b0cf16700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 0x7f3aec01abd0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:38.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.741 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b080daff0 con 0x7f3b0809f790 2021-02-10T11:51:38.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.744 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b00000990 con 0x7f3b0809f790 2021-02-10T11:51:38.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.875 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvolume_0000000000848281", "target": ["mgr", ""]}) v1 -- 0x7f3b0809fb90 con 0x7f3aec0186d0 2021-02-10T11:51:38.877 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:51:38.879 7f59c98e1700 -1 mgr.server reply reply (95) Operation not supported operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000848281' of type subvolume 2021-02-10T11:51:38.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.878 7f3b057fa700 1 -- 172.21.2.117:0/573763041 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -95 operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000848281' of type subvolume) v1 ==== 107+0+0 (crc 0 0 0) 0x7f3af0009440 con 0x7f3aec0186d0 2021-02-10T11:51:38.878 INFO:teuthology.orchestra.run.gibba017.stderr:Error ENOTSUP: operation 'clone-status' is not allowed on subvolume 'subvolume_0000000000848281' of type subvolume 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 msgr2=0x7f3aec01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 0x7f3aec01abd0 crc :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 msgr2=0x7f3b080b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 rx=0x7f3b0000b5c0 tx=0x7f3b0000b960).stop 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 shutdown_connections 2021-02-10T11:51:38.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3aec0186d0 0x7f3aec01abd0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b0809f790 0x7f3b080b8160 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b0809ed70 0x7f3b0809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 --2- 172.21.2.117:0/573763041 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b081a4240 0x7f3b081a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.880 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 >> 172.21.2.117:0/573763041 conn(0x7f3b08087090 msgr2=0x7f3b0808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.881 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 shutdown_connections 2021-02-10T11:51:38.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:38.881 7f3b0f0fa700 1 -- 172.21.2.117:0/573763041 wait complete. 2021-02-10T11:51:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 95 2021-02-10T11:51:38.888 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000848281 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.304 7fdadc2ba700 1 Processor -- start 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.304 7fdadc2ba700 1 -- start start 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdadc2ba700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdadc2ba700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 0x7fdad41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdadc2ba700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40ef7f0 con 0x7fdad40d18f0 2021-02-10T11:51:39.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdadc2ba700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad40eded0 con 0x7fdad40eb830 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdadc2ba700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40d1cf0 con 0x7fdad40ef3f0 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.305 7fdad98d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdada8d7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 0x7fdad41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53424/0 (socket says 172.21.2.117:53424) 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdada0d6700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fdad40d18f0 0x7fdad40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48920/0 (socket says 172.21.2.117:48920) 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 -- 172.21.2.117:0/163852579 learned_addr learned my addr 172.21.2.117:0/163852579 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 -- 172.21.2.117:0/163852579 >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 msgr2=0x7fdad41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 --2- 172.21.2.117:0/163852579 >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 0x7fdad41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 -- 172.21.2.117:0/163852579 >> v1:172.21.2.125:6789/0 conn(0x7fdad40d18f0 legacy=0x7fdad40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad90d4700 1 -- 172.21.2.117:0/163852579 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069545778 0 0) 0x7fdad00010c0 con 0x7fdad40d18f0 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 -- 172.21.2.117:0/163852579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad41b8fd0 con 0x7fdad40ef3f0 2021-02-10T11:51:39.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdada8d7700 1 --2- 172.21.2.117:0/163852579 >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 0x7fdad41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.306 7fdad98d5700 1 --2- 172.21.2.117:0/163852579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fdac8006700 tx=0x7fdac8009b80).ready entity=mon.2 client_cookie=54b4f1a693f76b1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.307 7fdad90d4700 1 -- 172.21.2.117:0/163852579 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac800c0c0 con 0x7fdad40ef3f0 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.307 7fdad90d4700 1 -- 172.21.2.117:0/163852579 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac800c400 con 0x7fdad40ef3f0 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.307 7fdad90d4700 1 -- 172.21.2.117:0/163852579 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac800c920 con 0x7fdad40ef3f0 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.307 7fdadc2ba700 1 -- 172.21.2.117:0/163852579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 msgr2=0x7fdad40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.307 7fdadc2ba700 1 --2- 172.21.2.117:0/163852579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fdac8006700 tx=0x7fdac8009b80).stop 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 -- 172.21.2.117:0/163852579 shutdown_connections 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 --2- 172.21.2.117:0/163852579 >> v2:172.21.2.125:3300/0 conn(0x7fdad40eb830 0x7fdad41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 --2- 172.21.2.117:0/163852579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40ef3f0 0x7fdad40eb210 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 -- 172.21.2.117:0/163852579 >> 172.21.2.117:0/163852579 conn(0x7fdad40cf060 msgr2=0x7fdad40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 -- 172.21.2.117:0/163852579 shutdown_connections 2021-02-10T11:51:39.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.308 7fdadc2ba700 1 -- 172.21.2.117:0/163852579 wait complete. 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 Processor -- start 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 -- start start 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdad40a6ce0 0x7fdad40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40a2040 0x7fdad40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40a67e0 con 0x7fdad4085ec0 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40a7120 con 0x7fdad40a6ce0 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdadc2ba700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40a2a60 con 0x7fdad40a2040 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdad98d5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdad98d5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55752/0 (socket says 172.21.2.117:55752) 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.323 7fdad98d5700 1 -- 172.21.2.117:0/4230017780 learned_addr learned my addr 172.21.2.117:0/4230017780 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:39.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 -- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40a2040 msgr2=0x7fdad40a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40a2040 0x7fdad40a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 -- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdad40a6ce0 msgr2=0x7fdad40dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdad40a6ce0 0x7fdad40dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 -- 172.21.2.117:0/4230017780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad41a2dc0 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdad98d5700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fdac80066a0 tx=0x7fdac800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac800bd80 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac80052e0 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac8005740 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad41a2dc0 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.324 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad41a3350 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.325 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdac8006030 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.325 7fdac6ffd700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 0x7fdabc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.325 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdac8032540 con 0x7fdad4085ec0 2021-02-10T11:51:39.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.326 7fdada0d6700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 0x7fdabc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:39.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.327 7fdada0d6700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 0x7fdabc01abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:39.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.355 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad40daff0 con 0x7fdad4085ec0 2021-02-10T11:51:39.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.357 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdac8030d20 con 0x7fdad4085ec0 2021-02-10T11:51:39.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.481 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000848281", "target": ["mgr", ""]}) v1 -- 0x7fdad41a3350 con 0x7fdabc0186c0 2021-02-10T11:51:39.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.490 7fdac6ffd700 1 -- 172.21.2.117:0/4230017780 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fdad0007770 con 0x7fdabc0186c0 2021-02-10T11:51:39.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.498 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 msgr2=0x7fdabc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.498 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 0x7fdabc01abc0 crc :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.498 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 msgr2=0x7fdad40862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.498 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 rx=0x7fdac80066a0 tx=0x7fdac800b990).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 shutdown_connections 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdabc0186c0 0x7fdabc01abc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdad4085ec0 0x7fdad40862c0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdad40a2040 0x7fdad40a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 --2- 172.21.2.117:0/4230017780 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdad40a6ce0 0x7fdad40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 >> 172.21.2.117:0/4230017780 conn(0x7fdad40b6540 msgr2=0x7fdad408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 shutdown_connections 2021-02-10T11:51:39.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:39.499 7fdadc2ba700 1 -- 172.21.2.117:0/4230017780 wait complete. 2021-02-10T11:51:39.507 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:51:39.857 INFO:teuthology.orchestra.run.gibba017.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:51:39.857 INFO:teuthology.orchestra.run.gibba017.stdout:0 2021-02-10T11:51:39.858 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:51:39.858 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:51:39.858 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:51:40.204 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:51:40.550 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:51:40.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:40.550 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:40.550 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:40.885 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:51:40.886 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:51:40.886 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:51:40.886 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:51:40.886 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:41.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[166003]: fuse finished with error 0 and tester_r 0 2021-02-10T11:51:41.238 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:41.238 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:41.238 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:41.574 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:51:41.574 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:51:41.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:51:41.575 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:41.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:41.911 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:41.911 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:42.247 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:51:42.248 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:51:42.248 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:51:42.248 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:51:42.248 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:42.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[166643]: fuse finished with error 0 and tester_r 0 2021-02-10T11:51:42.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:42.601 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:42.601 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:42.936 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:51:42.936 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:51:42.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:51:42.937 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:43.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:43.273 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:43.273 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:51:43.608 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:51:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:43.608 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:51:43.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:51:43.608 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:51:43.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:43.944 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:43.944 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:51:44.278 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:51:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:44.278 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:51:44.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:51:44.278 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:51:44.614 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:51:44.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:44.677 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:51:45.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 Processor -- start 2021-02-10T11:51:45.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 -- start start 2021-02-10T11:51:45.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 0x7f6d541b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6d540e5240 0x7f6d54084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d54085930 con 0x7f6d541b0810 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d540e5cc0 con 0x7f6d540e4e40 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.032 7f6d5aa91700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d54084840 con 0x7f6d540e5240 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d590ae700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6d541b0810 0x7f6d54085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:48938/0 (socket says 172.21.2.117:48938) 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d588ad700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 0x7f6d541b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d590ae700 1 -- 172.21.2.117:0/2139071970 learned_addr learned my addr 172.21.2.117:0/2139071970 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 --2- 172.21.2.117:0/2139071970 >> v2:172.21.2.125:3300/0 conn(0x7f6d540e5240 0x7f6d54084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d537fe700 1 -- 172.21.2.117:0/2139071970 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950365471 0 0) 0x7f6d4c001440 con 0x7f6d541b0810 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d537fe700 1 -- 172.21.2.117:0/2139071970 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d40002fe0 con 0x7f6d541b0810 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 -- 172.21.2.117:0/2139071970 >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 msgr2=0x7f6d541b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 --2- 172.21.2.117:0/2139071970 >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 0x7f6d541b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 -- 172.21.2.117:0/2139071970 >> v1:172.21.2.125:6789/0 conn(0x7f6d541b0810 legacy=0x7f6d54085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 -- 172.21.2.117:0/2139071970 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d54085280 con 0x7f6d540e5240 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d588ad700 1 --2- 172.21.2.117:0/2139071970 >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 0x7f6d541b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d53fff700 1 --2- 172.21.2.117:0/2139071970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540e5240 0x7f6d54084c60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f6d44006650 tx=0x7f6d44009a80).ready entity=mon.1 client_cookie=4d2a722b86b298d3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d537fe700 1 -- 172.21.2.117:0/2139071970 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d440009b0 con 0x7f6d540e5240 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.033 7f6d537fe700 1 -- 172.21.2.117:0/2139071970 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4400c0c0 con 0x7f6d540e5240 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d537fe700 1 -- 172.21.2.117:0/2139071970 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4400c5e0 con 0x7f6d540e5240 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 -- 172.21.2.117:0/2139071970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540e5240 msgr2=0x7f6d54084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 --2- 172.21.2.117:0/2139071970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540e5240 0x7f6d54084c60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f6d44006650 tx=0x7f6d44009a80).stop 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 -- 172.21.2.117:0/2139071970 shutdown_connections 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 --2- 172.21.2.117:0/2139071970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540e5240 0x7f6d54084c60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 --2- 172.21.2.117:0/2139071970 >> v2:172.21.2.117:3300/0 conn(0x7f6d540e4e40 0x7f6d541b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 -- 172.21.2.117:0/2139071970 >> 172.21.2.117:0/2139071970 conn(0x7f6d540ce4e0 msgr2=0x7f6d540e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 -- 172.21.2.117:0/2139071970 shutdown_connections 2021-02-10T11:51:45.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.034 7f6d5aa91700 1 -- 172.21.2.117:0/2139071970 wait complete. 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 Processor -- start 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 -- start start 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540b79b0 0x7f6d540b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d540b9e40 0x7f6d540ba240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540baab0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d540bb030 con 0x7f6d540b79b0 2021-02-10T11:51:45.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.049 7f6d5aa91700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d541a37f0 con 0x7f6d540b9e40 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d588ad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540b79b0 0x7f6d540b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55768/0 (socket says 172.21.2.117:55768) 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 -- 172.21.2.117:0/1671936089 learned_addr learned my addr 172.21.2.117:0/1671936089 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d53fff700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d540b9e40 0x7f6d540ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 -- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d540b9e40 msgr2=0x7f6d540ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d540b9e40 0x7f6d540ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 -- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540b79b0 msgr2=0x7f6d540b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540b79b0 0x7f6d540b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 -- 172.21.2.117:0/1671936089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d541a3a10 con 0x7f6d541a2dd0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d590ae700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7f6d4c00a8f0 tx=0x7f6d4c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4c0079c0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c0048b0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d541a3a10 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d4c004d90 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.050 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5408e920 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.051 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d4c00add0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.051 7f6d517fa700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 0x7f6d3801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.051 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6d4c032ac0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.051 7f6d588ad700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 0x7f6d3801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.052 7f6d588ad700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 0x7f6d3801abd0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.082 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5410c290 con 0x7f6d541a2dd0 2021-02-10T11:51:45.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.085 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d4c000f40 con 0x7f6d541a2dd0 2021-02-10T11:51:45.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.233 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6d54085930 con 0x7f6d541a2dd0 2021-02-10T11:51:45.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.237 7f6d517fa700 1 -- 172.21.2.117:0/1671936089 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6d4c00add0 con 0x7f6d541a2dd0 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.241 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 msgr2=0x7f6d3801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.241 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 0x7f6d3801abd0 crc :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.241 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 msgr2=0x7f6d541a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.241 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 rx=0x7f6d4c00a8f0 tx=0x7f6d4c00ac90).stop 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 shutdown_connections 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d380186d0 0x7f6d3801abd0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6d541a2dd0 0x7f6d541a31d0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6d540b9e40 0x7f6d540ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 --2- 172.21.2.117:0/1671936089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6d540b79b0 0x7f6d540b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 >> 172.21.2.117:0/1671936089 conn(0x7f6d540ace70 msgr2=0x7f6d540b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:45.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 shutdown_connections 2021-02-10T11:51:45.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.242 7f6d5aa91700 1 -- 172.21.2.117:0/1671936089 wait complete. 2021-02-10T11:51:45.251 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status' 2021-02-10T11:51:45.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.663 7f9d42450700 1 Processor -- start 2021-02-10T11:51:45.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.663 7f9d42450700 1 -- start start 2021-02-10T11:51:45.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d42450700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9d3c0d18e0 0x7f9d3c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d42450700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0ef3f0 0x7f9d3c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d42450700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d3c0ede60 con 0x7f9d3c0eb000 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d42450700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d3c0ef7f0 con 0x7f9d3c0d18e0 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d42450700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0d1ce0 con 0x7f9d3c0ef3f0 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9d3c0eb000 0x7f9d3c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56026/0 (socket says 172.21.2.117:56026) 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3b7fe700 1 -- 172.21.2.117:0/2705540000 learned_addr learned my addr 172.21.2.117:0/2705540000 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612777392 0 0) 0x7f9d300009b0 con 0x7f9d3c0eb000 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d24002fe0 con 0x7f9d3c0eb000 2021-02-10T11:51:45.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d40a6d700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0ef3f0 0x7f9d3c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> v2:172.21.2.125:3300/0 conn(0x7f9d3c0d18e0 0x7f9d3c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.664 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2706402502 0 0) 0x7f9d30001010 con 0x7f9d3c0eb000 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0ef3f0 msgr2=0x7f9d3c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0ef3f0 0x7f9d3c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 >> v2:172.21.2.125:3300/0 conn(0x7f9d3c0d18e0 msgr2=0x7f9d3c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:45.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 --2- 172.21.2.117:0/2705540000 >> v2:172.21.2.125:3300/0 conn(0x7f9d3c0d18e0 0x7f9d3c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c1b8f90 con 0x7f9d3c0eb000 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f9d30001590 con 0x7f9d3c0eb000 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 >> v1:172.21.2.117:6789/0 conn(0x7f9d3c0eb000 legacy=0x7f9d3c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d24003b90 0x7f9d240060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d24006950 0x7f9d24008e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d24009770 0x7f9d2400bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2400c4b0 con 0x7f9d24003b90 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2400c6d0 con 0x7f9d24006950 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2400c8f0 con 0x7f9d24009770 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d40a6d700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0ef3f0 0x7f9d3c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> v2:172.21.2.125:3300/0 conn(0x7f9d3c0d18e0 0x7f9d3c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d24003b90 0x7f9d240060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 -- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d24009770 msgr2=0x7f9d2400bc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d24009770 0x7f9d2400bc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 -- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d24006950 msgr2=0x7f9d24008e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d24006950 0x7f9d24008e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 -- 172.21.2.117:0/2705540000 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24050ba0 con 0x7f9d24003b90 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3bfff700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d24003b90 0x7f9d240060b0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7f9d2c009970 tx=0x7f9d2c009d10).ready entity=mon.0 client_cookie=861d31233497723a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d2c0065d0 con 0x7f9d24003b90 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c000d20 con 0x7f9d24003b90 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d3affd700 1 -- 172.21.2.117:0/2705540000 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d2c002220 con 0x7f9d24003b90 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d42450700 1 -- 172.21.2.117:0/2705540000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d24003b90 msgr2=0x7f9d240060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d42450700 1 --2- 172.21.2.117:0/2705540000 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d24003b90 0x7f9d240060b0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 rx=0x7f9d2c009970 tx=0x7f9d2c009d10).stop 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.665 7f9d40a6d700 1 -- 172.21.2.117:0/2705540000 reap_dead start 2021-02-10T11:51:45.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.666 7f9d42450700 1 -- 172.21.2.117:0/2705540000 shutdown_connections 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.666 7f9d42450700 1 -- 172.21.2.117:0/2705540000 >> 172.21.2.117:0/2705540000 conn(0x7f9d3c0ced60 msgr2=0x7f9d3c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.666 7f9d42450700 1 -- 172.21.2.117:0/2705540000 shutdown_connections 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.666 7f9d42450700 1 -- 172.21.2.117:0/2705540000 wait complete. 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 Processor -- start 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 -- start start 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d3c0b59a0 0x7f9d3c08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0b5da0 0x7f9d3c1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0b61a0 con 0x7f9d3c1a3480 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c08c8b0 con 0x7f9d3c0b59a0 2021-02-10T11:51:45.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d42450700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c0ba1c0 con 0x7f9d3c0b5da0 2021-02-10T11:51:45.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d40a6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d40a6d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55786/0 (socket says 172.21.2.117:55786) 2021-02-10T11:51:45.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.681 7f9d40a6d700 1 -- 172.21.2.117:0/262665057 learned_addr learned my addr 172.21.2.117:0/262665057 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 -- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0b5da0 msgr2=0x7f9d3c1a2e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0b5da0 0x7f9d3c1a2e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 -- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d3c0b59a0 msgr2=0x7f9d3c08c290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d3c0b59a0 0x7f9d3c08c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 -- 172.21.2.117:0/262665057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c1a3880 con 0x7f9d3c1a3480 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d40a6d700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7f9d340099a0 tx=0x7f9d3400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d340066f0 con 0x7f9d3c1a3480 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d34003030 con 0x7f9d3c1a3480 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9d340013e0 con 0x7f9d3c1a3480 2021-02-10T11:51:45.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d42450700 1 -- 172.21.2.117:0/262665057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c1a3880 con 0x7f9d3c1a3480 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.682 7f9d42450700 1 -- 172.21.2.117:0/262665057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c0b2bb0 con 0x7f9d3c1a3480 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.683 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9d340011f0 con 0x7f9d3c1a3480 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.683 7f9d38ff9700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 0x7f9d1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.683 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9d34032a20 con 0x7f9d3c1a3480 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.683 7f9d3bfff700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 0x7f9d1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:45.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.684 7f9d3bfff700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 0x7f9d1c01abc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:45.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.713 7f9d42450700 1 -- 172.21.2.117:0/262665057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c0ede60 con 0x7f9d3c1a3480 2021-02-10T11:51:45.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.716 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9d34000f60 con 0x7f9d3c1a3480 2021-02-10T11:51:45.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:45.847 7f9d42450700 1 -- 172.21.2.117:0/262665057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]} v 0) v1 -- 0x7f9d3c0ede60 con 0x7f9d3c1a3480 2021-02-10T11:51:46.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.291 7f9d38ff9700 1 -- 172.21.2.117:0/262665057 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_non_clone_status"]}]=0 v662) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9d340011f0 con 0x7f9d3c1a3480 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 msgr2=0x7f9d1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 0x7f9d1c01abc0 crc :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 msgr2=0x7f9d3c0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 rx=0x7f9d340099a0 tx=0x7f9d3400bcf0).stop 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 shutdown_connections 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9d1c0186c0 0x7f9d1c01abc0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9d3c1a3480 0x7f9d3c0b2590 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9d3c0b5da0 0x7f9d3c1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:46.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 --2- 172.21.2.117:0/262665057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9d3c0b59a0 0x7f9d3c08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:46.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 >> 172.21.2.117:0/262665057 conn(0x7f9d3c0bb5e0 msgr2=0x7f9d3c090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:46.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 shutdown_connections 2021-02-10T11:51:46.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.315 7f9d42450700 1 -- 172.21.2.117:0/262665057 wait complete. 2021-02-10T11:51:46.324 INFO:tasks.cephfs_test_runner:test_non_clone_status (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:51:46.325 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:51:46.325 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:51:46.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 Processor -- start 2021-02-10T11:51:46.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 -- start start 2021-02-10T11:51:46.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2ca0085450 0x7f2ca0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca0085930 con 0x7f2ca00e4e40 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca01b09e0 con 0x7f2ca00e5240 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2ca5324700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca00e5cc0 con 0x7f2ca0085450 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2c9f07d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2ca00e4e40 0x7f2ca01b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56042/0 (socket says 172.21.2.117:56042) 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.736 7f2c9f07d700 1 -- 172.21.2.117:0/3111853444 learned_addr learned my addr 172.21.2.117:0/3111853444 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437437495 0 0) 0x7f2c880010b0 con 0x7f2ca00e4e40 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c8c002fe0 con 0x7f2ca00e4e40 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 --2- 172.21.2.117:0/3111853444 >> v2:172.21.2.125:3300/0 conn(0x7f2ca0085450 0x7f2ca0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3239417072 0 0) 0x7f2c880010b0 con 0x7f2ca00e4e40 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 >> v2:172.21.2.125:3300/0 conn(0x7f2ca0085450 msgr2=0x7f2ca0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 --2- 172.21.2.117:0/3111853444 >> v2:172.21.2.125:3300/0 conn(0x7f2ca0085450 0x7f2ca0086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 >> v1:172.21.2.125:6789/0 conn(0x7f2ca00e5240 legacy=0x7f2ca0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:46.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca0084840 con 0x7f2ca00e4e40 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2c880015b0 con 0x7f2ca00e4e40 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 >> v1:172.21.2.117:6789/0 conn(0x7f2ca00e4e40 legacy=0x7f2ca01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c8c003a70 0x7f2c8c005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c8c006790 0x7f2c8c008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 0x7f2c8c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00c300 con 0x7f2c8c0095b0 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00c520 con 0x7f2c8c006790 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00c740 con 0x7f2c8c003a70 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 0x7f2c8c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:46.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f07d700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c8c003a70 0x7f2c8c005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 -- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c8c003a70 msgr2=0x7f2c8c005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c8c003a70 0x7f2c8c005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 -- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c8c006790 msgr2=0x7f2c8c008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c8c006790 0x7f2c8c008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.737 7f2c9f87e700 1 -- 172.21.2.117:0/3111853444 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0509b0 con 0x7f2c8c0095b0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2c9f87e700 1 -- 172.21.2.117:0/3111853444 reap_dead start 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2c9f87e700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 0x7f2c8c00bab0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 rx=0x7f2c98006a40 tx=0x7f2c98009f40).ready entity=mon.0 client_cookie=90b8ce616709ecad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c98006ab0 con 0x7f2c8c0095b0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98007070 con 0x7f2c8c0095b0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2c9e07b700 1 -- 172.21.2.117:0/3111853444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c98007590 con 0x7f2c8c0095b0 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 -- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 msgr2=0x7f2c8c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 0x7f2c8c00bab0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 rx=0x7f2c98006a40 tx=0x7f2c98009f40).stop 2021-02-10T11:51:46.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 -- 172.21.2.117:0/3111853444 shutdown_connections 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 --2- 172.21.2.117:0/3111853444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c8c0095b0 0x7f2c8c00bab0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 -- 172.21.2.117:0/3111853444 >> 172.21.2.117:0/3111853444 conn(0x7f2ca00ce4e0 msgr2=0x7f2ca00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 -- 172.21.2.117:0/3111853444 shutdown_connections 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.738 7f2ca5324700 1 -- 172.21.2.117:0/3111853444 wait complete. 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.753 7f2ca5324700 1 Processor -- start 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.753 7f2ca5324700 1 -- start start 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca00ad550 0x7f2ca00bee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca00ab150 0x7f2ca00ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00ac330 con 0x7f2ca00abb70 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00adad0 con 0x7f2ca00ad550 2021-02-10T11:51:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00bf490 con 0x7f2ca00ab150 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55804/0 (socket says 172.21.2.117:55804) 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 -- 172.21.2.117:0/3481694993 learned_addr learned my addr 172.21.2.117:0/3481694993 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 -- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca00ab150 msgr2=0x7f2ca00ab550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca00ab150 0x7f2ca00ab550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 -- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca00ad550 msgr2=0x7f2ca00bee70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca00ad550 0x7f2ca00bee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 -- 172.21.2.117:0/3481694993 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00abf70 con 0x7f2ca00abb70 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c9f87e700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f2c980096f0 tx=0x7f2c9800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c98000f90 con 0x7f2ca00abb70 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98003030 con 0x7f2ca00abb70 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c980015a0 con 0x7f2ca00abb70 2021-02-10T11:51:46.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca00abf70 con 0x7f2ca00abb70 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.754 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca00abf70 con 0x7f2ca00abb70 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.755 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c980013b0 con 0x7f2ca00abb70 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.755 7f2c97fff700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 0x7f2c8401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.755 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2c98032d20 con 0x7f2ca00abb70 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.756 7f2c9f07d700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 0x7f2c8401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.756 7f2c9f07d700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 0x7f2c8401abd0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:46.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.786 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca00adad0 con 0x7f2ca00abb70 2021-02-10T11:51:46.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.789 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c98003010 con 0x7f2ca00abb70 2021-02-10T11:51:46.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:46.922 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7f2ca00adad0 con 0x7f2ca00abb70 2021-02-10T11:51:47.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.296 7f2c97fff700 1 -- 172.21.2.117:0/3481694993 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v663) v1 ==== 159+0+0 (secure 0 0 0) 0x7f2c98003010 con 0x7f2ca00abb70 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 msgr2=0x7f2c8401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 0x7f2c8401abd0 crc :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 msgr2=0x7f2ca00ae050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 rx=0x7f2c980096f0 tx=0x7f2c9800bd40).stop 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 shutdown_connections 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186d0 0x7f2c8401abd0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca00abb70 0x7f2ca00ae050 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca00ab150 0x7f2ca00ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 --2- 172.21.2.117:0/3481694993 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca00ad550 0x7f2ca00bee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.341 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 >> 172.21.2.117:0/3481694993 conn(0x7f2ca00ace80 msgr2=0x7f2ca00c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:47.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.342 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 shutdown_connections 2021-02-10T11:51:47.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.342 7f2ca5324700 1 -- 172.21.2.117:0/3481694993 wait complete. 2021-02-10T11:51:47.351 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 Processor -- start 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 -- start start 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f59400d18f0 0x7f59400eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f59400eb040 0x7f59400eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59400ee3e0 0x7f59400ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400edee0 con 0x7f59400d18f0 2021-02-10T11:51:47.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400ebaa0 con 0x7f59400eb040 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.766 7f5945c54700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59400d1cf0 con 0x7f59400ee3e0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f59400d18f0 0x7f59400eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f59400d18f0 0x7f59400eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55812/0 (socket says 172.21.2.117:55812) 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 -- 172.21.2.117:0/697002772 learned_addr learned my addr 172.21.2.117:0/697002772 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593ffff700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59400ee3e0 0x7f59400ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593effd700 1 --2- 172.21.2.117:0/697002772 >> v2:172.21.2.125:3300/0 conn(0x7f59400eb040 0x7f59400eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 -- 172.21.2.117:0/697002772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59400ee3e0 msgr2=0x7f59400ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59400ee3e0 0x7f59400ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 -- 172.21.2.117:0/697002772 >> v2:172.21.2.125:3300/0 conn(0x7f59400eb040 msgr2=0x7f59400eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 --2- 172.21.2.117:0/697002772 >> v2:172.21.2.125:3300/0 conn(0x7f59400eb040 0x7f59400eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 -- 172.21.2.117:0/697002772 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400eee00 con 0x7f59400d18f0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593f7fe700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59400d18f0 0x7f59400eaa20 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7f5930006700 tx=0x7f5930009b80).ready entity=mon.0 client_cookie=1c9cbe0c7a913e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:47.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593e7fc700 1 -- 172.21.2.117:0/697002772 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5930000c90 con 0x7f59400d18f0 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593e7fc700 1 -- 172.21.2.117:0/697002772 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593000c0c0 con 0x7f59400d18f0 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f593e7fc700 1 -- 172.21.2.117:0/697002772 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f593000c5a0 con 0x7f59400d18f0 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f5945c54700 1 -- 172.21.2.117:0/697002772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59400d18f0 msgr2=0x7f59400eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.767 7f5945c54700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59400d18f0 0x7f59400eaa20 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 rx=0x7f5930006700 tx=0x7f5930009b80).stop 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 -- 172.21.2.117:0/697002772 shutdown_connections 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 --2- 172.21.2.117:0/697002772 >> v2:172.21.2.125:3300/0 conn(0x7f59400eb040 0x7f59400eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59400ee3e0 0x7f59400ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 --2- 172.21.2.117:0/697002772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59400d18f0 0x7f59400eaa20 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 -- 172.21.2.117:0/697002772 >> 172.21.2.117:0/697002772 conn(0x7f59400cf060 msgr2=0x7f59400cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 -- 172.21.2.117:0/697002772 shutdown_connections 2021-02-10T11:51:47.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.768 7f5945c54700 1 -- 172.21.2.117:0/697002772 wait complete. 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.782 7f5945c54700 1 Processor -- start 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.782 7f5945c54700 1 -- start start 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5940083990 0x7f59400863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59401a4180 0x7f59400dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940085e50 con 0x7f59401a4580 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5940097270 con 0x7f5940083990 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f5945c54700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5940097450 con 0x7f59401a4180 2021-02-10T11:51:47.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55816/0 (socket says 172.21.2.117:55816) 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 -- 172.21.2.117:0/2807713291 learned_addr learned my addr 172.21.2.117:0/2807713291 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 -- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59401a4180 msgr2=0x7f59400dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59401a4180 0x7f59400dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 -- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5940083990 msgr2=0x7f59400863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5940083990 0x7f59400863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.783 7f593ffff700 1 -- 172.21.2.117:0/2807713291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594009ed70 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f593ffff700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f59380099a0 tx=0x7f593800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59380066f0 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938003030 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f59380013e0 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5945c54700 1 -- 172.21.2.117:0/2807713291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594009ed70 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5945c54700 1 -- 172.21.2.117:0/2807713291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594009f2c0 con 0x7f59401a4580 2021-02-10T11:51:47.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.784 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f59380011f0 con 0x7f59401a4580 2021-02-10T11:51:47.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.785 7f5937fff700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 0x7f592401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:47.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.785 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5938032880 con 0x7f59401a4580 2021-02-10T11:51:47.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.785 7f593f7fe700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 0x7f592401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:47.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.785 7f593f7fe700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 0x7f592401abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:47.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.817 7f5945c54700 1 -- 172.21.2.117:0/2807713291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594009fb50 con 0x7f59401a4580 2021-02-10T11:51:47.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.819 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5938000f90 con 0x7f59401a4580 2021-02-10T11:51:47.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.967 7f5945c54700 1 -- 172.21.2.117:0/2807713291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f594009fb50 con 0x7f59401a4580 2021-02-10T11:51:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.971 7f5937fff700 1 -- 172.21.2.117:0/2807713291 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f593801f5f0 con 0x7f59401a4580 2021-02-10T11:51:47.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.975 7f5945c54700 1 -- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 msgr2=0x7f592401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.975 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 0x7f592401abc0 crc :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 -- 172.21.2.117:0/2807713291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 msgr2=0x7f59400dc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f59380099a0 tx=0x7f593800bcf0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 -- 172.21.2.117:0/2807713291 shutdown_connections 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f59240186c0 0x7f592401abc0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f59401a4580 0x7f59400dc460 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f59401a4180 0x7f59400dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 --2- 172.21.2.117:0/2807713291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5940083990 0x7f59400863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:47.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 -- 172.21.2.117:0/2807713291 >> 172.21.2.117:0/2807713291 conn(0x7f59400b6540 msgr2=0x7f594008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:47.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 -- 172.21.2.117:0/2807713291 shutdown_connections 2021-02-10T11:51:47.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:47.976 7f5945c54700 1 -- 172.21.2.117:0/2807713291 wait complete. 2021-02-10T11:51:47.985 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:51:47.985 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:51:48.330 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:51:48.673 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:51:48.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:48.673 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:48.673 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:51:49.008 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:51:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:49.008 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:51:49.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:49.008 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:49.008 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:51:49.343 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:51:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:49.344 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:51:49.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:49.344 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:49.344 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:51:49.678 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:51:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:49.678 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:51:49.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:51:49.679 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:51:49.679 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:51:50.013 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:51:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:51:50.014 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:51:50.014 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.426 7fd14eb0c700 1 Processor -- start 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.426 7fd14eb0c700 1 -- start start 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14eb0c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd1480d18e0 0x7fd1480ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14eb0c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd1480ef3f0 0x7fd1481b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14eb0c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1480ede60 con 0x7fd1480eb000 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14eb0c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480ef7f0 con 0x7fd1480d18e0 2021-02-10T11:51:50.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14eb0c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480d1ce0 con 0x7fd1480ef3f0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14c928700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd1480d18e0 0x7fd1480ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd147fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd1480eb000 0x7fd1480eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56076/0 (socket says 172.21.2.117:56076) 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd147fff700 1 -- 172.21.2.117:0/3748181634 learned_addr learned my addr 172.21.2.117:0/3748181634 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14c928700 1 -- 172.21.2.117:0/3748181634 >> v2:172.21.2.125:3300/0 conn(0x7fd1480ef3f0 msgr2=0x7fd1481b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14c928700 1 --2- 172.21.2.117:0/3748181634 >> v2:172.21.2.125:3300/0 conn(0x7fd1480ef3f0 0x7fd1481b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14c928700 1 -- 172.21.2.117:0/3748181634 >> v1:172.21.2.117:6789/0 conn(0x7fd1480eb000 legacy=0x7fd1480eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd1477fe700 1 -- 172.21.2.117:0/3748181634 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792038580 0 0) 0x7fd138000ce0 con 0x7fd1480eb000 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.427 7fd14c928700 1 -- 172.21.2.117:0/3748181634 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1481b8f90 con 0x7fd1480d18e0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd14c928700 1 --2- 172.21.2.117:0/3748181634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480d18e0 0x7fd1480ea9e0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7fd130006650 tx=0x7fd130009a80).ready entity=mon.0 client_cookie=184ad4295032eb11 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd1477fe700 1 -- 172.21.2.117:0/3748181634 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd13000c0c0 con 0x7fd1480d18e0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd1477fe700 1 -- 172.21.2.117:0/3748181634 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13000c440 con 0x7fd1480d18e0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd1477fe700 1 -- 172.21.2.117:0/3748181634 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd13000c960 con 0x7fd1480d18e0 2021-02-10T11:51:50.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd14eb0c700 1 -- 172.21.2.117:0/3748181634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480d18e0 msgr2=0x7fd1480ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.428 7fd14eb0c700 1 --2- 172.21.2.117:0/3748181634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480d18e0 0x7fd1480ea9e0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 rx=0x7fd130006650 tx=0x7fd130009a80).stop 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 -- 172.21.2.117:0/3748181634 shutdown_connections 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 --2- 172.21.2.117:0/3748181634 >> v2:172.21.2.125:3300/0 conn(0x7fd1480ef3f0 0x7fd1481b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 --2- 172.21.2.117:0/3748181634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480d18e0 0x7fd1480ea9e0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 -- 172.21.2.117:0/3748181634 >> 172.21.2.117:0/3748181634 conn(0x7fd1480ced60 msgr2=0x7fd1480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 -- 172.21.2.117:0/3748181634 shutdown_connections 2021-02-10T11:51:50.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.429 7fd14eb0c700 1 -- 172.21.2.117:0/3748181634 wait complete. 2021-02-10T11:51:50.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.443 7fd14eb0c700 1 Processor -- start 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.443 7fd14eb0c700 1 -- start start 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14808a8a0 0x7fd148095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1480960c0 0x7fd1480e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480964c0 con 0x7fd1480ac7b0 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480e7bd0 con 0x7fd14808a8a0 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14eb0c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480e7df0 con 0x7fd1480960c0 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55832/0 (socket says 172.21.2.117:55832) 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 -- 172.21.2.117:0/1916452128 learned_addr learned my addr 172.21.2.117:0/1916452128 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 -- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1480960c0 msgr2=0x7fd1480e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1480960c0 0x7fd1480e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 -- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14808a8a0 msgr2=0x7fd148095aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14808a8a0 0x7fd148095aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.444 7fd14c928700 1 -- 172.21.2.117:0/1916452128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480e8010 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd14c928700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 rx=0x7fd13000b660 tx=0x7fd13000ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd130000dd0 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130000990 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd130002030 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1480e8010 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.445 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14808b2b0 con 0x7fd1480ac7b0 2021-02-10T11:51:50.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.446 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1300091d0 con 0x7fd1480ac7b0 2021-02-10T11:51:50.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.446 7fd1457fa700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 0x7fd12c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:50.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.446 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd130032740 con 0x7fd1480ac7b0 2021-02-10T11:51:50.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.446 7fd147fff700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 0x7fd12c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:50.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.447 7fd147fff700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 0x7fd12c01abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:50.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.476 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd148087210 con 0x7fd1480ac7b0 2021-02-10T11:51:50.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.479 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd13001f030 con 0x7fd1480ac7b0 2021-02-10T11:51:50.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.604 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd1480ede60 con 0x7fd1480ac7b0 2021-02-10T11:51:50.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.605 7fd1457fa700 1 -- 172.21.2.117:0/1916452128 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v125) v1 ==== 81+0+30766 (secure 0 0 0) 0x7fd13001f030 con 0x7fd1480ac7b0 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 125, 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:51:14.819978", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:51:50.607 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 27, 2021-02-10T11:51:50.608 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:50.609 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:50.610 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:50.611 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:50.612 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:50.613 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 26, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:51:11.788181", 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:50.614 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:50.615 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "124", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:50.616 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:50.617 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:50.618 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:51:50.619 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 27, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:51:12.863682", 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:51:50.620 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:51:50.621 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "125", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:51:50.622 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:51:50.623 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:51:50.624 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.625 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.626 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:51:50.627 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:51:50.628 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.629 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:51:50.630 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.631 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 122, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.633 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.634 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.635 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.636 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:51:50.637 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.638 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.639 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.640 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:51:50.641 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.642 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.643 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.645 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.646 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:51:50.648 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.651 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.652 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:51:50.653 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.654 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:51:50.656 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.658 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:51:50.659 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.660 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.661 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.662 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.664 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.665 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:51:50.666 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 122, 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:51:50.667 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.669 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.670 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:51:50.671 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.672 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.673 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.674 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.675 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:51:50.676 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:51:50.677 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:51:50.678 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:51:50.679 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:50.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.606 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 msgr2=0x7fd12c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.606 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 0x7fd12c01abc0 crc :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.606 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 msgr2=0x7fd14808a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.606 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 rx=0x7fd13000b660 tx=0x7fd13000ba00).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 shutdown_connections 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1480960c0 0x7fd1480e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd14808a8a0 0x7fd148095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd12c0186c0 0x7fd12c01abc0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 --2- 172.21.2.117:0/1916452128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1480ac7b0 0x7fd14808a280 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 >> 172.21.2.117:0/1916452128 conn(0x7fd14819f610 msgr2=0x7fd1480e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 shutdown_connections 2021-02-10T11:51:50.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:50.607 7fd14eb0c700 1 -- 172.21.2.117:0/1916452128 wait complete. 2021-02-10T11:51:50.682 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 Processor -- start 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 -- start start 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff3e80e4e40 0x7ff3e81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e80d1db0 con 0x7ff3e80e5240 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e8085930 con 0x7ff3e8085450 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.092 7ff3ee7fc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e8084840 con 0x7ff3e80e4e40 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff3e80e5240 0x7ff3e8084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56090/0 (socket says 172.21.2.117:56090) 2021-02-10T11:51:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e77fe700 1 -- 172.21.2.117:0/3101210254 learned_addr learned my addr 172.21.2.117:0/3101210254 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e7fff700 1 --2- 172.21.2.117:0/3101210254 >> v2:172.21.2.125:3300/0 conn(0x7ff3e80e4e40 0x7ff3e81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520023249 0 0) 0x7ff3dc001180 con 0x7ff3e80e5240 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d0002fe0 con 0x7ff3e80e5240 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e7fff700 1 -- 172.21.2.117:0/3101210254 >> v1:172.21.2.125:6789/0 conn(0x7ff3e8085450 legacy=0x7ff3e80d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937306372 0 0) 0x7ff3e0001400 con 0x7ff3e8085450 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e7fff700 1 -- 172.21.2.117:0/3101210254 >> v1:172.21.2.117:6789/0 conn(0x7ff3e80e5240 legacy=0x7ff3e8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.093 7ff3e7fff700 1 -- 172.21.2.117:0/3101210254 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e81b0810 con 0x7ff3e80e4e40 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3e7fff700 1 --2- 172.21.2.117:0/3101210254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e80e4e40 0x7ff3e81b01f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7ff3d8006650 tx=0x7ff3d8009a80).ready entity=mon.1 client_cookie=8a467133844c6cc4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d80009b0 con 0x7ff3e80e4e40 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d800c0c0 con 0x7ff3e80e4e40 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3e6ffd700 1 -- 172.21.2.117:0/3101210254 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3d800c5e0 con 0x7ff3e80e4e40 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3ee7fc700 1 -- 172.21.2.117:0/3101210254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e80e4e40 msgr2=0x7ff3e81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.094 7ff3ee7fc700 1 --2- 172.21.2.117:0/3101210254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e80e4e40 0x7ff3e81b01f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7ff3d8006650 tx=0x7ff3d8009a80).stop 2021-02-10T11:51:51.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.095 7ff3ee7fc700 1 -- 172.21.2.117:0/3101210254 shutdown_connections 2021-02-10T11:51:51.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.095 7ff3ee7fc700 1 --2- 172.21.2.117:0/3101210254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e80e4e40 0x7ff3e81b01f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.095 7ff3ee7fc700 1 -- 172.21.2.117:0/3101210254 >> 172.21.2.117:0/3101210254 conn(0x7ff3e80ce4e0 msgr2=0x7ff3e80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:51.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.095 7ff3ee7fc700 1 -- 172.21.2.117:0/3101210254 shutdown_connections 2021-02-10T11:51:51.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.095 7ff3ee7fc700 1 -- 172.21.2.117:0/3101210254 wait complete. 2021-02-10T11:51:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.109 7ff3ee7fc700 1 Processor -- start 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.109 7ff3ee7fc700 1 -- start start 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e819e470 0x7ff3e818cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3e8095330 0x7ff3e8190810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e8190290 con 0x7ff3e8095730 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e819e870 con 0x7ff3e819e470 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ee7fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e818d230 con 0x7ff3e8095330 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55846/0 (socket says 172.21.2.117:55846) 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 -- 172.21.2.117:0/4083214388 learned_addr learned my addr 172.21.2.117:0/4083214388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 -- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3e8095330 msgr2=0x7ff3e8190810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3e8095330 0x7ff3e8190810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 -- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e819e470 msgr2=0x7ff3e818cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e819e470 0x7ff3e818cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 -- 172.21.2.117:0/4083214388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e818fef0 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.110 7ff3ece19700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7ff3e0009940 tx=0x7ff3e000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e0000f40 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e000c2f0 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3e000c750 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e818fef0 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e818fef0 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff3e0000d30 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 0x7ff3c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.111 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff3e0032af0 con 0x7ff3e8095730 2021-02-10T11:51:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.112 7ff3e7fff700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 0x7ff3c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.112 7ff3e7fff700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 0x7ff3c801abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.142 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e810c250 con 0x7ff3e8095730 2021-02-10T11:51:51.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.145 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff3e000be30 con 0x7ff3e8095730 2021-02-10T11:51:51.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.271 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3e810c250 con 0x7ff3e8095730 2021-02-10T11:51:51.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.271 7ff3e4ff9700 1 -- 172.21.2.117:0/4083214388 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+2106 (secure 0 0 0) 0x7ff3e0000d30 con 0x7ff3e8095730 2021-02-10T11:51:51.273 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:51.273 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":103,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:15.824496","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11124":{"gid":11124,"name":"b","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1123044872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1123044872},{"type":"v1","addr":"172.21.2.125:6835","nonce":1123044872}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:51.273 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 103 2021-02-10T11:51:51.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 msgr2=0x7ff3c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 0x7ff3c801abc0 crc :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 msgr2=0x7ff3e8190e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 rx=0x7ff3e0009940 tx=0x7ff3e000bcf0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 shutdown_connections 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3c80186c0 0x7ff3c801abc0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3e8095730 0x7ff3e8190e30 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3e8095330 0x7ff3e8190810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 --2- 172.21.2.117:0/4083214388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3e819e470 0x7ff3e818cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:51.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.273 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 >> 172.21.2.117:0/4083214388 conn(0x7ff3e810de60 msgr2=0x7ff3e81af660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:51.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.274 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 shutdown_connections 2021-02-10T11:51:51.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.274 7ff3ee7fc700 1 -- 172.21.2.117:0/4083214388 wait complete. 2021-02-10T11:51:51.284 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 Processor -- start 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 -- start start 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f01601b07f0 0x7f01600e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01600e6170 0x7f01600e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600e77f0 con 0x7f01600e6b90 2021-02-10T11:51:51.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600e5ce0 con 0x7f01601b07f0 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.695 7f016534d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600e54b0 con 0x7f01600e6170 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f01601b07f0 0x7f01600e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f01601b07f0 0x7f01600e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55854/0 (socket says 172.21.2.117:55854) 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 -- 172.21.2.117:0/4184591732 learned_addr learned my addr 172.21.2.117:0/4184591732 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f87e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f01600e6b90 0x7f01600e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:49030/0 (socket says 172.21.2.117:49030) 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015e87c700 1 --2- 172.21.2.117:0/4184591732 >> v2:172.21.2.125:3300/0 conn(0x7f01600e6170 0x7f01600e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 -- 172.21.2.117:0/4184591732 >> v2:172.21.2.125:3300/0 conn(0x7f01600e6170 msgr2=0x7f01600e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 --2- 172.21.2.117:0/4184591732 >> v2:172.21.2.125:3300/0 conn(0x7f01600e6170 0x7f01600e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 -- 172.21.2.117:0/4184591732 >> v1:172.21.2.125:6789/0 conn(0x7f01600e6b90 legacy=0x7f01600e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.696 7f015f07d700 1 -- 172.21.2.117:0/4184591732 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01600e8d80 con 0x7f01601b07f0 2021-02-10T11:51:51.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f015f07d700 1 --2- 172.21.2.117:0/4184591732 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01601b07f0 0x7f01600e4e90 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 rx=0x7f0150006700 tx=0x7f0150009b00).ready entity=mon.0 client_cookie=6d7899545b544876 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f015e07b700 1 -- 172.21.2.117:0/4184591732 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c0c0 con 0x7f01601b07f0 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f015e07b700 1 -- 172.21.2.117:0/4184591732 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000c400 con 0x7f01601b07f0 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f015e07b700 1 -- 172.21.2.117:0/4184591732 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f015000c920 con 0x7f01601b07f0 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f016534d700 1 -- 172.21.2.117:0/4184591732 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01601b07f0 msgr2=0x7f01600e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f016534d700 1 --2- 172.21.2.117:0/4184591732 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01601b07f0 0x7f01600e4e90 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 rx=0x7f0150006700 tx=0x7f0150009b00).stop 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f016534d700 1 -- 172.21.2.117:0/4184591732 shutdown_connections 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f016534d700 1 --2- 172.21.2.117:0/4184591732 >> v2:172.21.2.125:3300/0 conn(0x7f01600e6170 0x7f01600e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.697 7f016534d700 1 --2- 172.21.2.117:0/4184591732 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01601b07f0 0x7f01600e4e90 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.698 7f016534d700 1 -- 172.21.2.117:0/4184591732 >> 172.21.2.117:0/4184591732 conn(0x7f01600ce500 msgr2=0x7f01600e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:51.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.698 7f016534d700 1 -- 172.21.2.117:0/4184591732 shutdown_connections 2021-02-10T11:51:51.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.698 7f016534d700 1 -- 172.21.2.117:0/4184591732 wait complete. 2021-02-10T11:51:51.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.712 7f016534d700 1 Processor -- start 2021-02-10T11:51:51.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.712 7f016534d700 1 -- start start 2021-02-10T11:51:51.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 0x7f016008bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f016008c780 0x7f0160199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016008cca0 con 0x7f016008c380 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160199c90 con 0x7f016008c780 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f016534d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0160199e70 con 0x7f01600af5f0 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55860/0 (socket says 172.21.2.117:55860) 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 -- 172.21.2.117:0/1922008151 learned_addr learned my addr 172.21.2.117:0/1922008151 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015f87e700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f016008c780 0x7f0160199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015f07d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 0x7f016008bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 -- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 msgr2=0x7f016008bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 0x7f016008bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 -- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f016008c780 msgr2=0x7f0160199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f016008c780 0x7f0160199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015e87c700 1 -- 172.21.2.117:0/1922008151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016009d790 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.713 7f015f07d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 0x7f016008bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f015e87c700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f0148006f60 tx=0x7f014800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f014800bd60 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0148000c90 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01480022b0 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f016534d700 1 -- 172.21.2.117:0/1922008151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016009d790 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.714 7f016534d700 1 -- 172.21.2.117:0/1922008151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016009dd20 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.715 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0148006b00 con 0x7f016008c380 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.715 7f0157fff700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 0x7f014401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:51.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.715 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0148032680 con 0x7f016008c380 2021-02-10T11:51:51.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.715 7f015f07d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 0x7f014401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:51.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.715 7f015f07d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 0x7f014401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:51.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.745 7f016534d700 1 -- 172.21.2.117:0/1922008151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0160106e10 con 0x7f016008c380 2021-02-10T11:51:51.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.748 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0148000940 con 0x7f016008c380 2021-02-10T11:51:51.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:51.874 7f016534d700 1 -- 172.21.2.117:0/1922008151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0160106e10 con 0x7f016008c380 2021-02-10T11:51:52.664 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:51:52.663 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:51:52.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.670 7f0157fff700 1 -- 172.21.2.117:0/1922008151 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0148006b00 con 0x7f016008c380 2021-02-10T11:51:52.696 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 msgr2=0x7f014401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 0x7f014401abc0 crc :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 msgr2=0x7f0160199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 rx=0x7f0148006f60 tx=0x7f014800b970).stop 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 shutdown_connections 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01440186c0 0x7f014401abc0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f016008c380 0x7f0160199050 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f016008c780 0x7f0160199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 --2- 172.21.2.117:0/1922008151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01600af5f0 0x7f016008bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 >> 172.21.2.117:0/1922008151 conn(0x7f01600953a0 msgr2=0x7f016009e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 shutdown_connections 2021-02-10T11:51:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:52.697 7f016534d700 1 -- 172.21.2.117:0/1922008151 wait complete. 2021-02-10T11:51:52.706 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:51:52.722 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 Processor -- start 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 -- start start 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f04a80d18e0 0x7f04a80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a80ede60 con 0x7f04a80ef3f0 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a80d1ce0 con 0x7f04a80eb000 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04ad4d1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a80ef7f0 con 0x7f04a80d18e0 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04a7ac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f04a80ef3f0 0x7f04a81b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56116/0 (socket says 172.21.2.117:56116) 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.117 7f04a7ac0700 1 -- 172.21.2.117:0/3299814063 learned_addr learned my addr 172.21.2.117:0/3299814063 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:53.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055207738 0 0) 0x7f04a0001500 con 0x7f04a80ef3f0 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0494002fe0 con 0x7f04a80ef3f0 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3825084419 0 0) 0x7f04a0001460 con 0x7f04a80ef3f0 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 >> v2:172.21.2.125:3300/0 conn(0x7f04a80d18e0 msgr2=0x7f04a80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 --2- 172.21.2.117:0/3299814063 >> v2:172.21.2.125:3300/0 conn(0x7f04a80d18e0 0x7f04a80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 >> v1:172.21.2.125:6789/0 conn(0x7f04a80eb000 legacy=0x7f04a80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a81bb180 con 0x7f04a80ef3f0 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f04a0001980 con 0x7f04a80ef3f0 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 >> v1:172.21.2.117:6789/0 conn(0x7f04a80ef3f0 legacy=0x7f04a81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0494003a60 0x7f0494005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0494006790 0x7f0494008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 0x7f049400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049400c300 con 0x7f0494009590 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049400c520 con 0x7f0494003a60 2021-02-10T11:51:53.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049400c740 con 0x7f0494006790 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 0x7f049400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a72bf700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0494003a60 0x7f0494005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 -- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0494006790 msgr2=0x7f0494008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0494006790 0x7f0494008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 -- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0494003a60 msgr2=0x7f0494005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0494003a60 0x7f0494005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a7ac0700 1 -- 172.21.2.117:0/3299814063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049400c9e0 con 0x7f0494009590 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.118 7f04a72bf700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0494003a60 0x7f0494005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04a7ac0700 1 -- 172.21.2.117:0/3299814063 reap_dead start 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04a7ac0700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 0x7f049400bab0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f04a0007ce0 tx=0x7f04a0050310).ready entity=mon.0 client_cookie=ce107b1e7fc7962d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04a0007dd0 con 0x7f0494009590 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0007030 con 0x7f0494009590 2021-02-10T11:51:53.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04a62bd700 1 -- 172.21.2.117:0/3299814063 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04a0007550 con 0x7f0494009590 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 -- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 msgr2=0x7f049400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 0x7f049400bab0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 rx=0x7f04a0007ce0 tx=0x7f04a0050310).stop 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 -- 172.21.2.117:0/3299814063 shutdown_connections 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 --2- 172.21.2.117:0/3299814063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0494009590 0x7f049400bab0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 -- 172.21.2.117:0/3299814063 >> 172.21.2.117:0/3299814063 conn(0x7f04a80ced60 msgr2=0x7f04a80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 -- 172.21.2.117:0/3299814063 shutdown_connections 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.119 7f04ad4d1700 1 -- 172.21.2.117:0/3299814063 wait complete. 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 Processor -- start 2021-02-10T11:51:53.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 -- start start 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04a808c6f0 0x7f04a81a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04a808b220 0x7f04a808b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a80a51e0 con 0x7f04a808caf0 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a808f720 con 0x7f04a808b220 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04ad4d1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a81a7e30 con 0x7f04a808c6f0 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04a6abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04a6abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55876/0 (socket says 172.21.2.117:55876) 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.134 7f04a6abe700 1 -- 172.21.2.117:0/1981562283 learned_addr learned my addr 172.21.2.117:0/1981562283 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:53.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 -- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04a808c6f0 msgr2=0x7f04a81a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04a808c6f0 0x7f04a81a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 -- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04a808b220 msgr2=0x7f04a808b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04a808b220 0x7f04a808b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 -- 172.21.2.117:0/1981562283 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a808bc40 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04a6abe700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7f0498006f00 tx=0x7f049800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04980009b0 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049800bd30 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04980020d0 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a808bc40 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.135 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a808c1d0 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.136 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0498007000 con 0x7f04a808caf0 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.136 7f049ffff700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 0x7f048c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.136 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f0498032770 con 0x7f04a808caf0 2021-02-10T11:51:53.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.136 7f04a72bf700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 0x7f048c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.137 7f04a72bf700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 0x7f048c01abc0 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.167 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a80ede60 con 0x7f04a808caf0 2021-02-10T11:51:53.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.170 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04980009b0 con 0x7f04a808caf0 2021-02-10T11:51:53.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.297 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04a80ede60 con 0x7f04a808caf0 2021-02-10T11:51:53.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.297 7f049ffff700 1 -- 172.21.2.117:0/1981562283 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1759 (secure 0 0 0) 0x7f0498007000 con 0x7f04a808caf0 2021-02-10T11:51:53.299 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:53.299 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":104,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11127,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/810156774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":810156774},{"type":"v1","addr":"172.21.2.117:6833","nonce":810156774}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":98}],"filesystems":[{"mdsmap":{"epoch":104,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:51:13.818137","modified":"2021-02-10 11:51:52.664630","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2021-02-10T11:51:53.300 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 104 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.299 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 msgr2=0x7f048c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.299 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 0x7f048c01abc0 crc :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.299 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 msgr2=0x7f04a81a7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.299 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 rx=0x7f0498006f00 tx=0x7f049800b940).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 shutdown_connections 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f048c0186c0 0x7f048c01abc0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04a808caf0 0x7f04a81a7810 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04a808b220 0x7f04a808b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 --2- 172.21.2.117:0/1981562283 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04a808c6f0 0x7f04a81a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 >> 172.21.2.117:0/1981562283 conn(0x7f04a80bb5e0 msgr2=0x7f04a80e9440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:53.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 shutdown_connections 2021-02-10T11:51:53.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.300 7f04ad4d1700 1 -- 172.21.2.117:0/1981562283 wait complete. 2021-02-10T11:51:53.309 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.726 7f2ba1fd9700 1 Processor -- start 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.726 7f2ba1fd9700 1 -- start start 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2ba1fd9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2ba1fd9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c0e8d80 0x7f2b9c0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2ba1fd9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9c0e77f0 con 0x7f2b9c0e6170 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2ba1fd9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9c0e54b0 con 0x7f2b9c1b07f0 2021-02-10T11:51:53.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2ba1fd9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0e9180 con 0x7f2b9c0e8d80 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55884/0 (socket says 172.21.2.117:55884) 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 -- 172.21.2.117:0/2409570137 learned_addr learned my addr 172.21.2.117:0/2409570137 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 -- 172.21.2.117:0/2409570137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c0e8d80 msgr2=0x7f2b9c0e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 --2- 172.21.2.117:0/2409570137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c0e8d80 0x7f2b9c0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 -- 172.21.2.117:0/2409570137 >> v1:172.21.2.125:6789/0 conn(0x7f2b9c0e6170 legacy=0x7f2b9c0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.727 7f2b9b7fe700 1 -- 172.21.2.117:0/2409570137 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c0e7380 con 0x7f2b9c1b07f0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2b9b7fe700 1 --2- 172.21.2.117:0/2409570137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7f2b84006700 tx=0x7f2b84009b80).ready entity=mon.0 client_cookie=1bc992072dfe6ad3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2b9a7fc700 1 -- 172.21.2.117:0/2409570137 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b8400c0c0 con 0x7f2b9c1b07f0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2b9a7fc700 1 -- 172.21.2.117:0/2409570137 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8400c340 con 0x7f2b9c1b07f0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2b9a7fc700 1 -- 172.21.2.117:0/2409570137 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b8400c860 con 0x7f2b9c1b07f0 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 -- 172.21.2.117:0/2409570137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c1b07f0 msgr2=0x7f2b9c0e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 --2- 172.21.2.117:0/2409570137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7f2b84006700 tx=0x7f2b84009b80).stop 2021-02-10T11:51:53.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 -- 172.21.2.117:0/2409570137 shutdown_connections 2021-02-10T11:51:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 --2- 172.21.2.117:0/2409570137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c0e8d80 0x7f2b9c0e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 --2- 172.21.2.117:0/2409570137 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c1b07f0 0x7f2b9c0e4e90 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.728 7f2ba1fd9700 1 -- 172.21.2.117:0/2409570137 >> 172.21.2.117:0/2409570137 conn(0x7f2b9c0ce500 msgr2=0x7f2b9c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.729 7f2ba1fd9700 1 -- 172.21.2.117:0/2409570137 shutdown_connections 2021-02-10T11:51:53.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.729 7f2ba1fd9700 1 -- 172.21.2.117:0/2409570137 wait complete. 2021-02-10T11:51:53.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.743 7f2ba1fd9700 1 Processor -- start 2021-02-10T11:51:53.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.743 7f2ba1fd9700 1 -- start start 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c10bcb0 0x7f2b9c0b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b9c08f180 0x7f2b9c08f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c10c1e0 con 0x7f2b9c10c760 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0b3e60 con 0x7f2b9c08f180 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9c0b4080 con 0x7f2b9c10bcb0 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55888/0 (socket says 172.21.2.117:55888) 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 -- 172.21.2.117:0/447116223 learned_addr learned my addr 172.21.2.117:0/447116223 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 -- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c10bcb0 msgr2=0x7f2b9c0b3840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c10bcb0 0x7f2b9c0b3840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 -- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b9c08f180 msgr2=0x7f2b9c08f580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:53.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b9c08f180 0x7f2b9c08f580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 -- 172.21.2.117:0/447116223 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c08fba0 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b9b7fe700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f2b8400b5e0 tx=0x7f2b8400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b8400bd70 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b84002030 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.744 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9c08fba0 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9c08fba0 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2b84002550 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2b84000c90 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2b93fff700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 0x7f2b8001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f2b84032800 con 0x7f2b9c10c760 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.745 7f2b9affd700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 0x7f2b8001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:53.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.746 7f2b9affd700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 0x7f2b8001ac10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:53.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.777 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9c108a30 con 0x7f2b9c10c760 2021-02-10T11:51:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.779 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2b8401f510 con 0x7f2b9c10c760 2021-02-10T11:51:53.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:53.907 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2b9c0e77f0 con 0x7f2b9c10c760 2021-02-10T11:51:54.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.678 7f2b93fff700 1 -- 172.21.2.117:0/447116223 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2b84000c90 con 0x7f2b9c10c760 2021-02-10T11:51:54.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.729 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 msgr2=0x7f2b8001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:54.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.729 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 0x7f2b8001ac10 crc :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:54.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.729 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 msgr2=0x7f2b9c0b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.729 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 rx=0x7f2b8400b5e0 tx=0x7f2b8400b980).stop 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 shutdown_connections 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2b9c08f180 0x7f2b9c08f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2b9c10bcb0 0x7f2b9c0b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b80018730 0x7f2b8001ac10 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 --2- 172.21.2.117:0/447116223 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2b9c10c760 0x7f2b9c0b3220 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 >> 172.21.2.117:0/447116223 conn(0x7f2b9c198490 msgr2=0x7f2b9c111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 shutdown_connections 2021-02-10T11:51:54.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:54.730 7f2ba1fd9700 1 -- 172.21.2.117:0/447116223 wait complete. 2021-02-10T11:51:54.738 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:51:55.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 Processor -- start 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 -- start start 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0e5240 0x7f4e6c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c085930 con 0x7f4e6c084c60 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c1b09e0 con 0x7f4e6c0e4e40 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e72dd0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0e5cc0 con 0x7f4e6c0e5240 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e713ed700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4e6c084c60 0x7f4e6c085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56148/0 (socket says 172.21.2.117:56148) 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e713ed700 1 -- 172.21.2.117:0/3599905737 learned_addr learned my addr 172.21.2.117:0/3599905737 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251693498 0 0) 0x7f4e64001500 con 0x7f4e6c084c60 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.155 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e58002fe0 con 0x7f4e6c084c60 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2170841283 0 0) 0x7f4e64001270 con 0x7f4e6c084c60 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0e5240 msgr2=0x7f4e6c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:55.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0e5240 0x7f4e6c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 >> v1:172.21.2.125:6789/0 conn(0x7f4e6c0e4e40 legacy=0x7f4e6c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c084840 con 0x7f4e6c084c60 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4e64001b50 con 0x7f4e6c084c60 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 >> v1:172.21.2.117:6789/0 conn(0x7f4e6c084c60 legacy=0x7f4e6c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e58003b90 0x7f4e58005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e580067a0 0x7f4e58008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e58009510 0x7f4e5800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5800c2e0 con 0x7f4e58009510 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5800c500 con 0x7f4e58003b90 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5800c720 con 0x7f4e580067a0 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e58009510 0x7f4e5800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e70bec700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e58003b90 0x7f4e58005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:55.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6bfff700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e580067a0 0x7f4e58008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 -- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e580067a0 msgr2=0x7f4e58008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e580067a0 0x7f4e58008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 -- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e58003b90 msgr2=0x7f4e58005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e58003b90 0x7f4e58005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 -- 172.21.2.117:0/3599905737 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5800c9c0 con 0x7f4e58009510 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6bfff700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e580067a0 0x7f4e58008c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e713ed700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e58009510 0x7f4e5800ba30 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7f4e64007cb0 tx=0x7f4e64050310).ready entity=mon.0 client_cookie=5f631ae24969156a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e64001e80 con 0x7f4e58009510 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64007280 con 0x7f4e58009510 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e6b7fe700 1 -- 172.21.2.117:0/3599905737 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e640090d0 con 0x7f4e58009510 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e72dd0700 1 -- 172.21.2.117:0/3599905737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e58009510 msgr2=0x7f4e5800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.156 7f4e72dd0700 1 --2- 172.21.2.117:0/3599905737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e58009510 0x7f4e5800ba30 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 rx=0x7f4e64007cb0 tx=0x7f4e64050310).stop 2021-02-10T11:51:55.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.157 7f4e713ed700 1 -- 172.21.2.117:0/3599905737 reap_dead start 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.157 7f4e72dd0700 1 -- 172.21.2.117:0/3599905737 shutdown_connections 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.157 7f4e72dd0700 1 -- 172.21.2.117:0/3599905737 >> 172.21.2.117:0/3599905737 conn(0x7f4e6c0ce4e0 msgr2=0x7f4e6c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.157 7f4e72dd0700 1 -- 172.21.2.117:0/3599905737 shutdown_connections 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.157 7f4e72dd0700 1 -- 172.21.2.117:0/3599905737 wait complete. 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.171 7f4e72dd0700 1 Processor -- start 2021-02-10T11:51:55.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.171 7f4e72dd0700 1 -- start start 2021-02-10T11:51:55.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0a0aa0 0x7f4e6c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e6c10ffa0 0x7f4e6c1103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1109c0 con 0x7f4e6c090ae0 2021-02-10T11:51:55.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c0a0ea0 con 0x7f4e6c10ffa0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e72dd0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1a0360 con 0x7f4e6c0a0aa0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55910/0 (socket says 172.21.2.117:55910) 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 -- 172.21.2.117:0/4221899174 learned_addr learned my addr 172.21.2.117:0/4221899174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 -- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0a0aa0 msgr2=0x7f4e6c19fd40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0a0aa0 0x7f4e6c19fd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 -- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e6c10ffa0 msgr2=0x7f4e6c1103a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e6c10ffa0 0x7f4e6c1103a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 -- 172.21.2.117:0/4221899174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c110d00 con 0x7f4e6c090ae0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.172 7f4e713ed700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f4e64001c30 tx=0x7f4e64051cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e640014d0 con 0x7f4e6c090ae0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64007280 con 0x7f4e6c090ae0 2021-02-10T11:51:55.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e64004720 con 0x7f4e6c090ae0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c110d00 con 0x7f4e6c090ae0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c110d00 con 0x7f4e6c090ae0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e64001400 con 0x7f4e6c090ae0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 0x7f4e5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.173 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f4e64076a80 con 0x7f4e6c090ae0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.174 7f4e70bec700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 0x7f4e5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:55.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.174 7f4e70bec700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 0x7f4e5001abc0 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:55.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.205 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c10c2a0 con 0x7f4e6c090ae0 2021-02-10T11:51:55.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.207 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e64004720 con 0x7f4e6c090ae0 2021-02-10T11:51:55.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.333 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e6c085930 con 0x7f4e6c090ae0 2021-02-10T11:51:55.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.683 7f4e697fa700 1 -- 172.21.2.117:0/4221899174 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v127) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4e64063860 con 0x7f4e6c090ae0 2021-02-10T11:51:55.700 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:51:55.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.701 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 msgr2=0x7f4e5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:55.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.701 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 0x7f4e5001abc0 crc :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:55.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.701 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 msgr2=0x7f4e6c090ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:55.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.701 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 rx=0x7f4e64001c30 tx=0x7f4e64051cc0).stop 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 shutdown_connections 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e500186c0 0x7f4e5001abc0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e6c090ae0 0x7f4e6c090ee0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e6c10ffa0 0x7f4e6c1103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 --2- 172.21.2.117:0/4221899174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e6c0a0aa0 0x7f4e6c19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 >> 172.21.2.117:0/4221899174 conn(0x7f4e6c0a5e90 msgr2=0x7f4e6c0b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 shutdown_connections 2021-02-10T11:51:55.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:55.702 7f4e72dd0700 1 -- 172.21.2.117:0/4221899174 wait complete. 2021-02-10T11:51:55.710 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.126 7f4cf022a700 1 Processor -- start 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.126 7f4cf022a700 1 -- start start 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.126 7f4cf022a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ce80e5240 0x7f4ce8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4cf022a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce8085930 con 0x7f4ce80e4e40 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4cf022a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce81b09e0 con 0x7f4ce8084c60 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4cf022a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce80e5cc0 con 0x7f4ce80e5240 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4ced845700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ce80e5240 0x7f4ce8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4cee046700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4ce80e4e40 0x7f4ce81b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56166/0 (socket says 172.21.2.117:56166) 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4cee046700 1 -- 172.21.2.117:0/1090551182 learned_addr learned my addr 172.21.2.117:0/1090551182 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:56.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4ced845700 1 -- 172.21.2.117:0/1090551182 >> v1:172.21.2.125:6789/0 conn(0x7f4ce8084c60 legacy=0x7f4ce8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4ced845700 1 -- 172.21.2.117:0/1090551182 >> v1:172.21.2.117:6789/0 conn(0x7f4ce80e4e40 legacy=0x7f4ce81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4ced845700 1 -- 172.21.2.117:0/1090551182 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce8084840 con 0x7f4ce80e5240 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.127 7f4ced845700 1 --2- 172.21.2.117:0/1090551182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80e5240 0x7f4ce8086ec0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7f4cd8006650 tx=0x7f4cd8009a80).ready entity=mon.0 client_cookie=7683837f0e4fc185 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4ced044700 1 -- 172.21.2.117:0/1090551182 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd80009b0 con 0x7f4ce80e5240 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4ced044700 1 -- 172.21.2.117:0/1090551182 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd800c0c0 con 0x7f4ce80e5240 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4ced044700 1 -- 172.21.2.117:0/1090551182 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd800c5e0 con 0x7f4ce80e5240 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 -- 172.21.2.117:0/1090551182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80e5240 msgr2=0x7f4ce8086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 --2- 172.21.2.117:0/1090551182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80e5240 0x7f4ce8086ec0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 rx=0x7f4cd8006650 tx=0x7f4cd8009a80).stop 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 -- 172.21.2.117:0/1090551182 shutdown_connections 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 --2- 172.21.2.117:0/1090551182 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80e5240 0x7f4ce8086ec0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 -- 172.21.2.117:0/1090551182 >> 172.21.2.117:0/1090551182 conn(0x7f4ce80ce4e0 msgr2=0x7f4ce80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.128 7f4cf022a700 1 -- 172.21.2.117:0/1090551182 shutdown_connections 2021-02-10T11:51:56.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.129 7f4cf022a700 1 -- 172.21.2.117:0/1090551182 wait complete. 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 Processor -- start 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 -- start start 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce80b7800 0x7f4ce80b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce80b72d0 0x7f4ce80b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80b7ca0 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce80b53c0 con 0x7f4ce80b72d0 2021-02-10T11:51:56.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.143 7f4cf022a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce81b2580 con 0x7f4ce80b7800 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55918/0 (socket says 172.21.2.117:55918) 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 -- 172.21.2.117:0/3150911951 learned_addr learned my addr 172.21.2.117:0/3150911951 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 -- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce80b7800 msgr2=0x7f4ce80b68b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce80b7800 0x7f4ce80b68b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 -- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce80b72d0 msgr2=0x7f4ce80b8840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce80b72d0 0x7f4ce80b8840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 -- 172.21.2.117:0/3150911951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80b8e60 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4ced845700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f4cd8009630 tx=0x7f4cd800b8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd8009360 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8002030 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4cd8002510 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce80b8e60 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.144 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce80b8e60 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.145 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4cd8002210 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.145 7f4cdeffd700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 0x7f4cd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.145 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f4cd8032780 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.145 7f4cee046700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 0x7f4cd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:56.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.146 7f4cee046700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 0x7f4cd001abc0 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:56.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.177 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce80e5eb0 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.179 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4cd8009300 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.306 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ce80e5eb0 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.690 7f4cdeffd700 1 -- 172.21.2.117:0/3150911951 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v128) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4cd8002030 con 0x7f4ce80b6ed0 2021-02-10T11:51:56.723 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:51:56.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.725 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 msgr2=0x7f4cd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:56.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.725 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 0x7f4cd001abc0 crc :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:56.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.725 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 msgr2=0x7f4ce80b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:56.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.725 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 rx=0x7f4cd8009630 tx=0x7f4cd800b8f0).stop 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 shutdown_connections 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4cd00186c0 0x7f4cd001abc0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ce80b6ed0 0x7f4ce80b8220 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ce80b72d0 0x7f4ce80b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 --2- 172.21.2.117:0/3150911951 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ce80b7800 0x7f4ce80b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 >> 172.21.2.117:0/3150911951 conn(0x7f4ce809fb50 msgr2=0x7f4ce8110ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:56.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 shutdown_connections 2021-02-10T11:51:56.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:56.726 7f4cf022a700 1 -- 172.21.2.117:0/3150911951 wait complete. 2021-02-10T11:51:56.734 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:51:56.734 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:51:56.735 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:51:56.735 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:56.735 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:51:56.735 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:51:56.735 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:51:56.735 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:51:56.740 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:51:56.740 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:51:56.741 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:51:56.741 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:51:56.782 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:51:56.782 DEBUG:teuthology.parallel:result is None 2021-02-10T11:51:56.783 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:51:56.784 DEBUG:teuthology.parallel:result is None 2021-02-10T11:51:56.784 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:51:57.188 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:51:57.190 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:51:57.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 Processor -- start 2021-02-10T11:51:57.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 -- start start 2021-02-10T11:51:57.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69781b07f0 0x7f69780e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f69780e6170 0x7f69780e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69780e77f0 con 0x7f69780e6b90 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69780e5ce0 con 0x7f69781b07f0 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697f3ca700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69780e54b0 con 0x7f69780e6170 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697d1e6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69781b07f0 0x7f69780e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697d9e7700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f69780e6b90 0x7f69780e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56192/0 (socket says 172.21.2.117:56192) 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697d9e7700 1 -- 172.21.2.117:0/190410715 learned_addr learned my addr 172.21.2.117:0/190410715 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:57.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.193 7f697c9e5700 1 --2- 172.21.2.117:0/190410715 >> v2:172.21.2.125:3300/0 conn(0x7f69780e6170 0x7f69780e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f696ffff700 1 -- 172.21.2.117:0/190410715 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008958841 0 0) 0x7f6970000d30 con 0x7f69780e6b90 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697d1e6700 1 -- 172.21.2.117:0/190410715 >> v2:172.21.2.125:3300/0 conn(0x7f69780e6170 msgr2=0x7f69780e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697d1e6700 1 --2- 172.21.2.117:0/190410715 >> v2:172.21.2.125:3300/0 conn(0x7f69780e6170 0x7f69780e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697d1e6700 1 -- 172.21.2.117:0/190410715 >> v1:172.21.2.117:6789/0 conn(0x7f69780e6b90 legacy=0x7f69780e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697d1e6700 1 -- 172.21.2.117:0/190410715 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69780d3150 con 0x7f69781b07f0 2021-02-10T11:51:57.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697d1e6700 1 --2- 172.21.2.117:0/190410715 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69781b07f0 0x7f69780e4e90 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f6974006650 tx=0x7f6974009a80).ready entity=mon.0 client_cookie=e12483cec4068b8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f696ffff700 1 -- 172.21.2.117:0/190410715 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69740009b0 con 0x7f69781b07f0 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f696ffff700 1 -- 172.21.2.117:0/190410715 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697400c0c0 con 0x7f69781b07f0 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f696ffff700 1 -- 172.21.2.117:0/190410715 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f697400c520 con 0x7f69781b07f0 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697f3ca700 1 -- 172.21.2.117:0/190410715 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69781b07f0 msgr2=0x7f69780e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697f3ca700 1 --2- 172.21.2.117:0/190410715 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69781b07f0 0x7f69780e4e90 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 rx=0x7f6974006650 tx=0x7f6974009a80).stop 2021-02-10T11:51:57.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697f3ca700 1 -- 172.21.2.117:0/190410715 shutdown_connections 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.194 7f697f3ca700 1 --2- 172.21.2.117:0/190410715 >> v2:172.21.2.125:3300/0 conn(0x7f69780e6170 0x7f69780e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.195 7f697f3ca700 1 --2- 172.21.2.117:0/190410715 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69781b07f0 0x7f69780e4e90 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.195 7f697f3ca700 1 -- 172.21.2.117:0/190410715 >> 172.21.2.117:0/190410715 conn(0x7f69780ce500 msgr2=0x7f69780e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.195 7f697f3ca700 1 -- 172.21.2.117:0/190410715 shutdown_connections 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.195 7f697f3ca700 1 -- 172.21.2.117:0/190410715 wait complete. 2021-02-10T11:51:57.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.209 7f697f3ca700 1 Processor -- start 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.209 7f697f3ca700 1 -- start start 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69780af5f0 0x7f697808bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f697808c380 0x7f6978199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697808cca0 con 0x7f697808c780 2021-02-10T11:51:57.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6978199c90 con 0x7f69780af5f0 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697f3ca700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6978199e70 con 0x7f697808c380 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d9e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d9e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55948/0 (socket says 172.21.2.117:55948) 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d9e7700 1 -- 172.21.2.117:0/4034982274 learned_addr learned my addr 172.21.2.117:0/4034982274 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697c9e5700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f697808c380 0x7f6978199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d1e6700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69780af5f0 0x7f697808bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d9e7700 1 -- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f697808c380 msgr2=0x7f6978199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.210 7f697d9e7700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f697808c380 0x7f6978199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697d9e7700 1 -- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69780af5f0 msgr2=0x7f697808bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697d9e7700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69780af5f0 0x7f697808bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697d9e7700 1 -- 172.21.2.117:0/4034982274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697809d790 con 0x7f697808c780 2021-02-10T11:51:57.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697d9e7700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f6970007f90 tx=0x7f697000abd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6970000cc0 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970007160 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6970007530 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697809d790 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.211 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697809dd20 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.212 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6970007340 con 0x7f697808c780 2021-02-10T11:51:57.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.212 7f696dffb700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 0x7f695801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:57.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.212 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f69700325c0 con 0x7f697808c780 2021-02-10T11:51:57.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.212 7f697d1e6700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 0x7f695801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:57.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.212 7f697d1e6700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 0x7f695801abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:57.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.242 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6978106e10 con 0x7f697808c780 2021-02-10T11:51:57.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.245 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6970001270 con 0x7f697808c780 2021-02-10T11:51:57.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.382 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f6978106e10 con 0x7f697808c780 2021-02-10T11:51:57.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.734 7f696dffb700 1 -- 172.21.2.117:0/4034982274 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v129) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6970007340 con 0x7f697808c780 2021-02-10T11:51:57.747 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:51:57.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 msgr2=0x7f695801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 0x7f695801abc0 crc :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 msgr2=0x7f6978199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:57.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f6970007f90 tx=0x7f697000abd0).stop 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 shutdown_connections 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69580186c0 0x7f695801abc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f697808c780 0x7f6978199670 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f697808c380 0x7f6978199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 --2- 172.21.2.117:0/4034982274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69780af5f0 0x7f697808bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.749 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 >> 172.21.2.117:0/4034982274 conn(0x7f69780953a0 msgr2=0x7f697809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.750 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 shutdown_connections 2021-02-10T11:51:57.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:57.750 7f697f3ca700 1 -- 172.21.2.117:0/4034982274 wait complete. 2021-02-10T11:51:57.758 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:51:57.759 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:51:58.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.170 7feaf9ad6700 1 Processor -- start 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.170 7feaf9ad6700 1 -- start start 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf9ad6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feaf40d18e0 0x7feaf40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf9ad6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf40ede60 con 0x7feaf40eb000 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf9ad6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf40d1ce0 con 0x7feaf40ef3f0 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf9ad6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf40ef7f0 con 0x7feaf40d18e0 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf2ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7feaf40eb000 0x7feaf40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56202/0 (socket says 172.21.2.117:56202) 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf2ffd700 1 -- 172.21.2.117:0/1342877777 learned_addr learned my addr 172.21.2.117:0/1342877777 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf37fe700 1 --2- 172.21.2.117:0/1342877777 >> v2:172.21.2.117:3300/0 conn(0x7feaf40d18e0 0x7feaf40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf27fc700 1 -- 172.21.2.117:0/1342877777 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276425216 0 0) 0x7feadc0009b0 con 0x7feaf40eb000 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf27fc700 1 -- 172.21.2.117:0/1342877777 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae0002fe0 con 0x7feaf40eb000 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf37fe700 1 -- 172.21.2.117:0/1342877777 >> v1:172.21.2.125:6789/0 conn(0x7feaf40ef3f0 legacy=0x7feaf41b7960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:51:58.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf37fe700 1 -- 172.21.2.117:0/1342877777 >> v1:172.21.2.117:6789/0 conn(0x7feaf40eb000 legacy=0x7feaf40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf37fe700 1 -- 172.21.2.117:0/1342877777 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf41bb180 con 0x7feaf40d18e0 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.171 7feaf37fe700 1 --2- 172.21.2.117:0/1342877777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40d18e0 0x7feaf40ea9e0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7feae4006650 tx=0x7feae4009a80).ready entity=mon.0 client_cookie=df6e97732d7b5790 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf27fc700 1 -- 172.21.2.117:0/1342877777 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feae40009b0 con 0x7feaf40d18e0 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf27fc700 1 -- 172.21.2.117:0/1342877777 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae4006740 con 0x7feaf40d18e0 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf27fc700 1 -- 172.21.2.117:0/1342877777 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feae400c0c0 con 0x7feaf40d18e0 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 -- 172.21.2.117:0/1342877777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40d18e0 msgr2=0x7feaf40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 --2- 172.21.2.117:0/1342877777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40d18e0 0x7feaf40ea9e0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 rx=0x7feae4006650 tx=0x7feae4009a80).stop 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 -- 172.21.2.117:0/1342877777 shutdown_connections 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 --2- 172.21.2.117:0/1342877777 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40d18e0 0x7feaf40ea9e0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 -- 172.21.2.117:0/1342877777 >> 172.21.2.117:0/1342877777 conn(0x7feaf40ced60 msgr2=0x7feaf40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 -- 172.21.2.117:0/1342877777 shutdown_connections 2021-02-10T11:51:58.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.172 7feaf9ad6700 1 -- 172.21.2.117:0/1342877777 wait complete. 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 Processor -- start 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 -- start start 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaf40980c0 0x7feaf419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaf419dd40 0x7feaf419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf40b1f30 con 0x7feaf40984c0 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf419d830 con 0x7feaf419dd40 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf9ad6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf41a55e0 con 0x7feaf40980c0 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55956/0 (socket says 172.21.2.117:55956) 2021-02-10T11:51:58.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.187 7feaf2ffd700 1 -- 172.21.2.117:0/1294996345 learned_addr learned my addr 172.21.2.117:0/1294996345 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 -- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaf40980c0 msgr2=0x7feaf419ca80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaf40980c0 0x7feaf419ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 -- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaf419dd40 msgr2=0x7feaf419e140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaf419dd40 0x7feaf419e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 -- 172.21.2.117:0/1294996345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf419e760 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf2ffd700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 rx=0x7feadc001c60 tx=0x7feadc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feadc00acc0 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc004140 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feadc004620 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf419e760 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.188 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf419e760 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.189 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feadc004320 con 0x7feaf40984c0 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.189 7feaebfff700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 0x7fead801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.189 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5100+0+0 (secure 0 0 0) 0x7feadc032380 con 0x7feaf40984c0 2021-02-10T11:51:58.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.189 7feaf37fe700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 0x7fead801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.190 7feaf37fe700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 0x7fead801abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.221 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf4087210 con 0x7feaf40984c0 2021-02-10T11:51:58.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.223 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feadc000e20 con 0x7feaf40984c0 2021-02-10T11:51:58.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.354 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7feaf4087210 con 0x7feaf40984c0 2021-02-10T11:51:58.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.355 7feaebfff700 1 -- 172.21.2.117:0/1294996345 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v76) v1 ==== 80+0+4513 (secure 0 0 0) 0x7feadc004300 con 0x7feaf40984c0 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.357 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.358 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.361 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:51:58.362 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.363 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.364 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.365 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:51:58.366 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:51:58.367 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:51:58.368 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:51:58.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 msgr2=0x7fead801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 0x7fead801abc0 crc :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 msgr2=0x7feaf419d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 rx=0x7feadc001c60 tx=0x7feadc00a8d0).stop 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 shutdown_connections 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fead80186c0 0x7fead801abc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaf40984c0 0x7feaf419d0a0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaf419dd40 0x7feaf419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 --2- 172.21.2.117:0/1294996345 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaf40980c0 0x7feaf419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 >> 172.21.2.117:0/1294996345 conn(0x7feaf419f610 msgr2=0x7feaf41a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.357 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 shutdown_connections 2021-02-10T11:51:58.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.358 7feaf9ad6700 1 -- 172.21.2.117:0/1294996345 wait complete. 2021-02-10T11:51:58.371 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.782 7f6ec5947700 1 Processor -- start 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.782 7f6ec5947700 1 -- start start 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ec5947700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 0x7f6ec00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ec5947700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec00ede60 con 0x7f6ec00d18e0 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ec5947700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec00d1ce0 con 0x7f6ec00eb7f0 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ec5947700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec00ef7f0 con 0x7f6ec00ef3f0 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ebf574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6ec00d18e0 0x7f6ec00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56214/0 (socket says 172.21.2.117:56214) 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ebf574700 1 -- 172.21.2.117:0/3023492315 learned_addr learned my addr 172.21.2.117:0/3023492315 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.783 7f6ebed73700 1 --2- 172.21.2.117:0/3023492315 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 0x7f6ec00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769900000 0 0) 0x7f6eb0001180 con 0x7f6ec00d18e0 2021-02-10T11:51:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eac002fe0 con 0x7f6ec00d18e0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639656800 0 0) 0x7f6eb8001400 con 0x7f6ec00eb7f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eac003210 con 0x7f6ec00eb7f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebed73700 1 -- 172.21.2.117:0/3023492315 >> v1:172.21.2.125:6789/0 conn(0x7f6ec00eb7f0 legacy=0x7f6ec01b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebed73700 1 -- 172.21.2.117:0/3023492315 >> v1:172.21.2.117:6789/0 conn(0x7f6ec00d18e0 legacy=0x7f6ec00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebed73700 1 -- 172.21.2.117:0/3023492315 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec01bb180 con 0x7f6ec00ef3f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebed73700 1 --2- 172.21.2.117:0/3023492315 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 0x7f6ec00eb1d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6ea8006650 tx=0x7f6ea8009a00).ready entity=mon.2 client_cookie=86ea6636f21206d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea80009b0 con 0x7f6ec00ef3f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea800c0c0 con 0x7f6ec00ef3f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ebe572700 1 -- 172.21.2.117:0/3023492315 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ea800c5e0 con 0x7f6ec00ef3f0 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ec5947700 1 -- 172.21.2.117:0/3023492315 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 msgr2=0x7f6ec00eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.784 7f6ec5947700 1 --2- 172.21.2.117:0/3023492315 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 0x7f6ec00eb1d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6ea8006650 tx=0x7f6ea8009a00).stop 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.785 7f6ec5947700 1 -- 172.21.2.117:0/3023492315 shutdown_connections 2021-02-10T11:51:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.785 7f6ec5947700 1 --2- 172.21.2.117:0/3023492315 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00ef3f0 0x7f6ec00eb1d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.785 7f6ec5947700 1 -- 172.21.2.117:0/3023492315 >> 172.21.2.117:0/3023492315 conn(0x7f6ec00ced60 msgr2=0x7f6ec00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.785 7f6ec5947700 1 -- 172.21.2.117:0/3023492315 shutdown_connections 2021-02-10T11:51:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.785 7f6ec5947700 1 -- 172.21.2.117:0/3023492315 wait complete. 2021-02-10T11:51:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 Processor -- start 2021-02-10T11:51:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 -- start start 2021-02-10T11:51:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00e6190 0x7f6ec00e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ec00e6bb0 0x7f6ec00e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec008a5a0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec008a780 con 0x7f6ec00e6bb0 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ec5947700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec008a9a0 con 0x7f6ec00e6190 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.800 7f6ebf574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55970/0 (socket says 172.21.2.117:55970) 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 -- 172.21.2.117:0/3871635360 learned_addr learned my addr 172.21.2.117:0/3871635360 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 -- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00e6190 msgr2=0x7f6ec00e6590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00e6190 0x7f6ec00e6590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 -- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ec00e6bb0 msgr2=0x7f6ec00e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ec00e6bb0 0x7f6ec00e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 -- 172.21.2.117:0/3871635360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec008abc0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ebf574700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7f6eb0000b80 tx=0x7f6eb000a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb000ad70 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb00070d0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec008abc0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec008b150 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.801 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6eb0004830 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.802 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6eb00043a0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.802 7f6eb7fff700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.802 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f6eb00323c0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.802 7f6ebed73700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.803 7f6ebed73700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 0x7f6ea401abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:58.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.833 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec00e5c50 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.835 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6eb0001490 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.962 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ec00e5c50 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.962 7f6eb7fff700 1 -- 172.21.2.117:0/3871635360 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f6eb00043a0 con 0x7f6ec00ae5b0 2021-02-10T11:51:58.963 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:51:58.963 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":129,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:51:57.729449","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":122,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:51:58.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 msgr2=0x7f6ea401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 0x7f6ea401abc0 crc :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 msgr2=0x7f6ec00ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 rx=0x7f6eb0000b80 tx=0x7f6eb000a980).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 shutdown_connections 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ec00e6bb0 0x7f6ec00e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ec00e6190 0x7f6ec00e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ea40186c0 0x7f6ea401abc0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 --2- 172.21.2.117:0/3871635360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ec00ae5b0 0x7f6ec00ae9b0 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:58.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.964 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 >> 172.21.2.117:0/3871635360 conn(0x7f6ec01a3620 msgr2=0x7f6ec00e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:58.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.965 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 shutdown_connections 2021-02-10T11:51:58.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:58.965 7f6ec5947700 1 -- 172.21.2.117:0/3871635360 wait complete. 2021-02-10T11:51:58.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:51:58.974 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:51:59.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.389 7f3874316700 1 Processor -- start 2021-02-10T11:51:59.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.389 7f3874316700 1 -- start start 2021-02-10T11:51:59.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3874316700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f386c0d18f0 0x7f386c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3874316700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0ef3f0 0x7f386c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3874316700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386c0ef7f0 con 0x7f386c0eb040 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3874316700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f386c0eded0 con 0x7f386c0d18f0 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3874316700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0d1cf0 con 0x7f386c0ef3f0 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3871931700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f386c0eb040 0x7f386c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56228/0 (socket says 172.21.2.117:56228) 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3871931700 1 -- 172.21.2.117:0/3360496159 learned_addr learned my addr 172.21.2.117:0/3360496159 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3872132700 1 --2- 172.21.2.117:0/3360496159 >> v2:172.21.2.117:3300/0 conn(0x7f386c0d18f0 0x7f386c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.390 7f3872933700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0ef3f0 0x7f386c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3871130700 1 -- 172.21.2.117:0/3360496159 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439438599 0 0) 0x7f38680010b0 con 0x7f386c0eb040 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3871130700 1 -- 172.21.2.117:0/3360496159 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3858002fe0 con 0x7f386c0eb040 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3872132700 1 -- 172.21.2.117:0/3360496159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0ef3f0 msgr2=0x7f386c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:51:59.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3872132700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0ef3f0 0x7f386c1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3872132700 1 -- 172.21.2.117:0/3360496159 >> v1:172.21.2.117:6789/0 conn(0x7f386c0eb040 legacy=0x7f386c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3872132700 1 -- 172.21.2.117:0/3360496159 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c1b8fd0 con 0x7f386c0d18f0 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3872132700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0d18f0 0x7f386c0eaa20 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f3860006650 tx=0x7f3860009a80).ready entity=mon.0 client_cookie=729e675a30aeabd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3871130700 1 -- 172.21.2.117:0/3360496159 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f386000c0c0 con 0x7f386c0d18f0 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3871130700 1 -- 172.21.2.117:0/3360496159 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386000c340 con 0x7f386c0d18f0 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3871130700 1 -- 172.21.2.117:0/3360496159 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f386000c860 con 0x7f386c0d18f0 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3874316700 1 -- 172.21.2.117:0/3360496159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0d18f0 msgr2=0x7f386c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.391 7f3874316700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0d18f0 0x7f386c0eaa20 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 rx=0x7f3860006650 tx=0x7f3860009a80).stop 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 -- 172.21.2.117:0/3360496159 shutdown_connections 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0ef3f0 0x7f386c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 --2- 172.21.2.117:0/3360496159 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0d18f0 0x7f386c0eaa20 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 -- 172.21.2.117:0/3360496159 >> 172.21.2.117:0/3360496159 conn(0x7f386c0cf060 msgr2=0x7f386c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 -- 172.21.2.117:0/3360496159 shutdown_connections 2021-02-10T11:51:59.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.392 7f3874316700 1 -- 172.21.2.117:0/3360496159 wait complete. 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.406 7f3874316700 1 Processor -- start 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.406 7f3874316700 1 -- start start 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0a6ce0 0x7f386c0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f386c085ec0 0x7f386c0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0a67e0 con 0x7f386c0a2040 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0a7120 con 0x7f386c085ec0 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3874316700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386c0a2a60 con 0x7f386c0a6ce0 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55982/0 (socket says 172.21.2.117:55982) 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 -- 172.21.2.117:0/1480798735 learned_addr learned my addr 172.21.2.117:0/1480798735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872132700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0a6ce0 0x7f386c0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3871931700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f386c085ec0 0x7f386c0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 -- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0a6ce0 msgr2=0x7f386c0dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0a6ce0 0x7f386c0dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 -- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f386c085ec0 msgr2=0x7f386c0862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f386c085ec0 0x7f386c0862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 -- 172.21.2.117:0/1480798735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386c1a2dc0 con 0x7f386c0a2040 2021-02-10T11:51:59.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.407 7f3872933700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7f386400b910 tx=0x7f386400bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3864006aa0 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3864003030 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38640013e0 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f3874316700 1 -- 172.21.2.117:0/1480798735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f386c1a2dc0 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f3874316700 1 -- 172.21.2.117:0/1480798735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386c1a2dc0 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3864000ec0 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 0x7f385401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.408 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f3864032740 con 0x7f386c0a2040 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.409 7f3872132700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 0x7f385401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:51:59.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.409 7f3872132700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 0x7f385401abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:51:59.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.440 7f3874316700 1 -- 172.21.2.117:0/1480798735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386c0daff0 con 0x7f386c0a2040 2021-02-10T11:51:59.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.442 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f386400bdf0 con 0x7f386c0a2040 2021-02-10T11:51:59.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.568 7f3874316700 1 -- 172.21.2.117:0/1480798735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f386c0daff0 con 0x7f386c0a2040 2021-02-10T11:51:59.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.569 7f385effd700 1 -- 172.21.2.117:0/1480798735 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v129) v1 ==== 92+0+10 (secure 0 0 0) 0x7f38640066f0 con 0x7f386c0a2040 2021-02-10T11:51:59.570 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:51:59.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 msgr2=0x7f385401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 0x7f385401abc0 crc :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 msgr2=0x7f386c0a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 rx=0x7f386400b910 tx=0x7f386400bcb0).stop 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 shutdown_connections 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38540186c0 0x7f385401abc0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f386c0a2040 0x7f386c0a2440 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f386c085ec0 0x7f386c0862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 --2- 172.21.2.117:0/1480798735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f386c0a6ce0 0x7f386c0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 >> 172.21.2.117:0/1480798735 conn(0x7f386c0b6540 msgr2=0x7f386c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 shutdown_connections 2021-02-10T11:51:59.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.571 7f3874316700 1 -- 172.21.2.117:0/1480798735 wait complete. 2021-02-10T11:51:59.580 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.990 7f03f8236700 1 Processor -- start 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.990 7f03f8236700 1 -- start start 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f8236700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03f00e4e40 0x7f03f01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f8236700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f03f00e5240 0x7f03f0084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f8236700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f0085930 con 0x7f03f01b0810 2021-02-10T11:52:00.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f8236700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00e5cc0 con 0x7f03f00e4e40 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f8236700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f0084840 con 0x7f03f00e5240 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03f00e4e40 0x7f03f01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f03f00e4e40 0x7f03f01b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55990/0 (socket says 172.21.2.117:55990) 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 -- 172.21.2.117:0/1971344279 learned_addr learned my addr 172.21.2.117:0/1971344279 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 -- 172.21.2.117:0/1971344279 >> v2:172.21.2.125:3300/0 conn(0x7f03f00e5240 msgr2=0x7f03f0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 --2- 172.21.2.117:0/1971344279 >> v2:172.21.2.125:3300/0 conn(0x7f03f00e5240 0x7f03f0084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.991 7f03f6052700 1 -- 172.21.2.117:0/1971344279 >> v1:172.21.2.125:6789/0 conn(0x7f03f01b0810 legacy=0x7f03f0085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f6052700 1 -- 172.21.2.117:0/1971344279 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f0085280 con 0x7f03f00e4e40 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f6052700 1 --2- 172.21.2.117:0/1971344279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00e4e40 0x7f03f01b01f0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f03ec006700 tx=0x7f03ec009b80).ready entity=mon.0 client_cookie=b2edfb1400489482 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f5050700 1 -- 172.21.2.117:0/1971344279 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ec00c0c0 con 0x7f03f00e4e40 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f5050700 1 -- 172.21.2.117:0/1971344279 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec00c340 con 0x7f03f00e4e40 2021-02-10T11:52:00.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f5050700 1 -- 172.21.2.117:0/1971344279 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03ec00c860 con 0x7f03f00e4e40 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f8236700 1 -- 172.21.2.117:0/1971344279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00e4e40 msgr2=0x7f03f01b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.992 7f03f8236700 1 --2- 172.21.2.117:0/1971344279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00e4e40 0x7f03f01b01f0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 rx=0x7f03ec006700 tx=0x7f03ec009b80).stop 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 -- 172.21.2.117:0/1971344279 shutdown_connections 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 --2- 172.21.2.117:0/1971344279 >> v2:172.21.2.125:3300/0 conn(0x7f03f00e5240 0x7f03f0084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 --2- 172.21.2.117:0/1971344279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00e4e40 0x7f03f01b01f0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 -- 172.21.2.117:0/1971344279 >> 172.21.2.117:0/1971344279 conn(0x7f03f00ce4e0 msgr2=0x7f03f00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 -- 172.21.2.117:0/1971344279 shutdown_connections 2021-02-10T11:52:00.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:51:59.993 7f03f8236700 1 -- 172.21.2.117:0/1971344279 wait complete. 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 Processor -- start 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 -- start start 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03f00b0700 0x7f03f018c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03f0190f20 0x7f03f0097f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f0097a10 con 0x7f03f00985b0 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f018c720 con 0x7f03f0190f20 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.008 7f03f8236700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00989b0 con 0x7f03f00b0700 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:00.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:55998/0 (socket says 172.21.2.117:55998) 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 -- 172.21.2.117:0/393939091 learned_addr learned my addr 172.21.2.117:0/393939091 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 -- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03f00b0700 msgr2=0x7f03f018c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03f00b0700 0x7f03f018c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 -- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03f0190f20 msgr2=0x7f03f0097f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03f0190f20 0x7f03f0097f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 -- 172.21.2.117:0/393939091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f00965f0 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.009 7f03f6853700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f03e8006ed0 tx=0x7f03e800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e800bd50 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e80009b0 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f03e80062a0 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03f8236700 1 -- 172.21.2.117:0/393939091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f00965f0 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03f8236700 1 -- 172.21.2.117:0/393939091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f0096b80 con 0x7f03f00985b0 2021-02-10T11:52:00.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f03e8006020 con 0x7f03f00985b0 2021-02-10T11:52:00.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.010 7f03e2ffd700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 0x7f03d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:00.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.011 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f03e8032350 con 0x7f03f00985b0 2021-02-10T11:52:00.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.011 7f03f6052700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 0x7f03d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:00.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.011 7f03f6052700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 0x7f03d801abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:00.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.041 7f03f8236700 1 -- 172.21.2.117:0/393939091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f010c250 con 0x7f03f00985b0 2021-02-10T11:52:00.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.043 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f03e8000c10 con 0x7f03f00985b0 2021-02-10T11:52:00.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.166 7f03f8236700 1 -- 172.21.2.117:0/393939091 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f03f0085930 con 0x7f03f00985b0 2021-02-10T11:52:00.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.166 7f03e2ffd700 1 -- 172.21.2.117:0/393939091 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v129) v1 ==== 84+0+64 (secure 0 0 0) 0x7f03e8006000 con 0x7f03f00985b0 2021-02-10T11:52:00.167 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:00.167 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:52:00.168 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:00.168 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:52:00.168 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:52:00.168 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:00.168 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:52:00.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.168 7f03f8236700 1 -- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 msgr2=0x7f03d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:00.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.168 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 0x7f03d801abc0 crc :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:00.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.168 7f03f8236700 1 -- 172.21.2.117:0/393939091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 msgr2=0x7f03f0095fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:00.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.168 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 rx=0x7f03e8006ed0 tx=0x7f03e800b960).stop 2021-02-10T11:52:00.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 -- 172.21.2.117:0/393939091 shutdown_connections 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f03d80186c0 0x7f03d801abc0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f03f00985b0 0x7f03f0095fd0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f03f0190f20 0x7f03f0097f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 --2- 172.21.2.117:0/393939091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f03f00b0700 0x7f03f018c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 -- 172.21.2.117:0/393939091 >> 172.21.2.117:0/393939091 conn(0x7f03f0199160 msgr2=0x7f03f008d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 -- 172.21.2.117:0/393939091 shutdown_connections 2021-02-10T11:52:00.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.169 7f03f8236700 1 -- 172.21.2.117:0/393939091 wait complete. 2021-02-10T11:52:00.177 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:52:00.177 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:52:00.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 Processor -- start 2021-02-10T11:52:01.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 -- start start 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 0x7f24580e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24580e8390 0x7f24580e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24580e7580 con 0x7f24581b07f0 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24580e5ce0 con 0x7f24580e8390 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.594 7f245d8b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580e77f0 con 0x7f24580e7180 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24580e8390 0x7f24580e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2456d73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 0x7f24580e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24580e8390 0x7f24580e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60390/0 (socket says 172.21.2.117:60390) 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 -- 172.21.2.117:0/380230572 learned_addr learned my addr 172.21.2.117:0/380230572 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 -- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 msgr2=0x7f24580e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 0x7f24580e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 -- 172.21.2.117:0/380230572 >> v1:172.21.2.125:6789/0 conn(0x7f24581b07f0 legacy=0x7f24580e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2457d75700 1 -- 172.21.2.117:0/380230572 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24580e8db0 con 0x7f24580e8390 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.595 7f2456d73700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 0x7f24580e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f2457d75700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580e8390 0x7f24580e8790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f2450006aa0 tx=0x7f2450009f20).ready entity=mon.1 client_cookie=525527c95ffd46b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f2456572700 1 -- 172.21.2.117:0/380230572 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245000b5c0 con 0x7f24580e8390 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f2456572700 1 -- 172.21.2.117:0/380230572 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2450003030 con 0x7f24580e8390 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f2456572700 1 -- 172.21.2.117:0/380230572 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245000ba30 con 0x7f24580e8390 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f245d8b3700 1 -- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580e8390 msgr2=0x7f24580e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.596 7f245d8b3700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580e8390 0x7f24580e8790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f2450006aa0 tx=0x7f2450009f20).stop 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 -- 172.21.2.117:0/380230572 shutdown_connections 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580e8390 0x7f24580e8790 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 --2- 172.21.2.117:0/380230572 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24580e7180 0x7f24580e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 -- 172.21.2.117:0/380230572 >> 172.21.2.117:0/380230572 conn(0x7f24580ce500 msgr2=0x7f24580e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 -- 172.21.2.117:0/380230572 shutdown_connections 2021-02-10T11:52:01.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.597 7f245d8b3700 1 -- 172.21.2.117:0/380230572 wait complete. 2021-02-10T11:52:01.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.611 7f245d8b3700 1 Processor -- start 2021-02-10T11:52:01.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.611 7f245d8b3700 1 -- start start 2021-02-10T11:52:01.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580b8ac0 0x7f245819d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f245819df40 0x7f24580bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580bb430 con 0x7f24580bbfd0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580bc9f0 con 0x7f24580b8ac0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f245d8b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580bcc10 con 0x7f245819df40 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56010/0 (socket says 172.21.2.117:56010) 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 -- 172.21.2.117:0/2333101769 learned_addr learned my addr 172.21.2.117:0/2333101769 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 -- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f245819df40 msgr2=0x7f24580bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f245819df40 0x7f24580bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 -- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580b8ac0 msgr2=0x7f245819d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580b8ac0 0x7f245819d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.612 7f2457d75700 1 -- 172.21.2.117:0/2333101769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24580b6f70 con 0x7f24580bbfd0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f2457d75700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7f245000b970 tx=0x7f245000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2450009450 con 0x7f24580bbfd0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245000c2f0 con 0x7f24580bbfd0 2021-02-10T11:52:01.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f245000c790 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24580b6f70 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.613 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24580b7500 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.614 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2450009fb0 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.614 7f244ffff700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 0x7f243c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.614 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f2450032550 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.614 7f2457574700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 0x7f243c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.615 7f2457574700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 0x7f243c01abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.644 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245807e800 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.647 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2450009450 con 0x7f24580bbfd0 2021-02-10T11:52:01.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:00.773 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f24580e77f0 con 0x7f24580bbfd0 2021-02-10T11:52:01.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.197 7f244ffff700 1 -- 172.21.2.117:0/2333101769 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v130) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2450009fb0 con 0x7f24580bbfd0 2021-02-10T11:52:01.240 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:52:01.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 msgr2=0x7f243c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 0x7f243c01abc0 crc :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 msgr2=0x7f24580bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 rx=0x7f245000b970 tx=0x7f245000bd10).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 shutdown_connections 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f243c0186c0 0x7f243c01abc0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24580bbfd0 0x7f24580bc3d0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f245819df40 0x7f24580bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 --2- 172.21.2.117:0/2333101769 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24580b8ac0 0x7f245819d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 >> 172.21.2.117:0/2333101769 conn(0x7f2458110940 msgr2=0x7f245807fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:01.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 shutdown_connections 2021-02-10T11:52:01.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.242 7f245d8b3700 1 -- 172.21.2.117:0/2333101769 wait complete. 2021-02-10T11:52:01.251 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:52:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.666 7fb497626700 1 Processor -- start 2021-02-10T11:52:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.666 7fb497626700 1 -- start start 2021-02-10T11:52:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb497626700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4900d18f0 0x7fb4900eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb497626700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4900ef3f0 0x7fb4901b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb497626700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4900ede60 con 0x7fb4900eb040 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb497626700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb490088020 con 0x7fb4900ef3f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb497626700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900d1d30 con 0x7fb4900d18f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4900ef3f0 0x7fb4901b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4900ef3f0 0x7fb4901b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56016/0 (socket says 172.21.2.117:56016) 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 -- 172.21.2.117:0/2984785649 learned_addr learned my addr 172.21.2.117:0/2984785649 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 -- 172.21.2.117:0/2984785649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4900d18f0 msgr2=0x7fb4900eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 --2- 172.21.2.117:0/2984785649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4900d18f0 0x7fb4900eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 -- 172.21.2.117:0/2984785649 >> v1:172.21.2.125:6789/0 conn(0x7fb4900eb040 legacy=0x7fb4900eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 -- 172.21.2.117:0/2984785649 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4901b8fd0 con 0x7fb4900ef3f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.667 7fb495c43700 1 --2- 172.21.2.117:0/2984785649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4900ef3f0 0x7fb4901b79a0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 rx=0x7fb488006f00 tx=0x7fb488009f10).ready entity=mon.0 client_cookie=b7453a4be6da8726 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb483fff700 1 -- 172.21.2.117:0/2984785649 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48800b3b0 con 0x7fb4900ef3f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb483fff700 1 -- 172.21.2.117:0/2984785649 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48800b730 con 0x7fb4900ef3f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb483fff700 1 -- 172.21.2.117:0/2984785649 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb488000f90 con 0x7fb4900ef3f0 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 -- 172.21.2.117:0/2984785649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4900ef3f0 msgr2=0x7fb4901b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 --2- 172.21.2.117:0/2984785649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4900ef3f0 0x7fb4901b79a0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 rx=0x7fb488006f00 tx=0x7fb488009f10).stop 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 -- 172.21.2.117:0/2984785649 shutdown_connections 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 --2- 172.21.2.117:0/2984785649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4900ef3f0 0x7fb4901b79a0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 --2- 172.21.2.117:0/2984785649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4900d18f0 0x7fb4900eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.668 7fb497626700 1 -- 172.21.2.117:0/2984785649 >> 172.21.2.117:0/2984785649 conn(0x7fb4900cf060 msgr2=0x7fb4900cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.669 7fb497626700 1 -- 172.21.2.117:0/2984785649 shutdown_connections 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.669 7fb497626700 1 -- 172.21.2.117:0/2984785649 wait complete. 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 Processor -- start 2021-02-10T11:52:01.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 -- start start 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4901a4240 0x7fb4901a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb49009ed70 0x7fb49009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49009fb90 con 0x7fb49009f790 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900dc640 con 0x7fb4901a4240 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb497626700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb490097270 con 0x7fb49009ed70 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb495c43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb495c43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56024/0 (socket says 172.21.2.117:56024) 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.684 7fb495c43700 1 -- 172.21.2.117:0/1040553998 learned_addr learned my addr 172.21.2.117:0/1040553998 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb494c41700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb49009ed70 0x7fb49009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495442700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4901a4240 0x7fb4901a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 -- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb49009ed70 msgr2=0x7fb49009f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb49009ed70 0x7fb49009f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 -- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4901a4240 msgr2=0x7fb4901a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4901a4240 0x7fb4901a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 -- 172.21.2.117:0/1040553998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900b8780 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb495c43700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fb488009390 tx=0x7fb48800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb488003030 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4880068e0 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb497626700 1 -- 172.21.2.117:0/1040553998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4900b8780 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb497626700 1 -- 172.21.2.117:0/1040553998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4900b8780 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.685 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48800c570 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.686 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb488001270 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.686 7fb481ffb700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 0x7fb47001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.686 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5454+0+0 (secure 0 0 0) 0x7fb488032aa0 con 0x7fb49009f790 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.687 7fb495442700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 0x7fb47001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:01.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.687 7fb495442700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 0x7fb47001abc0 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:01.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.716 7fb497626700 1 -- 172.21.2.117:0/1040553998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4900daff0 con 0x7fb49009f790 2021-02-10T11:52:01.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.719 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb488006c80 con 0x7fb49009f790 2021-02-10T11:52:01.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:01.847 7fb497626700 1 -- 172.21.2.117:0/1040553998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fb4900ede60 con 0x7fb49009f790 2021-02-10T11:52:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.204 7fb481ffb700 1 -- 172.21.2.117:0/1040553998 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v132) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb488001270 con 0x7fb49009f790 2021-02-10T11:52:02.213 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 msgr2=0x7fb47001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 0x7fb47001abc0 crc :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 msgr2=0x7fb4900b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7fb488009390 tx=0x7fb48800bd40).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 shutdown_connections 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4700186c0 0x7fb47001abc0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb49009f790 0x7fb4900b8160 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb49009ed70 0x7fb49009f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 --2- 172.21.2.117:0/1040553998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4901a4240 0x7fb4901a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 >> 172.21.2.117:0/1040553998 conn(0x7fb490087090 msgr2=0x7fb49008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:02.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 shutdown_connections 2021-02-10T11:52:02.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.215 7fb497626700 1 -- 172.21.2.117:0/1040553998 wait complete. 2021-02-10T11:52:02.224 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.637 7f638f66d700 1 Processor -- start 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.637 7f638f66d700 1 -- start start 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.638 7f638f66d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f63880e4e40 0x7f63881b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.638 7f638f66d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 0x7f6388086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.638 7f638f66d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6388085930 con 0x7f63880e5240 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.638 7f638f66d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63880e5cc0 con 0x7f63880e4e40 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.638 7f638f66d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6388084840 con 0x7f6388085450 2021-02-10T11:52:02.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638dc8a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 0x7f6388086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638dc8a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 0x7f6388086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:53710/0 (socket says 172.21.2.117:53710) 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f63880e4e40 0x7f63881b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638dc8a700 1 -- 172.21.2.117:0/1820873171 learned_addr learned my addr 172.21.2.117:0/1820873171 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 -- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 msgr2=0x7f6388086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 --2- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 0x7f6388086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 -- 172.21.2.117:0/1820873171 >> v1:172.21.2.125:6789/0 conn(0x7f63880e5240 legacy=0x7f6388084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 -- 172.21.2.117:0/1820873171 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63881b0810 con 0x7f63880e4e40 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.639 7f638d489700 1 --2- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63880e4e40 0x7f63881b01f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).ready entity=mon.1 client_cookie=afda6c2f9aa993 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f637bfff700 1 -- 172.21.2.117:0/1820873171 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c00c0c0 con 0x7f63880e4e40 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f637bfff700 1 -- 172.21.2.117:0/1820873171 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00c340 con 0x7f63880e4e40 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f637bfff700 1 -- 172.21.2.117:0/1820873171 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c00c860 con 0x7f63880e4e40 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 -- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63880e4e40 msgr2=0x7f63881b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 --2- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63880e4e40 0x7f63881b01f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).stop 2021-02-10T11:52:02.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 -- 172.21.2.117:0/1820873171 shutdown_connections 2021-02-10T11:52:02.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 --2- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6388085450 0x7f6388086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 --2- 172.21.2.117:0/1820873171 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63880e4e40 0x7f63881b01f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.640 7f638f66d700 1 -- 172.21.2.117:0/1820873171 >> 172.21.2.117:0/1820873171 conn(0x7f63880ce4e0 msgr2=0x7f63880e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:02.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.641 7f638f66d700 1 -- 172.21.2.117:0/1820873171 shutdown_connections 2021-02-10T11:52:02.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.641 7f638f66d700 1 -- 172.21.2.117:0/1820873171 wait complete. 2021-02-10T11:52:02.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 Processor -- start 2021-02-10T11:52:02.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 -- start start 2021-02-10T11:52:02.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 0x7f63881b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63880a92b0 0x7f63880a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638810f580 con 0x7f63881b0ca0 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638818ba20 con 0x7f638818b460 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.655 7f638f66d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63881b16a0 con 0x7f63880a92b0 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56034/0 (socket says 172.21.2.117:56034) 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 -- 172.21.2.117:0/3303981313 learned_addr learned my addr 172.21.2.117:0/3303981313 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638dc8a700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63880a92b0 0x7f63880a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638d489700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 0x7f63881b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 -- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63880a92b0 msgr2=0x7f63880a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63880a92b0 0x7f63880a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 -- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 msgr2=0x7f63881b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 0x7f63881b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 -- 172.21.2.117:0/3303981313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63881b18c0 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638d489700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 0x7f63881b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638cc88700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7f6384006ed0 tx=0x7f638400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6384006240 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63840054b0 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.656 7f638f66d700 1 -- 172.21.2.117:0/3303981313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63881b18c0 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f638f66d700 1 -- 172.21.2.117:0/3303981313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63881b1e50 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6384005910 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6384006030 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f6379ffb700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 0x7f636801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f6384032610 con 0x7f63881b0ca0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.657 7f638d489700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 0x7f636801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.658 7f638d489700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 0x7f636801abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:02.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.688 7f638f66d700 1 -- 172.21.2.117:0/3303981313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638810c2b0 con 0x7f63881b0ca0 2021-02-10T11:52:02.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.691 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6384030ce0 con 0x7f63881b0ca0 2021-02-10T11:52:02.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.817 7f638f66d700 1 -- 172.21.2.117:0/3303981313 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f6388085930 con 0x7f63881b0ca0 2021-02-10T11:52:02.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.821 7f6379ffb700 1 -- 172.21.2.117:0/3303981313 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 28 and data pool 29 v108) v1 ==== 189+0+0 (secure 0 0 0) 0x7f638401f6c0 con 0x7f63881b0ca0 2021-02-10T11:52:02.825 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 28 and data pool 29 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 msgr2=0x7f636801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 0x7f636801abc0 crc :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 msgr2=0x7f63880a8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 rx=0x7f6384006ed0 tx=0x7f638400b960).stop 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 shutdown_connections 2021-02-10T11:52:02.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63680186c0 0x7f636801abc0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63881b0ca0 0x7f63880a8c90 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63880a92b0 0x7f63880a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 --2- 172.21.2.117:0/3303981313 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638818b460 0x7f63881b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 >> 172.21.2.117:0/3303981313 conn(0x7f638810dec0 msgr2=0x7f6388196850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 shutdown_connections 2021-02-10T11:52:02.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:02.827 7f638f66d700 1 -- 172.21.2.117:0/3303981313 wait complete. 2021-02-10T11:52:02.836 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.245 7f33cc55b700 1 Processor -- start 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.245 7f33cc55b700 1 -- start start 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33cc55b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33c40d18e0 0x7f33c40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33cc55b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c40ef3f0 0x7f33c41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33cc55b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c40ede60 con 0x7f33c40eb000 2021-02-10T11:52:03.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33cc55b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c40ef7f0 con 0x7f33c40d18e0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33cc55b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40d1ce0 con 0x7f33c40ef3f0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33c40d18e0 0x7f33c40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33c40d18e0 0x7f33c40ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56042/0 (socket says 172.21.2.117:56042) 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 -- 172.21.2.117:0/3539406279 learned_addr learned my addr 172.21.2.117:0/3539406279 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 -- 172.21.2.117:0/3539406279 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c40ef3f0 msgr2=0x7f33c41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 --2- 172.21.2.117:0/3539406279 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c40ef3f0 0x7f33c41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 -- 172.21.2.117:0/3539406279 >> v1:172.21.2.125:6789/0 conn(0x7f33c40eb000 legacy=0x7f33c40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 -- 172.21.2.117:0/3539406279 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c41b8f90 con 0x7f33c40d18e0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.246 7f33ca377700 1 --2- 172.21.2.117:0/3539406279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c40d18e0 0x7f33c40ea9e0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f33b8006700 tx=0x7f33b8009b00).ready entity=mon.0 client_cookie=556233ee6ef76c9f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33c9375700 1 -- 172.21.2.117:0/3539406279 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b800c0c0 con 0x7f33c40d18e0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33c9375700 1 -- 172.21.2.117:0/3539406279 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b800c340 con 0x7f33c40d18e0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33c9375700 1 -- 172.21.2.117:0/3539406279 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b800c820 con 0x7f33c40d18e0 2021-02-10T11:52:03.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 -- 172.21.2.117:0/3539406279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c40d18e0 msgr2=0x7f33c40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 --2- 172.21.2.117:0/3539406279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c40d18e0 0x7f33c40ea9e0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 rx=0x7f33b8006700 tx=0x7f33b8009b00).stop 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 -- 172.21.2.117:0/3539406279 shutdown_connections 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 --2- 172.21.2.117:0/3539406279 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c40ef3f0 0x7f33c41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 --2- 172.21.2.117:0/3539406279 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c40d18e0 0x7f33c40ea9e0 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.247 7f33cc55b700 1 -- 172.21.2.117:0/3539406279 >> 172.21.2.117:0/3539406279 conn(0x7f33c40ced60 msgr2=0x7f33c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.248 7f33cc55b700 1 -- 172.21.2.117:0/3539406279 shutdown_connections 2021-02-10T11:52:03.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.248 7f33cc55b700 1 -- 172.21.2.117:0/3539406279 wait complete. 2021-02-10T11:52:03.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 Processor -- start 2021-02-10T11:52:03.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 -- start start 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c419b760 0x7f33c40b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33c40b5370 0x7f33c419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c41af950 con 0x7f33c419e620 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c419bb60 con 0x7f33c40b5370 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.262 7f33cc55b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c419ea20 con 0x7f33c419b760 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56048/0 (socket says 172.21.2.117:56048) 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 -- 172.21.2.117:0/1622756341 learned_addr learned my addr 172.21.2.117:0/1622756341 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33c9b76700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33c40b5370 0x7f33c419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33ca377700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c419b760 0x7f33c40b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 -- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c419b760 msgr2=0x7f33c40b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c419b760 0x7f33c40b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 -- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33c40b5370 msgr2=0x7f33c419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33c40b5370 0x7f33c419e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 -- 172.21.2.117:0/1622756341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c419ec40 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cab78700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7f33bc00b8c0 tx=0x7f33bc00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33bc005af0 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc005df0 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33bc0062b0 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c419ec40 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.263 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c419ec40 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.264 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33bc005af0 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.264 7f33b6ffd700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 0x7f33ac01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.264 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f33bc032b70 con 0x7f33c419e620 2021-02-10T11:52:03.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.264 7f33ca377700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 0x7f33ac01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.265 7f33ca377700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 0x7f33ac01abd0 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.296 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c4087210 con 0x7f33c419e620 2021-02-10T11:52:03.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.299 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33bc005df0 con 0x7f33c419e620 2021-02-10T11:52:03.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.422 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33c4087210 con 0x7f33c419e620 2021-02-10T11:52:03.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.423 7f33b6ffd700 1 -- 172.21.2.117:0/1622756341 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f33bc005df0 con 0x7f33c419e620 2021-02-10T11:52:03.424 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:03.424 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":133,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:02.818716","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:00.791090","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:01.865056","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 msgr2=0x7f33ac01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 0x7f33ac01abd0 crc :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 msgr2=0x7f33c41afe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 rx=0x7f33bc00b8c0 tx=0x7f33bc00bc60).stop 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 shutdown_connections 2021-02-10T11:52:03.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33ac0186d0 0x7f33ac01abd0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33c419e620 0x7f33c41afe50 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33c40b5370 0x7f33c419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 --2- 172.21.2.117:0/1622756341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33c419b760 0x7f33c40b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 >> 172.21.2.117:0/1622756341 conn(0x7f33c419f610 msgr2=0x7f33c41ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 shutdown_connections 2021-02-10T11:52:03.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.425 7f33cc55b700 1 -- 172.21.2.117:0/1622756341 wait complete. 2021-02-10T11:52:03.434 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.846 7f329e9fa700 1 Processor -- start 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.846 7f329e9fa700 1 -- start start 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f32980d18e0 0x7f32980ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32980eb000 0x7f32980eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32980ee3e0 0x7f32980ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980ede60 con 0x7f32980ee3e0 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980eba20 con 0x7f32980d18e0 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32980d1ce0 con 0x7f32980eb000 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32980ee3e0 0x7f32980ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32980ee3e0 0x7f32980ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56060/0 (socket says 172.21.2.117:56060) 2021-02-10T11:52:03.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 -- 172.21.2.117:0/646479743 learned_addr learned my addr 172.21.2.117:0/646479743 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 -- 172.21.2.117:0/646479743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32980eb000 msgr2=0x7f32980eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 --2- 172.21.2.117:0/646479743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32980eb000 0x7f32980eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 -- 172.21.2.117:0/646479743 >> v2:172.21.2.125:3300/0 conn(0x7f32980d18e0 msgr2=0x7f32980ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 --2- 172.21.2.117:0/646479743 >> v2:172.21.2.125:3300/0 conn(0x7f32980d18e0 0x7f32980ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 -- 172.21.2.117:0/646479743 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32980eee00 con 0x7f32980ee3e0 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329d017700 1 --2- 172.21.2.117:0/646479743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32980ee3e0 0x7f32980ee7e0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f3290006b80 tx=0x7f3290009b90).ready entity=mon.0 client_cookie=3d610acc669486df server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f32977fe700 1 -- 172.21.2.117:0/646479743 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32900022e0 con 0x7f32980ee3e0 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f32977fe700 1 -- 172.21.2.117:0/646479743 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32900009b0 con 0x7f32980ee3e0 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f32977fe700 1 -- 172.21.2.117:0/646479743 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32900027e0 con 0x7f32980ee3e0 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 -- 172.21.2.117:0/646479743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32980ee3e0 msgr2=0x7f32980ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.847 7f329e9fa700 1 --2- 172.21.2.117:0/646479743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32980ee3e0 0x7f32980ee7e0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 rx=0x7f3290006b80 tx=0x7f3290009b90).stop 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 -- 172.21.2.117:0/646479743 shutdown_connections 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 --2- 172.21.2.117:0/646479743 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32980ee3e0 0x7f32980ee7e0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 --2- 172.21.2.117:0/646479743 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32980eb000 0x7f32980eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 --2- 172.21.2.117:0/646479743 >> v2:172.21.2.125:3300/0 conn(0x7f32980d18e0 0x7f32980ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:03.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 -- 172.21.2.117:0/646479743 >> 172.21.2.117:0/646479743 conn(0x7f32980ced60 msgr2=0x7f32980cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:03.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 -- 172.21.2.117:0/646479743 shutdown_connections 2021-02-10T11:52:03.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.848 7f329e9fa700 1 -- 172.21.2.117:0/646479743 wait complete. 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 Processor -- start 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 -- start start 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32981a3230 0x7f329819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32981a1420 0x7f32981a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32981a0ea0 con 0x7f32981a0300 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32980def00 con 0x7f32981a3230 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f329e9fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32981a1e40 con 0x7f32981a1420 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f3297fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f3297fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56062/0 (socket says 172.21.2.117:56062) 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.863 7f3297fff700 1 -- 172.21.2.117:0/1407685570 learned_addr learned my addr 172.21.2.117:0/1407685570 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f329c816700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32981a3230 0x7f329819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f329d017700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32981a1420 0x7f32981a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 -- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32981a1420 msgr2=0x7f32981a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32981a1420 0x7f32981a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 -- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32981a3230 msgr2=0x7f329819fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32981a3230 0x7f329819fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 -- 172.21.2.117:0/1407685570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32981a2060 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f3297fff700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7f3288009960 tx=0x7f328800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3288003030 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3288001310 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32981a2060 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32880017f0 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.864 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32981a2060 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.865 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f32880014f0 con 0x7f32981a0300 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.865 7f32957fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 0x7f327c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:03.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.865 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3288032b10 con 0x7f32981a0300 2021-02-10T11:52:03.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.865 7f329c816700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 0x7f327c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:03.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.866 7f329c816700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 0x7f327c01ac10 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:03.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.896 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3298087210 con 0x7f32981a0300 2021-02-10T11:52:03.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:03.899 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f328800be30 con 0x7f32981a0300 2021-02-10T11:52:04.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.024 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32980ede60 con 0x7f32981a0300 2021-02-10T11:52:04.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.024 7f32957fa700 1 -- 172.21.2.117:0/1407685570 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f3288001310 con 0x7f32981a0300 2021-02-10T11:52:04.025 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:04.025 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":134,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:03.821388","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:00.791090","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:01.865056","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:04.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 msgr2=0x7f327c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 0x7f327c01ac10 crc :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 msgr2=0x7f32981a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 rx=0x7f3288009960 tx=0x7f328800bcf0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 shutdown_connections 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f327c018730 0x7f327c01ac10 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32981a0300 0x7f32981a0700 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32981a1420 0x7f32981a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 --2- 172.21.2.117:0/1407685570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32981a3230 0x7f329819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 >> 172.21.2.117:0/1407685570 conn(0x7f329819f610 msgr2=0x7f3298199b70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 shutdown_connections 2021-02-10T11:52:04.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.026 7f329e9fa700 1 -- 172.21.2.117:0/1407685570 wait complete. 2021-02-10T11:52:04.035 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:52:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.450 7f8d6bfef700 1 Processor -- start 2021-02-10T11:52:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.450 7f8d6bfef700 1 -- start start 2021-02-10T11:52:04.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8d640d18f0 0x7f8d640eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 0x7f8d640eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640edee0 con 0x7f8d640ee3e0 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640eba60 con 0x7f8d640d18f0 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6bfef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d1cf0 con 0x7f8d640eb040 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6a60c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6a60c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56070/0 (socket says 172.21.2.117:56070) 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6a60c700 1 -- 172.21.2.117:0/298365931 learned_addr learned my addr 172.21.2.117:0/298365931 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 --2- 172.21.2.117:0/298365931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 0x7f8d640eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d69e0b700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.125:3300/0 conn(0x7f8d640d18f0 0x7f8d640eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 -- 172.21.2.117:0/298365931 >> v2:172.21.2.125:3300/0 conn(0x7f8d640d18f0 msgr2=0x7f8d640eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.125:3300/0 conn(0x7f8d640d18f0 0x7f8d640eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 -- 172.21.2.117:0/298365931 >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 msgr2=0x7f8d640ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.451 7f8d6960a700 1 -- 172.21.2.117:0/298365931 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640eee00 con 0x7f8d640eb040 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d6a60c700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d6960a700 1 --2- 172.21.2.117:0/298365931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 0x7f8d640eb440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8d60006650 tx=0x7f8d60009a00).ready entity=mon.2 client_cookie=e5ffa19a4b9a4292 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d68e09700 1 -- 172.21.2.117:0/298365931 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d60009df0 con 0x7f8d640eb040 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d68e09700 1 -- 172.21.2.117:0/298365931 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d600009b0 con 0x7f8d640eb040 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d68e09700 1 -- 172.21.2.117:0/298365931 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d6000c2b0 con 0x7f8d640eb040 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d6bfef700 1 -- 172.21.2.117:0/298365931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 msgr2=0x7f8d640eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d6bfef700 1 --2- 172.21.2.117:0/298365931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 0x7f8d640eb440 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8d60006650 tx=0x7f8d60009a00).stop 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.452 7f8d6bfef700 1 -- 172.21.2.117:0/298365931 shutdown_connections 2021-02-10T11:52:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.117:3300/0 conn(0x7f8d640ee3e0 0x7f8d640ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 --2- 172.21.2.117:0/298365931 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d640eb040 0x7f8d640eb440 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 --2- 172.21.2.117:0/298365931 >> v2:172.21.2.125:3300/0 conn(0x7f8d640d18f0 0x7f8d640eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 -- 172.21.2.117:0/298365931 >> 172.21.2.117:0/298365931 conn(0x7f8d640cf060 msgr2=0x7f8d640cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:04.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 -- 172.21.2.117:0/298365931 shutdown_connections 2021-02-10T11:52:04.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.453 7f8d6bfef700 1 -- 172.21.2.117:0/298365931 wait complete. 2021-02-10T11:52:04.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.467 7f8d6bfef700 1 Processor -- start 2021-02-10T11:52:04.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.467 7f8d6bfef700 1 -- start start 2021-02-10T11:52:04.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 0x7f8d640dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 0x7f8d640dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64085e50 con 0x7f8d64083990 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64097270 con 0x7f8d641a4580 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d6bfef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64097450 con 0x7f8d641a4180 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d69e0b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d69e0b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56072/0 (socket says 172.21.2.117:56072) 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.468 7f8d69e0b700 1 -- 172.21.2.117:0/1570337604 learned_addr learned my addr 172.21.2.117:0/1570337604 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6a60c700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 0x7f8d640dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 -- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 msgr2=0x7f8d640dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6960a700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 0x7f8d640dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 0x7f8d640dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 -- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 msgr2=0x7f8d640dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 0x7f8d640dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 -- 172.21.2.117:0/1570337604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6409ed70 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6960a700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 0x7f8d640dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6a60c700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 0x7f8d640dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d69e0b700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f8d58006f60 tx=0x7f8d5800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d58006bb0 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d58006340 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d58002030 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6409ed70 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.469 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6409ed70 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.470 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d58006b50 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.470 7f8d567fc700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 0x7f8d4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.470 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d58032680 con 0x7f8d64083990 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.471 7f8d6960a700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 0x7f8d4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:04.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.471 7f8d6960a700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 0x7f8d4c01abc0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:04.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.500 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d64097450 con 0x7f8d64083990 2021-02-10T11:52:04.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.503 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d58030ce0 con 0x7f8d64083990 2021-02-10T11:52:04.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.638 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f8d64097450 con 0x7f8d64083990 2021-02-10T11:52:04.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.830 7f8d567fc700 1 -- 172.21.2.117:0/1570337604 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v111) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8d58006b50 con 0x7f8d64083990 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 msgr2=0x7f8d4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 0x7f8d4c01abc0 crc :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 msgr2=0x7f8d640863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 rx=0x7f8d58006f60 tx=0x7f8d5800b970).stop 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 shutdown_connections 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d641a4580 0x7f8d640dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d641a4180 0x7f8d640dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d4c0186c0 0x7f8d4c01abc0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 --2- 172.21.2.117:0/1570337604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d64083990 0x7f8d640863d0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:04.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.856 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 >> 172.21.2.117:0/1570337604 conn(0x7f8d640b6540 msgr2=0x7f8d6408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:04.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.857 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 shutdown_connections 2021-02-10T11:52:04.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:04.857 7f8d6bfef700 1 -- 172.21.2.117:0/1570337604 wait complete. 2021-02-10T11:52:04.865 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:05.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 Processor -- start 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 -- start start 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09fc0d18e0 0x7f09fc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc0ede60 con 0x7f09fc0eb000 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc0d1ce0 con 0x7f09fc0ef3f0 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a035fd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0ef7f0 con 0x7f09fc0d18e0 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a01419700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09fc0d18e0 0x7f09fc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a00c18700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f09fc0eb000 0x7f09fc0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56330/0 (socket says 172.21.2.117:56330) 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.276 7f0a00c18700 1 -- 172.21.2.117:0/3905755999 learned_addr learned my addr 172.21.2.117:0/3905755999 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:05.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f09effff700 1 -- 172.21.2.117:0/3905755999 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935801695 0 0) 0x7f09f80010b0 con 0x7f09fc0eb000 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f09effff700 1 -- 172.21.2.117:0/3905755999 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e8002fe0 con 0x7f09fc0eb000 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a01419700 1 -- 172.21.2.117:0/3905755999 >> v1:172.21.2.125:6789/0 conn(0x7f09fc0ef3f0 legacy=0x7f09fc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a01419700 1 -- 172.21.2.117:0/3905755999 >> v1:172.21.2.117:6789/0 conn(0x7f09fc0eb000 legacy=0x7f09fc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a01419700 1 -- 172.21.2.117:0/3905755999 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc1bb180 con 0x7f09fc0d18e0 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a01419700 1 --2- 172.21.2.117:0/3905755999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc0d18e0 0x7f09fc0ea9e0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7f09f0006650 tx=0x7f09f0009a80).ready entity=mon.0 client_cookie=7aea8299089fd18d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f09effff700 1 -- 172.21.2.117:0/3905755999 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09f00009b0 con 0x7f09fc0d18e0 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f09effff700 1 -- 172.21.2.117:0/3905755999 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000c0c0 con 0x7f09fc0d18e0 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f09effff700 1 -- 172.21.2.117:0/3905755999 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09f000c520 con 0x7f09fc0d18e0 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a035fd700 1 -- 172.21.2.117:0/3905755999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc0d18e0 msgr2=0x7f09fc0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.277 7f0a035fd700 1 --2- 172.21.2.117:0/3905755999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc0d18e0 0x7f09fc0ea9e0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 rx=0x7f09f0006650 tx=0x7f09f0009a80).stop 2021-02-10T11:52:05.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.278 7f0a035fd700 1 -- 172.21.2.117:0/3905755999 shutdown_connections 2021-02-10T11:52:05.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.278 7f0a035fd700 1 --2- 172.21.2.117:0/3905755999 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc0d18e0 0x7f09fc0ea9e0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.278 7f0a035fd700 1 -- 172.21.2.117:0/3905755999 >> 172.21.2.117:0/3905755999 conn(0x7f09fc0ced60 msgr2=0x7f09fc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:05.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.278 7f0a035fd700 1 -- 172.21.2.117:0/3905755999 shutdown_connections 2021-02-10T11:52:05.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.278 7f0a035fd700 1 -- 172.21.2.117:0/3905755999 wait complete. 2021-02-10T11:52:05.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 Processor -- start 2021-02-10T11:52:05.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 -- start start 2021-02-10T11:52:05.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09fc0e9560 0x7f09fc0e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09fc0e9960 0x7f09fc0e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc196810 con 0x7f09fc196390 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc098ad0 con 0x7f09fc0e9560 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a035fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0e8a40 con 0x7f09fc0e9960 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a01419700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a01419700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56090/0 (socket says 172.21.2.117:56090) 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.293 7f0a01419700 1 -- 172.21.2.117:0/2577045502 learned_addr learned my addr 172.21.2.117:0/2577045502 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 -- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09fc0e9960 msgr2=0x7f09fc0e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09fc0e9960 0x7f09fc0e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 -- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09fc0e9560 msgr2=0x7f09fc0e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09fc0e9560 0x7f09fc0e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 -- 172.21.2.117:0/2577045502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0e8c60 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a01419700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 rx=0x7f09f0009630 tx=0x7f09f000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09f00093a0 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f000bd70 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09f0005560 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc0e8c60 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.294 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc19cd70 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.295 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09f000a000 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.295 7f09edffb700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 0x7f09dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.295 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f09f0032680 con 0x7f09fc196390 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.295 7f0a00c18700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 0x7f09dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.296 7f0a00c18700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 0x7f09dc01abc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.326 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc0e8c60 con 0x7f09fc196390 2021-02-10T11:52:05.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.329 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09f0025f60 con 0x7f09fc196390 2021-02-10T11:52:05.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.455 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09fc0e8c60 con 0x7f09fc196390 2021-02-10T11:52:05.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.456 7f09edffb700 1 -- 172.21.2.117:0/2577045502 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f09f0025f60 con 0x7f09fc196390 2021-02-10T11:52:05.457 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:05.457 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":111,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:04.828714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7286":{"gid":7286,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/604396797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":604396797},{"type":"v1","addr":"172.21.2.125:6835","nonce":604396797}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:05.457 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 111 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.457 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 msgr2=0x7f09dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.457 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 0x7f09dc01abc0 crc :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.457 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 msgr2=0x7f09fc0e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.457 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 rx=0x7f09f0009630 tx=0x7f09f000b980).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 shutdown_connections 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09fc0e9960 0x7f09fc0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09fc0e9560 0x7f09fc0e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09dc0186c0 0x7f09dc01abc0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 --2- 172.21.2.117:0/2577045502 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09fc196390 0x7f09fc0e8f40 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 >> 172.21.2.117:0/2577045502 conn(0x7f09fc19f610 msgr2=0x7f09fc1a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:05.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 shutdown_connections 2021-02-10T11:52:05.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.458 7f0a035fd700 1 -- 172.21.2.117:0/2577045502 wait complete. 2021-02-10T11:52:05.467 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 Processor -- start 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 -- start start 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 0x7ff0600ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0600ede60 con 0x7ff0600ef3f0 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0600d1ce0 con 0x7ff0600eb000 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.879 7ff065a58700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff060088020 con 0x7ff0600d18e0 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff0600ef3f0 0x7ff0601b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56344/0 (socket says 172.21.2.117:56344) 2021-02-10T11:52:05.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05ffff700 1 -- 172.21.2.117:0/1706722478 learned_addr learned my addr 172.21.2.117:0/1706722478 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05f7fe700 1 --2- 172.21.2.117:0/1706722478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 0x7ff0600ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05f7fe700 1 -- 172.21.2.117:0/1706722478 >> v1:172.21.2.125:6789/0 conn(0x7ff0600eb000 legacy=0x7ff0600eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05e7fc700 1 -- 172.21.2.117:0/1706722478 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998141462 0 0) 0x7ff058001500 con 0x7ff0600ef3f0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05f7fe700 1 -- 172.21.2.117:0/1706722478 >> v1:172.21.2.117:6789/0 conn(0x7ff0600ef3f0 legacy=0x7ff0601b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05f7fe700 1 -- 172.21.2.117:0/1706722478 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0601bb180 con 0x7ff0600d18e0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.880 7ff05f7fe700 1 --2- 172.21.2.117:0/1706722478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 0x7ff0600ea9e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff048006650 tx=0x7ff048009a00).ready entity=mon.2 client_cookie=14e949ecec43fc9c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff05e7fc700 1 -- 172.21.2.117:0/1706722478 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0480009b0 con 0x7ff0600d18e0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff05e7fc700 1 -- 172.21.2.117:0/1706722478 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04800c0c0 con 0x7ff0600d18e0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff05e7fc700 1 -- 172.21.2.117:0/1706722478 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff04800c5a0 con 0x7ff0600d18e0 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 -- 172.21.2.117:0/1706722478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 msgr2=0x7ff0600ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 --2- 172.21.2.117:0/1706722478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 0x7ff0600ea9e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff048006650 tx=0x7ff048009a00).stop 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 -- 172.21.2.117:0/1706722478 shutdown_connections 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 --2- 172.21.2.117:0/1706722478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0600d18e0 0x7ff0600ea9e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 -- 172.21.2.117:0/1706722478 >> 172.21.2.117:0/1706722478 conn(0x7ff0600ced60 msgr2=0x7ff0600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:05.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 -- 172.21.2.117:0/1706722478 shutdown_connections 2021-02-10T11:52:05.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.881 7ff065a58700 1 -- 172.21.2.117:0/1706722478 wait complete. 2021-02-10T11:52:05.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 Processor -- start 2021-02-10T11:52:05.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 -- start start 2021-02-10T11:52:05.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0600b5370 0x7ff06019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff06019e620 0x7ff0601afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0601af950 con 0x7ff0600b5370 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06019bb60 con 0x7ff06019b760 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.896 7ff065a58700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06019ea20 con 0x7ff06019e620 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60482/0 (socket says 172.21.2.117:60482) 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 -- 172.21.2.117:0/710048906 learned_addr learned my addr 172.21.2.117:0/710048906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05effd700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0600b5370 0x7ff06019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05ffff700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff06019e620 0x7ff0601afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 -- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff06019e620 msgr2=0x7ff0601afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff06019e620 0x7ff0601afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 -- 172.21.2.117:0/710048906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0600b5370 msgr2=0x7ff06019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0600b5370 0x7ff06019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 -- 172.21.2.117:0/710048906 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06019ec40 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff05f7fe700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7ff04800b5e0 tx=0x7ff04800b980).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0480090c0 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0480009b0 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0480022e0 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff065a58700 1 -- 172.21.2.117:0/710048906 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06019ec40 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.897 7ff065a58700 1 -- 172.21.2.117:0/710048906 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06019ec40 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.898 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff048000c90 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.898 7ff057fff700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 0x7ff04401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.898 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff048032920 con 0x7ff06019b760 2021-02-10T11:52:05.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.898 7ff05effd700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 0x7ff04401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:05.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.898 7ff05effd700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 0x7ff04401abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:05.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.930 7ff065a58700 1 -- 172.21.2.117:0/710048906 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff060087210 con 0x7ff06019b760 2021-02-10T11:52:05.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:05.933 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff048000940 con 0x7ff06019b760 2021-02-10T11:52:06.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.059 7ff065a58700 1 -- 172.21.2.117:0/710048906 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff060087210 con 0x7ff06019b760 2021-02-10T11:52:06.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.061 7ff057fff700 1 -- 172.21.2.117:0/710048906 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14240 (secure 0 0 0) 0x7ff048000c90 con 0x7ff06019b760 2021-02-10T11:52:06.063 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:06.063 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":134,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:03.821388","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:00.791090","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:01.865056","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:06.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 msgr2=0x7ff04401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 0x7ff04401abc0 crc :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 msgr2=0x7ff0600b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7ff04800b5e0 tx=0x7ff04800b980).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 shutdown_connections 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff044018700 0x7ff04401abc0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0600b5370 0x7ff06019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff06019e620 0x7ff0601afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 --2- 172.21.2.117:0/710048906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff06019b760 0x7ff0600b4d50 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 >> 172.21.2.117:0/710048906 conn(0x7ff06019f610 msgr2=0x7ff0601ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:06.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 shutdown_connections 2021-02-10T11:52:06.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.064 7ff065a58700 1 -- 172.21.2.117:0/710048906 wait complete. 2021-02-10T11:52:06.074 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:06.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 Processor -- start 2021-02-10T11:52:06.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 -- start start 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe9407ab40 0x7fbe940794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe94079b10 0x7fbe94079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe940e5770 con 0x7fbe9407af40 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.498 7fbe9a214700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940d18c0 con 0x7fbe9407ab40 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe9a214700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe940d1c60 con 0x7fbe94079b10 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe9407ab40 0x7fbe940794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbe9407ab40 0x7fbe940794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56106/0 (socket says 172.21.2.117:56106) 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 -- 172.21.2.117:0/1087721907 learned_addr learned my addr 172.21.2.117:0/1087721907 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 -- 172.21.2.117:0/1087721907 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe94079b10 msgr2=0x7fbe94079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 --2- 172.21.2.117:0/1087721907 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe94079b10 0x7fbe94079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 -- 172.21.2.117:0/1087721907 >> v1:172.21.2.125:6789/0 conn(0x7fbe9407af40 legacy=0x7fbe9407b340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:06.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 -- 172.21.2.117:0/1087721907 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9407bb30 con 0x7fbe9407ab40 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.499 7fbe93fff700 1 --2- 172.21.2.117:0/1087721907 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe9407ab40 0x7fbe940794f0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 rx=0x7fbe84006700 tx=0x7fbe84009b80).ready entity=mon.0 client_cookie=10909155c4295781 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe92ffd700 1 -- 172.21.2.117:0/1087721907 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe8400c0c0 con 0x7fbe9407ab40 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe92ffd700 1 -- 172.21.2.117:0/1087721907 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8400c340 con 0x7fbe9407ab40 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe92ffd700 1 -- 172.21.2.117:0/1087721907 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe8400c7a0 con 0x7fbe9407ab40 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 -- 172.21.2.117:0/1087721907 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe9407ab40 msgr2=0x7fbe940794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 --2- 172.21.2.117:0/1087721907 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe9407ab40 0x7fbe940794f0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 rx=0x7fbe84006700 tx=0x7fbe84009b80).stop 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 -- 172.21.2.117:0/1087721907 shutdown_connections 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 --2- 172.21.2.117:0/1087721907 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe94079b10 0x7fbe94079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 --2- 172.21.2.117:0/1087721907 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe9407ab40 0x7fbe940794f0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 -- 172.21.2.117:0/1087721907 >> 172.21.2.117:0/1087721907 conn(0x7fbe940d0910 msgr2=0x7fbe940d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 -- 172.21.2.117:0/1087721907 shutdown_connections 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.500 7fbe9a214700 1 -- 172.21.2.117:0/1087721907 wait complete. 2021-02-10T11:52:06.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 Processor -- start 2021-02-10T11:52:06.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 -- start start 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 0x7fbe94100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe9409bac0 0x7fbe9409bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe94100690 con 0x7fbe940b08a0 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9409c4e0 con 0x7fbe941a5510 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.515 7fbe9a214700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9409c700 con 0x7fbe9409bac0 2021-02-10T11:52:06.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56114/0 (socket says 172.21.2.117:56114) 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 -- 172.21.2.117:0/899487406 learned_addr learned my addr 172.21.2.117:0/899487406 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe98831700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe9409bac0 0x7fbe9409bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe93fff700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 0x7fbe94100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 -- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe9409bac0 msgr2=0x7fbe9409bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe9409bac0 0x7fbe9409bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 -- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 msgr2=0x7fbe94100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 0x7fbe94100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 -- 172.21.2.117:0/899487406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94090560 con 0x7fbe940b08a0 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe93fff700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 0x7fbe94100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe937fe700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7fbe8800b5b0 tx=0x7fbe8800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:06.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.516 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe8800bd40 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe88002080 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe880025a0 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe9a214700 1 -- 172.21.2.117:0/899487406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe94090560 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe9a214700 1 -- 172.21.2.117:0/899487406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe94090af0 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe88000c90 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe90ff9700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 0x7fbe7401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.517 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbe88032530 con 0x7fbe940b08a0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.518 7fbe93fff700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 0x7fbe7401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:06.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.518 7fbe93fff700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 0x7fbe7401abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:06.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.548 7fbe9a214700 1 -- 172.21.2.117:0/899487406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe940daa00 con 0x7fbe940b08a0 2021-02-10T11:52:06.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.551 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe88002df0 con 0x7fbe940b08a0 2021-02-10T11:52:06.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.679 7fbe9a214700 1 -- 172.21.2.117:0/899487406 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbe940e5770 con 0x7fbe940b08a0 2021-02-10T11:52:06.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.682 7fbe90ff9700 1 -- 172.21.2.117:0/899487406 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v77) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbe8801f4d0 con 0x7fbe940b08a0 2021-02-10T11:52:06.687 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:52:06.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.687 7fbe9a214700 1 -- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 msgr2=0x7fbe7401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.687 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 0x7fbe7401abc0 crc :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.687 7fbe9a214700 1 -- 172.21.2.117:0/899487406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 msgr2=0x7fbe940b0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.687 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 rx=0x7fbe8800b5b0 tx=0x7fbe8800b950).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 -- 172.21.2.117:0/899487406 shutdown_connections 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe740186c0 0x7fbe7401abc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe940b08a0 0x7fbe940b0ca0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe9409bac0 0x7fbe9409bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 --2- 172.21.2.117:0/899487406 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe941a5510 0x7fbe94100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 -- 172.21.2.117:0/899487406 >> 172.21.2.117:0/899487406 conn(0x7fbe9419f690 msgr2=0x7fbe94091c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 -- 172.21.2.117:0/899487406 shutdown_connections 2021-02-10T11:52:06.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:06.688 7fbe9a214700 1 -- 172.21.2.117:0/899487406 wait complete. 2021-02-10T11:52:06.697 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 Processor -- start 2021-02-10T11:52:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 -- start start 2021-02-10T11:52:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe38c07ab40 0x7fe38c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c079b10 0x7fe38c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0e5770 con 0x7fe38c07af40 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38c0d18c0 con 0x7fe38c07ab40 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.118 7fe394578700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0d1c60 con 0x7fe38c079b10 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe38c07ab40 0x7fe38c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe38c07ab40 0x7fe38c0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56118/0 (socket says 172.21.2.117:56118) 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 -- 172.21.2.117:0/3250493549 learned_addr learned my addr 172.21.2.117:0/3250493549 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 -- 172.21.2.117:0/3250493549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c079b10 msgr2=0x7fe38c079f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 --2- 172.21.2.117:0/3250493549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c079b10 0x7fe38c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 -- 172.21.2.117:0/3250493549 >> v1:172.21.2.125:6789/0 conn(0x7fe38c07af40 legacy=0x7fe38c07b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 -- 172.21.2.117:0/3250493549 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c07bb30 con 0x7fe38c07ab40 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.119 7fe392394700 1 --2- 172.21.2.117:0/3250493549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c07ab40 0x7fe38c0794f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7fe37c006700 tx=0x7fe37c009b00).ready entity=mon.0 client_cookie=3265177bcc3df4d6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe391392700 1 -- 172.21.2.117:0/3250493549 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe37c00c0c0 con 0x7fe38c07ab40 2021-02-10T11:52:07.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe391392700 1 -- 172.21.2.117:0/3250493549 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c00c340 con 0x7fe38c07ab40 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe391392700 1 -- 172.21.2.117:0/3250493549 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe37c00c820 con 0x7fe38c07ab40 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 -- 172.21.2.117:0/3250493549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c07ab40 msgr2=0x7fe38c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 --2- 172.21.2.117:0/3250493549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c07ab40 0x7fe38c0794f0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7fe37c006700 tx=0x7fe37c009b00).stop 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 -- 172.21.2.117:0/3250493549 shutdown_connections 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 --2- 172.21.2.117:0/3250493549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c079b10 0x7fe38c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 --2- 172.21.2.117:0/3250493549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c07ab40 0x7fe38c0794f0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.120 7fe394578700 1 -- 172.21.2.117:0/3250493549 >> 172.21.2.117:0/3250493549 conn(0x7fe38c0d0910 msgr2=0x7fe38c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.121 7fe394578700 1 -- 172.21.2.117:0/3250493549 shutdown_connections 2021-02-10T11:52:07.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.121 7fe394578700 1 -- 172.21.2.117:0/3250493549 wait complete. 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.135 7fe394578700 1 Processor -- start 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.135 7fe394578700 1 -- start start 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 0x7fe38c100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c0fceb0 0x7fe38c0907e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c1002d0 con 0x7fe38c100e70 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0fc480 con 0x7fe38c0fcab0 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe394578700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38c0f95e0 con 0x7fe38c0fceb0 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56122/0 (socket says 172.21.2.117:56122) 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 -- 172.21.2.117:0/253845584 learned_addr learned my addr 172.21.2.117:0/253845584 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe392394700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 0x7fe38c100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 -- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c0fceb0 msgr2=0x7fe38c0907e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c0fceb0 0x7fe38c0907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 -- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 msgr2=0x7fe38c100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 0x7fe38c100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 -- 172.21.2.117:0/253845584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c0f9800 con 0x7fe38c100e70 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe392394700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 0x7fe38c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.136 7fe391b93700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7fe380009980 tx=0x7fe38000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe38000c2f0 con 0x7fe38c100e70 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000c570 con 0x7fe38c100e70 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe394578700 1 -- 172.21.2.117:0/253845584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c0f9800 con 0x7fe38c100e70 2021-02-10T11:52:07.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe394578700 1 -- 172.21.2.117:0/253845584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c0910f0 con 0x7fe38c100e70 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe38000ca90 con 0x7fe38c100e70 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe380001010 con 0x7fe38c100e70 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 0x7fe36c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.137 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe380032c40 con 0x7fe38c100e70 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.138 7fe392394700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 0x7fe36c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.138 7fe392394700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 0x7fe36c01ac10 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.168 7fe394578700 1 -- 172.21.2.117:0/253845584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c0daa00 con 0x7fe38c100e70 2021-02-10T11:52:07.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.171 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe380000ce0 con 0x7fe38c100e70 2021-02-10T11:52:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.300 7fe394578700 1 -- 172.21.2.117:0/253845584 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe38c0e5770 con 0x7fe38c100e70 2021-02-10T11:52:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.303 7fe38affd700 1 -- 172.21.2.117:0/253845584 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v78) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe380001010 con 0x7fe38c100e70 2021-02-10T11:52:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 -- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 msgr2=0x7fe36c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 0x7fe36c01ac10 crc :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 -- 172.21.2.117:0/253845584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 msgr2=0x7fe38c0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 rx=0x7fe380009980 tx=0x7fe38000bcd0).stop 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 -- 172.21.2.117:0/253845584 shutdown_connections 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe36c018730 0x7fe36c01ac10 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.308 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe38c100e70 0x7fe38c0f8fc0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.309 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe38c0fceb0 0x7fe38c0907e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.309 7fe394578700 1 --2- 172.21.2.117:0/253845584 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe38c0fcab0 0x7fe38c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.309 7fe394578700 1 -- 172.21.2.117:0/253845584 >> 172.21.2.117:0/253845584 conn(0x7fe38c19f690 msgr2=0x7fe38c091c40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.309 7fe394578700 1 -- 172.21.2.117:0/253845584 shutdown_connections 2021-02-10T11:52:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.309 7fe394578700 1 -- 172.21.2.117:0/253845584 wait complete. 2021-02-10T11:52:07.318 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 Processor -- start 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 -- start start 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42280ef400 0x7f42280eb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280ef800 con 0x7f42280eb840 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280d1d00 con 0x7f42280d1900 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.741 7f422e6e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42280edee0 con 0x7f42280ef400 2021-02-10T11:52:07.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f422cd01700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f42280eb840 0x7f42281b79b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56382/0 (socket says 172.21.2.117:56382) 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f422cd01700 1 -- 172.21.2.117:0/3706936356 learned_addr learned my addr 172.21.2.117:0/3706936356 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539035346 0 0) 0x7f4220000f90 con 0x7f42280eb840 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4210002fe0 con 0x7f42280eb840 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f42277fe700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42280ef400 0x7f42280eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1891922227 0 0) 0x7f4220001460 con 0x7f42280eb840 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42280ef400 msgr2=0x7f42280eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42280ef400 0x7f42280eb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 >> v1:172.21.2.125:6789/0 conn(0x7f42280d1900 legacy=0x7f42280eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42281bb1d0 con 0x7f42280eb840 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f42200019e0 con 0x7f42280eb840 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.742 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 >> v1:172.21.2.117:6789/0 conn(0x7f42280eb840 legacy=0x7f42281b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4210003b90 0x7f42100060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42100067f0 0x7f4210008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42100095a0 0x7f421000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421000c2c0 con 0x7f4210003b90 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421000c4e0 con 0x7f42100067f0 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421000c700 con 0x7f42100095a0 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4210003b90 0x7f42100060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f422cd01700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42100095a0 0x7f421000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f42277fe700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42100067f0 0x7f4210008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 -- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42100095a0 msgr2=0x7f421000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42100095a0 0x7f421000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 -- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42100067f0 msgr2=0x7f4210008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42100067f0 0x7f4210008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 -- 172.21.2.117:0/3706936356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4210050970 con 0x7f4210003b90 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f422cd01700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42100095a0 0x7f421000ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:07.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4227fff700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4210003b90 0x7f42100060b0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7f4218006ab0 tx=0x7f4218009b40).ready entity=mon.0 client_cookie=616a61d598d2347 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f421800b2e0 con 0x7f4210003b90 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421800b5e0 con 0x7f4210003b90 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f4226ffd700 1 -- 172.21.2.117:0/3706936356 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f421800ba30 con 0x7f4210003b90 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f422e6e4700 1 -- 172.21.2.117:0/3706936356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4210003b90 msgr2=0x7f42100060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f422e6e4700 1 --2- 172.21.2.117:0/3706936356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4210003b90 0x7f42100060b0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 rx=0x7f4218006ab0 tx=0x7f4218009b40).stop 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.743 7f422cd01700 1 -- 172.21.2.117:0/3706936356 reap_dead start 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.744 7f422e6e4700 1 -- 172.21.2.117:0/3706936356 shutdown_connections 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.744 7f422e6e4700 1 -- 172.21.2.117:0/3706936356 >> 172.21.2.117:0/3706936356 conn(0x7f42280cf070 msgr2=0x7f42280cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.744 7f422e6e4700 1 -- 172.21.2.117:0/3706936356 shutdown_connections 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.744 7f422e6e4700 1 -- 172.21.2.117:0/3706936356 wait complete. 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.758 7f422e6e4700 1 Processor -- start 2021-02-10T11:52:07.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 -- start start 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 0x7f42280a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 0x7f42280a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280a5700 con 0x7f42280a68c0 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c0a40 con 0x7f42281a48c0 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422e6e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42280c0c20 con 0x7f422808c720 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422cd01700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422cd01700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56140/0 (socket says 172.21.2.117:56140) 2021-02-10T11:52:07.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422cd01700 1 -- 172.21.2.117:0/3359949765 learned_addr learned my addr 172.21.2.117:0/3359949765 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f4227fff700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 0x7f42280a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f42277fe700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 0x7f42280a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.759 7f422cd01700 1 -- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 msgr2=0x7f42280a62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422cd01700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 0x7f42280a62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422cd01700 1 -- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 msgr2=0x7f42280a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422cd01700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 0x7f42280a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422cd01700 1 -- 172.21.2.117:0/3359949765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280a4390 con 0x7f42280a68c0 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f42277fe700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 0x7f42280a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f4227fff700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 0x7f42280a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422cd01700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7f4220001b90 tx=0x7f422000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4220000cf0 con 0x7f42280a68c0 2021-02-10T11:52:07.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42200070d0 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42280a4390 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4220004930 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.760 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42280a4920 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.761 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42200072b0 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.761 7f4224ff9700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 0x7f420801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.761 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4220032c40 con 0x7f42280a68c0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.761 7f4227fff700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 0x7f420801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:07.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.762 7f4227fff700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 0x7f420801abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:07.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.791 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42280ede70 con 0x7f42280a68c0 2021-02-10T11:52:07.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.793 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4220001270 con 0x7f42280a68c0 2021-02-10T11:52:07.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.923 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f42280ede70 con 0x7f42280a68c0 2021-02-10T11:52:07.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.926 7f4224ff9700 1 -- 172.21.2.117:0/3359949765 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v79) v1 ==== 184+0+0 (secure 0 0 0) 0x7f42200070d0 con 0x7f42280a68c0 2021-02-10T11:52:07.927 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.927 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 msgr2=0x7f420801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.927 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 0x7f420801abc0 crc :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.927 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 msgr2=0x7f42280a3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.927 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 rx=0x7f4220001b90 tx=0x7f422000ac80).stop 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 shutdown_connections 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4208018730 0x7f420801abc0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42280a68c0 0x7f42280a3d70 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f422808c720 0x7f42280a62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 --2- 172.21.2.117:0/3359949765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42281a48c0 0x7f42280a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:07.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 >> 172.21.2.117:0/3359949765 conn(0x7f4228098ef0 msgr2=0x7f422808e770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:07.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 shutdown_connections 2021-02-10T11:52:07.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:07.928 7f422e6e4700 1 -- 172.21.2.117:0/3359949765 wait complete. 2021-02-10T11:52:07.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.358 7f48080f2700 1 Processor -- start 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.358 7f48080f2700 1 -- start start 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f48080f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 0x7f48001b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f48080f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f480007a0e0 0x7f48001b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f48080f2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48000e5770 con 0x7f480007ab40 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f48080f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48000d18c0 con 0x7f480007a0e0 2021-02-10T11:52:08.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f48080f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480007bf50 con 0x7f4800079ce0 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f480670f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f480007a0e0 0x7f48001b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f4805f0e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f480007ab40 0x7f48000794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:49324/0 (socket says 172.21.2.117:49324) 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f480670f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f480007a0e0 0x7f48001b8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60536/0 (socket says 172.21.2.117:60536) 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f4805f0e700 1 -- 172.21.2.117:0/3535592459 learned_addr learned my addr 172.21.2.117:0/3535592459 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.359 7f480570d700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 0x7f48001b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480670f700 1 -- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 msgr2=0x7f48001b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480670f700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 0x7f48001b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480670f700 1 -- 172.21.2.117:0/3535592459 >> v1:172.21.2.125:6789/0 conn(0x7f480007ab40 legacy=0x7f48000794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480670f700 1 -- 172.21.2.117:0/3535592459 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000d1c60 con 0x7f480007a0e0 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480570d700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 0x7f48001b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.360 7f480670f700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f480007a0e0 0x7f48001b8000 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f47f8006e50 tx=0x7f47f8009e70).ready entity=mon.1 client_cookie=48e7681dfd2cc40d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:08.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f4804f0c700 1 -- 172.21.2.117:0/3535592459 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f800b510 con 0x7f480007a0e0 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f4804f0c700 1 -- 172.21.2.117:0/3535592459 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8006790 con 0x7f480007a0e0 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f4804f0c700 1 -- 172.21.2.117:0/3535592459 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f800b940 con 0x7f480007a0e0 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 -- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f480007a0e0 msgr2=0x7f48001b8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f480007a0e0 0x7f48001b8000 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f47f8006e50 tx=0x7f47f8009e70).stop 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 -- 172.21.2.117:0/3535592459 shutdown_connections 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f480007a0e0 0x7f48001b8000 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 --2- 172.21.2.117:0/3535592459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4800079ce0 0x7f48001b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.361 7f48080f2700 1 -- 172.21.2.117:0/3535592459 >> 172.21.2.117:0/3535592459 conn(0x7f48000d0910 msgr2=0x7f48000d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.362 7f48080f2700 1 -- 172.21.2.117:0/3535592459 shutdown_connections 2021-02-10T11:52:08.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.362 7f48080f2700 1 -- 172.21.2.117:0/3535592459 wait complete. 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 Processor -- start 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 -- start start 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48000ab1f0 0x7f48000ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48000db5b0 0x7f48000db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48001001d0 con 0x7f4800094ea0 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48000b8ae0 con 0x7f48000ab1f0 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f48080f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48000dcdc0 con 0x7f48000db5b0 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.376 7f4805f0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56156/0 (socket says 172.21.2.117:56156) 2021-02-10T11:52:08.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 -- 172.21.2.117:0/1715721269 learned_addr learned my addr 172.21.2.117:0/1715721269 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 -- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48000db5b0 msgr2=0x7f48000db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48000db5b0 0x7f48000db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 -- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48000ab1f0 msgr2=0x7f48000ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48000ab1f0 0x7f48000ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 -- 172.21.2.117:0/1715721269 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800100d70 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f4805f0e700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 rx=0x7f47f4001c60 tx=0x7f47f400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f400acc0 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f4004140 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47f4004660 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.377 7f48080f2700 1 -- 172.21.2.117:0/1715721269 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4800100d70 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.378 7f48080f2700 1 -- 172.21.2.117:0/1715721269 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4800100d70 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.378 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f47f4004320 con 0x7f4800094ea0 2021-02-10T11:52:08.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.378 7f47f27fc700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.378 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f47f4032660 con 0x7f4800094ea0 2021-02-10T11:52:08.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.379 7f480570d700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.379 7f480570d700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 0x7f47e801abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:08.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.409 7f48080f2700 1 -- 172.21.2.117:0/1715721269 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48000daa50 con 0x7f4800094ea0 2021-02-10T11:52:08.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.411 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f47f4000980 con 0x7f4800094ea0 2021-02-10T11:52:08.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.540 7f48080f2700 1 -- 172.21.2.117:0/1715721269 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f48000daa50 con 0x7f4800094ea0 2021-02-10T11:52:08.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.544 7f47f27fc700 1 -- 172.21.2.117:0/1715721269 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v80) v1 ==== 184+0+0 (secure 0 0 0) 0x7f47f4004680 con 0x7f4800094ea0 2021-02-10T11:52:08.549 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:52:08.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 msgr2=0x7f47e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 0x7f47e801abc0 crc :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 msgr2=0x7f4800100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 rx=0x7f47f4001c60 tx=0x7f47f400a8d0).stop 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 shutdown_connections 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48000db5b0 0x7f48000db9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48000ab1f0 0x7f48000ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47e80186c0 0x7f47e801abc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 --2- 172.21.2.117:0/1715721269 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4800094ea0 0x7f4800100750 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 >> 172.21.2.117:0/1715721269 conn(0x7f48000bacc0 msgr2=0x7f48000b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 shutdown_connections 2021-02-10T11:52:08.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.549 7f48080f2700 1 -- 172.21.2.117:0/1715721269 wait complete. 2021-02-10T11:52:08.559 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 Processor -- start 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 -- start start 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb8ec0ef3f0 0x7fb8ec0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0ede60 con 0x7fb8ec0d18e0 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ec0d1ce0 con 0x7fb8ec0eb7f0 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.970 7fb8f33a4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ec0ef7f0 con 0x7fb8ec0ef3f0 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f11c0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb8ec0d18e0 0x7fb8ec0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56412/0 (socket says 172.21.2.117:56412) 2021-02-10T11:52:08.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f11c0700 1 -- 172.21.2.117:0/1131574767 learned_addr learned my addr 172.21.2.117:0/1131574767 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f09bf700 1 --2- 172.21.2.117:0/1131574767 >> v2:172.21.2.117:3300/0 conn(0x7fb8ec0ef3f0 0x7fb8ec0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f09bf700 1 -- 172.21.2.117:0/1131574767 >> v1:172.21.2.125:6789/0 conn(0x7fb8ec0eb7f0 legacy=0x7fb8ec1b7960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f09bf700 1 -- 172.21.2.117:0/1131574767 >> v1:172.21.2.117:6789/0 conn(0x7fb8ec0d18e0 legacy=0x7fb8ec0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f09bf700 1 -- 172.21.2.117:0/1131574767 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec1bb180 con 0x7fb8ec0ef3f0 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f09bf700 1 --2- 172.21.2.117:0/1131574767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0ef3f0 0x7fb8ec0eb1d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7fb8e0006650 tx=0x7fb8e0009a80).ready entity=mon.0 client_cookie=6653db3b7d6e82e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:08.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8dffff700 1 -- 172.21.2.117:0/1131574767 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e00009b0 con 0x7fb8ec0ef3f0 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8dffff700 1 -- 172.21.2.117:0/1131574767 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e000c0c0 con 0x7fb8ec0ef3f0 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8dffff700 1 -- 172.21.2.117:0/1131574767 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e000c5e0 con 0x7fb8ec0ef3f0 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f33a4700 1 -- 172.21.2.117:0/1131574767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0ef3f0 msgr2=0x7fb8ec0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.971 7fb8f33a4700 1 --2- 172.21.2.117:0/1131574767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0ef3f0 0x7fb8ec0eb1d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 rx=0x7fb8e0006650 tx=0x7fb8e0009a80).stop 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.972 7fb8f33a4700 1 -- 172.21.2.117:0/1131574767 shutdown_connections 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.972 7fb8f33a4700 1 --2- 172.21.2.117:0/1131574767 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0ef3f0 0x7fb8ec0eb1d0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.972 7fb8f33a4700 1 -- 172.21.2.117:0/1131574767 >> 172.21.2.117:0/1131574767 conn(0x7fb8ec0ced60 msgr2=0x7fb8ec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.972 7fb8f33a4700 1 -- 172.21.2.117:0/1131574767 shutdown_connections 2021-02-10T11:52:08.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.972 7fb8f33a4700 1 -- 172.21.2.117:0/1131574767 wait complete. 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 Processor -- start 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 -- start start 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8ec0f4d60 0x7fb8ec0ba070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8ec0bad40 0x7fb8ec0f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0f5bc0 con 0x7fb8ec0bb140 2021-02-10T11:52:08.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0f5160 con 0x7fb8ec0f4d60 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f33a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec0a73f0 con 0x7fb8ec0bad40 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f19c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f19c1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56172/0 (socket says 172.21.2.117:56172) 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.987 7fb8f19c1700 1 -- 172.21.2.117:0/3926309097 learned_addr learned my addr 172.21.2.117:0/3926309097 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f11c0700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8ec0f4d60 0x7fb8ec0ba070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f09bf700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8ec0bad40 0x7fb8ec0f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 -- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8ec0bad40 msgr2=0x7fb8ec0f55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8ec0bad40 0x7fb8ec0f55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 -- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8ec0f4d60 msgr2=0x7fb8ec0ba070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8ec0f4d60 0x7fb8ec0ba070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 -- 172.21.2.117:0/3926309097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec0a7610 con 0x7fb8ec0bb140 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f19c1700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7fb8e4009940 tx=0x7fb8e400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:08.993 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e4003030 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e4000d30 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb8e40014f0 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec0a7610 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.988 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec0a7610 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.989 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb8e4001270 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.989 7fb8ddffb700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 0x7fb8cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.989 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb8e4032b30 con 0x7fb8ec0bb140 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.989 7fb8f11c0700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 0x7fb8cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:08.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:08.990 7fb8f11c0700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 0x7fb8cc01abc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:09.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.020 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec0f5bc0 con 0x7fb8ec0bb140 2021-02-10T11:52:09.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.023 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb8e400be30 con 0x7fb8ec0bb140 2021-02-10T11:52:09.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.150 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8ec0f5bc0 con 0x7fb8ec0bb140 2021-02-10T11:52:09.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.150 7fb8ddffb700 1 -- 172.21.2.117:0/3926309097 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fb8e4000d30 con 0x7fb8ec0bb140 2021-02-10T11:52:09.152 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:09.153 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":111,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:04.828714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7286":{"gid":7286,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/604396797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":604396797},{"type":"v1","addr":"172.21.2.125:6835","nonce":604396797}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:09.153 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 111 2021-02-10T11:52:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.152 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 msgr2=0x7fb8cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.152 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 0x7fb8cc01abc0 crc :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.152 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 msgr2=0x7fb8ec0a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.152 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 rx=0x7fb8e4009940 tx=0x7fb8e400bcf0).stop 2021-02-10T11:52:09.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 shutdown_connections 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb8cc018730 0x7fb8cc01abc0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb8ec0bb140 0x7fb8ec0a6dd0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb8ec0bad40 0x7fb8ec0f55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 --2- 172.21.2.117:0/3926309097 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb8ec0f4d60 0x7fb8ec0ba070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 >> 172.21.2.117:0/3926309097 conn(0x7fb8ec099f80 msgr2=0x7fb8ec092940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 shutdown_connections 2021-02-10T11:52:09.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:09.153 7fb8f33a4700 1 -- 172.21.2.117:0/3926309097 wait complete. 2021-02-10T11:52:09.364 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:02.818807', 'modified': '2021-02-10 11:52:04.828714', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7286': {'gid': 7286, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/604396797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 604396797}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 604396797}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:52:09.364 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:52:09.364 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:52:09.762 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7286, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9407363579999997, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 111, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 134, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.572809229000001 2021-02-10T11:52:09.763 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:52:09.767 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7286, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9407363579999997, "mdsmap_epoch": 111, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 12.572809229000001 } 2021-02-10T11:52:09.767 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:10.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 Processor -- start 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 -- start start 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 0x7fae3c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fae3c0eb000 0x7fae3c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0ede60 con 0x7fae3c0ee3e0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0eba20 con 0x7fae3c0eb000 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.178 7fae44514700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0d1ce0 con 0x7fae3c0d18e0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae42b31700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fae3c0ee3e0 0x7fae3c0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56430/0 (socket says 172.21.2.117:56430) 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae42b31700 1 -- 172.21.2.117:0/3745721108 learned_addr learned my addr 172.21.2.117:0/3745721108 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 --2- 172.21.2.117:0/3745721108 >> v2:172.21.2.117:3300/0 conn(0x7fae3c0eb000 0x7fae3c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae4132e700 1 -- 172.21.2.117:0/3745721108 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366570546 0 0) 0x7fae34001500 con 0x7fae3c0ee3e0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae4132e700 1 -- 172.21.2.117:0/3745721108 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28002fe0 con 0x7fae3c0ee3e0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae42330700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 0x7fae3c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 -- 172.21.2.117:0/3745721108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 msgr2=0x7fae3c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 0x7fae3c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 -- 172.21.2.117:0/3745721108 >> v1:172.21.2.117:6789/0 conn(0x7fae3c0ee3e0 legacy=0x7fae3c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae4132e700 1 -- 172.21.2.117:0/3745721108 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3638862972 0 0) 0x7fae34001270 con 0x7fae3c0ee3e0 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 -- 172.21.2.117:0/3745721108 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0eefd0 con 0x7fae3c0eb000 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae42330700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 0x7fae3c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.179 7fae41b2f700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c0eb000 0x7fae3c0eb400 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7fae38006650 tx=0x7fae38009a80).ready entity=mon.0 client_cookie=3b4c45332b761b83 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.180 7fae4132e700 1 -- 172.21.2.117:0/3745721108 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae380009b0 con 0x7fae3c0eb000 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.180 7fae4132e700 1 -- 172.21.2.117:0/3745721108 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3800c0c0 con 0x7fae3c0eb000 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.180 7fae4132e700 1 -- 172.21.2.117:0/3745721108 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae3800c5e0 con 0x7fae3c0eb000 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.180 7fae44514700 1 -- 172.21.2.117:0/3745721108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c0eb000 msgr2=0x7fae3c0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.180 7fae44514700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c0eb000 0x7fae3c0eb400 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 rx=0x7fae38006650 tx=0x7fae38009a80).stop 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 -- 172.21.2.117:0/3745721108 shutdown_connections 2021-02-10T11:52:10.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c0eb000 0x7fae3c0eb400 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 --2- 172.21.2.117:0/3745721108 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c0d18e0 0x7fae3c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 -- 172.21.2.117:0/3745721108 >> 172.21.2.117:0/3745721108 conn(0x7fae3c0ced60 msgr2=0x7fae3c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 -- 172.21.2.117:0/3745721108 shutdown_connections 2021-02-10T11:52:10.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.181 7fae44514700 1 -- 172.21.2.117:0/3745721108 wait complete. 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.195 7fae44514700 1 Processor -- start 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.195 7fae44514700 1 -- start start 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 0x7fae3c091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 0x7fae3c095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0967f0 con 0x7fae3c096270 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1972b0 con 0x7fae3c091b00 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae44514700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c197490 con 0x7fae3c095850 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae42b31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae42b31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56186/0 (socket says 172.21.2.117:56186) 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.196 7fae42b31700 1 -- 172.21.2.117:0/2473613847 learned_addr learned my addr 172.21.2.117:0/2473613847 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42330700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 0x7fae3c091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae41b2f700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 0x7fae3c095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 -- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 msgr2=0x7fae3c095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 0x7fae3c095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 -- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 msgr2=0x7fae3c091f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 0x7fae3c091f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 -- 172.21.2.117:0/2473613847 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c1976b0 con 0x7fae3c096270 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42330700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 0x7fae3c091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae41b2f700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 0x7fae3c095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae42b31700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7fae3400a8f0 tx=0x7fae3400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae34007d10 con 0x7fae3c096270 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae340070d0 con 0x7fae3c096270 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fae34004930 con 0x7fae3c096270 2021-02-10T11:52:10.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae44514700 1 -- 172.21.2.117:0/2473613847 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1976b0 con 0x7fae3c096270 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.197 7fae44514700 1 -- 172.21.2.117:0/2473613847 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c1976b0 con 0x7fae3c096270 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.198 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fae340072b0 con 0x7fae3c096270 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.198 7fae2effd700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 0x7fae2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.198 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fae34032a00 con 0x7fae3c096270 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.198 7fae42330700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 0x7fae2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.199 7fae42330700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 0x7fae2401abc0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.228 7fae44514700 1 -- 172.21.2.117:0/2473613847 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c087290 con 0x7fae3c096270 2021-02-10T11:52:10.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.231 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fae34007d10 con 0x7fae3c096270 2021-02-10T11:52:10.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.357 7fae44514700 1 -- 172.21.2.117:0/2473613847 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae3c0ede60 con 0x7fae3c096270 2021-02-10T11:52:10.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.357 7fae2effd700 1 -- 172.21.2.117:0/2473613847 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fae340071b0 con 0x7fae3c096270 2021-02-10T11:52:10.359 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:10.359 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":134,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:03.821388","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:00.791090","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:01.865056","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 msgr2=0x7fae2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 0x7fae2401abc0 crc :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 msgr2=0x7fae3c196c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 rx=0x7fae3400a8f0 tx=0x7fae3400ac90).stop 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 shutdown_connections 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fae240186c0 0x7fae2401abc0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fae3c096270 0x7fae3c196c90 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fae3c095850 0x7fae3c095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 --2- 172.21.2.117:0/2473613847 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fae3c091b00 0x7fae3c091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 >> 172.21.2.117:0/2473613847 conn(0x7fae3c1a3620 msgr2=0x7fae3c092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 shutdown_connections 2021-02-10T11:52:10.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.359 7fae44514700 1 -- 172.21.2.117:0/2473613847 wait complete. 2021-02-10T11:52:10.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:52:00.791090', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:52:01.865056', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:52:10.370 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:52:10.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.783 7f6620f16700 1 Processor -- start 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.783 7f6620f16700 1 -- start start 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f6620f16700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f661c0d18f0 0x7f661c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f6620f16700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 0x7f661c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f6620f16700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c0ef7f0 con 0x7f661c0eb040 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f6620f16700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c0eded0 con 0x7f661c0d18f0 2021-02-10T11:52:10.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f6620f16700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0d1cf0 con 0x7f661c0ef3f0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f661ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f661c0d18f0 0x7f661c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f661ab13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f661c0d18f0 0x7f661c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56190/0 (socket says 172.21.2.117:56190) 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f661ab13700 1 -- 172.21.2.117:0/2921902969 learned_addr learned my addr 172.21.2.117:0/2921902969 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.784 7f661b314700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 0x7f661c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661ab13700 1 -- 172.21.2.117:0/2921902969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 msgr2=0x7f661c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661ab13700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 0x7f661c1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661ab13700 1 -- 172.21.2.117:0/2921902969 >> v1:172.21.2.125:6789/0 conn(0x7f661c0eb040 legacy=0x7f661c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661ab13700 1 -- 172.21.2.117:0/2921902969 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c1b8fd0 con 0x7f661c0d18f0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661b314700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 0x7f661c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f661ab13700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0d18f0 0x7f661c0eaa20 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f660c006700 tx=0x7f660c009b80).ready entity=mon.0 client_cookie=968b6d34dd56c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f6619b11700 1 -- 172.21.2.117:0/2921902969 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c00c0c0 con 0x7f661c0d18f0 2021-02-10T11:52:10.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f6619b11700 1 -- 172.21.2.117:0/2921902969 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c00c440 con 0x7f661c0d18f0 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f6619b11700 1 -- 172.21.2.117:0/2921902969 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660c00c960 con 0x7f661c0d18f0 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f6620f16700 1 -- 172.21.2.117:0/2921902969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0d18f0 msgr2=0x7f661c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.785 7f6620f16700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0d18f0 0x7f661c0eaa20 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f660c006700 tx=0x7f660c009b80).stop 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 -- 172.21.2.117:0/2921902969 shutdown_connections 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c0ef3f0 0x7f661c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 --2- 172.21.2.117:0/2921902969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c0d18f0 0x7f661c0eaa20 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 -- 172.21.2.117:0/2921902969 >> 172.21.2.117:0/2921902969 conn(0x7f661c0cf060 msgr2=0x7f661c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 -- 172.21.2.117:0/2921902969 shutdown_connections 2021-02-10T11:52:10.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.786 7f6620f16700 1 -- 172.21.2.117:0/2921902969 wait complete. 2021-02-10T11:52:10.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.800 7f6620f16700 1 Processor -- start 2021-02-10T11:52:10.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.800 7f6620f16700 1 -- start start 2021-02-10T11:52:10.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c07c7c0 0x7f661c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c0a13a0 0x7f661c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c07cca0 con 0x7f661c1a3930 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c18fb70 con 0x7f661c0a13a0 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f6620f16700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0dc6d0 con 0x7f661c07c7c0 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56198/0 (socket says 172.21.2.117:56198) 2021-02-10T11:52:10.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 -- 172.21.2.117:0/3774881048 learned_addr learned my addr 172.21.2.117:0/3774881048 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:10.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 -- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c07c7c0 msgr2=0x7f661c1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c07c7c0 0x7f661c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:10.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 -- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c0a13a0 msgr2=0x7f661c0a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.801 7f661a312700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c0a13a0 0x7f661c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:10.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f661a312700 1 -- 172.21.2.117:0/3774881048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c07d220 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f661a312700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 rx=0x7f660400b5b0 tx=0x7f660400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f660400bd40 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6604006240 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f6620f16700 1 -- 172.21.2.117:0/3774881048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c07d220 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f6620f16700 1 -- 172.21.2.117:0/3774881048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c07d7b0 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.802 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6604005720 con 0x7f661c1a3930 2021-02-10T11:52:10.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.803 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f66040054a0 con 0x7f661c1a3930 2021-02-10T11:52:10.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.803 7f66137fe700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 0x7f660001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:10.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.803 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6604032970 con 0x7f661c1a3930 2021-02-10T11:52:10.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.803 7f661ab13700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 0x7f660001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:10.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.804 7f661ab13700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 0x7f660001ac10 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:10.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.834 7f6620f16700 1 -- 172.21.2.117:0/3774881048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0daff0 con 0x7f661c1a3930 2021-02-10T11:52:10.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.836 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6604000940 con 0x7f661c1a3930 2021-02-10T11:52:10.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.963 7f6620f16700 1 -- 172.21.2.117:0/3774881048 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f661c0ede60 con 0x7f661c1a3930 2021-02-10T11:52:10.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.964 7f66137fe700 1 -- 172.21.2.117:0/3774881048 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7f660400bd40 con 0x7f661c1a3930 2021-02-10T11:52:10.966 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:10.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 msgr2=0x7f660001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 0x7f660001ac10 crc :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 msgr2=0x7f661c0a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:10.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 rx=0x7f660400b5b0 tx=0x7f660400b950).stop 2021-02-10T11:52:10.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 shutdown_connections 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6600018730 0x7f660001ac10 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f661c1a3930 0x7f661c0a0d80 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f661c0a13a0 0x7f661c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 --2- 172.21.2.117:0/3774881048 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f661c07c7c0 0x7f661c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 >> 172.21.2.117:0/3774881048 conn(0x7f661c087090 msgr2=0x7f661c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 shutdown_connections 2021-02-10T11:52:10.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:10.966 7f6620f16700 1 -- 172.21.2.117:0/3774881048 wait complete. 2021-02-10T11:52:10.976 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:52:11.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 Processor -- start 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 -- start start 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd5e40ef3f0 0x7fd5e40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 0x7fd5e41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e40ef7f0 con 0x7fd5e40d18f0 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e40eded0 con 0x7fd5e40ef3f0 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e9528700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e40d1cf0 con 0x7fd5e40eb830 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e32bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd5e40d18f0 0x7fd5e40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56454/0 (socket says 172.21.2.117:56454) 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.394 7fd5e32bf700 1 -- 172.21.2.117:0/3914796807 learned_addr learned my addr 172.21.2.117:0/3914796807 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 --2- 172.21.2.117:0/3914796807 >> v2:172.21.2.125:3300/0 conn(0x7fd5e40ef3f0 0x7fd5e40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e3ac0700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 0x7fd5e41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e22bd700 1 -- 172.21.2.117:0/3914796807 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273356745 0 0) 0x7fd5cc000ce0 con 0x7fd5e40d18f0 2021-02-10T11:52:11.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e22bd700 1 -- 172.21.2.117:0/3914796807 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d0002fe0 con 0x7fd5e40d18f0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 -- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 msgr2=0x7fd5e41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 0x7fd5e41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 -- 172.21.2.117:0/3914796807 >> v1:172.21.2.117:6789/0 conn(0x7fd5e40d18f0 legacy=0x7fd5e40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 -- 172.21.2.117:0/3914796807 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e41b8fd0 con 0x7fd5e40ef3f0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e3ac0700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 0x7fd5e41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.395 7fd5e2abe700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e40ef3f0 0x7fd5e40eb210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fd5d4006650 tx=0x7fd5d4009a80).ready entity=mon.1 client_cookie=6275632487ac034a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.396 7fd5e22bd700 1 -- 172.21.2.117:0/3914796807 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d400c0c0 con 0x7fd5e40ef3f0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.396 7fd5e22bd700 1 -- 172.21.2.117:0/3914796807 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400c340 con 0x7fd5e40ef3f0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.396 7fd5e22bd700 1 -- 172.21.2.117:0/3914796807 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5d400c860 con 0x7fd5e40ef3f0 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.396 7fd5e9528700 1 -- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e40ef3f0 msgr2=0x7fd5e40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.396 7fd5e9528700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e40ef3f0 0x7fd5e40eb210 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fd5d4006650 tx=0x7fd5d4009a80).stop 2021-02-10T11:52:11.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 -- 172.21.2.117:0/3914796807 shutdown_connections 2021-02-10T11:52:11.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40eb830 0x7fd5e41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 --2- 172.21.2.117:0/3914796807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e40ef3f0 0x7fd5e40eb210 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 -- 172.21.2.117:0/3914796807 >> 172.21.2.117:0/3914796807 conn(0x7fd5e40cf060 msgr2=0x7fd5e40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:11.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 -- 172.21.2.117:0/3914796807 shutdown_connections 2021-02-10T11:52:11.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.397 7fd5e9528700 1 -- 172.21.2.117:0/3914796807 wait complete. 2021-02-10T11:52:11.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.411 7fd5e9528700 1 Processor -- start 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.411 7fd5e9528700 1 -- start start 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e408c210 0x7fd5e408c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 0x7fd5e40b7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e408cc30 con 0x7fd5e408b7f0 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e40b8460 con 0x7fd5e408c210 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e9528700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e40b8680 con 0x7fd5e40b7a40 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e32bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e32bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56210/0 (socket says 172.21.2.117:56210) 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e32bf700 1 -- 172.21.2.117:0/841996243 learned_addr learned my addr 172.21.2.117:0/841996243 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e3ac0700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 0x7fd5e40b7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e2abe700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e408c210 0x7fd5e408c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e3ac0700 1 -- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e408c210 msgr2=0x7fd5e408c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.412 7fd5e3ac0700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e408c210 0x7fd5e408c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e3ac0700 1 -- 172.21.2.117:0/841996243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 msgr2=0x7fd5e408bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e3ac0700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e3ac0700 1 -- 172.21.2.117:0/841996243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e40b88a0 con 0x7fd5e40b7a40 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e32bf700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e3ac0700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 0x7fd5e40b7e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fd5dc00b8a0 tx=0x7fd5dc00bc40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5dc006790 con 0x7fd5e40b7a40 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc000f90 con 0x7fd5e40b7a40 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e9528700 1 -- 172.21.2.117:0/841996243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e40b88a0 con 0x7fd5e40b7a40 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd5dc006790 con 0x7fd5e40b7a40 2021-02-10T11:52:11.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.413 7fd5e9528700 1 -- 172.21.2.117:0/841996243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e40b8e30 con 0x7fd5e40b7a40 2021-02-10T11:52:11.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.414 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd5dc00a000 con 0x7fd5e40b7a40 2021-02-10T11:52:11.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.414 7fd5dbfff700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 0x7fd5c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:11.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.414 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd5dc0327c0 con 0x7fd5e40b7a40 2021-02-10T11:52:11.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.414 7fd5e32bf700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 0x7fd5c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:11.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.414 7fd5e32bf700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 0x7fd5c801abc0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.444 7fd5e9528700 1 -- 172.21.2.117:0/841996243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e40ede60 con 0x7fd5e40b7a40 2021-02-10T11:52:11.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.448 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd5dc0011f0 con 0x7fd5e40b7a40 2021-02-10T11:52:11.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.575 7fd5e9528700 1 -- 172.21.2.117:0/841996243 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd5e40ede60 con 0x7fd5e40b7a40 2021-02-10T11:52:11.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.577 7fd5dbfff700 1 -- 172.21.2.117:0/841996243 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v134) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd5dc00a000 con 0x7fd5e40b7a40 2021-02-10T11:52:11.580 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:11.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 msgr2=0x7fd5c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 0x7fd5c801abc0 crc :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 msgr2=0x7fd5e40b7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 0x7fd5e40b7e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fd5dc00b8a0 tx=0x7fd5dc00bc40).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 shutdown_connections 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5e40b7a40 0x7fd5e40b7e40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd5e408c210 0x7fd5e408c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5c80186c0 0x7fd5c801abc0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 --2- 172.21.2.117:0/841996243 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5e408b7f0 0x7fd5e408bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:11.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 >> 172.21.2.117:0/841996243 conn(0x7fd5e408ede0 msgr2=0x7fd5e40bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:11.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 shutdown_connections 2021-02-10T11:52:11.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:11.580 7fd5e9528700 1 -- 172.21.2.117:0/841996243 wait complete. 2021-02-10T11:52:11.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 Processor -- start 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 -- start start 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf200d18f0 0x7fbf200eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf200eb040 0x7fbf200eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200ede60 con 0x7fbf200ee3e0 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf200ebad0 con 0x7fbf200d18f0 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf255f3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf200d1cf0 con 0x7fbf200eb040 2021-02-10T11:52:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf1f2bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf200d18f0 0x7fbf200eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf1f2bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf200d18f0 0x7fbf200eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56218/0 (socket says 172.21.2.117:56218) 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf1f2bf700 1 -- 172.21.2.117:0/1513565465 learned_addr learned my addr 172.21.2.117:0/1513565465 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.004 7fbf1eabe700 1 --2- 172.21.2.117:0/1513565465 >> v2:172.21.2.125:3300/0 conn(0x7fbf200eb040 0x7fbf200eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1f2bf700 1 -- 172.21.2.117:0/1513565465 >> v2:172.21.2.125:3300/0 conn(0x7fbf200eb040 msgr2=0x7fbf200eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1f2bf700 1 --2- 172.21.2.117:0/1513565465 >> v2:172.21.2.125:3300/0 conn(0x7fbf200eb040 0x7fbf200eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1f2bf700 1 -- 172.21.2.117:0/1513565465 >> v1:172.21.2.125:6789/0 conn(0x7fbf200ee3e0 legacy=0x7fbf200ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1f2bf700 1 -- 172.21.2.117:0/1513565465 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200eefd0 con 0x7fbf200d18f0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1f2bf700 1 --2- 172.21.2.117:0/1513565465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf200d18f0 0x7fbf200eaa20 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 rx=0x7fbf10006700 tx=0x7fbf10009b80).ready entity=mon.0 client_cookie=ed423687f70f20d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1e2bd700 1 -- 172.21.2.117:0/1513565465 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1000c0c0 con 0x7fbf200d18f0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1e2bd700 1 -- 172.21.2.117:0/1513565465 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1000c340 con 0x7fbf200d18f0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf1e2bd700 1 -- 172.21.2.117:0/1513565465 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1000c860 con 0x7fbf200d18f0 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf255f3700 1 -- 172.21.2.117:0/1513565465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf200d18f0 msgr2=0x7fbf200eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.005 7fbf255f3700 1 --2- 172.21.2.117:0/1513565465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf200d18f0 0x7fbf200eaa20 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 rx=0x7fbf10006700 tx=0x7fbf10009b80).stop 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 -- 172.21.2.117:0/1513565465 shutdown_connections 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 --2- 172.21.2.117:0/1513565465 >> v2:172.21.2.125:3300/0 conn(0x7fbf200eb040 0x7fbf200eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 --2- 172.21.2.117:0/1513565465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf200d18f0 0x7fbf200eaa20 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 -- 172.21.2.117:0/1513565465 >> 172.21.2.117:0/1513565465 conn(0x7fbf200cf060 msgr2=0x7fbf200cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 -- 172.21.2.117:0/1513565465 shutdown_connections 2021-02-10T11:52:12.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.006 7fbf255f3700 1 -- 172.21.2.117:0/1513565465 wait complete. 2021-02-10T11:52:12.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 Processor -- start 2021-02-10T11:52:12.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 -- start start 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf201a4240 0x7fbf201a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf2009ed70 0x7fbf2009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2009fb90 con 0x7fbf2009f790 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf200dc640 con 0x7fbf201a4240 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf255f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf20097270 con 0x7fbf2009ed70 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf1fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf1fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56226/0 (socket says 172.21.2.117:56226) 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.021 7fbf1fac0700 1 -- 172.21.2.117:0/1192633034 learned_addr learned my addr 172.21.2.117:0/1192633034 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 -- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf2009ed70 msgr2=0x7fbf2009f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf2009ed70 0x7fbf2009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 -- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf201a4240 msgr2=0x7fbf201a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf201a4240 0x7fbf201a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 -- 172.21.2.117:0/1192633034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200b8780 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf1fac0700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7fbf18009940 tx=0x7fbf1800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf18000ce0 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1800c2f0 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf1800c810 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf200b8780 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.022 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf200b8d10 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.023 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf18001220 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.023 7fbf07fff700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 0x7fbf0001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.023 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf18032a40 con 0x7fbf2009f790 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.023 7fbf1f2bf700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 0x7fbf0001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.024 7fbf1f2bf700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 0x7fbf0001abc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.054 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf200daff0 con 0x7fbf2009f790 2021-02-10T11:52:12.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.057 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf18000cc0 con 0x7fbf2009f790 2021-02-10T11:52:12.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.183 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fbf200ede60 con 0x7fbf2009f790 2021-02-10T11:52:12.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.184 7fbf07fff700 1 -- 172.21.2.117:0/1192633034 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v134) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbf1801f690 con 0x7fbf2009f790 2021-02-10T11:52:12.186 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:12.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.185 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 msgr2=0x7fbf0001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.185 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 0x7fbf0001abc0 crc :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.185 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 msgr2=0x7fbf200b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.185 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 rx=0x7fbf18009940 tx=0x7fbf1800bcf0).stop 2021-02-10T11:52:12.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 shutdown_connections 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf000186c0 0x7fbf0001abc0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf2009f790 0x7fbf200b8160 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf2009ed70 0x7fbf2009f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 --2- 172.21.2.117:0/1192633034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf201a4240 0x7fbf201a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 >> 172.21.2.117:0/1192633034 conn(0x7fbf20087090 msgr2=0x7fbf2008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 shutdown_connections 2021-02-10T11:52:12.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.186 7fbf255f3700 1 -- 172.21.2.117:0/1192633034 wait complete. 2021-02-10T11:52:12.196 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:52:12.197 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.607 7fa401c24700 1 Processor -- start 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.607 7fa401c24700 1 -- start start 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa401c24700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa3fc0d18e0 0x7fa3fc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa401c24700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa401c24700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc0ede60 con 0x7fa3fc0eb000 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa401c24700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0ef7f0 con 0x7fa3fc0ef3f0 2021-02-10T11:52:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa401c24700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0d1ce0 con 0x7fa3fc0d18e0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56232/0 (socket says 172.21.2.117:56232) 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 -- 172.21.2.117:0/2078687898 learned_addr learned my addr 172.21.2.117:0/2078687898 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 -- 172.21.2.117:0/2078687898 >> v2:172.21.2.125:3300/0 conn(0x7fa3fc0d18e0 msgr2=0x7fa3fc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 --2- 172.21.2.117:0/2078687898 >> v2:172.21.2.125:3300/0 conn(0x7fa3fc0d18e0 0x7fa3fc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 -- 172.21.2.117:0/2078687898 >> v1:172.21.2.125:6789/0 conn(0x7fa3fc0eb000 legacy=0x7fa3fc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.608 7fa3fbfff700 1 -- 172.21.2.117:0/2078687898 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc1b8f90 con 0x7fa3fc0ef3f0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa3fbfff700 1 --2- 172.21.2.117:0/2078687898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7fa3f4006f00 tx=0x7fa3f4009f10).ready entity=mon.0 client_cookie=64b89a4f94580ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa3fa7fc700 1 -- 172.21.2.117:0/2078687898 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3f4003030 con 0x7fa3fc0ef3f0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa3fa7fc700 1 -- 172.21.2.117:0/2078687898 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f400b3b0 con 0x7fa3fc0ef3f0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa3fa7fc700 1 -- 172.21.2.117:0/2078687898 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3f400b8d0 con 0x7fa3fc0ef3f0 2021-02-10T11:52:12.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa401c24700 1 -- 172.21.2.117:0/2078687898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc0ef3f0 msgr2=0x7fa3fc1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.609 7fa401c24700 1 --2- 172.21.2.117:0/2078687898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 rx=0x7fa3f4006f00 tx=0x7fa3f4009f10).stop 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 -- 172.21.2.117:0/2078687898 shutdown_connections 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 --2- 172.21.2.117:0/2078687898 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc0ef3f0 0x7fa3fc1b7960 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 --2- 172.21.2.117:0/2078687898 >> v2:172.21.2.125:3300/0 conn(0x7fa3fc0d18e0 0x7fa3fc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 -- 172.21.2.117:0/2078687898 >> 172.21.2.117:0/2078687898 conn(0x7fa3fc0ced60 msgr2=0x7fa3fc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 -- 172.21.2.117:0/2078687898 shutdown_connections 2021-02-10T11:52:12.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.610 7fa401c24700 1 -- 172.21.2.117:0/2078687898 wait complete. 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 Processor -- start 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 -- start start 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 0x7fa3fc0e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 0x7fa3fc0e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc196810 con 0x7fa3fc196390 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc098ad0 con 0x7fa3fc0e9960 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.625 7fa401c24700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0e8a40 con 0x7fa3fc0e9560 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56236/0 (socket says 172.21.2.117:56236) 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 -- 172.21.2.117:0/858984271 learned_addr learned my addr 172.21.2.117:0/858984271 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fbfff700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 0x7fa3fc0e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3faffd700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 0x7fa3fc0e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 -- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 msgr2=0x7fa3fc0e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 0x7fa3fc0e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 -- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 msgr2=0x7fa3fc0e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 0x7fa3fc0e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 -- 172.21.2.117:0/858984271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc0e8c60 con 0x7fa3fc196390 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3faffd700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 0x7fa3fc0e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fbfff700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 0x7fa3fc0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3fb7fe700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa3ec006f00 tx=0x7fa3ec00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3ec0009b0 con 0x7fa3fc196390 2021-02-10T11:52:12.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.626 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec006350 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3ec000c90 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa401c24700 1 -- 172.21.2.117:0/858984271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc0e8c60 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa401c24700 1 -- 172.21.2.117:0/858984271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc19cd70 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3ec000c90 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa3e3fff700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 0x7fa3dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.627 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa3ec032700 con 0x7fa3fc196390 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.628 7fa3faffd700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 0x7fa3dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:12.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.628 7fa3faffd700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 0x7fa3dc01abc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:12.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.657 7fa401c24700 1 -- 172.21.2.117:0/858984271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc0e8c60 con 0x7fa3fc196390 2021-02-10T11:52:12.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.660 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3ec000940 con 0x7fa3fc196390 2021-02-10T11:52:12.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.786 7fa401c24700 1 -- 172.21.2.117:0/858984271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3fc0e8c60 con 0x7fa3fc196390 2021-02-10T11:52:12.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.786 7fa3e3fff700 1 -- 172.21.2.117:0/858984271 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fa3ec00bd30 con 0x7fa3fc196390 2021-02-10T11:52:12.789 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 111 2021-02-10T11:52:12.789 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:12.789 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":111,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:04.828714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7286":{"gid":7286,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/604396797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":604396797},{"type":"v1","addr":"172.21.2.125:6835","nonce":604396797}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:12.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.788 7fa401c24700 1 -- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 msgr2=0x7fa3dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.788 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 0x7fa3dc01abc0 crc :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.788 7fa401c24700 1 -- 172.21.2.117:0/858984271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 msgr2=0x7fa3fc0e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:12.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.788 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 rx=0x7fa3ec006f00 tx=0x7fa3ec00b940).stop 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 -- 172.21.2.117:0/858984271 shutdown_connections 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3fc0e9960 0x7fa3fc0e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3fc0e9560 0x7fa3fc0e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3dc0186c0 0x7fa3dc01abc0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 --2- 172.21.2.117:0/858984271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3fc196390 0x7fa3fc0e8f40 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 -- 172.21.2.117:0/858984271 >> 172.21.2.117:0/858984271 conn(0x7fa3fc19f610 msgr2=0x7fa3fc1a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 -- 172.21.2.117:0/858984271 shutdown_connections 2021-02-10T11:52:12.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:12.789 7fa401c24700 1 -- 172.21.2.117:0/858984271 wait complete. 2021-02-10T11:52:13.000 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:02.818807', 'modified': '2021-02-10 11:52:04.828714', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7286': {'gid': 7286, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/604396797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 604396797}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 604396797}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:52:13.000 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:52:13.001 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:52:13.400 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:52:13.400 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7286, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.578493791, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 111, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 134, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.210568903999999 2021-02-10T11:52:13.401 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:52:13.404 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7286, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.578493791, "mdsmap_epoch": 111, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 16.210568903999999 } 2021-02-10T11:52:13.405 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:52:13.405 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:52:13.405 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:52:13.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:13.405 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:13.739 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:52:14.083 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:52:14.084 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:52:14.427 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:52:14.427 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:52:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:52:14.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:14.428 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:52:14.763 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:52:14.763 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:14.766 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:52:15.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:52:15.046 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:52:15.110 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:52:15.110 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:52:15.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[169730]: starting ceph client 2021-02-10T11:52:15.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:52:15.175 7f5c2d9bbf80 -1 init, newargv = 0x561f79550990 newargc=9 2021-02-10T11:52:15.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[169730]: starting fuse 2021-02-10T11:52:15.389 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:52:15.389 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:52:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:52:15.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:15.390 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:52:15.726 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:52:15.726 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:52:15.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:15.726 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:52:15.727 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:52:15.728 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:52:16.090 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.169730.asok 2021-02-10T11:52:16.093 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.169730.asok 2021-02-10T11:52:16.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:16.093 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.169730.asok status 2021-02-10T11:52:16.482 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:52:16.482 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:52:16.482 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:52:16.482 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:52:16.482 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "169730", 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11595, 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11595 2021-02-10T11:52:16.483 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1060224965 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1060224965 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11595 172.21.2.117:0/1060224965", 2021-02-10T11:52:16.484 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1060224965", 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 111, 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 134, 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:52:16.485 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:52:16.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:16.487 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:16.487 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:16.822 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:52:16.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:52:16.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:16.823 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:17.172 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:17.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.581 7f44bd3de700 1 Processor -- start 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.581 7f44bd3de700 1 -- start start 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44bd3de700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f44b80d18e0 0x7f44b80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44bd3de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80eb000 0x7f44b80eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44bd3de700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b80ede60 con 0x7f44b80ee3e0 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44bd3de700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b80eba90 con 0x7f44b80d18e0 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44bd3de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80d1ce0 con 0x7f44b80eb000 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b787e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f44b80ee3e0 0x7f44b80ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56506/0 (socket says 172.21.2.117:56506) 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b787e700 1 -- 172.21.2.117:0/3187292941 learned_addr learned my addr 172.21.2.117:0/3187292941 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b707d700 1 --2- 172.21.2.117:0/3187292941 >> v2:172.21.2.117:3300/0 conn(0x7f44b80d18e0 0x7f44b80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:17.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b687c700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80eb000 0x7f44b80eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b607b700 1 -- 172.21.2.117:0/3187292941 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748159445 0 0) 0x7f44b0001500 con 0x7f44b80ee3e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b607b700 1 -- 172.21.2.117:0/3187292941 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a4002fe0 con 0x7f44b80ee3e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b707d700 1 -- 172.21.2.117:0/3187292941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80eb000 msgr2=0x7f44b80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b707d700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80eb000 0x7f44b80eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.582 7f44b707d700 1 -- 172.21.2.117:0/3187292941 >> v1:172.21.2.117:6789/0 conn(0x7f44b80ee3e0 legacy=0x7f44b80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44b707d700 1 -- 172.21.2.117:0/3187292941 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80eefd0 con 0x7f44b80d18e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44b707d700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80d18e0 0x7f44b80ea9e0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f44a0006650 tx=0x7f44a0009a80).ready entity=mon.0 client_cookie=31b57f65bea8bbc6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44b607b700 1 -- 172.21.2.117:0/3187292941 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44a00009b0 con 0x7f44b80d18e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44b607b700 1 -- 172.21.2.117:0/3187292941 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a000c0c0 con 0x7f44b80d18e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44b607b700 1 -- 172.21.2.117:0/3187292941 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44a000c5e0 con 0x7f44b80d18e0 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 -- 172.21.2.117:0/3187292941 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80d18e0 msgr2=0x7f44b80ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80d18e0 0x7f44b80ea9e0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 rx=0x7f44a0006650 tx=0x7f44a0009a80).stop 2021-02-10T11:52:17.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 -- 172.21.2.117:0/3187292941 shutdown_connections 2021-02-10T11:52:17.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80eb000 0x7f44b80eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 --2- 172.21.2.117:0/3187292941 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80d18e0 0x7f44b80ea9e0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.583 7f44bd3de700 1 -- 172.21.2.117:0/3187292941 >> 172.21.2.117:0/3187292941 conn(0x7f44b80ced60 msgr2=0x7f44b80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:17.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.584 7f44bd3de700 1 -- 172.21.2.117:0/3187292941 shutdown_connections 2021-02-10T11:52:17.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.584 7f44bd3de700 1 -- 172.21.2.117:0/3187292941 wait complete. 2021-02-10T11:52:17.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.598 7f44bd3de700 1 Processor -- start 2021-02-10T11:52:17.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.598 7f44bd3de700 1 -- start start 2021-02-10T11:52:17.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44b80b77e0 0x7f44b80b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80b7310 0x7f44b80aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80ab8c0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80ac460 con 0x7f44b80b77e0 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44bd3de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80ac680 con 0x7f44b80b7310 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56262/0 (socket says 172.21.2.117:56262) 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 -- 172.21.2.117:0/649831752 learned_addr learned my addr 172.21.2.117:0/649831752 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 -- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80b7310 msgr2=0x7f44b80aaca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80b7310 0x7f44b80aaca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 -- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44b80b77e0 msgr2=0x7f44b80b6cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44b80b77e0 0x7f44b80b6cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:17.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 -- 172.21.2.117:0/649831752 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b80ac8a0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44b787e700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f44b0009980 tx=0x7f44b000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.599 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b000be10 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b0006030 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44bd3de700 1 -- 172.21.2.117:0/649831752 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b80ac8a0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44b00064c0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44bd3de700 1 -- 172.21.2.117:0/649831752 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b80ace30 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f44b0006210 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44affff700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 0x7f449c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.600 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f44b00328e0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.601 7f44b707d700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 0x7f449c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:17.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.603 7f44b707d700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 0x7f449c01abc0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:17.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.632 7f44bd3de700 1 -- 172.21.2.117:0/649831752 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b8087290 con 0x7f44b80ab2c0 2021-02-10T11:52:17.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.634 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44b0000f90 con 0x7f44b80ab2c0 2021-02-10T11:52:17.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.758 7f44bd3de700 1 -- 172.21.2.117:0/649831752 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44b80ede60 con 0x7f44b80ab2c0 2021-02-10T11:52:17.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.759 7f44affff700 1 -- 172.21.2.117:0/649831752 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f44b00060f0 con 0x7f44b80ab2c0 2021-02-10T11:52:17.761 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:17.761 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":134,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:03.821388","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:00.791090","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":29,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:01.865056","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:17.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 msgr2=0x7f449c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:17.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 0x7f449c01abc0 crc :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 msgr2=0x7f44b80abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:17.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 rx=0x7f44b0009980 tx=0x7f44b000bcd0).stop 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 shutdown_connections 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f449c0186c0 0x7f449c01abc0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44b80ab2c0 0x7f44b80abe40 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44b80b7310 0x7f44b80aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 --2- 172.21.2.117:0/649831752 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44b80b77e0 0x7f44b80b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 >> 172.21.2.117:0/649831752 conn(0x7f44b80874c0 msgr2=0x7f44b80af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 shutdown_connections 2021-02-10T11:52:17.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:17.761 7f44bd3de700 1 -- 172.21.2.117:0/649831752 wait complete. 2021-02-10T11:52:17.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:52:00.791090', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:52:01.865056', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '134', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:52:17.772 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.185 7f810e19a700 1 Processor -- start 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.185 7f810e19a700 1 -- start start 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.185 7f810e19a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f81081b07f0 0x7f81080e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.185 7f810e19a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81080e8d80 0x7f81080e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f810e19a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080e77f0 con 0x7f81080e6170 2021-02-10T11:52:18.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f810e19a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81080e54b0 con 0x7f81081b07f0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f810e19a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81080e9180 con 0x7f81080e8d80 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f81081b07f0 0x7f81080e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f81081b07f0 0x7f81080e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56270/0 (socket says 172.21.2.117:56270) 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 -- 172.21.2.117:0/2211005652 learned_addr learned my addr 172.21.2.117:0/2211005652 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 -- 172.21.2.117:0/2211005652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81080e8d80 msgr2=0x7f81080e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 --2- 172.21.2.117:0/2211005652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81080e8d80 0x7f81080e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 -- 172.21.2.117:0/2211005652 >> v1:172.21.2.117:6789/0 conn(0x7f81080e6170 legacy=0x7f81080e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 -- 172.21.2.117:0/2211005652 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81080e7380 con 0x7f81081b07f0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81077fe700 1 --2- 172.21.2.117:0/2211005652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081b07f0 0x7f81080e4e90 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 rx=0x7f80f8006700 tx=0x7f80f8009b80).ready entity=mon.0 client_cookie=46ee3953e833dda3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81067fc700 1 -- 172.21.2.117:0/2211005652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80f800c0c0 con 0x7f81081b07f0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81067fc700 1 -- 172.21.2.117:0/2211005652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f800c440 con 0x7f81081b07f0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.186 7f81067fc700 1 -- 172.21.2.117:0/2211005652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80f800c960 con 0x7f81081b07f0 2021-02-10T11:52:18.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 -- 172.21.2.117:0/2211005652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081b07f0 msgr2=0x7f81080e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 --2- 172.21.2.117:0/2211005652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081b07f0 0x7f81080e4e90 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 rx=0x7f80f8006700 tx=0x7f80f8009b80).stop 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 -- 172.21.2.117:0/2211005652 shutdown_connections 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 --2- 172.21.2.117:0/2211005652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f81080e8d80 0x7f81080e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 --2- 172.21.2.117:0/2211005652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081b07f0 0x7f81080e4e90 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 -- 172.21.2.117:0/2211005652 >> 172.21.2.117:0/2211005652 conn(0x7f81080ce500 msgr2=0x7f81080e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 -- 172.21.2.117:0/2211005652 shutdown_connections 2021-02-10T11:52:18.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.187 7f810e19a700 1 -- 172.21.2.117:0/2211005652 wait complete. 2021-02-10T11:52:18.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 Processor -- start 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 -- start start 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f810810aa40 0x7f810810ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f810810b460 0x7f810810b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080a2c40 con 0x7f81081a09d0 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080a2870 con 0x7f810810b460 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f810e19a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81080a2a50 con 0x7f810810aa40 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f81077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f81077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56280/0 (socket says 172.21.2.117:56280) 2021-02-10T11:52:18.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.202 7f81077fe700 1 -- 172.21.2.117:0/3217888915 learned_addr learned my addr 172.21.2.117:0/3217888915 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 -- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f810810aa40 msgr2=0x7f810810ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f810810aa40 0x7f810810ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 -- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f810810b460 msgr2=0x7f810810b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f810810b460 0x7f810810b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 -- 172.21.2.117:0/3217888915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81080a31c0 con 0x7f81081a09d0 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f81077fe700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f80f800b580 tx=0x7f80f800b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80f800bd10 con 0x7f81081a09d0 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8002030 con 0x7f81081a09d0 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f810e19a700 1 -- 172.21.2.117:0/3217888915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81080a31c0 con 0x7f81081a09d0 2021-02-10T11:52:18.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f810e19a700 1 -- 172.21.2.117:0/3217888915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81080a3750 con 0x7f81081a09d0 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.203 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80f8002480 con 0x7f81081a09d0 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.204 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80f8000c90 con 0x7f81081a09d0 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.204 7f80fffff700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 0x7f80ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.204 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f80f8032780 con 0x7f81081a09d0 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.204 7f8106ffd700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 0x7f80ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.205 7f8106ffd700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 0x7f80ec01abc0 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.236 7f810e19a700 1 -- 172.21.2.117:0/3217888915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81080e77f0 con 0x7f81081a09d0 2021-02-10T11:52:18.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.238 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80f80009b0 con 0x7f81081a09d0 2021-02-10T11:52:18.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.369 7f810e19a700 1 -- 172.21.2.117:0/3217888915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f81080a3180 con 0x7f81081a09d0 2021-02-10T11:52:18.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.369 7f80fffff700 1 -- 172.21.2.117:0/3217888915 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7f80f800bd10 con 0x7f81081a09d0 2021-02-10T11:52:18.372 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:18.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 msgr2=0x7f80ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 0x7f80ec01abc0 crc :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 msgr2=0x7f81080a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 rx=0x7f80f800b580 tx=0x7f80f800b920).stop 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 shutdown_connections 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f810810b460 0x7f810810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f810810aa40 0x7f810810ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80ec0186c0 0x7f80ec01abc0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 --2- 172.21.2.117:0/3217888915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f81081a09d0 0x7f81080a2250 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 >> 172.21.2.117:0/3217888915 conn(0x7f81080bb1c0 msgr2=0x7f810809b940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:18.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 shutdown_connections 2021-02-10T11:52:18.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.372 7f810e19a700 1 -- 172.21.2.117:0/3217888915 wait complete. 2021-02-10T11:52:18.382 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 Processor -- start 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 -- start start 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca80d18f0 0x7f2ca80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2ca80eb040 0x7f2ca80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca80ede60 con 0x7f2ca80ee3e0 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca80ebad0 con 0x7f2ca80d18f0 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2cacf3f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca80d1cf0 con 0x7f2ca80eb040 2021-02-10T11:52:18.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2ca6b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca80d18f0 0x7f2ca80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.797 7f2ca6b13700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca80d18f0 0x7f2ca80eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56284/0 (socket says 172.21.2.117:56284) 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 -- 172.21.2.117:0/3502098964 learned_addr learned my addr 172.21.2.117:0/3502098964 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 -- 172.21.2.117:0/3502098964 >> v2:172.21.2.125:3300/0 conn(0x7f2ca80eb040 msgr2=0x7f2ca80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 --2- 172.21.2.117:0/3502098964 >> v2:172.21.2.125:3300/0 conn(0x7f2ca80eb040 0x7f2ca80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 -- 172.21.2.117:0/3502098964 >> v1:172.21.2.125:6789/0 conn(0x7f2ca80ee3e0 legacy=0x7f2ca80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 -- 172.21.2.117:0/3502098964 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca80eefd0 con 0x7f2ca80d18f0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca6b13700 1 --2- 172.21.2.117:0/3502098964 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca80d18f0 0x7f2ca80eaa20 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f2c98006700 tx=0x7f2c98009b80).ready entity=mon.0 client_cookie=bb6d641ea7dd9b68 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca5b11700 1 -- 172.21.2.117:0/3502098964 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800c0c0 con 0x7f2ca80d18f0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca5b11700 1 -- 172.21.2.117:0/3502098964 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9800c440 con 0x7f2ca80d18f0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2ca5b11700 1 -- 172.21.2.117:0/3502098964 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800c8a0 con 0x7f2ca80d18f0 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2cacf3f700 1 -- 172.21.2.117:0/3502098964 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca80d18f0 msgr2=0x7f2ca80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.798 7f2cacf3f700 1 --2- 172.21.2.117:0/3502098964 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca80d18f0 0x7f2ca80eaa20 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 rx=0x7f2c98006700 tx=0x7f2c98009b80).stop 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 -- 172.21.2.117:0/3502098964 shutdown_connections 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 --2- 172.21.2.117:0/3502098964 >> v2:172.21.2.125:3300/0 conn(0x7f2ca80eb040 0x7f2ca80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 --2- 172.21.2.117:0/3502098964 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca80d18f0 0x7f2ca80eaa20 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 -- 172.21.2.117:0/3502098964 >> 172.21.2.117:0/3502098964 conn(0x7f2ca80cf060 msgr2=0x7f2ca80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 -- 172.21.2.117:0/3502098964 shutdown_connections 2021-02-10T11:52:18.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.799 7f2cacf3f700 1 -- 172.21.2.117:0/3502098964 wait complete. 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.813 7f2cacf3f700 1 Processor -- start 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 -- start start 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca81a3930 0x7f2ca80a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca80a13a0 0x7f2ca80a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca807cca0 con 0x7f2ca807c7c0 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca818fb70 con 0x7f2ca81a3930 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2cacf3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca80dc6d0 con 0x7f2ca80a13a0 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2ca6b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2ca6b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56294/0 (socket says 172.21.2.117:56294) 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.814 7f2ca6b13700 1 -- 172.21.2.117:0/3833894200 learned_addr learned my addr 172.21.2.117:0/3833894200 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 -- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca80a13a0 msgr2=0x7f2ca80a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca80a13a0 0x7f2ca80a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 -- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca81a3930 msgr2=0x7f2ca80a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca81a3930 0x7f2ca80a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 -- 172.21.2.117:0/3833894200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca807d220 con 0x7f2ca807c7c0 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2ca6b13700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f2c9800b5e0 tx=0x7f2c9800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800bd70 con 0x7f2ca807c7c0 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98002030 con 0x7f2ca807c7c0 2021-02-10T11:52:18.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c980024d0 con 0x7f2ca807c7c0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca807d220 con 0x7f2ca807c7c0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.815 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca807d220 con 0x7f2ca807c7c0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.816 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c98000c90 con 0x7f2ca807c7c0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.816 7f2c9f7fe700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 0x7f2c8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.816 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2c98032760 con 0x7f2ca807c7c0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.816 7f2ca6312700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 0x7f2c8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:18.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.817 7f2ca6312700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 0x7f2c8c01abc0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:18.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.847 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca80daff0 con 0x7f2ca807c7c0 2021-02-10T11:52:18.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.850 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c9801f4d0 con 0x7f2ca807c7c0 2021-02-10T11:52:18.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.975 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2ca80daff0 con 0x7f2ca807c7c0 2021-02-10T11:52:18.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.976 7f2c9f7fe700 1 -- 172.21.2.117:0/3833894200 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v134) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2c98000c90 con 0x7f2ca807c7c0 2021-02-10T11:52:18.977 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:18.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 msgr2=0x7f2c8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 0x7f2c8c01abc0 crc :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 msgr2=0x7f2ca81a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 rx=0x7f2c9800b5e0 tx=0x7f2c9800b980).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 shutdown_connections 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca80a13a0 0x7f2ca80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca81a3930 0x7f2ca80a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c8c018700 0x7f2c8c01abc0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 --2- 172.21.2.117:0/3833894200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca807c7c0 0x7f2ca81a3310 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 >> 172.21.2.117:0/3833894200 conn(0x7f2ca8087090 msgr2=0x7f2ca808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 shutdown_connections 2021-02-10T11:52:18.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:18.978 7f2cacf3f700 1 -- 172.21.2.117:0/3833894200 wait complete. 2021-02-10T11:52:18.987 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 Processor -- start 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 -- start start 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90480d18f0 0x7f90480eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480eb040 0x7f90480eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90480eba60 con 0x7f90480ee3e0 2021-02-10T11:52:19.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90480d1cf0 con 0x7f90480d18f0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f904d8cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90480eefd0 con 0x7f90480eb040 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.401 7f9047574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90480d18f0 0x7f90480eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90480d18f0 0x7f90480eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56306/0 (socket says 172.21.2.117:56306) 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 -- 172.21.2.117:0/3080872579 learned_addr learned my addr 172.21.2.117:0/3080872579 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9046d73700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480eb040 0x7f90480eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 -- 172.21.2.117:0/3080872579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480eb040 msgr2=0x7f90480eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480eb040 0x7f90480eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 -- 172.21.2.117:0/3080872579 >> v1:172.21.2.125:6789/0 conn(0x7f90480ee3e0 legacy=0x7f90480ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 -- 172.21.2.117:0/3080872579 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480ef1f0 con 0x7f90480d18f0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9047574700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90480d18f0 0x7f90480eaa20 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f9030006700 tx=0x7f9030009b80).ready entity=mon.0 client_cookie=f0edfc296cd7e9c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9046572700 1 -- 172.21.2.117:0/3080872579 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f903000c0c0 con 0x7f90480d18f0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9046572700 1 -- 172.21.2.117:0/3080872579 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903000c3c0 con 0x7f90480d18f0 2021-02-10T11:52:19.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f9046572700 1 -- 172.21.2.117:0/3080872579 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f903000c8e0 con 0x7f90480d18f0 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f904d8cc700 1 -- 172.21.2.117:0/3080872579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90480d18f0 msgr2=0x7f90480eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.402 7f904d8cc700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90480d18f0 0x7f90480eaa20 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 rx=0x7f9030006700 tx=0x7f9030009b80).stop 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 -- 172.21.2.117:0/3080872579 shutdown_connections 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480eb040 0x7f90480eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 --2- 172.21.2.117:0/3080872579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90480d18f0 0x7f90480eaa20 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 -- 172.21.2.117:0/3080872579 >> 172.21.2.117:0/3080872579 conn(0x7f90480cf060 msgr2=0x7f90480cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 -- 172.21.2.117:0/3080872579 shutdown_connections 2021-02-10T11:52:19.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.403 7f904d8cc700 1 -- 172.21.2.117:0/3080872579 wait complete. 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 Processor -- start 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 -- start start 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 0x7f90481a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480a13a0 0x7f90480a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904807cca0 con 0x7f90481a3930 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904818fb70 con 0x7f904807c7c0 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f904d8cc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90480dc6d0 con 0x7f90480a13a0 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f9046d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f9046d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56314/0 (socket says 172.21.2.117:56314) 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f9046d73700 1 -- 172.21.2.117:0/302977176 learned_addr learned my addr 172.21.2.117:0/302977176 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f9047574700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 0x7f90481a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.418 7f9047d75700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480a13a0 0x7f90480a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 -- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480a13a0 msgr2=0x7f90480a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480a13a0 0x7f90480a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 -- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 msgr2=0x7f90481a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 0x7f90481a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 -- 172.21.2.117:0/302977176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904807d220 con 0x7f90481a3930 2021-02-10T11:52:19.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9047574700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 0x7f90481a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f9046d73700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7f9038006f60 tx=0x7f903800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f903800bd60 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9038000c90 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f904d8cc700 1 -- 172.21.2.117:0/302977176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904807d220 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f904d8cc700 1 -- 172.21.2.117:0/302977176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904807d7b0 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.419 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90380022b0 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.420 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9038006b00 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.420 7f903ffff700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 0x7f902c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.420 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9038032670 con 0x7f90481a3930 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.420 7f9047574700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 0x7f902c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:19.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.420 7f9047574700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 0x7f902c01abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:19.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.451 7f904d8cc700 1 -- 172.21.2.117:0/302977176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90480daff0 con 0x7f90481a3930 2021-02-10T11:52:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.454 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f90380307b0 con 0x7f90481a3930 2021-02-10T11:52:19.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.582 7f904d8cc700 1 -- 172.21.2.117:0/302977176 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f90480ede60 con 0x7f90481a3930 2021-02-10T11:52:19.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.582 7f903ffff700 1 -- 172.21.2.117:0/302977176 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v134) v1 ==== 100+0+10 (secure 0 0 0) 0x7f90380307b0 con 0x7f90481a3930 2021-02-10T11:52:19.583 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 msgr2=0x7f902c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 0x7f902c01abc0 crc :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 msgr2=0x7f90480a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 rx=0x7f9038006f60 tx=0x7f903800b970).stop 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 shutdown_connections 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f902c0186c0 0x7f902c01abc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90481a3930 0x7f90480a0d80 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90480a13a0 0x7f90480a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 --2- 172.21.2.117:0/302977176 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f904807c7c0 0x7f90481a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:19.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 >> 172.21.2.117:0/302977176 conn(0x7f9048087090 msgr2=0x7f904808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:19.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 shutdown_connections 2021-02-10T11:52:19.586 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:19.584 7f904d8cc700 1 -- 172.21.2.117:0/302977176 wait complete. 2021-02-10T11:52:19.593 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:52:19.593 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:20.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.003 7f9019331700 1 Processor -- start 2021-02-10T11:52:20.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.003 7f9019331700 1 -- start start 2021-02-10T11:52:20.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f9019331700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90140e5240 0x7f9014086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f9019331700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9014084c60 0x7f9014085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f9019331700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9014085930 con 0x7f90140e4e40 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f9019331700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140e5cc0 con 0x7f90140e5240 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f9019331700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140856b0 con 0x7f9014084c60 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f90140e5240 0x7f9014086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901307d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f90140e4e40 0x7f90141b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56568/0 (socket says 172.21.2.117:56568) 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901307d700 1 -- 172.21.2.117:0/2037035771 learned_addr learned my addr 172.21.2.117:0/2037035771 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901287c700 1 -- 172.21.2.117:0/2037035771 >> v2:172.21.2.125:3300/0 conn(0x7f9014084c60 msgr2=0x7f9014085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901287c700 1 --2- 172.21.2.117:0/2037035771 >> v2:172.21.2.125:3300/0 conn(0x7f9014084c60 0x7f9014085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901287c700 1 -- 172.21.2.117:0/2037035771 >> v1:172.21.2.117:6789/0 conn(0x7f90140e4e40 legacy=0x7f90141b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.004 7f901287c700 1 -- 172.21.2.117:0/2037035771 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90141b09e0 con 0x7f90140e5240 2021-02-10T11:52:20.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f901287c700 1 --2- 172.21.2.117:0/2037035771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90140e5240 0x7f9014086ec0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7f9008006650 tx=0x7f9008009a80).ready entity=mon.0 client_cookie=1aac8e8b02d2b231 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f901207b700 1 -- 172.21.2.117:0/2037035771 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f90080009b0 con 0x7f90140e5240 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f901207b700 1 -- 172.21.2.117:0/2037035771 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900800c0c0 con 0x7f90140e5240 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f901207b700 1 -- 172.21.2.117:0/2037035771 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f900800c5e0 con 0x7f90140e5240 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 -- 172.21.2.117:0/2037035771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90140e5240 msgr2=0x7f9014086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 --2- 172.21.2.117:0/2037035771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90140e5240 0x7f9014086ec0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 rx=0x7f9008006650 tx=0x7f9008009a80).stop 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 -- 172.21.2.117:0/2037035771 shutdown_connections 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 --2- 172.21.2.117:0/2037035771 >> v2:172.21.2.125:3300/0 conn(0x7f9014084c60 0x7f9014085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 --2- 172.21.2.117:0/2037035771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90140e5240 0x7f9014086ec0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 -- 172.21.2.117:0/2037035771 >> 172.21.2.117:0/2037035771 conn(0x7f90140ce4e0 msgr2=0x7f90140e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.005 7f9019331700 1 -- 172.21.2.117:0/2037035771 shutdown_connections 2021-02-10T11:52:20.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.006 7f9019331700 1 -- 172.21.2.117:0/2037035771 wait complete. 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 Processor -- start 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 -- start start 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 0x7f901410c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 0x7f9014098010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.020 7f9019331700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141b2610 con 0x7f90141897b0 2021-02-10T11:52:20.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f9019331700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90141b2830 con 0x7f901410caa0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f9019331700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140986a0 con 0x7f90141893b0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56328/0 (socket says 172.21.2.117:56328) 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 -- 172.21.2.117:0/1626943015 learned_addr learned my addr 172.21.2.117:0/1626943015 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 -- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 msgr2=0x7f901410c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901307d700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 0x7f901410c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901287c700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 0x7f9014098010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 0x7f901410c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 -- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 msgr2=0x7f9014098010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 0x7f9014098010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901387e700 1 -- 172.21.2.117:0/1626943015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140988c0 con 0x7f90141897b0 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.021 7f901307d700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 0x7f901410c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:20.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f901287c700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 0x7f9014098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f901387e700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f900c009980 tx=0x7f900c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f900c003020 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c005a80 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f900c006030 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f9019331700 1 -- 172.21.2.117:0/1626943015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90140988c0 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f9019331700 1 -- 172.21.2.117:0/1626943015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90140988c0 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f900c006030 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 0x7f8ff401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.022 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f900c032c90 con 0x7f90141897b0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.023 7f901307d700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 0x7f8ff401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:20.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.023 7f901307d700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 0x7f8ff401abd0 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:20.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.054 7f9019331700 1 -- 172.21.2.117:0/1626943015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901410c250 con 0x7f90141897b0 2021-02-10T11:52:20.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.056 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f900c005870 con 0x7f90141897b0 2021-02-10T11:52:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.184 7f9019331700 1 -- 172.21.2.117:0/1626943015 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f901410c250 con 0x7f90141897b0 2021-02-10T11:52:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.184 7f8ffbfff700 1 -- 172.21.2.117:0/1626943015 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f900c005a80 con 0x7f90141897b0 2021-02-10T11:52:20.186 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 111 2021-02-10T11:52:20.186 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:20.186 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":111,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:04.828714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7286":{"gid":7286,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/604396797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":604396797},{"type":"v1","addr":"172.21.2.125:6835","nonce":604396797}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 msgr2=0x7f8ff401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 0x7f8ff401abd0 crc :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 msgr2=0x7f9014187fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f900c009980 tx=0x7f900c00bcd0).stop 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 shutdown_connections 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ff40186d0 0x7f8ff401abd0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f90141897b0 0x7f9014187fe0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f901410caa0 0x7f9014098010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 --2- 172.21.2.117:0/1626943015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90141893b0 0x7f901410c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 >> 172.21.2.117:0/1626943015 conn(0x7f9014199160 msgr2=0x7f901408d310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 shutdown_connections 2021-02-10T11:52:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:20.186 7f9019331700 1 -- 172.21.2.117:0/1626943015 wait complete. 2021-02-10T11:52:20.398 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:02.818807', 'modified': '2021-02-10 11:52:04.828714', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7286}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7286': {'gid': 7286, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/604396797', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 604396797}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 604396797}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:52:20.398 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:52:20.398 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7286, 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 17.972729365999999, 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 111, 2021-02-10T11:52:20.795 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 134, 2021-02-10T11:52:20.796 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 134, 2021-02-10T11:52:20.796 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.60480424 2021-02-10T11:52:20.796 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:52:20.799 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7286, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.972729365999999, "mdsmap_epoch": 111, "osdmap_epoch": 134, "osdmap_epoch_barrier": 134, "uptime": 23.60480424 } 2021-02-10T11:52:20.799 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:52:20.799 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:52:20.799 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:52:20.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:20.799 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:21.136 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:52:21.481 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:52:21.481 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:52:21.824 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:52:21.824 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:52:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:52:21.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:21.825 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:52:22.166 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:52:22.166 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:52:22.166 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:22.169 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:52:22.513 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:52:22.513 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:52:22.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:52:22.579 7fdecfe67f80 -1 init, newargv = 0x55bbe1cbaf30 newargc=9 2021-02-10T11:52:22.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[170379]: starting ceph client 2021-02-10T11:52:22.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[170379]: starting fuse 2021-02-10T11:52:22.856 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:52:22.856 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:52:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:52:22.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:22.857 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:52:23.192 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:52:23.193 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:52:23.193 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:52:23.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:23.193 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:52:23.193 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:52:23.193 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:52:23.193 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:52:23.193 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:52:23.194 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:52:23.195 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:52:23.195 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:52:23.195 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:52:23.195 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:52:23.558 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.170379.asok 2021-02-10T11:52:23.560 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.170379.asok 2021-02-10T11:52:23.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:23.561 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.170379.asok status 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:52:23.951 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "170379", 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11640, 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11640 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:23.952 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3570959595 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3570959595 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11640 172.21.2.117:0/3570959595", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3570959595", 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:52:23.953 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 111, 2021-02-10T11:52:23.954 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 134, 2021-02-10T11:52:23.954 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:52:23.954 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:52:23.954 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:52:23.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:23.955 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:23.955 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:24.292 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:52:24.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:52:24.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:24.292 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:24.640 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 Processor -- start 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 -- start start 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fada80e5240 0x7fada8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8085930 con 0x7fada8084c60 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada81b09e0 con 0x7fada80e4e40 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadae90f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80e5cc0 con 0x7fada80e5240 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadacf2c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fada8084c60 0x7fada8085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56600/0 (socket says 172.21.2.117:56600) 2021-02-10T11:52:25.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.056 7fadacf2c700 1 -- 172.21.2.117:0/2193979628 learned_addr learned my addr 172.21.2.117:0/2193979628 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381712827 0 0) 0x7fada0001500 con 0x7fada8084c60 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad94002fe0 con 0x7fada8084c60 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fad9ffff700 1 --2- 172.21.2.117:0/2193979628 >> v2:172.21.2.125:3300/0 conn(0x7fada80e5240 0x7fada8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1139603728 0 0) 0x7fada0001500 con 0x7fada8084c60 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 >> v2:172.21.2.125:3300/0 conn(0x7fada80e5240 msgr2=0x7fada8086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 --2- 172.21.2.117:0/2193979628 >> v2:172.21.2.125:3300/0 conn(0x7fada80e5240 0x7fada8086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 >> v1:172.21.2.125:6789/0 conn(0x7fada80e4e40 legacy=0x7fada81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8084840 con 0x7fada8084c60 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fada0001a00 con 0x7fada8084c60 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 >> v1:172.21.2.117:6789/0 conn(0x7fada8084c60 legacy=0x7fada8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad94003ae0 0x7fad94006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad94006820 0x7fad94008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 0x7fad9400bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9400c310 con 0x7fad94009640 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9400c530 con 0x7fad94003ae0 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fada77fe700 1 -- 172.21.2.117:0/2193979628 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9400c750 con 0x7fad94006820 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fadacf2c700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 0x7fad9400bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.057 7fad9ffff700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad94006820 0x7fad94008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 -- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad94006820 msgr2=0x7fad94008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad94006820 0x7fad94008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 -- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad94003ae0 msgr2=0x7fad94006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad94003ae0 0x7fad94006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 -- 172.21.2.117:0/2193979628 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad940509c0 con 0x7fad94009640 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 -- 172.21.2.117:0/2193979628 reap_dead start 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadacf2c700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 0x7fad9400bb40 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fada0007c60 tx=0x7fada000a060).ready entity=mon.0 client_cookie=2dc804d9964d65a1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada00070c0 con 0x7fad94009640 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada0007340 con 0x7fad94009640 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fada77fe700 1 -- 172.21.2.117:0/2193979628 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada00049a0 con 0x7fad94009640 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 -- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 msgr2=0x7fad9400bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 0x7fad9400bb40 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 rx=0x7fada0007c60 tx=0x7fada000a060).stop 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 -- 172.21.2.117:0/2193979628 shutdown_connections 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 --2- 172.21.2.117:0/2193979628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad94009640 0x7fad9400bb40 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 -- 172.21.2.117:0/2193979628 >> 172.21.2.117:0/2193979628 conn(0x7fada80ce4e0 msgr2=0x7fada80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 -- 172.21.2.117:0/2193979628 shutdown_connections 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.058 7fadae90f700 1 -- 172.21.2.117:0/2193979628 wait complete. 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 Processor -- start 2021-02-10T11:52:25.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 -- start start 2021-02-10T11:52:25.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fada80be700 0x7fada80bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fada80bfa00 0x7fada80bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80bee60 con 0x7fada80c0420 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80beb00 con 0x7fada80bfa00 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.073 7fadae90f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada80912f0 con 0x7fada80be700 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56364/0 (socket says 172.21.2.117:56364) 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 -- 172.21.2.117:0/3757746343 learned_addr learned my addr 172.21.2.117:0/3757746343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 -- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fada80be700 msgr2=0x7fada80bf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fada80be700 0x7fada80bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 -- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fada80bfa00 msgr2=0x7fada80bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fada80bfa00 0x7fada80bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 -- 172.21.2.117:0/3757746343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada80a9bd0 con 0x7fada80c0420 2021-02-10T11:52:25.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.074 7fadacf2c700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7fada0007f10 tx=0x7fada000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada0007860 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada0004930 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fada0007020 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fadae90f700 1 -- 172.21.2.117:0/3757746343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada80a9bd0 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fadae90f700 1 -- 172.21.2.117:0/3757746343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada80aa160 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fada0007860 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 0x7fad8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.075 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fada0032a60 con 0x7fada80c0420 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.076 7fada7fff700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 0x7fad8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.076 7fada7fff700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 0x7fad8c01abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.106 7fadae90f700 1 -- 172.21.2.117:0/3757746343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada810c2f0 con 0x7fada80c0420 2021-02-10T11:52:25.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.108 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fada0004bb0 con 0x7fada80c0420 2021-02-10T11:52:25.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.243 7fadae90f700 1 -- 172.21.2.117:0/3757746343 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fada8085930 con 0x7fada80c0420 2021-02-10T11:52:25.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.297 7fada57fa700 1 -- 172.21.2.117:0/3757746343 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v112) v1 ==== 136+0+0 (secure 0 0 0) 0x7fada0007860 con 0x7fada80c0420 2021-02-10T11:52:25.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.309 7fadae90f700 1 -- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 msgr2=0x7fad8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.309 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 0x7fad8c01abc0 crc :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.309 7fadae90f700 1 -- 172.21.2.117:0/3757746343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 msgr2=0x7fada80a95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.309 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 rx=0x7fada0007f10 tx=0x7fada000ac80).stop 2021-02-10T11:52:25.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 -- 172.21.2.117:0/3757746343 shutdown_connections 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad8c0186c0 0x7fad8c01abc0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fada80c0420 0x7fada80a95b0 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fada80bfa00 0x7fada80bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 --2- 172.21.2.117:0/3757746343 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fada80be700 0x7fada80bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 -- 172.21.2.117:0/3757746343 >> 172.21.2.117:0/3757746343 conn(0x7fada80bd990 msgr2=0x7fada80c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 -- 172.21.2.117:0/3757746343 shutdown_connections 2021-02-10T11:52:25.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.310 7fadae90f700 1 -- 172.21.2.117:0/3757746343 wait complete. 2021-02-10T11:52:25.319 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.730 7ffb1ae10700 1 Processor -- start 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.730 7ffb1ae10700 1 -- start start 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.731 7ffb1ae10700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffb140e7180 0x7ffb140e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.731 7ffb1ae10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 0x7ffb140e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.731 7ffb1ae10700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140e7580 con 0x7ffb141b07f0 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.731 7ffb1ae10700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb140e5ce0 con 0x7ffb140e7180 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.731 7ffb1ae10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140e77f0 con 0x7ffb140e8390 2021-02-10T11:52:25.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb1942d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 0x7ffb140e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffb140e7180 0x7ffb140e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffb140e7180 0x7ffb140e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60752/0 (socket says 172.21.2.117:60752) 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb1942d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 0x7ffb140e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54052/0 (socket says 172.21.2.117:54052) 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 -- 172.21.2.117:0/3197393254 learned_addr learned my addr 172.21.2.117:0/3197393254 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 -- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 msgr2=0x7ffb140e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 --2- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 0x7ffb140e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 -- 172.21.2.117:0/3197393254 >> v1:172.21.2.125:6789/0 conn(0x7ffb141b07f0 legacy=0x7ffb140e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.732 7ffb13fff700 1 -- 172.21.2.117:0/3197393254 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140e8db0 con 0x7ffb140e7180 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb13fff700 1 --2- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140e7180 0x7ffb140e7d70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7ffb00006700 tx=0x7ffb00009b80).ready entity=mon.1 client_cookie=418a4db137bd51fd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb137fe700 1 -- 172.21.2.117:0/3197393254 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0000c0c0 con 0x7ffb140e7180 2021-02-10T11:52:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb137fe700 1 -- 172.21.2.117:0/3197393254 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0000c340 con 0x7ffb140e7180 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb137fe700 1 -- 172.21.2.117:0/3197393254 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0000c860 con 0x7ffb140e7180 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb1ae10700 1 -- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140e7180 msgr2=0x7ffb140e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.733 7ffb1ae10700 1 --2- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140e7180 0x7ffb140e7d70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7ffb00006700 tx=0x7ffb00009b80).stop 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 -- 172.21.2.117:0/3197393254 shutdown_connections 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 --2- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb140e8390 0x7ffb140e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 --2- 172.21.2.117:0/3197393254 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140e7180 0x7ffb140e7d70 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 -- 172.21.2.117:0/3197393254 >> 172.21.2.117:0/3197393254 conn(0x7ffb140ce500 msgr2=0x7ffb140e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 -- 172.21.2.117:0/3197393254 shutdown_connections 2021-02-10T11:52:25.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.734 7ffb1ae10700 1 -- 172.21.2.117:0/3197393254 wait complete. 2021-02-10T11:52:25.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.748 7ffb1ae10700 1 Processor -- start 2021-02-10T11:52:25.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.748 7ffb1ae10700 1 -- start start 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140af5f0 0x7ffb1408bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb1408c380 0x7ffb14199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1408cca0 con 0x7ffb1408c780 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14199c90 con 0x7ffb140af5f0 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1ae10700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14199e70 con 0x7ffb1408c380 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56374/0 (socket says 172.21.2.117:56374) 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 -- 172.21.2.117:0/3877564866 learned_addr learned my addr 172.21.2.117:0/3877564866 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb18c2c700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140af5f0 0x7ffb1408bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb13fff700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb1408c380 0x7ffb14199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 -- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb1408c380 msgr2=0x7ffb14199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb1408c380 0x7ffb14199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 -- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140af5f0 msgr2=0x7ffb1408bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140af5f0 0x7ffb1408bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.749 7ffb1942d700 1 -- 172.21.2.117:0/3877564866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1409d790 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb1942d700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7ffb0c00b970 tx=0x7ffb0c00bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0c000f90 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c003030 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffb0c0015f0 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1409d790 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.750 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1409dd20 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.751 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffb0c0011f0 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.751 7ffb117fa700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 0x7ffaf001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.751 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffb0c032a60 con 0x7ffb1408c780 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.751 7ffb18c2c700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 0x7ffaf001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:25.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.752 7ffb18c2c700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 0x7ffaf001abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:25.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.781 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14106e10 con 0x7ffb1408c780 2021-02-10T11:52:25.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.783 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffb0c000f40 con 0x7ffb1408c780 2021-02-10T11:52:25.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.917 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7ffb1408cca0 con 0x7ffaf00186c0 2021-02-10T11:52:25.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.919 7ffb117fa700 1 -- 172.21.2.117:0/3877564866 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ffb04006a20 con 0x7ffaf00186c0 2021-02-10T11:52:25.921 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:52:25.921 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:25.921 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:52:25.922 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:25.922 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 msgr2=0x7ffaf001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 0x7ffaf001abc0 crc :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 msgr2=0x7ffb14199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 rx=0x7ffb0c00b970 tx=0x7ffb0c00bd10).stop 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 shutdown_connections 2021-02-10T11:52:25.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffaf00186c0 0x7ffaf001abc0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffb1408c780 0x7ffb14199670 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffb1408c380 0x7ffb14199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 --2- 172.21.2.117:0/3877564866 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffb140af5f0 0x7ffb1408bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 >> 172.21.2.117:0/3877564866 conn(0x7ffb140953a0 msgr2=0x7ffb1409e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 shutdown_connections 2021-02-10T11:52:25.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:25.922 7ffb1ae10700 1 -- 172.21.2.117:0/3877564866 wait complete. 2021-02-10T11:52:25.931 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:52:26.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.346 7f04504db700 1 Processor -- start 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.346 7f04504db700 1 -- start start 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f04504db700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f04480d18f0 0x7f04480eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f04504db700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480ef7f0 con 0x7f04480ef3f0 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f04504db700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480ede60 con 0x7f04480eb040 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f04504db700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04480d1da0 con 0x7f04480d18f0 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f044e2f7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f04480d18f0 0x7f04480eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f044eaf8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f04480ef3f0 0x7f04481b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56630/0 (socket says 172.21.2.117:56630) 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.347 7f044eaf8700 1 -- 172.21.2.117:0/2833465600 learned_addr learned my addr 172.21.2.117:0/2833465600 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044e2f7700 1 -- 172.21.2.117:0/2833465600 >> v1:172.21.2.125:6789/0 conn(0x7f04480eb040 legacy=0x7f04480eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:26.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044e2f7700 1 -- 172.21.2.117:0/2833465600 >> v1:172.21.2.117:6789/0 conn(0x7f04480ef3f0 legacy=0x7f04481b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044d2f5700 1 -- 172.21.2.117:0/2833465600 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462578229 0 0) 0x7f0440001500 con 0x7f04480ef3f0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044e2f7700 1 -- 172.21.2.117:0/2833465600 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04481bb1c0 con 0x7f04480d18f0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044e2f7700 1 --2- 172.21.2.117:0/2833465600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04480d18f0 0x7f04480eaa20 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f0444006650 tx=0x7f0444009a00).ready entity=mon.0 client_cookie=4994909d4af37674 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044d2f5700 1 -- 172.21.2.117:0/2833465600 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04440009b0 con 0x7f04480d18f0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044d2f5700 1 -- 172.21.2.117:0/2833465600 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044400c0c0 con 0x7f04480d18f0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f044d2f5700 1 -- 172.21.2.117:0/2833465600 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f044400c5a0 con 0x7f04480d18f0 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f04504db700 1 -- 172.21.2.117:0/2833465600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04480d18f0 msgr2=0x7f04480eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.348 7f04504db700 1 --2- 172.21.2.117:0/2833465600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04480d18f0 0x7f04480eaa20 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f0444006650 tx=0x7f0444009a00).stop 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.349 7f04504db700 1 -- 172.21.2.117:0/2833465600 shutdown_connections 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.349 7f04504db700 1 --2- 172.21.2.117:0/2833465600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04480d18f0 0x7f04480eaa20 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.349 7f04504db700 1 -- 172.21.2.117:0/2833465600 >> 172.21.2.117:0/2833465600 conn(0x7f04480cf060 msgr2=0x7f04480cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.349 7f04504db700 1 -- 172.21.2.117:0/2833465600 shutdown_connections 2021-02-10T11:52:26.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.349 7f04504db700 1 -- 172.21.2.117:0/2833465600 wait complete. 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 Processor -- start 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 -- start start 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04481a4740 0x7f044809bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f044818f950 0x7f044818fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044809c190 con 0x7f0448190370 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044809c3b0 con 0x7f044818f950 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.364 7f04504db700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0448085350 con 0x7f04481a4740 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56384/0 (socket says 172.21.2.117:56384) 2021-02-10T11:52:26.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 -- 172.21.2.117:0/2850367694 learned_addr learned my addr 172.21.2.117:0/2850367694 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044e2f7700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04481a4740 0x7f044809bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044daf6700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f044818f950 0x7f044818fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 -- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04481a4740 msgr2=0x7f044809bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04481a4740 0x7f044809bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 -- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f044818f950 msgr2=0x7f044818fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f044818f950 0x7f044818fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 -- 172.21.2.117:0/2850367694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448085570 con 0x7f0448190370 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f044eaf8700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f0440001c10 tx=0x7f044000ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0440000f90 con 0x7f0448190370 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0440007020 con 0x7f0448190370 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04400074c0 con 0x7f0448190370 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.365 7f04504db700 1 -- 172.21.2.117:0/2850367694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0448085570 con 0x7f0448190370 2021-02-10T11:52:26.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.366 7f04504db700 1 -- 172.21.2.117:0/2850367694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0448085570 con 0x7f0448190370 2021-02-10T11:52:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.366 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0440007200 con 0x7f0448190370 2021-02-10T11:52:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.366 7f043effd700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 0x7f043001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.366 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0440032c90 con 0x7f0448190370 2021-02-10T11:52:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.366 7f044e2f7700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 0x7f043001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.367 7f044e2f7700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 0x7f043001abd0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:26.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.397 7f04504db700 1 -- 172.21.2.117:0/2850367694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04480f67e0 con 0x7f0448190370 2021-02-10T11:52:26.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.399 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0440000cc0 con 0x7f0448190370 2021-02-10T11:52:26.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.548 7f04504db700 1 -- 172.21.2.117:0/2850367694 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f04480f67e0 con 0x7f0448190370 2021-02-10T11:52:26.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.549 7f043effd700 1 -- 172.21.2.117:0/2850367694 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0440007020 con 0x7f0448190370 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 msgr2=0x7f043001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 0x7f043001abd0 crc :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 msgr2=0x7f0448084d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 rx=0x7f0440001c10 tx=0x7f044000ac60).stop 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 shutdown_connections 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04300186d0 0x7f043001abd0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0448190370 0x7f0448084d30 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f044818f950 0x7f044818fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 --2- 172.21.2.117:0/2850367694 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04481a4740 0x7f044809bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 >> 172.21.2.117:0/2850367694 conn(0x7f0448087250 msgr2=0x7f044808e490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:26.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 shutdown_connections 2021-02-10T11:52:26.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.551 7f04504db700 1 -- 172.21.2.117:0/2850367694 wait complete. 2021-02-10T11:52:26.560 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000968030 --group_name non_existent_group 2021-02-10T11:52:26.994 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.975 7f0b84345700 1 Processor -- start 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.975 7f0b84345700 1 -- start start 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b84345700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0b7c0e5240 0x7f0b7c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b84345700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c085930 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b84345700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c1b09e0 con 0x7f0b7c084c60 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b84345700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0e5cc0 con 0x7f0b7c0e5240 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b82161700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0b7c0e4e40 0x7f0b7c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56642/0 (socket says 172.21.2.117:56642) 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.976 7f0b82161700 1 -- 172.21.2.117:0/3707345698 learned_addr learned my addr 172.21.2.117:0/3707345698 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809370354 0 0) 0x7f0b70001180 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b68002fe0 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b81960700 1 --2- 172.21.2.117:0/3707345698 >> v2:172.21.2.125:3300/0 conn(0x7f0b7c0e5240 0x7f0b7c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2567604183 0 0) 0x7f0b70001180 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 >> v2:172.21.2.125:3300/0 conn(0x7f0b7c0e5240 msgr2=0x7f0b7c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 --2- 172.21.2.117:0/3707345698 >> v2:172.21.2.125:3300/0 conn(0x7f0b7c0e5240 0x7f0b7c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 >> v1:172.21.2.125:6789/0 conn(0x7f0b7c084c60 legacy=0x7f0b7c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c084840 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0b70001680 con 0x7f0b7c0e4e40 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 >> v1:172.21.2.117:6789/0 conn(0x7f0b7c0e4e40 legacy=0x7f0b7c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b68003af0 0x7f0b68005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b68006780 0x7f0b68008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 0x7f0b6800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6800c2f0 con 0x7f0b680095a0 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6800c510 con 0x7f0b68006780 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6800c730 con 0x7f0b68003af0 2021-02-10T11:52:26.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b82962700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 0x7f0b6800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b81960700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b68006780 0x7f0b68008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.977 7f0b82161700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b68003af0 0x7f0b68005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 -- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b68003af0 msgr2=0x7f0b68005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b68003af0 0x7f0b68005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 -- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b68006780 msgr2=0x7f0b68008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b68006780 0x7f0b68008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 -- 172.21.2.117:0/3707345698 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680509a0 con 0x7f0b680095a0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 -- 172.21.2.117:0/3707345698 reap_dead start 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b82962700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 0x7f0b6800baa0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f0b74006e40 tx=0x7f0b74009e30).ready entity=mon.0 client_cookie=496cb5d1da78569e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b74006a60 con 0x7f0b680095a0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7400b300 con 0x7f0b680095a0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b8115f700 1 -- 172.21.2.117:0/3707345698 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b7400b7e0 con 0x7f0b680095a0 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b84345700 1 -- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 msgr2=0x7f0b6800baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:26.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.978 7f0b84345700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 0x7f0b6800baa0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 rx=0x7f0b74006e40 tx=0x7f0b74009e30).stop 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.979 7f0b84345700 1 -- 172.21.2.117:0/3707345698 shutdown_connections 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.979 7f0b84345700 1 --2- 172.21.2.117:0/3707345698 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b680095a0 0x7f0b6800baa0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.979 7f0b84345700 1 -- 172.21.2.117:0/3707345698 >> 172.21.2.117:0/3707345698 conn(0x7f0b7c0ce4e0 msgr2=0x7f0b7c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.979 7f0b84345700 1 -- 172.21.2.117:0/3707345698 shutdown_connections 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.979 7f0b84345700 1 -- 172.21.2.117:0/3707345698 wait complete. 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.993 7f0b84345700 1 Processor -- start 2021-02-10T11:52:26.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 -- start start 2021-02-10T11:52:26.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0b24f0 0x7f0b7c0c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c0a36b0 0x7f0b7c0a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0c2cf0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a06d0 con 0x7f0b7c0b24f0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b84345700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a40d0 con 0x7f0b7c0a36b0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b81960700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b81960700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56408/0 (socket says 172.21.2.117:56408) 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b81960700 1 -- 172.21.2.117:0/2851166454 learned_addr learned my addr 172.21.2.117:0/2851166454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.994 7f0b82962700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c0a36b0 0x7f0b7c0a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b82161700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0b24f0 0x7f0b7c0c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 -- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c0a36b0 msgr2=0x7f0b7c0a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:27.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c0a36b0 0x7f0b7c0a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 -- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0b24f0 msgr2=0x7f0b7c0c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0b24f0 0x7f0b7c0c20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 -- 172.21.2.117:0/2851166454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0a42f0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b81960700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f0b78006f60 tx=0x7f0b7800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b780009b0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7800bdf0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0b78002030 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b84345700 1 -- 172.21.2.117:0/2851166454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c0a42f0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.995 7f0b84345700 1 -- 172.21.2.117:0/2851166454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c0a4880 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.996 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0b78000c90 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.996 7f0b6effd700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 0x7f0b6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.996 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0b780325a0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.996 7f0b82161700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 0x7f0b6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:27.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:26.997 7f0b82161700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 0x7f0b6401abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:27.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.026 7f0b84345700 1 -- 172.21.2.117:0/2851166454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c10c2a0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.028 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0b780009b0 con 0x7f0b7c0b28f0 2021-02-10T11:52:27.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.153 7f0b84345700 1 -- 172.21.2.117:0/2851166454 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000968030", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7f0b7c0a4880 con 0x7f0b640186c0 2021-02-10T11:52:27.158 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.160 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:52:27.159 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.160 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:52:27.159 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.160 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:52:27.159 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.160 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:52:27.159 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.160 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:52:27.163 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:52:27.165 7f59c98e1700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:52:27.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.163 7f0b6effd700 1 -- 172.21.2.117:0/2851166454 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f0b70007af0 con 0x7f0b640186c0 2021-02-10T11:52:27.169 INFO:teuthology.orchestra.run.gibba017.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:52:27.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.170 7f0b84345700 1 -- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 msgr2=0x7f0b6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:27.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.170 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 0x7f0b6401abc0 crc :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.170 7f0b84345700 1 -- 172.21.2.117:0/2851166454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 msgr2=0x7f0b7c0c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:27.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.170 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 rx=0x7f0b78006f60 tx=0x7f0b7800b970).stop 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 -- 172.21.2.117:0/2851166454 shutdown_connections 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0b640186c0 0x7f0b6401abc0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0b7c0b28f0 0x7f0b7c0c26d0 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0b7c0a36b0 0x7f0b7c0a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 --2- 172.21.2.117:0/2851166454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0b7c0b24f0 0x7f0b7c0c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:27.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 -- 172.21.2.117:0/2851166454 >> 172.21.2.117:0/2851166454 conn(0x7f0b7c0a0e40 msgr2=0x7f0b7c0c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:27.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 -- 172.21.2.117:0/2851166454 shutdown_connections 2021-02-10T11:52:27.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:27.171 7f0b84345700 1 -- 172.21.2.117:0/2851166454 wait complete. 2021-02-10T11:52:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:52:27.180 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:52:27.180 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:52:27.526 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:52:27.872 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:52:27.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:27.872 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:27.872 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:28.210 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:52:28.210 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:52:28.210 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:52:28.210 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:52:28.210 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:28.563 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[169730]: fuse finished with error 0 and tester_r 0 2021-02-10T11:52:28.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:28.564 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:28.565 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:28.900 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:52:28.900 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:52:28.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:52:28.901 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:29.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:29.237 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:29.237 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:29.574 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:52:29.574 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:52:29.574 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:52:29.574 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:52:29.575 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:29.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[170379]: fuse finished with error 0 and tester_r 0 2021-02-10T11:52:29.928 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:29.928 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:29.928 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:30.264 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:52:30.265 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:52:30.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:52:30.265 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:30.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:30.601 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:30.601 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:52:30.934 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:52:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:30.935 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:52:30.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:52:30.935 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:52:31.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:31.271 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:31.271 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:52:31.604 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:52:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:31.605 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:52:31.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:52:31.605 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:52:31.940 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:52:32.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.448 7ff35dfdf700 1 Processor -- start 2021-02-10T11:52:32.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff35dfdf700 1 -- start start 2021-02-10T11:52:32.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff35dfdf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff3580e7180 0x7ff3580e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff35dfdf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580e7580 con 0x7ff3580e8390 2021-02-10T11:52:32.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff35dfdf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580e77f0 con 0x7ff3581b07f0 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff35dfdf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580e5ce0 con 0x7ff3580e7180 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff357fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff3580e8390 0x7ff3580e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56666/0 (socket says 172.21.2.117:56666) 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.449 7ff357fff700 1 -- 172.21.2.117:0/3358591057 learned_addr learned my addr 172.21.2.117:0/3358591057 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527193519 0 0) 0x7ff350001500 con 0x7ff3580e8390 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff344002fe0 con 0x7ff3580e8390 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 --2- 172.21.2.117:0/3358591057 >> v2:172.21.2.125:3300/0 conn(0x7ff3580e7180 0x7ff3580e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2927994639 0 0) 0x7ff350001460 con 0x7ff3580e8390 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 >> v2:172.21.2.125:3300/0 conn(0x7ff3580e7180 msgr2=0x7ff3580e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 --2- 172.21.2.117:0/3358591057 >> v2:172.21.2.125:3300/0 conn(0x7ff3580e7180 0x7ff3580e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 >> v1:172.21.2.125:6789/0 conn(0x7ff3581b07f0 legacy=0x7ff3580e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580e8f80 con 0x7ff3580e8390 2021-02-10T11:52:32.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff350001980 con 0x7ff3580e8390 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 >> v1:172.21.2.117:6789/0 conn(0x7ff3580e8390 legacy=0x7ff3580e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff344003ac0 0x7ff344005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 0x7ff344008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400c300 con 0x7ff3440067b0 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34400c520 con 0x7ff344003ac0 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff344050750 con 0x7ff344009590 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 0x7ff344008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff3577fe700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff344003ac0 0x7ff344005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff357fff700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 -- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff344009590 msgr2=0x7ff34400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 -- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff344003ac0 msgr2=0x7ff344005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff344003ac0 0x7ff344005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.450 7ff356ffd700 1 -- 172.21.2.117:0/3358591057 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440509b0 con 0x7ff3440067b0 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff357fff700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff344009590 0x7ff34400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff357fff700 1 -- 172.21.2.117:0/3358591057 reap_dead start 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff356ffd700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 0x7ff344008c90 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7ff34000c8f0 tx=0x7ff34000cc90).ready entity=mon.0 client_cookie=8639289f0c51702f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff3400009b0 con 0x7ff3440067b0 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff340009310 con 0x7ff3440067b0 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff3567fc700 1 -- 172.21.2.117:0/3358591057 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff340009760 con 0x7ff3440067b0 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 -- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 msgr2=0x7ff344008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 0x7ff344008c90 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 rx=0x7ff34000c8f0 tx=0x7ff34000cc90).stop 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 -- 172.21.2.117:0/3358591057 shutdown_connections 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 --2- 172.21.2.117:0/3358591057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440067b0 0x7ff344008c90 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 -- 172.21.2.117:0/3358591057 >> 172.21.2.117:0/3358591057 conn(0x7ff3580ce500 msgr2=0x7ff3580e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:32.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 -- 172.21.2.117:0/3358591057 shutdown_connections 2021-02-10T11:52:32.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.451 7ff35dfdf700 1 -- 172.21.2.117:0/3358591057 wait complete. 2021-02-10T11:52:32.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.466 7ff35dfdf700 1 Processor -- start 2021-02-10T11:52:32.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.466 7ff35dfdf700 1 -- start start 2021-02-10T11:52:32.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3580b3f30 0x7ff35810e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3581af980 0x7ff35809ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35810e2b0 con 0x7ff3581afd80 2021-02-10T11:52:32.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35809a8f0 con 0x7ff3581af980 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff35dfdf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580992b0 con 0x7ff3580b3f30 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56424/0 (socket says 172.21.2.117:56424) 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 -- 172.21.2.117:0/1409086551 learned_addr learned my addr 172.21.2.117:0/1409086551 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 -- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3580b3f30 msgr2=0x7ff35810e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3580b3f30 0x7ff35810e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 -- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3581af980 msgr2=0x7ff35809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3581af980 0x7ff35809ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 -- 172.21.2.117:0/1409086551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580994d0 con 0x7ff3581afd80 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.467 7ff357fff700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff350002190 tx=0x7ff35000ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff350001650 con 0x7ff3581afd80 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff350004930 con 0x7ff3581afd80 2021-02-10T11:52:32.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3580994d0 con 0x7ff3581afd80 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff358099a60 con 0x7ff3581afd80 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff350007020 con 0x7ff3581afd80 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff350001650 con 0x7ff3581afd80 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 0x7ff33c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.468 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff350032a50 con 0x7ff3581afd80 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.469 7ff3577fe700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 0x7ff33c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:32.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.470 7ff3577fe700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 0x7ff33c01abc0 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:32.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.499 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3580e77f0 con 0x7ff3581afd80 2021-02-10T11:52:32.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.502 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff350001270 con 0x7ff3581afd80 2021-02-10T11:52:32.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.650 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff3580e77f0 con 0x7ff3581afd80 2021-02-10T11:52:32.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.654 7ff34ffff700 1 -- 172.21.2.117:0/1409086551 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff350001650 con 0x7ff3581afd80 2021-02-10T11:52:32.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 msgr2=0x7ff33c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 0x7ff33c01abc0 crc :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 msgr2=0x7ff35809b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 rx=0x7ff350002190 tx=0x7ff35000ac90).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 shutdown_connections 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff33c0186c0 0x7ff33c01abc0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3581afd80 0x7ff35809b490 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3581af980 0x7ff35809ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 --2- 172.21.2.117:0/1409086551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3580b3f30 0x7ff35810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.659 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 >> 172.21.2.117:0/1409086551 conn(0x7ff35819f010 msgr2=0x7ff358191b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.660 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 shutdown_connections 2021-02-10T11:52:32.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:32.660 7ff35dfdf700 1 -- 172.21.2.117:0/1409086551 wait complete. 2021-02-10T11:52:32.670 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create' 2021-02-10T11:52:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 Processor -- start 2021-02-10T11:52:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 -- start start 2021-02-10T11:52:33.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35780ef3f0 0x7f35780eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35780ede60 con 0x7f35780d18e0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35780d1ce0 con 0x7f35780eb7f0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.080 7f357e92c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35780ef7f0 con 0x7f35780ef3f0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f35777fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35780ef3f0 0x7f35780eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f3577fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f35780d18e0 0x7f35780ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56682/0 (socket says 172.21.2.117:56682) 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f3577fff700 1 -- 172.21.2.117:0/3166099538 learned_addr learned my addr 172.21.2.117:0/3166099538 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f35777fe700 1 -- 172.21.2.117:0/3166099538 >> v1:172.21.2.125:6789/0 conn(0x7f35780eb7f0 legacy=0x7f35781b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f35777fe700 1 -- 172.21.2.117:0/3166099538 >> v1:172.21.2.117:6789/0 conn(0x7f35780d18e0 legacy=0x7f35780ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f35777fe700 1 -- 172.21.2.117:0/3166099538 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35781bb180 con 0x7f35780ef3f0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f35777fe700 1 --2- 172.21.2.117:0/3166099538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780ef3f0 0x7f35780eb1d0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f356c006650 tx=0x7f356c009a80).ready entity=mon.0 client_cookie=6854a4b578c2303a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f3576ffd700 1 -- 172.21.2.117:0/3166099538 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f356c00c0c0 con 0x7f35780ef3f0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f3576ffd700 1 -- 172.21.2.117:0/3166099538 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c00c340 con 0x7f35780ef3f0 2021-02-10T11:52:33.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f3576ffd700 1 -- 172.21.2.117:0/3166099538 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f356c00c860 con 0x7f35780ef3f0 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f357e92c700 1 -- 172.21.2.117:0/3166099538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780ef3f0 msgr2=0x7f35780eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.081 7f357e92c700 1 --2- 172.21.2.117:0/3166099538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780ef3f0 0x7f35780eb1d0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 rx=0x7f356c006650 tx=0x7f356c009a80).stop 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.082 7f357e92c700 1 -- 172.21.2.117:0/3166099538 shutdown_connections 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.082 7f357e92c700 1 --2- 172.21.2.117:0/3166099538 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780ef3f0 0x7f35780eb1d0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.082 7f357e92c700 1 -- 172.21.2.117:0/3166099538 >> 172.21.2.117:0/3166099538 conn(0x7f35780ced60 msgr2=0x7f35780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.082 7f357e92c700 1 -- 172.21.2.117:0/3166099538 shutdown_connections 2021-02-10T11:52:33.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.082 7f357e92c700 1 -- 172.21.2.117:0/3166099538 wait complete. 2021-02-10T11:52:33.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 Processor -- start 2021-02-10T11:52:33.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 -- start start 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35780b27a0 0x7f3578097b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3578098170 0x7f35781c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35780b2ba0 con 0x7f35780f2b10 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35780f2f10 con 0x7f3578098170 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f357e92c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3578098570 con 0x7f35780b27a0 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f3577fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f3577fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56436/0 (socket says 172.21.2.117:56436) 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.097 7f3577fff700 1 -- 172.21.2.117:0/478456333 learned_addr learned my addr 172.21.2.117:0/478456333 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 -- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35780b27a0 msgr2=0x7f3578097b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35780b27a0 0x7f3578097b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 -- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3578098170 msgr2=0x7f35781c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3578098170 0x7f35781c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 -- 172.21.2.117:0/478456333 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3578098790 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3577fff700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f3568001410 tx=0x7f356800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f356800ad40 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35680043a0 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35680047b0 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f357e92c700 1 -- 172.21.2.117:0/478456333 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3578098790 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.098 7f357e92c700 1 -- 172.21.2.117:0/478456333 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3578098790 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.099 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3568004580 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.099 7f3574ff9700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 0x7f355801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.099 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3568032920 con 0x7f35780f2b10 2021-02-10T11:52:33.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.100 7f35777fe700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 0x7f355801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:33.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.100 7f35777fe700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 0x7f355801abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:33.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.131 7f357e92c700 1 -- 172.21.2.117:0/478456333 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3578086140 con 0x7f35780f2b10 2021-02-10T11:52:33.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.133 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35680045b0 con 0x7f35780f2b10 2021-02-10T11:52:33.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.266 7f357e92c700 1 -- 172.21.2.117:0/478456333 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]} v 0) v1 -- 0x7f35780d18e0 con 0x7f35780f2b10 2021-02-10T11:52:33.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.686 7f3574ff9700 1 -- 172.21.2.117:0/478456333 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_create"]}]=0 v696) v1 ==== 156+0+0 (secure 0 0 0) 0x7f35680043a0 con 0x7f35780f2b10 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 -- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 msgr2=0x7f355801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 0x7f355801abc0 crc :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 -- 172.21.2.117:0/478456333 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 msgr2=0x7f35780b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 rx=0x7f3568001410 tx=0x7f356800a950).stop 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 -- 172.21.2.117:0/478456333 shutdown_connections 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3578098170 0x7f35781c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:33.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35780b27a0 0x7f3578097b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35580186c0 0x7f355801abc0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 --2- 172.21.2.117:0/478456333 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35780f2b10 0x7f35780b2180 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.735 7f357e92c700 1 -- 172.21.2.117:0/478456333 >> 172.21.2.117:0/478456333 conn(0x7f35780f3230 msgr2=0x7f35780b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.736 7f357e92c700 1 -- 172.21.2.117:0/478456333 shutdown_connections 2021-02-10T11:52:33.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:33.736 7f357e92c700 1 -- 172.21.2.117:0/478456333 wait complete. 2021-02-10T11:52:33.746 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_create (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:52:33.746 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:52:33.746 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.156 7f2bd2272700 1 Processor -- start 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.156 7f2bd2272700 1 -- start start 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd2272700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0ef3f0 0x7f2bcc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd2272700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0ede60 con 0x7f2bcc0eb7f0 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd2272700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0d1ce0 con 0x7f2bcc0d18e0 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd2272700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc0ef7f0 con 0x7f2bcc0ef3f0 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd088f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2bcc0eb7f0 0x7f2bcc1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56698/0 (socket says 172.21.2.117:56698) 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bd088f700 1 -- 172.21.2.117:0/1426877550 learned_addr learned my addr 172.21.2.117:0/1426877550 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.157 7f2bcb7fe700 1 --2- 172.21.2.117:0/1426877550 >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0ef3f0 0x7f2bcc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761287786 0 0) 0x7f2bbc000c60 con 0x7f2bcc0d18e0 2021-02-10T11:52:34.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb4002fe0 con 0x7f2bcc0d18e0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662046501 0 0) 0x7f2bc4001480 con 0x7f2bcc0eb7f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb4003210 con 0x7f2bcc0eb7f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcb7fe700 1 -- 172.21.2.117:0/1426877550 >> v1:172.21.2.125:6789/0 conn(0x7f2bcc0d18e0 legacy=0x7f2bcc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcb7fe700 1 -- 172.21.2.117:0/1426877550 >> v1:172.21.2.117:6789/0 conn(0x7f2bcc0eb7f0 legacy=0x7f2bcc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcb7fe700 1 -- 172.21.2.117:0/1426877550 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc1bb180 con 0x7f2bcc0ef3f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcb7fe700 1 --2- 172.21.2.117:0/1426877550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb1d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f2bc0006650 tx=0x7f2bc0009a80).ready entity=mon.1 client_cookie=e18eca2a44bda7a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc000c0c0 con 0x7f2bcc0ef3f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc000c340 con 0x7f2bcc0ef3f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bcaffd700 1 -- 172.21.2.117:0/1426877550 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc000c860 con 0x7f2bcc0ef3f0 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bd2272700 1 -- 172.21.2.117:0/1426877550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ef3f0 msgr2=0x7f2bcc0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.158 7f2bd2272700 1 --2- 172.21.2.117:0/1426877550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb1d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f2bc0006650 tx=0x7f2bc0009a80).stop 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.159 7f2bd2272700 1 -- 172.21.2.117:0/1426877550 shutdown_connections 2021-02-10T11:52:34.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.159 7f2bd2272700 1 --2- 172.21.2.117:0/1426877550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb1d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.159 7f2bd2272700 1 -- 172.21.2.117:0/1426877550 >> 172.21.2.117:0/1426877550 conn(0x7f2bcc0ced60 msgr2=0x7f2bcc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:34.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.159 7f2bd2272700 1 -- 172.21.2.117:0/1426877550 shutdown_connections 2021-02-10T11:52:34.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.159 7f2bd2272700 1 -- 172.21.2.117:0/1426877550 wait complete. 2021-02-10T11:52:34.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 Processor -- start 2021-02-10T11:52:34.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 -- start start 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0e6190 0x7f2bcc0e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc0e6bb0 0x7f2bcc0e6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc08a5a0 con 0x7f2bcc0e6190 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc08a780 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.174 7f2bd2272700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc08a9a0 con 0x7f2bcc0e6bb0 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0e6190 0x7f2bcc0e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60834/0 (socket says 172.21.2.117:60834) 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 -- 172.21.2.117:0/3108344709 learned_addr learned my addr 172.21.2.117:0/3108344709 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bd088f700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc0e6bb0 0x7f2bcc0e6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 -- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc0e6bb0 msgr2=0x7f2bcc0e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc0e6bb0 0x7f2bcc0e6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 -- 172.21.2.117:0/3108344709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0e6190 msgr2=0x7f2bcc0e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0e6190 0x7f2bcc0e6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 -- 172.21.2.117:0/3108344709 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc08abc0 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bcbfff700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f2bbc006f00 tx=0x7f2bbc00b940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.175 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bbc006b90 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc00bd30 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bbc002280 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc08abc0 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc08abc0 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bbc000aa0 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bc8ff9700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 0x7f2bac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2bbc032680 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.176 7f2bcb7fe700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 0x7f2bac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:34.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.177 7f2bcb7fe700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 0x7f2bac01abc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:34.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.207 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc087290 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.210 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bbc006b90 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.346 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f2bcc0ede60 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.691 7f2bc8ff9700 1 -- 172.21.2.117:0/3108344709 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v697) v1 ==== 155+0+0 (secure 0 0 0) 0x7f2bbc000aa0 con 0x7f2bcc0ae5b0 2021-02-10T11:52:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 msgr2=0x7f2bac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 0x7f2bac01abc0 crc :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 msgr2=0x7f2bcc0ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f2bbc006f00 tx=0x7f2bbc00b940).stop 2021-02-10T11:52:34.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 shutdown_connections 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bac0186c0 0x7f2bac01abc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0e6190 0x7f2bcc0e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc0e6bb0 0x7f2bcc0e6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 --2- 172.21.2.117:0/3108344709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc0ae5b0 0x7f2bcc0ae9b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 >> 172.21.2.117:0/3108344709 conn(0x7f2bcc1a3620 msgr2=0x7f2bcc0e0670 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.713 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 shutdown_connections 2021-02-10T11:52:34.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:34.714 7f2bd2272700 1 -- 172.21.2.117:0/3108344709 wait complete. 2021-02-10T11:52:34.724 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:52:35.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 Processor -- start 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 -- start start 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efc580d15e0 0x7efc580e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580e7580 con 0x7efc580e7180 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580e77f0 con 0x7efc581b07f0 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.138 7efc5fdb4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc580e5ce0 con 0x7efc580d15e0 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efc580d15e0 0x7efc580e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efc580d15e0 0x7efc580e8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56462/0 (socket says 172.21.2.117:56462) 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 -- 172.21.2.117:0/866871919 learned_addr learned my addr 172.21.2.117:0/866871919 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5d3cf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efc580e7180 0x7efc580e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56708/0 (socket says 172.21.2.117:56708) 2021-02-10T11:52:35.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 -- 172.21.2.117:0/866871919 >> v1:172.21.2.125:6789/0 conn(0x7efc581b07f0 legacy=0x7efc580e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 -- 172.21.2.117:0/866871919 >> v1:172.21.2.117:6789/0 conn(0x7efc580e7180 legacy=0x7efc580e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 -- 172.21.2.117:0/866871919 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580d19e0 con 0x7efc580d15e0 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5e3d1700 1 --2- 172.21.2.117:0/866871919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc580d15e0 0x7efc580e8560 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7efc50006f00 tx=0x7efc50009f10).ready entity=mon.0 client_cookie=edda69820523cbb1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5cbce700 1 -- 172.21.2.117:0/866871919 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5000b3b0 con 0x7efc580d15e0 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5cbce700 1 -- 172.21.2.117:0/866871919 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5000b630 con 0x7efc580d15e0 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5cbce700 1 -- 172.21.2.117:0/866871919 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc5000bb50 con 0x7efc580d15e0 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5fdb4700 1 -- 172.21.2.117:0/866871919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc580d15e0 msgr2=0x7efc580e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.139 7efc5fdb4700 1 --2- 172.21.2.117:0/866871919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc580d15e0 0x7efc580e8560 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 rx=0x7efc50006f00 tx=0x7efc50009f10).stop 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.140 7efc5fdb4700 1 -- 172.21.2.117:0/866871919 shutdown_connections 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.140 7efc5fdb4700 1 --2- 172.21.2.117:0/866871919 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc580d15e0 0x7efc580e8560 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.140 7efc5fdb4700 1 -- 172.21.2.117:0/866871919 >> 172.21.2.117:0/866871919 conn(0x7efc580ce500 msgr2=0x7efc580e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.140 7efc5fdb4700 1 -- 172.21.2.117:0/866871919 shutdown_connections 2021-02-10T11:52:35.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.140 7efc5fdb4700 1 -- 172.21.2.117:0/866871919 wait complete. 2021-02-10T11:52:35.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 Processor -- start 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 -- start start 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc580ab960 0x7efc580abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc580ac380 0x7efc580ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580ab3e0 con 0x7efc581c5370 2021-02-10T11:52:35.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5807e130 con 0x7efc580ab960 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5fdb4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5807e350 con 0x7efc580ac380 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5dbd0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5dbd0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56466/0 (socket says 172.21.2.117:56466) 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.155 7efc5dbd0700 1 -- 172.21.2.117:0/3988679977 learned_addr learned my addr 172.21.2.117:0/3988679977 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 -- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc580ac380 msgr2=0x7efc580ac780 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc580ac380 0x7efc580ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 -- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc580ab960 msgr2=0x7efc580abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc580ab960 0x7efc580abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 -- 172.21.2.117:0/3988679977 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580982f0 con 0x7efc581c5370 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5dbd0700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7efc4c00b5b0 tx=0x7efc4c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc4c00bd40 con 0x7efc581c5370 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c006240 con 0x7efc581c5370 2021-02-10T11:52:35.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc580982f0 con 0x7efc581c5370 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.156 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc4c005720 con 0x7efc581c5370 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.157 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc580982f0 con 0x7efc581c5370 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.157 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc4c0054a0 con 0x7efc581c5370 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.157 7efc4a7fc700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 0x7efc4001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.157 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efc4c032940 con 0x7efc581c5370 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.158 7efc5d3cf700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 0x7efc4001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:35.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.158 7efc5d3cf700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 0x7efc4001abd0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:35.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.188 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5807c0a0 con 0x7efc581c5370 2021-02-10T11:52:35.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.191 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc4c006030 con 0x7efc581c5370 2021-02-10T11:52:35.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.340 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7efc5807c0a0 con 0x7efc581c5370 2021-02-10T11:52:35.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.344 7efc4a7fc700 1 -- 172.21.2.117:0/3988679977 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7efc4c0054a0 con 0x7efc581c5370 2021-02-10T11:52:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 msgr2=0x7efc4001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 0x7efc4001abd0 crc :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 msgr2=0x7efc581c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:35.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 rx=0x7efc4c00b5b0 tx=0x7efc4c00b950).stop 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 shutdown_connections 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc580ac380 0x7efc580ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc580ab960 0x7efc580abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc400186d0 0x7efc4001abd0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 --2- 172.21.2.117:0/3988679977 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc581c5370 0x7efc581c5770 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:35.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 >> 172.21.2.117:0/3988679977 conn(0x7efc58084ee0 msgr2=0x7efc580873c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:35.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 shutdown_connections 2021-02-10T11:52:35.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:35.349 7efc5fdb4700 1 -- 172.21.2.117:0/3988679977 wait complete. 2021-02-10T11:52:35.359 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:52:35.359 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:52:35.707 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:52:36.051 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:52:36.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:36.052 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:36.052 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:52:36.388 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:52:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:36.389 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:52:36.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:36.389 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:36.389 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:52:36.725 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:52:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:36.725 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:52:36.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:36.726 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:36.726 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:52:37.060 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:52:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:37.061 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:52:37.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:52:37.061 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:52:37.061 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:52:37.396 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:52:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:52:37.396 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:52:37.397 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:52:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.807 7f8b9b0ec700 1 Processor -- start 2021-02-10T11:52:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.807 7f8b9b0ec700 1 -- start start 2021-02-10T11:52:37.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b9b0ec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8b941b6960 0x7f8b941b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b9b0ec700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940ede60 con 0x7f8b940d18e0 2021-02-10T11:52:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b9b0ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b940eb9c0 con 0x7f8b940ef3f0 2021-02-10T11:52:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b9b0ec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b940ef7f0 con 0x7f8b941b6960 2021-02-10T11:52:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b98f08700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8b940d18e0 0x7f8b940ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56722/0 (socket says 172.21.2.117:56722) 2021-02-10T11:52:37.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b98f08700 1 -- 172.21.2.117:0/336519814 learned_addr learned my addr 172.21.2.117:0/336519814 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.808 7f8b99709700 1 --2- 172.21.2.117:0/336519814 >> v2:172.21.2.125:3300/0 conn(0x7f8b941b6960 0x7f8b941b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17252145 0 0) 0x7f8b84001400 con 0x7f8b940ef3f0 2021-02-10T11:52:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b80002fe0 con 0x7f8b940ef3f0 2021-02-10T11:52:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259091529 0 0) 0x7f8b7c001100 con 0x7f8b940d18e0 2021-02-10T11:52:37.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b80003090 con 0x7f8b940d18e0 2021-02-10T11:52:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b99709700 1 -- 172.21.2.117:0/336519814 >> v1:172.21.2.125:6789/0 conn(0x7f8b940ef3f0 legacy=0x7f8b940eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b99709700 1 -- 172.21.2.117:0/336519814 >> v1:172.21.2.117:6789/0 conn(0x7f8b940d18e0 legacy=0x7f8b940ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b99709700 1 -- 172.21.2.117:0/336519814 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b940d1d20 con 0x7f8b941b6960 2021-02-10T11:52:37.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b99709700 1 --2- 172.21.2.117:0/336519814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b941b6960 0x7f8b941b8d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f8b8c006ad0 tx=0x7f8b8c009a50).ready entity=mon.1 client_cookie=9acdbc1586b9c90c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8c0066a0 con 0x7f8b941b6960 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c006410 con 0x7f8b941b6960 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b937fe700 1 -- 172.21.2.117:0/336519814 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b8c000c10 con 0x7f8b941b6960 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b9b0ec700 1 -- 172.21.2.117:0/336519814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b941b6960 msgr2=0x7f8b941b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.809 7f8b9b0ec700 1 --2- 172.21.2.117:0/336519814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b941b6960 0x7f8b941b8d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7f8b8c006ad0 tx=0x7f8b8c009a50).stop 2021-02-10T11:52:37.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.810 7f8b9b0ec700 1 -- 172.21.2.117:0/336519814 shutdown_connections 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.810 7f8b9b0ec700 1 --2- 172.21.2.117:0/336519814 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b941b6960 0x7f8b941b8d70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.810 7f8b9b0ec700 1 -- 172.21.2.117:0/336519814 >> 172.21.2.117:0/336519814 conn(0x7f8b940ced60 msgr2=0x7f8b940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.810 7f8b9b0ec700 1 -- 172.21.2.117:0/336519814 shutdown_connections 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.810 7f8b9b0ec700 1 -- 172.21.2.117:0/336519814 wait complete. 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 Processor -- start 2021-02-10T11:52:37.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 -- start start 2021-02-10T11:52:37.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:37.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b940ae480 0x7f8b940ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:37.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b9408a280 0x7f8b9408a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:37.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9408b270 con 0x7f8b9408a280 2021-02-10T11:52:37.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9408aca0 con 0x7f8b940979b0 2021-02-10T11:52:37.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b9b0ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9408ae80 con 0x7f8b940ae480 2021-02-10T11:52:37.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.825 7f8b98f08700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:37.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b99709700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b9408a280 0x7f8b9408a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60862/0 (socket says 172.21.2.117:60862) 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 -- 172.21.2.117:0/3563992779 learned_addr learned my addr 172.21.2.117:0/3563992779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b93fff700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b940ae480 0x7f8b940ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 -- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b940ae480 msgr2=0x7f8b940ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b940ae480 0x7f8b940ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:37.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 -- 172.21.2.117:0/3563992779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b9408a280 msgr2=0x7f8b9408a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:37.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b9408a280 0x7f8b9408a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:37.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 -- 172.21.2.117:0/3563992779 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9408b7f0 con 0x7f8b940979b0 2021-02-10T11:52:37.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b98f08700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f8b7c00b5c0 tx=0x7f8b7c00b960).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:37.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7c006030 con 0x7f8b940979b0 2021-02-10T11:52:37.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c000940 con 0x7f8b940979b0 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7c004590 con 0x7f8b940979b0 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9408b7f0 con 0x7f8b940979b0 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.826 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9408b7f0 con 0x7f8b940979b0 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.827 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b7c0043a0 con 0x7f8b940979b0 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.827 7f8b917fa700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 0x7f8b7801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:37.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.827 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b7c032920 con 0x7f8b940979b0 2021-02-10T11:52:37.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.827 7f8b93fff700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 0x7f8b7801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:37.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.828 7f8b93fff700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 0x7f8b7801ac10 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:37.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.857 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b94087290 con 0x7f8b940979b0 2021-02-10T11:52:37.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.860 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b7c006b80 con 0x7f8b940979b0 2021-02-10T11:52:37.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.989 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8b94087290 con 0x7f8b940979b0 2021-02-10T11:52:37.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.990 7f8b917fa700 1 -- 172.21.2.117:0/3563992779 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v134) v1 ==== 81+0+30766 (secure 0 0 0) 0x7f8b7c0043a0 con 0x7f8b940979b0 2021-02-10T11:52:37.994 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 134, 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:52:03.821388", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:52:37.995 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 29, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:52:37.996 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:52:37.997 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:52:37.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:52:37.999 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:52:38.000 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:52:38.001 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 28, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:52:00.791090", 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:52:38.002 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:52:38.003 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "133", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:52:38.004 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:52:38.005 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.006 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:52:38.007 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 29, 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:52:38.008 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:52:01.865056", 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:52:38.009 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "134", 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:52:38.010 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:52:38.011 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.012 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:52:38.013 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.014 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 130, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:52:38.017 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.018 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:52:38.019 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.020 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 132, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.023 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.024 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:52:38.025 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.026 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 123, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.027 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.028 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.029 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.030 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:52:38.031 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:52:38.032 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.033 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 115, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.036 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.037 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:52:38.038 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:52:38.039 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.040 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 132, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.042 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.043 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.044 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:52:38.045 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.046 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 130, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.047 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:52:38.051 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.052 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 132, 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.053 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.054 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.055 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:52:38.057 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:52:38.058 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 132, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:52:38.059 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:52:38.060 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:52:38.063 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.064 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.065 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:52:38.066 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.067 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.068 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:52:38.069 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:52:38.070 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:38.071 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:52:38.072 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:52:38.072 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:52:38.072 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:52:38.072 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:52:38.072 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:52:38.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 msgr2=0x7f8b7801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 0x7f8b7801ac10 crc :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 msgr2=0x7f8b9408ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7f8b7c00b5c0 tx=0x7f8b7c00b960).stop 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 shutdown_connections 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b78018730 0x7f8b7801ac10 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b9408a280 0x7f8b9408a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b940ae480 0x7f8b940ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 --2- 172.21.2.117:0/3563992779 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b940979b0 0x7f8b9408ddd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.993 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 >> 172.21.2.117:0/3563992779 conn(0x7f8b941a3620 msgr2=0x7f8b941a93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.994 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 shutdown_connections 2021-02-10T11:52:38.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:37.994 7f8b9b0ec700 1 -- 172.21.2.117:0/3563992779 wait complete. 2021-02-10T11:52:38.075 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:38.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.485 7f623807d700 1 Processor -- start 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.485 7f623807d700 1 -- start start 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623807d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62300ef3f0 0x7f62300eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623807d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f62300eb7f0 0x7f62301b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623807d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62300ede60 con 0x7f62300d18e0 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623807d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62300ef7f0 con 0x7f62300eb7f0 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623807d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62300d1ce0 con 0x7f62300ef3f0 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f62300eb7f0 0x7f62301b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f62300eb7f0 0x7f62301b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56488/0 (socket says 172.21.2.117:56488) 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 -- 172.21.2.117:0/3425235491 learned_addr learned my addr 172.21.2.117:0/3425235491 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:38.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 -- 172.21.2.117:0/3425235491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62300ef3f0 msgr2=0x7f62300eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 --2- 172.21.2.117:0/3425235491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62300ef3f0 0x7f62300eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 -- 172.21.2.117:0/3425235491 >> v1:172.21.2.125:6789/0 conn(0x7f62300d18e0 legacy=0x7f62300ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 -- 172.21.2.117:0/3425235491 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62301b8f90 con 0x7f62300eb7f0 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.486 7f623669a700 1 --2- 172.21.2.117:0/3425235491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300eb7f0 0x7f62301b7960 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7f6228006f00 tx=0x7f6228009f10).ready entity=mon.0 client_cookie=30eb87cc2a935a72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f6234e97700 1 -- 172.21.2.117:0/3425235491 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622800b3b0 con 0x7f62300eb7f0 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f6234e97700 1 -- 172.21.2.117:0/3425235491 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622800b730 con 0x7f62300eb7f0 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f6234e97700 1 -- 172.21.2.117:0/3425235491 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622800bc50 con 0x7f62300eb7f0 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 -- 172.21.2.117:0/3425235491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300eb7f0 msgr2=0x7f62301b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 --2- 172.21.2.117:0/3425235491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300eb7f0 0x7f62301b7960 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 rx=0x7f6228006f00 tx=0x7f6228009f10).stop 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 -- 172.21.2.117:0/3425235491 shutdown_connections 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 --2- 172.21.2.117:0/3425235491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300eb7f0 0x7f62301b7960 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 --2- 172.21.2.117:0/3425235491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f62300ef3f0 0x7f62300eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:38.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.487 7f623807d700 1 -- 172.21.2.117:0/3425235491 >> 172.21.2.117:0/3425235491 conn(0x7f62300ced60 msgr2=0x7f62300cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:38.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.488 7f623807d700 1 -- 172.21.2.117:0/3425235491 shutdown_connections 2021-02-10T11:52:38.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.488 7f623807d700 1 -- 172.21.2.117:0/3425235491 wait complete. 2021-02-10T11:52:38.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 Processor -- start 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 -- start start 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 0x7f62300e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62300e9960 0x7f62300e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230196810 con 0x7f62300e9560 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230098ad0 con 0x7f62300e9960 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.502 7f623807d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62300e8a40 con 0x7f6230196390 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56490/0 (socket says 172.21.2.117:56490) 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 -- 172.21.2.117:0/3085586118 learned_addr learned my addr 172.21.2.117:0/3085586118 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:38.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 -- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 msgr2=0x7f62300e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235e99700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 0x7f62300e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 0x7f62300e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 -- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62300e9960 msgr2=0x7f62300e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62300e9960 0x7f62300e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 -- 172.21.2.117:0/3085586118 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62300e8c60 con 0x7f62300e9560 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235e99700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 0x7f62300e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f6235698700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 rx=0x7f622c006f00 tx=0x7f622c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622c006b90 con 0x7f62300e9560 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c0063d0 con 0x7f62300e9560 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f622c000c90 con 0x7f62300e9560 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.503 7f623807d700 1 -- 172.21.2.117:0/3085586118 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62300e8c60 con 0x7f62300e9560 2021-02-10T11:52:38.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.504 7f623807d700 1 -- 172.21.2.117:0/3085586118 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623019cd70 con 0x7f62300e9560 2021-02-10T11:52:38.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.504 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f622c007000 con 0x7f62300e9560 2021-02-10T11:52:38.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.504 7f62227fc700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 0x7f621801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:38.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.504 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f622c032670 con 0x7f62300e9560 2021-02-10T11:52:38.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.504 7f6235e99700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 0x7f621801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:38.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.505 7f6235e99700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 0x7f621801abc0 crc :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.536 7f623807d700 1 -- 172.21.2.117:0/3085586118 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230087210 con 0x7f62300e9560 2021-02-10T11:52:38.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.538 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f622c006b90 con 0x7f62300e9560 2021-02-10T11:52:38.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.664 7f623807d700 1 -- 172.21.2.117:0/3085586118 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62300ede60 con 0x7f62300e9560 2021-02-10T11:52:38.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.664 7f62227fc700 1 -- 172.21.2.117:0/3085586118 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f622c00bd30 con 0x7f62300e9560 2021-02-10T11:52:38.667 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 112 2021-02-10T11:52:38.667 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:38.667 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":112,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:04.828714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7286},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7286":{"gid":7286,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/604396797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":604396797},{"type":"v1","addr":"172.21.2.125:6835","nonce":604396797}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:38.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 -- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 msgr2=0x7f621801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 0x7f621801abc0 crc :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 -- 172.21.2.117:0/3085586118 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 msgr2=0x7f62300e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:38.668 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 rx=0x7f622c006f00 tx=0x7f622c00b940).stop 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 -- 172.21.2.117:0/3085586118 shutdown_connections 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f62180186c0 0x7f621801abc0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f62300e9560 0x7f62300e7e00 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f62300e9960 0x7f62300e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 --2- 172.21.2.117:0/3085586118 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6230196390 0x7f62300e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:38.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.666 7f623807d700 1 -- 172.21.2.117:0/3085586118 >> 172.21.2.117:0/3085586118 conn(0x7f623019f610 msgr2=0x7f62301a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:38.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.667 7f623807d700 1 -- 172.21.2.117:0/3085586118 shutdown_connections 2021-02-10T11:52:38.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:38.667 7f623807d700 1 -- 172.21.2.117:0/3085586118 wait complete. 2021-02-10T11:52:38.677 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:52:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 Processor -- start 2021-02-10T11:52:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 -- start start 2021-02-10T11:52:39.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 0x7f74180e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74180e7580 con 0x7f74181b07f0 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74180e5ce0 con 0x7f74180e8390 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.087 7f741fcd5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74180e77f0 con 0x7f74180e7180 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.088 7f741d2f0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.088 7f741d2f0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54178/0 (socket says 172.21.2.117:54178) 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.088 7f741d2f0700 1 -- 172.21.2.117:0/1621954010 learned_addr learned my addr 172.21.2.117:0/1621954010 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.088 7f741e2f2700 1 --2- 172.21.2.117:0/1621954010 >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 0x7f74180e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741d2f0700 1 -- 172.21.2.117:0/1621954010 >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 msgr2=0x7f74180e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:39.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741d2f0700 1 --2- 172.21.2.117:0/1621954010 >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 0x7f74180e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741d2f0700 1 -- 172.21.2.117:0/1621954010 >> v1:172.21.2.125:6789/0 conn(0x7f74181b07f0 legacy=0x7f74180e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741d2f0700 1 -- 172.21.2.117:0/1621954010 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74180e8db0 con 0x7f74180e7180 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741e2f2700 1 --2- 172.21.2.117:0/1621954010 >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 0x7f74180e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741d2f0700 1 --2- 172.21.2.117:0/1621954010 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7414006700 tx=0x7f7414009b80).ready entity=mon.2 client_cookie=e9c55b4410c3ef95 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741caef700 1 -- 172.21.2.117:0/1621954010 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400c0c0 con 0x7f74180e7180 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741caef700 1 -- 172.21.2.117:0/1621954010 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741400c340 con 0x7f74180e7180 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741caef700 1 -- 172.21.2.117:0/1621954010 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f741400c880 con 0x7f74180e7180 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741fcd5700 1 -- 172.21.2.117:0/1621954010 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 msgr2=0x7f74180e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.089 7f741fcd5700 1 --2- 172.21.2.117:0/1621954010 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7414006700 tx=0x7f7414009b80).stop 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 -- 172.21.2.117:0/1621954010 shutdown_connections 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 --2- 172.21.2.117:0/1621954010 >> v2:172.21.2.125:3300/0 conn(0x7f74180e8390 0x7f74180e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 --2- 172.21.2.117:0/1621954010 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180e7180 0x7f74180e7d70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 -- 172.21.2.117:0/1621954010 >> 172.21.2.117:0/1621954010 conn(0x7f74180ce500 msgr2=0x7f74180e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:39.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 -- 172.21.2.117:0/1621954010 shutdown_connections 2021-02-10T11:52:39.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.090 7f741fcd5700 1 -- 172.21.2.117:0/1621954010 wait complete. 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.104 7f741fcd5700 1 Processor -- start 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.104 7f741fcd5700 1 -- start start 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180b2960 0x7f741807f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74180b23f0 0x7f741819d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741807f900 con 0x7f74180b19d0 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180b1450 con 0x7f74180b23f0 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741fcd5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741819d910 con 0x7f74180b2960 2021-02-10T11:52:39.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741d2f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741d2f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56504/0 (socket says 172.21.2.117:56504) 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.105 7f741d2f0700 1 -- 172.21.2.117:0/3866255478 learned_addr learned my addr 172.21.2.117:0/3866255478 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 -- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180b2960 msgr2=0x7f741807f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180b2960 0x7f741807f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 -- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74180b23f0 msgr2=0x7f741819d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74180b23f0 0x7f741819d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 -- 172.21.2.117:0/3866255478 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741819daf0 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741d2f0700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 rx=0x7f7414009720 tx=0x7f741400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74140090c0 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414000940 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741819daf0 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7414002220 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.106 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741819daf0 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.107 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f74140052e0 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.107 7f740a7fc700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 0x7f740001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.107 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7414032760 con 0x7f74180b19d0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.107 7f741daf1700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 0x7f740001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.107 7f741daf1700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 0x7f740001ac10 crc :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.137 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741819e100 con 0x7f74180b19d0 2021-02-10T11:52:39.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.139 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74140063a0 con 0x7f74180b19d0 2021-02-10T11:52:39.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.265 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f741819e100 con 0x7f74180b19d0 2021-02-10T11:52:39.299 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:52:39.298 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:52:39.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.300 7f740a7fc700 1 -- 172.21.2.117:0/3866255478 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) v1 ==== 161+0+0 (secure 0 0 0) 0x7f74140052e0 con 0x7f74180b19d0 2021-02-10T11:52:39.331 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:52:39.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 msgr2=0x7f740001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 0x7f740001ac10 crc :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 msgr2=0x7f74180b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 rx=0x7f7414009720 tx=0x7f741400b960).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 shutdown_connections 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7400018730 0x7f740001ac10 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f74180b19d0 0x7f74180b1dd0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74180b23f0 0x7f741819d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 --2- 172.21.2.117:0/3866255478 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74180b2960 0x7f741807f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 >> 172.21.2.117:0/3866255478 conn(0x7f74180ae8b0 msgr2=0x7f7418090900 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:39.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 shutdown_connections 2021-02-10T11:52:39.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.331 7f741fcd5700 1 -- 172.21.2.117:0/3866255478 wait complete. 2021-02-10T11:52:39.342 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:39.347 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:52:39.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.750 7fe851703700 1 Processor -- start 2021-02-10T11:52:39.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.750 7fe851703700 1 -- start start 2021-02-10T11:52:39.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe84c0d18e0 0x7fe84c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c0eb000 0x7fe84c0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c0ede60 con 0x7fe84c0ee3e0 2021-02-10T11:52:39.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c0eba20 con 0x7fe84c0d18e0 2021-02-10T11:52:39.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe851703700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0d1ce0 con 0x7fe84c0eb000 2021-02-10T11:52:39.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56512/0 (socket says 172.21.2.117:56512) 2021-02-10T11:52:39.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 -- 172.21.2.117:0/3022837162 learned_addr learned my addr 172.21.2.117:0/3022837162 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:39.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 -- 172.21.2.117:0/3022837162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c0eb000 msgr2=0x7fe84c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:39.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 --2- 172.21.2.117:0/3022837162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c0eb000 0x7fe84c0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 -- 172.21.2.117:0/3022837162 >> v2:172.21.2.125:3300/0 conn(0x7fe84c0d18e0 msgr2=0x7fe84c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 --2- 172.21.2.117:0/3022837162 >> v2:172.21.2.125:3300/0 conn(0x7fe84c0d18e0 0x7fe84c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 -- 172.21.2.117:0/3022837162 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0eee00 con 0x7fe84c0ee3e0 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.751 7fe84bac0700 1 --2- 172.21.2.117:0/3022837162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7fe844006f00 tx=0x7fe844009f10).ready entity=mon.0 client_cookie=9b966e937413dc46 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe84a2bd700 1 -- 172.21.2.117:0/3022837162 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe84400b3b0 con 0x7fe84c0ee3e0 2021-02-10T11:52:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe84a2bd700 1 -- 172.21.2.117:0/3022837162 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400b630 con 0x7fe84c0ee3e0 2021-02-10T11:52:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe84a2bd700 1 -- 172.21.2.117:0/3022837162 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe84400bb10 con 0x7fe84c0ee3e0 2021-02-10T11:52:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 -- 172.21.2.117:0/3022837162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c0ee3e0 msgr2=0x7fe84c0ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 --2- 172.21.2.117:0/3022837162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 rx=0x7fe844006f00 tx=0x7fe844009f10).stop 2021-02-10T11:52:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 -- 172.21.2.117:0/3022837162 shutdown_connections 2021-02-10T11:52:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 --2- 172.21.2.117:0/3022837162 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c0ee3e0 0x7fe84c0ee7e0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 --2- 172.21.2.117:0/3022837162 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c0eb000 0x7fe84c0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 --2- 172.21.2.117:0/3022837162 >> v2:172.21.2.125:3300/0 conn(0x7fe84c0d18e0 0x7fe84c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 -- 172.21.2.117:0/3022837162 >> 172.21.2.117:0/3022837162 conn(0x7fe84c0ced60 msgr2=0x7fe84c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 -- 172.21.2.117:0/3022837162 shutdown_connections 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.752 7fe851703700 1 -- 172.21.2.117:0/3022837162 wait complete. 2021-02-10T11:52:39.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 Processor -- start 2021-02-10T11:52:39.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 -- start start 2021-02-10T11:52:39.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c08a280 0x7fe84c08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe84c0e8f90 0x7fe84c0e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c0a9aa0 con 0x7fe84c19c7c0 2021-02-10T11:52:39.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c19cbc0 con 0x7fe84c0e8f90 2021-02-10T11:52:39.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe851703700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c08aca0 con 0x7fe84c08a280 2021-02-10T11:52:39.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe84b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe84b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56518/0 (socket says 172.21.2.117:56518) 2021-02-10T11:52:39.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.767 7fe84b2bf700 1 -- 172.21.2.117:0/863395849 learned_addr learned my addr 172.21.2.117:0/863395849 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:39.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 -- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c08a280 msgr2=0x7fe84c08a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c08a280 0x7fe84c08a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 -- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe84c0e8f90 msgr2=0x7fe84c0e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe84c0e8f90 0x7fe84c0e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 -- 172.21.2.117:0/863395849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0e99b0 con 0x7fe84c19c7c0 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe84b2bf700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 rx=0x7fe83c006f00 tx=0x7fe83c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c00bd30 con 0x7fe84c19c7c0 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c000c90 con 0x7fe84c19c7c0 2021-02-10T11:52:39.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe83c0022a0 con 0x7fe84c19c7c0 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe851703700 1 -- 172.21.2.117:0/863395849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c0e99b0 con 0x7fe84c19c7c0 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.768 7fe851703700 1 -- 172.21.2.117:0/863395849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c19b330 con 0x7fe84c19c7c0 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.769 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe83c005310 con 0x7fe84c19c7c0 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.769 7fe833fff700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 0x7fe82c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.769 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fe83c032730 con 0x7fe84c19c7c0 2021-02-10T11:52:39.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.769 7fe84aabe700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 0x7fe82c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:39.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.770 7fe84aabe700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 0x7fe82c01abc0 crc :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:39.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.801 7fe851703700 1 -- 172.21.2.117:0/863395849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c0e8a60 con 0x7fe84c19c7c0 2021-02-10T11:52:39.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.803 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe83c000960 con 0x7fe84c19c7c0 2021-02-10T11:52:39.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.929 7fe851703700 1 -- 172.21.2.117:0/863395849 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe84c0e8a60 con 0x7fe84c19c7c0 2021-02-10T11:52:39.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.929 7fe833fff700 1 -- 172.21.2.117:0/863395849 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+1763 (secure 0 0 0) 0x7fe83c00bd30 con 0x7fe84c19c7c0 2021-02-10T11:52:39.932 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 113 2021-02-10T11:52:39.932 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:39.932 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":113,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1292598946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1292598946},{"type":"v1","addr":"172.21.2.117:6833","nonce":1292598946}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":113,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:02.818807","modified":"2021-02-10 11:52:39.298957","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":135,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2021-02-10T11:52:39.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 -- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 msgr2=0x7fe82c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 0x7fe82c01abc0 crc :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 -- 172.21.2.117:0/863395849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 msgr2=0x7fe84c092f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 rx=0x7fe83c006f00 tx=0x7fe83c00b940).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 -- 172.21.2.117:0/863395849 shutdown_connections 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe84c0e8f90 0x7fe84c0e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe84c08a280 0x7fe84c08a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe82c0186c0 0x7fe82c01abc0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 --2- 172.21.2.117:0/863395849 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe84c19c7c0 0x7fe84c092f90 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.931 7fe851703700 1 -- 172.21.2.117:0/863395849 >> 172.21.2.117:0/863395849 conn(0x7fe84c19f610 msgr2=0x7fe84c098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.932 7fe851703700 1 -- 172.21.2.117:0/863395849 shutdown_connections 2021-02-10T11:52:39.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:39.932 7fe851703700 1 -- 172.21.2.117:0/863395849 wait complete. 2021-02-10T11:52:39.942 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 Processor -- start 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 -- start start 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c0e8390 0x7fdc6c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0e7580 con 0x7fdc6c1b07f0 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc6c0e5ce0 con 0x7fdc6c0e7180 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc70edb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0e77f0 con 0x7fdc6c0e8390 2021-02-10T11:52:40.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc6a312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc6a312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56526/0 (socket says 172.21.2.117:56526) 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.358 7fdc6a312700 1 -- 172.21.2.117:0/3936972176 learned_addr learned my addr 172.21.2.117:0/3936972176 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6b314700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c0e8390 0x7fdc6c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6a312700 1 -- 172.21.2.117:0/3936972176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c0e8390 msgr2=0x7fdc6c0e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6a312700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c0e8390 0x7fdc6c0e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6a312700 1 -- 172.21.2.117:0/3936972176 >> v1:172.21.2.125:6789/0 conn(0x7fdc6c1b07f0 legacy=0x7fdc6c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:40.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6a312700 1 -- 172.21.2.117:0/3936972176 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0e8db0 con 0x7fdc6c0e7180 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc6a312700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7fdc5c006700 tx=0x7fdc5c009b80).ready entity=mon.0 client_cookie=83281aa36f3b6e31 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc69b11700 1 -- 172.21.2.117:0/3936972176 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc5c00c0c0 con 0x7fdc6c0e7180 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc69b11700 1 -- 172.21.2.117:0/3936972176 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c00c340 con 0x7fdc6c0e7180 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc69b11700 1 -- 172.21.2.117:0/3936972176 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc5c00c860 con 0x7fdc6c0e7180 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc70edb700 1 -- 172.21.2.117:0/3936972176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c0e7180 msgr2=0x7fdc6c0e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.359 7fdc70edb700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 rx=0x7fdc5c006700 tx=0x7fdc5c009b80).stop 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 -- 172.21.2.117:0/3936972176 shutdown_connections 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c0e8390 0x7fdc6c0e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 --2- 172.21.2.117:0/3936972176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c0e7180 0x7fdc6c0e7d70 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 -- 172.21.2.117:0/3936972176 >> 172.21.2.117:0/3936972176 conn(0x7fdc6c0ce500 msgr2=0x7fdc6c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 -- 172.21.2.117:0/3936972176 shutdown_connections 2021-02-10T11:52:40.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.360 7fdc70edb700 1 -- 172.21.2.117:0/3936972176 wait complete. 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 Processor -- start 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 -- start start 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 0x7fdc6c0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 0x7fdc6c197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 0x7fdc6c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c198390 con 0x7fdc6c198800 2021-02-10T11:52:40.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0aeac0 con 0x7fdc6c197970 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.375 7fdc70edb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c0aece0 con 0x7fdc6c10b430 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6a312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 0x7fdc6c197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 0x7fdc6c0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6a312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 0x7fdc6c197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54210/0 (socket says 172.21.2.117:54210) 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6a312700 1 -- 172.21.2.117:0/494187115 learned_addr learned my addr 172.21.2.117:0/494187115 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6b314700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 0x7fdc6c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 -- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 msgr2=0x7fdc6c197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 0x7fdc6c197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 -- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 msgr2=0x7fdc6c197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 0x7fdc6c197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 -- 172.21.2.117:0/494187115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c07eea0 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6b314700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 0x7fdc6c197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.376 7fdc6ab13700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 0x7fdc6c0ae4a0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7fdc5400b510 tx=0x7fdc5400b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc54000960 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5400bca0 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc70edb700 1 -- 172.21.2.117:0/494187115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c07eea0 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc70edb700 1 -- 172.21.2.117:0/494187115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c07f430 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdc540024d0 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdc54006020 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 0x7fdc5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.377 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fdc54032870 con 0x7fdc6c198800 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.378 7fdc6a312700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 0x7fdc5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:40.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.378 7fdc6a312700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 0x7fdc5001abc0 crc :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:40.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.408 7fdc70edb700 1 -- 172.21.2.117:0/494187115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c0e77f0 con 0x7fdc6c198800 2021-02-10T11:52:40.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.411 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdc54000940 con 0x7fdc6c198800 2021-02-10T11:52:40.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:40.538 7fdc70edb700 1 -- 172.21.2.117:0/494187115 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdc6c0e77f0 con 0x7fdc6c198800 2021-02-10T11:52:41.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.310 7fdc637fe700 1 -- 172.21.2.117:0/494187115 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdc54006020 con 0x7fdc6c198800 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 -- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 msgr2=0x7fdc5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 0x7fdc5001abc0 crc :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 -- 172.21.2.117:0/494187115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 msgr2=0x7fdc6c0ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 0x7fdc6c0ae4a0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 rx=0x7fdc5400b510 tx=0x7fdc5400b8b0).stop 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 -- 172.21.2.117:0/494187115 shutdown_connections 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdc6c197970 0x7fdc6c197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdc6c10b430 0x7fdc6c197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc500186c0 0x7fdc5001abc0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 --2- 172.21.2.117:0/494187115 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdc6c198800 0x7fdc6c0ae4a0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.359 7fdc70edb700 1 -- 172.21.2.117:0/494187115 >> 172.21.2.117:0/494187115 conn(0x7fdc6c1998b0 msgr2=0x7fdc6c1a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:41.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.360 7fdc70edb700 1 -- 172.21.2.117:0/494187115 shutdown_connections 2021-02-10T11:52:41.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.360 7fdc70edb700 1 -- 172.21.2.117:0/494187115 wait complete. 2021-02-10T11:52:41.370 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.783 7fa9aa252700 1 Processor -- start 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.783 7fa9aa252700 1 -- start start 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9aa252700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4085450 0x7fa9a4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9aa252700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a4085930 con 0x7fa9a40e4e40 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9aa252700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a41b09e0 con 0x7fa9a40e5240 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9aa252700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40e5cc0 con 0x7fa9a4085450 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9a3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa9a40e4e40 0x7fa9a41b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56788/0 (socket says 172.21.2.117:56788) 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9a3fff700 1 -- 172.21.2.117:0/878380614 learned_addr learned my addr 172.21.2.117:0/878380614 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:41.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914715274 0 0) 0x7fa9940009b0 con 0x7fa9a40e4e40 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa98c002fe0 con 0x7fa9a40e4e40 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.784 7fa9a886f700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4085450 0x7fa9a4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 690429986 0 0) 0x7fa994001010 con 0x7fa9a40e4e40 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4085450 msgr2=0x7fa9a4086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4085450 0x7fa9a4086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 >> v1:172.21.2.125:6789/0 conn(0x7fa9a40e5240 legacy=0x7fa9a4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4084840 con 0x7fa9a40e4e40 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa994001590 con 0x7fa9a40e4e40 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 >> v1:172.21.2.117:6789/0 conn(0x7fa9a40e4e40 legacy=0x7fa9a41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa98c003af0 0x7fa98c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa98c006770 0x7fa98c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa98c009530 0x7fa98c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c00c2c0 con 0x7fa98c006770 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c00c4e0 con 0x7fa98c009530 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c050710 con 0x7fa98c003af0 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa98c006770 0x7fa98c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a3fff700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa98c003af0 0x7fa98c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 -- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa98c003af0 msgr2=0x7fa98c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa98c003af0 0x7fa98c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 -- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa98c009530 msgr2=0x7fa98c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa98c009530 0x7fa98c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 -- 172.21.2.117:0/878380614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c050970 con 0x7fa98c006770 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a37fe700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa98c006770 0x7fa98c008c90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7fa998006ac0 tx=0x7fa998009af0).ready entity=mon.0 client_cookie=a4cea15be04f2d30 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa99800c340 con 0x7fa98c006770 2021-02-10T11:52:41.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99800c6c0 con 0x7fa98c006770 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a2ffd700 1 -- 172.21.2.117:0/878380614 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa99800cb10 con 0x7fa98c006770 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9aa252700 1 -- 172.21.2.117:0/878380614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa98c006770 msgr2=0x7fa98c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9aa252700 1 --2- 172.21.2.117:0/878380614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa98c006770 0x7fa98c008c90 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 rx=0x7fa998006ac0 tx=0x7fa998009af0).stop 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.785 7fa9a886f700 1 -- 172.21.2.117:0/878380614 reap_dead start 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.786 7fa9aa252700 1 -- 172.21.2.117:0/878380614 shutdown_connections 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.786 7fa9aa252700 1 -- 172.21.2.117:0/878380614 >> 172.21.2.117:0/878380614 conn(0x7fa9a40ce4e0 msgr2=0x7fa9a40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.786 7fa9aa252700 1 -- 172.21.2.117:0/878380614 shutdown_connections 2021-02-10T11:52:41.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.786 7fa9aa252700 1 -- 172.21.2.117:0/878380614 wait complete. 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 Processor -- start 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 -- start start 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9a410ffa0 0x7fa9a41103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4090ae0 0x7fa9a4090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a41109c0 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40a0ea0 con 0x7fa9a410ffa0 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.801 7fa9aa252700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a41a0360 con 0x7fa9a4090ae0 2021-02-10T11:52:41.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56552/0 (socket says 172.21.2.117:56552) 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 -- 172.21.2.117:0/2877908282 learned_addr learned my addr 172.21.2.117:0/2877908282 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 -- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4090ae0 msgr2=0x7fa9a4090ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4090ae0 0x7fa9a4090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 -- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9a410ffa0 msgr2=0x7fa9a41103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9a410ffa0 0x7fa9a41103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 -- 172.21.2.117:0/2877908282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4110d00 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a3fff700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7fa99400a4d0 tx=0x7fa99400a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9940009b0 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99400ac60 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9940045d0 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.802 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a4110d00 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.803 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a4110d00 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.803 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa994004350 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.803 7fa9a0ff9700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 0x7fa98401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.803 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fa9940327e0 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.803 7fa9a37fe700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 0x7fa98401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:41.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.804 7fa9a37fe700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 0x7fa98401abd0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:41.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.833 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a41109c0 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.836 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa994004140 con 0x7fa9a40a0aa0 2021-02-10T11:52:41.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:41.964 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa9a41109c0 con 0x7fa9a40a0aa0 2021-02-10T11:52:42.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.312 7fa9a0ff9700 1 -- 172.21.2.117:0/2877908282 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v136) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa99401f4f0 con 0x7fa9a40a0aa0 2021-02-10T11:52:42.334 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:52:42.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 msgr2=0x7fa98401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:42.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 0x7fa98401abd0 crc :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 msgr2=0x7fa9a419fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7fa99400a4d0 tx=0x7fa99400a870).stop 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 shutdown_connections 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9a4090ae0 0x7fa9a4090ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9a410ffa0 0x7fa9a41103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9840186d0 0x7fa98401abd0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 --2- 172.21.2.117:0/2877908282 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9a40a0aa0 0x7fa9a419fd40 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 >> 172.21.2.117:0/2877908282 conn(0x7fa9a40a5e90 msgr2=0x7fa9a40b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:42.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 shutdown_connections 2021-02-10T11:52:42.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.333 7fa9aa252700 1 -- 172.21.2.117:0/2877908282 wait complete. 2021-02-10T11:52:42.345 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:52:42.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 Processor -- start 2021-02-10T11:52:42.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 -- start start 2021-02-10T11:52:42.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92580e7170 0x7f92580e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f92580e8380 0x7f92580e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580e7570 con 0x7f92581b07e0 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.758 7f925ddcf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92580e5cd0 con 0x7f92580e8380 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f925ddcf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92580e77e0 con 0x7f92580e7170 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f92577fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f92581b07e0 0x7f92580e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56808/0 (socket says 172.21.2.117:56808) 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f92577fe700 1 -- 172.21.2.117:0/264397952 learned_addr learned my addr 172.21.2.117:0/264397952 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f92567fc700 1 -- 172.21.2.117:0/264397952 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031587038 0 0) 0x7f92480010b0 con 0x7f92581b07e0 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f92567fc700 1 -- 172.21.2.117:0/264397952 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9244002fe0 con 0x7f92581b07e0 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f9257fff700 1 --2- 172.21.2.117:0/264397952 >> v2:172.21.2.125:3300/0 conn(0x7f92580e8380 0x7f92580e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.759 7f9256ffd700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92580e7170 0x7f92580e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f9257fff700 1 -- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92580e7170 msgr2=0x7f92580e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:42.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f9257fff700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92580e7170 0x7f92580e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f9257fff700 1 -- 172.21.2.117:0/264397952 >> v1:172.21.2.117:6789/0 conn(0x7f92581b07e0 legacy=0x7f92580e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f9257fff700 1 -- 172.21.2.117:0/264397952 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580e8da0 con 0x7f92580e8380 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f9257fff700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580e8380 0x7f92580e8780 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f9250006e50 tx=0x7f9250009e70).ready entity=mon.1 client_cookie=8b221c52ac5dc84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f92567fc700 1 -- 172.21.2.117:0/264397952 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9250006a20 con 0x7f92580e8380 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f92567fc700 1 -- 172.21.2.117:0/264397952 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250006790 con 0x7f92580e8380 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f92567fc700 1 -- 172.21.2.117:0/264397952 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925000b700 con 0x7f92580e8380 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f925ddcf700 1 -- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580e8380 msgr2=0x7f92580e8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.760 7f925ddcf700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580e8380 0x7f92580e8780 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 rx=0x7f9250006e50 tx=0x7f9250009e70).stop 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 -- 172.21.2.117:0/264397952 shutdown_connections 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580e8380 0x7f92580e8780 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 --2- 172.21.2.117:0/264397952 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92580e7170 0x7f92580e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:42.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 -- 172.21.2.117:0/264397952 >> 172.21.2.117:0/264397952 conn(0x7f92580ce4f0 msgr2=0x7f92580e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:42.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 -- 172.21.2.117:0/264397952 shutdown_connections 2021-02-10T11:52:42.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.762 7f925ddcf700 1 -- 172.21.2.117:0/264397952 wait complete. 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.775 7f925ddcf700 1 Processor -- start 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 -- start start 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580a3cf0 0x7f92581a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92581a41e0 0x7f925810f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580a4280 con 0x7f92581a45e0 2021-02-10T11:52:42.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9258110300 con 0x7f92580a3cf0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f925ddcf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9258110520 con 0x7f92581a41e0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56566/0 (socket says 172.21.2.117:56566) 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 -- 172.21.2.117:0/2159797321 learned_addr learned my addr 172.21.2.117:0/2159797321 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 -- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92581a41e0 msgr2=0x7f925810f6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92581a41e0 0x7f925810f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 -- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580a3cf0 msgr2=0x7f92581a3bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580a3cf0 0x7f92581a3bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.776 7f9257fff700 1 -- 172.21.2.117:0/2159797321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9258110790 con 0x7f92581a45e0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f9257fff700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f9250006e60 tx=0x7f925000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9250009130 con 0x7f92581a45e0 2021-02-10T11:52:42.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250000e80 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9250001390 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9258110790 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9258110790 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9250009130 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.777 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f9250032980 con 0x7f92581a45e0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.778 7f92577fe700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:42.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.779 7f92577fe700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:42.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.808 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925810c250 con 0x7f92581a45e0 2021-02-10T11:52:42.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.811 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f925000be30 con 0x7f92581a45e0 2021-02-10T11:52:42.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:42.936 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f925810c250 con 0x7f92581a45e0 2021-02-10T11:52:43.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.317 7f924ffff700 1 -- 172.21.2.117:0/2159797321 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v137) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9250009130 con 0x7f92581a45e0 2021-02-10T11:52:43.355 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:52:43.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 msgr2=0x7f923c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 crc :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 msgr2=0x7f925810fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 rx=0x7f9250006e60 tx=0x7f925000bcf0).stop 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 shutdown_connections 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92581a45e0 0x7f925810fce0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92581a41e0 0x7f925810f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 --2- 172.21.2.117:0/2159797321 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92580a3cf0 0x7f92581a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 >> 172.21.2.117:0/2159797321 conn(0x7f925819f020 msgr2=0x7f92581af160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 shutdown_connections 2021-02-10T11:52:43.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.355 7f925ddcf700 1 -- 172.21.2.117:0/2159797321 wait complete. 2021-02-10T11:52:43.367 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:52:43.367 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:52:43.367 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:52:43.367 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:52:43.367 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:52:43.367 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:52:43.368 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:52:43.368 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:52:43.373 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:52:43.373 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:52:43.374 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:52:43.374 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:52:43.415 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:52:43.415 DEBUG:teuthology.parallel:result is None 2021-02-10T11:52:43.416 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:52:43.416 DEBUG:teuthology.parallel:result is None 2021-02-10T11:52:43.416 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:52:43.820 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:52:43.826 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:52:43.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 Processor -- start 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 -- start start 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f78141b07f0 0x7f78140e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140e77f0 con 0x7f78140e6b90 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78140e5ce0 con 0x7f78141b07f0 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.832 7f7819d38700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78140e54b0 con 0x7f78140e6170 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f7812ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f78141b07f0 0x7f78140e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f78141b07f0 0x7f78140e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:60970/0 (socket says 172.21.2.117:60970) 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f7812ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54266/0 (socket says 172.21.2.117:54266) 2021-02-10T11:52:43.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 -- 172.21.2.117:0/1208981777 learned_addr learned my addr 172.21.2.117:0/1208981777 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 -- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 msgr2=0x7f78140e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 -- 172.21.2.117:0/1208981777 >> v1:172.21.2.125:6789/0 conn(0x7f78140e6b90 legacy=0x7f78140e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 -- 172.21.2.117:0/1208981777 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140e8d80 con 0x7f78141b07f0 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f7812ffd700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.833 7f78137fe700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78141b07f0 0x7f78140e4e90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f77fc006700 tx=0x7f77fc009b80).ready entity=mon.1 client_cookie=44c12a2bb86bc032 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f78127fc700 1 -- 172.21.2.117:0/1208981777 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc00c0c0 con 0x7f78141b07f0 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f78127fc700 1 -- 172.21.2.117:0/1208981777 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc00c340 con 0x7f78141b07f0 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f78127fc700 1 -- 172.21.2.117:0/1208981777 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc00c860 con 0x7f78141b07f0 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 -- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78141b07f0 msgr2=0x7f78140e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78141b07f0 0x7f78140e4e90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f77fc006700 tx=0x7f77fc009b80).stop 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 -- 172.21.2.117:0/1208981777 shutdown_connections 2021-02-10T11:52:43.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78140e6170 0x7f78140e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 --2- 172.21.2.117:0/1208981777 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78141b07f0 0x7f78140e4e90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 -- 172.21.2.117:0/1208981777 >> 172.21.2.117:0/1208981777 conn(0x7f78140ce500 msgr2=0x7f78140e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 -- 172.21.2.117:0/1208981777 shutdown_connections 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.834 7f7819d38700 1 -- 172.21.2.117:0/1208981777 wait complete. 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.849 7f7819d38700 1 Processor -- start 2021-02-10T11:52:43.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.849 7f7819d38700 1 -- start start 2021-02-10T11:52:43.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781408c380 0x7f7814199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781408c780 0x7f7814199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781408cca0 con 0x7f78140af5f0 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7814199c90 con 0x7f781408c780 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7819d38700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7814199e70 con 0x7f781408c380 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56592/0 (socket says 172.21.2.117:56592) 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 -- 172.21.2.117:0/411848013 learned_addr learned my addr 172.21.2.117:0/411848013 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f7812ffd700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781408c380 0x7f7814199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 -- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781408c380 msgr2=0x7f7814199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781408c380 0x7f7814199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 -- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781408c780 msgr2=0x7f7814199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:43.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781408c780 0x7f7814199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 -- 172.21.2.117:0/411848013 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781409d790 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.850 7f78137fe700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 rx=0x7f77fc00b5e0 tx=0x7f77fc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc00bd70 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77fc002030 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f77fc0024d0 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f7819d38700 1 -- 172.21.2.117:0/411848013 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781409d790 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f7819d38700 1 -- 172.21.2.117:0/411848013 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781409d790 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77fc000c90 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 0x7f77f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f77fc032530 con 0x7f78140af5f0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.851 7f7812ffd700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 0x7f77f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:43.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.852 7f7812ffd700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 0x7f77f801abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:43.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.881 7f7819d38700 1 -- 172.21.2.117:0/411848013 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7814106e10 con 0x7f78140af5f0 2021-02-10T11:52:43.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:43.884 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f77fc01f570 con 0x7f78140af5f0 2021-02-10T11:52:44.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.022 7f7819d38700 1 -- 172.21.2.117:0/411848013 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f7814106e10 con 0x7f78140af5f0 2021-02-10T11:52:44.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.354 7f780bfff700 1 -- 172.21.2.117:0/411848013 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v138) v1 ==== 97+0+0 (secure 0 0 0) 0x7f77fc000c90 con 0x7f78140af5f0 2021-02-10T11:52:44.390 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:52:44.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 -- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 msgr2=0x7f77f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 0x7f77f801abc0 crc :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 -- 172.21.2.117:0/411848013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 msgr2=0x7f781408bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 rx=0x7f77fc00b5e0 tx=0x7f77fc00b980).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 -- 172.21.2.117:0/411848013 shutdown_connections 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f781408c780 0x7f7814199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f781408c380 0x7f7814199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77f8018730 0x7f77f801abc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 --2- 172.21.2.117:0/411848013 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78140af5f0 0x7f781408bd60 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.390 7f7819d38700 1 -- 172.21.2.117:0/411848013 >> 172.21.2.117:0/411848013 conn(0x7f78140953a0 msgr2=0x7f781409e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:44.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.391 7f7819d38700 1 -- 172.21.2.117:0/411848013 shutdown_connections 2021-02-10T11:52:44.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.391 7f7819d38700 1 -- 172.21.2.117:0/411848013 wait complete. 2021-02-10T11:52:44.400 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:52:44.400 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:52:44.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 Processor -- start 2021-02-10T11:52:44.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 -- start start 2021-02-10T11:52:44.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 0x7f4e780eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780ede60 con 0x7f4e780eb7f0 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780d1ce0 con 0x7f4e780d18e0 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.811 7f4e800f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780ef7f0 con 0x7f4e780ef3f0 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7e714700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4e780eb7f0 0x7f4e781b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56852/0 (socket says 172.21.2.117:56852) 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7e714700 1 -- 172.21.2.117:0/3384292085 learned_addr learned my addr 172.21.2.117:0/3384292085 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7df13700 1 --1- 172.21.2.117:0/3384292085 >> v1:172.21.2.125:6789/0 conn(0x7f4e780d18e0 0x7f4e780ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:49774/0 (socket says 172.21.2.117:49774) 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7d712700 1 --2- 172.21.2.117:0/3384292085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 0x7f4e780eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7d712700 1 -- 172.21.2.117:0/3384292085 >> v1:172.21.2.125:6789/0 conn(0x7f4e780d18e0 legacy=0x7f4e780ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7d712700 1 -- 172.21.2.117:0/3384292085 >> v1:172.21.2.117:6789/0 conn(0x7f4e780eb7f0 legacy=0x7f4e781b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7cf11700 1 -- 172.21.2.117:0/3384292085 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152435608 0 0) 0x7f4e68000c60 con 0x7f4e780d18e0 2021-02-10T11:52:44.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7d712700 1 -- 172.21.2.117:0/3384292085 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780d34b0 con 0x7f4e780ef3f0 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.812 7f4e7d712700 1 --2- 172.21.2.117:0/3384292085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 0x7f4e780eb1d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4e700069d0 tx=0x7f4e70009e40).ready entity=mon.2 client_cookie=80fbfa6b39fd73c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e7cf11700 1 -- 172.21.2.117:0/3384292085 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000b510 con 0x7f4e780ef3f0 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e7cf11700 1 -- 172.21.2.117:0/3384292085 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000b890 con 0x7f4e780ef3f0 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e7cf11700 1 -- 172.21.2.117:0/3384292085 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7000bdb0 con 0x7f4e780ef3f0 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 -- 172.21.2.117:0/3384292085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 msgr2=0x7f4e780eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 --2- 172.21.2.117:0/3384292085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 0x7f4e780eb1d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4e700069d0 tx=0x7f4e70009e40).stop 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 -- 172.21.2.117:0/3384292085 shutdown_connections 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 --2- 172.21.2.117:0/3384292085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780ef3f0 0x7f4e780eb1d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 -- 172.21.2.117:0/3384292085 >> 172.21.2.117:0/3384292085 conn(0x7f4e780ced60 msgr2=0x7f4e780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.813 7f4e800f7700 1 -- 172.21.2.117:0/3384292085 shutdown_connections 2021-02-10T11:52:44.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.814 7f4e800f7700 1 -- 172.21.2.117:0/3384292085 wait complete. 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.828 7f4e800f7700 1 Processor -- start 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.828 7f4e800f7700 1 -- start start 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e781aa550 0x7f4e781a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e781a3bf0 0x7f4e781a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781a9340 con 0x7f4e781a3bf0 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781aa990 con 0x7f4e781aa550 2021-02-10T11:52:44.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e800f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781a9e60 con 0x7f4e780df870 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54286/0 (socket says 172.21.2.117:54286) 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 -- 172.21.2.117:0/2145955774 learned_addr learned my addr 172.21.2.117:0/2145955774 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7d712700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e781a3bf0 0x7f4e781a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7df13700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e781aa550 0x7f4e781a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 -- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e781aa550 msgr2=0x7f4e781a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e781aa550 0x7f4e781a9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 -- 172.21.2.117:0/2145955774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e781a3bf0 msgr2=0x7f4e781a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e781a3bf0 0x7f4e781a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.829 7f4e7e714700 1 -- 172.21.2.117:0/2145955774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780e0290 con 0x7f4e780df870 2021-02-10T11:52:44.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e7e714700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4e7400a590 tx=0x7f4e7400a930).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e7400ad20 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e740043a0 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4e74004880 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e780e0290 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.830 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e780e0290 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.831 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4e74004790 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.831 7f4e6e7fc700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 0x7f4e6001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.831 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4e74032640 con 0x7f4e780df870 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.831 7f4e7df13700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 0x7f4e6001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:44.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.831 7f4e7df13700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 0x7f4e6001ac10 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:44.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.861 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e78087290 con 0x7f4e780df870 2021-02-10T11:52:44.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.865 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4e740043a0 con 0x7f4e780df870 2021-02-10T11:52:45.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:44.999 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4e781a92c0 con 0x7f4e780df870 2021-02-10T11:52:45.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.000 7f4e6e7fc700 1 -- 172.21.2.117:0/2145955774 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v80) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f4e7400ad20 con 0x7f4e780df870 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.003 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:52:45.004 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:52:45.005 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.005 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.005 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.005 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:52:45.007 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:52:45.009 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:52:45.011 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.014 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.015 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:52:45.016 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:52:45.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 msgr2=0x7f4e6001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 0x7f4e6001ac10 crc :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 msgr2=0x7f4e780dfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4e7400a590 tx=0x7f4e7400a930).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 shutdown_connections 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4e60018730 0x7f4e6001ac10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4e781a3bf0 0x7f4e781a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4e780df870 0x7f4e780dfc70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 --2- 172.21.2.117:0/2145955774 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4e781aa550 0x7f4e781a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 >> 172.21.2.117:0/2145955774 conn(0x7f4e781a3620 msgr2=0x7f4e780e0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:45.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 shutdown_connections 2021-02-10T11:52:45.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.002 7f4e800f7700 1 -- 172.21.2.117:0/2145955774 wait complete. 2021-02-10T11:52:45.019 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:45.413 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:52:45.415 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:52:45.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 Processor -- start 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 -- start start 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc0ede60 con 0x7f4adc0eb000 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc0ef7f0 con 0x7f4adc0d18e0 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.430 7f4ae3a9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0d1ce0 con 0x7f4adc0ef3f0 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae10ba700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4adc0eb000 0x7f4adc0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56862/0 (socket says 172.21.2.117:56862) 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae10ba700 1 -- 172.21.2.117:0/2085655989 learned_addr learned my addr 172.21.2.117:0/2085655989 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae18bb700 1 --2- 172.21.2.117:0/2085655989 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 --2- 172.21.2.117:0/2085655989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae08b9700 1 -- 172.21.2.117:0/2085655989 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498914757 0 0) 0x7f4acc000ce0 con 0x7f4adc0eb000 2021-02-10T11:52:45.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae08b9700 1 -- 172.21.2.117:0/2085655989 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac8002fe0 con 0x7f4adc0eb000 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 -- 172.21.2.117:0/2085655989 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 msgr2=0x7f4adc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 --2- 172.21.2.117:0/2085655989 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 -- 172.21.2.117:0/2085655989 >> v1:172.21.2.117:6789/0 conn(0x7f4adc0eb000 legacy=0x7f4adc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 -- 172.21.2.117:0/2085655989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc1b8f90 con 0x7f4adc0ef3f0 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae18bb700 1 --2- 172.21.2.117:0/2085655989 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.431 7f4ae20bc700 1 --2- 172.21.2.117:0/2085655989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc1b7960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4ad4006e50 tx=0x7f4ad4009e70).ready entity=mon.2 client_cookie=95e0dbace36b6c4a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae08b9700 1 -- 172.21.2.117:0/2085655989 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad4006a20 con 0x7f4adc0ef3f0 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae08b9700 1 -- 172.21.2.117:0/2085655989 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad4006790 con 0x7f4adc0ef3f0 2021-02-10T11:52:45.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae08b9700 1 -- 172.21.2.117:0/2085655989 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad400b700 con 0x7f4adc0ef3f0 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 -- 172.21.2.117:0/2085655989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 msgr2=0x7f4adc1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 --2- 172.21.2.117:0/2085655989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc1b7960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4ad4006e50 tx=0x7f4ad4009e70).stop 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 -- 172.21.2.117:0/2085655989 shutdown_connections 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 --2- 172.21.2.117:0/2085655989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0ef3f0 0x7f4adc1b7960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 --2- 172.21.2.117:0/2085655989 >> v2:172.21.2.117:3300/0 conn(0x7f4adc0d18e0 0x7f4adc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.432 7f4ae3a9f700 1 -- 172.21.2.117:0/2085655989 >> 172.21.2.117:0/2085655989 conn(0x7f4adc0ced60 msgr2=0x7f4adc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.433 7f4ae3a9f700 1 -- 172.21.2.117:0/2085655989 shutdown_connections 2021-02-10T11:52:45.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.433 7f4ae3a9f700 1 -- 172.21.2.117:0/2085655989 wait complete. 2021-02-10T11:52:45.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.447 7f4ae3a9f700 1 Processor -- start 2021-02-10T11:52:45.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 -- start start 2021-02-10T11:52:45.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 0x7f4adc1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0b8d10 0x7f4adc0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0c0670 con 0x7f4adc0b82f0 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0deb20 con 0x7f4adc1a33d0 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae3a9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0ded40 con 0x7f4adc0b8d10 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae10ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae10ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56620/0 (socket says 172.21.2.117:56620) 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae10ba700 1 -- 172.21.2.117:0/48777402 learned_addr learned my addr 172.21.2.117:0/48777402 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae20bc700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0b8d10 0x7f4adc0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.448 7f4ae18bb700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 0x7f4adc1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 -- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0b8d10 msgr2=0x7f4adc0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0b8d10 0x7f4adc0b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 -- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 msgr2=0x7f4adc1a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 0x7f4adc1a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 -- 172.21.2.117:0/48777402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc0def60 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae18bb700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 0x7f4adc1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae10ba700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7f4acc0014f0 tx=0x7f4acc00a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4acc00ace0 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc0043a0 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4acc004840 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc0def60 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.449 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc0def60 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.450 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4acc000ed0 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.450 7f4ad27fc700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 0x7f4ac401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.450 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f4acc032380 con 0x7f4adc0b82f0 2021-02-10T11:52:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.450 7f4ae18bb700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 0x7f4ac401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.450 7f4ae18bb700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 0x7f4ac401abc0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:45.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.479 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc087290 con 0x7f4adc0b82f0 2021-02-10T11:52:45.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.482 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4acc000c90 con 0x7f4adc0b82f0 2021-02-10T11:52:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.608 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4adc0ede60 con 0x7f4adc0b82f0 2021-02-10T11:52:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.608 7f4ad27fc700 1 -- 172.21.2.117:0/48777402 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f4acc000ed0 con 0x7f4adc0b82f0 2021-02-10T11:52:45.610 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:45.610 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":138,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:44.351508","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":132,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 msgr2=0x7f4ac401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 0x7f4ac401abc0 crc :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 msgr2=0x7f4adc0b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 rx=0x7f4acc0014f0 tx=0x7f4acc00a8f0).stop 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 shutdown_connections 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac40186c0 0x7f4ac401abc0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4adc0b82f0 0x7f4adc0b86f0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4adc0b8d10 0x7f4adc0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 --2- 172.21.2.117:0/48777402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4adc1a33d0 0x7f4adc1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.610 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 >> 172.21.2.117:0/48777402 conn(0x7f4adc0874c0 msgr2=0x7f4adc096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.611 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 shutdown_connections 2021-02-10T11:52:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:45.611 7f4ae3a9f700 1 -- 172.21.2.117:0/48777402 wait complete. 2021-02-10T11:52:45.621 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:52:45.622 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.072 7ff29b299700 1 Processor -- start 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 -- start start 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 0x7ff2940eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff2940eb040 0x7ff2940eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2940ede60 con 0x7ff2940ee3e0 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940eba60 con 0x7ff2940eb040 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff29b299700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940d1cf0 con 0x7ff2940d18f0 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff2998b6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff2940ee3e0 0x7ff2940ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56876/0 (socket says 172.21.2.117:56876) 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff2998b6700 1 -- 172.21.2.117:0/174372242 learned_addr learned my addr 172.21.2.117:0/174372242 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:46.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.073 7ff2988b4700 1 --2- 172.21.2.117:0/174372242 >> v2:172.21.2.117:3300/0 conn(0x7ff2940eb040 0x7ff2940eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff287fff700 1 -- 172.21.2.117:0/174372242 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732716202 0 0) 0x7ff28c001500 con 0x7ff2940ee3e0 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff287fff700 1 -- 172.21.2.117:0/174372242 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280002fe0 con 0x7ff2940ee3e0 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2988b4700 1 -- 172.21.2.117:0/174372242 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 msgr2=0x7ff2940eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2990b5700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 0x7ff2940eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2988b4700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 0x7ff2940eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2988b4700 1 -- 172.21.2.117:0/174372242 >> v1:172.21.2.117:6789/0 conn(0x7ff2940ee3e0 legacy=0x7ff2940ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2988b4700 1 -- 172.21.2.117:0/174372242 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940eefd0 con 0x7ff2940eb040 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2990b5700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 0x7ff2940eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff2988b4700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2940eb040 0x7ff2940eb440 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 rx=0x7ff290006650 tx=0x7ff290009a80).ready entity=mon.0 client_cookie=f14ea20d7dab8a0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff287fff700 1 -- 172.21.2.117:0/174372242 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29000c0c0 con 0x7ff2940eb040 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff287fff700 1 -- 172.21.2.117:0/174372242 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29000c340 con 0x7ff2940eb040 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff287fff700 1 -- 172.21.2.117:0/174372242 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff29000c860 con 0x7ff2940eb040 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff29b299700 1 -- 172.21.2.117:0/174372242 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2940eb040 msgr2=0x7ff2940eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.074 7ff29b299700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2940eb040 0x7ff2940eb440 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 rx=0x7ff290006650 tx=0x7ff290009a80).stop 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 -- 172.21.2.117:0/174372242 shutdown_connections 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2940eb040 0x7ff2940eb440 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 --2- 172.21.2.117:0/174372242 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff2940d18f0 0x7ff2940eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 -- 172.21.2.117:0/174372242 >> 172.21.2.117:0/174372242 conn(0x7ff2940cf060 msgr2=0x7ff2940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 -- 172.21.2.117:0/174372242 shutdown_connections 2021-02-10T11:52:46.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.075 7ff29b299700 1 -- 172.21.2.117:0/174372242 wait complete. 2021-02-10T11:52:46.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 Processor -- start 2021-02-10T11:52:46.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 -- start start 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff29409ed70 0x7ff29409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff29409f790 0x7ff2940b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29409fb90 con 0x7ff2941a4240 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940dc640 con 0x7ff29409ed70 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.090 7ff29b299700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff294097270 con 0x7ff29409f790 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56630/0 (socket says 172.21.2.117:56630) 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 -- 172.21.2.117:0/2787073874 learned_addr learned my addr 172.21.2.117:0/2787073874 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 -- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff29409f790 msgr2=0x7ff2940b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff29409f790 0x7ff2940b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 -- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff29409ed70 msgr2=0x7ff29409f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff29409ed70 0x7ff29409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 -- 172.21.2.117:0/2787073874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940b8780 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff2990b5700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7ff288006f20 tx=0x7ff28800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff2880009b0 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28800bd80 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff29b299700 1 -- 172.21.2.117:0/2787073874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2940b8780 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff288002220 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.091 7ff29b299700 1 -- 172.21.2.117:0/2787073874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2940b8780 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.092 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff28800a000 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.092 7ff285ffb700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 0x7ff27401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.092 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5100+0+0 (secure 0 0 0) 0x7ff288032420 con 0x7ff2941a4240 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.092 7ff2988b4700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 0x7ff27401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.093 7ff2988b4700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 0x7ff27401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.123 7ff29b299700 1 -- 172.21.2.117:0/2787073874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29409fb90 con 0x7ff2941a4240 2021-02-10T11:52:46.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.126 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff288006bb0 con 0x7ff2941a4240 2021-02-10T11:52:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.253 7ff29b299700 1 -- 172.21.2.117:0/2787073874 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff29409fb90 con 0x7ff2941a4240 2021-02-10T11:52:46.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.254 7ff285ffb700 1 -- 172.21.2.117:0/2787073874 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v138) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff28800bd80 con 0x7ff2941a4240 2021-02-10T11:52:46.256 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:46.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 msgr2=0x7ff27401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 0x7ff27401abc0 crc :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 msgr2=0x7ff2941a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 rx=0x7ff288006f20 tx=0x7ff28800b990).stop 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 shutdown_connections 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff29409f790 0x7ff2940b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff29409ed70 0x7ff29409f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff274018730 0x7ff27401abc0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 --2- 172.21.2.117:0/2787073874 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff2941a4240 0x7ff2941a4640 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 >> 172.21.2.117:0/2787073874 conn(0x7ff294087090 msgr2=0x7ff29408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:46.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 shutdown_connections 2021-02-10T11:52:46.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.256 7ff29b299700 1 -- 172.21.2.117:0/2787073874 wait complete. 2021-02-10T11:52:46.266 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:52:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.677 7f653015b700 1 Processor -- start 2021-02-10T11:52:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.677 7f653015b700 1 -- start start 2021-02-10T11:52:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f653015b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f65280d18e0 0x7f65280ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f653015b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280ef3f0 0x7f65281b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f653015b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65280ede60 con 0x7f65280eb000 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f653015b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65280ef7f0 con 0x7f65280d18e0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f653015b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65280d1ce0 con 0x7f65280ef3f0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f65280d18e0 0x7f65280ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f65280d18e0 0x7f65280ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56638/0 (socket says 172.21.2.117:56638) 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 -- 172.21.2.117:0/2171310081 learned_addr learned my addr 172.21.2.117:0/2171310081 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652e778700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280ef3f0 0x7f65281b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 -- 172.21.2.117:0/2171310081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280ef3f0 msgr2=0x7f65281b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280ef3f0 0x7f65281b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 -- 172.21.2.117:0/2171310081 >> v1:172.21.2.125:6789/0 conn(0x7f65280eb000 legacy=0x7f65280eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 -- 172.21.2.117:0/2171310081 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65281b8f90 con 0x7f65280d18e0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.678 7f652df77700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280d18e0 0x7f65280ea9e0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f6524006700 tx=0x7f6524009b80).ready entity=mon.0 client_cookie=c26efdcefffbb00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f652cf75700 1 -- 172.21.2.117:0/2171310081 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f652400c0c0 con 0x7f65280d18e0 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f652cf75700 1 -- 172.21.2.117:0/2171310081 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652400c440 con 0x7f65280d18e0 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f652cf75700 1 -- 172.21.2.117:0/2171310081 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f652400c960 con 0x7f65280d18e0 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 -- 172.21.2.117:0/2171310081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280d18e0 msgr2=0x7f65280ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280d18e0 0x7f65280ea9e0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 rx=0x7f6524006700 tx=0x7f6524009b80).stop 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 -- 172.21.2.117:0/2171310081 shutdown_connections 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280ef3f0 0x7f65281b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 --2- 172.21.2.117:0/2171310081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280d18e0 0x7f65280ea9e0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.679 7f653015b700 1 -- 172.21.2.117:0/2171310081 >> 172.21.2.117:0/2171310081 conn(0x7f65280ced60 msgr2=0x7f65280cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.680 7f653015b700 1 -- 172.21.2.117:0/2171310081 shutdown_connections 2021-02-10T11:52:46.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.680 7f653015b700 1 -- 172.21.2.117:0/2171310081 wait complete. 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.694 7f653015b700 1 Processor -- start 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.694 7f653015b700 1 -- start start 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65281a33d0 0x7f65281a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280b8d10 0x7f65280b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65280c0670 con 0x7f65280b82f0 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65280deb20 con 0x7f65281a33d0 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65280ded40 con 0x7f65280b8d10 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56640/0 (socket says 172.21.2.117:56640) 2021-02-10T11:52:46.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 -- 172.21.2.117:0/197334439 learned_addr learned my addr 172.21.2.117:0/197334439 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 -- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280b8d10 msgr2=0x7f65280b9110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280b8d10 0x7f65280b9110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 -- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65281a33d0 msgr2=0x7f65281a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65281a33d0 0x7f65281a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 -- 172.21.2.117:0/197334439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65280def60 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f652d776700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f651c006ed0 tx=0x7f651c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f651c006240 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651c0054b0 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f651c005990 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.695 7f653015b700 1 -- 172.21.2.117:0/197334439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65280def60 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.696 7f653015b700 1 -- 172.21.2.117:0/197334439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65280def60 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.696 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f651c006030 con 0x7f65280b82f0 2021-02-10T11:52:46.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.696 7f651a7fc700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 0x7f651001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:46.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.696 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f651c032200 con 0x7f65280b82f0 2021-02-10T11:52:46.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.697 7f652df77700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 0x7f651001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:46.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.697 7f652df77700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 0x7f651001abc0 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:46.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.727 7f653015b700 1 -- 172.21.2.117:0/197334439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6528087290 con 0x7f65280b82f0 2021-02-10T11:52:46.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.730 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f651c000de0 con 0x7f65280b82f0 2021-02-10T11:52:46.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.854 7f653015b700 1 -- 172.21.2.117:0/197334439 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f65280ede60 con 0x7f65280b82f0 2021-02-10T11:52:46.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.854 7f651a7fc700 1 -- 172.21.2.117:0/197334439 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v138) v1 ==== 84+0+64 (secure 0 0 0) 0x7f651c006030 con 0x7f65280b82f0 2021-02-10T11:52:46.856 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:46.856 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:52:46.856 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:52:46.856 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:52:46.857 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:52:46.857 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:52:46.857 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.856 7f653015b700 1 -- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 msgr2=0x7f651001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.856 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 0x7f651001abc0 crc :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.856 7f653015b700 1 -- 172.21.2.117:0/197334439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 msgr2=0x7f65280b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.856 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 rx=0x7f651c006ed0 tx=0x7f651c00b960).stop 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 -- 172.21.2.117:0/197334439 shutdown_connections 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65100186c0 0x7f651001abc0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65280b82f0 0x7f65280b86f0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65280b8d10 0x7f65280b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 --2- 172.21.2.117:0/197334439 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65281a33d0 0x7f65281a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:46.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 -- 172.21.2.117:0/197334439 >> 172.21.2.117:0/197334439 conn(0x7f65280874c0 msgr2=0x7f6528096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:46.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 -- 172.21.2.117:0/197334439 shutdown_connections 2021-02-10T11:52:46.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:46.857 7f653015b700 1 -- 172.21.2.117:0/197334439 wait complete. 2021-02-10T11:52:46.867 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:52:46.867 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:52:47.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 Processor -- start 2021-02-10T11:52:47.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 -- start start 2021-02-10T11:52:47.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f31f00d18f0 0x7f31f00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f00ef3f0 0x7f31f01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f00ef7f0 con 0x7f31f00eb040 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00eded0 con 0x7f31f00d18f0 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31f5f1d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00d1cf0 con 0x7f31f00ef3f0 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.281 7f31eeffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f31f00eb040 0x7f31f00eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56898/0 (socket says 172.21.2.117:56898) 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31eeffd700 1 -- 172.21.2.117:0/1244425246 learned_addr learned my addr 172.21.2.117:0/1244425246 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524899911 0 0) 0x7f31e00010b0 con 0x7f31f00eb040 2021-02-10T11:52:47.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31dc002fe0 con 0x7f31f00eb040 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31effff700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f00ef3f0 0x7f31f01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ef7fe700 1 --2- 172.21.2.117:0/1244425246 >> v2:172.21.2.125:3300/0 conn(0x7f31f00d18f0 0x7f31f00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2415641920 0 0) 0x7f31e00010b0 con 0x7f31f00eb040 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f00ef3f0 msgr2=0x7f31f01b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f00ef3f0 0x7f31f01b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 >> v2:172.21.2.125:3300/0 conn(0x7f31f00d18f0 msgr2=0x7f31f00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 --2- 172.21.2.117:0/1244425246 >> v2:172.21.2.125:3300/0 conn(0x7f31f00d18f0 0x7f31f00eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f01b8fd0 con 0x7f31f00eb040 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f31e00015b0 con 0x7f31f00eb040 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 >> v1:172.21.2.117:6789/0 conn(0x7f31f00eb040 legacy=0x7f31f00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31dc003b90 0x7f31dc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31dc006840 0x7f31dc008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31dc009640 0x7f31dc00bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.282 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc050390 con 0x7f31dc006840 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0505b0 con 0x7f31dc003b90 2021-02-10T11:52:47.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0507d0 con 0x7f31dc009640 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31dc006840 0x7f31dc008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ef7fe700 1 --2- 172.21.2.117:0/1244425246 >> v2:172.21.2.125:3300/0 conn(0x7f31f00d18f0 0x7f31f00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31effff700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31dc009640 0x7f31dc00bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ef7fe700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31dc003b90 0x7f31dc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 -- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31dc009640 msgr2=0x7f31dc00bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31dc009640 0x7f31dc00bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 -- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31dc003b90 msgr2=0x7f31dc005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31dc003b90 0x7f31dc005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 -- 172.21.2.117:0/1244425246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc050a70 con 0x7f31dc006840 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31effff700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31dc009640 0x7f31dc00bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31eeffd700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31dc006840 0x7f31dc008d60 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7f31e0001270 tx=0x7f31e0007d70).ready entity=mon.0 client_cookie=b4ac29a8d3184c5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31e00009b0 con 0x7f31dc006840 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e000a520 con 0x7f31dc006840 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31ee7fc700 1 -- 172.21.2.117:0/1244425246 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31e000a9f0 con 0x7f31dc006840 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31f5f1d700 1 -- 172.21.2.117:0/1244425246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31dc006840 msgr2=0x7f31dc008d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31f5f1d700 1 --2- 172.21.2.117:0/1244425246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31dc006840 0x7f31dc008d60 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 rx=0x7f31e0001270 tx=0x7f31e0007d70).stop 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.283 7f31effff700 1 -- 172.21.2.117:0/1244425246 reap_dead start 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.284 7f31f5f1d700 1 -- 172.21.2.117:0/1244425246 shutdown_connections 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.284 7f31f5f1d700 1 -- 172.21.2.117:0/1244425246 >> 172.21.2.117:0/1244425246 conn(0x7f31f00cf060 msgr2=0x7f31f00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.284 7f31f5f1d700 1 -- 172.21.2.117:0/1244425246 shutdown_connections 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.284 7f31f5f1d700 1 -- 172.21.2.117:0/1244425246 wait complete. 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 Processor -- start 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 -- start start 2021-02-10T11:52:47.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f0094090 0x7f31f0090e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f01a7c10 0x7f31f00a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00b20e0 con 0x7f31f01a71f0 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00945f0 con 0x7f31f01a7c10 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.299 7f31f5f1d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00a1280 con 0x7f31f0094090 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56660/0 (socket says 172.21.2.117:56660) 2021-02-10T11:52:47.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 -- 172.21.2.117:0/3473304957 learned_addr learned my addr 172.21.2.117:0/3473304957 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 -- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f0094090 msgr2=0x7f31f0090e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f0094090 0x7f31f0090e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 -- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f01a7c10 msgr2=0x7f31f00a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f01a7c10 0x7f31f00a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:47.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 -- 172.21.2.117:0/3473304957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00a14a0 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31eeffd700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f31e000a5f0 tx=0x7f31e000a990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31e0001060 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e000ad80 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.300 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f31e00045d0 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.301 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f00a14a0 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.301 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f00a1a30 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.301 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f31e0004140 con 0x7f31f01a71f0 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.301 7f31e7fff700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 0x7f31d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:47.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.301 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f31e0032410 con 0x7f31f01a71f0 2021-02-10T11:52:47.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.302 7f31ef7fe700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 0x7f31d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:47.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.303 7f31ef7fe700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 0x7f31d401abc0 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:47.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.331 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00daff0 con 0x7f31f01a71f0 2021-02-10T11:52:47.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.334 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f31e00009b0 con 0x7f31f01a71f0 2021-02-10T11:52:47.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.460 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f31f00ede60 con 0x7f31f01a71f0 2021-02-10T11:52:47.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.827 7f31e7fff700 1 -- 172.21.2.117:0/3473304957 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v139) v1 ==== 133+0+0 (secure 0 0 0) 0x7f31e0004140 con 0x7f31f01a71f0 2021-02-10T11:52:47.877 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.877 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 msgr2=0x7f31d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.877 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 0x7f31d401abc0 crc :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 msgr2=0x7f31f01a75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 rx=0x7f31e000a5f0 tx=0x7f31e000a990).stop 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 shutdown_connections 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f31d40186c0 0x7f31d401abc0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31f01a71f0 0x7f31f01a75f0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31f01a7c10 0x7f31f00a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:47.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 --2- 172.21.2.117:0/3473304957 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31f0094090 0x7f31f0090e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:47.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 >> 172.21.2.117:0/3473304957 conn(0x7f31f00b8180 msgr2=0x7f31f00b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:47.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 shutdown_connections 2021-02-10T11:52:47.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:47.878 7f31f5f1d700 1 -- 172.21.2.117:0/3473304957 wait complete. 2021-02-10T11:52:47.887 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.301 7f2c08f57700 1 Processor -- start 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.301 7f2c08f57700 1 -- start start 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c08f57700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 0x7f2c040eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c08f57700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 0x7f2c041b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c08f57700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c040ef7f0 con 0x7f2c040d18f0 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c08f57700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c040eded0 con 0x7f2c040eb830 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c08f57700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c040d1cf0 con 0x7f2c040ef3f0 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c03314700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 0x7f2c041b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c02312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 0x7f2c040eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c02b13700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f2c040d18f0 0x7f2c040eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:49840/0 (socket says 172.21.2.117:49840) 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.302 7f2c02b13700 1 -- 172.21.2.117:0/844596504 learned_addr learned my addr 172.21.2.117:0/844596504 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c02312700 1 -- 172.21.2.117:0/844596504 >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 msgr2=0x7f2c041b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c02312700 1 --2- 172.21.2.117:0/844596504 >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 0x7f2c041b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c02312700 1 -- 172.21.2.117:0/844596504 >> v1:172.21.2.125:6789/0 conn(0x7f2c040d18f0 legacy=0x7f2c040eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c02312700 1 -- 172.21.2.117:0/844596504 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c041b8fd0 con 0x7f2c040ef3f0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c03314700 1 --2- 172.21.2.117:0/844596504 >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 0x7f2c041b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c02312700 1 --2- 172.21.2.117:0/844596504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 0x7f2c040eb210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2bec006650 tx=0x7f2bec009a80).ready entity=mon.2 client_cookie=a71baa94764c0507 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c01b11700 1 -- 172.21.2.117:0/844596504 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec00c0c0 con 0x7f2c040ef3f0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c01b11700 1 -- 172.21.2.117:0/844596504 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec00c340 con 0x7f2c040ef3f0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c01b11700 1 -- 172.21.2.117:0/844596504 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec00c860 con 0x7f2c040ef3f0 2021-02-10T11:52:48.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c08f57700 1 -- 172.21.2.117:0/844596504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 msgr2=0x7f2c040eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.303 7f2c08f57700 1 --2- 172.21.2.117:0/844596504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 0x7f2c040eb210 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2bec006650 tx=0x7f2bec009a80).stop 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 -- 172.21.2.117:0/844596504 shutdown_connections 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 --2- 172.21.2.117:0/844596504 >> v2:172.21.2.125:3300/0 conn(0x7f2c040eb830 0x7f2c041b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 --2- 172.21.2.117:0/844596504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c040ef3f0 0x7f2c040eb210 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 -- 172.21.2.117:0/844596504 >> 172.21.2.117:0/844596504 conn(0x7f2c040cf060 msgr2=0x7f2c040cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 -- 172.21.2.117:0/844596504 shutdown_connections 2021-02-10T11:52:48.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.304 7f2c08f57700 1 -- 172.21.2.117:0/844596504 wait complete. 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 Processor -- start 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 -- start start 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 0x7f2c0409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c0409f790 0x7f2c040b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0409fb90 con 0x7f2c041a4240 2021-02-10T11:52:48.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c040dc640 con 0x7f2c0409ed70 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.319 7f2c08f57700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04097270 con 0x7f2c0409f790 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 0x7f2c0409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56672/0 (socket says 172.21.2.117:56672) 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02b13700 1 -- 172.21.2.117:0/2567614857 learned_addr learned my addr 172.21.2.117:0/2567614857 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c03314700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c0409f790 0x7f2c040b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 -- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c0409f790 msgr2=0x7f2c040b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c0409f790 0x7f2c040b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 -- 172.21.2.117:0/2567614857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 msgr2=0x7f2c041a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 -- 172.21.2.117:0/2567614857 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c040b8780 con 0x7f2c0409ed70 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02b13700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:48.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2c02312700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 0x7f2c0409f170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f2bec00b530 tx=0x7f2bec00b8d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec0091c0 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.320 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec002030 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c040b8780 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bec0024c0 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c040b8780 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bec0091c0 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2bfb7fe700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 0x7f2be801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f2bec032560 con 0x7f2c0409ed70 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.321 7f2c02b13700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 0x7f2be801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:48.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.322 7f2c02b13700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 0x7f2be801ac10 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:48.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.351 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0409fb90 con 0x7f2c0409ed70 2021-02-10T11:52:48.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.356 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bec000c90 con 0x7f2c0409ed70 2021-02-10T11:52:48.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.487 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f2c0409fb90 con 0x7f2c0409ed70 2021-02-10T11:52:48.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.836 7f2bfb7fe700 1 -- 172.21.2.117:0/2567614857 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v141) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2bec0091c0 con 0x7f2c0409ed70 2021-02-10T11:52:48.854 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.854 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 msgr2=0x7f2be801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.854 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 0x7f2be801ac10 crc :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.854 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 msgr2=0x7f2c0409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.854 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 0x7f2c0409f170 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 rx=0x7f2bec00b530 tx=0x7f2bec00b8d0).stop 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 shutdown_connections 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2c0409f790 0x7f2c040b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2c0409ed70 0x7f2c0409f170 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2be8018730 0x7f2be801ac10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 --2- 172.21.2.117:0/2567614857 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2c041a4240 0x7f2c041a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:48.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 >> 172.21.2.117:0/2567614857 conn(0x7f2c04087090 msgr2=0x7f2c0408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:48.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 shutdown_connections 2021-02-10T11:52:48.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:48.855 7f2c08f57700 1 -- 172.21.2.117:0/2567614857 wait complete. 2021-02-10T11:52:48.865 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:52:49.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 Processor -- start 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 -- start start 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7cd80e4e40 0x7f7cd81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd8085450 0x7f7cd8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8085930 con 0x7f7cd80e5240 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd80e5cc0 con 0x7f7cd80e4e40 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.279 7f7cdf25d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8084840 con 0x7f7cd8085450 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7cd80e4e40 0x7f7cd81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdc878700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7cd80e5240 0x7f7cd8084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56930/0 (socket says 172.21.2.117:56930) 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdc878700 1 -- 172.21.2.117:0/36486723 learned_addr learned my addr 172.21.2.117:0/36486723 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 -- 172.21.2.117:0/36486723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd8085450 msgr2=0x7f7cd8086ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 --2- 172.21.2.117:0/36486723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd8085450 0x7f7cd8086ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 -- 172.21.2.117:0/36486723 >> v1:172.21.2.117:6789/0 conn(0x7f7cd80e5240 legacy=0x7f7cd8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 -- 172.21.2.117:0/36486723 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd81b0810 con 0x7f7cd80e4e40 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdd079700 1 --2- 172.21.2.117:0/36486723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd80e4e40 0x7f7cd81b01f0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f7ccc006650 tx=0x7f7ccc009a80).ready entity=mon.0 client_cookie=68a8fe1cbae3bd56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7ccbfff700 1 -- 172.21.2.117:0/36486723 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc00c0c0 con 0x7f7cd80e4e40 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7ccbfff700 1 -- 172.21.2.117:0/36486723 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc00c340 con 0x7f7cd80e4e40 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7ccbfff700 1 -- 172.21.2.117:0/36486723 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc00c860 con 0x7f7cd80e4e40 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdf25d700 1 -- 172.21.2.117:0/36486723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd80e4e40 msgr2=0x7f7cd81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.280 7f7cdf25d700 1 --2- 172.21.2.117:0/36486723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd80e4e40 0x7f7cd81b01f0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 rx=0x7f7ccc006650 tx=0x7f7ccc009a80).stop 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 -- 172.21.2.117:0/36486723 shutdown_connections 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 --2- 172.21.2.117:0/36486723 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd8085450 0x7f7cd8086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 --2- 172.21.2.117:0/36486723 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd80e4e40 0x7f7cd81b01f0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 -- 172.21.2.117:0/36486723 >> 172.21.2.117:0/36486723 conn(0x7f7cd80ce4e0 msgr2=0x7f7cd80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:49.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 -- 172.21.2.117:0/36486723 shutdown_connections 2021-02-10T11:52:49.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.281 7f7cdf25d700 1 -- 172.21.2.117:0/36486723 wait complete. 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.295 7f7cdf25d700 1 Processor -- start 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.295 7f7cdf25d700 1 -- start start 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd81b1f90 0x7f7cd81b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd8094390 0x7f7cd8094790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81110e0 con 0x7f7cd81b1570 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd81b0ff0 con 0x7f7cd8094390 2021-02-10T11:52:49.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdf25d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8094db0 con 0x7f7cd81b1f90 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56686/0 (socket says 172.21.2.117:56686) 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 -- 172.21.2.117:0/3386678966 learned_addr learned my addr 172.21.2.117:0/3386678966 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 -- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd81b1f90 msgr2=0x7f7cd81b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd81b1f90 0x7f7cd81b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 -- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd8094390 msgr2=0x7f7cd8094790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.296 7f7cdd079700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd8094390 0x7f7cd8094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cdd079700 1 -- 172.21.2.117:0/3386678966 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8094f90 con 0x7f7cd81b1570 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cdd079700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 rx=0x7f7ccc009630 tx=0x7f7ccc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc0093a0 con 0x7f7cd81b1570 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc00bd70 con 0x7f7cd81b1570 2021-02-10T11:52:49.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd8094f90 con 0x7f7cd81b1570 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd8095520 con 0x7f7cd81b1570 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.297 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7ccc0054d0 con 0x7f7cd81b1570 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.298 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7ccc00a000 con 0x7f7cd81b1570 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.298 7f7cc9ffb700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.298 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f7ccc032700 con 0x7f7cd81b1570 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.298 7f7cdc878700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.299 7f7cdc878700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.329 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd810c2f0 con 0x7f7cd81b1570 2021-02-10T11:52:49.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.332 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7ccc009340 con 0x7f7cd81b1570 2021-02-10T11:52:49.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.461 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f7cd8085930 con 0x7f7cd81b1570 2021-02-10T11:52:49.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.465 7f7cc9ffb700 1 -- 172.21.2.117:0/3386678966 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 30 and data pool 31 v117) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7ccc01f540 con 0x7f7cd81b1570 2021-02-10T11:52:49.469 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 30 and data pool 31 2021-02-10T11:52:49.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 msgr2=0x7f7cb801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 0x7f7cb801abc0 crc :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 msgr2=0x7f7cd81b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 rx=0x7f7ccc009630 tx=0x7f7ccc00b980).stop 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 shutdown_connections 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cd8094390 0x7f7cd8094790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cd81b1f90 0x7f7cd81b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7cb80186c0 0x7f7cb801abc0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 --2- 172.21.2.117:0/3386678966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cd81b1570 0x7f7cd81b1970 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 >> 172.21.2.117:0/3386678966 conn(0x7f7cd81b3150 msgr2=0x7f7cd81ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:49.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 shutdown_connections 2021-02-10T11:52:49.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.470 7f7cdf25d700 1 -- 172.21.2.117:0/3386678966 wait complete. 2021-02-10T11:52:49.479 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 Processor -- start 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 -- start start 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ef3f0 0x7fa8180eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8180eb7f0 0x7fa8181b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180ede60 con 0x7fa8180d18e0 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.888 7fa8207b5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180ef7f0 con 0x7fa8180eb7f0 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa8207b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180d1ce0 con 0x7fa8180ef3f0 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8180eb7f0 0x7fa8181b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8180eb7f0 0x7fa8181b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56694/0 (socket says 172.21.2.117:56694) 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 -- 172.21.2.117:0/899094460 learned_addr learned my addr 172.21.2.117:0/899094460 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81ddd0700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ef3f0 0x7fa8180eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 -- 172.21.2.117:0/899094460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ef3f0 msgr2=0x7fa8180eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ef3f0 0x7fa8180eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 -- 172.21.2.117:0/899094460 >> v1:172.21.2.125:6789/0 conn(0x7fa8180d18e0 legacy=0x7fa8180ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 -- 172.21.2.117:0/899094460 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8181b8f90 con 0x7fa8180eb7f0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.889 7fa81edd2700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8180eb7f0 0x7fa8181b7960 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7fa810006b80 tx=0x7fa810009b90).ready entity=mon.0 client_cookie=c330f36a78b976b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa81d5cf700 1 -- 172.21.2.117:0/899094460 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8100022e0 con 0x7fa8180eb7f0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa81d5cf700 1 -- 172.21.2.117:0/899094460 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810002620 con 0x7fa8180eb7f0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa81d5cf700 1 -- 172.21.2.117:0/899094460 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa810002a80 con 0x7fa8180eb7f0 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 -- 172.21.2.117:0/899094460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8180eb7f0 msgr2=0x7fa8181b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8180eb7f0 0x7fa8181b7960 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 rx=0x7fa810006b80 tx=0x7fa810009b90).stop 2021-02-10T11:52:49.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 -- 172.21.2.117:0/899094460 shutdown_connections 2021-02-10T11:52:49.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8180eb7f0 0x7fa8181b7960 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 --2- 172.21.2.117:0/899094460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ef3f0 0x7fa8180eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 -- 172.21.2.117:0/899094460 >> 172.21.2.117:0/899094460 conn(0x7fa8180ced60 msgr2=0x7fa8180cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:49.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 -- 172.21.2.117:0/899094460 shutdown_connections 2021-02-10T11:52:49.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.890 7fa8207b5700 1 -- 172.21.2.117:0/899094460 wait complete. 2021-02-10T11:52:49.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 Processor -- start 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 -- start start 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ab280 0x7fa8180ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 0x7fa8180a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180a7d10 con 0x7fa8181a4290 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8180a7a00 con 0x7fa8180a6fe0 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa8207b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa818096370 con 0x7fa8180ab280 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa81e5d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa81e5d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56698/0 (socket says 172.21.2.117:56698) 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.905 7fa81e5d1700 1 -- 172.21.2.117:0/1333284060 learned_addr learned my addr 172.21.2.117:0/1333284060 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81ddd0700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ab280 0x7fa8180ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 -- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ab280 msgr2=0x7fa8180ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ab280 0x7fa8180ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 -- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 msgr2=0x7fa8180a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81edd2700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 0x7fa8180a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 0x7fa8180a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 -- 172.21.2.117:0/1333284060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180a8290 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81edd2700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 0x7fa8180a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.906 7fa81e5d1700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7fa80c00b8e0 tx=0x7fa80c00bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80c000ce0 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80c00c2f0 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa80c00c810 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8180a8290 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8180a8290 con 0x7fa8181a4290 2021-02-10T11:52:49.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa80c001230 con 0x7fa8181a4290 2021-02-10T11:52:49.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 0x7fa80001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:49.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.907 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa80c032ce0 con 0x7fa8181a4290 2021-02-10T11:52:49.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.908 7fa81ddd0700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 0x7fa80001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:49.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.908 7fa81ddd0700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 0x7fa80001abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:49.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.939 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa818087290 con 0x7fa8181a4290 2021-02-10T11:52:49.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:49.941 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa80c00bdc0 con 0x7fa8181a4290 2021-02-10T11:52:50.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.064 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8180ede60 con 0x7fa8181a4290 2021-02-10T11:52:50.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.065 7fa80affd700 1 -- 172.21.2.117:0/1333284060 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+14253 (secure 0 0 0) 0x7fa80c01f8d0 con 0x7fa8181a4290 2021-02-10T11:52:50.067 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:50.067 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":142,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:49.462039","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:47.477707","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:48.506496","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:50.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 msgr2=0x7fa80001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 0x7fa80001abc0 crc :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 msgr2=0x7fa8181a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 rx=0x7fa80c00b8e0 tx=0x7fa80c00bc80).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 shutdown_connections 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8180a6fe0 0x7fa8180a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180ab280 0x7fa8180ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa800018730 0x7fa80001abc0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 --2- 172.21.2.117:0/1333284060 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8181a4290 0x7fa8181a4690 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 >> 172.21.2.117:0/1333284060 conn(0x7fa8180874c0 msgr2=0x7fa818096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 shutdown_connections 2021-02-10T11:52:50.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.067 7fa8207b5700 1 -- 172.21.2.117:0/1333284060 wait complete. 2021-02-10T11:52:50.077 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 Processor -- start 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 -- start start 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdacc0ef3f0 0x7fdacc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 0x7fdacc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0ede60 con 0x7fdacc0d18e0 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc0ef7f0 con 0x7fdacc0ef3f0 2021-02-10T11:52:50.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.486 7fdad3d05700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0d34b0 con 0x7fdacc0eb7f0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1b21700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fdacc0d18e0 0x7fdacc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:49882/0 (socket says 172.21.2.117:49882) 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1b21700 1 -- 172.21.2.117:0/432582966 learned_addr learned my addr 172.21.2.117:0/432582966 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 --2- 172.21.2.117:0/432582966 >> v2:172.21.2.117:3300/0 conn(0x7fdacc0ef3f0 0x7fdacc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad2322700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 0x7fdacc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad0b1f700 1 -- 172.21.2.117:0/432582966 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158681139 0 0) 0x7fdac8001440 con 0x7fdacc0d18e0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad0b1f700 1 -- 172.21.2.117:0/432582966 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8002fe0 con 0x7fdacc0d18e0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 -- 172.21.2.117:0/432582966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 msgr2=0x7fdacc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 0x7fdacc1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 -- 172.21.2.117:0/432582966 >> v1:172.21.2.125:6789/0 conn(0x7fdacc0d18e0 legacy=0x7fdacc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 -- 172.21.2.117:0/432582966 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0d1ce0 con 0x7fdacc0ef3f0 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad2322700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 0x7fdacc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.487 7fdad1320700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0ef3f0 0x7fdacc0eb1d0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7fdabc006650 tx=0x7fdabc009a80).ready entity=mon.0 client_cookie=318ccb785ed23075 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad0b1f700 1 -- 172.21.2.117:0/432582966 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdabc0009b0 con 0x7fdacc0ef3f0 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad0b1f700 1 -- 172.21.2.117:0/432582966 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc00c0c0 con 0x7fdacc0ef3f0 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad0b1f700 1 -- 172.21.2.117:0/432582966 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdabc00c520 con 0x7fdacc0ef3f0 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 -- 172.21.2.117:0/432582966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0ef3f0 msgr2=0x7fdacc0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0ef3f0 0x7fdacc0eb1d0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 rx=0x7fdabc006650 tx=0x7fdabc009a80).stop 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 -- 172.21.2.117:0/432582966 shutdown_connections 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc0eb7f0 0x7fdacc1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 --2- 172.21.2.117:0/432582966 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0ef3f0 0x7fdacc0eb1d0 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 -- 172.21.2.117:0/432582966 >> 172.21.2.117:0/432582966 conn(0x7fdacc0ced60 msgr2=0x7fdacc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 -- 172.21.2.117:0/432582966 shutdown_connections 2021-02-10T11:52:50.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.488 7fdad3d05700 1 -- 172.21.2.117:0/432582966 wait complete. 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 Processor -- start 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 -- start start 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc1a4290 0x7fdacc1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdacc0ab280 0x7fdacc0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0a7d10 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0a7a00 con 0x7fdacc0ab280 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.503 7fdad3d05700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc096370 con 0x7fdacc1a4290 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56712/0 (socket says 172.21.2.117:56712) 2021-02-10T11:52:50.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 -- 172.21.2.117:0/3912798090 learned_addr learned my addr 172.21.2.117:0/3912798090 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 -- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc1a4290 msgr2=0x7fdacc1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc1a4290 0x7fdacc1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 -- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdacc0ab280 msgr2=0x7fdacc0ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdacc0ab280 0x7fdacc0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 -- 172.21.2.117:0/3912798090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0a8290 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad2322700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7fdac400b5e0 tx=0x7fdac400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac4006370 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac400bd70 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdac4000c10 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc0a8290 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.504 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc0a8290 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.505 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdac4005330 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.505 7fdac27fc700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 0x7fdab401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:50.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.505 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdac4032550 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.505 7fdad1b21700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 0x7fdab401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:50.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.505 7fdad1b21700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 0x7fdab401abc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:50.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.536 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc087290 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.539 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdac4000c10 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.665 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdacc087290 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.666 7fdac27fc700 1 -- 172.21.2.117:0/3912798090 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fdac4005330 con 0x7fdacc0a6fe0 2021-02-10T11:52:50.668 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:50.668 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":143,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:50.464421","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:47.477707","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:48.506496","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:50.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 msgr2=0x7fdab401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 0x7fdab401abc0 crc :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 msgr2=0x7fdacc0a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 rx=0x7fdac400b5e0 tx=0x7fdac400b980).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 shutdown_connections 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdab40186c0 0x7fdab401abc0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdacc0a6fe0 0x7fdacc0a73e0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdacc0ab280 0x7fdacc0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 --2- 172.21.2.117:0/3912798090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdacc1a4290 0x7fdacc1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 >> 172.21.2.117:0/3912798090 conn(0x7fdacc0874c0 msgr2=0x7fdacc096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 shutdown_connections 2021-02-10T11:52:50.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:50.668 7fdad3d05700 1 -- 172.21.2.117:0/3912798090 wait complete. 2021-02-10T11:52:50.678 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:52:51.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.091 7f7b6bb0b700 1 Processor -- start 2021-02-10T11:52:51.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.091 7f7b6bb0b700 1 -- start start 2021-02-10T11:52:51.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6bb0b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b640d18f0 0x7f7b640eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6bb0b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7b640eb040 0x7f7b640eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6bb0b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b640ede60 con 0x7f7b640ee3e0 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6bb0b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b640eba60 con 0x7f7b640eb040 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6bb0b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640d1cf0 con 0x7f7b640d18f0 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7b640eb040 0x7f7b640eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6a128700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7b640ee3e0 0x7f7b640ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56970/0 (socket says 172.21.2.117:56970) 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b6a128700 1 -- 172.21.2.117:0/4292225313 learned_addr learned my addr 172.21.2.117:0/4292225313 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 -- 172.21.2.117:0/4292225313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b640d18f0 msgr2=0x7f7b640eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 --2- 172.21.2.117:0/4292225313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b640d18f0 0x7f7b640eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 -- 172.21.2.117:0/4292225313 >> v1:172.21.2.117:6789/0 conn(0x7f7b640ee3e0 legacy=0x7f7b640ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 -- 172.21.2.117:0/4292225313 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640eefd0 con 0x7f7b640eb040 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.092 7f7b69126700 1 --2- 172.21.2.117:0/4292225313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b640eb040 0x7f7b640eb440 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7f7b60006650 tx=0x7f7b60009a80).ready entity=mon.0 client_cookie=aa182d590a346264 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b68925700 1 -- 172.21.2.117:0/4292225313 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b600009b0 con 0x7f7b640eb040 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b68925700 1 -- 172.21.2.117:0/4292225313 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6000c0c0 con 0x7f7b640eb040 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b68925700 1 -- 172.21.2.117:0/4292225313 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b6000c5e0 con 0x7f7b640eb040 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 -- 172.21.2.117:0/4292225313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b640eb040 msgr2=0x7f7b640eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 --2- 172.21.2.117:0/4292225313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b640eb040 0x7f7b640eb440 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 rx=0x7f7b60006650 tx=0x7f7b60009a80).stop 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 -- 172.21.2.117:0/4292225313 shutdown_connections 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 --2- 172.21.2.117:0/4292225313 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b640eb040 0x7f7b640eb440 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 --2- 172.21.2.117:0/4292225313 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b640d18f0 0x7f7b640eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.093 7f7b6bb0b700 1 -- 172.21.2.117:0/4292225313 >> 172.21.2.117:0/4292225313 conn(0x7f7b640cf060 msgr2=0x7f7b640cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.094 7f7b6bb0b700 1 -- 172.21.2.117:0/4292225313 shutdown_connections 2021-02-10T11:52:51.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.094 7f7b6bb0b700 1 -- 172.21.2.117:0/4292225313 wait complete. 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.108 7f7b6bb0b700 1 Processor -- start 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.108 7f7b6bb0b700 1 -- start start 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b641a4240 0x7f7b641a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b6409f790 0x7f7b640b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6409fb90 con 0x7f7b6409ed70 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b640dc640 con 0x7f7b6409f790 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b6bb0b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64097270 con 0x7f7b641a4240 2021-02-10T11:52:51.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56726/0 (socket says 172.21.2.117:56726) 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 -- 172.21.2.117:0/3212159816 learned_addr learned my addr 172.21.2.117:0/3212159816 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 -- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b641a4240 msgr2=0x7f7b641a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b641a4240 0x7f7b641a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 -- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b6409f790 msgr2=0x7f7b640b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b6409f790 0x7f7b640b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 -- 172.21.2.117:0/3212159816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b640b8780 con 0x7f7b6409ed70 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.109 7f7b69126700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f7b60009630 tx=0x7f7b6000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b600093a0 con 0x7f7b6409ed70 2021-02-10T11:52:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6000bcf0 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7b600054d0 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b640b8780 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b640b8780 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7b600054f0 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 0x7f7b4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.110 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7b600326b0 con 0x7f7b6409ed70 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.111 7f7b69927700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 0x7f7b4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.111 7f7b69927700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 0x7f7b4c01abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.142 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b640daff0 con 0x7f7b6409ed70 2021-02-10T11:52:51.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.144 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7b600052e0 con 0x7f7b6409ed70 2021-02-10T11:52:51.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.279 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f7b640ede60 con 0x7f7b6409ed70 2021-02-10T11:52:51.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.472 7f7b567fc700 1 -- 172.21.2.117:0/3212159816 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v120) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7b600054f0 con 0x7f7b6409ed70 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 msgr2=0x7f7b4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 0x7f7b4c01abc0 crc :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 msgr2=0x7f7b6409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f7b60009630 tx=0x7f7b6000b900).stop 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 shutdown_connections 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7b4c0186c0 0x7f7b4c01abc0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7b6409ed70 0x7f7b6409f170 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7b6409f790 0x7f7b640b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 --2- 172.21.2.117:0/3212159816 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7b641a4240 0x7f7b641a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 >> 172.21.2.117:0/3212159816 conn(0x7f7b64087090 msgr2=0x7f7b6408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:51.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 shutdown_connections 2021-02-10T11:52:51.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.497 7f7b6bb0b700 1 -- 172.21.2.117:0/3212159816 wait complete. 2021-02-10T11:52:51.507 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:51.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 Processor -- start 2021-02-10T11:52:51.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 -- start start 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8d240d18e0 0x7f8d240ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d240ef3f0 0x7f8d241b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240ede60 con 0x7f8d240eb000 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d240ef7f0 con 0x7f8d240ef3f0 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.919 7f8d2acec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d240d1ce0 con 0x7f8d240d18e0 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d240ef3f0 0x7f8d241b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d240ef3f0 0x7f8d241b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56736/0 (socket says 172.21.2.117:56736) 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 -- 172.21.2.117:0/887958908 learned_addr learned my addr 172.21.2.117:0/887958908 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d28b08700 1 --2- 172.21.2.117:0/887958908 >> v2:172.21.2.125:3300/0 conn(0x7f8d240d18e0 0x7f8d240ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 -- 172.21.2.117:0/887958908 >> v2:172.21.2.125:3300/0 conn(0x7f8d240d18e0 msgr2=0x7f8d240ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:51.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 --2- 172.21.2.117:0/887958908 >> v2:172.21.2.125:3300/0 conn(0x7f8d240d18e0 0x7f8d240ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 -- 172.21.2.117:0/887958908 >> v1:172.21.2.125:6789/0 conn(0x7f8d240eb000 legacy=0x7f8d240eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 -- 172.21.2.117:0/887958908 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d241b8f90 con 0x7f8d240ef3f0 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.920 7f8d29309700 1 --2- 172.21.2.117:0/887958908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d240ef3f0 0x7f8d241b7960 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 rx=0x7f8d1c006f00 tx=0x7f8d1c009e90).ready entity=mon.0 client_cookie=f7633577470d6af0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.921 7f8d237fe700 1 -- 172.21.2.117:0/887958908 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c00b300 con 0x7f8d240ef3f0 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.921 7f8d237fe700 1 -- 172.21.2.117:0/887958908 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c00b640 con 0x7f8d240ef3f0 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.921 7f8d237fe700 1 -- 172.21.2.117:0/887958908 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c00ba90 con 0x7f8d240ef3f0 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.921 7f8d2acec700 1 -- 172.21.2.117:0/887958908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d240ef3f0 msgr2=0x7f8d241b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.921 7f8d2acec700 1 --2- 172.21.2.117:0/887958908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d240ef3f0 0x7f8d241b7960 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 rx=0x7f8d1c006f00 tx=0x7f8d1c009e90).stop 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 -- 172.21.2.117:0/887958908 shutdown_connections 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 --2- 172.21.2.117:0/887958908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d240ef3f0 0x7f8d241b7960 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 --2- 172.21.2.117:0/887958908 >> v2:172.21.2.125:3300/0 conn(0x7f8d240d18e0 0x7f8d240ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 -- 172.21.2.117:0/887958908 >> 172.21.2.117:0/887958908 conn(0x7f8d240ced60 msgr2=0x7f8d240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:51.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 -- 172.21.2.117:0/887958908 shutdown_connections 2021-02-10T11:52:51.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.922 7f8d2acec700 1 -- 172.21.2.117:0/887958908 wait complete. 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.936 7f8d2acec700 1 Processor -- start 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.936 7f8d2acec700 1 -- start start 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d24091d40 0x7f8d240e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d241b24b0 0x7f8d241b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24197cd0 con 0x7f8d24198250 2021-02-10T11:52:51.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d24092140 con 0x7f8d24091d40 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d2acec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d241b2ed0 con 0x7f8d241b24b0 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d29309700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d29309700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56738/0 (socket says 172.21.2.117:56738) 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d29309700 1 -- 172.21.2.117:0/3481749317 learned_addr learned my addr 172.21.2.117:0/3481749317 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.937 7f8d29309700 1 -- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d241b24b0 msgr2=0x7f8d241b28b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d29309700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d241b24b0 0x7f8d241b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d29309700 1 -- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d24091d40 msgr2=0x7f8d240e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d29309700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d24091d40 0x7f8d240e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d29309700 1 -- 172.21.2.117:0/3481749317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d24198c70 con 0x7f8d24198250 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d29309700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f8d1c00b910 tx=0x7f8d1c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c00bdf0 con 0x7f8d24198250 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c005820 con 0x7f8d24198250 2021-02-10T11:52:51.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d24198c70 con 0x7f8d24198250 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d24196c90 con 0x7f8d24198250 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.938 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d1c005d40 con 0x7f8d24198250 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.939 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d1c005a00 con 0x7f8d24198250 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.939 7f8d217fa700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 0x7f8d0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.939 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d1c032a50 con 0x7f8d24198250 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.939 7f8d28b08700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 0x7f8d0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:51.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.939 7f8d28b08700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 0x7f8d0801abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:51.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.969 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24087290 con 0x7f8d24198250 2021-02-10T11:52:51.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:51.971 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d1c000f90 con 0x7f8d24198250 2021-02-10T11:52:52.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.099 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d240ede60 con 0x7f8d24198250 2021-02-10T11:52:52.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.099 7f8d217fa700 1 -- 172.21.2.117:0/3481749317 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f8d1c006ef0 con 0x7f8d24198250 2021-02-10T11:52:52.101 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:52.101 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:52:51.468783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7415":{"gid":7415,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/154249133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":154249133},{"type":"v1","addr":"172.21.2.125:6835","nonce":154249133}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:52:52.101 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 120 2021-02-10T11:52:52.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 msgr2=0x7f8d0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 0x7f8d0801abc0 crc :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 msgr2=0x7f8d24198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 rx=0x7f8d1c00b910 tx=0x7f8d1c00bcb0).stop 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 shutdown_connections 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d080186c0 0x7f8d0801abc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d24198250 0x7f8d24198650 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d241b24b0 0x7f8d241b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 --2- 172.21.2.117:0/3481749317 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d24091d40 0x7f8d240e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 >> 172.21.2.117:0/3481749317 conn(0x7f8d241a3620 msgr2=0x7f8d240dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 shutdown_connections 2021-02-10T11:52:52.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.101 7f8d2acec700 1 -- 172.21.2.117:0/3481749317 wait complete. 2021-02-10T11:52:52.111 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.520 7f9bf3901700 1 Processor -- start 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.520 7f9bf3901700 1 -- start start 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf3901700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf3901700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9bec0eb7f0 0x7f9bec1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf3901700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0ede60 con 0x7f9bec0d18e0 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf3901700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec0ef7f0 con 0x7f9bec0ef3f0 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf3901700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec0d1ce0 con 0x7f9bec0eb7f0 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf0f1c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf0f1c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56746/0 (socket says 172.21.2.117:56746) 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf0f1c700 1 -- 172.21.2.117:0/3208635576 learned_addr learned my addr 172.21.2.117:0/3208635576 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:52.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf171d700 1 --1- 172.21.2.117:0/3208635576 >> v1:172.21.2.117:6789/0 conn(0x7f9bec0d18e0 0x7f9bec0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:56996/0 (socket says 172.21.2.117:56996) 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.521 7f9bf1f1e700 1 --2- 172.21.2.117:0/3208635576 >> v2:172.21.2.125:3300/0 conn(0x7f9bec0eb7f0 0x7f9bec1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf0f1c700 1 -- 172.21.2.117:0/3208635576 >> v2:172.21.2.125:3300/0 conn(0x7f9bec0eb7f0 msgr2=0x7f9bec1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf0f1c700 1 --2- 172.21.2.117:0/3208635576 >> v2:172.21.2.125:3300/0 conn(0x7f9bec0eb7f0 0x7f9bec1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf0f1c700 1 -- 172.21.2.117:0/3208635576 >> v1:172.21.2.117:6789/0 conn(0x7f9bec0d18e0 legacy=0x7f9bec0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf0f1c700 1 -- 172.21.2.117:0/3208635576 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec1b8f90 con 0x7f9bec0ef3f0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf0f1c700 1 --2- 172.21.2.117:0/3208635576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f9be8006700 tx=0x7f9be8009b80).ready entity=mon.0 client_cookie=8ef3aa3d028ecacf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bdffff700 1 -- 172.21.2.117:0/3208635576 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be800c0c0 con 0x7f9bec0ef3f0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bdffff700 1 -- 172.21.2.117:0/3208635576 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be800c340 con 0x7f9bec0ef3f0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bdffff700 1 -- 172.21.2.117:0/3208635576 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be800c860 con 0x7f9bec0ef3f0 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf3901700 1 -- 172.21.2.117:0/3208635576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec0ef3f0 msgr2=0x7f9bec0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.522 7f9bf3901700 1 --2- 172.21.2.117:0/3208635576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 rx=0x7f9be8006700 tx=0x7f9be8009b80).stop 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 -- 172.21.2.117:0/3208635576 shutdown_connections 2021-02-10T11:52:52.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 --2- 172.21.2.117:0/3208635576 >> v2:172.21.2.125:3300/0 conn(0x7f9bec0eb7f0 0x7f9bec1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 --2- 172.21.2.117:0/3208635576 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec0ef3f0 0x7f9bec0eb1d0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 -- 172.21.2.117:0/3208635576 >> 172.21.2.117:0/3208635576 conn(0x7f9bec0ced60 msgr2=0x7f9bec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:52.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 -- 172.21.2.117:0/3208635576 shutdown_connections 2021-02-10T11:52:52.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.523 7f9bf3901700 1 -- 172.21.2.117:0/3208635576 wait complete. 2021-02-10T11:52:52.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 Processor -- start 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 -- start start 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bec090c40 0x7f9bec091040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bec08c1c0 0x7f9bec08c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0906c0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec1a3f50 con 0x7f9bec090c40 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf3901700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec091e40 con 0x7f9bec08c1c0 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf171d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf171d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56754/0 (socket says 172.21.2.117:56754) 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.538 7f9bf171d700 1 -- 172.21.2.117:0/1452330505 learned_addr learned my addr 172.21.2.117:0/1452330505 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 -- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bec08c1c0 msgr2=0x7f9bec08c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bec08c1c0 0x7f9bec08c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 -- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bec090c40 msgr2=0x7f9bec091040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bec090c40 0x7f9bec091040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 -- 172.21.2.117:0/1452330505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec08cbe0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf171d700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f9be000a590 tx=0x7f9be000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be0000ef0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000ad20 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be0004620 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec08cbe0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.539 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec08cbe0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.540 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9be00045b0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.540 7f9bddffb700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 0x7f9bcc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.540 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9be0032770 con 0x7f9bec1a3b50 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.540 7f9bf0f1c700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 0x7f9bcc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:52.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.541 7f9bf0f1c700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 0x7f9bcc01abd0 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:52.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.570 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec087290 con 0x7f9bec1a3b50 2021-02-10T11:52:52.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.573 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9be00043a0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.697 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bec087290 con 0x7f9bec1a3b50 2021-02-10T11:52:52.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.697 7f9bddffb700 1 -- 172.21.2.117:0/1452330505 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f9be00045b0 con 0x7f9bec1a3b50 2021-02-10T11:52:52.699 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:52.699 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":143,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:50.464421","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:47.477707","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:48.506496","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:52.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 msgr2=0x7f9bcc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 0x7f9bcc01abd0 crc :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 msgr2=0x7f9bec091820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:52.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 rx=0x7f9be000a590 tx=0x7f9be000a930).stop 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 shutdown_connections 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9bec08c1c0 0x7f9bec08c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9bec090c40 0x7f9bec091040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bcc0186d0 0x7f9bcc01abd0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 --2- 172.21.2.117:0/1452330505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9bec1a3b50 0x7f9bec091820 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 >> 172.21.2.117:0/1452330505 conn(0x7f9bec0874c0 msgr2=0x7f9bec199ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 shutdown_connections 2021-02-10T11:52:52.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:52.699 7f9bf3901700 1 -- 172.21.2.117:0/1452330505 wait complete. 2021-02-10T11:52:52.709 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:53.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.132 7f63916af700 1 Processor -- start 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.132 7f63916af700 1 -- start start 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f63916af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f638c079ce0 0x7f638c1b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f63916af700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c07a5c0 con 0x7f638c07ab40 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f63916af700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c0d18c0 con 0x7f638c07a0e0 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f63916af700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c07bf50 con 0x7f638c079ce0 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638aabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f638c079ce0 0x7f638c1b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638b2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f638c07ab40 0x7f638c0794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57010/0 (socket says 172.21.2.117:57010) 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638b2bf700 1 -- 172.21.2.117:0/2068270905 learned_addr learned my addr 172.21.2.117:0/2068270905 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638a2bd700 1 -- 172.21.2.117:0/2068270905 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887661063 0 0) 0x7f63740009b0 con 0x7f638c07ab40 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638a2bd700 1 -- 172.21.2.117:0/2068270905 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6378002fe0 con 0x7f638c07ab40 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638aabe700 1 -- 172.21.2.117:0/2068270905 >> v1:172.21.2.125:6789/0 conn(0x7f638c07a0e0 legacy=0x7f638c1b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:53.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638aabe700 1 -- 172.21.2.117:0/2068270905 >> v1:172.21.2.117:6789/0 conn(0x7f638c07ab40 legacy=0x7f638c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638aabe700 1 -- 172.21.2.117:0/2068270905 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c0d1c60 con 0x7f638c079ce0 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.133 7f638aabe700 1 --2- 172.21.2.117:0/2068270905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c079ce0 0x7f638c1b59d0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).ready entity=mon.0 client_cookie=3cba3d49db3356fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f638a2bd700 1 -- 172.21.2.117:0/2068270905 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c0009b0 con 0x7f638c079ce0 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f638a2bd700 1 -- 172.21.2.117:0/2068270905 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00c0c0 con 0x7f638c079ce0 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f638a2bd700 1 -- 172.21.2.117:0/2068270905 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c00c5e0 con 0x7f638c079ce0 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f63916af700 1 -- 172.21.2.117:0/2068270905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c079ce0 msgr2=0x7f638c1b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f63916af700 1 --2- 172.21.2.117:0/2068270905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c079ce0 0x7f638c1b59d0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 rx=0x7f637c006650 tx=0x7f637c009a80).stop 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f63916af700 1 -- 172.21.2.117:0/2068270905 shutdown_connections 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f63916af700 1 --2- 172.21.2.117:0/2068270905 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c079ce0 0x7f638c1b59d0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.134 7f63916af700 1 -- 172.21.2.117:0/2068270905 >> 172.21.2.117:0/2068270905 conn(0x7f638c0d0910 msgr2=0x7f638c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.135 7f63916af700 1 -- 172.21.2.117:0/2068270905 shutdown_connections 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.135 7f63916af700 1 -- 172.21.2.117:0/2068270905 wait complete. 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 Processor -- start 2021-02-10T11:52:53.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 -- start start 2021-02-10T11:52:53.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f638c09ae40 0x7f638c19c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638c0ff4c0 0x7f638c0ff8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c100cc0 con 0x7f638c09b240 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0ffee0 con 0x7f638c0ff4c0 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.149 7f63916af700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c100100 con 0x7f638c09ae40 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56768/0 (socket says 172.21.2.117:56768) 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 -- 172.21.2.117:0/54773783 learned_addr learned my addr 172.21.2.117:0/54773783 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 -- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f638c09ae40 msgr2=0x7f638c19c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f638c09ae40 0x7f638c19c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 -- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638c0ff4c0 msgr2=0x7f638c0ff8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638c0ff4c0 0x7f638c0ff8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 -- 172.21.2.117:0/54773783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c100320 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f638aabe700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7f637c009630 tx=0x7f637c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c0093a0 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c00bd70 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f637c005560 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f63916af700 1 -- 172.21.2.117:0/54773783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c100320 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.150 7f63916af700 1 -- 172.21.2.117:0/54773783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c1008b0 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.151 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f637c00a000 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.151 7f6383fff700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 0x7f637001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.151 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f637c0328c0 con 0x7f638c09b240 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.151 7f638b2bf700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 0x7f637001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.152 7f638b2bf700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 0x7f637001abc0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.183 7f63916af700 1 -- 172.21.2.117:0/54773783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c0e5770 con 0x7f638c09b240 2021-02-10T11:52:53.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.185 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f637c0093a0 con 0x7f638c09b240 2021-02-10T11:52:53.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.315 7f63916af700 1 -- 172.21.2.117:0/54773783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f638c0e5770 con 0x7f638c09b240 2021-02-10T11:52:53.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.318 7f6383fff700 1 -- 172.21.2.117:0/54773783 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v81) v1 ==== 184+0+0 (secure 0 0 0) 0x7f637c01f4b0 con 0x7f638c09b240 2021-02-10T11:52:53.323 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:52:53.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 msgr2=0x7f637001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 0x7f637001abc0 crc :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 msgr2=0x7f638c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 rx=0x7f637c009630 tx=0x7f637c00b980).stop 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 shutdown_connections 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63700186c0 0x7f637001abc0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f638c09b240 0x7f638c19c7b0 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f638c0ff4c0 0x7f638c0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 --2- 172.21.2.117:0/54773783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f638c09ae40 0x7f638c19c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 >> 172.21.2.117:0/54773783 conn(0x7f638c19f690 msgr2=0x7f638c1950e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 shutdown_connections 2021-02-10T11:52:53.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.324 7f63916af700 1 -- 172.21.2.117:0/54773783 wait complete. 2021-02-10T11:52:53.334 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.757 7fc9e433c700 1 Processor -- start 2021-02-10T11:52:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.757 7fc9e433c700 1 -- start start 2021-02-10T11:52:53.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e433c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc9dc07ab40 0x7fc9dc0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e433c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0e5770 con 0x7fc9dc079f10 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e433c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc07a310 con 0x7fc9dc079b10 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e433c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc0d18c0 con 0x7fc9dc07ab40 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc9dc07ab40 0x7fc9dc0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc9dc07ab40 0x7fc9dc0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56774/0 (socket says 172.21.2.117:56774) 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 -- 172.21.2.117:0/3914069278 learned_addr learned my addr 172.21.2.117:0/3914069278 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 -- 172.21.2.117:0/3914069278 >> v1:172.21.2.125:6789/0 conn(0x7fc9dc079b10 legacy=0x7fc9dc07af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 -- 172.21.2.117:0/3914069278 >> v1:172.21.2.117:6789/0 conn(0x7fc9dc079f10 legacy=0x7fc9dc07b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 -- 172.21.2.117:0/3914069278 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc07bf20 con 0x7fc9dc07ab40 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.758 7fc9e2158700 1 --2- 172.21.2.117:0/3914069278 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc07ab40 0x7fc9dc0794f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7fc9d8006700 tx=0x7fc9d8009b80).ready entity=mon.0 client_cookie=cdfaf9ba785a032f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e1156700 1 -- 172.21.2.117:0/3914069278 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9d800c0c0 con 0x7fc9dc07ab40 2021-02-10T11:52:53.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e1156700 1 -- 172.21.2.117:0/3914069278 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d800c440 con 0x7fc9dc07ab40 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e1156700 1 -- 172.21.2.117:0/3914069278 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9d800c960 con 0x7fc9dc07ab40 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e433c700 1 -- 172.21.2.117:0/3914069278 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc07ab40 msgr2=0x7fc9dc0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e433c700 1 --2- 172.21.2.117:0/3914069278 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc07ab40 0x7fc9dc0794f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 rx=0x7fc9d8006700 tx=0x7fc9d8009b80).stop 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e433c700 1 -- 172.21.2.117:0/3914069278 shutdown_connections 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e433c700 1 --2- 172.21.2.117:0/3914069278 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc07ab40 0x7fc9dc0794f0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.759 7fc9e433c700 1 -- 172.21.2.117:0/3914069278 >> 172.21.2.117:0/3914069278 conn(0x7fc9dc0d0910 msgr2=0x7fc9dc0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.760 7fc9e433c700 1 -- 172.21.2.117:0/3914069278 shutdown_connections 2021-02-10T11:52:53.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.760 7fc9e433c700 1 -- 172.21.2.117:0/3914069278 wait complete. 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.774 7fc9e433c700 1 Processor -- start 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.774 7fc9e433c700 1 -- start start 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 0x7fc9dc098750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc9dc099790 0x7fc9dc0b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0f8d40 con 0x7fc9dc098d70 2021-02-10T11:52:53.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0b1b70 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e433c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc0b1d90 con 0x7fc9dc099790 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e1957700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e1957700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56782/0 (socket says 172.21.2.117:56782) 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e1957700 1 -- 172.21.2.117:0/1622698718 learned_addr learned my addr 172.21.2.117:0/1622698718 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 0x7fc9dc098750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2959700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc9dc099790 0x7fc9dc0b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 -- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc9dc099790 msgr2=0x7fc9dc0b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc9dc099790 0x7fc9dc0b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 -- 172.21.2.117:0/1622698718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 msgr2=0x7fc9dc099170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e2158700 1 -- 172.21.2.117:0/1622698718 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0b1fb0 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.775 7fc9e1957700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9e2158700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 0x7fc9dc098750 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7fc9d80096d0 tx=0x7fc9d800b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9d800bcb0 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d8002030 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc0b1fb0 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc9d8002590 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc0b1fb0 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc9d8000c90 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 0x7fc9c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.776 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc9d8032880 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.777 7fc9e1957700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 0x7fc9c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:53.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.777 7fc9e1957700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 0x7fc9c401abc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:53.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.807 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc0d1c60 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.812 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc9d801f470 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.944 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc9dc0d1c60 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.948 7fc9ceffd700 1 -- 172.21.2.117:0/1622698718 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v82) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc9d8000c90 con 0x7fc9dc1c68b0 2021-02-10T11:52:53.952 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:52:53.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 msgr2=0x7fc9c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 0x7fc9c401abc0 crc :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 msgr2=0x7fc9dc098750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 0x7fc9dc098750 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 rx=0x7fc9d80096d0 tx=0x7fc9d800b8c0).stop 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 shutdown_connections 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc9c4018730 0x7fc9c401abc0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc9dc098d70 0x7fc9dc099170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc9dc099790 0x7fc9dc0b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 --2- 172.21.2.117:0/1622698718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc9dc1c68b0 0x7fc9dc098750 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 >> 172.21.2.117:0/1622698718 conn(0x7fc9dc0fc450 msgr2=0x7fc9dc0bad50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 shutdown_connections 2021-02-10T11:52:53.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:53.953 7fc9e433c700 1 -- 172.21.2.117:0/1622698718 wait complete. 2021-02-10T11:52:53.963 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:54.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 Processor -- start 2021-02-10T11:52:54.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 -- start start 2021-02-10T11:52:54.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f064007ab40 0x7f06400794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06400e5770 con 0x7f0640079b10 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064007a310 con 0x7f064007ab40 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.385 7f0644a99700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400d18c0 con 0x7f0640079f10 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54470/0 (socket says 172.21.2.117:54470) 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 -- 172.21.2.117:0/573655569 learned_addr learned my addr 172.21.2.117:0/573655569 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063e85e700 1 --2- 172.21.2.117:0/573655569 >> v2:172.21.2.125:3300/0 conn(0x7f064007ab40 0x7f06400794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 -- 172.21.2.117:0/573655569 >> v2:172.21.2.125:3300/0 conn(0x7f064007ab40 msgr2=0x7f06400794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 --2- 172.21.2.117:0/573655569 >> v2:172.21.2.125:3300/0 conn(0x7f064007ab40 0x7f06400794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 -- 172.21.2.117:0/573655569 >> v1:172.21.2.125:6789/0 conn(0x7f0640079b10 legacy=0x7f064007af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 -- 172.21.2.117:0/573655569 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06400d1c60 con 0x7f0640079f10 2021-02-10T11:52:54.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.386 7f063f05f700 1 --2- 172.21.2.117:0/573655569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0638006aa0 tx=0x7f0638009f20).ready entity=mon.2 client_cookie=b2e40941e6aa37c4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f063d85c700 1 -- 172.21.2.117:0/573655569 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f063800b5c0 con 0x7f0640079f10 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f063d85c700 1 -- 172.21.2.117:0/573655569 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0638003030 con 0x7f0640079f10 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f063d85c700 1 -- 172.21.2.117:0/573655569 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f063800bb00 con 0x7f0640079f10 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 -- 172.21.2.117:0/573655569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 msgr2=0x7f064007b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 --2- 172.21.2.117:0/573655569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0638006aa0 tx=0x7f0638009f20).stop 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 -- 172.21.2.117:0/573655569 shutdown_connections 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 --2- 172.21.2.117:0/573655569 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0640079f10 0x7f064007b730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 --2- 172.21.2.117:0/573655569 >> v2:172.21.2.125:3300/0 conn(0x7f064007ab40 0x7f06400794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.387 7f0644a99700 1 -- 172.21.2.117:0/573655569 >> 172.21.2.117:0/573655569 conn(0x7f06400d0910 msgr2=0x7f06400d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.388 7f0644a99700 1 -- 172.21.2.117:0/573655569 shutdown_connections 2021-02-10T11:52:54.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.388 7f0644a99700 1 -- 172.21.2.117:0/573655569 wait complete. 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.402 7f0644a99700 1 Processor -- start 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.402 7f0644a99700 1 -- start start 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06400fe660 0x7f06400fea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06400ff800 0x7f06400ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400ff300 con 0x7f06400dc6e0 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400a8b30 con 0x7f06400ff800 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f0644a99700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400dc230 con 0x7f06400fe660 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56796/0 (socket says 172.21.2.117:56796) 2021-02-10T11:52:54.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 -- 172.21.2.117:0/569498726 learned_addr learned my addr 172.21.2.117:0/569498726 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:54.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 -- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06400fe660 msgr2=0x7f06400fea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06400fe660 0x7f06400fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:54.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 -- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06400ff800 msgr2=0x7f06400ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06400ff800 0x7f06400ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:54.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 -- 172.21.2.117:0/569498726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0640100220 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.403 7f063e85e700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f0628006f60 tx=0x7f062800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.404 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06280009b0 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.404 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062800bdf0 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.404 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0628002030 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.404 7f0644a99700 1 -- 172.21.2.117:0/569498726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0640100220 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.404 7f0644a99700 1 -- 172.21.2.117:0/569498726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06401007b0 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.405 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0628006b00 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.405 7f06377fe700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 0x7f062401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.405 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f06280326b0 con 0x7f06400dc6e0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.405 7f063e05d700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 0x7f062401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:54.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.405 7f063e05d700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 0x7f062401abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:54.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.435 7f0644a99700 1 -- 172.21.2.117:0/569498726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06400da9a0 con 0x7f06400dc6e0 2021-02-10T11:52:54.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.438 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06280009b0 con 0x7f06400dc6e0 2021-02-10T11:52:54.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.568 7f0644a99700 1 -- 172.21.2.117:0/569498726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f06400e5770 con 0x7f06400dc6e0 2021-02-10T11:52:54.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.571 7f06377fe700 1 -- 172.21.2.117:0/569498726 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v83) v1 ==== 184+0+0 (secure 0 0 0) 0x7f062801f2a0 con 0x7f06400dc6e0 2021-02-10T11:52:54.576 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:52:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 msgr2=0x7f062401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 0x7f062401abc0 crc :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 msgr2=0x7f0640191c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:54.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 rx=0x7f0628006f60 tx=0x7f062800b970).stop 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 shutdown_connections 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06400ff800 0x7f06400ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06400fe660 0x7f06400fea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06240186c0 0x7f062401abc0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 --2- 172.21.2.117:0/569498726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06400dc6e0 0x7f0640191c90 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 >> 172.21.2.117:0/569498726 conn(0x7f064019f690 msgr2=0x7f0640096490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 shutdown_connections 2021-02-10T11:52:54.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:54.577 7f0644a99700 1 -- 172.21.2.117:0/569498726 wait complete. 2021-02-10T11:52:54.587 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 Processor -- start 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 -- start start 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f332c079ce0 0x7f332c1b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c07a5c0 con 0x7f332c07a0e0 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c0d18c0 con 0x7f332c07ab40 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f3331d51700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c07bf50 con 0x7f332c079ce0 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f332affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f332c079ce0 0x7f332c1b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f332bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f332c07a0e0 0x7f332c1b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57052/0 (socket says 172.21.2.117:57052) 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.001 7f332bfff700 1 -- 172.21.2.117:0/2538348462 learned_addr learned my addr 172.21.2.117:0/2538348462 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332affd700 1 -- 172.21.2.117:0/2538348462 >> v1:172.21.2.125:6789/0 conn(0x7f332c07ab40 legacy=0x7f332c0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332affd700 1 -- 172.21.2.117:0/2538348462 >> v1:172.21.2.117:6789/0 conn(0x7f332c07a0e0 legacy=0x7f332c1b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332affd700 1 -- 172.21.2.117:0/2538348462 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c0d1c60 con 0x7f332c079ce0 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332affd700 1 --2- 172.21.2.117:0/2538348462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c079ce0 0x7f332c1b59d0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7f3324006650 tx=0x7f3324009a80).ready entity=mon.0 client_cookie=a0241e8ec6167225 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332a7fc700 1 -- 172.21.2.117:0/2538348462 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33240009b0 con 0x7f332c079ce0 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332a7fc700 1 -- 172.21.2.117:0/2538348462 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332400c0c0 con 0x7f332c079ce0 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f332a7fc700 1 -- 172.21.2.117:0/2538348462 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f332400c5e0 con 0x7f332c079ce0 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f3331d51700 1 -- 172.21.2.117:0/2538348462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c079ce0 msgr2=0x7f332c1b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.002 7f3331d51700 1 --2- 172.21.2.117:0/2538348462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c079ce0 0x7f332c1b59d0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 rx=0x7f3324006650 tx=0x7f3324009a80).stop 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.003 7f3331d51700 1 -- 172.21.2.117:0/2538348462 shutdown_connections 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.003 7f3331d51700 1 --2- 172.21.2.117:0/2538348462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c079ce0 0x7f332c1b59d0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.003 7f3331d51700 1 -- 172.21.2.117:0/2538348462 >> 172.21.2.117:0/2538348462 conn(0x7f332c0d0910 msgr2=0x7f332c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:55.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.003 7f3331d51700 1 -- 172.21.2.117:0/2538348462 shutdown_connections 2021-02-10T11:52:55.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.003 7f3331d51700 1 -- 172.21.2.117:0/2538348462 wait complete. 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 Processor -- start 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 -- start start 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 0x7f332c0ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f332c0956d0 0x7f332c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c095ad0 con 0x7f332c0b4320 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c095cb0 con 0x7f332c0ff230 2021-02-10T11:52:55.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.018 7f3331d51700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c095ed0 con 0x7f332c0956d0 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56808/0 (socket says 172.21.2.117:56808) 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 -- 172.21.2.117:0/1968649454 learned_addr learned my addr 172.21.2.117:0/1968649454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332affd700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f332c0956d0 0x7f332c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332b7fe700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 0x7f332c0ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 -- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f332c0956d0 msgr2=0x7f332c0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f332c0956d0 0x7f332c0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 -- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 msgr2=0x7f332c0ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 0x7f332c0ff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 -- 172.21.2.117:0/1968649454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c1c5510 con 0x7f332c0b4320 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332b7fe700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 0x7f332c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f332bfff700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 rx=0x7f331c00a570 tx=0x7f331c00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f331c0070d0 con 0x7f332c0b4320 2021-02-10T11:52:55.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c00ad00 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f3331d51700 1 -- 172.21.2.117:0/1968649454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c1c5510 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.019 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f331c004590 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.020 7f3331d51700 1 -- 172.21.2.117:0/1968649454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c1c5510 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.020 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f331c0043a0 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.020 7f3313fff700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 0x7f330c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.020 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f331c0326a0 con 0x7f332c0b4320 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.020 7f332b7fe700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 0x7f330c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.021 7f332b7fe700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 0x7f330c01abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.051 7f3331d51700 1 -- 172.21.2.117:0/1968649454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c07d4f0 con 0x7f332c0b4320 2021-02-10T11:52:55.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.054 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f331c030d80 con 0x7f332c0b4320 2021-02-10T11:52:55.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.181 7f3331d51700 1 -- 172.21.2.117:0/1968649454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f332c07d4f0 con 0x7f332c0b4320 2021-02-10T11:52:55.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.185 7f3313fff700 1 -- 172.21.2.117:0/1968649454 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v84) v1 ==== 184+0+0 (secure 0 0 0) 0x7f331c01f4d0 con 0x7f332c0b4320 2021-02-10T11:52:55.189 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:52:55.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.189 7f3331d51700 1 -- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 msgr2=0x7f330c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.189 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 0x7f330c01abc0 crc :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.189 7f3331d51700 1 -- 172.21.2.117:0/1968649454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 msgr2=0x7f332c1c4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.189 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 rx=0x7f331c00a570 tx=0x7f331c00a910).stop 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 -- 172.21.2.117:0/1968649454 shutdown_connections 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f330c0186c0 0x7f330c01abc0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f332c0b4320 0x7f332c1c4ef0 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f332c0956d0 0x7f332c0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 --2- 172.21.2.117:0/1968649454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f332c0ff230 0x7f332c0ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 -- 172.21.2.117:0/1968649454 >> 172.21.2.117:0/1968649454 conn(0x7f332c09e020 msgr2=0x7f332c0a7140 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 -- 172.21.2.117:0/1968649454 shutdown_connections 2021-02-10T11:52:55.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.190 7f3331d51700 1 -- 172.21.2.117:0/1968649454 wait complete. 2021-02-10T11:52:55.200 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:55.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 Processor -- start 2021-02-10T11:52:55.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 -- start start 2021-02-10T11:52:55.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcfc80ef3f0 0x7fcfc80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 0x7fcfc81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc80ede60 con 0x7fcfc80d18e0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc80ef7f0 con 0x7fcfc80ef3f0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.610 7fcfcd391700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc80d1ce0 con 0x7fcfc80eb7f0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc707d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcfc80d18e0 0x7fcfc80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57062/0 (socket says 172.21.2.117:57062) 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc707d700 1 -- 172.21.2.117:0/1382670813 learned_addr learned my addr 172.21.2.117:0/1382670813 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc787e700 1 --2- 172.21.2.117:0/1382670813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 0x7fcfc81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc687c700 1 --2- 172.21.2.117:0/1382670813 >> v2:172.21.2.125:3300/0 conn(0x7fcfc80ef3f0 0x7fcfc80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc607b700 1 -- 172.21.2.117:0/1382670813 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267751886 0 0) 0x7fcfb8001180 con 0x7fcfc80d18e0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc607b700 1 -- 172.21.2.117:0/1382670813 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb4002fe0 con 0x7fcfc80d18e0 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc787e700 1 -- 172.21.2.117:0/1382670813 >> v2:172.21.2.125:3300/0 conn(0x7fcfc80ef3f0 msgr2=0x7fcfc80eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:55.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc787e700 1 --2- 172.21.2.117:0/1382670813 >> v2:172.21.2.125:3300/0 conn(0x7fcfc80ef3f0 0x7fcfc80eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc787e700 1 -- 172.21.2.117:0/1382670813 >> v1:172.21.2.117:6789/0 conn(0x7fcfc80d18e0 legacy=0x7fcfc80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.611 7fcfc787e700 1 -- 172.21.2.117:0/1382670813 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc81b8f90 con 0x7fcfc80eb7f0 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfc787e700 1 --2- 172.21.2.117:0/1382670813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 0x7fcfc81b7960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fcfc0006e50 tx=0x7fcfc0009e70).ready entity=mon.2 client_cookie=6dc8f7b712dbf8c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfc607b700 1 -- 172.21.2.117:0/1382670813 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc0006a20 con 0x7fcfc80eb7f0 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfc607b700 1 -- 172.21.2.117:0/1382670813 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc0006790 con 0x7fcfc80eb7f0 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfc607b700 1 -- 172.21.2.117:0/1382670813 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfc000b700 con 0x7fcfc80eb7f0 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfcd391700 1 -- 172.21.2.117:0/1382670813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 msgr2=0x7fcfc81b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.612 7fcfcd391700 1 --2- 172.21.2.117:0/1382670813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 0x7fcfc81b7960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fcfc0006e50 tx=0x7fcfc0009e70).stop 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 -- 172.21.2.117:0/1382670813 shutdown_connections 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 --2- 172.21.2.117:0/1382670813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc80eb7f0 0x7fcfc81b7960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 --2- 172.21.2.117:0/1382670813 >> v2:172.21.2.125:3300/0 conn(0x7fcfc80ef3f0 0x7fcfc80eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 -- 172.21.2.117:0/1382670813 >> 172.21.2.117:0/1382670813 conn(0x7fcfc80ced60 msgr2=0x7fcfc80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:55.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 -- 172.21.2.117:0/1382670813 shutdown_connections 2021-02-10T11:52:55.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.613 7fcfcd391700 1 -- 172.21.2.117:0/1382670813 wait complete. 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.627 7fcfcd391700 1 Processor -- start 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.627 7fcfcd391700 1 -- start start 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc81afd30 0x7fcfc80b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfc81b0130 0x7fcfc80e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc80e80e0 con 0x7fcfc80b43f0 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc80e7e20 con 0x7fcfc81b0130 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfcd391700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc8096a70 con 0x7fcfc81afd30 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56818/0 (socket says 172.21.2.117:56818) 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 -- 172.21.2.117:0/695063910 learned_addr learned my addr 172.21.2.117:0/695063910 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc707d700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc81afd30 0x7fcfc80b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 -- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc81afd30 msgr2=0x7fcfc80b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc81afd30 0x7fcfc80b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 -- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfc81b0130 msgr2=0x7fcfc80e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfc81b0130 0x7fcfc80e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 -- 172.21.2.117:0/695063910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc80e8660 con 0x7fcfc80b43f0 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.628 7fcfc687c700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7fcfb0006f60 tx=0x7fcfb000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfb000bd60 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0000c90 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfcd391700 1 -- 172.21.2.117:0/695063910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc80e8660 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcfb0002220 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfcd391700 1 -- 172.21.2.117:0/695063910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc80e8660 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcfb00052e0 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 0x7fcfac01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.629 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcfb00328e0 con 0x7fcfc80b43f0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.630 7fcfc707d700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 0x7fcfac01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:55.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.630 7fcfc707d700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 0x7fcfac01ac10 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:55.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.660 7fcfcd391700 1 -- 172.21.2.117:0/695063910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc80e7e20 con 0x7fcfc80b43f0 2021-02-10T11:52:55.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.663 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcfb00052e0 con 0x7fcfc80b43f0 2021-02-10T11:52:55.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.790 7fcfcd391700 1 -- 172.21.2.117:0/695063910 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc80e7e20 con 0x7fcfc80b43f0 2021-02-10T11:52:55.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.791 7fcfbffff700 1 -- 172.21.2.117:0/695063910 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fcfb000bd60 con 0x7fcfc80b43f0 2021-02-10T11:52:55.793 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:55.793 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:52:51.468783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7415":{"gid":7415,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/154249133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":154249133},{"type":"v1","addr":"172.21.2.125:6835","nonce":154249133}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:52:55.793 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 120 2021-02-10T11:52:55.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 msgr2=0x7fcfac01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 0x7fcfac01ac10 crc :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 msgr2=0x7fcfc80e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 rx=0x7fcfb0006f60 tx=0x7fcfb000b970).stop 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 shutdown_connections 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcfac018730 0x7fcfac01ac10 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcfc80b43f0 0x7fcfc80e71e0 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcfc81b0130 0x7fcfc80e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 --2- 172.21.2.117:0/695063910 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcfc81afd30 0x7fcfc80b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 >> 172.21.2.117:0/695063910 conn(0x7fcfc81a3620 msgr2=0x7fcfc80dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 shutdown_connections 2021-02-10T11:52:55.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:55.793 7fcfcd391700 1 -- 172.21.2.117:0/695063910 wait complete. 2021-02-10T11:52:56.005 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 120, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:49.462120', 'modified': '2021-02-10 11:52:51.468783', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7415': {'gid': 7415, 'name': 'b', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/154249133', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 154249133}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 154249133}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:52:56.005 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:52:56.005 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7415, 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:52:56.405 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9388608930000002, 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 120, 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 143, 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.582890447 2021-02-10T11:52:56.406 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:52:56.409 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7415, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9388608930000002, "mdsmap_epoch": 120, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 12.582890447 } 2021-02-10T11:52:56.410 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 Processor -- start 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 -- start start 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8640d18e0 0x7fc8640ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc8640ef3f0 0x7fc8641b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8640ede60 con 0x7fc8640eb000 2021-02-10T11:52:56.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640ef7f0 con 0x7fc8640d18e0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.820 7fc86b960700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8640d1ce0 con 0x7fc8640ef3f0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc8640d18e0 0x7fc8640ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc868f7b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc8640eb000 0x7fc8640eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57076/0 (socket says 172.21.2.117:57076) 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc868f7b700 1 -- 172.21.2.117:0/2738328608 learned_addr learned my addr 172.21.2.117:0/2738328608 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc869f7d700 1 --2- 172.21.2.117:0/2738328608 >> v2:172.21.2.125:3300/0 conn(0x7fc8640ef3f0 0x7fc8641b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 -- 172.21.2.117:0/2738328608 >> v2:172.21.2.125:3300/0 conn(0x7fc8640ef3f0 msgr2=0x7fc8641b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 --2- 172.21.2.117:0/2738328608 >> v2:172.21.2.125:3300/0 conn(0x7fc8640ef3f0 0x7fc8641b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 -- 172.21.2.117:0/2738328608 >> v1:172.21.2.117:6789/0 conn(0x7fc8640eb000 legacy=0x7fc8640eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 -- 172.21.2.117:0/2738328608 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8641b8f90 con 0x7fc8640d18e0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.821 7fc86977c700 1 --2- 172.21.2.117:0/2738328608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8640d18e0 0x7fc8640ea9e0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 rx=0x7fc860006650 tx=0x7fc860009a80).ready entity=mon.0 client_cookie=4244889239b37ac6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc85bfff700 1 -- 172.21.2.117:0/2738328608 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8600009b0 con 0x7fc8640d18e0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc85bfff700 1 -- 172.21.2.117:0/2738328608 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86000c0c0 con 0x7fc8640d18e0 2021-02-10T11:52:56.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc85bfff700 1 -- 172.21.2.117:0/2738328608 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc86000c520 con 0x7fc8640d18e0 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 -- 172.21.2.117:0/2738328608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8640d18e0 msgr2=0x7fc8640ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 --2- 172.21.2.117:0/2738328608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8640d18e0 0x7fc8640ea9e0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 rx=0x7fc860006650 tx=0x7fc860009a80).stop 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 -- 172.21.2.117:0/2738328608 shutdown_connections 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 --2- 172.21.2.117:0/2738328608 >> v2:172.21.2.125:3300/0 conn(0x7fc8640ef3f0 0x7fc8641b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 --2- 172.21.2.117:0/2738328608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8640d18e0 0x7fc8640ea9e0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 -- 172.21.2.117:0/2738328608 >> 172.21.2.117:0/2738328608 conn(0x7fc8640ced60 msgr2=0x7fc8640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 -- 172.21.2.117:0/2738328608 shutdown_connections 2021-02-10T11:52:56.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.822 7fc86b960700 1 -- 172.21.2.117:0/2738328608 wait complete. 2021-02-10T11:52:56.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 Processor -- start 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 -- start start 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8640b43f0 0x7fc8640e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8641b0130 0x7fc8640e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640e80e0 con 0x7fc8641afd30 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8640e7e20 con 0x7fc8640b43f0 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.837 7fc86b960700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc864096a70 con 0x7fc8641b0130 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56830/0 (socket says 172.21.2.117:56830) 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 -- 172.21.2.117:0/1743881917 learned_addr learned my addr 172.21.2.117:0/1743881917 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:56.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 -- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8641b0130 msgr2=0x7fc8640e7800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8641b0130 0x7fc8640e7800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 -- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8640b43f0 msgr2=0x7fc8640e71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8640b43f0 0x7fc8640e71e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 -- 172.21.2.117:0/1743881917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8640e8660 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86977c700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 rx=0x7fc860009630 tx=0x7fc86000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc8600093a0 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86000bd70 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc860005560 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86b960700 1 -- 172.21.2.117:0/1743881917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8640e8660 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.838 7fc86b960700 1 -- 172.21.2.117:0/1743881917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8640e8660 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.839 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc860005700 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.839 7fc859ffb700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 0x7fc84401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.839 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc8600327a0 con 0x7fc8641afd30 2021-02-10T11:52:56.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.840 7fc868f7b700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 0x7fc84401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:56.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.840 7fc868f7b700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 0x7fc84401abd0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:56.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.870 7fc86b960700 1 -- 172.21.2.117:0/1743881917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8640e7e20 con 0x7fc8641afd30 2021-02-10T11:52:56.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.873 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc8600054f0 con 0x7fc8641afd30 2021-02-10T11:52:56.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.996 7fc86b960700 1 -- 172.21.2.117:0/1743881917 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8640e7e20 con 0x7fc8641afd30 2021-02-10T11:52:56.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.996 7fc859ffb700 1 -- 172.21.2.117:0/1743881917 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fc860005700 con 0x7fc8641afd30 2021-02-10T11:52:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:56.998 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":143,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:50.464421","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:47.477707","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:48.506496","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:52:56.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 msgr2=0x7fc84401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:56.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 0x7fc84401abd0 crc :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:56.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 msgr2=0x7fc8640b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:56.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 rx=0x7fc860009630 tx=0x7fc86000b980).stop 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 shutdown_connections 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc8641b0130 0x7fc8640e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc8640b43f0 0x7fc8640e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc8440186d0 0x7fc84401abd0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 --2- 172.21.2.117:0/1743881917 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc8641afd30 0x7fc8640b3dd0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 >> 172.21.2.117:0/1743881917 conn(0x7fc8641a3620 msgr2=0x7fc8640dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 shutdown_connections 2021-02-10T11:52:57.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:56.998 7fc86b960700 1 -- 172.21.2.117:0/1743881917 wait complete. 2021-02-10T11:52:57.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:52:47.477707', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:52:48.506496', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:52:57.009 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:52:57.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.421 7f74994e1700 1 Processor -- start 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.421 7f74994e1700 1 -- start start 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f74994e1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74940d18f0 0x7f74940eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f74994e1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74940ede60 con 0x7f74940eb040 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f74994e1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74940d1cf0 con 0x7f74940ef3f0 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f74994e1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7494088020 con 0x7f74940d18f0 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f7492abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f74940eb040 0x7f74940eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57088/0 (socket says 172.21.2.117:57088) 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f7492abe700 1 -- 172.21.2.117:0/227380742 learned_addr learned my addr 172.21.2.117:0/227380742 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.422 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289718945 0 0) 0x7f74880010b0 con 0x7f74940eb040 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c002fe0 con 0x7f74940eb040 2021-02-10T11:52:57.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74932bf700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74940d18f0 0x7f74940eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3472622245 0 0) 0x7f74880010b0 con 0x7f74940eb040 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74940d18f0 msgr2=0x7f74940eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74940d18f0 0x7f74940eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 >> v1:172.21.2.125:6789/0 conn(0x7f74940ef3f0 legacy=0x7f74941b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74941bb200 con 0x7f74940eb040 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f74880015b0 con 0x7f74940eb040 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 >> v1:172.21.2.117:6789/0 conn(0x7f74940eb040 legacy=0x7f74940eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f747c003a60 0x7f747c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f747c006750 0x7f747c008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f747c009510 0x7f747c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c2c0 con 0x7f747c006750 2021-02-10T11:52:57.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c4e0 con 0x7f747c009510 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74922bd700 1 -- 172.21.2.117:0/227380742 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00c700 con 0x7f747c003a60 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f7492abe700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f747c006750 0x7f747c008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f7493ac0700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f747c009510 0x7f747c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f74932bf700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f747c003a60 0x7f747c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f7492abe700 1 -- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f747c003a60 msgr2=0x7f747c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.423 7f7492abe700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f747c003a60 0x7f747c005f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7492abe700 1 -- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f747c009510 msgr2=0x7f747c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7492abe700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f747c009510 0x7f747c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7492abe700 1 -- 172.21.2.117:0/227380742 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c050970 con 0x7f747c006750 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7493ac0700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f747c009510 0x7f747c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7492abe700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f747c006750 0x7f747c008c70 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7f7488007990 tx=0x7f7488007d30).ready entity=mon.0 client_cookie=6dfa1fbe488d619f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:57.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74880009b0 con 0x7f747c006750 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748800a310 con 0x7f747c006750 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74922bd700 1 -- 172.21.2.117:0/227380742 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748800a7a0 con 0x7f747c006750 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74994e1700 1 -- 172.21.2.117:0/227380742 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f747c006750 msgr2=0x7f747c008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74994e1700 1 --2- 172.21.2.117:0/227380742 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f747c006750 0x7f747c008c70 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 rx=0x7f7488007990 tx=0x7f7488007d30).stop 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f7493ac0700 1 -- 172.21.2.117:0/227380742 reap_dead start 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74994e1700 1 -- 172.21.2.117:0/227380742 shutdown_connections 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.424 7f74994e1700 1 -- 172.21.2.117:0/227380742 >> 172.21.2.117:0/227380742 conn(0x7f74940cf060 msgr2=0x7f74940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.425 7f74994e1700 1 -- 172.21.2.117:0/227380742 shutdown_connections 2021-02-10T11:52:57.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.425 7f74994e1700 1 -- 172.21.2.117:0/227380742 wait complete. 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.439 7f74994e1700 1 Processor -- start 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.439 7f74994e1700 1 -- start start 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74941a4180 0x7f74940dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74941a4580 0x7f74940dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494085e50 con 0x7f7494083990 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494097270 con 0x7f74941a4180 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74994e1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7494097450 con 0x7f74941a4580 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74932bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74932bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56848/0 (socket says 172.21.2.117:56848) 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f74932bf700 1 -- 172.21.2.117:0/41677998 learned_addr learned my addr 172.21.2.117:0/41677998 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f7493ac0700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74941a4580 0x7f74940dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.440 7f7492abe700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74941a4180 0x7f74940dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 -- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74941a4580 msgr2=0x7f74940dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74941a4580 0x7f74940dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 -- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74941a4180 msgr2=0x7f74940dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74941a4180 0x7f74940dbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 -- 172.21.2.117:0/41677998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749409ed70 con 0x7f7494083990 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74932bf700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f7484006f60 tx=0x7f748400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f748400bd60 con 0x7f7494083990 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7484000c90 con 0x7f7494083990 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f74840022b0 con 0x7f7494083990 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74994e1700 1 -- 172.21.2.117:0/41677998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749409ed70 con 0x7f7494083990 2021-02-10T11:52:57.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.441 7f74994e1700 1 -- 172.21.2.117:0/41677998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749409f2c0 con 0x7f7494083990 2021-02-10T11:52:57.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.442 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7484006b00 con 0x7f7494083990 2021-02-10T11:52:57.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.442 7f747bfff700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 0x7f747401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:57.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.442 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f74840326a0 con 0x7f7494083990 2021-02-10T11:52:57.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.442 7f7492abe700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 0x7f747401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:57.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.442 7f7492abe700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 0x7f747401abc0 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:57.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.472 7f74994e1700 1 -- 172.21.2.117:0/41677998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749409fb50 con 0x7f7494083990 2021-02-10T11:52:57.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.475 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f74840009b0 con 0x7f7494083990 2021-02-10T11:52:57.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.604 7f74994e1700 1 -- 172.21.2.117:0/41677998 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f749409fb50 con 0x7f7494083990 2021-02-10T11:52:57.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.604 7f747bfff700 1 -- 172.21.2.117:0/41677998 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f748400bd60 con 0x7f7494083990 2021-02-10T11:52:57.607 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:57.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 msgr2=0x7f747401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 0x7f747401abc0 crc :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 msgr2=0x7f74940863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:57.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 rx=0x7f7484006f60 tx=0x7f748400b970).stop 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 shutdown_connections 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f74941a4580 0x7f74940dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f74941a4180 0x7f74940dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f74740186c0 0x7f747401abc0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 --2- 172.21.2.117:0/41677998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7494083990 0x7f74940863d0 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 >> 172.21.2.117:0/41677998 conn(0x7f74940b6540 msgr2=0x7f749408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:57.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 shutdown_connections 2021-02-10T11:52:57.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:57.606 7f74994e1700 1 -- 172.21.2.117:0/41677998 wait complete. 2021-02-10T11:52:57.617 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.029 7f58303d5700 1 Processor -- start 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.029 7f58303d5700 1 -- start start 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58280d18f0 0x7f58280eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58280eb040 0x7f58280eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f58280ee3e0 0x7f58280ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280ede60 con 0x7f58280eb040 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280eba60 con 0x7f58280ee3e0 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f58303d5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280d1cf0 con 0x7f58280d18f0 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58280eb040 0x7f58280eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f58280eb040 0x7f58280eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56856/0 (socket says 172.21.2.117:56856) 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 -- 172.21.2.117:0/2550818542 learned_addr learned my addr 172.21.2.117:0/2550818542 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 -- 172.21.2.117:0/2550818542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58280d18f0 msgr2=0x7f58280eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 --2- 172.21.2.117:0/2550818542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58280d18f0 0x7f58280eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 -- 172.21.2.117:0/2550818542 >> v2:172.21.2.125:3300/0 conn(0x7f58280ee3e0 msgr2=0x7f58280ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 --2- 172.21.2.117:0/2550818542 >> v2:172.21.2.125:3300/0 conn(0x7f58280ee3e0 0x7f58280ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 -- 172.21.2.117:0/2550818542 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280eee00 con 0x7f58280eb040 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d9f0700 1 --2- 172.21.2.117:0/2550818542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58280eb040 0x7f58280eb440 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7f581c006700 tx=0x7f581c009b80).ready entity=mon.0 client_cookie=7c49d63b35fa276f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.030 7f582d1ef700 1 -- 172.21.2.117:0/2550818542 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581c00c0c0 con 0x7f58280eb040 2021-02-10T11:52:58.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f582d1ef700 1 -- 172.21.2.117:0/2550818542 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c00c340 con 0x7f58280eb040 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f582d1ef700 1 -- 172.21.2.117:0/2550818542 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f581c00c860 con 0x7f58280eb040 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 -- 172.21.2.117:0/2550818542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58280eb040 msgr2=0x7f58280eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 --2- 172.21.2.117:0/2550818542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58280eb040 0x7f58280eb440 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 rx=0x7f581c006700 tx=0x7f581c009b80).stop 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 -- 172.21.2.117:0/2550818542 shutdown_connections 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 --2- 172.21.2.117:0/2550818542 >> v2:172.21.2.125:3300/0 conn(0x7f58280ee3e0 0x7f58280ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 --2- 172.21.2.117:0/2550818542 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58280eb040 0x7f58280eb440 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 --2- 172.21.2.117:0/2550818542 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58280d18f0 0x7f58280eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 -- 172.21.2.117:0/2550818542 >> 172.21.2.117:0/2550818542 conn(0x7f58280cf060 msgr2=0x7f58280cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 -- 172.21.2.117:0/2550818542 shutdown_connections 2021-02-10T11:52:58.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.031 7f58303d5700 1 -- 172.21.2.117:0/2550818542 wait complete. 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 Processor -- start 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 -- start start 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58281a7ed0 0x7f582808c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f582808c850 0x7f582808cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828094280 con 0x7f5828093be0 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828093fe0 con 0x7f582808c850 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.046 7f58303d5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280ace00 con 0x7f58281a7ed0 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56860/0 (socket says 172.21.2.117:56860) 2021-02-10T11:52:58.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 -- 172.21.2.117:0/47585949 learned_addr learned my addr 172.21.2.117:0/47585949 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 -- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58281a7ed0 msgr2=0x7f582808c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58281a7ed0 0x7f582808c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 -- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f582808c850 msgr2=0x7f582808cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f582808c850 0x7f582808cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 -- 172.21.2.117:0/47585949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5828094800 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f582e1f1700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f582400b530 tx=0x7f582400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f582400bcc0 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5824000c90 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f58303d5700 1 -- 172.21.2.117:0/47585949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5828094800 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58240022a0 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.047 7f58303d5700 1 -- 172.21.2.117:0/47585949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5828094d90 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.048 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5824005310 con 0x7f5828093be0 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.048 7f581affd700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 0x7f581001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.048 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5824032900 con 0x7f5828093be0 2021-02-10T11:52:58.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.048 7f582d9f0700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 0x7f581001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.049 7f582d9f0700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 0x7f581001ac10 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.079 7f58303d5700 1 -- 172.21.2.117:0/47585949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58280ede60 con 0x7f5828093be0 2021-02-10T11:52:58.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.082 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5824000940 con 0x7f5828093be0 2021-02-10T11:52:58.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.209 7f58303d5700 1 -- 172.21.2.117:0/47585949 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f58280ede60 con 0x7f5828093be0 2021-02-10T11:52:58.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.210 7f581affd700 1 -- 172.21.2.117:0/47585949 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v143) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5824000c90 con 0x7f5828093be0 2021-02-10T11:52:58.213 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:58.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 msgr2=0x7f581001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 0x7f581001ac10 crc :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 msgr2=0x7f58281a78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 rx=0x7f582400b530 tx=0x7f582400b8d0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 shutdown_connections 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f582808c850 0x7f582808cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58281a7ed0 0x7f582808c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5810018730 0x7f581001ac10 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 --2- 172.21.2.117:0/47585949 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5828093be0 0x7f58281a78b0 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 >> 172.21.2.117:0/47585949 conn(0x7f5828092d00 msgr2=0x7f582808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 shutdown_connections 2021-02-10T11:52:58.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.212 7f58303d5700 1 -- 172.21.2.117:0/47585949 wait complete. 2021-02-10T11:52:58.223 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 Processor -- start 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 -- start start 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9140ef3f0 0x7fa9140eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9140eb830 0x7fa9141b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140ef7f0 con 0x7fa9140d18f0 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9140eded0 con 0x7fa9140ef3f0 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.635 7fa91b09c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140d1cf0 con 0x7fa9140eb830 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9140ef3f0 0x7fa9140eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9140ef3f0 0x7fa9140eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56870/0 (socket says 172.21.2.117:56870) 2021-02-10T11:52:58.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 -- 172.21.2.117:0/233225438 learned_addr learned my addr 172.21.2.117:0/233225438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa9196b9700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9140eb830 0x7fa9141b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 -- 172.21.2.117:0/233225438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9140eb830 msgr2=0x7fa9141b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9140eb830 0x7fa9141b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 -- 172.21.2.117:0/233225438 >> v1:172.21.2.125:6789/0 conn(0x7fa9140d18f0 legacy=0x7fa9140eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 -- 172.21.2.117:0/233225438 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9141b8fd0 con 0x7fa9140ef3f0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa913fff700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9140ef3f0 0x7fa9140eb210 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 rx=0x7fa904006700 tx=0x7fa904009b80).ready entity=mon.0 client_cookie=d6a448b41cb48cb0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa9137fe700 1 -- 172.21.2.117:0/233225438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa90400c0c0 con 0x7fa9140ef3f0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa9137fe700 1 -- 172.21.2.117:0/233225438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90400c440 con 0x7fa9140ef3f0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa9137fe700 1 -- 172.21.2.117:0/233225438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa90400c920 con 0x7fa9140ef3f0 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa91b09c700 1 -- 172.21.2.117:0/233225438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9140ef3f0 msgr2=0x7fa9140eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.636 7fa91b09c700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9140ef3f0 0x7fa9140eb210 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 rx=0x7fa904006700 tx=0x7fa904009b80).stop 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 -- 172.21.2.117:0/233225438 shutdown_connections 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9140eb830 0x7fa9141b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 --2- 172.21.2.117:0/233225438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9140ef3f0 0x7fa9140eb210 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 -- 172.21.2.117:0/233225438 >> 172.21.2.117:0/233225438 conn(0x7fa9140cf060 msgr2=0x7fa9140cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 -- 172.21.2.117:0/233225438 shutdown_connections 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.637 7fa91b09c700 1 -- 172.21.2.117:0/233225438 wait complete. 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 Processor -- start 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 -- start start 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa914084500 0x7fa91418f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa91408d090 0x7fa91408d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91408dab0 con 0x7fa9141a7800 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91408dcd0 con 0x7fa91408d090 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa91b09c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91408def0 con 0x7fa914084500 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa913fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa913fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56876/0 (socket says 172.21.2.117:56876) 2021-02-10T11:52:58.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.652 7fa913fff700 1 -- 172.21.2.117:0/3009051004 learned_addr learned my addr 172.21.2.117:0/3009051004 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 -- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa914084500 msgr2=0x7fa91418f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa914084500 0x7fa91418f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 -- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa91408d090 msgr2=0x7fa91408d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa91408d090 0x7fa91408d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 -- 172.21.2.117:0/3009051004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91408e110 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa913fff700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7fa90400b5e0 tx=0x7fa90400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa90400bd70 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa904002030 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91408e110 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa904002550 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.653 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914190860 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.654 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa904002210 con 0x7fa9141a7800 2021-02-10T11:52:58.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.654 7fa9117fa700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 0x7fa8f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:58.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.654 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa904032700 con 0x7fa9141a7800 2021-02-10T11:52:58.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.654 7fa918eb8700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 0x7fa8f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:58.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.655 7fa918eb8700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 0x7fa8f801abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:58.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.685 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9140ede60 con 0x7fa9141a7800 2021-02-10T11:52:58.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.688 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa90402df10 con 0x7fa9141a7800 2021-02-10T11:52:58.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.815 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa9140ede60 con 0x7fa9141a7800 2021-02-10T11:52:58.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.815 7fa9117fa700 1 -- 172.21.2.117:0/3009051004 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v143) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa90402df10 con 0x7fa9141a7800 2021-02-10T11:52:58.818 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 msgr2=0x7fa8f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 0x7fa8f801abc0 crc :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 msgr2=0x7fa9141a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 rx=0x7fa90400b5e0 tx=0x7fa90400b980).stop 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 shutdown_connections 2021-02-10T11:52:58.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8f80186c0 0x7fa8f801abc0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9141a7800 0x7fa9141a7c00 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa91408d090 0x7fa91408d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 --2- 172.21.2.117:0/3009051004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa914084500 0x7fa91418f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.817 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 >> 172.21.2.117:0/3009051004 conn(0x7fa9140b1610 msgr2=0x7fa9140bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.818 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 shutdown_connections 2021-02-10T11:52:58.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:58.818 7fa91b09c700 1 -- 172.21.2.117:0/3009051004 wait complete. 2021-02-10T11:52:58.829 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:52:58.829 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.239 7f8ccaddf700 1 Processor -- start 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 -- start start 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8cc40d18e0 0x7f8cc40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8cc40eb000 0x7f8cc40eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40ede60 con 0x7f8cc40ee3e0 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc40eba20 con 0x7f8cc40d18e0 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8ccaddf700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc40d1ce0 con 0x7f8cc40eb000 2021-02-10T11:52:59.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8cc93fc700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8cc40ee3e0 0x7f8cc40ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57132/0 (socket says 172.21.2.117:57132) 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8cc93fc700 1 -- 172.21.2.117:0/392626119 learned_addr learned my addr 172.21.2.117:0/392626119 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.240 7f8cc8bfb700 1 --2- 172.21.2.117:0/392626119 >> v2:172.21.2.117:3300/0 conn(0x7f8cc40d18e0 0x7f8cc40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc37fe700 1 -- 172.21.2.117:0/392626119 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982595907 0 0) 0x7f8cbc000d30 con 0x7f8cc40ee3e0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc3fff700 1 --2- 172.21.2.117:0/392626119 >> v2:172.21.2.125:3300/0 conn(0x7f8cc40eb000 0x7f8cc40eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc37fe700 1 -- 172.21.2.117:0/392626119 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0002fe0 con 0x7f8cc40ee3e0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc8bfb700 1 -- 172.21.2.117:0/392626119 >> v2:172.21.2.125:3300/0 conn(0x7f8cc40eb000 msgr2=0x7f8cc40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc8bfb700 1 --2- 172.21.2.117:0/392626119 >> v2:172.21.2.125:3300/0 conn(0x7f8cc40eb000 0x7f8cc40eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc8bfb700 1 -- 172.21.2.117:0/392626119 >> v1:172.21.2.117:6789/0 conn(0x7f8cc40ee3e0 legacy=0x7f8cc40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc8bfb700 1 -- 172.21.2.117:0/392626119 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40eefd0 con 0x7f8cc40d18e0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc8bfb700 1 --2- 172.21.2.117:0/392626119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc40d18e0 0x7f8cc40ea9e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7f8cac006650 tx=0x7f8cac009a80).ready entity=mon.0 client_cookie=f4b1ed91c8042f56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc37fe700 1 -- 172.21.2.117:0/392626119 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac0009b0 con 0x7f8cc40d18e0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc37fe700 1 -- 172.21.2.117:0/392626119 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00c0c0 con 0x7f8cc40d18e0 2021-02-10T11:52:59.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8cc37fe700 1 -- 172.21.2.117:0/392626119 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac00c5e0 con 0x7f8cc40d18e0 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8ccaddf700 1 -- 172.21.2.117:0/392626119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc40d18e0 msgr2=0x7f8cc40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.241 7f8ccaddf700 1 --2- 172.21.2.117:0/392626119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc40d18e0 0x7f8cc40ea9e0 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 rx=0x7f8cac006650 tx=0x7f8cac009a80).stop 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 -- 172.21.2.117:0/392626119 shutdown_connections 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 --2- 172.21.2.117:0/392626119 >> v2:172.21.2.125:3300/0 conn(0x7f8cc40eb000 0x7f8cc40eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 --2- 172.21.2.117:0/392626119 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc40d18e0 0x7f8cc40ea9e0 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 -- 172.21.2.117:0/392626119 >> 172.21.2.117:0/392626119 conn(0x7f8cc40ced60 msgr2=0x7f8cc40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 -- 172.21.2.117:0/392626119 shutdown_connections 2021-02-10T11:52:59.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.242 7f8ccaddf700 1 -- 172.21.2.117:0/392626119 wait complete. 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 Processor -- start 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 -- start start 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cc41aeb20 0x7f8cc419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cc40b2c30 0x7f8cc40b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc41b87a0 con 0x7f8cc41af4e0 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc41aef90 con 0x7f8cc40b2c30 2021-02-10T11:52:59.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.257 7f8ccaddf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc419c760 con 0x7f8cc41aeb20 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56886/0 (socket says 172.21.2.117:56886) 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 -- 172.21.2.117:0/3517513402 learned_addr learned my addr 172.21.2.117:0/3517513402 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc3fff700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cc41aeb20 0x7f8cc419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc93fc700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cc40b2c30 0x7f8cc40b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 -- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cc41aeb20 msgr2=0x7f8cc419c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cc41aeb20 0x7f8cc419c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 -- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cc40b2c30 msgr2=0x7f8cc40b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cc40b2c30 0x7f8cc40b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 -- 172.21.2.117:0/3517513402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b3650 con 0x7f8cc41af4e0 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc8bfb700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7f8cac009630 tx=0x7f8cac00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:59.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac0093a0 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cac00bd70 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.258 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40b3650 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.259 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8cac005560 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.259 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4093840 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.259 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8cac0052e0 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.259 7f8cc17fa700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.259 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8cac0326e0 con 0x7f8cc41af4e0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.260 7f8cc3fff700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:52:59.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.260 7f8cc3fff700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 0x7f8ca801abc0 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:52:59.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.290 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc4087290 con 0x7f8cc41af4e0 2021-02-10T11:52:59.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.292 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8cac030d40 con 0x7f8cc41af4e0 2021-02-10T11:52:59.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.420 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cc40ede60 con 0x7f8cc41af4e0 2021-02-10T11:52:59.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.420 7f8cc17fa700 1 -- 172.21.2.117:0/3517513402 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f8cac0052e0 con 0x7f8cc41af4e0 2021-02-10T11:52:59.423 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:52:59.423 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:52:51.468783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7415":{"gid":7415,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/154249133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":154249133},{"type":"v1","addr":"172.21.2.125:6835","nonce":154249133}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:52:59.423 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 120 2021-02-10T11:52:59.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 msgr2=0x7f8ca801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 0x7f8ca801abc0 crc :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 msgr2=0x7f8cc419b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:52:59.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 rx=0x7f8cac009630 tx=0x7f8cac00b980).stop 2021-02-10T11:52:59.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 shutdown_connections 2021-02-10T11:52:59.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8cc40b2c30 0x7f8cc40b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8cc41aeb20 0x7f8cc419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ca80186c0 0x7f8ca801abc0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 --2- 172.21.2.117:0/3517513402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8cc41af4e0 0x7f8cc419b900 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:52:59.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 >> 172.21.2.117:0/3517513402 conn(0x7f8cc41a3620 msgr2=0x7f8cc4092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:52:59.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 shutdown_connections 2021-02-10T11:52:59.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:52:59.422 7f8ccaddf700 1 -- 172.21.2.117:0/3517513402 wait complete. 2021-02-10T11:52:59.635 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 120, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:49.462120', 'modified': '2021-02-10 11:52:51.468783', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7415': {'gid': 7415, 'name': 'b', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/154249133', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 154249133}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 154249133}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:52:59.635 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:52:59.635 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:53:00.033 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:53:00.033 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7415, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.567358387000001, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 120, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 143, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.211390169000001 2021-02-10T11:53:00.034 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:53:00.037 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7415, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.567358387000001, "mdsmap_epoch": 120, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 16.211390169000001 } 2021-02-10T11:53:00.038 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:53:00.038 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:53:00.038 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:53:00.038 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:00.038 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:00.374 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:00.719 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:00.720 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:01.063 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:01.063 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:01.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:01.064 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:01.398 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:53:01.399 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:01.401 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:01.744 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:01.744 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:01.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[173305]: starting ceph client 2021-02-10T11:53:01.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:53:01.808 7f985fb31f80 -1 init, newargv = 0x55e62bcadb90 newargc=9 2021-02-10T11:53:01.817 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[173305]: starting fuse 2021-02-10T11:53:02.086 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:02.086 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:02.088 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:02.088 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:02.424 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:02.424 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:53:02.424 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:02.424 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:53:02.424 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:53:02.424 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:53:02.424 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:53:02.425 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:53:02.426 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:53:02.426 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:02.426 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:53:02.426 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:53:02.793 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.173305.asok 2021-02-10T11:53:02.796 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.173305.asok 2021-02-10T11:53:02.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:02.797 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.173305.asok status 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:53:03.189 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "173305", 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11907, 2021-02-10T11:53:03.190 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11907 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2547953603 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:03.191 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 2547953603 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11907 172.21.2.117:0/2547953603", 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/2547953603", 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 120, 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 143, 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:53:03.192 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:03.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:03.194 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:03.194 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:03.529 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:03.529 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:53:03.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:03.530 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:03.877 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:04.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.285 7fa7654f2700 1 Processor -- start 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 -- start start 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa7600ef3f0 0x7fa7600eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7600eb7f0 0x7fa7601b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7600ede60 con 0x7fa7600d18e0 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7600ef7f0 con 0x7fa7600eb7f0 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa7654f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7600d1ce0 con 0x7fa7600ef3f0 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa75fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7600eb7f0 0x7fa7601b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa75fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7600eb7f0 0x7fa7601b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56906/0 (socket says 172.21.2.117:56906) 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.286 7fa75fac0700 1 -- 172.21.2.117:0/1417598547 learned_addr learned my addr 172.21.2.117:0/1417598547 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75fac0700 1 -- 172.21.2.117:0/1417598547 >> v2:172.21.2.125:3300/0 conn(0x7fa7600ef3f0 msgr2=0x7fa7600eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:04.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75fac0700 1 --2- 172.21.2.117:0/1417598547 >> v2:172.21.2.125:3300/0 conn(0x7fa7600ef3f0 0x7fa7600eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75fac0700 1 -- 172.21.2.117:0/1417598547 >> v1:172.21.2.125:6789/0 conn(0x7fa7600d18e0 legacy=0x7fa7600ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75fac0700 1 -- 172.21.2.117:0/1417598547 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7601b8f90 con 0x7fa7600eb7f0 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75fac0700 1 --2- 172.21.2.117:0/1417598547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600eb7f0 0x7fa7601b7960 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 rx=0x7fa758006f00 tx=0x7fa758009f10).ready entity=mon.0 client_cookie=20081be3689ddcc5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75eabe700 1 -- 172.21.2.117:0/1417598547 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa75800b3b0 con 0x7fa7600eb7f0 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75eabe700 1 -- 172.21.2.117:0/1417598547 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800b630 con 0x7fa7600eb7f0 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa75eabe700 1 -- 172.21.2.117:0/1417598547 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa75800bb50 con 0x7fa7600eb7f0 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa7654f2700 1 -- 172.21.2.117:0/1417598547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600eb7f0 msgr2=0x7fa7601b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.287 7fa7654f2700 1 --2- 172.21.2.117:0/1417598547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600eb7f0 0x7fa7601b7960 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 rx=0x7fa758006f00 tx=0x7fa758009f10).stop 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 -- 172.21.2.117:0/1417598547 shutdown_connections 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 --2- 172.21.2.117:0/1417598547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600eb7f0 0x7fa7601b7960 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 --2- 172.21.2.117:0/1417598547 >> v2:172.21.2.125:3300/0 conn(0x7fa7600ef3f0 0x7fa7600eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 -- 172.21.2.117:0/1417598547 >> 172.21.2.117:0/1417598547 conn(0x7fa7600ced60 msgr2=0x7fa7600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 -- 172.21.2.117:0/1417598547 shutdown_connections 2021-02-10T11:53:04.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.288 7fa7654f2700 1 -- 172.21.2.117:0/1417598547 wait complete. 2021-02-10T11:53:04.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.302 7fa7654f2700 1 Processor -- start 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.302 7fa7654f2700 1 -- start start 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa760196390 0x7fa7600e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7600e9960 0x7fa7600e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760196810 con 0x7fa7600e9560 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760098ad0 con 0x7fa760196390 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa7654f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600e8a40 con 0x7fa7600e9960 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56910/0 (socket says 172.21.2.117:56910) 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 -- 172.21.2.117:0/3349878434 learned_addr learned my addr 172.21.2.117:0/3349878434 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 -- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7600e9960 msgr2=0x7fa7600e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7600e9960 0x7fa7600e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 -- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa760196390 msgr2=0x7fa7600e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.303 7fa756abe700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa760196390 0x7fa7600e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa756abe700 1 -- 172.21.2.117:0/3349878434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600e8c60 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa756abe700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7fa74800b530 tx=0x7fa74800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa74800bcc0 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa748000c90 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7480022a0 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7600e8c60 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.304 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76019cd70 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.305 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa748005310 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.305 7fa75caba700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 0x7fa74401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.305 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa7480326b0 con 0x7fa7600e9560 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.305 7fa75f2bf700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 0x7fa74401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.306 7fa75f2bf700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 0x7fa74401abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.336 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa760087210 con 0x7fa7600e9560 2021-02-10T11:53:04.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.339 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa748022620 con 0x7fa7600e9560 2021-02-10T11:53:04.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.462 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7600ede60 con 0x7fa7600e9560 2021-02-10T11:53:04.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.463 7fa75caba700 1 -- 172.21.2.117:0/3349878434 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fa748000940 con 0x7fa7600e9560 2021-02-10T11:53:04.466 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:04.466 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":143,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:52:50.464421","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:52:47.477707","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":31,"pool_name":"cephfs_data","create_time":"2021-02-10 11:52:48.506496","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"143","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.464 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 msgr2=0x7fa74401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.464 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 0x7fa74401abc0 crc :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.464 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 msgr2=0x7fa7600e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.464 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 rx=0x7fa74800b530 tx=0x7fa74800b8d0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 shutdown_connections 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7440186c0 0x7fa74401abc0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7600e9560 0x7fa7600e7e00 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7600e9960 0x7fa7600e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 --2- 172.21.2.117:0/3349878434 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa760196390 0x7fa7600e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 >> 172.21.2.117:0/3349878434 conn(0x7fa76019f610 msgr2=0x7fa7601a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:04.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 shutdown_connections 2021-02-10T11:53:04.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.465 7fa7654f2700 1 -- 172.21.2.117:0/3349878434 wait complete. 2021-02-10T11:53:04.476 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:52:47.477707', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:52:48.506496', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:53:04.476 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 Processor -- start 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 -- start start 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 0x7f0bfc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0bfc0eb040 0x7f0bfc0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc0ede60 con 0x7f0bfc0ee3e0 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bfc0eba60 con 0x7f0bfc0eb040 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.889 7f0c029d2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc0d1cf0 con 0x7f0bfc0d18f0 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.890 7f0bfb7fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0bfc0eb040 0x7f0bfc0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.890 7f0bfbfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 0x7f0bfc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0c00fef700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f0bfc0ee3e0 0x7f0bfc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50094/0 (socket says 172.21.2.117:50094) 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0c00fef700 1 -- 172.21.2.117:0/2573813298 learned_addr learned my addr 172.21.2.117:0/2573813298 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfb7fe700 1 -- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 msgr2=0x7f0bfc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfb7fe700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 0x7f0bfc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfb7fe700 1 -- 172.21.2.117:0/2573813298 >> v1:172.21.2.125:6789/0 conn(0x7f0bfc0ee3e0 legacy=0x7f0bfc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfb7fe700 1 -- 172.21.2.117:0/2573813298 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc0eefd0 con 0x7f0bfc0eb040 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfbfff700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 0x7f0bfc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.891 7f0bfb7fe700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc0eb040 0x7f0bfc0eb440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f0bf0006650 tx=0x7f0bf0009a80).ready entity=mon.1 client_cookie=370d9230dbbc10ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0bfaffd700 1 -- 172.21.2.117:0/2573813298 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf00009b0 con 0x7f0bfc0eb040 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0bfaffd700 1 -- 172.21.2.117:0/2573813298 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf000c0c0 con 0x7f0bfc0eb040 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0bfaffd700 1 -- 172.21.2.117:0/2573813298 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf000c5e0 con 0x7f0bfc0eb040 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 -- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc0eb040 msgr2=0x7f0bfc0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc0eb040 0x7f0bfc0eb440 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f0bf0006650 tx=0x7f0bf0009a80).stop 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 -- 172.21.2.117:0/2573813298 shutdown_connections 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc0eb040 0x7f0bfc0eb440 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 --2- 172.21.2.117:0/2573813298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc0d18f0 0x7f0bfc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 -- 172.21.2.117:0/2573813298 >> 172.21.2.117:0/2573813298 conn(0x7f0bfc0cf060 msgr2=0x7f0bfc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 -- 172.21.2.117:0/2573813298 shutdown_connections 2021-02-10T11:53:04.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.892 7f0c029d2700 1 -- 172.21.2.117:0/2573813298 wait complete. 2021-02-10T11:53:04.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 Processor -- start 2021-02-10T11:53:04.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 -- start start 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 0x7f0bfc1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc09f790 0x7f0bfc0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc09fb90 con 0x7f0bfc09ed70 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc0dc640 con 0x7f0bfc1a4240 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0c029d2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bfc097270 con 0x7f0bfc09f790 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0bfb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0bfb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56924/0 (socket says 172.21.2.117:56924) 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.907 7f0bfb7fe700 1 -- 172.21.2.117:0/4280992614 learned_addr learned my addr 172.21.2.117:0/4280992614 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0c00fef700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc09f790 0x7f0bfc0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfbfff700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 0x7f0bfc1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 -- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc09f790 msgr2=0x7f0bfc0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc09f790 0x7f0bfc0b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 -- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 msgr2=0x7f0bfc1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 0x7f0bfc1a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 -- 172.21.2.117:0/4280992614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc0b8780 con 0x7f0bfc09ed70 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfbfff700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 0x7f0bfc1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bfb7fe700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7f0bf0009630 tx=0x7f0bf000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf00093a0 con 0x7f0bfc09ed70 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf000bd70 con 0x7f0bfc09ed70 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.908 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc0b8780 con 0x7f0bfc09ed70 2021-02-10T11:53:04.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.909 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bf0005560 con 0x7f0bfc09ed70 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.909 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc0b8780 con 0x7f0bfc09ed70 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.909 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bf00052e0 con 0x7f0bfc09ed70 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.909 7f0bf8ff9700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 0x7f0bdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.909 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0bf0032920 con 0x7f0bfc09ed70 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.910 7f0bfbfff700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 0x7f0bdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:04.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.911 7f0bfbfff700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 0x7f0bdc01abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:04.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.939 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bfc09fb90 con 0x7f0bfc09ed70 2021-02-10T11:53:04.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:04.942 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bf001f510 con 0x7f0bfc09ed70 2021-02-10T11:53:05.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.069 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0bfc09fb90 con 0x7f0bfc09ed70 2021-02-10T11:53:05.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.070 7f0bf8ff9700 1 -- 172.21.2.117:0/4280992614 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0bf000bd70 con 0x7f0bfc09ed70 2021-02-10T11:53:05.073 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 msgr2=0x7f0bdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 0x7f0bdc01abc0 crc :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 msgr2=0x7f0bfc09f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 rx=0x7f0bf0009630 tx=0x7f0bf000b980).stop 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 shutdown_connections 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bdc0186c0 0x7f0bdc01abc0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bfc09ed70 0x7f0bfc09f170 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bfc09f790 0x7f0bfc0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 --2- 172.21.2.117:0/4280992614 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bfc1a4240 0x7f0bfc1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 >> 172.21.2.117:0/4280992614 conn(0x7f0bfc087090 msgr2=0x7f0bfc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 shutdown_connections 2021-02-10T11:53:05.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.072 7f0c029d2700 1 -- 172.21.2.117:0/4280992614 wait complete. 2021-02-10T11:53:05.083 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 Processor -- start 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 -- start start 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c0ef3f0 0x7fce0c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce0c0eb830 0x7fce0c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0c0ef7f0 con 0x7fce0c0d18f0 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce0c0eded0 con 0x7fce0c0eb830 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce12e4e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0d1cf0 con 0x7fce0c0ef3f0 2021-02-10T11:53:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce1146b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce0c0eb830 0x7fce0c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.495 7fce1146b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce0c0eb830 0x7fce0c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56932/0 (socket says 172.21.2.117:56932) 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 -- 172.21.2.117:0/1657148158 learned_addr learned my addr 172.21.2.117:0/1657148158 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 -- 172.21.2.117:0/1657148158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c0ef3f0 msgr2=0x7fce0c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 --2- 172.21.2.117:0/1657148158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c0ef3f0 0x7fce0c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 -- 172.21.2.117:0/1657148158 >> v1:172.21.2.125:6789/0 conn(0x7fce0c0d18f0 legacy=0x7fce0c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 -- 172.21.2.117:0/1657148158 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c1b8fd0 con 0x7fce0c0eb830 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce1146b700 1 --2- 172.21.2.117:0/1657148158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c0eb830 0x7fce0c1b79a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7fce04006b80 tx=0x7fce04009b90).ready entity=mon.0 client_cookie=d0cb0dc5c2c1b3e5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce0b7fe700 1 -- 172.21.2.117:0/1657148158 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce040022e0 con 0x7fce0c0eb830 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce0b7fe700 1 -- 172.21.2.117:0/1657148158 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04002560 con 0x7fce0c0eb830 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce0b7fe700 1 -- 172.21.2.117:0/1657148158 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce040029c0 con 0x7fce0c0eb830 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce12e4e700 1 -- 172.21.2.117:0/1657148158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c0eb830 msgr2=0x7fce0c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.496 7fce12e4e700 1 --2- 172.21.2.117:0/1657148158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c0eb830 0x7fce0c1b79a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7fce04006b80 tx=0x7fce04009b90).stop 2021-02-10T11:53:05.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 -- 172.21.2.117:0/1657148158 shutdown_connections 2021-02-10T11:53:05.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 --2- 172.21.2.117:0/1657148158 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c0eb830 0x7fce0c1b79a0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 --2- 172.21.2.117:0/1657148158 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c0ef3f0 0x7fce0c0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:05.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 -- 172.21.2.117:0/1657148158 >> 172.21.2.117:0/1657148158 conn(0x7fce0c0cf060 msgr2=0x7fce0c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:05.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 -- 172.21.2.117:0/1657148158 shutdown_connections 2021-02-10T11:53:05.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.497 7fce12e4e700 1 -- 172.21.2.117:0/1657148158 wait complete. 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 Processor -- start 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 -- start start 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 0x7fce0c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 0x7fce0c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c07cca0 con 0x7fce0c1a3930 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c18fb70 con 0x7fce0c0a13a0 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce12e4e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c0dc6d0 con 0x7fce0c07c7c0 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce0bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce0bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56936/0 (socket says 172.21.2.117:56936) 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.512 7fce0bfff700 1 -- 172.21.2.117:0/1253442969 learned_addr learned my addr 172.21.2.117:0/1253442969 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:05.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce10c6a700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 0x7fce0c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 -- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 msgr2=0x7fce0c1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce1146b700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 0x7fce0c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 0x7fce0c1a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 -- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 msgr2=0x7fce0c0a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 0x7fce0c0a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 -- 172.21.2.117:0/1253442969 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c07d220 con 0x7fce0c1a3930 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce10c6a700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 0x7fce0c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce1146b700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 0x7fce0c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce0bfff700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7fcdfc009980 tx=0x7fcdfc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdfc0066d0 con 0x7fce0c1a3930 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc00c2f0 con 0x7fce0c1a3930 2021-02-10T11:53:05.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c07d220 con 0x7fce0c1a3930 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c07d7b0 con 0x7fce0c1a3930 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.513 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcdfc00c810 con 0x7fce0c1a3930 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.514 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcdfc001010 con 0x7fce0c1a3930 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.514 7fce097fa700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 0x7fcdf001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.514 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcdfc032a40 con 0x7fce0c1a3930 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.514 7fce10c6a700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 0x7fcdf001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:05.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.515 7fce10c6a700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 0x7fcdf001abc0 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:05.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.545 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c0daff0 con 0x7fce0c1a3930 2021-02-10T11:53:05.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.548 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcdfc000cc0 con 0x7fce0c1a3930 2021-02-10T11:53:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.674 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fce0c0ede60 con 0x7fce0c1a3930 2021-02-10T11:53:05.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.675 7fce097fa700 1 -- 172.21.2.117:0/1253442969 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v143) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcdfc001010 con 0x7fce0c1a3930 2021-02-10T11:53:05.678 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 msgr2=0x7fcdf001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 0x7fcdf001abc0 crc :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 msgr2=0x7fce0c0a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:05.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 rx=0x7fcdfc009980 tx=0x7fcdfc00bcd0).stop 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 shutdown_connections 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcdf00186c0 0x7fcdf001abc0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce0c1a3930 0x7fce0c0a0d80 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce0c0a13a0 0x7fce0c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 --2- 172.21.2.117:0/1253442969 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce0c07c7c0 0x7fce0c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 >> 172.21.2.117:0/1253442969 conn(0x7fce0c087090 msgr2=0x7fce0c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 shutdown_connections 2021-02-10T11:53:05.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:05.677 7fce12e4e700 1 -- 172.21.2.117:0/1253442969 wait complete. 2021-02-10T11:53:05.688 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 Processor -- start 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 -- start start 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d280d18f0 0x7f8d280eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8d280eb040 0x7f8d280eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280ede60 con 0x7f8d280ee3e0 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d280ebad0 con 0x7f8d280d18f0 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2fd43700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d280d1cf0 con 0x7f8d280eb040 2021-02-10T11:53:06.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.100 7f8d2db5f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d280d18f0 0x7f8d280eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d280d18f0 0x7f8d280eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56944/0 (socket says 172.21.2.117:56944) 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 -- 172.21.2.117:0/3763226354 learned_addr learned my addr 172.21.2.117:0/3763226354 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 -- 172.21.2.117:0/3763226354 >> v2:172.21.2.125:3300/0 conn(0x7f8d280eb040 msgr2=0x7f8d280eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 --2- 172.21.2.117:0/3763226354 >> v2:172.21.2.125:3300/0 conn(0x7f8d280eb040 0x7f8d280eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 -- 172.21.2.117:0/3763226354 >> v1:172.21.2.125:6789/0 conn(0x7f8d280ee3e0 legacy=0x7f8d280ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 -- 172.21.2.117:0/3763226354 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d280eefd0 con 0x7f8d280d18f0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2db5f700 1 --2- 172.21.2.117:0/3763226354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d280d18f0 0x7f8d280eaa20 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 rx=0x7f8d24006700 tx=0x7f8d24009b80).ready entity=mon.0 client_cookie=c6a4819899e1dbd4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2cb5d700 1 -- 172.21.2.117:0/3763226354 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2400c0c0 con 0x7f8d280d18f0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2cb5d700 1 -- 172.21.2.117:0/3763226354 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2400c340 con 0x7f8d280d18f0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2cb5d700 1 -- 172.21.2.117:0/3763226354 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2400c860 con 0x7f8d280d18f0 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2fd43700 1 -- 172.21.2.117:0/3763226354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d280d18f0 msgr2=0x7f8d280eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.101 7f8d2fd43700 1 --2- 172.21.2.117:0/3763226354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d280d18f0 0x7f8d280eaa20 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 rx=0x7f8d24006700 tx=0x7f8d24009b80).stop 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 -- 172.21.2.117:0/3763226354 shutdown_connections 2021-02-10T11:53:06.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 --2- 172.21.2.117:0/3763226354 >> v2:172.21.2.125:3300/0 conn(0x7f8d280eb040 0x7f8d280eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 --2- 172.21.2.117:0/3763226354 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d280d18f0 0x7f8d280eaa20 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 -- 172.21.2.117:0/3763226354 >> 172.21.2.117:0/3763226354 conn(0x7f8d280cf060 msgr2=0x7f8d280cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 -- 172.21.2.117:0/3763226354 shutdown_connections 2021-02-10T11:53:06.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.102 7f8d2fd43700 1 -- 172.21.2.117:0/3763226354 wait complete. 2021-02-10T11:53:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 Processor -- start 2021-02-10T11:53:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 -- start start 2021-02-10T11:53:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d2809ed70 0x7f8d2809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2809f790 0x7f8d280b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2809fb90 con 0x7f8d281a4240 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d280dc640 con 0x7f8d2809ed70 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2fd43700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d28097270 con 0x7f8d2809f790 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2db5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2db5f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56946/0 (socket says 172.21.2.117:56946) 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.117 7f8d2db5f700 1 -- 172.21.2.117:0/2423511877 learned_addr learned my addr 172.21.2.117:0/2423511877 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 -- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2809f790 msgr2=0x7f8d280b8160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2809f790 0x7f8d280b8160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 -- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d2809ed70 msgr2=0x7f8d2809f170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d2809ed70 0x7f8d2809f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 -- 172.21.2.117:0/2423511877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d280b8780 con 0x7f8d281a4240 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2db5f700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f8d2400b5e0 tx=0x7f8d2400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d2400bd70 con 0x7f8d281a4240 2021-02-10T11:53:06.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d24002030 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d24002480 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d280b8780 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.118 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d280b8780 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.119 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d24002210 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.119 7f8d1e7fc700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 0x7f8d1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.119 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d24032560 con 0x7f8d281a4240 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.120 7f8d2d35e700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 0x7f8d1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.120 7f8d2d35e700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 0x7f8d1001abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.151 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d280daff0 con 0x7f8d281a4240 2021-02-10T11:53:06.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.153 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d24030d40 con 0x7f8d281a4240 2021-02-10T11:53:06.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.279 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8d280daff0 con 0x7f8d281a4240 2021-02-10T11:53:06.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.279 7f8d1e7fc700 1 -- 172.21.2.117:0/2423511877 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v143) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8d2400bd70 con 0x7f8d281a4240 2021-02-10T11:53:06.282 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.281 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 msgr2=0x7f8d1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.281 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 0x7f8d1001abc0 crc :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.281 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 msgr2=0x7f8d281a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.281 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 rx=0x7f8d2400b5e0 tx=0x7f8d2400b980).stop 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 shutdown_connections 2021-02-10T11:53:06.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d2809f790 0x7f8d280b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d2809ed70 0x7f8d2809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d100186c0 0x7f8d1001abc0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 --2- 172.21.2.117:0/2423511877 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d281a4240 0x7f8d281a4640 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 >> 172.21.2.117:0/2423511877 conn(0x7f8d28087090 msgr2=0x7f8d2808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 shutdown_connections 2021-02-10T11:53:06.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.282 7f8d2fd43700 1 -- 172.21.2.117:0/2423511877 wait complete. 2021-02-10T11:53:06.293 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:53:06.293 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.702 7f1eb5cda700 1 Processor -- start 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.702 7f1eb5cda700 1 -- start start 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eb5cda700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1eb00ef3f0 0x7f1eb00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eb5cda700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 0x7f1eb01b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eb5cda700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00ede60 con 0x7f1eb00d18e0 2021-02-10T11:53:06.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eb5cda700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00ef7f0 con 0x7f1eb00ef3f0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eb5cda700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00d1ce0 con 0x7f1eb00eb7f0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eaf7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1eb00d18e0 0x7f1eb00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57204/0 (socket says 172.21.2.117:57204) 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eaf7fe700 1 -- 172.21.2.117:0/2246093408 learned_addr learned my addr 172.21.2.117:0/2246093408 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eaeffd700 1 --2- 172.21.2.117:0/2246093408 >> v2:172.21.2.125:3300/0 conn(0x7f1eb00ef3f0 0x7f1eb00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.703 7f1eaffff700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 0x7f1eb01b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eae7fc700 1 -- 172.21.2.117:0/2246093408 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204114955 0 0) 0x7f1ea0000ce0 con 0x7f1eb00d18e0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eae7fc700 1 -- 172.21.2.117:0/2246093408 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c002fe0 con 0x7f1eb00d18e0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaeffd700 1 -- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 msgr2=0x7f1eb01b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaeffd700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 0x7f1eb01b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaeffd700 1 -- 172.21.2.117:0/2246093408 >> v1:172.21.2.117:6789/0 conn(0x7f1eb00d18e0 legacy=0x7f1eb00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaeffd700 1 -- 172.21.2.117:0/2246093408 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb01b8f90 con 0x7f1eb00ef3f0 2021-02-10T11:53:06.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaffff700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 0x7f1eb01b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eaeffd700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00ef3f0 0x7f1eb00eb1d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f1e98006650 tx=0x7f1e98009a80).ready entity=mon.1 client_cookie=d8ecec1e71534b51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eae7fc700 1 -- 172.21.2.117:0/2246093408 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e98006390 con 0x7f1eb00ef3f0 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eae7fc700 1 -- 172.21.2.117:0/2246093408 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e980009b0 con 0x7f1eb00ef3f0 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eb5cda700 1 -- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00ef3f0 msgr2=0x7f1eb00eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eb5cda700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00ef3f0 0x7f1eb00eb1d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f1e98006650 tx=0x7f1e98009a80).stop 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.704 7f1eae7fc700 1 -- 172.21.2.117:0/2246093408 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e9800c2b0 con 0x7f1eb00ef3f0 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 -- 172.21.2.117:0/2246093408 shutdown_connections 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb00eb7f0 0x7f1eb01b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 --2- 172.21.2.117:0/2246093408 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00ef3f0 0x7f1eb00eb1d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 -- 172.21.2.117:0/2246093408 >> 172.21.2.117:0/2246093408 conn(0x7f1eb00ced60 msgr2=0x7f1eb00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 -- 172.21.2.117:0/2246093408 shutdown_connections 2021-02-10T11:53:06.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.705 7f1eb5cda700 1 -- 172.21.2.117:0/2246093408 wait complete. 2021-02-10T11:53:06.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.720 7f1eb5cda700 1 Processor -- start 2021-02-10T11:53:06.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.720 7f1eb5cda700 1 -- start start 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 0x7f1eb00b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 0x7f1eb00e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00e80e0 con 0x7f1eb01afd30 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00e7e20 con 0x7f1eb00b43f0 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eb5cda700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0096a70 con 0x7f1eb01b0130 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaeffd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 0x7f1eb00e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 0x7f1eb00b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54640/0 (socket says 172.21.2.117:54640) 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 -- 172.21.2.117:0/1976375764 learned_addr learned my addr 172.21.2.117:0/1976375764 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaeffd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 0x7f1eb00e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:33112/0 (socket says 172.21.2.117:33112) 2021-02-10T11:53:06.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 -- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 msgr2=0x7f1eb00e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 0x7f1eb00e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 -- 172.21.2.117:0/1976375764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 msgr2=0x7f1eb00b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 0x7f1eb00b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.721 7f1eaffff700 1 -- 172.21.2.117:0/1976375764 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00e8660 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1eaf7fe700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 0x7f1eb00b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1eaffff700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1ea800b970 tx=0x7f1ea800bd10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea8000f90 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea8006790 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ea80015f0 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb00e8660 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.722 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb00e8660 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.723 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ea8000df0 con 0x7f1eb01b0130 2021-02-10T11:53:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.723 7f1ea7fff700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:06.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.723 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1ea80329d0 con 0x7f1eb01b0130 2021-02-10T11:53:06.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.723 7f1eaf7fe700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:06.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.724 7f1eaf7fe700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 0x7f1e9401abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:06.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.753 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00e7e20 con 0x7f1eb01b0130 2021-02-10T11:53:06.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.756 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ea801dee0 con 0x7f1eb01b0130 2021-02-10T11:53:06.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.887 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb00e7e20 con 0x7f1eb01b0130 2021-02-10T11:53:06.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.889 7f1ea7fff700 1 -- 172.21.2.117:0/1976375764 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f1ea801dee0 con 0x7f1eb01b0130 2021-02-10T11:53:06.891 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 120 2021-02-10T11:53:06.891 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:06.891 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":120,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:52:51.468783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7415":{"gid":7415,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/154249133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":154249133},{"type":"v1","addr":"172.21.2.125:6835","nonce":154249133}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:53:06.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.889 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 msgr2=0x7f1e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.892 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.889 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 0x7f1e9401abc0 crc :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.889 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 msgr2=0x7f1eb00e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.889 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1ea800b970 tx=0x7f1ea800bd10).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 shutdown_connections 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1eb01b0130 0x7f1eb00e7800 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1eb00b43f0 0x7f1eb00e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e940186c0 0x7f1e9401abc0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 --2- 172.21.2.117:0/1976375764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1eb01afd30 0x7f1eb00b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 >> 172.21.2.117:0/1976375764 conn(0x7f1eb01a3620 msgr2=0x7f1eb00dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 shutdown_connections 2021-02-10T11:53:06.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:06.890 7f1eb5cda700 1 -- 172.21.2.117:0/1976375764 wait complete. 2021-02-10T11:53:07.102 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 120, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:52:49.462120', 'modified': '2021-02-10 11:52:51.468783', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7415}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7415': {'gid': 7415, 'name': 'b', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/154249133', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 154249133}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 154249133}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:53:07.103 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:53:07.103 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:53:07.502 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:53:07.502 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:07.502 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:53:07.502 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7415, 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.036088533000001, 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 120, 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 143, 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 143, 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.680118615000001 2021-02-10T11:53:07.503 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:53:07.507 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7415, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.036088533000001, "mdsmap_epoch": 120, "osdmap_epoch": 143, "osdmap_epoch_barrier": 143, "uptime": 23.680118615000001 } 2021-02-10T11:53:07.507 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:53:07.507 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:53:07.507 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:53:07.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:07.507 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:07.843 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:08.188 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:08.189 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:08.533 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:08.533 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:08.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:08.534 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:08.869 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:08.869 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:53:08.870 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:08.872 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:09.216 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:09.216 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:09.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[173946]: starting ceph client 2021-02-10T11:53:09.283 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:53:09.280 7fc7e1b4cf80 -1 init, newargv = 0x55c6c7c44f30 newargc=9 2021-02-10T11:53:09.287 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[173946]: starting fuse 2021-02-10T11:53:09.559 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:09.559 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:09.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:09.561 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:09.894 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:09.895 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:53:09.895 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:53:09.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:09.895 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:53:09.895 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:53:09.895 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:53:09.895 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:53:09.896 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:53:09.897 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:53:10.259 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.173946.asok 2021-02-10T11:53:10.262 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.173946.asok 2021-02-10T11:53:10.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:10.262 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.173946.asok status 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "173946", 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:10.653 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 11940, 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 11940 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3382210501 2021-02-10T11:53:10.654 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3382210501 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.11940 172.21.2.117:0/3382210501", 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3382210501", 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 120, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 143, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:53:10.655 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:10.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:10.657 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:10.657 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:10.993 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:10.994 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:53:10.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:10.994 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:11.342 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.756 7fe9e37a4700 1 Processor -- start 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.756 7fe9e37a4700 1 -- start start 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e37a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e37a4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe9dc084c60 0x7fe9dc085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e37a4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc085930 con 0x7fe9dc0e4e40 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e37a4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9dc085680 con 0x7fe9dc084c60 2021-02-10T11:53:11.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e37a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc1b09e0 con 0x7fe9dc0e5240 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e0dbf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e0dbf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54658/0 (socket says 172.21.2.117:54658) 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e0dbf700 1 -- 172.21.2.117:0/1218166039 learned_addr learned my addr 172.21.2.117:0/1218166039 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.757 7fe9e1dc1700 1 --2- 172.21.2.117:0/1218166039 >> v2:172.21.2.125:3300/0 conn(0x7fe9dc084c60 0x7fe9dc085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e0dbf700 1 -- 172.21.2.117:0/1218166039 >> v2:172.21.2.125:3300/0 conn(0x7fe9dc084c60 msgr2=0x7fe9dc085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e0dbf700 1 --2- 172.21.2.117:0/1218166039 >> v2:172.21.2.125:3300/0 conn(0x7fe9dc084c60 0x7fe9dc085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e0dbf700 1 -- 172.21.2.117:0/1218166039 >> v1:172.21.2.125:6789/0 conn(0x7fe9dc0e4e40 legacy=0x7fe9dc1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e0dbf700 1 -- 172.21.2.117:0/1218166039 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc084840 con 0x7fe9dc0e5240 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e0dbf700 1 --2- 172.21.2.117:0/1218166039 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe9d8006700 tx=0x7fe9d8009b80).ready entity=mon.2 client_cookie=176aa932d96a5b6a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9cffff700 1 -- 172.21.2.117:0/1218166039 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d800c0c0 con 0x7fe9dc0e5240 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9cffff700 1 -- 172.21.2.117:0/1218166039 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d800c340 con 0x7fe9dc0e5240 2021-02-10T11:53:11.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9cffff700 1 -- 172.21.2.117:0/1218166039 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d800c860 con 0x7fe9dc0e5240 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e37a4700 1 -- 172.21.2.117:0/1218166039 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 msgr2=0x7fe9dc086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.758 7fe9e37a4700 1 --2- 172.21.2.117:0/1218166039 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe9d8006700 tx=0x7fe9d8009b80).stop 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 -- 172.21.2.117:0/1218166039 shutdown_connections 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 --2- 172.21.2.117:0/1218166039 >> v2:172.21.2.125:3300/0 conn(0x7fe9dc084c60 0x7fe9dc085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 --2- 172.21.2.117:0/1218166039 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0e5240 0x7fe9dc086ec0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 -- 172.21.2.117:0/1218166039 >> 172.21.2.117:0/1218166039 conn(0x7fe9dc0ce4e0 msgr2=0x7fe9dc0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 -- 172.21.2.117:0/1218166039 shutdown_connections 2021-02-10T11:53:11.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.759 7fe9e37a4700 1 -- 172.21.2.117:0/1218166039 wait complete. 2021-02-10T11:53:11.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 Processor -- start 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 -- start start 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0ad1b0 0x7fe9dc0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9dc0adbd0 0x7fe9dc09b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc18d0c0 con 0x7fe9dc0973b0 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc09ae70 con 0x7fe9dc0adbd0 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e37a4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9dc09ba10 con 0x7fe9dc0ad1b0 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e15c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e15c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56984/0 (socket says 172.21.2.117:56984) 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.774 7fe9e15c0700 1 -- 172.21.2.117:0/1208546305 learned_addr learned my addr 172.21.2.117:0/1208546305 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 -- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0ad1b0 msgr2=0x7fe9dc0ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0ad1b0 0x7fe9dc0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:11.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 -- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9dc0adbd0 msgr2=0x7fe9dc09b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9dc0adbd0 0x7fe9dc09b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 -- 172.21.2.117:0/1208546305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc09bbf0 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e15c0700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7fe9d0006ed0 tx=0x7fe9d000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d0006240 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d00009b0 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc09bbf0 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9d0005730 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.775 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc09c180 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.776 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe9d0006030 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.776 7fe9cdffb700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.776 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe9d0032510 con 0x7fe9dc0973b0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.776 7fe9e0dbf700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:11.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.777 7fe9e0dbf700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:11.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.806 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc085930 con 0x7fe9dc0973b0 2021-02-10T11:53:11.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.809 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe9d0022370 con 0x7fe9dc0973b0 2021-02-10T11:53:11.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:11.944 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe9dc085930 con 0x7fe9dc0973b0 2021-02-10T11:53:12.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.684 7fe9cdffb700 1 -- 172.21.2.117:0/1208546305 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe9d0022370 con 0x7fe9dc0973b0 2021-02-10T11:53:12.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 msgr2=0x7fe9bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:12.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 crc :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:12.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 msgr2=0x7fe9dc097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:12.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 rx=0x7fe9d0006ed0 tx=0x7fe9d000b960).stop 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 shutdown_connections 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9dc0adbd0 0x7fe9dc09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9dc0ad1b0 0x7fe9dc0ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9bc0186c0 0x7fe9bc01abc0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 --2- 172.21.2.117:0/1208546305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9dc0973b0 0x7fe9dc097980 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:12.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 >> 172.21.2.117:0/1208546305 conn(0x7fe9dc18cc70 msgr2=0x7fe9dc093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 shutdown_connections 2021-02-10T11:53:12.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:12.714 7fe9e37a4700 1 -- 172.21.2.117:0/1208546305 wait complete. 2021-02-10T11:53:12.724 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 Processor -- start 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 -- start start 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 0x7f0c180e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0c180e7d70 0x7f0c180e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c180e77f0 con 0x7f0c180e8790 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c180e92f0 con 0x7f0c180e7d70 2021-02-10T11:53:13.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1eebe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c180e5ce0 con 0x7f0c181b07f0 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1d4db700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0c180e8790 0x7f0c180e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57246/0 (socket says 172.21.2.117:57246) 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.134 7f0c1d4db700 1 -- 172.21.2.117:0/1031661578 learned_addr learned my addr 172.21.2.117:0/1031661578 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 --2- 172.21.2.117:0/1031661578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 0x7f0c180e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c17fff700 1 --2- 172.21.2.117:0/1031661578 >> v2:172.21.2.125:3300/0 conn(0x7f0c180e7d70 0x7f0c180e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782990092 0 0) 0x7f0c10001180 con 0x7f0c180e8790 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c04002fe0 con 0x7f0c180e8790 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 621102957 0 0) 0x7f0c10000ef0 con 0x7f0c180e8790 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 -- 172.21.2.117:0/1031661578 >> v2:172.21.2.125:3300/0 conn(0x7f0c180e7d70 msgr2=0x7f0c180e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 --2- 172.21.2.117:0/1031661578 >> v2:172.21.2.125:3300/0 conn(0x7f0c180e7d70 0x7f0c180e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 -- 172.21.2.117:0/1031661578 >> v1:172.21.2.117:6789/0 conn(0x7f0c180e8790 legacy=0x7f0c180e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 -- 172.21.2.117:0/1031661578 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c180e54b0 con 0x7f0c181b07f0 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1ccda700 1 --2- 172.21.2.117:0/1031661578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 0x7f0c180e4e90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0c00006650 tx=0x7f0c00009a80).ready entity=mon.2 client_cookie=b28c9eb891e10f2b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c0000c0c0 con 0x7f0c181b07f0 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0000c340 con 0x7f0c181b07f0 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c177fe700 1 -- 172.21.2.117:0/1031661578 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c000009b0 con 0x7f0c181b07f0 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1eebe700 1 -- 172.21.2.117:0/1031661578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 msgr2=0x7f0c180e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.135 7f0c1eebe700 1 --2- 172.21.2.117:0/1031661578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 0x7f0c180e4e90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0c00006650 tx=0x7f0c00009a80).stop 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 -- 172.21.2.117:0/1031661578 shutdown_connections 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 --2- 172.21.2.117:0/1031661578 >> v2:172.21.2.125:3300/0 conn(0x7f0c180e7d70 0x7f0c180e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 --2- 172.21.2.117:0/1031661578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c181b07f0 0x7f0c180e4e90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 -- 172.21.2.117:0/1031661578 >> 172.21.2.117:0/1031661578 conn(0x7f0c180ce500 msgr2=0x7f0c180e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 -- 172.21.2.117:0/1031661578 shutdown_connections 2021-02-10T11:53:13.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.136 7f0c1eebe700 1 -- 172.21.2.117:0/1031661578 wait complete. 2021-02-10T11:53:13.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 Processor -- start 2021-02-10T11:53:13.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 -- start start 2021-02-10T11:53:13.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c180b15a0 0x7f0c18093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c180b19a0 con 0x7f0c180a3a60 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c180930b0 con 0x7f0c180b15a0 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.151 7f0c1eebe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c181a3450 con 0x7f0c180b0a50 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1ccda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1ccda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:56998/0 (socket says 172.21.2.117:56998) 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54680/0 (socket says 172.21.2.117:54680) 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1ccda700 1 -- 172.21.2.117:0/1041257238 learned_addr learned my addr 172.21.2.117:0/1041257238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1d4db700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c180b15a0 0x7f0c18093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 -- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c180b15a0 msgr2=0x7f0c18093630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c180b15a0 0x7f0c18093630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 -- 172.21.2.117:0/1041257238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 msgr2=0x7f0c181a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 -- 172.21.2.117:0/1041257238 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c181a3630 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1ccda700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c17fff700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f0c080098a0 tx=0x7f0c0800bc20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c08006030 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c08006330 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c181a3630 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.152 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0c08000d30 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.153 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c181a3630 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.153 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0c08006510 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.153 7f0c157fa700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 0x7f0bfc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.153 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0c080327f0 con 0x7f0c180b0a50 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.153 7f0c1ccda700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 0x7f0bfc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.154 7f0c1ccda700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 0x7f0bfc01abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.184 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c180b19a0 con 0x7f0c180b0a50 2021-02-10T11:53:13.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.187 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0c0800bd60 con 0x7f0c180b0a50 2021-02-10T11:53:13.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.320 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f0c180b19a0 con 0x7f0bfc0186c0 2021-02-10T11:53:13.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.322 7f0c157fa700 1 -- 172.21.2.117:0/1041257238 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f0c00009380 con 0x7f0bfc0186c0 2021-02-10T11:53:13.326 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:53:13.326 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:13.326 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:53:13.326 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:13.326 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:53:13.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.324 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 msgr2=0x7f0bfc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.324 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 0x7f0bfc01abc0 crc :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.324 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 msgr2=0x7f0c181a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.324 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f0c080098a0 tx=0x7f0c0800bc20).stop 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 shutdown_connections 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0c180b15a0 0x7f0c18093630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0c180b0a50 0x7f0c181a2e30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0bfc0186c0 0x7f0bfc01abc0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 --2- 172.21.2.117:0/1041257238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0c180a3a60 0x7f0c181a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 >> 172.21.2.117:0/1041257238 conn(0x7f0c180a7260 msgr2=0x7f0c180bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 shutdown_connections 2021-02-10T11:53:13.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.325 7f0c1eebe700 1 -- 172.21.2.117:0/1041257238 wait complete. 2021-02-10T11:53:13.336 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 Processor -- start 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 -- start start 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e901b69a0 0x7f1e901b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e900ede60 con 0x7f1e900ef3f0 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e900eba00 con 0x7f1e900d18f0 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e966b3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e900d1cf0 con 0x7f1e901b69a0 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.750 7f1e94cd0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e901b69a0 0x7f1e901b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e901b69a0 0x7f1e901b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57008/0 (socket says 172.21.2.117:57008) 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 -- 172.21.2.117:0/3279628418 learned_addr learned my addr 172.21.2.117:0/3279628418 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 -- 172.21.2.117:0/3279628418 >> v1:172.21.2.125:6789/0 conn(0x7f1e900d18f0 legacy=0x7f1e900eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 -- 172.21.2.117:0/3279628418 >> v1:172.21.2.117:6789/0 conn(0x7f1e900ef3f0 legacy=0x7f1e900eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 -- 172.21.2.117:0/3279628418 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e901bb3f0 con 0x7f1e901b69a0 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e94cd0700 1 --2- 172.21.2.117:0/3279628418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e901b69a0 0x7f1e901b8db0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f1e88006f00 tx=0x7f1e88009f10).ready entity=mon.0 client_cookie=84ac41ab7f6811b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e8effd700 1 -- 172.21.2.117:0/3279628418 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8800b3b0 con 0x7f1e901b69a0 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e8effd700 1 -- 172.21.2.117:0/3279628418 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8800b730 con 0x7f1e901b69a0 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e8effd700 1 -- 172.21.2.117:0/3279628418 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8800bc10 con 0x7f1e901b69a0 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e966b3700 1 -- 172.21.2.117:0/3279628418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e901b69a0 msgr2=0x7f1e901b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.751 7f1e966b3700 1 --2- 172.21.2.117:0/3279628418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e901b69a0 0x7f1e901b8db0 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 rx=0x7f1e88006f00 tx=0x7f1e88009f10).stop 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.752 7f1e966b3700 1 -- 172.21.2.117:0/3279628418 shutdown_connections 2021-02-10T11:53:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.752 7f1e966b3700 1 --2- 172.21.2.117:0/3279628418 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e901b69a0 0x7f1e901b8db0 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.752 7f1e966b3700 1 -- 172.21.2.117:0/3279628418 >> 172.21.2.117:0/3279628418 conn(0x7f1e900cf060 msgr2=0x7f1e900cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:13.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.752 7f1e966b3700 1 -- 172.21.2.117:0/3279628418 shutdown_connections 2021-02-10T11:53:13.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.752 7f1e966b3700 1 -- 172.21.2.117:0/3279628418 wait complete. 2021-02-10T11:53:13.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 Processor -- start 2021-02-10T11:53:13.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 -- start start 2021-02-10T11:53:13.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e900e9630 0x7f1e901c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e901c9820 0x7f1e9009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9009c050 con 0x7f1e900dedc0 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e900affd0 con 0x7f1e901c9820 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e966b3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e900e9a30 con 0x7f1e900e9630 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e8ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e8ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57012/0 (socket says 172.21.2.117:57012) 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e94cd0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e901c9820 0x7f1e9009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.767 7f1e8ffff700 1 -- 172.21.2.117:0/801423634 learned_addr learned my addr 172.21.2.117:0/801423634 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8f7fe700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e900e9630 0x7f1e901c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 -- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e900e9630 msgr2=0x7f1e901c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e900e9630 0x7f1e901c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 -- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e901c9820 msgr2=0x7f1e9009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e901c9820 0x7f1e9009c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 -- 172.21.2.117:0/801423634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e901c9c20 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8ffff700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 rx=0x7f1e8000b5b0 tx=0x7f1e8000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e8000bd40 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80006020 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e966b3700 1 -- 172.21.2.117:0/801423634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e901c9c20 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e800054a0 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.768 7f1e966b3700 1 -- 172.21.2.117:0/801423634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e901c9c20 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.769 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e80006200 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.769 7f1e8cff9700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 0x7f1e7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.769 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1e80032790 con 0x7f1e900dedc0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.770 7f1e8f7fe700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 0x7f1e7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:13.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.770 7f1e8f7fe700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 0x7f1e7001abc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:13.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.800 7f1e966b3700 1 -- 172.21.2.117:0/801423634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e90087390 con 0x7f1e900dedc0 2021-02-10T11:53:13.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.803 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e80000c60 con 0x7f1e900dedc0 2021-02-10T11:53:13.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.956 7f1e966b3700 1 -- 172.21.2.117:0/801423634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1e90087390 con 0x7f1e900dedc0 2021-02-10T11:53:13.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.957 7f1e8cff9700 1 -- 172.21.2.117:0/801423634 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1e80006020 con 0x7f1e900dedc0 2021-02-10T11:53:13.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.958 7f1e966b3700 1 -- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 msgr2=0x7f1e7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.958 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 0x7f1e7001abc0 crc :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.958 7f1e966b3700 1 -- 172.21.2.117:0/801423634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 msgr2=0x7f1e900af920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.958 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 rx=0x7f1e8000b5b0 tx=0x7f1e8000b950).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 -- 172.21.2.117:0/801423634 shutdown_connections 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e901c9820 0x7f1e9009c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e900e9630 0x7f1e901c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e70018700 0x7f1e7001abc0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 --2- 172.21.2.117:0/801423634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e900dedc0 0x7f1e900af920 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 -- 172.21.2.117:0/801423634 >> 172.21.2.117:0/801423634 conn(0x7f1e900b74a0 msgr2=0x7f1e900b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 -- 172.21.2.117:0/801423634 shutdown_connections 2021-02-10T11:53:13.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:13.959 7f1e966b3700 1 -- 172.21.2.117:0/801423634 wait complete. 2021-02-10T11:53:13.970 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs non_existent_group 2021-02-10T11:53:14.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.383 7fec9bbe4700 1 Processor -- start 2021-02-10T11:53:14.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.383 7fec9bbe4700 1 -- start start 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9bbe4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fec940d18f0 0x7fec940eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9bbe4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec940eb040 0x7fec940eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9bbe4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940ede60 con 0x7fec940ee3e0 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9bbe4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940ebad0 con 0x7fec940eb040 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9bbe4700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940d1cf0 con 0x7fec940d18f0 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec940eb040 0x7fec940eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9a201700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fec940ee3e0 0x7fec940ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57274/0 (socket says 172.21.2.117:57274) 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fec940eb040 0x7fec940eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57020/0 (socket says 172.21.2.117:57020) 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec9a201700 1 -- 172.21.2.117:0/2781514167 learned_addr learned my addr 172.21.2.117:0/2781514167 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec989fe700 1 -- 172.21.2.117:0/2781514167 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572203578 0 0) 0x7fec8c001100 con 0x7fec940ee3e0 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 -- 172.21.2.117:0/2781514167 >> v2:172.21.2.125:3300/0 conn(0x7fec940d18f0 msgr2=0x7fec940eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:14.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 --2- 172.21.2.117:0/2781514167 >> v2:172.21.2.125:3300/0 conn(0x7fec940d18f0 0x7fec940eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 -- 172.21.2.117:0/2781514167 >> v1:172.21.2.117:6789/0 conn(0x7fec940ee3e0 legacy=0x7fec940ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.384 7fec991ff700 1 -- 172.21.2.117:0/2781514167 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940eefd0 con 0x7fec940eb040 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec991ff700 1 --2- 172.21.2.117:0/2781514167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940eb040 0x7fec940eb440 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7fec900069d0 tx=0x7fec90009e40).ready entity=mon.0 client_cookie=bb5bad9d125a53c1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec989fe700 1 -- 172.21.2.117:0/2781514167 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9000b510 con 0x7fec940eb040 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec989fe700 1 -- 172.21.2.117:0/2781514167 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9000b890 con 0x7fec940eb040 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec989fe700 1 -- 172.21.2.117:0/2781514167 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9000bd70 con 0x7fec940eb040 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 -- 172.21.2.117:0/2781514167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940eb040 msgr2=0x7fec940eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 --2- 172.21.2.117:0/2781514167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940eb040 0x7fec940eb440 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 rx=0x7fec900069d0 tx=0x7fec90009e40).stop 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 -- 172.21.2.117:0/2781514167 shutdown_connections 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 --2- 172.21.2.117:0/2781514167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940eb040 0x7fec940eb440 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 --2- 172.21.2.117:0/2781514167 >> v2:172.21.2.125:3300/0 conn(0x7fec940d18f0 0x7fec940eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:14.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.385 7fec9bbe4700 1 -- 172.21.2.117:0/2781514167 >> 172.21.2.117:0/2781514167 conn(0x7fec940cf060 msgr2=0x7fec940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:14.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.386 7fec9bbe4700 1 -- 172.21.2.117:0/2781514167 shutdown_connections 2021-02-10T11:53:14.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.386 7fec9bbe4700 1 -- 172.21.2.117:0/2781514167 wait complete. 2021-02-10T11:53:14.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 Processor -- start 2021-02-10T11:53:14.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 -- start start 2021-02-10T11:53:14.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 0x7fec940b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec940c0a20 0x7fec940c1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c1ab0 con 0x7fec940c0000 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c1ca0 con 0x7fec940b09c0 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.400 7fec9bbe4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c1ec0 con 0x7fec940c0a20 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57028/0 (socket says 172.21.2.117:57028) 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 -- 172.21.2.117:0/1696579175 learned_addr learned my addr 172.21.2.117:0/1696579175 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec9a201700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec940c0a20 0x7fec940c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec99a00700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 0x7fec940b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 -- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec940c0a20 msgr2=0x7fec940c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec940c0a20 0x7fec940c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 -- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 msgr2=0x7fec940b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 0x7fec940b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 -- 172.21.2.117:0/1696579175 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940c20e0 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec99a00700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 0x7fec940b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec991ff700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7fec90009f40 tx=0x7fec9000bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec90006240 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec900065c0 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec940c20e0 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.401 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec900058e0 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.402 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec940c20e0 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.402 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec90005660 con 0x7fec940c0000 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.402 7fec867fc700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 0x7fec7c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:14.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.402 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fec90032b00 con 0x7fec940c0000 2021-02-10T11:53:14.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.402 7fec99a00700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 0x7fec7c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:14.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.403 7fec99a00700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 0x7fec7c01ac10 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:14.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.433 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec940f67e0 con 0x7fec940c0000 2021-02-10T11:53:14.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.436 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec900065c0 con 0x7fec940c0000 2021-02-10T11:53:14.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.562 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "non_existent_group"}) v1 -- 0x7fec940f67e0 con 0x7fec7c018730 2021-02-10T11:53:14.569 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:53:14.569 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:53:14.569 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:53:14.569 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:53:14.569 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:53:14.573 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:53:14.573 7f59c98e1700 -1 mgr.server reply reply (2) No such file or directory subvolume group 'non_existent_group' does not exist 2021-02-10T11:53:14.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.571 7fec867fc700 1 -- 172.21.2.117:0/1696579175 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -2 subvolume group 'non_existent_group' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7fec880009b0 con 0x7fec7c018730 2021-02-10T11:53:14.580 INFO:teuthology.orchestra.run.gibba017.stderr:Error ENOENT: subvolume group 'non_existent_group' does not exist 2021-02-10T11:53:14.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 msgr2=0x7fec7c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 0x7fec7c01ac10 crc :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 msgr2=0x7fec940c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:14.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 rx=0x7fec90009f40 tx=0x7fec9000bd50).stop 2021-02-10T11:53:14.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 shutdown_connections 2021-02-10T11:53:14.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec7c018730 0x7fec7c01ac10 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec940c0000 0x7fec940c0400 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec940c0a20 0x7fec940c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 --2- 172.21.2.117:0/1696579175 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec940b09c0 0x7fec940b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:14.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 >> 172.21.2.117:0/1696579175 conn(0x7fec9408e930 msgr2=0x7fec940bd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:14.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 shutdown_connections 2021-02-10T11:53:14.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:14.579 7fec9bbe4700 1 -- 172.21.2.117:0/1696579175 wait complete. 2021-02-10T11:53:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:53:14.590 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:53:14.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:53:14.936 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:53:15.279 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:53:15.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:15.279 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:15.279 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:15.615 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:15.615 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:53:15.615 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:53:15.616 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:53:15.616 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:15.780 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:53:15.784 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:53:15.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[173305]: fuse finished with error 0 and tester_r 0 2021-02-10T11:53:15.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:15.967 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:15.967 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:16.302 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:53:16.302 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:53:16.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:53:16.302 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:16.639 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:16.639 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:16.639 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:16.975 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:16.975 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:53:16.976 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:53:16.976 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:53:16.976 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:17.328 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[173946]: fuse finished with error 0 and tester_r 0 2021-02-10T11:53:17.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:17.330 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:17.330 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:17.666 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:53:17.666 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:53:17.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:53:17.666 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:18.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:18.002 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:18.002 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:53:18.337 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:53:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:18.338 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:53:18.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:53:18.338 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:53:18.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:18.673 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:18.673 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:53:19.007 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:53:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:19.007 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:53:19.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:53:19.008 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:53:19.341 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:53:19.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 Processor -- start 2021-02-10T11:53:19.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 -- start start 2021-02-10T11:53:19.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441b07f0 0x7f20440e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20440e7d70 0x7f20440e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20440e8790 0x7f20440e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20440e92f0 con 0x7f20440e7d70 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20440e5ce0 con 0x7f20440e8790 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.757 7f2048ff1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20440e77f0 con 0x7f20441b07f0 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20440e7d70 0x7f20440e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20440e7d70 0x7f20440e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57036/0 (socket says 172.21.2.117:57036) 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 -- 172.21.2.117:0/3886037642 learned_addr learned my addr 172.21.2.117:0/3886037642 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 -- 172.21.2.117:0/3886037642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441b07f0 msgr2=0x7f20440e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 --2- 172.21.2.117:0/3886037642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441b07f0 0x7f20440e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 -- 172.21.2.117:0/3886037642 >> v2:172.21.2.125:3300/0 conn(0x7f20440e8790 msgr2=0x7f20440e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 --2- 172.21.2.117:0/3886037642 >> v2:172.21.2.125:3300/0 conn(0x7f20440e8790 0x7f20440e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 -- 172.21.2.117:0/3886037642 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440e54b0 con 0x7f20440e7d70 2021-02-10T11:53:19.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.758 7f204259c700 1 --2- 172.21.2.117:0/3886037642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20440e7d70 0x7f20440e8170 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f202c006700 tx=0x7f202c009b80).ready entity=mon.0 client_cookie=f14eca58925590bd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2041d9b700 1 -- 172.21.2.117:0/3886037642 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f202c00c0c0 con 0x7f20440e7d70 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2041d9b700 1 -- 172.21.2.117:0/3886037642 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c00c340 con 0x7f20440e7d70 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2041d9b700 1 -- 172.21.2.117:0/3886037642 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f202c00c860 con 0x7f20440e7d70 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 -- 172.21.2.117:0/3886037642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20440e7d70 msgr2=0x7f20440e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 --2- 172.21.2.117:0/3886037642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20440e7d70 0x7f20440e8170 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 rx=0x7f202c006700 tx=0x7f202c009b80).stop 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 -- 172.21.2.117:0/3886037642 shutdown_connections 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 --2- 172.21.2.117:0/3886037642 >> v2:172.21.2.125:3300/0 conn(0x7f20440e8790 0x7f20440e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 --2- 172.21.2.117:0/3886037642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20440e7d70 0x7f20440e8170 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 --2- 172.21.2.117:0/3886037642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441b07f0 0x7f20440e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.759 7f2048ff1700 1 -- 172.21.2.117:0/3886037642 >> 172.21.2.117:0/3886037642 conn(0x7f20440ce500 msgr2=0x7f20440e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.760 7f2048ff1700 1 -- 172.21.2.117:0/3886037642 shutdown_connections 2021-02-10T11:53:19.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.760 7f2048ff1700 1 -- 172.21.2.117:0/3886037642 wait complete. 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 Processor -- start 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 -- start start 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2044195540 0x7f20441af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441afc90 0x7f20441108d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20441103d0 con 0x7f2044110ef0 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204419ee20 con 0x7f2044195540 2021-02-10T11:53:19.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.774 7f2048ff1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204419f260 con 0x7f20441afc90 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57042/0 (socket says 172.21.2.117:57042) 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 -- 172.21.2.117:0/4030765201 learned_addr learned my addr 172.21.2.117:0/4030765201 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 -- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441afc90 msgr2=0x7f20441108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441afc90 0x7f20441108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 -- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2044195540 msgr2=0x7f20441af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2044195540 0x7f20441af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 -- 172.21.2.117:0/4030765201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440aeb10 con 0x7f2044110ef0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f204359e700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f203c009980 tx=0x7f203c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f203c005af0 con 0x7f2044110ef0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c005d70 con 0x7f2044110ef0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f203c0062b0 con 0x7f2044110ef0 2021-02-10T11:53:19.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20440aeb10 con 0x7f2044110ef0 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.775 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2044195c30 con 0x7f2044110ef0 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.776 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f203c006030 con 0x7f2044110ef0 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.776 7f203b7fe700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 0x7f202801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.776 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f203c032a00 con 0x7f2044110ef0 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.777 7f2042d9d700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 0x7f202801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:19.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.777 7f2042d9d700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 0x7f202801abc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:19.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.807 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20440e77f0 con 0x7f2044110ef0 2021-02-10T11:53:19.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.810 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f203c006570 con 0x7f2044110ef0 2021-02-10T11:53:19.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.959 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f20440e77f0 con 0x7f2044110ef0 2021-02-10T11:53:19.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.963 7f203b7fe700 1 -- 172.21.2.117:0/4030765201 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f203c006030 con 0x7f2044110ef0 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 msgr2=0x7f202801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 0x7f202801abc0 crc :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 msgr2=0x7f204410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 rx=0x7f203c009980 tx=0x7f203c00bcd0).stop 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 shutdown_connections 2021-02-10T11:53:19.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f20280186c0 0x7f202801abc0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2044110ef0 0x7f204410f740 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20441afc90 0x7f20441108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 --2- 172.21.2.117:0/4030765201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2044195540 0x7f20441af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 >> 172.21.2.117:0/4030765201 conn(0x7f20441998b0 msgr2=0x7f204410cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.968 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 shutdown_connections 2021-02-10T11:53:19.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:19.969 7f2048ff1700 1 -- 172.21.2.117:0/4030765201 wait complete. 2021-02-10T11:53:19.978 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm' 2021-02-10T11:53:20.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.389 7f2bbd033700 1 Processor -- start 2021-02-10T11:53:20.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.389 7f2bbd033700 1 -- start start 2021-02-10T11:53:20.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bbd033700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bbd033700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2bb80ef3f0 0x7f2bb81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bbd033700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80ede60 con 0x7f2bb80eb000 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bbd033700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80ef7f0 con 0x7f2bb80d18e0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bbd033700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80d1ce0 con 0x7f2bb80ef3f0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57048/0 (socket says 172.21.2.117:57048) 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 -- 172.21.2.117:0/1521279651 learned_addr learned my addr 172.21.2.117:0/1521279651 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 -- 172.21.2.117:0/1521279651 >> v2:172.21.2.125:3300/0 conn(0x7f2bb80ef3f0 msgr2=0x7f2bb81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 --2- 172.21.2.117:0/1521279651 >> v2:172.21.2.125:3300/0 conn(0x7f2bb80ef3f0 0x7f2bb81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 -- 172.21.2.117:0/1521279651 >> v1:172.21.2.125:6789/0 conn(0x7f2bb80eb000 legacy=0x7f2bb80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 -- 172.21.2.117:0/1521279651 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb81b8f90 con 0x7f2bb80d18e0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.390 7f2bb6d9d700 1 --2- 172.21.2.117:0/1521279651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f2ba8006700 tx=0x7f2ba8009b80).ready entity=mon.0 client_cookie=aa532525c80da93d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bb5d9b700 1 -- 172.21.2.117:0/1521279651 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ba800c0c0 con 0x7f2bb80d18e0 2021-02-10T11:53:20.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bb5d9b700 1 -- 172.21.2.117:0/1521279651 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba800c340 con 0x7f2bb80d18e0 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bb5d9b700 1 -- 172.21.2.117:0/1521279651 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ba800c860 con 0x7f2bb80d18e0 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 -- 172.21.2.117:0/1521279651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80d18e0 msgr2=0x7f2bb80ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 --2- 172.21.2.117:0/1521279651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 rx=0x7f2ba8006700 tx=0x7f2ba8009b80).stop 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 -- 172.21.2.117:0/1521279651 shutdown_connections 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 --2- 172.21.2.117:0/1521279651 >> v2:172.21.2.125:3300/0 conn(0x7f2bb80ef3f0 0x7f2bb81b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 --2- 172.21.2.117:0/1521279651 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80d18e0 0x7f2bb80ea9e0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 -- 172.21.2.117:0/1521279651 >> 172.21.2.117:0/1521279651 conn(0x7f2bb80ced60 msgr2=0x7f2bb80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 -- 172.21.2.117:0/1521279651 shutdown_connections 2021-02-10T11:53:20.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.391 7f2bbd033700 1 -- 172.21.2.117:0/1521279651 wait complete. 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 Processor -- start 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 -- start start 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bb8196390 0x7f2bb80e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bb80e9560 0x7f2bb80e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8196810 con 0x7f2bb80e9960 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb8098ad0 con 0x7f2bb80e9560 2021-02-10T11:53:20.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.406 7f2bbd033700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80e8a40 con 0x7f2bb8196390 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57058/0 (socket says 172.21.2.117:57058) 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 -- 172.21.2.117:0/2189861058 learned_addr learned my addr 172.21.2.117:0/2189861058 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb659c700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bb80e9560 0x7f2bb80e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb6d9d700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bb8196390 0x7f2bb80e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 -- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bb8196390 msgr2=0x7f2bb80e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bb8196390 0x7f2bb80e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 -- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bb80e9560 msgr2=0x7f2bb80e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bb80e9560 0x7f2bb80e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 -- 172.21.2.117:0/2189861058 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80e8c60 con 0x7f2bb80e9960 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.407 7f2bb759e700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 rx=0x7f2bb0009980 tx=0x7f2bb000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.408 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb0005870 con 0x7f2bb80e9960 2021-02-10T11:53:20.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.408 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb80e8c60 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.408 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb000be10 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.408 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bb0005b30 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.408 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb80e8c60 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.409 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bb0005d00 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.409 7f2baf7fe700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 0x7f2b9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.409 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2bb0032b40 con 0x7f2bb80e9960 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.409 7f2bb6d9d700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 0x7f2b9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:20.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.409 7f2bb6d9d700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 0x7f2b9c01abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.439 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb8087210 con 0x7f2bb80e9960 2021-02-10T11:53:20.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.442 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bb0005ad0 con 0x7f2bb80e9960 2021-02-10T11:53:20.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:20.574 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]} v 0) v1 -- 0x7f2bb80ede60 con 0x7f2bb80e9960 2021-02-10T11:53:21.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.312 7f2baf7fe700 1 -- 172.21.2.117:0/2189861058 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_group_rm"]}]=0 v730) v1 ==== 152+0+0 (secure 0 0 0) 0x7f2bb0005d00 con 0x7f2bb80e9960 2021-02-10T11:53:21.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 msgr2=0x7f2b9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 0x7f2b9c01abc0 crc :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 msgr2=0x7f2bb80e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 rx=0x7f2bb0009980 tx=0x7f2bb000bcd0).stop 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 shutdown_connections 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2b9c018730 0x7f2b9c01abc0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bb80e9960 0x7f2bb80e8420 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bb80e9560 0x7f2bb80e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 --2- 172.21.2.117:0/2189861058 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bb8196390 0x7f2bb80e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.345 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 >> 172.21.2.117:0/2189861058 conn(0x7f2bb819f610 msgr2=0x7f2bb81a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.346 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 shutdown_connections 2021-02-10T11:53:21.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.346 7f2bbd033700 1 -- 172.21.2.117:0/2189861058 wait complete. 2021-02-10T11:53:21.356 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:53:21.356 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:53:21.356 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_group_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... skipped 'skipping subvolumegroup snapshot tests' 2021-02-10T11:53:21.356 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:53:21.356 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:53:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.765 7f92e1977700 1 Processor -- start 2021-02-10T11:53:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.765 7f92e1977700 1 -- start start 2021-02-10T11:53:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92e1977700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0ef3f0 0x7f92dc0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92e1977700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc0ede60 con 0x7f92dc0d18e0 2021-02-10T11:53:21.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92e1977700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc0d1ce0 con 0x7f92dc0eb7f0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92e1977700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0ef7f0 con 0x7f92dc0ef3f0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92db574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f92dc0d18e0 0x7f92dc0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57312/0 (socket says 172.21.2.117:57312) 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92db574700 1 -- 172.21.2.117:0/3224661503 learned_addr learned my addr 172.21.2.117:0/3224661503 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825171302 0 0) 0x7f92c4001180 con 0x7f92dc0d18e0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92da572700 1 -- 172.21.2.117:0/3224661503 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c8002fe0 con 0x7f92dc0d18e0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.766 7f92dad73700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0ef3f0 0x7f92dc0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 99585444 0 0) 0x7f92c4001180 con 0x7f92dc0d18e0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0ef3f0 msgr2=0x7f92dc0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0ef3f0 0x7f92dc0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 >> v1:172.21.2.125:6789/0 conn(0x7f92dc0eb7f0 legacy=0x7f92dc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc1bb180 con 0x7f92dc0d18e0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f92c4001680 con 0x7f92dc0d18e0 2021-02-10T11:53:21.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 >> v1:172.21.2.117:6789/0 conn(0x7f92dc0d18e0 legacy=0x7f92dc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c8003a60 0x7f92c8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8006780 0x7f92c8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c80094f0 0x7f92c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c800c2c0 con 0x7f92c8003a60 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c800c4e0 con 0x7f92c80094f0 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92da572700 1 -- 172.21.2.117:0/3224661503 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8050710 con 0x7f92c8006780 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c8003a60 0x7f92c8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92dad73700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8006780 0x7f92c8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92dbd75700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c80094f0 0x7f92c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 -- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8006780 msgr2=0x7f92c8008c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92c8006780 0x7f92c8008c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 -- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c80094f0 msgr2=0x7f92c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92c80094f0 0x7f92c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 -- 172.21.2.117:0/3224661503 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c8050970 con 0x7f92c8003a60 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.767 7f92db574700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c8003a60 0x7f92c8005f60 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f92c4007970 tx=0x7f92c4007d10).ready entity=mon.0 client_cookie=d2d02e6877f04c70 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c40070c0 con 0x7f92c8003a60 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c40043a0 con 0x7f92c8003a60 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92da572700 1 -- 172.21.2.117:0/3224661503 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c40048c0 con 0x7f92c8003a60 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 -- 172.21.2.117:0/3224661503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c8003a60 msgr2=0x7f92c8005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 --2- 172.21.2.117:0/3224661503 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92c8003a60 0x7f92c8005f60 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 rx=0x7f92c4007970 tx=0x7f92c4007d10).stop 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92dbd75700 1 -- 172.21.2.117:0/3224661503 reap_dead start 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 -- 172.21.2.117:0/3224661503 shutdown_connections 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 -- 172.21.2.117:0/3224661503 >> 172.21.2.117:0/3224661503 conn(0x7f92dc0ced60 msgr2=0x7f92dc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:21.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 -- 172.21.2.117:0/3224661503 shutdown_connections 2021-02-10T11:53:21.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.768 7f92e1977700 1 -- 172.21.2.117:0/3224661503 wait complete. 2021-02-10T11:53:21.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 Processor -- start 2021-02-10T11:53:21.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 -- start start 2021-02-10T11:53:21.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 0x7f92dc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 0x7f92dc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0b61a0 con 0x7f92dc0b59a0 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc08c8b0 con 0x7f92dc1a3480 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92e1977700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0ba1c0 con 0x7f92dc0b5da0 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92db574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92db574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57074/0 (socket says 172.21.2.117:57074) 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.783 7f92db574700 1 -- 172.21.2.117:0/3612591096 learned_addr learned my addr 172.21.2.117:0/3612591096 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 -- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 msgr2=0x7f92dc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92dad73700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 0x7f92dc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92dbd75700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 0x7f92dc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 0x7f92dc1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 -- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 msgr2=0x7f92dc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 0x7f92dc0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 -- 172.21.2.117:0/3612591096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc1a3880 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92dad73700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 0x7f92dc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92dbd75700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 0x7f92dc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92db574700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f92c400a580 tx=0x7f92c400a920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c4001130 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c40043a0 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92e1977700 1 -- 172.21.2.117:0/3612591096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc1a3880 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92c4004880 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.784 7f92e1977700 1 -- 172.21.2.117:0/3612591096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc0b2bb0 con 0x7f92dc0b59a0 2021-02-10T11:53:21.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.785 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92c4004580 con 0x7f92dc0b59a0 2021-02-10T11:53:21.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.785 7f92d3fff700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 0x7f92c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:21.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.785 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f92c4032520 con 0x7f92dc0b59a0 2021-02-10T11:53:21.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.786 7f92dad73700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 0x7f92c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:21.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.787 7f92dad73700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 0x7f92c001abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:21.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.815 7f92e1977700 1 -- 172.21.2.117:0/3612591096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc0ede60 con 0x7f92dc0b59a0 2021-02-10T11:53:21.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.818 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92c4030d00 con 0x7f92dc0b59a0 2021-02-10T11:53:21.951 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:21.949 7f92e1977700 1 -- 172.21.2.117:0/3612591096 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7f92dc0ede60 con 0x7f92dc0b59a0 2021-02-10T11:53:22.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.315 7f92d3fff700 1 -- 172.21.2.117:0/3612591096 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v731) v1 ==== 149+0+0 (secure 0 0 0) 0x7f92c4004580 con 0x7f92dc0b59a0 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 -- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 msgr2=0x7f92c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 0x7f92c001abc0 crc :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 -- 172.21.2.117:0/3612591096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 msgr2=0x7f92dc08c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 rx=0x7f92c400a580 tx=0x7f92c400a920).stop 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 -- 172.21.2.117:0/3612591096 shutdown_connections 2021-02-10T11:53:22.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92dc1a3480 0x7f92dc0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92dc0b5da0 0x7f92dc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92c00186c0 0x7f92c001abc0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 --2- 172.21.2.117:0/3612591096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92dc0b59a0 0x7f92dc08c290 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.317 7f92e1977700 1 -- 172.21.2.117:0/3612591096 >> 172.21.2.117:0/3612591096 conn(0x7f92dc0bb5e0 msgr2=0x7f92dc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.318 7f92e1977700 1 -- 172.21.2.117:0/3612591096 shutdown_connections 2021-02-10T11:53:22.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.318 7f92e1977700 1 -- 172.21.2.117:0/3612591096 wait complete. 2021-02-10T11:53:22.328 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:53:22.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 Processor -- start 2021-02-10T11:53:22.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 -- start start 2021-02-10T11:53:22.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 0x7f15a40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f15a40e7d70 0x7f15a40e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40e77f0 con 0x7f15a40e8790 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40e92f0 con 0x7f15a40e7d70 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.742 7f15aafab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40e5ce0 con 0x7f15a41b07f0 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.743 7f15a95c8700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f15a40e8790 0x7f15a40e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50262/0 (socket says 172.21.2.117:50262) 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.743 7f15a95c8700 1 -- 172.21.2.117:0/3732295463 learned_addr learned my addr 172.21.2.117:0/3732295463 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:22.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.743 7f15a3fff700 1 --2- 172.21.2.117:0/3732295463 >> v2:172.21.2.125:3300/0 conn(0x7f15a40e7d70 0x7f15a40e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.743 7f15a8dc7700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 0x7f15a40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a37fe700 1 -- 172.21.2.117:0/3732295463 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420107601 0 0) 0x7f159c001510 con 0x7f15a40e8790 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a37fe700 1 -- 172.21.2.117:0/3732295463 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1590002fe0 con 0x7f15a40e8790 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a3fff700 1 -- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 msgr2=0x7f15a40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a3fff700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 0x7f15a40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a3fff700 1 -- 172.21.2.117:0/3732295463 >> v1:172.21.2.125:6789/0 conn(0x7f15a40e8790 legacy=0x7f15a40e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a3fff700 1 -- 172.21.2.117:0/3732295463 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40e54b0 con 0x7f15a40e7d70 2021-02-10T11:53:22.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a8dc7700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 0x7f15a40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a3fff700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a40e7d70 0x7f15a40e8170 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f1594006650 tx=0x7f1594009a80).ready entity=mon.1 client_cookie=ac470ec4a91b9034 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a37fe700 1 -- 172.21.2.117:0/3732295463 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f159400c0c0 con 0x7f15a40e7d70 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a37fe700 1 -- 172.21.2.117:0/3732295463 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400c340 con 0x7f15a40e7d70 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15a37fe700 1 -- 172.21.2.117:0/3732295463 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f159400c860 con 0x7f15a40e7d70 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15aafab700 1 -- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a40e7d70 msgr2=0x7f15a40e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.744 7f15aafab700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a40e7d70 0x7f15a40e8170 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f1594006650 tx=0x7f1594009a80).stop 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 -- 172.21.2.117:0/3732295463 shutdown_connections 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a40e7d70 0x7f15a40e8170 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 --2- 172.21.2.117:0/3732295463 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a41b07f0 0x7f15a40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 -- 172.21.2.117:0/3732295463 >> 172.21.2.117:0/3732295463 conn(0x7f15a40ce500 msgr2=0x7f15a40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 -- 172.21.2.117:0/3732295463 shutdown_connections 2021-02-10T11:53:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.745 7f15aafab700 1 -- 172.21.2.117:0/3732295463 wait complete. 2021-02-10T11:53:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.759 7f15aafab700 1 Processor -- start 2021-02-10T11:53:22.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.759 7f15aafab700 1 -- start start 2021-02-10T11:53:22.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a408f410 0x7f15a41a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a41a31d0 0x7f15a40bb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4095b70 con 0x7f15a41a2dd0 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a408f810 con 0x7f15a41a31d0 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15aafab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40bb980 con 0x7f15a408f410 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57090/0 (socket says 172.21.2.117:57090) 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 -- 172.21.2.117:0/1626698601 learned_addr learned my addr 172.21.2.117:0/1626698601 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:22.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 -- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a408f410 msgr2=0x7f15a41a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a408f410 0x7f15a41a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 -- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a41a31d0 msgr2=0x7f15a40bb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a41a31d0 0x7f15a40bb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.760 7f15a3fff700 1 -- 172.21.2.117:0/1626698601 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40bbba0 con 0x7f15a41a2dd0 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15a3fff700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f1594009630 tx=0x7f159400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15940093a0 con 0x7f15a41a2dd0 2021-02-10T11:53:22.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159400bd70 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1594005560 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15aafab700 1 -- 172.21.2.117:0/1626698601 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a40bbba0 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.761 7f15aafab700 1 -- 172.21.2.117:0/1626698601 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40bbba0 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.762 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1594008240 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.762 7f15a17fa700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 0x7f158801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.762 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f15940326e0 con 0x7f15a41a2dd0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.762 7f15a8dc7700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 0x7f158801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:22.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.763 7f15a8dc7700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 0x7f158801abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:22.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.792 7f15aafab700 1 -- 172.21.2.117:0/1626698601 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a4095b70 con 0x7f15a41a2dd0 2021-02-10T11:53:22.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.794 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1594008030 con 0x7f15a41a2dd0 2021-02-10T11:53:22.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.943 7f15aafab700 1 -- 172.21.2.117:0/1626698601 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f15a4095b70 con 0x7f15a41a2dd0 2021-02-10T11:53:22.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.947 7f15a17fa700 1 -- 172.21.2.117:0/1626698601 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1594008240 con 0x7f15a41a2dd0 2021-02-10T11:53:22.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 msgr2=0x7f158801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 0x7f158801abc0 crc :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 msgr2=0x7f15a40bad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:22.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 rx=0x7f1594009630 tx=0x7f159400b980).stop 2021-02-10T11:53:22.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 shutdown_connections 2021-02-10T11:53:22.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15880186c0 0x7f158801abc0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15a41a2dd0 0x7f15a40bad40 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15a41a31d0 0x7f15a40bb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 --2- 172.21.2.117:0/1626698601 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15a408f410 0x7f15a41a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 >> 172.21.2.117:0/1626698601 conn(0x7f15a408fb00 msgr2=0x7f15a408fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 shutdown_connections 2021-02-10T11:53:22.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:22.952 7f15aafab700 1 -- 172.21.2.117:0/1626698601 wait complete. 2021-02-10T11:53:22.962 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:53:22.962 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:53:23.309 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:53:23.653 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:53:23.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:23.654 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:23.654 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:23.990 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:53:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:23.990 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:53:23.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:23.991 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:23.991 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:24.326 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:53:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:24.326 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:53:24.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:24.327 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:24.327 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:53:24.665 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:53:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:24.665 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:53:24.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:24.666 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:24.666 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:53:25.001 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:53:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:53:25.001 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:53:25.002 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 Processor -- start 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 -- start start 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0bc00d18e0 0x7f0bc00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0bc00eb000 0x7f0bc00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ee3e0 0x7f0bc00ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00edee0 con 0x7f0bc00eb000 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00eba20 con 0x7f0bc00d18e0 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bc7086700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00d1ce0 con 0x7f0bc00ee3e0 2021-02-10T11:53:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bbffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0bc00eb000 0x7f0bc00eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bbffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0bc00eb000 0x7f0bc00eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57096/0 (socket says 172.21.2.117:57096) 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.413 7f0bbffff700 1 -- 172.21.2.117:0/4110559984 learned_addr learned my addr 172.21.2.117:0/4110559984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bc56a3700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ee3e0 0x7f0bc00ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bc4ea2700 1 --2- 172.21.2.117:0/4110559984 >> v2:172.21.2.125:3300/0 conn(0x7f0bc00d18e0 0x7f0bc00ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 -- 172.21.2.117:0/4110559984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ee3e0 msgr2=0x7f0bc00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ee3e0 0x7f0bc00ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 -- 172.21.2.117:0/4110559984 >> v2:172.21.2.125:3300/0 conn(0x7f0bc00d18e0 msgr2=0x7f0bc00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 --2- 172.21.2.117:0/4110559984 >> v2:172.21.2.125:3300/0 conn(0x7f0bc00d18e0 0x7f0bc00ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 -- 172.21.2.117:0/4110559984 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00eee00 con 0x7f0bc00eb000 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbffff700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc00eb000 0x7f0bc00eb400 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7f0bac006700 tx=0x7f0bac009b80).ready entity=mon.0 client_cookie=7857435f70b51bca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbf7fe700 1 -- 172.21.2.117:0/4110559984 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bac00c0c0 con 0x7f0bc00eb000 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbf7fe700 1 -- 172.21.2.117:0/4110559984 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00c400 con 0x7f0bc00eb000 2021-02-10T11:53:25.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bbf7fe700 1 -- 172.21.2.117:0/4110559984 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bac00c860 con 0x7f0bc00eb000 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bc7086700 1 -- 172.21.2.117:0/4110559984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc00eb000 msgr2=0x7f0bc00eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.414 7f0bc7086700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc00eb000 0x7f0bc00eb400 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 rx=0x7f0bac006700 tx=0x7f0bac009b80).stop 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 -- 172.21.2.117:0/4110559984 shutdown_connections 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ee3e0 0x7f0bc00ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 --2- 172.21.2.117:0/4110559984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc00eb000 0x7f0bc00eb400 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 --2- 172.21.2.117:0/4110559984 >> v2:172.21.2.125:3300/0 conn(0x7f0bc00d18e0 0x7f0bc00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 -- 172.21.2.117:0/4110559984 >> 172.21.2.117:0/4110559984 conn(0x7f0bc00ced60 msgr2=0x7f0bc00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 -- 172.21.2.117:0/4110559984 shutdown_connections 2021-02-10T11:53:25.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.415 7f0bc7086700 1 -- 172.21.2.117:0/4110559984 wait complete. 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 Processor -- start 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 -- start start 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ab280 0x7f0bc00ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00a6fe0 0x7f0bc00a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00a7d10 con 0x7f0bc01a4290 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00a7a00 con 0x7f0bc00a6fe0 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc7086700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0096370 con 0x7f0bc00ab280 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc4ea2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc4ea2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57100/0 (socket says 172.21.2.117:57100) 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc4ea2700 1 -- 172.21.2.117:0/48630790 learned_addr learned my addr 172.21.2.117:0/48630790 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:25.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bbffff700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ab280 0x7f0bc00ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.430 7f0bc56a3700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00a6fe0 0x7f0bc00a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 -- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ab280 msgr2=0x7f0bc00ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ab280 0x7f0bc00ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 -- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00a6fe0 msgr2=0x7f0bc00a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00a6fe0 0x7f0bc00a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 -- 172.21.2.117:0/48630790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00a8290 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc4ea2700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f0bb4006f60 tx=0x7f0bb400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb40009b0 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb400bd60 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc7086700 1 -- 172.21.2.117:0/48630790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc00a8290 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bc7086700 1 -- 172.21.2.117:0/48630790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc00a8290 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0bb4002030 con 0x7f0bc01a4290 2021-02-10T11:53:25.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.431 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0bb4000c90 con 0x7f0bc01a4290 2021-02-10T11:53:25.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.432 7f0bbd7fa700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 0x7f0ba001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:25.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.432 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0bb4032790 con 0x7f0bc01a4290 2021-02-10T11:53:25.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.432 7f0bbffff700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 0x7f0ba001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:25.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.432 7f0bbffff700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 0x7f0ba001abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:25.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.463 7f0bc7086700 1 -- 172.21.2.117:0/48630790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc0087290 con 0x7f0bc01a4290 2021-02-10T11:53:25.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.465 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0bb4006b00 con 0x7f0bc01a4290 2021-02-10T11:53:25.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.588 7f0bc7086700 1 -- 172.21.2.117:0/48630790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0bc0087290 con 0x7f0bc01a4290 2021-02-10T11:53:25.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.589 7f0bbd7fa700 1 -- 172.21.2.117:0/48630790 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+30766 (secure 0 0 0) 0x7f0bb4000c90 con 0x7f0bc01a4290 2021-02-10T11:53:25.591 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:25.591 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:25.591 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 143, 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:52:50.464421", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:53:25.592 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 31, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:53:25.593 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:53:25.594 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:53:25.595 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:53:25.596 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:53:25.597 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:53:25.598 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 30, 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:53:25.599 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:52:47.477707", 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:53:25.600 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "142", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:53:25.601 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:53:25.602 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.603 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:53:25.604 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 31, 2021-02-10T11:53:25.605 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:52:48.506496", 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:53:25.606 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "143", 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:53:25.607 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:53:25.608 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.609 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:53:25.610 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.611 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:53:25.612 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.613 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.614 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.615 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.616 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.617 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.618 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.619 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:53:25.620 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.621 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.622 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.623 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 140, 2021-02-10T11:53:25.624 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.625 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:53:25.626 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:53:25.627 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:53:25.628 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.629 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 139, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.630 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:53:25.631 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.632 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.633 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.634 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:53:25.635 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.636 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.637 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.638 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.640 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:53:25.641 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.642 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 139, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:53:25.643 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.644 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.645 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.646 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:53:25.647 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.648 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:53:25.649 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.650 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.651 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.652 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:53:25.653 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:53:25.654 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 140, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:53:25.655 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.656 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.657 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:53:25.658 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:53:25.659 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.660 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.661 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.662 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.664 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:53:25.665 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:53:25.666 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:53:25.667 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:53:25.668 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:53:25.668 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:53:25.668 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.590 7f0bc7086700 1 -- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 msgr2=0x7f0ba001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.590 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 0x7f0ba001abc0 crc :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.590 7f0bc7086700 1 -- 172.21.2.117:0/48630790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 msgr2=0x7f0bc01a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.590 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 rx=0x7f0bb4006f60 tx=0x7f0bb400b970).stop 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 -- 172.21.2.117:0/48630790 shutdown_connections 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0bc00a6fe0 0x7f0bc00a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.669 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0bc00ab280 0x7f0bc00ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ba0018730 0x7f0ba001abc0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 --2- 172.21.2.117:0/48630790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0bc01a4290 0x7f0bc01a4690 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:25.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 -- 172.21.2.117:0/48630790 >> 172.21.2.117:0/48630790 conn(0x7f0bc00874c0 msgr2=0x7f0bc0096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:25.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 -- 172.21.2.117:0/48630790 shutdown_connections 2021-02-10T11:53:25.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:25.591 7f0bc7086700 1 -- 172.21.2.117:0/48630790 wait complete. 2021-02-10T11:53:25.671 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.079 7fb989831700 1 Processor -- start 2021-02-10T11:53:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.079 7fb989831700 1 -- start start 2021-02-10T11:53:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb989831700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9840ef3f0 0x7fb9840eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb989831700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9840eb7f0 0x7fb9841b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb989831700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9840ede60 con 0x7fb9840d18e0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb989831700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9840ef7f0 con 0x7fb9840eb7f0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb989831700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840d1ce0 con 0x7fb9840ef3f0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9840eb7f0 0x7fb9841b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb9840eb7f0 0x7fb9841b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57112/0 (socket says 172.21.2.117:57112) 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 -- 172.21.2.117:0/3867368128 learned_addr learned my addr 172.21.2.117:0/3867368128 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 -- 172.21.2.117:0/3867368128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9840ef3f0 msgr2=0x7fb9840eb1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 --2- 172.21.2.117:0/3867368128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9840ef3f0 0x7fb9840eb1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 -- 172.21.2.117:0/3867368128 >> v1:172.21.2.125:6789/0 conn(0x7fb9840d18e0 legacy=0x7fb9840ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 -- 172.21.2.117:0/3867368128 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9841b8f90 con 0x7fb9840eb7f0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.080 7fb983d75700 1 --2- 172.21.2.117:0/3867368128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840eb7f0 0x7fb9841b7960 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7fb97c006f00 tx=0x7fb97c009f10).ready entity=mon.0 client_cookie=aef7174204563fbd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb982572700 1 -- 172.21.2.117:0/3867368128 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c00b3b0 con 0x7fb9840eb7f0 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb982572700 1 -- 172.21.2.117:0/3867368128 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00b730 con 0x7fb9840eb7f0 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb982572700 1 -- 172.21.2.117:0/3867368128 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb97c00bc50 con 0x7fb9840eb7f0 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 -- 172.21.2.117:0/3867368128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840eb7f0 msgr2=0x7fb9841b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 --2- 172.21.2.117:0/3867368128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840eb7f0 0x7fb9841b7960 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 rx=0x7fb97c006f00 tx=0x7fb97c009f10).stop 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 -- 172.21.2.117:0/3867368128 shutdown_connections 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 --2- 172.21.2.117:0/3867368128 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840eb7f0 0x7fb9841b7960 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 --2- 172.21.2.117:0/3867368128 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9840ef3f0 0x7fb9840eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 -- 172.21.2.117:0/3867368128 >> 172.21.2.117:0/3867368128 conn(0x7fb9840ced60 msgr2=0x7fb9840cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 -- 172.21.2.117:0/3867368128 shutdown_connections 2021-02-10T11:53:26.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.081 7fb989831700 1 -- 172.21.2.117:0/3867368128 wait complete. 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 Processor -- start 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 -- start start 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9840984c0 0x7fb98419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb98419dd40 0x7fb98419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840b1f30 con 0x7fb9840980c0 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98419d830 con 0x7fb9840984c0 2021-02-10T11:53:26.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb989831700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9841a55e0 con 0x7fb98419dd40 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb983574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb983574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57114/0 (socket says 172.21.2.117:57114) 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.096 7fb983574700 1 -- 172.21.2.117:0/150673192 learned_addr learned my addr 172.21.2.117:0/150673192 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 -- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb98419dd40 msgr2=0x7fb98419e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb98419dd40 0x7fb98419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 -- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9840984c0 msgr2=0x7fb98419d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9840984c0 0x7fb98419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 -- 172.21.2.117:0/150673192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98419e760 con 0x7fb9840980c0 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb983574700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 rx=0x7fb96c00b5b0 tx=0x7fb96c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb96c00bd40 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c000c60 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb989831700 1 -- 172.21.2.117:0/150673192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98419e760 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb989831700 1 -- 172.21.2.117:0/150673192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98419ecf0 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.097 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb96c0062a0 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.098 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb96c006020 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.098 7fb97bfff700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.098 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb96c032700 con 0x7fb9840980c0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.098 7fb982d73700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.099 7fb982d73700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 0x7fb96801abc0 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.129 7fb989831700 1 -- 172.21.2.117:0/150673192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb984087210 con 0x7fb9840980c0 2021-02-10T11:53:26.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.132 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb96c006230 con 0x7fb9840980c0 2021-02-10T11:53:26.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.260 7fb989831700 1 -- 172.21.2.117:0/150673192 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9840ede60 con 0x7fb9840980c0 2021-02-10T11:53:26.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.261 7fb97bfff700 1 -- 172.21.2.117:0/150673192 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fb96c006000 con 0x7fb9840980c0 2021-02-10T11:53:26.263 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:26.263 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":121,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:52:51.468783","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7415},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7415":{"gid":7415,"name":"b","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/154249133","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":154249133},{"type":"v1","addr":"172.21.2.125:6835","nonce":154249133}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:53:26.263 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 121 2021-02-10T11:53:26.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.262 7fb989831700 1 -- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 msgr2=0x7fb96801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.262 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 0x7fb96801abc0 crc :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:26.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.262 7fb989831700 1 -- 172.21.2.117:0/150673192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 msgr2=0x7fb98419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.262 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 rx=0x7fb96c00b5b0 tx=0x7fb96c00b950).stop 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 -- 172.21.2.117:0/150673192 shutdown_connections 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb98419dd40 0x7fb98419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9840984c0 0x7fb98419d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb9680186c0 0x7fb96801abc0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 --2- 172.21.2.117:0/150673192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9840980c0 0x7fb98419ca80 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 -- 172.21.2.117:0/150673192 >> 172.21.2.117:0/150673192 conn(0x7fb98419f610 msgr2=0x7fb9841a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 -- 172.21.2.117:0/150673192 shutdown_connections 2021-02-10T11:53:26.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.263 7fb989831700 1 -- 172.21.2.117:0/150673192 wait complete. 2021-02-10T11:53:26.273 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:53:26.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 Processor -- start 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 -- start start 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4981b07f0 0x7fb4980e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980e6170 0x7fb4980e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb4980e6b90 0x7fb4980e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4980e5ce0 con 0x7fb4981b07f0 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4980e75b0 con 0x7fb4980e6b90 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49d3b2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980e77f0 con 0x7fb4980e6170 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4981b07f0 0x7fb4980e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb4981b07f0 0x7fb4980e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57122/0 (socket says 172.21.2.117:57122) 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.684 7fb49707d700 1 -- 172.21.2.117:0/3866043606 learned_addr learned my addr 172.21.2.117:0/3866043606 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 -- 172.21.2.117:0/3866043606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980e6170 msgr2=0x7fb4980e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:26.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 --2- 172.21.2.117:0/3866043606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980e6170 0x7fb4980e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 -- 172.21.2.117:0/3866043606 >> v2:172.21.2.125:3300/0 conn(0x7fb4980e6b90 msgr2=0x7fb4980e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 --2- 172.21.2.117:0/3866043606 >> v2:172.21.2.125:3300/0 conn(0x7fb4980e6b90 0x7fb4980e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 -- 172.21.2.117:0/3866043606 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980e54b0 con 0x7fb4981b07f0 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49707d700 1 --2- 172.21.2.117:0/3866043606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4981b07f0 0x7fb4980e4e90 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7fb48c006700 tx=0x7fb48c009b80).ready entity=mon.0 client_cookie=df57a0b0e6927bbf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49607b700 1 -- 172.21.2.117:0/3866043606 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48c00c0c0 con 0x7fb4981b07f0 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49607b700 1 -- 172.21.2.117:0/3866043606 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00c3c0 con 0x7fb4981b07f0 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49607b700 1 -- 172.21.2.117:0/3866043606 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb48c00c8e0 con 0x7fb4981b07f0 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 -- 172.21.2.117:0/3866043606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4981b07f0 msgr2=0x7fb4980e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 --2- 172.21.2.117:0/3866043606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4981b07f0 0x7fb4980e4e90 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 rx=0x7fb48c006700 tx=0x7fb48c009b80).stop 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 -- 172.21.2.117:0/3866043606 shutdown_connections 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 --2- 172.21.2.117:0/3866043606 >> v2:172.21.2.125:3300/0 conn(0x7fb4980e6b90 0x7fb4980e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 --2- 172.21.2.117:0/3866043606 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980e6170 0x7fb4980e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 --2- 172.21.2.117:0/3866043606 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4981b07f0 0x7fb4980e4e90 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.685 7fb49d3b2700 1 -- 172.21.2.117:0/3866043606 >> 172.21.2.117:0/3866043606 conn(0x7fb4980ce500 msgr2=0x7fb4980e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.686 7fb49d3b2700 1 -- 172.21.2.117:0/3866043606 shutdown_connections 2021-02-10T11:53:26.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.686 7fb49d3b2700 1 -- 172.21.2.117:0/3866043606 wait complete. 2021-02-10T11:53:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.700 7fb49d3b2700 1 Processor -- start 2021-02-10T11:53:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.700 7fb49d3b2700 1 -- start start 2021-02-10T11:53:26.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980a9330 0x7fb4980a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4980a7700 0x7fb4980a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980a8db0 con 0x7fb4980a9d50 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb498105f50 con 0x7fb4980a7700 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49d3b2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980ab4c0 con 0x7fb4980a9330 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57130/0 (socket says 172.21.2.117:57130) 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 -- 172.21.2.117:0/2716270735 learned_addr learned my addr 172.21.2.117:0/2716270735 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 -- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980a9330 msgr2=0x7fb4980a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980a9330 0x7fb4980a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 -- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4980a7700 msgr2=0x7fb4980a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4980a7700 0x7fb4980a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 -- 172.21.2.117:0/2716270735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980a8120 con 0x7fb4980a9d50 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.701 7fb49687c700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7fb490009980 tx=0x7fb49000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49000c2f0 con 0x7fb4980a9d50 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49000c570 con 0x7fb4980a9d50 2021-02-10T11:53:26.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4980a8120 con 0x7fb4980a9d50 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4980a86b0 con 0x7fb4980a9d50 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb49000ca90 con 0x7fb4980a9d50 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb490001010 con 0x7fb4980a9d50 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 0x7fb47801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.702 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb490032a00 con 0x7fb4980a9d50 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.703 7fb49707d700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 0x7fb47801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:26.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.703 7fb49707d700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 0x7fb47801abc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:26.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.734 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4981b8070 con 0x7fb4980a9d50 2021-02-10T11:53:26.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.737 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4900221f0 con 0x7fb4980a9d50 2021-02-10T11:53:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:26.862 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb4980e77f0 con 0x7fb4980a9d50 2021-02-10T11:53:27.312 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:53:27.311 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:53:27.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.317 7fb47ffff700 1 -- 172.21.2.117:0/2716270735 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v122) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb4900221f0 con 0x7fb4980a9d50 2021-02-10T11:53:27.330 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 msgr2=0x7fb47801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 0x7fb47801abc0 crc :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 msgr2=0x7fb4980aa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 rx=0x7fb490009980 tx=0x7fb49000bcd0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 shutdown_connections 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4780186c0 0x7fb47801abc0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4980a9d50 0x7fb4980aa150 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4980a7700 0x7fb4980a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 --2- 172.21.2.117:0/2716270735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4980a9330 0x7fb4980a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.330 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 >> 172.21.2.117:0/2716270735 conn(0x7fb4980b97b0 msgr2=0x7fb498098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:27.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.331 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 shutdown_connections 2021-02-10T11:53:27.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.331 7fb49d3b2700 1 -- 172.21.2.117:0/2716270735 wait complete. 2021-02-10T11:53:27.340 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:27.362 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:53:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 Processor -- start 2021-02-10T11:53:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 -- start start 2021-02-10T11:53:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13c40d18e0 0x7f13c40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c40ede60 con 0x7f13c40eb000 2021-02-10T11:53:27.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c40d1ce0 con 0x7f13c40ef3f0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.752 7f13c9869700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c40ef7f0 con 0x7f13c40d18e0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c3574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13c40d18e0 0x7f13c40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c2d73700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f13c40eb000 0x7f13c40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57386/0 (socket says 172.21.2.117:57386) 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c2d73700 1 -- 172.21.2.117:0/3645955815 learned_addr learned my addr 172.21.2.117:0/3645955815 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c3574700 1 -- 172.21.2.117:0/3645955815 >> v1:172.21.2.125:6789/0 conn(0x7f13c40ef3f0 legacy=0x7f13c41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c3574700 1 -- 172.21.2.117:0/3645955815 >> v1:172.21.2.117:6789/0 conn(0x7f13c40eb000 legacy=0x7f13c40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c3574700 1 -- 172.21.2.117:0/3645955815 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c41bb180 con 0x7f13c40d18e0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c3574700 1 --2- 172.21.2.117:0/3645955815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40d18e0 0x7f13c40ea9e0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 rx=0x7f13b4006650 tx=0x7f13b4009a80).ready entity=mon.0 client_cookie=ab880164ae0991b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c2572700 1 -- 172.21.2.117:0/3645955815 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b40009b0 con 0x7f13c40d18e0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c2572700 1 -- 172.21.2.117:0/3645955815 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b400c0c0 con 0x7f13c40d18e0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c2572700 1 -- 172.21.2.117:0/3645955815 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b400c510 con 0x7f13c40d18e0 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c9869700 1 -- 172.21.2.117:0/3645955815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40d18e0 msgr2=0x7f13c40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.753 7f13c9869700 1 --2- 172.21.2.117:0/3645955815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40d18e0 0x7f13c40ea9e0 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 rx=0x7f13b4006650 tx=0x7f13b4009a80).stop 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.754 7f13c9869700 1 -- 172.21.2.117:0/3645955815 shutdown_connections 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.754 7f13c9869700 1 --2- 172.21.2.117:0/3645955815 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40d18e0 0x7f13c40ea9e0 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.754 7f13c9869700 1 -- 172.21.2.117:0/3645955815 >> 172.21.2.117:0/3645955815 conn(0x7f13c40ced60 msgr2=0x7f13c40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.754 7f13c9869700 1 -- 172.21.2.117:0/3645955815 shutdown_connections 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.754 7f13c9869700 1 -- 172.21.2.117:0/3645955815 wait complete. 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 Processor -- start 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 -- start start 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c40f4d60 0x7f13c40ba070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c40bb140 0x7f13c40a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40f5bc0 con 0x7f13c40bad40 2021-02-10T11:53:27.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40f5160 con 0x7f13c40f4d60 2021-02-10T11:53:27.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c9869700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40a73f0 con 0x7f13c40bb140 2021-02-10T11:53:27.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.769 7f13c2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:27.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57138/0 (socket says 172.21.2.117:57138) 2021-02-10T11:53:27.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 -- 172.21.2.117:0/3193552117 learned_addr learned my addr 172.21.2.117:0/3193552117 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:27.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c3d75700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c40bb140 0x7f13c40a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:27.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c3574700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c40f4d60 0x7f13c40ba070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:27.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 -- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c40bb140 msgr2=0x7f13c40a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c40bb140 0x7f13c40a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 -- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c40f4d60 msgr2=0x7f13c40ba070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c40f4d60 0x7f13c40ba070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 -- 172.21.2.117:0/3193552117 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c40a7610 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c2d73700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f13b8001c60 tx=0x7f13b800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b800acc0 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b8004140 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13b8004620 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c9869700 1 -- 172.21.2.117:0/3193552117 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c40a7610 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.770 7f13c9869700 1 -- 172.21.2.117:0/3193552117 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c41c5140 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.771 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13b8004320 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.771 7f13abfff700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 0x7f13a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.771 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f13b80326f0 con 0x7f13c40bad40 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.772 7f13c3574700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 0x7f13a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:27.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.772 7f13c3574700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 0x7f13a401abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:27.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.802 7f13c9869700 1 -- 172.21.2.117:0/3193552117 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4086140 con 0x7f13c40bad40 2021-02-10T11:53:27.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.805 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f13b8001090 con 0x7f13c40bad40 2021-02-10T11:53:27.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.933 7f13c9869700 1 -- 172.21.2.117:0/3193552117 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13c40d18e0 con 0x7f13c40bad40 2021-02-10T11:53:27.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.933 7f13abfff700 1 -- 172.21.2.117:0/3193552117 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f13b800acc0 con 0x7f13c40bad40 2021-02-10T11:53:27.936 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:27.936 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":122,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11745,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3954980866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3954980866},{"type":"v1","addr":"172.21.2.117:6833","nonce":3954980866}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":116}],"filesystems":[{"mdsmap":{"epoch":122,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:52:49.462120","modified":"2021-02-10 11:53:27.311618","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":144,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2021-02-10T11:53:27.936 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 122 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 -- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 msgr2=0x7f13a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 0x7f13a401abc0 crc :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 -- 172.21.2.117:0/3193552117 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 msgr2=0x7f13c40f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 rx=0x7f13b8001c60 tx=0x7f13b800a8d0).stop 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 -- 172.21.2.117:0/3193552117 shutdown_connections 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f13a40186c0 0x7f13a401abc0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13c40bad40 0x7f13c40f55a0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13c40bb140 0x7f13c40a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 --2- 172.21.2.117:0/3193552117 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13c40f4d60 0x7f13c40ba070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:27.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.935 7f13c9869700 1 -- 172.21.2.117:0/3193552117 >> 172.21.2.117:0/3193552117 conn(0x7f13c4099f80 msgr2=0x7f13c4092940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:27.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.936 7f13c9869700 1 -- 172.21.2.117:0/3193552117 shutdown_connections 2021-02-10T11:53:27.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:27.936 7f13c9869700 1 -- 172.21.2.117:0/3193552117 wait complete. 2021-02-10T11:53:27.945 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 Processor -- start 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 -- start start 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0ff40d15e0 0x7f0ff40e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40e7580 con 0x7f0ff40e7180 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40e77f0 con 0x7f0ff41b07f0 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ffa843700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff40e5ce0 con 0x7f0ff40d15e0 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ff37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0ff40e7180 0x7f0ff40e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57396/0 (socket says 172.21.2.117:57396) 2021-02-10T11:53:28.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.360 7f0ff37fe700 1 -- 172.21.2.117:0/1660597186 learned_addr learned my addr 172.21.2.117:0/1660597186 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551478230 0 0) 0x7f0fe80010b0 con 0x7f0ff40e7180 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fdc002fe0 con 0x7f0ff40e7180 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff8e60700 1 --2- 172.21.2.117:0/1660597186 >> v2:172.21.2.125:3300/0 conn(0x7f0ff40d15e0 0x7f0ff40e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1693773469 0 0) 0x7f0fe8001010 con 0x7f0ff40e7180 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 >> v2:172.21.2.125:3300/0 conn(0x7f0ff40d15e0 msgr2=0x7f0ff40e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 --2- 172.21.2.117:0/1660597186 >> v2:172.21.2.125:3300/0 conn(0x7f0ff40d15e0 0x7f0ff40e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 >> v1:172.21.2.125:6789/0 conn(0x7f0ff41b07f0 legacy=0x7f0ff40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40d19e0 con 0x7f0ff40e7180 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f0fe8001530 con 0x7f0ff40e7180 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 >> v1:172.21.2.117:6789/0 conn(0x7f0ff40e7180 legacy=0x7f0ff40e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fdc003a70 0x7f0fdc005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 0x7f0fdc008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fdc0095a0 0x7f0fdc00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00c310 con 0x7f0fdc0067a0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00c530 con 0x7f0fdc0095a0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00c750 con 0x7f0fdc003a70 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.361 7f0ff37fe700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 0x7f0fdc008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff8e60700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fdc0095a0 0x7f0fdc00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff3fff700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fdc003a70 0x7f0fdc005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 -- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fdc003a70 msgr2=0x7f0fdc005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0fdc003a70 0x7f0fdc005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 -- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fdc0095a0 msgr2=0x7f0fdc00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0fdc0095a0 0x7f0fdc00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 -- 172.21.2.117:0/1660597186 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc0509c0 con 0x7f0fdc0067a0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff8e60700 1 -- 172.21.2.117:0/1660597186 reap_dead start 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff37fe700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 0x7f0fdc008ca0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 rx=0x7f0fe8001380 tx=0x7f0fe8007d30).ready entity=mon.0 client_cookie=2c5117874e7b92c7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:28.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe8001010 con 0x7f0fdc0067a0 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800a310 con 0x7f0fdc0067a0 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ff2ffd700 1 -- 172.21.2.117:0/1660597186 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe800a830 con 0x7f0fdc0067a0 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ffa843700 1 -- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 msgr2=0x7f0fdc008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.362 7f0ffa843700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 0x7f0fdc008ca0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 rx=0x7f0fe8001380 tx=0x7f0fe8007d30).stop 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.363 7f0ffa843700 1 -- 172.21.2.117:0/1660597186 shutdown_connections 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.363 7f0ffa843700 1 --2- 172.21.2.117:0/1660597186 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0fdc0067a0 0x7f0fdc008ca0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.363 7f0ffa843700 1 -- 172.21.2.117:0/1660597186 >> 172.21.2.117:0/1660597186 conn(0x7f0ff40ce500 msgr2=0x7f0ff40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.363 7f0ffa843700 1 -- 172.21.2.117:0/1660597186 shutdown_connections 2021-02-10T11:53:28.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.363 7f0ffa843700 1 -- 172.21.2.117:0/1660597186 wait complete. 2021-02-10T11:53:28.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 Processor -- start 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 -- start start 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ff40b3f30 0x7f0ff410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 0x7f0ff409b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff410e2b0 con 0x7f0ff41af980 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff409a8f0 con 0x7f0ff41afd80 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ffa843700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40992b0 con 0x7f0ff40b3f30 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57162/0 (socket says 172.21.2.117:57162) 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.378 7f0ff37fe700 1 -- 172.21.2.117:0/2641541440 learned_addr learned my addr 172.21.2.117:0/2641541440 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 -- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ff40b3f30 msgr2=0x7f0ff410e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:28.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ff40b3f30 0x7f0ff410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 -- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 msgr2=0x7f0ff409b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff8e60700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 0x7f0ff409b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 0x7f0ff409b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 -- 172.21.2.117:0/2641541440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40994d0 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff8e60700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 0x7f0ff409b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff37fe700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7f0fe8001200 tx=0x7f0fe800a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe800ac60 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe8004350 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40994d0 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff40994d0 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.379 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0fe8004870 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.380 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0fe8004530 con 0x7f0ff41af980 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.380 7f0ff0ff9700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 0x7f0fd401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:28.385 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.380 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f0fe8032740 con 0x7f0ff41af980 2021-02-10T11:53:28.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.381 7f0ff3fff700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 0x7f0fd401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:28.386 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.381 7f0ff3fff700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 0x7f0fd401abc0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:28.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.410 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff40e77f0 con 0x7f0ff41af980 2021-02-10T11:53:28.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.412 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0fe801f750 con 0x7f0ff41af980 2021-02-10T11:53:28.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:28.538 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0ff40e77f0 con 0x7f0ff41af980 2021-02-10T11:53:29.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.322 7f0ff0ff9700 1 -- 172.21.2.117:0/2641541440 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v123) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0fe801f510 con 0x7f0ff41af980 2021-02-10T11:53:29.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.358 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 msgr2=0x7f0fd401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.358 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 0x7f0fd401abc0 crc :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.358 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 msgr2=0x7f0ff409ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.360 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.358 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 rx=0x7f0fe8001200 tx=0x7f0fe800a870).stop 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 shutdown_connections 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0fd40186c0 0x7f0fd401abc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ff41af980 0x7f0ff409ae70 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ff41afd80 0x7f0ff409b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 --2- 172.21.2.117:0/2641541440 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ff40b3f30 0x7f0ff410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 >> 172.21.2.117:0/2641541440 conn(0x7f0ff419f010 msgr2=0x7f0ff4191b20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 shutdown_connections 2021-02-10T11:53:29.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.359 7f0ffa843700 1 -- 172.21.2.117:0/2641541440 wait complete. 2021-02-10T11:53:29.369 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:53:29.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.781 7f5643b0f700 1 Processor -- start 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 -- start start 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f563c1b07e0 0x7f563c0e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c0e6160 0x7f563c0e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563c0e77e0 con 0x7f563c0e6b80 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563c0e5cd0 con 0x7f563c1b07e0 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f5643b0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c0e54a0 con 0x7f563c0e6160 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f564192b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f563c1b07e0 0x7f563c0e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f564192b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f563c1b07e0 0x7f563c0e4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57166/0 (socket says 172.21.2.117:57166) 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.782 7f564192b700 1 -- 172.21.2.117:0/2442676482 learned_addr learned my addr 172.21.2.117:0/2442676482 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564112a700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c0e6160 0x7f563c0e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564192b700 1 -- 172.21.2.117:0/2442676482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c0e6160 msgr2=0x7f563c0e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564192b700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c0e6160 0x7f563c0e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564192b700 1 -- 172.21.2.117:0/2442676482 >> v1:172.21.2.125:6789/0 conn(0x7f563c0e6b80 legacy=0x7f563c0e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564192b700 1 -- 172.21.2.117:0/2442676482 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c0e8d70 con 0x7f563c1b07e0 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f564192b700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c1b07e0 0x7f563c0e4e80 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b80).ready entity=mon.0 client_cookie=835e45c736f752d1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f5640929700 1 -- 172.21.2.117:0/2442676482 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56300009b0 con 0x7f563c1b07e0 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f5640929700 1 -- 172.21.2.117:0/2442676482 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563000c0c0 con 0x7f563c1b07e0 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f5640929700 1 -- 172.21.2.117:0/2442676482 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f563000c600 con 0x7f563c1b07e0 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f5643b0f700 1 -- 172.21.2.117:0/2442676482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c1b07e0 msgr2=0x7f563c0e4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.783 7f5643b0f700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c1b07e0 0x7f563c0e4e80 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 rx=0x7f5630006700 tx=0x7f5630009b80).stop 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 -- 172.21.2.117:0/2442676482 shutdown_connections 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c0e6160 0x7f563c0e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 --2- 172.21.2.117:0/2442676482 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c1b07e0 0x7f563c0e4e80 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 -- 172.21.2.117:0/2442676482 >> 172.21.2.117:0/2442676482 conn(0x7f563c0ce4f0 msgr2=0x7f563c0e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 -- 172.21.2.117:0/2442676482 shutdown_connections 2021-02-10T11:53:29.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.784 7f5643b0f700 1 -- 172.21.2.117:0/2442676482 wait complete. 2021-02-10T11:53:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.798 7f5643b0f700 1 Processor -- start 2021-02-10T11:53:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.798 7f5643b0f700 1 -- start start 2021-02-10T11:53:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f563c0804b0 0x7f563c1a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c1a3460 0x7f563c195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c197250 con 0x7f563c0800b0 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c195750 con 0x7f563c0804b0 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f5643b0f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563c1962f0 con 0x7f563c1a3460 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f564192b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f564192b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57170/0 (socket says 172.21.2.117:57170) 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f564192b700 1 -- 172.21.2.117:0/4151138113 learned_addr learned my addr 172.21.2.117:0/4151138113 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f564112a700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f563c0804b0 0x7f563c1a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.799 7f564212c700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c1a3460 0x7f563c195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 -- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c1a3460 msgr2=0x7f563c195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c1a3460 0x7f563c195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 -- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f563c0804b0 msgr2=0x7f563c1a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f563c0804b0 0x7f563c1a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:29.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 -- 172.21.2.117:0/4151138113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c1964d0 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f564192b700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f563000b580 tx=0x7f563000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630002030 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5630002370 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c1964d0 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5630002890 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.800 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c1964d0 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.801 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5630000c90 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.801 7f562e7fc700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 0x7f562401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.801 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f56300329e0 con 0x7f563c0800b0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.801 7f564112a700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 0x7f562401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:29.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.801 7f564112a700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 0x7f562401ac10 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:29.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.832 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c07e8b0 con 0x7f563c0800b0 2021-02-10T11:53:29.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.835 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5630009020 con 0x7f563c0800b0 2021-02-10T11:53:29.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:29.961 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f563c0e77e0 con 0x7f563c0800b0 2021-02-10T11:53:30.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.326 7f562e7fc700 1 -- 172.21.2.117:0/4151138113 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) v1 ==== 185+0+0 (secure 0 0 0) 0x7f5630000c90 con 0x7f563c0800b0 2021-02-10T11:53:30.329 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 msgr2=0x7f562401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 0x7f562401ac10 crc :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 msgr2=0x7f563c1a2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f563000b580 tx=0x7f563000b920).stop 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 shutdown_connections 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f563c1a3460 0x7f563c195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f563c0804b0 0x7f563c1a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5624018730 0x7f562401ac10 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 --2- 172.21.2.117:0/4151138113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f563c0800b0 0x7f563c1a2820 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 >> 172.21.2.117:0/4151138113 conn(0x7f563c1998a0 msgr2=0x7f563c1aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:30.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 shutdown_connections 2021-02-10T11:53:30.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.329 7f5643b0f700 1 -- 172.21.2.117:0/4151138113 wait complete. 2021-02-10T11:53:30.339 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:53:30.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.753 7fd251a66700 1 Processor -- start 2021-02-10T11:53:30.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.753 7fd251a66700 1 -- start start 2021-02-10T11:53:30.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd251a66700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd24c085450 0x7fd24c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:30.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd251a66700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c085930 con 0x7fd24c0e5240 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd251a66700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c1b09e0 con 0x7fd24c0e4e40 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd251a66700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c0e5cc0 con 0x7fd24c085450 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd24c085450 0x7fd24c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24affd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd24c0e5240 0x7fd24c084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57428/0 (socket says 172.21.2.117:57428) 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24affd700 1 -- 172.21.2.117:0/2328869444 learned_addr learned my addr 172.21.2.117:0/2328869444 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24bfff700 1 -- 172.21.2.117:0/2328869444 >> v1:172.21.2.125:6789/0 conn(0x7fd24c0e4e40 legacy=0x7fd24c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24bfff700 1 -- 172.21.2.117:0/2328869444 >> v1:172.21.2.117:6789/0 conn(0x7fd24c0e5240 legacy=0x7fd24c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.754 7fd24bfff700 1 -- 172.21.2.117:0/2328869444 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c084840 con 0x7fd24c085450 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd24bfff700 1 --2- 172.21.2.117:0/2328869444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c085450 0x7fd24c086ec0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7fd23c006ab0 tx=0x7fd23c009ac0).ready entity=mon.0 client_cookie=c81f4167d7843c2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd24a7fc700 1 -- 172.21.2.117:0/2328869444 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c0066d0 con 0x7fd24c085450 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd24a7fc700 1 -- 172.21.2.117:0/2328869444 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c0022e0 con 0x7fd24c085450 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd24a7fc700 1 -- 172.21.2.117:0/2328869444 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd23c0027c0 con 0x7fd24c085450 2021-02-10T11:53:30.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 -- 172.21.2.117:0/2328869444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c085450 msgr2=0x7fd24c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 --2- 172.21.2.117:0/2328869444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c085450 0x7fd24c086ec0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 rx=0x7fd23c006ab0 tx=0x7fd23c009ac0).stop 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 -- 172.21.2.117:0/2328869444 shutdown_connections 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 --2- 172.21.2.117:0/2328869444 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c085450 0x7fd24c086ec0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 -- 172.21.2.117:0/2328869444 >> 172.21.2.117:0/2328869444 conn(0x7fd24c0ce4e0 msgr2=0x7fd24c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 -- 172.21.2.117:0/2328869444 shutdown_connections 2021-02-10T11:53:30.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.755 7fd251a66700 1 -- 172.21.2.117:0/2328869444 wait complete. 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.770 7fd251a66700 1 Processor -- start 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.770 7fd251a66700 1 -- start start 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.770 7fd251a66700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 0x7fd24c0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.770 7fd251a66700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 0x7fd24c0a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd251a66700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd24c0a35e0 0x7fd24c1c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd251a66700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0a4f40 con 0x7fd24c0a49c0 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd251a66700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0bbe50 con 0x7fd24c0a35e0 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd251a66700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c0bc030 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 0x7fd24c0a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 0x7fd24c0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24affd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 0x7fd24c0a2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57188/0 (socket says 172.21.2.117:57188) 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24affd700 1 -- 172.21.2.117:0/59559755 learned_addr learned my addr 172.21.2.117:0/59559755 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24bfff700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd24c0a35e0 0x7fd24c1c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 -- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd24c0a35e0 msgr2=0x7fd24c1c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd24c0a35e0 0x7fd24c1c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 -- 172.21.2.117:0/59559755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 msgr2=0x7fd24c0a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 0x7fd24c0a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 -- 172.21.2.117:0/59559755 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c0a39e0 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd24b7fe700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 0x7fd24c0a43a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fd240006e90 tx=0x7fd24000b860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.771 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd240006340 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24000bc50 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2400022b0 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd251a66700 1 -- 172.21.2.117:0/59559755 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c0a39e0 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd251a66700 1 -- 172.21.2.117:0/59559755 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c1c9c00 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2400052e0 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd233fff700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 0x7fd22c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.772 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5568+0+0 (secure 0 0 0) 0x7fd240032440 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.773 7fd24affd700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 0x7fd22c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:30.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.773 7fd24affd700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 0x7fd22c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:30.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.803 7fd251a66700 1 -- 172.21.2.117:0/59559755 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c0e5eb0 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.806 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd240000960 con 0x7fd24c0a3fa0 2021-02-10T11:53:30.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:30.935 7fd251a66700 1 -- 172.21.2.117:0/59559755 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd24c0e5eb0 con 0x7fd24c0a3fa0 2021-02-10T11:53:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.333 7fd233fff700 1 -- 172.21.2.117:0/59559755 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd2400052e0 con 0x7fd24c0a3fa0 2021-02-10T11:53:31.354 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:53:31.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 -- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 msgr2=0x7fd22c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 0x7fd22c01abc0 crc :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 -- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 msgr2=0x7fd24c0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 0x7fd24c0a43a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fd240006e90 tx=0x7fd24000b860).stop 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 -- 172.21.2.117:0/59559755 shutdown_connections 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd22c0186c0 0x7fd22c01abc0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd24c0a49c0 0x7fd24c0a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd24c0a35e0 0x7fd24c1c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 --2- 172.21.2.117:0/59559755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd24c0a3fa0 0x7fd24c0a43a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 -- 172.21.2.117:0/59559755 >> 172.21.2.117:0/59559755 conn(0x7fd24c09fb50 msgr2=0x7fd24c0a6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:31.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.354 7fd251a66700 1 -- 172.21.2.117:0/59559755 shutdown_connections 2021-02-10T11:53:31.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.355 7fd251a66700 1 -- 172.21.2.117:0/59559755 wait complete. 2021-02-10T11:53:31.364 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:53:31.364 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:53:31.364 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:53:31.364 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:53:31.365 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:53:31.365 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:53:31.365 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:53:31.365 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:53:31.371 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:53:31.371 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:53:31.372 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:53:31.372 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:53:31.413 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:53:31.413 DEBUG:teuthology.parallel:result is None 2021-02-10T11:53:31.414 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:53:31.415 DEBUG:teuthology.parallel:result is None 2021-02-10T11:53:31.415 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:53:31.820 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:53:31.824 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:53:31.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.829 7fd908ca2700 1 Processor -- start 2021-02-10T11:53:31.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 -- start start 2021-02-10T11:53:31.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 0x7fd904084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904085930 con 0x7fd9041b0810 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd904087430 con 0x7fd9040e5240 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd908ca2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd904084840 con 0x7fd9040e4e40 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 0x7fd904084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54884/0 (socket says 172.21.2.117:54884) 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 -- 172.21.2.117:0/470458023 learned_addr learned my addr 172.21.2.117:0/470458023 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90205d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 0x7fd904084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57208/0 (socket says 172.21.2.117:57208) 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 -- 172.21.2.117:0/470458023 >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 msgr2=0x7fd904084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 --2- 172.21.2.117:0/470458023 >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 0x7fd904084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 -- 172.21.2.117:0/470458023 >> v1:172.21.2.125:6789/0 conn(0x7fd9041b0810 legacy=0x7fd904085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.830 7fd90285e700 1 -- 172.21.2.117:0/470458023 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9040e5cc0 con 0x7fd9040e4e40 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd90285e700 1 --2- 172.21.2.117:0/470458023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fd8ec006700 tx=0x7fd8ec009b00).ready entity=mon.2 client_cookie=2e8010b7f993b02f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd90185c700 1 -- 172.21.2.117:0/470458023 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec00c0c0 con 0x7fd9040e4e40 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd90185c700 1 -- 172.21.2.117:0/470458023 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec00c400 con 0x7fd9040e4e40 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd90185c700 1 -- 172.21.2.117:0/470458023 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec00c8e0 con 0x7fd9040e4e40 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 -- 172.21.2.117:0/470458023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 msgr2=0x7fd9041b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 --2- 172.21.2.117:0/470458023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fd8ec006700 tx=0x7fd8ec009b00).stop 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 -- 172.21.2.117:0/470458023 shutdown_connections 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 --2- 172.21.2.117:0/470458023 >> v2:172.21.2.117:3300/0 conn(0x7fd9040e5240 0x7fd904084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 --2- 172.21.2.117:0/470458023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9040e4e40 0x7fd9041b01f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 -- 172.21.2.117:0/470458023 >> 172.21.2.117:0/470458023 conn(0x7fd9040ce4e0 msgr2=0x7fd9040e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 -- 172.21.2.117:0/470458023 shutdown_connections 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.831 7fd908ca2700 1 -- 172.21.2.117:0/470458023 wait complete. 2021-02-10T11:53:31.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.846 7fd908ca2700 1 Processor -- start 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.846 7fd908ca2700 1 -- start start 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 0x7fd9041b8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9040aa8a0 0x7fd90418ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90418f9b0 con 0x7fd90418f100 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90419e960 con 0x7fd9040aa8a0 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd908ca2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90419eb40 con 0x7fd90419e560 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90205d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57212/0 (socket says 172.21.2.117:57212) 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90205d700 1 -- 172.21.2.117:0/1104926340 learned_addr learned my addr 172.21.2.117:0/1104926340 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 0x7fd9041b8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90305f700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9040aa8a0 0x7fd90418ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 -- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9040aa8a0 msgr2=0x7fd90418ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9040aa8a0 0x7fd90418ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 -- 172.21.2.117:0/1104926340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 msgr2=0x7fd9040aa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 -- 172.21.2.117:0/1104926340 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90418f500 con 0x7fd90419e560 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90205d700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd90285e700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 0x7fd9041b8220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fd8ec000bd0 tx=0x7fd8ec00b890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec000c90 con 0x7fd90419e560 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.847 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec009090 con 0x7fd90419e560 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90418f500 con 0x7fd90419e560 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904190840 con 0x7fd90419e560 2021-02-10T11:53:31.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd8ec002030 con 0x7fd90419e560 2021-02-10T11:53:31.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd8ec009f30 con 0x7fd90419e560 2021-02-10T11:53:31.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd8fb7fe700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 0x7fd8e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:31.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.848 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fd8ec032170 con 0x7fd90419e560 2021-02-10T11:53:31.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.849 7fd90205d700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 0x7fd8e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:31.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.849 7fd90205d700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 0x7fd8e801abc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:31.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.879 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd904085930 con 0x7fd90419e560 2021-02-10T11:53:31.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:31.882 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd8ec0167a0 con 0x7fd90419e560 2021-02-10T11:53:32.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.022 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd904085930 con 0x7fd90419e560 2021-02-10T11:53:32.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.370 7fd8fb7fe700 1 -- 172.21.2.117:0/1104926340 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v147) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd8ec0167a0 con 0x7fd90419e560 2021-02-10T11:53:32.390 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 msgr2=0x7fd8e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 0x7fd8e801abc0 crc :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 msgr2=0x7fd9041b8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 0x7fd9041b8220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fd8ec000bd0 tx=0x7fd8ec00b890).stop 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 shutdown_connections 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd8e80186c0 0x7fd8e801abc0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd90418f100 0x7fd9040aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9040aa8a0 0x7fd90418ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 --2- 172.21.2.117:0/1104926340 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd90419e560 0x7fd9041b8220 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 >> 172.21.2.117:0/1104926340 conn(0x7fd90419f650 msgr2=0x7fd90418c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:32.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 shutdown_connections 2021-02-10T11:53:32.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.391 7fd908ca2700 1 -- 172.21.2.117:0/1104926340 wait complete. 2021-02-10T11:53:32.401 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:53:32.402 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:53:32.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.811 7f6c787ec700 1 Processor -- start 2021-02-10T11:53:32.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.811 7f6c787ec700 1 -- start start 2021-02-10T11:53:32.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.812 7f6c787ec700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6c700d18e0 0x7f6c700ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.812 7f6c787ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 0x7f6c701b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.812 7f6c787ec700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c700ede60 con 0x7f6c700eb000 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.812 7f6c787ec700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c700ef7f0 con 0x7f6c700d18e0 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.812 7f6c787ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700d1ce0 con 0x7f6c700ef3f0 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76e09700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 0x7f6c701b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6c700d18e0 0x7f6c700ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c75e07700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6c700eb000 0x7f6c700eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50394/0 (socket says 172.21.2.117:50394) 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c75e07700 1 -- 172.21.2.117:0/3260513573 learned_addr learned my addr 172.21.2.117:0/3260513573 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 -- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 msgr2=0x7f6c701b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 0x7f6c701b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 -- 172.21.2.117:0/3260513573 >> v1:172.21.2.125:6789/0 conn(0x7f6c700eb000 legacy=0x7f6c700eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 -- 172.21.2.117:0/3260513573 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c701b8f90 con 0x7f6c700d18e0 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76e09700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 0x7f6c701b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c76608700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700d18e0 0x7f6c700ea9e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f6c6c006650 tx=0x7f6c6c009a80).ready entity=mon.1 client_cookie=a08a8a1bccbab050 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.813 7f6c75606700 1 -- 172.21.2.117:0/3260513573 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6c006390 con 0x7f6c700d18e0 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c75606700 1 -- 172.21.2.117:0/3260513573 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c0009b0 con 0x7f6c700d18e0 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c75606700 1 -- 172.21.2.117:0/3260513573 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6c00c2b0 con 0x7f6c700d18e0 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 -- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700d18e0 msgr2=0x7f6c700ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700d18e0 0x7f6c700ea9e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f6c6c006650 tx=0x7f6c6c009a80).stop 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 -- 172.21.2.117:0/3260513573 shutdown_connections 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700ef3f0 0x7f6c701b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 --2- 172.21.2.117:0/3260513573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700d18e0 0x7f6c700ea9e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:32.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 -- 172.21.2.117:0/3260513573 >> 172.21.2.117:0/3260513573 conn(0x7f6c700ced60 msgr2=0x7f6c700cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:32.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 -- 172.21.2.117:0/3260513573 shutdown_connections 2021-02-10T11:53:32.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.814 7f6c787ec700 1 -- 172.21.2.117:0/3260513573 wait complete. 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 Processor -- start 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 -- start start 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700b7310 0x7f6c700aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700ab2c0 0x7f6c700abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700ab8c0 con 0x7f6c700b77e0 2021-02-10T11:53:32.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700ac460 con 0x7f6c700ab2c0 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c787ec700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700ac680 con 0x7f6c700b7310 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c76608700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c76608700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57224/0 (socket says 172.21.2.117:57224) 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.829 7f6c76608700 1 -- 172.21.2.117:0/1103514888 learned_addr learned my addr 172.21.2.117:0/1103514888 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 -- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700b7310 msgr2=0x7f6c700aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700b7310 0x7f6c700aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 -- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700ab2c0 msgr2=0x7f6c700abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700ab2c0 0x7f6c700abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 -- 172.21.2.117:0/1103514888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c700ac8a0 con 0x7f6c700b77e0 2021-02-10T11:53:32.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c76608700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 rx=0x7f6c6c006f70 tx=0x7f6c6c00b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6c0009b0 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c006070 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6c6c0052e0 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c700ac8a0 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.830 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c700ac8a0 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.831 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6c6c0052e0 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.831 7f6c66ffd700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 0x7f6c5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.831 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f6c6c032360 con 0x7f6c700b77e0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.831 7f6c75e07700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 0x7f6c5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:32.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.832 7f6c75e07700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 0x7f6c5801abc0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:32.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.862 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c70087290 con 0x7f6c700b77e0 2021-02-10T11:53:32.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.864 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6c6c006bc0 con 0x7f6c700b77e0 2021-02-10T11:53:32.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.997 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6c700ede60 con 0x7f6c700b77e0 2021-02-10T11:53:32.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:32.998 7f6c66ffd700 1 -- 172.21.2.117:0/1103514888 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v84) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f6c6c0052e0 con 0x7f6c700b77e0 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.000 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.001 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:53:33.003 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.005 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.006 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:53:33.008 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:53:33.010 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:53:33.011 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:33.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:53:33.013 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:33.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 msgr2=0x7f6c5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 0x7f6c5801abc0 crc :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 msgr2=0x7f6c700b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 rx=0x7f6c6c006f70 tx=0x7f6c6c00b8e0).stop 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 shutdown_connections 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c700ab2c0 0x7f6c700abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c700b7310 0x7f6c700aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6c580186c0 0x7f6c5801abc0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 --2- 172.21.2.117:0/1103514888 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c700b77e0 0x7f6c700b6cf0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 >> 172.21.2.117:0/1103514888 conn(0x7f6c700874c0 msgr2=0x7f6c700af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 shutdown_connections 2021-02-10T11:53:33.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.000 7f6c787ec700 1 -- 172.21.2.117:0/1103514888 wait complete. 2021-02-10T11:53:33.016 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.426 7fbf9201d700 1 Processor -- start 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.426 7fbf9201d700 1 -- start start 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf9201d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf8c0ef3f0 0x7fbf8c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf9201d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf8c0eb7f0 0x7fbf8c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf9201d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c0ede60 con 0x7fbf8c0d18e0 2021-02-10T11:53:33.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf9201d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c0ef7f0 con 0x7fbf8c0eb7f0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf9201d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c0d1ce0 con 0x7fbf8c0ef3f0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf8b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbf8c0d18e0 0x7fbf8c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57482/0 (socket says 172.21.2.117:57482) 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf8b7fe700 1 -- 172.21.2.117:0/3832451586 learned_addr learned my addr 172.21.2.117:0/3832451586 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf8bfff700 1 --2- 172.21.2.117:0/3832451586 >> v2:172.21.2.117:3300/0 conn(0x7fbf8c0eb7f0 0x7fbf8c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.427 7fbf8affd700 1 --2- 172.21.2.117:0/3832451586 >> v2:172.21.2.125:3300/0 conn(0x7fbf8c0ef3f0 0x7fbf8c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8a7fc700 1 -- 172.21.2.117:0/3832451586 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958928400 0 0) 0x7fbf84001060 con 0x7fbf8c0d18e0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8a7fc700 1 -- 172.21.2.117:0/3832451586 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf74002fe0 con 0x7fbf8c0d18e0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8bfff700 1 -- 172.21.2.117:0/3832451586 >> v2:172.21.2.125:3300/0 conn(0x7fbf8c0ef3f0 msgr2=0x7fbf8c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8bfff700 1 --2- 172.21.2.117:0/3832451586 >> v2:172.21.2.125:3300/0 conn(0x7fbf8c0ef3f0 0x7fbf8c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8bfff700 1 -- 172.21.2.117:0/3832451586 >> v1:172.21.2.117:6789/0 conn(0x7fbf8c0d18e0 legacy=0x7fbf8c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8bfff700 1 -- 172.21.2.117:0/3832451586 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c1b8f90 con 0x7fbf8c0eb7f0 2021-02-10T11:53:33.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8bfff700 1 --2- 172.21.2.117:0/3832451586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0eb7f0 0x7fbf8c1b7960 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7fbf7c006ab0 tx=0x7fbf7c009b40).ready entity=mon.0 client_cookie=ce421ef3f4f73297 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8a7fc700 1 -- 172.21.2.117:0/3832451586 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf7c0022e0 con 0x7fbf8c0eb7f0 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8a7fc700 1 -- 172.21.2.117:0/3832451586 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c002560 con 0x7fbf8c0eb7f0 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf8a7fc700 1 -- 172.21.2.117:0/3832451586 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf7c0029f0 con 0x7fbf8c0eb7f0 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf9201d700 1 -- 172.21.2.117:0/3832451586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0eb7f0 msgr2=0x7fbf8c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.428 7fbf9201d700 1 --2- 172.21.2.117:0/3832451586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0eb7f0 0x7fbf8c1b7960 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 rx=0x7fbf7c006ab0 tx=0x7fbf7c009b40).stop 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 -- 172.21.2.117:0/3832451586 shutdown_connections 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 --2- 172.21.2.117:0/3832451586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0eb7f0 0x7fbf8c1b7960 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 --2- 172.21.2.117:0/3832451586 >> v2:172.21.2.125:3300/0 conn(0x7fbf8c0ef3f0 0x7fbf8c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 -- 172.21.2.117:0/3832451586 >> 172.21.2.117:0/3832451586 conn(0x7fbf8c0ced60 msgr2=0x7fbf8c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 -- 172.21.2.117:0/3832451586 shutdown_connections 2021-02-10T11:53:33.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.429 7fbf9201d700 1 -- 172.21.2.117:0/3832451586 wait complete. 2021-02-10T11:53:33.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 Processor -- start 2021-02-10T11:53:33.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 -- start start 2021-02-10T11:53:33.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 0x7fbf8c0b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 0x7fbf8c0aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0ab8c0 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0ac460 con 0x7fbf8c0b77e0 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.444 7fbf9201d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0ac680 con 0x7fbf8c0b7310 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57238/0 (socket says 172.21.2.117:57238) 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 -- 172.21.2.117:0/889004131 learned_addr learned my addr 172.21.2.117:0/889004131 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8affd700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 0x7fbf8c0aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8b7fe700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 0x7fbf8c0b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 -- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 msgr2=0x7fbf8c0aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 0x7fbf8c0aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 -- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 msgr2=0x7fbf8c0b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 0x7fbf8c0b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 -- 172.21.2.117:0/889004131 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0ac8a0 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8b7fe700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 0x7fbf8c0b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8affd700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 0x7fbf8c0aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf8bfff700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7fbf7c006a30 tx=0x7fbf7c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf7c00bd80 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c009020 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf9201d700 1 -- 172.21.2.117:0/889004131 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c0ac8a0 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf9201d700 1 -- 172.21.2.117:0/889004131 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c0ace30 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.445 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf7c002270 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.446 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf7c002020 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.446 7fbf73fff700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 0x7fbf6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.446 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fbf7c032370 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.446 7fbf8b7fe700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 0x7fbf6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:33.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.447 7fbf8b7fe700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 0x7fbf6c01abc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:33.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.477 7fbf9201d700 1 -- 172.21.2.117:0/889004131 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c087290 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.479 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf7c0093e0 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.603 7fbf9201d700 1 -- 172.21.2.117:0/889004131 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf8c0ede60 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.604 7fbf73fff700 1 -- 172.21.2.117:0/889004131 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+11387 (secure 0 0 0) 0x7fbf7c002000 con 0x7fbf8c0ab2c0 2021-02-10T11:53:33.606 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:33.606 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":147,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:32.366200","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.605 7fbf9201d700 1 -- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 msgr2=0x7fbf6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.605 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 0x7fbf6c01abc0 crc :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.605 7fbf9201d700 1 -- 172.21.2.117:0/889004131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 msgr2=0x7fbf8c0abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.605 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 rx=0x7fbf7c006a30 tx=0x7fbf7c00b990).stop 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 -- 172.21.2.117:0/889004131 shutdown_connections 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf6c0186c0 0x7fbf6c01abc0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf8c0ab2c0 0x7fbf8c0abe40 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf8c0b7310 0x7fbf8c0aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 --2- 172.21.2.117:0/889004131 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf8c0b77e0 0x7fbf8c0b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 -- 172.21.2.117:0/889004131 >> 172.21.2.117:0/889004131 conn(0x7fbf8c0874c0 msgr2=0x7fbf8c0af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 -- 172.21.2.117:0/889004131 shutdown_connections 2021-02-10T11:53:33.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:33.606 7fbf9201d700 1 -- 172.21.2.117:0/889004131 wait complete. 2021-02-10T11:53:33.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:53:33.617 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:53:34.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.030 7fe18c054700 1 Processor -- start 2021-02-10T11:53:34.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.030 7fe18c054700 1 -- start start 2021-02-10T11:53:34.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1840d18f0 0x7fe1840eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 0x7fe1840eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 0x7fe1840ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1840edee0 con 0x7fe1840d18f0 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1840eba60 con 0x7fe1840ee3e0 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18c054700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840d1cf0 con 0x7fe1840eb040 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1840d18f0 0x7fe1840eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1840d18f0 0x7fe1840eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57246/0 (socket says 172.21.2.117:57246) 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 -- 172.21.2.117:0/3672718994 learned_addr learned my addr 172.21.2.117:0/3672718994 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18a671700 1 --2- 172.21.2.117:0/3672718994 >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 0x7fe1840ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 -- 172.21.2.117:0/3672718994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 msgr2=0x7fe1840eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18966f700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 0x7fe1840eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 0x7fe1840eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 -- 172.21.2.117:0/3672718994 >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 msgr2=0x7fe1840ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 --2- 172.21.2.117:0/3672718994 >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 0x7fe1840ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe189e70700 1 -- 172.21.2.117:0/3672718994 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840eee00 con 0x7fe1840d18f0 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18a671700 1 --2- 172.21.2.117:0/3672718994 >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 0x7fe1840ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.031 7fe18966f700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 0x7fe1840eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe189e70700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840d18f0 0x7fe1840eaa20 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 rx=0x7fe178006700 tx=0x7fe178009b80).ready entity=mon.0 client_cookie=8297dd0af604d7ae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe188e6e700 1 -- 172.21.2.117:0/3672718994 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17800c0c0 con 0x7fe1840d18f0 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe188e6e700 1 -- 172.21.2.117:0/3672718994 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17800c3c0 con 0x7fe1840d18f0 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe188e6e700 1 -- 172.21.2.117:0/3672718994 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17800c8e0 con 0x7fe1840d18f0 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 -- 172.21.2.117:0/3672718994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840d18f0 msgr2=0x7fe1840eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840d18f0 0x7fe1840eaa20 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 rx=0x7fe178006700 tx=0x7fe178009b80).stop 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 -- 172.21.2.117:0/3672718994 shutdown_connections 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 --2- 172.21.2.117:0/3672718994 >> v2:172.21.2.125:3300/0 conn(0x7fe1840ee3e0 0x7fe1840ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1840eb040 0x7fe1840eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 --2- 172.21.2.117:0/3672718994 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840d18f0 0x7fe1840eaa20 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.032 7fe18c054700 1 -- 172.21.2.117:0/3672718994 >> 172.21.2.117:0/3672718994 conn(0x7fe1840cf060 msgr2=0x7fe1840cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.033 7fe18c054700 1 -- 172.21.2.117:0/3672718994 shutdown_connections 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.033 7fe18c054700 1 -- 172.21.2.117:0/3672718994 wait complete. 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.047 7fe18c054700 1 Processor -- start 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.047 7fe18c054700 1 -- start start 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1840a6ce0 0x7fe1840dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe184085ec0 0x7fe1840862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840a67e0 con 0x7fe1840a2040 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840a7120 con 0x7fe1840a6ce0 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18c054700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1840a2a60 con 0x7fe184085ec0 2021-02-10T11:53:34.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57250/0 (socket says 172.21.2.117:57250) 2021-02-10T11:53:34.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 -- 172.21.2.117:0/3706394513 learned_addr learned my addr 172.21.2.117:0/3706394513 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 -- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe184085ec0 msgr2=0x7fe1840862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe184085ec0 0x7fe1840862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 -- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1840a6ce0 msgr2=0x7fe1840dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1840a6ce0 0x7fe1840dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 -- 172.21.2.117:0/3706394513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1841a2dc0 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.048 7fe18a671700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7fe17c0099a0 tx=0x7fe17c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17c0011f0 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17c003030 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe17c001570 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe18c054700 1 -- 172.21.2.117:0/3706394513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1841a2dc0 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe18c054700 1 -- 172.21.2.117:0/3706394513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1841a2dc0 con 0x7fe1840a2040 2021-02-10T11:53:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe17c006950 con 0x7fe1840a2040 2021-02-10T11:53:34.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 0x7fe16c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.049 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fe17c0326e0 con 0x7fe1840a2040 2021-02-10T11:53:34.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.050 7fe189e70700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 0x7fe16c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.050 7fe189e70700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 0x7fe16c01abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.081 7fe18c054700 1 -- 172.21.2.117:0/3706394513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1840daff0 con 0x7fe1840a2040 2021-02-10T11:53:34.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.084 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe17c00be30 con 0x7fe1840a2040 2021-02-10T11:53:34.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.210 7fe18c054700 1 -- 172.21.2.117:0/3706394513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe1840daff0 con 0x7fe1840a2040 2021-02-10T11:53:34.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.211 7fe1767fc700 1 -- 172.21.2.117:0/3706394513 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v147) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe17c006950 con 0x7fe1840a2040 2021-02-10T11:53:34.213 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:34.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 msgr2=0x7fe16c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 0x7fe16c01abc0 crc :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 msgr2=0x7fe1840a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 rx=0x7fe17c0099a0 tx=0x7fe17c00bcf0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 shutdown_connections 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe16c0186c0 0x7fe16c01abc0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1840a2040 0x7fe1840a2440 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe184085ec0 0x7fe1840862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 --2- 172.21.2.117:0/3706394513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1840a6ce0 0x7fe1840dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 >> 172.21.2.117:0/3706394513 conn(0x7fe1840b6540 msgr2=0x7fe18408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 shutdown_connections 2021-02-10T11:53:34.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.213 7fe18c054700 1 -- 172.21.2.117:0/3706394513 wait complete. 2021-02-10T11:53:34.223 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:53:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 Processor -- start 2021-02-10T11:53:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 -- start start 2021-02-10T11:53:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 0x7f796c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 0x7f796c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c0ede60 con 0x7f796c0eb000 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0ef7f0 con 0x7f796c0d18e0 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f7972c1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0d1ce0 con 0x7f796c0ef3f0 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f796bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f796c0eb000 0x7f796c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57508/0 (socket says 172.21.2.117:57508) 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.633 7f796bfff700 1 -- 172.21.2.117:0/1547947736 learned_addr learned my addr 172.21.2.117:0/1547947736 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f7970a3b700 1 --2- 172.21.2.117:0/1547947736 >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 0x7f796c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 --2- 172.21.2.117:0/1547947736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 0x7f796c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f796b7fe700 1 -- 172.21.2.117:0/1547947736 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687176086 0 0) 0x7f795c001060 con 0x7f796c0eb000 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f796b7fe700 1 -- 172.21.2.117:0/1547947736 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7958002fe0 con 0x7f796c0eb000 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 -- 172.21.2.117:0/1547947736 >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 msgr2=0x7f796c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 --2- 172.21.2.117:0/1547947736 >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 0x7f796c0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 -- 172.21.2.117:0/1547947736 >> v1:172.21.2.117:6789/0 conn(0x7f796c0eb000 legacy=0x7f796c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 -- 172.21.2.117:0/1547947736 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c1b8f90 con 0x7f796c0ef3f0 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f7970a3b700 1 --2- 172.21.2.117:0/1547947736 >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 0x7f796c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.634 7f797123c700 1 --2- 172.21.2.117:0/1547947736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 0x7f796c1b7960 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7964006ad0 tx=0x7f7964009ad0).ready entity=mon.2 client_cookie=adfa81517a98d5b5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f796b7fe700 1 -- 172.21.2.117:0/1547947736 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f79640022e0 con 0x7f796c0ef3f0 2021-02-10T11:53:34.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f796b7fe700 1 -- 172.21.2.117:0/1547947736 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964006410 con 0x7f796c0ef3f0 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f796b7fe700 1 -- 172.21.2.117:0/1547947736 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7964000c10 con 0x7f796c0ef3f0 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 -- 172.21.2.117:0/1547947736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 msgr2=0x7f796c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 --2- 172.21.2.117:0/1547947736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 0x7f796c1b7960 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f7964006ad0 tx=0x7f7964009ad0).stop 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 -- 172.21.2.117:0/1547947736 shutdown_connections 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 --2- 172.21.2.117:0/1547947736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0ef3f0 0x7f796c1b7960 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 --2- 172.21.2.117:0/1547947736 >> v2:172.21.2.125:3300/0 conn(0x7f796c0d18e0 0x7f796c0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 -- 172.21.2.117:0/1547947736 >> 172.21.2.117:0/1547947736 conn(0x7f796c0ced60 msgr2=0x7f796c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:34.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 -- 172.21.2.117:0/1547947736 shutdown_connections 2021-02-10T11:53:34.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.635 7f7972c1f700 1 -- 172.21.2.117:0/1547947736 wait complete. 2021-02-10T11:53:34.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 Processor -- start 2021-02-10T11:53:34.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 -- start start 2021-02-10T11:53:34.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 0x7f796c1a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f796c0ab280 0x7f796c0ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0a7d10 con 0x7f796c1a4290 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0a7a00 con 0x7f796c0ab280 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.650 7f7972c1f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c096370 con 0x7f796c0a6fe0 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f7970a3b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 0x7f796c1a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:54948/0 (socket says 172.21.2.117:54948) 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 -- 172.21.2.117:0/3199092846 learned_addr learned my addr 172.21.2.117:0/3199092846 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f796bfff700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f796c0ab280 0x7f796c0ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 -- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f796c0ab280 msgr2=0x7f796c0ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f796c0ab280 0x7f796c0ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 -- 172.21.2.117:0/3199092846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 msgr2=0x7f796c1a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 0x7f796c1a4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 -- 172.21.2.117:0/3199092846 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0a8290 con 0x7f796c0a6fe0 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f7970a3b700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 0x7f796c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f797123c700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f796400b610 tx=0x7f796400b9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7964009330 con 0x7f796c0a6fe0 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964000c80 con 0x7f796c0a6fe0 2021-02-10T11:53:34.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0a8290 con 0x7f796c0a6fe0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.651 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c0a8290 con 0x7f796c0a6fe0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.652 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7964002300 con 0x7f796c0a6fe0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.652 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f79640092e0 con 0x7f796c0a6fe0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.652 7f79697fa700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 0x7f795001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.652 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f7964032620 con 0x7f796c0a6fe0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.652 7f7970a3b700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 0x7f795001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:34.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.653 7f7970a3b700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 0x7f795001abc0 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:34.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.683 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c087290 con 0x7f796c0a6fe0 2021-02-10T11:53:34.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.687 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7964000a70 con 0x7f796c0a6fe0 2021-02-10T11:53:34.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.814 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f796c0ede60 con 0x7f796c0a6fe0 2021-02-10T11:53:34.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.815 7f79697fa700 1 -- 172.21.2.117:0/3199092846 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) v1 ==== 84+0+64 (secure 0 0 0) 0x7f796400bda0 con 0x7f796c0a6fe0 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:34.817 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.816 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 msgr2=0x7f795001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.816 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 0x7f795001abc0 crc :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.816 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 msgr2=0x7f796c0a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.816 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f796400b610 tx=0x7f796400b9b0).stop 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 shutdown_connections 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f796c0a6fe0 0x7f796c0a73e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f796c0ab280 0x7f796c0ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f79500186c0 0x7f795001abc0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 --2- 172.21.2.117:0/3199092846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f796c1a4290 0x7f796c1a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:34.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 >> 172.21.2.117:0/3199092846 conn(0x7f796c0874c0 msgr2=0x7f796c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:34.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 shutdown_connections 2021-02-10T11:53:34.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:34.817 7f7972c1f700 1 -- 172.21.2.117:0/3199092846 wait complete. 2021-02-10T11:53:34.827 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:53:34.828 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:53:35.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.243 7f5233144700 1 Processor -- start 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.243 7f5233144700 1 -- start start 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f5233144700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f522c0d18f0 0x7f522c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f5233144700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0ef7f0 con 0x7f522c0eb040 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f5233144700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0ede60 con 0x7f522c0ef3f0 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f5233144700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522c0d1da0 con 0x7f522c0d18f0 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f522c0eb040 0x7f522c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57526/0 (socket says 172.21.2.117:57526) 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522bfff700 1 -- 172.21.2.117:0/2248333958 learned_addr learned my addr 172.21.2.117:0/2248333958 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848353278 0 0) 0x7f521c0009b0 con 0x7f522c0eb040 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5218002fe0 con 0x7f522c0eb040 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> v2:172.21.2.125:3300/0 conn(0x7f522c0d18f0 0x7f522c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1282899656 0 0) 0x7f521c001010 con 0x7f522c0eb040 2021-02-10T11:53:35.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 >> v2:172.21.2.125:3300/0 conn(0x7f522c0d18f0 msgr2=0x7f522c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 --2- 172.21.2.117:0/2248333958 >> v2:172.21.2.125:3300/0 conn(0x7f522c0d18f0 0x7f522c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 >> v1:172.21.2.125:6789/0 conn(0x7f522c0ef3f0 legacy=0x7f522c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c1bb1c0 con 0x7f522c0eb040 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f521c001590 con 0x7f522c0eb040 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 >> v1:172.21.2.117:6789/0 conn(0x7f522c0eb040 legacy=0x7f522c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 0x7f5218005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.244 7f522b7fe700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5218006830 0x7f5218008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5218009630 0x7f521800bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521800c3a0 con 0x7f5218003b20 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521800c5c0 con 0x7f5218006830 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521800c7e0 con 0x7f5218009630 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> v2:172.21.2.125:3300/0 conn(0x7f522c0d18f0 0x7f522c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:35.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 0x7f5218005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522bfff700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5218006830 0x7f5218008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5231761700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5218009630 0x7f521800bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 -- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5218009630 msgr2=0x7f521800bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5218009630 0x7f521800bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 -- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5218006830 msgr2=0x7f5218008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5218006830 0x7f5218008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 -- 172.21.2.117:0/2248333958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521800ca80 con 0x7f5218003b20 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5231761700 1 -- 172.21.2.117:0/2248333958 reap_dead start 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522bfff700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5218006830 0x7f5218008d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5230f60700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 0x7f5218005f90 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f5214006cf0 tx=0x7f521404dd20).ready entity=mon.0 client_cookie=33eba273963ec871 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5214006910 con 0x7f5218003b20 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214000960 con 0x7f5218003b20 2021-02-10T11:53:35.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f522b7fe700 1 -- 172.21.2.117:0/2248333958 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52140020d0 con 0x7f5218003b20 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5233144700 1 -- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 msgr2=0x7f5218005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.245 7f5233144700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 0x7f5218005f90 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 rx=0x7f5214006cf0 tx=0x7f521404dd20).stop 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.246 7f5233144700 1 -- 172.21.2.117:0/2248333958 shutdown_connections 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.246 7f5233144700 1 --2- 172.21.2.117:0/2248333958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5218003b20 0x7f5218005f90 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.246 7f5233144700 1 -- 172.21.2.117:0/2248333958 >> 172.21.2.117:0/2248333958 conn(0x7f522c0cf060 msgr2=0x7f522c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.246 7f5233144700 1 -- 172.21.2.117:0/2248333958 shutdown_connections 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.246 7f5233144700 1 -- 172.21.2.117:0/2248333958 wait complete. 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 Processor -- start 2021-02-10T11:53:35.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 -- start start 2021-02-10T11:53:35.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f522c085ec0 0x7f522c0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f522c0a2040 0x7f522c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0a67e0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0a7120 con 0x7f522c0a2040 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.261 7f5233144700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0a2a60 con 0x7f522c085ec0 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57284/0 (socket says 172.21.2.117:57284) 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 -- 172.21.2.117:0/912607033 learned_addr learned my addr 172.21.2.117:0/912607033 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 -- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f522c085ec0 msgr2=0x7f522c0862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f522c085ec0 0x7f522c0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 -- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f522c0a2040 msgr2=0x7f522c0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f522c0a2040 0x7f522c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:35.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 -- 172.21.2.117:0/912607033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c1a2dc0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5230f60700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 rx=0x7f521404e660 tx=0x7f5214050940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5214050d30 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214006030 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52140022b0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5233144700 1 -- 172.21.2.117:0/912607033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1a2dc0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.262 7f5233144700 1 -- 172.21.2.117:0/912607033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c1a3350 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.263 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5214000c90 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.263 7f52297fa700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 0x7f521001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.263 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f52140763b0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.263 7f522bfff700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 0x7f521001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:35.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.264 7f522bfff700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 0x7f521001abc0 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:35.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.293 7f5233144700 1 -- 172.21.2.117:0/912607033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c0daff0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.296 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5214059ff0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.423 7f5233144700 1 -- 172.21.2.117:0/912607033 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f522c0ede60 con 0x7f522c0a6ce0 2021-02-10T11:53:35.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.825 7f52297fa700 1 -- 172.21.2.117:0/912607033 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v148) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5214059ff0 con 0x7f522c0a6ce0 2021-02-10T11:53:35.840 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:53:35.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.840 7f5233144700 1 -- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 msgr2=0x7f521001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.840 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 0x7f521001abc0 crc :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.840 7f5233144700 1 -- 172.21.2.117:0/912607033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 msgr2=0x7f522c0dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:35.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.840 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 rx=0x7f521404e660 tx=0x7f5214050940).stop 2021-02-10T11:53:35.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 -- 172.21.2.117:0/912607033 shutdown_connections 2021-02-10T11:53:35.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f522c0a2040 0x7f522c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:35.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f522c085ec0 0x7f522c0862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:35.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52100186c0 0x7f521001abc0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 --2- 172.21.2.117:0/912607033 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f522c0a6ce0 0x7f522c0dc420 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:35.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 -- 172.21.2.117:0/912607033 >> 172.21.2.117:0/912607033 conn(0x7f522c0b6540 msgr2=0x7f522c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:35.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 -- 172.21.2.117:0/912607033 shutdown_connections 2021-02-10T11:53:35.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:35.841 7f5233144700 1 -- 172.21.2.117:0/912607033 wait complete. 2021-02-10T11:53:35.851 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.264 7f6993e95700 1 Processor -- start 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.264 7f6993e95700 1 -- start start 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c0d18f0 0x7f698c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 0x7f698c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f698c0ee3e0 0x7f698c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c0ede60 con 0x7f698c0ee3e0 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c0eba60 con 0x7f698c0eb040 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6993e95700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c0d1cf0 con 0x7f698c0d18f0 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f698c0ee3e0 0x7f698c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f698c0ee3e0 0x7f698c0ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57292/0 (socket says 172.21.2.117:57292) 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f6991cb1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c0d18f0 0x7f698c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 -- 172.21.2.117:0/147664939 learned_addr learned my addr 172.21.2.117:0/147664939 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69914b0700 1 --2- 172.21.2.117:0/147664939 >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 0x7f698c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 -- 172.21.2.117:0/147664939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c0d18f0 msgr2=0x7f698c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 --2- 172.21.2.117:0/147664939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c0d18f0 0x7f698c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 -- 172.21.2.117:0/147664939 >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 msgr2=0x7f698c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.265 7f69924b2700 1 --2- 172.21.2.117:0/147664939 >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 0x7f698c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f69924b2700 1 -- 172.21.2.117:0/147664939 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c0eee00 con 0x7f698c0ee3e0 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f69914b0700 1 --2- 172.21.2.117:0/147664939 >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 0x7f698c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f69924b2700 1 --2- 172.21.2.117:0/147664939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0ee3e0 0x7f698c0ee7e0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f6984006f00 tx=0x7f6984009f10).ready entity=mon.0 client_cookie=85637ef878c03ef server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:36.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f6990caf700 1 -- 172.21.2.117:0/147664939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6984003030 con 0x7f698c0ee3e0 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f6990caf700 1 -- 172.21.2.117:0/147664939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698400b3b0 con 0x7f698c0ee3e0 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f6990caf700 1 -- 172.21.2.117:0/147664939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698400b8d0 con 0x7f698c0ee3e0 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f6993e95700 1 -- 172.21.2.117:0/147664939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0ee3e0 msgr2=0x7f698c0ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.266 7f6993e95700 1 --2- 172.21.2.117:0/147664939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0ee3e0 0x7f698c0ee7e0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 rx=0x7f6984006f00 tx=0x7f6984009f10).stop 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 -- 172.21.2.117:0/147664939 shutdown_connections 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 --2- 172.21.2.117:0/147664939 >> v2:172.21.2.125:3300/0 conn(0x7f698c0eb040 0x7f698c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 --2- 172.21.2.117:0/147664939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0ee3e0 0x7f698c0ee7e0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 --2- 172.21.2.117:0/147664939 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c0d18f0 0x7f698c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 -- 172.21.2.117:0/147664939 >> 172.21.2.117:0/147664939 conn(0x7f698c0cf060 msgr2=0x7f698c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 -- 172.21.2.117:0/147664939 shutdown_connections 2021-02-10T11:53:36.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.267 7f6993e95700 1 -- 172.21.2.117:0/147664939 wait complete. 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.281 7f6993e95700 1 Processor -- start 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 -- start start 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c1a48b0 0x7f698c08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 0x7f698c0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c0aae30 con 0x7f698c0a9f90 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c0aa9f0 con 0x7f698c08cf40 2021-02-10T11:53:36.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6993e95700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c0ab330 con 0x7f698c1a48b0 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f69924b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f69924b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57296/0 (socket says 172.21.2.117:57296) 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f69924b2700 1 -- 172.21.2.117:0/497207786 learned_addr learned my addr 172.21.2.117:0/497207786 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.282 7f6991cb1700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c1a48b0 0x7f698c08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69914b0700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 0x7f698c0a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 -- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c1a48b0 msgr2=0x7f698c08c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c1a48b0 0x7f698c08c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 -- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 msgr2=0x7f698c0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 0x7f698c0a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 -- 172.21.2.117:0/497207786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c0ab550 con 0x7f698c0a9f90 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69914b0700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 0x7f698c0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f69924b2700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f698400b920 tx=0x7f698400bcc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f698400be00 con 0x7f698c0a9f90 2021-02-10T11:53:36.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6984000cc0 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6984005aa0 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f6993e95700 1 -- 172.21.2.117:0/497207786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c0ab550 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.283 7f6993e95700 1 -- 172.21.2.117:0/497207786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c0ab550 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.284 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6984005820 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.284 7f697e7fc700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 0x7f697401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.284 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f6984032860 con 0x7f698c0a9f90 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.284 7f6991cb1700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 0x7f697401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:36.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.285 7f6991cb1700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 0x7f697401abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:36.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.315 7f6993e95700 1 -- 172.21.2.117:0/497207786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c0ede60 con 0x7f698c0a9f90 2021-02-10T11:53:36.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.318 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f698400be00 con 0x7f698c0a9f90 2021-02-10T11:53:36.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.444 7f6993e95700 1 -- 172.21.2.117:0/497207786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f698c0ede60 con 0x7f698c0a9f90 2021-02-10T11:53:36.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.833 7f697e7fc700 1 -- 172.21.2.117:0/497207786 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v150) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6984005820 con 0x7f698c0a9f90 2021-02-10T11:53:36.862 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:53:36.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 msgr2=0x7f697401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 0x7f697401abc0 crc :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 msgr2=0x7f698c0aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f698400b920 tx=0x7f698400bcc0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 shutdown_connections 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69740186c0 0x7f697401abc0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f698c0a9f90 0x7f698c0aa390 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f698c08cf40 0x7f698c0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 --2- 172.21.2.117:0/497207786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f698c1a48b0 0x7f698c08c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 >> 172.21.2.117:0/497207786 conn(0x7f698c092d00 msgr2=0x7f698c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 shutdown_connections 2021-02-10T11:53:36.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:36.863 7f6993e95700 1 -- 172.21.2.117:0/497207786 wait complete. 2021-02-10T11:53:36.873 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:53:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 Processor -- start 2021-02-10T11:53:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 -- start start 2021-02-10T11:53:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fab300e5240 0x7fab30086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab30084c60 0x7fab30085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab30085930 con 0x7fab300e4e40 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab30085680 con 0x7fab300e5240 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.287 7fab36559700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab301b09e0 con 0x7fab30084c60 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fab300e5240 0x7fab30086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fab300e4e40 0x7fab301b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57556/0 (socket says 172.21.2.117:57556) 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2ffff700 1 -- 172.21.2.117:0/3543041185 learned_addr learned my addr 172.21.2.117:0/3543041185 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 -- 172.21.2.117:0/3543041185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab30084c60 msgr2=0x7fab30085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 --2- 172.21.2.117:0/3543041185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab30084c60 0x7fab30085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 -- 172.21.2.117:0/3543041185 >> v1:172.21.2.117:6789/0 conn(0x7fab300e4e40 legacy=0x7fab301b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 -- 172.21.2.117:0/3543041185 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab30084840 con 0x7fab300e5240 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2f7fe700 1 --2- 172.21.2.117:0/3543041185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab300e5240 0x7fab30086ec0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7fab24006650 tx=0x7fab24009a00).ready entity=mon.0 client_cookie=a0476c89e1326dd7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2effd700 1 -- 172.21.2.117:0/3543041185 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab24009df0 con 0x7fab300e5240 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2effd700 1 -- 172.21.2.117:0/3543041185 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab240009b0 con 0x7fab300e5240 2021-02-10T11:53:37.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab2effd700 1 -- 172.21.2.117:0/3543041185 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab2400c2b0 con 0x7fab300e5240 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab36559700 1 -- 172.21.2.117:0/3543041185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab300e5240 msgr2=0x7fab30086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.288 7fab36559700 1 --2- 172.21.2.117:0/3543041185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab300e5240 0x7fab30086ec0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 rx=0x7fab24006650 tx=0x7fab24009a00).stop 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 -- 172.21.2.117:0/3543041185 shutdown_connections 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 --2- 172.21.2.117:0/3543041185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab30084c60 0x7fab30085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 --2- 172.21.2.117:0/3543041185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab300e5240 0x7fab30086ec0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 -- 172.21.2.117:0/3543041185 >> 172.21.2.117:0/3543041185 conn(0x7fab300ce4e0 msgr2=0x7fab300e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 -- 172.21.2.117:0/3543041185 shutdown_connections 2021-02-10T11:53:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.289 7fab36559700 1 -- 172.21.2.117:0/3543041185 wait complete. 2021-02-10T11:53:37.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 Processor -- start 2021-02-10T11:53:37.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 -- start start 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab3018e5b0 0x7fab3018e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab300b30d0 0x7fab300b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab301ae8d0 con 0x7fab30099bf0 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300b3af0 con 0x7fab300b30d0 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.304 7fab36559700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab300b3d10 con 0x7fab3018e5b0 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57312/0 (socket says 172.21.2.117:57312) 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 -- 172.21.2.117:0/1879867157 learned_addr learned my addr 172.21.2.117:0/1879867157 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 -- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab3018e5b0 msgr2=0x7fab3018e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab3018e5b0 0x7fab3018e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 -- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab300b30d0 msgr2=0x7fab300b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab300b30d0 0x7fab300b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 -- 172.21.2.117:0/1879867157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300b3f30 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2ffff700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7fab2000a4c0 tx=0x7fab2000a860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab2000ac50 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab20004140 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab36559700 1 -- 172.21.2.117:0/1879867157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab300b3f30 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fab20004620 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.305 7fab36559700 1 -- 172.21.2.117:0/1879867157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab301a19b0 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.306 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fab20004320 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.306 7fab2cff9700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 0x7fab1001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.306 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fab200324f0 con 0x7fab30099bf0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.307 7fab2f7fe700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 0x7fab1001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.307 7fab2f7fe700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 0x7fab1001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.337 7fab36559700 1 -- 172.21.2.117:0/1879867157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3018e080 con 0x7fab30099bf0 2021-02-10T11:53:37.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.340 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fab20000940 con 0x7fab30099bf0 2021-02-10T11:53:37.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.468 7fab36559700 1 -- 172.21.2.117:0/1879867157 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fab3018e080 con 0x7fab30099bf0 2021-02-10T11:53:37.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.472 7fab2cff9700 1 -- 172.21.2.117:0/1879867157 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 32 and data pool 33 v126) v1 ==== 189+0+0 (secure 0 0 0) 0x7fab20004650 con 0x7fab30099bf0 2021-02-10T11:53:37.476 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 32 and data pool 33 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 msgr2=0x7fab1001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 0x7fab1001abc0 crc :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 msgr2=0x7fab301ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 rx=0x7fab2000a4c0 tx=0x7fab2000a860).stop 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 shutdown_connections 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fab300b30d0 0x7fab300b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fab3018e5b0 0x7fab3018e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fab100186c0 0x7fab1001abc0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 --2- 172.21.2.117:0/1879867157 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fab30099bf0 0x7fab301ae2b0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 >> 172.21.2.117:0/1879867157 conn(0x7fab301ad6b0 msgr2=0x7fab30190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:37.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 shutdown_connections 2021-02-10T11:53:37.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.478 7fab36559700 1 -- 172.21.2.117:0/1879867157 wait complete. 2021-02-10T11:53:37.488 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 Processor -- start 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 -- start start 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f56d40ef3f0 0x7f56d41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d40ede60 con 0x7f56d40eb000 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d4088020 con 0x7f56d40ef3f0 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.897 7f56d9817700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40d1ce0 con 0x7f56d40d18e0 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f56d40ef3f0 0x7f56d41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3574700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f56d40ef3f0 0x7f56d41b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:33472/0 (socket says 172.21.2.117:33472) 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3574700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:55000/0 (socket says 172.21.2.117:55000) 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 -- 172.21.2.117:0/3722800376 learned_addr learned my addr 172.21.2.117:0/3722800376 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d2d73700 1 --1- 172.21.2.117:0/3722800376 >> v1:172.21.2.125:6789/0 conn(0x7f56d40eb000 0x7f56d40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50496/0 (socket says 172.21.2.117:50496) 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 -- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 msgr2=0x7f56d40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 -- 172.21.2.117:0/3722800376 >> v1:172.21.2.125:6789/0 conn(0x7f56d40eb000 legacy=0x7f56d40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3d75700 1 -- 172.21.2.117:0/3722800376 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d41b8f90 con 0x7f56d40ef3f0 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.898 7f56d3574700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d3d75700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d40ef3f0 0x7f56d41b7960 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f56cc006aa0 tx=0x7f56cc009f20).ready entity=mon.1 client_cookie=a4f7d1b4fff4f3ba server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d2572700 1 -- 172.21.2.117:0/3722800376 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56cc00b5c0 con 0x7f56d40ef3f0 2021-02-10T11:53:37.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d2572700 1 -- 172.21.2.117:0/3722800376 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc003030 con 0x7f56d40ef3f0 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d2572700 1 -- 172.21.2.117:0/3722800376 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56cc00b9f0 con 0x7f56d40ef3f0 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d9817700 1 -- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d40ef3f0 msgr2=0x7f56d41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d9817700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d40ef3f0 0x7f56d41b7960 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f56cc006aa0 tx=0x7f56cc009f20).stop 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d9817700 1 -- 172.21.2.117:0/3722800376 shutdown_connections 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.899 7f56d9817700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d40ef3f0 0x7f56d41b7960 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.900 7f56d9817700 1 --2- 172.21.2.117:0/3722800376 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40d18e0 0x7f56d40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.900 7f56d9817700 1 -- 172.21.2.117:0/3722800376 >> 172.21.2.117:0/3722800376 conn(0x7f56d40ced60 msgr2=0x7f56d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.900 7f56d9817700 1 -- 172.21.2.117:0/3722800376 shutdown_connections 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.900 7f56d9817700 1 -- 172.21.2.117:0/3722800376 wait complete. 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.914 7f56d9817700 1 Processor -- start 2021-02-10T11:53:37.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.914 7f56d9817700 1 -- start start 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40bd520 0x7f56d41a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56d41a37d0 0x7f56d40be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40bd9f0 con 0x7f56d41a37d0 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40be9f0 con 0x7f56d41a33d0 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d9817700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40bec10 con 0x7f56d40bd520 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:33476/0 (socket says 172.21.2.117:33476) 2021-02-10T11:53:37.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 -- 172.21.2.117:0/3238420329 learned_addr learned my addr 172.21.2.117:0/3238420329 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d3574700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40bd520 0x7f56d41a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d3d75700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56d41a37d0 0x7f56d40be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 -- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40bd520 msgr2=0x7f56d41a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40bd520 0x7f56d41a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 -- 172.21.2.117:0/3238420329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56d41a37d0 msgr2=0x7f56d40be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56d41a37d0 0x7f56d40be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.915 7f56d2d73700 1 -- 172.21.2.117:0/3238420329 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40e8430 con 0x7f56d41a33d0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56d2d73700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f56c4007650 tx=0x7f56c400a880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c4000960 con 0x7f56d41a33d0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c400ac70 con 0x7f56d41a33d0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f56c40045d0 con 0x7f56d41a33d0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56d9817700 1 -- 172.21.2.117:0/3238420329 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d40e8430 con 0x7f56d41a33d0 2021-02-10T11:53:37.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.916 7f56d9817700 1 -- 172.21.2.117:0/3238420329 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d40e89c0 con 0x7f56d41a33d0 2021-02-10T11:53:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.917 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f56c4004140 con 0x7f56d41a33d0 2021-02-10T11:53:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.917 7f56cbfff700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 0x7f56b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.917 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f56c40326b0 con 0x7f56d41a33d0 2021-02-10T11:53:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.917 7f56d3574700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 0x7f56b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.918 7f56d3574700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 0x7f56b801abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:37.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.947 7f56d9817700 1 -- 172.21.2.117:0/3238420329 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d4087290 con 0x7f56d41a33d0 2021-02-10T11:53:37.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:37.952 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f56c4000e20 con 0x7f56d41a33d0 2021-02-10T11:53:38.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.079 7f56d9817700 1 -- 172.21.2.117:0/3238420329 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56d40ede60 con 0x7f56d41a33d0 2021-02-10T11:53:38.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.080 7f56cbfff700 1 -- 172.21.2.117:0/3238420329 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f56c4004140 con 0x7f56d41a33d0 2021-02-10T11:53:38.084 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:38.084 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":151,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:37.468828","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:53:35.441170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:53:36.462661","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:38.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 msgr2=0x7f56b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 0x7f56b801abc0 crc :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 msgr2=0x7f56d40bddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f56c4007650 tx=0x7f56c400a880).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 shutdown_connections 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f56b80186c0 0x7f56b801abc0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f56d41a37d0 0x7f56d40be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f56d41a33d0 0x7f56d40bddb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 --2- 172.21.2.117:0/3238420329 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f56d40bd520 0x7f56d41a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 >> 172.21.2.117:0/3238420329 conn(0x7f56d40874c0 msgr2=0x7f56d40dfdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 shutdown_connections 2021-02-10T11:53:38.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.084 7f56d9817700 1 -- 172.21.2.117:0/3238420329 wait complete. 2021-02-10T11:53:38.094 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 Processor -- start 2021-02-10T11:53:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 -- start start 2021-02-10T11:53:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 0x7f24f40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24f40eb7f0 0x7f24f41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40ede60 con 0x7f24f40d18e0 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f40ef7f0 con 0x7f24f40eb7f0 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.507 7f24fc6f0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40d1ce0 con 0x7f24f40ef3f0 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24f9d0b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 0x7f24f40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24f40eb7f0 0x7f24f41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24f40eb7f0 0x7f24f41b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:33488/0 (socket says 172.21.2.117:33488) 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24f9d0b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 0x7f24f40eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:55012/0 (socket says 172.21.2.117:55012) 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 -- 172.21.2.117:0/2804002137 learned_addr learned my addr 172.21.2.117:0/2804002137 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 -- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 msgr2=0x7f24f40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 --2- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 0x7f24f40eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 -- 172.21.2.117:0/2804002137 >> v1:172.21.2.125:6789/0 conn(0x7f24f40d18e0 legacy=0x7f24f40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.508 7f24fad0d700 1 -- 172.21.2.117:0/2804002137 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f41b8f90 con 0x7f24f40eb7f0 2021-02-10T11:53:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24fad0d700 1 --2- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f40eb7f0 0x7f24f41b7960 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f24ec006720 tx=0x7f24ec009ba0).ready entity=mon.1 client_cookie=811a255fb2b0fc13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24f950a700 1 -- 172.21.2.117:0/2804002137 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ec0022e0 con 0x7f24f40eb7f0 2021-02-10T11:53:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24f950a700 1 -- 172.21.2.117:0/2804002137 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec002560 con 0x7f24f40eb7f0 2021-02-10T11:53:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24f950a700 1 -- 172.21.2.117:0/2804002137 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24ec002a80 con 0x7f24f40eb7f0 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24fc6f0700 1 -- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f40eb7f0 msgr2=0x7f24f41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.509 7f24fc6f0700 1 --2- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f40eb7f0 0x7f24f41b7960 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f24ec006720 tx=0x7f24ec009ba0).stop 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 -- 172.21.2.117:0/2804002137 shutdown_connections 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 --2- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f40eb7f0 0x7f24f41b7960 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 --2- 172.21.2.117:0/2804002137 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f40ef3f0 0x7f24f40eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 -- 172.21.2.117:0/2804002137 >> 172.21.2.117:0/2804002137 conn(0x7f24f40ced60 msgr2=0x7f24f40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 -- 172.21.2.117:0/2804002137 shutdown_connections 2021-02-10T11:53:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.510 7f24fc6f0700 1 -- 172.21.2.117:0/2804002137 wait complete. 2021-02-10T11:53:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.524 7f24fc6f0700 1 Processor -- start 2021-02-10T11:53:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 -- start start 2021-02-10T11:53:38.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 0x7f24f40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f41b0130 0x7f24f40e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40e80e0 con 0x7f24f40b43f0 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40e7e20 con 0x7f24f41afd30 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24fc6f0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4096a70 con 0x7f24f41b0130 2021-02-10T11:53:38.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24f9d0b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24f9d0b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57340/0 (socket says 172.21.2.117:57340) 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.525 7f24f9d0b700 1 -- 172.21.2.117:0/3500000562 learned_addr learned my addr 172.21.2.117:0/3500000562 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24fad0d700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f41b0130 0x7f24f40e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24fa50c700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 0x7f24f40b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 -- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f41b0130 msgr2=0x7f24f40e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f41b0130 0x7f24f40e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 -- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 msgr2=0x7f24f40b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 0x7f24f40b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 -- 172.21.2.117:0/3500000562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f40e8660 con 0x7f24f40b43f0 2021-02-10T11:53:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24fa50c700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 0x7f24f40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:38.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24f9d0b700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f24f0006f60 tx=0x7f24f000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f000bd60 con 0x7f24f40b43f0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0000c90 con 0x7f24f40b43f0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f24f0002220 con 0x7f24f40b43f0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f40e8660 con 0x7f24f40b43f0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.526 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f40e8660 con 0x7f24f40b43f0 2021-02-10T11:53:38.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.527 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f24f00052e0 con 0x7f24f40b43f0 2021-02-10T11:53:38.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.527 7f24e6ffd700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 0x7f24dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:38.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.527 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f24f00326b0 con 0x7f24f40b43f0 2021-02-10T11:53:38.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.528 7f24fa50c700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 0x7f24dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:38.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.528 7f24fa50c700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 0x7f24dc01abc0 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:38.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.557 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f40e7e20 con 0x7f24f40b43f0 2021-02-10T11:53:38.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.560 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f24f0006b00 con 0x7f24f40b43f0 2021-02-10T11:53:38.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.687 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24f40e7e20 con 0x7f24f40b43f0 2021-02-10T11:53:38.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.688 7f24e6ffd700 1 -- 172.21.2.117:0/3500000562 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f24f00052e0 con 0x7f24f40b43f0 2021-02-10T11:53:38.690 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:38.690 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":152,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:38.470991","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:53:35.441170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:53:36.462661","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:38.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.689 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 msgr2=0x7f24dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.689 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 0x7f24dc01abc0 crc :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.689 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 msgr2=0x7f24f40e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.689 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 rx=0x7f24f0006f60 tx=0x7f24f000b970).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 shutdown_connections 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24dc0186c0 0x7f24dc01abc0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24f40b43f0 0x7f24f40e71e0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24f41b0130 0x7f24f40e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 --2- 172.21.2.117:0/3500000562 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24f41afd30 0x7f24f40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 >> 172.21.2.117:0/3500000562 conn(0x7f24f41a3620 msgr2=0x7f24f40dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 shutdown_connections 2021-02-10T11:53:38.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:38.690 7f24fc6f0700 1 -- 172.21.2.117:0/3500000562 wait complete. 2021-02-10T11:53:38.700 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:53:39.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 Processor -- start 2021-02-10T11:53:39.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 -- start start 2021-02-10T11:53:39.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d500d18f0 0x7f5d500eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500eb040 0x7f5d500eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d500eba60 con 0x7f5d500ee3e0 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d500d1cf0 con 0x7f5d500d18f0 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.115 7f5d5696c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d500eefd0 con 0x7f5d500eb040 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4f7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500eb040 0x7f5d500eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5d500d18f0 0x7f5d500eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d54f89700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f5d500ee3e0 0x7f5d500ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50522/0 (socket says 172.21.2.117:50522) 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d54f89700 1 -- 172.21.2.117:0/2348141582 learned_addr learned my addr 172.21.2.117:0/2348141582 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 -- 172.21.2.117:0/2348141582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500eb040 msgr2=0x7f5d500eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 --2- 172.21.2.117:0/2348141582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500eb040 0x7f5d500eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 -- 172.21.2.117:0/2348141582 >> v1:172.21.2.125:6789/0 conn(0x7f5d500ee3e0 legacy=0x7f5d500ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 -- 172.21.2.117:0/2348141582 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d500ef1f0 con 0x7f5d500d18f0 2021-02-10T11:53:39.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.116 7f5d4ffff700 1 --2- 172.21.2.117:0/2348141582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d500d18f0 0x7f5d500eaa20 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7f5d480069d0 tx=0x7f5d48009e40).ready entity=mon.0 client_cookie=c2bef064803842cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d4effd700 1 -- 172.21.2.117:0/2348141582 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d4800b510 con 0x7f5d500d18f0 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d4effd700 1 -- 172.21.2.117:0/2348141582 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4800b790 con 0x7f5d500d18f0 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d4effd700 1 -- 172.21.2.117:0/2348141582 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d4800bcb0 con 0x7f5d500d18f0 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 -- 172.21.2.117:0/2348141582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d500d18f0 msgr2=0x7f5d500eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 --2- 172.21.2.117:0/2348141582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d500d18f0 0x7f5d500eaa20 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 rx=0x7f5d480069d0 tx=0x7f5d48009e40).stop 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 -- 172.21.2.117:0/2348141582 shutdown_connections 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 --2- 172.21.2.117:0/2348141582 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500eb040 0x7f5d500eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 --2- 172.21.2.117:0/2348141582 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d500d18f0 0x7f5d500eaa20 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 -- 172.21.2.117:0/2348141582 >> 172.21.2.117:0/2348141582 conn(0x7f5d500cf060 msgr2=0x7f5d500cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 -- 172.21.2.117:0/2348141582 shutdown_connections 2021-02-10T11:53:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.117 7f5d5696c700 1 -- 172.21.2.117:0/2348141582 wait complete. 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 Processor -- start 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 -- start start 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d5008cf40 0x7f5d500a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500a9f90 0x7f5d500aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d500aae30 con 0x7f5d501a48b0 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d500aa9f0 con 0x7f5d5008cf40 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.132 7f5d5696c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d500ab330 con 0x7f5d500a9f90 2021-02-10T11:53:39.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57350/0 (socket says 172.21.2.117:57350) 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4f7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d5008cf40 0x7f5d500a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 -- 172.21.2.117:0/724620644 learned_addr learned my addr 172.21.2.117:0/724620644 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d54f89700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500a9f90 0x7f5d500aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 -- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500a9f90 msgr2=0x7f5d500aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500a9f90 0x7f5d500aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 -- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d5008cf40 msgr2=0x7f5d500a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d5008cf40 0x7f5d500a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 -- 172.21.2.117:0/724620644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d500ab550 con 0x7f5d501a48b0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4ffff700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f5d48009f40 tx=0x7f5d4800bd50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d48006240 con 0x7f5d501a48b0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d480065c0 con 0x7f5d501a48b0 2021-02-10T11:53:39.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5d480058e0 con 0x7f5d501a48b0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d5696c700 1 -- 172.21.2.117:0/724620644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d500ab550 con 0x7f5d501a48b0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.133 7f5d5696c700 1 -- 172.21.2.117:0/724620644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d500abae0 con 0x7f5d501a48b0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.134 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5d48005660 con 0x7f5d501a48b0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.134 7f5d4cff9700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 0x7f5d3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.134 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5d48032850 con 0x7f5d501a48b0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.134 7f5d4f7fe700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 0x7f5d3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.135 7f5d4f7fe700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 0x7f5d3001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.165 7f5d5696c700 1 -- 172.21.2.117:0/724620644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d500ede60 con 0x7f5d501a48b0 2021-02-10T11:53:39.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.168 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5d4802e400 con 0x7f5d501a48b0 2021-02-10T11:53:39.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.303 7f5d5696c700 1 -- 172.21.2.117:0/724620644 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f5d500ab2f0 con 0x7f5d501a48b0 2021-02-10T11:53:39.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.478 7f5d4cff9700 1 -- 172.21.2.117:0/724620644 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v128) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5d4802eee0 con 0x7f5d501a48b0 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 msgr2=0x7f5d3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 0x7f5d3001abc0 crc :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 msgr2=0x7f5d5008c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 rx=0x7f5d48009f40 tx=0x7f5d4800bd50).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 shutdown_connections 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5d500a9f90 0x7f5d500aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5d5008cf40 0x7f5d500a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5d300186c0 0x7f5d3001abc0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 --2- 172.21.2.117:0/724620644 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5d501a48b0 0x7f5d5008c920 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 >> 172.21.2.117:0/724620644 conn(0x7f5d50092d00 msgr2=0x7f5d5008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:39.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 shutdown_connections 2021-02-10T11:53:39.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.520 7f5d5696c700 1 -- 172.21.2.117:0/724620644 wait complete. 2021-02-10T11:53:39.530 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 Processor -- start 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 -- start start 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc0d18e0 0x7f8ebc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ebc0eb000 0x7f8ebc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc0ede60 con 0x7f8ebc0ee3e0 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0eba20 con 0x7f8ebc0eb000 2021-02-10T11:53:39.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.940 7f8ec1166700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0d1ce0 con 0x7f8ebc0d18e0 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ebc0eb000 0x7f8ebc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ebc0eb000 0x7f8ebc0eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57360/0 (socket says 172.21.2.117:57360) 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 -- 172.21.2.117:0/2277175374 learned_addr learned my addr 172.21.2.117:0/2277175374 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 -- 172.21.2.117:0/2277175374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc0d18e0 msgr2=0x7f8ebc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 --2- 172.21.2.117:0/2277175374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc0d18e0 0x7f8ebc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 -- 172.21.2.117:0/2277175374 >> v1:172.21.2.125:6789/0 conn(0x7f8ebc0ee3e0 legacy=0x7f8ebc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 -- 172.21.2.117:0/2277175374 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0eefd0 con 0x7f8ebc0eb000 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eba59c700 1 --2- 172.21.2.117:0/2277175374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0eb000 0x7f8ebc0eb400 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f8eac006700 tx=0x7f8eac009b80).ready entity=mon.0 client_cookie=5d177d76edff8f7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eb9d9b700 1 -- 172.21.2.117:0/2277175374 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eac00c0c0 con 0x7f8ebc0eb000 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eb9d9b700 1 -- 172.21.2.117:0/2277175374 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac00c3c0 con 0x7f8ebc0eb000 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8eb9d9b700 1 -- 172.21.2.117:0/2277175374 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eac00c8e0 con 0x7f8ebc0eb000 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8ec1166700 1 -- 172.21.2.117:0/2277175374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0eb000 msgr2=0x7f8ebc0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.941 7f8ec1166700 1 --2- 172.21.2.117:0/2277175374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0eb000 0x7f8ebc0eb400 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 rx=0x7f8eac006700 tx=0x7f8eac009b80).stop 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 -- 172.21.2.117:0/2277175374 shutdown_connections 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 --2- 172.21.2.117:0/2277175374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0eb000 0x7f8ebc0eb400 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 --2- 172.21.2.117:0/2277175374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc0d18e0 0x7f8ebc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 -- 172.21.2.117:0/2277175374 >> 172.21.2.117:0/2277175374 conn(0x7f8ebc0ced60 msgr2=0x7f8ebc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 -- 172.21.2.117:0/2277175374 shutdown_connections 2021-02-10T11:53:39.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.942 7f8ec1166700 1 -- 172.21.2.117:0/2277175374 wait complete. 2021-02-10T11:53:39.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 Processor -- start 2021-02-10T11:53:39.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 -- start start 2021-02-10T11:53:39.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ebc0a0bd0 0x7f8ebc08f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc08cb70 0x7f8ebc08cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc08fab0 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc08d590 con 0x7f8ebc0a0bd0 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ec1166700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc08c290 con 0x7f8ebc08cb70 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ebb59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ebb59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57368/0 (socket says 172.21.2.117:57368) 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.957 7f8ebb59e700 1 -- 172.21.2.117:0/4106847331 learned_addr learned my addr 172.21.2.117:0/4106847331 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 -- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc08cb70 msgr2=0x7f8ebc08cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc08cb70 0x7f8ebc08cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 -- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ebc0a0bd0 msgr2=0x7f8ebc08f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:39.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ebc0a0bd0 0x7f8ebc08f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 -- 172.21.2.117:0/4106847331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0a1170 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ebb59e700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 rx=0x7f8eb4009940 tx=0x7f8eb400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb4000f90 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400c2f0 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8eb400c810 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc0a1170 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.958 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc0e0870 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.959 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8eb4001220 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.959 7f8eb37fe700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 0x7f8ea001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.959 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8eb4032a50 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.959 7f8ebad9d700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 0x7f8ea001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:39.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.960 7f8ebad9d700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 0x7f8ea001abc0 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:39.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.990 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc08c640 con 0x7f8ebc0dfb60 2021-02-10T11:53:39.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:39.993 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8eb4000f40 con 0x7f8ebc0dfb60 2021-02-10T11:53:40.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.121 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ebc08c640 con 0x7f8ebc0dfb60 2021-02-10T11:53:40.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.122 7f8eb37fe700 1 -- 172.21.2.117:0/4106847331 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f8eb401f6a0 con 0x7f8ebc0dfb60 2021-02-10T11:53:40.124 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:40.124 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":128,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:53:39.473419","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8737},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8737":{"gid":8737,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/277081047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":277081047},{"type":"v1","addr":"172.21.2.125:6835","nonce":277081047}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:53:40.124 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 128 2021-02-10T11:53:40.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.123 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 msgr2=0x7f8ea001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.123 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 0x7f8ea001abc0 crc :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.123 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 msgr2=0x7f8ebc0dff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.123 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 rx=0x7f8eb4009940 tx=0x7f8eb400bcf0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 shutdown_connections 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ea00186c0 0x7f8ea001abc0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ebc0dfb60 0x7f8ebc0dff60 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ebc08cb70 0x7f8ebc08cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 --2- 172.21.2.117:0/4106847331 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ebc0a0bd0 0x7f8ebc08f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 >> 172.21.2.117:0/4106847331 conn(0x7f8ebc1a3620 msgr2=0x7f8ebc0e0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:40.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 shutdown_connections 2021-02-10T11:53:40.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.124 7f8ec1166700 1 -- 172.21.2.117:0/4106847331 wait complete. 2021-02-10T11:53:40.134 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.545 7f8ff9ca8700 1 Processor -- start 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.545 7f8ff9ca8700 1 -- start start 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff9ca8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8ff40e5240 0x7f8ff4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff9ca8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ff4084c60 0x7f8ff4085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff9ca8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff4085930 con 0x7f8ff40e4e40 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff9ca8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40e5cc0 con 0x7f8ff4084c60 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff9ca8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40856b0 con 0x7f8ff40e5240 2021-02-10T11:53:40.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ff4084c60 0x7f8ff4085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8ff4084c60 0x7f8ff4085060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57376/0 (socket says 172.21.2.117:57376) 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 -- 172.21.2.117:0/2180560795 learned_addr learned my addr 172.21.2.117:0/2180560795 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 -- 172.21.2.117:0/2180560795 >> v2:172.21.2.125:3300/0 conn(0x7f8ff40e5240 msgr2=0x7f8ff4086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 --2- 172.21.2.117:0/2180560795 >> v2:172.21.2.125:3300/0 conn(0x7f8ff40e5240 0x7f8ff4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 -- 172.21.2.117:0/2180560795 >> v1:172.21.2.125:6789/0 conn(0x7f8ff40e4e40 legacy=0x7f8ff41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 -- 172.21.2.117:0/2180560795 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff41b09e0 con 0x7f8ff4084c60 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.546 7f8ff3fff700 1 --2- 172.21.2.117:0/2180560795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff4084c60 0x7f8ff4085060 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f8fec006f00 tx=0x7f8fec009f10).ready entity=mon.0 client_cookie=31c334f6c6886abe server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff27fc700 1 -- 172.21.2.117:0/2180560795 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fec00b3b0 con 0x7f8ff4084c60 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff27fc700 1 -- 172.21.2.117:0/2180560795 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00b630 con 0x7f8ff4084c60 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff27fc700 1 -- 172.21.2.117:0/2180560795 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fec00bb50 con 0x7f8ff4084c60 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 -- 172.21.2.117:0/2180560795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff4084c60 msgr2=0x7f8ff4085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 --2- 172.21.2.117:0/2180560795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff4084c60 0x7f8ff4085060 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 rx=0x7f8fec006f00 tx=0x7f8fec009f10).stop 2021-02-10T11:53:40.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 -- 172.21.2.117:0/2180560795 shutdown_connections 2021-02-10T11:53:40.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 --2- 172.21.2.117:0/2180560795 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff4084c60 0x7f8ff4085060 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 --2- 172.21.2.117:0/2180560795 >> v2:172.21.2.125:3300/0 conn(0x7f8ff40e5240 0x7f8ff4086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 -- 172.21.2.117:0/2180560795 >> 172.21.2.117:0/2180560795 conn(0x7f8ff40ce4e0 msgr2=0x7f8ff40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:40.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 -- 172.21.2.117:0/2180560795 shutdown_connections 2021-02-10T11:53:40.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.547 7f8ff9ca8700 1 -- 172.21.2.117:0/2180560795 wait complete. 2021-02-10T11:53:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 Processor -- start 2021-02-10T11:53:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 -- start start 2021-02-10T11:53:40.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ff40b4af0 0x7f8ff41af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ff41afca0 0x7f8ff41984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41b00a0 con 0x7f8ff409eb20 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4198ae0 con 0x7f8ff41afca0 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff9ca8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4198d00 con 0x7f8ff40b4af0 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57380/0 (socket says 172.21.2.117:57380) 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.562 7f8ff37fe700 1 -- 172.21.2.117:0/1203926914 learned_addr learned my addr 172.21.2.117:0/1203926914 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 -- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ff40b4af0 msgr2=0x7f8ff41af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ff40b4af0 0x7f8ff41af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 -- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ff41afca0 msgr2=0x7f8ff41984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ff41afca0 0x7f8ff41984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 -- 172.21.2.117:0/1203926914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4198f20 con 0x7f8ff409eb20 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff37fe700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 rx=0x7f8fdc00b530 tx=0x7f8fdc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fdc0009b0 con 0x7f8ff409eb20 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00bcc0 con 0x7f8ff409eb20 2021-02-10T11:53:40.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff4198f20 con 0x7f8ff409eb20 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fdc0062a0 con 0x7f8ff409eb20 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.563 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff4198f20 con 0x7f8ff409eb20 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.564 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fdc006230 con 0x7f8ff409eb20 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.564 7f8febfff700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 0x7f8fd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.564 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8fdc032590 con 0x7f8ff409eb20 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.564 7f8ff2ffd700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 0x7f8fd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:40.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.565 7f8ff2ffd700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 0x7f8fd801abc0 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:40.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.595 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff410c270 con 0x7f8ff409eb20 2021-02-10T11:53:40.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.598 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fdc02ef70 con 0x7f8ff409eb20 2021-02-10T11:53:40.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.722 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ff4085930 con 0x7f8ff409eb20 2021-02-10T11:53:40.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.722 7f8febfff700 1 -- 172.21.2.117:0/1203926914 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f8fdc02ef70 con 0x7f8ff409eb20 2021-02-10T11:53:40.724 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:40.724 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":152,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:38.470991","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:53:35.441170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:53:36.462661","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:40.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 msgr2=0x7f8fd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 0x7f8fd801abc0 crc :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 msgr2=0x7f8ff40b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 rx=0x7f8fdc00b530 tx=0x7f8fdc00b8d0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 shutdown_connections 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ff41afca0 0x7f8ff41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ff40b4af0 0x7f8ff41af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fd80186c0 0x7f8fd801abc0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 --2- 172.21.2.117:0/1203926914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ff409eb20 0x7f8ff40b44d0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:40.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 >> 172.21.2.117:0/1203926914 conn(0x7f8ff4110e50 msgr2=0x7f8ff418c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:40.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 shutdown_connections 2021-02-10T11:53:40.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:40.724 7f8ff9ca8700 1 -- 172.21.2.117:0/1203926914 wait complete. 2021-02-10T11:53:40.735 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:53:41.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 Processor -- start 2021-02-10T11:53:41.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 -- start start 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd30807ab40 0x7fd3080794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd308079b10 0x7fd308079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3080e5770 con 0x7fd30807af40 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3080d18c0 con 0x7fd30807ab40 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd30c80f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080d1c60 con 0x7fd308079b10 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd306e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd30807af40 0x7fd30807b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57636/0 (socket says 172.21.2.117:57636) 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.158 7fd306e1d700 1 -- 172.21.2.117:0/3904065608 learned_addr learned my addr 172.21.2.117:0/3904065608 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182603772 0 0) 0x7fd300001500 con 0x7fd30807af40 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f0002fe0 con 0x7fd30807af40 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> v2:172.21.2.125:3300/0 conn(0x7fd30807ab40 0x7fd3080794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd305e1b700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd308079b10 0x7fd308079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3294064171 0 0) 0x7fd300001270 con 0x7fd30807af40 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd308079b10 msgr2=0x7fd308079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd308079b10 0x7fd308079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 >> v2:172.21.2.125:3300/0 conn(0x7fd30807ab40 msgr2=0x7fd3080794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 --2- 172.21.2.117:0/3904065608 >> v2:172.21.2.125:3300/0 conn(0x7fd30807ab40 0x7fd3080794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30807bb30 con 0x7fd30807af40 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd300001b50 con 0x7fd30807af40 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 >> v1:172.21.2.117:6789/0 conn(0x7fd30807af40 legacy=0x7fd30807b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2f0003b10 0x7fd2f0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2f0006840 0x7fd2f0008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2f0009640 0x7fd2f000bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0050390 con 0x7fd2f0003b10 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00505b0 con 0x7fd2f0006840 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30561a700 1 -- 172.21.2.117:0/3904065608 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00507d0 con 0x7fd2f0009640 2021-02-10T11:53:41.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> v2:172.21.2.125:3300/0 conn(0x7fd30807ab40 0x7fd3080794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.159 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2f0003b10 0x7fd2f0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd305e1b700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2f0006840 0x7fd2f0008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd306e1d700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2f0009640 0x7fd2f000bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 -- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2f0009640 msgr2=0x7fd2f000bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2f0009640 0x7fd2f000bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 -- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2f0006840 msgr2=0x7fd2f0008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2f0006840 0x7fd2f0008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 -- 172.21.2.117:0/3904065608 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0050a70 con 0x7fd2f0003b10 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30661c700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2f0003b10 0x7fd2f0005fa0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fd2f8009950 tx=0x7fd2f8009cf0).ready entity=mon.0 client_cookie=b7be243b6983b55d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2f8000960 con 0x7fd2f0003b10 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f8000c90 con 0x7fd2f0003b10 2021-02-10T11:53:41.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30561a700 1 -- 172.21.2.117:0/3904065608 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2f80022b0 con 0x7fd2f0003b10 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30c80f700 1 -- 172.21.2.117:0/3904065608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2f0003b10 msgr2=0x7fd2f0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd30c80f700 1 --2- 172.21.2.117:0/3904065608 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2f0003b10 0x7fd2f0005fa0 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 rx=0x7fd2f8009950 tx=0x7fd2f8009cf0).stop 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.160 7fd306e1d700 1 -- 172.21.2.117:0/3904065608 reap_dead start 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.161 7fd30c80f700 1 -- 172.21.2.117:0/3904065608 shutdown_connections 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.161 7fd30c80f700 1 -- 172.21.2.117:0/3904065608 >> 172.21.2.117:0/3904065608 conn(0x7fd3080d0910 msgr2=0x7fd3080d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.161 7fd30c80f700 1 -- 172.21.2.117:0/3904065608 shutdown_connections 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.161 7fd30c80f700 1 -- 172.21.2.117:0/3904065608 wait complete. 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 Processor -- start 2021-02-10T11:53:41.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 -- start start 2021-02-10T11:53:41.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3081a48f0 0x7fd3080ba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd30809da50 0x7fd3080ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080a1da0 con 0x7fd3080ac640 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30809de50 con 0x7fd3081a48f0 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd30c80f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30809e070 con 0x7fd30809da50 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd306e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd306e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57398/0 (socket says 172.21.2.117:57398) 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.176 7fd306e1d700 1 -- 172.21.2.117:0/3395100798 learned_addr learned my addr 172.21.2.117:0/3395100798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 -- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd30809da50 msgr2=0x7fd3080ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd30809da50 0x7fd3080ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 -- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3081a48f0 msgr2=0x7fd3080ba7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3081a48f0 0x7fd3080ba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 -- 172.21.2.117:0/3395100798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30809ef40 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd306e1d700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7fd300001c10 tx=0x7fd30000ad20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd300000f30 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd300007020 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd300007540 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30809ef40 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.177 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30809f4d0 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.178 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd300007200 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.178 7fd2f6ffd700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 0x7fd2ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.178 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd300032a40 con 0x7fd3080ac640 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.178 7fd30661c700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 0x7fd2ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.179 7fd30661c700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 0x7fd2ec01abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.208 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3080daa10 con 0x7fd3080ac640 2021-02-10T11:53:41.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.211 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd300021370 con 0x7fd3080ac640 2021-02-10T11:53:41.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.340 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd30807a5f0 con 0x7fd3080ac640 2021-02-10T11:53:41.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.343 7fd2f6ffd700 1 -- 172.21.2.117:0/3395100798 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v85) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd300007200 con 0x7fd3080ac640 2021-02-10T11:53:41.349 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:53:41.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 msgr2=0x7fd2ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 0x7fd2ec01abc0 crc :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 msgr2=0x7fd30809e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 rx=0x7fd300001c10 tx=0x7fd30000ad20).stop 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 shutdown_connections 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2ec0186c0 0x7fd2ec01abc0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3080ac640 0x7fd30809e920 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd30809da50 0x7fd3080ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 --2- 172.21.2.117:0/3395100798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3081a48f0 0x7fd3080ba7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 >> 172.21.2.117:0/3395100798 conn(0x7fd3080b82d0 msgr2=0x7fd3080c2150 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:41.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 shutdown_connections 2021-02-10T11:53:41.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.349 7fd30c80f700 1 -- 172.21.2.117:0/3395100798 wait complete. 2021-02-10T11:53:41.359 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.772 7fe42729e700 1 Processor -- start 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.772 7fe42729e700 1 -- start start 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe42729e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4201b69b0 0x7fe4201b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe42729e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200ede70 con 0x7fe4200ef400 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe42729e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4200eba10 con 0x7fe4200d1900 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe42729e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200d1d00 con 0x7fe4201b69b0 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe4248b9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe4200ef400 0x7fe4200eb220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57652/0 (socket says 172.21.2.117:57652) 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe4248b9700 1 -- 172.21.2.117:0/2991926984 learned_addr learned my addr 172.21.2.117:0/2991926984 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:41.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe4258bb700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4201b69b0 0x7fe4201b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885313599 0 0) 0x7fe414000ce0 con 0x7fe4200ef400 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.773 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40c002fe0 con 0x7fe4200ef400 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280150468 0 0) 0x7fe41c001080 con 0x7fe4200d1900 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe40c003210 con 0x7fe4200d1900 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 715400855 0 0) 0x7fe414000ed0 con 0x7fe4200ef400 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4201b69b0 msgr2=0x7fe4201b8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4201b69b0 0x7fe4201b8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 >> v1:172.21.2.125:6789/0 conn(0x7fe4200d1900 legacy=0x7fe4200eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4201bb400 con 0x7fe4200ef400 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe4140016a0 con 0x7fe4200ef400 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 >> v1:172.21.2.117:6789/0 conn(0x7fe4200ef400 legacy=0x7fe4200eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe40c003b90 0x7fe40c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe40c006770 0x7fe40c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe40c009590 0x7fe40c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c00c2d0 con 0x7fe40c003b90 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0504c0 con 0x7fe40c006770 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe413fff700 1 -- 172.21.2.117:0/2991926984 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0506e0 con 0x7fe40c009590 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe40c006770 0x7fe40c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4250ba700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe40c003b90 0x7fe40c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4258bb700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe40c009590 0x7fe40c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 -- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe40c009590 msgr2=0x7fe40c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe40c009590 0x7fe40c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 -- 172.21.2.117:0/2991926984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe40c003b90 msgr2=0x7fe40c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe40c003b90 0x7fe40c005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.774 7fe4248b9700 1 -- 172.21.2.117:0/2991926984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c050980 con 0x7fe40c006770 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe4258bb700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe40c009590 0x7fe40c00ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe4248b9700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe40c006770 0x7fe40c008c90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fe4140012c0 tx=0x7fe414007cf0).ready entity=mon.1 client_cookie=d7df5c46b2cc6f72 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe414000ed0 con 0x7fe40c006770 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4140070c0 con 0x7fe40c006770 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe413fff700 1 -- 172.21.2.117:0/2991926984 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe414000ed0 con 0x7fe40c006770 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe42729e700 1 -- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe40c006770 msgr2=0x7fe40c008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe42729e700 1 --2- 172.21.2.117:0/2991926984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe40c006770 0x7fe40c008c90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fe4140012c0 tx=0x7fe414007cf0).stop 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.775 7fe4258bb700 1 -- 172.21.2.117:0/2991926984 reap_dead start 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.776 7fe42729e700 1 -- 172.21.2.117:0/2991926984 shutdown_connections 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.776 7fe42729e700 1 -- 172.21.2.117:0/2991926984 >> 172.21.2.117:0/2991926984 conn(0x7fe4200cf070 msgr2=0x7fe4200cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.776 7fe42729e700 1 -- 172.21.2.117:0/2991926984 shutdown_connections 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.776 7fe42729e700 1 -- 172.21.2.117:0/2991926984 wait complete. 2021-02-10T11:53:41.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.790 7fe42729e700 1 Processor -- start 2021-02-10T11:53:41.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.790 7fe42729e700 1 -- start start 2021-02-10T11:53:41.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4200b9d10 0x7fe4200b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4200ba110 0x7fe4200b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200b2df0 con 0x7fe4200dc530 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200dc930 con 0x7fe4200b9d10 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe42729e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4200b88a0 con 0x7fe4200ba110 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe4250ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe4250ba700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57414/0 (socket says 172.21.2.117:57414) 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe4250ba700 1 -- 172.21.2.117:0/993249968 learned_addr learned my addr 172.21.2.117:0/993249968 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe4248b9700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4200b9d10 0x7fe4200b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.791 7fe4258bb700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4200ba110 0x7fe4200b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 -- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4200ba110 msgr2=0x7fe4200b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4200ba110 0x7fe4200b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 -- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4200b9d10 msgr2=0x7fe4200b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4200b9d10 0x7fe4200b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 -- 172.21.2.117:0/993249968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4200b8ac0 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe4250ba700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7fe41c006ed0 tx=0x7fe41c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c006240 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe41c000c10 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe41c005770 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe42729e700 1 -- 172.21.2.117:0/993249968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4200b8ac0 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.792 7fe42729e700 1 -- 172.21.2.117:0/993249968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4200b8ac0 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.793 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe41c00a000 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.793 7fe411ffb700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 0x7fe40001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.793 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe41c032950 con 0x7fe4200dc530 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.793 7fe4248b9700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 0x7fe40001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:41.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.794 7fe4248b9700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 0x7fe40001abc0 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:41.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.823 7fe42729e700 1 -- 172.21.2.117:0/993249968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4200b2df0 con 0x7fe4200dc530 2021-02-10T11:53:41.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.826 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe41c000c10 con 0x7fe4200dc530 2021-02-10T11:53:41.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.957 7fe42729e700 1 -- 172.21.2.117:0/993249968 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe4200b2df0 con 0x7fe4200dc530 2021-02-10T11:53:41.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.961 7fe411ffb700 1 -- 172.21.2.117:0/993249968 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v86) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe41c00a000 con 0x7fe4200dc530 2021-02-10T11:53:41.966 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 -- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 msgr2=0x7fe40001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 0x7fe40001abc0 crc :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 -- 172.21.2.117:0/993249968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 msgr2=0x7fe420086f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 rx=0x7fe41c006ed0 tx=0x7fe41c00b960).stop 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 -- 172.21.2.117:0/993249968 shutdown_connections 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4200ba110 0x7fe4200b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4200b9d10 0x7fe4200b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4000186c0 0x7fe40001abc0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 --2- 172.21.2.117:0/993249968 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4200dc530 0x7fe420086f60 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:41.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.966 7fe42729e700 1 -- 172.21.2.117:0/993249968 >> 172.21.2.117:0/993249968 conn(0x7fe4200af230 msgr2=0x7fe4200ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:41.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.967 7fe42729e700 1 -- 172.21.2.117:0/993249968 shutdown_connections 2021-02-10T11:53:41.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:41.967 7fe42729e700 1 -- 172.21.2.117:0/993249968 wait complete. 2021-02-10T11:53:41.977 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 Processor -- start 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 -- start start 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af00d1900 0x7f8af00eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8af00ef400 0x7f8af01b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00ede70 con 0x7f8af00eb050 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af0088020 con 0x7f8af00ef400 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af83eb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af00d1d40 con 0x7f8af00d1900 2021-02-10T11:53:42.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af6a08700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8af00ef400 0x7f8af01b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af5a06700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8af00eb050 0x7f8af00eb450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57672/0 (socket says 172.21.2.117:57672) 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.390 7f8af5a06700 1 -- 172.21.2.117:0/2418379096 learned_addr learned my addr 172.21.2.117:0/2418379096 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af6a08700 1 -- 172.21.2.117:0/2418379096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af00d1900 msgr2=0x7f8af00eaa30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af6a08700 1 --2- 172.21.2.117:0/2418379096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af00d1900 0x7f8af00eaa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af6a08700 1 -- 172.21.2.117:0/2418379096 >> v1:172.21.2.117:6789/0 conn(0x7f8af00eb050 legacy=0x7f8af00eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af6a08700 1 -- 172.21.2.117:0/2418379096 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af01b8fe0 con 0x7f8af00ef400 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af6a08700 1 --2- 172.21.2.117:0/2418379096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00ef400 0x7f8af01b79b0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7f8ae8006e30 tx=0x7f8ae8009e60).ready entity=mon.0 client_cookie=2cb96a8c4f1f634a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af5205700 1 -- 172.21.2.117:0/2418379096 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae800b300 con 0x7f8af00ef400 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af5205700 1 -- 172.21.2.117:0/2418379096 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae8000d30 con 0x7f8af00ef400 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af5205700 1 -- 172.21.2.117:0/2418379096 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae800b770 con 0x7f8af00ef400 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af83eb700 1 -- 172.21.2.117:0/2418379096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00ef400 msgr2=0x7f8af01b79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.391 7f8af83eb700 1 --2- 172.21.2.117:0/2418379096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00ef400 0x7f8af01b79b0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 rx=0x7f8ae8006e30 tx=0x7f8ae8009e60).stop 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 -- 172.21.2.117:0/2418379096 shutdown_connections 2021-02-10T11:53:42.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 --2- 172.21.2.117:0/2418379096 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00ef400 0x7f8af01b79b0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 --2- 172.21.2.117:0/2418379096 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af00d1900 0x7f8af00eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:42.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 -- 172.21.2.117:0/2418379096 >> 172.21.2.117:0/2418379096 conn(0x7f8af00cf070 msgr2=0x7f8af00cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:42.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 -- 172.21.2.117:0/2418379096 shutdown_connections 2021-02-10T11:53:42.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.392 7f8af83eb700 1 -- 172.21.2.117:0/2418379096 wait complete. 2021-02-10T11:53:42.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 Processor -- start 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 -- start start 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8af0191250 0x7f8af0191650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af0191c70 0x7f8af008c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01920f0 con 0x7f8af00b0810 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01927b0 con 0x7f8af0191250 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af83eb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0190d40 con 0x7f8af0191c70 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af6207700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af6207700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57428/0 (socket says 172.21.2.117:57428) 2021-02-10T11:53:42.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.407 7f8af6207700 1 -- 172.21.2.117:0/1479043627 learned_addr learned my addr 172.21.2.117:0/1479043627 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6a08700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af0191c70 0x7f8af008c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af5a06700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8af0191250 0x7f8af0191650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 -- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af0191c70 msgr2=0x7f8af008c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af0191c70 0x7f8af008c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 -- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8af0191250 msgr2=0x7f8af0191650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8af0191250 0x7f8af0191650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 -- 172.21.2.117:0/1479043627 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0190980 con 0x7f8af00b0810 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af6207700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f8ae400b5c0 tx=0x7f8ae400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae400bd50 con 0x7f8af00b0810 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae4006030 con 0x7f8af00b0810 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ae40054d0 con 0x7f8af00b0810 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.408 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0190980 con 0x7f8af00b0810 2021-02-10T11:53:42.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.409 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0190980 con 0x7f8af00b0810 2021-02-10T11:53:42.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.409 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8ae4006210 con 0x7f8af00b0810 2021-02-10T11:53:42.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.409 7f8ae2ffd700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:42.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.409 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8ae40326b0 con 0x7f8af00b0810 2021-02-10T11:53:42.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.409 7f8af5a06700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:42.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.410 7f8af5a06700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 0x7f8ad801abc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:42.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.440 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af00ede70 con 0x7f8af00b0810 2021-02-10T11:53:42.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.442 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8ae4006b80 con 0x7f8af00b0810 2021-02-10T11:53:42.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.571 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8af00ede70 con 0x7f8af00b0810 2021-02-10T11:53:42.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.574 7f8ae2ffd700 1 -- 172.21.2.117:0/1479043627 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v87) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8ae4006210 con 0x7f8af00b0810 2021-02-10T11:53:42.580 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:53:42.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 msgr2=0x7f8ad801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 0x7f8ad801abc0 crc :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 msgr2=0x7f8af00b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 rx=0x7f8ae400b5c0 tx=0x7f8ae400b960).stop 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 shutdown_connections 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8af0191c70 0x7f8af008c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8af0191250 0x7f8af0191650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8ad80186c0 0x7f8ad801abc0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 --2- 172.21.2.117:0/1479043627 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8af00b0810 0x7f8af00b0c10 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:42.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 >> 172.21.2.117:0/1479043627 conn(0x7f8af0084ac0 msgr2=0x7f8af019ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:42.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 shutdown_connections 2021-02-10T11:53:42.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:42.580 7f8af83eb700 1 -- 172.21.2.117:0/1479043627 wait complete. 2021-02-10T11:53:42.590 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.012 7ff912d88700 1 Processor -- start 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.012 7ff912d88700 1 -- start start 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff912d88700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff90c07ab40 0x7ff90c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff912d88700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c0e5770 con 0x7ff90c079f10 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff912d88700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c07a310 con 0x7ff90c079b10 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff912d88700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c0d18c0 con 0x7ff90c07ab40 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff910ba4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff90c07ab40 0x7ff90c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff9113a5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff90c079f10 0x7ff90c07b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57684/0 (socket says 172.21.2.117:57684) 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff9113a5700 1 -- 172.21.2.117:0/2900372258 learned_addr learned my addr 172.21.2.117:0/2900372258 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:43.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.013 7ff910ba4700 1 -- 172.21.2.117:0/2900372258 >> v1:172.21.2.125:6789/0 conn(0x7ff90c079b10 legacy=0x7ff90c07af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff910ba4700 1 -- 172.21.2.117:0/2900372258 >> v1:172.21.2.117:6789/0 conn(0x7ff90c079f10 legacy=0x7ff90c07b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff910ba4700 1 -- 172.21.2.117:0/2900372258 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c07bf20 con 0x7ff90c07ab40 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff90b7fe700 1 -- 172.21.2.117:0/2900372258 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544061878 0 0) 0x7ff904000c10 con 0x7ff90c079f10 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff910ba4700 1 --2- 172.21.2.117:0/2900372258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c07ab40 0x7ff90c0794f0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7ff8f4006650 tx=0x7ff8f4009a80).ready entity=mon.0 client_cookie=4e48856abd3d672e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff90b7fe700 1 -- 172.21.2.117:0/2900372258 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f40009b0 con 0x7ff90c07ab40 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff90b7fe700 1 -- 172.21.2.117:0/2900372258 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f400c0c0 con 0x7ff90c07ab40 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff90b7fe700 1 -- 172.21.2.117:0/2900372258 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f400c5e0 con 0x7ff90c07ab40 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff912d88700 1 -- 172.21.2.117:0/2900372258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c07ab40 msgr2=0x7ff90c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.014 7ff912d88700 1 --2- 172.21.2.117:0/2900372258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c07ab40 0x7ff90c0794f0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 rx=0x7ff8f4006650 tx=0x7ff8f4009a80).stop 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.015 7ff912d88700 1 -- 172.21.2.117:0/2900372258 shutdown_connections 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.015 7ff912d88700 1 --2- 172.21.2.117:0/2900372258 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c07ab40 0x7ff90c0794f0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.015 7ff912d88700 1 -- 172.21.2.117:0/2900372258 >> 172.21.2.117:0/2900372258 conn(0x7ff90c0d0910 msgr2=0x7ff90c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:43.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.015 7ff912d88700 1 -- 172.21.2.117:0/2900372258 shutdown_connections 2021-02-10T11:53:43.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.015 7ff912d88700 1 -- 172.21.2.117:0/2900372258 wait complete. 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.029 7ff912d88700 1 Processor -- start 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 -- start start 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff90c193770 0x7ff90c1b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff90c1b1a50 0x7ff90c1b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1b14d0 con 0x7ff90c193370 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1b10c0 con 0x7ff90c193770 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff912d88700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1b12e0 con 0x7ff90c1b1a50 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff910ba4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff910ba4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57442/0 (socket says 172.21.2.117:57442) 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.030 7ff910ba4700 1 -- 172.21.2.117:0/4156610973 learned_addr learned my addr 172.21.2.117:0/4156610973 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff90bfff700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff90c193770 0x7ff90c1b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff9113a5700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff90c1b1a50 0x7ff90c1b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 -- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff90c1b1a50 msgr2=0x7ff90c1b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff90c1b1a50 0x7ff90c1b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 -- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff90c193770 msgr2=0x7ff90c1b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff90c193770 0x7ff90c1b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 -- 172.21.2.117:0/4156610973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c1b2470 con 0x7ff90c193370 2021-02-10T11:53:43.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff910ba4700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7ff8f4009630 tx=0x7ff8f400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f40093a0 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f400bd70 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8f4005560 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff912d88700 1 -- 172.21.2.117:0/4156610973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c1b2470 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.031 7ff912d88700 1 -- 172.21.2.117:0/4156610973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c1b2a00 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.032 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8f4007020 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.032 7ff9097fa700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 0x7ff8f001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.032 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8f4032530 con 0x7ff90c193370 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.032 7ff90bfff700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 0x7ff8f001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.033 7ff90bfff700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 0x7ff8f001abc0 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.062 7ff912d88700 1 -- 172.21.2.117:0/4156610973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c0adbc0 con 0x7ff90c193370 2021-02-10T11:53:43.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.065 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8f402ebc0 con 0x7ff90c193370 2021-02-10T11:53:43.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.194 7ff912d88700 1 -- 172.21.2.117:0/4156610973 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7ff90c0adbc0 con 0x7ff90c193370 2021-02-10T11:53:43.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.197 7ff9097fa700 1 -- 172.21.2.117:0/4156610973 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v88) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff8f4007020 con 0x7ff90c193370 2021-02-10T11:53:43.199 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:53:43.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 msgr2=0x7ff8f001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 0x7ff8f001abc0 crc :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 msgr2=0x7ff90c1b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7ff8f4009630 tx=0x7ff8f400b980).stop 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 shutdown_connections 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff90c1b1a50 0x7ff90c1b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff90c193770 0x7ff90c1b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8f00186c0 0x7ff8f001abc0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 --2- 172.21.2.117:0/4156610973 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff90c193370 0x7ff90c1b0480 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 >> 172.21.2.117:0/4156610973 conn(0x7ff90c192380 msgr2=0x7ff90c19eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:43.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 shutdown_connections 2021-02-10T11:53:43.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.199 7ff912d88700 1 -- 172.21.2.117:0/4156610973 wait complete. 2021-02-10T11:53:43.209 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:43.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.619 7f1f130f2700 1 Processor -- start 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.619 7f1f130f2700 1 -- start start 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f130f2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1f0c0ef3f0 0x7f1f0c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f130f2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f130f2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0c0ede60 con 0x7f1f0c0d18e0 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f130f2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0c0ef7f0 con 0x7f1f0c0eb7f0 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f130f2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0c0d1ce0 con 0x7f1f0c0ef3f0 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57448/0 (socket says 172.21.2.117:57448) 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 -- 172.21.2.117:0/3762218473 learned_addr learned my addr 172.21.2.117:0/3762218473 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:43.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f0bfff700 1 --2- 172.21.2.117:0/3762218473 >> v2:172.21.2.125:3300/0 conn(0x7f1f0c0ef3f0 0x7f1f0c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 -- 172.21.2.117:0/3762218473 >> v2:172.21.2.125:3300/0 conn(0x7f1f0c0ef3f0 msgr2=0x7f1f0c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 --2- 172.21.2.117:0/3762218473 >> v2:172.21.2.125:3300/0 conn(0x7f1f0c0ef3f0 0x7f1f0c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 -- 172.21.2.117:0/3762218473 >> v1:172.21.2.125:6789/0 conn(0x7f1f0c0d18e0 legacy=0x7f1f0c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f0b7fe700 1 -- 172.21.2.117:0/3762218473 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671075002 0 0) 0x7f1ef4000c60 con 0x7f1f0c0d18e0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.620 7f1f1170f700 1 -- 172.21.2.117:0/3762218473 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c1b8f90 con 0x7f1f0c0eb7f0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f1170f700 1 --2- 172.21.2.117:0/3762218473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7f1f04006f00 tx=0x7f1f04009f10).ready entity=mon.0 client_cookie=b9bf099773241803 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f0b7fe700 1 -- 172.21.2.117:0/3762218473 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0400b3b0 con 0x7f1f0c0eb7f0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f0b7fe700 1 -- 172.21.2.117:0/3762218473 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0400b730 con 0x7f1f0c0eb7f0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f0b7fe700 1 -- 172.21.2.117:0/3762218473 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0400bc10 con 0x7f1f0c0eb7f0 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f130f2700 1 -- 172.21.2.117:0/3762218473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0eb7f0 msgr2=0x7f1f0c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.621 7f1f130f2700 1 --2- 172.21.2.117:0/3762218473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 rx=0x7f1f04006f00 tx=0x7f1f04009f10).stop 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 -- 172.21.2.117:0/3762218473 shutdown_connections 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 --2- 172.21.2.117:0/3762218473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0eb7f0 0x7f1f0c1b7960 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 --2- 172.21.2.117:0/3762218473 >> v2:172.21.2.125:3300/0 conn(0x7f1f0c0ef3f0 0x7f1f0c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 -- 172.21.2.117:0/3762218473 >> 172.21.2.117:0/3762218473 conn(0x7f1f0c0ced60 msgr2=0x7f1f0c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:43.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 -- 172.21.2.117:0/3762218473 shutdown_connections 2021-02-10T11:53:43.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.622 7f1f130f2700 1 -- 172.21.2.117:0/3762218473 wait complete. 2021-02-10T11:53:43.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.636 7f1f130f2700 1 Processor -- start 2021-02-10T11:53:43.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.636 7f1f130f2700 1 -- start start 2021-02-10T11:53:43.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f0c1a33d0 0x7f1f0c1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f0c0b82f0 0x7f1f0c0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0c0670 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0deb20 con 0x7f1f0c0b82f0 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f130f2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0c0ded40 con 0x7f1f0c1a33d0 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f1170f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f0bfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f0c0b82f0 0x7f1f0c0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f1170f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57452/0 (socket says 172.21.2.117:57452) 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f1170f700 1 -- 172.21.2.117:0/938127432 learned_addr learned my addr 172.21.2.117:0/938127432 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.637 7f1f10f0e700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f0c1a33d0 0x7f1f0c1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 -- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f0c1a33d0 msgr2=0x7f1f0c1a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f0c1a33d0 0x7f1f0c1a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 -- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f0c0b82f0 msgr2=0x7f1f0c0b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f0c0b82f0 0x7f1f0c0b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 -- 172.21.2.117:0/938127432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0def60 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f1170f700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7f1f04009390 tx=0x7f1f0400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f04003030 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f040068e0 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f130f2700 1 -- 172.21.2.117:0/938127432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0c0def60 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f130f2700 1 -- 172.21.2.117:0/938127432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0c0df4f0 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.638 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1f0400c570 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.639 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1f04005820 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.639 7f1f097fa700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.639 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1f04032a30 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.639 7f1f10f0e700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:43.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.639 7f1f10f0e700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 0x7f1ef001abc0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:43.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.669 7f1f130f2700 1 -- 172.21.2.117:0/938127432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0c087290 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.672 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1f04009450 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.801 7f1f130f2700 1 -- 172.21.2.117:0/938127432 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f0c0ede60 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.801 7f1f097fa700 1 -- 172.21.2.117:0/938127432 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f1f04005820 con 0x7f1f0c0b8d10 2021-02-10T11:53:43.804 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:43.804 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":128,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:53:39.473419","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8737},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8737":{"gid":8737,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/277081047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":277081047},{"type":"v1","addr":"172.21.2.125:6835","nonce":277081047}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:53:43.805 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 128 2021-02-10T11:53:43.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 msgr2=0x7f1ef001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 0x7f1ef001abc0 crc :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 msgr2=0x7f1f0c0b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:43.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 rx=0x7f1f04009390 tx=0x7f1f0400bd40).stop 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 shutdown_connections 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1ef00186c0 0x7f1ef001abc0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1f0c0b8d10 0x7f1f0c0b9110 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1f0c0b82f0 0x7f1f0c0b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 --2- 172.21.2.117:0/938127432 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1f0c1a33d0 0x7f1f0c1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:43.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 >> 172.21.2.117:0/938127432 conn(0x7f1f0c0874c0 msgr2=0x7f1f0c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:43.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 shutdown_connections 2021-02-10T11:53:43.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:43.803 7f1f130f2700 1 -- 172.21.2.117:0/938127432 wait complete. 2021-02-10T11:53:44.016 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:53:37.468914', 'modified': '2021-02-10 11:53:39.473419', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8737}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8737': {'gid': 8737, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/277081047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 277081047}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 277081047}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:53:44.016 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:53:44.016 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:53:44.416 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:53:44.416 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8737, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9430161569999997, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 128, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 152, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 152, 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.595272868 2021-02-10T11:53:44.417 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:53:44.420 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8737, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9430161569999997, "mdsmap_epoch": 128, "osdmap_epoch": 152, "osdmap_epoch_barrier": 152, "uptime": 12.595272868 } 2021-02-10T11:53:44.421 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:44.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.831 7fe22853e700 1 Processor -- start 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.831 7fe22853e700 1 -- start start 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe22853e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200d18e0 0x7fe2200ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe22853e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe2200eb000 0x7fe2200eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe22853e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2200ede60 con 0x7fe2200ee3e0 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe22853e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2200eba20 con 0x7fe2200eb000 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe22853e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200d1ce0 con 0x7fe2200d18e0 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe226b5b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe2200ee3e0 0x7fe2200ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57708/0 (socket says 172.21.2.117:57708) 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.832 7fe226b5b700 1 -- 172.21.2.117:0/2938866567 learned_addr learned my addr 172.21.2.117:0/2938866567 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240488843 0 0) 0x7fe218001500 con 0x7fe2200ee3e0 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c002fe0 con 0x7fe2200ee3e0 2021-02-10T11:53:44.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200d18e0 0x7fe2200ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225b59700 1 --2- 172.21.2.117:0/2938866567 >> v2:172.21.2.125:3300/0 conn(0x7fe2200eb000 0x7fe2200eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1384154622 0 0) 0x7fe218001460 con 0x7fe2200ee3e0 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200d18e0 msgr2=0x7fe2200ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200d18e0 0x7fe2200ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 >> v2:172.21.2.125:3300/0 conn(0x7fe2200eb000 msgr2=0x7fe2200eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 --2- 172.21.2.117:0/2938866567 >> v2:172.21.2.125:3300/0 conn(0x7fe2200eb000 0x7fe2200eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200eefd0 con 0x7fe2200ee3e0 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe218001980 con 0x7fe2200ee3e0 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 >> v1:172.21.2.117:6789/0 conn(0x7fe2200ee3e0 legacy=0x7fe2200ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe20c003b10 0x7fe20c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe20c006820 0x7fe20c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe20c009620 0x7fe20c00bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c050370 con 0x7fe20c003b10 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c050590 con 0x7fe20c006820 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225358700 1 -- 172.21.2.117:0/2938866567 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c0507b0 con 0x7fe20c009620 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200d18e0 0x7fe2200ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe20c003b10 0x7fe20c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe225b59700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe20c006820 0x7fe20c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.833 7fe226b5b700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe20c009620 0x7fe20c00bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 -- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe20c009620 msgr2=0x7fe20c00bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe20c009620 0x7fe20c00bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 -- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe20c006820 msgr2=0x7fe20c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe20c006820 0x7fe20c008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 -- 172.21.2.117:0/2938866567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c050a50 con 0x7fe20c003b10 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22635a700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe20c003b10 0x7fe20c005fa0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 rx=0x7fe21c006890 tx=0x7fe21c009aa0).ready entity=mon.0 client_cookie=a504b1919a3b7f04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:44.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c00c0c0 con 0x7fe20c003b10 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c00c340 con 0x7fe20c003b10 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe225358700 1 -- 172.21.2.117:0/2938866567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c00c860 con 0x7fe20c003b10 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22853e700 1 -- 172.21.2.117:0/2938866567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe20c003b10 msgr2=0x7fe20c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe22853e700 1 --2- 172.21.2.117:0/2938866567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe20c003b10 0x7fe20c005fa0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 rx=0x7fe21c006890 tx=0x7fe21c009aa0).stop 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.834 7fe226b5b700 1 -- 172.21.2.117:0/2938866567 reap_dead start 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.835 7fe22853e700 1 -- 172.21.2.117:0/2938866567 shutdown_connections 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.835 7fe22853e700 1 -- 172.21.2.117:0/2938866567 >> 172.21.2.117:0/2938866567 conn(0x7fe2200ced60 msgr2=0x7fe2200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.835 7fe22853e700 1 -- 172.21.2.117:0/2938866567 shutdown_connections 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.835 7fe22853e700 1 -- 172.21.2.117:0/2938866567 wait complete. 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.849 7fe22853e700 1 Processor -- start 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.849 7fe22853e700 1 -- start start 2021-02-10T11:53:44.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 0x7fe220098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe22008b3c0 0x7fe22008b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220097460 con 0x7fe2200a3210 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22008bde0 con 0x7fe22008b3c0 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22853e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22008bfc0 con 0x7fe2200a3610 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22635a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22635a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57468/0 (socket says 172.21.2.117:57468) 2021-02-10T11:53:44.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22635a700 1 -- 172.21.2.117:0/3725960504 learned_addr learned my addr 172.21.2.117:0/3725960504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:44.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe225b59700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 0x7fe220098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe226b5b700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe22008b3c0 0x7fe22008b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22635a700 1 -- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 msgr2=0x7fe220098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.850 7fe22635a700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 0x7fe220098000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22635a700 1 -- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe22008b3c0 msgr2=0x7fe22008b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22635a700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe22008b3c0 0x7fe22008b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22635a700 1 -- 172.21.2.117:0/3725960504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22008c1e0 con 0x7fe2200a3210 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe225b59700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 0x7fe220098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22635a700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7fe21c009680 tx=0x7fe21c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c00bd50 con 0x7fe2200a3210 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c006240 con 0x7fe2200a3210 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22853e700 1 -- 172.21.2.117:0/3725960504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22008c1e0 con 0x7fe2200a3210 2021-02-10T11:53:44.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe21c0054d0 con 0x7fe2200a3210 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe22853e700 1 -- 172.21.2.117:0/3725960504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22008c1e0 con 0x7fe2200a3210 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.851 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe21c0052e0 con 0x7fe2200a3210 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.852 7fe212ffd700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 0x7fe20801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.852 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe21c032790 con 0x7fe2200a3210 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.852 7fe225b59700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 0x7fe20801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:44.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.852 7fe225b59700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 0x7fe20801ac10 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:44.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.882 7fe22853e700 1 -- 172.21.2.117:0/3725960504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2200ede60 con 0x7fe2200a3210 2021-02-10T11:53:44.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:44.884 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe21c006030 con 0x7fe2200a3210 2021-02-10T11:53:45.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.009 7fe22853e700 1 -- 172.21.2.117:0/3725960504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2200ede60 con 0x7fe2200a3210 2021-02-10T11:53:45.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.009 7fe212ffd700 1 -- 172.21.2.117:0/3725960504 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fe21c01f500 con 0x7fe2200a3210 2021-02-10T11:53:45.012 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:45.013 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":152,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:38.470991","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:53:35.441170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:53:36.462661","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 msgr2=0x7fe20801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 0x7fe20801ac10 crc :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 msgr2=0x7fe2200979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 rx=0x7fe21c009680 tx=0x7fe21c00b960).stop 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 shutdown_connections 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe22008b3c0 0x7fe22008b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe2200a3610 0x7fe220098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe208018730 0x7fe20801ac10 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 --2- 172.21.2.117:0/3725960504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe2200a3210 0x7fe2200979e0 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 >> 172.21.2.117:0/3725960504 conn(0x7fe220087440 msgr2=0x7fe220098790 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:45.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 shutdown_connections 2021-02-10T11:53:45.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.011 7fe22853e700 1 -- 172.21.2.117:0/3725960504 wait complete. 2021-02-10T11:53:45.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:53:35.441170', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:53:36.462661', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:53:45.023 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.436 7feab4452700 1 Processor -- start 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.436 7feab4452700 1 -- start start 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab4452700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feaac0d18f0 0x7feaac0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab4452700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7feaac0ef3f0 0x7feaac1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab4452700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0ef7f0 con 0x7feaac0eb040 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab4452700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0ede60 con 0x7feaac0d18f0 2021-02-10T11:53:45.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab4452700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaac0d1cf0 con 0x7feaac0ef3f0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab226e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7feaac0d18f0 0x7feaac0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab1a6d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7feaac0eb040 0x7feaac0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57726/0 (socket says 172.21.2.117:57726) 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab1a6d700 1 -- 172.21.2.117:0/4144298337 learned_addr learned my addr 172.21.2.117:0/4144298337 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab226e700 1 -- 172.21.2.117:0/4144298337 >> v2:172.21.2.125:3300/0 conn(0x7feaac0ef3f0 msgr2=0x7feaac1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab126c700 1 -- 172.21.2.117:0/4144298337 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420372054 0 0) 0x7feaa80010b0 con 0x7feaac0eb040 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab226e700 1 --2- 172.21.2.117:0/4144298337 >> v2:172.21.2.125:3300/0 conn(0x7feaac0ef3f0 0x7feaac1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab226e700 1 -- 172.21.2.117:0/4144298337 >> v1:172.21.2.117:6789/0 conn(0x7feaac0eb040 legacy=0x7feaac0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.437 7feab226e700 1 -- 172.21.2.117:0/4144298337 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac1b8fd0 con 0x7feaac0d18f0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab226e700 1 --2- 172.21.2.117:0/4144298337 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0d18f0 0x7feaac0eaa20 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a80).ready entity=mon.0 client_cookie=b4739288cd673ce4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab126c700 1 -- 172.21.2.117:0/4144298337 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa000c0c0 con 0x7feaac0d18f0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab126c700 1 -- 172.21.2.117:0/4144298337 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa000c340 con 0x7feaac0d18f0 2021-02-10T11:53:45.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab126c700 1 -- 172.21.2.117:0/4144298337 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa000c860 con 0x7feaac0d18f0 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab4452700 1 -- 172.21.2.117:0/4144298337 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0d18f0 msgr2=0x7feaac0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab4452700 1 --2- 172.21.2.117:0/4144298337 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0d18f0 0x7feaac0eaa20 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 rx=0x7feaa0006650 tx=0x7feaa0009a80).stop 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab4452700 1 -- 172.21.2.117:0/4144298337 shutdown_connections 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.438 7feab4452700 1 --2- 172.21.2.117:0/4144298337 >> v2:172.21.2.125:3300/0 conn(0x7feaac0ef3f0 0x7feaac1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.439 7feab4452700 1 --2- 172.21.2.117:0/4144298337 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0d18f0 0x7feaac0eaa20 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.439 7feab4452700 1 -- 172.21.2.117:0/4144298337 >> 172.21.2.117:0/4144298337 conn(0x7feaac0cf060 msgr2=0x7feaac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.439 7feab4452700 1 -- 172.21.2.117:0/4144298337 shutdown_connections 2021-02-10T11:53:45.457 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.439 7feab4452700 1 -- 172.21.2.117:0/4144298337 wait complete. 2021-02-10T11:53:45.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 Processor -- start 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 -- start start 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaac0b3e10 0x7feaac0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaac0b4c10 0x7feaac0aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0b4320 con 0x7feaac0b5010 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac08a220 con 0x7feaac0b4c10 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.453 7feab4452700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaac0abbc0 con 0x7feaac0b3e10 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57484/0 (socket says 172.21.2.117:57484) 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 -- 172.21.2.117:0/335829544 learned_addr learned my addr 172.21.2.117:0/335829544 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 -- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaac0b3e10 msgr2=0x7feaac0b45f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:45.459 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaac0b3e10 0x7feaac0b45f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 -- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaac0b4c10 msgr2=0x7feaac0aaf80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaac0b4c10 0x7feaac0aaf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 -- 172.21.2.117:0/335829544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0abde0 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab2a6f700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7feaa4009980 tx=0x7feaa400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa4003020 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa4005a80 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab4452700 1 -- 172.21.2.117:0/335829544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac0abde0 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7feaa4006030 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.454 7feab4452700 1 -- 172.21.2.117:0/335829544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac0ac420 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.455 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7feaa4006030 con 0x7feaac0b5010 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.455 7fea9effd700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 0x7fea9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:45.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.455 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7feaa4032cc0 con 0x7feaac0b5010 2021-02-10T11:53:45.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.455 7feab226e700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 0x7fea9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:45.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.456 7feab226e700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 0x7fea9401abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:45.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.487 7feab4452700 1 -- 172.21.2.117:0/335829544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac08a220 con 0x7feaac0b5010 2021-02-10T11:53:45.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.489 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7feaa4005a80 con 0x7feaac0b5010 2021-02-10T11:53:45.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.619 7feab4452700 1 -- 172.21.2.117:0/335829544 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7feaac08a220 con 0x7feaac0b5010 2021-02-10T11:53:45.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.619 7fea9effd700 1 -- 172.21.2.117:0/335829544 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v152) v1 ==== 92+0+10 (secure 0 0 0) 0x7feaa4006030 con 0x7feaac0b5010 2021-02-10T11:53:45.622 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 msgr2=0x7fea9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 0x7fea9401abc0 crc :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 msgr2=0x7feaac0ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 rx=0x7feaa4009980 tx=0x7feaa400bcd0).stop 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 shutdown_connections 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fea94018730 0x7fea9401abc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feaac0b5010 0x7feaac0ab5a0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feaac0b4c10 0x7feaac0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 --2- 172.21.2.117:0/335829544 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feaac0b3e10 0x7feaac0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:45.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 >> 172.21.2.117:0/335829544 conn(0x7feaac0b6c70 msgr2=0x7feaac0970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:45.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 shutdown_connections 2021-02-10T11:53:45.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:45.621 7feab4452700 1 -- 172.21.2.117:0/335829544 wait complete. 2021-02-10T11:53:45.633 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:53:46.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.047 7f7d043c6700 1 Processor -- start 2021-02-10T11:53:46.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 -- start start 2021-02-10T11:53:46.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0d18f0 0x7f7cfc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cfc0ede60 con 0x7f7cfc0eb040 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cfc088020 con 0x7f7cfc0ef3f0 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d043c6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0d1d30 con 0x7f7cfc0d18f0 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d029e3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d029e3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57488/0 (socket says 172.21.2.117:57488) 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d029e3700 1 -- 172.21.2.117:0/3043874141 learned_addr learned my addr 172.21.2.117:0/3043874141 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.048 7f7d019e1700 1 --1- 172.21.2.117:0/3043874141 >> v1:172.21.2.117:6789/0 conn(0x7f7cfc0eb040 0x7f7cfc0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57738/0 (socket says 172.21.2.117:57738) 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d029e3700 1 -- 172.21.2.117:0/3043874141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0d18f0 msgr2=0x7f7cfc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d029e3700 1 --2- 172.21.2.117:0/3043874141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0d18f0 0x7f7cfc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d029e3700 1 -- 172.21.2.117:0/3043874141 >> v1:172.21.2.117:6789/0 conn(0x7f7cfc0eb040 legacy=0x7f7cfc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d029e3700 1 -- 172.21.2.117:0/3043874141 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc1b8fd0 con 0x7f7cfc0ef3f0 2021-02-10T11:53:46.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d029e3700 1 --2- 172.21.2.117:0/3043874141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f7cf4006f00 tx=0x7f7cf4009f10).ready entity=mon.0 client_cookie=c05c0a2933d140a7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d011e0700 1 -- 172.21.2.117:0/3043874141 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf400b3b0 con 0x7f7cfc0ef3f0 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d011e0700 1 -- 172.21.2.117:0/3043874141 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf400b730 con 0x7f7cfc0ef3f0 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d011e0700 1 -- 172.21.2.117:0/3043874141 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf400bc50 con 0x7f7cfc0ef3f0 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d043c6700 1 -- 172.21.2.117:0/3043874141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0ef3f0 msgr2=0x7f7cfc1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.049 7f7d043c6700 1 --2- 172.21.2.117:0/3043874141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 rx=0x7f7cf4006f00 tx=0x7f7cf4009f10).stop 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 -- 172.21.2.117:0/3043874141 shutdown_connections 2021-02-10T11:53:46.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 --2- 172.21.2.117:0/3043874141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0ef3f0 0x7f7cfc1b79a0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 --2- 172.21.2.117:0/3043874141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0d18f0 0x7f7cfc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 -- 172.21.2.117:0/3043874141 >> 172.21.2.117:0/3043874141 conn(0x7f7cfc0cf060 msgr2=0x7f7cfc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 -- 172.21.2.117:0/3043874141 shutdown_connections 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.050 7f7d043c6700 1 -- 172.21.2.117:0/3043874141 wait complete. 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.064 7f7d043c6700 1 Processor -- start 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.064 7f7d043c6700 1 -- start start 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 0x7f7cfc0b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 0x7f7cfc0aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b4320 con 0x7f7cfc0b5010 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc08a220 con 0x7f7cfc0b3e10 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d043c6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc0abbc0 con 0x7f7cfc0b4c10 2021-02-10T11:53:46.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d029e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d029e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57496/0 (socket says 172.21.2.117:57496) 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d029e3700 1 -- 172.21.2.117:0/3645042477 learned_addr learned my addr 172.21.2.117:0/3645042477 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d029e3700 1 -- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 msgr2=0x7f7cfc0aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d021e2700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 0x7f7cfc0b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.065 7f7d019e1700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 0x7f7cfc0aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d029e3700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 0x7f7cfc0aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d029e3700 1 -- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 msgr2=0x7f7cfc0b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d029e3700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 0x7f7cfc0b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d029e3700 1 -- 172.21.2.117:0/3645042477 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc0abde0 con 0x7f7cfc0b5010 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d021e2700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 0x7f7cfc0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d019e1700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 0x7f7cfc0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:46.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d029e3700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f7cf4009390 tx=0x7f7cf400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf4000cc0 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf4003030 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc0abde0 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7cf4001480 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.066 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc0ac420 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.067 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7cf4005820 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.067 7f7ceeffd700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.067 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7cf4032a20 con 0x7f7cfc0b5010 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.067 7f7d021e2700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.068 7f7d021e2700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 0x7f7ce401abc0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.098 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc0f67e0 con 0x7f7cfc0b5010 2021-02-10T11:53:46.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.101 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7cf402e700 con 0x7f7cfc0b5010 2021-02-10T11:53:46.158 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:53:46.161 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:53:46.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.231 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f7cfc0f67e0 con 0x7f7cfc0b5010 2021-02-10T11:53:46.234 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.232 7f7ceeffd700 1 -- 172.21.2.117:0/3645042477 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v152) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7cf402e700 con 0x7f7cfc0b5010 2021-02-10T11:53:46.235 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:46.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 msgr2=0x7f7ce401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 0x7f7ce401abc0 crc :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 msgr2=0x7f7cfc0ab5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 rx=0x7f7cf4009390 tx=0x7f7cf400bd40).stop 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 shutdown_connections 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7ce40186c0 0x7f7ce401abc0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7cfc0b5010 0x7f7cfc0ab5a0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7cfc0b4c10 0x7f7cfc0aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 --2- 172.21.2.117:0/3645042477 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7cfc0b3e10 0x7f7cfc0b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 >> 172.21.2.117:0/3645042477 conn(0x7f7cfc0b6c70 msgr2=0x7f7cfc0970a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:46.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.234 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 shutdown_connections 2021-02-10T11:53:46.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.235 7f7d043c6700 1 -- 172.21.2.117:0/3645042477 wait complete. 2021-02-10T11:53:46.245 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:53:46.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.577 7fa058cfe700 1 Processor -- start 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.577 7fa058cfe700 1 -- start start 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa058cfe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0540d15e0 0x7fa0540e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa058cfe700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0540e7580 con 0x7fa0540e7180 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa058cfe700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0540e77f0 con 0x7fa0541b07f0 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa058cfe700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0540e5ce0 con 0x7fa0540d15e0 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa052312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa0540e7180 0x7fa0540e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57752/0 (socket says 172.21.2.117:57752) 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa052312700 1 -- 172.21.2.117:0/4101023432 learned_addr learned my addr 172.21.2.117:0/4101023432 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa053314700 1 --2- 172.21.2.117:0/4101023432 >> v2:172.21.2.117:3300/0 conn(0x7fa0540d15e0 0x7fa0540e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa051b11700 1 -- 172.21.2.117:0/4101023432 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614361684 0 0) 0x7fa0440009b0 con 0x7fa0540e7180 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa051b11700 1 -- 172.21.2.117:0/4101023432 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa040002fe0 con 0x7fa0540e7180 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa053314700 1 -- 172.21.2.117:0/4101023432 >> v1:172.21.2.125:6789/0 conn(0x7fa0541b07f0 legacy=0x7fa0540e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa053314700 1 -- 172.21.2.117:0/4101023432 >> v1:172.21.2.117:6789/0 conn(0x7fa0540e7180 legacy=0x7fa0540e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa053314700 1 -- 172.21.2.117:0/4101023432 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0540d19e0 con 0x7fa0540d15e0 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.578 7fa053314700 1 --2- 172.21.2.117:0/4101023432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540d15e0 0x7fa0540e8560 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7fa04c006e30 tx=0x7fa04c009e60).ready entity=mon.0 client_cookie=1d3335afa546f47b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.579 7fa051b11700 1 -- 172.21.2.117:0/4101023432 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa04c006a50 con 0x7fa0540d15e0 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.579 7fa051b11700 1 -- 172.21.2.117:0/4101023432 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c00b300 con 0x7fa0540d15e0 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.579 7fa051b11700 1 -- 172.21.2.117:0/4101023432 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa04c00b7e0 con 0x7fa0540d15e0 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.579 7fa058cfe700 1 -- 172.21.2.117:0/4101023432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540d15e0 msgr2=0x7fa0540e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.579 7fa058cfe700 1 --2- 172.21.2.117:0/4101023432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540d15e0 0x7fa0540e8560 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 rx=0x7fa04c006e30 tx=0x7fa04c009e60).stop 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.580 7fa058cfe700 1 -- 172.21.2.117:0/4101023432 shutdown_connections 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.580 7fa058cfe700 1 --2- 172.21.2.117:0/4101023432 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540d15e0 0x7fa0540e8560 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.580 7fa058cfe700 1 -- 172.21.2.117:0/4101023432 >> 172.21.2.117:0/4101023432 conn(0x7fa0540ce500 msgr2=0x7fa0540e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.580 7fa058cfe700 1 -- 172.21.2.117:0/4101023432 shutdown_connections 2021-02-10T11:53:46.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.580 7fa058cfe700 1 -- 172.21.2.117:0/4101023432 wait complete. 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.594 7fa058cfe700 1 Processor -- start 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.594 7fa058cfe700 1 -- start start 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa05419c4e0 0x7fa0540bb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 0x7fa0541091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540adc60 con 0x7fa0540ace40 2021-02-10T11:53:46.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540beba0 con 0x7fa0540ad860 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540bedc0 con 0x7fa05419c4e0 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57508/0 (socket says 172.21.2.117:57508) 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 -- 172.21.2.117:0/2383036227 learned_addr learned my addr 172.21.2.117:0/2383036227 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa053314700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 0x7fa0541091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052b13700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa05419c4e0 0x7fa0540bb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 -- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa05419c4e0 msgr2=0x7fa0540bb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa05419c4e0 0x7fa0540bb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 -- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 msgr2=0x7fa0541091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 0x7fa0541091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 -- 172.21.2.117:0/2383036227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054109800 con 0x7fa0540ace40 2021-02-10T11:53:46.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa053314700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 0x7fa0541091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa052312700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7fa044001c60 tx=0x7fa04400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa044000e00 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04400acc0 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa0440045d0 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa054109800 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.595 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa054109800 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.596 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa044007f30 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.596 7fa04b7fe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 0x7fa03801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.596 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa044032500 con 0x7fa0540ace40 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.596 7fa052b13700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 0x7fa03801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:46.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.597 7fa052b13700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 0x7fa03801abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:46.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.627 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0540e77f0 con 0x7fa0540ace40 2021-02-10T11:53:46.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.630 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa044004140 con 0x7fa0540ace40 2021-02-10T11:53:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.757 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa0540e77f0 con 0x7fa0540ace40 2021-02-10T11:53:46.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.758 7fa04b7fe700 1 -- 172.21.2.117:0/2383036227 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v152) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa044007f30 con 0x7fa0540ace40 2021-02-10T11:53:46.761 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.759 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 msgr2=0x7fa03801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.759 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 0x7fa03801abc0 crc :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 msgr2=0x7fa0540ad240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 rx=0x7fa044001c60 tx=0x7fa04400a8d0).stop 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 shutdown_connections 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0380186c0 0x7fa03801abc0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0540ace40 0x7fa0540ad240 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa0540ad860 0x7fa0541091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 --2- 172.21.2.117:0/2383036227 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa05419c4e0 0x7fa0540bb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 >> 172.21.2.117:0/2383036227 conn(0x7fa0540c3f30 msgr2=0x7fa054111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 shutdown_connections 2021-02-10T11:53:46.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:46.760 7fa058cfe700 1 -- 172.21.2.117:0/2383036227 wait complete. 2021-02-10T11:53:46.771 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:53:46.772 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.182 7fbf62f86700 1 Processor -- start 2021-02-10T11:53:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.182 7fbf62f86700 1 -- start start 2021-02-10T11:53:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf62f86700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf5c1b6960 0x7fbf5c1b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf62f86700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0ede60 con 0x7fbf5c0d18e0 2021-02-10T11:53:47.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf62f86700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0eb9c0 con 0x7fbf5c0ef3f0 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf62f86700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c0ef7f0 con 0x7fbf5c1b6960 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf615a3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbf5c1b6960 0x7fbf5c1b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf60da2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbf5c0d18e0 0x7fbf5c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57764/0 (socket says 172.21.2.117:57764) 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf60da2700 1 -- 172.21.2.117:0/1430874052 learned_addr learned my addr 172.21.2.117:0/1430874052 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf615a3700 1 -- 172.21.2.117:0/1430874052 >> v1:172.21.2.125:6789/0 conn(0x7fbf5c0ef3f0 legacy=0x7fbf5c0eb1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf615a3700 1 -- 172.21.2.117:0/1430874052 >> v1:172.21.2.117:6789/0 conn(0x7fbf5c0d18e0 legacy=0x7fbf5c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf615a3700 1 -- 172.21.2.117:0/1430874052 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0d1d20 con 0x7fbf5c1b6960 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.183 7fbf615a3700 1 --2- 172.21.2.117:0/1430874052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c1b6960 0x7fbf5c1b8d70 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fbf54006e30 tx=0x7fbf54009e60).ready entity=mon.0 client_cookie=293dcca60b96c796 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf5b7fe700 1 -- 172.21.2.117:0/1430874052 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf54006a50 con 0x7fbf5c1b6960 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf5b7fe700 1 -- 172.21.2.117:0/1430874052 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5400b300 con 0x7fbf5c1b6960 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf5b7fe700 1 -- 172.21.2.117:0/1430874052 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf5400b820 con 0x7fbf5c1b6960 2021-02-10T11:53:47.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 -- 172.21.2.117:0/1430874052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c1b6960 msgr2=0x7fbf5c1b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 --2- 172.21.2.117:0/1430874052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c1b6960 0x7fbf5c1b8d70 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 rx=0x7fbf54006e30 tx=0x7fbf54009e60).stop 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 -- 172.21.2.117:0/1430874052 shutdown_connections 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 --2- 172.21.2.117:0/1430874052 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c1b6960 0x7fbf5c1b8d70 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 -- 172.21.2.117:0/1430874052 >> 172.21.2.117:0/1430874052 conn(0x7fbf5c0ced60 msgr2=0x7fbf5c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 -- 172.21.2.117:0/1430874052 shutdown_connections 2021-02-10T11:53:47.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.184 7fbf62f86700 1 -- 172.21.2.117:0/1430874052 wait complete. 2021-02-10T11:53:47.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 Processor -- start 2021-02-10T11:53:47.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 -- start start 2021-02-10T11:53:47.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:47.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c099780 0x7fbf5c1c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:47.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c1c68d0 0x7fbf5c0edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.199 7fbf62f86700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c099200 con 0x7fbf5c0a3200 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf62f86700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0f2530 con 0x7fbf5c099780 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf62f86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0f2710 con 0x7fbf5c1c68d0 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57518/0 (socket says 172.21.2.117:57518) 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 -- 172.21.2.117:0/3276887197 learned_addr learned my addr 172.21.2.117:0/3276887197 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 -- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c1c68d0 msgr2=0x7fbf5c0edd00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c1c68d0 0x7fbf5c0edd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 -- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c099780 msgr2=0x7fbf5c1c62b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c099780 0x7fbf5c1c62b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 -- 172.21.2.117:0/3276887197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c099b80 con 0x7fbf5c0a3200 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf60da2700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fbf44001400 tx=0x7fbf4400a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:47.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.200 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4400acf0 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf440043a0 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf440048c0 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c099b80 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0ee610 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf44004580 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf597fa700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 0x7fbf4001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.201 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf44032690 con 0x7fbf5c0a3200 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.202 7fbf5bfff700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 0x7fbf4001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:47.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.203 7fbf5bfff700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 0x7fbf4001abc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:47.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.232 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c086140 con 0x7fbf5c0a3200 2021-02-10T11:53:47.237 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.235 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf44004790 con 0x7fbf5c0a3200 2021-02-10T11:53:47.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.362 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf5c086140 con 0x7fbf5c0a3200 2021-02-10T11:53:47.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.362 7fbf597fa700 1 -- 172.21.2.117:0/3276887197 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2102 (secure 0 0 0) 0x7fbf44004580 con 0x7fbf5c0a3200 2021-02-10T11:53:47.366 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:47.366 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":128,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:53:39.473419","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8737},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8737":{"gid":8737,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/277081047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":277081047},{"type":"v1","addr":"172.21.2.125:6835","nonce":277081047}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:53:47.366 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 128 2021-02-10T11:53:47.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.364 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 msgr2=0x7fbf4001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:47.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.364 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 0x7fbf4001abc0 crc :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:47.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.364 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 msgr2=0x7fbf5c0f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:47.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.364 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7fbf44001400 tx=0x7fbf4400a900).stop 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 shutdown_connections 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf5c1c68d0 0x7fbf5c0edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf5c099780 0x7fbf5c1c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf400186c0 0x7fbf4001abc0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 --2- 172.21.2.117:0/3276887197 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf5c0a3200 0x7fbf5c0f29d0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 >> 172.21.2.117:0/3276887197 conn(0x7fbf5c0f3230 msgr2=0x7fbf5c0b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 shutdown_connections 2021-02-10T11:53:47.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:47.365 7fbf62f86700 1 -- 172.21.2.117:0/3276887197 wait complete. 2021-02-10T11:53:47.577 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:53:37.468914', 'modified': '2021-02-10 11:53:39.473419', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8737}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8737': {'gid': 8737, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/277081047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 277081047}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 277081047}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:53:47.577 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:53:47.578 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:53:47.976 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8737, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.503054155999999, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 128, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 152, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 152, 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.155314465 2021-02-10T11:53:47.977 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:53:47.981 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8737, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.503054155999999, "mdsmap_epoch": 128, "osdmap_epoch": 152, "osdmap_epoch_barrier": 152, "uptime": 16.155314465 } 2021-02-10T11:53:47.981 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:53:47.981 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:53:47.981 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:53:47.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:47.981 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:48.317 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:48.661 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:48.661 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:49.003 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:49.004 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:49.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:49.005 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:49.339 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:53:49.340 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:49.342 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:49.686 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:49.686 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:49.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:53:49.750 7f98f8f53f80 -1 init, newargv = 0x56442caaff70 newargc=9 2021-02-10T11:53:49.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[176898]: starting ceph client 2021-02-10T11:53:49.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[176898]: starting fuse 2021-02-10T11:53:50.028 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:50.028 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:50.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:50.030 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:50.364 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:50.364 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:53:50.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:50.364 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:53:50.364 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:53:50.364 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:53:50.365 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:53:50.366 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:53:50.366 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:53:50.366 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:50.366 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:53:50.366 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:53:50.728 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.176898.asok 2021-02-10T11:53:50.731 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.176898.asok 2021-02-10T11:53:50.731 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:50.731 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.176898.asok status 2021-02-10T11:53:51.121 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "176898", 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:53:51.122 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12219, 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12219 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:51.123 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1490004070 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1490004070 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12219 172.21.2.117:0/1490004070", 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1490004070", 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 128, 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 152, 2021-02-10T11:53:51.124 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:53:51.125 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:53:51.125 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:51.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:51.126 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:51.126 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:51.462 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:51.463 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:53:51.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:51.463 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:53:51.809 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 Processor -- start 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 -- start start 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fedb80ef3f0 0x7fedb80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb80ede60 con 0x7fedb80eb7f0 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb80d1ce0 con 0x7fedb80d18e0 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedbcfb8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb80ef7f0 con 0x7fedb80ef3f0 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.218 7fedb759e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fedb80eb7f0 0x7fedb81b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57788/0 (socket says 172.21.2.117:57788) 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb759e700 1 -- 172.21.2.117:0/2917817976 learned_addr learned my addr 172.21.2.117:0/2917817976 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878840762 0 0) 0x7fedb0001500 con 0x7fedb80eb7f0 2021-02-10T11:53:52.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0002fe0 con 0x7fedb80eb7f0 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb659c700 1 --2- 172.21.2.117:0/2917817976 >> v2:172.21.2.125:3300/0 conn(0x7fedb80ef3f0 0x7fedb80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3231180113 0 0) 0x7fedb0001460 con 0x7fedb80eb7f0 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 >> v2:172.21.2.125:3300/0 conn(0x7fedb80ef3f0 msgr2=0x7fedb80eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 --2- 172.21.2.117:0/2917817976 >> v2:172.21.2.125:3300/0 conn(0x7fedb80ef3f0 0x7fedb80eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 >> v1:172.21.2.125:6789/0 conn(0x7fedb80d18e0 legacy=0x7fedb80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb81bb180 con 0x7fedb80eb7f0 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.219 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fedb0001980 con 0x7fedb80eb7f0 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 >> v1:172.21.2.117:6789/0 conn(0x7fedb80eb7f0 legacy=0x7fedb81b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda0003a60 0x7feda0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 0x7feda0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda0009590 0x7feda000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda000c300 con 0x7feda0006790 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda000c520 con 0x7feda0003a60 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda0050750 con 0x7feda0009590 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 0x7feda0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb759e700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda0009590 0x7feda000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb6d9d700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda0003a60 0x7feda0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 -- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda0009590 msgr2=0x7feda000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7feda0009590 0x7feda000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 -- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda0003a60 msgr2=0x7feda0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda0003a60 0x7feda0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 -- 172.21.2.117:0/2917817976 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0050980 con 0x7feda0006790 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb6d9d700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7feda0003a60 0x7feda0005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:53:52.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb759e700 1 -- 172.21.2.117:0/2917817976 reap_dead start 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.220 7fedb659c700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 0x7feda0008c90 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 rx=0x7fedac00c870 tx=0x7fedac00cc10).ready entity=mon.0 client_cookie=ceb10f0e59e62a32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.221 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac0090d0 con 0x7feda0006790 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.221 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac009350 con 0x7feda0006790 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.221 7fedb5d9b700 1 -- 172.21.2.117:0/2917817976 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac009870 con 0x7feda0006790 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.221 7fedbcfb8700 1 -- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 msgr2=0x7feda0008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.221 7fedbcfb8700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 0x7feda0008c90 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 rx=0x7fedac00c870 tx=0x7fedac00cc10).stop 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.222 7fedbcfb8700 1 -- 172.21.2.117:0/2917817976 shutdown_connections 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.222 7fedbcfb8700 1 --2- 172.21.2.117:0/2917817976 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7feda0006790 0x7feda0008c90 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.222 7fedbcfb8700 1 -- 172.21.2.117:0/2917817976 >> 172.21.2.117:0/2917817976 conn(0x7fedb80ced60 msgr2=0x7fedb80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.222 7fedbcfb8700 1 -- 172.21.2.117:0/2917817976 shutdown_connections 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.222 7fedbcfb8700 1 -- 172.21.2.117:0/2917817976 wait complete. 2021-02-10T11:53:52.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 Processor -- start 2021-02-10T11:53:52.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 -- start start 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fedb80b59a0 0x7fedb808c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fedb81a3480 0x7fedb80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb80b61a0 con 0x7fedb80b5da0 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb808c8b0 con 0x7fedb81a3480 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.236 7fedbcfb8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb80ba1c0 con 0x7fedb80b59a0 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57548/0 (socket says 172.21.2.117:57548) 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 -- 172.21.2.117:0/705035634 learned_addr learned my addr 172.21.2.117:0/705035634 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 -- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fedb80b59a0 msgr2=0x7fedb808c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fedb80b59a0 0x7fedb808c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 -- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fedb81a3480 msgr2=0x7fedb80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fedb81a3480 0x7fedb80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 -- 172.21.2.117:0/705035634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb81a3880 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedb659c700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7fedac0065a0 tx=0x7fedac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac006240 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac006740 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fedac002210 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb81a3880 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.237 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb81a3880 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.238 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fedac0052e0 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.238 7feda77fe700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 0x7fed9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.238 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fedac032930 con 0x7fedb80b5da0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.238 7fedb6d9d700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 0x7fed9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.238 7fedb6d9d700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 0x7fed9c01abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.268 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb80ede60 con 0x7fedb80b5da0 2021-02-10T11:53:52.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.271 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fedac0052e0 con 0x7fedb80b5da0 2021-02-10T11:53:52.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.395 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedb80ede60 con 0x7fedb80b5da0 2021-02-10T11:53:52.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.396 7feda77fe700 1 -- 172.21.2.117:0/705035634 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fedac006740 con 0x7fedb80b5da0 2021-02-10T11:53:52.398 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:52.399 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":152,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:53:38.470991","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:53:35.441170","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":33,"pool_name":"cephfs_data","create_time":"2021-02-10 11:53:36.462661","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 msgr2=0x7fed9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 0x7fed9c01abc0 crc :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 msgr2=0x7fedb81a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 rx=0x7fedac0065a0 tx=0x7fedac00b960).stop 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 shutdown_connections 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fed9c0186c0 0x7fed9c01abc0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fedb80b5da0 0x7fedb81a2e60 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fedb81a3480 0x7fedb80b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 --2- 172.21.2.117:0/705035634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fedb80b59a0 0x7fedb808c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.397 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 >> 172.21.2.117:0/705035634 conn(0x7fedb80bb5e0 msgr2=0x7fedb8090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:52.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.398 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 shutdown_connections 2021-02-10T11:53:52.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.398 7fedbcfb8700 1 -- 172.21.2.117:0/705035634 wait complete. 2021-02-10T11:53:52.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:53:35.441170', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:53:36.462661', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:53:52.409 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:53:52.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 Processor -- start 2021-02-10T11:53:52.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 -- start start 2021-02-10T11:53:52.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 0x7f2d140eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2d140eb830 0x7f2d141b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d140ef7f0 con 0x7f2d140d18f0 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d140eded0 con 0x7f2d140eb830 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d1aacd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140d1cf0 con 0x7f2d140ef3f0 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d190ea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2d140eb830 0x7f2d141b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d190ea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2d140eb830 0x7f2d141b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57556/0 (socket says 172.21.2.117:57556) 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.820 7f2d190ea700 1 -- 172.21.2.117:0/295394023 learned_addr learned my addr 172.21.2.117:0/295394023 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d13fff700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 0x7f2d140eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d190ea700 1 -- 172.21.2.117:0/295394023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 msgr2=0x7f2d140eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d190ea700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 0x7f2d140eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d190ea700 1 -- 172.21.2.117:0/295394023 >> v1:172.21.2.125:6789/0 conn(0x7f2d140d18f0 legacy=0x7f2d140eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d190ea700 1 -- 172.21.2.117:0/295394023 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d141b8fd0 con 0x7f2d140eb830 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d13fff700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 0x7f2d140eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d190ea700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140eb830 0x7f2d141b79a0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f2d0c006f00 tx=0x7f2d0c009f10).ready entity=mon.0 client_cookie=46e9725b75013049 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d137fe700 1 -- 172.21.2.117:0/295394023 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d0c00b3b0 con 0x7f2d140eb830 2021-02-10T11:53:52.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d137fe700 1 -- 172.21.2.117:0/295394023 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0c00b630 con 0x7f2d140eb830 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d137fe700 1 -- 172.21.2.117:0/295394023 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d0c00bb50 con 0x7f2d140eb830 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d1aacd700 1 -- 172.21.2.117:0/295394023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140eb830 msgr2=0x7f2d141b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.821 7f2d1aacd700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140eb830 0x7f2d141b79a0 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 rx=0x7f2d0c006f00 tx=0x7f2d0c009f10).stop 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 -- 172.21.2.117:0/295394023 shutdown_connections 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140eb830 0x7f2d141b79a0 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 --2- 172.21.2.117:0/295394023 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d140ef3f0 0x7f2d140eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 -- 172.21.2.117:0/295394023 >> 172.21.2.117:0/295394023 conn(0x7f2d140cf060 msgr2=0x7f2d140cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 -- 172.21.2.117:0/295394023 shutdown_connections 2021-02-10T11:53:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.822 7f2d1aacd700 1 -- 172.21.2.117:0/295394023 wait complete. 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 Processor -- start 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 -- start start 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d1407e7d0 0x7f2d1407ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d141a2e50 0x7f2d141a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1407f890 con 0x7f2d140a67e0 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1407f310 con 0x7f2d141a2e50 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d1aacd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d141a3870 con 0x7f2d1407e7d0 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d188e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d188e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57562/0 (socket says 172.21.2.117:57562) 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.837 7f2d188e9700 1 -- 172.21.2.117:0/740726265 learned_addr learned my addr 172.21.2.117:0/740726265 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 -- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d1407e7d0 msgr2=0x7f2d1407ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d1407e7d0 0x7f2d1407ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 -- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d141a2e50 msgr2=0x7f2d141a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d141a2e50 0x7f2d141a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 -- 172.21.2.117:0/740726265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d141a3a90 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d188e9700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f2cfc00b5b0 tx=0x7f2cfc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cfc00bd40 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc006240 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cfc005720 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d141a3a90 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.838 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1409f8e0 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.839 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cfc0054a0 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.839 7f2d117fa700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 0x7f2cf801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.839 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2cfc0325b0 con 0x7f2d140a67e0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.839 7f2d13fff700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 0x7f2cf801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.840 7f2d13fff700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 0x7f2cf801abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:52.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.870 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d140ede60 con 0x7f2d140a67e0 2021-02-10T11:53:52.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:52.873 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cfc030d90 con 0x7f2d140a67e0 2021-02-10T11:53:53.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.000 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2d140ede60 con 0x7f2d140a67e0 2021-02-10T11:53:53.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.000 7f2d117fa700 1 -- 172.21.2.117:0/740726265 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v152) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2cfc00bd40 con 0x7f2d140a67e0 2021-02-10T11:53:53.003 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:53.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.002 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 msgr2=0x7f2cf801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.002 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 0x7f2cf801abc0 crc :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.002 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 msgr2=0x7f2d140a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.002 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 rx=0x7f2cfc00b5b0 tx=0x7f2cfc00b950).stop 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 shutdown_connections 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d141a2e50 0x7f2d141a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d1407e7d0 0x7f2d1407ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cf80186c0 0x7f2cf801abc0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 --2- 172.21.2.117:0/740726265 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d140a67e0 0x7f2d140a5f80 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 >> 172.21.2.117:0/740726265 conn(0x7f2d1409e660 msgr2=0x7f2d140bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 shutdown_connections 2021-02-10T11:53:53.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.003 7f2d1aacd700 1 -- 172.21.2.117:0/740726265 wait complete. 2021-02-10T11:53:53.013 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.425 7f9043fc6700 1 Processor -- start 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.425 7f9043fc6700 1 -- start start 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f9043fc6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f903c0d18f0 0x7f903c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f9043fc6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f903c0eb040 0x7f903c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f9043fc6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c0ede60 con 0x7f903c0ee3e0 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f9043fc6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c0ebad0 con 0x7f903c0eb040 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f9043fc6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c0d1cf0 con 0x7f903c0d18f0 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f903c0eb040 0x7f903c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:53.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f903c0eb040 0x7f903c0eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57570/0 (socket says 172.21.2.117:57570) 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 -- 172.21.2.117:0/2828999545 learned_addr learned my addr 172.21.2.117:0/2828999545 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 -- 172.21.2.117:0/2828999545 >> v2:172.21.2.125:3300/0 conn(0x7f903c0d18f0 msgr2=0x7f903c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 --2- 172.21.2.117:0/2828999545 >> v2:172.21.2.125:3300/0 conn(0x7f903c0d18f0 0x7f903c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 -- 172.21.2.117:0/2828999545 >> v1:172.21.2.125:6789/0 conn(0x7f903c0ee3e0 legacy=0x7f903c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.426 7f90415e1700 1 -- 172.21.2.117:0/2828999545 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0eefd0 con 0x7f903c0eb040 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f90415e1700 1 --2- 172.21.2.117:0/2828999545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c0eb040 0x7f903c0eb440 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f902c006700 tx=0x7f902c009b80).ready entity=mon.0 client_cookie=744b8c2b4692869c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f9040de0700 1 -- 172.21.2.117:0/2828999545 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f902c00c0c0 con 0x7f903c0eb040 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f9040de0700 1 -- 172.21.2.117:0/2828999545 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c00c440 con 0x7f903c0eb040 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f9040de0700 1 -- 172.21.2.117:0/2828999545 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f902c00c960 con 0x7f903c0eb040 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f9043fc6700 1 -- 172.21.2.117:0/2828999545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c0eb040 msgr2=0x7f903c0eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.427 7f9043fc6700 1 --2- 172.21.2.117:0/2828999545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c0eb040 0x7f903c0eb440 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f902c006700 tx=0x7f902c009b80).stop 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 -- 172.21.2.117:0/2828999545 shutdown_connections 2021-02-10T11:53:53.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 --2- 172.21.2.117:0/2828999545 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c0eb040 0x7f903c0eb440 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 --2- 172.21.2.117:0/2828999545 >> v2:172.21.2.125:3300/0 conn(0x7f903c0d18f0 0x7f903c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 -- 172.21.2.117:0/2828999545 >> 172.21.2.117:0/2828999545 conn(0x7f903c0cf060 msgr2=0x7f903c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:53.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 -- 172.21.2.117:0/2828999545 shutdown_connections 2021-02-10T11:53:53.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.428 7f9043fc6700 1 -- 172.21.2.117:0/2828999545 wait complete. 2021-02-10T11:53:53.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 Processor -- start 2021-02-10T11:53:53.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 -- start start 2021-02-10T11:53:53.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f903c1a4240 0x7f903c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f903c09ed70 0x7f903c09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c09fb90 con 0x7f903c09f790 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0dc640 con 0x7f903c1a4240 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f9043fc6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c097270 con 0x7f903c09ed70 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f90425e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f90425e3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57576/0 (socket says 172.21.2.117:57576) 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.443 7f90425e3700 1 -- 172.21.2.117:0/382502832 learned_addr learned my addr 172.21.2.117:0/382502832 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 -- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f903c09ed70 msgr2=0x7f903c09f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f903c09ed70 0x7f903c09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 -- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f903c1a4240 msgr2=0x7f903c1a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f903c1a4240 0x7f903c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 -- 172.21.2.117:0/382502832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0b8780 con 0x7f903c09f790 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90425e3700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 rx=0x7f9034009940 tx=0x7f903400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:53.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9034000f90 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034000d30 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f9043fc6700 1 -- 172.21.2.117:0/382502832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c0b8780 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f903400c570 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.444 7f9043fc6700 1 -- 172.21.2.117:0/382502832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c0b8780 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.445 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9034001220 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.445 7f90327fc700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 0x7f902401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.445 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9034032af0 con 0x7f903c09f790 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.445 7f9041de2700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 0x7f902401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:53.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.445 7f9041de2700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 0x7f902401ac10 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:53.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.476 7f9043fc6700 1 -- 172.21.2.117:0/382502832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c0daff0 con 0x7f903c09f790 2021-02-10T11:53:53.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.478 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f903400be30 con 0x7f903c09f790 2021-02-10T11:53:53.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.605 7f9043fc6700 1 -- 172.21.2.117:0/382502832 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f903c0daff0 con 0x7f903c09f790 2021-02-10T11:53:53.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.605 7f90327fc700 1 -- 172.21.2.117:0/382502832 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v152) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9034000d30 con 0x7f903c09f790 2021-02-10T11:53:53.608 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 msgr2=0x7f902401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 0x7f902401ac10 crc :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 msgr2=0x7f903c0b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 rx=0x7f9034009940 tx=0x7f903400bcf0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 shutdown_connections 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9024018730 0x7f902401ac10 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f903c09f790 0x7f903c0b8160 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f903c09ed70 0x7f903c09f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 --2- 172.21.2.117:0/382502832 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f903c1a4240 0x7f903c1a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 >> 172.21.2.117:0/382502832 conn(0x7f903c087090 msgr2=0x7f903c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 shutdown_connections 2021-02-10T11:53:53.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:53.607 7f9043fc6700 1 -- 172.21.2.117:0/382502832 wait complete. 2021-02-10T11:53:53.618 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:53:54.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 Processor -- start 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 -- start start 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0eb830 0x7f2bcc1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc0ef7f0 con 0x7f2bcc0d18f0 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc0ede60 con 0x7f2bcc0ef3f0 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.030 7f2bd0b37700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc0d1cf0 con 0x7f2bcc0eb830 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57584/0 (socket says 172.21.2.117:57584) 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 -- 172.21.2.117:0/989882558 learned_addr learned my addr 172.21.2.117:0/989882558 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 -- 172.21.2.117:0/989882558 >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0eb830 msgr2=0x7f2bcc1b79a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:54.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 --2- 172.21.2.117:0/989882558 >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0eb830 0x7f2bcc1b79a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 -- 172.21.2.117:0/989882558 >> v1:172.21.2.125:6789/0 conn(0x7f2bcc0d18f0 legacy=0x7f2bcc0eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 -- 172.21.2.117:0/989882558 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc1b8fd0 con 0x7f2bcc0ef3f0 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bca05d700 1 --2- 172.21.2.117:0/989882558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f2bbc006700 tx=0x7f2bbc009b80).ready entity=mon.0 client_cookie=f6ed88a66d9a830d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bc985c700 1 -- 172.21.2.117:0/989882558 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bbc00c0c0 con 0x7f2bcc0ef3f0 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bc985c700 1 -- 172.21.2.117:0/989882558 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc00c340 con 0x7f2bcc0ef3f0 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bc985c700 1 -- 172.21.2.117:0/989882558 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bbc00c860 con 0x7f2bcc0ef3f0 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bd0b37700 1 -- 172.21.2.117:0/989882558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0ef3f0 msgr2=0x7f2bcc0eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.031 7f2bd0b37700 1 --2- 172.21.2.117:0/989882558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f2bbc006700 tx=0x7f2bbc009b80).stop 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 -- 172.21.2.117:0/989882558 shutdown_connections 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 --2- 172.21.2.117:0/989882558 >> v2:172.21.2.125:3300/0 conn(0x7f2bcc0eb830 0x7f2bcc1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 --2- 172.21.2.117:0/989882558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0ef3f0 0x7f2bcc0eb210 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 -- 172.21.2.117:0/989882558 >> 172.21.2.117:0/989882558 conn(0x7f2bcc0cf060 msgr2=0x7f2bcc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:54.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 -- 172.21.2.117:0/989882558 shutdown_connections 2021-02-10T11:53:54.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.032 7f2bd0b37700 1 -- 172.21.2.117:0/989882558 wait complete. 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.046 7f2bd0b37700 1 Processor -- start 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.046 7f2bd0b37700 1 -- start start 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc07c7c0 0x7f2bcc1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc1a3930 0x7f2bcc0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc07cca0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc18fb70 con 0x7f2bcc07c7c0 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bd0b37700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0dc6d0 con 0x7f2bcc1a3930 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bcb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bcb05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57590/0 (socket says 172.21.2.117:57590) 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.047 7f2bcb05f700 1 -- 172.21.2.117:0/1270339547 learned_addr learned my addr 172.21.2.117:0/1270339547 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 -- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc1a3930 msgr2=0x7f2bcc0a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc1a3930 0x7f2bcc0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 -- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc07c7c0 msgr2=0x7f2bcc1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc07c7c0 0x7f2bcc1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 -- 172.21.2.117:0/1270339547 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc07d220 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bcb05f700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7f2bc4009960 tx=0x7f2bc400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc4003030 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4001310 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2bc4001830 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc07d220 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.048 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc07d7b0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.049 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2bc40014f0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.049 7f2bc37fe700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 0x7f2bb001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.049 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2bc4032ab0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.049 7f2bca85e700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 0x7f2bb001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.050 7f2bca85e700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 0x7f2bb001abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.080 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc0daff0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.083 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2bc4000f40 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.209 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f2bcc0ede60 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.209 7f2bc37fe700 1 -- 172.21.2.117:0/1270339547 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v152) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2bc401f6a0 con 0x7f2bcc0a13a0 2021-02-10T11:53:54.212 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:53:54.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 msgr2=0x7f2bb001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 0x7f2bb001abc0 crc :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 msgr2=0x7f2bcc0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 rx=0x7f2bc4009960 tx=0x7f2bc400bcf0).stop 2021-02-10T11:53:54.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 shutdown_connections 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2bb00186c0 0x7f2bb001abc0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2bcc0a13a0 0x7f2bcc0a17a0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2bcc1a3930 0x7f2bcc0a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.211 7f2bd0b37700 1 --2- 172.21.2.117:0/1270339547 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2bcc07c7c0 0x7f2bcc1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.212 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 >> 172.21.2.117:0/1270339547 conn(0x7f2bcc087090 msgr2=0x7f2bcc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.212 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 shutdown_connections 2021-02-10T11:53:54.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.212 7f2bd0b37700 1 -- 172.21.2.117:0/1270339547 wait complete. 2021-02-10T11:53:54.222 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:53:54.223 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.630 7f43806bf700 1 Processor -- start 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.630 7f43806bf700 1 -- start start 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f43806bf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43780d18e0 0x7f43780ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f43806bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780eb000 0x7f43780eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f43806bf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43780ede60 con 0x7f43780ee3e0 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f43806bf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43780eba90 con 0x7f43780d18e0 2021-02-10T11:53:54.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f43806bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43780d1ce0 con 0x7f43780eb000 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43780d18e0 0x7f43780ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f43780d18e0 0x7f43780ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57596/0 (socket says 172.21.2.117:57596) 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 -- 172.21.2.117:0/1582265524 learned_addr learned my addr 172.21.2.117:0/1582265524 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 -- 172.21.2.117:0/1582265524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780eb000 msgr2=0x7f43780eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 --2- 172.21.2.117:0/1582265524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780eb000 0x7f43780eb400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 -- 172.21.2.117:0/1582265524 >> v1:172.21.2.125:6789/0 conn(0x7f43780ee3e0 legacy=0x7f43780ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.631 7f437e4db700 1 -- 172.21.2.117:0/1582265524 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43780eefd0 con 0x7f43780d18e0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f437e4db700 1 --2- 172.21.2.117:0/1582265524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43780d18e0 0x7f43780ea9e0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7f4370006a80 tx=0x7f4370009f00).ready entity=mon.0 client_cookie=2f2d52c8d4777ccb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f437d4d9700 1 -- 172.21.2.117:0/1582265524 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f437000b5c0 con 0x7f43780d18e0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f437d4d9700 1 -- 172.21.2.117:0/1582265524 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437000b8c0 con 0x7f43780d18e0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f437d4d9700 1 -- 172.21.2.117:0/1582265524 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f437000bda0 con 0x7f43780d18e0 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f43806bf700 1 -- 172.21.2.117:0/1582265524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43780d18e0 msgr2=0x7f43780ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.632 7f43806bf700 1 --2- 172.21.2.117:0/1582265524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43780d18e0 0x7f43780ea9e0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7f4370006a80 tx=0x7f4370009f00).stop 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 -- 172.21.2.117:0/1582265524 shutdown_connections 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 --2- 172.21.2.117:0/1582265524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780eb000 0x7f43780eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 --2- 172.21.2.117:0/1582265524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43780d18e0 0x7f43780ea9e0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 -- 172.21.2.117:0/1582265524 >> 172.21.2.117:0/1582265524 conn(0x7f43780ced60 msgr2=0x7f43780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 -- 172.21.2.117:0/1582265524 shutdown_connections 2021-02-10T11:53:54.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.633 7f43806bf700 1 -- 172.21.2.117:0/1582265524 wait complete. 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 Processor -- start 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 -- start start 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780e9560 0x7f43780e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43780e9960 0x7f43780e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4378196810 con 0x7f4378196390 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4378098ad0 con 0x7f43780e9960 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f43806bf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43780e8a40 con 0x7f43780e9560 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f437e4db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f437e4db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57604/0 (socket says 172.21.2.117:57604) 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.648 7f437e4db700 1 -- 172.21.2.117:0/1178759014 learned_addr learned my addr 172.21.2.117:0/1178759014 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:54.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 -- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780e9560 msgr2=0x7f43780e7e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780e9560 0x7f43780e7e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 -- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43780e9960 msgr2=0x7f43780e8420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43780e9960 0x7f43780e8420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 -- 172.21.2.117:0/1178759014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43780e8c60 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f437e4db700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 rx=0x7f4370009450 tx=0x7f437000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4370006e40 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437000be60 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f43806bf700 1 -- 172.21.2.117:0/1178759014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43780e8c60 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f43806bf700 1 -- 172.21.2.117:0/1178759014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437819cd70 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.649 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f437000c4e0 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.650 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4370005660 con 0x7f4378196390 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.650 7f436effd700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 0x7f436001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:54.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.650 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4370032c70 con 0x7f4378196390 2021-02-10T11:53:54.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.650 7f437dcda700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 0x7f436001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:54.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.651 7f437dcda700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 0x7f436001ac10 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:54.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.681 7f43806bf700 1 -- 172.21.2.117:0/1178759014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43780e8c60 con 0x7f4378196390 2021-02-10T11:53:54.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.684 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4370000d30 con 0x7f4378196390 2021-02-10T11:53:54.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.812 7f43806bf700 1 -- 172.21.2.117:0/1178759014 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43780e8c60 con 0x7f4378196390 2021-02-10T11:53:54.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.812 7f436effd700 1 -- 172.21.2.117:0/1178759014 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f4370005660 con 0x7f4378196390 2021-02-10T11:53:54.815 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:53:54.815 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":128,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:53:39.473419","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8737},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8737":{"gid":8737,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/277081047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":277081047},{"type":"v1","addr":"172.21.2.125:6835","nonce":277081047}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:53:54.816 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 128 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 msgr2=0x7f436001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 0x7f436001ac10 crc :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 msgr2=0x7f43780e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 rx=0x7f4370009450 tx=0x7f437000bd20).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 shutdown_connections 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43780e9960 0x7f43780e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43780e9560 0x7f43780e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4360018730 0x7f436001ac10 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 --2- 172.21.2.117:0/1178759014 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4378196390 0x7f43780e8f40 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:54.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 >> 172.21.2.117:0/1178759014 conn(0x7f437819f610 msgr2=0x7f43781a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 shutdown_connections 2021-02-10T11:53:54.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:54.814 7f43806bf700 1 -- 172.21.2.117:0/1178759014 wait complete. 2021-02-10T11:53:55.026 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:53:37.468914', 'modified': '2021-02-10 11:53:39.473419', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8737}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8737': {'gid': 8737, 'name': 'b', 'rank': 0, 'incarnation': 126, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/277081047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 277081047}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 277081047}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:53:55.027 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:53:55.027 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:53:55.427 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:53:55.427 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:53:55.427 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8737, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 17.953531653999999, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 128, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 152, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 152, 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.605791794999998 2021-02-10T11:53:55.428 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:53:55.431 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8737, "want_state": "up:active", "state": "up:active", "rank_uptime": 17.953531653999999, "mdsmap_epoch": 128, "osdmap_epoch": 152, "osdmap_epoch_barrier": 152, "uptime": 23.605791794999998 } 2021-02-10T11:53:55.431 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:53:55.431 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:53:55.432 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:53:55.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:55.432 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:55.768 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:56.113 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:56.114 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:56.458 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:56.458 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:56.459 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:56.459 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:56.794 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:56.794 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:53:56.794 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:56.797 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:53:57.142 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:53:57.142 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:53:57.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[177539]: starting ceph client 2021-02-10T11:53:57.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:53:57.206 7fe7523b7f80 -1 init, newargv = 0x560ca3e53f30 newargc=9 2021-02-10T11:53:57.214 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[177539]: starting fuse 2021-02-10T11:53:57.485 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:53:57.486 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:53:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:53:57.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:57.487 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:53:57.823 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:53:57.823 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:53:57.823 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:53:57.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:53:57.824 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:53:57.825 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:53:58.188 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.177539.asok 2021-02-10T11:53:58.190 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.177539.asok 2021-02-10T11:53:58.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:58.191 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.177539.asok status 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:53:58.582 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "177539", 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12255, 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12255 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:58.583 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 697719067 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 697719067 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12255 172.21.2.117:0/697719067", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/697719067", 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:53:58.584 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 128, 2021-02-10T11:53:58.585 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 152, 2021-02-10T11:53:58.585 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:53:58.585 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:53:58.585 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:53:58.586 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:58.586 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:53:58.586 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:58.923 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:53:58.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:53:58.944 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:53:58.944 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:53:59.293 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:53:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.705 7f7a43c46700 1 Processor -- start 2021-02-10T11:53:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.705 7f7a43c46700 1 -- start start 2021-02-10T11:53:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a43c46700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a43c46700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c085450 0x7f7a3c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a43c46700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c085930 con 0x7f7a3c0e5240 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a43c46700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c0e5cc0 con 0x7f7a3c0e4e40 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a43c46700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c084840 con 0x7f7a3c085450 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a41a62700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a41a62700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57622/0 (socket says 172.21.2.117:57622) 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a41a62700 1 -- 172.21.2.117:0/372795150 learned_addr learned my addr 172.21.2.117:0/372795150 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.706 7f7a42263700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c085450 0x7f7a3c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a41a62700 1 -- 172.21.2.117:0/372795150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c085450 msgr2=0x7f7a3c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a41a62700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c085450 0x7f7a3c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a41a62700 1 -- 172.21.2.117:0/372795150 >> v1:172.21.2.125:6789/0 conn(0x7f7a3c0e5240 legacy=0x7f7a3c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a41a62700 1 -- 172.21.2.117:0/372795150 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c1b0810 con 0x7f7a3c0e4e40 2021-02-10T11:53:59.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a41a62700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 rx=0x7f7a30006700 tx=0x7f7a30009b80).ready entity=mon.0 client_cookie=b2cc09a397ec385a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a40a60700 1 -- 172.21.2.117:0/372795150 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a3000c0c0 con 0x7f7a3c0e4e40 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a40a60700 1 -- 172.21.2.117:0/372795150 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3000c340 con 0x7f7a3c0e4e40 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a40a60700 1 -- 172.21.2.117:0/372795150 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a3000c860 con 0x7f7a3c0e4e40 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a43c46700 1 -- 172.21.2.117:0/372795150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0e4e40 msgr2=0x7f7a3c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.707 7f7a43c46700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 rx=0x7f7a30006700 tx=0x7f7a30009b80).stop 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 -- 172.21.2.117:0/372795150 shutdown_connections 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c085450 0x7f7a3c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 --2- 172.21.2.117:0/372795150 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0e4e40 0x7f7a3c1b01f0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 -- 172.21.2.117:0/372795150 >> 172.21.2.117:0/372795150 conn(0x7f7a3c0ce4e0 msgr2=0x7f7a3c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 -- 172.21.2.117:0/372795150 shutdown_connections 2021-02-10T11:53:59.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.708 7f7a43c46700 1 -- 172.21.2.117:0/372795150 wait complete. 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.723 7f7a43c46700 1 Processor -- start 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.723 7f7a43c46700 1 -- start start 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c0c14d0 0x7f7a3c0c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 0x7f7a3c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c191ab0 con 0x7f7a3c0c2000 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c1916a0 con 0x7f7a3c0b6c00 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a43c46700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c191880 con 0x7f7a3c0c14d0 2021-02-10T11:53:59.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57624/0 (socket says 172.21.2.117:57624) 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 -- 172.21.2.117:0/3991061570 learned_addr learned my addr 172.21.2.117:0/3991061570 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a42263700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 0x7f7a3c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41261700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c0c14d0 0x7f7a3c0c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 -- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c0c14d0 msgr2=0x7f7a3c0c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c0c14d0 0x7f7a3c0c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 -- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 msgr2=0x7f7a3c0b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 0x7f7a3c0b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 -- 172.21.2.117:0/3991061570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c0b7620 con 0x7f7a3c0c2000 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a42263700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 0x7f7a3c0b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.724 7f7a41a62700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f7a3000b5e0 tx=0x7f7a3000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.725 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a3000bd70 con 0x7f7a3c0c2000 2021-02-10T11:53:59.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.725 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30002030 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.725 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a30002550 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.725 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c0b7620 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.725 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c0b7620 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.726 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a30000c90 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.726 7f7a2e7fc700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 0x7f7a2401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.726 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7a30032700 con 0x7f7a3c0c2000 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.726 7f7a41261700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 0x7f7a2401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:53:59.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.727 7f7a41261700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 0x7f7a2401abc0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:53:59.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.756 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c10c2b0 con 0x7f7a3c0c2000 2021-02-10T11:53:59.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.758 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a30030d60 con 0x7f7a3c0c2000 2021-02-10T11:53:59.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:53:59.892 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7a3c10c2b0 con 0x7f7a3c0c2000 2021-02-10T11:54:00.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.319 7f7a2e7fc700 1 -- 172.21.2.117:0/3991061570 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v129) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7a30000c90 con 0x7f7a3c0c2000 2021-02-10T11:54:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.360 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 msgr2=0x7f7a2401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.360 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 0x7f7a2401abc0 crc :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.360 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 msgr2=0x7f7a3c191080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.360 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f7a3000b5e0 tx=0x7f7a3000b980).stop 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 shutdown_connections 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a3c0b6c00 0x7f7a3c0b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a3c0c14d0 0x7f7a3c0c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a240186c0 0x7f7a2401abc0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 --2- 172.21.2.117:0/3991061570 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a3c0c2000 0x7f7a3c191080 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 >> 172.21.2.117:0/3991061570 conn(0x7f7a3c0acec0 msgr2=0x7f7a3c0b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 shutdown_connections 2021-02-10T11:54:00.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.361 7f7a43c46700 1 -- 172.21.2.117:0/3991061570 wait complete. 2021-02-10T11:54:00.372 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.780 7f182e39b700 1 Processor -- start 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.780 7f182e39b700 1 -- start start 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f182e39b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18280e4e40 0x7f18281b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f182e39b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f18280e5240 0x7f1828084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f182e39b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828085930 con 0x7f18281b0810 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f182e39b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18280e5cc0 con 0x7f18280e4e40 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f182e39b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828084840 con 0x7f18280e5240 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18280e4e40 0x7f18281b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f18280e4e40 0x7f18281b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57632/0 (socket says 172.21.2.117:57632) 2021-02-10T11:54:00.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 -- 172.21.2.117:0/1377150291 learned_addr learned my addr 172.21.2.117:0/1377150291 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 -- 172.21.2.117:0/1377150291 >> v2:172.21.2.125:3300/0 conn(0x7f18280e5240 msgr2=0x7f1828084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 --2- 172.21.2.117:0/1377150291 >> v2:172.21.2.125:3300/0 conn(0x7f18280e5240 0x7f1828084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 -- 172.21.2.117:0/1377150291 >> v1:172.21.2.125:6789/0 conn(0x7f18281b0810 legacy=0x7f1828085eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 -- 172.21.2.117:0/1377150291 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828085280 con 0x7f18280e4e40 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.781 7f1827fff700 1 --2- 172.21.2.117:0/1377150291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280e4e40 0x7f18281b01f0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f1818006700 tx=0x7f1818009b80).ready entity=mon.0 client_cookie=4c04d3e971d71292 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f1826ffd700 1 -- 172.21.2.117:0/1377150291 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f181800c0c0 con 0x7f18280e4e40 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f1826ffd700 1 -- 172.21.2.117:0/1377150291 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181800c0c0 con 0x7f18280e4e40 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f1826ffd700 1 -- 172.21.2.117:0/1377150291 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f181800c440 con 0x7f18280e4e40 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 -- 172.21.2.117:0/1377150291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280e4e40 msgr2=0x7f18281b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 --2- 172.21.2.117:0/1377150291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280e4e40 0x7f18281b01f0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 rx=0x7f1818006700 tx=0x7f1818009b80).stop 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 -- 172.21.2.117:0/1377150291 shutdown_connections 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 --2- 172.21.2.117:0/1377150291 >> v2:172.21.2.125:3300/0 conn(0x7f18280e5240 0x7f1828084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 --2- 172.21.2.117:0/1377150291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280e4e40 0x7f18281b01f0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 -- 172.21.2.117:0/1377150291 >> 172.21.2.117:0/1377150291 conn(0x7f18280ce4e0 msgr2=0x7f18280e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:00.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 -- 172.21.2.117:0/1377150291 shutdown_connections 2021-02-10T11:54:00.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.782 7f182e39b700 1 -- 172.21.2.117:0/1377150291 wait complete. 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 Processor -- start 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 -- start start 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828194fb0 0x7f1828095db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18280963d0 0x7f18281a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1828090410 con 0x7f18280967d0 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18281953b0 con 0x7f1828194fb0 2021-02-10T11:54:00.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182e39b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18281a5f00 con 0x7f18280963d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182c9b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182c9b8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57640/0 (socket says 172.21.2.117:57640) 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.797 7f182c9b8700 1 -- 172.21.2.117:0/1687891120 learned_addr learned my addr 172.21.2.117:0/1687891120 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 -- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18280963d0 msgr2=0x7f18281a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18280963d0 0x7f18281a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 -- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828194fb0 msgr2=0x7f1828095db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828194fb0 0x7f1828095db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 -- 172.21.2.117:0/1687891120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18281a5b10 con 0x7f18280967d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182c9b8700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7f1820009940 tx=0x7f182000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1820003030 con 0x7f18280967d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820001270 con 0x7f18280967d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1820001750 con 0x7f18280967d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182e39b700 1 -- 172.21.2.117:0/1687891120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18281a5b10 con 0x7f18280967d0 2021-02-10T11:54:00.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.798 7f182e39b700 1 -- 172.21.2.117:0/1687891120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18281a5b10 con 0x7f18280967d0 2021-02-10T11:54:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.799 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f182000a000 con 0x7f18280967d0 2021-02-10T11:54:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.799 7f1824ff9700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 0x7f180801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.799 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1820032c70 con 0x7f18280967d0 2021-02-10T11:54:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.800 7f1827fff700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 0x7f180801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:00.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.801 7f1827fff700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 0x7f180801abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:00.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.831 7f182e39b700 1 -- 172.21.2.117:0/1687891120 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1828085930 con 0x7f18280967d0 2021-02-10T11:54:00.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.833 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f182000be30 con 0x7f18280967d0 2021-02-10T11:54:00.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.966 7f182e39b700 1 -- 172.21.2.117:0/1687891120 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f1828090410 con 0x7f18080186c0 2021-02-10T11:54:00.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.968 7f1824ff9700 1 -- 172.21.2.117:0/1687891120 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f1818000970 con 0x7f18080186c0 2021-02-10T11:54:00.972 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:54:00.972 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:00.972 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:54:00.972 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:00.972 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:54:00.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 msgr2=0x7f180801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 0x7f180801abc0 crc :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 msgr2=0x7f18281a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 rx=0x7f1820009940 tx=0x7f182000bcf0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 shutdown_connections 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f18080186c0 0x7f180801abc0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f18280967d0 0x7f18281a54f0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f18280963d0 0x7f18281a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 --2- 172.21.2.117:0/1687891120 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1828194fb0 0x7f1828095db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:00.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 >> 172.21.2.117:0/1687891120 conn(0x7f18281a6900 msgr2=0x7f182818c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:00.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 shutdown_connections 2021-02-10T11:54:00.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:00.971 7f182e39b700 1 -- 172.21.2.117:0/1687891120 wait complete. 2021-02-10T11:54:00.982 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 Processor -- start 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 -- start start 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f045c0e7180 0x7f045c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f045c0e8390 0x7f045c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0e7580 con 0x7f045c1b07f0 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045c0e5ce0 con 0x7f045c0e8390 2021-02-10T11:54:01.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f046132a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045c0e77f0 con 0x7f045c0e7180 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f045b87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f045c0e8390 0x7f045c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f045b87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f045c0e8390 0x7f045c0e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57650/0 (socket says 172.21.2.117:57650) 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.397 7f045b87e700 1 -- 172.21.2.117:0/222130169 learned_addr learned my addr 172.21.2.117:0/222130169 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045b87e700 1 -- 172.21.2.117:0/222130169 >> v2:172.21.2.125:3300/0 conn(0x7f045c0e7180 msgr2=0x7f045c0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045b87e700 1 --2- 172.21.2.117:0/222130169 >> v2:172.21.2.125:3300/0 conn(0x7f045c0e7180 0x7f045c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045b87e700 1 -- 172.21.2.117:0/222130169 >> v1:172.21.2.117:6789/0 conn(0x7f045c1b07f0 legacy=0x7f045c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045b87e700 1 -- 172.21.2.117:0/222130169 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c0e8db0 con 0x7f045c0e8390 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045b87e700 1 --2- 172.21.2.117:0/222130169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c0e8390 0x7f045c0e8790 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7f0454006f00 tx=0x7f0454009f10).ready entity=mon.0 client_cookie=d90f26b2ff3237e1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045a07b700 1 -- 172.21.2.117:0/222130169 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0454003030 con 0x7f045c0e8390 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045a07b700 1 -- 172.21.2.117:0/222130169 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045400b3b0 con 0x7f045c0e8390 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f045a07b700 1 -- 172.21.2.117:0/222130169 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f045400b810 con 0x7f045c0e8390 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f046132a700 1 -- 172.21.2.117:0/222130169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c0e8390 msgr2=0x7f045c0e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.398 7f046132a700 1 --2- 172.21.2.117:0/222130169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c0e8390 0x7f045c0e8790 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 rx=0x7f0454006f00 tx=0x7f0454009f10).stop 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 -- 172.21.2.117:0/222130169 shutdown_connections 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 --2- 172.21.2.117:0/222130169 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c0e8390 0x7f045c0e8790 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 --2- 172.21.2.117:0/222130169 >> v2:172.21.2.125:3300/0 conn(0x7f045c0e7180 0x7f045c0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 -- 172.21.2.117:0/222130169 >> 172.21.2.117:0/222130169 conn(0x7f045c0ce500 msgr2=0x7f045c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 -- 172.21.2.117:0/222130169 shutdown_connections 2021-02-10T11:54:01.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.399 7f046132a700 1 -- 172.21.2.117:0/222130169 wait complete. 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 Processor -- start 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 -- start start 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f045c1a39e0 0x7f045c1a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f045c08b2b0 0x7f045c08b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c08a310 con 0x7f045c08a890 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c0c06f0 con 0x7f045c1a39e0 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f046132a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c0c08d0 con 0x7f045c08b2b0 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f045a87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:01.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f045a87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57654/0 (socket says 172.21.2.117:57654) 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.414 7f045a87c700 1 -- 172.21.2.117:0/822164662 learned_addr learned my addr 172.21.2.117:0/822164662 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045b07d700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f045c1a39e0 0x7f045c1a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045b87e700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f045c08b2b0 0x7f045c08b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 -- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f045c08b2b0 msgr2=0x7f045c08b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f045c08b2b0 0x7f045c08b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 -- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f045c1a39e0 msgr2=0x7f045c1a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f045c1a39e0 0x7f045c1a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 -- 172.21.2.117:0/822164662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c0c0af0 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f045a87c700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7f044c006f00 tx=0x7f044c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f044c0009b0 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c006350 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f044c000c90 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f046132a700 1 -- 172.21.2.117:0/822164662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c0c0af0 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.415 7f046132a700 1 -- 172.21.2.117:0/822164662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c0c1080 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.416 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f044c007000 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.416 7f0453fff700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 0x7f044001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.416 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f044c032750 con 0x7f045c08a890 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.416 7f045b07d700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 0x7f044001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:01.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.416 7f045b07d700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 0x7f044001abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:01.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.447 7f046132a700 1 -- 172.21.2.117:0/822164662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c108a30 con 0x7f045c08a890 2021-02-10T11:54:01.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.450 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f044c006b90 con 0x7f045c08a890 2021-02-10T11:54:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.599 7f046132a700 1 -- 172.21.2.117:0/822164662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f045c0e77f0 con 0x7f045c08a890 2021-02-10T11:54:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.600 7f0453fff700 1 -- 172.21.2.117:0/822164662 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f044c007000 con 0x7f045c08a890 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.601 7f046132a700 1 -- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 msgr2=0x7f044001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.601 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 0x7f044001abc0 crc :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 -- 172.21.2.117:0/822164662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 msgr2=0x7f045c08ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 rx=0x7f044c006f00 tx=0x7f044c00b940).stop 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 -- 172.21.2.117:0/822164662 shutdown_connections 2021-02-10T11:54:01.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04400186c0 0x7f044001abc0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f045c08a890 0x7f045c08ac90 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f045c08b2b0 0x7f045c08b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 --2- 172.21.2.117:0/822164662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f045c1a39e0 0x7f045c1a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 -- 172.21.2.117:0/822164662 >> 172.21.2.117:0/822164662 conn(0x7f045c0a6b70 msgr2=0x7f045c0be660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 -- 172.21.2.117:0/822164662 shutdown_connections 2021-02-10T11:54:01.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:01.602 7f046132a700 1 -- 172.21.2.117:0/822164662 wait complete. 2021-02-10T11:54:01.613 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs non_existent_subvolume 2021-02-10T11:54:02.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.026 7fc27503b700 1 Processor -- start 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.026 7fc27503b700 1 -- start start 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc27503b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc2700d18f0 0x7fc2700eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc27503b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2700ef7f0 con 0x7fc2700eb040 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc27503b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2700ede60 con 0x7fc2700ef3f0 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc27503b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2700d1da0 con 0x7fc2700d18f0 2021-02-10T11:54:02.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc26e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc2700eb040 0x7fc2700eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57912/0 (socket says 172.21.2.117:57912) 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc26e59c700 1 -- 172.21.2.117:0/3499607611 learned_addr learned my addr 172.21.2.117:0/3499607611 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc26ed9d700 1 --2- 172.21.2.117:0/3499607611 >> v2:172.21.2.125:3300/0 conn(0x7fc2700d18f0 0x7fc2700eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.027 7fc26dd9b700 1 -- 172.21.2.117:0/3499607611 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583482990 0 0) 0x7fc2640010b0 con 0x7fc2700eb040 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26dd9b700 1 -- 172.21.2.117:0/3499607611 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258002fe0 con 0x7fc2700eb040 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26ed9d700 1 -- 172.21.2.117:0/3499607611 >> v1:172.21.2.125:6789/0 conn(0x7fc2700ef3f0 legacy=0x7fc2701b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26ed9d700 1 -- 172.21.2.117:0/3499607611 >> v1:172.21.2.117:6789/0 conn(0x7fc2700eb040 legacy=0x7fc2700eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26ed9d700 1 -- 172.21.2.117:0/3499607611 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2701bb1c0 con 0x7fc2700d18f0 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26ed9d700 1 --2- 172.21.2.117:0/3499607611 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2700d18f0 0x7fc2700eaa20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fc260006650 tx=0x7fc260009a80).ready entity=mon.1 client_cookie=95ebb0ecf1acd0d5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26dd9b700 1 -- 172.21.2.117:0/3499607611 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2600009b0 con 0x7fc2700d18f0 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26dd9b700 1 -- 172.21.2.117:0/3499607611 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000c0c0 con 0x7fc2700d18f0 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc26dd9b700 1 -- 172.21.2.117:0/3499607611 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc26000c5a0 con 0x7fc2700d18f0 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc27503b700 1 -- 172.21.2.117:0/3499607611 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2700d18f0 msgr2=0x7fc2700eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.028 7fc27503b700 1 --2- 172.21.2.117:0/3499607611 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2700d18f0 0x7fc2700eaa20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fc260006650 tx=0x7fc260009a80).stop 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.029 7fc27503b700 1 -- 172.21.2.117:0/3499607611 shutdown_connections 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.029 7fc27503b700 1 --2- 172.21.2.117:0/3499607611 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc2700d18f0 0x7fc2700eaa20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.029 7fc27503b700 1 -- 172.21.2.117:0/3499607611 >> 172.21.2.117:0/3499607611 conn(0x7fc2700cf060 msgr2=0x7fc2700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.029 7fc27503b700 1 -- 172.21.2.117:0/3499607611 shutdown_connections 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.029 7fc27503b700 1 -- 172.21.2.117:0/3499607611 wait complete. 2021-02-10T11:54:02.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.043 7fc27503b700 1 Processor -- start 2021-02-10T11:54:02.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 -- start start 2021-02-10T11:54:02.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc270093be0 0x7fc2701a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:02.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:02.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc27008c850 0x7fc27008cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270094280 con 0x7fc2701a7ed0 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270093fe0 con 0x7fc270093be0 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc27503b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700ace00 con 0x7fc27008c850 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc26e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc26e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57668/0 (socket says 172.21.2.117:57668) 2021-02-10T11:54:02.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc26e59c700 1 -- 172.21.2.117:0/1312969305 learned_addr learned my addr 172.21.2.117:0/1312969305 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc26ed9d700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc270093be0 0x7fc2701a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.044 7fc26f59e700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc27008c850 0x7fc27008cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 -- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc27008c850 msgr2=0x7fc27008cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc27008c850 0x7fc27008cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 -- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc270093be0 msgr2=0x7fc2701a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc270093be0 0x7fc2701a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 -- 172.21.2.117:0/1312969305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270094800 con 0x7fc2701a7ed0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc26e59c700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7fc264001c60 tx=0x7fc26400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc264000e00 con 0x7fc2701a7ed0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26400acc0 con 0x7fc2701a7ed0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc27503b700 1 -- 172.21.2.117:0/1312969305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc270094800 con 0x7fc2701a7ed0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc27503b700 1 -- 172.21.2.117:0/1312969305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc270094d90 con 0x7fc2701a7ed0 2021-02-10T11:54:02.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.045 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc2640045d0 con 0x7fc2701a7ed0 2021-02-10T11:54:02.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.046 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc264004140 con 0x7fc2701a7ed0 2021-02-10T11:54:02.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.046 7fc25f7fe700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 0x7fc25401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:02.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.046 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc264032910 con 0x7fc2701a7ed0 2021-02-10T11:54:02.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.046 7fc26ed9d700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 0x7fc25401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:02.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.046 7fc26ed9d700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 0x7fc25401abd0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.077 7fc27503b700 1 -- 172.21.2.117:0/1312969305 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2700ede60 con 0x7fc2701a7ed0 2021-02-10T11:54:02.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.079 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc264000960 con 0x7fc2701a7ed0 2021-02-10T11:54:02.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.204 7fc27503b700 1 -- 172.21.2.117:0/1312969305 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "non_existent_subvolume", "target": ["mgr", ""]}) v1 -- 0x7fc270094d90 con 0x7fc2540186d0 2021-02-10T11:54:02.211 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.211 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:02.212 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.211 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:02.212 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.211 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:02.212 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.211 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:02.212 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.211 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:02.216 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:02.216 7f59c98e1700 -1 mgr.server reply reply (2) No such file or directory subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:54:02.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.215 7fc25f7fe700 1 -- 172.21.2.117:0/1312969305 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -2 subvolume 'non_existent_subvolume' does not exist) v1 ==== 57+0+0 (crc 0 0 0) 0x7fc260006ae0 con 0x7fc2540186d0 2021-02-10T11:54:02.222 INFO:teuthology.orchestra.run.gibba017.stderr:Error ENOENT: subvolume 'non_existent_subvolume' does not exist 2021-02-10T11:54:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 msgr2=0x7fc25401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 0x7fc25401abd0 crc :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 msgr2=0x7fc27008c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:02.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 rx=0x7fc264001c60 tx=0x7fc26400a8d0).stop 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 shutdown_connections 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc2540186d0 0x7fc25401abd0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc2701a7ed0 0x7fc27008c230 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc27008c850 0x7fc27008cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 --2- 172.21.2.117:0/1312969305 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc270093be0 0x7fc2701a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 >> 172.21.2.117:0/1312969305 conn(0x7fc270092d00 msgr2=0x7fc27008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 shutdown_connections 2021-02-10T11:54:02.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:02.222 7fc27503b700 1 -- 172.21.2.117:0/1312969305 wait complete. 2021-02-10T11:54:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:54:02.233 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:54:02.233 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:02.579 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:02.925 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:54:02.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:02.925 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:02.926 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:03.261 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:03.262 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:54:03.262 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:54:03.262 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:54:03.262 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:03.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[176898]: fuse finished with error 0 and tester_r 0 2021-02-10T11:54:03.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:03.621 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:03.622 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:03.957 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:54:03.957 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:54:03.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:03.958 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:04.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:04.294 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:04.294 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:04.630 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:04.631 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:54:04.631 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:54:04.631 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:54:04.631 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:04.984 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[177539]: fuse finished with error 0 and tester_r 0 2021-02-10T11:54:04.985 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:04.985 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:04.985 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:05.322 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:54:05.322 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:54:05.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:05.323 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:05.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:05.659 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:05.659 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:54:05.993 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:54:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:05.994 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:54:05.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:05.994 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:54:06.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:06.330 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:06.330 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:54:06.667 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:54:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:06.668 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:54:06.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:06.668 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:54:07.002 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:54:07.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 Processor -- start 2021-02-10T11:54:07.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 -- start start 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8d400d15e0 0x7f8d400e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400e7580 con 0x7f8d401b07f0 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400e77f0 con 0x7f8d400e7180 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.415 7f8d48515700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d400e5ce0 con 0x7f8d400d15e0 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46331700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8d401b07f0 0x7f8d400e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57924/0 (socket says 172.21.2.117:57924) 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46331700 1 -- 172.21.2.117:0/2850921341 learned_addr learned my addr 172.21.2.117:0/2850921341 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46b32700 1 --2- 172.21.2.117:0/2850921341 >> v2:172.21.2.117:3300/0 conn(0x7f8d400d15e0 0x7f8d400e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46b32700 1 -- 172.21.2.117:0/2850921341 >> v1:172.21.2.125:6789/0 conn(0x7f8d400e7180 legacy=0x7f8d400e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46b32700 1 -- 172.21.2.117:0/2850921341 >> v1:172.21.2.117:6789/0 conn(0x7f8d401b07f0 legacy=0x7f8d400e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.416 7f8d46b32700 1 -- 172.21.2.117:0/2850921341 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400d19e0 con 0x7f8d400d15e0 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d46b32700 1 --2- 172.21.2.117:0/2850921341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d400d15e0 0x7f8d400e8560 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f8d38006e30 tx=0x7f8d38009de0).ready entity=mon.0 client_cookie=d29bf264ae20e712 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:07.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d4532f700 1 -- 172.21.2.117:0/2850921341 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3800b300 con 0x7f8d400d15e0 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d4532f700 1 -- 172.21.2.117:0/2850921341 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3800b580 con 0x7f8d400d15e0 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d4532f700 1 -- 172.21.2.117:0/2850921341 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3800baa0 con 0x7f8d400d15e0 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d48515700 1 -- 172.21.2.117:0/2850921341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d400d15e0 msgr2=0x7f8d400e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.417 7f8d48515700 1 --2- 172.21.2.117:0/2850921341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d400d15e0 0x7f8d400e8560 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f8d38006e30 tx=0x7f8d38009de0).stop 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.418 7f8d48515700 1 -- 172.21.2.117:0/2850921341 shutdown_connections 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.418 7f8d48515700 1 --2- 172.21.2.117:0/2850921341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d400d15e0 0x7f8d400e8560 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.418 7f8d48515700 1 -- 172.21.2.117:0/2850921341 >> 172.21.2.117:0/2850921341 conn(0x7f8d400ce500 msgr2=0x7f8d400e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.418 7f8d48515700 1 -- 172.21.2.117:0/2850921341 shutdown_connections 2021-02-10T11:54:07.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.418 7f8d48515700 1 -- 172.21.2.117:0/2850921341 wait complete. 2021-02-10T11:54:07.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.432 7f8d48515700 1 Processor -- start 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.432 7f8d48515700 1 -- start start 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d400bb840 0x7f8d400bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401c4bb0 0x7f8d401c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d401c59d0 con 0x7f8d401c55d0 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4008de90 con 0x7f8d400bb840 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d48515700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40096fc0 con 0x7f8d401c4bb0 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57684/0 (socket says 172.21.2.117:57684) 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 -- 172.21.2.117:0/230516586 learned_addr learned my addr 172.21.2.117:0/230516586 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:07.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 -- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401c4bb0 msgr2=0x7f8d401c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401c4bb0 0x7f8d401c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 -- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d400bb840 msgr2=0x7f8d400bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d400bb840 0x7f8d400bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 -- 172.21.2.117:0/230516586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400971e0 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.433 7f8d46b32700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7f8d38009310 tx=0x7f8d3800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.434 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d38000f90 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.434 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3800c2f0 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.434 7f8d48515700 1 -- 172.21.2.117:0/230516586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d400971e0 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.434 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8d3800c810 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.434 7f8d48515700 1 -- 172.21.2.117:0/230516586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d40097770 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.435 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8d38001270 con 0x7f8d401c55d0 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.435 7f8d32ffd700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:07.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.435 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8d38032c90 con 0x7f8d401c55d0 2021-02-10T11:54:07.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.435 7f8d46331700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:07.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.435 7f8d46331700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:07.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.465 7f8d48515700 1 -- 172.21.2.117:0/230516586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4007c0b0 con 0x7f8d401c55d0 2021-02-10T11:54:07.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.468 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8d38003000 con 0x7f8d401c55d0 2021-02-10T11:54:07.620 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.619 7f8d48515700 1 -- 172.21.2.117:0/230516586 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8d400d1c50 con 0x7f8d401c55d0 2021-02-10T11:54:07.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.622 7f8d32ffd700 1 -- 172.21.2.117:0/230516586 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8d38001270 con 0x7f8d401c55d0 2021-02-10T11:54:07.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 -- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 msgr2=0x7f8d2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 crc :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:07.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 -- 172.21.2.117:0/230516586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 msgr2=0x7f8d400969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:07.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 rx=0x7f8d38009310 tx=0x7f8d3800bcf0).stop 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 -- 172.21.2.117:0/230516586 shutdown_connections 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8d280186c0 0x7f8d2801abc0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8d401c55d0 0x7f8d400969a0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8d401c4bb0 0x7f8d401c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 --2- 172.21.2.117:0/230516586 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8d400bb840 0x7f8d400bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.627 7f8d48515700 1 -- 172.21.2.117:0/230516586 >> 172.21.2.117:0/230516586 conn(0x7f8d4007cb30 msgr2=0x7f8d400bc590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.628 7f8d48515700 1 -- 172.21.2.117:0/230516586 shutdown_connections 2021-02-10T11:54:07.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:07.628 7f8d48515700 1 -- 172.21.2.117:0/230516586 wait complete. 2021-02-10T11:54:07.638 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm' 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.047 7f142eb11700 1 Processor -- start 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 -- start start 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14280e4e40 0x7f14281b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280e5240 0x7f1428084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1428085930 con 0x7f14281b0810 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1428084840 con 0x7f14280e4e40 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142eb11700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1428084a30 con 0x7f14280e5240 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142d12e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f14281b0810 0x7f1428085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57942/0 (socket says 172.21.2.117:57942) 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142d12e700 1 -- 172.21.2.117:0/1222761802 learned_addr learned my addr 172.21.2.117:0/1222761802 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.048 7f142c92d700 1 --2- 172.21.2.117:0/1222761802 >> v2:172.21.2.117:3300/0 conn(0x7f14280e4e40 0x7f14281b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142c92d700 1 -- 172.21.2.117:0/1222761802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280e5240 msgr2=0x7f1428084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142c92d700 1 --2- 172.21.2.117:0/1222761802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280e5240 0x7f1428084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142c92d700 1 -- 172.21.2.117:0/1222761802 >> v1:172.21.2.117:6789/0 conn(0x7f14281b0810 legacy=0x7f1428085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142c92d700 1 -- 172.21.2.117:0/1222761802 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428085280 con 0x7f14280e4e40 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142c92d700 1 --2- 172.21.2.117:0/1222761802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14280e4e40 0x7f14281b01f0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 rx=0x7f1410006650 tx=0x7f1410009a80).ready entity=mon.0 client_cookie=4c6b59ba85f3b465 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f14277fe700 1 -- 172.21.2.117:0/1222761802 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141000c0c0 con 0x7f14280e4e40 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f14277fe700 1 -- 172.21.2.117:0/1222761802 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000c340 con 0x7f14280e4e40 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f14277fe700 1 -- 172.21.2.117:0/1222761802 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141000c860 con 0x7f14280e4e40 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142eb11700 1 -- 172.21.2.117:0/1222761802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14280e4e40 msgr2=0x7f14281b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.049 7f142eb11700 1 --2- 172.21.2.117:0/1222761802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14280e4e40 0x7f14281b01f0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 rx=0x7f1410006650 tx=0x7f1410009a80).stop 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 -- 172.21.2.117:0/1222761802 shutdown_connections 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 --2- 172.21.2.117:0/1222761802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280e5240 0x7f1428084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 --2- 172.21.2.117:0/1222761802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14280e4e40 0x7f14281b01f0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 -- 172.21.2.117:0/1222761802 >> 172.21.2.117:0/1222761802 conn(0x7f14280ce4e0 msgr2=0x7f14280e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 -- 172.21.2.117:0/1222761802 shutdown_connections 2021-02-10T11:54:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.050 7f142eb11700 1 -- 172.21.2.117:0/1222761802 wait complete. 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.064 7f142eb11700 1 Processor -- start 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.064 7f142eb11700 1 -- start start 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14281a5640 0x7f142810c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280b7c00 0x7f142810d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142810cf00 con 0x7f142810caa0 2021-02-10T11:54:08.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14281a5a40 con 0x7f14281a5640 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f142eb11700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14281a5c20 con 0x7f14280b7c00 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57696/0 (socket says 172.21.2.117:57696) 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 -- 172.21.2.117:0/3891103380 learned_addr learned my addr 172.21.2.117:0/3891103380 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 -- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280b7c00 msgr2=0x7f142810d480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280b7c00 0x7f142810d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 -- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14281a5640 msgr2=0x7f142810c480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14281a5640 0x7f142810c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.065 7f1427fff700 1 -- 172.21.2.117:0/3891103380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428110fe0 con 0x7f142810caa0 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f1427fff700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7f141800b5c0 tx=0x7f141800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141800bd50 con 0x7f142810caa0 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1418006030 con 0x7f142810caa0 2021-02-10T11:54:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14180054d0 con 0x7f142810caa0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f142eb11700 1 -- 172.21.2.117:0/3891103380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428110fe0 con 0x7f142810caa0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f142eb11700 1 -- 172.21.2.117:0/3891103380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14280b82f0 con 0x7f142810caa0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1418006210 con 0x7f142810caa0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 0x7f140c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.066 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f14180325d0 con 0x7f142810caa0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.067 7f142c92d700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 0x7f140c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.067 7f142c92d700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 0x7f140c01abc0 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.098 7f142eb11700 1 -- 172.21.2.117:0/3891103380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142810c250 con 0x7f142810caa0 2021-02-10T11:54:08.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.100 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1418030db0 con 0x7f142810caa0 2021-02-10T11:54:08.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.230 7f142eb11700 1 -- 172.21.2.117:0/3891103380 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]} v 0) v1 -- 0x7f1428085930 con 0x7f142810caa0 2021-02-10T11:54:08.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.325 7f14257fa700 1 -- 172.21.2.117:0/3891103380 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_rm"]}]=0 v762) v1 ==== 146+0+0 (secure 0 0 0) 0x7f141801fdf0 con 0x7f142810caa0 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 -- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 msgr2=0x7f140c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 0x7f140c01abc0 crc :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 -- 172.21.2.117:0/3891103380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 msgr2=0x7f14280b75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 rx=0x7f141800b5c0 tx=0x7f141800b960).stop 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 -- 172.21.2.117:0/3891103380 shutdown_connections 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f140c0186c0 0x7f140c01abc0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f142810caa0 0x7f14280b75e0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14280b7c00 0x7f142810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 --2- 172.21.2.117:0/3891103380 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14281a5640 0x7f142810c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 -- 172.21.2.117:0/3891103380 >> 172.21.2.117:0/3891103380 conn(0x7f14281adc00 msgr2=0x7f14281a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.347 7f142eb11700 1 -- 172.21.2.117:0/3891103380 shutdown_connections 2021-02-10T11:54:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.348 7f142eb11700 1 -- 172.21.2.117:0/3891103380 wait complete. 2021-02-10T11:54:08.357 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:54:08.358 INFO:tasks.cephfs_test_runner:Starting test: test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:54:08.358 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:54:08.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.767 7fc1a56cd700 1 Processor -- start 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.767 7fc1a56cd700 1 -- start start 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc1a56cd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1a01b6960 0x7fc1a01b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc1a56cd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a00ede60 con 0x7fc1a00d18e0 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc1a56cd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a00eb9c0 con 0x7fc1a00ef3f0 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc1a56cd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a00ef7f0 con 0x7fc1a01b6960 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1a01b6960 0x7fc1a01b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19f2bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc1a00d18e0 0x7fc1a00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57952/0 (socket says 172.21.2.117:57952) 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19fac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc1a01b6960 0x7fc1a01b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57706/0 (socket says 172.21.2.117:57706) 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19f2bf700 1 -- 172.21.2.117:0/1630988534 learned_addr learned my addr 172.21.2.117:0/1630988534 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19fac0700 1 -- 172.21.2.117:0/1630988534 >> v1:172.21.2.125:6789/0 conn(0x7fc1a00ef3f0 legacy=0x7fc1a00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19fac0700 1 -- 172.21.2.117:0/1630988534 >> v1:172.21.2.117:6789/0 conn(0x7fc1a00d18e0 legacy=0x7fc1a00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.768 7fc19fac0700 1 -- 172.21.2.117:0/1630988534 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00d1d20 con 0x7fc1a01b6960 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc19fac0700 1 --2- 172.21.2.117:0/1630988534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a01b6960 0x7fc1a01b8d70 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7fc198006e30 tx=0x7fc198009e60).ready entity=mon.0 client_cookie=6a5fb9be3bb9a047 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc19e2bd700 1 -- 172.21.2.117:0/1630988534 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc19800b300 con 0x7fc1a01b6960 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc19e2bd700 1 -- 172.21.2.117:0/1630988534 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19800b580 con 0x7fc1a01b6960 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc19e2bd700 1 -- 172.21.2.117:0/1630988534 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc19800baa0 con 0x7fc1a01b6960 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 -- 172.21.2.117:0/1630988534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a01b6960 msgr2=0x7fc1a01b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 --2- 172.21.2.117:0/1630988534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a01b6960 0x7fc1a01b8d70 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7fc198006e30 tx=0x7fc198009e60).stop 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 -- 172.21.2.117:0/1630988534 shutdown_connections 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 --2- 172.21.2.117:0/1630988534 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a01b6960 0x7fc1a01b8d70 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 -- 172.21.2.117:0/1630988534 >> 172.21.2.117:0/1630988534 conn(0x7fc1a00ced60 msgr2=0x7fc1a00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 -- 172.21.2.117:0/1630988534 shutdown_connections 2021-02-10T11:54:08.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.769 7fc1a56cd700 1 -- 172.21.2.117:0/1630988534 wait complete. 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.784 7fc1a56cd700 1 Processor -- start 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.784 7fc1a56cd700 1 -- start start 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1a008d6b0 0x7fc1a00be490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1a00beab0 0x7fc1a008c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a008d220 con 0x7fc1a008dab0 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a008d410 con 0x7fc1a00beab0 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc1a56cd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a00b0100 con 0x7fc1a008d6b0 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57712/0 (socket says 172.21.2.117:57712) 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 -- 172.21.2.117:0/2471754527 learned_addr learned my addr 172.21.2.117:0/2471754527 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:08.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19f2bf700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1a008d6b0 0x7fc1a00be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19eabe700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1a00beab0 0x7fc1a008c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 -- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1a008d6b0 msgr2=0x7fc1a00be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1a008d6b0 0x7fc1a00be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 -- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1a00beab0 msgr2=0x7fc1a008c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1a00beab0 0x7fc1a008c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 -- 172.21.2.117:0/2471754527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00b02f0 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.785 7fc19fac0700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fc19800b990 tx=0x7fc19800bd30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc198009380 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198001480 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc1980018d0 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a00b02f0 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a00b02f0 con 0x7fc1a008dab0 2021-02-10T11:54:08.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc1980069b0 con 0x7fc1a008dab0 2021-02-10T11:54:08.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 0x7fc18401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:08.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc1980328b0 con 0x7fc1a008dab0 2021-02-10T11:54:08.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.786 7fc19f2bf700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 0x7fc18401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:08.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.787 7fc19f2bf700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 0x7fc18401abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:08.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.818 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a0086140 con 0x7fc1a008dab0 2021-02-10T11:54:08.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.820 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc198001270 con 0x7fc1a008dab0 2021-02-10T11:54:08.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:08.951 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7fc1a0086140 con 0x7fc1a008dab0 2021-02-10T11:54:09.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.328 7fc197fff700 1 -- 172.21.2.117:0/2471754527 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v763) v1 ==== 158+0+0 (secure 0 0 0) 0x7fc1980069b0 con 0x7fc1a008dab0 2021-02-10T11:54:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 msgr2=0x7fc18401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 0x7fc18401abc0 crc :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 msgr2=0x7fc1a008cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 rx=0x7fc19800b990 tx=0x7fc19800bd30).stop 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 shutdown_connections 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.370 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc1840186c0 0x7fc18401abc0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc1a008dab0 0x7fc1a008cc00 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc1a00beab0 0x7fc1a008c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 --2- 172.21.2.117:0/2471754527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc1a008d6b0 0x7fc1a00be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 >> 172.21.2.117:0/2471754527 conn(0x7fc1a009ace0 msgr2=0x7fc1a00eef20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 shutdown_connections 2021-02-10T11:54:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.371 7fc1a56cd700 1 -- 172.21.2.117:0/2471754527 wait complete. 2021-02-10T11:54:09.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:54:09.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 Processor -- start 2021-02-10T11:54:09.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 -- start start 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1e800d18f0 0x7f1e800eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800ef7f0 con 0x7f1e800eb040 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800ede60 con 0x7f1e800ef3f0 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.793 7f1e84d1c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e800d1da0 con 0x7f1e800d18f0 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7e312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1e800eb040 0x7f1e800eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57966/0 (socket says 172.21.2.117:57966) 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7e312700 1 -- 172.21.2.117:0/1366353341 learned_addr learned my addr 172.21.2.117:0/1366353341 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703791620 0 0) 0x7f1e700009b0 con 0x7f1e800eb040 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c002fe0 con 0x7f1e800eb040 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7eb13700 1 --2- 172.21.2.117:0/1366353341 >> v2:172.21.2.125:3300/0 conn(0x7f1e800d18f0 0x7f1e800eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1309650374 0 0) 0x7f1e70001010 con 0x7f1e800eb040 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 >> v2:172.21.2.125:3300/0 conn(0x7f1e800d18f0 msgr2=0x7f1e800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 --2- 172.21.2.117:0/1366353341 >> v2:172.21.2.125:3300/0 conn(0x7f1e800d18f0 0x7f1e800eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 >> v1:172.21.2.125:6789/0 conn(0x7f1e800ef3f0 legacy=0x7f1e801b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e801bb1c0 con 0x7f1e800eb040 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1e70001590 con 0x7f1e800eb040 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 >> v1:172.21.2.117:6789/0 conn(0x7f1e800eb040 legacy=0x7f1e800eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e6c003ac0 0x7f1e6c005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e6c006770 0x7f1e6c008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.794 7f1e7db11700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 0x7f1e6c00bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c00c300 con 0x7f1e6c009590 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c00c520 con 0x7f1e6c003ac0 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c00c740 con 0x7f1e6c006770 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 0x7f1e6c00bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7e312700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e6c006770 0x7f1e6c008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7eb13700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e6c003ac0 0x7f1e6c005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 -- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e6c006770 msgr2=0x7f1e6c008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e6c006770 0x7f1e6c008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 -- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e6c003ac0 msgr2=0x7f1e6c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e6c003ac0 0x7f1e6c005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 -- 172.21.2.117:0/1366353341 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0509b0 con 0x7f1e6c009590 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 -- 172.21.2.117:0/1366353341 reap_dead start 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7f314700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 0x7f1e6c00bab0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 rx=0x7f1e78006e40 tx=0x7f1e78009e30).ready entity=mon.0 client_cookie=68e4e7f8f4e12cd1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e78006a30 con 0x7f1e6c009590 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7800b300 con 0x7f1e6c009590 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e7db11700 1 -- 172.21.2.117:0/1366353341 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e7800b7e0 con 0x7f1e6c009590 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e84d1c700 1 -- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 msgr2=0x7f1e6c00bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.795 7f1e84d1c700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 0x7f1e6c00bab0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 rx=0x7f1e78006e40 tx=0x7f1e78009e30).stop 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.796 7f1e84d1c700 1 -- 172.21.2.117:0/1366353341 shutdown_connections 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.796 7f1e84d1c700 1 --2- 172.21.2.117:0/1366353341 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e6c009590 0x7f1e6c00bab0 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.796 7f1e84d1c700 1 -- 172.21.2.117:0/1366353341 >> 172.21.2.117:0/1366353341 conn(0x7f1e800cf060 msgr2=0x7f1e800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:09.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.796 7f1e84d1c700 1 -- 172.21.2.117:0/1366353341 shutdown_connections 2021-02-10T11:54:09.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.796 7f1e84d1c700 1 -- 172.21.2.117:0/1366353341 wait complete. 2021-02-10T11:54:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.811 7f1e84d1c700 1 Processor -- start 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.811 7f1e84d1c700 1 -- start start 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e80085ec0 0x7f1e800862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e800a2040 0x7f1e800a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800a67e0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800a7120 con 0x7f1e800a2040 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e84d1c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800a2a60 con 0x7f1e80085ec0 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57726/0 (socket says 172.21.2.117:57726) 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 -- 172.21.2.117:0/4133762513 learned_addr learned my addr 172.21.2.117:0/4133762513 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7e312700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e80085ec0 0x7f1e800862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7f314700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e800a2040 0x7f1e800a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 -- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e80085ec0 msgr2=0x7f1e800862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e80085ec0 0x7f1e800862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 -- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e800a2040 msgr2=0x7f1e800a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e800a2040 0x7f1e800a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 -- 172.21.2.117:0/4133762513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e801a2dc0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.812 7f1e7eb13700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f1e68006f60 tx=0x7f1e6800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6800bd60 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e68000c90 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e680022b0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e801a2dc0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e801a2dc0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.813 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e68006b00 con 0x7f1e800a6ce0 2021-02-10T11:54:09.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.814 7f1e777fe700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 0x7f1e6401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:09.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.814 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1e68032790 con 0x7f1e800a6ce0 2021-02-10T11:54:09.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.814 7f1e7e312700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 0x7f1e6401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:09.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.814 7f1e7e312700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 0x7f1e6401abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:09.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.844 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e800daff0 con 0x7f1e800a6ce0 2021-02-10T11:54:09.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.846 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e68000c90 con 0x7f1e800a6ce0 2021-02-10T11:54:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.995 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1e800daff0 con 0x7f1e800a6ce0 2021-02-10T11:54:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:09.998 7f1e777fe700 1 -- 172.21.2.117:0/4133762513 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1e68006b00 con 0x7f1e800a6ce0 2021-02-10T11:54:10.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 msgr2=0x7f1e6401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:10.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 0x7f1e6401abc0 crc :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:10.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 msgr2=0x7f1e800dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 rx=0x7f1e68006f60 tx=0x7f1e6800b970).stop 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 shutdown_connections 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e800a2040 0x7f1e800a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e80085ec0 0x7f1e800862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e64018730 0x7f1e6401abc0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 --2- 172.21.2.117:0/4133762513 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e800a6ce0 0x7f1e800dc420 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.003 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 >> 172.21.2.117:0/4133762513 conn(0x7f1e800b6540 msgr2=0x7f1e8008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.004 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 shutdown_connections 2021-02-10T11:54:10.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:10.004 7f1e84d1c700 1 -- 172.21.2.117:0/4133762513 wait complete. 2021-02-10T11:54:10.014 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:54:10.014 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:10.358 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:10.703 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:54:10.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:10.704 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:10.704 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:11.040 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:54:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:11.040 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:54:11.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:11.041 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:11.041 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:11.374 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:54:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:11.375 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:54:11.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:11.375 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:11.375 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:54:11.710 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:54:11.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:11.710 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:54:11.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:11.710 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:11.711 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:54:12.045 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:54:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:12.046 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:54:12.046 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 Processor -- start 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 -- start start 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 0x7f8ab81b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80ede60 con 0x7f8ab80d18e0 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80eb9c0 con 0x7f8ab80ef3f0 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.456 7f8abdbae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab80d1ce0 con 0x7f8ab81b6960 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8ab80d18e0 0x7f8ab80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:57984/0 (socket says 172.21.2.117:57984) 2021-02-10T11:54:12.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab77fe700 1 -- 172.21.2.117:0/4009758142 learned_addr learned my addr 172.21.2.117:0/4009758142 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab7fff700 1 --2- 172.21.2.117:0/4009758142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 0x7f8ab81b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117864727 0 0) 0x7f8aa80009b0 con 0x7f8ab80d18e0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa4002fe0 con 0x7f8ab80d18e0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635102173 0 0) 0x7f8aa0001080 con 0x7f8ab80ef3f0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa40042c0 con 0x7f8ab80ef3f0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab7fff700 1 -- 172.21.2.117:0/4009758142 >> v1:172.21.2.125:6789/0 conn(0x7f8ab80ef3f0 legacy=0x7f8ab80eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab7fff700 1 -- 172.21.2.117:0/4009758142 >> v1:172.21.2.117:6789/0 conn(0x7f8ab80d18e0 legacy=0x7f8ab80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2097771373 0 0) 0x7f8aa8001010 con 0x7f8ab80d18e0 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.457 7f8ab7fff700 1 -- 172.21.2.117:0/4009758142 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab81bb3b0 con 0x7f8ab81b6960 2021-02-10T11:54:12.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8ab7fff700 1 --2- 172.21.2.117:0/4009758142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 0x7f8ab81b8d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8ab0006e50 tx=0x7f8ab0009e70).ready entity=mon.2 client_cookie=8679bf68ac262330 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ab0006a20 con 0x7f8ab81b6960 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab0006790 con 0x7f8ab81b6960 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8ab67fc700 1 -- 172.21.2.117:0/4009758142 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8ab000b700 con 0x7f8ab81b6960 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8abdbae700 1 -- 172.21.2.117:0/4009758142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 msgr2=0x7f8ab81b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.458 7f8abdbae700 1 --2- 172.21.2.117:0/4009758142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 0x7f8ab81b8d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f8ab0006e50 tx=0x7f8ab0009e70).stop 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.459 7f8abdbae700 1 -- 172.21.2.117:0/4009758142 shutdown_connections 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.459 7f8abdbae700 1 --2- 172.21.2.117:0/4009758142 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab81b6960 0x7f8ab81b8d70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.459 7f8abdbae700 1 -- 172.21.2.117:0/4009758142 >> 172.21.2.117:0/4009758142 conn(0x7f8ab80ced60 msgr2=0x7f8ab80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.459 7f8abdbae700 1 -- 172.21.2.117:0/4009758142 shutdown_connections 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.459 7f8abdbae700 1 -- 172.21.2.117:0/4009758142 wait complete. 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.473 7f8abdbae700 1 Processor -- start 2021-02-10T11:54:12.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.473 7f8abdbae700 1 -- start start 2021-02-10T11:54:12.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:12.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab809acc0 0x7f8ab809b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:12.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab809b6e0 0x7f8ab80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:12.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab809a420 con 0x7f8ab808a280 2021-02-10T11:54:12.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab809bae0 con 0x7f8ab809acc0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8abdbae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab809bd00 con 0x7f8ab809b6e0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57738/0 (socket says 172.21.2.117:57738) 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 -- 172.21.2.117:0/1070325720 learned_addr learned my addr 172.21.2.117:0/1070325720 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab6ffd700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab809acc0 0x7f8ab809b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab7fff700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab809b6e0 0x7f8ab80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 -- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab809b6e0 msgr2=0x7f8ab80b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab809b6e0 0x7f8ab80b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 -- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab809acc0 msgr2=0x7f8ab809b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab809acc0 0x7f8ab809b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.474 7f8ab77fe700 1 -- 172.21.2.117:0/1070325720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80b3f00 con 0x7f8ab808a280 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8ab77fe700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f8aa8001c60 tx=0x7f8aa800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:12.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa800acc0 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa8004140 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8aa8004620 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab80b3f00 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab80b4490 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.475 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8aa8004320 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.476 7f8aaffff700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 0x7f8a9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.476 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8aa80326a0 con 0x7f8ab808a280 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.476 7f8ab6ffd700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 0x7f8a9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:12.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.476 7f8ab6ffd700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 0x7f8a9c01abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:12.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.506 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab8087290 con 0x7f8ab808a280 2021-02-10T11:54:12.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.509 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8aa8000e20 con 0x7f8ab808a280 2021-02-10T11:54:12.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.632 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8ab80ede60 con 0x7f8ab808a280 2021-02-10T11:54:12.635 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.633 7f8aaffff700 1 -- 172.21.2.117:0/1070325720 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v152) v1 ==== 81+0+30766 (secure 0 0 0) 0x7f8aa8030d00 con 0x7f8ab808a280 2021-02-10T11:54:12.635 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:12.635 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:54:12.635 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 152, 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:53:38.470991", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:54:12.636 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 33, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:54:12.637 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:54:12.638 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:54:12.639 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:54:12.640 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.641 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:54:12.642 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:54:12.643 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:54:12.644 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:54:12.645 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:54:12.646 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 32, 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:54:12.647 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:53:35.441170", 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:54:12.648 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:54:12.649 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "151", 2021-02-10T11:54:12.650 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:54:12.651 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:54:12.652 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:54:12.653 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.654 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 33, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:53:36.462661", 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:54:12.655 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "152", 2021-02-10T11:54:12.656 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:54:12.657 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:54:12.658 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:54:12.659 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:54:12.660 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 148, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.661 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.662 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.663 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.664 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.665 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.666 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:54:12.667 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.668 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:54:12.669 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.670 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.671 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.672 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:54:12.673 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.674 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.674 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.674 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.674 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.674 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 150, 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.675 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:54:12.676 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.677 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.678 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.679 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.680 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.681 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.682 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.683 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.684 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.685 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 150, 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.686 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.687 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.688 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.689 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:54:12.690 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.691 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:54:12.693 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.694 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.695 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:54:12.696 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.697 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 150, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.698 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:54:12.699 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:54:12.700 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.701 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.702 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.703 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 148, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.704 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.705 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.706 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.707 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.712 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.713 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:54:12.714 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 140, 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.715 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:54:12.716 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:54:12.717 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:12.718 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:12.719 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.720 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.721 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.722 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.723 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:54:12.724 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:54:12.725 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:12.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:54:12.727 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.634 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 msgr2=0x7f8a9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.634 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 0x7f8a9c01abc0 crc :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.634 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 msgr2=0x7f8ab8099e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f8aa8001c60 tx=0x7f8aa800a8d0).stop 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 shutdown_connections 2021-02-10T11:54:12.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8ab809b6e0 0x7f8ab80b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8ab809acc0 0x7f8ab809b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8a9c0186c0 0x7f8a9c01abc0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 --2- 172.21.2.117:0/1070325720 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8ab808a280 0x7f8ab8099e00 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 >> 172.21.2.117:0/1070325720 conn(0x7f8ab80874c0 msgr2=0x7f8ab8091940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 shutdown_connections 2021-02-10T11:54:12.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:12.635 7f8abdbae700 1 -- 172.21.2.117:0/1070325720 wait complete. 2021-02-10T11:54:12.730 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:13.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.140 7f2fc9245700 1 Processor -- start 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.140 7f2fc9245700 1 -- start start 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc9245700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2fc40d18e0 0x7f2fc40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc9245700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 0x7f2fc40eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc9245700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc40ede60 con 0x7f2fc40ee3e0 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc9245700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40eba20 con 0x7f2fc40d18e0 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc9245700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40d1ce0 con 0x7f2fc40eb000 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc2d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2fc40d18e0 0x7f2fc40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc359e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2fc40ee3e0 0x7f2fc40ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58000/0 (socket says 172.21.2.117:58000) 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc359e700 1 -- 172.21.2.117:0/302849988 learned_addr learned my addr 172.21.2.117:0/302849988 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc259c700 1 --2- 172.21.2.117:0/302849988 >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 0x7f2fc40eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc2d9d700 1 -- 172.21.2.117:0/302849988 >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 msgr2=0x7f2fc40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:13.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc2d9d700 1 --2- 172.21.2.117:0/302849988 >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 0x7f2fc40eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc2d9d700 1 -- 172.21.2.117:0/302849988 >> v1:172.21.2.117:6789/0 conn(0x7f2fc40ee3e0 legacy=0x7f2fc40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.141 7f2fc1d9b700 1 -- 172.21.2.117:0/302849988 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177458845 0 0) 0x7f2fbc001500 con 0x7f2fc40ee3e0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc2d9d700 1 -- 172.21.2.117:0/302849988 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40eefd0 con 0x7f2fc40d18e0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc259c700 1 --2- 172.21.2.117:0/302849988 >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 0x7f2fc40eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc2d9d700 1 --2- 172.21.2.117:0/302849988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc40d18e0 0x7f2fc40ea9e0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f2fac006650 tx=0x7f2fac009a80).ready entity=mon.0 client_cookie=561c51bc620474ea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc1d9b700 1 -- 172.21.2.117:0/302849988 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fac00c0c0 con 0x7f2fc40d18e0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc1d9b700 1 -- 172.21.2.117:0/302849988 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac00c340 con 0x7f2fc40d18e0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc1d9b700 1 -- 172.21.2.117:0/302849988 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fac00c860 con 0x7f2fc40d18e0 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc9245700 1 -- 172.21.2.117:0/302849988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc40d18e0 msgr2=0x7f2fc40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.142 7f2fc9245700 1 --2- 172.21.2.117:0/302849988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc40d18e0 0x7f2fc40ea9e0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 rx=0x7f2fac006650 tx=0x7f2fac009a80).stop 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 -- 172.21.2.117:0/302849988 shutdown_connections 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 --2- 172.21.2.117:0/302849988 >> v2:172.21.2.125:3300/0 conn(0x7f2fc40eb000 0x7f2fc40eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 --2- 172.21.2.117:0/302849988 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc40d18e0 0x7f2fc40ea9e0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 -- 172.21.2.117:0/302849988 >> 172.21.2.117:0/302849988 conn(0x7f2fc40ced60 msgr2=0x7f2fc40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:13.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 -- 172.21.2.117:0/302849988 shutdown_connections 2021-02-10T11:54:13.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.143 7f2fc9245700 1 -- 172.21.2.117:0/302849988 wait complete. 2021-02-10T11:54:13.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.157 7f2fc9245700 1 Processor -- start 2021-02-10T11:54:13.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.157 7f2fc9245700 1 -- start start 2021-02-10T11:54:13.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc4091b00 0x7f2fc4091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc4096270 0x7f2fc4196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40967f0 con 0x7f2fc4095850 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc41972b0 con 0x7f2fc4091b00 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc9245700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc4197490 con 0x7f2fc4096270 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57754/0 (socket says 172.21.2.117:57754) 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 -- 172.21.2.117:0/2059552892 learned_addr learned my addr 172.21.2.117:0/2059552892 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 -- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc4096270 msgr2=0x7f2fc4196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc4096270 0x7f2fc4196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 -- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc4091b00 msgr2=0x7f2fc4091f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc4091b00 0x7f2fc4091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 -- 172.21.2.117:0/2059552892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc41976b0 con 0x7f2fc4095850 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.158 7f2fc259c700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 rx=0x7f2fb4006f20 tx=0x7f2fb400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb400bd80 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4000c90 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc41976b0 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fb40022b0 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc41976b0 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2fb400a000 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 0x7f2fa801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.159 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2fb4032760 con 0x7f2fc4095850 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.160 7f2fc2d9d700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 0x7f2fa801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.160 7f2fc2d9d700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 0x7f2fa801abc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.190 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc4087290 con 0x7f2fc4095850 2021-02-10T11:54:13.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.193 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2fb4006bb0 con 0x7f2fc4095850 2021-02-10T11:54:13.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.321 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc40ede60 con 0x7f2fc4095850 2021-02-10T11:54:13.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.321 7f2fbb7fe700 1 -- 172.21.2.117:0/2059552892 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+2102 (secure 0 0 0) 0x7f2fb400bd80 con 0x7f2fc4095850 2021-02-10T11:54:13.324 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:13.324 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":129,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:53:39.473419","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8737},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8737":{"gid":8737,"name":"b","rank":0,"incarnation":126,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/277081047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":277081047},{"type":"v1","addr":"172.21.2.125:6835","nonce":277081047}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:54:13.324 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 129 2021-02-10T11:54:13.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.323 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 msgr2=0x7f2fa801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.323 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 0x7f2fa801abc0 crc :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.323 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 msgr2=0x7f2fc4095c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.323 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 rx=0x7f2fb4006f20 tx=0x7f2fb400b990).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 shutdown_connections 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fa8018730 0x7f2fa801abc0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc4095850 0x7f2fc4095c50 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc4096270 0x7f2fc4196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 --2- 172.21.2.117:0/2059552892 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc4091b00 0x7f2fc4091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 >> 172.21.2.117:0/2059552892 conn(0x7f2fc41a3620 msgr2=0x7f2fc4092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 shutdown_connections 2021-02-10T11:54:13.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.324 7f2fc9245700 1 -- 172.21.2.117:0/2059552892 wait complete. 2021-02-10T11:54:13.334 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:54:13.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.746 7fa299d74700 1 Processor -- start 2021-02-10T11:54:13.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.746 7fa299d74700 1 -- start start 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2941b07f0 0x7fa2940e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2940e7d70 0x7fa2940e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa2940e8790 0x7fa2940e8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940e92f0 con 0x7fa2940e7d70 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2940e5ce0 con 0x7fa2940e8790 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa299d74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940e77f0 con 0x7fa2941b07f0 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2940e7d70 0x7fa2940e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2940e7d70 0x7fa2940e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57760/0 (socket says 172.21.2.117:57760) 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 -- 172.21.2.117:0/1208509750 learned_addr learned my addr 172.21.2.117:0/1208509750 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 -- 172.21.2.117:0/1208509750 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2941b07f0 msgr2=0x7fa2940e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 --2- 172.21.2.117:0/1208509750 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2941b07f0 0x7fa2940e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 -- 172.21.2.117:0/1208509750 >> v2:172.21.2.125:3300/0 conn(0x7fa2940e8790 msgr2=0x7fa2940e8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:13.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 --2- 172.21.2.117:0/1208509750 >> v2:172.21.2.125:3300/0 conn(0x7fa2940e8790 0x7fa2940e8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.747 7fa292ffd700 1 -- 172.21.2.117:0/1208509750 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2940e54b0 con 0x7fa2940e7d70 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa292ffd700 1 --2- 172.21.2.117:0/1208509750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940e7d70 0x7fa2940e8170 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7fa284006700 tx=0x7fa284009b80).ready entity=mon.0 client_cookie=3ea5eefe0543d2fb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa2927fc700 1 -- 172.21.2.117:0/1208509750 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28400c0c0 con 0x7fa2940e7d70 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa2927fc700 1 -- 172.21.2.117:0/1208509750 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28400c340 con 0x7fa2940e7d70 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa2927fc700 1 -- 172.21.2.117:0/1208509750 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28400c860 con 0x7fa2940e7d70 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 -- 172.21.2.117:0/1208509750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940e7d70 msgr2=0x7fa2940e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 --2- 172.21.2.117:0/1208509750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940e7d70 0x7fa2940e8170 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 rx=0x7fa284006700 tx=0x7fa284009b80).stop 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 -- 172.21.2.117:0/1208509750 shutdown_connections 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 --2- 172.21.2.117:0/1208509750 >> v2:172.21.2.125:3300/0 conn(0x7fa2940e8790 0x7fa2940e8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 --2- 172.21.2.117:0/1208509750 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2940e7d70 0x7fa2940e8170 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 --2- 172.21.2.117:0/1208509750 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2941b07f0 0x7fa2940e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.748 7fa299d74700 1 -- 172.21.2.117:0/1208509750 >> 172.21.2.117:0/1208509750 conn(0x7fa2940ce500 msgr2=0x7fa2940e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:13.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.749 7fa299d74700 1 -- 172.21.2.117:0/1208509750 shutdown_connections 2021-02-10T11:54:13.767 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.749 7fa299d74700 1 -- 172.21.2.117:0/1208509750 wait complete. 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.763 7fa299d74700 1 Processor -- start 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.763 7fa299d74700 1 -- start start 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2940a70c0 0x7fa2940a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29419d3e0 0x7fa29419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.768 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2940a7760 con 0x7fa294110bd0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29419de00 con 0x7fa29419d3e0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa299d74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29419dfe0 con 0x7fa2940a70c0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57768/0 (socket says 172.21.2.117:57768) 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa2937fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2940a70c0 0x7fa2940a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 -- 172.21.2.117:0/3201806055 learned_addr learned my addr 172.21.2.117:0/3201806055 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa293fff700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29419d3e0 0x7fa29419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 -- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2940a70c0 msgr2=0x7fa2940a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2940a70c0 0x7fa2940a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 -- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29419d3e0 msgr2=0x7fa29419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29419d3e0 0x7fa29419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 -- 172.21.2.117:0/3201806055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29419e200 con 0x7fa294110bd0 2021-02-10T11:54:13.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.764 7fa292ffd700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 rx=0x7fa28400b5e0 tx=0x7fa28400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28400bd70 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa284002030 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa284002550 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa299d74700 1 -- 172.21.2.117:0/3201806055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29419e200 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa299d74700 1 -- 172.21.2.117:0/3201806055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2940a7fd0 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa284000c90 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 0x7fa27801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.765 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa284032730 con 0x7fa294110bd0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.766 7fa2937fe700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 0x7fa27801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:13.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.767 7fa2937fe700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 0x7fa27801abc0 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:13.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.796 7fa299d74700 1 -- 172.21.2.117:0/3201806055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2941b8070 con 0x7fa294110bd0 2021-02-10T11:54:13.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.799 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa28402df40 con 0x7fa294110bd0 2021-02-10T11:54:13.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:13.926 7fa299d74700 1 -- 172.21.2.117:0/3201806055 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa2940e77f0 con 0x7fa294110bd0 2021-02-10T11:54:14.693 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:54:14.692 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:54:14.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.697 7fa28bfff700 1 -- 172.21.2.117:0/3201806055 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v130) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa28402df40 con 0x7fa294110bd0 2021-02-10T11:54:14.743 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:54:14.746 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:54:14.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 -- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 msgr2=0x7fa27801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:14.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 0x7fa27801abc0 crc :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:14.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 -- 172.21.2.117:0/3201806055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 msgr2=0x7fa294110fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:14.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 rx=0x7fa28400b5e0 tx=0x7fa28400b980).stop 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 -- 172.21.2.117:0/3201806055 shutdown_connections 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2780186c0 0x7fa27801abc0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa294110bd0 0x7fa294110fd0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa29419d3e0 0x7fa29419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 --2- 172.21.2.117:0/3201806055 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2940a70c0 0x7fa2940a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.746 7fa299d74700 1 -- 172.21.2.117:0/3201806055 >> 172.21.2.117:0/3201806055 conn(0x7fa2940bff40 msgr2=0x7fa294098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.747 7fa299d74700 1 -- 172.21.2.117:0/3201806055 shutdown_connections 2021-02-10T11:54:14.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:14.747 7fa299d74700 1 -- 172.21.2.117:0/3201806055 wait complete. 2021-02-10T11:54:14.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 Processor -- start 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 -- start start 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcafc0d18e0 0x7fcafc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 0x7fcafc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcafc0ede60 con 0x7fcafc0ee3e0 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcafc0eba90 con 0x7fcafc0d18e0 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.167 7fcb01bf3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0d1ce0 con 0x7fcafc0eb000 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafbfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcafc0ee3e0 0x7fcafc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:50950/0 (socket says 172.21.2.117:50950) 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafb7fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcafc0d18e0 0x7fcafc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafbfff700 1 -- 172.21.2.117:0/2648751783 learned_addr learned my addr 172.21.2.117:0/2648751783 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:15.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafaffd700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 0x7fcafc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafb7fe700 1 -- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 msgr2=0x7fcafc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafb7fe700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 0x7fcafc0eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafb7fe700 1 -- 172.21.2.117:0/2648751783 >> v1:172.21.2.125:6789/0 conn(0x7fcafc0ee3e0 legacy=0x7fcafc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.168 7fcafb7fe700 1 -- 172.21.2.117:0/2648751783 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0eefd0 con 0x7fcafc0d18e0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcafaffd700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 0x7fcafc0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcafb7fe700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc0d18e0 0x7fcafc0ea9e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7fcae4006650 tx=0x7fcae4009a80).ready entity=mon.1 client_cookie=e37f20fc50a43855 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcafa7fc700 1 -- 172.21.2.117:0/2648751783 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcae400c0c0 con 0x7fcafc0d18e0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcafa7fc700 1 -- 172.21.2.117:0/2648751783 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae400c340 con 0x7fcafc0d18e0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcafa7fc700 1 -- 172.21.2.117:0/2648751783 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcae400c860 con 0x7fcafc0d18e0 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcb01bf3700 1 -- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc0d18e0 msgr2=0x7fcafc0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.169 7fcb01bf3700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc0d18e0 0x7fcafc0ea9e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7fcae4006650 tx=0x7fcae4009a80).stop 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 -- 172.21.2.117:0/2648751783 shutdown_connections 2021-02-10T11:54:15.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc0eb000 0x7fcafc0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 --2- 172.21.2.117:0/2648751783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc0d18e0 0x7fcafc0ea9e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 -- 172.21.2.117:0/2648751783 >> 172.21.2.117:0/2648751783 conn(0x7fcafc0ced60 msgr2=0x7fcafc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:15.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 -- 172.21.2.117:0/2648751783 shutdown_connections 2021-02-10T11:54:15.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.170 7fcb01bf3700 1 -- 172.21.2.117:0/2648751783 wait complete. 2021-02-10T11:54:15.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.184 7fcb01bf3700 1 Processor -- start 2021-02-10T11:54:15.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.184 7fcb01bf3700 1 -- start start 2021-02-10T11:54:15.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc095850 0x7fcafc095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc096270 0x7fcafc196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0967f0 con 0x7fcafc091b00 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1972b0 con 0x7fcafc095850 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcb01bf3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc197490 con 0x7fcafc096270 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcafb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcafb7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57778/0 (socket says 172.21.2.117:57778) 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.185 7fcafb7fe700 1 -- 172.21.2.117:0/3929700141 learned_addr learned my addr 172.21.2.117:0/3929700141 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 -- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc096270 msgr2=0x7fcafc196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc096270 0x7fcafc196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 -- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc095850 msgr2=0x7fcafc095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc095850 0x7fcafc095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 -- 172.21.2.117:0/3929700141 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc1976b0 con 0x7fcafc091b00 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcafb7fe700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7fcae4009630 tx=0x7fcae400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcae40093a0 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae400bd70 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcae4005560 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc1976b0 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.186 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc197c40 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.187 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcae400a000 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.187 7fcaf3fff700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 0x7fcae001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.187 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fcae4032530 con 0x7fcafc091b00 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.187 7fcafaffd700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 0x7fcae001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.188 7fcafaffd700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 0x7fcae001abc0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.217 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc087290 con 0x7fcafc091b00 2021-02-10T11:54:15.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.220 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcae4020f20 con 0x7fcafc091b00 2021-02-10T11:54:15.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.347 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcafc0ede60 con 0x7fcafc091b00 2021-02-10T11:54:15.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.347 7fcaf3fff700 1 -- 172.21.2.117:0/3929700141 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+1760 (secure 0 0 0) 0x7fcae4020f20 con 0x7fcafc091b00 2021-02-10T11:54:15.349 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 130 2021-02-10T11:54:15.349 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:15.349 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":130,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12057,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/803941046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":803941046},{"type":"v1","addr":"172.21.2.117:6833","nonce":803941046}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:53:37.468914","modified":"2021-02-10 11:54:14.692581","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":153,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":16}]} 2021-02-10T11:54:15.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 msgr2=0x7fcae001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 0x7fcae001abc0 crc :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 msgr2=0x7fcafc091f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 rx=0x7fcae4009630 tx=0x7fcae400b980).stop 2021-02-10T11:54:15.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 shutdown_connections 2021-02-10T11:54:15.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcafc096270 0x7fcafc196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcafc095850 0x7fcafc095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcae00186c0 0x7fcae001abc0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 --2- 172.21.2.117:0/3929700141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcafc091b00 0x7fcafc091f00 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.349 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 >> 172.21.2.117:0/3929700141 conn(0x7fcafc1a3620 msgr2=0x7fcafc092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.350 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 shutdown_connections 2021-02-10T11:54:15.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.350 7fcb01bf3700 1 -- 172.21.2.117:0/3929700141 wait complete. 2021-02-10T11:54:15.359 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:54:15.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.775 7f80e46ae700 1 Processor -- start 2021-02-10T11:54:15.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.775 7f80e46ae700 1 -- start start 2021-02-10T11:54:15.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e46ae700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f80dc1b07f0 0x7f80dc0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e46ae700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80dc0e8d80 0x7f80dc0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e46ae700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc0e77f0 con 0x7f80dc0e6170 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e46ae700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc0e5ce0 con 0x7f80dc0e8d80 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e46ae700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc0e54b0 con 0x7f80dc1b07f0 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80dc0e8d80 0x7f80dc0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f80dc0e8d80 0x7f80dc0e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57784/0 (socket says 172.21.2.117:57784) 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 -- 172.21.2.117:0/2253648378 learned_addr learned my addr 172.21.2.117:0/2253648378 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e24ca700 1 --2- 172.21.2.117:0/2253648378 >> v2:172.21.2.125:3300/0 conn(0x7f80dc1b07f0 0x7f80dc0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 -- 172.21.2.117:0/2253648378 >> v2:172.21.2.125:3300/0 conn(0x7f80dc1b07f0 msgr2=0x7f80dc0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 --2- 172.21.2.117:0/2253648378 >> v2:172.21.2.125:3300/0 conn(0x7f80dc1b07f0 0x7f80dc0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 -- 172.21.2.117:0/2253648378 >> v1:172.21.2.117:6789/0 conn(0x7f80dc0e6170 legacy=0x7f80dc0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e14c8700 1 -- 172.21.2.117:0/2253648378 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848747834 0 0) 0x7f80d8000ff0 con 0x7f80dc0e6170 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 -- 172.21.2.117:0/2253648378 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc0e9180 con 0x7f80dc0e8d80 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.776 7f80e2ccb700 1 --2- 172.21.2.117:0/2253648378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc0e8d80 0x7f80dc0e6d60 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 rx=0x7f80d4006f00 tx=0x7f80d4009f10).ready entity=mon.0 client_cookie=d5d18ff10b3f5a7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e14c8700 1 -- 172.21.2.117:0/2253648378 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d400b3b0 con 0x7f80dc0e8d80 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e14c8700 1 -- 172.21.2.117:0/2253648378 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d400b630 con 0x7f80dc0e8d80 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e14c8700 1 -- 172.21.2.117:0/2253648378 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d400bb50 con 0x7f80dc0e8d80 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 -- 172.21.2.117:0/2253648378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc0e8d80 msgr2=0x7f80dc0e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 --2- 172.21.2.117:0/2253648378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc0e8d80 0x7f80dc0e6d60 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 rx=0x7f80d4006f00 tx=0x7f80d4009f10).stop 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 -- 172.21.2.117:0/2253648378 shutdown_connections 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 --2- 172.21.2.117:0/2253648378 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc0e8d80 0x7f80dc0e6d60 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 --2- 172.21.2.117:0/2253648378 >> v2:172.21.2.125:3300/0 conn(0x7f80dc1b07f0 0x7f80dc0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 -- 172.21.2.117:0/2253648378 >> 172.21.2.117:0/2253648378 conn(0x7f80dc0ce500 msgr2=0x7f80dc0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:15.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 -- 172.21.2.117:0/2253648378 shutdown_connections 2021-02-10T11:54:15.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.777 7f80e46ae700 1 -- 172.21.2.117:0/2253648378 wait complete. 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 Processor -- start 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 -- start start 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 0x7f80dc0ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 0x7f80dc197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc198390 con 0x7f80dc10b430 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc0aeac0 con 0x7f80dc198800 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e46ae700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc0aece0 con 0x7f80dc197970 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.792 7f80e1cc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57788/0 (socket says 172.21.2.117:57788) 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 -- 172.21.2.117:0/2613381980 learned_addr learned my addr 172.21.2.117:0/2613381980 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e24ca700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 0x7f80dc0ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e2ccb700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 0x7f80dc197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 -- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 msgr2=0x7f80dc197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 0x7f80dc197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 -- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 msgr2=0x7f80dc0ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 0x7f80dc0ae4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:15.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 -- 172.21.2.117:0/2613381980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc07eea0 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e2ccb700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 0x7f80dc197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e24ca700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 0x7f80dc0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e1cc9700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f80d800a570 tx=0x7f80d800a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d800c0d0 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d800c3d0 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc07eea0 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f80d800c820 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.793 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc07eea0 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.794 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f80d8006540 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.794 7f80ceffd700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 0x7f80c401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.794 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f80d8032810 con 0x7f80dc10b430 2021-02-10T11:54:15.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.794 7f80e24ca700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 0x7f80c401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:15.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.794 7f80e24ca700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 0x7f80c401ac10 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:15.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.825 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80dc07e840 con 0x7f80dc10b430 2021-02-10T11:54:15.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.828 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f80d8000990 con 0x7f80dc10b430 2021-02-10T11:54:15.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:15.954 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f80dc0e77f0 con 0x7f80dc10b430 2021-02-10T11:54:16.529 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:54:16.532 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:54:16.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.706 7f80ceffd700 1 -- 172.21.2.117:0/2613381980 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 104+0+0 (secure 0 0 0) 0x7f80d8006540 con 0x7f80dc10b430 2021-02-10T11:54:16.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 msgr2=0x7f80c401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:16.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 0x7f80c401ac10 crc :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 msgr2=0x7f80dc197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 rx=0x7f80d800a570 tx=0x7f80d800a910).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 shutdown_connections 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f80c4018730 0x7f80c401ac10 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f80dc10b430 0x7f80dc197350 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f80dc197970 0x7f80dc197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 --2- 172.21.2.117:0/2613381980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f80dc198800 0x7f80dc0ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:16.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 >> 172.21.2.117:0/2613381980 conn(0x7f80dc1998b0 msgr2=0x7f80dc1a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:16.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 shutdown_connections 2021-02-10T11:54:16.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:16.726 7f80e46ae700 1 -- 172.21.2.117:0/2613381980 wait complete. 2021-02-10T11:54:16.736 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:54:17.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.149 7f687c428700 1 Processor -- start 2021-02-10T11:54:17.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.149 7f687c428700 1 -- start start 2021-02-10T11:54:17.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687c428700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68740e5240 0x7f6874086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687c428700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6874084c60 0x7f6874085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687c428700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6874085930 con 0x7f68740e4e40 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687c428700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740e5cc0 con 0x7f68740e5240 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687c428700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68740856b0 con 0x7f6874084c60 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f6879a43700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f68740e5240 0x7f6874086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687a244700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f68740e4e40 0x7f68741b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58044/0 (socket says 172.21.2.117:58044) 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.150 7f687a244700 1 -- 172.21.2.117:0/3724194839 learned_addr learned my addr 172.21.2.117:0/3724194839 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879a43700 1 -- 172.21.2.117:0/3724194839 >> v2:172.21.2.125:3300/0 conn(0x7f6874084c60 msgr2=0x7f6874085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879a43700 1 --2- 172.21.2.117:0/3724194839 >> v2:172.21.2.125:3300/0 conn(0x7f6874084c60 0x7f6874085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879a43700 1 -- 172.21.2.117:0/3724194839 >> v1:172.21.2.117:6789/0 conn(0x7f68740e4e40 legacy=0x7f68741b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879a43700 1 -- 172.21.2.117:0/3724194839 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68741b09e0 con 0x7f68740e5240 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879a43700 1 --2- 172.21.2.117:0/3724194839 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68740e5240 0x7f6874086ec0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f6870006650 tx=0x7f6870009a80).ready entity=mon.0 client_cookie=d081a7719e2ce1ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879242700 1 -- 172.21.2.117:0/3724194839 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68700009b0 con 0x7f68740e5240 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879242700 1 -- 172.21.2.117:0/3724194839 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687000c0c0 con 0x7f68740e5240 2021-02-10T11:54:17.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f6879242700 1 -- 172.21.2.117:0/3724194839 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f687000c520 con 0x7f68740e5240 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f687c428700 1 -- 172.21.2.117:0/3724194839 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68740e5240 msgr2=0x7f6874086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.151 7f687c428700 1 --2- 172.21.2.117:0/3724194839 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68740e5240 0x7f6874086ec0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f6870006650 tx=0x7f6870009a80).stop 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 -- 172.21.2.117:0/3724194839 shutdown_connections 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 --2- 172.21.2.117:0/3724194839 >> v2:172.21.2.125:3300/0 conn(0x7f6874084c60 0x7f6874085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 --2- 172.21.2.117:0/3724194839 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68740e5240 0x7f6874086ec0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 -- 172.21.2.117:0/3724194839 >> 172.21.2.117:0/3724194839 conn(0x7f68740ce4e0 msgr2=0x7f68740e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 -- 172.21.2.117:0/3724194839 shutdown_connections 2021-02-10T11:54:17.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.152 7f687c428700 1 -- 172.21.2.117:0/3724194839 wait complete. 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 Processor -- start 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 -- start start 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68741b1f90 0x7f68741b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6874094390 0x7f6874094790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741110e0 con 0x7f68741b1570 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68741b0ff0 con 0x7f6874094390 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687c428700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6874094db0 con 0x7f68741b1f90 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687a244700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687a244700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57800/0 (socket says 172.21.2.117:57800) 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.167 7f687a244700 1 -- 172.21.2.117:0/418399224 learned_addr learned my addr 172.21.2.117:0/418399224 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:17.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 -- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68741b1f90 msgr2=0x7f68741b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68741b1f90 0x7f68741b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 -- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6874094390 msgr2=0x7f6874094790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6874094390 0x7f6874094790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 -- 172.21.2.117:0/418399224 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874094f90 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687a244700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f6868001c60 tx=0x7f686800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f68680010b0 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868004140 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6868004660 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687c428700 1 -- 172.21.2.117:0/418399224 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6874094f90 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.168 7f687c428700 1 -- 172.21.2.117:0/418399224 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6874095520 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.169 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f68680010b0 con 0x7f68741b1570 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.169 7f6866ffd700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 0x7f685c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:17.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.169 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f68680325b0 con 0x7f68741b1570 2021-02-10T11:54:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.169 7f6879a43700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 0x7f685c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:17.174 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.170 7f6879a43700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 0x7f685c01abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:17.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.200 7f687c428700 1 -- 172.21.2.117:0/418399224 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687410c2f0 con 0x7f68741b1570 2021-02-10T11:54:17.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.202 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6868030d30 con 0x7f68741b1570 2021-02-10T11:54:17.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.328 7f687c428700 1 -- 172.21.2.117:0/418399224 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6874085930 con 0x7f68741b1570 2021-02-10T11:54:17.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.710 7f6866ffd700 1 -- 172.21.2.117:0/418399224 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v154) v1 ==== 185+0+0 (secure 0 0 0) 0x7f686801f4e0 con 0x7f68741b1570 2021-02-10T11:54:17.744 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:54:17.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.744 7f687c428700 1 -- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 msgr2=0x7f685c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.744 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 0x7f685c01abc0 crc :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:17.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.744 7f687c428700 1 -- 172.21.2.117:0/418399224 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 msgr2=0x7f68741b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:17.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.744 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 rx=0x7f6868001c60 tx=0x7f686800a8d0).stop 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 -- 172.21.2.117:0/418399224 shutdown_connections 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6874094390 0x7f6874094790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f68741b1f90 0x7f68741b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f685c0186c0 0x7f685c01abc0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 --2- 172.21.2.117:0/418399224 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f68741b1570 0x7f68741b1970 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 -- 172.21.2.117:0/418399224 >> 172.21.2.117:0/418399224 conn(0x7f68741b3150 msgr2=0x7f68741ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 -- 172.21.2.117:0/418399224 shutdown_connections 2021-02-10T11:54:17.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:17.745 7f687c428700 1 -- 172.21.2.117:0/418399224 wait complete. 2021-02-10T11:54:17.755 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:54:18.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 Processor -- start 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 -- start start 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd1e8085450 0x7fd1e8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8085930 con 0x7fd1e80e5240 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e81b09e0 con 0x7fd1e80e4e40 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.170 7fd1ee382700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80e5cc0 con 0x7fd1e8085450 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd1e80e5240 0x7fd1e8084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58060/0 (socket says 172.21.2.117:58060) 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e77fe700 1 -- 172.21.2.117:0/83591740 learned_addr learned my addr 172.21.2.117:0/83591740 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257457180 0 0) 0x7fd1dc0010b0 con 0x7fd1e80e5240 2021-02-10T11:54:18.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d0002fe0 con 0x7fd1e80e5240 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1ec99f700 1 --2- 172.21.2.117:0/83591740 >> v2:172.21.2.125:3300/0 conn(0x7fd1e8085450 0x7fd1e8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1870955869 0 0) 0x7fd1dc0010b0 con 0x7fd1e80e5240 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 >> v2:172.21.2.125:3300/0 conn(0x7fd1e8085450 msgr2=0x7fd1e8086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 --2- 172.21.2.117:0/83591740 >> v2:172.21.2.125:3300/0 conn(0x7fd1e8085450 0x7fd1e8086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 >> v1:172.21.2.125:6789/0 conn(0x7fd1e80e4e40 legacy=0x7fd1e81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e8084840 con 0x7fd1e80e5240 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd1dc0015b0 con 0x7fd1e80e5240 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 >> v1:172.21.2.117:6789/0 conn(0x7fd1e80e5240 legacy=0x7fd1e8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1d0003a60 0x7fd1d0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 0x7fd1d0008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1d0009590 0x7fd1d000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d000c300 con 0x7fd1d0006790 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.171 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d000c520 con 0x7fd1d0009590 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d0050750 con 0x7fd1d0003a60 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 0x7fd1d0008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e7fff700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1d0003a60 0x7fd1d0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1ec99f700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1d0009590 0x7fd1d000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 -- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1d0003a60 msgr2=0x7fd1d0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1d0003a60 0x7fd1d0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 -- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1d0009590 msgr2=0x7fd1d000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1d0009590 0x7fd1d000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1ec99f700 1 -- 172.21.2.117:0/83591740 reap_dead start 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 -- 172.21.2.117:0/83591740 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d00509b0 con 0x7fd1d0006790 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e77fe700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 0x7fd1d0008c90 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7fd1dc000b50 tx=0x7fd1dc007d40).ready entity=mon.0 client_cookie=3e5a035595babc1e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc00a310 con 0x7fd1d0006790 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc00a5d0 con 0x7fd1d0006790 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1e6ffd700 1 -- 172.21.2.117:0/83591740 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1dc00aa50 con 0x7fd1d0006790 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1ee382700 1 -- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 msgr2=0x7fd1d0008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.172 7fd1ee382700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 0x7fd1d0008c90 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7fd1dc000b50 tx=0x7fd1dc007d40).stop 2021-02-10T11:54:18.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.173 7fd1ee382700 1 -- 172.21.2.117:0/83591740 shutdown_connections 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.173 7fd1ee382700 1 --2- 172.21.2.117:0/83591740 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1d0006790 0x7fd1d0008c90 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.173 7fd1ee382700 1 -- 172.21.2.117:0/83591740 >> 172.21.2.117:0/83591740 conn(0x7fd1e80ce4e0 msgr2=0x7fd1e80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.173 7fd1ee382700 1 -- 172.21.2.117:0/83591740 shutdown_connections 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.173 7fd1ee382700 1 -- 172.21.2.117:0/83591740 wait complete. 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 Processor -- start 2021-02-10T11:54:18.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 -- start start 2021-02-10T11:54:18.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1e80b28f0 0x7fd1e80c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1e80a36b0 0x7fd1e80a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80c2cf0 con 0x7fd1e80a36b0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80a06d0 con 0x7fd1e80b28f0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ee382700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80a40d0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1e7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1e7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:55500/0 (socket says 172.21.2.117:55500) 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1e7fff700 1 -- 172.21.2.117:0/1435327579 learned_addr learned my addr 172.21.2.117:0/1435327579 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1ec99f700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1e80a36b0 0x7fd1e80a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.188 7fd1e77fe700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1e80b28f0 0x7fd1e80c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 -- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1e80b28f0 msgr2=0x7fd1e80c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1e80b28f0 0x7fd1e80c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 -- 172.21.2.117:0/1435327579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1e80a36b0 msgr2=0x7fd1e80a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1e80a36b0 0x7fd1e80a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 -- 172.21.2.117:0/1435327579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80a42f0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e7fff700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd1d800b610 tx=0x7fd1d800b9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d8000960 con 0x7fd1e80b24f0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d80063c0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd1d8000ca0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e80a42f0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.189 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80a42f0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.190 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd1d8000ca0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.190 7fd1e4ff9700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 0x7fd1c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:18.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.190 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5568+0+0 (secure 0 0 0) 0x7fd1d80323c0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.190 7fd1e77fe700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 0x7fd1c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:18.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.190 7fd1e77fe700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 0x7fd1c801abc0 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:18.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.220 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e80c2cf0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.223 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd1d8006bb0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.351 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1e80c2cf0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.716 7fd1e4ff9700 1 -- 172.21.2.117:0/1435327579 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v155) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd1d8000ca0 con 0x7fd1e80b24f0 2021-02-10T11:54:18.719 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:54:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.719 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 msgr2=0x7fd1c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.719 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 0x7fd1c801abc0 crc :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.719 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 msgr2=0x7fd1e80c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.719 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fd1d800b610 tx=0x7fd1d800b9b0).stop 2021-02-10T11:54:18.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 shutdown_connections 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1c80186c0 0x7fd1c801abc0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd1e80a36b0 0x7fd1e80a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd1e80b28f0 0x7fd1e80c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 --2- 172.21.2.117:0/1435327579 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd1e80b24f0 0x7fd1e80c20b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 >> 172.21.2.117:0/1435327579 conn(0x7fd1e80a0e40 msgr2=0x7fd1e80c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 shutdown_connections 2021-02-10T11:54:18.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:18.720 7fd1ee382700 1 -- 172.21.2.117:0/1435327579 wait complete. 2021-02-10T11:54:18.730 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:54:18.730 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:54:18.730 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:54:18.730 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:54:18.731 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:54:18.731 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:54:18.731 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:54:18.731 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:54:18.736 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:54:18.736 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:54:18.737 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:54:18.737 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:54:18.779 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:54:18.779 DEBUG:teuthology.parallel:result is None 2021-02-10T11:54:18.780 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:54:18.780 DEBUG:teuthology.parallel:result is None 2021-02-10T11:54:18.780 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:54:19.184 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:54:19.186 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.188 7f43a653a700 1 Processor -- start 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.188 7f43a653a700 1 -- start start 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f43a653a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f43a00d15e0 0x7f43a00e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f43a653a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a00e7580 con 0x7f43a01b07f0 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f43a653a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a00e77f0 con 0x7f43a00e7180 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f43a653a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a00e5ce0 con 0x7f43a00d15e0 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f439ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f43a01b07f0 0x7f43a00e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58092/0 (socket says 172.21.2.117:58092) 2021-02-10T11:54:19.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f439ffff700 1 -- 172.21.2.117:0/321596784 learned_addr learned my addr 172.21.2.117:0/321596784 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.189 7f43a4b57700 1 --2- 172.21.2.117:0/321596784 >> v2:172.21.2.125:3300/0 conn(0x7f43a00d15e0 0x7f43a00e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409648001 0 0) 0x7f4394000ff0 con 0x7f43a00e7180 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4388002fe0 con 0x7f43a00e7180 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871678943 0 0) 0x7f43900009b0 con 0x7f43a01b07f0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43880042c0 con 0x7f43a01b07f0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a4b57700 1 -- 172.21.2.117:0/321596784 >> v1:172.21.2.125:6789/0 conn(0x7f43a00e7180 legacy=0x7f43a00e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a4b57700 1 -- 172.21.2.117:0/321596784 >> v1:172.21.2.117:6789/0 conn(0x7f43a01b07f0 legacy=0x7f43a00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a4b57700 1 -- 172.21.2.117:0/321596784 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a00d19e0 con 0x7f43a00d15e0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a4b57700 1 --2- 172.21.2.117:0/321596784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a00d15e0 0x7f43a00e8560 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f4398006e50 tx=0x7f4398009e70).ready entity=mon.1 client_cookie=51fa241bb2c9dd22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f439800b510 con 0x7f43a00d15e0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4398006790 con 0x7f43a00d15e0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f439effd700 1 -- 172.21.2.117:0/321596784 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f439800b940 con 0x7f43a00d15e0 2021-02-10T11:54:19.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a653a700 1 -- 172.21.2.117:0/321596784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a00d15e0 msgr2=0x7f43a00e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.190 7f43a653a700 1 --2- 172.21.2.117:0/321596784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a00d15e0 0x7f43a00e8560 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f4398006e50 tx=0x7f4398009e70).stop 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.191 7f43a653a700 1 -- 172.21.2.117:0/321596784 shutdown_connections 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.191 7f43a653a700 1 --2- 172.21.2.117:0/321596784 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a00d15e0 0x7f43a00e8560 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.191 7f43a653a700 1 -- 172.21.2.117:0/321596784 >> 172.21.2.117:0/321596784 conn(0x7f43a00ce500 msgr2=0x7f43a00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.191 7f43a653a700 1 -- 172.21.2.117:0/321596784 shutdown_connections 2021-02-10T11:54:19.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.191 7f43a653a700 1 -- 172.21.2.117:0/321596784 wait complete. 2021-02-10T11:54:19.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 Processor -- start 2021-02-10T11:54:19.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 -- start start 2021-02-10T11:54:19.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 0x7f43a00b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:19.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:19.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43a019dd10 0x7f43a0093af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a019e110 con 0x7f43a019d2f0 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a00ae1b0 con 0x7f43a0094c60 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f43a653a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a0095060 con 0x7f43a019dd10 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f439f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f439f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57846/0 (socket says 172.21.2.117:57846) 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.206 7f439f7fe700 1 -- 172.21.2.117:0/2271124112 learned_addr learned my addr 172.21.2.117:0/2271124112 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f43a4b57700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43a019dd10 0x7f43a0093af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439ffff700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 0x7f43a00b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 -- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43a019dd10 msgr2=0x7f43a0093af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43a019dd10 0x7f43a0093af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 -- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 msgr2=0x7f43a00b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 0x7f43a00b0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 -- 172.21.2.117:0/2271124112 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0094110 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439ffff700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 0x7f43a00b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439f7fe700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f4394006ed0 tx=0x7f439400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f439400bd50 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394006240 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4394005730 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f43a653a700 1 -- 172.21.2.117:0/2271124112 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a0094110 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.207 7f43a653a700 1 -- 172.21.2.117:0/2271124112 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a00946a0 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.208 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f43940054b0 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.208 7f439cff9700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 0x7f438001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.208 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f43940323c0 con 0x7f43a019d2f0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.208 7f439ffff700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 0x7f438001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:19.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.209 7f439ffff700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 0x7f438001abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.238 7f43a653a700 1 -- 172.21.2.117:0/2271124112 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a00ae1b0 con 0x7f43a019d2f0 2021-02-10T11:54:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.240 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4394000940 con 0x7f43a019d2f0 2021-02-10T11:54:19.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.377 7f43a653a700 1 -- 172.21.2.117:0/2271124112 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f43a00ae1b0 con 0x7f43a019d2f0 2021-02-10T11:54:19.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.721 7f439cff9700 1 -- 172.21.2.117:0/2271124112 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v156) v1 ==== 97+0+0 (secure 0 0 0) 0x7f43940054b0 con 0x7f43a019d2f0 2021-02-10T11:54:19.745 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:54:19.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 -- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 msgr2=0x7f438001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 0x7f438001abc0 crc :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 -- 172.21.2.117:0/2271124112 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 msgr2=0x7f43a019d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:19.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 rx=0x7f4394006ed0 tx=0x7f439400b960).stop 2021-02-10T11:54:19.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 -- 172.21.2.117:0/2271124112 shutdown_connections 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f43800186c0 0x7f438001abc0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f43a019d2f0 0x7f43a019d6f0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.745 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f43a019dd10 0x7f43a0093af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.746 7f43a653a700 1 --2- 172.21.2.117:0/2271124112 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f43a0094c60 0x7f43a00b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.746 7f43a653a700 1 -- 172.21.2.117:0/2271124112 >> 172.21.2.117:0/2271124112 conn(0x7f43a0096150 msgr2=0x7f43a007ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.746 7f43a653a700 1 -- 172.21.2.117:0/2271124112 shutdown_connections 2021-02-10T11:54:19.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:19.746 7f43a653a700 1 -- 172.21.2.117:0/2271124112 wait complete. 2021-02-10T11:54:19.756 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:54:19.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.164 7fee62fa5700 1 Processor -- start 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.164 7fee62fa5700 1 -- start start 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee62fa5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fee5c0e4e40 0x7fee5c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee62fa5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c0d1db0 con 0x7fee5c085450 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee62fa5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5c085930 con 0x7fee5c0e5240 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee62fa5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5c084840 con 0x7fee5c0e4e40 2021-02-10T11:54:20.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee615c2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fee5c085450 0x7fee5c0d05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58104/0 (socket says 172.21.2.117:58104) 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee615c2700 1 -- 172.21.2.117:0/3470943187 learned_addr learned my addr 172.21.2.117:0/3470943187 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee60dc1700 1 --2- 172.21.2.117:0/3470943187 >> v2:172.21.2.117:3300/0 conn(0x7fee5c0e4e40 0x7fee5c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938201271 0 0) 0x7fee54001500 con 0x7fee5c085450 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee48002fe0 con 0x7fee5c085450 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.165 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020090760 0 0) 0x7fee4c000c60 con 0x7fee5c0e5240 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee48004370 con 0x7fee5c0e5240 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee60dc1700 1 -- 172.21.2.117:0/3470943187 >> v1:172.21.2.125:6789/0 conn(0x7fee5c0e5240 legacy=0x7fee5c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee60dc1700 1 -- 172.21.2.117:0/3470943187 >> v1:172.21.2.117:6789/0 conn(0x7fee5c085450 legacy=0x7fee5c0d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee60dc1700 1 -- 172.21.2.117:0/3470943187 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c1b0810 con 0x7fee5c0e4e40 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee60dc1700 1 --2- 172.21.2.117:0/3470943187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c0e4e40 0x7fee5c1b01f0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 rx=0x7fee44006650 tx=0x7fee44009a80).ready entity=mon.0 client_cookie=7e2adfba68e2e512 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4400c0c0 con 0x7fee5c0e4e40 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee4400c340 con 0x7fee5c0e4e40 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee5b7fe700 1 -- 172.21.2.117:0/3470943187 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee4400c860 con 0x7fee5c0e4e40 2021-02-10T11:54:20.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee62fa5700 1 -- 172.21.2.117:0/3470943187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c0e4e40 msgr2=0x7fee5c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.166 7fee62fa5700 1 --2- 172.21.2.117:0/3470943187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c0e4e40 0x7fee5c1b01f0 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 rx=0x7fee44006650 tx=0x7fee44009a80).stop 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.167 7fee62fa5700 1 -- 172.21.2.117:0/3470943187 shutdown_connections 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.167 7fee62fa5700 1 --2- 172.21.2.117:0/3470943187 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c0e4e40 0x7fee5c1b01f0 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.167 7fee62fa5700 1 -- 172.21.2.117:0/3470943187 >> 172.21.2.117:0/3470943187 conn(0x7fee5c0ce4e0 msgr2=0x7fee5c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.167 7fee62fa5700 1 -- 172.21.2.117:0/3470943187 shutdown_connections 2021-02-10T11:54:20.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.167 7fee62fa5700 1 -- 172.21.2.117:0/3470943187 wait complete. 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 Processor -- start 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 -- start start 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 0x7fee5c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee5c189d00 0x7fee5c18a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 0x7fee5c18b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c18d4b0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c18ac30 con 0x7fee5c18a720 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee62fa5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5c19a480 con 0x7fee5c189d00 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee615c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 0x7fee5c18b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee615c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 0x7fee5c18b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:34012/0 (socket says 172.21.2.117:34012) 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee615c2700 1 -- 172.21.2.117:0/2242199090 learned_addr learned my addr 172.21.2.117:0/2242199090 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee60dc1700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 0x7fee5c19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.182 7fee5bfff700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee5c189d00 0x7fee5c18a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 -- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee5c189d00 msgr2=0x7fee5c18a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee5c189d00 0x7fee5c18a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 -- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 msgr2=0x7fee5c18b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 0x7fee5c18b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 -- 172.21.2.117:0/2242199090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5c18b7d0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee60dc1700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 0x7fee5c19e4e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7fee4400b530 tx=0x7fee4400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee44000c90 con 0x7fee5c18d0b0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44002030 con 0x7fee5c18d0b0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5c18b7d0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fee44002490 con 0x7fee5c18d0b0 2021-02-10T11:54:20.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.183 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5c18b7d0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.184 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fee44002210 con 0x7fee5c18d0b0 2021-02-10T11:54:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.184 7fee597fa700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 0x7fee4001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.184 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fee440323a0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.184 7fee5bfff700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 0x7fee4001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.184 7fee5bfff700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 0x7fee4001ac10 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.215 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5c10c260 con 0x7fee5c18d0b0 2021-02-10T11:54:20.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.217 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fee44000960 con 0x7fee5c18d0b0 2021-02-10T11:54:20.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.347 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fee5c085930 con 0x7fee5c18d0b0 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.347 7fee597fa700 1 -- 172.21.2.117:0/2242199090 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v88) v1 ==== 80+0+4513 (secure 0 0 0) 0x7fee4400bcc0 con 0x7fee5c18d0b0 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.349 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:54:20.350 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.351 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.352 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.353 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.354 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.355 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:54:20.356 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.357 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.358 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.359 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:54:20.360 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.361 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:54:20.362 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:54:20.363 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:54:20.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 msgr2=0x7fee4001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 0x7fee4001ac10 crc :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 msgr2=0x7fee5c19e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 0x7fee5c19e4e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 rx=0x7fee4400b530 tx=0x7fee4400b8d0).stop 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 shutdown_connections 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fee5c18a720 0x7fee5c18b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fee5c189d00 0x7fee5c18a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fee40018730 0x7fee4001ac10 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 --2- 172.21.2.117:0/2242199090 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fee5c18d0b0 0x7fee5c19e4e0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 >> 172.21.2.117:0/2242199090 conn(0x7fee5c1a60a0 msgr2=0x7fee5c10c490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 shutdown_connections 2021-02-10T11:54:20.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.349 7fee62fa5700 1 -- 172.21.2.117:0/2242199090 wait complete. 2021-02-10T11:54:20.366 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:20.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.777 7f71512e1700 1 Processor -- start 2021-02-10T11:54:20.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 -- start start 2021-02-10T11:54:20.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f714c0d18e0 0x7f714c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0ef3f0 0x7f714c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0ede60 con 0x7f714c0eb000 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714c0ef7f0 con 0x7f714c0d18e0 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f71512e1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0d1ce0 con 0x7f714c0ef3f0 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f714b07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f714c0d18e0 0x7f714c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f714b07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f714c0d18e0 0x7f714c0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57866/0 (socket says 172.21.2.117:57866) 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f714b07d700 1 -- 172.21.2.117:0/783705266 learned_addr learned my addr 172.21.2.117:0/783705266 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.778 7f714b87e700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0ef3f0 0x7f714c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714b07d700 1 -- 172.21.2.117:0/783705266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0ef3f0 msgr2=0x7f714c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714b07d700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0ef3f0 0x7f714c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714b07d700 1 -- 172.21.2.117:0/783705266 >> v1:172.21.2.125:6789/0 conn(0x7f714c0eb000 legacy=0x7f714c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714b07d700 1 -- 172.21.2.117:0/783705266 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c1b8f90 con 0x7f714c0d18e0 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714b07d700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0d18e0 0x7f714c0ea9e0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f713c006700 tx=0x7f713c009b80).ready entity=mon.0 client_cookie=12280f833f851aca server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714a07b700 1 -- 172.21.2.117:0/783705266 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713c006340 con 0x7f714c0d18e0 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714a07b700 1 -- 172.21.2.117:0/783705266 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c00c0c0 con 0x7f714c0d18e0 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f714a07b700 1 -- 172.21.2.117:0/783705266 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713c00c5e0 con 0x7f714c0d18e0 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f71512e1700 1 -- 172.21.2.117:0/783705266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0d18e0 msgr2=0x7f714c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.779 7f71512e1700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0d18e0 0x7f714c0ea9e0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f713c006700 tx=0x7f713c009b80).stop 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 -- 172.21.2.117:0/783705266 shutdown_connections 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0ef3f0 0x7f714c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 --2- 172.21.2.117:0/783705266 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0d18e0 0x7f714c0ea9e0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 -- 172.21.2.117:0/783705266 >> 172.21.2.117:0/783705266 conn(0x7f714c0ced60 msgr2=0x7f714c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 -- 172.21.2.117:0/783705266 shutdown_connections 2021-02-10T11:54:20.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.780 7f71512e1700 1 -- 172.21.2.117:0/783705266 wait complete. 2021-02-10T11:54:20.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 Processor -- start 2021-02-10T11:54:20.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 -- start start 2021-02-10T11:54:20.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f714c1a33d0 0x7f714c1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0b8d10 0x7f714c0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0c0670 con 0x7f714c0b82f0 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0deb20 con 0x7f714c1a33d0 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f71512e1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c0ded40 con 0x7f714c0b8d10 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f714a87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f714a87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57870/0 (socket says 172.21.2.117:57870) 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f714a87c700 1 -- 172.21.2.117:0/1573325906 learned_addr learned my addr 172.21.2.117:0/1573325906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f714b87e700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0b8d10 0x7f714c0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.795 7f714b07d700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f714c1a33d0 0x7f714c1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 -- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0b8d10 msgr2=0x7f714c0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0b8d10 0x7f714c0b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 -- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f714c1a33d0 msgr2=0x7f714c1a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f714c1a33d0 0x7f714c1a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 -- 172.21.2.117:0/1573325906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c0def60 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f714a87c700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 rx=0x7f7134006ed0 tx=0x7f713400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7134006240 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71340054b0 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f71512e1700 1 -- 172.21.2.117:0/1573325906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c0def60 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f71512e1700 1 -- 172.21.2.117:0/1573325906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c0df4f0 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71340059d0 con 0x7f714c0b82f0 2021-02-10T11:54:20.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7134006030 con 0x7f714c0b82f0 2021-02-10T11:54:20.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.796 7f7143fff700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 0x7f713001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:20.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.797 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f7134032600 con 0x7f714c0b82f0 2021-02-10T11:54:20.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.797 7f714b07d700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 0x7f713001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:20.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.797 7f714b07d700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 0x7f713001abc0 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:20.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.828 7f71512e1700 1 -- 172.21.2.117:0/1573325906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7124002fd0 con 0x7f714c0b82f0 2021-02-10T11:54:20.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.831 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f713401f520 con 0x7f714c0b82f0 2021-02-10T11:54:20.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.954 7f71512e1700 1 -- 172.21.2.117:0/1573325906 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7124002de0 con 0x7f714c0b82f0 2021-02-10T11:54:20.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.955 7f7143fff700 1 -- 172.21.2.117:0/1573325906 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v156) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f7134006030 con 0x7f714c0b82f0 2021-02-10T11:54:20.957 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:20.957 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":156,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:19.715911","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:20.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 msgr2=0x7f713001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 0x7f713001abc0 crc :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 msgr2=0x7f714c0b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 rx=0x7f7134006ed0 tx=0x7f713400b960).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 shutdown_connections 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71300186c0 0x7f713001abc0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f714c0b82f0 0x7f714c0b86f0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f714c0b8d10 0x7f714c0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 --2- 172.21.2.117:0/1573325906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f714c1a33d0 0x7f714c1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:20.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 >> 172.21.2.117:0/1573325906 conn(0x7f714c0874c0 msgr2=0x7f714c096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:20.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 shutdown_connections 2021-02-10T11:54:20.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:20.957 7f71512e1700 1 -- 172.21.2.117:0/1573325906 wait complete. 2021-02-10T11:54:20.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:54:20.968 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:54:21.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.382 7f0e79381700 1 Processor -- start 2021-02-10T11:54:21.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.382 7f0e79381700 1 -- start start 2021-02-10T11:54:21.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e79381700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e740ef3f0 0x7f0e740eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e79381700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e740eb830 0x7f0e741b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e79381700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740ef7f0 con 0x7f0e740d18f0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e79381700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e740eded0 con 0x7f0e740ef3f0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e79381700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e740d1cf0 con 0x7f0e740eb830 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e740ef3f0 0x7f0e740eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0e740ef3f0 0x7f0e740eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57876/0 (socket says 172.21.2.117:57876) 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 -- 172.21.2.117:0/3421917181 learned_addr learned my addr 172.21.2.117:0/3421917181 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 -- 172.21.2.117:0/3421917181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e740eb830 msgr2=0x7f0e741b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 --2- 172.21.2.117:0/3421917181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e740eb830 0x7f0e741b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.383 7f0e7287c700 1 -- 172.21.2.117:0/3421917181 >> v1:172.21.2.117:6789/0 conn(0x7f0e740d18f0 legacy=0x7f0e740eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e7287c700 1 -- 172.21.2.117:0/3421917181 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e741b8fd0 con 0x7f0e740ef3f0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e7287c700 1 --2- 172.21.2.117:0/3421917181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e740ef3f0 0x7f0e740eb210 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7f0e6c006a80 tx=0x7f0e6c009f00).ready entity=mon.0 client_cookie=e8ba927bbee74b0e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e7207b700 1 -- 172.21.2.117:0/3421917181 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e6c00b5c0 con 0x7f0e740ef3f0 2021-02-10T11:54:21.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e7207b700 1 -- 172.21.2.117:0/3421917181 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6c00b8c0 con 0x7f0e740ef3f0 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e7207b700 1 -- 172.21.2.117:0/3421917181 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e6c00bde0 con 0x7f0e740ef3f0 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e79381700 1 -- 172.21.2.117:0/3421917181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e740ef3f0 msgr2=0x7f0e740eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.384 7f0e79381700 1 --2- 172.21.2.117:0/3421917181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e740ef3f0 0x7f0e740eb210 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7f0e6c006a80 tx=0x7f0e6c009f00).stop 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 -- 172.21.2.117:0/3421917181 shutdown_connections 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 --2- 172.21.2.117:0/3421917181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e740eb830 0x7f0e741b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 --2- 172.21.2.117:0/3421917181 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e740ef3f0 0x7f0e740eb210 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 -- 172.21.2.117:0/3421917181 >> 172.21.2.117:0/3421917181 conn(0x7f0e740cf060 msgr2=0x7f0e740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 -- 172.21.2.117:0/3421917181 shutdown_connections 2021-02-10T11:54:21.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.385 7f0e79381700 1 -- 172.21.2.117:0/3421917181 wait complete. 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.399 7f0e79381700 1 Processor -- start 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.399 7f0e79381700 1 -- start start 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e740a67e0 0x7f0e740a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e7407e7d0 0x7f0e7407ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7407f890 con 0x7f0e741a2e50 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7407f310 con 0x7f0e740a67e0 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e79381700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e741a3870 con 0x7f0e7407e7d0 2021-02-10T11:54:21.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57884/0 (socket says 172.21.2.117:57884) 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 -- 172.21.2.117:0/2210581567 learned_addr learned my addr 172.21.2.117:0/2210581567 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7307d700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e740a67e0 0x7f0e740a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7287c700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e7407e7d0 0x7f0e7407ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 -- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e7407e7d0 msgr2=0x7f0e7407ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e7407e7d0 0x7f0e7407ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 -- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e740a67e0 msgr2=0x7f0e740a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e740a67e0 0x7f0e740a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.400 7f0e7387e700 1 -- 172.21.2.117:0/2210581567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e741a3a90 con 0x7f0e741a2e50 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e7387e700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 rx=0x7f0e6400b580 tx=0x7f0e6400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e64006240 con 0x7f0e741a2e50 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64000c10 con 0x7f0e741a2e50 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0e640054f0 con 0x7f0e741a2e50 2021-02-10T11:54:21.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e79381700 1 -- 172.21.2.117:0/2210581567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e741a3a90 con 0x7f0e741a2e50 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e79381700 1 -- 172.21.2.117:0/2210581567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7409f8e0 con 0x7f0e741a2e50 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.401 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0e64006030 con 0x7f0e741a2e50 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.402 7f0e5bfff700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 0x7f0e5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.402 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f0e64032240 con 0x7f0e741a2e50 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.402 7f0e7307d700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 0x7f0e5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:21.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.402 7f0e7307d700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 0x7f0e5401abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:21.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.433 7f0e79381700 1 -- 172.21.2.117:0/2210581567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e740ede60 con 0x7f0e741a2e50 2021-02-10T11:54:21.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.435 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0e64000df0 con 0x7f0e741a2e50 2021-02-10T11:54:21.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.563 7f0e79381700 1 -- 172.21.2.117:0/2210581567 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f0e740ede60 con 0x7f0e741a2e50 2021-02-10T11:54:21.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.564 7f0e5bfff700 1 -- 172.21.2.117:0/2210581567 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v156) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0e64006030 con 0x7f0e741a2e50 2021-02-10T11:54:21.566 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:21.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 msgr2=0x7f0e5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 0x7f0e5401abc0 crc :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 msgr2=0x7f0e741a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 rx=0x7f0e6400b580 tx=0x7f0e6400b920).stop 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 shutdown_connections 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e540186c0 0x7f0e5401abc0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0e741a2e50 0x7f0e741a3250 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0e7407e7d0 0x7f0e7407ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 --2- 172.21.2.117:0/2210581567 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0e740a67e0 0x7f0e740a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 >> 172.21.2.117:0/2210581567 conn(0x7f0e7409e660 msgr2=0x7f0e740bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:21.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 shutdown_connections 2021-02-10T11:54:21.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.566 7f0e79381700 1 -- 172.21.2.117:0/2210581567 wait complete. 2021-02-10T11:54:21.576 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 Processor -- start 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 -- start start 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f1e7c0eb7f0 0x7f1e7c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c0ede60 con 0x7f1e7c0d18e0 2021-02-10T11:54:22.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0ef7f0 con 0x7f1e7c0ef3f0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.987 7f1e80a78700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0d1ce0 con 0x7f1e7c0eb7f0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57890/0 (socket says 172.21.2.117:57890) 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 -- 172.21.2.117:0/1237747477 learned_addr learned my addr 172.21.2.117:0/1237747477 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 -- 172.21.2.117:0/1237747477 >> v2:172.21.2.125:3300/0 conn(0x7f1e7c0eb7f0 msgr2=0x7f1e7c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 --2- 172.21.2.117:0/1237747477 >> v2:172.21.2.125:3300/0 conn(0x7f1e7c0eb7f0 0x7f1e7c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 -- 172.21.2.117:0/1237747477 >> v1:172.21.2.125:6789/0 conn(0x7f1e7c0d18e0 legacy=0x7f1e7c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 -- 172.21.2.117:0/1237747477 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c1b8f90 con 0x7f1e7c0ef3f0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7a05d700 1 --2- 172.21.2.117:0/1237747477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7f1e6c006700 tx=0x7f1e6c009b80).ready entity=mon.0 client_cookie=f244282ee847753a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7985c700 1 -- 172.21.2.117:0/1237747477 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6c00c0c0 con 0x7f1e7c0ef3f0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7985c700 1 -- 172.21.2.117:0/1237747477 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c00c340 con 0x7f1e7c0ef3f0 2021-02-10T11:54:22.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e7985c700 1 -- 172.21.2.117:0/1237747477 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6c00c860 con 0x7f1e7c0ef3f0 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e80a78700 1 -- 172.21.2.117:0/1237747477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c0ef3f0 msgr2=0x7f1e7c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.988 7f1e80a78700 1 --2- 172.21.2.117:0/1237747477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 rx=0x7f1e6c006700 tx=0x7f1e6c009b80).stop 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 -- 172.21.2.117:0/1237747477 shutdown_connections 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 --2- 172.21.2.117:0/1237747477 >> v2:172.21.2.125:3300/0 conn(0x7f1e7c0eb7f0 0x7f1e7c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 --2- 172.21.2.117:0/1237747477 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c0ef3f0 0x7f1e7c0eb1d0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 -- 172.21.2.117:0/1237747477 >> 172.21.2.117:0/1237747477 conn(0x7f1e7c0ced60 msgr2=0x7f1e7c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 -- 172.21.2.117:0/1237747477 shutdown_connections 2021-02-10T11:54:22.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:21.989 7f1e80a78700 1 -- 172.21.2.117:0/1237747477 wait complete. 2021-02-10T11:54:22.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 Processor -- start 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 -- start start 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e7c09fc50 0x7f1e7c0e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7c0e6080 0x7f1e7c0a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c1b29b0 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0e6480 con 0x7f1e7c0e6080 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e80a78700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0e66a0 con 0x7f1e7c09fc50 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e7a85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e7a85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57894/0 (socket says 172.21.2.117:57894) 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.004 7f1e7a85e700 1 -- 172.21.2.117:0/1285529374 learned_addr learned my addr 172.21.2.117:0/1285529374 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 -- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e7c09fc50 msgr2=0x7f1e7c0e5a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:22.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e7c09fc50 0x7f1e7c0e5a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 -- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7c0e6080 msgr2=0x7f1e7c0a01b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7c0e6080 0x7f1e7c0a01b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 -- 172.21.2.117:0/1285529374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c0a07d0 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e7a85e700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f1e6400b5b0 tx=0x7f1e6400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e6400bd40 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e640009b0 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1e64006210 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c0a07d0 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.005 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c0a0d60 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.006 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1e64006020 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.006 7f1e737fe700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 0x7f1e6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.006 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f1e64032230 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.006 7f1e7a05d700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 0x7f1e6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.007 7f1e7a05d700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 0x7f1e6001abc0 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.037 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c087280 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.040 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1e64023190 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.162 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1e7c0ede60 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.162 7f1e737fe700 1 -- 172.21.2.117:0/1285529374 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v156) v1 ==== 84+0+64 (secure 0 0 0) 0x7f1e64023190 con 0x7f1e7c1b2eb0 2021-02-10T11:54:22.164 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:22.165 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:54:22.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.164 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 msgr2=0x7f1e6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.164 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 0x7f1e6001abc0 crc :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.164 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 msgr2=0x7f1e7c09f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.164 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7f1e6400b5b0 tx=0x7f1e6400b950).stop 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 shutdown_connections 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1e7c0e6080 0x7f1e7c0a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1e7c09fc50 0x7f1e7c0e5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1e600186c0 0x7f1e6001abc0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 --2- 172.21.2.117:0/1285529374 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1e7c1b2eb0 0x7f1e7c09f630 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 >> 172.21.2.117:0/1285529374 conn(0x7f1e7c0874b0 msgr2=0x7f1e7c198230 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 shutdown_connections 2021-02-10T11:54:22.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.165 7f1e80a78700 1 -- 172.21.2.117:0/1285529374 wait complete. 2021-02-10T11:54:22.175 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:54:22.175 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:54:22.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.589 7f0dd6c27700 1 Processor -- start 2021-02-10T11:54:22.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.589 7f0dd6c27700 1 -- start start 2021-02-10T11:54:22.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd6c27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd01b07f0 0x7f0dd00e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd6c27700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0dd00e7180 0x7f0dd00e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd6c27700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd00e7580 con 0x7f0dd00e7d70 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd6c27700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00e92f0 con 0x7f0dd00e7180 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd6c27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00e5ce0 con 0x7f0dd01b07f0 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0dd00e7180 0x7f0dd00e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0dd00e7180 0x7f0dd00e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57906/0 (socket says 172.21.2.117:57906) 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 -- 172.21.2.117:0/258415068 learned_addr learned my addr 172.21.2.117:0/258415068 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd4a43700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd01b07f0 0x7f0dd00e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 -- 172.21.2.117:0/258415068 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd01b07f0 msgr2=0x7f0dd00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd01b07f0 0x7f0dd00e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 -- 172.21.2.117:0/258415068 >> v1:172.21.2.125:6789/0 conn(0x7f0dd00e7d70 legacy=0x7f0dd00e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 -- 172.21.2.117:0/258415068 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00e77f0 con 0x7f0dd00e7180 2021-02-10T11:54:22.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.590 7f0dd5244700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00e7180 0x7f0dd00e8960 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f0dc8006b80 tx=0x7f0dc8009b90).ready entity=mon.0 client_cookie=886676995c50eaae server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dcf7fe700 1 -- 172.21.2.117:0/258415068 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc80022e0 con 0x7f0dd00e7180 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dcf7fe700 1 -- 172.21.2.117:0/258415068 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc80009b0 con 0x7f0dd00e7180 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dcf7fe700 1 -- 172.21.2.117:0/258415068 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0dc80028e0 con 0x7f0dd00e7180 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 -- 172.21.2.117:0/258415068 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00e7180 msgr2=0x7f0dd00e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00e7180 0x7f0dd00e8960 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f0dc8006b80 tx=0x7f0dc8009b90).stop 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 -- 172.21.2.117:0/258415068 shutdown_connections 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00e7180 0x7f0dd00e8960 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 --2- 172.21.2.117:0/258415068 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd01b07f0 0x7f0dd00e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 -- 172.21.2.117:0/258415068 >> 172.21.2.117:0/258415068 conn(0x7f0dd00ce500 msgr2=0x7f0dd00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 -- 172.21.2.117:0/258415068 shutdown_connections 2021-02-10T11:54:22.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.591 7f0dd6c27700 1 -- 172.21.2.117:0/258415068 wait complete. 2021-02-10T11:54:22.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 Processor -- start 2021-02-10T11:54:22.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 -- start start 2021-02-10T11:54:22.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd019df40 0x7f0dd00bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0dd00bbfd0 0x7f0dd00bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00bb430 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00bc9f0 con 0x7f0dd00bbfd0 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.606 7f0dd6c27700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00bcc10 con 0x7f0dd019df40 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57912/0 (socket says 172.21.2.117:57912) 2021-02-10T11:54:22.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 -- 172.21.2.117:0/2587676527 learned_addr learned my addr 172.21.2.117:0/2587676527 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 -- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd019df40 msgr2=0x7f0dd00bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd019df40 0x7f0dd00bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 -- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0dd00bbfd0 msgr2=0x7f0dd00bc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0dd00bbfd0 0x7f0dd00bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 -- 172.21.2.117:0/2587676527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00b6f70 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd4a43700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7f0db8006f60 tx=0x7f0db800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db8006bb0 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db8006340 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd00b6f70 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd00b7500 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.607 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0db8002030 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.608 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0db8006b50 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.608 7f0dcd7fa700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 0x7f0db401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.608 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f0db8032340 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.608 7f0dcffff700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 0x7f0db401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:22.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.609 7f0dcffff700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 0x7f0db401abc0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:22.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.639 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd007e800 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.642 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0db8000960 con 0x7f0dd00b8ac0 2021-02-10T11:54:22.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:22.769 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f0dd00e77f0 con 0x7f0dd00b8ac0 2021-02-10T11:54:23.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.192 7f0dcd7fa700 1 -- 172.21.2.117:0/2587676527 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v157) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0db8006b50 con 0x7f0dd00b8ac0 2021-02-10T11:54:23.237 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:54:23.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 msgr2=0x7f0db401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 0x7f0db401abc0 crc :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 msgr2=0x7f0dd019d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 rx=0x7f0db8006f60 tx=0x7f0db800b970).stop 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 shutdown_connections 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0dd00bbfd0 0x7f0dd00bc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:23.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0dd019df40 0x7f0dd00bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0db40186c0 0x7f0db401abc0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 --2- 172.21.2.117:0/2587676527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0dd00b8ac0 0x7f0dd019d920 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 >> 172.21.2.117:0/2587676527 conn(0x7f0dd0110940 msgr2=0x7f0dd007fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 shutdown_connections 2021-02-10T11:54:23.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.237 7f0dd6c27700 1 -- 172.21.2.117:0/2587676527 wait complete. 2021-02-10T11:54:23.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 Processor -- start 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 -- start start 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa82c1b69a0 0x7fa82c1b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0ede60 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0eba00 con 0x7fa82c0d18f0 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.662 7fa831f7c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c0d1cf0 con 0x7fa82c1b69a0 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82affd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa82c0ef3f0 0x7fa82c0eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58166/0 (socket says 172.21.2.117:58166) 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82affd700 1 -- 172.21.2.117:0/3403964699 learned_addr learned my addr 172.21.2.117:0/3403964699 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82bfff700 1 --2- 172.21.2.117:0/3403964699 >> v2:172.21.2.125:3300/0 conn(0x7fa82c1b69a0 0x7fa82c1b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246372532 0 0) 0x7fa814001180 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa818002fe0 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 325748999 0 0) 0x7fa814001180 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 >> v2:172.21.2.125:3300/0 conn(0x7fa82c1b69a0 msgr2=0x7fa82c1b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 --2- 172.21.2.117:0/3403964699 >> v2:172.21.2.125:3300/0 conn(0x7fa82c1b69a0 0x7fa82c1b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 >> v1:172.21.2.125:6789/0 conn(0x7fa82c0d18f0 legacy=0x7fa82c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c1bb3f0 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa814001680 con 0x7fa82c0ef3f0 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 >> v1:172.21.2.117:6789/0 conn(0x7fa82c0ef3f0 legacy=0x7fa82c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa818003ac0 0x7fa818005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 0x7fa818008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180095a0 0x7fa81800baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800c2f0 con 0x7fa818006780 2021-02-10T11:54:23.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800c510 con 0x7fa818003ac0 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81800c730 con 0x7fa8180095a0 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.663 7fa82affd700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 0x7fa818008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82b7fe700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa818003ac0 0x7fa818005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 -- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180095a0 msgr2=0x7fa81800baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8180095a0 0x7fa81800baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 -- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa818003ac0 msgr2=0x7fa818005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa818003ac0 0x7fa818005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 -- 172.21.2.117:0/3403964699 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180509a0 con 0x7fa818006780 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82bfff700 1 -- 172.21.2.117:0/3403964699 reap_dead start 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82affd700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 0x7fa818008ca0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7fa814000b60 tx=0x7fa814007da0).ready entity=mon.0 client_cookie=a59a9db97d6a3b2e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa814007810 con 0x7fa818006780 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8140070c0 con 0x7fa818006780 2021-02-10T11:54:23.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa82a7fc700 1 -- 172.21.2.117:0/3403964699 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa814004590 con 0x7fa818006780 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 -- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 msgr2=0x7fa818008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 0x7fa818008ca0 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 rx=0x7fa814000b60 tx=0x7fa814007da0).stop 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 -- 172.21.2.117:0/3403964699 shutdown_connections 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 --2- 172.21.2.117:0/3403964699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa818006780 0x7fa818008ca0 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 -- 172.21.2.117:0/3403964699 >> 172.21.2.117:0/3403964699 conn(0x7fa82c0cf060 msgr2=0x7fa82c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 -- 172.21.2.117:0/3403964699 shutdown_connections 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.664 7fa831f7c700 1 -- 172.21.2.117:0/3403964699 wait complete. 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.679 7fa831f7c700 1 Processor -- start 2021-02-10T11:54:23.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.679 7fa831f7c700 1 -- start start 2021-02-10T11:54:23.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.679 7fa831f7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa82c1a48b0 0x7fa82c08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa82c08cf40 0x7fa82c0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0aae30 con 0x7fa82c0a9f90 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0aa9f0 con 0x7fa82c1a48b0 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0ab330 con 0x7fa82c08cf40 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57928/0 (socket says 172.21.2.117:57928) 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 -- 172.21.2.117:0/1850268338 learned_addr learned my addr 172.21.2.117:0/1850268338 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 -- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa82c08cf40 msgr2=0x7fa82c0a9970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa82c08cf40 0x7fa82c0a9970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 -- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa82c1a48b0 msgr2=0x7fa82c08c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa82c1a48b0 0x7fa82c08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 -- 172.21.2.117:0/1850268338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0ab550 con 0x7fa82c0a9f90 2021-02-10T11:54:23.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa82bfff700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fa8240099a0 tx=0x7fa82400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8240011f0 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824003030 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.680 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0ab550 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.681 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0abae0 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.681 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa824001760 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.681 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa8240011f0 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.681 7fa823fff700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.681 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5454+0+0 (secure 0 0 0) 0x7fa824032880 con 0x7fa82c0a9f90 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.682 7fa82b7fe700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:23.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.682 7fa82b7fe700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:23.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.712 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0ede60 con 0x7fa82c0a9f90 2021-02-10T11:54:23.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.715 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa8240066f0 con 0x7fa82c0a9f90 2021-02-10T11:54:23.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:23.843 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7fa82c0aa9b0 con 0x7fa82c0a9f90 2021-02-10T11:54:24.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.200 7fa823fff700 1 -- 172.21.2.117:0/1850268338 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v158) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa82401f8b0 con 0x7fa82c0a9f90 2021-02-10T11:54:24.209 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:54:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 msgr2=0x7fa81001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 crc :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 msgr2=0x7fa82c0aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 rx=0x7fa8240099a0 tx=0x7fa82400bcf0).stop 2021-02-10T11:54:24.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 shutdown_connections 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa82c0a9f90 0x7fa82c0aa390 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa82c08cf40 0x7fa82c0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 --2- 172.21.2.117:0/1850268338 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa82c1a48b0 0x7fa82c08c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 >> 172.21.2.117:0/1850268338 conn(0x7fa82c092d00 msgr2=0x7fa82c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 shutdown_connections 2021-02-10T11:54:24.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.209 7fa831f7c700 1 -- 172.21.2.117:0/1850268338 wait complete. 2021-02-10T11:54:24.219 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:54:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.634 7fcccb254700 1 Processor -- start 2021-02-10T11:54:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.634 7fcccb254700 1 -- start start 2021-02-10T11:54:24.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fcccb254700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccc40e5240 0x7fccc4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fcccb254700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4084c60 0x7fccc4085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fcccb254700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc4085930 con 0x7fccc40e4e40 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fcccb254700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc4085680 con 0x7fccc40e5240 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fcccb254700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41b09e0 con 0x7fccc4084c60 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fccc40e5240 0x7fccc4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc9070700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fccc40e4e40 0x7fccc41b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58188/0 (socket says 172.21.2.117:58188) 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc9070700 1 -- 172.21.2.117:0/3976256462 learned_addr learned my addr 172.21.2.117:0/3976256462 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 -- 172.21.2.117:0/3976256462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4084c60 msgr2=0x7fccc4085060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 --2- 172.21.2.117:0/3976256462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4084c60 0x7fccc4085060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 -- 172.21.2.117:0/3976256462 >> v1:172.21.2.117:6789/0 conn(0x7fccc40e4e40 legacy=0x7fccc41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 -- 172.21.2.117:0/3976256462 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4084840 con 0x7fccc40e5240 2021-02-10T11:54:24.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.635 7fccc886f700 1 --2- 172.21.2.117:0/3976256462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc40e5240 0x7fccc4086ec0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7fccc0006650 tx=0x7fccc0009a80).ready entity=mon.0 client_cookie=4b9baaa58353587e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fccb7fff700 1 -- 172.21.2.117:0/3976256462 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc00009b0 con 0x7fccc40e5240 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fccb7fff700 1 -- 172.21.2.117:0/3976256462 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc000c0c0 con 0x7fccc40e5240 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fccb7fff700 1 -- 172.21.2.117:0/3976256462 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc000c5e0 con 0x7fccc40e5240 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 -- 172.21.2.117:0/3976256462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc40e5240 msgr2=0x7fccc4086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 --2- 172.21.2.117:0/3976256462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc40e5240 0x7fccc4086ec0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7fccc0006650 tx=0x7fccc0009a80).stop 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 -- 172.21.2.117:0/3976256462 shutdown_connections 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 --2- 172.21.2.117:0/3976256462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4084c60 0x7fccc4085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 --2- 172.21.2.117:0/3976256462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc40e5240 0x7fccc4086ec0 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.636 7fcccb254700 1 -- 172.21.2.117:0/3976256462 >> 172.21.2.117:0/3976256462 conn(0x7fccc40ce4e0 msgr2=0x7fccc40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.637 7fcccb254700 1 -- 172.21.2.117:0/3976256462 shutdown_connections 2021-02-10T11:54:24.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.637 7fcccb254700 1 -- 172.21.2.117:0/3976256462 wait complete. 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 Processor -- start 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 -- start start 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4099bf0 0x7fccc41ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc40b30d0 0x7fccc40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41ae8d0 con 0x7fccc418e5b0 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b3af0 con 0x7fccc40b30d0 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.651 7fcccb254700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b3d10 con 0x7fccc4099bf0 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57946/0 (socket says 172.21.2.117:57946) 2021-02-10T11:54:24.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 -- 172.21.2.117:0/2265059063 learned_addr learned my addr 172.21.2.117:0/2265059063 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 -- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4099bf0 msgr2=0x7fccc41ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4099bf0 0x7fccc41ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 -- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc40b30d0 msgr2=0x7fccc40b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc40b30d0 0x7fccc40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 -- 172.21.2.117:0/2265059063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40b3f30 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccc886f700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7fccc0009630 tx=0x7fccc000b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc00093a0 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc000bcf0 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fcccb254700 1 -- 172.21.2.117:0/2265059063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc40b3f30 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fcccb254700 1 -- 172.21.2.117:0/2265059063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc41a19b0 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.652 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fccc0005560 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.653 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fccc000a000 con 0x7fccc418e5b0 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.653 7fccb5ffb700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 0x7fcca401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:24.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.653 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fccc0032660 con 0x7fccc418e5b0 2021-02-10T11:54:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.654 7fccc9070700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 0x7fcca401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:24.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.654 7fccc9070700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 0x7fcca401abc0 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:24.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.685 7fcccb254700 1 -- 172.21.2.117:0/2265059063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc418e080 con 0x7fccc418e5b0 2021-02-10T11:54:24.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.687 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fccc0009340 con 0x7fccc418e5b0 2021-02-10T11:54:24.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.816 7fcccb254700 1 -- 172.21.2.117:0/2265059063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fccc418e080 con 0x7fccc418e5b0 2021-02-10T11:54:24.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.820 7fccb5ffb700 1 -- 172.21.2.117:0/2265059063 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 34 and data pool 35 v133) v1 ==== 189+0+0 (secure 0 0 0) 0x7fccc000a300 con 0x7fccc418e5b0 2021-02-10T11:54:24.825 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 34 and data pool 35 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.825 7fcccb254700 1 -- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 msgr2=0x7fcca401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.825 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 0x7fcca401abc0 crc :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.825 7fcccb254700 1 -- 172.21.2.117:0/2265059063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 msgr2=0x7fccc418e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.825 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 rx=0x7fccc0009630 tx=0x7fccc000b900).stop 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 -- 172.21.2.117:0/2265059063 shutdown_connections 2021-02-10T11:54:24.827 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcca40186c0 0x7fcca401abc0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fccc418e5b0 0x7fccc418e9b0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:24.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fccc40b30d0 0x7fccc40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 --2- 172.21.2.117:0/2265059063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fccc4099bf0 0x7fccc41ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:24.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 -- 172.21.2.117:0/2265059063 >> 172.21.2.117:0/2265059063 conn(0x7fccc41ad6b0 msgr2=0x7fccc4190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:24.828 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 -- 172.21.2.117:0/2265059063 shutdown_connections 2021-02-10T11:54:24.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:24.826 7fcccb254700 1 -- 172.21.2.117:0/2265059063 wait complete. 2021-02-10T11:54:24.836 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:25.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.245 7fd4b43ac700 1 Processor -- start 2021-02-10T11:54:25.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.245 7fd4b43ac700 1 -- start start 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b43ac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b43ac700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd4ac0eb7f0 0x7fd4ac1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b43ac700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ac0ede60 con 0x7fd4ac0d18e0 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b43ac700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0ef7f0 con 0x7fd4ac0ef3f0 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b43ac700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ac0d1ce0 con 0x7fd4ac0eb7f0 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57952/0 (socket says 172.21.2.117:57952) 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 -- 172.21.2.117:0/1549112785 learned_addr learned my addr 172.21.2.117:0/1549112785 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 -- 172.21.2.117:0/1549112785 >> v2:172.21.2.125:3300/0 conn(0x7fd4ac0eb7f0 msgr2=0x7fd4ac1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 --2- 172.21.2.117:0/1549112785 >> v2:172.21.2.125:3300/0 conn(0x7fd4ac0eb7f0 0x7fd4ac1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 -- 172.21.2.117:0/1549112785 >> v1:172.21.2.125:6789/0 conn(0x7fd4ac0d18e0 legacy=0x7fd4ac0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 -- 172.21.2.117:0/1549112785 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac1b8f90 con 0x7fd4ac0ef3f0 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.246 7fd4b19c7700 1 --2- 172.21.2.117:0/1549112785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7fd4a8006700 tx=0x7fd4a8009b80).ready entity=mon.0 client_cookie=86d9f9e885e899d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b11c6700 1 -- 172.21.2.117:0/1549112785 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a800c0c0 con 0x7fd4ac0ef3f0 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b11c6700 1 -- 172.21.2.117:0/1549112785 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a800c400 con 0x7fd4ac0ef3f0 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b11c6700 1 -- 172.21.2.117:0/1549112785 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a800c920 con 0x7fd4ac0ef3f0 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 -- 172.21.2.117:0/1549112785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac0ef3f0 msgr2=0x7fd4ac0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 --2- 172.21.2.117:0/1549112785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 rx=0x7fd4a8006700 tx=0x7fd4a8009b80).stop 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 -- 172.21.2.117:0/1549112785 shutdown_connections 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 --2- 172.21.2.117:0/1549112785 >> v2:172.21.2.125:3300/0 conn(0x7fd4ac0eb7f0 0x7fd4ac1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 --2- 172.21.2.117:0/1549112785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac0ef3f0 0x7fd4ac0eb1d0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.247 7fd4b43ac700 1 -- 172.21.2.117:0/1549112785 >> 172.21.2.117:0/1549112785 conn(0x7fd4ac0ced60 msgr2=0x7fd4ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.248 7fd4b43ac700 1 -- 172.21.2.117:0/1549112785 shutdown_connections 2021-02-10T11:54:25.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.248 7fd4b43ac700 1 -- 172.21.2.117:0/1549112785 wait complete. 2021-02-10T11:54:25.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.262 7fd4b43ac700 1 Processor -- start 2021-02-10T11:54:25.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.262 7fd4b43ac700 1 -- start start 2021-02-10T11:54:25.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4ac0b51d0 0x7fd4ac19b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4ac091740 0x7fd4ac091b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac1b1b50 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac19b770 con 0x7fd4ac091740 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b43ac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac19b950 con 0x7fd4ac0b51d0 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57960/0 (socket says 172.21.2.117:57960) 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 -- 172.21.2.117:0/3832515581 learned_addr learned my addr 172.21.2.117:0/3832515581 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 -- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4ac0b51d0 msgr2=0x7fd4ac19b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4ac0b51d0 0x7fd4ac19b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 -- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4ac091740 msgr2=0x7fd4ac091b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4ac091740 0x7fd4ac091b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 -- 172.21.2.117:0/3832515581 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac092190 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd4b19c7700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fd4a80066a0 tx=0x7fd4a800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.263 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a800bd80 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4a80052e0 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd4a80057c0 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ac092190 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ac092190 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd4a8006030 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd49effd700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.264 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd4a8032700 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.265 7fd4b21c8700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.269 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.266 7fd4b21c8700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 0x7fd49401abc0 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.295 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ac19b990 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.298 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd4a8030d60 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.426 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4ac19b990 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.426 7fd49effd700 1 -- 172.21.2.117:0/3832515581 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+14253 (secure 0 0 0) 0x7fd4a8006030 con 0x7fd4ac1b20d0 2021-02-10T11:54:25.428 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:25.429 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":159,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:24.817089","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:54:22.787648","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:54:23.861446","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 msgr2=0x7fd49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 0x7fd49401abc0 crc :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 msgr2=0x7fd4ac1b24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 rx=0x7fd4a80066a0 tx=0x7fd4a800b990).stop 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 shutdown_connections 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd4940186c0 0x7fd49401abc0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd4ac1b20d0 0x7fd4ac1b24d0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd4ac091740 0x7fd4ac091b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 --2- 172.21.2.117:0/3832515581 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd4ac0b51d0 0x7fd4ac19b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 >> 172.21.2.117:0/3832515581 conn(0x7fd4ac19f610 msgr2=0x7fd4ac1ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.428 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 shutdown_connections 2021-02-10T11:54:25.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.429 7fd4b43ac700 1 -- 172.21.2.117:0/3832515581 wait complete. 2021-02-10T11:54:25.439 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.852 7f16bbe21700 1 Processor -- start 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.852 7f16bbe21700 1 -- start start 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16bbe21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 0x7f16b41b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16bbe21700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40ede60 con 0x7f16b40d18e0 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16bbe21700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40eb9c0 con 0x7f16b40ef3f0 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16bbe21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40d1ce0 con 0x7f16b41b6960 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b9c3d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f16b40d18e0 0x7f16b40ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58216/0 (socket says 172.21.2.117:58216) 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b9c3d700 1 -- 172.21.2.117:0/2453692965 learned_addr learned my addr 172.21.2.117:0/2453692965 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:25.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16ba43e700 1 --2- 172.21.2.117:0/2453692965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 0x7f16b41b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711233414 0 0) 0x7f16a80010b0 con 0x7f16b40d18e0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0002fe0 con 0x7f16b40d18e0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110805785 0 0) 0x7f16b0000c60 con 0x7f16b40ef3f0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.853 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0004370 con 0x7f16b40ef3f0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16ba43e700 1 -- 172.21.2.117:0/2453692965 >> v1:172.21.2.125:6789/0 conn(0x7f16b40ef3f0 legacy=0x7f16b40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16ba43e700 1 -- 172.21.2.117:0/2453692965 >> v1:172.21.2.117:6789/0 conn(0x7f16b40d18e0 legacy=0x7f16b40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16ba43e700 1 -- 172.21.2.117:0/2453692965 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b41bb3b0 con 0x7f16b41b6960 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16ba43e700 1 --2- 172.21.2.117:0/2453692965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 0x7f16b41b8d70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f16ac006e50 tx=0x7f16ac009e70).ready entity=mon.2 client_cookie=35a742be998180e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16ac00b510 con 0x7f16b41b6960 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac006790 con 0x7f16b41b6960 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16b8c3b700 1 -- 172.21.2.117:0/2453692965 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16ac00b940 con 0x7f16b41b6960 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16bbe21700 1 -- 172.21.2.117:0/2453692965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 msgr2=0x7f16b41b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.854 7f16bbe21700 1 --2- 172.21.2.117:0/2453692965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 0x7f16b41b8d70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f16ac006e50 tx=0x7f16ac009e70).stop 2021-02-10T11:54:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.855 7f16bbe21700 1 -- 172.21.2.117:0/2453692965 shutdown_connections 2021-02-10T11:54:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.855 7f16bbe21700 1 --2- 172.21.2.117:0/2453692965 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b41b6960 0x7f16b41b8d70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.855 7f16bbe21700 1 -- 172.21.2.117:0/2453692965 >> 172.21.2.117:0/2453692965 conn(0x7f16b40ced60 msgr2=0x7f16b40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.855 7f16bbe21700 1 -- 172.21.2.117:0/2453692965 shutdown_connections 2021-02-10T11:54:25.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.855 7f16bbe21700 1 -- 172.21.2.117:0/2453692965 wait complete. 2021-02-10T11:54:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 Processor -- start 2021-02-10T11:54:25.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 -- start start 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16b40e3ad0 0x7f16b40b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b40e3ed0 0x7f16b40e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40e44e0 con 0x7f16b40b2e30 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40e56a0 con 0x7f16b40e3ad0 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16bbe21700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b40e5880 con 0x7f16b40e3ed0 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16b943c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16b943c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57974/0 (socket says 172.21.2.117:57974) 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16b943c700 1 -- 172.21.2.117:0/257043960 learned_addr learned my addr 172.21.2.117:0/257043960 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16b9c3d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16b40e3ad0 0x7f16b40b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.870 7f16ba43e700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b40e3ed0 0x7f16b40e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 -- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b40e3ed0 msgr2=0x7f16b40e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b40e3ed0 0x7f16b40e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 -- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16b40e3ad0 msgr2=0x7f16b40b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16b40e3ad0 0x7f16b40b2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 -- 172.21.2.117:0/257043960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40e5aa0 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16b943c700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f16b0006f10 tx=0x7f16b000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b000bd80 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b0000e20 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f16b00054f0 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16bbe21700 1 -- 172.21.2.117:0/257043960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b40e5aa0 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.871 7f16bbe21700 1 -- 172.21.2.117:0/257043960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b40e5aa0 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.872 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f16b0009fe0 con 0x7f16b40b2e30 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.872 7f16a67fc700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 0x7f169c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:25.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.872 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f16b0032710 con 0x7f16b40b2e30 2021-02-10T11:54:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.872 7f16b9c3d700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 0x7f169c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:25.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.872 7f16b9c3d700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 0x7f169c01abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:25.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.902 7f16bbe21700 1 -- 172.21.2.117:0/257043960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b4087290 con 0x7f16b40b2e30 2021-02-10T11:54:25.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:25.905 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f16b0030d70 con 0x7f16b40b2e30 2021-02-10T11:54:26.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.029 7f16bbe21700 1 -- 172.21.2.117:0/257043960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16b40ede60 con 0x7f16b40b2e30 2021-02-10T11:54:26.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.030 7f16a67fc700 1 -- 172.21.2.117:0/257043960 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f16b0009fe0 con 0x7f16b40b2e30 2021-02-10T11:54:26.032 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:26.032 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":160,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:25.819611","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:54:22.787648","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:54:23.861446","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:26.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.031 7f16bbe21700 1 -- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 msgr2=0x7f169c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.031 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 0x7f169c01abc0 crc :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.031 7f16bbe21700 1 -- 172.21.2.117:0/257043960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 msgr2=0x7f16b40e4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.031 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7f16b0006f10 tx=0x7f16b000b990).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 -- 172.21.2.117:0/257043960 shutdown_connections 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f169c0186c0 0x7f169c01abc0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f16b40b2e30 0x7f16b40e4a60 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f16b40e3ed0 0x7f16b40e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 --2- 172.21.2.117:0/257043960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f16b40e3ad0 0x7f16b40b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 -- 172.21.2.117:0/257043960 >> 172.21.2.117:0/257043960 conn(0x7f16b41a3620 msgr2=0x7f16b408d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 -- 172.21.2.117:0/257043960 shutdown_connections 2021-02-10T11:54:26.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.032 7f16bbe21700 1 -- 172.21.2.117:0/257043960 wait complete. 2021-02-10T11:54:26.042 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:54:26.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.456 7fa24421d700 1 Processor -- start 2021-02-10T11:54:26.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.456 7fa24421d700 1 -- start start 2021-02-10T11:54:26.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa24421d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0d18f0 0x7fa23c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa24421d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa23c0ef3f0 0x7fa23c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa24421d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23c0ede60 con 0x7fa23c0eb040 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa24421d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23c088020 con 0x7fa23c0ef3f0 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa24421d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0d1d30 con 0x7fa23c0d18f0 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa242039700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0d18f0 0x7fa23c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.457 7fa241838700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fa23c0eb040 0x7fa23c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51156/0 (socket says 172.21.2.117:51156) 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa241838700 1 -- 172.21.2.117:0/1490612995 learned_addr learned my addr 172.21.2.117:0/1490612995 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 --2- 172.21.2.117:0/1490612995 >> v2:172.21.2.125:3300/0 conn(0x7fa23c0ef3f0 0x7fa23c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 -- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0d18f0 msgr2=0x7fa23c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 --2- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0d18f0 0x7fa23c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 -- 172.21.2.117:0/1490612995 >> v1:172.21.2.125:6789/0 conn(0x7fa23c0eb040 legacy=0x7fa23c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa241037700 1 -- 172.21.2.117:0/1490612995 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595911710 0 0) 0x7fa2380010b0 con 0x7fa23c0eb040 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 -- 172.21.2.117:0/1490612995 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c1b8fd0 con 0x7fa23c0ef3f0 2021-02-10T11:54:26.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa24283a700 1 --2- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0ef3f0 0x7fa23c1b79a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fa234006e50 tx=0x7fa234009df0).ready entity=mon.1 client_cookie=39656544fa22531e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa241037700 1 -- 172.21.2.117:0/1490612995 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23400b510 con 0x7fa23c0ef3f0 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa241037700 1 -- 172.21.2.117:0/1490612995 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa234006790 con 0x7fa23c0ef3f0 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.458 7fa241037700 1 -- 172.21.2.117:0/1490612995 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa23400b750 con 0x7fa23c0ef3f0 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 -- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0ef3f0 msgr2=0x7fa23c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 --2- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0ef3f0 0x7fa23c1b79a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fa234006e50 tx=0x7fa234009df0).stop 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 -- 172.21.2.117:0/1490612995 shutdown_connections 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 --2- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0ef3f0 0x7fa23c1b79a0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 --2- 172.21.2.117:0/1490612995 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0d18f0 0x7fa23c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 -- 172.21.2.117:0/1490612995 >> 172.21.2.117:0/1490612995 conn(0x7fa23c0cf060 msgr2=0x7fa23c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 -- 172.21.2.117:0/1490612995 shutdown_connections 2021-02-10T11:54:26.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.459 7fa24421d700 1 -- 172.21.2.117:0/1490612995 wait complete. 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 Processor -- start 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 -- start start 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0a6ce0 0x7fa23c0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0a2040 0x7fa23c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0a67e0 con 0x7fa23c085ec0 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0a7120 con 0x7fa23c0a2040 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa24421d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c0a2a60 con 0x7fa23c0a6ce0 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa241838700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa241838700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:57988/0 (socket says 172.21.2.117:57988) 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.474 7fa241838700 1 -- 172.21.2.117:0/1667744276 learned_addr learned my addr 172.21.2.117:0/1667744276 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 -- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0a6ce0 msgr2=0x7fa23c0dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0a6ce0 0x7fa23c0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 -- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0a2040 msgr2=0x7fa23c0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0a2040 0x7fa23c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 -- 172.21.2.117:0/1667744276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c1a2dc0 con 0x7fa23c085ec0 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa241838700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7fa238001c10 tx=0x7fa23800a8f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:26.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2380009b0 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2380012c0 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa238004350 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa24421d700 1 -- 172.21.2.117:0/1667744276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23c1a2dc0 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.475 7fa24421d700 1 -- 172.21.2.117:0/1667744276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23c1a3350 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.476 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa238004140 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.476 7fa22effd700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 0x7fa22401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.476 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2380326d0 con 0x7fa23c085ec0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.476 7fa242039700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 0x7fa22401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:26.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.477 7fa242039700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 0x7fa22401abc0 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:26.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.506 7fa24421d700 1 -- 172.21.2.117:0/1667744276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23c0daff0 con 0x7fa23c085ec0 2021-02-10T11:54:26.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.509 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa2380009b0 con 0x7fa23c085ec0 2021-02-10T11:54:26.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.644 7fa24421d700 1 -- 172.21.2.117:0/1667744276 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fa23c0ede60 con 0x7fa23c085ec0 2021-02-10T11:54:26.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.828 7fa22effd700 1 -- 172.21.2.117:0/1667744276 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v136) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa238004140 con 0x7fa23c085ec0 2021-02-10T11:54:26.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.861 7fa24421d700 1 -- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 msgr2=0x7fa22401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.861 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 0x7fa22401abc0 crc :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.861 7fa24421d700 1 -- 172.21.2.117:0/1667744276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 msgr2=0x7fa23c0862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.861 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7fa238001c10 tx=0x7fa23800a8f0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 -- 172.21.2.117:0/1667744276 shutdown_connections 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2240186c0 0x7fa22401abc0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa23c085ec0 0x7fa23c0862c0 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa23c0a2040 0x7fa23c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 --2- 172.21.2.117:0/1667744276 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa23c0a6ce0 0x7fa23c0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 -- 172.21.2.117:0/1667744276 >> 172.21.2.117:0/1667744276 conn(0x7fa23c0b6540 msgr2=0x7fa23c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 -- 172.21.2.117:0/1667744276 shutdown_connections 2021-02-10T11:54:26.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:26.862 7fa24421d700 1 -- 172.21.2.117:0/1667744276 wait complete. 2021-02-10T11:54:26.871 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.280 7f4e029ab700 1 Processor -- start 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.280 7f4e029ab700 1 -- start start 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e029ab700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4dfc0d18e0 0x7f4dfc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e029ab700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc0ede60 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e029ab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc0d1ce0 con 0x7f4dfc0eb000 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e029ab700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc0ef7f0 con 0x7f4dfc0d18e0 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e00fc8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4dfc0ef3f0 0x7f4dfc1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58242/0 (socket says 172.21.2.117:58242) 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.281 7f4e00fc8700 1 -- 172.21.2.117:0/1303766926 learned_addr learned my addr 172.21.2.117:0/1303766926 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:27.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598249584 0 0) 0x7f4df4001500 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4002fe0 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfbfff700 1 --2- 172.21.2.117:0/1303766926 >> v2:172.21.2.125:3300/0 conn(0x7f4dfc0d18e0 0x7f4dfc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 555481973 0 0) 0x7f4df4001500 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 >> v2:172.21.2.125:3300/0 conn(0x7f4dfc0d18e0 msgr2=0x7f4dfc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 --2- 172.21.2.117:0/1303766926 >> v2:172.21.2.125:3300/0 conn(0x7f4dfc0d18e0 0x7f4dfc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 >> v1:172.21.2.125:6789/0 conn(0x7f4dfc0eb000 legacy=0x7f4dfc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc1bb180 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4df4001a00 con 0x7f4dfc0ef3f0 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 >> v1:172.21.2.117:6789/0 conn(0x7f4dfc0ef3f0 legacy=0x7f4dfc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4de4003ac0 0x7f4de4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4de4006770 0x7f4de4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 0x7f4de400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de400c300 con 0x7f4de4009590 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de400c520 con 0x7f4de4003ac0 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4050750 con 0x7f4de4006770 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4e00fc8700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 0x7f4de400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.282 7f4dfbfff700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4de4003ac0 0x7f4de4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfb7fe700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4de4006770 0x7f4de4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 -- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4de4006770 msgr2=0x7f4de4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4de4006770 0x7f4de4008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 -- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4de4003ac0 msgr2=0x7f4de4005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4de4003ac0 0x7f4de4005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 -- 172.21.2.117:0/1303766926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40509b0 con 0x7f4de4009590 2021-02-10T11:54:27.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfbfff700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4de4003ac0 0x7f4de4005ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfb7fe700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4de4006770 0x7f4de4008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 -- 172.21.2.117:0/1303766926 reap_dead start 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e00fc8700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 0x7f4de400bab0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7f4df4000f00 tx=0x7f4df400a310).ready entity=mon.0 client_cookie=c0a2eec38f4a6519 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df40070c0 con 0x7f4de4009590 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df40073c0 con 0x7f4de4009590 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4dfaffd700 1 -- 172.21.2.117:0/1303766926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df40049a0 con 0x7f4de4009590 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e029ab700 1 -- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 msgr2=0x7f4de400bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.283 7f4e029ab700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 0x7f4de400bab0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 rx=0x7f4df4000f00 tx=0x7f4df400a310).stop 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.284 7f4e029ab700 1 -- 172.21.2.117:0/1303766926 shutdown_connections 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.284 7f4e029ab700 1 --2- 172.21.2.117:0/1303766926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4de4009590 0x7f4de400bab0 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.284 7f4e029ab700 1 -- 172.21.2.117:0/1303766926 >> 172.21.2.117:0/1303766926 conn(0x7f4dfc0ced60 msgr2=0x7f4dfc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:27.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.284 7f4e029ab700 1 -- 172.21.2.117:0/1303766926 shutdown_connections 2021-02-10T11:54:27.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.284 7f4e029ab700 1 -- 172.21.2.117:0/1303766926 wait complete. 2021-02-10T11:54:27.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.298 7f4e029ab700 1 Processor -- start 2021-02-10T11:54:27.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 -- start start 2021-02-10T11:54:27.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 0x7f4dfc08c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 0x7f4dfc1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0b61a0 con 0x7f4dfc1a3480 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc08c8b0 con 0x7f4dfc0b59a0 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e029ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0ba1c0 con 0x7f4dfc0b5da0 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e00fc8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e00fc8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58006/0 (socket says 172.21.2.117:58006) 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.299 7f4e00fc8700 1 -- 172.21.2.117:0/2009260310 learned_addr learned my addr 172.21.2.117:0/2009260310 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4dfb7fe700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 0x7f4dfc1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4dfbfff700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 0x7f4dfc08c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 -- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 msgr2=0x7f4dfc1a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 0x7f4dfc1a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 -- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 msgr2=0x7f4dfc08c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 0x7f4dfc08c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 -- 172.21.2.117:0/2009260310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc1a3880 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4dfbfff700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 0x7f4dfc08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4dfb7fe700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 0x7f4dfc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e00fc8700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f4df400a8c0 tx=0x7f4df400ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df4004930 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4001860 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc1a3880 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc0b2bb0 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.300 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4df4007020 con 0x7f4dfc1a3480 2021-02-10T11:54:27.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.301 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4df4004720 con 0x7f4dfc1a3480 2021-02-10T11:54:27.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.301 7f4df8ff9700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 0x7f4ddc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.301 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4df40329f0 con 0x7f4dfc1a3480 2021-02-10T11:54:27.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.302 7f4dfbfff700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 0x7f4ddc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.302 7f4dfbfff700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 0x7f4ddc01abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.331 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc0ede60 con 0x7f4dfc1a3480 2021-02-10T11:54:27.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.333 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4df4004da0 con 0x7f4dfc1a3480 2021-02-10T11:54:27.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.460 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dfc0ede60 con 0x7f4dfc1a3480 2021-02-10T11:54:27.462 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.461 7f4df8ff9700 1 -- 172.21.2.117:0/2009260310 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+2099 (secure 0 0 0) 0x7f4df4004720 con 0x7f4dfc1a3480 2021-02-10T11:54:27.463 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:27.463 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:54:26.825269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7697":{"gid":7697,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/95294604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":95294604},{"type":"v1","addr":"172.21.2.125:6835","nonce":95294604}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:54:27.463 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 136 2021-02-10T11:54:27.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.462 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 msgr2=0x7f4ddc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.462 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 0x7f4ddc01abc0 crc :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.462 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 msgr2=0x7f4dfc0b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.462 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f4df400a8c0 tx=0x7f4df400ac60).stop 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 shutdown_connections 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ddc0186c0 0x7f4ddc01abc0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4dfc1a3480 0x7f4dfc0b2590 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4dfc0b5da0 0x7f4dfc1a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 --2- 172.21.2.117:0/2009260310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4dfc0b59a0 0x7f4dfc08c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 >> 172.21.2.117:0/2009260310 conn(0x7f4dfc0bb5e0 msgr2=0x7f4dfc090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 shutdown_connections 2021-02-10T11:54:27.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.463 7f4e029ab700 1 -- 172.21.2.117:0/2009260310 wait complete. 2021-02-10T11:54:27.473 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:27.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 Processor -- start 2021-02-10T11:54:27.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 -- start start 2021-02-10T11:54:27.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84c40e4e40 0x7f84c41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40d1db0 con 0x7f84c40e5240 2021-02-10T11:54:27.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c4085930 con 0x7f84c4085450 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.885 7f84cb085700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c4084840 con 0x7f84c40e4e40 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c8ea1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84c40e4e40 0x7f84c41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84c40e5240 0x7f84c4084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58260/0 (socket says 172.21.2.117:58260) 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c3fff700 1 -- 172.21.2.117:0/2640422274 learned_addr learned my addr 172.21.2.117:0/2640422274 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c8ea1700 1 -- 172.21.2.117:0/2640422274 >> v1:172.21.2.125:6789/0 conn(0x7f84c4085450 legacy=0x7f84c40d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c8ea1700 1 -- 172.21.2.117:0/2640422274 >> v1:172.21.2.117:6789/0 conn(0x7f84c40e5240 legacy=0x7f84c4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c8ea1700 1 -- 172.21.2.117:0/2640422274 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c41b0810 con 0x7f84c40e4e40 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.886 7f84c8ea1700 1 --2- 172.21.2.117:0/2640422274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40e4e40 0x7f84c41b01f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f84ac006650 tx=0x7f84ac009a00).ready entity=mon.0 client_cookie=b37d6c1c90969b0a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84c37fe700 1 -- 172.21.2.117:0/2640422274 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84ac0009b0 con 0x7f84c40e4e40 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84c37fe700 1 -- 172.21.2.117:0/2640422274 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00c0c0 con 0x7f84c40e4e40 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84c37fe700 1 -- 172.21.2.117:0/2640422274 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84ac00c5e0 con 0x7f84c40e4e40 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 -- 172.21.2.117:0/2640422274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40e4e40 msgr2=0x7f84c41b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 --2- 172.21.2.117:0/2640422274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40e4e40 0x7f84c41b01f0 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 rx=0x7f84ac006650 tx=0x7f84ac009a00).stop 2021-02-10T11:54:27.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 -- 172.21.2.117:0/2640422274 shutdown_connections 2021-02-10T11:54:27.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 --2- 172.21.2.117:0/2640422274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40e4e40 0x7f84c41b01f0 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 -- 172.21.2.117:0/2640422274 >> 172.21.2.117:0/2640422274 conn(0x7f84c40ce4e0 msgr2=0x7f84c40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:27.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 -- 172.21.2.117:0/2640422274 shutdown_connections 2021-02-10T11:54:27.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.887 7f84cb085700 1 -- 172.21.2.117:0/2640422274 wait complete. 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 Processor -- start 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 -- start start 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 0x7f84c41b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 0x7f84c40b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c410e2d0 con 0x7f84c409d010 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c41b39b0 con 0x7f84c409db10 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.902 7f84cb085700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c41b3600 con 0x7f84c409d410 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58018/0 (socket says 172.21.2.117:58018) 2021-02-10T11:54:27.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 -- 172.21.2.117:0/3313739872 learned_addr learned my addr 172.21.2.117:0/3313739872 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 -- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 msgr2=0x7f84c40b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c96a2700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 0x7f84c40b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c8ea1700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 0x7f84c41b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 0x7f84c40b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 -- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 msgr2=0x7f84c41b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 0x7f84c41b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 -- 172.21.2.117:0/3313739872 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40b5000 con 0x7f84c409d010 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c96a2700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 0x7f84c40b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c8ea1700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 0x7f84c41b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.903 7f84c3fff700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f84b4001490 tx=0x7f84b400a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.904 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b4000ef0 con 0x7f84c409d010 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.904 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b400ad40 con 0x7f84c409d010 2021-02-10T11:54:27.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.904 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b4004590 con 0x7f84c409d010 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.904 7f84cb085700 1 -- 172.21.2.117:0/3313739872 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c40b5000 con 0x7f84c409d010 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.904 7f84cb085700 1 -- 172.21.2.117:0/3313739872 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c40b5590 con 0x7f84c409d010 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.905 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84b40043a0 con 0x7f84c409d010 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.905 7f84c17fa700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.905 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f84b40326d0 con 0x7f84c409d010 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.905 7f84c8ea1700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:27.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.905 7f84c8ea1700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:27.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.935 7f84cb085700 1 -- 172.21.2.117:0/3313739872 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c410e2d0 con 0x7f84c409d010 2021-02-10T11:54:27.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:27.938 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84b4030d30 con 0x7f84c409d010 2021-02-10T11:54:28.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.061 7f84cb085700 1 -- 172.21.2.117:0/3313739872 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c410e2d0 con 0x7f84c409d010 2021-02-10T11:54:28.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.062 7f84c17fa700 1 -- 172.21.2.117:0/3313739872 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f84b40043a0 con 0x7f84c409d010 2021-02-10T11:54:28.064 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:28.064 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":160,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:25.819611","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:54:22.787648","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:54:23.861446","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 msgr2=0x7f84a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 crc :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 msgr2=0x7f84c40b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f84b4001490 tx=0x7f84b400a950).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 shutdown_connections 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c409d010 0x7f84c40b43c0 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c409d410 0x7f84c40b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 --2- 172.21.2.117:0/3313739872 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409db10 0x7f84c41b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 >> 172.21.2.117:0/3313739872 conn(0x7f84c4108ad0 msgr2=0x7f84c40c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:28.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 shutdown_connections 2021-02-10T11:54:28.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.064 7f84cb085700 1 -- 172.21.2.117:0/3313739872 wait complete. 2021-02-10T11:54:28.075 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.489 7f1993e1b700 1 Processor -- start 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.489 7f1993e1b700 1 -- start start 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f198c07ab40 0x7f198c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f198c079b10 0x7f198c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 0x7f198c07b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198c0e5770 con 0x7f198c07ab40 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198c0d18c0 con 0x7f198c079b10 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1993e1b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198c0d1c60 con 0x7f198c07af40 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f198c07ab40 0x7f198c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f198c07ab40 0x7f198c0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58024/0 (socket says 172.21.2.117:58024) 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 -- 172.21.2.117:0/1862817015 learned_addr learned my addr 172.21.2.117:0/1862817015 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991436700 1 --2- 172.21.2.117:0/1862817015 >> v2:172.21.2.125:3300/0 conn(0x7f198c079b10 0x7f198c079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1992438700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 0x7f198c07b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 -- 172.21.2.117:0/1862817015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 msgr2=0x7f198c07b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:28.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 0x7f198c07b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 -- 172.21.2.117:0/1862817015 >> v2:172.21.2.125:3300/0 conn(0x7f198c079b10 msgr2=0x7f198c079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 --2- 172.21.2.117:0/1862817015 >> v2:172.21.2.125:3300/0 conn(0x7f198c079b10 0x7f198c079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1991c37700 1 -- 172.21.2.117:0/1862817015 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c07b960 con 0x7f198c07ab40 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.490 7f1992438700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 0x7f198c07b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1991c37700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c07ab40 0x7f198c0794f0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f1988006700 tx=0x7f1988009b80).ready entity=mon.0 client_cookie=e5d7d65c6318c87d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1990c35700 1 -- 172.21.2.117:0/1862817015 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198800c0c0 con 0x7f198c07ab40 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1990c35700 1 -- 172.21.2.117:0/1862817015 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198800c440 con 0x7f198c07ab40 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1990c35700 1 -- 172.21.2.117:0/1862817015 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198800c960 con 0x7f198c07ab40 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1993e1b700 1 -- 172.21.2.117:0/1862817015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c07ab40 msgr2=0x7f198c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.491 7f1993e1b700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c07ab40 0x7f198c0794f0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7f1988006700 tx=0x7f1988009b80).stop 2021-02-10T11:54:28.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 -- 172.21.2.117:0/1862817015 shutdown_connections 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 --2- 172.21.2.117:0/1862817015 >> v2:172.21.2.125:3300/0 conn(0x7f198c079b10 0x7f198c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c07af40 0x7f198c07b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 --2- 172.21.2.117:0/1862817015 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c07ab40 0x7f198c0794f0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 -- 172.21.2.117:0/1862817015 >> 172.21.2.117:0/1862817015 conn(0x7f198c0d0910 msgr2=0x7f198c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 -- 172.21.2.117:0/1862817015 shutdown_connections 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.492 7f1993e1b700 1 -- 172.21.2.117:0/1862817015 wait complete. 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.507 7f1993e1b700 1 Processor -- start 2021-02-10T11:54:28.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.507 7f1993e1b700 1 -- start start 2021-02-10T11:54:28.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c0b0970 0x7f198c0fe930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f198c0afe80 0x7f198c1a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c0b0d70 con 0x7f198c0fef50 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1a3a20 con 0x7f198c0afe80 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1993e1b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198c1a3c40 con 0x7f198c0b0970 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58032/0 (socket says 172.21.2.117:58032) 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 -- 172.21.2.117:0/3264664323 learned_addr learned my addr 172.21.2.117:0/3264664323 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1991c37700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c0b0970 0x7f198c0fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1991436700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f198c0afe80 0x7f198c1a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 -- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c0b0970 msgr2=0x7f198c0fe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c0b0970 0x7f198c0fe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 -- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f198c0afe80 msgr2=0x7f198c1a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f198c0afe80 0x7f198c1a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.508 7f1992438700 1 -- 172.21.2.117:0/3264664323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0ffa00 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f1992438700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f1984009980 tx=0x7f198400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198400c2f0 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984006790 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198c0ffa00 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f198400c870 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.509 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198c0ffa00 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.510 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f19840011f0 con 0x7f198c0fef50 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.510 7f197e7fc700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 0x7f197401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:28.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.510 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f19840011f0 con 0x7f198c0fef50 2021-02-10T11:54:28.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.510 7f1991c37700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 0x7f197401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:28.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.511 7f1991c37700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 0x7f197401ac10 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:28.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.540 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198c0e5770 con 0x7f198c0fef50 2021-02-10T11:54:28.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.542 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1984000f40 con 0x7f198c0fef50 2021-02-10T11:54:28.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.670 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f198c0e5770 con 0x7f198c0fef50 2021-02-10T11:54:28.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.674 7f197e7fc700 1 -- 172.21.2.117:0/3264664323 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v89) v1 ==== 184+0+0 (secure 0 0 0) 0x7f19840011f0 con 0x7f198c0fef50 2021-02-10T11:54:28.679 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:54:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 msgr2=0x7f197401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 0x7f197401ac10 crc :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 msgr2=0x7f198c1a3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 rx=0x7f1984009980 tx=0x7f198400bcd0).stop 2021-02-10T11:54:28.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 shutdown_connections 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1974018730 0x7f197401ac10 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f198c0fef50 0x7f198c1a3400 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f198c0afe80 0x7f198c1a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 --2- 172.21.2.117:0/3264664323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f198c0b0970 0x7f198c0fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 >> 172.21.2.117:0/3264664323 conn(0x7f198c0bb610 msgr2=0x7f198c08d330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 shutdown_connections 2021-02-10T11:54:28.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:28.679 7f1993e1b700 1 -- 172.21.2.117:0/3264664323 wait complete. 2021-02-10T11:54:28.690 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:54:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.111 7efd083a8700 1 Processor -- start 2021-02-10T11:54:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.111 7efd083a8700 1 -- start start 2021-02-10T11:54:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd083a8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efd0007a0e0 0x7efd001b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd083a8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd000e5770 con 0x7efd00079ce0 2021-02-10T11:54:29.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd083a8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd000d18c0 con 0x7efd0007ab40 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd083a8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0007a640 con 0x7efd0007a0e0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd069c5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7efd0007a0e0 0x7efd001b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd059c3700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efd00079ce0 0x7efd001b59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58286/0 (socket says 172.21.2.117:58286) 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.112 7efd059c3700 1 -- 172.21.2.117:0/4204065759 learned_addr learned my addr 172.21.2.117:0/4204065759 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd069c5700 1 -- 172.21.2.117:0/4204065759 >> v1:172.21.2.125:6789/0 conn(0x7efd0007ab40 legacy=0x7efd000794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd069c5700 1 -- 172.21.2.117:0/4204065759 >> v1:172.21.2.117:6789/0 conn(0x7efd00079ce0 legacy=0x7efd001b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd069c5700 1 -- 172.21.2.117:0/4204065759 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0007bf90 con 0x7efd0007a0e0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd069c5700 1 --2- 172.21.2.117:0/4204065759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0007a0e0 0x7efd001b91e0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7efcf8006e30 tx=0x7efcf8009e60).ready entity=mon.0 client_cookie=75f6d20ae6d6196d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd051c2700 1 -- 172.21.2.117:0/4204065759 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcf8006a50 con 0x7efd0007a0e0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd051c2700 1 -- 172.21.2.117:0/4204065759 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf800b300 con 0x7efd0007a0e0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd051c2700 1 -- 172.21.2.117:0/4204065759 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcf800b7a0 con 0x7efd0007a0e0 2021-02-10T11:54:29.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd083a8700 1 -- 172.21.2.117:0/4204065759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0007a0e0 msgr2=0x7efd001b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.113 7efd083a8700 1 --2- 172.21.2.117:0/4204065759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0007a0e0 0x7efd001b91e0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 rx=0x7efcf8006e30 tx=0x7efcf8009e60).stop 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.114 7efd083a8700 1 -- 172.21.2.117:0/4204065759 shutdown_connections 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.114 7efd083a8700 1 --2- 172.21.2.117:0/4204065759 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0007a0e0 0x7efd001b91e0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.114 7efd083a8700 1 -- 172.21.2.117:0/4204065759 >> 172.21.2.117:0/4204065759 conn(0x7efd000d0910 msgr2=0x7efd000d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.114 7efd083a8700 1 -- 172.21.2.117:0/4204065759 shutdown_connections 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.114 7efd083a8700 1 -- 172.21.2.117:0/4204065759 wait complete. 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 Processor -- start 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 -- start start 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd0009b8c0 0x7efd0009bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd000fff40 0x7efd00100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd000c1ac0 con 0x7efd0009c2e0 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd00100960 con 0x7efd000fff40 2021-02-10T11:54:29.133 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd083a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd00100b80 con 0x7efd0009b8c0 2021-02-10T11:54:29.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.129 7efd059c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58046/0 (socket says 172.21.2.117:58046) 2021-02-10T11:54:29.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 -- 172.21.2.117:0/624999445 learned_addr learned my addr 172.21.2.117:0/624999445 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 -- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd0009b8c0 msgr2=0x7efd0009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd0009b8c0 0x7efd0009bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 -- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd000fff40 msgr2=0x7efd00100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd000fff40 0x7efd00100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 -- 172.21.2.117:0/624999445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd000c05c0 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd059c3700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7efcf0001410 tx=0x7efcf000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcf000ad40 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf0000c90 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcf0004590 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd083a8700 1 -- 172.21.2.117:0/624999445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd000c05c0 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.130 7efd083a8700 1 -- 172.21.2.117:0/624999445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd000c05c0 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.131 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcf00043a0 con 0x7efd0009c2e0 2021-02-10T11:54:29.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.131 7efcf6ffd700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 0x7efce801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.131 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efcf0032960 con 0x7efd0009c2e0 2021-02-10T11:54:29.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.131 7efd061c4700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 0x7efce801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.132 7efd061c4700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 0x7efce801abc0 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.162 7efd083a8700 1 -- 172.21.2.117:0/624999445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd000d1c60 con 0x7efd0009c2e0 2021-02-10T11:54:29.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.164 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcf001f550 con 0x7efd0009c2e0 2021-02-10T11:54:29.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.293 7efd083a8700 1 -- 172.21.2.117:0/624999445 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7efd000d1c60 con 0x7efd0009c2e0 2021-02-10T11:54:29.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.296 7efcf6ffd700 1 -- 172.21.2.117:0/624999445 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v90) v1 ==== 184+0+0 (secure 0 0 0) 0x7efcf00043a0 con 0x7efd0009c2e0 2021-02-10T11:54:29.302 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 msgr2=0x7efce801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 0x7efce801abc0 crc :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 msgr2=0x7efd000ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 rx=0x7efcf0001410 tx=0x7efcf000a950).stop 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 shutdown_connections 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efce80186c0 0x7efce801abc0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efd0009c2e0 0x7efd000ff920 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efd000fff40 0x7efd00100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 --2- 172.21.2.117:0/624999445 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efd0009b8c0 0x7efd0009bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 >> 172.21.2.117:0/624999445 conn(0x7efd0009e020 msgr2=0x7efd000fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:29.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 shutdown_connections 2021-02-10T11:54:29.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.302 7efd083a8700 1 -- 172.21.2.117:0/624999445 wait complete. 2021-02-10T11:54:29.312 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:54:29.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 Processor -- start 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 -- start start 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc079ce0 0x7fb3dc1b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0e5770 con 0x7fb3dc07ab40 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc0d18c0 con 0x7fb3dc07a0e0 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.735 7fb3e393e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc07bf50 con 0x7fb3dc079ce0 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3e175a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb3dc07ab40 0x7fb3dc0794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58300/0 (socket says 172.21.2.117:58300) 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3e175a700 1 -- 172.21.2.117:0/2681582021 learned_addr learned my addr 172.21.2.117:0/2681582021 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036379583 0 0) 0x7fb3d0001180 con 0x7fb3dc07ab40 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c8002fe0 con 0x7fb3dc07ab40 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3e0f59700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc079ce0 0x7fb3dc1b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 161639420 0 0) 0x7fb3d0001180 con 0x7fb3dc07ab40 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc079ce0 msgr2=0x7fb3dc1b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc079ce0 0x7fb3dc1b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 >> v1:172.21.2.125:6789/0 conn(0x7fb3dc07a0e0 legacy=0x7fb3dc1b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0d1c60 con 0x7fb3dc07ab40 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fb3d0001680 con 0x7fb3dc07ab40 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 >> v1:172.21.2.117:6789/0 conn(0x7fb3dc07ab40 legacy=0x7fb3dc0794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3c8003a60 0x7fb3c8005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3c80067d0 0x7fb3c8008c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3c8009520 0x7fb3c800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.736 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800c2d0 con 0x7fb3c80067d0 2021-02-10T11:54:29.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800c4f0 con 0x7fb3c8009520 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c800c710 con 0x7fb3c8003a60 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3c80067d0 0x7fb3c8008c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e175a700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3c8003a60 0x7fb3c8005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e1f5b700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3c8009520 0x7fb3c800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 -- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3c8003a60 msgr2=0x7fb3c8005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3c8003a60 0x7fb3c8005f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 -- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3c8009520 msgr2=0x7fb3c800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3c8009520 0x7fb3c800ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 -- 172.21.2.117:0/2681582021 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c8050980 con 0x7fb3c80067d0 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e0f59700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3c80067d0 0x7fb3c8008c80 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7fb3d8009310 tx=0x7fb3d8009a10).ready entity=mon.0 client_cookie=69115bdac10710ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3d80009b0 con 0x7fb3c80067d0 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d8006400 con 0x7fb3c80067d0 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3cffff700 1 -- 172.21.2.117:0/2681582021 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3d800c340 con 0x7fb3c80067d0 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e393e700 1 -- 172.21.2.117:0/2681582021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3c80067d0 msgr2=0x7fb3c8008c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e393e700 1 --2- 172.21.2.117:0/2681582021 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3c80067d0 0x7fb3c8008c80 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 rx=0x7fb3d8009310 tx=0x7fb3d8009a10).stop 2021-02-10T11:54:29.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.737 7fb3e1f5b700 1 -- 172.21.2.117:0/2681582021 reap_dead start 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.738 7fb3e393e700 1 -- 172.21.2.117:0/2681582021 shutdown_connections 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.738 7fb3e393e700 1 -- 172.21.2.117:0/2681582021 >> 172.21.2.117:0/2681582021 conn(0x7fb3dc0d0910 msgr2=0x7fb3dc0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.738 7fb3e393e700 1 -- 172.21.2.117:0/2681582021 shutdown_connections 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.738 7fb3e393e700 1 -- 172.21.2.117:0/2681582021 wait complete. 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 Processor -- start 2021-02-10T11:54:29.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 -- start start 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc0b2190 0x7fb3dc1a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3dc08c3e0 0x7fb3dc100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0b2770 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0dce90 con 0x7fb3dc08c3e0 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.753 7fb3e393e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0ffdd0 con 0x7fb3dc0b2190 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58064/0 (socket says 172.21.2.117:58064) 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 -- 172.21.2.117:0/889187384 learned_addr learned my addr 172.21.2.117:0/889187384 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 -- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc0b2190 msgr2=0x7fb3dc1a34c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc0b2190 0x7fb3dc1a34c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 -- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3dc08c3e0 msgr2=0x7fb3dc100320 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:29.761 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3dc08c3e0 0x7fb3dc100320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 -- 172.21.2.117:0/889187384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc100940 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e0f59700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7fb3d8009fc0 tx=0x7fb3d800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3d800bd60 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d8006030 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3d80052e0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e393e700 1 -- 172.21.2.117:0/889187384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc100940 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.754 7fb3e393e700 1 -- 172.21.2.117:0/889187384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc100ed0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.755 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3d80052e0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.755 7fb3cdffb700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 0x7fb3bc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:29.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.755 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb3d8032700 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.755 7fb3e175a700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 0x7fb3bc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:29.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.756 7fb3e175a700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 0x7fb3bc01abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:29.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.785 7fb3e393e700 1 -- 172.21.2.117:0/889187384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc0ffda0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.788 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3d80009b0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.920 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.918 7fb3e393e700 1 -- 172.21.2.117:0/889187384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb3dc0ffda0 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.921 7fb3cdffb700 1 -- 172.21.2.117:0/889187384 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v91) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb3d801f510 con 0x7fb3dc1a3ae0 2021-02-10T11:54:29.923 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:54:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.922 7fb3e393e700 1 -- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 msgr2=0x7fb3bc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.922 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 0x7fb3bc01abc0 crc :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.922 7fb3e393e700 1 -- 172.21.2.117:0/889187384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 msgr2=0x7fb3dc08bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:29.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.922 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7fb3d8009fc0 tx=0x7fb3d800b970).stop 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 -- 172.21.2.117:0/889187384 shutdown_connections 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3bc0186c0 0x7fb3bc01abc0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3dc1a3ae0 0x7fb3dc08bdc0 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3dc08c3e0 0x7fb3dc100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 --2- 172.21.2.117:0/889187384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3dc0b2190 0x7fb3dc1a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 -- 172.21.2.117:0/889187384 >> 172.21.2.117:0/889187384 conn(0x7fb3dc09f900 msgr2=0x7fb3dc0fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 -- 172.21.2.117:0/889187384 shutdown_connections 2021-02-10T11:54:29.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:29.923 7fb3e393e700 1 -- 172.21.2.117:0/889187384 wait complete. 2021-02-10T11:54:29.933 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:54:30.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.354 7f2d4d792700 1 Processor -- start 2021-02-10T11:54:30.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.354 7f2d4d792700 1 -- start start 2021-02-10T11:54:30.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d4d792700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2d48079ce0 0x7f2d481b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d4d792700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4807a5c0 con 0x7f2d4807ab40 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d4d792700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d480d18c0 con 0x7f2d4807a0e0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d4d792700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4807bf50 con 0x7f2d48079ce0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d46d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2d48079ce0 0x7f2d481b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d47574700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2d4807ab40 0x7f2d480794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58320/0 (socket says 172.21.2.117:58320) 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d47574700 1 -- 172.21.2.117:0/994083380 learned_addr learned my addr 172.21.2.117:0/994083380 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d46d73700 1 -- 172.21.2.117:0/994083380 >> v1:172.21.2.125:6789/0 conn(0x7f2d4807a0e0 legacy=0x7f2d481b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d46d73700 1 -- 172.21.2.117:0/994083380 >> v1:172.21.2.117:6789/0 conn(0x7f2d4807ab40 legacy=0x7f2d480794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d46d73700 1 -- 172.21.2.117:0/994083380 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d480d1c60 con 0x7f2d48079ce0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.355 7f2d46d73700 1 --2- 172.21.2.117:0/994083380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d48079ce0 0x7f2d481b59d0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7f2d400069d0 tx=0x7f2d40009e40).ready entity=mon.0 client_cookie=7c2420125ede0e6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d46572700 1 -- 172.21.2.117:0/994083380 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d4000b510 con 0x7f2d48079ce0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d46572700 1 -- 172.21.2.117:0/994083380 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4000b790 con 0x7f2d48079ce0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d46572700 1 -- 172.21.2.117:0/994083380 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d4000bcb0 con 0x7f2d48079ce0 2021-02-10T11:54:30.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 -- 172.21.2.117:0/994083380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d48079ce0 msgr2=0x7f2d481b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 --2- 172.21.2.117:0/994083380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d48079ce0 0x7f2d481b59d0 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 rx=0x7f2d400069d0 tx=0x7f2d40009e40).stop 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 -- 172.21.2.117:0/994083380 shutdown_connections 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 --2- 172.21.2.117:0/994083380 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d48079ce0 0x7f2d481b59d0 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 -- 172.21.2.117:0/994083380 >> 172.21.2.117:0/994083380 conn(0x7f2d480d0910 msgr2=0x7f2d480d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 -- 172.21.2.117:0/994083380 shutdown_connections 2021-02-10T11:54:30.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.356 7f2d4d792700 1 -- 172.21.2.117:0/994083380 wait complete. 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.371 7f2d4d792700 1 Processor -- start 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.371 7f2d4d792700 1 -- start start 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d481c9e60 0x7f2d480c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 0x7f2d480c06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480b90e0 con 0x7f2d480b8c10 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d48100c20 con 0x7f2d481c9e60 2021-02-10T11:54:30.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d4d792700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d480c31a0 con 0x7f2d480c2da0 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58076/0 (socket says 172.21.2.117:58076) 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 -- 172.21.2.117:0/3984872902 learned_addr learned my addr 172.21.2.117:0/3984872902 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d46d73700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d481c9e60 0x7f2d480c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47d75700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 0x7f2d480c06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 -- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 msgr2=0x7f2d480c06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 0x7f2d480c06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 -- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d481c9e60 msgr2=0x7f2d480c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d481c9e60 0x7f2d480c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 -- 172.21.2.117:0/3984872902 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d480c33c0 con 0x7f2d480b8c10 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47d75700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 0x7f2d480c06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.372 7f2d47574700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7f2d3c001410 tx=0x7f2d3c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:30.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.373 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d3c00ad40 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.373 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3c0009b0 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.373 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2d3c004590 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.373 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d480c33c0 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.373 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d480c0fd0 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.374 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2d3c0043a0 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.374 7f2d2ffff700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 0x7f2d2801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.374 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2d3c0326d0 con 0x7f2d480b8c10 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.374 7f2d46d73700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 0x7f2d2801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.374 7f2d46d73700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 0x7f2d2801abc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:30.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.405 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d480d0910 con 0x7f2d480b8c10 2021-02-10T11:54:30.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.407 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2d3c030d30 con 0x7f2d480b8c10 2021-02-10T11:54:30.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.538 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2d480d0910 con 0x7f2d480b8c10 2021-02-10T11:54:30.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.541 7f2d2ffff700 1 -- 172.21.2.117:0/3984872902 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v92) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2d3c01f6f0 con 0x7f2d480b8c10 2021-02-10T11:54:30.547 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:54:30.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 msgr2=0x7f2d2801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 0x7f2d2801abc0 crc :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 msgr2=0x7f2d481c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 rx=0x7f2d3c001410 tx=0x7f2d3c00a950).stop 2021-02-10T11:54:30.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 shutdown_connections 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2d480c2da0 0x7f2d480c06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2d481c9e60 0x7f2d480c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2d280186c0 0x7f2d2801abc0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 --2- 172.21.2.117:0/3984872902 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2d480b8c10 0x7f2d481c9840 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.547 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 >> 172.21.2.117:0/3984872902 conn(0x7f2d4809e020 msgr2=0x7f2d4809cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.548 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 shutdown_connections 2021-02-10T11:54:30.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.548 7f2d4d792700 1 -- 172.21.2.117:0/3984872902 wait complete. 2021-02-10T11:54:30.557 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:30.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 Processor -- start 2021-02-10T11:54:30.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 -- start start 2021-02-10T11:54:30.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e640ef3f0 0x7f5e640eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e640eb7f0 0x7f5e641b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e640ede60 con 0x7f5e640d18e0 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e640ef7f0 con 0x7f5e640ef3f0 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.966 7f5e6a2de700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e640d1ce0 con 0x7f5e640eb7f0 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e640eb7f0 0x7f5e641b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e640eb7f0 0x7f5e641b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:34238/0 (socket says 172.21.2.117:34238) 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 -- 172.21.2.117:0/2169686138 learned_addr learned my addr 172.21.2.117:0/2169686138 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e637fe700 1 --2- 172.21.2.117:0/2169686138 >> v2:172.21.2.117:3300/0 conn(0x7f5e640ef3f0 0x7f5e640eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 -- 172.21.2.117:0/2169686138 >> v2:172.21.2.117:3300/0 conn(0x7f5e640ef3f0 msgr2=0x7f5e640eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 --2- 172.21.2.117:0/2169686138 >> v2:172.21.2.117:3300/0 conn(0x7f5e640ef3f0 0x7f5e640eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 -- 172.21.2.117:0/2169686138 >> v1:172.21.2.117:6789/0 conn(0x7f5e640d18e0 legacy=0x7f5e640ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 -- 172.21.2.117:0/2169686138 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e641b8f90 con 0x7f5e640eb7f0 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e688fb700 1 --2- 172.21.2.117:0/2169686138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e640eb7f0 0x7f5e641b7960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f5e5c006aa0 tx=0x7f5e5c009f20).ready entity=mon.1 client_cookie=26c123e789348278 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e62ffd700 1 -- 172.21.2.117:0/2169686138 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e5c00b5c0 con 0x7f5e640eb7f0 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e62ffd700 1 -- 172.21.2.117:0/2169686138 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c003030 con 0x7f5e640eb7f0 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e62ffd700 1 -- 172.21.2.117:0/2169686138 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e5c00b9f0 con 0x7f5e640eb7f0 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e6a2de700 1 -- 172.21.2.117:0/2169686138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e640eb7f0 msgr2=0x7f5e641b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.967 7f5e6a2de700 1 --2- 172.21.2.117:0/2169686138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e640eb7f0 0x7f5e641b7960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 rx=0x7f5e5c006aa0 tx=0x7f5e5c009f20).stop 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 -- 172.21.2.117:0/2169686138 shutdown_connections 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 --2- 172.21.2.117:0/2169686138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e640eb7f0 0x7f5e641b7960 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 --2- 172.21.2.117:0/2169686138 >> v2:172.21.2.117:3300/0 conn(0x7f5e640ef3f0 0x7f5e640eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 -- 172.21.2.117:0/2169686138 >> 172.21.2.117:0/2169686138 conn(0x7f5e640ced60 msgr2=0x7f5e640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 -- 172.21.2.117:0/2169686138 shutdown_connections 2021-02-10T11:54:30.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.968 7f5e6a2de700 1 -- 172.21.2.117:0/2169686138 wait complete. 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 Processor -- start 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 -- start start 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e641a33d0 0x7f5e641a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e640b8d10 0x7f5e640b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640c0670 con 0x7f5e640b82f0 2021-02-10T11:54:30.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640deb20 con 0x7f5e641a33d0 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.983 7f5e6a2de700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e640ded40 con 0x7f5e640b8d10 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58088/0 (socket says 172.21.2.117:58088) 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 -- 172.21.2.117:0/2263294837 learned_addr learned my addr 172.21.2.117:0/2263294837 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e63fff700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e641a33d0 0x7f5e641a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e688fb700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e640b8d10 0x7f5e640b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 -- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e640b8d10 msgr2=0x7f5e640b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e640b8d10 0x7f5e640b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 -- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e641a33d0 msgr2=0x7f5e641a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e641a33d0 0x7f5e641a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 -- 172.21.2.117:0/2263294837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e640def60 con 0x7f5e640b82f0 2021-02-10T11:54:30.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e637fe700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f5e5800b560 tx=0x7f5e5800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e580009b0 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5800bdc0 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e58002030 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e640def60 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.984 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e640df4f0 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.985 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e58000c90 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.985 7f5e60ff9700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 0x7f5e4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.985 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e580327b0 con 0x7f5e640b82f0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.985 7f5e63fff700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 0x7f5e4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:30.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:30.986 7f5e63fff700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 0x7f5e4401abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:31.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.016 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e64087290 con 0x7f5e640b82f0 2021-02-10T11:54:31.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.019 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e58000940 con 0x7f5e640b82f0 2021-02-10T11:54:31.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.147 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e640ede60 con 0x7f5e640b82f0 2021-02-10T11:54:31.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.147 7f5e60ff9700 1 -- 172.21.2.117:0/2263294837 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+2099 (secure 0 0 0) 0x7f5e5800bdc0 con 0x7f5e640b82f0 2021-02-10T11:54:31.150 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:31.150 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:54:26.825269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7697":{"gid":7697,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/95294604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":95294604},{"type":"v1","addr":"172.21.2.125:6835","nonce":95294604}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:54:31.151 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 136 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 msgr2=0x7f5e4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 0x7f5e4401abc0 crc :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 msgr2=0x7f5e640b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f5e5800b560 tx=0x7f5e5800b900).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 shutdown_connections 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e440186c0 0x7f5e4401abc0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e640b82f0 0x7f5e640b86f0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e640b8d10 0x7f5e640b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 --2- 172.21.2.117:0/2263294837 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e641a33d0 0x7f5e641a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:31.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 >> 172.21.2.117:0/2263294837 conn(0x7f5e640874c0 msgr2=0x7f5e64096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:31.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 shutdown_connections 2021-02-10T11:54:31.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:31.149 7f5e6a2de700 1 -- 172.21.2.117:0/2263294837 wait complete. 2021-02-10T11:54:31.362 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 136, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:54:24.817179', 'modified': '2021-02-10 11:54:26.825269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7697': {'gid': 7697, 'name': 'b', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/95294604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 95294604}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 95294604}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:54:31.362 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:54:31.362 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:54:31.761 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:54:31.761 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:54:31.761 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:54:31.761 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7697, 2021-02-10T11:54:31.761 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.939803779, 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 136, 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 160, 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 160, 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.575286257 2021-02-10T11:54:31.762 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:54:31.765 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7697, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.939803779, "mdsmap_epoch": 136, "osdmap_epoch": 160, "osdmap_epoch_barrier": 160, "uptime": 12.575286257 } 2021-02-10T11:54:31.765 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.174 7fbf4d5ad700 1 Processor -- start 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.174 7fbf4d5ad700 1 -- start start 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf4d5ad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbf481b6960 0x7fbf481b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf4d5ad700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480ede60 con 0x7fbf480ef3f0 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf4d5ad700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf480eb9c0 con 0x7fbf480d18e0 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf4d5ad700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf480ef7f0 con 0x7fbf481b6960 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf46abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbf480ef3f0 0x7fbf480eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58346/0 (socket says 172.21.2.117:58346) 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf46abe700 1 -- 172.21.2.117:0/716217641 learned_addr learned my addr 172.21.2.117:0/716217641 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:32.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf47ac0700 1 --2- 172.21.2.117:0/716217641 >> v2:172.21.2.125:3300/0 conn(0x7fbf481b6960 0x7fbf481b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf462bd700 1 -- 172.21.2.117:0/716217641 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870727274 0 0) 0x7fbf30001180 con 0x7fbf480ef3f0 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.175 7fbf462bd700 1 -- 172.21.2.117:0/716217641 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34002fe0 con 0x7fbf480ef3f0 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf462bd700 1 -- 172.21.2.117:0/716217641 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 667278217 0 0) 0x7fbf30001180 con 0x7fbf480ef3f0 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf47ac0700 1 -- 172.21.2.117:0/716217641 >> v1:172.21.2.125:6789/0 conn(0x7fbf480d18e0 legacy=0x7fbf480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf47ac0700 1 -- 172.21.2.117:0/716217641 >> v1:172.21.2.117:6789/0 conn(0x7fbf480ef3f0 legacy=0x7fbf480eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf47ac0700 1 -- 172.21.2.117:0/716217641 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480d1d20 con 0x7fbf481b6960 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf47ac0700 1 --2- 172.21.2.117:0/716217641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf481b6960 0x7fbf481b8d70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbf40006e50 tx=0x7fbf40009e70).ready entity=mon.1 client_cookie=d981561b0e762466 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf462bd700 1 -- 172.21.2.117:0/716217641 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4000b510 con 0x7fbf481b6960 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf462bd700 1 -- 172.21.2.117:0/716217641 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf40006790 con 0x7fbf481b6960 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf462bd700 1 -- 172.21.2.117:0/716217641 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4000b940 con 0x7fbf481b6960 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf4d5ad700 1 -- 172.21.2.117:0/716217641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf481b6960 msgr2=0x7fbf481b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.176 7fbf4d5ad700 1 --2- 172.21.2.117:0/716217641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf481b6960 0x7fbf481b8d70 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 rx=0x7fbf40006e50 tx=0x7fbf40009e70).stop 2021-02-10T11:54:32.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.177 7fbf4d5ad700 1 -- 172.21.2.117:0/716217641 shutdown_connections 2021-02-10T11:54:32.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.177 7fbf4d5ad700 1 --2- 172.21.2.117:0/716217641 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf481b6960 0x7fbf481b8d70 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.177 7fbf4d5ad700 1 -- 172.21.2.117:0/716217641 >> 172.21.2.117:0/716217641 conn(0x7fbf480ced60 msgr2=0x7fbf480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:32.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.177 7fbf4d5ad700 1 -- 172.21.2.117:0/716217641 shutdown_connections 2021-02-10T11:54:32.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.177 7fbf4d5ad700 1 -- 172.21.2.117:0/716217641 wait complete. 2021-02-10T11:54:32.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.191 7fbf4d5ad700 1 Processor -- start 2021-02-10T11:54:32.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 -- start start 2021-02-10T11:54:32.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf480ab380 0x7fbf480dd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf4819ade0 0x7fbf4819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4819a860 con 0x7fbf4819b800 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48190cd0 con 0x7fbf480ab380 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf4d5ad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf48190eb0 con 0x7fbf4819ade0 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf47ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf47ac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58102/0 (socket says 172.21.2.117:58102) 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.192 7fbf47ac0700 1 -- 172.21.2.117:0/1313282959 learned_addr learned my addr 172.21.2.117:0/1313282959 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 -- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf4819ade0 msgr2=0x7fbf4819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf4819ade0 0x7fbf4819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 -- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf480ab380 msgr2=0x7fbf480dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf480ab380 0x7fbf480dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 -- 172.21.2.117:0/1313282959 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf481910d0 con 0x7fbf4819b800 2021-02-10T11:54:32.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf47ac0700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7fbf40009a90 tx=0x7fbf4000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf40002ef0 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf400092c0 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf481910d0 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf48191660 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.193 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbf4000c4e0 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.194 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbf400094a0 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.194 7fbf3ffff700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 0x7fbf2c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.194 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbf40032c90 con 0x7fbf4819b800 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.195 7fbf472bf700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 0x7fbf2c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.195 7fbf472bf700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 0x7fbf2c01abc0 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.225 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf48087210 con 0x7fbf4819b800 2021-02-10T11:54:32.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.227 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbf400094d0 con 0x7fbf4819b800 2021-02-10T11:54:32.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.351 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf480ede60 con 0x7fbf4819b800 2021-02-10T11:54:32.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.352 7fbf3ffff700 1 -- 172.21.2.117:0/1313282959 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fbf400092c0 con 0x7fbf4819b800 2021-02-10T11:54:32.355 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:32.355 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":160,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:25.819611","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:54:22.787648","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:54:23.861446","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:32.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 msgr2=0x7fbf2c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 0x7fbf2c01abc0 crc :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 msgr2=0x7fbf481906b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 rx=0x7fbf40009a90 tx=0x7fbf4000bcf0).stop 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 shutdown_connections 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbf2c0186c0 0x7fbf2c01abc0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbf4819b800 0x7fbf481906b0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbf4819ade0 0x7fbf4819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 --2- 172.21.2.117:0/1313282959 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbf480ab380 0x7fbf480dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.353 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 >> 172.21.2.117:0/1313282959 conn(0x7fbf481b67a0 msgr2=0x7fbf480e9560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.354 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 shutdown_connections 2021-02-10T11:54:32.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.354 7fbf4d5ad700 1 -- 172.21.2.117:0/1313282959 wait complete. 2021-02-10T11:54:32.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:54:22.787648', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:54:23.861446', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:54:32.365 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:54:32.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.777 7f13abe03700 1 Processor -- start 2021-02-10T11:54:32.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13abe03700 1 -- start start 2021-02-10T11:54:32.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13abe03700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13a41b69a0 0x7f13a41b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13abe03700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40ede60 con 0x7f13a40d18f0 2021-02-10T11:54:32.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13abe03700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40eba00 con 0x7f13a40ef3f0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13abe03700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40d1cf0 con 0x7f13a41b69a0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13a41b69a0 0x7f13a41b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f13a41b69a0 0x7f13a41b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58114/0 (socket says 172.21.2.117:58114) 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 -- 172.21.2.117:0/2384683628 learned_addr learned my addr 172.21.2.117:0/2384683628 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 -- 172.21.2.117:0/2384683628 >> v1:172.21.2.125:6789/0 conn(0x7f13a40ef3f0 legacy=0x7f13a40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 -- 172.21.2.117:0/2384683628 >> v1:172.21.2.117:6789/0 conn(0x7f13a40d18f0 legacy=0x7f13a40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.778 7f13aa420700 1 -- 172.21.2.117:0/2384683628 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a41bb3f0 con 0x7f13a41b69a0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13aa420700 1 --2- 172.21.2.117:0/2384683628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a41b69a0 0x7f13a41b8db0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f139c006f00 tx=0x7f139c009f10).ready entity=mon.0 client_cookie=b6518e07ef43ee84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13a8c1d700 1 -- 172.21.2.117:0/2384683628 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f139c00b3b0 con 0x7f13a41b69a0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13a8c1d700 1 -- 172.21.2.117:0/2384683628 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c00b730 con 0x7f13a41b69a0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13a8c1d700 1 -- 172.21.2.117:0/2384683628 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f139c00bc50 con 0x7f13a41b69a0 2021-02-10T11:54:32.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 -- 172.21.2.117:0/2384683628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a41b69a0 msgr2=0x7f13a41b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 --2- 172.21.2.117:0/2384683628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a41b69a0 0x7f13a41b8db0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 rx=0x7f139c006f00 tx=0x7f139c009f10).stop 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 -- 172.21.2.117:0/2384683628 shutdown_connections 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 --2- 172.21.2.117:0/2384683628 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a41b69a0 0x7f13a41b8db0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 -- 172.21.2.117:0/2384683628 >> 172.21.2.117:0/2384683628 conn(0x7f13a40cf060 msgr2=0x7f13a40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 -- 172.21.2.117:0/2384683628 shutdown_connections 2021-02-10T11:54:32.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.779 7f13abe03700 1 -- 172.21.2.117:0/2384683628 wait complete. 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 Processor -- start 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 -- start start 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13a40a66b0 0x7f13a41c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13a40a6ab0 0x7f13a41c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40df000 con 0x7f13a40e96c0 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a409f8d0 con 0x7f13a40a6ab0 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13abe03700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41c5a30 con 0x7f13a40a66b0 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13a9c1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13a9c1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58116/0 (socket says 172.21.2.117:58116) 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.794 7f13a9c1f700 1 -- 172.21.2.117:0/1099742578 learned_addr learned my addr 172.21.2.117:0/1099742578 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 -- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13a40a66b0 msgr2=0x7f13a41c4df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:32.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13a40a66b0 0x7f13a41c4df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 -- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13a40a6ab0 msgr2=0x7f13a41c5410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13a40a6ab0 0x7f13a41c5410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 -- 172.21.2.117:0/1099742578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a41c5c10 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13a9c1f700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f1398006ed0 tx=0x7f139800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1398006020 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13980009b0 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13abe03700 1 -- 172.21.2.117:0/1099742578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a41c5c10 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.795 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f13980054b0 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.796 7f13abe03700 1 -- 172.21.2.117:0/1099742578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a41c5c10 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.796 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f13980054b0 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.796 7f13967fc700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 0x7f138c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.796 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f13980327d0 con 0x7f13a40e96c0 2021-02-10T11:54:32.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.797 7f13a941e700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 0x7f138c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:32.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.797 7f13a941e700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 0x7f138c01abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:32.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.828 7f13abe03700 1 -- 172.21.2.117:0/1099742578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a4087390 con 0x7f13a40e96c0 2021-02-10T11:54:32.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.830 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1398006260 con 0x7f13a40e96c0 2021-02-10T11:54:32.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.956 7f13abe03700 1 -- 172.21.2.117:0/1099742578 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f13a40df000 con 0x7f13a40e96c0 2021-02-10T11:54:32.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.957 7f13967fc700 1 -- 172.21.2.117:0/1099742578 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7f13980054b0 con 0x7f13a40e96c0 2021-02-10T11:54:32.960 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:32.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.958 7f13abe03700 1 -- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 msgr2=0x7f138c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.958 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 0x7f138c01abc0 crc :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.958 7f13abe03700 1 -- 172.21.2.117:0/1099742578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 msgr2=0x7f13a40e8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.958 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f1398006ed0 tx=0x7f139800b960).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 -- 172.21.2.117:0/1099742578 shutdown_connections 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f13a40a6ab0 0x7f13a41c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f13a40a66b0 0x7f13a41c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f138c018730 0x7f138c01abc0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 --2- 172.21.2.117:0/1099742578 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f13a40e96c0 0x7f13a40e8e80 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 -- 172.21.2.117:0/1099742578 >> 172.21.2.117:0/1099742578 conn(0x7f13a40e71d0 msgr2=0x7f13a40b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 -- 172.21.2.117:0/1099742578 shutdown_connections 2021-02-10T11:54:32.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:32.959 7f13abe03700 1 -- 172.21.2.117:0/1099742578 wait complete. 2021-02-10T11:54:32.970 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:54:33.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 Processor -- start 2021-02-10T11:54:33.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 -- start start 2021-02-10T11:54:33.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a780ef3f0 0x7f9a780eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a780eb830 0x7f9a781b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a780ef7f0 con 0x7f9a780d18f0 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a780eded0 con 0x7f9a780eb830 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.382 7f9a7f995700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780d1cf0 con 0x7f9a780ef3f0 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a780eb830 0x7f9a781b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9a780eb830 0x7f9a781b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58128/0 (socket says 172.21.2.117:58128) 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 -- 172.21.2.117:0/2735605474 learned_addr learned my addr 172.21.2.117:0/2735605474 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 -- 172.21.2.117:0/2735605474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a780ef3f0 msgr2=0x7f9a780eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 --2- 172.21.2.117:0/2735605474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a780ef3f0 0x7f9a780eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 -- 172.21.2.117:0/2735605474 >> v1:172.21.2.125:6789/0 conn(0x7f9a780d18f0 legacy=0x7f9a780eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:33.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 -- 172.21.2.117:0/2735605474 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a781b8fd0 con 0x7f9a780eb830 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.383 7f9a7dfb2700 1 --2- 172.21.2.117:0/2735605474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780eb830 0x7f9a781b79a0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f9a70006f00 tx=0x7f9a70009e90).ready entity=mon.0 client_cookie=9d6193ae6cc968eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a6bfff700 1 -- 172.21.2.117:0/2735605474 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7000b300 con 0x7f9a780eb830 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a6bfff700 1 -- 172.21.2.117:0/2735605474 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7000b640 con 0x7f9a780eb830 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a6bfff700 1 -- 172.21.2.117:0/2735605474 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7000ba90 con 0x7f9a780eb830 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 -- 172.21.2.117:0/2735605474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780eb830 msgr2=0x7f9a781b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 --2- 172.21.2.117:0/2735605474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780eb830 0x7f9a781b79a0 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f9a70006f00 tx=0x7f9a70009e90).stop 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 -- 172.21.2.117:0/2735605474 shutdown_connections 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 --2- 172.21.2.117:0/2735605474 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780eb830 0x7f9a781b79a0 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 --2- 172.21.2.117:0/2735605474 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a780ef3f0 0x7f9a780eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.384 7f9a7f995700 1 -- 172.21.2.117:0/2735605474 >> 172.21.2.117:0/2735605474 conn(0x7f9a780cf060 msgr2=0x7f9a780cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.385 7f9a7f995700 1 -- 172.21.2.117:0/2735605474 shutdown_connections 2021-02-10T11:54:33.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.385 7f9a7f995700 1 -- 172.21.2.117:0/2735605474 wait complete. 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.399 7f9a7f995700 1 Processor -- start 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.399 7f9a7f995700 1 -- start start 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a7809e160 0x7f9a781a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a781a4860 0x7f9a780a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780a1820 con 0x7f9a780a65a0 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a780a69a0 con 0x7f9a7809e160 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7f995700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7809dc80 con 0x7f9a781a4860 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:33.405 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58132/0 (socket says 172.21.2.117:58132) 2021-02-10T11:54:33.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 -- 172.21.2.117:0/576831391 learned_addr learned my addr 172.21.2.117:0/576831391 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:33.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7cfb0700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a781a4860 0x7f9a780a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:33.406 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7d7b1700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a7809e160 0x7f9a781a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 -- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a781a4860 msgr2=0x7f9a780a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a781a4860 0x7f9a780a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 -- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a7809e160 msgr2=0x7f9a781a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a7809e160 0x7f9a781a4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.400 7f9a7dfb2700 1 -- 172.21.2.117:0/576831391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780a23c0 con 0x7f9a780a65a0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a7dfb2700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f9a7000b890 tx=0x7f9a7000bc30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a7000bd70 con 0x7f9a780a65a0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70005820 con 0x7f9a780a65a0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a7f995700 1 -- 172.21.2.117:0/576831391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a780a23c0 con 0x7f9a780a65a0 2021-02-10T11:54:33.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a7f995700 1 -- 172.21.2.117:0/576831391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a780a2950 con 0x7f9a780a65a0 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.401 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a70005cb0 con 0x7f9a780a65a0 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.402 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a7000bd70 con 0x7f9a780a65a0 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.402 7f9a69ffb700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 0x7f9a5801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.402 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9a70032a60 con 0x7f9a780a65a0 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.402 7f9a7d7b1700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 0x7f9a5801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:33.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.402 7f9a7d7b1700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 0x7f9a5801abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:33.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.432 7f9a7f995700 1 -- 172.21.2.117:0/576831391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a780ede60 con 0x7f9a780a65a0 2021-02-10T11:54:33.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.435 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a70029e30 con 0x7f9a780a65a0 2021-02-10T11:54:33.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.562 7f9a7f995700 1 -- 172.21.2.117:0/576831391 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f9a780ede60 con 0x7f9a780a65a0 2021-02-10T11:54:33.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.563 7f9a69ffb700 1 -- 172.21.2.117:0/576831391 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v160) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9a70029e30 con 0x7f9a780a65a0 2021-02-10T11:54:33.566 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:33.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.564 7f9a7f995700 1 -- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 msgr2=0x7f9a5801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:33.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.564 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 0x7f9a5801abc0 crc :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.564 7f9a7f995700 1 -- 172.21.2.117:0/576831391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 msgr2=0x7f9a780a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:33.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.564 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f9a7000b890 tx=0x7f9a7000bc30).stop 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 -- 172.21.2.117:0/576831391 shutdown_connections 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a580186c0 0x7f9a5801abc0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a780a65a0 0x7f9a780a1da0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a781a4860 0x7f9a780a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 --2- 172.21.2.117:0/576831391 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a7809e160 0x7f9a781a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 -- 172.21.2.117:0/576831391 >> 172.21.2.117:0/576831391 conn(0x7f9a7809e660 msgr2=0x7f9a780bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 -- 172.21.2.117:0/576831391 shutdown_connections 2021-02-10T11:54:33.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.565 7f9a7f995700 1 -- 172.21.2.117:0/576831391 wait complete. 2021-02-10T11:54:33.576 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:54:34.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 Processor -- start 2021-02-10T11:54:34.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 -- start start 2021-02-10T11:54:34.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28641b69a0 0x7f28641b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28640ede60 con 0x7f28640ef3f0 2021-02-10T11:54:34.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28640eba00 con 0x7f28640d18f0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.989 7f28692e7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28640d1cf0 con 0x7f28641b69a0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28641b69a0 0x7f28641b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28641b69a0 0x7f28641b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58142/0 (socket says 172.21.2.117:58142) 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 -- 172.21.2.117:0/3134302741 learned_addr learned my addr 172.21.2.117:0/3134302741 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 -- 172.21.2.117:0/3134302741 >> v1:172.21.2.125:6789/0 conn(0x7f28640d18f0 legacy=0x7f28640eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 -- 172.21.2.117:0/3134302741 >> v1:172.21.2.117:6789/0 conn(0x7f28640ef3f0 legacy=0x7f28640eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 -- 172.21.2.117:0/3134302741 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28641bb3f0 con 0x7f28641b69a0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286387e700 1 --2- 172.21.2.117:0/3134302741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28641b69a0 0x7f28641b8db0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f285c006f00 tx=0x7f285c009f10).ready entity=mon.0 client_cookie=af41e260dfe5d400 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286207b700 1 -- 172.21.2.117:0/3134302741 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285c00b3b0 con 0x7f28641b69a0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286207b700 1 -- 172.21.2.117:0/3134302741 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c00b630 con 0x7f28641b69a0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f286207b700 1 -- 172.21.2.117:0/3134302741 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285c00bb50 con 0x7f28641b69a0 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f28692e7700 1 -- 172.21.2.117:0/3134302741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28641b69a0 msgr2=0x7f28641b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.990 7f28692e7700 1 --2- 172.21.2.117:0/3134302741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28641b69a0 0x7f28641b8db0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7f285c006f00 tx=0x7f285c009f10).stop 2021-02-10T11:54:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.991 7f28692e7700 1 -- 172.21.2.117:0/3134302741 shutdown_connections 2021-02-10T11:54:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.991 7f28692e7700 1 --2- 172.21.2.117:0/3134302741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28641b69a0 0x7f28641b8db0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.991 7f28692e7700 1 -- 172.21.2.117:0/3134302741 >> 172.21.2.117:0/3134302741 conn(0x7f28640cf060 msgr2=0x7f28640cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.991 7f28692e7700 1 -- 172.21.2.117:0/3134302741 shutdown_connections 2021-02-10T11:54:34.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:33.991 7f28692e7700 1 -- 172.21.2.117:0/3134302741 wait complete. 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 Processor -- start 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 -- start start 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28640e94a0 0x7f28641c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28641c9880 0x7f28640a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28641b7740 con 0x7f286409e4e0 2021-02-10T11:54:34.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28640e98a0 con 0x7f28641c9880 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.006 7f28692e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28640e17d0 con 0x7f28640e94a0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58144/0 (socket says 172.21.2.117:58144) 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 -- 172.21.2.117:0/2583991527 learned_addr learned my addr 172.21.2.117:0/2583991527 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 -- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28640e94a0 msgr2=0x7f28641c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28640e94a0 0x7f28641c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 -- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28641c9880 msgr2=0x7f28640a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28641c9880 0x7f28640a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 -- 172.21.2.117:0/2583991527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28641c9c80 con 0x7f286409e4e0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f286307d700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f285400b530 tx=0x7f285400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f285400bcc0 con 0x7f286409e4e0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854006020 con 0x7f286409e4e0 2021-02-10T11:54:34.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f28692e7700 1 -- 172.21.2.117:0/2583991527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28641c9c80 con 0x7f286409e4e0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f28692e7700 1 -- 172.21.2.117:0/2583991527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28641c9c80 con 0x7f286409e4e0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.007 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28540054a0 con 0x7f286409e4e0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.008 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2854006200 con 0x7f286409e4e0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.008 7f285bfff700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 0x7f284801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.008 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2854032950 con 0x7f286409e4e0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.008 7f286287c700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 0x7f284801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.009 7f286287c700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 0x7f284801abc0 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.039 7f28692e7700 1 -- 172.21.2.117:0/2583991527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2864087390 con 0x7f286409e4e0 2021-02-10T11:54:34.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.042 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f285400bcc0 con 0x7f286409e4e0 2021-02-10T11:54:34.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.167 7f28692e7700 1 -- 172.21.2.117:0/2583991527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f28640d18f0 con 0x7f286409e4e0 2021-02-10T11:54:34.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.167 7f285bfff700 1 -- 172.21.2.117:0/2583991527 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v160) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2854006020 con 0x7f286409e4e0 2021-02-10T11:54:34.171 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 -- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 msgr2=0x7f284801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 0x7f284801abc0 crc :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 -- 172.21.2.117:0/2583991527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 msgr2=0x7f28640e8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f285400b530 tx=0x7f285400b8d0).stop 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 -- 172.21.2.117:0/2583991527 shutdown_connections 2021-02-10T11:54:34.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.169 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28641c9880 0x7f28640a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28640e94a0 0x7f28641c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2848018730 0x7f284801abc0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 --2- 172.21.2.117:0/2583991527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f286409e4e0 0x7f28640e8e80 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 -- 172.21.2.117:0/2583991527 >> 172.21.2.117:0/2583991527 conn(0x7f28640e71d0 msgr2=0x7f28640b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 -- 172.21.2.117:0/2583991527 shutdown_connections 2021-02-10T11:54:34.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.170 7f28692e7700 1 -- 172.21.2.117:0/2583991527 wait complete. 2021-02-10T11:54:34.181 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:54:34.181 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 Processor -- start 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 -- start start 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 0x7f2ed41b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40ede60 con 0x7f2ed40d18e0 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40eb9c0 con 0x7f2ed40ef3f0 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.590 7f2edb802700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40d1ce0 con 0x7f2ed41b6960 2021-02-10T11:54:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.591 7f2ed961e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2ed40d18e0 0x7f2ed40ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58402/0 (socket says 172.21.2.117:58402) 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.591 7f2ed961e700 1 -- 172.21.2.117:0/1178367652 learned_addr learned my addr 172.21.2.117:0/1178367652 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.591 7f2ed9e1f700 1 --2- 172.21.2.117:0/1178367652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 0x7f2ed41b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.591 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905001702 0 0) 0x7f2ec8001180 con 0x7f2ed40d18e0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.591 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec0002fe0 con 0x7f2ed40d18e0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282303785 0 0) 0x7f2ed0001080 con 0x7f2ed40ef3f0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec0003210 con 0x7f2ed40ef3f0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ed9e1f700 1 -- 172.21.2.117:0/1178367652 >> v1:172.21.2.125:6789/0 conn(0x7f2ed40ef3f0 legacy=0x7f2ed40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ed9e1f700 1 -- 172.21.2.117:0/1178367652 >> v1:172.21.2.117:6789/0 conn(0x7f2ed40d18e0 legacy=0x7f2ed40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ed9e1f700 1 -- 172.21.2.117:0/1178367652 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed41bb3b0 con 0x7f2ed41b6960 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ed9e1f700 1 --2- 172.21.2.117:0/1178367652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 0x7f2ed41b8d70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2ecc006e50 tx=0x7f2ecc009e70).ready entity=mon.2 client_cookie=ad6bc94bc92c9166 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ecc00b510 con 0x7f2ed41b6960 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc006790 con 0x7f2ed41b6960 2021-02-10T11:54:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.592 7f2ec7fff700 1 -- 172.21.2.117:0/1178367652 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ecc00b940 con 0x7f2ed41b6960 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 -- 172.21.2.117:0/1178367652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 msgr2=0x7f2ed41b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 --2- 172.21.2.117:0/1178367652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 0x7f2ed41b8d70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2ecc006e50 tx=0x7f2ecc009e70).stop 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 -- 172.21.2.117:0/1178367652 shutdown_connections 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 --2- 172.21.2.117:0/1178367652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed41b6960 0x7f2ed41b8d70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 -- 172.21.2.117:0/1178367652 >> 172.21.2.117:0/1178367652 conn(0x7f2ed40ced60 msgr2=0x7f2ed40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.593 7f2edb802700 1 -- 172.21.2.117:0/1178367652 shutdown_connections 2021-02-10T11:54:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.594 7f2edb802700 1 -- 172.21.2.117:0/1178367652 wait complete. 2021-02-10T11:54:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 Processor -- start 2021-02-10T11:54:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 -- start start 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed40e3ad0 0x7f2ed40b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ed40e3ed0 0x7f2ed40e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40e44e0 con 0x7f2ed40b2e30 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40e56a0 con 0x7f2ed40e3ed0 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2edb802700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40e5880 con 0x7f2ed40e3ad0 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2ed8e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2ed8e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58162/0 (socket says 172.21.2.117:58162) 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.608 7f2ed8e1d700 1 -- 172.21.2.117:0/1188024240 learned_addr learned my addr 172.21.2.117:0/1188024240 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 -- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed40e3ad0 msgr2=0x7f2ed40b2810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed40e3ad0 0x7f2ed40b2810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 -- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ed40e3ed0 msgr2=0x7f2ed40e5080 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ed40e3ed0 0x7f2ed40e5080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 -- 172.21.2.117:0/1188024240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40e5aa0 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ed8e1d700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f2ed000b5b0 tx=0x7f2ed000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ed0006240 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed00054a0 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2edb802700 1 -- 172.21.2.117:0/1188024240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed40e5aa0 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2edb802700 1 -- 172.21.2.117:0/1188024240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed40e5aa0 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.609 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2ed0006240 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.610 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2ed0006240 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.610 7f2ec5ffb700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 0x7f2eb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.610 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2ed0032930 con 0x7f2ed40b2e30 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.610 7f2ed961e700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 0x7f2eb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.611 7f2ed961e700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 0x7f2eb401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:34.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.641 7f2edb802700 1 -- 172.21.2.117:0/1188024240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed4087290 con 0x7f2ed40b2e30 2021-02-10T11:54:34.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.643 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2ed0006030 con 0x7f2ed40b2e30 2021-02-10T11:54:34.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.768 7f2edb802700 1 -- 172.21.2.117:0/1188024240 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed4087290 con 0x7f2ed40b2e30 2021-02-10T11:54:34.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.769 7f2ec5ffb700 1 -- 172.21.2.117:0/1188024240 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+2099 (secure 0 0 0) 0x7f2ed000bd40 con 0x7f2ed40b2e30 2021-02-10T11:54:34.772 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 136 2021-02-10T11:54:34.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:34.772 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:54:26.825269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7697":{"gid":7697,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/95294604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":95294604},{"type":"v1","addr":"172.21.2.125:6835","nonce":95294604}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:54:34.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 msgr2=0x7f2eb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 0x7f2eb401abc0 crc :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 msgr2=0x7f2ed40e4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:34.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f2ed000b5b0 tx=0x7f2ed000b950).stop 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 shutdown_connections 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2eb4018730 0x7f2eb401abc0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ed40b2e30 0x7f2ed40e4a60 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ed40e3ed0 0x7f2ed40e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 --2- 172.21.2.117:0/1188024240 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ed40e3ad0 0x7f2ed40b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 >> 172.21.2.117:0/1188024240 conn(0x7f2ed41a3620 msgr2=0x7f2ed408d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:34.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 shutdown_connections 2021-02-10T11:54:34.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:34.771 7f2edb802700 1 -- 172.21.2.117:0/1188024240 wait complete. 2021-02-10T11:54:34.987 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 136, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:54:24.817179', 'modified': '2021-02-10 11:54:26.825269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7697': {'gid': 7697, 'name': 'b', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/95294604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 95294604}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 95294604}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:54:34.987 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:54:34.987 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7697, 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.565820929999999, 2021-02-10T11:54:35.387 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 136, 2021-02-10T11:54:35.388 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 160, 2021-02-10T11:54:35.388 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 160, 2021-02-10T11:54:35.388 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.201304986 2021-02-10T11:54:35.388 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:54:35.391 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7697, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.565820929999999, "mdsmap_epoch": 136, "osdmap_epoch": 160, "osdmap_epoch_barrier": 160, "uptime": 16.201304986 } 2021-02-10T11:54:35.391 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:54:35.391 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:54:35.391 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:54:35.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:35.391 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:35.726 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:54:36.071 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:54:36.071 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:54:36.416 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:54:36.416 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:54:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:54:36.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:36.417 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:54:36.753 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:54:36.753 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:36.756 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:54:37.101 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:54:37.102 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:54:37.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[180463]: starting ceph client 2021-02-10T11:54:37.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:54:37.166 7fc6ea6eff80 -1 init, newargv = 0x559538468f70 newargc=9 2021-02-10T11:54:37.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[180463]: starting fuse 2021-02-10T11:54:37.444 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:54:37.444 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:54:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:54:37.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:37.445 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:54:37.780 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:54:37.780 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:54:37.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:37.780 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:54:37.780 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:37.781 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:54:37.782 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:54:38.144 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.180463.asok 2021-02-10T11:54:38.147 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.180463.asok 2021-02-10T11:54:38.147 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:38.147 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.180463.asok status 2021-02-10T11:54:38.538 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:54:38.538 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:54:38.538 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:54:38.538 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:54:38.538 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "180463", 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12543, 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12543 2021-02-10T11:54:38.539 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3907464943 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3907464943 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:38.540 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12543 172.21.2.117:0/3907464943", 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3907464943", 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 136, 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 160, 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:54:38.541 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:54:38.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:38.543 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:38.543 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:38.879 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:38.879 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:54:38.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:38.880 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:39.227 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.636 7fef1cfee700 1 Processor -- start 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 -- start start 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180ef3f0 0x7fef180eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef180eb7f0 0x7fef181b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180ede60 con 0x7fef180d18e0 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef180ef7f0 con 0x7fef180eb7f0 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1cfee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef180d1ce0 con 0x7fef180ef3f0 2021-02-10T11:54:39.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1759e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef180eb7f0 0x7fef181b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1759e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fef180eb7f0 0x7fef181b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58182/0 (socket says 172.21.2.117:58182) 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.637 7fef1759e700 1 -- 172.21.2.117:0/1895395141 learned_addr learned my addr 172.21.2.117:0/1895395141 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1759e700 1 -- 172.21.2.117:0/1895395141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180ef3f0 msgr2=0x7fef180eb1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1759e700 1 --2- 172.21.2.117:0/1895395141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180ef3f0 0x7fef180eb1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1759e700 1 -- 172.21.2.117:0/1895395141 >> v1:172.21.2.125:6789/0 conn(0x7fef180d18e0 legacy=0x7fef180ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1759e700 1 -- 172.21.2.117:0/1895395141 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef181b8f90 con 0x7fef180eb7f0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1759e700 1 --2- 172.21.2.117:0/1895395141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef180eb7f0 0x7fef181b7960 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7fef10006f00 tx=0x7fef10009f10).ready entity=mon.0 client_cookie=4fb2cc568840acda server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef15d9b700 1 -- 172.21.2.117:0/1895395141 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1000b3b0 con 0x7fef180eb7f0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef15d9b700 1 -- 172.21.2.117:0/1895395141 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1000b630 con 0x7fef180eb7f0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef15d9b700 1 -- 172.21.2.117:0/1895395141 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef1000bb50 con 0x7fef180eb7f0 2021-02-10T11:54:39.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1cfee700 1 -- 172.21.2.117:0/1895395141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef180eb7f0 msgr2=0x7fef181b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.638 7fef1cfee700 1 --2- 172.21.2.117:0/1895395141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef180eb7f0 0x7fef181b7960 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 rx=0x7fef10006f00 tx=0x7fef10009f10).stop 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 -- 172.21.2.117:0/1895395141 shutdown_connections 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 --2- 172.21.2.117:0/1895395141 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef180eb7f0 0x7fef181b7960 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 --2- 172.21.2.117:0/1895395141 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180ef3f0 0x7fef180eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 -- 172.21.2.117:0/1895395141 >> 172.21.2.117:0/1895395141 conn(0x7fef180ced60 msgr2=0x7fef180cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 -- 172.21.2.117:0/1895395141 shutdown_connections 2021-02-10T11:54:39.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.639 7fef1cfee700 1 -- 172.21.2.117:0/1895395141 wait complete. 2021-02-10T11:54:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 Processor -- start 2021-02-10T11:54:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 -- start start 2021-02-10T11:54:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180e9560 0x7fef180e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef180e9960 0x7fef180e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef18196810 con 0x7fef18196390 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef18098ad0 con 0x7fef180e9960 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef1cfee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef180e8a40 con 0x7fef180e9560 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef16d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef16d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58184/0 (socket says 172.21.2.117:58184) 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.654 7fef16d9d700 1 -- 172.21.2.117:0/4156217241 learned_addr learned my addr 172.21.2.117:0/4156217241 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 -- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180e9560 msgr2=0x7fef180e7e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180e9560 0x7fef180e7e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 -- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef180e9960 msgr2=0x7fef180e8420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef180e9960 0x7fef180e8420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 -- 172.21.2.117:0/4156217241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180e8c60 con 0x7fef18196390 2021-02-10T11:54:39.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef16d9d700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7fef0800b5b0 tx=0x7fef0800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef0800bd40 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef080009b0 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef180e8c60 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fef080062a0 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.655 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1819cd70 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.656 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fef08006020 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.656 7fef0f7fe700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 0x7feefc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.656 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fef08032730 con 0x7fef18196390 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.656 7fef1659c700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 0x7feefc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:39.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.657 7fef1659c700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 0x7feefc01abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:39.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.687 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef18087210 con 0x7fef18196390 2021-02-10T11:54:39.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.690 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fef08006230 con 0x7fef18196390 2021-02-10T11:54:39.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.813 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef180ede60 con 0x7fef18196390 2021-02-10T11:54:39.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.814 7fef0f7fe700 1 -- 172.21.2.117:0/4156217241 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v160) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fef08006000 con 0x7fef18196390 2021-02-10T11:54:39.817 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:39.817 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":160,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:54:25.819611","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:54:22.787648","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":35,"pool_name":"cephfs_data","create_time":"2021-02-10 11:54:23.861446","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"160","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 msgr2=0x7feefc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 0x7feefc01abc0 crc :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 msgr2=0x7fef180e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7fef0800b5b0 tx=0x7fef0800b950).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 shutdown_connections 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fef180e9960 0x7fef180e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fef180e9560 0x7fef180e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7feefc0186c0 0x7feefc01abc0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 --2- 172.21.2.117:0/4156217241 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fef18196390 0x7fef180e8f40 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 >> 172.21.2.117:0/4156217241 conn(0x7fef1819f610 msgr2=0x7fef181a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 shutdown_connections 2021-02-10T11:54:39.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:39.816 7fef1cfee700 1 -- 172.21.2.117:0/4156217241 wait complete. 2021-02-10T11:54:39.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:54:22.787648', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '159', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:54:23.861446', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '160', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:54:39.827 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:54:40.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 Processor -- start 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 -- start start 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc5ac0d18f0 0x7fc5ac0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0eb040 0x7fc5ac0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ac0eba60 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ac0d1cf0 con 0x7fc5ac0d18f0 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.240 7fc5b2d35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac0eefd0 con 0x7fc5ac0eb040 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5b1352700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc5ac0ee3e0 0x7fc5ac0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58442/0 (socket says 172.21.2.117:58442) 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5b1352700 1 -- 172.21.2.117:0/2525673446 learned_addr learned my addr 172.21.2.117:0/2525673446 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222355822 0 0) 0x7fc5a4001180 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598002fe0 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.261 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5abfff700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0eb040 0x7fc5ac0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5b0b51700 1 --2- 172.21.2.117:0/2525673446 >> v2:172.21.2.125:3300/0 conn(0x7fc5ac0d18f0 0x7fc5ac0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2471205337 0 0) 0x7fc5a4001180 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0eb040 msgr2=0x7fc5ac0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0eb040 0x7fc5ac0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 >> v2:172.21.2.125:3300/0 conn(0x7fc5ac0d18f0 msgr2=0x7fc5ac0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 --2- 172.21.2.117:0/2525673446 >> v2:172.21.2.125:3300/0 conn(0x7fc5ac0d18f0 0x7fc5ac0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac0ef1f0 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.241 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fc5a4001680 con 0x7fc5ac0ee3e0 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 >> v1:172.21.2.117:6789/0 conn(0x7fc5ac0ee3e0 legacy=0x7fc5ac0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc598003a60 0x7fc598005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc59804a7a0 0x7fc59804cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.262 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc59804d510 0x7fc59804fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5980502e0 con 0x7fc59804d510 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc598050500 con 0x7fc598003a60 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc598050720 con 0x7fc59804a7a0 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc598003a60 0x7fc598005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b1352700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc59804d510 0x7fc59804fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5abfff700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc59804a7a0 0x7fc59804cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 -- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc59804a7a0 msgr2=0x7fc59804cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc59804a7a0 0x7fc59804cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 -- 172.21.2.117:0/2525673446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc59804d510 msgr2=0x7fc59804fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc59804d510 0x7fc59804fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 -- 172.21.2.117:0/2525673446 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980509c0 con 0x7fc598003a60 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5b0b51700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc598003a60 0x7fc598005ef0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fc59400c8f0 tx=0x7fc59400cc90).ready entity=mon.1 client_cookie=bced99d33268809f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400cdd0 con 0x7fc598003a60 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.242 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594009310 con 0x7fc598003a60 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 -- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc598003a60 msgr2=0x7fc598005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5ab7fe700 1 -- 172.21.2.117:0/2525673446 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400cdd0 con 0x7fc598003a60 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 --2- 172.21.2.117:0/2525673446 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc598003a60 0x7fc598005ef0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fc59400c8f0 tx=0x7fc59400cc90).stop 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b1352700 1 -- 172.21.2.117:0/2525673446 reap_dead start 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 -- 172.21.2.117:0/2525673446 shutdown_connections 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 -- 172.21.2.117:0/2525673446 >> 172.21.2.117:0/2525673446 conn(0x7fc5ac0cf060 msgr2=0x7fc5ac0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 -- 172.21.2.117:0/2525673446 shutdown_connections 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.243 7fc5b2d35700 1 -- 172.21.2.117:0/2525673446 wait complete. 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 Processor -- start 2021-02-10T11:54:40.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 -- start start 2021-02-10T11:54:40.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5ac08cf40 0x7fc5ac0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0a9f90 0x7fc5ac0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac0aae30 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac0aa9f0 con 0x7fc5ac08cf40 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b2d35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac0ab330 con 0x7fc5ac0a9f90 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.258 7fc5b0b51700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58202/0 (socket says 172.21.2.117:58202) 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 -- 172.21.2.117:0/3463056427 learned_addr learned my addr 172.21.2.117:0/3463056427 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 -- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0a9f90 msgr2=0x7fc5ac0aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0a9f90 0x7fc5ac0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 -- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5ac08cf40 msgr2=0x7fc5ac0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5ac08cf40 0x7fc5ac0a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 -- 172.21.2.117:0/3463056427 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ac0ab550 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b0b51700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7fc594009e00 tx=0x7fc59400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400c7c0 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59400cb40 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.259 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ac0ab550 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.260 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc59400b6f0 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.260 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ac0ab550 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.260 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc59400c7c0 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.260 7fc5a97fa700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 0x7fc59001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.260 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc5940326b0 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.261 7fc5abfff700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 0x7fc59001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.261 7fc5abfff700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 0x7fc59001abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.290 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ac0ede60 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.293 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc594000c90 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.422 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc5ac0ede60 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.423 7fc5a97fa700 1 -- 172.21.2.117:0/3463056427 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v160) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc594000f20 con 0x7fc5ac1a48b0 2021-02-10T11:54:40.426 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:40.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 msgr2=0x7fc59001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 0x7fc59001abc0 crc :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 msgr2=0x7fc5ac08c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 rx=0x7fc594009e00 tx=0x7fc59400b300).stop 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 shutdown_connections 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc5ac0a9f90 0x7fc5ac0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc5ac08cf40 0x7fc5ac0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc5900186c0 0x7fc59001abc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 --2- 172.21.2.117:0/3463056427 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc5ac1a48b0 0x7fc5ac08c920 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 >> 172.21.2.117:0/3463056427 conn(0x7fc5ac092d00 msgr2=0x7fc5ac08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:40.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 shutdown_connections 2021-02-10T11:54:40.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.425 7fc5b2d35700 1 -- 172.21.2.117:0/3463056427 wait complete. 2021-02-10T11:54:40.436 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:54:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.849 7f06d4868700 1 Processor -- start 2021-02-10T11:54:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.849 7f06d4868700 1 -- start start 2021-02-10T11:54:40.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.850 7f06d4868700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 0x7f06d00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.850 7f06d4868700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 0x7f06d01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.850 7f06d4868700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d00ef7f0 con 0x7f06d00eb040 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.850 7f06d4868700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00eded0 con 0x7f06d00d18f0 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.850 7f06d4868700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00d1cf0 con 0x7f06d00ef3f0 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cde1b700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f06d00eb040 0x7f06d00eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51386/0 (socket says 172.21.2.117:51386) 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cde1b700 1 -- 172.21.2.117:0/3174011368 learned_addr learned my addr 172.21.2.117:0/3174011368 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06ce61c700 1 --2- 172.21.2.117:0/3174011368 >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 0x7f06d00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 --2- 172.21.2.117:0/3174011368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 0x7f06d01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 -- 172.21.2.117:0/3174011368 >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 msgr2=0x7f06d00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 --2- 172.21.2.117:0/3174011368 >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 0x7f06d00eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 -- 172.21.2.117:0/3174011368 >> v1:172.21.2.125:6789/0 conn(0x7f06d00eb040 legacy=0x7f06d00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 -- 172.21.2.117:0/3174011368 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d01b8fd0 con 0x7f06d00ef3f0 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06ce61c700 1 --2- 172.21.2.117:0/3174011368 >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 0x7f06d00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.851 7f06cee1d700 1 --2- 172.21.2.117:0/3174011368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 0x7f06d01b79a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f06c8006e50 tx=0x7f06c8009e70).ready entity=mon.2 client_cookie=83b612fcbe5b0522 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06cd61a700 1 -- 172.21.2.117:0/3174011368 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06c80066f0 con 0x7f06d00ef3f0 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06cd61a700 1 -- 172.21.2.117:0/3174011368 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c8003030 con 0x7f06d00ef3f0 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06cd61a700 1 -- 172.21.2.117:0/3174011368 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06c800b700 con 0x7f06d00ef3f0 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 -- 172.21.2.117:0/3174011368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 msgr2=0x7f06d01b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 --2- 172.21.2.117:0/3174011368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 0x7f06d01b79a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f06c8006e50 tx=0x7f06c8009e70).stop 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 -- 172.21.2.117:0/3174011368 shutdown_connections 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 --2- 172.21.2.117:0/3174011368 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d00ef3f0 0x7f06d01b79a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 --2- 172.21.2.117:0/3174011368 >> v2:172.21.2.125:3300/0 conn(0x7f06d00d18f0 0x7f06d00eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:40.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.852 7f06d4868700 1 -- 172.21.2.117:0/3174011368 >> 172.21.2.117:0/3174011368 conn(0x7f06d00cf060 msgr2=0x7f06d00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:40.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.853 7f06d4868700 1 -- 172.21.2.117:0/3174011368 shutdown_connections 2021-02-10T11:54:40.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.853 7f06d4868700 1 -- 172.21.2.117:0/3174011368 wait complete. 2021-02-10T11:54:40.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 Processor -- start 2021-02-10T11:54:40.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 -- start start 2021-02-10T11:54:40.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d007c7c0 0x7f06d01a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06d01a3930 0x7f06d00a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d007cca0 con 0x7f06d00a13a0 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d018fb70 con 0x7f06d01a3930 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.867 7f06d4868700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00dc6d0 con 0x7f06d007c7c0 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58216/0 (socket says 172.21.2.117:58216) 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 -- 172.21.2.117:0/2365500938 learned_addr learned my addr 172.21.2.117:0/2365500938 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 -- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d007c7c0 msgr2=0x7f06d01a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d007c7c0 0x7f06d01a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 -- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06d01a3930 msgr2=0x7f06d00a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06d01a3930 0x7f06d00a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 -- 172.21.2.117:0/2365500938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d007d220 con 0x7f06d00a13a0 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06cee1d700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f06c8009390 tx=0x7f06c800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06c80093d0 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c800c2f0 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f06c800c7d0 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06d4868700 1 -- 172.21.2.117:0/2365500938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d007d220 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.868 7f06d4868700 1 -- 172.21.2.117:0/2365500938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d007d7b0 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.869 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f06c80056b0 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.869 7f06c6ffd700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 0x7f06b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.869 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f06c8032a80 con 0x7f06d00a13a0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.869 7f06ce61c700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 0x7f06b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:40.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.870 7f06ce61c700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 0x7f06b401abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:40.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.900 7f06d4868700 1 -- 172.21.2.117:0/2365500938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d00daff0 con 0x7f06d00a13a0 2021-02-10T11:54:40.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:40.902 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f06c80093d0 con 0x7f06d00a13a0 2021-02-10T11:54:41.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.029 7f06d4868700 1 -- 172.21.2.117:0/2365500938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f06d00daff0 con 0x7f06d00a13a0 2021-02-10T11:54:41.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.030 7f06c6ffd700 1 -- 172.21.2.117:0/2365500938 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v160) v1 ==== 104+0+10 (secure 0 0 0) 0x7f06c80056b0 con 0x7f06d00a13a0 2021-02-10T11:54:41.033 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 msgr2=0x7f06b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 0x7f06b401abc0 crc :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 msgr2=0x7f06d00a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f06c8009390 tx=0x7f06c800bd40).stop 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 shutdown_connections 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f06b40186c0 0x7f06b401abc0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f06d00a13a0 0x7f06d00a17a0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f06d01a3930 0x7f06d00a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:41.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 --2- 172.21.2.117:0/2365500938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f06d007c7c0 0x7f06d01a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:41.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 >> 172.21.2.117:0/2365500938 conn(0x7f06d0087090 msgr2=0x7f06d008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:41.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 shutdown_connections 2021-02-10T11:54:41.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.032 7f06d4868700 1 -- 172.21.2.117:0/2365500938 wait complete. 2021-02-10T11:54:41.043 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.455 7fbd7b3e7700 1 Processor -- start 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.455 7fbd7b3e7700 1 -- start start 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd7b3e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbd740d18f0 0x7fbd740eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd7b3e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740ef3f0 0x7fbd741b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd7b3e7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd740ef7f0 con 0x7fbd740eb040 2021-02-10T11:54:41.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd7b3e7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd740eded0 con 0x7fbd740d18f0 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd7b3e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740d1cf0 con 0x7fbd740ef3f0 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd79203700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbd740d18f0 0x7fbd740eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd78a02700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbd740eb040 0x7fbd740eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58474/0 (socket says 172.21.2.117:58474) 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd78a02700 1 -- 172.21.2.117:0/3711971163 learned_addr learned my addr 172.21.2.117:0/3711971163 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd79a04700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740ef3f0 0x7fbd741b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd67fff700 1 -- 172.21.2.117:0/3711971163 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960797057 0 0) 0x7fbd680010b0 con 0x7fbd740eb040 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.456 7fbd67fff700 1 -- 172.21.2.117:0/3711971163 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd60002fe0 con 0x7fbd740eb040 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd79203700 1 -- 172.21.2.117:0/3711971163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740ef3f0 msgr2=0x7fbd741b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd79203700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740ef3f0 0x7fbd741b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd79203700 1 -- 172.21.2.117:0/3711971163 >> v1:172.21.2.117:6789/0 conn(0x7fbd740eb040 legacy=0x7fbd740eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd79203700 1 -- 172.21.2.117:0/3711971163 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd741b8fd0 con 0x7fbd740d18f0 2021-02-10T11:54:41.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd79203700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd740d18f0 0x7fbd740eaa20 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fbd70006650 tx=0x7fbd70009a80).ready entity=mon.0 client_cookie=2854c1f907c329a5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd67fff700 1 -- 172.21.2.117:0/3711971163 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd700009b0 con 0x7fbd740d18f0 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd67fff700 1 -- 172.21.2.117:0/3711971163 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7000c0c0 con 0x7fbd740d18f0 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd67fff700 1 -- 172.21.2.117:0/3711971163 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd7000c5e0 con 0x7fbd740d18f0 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd7b3e7700 1 -- 172.21.2.117:0/3711971163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd740d18f0 msgr2=0x7fbd740eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.457 7fbd7b3e7700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd740d18f0 0x7fbd740eaa20 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fbd70006650 tx=0x7fbd70009a80).stop 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 -- 172.21.2.117:0/3711971163 shutdown_connections 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740ef3f0 0x7fbd741b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 --2- 172.21.2.117:0/3711971163 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd740d18f0 0x7fbd740eaa20 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 -- 172.21.2.117:0/3711971163 >> 172.21.2.117:0/3711971163 conn(0x7fbd740cf060 msgr2=0x7fbd740cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 -- 172.21.2.117:0/3711971163 shutdown_connections 2021-02-10T11:54:41.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.458 7fbd7b3e7700 1 -- 172.21.2.117:0/3711971163 wait complete. 2021-02-10T11:54:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 Processor -- start 2021-02-10T11:54:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 -- start start 2021-02-10T11:54:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd7407c7c0 0x7fbd741a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740a13a0 0x7fbd740a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7407cca0 con 0x7fbd741a3930 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7418fb70 con 0x7fbd7407c7c0 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd7b3e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd740dc6d0 con 0x7fbd740a13a0 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd78a02700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd78a02700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58230/0 (socket says 172.21.2.117:58230) 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.473 7fbd78a02700 1 -- 172.21.2.117:0/1576420383 learned_addr learned my addr 172.21.2.117:0/1576420383 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd79a04700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740a13a0 0x7fbd740a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd79203700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd7407c7c0 0x7fbd741a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 -- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740a13a0 msgr2=0x7fbd740a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740a13a0 0x7fbd740a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 -- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd7407c7c0 msgr2=0x7fbd741a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.480 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd7407c7c0 0x7fbd741a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 -- 172.21.2.117:0/1576420383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7407d220 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd78a02700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fbd68001c60 tx=0x7fbd6800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd68000e00 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6800acc0 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbd680045d0 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7407d220 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.474 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7407d7b0 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.475 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbd68004140 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.475 7fbd65ffb700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 0x7fbd5401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.475 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbd68032520 con 0x7fbd741a3930 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.475 7fbd79203700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 0x7fbd5401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:41.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.476 7fbd79203700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 0x7fbd5401abc0 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.506 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd740daff0 con 0x7fbd741a3930 2021-02-10T11:54:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.508 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbd68000e00 con 0x7fbd741a3930 2021-02-10T11:54:41.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.634 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fbd740ede60 con 0x7fbd741a3930 2021-02-10T11:54:41.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.634 7fbd65ffb700 1 -- 172.21.2.117:0/1576420383 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v160) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbd68004140 con 0x7fbd741a3930 2021-02-10T11:54:41.638 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:54:41.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.636 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 msgr2=0x7fbd5401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.636 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 0x7fbd5401abc0 crc :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.636 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 msgr2=0x7fbd740a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:41.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.636 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7fbd68001c60 tx=0x7fbd6800a8d0).stop 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 shutdown_connections 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd540186c0 0x7fbd5401abc0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbd741a3930 0x7fbd740a0d80 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbd740a13a0 0x7fbd740a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 --2- 172.21.2.117:0/1576420383 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbd7407c7c0 0x7fbd741a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 >> 172.21.2.117:0/1576420383 conn(0x7fbd74087090 msgr2=0x7fbd7408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 shutdown_connections 2021-02-10T11:54:41.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:41.637 7fbd7b3e7700 1 -- 172.21.2.117:0/1576420383 wait complete. 2021-02-10T11:54:41.648 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:54:41.648 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:54:42.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 Processor -- start 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 -- start start 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fda880d18e0 0x7fda880ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 0x7fda881b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda880ede60 con 0x7fda880eb000 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda880ef7f0 con 0x7fda880d18e0 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.058 7fda9051b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880d1ce0 con 0x7fda880ef3f0 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fda880d18e0 0x7fda880ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8eb38700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 0x7fda881b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8db36700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fda880eb000 0x7fda880eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51414/0 (socket says 172.21.2.117:51414) 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8db36700 1 -- 172.21.2.117:0/1116000181 learned_addr learned my addr 172.21.2.117:0/1116000181 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:42.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 -- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 msgr2=0x7fda881b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 0x7fda881b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 -- 172.21.2.117:0/1116000181 >> v1:172.21.2.125:6789/0 conn(0x7fda880eb000 legacy=0x7fda880eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 -- 172.21.2.117:0/1116000181 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda881b8f90 con 0x7fda880d18e0 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8eb38700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 0x7fda881b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.060 7fda8e337700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda880d18e0 0x7fda880ea9e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fda84006650 tx=0x7fda84009a80).ready entity=mon.1 client_cookie=4dbcdc571d598c7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.061 7fda8d335700 1 -- 172.21.2.117:0/1116000181 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda840009b0 con 0x7fda880d18e0 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.061 7fda8d335700 1 -- 172.21.2.117:0/1116000181 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8400c0c0 con 0x7fda880d18e0 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.061 7fda8d335700 1 -- 172.21.2.117:0/1116000181 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda8400c5e0 con 0x7fda880d18e0 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.061 7fda9051b700 1 -- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda880d18e0 msgr2=0x7fda880ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.061 7fda9051b700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda880d18e0 0x7fda880ea9e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fda84006650 tx=0x7fda84009a80).stop 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 -- 172.21.2.117:0/1116000181 shutdown_connections 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880ef3f0 0x7fda881b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 --2- 172.21.2.117:0/1116000181 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda880d18e0 0x7fda880ea9e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 -- 172.21.2.117:0/1116000181 >> 172.21.2.117:0/1116000181 conn(0x7fda880ced60 msgr2=0x7fda880cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:42.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 -- 172.21.2.117:0/1116000181 shutdown_connections 2021-02-10T11:54:42.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.062 7fda9051b700 1 -- 172.21.2.117:0/1116000181 wait complete. 2021-02-10T11:54:42.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 Processor -- start 2021-02-10T11:54:42.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 -- start start 2021-02-10T11:54:42.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda881afd30 0x7fda880b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880e80e0 con 0x7fda881b0130 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880e7e20 con 0x7fda881afd30 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.076 7fda9051b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda88096a70 con 0x7fda880b43f0 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8eb38700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:55922/0 (socket says 172.21.2.117:55922) 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 -- 172.21.2.117:0/1262603753 learned_addr learned my addr 172.21.2.117:0/1262603753 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8eb38700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58246/0 (socket says 172.21.2.117:58246) 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8e337700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda881afd30 0x7fda880b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 -- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda881afd30 msgr2=0x7fda880b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda881afd30 0x7fda880b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 -- 172.21.2.117:0/1262603753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 msgr2=0x7fda880e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 -- 172.21.2.117:0/1262603753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda880e8660 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8eb38700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda8db36700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fda7c001c60 tx=0x7fda7c00a8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda7c0010b0 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c004350 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda9051b700 1 -- 172.21.2.117:0/1262603753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda880e8660 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.077 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fda7c004870 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.078 7fda9051b700 1 -- 172.21.2.117:0/1262603753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda880e8660 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.078 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fda7c001090 con 0x7fda880b43f0 2021-02-10T11:54:42.084 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.078 7fda7affd700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 0x7fda7001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:42.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.078 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fda7c0326a0 con 0x7fda880b43f0 2021-02-10T11:54:42.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.078 7fda8e337700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 0x7fda7001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:42.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.079 7fda8e337700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 0x7fda7001abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:42.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.108 7fda9051b700 1 -- 172.21.2.117:0/1262603753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda880e7e20 con 0x7fda880b43f0 2021-02-10T11:54:42.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.111 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fda7c004140 con 0x7fda880b43f0 2021-02-10T11:54:42.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.237 7fda9051b700 1 -- 172.21.2.117:0/1262603753 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda880e7e20 con 0x7fda880b43f0 2021-02-10T11:54:42.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.238 7fda7affd700 1 -- 172.21.2.117:0/1262603753 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+2099 (secure 0 0 0) 0x7fda7c001090 con 0x7fda880b43f0 2021-02-10T11:54:42.241 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 136 2021-02-10T11:54:42.241 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:54:42.241 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":136,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:54:26.825269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7697":{"gid":7697,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/95294604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":95294604},{"type":"v1","addr":"172.21.2.125:6835","nonce":95294604}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 msgr2=0x7fda7001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 0x7fda7001abc0 crc :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 msgr2=0x7fda880e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fda7c001c60 tx=0x7fda7c00a8d0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 shutdown_connections 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fda700186c0 0x7fda7001abc0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fda881b0130 0x7fda880e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fda880b43f0 0x7fda880e71e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 --2- 172.21.2.117:0/1262603753 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fda881afd30 0x7fda880b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 >> 172.21.2.117:0/1262603753 conn(0x7fda881a3620 msgr2=0x7fda880dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:42.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 shutdown_connections 2021-02-10T11:54:42.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:42.240 7fda9051b700 1 -- 172.21.2.117:0/1262603753 wait complete. 2021-02-10T11:54:42.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 136, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:54:24.817179', 'modified': '2021-02-10 11:54:26.825269', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7697}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7697': {'gid': 7697, 'name': 'b', 'rank': 0, 'incarnation': 133, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/95294604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 95294604}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 95294604}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:54:42.452 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:54:42.453 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7697, 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.030395293000002, 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 136, 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 160, 2021-02-10T11:54:42.852 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 160, 2021-02-10T11:54:42.853 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.665877174999999 2021-02-10T11:54:42.853 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:54:42.856 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7697, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.030395293000002, "mdsmap_epoch": 136, "osdmap_epoch": 160, "osdmap_epoch_barrier": 160, "uptime": 23.665877174999999 } 2021-02-10T11:54:42.856 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:54:42.856 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:54:42.856 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:54:42.856 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:42.857 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:43.192 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:54:43.538 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:54:43.538 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:54:43.881 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:54:43.881 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:54:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:54:43.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:43.882 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:54:44.217 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:54:44.217 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:54:44.217 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:44.220 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:54:44.563 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:54:44.564 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:54:44.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[181104]: starting ceph client 2021-02-10T11:54:44.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:54:44.628 7f3220211f80 -1 init, newargv = 0x55870c50b960 newargc=9 2021-02-10T11:54:44.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[181104]: starting fuse 2021-02-10T11:54:44.906 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:54:44.907 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:54:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:54:44.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:44.908 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:54:45.243 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:54:45.244 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:54:45.244 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:54:45.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:45.244 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:54:45.245 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:54:45.246 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:54:45.247 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:54:45.247 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:54:45.247 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:54:45.609 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.181104.asok 2021-02-10T11:54:45.612 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.181104.asok 2021-02-10T11:54:45.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:45.612 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.181104.asok status 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "181104", 2021-02-10T11:54:46.003 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12573, 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12573 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3074040925 2021-02-10T11:54:46.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3074040925 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12573 172.21.2.117:0/3074040925", 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3074040925", 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 136, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 160, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:54:46.005 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:54:46.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:46.007 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:46.007 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:46.343 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:46.343 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:54:46.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:46.344 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:46.691 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:54:46.997 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:54:47.000 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:54:47.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 Processor -- start 2021-02-10T11:54:47.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 -- start start 2021-02-10T11:54:47.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f48340e4e40 0x7f48341b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4834085450 0x7f4834086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4834085930 con 0x7f48340e5240 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48340e5cc0 con 0x7f4834085450 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.105 7f48390dd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4834084840 con 0x7f48340e4e40 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4834085450 0x7f4834086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4834085450 0x7f4834086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58266/0 (socket says 172.21.2.117:58266) 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 -- 172.21.2.117:0/4254132334 learned_addr learned my addr 172.21.2.117:0/4254132334 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f4832d9d700 1 --2- 172.21.2.117:0/4254132334 >> v2:172.21.2.125:3300/0 conn(0x7f48340e4e40 0x7f48341b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 -- 172.21.2.117:0/4254132334 >> v2:172.21.2.125:3300/0 conn(0x7f48340e4e40 msgr2=0x7f48341b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 --2- 172.21.2.117:0/4254132334 >> v2:172.21.2.125:3300/0 conn(0x7f48340e4e40 0x7f48341b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 -- 172.21.2.117:0/4254132334 >> v1:172.21.2.117:6789/0 conn(0x7f48340e5240 legacy=0x7f4834084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 -- 172.21.2.117:0/4254132334 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48341b0810 con 0x7f4834085450 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483359e700 1 --2- 172.21.2.117:0/4254132334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4834085450 0x7f4834086ec0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f482c006f00 tx=0x7f482c009f10).ready entity=mon.0 client_cookie=e30178745f13e884 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483259c700 1 -- 172.21.2.117:0/4254132334 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f482c00b3b0 con 0x7f4834085450 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.106 7f483259c700 1 -- 172.21.2.117:0/4254132334 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c00b630 con 0x7f4834085450 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f483259c700 1 -- 172.21.2.117:0/4254132334 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f482c00bb50 con 0x7f4834085450 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 -- 172.21.2.117:0/4254132334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4834085450 msgr2=0x7f4834086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 --2- 172.21.2.117:0/4254132334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4834085450 0x7f4834086ec0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f482c006f00 tx=0x7f482c009f10).stop 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 -- 172.21.2.117:0/4254132334 shutdown_connections 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 --2- 172.21.2.117:0/4254132334 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4834085450 0x7f4834086ec0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 --2- 172.21.2.117:0/4254132334 >> v2:172.21.2.125:3300/0 conn(0x7f48340e4e40 0x7f48341b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 -- 172.21.2.117:0/4254132334 >> 172.21.2.117:0/4254132334 conn(0x7f48340ce4e0 msgr2=0x7f48340e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 -- 172.21.2.117:0/4254132334 shutdown_connections 2021-02-10T11:54:47.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.107 7f48390dd700 1 -- 172.21.2.117:0/4254132334 wait complete. 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 Processor -- start 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 -- start start 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340c2000 0x7f4834191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48340b6c00 0x7f48340b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834191ab0 con 0x7f48340c14d0 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48341916a0 con 0x7f48340c2000 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.122 7f48390dd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4834191880 con 0x7f48340b6c00 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f483359e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48340b6c00 0x7f48340b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58268/0 (socket says 172.21.2.117:58268) 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 -- 172.21.2.117:0/2814328772 learned_addr learned my addr 172.21.2.117:0/2814328772 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f4832d9d700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340c2000 0x7f4834191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 -- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48340b6c00 msgr2=0x7f48340b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48340b6c00 0x7f48340b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 -- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340c2000 msgr2=0x7f4834191080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340c2000 0x7f4834191080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 -- 172.21.2.117:0/2814328772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48340b7620 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.123 7f482a59c700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f481c006f00 tx=0x7f481c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c00bd30 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c007070 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c007590 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f48390dd700 1 -- 172.21.2.117:0/2814328772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48340b7620 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f48390dd700 1 -- 172.21.2.117:0/2814328772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48340b7620 con 0x7f48340c14d0 2021-02-10T11:54:47.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f481c002020 con 0x7f48340c14d0 2021-02-10T11:54:47.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 0x7f481801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.124 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f481c032590 con 0x7f48340c14d0 2021-02-10T11:54:47.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.125 7f4832d9d700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 0x7f481801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.125 7f4832d9d700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 0x7f481801abc0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.156 7f48390dd700 1 -- 172.21.2.117:0/2814328772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483410c2b0 con 0x7f48340c14d0 2021-02-10T11:54:47.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.159 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f481c006b90 con 0x7f48340c14d0 2021-02-10T11:54:47.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.295 7f48390dd700 1 -- 172.21.2.117:0/2814328772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f483410c2b0 con 0x7f48340c14d0 2021-02-10T11:54:47.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.348 7f482bfff700 1 -- 172.21.2.117:0/2814328772 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v137) v1 ==== 136+0+0 (secure 0 0 0) 0x7f481c002f20 con 0x7f48340c14d0 2021-02-10T11:54:47.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 msgr2=0x7f481801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 0x7f481801abc0 crc :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 msgr2=0x7f48340c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f481c006f00 tx=0x7f481c00b940).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 shutdown_connections 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f48180186c0 0x7f481801abc0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48340c14d0 0x7f48340c18d0 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48340b6c00 0x7f48340b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 --2- 172.21.2.117:0/2814328772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48340c2000 0x7f4834191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 >> 172.21.2.117:0/2814328772 conn(0x7f48340acec0 msgr2=0x7f48340b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:47.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 shutdown_connections 2021-02-10T11:54:47.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.361 7f48390dd700 1 -- 172.21.2.117:0/2814328772 wait complete. 2021-02-10T11:54:47.372 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.780 7ffbb3fd8700 1 Processor -- start 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.780 7ffbb3fd8700 1 -- start start 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb3fd8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 0x7ffbac0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb3fd8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 0x7ffbac0e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb3fd8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac0e77f0 con 0x7ffbac0e8790 2021-02-10T11:54:47.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb3fd8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac0e92f0 con 0x7ffbac0e7d70 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb3fd8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0e5ce0 con 0x7ffbac1b07f0 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb25f5700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7ffbac0e8790 0x7ffbac0e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51452/0 (socket says 172.21.2.117:51452) 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb25f5700 1 -- 172.21.2.117:0/1291612217 learned_addr learned my addr 172.21.2.117:0/1291612217 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb15f3700 1 --2- 172.21.2.117:0/1291612217 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 0x7ffbac0e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.781 7ffbb1df4700 1 --2- 172.21.2.117:0/1291612217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 0x7ffbac0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb0df2700 1 -- 172.21.2.117:0/1291612217 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754032718 0 0) 0x7ffba4000f90 con 0x7ffbac0e8790 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb0df2700 1 -- 172.21.2.117:0/1291612217 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb98002fe0 con 0x7ffbac0e8790 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb1df4700 1 -- 172.21.2.117:0/1291612217 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 msgr2=0x7ffbac0e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb1df4700 1 --2- 172.21.2.117:0/1291612217 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 0x7ffbac0e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb1df4700 1 -- 172.21.2.117:0/1291612217 >> v1:172.21.2.125:6789/0 conn(0x7ffbac0e8790 legacy=0x7ffbac0e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb1df4700 1 -- 172.21.2.117:0/1291612217 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac0e54b0 con 0x7ffbac1b07f0 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb15f3700 1 --2- 172.21.2.117:0/1291612217 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 0x7ffbac0e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:47.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb1df4700 1 --2- 172.21.2.117:0/1291612217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 0x7ffbac0e4e90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7ffba0006650 tx=0x7ffba0009a80).ready entity=mon.2 client_cookie=5cb1aadeb5cce724 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb0df2700 1 -- 172.21.2.117:0/1291612217 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba0006390 con 0x7ffbac1b07f0 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb0df2700 1 -- 172.21.2.117:0/1291612217 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba00009b0 con 0x7ffbac1b07f0 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 -- 172.21.2.117:0/1291612217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 msgr2=0x7ffbac0e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 --2- 172.21.2.117:0/1291612217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 0x7ffbac0e4e90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7ffba0006650 tx=0x7ffba0009a80).stop 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.782 7ffbb0df2700 1 -- 172.21.2.117:0/1291612217 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba000c2b0 con 0x7ffbac1b07f0 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 -- 172.21.2.117:0/1291612217 shutdown_connections 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 --2- 172.21.2.117:0/1291612217 >> v2:172.21.2.125:3300/0 conn(0x7ffbac0e7d70 0x7ffbac0e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 --2- 172.21.2.117:0/1291612217 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac1b07f0 0x7ffbac0e4e90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 -- 172.21.2.117:0/1291612217 >> 172.21.2.117:0/1291612217 conn(0x7ffbac0ce500 msgr2=0x7ffbac0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 -- 172.21.2.117:0/1291612217 shutdown_connections 2021-02-10T11:54:47.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.783 7ffbb3fd8700 1 -- 172.21.2.117:0/1291612217 wait complete. 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 Processor -- start 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 -- start start 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 0x7ffbac08bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 0x7ffbac199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac08cca0 con 0x7ffbac08c780 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac199c90 con 0x7ffbac08c380 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.798 7ffbb3fd8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac199e70 con 0x7ffbac0af5f0 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58282/0 (socket says 172.21.2.117:58282) 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 -- 172.21.2.117:0/3863004531 learned_addr learned my addr 172.21.2.117:0/3863004531 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb1df4700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 0x7ffbac08bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb15f3700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 0x7ffbac199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 -- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 msgr2=0x7ffbac08bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 0x7ffbac08bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 -- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 msgr2=0x7ffbac199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 0x7ffbac199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 -- 172.21.2.117:0/3863004531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac09d790 con 0x7ffbac08c780 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb1df4700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 0x7ffbac08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:47.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb15f3700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 0x7ffbac199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb25f5700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7ffba4007fc0 tx=0x7ffba400ac70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba40070d0 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba40072d0 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffba4004920 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.799 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac09d790 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.800 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac09d790 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.800 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffba40046a0 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.800 7ffb9e7fc700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 0x7ffb9401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.800 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffba4032cd0 con 0x7ffbac08c780 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.801 7ffbb1df4700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 0x7ffb9401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:47.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.801 7ffbb1df4700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 0x7ffb9401abd0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:47.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.831 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac106e10 con 0x7ffbac08c780 2021-02-10T11:54:47.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.833 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffba40046a0 con 0x7ffbac08c780 2021-02-10T11:54:47.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.966 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7ffbac106e10 con 0x7ffb940186d0 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffb9e7fc700 1 -- 172.21.2.117:0/3863004531 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ffba0006aa0 con 0x7ffb940186d0 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:54:47.970 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:54:47.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 msgr2=0x7ffb9401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 0x7ffb9401abd0 crc :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 msgr2=0x7ffbac199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:47.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 rx=0x7ffba4007fc0 tx=0x7ffba400ac70).stop 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 shutdown_connections 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffb940186d0 0x7ffb9401abd0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffbac08c780 0x7ffbac199670 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffbac08c380 0x7ffbac199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 --2- 172.21.2.117:0/3863004531 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffbac0af5f0 0x7ffbac08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.968 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 >> 172.21.2.117:0/3863004531 conn(0x7ffbac0953a0 msgr2=0x7ffbac09e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.969 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 shutdown_connections 2021-02-10T11:54:47.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:47.969 7ffbb3fd8700 1 -- 172.21.2.117:0/3863004531 wait complete. 2021-02-10T11:54:47.979 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.393 7f9275aea700 1 Processor -- start 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.393 7f9275aea700 1 -- start start 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 0x7f92700eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f92700eb040 0x7f92700eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92700ee3e0 0x7f92700ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92700ede60 con 0x7f92700ee3e0 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92700eba60 con 0x7f92700eb040 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f9275aea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92700d1cf0 con 0x7f92700d18f0 2021-02-10T11:54:48.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f926ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92700ee3e0 0x7f92700ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f926ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92700ee3e0 0x7f92700ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58288/0 (socket says 172.21.2.117:58288) 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f926ffff700 1 -- 172.21.2.117:0/2517708325 learned_addr learned my addr 172.21.2.117:0/2517708325 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f926f7fe700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 0x7f92700eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.394 7f926effd700 1 --2- 172.21.2.117:0/2517708325 >> v2:172.21.2.125:3300/0 conn(0x7f92700eb040 0x7f92700eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 -- 172.21.2.117:0/2517708325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 msgr2=0x7f92700eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 0x7f92700eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 -- 172.21.2.117:0/2517708325 >> v2:172.21.2.125:3300/0 conn(0x7f92700eb040 msgr2=0x7f92700eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 --2- 172.21.2.117:0/2517708325 >> v2:172.21.2.125:3300/0 conn(0x7f92700eb040 0x7f92700eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 -- 172.21.2.117:0/2517708325 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92700eee00 con 0x7f92700ee3e0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926f7fe700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 0x7f92700eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926ffff700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700ee3e0 0x7f92700ee7e0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f9268006f00 tx=0x7f9268009f10).ready entity=mon.0 client_cookie=170fd4f2448bfee8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926e7fc700 1 -- 172.21.2.117:0/2517708325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f926800b3b0 con 0x7f92700ee3e0 2021-02-10T11:54:48.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926e7fc700 1 -- 172.21.2.117:0/2517708325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800b630 con 0x7f92700ee3e0 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f926e7fc700 1 -- 172.21.2.117:0/2517708325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f926800bb50 con 0x7f92700ee3e0 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f9275aea700 1 -- 172.21.2.117:0/2517708325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700ee3e0 msgr2=0x7f92700ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.395 7f9275aea700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700ee3e0 0x7f92700ee7e0 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f9268006f00 tx=0x7f9268009f10).stop 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 -- 172.21.2.117:0/2517708325 shutdown_connections 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 --2- 172.21.2.117:0/2517708325 >> v2:172.21.2.125:3300/0 conn(0x7f92700eb040 0x7f92700eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700ee3e0 0x7f92700ee7e0 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 --2- 172.21.2.117:0/2517708325 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92700d18f0 0x7f92700eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 -- 172.21.2.117:0/2517708325 >> 172.21.2.117:0/2517708325 conn(0x7f92700cf060 msgr2=0x7f92700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 -- 172.21.2.117:0/2517708325 shutdown_connections 2021-02-10T11:54:48.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.396 7f9275aea700 1 -- 172.21.2.117:0/2517708325 wait complete. 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.410 7f9275aea700 1 Processor -- start 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 -- start start 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92701a48b0 0x7f927008c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f927008cf40 0x7f92700a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92700aae30 con 0x7f92700a9f90 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92700aa9f0 con 0x7f927008cf40 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f9275aea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92700ab330 con 0x7f92701a48b0 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f926ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f926ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58294/0 (socket says 172.21.2.117:58294) 2021-02-10T11:54:48.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.411 7f926ffff700 1 -- 172.21.2.117:0/3802468982 learned_addr learned my addr 172.21.2.117:0/3802468982 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 -- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92701a48b0 msgr2=0x7f927008c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92701a48b0 0x7f927008c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 -- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f927008cf40 msgr2=0x7f92700a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f927008cf40 0x7f92700a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 -- 172.21.2.117:0/3802468982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92700ab550 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f926ffff700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7f9268009390 tx=0x7f926800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9268003030 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92680068e0 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f926800c570 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f9275aea700 1 -- 172.21.2.117:0/3802468982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92700ab550 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.412 7f9275aea700 1 -- 172.21.2.117:0/3802468982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92700abae0 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.413 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9268001270 con 0x7f92700a9f90 2021-02-10T11:54:48.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.413 7f9267fff700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 0x7f925401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:48.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.413 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9268032a90 con 0x7f92700a9f90 2021-02-10T11:54:48.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.413 7f926f7fe700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 0x7f925401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:48.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.414 7f926f7fe700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 0x7f925401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:48.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.444 7f9275aea700 1 -- 172.21.2.117:0/3802468982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92700ede60 con 0x7f92700a9f90 2021-02-10T11:54:48.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.447 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92680068e0 con 0x7f92700a9f90 2021-02-10T11:54:48.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.595 7f9275aea700 1 -- 172.21.2.117:0/3802468982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f92700ede60 con 0x7f92700a9f90 2021-02-10T11:54:48.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.595 7f9267fff700 1 -- 172.21.2.117:0/3802468982 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9268001270 con 0x7f92700a9f90 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 -- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 msgr2=0x7f925401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 0x7f925401abc0 crc :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 -- 172.21.2.117:0/3802468982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 msgr2=0x7f92700aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7f9268009390 tx=0x7f926800bd40).stop 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 -- 172.21.2.117:0/3802468982 shutdown_connections 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92540186c0 0x7f925401abc0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92700a9f90 0x7f92700aa390 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f927008cf40 0x7f92700a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:48.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 --2- 172.21.2.117:0/3802468982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92701a48b0 0x7f927008c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:48.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.597 7f9275aea700 1 -- 172.21.2.117:0/3802468982 >> 172.21.2.117:0/3802468982 conn(0x7f9270092d00 msgr2=0x7f927008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:48.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.598 7f9275aea700 1 -- 172.21.2.117:0/3802468982 shutdown_connections 2021-02-10T11:54:48.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:48.598 7f9275aea700 1 -- 172.21.2.117:0/3802468982 wait complete. 2021-02-10T11:54:48.608 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000439363 2021-02-10T11:54:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.021 7f6f52e40700 1 Processor -- start 2021-02-10T11:54:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.021 7f6f52e40700 1 -- start start 2021-02-10T11:54:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f52e40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f52e40700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f52e40700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0ede60 con 0x7f6f4c0eb040 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f52e40700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c088020 con 0x7f6f4c0ef3f0 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f52e40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0d1d30 con 0x7f6f4c0d18f0 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f50c5c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f5145d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f50c5c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:55982/0 (socket says 172.21.2.117:55982) 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f5145d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:34456/0 (socket says 172.21.2.117:34456) 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f50c5c700 1 -- 172.21.2.117:0/2418647650 learned_addr learned my addr 172.21.2.117:0/2418647650 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.022 7f6f4bfff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f6f4c0eb040 0x7f6f4c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51476/0 (socket says 172.21.2.117:51476) 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f50c5c700 1 -- 172.21.2.117:0/2418647650 >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 msgr2=0x7f6f4c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f50c5c700 1 --2- 172.21.2.117:0/2418647650 >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f50c5c700 1 -- 172.21.2.117:0/2418647650 >> v1:172.21.2.125:6789/0 conn(0x7f6f4c0eb040 legacy=0x7f6f4c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f50c5c700 1 -- 172.21.2.117:0/2418647650 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c1b8fd0 con 0x7f6f4c0d18f0 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f4b7fe700 1 -- 172.21.2.117:0/2418647650 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019218388 0 0) 0x7f6f340010c0 con 0x7f6f4c0eb040 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f5145d700 1 --2- 172.21.2.117:0/2418647650 >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.023 7f6f50c5c700 1 --2- 172.21.2.117:0/2418647650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f6f3c006700 tx=0x7f6f3c009b80).ready entity=mon.2 client_cookie=c1e3e48fa9d8d75a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f4b7fe700 1 -- 172.21.2.117:0/2418647650 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3c00c0c0 con 0x7f6f4c0d18f0 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f4b7fe700 1 -- 172.21.2.117:0/2418647650 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c00c440 con 0x7f6f4c0d18f0 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f4b7fe700 1 -- 172.21.2.117:0/2418647650 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3c00c960 con 0x7f6f4c0d18f0 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 -- 172.21.2.117:0/2418647650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 msgr2=0x7f6f4c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 --2- 172.21.2.117:0/2418647650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f6f3c006700 tx=0x7f6f3c009b80).stop 2021-02-10T11:54:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 -- 172.21.2.117:0/2418647650 shutdown_connections 2021-02-10T11:54:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 --2- 172.21.2.117:0/2418647650 >> v2:172.21.2.125:3300/0 conn(0x7f6f4c0ef3f0 0x7f6f4c1b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 --2- 172.21.2.117:0/2418647650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0d18f0 0x7f6f4c0eaa20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 -- 172.21.2.117:0/2418647650 >> 172.21.2.117:0/2418647650 conn(0x7f6f4c0cf060 msgr2=0x7f6f4c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 -- 172.21.2.117:0/2418647650 shutdown_connections 2021-02-10T11:54:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.024 7f6f52e40700 1 -- 172.21.2.117:0/2418647650 wait complete. 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 Processor -- start 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 -- start start 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f4c0a8eb0 0x7f6f4c0a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0a98f0 0x7f6f4c07ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0a9cf0 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c09f9f0 con 0x7f6f4c0a8eb0 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.039 7f6f52e40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c09fc10 con 0x7f6f4c0a98f0 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58310/0 (socket says 172.21.2.117:58310) 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 -- 172.21.2.117:0/1218335650 learned_addr learned my addr 172.21.2.117:0/1218335650 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:49.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f5145d700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0a98f0 0x7f6f4c07ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f50c5c700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f4c0a8eb0 0x7f6f4c0a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 -- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0a98f0 msgr2=0x7f6f4c07ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0a98f0 0x7f6f4c07ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 -- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f4c0a8eb0 msgr2=0x7f6f4c0a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f4c0a8eb0 0x7f6f4c0a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 -- 172.21.2.117:0/1218335650 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c07f310 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f4bfff700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f6f34001450 tx=0x7f6f3400a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f3400ad20 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f340075a0 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6f34004320 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.040 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c07f310 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.041 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c07f310 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.041 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6f34004320 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.041 7f6f497fa700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 0x7f6f3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.041 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6f340328f0 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.041 7f6f50c5c700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 0x7f6f3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.042 7f6f50c5c700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 0x7f6f3001abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.072 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c09fc10 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.074 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6f34000c10 con 0x7f6f4c09f5f0 2021-02-10T11:54:49.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.199 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000439363", "target": ["mgr", ""]}) v1 -- 0x7f6f4c0a9cf0 con 0x7f6f300186c0 2021-02-10T11:54:49.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:49.204 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:49.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:49.204 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:49.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:49.204 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:49.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:49.204 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:49.206 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:49.204 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:54:49.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.233 7f6f497fa700 1 -- 172.21.2.117:0/1218335650 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6f3c009100 con 0x7f6f300186c0 2021-02-10T11:54:49.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 msgr2=0x7f6f3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 0x7f6f3001abc0 crc :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 msgr2=0x7f6f4c0a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.267 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 rx=0x7f6f34001450 tx=0x7f6f3400a930).stop 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 shutdown_connections 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.264 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6f300186c0 0x7f6f3001abc0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6f4c09f5f0 0x7f6f4c0a92d0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6f4c0a98f0 0x7f6f4c07ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 --2- 172.21.2.117:0/1218335650 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6f4c0a8eb0 0x7f6f4c0a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 >> 172.21.2.117:0/1218335650 conn(0x7f6f4c0a71e0 msgr2=0x7f6f4c0a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 shutdown_connections 2021-02-10T11:54:49.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.265 7f6f52e40700 1 -- 172.21.2.117:0/1218335650 wait complete. 2021-02-10T11:54:49.275 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000439363 snapshot_0000000000755094 2021-02-10T11:54:49.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.687 7f3773502700 1 Processor -- start 2021-02-10T11:54:49.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.687 7f3773502700 1 -- start start 2021-02-10T11:54:49.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3773502700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c0e4e40 0x7f376c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3773502700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f376c0e5240 0x7f376c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3773502700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c085930 con 0x7f376c1b0810 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3773502700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c087430 con 0x7f376c0e5240 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3773502700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c0e5cc0 con 0x7f376c0e4e40 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3771b1f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f376c1b0810 0x7f376c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58568/0 (socket says 172.21.2.117:58568) 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f3771b1f700 1 -- 172.21.2.117:0/3176862047 learned_addr learned my addr 172.21.2.117:0/3176862047 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575808246 0 0) 0x7f3764001500 con 0x7f376c1b0810 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f375ffff700 1 -- 172.21.2.117:0/3176862047 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3758002fe0 con 0x7f376c1b0810 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.688 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 596726708 0 0) 0x7f3764001460 con 0x7f376c1b0810 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c0e4e40 msgr2=0x7f376c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f3770b1d700 1 --2- 172.21.2.117:0/3176862047 >> v2:172.21.2.125:3300/0 conn(0x7f376c0e5240 0x7f376c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c0e4e40 0x7f376c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c0e4e40 0x7f376c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 >> v2:172.21.2.125:3300/0 conn(0x7f376c0e5240 msgr2=0x7f376c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:49.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 --2- 172.21.2.117:0/3176862047 >> v2:172.21.2.125:3300/0 conn(0x7f376c0e5240 0x7f376c084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c084840 con 0x7f376c1b0810 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3764001980 con 0x7f376c1b0810 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 >> v1:172.21.2.117:6789/0 conn(0x7f376c1b0810 legacy=0x7f376c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3758003b90 0x7f37580060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37580068d0 0x7f3758008df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37580096f0 0x7f375800bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375800c410 con 0x7f3758003b90 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375800c630 con 0x7f37580068d0 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f375ffff700 1 -- 172.21.2.117:0/3176862047 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375800c850 con 0x7f37580096f0 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c0e4e40 0x7f376c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f3770b1d700 1 --2- 172.21.2.117:0/3176862047 >> v2:172.21.2.125:3300/0 conn(0x7f376c0e5240 0x7f376c084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3758003b90 0x7f37580060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f3771b1f700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37580096f0 0x7f375800bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f3770b1d700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37580068d0 0x7f3758008df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 -- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37580096f0 msgr2=0x7f375800bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37580096f0 0x7f375800bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 -- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37580068d0 msgr2=0x7f3758008df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37580068d0 0x7f3758008df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f377131e700 1 -- 172.21.2.117:0/3176862047 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375800caf0 con 0x7f3758003b90 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.689 7f3770b1d700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37580068d0 0x7f3758008df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f377131e700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3758003b90 0x7f37580060b0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f3768002e00 tx=0x7f376804eab0).ready entity=mon.0 client_cookie=17f816e5616e6914 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3768000c90 con 0x7f3758003b90 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37680510c0 con 0x7f3758003b90 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f375ffff700 1 -- 172.21.2.117:0/3176862047 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3768051600 con 0x7f3758003b90 2021-02-10T11:54:49.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 -- 172.21.2.117:0/3176862047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3758003b90 msgr2=0x7f37580060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 --2- 172.21.2.117:0/3176862047 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3758003b90 0x7f37580060b0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 rx=0x7f3768002e00 tx=0x7f376804eab0).stop 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3771b1f700 1 -- 172.21.2.117:0/3176862047 reap_dead start 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 -- 172.21.2.117:0/3176862047 shutdown_connections 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 -- 172.21.2.117:0/3176862047 >> 172.21.2.117:0/3176862047 conn(0x7f376c0ce4e0 msgr2=0x7f376c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 -- 172.21.2.117:0/3176862047 shutdown_connections 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.690 7f3773502700 1 -- 172.21.2.117:0/3176862047 wait complete. 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 Processor -- start 2021-02-10T11:54:49.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 -- start start 2021-02-10T11:54:49.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f376c08c290 0x7f376c1105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c110be0 0x7f376c0aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c08c8a0 con 0x7f376c1a4170 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1a4570 con 0x7f376c08c290 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f3773502700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1a4790 con 0x7f376c110be0 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f377131e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f377131e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58326/0 (socket says 172.21.2.117:58326) 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.705 7f377131e700 1 -- 172.21.2.117:0/2798966927 learned_addr learned my addr 172.21.2.117:0/2798966927 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 -- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c110be0 msgr2=0x7f376c0aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c110be0 0x7f376c0aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 -- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f376c08c290 msgr2=0x7f376c1105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f376c08c290 0x7f376c1105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 -- 172.21.2.117:0/2798966927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0af350 con 0x7f376c1a4170 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f377131e700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f376804e510 tx=0x7f3768050940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3768050d30 con 0x7f376c1a4170 2021-02-10T11:54:49.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37680052e0 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f3773502700 1 -- 172.21.2.117:0/2798966927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376c0af350 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f3773502700 1 -- 172.21.2.117:0/2798966927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c0af8e0 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.706 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3768005740 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.707 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37680054c0 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.707 7f375dffb700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 0x7f374c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.707 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3768076840 con 0x7f376c1a4170 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.707 7f3770b1d700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 0x7f374c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:49.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.708 7f3770b1d700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 0x7f374c01abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:49.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.738 7f3773502700 1 -- 172.21.2.117:0/2798966927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376c10c2a0 con 0x7f376c1a4170 2021-02-10T11:54:49.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.740 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3768000940 con 0x7f376c1a4170 2021-02-10T11:54:49.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.866 7f3773502700 1 -- 172.21.2.117:0/2798966927 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000439363", "target": ["mgr", ""], "snap_name": "snapshot_0000000000755094"}) v1 -- 0x7f376c0af8e0 con 0x7f374c0186c0 2021-02-10T11:54:49.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.877 7f375dffb700 1 -- 172.21.2.117:0/2798966927 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3760007c80 con 0x7f374c0186c0 2021-02-10T11:54:49.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 msgr2=0x7f374c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 0x7f374c01abc0 crc :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 msgr2=0x7f376c10ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f376804e510 tx=0x7f3768050940).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 shutdown_connections 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f376c110be0 0x7f376c0aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f376c08c290 0x7f376c1105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f374c0186c0 0x7f374c01abc0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 --2- 172.21.2.117:0/2798966927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f376c1a4170 0x7f376c10ffa0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:49.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 >> 172.21.2.117:0/2798966927 conn(0x7f376c0a5e90 msgr2=0x7f376c0b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:49.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 shutdown_connections 2021-02-10T11:54:49.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:49.884 7f3773502700 1 -- 172.21.2.117:0/2798966927 wait complete. 2021-02-10T11:54:49.895 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000439363 snapshot_0000000000755094 2021-02-10T11:54:50.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.308 7f6eef0a8700 1 Processor -- start 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.308 7f6eef0a8700 1 -- start start 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.308 7f6eef0a8700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6ee80e4e40 0x7f6ee81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.308 7f6eef0a8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 0x7f6ee8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.308 7f6eef0a8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee8085930 con 0x7f6ee81b0810 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eef0a8700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee80e5cc0 con 0x7f6ee80e4e40 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eef0a8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee8084840 con 0x7f6ee80e5240 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eed6c5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6ee81b0810 0x7f6ee8085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58584/0 (socket says 172.21.2.117:58584) 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eed6c5700 1 -- 172.21.2.117:0/1063979908 learned_addr learned my addr 172.21.2.117:0/1063979908 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 --2- 172.21.2.117:0/1063979908 >> v2:172.21.2.117:3300/0 conn(0x7f6ee80e4e40 0x7f6ee81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6ee77fe700 1 -- 172.21.2.117:0/1063979908 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989842102 0 0) 0x7f6ee0001180 con 0x7f6ee81b0810 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6ee77fe700 1 -- 172.21.2.117:0/1063979908 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed4002fe0 con 0x7f6ee81b0810 2021-02-10T11:54:50.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 -- 172.21.2.117:0/1063979908 >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 msgr2=0x7f6ee8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6ee7fff700 1 --2- 172.21.2.117:0/1063979908 >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 0x7f6ee8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 --2- 172.21.2.117:0/1063979908 >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 0x7f6ee8084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 -- 172.21.2.117:0/1063979908 >> v1:172.21.2.117:6789/0 conn(0x7f6ee81b0810 legacy=0x7f6ee8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 -- 172.21.2.117:0/1063979908 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee8085280 con 0x7f6ee80e4e40 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6ee7fff700 1 --2- 172.21.2.117:0/1063979908 >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 0x7f6ee8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.309 7f6eecec4700 1 --2- 172.21.2.117:0/1063979908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee80e4e40 0x7f6ee81b01f0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f6ed0006650 tx=0x7f6ed0009a80).ready entity=mon.0 client_cookie=7ad400b3fda61644 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6ee77fe700 1 -- 172.21.2.117:0/1063979908 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ed000c0c0 con 0x7f6ee80e4e40 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6ee77fe700 1 -- 172.21.2.117:0/1063979908 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed000c340 con 0x7f6ee80e4e40 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6ee77fe700 1 -- 172.21.2.117:0/1063979908 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ed000c860 con 0x7f6ee80e4e40 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 -- 172.21.2.117:0/1063979908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee80e4e40 msgr2=0x7f6ee81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 --2- 172.21.2.117:0/1063979908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee80e4e40 0x7f6ee81b01f0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 rx=0x7f6ed0006650 tx=0x7f6ed0009a80).stop 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 -- 172.21.2.117:0/1063979908 shutdown_connections 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 --2- 172.21.2.117:0/1063979908 >> v2:172.21.2.125:3300/0 conn(0x7f6ee80e5240 0x7f6ee8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 --2- 172.21.2.117:0/1063979908 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee80e4e40 0x7f6ee81b01f0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.310 7f6eef0a8700 1 -- 172.21.2.117:0/1063979908 >> 172.21.2.117:0/1063979908 conn(0x7f6ee80ce4e0 msgr2=0x7f6ee80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:50.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.311 7f6eef0a8700 1 -- 172.21.2.117:0/1063979908 shutdown_connections 2021-02-10T11:54:50.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.311 7f6eef0a8700 1 -- 172.21.2.117:0/1063979908 wait complete. 2021-02-10T11:54:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.325 7f6eef0a8700 1 Processor -- start 2021-02-10T11:54:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.325 7f6eef0a8700 1 -- start start 2021-02-10T11:54:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ee8194e10 0x7f6ee81915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ee80973f0 0x7f6ee80977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8195290 con 0x7f6ee8191bf0 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80b1460 con 0x7f6ee80973f0 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6eef0a8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee80b1680 con 0x7f6ee8194e10 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58338/0 (socket says 172.21.2.117:58338) 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 -- 172.21.2.117:0/4089119310 learned_addr learned my addr 172.21.2.117:0/4089119310 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 -- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ee8194e10 msgr2=0x7f6ee81915d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ee8194e10 0x7f6ee81915d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 -- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ee80973f0 msgr2=0x7f6ee80977f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ee80973f0 0x7f6ee80977f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 -- 172.21.2.117:0/4089119310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee80b18a0 con 0x7f6ee8191bf0 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.326 7f6ee7fff700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f6ed80099a0 tx=0x7f6ed800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ed800c2f0 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed800c630 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6ed800cb50 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee80b18a0 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee80b1e30 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6ed800c2f0 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 0x7f6ecc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.327 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6ed8032a50 con 0x7f6ee8191bf0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.328 7f6eecec4700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 0x7f6ecc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.328 7f6eecec4700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 0x7f6ecc01abc0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.361 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.359 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee8085930 con 0x7f6ee8191bf0 2021-02-10T11:54:50.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.361 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6ed8000cc0 con 0x7f6ee8191bf0 2021-02-10T11:54:50.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.490 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000439363", "target": ["mgr", ""], "snap_name": "snapshot_0000000000755094"}) v1 -- 0x7f6ee80b1e30 con 0x7f6ecc0186c0 2021-02-10T11:54:50.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.503 7f6ee57fa700 1 -- 172.21.2.117:0/4089119310 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f6ed0006740 con 0x7f6ecc0186c0 2021-02-10T11:54:50.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.508 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 msgr2=0x7f6ecc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.508 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 0x7f6ecc01abc0 crc :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.508 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 msgr2=0x7f6ee8096dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.508 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f6ed80099a0 tx=0x7f6ed800bcf0).stop 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 shutdown_connections 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6ecc0186c0 0x7f6ecc01abc0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6ee8191bf0 0x7f6ee8096dd0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6ee80973f0 0x7f6ee80977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 --2- 172.21.2.117:0/4089119310 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6ee8194e10 0x7f6ee81915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 >> 172.21.2.117:0/4089119310 conn(0x7f6ee818cc70 msgr2=0x7f6ee8093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 shutdown_connections 2021-02-10T11:54:50.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.509 7f6eef0a8700 1 -- 172.21.2.117:0/4089119310 wait complete. 2021-02-10T11:54:50.519 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot rm cephfs subvolume_0000000000439363 snapshot_0000000000755094 2021-02-10T11:54:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.931 7f42e9d17700 1 Processor -- start 2021-02-10T11:54:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.931 7f42e9d17700 1 -- start start 2021-02-10T11:54:50.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e9d17700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e40e4e40 0x7f42e41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e9d17700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f42e4085450 0x7f42e40d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e9d17700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e4085930 con 0x7f42e40e5240 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e9d17700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e40e5cc0 con 0x7f42e4085450 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e9d17700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4084840 con 0x7f42e40e4e40 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e2ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f42e40e5240 0x7f42e4084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58596/0 (socket says 172.21.2.117:58596) 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e2ffd700 1 -- 172.21.2.117:0/2666699989 learned_addr learned my addr 172.21.2.117:0/2666699989 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:50.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39373002 0 0) 0x7f42d40010b0 con 0x7f42e40e5240 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d0002fe0 con 0x7f42e40e5240 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e3fff700 1 --2- 172.21.2.117:0/2666699989 >> v2:172.21.2.125:3300/0 conn(0x7f42e4085450 0x7f42e40d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e37fe700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e40e4e40 0x7f42e41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 430109687 0 0) 0x7f42d40010b0 con 0x7f42e40e5240 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e40e4e40 msgr2=0x7f42e41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e40e4e40 0x7f42e41b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 >> v2:172.21.2.125:3300/0 conn(0x7f42e4085450 msgr2=0x7f42e40d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.932 7f42e27fc700 1 --2- 172.21.2.117:0/2666699989 >> v2:172.21.2.125:3300/0 conn(0x7f42e4085450 0x7f42e40d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40d1be0 con 0x7f42e40e5240 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f42d40015b0 con 0x7f42e40e5240 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 >> v1:172.21.2.117:6789/0 conn(0x7f42e40e5240 legacy=0x7f42e4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42d0003ae0 0x7f42d0005f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42d004a800 0x7f42d004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42d004d510 0x7f42d004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d00502e0 con 0x7f42d004a800 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0050500 con 0x7f42d004d510 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d0050720 con 0x7f42d0003ae0 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42d004a800 0x7f42d004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e37fe700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42d0003ae0 0x7f42d0005f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e3fff700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42d004d510 0x7f42d004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 -- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42d0003ae0 msgr2=0x7f42d0005f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42d0003ae0 0x7f42d0005f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 -- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42d004d510 msgr2=0x7f42d004fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42d004d510 0x7f42d004fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 -- 172.21.2.117:0/2666699989 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00509c0 con 0x7f42d004a800 2021-02-10T11:54:50.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e2ffd700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42d004a800 0x7f42d004cc70 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f42d40017d0 tx=0x7f42d4007d30).ready entity=mon.0 client_cookie=21f5caea97edd2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42d40013a0 con 0x7f42d004a800 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d400a310 con 0x7f42d004a800 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e27fc700 1 -- 172.21.2.117:0/2666699989 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42d400a7a0 con 0x7f42d004a800 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e9d17700 1 -- 172.21.2.117:0/2666699989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42d004a800 msgr2=0x7f42d004cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e9d17700 1 --2- 172.21.2.117:0/2666699989 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42d004a800 0x7f42d004cc70 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f42d40017d0 tx=0x7f42d4007d30).stop 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.933 7f42e3fff700 1 -- 172.21.2.117:0/2666699989 reap_dead start 2021-02-10T11:54:50.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.934 7f42e9d17700 1 -- 172.21.2.117:0/2666699989 shutdown_connections 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.934 7f42e9d17700 1 -- 172.21.2.117:0/2666699989 >> 172.21.2.117:0/2666699989 conn(0x7f42e40ce4e0 msgr2=0x7f42e40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.934 7f42e9d17700 1 -- 172.21.2.117:0/2666699989 shutdown_connections 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.934 7f42e9d17700 1 -- 172.21.2.117:0/2666699989 wait complete. 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 Processor -- start 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 -- start start 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42e40bf500 0x7f42e408bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e408c560 0x7f42e409d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40bfdf0 con 0x7f42e40bf900 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40b5c10 con 0x7f42e40bf500 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e9d17700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4190ae0 con 0x7f42e408c560 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58362/0 (socket says 172.21.2.117:58362) 2021-02-10T11:54:50.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.949 7f42e3fff700 1 -- 172.21.2.117:0/2354929083 learned_addr learned my addr 172.21.2.117:0/2354929083 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 -- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e408c560 msgr2=0x7f42e409d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e408c560 0x7f42e409d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 -- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42e40bf500 msgr2=0x7f42e408bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42e40bf500 0x7f42e408bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 -- 172.21.2.117:0/2354929083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e41a7240 con 0x7f42e40bf900 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e3fff700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f42dc00b910 tx=0x7f42dc00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42dc00c2f0 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc00c530 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42dc00ca50 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e41a7240 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.950 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e41a77d0 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.951 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f42dc006b90 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.951 7f42dbfff700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 0x7f42c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.951 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f42dc032af0 con 0x7f42e40bf900 2021-02-10T11:54:50.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.951 7f42e37fe700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 0x7f42c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:50.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.952 7f42e37fe700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 0x7f42c801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:50.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.982 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e410c2f0 con 0x7f42e40bf900 2021-02-10T11:54:50.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:50.984 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f42dc01f900 con 0x7f42e40bf900 2021-02-10T11:54:51.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.111 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000439363", "target": ["mgr", ""], "snap_name": "snapshot_0000000000755094"}) v1 -- 0x7f42e41a8060 con 0x7f42c80186c0 2021-02-10T11:54:51.118 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:54:51.116 7f59c98e1700 -1 mgr.server reply reply (2) No such file or directory snapshot 'snapshot_0000000000755094' does not exist 2021-02-10T11:54:51.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.117 7f42dbfff700 1 -- 172.21.2.117:0/2354929083 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: -2 snapshot 'snapshot_0000000000755094' does not exist) v1 ==== 59+0+0 (crc 0 0 0) 0x7f42cc006e50 con 0x7f42c80186c0 2021-02-10T11:54:51.122 INFO:teuthology.orchestra.run.gibba017.stderr:Error ENOENT: snapshot 'snapshot_0000000000755094' does not exist 2021-02-10T11:54:51.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 msgr2=0x7f42c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 0x7f42c801abc0 crc :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 msgr2=0x7f42e409d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 rx=0x7f42dc00b910 tx=0x7f42dc00bcb0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 shutdown_connections 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42c80186c0 0x7f42c801abc0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f42e40bf900 0x7f42e409d6c0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f42e408c560 0x7f42e409d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 --2- 172.21.2.117:0/2354929083 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f42e40bf500 0x7f42e408bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:51.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 >> 172.21.2.117:0/2354929083 conn(0x7f42e409e8a0 msgr2=0x7f42e410c520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:51.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 shutdown_connections 2021-02-10T11:54:51.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.121 7f42e9d17700 1 -- 172.21.2.117:0/2354929083 wait complete. 2021-02-10T11:54:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-02-10T11:54:51.132 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000439363 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 Processor -- start 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 -- start start 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ca00d18f0 0x7f3ca00eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 0x7f3ca01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca00ef7f0 con 0x7f3ca00eb040 2021-02-10T11:54:51.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca00eded0 con 0x7f3ca00d18f0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca7cc0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca00d1cf0 con 0x7f3ca00ef3f0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.545 7f3ca5adc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ca00d18f0 0x7f3ca00eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3ca00d18f0 0x7f3ca00eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58366/0 (socket says 172.21.2.117:58366) 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 -- 172.21.2.117:0/3926866087 learned_addr learned my addr 172.21.2.117:0/3926866087 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca62dd700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 0x7f3ca01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 -- 172.21.2.117:0/3926866087 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 msgr2=0x7f3ca01b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 0x7f3ca01b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 -- 172.21.2.117:0/3926866087 >> v1:172.21.2.125:6789/0 conn(0x7f3ca00eb040 legacy=0x7f3ca00eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 -- 172.21.2.117:0/3926866087 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca01b8fd0 con 0x7f3ca00d18f0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca62dd700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 0x7f3ca01b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca5adc700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca00d18f0 0x7f3ca00eaa20 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f3c94006700 tx=0x7f3c94009b80).ready entity=mon.0 client_cookie=47e55cb0a6bc6cdd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca4ada700 1 -- 172.21.2.117:0/3926866087 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c9400c0c0 con 0x7f3ca00d18f0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca4ada700 1 -- 172.21.2.117:0/3926866087 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400c340 con 0x7f3ca00d18f0 2021-02-10T11:54:51.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca4ada700 1 -- 172.21.2.117:0/3926866087 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c9400c860 con 0x7f3ca00d18f0 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca7cc0700 1 -- 172.21.2.117:0/3926866087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca00d18f0 msgr2=0x7f3ca00eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.546 7f3ca7cc0700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca00d18f0 0x7f3ca00eaa20 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7f3c94006700 tx=0x7f3c94009b80).stop 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 -- 172.21.2.117:0/3926866087 shutdown_connections 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca00ef3f0 0x7f3ca01b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 --2- 172.21.2.117:0/3926866087 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca00d18f0 0x7f3ca00eaa20 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 -- 172.21.2.117:0/3926866087 >> 172.21.2.117:0/3926866087 conn(0x7f3ca00cf060 msgr2=0x7f3ca00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 -- 172.21.2.117:0/3926866087 shutdown_connections 2021-02-10T11:54:51.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.547 7f3ca7cc0700 1 -- 172.21.2.117:0/3926866087 wait complete. 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 Processor -- start 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 -- start start 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ca00a67e0 0x7f3ca00a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca007e7d0 0x7f3ca007ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca007f890 con 0x7f3ca01a2e50 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca007f310 con 0x7f3ca00a67e0 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca7cc0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca01a3870 con 0x7f3ca007e7d0 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca62dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca62dd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58374/0 (socket says 172.21.2.117:58374) 2021-02-10T11:54:51.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca62dd700 1 -- 172.21.2.117:0/2706308805 learned_addr learned my addr 172.21.2.117:0/2706308805 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca52db700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca007e7d0 0x7f3ca007ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.562 7f3ca5adc700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ca00a67e0 0x7f3ca00a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 -- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca007e7d0 msgr2=0x7f3ca007ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca007e7d0 0x7f3ca007ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 -- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ca00a67e0 msgr2=0x7f3ca00a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ca00a67e0 0x7f3ca00a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 -- 172.21.2.117:0/2706308805 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca01a3a90 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca62dd700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f3c98009980 tx=0x7f3c9800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c9800c2f0 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98006790 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca01a3a90 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3c9800c870 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.563 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca01a3a90 con 0x7f3ca01a2e50 2021-02-10T11:54:51.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.564 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3c980011f0 con 0x7f3ca01a2e50 2021-02-10T11:54:51.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.564 7f3c927fc700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 0x7f3c8801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:51.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.564 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3c98032ba0 con 0x7f3ca01a2e50 2021-02-10T11:54:51.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.564 7f3ca5adc700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 0x7f3c8801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:51.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.564 7f3ca5adc700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 0x7f3c8801ac10 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:51.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.595 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca00ede60 con 0x7f3ca01a2e50 2021-02-10T11:54:51.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.598 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3c9800be10 con 0x7f3ca01a2e50 2021-02-10T11:54:51.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.724 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000439363", "target": ["mgr", ""]}) v1 -- 0x7f3ca00ede60 con 0x7f3c88018730 2021-02-10T11:54:51.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.734 7f3c927fc700 1 -- 172.21.2.117:0/2706308805 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f3c94006430 con 0x7f3c88018730 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 msgr2=0x7f3c8801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 0x7f3c8801ac10 crc :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 msgr2=0x7f3ca01a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f3c98009980 tx=0x7f3c9800bcd0).stop 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 shutdown_connections 2021-02-10T11:54:51.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3c88018730 0x7f3c8801ac10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3ca01a2e50 0x7f3ca01a3250 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3ca007e7d0 0x7f3ca007ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 --2- 172.21.2.117:0/2706308805 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3ca00a67e0 0x7f3ca00a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.741 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 >> 172.21.2.117:0/2706308805 conn(0x7f3ca009e660 msgr2=0x7f3ca00bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.742 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 shutdown_connections 2021-02-10T11:54:51.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:51.742 7f3ca7cc0700 1 -- 172.21.2.117:0/2706308805 wait complete. 2021-02-10T11:54:51.752 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.entries /home/ubuntu/cephtest/mnt.0/./volumes/_deleting 2021-02-10T11:54:52.104 INFO:teuthology.orchestra.run.gibba017.stderr:getfattr: Removing leading '/' from absolute path names 2021-02-10T11:54:52.105 INFO:teuthology.orchestra.run.gibba017.stdout:0 2021-02-10T11:54:52.105 DEBUG:tasks.cephfs.mount:Directory ./volumes/_deleting seen empty from 0 after 0s 2021-02-10T11:54:52.105 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:54:52.105 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:52.452 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:54:52.798 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:54:52.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:52.798 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:52.798 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:53.134 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:53.134 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:54:53.134 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:54:53.135 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:54:53.135 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:53.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[180463]: fuse finished with error 0 and tester_r 0 2021-02-10T11:54:53.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:53.490 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:53.490 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:53.826 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:54:53.826 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:54:53.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:53.827 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:54:54.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:54.163 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:54.163 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:54.498 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:54:54.499 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:54:54.499 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:54:54.499 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:54:54.499 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:54.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[181104]: fuse finished with error 0 and tester_r 0 2021-02-10T11:54:54.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:54.848 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:54.848 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:55.184 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:54:55.184 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:54:55.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:55.185 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:54:55.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:55.527 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:55.527 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:54:55.862 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:54:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:55.863 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:54:55.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:55.863 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:54:56.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:54:56.197 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:54:56.197 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:54:56.531 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:54:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:54:56.532 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:54:56.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:54:56.532 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:54:56.867 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:54:57.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.281 7f17f46af700 1 Processor -- start 2021-02-10T11:54:57.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.281 7f17f46af700 1 -- start start 2021-02-10T11:54:57.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f46af700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17ec1b07f0 0x7f17ec0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f46af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec0e8d80 0x7f17ec0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f46af700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17ec0e77f0 con 0x7f17ec0e6170 2021-02-10T11:54:57.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f46af700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec0e5ce0 con 0x7f17ec0e8d80 2021-02-10T11:54:57.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f46af700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ec0e54b0 con 0x7f17ec1b07f0 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec0e8d80 0x7f17ec0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17ec0e8d80 0x7f17ec0e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58378/0 (socket says 172.21.2.117:58378) 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 -- 172.21.2.117:0/1550061623 learned_addr learned my addr 172.21.2.117:0/1550061623 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 -- 172.21.2.117:0/1550061623 >> v2:172.21.2.125:3300/0 conn(0x7f17ec1b07f0 msgr2=0x7f17ec0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 --2- 172.21.2.117:0/1550061623 >> v2:172.21.2.125:3300/0 conn(0x7f17ec1b07f0 0x7f17ec0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 -- 172.21.2.117:0/1550061623 >> v1:172.21.2.125:6789/0 conn(0x7f17ec0e6170 legacy=0x7f17ec0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 -- 172.21.2.117:0/1550061623 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0e9180 con 0x7f17ec0e8d80 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.282 7f17f2ccc700 1 --2- 172.21.2.117:0/1550061623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec0e8d80 0x7f17ec0e6d60 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f17e4006f00 tx=0x7f17e4009f10).ready entity=mon.0 client_cookie=8f93dfeeac582542 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f14c9700 1 -- 172.21.2.117:0/1550061623 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400b3b0 con 0x7f17ec0e8d80 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f14c9700 1 -- 172.21.2.117:0/1550061623 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400b730 con 0x7f17ec0e8d80 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f14c9700 1 -- 172.21.2.117:0/1550061623 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e400bc50 con 0x7f17ec0e8d80 2021-02-10T11:54:57.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 -- 172.21.2.117:0/1550061623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec0e8d80 msgr2=0x7f17ec0e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 --2- 172.21.2.117:0/1550061623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec0e8d80 0x7f17ec0e6d60 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 rx=0x7f17e4006f00 tx=0x7f17e4009f10).stop 2021-02-10T11:54:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 -- 172.21.2.117:0/1550061623 shutdown_connections 2021-02-10T11:54:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 --2- 172.21.2.117:0/1550061623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec0e8d80 0x7f17ec0e6d60 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 --2- 172.21.2.117:0/1550061623 >> v2:172.21.2.125:3300/0 conn(0x7f17ec1b07f0 0x7f17ec0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 -- 172.21.2.117:0/1550061623 >> 172.21.2.117:0/1550061623 conn(0x7f17ec0ce500 msgr2=0x7f17ec0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 -- 172.21.2.117:0/1550061623 shutdown_connections 2021-02-10T11:54:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.283 7f17f46af700 1 -- 172.21.2.117:0/1550061623 wait complete. 2021-02-10T11:54:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 Processor -- start 2021-02-10T11:54:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 -- start start 2021-02-10T11:54:57.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec10c760 0x7f17ec0b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec08f180 0x7f17ec08f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec10c1e0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0b3e60 con 0x7f17ec08f180 2021-02-10T11:54:57.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.298 7f17f46af700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0b4080 con 0x7f17ec10c760 2021-02-10T11:54:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58384/0 (socket says 172.21.2.117:58384) 2021-02-10T11:54:57.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 -- 172.21.2.117:0/3201031861 learned_addr learned my addr 172.21.2.117:0/3201031861 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 -- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec10c760 msgr2=0x7f17ec0b3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec10c760 0x7f17ec0b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 -- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec08f180 msgr2=0x7f17ec08f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec08f180 0x7f17ec08f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 -- 172.21.2.117:0/3201031861 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec08fba0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f1cca700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f17e800b5b0 tx=0x7f17e800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e80009b0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e800bd40 con 0x7f17ec10bcb0 2021-02-10T11:54:57.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17e8006210 con 0x7f17ec10bcb0 2021-02-10T11:54:57.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f46af700 1 -- 172.21.2.117:0/3201031861 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec08fba0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.299 7f17f46af700 1 -- 172.21.2.117:0/3201031861 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec08fba0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.300 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17e8006230 con 0x7f17ec10bcb0 2021-02-10T11:54:57.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.300 7f17deffd700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 0x7f17d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.300 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f17e8032750 con 0x7f17ec10bcb0 2021-02-10T11:54:57.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.300 7f17f24cb700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 0x7f17d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.301 7f17f24cb700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 0x7f17d401abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.332 7f17f46af700 1 -- 172.21.2.117:0/3201031861 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec108a30 con 0x7f17ec10bcb0 2021-02-10T11:54:57.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.334 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17e8006000 con 0x7f17ec10bcb0 2021-02-10T11:54:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.483 7f17f46af700 1 -- 172.21.2.117:0/3201031861 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f17ec0e77f0 con 0x7f17ec10bcb0 2021-02-10T11:54:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.487 7f17deffd700 1 -- 172.21.2.117:0/3201031861 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f17e8006210 con 0x7f17ec10bcb0 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 -- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 msgr2=0x7f17d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 0x7f17d401abc0 crc :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 -- 172.21.2.117:0/3201031861 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 msgr2=0x7f17ec0b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7f17e800b5b0 tx=0x7f17e800b950).stop 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 -- 172.21.2.117:0/3201031861 shutdown_connections 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17d40186c0 0x7f17d401abc0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17ec10bcb0 0x7f17ec0b3840 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:57.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17ec08f180 0x7f17ec08f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 --2- 172.21.2.117:0/3201031861 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17ec10c760 0x7f17ec0b3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 -- 172.21.2.117:0/3201031861 >> 172.21.2.117:0/3201031861 conn(0x7f17ec198490 msgr2=0x7f17ec111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:57.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.492 7f17f46af700 1 -- 172.21.2.117:0/3201031861 shutdown_connections 2021-02-10T11:54:57.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.493 7f17f46af700 1 -- 172.21.2.117:0/3201031861 wait complete. 2021-02-10T11:54:57.502 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm' 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 Processor -- start 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 -- start start 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa9380ef3f0 0x7fa9380eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9380eb7f0 0x7fa9381b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9380ede60 con 0x7fa9380d18e0 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9380ef7f0 con 0x7fa9380eb7f0 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.911 7fa93d986700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9380d1ce0 con 0x7fa9380ef3f0 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9380eb7f0 0x7fa9381b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa9380eb7f0 0x7fa9381b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58388/0 (socket says 172.21.2.117:58388) 2021-02-10T11:54:57.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 -- 172.21.2.117:0/274674365 learned_addr learned my addr 172.21.2.117:0/274674365 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 -- 172.21.2.117:0/274674365 >> v2:172.21.2.125:3300/0 conn(0x7fa9380ef3f0 msgr2=0x7fa9380eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 --2- 172.21.2.117:0/274674365 >> v2:172.21.2.125:3300/0 conn(0x7fa9380ef3f0 0x7fa9380eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 -- 172.21.2.117:0/274674365 >> v1:172.21.2.125:6789/0 conn(0x7fa9380d18e0 legacy=0x7fa9380ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 -- 172.21.2.117:0/274674365 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9381b8f90 con 0x7fa9380eb7f0 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.912 7fa937d75700 1 --2- 172.21.2.117:0/274674365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380eb7f0 0x7fa9381b7960 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7fa930006f00 tx=0x7fa930009f10).ready entity=mon.0 client_cookie=17c057414a401773 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa936572700 1 -- 172.21.2.117:0/274674365 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93000b3b0 con 0x7fa9380eb7f0 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa936572700 1 -- 172.21.2.117:0/274674365 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93000b630 con 0x7fa9380eb7f0 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa936572700 1 -- 172.21.2.117:0/274674365 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa93000bb50 con 0x7fa9380eb7f0 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 -- 172.21.2.117:0/274674365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380eb7f0 msgr2=0x7fa9381b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 --2- 172.21.2.117:0/274674365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380eb7f0 0x7fa9381b7960 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7fa930006f00 tx=0x7fa930009f10).stop 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 -- 172.21.2.117:0/274674365 shutdown_connections 2021-02-10T11:54:57.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 --2- 172.21.2.117:0/274674365 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380eb7f0 0x7fa9381b7960 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:57.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 --2- 172.21.2.117:0/274674365 >> v2:172.21.2.125:3300/0 conn(0x7fa9380ef3f0 0x7fa9380eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 -- 172.21.2.117:0/274674365 >> 172.21.2.117:0/274674365 conn(0x7fa9380ced60 msgr2=0x7fa9380cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:57.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 -- 172.21.2.117:0/274674365 shutdown_connections 2021-02-10T11:54:57.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.913 7fa93d986700 1 -- 172.21.2.117:0/274674365 wait complete. 2021-02-10T11:54:57.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.927 7fa93d986700 1 Processor -- start 2021-02-10T11:54:57.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 -- start start 2021-02-10T11:54:57.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9380980c0 0x7fa93819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa93819dd40 0x7fa93819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380b1f30 con 0x7fa9380984c0 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93819d830 con 0x7fa9380980c0 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa93d986700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9381a55e0 con 0x7fa93819dd40 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58392/0 (socket says 172.21.2.117:58392) 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 -- 172.21.2.117:0/3287602529 learned_addr learned my addr 172.21.2.117:0/3287602529 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 -- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa93819dd40 msgr2=0x7fa93819e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa93819dd40 0x7fa93819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 -- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9380980c0 msgr2=0x7fa93819ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9380980c0 0x7fa93819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:57.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.928 7fa936d73700 1 -- 172.21.2.117:0/3287602529 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93819e760 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa936d73700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7fa92c006f00 tx=0x7fa92c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa92c00bd30 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92c000c90 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa93d986700 1 -- 172.21.2.117:0/3287602529 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93819e760 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa92c0022a0 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.929 7fa93d986700 1 -- 172.21.2.117:0/3287602529 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93819e760 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.930 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa92c005310 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.930 7fa91ffff700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 0x7fa91801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.930 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5992+0+0 (secure 0 0 0) 0x7fa92c032910 con 0x7fa9380984c0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.930 7fa937574700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 0x7fa91801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:57.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.931 7fa937574700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 0x7fa91801abd0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:57.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.962 7fa93d986700 1 -- 172.21.2.117:0/3287602529 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa938087210 con 0x7fa9380984c0 2021-02-10T11:54:57.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:57.964 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa92c006b90 con 0x7fa9380984c0 2021-02-10T11:54:58.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.095 7fa93d986700 1 -- 172.21.2.117:0/3287602529 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]} v 0) v1 -- 0x7fa938087210 con 0x7fa9380984c0 2021-02-10T11:54:58.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.338 7fa91ffff700 1 -- 172.21.2.117:0/3287602529 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_nonexistent_subvolume_snapshot_rm"]}]=0 v796) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa92c005310 con 0x7fa9380984c0 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 msgr2=0x7fa91801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 0x7fa91801abd0 crc :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 msgr2=0x7fa93819d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7fa92c006f00 tx=0x7fa92c00b940).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 shutdown_connections 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186d0 0x7fa91801abd0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9380984c0 0x7fa93819d0a0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa93819dd40 0x7fa93819e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 --2- 172.21.2.117:0/3287602529 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9380980c0 0x7fa93819ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:54:58.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 >> 172.21.2.117:0/3287602529 conn(0x7fa93819f610 msgr2=0x7fa9381a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:58.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 shutdown_connections 2021-02-10T11:54:58.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.362 7fa93d986700 1 -- 172.21.2.117:0/3287602529 wait complete. 2021-02-10T11:54:58.372 INFO:tasks.cephfs_test_runner:test_nonexistent_subvolume_snapshot_rm (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:54:58.372 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:54:58.372 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.781 7f4963cf9700 1 Processor -- start 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.781 7f4963cf9700 1 -- start start 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4963cf9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f495c0e4e40 0x7f495c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4963cf9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 0x7f495c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4963cf9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c085930 con 0x7f495c1b0810 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4963cf9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c084840 con 0x7f495c0e4e40 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4963cf9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c084a30 con 0x7f495c0e5240 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961b15700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f495c0e4e40 0x7f495c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 0x7f495c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4962316700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f495c1b0810 0x7f495c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58650/0 (socket says 172.21.2.117:58650) 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4962316700 1 -- 172.21.2.117:0/1354891821 learned_addr learned my addr 172.21.2.117:0/1354891821 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961314700 1 -- 172.21.2.117:0/1354891821 >> v2:172.21.2.117:3300/0 conn(0x7f495c0e4e40 msgr2=0x7f495c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961314700 1 --2- 172.21.2.117:0/1354891821 >> v2:172.21.2.117:3300/0 conn(0x7f495c0e4e40 0x7f495c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961314700 1 -- 172.21.2.117:0/1354891821 >> v1:172.21.2.117:6789/0 conn(0x7f495c1b0810 legacy=0x7f495c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.782 7f4961314700 1 -- 172.21.2.117:0/1354891821 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c085280 con 0x7f495c0e5240 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4961314700 1 --2- 172.21.2.117:0/1354891821 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 0x7f495c084c60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4958006650 tx=0x7f4958009a80).ready entity=mon.2 client_cookie=6e4e9b851c4fbcd6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4960b13700 1 -- 172.21.2.117:0/1354891821 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f495800c0c0 con 0x7f495c0e5240 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4960b13700 1 -- 172.21.2.117:0/1354891821 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495800c340 con 0x7f495c0e5240 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4960b13700 1 -- 172.21.2.117:0/1354891821 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f495800c860 con 0x7f495c0e5240 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4963cf9700 1 -- 172.21.2.117:0/1354891821 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 msgr2=0x7f495c084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.783 7f4963cf9700 1 --2- 172.21.2.117:0/1354891821 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 0x7f495c084c60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4958006650 tx=0x7f4958009a80).stop 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 -- 172.21.2.117:0/1354891821 shutdown_connections 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 --2- 172.21.2.117:0/1354891821 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c0e5240 0x7f495c084c60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 --2- 172.21.2.117:0/1354891821 >> v2:172.21.2.117:3300/0 conn(0x7f495c0e4e40 0x7f495c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 -- 172.21.2.117:0/1354891821 >> 172.21.2.117:0/1354891821 conn(0x7f495c0ce4e0 msgr2=0x7f495c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 -- 172.21.2.117:0/1354891821 shutdown_connections 2021-02-10T11:54:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.784 7f4963cf9700 1 -- 172.21.2.117:0/1354891821 wait complete. 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.798 7f4963cf9700 1 Processor -- start 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.798 7f4963cf9700 1 -- start start 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 0x7f495c18b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c18baa0 0x7f495c1aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 0x7f495c1ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1ae4c0 con 0x7f495c1ab3e0 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1ae6e0 con 0x7f495c18cc40 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4963cf9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c1ae900 con 0x7f495c18baa0 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4962316700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 0x7f495c1ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4962316700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 0x7f495c1ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58408/0 (socket says 172.21.2.117:58408) 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4961b15700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 0x7f495c18b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4961314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c18baa0 0x7f495c1aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.799 7f4962316700 1 -- 172.21.2.117:0/3010297311 learned_addr learned my addr 172.21.2.117:0/3010297311 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 -- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c18baa0 msgr2=0x7f495c1aadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c18baa0 0x7f495c1aadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 -- 172.21.2.117:0/3010297311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 msgr2=0x7f495c1ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 0x7f495c1ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 -- 172.21.2.117:0/3010297311 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c1aeb20 con 0x7f495c18cc40 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4961b15700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 0x7f495c18b480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f4950009540 tx=0x7f495000b8e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f495000bcd0 con 0x7f495c18cc40 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4950006740 con 0x7f495c18cc40 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c1aeb20 con 0x7f495c18cc40 2021-02-10T11:54:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4950002030 con 0x7f495c18cc40 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.800 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c1aeb20 con 0x7f495c18cc40 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.801 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4950000c90 con 0x7f495c18cc40 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.801 7f494e7fc700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 0x7f494401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.801 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f4950032650 con 0x7f495c18cc40 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.801 7f4961314700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 0x7f494401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:58.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.802 7f4961314700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 0x7f494401abc0 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:58.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.832 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c10c280 con 0x7f495c18cc40 2021-02-10T11:54:58.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.835 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4950006ae0 con 0x7f495c18cc40 2021-02-10T11:54:58.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:58.970 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7f495c085930 con 0x7f495c18cc40 2021-02-10T11:54:59.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.342 7f494e7fc700 1 -- 172.21.2.117:0/3010297311 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v797) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4950000c90 con 0x7f495c18cc40 2021-02-10T11:54:59.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 msgr2=0x7f494401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 0x7f494401abc0 crc :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 msgr2=0x7f495c18b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 0x7f495c18b480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f4950009540 tx=0x7f495000b8e0).stop 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 shutdown_connections 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49440186c0 0x7f494401abc0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f495c1ab3e0 0x7f495c1ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f495c18baa0 0x7f495c1aadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 --2- 172.21.2.117:0/3010297311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f495c18cc40 0x7f495c18b480 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.389 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 >> 172.21.2.117:0/3010297311 conn(0x7f495c18c310 msgr2=0x7f495c10c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.390 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 shutdown_connections 2021-02-10T11:54:59.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.390 7f4963cf9700 1 -- 172.21.2.117:0/3010297311 wait complete. 2021-02-10T11:54:59.399 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:54:59.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 Processor -- start 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 -- start start 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 0x7f833c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 0x7f833c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0ef7f0 con 0x7f833c0d18f0 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0eded0 con 0x7f833c0ef3f0 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.813 7f8342fc6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0d1cf0 con 0x7f833c0eb830 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f833bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 0x7f833c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f833bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 0x7f833c0eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58414/0 (socket says 172.21.2.117:58414) 2021-02-10T11:54:59.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f833bfff700 1 -- 172.21.2.117:0/379674621 learned_addr learned my addr 172.21.2.117:0/379674621 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 --2- 172.21.2.117:0/379674621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 0x7f833c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 -- 172.21.2.117:0/379674621 >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 msgr2=0x7f833c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 --2- 172.21.2.117:0/379674621 >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 0x7f833c0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 -- 172.21.2.117:0/379674621 >> v1:172.21.2.125:6789/0 conn(0x7f833c0d18f0 legacy=0x7f833c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 -- 172.21.2.117:0/379674621 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0d3170 con 0x7f833c0eb830 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.814 7f83415e3700 1 --2- 172.21.2.117:0/379674621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 0x7f833c1b79a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8334006e50 tx=0x7f8334009e70).ready entity=mon.2 client_cookie=ef4bf7588b8b90ad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f833b7fe700 1 -- 172.21.2.117:0/379674621 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f833400b510 con 0x7f833c0eb830 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f833b7fe700 1 -- 172.21.2.117:0/379674621 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334006790 con 0x7f833c0eb830 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f833b7fe700 1 -- 172.21.2.117:0/379674621 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f833400b940 con 0x7f833c0eb830 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 -- 172.21.2.117:0/379674621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 msgr2=0x7f833c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 --2- 172.21.2.117:0/379674621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 0x7f833c1b79a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8334006e50 tx=0x7f8334009e70).stop 2021-02-10T11:54:59.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 -- 172.21.2.117:0/379674621 shutdown_connections 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 --2- 172.21.2.117:0/379674621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c0eb830 0x7f833c1b79a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 --2- 172.21.2.117:0/379674621 >> v2:172.21.2.117:3300/0 conn(0x7f833c0ef3f0 0x7f833c0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 -- 172.21.2.117:0/379674621 >> 172.21.2.117:0/379674621 conn(0x7f833c0cf060 msgr2=0x7f833c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.815 7f8342fc6700 1 -- 172.21.2.117:0/379674621 shutdown_connections 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.816 7f8342fc6700 1 -- 172.21.2.117:0/379674621 wait complete. 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 Processor -- start 2021-02-10T11:54:59.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 -- start start 2021-02-10T11:54:59.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f833c07e7d0 0x7f833c07ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c1a2e50 0x7f833c1a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c07f890 con 0x7f833c0a67e0 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c07f310 con 0x7f833c07e7d0 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.830 7f8342fc6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c1a3870 con 0x7f833c1a2e50 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f833bfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f833c07e7d0 0x7f833c07ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58422/0 (socket says 172.21.2.117:58422) 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 -- 172.21.2.117:0/135648314 learned_addr learned my addr 172.21.2.117:0/135648314 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f83415e3700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c1a2e50 0x7f833c1a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 -- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c1a2e50 msgr2=0x7f833c1a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c1a2e50 0x7f833c1a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 -- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f833c07e7d0 msgr2=0x7f833c07ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f833c07e7d0 0x7f833c07ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:54:59.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 -- 172.21.2.117:0/135648314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c1a3a90 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.831 7f8340de2700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f832c00b5b0 tx=0x7f832c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f832c00bd40 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c006240 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f832c005720 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f8342fc6700 1 -- 172.21.2.117:0/135648314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c1a3a90 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f8342fc6700 1 -- 172.21.2.117:0/135648314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c09f8e0 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f832c0054a0 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 0x7f832001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.832 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f832c032770 con 0x7f833c0a67e0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.833 7f833bfff700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 0x7f832001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:54:59.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.833 7f833bfff700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 0x7f832001abc0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:54:59.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.863 7f8342fc6700 1 -- 172.21.2.117:0/135648314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c0ede60 con 0x7f833c0a67e0 2021-02-10T11:54:59.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:54:59.866 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f832c000940 con 0x7f833c0a67e0 2021-02-10T11:55:00.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.013 7f8342fc6700 1 -- 172.21.2.117:0/135648314 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f833c0ede60 con 0x7f833c0a67e0 2021-02-10T11:55:00.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.016 7f83397fa700 1 -- 172.21.2.117:0/135648314 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f832c0054a0 con 0x7f833c0a67e0 2021-02-10T11:55:00.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.021 7f8342fc6700 1 -- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 msgr2=0x7f832001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:00.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.021 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 0x7f832001abc0 crc :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:00.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.021 7f8342fc6700 1 -- 172.21.2.117:0/135648314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 msgr2=0x7f833c0a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:00.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.021 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f832c00b5b0 tx=0x7f832c00b950).stop 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 -- 172.21.2.117:0/135648314 shutdown_connections 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f833c1a2e50 0x7f833c1a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f833c07e7d0 0x7f833c07ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f83200186c0 0x7f832001abc0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 --2- 172.21.2.117:0/135648314 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f833c0a67e0 0x7f833c0a5f80 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 -- 172.21.2.117:0/135648314 >> 172.21.2.117:0/135648314 conn(0x7f833c09e660 msgr2=0x7f833c0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 -- 172.21.2.117:0/135648314 shutdown_connections 2021-02-10T11:55:00.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:00.022 7f8342fc6700 1 -- 172.21.2.117:0/135648314 wait complete. 2021-02-10T11:55:00.031 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:55:00.032 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:00.378 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:00.725 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:55:00.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:00.725 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:00.725 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:01.062 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:55:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:01.062 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:55:01.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:01.063 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:01.063 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:55:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:01.399 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:55:01.400 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:01.400 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:01.400 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:55:01.733 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:55:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:01.734 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:55:01.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:01.734 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:01.734 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:55:02.068 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:55:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:02.068 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:55:02.068 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.480 7f4a0ed1b700 1 Processor -- start 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.480 7f4a0ed1b700 1 -- start start 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0ed1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4a080d18e0 0x7f4a080ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0ed1b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a080ede60 con 0x7f4a080eb000 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0ed1b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a080d1ce0 con 0x7f4a080ef3f0 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0ed1b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a080ef7f0 con 0x7f4a080d18e0 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a07fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4a080eb000 0x7f4a080eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58676/0 (socket says 172.21.2.117:58676) 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a07fff700 1 -- 172.21.2.117:0/151294558 learned_addr learned my addr 172.21.2.117:0/151294558 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0cb37700 1 --2- 172.21.2.117:0/151294558 >> v2:172.21.2.117:3300/0 conn(0x7f4a080d18e0 0x7f4a080ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:02.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0cb37700 1 -- 172.21.2.117:0/151294558 >> v1:172.21.2.125:6789/0 conn(0x7f4a080ef3f0 legacy=0x7f4a081b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0cb37700 1 -- 172.21.2.117:0/151294558 >> v1:172.21.2.117:6789/0 conn(0x7f4a080eb000 legacy=0x7f4a080eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.481 7f4a0cb37700 1 -- 172.21.2.117:0/151294558 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a081bb180 con 0x7f4a080d18e0 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a0cb37700 1 --2- 172.21.2.117:0/151294558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080d18e0 0x7f4a080ea9e0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f49f0006650 tx=0x7f49f0009a80).ready entity=mon.0 client_cookie=2a7230d73400d1fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a077fe700 1 -- 172.21.2.117:0/151294558 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49f00009b0 con 0x7f4a080d18e0 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a077fe700 1 -- 172.21.2.117:0/151294558 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f000c0c0 con 0x7f4a080d18e0 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a077fe700 1 -- 172.21.2.117:0/151294558 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49f000c5e0 con 0x7f4a080d18e0 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a0ed1b700 1 -- 172.21.2.117:0/151294558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080d18e0 msgr2=0x7f4a080ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.482 7f4a0ed1b700 1 --2- 172.21.2.117:0/151294558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080d18e0 0x7f4a080ea9e0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f49f0006650 tx=0x7f49f0009a80).stop 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.483 7f4a0ed1b700 1 -- 172.21.2.117:0/151294558 shutdown_connections 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.483 7f4a0ed1b700 1 --2- 172.21.2.117:0/151294558 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080d18e0 0x7f4a080ea9e0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.483 7f4a0ed1b700 1 -- 172.21.2.117:0/151294558 >> 172.21.2.117:0/151294558 conn(0x7f4a080ced60 msgr2=0x7f4a080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:02.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.483 7f4a0ed1b700 1 -- 172.21.2.117:0/151294558 shutdown_connections 2021-02-10T11:55:02.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.483 7f4a0ed1b700 1 -- 172.21.2.117:0/151294558 wait complete. 2021-02-10T11:55:02.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.497 7f4a0ed1b700 1 Processor -- start 2021-02-10T11:55:02.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.497 7f4a0ed1b700 1 -- start start 2021-02-10T11:55:02.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:02.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a080b27a0 0x7f4a08097b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:02.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a08098170 0x7f4a081c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080b2ba0 con 0x7f4a080f2b10 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a080f2f10 con 0x7f4a08098170 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0ed1b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08098570 con 0x7f4a080b27a0 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58434/0 (socket says 172.21.2.117:58434) 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 -- 172.21.2.117:0/3099886725 learned_addr learned my addr 172.21.2.117:0/3099886725 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a07fff700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a080b27a0 0x7f4a08097b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 -- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a080b27a0 msgr2=0x7f4a08097b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a080b27a0 0x7f4a08097b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 -- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a08098170 msgr2=0x7f4a081c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a08098170 0x7f4a081c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:02.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 -- 172.21.2.117:0/3099886725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08098790 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.498 7f4a0cb37700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7f49f0009630 tx=0x7f49f000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49f00093a0 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49f000bd70 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f49f0005560 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a08098790 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a081c5760 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f49f000a000 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 0x7f49ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.499 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 5992+0+0 (secure 0 0 0) 0x7f49f0032740 con 0x7f4a080f2b10 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.500 7f4a07fff700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 0x7f49ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:02.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.500 7f4a07fff700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 0x7f49ec01abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:02.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.530 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08086140 con 0x7f4a080f2b10 2021-02-10T11:55:02.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.533 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f49f0009340 con 0x7f4a080f2b10 2021-02-10T11:55:02.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.659 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4a08086140 con 0x7f4a080f2b10 2021-02-10T11:55:02.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.659 7f4a057fa700 1 -- 172.21.2.117:0/3099886725 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v162) v1 ==== 81+0+30978 (secure 0 0 0) 0x7f49f0030d50 con 0x7f4a080f2b10 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 162, 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:55:02.533889", 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:55:02.662 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:55:02.663 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 35, 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:55:02.664 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:02.665 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:02.666 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:02.667 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.668 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:55:02.669 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 34, 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:54:22.787648", 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:02.670 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:02.671 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "159", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:55:02.672 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:02.673 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:02.674 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:55:02.675 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 35, 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:54:23.861446", 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 8193, 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-02-10T11:55:02.676 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:02.677 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "161", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:02.678 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 3, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 161, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[2~2]", 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:02.679 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.680 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:55:02.681 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:55:02.682 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 157, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.683 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.684 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.685 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.686 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.687 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:55:02.688 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.689 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 141, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.690 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.691 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.692 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:55:02.693 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:55:02.694 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.695 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 158, 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:55:02.696 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:55:02.697 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.698 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.699 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.700 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 158, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.701 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.702 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.703 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.704 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.705 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.706 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 158, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.707 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.708 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.709 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.710 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.711 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.712 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 158, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.713 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:55:02.714 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:55:02.715 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:55:02.716 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.717 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.718 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 158, 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.719 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.720 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.721 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:55:02.722 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.723 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 157, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.724 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.725 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:02.726 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:02.727 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:55:02.728 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.729 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.730 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.731 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.732 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.733 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:55:02.734 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:55:02.735 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [ 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 35, 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "snaps": [ 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:02.736 INFO:teuthology.orchestra.run.gibba017.stdout: "begin": 2, 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: "length": 2 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:55:02.737 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:55:02.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 msgr2=0x7f49ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 0x7f49ec01abc0 crc :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 msgr2=0x7f4a080b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 rx=0x7f49f0009630 tx=0x7f49f000b980).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 shutdown_connections 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4a08098170 0x7f4a081c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4a080b27a0 0x7f4a08097b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f49ec0186c0 0x7f49ec01abc0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 --2- 172.21.2.117:0/3099886725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4a080f2b10 0x7f4a080b2180 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 >> 172.21.2.117:0/3099886725 conn(0x7f4a080f3230 msgr2=0x7f4a080b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 shutdown_connections 2021-02-10T11:55:02.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:02.661 7f4a0ed1b700 1 -- 172.21.2.117:0/3099886725 wait complete. 2021-02-10T11:55:02.740 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:03.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.150 7fa656506700 1 Processor -- start 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.150 7fa656506700 1 -- start start 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa656506700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6501b6960 0x7fa6501b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa656506700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6500ede60 con 0x7fa6500d18e0 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa656506700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6500eb9c0 con 0x7fa6500ef3f0 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa656506700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500d1ce0 con 0x7fa6501b6960 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa64ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa6500d18e0 0x7fa6500ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58690/0 (socket says 172.21.2.117:58690) 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa64ffff700 1 -- 172.21.2.117:0/2958361113 learned_addr learned my addr 172.21.2.117:0/2958361113 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071423712 0 0) 0x7fa640000ce0 con 0x7fa6500d18e0 2021-02-10T11:55:03.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.151 7fa64effd700 1 -- 172.21.2.117:0/2958361113 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638002fe0 con 0x7fa6500d18e0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 450658471 0 0) 0x7fa6400010e0 con 0x7fa6500d18e0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6501b6960 0x7fa6501b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6501b6960 msgr2=0x7fa6501b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6501b6960 0x7fa6501b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 >> v1:172.21.2.125:6789/0 conn(0x7fa6500ef3f0 legacy=0x7fa6500eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6501bb3b0 con 0x7fa6500d18e0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa640001660 con 0x7fa6500d18e0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 >> v1:172.21.2.117:6789/0 conn(0x7fa6500d18e0 legacy=0x7fa6500ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa638003af0 0x7fa638005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa638006820 0x7fa638008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6380095b0 0x7fa63800bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800c380 con 0x7fa6380095b0 2021-02-10T11:55:03.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800c5a0 con 0x7fa638003af0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64effd700 1 -- 172.21.2.117:0/2958361113 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800c7c0 con 0x7fa638006820 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6501b6960 0x7fa6501b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6380095b0 0x7fa63800bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64f7fe700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa638006820 0x7fa638008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa64ffff700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa638003af0 0x7fa638005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 -- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa638006820 msgr2=0x7fa638008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa638006820 0x7fa638008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 -- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa638003af0 msgr2=0x7fa638005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa638003af0 0x7fa638005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 -- 172.21.2.117:0/2958361113 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63800ca60 con 0x7fa6380095b0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.152 7fa654b23700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6380095b0 0x7fa63800bad0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7fa648001140 tx=0x7fa64804dde0).ready entity=mon.0 client_cookie=199a84dcf8a7dacc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa648006240 con 0x7fa6380095b0 2021-02-10T11:55:03.172 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648000f60 con 0x7fa6380095b0 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa64effd700 1 -- 172.21.2.117:0/2958361113 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa6480069a0 con 0x7fa6380095b0 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 -- 172.21.2.117:0/2958361113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6380095b0 msgr2=0x7fa63800bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 --2- 172.21.2.117:0/2958361113 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6380095b0 0x7fa63800bad0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7fa648001140 tx=0x7fa64804dde0).stop 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa654b23700 1 -- 172.21.2.117:0/2958361113 reap_dead start 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 -- 172.21.2.117:0/2958361113 shutdown_connections 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 -- 172.21.2.117:0/2958361113 >> 172.21.2.117:0/2958361113 conn(0x7fa6500ced60 msgr2=0x7fa6500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 -- 172.21.2.117:0/2958361113 shutdown_connections 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.153 7fa656506700 1 -- 172.21.2.117:0/2958361113 wait complete. 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 Processor -- start 2021-02-10T11:55:03.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 -- start start 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa65008bbe0 0x7fa65008bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa65008c600 0x7fa65008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65008b660 con 0x7fa65008c600 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65008ca00 con 0x7fa65008bbe0 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.168 7fa656506700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500e9a80 con 0x7fa6500b2160 2021-02-10T11:55:03.175 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:56132/0 (socket says 172.21.2.117:56132) 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 -- 172.21.2.117:0/964416906 learned_addr learned my addr 172.21.2.117:0/964416906 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa654b23700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa65008c600 0x7fa65008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64f7fe700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa65008bbe0 0x7fa65008bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 -- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa65008bbe0 msgr2=0x7fa65008bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa65008bbe0 0x7fa65008bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 -- 172.21.2.117:0/964416906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa65008c600 msgr2=0x7fa65008a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa65008c600 0x7fa65008a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 -- 172.21.2.117:0/964416906 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65008a8a0 con 0x7fa6500b2160 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64ffff700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fa640009530 tx=0x7fa6400098d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa640007020 con 0x7fa6500b2160 2021-02-10T11:55:03.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6400043a0 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa656506700 1 -- 172.21.2.117:0/964416906 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65008a8a0 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.169 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa640004880 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.170 7fa656506700 1 -- 172.21.2.117:0/964416906 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65008a8a0 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.170 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa640009cc0 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.170 7fa64cff9700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 0x7fa63001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.170 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fa640032810 con 0x7fa6500b2160 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.170 7fa64f7fe700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 0x7fa63001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.171 7fa64f7fe700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 0x7fa63001ac10 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.200 7fa656506700 1 -- 172.21.2.117:0/964416906 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65008b410 con 0x7fa6500b2160 2021-02-10T11:55:03.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.205 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6400043a0 con 0x7fa6500b2160 2021-02-10T11:55:03.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.335 7fa656506700 1 -- 172.21.2.117:0/964416906 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa65008b410 con 0x7fa6500b2160 2021-02-10T11:55:03.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.336 7fa64cff9700 1 -- 172.21.2.117:0/964416906 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2099 (secure 0 0 0) 0x7fa6400043a0 con 0x7fa6500b2160 2021-02-10T11:55:03.337 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 137 2021-02-10T11:55:03.338 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:03.338 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":137,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:54:26.825269","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7697},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7697":{"gid":7697,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/95294604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":95294604},{"type":"v1","addr":"172.21.2.125:6835","nonce":95294604}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 -- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 msgr2=0x7fa63001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 0x7fa63001ac10 crc :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 -- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 msgr2=0x7fa6500e4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fa640009530 tx=0x7fa6400098d0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 -- 172.21.2.117:0/964416906 shutdown_connections 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa630018730 0x7fa63001ac10 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa65008c600 0x7fa65008a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa65008bbe0 0x7fa65008bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 --2- 172.21.2.117:0/964416906 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6500b2160 0x7fa6500e4dd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.337 7fa656506700 1 -- 172.21.2.117:0/964416906 >> 172.21.2.117:0/964416906 conn(0x7fa6500bb5e0 msgr2=0x7fa650090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:03.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.338 7fa656506700 1 -- 172.21.2.117:0/964416906 shutdown_connections 2021-02-10T11:55:03.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.338 7fa656506700 1 -- 172.21.2.117:0/964416906 wait complete. 2021-02-10T11:55:03.347 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:55:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.756 7fa59bae5700 1 Processor -- start 2021-02-10T11:55:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.756 7fa59bae5700 1 -- start start 2021-02-10T11:55:03.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59bae5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5941b07f0 0x7fa5940e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59bae5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5940e7d70 0x7fa5940e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59bae5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5940e77f0 con 0x7fa5940e8790 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59bae5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940e92f0 con 0x7fa5940e7d70 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59bae5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940e5ce0 con 0x7fa5941b07f0 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5940e7d70 0x7fa5940e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa5940e7d70 0x7fa5940e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58460/0 (socket says 172.21.2.117:58460) 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa59a102700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa5940e8790 0x7fa5940e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58710/0 (socket says 172.21.2.117:58710) 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 -- 172.21.2.117:0/1210657699 learned_addr learned my addr 172.21.2.117:0/1210657699 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 -- 172.21.2.117:0/1210657699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5941b07f0 msgr2=0x7fa5940e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 --2- 172.21.2.117:0/1210657699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5941b07f0 0x7fa5940e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 -- 172.21.2.117:0/1210657699 >> v1:172.21.2.117:6789/0 conn(0x7fa5940e8790 legacy=0x7fa5940e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 -- 172.21.2.117:0/1210657699 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940e54b0 con 0x7fa5940e7d70 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.757 7fa599100700 1 --2- 172.21.2.117:0/1210657699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5940e7d70 0x7fa5940e8170 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7fa590006700 tx=0x7fa590009b00).ready entity=mon.0 client_cookie=45c027237d93c8e8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa5988ff700 1 -- 172.21.2.117:0/1210657699 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa59000c0c0 con 0x7fa5940e7d70 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa5988ff700 1 -- 172.21.2.117:0/1210657699 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59000c0c0 con 0x7fa5940e7d70 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa5988ff700 1 -- 172.21.2.117:0/1210657699 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa59000c340 con 0x7fa5940e7d70 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 -- 172.21.2.117:0/1210657699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5940e7d70 msgr2=0x7fa5940e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 --2- 172.21.2.117:0/1210657699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5940e7d70 0x7fa5940e8170 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 rx=0x7fa590006700 tx=0x7fa590009b00).stop 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 -- 172.21.2.117:0/1210657699 shutdown_connections 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 --2- 172.21.2.117:0/1210657699 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa5940e7d70 0x7fa5940e8170 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 --2- 172.21.2.117:0/1210657699 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5941b07f0 0x7fa5940e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 -- 172.21.2.117:0/1210657699 >> 172.21.2.117:0/1210657699 conn(0x7fa5940ce500 msgr2=0x7fa5940e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 -- 172.21.2.117:0/1210657699 shutdown_connections 2021-02-10T11:55:03.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.758 7fa59bae5700 1 -- 172.21.2.117:0/1210657699 wait complete. 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 Processor -- start 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 -- start start 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa594199a70 0x7fa5940b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5940b1af0 0x7fa5940b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa594105f50 con 0x7fa59408ef00 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59408f300 con 0x7fa594199a70 2021-02-10T11:55:03.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.773 7fa59bae5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940c3f80 con 0x7fa5940b1af0 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58466/0 (socket says 172.21.2.117:58466) 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 -- 172.21.2.117:0/4172058100 learned_addr learned my addr 172.21.2.117:0/4172058100 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 -- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5940b1af0 msgr2=0x7fa5940b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5940b1af0 0x7fa5940b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 -- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa594199a70 msgr2=0x7fa5940b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa594199a70 0x7fa5940b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 -- 172.21.2.117:0/4172058100 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940b2510 con 0x7fa59408ef00 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa599901700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7fa588006f00 tx=0x7fa58800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa588006350 con 0x7fa59408ef00 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588006b90 con 0x7fa59408ef00 2021-02-10T11:55:03.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940b2510 con 0x7fa59408ef00 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.774 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa588000c90 con 0x7fa59408ef00 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.775 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5940b2510 con 0x7fa59408ef00 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.775 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa588007000 con 0x7fa59408ef00 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.775 7fa5867fc700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 0x7fa57c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.775 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 5964+0+0 (secure 0 0 0) 0x7fa588032920 con 0x7fa59408ef00 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.776 7fa599100700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 0x7fa57c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:03.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.776 7fa599100700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 0x7fa57c01abc0 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:03.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.807 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5941b8070 con 0x7fa59408ef00 2021-02-10T11:55:03.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.809 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa588007f00 con 0x7fa59408ef00 2021-02-10T11:55:03.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:03.935 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa5941b8070 con 0x7fa59408ef00 2021-02-10T11:55:04.543 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:55:04.541 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:55:04.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.547 7fa5867fc700 1 -- 172.21.2.117:0/4172058100 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v138) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa588006b90 con 0x7fa59408ef00 2021-02-10T11:55:04.554 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:55:04.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 msgr2=0x7fa57c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:04.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 0x7fa57c01abc0 crc :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 msgr2=0x7fa594199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7fa588006f00 tx=0x7fa58800b940).stop 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 shutdown_connections 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa5940b1af0 0x7fa5940b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa594199a70 0x7fa5940b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa57c018700 0x7fa57c01abc0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 --2- 172.21.2.117:0/4172058100 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa59408ef00 0x7fa594199450 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 >> 172.21.2.117:0/4172058100 conn(0x7fa5940b97b0 msgr2=0x7fa594098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.554 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 shutdown_connections 2021-02-10T11:55:04.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.555 7fa59bae5700 1 -- 172.21.2.117:0/4172058100 wait complete. 2021-02-10T11:55:04.564 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:04.600 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:55:04.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.976 7fe4bb4a0700 1 Processor -- start 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.976 7fe4bb4a0700 1 -- start start 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4bb4a0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4bb4a0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 0x7fe4b41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4bb4a0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40ede60 con 0x7fe4b40eb000 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4bb4a0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b40ef7f0 con 0x7fe4b40d18e0 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4bb4a0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b40d1ce0 con 0x7fe4b40ef3f0 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4b92bc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4b92bc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58470/0 (socket says 172.21.2.117:58470) 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4b92bc700 1 -- 172.21.2.117:0/1994016347 learned_addr learned my addr 172.21.2.117:0/1994016347 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.977 7fe4b92bc700 1 -- 172.21.2.117:0/1994016347 >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 msgr2=0x7fe4b41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:04.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b9abd700 1 --2- 172.21.2.117:0/1994016347 >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 0x7fe4b41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b92bc700 1 --2- 172.21.2.117:0/1994016347 >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 0x7fe4b41b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b92bc700 1 -- 172.21.2.117:0/1994016347 >> v1:172.21.2.125:6789/0 conn(0x7fe4b40eb000 legacy=0x7fe4b40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b92bc700 1 -- 172.21.2.117:0/1994016347 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b41b8f90 con 0x7fe4b40d18e0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b9abd700 1 --2- 172.21.2.117:0/1994016347 >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 0x7fe4b41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4b92bc700 1 --2- 172.21.2.117:0/1994016347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fe4a8006700 tx=0x7fe4a8009b80).ready entity=mon.0 client_cookie=f54f73a091265ba8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4a7fff700 1 -- 172.21.2.117:0/1994016347 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a800c0c0 con 0x7fe4b40d18e0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4a7fff700 1 -- 172.21.2.117:0/1994016347 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a800c400 con 0x7fe4b40d18e0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4a7fff700 1 -- 172.21.2.117:0/1994016347 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4a800c920 con 0x7fe4b40d18e0 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4bb4a0700 1 -- 172.21.2.117:0/1994016347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b40d18e0 msgr2=0x7fe4b40ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.978 7fe4bb4a0700 1 --2- 172.21.2.117:0/1994016347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7fe4a8006700 tx=0x7fe4a8009b80).stop 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 -- 172.21.2.117:0/1994016347 shutdown_connections 2021-02-10T11:55:04.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 --2- 172.21.2.117:0/1994016347 >> v2:172.21.2.125:3300/0 conn(0x7fe4b40ef3f0 0x7fe4b41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 --2- 172.21.2.117:0/1994016347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b40d18e0 0x7fe4b40ea9e0 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 -- 172.21.2.117:0/1994016347 >> 172.21.2.117:0/1994016347 conn(0x7fe4b40ced60 msgr2=0x7fe4b40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 -- 172.21.2.117:0/1994016347 shutdown_connections 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.979 7fe4bb4a0700 1 -- 172.21.2.117:0/1994016347 wait complete. 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 Processor -- start 2021-02-10T11:55:04.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 -- start start 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 0x7fe4b408a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 0x7fe4b40e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40964c0 con 0x7fe4b408a8a0 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40e7bd0 con 0x7fe4b40960c0 2021-02-10T11:55:05.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.994 7fe4bb4a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40e7df0 con 0x7fe4b40ac7b0 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58478/0 (socket says 172.21.2.117:58478) 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 -- 172.21.2.117:0/1725430622 learned_addr learned my addr 172.21.2.117:0/1725430622 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b92bc700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 0x7fe4b408a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b9abd700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 0x7fe4b40e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 -- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 msgr2=0x7fe4b408a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 0x7fe4b408a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 -- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 msgr2=0x7fe4b40e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 0x7fe4b40e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 -- 172.21.2.117:0/1725430622 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40e8010 con 0x7fe4b408a8a0 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b92bc700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 0x7fe4b408a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:05.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b9abd700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 0x7fe4b40e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4b8abb700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fe4b0006ed0 tx=0x7fe4b000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b0006240 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b00054c0 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.995 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b40e8010 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.996 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe4b00059e0 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.996 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40e8010 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.996 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe4b0006030 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.996 7fe4a5ffb700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.996 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fe4b0032720 con 0x7fe4b408a8a0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.997 7fe4b92bc700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:04.997 7fe4b92bc700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 0x7fe49401abc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:05.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.027 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b4087210 con 0x7fe4b408a8a0 2021-02-10T11:55:05.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.029 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe4b0000990 con 0x7fe4b408a8a0 2021-02-10T11:55:05.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.156 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4b4087210 con 0x7fe4b408a8a0 2021-02-10T11:55:05.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.157 7fe4a5ffb700 1 -- 172.21.2.117:0/1725430622 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+1760 (secure 0 0 0) 0x7fe4b000bd50 con 0x7fe4b408a8a0 2021-02-10T11:55:05.159 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:05.159 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":138,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12378,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/848449039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":848449039},{"type":"v1","addr":"172.21.2.117:6833","nonce":848449039}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":138,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:54:24.817179","modified":"2021-02-10 11:55:04.542199","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":163,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":17}]} 2021-02-10T11:55:05.159 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 138 2021-02-10T11:55:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.158 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 msgr2=0x7fe49401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.158 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 0x7fe49401abc0 crc :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.158 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 msgr2=0x7fe4b4095aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.158 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7fe4b0006ed0 tx=0x7fe4b000b960).stop 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 shutdown_connections 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4940186c0 0x7fe49401abc0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4b408a8a0 0x7fe4b4095aa0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4b40960c0 0x7fe4b40e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 --2- 172.21.2.117:0/1725430622 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4b40ac7b0 0x7fe4b408a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 >> 172.21.2.117:0/1725430622 conn(0x7fe4b419f610 msgr2=0x7fe4b40e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:05.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 shutdown_connections 2021-02-10T11:55:05.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.159 7fe4bb4a0700 1 -- 172.21.2.117:0/1725430622 wait complete. 2021-02-10T11:55:05.169 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:55:05.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 Processor -- start 2021-02-10T11:55:05.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 -- start start 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9f81b07f0 0x7ff9f80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80e8d80 0x7ff9f80e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80e77f0 con 0x7ff9f80e6170 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80e54b0 con 0x7ff9f81b07f0 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.586 7ff9ff220700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80e9180 con 0x7ff9f80e8d80 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9f81b07f0 0x7ff9f80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ff9f81b07f0 0x7ff9f80e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58484/0 (socket says 172.21.2.117:58484) 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 -- 172.21.2.117:0/4021398257 learned_addr learned my addr 172.21.2.117:0/4021398257 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 -- 172.21.2.117:0/4021398257 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80e8d80 msgr2=0x7ff9f80e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 --2- 172.21.2.117:0/4021398257 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80e8d80 0x7ff9f80e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 -- 172.21.2.117:0/4021398257 >> v1:172.21.2.125:6789/0 conn(0x7ff9f80e6170 legacy=0x7ff9f80e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:05.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 -- 172.21.2.117:0/4021398257 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80e7380 con 0x7ff9f81b07f0 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9fd03c700 1 --2- 172.21.2.117:0/4021398257 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f81b07f0 0x7ff9f80e4e90 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7ff9ec006700 tx=0x7ff9ec009b80).ready entity=mon.0 client_cookie=b3334d150d48cc13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9ebfff700 1 -- 172.21.2.117:0/4021398257 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9ec00c0c0 con 0x7ff9f81b07f0 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9ebfff700 1 -- 172.21.2.117:0/4021398257 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec00c340 con 0x7ff9f81b07f0 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9ebfff700 1 -- 172.21.2.117:0/4021398257 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9ec00c860 con 0x7ff9f81b07f0 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9ff220700 1 -- 172.21.2.117:0/4021398257 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f81b07f0 msgr2=0x7ff9f80e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.587 7ff9ff220700 1 --2- 172.21.2.117:0/4021398257 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f81b07f0 0x7ff9f80e4e90 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 rx=0x7ff9ec006700 tx=0x7ff9ec009b80).stop 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 -- 172.21.2.117:0/4021398257 shutdown_connections 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 --2- 172.21.2.117:0/4021398257 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80e8d80 0x7ff9f80e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 --2- 172.21.2.117:0/4021398257 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f81b07f0 0x7ff9f80e4e90 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 -- 172.21.2.117:0/4021398257 >> 172.21.2.117:0/4021398257 conn(0x7ff9f80ce500 msgr2=0x7ff9f80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:05.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 -- 172.21.2.117:0/4021398257 shutdown_connections 2021-02-10T11:55:05.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.588 7ff9ff220700 1 -- 172.21.2.117:0/4021398257 wait complete. 2021-02-10T11:55:05.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 Processor -- start 2021-02-10T11:55:05.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 -- start start 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9f8093010 0x7ff9f8189e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80b80e0 0x7ff9f8188d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8097690 con 0x7ff9f80b76c0 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8093410 con 0x7ff9f8093010 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.603 7ff9ff220700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f818a430 con 0x7ff9f80b80e0 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58488/0 (socket says 172.21.2.117:58488) 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 -- 172.21.2.117:0/1685631958 learned_addr learned my addr 172.21.2.117:0/1685631958 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fd03c700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9f8093010 0x7ff9f8189e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fd83d700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80b80e0 0x7ff9f8188d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 -- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80b80e0 msgr2=0x7ff9f8188d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80b80e0 0x7ff9f8188d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 -- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9f8093010 msgr2=0x7ff9f8189e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9f8093010 0x7ff9f8189e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 -- 172.21.2.117:0/1685631958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f81893a0 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9fc83b700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7ff9f400b5b0 tx=0x7ff9f400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.604 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f400bd40 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f4000e20 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9f40054a0 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f81893a0 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8189930 con 0x7ff9f80b76c0 2021-02-10T11:55:05.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff9f4006030 con 0x7ff9f80b76c0 2021-02-10T11:55:05.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9e9ffb700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 0x7ff9d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:05.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6022+0+0 (secure 0 0 0) 0x7ff9f40326d0 con 0x7ff9f80b76c0 2021-02-10T11:55:05.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.605 7ff9fd03c700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 0x7ff9d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:05.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.606 7ff9fd03c700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 0x7ff9d801abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:05.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.637 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f8093410 con 0x7ff9f80b76c0 2021-02-10T11:55:05.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.639 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff9f4030cc0 con 0x7ff9f80b76c0 2021-02-10T11:55:05.766 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:05.764 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff9f8093410 con 0x7ff9f80b76c0 2021-02-10T11:55:06.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.555 7ff9e9ffb700 1 -- 172.21.2.117:0/1685631958 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v139) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff9f4006030 con 0x7ff9f80b76c0 2021-02-10T11:55:06.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 msgr2=0x7ff9d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:06.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 0x7ff9d801abc0 crc :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:06.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 msgr2=0x7ff9f80b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:06.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7ff9f400b5b0 tx=0x7ff9f400b950).stop 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 shutdown_connections 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff9d80186c0 0x7ff9d801abc0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff9f80b76c0 0x7ff9f80b7ac0 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9f80b80e0 0x7ff9f8188d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 --2- 172.21.2.117:0/1685631958 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff9f8093010 0x7ff9f8189e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.587 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 >> 172.21.2.117:0/1685631958 conn(0x7ff9f8198490 msgr2=0x7ff9f8111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.588 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 shutdown_connections 2021-02-10T11:55:06.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:06.588 7ff9ff220700 1 -- 172.21.2.117:0/1685631958 wait complete. 2021-02-10T11:55:06.598 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:55:07.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 Processor -- start 2021-02-10T11:55:07.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 -- start start 2021-02-10T11:55:07.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe6801b07e0 0x7fe6800e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 0x7fe6800e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6800e77e0 con 0x7fe6800e6160 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6800e54a0 con 0x7fe6801b07e0 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe685fa0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6800e9170 con 0x7fe6800e8d70 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe67effd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe6800e6160 0x7fe6800e6560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58746/0 (socket says 172.21.2.117:58746) 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe67effd700 1 -- 172.21.2.117:0/3400403266 learned_addr learned my addr 172.21.2.117:0/3400403266 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.013 7fe67ffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 0x7fe6800e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67f7fe700 1 --2- 172.21.2.117:0/3400403266 >> v2:172.21.2.125:3300/0 conn(0x7fe6801b07e0 0x7fe6800e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67e7fc700 1 -- 172.21.2.117:0/3400403266 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056128175 0 0) 0x7fe6740010b0 con 0x7fe6800e6160 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67e7fc700 1 -- 172.21.2.117:0/3400403266 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe668002fe0 con 0x7fe6800e6160 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67ffff700 1 -- 172.21.2.117:0/3400403266 >> v2:172.21.2.125:3300/0 conn(0x7fe6801b07e0 msgr2=0x7fe6800e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67ffff700 1 --2- 172.21.2.117:0/3400403266 >> v2:172.21.2.125:3300/0 conn(0x7fe6801b07e0 0x7fe6800e4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67ffff700 1 -- 172.21.2.117:0/3400403266 >> v1:172.21.2.117:6789/0 conn(0x7fe6800e6160 legacy=0x7fe6800e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67ffff700 1 -- 172.21.2.117:0/3400403266 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800e7370 con 0x7fe6800e8d70 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.014 7fe67ffff700 1 --2- 172.21.2.117:0/3400403266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 0x7fe6800e6d50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fe678006e50 tx=0x7fe678009e70).ready entity=mon.2 client_cookie=4e57df459ca8b34 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe67e7fc700 1 -- 172.21.2.117:0/3400403266 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe67800b510 con 0x7fe6800e8d70 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe67e7fc700 1 -- 172.21.2.117:0/3400403266 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678006790 con 0x7fe6800e8d70 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe67e7fc700 1 -- 172.21.2.117:0/3400403266 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe67800b940 con 0x7fe6800e8d70 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 -- 172.21.2.117:0/3400403266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 msgr2=0x7fe6800e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 --2- 172.21.2.117:0/3400403266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 0x7fe6800e6d50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fe678006e50 tx=0x7fe678009e70).stop 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 -- 172.21.2.117:0/3400403266 shutdown_connections 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 --2- 172.21.2.117:0/3400403266 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe6800e8d70 0x7fe6800e6d50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 --2- 172.21.2.117:0/3400403266 >> v2:172.21.2.125:3300/0 conn(0x7fe6801b07e0 0x7fe6800e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.015 7fe685fa0700 1 -- 172.21.2.117:0/3400403266 >> 172.21.2.117:0/3400403266 conn(0x7fe6800ce4f0 msgr2=0x7fe6800e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.016 7fe685fa0700 1 -- 172.21.2.117:0/3400403266 shutdown_connections 2021-02-10T11:55:07.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.016 7fe685fa0700 1 -- 172.21.2.117:0/3400403266 wait complete. 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 Processor -- start 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 -- start start 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6800b4000 0x7fe68018dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe68018e310 0x7fe6801a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe680110250 con 0x7fe68010fca0 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6801a67b0 con 0x7fe6800b4000 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.030 7fe685fa0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6801a69d0 con 0x7fe68018e310 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:07.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58502/0 (socket says 172.21.2.117:58502) 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 -- 172.21.2.117:0/263343881 learned_addr learned my addr 172.21.2.117:0/263343881 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 -- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe68018e310 msgr2=0x7fe6801a6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe68018e310 0x7fe6801a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 -- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6800b4000 msgr2=0x7fe68018dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6800b4000 0x7fe68018dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 -- 172.21.2.117:0/263343881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe680110530 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe67f7fe700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7fe670006f60 tx=0x7fe67000b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe67000bd60 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670000c90 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe670002220 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe685fa0700 1 -- 172.21.2.117:0/263343881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe680110530 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.031 7fe685fa0700 1 -- 172.21.2.117:0/263343881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe680110530 con 0x7fe68010fca0 2021-02-10T11:55:07.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.032 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe6700052e0 con 0x7fe68010fca0 2021-02-10T11:55:07.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.032 7fe667fff700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 0x7fe66001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:07.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.032 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6022+0+0 (secure 0 0 0) 0x7fe6700327c0 con 0x7fe68010fca0 2021-02-10T11:55:07.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.032 7fe67effd700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 0x7fe66001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:07.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.033 7fe67effd700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 0x7fe66001abc0 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:07.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.063 7fe685fa0700 1 -- 172.21.2.117:0/263343881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68010c250 con 0x7fe68010fca0 2021-02-10T11:55:07.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.065 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe670006b00 con 0x7fe68010fca0 2021-02-10T11:55:07.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.191 7fe685fa0700 1 -- 172.21.2.117:0/263343881 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe68010c250 con 0x7fe68010fca0 2021-02-10T11:55:07.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.560 7fe667fff700 1 -- 172.21.2.117:0/263343881 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v164) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe6700052e0 con 0x7fe68010fca0 2021-02-10T11:55:07.609 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:55:07.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 msgr2=0x7fe66001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 0x7fe66001abc0 crc :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 msgr2=0x7fe6800b39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:07.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7fe670006f60 tx=0x7fe67000b970).stop 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 shutdown_connections 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe68018e310 0x7fe6801a6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe6800b4000 0x7fe68018dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe6600186c0 0x7fe66001abc0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 --2- 172.21.2.117:0/263343881 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe68010fca0 0x7fe6800b39e0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:07.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 >> 172.21.2.117:0/263343881 conn(0x7fe68019f020 msgr2=0x7fe6801af160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:07.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 shutdown_connections 2021-02-10T11:55:07.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:07.609 7fe685fa0700 1 -- 172.21.2.117:0/263343881 wait complete. 2021-02-10T11:55:07.619 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:55:08.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.033 7f4823806700 1 Processor -- start 2021-02-10T11:55:08.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 -- start start 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f481c0e4e40 0x7f481c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c0e5240 0x7f481c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f481c1b0810 0x7f481c085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c0e5cc0 con 0x7f481c1b0810 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c085930 con 0x7f481c0e4e40 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4823806700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c084840 con 0x7f481c0e5240 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4821e23700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f481c1b0810 0x7f481c085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4821e23700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f481c1b0810 0x7f481c085280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58508/0 (socket says 172.21.2.117:58508) 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.034 7f4821e23700 1 -- 172.21.2.117:0/1651020736 learned_addr learned my addr 172.21.2.117:0/1651020736 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 -- 172.21.2.117:0/1651020736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c0e5240 msgr2=0x7f481c084c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 --2- 172.21.2.117:0/1651020736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c0e5240 0x7f481c084c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 -- 172.21.2.117:0/1651020736 >> v2:172.21.2.125:3300/0 conn(0x7f481c0e4e40 msgr2=0x7f481c1b01f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:08.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 --2- 172.21.2.117:0/1651020736 >> v2:172.21.2.125:3300/0 conn(0x7f481c0e4e40 0x7f481c1b01f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 -- 172.21.2.117:0/1651020736 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c085eb0 con 0x7f481c1b0810 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4821e23700 1 --2- 172.21.2.117:0/1651020736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c1b0810 0x7f481c085280 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7f4814006f00 tx=0x7f4814009f10).ready entity=mon.0 client_cookie=6063e3ddfa7be1a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f480ffff700 1 -- 172.21.2.117:0/1651020736 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481400b3b0 con 0x7f481c1b0810 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f480ffff700 1 -- 172.21.2.117:0/1651020736 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481400b630 con 0x7f481c1b0810 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f480ffff700 1 -- 172.21.2.117:0/1651020736 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481400bb50 con 0x7f481c1b0810 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4823806700 1 -- 172.21.2.117:0/1651020736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c1b0810 msgr2=0x7f481c085280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.035 7f4823806700 1 --2- 172.21.2.117:0/1651020736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c1b0810 0x7f481c085280 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7f4814006f00 tx=0x7f4814009f10).stop 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 -- 172.21.2.117:0/1651020736 shutdown_connections 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 --2- 172.21.2.117:0/1651020736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c1b0810 0x7f481c085280 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 --2- 172.21.2.117:0/1651020736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c0e5240 0x7f481c084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 --2- 172.21.2.117:0/1651020736 >> v2:172.21.2.125:3300/0 conn(0x7f481c0e4e40 0x7f481c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.053 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 -- 172.21.2.117:0/1651020736 >> 172.21.2.117:0/1651020736 conn(0x7f481c0ce4e0 msgr2=0x7f481c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:08.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 -- 172.21.2.117:0/1651020736 shutdown_connections 2021-02-10T11:55:08.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.036 7f4823806700 1 -- 172.21.2.117:0/1651020736 wait complete. 2021-02-10T11:55:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.050 7f4823806700 1 Processor -- start 2021-02-10T11:55:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.050 7f4823806700 1 -- start start 2021-02-10T11:55:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f481c194e10 0x7f481c1915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c191bf0 0x7f481c096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c195290 con 0x7f481c0973f0 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0b1460 con 0x7f481c194e10 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4823806700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0b1680 con 0x7f481c191bf0 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58516/0 (socket says 172.21.2.117:58516) 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 -- 172.21.2.117:0/1807925813 learned_addr learned my addr 172.21.2.117:0/1807925813 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 -- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c191bf0 msgr2=0x7f481c096dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c191bf0 0x7f481c096dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 -- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f481c194e10 msgr2=0x7f481c1915d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f481c194e10 0x7f481c1915d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 -- 172.21.2.117:0/1807925813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0b18a0 con 0x7f481c0973f0 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.051 7f4821e23700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f4814009390 tx=0x7f481400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:08.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4814003030 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48140068e0 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f4823806700 1 -- 172.21.2.117:0/1807925813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c0b18a0 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f4823806700 1 -- 172.21.2.117:0/1807925813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c0b1e30 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481400c570 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4814001270 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 0x7f47fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5584+0+0 (secure 0 0 0) 0x7f4814032910 con 0x7f481c0973f0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.052 7f4821622700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 0x7f47fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:08.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.053 7f4821622700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 0x7f47fc01abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:08.085 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.084 7f4823806700 1 -- 172.21.2.117:0/1807925813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c085930 con 0x7f481c0973f0 2021-02-10T11:55:08.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.086 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4814026080 con 0x7f481c0973f0 2021-02-10T11:55:08.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.215 7f4823806700 1 -- 172.21.2.117:0/1807925813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f481c085930 con 0x7f481c0973f0 2021-02-10T11:55:08.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.562 7f480dffb700 1 -- 172.21.2.117:0/1807925813 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v165) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4814026080 con 0x7f481c0973f0 2021-02-10T11:55:08.582 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 -- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 msgr2=0x7f47fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 0x7f47fc01abc0 crc :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 -- 172.21.2.117:0/1807925813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 msgr2=0x7f481c0977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 rx=0x7f4814009390 tx=0x7f481400bd40).stop 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 -- 172.21.2.117:0/1807925813 shutdown_connections 2021-02-10T11:55:08.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f47fc0186c0 0x7f47fc01abc0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f481c0973f0 0x7f481c0977f0 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f481c191bf0 0x7f481c096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 --2- 172.21.2.117:0/1807925813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f481c194e10 0x7f481c1915d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 -- 172.21.2.117:0/1807925813 >> 172.21.2.117:0/1807925813 conn(0x7f481c18cc70 msgr2=0x7f481c093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.582 7f4823806700 1 -- 172.21.2.117:0/1807925813 shutdown_connections 2021-02-10T11:55:08.585 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:08.583 7f4823806700 1 -- 172.21.2.117:0/1807925813 wait complete. 2021-02-10T11:55:08.592 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:55:08.592 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:55:08.593 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:55:08.593 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:55:08.593 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:55:08.593 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:55:08.593 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:55:08.593 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:55:08.598 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:55:08.599 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:55:08.600 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:55:08.600 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:55:08.641 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:55:08.641 DEBUG:teuthology.parallel:result is None 2021-02-10T11:55:08.643 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:55:08.643 DEBUG:teuthology.parallel:result is None 2021-02-10T11:55:08.643 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:55:09.048 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:55:09.050 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 Processor -- start 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 -- start start 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 0x7f3a9c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3a9c0e8390 0x7f3a9c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c0e7580 con 0x7f3a9c1b07f0 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a9c0e5ce0 con 0x7f3a9c0e8390 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa22ac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c0e77f0 con 0x7f3a9c0e7180 2021-02-10T11:55:09.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3a9bfff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3a9c1b07f0 0x7f3a9c0e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58784/0 (socket says 172.21.2.117:58784) 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3a9bfff700 1 -- 172.21.2.117:0/2108700926 learned_addr learned my addr 172.21.2.117:0/2108700926 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3a9b7fe700 1 --2- 172.21.2.117:0/2108700926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 0x7f3a9c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.053 7f3aa08c9700 1 --2- 172.21.2.117:0/2108700926 >> v2:172.21.2.125:3300/0 conn(0x7f3a9c0e8390 0x7f3a9c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9affd700 1 -- 172.21.2.117:0/2108700926 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9474528 0 0) 0x7f3a94000d30 con 0x7f3a9c1b07f0 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9affd700 1 -- 172.21.2.117:0/2108700926 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a84002fe0 con 0x7f3a9c1b07f0 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9b7fe700 1 -- 172.21.2.117:0/2108700926 >> v2:172.21.2.125:3300/0 conn(0x7f3a9c0e8390 msgr2=0x7f3a9c0e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9b7fe700 1 --2- 172.21.2.117:0/2108700926 >> v2:172.21.2.125:3300/0 conn(0x7f3a9c0e8390 0x7f3a9c0e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9b7fe700 1 -- 172.21.2.117:0/2108700926 >> v1:172.21.2.117:6789/0 conn(0x7f3a9c1b07f0 legacy=0x7f3a9c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9b7fe700 1 -- 172.21.2.117:0/2108700926 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c0e8db0 con 0x7f3a9c0e7180 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9b7fe700 1 --2- 172.21.2.117:0/2108700926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 0x7f3a9c0e7d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3a90006650 tx=0x7f3a90009a80).ready entity=mon.2 client_cookie=2f14cfdcc2629ed4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9affd700 1 -- 172.21.2.117:0/2108700926 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9000c0c0 con 0x7f3a9c0e7180 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9affd700 1 -- 172.21.2.117:0/2108700926 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9000c340 con 0x7f3a9c0e7180 2021-02-10T11:55:09.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3a9affd700 1 -- 172.21.2.117:0/2108700926 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a9000c860 con 0x7f3a9c0e7180 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3aa22ac700 1 -- 172.21.2.117:0/2108700926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 msgr2=0x7f3a9c0e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.054 7f3aa22ac700 1 --2- 172.21.2.117:0/2108700926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 0x7f3a9c0e7d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3a90006650 tx=0x7f3a90009a80).stop 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 -- 172.21.2.117:0/2108700926 shutdown_connections 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 --2- 172.21.2.117:0/2108700926 >> v2:172.21.2.125:3300/0 conn(0x7f3a9c0e8390 0x7f3a9c0e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 --2- 172.21.2.117:0/2108700926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0e7180 0x7f3a9c0e7d70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 -- 172.21.2.117:0/2108700926 >> 172.21.2.117:0/2108700926 conn(0x7f3a9c0ce500 msgr2=0x7f3a9c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 -- 172.21.2.117:0/2108700926 shutdown_connections 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.055 7f3aa22ac700 1 -- 172.21.2.117:0/2108700926 wait complete. 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 Processor -- start 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 -- start start 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 0x7f3a9c1a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 0x7f3a9c1a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c0b19a0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c0930b0 con 0x7f3a9c0b0a50 2021-02-10T11:55:09.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa22ac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1a3450 con 0x7f3a9c0a3a60 2021-02-10T11:55:09.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa08c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.070 7f3aa08c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58542/0 (socket says 172.21.2.117:58542) 2021-02-10T11:55:09.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 -- 172.21.2.117:0/1172162527 learned_addr learned my addr 172.21.2.117:0/1172162527 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a9b7fe700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 0x7f3a9c1a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 -- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 msgr2=0x7f3a9c1a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a9bfff700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 0x7f3a9c1a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 0x7f3a9c1a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 -- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 msgr2=0x7f3a9c1a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 0x7f3a9c1a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 -- 172.21.2.117:0/1172162527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c1a3630 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a9b7fe700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 0x7f3a9c1a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a9bfff700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 0x7f3a9c1a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa08c9700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f3a8c00b5e0 tx=0x7f3a8c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:09.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a8c006370 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c00bd70 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c1a3630 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3a8c0024d0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.071 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c093fb0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.072 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3a8c005330 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.072 7f3a98ff9700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 0x7f3a7c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.072 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f3a8c032470 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.072 7f3a9bfff700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 0x7f3a7c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:09.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.073 7f3a9bfff700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 0x7f3a7c01abc0 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:09.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.102 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9c0b19a0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.104 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3a8c000bc0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.245 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f3a9c0b19a0 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.570 7f3a98ff9700 1 -- 172.21.2.117:0/1172162527 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v166) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3a8c01f570 con 0x7f3a9c0b15a0 2021-02-10T11:55:09.613 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:55:09.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.613 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 msgr2=0x7f3a7c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.613 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 0x7f3a7c01abc0 crc :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.613 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 msgr2=0x7f3a9c093630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.613 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f3a8c00b5e0 tx=0x7f3a8c00b980).stop 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 shutdown_connections 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3a7c0186c0 0x7f3a7c01abc0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3a9c0b15a0 0x7f3a9c093630 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3a9c0b0a50 0x7f3a9c1a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 --2- 172.21.2.117:0/1172162527 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3a9c0a3a60 0x7f3a9c1a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 >> 172.21.2.117:0/1172162527 conn(0x7f3a9c0a7260 msgr2=0x7f3a9c0bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:09.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 shutdown_connections 2021-02-10T11:55:09.617 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:09.614 7f3aa22ac700 1 -- 172.21.2.117:0/1172162527 wait complete. 2021-02-10T11:55:09.624 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:55:09.624 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.034 7f658640c700 1 Processor -- start 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.034 7f658640c700 1 -- start start 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f658640c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65801b6960 0x7f65801b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f658640c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65800ede60 con 0x7f65800d18e0 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f658640c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65800eb9c0 con 0x7f65800ef3f0 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f658640c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65800d1ce0 con 0x7f65801b6960 2021-02-10T11:55:10.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f657ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f65800d18e0 0x7f65800ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58796/0 (socket says 172.21.2.117:58796) 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f657ffff700 1 -- 172.21.2.117:0/2725817737 learned_addr learned my addr 172.21.2.117:0/2725817737 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978847680 0 0) 0x7f6570001180 con 0x7f65800d18e0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f657effd700 1 -- 172.21.2.117:0/2725817737 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568002fe0 con 0x7f65800d18e0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.035 7f6584a29700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65801b6960 0x7f65801b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3354120864 0 0) 0x7f6570001180 con 0x7f65800d18e0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65801b6960 msgr2=0x7f65801b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65801b6960 0x7f65801b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 >> v1:172.21.2.125:6789/0 conn(0x7f65800ef3f0 legacy=0x7f65800eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65801bb3b0 con 0x7f65800d18e0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6570001680 con 0x7f65800d18e0 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 >> v1:172.21.2.117:6789/0 conn(0x7f65800d18e0 legacy=0x7f65800ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6568003a40 0x7f6568005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6568006770 0x7f6568008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6568009590 0x7f656800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656800c2d0 con 0x7f6568006770 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656800c4f0 con 0x7f6568003a40 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657effd700 1 -- 172.21.2.117:0/2725817737 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6568050720 con 0x7f6568009590 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f6584a29700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6568009590 0x7f656800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657f7fe700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6568006770 0x7f6568008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.036 7f657ffff700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6568003a40 0x7f6568005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 -- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6568003a40 msgr2=0x7f6568005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6568003a40 0x7f6568005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 -- 172.21.2.117:0/2725817737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6568006770 msgr2=0x7f6568008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6568006770 0x7f6568008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 -- 172.21.2.117:0/2725817737 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568050980 con 0x7f6568009590 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f657f7fe700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6568006770 0x7f6568008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:10.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6568009590 0x7f656800ba80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6578006af0 tx=0x7f6578009e80).ready entity=mon.2 client_cookie=e3530d8935f2bfea server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657800b300 con 0x7f6568009590 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578003030 con 0x7f6568009590 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f657effd700 1 -- 172.21.2.117:0/2725817737 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f657800b730 con 0x7f6568009590 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f658640c700 1 -- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6568009590 msgr2=0x7f656800ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f658640c700 1 --2- 172.21.2.117:0/2725817737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6568009590 0x7f656800ba80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6578006af0 tx=0x7f6578009e80).stop 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.037 7f6584a29700 1 -- 172.21.2.117:0/2725817737 reap_dead start 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.038 7f658640c700 1 -- 172.21.2.117:0/2725817737 shutdown_connections 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.038 7f658640c700 1 -- 172.21.2.117:0/2725817737 >> 172.21.2.117:0/2725817737 conn(0x7f65800ced60 msgr2=0x7f65800cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.038 7f658640c700 1 -- 172.21.2.117:0/2725817737 shutdown_connections 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.038 7f658640c700 1 -- 172.21.2.117:0/2725817737 wait complete. 2021-02-10T11:55:10.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.052 7f658640c700 1 Processor -- start 2021-02-10T11:55:10.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 -- start start 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f658008bbe0 0x7f658008bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f658008c600 0x7f658008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658008b660 con 0x7f65800b2160 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658008ca00 con 0x7f658008c600 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f658640c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65800e9a80 con 0x7f658008bbe0 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58558/0 (socket says 172.21.2.117:58558) 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 -- 172.21.2.117:0/2408583492 learned_addr learned my addr 172.21.2.117:0/2408583492 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 -- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f658008bbe0 msgr2=0x7f658008bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f658008bbe0 0x7f658008bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:10.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 -- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f658008c600 msgr2=0x7f658008a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.053 7f657ffff700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f658008c600 0x7f658008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657ffff700 1 -- 172.21.2.117:0/2408583492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658008a8a0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657ffff700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f657000a540 tx=0x7f657000a8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6570001180 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65700070d0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f65700043a0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f658640c700 1 -- 172.21.2.117:0/2408583492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658008a8a0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f658640c700 1 -- 172.21.2.117:0/2408583492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658008ae30 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.054 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f65700043a0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.055 7f657cff9700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 0x7f656001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.055 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f65700321b0 con 0x7f65800b2160 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.055 7f657f7fe700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 0x7f656001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.055 7f657f7fe700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 0x7f656001abc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.084 7f658640c700 1 -- 172.21.2.117:0/2408583492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65800ede60 con 0x7f65800b2160 2021-02-10T11:55:10.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.087 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6570027830 con 0x7f65800b2160 2021-02-10T11:55:10.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.219 7f658640c700 1 -- 172.21.2.117:0/2408583492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f65800ede60 con 0x7f65800b2160 2021-02-10T11:55:10.221 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.219 7f657cff9700 1 -- 172.21.2.117:0/2408583492 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v92) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f6570027830 con 0x7f65800b2160 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.222 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.223 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.224 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.225 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.226 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:55:10.227 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.228 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.229 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:55:10.230 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.231 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:55:10.232 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.233 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.234 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:10.235 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:10.236 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 -- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 msgr2=0x7f656001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 0x7f656001abc0 crc :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 -- 172.21.2.117:0/2408583492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 msgr2=0x7f65800e4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f657000a540 tx=0x7f657000a8e0).stop 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 -- 172.21.2.117:0/2408583492 shutdown_connections 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f658008c600 0x7f658008a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f658008bbe0 0x7f658008bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:10.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65600186c0 0x7f656001abc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 --2- 172.21.2.117:0/2408583492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65800b2160 0x7f65800e4dd0 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.221 7f658640c700 1 -- 172.21.2.117:0/2408583492 >> 172.21.2.117:0/2408583492 conn(0x7f65800bb5e0 msgr2=0x7f6580090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:10.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.222 7f658640c700 1 -- 172.21.2.117:0/2408583492 shutdown_connections 2021-02-10T11:55:10.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.222 7f658640c700 1 -- 172.21.2.117:0/2408583492 wait complete. 2021-02-10T11:55:10.239 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:10.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.652 7f1771e3d700 1 Processor -- start 2021-02-10T11:55:10.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.652 7f1771e3d700 1 -- start start 2021-02-10T11:55:10.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f1771e3d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 0x7f176c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f1771e3d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 0x7f176c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f1771e3d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0ede60 con 0x7f176c0d18e0 2021-02-10T11:55:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f1771e3d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c0ef7f0 con 0x7f176c0ef3f0 2021-02-10T11:55:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f1771e3d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0d1ce0 con 0x7f176c0eb7f0 2021-02-10T11:55:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f176affd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 0x7f176c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f176b7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f176c0d18e0 0x7f176c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51742/0 (socket says 172.21.2.117:51742) 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f176b7fe700 1 -- 172.21.2.117:0/1796118589 learned_addr learned my addr 172.21.2.117:0/1796118589 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.653 7f176bfff700 1 --2- 172.21.2.117:0/1796118589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 0x7f176c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176bfff700 1 -- 172.21.2.117:0/1796118589 >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 msgr2=0x7f176c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176bfff700 1 --2- 172.21.2.117:0/1796118589 >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 0x7f176c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176bfff700 1 -- 172.21.2.117:0/1796118589 >> v1:172.21.2.125:6789/0 conn(0x7f176c0d18e0 legacy=0x7f176c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176bfff700 1 -- 172.21.2.117:0/1796118589 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c1b8f90 con 0x7f176c0eb7f0 2021-02-10T11:55:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176affd700 1 --2- 172.21.2.117:0/1796118589 >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 0x7f176c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.654 7f176bfff700 1 --2- 172.21.2.117:0/1796118589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 0x7f176c1b7960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1764006e50 tx=0x7f1764009e70).ready entity=mon.2 client_cookie=dc3520b0d7197888 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f176a7fc700 1 -- 172.21.2.117:0/1796118589 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176400b510 con 0x7f176c0eb7f0 2021-02-10T11:55:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f176a7fc700 1 -- 172.21.2.117:0/1796118589 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1764006790 con 0x7f176c0eb7f0 2021-02-10T11:55:10.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f176a7fc700 1 -- 172.21.2.117:0/1796118589 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176400b940 con 0x7f176c0eb7f0 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 -- 172.21.2.117:0/1796118589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 msgr2=0x7f176c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 --2- 172.21.2.117:0/1796118589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 0x7f176c1b7960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1764006e50 tx=0x7f1764009e70).stop 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 -- 172.21.2.117:0/1796118589 shutdown_connections 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 --2- 172.21.2.117:0/1796118589 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c0eb7f0 0x7f176c1b7960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 --2- 172.21.2.117:0/1796118589 >> v2:172.21.2.125:3300/0 conn(0x7f176c0ef3f0 0x7f176c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 -- 172.21.2.117:0/1796118589 >> 172.21.2.117:0/1796118589 conn(0x7f176c0ced60 msgr2=0x7f176c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:10.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 -- 172.21.2.117:0/1796118589 shutdown_connections 2021-02-10T11:55:10.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.655 7f1771e3d700 1 -- 172.21.2.117:0/1796118589 wait complete. 2021-02-10T11:55:10.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 Processor -- start 2021-02-10T11:55:10.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 -- start start 2021-02-10T11:55:10.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f176c091d40 0x7f176c0e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 0x7f176c1b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c197cd0 con 0x7f176c198250 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c092140 con 0x7f176c091d40 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.670 7f1771e3d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c1b2ed0 con 0x7f176c1b24b0 2021-02-10T11:55:10.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58574/0 (socket says 172.21.2.117:58574) 2021-02-10T11:55:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176bfff700 1 -- 172.21.2.117:0/3405437147 learned_addr learned my addr 172.21.2.117:0/3405437147 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 0x7f176c1b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176b7fe700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f176c091d40 0x7f176c0e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 -- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f176c091d40 msgr2=0x7f176c0e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f176c091d40 0x7f176c0e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 -- 172.21.2.117:0/3405437147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 msgr2=0x7f176c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 -- 172.21.2.117:0/3405437147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c198c70 con 0x7f176c1b24b0 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176bfff700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.671 7f176affd700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 0x7f176c1b28b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f175c009540 tx=0x7f175c00b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175c00bcd0 con 0x7f176c1b24b0 2021-02-10T11:55:10.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175c002030 con 0x7f176c1b24b0 2021-02-10T11:55:10.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175c002550 con 0x7f176c1b24b0 2021-02-10T11:55:10.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c198c70 con 0x7f176c1b24b0 2021-02-10T11:55:10.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c198c70 con 0x7f176c1b24b0 2021-02-10T11:55:10.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.672 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f175c002210 con 0x7f176c1b24b0 2021-02-10T11:55:10.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.673 7f1763fff700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 0x7f175001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:10.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.673 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f175c0322c0 con 0x7f176c1b24b0 2021-02-10T11:55:10.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.673 7f176b7fe700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 0x7f175001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:10.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.673 7f176b7fe700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 0x7f175001abc0 crc :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:10.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.702 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c087290 con 0x7f176c1b24b0 2021-02-10T11:55:10.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.707 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f175c021de0 con 0x7f176c1b24b0 2021-02-10T11:55:10.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.833 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176c0ede60 con 0x7f176c1b24b0 2021-02-10T11:55:10.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.834 7f1763fff700 1 -- 172.21.2.117:0/3405437147 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f175c021de0 con 0x7f176c1b24b0 2021-02-10T11:55:10.838 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:10.838 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":166,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:09.565353","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":141,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 msgr2=0x7f175001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 0x7f175001abc0 crc :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 msgr2=0x7f176c1b28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 0x7f176c1b28b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f175c009540 tx=0x7f175c00b8e0).stop 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 shutdown_connections 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f17500186c0 0x7f175001abc0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176c198250 0x7f176c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f176c1b24b0 0x7f176c1b28b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 --2- 172.21.2.117:0/3405437147 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f176c091d40 0x7f176c0e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:10.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.837 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 >> 172.21.2.117:0/3405437147 conn(0x7f176c1a3620 msgr2=0x7f176c0dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:10.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.838 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 shutdown_connections 2021-02-10T11:55:10.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:10.838 7f1771e3d700 1 -- 172.21.2.117:0/3405437147 wait complete. 2021-02-10T11:55:10.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:55:10.848 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 Processor -- start 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 -- start start 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed100d18f0 0x7fed100eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fed100eb040 0x7fed100eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100ede60 con 0x7fed100ee3e0 2021-02-10T11:55:11.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed100eba60 con 0x7fed100eb040 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.263 7fed151a9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed100d1cf0 con 0x7fed100d18f0 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fed100eb040 0x7fed100eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fed100eb040 0x7fed100eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58580/0 (socket says 172.21.2.117:58580) 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 -- 172.21.2.117:0/814753548 learned_addr learned my addr 172.21.2.117:0/814753548 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 -- 172.21.2.117:0/814753548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed100d18f0 msgr2=0x7fed100eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 --2- 172.21.2.117:0/814753548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed100d18f0 0x7fed100eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 -- 172.21.2.117:0/814753548 >> v1:172.21.2.125:6789/0 conn(0x7fed100ee3e0 legacy=0x7fed100ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 -- 172.21.2.117:0/814753548 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100eefd0 con 0x7fed100eb040 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0e59c700 1 --2- 172.21.2.117:0/814753548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100eb040 0x7fed100eb440 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fed00006700 tx=0x7fed00009b80).ready entity=mon.0 client_cookie=4fc3a2582aa461b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0dd9b700 1 -- 172.21.2.117:0/814753548 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed0000c0c0 con 0x7fed100eb040 2021-02-10T11:55:11.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0dd9b700 1 -- 172.21.2.117:0/814753548 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0000c340 con 0x7fed100eb040 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed0dd9b700 1 -- 172.21.2.117:0/814753548 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fed0000c860 con 0x7fed100eb040 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed151a9700 1 -- 172.21.2.117:0/814753548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100eb040 msgr2=0x7fed100eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.264 7fed151a9700 1 --2- 172.21.2.117:0/814753548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100eb040 0x7fed100eb440 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7fed00006700 tx=0x7fed00009b80).stop 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 -- 172.21.2.117:0/814753548 shutdown_connections 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 --2- 172.21.2.117:0/814753548 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100eb040 0x7fed100eb440 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 --2- 172.21.2.117:0/814753548 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed100d18f0 0x7fed100eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 -- 172.21.2.117:0/814753548 >> 172.21.2.117:0/814753548 conn(0x7fed100cf060 msgr2=0x7fed100cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 -- 172.21.2.117:0/814753548 shutdown_connections 2021-02-10T11:55:11.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.265 7fed151a9700 1 -- 172.21.2.117:0/814753548 wait complete. 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 Processor -- start 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 -- start start 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed1007e7d0 0x7fed1007ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fed101a2e50 0x7fed101a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1007f890 con 0x7fed100a67e0 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1007f310 con 0x7fed101a2e50 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed151a9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed101a3870 con 0x7fed1007e7d0 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed0ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed0ed9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58586/0 (socket says 172.21.2.117:58586) 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.280 7fed0ed9d700 1 -- 172.21.2.117:0/3366421053 learned_addr learned my addr 172.21.2.117:0/3366421053 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:11.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 -- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed1007e7d0 msgr2=0x7fed1007ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed1007e7d0 0x7fed1007ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 -- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fed101a2e50 msgr2=0x7fed101a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fed101a2e50 0x7fed101a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 -- 172.21.2.117:0/3366421053 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed101a3a90 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed0ed9d700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7fecf800b560 tx=0x7fecf800b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecf800bcf0 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf8006030 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fecf80054d0 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed151a9700 1 -- 172.21.2.117:0/3366421053 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed101a3a90 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.281 7fed151a9700 1 -- 172.21.2.117:0/3366421053 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1009f8e0 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.282 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fecf8006210 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.282 7fed077fe700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 0x7fecf401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.282 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fecf80323c0 con 0x7fed100a67e0 2021-02-10T11:55:11.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.282 7fed0e59c700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 0x7fecf401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.283 7fed0e59c700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 0x7fecf401abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.313 7fed151a9700 1 -- 172.21.2.117:0/3366421053 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100ede60 con 0x7fed100a67e0 2021-02-10T11:55:11.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.316 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fecf8000960 con 0x7fed100a67e0 2021-02-10T11:55:11.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.441 7fed151a9700 1 -- 172.21.2.117:0/3366421053 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fed100ede60 con 0x7fed100a67e0 2021-02-10T11:55:11.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.442 7fed077fe700 1 -- 172.21.2.117:0/3366421053 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v166) v1 ==== 92+0+10 (secure 0 0 0) 0x7fecf80061f0 con 0x7fed100a67e0 2021-02-10T11:55:11.444 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 msgr2=0x7fecf401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 0x7fecf401abc0 crc :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 msgr2=0x7fed100a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7fecf800b560 tx=0x7fecf800b900).stop 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 shutdown_connections 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fed101a2e50 0x7fed101a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fed1007e7d0 0x7fed1007ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fecf40186c0 0x7fecf401abc0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:11.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 --2- 172.21.2.117:0/3366421053 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fed100a67e0 0x7fed100a5f80 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:11.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 >> 172.21.2.117:0/3366421053 conn(0x7fed1009e660 msgr2=0x7fed100bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:11.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 shutdown_connections 2021-02-10T11:55:11.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.444 7fed151a9700 1 -- 172.21.2.117:0/3366421053 wait complete. 2021-02-10T11:55:11.454 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.864 7fe373de4700 1 Processor -- start 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe373de4700 1 -- start start 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe373de4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe36c0d18e0 0x7fe36c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe373de4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0ede60 con 0x7fe36c0eb000 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe373de4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0d1ce0 con 0x7fe36c0ef3f0 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe373de4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c0ef7f0 con 0x7fe36c0d18e0 2021-02-10T11:55:11.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe3713ff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe36c0eb000 0x7fe36c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58844/0 (socket says 172.21.2.117:58844) 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe3713ff700 1 -- 172.21.2.117:0/774562801 learned_addr learned my addr 172.21.2.117:0/774562801 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.865 7fe371c00700 1 --2- 172.21.2.117:0/774562801 >> v2:172.21.2.117:3300/0 conn(0x7fe36c0d18e0 0x7fe36c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe371c00700 1 -- 172.21.2.117:0/774562801 >> v1:172.21.2.125:6789/0 conn(0x7fe36c0ef3f0 legacy=0x7fe36c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe371c00700 1 -- 172.21.2.117:0/774562801 >> v1:172.21.2.117:6789/0 conn(0x7fe36c0eb000 legacy=0x7fe36c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe371c00700 1 -- 172.21.2.117:0/774562801 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c1bb180 con 0x7fe36c0d18e0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe371c00700 1 --2- 172.21.2.117:0/774562801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0d18e0 0x7fe36c0ea9e0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fe368006650 tx=0x7fe368009a80).ready entity=mon.0 client_cookie=32fb1b9f3bb95233 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe370bfe700 1 -- 172.21.2.117:0/774562801 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe3680009b0 con 0x7fe36c0d18e0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe370bfe700 1 -- 172.21.2.117:0/774562801 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36800c0c0 con 0x7fe36c0d18e0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe370bfe700 1 -- 172.21.2.117:0/774562801 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe36800c520 con 0x7fe36c0d18e0 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe373de4700 1 -- 172.21.2.117:0/774562801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0d18e0 msgr2=0x7fe36c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.866 7fe373de4700 1 --2- 172.21.2.117:0/774562801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0d18e0 0x7fe36c0ea9e0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fe368006650 tx=0x7fe368009a80).stop 2021-02-10T11:55:11.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.867 7fe373de4700 1 -- 172.21.2.117:0/774562801 shutdown_connections 2021-02-10T11:55:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.867 7fe373de4700 1 --2- 172.21.2.117:0/774562801 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0d18e0 0x7fe36c0ea9e0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.867 7fe373de4700 1 -- 172.21.2.117:0/774562801 >> 172.21.2.117:0/774562801 conn(0x7fe36c0ced60 msgr2=0x7fe36c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.867 7fe373de4700 1 -- 172.21.2.117:0/774562801 shutdown_connections 2021-02-10T11:55:11.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.867 7fe373de4700 1 -- 172.21.2.117:0/774562801 wait complete. 2021-02-10T11:55:11.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 Processor -- start 2021-02-10T11:55:11.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 -- start start 2021-02-10T11:55:11.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe36c0f4d60 0x7fe36c0ba070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe36c0bb140 0x7fe36c0a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0f5bc0 con 0x7fe36c0bad40 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0f5160 con 0x7fe36c0f4d60 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe373de4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0a73f0 con 0x7fe36c0bb140 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe3713ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe3713ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58600/0 (socket says 172.21.2.117:58600) 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.882 7fe3713ff700 1 -- 172.21.2.117:0/20281926 learned_addr learned my addr 172.21.2.117:0/20281926 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 -- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe36c0bb140 msgr2=0x7fe36c0a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe36c0bb140 0x7fe36c0a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 -- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe36c0f4d60 msgr2=0x7fe36c0ba070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe36c0f4d60 0x7fe36c0ba070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 -- 172.21.2.117:0/20281926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0a7610 con 0x7fe36c0bad40 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3713ff700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7fe35c001c60 tx=0x7fe35c00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c00acc0 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c004140 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe35c004660 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe373de4700 1 -- 172.21.2.117:0/20281926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0a7610 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.883 7fe373de4700 1 -- 172.21.2.117:0/20281926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c0a7610 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.884 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe35c000ce0 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.884 7fe3627fc700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.884 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fe35c0323e0 con 0x7fe36c0bad40 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.884 7fe371c00700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:11.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.885 7fe371c00700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:11.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.915 7fe373de4700 1 -- 172.21.2.117:0/20281926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c086140 con 0x7fe36c0bad40 2021-02-10T11:55:11.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:11.918 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe35c0009b0 con 0x7fe36c0bad40 2021-02-10T11:55:12.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.039 7fe373de4700 1 -- 172.21.2.117:0/20281926 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe36c0d18e0 con 0x7fe36c0bad40 2021-02-10T11:55:12.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.040 7fe3627fc700 1 -- 172.21.2.117:0/20281926 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v166) v1 ==== 84+0+64 (secure 0 0 0) 0x7fe35c000ce0 con 0x7fe36c0bad40 2021-02-10T11:55:12.042 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:12.042 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:55:12.042 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:12.042 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:55:12.042 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:55:12.043 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:12.043 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:55:12.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 msgr2=0x7fe35401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 crc :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 msgr2=0x7fe36c0f55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7fe35c001c60 tx=0x7fe35c00a8d0).stop 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 shutdown_connections 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe3540186c0 0x7fe35401abc0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe36c0bad40 0x7fe36c0f55a0 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe36c0bb140 0x7fe36c0a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:12.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 --2- 172.21.2.117:0/20281926 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe36c0f4d60 0x7fe36c0ba070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:12.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 >> 172.21.2.117:0/20281926 conn(0x7fe36c099f80 msgr2=0x7fe36c092940 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:12.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 shutdown_connections 2021-02-10T11:55:12.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.042 7fe373de4700 1 -- 172.21.2.117:0/20281926 wait complete. 2021-02-10T11:55:12.052 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:55:12.052 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:55:12.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 Processor -- start 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 -- start start 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0d18f0 0x7f54bc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc0ede60 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc0d1cf0 con 0x7f54bc0eb040 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.466 7f54c1539700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc088020 con 0x7f54bc0d18f0 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54bbac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f54bc0ef3f0 0x7f54bc1b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58854/0 (socket says 172.21.2.117:58854) 2021-02-10T11:55:12.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54bbac0700 1 -- 172.21.2.117:0/1904148566 learned_addr learned my addr 172.21.2.117:0/1904148566 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54bb2bf700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0d18f0 0x7f54bc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769225162 0 0) 0x7f54b4001500 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8002fe0 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199524893 0 0) 0x7f54a4000ff0 con 0x7f54bc0eb040 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8003210 con 0x7f54bc0eb040 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 243855498 0 0) 0x7f54b4001460 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0d18f0 msgr2=0x7f54bc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:12.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0d18f0 0x7f54bc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 >> v1:172.21.2.125:6789/0 conn(0x7f54bc0eb040 legacy=0x7f54bc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc1bb200 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f54b4001980 con 0x7f54bc0ef3f0 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 >> v1:172.21.2.117:6789/0 conn(0x7f54bc0ef3f0 legacy=0x7f54bc1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54a80037e0 0x7f54a8003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54a80067e0 0x7f54a8008c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54a8009510 0x7f54a800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80502f0 con 0x7f54a80067e0 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80504d0 con 0x7f54a8009510 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a80506f0 con 0x7f54a80037e0 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.467 7f54baabe700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54a80067e0 0x7f54a8008c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54bbac0700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54a8009510 0x7f54a800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54bb2bf700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54a80037e0 0x7f54a8003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 -- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54a80037e0 msgr2=0x7f54a8003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:12.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54a80037e0 0x7f54a8003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 -- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54a8009510 msgr2=0x7f54a800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54a8009510 0x7f54a800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 -- 172.21.2.117:0/1904148566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8050990 con 0x7f54a80067e0 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54baabe700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54a80067e0 0x7f54a8008c70 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f54a4000b80 tx=0x7f54a4009a10).ready entity=mon.0 client_cookie=c55bf6881bc0363e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a40066d0 con 0x7f54a80067e0 2021-02-10T11:55:12.490 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4007280 con 0x7f54a80067e0 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54ba2bd700 1 -- 172.21.2.117:0/1904148566 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a40077a0 con 0x7f54a80067e0 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54c1539700 1 -- 172.21.2.117:0/1904148566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54a80067e0 msgr2=0x7f54a8008c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54c1539700 1 --2- 172.21.2.117:0/1904148566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54a80067e0 0x7f54a8008c70 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f54a4000b80 tx=0x7f54a4009a10).stop 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.468 7f54bbac0700 1 -- 172.21.2.117:0/1904148566 reap_dead start 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.469 7f54c1539700 1 -- 172.21.2.117:0/1904148566 shutdown_connections 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.469 7f54c1539700 1 -- 172.21.2.117:0/1904148566 >> 172.21.2.117:0/1904148566 conn(0x7f54bc0cf060 msgr2=0x7f54bc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:12.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.469 7f54c1539700 1 -- 172.21.2.117:0/1904148566 shutdown_connections 2021-02-10T11:55:12.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.469 7f54c1539700 1 -- 172.21.2.117:0/1904148566 wait complete. 2021-02-10T11:55:12.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 Processor -- start 2021-02-10T11:55:12.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 -- start start 2021-02-10T11:55:12.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 0x7f54bc19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 0x7f54bc096440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 0x7f54bc096e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc095ec0 con 0x7f54bc096a60 2021-02-10T11:55:12.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc097480 con 0x7f54bc19f850 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54c1539700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc097660 con 0x7f54bc0bf010 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54bbac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 0x7f54bc096e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54bbac0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 0x7f54bc096e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58618/0 (socket says 172.21.2.117:58618) 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54bbac0700 1 -- 172.21.2.117:0/303151647 learned_addr learned my addr 172.21.2.117:0/303151647 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54baabe700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 0x7f54bc096440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.484 7f54bb2bf700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 0x7f54bc19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 -- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 msgr2=0x7f54bc19f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 0x7f54bc19f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 -- 172.21.2.117:0/303151647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 msgr2=0x7f54bc096e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 0x7f54bc096e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 -- 172.21.2.117:0/303151647 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc094450 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54bb2bf700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 0x7f54bc19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54baabe700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 0x7f54bc096440 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f54a4000bd0 tx=0x7f54a400c8f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a4006240 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4000940 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54c1539700 1 -- 172.21.2.117:0/303151647 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc094450 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54c1539700 1 -- 172.21.2.117:0/303151647 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc0949e0 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.485 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f54a4007030 con 0x7f54bc19f850 2021-02-10T11:55:12.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.486 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f54a40066d0 con 0x7f54bc19f850 2021-02-10T11:55:12.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.486 7f54b3fff700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 0x7f54a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:12.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.486 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f54a4032300 con 0x7f54bc19f850 2021-02-10T11:55:12.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.486 7f54bb2bf700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 0x7f54a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:12.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.486 7f54bb2bf700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 0x7f54a001abc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:12.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.516 7f54c1539700 1 -- 172.21.2.117:0/303151647 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc0ede60 con 0x7f54bc19f850 2021-02-10T11:55:12.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.520 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f54a40072f0 con 0x7f54bc19f850 2021-02-10T11:55:12.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:12.650 7f54c1539700 1 -- 172.21.2.117:0/303151647 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f54bc0ede60 con 0x7f54bc19f850 2021-02-10T11:55:13.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.058 7f54b3fff700 1 -- 172.21.2.117:0/303151647 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v167) v1 ==== 133+0+0 (secure 0 0 0) 0x7f54a40066d0 con 0x7f54bc19f850 2021-02-10T11:55:13.067 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 msgr2=0x7f54a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 0x7f54a001abc0 crc :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 msgr2=0x7f54bc096440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 0x7f54bc096440 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f54a4000bd0 tx=0x7f54a400c8f0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 shutdown_connections 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54a00186c0 0x7f54a001abc0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f54bc096a60 0x7f54bc096e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54bc19f850 0x7f54bc096440 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 --2- 172.21.2.117:0/303151647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54bc0bf010 0x7f54bc19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 >> 172.21.2.117:0/303151647 conn(0x7f54bc0ae940 msgr2=0x7f54bc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:13.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 shutdown_connections 2021-02-10T11:55:13.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.067 7f54c1539700 1 -- 172.21.2.117:0/303151647 wait complete. 2021-02-10T11:55:13.077 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.491 7ff734022700 1 Processor -- start 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.491 7ff734022700 1 -- start start 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff734022700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 0x7ff72c1b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff734022700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0ede60 con 0x7ff72c0ef3f0 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff734022700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72c0eba00 con 0x7ff72c0d18f0 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff734022700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c0d1cf0 con 0x7ff72c1b69a0 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff73163d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff72c0ef3f0 0x7ff72c0eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58872/0 (socket says 172.21.2.117:58872) 2021-02-10T11:55:13.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff73163d700 1 -- 172.21.2.117:0/19024551 learned_addr learned my addr 172.21.2.117:0/19024551 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff73263f700 1 --2- 172.21.2.117:0/19024551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 0x7ff72c1b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff730e3c700 1 -- 172.21.2.117:0/19024551 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512525605 0 0) 0x7ff71c001180 con 0x7ff72c0ef3f0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.492 7ff730e3c700 1 -- 172.21.2.117:0/19024551 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718002fe0 con 0x7ff72c0ef3f0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff730e3c700 1 -- 172.21.2.117:0/19024551 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668836765 0 0) 0x7ff728001080 con 0x7ff72c0d18f0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff730e3c700 1 -- 172.21.2.117:0/19024551 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718003210 con 0x7ff72c0d18f0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff73263f700 1 -- 172.21.2.117:0/19024551 >> v1:172.21.2.125:6789/0 conn(0x7ff72c0d18f0 legacy=0x7ff72c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff73263f700 1 -- 172.21.2.117:0/19024551 >> v1:172.21.2.117:6789/0 conn(0x7ff72c0ef3f0 legacy=0x7ff72c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff73263f700 1 -- 172.21.2.117:0/19024551 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c1bb3f0 con 0x7ff72c1b69a0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff73263f700 1 --2- 172.21.2.117:0/19024551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 0x7ff72c1b8db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff724006e50 tx=0x7ff724009df0).ready entity=mon.2 client_cookie=78261f4fbd0cc09d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff730e3c700 1 -- 172.21.2.117:0/19024551 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72400b510 con 0x7ff72c1b69a0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff730e3c700 1 -- 172.21.2.117:0/19024551 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724006790 con 0x7ff72c1b69a0 2021-02-10T11:55:13.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff730e3c700 1 -- 172.21.2.117:0/19024551 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff72400b750 con 0x7ff72c1b69a0 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff734022700 1 -- 172.21.2.117:0/19024551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 msgr2=0x7ff72c1b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.493 7ff734022700 1 --2- 172.21.2.117:0/19024551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 0x7ff72c1b8db0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7ff724006e50 tx=0x7ff724009df0).stop 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.494 7ff734022700 1 -- 172.21.2.117:0/19024551 shutdown_connections 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.494 7ff734022700 1 --2- 172.21.2.117:0/19024551 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c1b69a0 0x7ff72c1b8db0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.494 7ff734022700 1 -- 172.21.2.117:0/19024551 >> 172.21.2.117:0/19024551 conn(0x7ff72c0cf060 msgr2=0x7ff72c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.494 7ff734022700 1 -- 172.21.2.117:0/19024551 shutdown_connections 2021-02-10T11:55:13.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.494 7ff734022700 1 -- 172.21.2.117:0/19024551 wait complete. 2021-02-10T11:55:13.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 Processor -- start 2021-02-10T11:55:13.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 -- start start 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c0a67e0 0x7ff72c0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff72c1a2e50 0x7ff72c1a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c07f890 con 0x7ff72c07e7d0 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c07f310 con 0x7ff72c1a2e50 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff734022700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72c1a3870 con 0x7ff72c0a67e0 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff73163d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff73163d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58632/0 (socket says 172.21.2.117:58632) 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.509 7ff73163d700 1 -- 172.21.2.117:0/402702460 learned_addr learned my addr 172.21.2.117:0/402702460 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 -- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c0a67e0 msgr2=0x7ff72c0a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c0a67e0 0x7ff72c0a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:13.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 -- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff72c1a2e50 msgr2=0x7ff72c1a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff72c1a2e50 0x7ff72c1a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 -- 172.21.2.117:0/402702460 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c1a3a90 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff73163d700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7ff71c00a5d0 tx=0x7ff71c00a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c000ce0 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c0043a0 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff734022700 1 -- 172.21.2.117:0/402702460 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff72c1a3a90 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.510 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff71c0048c0 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.511 7ff734022700 1 -- 172.21.2.117:0/402702460 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff72c1a3a90 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.511 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff71c004580 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.511 7ff7227fc700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 0x7ff71401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.511 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5454+0+0 (secure 0 0 0) 0x7ff71c0324c0 con 0x7ff72c07e7d0 2021-02-10T11:55:13.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.512 7ff731e3e700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 0x7ff71401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:13.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.512 7ff731e3e700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 0x7ff71401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:13.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.542 7ff734022700 1 -- 172.21.2.117:0/402702460 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72c0ede60 con 0x7ff72c07e7d0 2021-02-10T11:55:13.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.544 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff71c000ce0 con 0x7ff72c07e7d0 2021-02-10T11:55:13.675 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:13.674 7ff734022700 1 -- 172.21.2.117:0/402702460 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7ff72c0ede60 con 0x7ff72c07e7d0 2021-02-10T11:55:14.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.061 7ff7227fc700 1 -- 172.21.2.117:0/402702460 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v168) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff71c004580 con 0x7ff72c07e7d0 2021-02-10T11:55:14.091 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:55:14.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 msgr2=0x7ff71401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 0x7ff71401abc0 crc :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 msgr2=0x7ff72c07ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7ff71c00a5d0 tx=0x7ff71c00a970).stop 2021-02-10T11:55:14.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 shutdown_connections 2021-02-10T11:55:14.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff7140186c0 0x7ff71401abc0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff72c07e7d0 0x7ff72c07ebd0 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff72c1a2e50 0x7ff72c1a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:14.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 --2- 172.21.2.117:0/402702460 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff72c0a67e0 0x7ff72c0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:14.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 >> 172.21.2.117:0/402702460 conn(0x7ff72c09e660 msgr2=0x7ff72c0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:14.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 shutdown_connections 2021-02-10T11:55:14.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.092 7ff734022700 1 -- 172.21.2.117:0/402702460 wait complete. 2021-02-10T11:55:14.103 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.517 7f7364ff6700 1 Processor -- start 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.517 7f7364ff6700 1 -- start start 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.518 7f7364ff6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 0x7f7360086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.518 7f7364ff6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7360084c60 0x7f7360085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.518 7f7364ff6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7360085930 con 0x7f73600e4e40 2021-02-10T11:55:14.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.518 7f7364ff6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7360085680 con 0x7f7360084c60 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.518 7f7364ff6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73601b09e0 con 0x7f73600e5240 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735ed9d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f73600e4e40 0x7f73601b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51814/0 (socket says 172.21.2.117:51814) 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735ed9d700 1 -- 172.21.2.117:0/629225677 learned_addr learned my addr 172.21.2.117:0/629225677 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735e59c700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 0x7f7360086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735f59e700 1 --2- 172.21.2.117:0/629225677 >> v2:172.21.2.125:3300/0 conn(0x7f7360084c60 0x7f7360085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735f59e700 1 -- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 msgr2=0x7f7360086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735f59e700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 0x7f7360086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735f59e700 1 -- 172.21.2.117:0/629225677 >> v1:172.21.2.125:6789/0 conn(0x7f73600e4e40 legacy=0x7f73601b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.519 7f735dd9b700 1 -- 172.21.2.117:0/629225677 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342679597 0 0) 0x7f7350001190 con 0x7f73600e4e40 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735f59e700 1 -- 172.21.2.117:0/629225677 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7360084840 con 0x7f7360084c60 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735e59c700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 0x7f7360086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:14.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735f59e700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7360084c60 0x7f7360085060 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f7358006e50 tx=0x7f7358009e70).ready entity=mon.1 client_cookie=5de377d00a11baad server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735dd9b700 1 -- 172.21.2.117:0/629225677 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7358006a20 con 0x7f7360084c60 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735dd9b700 1 -- 172.21.2.117:0/629225677 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7358006790 con 0x7f7360084c60 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f735dd9b700 1 -- 172.21.2.117:0/629225677 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f735800b700 con 0x7f7360084c60 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f7364ff6700 1 -- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7360084c60 msgr2=0x7f7360085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.520 7f7364ff6700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7360084c60 0x7f7360085060 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7f7358006e50 tx=0x7f7358009e70).stop 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 -- 172.21.2.117:0/629225677 shutdown_connections 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7360084c60 0x7f7360085060 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 --2- 172.21.2.117:0/629225677 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73600e5240 0x7f7360086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 -- 172.21.2.117:0/629225677 >> 172.21.2.117:0/629225677 conn(0x7f73600ce4e0 msgr2=0x7f73600e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 -- 172.21.2.117:0/629225677 shutdown_connections 2021-02-10T11:55:14.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.521 7f7364ff6700 1 -- 172.21.2.117:0/629225677 wait complete. 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 Processor -- start 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 -- start start 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 0x7f736008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736008a8a0 0x7f736008aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73601a7a80 con 0x7f73601a7060 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73601a7ca0 con 0x7f73600abe50 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f7364ff6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73601a7ec0 con 0x7f736008a8a0 2021-02-10T11:55:14.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f735f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f735f59e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58644/0 (socket says 172.21.2.117:58644) 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.536 7f735f59e700 1 -- 172.21.2.117:0/244245456 learned_addr learned my addr 172.21.2.117:0/244245456 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735e59c700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736008a8a0 0x7f736008aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735ed9d700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 0x7f736008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 -- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736008a8a0 msgr2=0x7f736008aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736008a8a0 0x7f736008aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 -- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 msgr2=0x7f736008a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 0x7f736008a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 -- 172.21.2.117:0/244245456 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600b7030 con 0x7f73601a7060 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735ed9d700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 0x7f736008a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f735f59e700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f7358006e60 tx=0x7f735800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:14.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7358000df0 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73580091c0 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7358001390 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f7364ff6700 1 -- 172.21.2.117:0/244245456 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73600b7030 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.537 7f7364ff6700 1 -- 172.21.2.117:0/244245456 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73600b75c0 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.538 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73580011a0 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.538 7f734f7fe700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 0x7f734401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.538 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f7358032980 con 0x7f73601a7060 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.539 7f735ed9d700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 0x7f734401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:14.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.539 7f735ed9d700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 0x7f734401abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:14.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.568 7f7364ff6700 1 -- 172.21.2.117:0/244245456 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736010c2b0 con 0x7f73601a7060 2021-02-10T11:55:14.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.570 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73580093d0 con 0x7f73601a7060 2021-02-10T11:55:14.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.698 7f7364ff6700 1 -- 172.21.2.117:0/244245456 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f7360085930 con 0x7f73601a7060 2021-02-10T11:55:14.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.702 7f734f7fe700 1 -- 172.21.2.117:0/244245456 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 36 and data pool 37 v141) v1 ==== 189+0+0 (secure 0 0 0) 0x7f735801f5e0 con 0x7f73601a7060 2021-02-10T11:55:14.708 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 36 and data pool 37 2021-02-10T11:55:14.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 msgr2=0x7f734401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 0x7f734401abc0 crc :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 msgr2=0x7f73601a7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:14.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f7358006e60 tx=0x7f735800bcf0).stop 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 shutdown_connections 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73440186c0 0x7f734401abc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73601a7060 0x7f73601a7460 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f736008a8a0 0x7f736008aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 --2- 172.21.2.117:0/244245456 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73600abe50 0x7f736008a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 >> 172.21.2.117:0/244245456 conn(0x7f73600acec0 msgr2=0x7f73600b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 shutdown_connections 2021-02-10T11:55:14.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:14.707 7f7364ff6700 1 -- 172.21.2.117:0/244245456 wait complete. 2021-02-10T11:55:14.717 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.126 7f32990d2700 1 Processor -- start 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.126 7f32990d2700 1 -- start start 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f32990d2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32940d18e0 0x7f32940ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f32990d2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32940eb000 0x7f32940eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f32990d2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940ede60 con 0x7f32940ee3e0 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f32990d2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32940eba20 con 0x7f32940eb000 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f32990d2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32940d1ce0 con 0x7f32940d18e0 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329259c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f32940eb000 0x7f32940eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329359e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f32940ee3e0 0x7f32940ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58904/0 (socket says 172.21.2.117:58904) 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329359e700 1 -- 172.21.2.117:0/3151576923 learned_addr learned my addr 172.21.2.117:0/3151576923 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f3292d9d700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32940d18e0 0x7f32940ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329259c700 1 -- 172.21.2.117:0/3151576923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32940d18e0 msgr2=0x7f32940ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329259c700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32940d18e0 0x7f32940ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329259c700 1 -- 172.21.2.117:0/3151576923 >> v1:172.21.2.117:6789/0 conn(0x7f32940ee3e0 legacy=0x7f32940ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.127 7f329259c700 1 -- 172.21.2.117:0/3151576923 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940eefd0 con 0x7f32940eb000 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f329259c700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940eb000 0x7f32940eb400 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f3288006650 tx=0x7f3288009a80).ready entity=mon.0 client_cookie=3fb57fd9cae447aa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f3291d9b700 1 -- 172.21.2.117:0/3151576923 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f328800c0c0 con 0x7f32940eb000 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f3291d9b700 1 -- 172.21.2.117:0/3151576923 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328800c340 con 0x7f32940eb000 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f3291d9b700 1 -- 172.21.2.117:0/3151576923 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f328800c860 con 0x7f32940eb000 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f32990d2700 1 -- 172.21.2.117:0/3151576923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940eb000 msgr2=0x7f32940eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.128 7f32990d2700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940eb000 0x7f32940eb400 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7f3288006650 tx=0x7f3288009a80).stop 2021-02-10T11:55:15.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 -- 172.21.2.117:0/3151576923 shutdown_connections 2021-02-10T11:55:15.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940eb000 0x7f32940eb400 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 --2- 172.21.2.117:0/3151576923 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32940d18e0 0x7f32940ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 -- 172.21.2.117:0/3151576923 >> 172.21.2.117:0/3151576923 conn(0x7f32940ced60 msgr2=0x7f32940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:15.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 -- 172.21.2.117:0/3151576923 shutdown_connections 2021-02-10T11:55:15.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.129 7f32990d2700 1 -- 172.21.2.117:0/3151576923 wait complete. 2021-02-10T11:55:15.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.143 7f32990d2700 1 Processor -- start 2021-02-10T11:55:15.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.143 7f32990d2700 1 -- start start 2021-02-10T11:55:15.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32941a33d0 0x7f32941a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32940b82f0 0x7f32940b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32940c0670 con 0x7f32940b8d10 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32940deb20 con 0x7f32940b82f0 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f32990d2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32940ded40 con 0x7f32941a33d0 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58660/0 (socket says 172.21.2.117:58660) 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 -- 172.21.2.117:0/126766956 learned_addr learned my addr 172.21.2.117:0/126766956 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 -- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32941a33d0 msgr2=0x7f32941a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32941a33d0 0x7f32941a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 -- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32940b82f0 msgr2=0x7f32940b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32940b82f0 0x7f32940b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.144 7f329359e700 1 -- 172.21.2.117:0/126766956 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940def60 con 0x7f32940b8d10 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f329359e700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f328c00a940 tx=0x7f328c00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f328c001270 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c0070b0 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f328c007540 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f32990d2700 1 -- 172.21.2.117:0/126766956 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32940def60 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.145 7f32990d2700 1 -- 172.21.2.117:0/126766956 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32940df4f0 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.146 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f328c000d30 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.146 7f32837fe700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 0x7f327801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.146 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f328c032a40 con 0x7f32940b8d10 2021-02-10T11:55:15.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.146 7f3292d9d700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 0x7f327801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.147 7f3292d9d700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 0x7f327801abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.176 7f32990d2700 1 -- 172.21.2.117:0/126766956 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3294087290 con 0x7f32940b8d10 2021-02-10T11:55:15.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.179 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f328c014cd0 con 0x7f32940b8d10 2021-02-10T11:55:15.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.304 7f32990d2700 1 -- 172.21.2.117:0/126766956 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32940ede60 con 0x7f32940b8d10 2021-02-10T11:55:15.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.305 7f32837fe700 1 -- 172.21.2.117:0/126766956 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f328c014cd0 con 0x7f32940b8d10 2021-02-10T11:55:15.307 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:15.307 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":169,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:14.699245","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:55:12.669422","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:55:13.692093","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:15.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.306 7f32990d2700 1 -- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 msgr2=0x7f327801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.306 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 0x7f327801abc0 crc :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.306 7f32990d2700 1 -- 172.21.2.117:0/126766956 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 msgr2=0x7f32940b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.306 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f328c00a940 tx=0x7f328c00ace0).stop 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 -- 172.21.2.117:0/126766956 shutdown_connections 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32780186c0 0x7f327801abc0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32940b8d10 0x7f32940b9110 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32940b82f0 0x7f32940b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 --2- 172.21.2.117:0/126766956 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f32941a33d0 0x7f32941a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 -- 172.21.2.117:0/126766956 >> 172.21.2.117:0/126766956 conn(0x7f32940874c0 msgr2=0x7f3294096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 -- 172.21.2.117:0/126766956 shutdown_connections 2021-02-10T11:55:15.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.307 7f32990d2700 1 -- 172.21.2.117:0/126766956 wait complete. 2021-02-10T11:55:15.317 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:15.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.726 7f5f6aefa700 1 Processor -- start 2021-02-10T11:55:15.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.726 7f5f6aefa700 1 -- start start 2021-02-10T11:55:15.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f6aefa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 0x7f5f640ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f6aefa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 0x7f5f640eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f6aefa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640ede60 con 0x7f5f640ee3e0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f6aefa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640eba90 con 0x7f5f640d18e0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f6aefa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640d1ce0 con 0x7f5f640eb000 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f69517700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f5f640ee3e0 0x7f5f640ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:51844/0 (socket says 172.21.2.117:51844) 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f69517700 1 -- 172.21.2.117:0/2052399922 learned_addr learned my addr 172.21.2.117:0/2052399922 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f68d16700 1 --2- 172.21.2.117:0/2052399922 >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 0x7f5f640ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f63fff700 1 --2- 172.21.2.117:0/2052399922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 0x7f5f640eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f637fe700 1 -- 172.21.2.117:0/2052399922 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810732524 0 0) 0x7f5f5c001440 con 0x7f5f640ee3e0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f637fe700 1 -- 172.21.2.117:0/2052399922 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f50002fe0 con 0x7f5f640ee3e0 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f63fff700 1 -- 172.21.2.117:0/2052399922 >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 msgr2=0x7f5f640ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f63fff700 1 --2- 172.21.2.117:0/2052399922 >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 0x7f5f640ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f63fff700 1 -- 172.21.2.117:0/2052399922 >> v1:172.21.2.125:6789/0 conn(0x7f5f640ee3e0 legacy=0x7f5f640ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f63fff700 1 -- 172.21.2.117:0/2052399922 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640eefd0 con 0x7f5f640eb000 2021-02-10T11:55:15.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.727 7f5f68d16700 1 --2- 172.21.2.117:0/2052399922 >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 0x7f5f640ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f63fff700 1 --2- 172.21.2.117:0/2052399922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 0x7f5f640eb400 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f5f54006650 tx=0x7f5f54009a80).ready entity=mon.2 client_cookie=84855f888e72ca6f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f637fe700 1 -- 172.21.2.117:0/2052399922 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f5400c0c0 con 0x7f5f640eb000 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f637fe700 1 -- 172.21.2.117:0/2052399922 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5400c340 con 0x7f5f640eb000 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 -- 172.21.2.117:0/2052399922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 msgr2=0x7f5f640eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 --2- 172.21.2.117:0/2052399922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 0x7f5f640eb400 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f5f54006650 tx=0x7f5f54009a80).stop 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f637fe700 1 -- 172.21.2.117:0/2052399922 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f5400c860 con 0x7f5f640eb000 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 -- 172.21.2.117:0/2052399922 shutdown_connections 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 --2- 172.21.2.117:0/2052399922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f640eb000 0x7f5f640eb400 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 --2- 172.21.2.117:0/2052399922 >> v2:172.21.2.125:3300/0 conn(0x7f5f640d18e0 0x7f5f640ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 -- 172.21.2.117:0/2052399922 >> 172.21.2.117:0/2052399922 conn(0x7f5f640ced60 msgr2=0x7f5f640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 -- 172.21.2.117:0/2052399922 shutdown_connections 2021-02-10T11:55:15.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.728 7f5f6aefa700 1 -- 172.21.2.117:0/2052399922 wait complete. 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 Processor -- start 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 -- start start 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f64095850 0x7f5f64095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f64096270 0x7f5f64196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f640967f0 con 0x7f5f64091b00 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f641972b0 con 0x7f5f64095850 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.743 7f5f6aefa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64197490 con 0x7f5f64096270 2021-02-10T11:55:15.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58672/0 (socket says 172.21.2.117:58672) 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 -- 172.21.2.117:0/670633731 learned_addr learned my addr 172.21.2.117:0/670633731 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f63fff700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f64095850 0x7f5f64095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 -- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f64096270 msgr2=0x7f5f64196c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f64096270 0x7f5f64196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 -- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f64095850 msgr2=0x7f5f64095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f64095850 0x7f5f64095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 -- 172.21.2.117:0/670633731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f641976b0 con 0x7f5f64091b00 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f68d16700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f5f4c006f40 tx=0x7f5f4c00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f4c00bd80 con 0x7f5f64091b00 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c000c90 con 0x7f5f64091b00 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5f4c002220 con 0x7f5f64091b00 2021-02-10T11:55:15.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f641976b0 con 0x7f5f64091b00 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.744 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f641976b0 con 0x7f5f64091b00 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.745 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5f4c006b00 con 0x7f5f64091b00 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.745 7f5f617fa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 0x7f5f4801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.745 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5f4c032740 con 0x7f5f64091b00 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.745 7f5f63fff700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 0x7f5f4801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:15.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.745 7f5f63fff700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 0x7f5f4801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:15.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.776 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f64087290 con 0x7f5f64091b00 2021-02-10T11:55:15.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.779 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5f4c01f4b0 con 0x7f5f64091b00 2021-02-10T11:55:15.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.901 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f640ede60 con 0x7f5f64091b00 2021-02-10T11:55:15.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.902 7f5f617fa700 1 -- 172.21.2.117:0/670633731 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f5f4c006b00 con 0x7f5f64091b00 2021-02-10T11:55:15.904 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:15.904 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:15.701452","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:55:12.669422","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:55:13.692093","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 msgr2=0x7f5f4801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 0x7f5f4801abc0 crc :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 msgr2=0x7f5f64091f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7f5f4c006f40 tx=0x7f5f4c00b990).stop 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 shutdown_connections 2021-02-10T11:55:15.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5f64096270 0x7f5f64196c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5f64095850 0x7f5f64095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5f48018730 0x7f5f4801abc0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 --2- 172.21.2.117:0/670633731 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5f64091b00 0x7f5f64091f00 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.903 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 >> 172.21.2.117:0/670633731 conn(0x7f5f641a3620 msgr2=0x7f5f64092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.904 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 shutdown_connections 2021-02-10T11:55:15.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:15.904 7f5f6aefa700 1 -- 172.21.2.117:0/670633731 wait complete. 2021-02-10T11:55:15.914 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:55:16.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 Processor -- start 2021-02-10T11:55:16.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 -- start start 2021-02-10T11:55:16.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47040d18f0 0x7f47040eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47040eb040 0x7f47040eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47040eba60 con 0x7f47040ee3e0 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47040d1cf0 con 0x7f47040d18f0 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f470ad82700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47040eefd0 con 0x7f47040eb040 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f4708b9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47040d18f0 0x7f47040eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f4708b9e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f47040d18f0 0x7f47040eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58680/0 (socket says 172.21.2.117:58680) 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.328 7f4708b9e700 1 -- 172.21.2.117:0/856956490 learned_addr learned my addr 172.21.2.117:0/856956490 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f4708b9e700 1 -- 172.21.2.117:0/856956490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47040eb040 msgr2=0x7f47040eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f4708b9e700 1 --2- 172.21.2.117:0/856956490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47040eb040 0x7f47040eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f4708b9e700 1 -- 172.21.2.117:0/856956490 >> v1:172.21.2.125:6789/0 conn(0x7f47040ee3e0 legacy=0x7f47040ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f4708b9e700 1 -- 172.21.2.117:0/856956490 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47040ef1f0 con 0x7f47040d18f0 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f4708b9e700 1 --2- 172.21.2.117:0/856956490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47040d18f0 0x7f47040eaa20 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f46ec006700 tx=0x7f46ec009b80).ready entity=mon.0 client_cookie=89d56f960fce5c49 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:16.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f47037fe700 1 -- 172.21.2.117:0/856956490 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec00c0c0 con 0x7f47040d18f0 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f47037fe700 1 -- 172.21.2.117:0/856956490 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec00c3c0 con 0x7f47040d18f0 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f47037fe700 1 -- 172.21.2.117:0/856956490 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46ec00c8e0 con 0x7f47040d18f0 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f470ad82700 1 -- 172.21.2.117:0/856956490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47040d18f0 msgr2=0x7f47040eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.329 7f470ad82700 1 --2- 172.21.2.117:0/856956490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47040d18f0 0x7f47040eaa20 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f46ec006700 tx=0x7f46ec009b80).stop 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 -- 172.21.2.117:0/856956490 shutdown_connections 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 --2- 172.21.2.117:0/856956490 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f47040eb040 0x7f47040eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 --2- 172.21.2.117:0/856956490 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f47040d18f0 0x7f47040eaa20 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 -- 172.21.2.117:0/856956490 >> 172.21.2.117:0/856956490 conn(0x7f47040cf060 msgr2=0x7f47040cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 -- 172.21.2.117:0/856956490 shutdown_connections 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.330 7f470ad82700 1 -- 172.21.2.117:0/856956490 wait complete. 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.344 7f470ad82700 1 Processor -- start 2021-02-10T11:55:16.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.344 7f470ad82700 1 -- start start 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47041a4240 0x7f47041a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f470409f790 0x7f47040b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470409fb90 con 0x7f470409ed70 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47040dc640 con 0x7f47041a4240 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f470ad82700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4704097270 con 0x7f470409f790 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58688/0 (socket says 172.21.2.117:58688) 2021-02-10T11:55:16.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 -- 172.21.2.117:0/2341774846 learned_addr learned my addr 172.21.2.117:0/2341774846 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 -- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f470409f790 msgr2=0x7f47040b8160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f470409f790 0x7f47040b8160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 -- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47041a4240 msgr2=0x7f47041a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47041a4240 0x7f47041a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 -- 172.21.2.117:0/2341774846 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47040b8780 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.345 7f4703fff700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f46f4006f00 tx=0x7f46f400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46f400bd30 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4000c90 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f470ad82700 1 -- 172.21.2.117:0/2341774846 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47040b8780 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f470ad82700 1 -- 172.21.2.117:0/2341774846 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47040b8d10 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46f40022a0 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.346 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f46f4005310 con 0x7f470409ed70 2021-02-10T11:55:16.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.347 7f47017fa700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 0x7f46e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:16.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.347 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f46f4032650 con 0x7f470409ed70 2021-02-10T11:55:16.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.347 7f4708b9e700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 0x7f46e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:16.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.347 7f4708b9e700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 0x7f46e801abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:16.380 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.378 7f470ad82700 1 -- 172.21.2.117:0/2341774846 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47040daff0 con 0x7f470409ed70 2021-02-10T11:55:16.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.381 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f46f4000940 con 0x7f470409ed70 2021-02-10T11:55:16.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.518 7f470ad82700 1 -- 172.21.2.117:0/2341774846 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f47040ede60 con 0x7f470409ed70 2021-02-10T11:55:16.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.710 7f47017fa700 1 -- 172.21.2.117:0/2341774846 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v144) v1 ==== 118+0+0 (secure 0 0 0) 0x7f46f4005310 con 0x7f470409ed70 2021-02-10T11:55:16.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 -- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 msgr2=0x7f46e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:16.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 0x7f46e801abc0 crc :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:16.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 -- 172.21.2.117:0/2341774846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 msgr2=0x7f470409f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:16.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f46f4006f00 tx=0x7f46f400b940).stop 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 -- 172.21.2.117:0/2341774846 shutdown_connections 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46e80186c0 0x7f46e801abc0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f470409ed70 0x7f470409f170 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f470409f790 0x7f47040b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 --2- 172.21.2.117:0/2341774846 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f47041a4240 0x7f47041a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.736 7f470ad82700 1 -- 172.21.2.117:0/2341774846 >> 172.21.2.117:0/2341774846 conn(0x7f4704087090 msgr2=0x7f470408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.737 7f470ad82700 1 -- 172.21.2.117:0/2341774846 shutdown_connections 2021-02-10T11:55:16.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:16.737 7f470ad82700 1 -- 172.21.2.117:0/2341774846 wait complete. 2021-02-10T11:55:16.746 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:17.176 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.157 7f8b4fa63700 1 Processor -- start 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.157 7f8b4fa63700 1 -- start start 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4fa63700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f8b480d18e0 0x7f8b480ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4fa63700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480ede60 con 0x7f8b480ef3f0 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4fa63700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480d1ce0 con 0x7f8b480eb000 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4fa63700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b480ef7f0 con 0x7f8b480d18e0 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4e080700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8b480ef3f0 0x7f8b481b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58942/0 (socket says 172.21.2.117:58942) 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4e080700 1 -- 172.21.2.117:0/1281512498 learned_addr learned my addr 172.21.2.117:0/1281512498 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265969583 0 0) 0x7f8b40001500 con 0x7f8b480ef3f0 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b34002fe0 con 0x7f8b480ef3f0 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 504093180 0 0) 0x7f8b40001460 con 0x7f8b480ef3f0 2021-02-10T11:55:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 >> v2:172.21.2.125:3300/0 conn(0x7f8b480d18e0 msgr2=0x7f8b480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 --2- 172.21.2.117:0/1281512498 >> v2:172.21.2.125:3300/0 conn(0x7f8b480d18e0 0x7f8b480ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 >> v1:172.21.2.125:6789/0 conn(0x7f8b480eb000 legacy=0x7f8b480eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b481bb180 con 0x7f8b480ef3f0 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8b40001980 con 0x7f8b480ef3f0 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 >> v1:172.21.2.117:6789/0 conn(0x7f8b480ef3f0 legacy=0x7f8b481b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 0x7f8b34005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b34006790 0x7f8b34008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b34009590 0x7f8b3400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3400c300 con 0x7f8b34003a60 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3400c520 con 0x7f8b34006790 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.158 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3400c740 con 0x7f8b34009590 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 0x7f8b34005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d07e700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b34006790 0x7f8b34008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4e080700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b34009590 0x7f8b3400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 -- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b34009590 msgr2=0x7f8b3400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b34009590 0x7f8b3400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 -- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b34006790 msgr2=0x7f8b34008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b34006790 0x7f8b34008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 -- 172.21.2.117:0/1281512498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3400c9e0 con 0x7f8b34003a60 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4e080700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b34009590 0x7f8b3400bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4e080700 1 -- 172.21.2.117:0/1281512498 reap_dead start 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4d87f700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 0x7f8b34005ef0 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f8b44006bf0 tx=0x7f8b4404fcd0).ready entity=mon.0 client_cookie=b10b00b5ce8e1967 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b440009b0 con 0x7f8b34003a60 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44002030 con 0x7f8b34003a60 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4c87d700 1 -- 172.21.2.117:0/1281512498 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b440009b0 con 0x7f8b34003a60 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4fa63700 1 -- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 msgr2=0x7f8b34005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.159 7f8b4fa63700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 0x7f8b34005ef0 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f8b44006bf0 tx=0x7f8b4404fcd0).stop 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.160 7f8b4fa63700 1 -- 172.21.2.117:0/1281512498 shutdown_connections 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.160 7f8b4fa63700 1 --2- 172.21.2.117:0/1281512498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b34003a60 0x7f8b34005ef0 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.160 7f8b4fa63700 1 -- 172.21.2.117:0/1281512498 >> 172.21.2.117:0/1281512498 conn(0x7f8b480ced60 msgr2=0x7f8b480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.160 7f8b4fa63700 1 -- 172.21.2.117:0/1281512498 shutdown_connections 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.160 7f8b4fa63700 1 -- 172.21.2.117:0/1281512498 wait complete. 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.174 7f8b4fa63700 1 Processor -- start 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 -- start start 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b480dde20 0x7f8b480be850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b480de220 0x7f8b481a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480befd0 con 0x7f8b481a1250 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480de620 con 0x7f8b480de220 2021-02-10T11:55:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4fa63700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480b9660 con 0x7f8b480dde20 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58706/0 (socket says 172.21.2.117:58706) 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 -- 172.21.2.117:0/3610856130 learned_addr learned my addr 172.21.2.117:0/3610856130 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 -- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b480dde20 msgr2=0x7f8b480be850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b480dde20 0x7f8b480be850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 -- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b480de220 msgr2=0x7f8b481a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b480de220 0x7f8b481a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.175 7f8b4e080700 1 -- 172.21.2.117:0/3610856130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b481a1650 con 0x7f8b481a1250 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b4e080700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f8b4000a940 tx=0x7f8b4000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b400070d0 con 0x7f8b481a1250 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b400073d0 con 0x7f8b481a1250 2021-02-10T11:55:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b400049a0 con 0x7f8b481a1250 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b481a1650 con 0x7f8b481a1250 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b481a1650 con 0x7f8b481a1250 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.176 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b40001f70 con 0x7f8b481a1250 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.177 7f8b3e7fc700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 0x7f8b3001eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.177 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b40032aa0 con 0x7f8b481a1250 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.177 7f8b4d87f700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 0x7f8b3001eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.177 7f8b4d87f700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 0x7f8b3001eff0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.208 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48087210 con 0x7f8b481a1250 2021-02-10T11:55:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.210 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b40026ff0 con 0x7f8b481a1250 2021-02-10T11:55:17.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.338 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b480ede60 con 0x7f8b481a1250 2021-02-10T11:55:17.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.339 7f8b3e7fc700 1 -- 172.21.2.117:0/3610856130 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f8b40026ff0 con 0x7f8b481a1250 2021-02-10T11:55:17.341 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:17.341 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:16.708700","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7844":{"gid":7844,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/321593523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":321593523},{"type":"v1","addr":"172.21.2.125:6835","nonce":321593523}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:17.341 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 144 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 msgr2=0x7f8b3001eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 0x7f8b3001eff0 crc :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 msgr2=0x7f8b480bf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f8b4000a940 tx=0x7f8b4000ace0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 shutdown_connections 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b3001cad0 0x7f8b3001eff0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b481a1250 0x7f8b480bf4d0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b480de220 0x7f8b481a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 --2- 172.21.2.117:0/3610856130 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b480dde20 0x7f8b480be850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 >> 172.21.2.117:0/3610856130 conn(0x7f8b480a5270 msgr2=0x7f8b4808a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:17.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 shutdown_connections 2021-02-10T11:55:17.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.341 7f8b4fa63700 1 -- 172.21.2.117:0/3610856130 wait complete. 2021-02-10T11:55:17.351 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:17.396 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:55:17.399 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:55:17.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.765 7f28db1fd700 1 Processor -- start 2021-02-10T11:55:17.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.765 7f28db1fd700 1 -- start start 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28db1fd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 0x7f28d40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28db1fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28d40eb7f0 0x7f28d41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28db1fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d40ede60 con 0x7f28d40d18e0 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28db1fd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d40ef7f0 con 0x7f28d40eb7f0 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28db1fd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d40d1ce0 con 0x7f28d40ef3f0 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28d40eb7f0 0x7f28d41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f28d40eb7f0 0x7f28d41b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58714/0 (socket says 172.21.2.117:58714) 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 -- 172.21.2.117:0/519686106 learned_addr learned my addr 172.21.2.117:0/519686106 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 -- 172.21.2.117:0/519686106 >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 msgr2=0x7f28d40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d8818700 1 --2- 172.21.2.117:0/519686106 >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 0x7f28d40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 --2- 172.21.2.117:0/519686106 >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 0x7f28d40eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 -- 172.21.2.117:0/519686106 >> v1:172.21.2.125:6789/0 conn(0x7f28d40d18e0 legacy=0x7f28d40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d981a700 1 -- 172.21.2.117:0/519686106 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d41b8f90 con 0x7f28d40eb7f0 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.766 7f28d8818700 1 --2- 172.21.2.117:0/519686106 >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 0x7f28d40eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28d981a700 1 --2- 172.21.2.117:0/519686106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d40eb7f0 0x7f28d41b7960 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f28cc006f00 tx=0x7f28cc009f10).ready entity=mon.0 client_cookie=9582112cc68e2a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28c7fff700 1 -- 172.21.2.117:0/519686106 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28cc00b3b0 con 0x7f28d40eb7f0 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28c7fff700 1 -- 172.21.2.117:0/519686106 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc00b630 con 0x7f28d40eb7f0 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28c7fff700 1 -- 172.21.2.117:0/519686106 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28cc00ba90 con 0x7f28d40eb7f0 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 -- 172.21.2.117:0/519686106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d40eb7f0 msgr2=0x7f28d41b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 --2- 172.21.2.117:0/519686106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d40eb7f0 0x7f28d41b7960 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f28cc006f00 tx=0x7f28cc009f10).stop 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 -- 172.21.2.117:0/519686106 shutdown_connections 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 --2- 172.21.2.117:0/519686106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d40eb7f0 0x7f28d41b7960 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 --2- 172.21.2.117:0/519686106 >> v2:172.21.2.125:3300/0 conn(0x7f28d40ef3f0 0x7f28d40eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.767 7f28db1fd700 1 -- 172.21.2.117:0/519686106 >> 172.21.2.117:0/519686106 conn(0x7f28d40ced60 msgr2=0x7f28d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:17.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.768 7f28db1fd700 1 -- 172.21.2.117:0/519686106 shutdown_connections 2021-02-10T11:55:17.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.768 7f28db1fd700 1 -- 172.21.2.117:0/519686106 wait complete. 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.782 7f28db1fd700 1 Processor -- start 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.782 7f28db1fd700 1 -- start start 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28d4190240 0x7f28d4094300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28d4094d20 0x7f28d40af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4095220 con 0x7f28d4094920 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4190680 con 0x7f28d4094d20 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40afe90 con 0x7f28d4190240 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58718/0 (socket says 172.21.2.117:58718) 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 -- 172.21.2.117:0/2690267984 learned_addr learned my addr 172.21.2.117:0/2690267984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 -- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28d4190240 msgr2=0x7f28d4094300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28d4190240 0x7f28d4094300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 -- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28d4094d20 msgr2=0x7f28d40af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28d4094d20 0x7f28d40af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 -- 172.21.2.117:0/2690267984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4095720 con 0x7f28d4094920 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28d8818700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f28c8006f20 tx=0x7f28c800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28c800bd80 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c8000c90 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.783 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d4095720 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.784 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4095cb0 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.784 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28c80022b0 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.784 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f28c800a000 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.784 7f28c5ffb700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 0x7f28b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.784 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f28c80326a0 con 0x7f28d4094920 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.785 7f28d9019700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 0x7f28b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:17.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.785 7f28d9019700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 0x7f28b401abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.815 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d4087210 con 0x7f28d4094920 2021-02-10T11:55:17.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.818 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f28c8000940 con 0x7f28d4094920 2021-02-10T11:55:17.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.942 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28d40ede60 con 0x7f28d4094920 2021-02-10T11:55:17.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.943 7f28c5ffb700 1 -- 172.21.2.117:0/2690267984 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f28c800a000 con 0x7f28d4094920 2021-02-10T11:55:17.945 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:17.946 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:15.701452","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:55:12.669422","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:55:13.692093","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:17.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.944 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 msgr2=0x7f28b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.944 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 0x7f28b401abc0 crc :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.944 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 msgr2=0x7f28d40af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:17.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.944 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7f28c8006f20 tx=0x7f28c800b990).stop 2021-02-10T11:55:17.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 shutdown_connections 2021-02-10T11:55:17.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28b40186c0 0x7f28b401abc0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28d4094920 0x7f28d40af250 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:17.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f28d4094d20 0x7f28d40af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.947 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 --2- 172.21.2.117:0/2690267984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28d4190240 0x7f28d4094300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:17.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 >> 172.21.2.117:0/2690267984 conn(0x7f28d41a0330 msgr2=0x7f28d40dd890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:17.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 shutdown_connections 2021-02-10T11:55:17.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:17.945 7f28db1fd700 1 -- 172.21.2.117:0/2690267984 wait complete. 2021-02-10T11:55:17.955 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:55:18.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 Processor -- start 2021-02-10T11:55:18.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 -- start start 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6da807ab40 0x7f6da80794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da80e5770 con 0x7f6da807af40 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da80d18c0 con 0x7f6da807ab40 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.377 7f6dad42a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da80d1c60 con 0x7f6da8079b10 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6da807ab40 0x7f6da80794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da687c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da787e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6da807af40 0x7f6da807b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58974/0 (socket says 172.21.2.117:58974) 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da687c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:56404/0 (socket says 172.21.2.117:56404) 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6da807ab40 0x7f6da80794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58724/0 (socket says 172.21.2.117:58724) 2021-02-10T11:55:18.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da787e700 1 -- 172.21.2.117:0/1147393326 learned_addr learned my addr 172.21.2.117:0/1147393326 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da607b700 1 -- 172.21.2.117:0/1147393326 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665190031 0 0) 0x7f6da0001500 con 0x7f6da807af40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da607b700 1 -- 172.21.2.117:0/1147393326 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94002fe0 con 0x7f6da807af40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 -- 172.21.2.117:0/1147393326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 msgr2=0x7f6da8079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 -- 172.21.2.117:0/1147393326 >> v1:172.21.2.117:6789/0 conn(0x7f6da807af40 legacy=0x7f6da807b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 -- 172.21.2.117:0/1147393326 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da807bb30 con 0x7f6da807ab40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da687c700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da707d700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da807ab40 0x7f6da80794f0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f6d90006650 tx=0x7f6d90009a80).ready entity=mon.0 client_cookie=a5e0ce69fc9e9ffb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da607b700 1 -- 172.21.2.117:0/1147393326 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9000c0c0 con 0x7f6da807ab40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.378 7f6da607b700 1 -- 172.21.2.117:0/1147393326 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9000c340 con 0x7f6da807ab40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6da607b700 1 -- 172.21.2.117:0/1147393326 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d9000c860 con 0x7f6da807ab40 2021-02-10T11:55:18.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 -- 172.21.2.117:0/1147393326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da807ab40 msgr2=0x7f6da80794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da807ab40 0x7f6da80794f0 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7f6d90006650 tx=0x7f6d90009a80).stop 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 -- 172.21.2.117:0/1147393326 shutdown_connections 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da8079b10 0x7f6da8079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 --2- 172.21.2.117:0/1147393326 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da807ab40 0x7f6da80794f0 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 -- 172.21.2.117:0/1147393326 >> 172.21.2.117:0/1147393326 conn(0x7f6da80d0910 msgr2=0x7f6da80d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 -- 172.21.2.117:0/1147393326 shutdown_connections 2021-02-10T11:55:18.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.379 7f6dad42a700 1 -- 172.21.2.117:0/1147393326 wait complete. 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.394 7f6dad42a700 1 Processor -- start 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.394 7f6dad42a700 1 -- start start 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da81a4930 0x7f6da8098270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6da80b0850 0x7f6da8098890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8098f30 con 0x7f6da80b02b0 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8099150 con 0x7f6da80b0850 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6dad42a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8099370 con 0x7f6da81a4930 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58730/0 (socket says 172.21.2.117:58730) 2021-02-10T11:55:18.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 -- 172.21.2.117:0/425601807 learned_addr learned my addr 172.21.2.117:0/425601807 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da687c700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da81a4930 0x7f6da8098270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da787e700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6da80b0850 0x7f6da8098890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 -- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da81a4930 msgr2=0x7f6da8098270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da81a4930 0x7f6da8098270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 -- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6da80b0850 msgr2=0x7f6da8098890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6da80b0850 0x7f6da8098890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 -- 172.21.2.117:0/425601807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da80b61d0 con 0x7f6da80b02b0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.395 7f6da707d700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 rx=0x7f6d9000a640 tx=0x7f6d9000c960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d90009600 con 0x7f6da80b02b0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9000cd50 con 0x7f6da80b02b0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6d90009940 con 0x7f6da80b02b0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6dad42a700 1 -- 172.21.2.117:0/425601807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da80b61d0 con 0x7f6da80b02b0 2021-02-10T11:55:18.402 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6dad42a700 1 -- 172.21.2.117:0/425601807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da80b6760 con 0x7f6da80b02b0 2021-02-10T11:55:18.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.396 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6d90000c90 con 0x7f6da80b02b0 2021-02-10T11:55:18.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.397 7f6d9ffff700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:18.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.397 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6d90032700 con 0x7f6da80b02b0 2021-02-10T11:55:18.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.397 7f6da687c700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:18.403 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.397 7f6da687c700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.427 7f6dad42a700 1 -- 172.21.2.117:0/425601807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da80daa20 con 0x7f6da80b02b0 2021-02-10T11:55:18.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.430 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6d90016a80 con 0x7f6da80b02b0 2021-02-10T11:55:18.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.559 7f6dad42a700 1 -- 172.21.2.117:0/425601807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6da807a5f0 con 0x7f6da80b02b0 2021-02-10T11:55:18.565 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.563 7f6d9ffff700 1 -- 172.21.2.117:0/425601807 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v93) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6d90000c90 con 0x7f6da80b02b0 2021-02-10T11:55:18.569 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 -- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 msgr2=0x7f6d8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 crc :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 -- 172.21.2.117:0/425601807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 msgr2=0x7f6da81a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 rx=0x7f6d9000a640 tx=0x7f6d9000c960).stop 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 -- 172.21.2.117:0/425601807 shutdown_connections 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6da80b0850 0x7f6da8098890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6da81a4930 0x7f6da8098270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6d8c0186c0 0x7f6d8c01abc0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 --2- 172.21.2.117:0/425601807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6da80b02b0 0x7f6da81a4310 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:18.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.568 7f6dad42a700 1 -- 172.21.2.117:0/425601807 >> 172.21.2.117:0/425601807 conn(0x7f6da80b0e40 msgr2=0x7f6da80a6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:18.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.569 7f6dad42a700 1 -- 172.21.2.117:0/425601807 shutdown_connections 2021-02-10T11:55:18.572 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.569 7f6dad42a700 1 -- 172.21.2.117:0/425601807 wait complete. 2021-02-10T11:55:18.579 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.992 7f7e76917700 1 Processor -- start 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.992 7f7e76917700 1 -- start start 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e76917700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e7007ab40 0x7f7e700794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e76917700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700e5770 con 0x7f7e70079b10 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e76917700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7007a310 con 0x7f7e70079f10 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e76917700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700d18c0 con 0x7f7e7007ab40 2021-02-10T11:55:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e6ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f7e7007ab40 0x7f7e700794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e6f7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f7e70079b10 0x7f7e7007af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:58988/0 (socket says 172.21.2.117:58988) 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e6f7fe700 1 -- 172.21.2.117:0/271343338 learned_addr learned my addr 172.21.2.117:0/271343338 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e6ffff700 1 -- 172.21.2.117:0/271343338 >> v1:172.21.2.125:6789/0 conn(0x7f7e70079f10 legacy=0x7f7e7007b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.993 7f7e6ffff700 1 -- 172.21.2.117:0/271343338 >> v1:172.21.2.117:6789/0 conn(0x7f7e70079b10 legacy=0x7f7e7007af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6effd700 1 -- 172.21.2.117:0/271343338 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622139293 0 0) 0x7f7e5c0009b0 con 0x7f7e70079b10 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6ffff700 1 -- 172.21.2.117:0/271343338 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7007bf20 con 0x7f7e7007ab40 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6ffff700 1 --2- 172.21.2.117:0/271343338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7007ab40 0x7f7e700794f0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f7e60006650 tx=0x7f7e60009a00).ready entity=mon.0 client_cookie=3b93848535755f80 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6effd700 1 -- 172.21.2.117:0/271343338 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e600009b0 con 0x7f7e7007ab40 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6effd700 1 -- 172.21.2.117:0/271343338 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000c0c0 con 0x7f7e7007ab40 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e6effd700 1 -- 172.21.2.117:0/271343338 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e6000c5a0 con 0x7f7e7007ab40 2021-02-10T11:55:19.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e76917700 1 -- 172.21.2.117:0/271343338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7007ab40 msgr2=0x7f7e700794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.994 7f7e76917700 1 --2- 172.21.2.117:0/271343338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7007ab40 0x7f7e700794f0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f7e60006650 tx=0x7f7e60009a00).stop 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.995 7f7e76917700 1 -- 172.21.2.117:0/271343338 shutdown_connections 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.995 7f7e76917700 1 --2- 172.21.2.117:0/271343338 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7007ab40 0x7f7e700794f0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.995 7f7e76917700 1 -- 172.21.2.117:0/271343338 >> 172.21.2.117:0/271343338 conn(0x7f7e700d0910 msgr2=0x7f7e700d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.995 7f7e76917700 1 -- 172.21.2.117:0/271343338 shutdown_connections 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:18.995 7f7e76917700 1 -- 172.21.2.117:0/271343338 wait complete. 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.010 7f7e76917700 1 Processor -- start 2021-02-10T11:55:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.010 7f7e76917700 1 -- start start 2021-02-10T11:55:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e701b3220 0x7f7e70198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e700b08e0 0x7f7e700b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e701903b0 con 0x7f7e7018f990 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70190c10 con 0x7f7e700b08e0 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e76917700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70190e30 con 0x7f7e701b3220 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58744/0 (socket says 172.21.2.117:58744) 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 -- 172.21.2.117:0/3868335030 learned_addr learned my addr 172.21.2.117:0/3868335030 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e6f7fe700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e700b08e0 0x7f7e700b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e6ffff700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e701b3220 0x7f7e70198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 -- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e701b3220 msgr2=0x7f7e70198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e701b3220 0x7f7e70198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 -- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e700b08e0 msgr2=0x7f7e700b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e700b08e0 0x7f7e700b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 -- 172.21.2.117:0/3868335030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70191050 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.011 7f7e74f34700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f7e68009940 tx=0x7f7e6800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e68000ce0 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6800c2f0 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7e6800c810 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e76917700 1 -- 172.21.2.117:0/3868335030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e70191050 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e76917700 1 -- 172.21.2.117:0/3868335030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70191050 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.012 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7e68001220 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.013 7f7e6cff9700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 0x7f7e4c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.013 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7e68032a30 con 0x7f7e7018f990 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.013 7f7e6ffff700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 0x7f7e4c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.013 7f7e6ffff700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 0x7f7e4c01abc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.043 7f7e76917700 1 -- 172.21.2.117:0/3868335030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e701903b0 con 0x7f7e7018f990 2021-02-10T11:55:19.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.045 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7e6800be30 con 0x7f7e7018f990 2021-02-10T11:55:19.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.175 7f7e76917700 1 -- 172.21.2.117:0/3868335030 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7e701903b0 con 0x7f7e7018f990 2021-02-10T11:55:19.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.178 7f7e6cff9700 1 -- 172.21.2.117:0/3868335030 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v94) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7e6801f850 con 0x7f7e7018f990 2021-02-10T11:55:19.185 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:55:19.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.183 7f7e76917700 1 -- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 msgr2=0x7f7e4c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.183 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 0x7f7e4c01abc0 crc :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.183 7f7e76917700 1 -- 172.21.2.117:0/3868335030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 msgr2=0x7f7e7018fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.183 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f7e68009940 tx=0x7f7e6800bcf0).stop 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 -- 172.21.2.117:0/3868335030 shutdown_connections 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7e4c0186c0 0x7f7e4c01abc0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7e7018f990 0x7f7e7018fd90 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7e700b08e0 0x7f7e700b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 --2- 172.21.2.117:0/3868335030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7e701b3220 0x7f7e70198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 -- 172.21.2.117:0/3868335030 >> 172.21.2.117:0/3868335030 conn(0x7f7e7009caa0 msgr2=0x7f7e701ad590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:19.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 -- 172.21.2.117:0/3868335030 shutdown_connections 2021-02-10T11:55:19.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.184 7f7e76917700 1 -- 172.21.2.117:0/3868335030 wait complete. 2021-02-10T11:55:19.195 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:55:19.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.617 7f907bb94700 1 Processor -- start 2021-02-10T11:55:19.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.617 7f907bb94700 1 -- start start 2021-02-10T11:55:19.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f907bb94700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 0x7f90740794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f907bb94700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9074079b10 0x7f9074079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f907bb94700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90740e5770 con 0x7f907407af40 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f907bb94700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740d18c0 con 0x7f9074079b10 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f907bb94700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740d1c60 con 0x7f907407ab40 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f90791af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9074079b10 0x7f9074079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f90791af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9074079b10 0x7f9074079f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58748/0 (socket says 172.21.2.117:58748) 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.618 7f90791af700 1 -- 172.21.2.117:0/2939465234 learned_addr learned my addr 172.21.2.117:0/2939465234 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90799b0700 1 --2- 172.21.2.117:0/2939465234 >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 0x7f90740794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90791af700 1 -- 172.21.2.117:0/2939465234 >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 msgr2=0x7f90740794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90791af700 1 --2- 172.21.2.117:0/2939465234 >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 0x7f90740794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90791af700 1 -- 172.21.2.117:0/2939465234 >> v1:172.21.2.125:6789/0 conn(0x7f907407af40 legacy=0x7f907407b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90791af700 1 -- 172.21.2.117:0/2939465234 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907407bb30 con 0x7f9074079b10 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90799b0700 1 --2- 172.21.2.117:0/2939465234 >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 0x7f90740794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90791af700 1 --2- 172.21.2.117:0/2939465234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074079b10 0x7f9074079f10 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f9064006700 tx=0x7f9064009b80).ready entity=mon.0 client_cookie=ab34cf2d834ae875 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90789ae700 1 -- 172.21.2.117:0/2939465234 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f906400c0c0 con 0x7f9074079b10 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90789ae700 1 -- 172.21.2.117:0/2939465234 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906400c400 con 0x7f9074079b10 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f90789ae700 1 -- 172.21.2.117:0/2939465234 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f906400c860 con 0x7f9074079b10 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f907bb94700 1 -- 172.21.2.117:0/2939465234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074079b10 msgr2=0x7f9074079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.619 7f907bb94700 1 --2- 172.21.2.117:0/2939465234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074079b10 0x7f9074079f10 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f9064006700 tx=0x7f9064009b80).stop 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 -- 172.21.2.117:0/2939465234 shutdown_connections 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 --2- 172.21.2.117:0/2939465234 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074079b10 0x7f9074079f10 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 --2- 172.21.2.117:0/2939465234 >> v2:172.21.2.125:3300/0 conn(0x7f907407ab40 0x7f90740794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 -- 172.21.2.117:0/2939465234 >> 172.21.2.117:0/2939465234 conn(0x7f90740d0910 msgr2=0x7f90740d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 -- 172.21.2.117:0/2939465234 shutdown_connections 2021-02-10T11:55:19.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.620 7f907bb94700 1 -- 172.21.2.117:0/2939465234 wait complete. 2021-02-10T11:55:19.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 Processor -- start 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 -- start start 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90740ae830 0x7f90740aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90740af250 0x7f9074191540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074194730 con 0x7f9074191b60 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90741b7970 con 0x7f90740af250 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907bb94700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90741b74d0 con 0x7f90740ae830 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58754/0 (socket says 172.21.2.117:58754) 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 -- 172.21.2.117:0/4044976384 learned_addr learned my addr 172.21.2.117:0/4044976384 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 -- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90740ae830 msgr2=0x7f90740aec30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:19.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90740ae830 0x7f90740aec30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 -- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90740af250 msgr2=0x7f9074191540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90740af250 0x7f9074191540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.635 7f907a1b1700 1 -- 172.21.2.117:0/4044976384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074194cb0 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f907a1b1700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 rx=0x7f906c009940 tx=0x7f906c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f906c000f90 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c00c2f0 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f906c00c810 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f907bb94700 1 -- 172.21.2.117:0/4044976384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9074194cb0 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f907bb94700 1 -- 172.21.2.117:0/4044976384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074194cb0 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.636 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f906c001220 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.637 7f906a7fc700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.637 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f906c032a80 con 0x7f9074191b60 2021-02-10T11:55:19.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.638 7f90799b0700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:19.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.639 7f90799b0700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 0x7f905c01abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:19.671 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.668 7f907bb94700 1 -- 172.21.2.117:0/4044976384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90740f6820 con 0x7f9074191b60 2021-02-10T11:55:19.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.671 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f906c00be30 con 0x7f9074191b60 2021-02-10T11:55:19.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.801 7f907bb94700 1 -- 172.21.2.117:0/4044976384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f90740f6820 con 0x7f9074191b60 2021-02-10T11:55:19.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.805 7f906a7fc700 1 -- 172.21.2.117:0/4044976384 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v95) v1 ==== 184+0+0 (secure 0 0 0) 0x7f906c001220 con 0x7f9074191b60 2021-02-10T11:55:19.811 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 msgr2=0x7f905c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 0x7f905c01abc0 crc :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 msgr2=0x7f9074191f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 rx=0x7f906c009940 tx=0x7f906c00bcf0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 shutdown_connections 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f905c0186c0 0x7f905c01abc0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9074191b60 0x7f9074191f60 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f90740af250 0x7f9074191540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 --2- 172.21.2.117:0/4044976384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f90740ae830 0x7f90740aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:19.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 >> 172.21.2.117:0/4044976384 conn(0x7f907419f6d0 msgr2=0x7f9074095d30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:19.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 shutdown_connections 2021-02-10T11:55:19.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:19.810 7f907bb94700 1 -- 172.21.2.117:0/4044976384 wait complete. 2021-02-10T11:55:19.821 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.234 7f0eb4ae2700 1 Processor -- start 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.234 7f0eb4ae2700 1 -- start start 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eb4ae2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb0079ce0 0x7f0eb01b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eb4ae2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0eb007a0e0 0x7f0eb01b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eb4ae2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00e5770 con 0x7f0eb007ab40 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eb4ae2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb00d18c0 con 0x7f0eb007a0e0 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eb4ae2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb007bf50 con 0x7f0eb0079ce0 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eae85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f0eb007ab40 0x7f0eb00794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59012/0 (socket says 172.21.2.117:59012) 2021-02-10T11:55:20.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eae85e700 1 -- 172.21.2.117:0/3200575212 learned_addr learned my addr 172.21.2.117:0/3200575212 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eaf05f700 1 --2- 172.21.2.117:0/3200575212 >> v2:172.21.2.117:3300/0 conn(0x7f0eb007a0e0 0x7f0eb01b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0ead85c700 1 -- 172.21.2.117:0/3200575212 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997665707 0 0) 0x7f0ea0001180 con 0x7f0eb007ab40 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0ead85c700 1 -- 172.21.2.117:0/3200575212 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e9c002fe0 con 0x7f0eb007ab40 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eaf05f700 1 -- 172.21.2.117:0/3200575212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb0079ce0 msgr2=0x7f0eb01b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eaf05f700 1 --2- 172.21.2.117:0/3200575212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb0079ce0 0x7f0eb01b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eaf05f700 1 -- 172.21.2.117:0/3200575212 >> v1:172.21.2.117:6789/0 conn(0x7f0eb007ab40 legacy=0x7f0eb00794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.235 7f0eaf05f700 1 -- 172.21.2.117:0/3200575212 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00d1c60 con 0x7f0eb007a0e0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eaf05f700 1 --2- 172.21.2.117:0/3200575212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb007a0e0 0x7f0eb01b8000 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f0ea8006e30 tx=0x7f0ea8009e60).ready entity=mon.0 client_cookie=2c691ef4213a7f62 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0ead85c700 1 -- 172.21.2.117:0/3200575212 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea8006a50 con 0x7f0eb007a0e0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0ead85c700 1 -- 172.21.2.117:0/3200575212 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea800b300 con 0x7f0eb007a0e0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0ead85c700 1 -- 172.21.2.117:0/3200575212 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea800b7e0 con 0x7f0eb007a0e0 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 -- 172.21.2.117:0/3200575212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb007a0e0 msgr2=0x7f0eb01b8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 --2- 172.21.2.117:0/3200575212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb007a0e0 0x7f0eb01b8000 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f0ea8006e30 tx=0x7f0ea8009e60).stop 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 -- 172.21.2.117:0/3200575212 shutdown_connections 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 --2- 172.21.2.117:0/3200575212 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb007a0e0 0x7f0eb01b8000 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 --2- 172.21.2.117:0/3200575212 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb0079ce0 0x7f0eb01b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.236 7f0eb4ae2700 1 -- 172.21.2.117:0/3200575212 >> 172.21.2.117:0/3200575212 conn(0x7f0eb00d0910 msgr2=0x7f0eb00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.237 7f0eb4ae2700 1 -- 172.21.2.117:0/3200575212 shutdown_connections 2021-02-10T11:55:20.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.237 7f0eb4ae2700 1 -- 172.21.2.117:0/3200575212 wait complete. 2021-02-10T11:55:20.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.251 7f0eb4ae2700 1 Processor -- start 2021-02-10T11:55:20.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.251 7f0eb4ae2700 1 -- start start 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0eb01009f0 0x7f0eb01b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb01b2b50 0x7f0eb0090310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00f8830 con 0x7f0eb01b2030 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00ac1b0 con 0x7f0eb01009f0 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eb4ae2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb00ac3d0 con 0x7f0eb01b2b50 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58768/0 (socket says 172.21.2.117:58768) 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 -- 172.21.2.117:0/2636796823 learned_addr learned my addr 172.21.2.117:0/2636796823 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 -- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb01b2b50 msgr2=0x7f0eb0090310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb01b2b50 0x7f0eb0090310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 -- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0eb01009f0 msgr2=0x7f0eb01b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0eb01009f0 0x7f0eb01b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 -- 172.21.2.117:0/2636796823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb0090930 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.252 7f0eae85e700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f0ea000a5d0 tx=0x7f0ea000a970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea0000ce0 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea00043a0 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ea00048c0 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb0090930 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb0090ec0 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ea0007070 con 0x7f0eb01b2030 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 0x7f0e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.253 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0ea00326a0 con 0x7f0eb01b2030 2021-02-10T11:55:20.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.254 7f0eae05d700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 0x7f0e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.254 7f0eae05d700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 0x7f0e9401abc0 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.285 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb0100ea0 con 0x7f0eb01b2030 2021-02-10T11:55:20.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.287 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ea0030d00 con 0x7f0eb01b2030 2021-02-10T11:55:20.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.416 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0eb0100ea0 con 0x7f0eb01b2030 2021-02-10T11:55:20.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.420 7f0ea77fe700 1 -- 172.21.2.117:0/2636796823 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v96) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0ea0007050 con 0x7f0eb01b2030 2021-02-10T11:55:20.426 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:55:20.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.425 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 msgr2=0x7f0e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.425 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 0x7f0e9401abc0 crc :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.425 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 msgr2=0x7f0eb00abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.425 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f0ea000a5d0 tx=0x7f0ea000a970).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 shutdown_connections 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0eb01b2b50 0x7f0eb0090310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0eb01009f0 0x7f0eb01b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0e940186c0 0x7f0e9401abc0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 --2- 172.21.2.117:0/2636796823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0eb01b2030 0x7f0eb00abb90 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 >> 172.21.2.117:0/2636796823 conn(0x7f0eb008f550 msgr2=0x7f0eb01afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 shutdown_connections 2021-02-10T11:55:20.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.426 7f0eb4ae2700 1 -- 172.21.2.117:0/2636796823 wait complete. 2021-02-10T11:55:20.437 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:20.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.846 7f27001ab700 1 Processor -- start 2021-02-10T11:55:20.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.846 7f27001ab700 1 -- start start 2021-02-10T11:55:20.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f27001ab700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f26f80d18e0 0x7f26f80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f27001ab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f26f80eb000 0x7f26f80eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f27001ab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f80ede60 con 0x7f26f80ee3e0 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f27001ab700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f80eba20 con 0x7f26f80eb000 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f27001ab700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f80d1ce0 con 0x7f26f80d18e0 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f26f80eb000 0x7f26f80eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f26f80eb000 0x7f26f80eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58776/0 (socket says 172.21.2.117:58776) 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 -- 172.21.2.117:0/2394757981 learned_addr learned my addr 172.21.2.117:0/2394757981 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 -- 172.21.2.117:0/2394757981 >> v2:172.21.2.125:3300/0 conn(0x7f26f80d18e0 msgr2=0x7f26f80ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 --2- 172.21.2.117:0/2394757981 >> v2:172.21.2.125:3300/0 conn(0x7f26f80d18e0 0x7f26f80ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 -- 172.21.2.117:0/2394757981 >> v1:172.21.2.125:6789/0 conn(0x7f26f80ee3e0 legacy=0x7f26f80ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 -- 172.21.2.117:0/2394757981 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f80eefd0 con 0x7f26f80eb000 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.847 7f26fd7c6700 1 --2- 172.21.2.117:0/2394757981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f80eb000 0x7f26f80eb400 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f26f4006700 tx=0x7f26f4009b00).ready entity=mon.0 client_cookie=8f768c5aa219405c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f26fcfc5700 1 -- 172.21.2.117:0/2394757981 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f400c0c0 con 0x7f26f80eb000 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f26fcfc5700 1 -- 172.21.2.117:0/2394757981 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f400c400 con 0x7f26f80eb000 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f26fcfc5700 1 -- 172.21.2.117:0/2394757981 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f400c8e0 con 0x7f26f80eb000 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 -- 172.21.2.117:0/2394757981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f80eb000 msgr2=0x7f26f80eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 --2- 172.21.2.117:0/2394757981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f80eb000 0x7f26f80eb400 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f26f4006700 tx=0x7f26f4009b00).stop 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 -- 172.21.2.117:0/2394757981 shutdown_connections 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 --2- 172.21.2.117:0/2394757981 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f80eb000 0x7f26f80eb400 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 --2- 172.21.2.117:0/2394757981 >> v2:172.21.2.125:3300/0 conn(0x7f26f80d18e0 0x7f26f80ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.848 7f27001ab700 1 -- 172.21.2.117:0/2394757981 >> 172.21.2.117:0/2394757981 conn(0x7f26f80ced60 msgr2=0x7f26f80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.849 7f27001ab700 1 -- 172.21.2.117:0/2394757981 shutdown_connections 2021-02-10T11:55:20.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.849 7f27001ab700 1 -- 172.21.2.117:0/2394757981 wait complete. 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 Processor -- start 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 -- start start 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26f80de0d0 0x7f26f80e8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26f81aad80 0x7f26f80b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80e01c0 con 0x7f26f81aa360 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80de6c0 con 0x7f26f80de0d0 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f27001ab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80b1d40 con 0x7f26f81aad80 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f26fd7c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f26fd7c6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58784/0 (socket says 172.21.2.117:58784) 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.863 7f26fd7c6700 1 -- 172.21.2.117:0/1172131573 learned_addr learned my addr 172.21.2.117:0/1172131573 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:20.869 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 -- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26f81aad80 msgr2=0x7f26f80b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26f81aad80 0x7f26f80b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 -- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26f80de0d0 msgr2=0x7f26f80e8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26f80de0d0 0x7f26f80e8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 -- 172.21.2.117:0/1172131573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f80b1f60 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26fd7c6700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f26f400b660 tx=0x7f26f400ba00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f4009440 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4000c90 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f26f4002030 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f27001ab700 1 -- 172.21.2.117:0/1172131573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f80b1f60 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.864 7f27001ab700 1 -- 172.21.2.117:0/1172131573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f80b1f60 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.865 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f26f40093e0 con 0x7f26f81aa360 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.865 7f26ee7fc700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 0x7f26e001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.865 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f26f4032570 con 0x7f26f81aa360 2021-02-10T11:55:20.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.865 7f26fdfc7700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 0x7f26e001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:20.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.866 7f26fdfc7700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 0x7f26e001abc0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:20.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.896 7f27001ab700 1 -- 172.21.2.117:0/1172131573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f8087210 con 0x7f26f81aa360 2021-02-10T11:55:20.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:20.899 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f26f40090c0 con 0x7f26f81aa360 2021-02-10T11:55:21.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.030 7f27001ab700 1 -- 172.21.2.117:0/1172131573 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26f80ede60 con 0x7f26f81aa360 2021-02-10T11:55:21.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.030 7f26ee7fc700 1 -- 172.21.2.117:0/1172131573 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f26f400bdf0 con 0x7f26f81aa360 2021-02-10T11:55:21.034 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:21.034 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:16.708700","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7844":{"gid":7844,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/321593523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":321593523},{"type":"v1","addr":"172.21.2.125:6835","nonce":321593523}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:21.034 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 144 2021-02-10T11:55:21.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 -- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 msgr2=0x7f26e001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:21.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 0x7f26e001abc0 crc :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:21.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 -- 172.21.2.117:0/1172131573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 msgr2=0x7f26f81aa760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f26f400b660 tx=0x7f26f400ba00).stop 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 -- 172.21.2.117:0/1172131573 shutdown_connections 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f26e00186c0 0x7f26e001abc0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f26f81aa360 0x7f26f81aa760 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f26f81aad80 0x7f26f80b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 --2- 172.21.2.117:0/1172131573 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f26f80de0d0 0x7f26f80e8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.032 7f27001ab700 1 -- 172.21.2.117:0/1172131573 >> 172.21.2.117:0/1172131573 conn(0x7f26f8087490 msgr2=0x7f26f80dd890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.033 7f27001ab700 1 -- 172.21.2.117:0/1172131573 shutdown_connections 2021-02-10T11:55:21.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:21.033 7f27001ab700 1 -- 172.21.2.117:0/1172131573 wait complete. 2021-02-10T11:55:21.245 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:55:14.699333', 'modified': '2021-02-10 11:55:16.708700', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7844}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7844': {'gid': 7844, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/321593523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 321593523}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 321593523}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:55:21.245 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:55:21.245 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:55:21.644 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:55:21.644 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:55:21.644 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:55:21.644 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7844, 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9406250189999996, 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 144, 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 170, 2021-02-10T11:55:21.645 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 170, 2021-02-10T11:55:21.646 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.594236134999999 2021-02-10T11:55:21.646 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:55:21.648 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7844, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9406250189999996, "mdsmap_epoch": 144, "osdmap_epoch": 170, "osdmap_epoch_barrier": 170, "uptime": 12.594236134999999 } 2021-02-10T11:55:21.649 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:22.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 Processor -- start 2021-02-10T11:55:22.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 -- start start 2021-02-10T11:55:22.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8fd41b6960 0x7f8fd41b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40ede60 con 0x7f8fd40d18e0 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd40eb9c0 con 0x7f8fd40ef3f0 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fda6d5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40ef7f0 con 0x7f8fd41b6960 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.058 7f8fd3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8fd40d18e0 0x7f8fd40ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59040/0 (socket says 172.21.2.117:59040) 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd3fff700 1 -- 172.21.2.117:0/3982674478 learned_addr learned my addr 172.21.2.117:0/3982674478 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd8cf2700 1 --2- 172.21.2.117:0/3982674478 >> v2:172.21.2.117:3300/0 conn(0x7f8fd41b6960 0x7f8fd41b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd8cf2700 1 -- 172.21.2.117:0/3982674478 >> v1:172.21.2.125:6789/0 conn(0x7f8fd40ef3f0 legacy=0x7f8fd40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd8cf2700 1 -- 172.21.2.117:0/3982674478 >> v1:172.21.2.117:6789/0 conn(0x7f8fd40d18e0 legacy=0x7f8fd40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd2ffd700 1 -- 172.21.2.117:0/3982674478 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189132899 0 0) 0x7f8fc4001180 con 0x7f8fd40d18e0 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd8cf2700 1 -- 172.21.2.117:0/3982674478 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40d1d20 con 0x7f8fd41b6960 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd8cf2700 1 --2- 172.21.2.117:0/3982674478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b6960 0x7f8fd41b8d70 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f8fcc006e30 tx=0x7f8fcc009de0).ready entity=mon.0 client_cookie=6c3de09d73d53d4d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd2ffd700 1 -- 172.21.2.117:0/3982674478 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fcc00b300 con 0x7f8fd41b6960 2021-02-10T11:55:22.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd2ffd700 1 -- 172.21.2.117:0/3982674478 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc00b580 con 0x7f8fd41b6960 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fd2ffd700 1 -- 172.21.2.117:0/3982674478 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fcc00baa0 con 0x7f8fd41b6960 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fda6d5700 1 -- 172.21.2.117:0/3982674478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b6960 msgr2=0x7f8fd41b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.059 7f8fda6d5700 1 --2- 172.21.2.117:0/3982674478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b6960 0x7f8fd41b8d70 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f8fcc006e30 tx=0x7f8fcc009de0).stop 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.060 7f8fda6d5700 1 -- 172.21.2.117:0/3982674478 shutdown_connections 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.060 7f8fda6d5700 1 --2- 172.21.2.117:0/3982674478 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b6960 0x7f8fd41b8d70 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.060 7f8fda6d5700 1 -- 172.21.2.117:0/3982674478 >> 172.21.2.117:0/3982674478 conn(0x7f8fd40ced60 msgr2=0x7f8fd40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:22.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.060 7f8fda6d5700 1 -- 172.21.2.117:0/3982674478 shutdown_connections 2021-02-10T11:55:22.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.060 7f8fda6d5700 1 -- 172.21.2.117:0/3982674478 wait complete. 2021-02-10T11:55:22.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 Processor -- start 2021-02-10T11:55:22.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 -- start start 2021-02-10T11:55:22.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 0x7f8fd40ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8fd40ee7c0 0x7f8fd40eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40efd40 con 0x7f8fd41b8410 2021-02-10T11:55:22.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41b8940 con 0x7f8fd40f5980 2021-02-10T11:55:22.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fda6d5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40ee270 con 0x7f8fd40ee7c0 2021-02-10T11:55:22.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.075 7f8fd3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58800/0 (socket says 172.21.2.117:58800) 2021-02-10T11:55:22.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 -- 172.21.2.117:0/219651489 learned_addr learned my addr 172.21.2.117:0/219651489 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:22.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 -- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8fd40ee7c0 msgr2=0x7f8fd40eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8fd40ee7c0 0x7f8fd40eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 -- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 msgr2=0x7f8fd40ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd37fe700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 0x7f8fd40ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 0x7f8fd40ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 -- 172.21.2.117:0/219651489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40ef1e0 con 0x7f8fd41b8410 2021-02-10T11:55:22.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd37fe700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 0x7f8fd40ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:22.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd3fff700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f8fc4001ea0 tx=0x7f8fc400a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc400ac70 con 0x7f8fd41b8410 2021-02-10T11:55:22.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4001180 con 0x7f8fd41b8410 2021-02-10T11:55:22.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8fc4004590 con 0x7f8fd41b8410 2021-02-10T11:55:22.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40ef1e0 con 0x7f8fd41b8410 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.076 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd40ef770 con 0x7f8fd41b8410 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.077 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8fc40045b0 con 0x7f8fd41b8410 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.077 7f8fd0ff9700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 0x7f8fb401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.077 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8fc40326c0 con 0x7f8fd41b8410 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.078 7f8fd37fe700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 0x7f8fb401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.079 7f8fd37fe700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 0x7f8fb401abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.108 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4086140 con 0x7f8fd41b8410 2021-02-10T11:55:22.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.111 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8fc40043a0 con 0x7f8fd41b8410 2021-02-10T11:55:22.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.235 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd40d18e0 con 0x7f8fd41b8410 2021-02-10T11:55:22.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.236 7f8fd0ff9700 1 -- 172.21.2.117:0/219651489 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f8fc40045b0 con 0x7f8fd41b8410 2021-02-10T11:55:22.239 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:22.239 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:15.701452","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:55:12.669422","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:55:13.692093","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:22.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 msgr2=0x7f8fb401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 0x7f8fb401abc0 crc :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 msgr2=0x7f8fd40eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f8fc4001ea0 tx=0x7f8fc400a880).stop 2021-02-10T11:55:22.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 shutdown_connections 2021-02-10T11:55:22.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8fd40ee7c0 0x7f8fd40eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8fd40f5980 0x7f8fd40ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8fb40186c0 0x7f8fb401abc0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 --2- 172.21.2.117:0/219651489 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8fd41b8410 0x7f8fd40eda80 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 >> 172.21.2.117:0/219651489 conn(0x7f8fd4086d40 msgr2=0x7f8fd409c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:22.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 shutdown_connections 2021-02-10T11:55:22.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.238 7f8fda6d5700 1 -- 172.21.2.117:0/219651489 wait complete. 2021-02-10T11:55:22.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:55:12.669422', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '169', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:55:13.692093', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:55:22.250 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.662 7f482faf5700 1 Processor -- start 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.662 7f482faf5700 1 -- start start 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48280d18f0 0x7f48280eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48280eb040 0x7f48280eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f48280ee3e0 0x7f48280ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48280ede60 con 0x7f48280eb040 2021-02-10T11:55:22.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48280eba60 con 0x7f48280ee3e0 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482faf5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280d1cf0 con 0x7f48280d18f0 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48280eb040 0x7f48280eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f48280eb040 0x7f48280eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58804/0 (socket says 172.21.2.117:58804) 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 -- 172.21.2.117:0/3555285807 learned_addr learned my addr 172.21.2.117:0/3555285807 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 -- 172.21.2.117:0/3555285807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48280d18f0 msgr2=0x7f48280eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 --2- 172.21.2.117:0/3555285807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48280d18f0 0x7f48280eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 -- 172.21.2.117:0/3555285807 >> v2:172.21.2.125:3300/0 conn(0x7f48280ee3e0 msgr2=0x7f48280ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 --2- 172.21.2.117:0/3555285807 >> v2:172.21.2.125:3300/0 conn(0x7f48280ee3e0 0x7f48280ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.663 7f482d110700 1 -- 172.21.2.117:0/3555285807 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280eee00 con 0x7f48280eb040 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482d110700 1 --2- 172.21.2.117:0/3555285807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280eb040 0x7f48280eb440 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7f4824006700 tx=0x7f4824009b00).ready entity=mon.0 client_cookie=fabd2cb24892166e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482c90f700 1 -- 172.21.2.117:0/3555285807 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f482400c0c0 con 0x7f48280eb040 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482c90f700 1 -- 172.21.2.117:0/3555285807 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482400c340 con 0x7f48280eb040 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482c90f700 1 -- 172.21.2.117:0/3555285807 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f482400c820 con 0x7f48280eb040 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 -- 172.21.2.117:0/3555285807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280eb040 msgr2=0x7f48280eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 --2- 172.21.2.117:0/3555285807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280eb040 0x7f48280eb440 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7f4824006700 tx=0x7f4824009b00).stop 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 -- 172.21.2.117:0/3555285807 shutdown_connections 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 --2- 172.21.2.117:0/3555285807 >> v2:172.21.2.125:3300/0 conn(0x7f48280ee3e0 0x7f48280ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 --2- 172.21.2.117:0/3555285807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280eb040 0x7f48280eb440 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 --2- 172.21.2.117:0/3555285807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48280d18f0 0x7f48280eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.664 7f482faf5700 1 -- 172.21.2.117:0/3555285807 >> 172.21.2.117:0/3555285807 conn(0x7f48280cf060 msgr2=0x7f48280cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:22.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.665 7f482faf5700 1 -- 172.21.2.117:0/3555285807 shutdown_connections 2021-02-10T11:55:22.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.665 7f482faf5700 1 -- 172.21.2.117:0/3555285807 wait complete. 2021-02-10T11:55:22.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.679 7f482faf5700 1 Processor -- start 2021-02-10T11:55:22.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.679 7f482faf5700 1 -- start start 2021-02-10T11:55:22.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48280ab940 0x7f48280abd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a7180 0x7f48281a7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280ac3e0 con 0x7f48280ab470 2021-02-10T11:55:22.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48281a7ba0 con 0x7f48280ab940 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48281a7d80 con 0x7f48281a7180 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58812/0 (socket says 172.21.2.117:58812) 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 -- 172.21.2.117:0/868499225 learned_addr learned my addr 172.21.2.117:0/868499225 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 -- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a7180 msgr2=0x7f48281a7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a7180 0x7f48281a7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 -- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48280ab940 msgr2=0x7f48280abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48280ab940 0x7f48280abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 -- 172.21.2.117:0/868499225 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48281a7fa0 con 0x7f48280ab470 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482d911700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f481c00b5c0 tx=0x7f481c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c00bd50 con 0x7f48280ab470 2021-02-10T11:55:22.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c006030 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f481c0054d0 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.680 7f482faf5700 1 -- 172.21.2.117:0/868499225 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48281a7fa0 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.681 7f482faf5700 1 -- 172.21.2.117:0/868499225 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48281a7fa0 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.681 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f481c006210 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.681 7f481a7fc700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 0x7f481001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.681 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f481c0327d0 con 0x7f48280ab470 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.682 7f482d110700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 0x7f481001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:22.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.682 7f482d110700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 0x7f481001ac10 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:22.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.713 7f482faf5700 1 -- 172.21.2.117:0/868499225 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48280daff0 con 0x7f48280ab470 2021-02-10T11:55:22.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.715 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f481c006b80 con 0x7f48280ab470 2021-02-10T11:55:22.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.841 7f482faf5700 1 -- 172.21.2.117:0/868499225 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f48280daff0 con 0x7f48280ab470 2021-02-10T11:55:22.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.842 7f481a7fc700 1 -- 172.21.2.117:0/868499225 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v170) v1 ==== 92+0+10 (secure 0 0 0) 0x7f481c006b80 con 0x7f48280ab470 2021-02-10T11:55:22.845 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:22.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 msgr2=0x7f481001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 0x7f481001ac10 crc :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 msgr2=0x7f48280aabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7f481c00b5c0 tx=0x7f481c00b960).stop 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 shutdown_connections 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f48281a7180 0x7f48281a7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f48280ab940 0x7f48280abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:22.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4810018730 0x7f481001ac10 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 --2- 172.21.2.117:0/868499225 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f48280ab470 0x7f48280aabe0 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:22.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 >> 172.21.2.117:0/868499225 conn(0x7f48280c3f00 msgr2=0x7f4828094db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:22.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 shutdown_connections 2021-02-10T11:55:22.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:22.844 7f482faf5700 1 -- 172.21.2.117:0/868499225 wait complete. 2021-02-10T11:55:22.855 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:55:23.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.268 7f2fe13e9700 1 Processor -- start 2021-02-10T11:55:23.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fe13e9700 1 -- start start 2021-02-10T11:55:23.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fe13e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1b69a0 0x7f2fdc1b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fe13e9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fdc0ede60 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fe13e9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fdc0eba00 con 0x7f2fdc0ef3f0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fe13e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0d1cf0 con 0x7f2fdc1b69a0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fdb07d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2fdc0d18f0 0x7f2fdc0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59066/0 (socket says 172.21.2.117:59066) 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.269 7f2fdb07d700 1 -- 172.21.2.117:0/1981477807 learned_addr learned my addr 172.21.2.117:0/1981477807 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828428856 0 0) 0x7f2fc4001180 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc8002fe0 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fdb87e700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1b69a0 0x7f2fdc1b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2254890595 0 0) 0x7f2fc4001180 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1b69a0 msgr2=0x7f2fdc1b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:23.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1b69a0 0x7f2fdc1b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 >> v1:172.21.2.125:6789/0 conn(0x7f2fdc0ef3f0 legacy=0x7f2fdc0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc1bb3f0 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2fc4001680 con 0x7f2fdc0d18f0 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 >> v1:172.21.2.117:6789/0 conn(0x7f2fdc0d18f0 legacy=0x7f2fdc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc8003a60 0x7f2fc8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc8006780 0x7f2fc8008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc80095a0 0x7f2fc800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc800c2c0 con 0x7f2fc8006780 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc800c4e0 con 0x7f2fc8003a60 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc800c700 con 0x7f2fc80095a0 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda87c700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc8006780 0x7f2fc8008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fdb07d700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc8003a60 0x7f2fc8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fdb87e700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc80095a0 0x7f2fc800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda87c700 1 -- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc80095a0 msgr2=0x7f2fc800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda87c700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fc80095a0 0x7f2fc800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.270 7f2fda87c700 1 -- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc8003a60 msgr2=0x7f2fc8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda87c700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fc8003a60 0x7f2fc8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda87c700 1 -- 172.21.2.117:0/1981477807 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc800c9a0 con 0x7f2fc8006780 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda87c700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc8006780 0x7f2fc8008ca0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7f2fcc006ab0 tx=0x7f2fcc04daa0).ready entity=mon.0 client_cookie=b43aeece413f7941 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fcc0066d0 con 0x7f2fc8006780 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc001290 con 0x7f2fc8006780 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fda07b700 1 -- 172.21.2.117:0/1981477807 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fcc0017b0 con 0x7f2fc8006780 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 -- 172.21.2.117:0/1981477807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc8006780 msgr2=0x7f2fc8008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 --2- 172.21.2.117:0/1981477807 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fc8006780 0x7f2fc8008ca0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7f2fcc006ab0 tx=0x7f2fcc04daa0).stop 2021-02-10T11:55:23.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fdb87e700 1 -- 172.21.2.117:0/1981477807 reap_dead start 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 -- 172.21.2.117:0/1981477807 shutdown_connections 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 -- 172.21.2.117:0/1981477807 >> 172.21.2.117:0/1981477807 conn(0x7f2fdc0cf060 msgr2=0x7f2fdc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 -- 172.21.2.117:0/1981477807 shutdown_connections 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.271 7f2fe13e9700 1 -- 172.21.2.117:0/1981477807 wait complete. 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.286 7f2fe13e9700 1 Processor -- start 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.286 7f2fe13e9700 1 -- start start 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.286 7f2fe13e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1a1250 0x7f2fdc1a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.286 7f2fe13e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fe13e9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fdc0c2bb0 0x7f2fdc0c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fe13e9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0a7b20 con 0x7f2fdc0c2190 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fe13e9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc098520 con 0x7f2fdc0c2bb0 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fe13e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0a7720 con 0x7f2fdc1a1250 2021-02-10T11:55:23.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58830/0 (socket says 172.21.2.117:58830) 2021-02-10T11:55:23.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 -- 172.21.2.117:0/3154405074 learned_addr learned my addr 172.21.2.117:0/3154405074 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fdb87e700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fdc0c2bb0 0x7f2fdc0c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fdb07d700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1a1250 0x7f2fdc1a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 -- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1a1250 msgr2=0x7f2fdc1a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1a1250 0x7f2fdc1a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 -- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fdc0c2bb0 msgr2=0x7f2fdc0c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fdc0c2bb0 0x7f2fdc0c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 -- 172.21.2.117:0/3154405074 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc0a80a0 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fda87c700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f2fcc0069e0 tx=0x7f2fcc050960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fcc050d50 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc006240 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc0a80a0 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.287 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fcc0057c0 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.288 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc0a80a0 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.288 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2fcc005540 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.288 7f2fd3fff700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 0x7f2fc001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.288 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2fcc076930 con 0x7f2fdc0c2190 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.289 7f2fdb07d700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 0x7f2fc001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.289 7f2fdb07d700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 0x7f2fc001abd0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.319 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc0ede60 con 0x7f2fdc0c2190 2021-02-10T11:55:23.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.321 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2fcc006240 con 0x7f2fdc0c2190 2021-02-10T11:55:23.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.448 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f2fdc0ede60 con 0x7f2fdc0c2190 2021-02-10T11:55:23.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.448 7f2fd3fff700 1 -- 172.21.2.117:0/3154405074 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v170) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2fcc050d50 con 0x7f2fdc0c2190 2021-02-10T11:55:23.452 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:23.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 msgr2=0x7f2fc001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 0x7f2fc001abd0 crc :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 msgr2=0x7f2fdc0c2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f2fcc0069e0 tx=0x7f2fcc050960).stop 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 shutdown_connections 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2fc00186d0 0x7f2fc001abd0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fdc0c2190 0x7f2fdc0c2590 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fdc0c2bb0 0x7f2fdc0c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 --2- 172.21.2.117:0/3154405074 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fdc1a1250 0x7f2fdc1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.450 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 >> 172.21.2.117:0/3154405074 conn(0x7f2fdc0b8180 msgr2=0x7f2fdc0b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.451 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 shutdown_connections 2021-02-10T11:55:23.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.451 7f2fe13e9700 1 -- 172.21.2.117:0/3154405074 wait complete. 2021-02-10T11:55:23.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:55:23.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.875 7f542e9c8700 1 Processor -- start 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.875 7f542e9c8700 1 -- start start 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542e9c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54280d18f0 0x7f54280eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542e9c8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f54280eb040 0x7f54280eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542e9c8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54280ede60 con 0x7f54280ee3e0 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542e9c8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54280eba60 con 0x7f54280eb040 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542e9c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54280d1cf0 con 0x7f54280d18f0 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542cfe5700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f54280ee3e0 0x7f54280ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59088/0 (socket says 172.21.2.117:59088) 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f542cfe5700 1 -- 172.21.2.117:0/4089065091 learned_addr learned my addr 172.21.2.117:0/4089065091 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f54277fe700 1 --2- 172.21.2.117:0/4089065091 >> v2:172.21.2.125:3300/0 conn(0x7f54280eb040 0x7f54280eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f5427fff700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54280d18f0 0x7f54280eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f5426ffd700 1 -- 172.21.2.117:0/4089065091 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328514922 0 0) 0x7f5420000d30 con 0x7f54280ee3e0 2021-02-10T11:55:23.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f54277fe700 1 -- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54280d18f0 msgr2=0x7f54280eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.876 7f54277fe700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54280d18f0 0x7f54280eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f54277fe700 1 -- 172.21.2.117:0/4089065091 >> v1:172.21.2.117:6789/0 conn(0x7f54280ee3e0 legacy=0x7f54280ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f54277fe700 1 -- 172.21.2.117:0/4089065091 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54280eefd0 con 0x7f54280eb040 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f54277fe700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54280eb040 0x7f54280eb440 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f541c006650 tx=0x7f541c009a80).ready entity=mon.1 client_cookie=7c05a41f3c9f5711 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f5426ffd700 1 -- 172.21.2.117:0/4089065091 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c00c0c0 con 0x7f54280eb040 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f5426ffd700 1 -- 172.21.2.117:0/4089065091 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00c340 con 0x7f54280eb040 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f5426ffd700 1 -- 172.21.2.117:0/4089065091 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c00c860 con 0x7f54280eb040 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f542e9c8700 1 -- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54280eb040 msgr2=0x7f54280eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.877 7f542e9c8700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54280eb040 0x7f54280eb440 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f541c006650 tx=0x7f541c009a80).stop 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 -- 172.21.2.117:0/4089065091 shutdown_connections 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54280eb040 0x7f54280eb440 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 --2- 172.21.2.117:0/4089065091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f54280d18f0 0x7f54280eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 -- 172.21.2.117:0/4089065091 >> 172.21.2.117:0/4089065091 conn(0x7f54280cf060 msgr2=0x7f54280cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:23.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 -- 172.21.2.117:0/4089065091 shutdown_connections 2021-02-10T11:55:23.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.878 7f542e9c8700 1 -- 172.21.2.117:0/4089065091 wait complete. 2021-02-10T11:55:23.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.892 7f542e9c8700 1 Processor -- start 2021-02-10T11:55:23.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.892 7f542e9c8700 1 -- start start 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54281a4240 0x7f54281a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 0x7f54280b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542809fb90 con 0x7f542809ed70 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54280dc640 con 0x7f54281a4240 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542e9c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5428097270 con 0x7f542809f790 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58842/0 (socket says 172.21.2.117:58842) 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 -- 172.21.2.117:0/3305184739 learned_addr learned my addr 172.21.2.117:0/3305184739 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f5427fff700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54281a4240 0x7f54281a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542cfe5700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 0x7f54280b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 -- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 msgr2=0x7f54280b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 0x7f54280b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 -- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54281a4240 msgr2=0x7f54281a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54281a4240 0x7f54281a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 -- 172.21.2.117:0/3305184739 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54280b8780 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f542cfe5700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 0x7f54280b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.893 7f54277fe700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f541c009630 tx=0x7f541c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c0093a0 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00bd70 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f541c005560 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54280b8780 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54280b8d10 con 0x7f542809ed70 2021-02-10T11:55:23.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f541c00a000 con 0x7f542809ed70 2021-02-10T11:55:23.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.894 7f5424ff9700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:23.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.895 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f541c032510 con 0x7f542809ed70 2021-02-10T11:55:23.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.895 7f5427fff700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:23.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.895 7f5427fff700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 0x7f540801abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:23.928 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.926 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54280b94e0 con 0x7f542809ed70 2021-02-10T11:55:23.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:23.928 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f541c025fa0 con 0x7f542809ed70 2021-02-10T11:55:24.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.055 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f54280b94e0 con 0x7f542809ed70 2021-02-10T11:55:24.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.055 7f5424ff9700 1 -- 172.21.2.117:0/3305184739 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v170) v1 ==== 100+0+10 (secure 0 0 0) 0x7f541c025fa0 con 0x7f542809ed70 2021-02-10T11:55:24.059 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.057 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 msgr2=0x7f540801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.057 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 0x7f540801abc0 crc :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.057 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 msgr2=0x7f542809f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.057 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f541c009630 tx=0x7f541c00b980).stop 2021-02-10T11:55:24.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 shutdown_connections 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f54080186c0 0x7f540801abc0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f542809ed70 0x7f542809f170 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f542809f790 0x7f54280b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 --2- 172.21.2.117:0/3305184739 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f54281a4240 0x7f54281a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 >> 172.21.2.117:0/3305184739 conn(0x7f5428087090 msgr2=0x7f542808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 shutdown_connections 2021-02-10T11:55:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.058 7f542e9c8700 1 -- 172.21.2.117:0/3305184739 wait complete. 2021-02-10T11:55:24.068 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:55:24.069 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:24.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 Processor -- start 2021-02-10T11:55:24.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 -- start start 2021-02-10T11:55:24.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f925c0e5240 0x7f925c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f925c084c60 0x7f925c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c085930 con 0x7f925c0e4e40 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c0e5cc0 con 0x7f925c0e5240 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9264638700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c0856b0 con 0x7f925c084c60 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.480 7f9261c53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f925c0e5240 0x7f925c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f925c0e5240 0x7f925c086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58850/0 (socket says 172.21.2.117:58850) 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9262454700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f925c0e4e40 0x7f925c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59100/0 (socket says 172.21.2.117:59100) 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 -- 172.21.2.117:0/3279675215 learned_addr learned my addr 172.21.2.117:0/3279675215 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9262c55700 1 --2- 172.21.2.117:0/3279675215 >> v2:172.21.2.125:3300/0 conn(0x7f925c084c60 0x7f925c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 -- 172.21.2.117:0/3279675215 >> v2:172.21.2.125:3300/0 conn(0x7f925c084c60 msgr2=0x7f925c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 --2- 172.21.2.117:0/3279675215 >> v2:172.21.2.125:3300/0 conn(0x7f925c084c60 0x7f925c085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 -- 172.21.2.117:0/3279675215 >> v1:172.21.2.117:6789/0 conn(0x7f925c0e4e40 legacy=0x7f925c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 -- 172.21.2.117:0/3279675215 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c1b09e0 con 0x7f925c0e5240 2021-02-10T11:55:24.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261c53700 1 --2- 172.21.2.117:0/3279675215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0e5240 0x7f925c086ec0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f9258006700 tx=0x7f9258009b00).ready entity=mon.0 client_cookie=f0a1274085af231b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261452700 1 -- 172.21.2.117:0/3279675215 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925800c0c0 con 0x7f925c0e5240 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261452700 1 -- 172.21.2.117:0/3279675215 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925800c340 con 0x7f925c0e5240 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9261452700 1 -- 172.21.2.117:0/3279675215 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f925800c820 con 0x7f925c0e5240 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9264638700 1 -- 172.21.2.117:0/3279675215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0e5240 msgr2=0x7f925c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.481 7f9264638700 1 --2- 172.21.2.117:0/3279675215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0e5240 0x7f925c086ec0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f9258006700 tx=0x7f9258009b00).stop 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 -- 172.21.2.117:0/3279675215 shutdown_connections 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 --2- 172.21.2.117:0/3279675215 >> v2:172.21.2.125:3300/0 conn(0x7f925c084c60 0x7f925c085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 --2- 172.21.2.117:0/3279675215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0e5240 0x7f925c086ec0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 -- 172.21.2.117:0/3279675215 >> 172.21.2.117:0/3279675215 conn(0x7f925c0ce4e0 msgr2=0x7f925c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 -- 172.21.2.117:0/3279675215 shutdown_connections 2021-02-10T11:55:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.482 7f9264638700 1 -- 172.21.2.117:0/3279675215 wait complete. 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 Processor -- start 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 -- start start 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c18cc40 0x7f925c18b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c18baa0 0x7f925c1aadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1ae4c0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1ae6e0 con 0x7f925c18cc40 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9264638700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1ae900 con 0x7f925c18baa0 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9262c55700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9262c55700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58856/0 (socket says 172.21.2.117:58856) 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9262c55700 1 -- 172.21.2.117:0/2129247085 learned_addr learned my addr 172.21.2.117:0/2129247085 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9262454700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c18cc40 0x7f925c18b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.497 7f9261c53700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c18baa0 0x7f925c1aadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 -- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c18baa0 msgr2=0x7f925c1aadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c18baa0 0x7f925c1aadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 -- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c18cc40 msgr2=0x7f925c18b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c18cc40 0x7f925c18b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 -- 172.21.2.117:0/2129247085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c1aeb20 con 0x7f925c1ab3e0 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9262c55700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f925400b970 tx=0x7f925400bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9254000f90 con 0x7f925c1ab3e0 2021-02-10T11:55:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9254006aa0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9254001680 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9264638700 1 -- 172.21.2.117:0/2129247085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c1aeb20 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f9264638700 1 -- 172.21.2.117:0/2129247085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1aeb20 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.498 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9254000df0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.499 7f924effd700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.499 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9254032c90 con 0x7f925c1ab3e0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.499 7f9262454700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.499 7f9262454700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 0x7f924401abc0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:24.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.530 7f9264638700 1 -- 172.21.2.117:0/2129247085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c1af3a0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.533 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9254000df0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.661 7f9264638700 1 -- 172.21.2.117:0/2129247085 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f925c10c280 con 0x7f925c1ab3e0 2021-02-10T11:55:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.661 7f924effd700 1 -- 172.21.2.117:0/2129247085 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f9254006aa0 con 0x7f925c1ab3e0 2021-02-10T11:55:24.664 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:24.665 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:16.708700","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7844":{"gid":7844,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/321593523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":321593523},{"type":"v1","addr":"172.21.2.125:6835","nonce":321593523}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 144 2021-02-10T11:55:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 -- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 msgr2=0x7f924401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 0x7f924401abc0 crc :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 -- 172.21.2.117:0/2129247085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 msgr2=0x7f925c1ab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f925400b970 tx=0x7f925400bd10).stop 2021-02-10T11:55:24.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 -- 172.21.2.117:0/2129247085 shutdown_connections 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f92440186c0 0x7f924401abc0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c1ab3e0 0x7f925c1ab7e0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c18baa0 0x7f925c1aadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 --2- 172.21.2.117:0/2129247085 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c18cc40 0x7f925c18b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.663 7f9264638700 1 -- 172.21.2.117:0/2129247085 >> 172.21.2.117:0/2129247085 conn(0x7f925c18c310 msgr2=0x7f925c10c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.664 7f9264638700 1 -- 172.21.2.117:0/2129247085 shutdown_connections 2021-02-10T11:55:24.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:24.664 7f9264638700 1 -- 172.21.2.117:0/2129247085 wait complete. 2021-02-10T11:55:24.876 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:55:14.699333', 'modified': '2021-02-10 11:55:16.708700', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7844}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7844': {'gid': 7844, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/321593523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 321593523}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 321593523}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:55:24.877 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:55:24.877 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:55:25.277 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:55:25.277 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:55:25.277 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:55:25.278 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7844, 2021-02-10T11:55:25.278 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:55:25.278 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:55:25.278 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.573840132999999, 2021-02-10T11:55:25.278 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 144, 2021-02-10T11:55:25.279 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 170, 2021-02-10T11:55:25.279 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 170, 2021-02-10T11:55:25.279 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.227450953999998 2021-02-10T11:55:25.279 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:55:25.281 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7844, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.573840132999999, "mdsmap_epoch": 144, "osdmap_epoch": 170, "osdmap_epoch_barrier": 170, "uptime": 16.227450953999998 } 2021-02-10T11:55:25.282 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:55:25.282 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:55:25.282 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:55:25.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:25.287 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:25.623 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:55:25.967 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:55:25.967 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:55:26.310 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:55:26.310 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:55:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:55:26.311 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:26.311 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:55:26.646 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:55:26.647 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:26.649 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:55:26.992 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:55:26.993 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:55:27.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:55:27.057 7f8f2d827f80 -1 init, newargv = 0x560d65328f30 newargc=9 2021-02-10T11:55:27.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[184311]: starting ceph client 2021-02-10T11:55:27.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[184311]: starting fuse 2021-02-10T11:55:27.336 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:55:27.337 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:55:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:55:27.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:27.338 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:55:27.674 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:55:27.674 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:55:27.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:27.674 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:55:27.674 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:55:27.674 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:55:27.675 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:55:27.676 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:55:28.038 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.184311.asok 2021-02-10T11:55:28.041 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.184311.asok 2021-02-10T11:55:28.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:28.041 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.184311.asok status 2021-02-10T11:55:28.431 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "184311", 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:55:28.432 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12864, 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12864 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3758128205 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:28.433 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3758128205 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12864 172.21.2.117:0/3758128205", 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3758128205", 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 144, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 170, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:55:28.434 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:55:28.435 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:55:28.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:28.436 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:28.436 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:28.772 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:55:28.773 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:55:28.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:28.773 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:29.118 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:55:29.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.526 7f10a62ee700 1 Processor -- start 2021-02-10T11:55:29.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.526 7f10a62ee700 1 -- start start 2021-02-10T11:55:29.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f10a62ee700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f10a00ef3f0 0x7f10a00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f10a62ee700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00ede60 con 0x7f10a00d18e0 2021-02-10T11:55:29.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f10a62ee700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00d1ce0 con 0x7f10a00eb7f0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f10a62ee700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a00ef7f0 con 0x7f10a00ef3f0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f109ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f10a00d18e0 0x7f10a00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59126/0 (socket says 172.21.2.117:59126) 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.527 7f109ffff700 1 -- 172.21.2.117:0/2953867577 learned_addr learned my addr 172.21.2.117:0/2953867577 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435652086 0 0) 0x7f10900009b0 con 0x7f10a00d18e0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109f7fe700 1 --2- 172.21.2.117:0/2953867577 >> v2:172.21.2.125:3300/0 conn(0x7f10a00ef3f0 0x7f10a00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1088002fe0 con 0x7f10a00d18e0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 716254016 0 0) 0x7f1090001010 con 0x7f10a00d18e0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 >> v2:172.21.2.125:3300/0 conn(0x7f10a00ef3f0 msgr2=0x7f10a00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 --2- 172.21.2.117:0/2953867577 >> v2:172.21.2.125:3300/0 conn(0x7f10a00ef3f0 0x7f10a00eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 >> v1:172.21.2.125:6789/0 conn(0x7f10a00eb7f0 legacy=0x7f10a01b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a01bb180 con 0x7f10a00d18e0 2021-02-10T11:55:29.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1090001590 con 0x7f10a00d18e0 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 >> v1:172.21.2.117:6789/0 conn(0x7f10a00d18e0 legacy=0x7f10a00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 0x7f1088005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1088006790 0x7f1088008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1088009590 0x7f108800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108800c300 con 0x7f1088003ac0 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108800c520 con 0x7f1088009590 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109effd700 1 -- 172.21.2.117:0/2953867577 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1088050750 con 0x7f1088006790 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109ffff700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 0x7f1088005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f109f7fe700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1088006790 0x7f1088008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.528 7f10a490b700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1088009590 0x7f108800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 -- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1088006790 msgr2=0x7f1088008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1088006790 0x7f1088008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 -- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1088009590 msgr2=0x7f108800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:29.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1088009590 0x7f108800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 -- 172.21.2.117:0/2953867577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880509b0 con 0x7f1088003ac0 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a490b700 1 -- 172.21.2.117:0/2953867577 reap_dead start 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109ffff700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 0x7f1088005ef0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7f1090007560 tx=0x7f1090007d10).ready entity=mon.0 client_cookie=e26ae9a12878a5ec server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10900009b0 con 0x7f1088003ac0 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10900013d0 con 0x7f1088003ac0 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f109effd700 1 -- 172.21.2.117:0/2953867577 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f109000a710 con 0x7f1088003ac0 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a62ee700 1 -- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 msgr2=0x7f1088005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a62ee700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 0x7f1088005ef0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7f1090007560 tx=0x7f1090007d10).stop 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a62ee700 1 -- 172.21.2.117:0/2953867577 shutdown_connections 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a62ee700 1 --2- 172.21.2.117:0/2953867577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1088003ac0 0x7f1088005ef0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.529 7f10a62ee700 1 -- 172.21.2.117:0/2953867577 >> 172.21.2.117:0/2953867577 conn(0x7f10a00ced60 msgr2=0x7f10a00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:29.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.530 7f10a62ee700 1 -- 172.21.2.117:0/2953867577 shutdown_connections 2021-02-10T11:55:29.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.530 7f10a62ee700 1 -- 172.21.2.117:0/2953867577 wait complete. 2021-02-10T11:55:29.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.544 7f10a62ee700 1 Processor -- start 2021-02-10T11:55:29.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.544 7f10a62ee700 1 -- start start 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10a00bfa80 0x7f10a01a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f10a00a1d70 0x7f10a01a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01a3c30 con 0x7f10a00bf680 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a0680 con 0x7f10a00bfa80 2021-02-10T11:55:29.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f10a62ee700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a08a0 con 0x7f10a00a1d70 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58888/0 (socket says 172.21.2.117:58888) 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 -- 172.21.2.117:0/2127916363 learned_addr learned my addr 172.21.2.117:0/2127916363 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 -- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f10a00a1d70 msgr2=0x7f10a01a3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f10a00a1d70 0x7f10a01a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 -- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10a00bfa80 msgr2=0x7f10a01a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10a00bfa80 0x7f10a01a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.545 7f109ffff700 1 -- 172.21.2.117:0/2127916363 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00a0ac0 con 0x7f10a00bf680 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f109ffff700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7f1090007f20 tx=0x7f109000a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f10900009b0 con 0x7f10a00bf680 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109000ad20 con 0x7f10a00bf680 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1090004350 con 0x7f10a00bf680 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a00a0ac0 con 0x7f10a00bf680 2021-02-10T11:55:29.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a00a0ac0 con 0x7f10a00bf680 2021-02-10T11:55:29.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.546 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f10900074e0 con 0x7f10a00bf680 2021-02-10T11:55:29.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.547 7f109cff9700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 0x7f108001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:29.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.547 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1090032780 con 0x7f10a00bf680 2021-02-10T11:55:29.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.547 7f109f7fe700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 0x7f108001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:29.554 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.547 7f109f7fe700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 0x7f108001abd0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:29.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.577 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a0087280 con 0x7f10a00bf680 2021-02-10T11:55:29.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.579 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1090001a90 con 0x7f10a00bf680 2021-02-10T11:55:29.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.703 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a00ede60 con 0x7f10a00bf680 2021-02-10T11:55:29.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.704 7f109cff9700 1 -- 172.21.2.117:0/2127916363 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f10900074e0 con 0x7f10a00bf680 2021-02-10T11:55:29.707 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:29.707 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:55:15.701452","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:55:12.669422","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"169","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":37,"pool_name":"cephfs_data","create_time":"2021-02-10 11:55:13.692093","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"170","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:55:29.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 msgr2=0x7f108001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 0x7f108001abd0 crc :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 msgr2=0x7f10a00a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:29.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7f1090007f20 tx=0x7f109000a930).stop 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 shutdown_connections 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f10a00a1d70 0x7f10a01a3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f10a00bfa80 0x7f10a01a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f10800186d0 0x7f108001abd0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 --2- 172.21.2.117:0/2127916363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f10a00bf680 0x7f10a00a1750 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.705 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 >> 172.21.2.117:0/2127916363 conn(0x7f10a00874b0 msgr2=0x7f10a0090b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.706 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 shutdown_connections 2021-02-10T11:55:29.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:29.706 7f10a62ee700 1 -- 172.21.2.117:0/2127916363 wait complete. 2021-02-10T11:55:29.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 36, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:55:12.669422', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '169', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 37, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:55:13.692093', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '170', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:55:29.717 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.130 7ff655c74700 1 Processor -- start 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.130 7ff655c74700 1 -- start start 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.130 7ff655c74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500d18f0 0x7ff6500eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff655c74700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff6500eb040 0x7ff6500eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff655c74700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500ede60 con 0x7ff6500ee3e0 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff655c74700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6500eba60 con 0x7ff6500eb040 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff655c74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500d1cf0 con 0x7ff6500d18f0 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64ffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff6500ee3e0 0x7ff6500ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59144/0 (socket says 172.21.2.117:59144) 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64ffff700 1 -- 172.21.2.117:0/2662493833 learned_addr learned my addr 172.21.2.117:0/2662493833 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:30.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640921481 0 0) 0x7ff648001500 con 0x7ff6500ee3e0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff63c002fe0 con 0x7ff6500ee3e0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64effd700 1 --2- 172.21.2.117:0/2662493833 >> v2:172.21.2.125:3300/0 conn(0x7ff6500eb040 0x7ff6500eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.131 7ff64f7fe700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500d18f0 0x7ff6500eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2146195461 0 0) 0x7ff648001500 con 0x7ff6500ee3e0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500d18f0 msgr2=0x7ff6500eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500d18f0 0x7ff6500eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 >> v2:172.21.2.125:3300/0 conn(0x7ff6500eb040 msgr2=0x7ff6500eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 --2- 172.21.2.117:0/2662493833 >> v2:172.21.2.125:3300/0 conn(0x7ff6500eb040 0x7ff6500eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6500eefd0 con 0x7ff6500ee3e0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff648001a00 con 0x7ff6500ee3e0 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64effd700 1 --2- 172.21.2.117:0/2662493833 >> v2:172.21.2.125:3300/0 conn(0x7ff6500eb040 0x7ff6500eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:30.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 >> v1:172.21.2.117:6789/0 conn(0x7ff6500ee3e0 legacy=0x7ff6500ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff63c003a60 0x7ff63c005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff63c04a7a0 0x7ff63c04cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff63c04d500 0x7ff63c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0502d0 con 0x7ff63c04d500 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0504f0 con 0x7ff63c04a7a0 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c050710 con 0x7ff63c003a60 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff63c04d500 0x7ff63c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64effd700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff63c04a7a0 0x7ff63c04cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64f7fe700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff63c003a60 0x7ff63c005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 -- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff63c003a60 msgr2=0x7ff63c005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff63c003a60 0x7ff63c005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 -- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff63c04a7a0 msgr2=0x7ff63c04cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:30.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff63c04a7a0 0x7ff63c04cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 -- 172.21.2.117:0/2662493833 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0509b0 con 0x7ff63c04d500 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64ffff700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff63c04d500 0x7ff63c04fa20 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff648007ad0 tx=0x7ff64800a310).ready entity=mon.0 client_cookie=43a7142a0d9e9e22 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.132 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff648001ff0 con 0x7ff63c04d500 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6480070c0 con 0x7ff63c04d500 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff64e7fc700 1 -- 172.21.2.117:0/2662493833 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff648004720 con 0x7ff63c04d500 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 -- 172.21.2.117:0/2662493833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff63c04d500 msgr2=0x7ff63c04fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 --2- 172.21.2.117:0/2662493833 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff63c04d500 0x7ff63c04fa20 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff648007ad0 tx=0x7ff64800a310).stop 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff64ffff700 1 -- 172.21.2.117:0/2662493833 reap_dead start 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 -- 172.21.2.117:0/2662493833 shutdown_connections 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 -- 172.21.2.117:0/2662493833 >> 172.21.2.117:0/2662493833 conn(0x7ff6500cf060 msgr2=0x7ff6500cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 -- 172.21.2.117:0/2662493833 shutdown_connections 2021-02-10T11:55:30.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.133 7ff655c74700 1 -- 172.21.2.117:0/2662493833 wait complete. 2021-02-10T11:55:30.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 Processor -- start 2021-02-10T11:55:30.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 -- start start 2021-02-10T11:55:30.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff650085ec0 0x7ff6500862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500a2040 0x7ff6500a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500a67e0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500a7120 con 0x7ff650085ec0 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.148 7ff655c74700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6500a2a60 con 0x7ff6500a2040 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58904/0 (socket says 172.21.2.117:58904) 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 -- 172.21.2.117:0/32305342 learned_addr learned my addr 172.21.2.117:0/32305342 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64effd700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff650085ec0 0x7ff6500862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64ffff700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500a2040 0x7ff6500a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 -- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500a2040 msgr2=0x7ff6500a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500a2040 0x7ff6500a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 -- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff650085ec0 msgr2=0x7ff6500862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff650085ec0 0x7ff6500862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 -- 172.21.2.117:0/32305342 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6501a2dc0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff64f7fe700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7ff638006f60 tx=0x7ff63800b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff63800bd60 con 0x7ff6500a6ce0 2021-02-10T11:55:30.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff638000c90 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6380022b0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.149 7ff655c74700 1 -- 172.21.2.117:0/32305342 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6501a2dc0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.150 7ff655c74700 1 -- 172.21.2.117:0/32305342 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6501a2dc0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.150 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff63800a000 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.150 7ff647fff700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 0x7ff63401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.150 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff6380326f0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.151 7ff64effd700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 0x7ff63401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.151 7ff64effd700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 0x7ff63401abc0 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.180 7ff655c74700 1 -- 172.21.2.117:0/32305342 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6500daff0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.183 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff638006b00 con 0x7ff6500a6ce0 2021-02-10T11:55:30.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.309 7ff655c74700 1 -- 172.21.2.117:0/32305342 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff6500daff0 con 0x7ff6500a6ce0 2021-02-10T11:55:30.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.310 7ff647fff700 1 -- 172.21.2.117:0/32305342 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v170) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff63800bd60 con 0x7ff6500a6ce0 2021-02-10T11:55:30.313 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:30.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.311 7ff655c74700 1 -- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 msgr2=0x7ff63401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.311 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 0x7ff63401abc0 crc :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.311 7ff655c74700 1 -- 172.21.2.117:0/32305342 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 msgr2=0x7ff6500dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.311 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7ff638006f60 tx=0x7ff63800b970).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 -- 172.21.2.117:0/32305342 shutdown_connections 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6500a2040 0x7ff6500a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff650085ec0 0x7ff6500862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff6340186c0 0x7ff63401abc0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 --2- 172.21.2.117:0/32305342 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6500a6ce0 0x7ff6500dc420 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 -- 172.21.2.117:0/32305342 >> 172.21.2.117:0/32305342 conn(0x7ff6500b6540 msgr2=0x7ff65008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 -- 172.21.2.117:0/32305342 shutdown_connections 2021-02-10T11:55:30.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.312 7ff655c74700 1 -- 172.21.2.117:0/32305342 wait complete. 2021-02-10T11:55:30.323 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.735 7f0743b86700 1 Processor -- start 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.735 7f0743b86700 1 -- start start 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f0743b86700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f073c0ef3f0 0x7f073c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f0743b86700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0ef7f0 con 0x7f073c0d18f0 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f0743b86700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0ede60 con 0x7f073c0eb830 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f0743b86700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c0d1da0 con 0x7f073c0ef3f0 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f07419a2700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f073c0d18f0 0x7f073c0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59162/0 (socket says 172.21.2.117:59162) 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f07419a2700 1 -- 172.21.2.117:0/1802917346 learned_addr learned my addr 172.21.2.117:0/1802917346 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f07409a0700 1 -- 172.21.2.117:0/1802917346 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868915806 0 0) 0x7f07300010b0 con 0x7f073c0d18f0 2021-02-10T11:55:30.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f07409a0700 1 -- 172.21.2.117:0/1802917346 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0728002fe0 con 0x7f073c0d18f0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.736 7f07411a1700 1 --2- 172.21.2.117:0/1802917346 >> v2:172.21.2.125:3300/0 conn(0x7f073c0ef3f0 0x7f073c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07411a1700 1 -- 172.21.2.117:0/1802917346 >> v1:172.21.2.125:6789/0 conn(0x7f073c0eb830 legacy=0x7f073c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07411a1700 1 -- 172.21.2.117:0/1802917346 >> v1:172.21.2.117:6789/0 conn(0x7f073c0d18f0 legacy=0x7f073c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07411a1700 1 -- 172.21.2.117:0/1802917346 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c1bb1c0 con 0x7f073c0ef3f0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07411a1700 1 --2- 172.21.2.117:0/1802917346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c0ef3f0 0x7f073c0eb210 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f0738006650 tx=0x7f0738009a80).ready entity=mon.1 client_cookie=261aa3f5a59aafd9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07409a0700 1 -- 172.21.2.117:0/1802917346 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f073800c0c0 con 0x7f073c0ef3f0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07409a0700 1 -- 172.21.2.117:0/1802917346 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073800c340 con 0x7f073c0ef3f0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f07409a0700 1 -- 172.21.2.117:0/1802917346 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f073800c860 con 0x7f073c0ef3f0 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f0743b86700 1 -- 172.21.2.117:0/1802917346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c0ef3f0 msgr2=0x7f073c0eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.737 7f0743b86700 1 --2- 172.21.2.117:0/1802917346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c0ef3f0 0x7f073c0eb210 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f0738006650 tx=0x7f0738009a80).stop 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.738 7f0743b86700 1 -- 172.21.2.117:0/1802917346 shutdown_connections 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.738 7f0743b86700 1 --2- 172.21.2.117:0/1802917346 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c0ef3f0 0x7f073c0eb210 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.738 7f0743b86700 1 -- 172.21.2.117:0/1802917346 >> 172.21.2.117:0/1802917346 conn(0x7f073c0cf060 msgr2=0x7f073c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:30.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.738 7f0743b86700 1 -- 172.21.2.117:0/1802917346 shutdown_connections 2021-02-10T11:55:30.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.738 7f0743b86700 1 -- 172.21.2.117:0/1802917346 wait complete. 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.752 7f0743b86700 1 Processor -- start 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.752 7f0743b86700 1 -- start start 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f073c09ed70 0x7f073c09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c09f790 0x7f073c0b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c09fb90 con 0x7f073c1a4240 2021-02-10T11:55:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0dc640 con 0x7f073c09f790 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f0743b86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c097270 con 0x7f073c09ed70 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58920/0 (socket says 172.21.2.117:58920) 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 -- 172.21.2.117:0/3873385454 learned_addr learned my addr 172.21.2.117:0/3873385454 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 -- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f073c09ed70 msgr2=0x7f073c09f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f073c09ed70 0x7f073c09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 -- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c09f790 msgr2=0x7f073c0b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c09f790 0x7f073c0b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 -- 172.21.2.117:0/3873385454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0b8780 con 0x7f073c1a4240 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.753 7f07419a2700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f0730001c60 tx=0x7f073000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.754 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f073000acc0 con 0x7f073c1a4240 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.754 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0730004140 con 0x7f073c1a4240 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.754 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0730004660 con 0x7f073c1a4240 2021-02-10T11:55:30.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.754 7f0743b86700 1 -- 172.21.2.117:0/3873385454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c0b8780 con 0x7f073c1a4240 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.754 7f0743b86700 1 -- 172.21.2.117:0/3873385454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c0b8780 con 0x7f073c1a4240 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.755 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0730004320 con 0x7f073c1a4240 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.755 7f072e7fc700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 0x7f072401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.755 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0730032530 con 0x7f073c1a4240 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.755 7f07411a1700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 0x7f072401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:30.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.755 7f07411a1700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 0x7f072401abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:30.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.786 7f0743b86700 1 -- 172.21.2.117:0/3873385454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c0daff0 con 0x7f073c1a4240 2021-02-10T11:55:30.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.789 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0730026a90 con 0x7f073c1a4240 2021-02-10T11:55:30.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.916 7f0743b86700 1 -- 172.21.2.117:0/3873385454 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f073c0daff0 con 0x7f073c1a4240 2021-02-10T11:55:30.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.916 7f072e7fc700 1 -- 172.21.2.117:0/3873385454 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v170) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0730026a90 con 0x7f073c1a4240 2021-02-10T11:55:30.920 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:30.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.918 7f0743b86700 1 -- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 msgr2=0x7f072401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.918 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 0x7f072401abc0 crc :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.921 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.918 7f0743b86700 1 -- 172.21.2.117:0/3873385454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 msgr2=0x7f073c1a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.918 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7f0730001c60 tx=0x7f073000a8d0).stop 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 -- 172.21.2.117:0/3873385454 shutdown_connections 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f073c09f790 0x7f073c0b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f073c09ed70 0x7f073c09f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07240186c0 0x7f072401abc0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 --2- 172.21.2.117:0/3873385454 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f073c1a4240 0x7f073c1a4640 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 -- 172.21.2.117:0/3873385454 >> 172.21.2.117:0/3873385454 conn(0x7f073c087090 msgr2=0x7f073c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 -- 172.21.2.117:0/3873385454 shutdown_connections 2021-02-10T11:55:30.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:30.919 7f0743b86700 1 -- 172.21.2.117:0/3873385454 wait complete. 2021-02-10T11:55:30.930 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 Processor -- start 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 -- start start 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 0x7f52e00eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f52e00eb830 0x7f52e01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e00ef7f0 con 0x7f52e00d18f0 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e00eded0 con 0x7f52e00eb830 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52e5f86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00d1cf0 con 0x7f52e00ef3f0 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52df7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f52e00d18f0 0x7f52e00eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52102/0 (socket says 172.21.2.117:52102) 2021-02-10T11:55:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52df7fe700 1 -- 172.21.2.117:0/1044288316 learned_addr learned my addr 172.21.2.117:0/1044288316 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.345 7f52deffd700 1 --2- 172.21.2.117:0/1044288316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 0x7f52e00eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52dffff700 1 --2- 172.21.2.117:0/1044288316 >> v2:172.21.2.125:3300/0 conn(0x7f52e00eb830 0x7f52e01b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52deffd700 1 -- 172.21.2.117:0/1044288316 >> v2:172.21.2.125:3300/0 conn(0x7f52e00eb830 msgr2=0x7f52e01b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52deffd700 1 --2- 172.21.2.117:0/1044288316 >> v2:172.21.2.125:3300/0 conn(0x7f52e00eb830 0x7f52e01b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52deffd700 1 -- 172.21.2.117:0/1044288316 >> v1:172.21.2.125:6789/0 conn(0x7f52e00d18f0 legacy=0x7f52e00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52deffd700 1 -- 172.21.2.117:0/1044288316 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e01b8fd0 con 0x7f52e00ef3f0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.346 7f52deffd700 1 --2- 172.21.2.117:0/1044288316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 0x7f52e00eb210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f52d80069d0 tx=0x7f52d8009e40).ready entity=mon.2 client_cookie=c362a7bcf65d8c8a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.347 7f52de7fc700 1 -- 172.21.2.117:0/1044288316 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d800b510 con 0x7f52e00ef3f0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.347 7f52de7fc700 1 -- 172.21.2.117:0/1044288316 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d800b890 con 0x7f52e00ef3f0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.347 7f52de7fc700 1 -- 172.21.2.117:0/1044288316 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d800bcf0 con 0x7f52e00ef3f0 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 -- 172.21.2.117:0/1044288316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 msgr2=0x7f52e00eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 --2- 172.21.2.117:0/1044288316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 0x7f52e00eb210 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f52d80069d0 tx=0x7f52d8009e40).stop 2021-02-10T11:55:31.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 -- 172.21.2.117:0/1044288316 shutdown_connections 2021-02-10T11:55:31.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 --2- 172.21.2.117:0/1044288316 >> v2:172.21.2.125:3300/0 conn(0x7f52e00eb830 0x7f52e01b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 --2- 172.21.2.117:0/1044288316 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e00ef3f0 0x7f52e00eb210 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 -- 172.21.2.117:0/1044288316 >> 172.21.2.117:0/1044288316 conn(0x7f52e00cf060 msgr2=0x7f52e00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:31.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 -- 172.21.2.117:0/1044288316 shutdown_connections 2021-02-10T11:55:31.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.348 7f52e5f86700 1 -- 172.21.2.117:0/1044288316 wait complete. 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.362 7f52e5f86700 1 Processor -- start 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.362 7f52e5f86700 1 -- start start 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52e009ed70 0x7f52e009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e009f790 0x7f52e00b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e009fb90 con 0x7f52e009ed70 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e00dc640 con 0x7f52e01a4240 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52e5f86700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e0097270 con 0x7f52e009f790 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52df7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52df7fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35086/0 (socket says 172.21.2.117:35086) 2021-02-10T11:55:31.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52deffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52e009ed70 0x7f52e009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52df7fe700 1 -- 172.21.2.117:0/4218684443 learned_addr learned my addr 172.21.2.117:0/4218684443 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.363 7f52dffff700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e009f790 0x7f52e00b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 -- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e009f790 msgr2=0x7f52e00b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e009f790 0x7f52e00b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 -- 172.21.2.117:0/4218684443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52e009ed70 msgr2=0x7f52e009f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52e009ed70 0x7f52e009f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 -- 172.21.2.117:0/4218684443 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e00b8780 con 0x7f52e01a4240 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52df7fe700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f52d4007650 tx=0x7f52d400a900).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d4000e20 con 0x7f52e01a4240 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d400acf0 con 0x7f52e01a4240 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f52d4004540 con 0x7f52e01a4240 2021-02-10T11:55:31.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e00b8780 con 0x7f52e01a4240 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.364 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e00b8d10 con 0x7f52e01a4240 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.365 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f52d4004140 con 0x7f52e01a4240 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.365 7f52c7fff700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 0x7f52c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.365 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f52d4032580 con 0x7f52e01a4240 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.365 7f52deffd700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 0x7f52c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.365 7f52deffd700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 0x7f52c001abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:31.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.395 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e00daff0 con 0x7f52e01a4240 2021-02-10T11:55:31.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.398 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f52d4000940 con 0x7f52e01a4240 2021-02-10T11:55:31.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.528 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f52e00ede60 con 0x7f52e01a4240 2021-02-10T11:55:31.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.529 7f52c7fff700 1 -- 172.21.2.117:0/4218684443 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v170) v1 ==== 100+0+10 (secure 0 0 0) 0x7f52d4000ce0 con 0x7f52e01a4240 2021-02-10T11:55:31.532 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:55:31.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.530 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 msgr2=0x7f52c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.530 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 0x7f52c001abc0 crc :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.530 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 msgr2=0x7f52e01a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.530 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f52d4007650 tx=0x7f52d400a900).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 shutdown_connections 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f52c00186c0 0x7f52c001abc0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f52e009ed70 0x7f52e009f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f52e009f790 0x7f52e00b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 --2- 172.21.2.117:0/4218684443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f52e01a4240 0x7f52e01a4640 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 >> 172.21.2.117:0/4218684443 conn(0x7f52e0087090 msgr2=0x7f52e008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 shutdown_connections 2021-02-10T11:55:31.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.531 7f52e5f86700 1 -- 172.21.2.117:0/4218684443 wait complete. 2021-02-10T11:55:31.542 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:55:31.542 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.952 7f61acb04700 1 Processor -- start 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.952 7f61acb04700 1 -- start start 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61acb04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 0x7f61a80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61acb04700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f61a80eb7f0 0x7f61a81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61acb04700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a80ede60 con 0x7f61a80d18e0 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61acb04700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a80ef7f0 con 0x7f61a80eb7f0 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61acb04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a80d1ce0 con 0x7f61a80ef3f0 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a685e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f61a80d18e0 0x7f61a80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52116/0 (socket says 172.21.2.117:52116) 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a685e700 1 -- 172.21.2.117:0/2521448144 learned_addr learned my addr 172.21.2.117:0/2521448144 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a705f700 1 --2- 172.21.2.117:0/2521448144 >> v2:172.21.2.125:3300/0 conn(0x7f61a80eb7f0 0x7f61a81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a605d700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 0x7f61a80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a705f700 1 -- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 msgr2=0x7f61a80eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a705f700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 0x7f61a80eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a705f700 1 -- 172.21.2.117:0/2521448144 >> v1:172.21.2.125:6789/0 conn(0x7f61a80d18e0 legacy=0x7f61a80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.953 7f61a705f700 1 -- 172.21.2.117:0/2521448144 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a81b8f90 con 0x7f61a80eb7f0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61a605d700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 0x7f61a80eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61a705f700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a80eb7f0 0x7f61a81b7960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f61a0006e50 tx=0x7f61a0009e70).ready entity=mon.1 client_cookie=14fe6ac4422f4b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61a585c700 1 -- 172.21.2.117:0/2521448144 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61a0000f90 con 0x7f61a80eb7f0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61a585c700 1 -- 172.21.2.117:0/2521448144 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a00066f0 con 0x7f61a80eb7f0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61a585c700 1 -- 172.21.2.117:0/2521448144 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61a000b700 con 0x7f61a80eb7f0 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61acb04700 1 -- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a80eb7f0 msgr2=0x7f61a81b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.954 7f61acb04700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a80eb7f0 0x7f61a81b7960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f61a0006e50 tx=0x7f61a0009e70).stop 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 -- 172.21.2.117:0/2521448144 shutdown_connections 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a80eb7f0 0x7f61a81b7960 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 --2- 172.21.2.117:0/2521448144 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a80ef3f0 0x7f61a80eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 -- 172.21.2.117:0/2521448144 >> 172.21.2.117:0/2521448144 conn(0x7f61a80ced60 msgr2=0x7f61a80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 -- 172.21.2.117:0/2521448144 shutdown_connections 2021-02-10T11:55:31.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.955 7f61acb04700 1 -- 172.21.2.117:0/2521448144 wait complete. 2021-02-10T11:55:31.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.969 7f61acb04700 1 Processor -- start 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.969 7f61acb04700 1 -- start start 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 0x7f61a81b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 0x7f61a8198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8197cd0 con 0x7f61a8091d40 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a8092140 con 0x7f61a81b24b0 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61acb04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a81b2ed0 con 0x7f61a8198250 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58946/0 (socket says 172.21.2.117:58946) 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 -- 172.21.2.117:0/4152224043 learned_addr learned my addr 172.21.2.117:0/4152224043 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a705f700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 0x7f61a8198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a605d700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 0x7f61a81b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 -- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 msgr2=0x7f61a8198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 0x7f61a8198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 -- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 msgr2=0x7f61a81b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 0x7f61a81b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a685e700 1 -- 172.21.2.117:0/4152224043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8198c70 con 0x7f61a8091d40 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a705f700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 0x7f61a8198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.970 7f61a605d700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 0x7f61a81b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f61a685e700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f6190007fa0 tx=0x7f619000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61900070d0 con 0x7f61a8091d40 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619000ad40 con 0x7f61a8091d40 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f61900045c0 con 0x7f61a8091d40 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f61acb04700 1 -- 172.21.2.117:0/4152224043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a8198c70 con 0x7f61a8091d40 2021-02-10T11:55:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.971 7f61acb04700 1 -- 172.21.2.117:0/4152224043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a8196c90 con 0x7f61a8091d40 2021-02-10T11:55:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.972 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6190000c10 con 0x7f61a8091d40 2021-02-10T11:55:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.972 7f619f7fe700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 0x7f618c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.972 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f61900326b0 con 0x7f61a8091d40 2021-02-10T11:55:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.972 7f61a605d700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 0x7f618c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:31.973 7f61a605d700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 0x7f618c01abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:32.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.002 7f61acb04700 1 -- 172.21.2.117:0/4152224043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a8087290 con 0x7f61a8091d40 2021-02-10T11:55:32.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.005 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6190000f00 con 0x7f61a8091d40 2021-02-10T11:55:32.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.132 7f61acb04700 1 -- 172.21.2.117:0/4152224043 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61a80ede60 con 0x7f61a8091d40 2021-02-10T11:55:32.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.133 7f619f7fe700 1 -- 172.21.2.117:0/4152224043 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+2105 (secure 0 0 0) 0x7f6190000c10 con 0x7f61a8091d40 2021-02-10T11:55:32.136 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 144 2021-02-10T11:55:32.136 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:32.136 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":144,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:16.708700","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7844":{"gid":7844,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/321593523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":321593523},{"type":"v1","addr":"172.21.2.125:6835","nonce":321593523}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 msgr2=0x7f618c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 0x7f618c01abc0 crc :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 msgr2=0x7f61a80e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7f6190007fa0 tx=0x7f619000a950).stop 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 shutdown_connections 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f61a8198250 0x7f61a8198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:32.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f61a81b24b0 0x7f61a81b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:32.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f618c0186c0 0x7f618c01abc0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:32.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 --2- 172.21.2.117:0/4152224043 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f61a8091d40 0x7f61a80e71e0 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:32.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 >> 172.21.2.117:0/4152224043 conn(0x7f61a81a3620 msgr2=0x7f61a80dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:32.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 shutdown_connections 2021-02-10T11:55:32.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:32.135 7f61acb04700 1 -- 172.21.2.117:0/4152224043 wait complete. 2021-02-10T11:55:32.348 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:55:14.699333', 'modified': '2021-02-10 11:55:16.708700', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7844}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7844': {'gid': 7844, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/321593523', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 321593523}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 321593523}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:55:32.348 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:55:32.348 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 7844, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.043477887000002, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 144, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 170, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 170, 2021-02-10T11:55:32.747 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.697089663 2021-02-10T11:55:32.748 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:55:32.751 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 7844, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.043477887000002, "mdsmap_epoch": 144, "osdmap_epoch": 170, "osdmap_epoch_barrier": 170, "uptime": 23.697089663 } 2021-02-10T11:55:32.751 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:55:32.751 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:55:32.751 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:55:32.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:32.751 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:33.088 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:55:33.433 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:55:33.433 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:55:33.777 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:55:33.777 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:55:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:55:33.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:33.778 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:55:34.114 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:55:34.114 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:55:34.114 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:34.117 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:55:34.461 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:55:34.461 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:55:34.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:55:34.525 7f64f9a14f80 -1 init, newargv = 0x560c27616f90 newargc=9 2021-02-10T11:55:34.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[184951]: starting ceph client 2021-02-10T11:55:34.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[184951]: starting fuse 2021-02-10T11:55:34.805 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:55:34.806 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:55:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:55:34.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:34.807 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:55:35.143 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:55:35.143 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:55:35.143 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:55:35.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:55:35.144 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:55:35.145 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:55:35.507 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.184951.asok 2021-02-10T11:55:35.510 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.184951.asok 2021-02-10T11:55:35.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:35.510 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.184951.asok status 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "184951", 2021-02-10T11:55:35.902 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 12897, 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 12897 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3172597060 2021-02-10T11:55:35.903 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3172597060 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.12897 172.21.2.117:0/3172597060", 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3172597060", 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 144, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 170, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:55:35.904 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:55:35.905 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:55:35.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:35.906 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:35.906 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:36.242 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:55:36.242 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:55:36.243 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:36.243 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:36.591 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 Processor -- start 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 -- start start 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efc6c0e7170 0x7efc6c0e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6c0e7570 con 0x7efc6c1b07e0 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6c0e77e0 con 0x7efc6c0e8380 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.005 7efc70ec9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c0e5cd0 con 0x7efc6c0e7170 2021-02-10T11:55:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc6ab13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efc6c1b07e0 0x7efc6c0e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59216/0 (socket says 172.21.2.117:59216) 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc6ab13700 1 -- 172.21.2.117:0/2574680791 learned_addr learned my addr 172.21.2.117:0/2574680791 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422594697 0 0) 0x7efc5c001180 con 0x7efc6c1b07e0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc69b11700 1 -- 172.21.2.117:0/2574680791 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54002fe0 con 0x7efc6c1b07e0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc6a312700 1 --2- 172.21.2.117:0/2574680791 >> v2:172.21.2.125:3300/0 conn(0x7efc6c0e7170 0x7efc6c0e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.006 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3406121521 0 0) 0x7efc5c001180 con 0x7efc6c1b07e0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 >> v2:172.21.2.125:3300/0 conn(0x7efc6c0e7170 msgr2=0x7efc6c0e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 --2- 172.21.2.117:0/2574680791 >> v2:172.21.2.125:3300/0 conn(0x7efc6c0e7170 0x7efc6c0e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 >> v1:172.21.2.125:6789/0 conn(0x7efc6c0e8380 legacy=0x7efc6c0e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0e8f70 con 0x7efc6c1b07e0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7efc5c001680 con 0x7efc6c1b07e0 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 >> v1:172.21.2.117:6789/0 conn(0x7efc6c1b07e0 legacy=0x7efc6c0e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc54003a70 0x7efc54005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 0x7efc54008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc540095a0 0x7efc5400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5400c310 con 0x7efc540067a0 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5400c530 con 0x7efc540095a0 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 0x7efc54008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc54050760 con 0x7efc54003a70 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6b314700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc540095a0 0x7efc5400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 -- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc54003a70 msgr2=0x7efc54005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6ab13700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc54003a70 0x7efc54005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc54003a70 0x7efc54005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 -- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc540095a0 msgr2=0x7efc5400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc540095a0 0x7efc5400bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 -- 172.21.2.117:0/2574680791 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc540509c0 con 0x7efc540067a0 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6b314700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc540095a0 0x7efc5400bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6ab13700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc54003a70 0x7efc54005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6b314700 1 -- 172.21.2.117:0/2574680791 reap_dead start 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc6a312700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 0x7efc54008ca0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7efc600066a0 tx=0x7efc6000cc80).ready entity=mon.0 client_cookie=10cffceeca8dbcfc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.007 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc60006470 con 0x7efc540067a0 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60009310 con 0x7efc540067a0 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc69b11700 1 -- 172.21.2.117:0/2574680791 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc600095c0 con 0x7efc540067a0 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc70ec9700 1 -- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 msgr2=0x7efc54008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc70ec9700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 0x7efc54008ca0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 rx=0x7efc600066a0 tx=0x7efc6000cc80).stop 2021-02-10T11:55:37.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc70ec9700 1 -- 172.21.2.117:0/2574680791 shutdown_connections 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc70ec9700 1 --2- 172.21.2.117:0/2574680791 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc540067a0 0x7efc54008ca0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.008 7efc70ec9700 1 -- 172.21.2.117:0/2574680791 >> 172.21.2.117:0/2574680791 conn(0x7efc6c0ce4f0 msgr2=0x7efc6c0e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.009 7efc70ec9700 1 -- 172.21.2.117:0/2574680791 shutdown_connections 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.009 7efc70ec9700 1 -- 172.21.2.117:0/2574680791 wait complete. 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 Processor -- start 2021-02-10T11:55:37.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 -- start start 2021-02-10T11:55:37.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc6c0a4480 0x7efc6c1a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6c0a3900 0x7efc6c0a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c1a2090 con 0x7efc6c1a15b0 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c1a2c30 con 0x7efc6c0a4480 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.023 7efc70ec9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c10c3b0 con 0x7efc6c0a3900 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58976/0 (socket says 172.21.2.117:58976) 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 -- 172.21.2.117:0/2264400165 learned_addr learned my addr 172.21.2.117:0/2264400165 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 -- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6c0a3900 msgr2=0x7efc6c0a3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6c0a3900 0x7efc6c0a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 -- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc6c0a4480 msgr2=0x7efc6c1a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc6c0a4480 0x7efc6c1a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 -- 172.21.2.117:0/2264400165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c10c5d0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc6b314700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7efc6400b940 tx=0x7efc6400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc640030c0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc640011a0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efc64001630 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c10c5d0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.024 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c10c5d0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.025 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efc6400a000 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.025 7efc5b7fe700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.025 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efc64032a60 con 0x7efc6c1a15b0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.026 7efc6ab13700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.026 7efc6ab13700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 0x7efc5001abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.055 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c0e77e0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.060 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.058 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efc6400be20 con 0x7efc6c1a15b0 2021-02-10T11:55:37.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.190 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc6c0e77e0 con 0x7efc6c1a15b0 2021-02-10T11:55:37.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.244 7efc5b7fe700 1 -- 172.21.2.117:0/2264400165 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v145) v1 ==== 136+0+0 (secure 0 0 0) 0x7efc64006f20 con 0x7efc6c1a15b0 2021-02-10T11:55:37.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.255 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 msgr2=0x7efc5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.255 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 0x7efc5001abc0 crc :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.255 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 msgr2=0x7efc6c1a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.255 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7efc6400b940 tx=0x7efc6400bce0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 shutdown_connections 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efc500186c0 0x7efc5001abc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efc6c1a15b0 0x7efc6c1a19b0 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efc6c0a3900 0x7efc6c0a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 --2- 172.21.2.117:0/2264400165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efc6c0a4480 0x7efc6c1a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 >> 172.21.2.117:0/2264400165 conn(0x7efc6c19efd0 msgr2=0x7efc6c1af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:37.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 shutdown_connections 2021-02-10T11:55:37.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.256 7efc70ec9700 1 -- 172.21.2.117:0/2264400165 wait complete. 2021-02-10T11:55:37.267 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 Processor -- start 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 -- start start 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f78901b07f0 0x7f78900e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78900e6170 0x7f78900e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78900e6b90 0x7f78900e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78900e5ce0 con 0x7f78900e6b90 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78900e75b0 con 0x7f78901b07f0 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7898423700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78900e77f0 con 0x7f78900e6170 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.676 7f7896a40700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78900e6b90 0x7f78900e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f78900e6b90 0x7f78900e6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58982/0 (socket says 172.21.2.117:58982) 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 -- 172.21.2.117:0/3605238226 learned_addr learned my addr 172.21.2.117:0/3605238226 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 -- 172.21.2.117:0/3605238226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78900e6170 msgr2=0x7f78900e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 --2- 172.21.2.117:0/3605238226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78900e6170 0x7f78900e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 -- 172.21.2.117:0/3605238226 >> v2:172.21.2.125:3300/0 conn(0x7f78901b07f0 msgr2=0x7f78900e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 --2- 172.21.2.117:0/3605238226 >> v2:172.21.2.125:3300/0 conn(0x7f78901b07f0 0x7f78900e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 -- 172.21.2.117:0/3605238226 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78900e54b0 con 0x7f78900e6b90 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7896a40700 1 --2- 172.21.2.117:0/3605238226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78900e6b90 0x7f78900e6f90 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7f7888006f00 tx=0x7f7888009f10).ready entity=mon.0 client_cookie=83a12a61854a013 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f789523d700 1 -- 172.21.2.117:0/3605238226 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788800b3b0 con 0x7f78900e6b90 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f789523d700 1 -- 172.21.2.117:0/3605238226 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7888000d30 con 0x7f78900e6b90 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f789523d700 1 -- 172.21.2.117:0/3605238226 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788800b820 con 0x7f78900e6b90 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7898423700 1 -- 172.21.2.117:0/3605238226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78900e6b90 msgr2=0x7f78900e6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.677 7f7898423700 1 --2- 172.21.2.117:0/3605238226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78900e6b90 0x7f78900e6f90 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7f7888006f00 tx=0x7f7888009f10).stop 2021-02-10T11:55:37.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 -- 172.21.2.117:0/3605238226 shutdown_connections 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 --2- 172.21.2.117:0/3605238226 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f78900e6b90 0x7f78900e6f90 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 --2- 172.21.2.117:0/3605238226 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f78900e6170 0x7f78900e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 --2- 172.21.2.117:0/3605238226 >> v2:172.21.2.125:3300/0 conn(0x7f78901b07f0 0x7f78900e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 -- 172.21.2.117:0/3605238226 >> 172.21.2.117:0/3605238226 conn(0x7f78900ce500 msgr2=0x7f78900e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 -- 172.21.2.117:0/3605238226 shutdown_connections 2021-02-10T11:55:37.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.678 7f7898423700 1 -- 172.21.2.117:0/3605238226 wait complete. 2021-02-10T11:55:37.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.692 7f7898423700 1 Processor -- start 2021-02-10T11:55:37.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.692 7f7898423700 1 -- start start 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789008ef00 0x7f7890199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78900b1af0 0x7f78900b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7890105f50 con 0x7f7890199a70 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789008f300 con 0x7f78900b1af0 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7898423700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78900c3f80 con 0x7f789008ef00 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:58992/0 (socket says 172.21.2.117:58992) 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 -- 172.21.2.117:0/3796977674 learned_addr learned my addr 172.21.2.117:0/3796977674 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 -- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789008ef00 msgr2=0x7f7890199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789008ef00 0x7f7890199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 -- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78900b1af0 msgr2=0x7f78900b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78900b1af0 0x7f78900b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.693 7f7895a3e700 1 -- 172.21.2.117:0/3796977674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78900b2510 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7895a3e700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7f788c006f00 tx=0x7f788c00b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788c00bd30 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c000c90 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f788c0022a0 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7898423700 1 -- 172.21.2.117:0/3796977674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78900b2510 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.694 7f7898423700 1 -- 172.21.2.117:0/3796977674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78900afca0 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.695 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f788c005310 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.695 7f7882ffd700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 0x7f787801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.695 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f788c032510 con 0x7f7890199a70 2021-02-10T11:55:37.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.695 7f789623f700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 0x7f787801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:37.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.696 7f789623f700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 0x7f787801abc0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:37.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.726 7f7898423700 1 -- 172.21.2.117:0/3796977674 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78901b8070 con 0x7f7890199a70 2021-02-10T11:55:37.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.729 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f788c000960 con 0x7f7890199a70 2021-02-10T11:55:37.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.862 7f7898423700 1 -- 172.21.2.117:0/3796977674 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f789008f300 con 0x7f78780186c0 2021-02-10T11:55:37.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.865 7f7882ffd700 1 -- 172.21.2.117:0/3796977674 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f7884006a10 con 0x7f78780186c0 2021-02-10T11:55:37.868 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:55:37.868 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:37.868 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:55:37.869 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:37.869 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 msgr2=0x7f787801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 0x7f787801abc0 crc :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 msgr2=0x7f78900b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7f788c006f00 tx=0x7f788c00b940).stop 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 shutdown_connections 2021-02-10T11:55:37.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f78780186c0 0x7f787801abc0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7890199a70 0x7f78900b14d0 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f78900b1af0 0x7f78900b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 --2- 172.21.2.117:0/3796977674 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f789008ef00 0x7f7890199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 >> 172.21.2.117:0/3796977674 conn(0x7f78900b97b0 msgr2=0x7f7890098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 shutdown_connections 2021-02-10T11:55:37.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:37.867 7f7898423700 1 -- 172.21.2.117:0/3796977674 wait complete. 2021-02-10T11:55:37.878 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 Processor -- start 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 -- start start 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 0x7f27980eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27980ef7f0 con 0x7f27980eb830 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27980d1cf0 con 0x7f27980d18f0 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f279ca13700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27980eded0 con 0x7f27980ef3f0 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f2796e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f27980eb830 0x7f27981b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59250/0 (socket says 172.21.2.117:59250) 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f2796e1d700 1 -- 172.21.2.117:0/2300291585 learned_addr learned my addr 172.21.2.117:0/2300291585 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:38.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.292 7f278de1b700 1 --2- 172.21.2.117:0/2300291585 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 0x7f27980eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f2795e1b700 1 -- 172.21.2.117:0/2300291585 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533702781 0 0) 0x7f2790001500 con 0x7f27980eb830 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f2795e1b700 1 -- 172.21.2.117:0/2300291585 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2784002fe0 con 0x7f27980eb830 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f278de1b700 1 -- 172.21.2.117:0/2300291585 >> v1:172.21.2.125:6789/0 conn(0x7f27980d18f0 legacy=0x7f27980eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f278de1b700 1 -- 172.21.2.117:0/2300291585 >> v1:172.21.2.117:6789/0 conn(0x7f27980eb830 legacy=0x7f27981b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f278de1b700 1 -- 172.21.2.117:0/2300291585 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27981bb1c0 con 0x7f27980ef3f0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f278de1b700 1 --2- 172.21.2.117:0/2300291585 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 0x7f27980eb210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2780006650 tx=0x7f2780009a80).ready entity=mon.2 client_cookie=4135cee2626cbcd0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f2795e1b700 1 -- 172.21.2.117:0/2300291585 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27800009b0 con 0x7f27980ef3f0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f2795e1b700 1 -- 172.21.2.117:0/2300291585 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278000c0c0 con 0x7f27980ef3f0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f2795e1b700 1 -- 172.21.2.117:0/2300291585 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f278000c510 con 0x7f27980ef3f0 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f279ca13700 1 -- 172.21.2.117:0/2300291585 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 msgr2=0x7f27980eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.293 7f279ca13700 1 --2- 172.21.2.117:0/2300291585 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 0x7f27980eb210 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f2780006650 tx=0x7f2780009a80).stop 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.294 7f279ca13700 1 -- 172.21.2.117:0/2300291585 shutdown_connections 2021-02-10T11:55:38.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.294 7f279ca13700 1 --2- 172.21.2.117:0/2300291585 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27980ef3f0 0x7f27980eb210 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:38.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.294 7f279ca13700 1 -- 172.21.2.117:0/2300291585 >> 172.21.2.117:0/2300291585 conn(0x7f27980cf060 msgr2=0x7f27980cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:38.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.294 7f279ca13700 1 -- 172.21.2.117:0/2300291585 shutdown_connections 2021-02-10T11:55:38.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.294 7f279ca13700 1 -- 172.21.2.117:0/2300291585 wait complete. 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.308 7f279ca13700 1 Processor -- start 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.308 7f279ca13700 1 -- start start 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279807c7c0 0x7f27981a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27981a3930 0x7f27980a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279807cca0 con 0x7f27980a13a0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279818fb70 con 0x7f279807c7c0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f279ca13700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27980dc6d0 con 0x7f27981a3930 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59004/0 (socket says 172.21.2.117:59004) 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 -- 172.21.2.117:0/1916347473 learned_addr learned my addr 172.21.2.117:0/1916347473 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 -- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27981a3930 msgr2=0x7f27980a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27981a3930 0x7f27980a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 -- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279807c7c0 msgr2=0x7f27981a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279807c7c0 0x7f27981a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 -- 172.21.2.117:0/1916347473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279807d220 con 0x7f27980a13a0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.309 7f2796e1d700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f279000a940 tx=0x7f279000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27900078b0 con 0x7f27980a13a0 2021-02-10T11:55:38.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2790004930 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f279ca13700 1 -- 172.21.2.117:0/1916347473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279807d220 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f279ca13700 1 -- 172.21.2.117:0/1916347473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279807d220 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2790007020 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2790007020 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 0x7f277c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.310 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2790032c70 con 0x7f27980a13a0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.311 7f279661c700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 0x7f277c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.311 7f279661c700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 0x7f277c01ac10 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.342 7f279ca13700 1 -- 172.21.2.117:0/1916347473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27980daff0 con 0x7f27980a13a0 2021-02-10T11:55:38.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.345 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2790004720 con 0x7f27980a13a0 2021-02-10T11:55:38.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.491 7f279ca13700 1 -- 172.21.2.117:0/1916347473 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f27980daff0 con 0x7f27980a13a0 2021-02-10T11:55:38.494 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.491 7f278f7fe700 1 -- 172.21.2.117:0/1916347473 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2790007020 con 0x7f27980a13a0 2021-02-10T11:55:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 -- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 msgr2=0x7f277c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 0x7f277c01ac10 crc :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 -- 172.21.2.117:0/1916347473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 msgr2=0x7f27980a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f279000a940 tx=0x7f279000ace0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 -- 172.21.2.117:0/1916347473 shutdown_connections 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f277c018730 0x7f277c01ac10 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27980a13a0 0x7f27980a17a0 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27981a3930 0x7f27980a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 --2- 172.21.2.117:0/1916347473 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f279807c7c0 0x7f27981a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.493 7f279ca13700 1 -- 172.21.2.117:0/1916347473 >> 172.21.2.117:0/1916347473 conn(0x7f2798087090 msgr2=0x7f279808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.494 7f279ca13700 1 -- 172.21.2.117:0/1916347473 shutdown_connections 2021-02-10T11:55:38.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.494 7f279ca13700 1 -- 172.21.2.117:0/1916347473 wait complete. 2021-02-10T11:55:38.505 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000979962 2021-02-10T11:55:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.919 7f2777097700 1 Processor -- start 2021-02-10T11:55:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.919 7f2777097700 1 -- start start 2021-02-10T11:55:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700d18f0 0x7f27700eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f27700eb040 0x7f27700eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f27700ee3e0 0x7f27700ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700ede60 con 0x7f27700eb040 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27700eba60 con 0x7f27700ee3e0 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f2777097700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27700d1cf0 con 0x7f27700d18f0 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f27700eb040 0x7f27700eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f27700eb040 0x7f27700eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59012/0 (socket says 172.21.2.117:59012) 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 -- 172.21.2.117:0/1042506235 learned_addr learned my addr 172.21.2.117:0/1042506235 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 -- 172.21.2.117:0/1042506235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700d18f0 msgr2=0x7f27700eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 --2- 172.21.2.117:0/1042506235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700d18f0 0x7f27700eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 -- 172.21.2.117:0/1042506235 >> v2:172.21.2.125:3300/0 conn(0x7f27700ee3e0 msgr2=0x7f27700ee7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 --2- 172.21.2.117:0/1042506235 >> v2:172.21.2.125:3300/0 conn(0x7f27700ee3e0 0x7f27700ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.920 7f276ffff700 1 -- 172.21.2.117:0/1042506235 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700eee00 con 0x7f27700eb040 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f276ffff700 1 --2- 172.21.2.117:0/1042506235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27700eb040 0x7f27700eb440 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f2760006700 tx=0x7f2760009b80).ready entity=mon.0 client_cookie=8b9caf4608fdd6fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f276f7fe700 1 -- 172.21.2.117:0/1042506235 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f276000c0c0 con 0x7f27700eb040 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f276f7fe700 1 -- 172.21.2.117:0/1042506235 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276000c440 con 0x7f27700eb040 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f276f7fe700 1 -- 172.21.2.117:0/1042506235 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f276000c920 con 0x7f27700eb040 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f2777097700 1 -- 172.21.2.117:0/1042506235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27700eb040 msgr2=0x7f27700eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.921 7f2777097700 1 --2- 172.21.2.117:0/1042506235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27700eb040 0x7f27700eb440 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7f2760006700 tx=0x7f2760009b80).stop 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 -- 172.21.2.117:0/1042506235 shutdown_connections 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 --2- 172.21.2.117:0/1042506235 >> v2:172.21.2.125:3300/0 conn(0x7f27700ee3e0 0x7f27700ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 --2- 172.21.2.117:0/1042506235 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f27700eb040 0x7f27700eb440 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 --2- 172.21.2.117:0/1042506235 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700d18f0 0x7f27700eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 -- 172.21.2.117:0/1042506235 >> 172.21.2.117:0/1042506235 conn(0x7f27700cf060 msgr2=0x7f27700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 -- 172.21.2.117:0/1042506235 shutdown_connections 2021-02-10T11:55:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.922 7f2777097700 1 -- 172.21.2.117:0/1042506235 wait complete. 2021-02-10T11:55:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.936 7f2777097700 1 Processor -- start 2021-02-10T11:55:38.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.936 7f2777097700 1 -- start start 2021-02-10T11:55:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27701a4860 0x7f27700a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700a65a0 0x7f27700a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27700a1820 con 0x7f277009e160 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27700a69a0 con 0x7f27701a4860 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2777097700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277009dc80 con 0x7f27700a65a0 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59014/0 (socket says 172.21.2.117:59014) 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 -- 172.21.2.117:0/1748917377 learned_addr learned my addr 172.21.2.117:0/1748917377 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 -- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700a65a0 msgr2=0x7f27700a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700a65a0 0x7f27700a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 -- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27701a4860 msgr2=0x7f27700a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27701a4860 0x7f27700a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.937 7f2774eb3700 1 -- 172.21.2.117:0/1748917377 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700a23c0 con 0x7f277009e160 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f2774eb3700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f275800b5c0 tx=0x7f275800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f275800bd50 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2758006030 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f27580054d0 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f2777097700 1 -- 172.21.2.117:0/1748917377 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27700a23c0 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f2777097700 1 -- 172.21.2.117:0/1748917377 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27700a23c0 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2758006210 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 0x7f275401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.938 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f27580328f0 con 0x7f277009e160 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.939 7f276ffff700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 0x7f275401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:38.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.940 7f276ffff700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 0x7f275401ac10 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.969 7f2777097700 1 -- 172.21.2.117:0/1748917377 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27700ede60 con 0x7f277009e160 2021-02-10T11:55:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:38.972 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2758006b80 con 0x7f277009e160 2021-02-10T11:55:39.102 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.099 7f2777097700 1 -- 172.21.2.117:0/1748917377 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000979962"}) v1 -- 0x7f27700ede60 con 0x7f2754018730 2021-02-10T11:55:39.105 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:55:39.103 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:55:39.105 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:55:39.103 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:55:39.106 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:55:39.103 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:55:39.106 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:55:39.103 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:55:39.106 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:55:39.103 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:55:39.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.119 7f276d7fa700 1 -- 172.21.2.117:0/1748917377 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2760000990 con 0x7f2754018730 2021-02-10T11:55:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 msgr2=0x7f275401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 0x7f275401ac10 crc :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 msgr2=0x7f27701a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f275800b5c0 tx=0x7f275800b960).stop 2021-02-10T11:55:39.135 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 shutdown_connections 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f27700a65a0 0x7f27700a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f27701a4860 0x7f27700a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2754018730 0x7f275401ac10 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 --2- 172.21.2.117:0/1748917377 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f277009e160 0x7f27701a4240 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 >> 172.21.2.117:0/1748917377 conn(0x7f277009e660 msgr2=0x7f27700bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 shutdown_connections 2021-02-10T11:55:39.136 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.132 7f2777097700 1 -- 172.21.2.117:0/1748917377 wait complete. 2021-02-10T11:55:39.143 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000165360 --group_name group_0000000000979962 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 Processor -- start 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 -- start start 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 0x7fbe4c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c085930 con 0x7fbe4c1b0810 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4c084840 con 0x7fbe4c0e4e40 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.558 7fbe52277700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c084a30 con 0x7fbe4c0e5240 2021-02-10T11:55:39.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4bfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4bfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35174/0 (socket says 172.21.2.117:35174) 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4bfff700 1 -- 172.21.2.117:0/1785797220 learned_addr learned my addr 172.21.2.117:0/1785797220 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 0x7fbe4c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 -- 172.21.2.117:0/1785797220 >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 msgr2=0x7fbe4c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 --2- 172.21.2.117:0/1785797220 >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 -- 172.21.2.117:0/1785797220 >> v1:172.21.2.125:6789/0 conn(0x7fbe4c1b0810 legacy=0x7fbe4c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 -- 172.21.2.117:0/1785797220 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c085280 con 0x7fbe4c0e5240 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4bfff700 1 --2- 172.21.2.117:0/1785797220 >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4b7fe700 1 --2- 172.21.2.117:0/1785797220 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 0x7fbe4c084c60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fbe40006650 tx=0x7fbe40009a80).ready entity=mon.2 client_cookie=6fa2d60a989af94a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4affd700 1 -- 172.21.2.117:0/1785797220 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe400009b0 con 0x7fbe4c0e5240 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4affd700 1 -- 172.21.2.117:0/1785797220 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4000c0c0 con 0x7fbe4c0e5240 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe4affd700 1 -- 172.21.2.117:0/1785797220 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe4000c5e0 con 0x7fbe4c0e5240 2021-02-10T11:55:39.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe52277700 1 -- 172.21.2.117:0/1785797220 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 msgr2=0x7fbe4c084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.560 7fbe52277700 1 --2- 172.21.2.117:0/1785797220 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 0x7fbe4c084c60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fbe40006650 tx=0x7fbe40009a80).stop 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 -- 172.21.2.117:0/1785797220 shutdown_connections 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 --2- 172.21.2.117:0/1785797220 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0e5240 0x7fbe4c084c60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 --2- 172.21.2.117:0/1785797220 >> v2:172.21.2.125:3300/0 conn(0x7fbe4c0e4e40 0x7fbe4c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 -- 172.21.2.117:0/1785797220 >> 172.21.2.117:0/1785797220 conn(0x7fbe4c0ce4e0 msgr2=0x7fbe4c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 -- 172.21.2.117:0/1785797220 shutdown_connections 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.561 7fbe52277700 1 -- 172.21.2.117:0/1785797220 wait complete. 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 Processor -- start 2021-02-10T11:55:39.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 -- start start 2021-02-10T11:55:39.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe4c0973b0 0x7fbe4c097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 0x7fbe4c09b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c18d0c0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c09ae70 con 0x7fbe4c0973b0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.575 7fbe52277700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c09ba10 con 0x7fbe4c0adbd0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59026/0 (socket says 172.21.2.117:59026) 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 -- 172.21.2.117:0/2751202214 learned_addr learned my addr 172.21.2.117:0/2751202214 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4bfff700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe4c0973b0 0x7fbe4c097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe50894700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 0x7fbe4c09b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 -- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 msgr2=0x7fbe4c09b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 0x7fbe4c09b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 -- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe4c0973b0 msgr2=0x7fbe4c097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe4c0973b0 0x7fbe4c097980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 -- 172.21.2.117:0/2751202214 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c09bbf0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe50894700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 0x7fbe4c09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe4b7fe700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7fbe40009630 tx=0x7fbe4000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:39.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe400093a0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4000bd70 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbe40005560 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe52277700 1 -- 172.21.2.117:0/2751202214 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c09bbf0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.576 7fbe52277700 1 -- 172.21.2.117:0/2751202214 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c09bbf0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.577 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbe40005770 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.577 7fbe48ff9700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 0x7fbe2c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.577 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbe400327a0 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.577 7fbe4bfff700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 0x7fbe2c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:39.584 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.578 7fbe4bfff700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 0x7fbe2c01ac10 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:39.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.608 7fbe52277700 1 -- 172.21.2.117:0/2751202214 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c085930 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.610 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbe40005560 con 0x7fbe4c0ad1b0 2021-02-10T11:55:39.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.736 7fbe52277700 1 -- 172.21.2.117:0/2751202214 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000165360", "target": ["mgr", ""], "group_name": "group_0000000000979962"}) v1 -- 0x7fbe4c09ae70 con 0x7fbe2c018730 2021-02-10T11:55:39.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.761 7fbe48ff9700 1 -- 172.21.2.117:0/2751202214 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fbe3c0009b0 con 0x7fbe2c018730 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 msgr2=0x7fbe2c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 0x7fbe2c01ac10 crc :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 msgr2=0x7fbe4c0ad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7fbe40009630 tx=0x7fbe4000b980).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 shutdown_connections 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbe2c018730 0x7fbe2c01ac10 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbe4c0ad1b0 0x7fbe4c0ad5b0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbe4c0adbd0 0x7fbe4c09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 --2- 172.21.2.117:0/2751202214 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbe4c0973b0 0x7fbe4c097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:39.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 >> 172.21.2.117:0/2751202214 conn(0x7fbe4c18cc70 msgr2=0x7fbe4c093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 shutdown_connections 2021-02-10T11:55:39.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:39.769 7fbe52277700 1 -- 172.21.2.117:0/2751202214 wait complete. 2021-02-10T11:55:39.780 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000165360 guest1 --group_name group_0000000000979962 --tenant_id tenant1 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.202 7f4624cc3700 1 Processor -- start 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.202 7f4624cc3700 1 -- start start 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f4624cc3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46201b69a0 0x7f46201b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f4624cc3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46200ede60 con 0x7f46200d18f0 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f4624cc3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46200eba00 con 0x7f46200ef3f0 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f4624cc3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46200d1cf0 con 0x7f46201b69a0 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f461e85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f46200d18f0 0x7f46200eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59284/0 (socket says 172.21.2.117:59284) 2021-02-10T11:55:40.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f461e85e700 1 -- 172.21.2.117:0/3924743735 learned_addr learned my addr 172.21.2.117:0/3924743735 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794004797 0 0) 0x7f4608001180 con 0x7f46200d18f0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.203 7f461d85c700 1 -- 172.21.2.117:0/3924743735 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c002fe0 con 0x7f46200d18f0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461f05f700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46201b69a0 0x7f46201b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3963964617 0 0) 0x7f4608001180 con 0x7f46200d18f0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46201b69a0 msgr2=0x7f46201b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46201b69a0 0x7f46201b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 >> v1:172.21.2.125:6789/0 conn(0x7f46200ef3f0 legacy=0x7f46200eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46201bb3f0 con 0x7f46200d18f0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4608001680 con 0x7f46200d18f0 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 >> v1:172.21.2.117:6789/0 conn(0x7f46200d18f0 legacy=0x7f46200eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f460c003b10 0x7f460c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f460c006840 0x7f460c008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f460c009620 0x7f460c00bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c00c360 con 0x7f460c003b10 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c00c580 con 0x7f460c006840 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461d85c700 1 -- 172.21.2.117:0/3924743735 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c00c7a0 con 0x7f460c009620 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461f05f700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f46201b69a0 0x7f46201b8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f460c003b10 0x7f460c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e05d700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f460c006840 0x7f460c008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461f05f700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f460c009620 0x7f460c00bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 -- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f460c009620 msgr2=0x7f460c00bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f460c009620 0x7f460c00bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 -- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f460c006840 msgr2=0x7f460c008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f460c006840 0x7f460c008d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 -- 172.21.2.117:0/3924743735 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c00ca40 con 0x7f460c003b10 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.204 7f461e85e700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f460c003b10 0x7f460c005fa0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7f4608000e80 tx=0x7f460804bd00).ready entity=mon.0 client_cookie=be731f48764bf500 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:40.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4608001180 con 0x7f460c003b10 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460804e310 con 0x7f460c003b10 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f461d85c700 1 -- 172.21.2.117:0/3924743735 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f460804e830 con 0x7f460c003b10 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 -- 172.21.2.117:0/3924743735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f460c003b10 msgr2=0x7f460c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 --2- 172.21.2.117:0/3924743735 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f460c003b10 0x7f460c005fa0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7f4608000e80 tx=0x7f460804bd00).stop 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f461f05f700 1 -- 172.21.2.117:0/3924743735 reap_dead start 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 -- 172.21.2.117:0/3924743735 shutdown_connections 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 -- 172.21.2.117:0/3924743735 >> 172.21.2.117:0/3924743735 conn(0x7f46200cf060 msgr2=0x7f46200cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 -- 172.21.2.117:0/3924743735 shutdown_connections 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.205 7f4624cc3700 1 -- 172.21.2.117:0/3924743735 wait complete. 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.220 7f4624cc3700 1 Processor -- start 2021-02-10T11:55:40.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.220 7f4624cc3700 1 -- start start 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f462007f6e0 0x7f46200a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46200a9550 0x7f462007e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46200aa220 con 0x7f46201a2ea0 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462007edf0 con 0x7f46200a9550 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46201a38c0 con 0x7f462007f6e0 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59046/0 (socket says 172.21.2.117:59046) 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 -- 172.21.2.117:0/1802036419 learned_addr learned my addr 172.21.2.117:0/1802036419 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461e05d700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46200a9550 0x7f462007e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461e85e700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f462007f6e0 0x7f46200a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 -- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f462007f6e0 msgr2=0x7f46200a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f462007f6e0 0x7f46200a9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 -- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46200a9550 msgr2=0x7f462007e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46200a9550 0x7f462007e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 -- 172.21.2.117:0/1802036419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46201a3ae0 con 0x7f46201a2ea0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f461f05f700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7f4618009940 tx=0x7f461800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4618006030 con 0x7f46201a2ea0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46180062b0 con 0x7f46201a2ea0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46201a3ae0 con 0x7f46201a2ea0 2021-02-10T11:55:40.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46180056b0 con 0x7f46201a2ea0 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.221 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46201a3ae0 con 0x7f46201a2ea0 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.222 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4618000df0 con 0x7f46201a2ea0 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.222 7f46177fe700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 0x7f460401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.222 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4618032c80 con 0x7f46201a2ea0 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.222 7f461e85e700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 0x7f460401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:40.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.223 7f461e85e700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 0x7f460401abc0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:40.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.253 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462007edf0 con 0x7f46201a2ea0 2021-02-10T11:55:40.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.256 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f461800be30 con 0x7f46201a2ea0 2021-02-10T11:55:40.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.381 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000165360", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000979962"}) v1 -- 0x7f462007edf0 con 0x7f46040186c0 2021-02-10T11:55:40.388 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:55:40.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.416 7f46177fe700 1 -- 172.21.2.117:0/1802036419 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f4608001410 con 0x7f46040186c0 2021-02-10T11:55:40.447 INFO:teuthology.orchestra.run.gibba017.stdout:AQC8ySNgKt+qFxAASaHXwOUfzJq1kGJUoQMeCQ== 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 msgr2=0x7f460401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 0x7f460401abc0 crc :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 msgr2=0x7f46201a32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7f4618009940 tx=0x7f461800bcf0).stop 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 shutdown_connections 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f46040186c0 0x7f460401abc0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46201a2ea0 0x7f46201a32a0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f46200a9550 0x7f462007e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 --2- 172.21.2.117:0/1802036419 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f462007f6e0 0x7f46200a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:40.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 >> 172.21.2.117:0/1802036419 conn(0x7f46200b19e0 msgr2=0x7f46200aab50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:40.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 shutdown_connections 2021-02-10T11:55:40.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:40.446 7f4624cc3700 1 -- 172.21.2.117:0/1802036419 wait complete. 2021-02-10T11:55:40.457 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:55:40.806 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000979962 2021-02-10T11:55:40.806 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000979962:subvolume_0000000000165360.meta 2021-02-10T11:55:40.806 INFO:teuthology.orchestra.run.gibba017.stdout:$guest1.meta 2021-02-10T11:55:40.807 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:55:41.173 INFO:teuthology.orchestra.run.gibba017.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000979962/subvolume_0000000000165360": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:55:41.174 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:55:41.538 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000165360 guest1 --group_name group_0000000000979962 --tenant_id tenant1 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.962 7f01bbbea700 1 Processor -- start 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.962 7f01bbbea700 1 -- start start 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01bbbea700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01b41b69a0 0x7f01b41b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01bbbea700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40ede60 con 0x7f01b40ef3f0 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01bbbea700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b40eba00 con 0x7f01b40d18f0 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01bbbea700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b40d1cf0 con 0x7f01b41b69a0 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b9205700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f01b40ef3f0 0x7f01b40eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59310/0 (socket says 172.21.2.117:59310) 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b9205700 1 -- 172.21.2.117:0/1066137737 learned_addr learned my addr 172.21.2.117:0/1066137737 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:41.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160209735 0 0) 0x7f01a4001180 con 0x7f01b40ef3f0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a0002fe0 con 0x7f01b40ef3f0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01ba207700 1 --2- 172.21.2.117:0/1066137737 >> v2:172.21.2.125:3300/0 conn(0x7f01b41b69a0 0x7f01b41b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 843648752 0 0) 0x7f01a4001180 con 0x7f01b40ef3f0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 >> v2:172.21.2.125:3300/0 conn(0x7f01b41b69a0 msgr2=0x7f01b41b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 --2- 172.21.2.117:0/1066137737 >> v2:172.21.2.125:3300/0 conn(0x7f01b41b69a0 0x7f01b41b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 >> v1:172.21.2.125:6789/0 conn(0x7f01b40d18f0 legacy=0x7f01b40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b41bb3f0 con 0x7f01b40ef3f0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f01a4001680 con 0x7f01b40ef3f0 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 >> v1:172.21.2.117:6789/0 conn(0x7f01b40ef3f0 legacy=0x7f01b40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.963 7f01b8a04700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01a0003a70 0x7f01a0005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01a00067a0 0x7f01a0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 0x7f01a000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.983 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a000c310 con 0x7f01a00095a0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a000c530 con 0x7f01a0003a70 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a000c750 con 0x7f01a00067a0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 0x7f01a000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b9205700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01a00067a0 0x7f01a0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b9a06700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01a0003a70 0x7f01a0005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 -- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01a00067a0 msgr2=0x7f01a0008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01a00067a0 0x7f01a0008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 -- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01a0003a70 msgr2=0x7f01a0005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01a0003a70 0x7f01a0005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 -- 172.21.2.117:0/1066137737 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a000c9f0 con 0x7f01a00095a0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b9a06700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01a0003a70 0x7f01a0005f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 -- 172.21.2.117:0/1066137737 reap_dead start 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01ba207700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 0x7f01a000bac0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f01ac0068b0 tx=0x7f01ac04f050).ready entity=mon.0 client_cookie=8cecc384e47fef8b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:41.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.964 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac04f440 con 0x7f01a00095a0 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac003030 con 0x7f01a00095a0 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01b8a04700 1 -- 172.21.2.117:0/1066137737 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac04f940 con 0x7f01a00095a0 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 -- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 msgr2=0x7f01a000bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 0x7f01a000bac0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7f01ac0068b0 tx=0x7f01ac04f050).stop 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 -- 172.21.2.117:0/1066137737 shutdown_connections 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 --2- 172.21.2.117:0/1066137737 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01a00095a0 0x7f01a000bac0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 -- 172.21.2.117:0/1066137737 >> 172.21.2.117:0/1066137737 conn(0x7f01b40cf060 msgr2=0x7f01b40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 -- 172.21.2.117:0/1066137737 shutdown_connections 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.965 7f01bbbea700 1 -- 172.21.2.117:0/1066137737 wait complete. 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 Processor -- start 2021-02-10T11:55:41.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 -- start start 2021-02-10T11:55:41.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b40a5e20 0x7f01b40a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b408c900 0x7f01b408cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b408dd10 con 0x7f01b408d320 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b41a4790 con 0x7f01b40a5e20 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01bbbea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b41a4970 con 0x7f01b408c900 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01ba207700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01ba207700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59074/0 (socket says 172.21.2.117:59074) 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.980 7f01ba207700 1 -- 172.21.2.117:0/1485401751 learned_addr learned my addr 172.21.2.117:0/1485401751 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 -- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b408c900 msgr2=0x7f01b408cd00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b408c900 0x7f01b408cd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 -- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b40a5e20 msgr2=0x7f01b40a6220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b40a5e20 0x7f01b40a6220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:41.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 -- 172.21.2.117:0/1485401751 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b408d790 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01ba207700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f01ac04d330 tx=0x7f01ac04fd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac04d8f0 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac006240 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01ac0056b0 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b408d790 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.981 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b408eba0 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.982 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01ac006420 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.982 7f01aa7fc700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.982 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f01ac076a90 con 0x7f01b408d320 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.982 7f01b9a06700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:41.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:41.983 7f01b9a06700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 0x7f019c01abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:42.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.012 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b40db020 con 0x7f01b408d320 2021-02-10T11:55:42.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.015 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01ac069130 con 0x7f01b408d320 2021-02-10T11:55:42.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.140 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000165360", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000979962"}) v1 -- 0x7f01b408eba0 con 0x7f019c0186c0 2021-02-10T11:55:42.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.205 7f01aa7fc700 1 -- 172.21.2.117:0/1485401751 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f01b0006bc0 con 0x7f019c0186c0 2021-02-10T11:55:42.255 INFO:teuthology.orchestra.run.gibba017.stdout:AQC8ySNgKt+qFxAASaHXwOUfzJq1kGJUoQMeCQ== 2021-02-10T11:55:42.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.255 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 msgr2=0x7f019c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:42.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.255 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 0x7f019c01abc0 crc :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:42.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.255 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 msgr2=0x7f01b408e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:42.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.255 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f01ac04d330 tx=0x7f01ac04fd10).stop 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 shutdown_connections 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f019c0186c0 0x7f019c01abc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b408d320 0x7f01b408e290 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b408c900 0x7f01b408cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 --2- 172.21.2.117:0/1485401751 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b40a5e20 0x7f01b40a6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 >> 172.21.2.117:0/1485401751 conn(0x7f01b4099ac0 msgr2=0x7f01b40c0740 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 shutdown_connections 2021-02-10T11:55:42.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:42.256 7f01bbbea700 1 -- 172.21.2.117:0/1485401751 wait complete. 2021-02-10T11:55:42.265 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:55:42.627 INFO:teuthology.orchestra.run.gibba017.stdout:{"tenant_id": "tenant1", "version": 5, "compat_version": 1, "subvolumes": {"group_0000000000979962/subvolume_0000000000165360": {"access_level": "rw", "dirty": false}}, "dirty": false} 2021-02-10T11:55:42.628 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000165360 guest1 --group_name group_0000000000979962 2021-02-10T11:55:43.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.043 7fa91c980700 1 Processor -- start 2021-02-10T11:55:43.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.043 7fa91c980700 1 -- start start 2021-02-10T11:55:43.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91c980700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 0x7fa9180e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91c980700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa9180e7180 0x7fa9180e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.062 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91c980700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9180e7580 con 0x7fa9180e7d70 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91c980700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180e92f0 con 0x7fa9180e7180 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91c980700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9180e5ce0 con 0x7fa9181b07f0 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa915e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa9180e7d70 0x7fa9180e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59332/0 (socket says 172.21.2.117:59332) 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa915e1b700 1 -- 172.21.2.117:0/387159120 learned_addr learned my addr 172.21.2.117:0/387159120 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa916e1d700 1 --2- 172.21.2.117:0/387159120 >> v2:172.21.2.125:3300/0 conn(0x7fa9180e7180 0x7fa9180e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.044 7fa91661c700 1 --2- 172.21.2.117:0/387159120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 0x7fa9180e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334800297 0 0) 0x7fa9000010b0 con 0x7fa9180e7d70 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa904002fe0 con 0x7fa9180e7d70 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91661c700 1 -- 172.21.2.117:0/387159120 >> v2:172.21.2.125:3300/0 conn(0x7fa9180e7180 msgr2=0x7fa9180e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91661c700 1 --2- 172.21.2.117:0/387159120 >> v2:172.21.2.125:3300/0 conn(0x7fa9180e7180 0x7fa9180e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91661c700 1 -- 172.21.2.117:0/387159120 >> v1:172.21.2.117:6789/0 conn(0x7fa9180e7d70 legacy=0x7fa9180e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3490418335 0 0) 0x7fa9000010b0 con 0x7fa9180e7d70 2021-02-10T11:55:43.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91661c700 1 -- 172.21.2.117:0/387159120 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180e77f0 con 0x7fa9181b07f0 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91661c700 1 --2- 172.21.2.117:0/387159120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 0x7fa9180e4e90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa908006650 tx=0x7fa908009a00).ready entity=mon.2 client_cookie=f3da07db0f0f2292 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9080009b0 con 0x7fa9181b07f0 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90800c0c0 con 0x7fa9181b07f0 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91561a700 1 -- 172.21.2.117:0/387159120 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa90800c5a0 con 0x7fa9181b07f0 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91c980700 1 -- 172.21.2.117:0/387159120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 msgr2=0x7fa9180e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.045 7fa91c980700 1 --2- 172.21.2.117:0/387159120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 0x7fa9180e4e90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa908006650 tx=0x7fa908009a00).stop 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 -- 172.21.2.117:0/387159120 shutdown_connections 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 --2- 172.21.2.117:0/387159120 >> v2:172.21.2.125:3300/0 conn(0x7fa9180e7180 0x7fa9180e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 --2- 172.21.2.117:0/387159120 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9181b07f0 0x7fa9180e4e90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 -- 172.21.2.117:0/387159120 >> 172.21.2.117:0/387159120 conn(0x7fa9180ce500 msgr2=0x7fa9180e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 -- 172.21.2.117:0/387159120 shutdown_connections 2021-02-10T11:55:43.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.046 7fa91c980700 1 -- 172.21.2.117:0/387159120 wait complete. 2021-02-10T11:55:43.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 Processor -- start 2021-02-10T11:55:43.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 -- start start 2021-02-10T11:55:43.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa91818cee0 0x7fa9180afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9180b0050 0x7fa91810cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9181a72b0 con 0x7fa91818cae0 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9181085d0 con 0x7fa9180b0050 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91c980700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91810d400 con 0x7fa91818cee0 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:43.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91661c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59090/0 (socket says 172.21.2.117:59090) 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.061 7fa91661c700 1 -- 172.21.2.117:0/1734946970 learned_addr learned my addr 172.21.2.117:0/1734946970 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 -- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa91818cee0 msgr2=0x7fa9180afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa91818cee0 0x7fa9180afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 -- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9180b0050 msgr2=0x7fa91810cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9180b0050 0x7fa91810cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 -- 172.21.2.117:0/1734946970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91810d620 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91661c700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fa90800b640 tx=0x7fa90800b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa908006ae0 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90800bdd0 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91c980700 1 -- 172.21.2.117:0/1734946970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91810d620 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa91c980700 1 -- 172.21.2.117:0/1734946970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91810d620 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.062 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9080022b0 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.063 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa908000c90 con 0x7fa91818cae0 2021-02-10T11:55:43.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.063 7fa90effd700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 0x7fa8fc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:43.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.063 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa908032770 con 0x7fa91818cae0 2021-02-10T11:55:43.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.063 7fa915e1b700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 0x7fa8fc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:43.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.064 7fa915e1b700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 0x7fa8fc01ac10 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:43.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.094 7fa91c980700 1 -- 172.21.2.117:0/1734946970 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa918080e20 con 0x7fa91818cae0 2021-02-10T11:55:43.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.096 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa908006a10 con 0x7fa91818cae0 2021-02-10T11:55:43.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.221 7fa91c980700 1 -- 172.21.2.117:0/1734946970 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000165360", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000979962"}) v1 -- 0x7fa9180e77f0 con 0x7fa8fc018730 2021-02-10T11:55:43.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.258 7fa90effd700 1 -- 172.21.2.117:0/1734946970 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa9000009b0 con 0x7fa8fc018730 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 msgr2=0x7fa8fc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 0x7fa8fc01ac10 crc :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 msgr2=0x7fa9181a6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7fa90800b640 tx=0x7fa90800b9e0).stop 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 shutdown_connections 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9180b0050 0x7fa91810cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:43.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa91818cee0 0x7fa9180afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:43.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8fc018730 0x7fa8fc01ac10 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 --2- 172.21.2.117:0/1734946970 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa91818cae0 0x7fa9181a6c90 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:43.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 >> 172.21.2.117:0/1734946970 conn(0x7fa91818a7e0 msgr2=0x7fa918195e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:43.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 shutdown_connections 2021-02-10T11:55:43.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:43.286 7fa91c980700 1 -- 172.21.2.117:0/1734946970 wait complete. 2021-02-10T11:55:43.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:43.297 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:43.297 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:43.634 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:55:43.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:55:43.634 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:55:43.634 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:55:43.635 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:43.988 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[184951]: fuse finished with error 0 and tester_r 0 2021-02-10T11:55:43.989 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:43.989 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:43.989 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:44.325 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:55:44.325 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:55:44.325 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:55:44.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:55:44.326 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:44.662 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:55:45.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.071 7fbadfc20700 1 Processor -- start 2021-02-10T11:55:45.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 -- start start 2021-02-10T11:55:45.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad81b07f0 0x7fbad80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbad80e7180 0x7fbad80e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad80e7580 con 0x7fbad80e7d70 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad80e92f0 con 0x7fbad80e7180 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadfc20700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80e5ce0 con 0x7fbad81b07f0 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadd23b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fbad80e7d70 0x7fbad80e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59344/0 (socket says 172.21.2.117:59344) 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbadd23b700 1 -- 172.21.2.117:0/818750960 learned_addr learned my addr 172.21.2.117:0/818750960 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.072 7fbade23d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbad80e7180 0x7fbad80e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbade23d700 1 -- 172.21.2.117:0/818750960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad81b07f0 msgr2=0x7fbad80e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbade23d700 1 --2- 172.21.2.117:0/818750960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad81b07f0 0x7fbad80e4e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbade23d700 1 -- 172.21.2.117:0/818750960 >> v1:172.21.2.117:6789/0 conn(0x7fbad80e7d70 legacy=0x7fbad80e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadca3a700 1 -- 172.21.2.117:0/818750960 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047190925 0 0) 0x7fbad40010b0 con 0x7fbad80e7d70 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbade23d700 1 -- 172.21.2.117:0/818750960 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80e77f0 con 0x7fbad80e7180 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbade23d700 1 --2- 172.21.2.117:0/818750960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad80e7180 0x7fbad80e8960 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 rx=0x7fbad0006e30 tx=0x7fbad0009de0).ready entity=mon.0 client_cookie=b12d2394d0444b00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadca3a700 1 -- 172.21.2.117:0/818750960 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad000b300 con 0x7fbad80e7180 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadca3a700 1 -- 172.21.2.117:0/818750960 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad000b580 con 0x7fbad80e7180 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadca3a700 1 -- 172.21.2.117:0/818750960 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbad000baa0 con 0x7fbad80e7180 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 -- 172.21.2.117:0/818750960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad80e7180 msgr2=0x7fbad80e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 --2- 172.21.2.117:0/818750960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad80e7180 0x7fbad80e8960 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 rx=0x7fbad0006e30 tx=0x7fbad0009de0).stop 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 -- 172.21.2.117:0/818750960 shutdown_connections 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 --2- 172.21.2.117:0/818750960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad80e7180 0x7fbad80e8960 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 --2- 172.21.2.117:0/818750960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad81b07f0 0x7fbad80e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.073 7fbadfc20700 1 -- 172.21.2.117:0/818750960 >> 172.21.2.117:0/818750960 conn(0x7fbad80ce500 msgr2=0x7fbad80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.074 7fbadfc20700 1 -- 172.21.2.117:0/818750960 shutdown_connections 2021-02-10T11:55:45.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.074 7fbadfc20700 1 -- 172.21.2.117:0/818750960 wait complete. 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 Processor -- start 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 -- start start 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbad8199a70 0x7fbad80b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad80b1af0 0x7fbad80b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8105f50 con 0x7fbad808ef00 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad808f300 con 0x7fbad8199a70 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadfc20700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80c3f80 con 0x7fbad80b1af0 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadda3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadda3c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59100/0 (socket says 172.21.2.117:59100) 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.088 7fbadda3c700 1 -- 172.21.2.117:0/751228979 learned_addr learned my addr 172.21.2.117:0/751228979 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 -- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad80b1af0 msgr2=0x7fbad80b1ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad80b1af0 0x7fbad80b1ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 -- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbad8199a70 msgr2=0x7fbad80b14d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbad8199a70 0x7fbad80b14d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 -- 172.21.2.117:0/751228979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80b2510 con 0x7fbad808ef00 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadda3c700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 rx=0x7fbacc00b5c0 tx=0x7fbacc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbacc00bd50 con 0x7fbad808ef00 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc006030 con 0x7fbad808ef00 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbacc0054d0 con 0x7fbad808ef00 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadfc20700 1 -- 172.21.2.117:0/751228979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad80b2510 con 0x7fbad808ef00 2021-02-10T11:55:45.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.089 7fbadfc20700 1 -- 172.21.2.117:0/751228979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad80b2510 con 0x7fbad808ef00 2021-02-10T11:55:45.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.090 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbacc006210 con 0x7fbad808ef00 2021-02-10T11:55:45.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.090 7fbaca7fc700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 0x7fbac001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.090 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbacc0326d0 con 0x7fbad808ef00 2021-02-10T11:55:45.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.090 7fbadd23b700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 0x7fbac001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.091 7fbadd23b700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 0x7fbac001abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.122 7fbadfc20700 1 -- 172.21.2.117:0/751228979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad81b8070 con 0x7fbad808ef00 2021-02-10T11:55:45.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.125 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbacc006b80 con 0x7fbad808ef00 2021-02-10T11:55:45.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.252 7fbadfc20700 1 -- 172.21.2.117:0/751228979 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7fbad81b8070 con 0x7fbad808ef00 2021-02-10T11:55:45.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.253 7fbaca7fc700 1 -- 172.21.2.117:0/751228979 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v99) v1 ==== 117+0+0 (secure 0 0 0) 0x7fbacc006210 con 0x7fbad808ef00 2021-02-10T11:55:45.255 INFO:teuthology.orchestra.run.gibba017.stderr:entity client.guest1 does not exist 2021-02-10T11:55:45.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.254 7fbadfc20700 1 -- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 msgr2=0x7fbac001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.254 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 0x7fbac001abc0 crc :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.254 7fbadfc20700 1 -- 172.21.2.117:0/751228979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 msgr2=0x7fbad8199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 rx=0x7fbacc00b5c0 tx=0x7fbacc00b960).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 -- 172.21.2.117:0/751228979 shutdown_connections 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbad80b1af0 0x7fbad80b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbad8199a70 0x7fbad80b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbac00186c0 0x7fbac001abc0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 --2- 172.21.2.117:0/751228979 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbad808ef00 0x7fbad8199450 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 -- 172.21.2.117:0/751228979 >> 172.21.2.117:0/751228979 conn(0x7fbad80b97b0 msgr2=0x7fbad8098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 -- 172.21.2.117:0/751228979 shutdown_connections 2021-02-10T11:55:45.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.255 7fbadfc20700 1 -- 172.21.2.117:0/751228979 wait complete. 2021-02-10T11:55:45.265 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000165360 --group_name group_0000000000979962 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 Processor -- start 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 -- start start 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f44a00e5240 0x7f44a0084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a01b0810 0x7f44a0085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a00e5cc0 con 0x7f44a00e5240 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a0085930 con 0x7f44a00e4e40 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.680 7f44a7c9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0084840 con 0x7f44a01b0810 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f44a00e5240 0x7f44a0084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a5abb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a5abb700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35258/0 (socket says 172.21.2.117:35258) 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f44a00e5240 0x7f44a0084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59106/0 (socket says 172.21.2.117:59106) 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a5abb700 1 -- 172.21.2.117:0/3376740221 learned_addr learned my addr 172.21.2.117:0/3376740221 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a62bc700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a01b0810 0x7f44a0085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 -- 172.21.2.117:0/3376740221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a01b0810 msgr2=0x7f44a0085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a01b0810 0x7f44a0085280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 -- 172.21.2.117:0/3376740221 >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 msgr2=0x7f44a01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 --2- 172.21.2.117:0/3376740221 >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 -- 172.21.2.117:0/3376740221 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a0085eb0 con 0x7f44a00e5240 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a5abb700 1 --2- 172.21.2.117:0/3376740221 >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a52ba700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00e5240 0x7f44a0084c60 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f4494006650 tx=0x7f4494009a80).ready entity=mon.0 client_cookie=fe8fbf0e24557cf2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.681 7f44a4ab9700 1 -- 172.21.2.117:0/3376740221 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f449400c0c0 con 0x7f44a00e5240 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a4ab9700 1 -- 172.21.2.117:0/3376740221 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449400c3c0 con 0x7f44a00e5240 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a4ab9700 1 -- 172.21.2.117:0/3376740221 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f449400c810 con 0x7f44a00e5240 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 -- 172.21.2.117:0/3376740221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00e5240 msgr2=0x7f44a0084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00e5240 0x7f44a0084c60 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f4494006650 tx=0x7f4494009a80).stop 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 -- 172.21.2.117:0/3376740221 shutdown_connections 2021-02-10T11:55:45.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a01b0810 0x7f44a0085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 --2- 172.21.2.117:0/3376740221 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00e5240 0x7f44a0084c60 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 --2- 172.21.2.117:0/3376740221 >> v2:172.21.2.125:3300/0 conn(0x7f44a00e4e40 0x7f44a01b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 -- 172.21.2.117:0/3376740221 >> 172.21.2.117:0/3376740221 conn(0x7f44a00ce4e0 msgr2=0x7f44a00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 -- 172.21.2.117:0/3376740221 shutdown_connections 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.682 7f44a7c9f700 1 -- 172.21.2.117:0/3376740221 wait complete. 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.697 7f44a7c9f700 1 Processor -- start 2021-02-10T11:55:45.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.697 7f44a7c9f700 1 -- start start 2021-02-10T11:55:45.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a00c2000 0x7f44a0191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44a00c14d0 0x7f44a00c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0191ab0 con 0x7f44a00b6c00 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a01916a0 con 0x7f44a00c14d0 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a7c9f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0191880 con 0x7f44a00c2000 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59112/0 (socket says 172.21.2.117:59112) 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 -- 172.21.2.117:0/839272561 learned_addr learned my addr 172.21.2.117:0/839272561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a52ba700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44a00c14d0 0x7f44a00c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a5abb700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a00c2000 0x7f44a0191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 -- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a00c2000 msgr2=0x7f44a0191080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a00c2000 0x7f44a0191080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 -- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44a00c14d0 msgr2=0x7f44a00c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44a00c14d0 0x7f44a00c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.698 7f44a62bc700 1 -- 172.21.2.117:0/839272561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a00b7620 con 0x7f44a00b6c00 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44a62bc700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f449800b970 tx=0x7f449800bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4498000f90 con 0x7f44a00b6c00 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498006aa0 con 0x7f44a00b6c00 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4498001680 con 0x7f44a00b6c00 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a00b7620 con 0x7f44a00b6c00 2021-02-10T11:55:45.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.699 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a00b7620 con 0x7f44a00b6c00 2021-02-10T11:55:45.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.700 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4498000df0 con 0x7f44a00b6c00 2021-02-10T11:55:45.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.700 7f44927fc700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 0x7f448801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:45.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.700 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f44980328d0 con 0x7f44a00b6c00 2021-02-10T11:55:45.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.700 7f44a5abb700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 0x7f448801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:45.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.700 7f44a5abb700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 0x7f448801abc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:45.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.730 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a010c2b0 con 0x7f44a00b6c00 2021-02-10T11:55:45.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.733 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f44980066f0 con 0x7f44a00b6c00 2021-02-10T11:55:45.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.859 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000165360", "target": ["mgr", ""], "group_name": "group_0000000000979962"}) v1 -- 0x7f44a010c2b0 con 0x7f44880186c0 2021-02-10T11:55:45.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.870 7f44927fc700 1 -- 172.21.2.117:0/839272561 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f449c0009b0 con 0x7f44880186c0 2021-02-10T11:55:45.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 msgr2=0x7f448801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 0x7f448801abc0 crc :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 msgr2=0x7f44a00b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f449800b970 tx=0x7f449800bd10).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 shutdown_connections 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44880186c0 0x7f448801abc0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f44a00b6c00 0x7f44a00b7000 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f44a00c14d0 0x7f44a00c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 --2- 172.21.2.117:0/839272561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f44a00c2000 0x7f44a0191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:45.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 >> 172.21.2.117:0/839272561 conn(0x7f44a00acec0 msgr2=0x7f44a00b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:45.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 shutdown_connections 2021-02-10T11:55:45.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:45.877 7f44a7c9f700 1 -- 172.21.2.117:0/839272561 wait complete. 2021-02-10T11:55:45.888 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000979962 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.301 7f177e049700 1 Processor -- start 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.301 7f177e049700 1 -- start start 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f177e049700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f17780d18f0 0x7f17780eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f177e049700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17780eb040 0x7f17780eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f177e049700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17780ede60 con 0x7f17780ee3e0 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f177e049700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780ebad0 con 0x7f17780d18f0 2021-02-10T11:55:46.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f177e049700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780d1cf0 con 0x7f17780eb040 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f1777fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17780ee3e0 0x7f17780ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59368/0 (socket says 172.21.2.117:59368) 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f1777fff700 1 -- 172.21.2.117:0/2975951492 learned_addr learned my addr 172.21.2.117:0/2975951492 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f17777fe700 1 --2- 172.21.2.117:0/2975951492 >> v2:172.21.2.117:3300/0 conn(0x7f17780d18f0 0x7f17780eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f1776ffd700 1 --2- 172.21.2.117:0/2975951492 >> v2:172.21.2.125:3300/0 conn(0x7f17780eb040 0x7f17780eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.302 7f17767fc700 1 -- 172.21.2.117:0/2975951492 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311617719 0 0) 0x7f1770000f90 con 0x7f17780ee3e0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17767fc700 1 -- 172.21.2.117:0/2975951492 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1760002fe0 con 0x7f17780ee3e0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17777fe700 1 -- 172.21.2.117:0/2975951492 >> v2:172.21.2.125:3300/0 conn(0x7f17780eb040 msgr2=0x7f17780eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17777fe700 1 --2- 172.21.2.117:0/2975951492 >> v2:172.21.2.125:3300/0 conn(0x7f17780eb040 0x7f17780eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17777fe700 1 -- 172.21.2.117:0/2975951492 >> v1:172.21.2.117:6789/0 conn(0x7f17780ee3e0 legacy=0x7f17780ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17777fe700 1 -- 172.21.2.117:0/2975951492 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780eefd0 con 0x7f17780d18f0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17777fe700 1 --2- 172.21.2.117:0/2975951492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17780d18f0 0x7f17780eaa20 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f1768006650 tx=0x7f1768009a80).ready entity=mon.0 client_cookie=4d5a6e94a2bb4a5e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:46.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17767fc700 1 -- 172.21.2.117:0/2975951492 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17680009b0 con 0x7f17780d18f0 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17767fc700 1 -- 172.21.2.117:0/2975951492 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176800c0c0 con 0x7f17780d18f0 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f17767fc700 1 -- 172.21.2.117:0/2975951492 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f176800c510 con 0x7f17780d18f0 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f177e049700 1 -- 172.21.2.117:0/2975951492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17780d18f0 msgr2=0x7f17780eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.303 7f177e049700 1 --2- 172.21.2.117:0/2975951492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17780d18f0 0x7f17780eaa20 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 rx=0x7f1768006650 tx=0x7f1768009a80).stop 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 -- 172.21.2.117:0/2975951492 shutdown_connections 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 --2- 172.21.2.117:0/2975951492 >> v2:172.21.2.125:3300/0 conn(0x7f17780eb040 0x7f17780eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 --2- 172.21.2.117:0/2975951492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17780d18f0 0x7f17780eaa20 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 -- 172.21.2.117:0/2975951492 >> 172.21.2.117:0/2975951492 conn(0x7f17780cf060 msgr2=0x7f17780cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 -- 172.21.2.117:0/2975951492 shutdown_connections 2021-02-10T11:55:46.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.304 7f177e049700 1 -- 172.21.2.117:0/2975951492 wait complete. 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.318 7f177e049700 1 Processor -- start 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 -- start start 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1778094090 0x7f1778090e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17781a71f0 0x7f17781a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780b20e0 con 0x7f17781a7c10 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780945f0 con 0x7f17781a71f0 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f177e049700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17780a1280 con 0x7f1778094090 2021-02-10T11:55:46.324 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f1777fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f1777fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59122/0 (socket says 172.21.2.117:59122) 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.319 7f1777fff700 1 -- 172.21.2.117:0/941111524 learned_addr learned my addr 172.21.2.117:0/941111524 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 -- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1778094090 msgr2=0x7f1778090e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1778094090 0x7f1778090e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 -- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17781a71f0 msgr2=0x7f17781a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17781a71f0 0x7f17781a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 -- 172.21.2.117:0/941111524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780a14a0 con 0x7f17781a7c10 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f1777fff700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f1770009980 tx=0x7f177000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f177000be10 con 0x7f17781a7c10 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770006080 con 0x7f17781a7c10 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1770004720 con 0x7f17781a7c10 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f177e049700 1 -- 172.21.2.117:0/941111524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17780a14a0 con 0x7f17781a7c10 2021-02-10T11:55:46.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.320 7f177e049700 1 -- 172.21.2.117:0/941111524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17780a14a0 con 0x7f17781a7c10 2021-02-10T11:55:46.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.321 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1770006260 con 0x7f17781a7c10 2021-02-10T11:55:46.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.321 7f175ffff700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 0x7f175801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:46.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.321 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1770032cc0 con 0x7f17781a7c10 2021-02-10T11:55:46.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.321 7f17777fe700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 0x7f175801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:46.326 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.322 7f17777fe700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 0x7f175801abc0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:46.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.351 7f177e049700 1 -- 172.21.2.117:0/941111524 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17780daff0 con 0x7f17781a7c10 2021-02-10T11:55:46.356 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.354 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1770006290 con 0x7f17781a7c10 2021-02-10T11:55:46.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.480 7f177e049700 1 -- 172.21.2.117:0/941111524 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000979962"}) v1 -- 0x7f17780ede60 con 0x7f1758018730 2021-02-10T11:55:46.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.484 7f175ffff700 1 -- 172.21.2.117:0/941111524 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f1768000940 con 0x7f1758018730 2021-02-10T11:55:46.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 -- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 msgr2=0x7f175801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 0x7f175801abc0 crc :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 -- 172.21.2.117:0/941111524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 msgr2=0x7f17780a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 rx=0x7f1770009980 tx=0x7f177000bcd0).stop 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 -- 172.21.2.117:0/941111524 shutdown_connections 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1758018730 0x7f175801abc0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f17781a7c10 0x7f17780a0c60 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:46.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17781a71f0 0x7f17781a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:46.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 --2- 172.21.2.117:0/941111524 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1778094090 0x7f1778090e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:46.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.489 7f177e049700 1 -- 172.21.2.117:0/941111524 >> 172.21.2.117:0/941111524 conn(0x7f17780b8180 msgr2=0x7f17780b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:46.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.490 7f177e049700 1 -- 172.21.2.117:0/941111524 shutdown_connections 2021-02-10T11:55:46.493 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:46.490 7f177e049700 1 -- 172.21.2.117:0/941111524 wait complete. 2021-02-10T11:55:46.500 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:55:46.500 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:46.847 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:47.191 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:55:47.192 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:47.192 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:47.192 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:47.526 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:55:47.527 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:55:47.527 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:55:47.527 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:55:47.527 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:47.776 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:55:47.780 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:55:47.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[184311]: fuse finished with error 0 and tester_r 0 2021-02-10T11:55:47.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:47.881 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:47.881 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:48.229 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:55:48.229 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:55:48.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:55:48.230 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:48.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:48.565 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:48.565 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:48.900 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:55:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:48.900 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:55:48.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:55:48.901 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:49.236 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:49.236 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:49.236 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:55:49.570 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:55:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:49.571 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:55:49.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:55:49.571 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:55:49.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:49.905 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:49.905 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:55:50.238 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:55:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:50.239 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:55:50.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:55:50.239 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:55:50.574 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:55:51.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 Processor -- start 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 -- start start 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee41b07f0 0x7f0ee40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ee40e7d70 0x7f0ee40e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee40e77f0 con 0x7f0ee40e8790 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee40e92f0 con 0x7f0ee40e7d70 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.988 7f0eeb295700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40e5ce0 con 0x7f0ee41b07f0 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ee40e7d70 0x7f0ee40e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f0ee40e7d70 0x7f0ee40e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59130/0 (socket says 172.21.2.117:59130) 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 -- 172.21.2.117:0/1319965192 learned_addr learned my addr 172.21.2.117:0/1319965192 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee90b1700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee41b07f0 0x7f0ee40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 -- 172.21.2.117:0/1319965192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee41b07f0 msgr2=0x7f0ee40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee41b07f0 0x7f0ee40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 -- 172.21.2.117:0/1319965192 >> v1:172.21.2.125:6789/0 conn(0x7f0ee40e8790 legacy=0x7f0ee40e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 -- 172.21.2.117:0/1319965192 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee40e54b0 con 0x7f0ee40e7d70 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ee88b0700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee40e7d70 0x7f0ee40e8170 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f0ee0006700 tx=0x7f0ee0009b80).ready entity=mon.0 client_cookie=8d4b1a24c6189886 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ed7fff700 1 -- 172.21.2.117:0/1319965192 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ee000c0c0 con 0x7f0ee40e7d70 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ed7fff700 1 -- 172.21.2.117:0/1319965192 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee000c440 con 0x7f0ee40e7d70 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0ed7fff700 1 -- 172.21.2.117:0/1319965192 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ee000c960 con 0x7f0ee40e7d70 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0eeb295700 1 -- 172.21.2.117:0/1319965192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee40e7d70 msgr2=0x7f0ee40e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.989 7f0eeb295700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee40e7d70 0x7f0ee40e8170 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f0ee0006700 tx=0x7f0ee0009b80).stop 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 -- 172.21.2.117:0/1319965192 shutdown_connections 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee40e7d70 0x7f0ee40e8170 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 --2- 172.21.2.117:0/1319965192 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee41b07f0 0x7f0ee40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 -- 172.21.2.117:0/1319965192 >> 172.21.2.117:0/1319965192 conn(0x7f0ee40ce500 msgr2=0x7f0ee40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 -- 172.21.2.117:0/1319965192 shutdown_connections 2021-02-10T11:55:51.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:50.990 7f0eeb295700 1 -- 172.21.2.117:0/1319965192 wait complete. 2021-02-10T11:55:51.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.005 7f0eeb295700 1 Processor -- start 2021-02-10T11:55:51.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.005 7f0eeb295700 1 -- start start 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.005 7f0eeb295700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.005 7f0eeb295700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee410b430 0x7f0ee4197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0eeb295700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 0x7f0ee4197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0eeb295700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4198390 con 0x7f0ee4198800 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0eeb295700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40aeac0 con 0x7f0ee4197970 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0eeb295700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40aece0 con 0x7f0ee410b430 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59134/0 (socket says 172.21.2.117:59134) 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 -- 172.21.2.117:0/1832467271 learned_addr learned my addr 172.21.2.117:0/1832467271 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee88b0700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee410b430 0x7f0ee4197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee98b2700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 0x7f0ee4197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 -- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee410b430 msgr2=0x7f0ee4197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee410b430 0x7f0ee4197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 -- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 msgr2=0x7f0ee4197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 0x7f0ee4197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 -- 172.21.2.117:0/1832467271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee407eea0 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee98b2700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 0x7f0ee4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.006 7f0ee90b1700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f0ed800b530 tx=0x7f0ed800b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed800bcc0 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed8002030 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ed8002510 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee407eea0 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee407f430 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.007 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0ed8000c90 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.008 7f0ed5ffb700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 0x7f0ec401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.008 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0ed8032670 con 0x7f0ee4198800 2021-02-10T11:55:51.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.008 7f0ee88b0700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 0x7f0ec401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.009 7f0ee88b0700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 0x7f0ec401abc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.038 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee40e77f0 con 0x7f0ee4198800 2021-02-10T11:55:51.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.041 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0ed80265a0 con 0x7f0ee4198800 2021-02-10T11:55:51.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.188 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0ee40e77f0 con 0x7f0ee4198800 2021-02-10T11:55:51.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.193 7f0ed5ffb700 1 -- 172.21.2.117:0/1832467271 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0ed80265a0 con 0x7f0ee4198800 2021-02-10T11:55:51.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.197 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 msgr2=0x7f0ec401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.197 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 0x7f0ec401abc0 crc :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.197 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 msgr2=0x7f0ee40ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.197 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7f0ed800b530 tx=0x7f0ed800b8d0).stop 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 shutdown_connections 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0ee4197970 0x7f0ee4197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0ee410b430 0x7f0ee4197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ec40186c0 0x7f0ec401abc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 --2- 172.21.2.117:0/1832467271 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0ee4198800 0x7f0ee40ae4a0 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 >> 172.21.2.117:0/1832467271 conn(0x7f0ee41998b0 msgr2=0x7f0ee41a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:51.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 shutdown_connections 2021-02-10T11:55:51.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.198 7f0eeb295700 1 -- 172.21.2.117:0/1832467271 wait complete. 2021-02-10T11:55:51.208 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize' 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.617 7f0ad2366700 1 Processor -- start 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 -- start start 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0d18e0 0x7f0acc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0acc0ef3f0 0x7f0acc1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0acc0ede60 con 0x7f0acc0eb000 2021-02-10T11:55:51.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0acc088020 con 0x7f0acc0ef3f0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.618 7f0ad2366700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0d1ce0 con 0x7f0acc0d18e0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0acc0ef3f0 0x7f0acc1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f0acc0ef3f0 0x7f0acc1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35294/0 (socket says 172.21.2.117:35294) 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 -- 172.21.2.117:0/3857555026 learned_addr learned my addr 172.21.2.117:0/3857555026 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0acbfff700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0d18e0 0x7f0acc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 -- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0d18e0 msgr2=0x7f0acc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0d18e0 0x7f0acc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 -- 172.21.2.117:0/3857555026 >> v1:172.21.2.125:6789/0 conn(0x7f0acc0eb000 legacy=0x7f0acc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 -- 172.21.2.117:0/3857555026 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc1b8f90 con 0x7f0acc0ef3f0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad0983700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0ef3f0 0x7f0acc1b7960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f0ac4006aa0 tx=0x7f0ac4009f20).ready entity=mon.1 client_cookie=afd5547d7de96654 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0acb7fe700 1 -- 172.21.2.117:0/3857555026 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac400b5c0 con 0x7f0acc0ef3f0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0acb7fe700 1 -- 172.21.2.117:0/3857555026 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4003030 con 0x7f0acc0ef3f0 2021-02-10T11:55:51.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0acb7fe700 1 -- 172.21.2.117:0/3857555026 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0ac400bb40 con 0x7f0acc0ef3f0 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad2366700 1 -- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0ef3f0 msgr2=0x7f0acc1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.619 7f0ad2366700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0ef3f0 0x7f0acc1b7960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f0ac4006aa0 tx=0x7f0ac4009f20).stop 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 -- 172.21.2.117:0/3857555026 shutdown_connections 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0ef3f0 0x7f0acc1b7960 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 --2- 172.21.2.117:0/3857555026 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0d18e0 0x7f0acc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 -- 172.21.2.117:0/3857555026 >> 172.21.2.117:0/3857555026 conn(0x7f0acc0ced60 msgr2=0x7f0acc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 -- 172.21.2.117:0/3857555026 shutdown_connections 2021-02-10T11:55:51.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.620 7f0ad2366700 1 -- 172.21.2.117:0/3857555026 wait complete. 2021-02-10T11:55:51.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 Processor -- start 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 -- start start 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 0x7f0acc1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0b82f0 0x7f0acc0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 0x7f0acc0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0c0670 con 0x7f0acc1a33d0 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0deb20 con 0x7f0acc0b8d10 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad2366700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0ded40 con 0x7f0acc0b82f0 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad0983700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 0x7f0acc0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0acbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 0x7f0acc1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad0983700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 0x7f0acc0b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35298/0 (socket says 172.21.2.117:35298) 2021-02-10T11:55:51.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ad0983700 1 -- 172.21.2.117:0/608550678 learned_addr learned my addr 172.21.2.117:0/608550678 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.635 7f0ac37fe700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0b82f0 0x7f0acc0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 -- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0b82f0 msgr2=0x7f0acc0b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0b82f0 0x7f0acc0b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 -- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 msgr2=0x7f0acc0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 0x7f0acc0b9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 -- 172.21.2.117:0/608550678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0def60 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0acbfff700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 0x7f0acc1a37d0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f0abc00b4e0 tx=0x7f0abc00b880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0abc000c90 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc006400 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0ad2366700 1 -- 172.21.2.117:0/608550678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc0def60 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0ad2366700 1 -- 172.21.2.117:0/608550678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc0df4f0 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.636 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0abc002030 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.637 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f0abc0052e0 con 0x7f0acc1a33d0 2021-02-10T11:55:51.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.637 7f0ac97fa700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 0x7f0ab001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:51.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.637 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f0abc032820 con 0x7f0acc1a33d0 2021-02-10T11:55:51.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.637 7f0ac37fe700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 0x7f0ab001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:51.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.638 7f0ac37fe700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 0x7f0ab001abd0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:51.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.668 7f0ad2366700 1 -- 172.21.2.117:0/608550678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc087290 con 0x7f0acc1a33d0 2021-02-10T11:55:51.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.671 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0abc0009b0 con 0x7f0acc1a33d0 2021-02-10T11:55:51.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:51.803 7f0ad2366700 1 -- 172.21.2.117:0/608550678 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]} v 0) v1 -- 0x7f0acc0ede60 con 0x7f0acc1a33d0 2021-02-10T11:55:52.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.691 7f0ac97fa700 1 -- 172.21.2.117:0/608550678 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_authorize"]}]=0 v834) v1 ==== 160+0+0 (secure 0 0 0) 0x7f0abc0063c0 con 0x7f0acc1a33d0 2021-02-10T11:55:52.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.725 7f0ad2366700 1 -- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 msgr2=0x7f0ab001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:52.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.725 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 0x7f0ab001abd0 crc :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:52.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.725 7f0ad2366700 1 -- 172.21.2.117:0/608550678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 msgr2=0x7f0acc1a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:52.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.725 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 0x7f0acc1a37d0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f0abc00b4e0 tx=0x7f0abc00b880).stop 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 -- 172.21.2.117:0/608550678 shutdown_connections 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f0acc0b8d10 0x7f0acc0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f0acc0b82f0 0x7f0acc0b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f0ab00186d0 0x7f0ab001abd0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 --2- 172.21.2.117:0/608550678 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f0acc1a33d0 0x7f0acc1a37d0 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 -- 172.21.2.117:0/608550678 >> 172.21.2.117:0/608550678 conn(0x7f0acc0874c0 msgr2=0x7f0acc096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:52.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 -- 172.21.2.117:0/608550678 shutdown_connections 2021-02-10T11:55:52.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:52.726 7f0ad2366700 1 -- 172.21.2.117:0/608550678 wait complete. 2021-02-10T11:55:52.735 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_authorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:55:52.736 INFO:tasks.cephfs_test_runner:Starting test: test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:55:52.736 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 Processor -- start 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 -- start start 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f302c0ef3f0 0x7f302c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f302c0eb7f0 0x7f302c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c0ede60 con 0x7f302c0d18e0 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c0ef7f0 con 0x7f302c0eb7f0 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3032e3a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c0d1ce0 con 0x7f302c0ef3f0 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3030c56700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f302c0d18e0 0x7f302c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59404/0 (socket says 172.21.2.117:59404) 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3030c56700 1 -- 172.21.2.117:0/641554714 learned_addr learned my addr 172.21.2.117:0/641554714 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:53.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.146 7f3031457700 1 --2- 172.21.2.117:0/641554714 >> v2:172.21.2.117:3300/0 conn(0x7f302c0eb7f0 0x7f302c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3031457700 1 -- 172.21.2.117:0/641554714 >> v2:172.21.2.125:3300/0 conn(0x7f302c0ef3f0 msgr2=0x7f302c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3031457700 1 --2- 172.21.2.117:0/641554714 >> v2:172.21.2.125:3300/0 conn(0x7f302c0ef3f0 0x7f302c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3031457700 1 -- 172.21.2.117:0/641554714 >> v1:172.21.2.117:6789/0 conn(0x7f302c0d18e0 legacy=0x7f302c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3031457700 1 -- 172.21.2.117:0/641554714 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c1b8f90 con 0x7f302c0eb7f0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3031457700 1 --2- 172.21.2.117:0/641554714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0eb7f0 0x7f302c1b7960 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f301c006e30 tx=0x7f301c009e60).ready entity=mon.0 client_cookie=95cda6bf09c8556a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f302b7fe700 1 -- 172.21.2.117:0/641554714 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f301c006a50 con 0x7f302c0eb7f0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f302b7fe700 1 -- 172.21.2.117:0/641554714 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00b300 con 0x7f302c0eb7f0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f302b7fe700 1 -- 172.21.2.117:0/641554714 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f301c00b820 con 0x7f302c0eb7f0 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3032e3a700 1 -- 172.21.2.117:0/641554714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0eb7f0 msgr2=0x7f302c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.147 7f3032e3a700 1 --2- 172.21.2.117:0/641554714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0eb7f0 0x7f302c1b7960 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7f301c006e30 tx=0x7f301c009e60).stop 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 -- 172.21.2.117:0/641554714 shutdown_connections 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 --2- 172.21.2.117:0/641554714 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0eb7f0 0x7f302c1b7960 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:53.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 --2- 172.21.2.117:0/641554714 >> v2:172.21.2.125:3300/0 conn(0x7f302c0ef3f0 0x7f302c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 -- 172.21.2.117:0/641554714 >> 172.21.2.117:0/641554714 conn(0x7f302c0ced60 msgr2=0x7f302c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:53.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 -- 172.21.2.117:0/641554714 shutdown_connections 2021-02-10T11:55:53.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.148 7f3032e3a700 1 -- 172.21.2.117:0/641554714 wait complete. 2021-02-10T11:55:53.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.162 7f3032e3a700 1 Processor -- start 2021-02-10T11:55:53.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.162 7f3032e3a700 1 -- start start 2021-02-10T11:55:53.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f302c19b760 0x7f302c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f302c19e620 0x7f302c1afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c1af950 con 0x7f302c0b5370 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c19bb60 con 0x7f302c19b760 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f3032e3a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c19ea20 con 0x7f302c19e620 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59160/0 (socket says 172.21.2.117:59160) 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 -- 172.21.2.117:0/189998920 learned_addr learned my addr 172.21.2.117:0/189998920 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 -- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f302c19e620 msgr2=0x7f302c1afe50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f302c19e620 0x7f302c1afe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 -- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f302c19b760 msgr2=0x7f302c0b4d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f302c19b760 0x7f302c0b4d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.163 7f302bfff700 1 -- 172.21.2.117:0/189998920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c19ec40 con 0x7f302c0b5370 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f302bfff700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f3024006f00 tx=0x7f302400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:53.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f302400bd30 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3024000c90 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f30240022a0 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f3032e3a700 1 -- 172.21.2.117:0/189998920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c19ec40 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f3032e3a700 1 -- 172.21.2.117:0/189998920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c1b0760 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3024005310 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.164 7f30297fa700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 0x7f301001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.165 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3024032700 con 0x7f302c0b5370 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.165 7f3030c56700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 0x7f301001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:53.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.165 7f3030c56700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 0x7f301001abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:53.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.196 7f3032e3a700 1 -- 172.21.2.117:0/189998920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c087210 con 0x7f302c0b5370 2021-02-10T11:55:53.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.198 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3024000940 con 0x7f302c0b5370 2021-02-10T11:55:53.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.330 7f3032e3a700 1 -- 172.21.2.117:0/189998920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7f302c0ede60 con 0x7f302c0b5370 2021-02-10T11:55:53.698 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.696 7f30297fa700 1 -- 172.21.2.117:0/189998920 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v835) v1 ==== 165+0+0 (secure 0 0 0) 0x7f3024005310 con 0x7f302c0b5370 2021-02-10T11:55:53.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.699 7f3032e3a700 1 -- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 msgr2=0x7f301001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:53.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.699 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 0x7f301001abc0 crc :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:53.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.699 7f3032e3a700 1 -- 172.21.2.117:0/189998920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 msgr2=0x7f302c19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:53.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.699 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f3024006f00 tx=0x7f302400b940).stop 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 -- 172.21.2.117:0/189998920 shutdown_connections 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f30100186c0 0x7f301001abc0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f302c0b5370 0x7f302c19e000 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f302c19e620 0x7f302c1afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 --2- 172.21.2.117:0/189998920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f302c19b760 0x7f302c0b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 -- 172.21.2.117:0/189998920 >> 172.21.2.117:0/189998920 conn(0x7f302c19f610 msgr2=0x7f302c1ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 -- 172.21.2.117:0/189998920 shutdown_connections 2021-02-10T11:55:53.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:53.700 7f3032e3a700 1 -- 172.21.2.117:0/189998920 wait complete. 2021-02-10T11:55:53.711 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:55:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.125 7f4bc8f6e700 1 Processor -- start 2021-02-10T11:55:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.125 7f4bc8f6e700 1 -- start start 2021-02-10T11:55:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40d18f0 0x7f4bc40eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bc40eb040 0x7f4bc40eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4bc40ee3e0 0x7f4bc40ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc40ede60 con 0x7f4bc40eb040 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc40eba60 con 0x7f4bc40ee3e0 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc8f6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40d1cf0 con 0x7f4bc40d18f0 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc2312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bc40eb040 0x7f4bc40eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc2312700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4bc40eb040 0x7f4bc40eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59168/0 (socket says 172.21.2.117:59168) 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.126 7f4bc2312700 1 -- 172.21.2.117:0/3313417926 learned_addr learned my addr 172.21.2.117:0/3313417926 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 -- 172.21.2.117:0/3313417926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40d18f0 msgr2=0x7f4bc40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 --2- 172.21.2.117:0/3313417926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40d18f0 0x7f4bc40eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 -- 172.21.2.117:0/3313417926 >> v2:172.21.2.125:3300/0 conn(0x7f4bc40ee3e0 msgr2=0x7f4bc40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:54.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 --2- 172.21.2.117:0/3313417926 >> v2:172.21.2.125:3300/0 conn(0x7f4bc40ee3e0 0x7f4bc40ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 -- 172.21.2.117:0/3313417926 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc40eee00 con 0x7f4bc40eb040 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc2312700 1 --2- 172.21.2.117:0/3313417926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc40eb040 0x7f4bc40eb440 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f4bb8006700 tx=0x7f4bb8009b80).ready entity=mon.0 client_cookie=98ca012008fa1b84 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc1b11700 1 -- 172.21.2.117:0/3313417926 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb800c0c0 con 0x7f4bc40eb040 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc1b11700 1 -- 172.21.2.117:0/3313417926 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb800c3c0 con 0x7f4bc40eb040 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc1b11700 1 -- 172.21.2.117:0/3313417926 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb800c8e0 con 0x7f4bc40eb040 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc8f6e700 1 -- 172.21.2.117:0/3313417926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc40eb040 msgr2=0x7f4bc40eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.127 7f4bc8f6e700 1 --2- 172.21.2.117:0/3313417926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc40eb040 0x7f4bc40eb440 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f4bb8006700 tx=0x7f4bb8009b80).stop 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 -- 172.21.2.117:0/3313417926 shutdown_connections 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 --2- 172.21.2.117:0/3313417926 >> v2:172.21.2.125:3300/0 conn(0x7f4bc40ee3e0 0x7f4bc40ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 --2- 172.21.2.117:0/3313417926 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc40eb040 0x7f4bc40eb440 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 --2- 172.21.2.117:0/3313417926 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40d18f0 0x7f4bc40eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.146 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 -- 172.21.2.117:0/3313417926 >> 172.21.2.117:0/3313417926 conn(0x7f4bc40cf060 msgr2=0x7f4bc40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 -- 172.21.2.117:0/3313417926 shutdown_connections 2021-02-10T11:55:54.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.128 7f4bc8f6e700 1 -- 172.21.2.117:0/3313417926 wait complete. 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 Processor -- start 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 -- start start 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bc409e160 0x7f4bc41a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40a65a0 0x7f4bc40a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40a1820 con 0x7f4bc41a4860 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40a69a0 con 0x7f4bc409e160 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.143 7f4bc8f6e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc409dc80 con 0x7f4bc40a65a0 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59172/0 (socket says 172.21.2.117:59172) 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 -- 172.21.2.117:0/2690349286 learned_addr learned my addr 172.21.2.117:0/2690349286 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 -- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40a65a0 msgr2=0x7f4bc40a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40a65a0 0x7f4bc40a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 -- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bc409e160 msgr2=0x7f4bc41a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bc409e160 0x7f4bc41a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 -- 172.21.2.117:0/2690349286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc40a23c0 con 0x7f4bc41a4860 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc2312700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f4bb8009720 tx=0x7f4bb800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb80090c0 con 0x7f4bc41a4860 2021-02-10T11:55:54.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8002030 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc40a23c0 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bb8002510 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.144 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc40a2950 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.145 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bb8000c90 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.145 7f4bb37fe700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 0x7f4ba801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.145 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4bb80327d0 con 0x7f4bc41a4860 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.146 7f4bc2b13700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 0x7f4ba801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:54.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.146 7f4bc2b13700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 0x7f4ba801abd0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:54.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.176 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc40ede60 con 0x7f4bc41a4860 2021-02-10T11:55:54.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.179 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bb80063a0 con 0x7f4bc41a4860 2021-02-10T11:55:54.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.329 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4bc40ede60 con 0x7f4bc41a4860 2021-02-10T11:55:54.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.333 7f4bb37fe700 1 -- 172.21.2.117:0/2690349286 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4bb8000c90 con 0x7f4bc41a4860 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 msgr2=0x7f4ba801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 0x7f4ba801abd0 crc :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 msgr2=0x7f4bc40a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f4bb8009720 tx=0x7f4bb800b960).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 shutdown_connections 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ba80186d0 0x7f4ba801abd0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4bc41a4860 0x7f4bc40a5f80 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4bc40a65a0 0x7f4bc40a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 --2- 172.21.2.117:0/2690349286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4bc409e160 0x7f4bc41a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 >> 172.21.2.117:0/2690349286 conn(0x7f4bc409e660 msgr2=0x7f4bc40bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:54.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 shutdown_connections 2021-02-10T11:55:54.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:54.338 7f4bc8f6e700 1 -- 172.21.2.117:0/2690349286 wait complete. 2021-02-10T11:55:54.348 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:55:54.348 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:54.695 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:55:55.042 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:55:55.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:55.042 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:55.042 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:55:55.379 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:55:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:55.380 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:55:55.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:55.380 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:55.380 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:55:55.715 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:55:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:55.715 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:55:55.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:55.716 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:55.716 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:55:56.049 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:55:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:56.049 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:55:56.050 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:55:56.050 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:55:56.050 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:55:56.384 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:55:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:55:56.384 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:55:56.385 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 Processor -- start 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 -- start start 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e80ef3f0 0x7fa7e80eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa7e80eb7f0 0x7fa7e81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e80ede60 con 0x7fa7e80d18e0 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e80ef7f0 con 0x7fa7e80eb7f0 2021-02-10T11:55:56.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.795 7fa7ede5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80d1ce0 con 0x7fa7e80ef3f0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa7e80d18e0 0x7fa7e80ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59428/0 (socket says 172.21.2.117:59428) 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e77fe700 1 -- 172.21.2.117:0/3255829675 learned_addr learned my addr 172.21.2.117:0/3255829675 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e7fff700 1 --2- 172.21.2.117:0/3255829675 >> v2:172.21.2.125:3300/0 conn(0x7fa7e80eb7f0 0x7fa7e81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e6ffd700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e80ef3f0 0x7fa7e80eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e67fc700 1 -- 172.21.2.117:0/3255829675 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158690942 0 0) 0x7fa7d0001180 con 0x7fa7e80d18e0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.796 7fa7e67fc700 1 -- 172.21.2.117:0/3255829675 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d4002fe0 con 0x7fa7e80d18e0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e7fff700 1 -- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e80ef3f0 msgr2=0x7fa7e80eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e7fff700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e80ef3f0 0x7fa7e80eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e7fff700 1 -- 172.21.2.117:0/3255829675 >> v1:172.21.2.117:6789/0 conn(0x7fa7e80d18e0 legacy=0x7fa7e80ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e7fff700 1 -- 172.21.2.117:0/3255829675 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e81b8f90 con 0x7fa7e80eb7f0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e7fff700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80eb7f0 0x7fa7e81b7960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fa7e0006e50 tx=0x7fa7e0009e70).ready entity=mon.1 client_cookie=54a4952eafeb8a02 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:56.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e67fc700 1 -- 172.21.2.117:0/3255829675 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7e000b510 con 0x7fa7e80eb7f0 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e67fc700 1 -- 172.21.2.117:0/3255829675 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e0006790 con 0x7fa7e80eb7f0 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7e67fc700 1 -- 172.21.2.117:0/3255829675 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7e000b940 con 0x7fa7e80eb7f0 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7ede5b700 1 -- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80eb7f0 msgr2=0x7fa7e81b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.797 7fa7ede5b700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80eb7f0 0x7fa7e81b7960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7fa7e0006e50 tx=0x7fa7e0009e70).stop 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 -- 172.21.2.117:0/3255829675 shutdown_connections 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80eb7f0 0x7fa7e81b7960 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 --2- 172.21.2.117:0/3255829675 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e80ef3f0 0x7fa7e80eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 -- 172.21.2.117:0/3255829675 >> 172.21.2.117:0/3255829675 conn(0x7fa7e80ced60 msgr2=0x7fa7e80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 -- 172.21.2.117:0/3255829675 shutdown_connections 2021-02-10T11:55:56.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.798 7fa7ede5b700 1 -- 172.21.2.117:0/3255829675 wait complete. 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 Processor -- start 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 -- start start 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e81a4290 0x7fa7e81a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80ab280 0x7fa7e80ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80a7d10 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e80a7a00 con 0x7fa7e80ab280 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7ede5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e8096370 con 0x7fa7e81a4290 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7e7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7e7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59186/0 (socket says 172.21.2.117:59186) 2021-02-10T11:55:56.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.813 7fa7e7fff700 1 -- 172.21.2.117:0/25878927 learned_addr learned my addr 172.21.2.117:0/25878927 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 -- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e81a4290 msgr2=0x7fa7e81a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e81a4290 0x7fa7e81a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 -- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80ab280 msgr2=0x7fa7e80ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80ab280 0x7fa7e80ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 -- 172.21.2.117:0/25878927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80a8290 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7e7fff700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fa7e0006c50 tx=0x7fa7e000bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7e00093d0 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e000be60 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7e000c570 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e80a8290 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.814 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e80a8820 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.815 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7e0002ef0 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.815 7fa7dffff700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 0x7fa7cc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.815 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa7e0032a70 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.816 7fa7e77fe700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 0x7fa7cc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:56.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.816 7fa7e77fe700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 0x7fa7cc01abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:56.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.845 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8087290 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.848 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa7e0025ef0 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.972 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa7e80ede60 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.973 7fa7dffff700 1 -- 172.21.2.117:0/25878927 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v170) v1 ==== 81+0+30766 (secure 0 0 0) 0x7fa7e0025ef0 con 0x7fa7e80a6fe0 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 170, 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:55:15.701452", 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:55:56.975 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:55:56.976 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 37, 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:55:56.977 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:56.978 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:55:56.979 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:56.980 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:56.981 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 36, 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:55:56.982 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:55:12.669422", 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:56.983 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "169", 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:56.984 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:56.985 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:56.986 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:55:56.987 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 37, 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:55:13.692093", 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:55:56.988 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:55:56.989 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "170", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:55:56.990 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:55:56.991 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:55:56.992 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.993 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:56.994 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:55:56.995 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.996 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.997 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.998 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:56.999 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.000 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:55:57.001 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.002 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.003 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.004 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.005 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.006 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.007 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:55:57.008 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:55:57.009 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.010 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.011 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.012 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.013 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.014 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.015 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:55:57.016 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.017 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.018 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.019 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.020 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.021 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.022 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.023 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.024 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.025 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:55:57.026 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.027 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.028 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.029 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.030 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 168, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.031 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.032 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.033 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:55:57.034 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:55:57.036 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:55:57.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 167, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.038 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:55:57.042 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.043 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.044 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:55:57.045 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:55:57.047 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:55:57.048 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:55:57.049 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:55:57.050 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:55:57.051 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 msgr2=0x7fa7cc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 0x7fa7cc01abc0 crc :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 msgr2=0x7fa7e80a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7fa7e0006c50 tx=0x7fa7e000bd20).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 shutdown_connections 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7cc0186c0 0x7fa7cc01abc0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7e80a6fe0 0x7fa7e80a73e0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7e80ab280 0x7fa7e80ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 --2- 172.21.2.117:0/25878927 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7e81a4290 0x7fa7e81a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.974 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 >> 172.21.2.117:0/25878927 conn(0x7fa7e80874c0 msgr2=0x7fa7e8096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.975 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 shutdown_connections 2021-02-10T11:55:57.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:56.975 7fa7ede5b700 1 -- 172.21.2.117:0/25878927 wait complete. 2021-02-10T11:55:57.053 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:57.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.463 7fe8f89e2700 1 Processor -- start 2021-02-10T11:55:57.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.463 7fe8f89e2700 1 -- start start 2021-02-10T11:55:57.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f89e2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8f40e4e40 0x7fe8f41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f89e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40e5240 0x7fe8f4084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f89e2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f4085930 con 0x7fe8f41b0810 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f89e2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f4084840 con 0x7fe8f40e4e40 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f89e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4084a30 con 0x7fe8f40e5240 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe8f40e4e40 0x7fe8f41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f2e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe8f41b0810 0x7fe8f4085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59440/0 (socket says 172.21.2.117:59440) 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f2e1d700 1 -- 172.21.2.117:0/1464800036 learned_addr learned my addr 172.21.2.117:0/1464800036 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 -- 172.21.2.117:0/1464800036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40e5240 msgr2=0x7fe8f4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 --2- 172.21.2.117:0/1464800036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40e5240 0x7fe8f4084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 -- 172.21.2.117:0/1464800036 >> v1:172.21.2.117:6789/0 conn(0x7fe8f41b0810 legacy=0x7fe8f4085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 -- 172.21.2.117:0/1464800036 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4085280 con 0x7fe8f40e4e40 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.464 7fe8f261c700 1 --2- 172.21.2.117:0/1464800036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f40e4e40 0x7fe8f41b01f0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7fe8dc006650 tx=0x7fe8dc009a80).ready entity=mon.0 client_cookie=a6ea445a20b4886b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f161a700 1 -- 172.21.2.117:0/1464800036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc0009b0 con 0x7fe8f40e4e40 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f161a700 1 -- 172.21.2.117:0/1464800036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc00c0c0 con 0x7fe8f40e4e40 2021-02-10T11:55:57.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f161a700 1 -- 172.21.2.117:0/1464800036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc00c5e0 con 0x7fe8f40e4e40 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 -- 172.21.2.117:0/1464800036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f40e4e40 msgr2=0x7fe8f41b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 --2- 172.21.2.117:0/1464800036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f40e4e40 0x7fe8f41b01f0 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7fe8dc006650 tx=0x7fe8dc009a80).stop 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 -- 172.21.2.117:0/1464800036 shutdown_connections 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 --2- 172.21.2.117:0/1464800036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40e5240 0x7fe8f4084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 --2- 172.21.2.117:0/1464800036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f40e4e40 0x7fe8f41b01f0 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.465 7fe8f89e2700 1 -- 172.21.2.117:0/1464800036 >> 172.21.2.117:0/1464800036 conn(0x7fe8f40ce4e0 msgr2=0x7fe8f40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.466 7fe8f89e2700 1 -- 172.21.2.117:0/1464800036 shutdown_connections 2021-02-10T11:55:57.484 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.466 7fe8f89e2700 1 -- 172.21.2.117:0/1464800036 wait complete. 2021-02-10T11:55:57.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 Processor -- start 2021-02-10T11:55:57.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 -- start start 2021-02-10T11:55:57.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8f410caa0 0x7fe8f40b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40b7c00 0x7fe8f410d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f410cf00 con 0x7fe8f41a5640 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f41a5a40 con 0x7fe8f410caa0 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f89e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f41a5c20 con 0x7fe8f40b7c00 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.481 7fe8f261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59194/0 (socket says 172.21.2.117:59194) 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 -- 172.21.2.117:0/4002563812 learned_addr learned my addr 172.21.2.117:0/4002563812 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f1e1b700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8f410caa0 0x7fe8f40b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f2e1d700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40b7c00 0x7fe8f410d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 -- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40b7c00 msgr2=0x7fe8f410d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40b7c00 0x7fe8f410d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 -- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8f410caa0 msgr2=0x7fe8f40b75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8f410caa0 0x7fe8f40b75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 -- 172.21.2.117:0/4002563812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4110fe0 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f261c700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7fe8dc009650 tx=0x7fe8dc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc00bd50 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc006030 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f4110fe0 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe8dc005560 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.482 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f40b82f0 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.483 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe8dc0054f0 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.483 7fe8eaffd700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 0x7fe8d801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.483 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe8dc032760 con 0x7fe8f41a5640 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.483 7fe8f1e1b700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 0x7fe8d801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:57.488 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.484 7fe8f1e1b700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 0x7fe8d801ac10 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:57.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.513 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f410c250 con 0x7fe8f41a5640 2021-02-10T11:55:57.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.516 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe8dc0052e0 con 0x7fe8f41a5640 2021-02-10T11:55:57.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.644 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f4085930 con 0x7fe8f41a5640 2021-02-10T11:55:57.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.645 7fe8eaffd700 1 -- 172.21.2.117:0/4002563812 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+2105 (secure 0 0 0) 0x7fe8dc00bd50 con 0x7fe8f41a5640 2021-02-10T11:55:57.647 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:57.647 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":145,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:16.708700","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7844},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7844":{"gid":7844,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/321593523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":321593523},{"type":"v1","addr":"172.21.2.125:6835","nonce":321593523}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:57.648 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 145 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 msgr2=0x7fe8d801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 0x7fe8d801ac10 crc :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 msgr2=0x7fe8f410c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7fe8dc009650 tx=0x7fe8dc00b960).stop 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 shutdown_connections 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe8f40b7c00 0x7fe8f410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe8f410caa0 0x7fe8f40b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe8d8018730 0x7fe8d801ac10 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 --2- 172.21.2.117:0/4002563812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe8f41a5640 0x7fe8f410c480 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:57.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 >> 172.21.2.117:0/4002563812 conn(0x7fe8f41adc00 msgr2=0x7fe8f41a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:57.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 shutdown_connections 2021-02-10T11:55:57.650 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:57.647 7fe8f89e2700 1 -- 172.21.2.117:0/4002563812 wait complete. 2021-02-10T11:55:57.658 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 Processor -- start 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 -- start start 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35181b07f0 0x7f35180e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f35180e8d80 0x7f35180e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180e77f0 con 0x7f35180e6170 2021-02-10T11:55:58.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180e5ce0 con 0x7f35181b07f0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.070 7f351f256700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180e54b0 con 0x7f35180e8d80 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35181b07f0 0x7f35180e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f35181b07f0 0x7f35180e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59202/0 (socket says 172.21.2.117:59202) 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 -- 172.21.2.117:0/3307461148 learned_addr learned my addr 172.21.2.117:0/3307461148 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 -- 172.21.2.117:0/3307461148 >> v2:172.21.2.125:3300/0 conn(0x7f35180e8d80 msgr2=0x7f35180e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 --2- 172.21.2.117:0/3307461148 >> v2:172.21.2.125:3300/0 conn(0x7f35180e8d80 0x7f35180e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 -- 172.21.2.117:0/3307461148 >> v1:172.21.2.125:6789/0 conn(0x7f35180e6170 legacy=0x7f35180e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 -- 172.21.2.117:0/3307461148 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180e9180 con 0x7f35181b07f0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351d072700 1 --2- 172.21.2.117:0/3307461148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35181b07f0 0x7f35180e4e90 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f350c006700 tx=0x7f350c009b80).ready entity=mon.0 client_cookie=c7564f3ac981751 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f350bfff700 1 -- 172.21.2.117:0/3307461148 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350c00c0c0 con 0x7f35181b07f0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f350bfff700 1 -- 172.21.2.117:0/3307461148 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00c340 con 0x7f35181b07f0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f350bfff700 1 -- 172.21.2.117:0/3307461148 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f350c00c860 con 0x7f35181b07f0 2021-02-10T11:55:58.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351f256700 1 -- 172.21.2.117:0/3307461148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35181b07f0 msgr2=0x7f35180e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.071 7f351f256700 1 --2- 172.21.2.117:0/3307461148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35181b07f0 0x7f35180e4e90 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f350c006700 tx=0x7f350c009b80).stop 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 -- 172.21.2.117:0/3307461148 shutdown_connections 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 --2- 172.21.2.117:0/3307461148 >> v2:172.21.2.125:3300/0 conn(0x7f35180e8d80 0x7f35180e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 --2- 172.21.2.117:0/3307461148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35181b07f0 0x7f35180e4e90 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 -- 172.21.2.117:0/3307461148 >> 172.21.2.117:0/3307461148 conn(0x7f35180ce500 msgr2=0x7f35180e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 -- 172.21.2.117:0/3307461148 shutdown_connections 2021-02-10T11:55:58.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.072 7f351f256700 1 -- 172.21.2.117:0/3307461148 wait complete. 2021-02-10T11:55:58.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 Processor -- start 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 -- start start 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f351809a180 0x7f35180a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3518093980 0x7f3518093d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518093480 con 0x7f35180943a0 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518091e90 con 0x7f351809a180 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351f256700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35180920b0 con 0x7f3518093980 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351d873700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351d873700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59212/0 (socket says 172.21.2.117:59212) 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.087 7f351d873700 1 -- 172.21.2.117:0/676093598 learned_addr learned my addr 172.21.2.117:0/676093598 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 -- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3518093980 msgr2=0x7f3518093d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:58.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3518093980 0x7f3518093d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 -- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f351809a180 msgr2=0x7f35180a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f351809a180 0x7f35180a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 -- 172.21.2.117:0/676093598 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180922d0 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351d873700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f3510009980 tx=0x7f351000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3510005870 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3510005af0 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351f256700 1 -- 172.21.2.117:0/676093598 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35180922d0 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f351f256700 1 -- 172.21.2.117:0/676093598 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3518092860 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.088 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3510006030 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.089 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3510005cd0 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.089 7f3509ffb700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 0x7f34f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.089 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3510032a40 con 0x7f35180943a0 2021-02-10T11:55:58.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.089 7f351d072700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 0x7f34f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:58.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.090 7f351d072700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 0x7f34f801abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:58.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.121 7f351f256700 1 -- 172.21.2.117:0/676093598 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180e77f0 con 0x7f35180943a0 2021-02-10T11:55:58.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.123 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f351001da80 con 0x7f35180943a0 2021-02-10T11:55:58.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:58.251 7f351f256700 1 -- 172.21.2.117:0/676093598 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3518093400 con 0x7f35180943a0 2021-02-10T11:55:59.032 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:55:59.030 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:55:59.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.033 7f3509ffb700 1 -- 172.21.2.117:0/676093598 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) v1 ==== 161+0+0 (secure 0 0 0) 0x7f351001da80 con 0x7f35180943a0 2021-02-10T11:55:59.072 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:55:59.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 -- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 msgr2=0x7f34f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 0x7f34f801abc0 crc :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 -- 172.21.2.117:0/676093598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 msgr2=0x7f35180947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f3510009980 tx=0x7f351000bcd0).stop 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 -- 172.21.2.117:0/676093598 shutdown_connections 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f34f80186c0 0x7f34f801abc0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35180943a0 0x7f35180947a0 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3518093980 0x7f3518093d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 --2- 172.21.2.117:0/676093598 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f351809a180 0x7f35180a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.072 7f351f256700 1 -- 172.21.2.117:0/676093598 >> 172.21.2.117:0/676093598 conn(0x7f351810c840 msgr2=0x7f35180c3720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.073 7f351f256700 1 -- 172.21.2.117:0/676093598 shutdown_connections 2021-02-10T11:55:59.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.073 7f351f256700 1 -- 172.21.2.117:0/676093598 wait complete. 2021-02-10T11:55:59.083 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:55:59.086 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 Processor -- start 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 -- start start 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcaac0e5240 0x7fcaac086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac084c60 0x7fcaac085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaac085930 con 0x7fcaac0e4e40 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaac085680 con 0x7fcaac0e5240 2021-02-10T11:55:59.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcab0c52700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac1b09e0 con 0x7fcaac084c60 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcaaa05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcaac0e5240 0x7fcaac086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcaaa05d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcaac0e5240 0x7fcaac086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59216/0 (socket says 172.21.2.117:59216) 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcaaa85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcaac0e4e40 0x7fcaac1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59466/0 (socket says 172.21.2.117:59466) 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.494 7fcaaa05d700 1 -- 172.21.2.117:0/3470528129 learned_addr learned my addr 172.21.2.117:0/3470528129 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaaa05d700 1 -- 172.21.2.117:0/3470528129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac084c60 msgr2=0x7fcaac085060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaaa05d700 1 --2- 172.21.2.117:0/3470528129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac084c60 0x7fcaac085060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaaa05d700 1 -- 172.21.2.117:0/3470528129 >> v1:172.21.2.117:6789/0 conn(0x7fcaac0e4e40 legacy=0x7fcaac1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaaa05d700 1 -- 172.21.2.117:0/3470528129 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac084840 con 0x7fcaac0e5240 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaaa05d700 1 --2- 172.21.2.117:0/3470528129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac0e5240 0x7fcaac086ec0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7fca9c006700 tx=0x7fca9c009b80).ready entity=mon.0 client_cookie=41b882d86a4ba472 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaa985c700 1 -- 172.21.2.117:0/3470528129 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9c00c0c0 con 0x7fcaac0e5240 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaa985c700 1 -- 172.21.2.117:0/3470528129 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c00c3c0 con 0x7fcaac0e5240 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcaa985c700 1 -- 172.21.2.117:0/3470528129 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca9c00c8e0 con 0x7fcaac0e5240 2021-02-10T11:55:59.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcab0c52700 1 -- 172.21.2.117:0/3470528129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac0e5240 msgr2=0x7fcaac086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.495 7fcab0c52700 1 --2- 172.21.2.117:0/3470528129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac0e5240 0x7fcaac086ec0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7fca9c006700 tx=0x7fca9c009b80).stop 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 -- 172.21.2.117:0/3470528129 shutdown_connections 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 --2- 172.21.2.117:0/3470528129 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac084c60 0x7fcaac085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 --2- 172.21.2.117:0/3470528129 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac0e5240 0x7fcaac086ec0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 -- 172.21.2.117:0/3470528129 >> 172.21.2.117:0/3470528129 conn(0x7fcaac0ce4e0 msgr2=0x7fcaac0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 -- 172.21.2.117:0/3470528129 shutdown_connections 2021-02-10T11:55:59.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.496 7fcab0c52700 1 -- 172.21.2.117:0/3470528129 wait complete. 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.510 7fcab0c52700 1 Processor -- start 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.510 7fcab0c52700 1 -- start start 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaac095330 0x7fcaac190810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac095730 0x7fcaac190e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac190290 con 0x7fcaac19e470 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac19e870 con 0x7fcaac095330 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcab0c52700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac18d230 con 0x7fcaac095730 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59226/0 (socket says 172.21.2.117:59226) 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 -- 172.21.2.117:0/3936670114 learned_addr learned my addr 172.21.2.117:0/3936670114 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 -- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac095730 msgr2=0x7fcaac190e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac095730 0x7fcaac190e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 -- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaac095330 msgr2=0x7fcaac190810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaac095330 0x7fcaac190810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 -- 172.21.2.117:0/3936670114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaac18fef0 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.511 7fcaaa85e700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7fca940014d0 tx=0x7fca9400a8a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca94000ef0 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9400ac90 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fca94004590 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaac18fef0 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac18d5d0 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.512 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fca940043a0 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.513 7fcaa37fe700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 0x7fca9001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.513 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fca94032740 con 0x7fcaac19e470 2021-02-10T11:55:59.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.513 7fcaaa05d700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 0x7fca9001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:55:59.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.513 7fcaaa05d700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 0x7fca9001abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:55:59.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.544 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac10c250 con 0x7fcaac19e470 2021-02-10T11:55:59.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.547 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fca94000ef0 con 0x7fcaac19e470 2021-02-10T11:55:59.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.674 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaac085930 con 0x7fcaac19e470 2021-02-10T11:55:59.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.675 7fcaa37fe700 1 -- 172.21.2.117:0/3936670114 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+1763 (secure 0 0 0) 0x7fca940043a0 con 0x7fcaac19e470 2021-02-10T11:55:59.677 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:55:59.677 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":146,"default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/1574265368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":1574265368},{"type":"v1","addr":"172.21.2.117:6833","nonce":1574265368}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":140}],"filesystems":[{"mdsmap":{"epoch":146,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:55:14.699333","modified":"2021-02-10 11:55:59.030674","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":171,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":18}]} 2021-02-10T11:55:59.677 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 146 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 msgr2=0x7fca9001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 0x7fca9001abc0 crc :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 msgr2=0x7fcaac18cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7fca940014d0 tx=0x7fca9400a8a0).stop 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 shutdown_connections 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcaac095730 0x7fcaac190e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcaac095330 0x7fcaac190810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fca900186c0 0x7fca9001abc0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 --2- 172.21.2.117:0/3936670114 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcaac19e470 0x7fcaac18cc10 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:55:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 >> 172.21.2.117:0/3936670114 conn(0x7fcaac10de60 msgr2=0x7fcaac1af660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:55:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 shutdown_connections 2021-02-10T11:55:59.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:55:59.677 7fcab0c52700 1 -- 172.21.2.117:0/3936670114 wait complete. 2021-02-10T11:55:59.688 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:56:00.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.104 7fc77f811700 1 Processor -- start 2021-02-10T11:56:00.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.104 7fc77f811700 1 -- start start 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77f811700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc7780e7180 0x7fc7780e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77f811700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780e7580 con 0x7fc7780e8390 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77f811700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780e77f0 con 0x7fc7781b07f0 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77f811700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780e5ce0 con 0x7fc7780e7180 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77d62d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fc7781b07f0 0x7fc7780e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52406/0 (socket says 172.21.2.117:52406) 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77d62d700 1 -- 172.21.2.117:0/3781239720 learned_addr learned my addr 172.21.2.117:0/3781239720 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77ce2c700 1 --2- 172.21.2.117:0/3781239720 >> v2:172.21.2.125:3300/0 conn(0x7fc7780e7180 0x7fc7780e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77ce2c700 1 -- 172.21.2.117:0/3781239720 >> v1:172.21.2.125:6789/0 conn(0x7fc7781b07f0 legacy=0x7fc7780e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77ce2c700 1 -- 172.21.2.117:0/3781239720 >> v1:172.21.2.117:6789/0 conn(0x7fc7780e8390 legacy=0x7fc7780e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc77ce2c700 1 -- 172.21.2.117:0/3781239720 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7780e8f80 con 0x7fc7780e7180 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.105 7fc76bfff700 1 -- 172.21.2.117:0/3781239720 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798739490 0 0) 0x7fc770000f90 con 0x7fc7780e8390 2021-02-10T11:56:00.123 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77ce2c700 1 --2- 172.21.2.117:0/3781239720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7780e7180 0x7fc7780e7d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fc774006650 tx=0x7fc774009a00).ready entity=mon.1 client_cookie=ec58377a75e77d7a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc76bfff700 1 -- 172.21.2.117:0/3781239720 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7740009b0 con 0x7fc7780e7180 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc76bfff700 1 -- 172.21.2.117:0/3781239720 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77400c0c0 con 0x7fc7780e7180 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc76bfff700 1 -- 172.21.2.117:0/3781239720 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc77400c5a0 con 0x7fc7780e7180 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77f811700 1 -- 172.21.2.117:0/3781239720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7780e7180 msgr2=0x7fc7780e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77f811700 1 --2- 172.21.2.117:0/3781239720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7780e7180 0x7fc7780e7d70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fc774006650 tx=0x7fc774009a00).stop 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77f811700 1 -- 172.21.2.117:0/3781239720 shutdown_connections 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77f811700 1 --2- 172.21.2.117:0/3781239720 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7780e7180 0x7fc7780e7d70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.106 7fc77f811700 1 -- 172.21.2.117:0/3781239720 >> 172.21.2.117:0/3781239720 conn(0x7fc7780ce500 msgr2=0x7fc7780e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.107 7fc77f811700 1 -- 172.21.2.117:0/3781239720 shutdown_connections 2021-02-10T11:56:00.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.107 7fc77f811700 1 -- 172.21.2.117:0/3781239720 wait complete. 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.121 7fc77f811700 1 Processor -- start 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.121 7fc77f811700 1 -- start start 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 0x7fc778189f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 0x7fc7781974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77809a400 con 0x7fc77809a000 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77809ab40 con 0x7fc7781970e0 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77f811700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77818ad80 con 0x7fc778099c00 2021-02-10T11:56:00.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77ce2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77ce2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59236/0 (socket says 172.21.2.117:59236) 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77ce2c700 1 -- 172.21.2.117:0/3946155974 learned_addr learned my addr 172.21.2.117:0/3946155974 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.122 7fc77de2e700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 0x7fc7781974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 -- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 msgr2=0x7fc778189f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77d62d700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 0x7fc778189f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 0x7fc778189f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 -- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 msgr2=0x7fc7781974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 0x7fc7781974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 -- 172.21.2.117:0/3946155974 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77809a630 con 0x7fc77809a000 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77de2e700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 0x7fc7781974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77d62d700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 0x7fc778189f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77ce2c700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7fc77400b640 tx=0x7fc77400b9e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:00.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc77400bdd0 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774000c90 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc7740022b0 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77f811700 1 -- 172.21.2.117:0/3946155974 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77809a630 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.123 7fc77f811700 1 -- 172.21.2.117:0/3946155974 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc778197df0 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.124 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc774006a10 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.124 7fc769ffb700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 0x7fc75801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.124 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fc7740327a0 con 0x7fc77809a000 2021-02-10T11:56:00.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.124 7fc77d62d700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 0x7fc75801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:00.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.124 7fc77d62d700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 0x7fc75801abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:00.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.155 7fc77f811700 1 -- 172.21.2.117:0/3946155974 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc778108a60 con 0x7fc77809a000 2021-02-10T11:56:00.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.157 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc774030da0 con 0x7fc77809a000 2021-02-10T11:56:00.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:00.283 7fc77f811700 1 -- 172.21.2.117:0/3946155974 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc7780e77f0 con 0x7fc77809a000 2021-02-10T11:56:01.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.047 7fc769ffb700 1 -- 172.21.2.117:0/3946155974 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v147) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc774006a10 con 0x7fc77809a000 2021-02-10T11:56:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 msgr2=0x7fc75801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 0x7fc75801abc0 crc :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 msgr2=0x7fc77818a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7fc77400b640 tx=0x7fc77400b9e0).stop 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 shutdown_connections 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc7580186c0 0x7fc75801abc0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc77809a000 0x7fc77818a580 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc7781970e0 0x7fc7781974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 --2- 172.21.2.117:0/3946155974 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc778099c00 0x7fc778189f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 >> 172.21.2.117:0/3946155974 conn(0x7fc7781110f0 msgr2=0x7fc7781095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:01.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 shutdown_connections 2021-02-10T11:56:01.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.056 7fc77f811700 1 -- 172.21.2.117:0/3946155974 wait complete. 2021-02-10T11:56:01.066 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.480 7fd219152700 1 Processor -- start 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.480 7fd219152700 1 -- start start 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd219152700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 0x7fd2141b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd219152700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd214085450 0x7fd2140d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd219152700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd214085930 con 0x7fd2140e5240 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd219152700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2140e5cc0 con 0x7fd214085450 2021-02-10T11:56:01.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd219152700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214084840 con 0x7fd2140e4e40 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd21259c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd2140e5240 0x7fd214084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59496/0 (socket says 172.21.2.117:59496) 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd21259c700 1 -- 172.21.2.117:0/3063022916 learned_addr learned my addr 172.21.2.117:0/3063022916 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd21359e700 1 --2- 172.21.2.117:0/3063022916 >> v2:172.21.2.125:3300/0 conn(0x7fd214085450 0x7fd2140d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.481 7fd212d9d700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 0x7fd2141b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd211d9b700 1 -- 172.21.2.117:0/3063022916 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054865118 0 0) 0x7fd2040010b0 con 0x7fd2140e5240 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd211d9b700 1 -- 172.21.2.117:0/3063022916 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200002fe0 con 0x7fd2140e5240 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd21359e700 1 -- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 msgr2=0x7fd2141b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd21359e700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 0x7fd2141b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd21359e700 1 -- 172.21.2.117:0/3063022916 >> v1:172.21.2.117:6789/0 conn(0x7fd2140e5240 legacy=0x7fd214084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd21359e700 1 -- 172.21.2.117:0/3063022916 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2140d1be0 con 0x7fd214085450 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd212d9d700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 0x7fd2141b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd21359e700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd214085450 0x7fd2140d05b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fd20c006e50 tx=0x7fd20c009e70).ready entity=mon.1 client_cookie=7e8ef709bdbf50f4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:01.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd211d9b700 1 -- 172.21.2.117:0/3063022916 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c00b510 con 0x7fd214085450 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd211d9b700 1 -- 172.21.2.117:0/3063022916 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c006790 con 0x7fd214085450 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd211d9b700 1 -- 172.21.2.117:0/3063022916 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd20c00b940 con 0x7fd214085450 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd219152700 1 -- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd214085450 msgr2=0x7fd2140d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.482 7fd219152700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd214085450 0x7fd2140d05b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7fd20c006e50 tx=0x7fd20c009e70).stop 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 -- 172.21.2.117:0/3063022916 shutdown_connections 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd214085450 0x7fd2140d05b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 --2- 172.21.2.117:0/3063022916 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140e4e40 0x7fd2141b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 -- 172.21.2.117:0/3063022916 >> 172.21.2.117:0/3063022916 conn(0x7fd2140ce4e0 msgr2=0x7fd2140e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 -- 172.21.2.117:0/3063022916 shutdown_connections 2021-02-10T11:56:01.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.483 7fd219152700 1 -- 172.21.2.117:0/3063022916 wait complete. 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.497 7fd219152700 1 Processor -- start 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.497 7fd219152700 1 -- start start 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2140a31e0 0x7fd2141a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140b0140 0x7fd2140a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2140a35e0 con 0x7fd2141a7f00 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214092fd0 con 0x7fd2140a31e0 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd219152700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd214093490 con 0x7fd2140b0140 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59254/0 (socket says 172.21.2.117:59254) 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 -- 172.21.2.117:0/3868301520 learned_addr learned my addr 172.21.2.117:0/3868301520 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd212d9d700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2140a31e0 0x7fd2141a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21359e700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140b0140 0x7fd2140a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 -- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140b0140 msgr2=0x7fd2140a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140b0140 0x7fd2140a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 -- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2140a31e0 msgr2=0x7fd2141a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2140a31e0 0x7fd2141a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 -- 172.21.2.117:0/3868301520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2140b0540 con 0x7fd2141a7f00 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.498 7fd21259c700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7fd204001c60 tx=0x7fd20400a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:01.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd204000e00 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20400acc0 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2040045d0 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd219152700 1 -- 172.21.2.117:0/3868301520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2140b0540 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd219152700 1 -- 172.21.2.117:0/3868301520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2140b0540 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd204004140 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.499 7fd20b7fe700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 0x7fd1f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.500 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fd2040326f0 con 0x7fd2141a7f00 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.500 7fd212d9d700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 0x7fd1f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:01.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.500 7fd212d9d700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 0x7fd1f801abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:01.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.531 7fd219152700 1 -- 172.21.2.117:0/3868301520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2140a35e0 con 0x7fd2141a7f00 2021-02-10T11:56:01.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.533 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd204026f30 con 0x7fd2141a7f00 2021-02-10T11:56:01.661 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:01.660 7fd219152700 1 -- 172.21.2.117:0/3868301520 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2140a35e0 con 0x7fd2141a7f00 2021-02-10T11:56:02.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.053 7fd20b7fe700 1 -- 172.21.2.117:0/3868301520 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v172) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd204004140 con 0x7fd2141a7f00 2021-02-10T11:56:02.076 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:56:02.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 msgr2=0x7fd1f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 0x7fd1f801abc0 crc :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 msgr2=0x7fd2140afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 rx=0x7fd204001c60 tx=0x7fd20400a8d0).stop 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 shutdown_connections 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd1f80186c0 0x7fd1f801abc0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2141a7f00 0x7fd2140afb20 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2140b0140 0x7fd2140a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 --2- 172.21.2.117:0/3868301520 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2140a31e0 0x7fd2141a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 >> 172.21.2.117:0/3868301520 conn(0x7fd2140b5d10 msgr2=0x7fd2140c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:02.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 shutdown_connections 2021-02-10T11:56:02.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.077 7fd219152700 1 -- 172.21.2.117:0/3868301520 wait complete. 2021-02-10T11:56:02.088 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:56:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 Processor -- start 2021-02-10T11:56:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 -- start start 2021-02-10T11:56:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f01c80e5240 0x7f01c8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c8085930 con 0x7f01c80e4e40 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c81b09e0 con 0x7f01c8084c60 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01ceb04700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c80e5cc0 con 0x7f01c80e5240 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01cc920700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f01c80e4e40 0x7f01c81b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59508/0 (socket says 172.21.2.117:59508) 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.503 7f01cc920700 1 -- 172.21.2.117:0/2155746280 learned_addr learned my addr 172.21.2.117:0/2155746280 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840324795 0 0) 0x7f01b0001180 con 0x7f01c80e4e40 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b4002fe0 con 0x7f01c80e4e40 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c7fff700 1 --2- 172.21.2.117:0/2155746280 >> v2:172.21.2.125:3300/0 conn(0x7f01c80e5240 0x7f01c8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1249062448 0 0) 0x7f01b0001180 con 0x7f01c80e4e40 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 >> v2:172.21.2.125:3300/0 conn(0x7f01c80e5240 msgr2=0x7f01c8086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 --2- 172.21.2.117:0/2155746280 >> v2:172.21.2.125:3300/0 conn(0x7f01c80e5240 0x7f01c8086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 >> v1:172.21.2.125:6789/0 conn(0x7f01c8084c60 legacy=0x7f01c8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8084840 con 0x7f01c80e4e40 2021-02-10T11:56:02.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f01b0001680 con 0x7f01c80e4e40 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.504 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 >> v1:172.21.2.117:6789/0 conn(0x7f01c80e4e40 legacy=0x7f01c81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 0x7f01b4005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b4006790 0x7f01b4008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b4009590 0x7f01b400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b400c300 con 0x7f01b4003ac0 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b400c520 con 0x7f01b4009590 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b400c740 con 0x7f01b4006790 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 0x7f01b4005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cd121700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b4009590 0x7f01b400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c7fff700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b4006790 0x7f01b4008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 -- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b4006790 msgr2=0x7f01b4008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b4006790 0x7f01b4008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 -- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b4009590 msgr2=0x7f01b400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:02.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01b4009590 0x7f01b400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 -- 172.21.2.117:0/2155746280 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b40509b0 con 0x7f01b4003ac0 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cd121700 1 -- 172.21.2.117:0/2155746280 reap_dead start 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c7fff700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01b4006790 0x7f01b4008c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01cc920700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 0x7f01b4005ef0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f01b0000e80 tx=0x7f01b0007cf0).ready entity=mon.0 client_cookie=feb2779b72cefcd7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b00070c0 con 0x7f01b4003ac0 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.505 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0001140 con 0x7f01b4003ac0 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01c77fe700 1 -- 172.21.2.117:0/2155746280 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b000a710 con 0x7f01b4003ac0 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 -- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 msgr2=0x7f01b4005ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 0x7f01b4005ef0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 rx=0x7f01b0000e80 tx=0x7f01b0007cf0).stop 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 -- 172.21.2.117:0/2155746280 shutdown_connections 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 --2- 172.21.2.117:0/2155746280 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01b4003ac0 0x7f01b4005ef0 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:02.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 -- 172.21.2.117:0/2155746280 >> 172.21.2.117:0/2155746280 conn(0x7f01c80ce4e0 msgr2=0x7f01c80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 -- 172.21.2.117:0/2155746280 shutdown_connections 2021-02-10T11:56:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.506 7f01ceb04700 1 -- 172.21.2.117:0/2155746280 wait complete. 2021-02-10T11:56:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.521 7f01ceb04700 1 Processor -- start 2021-02-10T11:56:02.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.521 7f01ceb04700 1 -- start start 2021-02-10T11:56:02.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01c80a0aa0 0x7f01c819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01c8090ae0 0x7f01c8090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c81109c0 con 0x7f01c810ffa0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80a0ea0 con 0x7f01c80a0aa0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01ceb04700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c81a0360 con 0x7f01c8090ae0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59272/0 (socket says 172.21.2.117:59272) 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 -- 172.21.2.117:0/1781312462 learned_addr learned my addr 172.21.2.117:0/1781312462 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 -- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01c8090ae0 msgr2=0x7f01c8090ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01c8090ae0 0x7f01c8090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 -- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01c80a0aa0 msgr2=0x7f01c819fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01c80a0aa0 0x7f01c819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.522 7f01c7fff700 1 -- 172.21.2.117:0/1781312462 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8110d00 con 0x7f01c810ffa0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01c7fff700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7f01b800b610 tx=0x7f01b800b9b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b80009b0 con 0x7f01c810ffa0 2021-02-10T11:56:02.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8006360 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f01b8002030 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c8110d00 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.523 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c80917f0 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.524 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f01b8000c90 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.524 7f01c57fa700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 0x7f01ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.524 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f01b8032590 con 0x7f01c810ffa0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.524 7f01cc920700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 0x7f01ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:02.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.525 7f01cc920700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 0x7f01ac01abc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:02.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.553 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c810c2a0 con 0x7f01c810ffa0 2021-02-10T11:56:02.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.556 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f01b8000940 con 0x7f01c810ffa0 2021-02-10T11:56:02.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:02.683 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f01c8085930 con 0x7f01c810ffa0 2021-02-10T11:56:03.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.056 7f01c57fa700 1 -- 172.21.2.117:0/1781312462 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v173) v1 ==== 173+0+0 (secure 0 0 0) 0x7f01b8000c90 con 0x7f01c810ffa0 2021-02-10T11:56:03.101 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:56:03.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 msgr2=0x7f01ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 0x7f01ac01abc0 crc :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 msgr2=0x7f01c81103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7f01b800b610 tx=0x7f01b800b9b0).stop 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 shutdown_connections 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f01ac0186c0 0x7f01ac01abc0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f01c810ffa0 0x7f01c81103a0 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f01c8090ae0 0x7f01c8090ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.101 7f01ceb04700 1 --2- 172.21.2.117:0/1781312462 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f01c80a0aa0 0x7f01c819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.102 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 >> 172.21.2.117:0/1781312462 conn(0x7f01c80a5e90 msgr2=0x7f01c80b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.102 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 shutdown_connections 2021-02-10T11:56:03.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.102 7f01ceb04700 1 -- 172.21.2.117:0/1781312462 wait complete. 2021-02-10T11:56:03.112 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:56:03.112 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:56:03.112 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:56:03.112 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:56:03.112 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:56:03.112 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:56:03.113 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:56:03.113 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:56:03.118 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:56:03.118 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:56:03.120 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:56:03.120 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:56:03.161 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:56:03.161 DEBUG:teuthology.parallel:result is None 2021-02-10T11:56:03.162 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:56:03.162 DEBUG:teuthology.parallel:result is None 2021-02-10T11:56:03.162 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:56:03.568 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:56:03.569 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:56:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.570 7f76836d6700 1 Processor -- start 2021-02-10T11:56:03.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.570 7f76836d6700 1 -- start start 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f76836d6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c0e7180 0x7f767c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f76836d6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f767c0e8390 0x7f767c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f76836d6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c0e7580 con 0x7f767c1b07f0 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f76836d6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c0e5ce0 con 0x7f767c0e8390 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f76836d6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0e77f0 con 0x7f767c0e7180 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f767c0e8390 0x7f767c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f767c0e8390 0x7f767c0e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59288/0 (socket says 172.21.2.117:59288) 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 -- 172.21.2.117:0/1829813297 learned_addr learned my addr 172.21.2.117:0/1829813297 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7680cf1700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c0e7180 0x7f767c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 -- 172.21.2.117:0/1829813297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c0e7180 msgr2=0x7f767c0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c0e7180 0x7f767c0e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.591 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 -- 172.21.2.117:0/1829813297 >> v1:172.21.2.125:6789/0 conn(0x7f767c1b07f0 legacy=0x7f767c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.571 7f7681cf3700 1 -- 172.21.2.117:0/1829813297 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0e8db0 con 0x7f767c0e8390 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f7681cf3700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c0e8390 0x7f767c0e8790 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f7678006f00 tx=0x7f7678009f10).ready entity=mon.0 client_cookie=7cd624f0aacfc8f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f7673fff700 1 -- 172.21.2.117:0/1829813297 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800b3b0 con 0x7f767c0e8390 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f7673fff700 1 -- 172.21.2.117:0/1829813297 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800b630 con 0x7f767c0e8390 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f7673fff700 1 -- 172.21.2.117:0/1829813297 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f767800bb50 con 0x7f767c0e8390 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 -- 172.21.2.117:0/1829813297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c0e8390 msgr2=0x7f767c0e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c0e8390 0x7f767c0e8790 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 rx=0x7f7678006f00 tx=0x7f7678009f10).stop 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 -- 172.21.2.117:0/1829813297 shutdown_connections 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c0e8390 0x7f767c0e8790 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 --2- 172.21.2.117:0/1829813297 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c0e7180 0x7f767c0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.572 7f76836d6700 1 -- 172.21.2.117:0/1829813297 >> 172.21.2.117:0/1829813297 conn(0x7f767c0ce500 msgr2=0x7f767c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.573 7f76836d6700 1 -- 172.21.2.117:0/1829813297 shutdown_connections 2021-02-10T11:56:03.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.573 7f76836d6700 1 -- 172.21.2.117:0/1829813297 wait complete. 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.587 7f76836d6700 1 Processor -- start 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.587 7f76836d6700 1 -- start start 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 0x7f767c0947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 0x7f767c08c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c106e10 con 0x7f767c19a820 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c08bf50 con 0x7f767c0af630 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76836d6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0c4090 con 0x7f767c094e00 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59296/0 (socket says 172.21.2.117:59296) 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 -- 172.21.2.117:0/3226296107 learned_addr learned my addr 172.21.2.117:0/3226296107 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f7681cf3700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 0x7f767c08c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f7680cf1700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 0x7f767c0947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 -- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 msgr2=0x7f767c08c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 0x7f767c08c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 -- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 msgr2=0x7f767c0947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 0x7f767c0947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.588 7f76814f2700 1 -- 172.21.2.117:0/3226296107 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c08caf0 con 0x7f767c19a820 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f7681cf3700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 0x7f767c08c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f7680cf1700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 0x7f767c0947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f76814f2700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f766c00b5b0 tx=0x7f766c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f766c00bd40 con 0x7f767c19a820 2021-02-10T11:56:03.595 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c006020 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f766c0054a0 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f76836d6700 1 -- 172.21.2.117:0/3226296107 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c08caf0 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.589 7f76836d6700 1 -- 172.21.2.117:0/3226296107 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c08caf0 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.590 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f766c006200 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.590 7f7671ffb700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 0x7f765c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.590 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f766c032460 con 0x7f767c19a820 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.590 7f7680cf1700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 0x7f765c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:03.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.590 7f7680cf1700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 0x7f765c01abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:03.621 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.619 7f76836d6700 1 -- 172.21.2.117:0/3226296107 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c07e800 con 0x7f767c19a820 2021-02-10T11:56:03.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.622 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f766c000c60 con 0x7f767c19a820 2021-02-10T11:56:03.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:03.760 7f76836d6700 1 -- 172.21.2.117:0/3226296107 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f767c0e77f0 con 0x7f767c19a820 2021-02-10T11:56:04.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.095 7f7671ffb700 1 -- 172.21.2.117:0/3226296107 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v174) v1 ==== 97+0+0 (secure 0 0 0) 0x7f766c006200 con 0x7f767c19a820 2021-02-10T11:56:04.128 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:56:04.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 msgr2=0x7f765c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 0x7f765c01abc0 crc :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 msgr2=0x7f767c0af010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 rx=0x7f766c00b5b0 tx=0x7f766c00b950).stop 2021-02-10T11:56:04.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 shutdown_connections 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f767c094e00 0x7f767c08c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f767c0af630 0x7f767c0947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f765c0186c0 0x7f765c01abc0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 --2- 172.21.2.117:0/3226296107 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f767c19a820 0x7f767c0af010 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 >> 172.21.2.117:0/3226296107 conn(0x7f767c0953a0 msgr2=0x7f767c09e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 shutdown_connections 2021-02-10T11:56:04.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.128 7f76836d6700 1 -- 172.21.2.117:0/3226296107 wait complete. 2021-02-10T11:56:04.138 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:56:04.139 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 Processor -- start 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 -- start start 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11780ef3f0 0x7f11780eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11780ede60 con 0x7f11780d18e0 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11780d1ce0 con 0x7f11780eb7f0 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f117cea0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11780ef7f0 con 0x7f11780ef3f0 2021-02-10T11:56:04.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f1176b13700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f11780d18e0 0x7f11780ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59552/0 (socket says 172.21.2.117:59552) 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.549 7f1176b13700 1 -- 172.21.2.117:0/4018477571 learned_addr learned my addr 172.21.2.117:0/4018477571 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214443035 0 0) 0x7f1160001180 con 0x7f11780d18e0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1164002fe0 con 0x7f11780d18e0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 563920697 0 0) 0x7f11600010e0 con 0x7f11780d18e0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11780ef3f0 msgr2=0x7f11780eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176312700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11780ef3f0 0x7f11780eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11780ef3f0 0x7f11780eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 >> v1:172.21.2.125:6789/0 conn(0x7f11780eb7f0 legacy=0x7f11781b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11781bb180 con 0x7f11780d18e0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f1160001600 con 0x7f11780d18e0 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 >> v1:172.21.2.117:6789/0 conn(0x7f11780d18e0 legacy=0x7f11780ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1164003b20 0x7f1164005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1164006810 0x7f1164008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1164009630 0x7f116400bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116400c370 con 0x7f1164003b20 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116400c590 con 0x7f1164006810 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1175b11700 1 -- 172.21.2.117:0/4018477571 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116400c7b0 con 0x7f1164009630 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176b13700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1164003b20 0x7f1164005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176312700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11780ef3f0 0x7f11780eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176b13700 1 -- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1164009630 msgr2=0x7f116400bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176b13700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1164009630 0x7f116400bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176b13700 1 -- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1164006810 msgr2=0x7f1164008d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.550 7f1176b13700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f1164006810 0x7f1164008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1176b13700 1 -- 172.21.2.117:0/4018477571 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116400ca50 con 0x7f1164003b20 2021-02-10T11:56:04.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1176b13700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1164003b20 0x7f1164005f90 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f11600018a0 tx=0x7f1160007d40).ready entity=mon.0 client_cookie=7c41e5cda3a69f40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1160001470 con 0x7f1164003b20 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1160007150 con 0x7f1164003b20 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1175b11700 1 -- 172.21.2.117:0/4018477571 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11600045d0 con 0x7f1164003b20 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 -- 172.21.2.117:0/4018477571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1164003b20 msgr2=0x7f1164005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 --2- 172.21.2.117:0/4018477571 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f1164003b20 0x7f1164005f90 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7f11600018a0 tx=0x7f1160007d40).stop 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f1177314700 1 -- 172.21.2.117:0/4018477571 reap_dead start 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 -- 172.21.2.117:0/4018477571 shutdown_connections 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 -- 172.21.2.117:0/4018477571 >> 172.21.2.117:0/4018477571 conn(0x7f11780ced60 msgr2=0x7f11780cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 -- 172.21.2.117:0/4018477571 shutdown_connections 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.551 7f117cea0700 1 -- 172.21.2.117:0/4018477571 wait complete. 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 Processor -- start 2021-02-10T11:56:04.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 -- start start 2021-02-10T11:56:04.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11780b5da0 0x7f11781a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11781a3480 0x7f11780b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780b61a0 con 0x7f11780b59a0 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117808c8b0 con 0x7f11780b5da0 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f117cea0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11780ba1c0 con 0x7f11781a3480 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f1176b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f1176b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59314/0 (socket says 172.21.2.117:59314) 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.566 7f1176b13700 1 -- 172.21.2.117:0/1555050755 learned_addr learned my addr 172.21.2.117:0/1555050755 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 -- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11781a3480 msgr2=0x7f11780b2590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11781a3480 0x7f11780b2590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 -- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11780b5da0 msgr2=0x7f11781a2e60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11780b5da0 0x7f11781a2e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 -- 172.21.2.117:0/1555050755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11781a3880 con 0x7f11780b59a0 2021-02-10T11:56:04.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f1176b13700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f1160007b60 tx=0x7f116000c990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1160007340 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11600043a0 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f11600048c0 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f117cea0700 1 -- 172.21.2.117:0/1555050755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11781a3880 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.567 7f117cea0700 1 -- 172.21.2.117:0/1555050755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11781a3880 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.568 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f116000b000 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.568 7f116f7fe700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 0x7f115c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.568 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f11600325c0 con 0x7f11780b59a0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.568 7f1176312700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 0x7f115c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:04.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.569 7f1176312700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 0x7f115c01abc0 crc :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:04.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.599 7f117cea0700 1 -- 172.21.2.117:0/1555050755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11780ede60 con 0x7f11780b59a0 2021-02-10T11:56:04.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.602 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1160007ba0 con 0x7f11780b59a0 2021-02-10T11:56:04.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.732 7f117cea0700 1 -- 172.21.2.117:0/1555050755 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f11780ede60 con 0x7f11780b59a0 2021-02-10T11:56:04.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.732 7f116f7fe700 1 -- 172.21.2.117:0/1555050755 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v99) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f116000cd80 con 0x7f11780b59a0 2021-02-10T11:56:04.734 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.735 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:56:04.736 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.737 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.738 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.739 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.740 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.741 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.742 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:56:04.743 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:56:04.744 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.745 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:56:04.746 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:04.747 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:56:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 msgr2=0x7f115c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 0x7f115c01abc0 crc :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 msgr2=0x7f117808c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f1160007b60 tx=0x7f116000c990).stop 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 shutdown_connections 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f11781a3480 0x7f11780b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f11780b5da0 0x7f11781a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f115c0186c0 0x7f115c01abc0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 --2- 172.21.2.117:0/1555050755 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f11780b59a0 0x7f117808c290 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 >> 172.21.2.117:0/1555050755 conn(0x7f11780bb5e0 msgr2=0x7f1178090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 shutdown_connections 2021-02-10T11:56:04.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:04.734 7f117cea0700 1 -- 172.21.2.117:0/1555050755 wait complete. 2021-02-10T11:56:04.750 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:05.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.161 7f3dca24c700 1 Processor -- start 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.161 7f3dca24c700 1 -- start start 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dca24c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40d18e0 0x7f3dc40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dca24c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3dc40eb000 0x7f3dc40eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dca24c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc40ede60 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dca24c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc40eba20 con 0x7f3dc40eb000 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dca24c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40d1ce0 con 0x7f3dc40d18e0 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dc8869700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3dc40ee3e0 0x7f3dc40ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59572/0 (socket says 172.21.2.117:59572) 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dc8869700 1 -- 172.21.2.117:0/3352264708 learned_addr learned my addr 172.21.2.117:0/3352264708 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823214355 0 0) 0x7f3dbc001500 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.162 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac002fe0 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40d18e0 0x7f3dc40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc37fe700 1 --2- 172.21.2.117:0/3352264708 >> v2:172.21.2.125:3300/0 conn(0x7f3dc40eb000 0x7f3dc40eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2205364873 0 0) 0x7f3dbc001460 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40d18e0 msgr2=0x7f3dc40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40d18e0 0x7f3dc40ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 >> v2:172.21.2.125:3300/0 conn(0x7f3dc40eb000 msgr2=0x7f3dc40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 --2- 172.21.2.117:0/3352264708 >> v2:172.21.2.125:3300/0 conn(0x7f3dc40eb000 0x7f3dc40eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40eefd0 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3dbc001980 con 0x7f3dc40ee3e0 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 >> v1:172.21.2.117:6789/0 conn(0x7f3dc40ee3e0 legacy=0x7f3dc40ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dac003ae0 0x7f3dac005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dac006810 0x7f3dac008d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dac009630 0x7f3dac00bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00c370 con 0x7f3dac003ae0 2021-02-10T11:56:05.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac050560 con 0x7f3dac006810 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac050780 con 0x7f3dac009630 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40d18e0 0x7f3dc40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dac003ae0 0x7f3dac005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc37fe700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dac006810 0x7f3dac008d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc8869700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dac009630 0x7f3dac00bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 -- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dac009630 msgr2=0x7f3dac00bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dac009630 0x7f3dac00bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 -- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dac006810 msgr2=0x7f3dac008d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dac006810 0x7f3dac008d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.163 7f3dc3fff700 1 -- 172.21.2.117:0/3352264708 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac050a20 con 0x7f3dac003ae0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dc3fff700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dac003ae0 0x7f3dac005f70 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7f3db4006890 tx=0x7f3db4009aa0).ready entity=mon.0 client_cookie=f3f7e830617dbbe8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db400c0c0 con 0x7f3dac003ae0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db400c300 con 0x7f3dac003ae0 2021-02-10T11:56:05.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dc2ffd700 1 -- 172.21.2.117:0/3352264708 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db400c820 con 0x7f3dac003ae0 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 -- 172.21.2.117:0/3352264708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dac003ae0 msgr2=0x7f3dac005f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 --2- 172.21.2.117:0/3352264708 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dac003ae0 0x7f3dac005f70 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7f3db4006890 tx=0x7f3db4009aa0).stop 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dc8869700 1 -- 172.21.2.117:0/3352264708 reap_dead start 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 -- 172.21.2.117:0/3352264708 shutdown_connections 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 -- 172.21.2.117:0/3352264708 >> 172.21.2.117:0/3352264708 conn(0x7f3dc40ced60 msgr2=0x7f3dc40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 -- 172.21.2.117:0/3352264708 shutdown_connections 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.164 7f3dca24c700 1 -- 172.21.2.117:0/3352264708 wait complete. 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.179 7f3dca24c700 1 Processor -- start 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.179 7f3dca24c700 1 -- start start 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 0x7f3dc41a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dc41a3480 0x7f3dc40b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40b61a0 con 0x7f3dc40b59a0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc408c8b0 con 0x7f3dc41a3480 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dca24c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc40ba1c0 con 0x7f3dc40b5da0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59332/0 (socket says 172.21.2.117:59332) 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 -- 172.21.2.117:0/2495945687 learned_addr learned my addr 172.21.2.117:0/2495945687 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc37fe700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 0x7f3dc41a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc8869700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dc41a3480 0x7f3dc40b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 -- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 msgr2=0x7f3dc41a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 0x7f3dc41a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 -- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dc41a3480 msgr2=0x7f3dc40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dc41a3480 0x7f3dc40b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc3fff700 1 -- 172.21.2.117:0/2495945687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc41a3880 con 0x7f3dc40b59a0 2021-02-10T11:56:05.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.180 7f3dc37fe700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 0x7f3dc41a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dc3fff700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7f3db400b630 tx=0x7f3db400b9d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db4002030 con 0x7f3dc40b59a0 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db4002270 con 0x7f3dc40b59a0 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3db4002790 con 0x7f3dc40b59a0 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc41a3880 con 0x7f3dc40b59a0 2021-02-10T11:56:05.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc40b2bb0 con 0x7f3dc40b59a0 2021-02-10T11:56:05.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.181 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3db4000c90 con 0x7f3dc40b59a0 2021-02-10T11:56:05.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.182 7f3dc0ff9700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 0x7f3da401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.182 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f3db40323d0 con 0x7f3dc40b59a0 2021-02-10T11:56:05.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.182 7f3dc37fe700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 0x7f3da401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.182 7f3dc37fe700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 0x7f3da401abc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.211 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc40ede60 con 0x7f3dc40b59a0 2021-02-10T11:56:05.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.213 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3db401abb0 con 0x7f3dc40b59a0 2021-02-10T11:56:05.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.337 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dc40ede60 con 0x7f3dc40b59a0 2021-02-10T11:56:05.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.338 7f3dc0ff9700 1 -- 172.21.2.117:0/2495945687 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f3db401abb0 con 0x7f3dc40b59a0 2021-02-10T11:56:05.341 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:05.341 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":174,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:04.090904","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":168,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":167,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.339 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 msgr2=0x7f3da401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.339 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 0x7f3da401abc0 crc :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.339 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 msgr2=0x7f3dc408c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.339 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7f3db400b630 tx=0x7f3db400b9d0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 shutdown_connections 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3dc41a3480 0x7f3dc40b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3dc40b5da0 0x7f3dc41a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3da40186c0 0x7f3da401abc0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 --2- 172.21.2.117:0/2495945687 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3dc40b59a0 0x7f3dc408c290 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 >> 172.21.2.117:0/2495945687 conn(0x7f3dc40bb5e0 msgr2=0x7f3dc4090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 shutdown_connections 2021-02-10T11:56:05.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.340 7f3dca24c700 1 -- 172.21.2.117:0/2495945687 wait complete. 2021-02-10T11:56:05.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:56:05.352 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:56:05.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.766 7fe4990ac700 1 Processor -- start 2021-02-10T11:56:05.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.766 7fe4990ac700 1 -- start start 2021-02-10T11:56:05.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe4990ac700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4940d18f0 0x7fe4940eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.785 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe4990ac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4940ef3f0 0x7fe4941b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe4990ac700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4940ef7f0 con 0x7fe4940eb040 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe4990ac700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4940eded0 con 0x7fe4940d18f0 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe4990ac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4940d1cf0 con 0x7fe4940ef3f0 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4940d18f0 0x7fe4940eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe4940d18f0 0x7fe4940eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59340/0 (socket says 172.21.2.117:59340) 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 -- 172.21.2.117:0/3873212441 learned_addr learned my addr 172.21.2.117:0/3873212441 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 -- 172.21.2.117:0/3873212441 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4940ef3f0 msgr2=0x7fe4941b79a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 --2- 172.21.2.117:0/3873212441 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4940ef3f0 0x7fe4941b79a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 -- 172.21.2.117:0/3873212441 >> v1:172.21.2.125:6789/0 conn(0x7fe4940eb040 legacy=0x7fe4940eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 -- 172.21.2.117:0/3873212441 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4941b8fd0 con 0x7fe4940d18f0 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.767 7fe492d9d700 1 --2- 172.21.2.117:0/3873212441 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940d18f0 0x7fe4940eaa20 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7fe47c006700 tx=0x7fe47c009b80).ready entity=mon.0 client_cookie=a1b88b102e8622f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.786 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe491d9b700 1 -- 172.21.2.117:0/3873212441 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe47c00c0c0 con 0x7fe4940d18f0 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe491d9b700 1 -- 172.21.2.117:0/3873212441 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00c400 con 0x7fe4940d18f0 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe491d9b700 1 -- 172.21.2.117:0/3873212441 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe47c00c920 con 0x7fe4940d18f0 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 -- 172.21.2.117:0/3873212441 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940d18f0 msgr2=0x7fe4940eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 --2- 172.21.2.117:0/3873212441 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940d18f0 0x7fe4940eaa20 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7fe47c006700 tx=0x7fe47c009b80).stop 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 -- 172.21.2.117:0/3873212441 shutdown_connections 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 --2- 172.21.2.117:0/3873212441 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe4940ef3f0 0x7fe4941b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 --2- 172.21.2.117:0/3873212441 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940d18f0 0x7fe4940eaa20 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 -- 172.21.2.117:0/3873212441 >> 172.21.2.117:0/3873212441 conn(0x7fe4940cf060 msgr2=0x7fe4940cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.768 7fe4990ac700 1 -- 172.21.2.117:0/3873212441 shutdown_connections 2021-02-10T11:56:05.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.769 7fe4990ac700 1 -- 172.21.2.117:0/3873212441 wait complete. 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 Processor -- start 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 -- start start 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe49407e7d0 0x7fe49407ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4941a2e50 0x7fe4941a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49407f890 con 0x7fe4940a67e0 2021-02-10T11:56:05.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49407f310 con 0x7fe4941a2e50 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe4990ac700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4941a3870 con 0x7fe49407e7d0 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe492d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe492d9d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59348/0 (socket says 172.21.2.117:59348) 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.783 7fe492d9d700 1 -- 172.21.2.117:0/2935123850 learned_addr learned my addr 172.21.2.117:0/2935123850 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 -- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe49407e7d0 msgr2=0x7fe49407ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe49407e7d0 0x7fe49407ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 -- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4941a2e50 msgr2=0x7fe4941a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4941a2e50 0x7fe4941a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 -- 172.21.2.117:0/2935123850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4941a3a90 con 0x7fe4940a67e0 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe492d9d700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7fe47c00b580 tx=0x7fe47c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe47c00bd10 con 0x7fe4940a67e0 2021-02-10T11:56:05.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c0052e0 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe47c005800 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.784 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4941a3a90 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.785 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49409f8e0 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.785 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe47c006030 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.785 7fe48b7fe700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 0x7fe47801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.785 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fe47c032260 con 0x7fe4940a67e0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.786 7fe49259c700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 0x7fe47801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:05.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.786 7fe49259c700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 0x7fe47801abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:05.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.817 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4940ede60 con 0x7fe4940a67e0 2021-02-10T11:56:05.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.819 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe47c030d70 con 0x7fe4940a67e0 2021-02-10T11:56:05.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.947 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe4940ede60 con 0x7fe4940a67e0 2021-02-10T11:56:05.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.948 7fe48b7fe700 1 -- 172.21.2.117:0/2935123850 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v174) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe47c00bd10 con 0x7fe4940a67e0 2021-02-10T11:56:05.951 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.949 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 msgr2=0x7fe47801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 0x7fe47801abc0 crc :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 msgr2=0x7fe4940a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7fe47c00b580 tx=0x7fe47c00b920).stop 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 shutdown_connections 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe4941a2e50 0x7fe4941a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.953 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe49407e7d0 0x7fe49407ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:05.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe4780186c0 0x7fe47801abc0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 --2- 172.21.2.117:0/2935123850 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe4940a67e0 0x7fe4940a5f80 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:05.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 >> 172.21.2.117:0/2935123850 conn(0x7fe49409e660 msgr2=0x7fe4940bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:05.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 shutdown_connections 2021-02-10T11:56:05.954 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:05.950 7fe4990ac700 1 -- 172.21.2.117:0/2935123850 wait complete. 2021-02-10T11:56:05.962 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:56:06.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 Processor -- start 2021-02-10T11:56:06.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 -- start start 2021-02-10T11:56:06.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9eb41b6960 0x7f9eb41b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40ede60 con 0x7f9eb40d18e0 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40eb9c0 con 0x7f9eb40ef3f0 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb9588700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb40ef7f0 con 0x7f9eb41b6960 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb3ac0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9eb41b6960 0x7f9eb41b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb32bf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9eb40d18e0 0x7f9eb40ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59604/0 (socket says 172.21.2.117:59604) 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.371 7f9eb32bf700 1 -- 172.21.2.117:0/4243753882 learned_addr learned my addr 172.21.2.117:0/4243753882 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb3ac0700 1 -- 172.21.2.117:0/4243753882 >> v1:172.21.2.125:6789/0 conn(0x7f9eb40ef3f0 legacy=0x7f9eb40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb3ac0700 1 -- 172.21.2.117:0/4243753882 >> v1:172.21.2.117:6789/0 conn(0x7f9eb40d18e0 legacy=0x7f9eb40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb3ac0700 1 -- 172.21.2.117:0/4243753882 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb40d1d20 con 0x7f9eb41b6960 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb3ac0700 1 --2- 172.21.2.117:0/4243753882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb41b6960 0x7f9eb41b8d70 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7f9eac006e30 tx=0x7f9eac009e60).ready entity=mon.0 client_cookie=82c9b68686b7e814 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb22bd700 1 -- 172.21.2.117:0/4243753882 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eac00b300 con 0x7f9eb41b6960 2021-02-10T11:56:06.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb22bd700 1 -- 172.21.2.117:0/4243753882 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac00b540 con 0x7f9eb41b6960 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb22bd700 1 -- 172.21.2.117:0/4243753882 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9eac00b9a0 con 0x7f9eb41b6960 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb9588700 1 -- 172.21.2.117:0/4243753882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb41b6960 msgr2=0x7f9eb41b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.372 7f9eb9588700 1 --2- 172.21.2.117:0/4243753882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb41b6960 0x7f9eb41b8d70 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7f9eac006e30 tx=0x7f9eac009e60).stop 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.373 7f9eb9588700 1 -- 172.21.2.117:0/4243753882 shutdown_connections 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.373 7f9eb9588700 1 --2- 172.21.2.117:0/4243753882 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb41b6960 0x7f9eb41b8d70 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.373 7f9eb9588700 1 -- 172.21.2.117:0/4243753882 >> 172.21.2.117:0/4243753882 conn(0x7f9eb40ced60 msgr2=0x7f9eb40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.373 7f9eb9588700 1 -- 172.21.2.117:0/4243753882 shutdown_connections 2021-02-10T11:56:06.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.373 7f9eb9588700 1 -- 172.21.2.117:0/4243753882 wait complete. 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 Processor -- start 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 -- start start 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9eb41b8410 0x7f9eb40eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9eb40ee7c0 0x7f9eb40eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb40efd40 con 0x7f9eb40f5980 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb41b8940 con 0x7f9eb40ee7c0 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb9588700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb40ee270 con 0x7f9eb41b8410 2021-02-10T11:56:06.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb2abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb2abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59360/0 (socket says 172.21.2.117:59360) 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.388 7f9eb2abe700 1 -- 172.21.2.117:0/3798855188 learned_addr learned my addr 172.21.2.117:0/3798855188 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 -- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9eb41b8410 msgr2=0x7f9eb40eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9eb41b8410 0x7f9eb40eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 -- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9eb40ee7c0 msgr2=0x7f9eb40eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9eb40ee7c0 0x7f9eb40eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 -- 172.21.2.117:0/3798855188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb40ef1e0 con 0x7f9eb40f5980 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb2abe700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f9ea800b5b0 tx=0x7f9ea800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea800bd40 con 0x7f9eb40f5980 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea80009b0 con 0x7f9eb40f5980 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ea8006210 con 0x7f9eb40f5980 2021-02-10T11:56:06.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb40ef1e0 con 0x7f9eb40f5980 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.389 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb40ef770 con 0x7f9eb40f5980 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.390 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ea8006020 con 0x7f9eb40f5980 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.390 7f9e9bfff700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 0x7f9e9401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.390 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f9ea8032450 con 0x7f9eb40f5980 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.390 7f9eb32bf700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 0x7f9e9401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.391 7f9eb32bf700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 0x7f9e9401abc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:06.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.421 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb40ef770 con 0x7f9eb40f5980 2021-02-10T11:56:06.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.424 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ea8000c60 con 0x7f9eb40f5980 2021-02-10T11:56:06.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.546 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9eb40ef770 con 0x7f9eb40f5980 2021-02-10T11:56:06.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.546 7f9e9bfff700 1 -- 172.21.2.117:0/3798855188 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v174) v1 ==== 84+0+64 (secure 0 0 0) 0x7f9ea8006000 con 0x7f9eb40f5980 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:06.550 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:56:06.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.548 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 msgr2=0x7f9e9401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.548 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 0x7f9e9401abc0 crc :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.548 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 msgr2=0x7f9eb40ec660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.548 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f9ea800b5b0 tx=0x7f9ea800b950).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 shutdown_connections 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9e940186c0 0x7f9e9401abc0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9eb40f5980 0x7f9eb40ec660 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9eb40ee7c0 0x7f9eb40eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 --2- 172.21.2.117:0/3798855188 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9eb41b8410 0x7f9eb40eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 >> 172.21.2.117:0/3798855188 conn(0x7f9eb4086d40 msgr2=0x7f9eb409c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 shutdown_connections 2021-02-10T11:56:06.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.549 7f9eb9588700 1 -- 172.21.2.117:0/3798855188 wait complete. 2021-02-10T11:56:06.560 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:56:06.560 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:56:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 Processor -- start 2021-02-10T11:56:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 -- start start 2021-02-10T11:56:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb4bc1b07f0 0x7fb4bc0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 0x7fb4bc0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4bc0e77f0 con 0x7fb4bc0e6170 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc0e54b0 con 0x7fb4bc1b07f0 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.975 7fb4c41f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0e9180 con 0x7fb4bc0e8d80 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.976 7fb4c2812700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 0x7fb4bc0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.976 7fb4c2011700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb4bc1b07f0 0x7fb4bc0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.976 7fb4c1810700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fb4bc0e6170 0x7fb4bc0e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52544/0 (socket says 172.21.2.117:52544) 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.976 7fb4c1810700 1 -- 172.21.2.117:0/2382649562 learned_addr learned my addr 172.21.2.117:0/2382649562 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c2812700 1 -- 172.21.2.117:0/2382649562 >> v2:172.21.2.125:3300/0 conn(0x7fb4bc1b07f0 msgr2=0x7fb4bc0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c2812700 1 --2- 172.21.2.117:0/2382649562 >> v2:172.21.2.125:3300/0 conn(0x7fb4bc1b07f0 0x7fb4bc0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c2812700 1 -- 172.21.2.117:0/2382649562 >> v1:172.21.2.125:6789/0 conn(0x7fb4bc0e6170 legacy=0x7fb4bc0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c2812700 1 -- 172.21.2.117:0/2382649562 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc0e7380 con 0x7fb4bc0e8d80 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c2812700 1 --2- 172.21.2.117:0/2382649562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 0x7fb4bc0e6d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb4b4006e50 tx=0x7fb4b4009e70).ready entity=mon.2 client_cookie=6753ba6fb69ca4f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:06.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c100f700 1 -- 172.21.2.117:0/2382649562 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b400b510 con 0x7fb4bc0e8d80 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c100f700 1 -- 172.21.2.117:0/2382649562 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b4006790 con 0x7fb4bc0e8d80 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c100f700 1 -- 172.21.2.117:0/2382649562 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b400b940 con 0x7fb4bc0e8d80 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c41f5700 1 -- 172.21.2.117:0/2382649562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 msgr2=0x7fb4bc0e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.977 7fb4c41f5700 1 --2- 172.21.2.117:0/2382649562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 0x7fb4bc0e6d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb4b4006e50 tx=0x7fb4b4009e70).stop 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 -- 172.21.2.117:0/2382649562 shutdown_connections 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 --2- 172.21.2.117:0/2382649562 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0e8d80 0x7fb4bc0e6d60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 --2- 172.21.2.117:0/2382649562 >> v2:172.21.2.125:3300/0 conn(0x7fb4bc1b07f0 0x7fb4bc0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 -- 172.21.2.117:0/2382649562 >> 172.21.2.117:0/2382649562 conn(0x7fb4bc0ce500 msgr2=0x7fb4bc0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 -- 172.21.2.117:0/2382649562 shutdown_connections 2021-02-10T11:56:06.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.978 7fb4c41f5700 1 -- 172.21.2.117:0/2382649562 wait complete. 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 Processor -- start 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 -- start start 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0b8ac0 0x7fb4bc19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4bc19df40 0x7fb4bc0bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0bb430 con 0x7fb4bc0bbfd0 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0bc9f0 con 0x7fb4bc19df40 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c41f5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0bcc10 con 0x7fb4bc0b8ac0 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c2812700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c2812700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59372/0 (socket says 172.21.2.117:59372) 2021-02-10T11:56:06.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c2812700 1 -- 172.21.2.117:0/3519263920 learned_addr learned my addr 172.21.2.117:0/3519263920 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.993 7fb4c2812700 1 -- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0b8ac0 msgr2=0x7fb4bc19d920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c2812700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0b8ac0 0x7fb4bc19d920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c2812700 1 -- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4bc19df40 msgr2=0x7fb4bc0bb9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c2812700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4bc19df40 0x7fb4bc0bb9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c2812700 1 -- 172.21.2.117:0/3519263920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc0b6f70 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c2812700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fb4b4006c50 tx=0x7fb4b400bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b40093d0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b400be60 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb4b400c4e0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc0b6f70 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.994 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc0b6f70 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.995 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb4b4002ef0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.995 7fb4aeffd700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 0x7fb4a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:07.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.995 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fb4b4032770 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.995 7fb4c2011700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 0x7fb4a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:07.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:06.996 7fb4c2011700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 0x7fb4a401abc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:07.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.025 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc07e800 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.028 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb4b4000df0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.155 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fb4bc0e77f0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.572 7fb4aeffd700 1 -- 172.21.2.117:0/3519263920 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v175) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb4b4002ef0 con 0x7fb4bc0bbfd0 2021-02-10T11:56:07.624 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:56:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 msgr2=0x7fb4a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 0x7fb4a401abc0 crc :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 msgr2=0x7fb4bc0bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:07.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fb4b4006c50 tx=0x7fb4b400bd20).stop 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 shutdown_connections 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb4a40186c0 0x7fb4a401abc0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb4bc0bbfd0 0x7fb4bc0bc3d0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb4bc19df40 0x7fb4bc0bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 --2- 172.21.2.117:0/3519263920 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb4bc0b8ac0 0x7fb4bc19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.623 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 >> 172.21.2.117:0/3519263920 conn(0x7fb4bc110940 msgr2=0x7fb4bc07fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.624 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 shutdown_connections 2021-02-10T11:56:07.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:07.624 7fb4c41f5700 1 -- 172.21.2.117:0/3519263920 wait complete. 2021-02-10T11:56:07.635 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:56:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.047 7f4ae64ed700 1 Processor -- start 2021-02-10T11:56:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.047 7f4ae64ed700 1 -- start start 2021-02-10T11:56:08.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4ae64ed700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ae00ef3f0 0x7f4ae00eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4ae64ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae00eb830 0x7f4ae01b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4ae64ed700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae00ef7f0 con 0x7f4ae00d18f0 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4ae64ed700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae00eded0 con 0x7f4ae00ef3f0 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4ae64ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae00d1cf0 con 0x7f4ae00eb830 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ae00ef3f0 0x7f4ae00eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ae00ef3f0 0x7f4ae00eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59380/0 (socket says 172.21.2.117:59380) 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 -- 172.21.2.117:0/3653148561 learned_addr learned my addr 172.21.2.117:0/3653148561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 -- 172.21.2.117:0/3653148561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae00eb830 msgr2=0x7f4ae01b79a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 --2- 172.21.2.117:0/3653148561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae00eb830 0x7f4ae01b79a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 -- 172.21.2.117:0/3653148561 >> v1:172.21.2.125:6789/0 conn(0x7f4ae00d18f0 legacy=0x7f4ae00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 -- 172.21.2.117:0/3653148561 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae01b8fd0 con 0x7f4ae00ef3f0 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.048 7f4adf7fe700 1 --2- 172.21.2.117:0/3653148561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae00ef3f0 0x7f4ae00eb210 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f4ad4006700 tx=0x7f4ad4009b80).ready entity=mon.0 client_cookie=2d543b01348b4315 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:08.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4adeffd700 1 -- 172.21.2.117:0/3653148561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad400c0c0 con 0x7f4ae00ef3f0 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4adeffd700 1 -- 172.21.2.117:0/3653148561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad400c440 con 0x7f4ae00ef3f0 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4adeffd700 1 -- 172.21.2.117:0/3653148561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad400c960 con 0x7f4ae00ef3f0 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 -- 172.21.2.117:0/3653148561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae00ef3f0 msgr2=0x7f4ae00eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 --2- 172.21.2.117:0/3653148561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae00ef3f0 0x7f4ae00eb210 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f4ad4006700 tx=0x7f4ad4009b80).stop 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 -- 172.21.2.117:0/3653148561 shutdown_connections 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 --2- 172.21.2.117:0/3653148561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae00eb830 0x7f4ae01b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 --2- 172.21.2.117:0/3653148561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae00ef3f0 0x7f4ae00eb210 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.049 7f4ae64ed700 1 -- 172.21.2.117:0/3653148561 >> 172.21.2.117:0/3653148561 conn(0x7f4ae00cf060 msgr2=0x7f4ae00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.050 7f4ae64ed700 1 -- 172.21.2.117:0/3653148561 shutdown_connections 2021-02-10T11:56:08.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.050 7f4ae64ed700 1 -- 172.21.2.117:0/3653148561 wait complete. 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 Processor -- start 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 -- start start 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 0x7f4ae01a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 0x7f4ae00b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae009fb90 con 0x7f4ae009ed70 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae00dc640 con 0x7f4ae01a4240 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.064 7f4ae64ed700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae0097270 con 0x7f4ae009f790 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:08.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59386/0 (socket says 172.21.2.117:59386) 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 -- 172.21.2.117:0/941254347 learned_addr learned my addr 172.21.2.117:0/941254347 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 -- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 msgr2=0x7f4ae00b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4ae4b0a700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 0x7f4ae00b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adffff700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 0x7f4ae01a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 0x7f4ae00b8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 -- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 msgr2=0x7f4ae01a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 0x7f4ae01a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 -- 172.21.2.117:0/941254347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae00b8780 con 0x7f4ae009ed70 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4ae4b0a700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 0x7f4ae00b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adffff700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 0x7f4ae01a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adf7fe700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f4ad400b5e0 tx=0x7f4ad400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:08.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad400bd70 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad4002030 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae00b8780 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.065 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4ad40024d0 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.066 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae00b8780 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.066 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4ad4002210 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.066 7f4adcff9700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 0x7f4ac001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.066 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f4ad4032500 con 0x7f4ae009ed70 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.066 7f4adffff700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 0x7f4ac001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:08.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.067 7f4adffff700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 0x7f4ac001abc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:08.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.097 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae00daff0 con 0x7f4ae009ed70 2021-02-10T11:56:08.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.100 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4ad402dee0 con 0x7f4ae009ed70 2021-02-10T11:56:08.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.229 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f4ae00ede60 con 0x7f4ae009ed70 2021-02-10T11:56:08.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.580 7f4adcff9700 1 -- 172.21.2.117:0/941254347 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v177) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4ad402dee0 con 0x7f4ae009ed70 2021-02-10T11:56:08.597 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.595 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 msgr2=0x7f4ac001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.595 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 0x7f4ac001abc0 crc :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 msgr2=0x7f4ae009f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7f4ad400b5e0 tx=0x7f4ad400b980).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 shutdown_connections 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4ac00186c0 0x7f4ac001abc0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ae009ed70 0x7f4ae009f170 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ae009f790 0x7f4ae00b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 --2- 172.21.2.117:0/941254347 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ae01a4240 0x7f4ae01a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 >> 172.21.2.117:0/941254347 conn(0x7f4ae0087090 msgr2=0x7f4ae008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 shutdown_connections 2021-02-10T11:56:08.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:08.596 7f4ae64ed700 1 -- 172.21.2.117:0/941254347 wait complete. 2021-02-10T11:56:08.607 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:56:09.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.020 7fe11d8d2700 1 Processor -- start 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.020 7fe11d8d2700 1 -- start start 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe11d8d2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe1180e4e40 0x7fe1181b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe11d8d2700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1180e5240 0x7fe118084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe11d8d2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe118085930 con 0x7fe1181b0810 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe11d8d2700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1180e5cc0 con 0x7fe1180e5240 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe11d8d2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe118084840 con 0x7fe1180e4e40 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1180e5240 0x7fe118084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe1180e5240 0x7fe118084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59396/0 (socket says 172.21.2.117:59396) 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe117d75700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe1181b0810 0x7fe118085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59646/0 (socket says 172.21.2.117:59646) 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 -- 172.21.2.117:0/980704724 learned_addr learned my addr 172.21.2.117:0/980704724 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe117574700 1 --2- 172.21.2.117:0/980704724 >> v2:172.21.2.125:3300/0 conn(0x7fe1180e4e40 0x7fe1181b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116572700 1 -- 172.21.2.117:0/980704724 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989455898 0 0) 0x7fe110001480 con 0x7fe1181b0810 2021-02-10T11:56:09.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116572700 1 -- 172.21.2.117:0/980704724 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0fc002fe0 con 0x7fe1181b0810 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 -- 172.21.2.117:0/980704724 >> v2:172.21.2.125:3300/0 conn(0x7fe1180e4e40 msgr2=0x7fe1181b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 --2- 172.21.2.117:0/980704724 >> v2:172.21.2.125:3300/0 conn(0x7fe1180e4e40 0x7fe1181b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 -- 172.21.2.117:0/980704724 >> v1:172.21.2.117:6789/0 conn(0x7fe1181b0810 legacy=0x7fe118085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.021 7fe116d73700 1 -- 172.21.2.117:0/980704724 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118085280 con 0x7fe1180e5240 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe116d73700 1 --2- 172.21.2.117:0/980704724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180e5240 0x7fe118084c60 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7fe104006700 tx=0x7fe104009b80).ready entity=mon.0 client_cookie=dbb258c8c4fff6f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe116572700 1 -- 172.21.2.117:0/980704724 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe10400c0c0 con 0x7fe1180e5240 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe116572700 1 -- 172.21.2.117:0/980704724 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10400c3c0 con 0x7fe1180e5240 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe116572700 1 -- 172.21.2.117:0/980704724 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe10400c8e0 con 0x7fe1180e5240 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 -- 172.21.2.117:0/980704724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180e5240 msgr2=0x7fe118084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 --2- 172.21.2.117:0/980704724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180e5240 0x7fe118084c60 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7fe104006700 tx=0x7fe104009b80).stop 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 -- 172.21.2.117:0/980704724 shutdown_connections 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 --2- 172.21.2.117:0/980704724 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180e5240 0x7fe118084c60 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 --2- 172.21.2.117:0/980704724 >> v2:172.21.2.125:3300/0 conn(0x7fe1180e4e40 0x7fe1181b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.022 7fe11d8d2700 1 -- 172.21.2.117:0/980704724 >> 172.21.2.117:0/980704724 conn(0x7fe1180ce4e0 msgr2=0x7fe1180e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:09.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.023 7fe11d8d2700 1 -- 172.21.2.117:0/980704724 shutdown_connections 2021-02-10T11:56:09.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.023 7fe11d8d2700 1 -- 172.21.2.117:0/980704724 wait complete. 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 Processor -- start 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 -- start start 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1180973b0 0x7fe118097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1180ad1b0 0x7fe1180ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11818d0c0 con 0x7fe1180adbd0 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11809ae70 con 0x7fe1180ad1b0 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.037 7fe11d8d2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11809ba10 con 0x7fe1180973b0 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59404/0 (socket says 172.21.2.117:59404) 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 -- 172.21.2.117:0/2578281360 learned_addr learned my addr 172.21.2.117:0/2578281360 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe116d73700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1180ad1b0 0x7fe1180ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117574700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1180973b0 0x7fe118097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 -- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1180973b0 msgr2=0x7fe118097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1180973b0 0x7fe118097980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 -- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1180ad1b0 msgr2=0x7fe1180ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1180ad1b0 0x7fe1180ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 -- 172.21.2.117:0/2578281360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11809bbf0 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.038 7fe117d75700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7fe110009960 tx=0x7fe11000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe110006030 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe110006370 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1100049a0 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11809bbf0 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11809c180 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe110004720 con 0x7fe1180adbd0 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 0x7fe0f401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.039 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fe110032990 con 0x7fe1180adbd0 2021-02-10T11:56:09.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.040 7fe117574700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 0x7fe0f401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.040 7fe117574700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 0x7fe0f401abc0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.071 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118085930 con 0x7fe1180adbd0 2021-02-10T11:56:09.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.074 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe110000f40 con 0x7fe1180adbd0 2021-02-10T11:56:09.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.203 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7fe118085930 con 0x7fe1180adbd0 2021-02-10T11:56:09.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.207 7fe10ffff700 1 -- 172.21.2.117:0/2578281360 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 38 and data pool 39 v150) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe11001f5f0 con 0x7fe1180adbd0 2021-02-10T11:56:09.213 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 38 and data pool 39 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 msgr2=0x7fe0f401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 0x7fe0f401abc0 crc :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 msgr2=0x7fe11809b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7fe110009960 tx=0x7fe11000bcf0).stop 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 shutdown_connections 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0f40186c0 0x7fe0f401abc0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.215 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1180adbd0 0x7fe11809b3f0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1180ad1b0 0x7fe1180ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 --2- 172.21.2.117:0/2578281360 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1180973b0 0x7fe118097980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 >> 172.21.2.117:0/2578281360 conn(0x7fe11818cc70 msgr2=0x7fe118093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:09.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 shutdown_connections 2021-02-10T11:56:09.216 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.212 7fe11d8d2700 1 -- 172.21.2.117:0/2578281360 wait complete. 2021-02-10T11:56:09.224 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.633 7f9a78406700 1 Processor -- start 2021-02-10T11:56:09.653 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.633 7f9a78406700 1 -- start start 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a78406700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a78406700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 0x7f9a701b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a78406700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a700ede60 con 0x7f9a700d18e0 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a78406700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a700ef7f0 con 0x7f9a700eb7f0 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a78406700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700d1ce0 con 0x7f9a700ef3f0 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a75a21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a75a21700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57090/0 (socket says 172.21.2.117:57090) 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a75a21700 1 -- 172.21.2.117:0/2835413647 learned_addr learned my addr 172.21.2.117:0/2835413647 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.634 7f9a76a23700 1 --2- 172.21.2.117:0/2835413647 >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 0x7f9a701b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a75a21700 1 -- 172.21.2.117:0/2835413647 >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 msgr2=0x7f9a701b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a75a21700 1 --2- 172.21.2.117:0/2835413647 >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 0x7f9a701b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.654 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a75a21700 1 -- 172.21.2.117:0/2835413647 >> v1:172.21.2.125:6789/0 conn(0x7f9a700d18e0 legacy=0x7f9a700ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a75a21700 1 -- 172.21.2.117:0/2835413647 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a701b8f90 con 0x7f9a700ef3f0 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a76a23700 1 --2- 172.21.2.117:0/2835413647 >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 0x7f9a701b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.635 7f9a75a21700 1 --2- 172.21.2.117:0/2835413647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9a6c006700 tx=0x7f9a6c009b80).ready entity=mon.2 client_cookie=4c62b6c3981033f6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.636 7f9a75220700 1 -- 172.21.2.117:0/2835413647 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c006340 con 0x7f9a700ef3f0 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.636 7f9a75220700 1 -- 172.21.2.117:0/2835413647 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00c0c0 con 0x7f9a700ef3f0 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.636 7f9a75220700 1 -- 172.21.2.117:0/2835413647 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a6c00c5e0 con 0x7f9a700ef3f0 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.636 7f9a78406700 1 -- 172.21.2.117:0/2835413647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 msgr2=0x7f9a700eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.636 7f9a78406700 1 --2- 172.21.2.117:0/2835413647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9a6c006700 tx=0x7f9a6c009b80).stop 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 -- 172.21.2.117:0/2835413647 shutdown_connections 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 --2- 172.21.2.117:0/2835413647 >> v2:172.21.2.125:3300/0 conn(0x7f9a700eb7f0 0x7f9a701b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 --2- 172.21.2.117:0/2835413647 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700ef3f0 0x7f9a700eb1d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 -- 172.21.2.117:0/2835413647 >> 172.21.2.117:0/2835413647 conn(0x7f9a700ced60 msgr2=0x7f9a700cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 -- 172.21.2.117:0/2835413647 shutdown_connections 2021-02-10T11:56:09.656 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.637 7f9a78406700 1 -- 172.21.2.117:0/2835413647 wait complete. 2021-02-10T11:56:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 Processor -- start 2021-02-10T11:56:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 -- start start 2021-02-10T11:56:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 0x7f9a700b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700b43f0 0x7f9a700e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700e80e0 con 0x7f9a701b0130 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a700e7e20 con 0x7f9a701afd30 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.651 7f9a78406700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a70096a70 con 0x7f9a700b43f0 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59418/0 (socket says 172.21.2.117:59418) 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 -- 172.21.2.117:0/937054811 learned_addr learned my addr 172.21.2.117:0/937054811 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76222700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 0x7f9a700b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a75a21700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700b43f0 0x7f9a700e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 -- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700b43f0 msgr2=0x7f9a700e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700b43f0 0x7f9a700e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 -- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 msgr2=0x7f9a700b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 0x7f9a700b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 -- 172.21.2.117:0/937054811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700e8660 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76222700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 0x7f9a700b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a76a23700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7f9a6800b910 tx=0x7f9a6800bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.652 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a680011f0 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a68006790 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a68001620 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a78406700 1 -- 172.21.2.117:0/937054811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a700e8660 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a78406700 1 -- 172.21.2.117:0/937054811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a700e8bf0 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a68006aa0 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a62ffd700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 0x7f9a5801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.653 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9a68032b00 con 0x7f9a701b0130 2021-02-10T11:56:09.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.654 7f9a76222700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 0x7f9a5801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:09.660 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.654 7f9a76222700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 0x7f9a5801ac10 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:09.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.683 7f9a78406700 1 -- 172.21.2.117:0/937054811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a70087290 con 0x7f9a701b0130 2021-02-10T11:56:09.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.686 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a68000f40 con 0x7f9a701b0130 2021-02-10T11:56:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.811 7f9a78406700 1 -- 172.21.2.117:0/937054811 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a700ede60 con 0x7f9a701b0130 2021-02-10T11:56:09.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.812 7f9a62ffd700 1 -- 172.21.2.117:0/937054811 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f9a680066f0 con 0x7f9a701b0130 2021-02-10T11:56:09.815 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:09.815 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":178,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:09.205339","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:56:07.173989","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:56:08.248299","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:09.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 msgr2=0x7f9a5801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 0x7f9a5801ac10 crc :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 msgr2=0x7f9a700e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7f9a6800b910 tx=0x7f9a6800bcb0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 shutdown_connections 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a58018730 0x7f9a5801ac10 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9a701b0130 0x7f9a700e7800 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9a700b43f0 0x7f9a700e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 --2- 172.21.2.117:0/937054811 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9a701afd30 0x7f9a700b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 >> 172.21.2.117:0/937054811 conn(0x7f9a701a3620 msgr2=0x7f9a700dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 shutdown_connections 2021-02-10T11:56:09.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:09.813 7f9a78406700 1 -- 172.21.2.117:0/937054811 wait complete. 2021-02-10T11:56:09.825 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.234 7f2006e7f700 1 Processor -- start 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.234 7f2006e7f700 1 -- start start 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 0x7f20000ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20000eb000 0x7f20000eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f20000ee3e0 0x7f20000ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20000ede60 con 0x7f20000eb000 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20000eba20 con 0x7f20000ee3e0 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2006e7f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20000d1ce0 con 0x7f20000d18e0 2021-02-10T11:56:10.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20000eb000 0x7f20000eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f20000eb000 0x7f20000eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59424/0 (socket says 172.21.2.117:59424) 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 -- 172.21.2.117:0/1816442359 learned_addr learned my addr 172.21.2.117:0/1816442359 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2004c9b700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 0x7f20000ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f200549c700 1 --2- 172.21.2.117:0/1816442359 >> v2:172.21.2.125:3300/0 conn(0x7f20000ee3e0 0x7f20000ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 -- 172.21.2.117:0/1816442359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 msgr2=0x7f20000ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 0x7f20000ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 -- 172.21.2.117:0/1816442359 >> v2:172.21.2.125:3300/0 conn(0x7f20000ee3e0 msgr2=0x7f20000ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 --2- 172.21.2.117:0/1816442359 >> v2:172.21.2.125:3300/0 conn(0x7f20000ee3e0 0x7f20000ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 -- 172.21.2.117:0/1816442359 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000eee00 con 0x7f20000eb000 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f2004c9b700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 0x7f20000ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.235 7f1ffffff700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20000eb000 0x7f20000eb400 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f1fe8006700 tx=0x7f1fe8009b80).ready entity=mon.0 client_cookie=d92a5f7af51ef9cd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f1fff7fe700 1 -- 172.21.2.117:0/1816442359 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe800c0c0 con 0x7f20000eb000 2021-02-10T11:56:10.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f1fff7fe700 1 -- 172.21.2.117:0/1816442359 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe800c3c0 con 0x7f20000eb000 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f1fff7fe700 1 -- 172.21.2.117:0/1816442359 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1fe800c8a0 con 0x7f20000eb000 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 -- 172.21.2.117:0/1816442359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20000eb000 msgr2=0x7f20000eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20000eb000 0x7f20000eb400 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7f1fe8006700 tx=0x7f1fe8009b80).stop 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 -- 172.21.2.117:0/1816442359 shutdown_connections 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 --2- 172.21.2.117:0/1816442359 >> v2:172.21.2.125:3300/0 conn(0x7f20000ee3e0 0x7f20000ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20000eb000 0x7f20000eb400 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 --2- 172.21.2.117:0/1816442359 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000d18e0 0x7f20000ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 -- 172.21.2.117:0/1816442359 >> 172.21.2.117:0/1816442359 conn(0x7f20000ced60 msgr2=0x7f20000cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 -- 172.21.2.117:0/1816442359 shutdown_connections 2021-02-10T11:56:10.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.236 7f2006e7f700 1 -- 172.21.2.117:0/1816442359 wait complete. 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.251 7f2006e7f700 1 Processor -- start 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.251 7f2006e7f700 1 -- start start 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000ab280 0x7f20000ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20000a6fe0 0x7f20000a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20000a7d10 con 0x7f20001a4290 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20000a7a00 con 0x7f20000a6fe0 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2006e7f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2000096370 con 0x7f20000ab280 2021-02-10T11:56:10.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59430/0 (socket says 172.21.2.117:59430) 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 -- 172.21.2.117:0/3027287458 learned_addr learned my addr 172.21.2.117:0/3027287458 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f1ffffff700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000ab280 0x7f20000ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f200549c700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20000a6fe0 0x7f20000a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 -- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000ab280 msgr2=0x7f20000ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000ab280 0x7f20000ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 -- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20000a6fe0 msgr2=0x7f20000a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20000a6fe0 0x7f20000a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 -- 172.21.2.117:0/3027287458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000a8290 con 0x7f20001a4290 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f2004c9b700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f1ff80099a0 tx=0x7f1ff800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.252 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff800c2f0 con 0x7f20001a4290 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff800c670 con 0x7f20001a4290 2021-02-10T11:56:10.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20000a8290 con 0x7f20001a4290 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1ff800cb90 con 0x7f20001a4290 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20000a8820 con 0x7f20001a4290 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1ff800c2f0 con 0x7f20001a4290 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffd7fa700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 0x7f1fe401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1ff8032a80 con 0x7f20001a4290 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.253 7f1ffffff700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 0x7f1fe401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.254 7f1ffffff700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 0x7f1fe401abc0 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.284 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2000087290 con 0x7f20001a4290 2021-02-10T11:56:10.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.287 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1ff8000cc0 con 0x7f20001a4290 2021-02-10T11:56:10.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.411 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20000ede60 con 0x7f20001a4290 2021-02-10T11:56:10.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.411 7f1ffd7fa700 1 -- 172.21.2.117:0/3027287458 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f1ff800c2f0 con 0x7f20001a4290 2021-02-10T11:56:10.415 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:10.415 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":179,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:10.207917","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:56:07.173989","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:56:08.248299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:10.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 msgr2=0x7f1fe401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 0x7f1fe401abc0 crc :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 msgr2=0x7f20001a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7f1ff80099a0 tx=0x7f1ff800bcf0).stop 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 shutdown_connections 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f20000a6fe0 0x7f20000a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f20000ab280 0x7f20000ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1fe40186c0 0x7f1fe401abc0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 --2- 172.21.2.117:0/3027287458 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f20001a4290 0x7f20001a4690 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.413 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 >> 172.21.2.117:0/3027287458 conn(0x7f20000874c0 msgr2=0x7f2000096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.414 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 shutdown_connections 2021-02-10T11:56:10.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.414 7f2006e7f700 1 -- 172.21.2.117:0/3027287458 wait complete. 2021-02-10T11:56:10.425 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:56:10.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 Processor -- start 2021-02-10T11:56:10.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 -- start start 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881b69a0 0x7f8e881b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880ede60 con 0x7f8e880d18f0 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880eba00 con 0x7f8e880ef3f0 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8f05c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880d1cf0 con 0x7f8e881b69a0 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8ce78700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8e880d18f0 0x7f8e880eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59686/0 (socket says 172.21.2.117:59686) 2021-02-10T11:56:10.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.841 7f8e8ce78700 1 -- 172.21.2.117:0/2584461396 learned_addr learned my addr 172.21.2.117:0/2584461396 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:10.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881b69a0 0x7f8e881b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566717717 0 0) 0x7f8e70000ce0 con 0x7f8e880d18f0 2021-02-10T11:56:10.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e74002fe0 con 0x7f8e880d18f0 2021-02-10T11:56:10.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3421618686 0 0) 0x7f8e700010e0 con 0x7f8e880d18f0 2021-02-10T11:56:10.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881b69a0 msgr2=0x7f8e881b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:10.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881b69a0 0x7f8e881b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 >> v1:172.21.2.125:6789/0 conn(0x7f8e880ef3f0 legacy=0x7f8e880eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e881bb3f0 con 0x7f8e880d18f0 2021-02-10T11:56:10.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8e70001600 con 0x7f8e880d18f0 2021-02-10T11:56:10.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 >> v1:172.21.2.117:6789/0 conn(0x7f8e880d18f0 legacy=0x7f8e880eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e74003a60 0x7f8e74005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e74006750 0x7f8e74008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e740094f0 0x7f8e7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400c2c0 con 0x7f8e740094f0 2021-02-10T11:56:10.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400c4e0 con 0x7f8e74006750 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400c700 con 0x7f8e74003a60 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e740094f0 0x7f8e7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8ce78700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e74003a60 0x7f8e74005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e87fff700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e74006750 0x7f8e74008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 -- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e74003a60 msgr2=0x7f8e74005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e74003a60 0x7f8e74005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 -- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e74006750 msgr2=0x7f8e74008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e74006750 0x7f8e74008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 -- 172.21.2.117:0/2584461396 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74050970 con 0x7f8e740094f0 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e740094f0 0x7f8e7400ba10 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f8e80006af0 tx=0x7f8e80009ad0).ready entity=mon.0 client_cookie=400a652654c4f745 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e800022e0 con 0x7f8e740094f0 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e800066c0 con 0x7f8e740094f0 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e877fe700 1 -- 172.21.2.117:0/2584461396 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e800027d0 con 0x7f8e740094f0 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8f05c700 1 -- 172.21.2.117:0/2584461396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e740094f0 msgr2=0x7f8e7400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8f05c700 1 --2- 172.21.2.117:0/2584461396 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e740094f0 0x7f8e7400ba10 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f8e80006af0 tx=0x7f8e80009ad0).stop 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.842 7f8e8d679700 1 -- 172.21.2.117:0/2584461396 reap_dead start 2021-02-10T11:56:10.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.843 7f8e8f05c700 1 -- 172.21.2.117:0/2584461396 shutdown_connections 2021-02-10T11:56:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.843 7f8e8f05c700 1 -- 172.21.2.117:0/2584461396 >> 172.21.2.117:0/2584461396 conn(0x7f8e880cf060 msgr2=0x7f8e880cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.843 7f8e8f05c700 1 -- 172.21.2.117:0/2584461396 shutdown_connections 2021-02-10T11:56:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.843 7f8e8f05c700 1 -- 172.21.2.117:0/2584461396 wait complete. 2021-02-10T11:56:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.858 7f8e8f05c700 1 Processor -- start 2021-02-10T11:56:10.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.858 7f8e8f05c700 1 -- start start 2021-02-10T11:56:10.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881a4180 0x7f8e880dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e881a4580 0x7f8e880dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88085e50 con 0x7f8e88083990 2021-02-10T11:56:10.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88097270 con 0x7f8e881a4580 2021-02-10T11:56:10.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88097450 con 0x7f8e881a4180 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59448/0 (socket says 172.21.2.117:59448) 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 -- 172.21.2.117:0/674298528 learned_addr learned my addr 172.21.2.117:0/674298528 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 -- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881a4180 msgr2=0x7f8e880dbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881a4180 0x7f8e880dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 -- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e881a4580 msgr2=0x7f8e880dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e881a4580 0x7f8e880dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:10.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 -- 172.21.2.117:0/674298528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8809ed70 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8ce78700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f8e7000a4f0 tx=0x7f8e7000a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e7000ac80 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e70007030 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8e700043a0 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8809ed70 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.859 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8809f2c0 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.860 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8e70007210 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.860 7f8e857fa700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 0x7f8e6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.860 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8e70032690 con 0x7f8e88083990 2021-02-10T11:56:10.873 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.861 7f8e87fff700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 0x7f8e6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:10.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.861 7f8e87fff700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 0x7f8e6c01abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:10.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.891 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8809fb50 con 0x7f8e88083990 2021-02-10T11:56:10.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:10.893 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8e700019b0 con 0x7f8e88083990 2021-02-10T11:56:11.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.029 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f8e8809fb50 con 0x7f8e88083990 2021-02-10T11:56:11.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.216 7f8e857fa700 1 -- 172.21.2.117:0/674298528 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v153) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8e70007210 con 0x7f8e88083990 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 msgr2=0x7f8e6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 0x7f8e6c01abc0 crc :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 msgr2=0x7f8e880863d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f8e7000a4f0 tx=0x7f8e7000a890).stop 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 shutdown_connections 2021-02-10T11:56:11.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8e881a4580 0x7f8e880dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8e881a4180 0x7f8e880dbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8e6c0186c0 0x7f8e6c01abc0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 --2- 172.21.2.117:0/674298528 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8e88083990 0x7f8e880863d0 unknown :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 >> 172.21.2.117:0/674298528 conn(0x7f8e880b6540 msgr2=0x7f8e8808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 shutdown_connections 2021-02-10T11:56:11.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.246 7f8e8f05c700 1 -- 172.21.2.117:0/674298528 wait complete. 2021-02-10T11:56:11.257 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.667 7f4ff28e9700 1 Processor -- start 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.667 7f4ff28e9700 1 -- start start 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff28e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0ef3f0 0x7f4fec0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff28e9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4fec0eb7f0 0x7f4fec1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff28e9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0ede60 con 0x7f4fec0d18e0 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff28e9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0ef7f0 con 0x7f4fec0eb7f0 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff28e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0d1ce0 con 0x7f4fec0ef3f0 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4fec0eb7f0 0x7f4fec1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:11.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4fec0eb7f0 0x7f4fec1b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59458/0 (socket says 172.21.2.117:59458) 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 -- 172.21.2.117:0/1446383097 learned_addr learned my addr 172.21.2.117:0/1446383097 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4febfff700 1 --1- 172.21.2.117:0/1446383097 >> v1:172.21.2.117:6789/0 conn(0x7f4fec0d18e0 0x7f4fec0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59706/0 (socket says 172.21.2.117:59706) 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 -- 172.21.2.117:0/1446383097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0ef3f0 msgr2=0x7f4fec0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 --2- 172.21.2.117:0/1446383097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0ef3f0 0x7f4fec0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 -- 172.21.2.117:0/1446383097 >> v1:172.21.2.117:6789/0 conn(0x7f4fec0d18e0 legacy=0x7f4fec0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.668 7f4ff0f06700 1 -- 172.21.2.117:0/1446383097 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec1b8f90 con 0x7f4fec0eb7f0 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4ff0f06700 1 --2- 172.21.2.117:0/1446383097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0eb7f0 0x7f4fec1b7960 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f4fe4006f00 tx=0x7f4fe4009f10).ready entity=mon.0 client_cookie=87393ee4e1bd71cc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4feaffd700 1 -- 172.21.2.117:0/1446383097 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe400b3b0 con 0x7f4fec0eb7f0 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4feaffd700 1 -- 172.21.2.117:0/1446383097 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe400b630 con 0x7f4fec0eb7f0 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4feaffd700 1 -- 172.21.2.117:0/1446383097 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe400bb50 con 0x7f4fec0eb7f0 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4ff28e9700 1 -- 172.21.2.117:0/1446383097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0eb7f0 msgr2=0x7f4fec1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.669 7f4ff28e9700 1 --2- 172.21.2.117:0/1446383097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0eb7f0 0x7f4fec1b7960 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f4fe4006f00 tx=0x7f4fe4009f10).stop 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 -- 172.21.2.117:0/1446383097 shutdown_connections 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 --2- 172.21.2.117:0/1446383097 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0eb7f0 0x7f4fec1b7960 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 --2- 172.21.2.117:0/1446383097 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0ef3f0 0x7f4fec0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 -- 172.21.2.117:0/1446383097 >> 172.21.2.117:0/1446383097 conn(0x7f4fec0ced60 msgr2=0x7f4fec0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 -- 172.21.2.117:0/1446383097 shutdown_connections 2021-02-10T11:56:11.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.670 7f4ff28e9700 1 -- 172.21.2.117:0/1446383097 wait complete. 2021-02-10T11:56:11.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.684 7f4ff28e9700 1 Processor -- start 2021-02-10T11:56:11.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.684 7f4ff28e9700 1 -- start start 2021-02-10T11:56:11.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 0x7f4fec095aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 0x7f4fec0e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0964c0 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0e7bd0 con 0x7f4fec08a8a0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff28e9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0e7df0 con 0x7f4fec0960c0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59466/0 (socket says 172.21.2.117:59466) 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 -- 172.21.2.117:0/1057642286 learned_addr learned my addr 172.21.2.117:0/1057642286 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 -- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 msgr2=0x7f4fec0e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4feb7fe700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 0x7f4fec095aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4ff0f06700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 0x7f4fec0e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 0x7f4fec0e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.685 7f4febfff700 1 -- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 msgr2=0x7f4fec095aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4febfff700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 0x7f4fec095aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4febfff700 1 -- 172.21.2.117:0/1057642286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0e8010 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4ff0f06700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 0x7f4fec0e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4feb7fe700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 0x7f4fec095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4febfff700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f4fdc00b5c0 tx=0x7f4fdc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fdc006030 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc0043a0 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec0e8010 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec0e8010 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.686 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fdc0048c0 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.687 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4fdc00a000 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.687 7f4fe8ff9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 0x7f4fcc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.687 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4fdc0327c0 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.687 7f4feb7fe700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 0x7f4fcc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:11.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.688 7f4feb7fe700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 0x7f4fcc01ac10 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:11.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.717 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec087210 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.720 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4fdc006b80 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.845 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fec087210 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.845 7f4fe8ff9700 1 -- 172.21.2.117:0/1057642286 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f4fdc00bd50 con 0x7f4fec0ac7b0 2021-02-10T11:56:11.849 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 153 2021-02-10T11:56:11.849 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:11.849 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":153,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:11.215531","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9310":{"gid":9310,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3249802432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3249802432},{"type":"v1","addr":"172.21.2.125:6835","nonce":3249802432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:56:11.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.847 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 msgr2=0x7f4fcc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.847 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 0x7f4fcc01ac10 crc :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.847 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 msgr2=0x7f4fec08a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:11.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.847 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f4fdc00b5c0 tx=0x7f4fdc00b960).stop 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 shutdown_connections 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4fec0960c0 0x7f4fec0e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4fec08a8a0 0x7f4fec095aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fcc018730 0x7f4fcc01ac10 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 --2- 172.21.2.117:0/1057642286 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4fec0ac7b0 0x7f4fec08a280 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 >> 172.21.2.117:0/1057642286 conn(0x7f4fec19f610 msgr2=0x7f4fec0e61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 shutdown_connections 2021-02-10T11:56:11.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:11.848 7f4ff28e9700 1 -- 172.21.2.117:0/1057642286 wait complete. 2021-02-10T11:56:11.860 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:12.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 Processor -- start 2021-02-10T11:56:12.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 -- start start 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 0x7fdd5c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdd5c0eb7f0 0x7fdd5c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0ede60 con 0x7fdd5c0d18e0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ef7f0 con 0x7fdd5c0eb7f0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd641ad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0d1ce0 con 0x7fdd5c0ef3f0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd61fc9700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fdd5c0d18e0 0x7fdd5c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59720/0 (socket says 172.21.2.117:59720) 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.272 7fdd61fc9700 1 -- 172.21.2.117:0/1990222028 learned_addr learned my addr 172.21.2.117:0/1990222028 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 --2- 172.21.2.117:0/1990222028 >> v2:172.21.2.125:3300/0 conn(0x7fdd5c0eb7f0 0x7fdd5c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd617c8700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 0x7fdd5c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd60fc7700 1 -- 172.21.2.117:0/1990222028 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357138137 0 0) 0x7fdd500010b0 con 0x7fdd5c0d18e0 2021-02-10T11:56:12.292 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd60fc7700 1 -- 172.21.2.117:0/1990222028 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd48002fe0 con 0x7fdd5c0d18e0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 -- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 msgr2=0x7fdd5c0eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 0x7fdd5c0eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 -- 172.21.2.117:0/1990222028 >> v1:172.21.2.117:6789/0 conn(0x7fdd5c0d18e0 legacy=0x7fdd5c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 -- 172.21.2.117:0/1990222028 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c1b8f90 con 0x7fdd5c0eb7f0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd617c8700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 0x7fdd5c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd627ca700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0eb7f0 0x7fdd5c1b7960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fdd54006e50 tx=0x7fdd54009e70).ready entity=mon.1 client_cookie=5a023351a4e72ea1 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd60fc7700 1 -- 172.21.2.117:0/1990222028 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd54000f90 con 0x7fdd5c0eb7f0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd60fc7700 1 -- 172.21.2.117:0/1990222028 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd540066f0 con 0x7fdd5c0eb7f0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.273 7fdd60fc7700 1 -- 172.21.2.117:0/1990222028 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd5400b700 con 0x7fdd5c0eb7f0 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 -- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0eb7f0 msgr2=0x7fdd5c1b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0eb7f0 0x7fdd5c1b7960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fdd54006e50 tx=0x7fdd54009e70).stop 2021-02-10T11:56:12.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 -- 172.21.2.117:0/1990222028 shutdown_connections 2021-02-10T11:56:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0eb7f0 0x7fdd5c1b7960 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 --2- 172.21.2.117:0/1990222028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0ef3f0 0x7fdd5c0eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 -- 172.21.2.117:0/1990222028 >> 172.21.2.117:0/1990222028 conn(0x7fdd5c0ced60 msgr2=0x7fdd5c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 -- 172.21.2.117:0/1990222028 shutdown_connections 2021-02-10T11:56:12.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.274 7fdd641ad700 1 -- 172.21.2.117:0/1990222028 wait complete. 2021-02-10T11:56:12.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 Processor -- start 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 -- start start 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 0x7fdd5c0b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0b7310 0x7fdd5c0aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 0x7fdd5c0abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ab8c0 con 0x7fdd5c0ab2c0 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ac460 con 0x7fdd5c0b7310 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.289 7fdd641ad700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ac680 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd627ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 0x7fdd5c0abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 0x7fdd5c0b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd627ca700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 0x7fdd5c0abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59478/0 (socket says 172.21.2.117:59478) 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd627ca700 1 -- 172.21.2.117:0/1350488961 learned_addr learned my addr 172.21.2.117:0/1350488961 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:12.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd617c8700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0b7310 0x7fdd5c0aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 -- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0b7310 msgr2=0x7fdd5c0aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0b7310 0x7fdd5c0aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 -- 172.21.2.117:0/1350488961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 msgr2=0x7fdd5c0abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 0x7fdd5c0abe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 -- 172.21.2.117:0/1350488961 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0ac8a0 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd61fc9700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 0x7fdd5c0b6cf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdd50001c60 tx=0x7fdd5000a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd50000cc0 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500009b0 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdd50004350 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.290 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0ac8a0 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.291 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0ac8a0 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.291 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdd50004140 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.291 7fdd4e7fc700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 0x7fdd4401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.291 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdd50032660 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.291 7fdd617c8700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 0x7fdd4401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.298 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.292 7fdd617c8700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 0x7fdd4401ac10 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.322 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c087290 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.327 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdd50013110 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.453 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd5c087290 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.456 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.454 7fdd4e7fc700 1 -- 172.21.2.117:0/1350488961 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fdd50012f00 con 0x7fdd5c0b77e0 2021-02-10T11:56:12.459 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:12.459 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":179,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:10.207917","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:56:07.173989","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:56:08.248299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:12.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.457 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 msgr2=0x7fdd4401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.457 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 0x7fdd4401ac10 crc :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.457 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 msgr2=0x7fdd5c0b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.457 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 0x7fdd5c0b6cf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdd50001c60 tx=0x7fdd5000a870).stop 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 shutdown_connections 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdd44018730 0x7fdd4401ac10 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdd5c0ab2c0 0x7fdd5c0abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdd5c0b7310 0x7fdd5c0aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 --2- 172.21.2.117:0/1350488961 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdd5c0b77e0 0x7fdd5c0b6cf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 >> 172.21.2.117:0/1350488961 conn(0x7fdd5c0874c0 msgr2=0x7fdd5c0af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 shutdown_connections 2021-02-10T11:56:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.458 7fdd641ad700 1 -- 172.21.2.117:0/1350488961 wait complete. 2021-02-10T11:56:12.470 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 Processor -- start 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 -- start start 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22e4079ce0 0x7f22e41b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f22e407a0e0 0x7f22e41b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e40e5770 con 0x7f22e407ab40 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e407a640 con 0x7f22e4079ce0 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e9ab1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e40d1900 con 0x7f22e407a0e0 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22e4079ce0 0x7f22e41b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f22e4079ce0 0x7f22e41b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59484/0 (socket says 172.21.2.117:59484) 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f22e407ab40 0x7f22e40794f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59734/0 (socket says 172.21.2.117:59734) 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.893 7f22e2ffd700 1 -- 172.21.2.117:0/3346561757 learned_addr learned my addr 172.21.2.117:0/3346561757 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e2ffd700 1 -- 172.21.2.117:0/3346561757 >> v2:172.21.2.125:3300/0 conn(0x7f22e407a0e0 msgr2=0x7f22e41b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e2ffd700 1 --2- 172.21.2.117:0/3346561757 >> v2:172.21.2.125:3300/0 conn(0x7f22e407a0e0 0x7f22e41b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e2ffd700 1 -- 172.21.2.117:0/3346561757 >> v1:172.21.2.117:6789/0 conn(0x7f22e407ab40 legacy=0x7f22e40794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e2ffd700 1 -- 172.21.2.117:0/3346561757 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e407bf50 con 0x7f22e4079ce0 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e2ffd700 1 --2- 172.21.2.117:0/3346561757 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4079ce0 0x7f22e41b59d0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f22d4006700 tx=0x7f22d4009b80).ready entity=mon.0 client_cookie=f69cd7ac8bb76c76 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e27fc700 1 -- 172.21.2.117:0/3346561757 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d400c0c0 con 0x7f22e4079ce0 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e27fc700 1 -- 172.21.2.117:0/3346561757 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d400c340 con 0x7f22e4079ce0 2021-02-10T11:56:12.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e27fc700 1 -- 172.21.2.117:0/3346561757 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22d400c820 con 0x7f22e4079ce0 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e9ab1700 1 -- 172.21.2.117:0/3346561757 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4079ce0 msgr2=0x7f22e41b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.894 7f22e9ab1700 1 --2- 172.21.2.117:0/3346561757 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4079ce0 0x7f22e41b59d0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f22d4006700 tx=0x7f22d4009b80).stop 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 -- 172.21.2.117:0/3346561757 shutdown_connections 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 --2- 172.21.2.117:0/3346561757 >> v2:172.21.2.125:3300/0 conn(0x7f22e407a0e0 0x7f22e41b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 --2- 172.21.2.117:0/3346561757 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4079ce0 0x7f22e41b59d0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 -- 172.21.2.117:0/3346561757 >> 172.21.2.117:0/3346561757 conn(0x7f22e40d0910 msgr2=0x7f22e40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 -- 172.21.2.117:0/3346561757 shutdown_connections 2021-02-10T11:56:12.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.895 7f22e9ab1700 1 -- 172.21.2.117:0/3346561757 wait complete. 2021-02-10T11:56:12.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.909 7f22e9ab1700 1 Processor -- start 2021-02-10T11:56:12.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.909 7f22e9ab1700 1 -- start start 2021-02-10T11:56:12.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22e4191440 0x7f22e4191840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22e4191e60 0x7f22e41b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e41b0d10 con 0x7f22e4192260 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e40989f0 con 0x7f22e4191440 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e9ab1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e4098bd0 con 0x7f22e4191e60 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59492/0 (socket says 172.21.2.117:59492) 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 -- 172.21.2.117:0/1237589741 learned_addr learned my addr 172.21.2.117:0/1237589741 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e37fe700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22e4191440 0x7f22e4191840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e2ffd700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22e4191e60 0x7f22e41b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 -- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22e4191e60 msgr2=0x7f22e41b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22e4191e60 0x7f22e41b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 -- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22e4191440 msgr2=0x7f22e4191840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.910 7f22e3fff700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22e4191440 0x7f22e4191840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22e3fff700 1 -- 172.21.2.117:0/1237589741 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4098e90 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22e3fff700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7f22dc009980 tx=0x7f22dc00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22dc005870 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22dc005b30 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e4098e90 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f22dc006030 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.911 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e41b21c0 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.912 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f22dc005d10 con 0x7f22e4192260 2021-02-10T11:56:12.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.912 7f22dbfff700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 0x7f22c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:12.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.912 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f22dc032930 con 0x7f22e4192260 2021-02-10T11:56:12.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.912 7f22e37fe700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 0x7f22c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:12.919 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.913 7f22e37fe700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 0x7f22c801abc0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:12.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.943 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e40da9e0 con 0x7f22e4192260 2021-02-10T11:56:12.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:12.945 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f22dc0062f0 con 0x7f22e4192260 2021-02-10T11:56:13.077 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.074 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f22e40e5770 con 0x7f22e4192260 2021-02-10T11:56:13.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.079 7f22dbfff700 1 -- 172.21.2.117:0/1237589741 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v100) v1 ==== 184+0+0 (secure 0 0 0) 0x7f22dc005c10 con 0x7f22e4192260 2021-02-10T11:56:13.084 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:56:13.086 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 msgr2=0x7f22c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 0x7f22c801abc0 crc :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 msgr2=0x7f22e41b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7f22dc009980 tx=0x7f22dc00bcd0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 shutdown_connections 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f22c80186c0 0x7f22c801abc0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f22e4192260 0x7f22e41b18b0 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f22e4191e60 0x7f22e41b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 --2- 172.21.2.117:0/1237589741 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f22e4191440 0x7f22e4191840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.083 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 >> 172.21.2.117:0/1237589741 conn(0x7f22e41b31d0 msgr2=0x7f22e41b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.084 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 shutdown_connections 2021-02-10T11:56:13.087 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.084 7f22e9ab1700 1 -- 172.21.2.117:0/1237589741 wait complete. 2021-02-10T11:56:13.095 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:56:13.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 Processor -- start 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 -- start start 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f150807ab40 0x7f15080794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080e5770 con 0x7f1508079f10 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150807a310 con 0x7f1508079b10 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.507 7f150ef0e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080d18c0 con 0x7f150807ab40 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f150807ab40 0x7f15080794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f150807ab40 0x7f15080794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59500/0 (socket says 172.21.2.117:59500) 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150d52b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f1508079f10 0x7f150807b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59752/0 (socket says 172.21.2.117:59752) 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 -- 172.21.2.117:0/613067381 learned_addr learned my addr 172.21.2.117:0/613067381 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:13.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f1507fff700 1 --1- 172.21.2.117:0/613067381 >> v1:172.21.2.125:6789/0 conn(0x7f1508079b10 0x7f150807af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52674/0 (socket says 172.21.2.117:52674) 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f15077fe700 1 -- 172.21.2.117:0/613067381 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139002332 0 0) 0x7f14f8001400 con 0x7f1508079b10 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f15077fe700 1 -- 172.21.2.117:0/613067381 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f4002fe0 con 0x7f1508079b10 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 -- 172.21.2.117:0/613067381 >> v1:172.21.2.125:6789/0 conn(0x7f1508079b10 legacy=0x7f150807af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 -- 172.21.2.117:0/613067381 >> v1:172.21.2.117:6789/0 conn(0x7f1508079f10 legacy=0x7f150807b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 -- 172.21.2.117:0/613067381 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150807bf20 con 0x7f150807ab40 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.508 7f150cd2a700 1 --2- 172.21.2.117:0/613067381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f150807ab40 0x7f15080794f0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f14f0006700 tx=0x7f14f0009b80).ready entity=mon.0 client_cookie=99267a9c75dfef60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f15077fe700 1 -- 172.21.2.117:0/613067381 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14f000c0c0 con 0x7f150807ab40 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f15077fe700 1 -- 172.21.2.117:0/613067381 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f000c440 con 0x7f150807ab40 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f15077fe700 1 -- 172.21.2.117:0/613067381 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14f000c920 con 0x7f150807ab40 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 -- 172.21.2.117:0/613067381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f150807ab40 msgr2=0x7f15080794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 --2- 172.21.2.117:0/613067381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f150807ab40 0x7f15080794f0 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f14f0006700 tx=0x7f14f0009b80).stop 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 -- 172.21.2.117:0/613067381 shutdown_connections 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 --2- 172.21.2.117:0/613067381 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f150807ab40 0x7f15080794f0 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 -- 172.21.2.117:0/613067381 >> 172.21.2.117:0/613067381 conn(0x7f15080d0910 msgr2=0x7f15080d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 -- 172.21.2.117:0/613067381 shutdown_connections 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.509 7f150ef0e700 1 -- 172.21.2.117:0/613067381 wait complete. 2021-02-10T11:56:13.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 Processor -- start 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 -- start start 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15080dca60 0x7f15080adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15080977d0 0x7f1508193210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1508193830 con 0x7f15080ae540 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1508193a50 con 0x7f15080dca60 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f150ef0e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1508193c70 con 0x7f15080977d0 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.524 7f1507fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:13.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59504/0 (socket says 172.21.2.117:59504) 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 -- 172.21.2.117:0/937504498 learned_addr learned my addr 172.21.2.117:0/937504498 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 -- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15080977d0 msgr2=0x7f1508193210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15080977d0 0x7f1508193210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 -- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15080dca60 msgr2=0x7f15080adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15080dca60 0x7f15080adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 -- 172.21.2.117:0/937504498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15081945b0 con 0x7f15080ae540 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f1507fff700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f14f800a8f0 tx=0x7f14f800ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14f80048b0 con 0x7f15080ae540 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8004bb0 con 0x7f15080ae540 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14f80070d0 con 0x7f15080ae540 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f150ef0e700 1 -- 172.21.2.117:0/937504498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15081945b0 con 0x7f15080ae540 2021-02-10T11:56:13.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.525 7f150ef0e700 1 -- 172.21.2.117:0/937504498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15081945b0 con 0x7f15080ae540 2021-02-10T11:56:13.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.526 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14f8000f90 con 0x7f15080ae540 2021-02-10T11:56:13.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.526 7f15057fa700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 0x7f14ec01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:13.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.526 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f14f8032ad0 con 0x7f15080ae540 2021-02-10T11:56:13.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.526 7f150cd2a700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 0x7f14ec01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:13.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.527 7f150cd2a700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 0x7f14ec01abc0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:13.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.557 7f150ef0e700 1 -- 172.21.2.117:0/937504498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1508194080 con 0x7f15080ae540 2021-02-10T11:56:13.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.560 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14f8004b70 con 0x7f15080ae540 2021-02-10T11:56:13.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.688 7f150ef0e700 1 -- 172.21.2.117:0/937504498 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1508194080 con 0x7f15080ae540 2021-02-10T11:56:13.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.692 7f15057fa700 1 -- 172.21.2.117:0/937504498 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v101) v1 ==== 184+0+0 (secure 0 0 0) 0x7f14f8000f90 con 0x7f15080ae540 2021-02-10T11:56:13.698 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:56:13.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 -- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 msgr2=0x7f14ec01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 0x7f14ec01abc0 crc :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 -- 172.21.2.117:0/937504498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 msgr2=0x7f1508192bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:13.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f14f800a8f0 tx=0x7f14f800ac90).stop 2021-02-10T11:56:13.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 -- 172.21.2.117:0/937504498 shutdown_connections 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14ec018700 0x7f14ec01abc0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f15080ae540 0x7f1508192bf0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f15080977d0 0x7f1508193210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 --2- 172.21.2.117:0/937504498 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f15080dca60 0x7f15080adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.697 7f150ef0e700 1 -- 172.21.2.117:0/937504498 >> 172.21.2.117:0/937504498 conn(0x7f15081b92d0 msgr2=0x7f1508097cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.698 7f150ef0e700 1 -- 172.21.2.117:0/937504498 shutdown_connections 2021-02-10T11:56:13.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:13.698 7f150ef0e700 1 -- 172.21.2.117:0/937504498 wait complete. 2021-02-10T11:56:13.708 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.130 7fd9f8c9b700 1 Processor -- start 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.130 7fd9f8c9b700 1 -- start start 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.130 7fd9f8c9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f4079ce0 0x7fd9f41b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f8c9b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40e5770 con 0x7fd9f407a0e0 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f8c9b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f40d18c0 con 0x7fd9f407ab40 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f8c9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f407bf50 con 0x7fd9f4079ce0 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f305f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd9f407a0e0 0x7fd9f41b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59766/0 (socket says 172.21.2.117:59766) 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f305f700 1 -- 172.21.2.117:0/1892800782 learned_addr learned my addr 172.21.2.117:0/1892800782 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540308740 0 0) 0x7fd9ec001500 con 0x7fd9f407a0e0 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9dc002fe0 con 0x7fd9f407a0e0 2021-02-10T11:56:14.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.131 7fd9f205d700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f4079ce0 0x7fd9f41b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2157864336 0 0) 0x7fd9ec001460 con 0x7fd9f407a0e0 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f4079ce0 msgr2=0x7fd9f41b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f4079ce0 0x7fd9f41b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 >> v1:172.21.2.125:6789/0 conn(0x7fd9f407ab40 legacy=0x7fd9f40794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40d1c60 con 0x7fd9f407a0e0 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd9ec001980 con 0x7fd9f407a0e0 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 >> v1:172.21.2.117:6789/0 conn(0x7fd9f407a0e0 legacy=0x7fd9f41b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9dc003af0 0x7fd9dc005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9dc006750 0x7fd9dc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9dc0094f0 0x7fd9dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c2c0 con 0x7fd9dc0094f0 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c4e0 con 0x7fd9dc006750 2021-02-10T11:56:14.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc00c700 con 0x7fd9dc003af0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9dc0094f0 0x7fd9dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f205d700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9dc006750 0x7fd9dc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f285e700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9dc003af0 0x7fd9dc005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 -- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9dc003af0 msgr2=0x7fd9dc005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9dc003af0 0x7fd9dc005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 -- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9dc006750 msgr2=0x7fd9dc008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9dc006750 0x7fd9dc008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 -- 172.21.2.117:0/1892800782 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc050970 con 0x7fd9dc0094f0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f305f700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9dc0094f0 0x7fd9dc00ba10 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fd9ec001160 tx=0x7fd9ec00a310).ready entity=mon.0 client_cookie=7c8043ca1fc5fffd server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9ec0070c0 con 0x7fd9dc0094f0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec007340 con 0x7fd9dc0094f0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.132 7fd9f185c700 1 -- 172.21.2.117:0/1892800782 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9ec00ab90 con 0x7fd9dc0094f0 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 -- 172.21.2.117:0/1892800782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9dc0094f0 msgr2=0x7fd9dc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.153 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 --2- 172.21.2.117:0/1892800782 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9dc0094f0 0x7fd9dc00ba10 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7fd9ec001160 tx=0x7fd9ec00a310).stop 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f305f700 1 -- 172.21.2.117:0/1892800782 reap_dead start 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 -- 172.21.2.117:0/1892800782 shutdown_connections 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 -- 172.21.2.117:0/1892800782 >> 172.21.2.117:0/1892800782 conn(0x7fd9f40d0910 msgr2=0x7fd9f40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 -- 172.21.2.117:0/1892800782 shutdown_connections 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.133 7fd9f8c9b700 1 -- 172.21.2.117:0/1892800782 wait complete. 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 Processor -- start 2021-02-10T11:56:14.154 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 -- start start 2021-02-10T11:56:14.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 0x7fd9f40a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f41a78c0 0x7fd9f41a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 0x7fd9f408a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f408b060 con 0x7fd9f40dc680 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f408acd0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.148 7fd9f8c9b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4094010 con 0x7fd9f41a78c0 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 0x7fd9f40a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f285e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 0x7fd9f40a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59524/0 (socket says 172.21.2.117:59524) 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f285e700 1 -- 172.21.2.117:0/3102863897 learned_addr learned my addr 172.21.2.117:0/3102863897 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 0x7fd9f408a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f205d700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f41a78c0 0x7fd9f41a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 -- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f41a78c0 msgr2=0x7fd9f41a7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f41a78c0 0x7fd9f41a7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 -- 172.21.2.117:0/3102863897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 msgr2=0x7fd9f40a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 0x7fd9f40a26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 -- 172.21.2.117:0/3102863897 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f408b560 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f305f700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 0x7fd9f408a6b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7fd9ec001520 tx=0x7fd9ec00abb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9ec000eb0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec0070d0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd9ec004720 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f408b560 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.149 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f408b560 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.150 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd9ec0072b0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.150 7fd9e37fe700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 0x7fd9d801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.150 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd9ec032ba0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.150 7fd9f285e700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 0x7fd9d801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.151 7fd9f285e700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 0x7fd9d801abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.180 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f408acd0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.183 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd9ec001ec0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.313 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd9f408acd0 con 0x7fd9f408a2b0 2021-02-10T11:56:14.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.318 7fd9e37fe700 1 -- 172.21.2.117:0/3102863897 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v102) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd9ec007290 con 0x7fd9f408a2b0 2021-02-10T11:56:14.323 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 msgr2=0x7fd9d801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 0x7fd9d801abc0 crc :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 msgr2=0x7fd9f408a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 0x7fd9f408a6b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7fd9ec001520 tx=0x7fd9ec00abb0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 shutdown_connections 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd9f408a2b0 0x7fd9f408a6b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd9f41a78c0 0x7fd9f41a7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd9d80186c0 0x7fd9d801abc0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 --2- 172.21.2.117:0/3102863897 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd9f40dc680 0x7fd9f40a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 >> 172.21.2.117:0/3102863897 conn(0x7fd9f40dccc0 msgr2=0x7fd9f40dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 shutdown_connections 2021-02-10T11:56:14.325 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.322 7fd9f8c9b700 1 -- 172.21.2.117:0/3102863897 wait complete. 2021-02-10T11:56:14.333 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:56:14.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 Processor -- start 2021-02-10T11:56:14.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 -- start start 2021-02-10T11:56:14.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2f50079ce0 0x7f2f501b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5007a5c0 con 0x7f2f5007a0e0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f500d18c0 con 0x7f2f5007ab40 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f5825b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5007bf50 con 0x7f2f50079ce0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f56878700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2f5007a0e0 0x7f2f501b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59786/0 (socket says 172.21.2.117:59786) 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f56878700 1 -- 172.21.2.117:0/4070965407 learned_addr learned my addr 172.21.2.117:0/4070965407 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.757 7f2f55876700 1 --2- 172.21.2.117:0/4070965407 >> v2:172.21.2.117:3300/0 conn(0x7f2f50079ce0 0x7f2f501b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55075700 1 -- 172.21.2.117:0/4070965407 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895231331 0 0) 0x7f2f48000f90 con 0x7f2f5007a0e0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55876700 1 -- 172.21.2.117:0/4070965407 >> v1:172.21.2.125:6789/0 conn(0x7f2f5007ab40 legacy=0x7f2f500794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55876700 1 -- 172.21.2.117:0/4070965407 >> v1:172.21.2.117:6789/0 conn(0x7f2f5007a0e0 legacy=0x7f2f501b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55876700 1 -- 172.21.2.117:0/4070965407 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500d1c60 con 0x7f2f50079ce0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55876700 1 --2- 172.21.2.117:0/4070965407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50079ce0 0x7f2f501b59d0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7f2f40006650 tx=0x7f2f40009a80).ready entity=mon.0 client_cookie=8eea85b55238e51b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55075700 1 -- 172.21.2.117:0/4070965407 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4000c0c0 con 0x7f2f50079ce0 2021-02-10T11:56:14.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55075700 1 -- 172.21.2.117:0/4070965407 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4000c340 con 0x7f2f50079ce0 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f55075700 1 -- 172.21.2.117:0/4070965407 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4000c860 con 0x7f2f50079ce0 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f5825b700 1 -- 172.21.2.117:0/4070965407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50079ce0 msgr2=0x7f2f501b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.758 7f2f5825b700 1 --2- 172.21.2.117:0/4070965407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50079ce0 0x7f2f501b59d0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7f2f40006650 tx=0x7f2f40009a80).stop 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.759 7f2f5825b700 1 -- 172.21.2.117:0/4070965407 shutdown_connections 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.759 7f2f5825b700 1 --2- 172.21.2.117:0/4070965407 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50079ce0 0x7f2f501b59d0 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.759 7f2f5825b700 1 -- 172.21.2.117:0/4070965407 >> 172.21.2.117:0/4070965407 conn(0x7f2f500d0910 msgr2=0x7f2f500d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.759 7f2f5825b700 1 -- 172.21.2.117:0/4070965407 shutdown_connections 2021-02-10T11:56:14.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.759 7f2f5825b700 1 -- 172.21.2.117:0/4070965407 wait complete. 2021-02-10T11:56:14.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 Processor -- start 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 -- start start 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f50096650 0x7f2f5008c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5008cbc0 0x7f2f5008a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5008ba80 con 0x7f2f50096250 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5008d000 con 0x7f2f50096650 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f5825b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5008d220 con 0x7f2f5008cbc0 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f56077700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f56077700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59540/0 (socket says 172.21.2.117:59540) 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.774 7f2f56077700 1 -- 172.21.2.117:0/1050323666 learned_addr learned my addr 172.21.2.117:0/1050323666 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 -- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5008cbc0 msgr2=0x7f2f5008a2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5008cbc0 0x7f2f5008a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:14.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 -- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f50096650 msgr2=0x7f2f5008c5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f50096650 0x7f2f5008c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 -- 172.21.2.117:0/1050323666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5008a8d0 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f56077700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f2f4c00b530 tx=0x7f2f4c00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4c00bcc0 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c006020 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2f4c0054a0 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5008a8d0 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.775 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5008ae60 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.776 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2f4c006200 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.776 7f2f46ffd700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 0x7f2f3801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.776 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2f4c0326c0 con 0x7f2f50096250 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.776 7f2f55876700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 0x7f2f3801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:14.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.777 7f2f55876700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 0x7f2f3801abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:14.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.807 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f500da950 con 0x7f2f50096250 2021-02-10T11:56:14.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.809 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2f4c000c10 con 0x7f2f50096250 2021-02-10T11:56:14.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.937 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2f500f6820 con 0x7f2f50096250 2021-02-10T11:56:14.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.941 7f2f46ffd700 1 -- 172.21.2.117:0/1050323666 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v103) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2f4c006240 con 0x7f2f50096250 2021-02-10T11:56:14.947 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:56:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 msgr2=0x7f2f3801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 0x7f2f3801abc0 crc :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 msgr2=0x7f2f5008bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f2f4c00b530 tx=0x7f2f4c00b8d0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 shutdown_connections 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2f5008cbc0 0x7f2f5008a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2f50096650 0x7f2f5008c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f380186c0 0x7f2f3801abc0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 --2- 172.21.2.117:0/1050323666 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2f50096250 0x7f2f5008bf80 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 >> 172.21.2.117:0/1050323666 conn(0x7f2f50192380 msgr2=0x7f2f5019eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 shutdown_connections 2021-02-10T11:56:14.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:14.946 7f2f5825b700 1 -- 172.21.2.117:0/1050323666 wait complete. 2021-02-10T11:56:14.958 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:15.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 Processor -- start 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 -- start start 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f94140ef3f0 0x7f94140eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140ede60 con 0x7f94140eb7f0 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140d1ce0 con 0x7f94140d18e0 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.368 7f9418a15700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94140ef7f0 con 0x7f94140ef3f0 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f9412e1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f94140eb7f0 0x7f94141b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59800/0 (socket says 172.21.2.117:59800) 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f9412e1d700 1 -- 172.21.2.117:0/3999394336 learned_addr learned my addr 172.21.2.117:0/3999394336 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:15.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433946326 0 0) 0x7f940c001500 con 0x7f94140eb7f0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9400002fe0 con 0x7f94140eb7f0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3687074422 0 0) 0x7f940c001460 con 0x7f94140eb7f0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 >> v2:172.21.2.125:3300/0 conn(0x7f94140ef3f0 msgr2=0x7f94140eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> v2:172.21.2.125:3300/0 conn(0x7f94140ef3f0 0x7f94140eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 --2- 172.21.2.117:0/3999394336 >> v2:172.21.2.125:3300/0 conn(0x7f94140ef3f0 0x7f94140eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 >> v1:172.21.2.125:6789/0 conn(0x7f94140d18e0 legacy=0x7f94140ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94141bb180 con 0x7f94140eb7f0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f940c001980 con 0x7f94140eb7f0 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> v2:172.21.2.125:3300/0 conn(0x7f94140ef3f0 0x7f94140eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 >> v1:172.21.2.117:6789/0 conn(0x7f94140eb7f0 legacy=0x7f94141b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9400003aa0 0x7f9400005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 0x7f9400008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9400009590 0x7f940000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940000c300 con 0x7f9400006790 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940000c520 con 0x7f9400003aa0 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f941161a700 1 -- 172.21.2.117:0/3999394336 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940000c740 con 0x7f9400009590 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.369 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 0x7f9400008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9412e1d700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9400009590 0x7f940000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f941261c700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9400003aa0 0x7f9400005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 -- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9400009590 msgr2=0x7f940000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9400009590 0x7f940000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 -- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9400003aa0 msgr2=0x7f9400005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9400003aa0 0x7f9400005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 -- 172.21.2.117:0/3999394336 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940000c9e0 con 0x7f9400006790 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9412e1d700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9400009590 0x7f940000bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:15.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9412e1d700 1 -- 172.21.2.117:0/3999394336 reap_dead start 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9411e1b700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 0x7f9400008c90 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f93fc04d980 tx=0x7f93fc04dd20).ready entity=mon.0 client_cookie=50f508d04a589769 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc006960 con 0x7f9400006790 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0009b0 con 0x7f9400006790 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f941161a700 1 -- 172.21.2.117:0/3999394336 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f93fc002030 con 0x7f9400006790 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9418a15700 1 -- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 msgr2=0x7f9400008c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.370 7f9418a15700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 0x7f9400008c90 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7f93fc04d980 tx=0x7f93fc04dd20).stop 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.371 7f9418a15700 1 -- 172.21.2.117:0/3999394336 shutdown_connections 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.371 7f9418a15700 1 --2- 172.21.2.117:0/3999394336 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9400006790 0x7f9400008c90 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.371 7f9418a15700 1 -- 172.21.2.117:0/3999394336 >> 172.21.2.117:0/3999394336 conn(0x7f94140ced60 msgr2=0x7f94140cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.371 7f9418a15700 1 -- 172.21.2.117:0/3999394336 shutdown_connections 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.371 7f9418a15700 1 -- 172.21.2.117:0/3999394336 wait complete. 2021-02-10T11:56:15.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.386 7f9418a15700 1 Processor -- start 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.386 7f9418a15700 1 -- start start 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 0x7f94140e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 0x7f941408bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941408b660 con 0x7f941408c600 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941408ca00 con 0x7f941408bbe0 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9418a15700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140e9a80 con 0x7f94140b2160 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59562/0 (socket says 172.21.2.117:59562) 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 -- 172.21.2.117:0/3803208823 learned_addr learned my addr 172.21.2.117:0/3803208823 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9411e1b700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 0x7f941408bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f941261c700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 0x7f94140e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 -- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 msgr2=0x7f94140e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 0x7f94140e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 -- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 msgr2=0x7f941408bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 0x7f941408bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 -- 172.21.2.117:0/3803208823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941408a8a0 con 0x7f941408c600 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9411e1b700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 0x7f941408bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f941261c700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 0x7f94140e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f9412e1d700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f940c0018c0 tx=0x7f940c00ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.387 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f940c001900 con 0x7f941408c600 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c004930 con 0x7f941408c600 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f940c007020 con 0x7f941408c600 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f9418a15700 1 -- 172.21.2.117:0/3803208823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941408a8a0 con 0x7f941408c600 2021-02-10T11:56:15.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f9418a15700 1 -- 172.21.2.117:0/3803208823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941408ae30 con 0x7f941408c600 2021-02-10T11:56:15.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f940c001900 con 0x7f941408c600 2021-02-10T11:56:15.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f940affd700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 0x7f93f801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:15.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.388 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f940c032890 con 0x7f941408c600 2021-02-10T11:56:15.397 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.389 7f941261c700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 0x7f93f801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:15.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.389 7f941261c700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 0x7f93f801abc0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:15.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.418 7f9418a15700 1 -- 172.21.2.117:0/3803208823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941408b410 con 0x7f941408c600 2021-02-10T11:56:15.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.421 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f940c000d30 con 0x7f941408c600 2021-02-10T11:56:15.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.547 7f9418a15700 1 -- 172.21.2.117:0/3803208823 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941408b410 con 0x7f941408c600 2021-02-10T11:56:15.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.547 7f940affd700 1 -- 172.21.2.117:0/3803208823 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f940c001900 con 0x7f941408c600 2021-02-10T11:56:15.551 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:15.551 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":153,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:11.215531","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9310":{"gid":9310,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3249802432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3249802432},{"type":"v1","addr":"172.21.2.125:6835","nonce":3249802432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:56:15.551 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 153 2021-02-10T11:56:15.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 -- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 msgr2=0x7f93f801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 0x7f93f801abc0 crc :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 -- 172.21.2.117:0/3803208823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 msgr2=0x7f941408a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f940c0018c0 tx=0x7f940c00ac80).stop 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 -- 172.21.2.117:0/3803208823 shutdown_connections 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f93f80186c0 0x7f93f801abc0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f941408c600 0x7f941408a280 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f941408bbe0 0x7f941408bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 --2- 172.21.2.117:0/3803208823 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f94140b2160 0x7f94140e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.549 7f9418a15700 1 -- 172.21.2.117:0/3803208823 >> 172.21.2.117:0/3803208823 conn(0x7f94140bb5e0 msgr2=0x7f9414090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.550 7f9418a15700 1 -- 172.21.2.117:0/3803208823 shutdown_connections 2021-02-10T11:56:15.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:15.550 7f9418a15700 1 -- 172.21.2.117:0/3803208823 wait complete. 2021-02-10T11:56:15.762 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:56:09.205433', 'modified': '2021-02-10 11:56:11.215531', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9310}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9310': {'gid': 9310, 'name': 'b', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3249802432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3249802432}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3249802432}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:56:15.763 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:56:15.763 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:56:16.162 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:56:16.162 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:56:16.162 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:56:16.162 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9310, 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9523465819999997, 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 153, 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 179, 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 179, 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.592952863000001 2021-02-10T11:56:16.163 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:56:16.166 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9310, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9523465819999997, "mdsmap_epoch": 153, "osdmap_epoch": 179, "osdmap_epoch_barrier": 179, "uptime": 12.592952863000001 } 2021-02-10T11:56:16.167 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.578 7f6aa6314700 1 Processor -- start 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.578 7f6aa6314700 1 -- start start 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa6314700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6aa00ef3f0 0x7f6aa00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa6314700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00ede60 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa6314700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00d1ce0 con 0x7f6aa00d18e0 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa6314700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa00ef7f0 con 0x7f6aa00ef3f0 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa4931700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6aa00eb7f0 0x7f6aa01b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59820/0 (socket says 172.21.2.117:59820) 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6aa4931700 1 -- 172.21.2.117:0/513997649 learned_addr learned my addr 172.21.2.117:0/513997649 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:16.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235474614 0 0) 0x7f6a98001500 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.579 7f6a9effd700 1 -- 172.21.2.117:0/513997649 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a88002fe0 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9f7fe700 1 --2- 172.21.2.117:0/513997649 >> v2:172.21.2.125:3300/0 conn(0x7f6aa00ef3f0 0x7f6aa00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1216946549 0 0) 0x7f6a98001460 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 >> v2:172.21.2.125:3300/0 conn(0x7f6aa00ef3f0 msgr2=0x7f6aa00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 --2- 172.21.2.117:0/513997649 >> v2:172.21.2.125:3300/0 conn(0x7f6aa00ef3f0 0x7f6aa00eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 >> v1:172.21.2.125:6789/0 conn(0x7f6aa00d18e0 legacy=0x7f6aa00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa01bb180 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6a98001980 con 0x7f6aa00eb7f0 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 >> v1:172.21.2.117:6789/0 conn(0x7f6aa00eb7f0 legacy=0x7f6aa01b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 0x7f6a88005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a88006770 0x7f6a88008c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a88009590 0x7f6a8800bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8800c300 con 0x7f6a88003a60 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8800c520 con 0x7f6a88006770 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9effd700 1 -- 172.21.2.117:0/513997649 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8800c740 con 0x7f6a88009590 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 0x7f6a88005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9f7fe700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a88006770 0x7f6a88008c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6aa4931700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a88009590 0x7f6a8800bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 -- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a88009590 msgr2=0x7f6a8800bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6a88009590 0x7f6a8800bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 -- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a88006770 msgr2=0x7f6a88008c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6a88006770 0x7f6a88008c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6a9ffff700 1 -- 172.21.2.117:0/513997649 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8800c9e0 con 0x7f6a88003a60 2021-02-10T11:56:16.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.580 7f6aa4931700 1 -- 172.21.2.117:0/513997649 reap_dead start 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6a9ffff700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 0x7f6a88005ed0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f6a9004aac0 tx=0x7f6a9004daf0).ready entity=mon.0 client_cookie=4eae6c6e7fe93abf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a9004a6b0 con 0x7f6a88003a60 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a900022e0 con 0x7f6a88003a60 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6a9effd700 1 -- 172.21.2.117:0/513997649 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a90002800 con 0x7f6a88003a60 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 -- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 msgr2=0x7f6a88005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 0x7f6a88005ed0 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f6a9004aac0 tx=0x7f6a9004daf0).stop 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 -- 172.21.2.117:0/513997649 shutdown_connections 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 --2- 172.21.2.117:0/513997649 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6a88003a60 0x7f6a88005ed0 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 -- 172.21.2.117:0/513997649 >> 172.21.2.117:0/513997649 conn(0x7f6aa00ced60 msgr2=0x7f6aa00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 -- 172.21.2.117:0/513997649 shutdown_connections 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.581 7f6aa6314700 1 -- 172.21.2.117:0/513997649 wait complete. 2021-02-10T11:56:16.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 Processor -- start 2021-02-10T11:56:16.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 -- start start 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6aa00bfa80 0x7f6aa01a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6aa00a1d70 0x7f6aa01a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01a3c30 con 0x7f6aa00bf680 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00a0680 con 0x7f6aa00bfa80 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.596 7f6aa6314700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00a08a0 con 0x7f6aa00a1d70 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59578/0 (socket says 172.21.2.117:59578) 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 -- 172.21.2.117:0/1292055200 learned_addr learned my addr 172.21.2.117:0/1292055200 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 -- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6aa00a1d70 msgr2=0x7f6aa01a3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6aa00a1d70 0x7f6aa01a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 -- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6aa00bfa80 msgr2=0x7f6aa01a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6aa00bfa80 0x7f6aa01a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 -- 172.21.2.117:0/1292055200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa00a0ac0 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9ffff700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7f6a90000bf0 tx=0x7f6a9004f9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a9004d3e0 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9004a570 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa00a0ac0 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa00a1050 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.597 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6a900024d0 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.598 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6a9004b000 con 0x7f6aa00bf680 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.598 7f6a9cff9700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 0x7f6a8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:16.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.598 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6a90076720 con 0x7f6aa00bf680 2021-02-10T11:56:16.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.598 7f6a9f7fe700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 0x7f6a8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:16.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.599 7f6a9f7fe700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 0x7f6a8001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:16.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.629 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa0087280 con 0x7f6aa00bf680 2021-02-10T11:56:16.634 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.631 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6a9004d3e0 con 0x7f6aa00bf680 2021-02-10T11:56:16.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.757 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aa00ede60 con 0x7f6aa00bf680 2021-02-10T11:56:16.760 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.758 7f6a9cff9700 1 -- 172.21.2.117:0/1292055200 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f6a9004bf00 con 0x7f6aa00bf680 2021-02-10T11:56:16.761 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:16.761 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":179,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:10.207917","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:56:07.173989","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:56:08.248299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:16.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 msgr2=0x7f6a8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.762 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 0x7f6a8001abc0 crc :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 msgr2=0x7f6aa00a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7f6a90000bf0 tx=0x7f6a9004f9f0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 shutdown_connections 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6aa00a1d70 0x7f6aa01a3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6aa00bfa80 0x7f6aa01a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6a800186c0 0x7f6a8001abc0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 --2- 172.21.2.117:0/1292055200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6aa00bf680 0x7f6aa00a1750 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.759 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 >> 172.21.2.117:0/1292055200 conn(0x7f6aa00874b0 msgr2=0x7f6aa0090b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.760 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 shutdown_connections 2021-02-10T11:56:16.763 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:16.760 7f6aa6314700 1 -- 172.21.2.117:0/1292055200 wait complete. 2021-02-10T11:56:16.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:56:07.173989', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:56:08.248299', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:56:16.771 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.186 7f0063670700 1 Processor -- start 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.186 7f0063670700 1 -- start start 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0063670700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 0x7f005c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0063670700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f005c0ef3f0 0x7f005c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0063670700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0ede60 con 0x7f005c0eb040 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0063670700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c088020 con 0x7f005c0ef3f0 2021-02-10T11:56:17.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0063670700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0d1d30 con 0x7f005c0d18f0 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0061c8d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f005c0ef3f0 0x7f005c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0060c8b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f005c0eb040 0x7f005c0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59836/0 (socket says 172.21.2.117:59836) 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0060c8b700 1 -- 172.21.2.117:0/3436229148 learned_addr learned my addr 172.21.2.117:0/3436229148 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0053fff700 1 -- 172.21.2.117:0/3436229148 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665169314 0 0) 0x7f004c0010b0 con 0x7f005c0eb040 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0053fff700 1 -- 172.21.2.117:0/3436229148 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0048002fe0 con 0x7f005c0eb040 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f006148c700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 0x7f005c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0061c8d700 1 -- 172.21.2.117:0/3436229148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 msgr2=0x7f005c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0061c8d700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 0x7f005c0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0061c8d700 1 -- 172.21.2.117:0/3436229148 >> v1:172.21.2.117:6789/0 conn(0x7f005c0eb040 legacy=0x7f005c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f0061c8d700 1 -- 172.21.2.117:0/3436229148 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c1b8fd0 con 0x7f005c0ef3f0 2021-02-10T11:56:17.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.187 7f006148c700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 0x7f005c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0061c8d700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0ef3f0 0x7f005c1b79a0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7f0054006e30 tx=0x7f0054009e60).ready entity=mon.0 client_cookie=32dd0530cd8c4aa3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0053fff700 1 -- 172.21.2.117:0/3436229148 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f0054006a50 con 0x7f005c0ef3f0 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0053fff700 1 -- 172.21.2.117:0/3436229148 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005400b300 con 0x7f005c0ef3f0 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0053fff700 1 -- 172.21.2.117:0/3436229148 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005400b820 con 0x7f005c0ef3f0 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0063670700 1 -- 172.21.2.117:0/3436229148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0ef3f0 msgr2=0x7f005c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.188 7f0063670700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0ef3f0 0x7f005c1b79a0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7f0054006e30 tx=0x7f0054009e60).stop 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 -- 172.21.2.117:0/3436229148 shutdown_connections 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c0ef3f0 0x7f005c1b79a0 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 --2- 172.21.2.117:0/3436229148 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c0d18f0 0x7f005c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 -- 172.21.2.117:0/3436229148 >> 172.21.2.117:0/3436229148 conn(0x7f005c0cf060 msgr2=0x7f005c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 -- 172.21.2.117:0/3436229148 shutdown_connections 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.189 7f0063670700 1 -- 172.21.2.117:0/3436229148 wait complete. 2021-02-10T11:56:17.208 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.203 7f0063670700 1 Processor -- start 2021-02-10T11:56:17.209 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.203 7f0063670700 1 -- start start 2021-02-10T11:56:17.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c1a4860 0x7f005c0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a65a0 0x7f005c0a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0a1820 con 0x7f005c09e160 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0a69a0 con 0x7f005c0a65a0 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f0063670700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c09dc80 con 0x7f005c1a4860 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59590/0 (socket says 172.21.2.117:59590) 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 -- 172.21.2.117:0/2049668561 learned_addr learned my addr 172.21.2.117:0/2049668561 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 -- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c1a4860 msgr2=0x7f005c0a5f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c1a4860 0x7f005c0a5f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 -- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a65a0 msgr2=0x7f005c0a1da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:17.211 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a65a0 0x7f005c0a1da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 -- 172.21.2.117:0/2049668561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0a23c0 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.204 7f006148c700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7f0058006f20 tx=0x7f005800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f005800bd80 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058000c90 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f00580022b0 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0063670700 1 -- 172.21.2.117:0/2049668561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c0a23c0 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0063670700 1 -- 172.21.2.117:0/2049668561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c0a23c0 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.205 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f005800a000 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.206 7f0051ffb700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 0x7f003c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.206 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f00580326b0 con 0x7f005c09e160 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.206 7f0060c8b700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 0x7f003c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.212 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.207 7f0060c8b700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 0x7f003c01abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.236 7f0063670700 1 -- 172.21.2.117:0/2049668561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c0ede60 con 0x7f005c09e160 2021-02-10T11:56:17.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.240 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f0058006bb0 con 0x7f005c09e160 2021-02-10T11:56:17.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.370 7f0063670700 1 -- 172.21.2.117:0/2049668561 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f005c0ede60 con 0x7f005c09e160 2021-02-10T11:56:17.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.371 7f0051ffb700 1 -- 172.21.2.117:0/2049668561 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v179) v1 ==== 92+0+10 (secure 0 0 0) 0x7f005800bd80 con 0x7f005c09e160 2021-02-10T11:56:17.374 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:17.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 msgr2=0x7f003c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 0x7f003c01abc0 crc :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 msgr2=0x7f005c1a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7f0058006f20 tx=0x7f005800b990).stop 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 shutdown_connections 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f005c0a65a0 0x7f005c0a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:17.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f005c1a4860 0x7f005c0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:17.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f003c0186c0 0x7f003c01abc0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 --2- 172.21.2.117:0/2049668561 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f005c09e160 0x7f005c1a4240 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 >> 172.21.2.117:0/2049668561 conn(0x7f005c09e660 msgr2=0x7f005c0bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:17.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 shutdown_connections 2021-02-10T11:56:17.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.373 7f0063670700 1 -- 172.21.2.117:0/2049668561 wait complete. 2021-02-10T11:56:17.384 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 Processor -- start 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 -- start start 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec0d18f0 0x7fd3ec0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.816 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec0ede60 con 0x7fd3ec0ee3e0 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec0eba60 con 0x7fd3ec0eb040 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.797 7fd3f4292700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0d1cf0 con 0x7fd3ec0d18f0 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f18ad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f18ad700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:35750/0 (socket says 172.21.2.117:35750) 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59598/0 (socket says 172.21.2.117:59598) 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f18ad700 1 -- 172.21.2.117:0/1494822306 learned_addr learned my addr 172.21.2.117:0/1494822306 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f20ae700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec0d18f0 0x7fd3ec0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 -- 172.21.2.117:0/1494822306 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec0d18f0 msgr2=0x7fd3ec0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec0d18f0 0x7fd3ec0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 -- 172.21.2.117:0/1494822306 >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 msgr2=0x7fd3ec0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:17.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 --2- 172.21.2.117:0/1494822306 >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 -- 172.21.2.117:0/1494822306 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0eee00 con 0x7fd3ec0ee3e0 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f18ad700 1 --2- 172.21.2.117:0/1494822306 >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f28af700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7fd3e4006e30 tx=0x7fd3e4009e60).ready entity=mon.0 client_cookie=a7edd4433e605dcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f10ac700 1 -- 172.21.2.117:0/1494822306 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e4006a50 con 0x7fd3ec0ee3e0 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.798 7fd3f10ac700 1 -- 172.21.2.117:0/1494822306 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e400b300 con 0x7fd3ec0ee3e0 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f10ac700 1 -- 172.21.2.117:0/1494822306 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e400b790 con 0x7fd3ec0ee3e0 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 -- 172.21.2.117:0/1494822306 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0ee3e0 msgr2=0x7fd3ec0ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7fd3e4006e30 tx=0x7fd3e4009e60).stop 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 -- 172.21.2.117:0/1494822306 shutdown_connections 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 --2- 172.21.2.117:0/1494822306 >> v2:172.21.2.125:3300/0 conn(0x7fd3ec0eb040 0x7fd3ec0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0ee3e0 0x7fd3ec0ee7e0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 --2- 172.21.2.117:0/1494822306 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec0d18f0 0x7fd3ec0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 -- 172.21.2.117:0/1494822306 >> 172.21.2.117:0/1494822306 conn(0x7fd3ec0cf060 msgr2=0x7fd3ec0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:17.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 -- 172.21.2.117:0/1494822306 shutdown_connections 2021-02-10T11:56:17.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.799 7fd3f4292700 1 -- 172.21.2.117:0/1494822306 wait complete. 2021-02-10T11:56:17.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.813 7fd3f4292700 1 Processor -- start 2021-02-10T11:56:17.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 -- start start 2021-02-10T11:56:17.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3ec1a48b0 0x7fd3ec08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec08cf40 0x7fd3ec0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0aae30 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0aa9f0 con 0x7fd3ec1a48b0 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f4292700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec0ab330 con 0x7fd3ec08cf40 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59604/0 (socket says 172.21.2.117:59604) 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 -- 172.21.2.117:0/855127357 learned_addr learned my addr 172.21.2.117:0/855127357 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f18ad700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec08cf40 0x7fd3ec0a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f20ae700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3ec1a48b0 0x7fd3ec08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 -- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec08cf40 msgr2=0x7fd3ec0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec08cf40 0x7fd3ec0a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 -- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3ec1a48b0 msgr2=0x7fd3ec08c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3ec1a48b0 0x7fd3ec08c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.814 7fd3f28af700 1 -- 172.21.2.117:0/855127357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0ab550 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3f28af700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7fd3e4001070 tx=0x7fd3e400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e4001270 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e4006df0 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3e4001830 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3f4292700 1 -- 172.21.2.117:0/855127357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec0ab550 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.815 7fd3f4292700 1 -- 172.21.2.117:0/855127357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec0ab550 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.816 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3e40093b0 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.816 7fd3deffd700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.816 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd3e4032a10 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.816 7fd3f20ae700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:17.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.817 7fd3f20ae700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 0x7fd3d401abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:17.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.847 7fd3f4292700 1 -- 172.21.2.117:0/855127357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec0ede60 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.850 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3e4003020 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.978 7fd3f4292700 1 -- 172.21.2.117:0/855127357 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fd3ec0ede60 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.979 7fd3deffd700 1 -- 172.21.2.117:0/855127357 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v179) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd3e40093b0 con 0x7fd3ec0a9f90 2021-02-10T11:56:17.982 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 msgr2=0x7fd3d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 0x7fd3d401abc0 crc :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 msgr2=0x7fd3ec0aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7fd3e4001070 tx=0x7fd3e400bd40).stop 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 shutdown_connections 2021-02-10T11:56:17.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3d40186c0 0x7fd3d401abc0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3ec0a9f90 0x7fd3ec0aa390 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3ec08cf40 0x7fd3ec0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 --2- 172.21.2.117:0/855127357 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3ec1a48b0 0x7fd3ec08c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 >> 172.21.2.117:0/855127357 conn(0x7fd3ec092d00 msgr2=0x7fd3ec08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 shutdown_connections 2021-02-10T11:56:17.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:17.981 7fd3f4292700 1 -- 172.21.2.117:0/855127357 wait complete. 2021-02-10T11:56:17.992 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:56:18.153 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:56:18.156 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 Processor -- start 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 -- start start 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af80d18f0 0x7f5af80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5af80ef3f0 0x7f5af81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af80ede60 con 0x7f5af80eb040 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af8088020 con 0x7f5af80ef3f0 2021-02-10T11:56:18.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.405 7f5afedbe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80d1d30 con 0x7f5af80d18f0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5af80ef3f0 0x7f5af81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5af7fff700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f5af80eb040 0x7f5af80eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:52786/0 (socket says 172.21.2.117:52786) 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5af7fff700 1 -- 172.21.2.117:0/3840099030 learned_addr learned my addr 172.21.2.117:0/3840099030 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afcbda700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af80d18f0 0x7f5af80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 -- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af80d18f0 msgr2=0x7f5af80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af80d18f0 0x7f5af80eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 -- 172.21.2.117:0/3840099030 >> v1:172.21.2.125:6789/0 conn(0x7f5af80eb040 legacy=0x7f5af80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 -- 172.21.2.117:0/3840099030 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af81b8fd0 con 0x7f5af80ef3f0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5afd3db700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80ef3f0 0x7f5af81b79a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f5af0006e50 tx=0x7f5af0009e70).ready entity=mon.1 client_cookie=2ddd2861ed9884ac server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.406 7f5af77fe700 1 -- 172.21.2.117:0/3840099030 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af00066f0 con 0x7f5af80ef3f0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5af77fe700 1 -- 172.21.2.117:0/3840099030 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0003030 con 0x7f5af80ef3f0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5af77fe700 1 -- 172.21.2.117:0/3840099030 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5af000b700 con 0x7f5af80ef3f0 2021-02-10T11:56:18.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 -- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80ef3f0 msgr2=0x7f5af81b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80ef3f0 0x7f5af81b79a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f5af0006e50 tx=0x7f5af0009e70).stop 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 -- 172.21.2.117:0/3840099030 shutdown_connections 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80ef3f0 0x7f5af81b79a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 --2- 172.21.2.117:0/3840099030 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af80d18f0 0x7f5af80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 -- 172.21.2.117:0/3840099030 >> 172.21.2.117:0/3840099030 conn(0x7f5af80cf060 msgr2=0x7f5af80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 -- 172.21.2.117:0/3840099030 shutdown_connections 2021-02-10T11:56:18.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.407 7f5afedbe700 1 -- 172.21.2.117:0/3840099030 wait complete. 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 Processor -- start 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 -- start start 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 0x7f5af81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 0x7f5af80a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80a13a0 0x7f5af80a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af807cca0 con 0x7f5af807c7c0 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af818fb70 con 0x7f5af80a13a0 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afedbe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80dc6d0 con 0x7f5af81a3930 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afcbda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 0x7f5af81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5af7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 0x7f5af80a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afcbda700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 0x7f5af81a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59618/0 (socket says 172.21.2.117:59618) 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afcbda700 1 -- 172.21.2.117:0/2737136200 learned_addr learned my addr 172.21.2.117:0/2737136200 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.422 7f5afd3db700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80a13a0 0x7f5af80a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 -- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80a13a0 msgr2=0x7f5af80a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80a13a0 0x7f5af80a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 -- 172.21.2.117:0/2737136200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 msgr2=0x7f5af81a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 0x7f5af81a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 -- 172.21.2.117:0/2737136200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af807d220 con 0x7f5af81a3930 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af7fff700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 0x7f5af80a0d80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5ae800a4d0 tx=0x7f5ae800a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae800ac60 con 0x7f5af81a3930 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8007030 con 0x7f5af81a3930 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af807d220 con 0x7f5af81a3930 2021-02-10T11:56:18.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af807d7b0 con 0x7f5af81a3930 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.423 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5ae80045a0 con 0x7f5af81a3930 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.424 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5ae8004320 con 0x7f5af81a3930 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.424 7f5af57fa700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 0x7f5adc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.424 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5ae8032620 con 0x7f5af81a3930 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.425 7f5afcbda700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 0x7f5adc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:18.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.426 7f5afcbda700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 0x7f5adc01abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:18.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.455 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af818fb00 con 0x7f5af81a3930 2021-02-10T11:56:18.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.459 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5ae8000e20 con 0x7f5af81a3930 2021-02-10T11:56:18.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.591 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f5af818fb00 con 0x7f5af81a3930 2021-02-10T11:56:18.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.591 7f5af57fa700 1 -- 172.21.2.117:0/2737136200 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v179) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5ae8004320 con 0x7f5af81a3930 2021-02-10T11:56:18.594 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 msgr2=0x7f5adc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 0x7f5adc01abc0 crc :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 msgr2=0x7f5af80a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 0x7f5af80a0d80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5ae800a4d0 tx=0x7f5ae800a870).stop 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 shutdown_connections 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5af80a13a0 0x7f5af80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5af81a3930 0x7f5af80a0d80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5adc0186c0 0x7f5adc01abc0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 --2- 172.21.2.117:0/2737136200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5af807c7c0 0x7f5af81a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 >> 172.21.2.117:0/2737136200 conn(0x7f5af8087090 msgr2=0x7f5af808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 shutdown_connections 2021-02-10T11:56:18.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:18.593 7f5afedbe700 1 -- 172.21.2.117:0/2737136200 wait complete. 2021-02-10T11:56:18.604 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:56:18.604 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:19.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.013 7f37ea1f3700 1 Processor -- start 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.013 7f37ea1f3700 1 -- start start 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37ea1f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4085450 0x7f37e4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37ea1f3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e4085930 con 0x7f37e40e5240 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37ea1f3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e41b09e0 con 0x7f37e40e4e40 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37ea1f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40e5cc0 con 0x7f37e4085450 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37e37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f37e40e5240 0x7f37e4084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59874/0 (socket says 172.21.2.117:59874) 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37e37fe700 1 -- 172.21.2.117:0/175900900 learned_addr learned my addr 172.21.2.117:0/175900900 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282037686 0 0) 0x7f37d8001180 con 0x7f37e40e5240 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37e8810700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4085450 0x7f37e4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.014 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc002fe0 con 0x7f37e40e5240 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2876202316 0 0) 0x7f37d8001180 con 0x7f37e40e5240 2021-02-10T11:56:19.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4085450 msgr2=0x7f37e4086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4085450 0x7f37e4086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 >> v1:172.21.2.125:6789/0 conn(0x7f37e40e4e40 legacy=0x7f37e41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e4084840 con 0x7f37e40e5240 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f37d8001680 con 0x7f37e40e5240 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 >> v1:172.21.2.117:6789/0 conn(0x7f37e40e5240 legacy=0x7f37e4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37cc003ac0 0x7f37cc005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37cc006750 0x7f37cc008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37cc0094f0 0x7f37cc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc00c2c0 con 0x7f37cc0094f0 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc00c4e0 con 0x7f37cc006750 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc050710 con 0x7f37cc003ac0 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37cc0094f0 0x7f37cc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e37fe700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37cc006750 0x7f37cc008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e3fff700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37cc003ac0 0x7f37cc005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 -- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37cc003ac0 msgr2=0x7f37cc005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37cc003ac0 0x7f37cc005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 -- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37cc006750 msgr2=0x7f37cc008c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37cc006750 0x7f37cc008c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 -- 172.21.2.117:0/175900900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc050970 con 0x7f37cc0094f0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e37fe700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37cc006750 0x7f37cc008c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e8810700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37cc0094f0 0x7f37cc00ba10 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f37dc009b10 tx=0x7f37dc009eb0).ready entity=mon.0 client_cookie=2f04ae1b7c7319fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00b510 con 0x7f37cc0094f0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc003030 con 0x7f37cc0094f0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.015 7f37e2ffd700 1 -- 172.21.2.117:0/175900900 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc00b510 con 0x7f37cc0094f0 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 -- 172.21.2.117:0/175900900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37cc0094f0 msgr2=0x7f37cc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 --2- 172.21.2.117:0/175900900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37cc0094f0 0x7f37cc00ba10 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f37dc009b10 tx=0x7f37dc009eb0).stop 2021-02-10T11:56:19.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37e8810700 1 -- 172.21.2.117:0/175900900 reap_dead start 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 -- 172.21.2.117:0/175900900 shutdown_connections 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 -- 172.21.2.117:0/175900900 >> 172.21.2.117:0/175900900 conn(0x7f37e40ce4e0 msgr2=0x7f37e40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 -- 172.21.2.117:0/175900900 shutdown_connections 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.016 7f37ea1f3700 1 -- 172.21.2.117:0/175900900 wait complete. 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 Processor -- start 2021-02-10T11:56:19.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 -- start start 2021-02-10T11:56:19.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e40b1a70 0x7f37e40b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e40a8320 0x7f37e40b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40a87f0 con 0x7f37e40b2520 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40b1590 con 0x7f37e40a8320 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37ea1f3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40b2920 con 0x7f37e40b1a70 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37e8810700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37e8810700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59638/0 (socket says 172.21.2.117:59638) 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.031 7f37e8810700 1 -- 172.21.2.117:0/3619077719 learned_addr learned my addr 172.21.2.117:0/3619077719 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 -- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e40b1a70 msgr2=0x7f37e40b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e40b1a70 0x7f37e40b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 -- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e40a8320 msgr2=0x7f37e40b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e40a8320 0x7f37e40b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 -- 172.21.2.117:0/3619077719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40aee30 con 0x7f37e40b2520 2021-02-10T11:56:19.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e8810700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f37dc00b8c0 tx=0x7f37dc00bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc000df0 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc0090c0 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37dc001420 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e40aee30 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.032 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e40af3c0 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.033 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37dc0056b0 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.033 7f37e0ff9700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 0x7f37c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.033 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f37dc032a30 con 0x7f37e40b2520 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.033 7f37e3fff700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 0x7f37c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:19.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.033 7f37e3fff700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 0x7f37c401abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:19.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.064 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e410c2d0 con 0x7f37e40b2520 2021-02-10T11:56:19.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.066 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37dc00bda0 con 0x7f37e40b2520 2021-02-10T11:56:19.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.195 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e410c2d0 con 0x7f37e40b2520 2021-02-10T11:56:19.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.196 7f37e0ff9700 1 -- 172.21.2.117:0/3619077719 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f37dc0056b0 con 0x7f37e40b2520 2021-02-10T11:56:19.199 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:19.199 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":153,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:11.215531","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9310":{"gid":9310,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3249802432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3249802432},{"type":"v1","addr":"172.21.2.125:6835","nonce":3249802432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:56:19.199 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 153 2021-02-10T11:56:19.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.197 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 msgr2=0x7f37c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.197 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 0x7f37c401abc0 crc :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.197 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 msgr2=0x7f37e40ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.197 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7f37dc00b8c0 tx=0x7f37dc00bc60).stop 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 shutdown_connections 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37c40186c0 0x7f37c401abc0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e40b2520 0x7f37e40ae810 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e40a8320 0x7f37e40b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 --2- 172.21.2.117:0/3619077719 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e40b1a70 0x7f37e40b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 >> 172.21.2.117:0/3619077719 conn(0x7f37e40b5cd0 msgr2=0x7f37e40c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 shutdown_connections 2021-02-10T11:56:19.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:19.198 7f37ea1f3700 1 -- 172.21.2.117:0/3619077719 wait complete. 2021-02-10T11:56:19.411 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:56:09.205433', 'modified': '2021-02-10 11:56:11.215531', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9310}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9310': {'gid': 9310, 'name': 'b', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3249802432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3249802432}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3249802432}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:56:19.411 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:56:19.411 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:56:19.809 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:56:19.809 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9310, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.599141302, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 153, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 179, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 179, 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.239748380000002 2021-02-10T11:56:19.810 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:56:19.813 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9310, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.599141302, "mdsmap_epoch": 153, "osdmap_epoch": 179, "osdmap_epoch_barrier": 179, "uptime": 16.239748380000002 } 2021-02-10T11:56:19.813 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:56:19.814 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:56:19.814 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:56:19.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:19.814 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:20.151 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:56:20.495 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:56:20.495 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:56:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:56:20.840 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:56:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:56:20.841 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:20.841 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:56:21.177 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:56:21.177 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:21.180 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:56:21.526 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:56:21.526 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:56:21.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[188477]: starting ceph client 2021-02-10T11:56:21.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:56:21.588 7f9dac5baf80 -1 init, newargv = 0x56148e129f30 newargc=9 2021-02-10T11:56:21.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[188477]: starting fuse 2021-02-10T11:56:21.869 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:56:21.870 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:56:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:56:21.871 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:21.871 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:56:22.206 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:56:26.927 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:56:26.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:26.927 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:26.928 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:56:26.929 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:56:27.292 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.188477.asok 2021-02-10T11:56:27.294 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.188477.asok 2021-02-10T11:56:27.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:27.295 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.188477.asok status 2021-02-10T11:56:27.686 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:56:27.686 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "188477", 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 13212, 2021-02-10T11:56:27.687 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 13212 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3808968048 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:56:27.688 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3808968048 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.13212 172.21.2.117:0/3808968048", 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3808968048", 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 153, 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 179, 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:56:27.689 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:56:27.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:27.691 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:27.691 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:28.027 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:56:28.027 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:56:28.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:28.028 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:28.374 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.785 7fa65e0a6700 1 Processor -- start 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.785 7fa65e0a6700 1 -- start start 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa65e0a6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa6580d18e0 0x7fa6580ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa65e0a6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580eb000 0x7fa6580eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa65e0a6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6580ede60 con 0x7fa6580ee3e0 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa65e0a6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6580eba90 con 0x7fa6580d18e0 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa65e0a6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580d1ce0 con 0x7fa6580eb000 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa6580d18e0 0x7fa6580ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa6580d18e0 0x7fa6580ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59658/0 (socket says 172.21.2.117:59658) 2021-02-10T11:56:28.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 -- 172.21.2.117:0/2703300702 learned_addr learned my addr 172.21.2.117:0/2703300702 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa656ffd700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580eb000 0x7fa6580eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 -- 172.21.2.117:0/2703300702 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580eb000 msgr2=0x7fa6580eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580eb000 0x7fa6580eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 -- 172.21.2.117:0/2703300702 >> v1:172.21.2.125:6789/0 conn(0x7fa6580ee3e0 legacy=0x7fa6580ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 -- 172.21.2.117:0/2703300702 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580eefd0 con 0x7fa6580d18e0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.786 7fa6577fe700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580d18e0 0x7fa6580ea9e0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fa640006700 tx=0x7fa640009b80).ready entity=mon.0 client_cookie=735956a00f2b7b0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa6567fc700 1 -- 172.21.2.117:0/2703300702 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64000c0c0 con 0x7fa6580d18e0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa6567fc700 1 -- 172.21.2.117:0/2703300702 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000c440 con 0x7fa6580d18e0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa6567fc700 1 -- 172.21.2.117:0/2703300702 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64000c960 con 0x7fa6580d18e0 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa65e0a6700 1 -- 172.21.2.117:0/2703300702 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580d18e0 msgr2=0x7fa6580ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa65e0a6700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580d18e0 0x7fa6580ea9e0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7fa640006700 tx=0x7fa640009b80).stop 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.787 7fa65e0a6700 1 -- 172.21.2.117:0/2703300702 shutdown_connections 2021-02-10T11:56:28.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.788 7fa65e0a6700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580eb000 0x7fa6580eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.788 7fa65e0a6700 1 --2- 172.21.2.117:0/2703300702 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580d18e0 0x7fa6580ea9e0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.788 7fa65e0a6700 1 -- 172.21.2.117:0/2703300702 >> 172.21.2.117:0/2703300702 conn(0x7fa6580ced60 msgr2=0x7fa6580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:28.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.788 7fa65e0a6700 1 -- 172.21.2.117:0/2703300702 shutdown_connections 2021-02-10T11:56:28.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.788 7fa65e0a6700 1 -- 172.21.2.117:0/2703300702 wait complete. 2021-02-10T11:56:28.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.802 7fa65e0a6700 1 Processor -- start 2021-02-10T11:56:28.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.802 7fa65e0a6700 1 -- start start 2021-02-10T11:56:28.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580b7310 0x7fa6580aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6580ab2c0 0x7fa6580abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580ab8c0 con 0x7fa6580b77e0 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580ac460 con 0x7fa6580ab2c0 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa65e0a6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6580ac680 con 0x7fa6580b7310 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59660/0 (socket says 172.21.2.117:59660) 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 -- 172.21.2.117:0/2230114765 learned_addr learned my addr 172.21.2.117:0/2230114765 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 -- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580b7310 msgr2=0x7fa6580aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580b7310 0x7fa6580aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 -- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6580ab2c0 msgr2=0x7fa6580abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6580ab2c0 0x7fa6580abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 -- 172.21.2.117:0/2230114765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6580ac8a0 con 0x7fa6580b77e0 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.803 7fa6577fe700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fa64000b5e0 tx=0x7fa64000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:28.808 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa64000bd70 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa640002030 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6580ac8a0 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa640002550 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6580ace30 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa640000c90 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 0x7fa63c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa6400327e0 con 0x7fa6580b77e0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.804 7fa656ffd700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 0x7fa63c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:28.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.805 7fa656ffd700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 0x7fa63c01abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:28.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.836 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa658087290 con 0x7fa6580b77e0 2021-02-10T11:56:28.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.838 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa6400009b0 con 0x7fa6580b77e0 2021-02-10T11:56:28.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.961 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6580ede60 con 0x7fa6580b77e0 2021-02-10T11:56:28.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.962 7fa64ffff700 1 -- 172.21.2.117:0/2230114765 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fa640000c90 con 0x7fa6580b77e0 2021-02-10T11:56:28.964 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:28.964 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":179,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:56:10.207917","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:56:07.173989","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":39,"pool_name":"cephfs_data","create_time":"2021-02-10 11:56:08.248299","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:56:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 msgr2=0x7fa63c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:28.965 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 0x7fa63c01abc0 crc :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 msgr2=0x7fa6580b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7fa64000b5e0 tx=0x7fa64000b980).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 shutdown_connections 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6580ab2c0 0x7fa6580abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6580b7310 0x7fa6580aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa63c0186c0 0x7fa63c01abc0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 --2- 172.21.2.117:0/2230114765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6580b77e0 0x7fa6580b6cf0 unknown :-1 s=CLOSED pgs=3017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 >> 172.21.2.117:0/2230114765 conn(0x7fa6580874c0 msgr2=0x7fa6580af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:28.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 shutdown_connections 2021-02-10T11:56:28.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:28.964 7fa65e0a6700 1 -- 172.21.2.117:0/2230114765 wait complete. 2021-02-10T11:56:28.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 38, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:56:07.173989', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 39, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:56:08.248299', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:56:28.975 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:56:29.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.389 7f3b7c537700 1 Processor -- start 2021-02-10T11:56:29.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.389 7f3b7c537700 1 -- start start 2021-02-10T11:56:29.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7c537700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b740d15e0 0x7f3b740e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7c537700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b740e7580 con 0x7f3b740e7180 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7c537700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b740e77f0 con 0x7f3b741b07f0 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7c537700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b740e5ce0 con 0x7f3b740d15e0 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7ab54700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b740d15e0 0x7f3b740e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b79b52700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3b740e7180 0x7f3b740e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59918/0 (socket says 172.21.2.117:59918) 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b79b52700 1 -- 172.21.2.117:0/2477883363 learned_addr learned my addr 172.21.2.117:0/2477883363 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7ab54700 1 -- 172.21.2.117:0/2477883363 >> v1:172.21.2.125:6789/0 conn(0x7f3b741b07f0 legacy=0x7f3b740e4e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7ab54700 1 -- 172.21.2.117:0/2477883363 >> v1:172.21.2.117:6789/0 conn(0x7f3b740e7180 legacy=0x7f3b740e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7ab54700 1 -- 172.21.2.117:0/2477883363 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740d19e0 con 0x7f3b740d15e0 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.390 7f3b7ab54700 1 --2- 172.21.2.117:0/2477883363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b740d15e0 0x7f3b740e8560 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7f3b6c006e30 tx=0x7f3b6c009de0).ready entity=mon.0 client_cookie=9ff988d26f27b3fc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b79351700 1 -- 172.21.2.117:0/2477883363 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b6c00b300 con 0x7f3b740d15e0 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b79351700 1 -- 172.21.2.117:0/2477883363 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c00b680 con 0x7f3b740d15e0 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b79351700 1 -- 172.21.2.117:0/2477883363 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b6c00bba0 con 0x7f3b740d15e0 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 -- 172.21.2.117:0/2477883363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b740d15e0 msgr2=0x7f3b740e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 --2- 172.21.2.117:0/2477883363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b740d15e0 0x7f3b740e8560 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7f3b6c006e30 tx=0x7f3b6c009de0).stop 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 -- 172.21.2.117:0/2477883363 shutdown_connections 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 --2- 172.21.2.117:0/2477883363 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b740d15e0 0x7f3b740e8560 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 -- 172.21.2.117:0/2477883363 >> 172.21.2.117:0/2477883363 conn(0x7f3b740ce500 msgr2=0x7f3b740e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:29.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 -- 172.21.2.117:0/2477883363 shutdown_connections 2021-02-10T11:56:29.410 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.391 7f3b7c537700 1 -- 172.21.2.117:0/2477883363 wait complete. 2021-02-10T11:56:29.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 Processor -- start 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 -- start start 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 0x7f3b740b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b740b3510 0x7f3b740b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7407e4a0 con 0x7f3b741c10e0 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740b3f30 con 0x7f3b740a6c80 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b7c537700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740b4150 con 0x7f3b740b3510 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.406 7f3b79b52700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59676/0 (socket says 172.21.2.117:59676) 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 -- 172.21.2.117:0/607422121 learned_addr learned my addr 172.21.2.117:0/607422121 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b7ab54700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b740b3510 0x7f3b740b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b7a353700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 0x7f3b740b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:29.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 -- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b740b3510 msgr2=0x7f3b740b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b740b3510 0x7f3b740b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 -- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 msgr2=0x7f3b740b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 0x7f3b740b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 -- 172.21.2.117:0/607422121 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b740b15c0 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b7a353700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 0x7f3b740b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b79b52700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f3b68001c60 tx=0x7f3b6800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b6800acc0 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68004140 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b7c537700 1 -- 172.21.2.117:0/607422121 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b740b15c0 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b680045e0 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.407 7f3b7c537700 1 -- 172.21.2.117:0/607422121 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b740b15c0 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.408 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b68004320 con 0x7f3b741c10e0 2021-02-10T11:56:29.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.408 7f3b66ffd700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 0x7f3b5c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:29.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.408 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b680328e0 con 0x7f3b741c10e0 2021-02-10T11:56:29.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.408 7f3b7a353700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 0x7f3b5c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:29.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.409 7f3b7a353700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 0x7f3b5c01ac10 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:29.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.440 7f3b7c537700 1 -- 172.21.2.117:0/607422121 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7407c0a0 con 0x7f3b741c10e0 2021-02-10T11:56:29.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.442 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b68000e20 con 0x7f3b741c10e0 2021-02-10T11:56:29.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.570 7f3b7c537700 1 -- 172.21.2.117:0/607422121 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f3b7407c0a0 con 0x7f3b741c10e0 2021-02-10T11:56:29.573 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.571 7f3b66ffd700 1 -- 172.21.2.117:0/607422121 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v179) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3b6800acc0 con 0x7f3b741c10e0 2021-02-10T11:56:29.573 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:29.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 msgr2=0x7f3b5c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 0x7f3b5c01ac10 crc :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 msgr2=0x7f3b741c14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:29.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7f3b68001c60 tx=0x7f3b6800a8d0).stop 2021-02-10T11:56:29.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 shutdown_connections 2021-02-10T11:56:29.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b5c018730 0x7f3b5c01ac10 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b741c10e0 0x7f3b741c14e0 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b740b3510 0x7f3b740b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 --2- 172.21.2.117:0/607422121 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b740a6c80 0x7f3b740b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:29.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 >> 172.21.2.117:0/607422121 conn(0x7f3b740a7f20 msgr2=0x7f3b740873c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:29.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 shutdown_connections 2021-02-10T11:56:29.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.573 7f3b7c537700 1 -- 172.21.2.117:0/607422121 wait complete. 2021-02-10T11:56:29.583 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 Processor -- start 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 -- start start 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f144c0d18f0 0x7f144c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f144c0eb040 0x7f144c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c0ee3e0 0x7f144c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c0edee0 con 0x7f144c0d18f0 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f144c0ebaa0 con 0x7f144c0eb040 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1453875700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0d1cf0 con 0x7f144c0ee3e0 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.997 7f1451691700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f144c0d18f0 0x7f144c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f144c0d18f0 0x7f144c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59682/0 (socket says 172.21.2.117:59682) 2021-02-10T11:56:30.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 -- 172.21.2.117:0/3865059395 learned_addr learned my addr 172.21.2.117:0/3865059395 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1450e90700 1 --2- 172.21.2.117:0/3865059395 >> v2:172.21.2.125:3300/0 conn(0x7f144c0eb040 0x7f144c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451e92700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c0ee3e0 0x7f144c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 -- 172.21.2.117:0/3865059395 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c0ee3e0 msgr2=0x7f144c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c0ee3e0 0x7f144c0ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 -- 172.21.2.117:0/3865059395 >> v2:172.21.2.125:3300/0 conn(0x7f144c0eb040 msgr2=0x7f144c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 --2- 172.21.2.117:0/3865059395 >> v2:172.21.2.125:3300/0 conn(0x7f144c0eb040 0x7f144c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 -- 172.21.2.117:0/3865059395 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0eee00 con 0x7f144c0d18f0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f1451691700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c0d18f0 0x7f144c0eaa20 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f1448006700 tx=0x7f1448009b80).ready entity=mon.0 client_cookie=2a80dd46ba7bb8c6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.998 7f143ffff700 1 -- 172.21.2.117:0/3865059395 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144800c0c0 con 0x7f144c0d18f0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f143ffff700 1 -- 172.21.2.117:0/3865059395 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144800c440 con 0x7f144c0d18f0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f143ffff700 1 -- 172.21.2.117:0/3865059395 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144800c960 con 0x7f144c0d18f0 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 -- 172.21.2.117:0/3865059395 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c0d18f0 msgr2=0x7f144c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c0d18f0 0x7f144c0eaa20 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7f1448006700 tx=0x7f1448009b80).stop 2021-02-10T11:56:30.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 -- 172.21.2.117:0/3865059395 shutdown_connections 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 --2- 172.21.2.117:0/3865059395 >> v2:172.21.2.125:3300/0 conn(0x7f144c0eb040 0x7f144c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c0ee3e0 0x7f144c0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 --2- 172.21.2.117:0/3865059395 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c0d18f0 0x7f144c0eaa20 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:29.999 7f1453875700 1 -- 172.21.2.117:0/3865059395 >> 172.21.2.117:0/3865059395 conn(0x7f144c0cf060 msgr2=0x7f144c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.000 7f1453875700 1 -- 172.21.2.117:0/3865059395 shutdown_connections 2021-02-10T11:56:30.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.000 7f1453875700 1 -- 172.21.2.117:0/3865059395 wait complete. 2021-02-10T11:56:30.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.014 7f1453875700 1 Processor -- start 2021-02-10T11:56:30.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.014 7f1453875700 1 -- start start 2021-02-10T11:56:30.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c08cf40 0x7f144c0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f144c0a9f90 0x7f144c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0aae30 con 0x7f144c1a48b0 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0aa9f0 con 0x7f144c0a9f90 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1453875700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144c0ab330 con 0x7f144c08cf40 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59686/0 (socket says 172.21.2.117:59686) 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 -- 172.21.2.117:0/274524367 learned_addr learned my addr 172.21.2.117:0/274524367 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 -- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c08cf40 msgr2=0x7f144c0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c08cf40 0x7f144c0a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 -- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f144c0a9f90 msgr2=0x7f144c0aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f144c0a9f90 0x7f144c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 -- 172.21.2.117:0/274524367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0ab550 con 0x7f144c1a48b0 2021-02-10T11:56:30.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.015 7f1451691700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f144800b650 tx=0x7f144800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.016 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f144800bde0 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.016 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448000c90 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.016 7f1453875700 1 -- 172.21.2.117:0/274524367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c0ab550 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.016 7f1453875700 1 -- 172.21.2.117:0/274524367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c0ab550 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.016 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14480022b0 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.017 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1448009f90 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.017 7f143dffb700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 0x7f142c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.017 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1448032910 con 0x7f144c1a48b0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.017 7f1450e90700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 0x7f142c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.018 7f1450e90700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 0x7f142c01ac10 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.048 7f1453875700 1 -- 172.21.2.117:0/274524367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c0ede60 con 0x7f144c1a48b0 2021-02-10T11:56:30.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.050 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14480091e0 con 0x7f144c1a48b0 2021-02-10T11:56:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.176 7f1453875700 1 -- 172.21.2.117:0/274524367 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f144c0ede60 con 0x7f144c1a48b0 2021-02-10T11:56:30.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.177 7f143dffb700 1 -- 172.21.2.117:0/274524367 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v179) v1 ==== 104+0+10 (secure 0 0 0) 0x7f144800bde0 con 0x7f144c1a48b0 2021-02-10T11:56:30.179 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:30.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.178 7f1453875700 1 -- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 msgr2=0x7f142c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.178 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 0x7f142c01ac10 crc :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.178 7f1453875700 1 -- 172.21.2.117:0/274524367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 msgr2=0x7f144c08c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.178 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f144800b650 tx=0x7f144800b9f0).stop 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 -- 172.21.2.117:0/274524367 shutdown_connections 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f144c0a9f90 0x7f144c0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f144c08cf40 0x7f144c0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f142c018730 0x7f142c01ac10 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 --2- 172.21.2.117:0/274524367 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f144c1a48b0 0x7f144c08c920 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 -- 172.21.2.117:0/274524367 >> 172.21.2.117:0/274524367 conn(0x7f144c092d00 msgr2=0x7f144c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:30.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 -- 172.21.2.117:0/274524367 shutdown_connections 2021-02-10T11:56:30.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.179 7f1453875700 1 -- 172.21.2.117:0/274524367 wait complete. 2021-02-10T11:56:30.189 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 Processor -- start 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 -- start start 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 0x7f589c0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f589c0e8390 0x7f589c0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589c0e7580 con 0x7f589c1b07f0 2021-02-10T11:56:30.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589c0e5ce0 con 0x7f589c0e8390 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f58a12ef700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589c0e77f0 con 0x7f589c0e7180 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f589b87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f589c0e8390 0x7f589c0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.603 7f589b07d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f589c1b07f0 0x7f589c0e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59944/0 (socket says 172.21.2.117:59944) 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b07d700 1 -- 172.21.2.117:0/3974348544 learned_addr learned my addr 172.21.2.117:0/3974348544 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b87e700 1 -- 172.21.2.117:0/3974348544 >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 msgr2=0x7f589c0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589a87c700 1 --2- 172.21.2.117:0/3974348544 >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 0x7f589c0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b87e700 1 --2- 172.21.2.117:0/3974348544 >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 0x7f589c0e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b87e700 1 -- 172.21.2.117:0/3974348544 >> v1:172.21.2.117:6789/0 conn(0x7f589c1b07f0 legacy=0x7f589c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b87e700 1 -- 172.21.2.117:0/3974348544 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0e8db0 con 0x7f589c0e8390 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589a87c700 1 --2- 172.21.2.117:0/3974348544 >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 0x7f589c0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.604 7f589b87e700 1 --2- 172.21.2.117:0/3974348544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0e8390 0x7f589c0e8790 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7f5894006e30 tx=0x7f5894009e60).ready entity=mon.0 client_cookie=9203711577df1208 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f589a07b700 1 -- 172.21.2.117:0/3974348544 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589400b300 con 0x7f589c0e8390 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f589a07b700 1 -- 172.21.2.117:0/3974348544 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589400b580 con 0x7f589c0e8390 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f589a07b700 1 -- 172.21.2.117:0/3974348544 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f589400ba60 con 0x7f589c0e8390 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 -- 172.21.2.117:0/3974348544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0e8390 msgr2=0x7f589c0e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 --2- 172.21.2.117:0/3974348544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0e8390 0x7f589c0e8790 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7f5894006e30 tx=0x7f5894009e60).stop 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 -- 172.21.2.117:0/3974348544 shutdown_connections 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 --2- 172.21.2.117:0/3974348544 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0e8390 0x7f589c0e8790 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 --2- 172.21.2.117:0/3974348544 >> v2:172.21.2.125:3300/0 conn(0x7f589c0e7180 0x7f589c0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.605 7f58a12ef700 1 -- 172.21.2.117:0/3974348544 >> 172.21.2.117:0/3974348544 conn(0x7f589c0ce500 msgr2=0x7f589c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.606 7f58a12ef700 1 -- 172.21.2.117:0/3974348544 shutdown_connections 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.606 7f58a12ef700 1 -- 172.21.2.117:0/3974348544 wait complete. 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.620 7f58a12ef700 1 Processor -- start 2021-02-10T11:56:30.624 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.620 7f58a12ef700 1 -- start start 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589c199630 0x7f589c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f589c091d40 0x7f589c092140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0aa6d0 con 0x7f589c0ac870 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0aa8c0 con 0x7f589c091d40 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f58a12ef700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589c0aaae0 con 0x7f589c199630 2021-02-10T11:56:30.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f589b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f589b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59702/0 (socket says 172.21.2.117:59702) 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f589b87e700 1 -- 172.21.2.117:0/1654424323 learned_addr learned my addr 172.21.2.117:0/1654424323 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f589b07d700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589c199630 0x7f589c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.621 7f589a87c700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f589c091d40 0x7f589c092140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 -- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589c199630 msgr2=0x7f589c199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589c199630 0x7f589c199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 -- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f589c091d40 msgr2=0x7f589c092140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f589c091d40 0x7f589c092140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 -- 172.21.2.117:0/1654424323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589c0aad00 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f589b87e700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f5894006ab0 tx=0x7f589400bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5894003030 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5894001270 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5894001790 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589c0aad00 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.622 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589c0aad00 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.623 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5894001450 con 0x7f589c0ac870 2021-02-10T11:56:30.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.623 7f5883fff700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 0x7f587c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:30.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.623 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5894032a70 con 0x7f589c0ac870 2021-02-10T11:56:30.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.623 7f589b07d700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 0x7f587c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:30.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.624 7f589b07d700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 0x7f587c01abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:30.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.653 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589c1b8010 con 0x7f589c0ac870 2021-02-10T11:56:30.657 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.656 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f589400be20 con 0x7f589c0ac870 2021-02-10T11:56:30.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.782 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f589c1b8010 con 0x7f589c0ac870 2021-02-10T11:56:30.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.782 7f5883fff700 1 -- 172.21.2.117:0/1654424323 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v179) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5894001450 con 0x7f589c0ac870 2021-02-10T11:56:30.785 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 msgr2=0x7f587c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 0x7f587c01abc0 crc :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 msgr2=0x7f589c0acc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f5894006ab0 tx=0x7f589400bce0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 shutdown_connections 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f587c0186c0 0x7f587c01abc0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f589c0ac870 0x7f589c0acc70 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f589c091d40 0x7f589c092140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 --2- 172.21.2.117:0/1654424323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f589c199630 0x7f589c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.784 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 >> 172.21.2.117:0/1654424323 conn(0x7f589c0b91b0 msgr2=0x7f589c0aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.785 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 shutdown_connections 2021-02-10T11:56:30.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:30.785 7f58a12ef700 1 -- 172.21.2.117:0/1654424323 wait complete. 2021-02-10T11:56:30.795 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:56:30.795 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:31.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.204 7f6b0139f700 1 Processor -- start 2021-02-10T11:56:31.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 -- start start 2021-02-10T11:56:31.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0eb000 0x7f6afc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6afc0ee3e0 0x7f6afc0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc0ede60 con 0x7f6afc0d18e0 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc0eba20 con 0x7f6afc0ee3e0 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6b0139f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0d1ce0 con 0x7f6afc0eb000 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6afb07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6afb07d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59706/0 (socket says 172.21.2.117:59706) 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.205 7f6afb07d700 1 -- 172.21.2.117:0/1318872219 learned_addr learned my addr 172.21.2.117:0/1318872219 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 -- 172.21.2.117:0/1318872219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0eb000 msgr2=0x7f6afc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 --2- 172.21.2.117:0/1318872219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0eb000 0x7f6afc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 -- 172.21.2.117:0/1318872219 >> v2:172.21.2.125:3300/0 conn(0x7f6afc0ee3e0 msgr2=0x7f6afc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 --2- 172.21.2.117:0/1318872219 >> v2:172.21.2.125:3300/0 conn(0x7f6afc0ee3e0 0x7f6afc0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 -- 172.21.2.117:0/1318872219 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0eee00 con 0x7f6afc0d18e0 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afb07d700 1 --2- 172.21.2.117:0/1318872219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f6aec006700 tx=0x7f6aec009b80).ready entity=mon.0 client_cookie=3c9b4770fa723b21 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:31.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afa07b700 1 -- 172.21.2.117:0/1318872219 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6aec00c0c0 con 0x7f6afc0d18e0 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afa07b700 1 -- 172.21.2.117:0/1318872219 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec00c400 con 0x7f6afc0d18e0 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.206 7f6afa07b700 1 -- 172.21.2.117:0/1318872219 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6aec00c920 con 0x7f6afc0d18e0 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 -- 172.21.2.117:0/1318872219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc0d18e0 msgr2=0x7f6afc0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 --2- 172.21.2.117:0/1318872219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7f6aec006700 tx=0x7f6aec009b80).stop 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 -- 172.21.2.117:0/1318872219 shutdown_connections 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 --2- 172.21.2.117:0/1318872219 >> v2:172.21.2.125:3300/0 conn(0x7f6afc0ee3e0 0x7f6afc0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 --2- 172.21.2.117:0/1318872219 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0eb000 0x7f6afc0eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 --2- 172.21.2.117:0/1318872219 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc0d18e0 0x7f6afc0ea9e0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 -- 172.21.2.117:0/1318872219 >> 172.21.2.117:0/1318872219 conn(0x7f6afc0ced60 msgr2=0x7f6afc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 -- 172.21.2.117:0/1318872219 shutdown_connections 2021-02-10T11:56:31.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.207 7f6b0139f700 1 -- 172.21.2.117:0/1318872219 wait complete. 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 Processor -- start 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 -- start start 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6afc19c7c0 0x7f6afc092f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0e8f90 0x7f6afc0e9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0a9aa0 con 0x7f6afc08a280 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc19cbc0 con 0x7f6afc19c7c0 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.222 7f6b0139f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc08aca0 con 0x7f6afc0e8f90 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59712/0 (socket says 172.21.2.117:59712) 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 -- 172.21.2.117:0/3039806492 learned_addr learned my addr 172.21.2.117:0/3039806492 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 -- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0e8f90 msgr2=0x7f6afc0e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:31.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0e8f90 0x7f6afc0e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 -- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6afc19c7c0 msgr2=0x7f6afc092f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6afc19c7c0 0x7f6afc092f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 -- 172.21.2.117:0/3039806492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0e99b0 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6afa87c700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f6af0006f00 tx=0x7f6af000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af00009b0 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0006350 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6af0000c90 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.223 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc0e99b0 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.224 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc0e99b0 con 0x7f6afc08a280 2021-02-10T11:56:31.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.224 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6af0007000 con 0x7f6afc08a280 2021-02-10T11:56:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.224 7f6ae3fff700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 0x7f6adc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.224 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6af0032690 con 0x7f6afc08a280 2021-02-10T11:56:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.224 7f6afb07d700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 0x7f6adc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:31.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.225 7f6afb07d700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 0x7f6adc01abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:31.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.255 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc0e8a60 con 0x7f6afc08a280 2021-02-10T11:56:31.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.258 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6af0006b90 con 0x7f6afc08a280 2021-02-10T11:56:31.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.385 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6afc0e8a60 con 0x7f6afc08a280 2021-02-10T11:56:31.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.385 7f6ae3fff700 1 -- 172.21.2.117:0/3039806492 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f6af000bd30 con 0x7f6afc08a280 2021-02-10T11:56:31.388 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:31.388 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":153,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:11.215531","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9310":{"gid":9310,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3249802432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3249802432},{"type":"v1","addr":"172.21.2.125:6835","nonce":3249802432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:56:31.389 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 153 2021-02-10T11:56:31.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 msgr2=0x7f6adc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:31.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 0x7f6adc01abc0 crc :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:31.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 msgr2=0x7f6afc08a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:31.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7f6af0006f00 tx=0x7f6af000b940).stop 2021-02-10T11:56:31.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 shutdown_connections 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6adc0186c0 0x7f6adc01abc0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6afc08a280 0x7f6afc08a680 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6afc0e8f90 0x7f6afc0e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 --2- 172.21.2.117:0/3039806492 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6afc19c7c0 0x7f6afc092f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.387 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 >> 172.21.2.117:0/3039806492 conn(0x7f6afc19f610 msgr2=0x7f6afc098640 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:31.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.388 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 shutdown_connections 2021-02-10T11:56:31.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:31.388 7f6b0139f700 1 -- 172.21.2.117:0/3039806492 wait complete. 2021-02-10T11:56:31.599 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 153, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:56:09.205433', 'modified': '2021-02-10 11:56:11.215531', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9310}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9310': {'gid': 9310, 'name': 'b', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/3249802432', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 3249802432}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 3249802432}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:56:31.600 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:56:31.600 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:56:31.997 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:56:31.997 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:56:31.997 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:56:31.997 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9310, 2021-02-10T11:56:31.997 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 22.787044358999999, 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 153, 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 179, 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 179, 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 28.427651415 2021-02-10T11:56:31.998 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:56:32.001 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9310, "want_state": "up:active", "state": "up:active", "rank_uptime": 22.787044358999999, "mdsmap_epoch": 153, "osdmap_epoch": 179, "osdmap_epoch_barrier": 179, "uptime": 28.427651415 } 2021-02-10T11:56:32.001 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:56:32.001 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:56:32.001 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:56:32.002 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:32.002 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:32.337 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:56:32.682 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:56:32.682 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:56:33.026 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:56:33.026 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:56:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:56:33.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:33.027 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:56:33.363 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:56:33.363 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:56:33.363 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:33.366 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:56:33.711 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:56:33.712 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:56:33.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[189117]: starting ceph client 2021-02-10T11:56:33.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:56:33.776 7fcfa9a55f80 -1 init, newargv = 0x55d4fd603f70 newargc=9 2021-02-10T11:56:33.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[189117]: starting fuse 2021-02-10T11:56:34.054 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:56:34.054 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:56:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:56:34.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:34.056 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:56:34.391 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:56:34.391 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:56:34.391 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:56:34.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:34.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:56:34.391 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:34.392 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:56:34.393 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:56:34.755 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.189117.asok 2021-02-10T11:56:34.758 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.189117.asok 2021-02-10T11:56:34.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:34.758 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.189117.asok status 2021-02-10T11:56:35.150 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "189117", 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:56:35.151 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 13257, 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 13257 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3112430487 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:56:35.152 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3112430487 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.13257 172.21.2.117:0/3112430487", 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3112430487", 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 153, 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 179, 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:56:35.153 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:56:35.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:35.155 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:35.155 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:35.490 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:56:35.491 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:56:35.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:35.491 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:35.837 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:56:36.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.251 7ff34951f700 1 Processor -- start 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.251 7ff34951f700 1 -- start start 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff34951f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440e4e40 0x7ff3441b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff34951f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff344085450 0x7ff3440d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff34951f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff344085930 con 0x7ff3440e5240 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff34951f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440e5cc0 con 0x7ff344085450 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff34951f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff344084840 con 0x7ff3440e4e40 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff342abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff3440e5240 0x7ff344084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:59984/0 (socket says 172.21.2.117:59984) 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff342abe700 1 -- 172.21.2.117:0/2876708792 learned_addr learned my addr 172.21.2.117:0/2876708792 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521845396 0 0) 0x7ff32c0010b0 con 0x7ff3440e5240 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff330002fe0 con 0x7ff3440e5240 2021-02-10T11:56:36.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff3432bf700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440e4e40 0x7ff3441b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.252 7ff343ac0700 1 --2- 172.21.2.117:0/2876708792 >> v2:172.21.2.125:3300/0 conn(0x7ff344085450 0x7ff3440d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 146098352 0 0) 0x7ff32c0010b0 con 0x7ff3440e5240 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440e4e40 msgr2=0x7ff3441b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440e4e40 0x7ff3441b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 >> v2:172.21.2.125:3300/0 conn(0x7ff344085450 msgr2=0x7ff3440d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 --2- 172.21.2.117:0/2876708792 >> v2:172.21.2.125:3300/0 conn(0x7ff344085450 0x7ff3440d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440d1be0 con 0x7ff3440e5240 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff32c0015b0 con 0x7ff3440e5240 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 >> v1:172.21.2.117:6789/0 conn(0x7ff3440e5240 legacy=0x7ff344084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff330003ac0 0x7ff330005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff33004a790 0x7ff33004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff33004d500 0x7ff33004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300502d0 con 0x7ff33004d500 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300504f0 con 0x7ff330003ac0 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff330050710 con 0x7ff33004a790 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff33004d500 0x7ff33004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff342abe700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff33004a790 0x7ff33004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff3432bf700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff330003ac0 0x7ff330005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 -- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff33004a790 msgr2=0x7ff33004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff33004a790 0x7ff33004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 -- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff330003ac0 msgr2=0x7ff330005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff330003ac0 0x7ff330005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.253 7ff343ac0700 1 -- 172.21.2.117:0/2876708792 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300509b0 con 0x7ff33004d500 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff343ac0700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff33004d500 0x7ff33004fa20 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7ff33c006a40 tx=0x7ff33c00c0c0).ready entity=mon.0 client_cookie=ba08435c00de589b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33c00c4b0 con 0x7ff33004d500 2021-02-10T11:56:36.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c00c6f0 con 0x7ff33004d500 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff3422bd700 1 -- 172.21.2.117:0/2876708792 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff33c00cb50 con 0x7ff33004d500 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 -- 172.21.2.117:0/2876708792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff33004d500 msgr2=0x7ff33004fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 --2- 172.21.2.117:0/2876708792 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff33004d500 0x7ff33004fa20 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 rx=0x7ff33c006a40 tx=0x7ff33c00c0c0).stop 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff343ac0700 1 -- 172.21.2.117:0/2876708792 reap_dead start 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 -- 172.21.2.117:0/2876708792 shutdown_connections 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 -- 172.21.2.117:0/2876708792 >> 172.21.2.117:0/2876708792 conn(0x7ff3440ce4e0 msgr2=0x7ff3440e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 -- 172.21.2.117:0/2876708792 shutdown_connections 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.254 7ff34951f700 1 -- 172.21.2.117:0/2876708792 wait complete. 2021-02-10T11:56:36.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 Processor -- start 2021-02-10T11:56:36.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 -- start start 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440be700 0x7ff3440bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3440c0420 0x7ff3440a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440bee60 con 0x7ff3440bfa00 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440beb00 con 0x7ff3440c0420 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.269 7ff34951f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440912f0 con 0x7ff3440be700 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59746/0 (socket says 172.21.2.117:59746) 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 -- 172.21.2.117:0/2949845522 learned_addr learned my addr 172.21.2.117:0/2949845522 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 -- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440be700 msgr2=0x7ff3440bf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440be700 0x7ff3440bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 -- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3440c0420 msgr2=0x7ff3440a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3440c0420 0x7ff3440a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 -- 172.21.2.117:0/2949845522 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440a9bd0 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff342abe700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 rx=0x7ff32c001660 tx=0x7ff32c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff32c00ad20 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0016a0 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff32c004330 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff34951f700 1 -- 172.21.2.117:0/2949845522 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3440a9bd0 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.270 7ff34951f700 1 -- 172.21.2.117:0/2949845522 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3440aa160 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.271 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff32c004140 con 0x7ff3440bfa00 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.271 7ff33bfff700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 0x7ff32801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.271 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff32c032640 con 0x7ff3440bfa00 2021-02-10T11:56:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.271 7ff3432bf700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 0x7ff32801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.272 7ff3432bf700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 0x7ff32801abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.301 7ff34951f700 1 -- 172.21.2.117:0/2949845522 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34410c2f0 con 0x7ff3440bfa00 2021-02-10T11:56:36.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.304 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff32c0009b0 con 0x7ff3440bfa00 2021-02-10T11:56:36.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.440 7ff34951f700 1 -- 172.21.2.117:0/2949845522 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff344085930 con 0x7ff3440bfa00 2021-02-10T11:56:36.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.494 7ff33bfff700 1 -- 172.21.2.117:0/2949845522 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v154) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff32c004140 con 0x7ff3440bfa00 2021-02-10T11:56:36.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 msgr2=0x7ff32801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 0x7ff32801abc0 crc :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 msgr2=0x7ff3440bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 rx=0x7ff32c001660 tx=0x7ff32c00a930).stop 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 shutdown_connections 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff3280186c0 0x7ff32801abc0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff3440bfa00 0x7ff3440bfe00 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff3440c0420 0x7ff3440a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 --2- 172.21.2.117:0/2949845522 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff3440be700 0x7ff3440bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 >> 172.21.2.117:0/2949845522 conn(0x7ff3440bd990 msgr2=0x7ff3440c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 shutdown_connections 2021-02-10T11:56:36.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.506 7ff34951f700 1 -- 172.21.2.117:0/2949845522 wait complete. 2021-02-10T11:56:36.516 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:56:36.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.925 7f63d7315700 1 Processor -- start 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.925 7f63d7315700 1 -- start start 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d7315700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63d00e7180 0x7f63d00e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d7315700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f63d00e8390 0x7f63d00e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d7315700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00e7580 con 0x7f63d01b07f0 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d7315700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00e5ce0 con 0x7f63d00e7180 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d7315700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00e77f0 con 0x7f63d00e8390 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f63d00e7180 0x7f63d00e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d5131700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f63d01b07f0 0x7f63d00e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60002/0 (socket says 172.21.2.117:60002) 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d5131700 1 -- 172.21.2.117:0/1209009176 learned_addr learned my addr 172.21.2.117:0/1209009176 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 -- 172.21.2.117:0/1209009176 >> v2:172.21.2.125:3300/0 conn(0x7f63d00e8390 msgr2=0x7f63d00e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 --2- 172.21.2.117:0/1209009176 >> v2:172.21.2.125:3300/0 conn(0x7f63d00e8390 0x7f63d00e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 -- 172.21.2.117:0/1209009176 >> v1:172.21.2.117:6789/0 conn(0x7f63d01b07f0 legacy=0x7f63d00e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 -- 172.21.2.117:0/1209009176 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00e8db0 con 0x7f63d00e7180 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63c7fff700 1 -- 172.21.2.117:0/1209009176 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350011792 0 0) 0x7f63cc0009b0 con 0x7f63d01b07f0 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.926 7f63d4930700 1 --2- 172.21.2.117:0/1209009176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00e7180 0x7f63d00e7d70 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f63c0006650 tx=0x7f63c0009a80).ready entity=mon.0 client_cookie=a7e183326d557d7e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63c7fff700 1 -- 172.21.2.117:0/1209009176 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c000c0c0 con 0x7f63d00e7180 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63c7fff700 1 -- 172.21.2.117:0/1209009176 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c000c340 con 0x7f63d00e7180 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63c7fff700 1 -- 172.21.2.117:0/1209009176 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63c000c860 con 0x7f63d00e7180 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 -- 172.21.2.117:0/1209009176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00e7180 msgr2=0x7f63d00e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 --2- 172.21.2.117:0/1209009176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00e7180 0x7f63d00e7d70 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f63c0006650 tx=0x7f63c0009a80).stop 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 -- 172.21.2.117:0/1209009176 shutdown_connections 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 --2- 172.21.2.117:0/1209009176 >> v2:172.21.2.125:3300/0 conn(0x7f63d00e8390 0x7f63d00e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 --2- 172.21.2.117:0/1209009176 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00e7180 0x7f63d00e7d70 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.927 7f63d7315700 1 -- 172.21.2.117:0/1209009176 >> 172.21.2.117:0/1209009176 conn(0x7f63d00ce500 msgr2=0x7f63d00e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.928 7f63d7315700 1 -- 172.21.2.117:0/1209009176 shutdown_connections 2021-02-10T11:56:36.946 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.928 7f63d7315700 1 -- 172.21.2.117:0/1209009176 wait complete. 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.942 7f63d7315700 1 Processor -- start 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.942 7f63d7315700 1 -- start start 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63d00a9d50 0x7f63d00aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63d00a7700 0x7f63d00a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00a8db0 con 0x7f63d00a9330 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0105f50 con 0x7f63d00a7700 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d7315700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00ab4c0 con 0x7f63d00a9d50 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59754/0 (socket says 172.21.2.117:59754) 2021-02-10T11:56:36.948 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 -- 172.21.2.117:0/3080779411 learned_addr learned my addr 172.21.2.117:0/3080779411 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 -- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63d00a9d50 msgr2=0x7f63d00aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63d00a9d50 0x7f63d00aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 -- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63d00a7700 msgr2=0x7f63d00a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63d00a7700 0x7f63d00a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 -- 172.21.2.117:0/3080779411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00a8120 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.943 7f63d5131700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f63cc001c60 tx=0x7f63cc00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63cc00acc0 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc004350 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f63cc0047b0 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63d7315700 1 -- 172.21.2.117:0/3080779411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d00a8120 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63d7315700 1 -- 172.21.2.117:0/3080779411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d00a86b0 con 0x7f63d00a9330 2021-02-10T11:56:36.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f63cc004530 con 0x7f63d00a9330 2021-02-10T11:56:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 0x7f63b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.944 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f63cc032510 con 0x7f63d00a9330 2021-02-10T11:56:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.945 7f63d4930700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 0x7f63b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:36.950 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.945 7f63d4930700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 0x7f63b001abc0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:36.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.976 7f63d7315700 1 -- 172.21.2.117:0/3080779411 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d01b8070 con 0x7f63d00a9330 2021-02-10T11:56:36.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:36.978 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f63cc000990 con 0x7f63d00a9330 2021-02-10T11:56:37.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.113 7f63d7315700 1 -- 172.21.2.117:0/3080779411 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f63d00a86b0 con 0x7f63b00186c0 2021-02-10T11:56:37.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.114 7f63c5ffb700 1 -- 172.21.2.117:0/3080779411 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f63c0006040 con 0x7f63b00186c0 2021-02-10T11:56:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:56:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:56:37.118 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:37.118 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:56:37.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.117 7f63d7315700 1 -- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 msgr2=0x7f63b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.117 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 0x7f63b001abc0 crc :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.117 7f63d7315700 1 -- 172.21.2.117:0/3080779411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 msgr2=0x7f63d00a9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.117 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 rx=0x7f63cc001c60 tx=0x7f63cc00a8d0).stop 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 -- 172.21.2.117:0/3080779411 shutdown_connections 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f63d00a7700 0x7f63d00a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f63d00a9d50 0x7f63d00aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f63b00186c0 0x7f63b001abc0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 --2- 172.21.2.117:0/3080779411 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f63d00a9330 0x7f63d00a9730 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 -- 172.21.2.117:0/3080779411 >> 172.21.2.117:0/3080779411 conn(0x7f63d00b97b0 msgr2=0x7f63d0098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 -- 172.21.2.117:0/3080779411 shutdown_connections 2021-02-10T11:56:37.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.118 7f63d7315700 1 -- 172.21.2.117:0/3080779411 wait complete. 2021-02-10T11:56:37.128 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:56:37.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.544 7f8b8d41f700 1 Processor -- start 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.544 7f8b8d41f700 1 -- start start 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8d41f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8b880ef3f0 0x7f8b880eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8d41f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b880ef7f0 con 0x7f8b880eb830 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8d41f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b880ede60 con 0x7f8b880d18f0 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8d41f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b880d1da0 con 0x7f8b880ef3f0 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8787e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f8b880eb830 0x7f8b881b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60016/0 (socket says 172.21.2.117:60016) 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8787e700 1 -- 172.21.2.117:0/3421049456 learned_addr learned my addr 172.21.2.117:0/3421049456 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8687c700 1 --2- 172.21.2.117:0/3421049456 >> v2:172.21.2.117:3300/0 conn(0x7f8b880ef3f0 0x7f8b880eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8687c700 1 -- 172.21.2.117:0/3421049456 >> v1:172.21.2.125:6789/0 conn(0x7f8b880d18f0 legacy=0x7f8b880eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8687c700 1 -- 172.21.2.117:0/3421049456 >> v1:172.21.2.117:6789/0 conn(0x7f8b880eb830 legacy=0x7f8b881b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.545 7f8b8687c700 1 -- 172.21.2.117:0/3421049456 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b881bb1c0 con 0x7f8b880ef3f0 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8687c700 1 --2- 172.21.2.117:0/3421049456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b880ef3f0 0x7f8b880eb210 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f8b78006650 tx=0x7f8b78009a80).ready entity=mon.0 client_cookie=a17bd4c98bb15194 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8607b700 1 -- 172.21.2.117:0/3421049456 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7800c0c0 con 0x7f8b880ef3f0 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8607b700 1 -- 172.21.2.117:0/3421049456 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7800c340 con 0x7f8b880ef3f0 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8607b700 1 -- 172.21.2.117:0/3421049456 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b7800c860 con 0x7f8b880ef3f0 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 -- 172.21.2.117:0/3421049456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b880ef3f0 msgr2=0x7f8b880eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 --2- 172.21.2.117:0/3421049456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b880ef3f0 0x7f8b880eb210 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 rx=0x7f8b78006650 tx=0x7f8b78009a80).stop 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 -- 172.21.2.117:0/3421049456 shutdown_connections 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 --2- 172.21.2.117:0/3421049456 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b880ef3f0 0x7f8b880eb210 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 -- 172.21.2.117:0/3421049456 >> 172.21.2.117:0/3421049456 conn(0x7f8b880cf060 msgr2=0x7f8b880cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 -- 172.21.2.117:0/3421049456 shutdown_connections 2021-02-10T11:56:37.564 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.546 7f8b8d41f700 1 -- 172.21.2.117:0/3421049456 wait complete. 2021-02-10T11:56:37.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.561 7f8b8d41f700 1 Processor -- start 2021-02-10T11:56:37.566 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.561 7f8b8d41f700 1 -- start start 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b8809e4e0 0x7f8b880e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b880e94a0 0x7f8b881c9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b881b7740 con 0x7f8b881c9880 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880e98a0 con 0x7f8b880e94a0 2021-02-10T11:56:37.567 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8d41f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880e17d0 con 0x7f8b8809e4e0 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59772/0 (socket says 172.21.2.117:59772) 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 -- 172.21.2.117:0/384844505 learned_addr learned my addr 172.21.2.117:0/384844505 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 -- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b8809e4e0 msgr2=0x7f8b880e8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b8809e4e0 0x7f8b880e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 -- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b880e94a0 msgr2=0x7f8b881c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b880e94a0 0x7f8b881c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 -- 172.21.2.117:0/384844505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b881c9c80 con 0x7f8b881c9880 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b8787e700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7f8b8000a940 tx=0x7f8b8000ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.562 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b80001270 con 0x7f8b881c9880 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b80000d30 con 0x7f8b881c9880 2021-02-10T11:56:37.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8b80004b20 con 0x7f8b881c9880 2021-02-10T11:56:37.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b881c9c80 con 0x7f8b881c9880 2021-02-10T11:56:37.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b881c9c80 con 0x7f8b881c9880 2021-02-10T11:56:37.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8b8000ae20 con 0x7f8b881c9880 2021-02-10T11:56:37.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b7ffff700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:37.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8b80032c40 con 0x7f8b881c9880 2021-02-10T11:56:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.563 7f8b8707d700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:37.571 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.564 7f8b8707d700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:37.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.595 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b88087390 con 0x7f8b881c9880 2021-02-10T11:56:37.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.597 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8b80007cb0 con 0x7f8b881c9880 2021-02-10T11:56:37.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.747 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8b880d18f0 con 0x7f8b881c9880 2021-02-10T11:56:37.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.748 7f8b7ffff700 1 -- 172.21.2.117:0/384844505 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8b8000ae20 con 0x7f8b881c9880 2021-02-10T11:56:37.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.749 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 msgr2=0x7f8b6c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.749 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 crc :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.749 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 msgr2=0x7f8b880a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 rx=0x7f8b8000a940 tx=0x7f8b8000ace0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 shutdown_connections 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8b6c0186c0 0x7f8b6c01abc0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8b881c9880 0x7f8b880a46a0 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8b880e94a0 0x7f8b881c9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 --2- 172.21.2.117:0/384844505 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8b8809e4e0 0x7f8b880e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 >> 172.21.2.117:0/384844505 conn(0x7f8b880e71d0 msgr2=0x7f8b880b66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 shutdown_connections 2021-02-10T11:56:37.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:37.750 7f8b8d41f700 1 -- 172.21.2.117:0/384844505 wait complete. 2021-02-10T11:56:37.760 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000826206 2021-02-10T11:56:38.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 Processor -- start 2021-02-10T11:56:38.193 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 -- start start 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7efcdc0ef3f0 0x7efcdc0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0ef7f0 con 0x7efcdc0eb830 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0ede60 con 0x7efcdc0d18f0 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.174 7efce3379700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0d1da0 con 0x7efcdc0ef3f0 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efce1996700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7efcdc0eb830 0x7efcdc1b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60030/0 (socket says 172.21.2.117:60030) 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efce1996700 1 -- 172.21.2.117:0/3148171788 learned_addr learned my addr 172.21.2.117:0/3148171788 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efce0994700 1 --2- 172.21.2.117:0/3148171788 >> v2:172.21.2.125:3300/0 conn(0x7efcdc0ef3f0 0x7efcdc0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179458425 0 0) 0x7efcd4000c10 con 0x7efcdc0eb830 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc8002fe0 con 0x7efcdc0eb830 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503007599 0 0) 0x7efcd0001400 con 0x7efcdc0d18f0 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc8003210 con 0x7efcdc0d18f0 2021-02-10T11:56:38.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.175 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3771008096 0 0) 0x7efcd4001180 con 0x7efcdc0eb830 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 >> v2:172.21.2.125:3300/0 conn(0x7efcdc0ef3f0 msgr2=0x7efcdc0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 --2- 172.21.2.117:0/3148171788 >> v2:172.21.2.125:3300/0 conn(0x7efcdc0ef3f0 0x7efcdc0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 >> v1:172.21.2.125:6789/0 conn(0x7efcdc0d18f0 legacy=0x7efcdc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc1bb1c0 con 0x7efcdc0eb830 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7efcd4001680 con 0x7efcdc0eb830 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 >> v1:172.21.2.117:6789/0 conn(0x7efcdc0eb830 legacy=0x7efcdc1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 0x7efcc8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcc8006820 0x7efcc8008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcc8009640 0x7efcc800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80503c0 con 0x7efcc8003b90 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80505a0 con 0x7efcc8006820 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80507c0 con 0x7efcc8009640 2021-02-10T11:56:38.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 0x7efcc8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce0994700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcc8006820 0x7efcc8008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1996700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcc8009640 0x7efcc800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 -- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcc8009640 msgr2=0x7efcc800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcc8009640 0x7efcc800bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 -- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcc8006820 msgr2=0x7efcc8008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcc8006820 0x7efcc8008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 -- 172.21.2.117:0/3148171788 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8050a60 con 0x7efcc8003b90 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1996700 1 -- 172.21.2.117:0/3148171788 reap_dead start 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce1195700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 0x7efcc8005fa0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7efcd0006f90 tx=0x7efcd0009de0).ready entity=mon.0 client_cookie=fa6c65fe5b8a8a13 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd00030c0 con 0x7efcc8003b90 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd000b510 con 0x7efcc8003b90 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efccffff700 1 -- 172.21.2.117:0/3148171788 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd000b960 con 0x7efcc8003b90 2021-02-10T11:56:38.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce3379700 1 -- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 msgr2=0x7efcc8005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.176 7efce3379700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 0x7efcc8005fa0 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 rx=0x7efcd0006f90 tx=0x7efcd0009de0).stop 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.177 7efce3379700 1 -- 172.21.2.117:0/3148171788 shutdown_connections 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.177 7efce3379700 1 --2- 172.21.2.117:0/3148171788 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcc8003b90 0x7efcc8005fa0 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.177 7efce3379700 1 -- 172.21.2.117:0/3148171788 >> 172.21.2.117:0/3148171788 conn(0x7efcdc0cf060 msgr2=0x7efcdc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.177 7efce3379700 1 -- 172.21.2.117:0/3148171788 shutdown_connections 2021-02-10T11:56:38.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.177 7efce3379700 1 -- 172.21.2.117:0/3148171788 wait complete. 2021-02-10T11:56:38.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 Processor -- start 2021-02-10T11:56:38.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 -- start start 2021-02-10T11:56:38.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcdc0aed60 0x7efcdc0a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcdc0aa040 0x7efcdc0aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0ae8c0 con 0x7efcdc0bda30 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0be450 con 0x7efcdc0aed60 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.192 7efce3379700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0be670 con 0x7efcdc0aa040 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59788/0 (socket says 172.21.2.117:59788) 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 -- 172.21.2.117:0/2522504201 learned_addr learned my addr 172.21.2.117:0/2522504201 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce0994700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcdc0aa040 0x7efcdc0aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1195700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcdc0aed60 0x7efcdc0a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 -- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcdc0aa040 msgr2=0x7efcdc0aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcdc0aa040 0x7efcdc0aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 -- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcdc0aed60 msgr2=0x7efcdc0a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcdc0aed60 0x7efcdc0a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 -- 172.21.2.117:0/2522504201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0be890 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce1996700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7efcd4009fa0 tx=0x7efcd400b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd400bd30 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd40090d0 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7efcd4009530 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.193 7efce3379700 1 -- 172.21.2.117:0/2522504201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc0be890 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.194 7efce3379700 1 -- 172.21.2.117:0/2522504201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc0be890 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.194 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7efcd40092b0 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.194 7efccdffb700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 0x7efcbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.194 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7efcd4032620 con 0x7efcdc0bda30 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.194 7efce1195700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 0x7efcbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.195 7efce1195700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 0x7efcbc01abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.224 7efce3379700 1 -- 172.21.2.117:0/2522504201 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc0daf70 con 0x7efcdc0bda30 2021-02-10T11:56:38.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.227 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7efcd4026fb0 con 0x7efcdc0bda30 2021-02-10T11:56:38.355 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.353 7efce3379700 1 -- 172.21.2.117:0/2522504201 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7efcdc0ede60 con 0x7efcbc0186c0 2021-02-10T11:56:38.358 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:56:38.355 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:56:38.358 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:56:38.355 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:56:38.359 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:56:38.355 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:56:38.359 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:56:38.355 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:56:38.359 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:56:38.355 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:56:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.370 7efccdffb700 1 -- 172.21.2.117:0/2522504201 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7efcd00031f0 con 0x7efcbc0186c0 2021-02-10T11:56:38.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.385 7efce3379700 1 -- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 msgr2=0x7efcbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.385 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 0x7efcbc01abc0 crc :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.387 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.385 7efce3379700 1 -- 172.21.2.117:0/2522504201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 msgr2=0x7efcdc0bde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 rx=0x7efcd4009fa0 tx=0x7efcd400b940).stop 2021-02-10T11:56:38.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 -- 172.21.2.117:0/2522504201 shutdown_connections 2021-02-10T11:56:38.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7efcbc0186c0 0x7efcbc01abc0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7efcdc0bda30 0x7efcdc0bde30 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7efcdc0aa040 0x7efcdc0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 --2- 172.21.2.117:0/2522504201 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7efcdc0aed60 0x7efcdc0a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 -- 172.21.2.117:0/2522504201 >> 172.21.2.117:0/2522504201 conn(0x7efcdc0ab150 msgr2=0x7efcdc0b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:38.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 -- 172.21.2.117:0/2522504201 shutdown_connections 2021-02-10T11:56:38.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.386 7efce3379700 1 -- 172.21.2.117:0/2522504201 wait complete. 2021-02-10T11:56:38.396 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000911369 --group_name group_0000000000826206 2021-02-10T11:56:38.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 Processor -- start 2021-02-10T11:56:38.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 -- start start 2021-02-10T11:56:38.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 0x7f5044086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.829 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5044084c60 0x7f5044085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5044085930 con 0x7f50440e4e40 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5044085680 con 0x7f5044084c60 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.810 7f504ba1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50441b09e0 con 0x7f50440e5240 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f5049836700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f50440e4e40 0x7f50441b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60046/0 (socket says 172.21.2.117:60046) 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f5049836700 1 -- 172.21.2.117:0/1902204327 learned_addr learned my addr 172.21.2.117:0/1902204327 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f5049035700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 0x7f5044086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f504a037700 1 --2- 172.21.2.117:0/1902204327 >> v2:172.21.2.125:3300/0 conn(0x7f5044084c60 0x7f5044085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f5048834700 1 -- 172.21.2.117:0/1902204327 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540410161 0 0) 0x7f50400010b0 con 0x7f50440e4e40 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.811 7f5048834700 1 -- 172.21.2.117:0/1902204327 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030002fe0 con 0x7f50440e4e40 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504a037700 1 -- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 msgr2=0x7f5044086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504a037700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 0x7f5044086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.830 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504a037700 1 -- 172.21.2.117:0/1902204327 >> v1:172.21.2.117:6789/0 conn(0x7f50440e4e40 legacy=0x7f50441b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f5048834700 1 -- 172.21.2.117:0/1902204327 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3249680500 0 0) 0x7f50400010b0 con 0x7f50440e4e40 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504a037700 1 -- 172.21.2.117:0/1902204327 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5044084840 con 0x7f5044084c60 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f5049035700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 0x7f5044086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504a037700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5044084c60 0x7f5044085060 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f503c006e50 tx=0x7f503c009e70).ready entity=mon.1 client_cookie=4291d9b1608fcb0b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f5048834700 1 -- 172.21.2.117:0/1902204327 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c006a20 con 0x7f5044084c60 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f5048834700 1 -- 172.21.2.117:0/1902204327 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c006790 con 0x7f5044084c60 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f5048834700 1 -- 172.21.2.117:0/1902204327 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503c00b700 con 0x7f5044084c60 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504ba1a700 1 -- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5044084c60 msgr2=0x7f5044085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.812 7f504ba1a700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5044084c60 0x7f5044085060 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f503c006e50 tx=0x7f503c009e70).stop 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 -- 172.21.2.117:0/1902204327 shutdown_connections 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5044084c60 0x7f5044085060 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 --2- 172.21.2.117:0/1902204327 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440e5240 0x7f5044086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 -- 172.21.2.117:0/1902204327 >> 172.21.2.117:0/1902204327 conn(0x7f50440ce4e0 msgr2=0x7f50440e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:38.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 -- 172.21.2.117:0/1902204327 shutdown_connections 2021-02-10T11:56:38.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.813 7f504ba1a700 1 -- 172.21.2.117:0/1902204327 wait complete. 2021-02-10T11:56:38.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 Processor -- start 2021-02-10T11:56:38.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 -- start start 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440a31e0 0x7f50441a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f50440b0140 0x7f50440a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50440a35e0 con 0x7f50441a7f00 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5044092fd0 con 0x7f50440b0140 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f504ba1a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5044093490 con 0x7f50440a31e0 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f5049035700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f5049035700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59800/0 (socket says 172.21.2.117:59800) 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.828 7f5049035700 1 -- 172.21.2.117:0/2350171945 learned_addr learned my addr 172.21.2.117:0/2350171945 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f504a037700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f50440b0140 0x7f50440a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049836700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440a31e0 0x7f50441a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 -- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440a31e0 msgr2=0x7f50441a78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440a31e0 0x7f50441a78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 -- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f50440b0140 msgr2=0x7f50440a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f50440b0140 0x7f50440a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 -- 172.21.2.117:0/2350171945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440b0540 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f5049035700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f5034006f40 tx=0x7f503400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f503400bd80 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034000c90 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5034002220 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.829 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50440b0540 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.830 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50440b0540 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.830 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f503400a000 con 0x7f50441a7f00 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.830 7f503a7fc700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 0x7f502c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:38.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.830 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f50340326b0 con 0x7f50441a7f00 2021-02-10T11:56:38.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.831 7f5049836700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 0x7f502c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:38.836 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.831 7f5049836700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 0x7f502c01abc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:38.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.860 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504410c2b0 con 0x7f50441a7f00 2021-02-10T11:56:38.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.863 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5034030d10 con 0x7f50441a7f00 2021-02-10T11:56:38.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:38.990 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911369", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7f5044092fd0 con 0x7f502c0186c0 2021-02-10T11:56:39.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.016 7f503a7fc700 1 -- 172.21.2.117:0/2350171945 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f50400009b0 con 0x7f502c0186c0 2021-02-10T11:56:39.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 msgr2=0x7f502c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 0x7f502c01abc0 crc :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 msgr2=0x7f50440afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f5034006f40 tx=0x7f503400b990).stop 2021-02-10T11:56:39.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 shutdown_connections 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f502c0186c0 0x7f502c01abc0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f50441a7f00 0x7f50440afb20 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f50440b0140 0x7f50440a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 --2- 172.21.2.117:0/2350171945 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f50440a31e0 0x7f50441a78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 >> 172.21.2.117:0/2350171945 conn(0x7f50440b5d10 msgr2=0x7f50440c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 shutdown_connections 2021-02-10T11:56:39.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.023 7f504ba1a700 1 -- 172.21.2.117:0/2350171945 wait complete. 2021-02-10T11:56:39.034 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000911370 --group_name group_0000000000826206 2021-02-10T11:56:39.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.449 7f4481650700 1 Processor -- start 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.449 7f4481650700 1 -- start start 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f4481650700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f447c0e4e40 0x7f447c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f4481650700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0e5240 0x7f447c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f4481650700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c085930 con 0x7f447c1b0810 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f4481650700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447c084840 con 0x7f447c0e4e40 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f4481650700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c084a30 con 0x7f447c0e5240 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447bac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f447c1b0810 0x7f447c085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60058/0 (socket says 172.21.2.117:60058) 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447bac0700 1 -- 172.21.2.117:0/1792500817 learned_addr learned my addr 172.21.2.117:0/1792500817 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 --2- 172.21.2.117:0/1792500817 >> v2:172.21.2.117:3300/0 conn(0x7f447c0e4e40 0x7f447c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 -- 172.21.2.117:0/1792500817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0e5240 msgr2=0x7f447c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 --2- 172.21.2.117:0/1792500817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0e5240 0x7f447c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 -- 172.21.2.117:0/1792500817 >> v1:172.21.2.117:6789/0 conn(0x7f447c1b0810 legacy=0x7f447c085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 -- 172.21.2.117:0/1792500817 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c085280 con 0x7f447c0e4e40 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.450 7f447b2bf700 1 --2- 172.21.2.117:0/1792500817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c0e4e40 0x7f447c1b01f0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 rx=0x7f4464006650 tx=0x7f4464009a80).ready entity=mon.0 client_cookie=256d80015a282cfa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f447a2bd700 1 -- 172.21.2.117:0/1792500817 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44640009b0 con 0x7f447c0e4e40 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f447a2bd700 1 -- 172.21.2.117:0/1792500817 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446400c0c0 con 0x7f447c0e4e40 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f447a2bd700 1 -- 172.21.2.117:0/1792500817 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f446400c5e0 con 0x7f447c0e4e40 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 -- 172.21.2.117:0/1792500817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c0e4e40 msgr2=0x7f447c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 --2- 172.21.2.117:0/1792500817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c0e4e40 0x7f447c1b01f0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 rx=0x7f4464006650 tx=0x7f4464009a80).stop 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 -- 172.21.2.117:0/1792500817 shutdown_connections 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 --2- 172.21.2.117:0/1792500817 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0e5240 0x7f447c084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 --2- 172.21.2.117:0/1792500817 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c0e4e40 0x7f447c1b01f0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.451 7f4481650700 1 -- 172.21.2.117:0/1792500817 >> 172.21.2.117:0/1792500817 conn(0x7f447c0ce4e0 msgr2=0x7f447c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.452 7f4481650700 1 -- 172.21.2.117:0/1792500817 shutdown_connections 2021-02-10T11:56:39.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.452 7f4481650700 1 -- 172.21.2.117:0/1792500817 wait complete. 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.466 7f4481650700 1 Processor -- start 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.466 7f4481650700 1 -- start start 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f447c18e5b0 0x7f447c18e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0b30d0 0x7f447c0b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1ae8d0 con 0x7f447c099bf0 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0b3af0 con 0x7f447c18e5b0 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f4481650700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0b3d10 con 0x7f447c0b30d0 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59814/0 (socket says 172.21.2.117:59814) 2021-02-10T11:56:39.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 -- 172.21.2.117:0/1735364523 learned_addr learned my addr 172.21.2.117:0/1735364523 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 -- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0b30d0 msgr2=0x7f447c0b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0b30d0 0x7f447c0b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 -- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f447c18e5b0 msgr2=0x7f447c18e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f447c18e5b0 0x7f447c18e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.467 7f447b2bf700 1 -- 172.21.2.117:0/1735364523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c0b3f30 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f447b2bf700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f4464009630 tx=0x7f446400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44640093a0 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446400bd70 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f44640054d0 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4481650700 1 -- 172.21.2.117:0/1735364523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c0b3f30 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4481650700 1 -- 172.21.2.117:0/1735364523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c1a19b0 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.468 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f446400a000 con 0x7f447c099bf0 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.469 7f4473fff700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 0x7f446001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:39.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.469 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f44640326c0 con 0x7f447c099bf0 2021-02-10T11:56:39.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.469 7f447aabe700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 0x7f446001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:39.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.469 7f447aabe700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 0x7f446001abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:39.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.499 7f4481650700 1 -- 172.21.2.117:0/1735364523 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c18e080 con 0x7f447c099bf0 2021-02-10T11:56:39.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.502 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4464024ee0 con 0x7f447c099bf0 2021-02-10T11:56:39.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.629 7f4481650700 1 -- 172.21.2.117:0/1735364523 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911370", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7f447c1ae8d0 con 0x7f44600186c0 2021-02-10T11:56:39.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.650 7f4473fff700 1 -- 172.21.2.117:0/1735364523 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f446c0009b0 con 0x7f44600186c0 2021-02-10T11:56:39.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.663 7f4481650700 1 -- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 msgr2=0x7f446001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.663 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 0x7f446001abc0 crc :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.663 7f4481650700 1 -- 172.21.2.117:0/1735364523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 msgr2=0x7f447c1ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.663 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 rx=0x7f4464009630 tx=0x7f446400b980).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 -- 172.21.2.117:0/1735364523 shutdown_connections 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f447c0b30d0 0x7f447c0b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f447c18e5b0 0x7f447c18e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44600186c0 0x7f446001abc0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 --2- 172.21.2.117:0/1735364523 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f447c099bf0 0x7f447c1ae2b0 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:39.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 -- 172.21.2.117:0/1735364523 >> 172.21.2.117:0/1735364523 conn(0x7f447c1ad6b0 msgr2=0x7f447c190560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:39.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 -- 172.21.2.117:0/1735364523 shutdown_connections 2021-02-10T11:56:39.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:39.664 7f4481650700 1 -- 172.21.2.117:0/1735364523 wait complete. 2021-02-10T11:56:39.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000911369 guest1 --group_name group_0000000000826206 --tenant_id tenant1 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.097 7f4ffec32700 1 Processor -- start 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.097 7f4ffec32700 1 -- start start 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ff80d18f0 0x7f4ff80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff80ef3f0 0x7f4ff81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff80ef7f0 con 0x7f4ff80eb040 2021-02-10T11:56:40.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff80eded0 con 0x7f4ff80d18f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80d1cf0 con 0x7f4ff80ef3f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ff80d18f0 0x7f4ff80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4ff80d18f0 0x7f4ff80eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59820/0 (socket says 172.21.2.117:59820) 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 -- 172.21.2.117:0/4277705486 learned_addr learned my addr 172.21.2.117:0/4277705486 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 -- 172.21.2.117:0/4277705486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff80ef3f0 msgr2=0x7f4ff81b79a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 --2- 172.21.2.117:0/4277705486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff80ef3f0 0x7f4ff81b79a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 -- 172.21.2.117:0/4277705486 >> v1:172.21.2.125:6789/0 conn(0x7f4ff80eb040 legacy=0x7f4ff80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 -- 172.21.2.117:0/4277705486 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff81b8fd0 con 0x7f4ff80d18f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffca4e700 1 --2- 172.21.2.117:0/4277705486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80d18f0 0x7f4ff80eaa20 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f4fe0006700 tx=0x7f4fe0009b00).ready entity=mon.0 client_cookie=225d7459418dd346 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ff77fe700 1 -- 172.21.2.117:0/4277705486 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe000c0c0 con 0x7f4ff80d18f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ff77fe700 1 -- 172.21.2.117:0/4277705486 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe000c400 con 0x7f4ff80d18f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ff77fe700 1 -- 172.21.2.117:0/4277705486 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe000c8e0 con 0x7f4ff80d18f0 2021-02-10T11:56:40.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 -- 172.21.2.117:0/4277705486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80d18f0 msgr2=0x7f4ff80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.098 7f4ffec32700 1 --2- 172.21.2.117:0/4277705486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80d18f0 0x7f4ff80eaa20 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 rx=0x7f4fe0006700 tx=0x7f4fe0009b00).stop 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 -- 172.21.2.117:0/4277705486 shutdown_connections 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 --2- 172.21.2.117:0/4277705486 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff80ef3f0 0x7f4ff81b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 --2- 172.21.2.117:0/4277705486 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80d18f0 0x7f4ff80eaa20 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 -- 172.21.2.117:0/4277705486 >> 172.21.2.117:0/4277705486 conn(0x7f4ff80cf060 msgr2=0x7f4ff80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 -- 172.21.2.117:0/4277705486 shutdown_connections 2021-02-10T11:56:40.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.099 7f4ffec32700 1 -- 172.21.2.117:0/4277705486 wait complete. 2021-02-10T11:56:40.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.113 7f4ffec32700 1 Processor -- start 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 -- start start 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff81a4770 0x7f4ff80b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ff81a4b70 0x7f4ff81a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80a0c90 con 0x7f4ff80b5270 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80a0e80 con 0x7f4ff81a4b70 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ffec32700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff81a3f30 con 0x7f4ff81a4770 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59824/0 (socket says 172.21.2.117:59824) 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 -- 172.21.2.117:0/1584145510 learned_addr learned my addr 172.21.2.117:0/1584145510 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 -- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff81a4770 msgr2=0x7f4ff80b4c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff81a4770 0x7f4ff80b4c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 -- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ff81a4b70 msgr2=0x7f4ff81a38e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ff81a4b70 0x7f4ff81a38e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.114 7f4ff7fff700 1 -- 172.21.2.117:0/1584145510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff81a4150 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff7fff700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 rx=0x7f4fe8006ed0 tx=0x7f4fe800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe8006240 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe80054c0 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff81a4150 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff808c2a0 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4fe80059a0 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4fe8005700 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.115 7f4ff57fa700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 0x7f4fdc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.116 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4fe8032680 con 0x7f4ff80b5270 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.116 7f4ffca4e700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 0x7f4fdc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:40.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.116 7f4ffca4e700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 0x7f4fdc01abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:40.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.147 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff80db020 con 0x7f4ff80b5270 2021-02-10T11:56:40.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.150 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4fe802cee0 con 0x7f4ff80b5270 2021-02-10T11:56:40.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.272 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911369", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000826206"}) v1 -- 0x7f4ff808c2a0 con 0x7f4fdc0186c0 2021-02-10T11:56:40.280 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest1 with tenant tenant1 2021-02-10T11:56:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.307 7f4ff57fa700 1 -- 172.21.2.117:0/1584145510 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f4fe0009440 con 0x7f4fdc0186c0 2021-02-10T11:56:40.338 INFO:teuthology.orchestra.run.gibba017.stdout:AQD4ySNg4ldNERAAodhBQlcUgsuW3XmpH7QMYA== 2021-02-10T11:56:40.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 msgr2=0x7f4fdc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:40.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 0x7f4fdc01abc0 crc :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:40.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 msgr2=0x7f4ff81a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:40.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 rx=0x7f4fe8006ed0 tx=0x7f4fe800b960).stop 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 shutdown_connections 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4fdc0186c0 0x7f4fdc01abc0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4ff80b5270 0x7f4ff81a32c0 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4ff81a4b70 0x7f4ff81a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 --2- 172.21.2.117:0/1584145510 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4ff81a4770 0x7f4ff80b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.338 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 >> 172.21.2.117:0/1584145510 conn(0x7f4ff80977b0 msgr2=0x7f4ff81a53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.339 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 shutdown_connections 2021-02-10T11:56:40.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:40.339 7f4ffec32700 1 -- 172.21.2.117:0/1584145510 wait complete. 2021-02-10T11:56:40.348 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo ls volumes 2021-02-10T11:56:40.698 INFO:teuthology.orchestra.run.gibba017.stdout:group_0000000000826206 2021-02-10T11:56:40.699 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000826206:subvolume_0000000000911369.meta 2021-02-10T11:56:40.699 INFO:teuthology.orchestra.run.gibba017.stdout:_group_0000000000826206:subvolume_0000000000911370.meta 2021-02-10T11:56:40.699 INFO:teuthology.orchestra.run.gibba017.stdout:$guest1.meta 2021-02-10T11:56:40.699 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:56:41.064 INFO:teuthology.orchestra.run.gibba017.stdout:{"compat_version": 1, "tenant_id": "tenant1", "version": 5, "dirty": false, "subvolumes": {"group_0000000000826206/subvolume_0000000000911369": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:56:41.065 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000911370 guest1 --group_name group_0000000000826206 --tenant_id tenant1 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.487 7f602275c700 1 Processor -- start 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.487 7f602275c700 1 -- start start 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f602275c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f601c0ef3f0 0x7f601c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f602275c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601c0ef7f0 con 0x7f601c0eb830 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f602275c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601c0ede60 con 0x7f601c0d18f0 2021-02-10T11:56:41.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f602275c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f601c0d1da0 con 0x7f601c0ef3f0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f6020d79700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f601c0eb830 0x7f601c1b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60088/0 (socket says 172.21.2.117:60088) 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f6020d79700 1 -- 172.21.2.117:0/4050931768 learned_addr learned my addr 172.21.2.117:0/4050931768 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f601b7fe700 1 --2- 172.21.2.117:0/4050931768 >> v2:172.21.2.117:3300/0 conn(0x7f601c0ef3f0 0x7f601c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.488 7f601b7fe700 1 -- 172.21.2.117:0/4050931768 >> v1:172.21.2.125:6789/0 conn(0x7f601c0d18f0 legacy=0x7f601c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601b7fe700 1 -- 172.21.2.117:0/4050931768 >> v1:172.21.2.117:6789/0 conn(0x7f601c0eb830 legacy=0x7f601c1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601affd700 1 -- 172.21.2.117:0/4050931768 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804525779 0 0) 0x7f6014001500 con 0x7f601c0eb830 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601b7fe700 1 -- 172.21.2.117:0/4050931768 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c1bb1c0 con 0x7f601c0ef3f0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601b7fe700 1 --2- 172.21.2.117:0/4050931768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c0ef3f0 0x7f601c0eb210 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f6010006650 tx=0x7f6010009a80).ready entity=mon.0 client_cookie=2e1046cd24f010ce server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601affd700 1 -- 172.21.2.117:0/4050931768 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f601000c0c0 con 0x7f601c0ef3f0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601affd700 1 -- 172.21.2.117:0/4050931768 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601000c340 con 0x7f601c0ef3f0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f601affd700 1 -- 172.21.2.117:0/4050931768 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f601000c860 con 0x7f601c0ef3f0 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f602275c700 1 -- 172.21.2.117:0/4050931768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c0ef3f0 msgr2=0x7f601c0eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.489 7f602275c700 1 --2- 172.21.2.117:0/4050931768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c0ef3f0 0x7f601c0eb210 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 rx=0x7f6010006650 tx=0x7f6010009a80).stop 2021-02-10T11:56:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.490 7f602275c700 1 -- 172.21.2.117:0/4050931768 shutdown_connections 2021-02-10T11:56:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.490 7f602275c700 1 --2- 172.21.2.117:0/4050931768 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c0ef3f0 0x7f601c0eb210 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.490 7f602275c700 1 -- 172.21.2.117:0/4050931768 >> 172.21.2.117:0/4050931768 conn(0x7f601c0cf060 msgr2=0x7f601c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.490 7f602275c700 1 -- 172.21.2.117:0/4050931768 shutdown_connections 2021-02-10T11:56:41.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.490 7f602275c700 1 -- 172.21.2.117:0/4050931768 wait complete. 2021-02-10T11:56:41.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 Processor -- start 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 -- start start 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 0x7f601c0a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 0x7f601c0a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1921b0 con 0x7f601c1918c0 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ab5a0 con 0x7f601c0a1b10 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f602275c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ac140 con 0x7f601c0a2d50 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f601bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f601bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59842/0 (socket says 172.21.2.117:59842) 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.505 7f601bfff700 1 -- 172.21.2.117:0/3136396298 learned_addr learned my addr 172.21.2.117:0/3136396298 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601b7fe700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 0x7f601c0a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:41.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 -- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 msgr2=0x7f601c0a3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 0x7f601c0a3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 -- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 msgr2=0x7f601c0a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f6020d79700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 0x7f601c0a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 0x7f601c0a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 -- 172.21.2.117:0/3136396298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c191cc0 con 0x7f601c1918c0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601b7fe700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 0x7f601c0a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f601bfff700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7f600c00b5b0 tx=0x7f600c00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f6020d79700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 0x7f601c0a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c00bd40 con 0x7f601c1918c0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600c006020 con 0x7f601c1918c0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f600c0054a0 con 0x7f601c1918c0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f602275c700 1 -- 172.21.2.117:0/3136396298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601c191cc0 con 0x7f601c1918c0 2021-02-10T11:56:41.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.506 7f602275c700 1 -- 172.21.2.117:0/3136396298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c0a2820 con 0x7f601c1918c0 2021-02-10T11:56:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.507 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f600c006200 con 0x7f601c1918c0 2021-02-10T11:56:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.507 7f6018ff9700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 0x7f5ffc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.507 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f600c032500 con 0x7f601c1918c0 2021-02-10T11:56:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.508 7f601b7fe700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 0x7f5ffc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:41.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.508 7f601b7fe700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 0x7f5ffc01abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:41.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.537 7f602275c700 1 -- 172.21.2.117:0/3136396298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c0f2f40 con 0x7f601c1918c0 2021-02-10T11:56:41.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.540 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f600c000c10 con 0x7f601c1918c0 2021-02-10T11:56:41.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.664 7f602275c700 1 -- 172.21.2.117:0/3136396298 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911370", "target": ["mgr", ""], "tenant_id": "tenant1", "auth_id": "guest1", "group_name": "group_0000000000826206"}) v1 -- 0x7f601c0a2820 con 0x7f5ffc0186c0 2021-02-10T11:56:41.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.697 7f6018ff9700 1 -- 172.21.2.117:0/3136396298 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f6010006310 con 0x7f5ffc0186c0 2021-02-10T11:56:41.730 INFO:teuthology.orchestra.run.gibba017.stdout:AQD4ySNg4ldNERAAodhBQlcUgsuW3XmpH7QMYA== 2021-02-10T11:56:41.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 msgr2=0x7f5ffc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.731 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 0x7f5ffc01abc0 crc :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 msgr2=0x7f601c1abb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 rx=0x7f600c00b5b0 tx=0x7f600c00b950).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 shutdown_connections 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f601c0a1b10 0x7f601c0a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f601c0a2d50 0x7f601c0a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ffc0186c0 0x7f5ffc01abc0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 --2- 172.21.2.117:0/3136396298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f601c1918c0 0x7f601c1abb20 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:41.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 >> 172.21.2.117:0/3136396298 conn(0x7f601c19f6d0 msgr2=0x7f601c08d990 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:41.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 shutdown_connections 2021-02-10T11:56:41.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:41.730 7f602275c700 1 -- 172.21.2.117:0/3136396298 wait complete. 2021-02-10T11:56:41.740 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.1 && sudo sed -i s/false/true/g 'volumes/$guest1.meta' 2021-02-10T11:56:42.109 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000911370 guest1 --group_name group_0000000000826206 2021-02-10T11:56:42.542 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.523 7fb05d635700 1 Processor -- start 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.523 7fb05d635700 1 -- start start 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb05d635700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 0x7fb0580e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb05d635700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 0x7fb0580e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb05d635700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0580e77f0 con 0x7fb0580e6170 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb05d635700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0580e54b0 con 0x7fb0581b07f0 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb05d635700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580e9180 con 0x7fb0580e8d80 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb056abe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb0580e6170 0x7fb0580e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60102/0 (socket says 172.21.2.117:60102) 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb057ac0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 0x7fb0580e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb056abe700 1 -- 172.21.2.117:0/2538525706 learned_addr learned my addr 172.21.2.117:0/2538525706 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.524 7fb0572bf700 1 --2- 172.21.2.117:0/2538525706 >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 0x7fb0580e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb057ac0700 1 -- 172.21.2.117:0/2538525706 >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 msgr2=0x7fb0580e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:42.543 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb057ac0700 1 --2- 172.21.2.117:0/2538525706 >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 0x7fb0580e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb057ac0700 1 -- 172.21.2.117:0/2538525706 >> v1:172.21.2.117:6789/0 conn(0x7fb0580e6170 legacy=0x7fb0580e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb057ac0700 1 -- 172.21.2.117:0/2538525706 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580e7380 con 0x7fb0580e8d80 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb0572bf700 1 --2- 172.21.2.117:0/2538525706 >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 0x7fb0580e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb057ac0700 1 --2- 172.21.2.117:0/2538525706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 0x7fb0580e6d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb050006e50 tx=0x7fb050009e70).ready entity=mon.2 client_cookie=b26191e2fdde18c8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb0562bd700 1 -- 172.21.2.117:0/2538525706 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb050006a20 con 0x7fb0580e8d80 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb0562bd700 1 -- 172.21.2.117:0/2538525706 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb050006790 con 0x7fb0580e8d80 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb0562bd700 1 -- 172.21.2.117:0/2538525706 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb05000b700 con 0x7fb0580e8d80 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb05d635700 1 -- 172.21.2.117:0/2538525706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 msgr2=0x7fb0580e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.525 7fb05d635700 1 --2- 172.21.2.117:0/2538525706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 0x7fb0580e6d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb050006e50 tx=0x7fb050009e70).stop 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 -- 172.21.2.117:0/2538525706 shutdown_connections 2021-02-10T11:56:42.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 --2- 172.21.2.117:0/2538525706 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb0580e8d80 0x7fb0580e6d60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 --2- 172.21.2.117:0/2538525706 >> v2:172.21.2.125:3300/0 conn(0x7fb0581b07f0 0x7fb0580e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 -- 172.21.2.117:0/2538525706 >> 172.21.2.117:0/2538525706 conn(0x7fb0580ce500 msgr2=0x7fb0580e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:42.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 -- 172.21.2.117:0/2538525706 shutdown_connections 2021-02-10T11:56:42.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.526 7fb05d635700 1 -- 172.21.2.117:0/2538525706 wait complete. 2021-02-10T11:56:42.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 Processor -- start 2021-02-10T11:56:42.546 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 -- start start 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb05818cee0 0x7fb0580afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0580b0050 0x7fb05810cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581a72b0 con 0x7fb05818cae0 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581085d0 con 0x7fb0580b0050 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.541 7fb05d635700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810d400 con 0x7fb05818cee0 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59858/0 (socket says 172.21.2.117:59858) 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 -- 172.21.2.117:0/1536928772 learned_addr learned my addr 172.21.2.117:0/1536928772 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 -- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb05818cee0 msgr2=0x7fb0580afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb05818cee0 0x7fb0580afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 -- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0580b0050 msgr2=0x7fb05810cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0580b0050 0x7fb05810cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 -- 172.21.2.117:0/1536928772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05810d620 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb0572bf700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fb048006f40 tx=0x7fb04800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb048006bd0 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04800bd80 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb0480022b0 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.542 7fb05d635700 1 -- 172.21.2.117:0/1536928772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05810d620 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.543 7fb05d635700 1 -- 172.21.2.117:0/1536928772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05818ae00 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.543 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb048000c90 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.543 7fb04ffff700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 0x7fb03c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.543 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb048032770 con 0x7fb05818cae0 2021-02-10T11:56:42.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.544 7fb056abe700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 0x7fb03c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:42.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.544 7fb056abe700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 0x7fb03c01abd0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:42.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.574 7fb05d635700 1 -- 172.21.2.117:0/1536928772 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb058080e20 con 0x7fb05818cae0 2021-02-10T11:56:42.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.576 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb0480009b0 con 0x7fb05818cae0 2021-02-10T11:56:42.704 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.702 7fb05d635700 1 -- 172.21.2.117:0/1536928772 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911370", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000826206"}) v1 -- 0x7fb05818ae00 con 0x7fb03c0186d0 2021-02-10T11:56:42.765 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.764 7fb04ffff700 1 -- 172.21.2.117:0/1536928772 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb040000e20 con 0x7fb03c0186d0 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.767 7fb05d635700 1 -- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 msgr2=0x7fb03c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.767 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 0x7fb03c01abd0 crc :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.767 7fb05d635700 1 -- 172.21.2.117:0/1536928772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 msgr2=0x7fb0581a6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.767 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 rx=0x7fb048006f40 tx=0x7fb04800b990).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 -- 172.21.2.117:0/1536928772 shutdown_connections 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb0580b0050 0x7fb05810cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb05818cee0 0x7fb0580afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb03c0186d0 0x7fb03c01abd0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 --2- 172.21.2.117:0/1536928772 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb05818cae0 0x7fb0581a6c90 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 -- 172.21.2.117:0/1536928772 >> 172.21.2.117:0/1536928772 conn(0x7fb05818a7e0 msgr2=0x7fb058195e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 -- 172.21.2.117:0/1536928772 shutdown_connections 2021-02-10T11:56:42.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:42.768 7fb05d635700 1 -- 172.21.2.117:0/1536928772 wait complete. 2021-02-10T11:56:42.778 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo sudo cat '/home/ubuntu/cephtest/mnt.0/volumes/$guest1.meta' 2021-02-10T11:56:43.142 INFO:teuthology.orchestra.run.gibba017.stdout:{"tenant_id": "tenant1", "version": 5, "dirty": false, "compat_version": 1, "subvolumes": {"group_0000000000826206/subvolume_0000000000911369": {"access_level": "rw", "dirty": false}}} 2021-02-10T11:56:43.143 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000911369 guest1 --group_name group_0000000000826206 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.555 7effabd40700 1 Processor -- start 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.555 7effabd40700 1 -- start start 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.555 7effabd40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41b07f0 0x7effa40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.555 7effabd40700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7effa40e7d70 0x7effa40e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effabd40700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa40e77f0 con 0x7effa40e8790 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effabd40700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa40e92f0 con 0x7effa40e7d70 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effabd40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa40e5ce0 con 0x7effa41b07f0 2021-02-10T11:56:43.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa935b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7effa40e7d70 0x7effa40e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effaa35d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7effa40e8790 0x7effa40e6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53048/0 (socket says 172.21.2.117:53048) 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effaa35d700 1 -- 172.21.2.117:0/719406437 learned_addr learned my addr 172.21.2.117:0/719406437 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa9b5c700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41b07f0 0x7effa40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa935b700 1 -- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41b07f0 msgr2=0x7effa40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa935b700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41b07f0 0x7effa40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa935b700 1 -- 172.21.2.117:0/719406437 >> v1:172.21.2.125:6789/0 conn(0x7effa40e8790 legacy=0x7effa40e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.556 7effa935b700 1 -- 172.21.2.117:0/719406437 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa40e54b0 con 0x7effa40e7d70 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effa935b700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa40e7d70 0x7effa40e8170 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7effa0006650 tx=0x7effa0009a80).ready entity=mon.1 client_cookie=c81bd53d962d7a07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effa8b5a700 1 -- 172.21.2.117:0/719406437 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa000c0c0 con 0x7effa40e7d70 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effa8b5a700 1 -- 172.21.2.117:0/719406437 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa000c340 con 0x7effa40e7d70 2021-02-10T11:56:43.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effa8b5a700 1 -- 172.21.2.117:0/719406437 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa000c860 con 0x7effa40e7d70 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 -- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa40e7d70 msgr2=0x7effa40e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa40e7d70 0x7effa40e8170 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7effa0006650 tx=0x7effa0009a80).stop 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 -- 172.21.2.117:0/719406437 shutdown_connections 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa40e7d70 0x7effa40e8170 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 --2- 172.21.2.117:0/719406437 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41b07f0 0x7effa40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.557 7effabd40700 1 -- 172.21.2.117:0/719406437 >> 172.21.2.117:0/719406437 conn(0x7effa40ce500 msgr2=0x7effa40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.558 7effabd40700 1 -- 172.21.2.117:0/719406437 shutdown_connections 2021-02-10T11:56:43.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.558 7effabd40700 1 -- 172.21.2.117:0/719406437 wait complete. 2021-02-10T11:56:43.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.572 7effabd40700 1 Processor -- start 2021-02-10T11:56:43.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.572 7effabd40700 1 -- start start 2021-02-10T11:56:43.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41a2de0 0x7effa41a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.578 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa41935a0 0x7effa41939a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa41a28e0 con 0x7effa41a3800 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa4193fc0 con 0x7effa41935a0 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effabd40700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa41941b0 con 0x7effa41a2de0 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59876/0 (socket says 172.21.2.117:59876) 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 -- 172.21.2.117:0/230543600 learned_addr learned my addr 172.21.2.117:0/230543600 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa9b5c700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41a2de0 0x7effa41a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effaa35d700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa41935a0 0x7effa41939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 -- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41a2de0 msgr2=0x7effa41a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41a2de0 0x7effa41a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 -- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa41935a0 msgr2=0x7effa41939a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa41935a0 0x7effa41939a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 -- 172.21.2.117:0/230543600 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa41943d0 con 0x7effa41a3800 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7effa935b700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7effa0009630 tx=0x7effa000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.573 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa00093a0 con 0x7effa41a3800 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa000bd70 con 0x7effa41a3800 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7effabd40700 1 -- 172.21.2.117:0/230543600 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa41943d0 con 0x7effa41a3800 2021-02-10T11:56:43.580 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7effa0005560 con 0x7effa41a3800 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7effabd40700 1 -- 172.21.2.117:0/230543600 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa41943d0 con 0x7effa41a3800 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7effa000a000 con 0x7effa41a3800 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7eff967fc700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 0x7eff8c01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.574 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7effa0032920 con 0x7effa41a3800 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.575 7effa9b5c700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 0x7eff8c01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:43.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.575 7effa9b5c700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 0x7eff8c01abd0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:43.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.605 7effabd40700 1 -- 172.21.2.117:0/230543600 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa4080e20 con 0x7effa41a3800 2021-02-10T11:56:43.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.608 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7effa001f510 con 0x7effa41a3800 2021-02-10T11:56:43.737 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.735 7effabd40700 1 -- 172.21.2.117:0/230543600 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911369", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000826206"}) v1 -- 0x7effa40e77f0 con 0x7eff8c0186d0 2021-02-10T11:56:43.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.769 7eff967fc700 1 -- 172.21.2.117:0/230543600 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7eff98006a70 con 0x7eff8c0186d0 2021-02-10T11:56:43.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 msgr2=0x7eff8c01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 0x7eff8c01abd0 crc :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 msgr2=0x7effa41a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 rx=0x7effa0009630 tx=0x7effa000b980).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 shutdown_connections 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7eff8c0186d0 0x7eff8c01abd0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7effa41a3800 0x7effa41a3c00 unknown :-1 s=CLOSED pgs=3055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7effa41935a0 0x7effa41939a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 --2- 172.21.2.117:0/230543600 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7effa41a2de0 0x7effa41a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 >> 172.21.2.117:0/230543600 conn(0x7effa41a7d10 msgr2=0x7effa4195270 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 shutdown_connections 2021-02-10T11:56:43.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:43.801 7effabd40700 1 -- 172.21.2.117:0/230543600 wait complete. 2021-02-10T11:56:43.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:43.811 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:43.811 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:44.146 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:56:44.147 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:56:44.147 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:56:44.147 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:56:44.147 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:44.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[189117]: fuse finished with error 0 and tester_r 0 2021-02-10T11:56:44.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:44.502 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:44.502 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:44.839 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:56:44.840 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:56:44.840 INFO:teuthology.orchestra.run:waiting for 900 2021-02-10T11:56:44.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:56:44.841 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:45.176 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth rm client.guest1 2021-02-10T11:56:45.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.587 7f6b0c392700 1 Processor -- start 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.587 7f6b0c392700 1 -- start start 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b0c392700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6b041b07f0 0x7f6b040e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b0c392700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b040e8d80 0x7f6b040e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b0c392700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b040e77f0 con 0x7f6b040e6170 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b0c392700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b040e54b0 con 0x7f6b041b07f0 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b0c392700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040e9180 con 0x7f6b040e8d80 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b099ad700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6b040e6170 0x7f6b040e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60132/0 (socket says 172.21.2.117:60132) 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.588 7f6b099ad700 1 -- 172.21.2.117:0/4040224200 learned_addr learned my addr 172.21.2.117:0/4040224200 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b0a9af700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b040e8d80 0x7f6b040e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b0a1ae700 1 --2- 172.21.2.117:0/4040224200 >> v2:172.21.2.125:3300/0 conn(0x7f6b041b07f0 0x7f6b040e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628246867 0 0) 0x7f6b00001430 con 0x7f6b040e6170 2021-02-10T11:56:45.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af0002fe0 con 0x7f6b040e6170 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2328818993 0 0) 0x7f6b00001430 con 0x7f6b040e6170 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b040e8d80 msgr2=0x7f6b040e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b040e8d80 0x7f6b040e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 >> v2:172.21.2.125:3300/0 conn(0x7f6b041b07f0 msgr2=0x7f6b040e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 --2- 172.21.2.117:0/4040224200 >> v2:172.21.2.125:3300/0 conn(0x7f6b041b07f0 0x7f6b040e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040e7380 con 0x7f6b040e6170 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f6b00001930 con 0x7f6b040e6170 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 >> v1:172.21.2.117:6789/0 conn(0x7f6b040e6170 legacy=0x7f6b040e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6af0003a60 0x7f6af0005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6af004a7a0 0x7f6af004cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6af004d510 0x7f6af004fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00502e0 con 0x7f6af004d510 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af0050500 con 0x7f6af0003a60 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.589 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af0050720 con 0x7f6af004a7a0 2021-02-10T11:56:45.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a1ae700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6af0003a60 0x7f6af0005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6af004d510 0x7f6af004fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b099ad700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6af004a7a0 0x7f6af004cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 -- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6af004a7a0 msgr2=0x7f6af004cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6af004a7a0 0x7f6af004cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 -- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6af0003a60 msgr2=0x7f6af0005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6af0003a60 0x7f6af0005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 -- 172.21.2.117:0/4040224200 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00509c0 con 0x7f6af004d510 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6af004d510 0x7f6af004fa30 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7f6afc006af0 tx=0x7f6afc009ad0).ready entity=mon.0 client_cookie=5904d243b2fef591 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6afc0066c0 con 0x7f6af004d510 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc0022e0 con 0x7f6af004d510 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b091ac700 1 -- 172.21.2.117:0/4040224200 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6afc002730 con 0x7f6af004d510 2021-02-10T11:56:45.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0c392700 1 -- 172.21.2.117:0/4040224200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6af004d510 msgr2=0x7f6af004fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0c392700 1 --2- 172.21.2.117:0/4040224200 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6af004d510 0x7f6af004fa30 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 rx=0x7f6afc006af0 tx=0x7f6afc009ad0).stop 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.590 7f6b0a9af700 1 -- 172.21.2.117:0/4040224200 reap_dead start 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.591 7f6b0c392700 1 -- 172.21.2.117:0/4040224200 shutdown_connections 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.591 7f6b0c392700 1 -- 172.21.2.117:0/4040224200 >> 172.21.2.117:0/4040224200 conn(0x7f6b040ce500 msgr2=0x7f6b040e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.591 7f6b0c392700 1 -- 172.21.2.117:0/4040224200 shutdown_connections 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.591 7f6b0c392700 1 -- 172.21.2.117:0/4040224200 wait complete. 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 Processor -- start 2021-02-10T11:56:45.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 -- start start 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b0410fd90 0x7f6b04110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 0x7f6b040bd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040bd950 con 0x7f6b04091d90 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040bded0 con 0x7f6b040bce00 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0c392700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040b21e0 con 0x7f6b0410fd90 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0a9af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0a9af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59894/0 (socket says 172.21.2.117:59894) 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.606 7f6b0a9af700 1 -- 172.21.2.117:0/722258428 learned_addr learned my addr 172.21.2.117:0/722258428 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b099ad700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 0x7f6b040bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 -- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b0410fd90 msgr2=0x7f6b04110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b0410fd90 0x7f6b04110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 -- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 msgr2=0x7f6b040bd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 0x7f6b040bd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 -- 172.21.2.117:0/722258428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040927b0 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b099ad700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 0x7f6b040bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0a9af700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7f6afc0096a0 tx=0x7f6afc00b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6afc000a70 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc002030 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0c392700 1 -- 172.21.2.117:0/722258428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b040927b0 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6afc002400 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.607 7f6b0c392700 1 -- 172.21.2.117:0/722258428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b04092d40 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.608 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6afc0067d0 con 0x7f6b04091d90 2021-02-10T11:56:45.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.608 7f6af6ffd700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 0x7f6aec01abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:45.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.608 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6afc032920 con 0x7f6b04091d90 2021-02-10T11:56:45.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.609 7f6b0a1ae700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 0x7f6aec01abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:45.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.609 7f6b0a1ae700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 0x7f6aec01abf0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:45.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.638 7f6b0c392700 1 -- 172.21.2.117:0/722258428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0410c1b0 con 0x7f6b04091d90 2021-02-10T11:56:45.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.641 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6afc0065c0 con 0x7f6b04091d90 2021-02-10T11:56:45.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.770 7f6b0c392700 1 -- 172.21.2.117:0/722258428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.guest1"} v 0) v1 -- 0x7f6b040e77f0 con 0x7f6b04091d90 2021-02-10T11:56:45.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.771 7f6af6ffd700 1 -- 172.21.2.117:0/722258428 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.guest1"}]=0 entity client.guest1 does not exist v107) v1 ==== 117+0+0 (secure 0 0 0) 0x7f6afc0067d0 con 0x7f6b04091d90 2021-02-10T11:56:45.774 INFO:teuthology.orchestra.run.gibba017.stderr:entity client.guest1 does not exist 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 msgr2=0x7f6aec01abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 0x7f6aec01abf0 crc :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 msgr2=0x7f6b04092190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 rx=0x7f6afc0096a0 tx=0x7f6afc00b9a0).stop 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 shutdown_connections 2021-02-10T11:56:45.776 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6aec018730 0x7f6aec01abf0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b04091d90 0x7f6b04092190 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b040bce00 0x7f6b040bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 --2- 172.21.2.117:0/722258428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b0410fd90 0x7f6b04110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 >> 172.21.2.117:0/722258428 conn(0x7f6b0408a780 msgr2=0x7f6b040c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:45.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 shutdown_connections 2021-02-10T11:56:45.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:45.773 7f6b0c392700 1 -- 172.21.2.117:0/722258428 wait complete. 2021-02-10T11:56:45.784 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000911369 --group_name group_0000000000826206 2021-02-10T11:56:46.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.199 7f84c91cf700 1 Processor -- start 2021-02-10T11:56:46.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.199 7f84c91cf700 1 -- start start 2021-02-10T11:56:46.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c91cf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c4085450 0x7f84c4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c91cf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c4085930 con 0x7f84c40e5240 2021-02-10T11:56:46.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c91cf700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c41b09e0 con 0x7f84c40e4e40 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c91cf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40e5cc0 con 0x7f84c4085450 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c259c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84c40e5240 0x7f84c4084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60150/0 (socket says 172.21.2.117:60150) 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c259c700 1 -- 172.21.2.117:0/2712564701 learned_addr learned my addr 172.21.2.117:0/2712564701 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703601365 0 0) 0x7f84b4000ce0 con 0x7f84c40e5240 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.200 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b0002fe0 con 0x7f84c40e5240 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c359e700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c4085450 0x7f84c4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 641868667 0 0) 0x7f84b40010e0 con 0x7f84c40e5240 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c4085450 msgr2=0x7f84c4086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c4085450 0x7f84c4086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 >> v1:172.21.2.125:6789/0 conn(0x7f84c40e4e40 legacy=0x7f84c41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4084840 con 0x7f84c40e5240 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f84b4001660 con 0x7f84c40e5240 2021-02-10T11:56:46.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 >> v1:172.21.2.117:6789/0 conn(0x7f84c40e5240 legacy=0x7f84c4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84b0003a70 0x7f84b0005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84b0006790 0x7f84b0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84b0009590 0x7f84b000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b000c2d0 con 0x7f84b0006790 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b000c4f0 con 0x7f84b0003a70 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b000c710 con 0x7f84b0009590 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84b0006790 0x7f84b0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c2d9d700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84b0003a70 0x7f84b0005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c359e700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84b0009590 0x7f84b000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 -- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84b0009590 msgr2=0x7f84b000ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84b0009590 0x7f84b000ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 -- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84b0003a70 msgr2=0x7f84b0005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84b0003a70 0x7f84b0005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c259c700 1 -- 172.21.2.117:0/2712564701 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b000c9b0 con 0x7f84b0006790 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.201 7f84c2d9d700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84b0003a70 0x7f84b0005f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c259c700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84b0006790 0x7f84b0008cb0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f84b4000eb0 tx=0x7f84b4007cf0).ready entity=mon.0 client_cookie=19b18c28c80bdadc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b40014d0 con 0x7f84b0006790 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b40070c0 con 0x7f84b0006790 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c1d9b700 1 -- 172.21.2.117:0/2712564701 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b4004590 con 0x7f84b0006790 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c91cf700 1 -- 172.21.2.117:0/2712564701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84b0006790 msgr2=0x7f84b0008cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c91cf700 1 --2- 172.21.2.117:0/2712564701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84b0006790 0x7f84b0008cb0 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 rx=0x7f84b4000eb0 tx=0x7f84b4007cf0).stop 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.202 7f84c359e700 1 -- 172.21.2.117:0/2712564701 reap_dead start 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.203 7f84c91cf700 1 -- 172.21.2.117:0/2712564701 shutdown_connections 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.203 7f84c91cf700 1 -- 172.21.2.117:0/2712564701 >> 172.21.2.117:0/2712564701 conn(0x7f84c40ce4e0 msgr2=0x7f84c40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.203 7f84c91cf700 1 -- 172.21.2.117:0/2712564701 shutdown_connections 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.203 7f84c91cf700 1 -- 172.21.2.117:0/2712564701 wait complete. 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 Processor -- start 2021-02-10T11:56:46.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 -- start start 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409cfa0 0x7f84c40c26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c41a2de0 0x7f84c41a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40c2170 con 0x7f84c40a7550 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40a7f70 con 0x7f84c409cfa0 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.218 7f84c91cf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c409d3a0 con 0x7f84c41a2de0 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59912/0 (socket says 172.21.2.117:59912) 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 -- 172.21.2.117:0/2830224002 learned_addr learned my addr 172.21.2.117:0/2830224002 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c2d9d700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409cfa0 0x7f84c40c26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c359e700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c41a2de0 0x7f84c41a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 -- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c41a2de0 msgr2=0x7f84c41a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c41a2de0 0x7f84c41a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 -- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409cfa0 msgr2=0x7f84c40c26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409cfa0 0x7f84c40c26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 -- 172.21.2.117:0/2830224002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c41a3800 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c259c700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f84b4001cd0 tx=0x7f84b404f950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b4007600 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b404fdc0 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84b40043a0 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.219 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c41a3800 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.220 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c41a3800 con 0x7f84c40a7550 2021-02-10T11:56:46.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.220 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f84b404e030 con 0x7f84c40a7550 2021-02-10T11:56:46.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.220 7f84bb7fe700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.220 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f84b4076510 con 0x7f84c40a7550 2021-02-10T11:56:46.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.220 7f84c2d9d700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.221 7f84c2d9d700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.250 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c410c2f0 con 0x7f84c40a7550 2021-02-10T11:56:46.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.252 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84b4001690 con 0x7f84c40a7550 2021-02-10T11:56:46.381 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.379 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911369", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7f84c4085930 con 0x7f84a80186c0 2021-02-10T11:56:46.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.390 7f84bb7fe700 1 -- 172.21.2.117:0/2830224002 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f84ac006950 con 0x7f84a80186c0 2021-02-10T11:56:46.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.396 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 msgr2=0x7f84a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.396 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 crc :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 msgr2=0x7f84c40a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 rx=0x7f84b4001cd0 tx=0x7f84b404f950).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 shutdown_connections 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84a80186c0 0x7f84a801abc0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84c40a7550 0x7f84c40a7950 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84c41a2de0 0x7f84c41a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 --2- 172.21.2.117:0/2830224002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84c409cfa0 0x7f84c40c26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 >> 172.21.2.117:0/2830224002 conn(0x7f84c40b4890 msgr2=0x7f84c40b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:46.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 shutdown_connections 2021-02-10T11:56:46.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.397 7f84c91cf700 1 -- 172.21.2.117:0/2830224002 wait complete. 2021-02-10T11:56:46.407 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000911370 --group_name group_0000000000826206 2021-02-10T11:56:46.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.824 7fa9c7b49700 1 Processor -- start 2021-02-10T11:56:46.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.824 7fa9c7b49700 1 -- start start 2021-02-10T11:56:46.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c7b49700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 0x7fa9c01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c7b49700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 0x7fa9c0084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c7b49700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0085930 con 0x7fa9c01b0810 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c7b49700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c0087430 con 0x7fa9c00e5240 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c7b49700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c00e5cc0 con 0x7fa9c00e4e40 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c6166700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fa9c01b0810 0x7fa9c0085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53094/0 (socket says 172.21.2.117:53094) 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c6166700 1 -- 172.21.2.117:0/1850489187 learned_addr learned my addr 172.21.2.117:0/1850489187 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c5965700 1 --2- 172.21.2.117:0/1850489187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 0x7fa9c01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.825 7fa9c5164700 1 --2- 172.21.2.117:0/1850489187 >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 0x7fa9c0084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5965700 1 -- 172.21.2.117:0/1850489187 >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 msgr2=0x7fa9c0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5965700 1 --2- 172.21.2.117:0/1850489187 >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 0x7fa9c0084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5965700 1 -- 172.21.2.117:0/1850489187 >> v1:172.21.2.125:6789/0 conn(0x7fa9c01b0810 legacy=0x7fa9c0085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5965700 1 -- 172.21.2.117:0/1850489187 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0084840 con 0x7fa9c00e4e40 2021-02-10T11:56:46.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5164700 1 --2- 172.21.2.117:0/1850489187 >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 0x7fa9c0084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c5965700 1 --2- 172.21.2.117:0/1850489187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 0x7fa9c01b01f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa9bc006650 tx=0x7fa9bc009a80).ready entity=mon.2 client_cookie=d02aebc523115bf6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c4963700 1 -- 172.21.2.117:0/1850489187 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc00c0c0 con 0x7fa9c00e4e40 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c4963700 1 -- 172.21.2.117:0/1850489187 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc00c340 con 0x7fa9c00e4e40 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.826 7fa9c4963700 1 -- 172.21.2.117:0/1850489187 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc00c860 con 0x7fa9c00e4e40 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 -- 172.21.2.117:0/1850489187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 msgr2=0x7fa9c01b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 --2- 172.21.2.117:0/1850489187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 0x7fa9c01b01f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa9bc006650 tx=0x7fa9bc009a80).stop 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 -- 172.21.2.117:0/1850489187 shutdown_connections 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 --2- 172.21.2.117:0/1850489187 >> v2:172.21.2.125:3300/0 conn(0x7fa9c00e5240 0x7fa9c0084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 --2- 172.21.2.117:0/1850489187 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00e4e40 0x7fa9c01b01f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 -- 172.21.2.117:0/1850489187 >> 172.21.2.117:0/1850489187 conn(0x7fa9c00ce4e0 msgr2=0x7fa9c00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 -- 172.21.2.117:0/1850489187 shutdown_connections 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.827 7fa9c7b49700 1 -- 172.21.2.117:0/1850489187 wait complete. 2021-02-10T11:56:46.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.841 7fa9c7b49700 1 Processor -- start 2021-02-10T11:56:46.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 -- start start 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9c00ad1b0 0x7fa9c00ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00adbd0 0x7fa9c009b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c018d0c0 con 0x7fa9c00973b0 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c009ae70 con 0x7fa9c00ad1b0 2021-02-10T11:56:46.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c7b49700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c009ba10 con 0x7fa9c00adbd0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c5965700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c5965700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59926/0 (socket says 172.21.2.117:59926) 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c5965700 1 -- 172.21.2.117:0/59013409 learned_addr learned my addr 172.21.2.117:0/59013409 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c5164700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9c00ad1b0 0x7fa9c00ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.842 7fa9c6166700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00adbd0 0x7fa9c009b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 -- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00adbd0 msgr2=0x7fa9c009b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00adbd0 0x7fa9c009b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 -- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9c00ad1b0 msgr2=0x7fa9c00ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9c00ad1b0 0x7fa9c00ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 -- 172.21.2.117:0/59013409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c009bbf0 con 0x7fa9c00973b0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c5965700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 rx=0x7fa9bc009630 tx=0x7fa9bc00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc0093a0 con 0x7fa9c00973b0 2021-02-10T11:56:46.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc00bcf0 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c009bbf0 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9bc005560 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.843 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c009bbf0 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.844 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa9bc008240 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.844 7fa9b67fc700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 0x7fa9a801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.844 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa9bc032750 con 0x7fa9c00973b0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.844 7fa9c5164700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 0x7fa9a801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:46.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.845 7fa9c5164700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 0x7fa9a801abd0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:46.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.875 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c009c4f0 con 0x7fa9c00973b0 2021-02-10T11:56:46.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:46.877 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa9bc008030 con 0x7fa9c00973b0 2021-02-10T11:56:47.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.002 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000911370", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7fa9c009c4f0 con 0x7fa9a80186d0 2021-02-10T11:56:47.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.012 7fa9b67fc700 1 -- 172.21.2.117:0/59013409 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa9b0006a30 con 0x7fa9a80186d0 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.020 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 msgr2=0x7fa9a801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.020 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 0x7fa9a801abd0 crc :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.020 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 msgr2=0x7fa9c0097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.020 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 rx=0x7fa9bc009630 tx=0x7fa9bc00b900).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 shutdown_connections 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa9c00adbd0 0x7fa9c009b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9c00ad1b0 0x7fa9c00ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9a80186d0 0x7fa9a801abd0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 --2- 172.21.2.117:0/59013409 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9c00973b0 0x7fa9c0097980 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 >> 172.21.2.117:0/59013409 conn(0x7fa9c018cc70 msgr2=0x7fa9c0093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:47.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 shutdown_connections 2021-02-10T11:56:47.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.021 7fa9c7b49700 1 -- 172.21.2.117:0/59013409 wait complete. 2021-02-10T11:56:47.031 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000826206 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.445 7f9262286700 1 Processor -- start 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.445 7f9262286700 1 -- start start 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f9262286700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f925c1b07f0 0x7f925c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f9262286700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c0e8d80 0x7f925c0e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f9262286700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0e77f0 con 0x7f925c0e6170 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f9262286700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c0e54b0 con 0x7f925c1b07f0 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f9262286700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0e9180 con 0x7f925c0e8d80 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f925b7fe700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f925c0e6170 0x7f925c0e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53106/0 (socket says 172.21.2.117:53106) 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f925b7fe700 1 -- 172.21.2.117:0/1273429182 learned_addr learned my addr 172.21.2.117:0/1273429182 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f925bfff700 1 --2- 172.21.2.117:0/1273429182 >> v2:172.21.2.125:3300/0 conn(0x7f925c1b07f0 0x7f925c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.446 7f92608a3700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c0e8d80 0x7f925c0e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925bfff700 1 -- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c0e8d80 msgr2=0x7f925c0e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925bfff700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c0e8d80 0x7f925c0e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925bfff700 1 -- 172.21.2.117:0/1273429182 >> v1:172.21.2.125:6789/0 conn(0x7f925c0e6170 legacy=0x7f925c0e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925bfff700 1 -- 172.21.2.117:0/1273429182 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c0e7380 con 0x7f925c1b07f0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925bfff700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c1b07f0 0x7f925c0e4e90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f924c006650 tx=0x7f924c009a00).ready entity=mon.1 client_cookie=987b56de311d5fc5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925affd700 1 -- 172.21.2.117:0/1273429182 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c0009b0 con 0x7f925c1b07f0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925affd700 1 -- 172.21.2.117:0/1273429182 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c00c0c0 con 0x7f925c1b07f0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f925affd700 1 -- 172.21.2.117:0/1273429182 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f924c00c5a0 con 0x7f925c1b07f0 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f9262286700 1 -- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c1b07f0 msgr2=0x7f925c0e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.447 7f9262286700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c1b07f0 0x7f925c0e4e90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7f924c006650 tx=0x7f924c009a00).stop 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 -- 172.21.2.117:0/1273429182 shutdown_connections 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c0e8d80 0x7f925c0e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 --2- 172.21.2.117:0/1273429182 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c1b07f0 0x7f925c0e4e90 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 -- 172.21.2.117:0/1273429182 >> 172.21.2.117:0/1273429182 conn(0x7f925c0ce500 msgr2=0x7f925c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 -- 172.21.2.117:0/1273429182 shutdown_connections 2021-02-10T11:56:47.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.448 7f9262286700 1 -- 172.21.2.117:0/1273429182 wait complete. 2021-02-10T11:56:47.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.462 7f9262286700 1 Processor -- start 2021-02-10T11:56:47.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.462 7f9262286700 1 -- start start 2021-02-10T11:56:47.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c08f410 0x7f925c0ba290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 0x7f925c093f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 0x7f925c0bb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c08f810 con 0x7f925c0bad40 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0935b0 con 0x7f925c08f410 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f9262286700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0ba8b0 con 0x7f925c093b30 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f92608a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 0x7f925c0bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 0x7f925c093f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f92608a3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 0x7f925c0bb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59934/0 (socket says 172.21.2.117:59934) 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f92608a3700 1 -- 172.21.2.117:0/1721453798 learned_addr learned my addr 172.21.2.117:0/1721453798 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925bfff700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c08f410 0x7f925c0ba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 -- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c08f410 msgr2=0x7f925c0ba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c08f410 0x7f925c0ba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 -- 172.21.2.117:0/1721453798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 msgr2=0x7f925c0bb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 0x7f925c0bb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 -- 172.21.2.117:0/1721453798 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c1a6bc0 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.463 7f925b7fe700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 0x7f925c093f30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f925000a4c0 tx=0x7f925000a860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.464 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92500010b0 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.464 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250004350 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.464 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9250004830 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.464 7f9262286700 1 -- 172.21.2.117:0/1721453798 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c1a6bc0 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.464 7f9262286700 1 -- 172.21.2.117:0/1721453798 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1a7150 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.465 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9250001090 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.465 7f9258ff9700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.465 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9250032550 con 0x7f925c093b30 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.465 7f925bfff700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:47.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.465 7f925bfff700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:47.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.496 7f9262286700 1 -- 172.21.2.117:0/1721453798 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c08f810 con 0x7f925c093b30 2021-02-10T11:56:47.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.498 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9250030bb0 con 0x7f925c093b30 2021-02-10T11:56:47.627 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.625 7f9262286700 1 -- 172.21.2.117:0/1721453798 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000826206"}) v1 -- 0x7f925c1a79e0 con 0x7f923c0186c0 2021-02-10T11:56:47.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.629 7f9258ff9700 1 -- 172.21.2.117:0/1721453798 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f924c006450 con 0x7f923c0186c0 2021-02-10T11:56:47.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.634 7f9262286700 1 -- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 msgr2=0x7f923c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.634 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 crc :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.634 7f9262286700 1 -- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 msgr2=0x7f925c093f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:47.636 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.634 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 0x7f925c093f30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f925000a4c0 tx=0x7f925000a860).stop 2021-02-10T11:56:47.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 -- 172.21.2.117:0/1721453798 shutdown_connections 2021-02-10T11:56:47.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f923c0186c0 0x7f923c01abc0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f925c0bad40 0x7f925c0bb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f925c093b30 0x7f925c093f30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.637 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 --2- 172.21.2.117:0/1721453798 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f925c08f410 0x7f925c0ba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:47.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 -- 172.21.2.117:0/1721453798 >> 172.21.2.117:0/1721453798 conn(0x7f925c08fb00 msgr2=0x7f925c08fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:47.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 -- 172.21.2.117:0/1721453798 shutdown_connections 2021-02-10T11:56:47.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:47.635 7f9262286700 1 -- 172.21.2.117:0/1721453798 wait complete. 2021-02-10T11:56:47.644 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:56:47.645 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:56:47.990 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:56:48.337 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:56:48.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:48.338 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:48.338 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:48.572 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:56:48.575 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:56:48.673 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:56:48.673 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:56:48.674 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:56:48.674 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:56:48.674 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:49.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[188477]: fuse finished with error 0 and tester_r 0 2021-02-10T11:56:49.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:49.039 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:49.040 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:49.374 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:56:49.375 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:56:49.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:56:49.375 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:49.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:49.710 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:49.710 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:50.047 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:56:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:50.048 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:56:50.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:56:50.048 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:50.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:50.383 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:50.383 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:56:50.717 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:56:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:50.718 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:56:50.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:56:50.718 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:56:51.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:51.053 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:51.053 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:56:51.387 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:56:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:51.387 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:56:51.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:56:51.388 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:56:51.723 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:56:52.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 Processor -- start 2021-02-10T11:56:52.156 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 -- start start 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 0x7f28300e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f28300e7180 0x7f28300e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300e7580 con 0x7f28300e7d70 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28300e92f0 con 0x7f28300e7180 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.136 7f2835a2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28300e5ce0 con 0x7f28301b07f0 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f28300e7180 0x7f28300e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 0x7f28300e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282effd700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f28300e7d70 0x7f28300e8170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53118/0 (socket says 172.21.2.117:53118) 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282effd700 1 -- 172.21.2.117:0/3719618966 learned_addr learned my addr 172.21.2.117:0/3719618966 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:52.157 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 -- 172.21.2.117:0/3719618966 >> v2:172.21.2.125:3300/0 conn(0x7f28300e7180 msgr2=0x7f28300e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 --2- 172.21.2.117:0/3719618966 >> v2:172.21.2.125:3300/0 conn(0x7f28300e7180 0x7f28300e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 -- 172.21.2.117:0/3719618966 >> v1:172.21.2.125:6789/0 conn(0x7f28300e7d70 legacy=0x7f28300e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 -- 172.21.2.117:0/3719618966 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28300e77f0 con 0x7f28301b07f0 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282f7fe700 1 --2- 172.21.2.117:0/3719618966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 0x7f28300e4e90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2818006650 tx=0x7f2818009a80).ready entity=mon.2 client_cookie=6cf2633e06db21eb server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.138 7f282e7fc700 1 -- 172.21.2.117:0/3719618966 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f281800c0c0 con 0x7f28301b07f0 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f282e7fc700 1 -- 172.21.2.117:0/3719618966 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281800c340 con 0x7f28301b07f0 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f282e7fc700 1 -- 172.21.2.117:0/3719618966 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f281800c860 con 0x7f28301b07f0 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 -- 172.21.2.117:0/3719618966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 msgr2=0x7f28300e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 --2- 172.21.2.117:0/3719618966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 0x7f28300e4e90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2818006650 tx=0x7f2818009a80).stop 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 -- 172.21.2.117:0/3719618966 shutdown_connections 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 --2- 172.21.2.117:0/3719618966 >> v2:172.21.2.125:3300/0 conn(0x7f28300e7180 0x7f28300e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 --2- 172.21.2.117:0/3719618966 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28301b07f0 0x7f28300e4e90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 -- 172.21.2.117:0/3719618966 >> 172.21.2.117:0/3719618966 conn(0x7f28300ce500 msgr2=0x7f28300e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:52.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 -- 172.21.2.117:0/3719618966 shutdown_connections 2021-02-10T11:56:52.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.139 7f2835a2c700 1 -- 172.21.2.117:0/3719618966 wait complete. 2021-02-10T11:56:52.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 Processor -- start 2021-02-10T11:56:52.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 -- start start 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2830110ae0 0x7f283009b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2830110ee0 0x7f2830187d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283009b1a0 con 0x7f28300966e0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830096ae0 con 0x7f2830110ae0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.154 7f2835a2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283009bd40 con 0x7f2830110ee0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59944/0 (socket says 172.21.2.117:59944) 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 -- 172.21.2.117:0/4204663662 learned_addr learned my addr 172.21.2.117:0/4204663662 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282effd700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2830110ae0 0x7f283009b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282ffff700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2830110ee0 0x7f2830187d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 -- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2830110ee0 msgr2=0x7f2830187d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2830110ee0 0x7f2830187d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 -- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2830110ae0 msgr2=0x7f283009b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2830110ae0 0x7f283009b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 -- 172.21.2.117:0/4204663662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28301a28c0 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f282f7fe700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7f2818009630 tx=0x7f281800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28180093a0 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281800bd70 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f28180054d0 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.155 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28301a28c0 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.156 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301a28c0 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.156 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2818000c90 con 0x7f28300966e0 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.156 7f2827fff700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 0x7f281401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.156 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2818032520 con 0x7f28300966e0 2021-02-10T11:56:52.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.157 7f282effd700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 0x7f281401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.157 7f282effd700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 0x7f281401abc0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.186 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283010c2c0 con 0x7f28300966e0 2021-02-10T11:56:52.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.189 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2818030d00 con 0x7f28300966e0 2021-02-10T11:56:52.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.337 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f28300e77f0 con 0x7f28300966e0 2021-02-10T11:56:52.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.341 7f2827fff700 1 -- 172.21.2.117:0/4204663662 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2818000c90 con 0x7f28300966e0 2021-02-10T11:56:52.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 msgr2=0x7f281401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 0x7f281401abc0 crc :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 msgr2=0x7f283009a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 rx=0x7f2818009630 tx=0x7f281800b980).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 shutdown_connections 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2830110ee0 0x7f2830187d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2830110ae0 0x7f283009b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28140186c0 0x7f281401abc0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 --2- 172.21.2.117:0/4204663662 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28300966e0 0x7f283009a9c0 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.346 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 >> 172.21.2.117:0/4204663662 conn(0x7f2830097720 msgr2=0x7f28301994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.347 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 shutdown_connections 2021-02-10T11:56:52.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.347 7f2835a2c700 1 -- 172.21.2.117:0/4204663662 wait complete. 2021-02-10T11:56:52.358 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize' 2021-02-10T11:56:52.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 Processor -- start 2021-02-10T11:56:52.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 -- start start 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8580d18e0 0x7ff8580ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580eb000 0x7ff8580eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8580ede60 con 0x7ff8580ee3e0 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8580eba90 con 0x7ff8580d18e0 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.768 7ff8602e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580d1ce0 con 0x7ff8580eb000 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85e8ff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8580ee3e0 0x7ff8580ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60202/0 (socket says 172.21.2.117:60202) 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85e8ff700 1 -- 172.21.2.117:0/2355871932 learned_addr learned my addr 172.21.2.117:0/2355871932 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991173885 0 0) 0x7ff850001500 con 0x7ff8580ee3e0 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85e0fe700 1 --2- 172.21.2.117:0/2355871932 >> v2:172.21.2.125:3300/0 conn(0x7ff8580d18e0 0x7ff8580ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d8fd700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580eb000 0x7ff8580eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff844002fe0 con 0x7ff8580ee3e0 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2305301505 0 0) 0x7ff850001500 con 0x7ff8580ee3e0 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580eb000 msgr2=0x7ff8580eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580eb000 0x7ff8580eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 >> v2:172.21.2.125:3300/0 conn(0x7ff8580d18e0 msgr2=0x7ff8580ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 --2- 172.21.2.117:0/2355871932 >> v2:172.21.2.125:3300/0 conn(0x7ff8580d18e0 0x7ff8580ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580eefd0 con 0x7ff8580ee3e0 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d8fd700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580eb000 0x7ff8580eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff850001960 con 0x7ff8580ee3e0 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.769 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 >> v1:172.21.2.117:6789/0 conn(0x7ff8580ee3e0 legacy=0x7ff8580ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff844003a60 0x7ff844005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff84404a7a0 0x7ff84404cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff84404d510 0x7ff84404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440502e0 con 0x7ff84404d510 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff844050500 con 0x7ff844003a60 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff844050720 con 0x7ff84404a7a0 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff84404d510 0x7ff84404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e0fe700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff844003a60 0x7ff844005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d8fd700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff84404a7a0 0x7ff84404cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 -- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff84404a7a0 msgr2=0x7ff84404cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff84404a7a0 0x7ff84404cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 -- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff844003a60 msgr2=0x7ff844005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff844003a60 0x7ff844005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 -- 172.21.2.117:0/2355871932 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440509c0 con 0x7ff84404d510 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff84404d510 0x7ff84404fa30 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7ff850001520 tx=0x7ff85000a310).ready entity=mon.0 client_cookie=b2c5c6f8e1ffb38b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff850007020 con 0x7ff84404d510 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff850001e80 con 0x7ff84404d510 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85d0fc700 1 -- 172.21.2.117:0/2355871932 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8500075e0 con 0x7ff84404d510 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff8602e2700 1 -- 172.21.2.117:0/2355871932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff84404d510 msgr2=0x7ff84404fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff8602e2700 1 --2- 172.21.2.117:0/2355871932 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff84404d510 0x7ff84404fa30 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 rx=0x7ff850001520 tx=0x7ff85000a310).stop 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.770 7ff85e8ff700 1 -- 172.21.2.117:0/2355871932 reap_dead start 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.771 7ff8602e2700 1 -- 172.21.2.117:0/2355871932 shutdown_connections 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.771 7ff8602e2700 1 -- 172.21.2.117:0/2355871932 >> 172.21.2.117:0/2355871932 conn(0x7ff8580ced60 msgr2=0x7ff8580cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.771 7ff8602e2700 1 -- 172.21.2.117:0/2355871932 shutdown_connections 2021-02-10T11:56:52.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.771 7ff8602e2700 1 -- 172.21.2.117:0/2355871932 wait complete. 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 Processor -- start 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 -- start start 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8580b59a0 0x7ff85808c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580b5da0 0x7ff8581a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580b61a0 con 0x7ff8581a3480 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85808c8b0 con 0x7ff8580b59a0 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff8602e2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580ba1c0 con 0x7ff8580b5da0 2021-02-10T11:56:52.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff85e8ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff85e8ff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59966/0 (socket says 172.21.2.117:59966) 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.786 7ff85e8ff700 1 -- 172.21.2.117:0/1445347098 learned_addr learned my addr 172.21.2.117:0/1445347098 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 -- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580b5da0 msgr2=0x7ff8581a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580b5da0 0x7ff8581a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 -- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8580b59a0 msgr2=0x7ff85808c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8580b59a0 0x7ff85808c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 -- 172.21.2.117:0/1445347098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8581a3880 con 0x7ff8581a3480 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff85e8ff700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7ff850007f00 tx=0x7ff85000ac80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff850004930 con 0x7ff8581a3480 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff850004c70 con 0x7ff8581a3480 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff850007350 con 0x7ff8581a3480 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8581a3880 con 0x7ff8581a3480 2021-02-10T11:56:52.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.787 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8580b2bb0 con 0x7ff8581a3480 2021-02-10T11:56:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.788 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff850004720 con 0x7ff8581a3480 2021-02-10T11:56:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.788 7ff84affd700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 0x7ff84001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.788 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8500329d0 con 0x7ff8581a3480 2021-02-10T11:56:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.788 7ff85e0fe700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 0x7ff84001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:52.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.789 7ff85e0fe700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 0x7ff84001abc0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:52.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.818 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8580ede60 con 0x7ff8581a3480 2021-02-10T11:56:52.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.821 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff850001270 con 0x7ff8581a3480 2021-02-10T11:56:52.955 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:52.951 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]} v 0) v1 -- 0x7ff8580ede60 con 0x7ff8581a3480 2021-02-10T11:56:53.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.698 7ff84affd700 1 -- 172.21.2.117:0/1445347098 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_recover_auth_metadata_during_deauthorize"]}]=0 v875) v1 ==== 162+0+0 (secure 0 0 0) 0x7ff850004720 con 0x7ff8581a3480 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 msgr2=0x7ff84001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 0x7ff84001abc0 crc :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 msgr2=0x7ff8580b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 rx=0x7ff850007f00 tx=0x7ff85000ac80).stop 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 shutdown_connections 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8400186c0 0x7ff84001abc0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:53.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8581a3480 0x7ff8580b2590 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:53.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8580b5da0 0x7ff8581a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:53.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 --2- 172.21.2.117:0/1445347098 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8580b59a0 0x7ff85808c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:53.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 >> 172.21.2.117:0/1445347098 conn(0x7ff8580bb5e0 msgr2=0x7ff858090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:53.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 shutdown_connections 2021-02-10T11:56:53.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:53.722 7ff8602e2700 1 -- 172.21.2.117:0/1445347098 wait complete. 2021-02-10T11:56:53.733 INFO:tasks.cephfs_test_runner:test_recover_auth_metadata_during_deauthorize (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:56:53.733 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:56:53.734 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:56:54.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.143 7ff86b977700 1 Processor -- start 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.143 7ff86b977700 1 -- start start 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff86b977700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8641b6960 0x7ff8641b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff86b977700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640ede60 con 0x7ff8640ef3f0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff86b977700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8640eb9c0 con 0x7ff8640d18e0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff86b977700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8640ef7f0 con 0x7ff8641b6960 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff868f92700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8640ef3f0 0x7ff8640eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60220/0 (socket says 172.21.2.117:60220) 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff868f92700 1 -- 172.21.2.117:0/672422938 learned_addr learned my addr 172.21.2.117:0/672422938 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff869f94700 1 --2- 172.21.2.117:0/672422938 >> v2:172.21.2.125:3300/0 conn(0x7ff8641b6960 0x7ff8641b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360282661 0 0) 0x7ff8600010b0 con 0x7ff8640ef3f0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 -- 172.21.2.117:0/672422938 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff850002fe0 con 0x7ff8640ef3f0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 428366354 0 0) 0x7ff8600010b0 con 0x7ff8640ef3f0 2021-02-10T11:56:54.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 -- 172.21.2.117:0/672422938 >> v2:172.21.2.125:3300/0 conn(0x7ff8641b6960 msgr2=0x7ff8641b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 --2- 172.21.2.117:0/672422938 >> v2:172.21.2.125:3300/0 conn(0x7ff8641b6960 0x7ff8641b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.144 7ff857fff700 1 -- 172.21.2.117:0/672422938 >> v1:172.21.2.125:6789/0 conn(0x7ff8640d18e0 legacy=0x7ff8640ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8640d1d20 con 0x7ff8640ef3f0 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff8600015b0 con 0x7ff8640ef3f0 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 >> v1:172.21.2.117:6789/0 conn(0x7ff8640ef3f0 legacy=0x7ff8640eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 0x7ff850005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff850006780 0x7ff850008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff850009580 0x7ff85000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85000c2f0 con 0x7ff850003aa0 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85000c510 con 0x7ff850006780 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85000c730 con 0x7ff850009580 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 0x7ff850005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869f94700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff850009580 0x7ff85000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff868f92700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff850006780 0x7ff850008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 -- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff850009580 msgr2=0x7ff85000baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff850009580 0x7ff85000baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 -- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff850006780 msgr2=0x7ff850008ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff850006780 0x7ff850008ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869f94700 1 -- 172.21.2.117:0/672422938 reap_dead start 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 -- 172.21.2.117:0/672422938 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850050970 con 0x7ff850003aa0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff869793700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 0x7ff850005f60 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 rx=0x7ff858006ad0 tx=0x7ff858009b10).ready entity=mon.0 client_cookie=f02e1b210539830a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff85800c0c0 con 0x7ff850003aa0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85800c400 con 0x7ff850003aa0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff857fff700 1 -- 172.21.2.117:0/672422938 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff85800c850 con 0x7ff850003aa0 2021-02-10T11:56:54.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff86b977700 1 -- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 msgr2=0x7ff850005f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.145 7ff86b977700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 0x7ff850005f60 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 rx=0x7ff858006ad0 tx=0x7ff858009b10).stop 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.146 7ff86b977700 1 -- 172.21.2.117:0/672422938 shutdown_connections 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.146 7ff86b977700 1 --2- 172.21.2.117:0/672422938 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff850003aa0 0x7ff850005f60 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.146 7ff86b977700 1 -- 172.21.2.117:0/672422938 >> 172.21.2.117:0/672422938 conn(0x7ff8640ced60 msgr2=0x7ff8640cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.146 7ff86b977700 1 -- 172.21.2.117:0/672422938 shutdown_connections 2021-02-10T11:56:54.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.146 7ff86b977700 1 -- 172.21.2.117:0/672422938 wait complete. 2021-02-10T11:56:54.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 Processor -- start 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 -- start start 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 0x7ff8640e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff86408c600 0x7ff86408a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86408b660 con 0x7ff86408bbe0 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86408ca00 con 0x7ff8640b2160 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.161 7ff86b977700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640e9a80 con 0x7ff86408c600 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:59982/0 (socket says 172.21.2.117:59982) 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 -- 172.21.2.117:0/549603802 learned_addr learned my addr 172.21.2.117:0/549603802 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff869f94700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff86408c600 0x7ff86408a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff869793700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 0x7ff8640e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 -- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff86408c600 msgr2=0x7ff86408a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff86408c600 0x7ff86408a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 -- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 msgr2=0x7ff8640e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 0x7ff8640e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 -- 172.21.2.117:0/549603802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86408a8a0 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff869793700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 0x7ff8640e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff868f92700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7ff860007650 tx=0x7ff86000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff86000acf0 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff860004350 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff860004870 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff86b977700 1 -- 172.21.2.117:0/549603802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86408a8a0 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.162 7ff86b977700 1 -- 172.21.2.117:0/549603802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86408ae30 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.163 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff860004530 con 0x7ff86408bbe0 2021-02-10T11:56:54.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.163 7ff855ffb700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:54.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.163 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff8600328e0 con 0x7ff86408bbe0 2021-02-10T11:56:54.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.163 7ff869793700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:54.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.164 7ff869793700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:54.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.194 7ff86b977700 1 -- 172.21.2.117:0/549603802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8640ede60 con 0x7ff86408bbe0 2021-02-10T11:56:54.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.196 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff860032af0 con 0x7ff86408bbe0 2021-02-10T11:56:54.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.329 7ff86b977700 1 -- 172.21.2.117:0/549603802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7ff8640ede60 con 0x7ff86408bbe0 2021-02-10T11:56:54.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.702 7ff855ffb700 1 -- 172.21.2.117:0/549603802 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v876) v1 ==== 150+0+0 (secure 0 0 0) 0x7ff860004350 con 0x7ff86408bbe0 2021-02-10T11:56:54.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 msgr2=0x7ff84401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 crc :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 msgr2=0x7ff86408bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:54.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 rx=0x7ff860007650 tx=0x7ff86000a900).stop 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 shutdown_connections 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8440186c0 0x7ff84401abc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff86408bbe0 0x7ff86408bfe0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff86408c600 0x7ff86408a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 --2- 172.21.2.117:0/549603802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8640b2160 0x7ff8640e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 >> 172.21.2.117:0/549603802 conn(0x7ff8640bb5e0 msgr2=0x7ff864090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 shutdown_connections 2021-02-10T11:56:54.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:54.748 7ff86b977700 1 -- 172.21.2.117:0/549603802 wait complete. 2021-02-10T11:56:54.759 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.174 7f37feeb1700 1 Processor -- start 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37feeb1700 1 -- start start 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37feeb1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80ef3f0 0x7f37f80eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37feeb1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80ef7f0 con 0x7f37f80d18f0 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37feeb1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80d1cf0 con 0x7f37f80eb830 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37feeb1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80eded0 con 0x7f37f80ef3f0 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37fcccd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f37f80d18f0 0x7f37f80eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60238/0 (socket says 172.21.2.117:60238) 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.175 7f37fcccd700 1 -- 172.21.2.117:0/3374086140 learned_addr learned my addr 172.21.2.117:0/3374086140 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:55.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656891990 0 0) 0x7f37e0001180 con 0x7f37f80d18f0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e4002fe0 con 0x7f37f80d18f0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80ef3f0 0x7f37f80eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2203786891 0 0) 0x7f37e00010e0 con 0x7f37f80d18f0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80ef3f0 msgr2=0x7f37f80eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80ef3f0 0x7f37f80eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 >> v1:172.21.2.125:6789/0 conn(0x7f37f80eb830 legacy=0x7f37f81b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f81bb1c0 con 0x7f37f80d18f0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f37e0001600 con 0x7f37f80d18f0 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 >> v1:172.21.2.117:6789/0 conn(0x7f37f80d18f0 legacy=0x7f37f80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4003ae0 0x7f37e4005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e4006870 0x7f37e4008d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e4009630 0x7f37e400bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c400 con 0x7f37e4006870 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c620 con 0x7f37e4009630 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c840 con 0x7f37e4003ae0 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80ef3f0 0x7f37f80eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e4006870 0x7f37e4008d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.176 7f37fcccd700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4003ae0 0x7f37e4005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37fd4ce700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e4009630 0x7f37e400bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 -- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4003ae0 msgr2=0x7f37e4005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37e4003ae0 0x7f37e4005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 -- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e4009630 msgr2=0x7f37e400bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37e4009630 0x7f37e400bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 -- 172.21.2.117:0/3374086140 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e400cae0 con 0x7f37e4006870 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f7fff700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e4006870 0x7f37e4008d90 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7f37e80064c0 tx=0x7f37e804dac0).ready entity=mon.0 client_cookie=5be6a00f9493d97e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:55.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37e80500c0 con 0x7f37e4006870 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e8050380 con 0x7f37e4006870 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37f77fe700 1 -- 172.21.2.117:0/3374086140 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37e8050860 con 0x7f37e4006870 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37feeb1700 1 -- 172.21.2.117:0/3374086140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e4006870 msgr2=0x7f37e4008d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37feeb1700 1 --2- 172.21.2.117:0/3374086140 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37e4006870 0x7f37e4008d90 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 rx=0x7f37e80064c0 tx=0x7f37e804dac0).stop 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.177 7f37fd4ce700 1 -- 172.21.2.117:0/3374086140 reap_dead start 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.178 7f37feeb1700 1 -- 172.21.2.117:0/3374086140 shutdown_connections 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.178 7f37feeb1700 1 -- 172.21.2.117:0/3374086140 >> 172.21.2.117:0/3374086140 conn(0x7f37f80cf060 msgr2=0x7f37f80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.178 7f37feeb1700 1 -- 172.21.2.117:0/3374086140 shutdown_connections 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.178 7f37feeb1700 1 -- 172.21.2.117:0/3374086140 wait complete. 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 Processor -- start 2021-02-10T11:56:55.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 -- start start 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37f8094090 0x7f37f80b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80b31e0 0x7f37f819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f80b3b50 con 0x7f37f80b35e0 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f8094490 con 0x7f37f8094090 2021-02-10T11:56:55.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.193 7f37feeb1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f81a08a0 con 0x7f37f80b31e0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60004/0 (socket says 172.21.2.117:60004) 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 -- 172.21.2.117:0/2995447756 learned_addr learned my addr 172.21.2.117:0/2995447756 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 -- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80b31e0 msgr2=0x7f37f819fc60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80b31e0 0x7f37f819fc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 -- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37f8094090 msgr2=0x7f37f80b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37f8094090 0x7f37f80b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 -- 172.21.2.117:0/2995447756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f807c7c0 con 0x7f37f80b35e0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37fd4ce700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7f37f000b8c0 tx=0x7f37f000bc60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f0006ba0 con 0x7f37f80b35e0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f000c2f0 con 0x7f37f80b35e0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f807c7c0 con 0x7f37f80b35e0 2021-02-10T11:56:55.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37f000c750 con 0x7f37f80b35e0 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.194 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f807cd50 con 0x7f37f80b35e0 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.195 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37f0006b80 con 0x7f37f80b35e0 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.195 7f37f57fa700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 0x7f37dc01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.195 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f37f0032b40 con 0x7f37f80b35e0 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.195 7f37fcccd700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 0x7f37dc01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:55.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.196 7f37fcccd700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 0x7f37dc01ac10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:55.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.225 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f80daff0 con 0x7f37f80b35e0 2021-02-10T11:56:55.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.227 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37f001f8b0 con 0x7f37f80b35e0 2021-02-10T11:56:55.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.376 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f37f80ede60 con 0x7f37f80b35e0 2021-02-10T11:56:55.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.380 7f37f57fa700 1 -- 172.21.2.117:0/2995447756 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37f0006b80 con 0x7f37f80b35e0 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.385 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 msgr2=0x7f37dc01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.385 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 0x7f37dc01ac10 crc :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.385 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 msgr2=0x7f37f81a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.385 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 rx=0x7f37f000b8c0 tx=0x7f37f000bc60).stop 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 shutdown_connections 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37dc018730 0x7f37dc01ac10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37f80b35e0 0x7f37f81a0280 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37f80b31e0 0x7f37f819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:55.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 --2- 172.21.2.117:0/2995447756 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37f8094090 0x7f37f80b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:55.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 >> 172.21.2.117:0/2995447756 conn(0x7f37f80b8180 msgr2=0x7f37f80b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:55.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 shutdown_connections 2021-02-10T11:56:55.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:55.386 7f37feeb1700 1 -- 172.21.2.117:0/2995447756 wait complete. 2021-02-10T11:56:55.397 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:56:55.397 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:56:55.743 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:56:56.087 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:56:56.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:56.087 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:56.088 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:56:56.422 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:56:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:56.422 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:56:56.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:56.422 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:56.423 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:56:56.759 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:56:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:56.760 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:56:56.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:56.760 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:56.760 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:56:57.095 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:56:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:57.096 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:56:57.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:56:57.096 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:56:57.096 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:56:57.429 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:56:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:56:57.430 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:56:57.430 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:56:57.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 Processor -- start 2021-02-10T11:56:57.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 -- start start 2021-02-10T11:56:57.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 0x7f2ccc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2ccc0eb000 0x7f2ccc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc0ede60 con 0x7f2ccc0ee3e0 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ccc0eba20 con 0x7f2ccc0eb000 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.841 7f2cd1394700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0d1ce0 con 0x7f2ccc0d18e0 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.842 7f2ccb87e700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f2ccc0ee3e0 0x7f2ccc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53184/0 (socket says 172.21.2.117:53184) 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.842 7f2cca87c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f2ccc0eb000 0x7f2ccc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.842 7f2ccb07d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 0x7f2ccc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.842 7f2ccb87e700 1 -- 172.21.2.117:0/2115145091 learned_addr learned my addr 172.21.2.117:0/2115145091 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca87c700 1 -- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 msgr2=0x7f2ccc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca87c700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 0x7f2ccc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca87c700 1 -- 172.21.2.117:0/2115145091 >> v1:172.21.2.125:6789/0 conn(0x7f2ccc0ee3e0 legacy=0x7f2ccc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca87c700 1 -- 172.21.2.117:0/2115145091 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc0eefd0 con 0x7f2ccc0eb000 2021-02-10T11:56:57.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2ccb07d700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 0x7f2ccc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca87c700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc0eb000 0x7f2ccc0eb400 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f2cb4006650 tx=0x7f2cb4009a80).ready entity=mon.1 client_cookie=63f6a7639cf5fc0b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca07b700 1 -- 172.21.2.117:0/2115145091 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb40009b0 con 0x7f2ccc0eb000 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca07b700 1 -- 172.21.2.117:0/2115145091 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb400c0c0 con 0x7f2ccc0eb000 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cca07b700 1 -- 172.21.2.117:0/2115145091 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cb400c5e0 con 0x7f2ccc0eb000 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cd1394700 1 -- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc0eb000 msgr2=0x7f2ccc0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.843 7f2cd1394700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc0eb000 0x7f2ccc0eb400 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f2cb4006650 tx=0x7f2cb4009a80).stop 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 -- 172.21.2.117:0/2115145091 shutdown_connections 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc0eb000 0x7f2ccc0eb400 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 --2- 172.21.2.117:0/2115145091 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc0d18e0 0x7f2ccc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 -- 172.21.2.117:0/2115145091 >> 172.21.2.117:0/2115145091 conn(0x7f2ccc0ced60 msgr2=0x7f2ccc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 -- 172.21.2.117:0/2115145091 shutdown_connections 2021-02-10T11:56:57.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.844 7f2cd1394700 1 -- 172.21.2.117:0/2115145091 wait complete. 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.858 7f2cd1394700 1 Processor -- start 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 -- start start 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc1af4e0 0x7f2ccc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc1aeb20 0x7f2ccc19c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc1b87a0 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc1aef90 con 0x7f2ccc1aeb20 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2cd1394700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc19c760 con 0x7f2ccc1af4e0 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2ccb87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2ccb87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60012/0 (socket says 172.21.2.117:60012) 2021-02-10T11:56:57.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.859 7f2ccb87e700 1 -- 172.21.2.117:0/1374857787 learned_addr learned my addr 172.21.2.117:0/1374857787 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb07d700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc1af4e0 0x7f2ccc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 -- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc1af4e0 msgr2=0x7f2ccc19b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc1af4e0 0x7f2ccc19b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 -- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc1aeb20 msgr2=0x7f2ccc19c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc1aeb20 0x7f2ccc19c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 -- 172.21.2.117:0/1374857787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc0b3650 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2ccb87e700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f2cc400a8a0 tx=0x7f2cc400ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc40070d0 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc4007410 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc0b3650 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc093840 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.860 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2cc4004920 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.861 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2cc40046a0 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.861 7f2cc3fff700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 0x7f2cb001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:57.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.861 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2cc4032c40 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.861 7f2ccb07d700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 0x7f2cb001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:57.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.861 7f2ccb07d700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 0x7f2cb001abd0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:57.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.891 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc087290 con 0x7f2ccc0b2c30 2021-02-10T11:56:57.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:57.893 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2cc4001280 con 0x7f2ccc0b2c30 2021-02-10T11:56:58.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.018 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2ccc0ede60 con 0x7f2ccc0b2c30 2021-02-10T11:56:58.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.019 7f2cc3fff700 1 -- 172.21.2.117:0/1374857787 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v179) v1 ==== 81+0+30766 (secure 0 0 0) 0x7f2cc40046a0 con 0x7f2ccc0b2c30 2021-02-10T11:56:58.027 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:58.027 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:56:58.027 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 179, 2021-02-10T11:56:58.027 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:56:58.027 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:56:10.207917", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:56:58.028 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 39, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:56:58.029 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:56:58.030 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:56:58.031 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:56:58.032 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:56:58.033 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:56:58.034 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 38, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:56:07.173989", 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:56:58.035 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:56:58.036 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "178", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:56:58.037 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:56:58.038 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:56:58.039 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:56:58.040 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 39, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:56:08.248299", 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:56:58.041 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:56:58.042 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "179", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:56:58.043 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:56:58.044 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:56:58.045 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.047 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.048 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.049 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.050 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:56:58.051 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.052 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.053 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.054 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.055 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.056 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:56:58.057 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.058 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.059 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.060 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.061 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:56:58.062 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.063 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.064 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.065 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.066 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:56:58.067 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.068 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.069 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.070 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.071 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.072 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.073 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.074 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.075 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.076 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.077 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.078 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.079 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:56:58.080 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.081 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:56:58.082 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.083 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 175, 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:56:58.084 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.085 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.086 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.087 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.088 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:56:58.089 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:56:58.090 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.091 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.092 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.093 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.094 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.095 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.096 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.097 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.098 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.099 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:56:58.100 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:56:58.101 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:56:58.102 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 msgr2=0x7f2cb001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 0x7f2cb001abd0 crc :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 msgr2=0x7f2ccc0b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 rx=0x7f2cc400a8a0 tx=0x7f2cc400ac40).stop 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 shutdown_connections 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2cb00186d0 0x7f2cb001abd0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ccc0b2c30 0x7f2ccc0b3030 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ccc1aeb20 0x7f2ccc19c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:58.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 --2- 172.21.2.117:0/1374857787 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ccc1af4e0 0x7f2ccc19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 >> 172.21.2.117:0/1374857787 conn(0x7f2ccc1a3620 msgr2=0x7f2ccc092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:58.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 shutdown_connections 2021-02-10T11:56:58.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.021 7f2cd1394700 1 -- 172.21.2.117:0/1374857787 wait complete. 2021-02-10T11:56:58.105 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 Processor -- start 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 -- start start 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 0x7f91940ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 0x7f91940eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940ede60 con 0x7f91940ee3e0 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91940eba90 con 0x7f91940d18e0 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.515 7f91997a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91940d1ce0 con 0x7f91940eb000 2021-02-10T11:56:58.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9193d75700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f91940ee3e0 0x7f91940ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53196/0 (socket says 172.21.2.117:53196) 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9193d75700 1 -- 172.21.2.117:0/4035137392 learned_addr learned my addr 172.21.2.117:0/4035137392 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9193574700 1 --2- 172.21.2.117:0/4035137392 >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 0x7f91940ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 --2- 172.21.2.117:0/4035137392 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 0x7f91940eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192572700 1 -- 172.21.2.117:0/4035137392 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693819345 0 0) 0x7f918c001510 con 0x7f91940ee3e0 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192572700 1 -- 172.21.2.117:0/4035137392 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9180002fe0 con 0x7f91940ee3e0 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 -- 172.21.2.117:0/4035137392 >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 msgr2=0x7f91940ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 --2- 172.21.2.117:0/4035137392 >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 0x7f91940ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 -- 172.21.2.117:0/4035137392 >> v1:172.21.2.125:6789/0 conn(0x7f91940ee3e0 legacy=0x7f91940ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 -- 172.21.2.117:0/4035137392 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91940eefd0 con 0x7f91940eb000 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9193574700 1 --2- 172.21.2.117:0/4035137392 >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 0x7f91940ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.516 7f9192d73700 1 --2- 172.21.2.117:0/4035137392 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 0x7f91940eb400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f917c006650 tx=0x7f917c009a80).ready entity=mon.2 client_cookie=3e5f298bb9c569bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f9192572700 1 -- 172.21.2.117:0/4035137392 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f917c0009b0 con 0x7f91940eb000 2021-02-10T11:56:58.536 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f9192572700 1 -- 172.21.2.117:0/4035137392 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c00c0c0 con 0x7f91940eb000 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f9192572700 1 -- 172.21.2.117:0/4035137392 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f917c00c5e0 con 0x7f91940eb000 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 -- 172.21.2.117:0/4035137392 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 msgr2=0x7f91940eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 --2- 172.21.2.117:0/4035137392 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 0x7f91940eb400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f917c006650 tx=0x7f917c009a80).stop 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 -- 172.21.2.117:0/4035137392 shutdown_connections 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 --2- 172.21.2.117:0/4035137392 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91940eb000 0x7f91940eb400 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 --2- 172.21.2.117:0/4035137392 >> v2:172.21.2.125:3300/0 conn(0x7f91940d18e0 0x7f91940ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.517 7f91997a7700 1 -- 172.21.2.117:0/4035137392 >> 172.21.2.117:0/4035137392 conn(0x7f91940ced60 msgr2=0x7f91940cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.518 7f91997a7700 1 -- 172.21.2.117:0/4035137392 shutdown_connections 2021-02-10T11:56:58.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.518 7f91997a7700 1 -- 172.21.2.117:0/4035137392 wait complete. 2021-02-10T11:56:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.532 7f91997a7700 1 Processor -- start 2021-02-10T11:56:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.532 7f91997a7700 1 -- start start 2021-02-10T11:56:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 0x7f9194091f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.539 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 0x7f9194095c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91940967f0 con 0x7f9194096270 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91941972b0 con 0x7f9194091b00 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f91997a7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9194197490 con 0x7f9194095850 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60026/0 (socket says 172.21.2.117:60026) 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 -- 172.21.2.117:0/1400690718 learned_addr learned my addr 172.21.2.117:0/1400690718 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9192d73700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 0x7f9194095c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193574700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 0x7f9194091f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 -- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 msgr2=0x7f9194095c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 0x7f9194095c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 -- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 msgr2=0x7f9194091f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 0x7f9194091f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.540 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 -- 172.21.2.117:0/1400690718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91941976b0 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193574700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 0x7f9194091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9192d73700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 0x7f9194095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.533 7f9193d75700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7f918c009980 tx=0x7f918c00bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f918c006080 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c006300 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f91997a7700 1 -- 172.21.2.117:0/1400690718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91941976b0 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f918c0046a0 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f91997a7700 1 -- 172.21.2.117:0/1400690718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9194197c40 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f918c006020 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 0x7f917801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f918c0328c0 con 0x7f9194096270 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.534 7f9193574700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 0x7f917801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:58.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.535 7f9193574700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 0x7f917801abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:58.568 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.565 7f91997a7700 1 -- 172.21.2.117:0/1400690718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9194087290 con 0x7f9194096270 2021-02-10T11:56:58.570 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.567 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f918c000f40 con 0x7f9194096270 2021-02-10T11:56:58.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.696 7f91997a7700 1 -- 172.21.2.117:0/1400690718 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91940ede60 con 0x7f9194096270 2021-02-10T11:56:58.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.696 7f918bfff700 1 -- 172.21.2.117:0/1400690718 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+2107 (secure 0 0 0) 0x7f918c006000 con 0x7f9194096270 2021-02-10T11:56:58.700 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:56:58.700 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":154,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:11.215531","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9310":{"gid":9310,"name":"b","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/3249802432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":3249802432},{"type":"v1","addr":"172.21.2.125:6835","nonce":3249802432}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:56:58.700 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 154 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 msgr2=0x7f917801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 0x7f917801abc0 crc :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 msgr2=0x7f9194196c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 rx=0x7f918c009980 tx=0x7f918c00bcd0).stop 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 shutdown_connections 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91780186c0 0x7f917801abc0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9194096270 0x7f9194196c90 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9194095850 0x7f9194095c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 --2- 172.21.2.117:0/1400690718 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9194091b00 0x7f9194091f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:58.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 >> 172.21.2.117:0/1400690718 conn(0x7f91941a3620 msgr2=0x7f9194092660 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:58.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 shutdown_connections 2021-02-10T11:56:58.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:58.698 7f91997a7700 1 -- 172.21.2.117:0/1400690718 wait complete. 2021-02-10T11:56:58.709 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:56:59.137 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.118 7fd2c2ec0700 1 Processor -- start 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.118 7fd2c2ec0700 1 -- start start 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c2ec0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 0x7fd2bc0e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c2ec0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2bc0e8390 0x7fd2bc0e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c2ec0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0e7580 con 0x7fd2bc1b07f0 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c2ec0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0e5ce0 con 0x7fd2bc0e8390 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c2ec0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0e77f0 con 0x7fd2bc0e7180 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2bc0e8390 0x7fd2bc0e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd2bc0e8390 0x7fd2bc0e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60030/0 (socket says 172.21.2.117:60030) 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 -- 172.21.2.117:0/1515018421 learned_addr learned my addr 172.21.2.117:0/1515018421 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:59.138 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2bbfff700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 0x7fd2bc0e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 -- 172.21.2.117:0/1515018421 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 msgr2=0x7fd2bc0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 0x7fd2bc0e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 -- 172.21.2.117:0/1515018421 >> v1:172.21.2.125:6789/0 conn(0x7fd2bc1b07f0 legacy=0x7fd2bc0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2c14dd700 1 -- 172.21.2.117:0/1515018421 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0e8db0 con 0x7fd2bc0e8390 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.119 7fd2bbfff700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 0x7fd2bc0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2c14dd700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0e8390 0x7fd2bc0e8790 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7fd2b4006b80 tx=0x7fd2b4009b90).ready entity=mon.0 client_cookie=140de05365cde8dc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2bb7fe700 1 -- 172.21.2.117:0/1515018421 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b40022e0 con 0x7fd2bc0e8390 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2bb7fe700 1 -- 172.21.2.117:0/1515018421 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4002620 con 0x7fd2bc0e8390 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2bb7fe700 1 -- 172.21.2.117:0/1515018421 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2b4002b00 con 0x7fd2bc0e8390 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2c2ec0700 1 -- 172.21.2.117:0/1515018421 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0e8390 msgr2=0x7fd2bc0e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.120 7fd2c2ec0700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0e8390 0x7fd2bc0e8790 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 rx=0x7fd2b4006b80 tx=0x7fd2b4009b90).stop 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 -- 172.21.2.117:0/1515018421 shutdown_connections 2021-02-10T11:56:59.139 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0e8390 0x7fd2bc0e8790 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 --2- 172.21.2.117:0/1515018421 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0e7180 0x7fd2bc0e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 -- 172.21.2.117:0/1515018421 >> 172.21.2.117:0/1515018421 conn(0x7fd2bc0ce500 msgr2=0x7fd2bc0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:59.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 -- 172.21.2.117:0/1515018421 shutdown_connections 2021-02-10T11:56:59.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.121 7fd2c2ec0700 1 -- 172.21.2.117:0/1515018421 wait complete. 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.135 7fd2c2ec0700 1 Processor -- start 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.135 7fd2c2ec0700 1 -- start start 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0a9330 0x7fd2bc0a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2bc0a7700 0x7fd2bc0a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.141 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0a8db0 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc105f50 con 0x7fd2bc0a7700 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0ab4c0 con 0x7fd2bc0a9330 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60034/0 (socket says 172.21.2.117:60034) 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 -- 172.21.2.117:0/3691915813 learned_addr learned my addr 172.21.2.117:0/3691915813 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 -- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0a9330 msgr2=0x7fd2bc0a9730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0a9330 0x7fd2bc0a9730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 -- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2bc0a7700 msgr2=0x7fd2bc0a7b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2bc0a7700 0x7fd2bc0a7b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 -- 172.21.2.117:0/3691915813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0a8120 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2bbfff700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7fd2ac0099a0 tx=0x7fd2ac00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2ac00c2f0 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac00c530 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd2ac00ca50 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc0a8120 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.136 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc0a86b0 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.137 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd2ac001010 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.137 7fd2b97fa700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 0x7fd2a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.137 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd2ac032a40 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.138 7fd2c0cdc700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 0x7fd2a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.138 7fd2c0cdc700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 0x7fd2a001abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.168 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc1b8070 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.173 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.171 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd2ac000ce0 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.297 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd2bc0e77f0 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.372 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:56:59.369 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:56:59.378 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.375 7fd2b97fa700 1 -- 172.21.2.117:0/3691915813 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v155) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd2ac001010 con 0x7fd2bc0a9d50 2021-02-10T11:56:59.414 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:56:59.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 msgr2=0x7fd2a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 0x7fd2a001abc0 crc :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 msgr2=0x7fd2bc0aa150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7fd2ac0099a0 tx=0x7fd2ac00bcf0).stop 2021-02-10T11:56:59.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 shutdown_connections 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd2a00186c0 0x7fd2a001abc0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd2bc0a9d50 0x7fd2bc0aa150 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd2bc0a7700 0x7fd2bc0a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 --2- 172.21.2.117:0/3691915813 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd2bc0a9330 0x7fd2bc0a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 >> 172.21.2.117:0/3691915813 conn(0x7fd2bc0b97b0 msgr2=0x7fd2bc098f40 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 shutdown_connections 2021-02-10T11:56:59.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.413 7fd2c2ec0700 1 -- 172.21.2.117:0/3691915813 wait complete. 2021-02-10T11:56:59.424 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:56:59.429 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:56:59.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.835 7fe02a77d700 1 Processor -- start 2021-02-10T11:56:59.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.835 7fe02a77d700 1 -- start start 2021-02-10T11:56:59.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe02a77d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe0240ef3f0 0x7fe0240eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe02a77d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe0240eb7f0 0x7fe0241b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe02a77d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0240ede60 con 0x7fe0240d18e0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe02a77d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240ef7f0 con 0x7fe0240ef3f0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe02a77d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0240d1ce0 con 0x7fe0240eb7f0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe0240ef3f0 0x7fe0240eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fe0240ef3f0 0x7fe0240eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60046/0 (socket says 172.21.2.117:60046) 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 -- 172.21.2.117:0/3052926900 learned_addr learned my addr 172.21.2.117:0/3052926900 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe028d9a700 1 --2- 172.21.2.117:0/3052926900 >> v2:172.21.2.125:3300/0 conn(0x7fe0240eb7f0 0x7fe0241b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 -- 172.21.2.117:0/3052926900 >> v2:172.21.2.125:3300/0 conn(0x7fe0240eb7f0 msgr2=0x7fe0241b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 --2- 172.21.2.117:0/3052926900 >> v2:172.21.2.125:3300/0 conn(0x7fe0240eb7f0 0x7fe0241b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.836 7fe0237fe700 1 -- 172.21.2.117:0/3052926900 >> v1:172.21.2.125:6789/0 conn(0x7fe0240d18e0 legacy=0x7fe0240ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe0237fe700 1 -- 172.21.2.117:0/3052926900 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0241b8f90 con 0x7fe0240ef3f0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe0237fe700 1 --2- 172.21.2.117:0/3052926900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240ef3f0 0x7fe0240eb1d0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7fe018006700 tx=0x7fe018009b80).ready entity=mon.0 client_cookie=7b68b141bef9c9e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe022ffd700 1 -- 172.21.2.117:0/3052926900 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01800c0c0 con 0x7fe0240ef3f0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe022ffd700 1 -- 172.21.2.117:0/3052926900 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01800c340 con 0x7fe0240ef3f0 2021-02-10T11:56:59.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe022ffd700 1 -- 172.21.2.117:0/3052926900 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01800c860 con 0x7fe0240ef3f0 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe02a77d700 1 -- 172.21.2.117:0/3052926900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240ef3f0 msgr2=0x7fe0240eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.837 7fe02a77d700 1 --2- 172.21.2.117:0/3052926900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240ef3f0 0x7fe0240eb1d0 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 rx=0x7fe018006700 tx=0x7fe018009b80).stop 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 -- 172.21.2.117:0/3052926900 shutdown_connections 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 --2- 172.21.2.117:0/3052926900 >> v2:172.21.2.125:3300/0 conn(0x7fe0240eb7f0 0x7fe0241b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 --2- 172.21.2.117:0/3052926900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240ef3f0 0x7fe0240eb1d0 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 -- 172.21.2.117:0/3052926900 >> 172.21.2.117:0/3052926900 conn(0x7fe0240ced60 msgr2=0x7fe0240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 -- 172.21.2.117:0/3052926900 shutdown_connections 2021-02-10T11:56:59.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.838 7fe02a77d700 1 -- 172.21.2.117:0/3052926900 wait complete. 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.852 7fe02a77d700 1 Processor -- start 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 -- start start 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0240c2910 0x7fe0240c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0240963f0 0x7fe0240967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe024095e70 con 0x7fe0240959c0 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0240e86d0 con 0x7fe0240c2910 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe02a77d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe024096e10 con 0x7fe0240963f0 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe023fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.859 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe023fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60048/0 (socket says 172.21.2.117:60048) 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.853 7fe023fff700 1 -- 172.21.2.117:0/765214493 learned_addr learned my addr 172.21.2.117:0/765214493 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 -- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0240963f0 msgr2=0x7fe0240967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0240963f0 0x7fe0240967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 -- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0240c2910 msgr2=0x7fe0240c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0240c2910 0x7fe0240c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 -- 172.21.2.117:0/765214493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe024097030 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe023fff700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7fe01400b5b0 tx=0x7fe01400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe01400bd40 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe014006240 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe02a77d700 1 -- 172.21.2.117:0/765214493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe024097030 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe014005720 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.854 7fe02a77d700 1 -- 172.21.2.117:0/765214493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe024097030 con 0x7fe0240959c0 2021-02-10T11:56:59.860 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.855 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe0140054a0 con 0x7fe0240959c0 2021-02-10T11:56:59.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.855 7fe020ff9700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 0x7fe00401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:56:59.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.855 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fe0140327e0 con 0x7fe0240959c0 2021-02-10T11:56:59.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.855 7fe0237fe700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 0x7fe00401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:56:59.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.856 7fe0237fe700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 0x7fe00401abd0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:56:59.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.885 7fe02a77d700 1 -- 172.21.2.117:0/765214493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe024087290 con 0x7fe0240959c0 2021-02-10T11:56:59.891 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:56:59.888 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe014000c60 con 0x7fe0240959c0 2021-02-10T11:57:00.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.014 7fe02a77d700 1 -- 172.21.2.117:0/765214493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0240ede60 con 0x7fe0240959c0 2021-02-10T11:57:00.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.015 7fe020ff9700 1 -- 172.21.2.117:0/765214493 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+1762 (secure 0 0 0) 0x7fe01400bd40 con 0x7fe0240959c0 2021-02-10T11:57:00.018 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:00.019 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":155,"default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9313,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/2021651122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":2021651122},{"type":"v1","addr":"172.21.2.117:6833","nonce":2021651122}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":149}],"filesystems":[{"mdsmap":{"epoch":155,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:56:09.205433","modified":"2021-02-10 11:56:59.371049","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":180,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":19}]} 2021-02-10T11:57:00.019 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 155 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 msgr2=0x7fe00401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 0x7fe00401abd0 crc :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 msgr2=0x7fe0240c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 rx=0x7fe01400b5b0 tx=0x7fe01400b950).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 shutdown_connections 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe0240963f0 0x7fe0240967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe0240c2910 0x7fe0240c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe0040186d0 0x7fe00401abd0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 --2- 172.21.2.117:0/765214493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe0240959c0 0x7fe0240c22f0 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 >> 172.21.2.117:0/765214493 conn(0x7fe0240e8c80 msgr2=0x7fe024092070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:00.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 shutdown_connections 2021-02-10T11:57:00.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.017 7fe02a77d700 1 -- 172.21.2.117:0/765214493 wait complete. 2021-02-10T11:57:00.029 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 Processor -- start 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 -- start start 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd5501b07f0 0x7fd5500e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd5500e8d80 0x7fd5500e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5500e77f0 con 0x7fd5500e6170 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5500e5ce0 con 0x7fd5500e8d80 2021-02-10T11:57:00.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.444 7fd557de6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5500e54b0 con 0x7fd5501b07f0 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd5500e8d80 0x7fd5500e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd5500e8d80 0x7fd5500e6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60056/0 (socket says 172.21.2.117:60056) 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 -- 172.21.2.117:0/792872689 learned_addr learned my addr 172.21.2.117:0/792872689 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 -- 172.21.2.117:0/792872689 >> v2:172.21.2.125:3300/0 conn(0x7fd5501b07f0 msgr2=0x7fd5500e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 --2- 172.21.2.117:0/792872689 >> v2:172.21.2.125:3300/0 conn(0x7fd5501b07f0 0x7fd5500e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 -- 172.21.2.117:0/792872689 >> v1:172.21.2.117:6789/0 conn(0x7fd5500e6170 legacy=0x7fd5500e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 -- 172.21.2.117:0/792872689 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5500e9180 con 0x7fd5500e8d80 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd556403700 1 --2- 172.21.2.117:0/792872689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500e8d80 0x7fd5500e6d60 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7fd548006f00 tx=0x7fd548009f10).ready entity=mon.0 client_cookie=f508dc025d28adc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.445 7fd554c00700 1 -- 172.21.2.117:0/792872689 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54800b3b0 con 0x7fd5500e8d80 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd554c00700 1 -- 172.21.2.117:0/792872689 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54800b630 con 0x7fd5500e8d80 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd554c00700 1 -- 172.21.2.117:0/792872689 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54800bb50 con 0x7fd5500e8d80 2021-02-10T11:57:00.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 -- 172.21.2.117:0/792872689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500e8d80 msgr2=0x7fd5500e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 --2- 172.21.2.117:0/792872689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500e8d80 0x7fd5500e6d60 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 rx=0x7fd548006f00 tx=0x7fd548009f10).stop 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 -- 172.21.2.117:0/792872689 shutdown_connections 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 --2- 172.21.2.117:0/792872689 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500e8d80 0x7fd5500e6d60 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 --2- 172.21.2.117:0/792872689 >> v2:172.21.2.125:3300/0 conn(0x7fd5501b07f0 0x7fd5500e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.446 7fd557de6700 1 -- 172.21.2.117:0/792872689 >> 172.21.2.117:0/792872689 conn(0x7fd5500ce500 msgr2=0x7fd5500e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.447 7fd557de6700 1 -- 172.21.2.117:0/792872689 shutdown_connections 2021-02-10T11:57:00.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.447 7fd557de6700 1 -- 172.21.2.117:0/792872689 wait complete. 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.461 7fd557de6700 1 Processor -- start 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.461 7fd557de6700 1 -- start start 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd550093010 0x7fd550189e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5500b80e0 0x7fd550188d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550097690 con 0x7fd5500b76c0 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd550093410 con 0x7fd550093010 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd557de6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55018a430 con 0x7fd5500b80e0 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd555401700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:00.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd555401700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60062/0 (socket says 172.21.2.117:60062) 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd555401700 1 -- 172.21.2.117:0/2309477268 learned_addr learned my addr 172.21.2.117:0/2309477268 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd555c02700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd550093010 0x7fd550189e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.462 7fd556403700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5500b80e0 0x7fd550188d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 -- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5500b80e0 msgr2=0x7fd550188d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5500b80e0 0x7fd550188d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 -- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd550093010 msgr2=0x7fd550189e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd550093010 0x7fd550189e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 -- 172.21.2.117:0/2309477268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5501893a0 con 0x7fd5500b76c0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd555401700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7fd54c006ed0 tx=0x7fd54c00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54c006240 con 0x7fd5500b76c0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c0054b0 con 0x7fd5500b76c0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd54c005900 con 0x7fd5500b76c0 2021-02-10T11:57:00.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.463 7fd557de6700 1 -- 172.21.2.117:0/2309477268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5501893a0 con 0x7fd5500b76c0 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.464 7fd557de6700 1 -- 172.21.2.117:0/2309477268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5501893a0 con 0x7fd5500b76c0 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.464 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd54c006030 con 0x7fd5500b76c0 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.464 7fd5427fc700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 0x7fd53801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.464 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fd54c0325e0 con 0x7fd5500b76c0 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.464 7fd555c02700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 0x7fd53801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:00.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.465 7fd555c02700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 0x7fd53801abc0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:00.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.494 7fd557de6700 1 -- 172.21.2.117:0/2309477268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd550108a30 con 0x7fd5500b76c0 2021-02-10T11:57:00.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.497 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd54c030d60 con 0x7fd5500b76c0 2021-02-10T11:57:00.625 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:00.623 7fd557de6700 1 -- 172.21.2.117:0/2309477268 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd5500e77f0 con 0x7fd5500b76c0 2021-02-10T11:57:01.383 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.380 7fd5427fc700 1 -- 172.21.2.117:0/2309477268 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v156) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd54c006030 con 0x7fd5500b76c0 2021-02-10T11:57:01.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 -- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 msgr2=0x7fd53801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 0x7fd53801abc0 crc :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 -- 172.21.2.117:0/2309477268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 msgr2=0x7fd5500b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 rx=0x7fd54c006ed0 tx=0x7fd54c00b960).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 -- 172.21.2.117:0/2309477268 shutdown_connections 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd5380186c0 0x7fd53801abc0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd5500b76c0 0x7fd5500b7ac0 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd5500b80e0 0x7fd550188d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 --2- 172.21.2.117:0/2309477268 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd550093010 0x7fd550189e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.395 7fd557de6700 1 -- 172.21.2.117:0/2309477268 >> 172.21.2.117:0/2309477268 conn(0x7fd550198490 msgr2=0x7fd550111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.396 7fd557de6700 1 -- 172.21.2.117:0/2309477268 shutdown_connections 2021-02-10T11:57:01.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.396 7fd557de6700 1 -- 172.21.2.117:0/2309477268 wait complete. 2021-02-10T11:57:01.407 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:57:01.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.821 7fd0a9f83700 1 Processor -- start 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.821 7fd0a9f83700 1 -- start start 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a9f83700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd0a40e5240 0x7fd0a4086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a9f83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4084c60 0x7fd0a4085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a9f83700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a4085930 con 0x7fd0a40e4e40 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a9f83700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a4085680 con 0x7fd0a40e5240 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a9f83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a41b09e0 con 0x7fd0a4084c60 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a37fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd0a40e4e40 0x7fd0a41b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60316/0 (socket says 172.21.2.117:60316) 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a37fe700 1 -- 172.21.2.117:0/4163626960 learned_addr learned my addr 172.21.2.117:0/4163626960 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173095771 0 0) 0x7fd094000ce0 con 0x7fd0a40e4e40 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd08c002fe0 con 0x7fd0a40e4e40 2021-02-10T11:57:01.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a2ffd700 1 --2- 172.21.2.117:0/4163626960 >> v2:172.21.2.125:3300/0 conn(0x7fd0a40e5240 0x7fd0a4086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.822 7fd0a3fff700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4084c60 0x7fd0a4085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 975878960 0 0) 0x7fd094001180 con 0x7fd0a40e4e40 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4084c60 msgr2=0x7fd0a4085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4084c60 0x7fd0a4085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 >> v2:172.21.2.125:3300/0 conn(0x7fd0a40e5240 msgr2=0x7fd0a4086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 --2- 172.21.2.117:0/4163626960 >> v2:172.21.2.125:3300/0 conn(0x7fd0a40e5240 0x7fd0a4086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4084840 con 0x7fd0a40e4e40 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fd094001680 con 0x7fd0a40e4e40 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 >> v1:172.21.2.117:6789/0 conn(0x7fd0a40e4e40 legacy=0x7fd0a41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd08c003af0 0x7fd08c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd08c04a8c0 0x7fd08c04cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd08c04d5b0 0x7fd08c04fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c050380 con 0x7fd08c003af0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0505a0 con 0x7fd08c04d5b0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c0507c0 con 0x7fd08c04a8c0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a2ffd700 1 --2- 172.21.2.117:0/4163626960 >> v2:172.21.2.125:3300/0 conn(0x7fd0a40e5240 0x7fd0a4086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a37fe700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd08c003af0 0x7fd08c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a3fff700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd08c04d5b0 0x7fd08c04fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.823 7fd0a2ffd700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd08c04a8c0 0x7fd08c04cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 -- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd08c04a8c0 msgr2=0x7fd08c04cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd08c04a8c0 0x7fd08c04cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 -- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd08c04d5b0 msgr2=0x7fd08c04fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd08c04d5b0 0x7fd08c04fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 -- 172.21.2.117:0/4163626960 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c050a60 con 0x7fd08c003af0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a37fe700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd08c003af0 0x7fd08c006010 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 rx=0x7fd0940017e0 tx=0x7fd094007d10).ready entity=mon.0 client_cookie=8edf3ed689b24b99 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd094001820 con 0x7fd08c003af0 2021-02-10T11:57:01.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd094007150 con 0x7fd08c003af0 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a27fc700 1 -- 172.21.2.117:0/4163626960 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd094004590 con 0x7fd08c003af0 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a9f83700 1 -- 172.21.2.117:0/4163626960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd08c003af0 msgr2=0x7fd08c006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a9f83700 1 --2- 172.21.2.117:0/4163626960 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd08c003af0 0x7fd08c006010 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 rx=0x7fd0940017e0 tx=0x7fd094007d10).stop 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.824 7fd0a3fff700 1 -- 172.21.2.117:0/4163626960 reap_dead start 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.825 7fd0a9f83700 1 -- 172.21.2.117:0/4163626960 shutdown_connections 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.825 7fd0a9f83700 1 -- 172.21.2.117:0/4163626960 >> 172.21.2.117:0/4163626960 conn(0x7fd0a40ce4e0 msgr2=0x7fd0a40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.825 7fd0a9f83700 1 -- 172.21.2.117:0/4163626960 shutdown_connections 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.825 7fd0a9f83700 1 -- 172.21.2.117:0/4163626960 wait complete. 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 Processor -- start 2021-02-10T11:57:01.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 -- start start 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0a410ffa0 0x7fd0a41103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 0x7fd0a4090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a41109c0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40a0ea0 con 0x7fd0a410ffa0 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a9f83700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a41a0360 con 0x7fd0a4090ae0 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.848 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60078/0 (socket says 172.21.2.117:60078) 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.840 7fd0a37fe700 1 -- 172.21.2.117:0/3362710597 learned_addr learned my addr 172.21.2.117:0/3362710597 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a3fff700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 0x7fd0a4090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a2ffd700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0a410ffa0 0x7fd0a41103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 -- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 msgr2=0x7fd0a4090ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 0x7fd0a4090ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 -- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0a410ffa0 msgr2=0x7fd0a41103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0a410ffa0 0x7fd0a41103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 -- 172.21.2.117:0/3362710597 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4110d00 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a3fff700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 0x7fd0a4090ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a37fe700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fd09400a5a0 tx=0x7fd09400a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd09400ad30 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0940070d0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.849 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a4110d00 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a40917f0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.841 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd0940043a0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.842 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd0940043a0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.842 7fd08bfff700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 0x7fd08401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.842 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6006+0+0 (secure 0 0 0) 0x7fd094032980 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.842 7fd0a2ffd700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 0x7fd08401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:01.850 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.843 7fd0a2ffd700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 0x7fd08401abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:01.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.872 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c002fd0 con 0x7fd0a40a0aa0 2021-02-10T11:57:01.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.874 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd0940013b0 con 0x7fd0a40a0aa0 2021-02-10T11:57:02.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:01.998 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd07c002ef0 con 0x7fd0a40a0aa0 2021-02-10T11:57:02.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.381 7fd08bfff700 1 -- 172.21.2.117:0/3362710597 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v181) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd0940043a0 con 0x7fd0a40a0aa0 2021-02-10T11:57:02.418 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:57:02.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.417 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 msgr2=0x7fd08401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.417 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 0x7fd08401abc0 crc :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.417 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 msgr2=0x7fd0a419fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.417 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 rx=0x7fd09400a5a0 tx=0x7fd09400a940).stop 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 shutdown_connections 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd0a4090ae0 0x7fd0a4090ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd0a410ffa0 0x7fd0a41103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd0840186c0 0x7fd08401abc0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 --2- 172.21.2.117:0/3362710597 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd0a40a0aa0 0x7fd0a419fd40 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 >> 172.21.2.117:0/3362710597 conn(0x7fd0a40a5e90 msgr2=0x7fd0a40b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 shutdown_connections 2021-02-10T11:57:02.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.418 7fd0a9f83700 1 -- 172.21.2.117:0/3362710597 wait complete. 2021-02-10T11:57:02.429 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:57:02.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 Processor -- start 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 -- start start 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f05f80e4e40 0x7f05f81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f80e5240 0x7f05f8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f8085930 con 0x7f05f81b0810 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f8084840 con 0x7f05f80e4e40 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.842 7f05ff0d1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f8084a30 con 0x7f05f80e5240 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fd6ee700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f05f81b0810 0x7f05f8085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60336/0 (socket says 172.21.2.117:60336) 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fd6ee700 1 -- 172.21.2.117:0/420056950 learned_addr learned my addr 172.21.2.117:0/420056950 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 --2- 172.21.2.117:0/420056950 >> v2:172.21.2.117:3300/0 conn(0x7f05f80e4e40 0x7f05f81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f7fff700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f80e5240 0x7f05f8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f77fe700 1 -- 172.21.2.117:0/420056950 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778540270 0 0) 0x7f05f0000c10 con 0x7f05f81b0810 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f77fe700 1 -- 172.21.2.117:0/420056950 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4002fe0 con 0x7f05f81b0810 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 -- 172.21.2.117:0/420056950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f80e5240 msgr2=0x7f05f8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f80e5240 0x7f05f8084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 -- 172.21.2.117:0/420056950 >> v1:172.21.2.117:6789/0 conn(0x7f05f81b0810 legacy=0x7f05f8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 -- 172.21.2.117:0/420056950 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f8085280 con 0x7f05f80e4e40 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05fceed700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f80e4e40 0x7f05f81b01f0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f05e0006650 tx=0x7f05e0009a80).ready entity=mon.0 client_cookie=41df1081afee5419 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f77fe700 1 -- 172.21.2.117:0/420056950 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05e000c0c0 con 0x7f05f80e4e40 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f77fe700 1 -- 172.21.2.117:0/420056950 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e000c340 con 0x7f05f80e4e40 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05f77fe700 1 -- 172.21.2.117:0/420056950 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05e000c860 con 0x7f05f80e4e40 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05ff0d1700 1 -- 172.21.2.117:0/420056950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f80e4e40 msgr2=0x7f05f81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.843 7f05ff0d1700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f80e4e40 0x7f05f81b01f0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 rx=0x7f05e0006650 tx=0x7f05e0009a80).stop 2021-02-10T11:57:02.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 -- 172.21.2.117:0/420056950 shutdown_connections 2021-02-10T11:57:02.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f80e5240 0x7f05f8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 --2- 172.21.2.117:0/420056950 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f80e4e40 0x7f05f81b01f0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 -- 172.21.2.117:0/420056950 >> 172.21.2.117:0/420056950 conn(0x7f05f80ce4e0 msgr2=0x7f05f80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:02.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 -- 172.21.2.117:0/420056950 shutdown_connections 2021-02-10T11:57:02.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.844 7f05ff0d1700 1 -- 172.21.2.117:0/420056950 wait complete. 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.859 7f05ff0d1700 1 Processor -- start 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.859 7f05ff0d1700 1 -- start start 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05f80abe50 0x7f05f808a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 0x7f05f808aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81a7a80 con 0x7f05f81a7060 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81a7ca0 con 0x7f05f80abe50 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81a7ec0 con 0x7f05f808a8a0 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60094/0 (socket says 172.21.2.117:60094) 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 -- 172.21.2.117:0/2624533775 learned_addr learned my addr 172.21.2.117:0/2624533775 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fceed700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05f80abe50 0x7f05f808a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05f7fff700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 0x7f05f808aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 -- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 msgr2=0x7f05f808aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 0x7f05f808aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 -- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05f80abe50 msgr2=0x7f05f808a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05f80abe50 0x7f05f808a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 -- 172.21.2.117:0/2624533775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f80b7030 con 0x7f05f81a7060 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05f7fff700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 0x7f05f808aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05fd6ee700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f05f0001400 tx=0x7f05f000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05f000acf0 con 0x7f05f81a7060 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0000c10 con 0x7f05f81a7060 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f80b7030 con 0x7f05f81a7060 2021-02-10T11:57:02.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f05f0004590 con 0x7f05f81a7060 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.860 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f80b7030 con 0x7f05f81a7060 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.861 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f05f00043a0 con 0x7f05f81a7060 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.861 7f05f57fa700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 0x7f05dc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.861 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f05f0032630 con 0x7f05f81a7060 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.861 7f05fceed700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 0x7f05dc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:02.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.862 7f05fceed700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 0x7f05dc01abc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:02.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.891 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f810c2b0 con 0x7f05f81a7060 2021-02-10T11:57:02.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:02.894 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f05f000acf0 con 0x7f05f81a7060 2021-02-10T11:57:03.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.023 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05f810c2b0 con 0x7f05f81a7060 2021-02-10T11:57:03.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.387 7f05f57fa700 1 -- 172.21.2.117:0/2624533775 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v182) v1 ==== 173+0+0 (secure 0 0 0) 0x7f05f00043a0 con 0x7f05f81a7060 2021-02-10T11:57:03.392 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.391 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 msgr2=0x7f05dc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.391 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 0x7f05dc01abc0 crc :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.391 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 msgr2=0x7f05f81a7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.391 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 rx=0x7f05f0001400 tx=0x7f05f000a900).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 shutdown_connections 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f05dc018700 0x7f05dc01abc0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f05f81a7060 0x7f05f81a7460 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f05f808a8a0 0x7f05f808aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 --2- 172.21.2.117:0/2624533775 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f05f80abe50 0x7f05f808a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 >> 172.21.2.117:0/2624533775 conn(0x7f05f80acec0 msgr2=0x7f05f80b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 shutdown_connections 2021-02-10T11:57:03.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.392 7f05ff0d1700 1 -- 172.21.2.117:0/2624533775 wait complete. 2021-02-10T11:57:03.403 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:57:03.403 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:57:03.404 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:57:03.404 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:57:03.404 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:57:03.404 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:57:03.404 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:57:03.404 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:57:03.409 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:57:03.410 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:57:03.410 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:57:03.411 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:57:03.452 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:57:03.452 DEBUG:teuthology.parallel:result is None 2021-02-10T11:57:03.453 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:57:03.453 DEBUG:teuthology.parallel:result is None 2021-02-10T11:57:03.453 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:57:03.858 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:57:03.863 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:57:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.863 7f3e59e1e700 1 Processor -- start 2021-02-10T11:57:03.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.863 7f3e59e1e700 1 -- start start 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e59e1e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 0x7f3e540e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e59e1e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540e7580 con 0x7f3e541b07f0 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e59e1e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540e77f0 con 0x7f3e540e8390 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e59e1e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540e8f80 con 0x7f3e540e7180 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e537fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3e541b07f0 0x7f3e540e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60360/0 (socket says 172.21.2.117:60360) 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e537fe700 1 -- 172.21.2.117:0/2462430354 learned_addr learned my addr 172.21.2.117:0/2462430354 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e52ffd700 1 --2- 172.21.2.117:0/2462430354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 0x7f3e540e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677013921 0 0) 0x7f3e440009b0 con 0x7f3e541b07f0 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40002fe0 con 0x7f3e541b07f0 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169429552 0 0) 0x7f3e4c001400 con 0x7f3e540e8390 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40004370 con 0x7f3e540e8390 2021-02-10T11:57:03.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e52ffd700 1 -- 172.21.2.117:0/2462430354 >> v1:172.21.2.125:6789/0 conn(0x7f3e540e8390 legacy=0x7f3e540e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.864 7f3e52ffd700 1 -- 172.21.2.117:0/2462430354 >> v1:172.21.2.117:6789/0 conn(0x7f3e541b07f0 legacy=0x7f3e540e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e52ffd700 1 -- 172.21.2.117:0/2462430354 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540e9170 con 0x7f3e540e7180 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e52ffd700 1 --2- 172.21.2.117:0/2462430354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 0x7f3e540e7d70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3e3c006650 tx=0x7f3e3c009a00).ready entity=mon.2 client_cookie=183c6338683af258 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c0009b0 con 0x7f3e540e7180 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c00c0c0 con 0x7f3e540e7180 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e527fc700 1 -- 172.21.2.117:0/2462430354 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e3c00c5a0 con 0x7f3e540e7180 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e59e1e700 1 -- 172.21.2.117:0/2462430354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 msgr2=0x7f3e540e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.865 7f3e59e1e700 1 --2- 172.21.2.117:0/2462430354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 0x7f3e540e7d70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3e3c006650 tx=0x7f3e3c009a00).stop 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.866 7f3e59e1e700 1 -- 172.21.2.117:0/2462430354 shutdown_connections 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.866 7f3e59e1e700 1 --2- 172.21.2.117:0/2462430354 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540e7180 0x7f3e540e7d70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.866 7f3e59e1e700 1 -- 172.21.2.117:0/2462430354 >> 172.21.2.117:0/2462430354 conn(0x7f3e540ce500 msgr2=0x7f3e540e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:03.885 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.866 7f3e59e1e700 1 -- 172.21.2.117:0/2462430354 shutdown_connections 2021-02-10T11:57:03.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.866 7f3e59e1e700 1 -- 172.21.2.117:0/2462430354 wait complete. 2021-02-10T11:57:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 Processor -- start 2021-02-10T11:57:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 -- start start 2021-02-10T11:57:03.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540af5f0 0x7f3e5408bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e5408c380 0x7f3e54199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5408cca0 con 0x7f3e5408c780 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54199c90 con 0x7f3e5408c380 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e59e1e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54199e70 con 0x7f3e540af5f0 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e53fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e53fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60120/0 (socket says 172.21.2.117:60120) 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.881 7f3e53fff700 1 -- 172.21.2.117:0/3089209388 learned_addr learned my addr 172.21.2.117:0/3089209388 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e52ffd700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e5408c380 0x7f3e54199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e537fe700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540af5f0 0x7f3e5408bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 -- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540af5f0 msgr2=0x7f3e5408bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540af5f0 0x7f3e5408bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 -- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e5408c380 msgr2=0x7f3e54199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e5408c380 0x7f3e54199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 -- 172.21.2.117:0/3089209388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5409d790 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e53fff700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7f3e4c009940 tx=0x7f3e4c00bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c00c2f0 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c00c570 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5409d790 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5409dd20 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.882 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3e4c00ca50 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.883 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3e4c00c750 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.883 7f3e4bfff700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 0x7f3e3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.883 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5186+0+0 (secure 0 0 0) 0x7f3e4c032a40 con 0x7f3e5408c780 2021-02-10T11:57:03.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.883 7f3e537fe700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 0x7f3e3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:03.890 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.884 7f3e537fe700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 0x7f3e3801ac10 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:03.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.913 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e54106e10 con 0x7f3e5408c780 2021-02-10T11:57:03.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:03.915 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3e4c00c780 con 0x7f3e5408c780 2021-02-10T11:57:04.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.054 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f3e54106e10 con 0x7f3e5408c780 2021-02-10T11:57:04.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.389 7f3e4bfff700 1 -- 172.21.2.117:0/3089209388 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v183) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3e4c00c570 con 0x7f3e5408c780 2021-02-10T11:57:04.423 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 msgr2=0x7f3e3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 0x7f3e3801ac10 crc :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 msgr2=0x7f3e54199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 rx=0x7f3e4c009940 tx=0x7f3e4c00bcf0).stop 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 shutdown_connections 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e38018730 0x7f3e3801ac10 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3e5408c780 0x7f3e54199670 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3e5408c380 0x7f3e54199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 --2- 172.21.2.117:0/3089209388 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3e540af5f0 0x7f3e5408bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 >> 172.21.2.117:0/3089209388 conn(0x7f3e540953a0 msgr2=0x7f3e5409e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:04.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.422 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 shutdown_connections 2021-02-10T11:57:04.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.423 7f3e59e1e700 1 -- 172.21.2.117:0/3089209388 wait complete. 2021-02-10T11:57:04.434 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:57:04.434 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:57:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.845 7f7a58857700 1 Processor -- start 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.845 7f7a58857700 1 -- start start 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.846 7f7a58857700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.846 7f7a58857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.846 7f7a58857700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a540ede60 con 0x7f7a540eb000 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.846 7f7a58857700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a540ef7f0 con 0x7f7a540d18e0 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.846 7f7a58857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540d1ce0 con 0x7f7a540ef3f0 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a5261c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57804/0 (socket says 172.21.2.117:57804) 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a5261c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36276/0 (socket says 172.21.2.117:36276) 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 -- 172.21.2.117:0/748762914 learned_addr learned my addr 172.21.2.117:0/748762914 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 -- 172.21.2.117:0/748762914 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 msgr2=0x7f7a540ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 --2- 172.21.2.117:0/748762914 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 -- 172.21.2.117:0/748762914 >> v1:172.21.2.125:6789/0 conn(0x7f7a540eb000 legacy=0x7f7a540eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a5161a700 1 -- 172.21.2.117:0/748762914 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752077208 0 0) 0x7f7a3c000ff0 con 0x7f7a540eb000 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 -- 172.21.2.117:0/748762914 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a541b8f90 con 0x7f7a540ef3f0 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a5261c700 1 --2- 172.21.2.117:0/748762914 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.847 7f7a52e1d700 1 --2- 172.21.2.117:0/748762914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7a4c006aa0 tx=0x7f7a4c009f20).ready entity=mon.2 client_cookie=976e18ab1c75094d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a5161a700 1 -- 172.21.2.117:0/748762914 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a4c00b5c0 con 0x7f7a540ef3f0 2021-02-10T11:57:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a5161a700 1 -- 172.21.2.117:0/748762914 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c003030 con 0x7f7a540ef3f0 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a5161a700 1 -- 172.21.2.117:0/748762914 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a4c00b9f0 con 0x7f7a540ef3f0 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 -- 172.21.2.117:0/748762914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 msgr2=0x7f7a541b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 --2- 172.21.2.117:0/748762914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f7a4c006aa0 tx=0x7f7a4c009f20).stop 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 -- 172.21.2.117:0/748762914 shutdown_connections 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 --2- 172.21.2.117:0/748762914 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540ef3f0 0x7f7a541b7960 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 --2- 172.21.2.117:0/748762914 >> v2:172.21.2.125:3300/0 conn(0x7f7a540d18e0 0x7f7a540ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.848 7f7a58857700 1 -- 172.21.2.117:0/748762914 >> 172.21.2.117:0/748762914 conn(0x7f7a540ced60 msgr2=0x7f7a540cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.849 7f7a58857700 1 -- 172.21.2.117:0/748762914 shutdown_connections 2021-02-10T11:57:04.868 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.849 7f7a58857700 1 -- 172.21.2.117:0/748762914 wait complete. 2021-02-10T11:57:04.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 Processor -- start 2021-02-10T11:57:04.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 -- start start 2021-02-10T11:57:04.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540b77e0 0x7f7a540b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540ab2c0 0x7f7a540abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540ab8c0 con 0x7f7a540b7310 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540ac460 con 0x7f7a540ab2c0 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a58857700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540ac680 con 0x7f7a540b77e0 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a51e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.863 7f7a51e1b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60128/0 (socket says 172.21.2.117:60128) 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 -- 172.21.2.117:0/3671605034 learned_addr learned my addr 172.21.2.117:0/3671605034 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 -- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540b77e0 msgr2=0x7f7a540b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540b77e0 0x7f7a540b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 -- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540ab2c0 msgr2=0x7f7a540abe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540ab2c0 0x7f7a540abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:04.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 -- 172.21.2.117:0/3671605034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540ac8a0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a51e1b700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7f7a3c00a570 tx=0x7f7a3c00a910).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a3c00c0d0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c00c3d0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a58857700 1 -- 172.21.2.117:0/3671605034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a540ac8a0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7a3c00c860 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.864 7f7a58857700 1 -- 172.21.2.117:0/3671605034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a540ac8a0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.865 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7a3c006540 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.865 7f7a4affd700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 0x7f7a3801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.865 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f7a3c0325d0 con 0x7f7a540b7310 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.865 7f7a5261c700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 0x7f7a3801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:04.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.866 7f7a5261c700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 0x7f7a3801ac10 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:04.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.895 7f7a58857700 1 -- 172.21.2.117:0/3671605034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54087290 con 0x7f7a540b7310 2021-02-10T11:57:04.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:04.898 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7a3c000990 con 0x7f7a540b7310 2021-02-10T11:57:05.032 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.029 7f7a58857700 1 -- 172.21.2.117:0/3671605034 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7a540ede60 con 0x7f7a540b7310 2021-02-10T11:57:05.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.030 7f7a4affd700 1 -- 172.21.2.117:0/3671605034 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v107) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f7a3c006540 con 0x7f7a540b7310 2021-02-10T11:57:05.033 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:05.033 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.034 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.035 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.036 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:57:05.037 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.038 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.039 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.040 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:57:05.041 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:57:05.042 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:05.043 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:57:05.044 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:05.045 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:05.046 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:05.047 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:57:05.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 msgr2=0x7f7a3801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 0x7f7a3801ac10 crc :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 msgr2=0x7f7a540aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 rx=0x7f7a3c00a570 tx=0x7f7a3c00a910).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 shutdown_connections 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7a38018730 0x7f7a3801ac10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7a540b7310 0x7f7a540aaca0 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7a540ab2c0 0x7f7a540abe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 --2- 172.21.2.117:0/3671605034 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7a540b77e0 0x7f7a540b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 >> 172.21.2.117:0/3671605034 conn(0x7f7a540874c0 msgr2=0x7f7a540af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 shutdown_connections 2021-02-10T11:57:05.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.032 7f7a58857700 1 -- 172.21.2.117:0/3671605034 wait complete. 2021-02-10T11:57:05.050 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.461 7faa28da3700 1 Processor -- start 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.461 7faa28da3700 1 -- start start 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa28da3700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7faa240ef3f0 0x7faa240eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa28da3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa28da3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa240ede60 con 0x7faa240d18e0 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa28da3700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa240ef7f0 con 0x7faa240ef3f0 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa28da3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240d1ce0 con 0x7faa240eb7f0 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.481 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57818/0 (socket says 172.21.2.117:57818) 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 -- 172.21.2.117:0/2726095174 learned_addr learned my addr 172.21.2.117:0/2726095174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa22312700 1 --2- 172.21.2.117:0/2726095174 >> v2:172.21.2.125:3300/0 conn(0x7faa240ef3f0 0x7faa240eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 -- 172.21.2.117:0/2726095174 >> v2:172.21.2.125:3300/0 conn(0x7faa240ef3f0 msgr2=0x7faa240eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 --2- 172.21.2.117:0/2726095174 >> v2:172.21.2.125:3300/0 conn(0x7faa240ef3f0 0x7faa240eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 -- 172.21.2.117:0/2726095174 >> v1:172.21.2.125:6789/0 conn(0x7faa240d18e0 legacy=0x7faa240ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.462 7faa23314700 1 -- 172.21.2.117:0/2726095174 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa241b8f90 con 0x7faa240eb7f0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa23314700 1 --2- 172.21.2.117:0/2726095174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7faa14006720 tx=0x7faa14009ba0).ready entity=mon.2 client_cookie=fe402643522370f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa21b11700 1 -- 172.21.2.117:0/2726095174 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa140022e0 con 0x7faa240eb7f0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa21b11700 1 -- 172.21.2.117:0/2726095174 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14002660 con 0x7faa240eb7f0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa21b11700 1 -- 172.21.2.117:0/2726095174 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa14002ab0 con 0x7faa240eb7f0 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa28da3700 1 -- 172.21.2.117:0/2726095174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 msgr2=0x7faa241b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.463 7faa28da3700 1 --2- 172.21.2.117:0/2726095174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7faa14006720 tx=0x7faa14009ba0).stop 2021-02-10T11:57:05.482 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 -- 172.21.2.117:0/2726095174 shutdown_connections 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 --2- 172.21.2.117:0/2726095174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa240eb7f0 0x7faa241b7960 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 --2- 172.21.2.117:0/2726095174 >> v2:172.21.2.125:3300/0 conn(0x7faa240ef3f0 0x7faa240eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 -- 172.21.2.117:0/2726095174 >> 172.21.2.117:0/2726095174 conn(0x7faa240ced60 msgr2=0x7faa240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 -- 172.21.2.117:0/2726095174 shutdown_connections 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.464 7faa28da3700 1 -- 172.21.2.117:0/2726095174 wait complete. 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 Processor -- start 2021-02-10T11:57:05.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 -- start start 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa24091d40 0x7faa240e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 0x7faa241b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa24197cd0 con 0x7faa24091d40 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa24092140 con 0x7faa24198250 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.478 7faa28da3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa241b2ed0 con 0x7faa241b24b0 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa22b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa24091d40 0x7faa240e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36292/0 (socket says 172.21.2.117:36292) 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 -- 172.21.2.117:0/4236849138 learned_addr learned my addr 172.21.2.117:0/4236849138 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa22312700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 0x7faa241b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 -- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 msgr2=0x7faa241b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.485 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 0x7faa241b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 -- 172.21.2.117:0/4236849138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa24091d40 msgr2=0x7faa240e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa24091d40 0x7faa240e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 -- 172.21.2.117:0/4236849138 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24198c70 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa22312700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 0x7faa241b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa23314700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7faa140095c0 tx=0x7faa1400b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa1400bd80 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1400bd80 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa28da3700 1 -- 172.21.2.117:0/4236849138 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa24198c70 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.479 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7faa14006240 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.480 7faa28da3700 1 -- 172.21.2.117:0/4236849138 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa24198c70 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.480 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7faa14006240 con 0x7faa24198250 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.480 7faa137fe700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 0x7faa0801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:05.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.480 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5100+0+0 (secure 0 0 0) 0x7faa14032450 con 0x7faa24198250 2021-02-10T11:57:05.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.480 7faa22b13700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 0x7faa0801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:05.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.481 7faa22b13700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 0x7faa0801abc0 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:05.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.512 7faa28da3700 1 -- 172.21.2.117:0/4236849138 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa24087290 con 0x7faa24198250 2021-02-10T11:57:05.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.515 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7faa14006030 con 0x7faa24198250 2021-02-10T11:57:05.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.640 7faa28da3700 1 -- 172.21.2.117:0/4236849138 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa240ede60 con 0x7faa24198250 2021-02-10T11:57:05.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.641 7faa137fe700 1 -- 172.21.2.117:0/4236849138 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+11387 (secure 0 0 0) 0x7faa14006240 con 0x7faa24198250 2021-02-10T11:57:05.644 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:05.645 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":183,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:04.387936","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:05.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 msgr2=0x7faa0801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 0x7faa0801abc0 crc :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 msgr2=0x7faa24198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7faa140095c0 tx=0x7faa1400b990).stop 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 shutdown_connections 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7faa24198250 0x7faa24198650 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7faa241b24b0 0x7faa241b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7faa080186c0 0x7faa0801abc0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 --2- 172.21.2.117:0/4236849138 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7faa24091d40 0x7faa240e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:05.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 >> 172.21.2.117:0/4236849138 conn(0x7faa241a3620 msgr2=0x7faa240dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:05.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 shutdown_connections 2021-02-10T11:57:05.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:05.643 7faa28da3700 1 -- 172.21.2.117:0/4236849138 wait complete. 2021-02-10T11:57:05.655 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:57:05.655 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:57:06.088 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.068 7fa045005700 1 Processor -- start 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 -- start start 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0400d18f0 0x7fa0400eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0400ef3f0 0x7fa0401b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0400ef7f0 con 0x7fa0400eb040 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0400eded0 con 0x7fa0400d18f0 2021-02-10T11:57:06.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa045005700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400d1cf0 con 0x7fa0400ef3f0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa03ed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0400d18f0 0x7fa0400eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa03e59c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa0400eb040 0x7fa0400eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60398/0 (socket says 172.21.2.117:60398) 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa03ed9d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa0400d18f0 0x7fa0400eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60148/0 (socket says 172.21.2.117:60148) 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa03e59c700 1 -- 172.21.2.117:0/2066574465 learned_addr learned my addr 172.21.2.117:0/2066574465 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.069 7fa03f59e700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0400ef3f0 0x7fa0401b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03ed9d700 1 -- 172.21.2.117:0/2066574465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0400ef3f0 msgr2=0x7fa0401b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03ed9d700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0400ef3f0 0x7fa0401b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03ed9d700 1 -- 172.21.2.117:0/2066574465 >> v1:172.21.2.117:6789/0 conn(0x7fa0400eb040 legacy=0x7fa0400eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03ed9d700 1 -- 172.21.2.117:0/2066574465 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0401b8fd0 con 0x7fa0400d18f0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03ed9d700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0400d18f0 0x7fa0400eaa20 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7fa028006650 tx=0x7fa028009a80).ready entity=mon.0 client_cookie=59ab013d6d1236da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03dd9b700 1 -- 172.21.2.117:0/2066574465 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa02800c0c0 con 0x7fa0400d18f0 2021-02-10T11:57:06.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03dd9b700 1 -- 172.21.2.117:0/2066574465 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02800c340 con 0x7fa0400d18f0 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa03dd9b700 1 -- 172.21.2.117:0/2066574465 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa02800c860 con 0x7fa0400d18f0 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa045005700 1 -- 172.21.2.117:0/2066574465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0400d18f0 msgr2=0x7fa0400eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.070 7fa045005700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0400d18f0 0x7fa0400eaa20 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 rx=0x7fa028006650 tx=0x7fa028009a80).stop 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 -- 172.21.2.117:0/2066574465 shutdown_connections 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa0400ef3f0 0x7fa0401b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 --2- 172.21.2.117:0/2066574465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0400d18f0 0x7fa0400eaa20 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 -- 172.21.2.117:0/2066574465 >> 172.21.2.117:0/2066574465 conn(0x7fa0400cf060 msgr2=0x7fa0400cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 -- 172.21.2.117:0/2066574465 shutdown_connections 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.071 7fa045005700 1 -- 172.21.2.117:0/2066574465 wait complete. 2021-02-10T11:57:06.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.085 7fa045005700 1 Processor -- start 2021-02-10T11:57:06.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.085 7fa045005700 1 -- start start 2021-02-10T11:57:06.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa040093be0 0x7fa04008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.094 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa04008ce70 0x7fa0400945e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040094060 con 0x7fa0401a4880 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400873f0 con 0x7fa04008ce70 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa045005700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400b5ee0 con 0x7fa040093be0 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60154/0 (socket says 172.21.2.117:60154) 2021-02-10T11:57:06.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 -- 172.21.2.117:0/1922450838 learned_addr learned my addr 172.21.2.117:0/1922450838 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03ed9d700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa040093be0 0x7fa04008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03f59e700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa04008ce70 0x7fa0400945e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 -- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa040093be0 msgr2=0x7fa04008c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa040093be0 0x7fa04008c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 -- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa04008ce70 msgr2=0x7fa0400945e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa04008ce70 0x7fa0400945e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 -- 172.21.2.117:0/1922450838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040094c00 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa03e59c700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7fa030001c60 tx=0x7fa03000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.086 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa03000acc0 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa030004140 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa030004590 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa045005700 1 -- 172.21.2.117:0/1922450838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa040094c00 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa045005700 1 -- 172.21.2.117:0/1922450838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa040094c00 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa030000ce0 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa0377fe700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 0x7fa02401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.087 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fa030032450 con 0x7fa0401a4880 2021-02-10T11:57:06.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.088 7fa03ed9d700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 0x7fa02401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.088 7fa03ed9d700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 0x7fa02401abd0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.119 7fa045005700 1 -- 172.21.2.117:0/1922450838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0400ede60 con 0x7fa0401a4880 2021-02-10T11:57:06.124 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.121 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa0300009b0 con 0x7fa0401a4880 2021-02-10T11:57:06.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.247 7fa045005700 1 -- 172.21.2.117:0/1922450838 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fa0400ede60 con 0x7fa0401a4880 2021-02-10T11:57:06.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.248 7fa0377fe700 1 -- 172.21.2.117:0/1922450838 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v183) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa030000ce0 con 0x7fa0401a4880 2021-02-10T11:57:06.251 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:06.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 msgr2=0x7fa02401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 0x7fa02401abd0 crc :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 msgr2=0x7fa04008c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 rx=0x7fa030001c60 tx=0x7fa03000a8d0).stop 2021-02-10T11:57:06.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 shutdown_connections 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa0240186d0 0x7fa02401abd0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa0401a4880 0x7fa04008c850 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa04008ce70 0x7fa0400945e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 --2- 172.21.2.117:0/1922450838 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa040093be0 0x7fa04008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 >> 172.21.2.117:0/1922450838 conn(0x7fa0400937a0 msgr2=0x7fa04008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 shutdown_connections 2021-02-10T11:57:06.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.250 7fa045005700 1 -- 172.21.2.117:0/1922450838 wait complete. 2021-02-10T11:57:06.262 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:57:06.691 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 Processor -- start 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 -- start start 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80d18e0 0x7f33b80ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33b80ef3f0 0x7f33b81b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33b80ede60 con 0x7f33b80eb000 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33b8088020 con 0x7f33b80ef3f0 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33c00c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80d1ce0 con 0x7f33b80d18e0 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.672 7f33be6dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33b80ef3f0 0x7f33b81b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f33b80ef3f0 0x7f33b81b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60160/0 (socket says 172.21.2.117:60160) 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 -- 172.21.2.117:0/3200707438 learned_addr learned my addr 172.21.2.117:0/3200707438 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 -- 172.21.2.117:0/3200707438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80d18e0 msgr2=0x7f33b80ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:06.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 --2- 172.21.2.117:0/3200707438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80d18e0 0x7f33b80ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 -- 172.21.2.117:0/3200707438 >> v1:172.21.2.117:6789/0 conn(0x7f33b80eb000 legacy=0x7f33b80eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 -- 172.21.2.117:0/3200707438 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b81b8f90 con 0x7f33b80ef3f0 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33be6dd700 1 --2- 172.21.2.117:0/3200707438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80ef3f0 0x7f33b81b7960 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f33b0006f00 tx=0x7f33b0009f10).ready entity=mon.0 client_cookie=c1f2c4b2cad78a87 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33bceda700 1 -- 172.21.2.117:0/3200707438 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000b3b0 con 0x7f33b80ef3f0 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33bceda700 1 -- 172.21.2.117:0/3200707438 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b000b630 con 0x7f33b80ef3f0 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33bceda700 1 -- 172.21.2.117:0/3200707438 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33b000bb50 con 0x7f33b80ef3f0 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33c00c0700 1 -- 172.21.2.117:0/3200707438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80ef3f0 msgr2=0x7f33b81b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.673 7f33c00c0700 1 --2- 172.21.2.117:0/3200707438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80ef3f0 0x7f33b81b7960 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 rx=0x7f33b0006f00 tx=0x7f33b0009f10).stop 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 -- 172.21.2.117:0/3200707438 shutdown_connections 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 --2- 172.21.2.117:0/3200707438 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80ef3f0 0x7f33b81b7960 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 --2- 172.21.2.117:0/3200707438 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80d18e0 0x7f33b80ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 -- 172.21.2.117:0/3200707438 >> 172.21.2.117:0/3200707438 conn(0x7f33b80ced60 msgr2=0x7f33b80cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:06.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 -- 172.21.2.117:0/3200707438 shutdown_connections 2021-02-10T11:57:06.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.674 7f33c00c0700 1 -- 172.21.2.117:0/3200707438 wait complete. 2021-02-10T11:57:06.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.688 7f33c00c0700 1 Processor -- start 2021-02-10T11:57:06.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 -- start start 2021-02-10T11:57:06.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33b8196390 0x7f33b80e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80e9960 0x7f33b80e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.695 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8196810 con 0x7f33b80e9560 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8098ad0 con 0x7f33b8196390 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33c00c0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33b80e8a40 con 0x7f33b80e9960 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60168/0 (socket says 172.21.2.117:60168) 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 -- 172.21.2.117:0/1390285829 learned_addr learned my addr 172.21.2.117:0/1390285829 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 -- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80e9960 msgr2=0x7f33b80e8420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80e9960 0x7f33b80e8420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 -- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33b8196390 msgr2=0x7f33b80e8f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33b8196390 0x7f33b80e8f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.689 7f33bd6db700 1 -- 172.21.2.117:0/1390285829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80e8c60 con 0x7f33b80e9560 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33bd6db700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7f33ac006ed0 tx=0x7f33ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.696 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33ac006020 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac006360 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f33ac005730 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b80e8c60 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.690 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b819cd70 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.691 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f33ac0054b0 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.691 7f33aa7fc700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 0x7f33a001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.691 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f33ac0323b0 con 0x7f33b80e9560 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.691 7f33bdedc700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 0x7f33a001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:06.697 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.692 7f33bdedc700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 0x7f33a001abc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:06.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.722 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b8087210 con 0x7f33b80e9560 2021-02-10T11:57:06.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.725 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f33ac021730 con 0x7f33b80e9560 2021-02-10T11:57:06.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.847 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f33b80ede60 con 0x7f33b80e9560 2021-02-10T11:57:06.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.848 7f33aa7fc700 1 -- 172.21.2.117:0/1390285829 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v183) v1 ==== 84+0+64 (secure 0 0 0) 0x7f33ac021730 con 0x7f33b80e9560 2021-02-10T11:57:06.851 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:06.852 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.849 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 msgr2=0x7f33a001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.849 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 0x7f33a001abc0 crc :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.849 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 msgr2=0x7f33b80e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.849 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 rx=0x7f33ac006ed0 tx=0x7f33ac00b960).stop 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 shutdown_connections 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33a00186c0 0x7f33a001abc0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33b80e9560 0x7f33b80e7e00 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:06.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33b80e9960 0x7f33b80e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 --2- 172.21.2.117:0/1390285829 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33b8196390 0x7f33b80e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:06.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 >> 172.21.2.117:0/1390285829 conn(0x7f33b819f610 msgr2=0x7f33b81a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:06.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 shutdown_connections 2021-02-10T11:57:06.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:06.850 7f33c00c0700 1 -- 172.21.2.117:0/1390285829 wait complete. 2021-02-10T11:57:06.861 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:57:06.861 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:57:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 Processor -- start 2021-02-10T11:57:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 -- start start 2021-02-10T11:57:07.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa1e81b07f0 0x7fa1e80e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e80e8d80 0x7fa1e80e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80e77f0 con 0x7fa1e80e6170 2021-02-10T11:57:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80e54b0 con 0x7fa1e81b07f0 2021-02-10T11:57:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.275 7fa1ee5f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80e9180 con 0x7fa1e80e8d80 2021-02-10T11:57:07.302 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e77fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa1e80e6170 0x7fa1e80e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60426/0 (socket says 172.21.2.117:60426) 2021-02-10T11:57:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e77fe700 1 -- 172.21.2.117:0/549502511 learned_addr learned my addr 172.21.2.117:0/549502511 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177903155 0 0) 0x7fa1dc0009b0 con 0x7fa1e80e6170 2021-02-10T11:57:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e7fff700 1 --2- 172.21.2.117:0/549502511 >> v2:172.21.2.125:3300/0 conn(0x7fa1e81b07f0 0x7fa1e80e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0002fe0 con 0x7fa1e80e6170 2021-02-10T11:57:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1ecc13700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e80e8d80 0x7fa1e80e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 368733588 0 0) 0x7fa1dc001010 con 0x7fa1e80e6170 2021-02-10T11:57:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e80e8d80 msgr2=0x7fa1e80e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e80e8d80 0x7fa1e80e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 >> v2:172.21.2.125:3300/0 conn(0x7fa1e81b07f0 msgr2=0x7fa1e80e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 --2- 172.21.2.117:0/549502511 >> v2:172.21.2.125:3300/0 conn(0x7fa1e81b07f0 0x7fa1e80e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80e7380 con 0x7fa1e80e6170 2021-02-10T11:57:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa1dc001590 con 0x7fa1e80e6170 2021-02-10T11:57:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 >> v1:172.21.2.117:6789/0 conn(0x7fa1e80e6170 legacy=0x7fa1e80e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1d0003a60 0x7fa1d0005e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1d004a740 0x7fa1d004cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1d004d500 0x7fa1d004fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d00502d0 con 0x7fa1d004d500 2021-02-10T11:57:07.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d00504f0 con 0x7fa1d004a740 2021-02-10T11:57:07.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.276 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d0050710 con 0x7fa1d0003a60 2021-02-10T11:57:07.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1d004d500 0x7fa1d004fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e77fe700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1d004a740 0x7fa1d004cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e7fff700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1d0003a60 0x7fa1d0005e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 -- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1d0003a60 msgr2=0x7fa1d0005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1d0003a60 0x7fa1d0005e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 -- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1d004a740 msgr2=0x7fa1d004cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1d004a740 0x7fa1d004cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 -- 172.21.2.117:0/549502511 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d00509b0 con 0x7fa1d004d500 2021-02-10T11:57:07.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e77fe700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1d004a740 0x7fa1d004cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:07.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1d004d500 0x7fa1d004fa20 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7fa1e0009a40 tx=0x7fa1e0009de0).ready entity=mon.0 client_cookie=f629d3018f149192 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:07.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e0006a90 con 0x7fa1d004d500 2021-02-10T11:57:07.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0003030 con 0x7fa1d004d500 2021-02-10T11:57:07.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1e6ffd700 1 -- 172.21.2.117:0/549502511 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1e0006a90 con 0x7fa1d004d500 2021-02-10T11:57:07.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ee5f6700 1 -- 172.21.2.117:0/549502511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1d004d500 msgr2=0x7fa1d004fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ee5f6700 1 --2- 172.21.2.117:0/549502511 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1d004d500 0x7fa1d004fa20 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 rx=0x7fa1e0009a40 tx=0x7fa1e0009de0).stop 2021-02-10T11:57:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.277 7fa1ecc13700 1 -- 172.21.2.117:0/549502511 reap_dead start 2021-02-10T11:57:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.278 7fa1ee5f6700 1 -- 172.21.2.117:0/549502511 shutdown_connections 2021-02-10T11:57:07.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.278 7fa1ee5f6700 1 -- 172.21.2.117:0/549502511 >> 172.21.2.117:0/549502511 conn(0x7fa1e80ce500 msgr2=0x7fa1e80e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.278 7fa1ee5f6700 1 -- 172.21.2.117:0/549502511 shutdown_connections 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.278 7fa1ee5f6700 1 -- 172.21.2.117:0/549502511 wait complete. 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 Processor -- start 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 -- start start 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 0x7fa1e81111b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1e80c0950 0x7fa1e80c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c4040 con 0x7fa1e80c0950 2021-02-10T11:57:07.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c1370 con 0x7fa1e8110db0 2021-02-10T11:57:07.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1ee5f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80c1590 con 0x7fa1e81a2df0 2021-02-10T11:57:07.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1e7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.293 7fa1e77fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 0x7fa1e81111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e7fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57864/0 (socket says 172.21.2.117:57864) 2021-02-10T11:57:07.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e7fff700 1 -- 172.21.2.117:0/3397099089 learned_addr learned my addr 172.21.2.117:0/3397099089 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:07.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1ecc13700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1e80c0950 0x7fa1e80c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 -- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 msgr2=0x7fa1e81a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 -- 172.21.2.117:0/3397099089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1e80c0950 msgr2=0x7fa1e80c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1e80c0950 0x7fa1e80c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 -- 172.21.2.117:0/3397099089 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80c17b0 con 0x7fa1e8110db0 2021-02-10T11:57:07.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e7fff700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:07.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e77fe700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 0x7fa1e81111b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7fa1dc00a4c0 tx=0x7fa1dc00a860).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:07.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1dc00ac50 con 0x7fa1e8110db0 2021-02-10T11:57:07.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc004140 con 0x7fa1e8110db0 2021-02-10T11:57:07.315 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa1dc004620 con 0x7fa1e8110db0 2021-02-10T11:57:07.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e80c17b0 con 0x7fa1e8110db0 2021-02-10T11:57:07.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.294 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e80c17b0 con 0x7fa1e8110db0 2021-02-10T11:57:07.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.295 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa1dc0010d0 con 0x7fa1e8110db0 2021-02-10T11:57:07.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.295 7fa1e4ff9700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:07.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.295 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fa1dc032190 con 0x7fa1e8110db0 2021-02-10T11:57:07.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.295 7fa1e7fff700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:07.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.295 7fa1e7fff700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 0x7fa1c801abc0 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:07.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.325 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e810c190 con 0x7fa1e8110db0 2021-02-10T11:57:07.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.328 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa1dc000e80 con 0x7fa1e8110db0 2021-02-10T11:57:07.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.455 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7fa1e810c190 con 0x7fa1e8110db0 2021-02-10T11:57:07.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.868 7fa1e4ff9700 1 -- 172.21.2.117:0/3397099089 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v184) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa1dc0010d0 con 0x7fa1e8110db0 2021-02-10T11:57:07.874 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 msgr2=0x7fa1c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 0x7fa1c801abc0 crc :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 msgr2=0x7fa1e81111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 0x7fa1e81111b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7fa1dc00a4c0 tx=0x7fa1dc00a860).stop 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 shutdown_connections 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa1c80186c0 0x7fa1c801abc0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa1e80c0950 0x7fa1e80c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa1e8110db0 0x7fa1e81111b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 --2- 172.21.2.117:0/3397099089 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa1e81a2df0 0x7fa1e81a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:07.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 >> 172.21.2.117:0/3397099089 conn(0x7fa1e80be140 msgr2=0x7fa1e80b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:07.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 shutdown_connections 2021-02-10T11:57:07.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:07.873 7fa1ee5f6700 1 -- 172.21.2.117:0/3397099089 wait complete. 2021-02-10T11:57:07.884 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:57:08.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.297 7f04dc341700 1 Processor -- start 2021-02-10T11:57:08.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.297 7f04dc341700 1 -- start start 2021-02-10T11:57:08.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04dc341700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f04d40ef3f0 0x7f04d40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04dc341700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04dc341700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d40ef7f0 con 0x7f04d40d18f0 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04dc341700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d40eded0 con 0x7f04d40ef3f0 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04dc341700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40d1cf0 con 0x7f04d40eb830 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04da95e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04d995c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f04d40ef3f0 0x7f04d40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04d995c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f04d40ef3f0 0x7f04d40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36344/0 (socket says 172.21.2.117:36344) 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04da95e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57876/0 (socket says 172.21.2.117:57876) 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.298 7f04d995c700 1 -- 172.21.2.117:0/3950823496 learned_addr learned my addr 172.21.2.117:0/3950823496 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04d995c700 1 -- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 msgr2=0x7f04d41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04d995c700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04d995c700 1 -- 172.21.2.117:0/3950823496 >> v1:172.21.2.125:6789/0 conn(0x7f04d40d18f0 legacy=0x7f04d40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04d995c700 1 -- 172.21.2.117:0/3950823496 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d41b8fd0 con 0x7f04d40ef3f0 2021-02-10T11:57:08.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04da95e700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.299 7f04d995c700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d40ef3f0 0x7f04d40eb210 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f04c8006700 tx=0x7f04c8009b80).ready entity=mon.1 client_cookie=899b4535bb4ebaa8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04d915b700 1 -- 172.21.2.117:0/3950823496 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c8006340 con 0x7f04d40ef3f0 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04d915b700 1 -- 172.21.2.117:0/3950823496 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c800c0c0 con 0x7f04d40ef3f0 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04d915b700 1 -- 172.21.2.117:0/3950823496 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c800c5e0 con 0x7f04d40ef3f0 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 -- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d40ef3f0 msgr2=0x7f04d40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d40ef3f0 0x7f04d40eb210 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7f04c8006700 tx=0x7f04c8009b80).stop 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 -- 172.21.2.117:0/3950823496 shutdown_connections 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40eb830 0x7f04d41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 --2- 172.21.2.117:0/3950823496 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d40ef3f0 0x7f04d40eb210 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 -- 172.21.2.117:0/3950823496 >> 172.21.2.117:0/3950823496 conn(0x7f04d40cf060 msgr2=0x7f04d40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 -- 172.21.2.117:0/3950823496 shutdown_connections 2021-02-10T11:57:08.319 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.300 7f04dc341700 1 -- 172.21.2.117:0/3950823496 wait complete. 2021-02-10T11:57:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 Processor -- start 2021-02-10T11:57:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 -- start start 2021-02-10T11:57:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 0x7f04d40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40a2040 0x7f04d40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40a67e0 con 0x7f04d40a6ce0 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40a7120 con 0x7f04d4085ec0 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.315 7f04dc341700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d40a2a60 con 0x7f04d40a2040 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04da15d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04da15d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60200/0 (socket says 172.21.2.117:60200) 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04da15d700 1 -- 172.21.2.117:0/721424467 learned_addr learned my addr 172.21.2.117:0/721424467 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 0x7f04d40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04da95e700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40a2040 0x7f04d40a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 -- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40a2040 msgr2=0x7f04d40a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40a2040 0x7f04d40a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 -- 172.21.2.117:0/721424467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 msgr2=0x7f04d40dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 -- 172.21.2.117:0/721424467 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d41a2dc0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04da15d700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04d995c700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 0x7f04d40862c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f04c80096e0 tx=0x7f04c800b910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c800bd00 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8006030 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04dc341700 1 -- 172.21.2.117:0/721424467 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d41a2dc0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f04c80052e0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.316 7f04dc341700 1 -- 172.21.2.117:0/721424467 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d41a2dc0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.317 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f04c80052e0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.317 7f04c6ffd700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 0x7f04bc01abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.317 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f04c80326c0 con 0x7f04d4085ec0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.317 7f04da15d700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 0x7f04bc01abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:08.323 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.318 7f04da15d700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 0x7f04bc01abd0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:08.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.347 7f04dc341700 1 -- 172.21.2.117:0/721424467 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40daff0 con 0x7f04d4085ec0 2021-02-10T11:57:08.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.351 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f04c8009340 con 0x7f04d4085ec0 2021-02-10T11:57:08.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.484 7f04dc341700 1 -- 172.21.2.117:0/721424467 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f04d40a6760 con 0x7f04d4085ec0 2021-02-10T11:57:08.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.874 7f04c6ffd700 1 -- 172.21.2.117:0/721424467 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v185) v1 ==== 125+0+0 (secure 0 0 0) 0x7f04c80052e0 con 0x7f04d4085ec0 2021-02-10T11:57:08.901 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 -- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 msgr2=0x7f04bc01abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 0x7f04bc01abd0 crc :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 -- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 msgr2=0x7f04d40862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 0x7f04d40862c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f04c80096e0 tx=0x7f04c800b910).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 -- 172.21.2.117:0/721424467 shutdown_connections 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f04d40a2040 0x7f04d40a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.900 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f04d4085ec0 0x7f04d40862c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.901 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f04bc0186d0 0x7f04bc01abd0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.901 7f04dc341700 1 --2- 172.21.2.117:0/721424467 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f04d40a6ce0 0x7f04d40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.901 7f04dc341700 1 -- 172.21.2.117:0/721424467 >> 172.21.2.117:0/721424467 conn(0x7f04d40b6540 msgr2=0x7f04d408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.901 7f04dc341700 1 -- 172.21.2.117:0/721424467 shutdown_connections 2021-02-10T11:57:08.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:08.901 7f04dc341700 1 -- 172.21.2.117:0/721424467 wait complete. 2021-02-10T11:57:08.912 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.326 7ff8e0393700 1 Processor -- start 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.326 7ff8e0393700 1 -- start start 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8e0393700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d80e7170 0x7ff8d80e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8e0393700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff8d80e8380 0x7ff8d80e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8e0393700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d80e7570 con 0x7ff8d81b07e0 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8e0393700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d80e5cd0 con 0x7ff8d80e8380 2021-02-10T11:57:09.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8e0393700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80e77e0 con 0x7ff8d80e7170 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8de1af700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff8d81b07e0 0x7ff8d80e4e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60458/0 (socket says 172.21.2.117:60458) 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8de1af700 1 -- 172.21.2.117:0/1776186009 learned_addr learned my addr 172.21.2.117:0/1776186009 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314639652 0 0) 0x7ff8c80010b0 con 0x7ff8d81b07e0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c4002fe0 con 0x7ff8d81b07e0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8dd9ae700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d80e7170 0x7ff8d80e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.327 7ff8de9b0700 1 --2- 172.21.2.117:0/1776186009 >> v2:172.21.2.125:3300/0 conn(0x7ff8d80e8380 0x7ff8d80e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1511640822 0 0) 0x7ff8c80010b0 con 0x7ff8d81b07e0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d80e7170 msgr2=0x7ff8d80e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d80e7170 0x7ff8d80e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 >> v2:172.21.2.125:3300/0 conn(0x7ff8d80e8380 msgr2=0x7ff8d80e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 --2- 172.21.2.117:0/1776186009 >> v2:172.21.2.125:3300/0 conn(0x7ff8d80e8380 0x7ff8d80e8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80e8da0 con 0x7ff8d81b07e0 2021-02-10T11:57:09.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff8c80015b0 con 0x7ff8d81b07e0 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 >> v1:172.21.2.117:6789/0 conn(0x7ff8d81b07e0 legacy=0x7ff8d80e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c4003b10 0x7ff8c4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c404a7f0 0x7ff8c404cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c404d500 0x7ff8c404fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c40502d0 con 0x7ff8c404d500 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c40504f0 con 0x7ff8c4003b10 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4050710 con 0x7ff8c404a7f0 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c404d500 0x7ff8c404fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd9ae700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c404a7f0 0x7ff8c404cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de1af700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c4003b10 0x7ff8c4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 -- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c404a7f0 msgr2=0x7ff8c404cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8c404a7f0 0x7ff8c404cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 -- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c4003b10 msgr2=0x7ff8c4006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8c4003b10 0x7ff8c4006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 -- 172.21.2.117:0/1776186009 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40509b0 con 0x7ff8c404d500 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c404d500 0x7ff8c404fa20 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 rx=0x7ff8d0006a40 tx=0x7ff8d000c0c0).ready entity=mon.0 client_cookie=2b6aa972218b00c9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d000c4b0 con 0x7ff8c404d500 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d000c6f0 con 0x7ff8c404d500 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8dd1ad700 1 -- 172.21.2.117:0/1776186009 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d000cb50 con 0x7ff8c404d500 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8e0393700 1 -- 172.21.2.117:0/1776186009 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c404d500 msgr2=0x7ff8c404fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8e0393700 1 --2- 172.21.2.117:0/1776186009 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8c404d500 0x7ff8c404fa20 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 rx=0x7ff8d0006a40 tx=0x7ff8d000c0c0).stop 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.328 7ff8de9b0700 1 -- 172.21.2.117:0/1776186009 reap_dead start 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.329 7ff8e0393700 1 -- 172.21.2.117:0/1776186009 shutdown_connections 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.329 7ff8e0393700 1 -- 172.21.2.117:0/1776186009 >> 172.21.2.117:0/1776186009 conn(0x7ff8d80ce4f0 msgr2=0x7ff8d80e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.329 7ff8e0393700 1 -- 172.21.2.117:0/1776186009 shutdown_connections 2021-02-10T11:57:09.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.329 7ff8e0393700 1 -- 172.21.2.117:0/1776186009 wait complete. 2021-02-10T11:57:09.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 Processor -- start 2021-02-10T11:57:09.351 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 -- start start 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d81112b0 0x7ff8d81a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 0x7ff8d81a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d81a1b90 con 0x7ff8d810c3b0 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d810d1d0 con 0x7ff8d810cdd0 2021-02-10T11:57:09.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.344 7ff8e0393700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d810d3f0 con 0x7ff8d81112b0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60220/0 (socket says 172.21.2.117:60220) 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 -- 172.21.2.117:0/65027264 learned_addr learned my addr 172.21.2.117:0/65027264 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8de9b0700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 0x7ff8d81a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8de1af700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d81112b0 0x7ff8d81a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 -- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d81112b0 msgr2=0x7ff8d81a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d81112b0 0x7ff8d81a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 -- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 msgr2=0x7ff8d81a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 0x7ff8d81a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 -- 172.21.2.117:0/65027264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d81a26e0 con 0x7ff8d810c3b0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8de9b0700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 0x7ff8d81a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8dd9ae700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 rx=0x7ff8d4006f60 tx=0x7ff8d400b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d400bd60 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d4000c90 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.345 7ff8e0393700 1 -- 172.21.2.117:0/65027264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d81a26e0 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8e0393700 1 -- 172.21.2.117:0/65027264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d81a2c70 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff8d4002300 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff8d4006b00 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8ceffd700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 0x7ff8c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ff8d40328b0 con 0x7ff8d810c3b0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.346 7ff8de1af700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 0x7ff8c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.347 7ff8de1af700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 0x7ff8c001ac10 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.379 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.376 7ff8e0393700 1 -- 172.21.2.117:0/65027264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d810c180 con 0x7ff8d810c3b0 2021-02-10T11:57:09.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.379 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff8d4000990 con 0x7ff8d810c3b0 2021-02-10T11:57:09.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.507 7ff8e0393700 1 -- 172.21.2.117:0/65027264 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7ff8d80e77e0 con 0x7ff8d810c3b0 2021-02-10T11:57:09.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.510 7ff8ceffd700 1 -- 172.21.2.117:0/65027264 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 40 and data pool 41 v158) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff8d4000c90 con 0x7ff8d810c3b0 2021-02-10T11:57:09.518 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 40 and data pool 41 2021-02-10T11:57:09.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.516 7ff8e0393700 1 -- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 msgr2=0x7ff8c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.516 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 0x7ff8c001ac10 crc :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.516 7ff8e0393700 1 -- 172.21.2.117:0/65027264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 msgr2=0x7ff8d810c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.516 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 secure :-1 s=READY pgs=3106 cs=0 l=1 rev1=1 rx=0x7ff8d4006f60 tx=0x7ff8d400b970).stop 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 -- 172.21.2.117:0/65027264 shutdown_connections 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8c0018730 0x7ff8c001ac10 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff8d810c3b0 0x7ff8d810c7b0 unknown :-1 s=CLOSED pgs=3106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff8d810cdd0 0x7ff8d81a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 --2- 172.21.2.117:0/65027264 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff8d81112b0 0x7ff8d81a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 -- 172.21.2.117:0/65027264 >> 172.21.2.117:0/65027264 conn(0x7ff8d819efd0 msgr2=0x7ff8d808a280 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:09.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 -- 172.21.2.117:0/65027264 shutdown_connections 2021-02-10T11:57:09.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.517 7ff8e0393700 1 -- 172.21.2.117:0/65027264 wait complete. 2021-02-10T11:57:09.528 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:09.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.936 7fa2a5572700 1 Processor -- start 2021-02-10T11:57:09.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.936 7fa2a5572700 1 -- start start 2021-02-10T11:57:09.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa2a5572700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa2a00d18e0 0x7fa2a00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa2a5572700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2a00eb000 0x7fa2a00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.956 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa2a5572700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a00ede60 con 0x7fa2a00ee3e0 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa2a5572700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a00eba20 con 0x7fa2a00eb000 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa2a5572700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a00d1ce0 con 0x7fa2a00d18e0 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2a00eb000 0x7fa2a00eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2a00eb000 0x7fa2a00eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60228/0 (socket says 172.21.2.117:60228) 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29fac0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa2a00ee3e0 0x7fa2a00ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60478/0 (socket says 172.21.2.117:60478) 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 -- 172.21.2.117:0/2047100057 learned_addr learned my addr 172.21.2.117:0/2047100057 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 -- 172.21.2.117:0/2047100057 >> v2:172.21.2.125:3300/0 conn(0x7fa2a00d18e0 msgr2=0x7fa2a00ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 --2- 172.21.2.117:0/2047100057 >> v2:172.21.2.125:3300/0 conn(0x7fa2a00d18e0 0x7fa2a00ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 -- 172.21.2.117:0/2047100057 >> v1:172.21.2.117:6789/0 conn(0x7fa2a00ee3e0 legacy=0x7fa2a00ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 -- 172.21.2.117:0/2047100057 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00eefd0 con 0x7fa2a00eb000 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.937 7fa29eabe700 1 --2- 172.21.2.117:0/2047100057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a00eb000 0x7fa2a00eb400 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7fa290006700 tx=0x7fa290009b00).ready entity=mon.0 client_cookie=3b293eed35207923 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.957 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa29e2bd700 1 -- 172.21.2.117:0/2047100057 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa29000c0c0 con 0x7fa2a00eb000 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa29e2bd700 1 -- 172.21.2.117:0/2047100057 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29000c400 con 0x7fa2a00eb000 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa29e2bd700 1 -- 172.21.2.117:0/2047100057 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa29000c920 con 0x7fa2a00eb000 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 -- 172.21.2.117:0/2047100057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a00eb000 msgr2=0x7fa2a00eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 --2- 172.21.2.117:0/2047100057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a00eb000 0x7fa2a00eb400 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 rx=0x7fa290006700 tx=0x7fa290009b00).stop 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 -- 172.21.2.117:0/2047100057 shutdown_connections 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 --2- 172.21.2.117:0/2047100057 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a00eb000 0x7fa2a00eb400 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 --2- 172.21.2.117:0/2047100057 >> v2:172.21.2.125:3300/0 conn(0x7fa2a00d18e0 0x7fa2a00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.938 7fa2a5572700 1 -- 172.21.2.117:0/2047100057 >> 172.21.2.117:0/2047100057 conn(0x7fa2a00ced60 msgr2=0x7fa2a00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.939 7fa2a5572700 1 -- 172.21.2.117:0/2047100057 shutdown_connections 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.939 7fa2a5572700 1 -- 172.21.2.117:0/2047100057 wait complete. 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.953 7fa2a5572700 1 Processor -- start 2021-02-10T11:57:09.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.953 7fa2a5572700 1 -- start start 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2a00b3230 0x7fa2a019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2a00b3630 0x7fa2a019ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00b3a30 con 0x7fa2a019eff0 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a01997b0 con 0x7fa2a00b3630 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a01a7450 con 0x7fa2a00b3230 2021-02-10T11:57:09.960 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60234/0 (socket says 172.21.2.117:60234) 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 -- 172.21.2.117:0/1243184011 learned_addr learned my addr 172.21.2.117:0/1243184011 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 -- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2a00b3230 msgr2=0x7fa2a019a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2a00b3230 0x7fa2a019a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 -- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2a00b3630 msgr2=0x7fa2a019ae10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2a00b3630 0x7fa2a019ae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 -- 172.21.2.117:0/1243184011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a019b430 con 0x7fa2a019eff0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa29f2bf700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7fa28800b5c0 tx=0x7fa28800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa28800bd50 con 0x7fa2a019eff0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa288006030 con 0x7fa2a019eff0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2880054d0 con 0x7fa2a019eff0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.954 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a019b430 con 0x7fa2a019eff0 2021-02-10T11:57:09.961 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.955 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a019b430 con 0x7fa2a019eff0 2021-02-10T11:57:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.955 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa288006210 con 0x7fa2a019eff0 2021-02-10T11:57:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.955 7fa297fff700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 0x7fa28401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.955 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2880327a0 con 0x7fa2a019eff0 2021-02-10T11:57:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.956 7fa29eabe700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 0x7fa28401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:09.962 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.957 7fa29eabe700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 0x7fa28401abc0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:09.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.987 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a0087210 con 0x7fa2a019eff0 2021-02-10T11:57:09.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:09.989 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa288006b80 con 0x7fa2a019eff0 2021-02-10T11:57:10.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.112 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2a00ede60 con 0x7fa2a019eff0 2021-02-10T11:57:10.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.113 7fa297fff700 1 -- 172.21.2.117:0/1243184011 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+14253 (secure 0 0 0) 0x7fa288006030 con 0x7fa2a019eff0 2021-02-10T11:57:10.116 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:10.116 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":186,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:09.509170","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:57:07.475412","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:57:08.504122","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:10.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 msgr2=0x7fa28401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 0x7fa28401abc0 crc :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 msgr2=0x7fa2a00b2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 rx=0x7fa28800b5c0 tx=0x7fa28800b960).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 shutdown_connections 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2a00b3630 0x7fa2a019ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2a00b3230 0x7fa2a019a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa284018730 0x7fa28401abc0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 --2- 172.21.2.117:0/1243184011 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2a019eff0 0x7fa2a00b2c10 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 >> 172.21.2.117:0/1243184011 conn(0x7fa2a019f610 msgr2=0x7fa2a0099520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 shutdown_connections 2021-02-10T11:57:10.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.114 7fa2a5572700 1 -- 172.21.2.117:0/1243184011 wait complete. 2021-02-10T11:57:10.127 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.537 7f35f5d73700 1 Processor -- start 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.537 7f35f5d73700 1 -- start start 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35f5d73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 0x7f35f01b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35f5d73700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00ede60 con 0x7f35f00ef3f0 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35f5d73700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00eb9c0 con 0x7f35f00d18e0 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35f5d73700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00d34b0 con 0x7f35f01b6960 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35eeffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f35f00ef3f0 0x7f35f00eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60492/0 (socket says 172.21.2.117:60492) 2021-02-10T11:57:10.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35eeffd700 1 -- 172.21.2.117:0/3932668999 learned_addr learned my addr 172.21.2.117:0/3932668999 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35effff700 1 --2- 172.21.2.117:0/3932668999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 0x7f35f01b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294563419 0 0) 0x7f35e40010b0 con 0x7f35f00ef3f0 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.538 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8002fe0 con 0x7f35f00ef3f0 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35effff700 1 -- 172.21.2.117:0/3932668999 >> v1:172.21.2.125:6789/0 conn(0x7f35f00d18e0 legacy=0x7f35f00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35effff700 1 -- 172.21.2.117:0/3932668999 >> v1:172.21.2.117:6789/0 conn(0x7f35f00ef3f0 legacy=0x7f35f00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584091030 0 0) 0x7f35e8000fe0 con 0x7f35f00d18e0 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35effff700 1 -- 172.21.2.117:0/3932668999 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00d1ce0 con 0x7f35f01b6960 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35effff700 1 --2- 172.21.2.117:0/3932668999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 0x7f35f01b8d70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f35e0006ad0 tx=0x7f35e0009ad0).ready entity=mon.2 client_cookie=2e058ba2a40f36fa server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e00022e0 con 0x7f35f01b6960 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0006410 con 0x7f35f01b6960 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35ee7fc700 1 -- 172.21.2.117:0/3932668999 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e0002560 con 0x7f35f01b6960 2021-02-10T11:57:10.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35f5d73700 1 -- 172.21.2.117:0/3932668999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 msgr2=0x7f35f01b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35f5d73700 1 --2- 172.21.2.117:0/3932668999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 0x7f35f01b8d70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f35e0006ad0 tx=0x7f35e0009ad0).stop 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.539 7f35f5d73700 1 -- 172.21.2.117:0/3932668999 shutdown_connections 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.540 7f35f5d73700 1 --2- 172.21.2.117:0/3932668999 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f01b6960 0x7f35f01b8d70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.540 7f35f5d73700 1 -- 172.21.2.117:0/3932668999 >> 172.21.2.117:0/3932668999 conn(0x7f35f00ced60 msgr2=0x7f35f00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.540 7f35f5d73700 1 -- 172.21.2.117:0/3932668999 shutdown_connections 2021-02-10T11:57:10.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.540 7f35f5d73700 1 -- 172.21.2.117:0/3932668999 wait complete. 2021-02-10T11:57:10.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.554 7f35f5d73700 1 Processor -- start 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.554 7f35f5d73700 1 -- start start 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35f00ab380 0x7f35f00dd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f019ade0 0x7f35f019b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f019a860 con 0x7f35f019b800 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0190cd0 con 0x7f35f00ab380 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35f5d73700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0190eb0 con 0x7f35f019ade0 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60248/0 (socket says 172.21.2.117:60248) 2021-02-10T11:57:10.561 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 -- 172.21.2.117:0/8264516 learned_addr learned my addr 172.21.2.117:0/8264516 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 -- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f019ade0 msgr2=0x7f35f019b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f019ade0 0x7f35f019b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 -- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35f00ab380 msgr2=0x7f35f00dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35f00ab380 0x7f35f00dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.555 7f35effff700 1 -- 172.21.2.117:0/8264516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f01910d0 con 0x7f35f019b800 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35effff700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f35e0009600 tx=0x7f35e000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e0002030 con 0x7f35f019b800 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0000e00 con 0x7f35f019b800 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f35e0002530 con 0x7f35f019b800 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35f5d73700 1 -- 172.21.2.117:0/8264516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f01910d0 con 0x7f35f019b800 2021-02-10T11:57:10.562 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35f5d73700 1 -- 172.21.2.117:0/8264516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f01910d0 con 0x7f35f019b800 2021-02-10T11:57:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f35e0005330 con 0x7f35f019b800 2021-02-10T11:57:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.556 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f35e0032770 con 0x7f35f019b800 2021-02-10T11:57:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.557 7f35ef7fe700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:10.563 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.557 7f35ef7fe700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 0x7f35d001abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:10.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.587 7f35f5d73700 1 -- 172.21.2.117:0/8264516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0087210 con 0x7f35f019b800 2021-02-10T11:57:10.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.590 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f35e0006370 con 0x7f35f019b800 2021-02-10T11:57:10.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.714 7f35f5d73700 1 -- 172.21.2.117:0/8264516 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35f00ede60 con 0x7f35f019b800 2021-02-10T11:57:10.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.715 7f35d7fff700 1 -- 172.21.2.117:0/8264516 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f35e0000de0 con 0x7f35f019b800 2021-02-10T11:57:10.718 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:10.719 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":187,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:10.511294","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:57:07.475412","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:57:08.504122","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:10.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.716 7f35f5d73700 1 -- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 msgr2=0x7f35d001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.716 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 0x7f35d001abc0 crc :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.716 7f35f5d73700 1 -- 172.21.2.117:0/8264516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 msgr2=0x7f35f01906b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.716 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 rx=0x7f35e0009600 tx=0x7f35e000b960).stop 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 -- 172.21.2.117:0/8264516 shutdown_connections 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f35d00186c0 0x7f35d001abc0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f35f019b800 0x7f35f01906b0 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f35f019ade0 0x7f35f019b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 --2- 172.21.2.117:0/8264516 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f35f00ab380 0x7f35f00dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 -- 172.21.2.117:0/8264516 >> 172.21.2.117:0/8264516 conn(0x7f35f01b67a0 msgr2=0x7f35f00e9560 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:10.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 -- 172.21.2.117:0/8264516 shutdown_connections 2021-02-10T11:57:10.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:10.717 7f35f5d73700 1 -- 172.21.2.117:0/8264516 wait complete. 2021-02-10T11:57:10.728 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.141 7fa939e3e700 1 Processor -- start 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa939e3e700 1 -- start start 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa939e3e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa9340d18f0 0x7fa9340eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa939e3e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340ef7f0 con 0x7fa9340eb040 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa939e3e700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340ede60 con 0x7fa9340ef3f0 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa939e3e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9340d1da0 con 0x7fa9340d18f0 2021-02-10T11:57:11.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa932ffd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa9340eb040 0x7fa9340eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60504/0 (socket says 172.21.2.117:60504) 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.142 7fa932ffd700 1 -- 172.21.2.117:0/4032067006 learned_addr learned my addr 172.21.2.117:0/4032067006 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391026968 0 0) 0x7fa9240010b0 con 0x7fa9340eb040 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa920002fe0 con 0x7fa9340eb040 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9337fe700 1 --2- 172.21.2.117:0/4032067006 >> v2:172.21.2.125:3300/0 conn(0x7fa9340d18f0 0x7fa9340eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2918567828 0 0) 0x7fa9240010b0 con 0x7fa9340eb040 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 >> v2:172.21.2.125:3300/0 conn(0x7fa9340d18f0 msgr2=0x7fa9340eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 --2- 172.21.2.117:0/4032067006 >> v2:172.21.2.125:3300/0 conn(0x7fa9340d18f0 0x7fa9340eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 >> v1:172.21.2.125:6789/0 conn(0x7fa9340ef3f0 legacy=0x7fa9341b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9341bb1c0 con 0x7fa9340eb040 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fa9240015b0 con 0x7fa9340eb040 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 >> v1:172.21.2.117:6789/0 conn(0x7fa9340eb040 legacy=0x7fa9340eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa920003b10 0x7fa920005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 0x7fa920008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa920009640 0x7fa92000bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000c3b0 con 0x7fa920006820 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000c5d0 con 0x7fa920009640 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000c7f0 con 0x7fa920003b10 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa932ffd700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 0x7fa920008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa9337fe700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa920003b10 0x7fa920005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.143 7fa933fff700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa920009640 0x7fa92000bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 -- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa920003b10 msgr2=0x7fa920005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa920003b10 0x7fa920005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 -- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa920009640 msgr2=0x7fa92000bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa920009640 0x7fa92000bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa933fff700 1 -- 172.21.2.117:0/4032067006 reap_dead start 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 -- 172.21.2.117:0/4032067006 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9200509f0 con 0x7fa920006820 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa932ffd700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 0x7fa920008d40 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7fa9240017d0 tx=0x7fa924007d30).ready entity=mon.0 client_cookie=6b3217389e5e5608 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa9240013a0 con 0x7fa920006820 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92400a310 con 0x7fa920006820 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa9327fc700 1 -- 172.21.2.117:0/4032067006 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa92400a7a0 con 0x7fa920006820 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa939e3e700 1 -- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 msgr2=0x7fa920008d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa939e3e700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 0x7fa920008d40 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 rx=0x7fa9240017d0 tx=0x7fa924007d30).stop 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa939e3e700 1 -- 172.21.2.117:0/4032067006 shutdown_connections 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa939e3e700 1 --2- 172.21.2.117:0/4032067006 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa920006820 0x7fa920008d40 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.144 7fa939e3e700 1 -- 172.21.2.117:0/4032067006 >> 172.21.2.117:0/4032067006 conn(0x7fa9340cf060 msgr2=0x7fa9340cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.145 7fa939e3e700 1 -- 172.21.2.117:0/4032067006 shutdown_connections 2021-02-10T11:57:11.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.145 7fa939e3e700 1 -- 172.21.2.117:0/4032067006 wait complete. 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.159 7fa939e3e700 1 Processor -- start 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.159 7fa939e3e700 1 -- start start 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa934085ec0 0x7fa9340862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9340a2040 0x7fa9340a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340a67e0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340a7120 con 0x7fa9340a2040 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa939e3e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340a2a60 con 0x7fa934085ec0 2021-02-10T11:57:11.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60268/0 (socket says 172.21.2.117:60268) 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 -- 172.21.2.117:0/1538258416 learned_addr learned my addr 172.21.2.117:0/1538258416 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 -- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa934085ec0 msgr2=0x7fa9340862c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa934085ec0 0x7fa9340862c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 -- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9340a2040 msgr2=0x7fa9340a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9340a2040 0x7fa9340a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.160 7fa9337fe700 1 -- 172.21.2.117:0/1538258416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9341a2dc0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa9337fe700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7fa91c006f60 tx=0x7fa91c00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91c00bd60 con 0x7fa9340a6ce0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c000c90 con 0x7fa9340a6ce0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa91c002220 con 0x7fa9340a6ce0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9341a2dc0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.161 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9341a3350 con 0x7fa9340a6ce0 2021-02-10T11:57:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.162 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa91c0052e0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.162 7fa92bfff700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 0x7fa91801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.162 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa91c0326e0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.162 7fa932ffd700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 0x7fa91801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.162 7fa932ffd700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 0x7fa91801abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:11.195 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.192 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9340daff0 con 0x7fa9340a6ce0 2021-02-10T11:57:11.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.194 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa91c028300 con 0x7fa9340a6ce0 2021-02-10T11:57:11.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.329 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7fa9340ede60 con 0x7fa9340a6ce0 2021-02-10T11:57:11.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.520 7fa92bfff700 1 -- 172.21.2.117:0/1538258416 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v161) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa91c028300 con 0x7fa9340a6ce0 2021-02-10T11:57:11.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 msgr2=0x7fa91801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 0x7fa91801abc0 crc :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 msgr2=0x7fa9340dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 rx=0x7fa91c006f60 tx=0x7fa91c00b970).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 shutdown_connections 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa9340a2040 0x7fa9340a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa934085ec0 0x7fa9340862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa9180186c0 0x7fa91801abc0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 --2- 172.21.2.117:0/1538258416 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa9340a6ce0 0x7fa9340dc420 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.546 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 >> 172.21.2.117:0/1538258416 conn(0x7fa9340b6540 msgr2=0x7fa93408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:11.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.547 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 shutdown_connections 2021-02-10T11:57:11.551 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.547 7fa939e3e700 1 -- 172.21.2.117:0/1538258416 wait complete. 2021-02-10T11:57:11.558 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 Processor -- start 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 -- start start 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f45140eb7f0 0x7f45141b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140ede60 con 0x7f45140d18e0 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140ef7f0 con 0x7f45140eb7f0 2021-02-10T11:57:11.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.967 7f451a4a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140d1ce0 con 0x7f45140ef3f0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:57952/0 (socket says 172.21.2.117:57952) 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 -- 172.21.2.117:0/2671540734 learned_addr learned my addr 172.21.2.117:0/2671540734 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f4518abd700 1 --2- 172.21.2.117:0/2671540734 >> v2:172.21.2.125:3300/0 conn(0x7f45140eb7f0 0x7f45141b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 -- 172.21.2.117:0/2671540734 >> v2:172.21.2.125:3300/0 conn(0x7f45140eb7f0 msgr2=0x7f45141b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 --2- 172.21.2.117:0/2671540734 >> v2:172.21.2.125:3300/0 conn(0x7f45140eb7f0 0x7f45141b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 -- 172.21.2.117:0/2671540734 >> v1:172.21.2.125:6789/0 conn(0x7f45140d18e0 legacy=0x7f45140ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 -- 172.21.2.117:0/2671540734 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45141b8f90 con 0x7f45140ef3f0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f45137fe700 1 --2- 172.21.2.117:0/2671540734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f4500006700 tx=0x7f4500009b80).ready entity=mon.2 client_cookie=1f8e790e0721cbb7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f4512ffd700 1 -- 172.21.2.117:0/2671540734 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4500000c90 con 0x7f45140ef3f0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f4512ffd700 1 -- 172.21.2.117:0/2671540734 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450000c0c0 con 0x7f45140ef3f0 2021-02-10T11:57:11.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f4512ffd700 1 -- 172.21.2.117:0/2671540734 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450000c5e0 con 0x7f45140ef3f0 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f451a4a0700 1 -- 172.21.2.117:0/2671540734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 msgr2=0x7f45140eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.969 7f451a4a0700 1 --2- 172.21.2.117:0/2671540734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f4500006700 tx=0x7f4500009b80).stop 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 -- 172.21.2.117:0/2671540734 shutdown_connections 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 --2- 172.21.2.117:0/2671540734 >> v2:172.21.2.125:3300/0 conn(0x7f45140eb7f0 0x7f45141b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 --2- 172.21.2.117:0/2671540734 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140ef3f0 0x7f45140eb1d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 -- 172.21.2.117:0/2671540734 >> 172.21.2.117:0/2671540734 conn(0x7f45140ced60 msgr2=0x7f45140cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 -- 172.21.2.117:0/2671540734 shutdown_connections 2021-02-10T11:57:11.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.970 7f451a4a0700 1 -- 172.21.2.117:0/2671540734 wait complete. 2021-02-10T11:57:11.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.984 7f451a4a0700 1 Processor -- start 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 -- start start 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45141afd30 0x7f45140b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140b43f0 0x7f45140e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140e80e0 con 0x7f45141b0130 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140e7e20 con 0x7f45141afd30 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f451a4a0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4514096a70 con 0x7f45140b43f0 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f4518abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f4518abd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60278/0 (socket says 172.21.2.117:60278) 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f4518abd700 1 -- 172.21.2.117:0/3053129785 learned_addr learned my addr 172.21.2.117:0/3053129785 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.985 7f4518abd700 1 -- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140b43f0 msgr2=0x7f45140e71e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:11.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4518abd700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140b43f0 0x7f45140e71e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4518abd700 1 -- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45141afd30 msgr2=0x7f45140b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4518abd700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45141afd30 0x7f45140b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4518abd700 1 -- 172.21.2.117:0/3053129785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140e8660 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4518abd700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 rx=0x7f450c00b910 tx=0x7f450c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450c0011f0 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c003030 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45140e8660 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140e8bf0 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.986 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f450c001620 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.987 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f450c0056b0 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.987 7f4510ff9700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 0x7f44f001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.987 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f450c032c90 con 0x7f45141b0130 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.987 7f4513fff700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 0x7f44f001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:11.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:11.987 7f4513fff700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 0x7f44f001abd0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:12.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.017 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4514087290 con 0x7f45141b0130 2021-02-10T11:57:12.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.019 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f450c003010 con 0x7f45141b0130 2021-02-10T11:57:12.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.145 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45140ede60 con 0x7f45141b0130 2021-02-10T11:57:12.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.145 7f4510ff9700 1 -- 172.21.2.117:0/3053129785 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f450c003010 con 0x7f45141b0130 2021-02-10T11:57:12.149 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 161 2021-02-10T11:57:12.149 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:12.149 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":161,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:11.517907","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8153":{"gid":8153,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/2333277299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2333277299},{"type":"v1","addr":"172.21.2.125:6835","nonce":2333277299}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:12.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 msgr2=0x7f44f001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 0x7f44f001abd0 crc :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 msgr2=0x7f45140e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 rx=0x7f450c00b910 tx=0x7f450c00bcb0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 shutdown_connections 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f44f00186d0 0x7f44f001abd0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f45141b0130 0x7f45140e7800 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f45140b43f0 0x7f45140e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 --2- 172.21.2.117:0/3053129785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f45141afd30 0x7f45140b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.147 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 >> 172.21.2.117:0/3053129785 conn(0x7f45141a3620 msgr2=0x7f45140dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.148 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 shutdown_connections 2021-02-10T11:57:12.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.148 7f451a4a0700 1 -- 172.21.2.117:0/3053129785 wait complete. 2021-02-10T11:57:12.159 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:12.588 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 Processor -- start 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 -- start start 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f3c085450 0x7f3f3c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c085930 con 0x7f3f3c0e5240 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c1b09e0 con 0x7f3f3c0e4e40 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f42302700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0e5cc0 con 0x7f3f3c085450 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f4091f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f3c085450 0x7f3f3c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f4091f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3f3c085450 0x7f3f3c086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60286/0 (socket says 172.21.2.117:60286) 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.570 7f3f4091f700 1 -- 172.21.2.117:0/3954143945 learned_addr learned my addr 172.21.2.117:0/3954143945 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f4091f700 1 -- 172.21.2.117:0/3954143945 >> v1:172.21.2.125:6789/0 conn(0x7f3f3c0e4e40 legacy=0x7f3f3c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f4091f700 1 -- 172.21.2.117:0/3954143945 >> v1:172.21.2.117:6789/0 conn(0x7f3f3c0e5240 legacy=0x7f3f3c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.589 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f4091f700 1 -- 172.21.2.117:0/3954143945 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c084840 con 0x7f3f3c085450 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f4091f700 1 --2- 172.21.2.117:0/3954143945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c085450 0x7f3f3c086ec0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f3f34006f00 tx=0x7f3f34009f10).ready entity=mon.0 client_cookie=774b89b5a34fb339 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f3affd700 1 -- 172.21.2.117:0/3954143945 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3400b3b0 con 0x7f3f3c085450 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f3affd700 1 -- 172.21.2.117:0/3954143945 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3400b630 con 0x7f3f3c085450 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f3affd700 1 -- 172.21.2.117:0/3954143945 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3400bb50 con 0x7f3f3c085450 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f42302700 1 -- 172.21.2.117:0/3954143945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c085450 msgr2=0x7f3f3c086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.571 7f3f42302700 1 --2- 172.21.2.117:0/3954143945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c085450 0x7f3f3c086ec0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 rx=0x7f3f34006f00 tx=0x7f3f34009f10).stop 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.572 7f3f42302700 1 -- 172.21.2.117:0/3954143945 shutdown_connections 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.572 7f3f42302700 1 --2- 172.21.2.117:0/3954143945 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c085450 0x7f3f3c086ec0 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.572 7f3f42302700 1 -- 172.21.2.117:0/3954143945 >> 172.21.2.117:0/3954143945 conn(0x7f3f3c0ce4e0 msgr2=0x7f3f3c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.572 7f3f42302700 1 -- 172.21.2.117:0/3954143945 shutdown_connections 2021-02-10T11:57:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.572 7f3f42302700 1 -- 172.21.2.117:0/3954143945 wait complete. 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.586 7f3f42302700 1 Processor -- start 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.586 7f3f42302700 1 -- start start 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f3c0a3830 0x7f3f3c1afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f3c1b0450 0x7f3f3c1b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1b0f40 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0aab30 con 0x7f3f3c1b0450 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f42302700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0a3100 con 0x7f3f3c0a3830 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60296/0 (socket says 172.21.2.117:60296) 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 -- 172.21.2.117:0/1121893065 learned_addr learned my addr 172.21.2.117:0/1121893065 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 -- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f3c0a3830 msgr2=0x7f3f3c1afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f3c0a3830 0x7f3f3c1afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 -- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f3c1b0450 msgr2=0x7f3f3c1b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f3c1b0450 0x7f3f3c1b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 -- 172.21.2.117:0/1121893065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0a3320 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f4091f700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 rx=0x7f3f34009390 tx=0x7f3f3400bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.587 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f34003030 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f340068e0 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3f3400c570 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f42302700 1 -- 172.21.2.117:0/1121893065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c0a3320 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f42302700 1 -- 172.21.2.117:0/1121893065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c106ed0 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3f34001270 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f38ff9700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 0x7f3f1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.588 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3f34032a60 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.589 7f3f3bfff700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 0x7f3f1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.589 7f3f3bfff700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 0x7f3f1c01abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:12.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.620 7f3f42302700 1 -- 172.21.2.117:0/1121893065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c105cd0 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.623 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3f34009430 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.746 7f3f42302700 1 -- 172.21.2.117:0/1121893065 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f3c0e5eb0 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.747 7f3f38ff9700 1 -- 172.21.2.117:0/1121893065 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f3f3401a200 con 0x7f3f3c0a26e0 2021-02-10T11:57:12.750 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:12.750 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":187,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:10.511294","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:57:07.475412","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:57:08.504122","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.748 7f3f42302700 1 -- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 msgr2=0x7f3f1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.748 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 0x7f3f1c01abc0 crc :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.748 7f3f42302700 1 -- 172.21.2.117:0/1121893065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 msgr2=0x7f3f3c0a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.748 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 rx=0x7f3f34009390 tx=0x7f3f3400bd40).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 -- 172.21.2.117:0/1121893065 shutdown_connections 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3f1c0186c0 0x7f3f1c01abc0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3f3c0a26e0 0x7f3f3c0a2ae0 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3f3c1b0450 0x7f3f3c1b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 --2- 172.21.2.117:0/1121893065 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3f3c0a3830 0x7f3f3c1afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:12.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 -- 172.21.2.117:0/1121893065 >> 172.21.2.117:0/1121893065 conn(0x7f3f3c0c3ed0 msgr2=0x7f3f3c0c06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:12.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 -- 172.21.2.117:0/1121893065 shutdown_connections 2021-02-10T11:57:12.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:12.749 7f3f42302700 1 -- 172.21.2.117:0/1121893065 wait complete. 2021-02-10T11:57:12.761 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:57:13.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.182 7fa31bb3e700 1 Processor -- start 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.182 7fa31bb3e700 1 -- start start 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31bb3e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa31407ab40 0x7fa3140794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31bb3e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa314079f10 0x7fa31407b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31bb3e700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3140e5770 con 0x7fa314079b10 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31bb3e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31407a310 con 0x7fa314079f10 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31bb3e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3140d18c0 con 0x7fa31407ab40 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa319159700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa314079b10 0x7fa31407af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60550/0 (socket says 172.21.2.117:60550) 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa319159700 1 -- 172.21.2.117:0/2215755493 learned_addr learned my addr 172.21.2.117:0/2215755493 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 --2- 172.21.2.117:0/2215755493 >> v2:172.21.2.117:3300/0 conn(0x7fa314079f10 0x7fa31407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa318958700 1 -- 172.21.2.117:0/2215755493 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572551669 0 0) 0x7fa3080009b0 con 0x7fa314079b10 2021-02-10T11:57:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa318958700 1 -- 172.21.2.117:0/2215755493 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa300002fe0 con 0x7fa314079b10 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 -- 172.21.2.117:0/2215755493 >> v2:172.21.2.125:3300/0 conn(0x7fa31407ab40 msgr2=0x7fa3140794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 --2- 172.21.2.117:0/2215755493 >> v2:172.21.2.125:3300/0 conn(0x7fa31407ab40 0x7fa3140794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 -- 172.21.2.117:0/2215755493 >> v1:172.21.2.117:6789/0 conn(0x7fa314079b10 legacy=0x7fa31407af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 -- 172.21.2.117:0/2215755493 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3140d1c60 con 0x7fa314079f10 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa31a15b700 1 --2- 172.21.2.117:0/2215755493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa314079f10 0x7fa31407b730 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7fa30c006e30 tx=0x7fa30c009e60).ready entity=mon.0 client_cookie=6cddaadde684b616 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.183 7fa318958700 1 -- 172.21.2.117:0/2215755493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30c006a50 con 0x7fa314079f10 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa318958700 1 -- 172.21.2.117:0/2215755493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00b300 con 0x7fa314079f10 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa318958700 1 -- 172.21.2.117:0/2215755493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30c00b750 con 0x7fa314079f10 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 -- 172.21.2.117:0/2215755493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa314079f10 msgr2=0x7fa31407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 --2- 172.21.2.117:0/2215755493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa314079f10 0x7fa31407b730 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 rx=0x7fa30c006e30 tx=0x7fa30c009e60).stop 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 -- 172.21.2.117:0/2215755493 shutdown_connections 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 --2- 172.21.2.117:0/2215755493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa314079f10 0x7fa31407b730 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 --2- 172.21.2.117:0/2215755493 >> v2:172.21.2.125:3300/0 conn(0x7fa31407ab40 0x7fa3140794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 -- 172.21.2.117:0/2215755493 >> 172.21.2.117:0/2215755493 conn(0x7fa3140d0910 msgr2=0x7fa3140d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:13.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 -- 172.21.2.117:0/2215755493 shutdown_connections 2021-02-10T11:57:13.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.184 7fa31bb3e700 1 -- 172.21.2.117:0/2215755493 wait complete. 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 Processor -- start 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 -- start start 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa314100bc0 0x7fa3141b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3141acc30 0x7fa3141ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa314092230 con 0x7fa3140917c0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3140927b0 con 0x7fa314100bc0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa31bb3e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa314100700 con 0x7fa3141acc30 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa319159700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa319159700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60310/0 (socket says 172.21.2.117:60310) 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.199 7fa319159700 1 -- 172.21.2.117:0/1894126002 learned_addr learned my addr 172.21.2.117:0/1894126002 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 -- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3141acc30 msgr2=0x7fa3141ad030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3141acc30 0x7fa3141ad030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 -- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa314100bc0 msgr2=0x7fa3141b7c30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa314100bc0 0x7fa3141b7c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 -- 172.21.2.117:0/1894126002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3141ad650 con 0x7fa3140917c0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa319159700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 rx=0x7fa308001c60 tx=0x7fa30800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa30800acc0 con 0x7fa3140917c0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308004140 con 0x7fa3140917c0 2021-02-10T11:57:13.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa308004660 con 0x7fa3140917c0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141ad650 con 0x7fa3140917c0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.200 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141ad650 con 0x7fa3140917c0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.201 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa308004320 con 0x7fa3140917c0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.201 7fa3067fc700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 0x7fa2fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.201 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa308032690 con 0x7fa3140917c0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.201 7fa31995a700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 0x7fa2fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.207 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.202 7fa31995a700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 0x7fa2fc01abc0 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.233 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31407a5f0 con 0x7fa3140917c0 2021-02-10T11:57:13.238 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.235 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa308000e20 con 0x7fa3140917c0 2021-02-10T11:57:13.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.364 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa31407a5f0 con 0x7fa3140917c0 2021-02-10T11:57:13.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.368 7fa3067fc700 1 -- 172.21.2.117:0/1894126002 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v108) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa308004680 con 0x7fa3140917c0 2021-02-10T11:57:13.374 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:57:13.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 msgr2=0x7fa2fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 0x7fa2fc01abc0 crc :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 msgr2=0x7fa314091bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.376 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 rx=0x7fa308001c60 tx=0x7fa30800a8d0).stop 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 shutdown_connections 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2fc0186c0 0x7fa2fc01abc0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3140917c0 0x7fa314091bc0 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3141acc30 0x7fa3141ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 --2- 172.21.2.117:0/1894126002 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa314100bc0 0x7fa3141b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.373 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 >> 172.21.2.117:0/1894126002 conn(0x7fa31419f690 msgr2=0x7fa3141919a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.374 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 shutdown_connections 2021-02-10T11:57:13.377 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.374 7fa31bb3e700 1 -- 172.21.2.117:0/1894126002 wait complete. 2021-02-10T11:57:13.384 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:57:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.798 7f72ea354700 1 Processor -- start 2021-02-10T11:57:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.798 7f72ea354700 1 -- start start 2021-02-10T11:57:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72ea354700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e4079ce0 0x7f72e41b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72ea354700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72e407a0e0 0x7f72e41b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72ea354700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40e5770 con 0x7f72e407ab40 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72ea354700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e40d18c0 con 0x7f72e407a0e0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72ea354700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e407bf50 con 0x7f72e4079ce0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72e407a0e0 0x7f72e41b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f72e407a0e0 0x7f72e41b8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60318/0 (socket says 172.21.2.117:60318) 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 -- 172.21.2.117:0/898589765 learned_addr learned my addr 172.21.2.117:0/898589765 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 -- 172.21.2.117:0/898589765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e4079ce0 msgr2=0x7f72e41b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 --2- 172.21.2.117:0/898589765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e4079ce0 0x7f72e41b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 -- 172.21.2.117:0/898589765 >> v1:172.21.2.125:6789/0 conn(0x7f72e407ab40 legacy=0x7f72e40794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.799 7f72e8971700 1 -- 172.21.2.117:0/898589765 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e40d1c60 con 0x7f72e407a0e0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72e8971700 1 --2- 172.21.2.117:0/898589765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e407a0e0 0x7f72e41b8000 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7f72dc006f00 tx=0x7f72dc009f10).ready entity=mon.0 client_cookie=9cfba954a867de24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72e2ffd700 1 -- 172.21.2.117:0/898589765 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72dc00b3b0 con 0x7f72e407a0e0 2021-02-10T11:57:13.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72e2ffd700 1 -- 172.21.2.117:0/898589765 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00b730 con 0x7f72e407a0e0 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72e2ffd700 1 -- 172.21.2.117:0/898589765 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72dc00bb90 con 0x7f72e407a0e0 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 -- 172.21.2.117:0/898589765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e407a0e0 msgr2=0x7f72e41b8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 --2- 172.21.2.117:0/898589765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e407a0e0 0x7f72e41b8000 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 rx=0x7f72dc006f00 tx=0x7f72dc009f10).stop 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 -- 172.21.2.117:0/898589765 shutdown_connections 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 --2- 172.21.2.117:0/898589765 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e407a0e0 0x7f72e41b8000 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 --2- 172.21.2.117:0/898589765 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e4079ce0 0x7f72e41b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 -- 172.21.2.117:0/898589765 >> 172.21.2.117:0/898589765 conn(0x7f72e40d0910 msgr2=0x7f72e40d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 -- 172.21.2.117:0/898589765 shutdown_connections 2021-02-10T11:57:13.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.800 7f72ea354700 1 -- 172.21.2.117:0/898589765 wait complete. 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 Processor -- start 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 -- start start 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 0x7f72e41919a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 0x7f72e40ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40ff820 con 0x7f72e40fe6d0 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40dc870 con 0x7f72e40ff0f0 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72ea354700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e40ff4f0 con 0x7f72e40ac9e0 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72e37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72e37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60322/0 (socket says 172.21.2.117:60322) 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.815 7f72e37fe700 1 -- 172.21.2.117:0/270491858 learned_addr learned my addr 172.21.2.117:0/270491858 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 -- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 msgr2=0x7f72e41919a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e3fff700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 0x7f72e41919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e8971700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 0x7f72e40ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 0x7f72e41919a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 -- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 msgr2=0x7f72e40ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 0x7f72e40ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 -- 172.21.2.117:0/270491858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e41003c0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e8971700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 0x7f72e40ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e3fff700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 0x7f72e41919a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e37fe700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7f72d8006f00 tx=0x7f72d800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.816 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d80009b0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8006350 con 0x7f72e40fe6d0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f72d8000c90 con 0x7f72e40fe6d0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72ea354700 1 -- 172.21.2.117:0/270491858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e41003c0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72ea354700 1 -- 172.21.2.117:0/270491858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e4100950 con 0x7f72e40fe6d0 2021-02-10T11:57:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f72d8007000 con 0x7f72e40fe6d0 2021-02-10T11:57:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72e0ff9700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 0x7f72c401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.817 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f72d80326e0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.818 7f72e3fff700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 0x7f72c401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:13.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.818 7f72e3fff700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 0x7f72c401abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:13.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.849 7f72ea354700 1 -- 172.21.2.117:0/270491858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e40ff4f0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.854 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.851 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f72d8006b90 con 0x7f72e40fe6d0 2021-02-10T11:57:13.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.979 7f72ea354700 1 -- 172.21.2.117:0/270491858 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f72e40ff4f0 con 0x7f72e40fe6d0 2021-02-10T11:57:13.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.983 7f72e0ff9700 1 -- 172.21.2.117:0/270491858 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v109) v1 ==== 184+0+0 (secure 0 0 0) 0x7f72d8007350 con 0x7f72e40fe6d0 2021-02-10T11:57:13.989 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:57:13.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 msgr2=0x7f72c401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 0x7f72c401abc0 crc :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 msgr2=0x7f72e40fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:13.991 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 rx=0x7f72d8006f00 tx=0x7f72d800b940).stop 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 shutdown_connections 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f72c40186c0 0x7f72c401abc0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f72e40fe6d0 0x7f72e40fead0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f72e40ff0f0 0x7f72e40ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 --2- 172.21.2.117:0/270491858 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f72e40ac9e0 0x7f72e41919a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 >> 172.21.2.117:0/270491858 conn(0x7f72e419f690 msgr2=0x7f72e4194fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 shutdown_connections 2021-02-10T11:57:13.992 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:13.989 7f72ea354700 1 -- 172.21.2.117:0/270491858 wait complete. 2021-02-10T11:57:14.000 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:57:14.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 Processor -- start 2021-02-10T11:57:14.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 -- start start 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f007ab40 0x7fa3f00794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 0x7fa3f0079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3f007af40 0x7fa3f007b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f007a5c0 con 0x7fa3f007af40 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f00e5770 con 0x7fa3f0079b10 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.423 7fa3f6d4a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00d18c0 con 0x7fa3f007ab40 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3f007af40 0x7fa3f007b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa3f007af40 0x7fa3f007b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60332/0 (socket says 172.21.2.117:60332) 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 -- 172.21.2.117:0/2289160942 learned_addr learned my addr 172.21.2.117:0/2289160942 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3effff700 1 --2- 172.21.2.117:0/2289160942 >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 0x7fa3f0079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f4b66700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f007ab40 0x7fa3f00794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 -- 172.21.2.117:0/2289160942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f007ab40 msgr2=0x7fa3f00794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f007ab40 0x7fa3f00794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 -- 172.21.2.117:0/2289160942 >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 msgr2=0x7fa3f0079f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 --2- 172.21.2.117:0/2289160942 >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 0x7fa3f0079f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 -- 172.21.2.117:0/2289160942 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00d1cb0 con 0x7fa3f007af40 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3effff700 1 --2- 172.21.2.117:0/2289160942 >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 0x7fa3f0079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f5367700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f007af40 0x7fa3f007b340 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7fa3e8006b80 tx=0x7fa3e8009b10).ready entity=mon.0 client_cookie=c7e9f2df80ebf0a8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3ef7fe700 1 -- 172.21.2.117:0/2289160942 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e80022e0 con 0x7fa3f007af40 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3ef7fe700 1 -- 172.21.2.117:0/2289160942 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e8002560 con 0x7fa3f007af40 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3ef7fe700 1 -- 172.21.2.117:0/2289160942 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e8002a40 con 0x7fa3f007af40 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f6d4a700 1 -- 172.21.2.117:0/2289160942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f007af40 msgr2=0x7fa3f007b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.424 7fa3f6d4a700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f007af40 0x7fa3f007b340 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 rx=0x7fa3e8006b80 tx=0x7fa3e8009b10).stop 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 -- 172.21.2.117:0/2289160942 shutdown_connections 2021-02-10T11:57:14.444 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 --2- 172.21.2.117:0/2289160942 >> v2:172.21.2.125:3300/0 conn(0x7fa3f0079b10 0x7fa3f0079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f007af40 0x7fa3f007b340 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 --2- 172.21.2.117:0/2289160942 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f007ab40 0x7fa3f00794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 -- 172.21.2.117:0/2289160942 >> 172.21.2.117:0/2289160942 conn(0x7fa3f00d0910 msgr2=0x7fa3f00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 -- 172.21.2.117:0/2289160942 shutdown_connections 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.425 7fa3f6d4a700 1 -- 172.21.2.117:0/2289160942 wait complete. 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 Processor -- start 2021-02-10T11:57:14.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 -- start start 2021-02-10T11:57:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f0100db0 0x7fa3f00ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3f009a4d0 0x7fa3f009a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0100b30 con 0x7fa3f0099960 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0099f50 con 0x7fa3f009a4d0 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f6d4a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f009aef0 con 0x7fa3f0100db0 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3effff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60334/0 (socket says 172.21.2.117:60334) 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3effff700 1 -- 172.21.2.117:0/1197697291 learned_addr learned my addr 172.21.2.117:0/1197697291 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f5367700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3f009a4d0 0x7fa3f009a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.440 7fa3f4b66700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f0100db0 0x7fa3f00ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 -- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f0100db0 msgr2=0x7fa3f00ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f0100db0 0x7fa3f00ffdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 -- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3f009a4d0 msgr2=0x7fa3f009a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3f009a4d0 0x7fa3f009a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 -- 172.21.2.117:0/1197697291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f009b0d0 con 0x7fa3f0099960 2021-02-10T11:57:14.447 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3effff700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7fa3e000b970 tx=0x7fa3e000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e0000d30 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e0006240 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa3e0005660 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f009b0d0 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f009b660 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa3e0006420 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 0x7fa3d401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa3e0032a70 con 0x7fa3f0099960 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3f4b66700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 0x7fa3d401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:14.448 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.442 7fa3f4b66700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 0x7fa3d401abc0 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:14.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.473 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f0099f50 con 0x7fa3f0099960 2021-02-10T11:57:14.479 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.477 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa3e0025f30 con 0x7fa3f0099960 2021-02-10T11:57:14.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.606 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa3f0099f50 con 0x7fa3f0099960 2021-02-10T11:57:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.608 7fa3ed7fa700 1 -- 172.21.2.117:0/1197697291 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v110) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa3e0025f30 con 0x7fa3f0099960 2021-02-10T11:57:14.611 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:57:14.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 msgr2=0x7fa3d401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:14.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 0x7fa3d401abc0 crc :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 msgr2=0x7fa3f0100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:14.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 rx=0x7fa3e000b970 tx=0x7fa3e000bd10).stop 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 shutdown_connections 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa3d40186c0 0x7fa3d401abc0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa3f0099960 0x7fa3f0100510 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa3f009a4d0 0x7fa3f009a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 --2- 172.21.2.117:0/1197697291 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa3f0100db0 0x7fa3f00ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 >> 172.21.2.117:0/1197697291 conn(0x7fa3f008f530 msgr2=0x7fa3f00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 shutdown_connections 2021-02-10T11:57:14.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:14.611 7fa3f6d4a700 1 -- 172.21.2.117:0/1197697291 wait complete. 2021-02-10T11:57:14.622 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:57:15.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.045 7f236eba1700 1 Processor -- start 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.045 7f236eba1700 1 -- start start 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236eba1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2368079ce0 0x7f23681b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236eba1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236807a5c0 con 0x7f236807a0e0 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236eba1700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23680d18c0 con 0x7f236807ab40 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236eba1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236807bf50 con 0x7f2368079ce0 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236d1be700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f236807a0e0 0x7f23681b8000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60596/0 (socket says 172.21.2.117:60596) 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f236d1be700 1 -- 172.21.2.117:0/1389736685 learned_addr learned my addr 172.21.2.117:0/1389736685 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f2367fff700 1 --2- 172.21.2.117:0/1389736685 >> v2:172.21.2.117:3300/0 conn(0x7f2368079ce0 0x7f23681b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f2367fff700 1 -- 172.21.2.117:0/1389736685 >> v1:172.21.2.125:6789/0 conn(0x7f236807ab40 legacy=0x7f23680794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f2367fff700 1 -- 172.21.2.117:0/1389736685 >> v1:172.21.2.117:6789/0 conn(0x7f236807a0e0 legacy=0x7f23681b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f2367fff700 1 -- 172.21.2.117:0/1389736685 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680d1c60 con 0x7f2368079ce0 2021-02-10T11:57:15.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.046 7f2367fff700 1 --2- 172.21.2.117:0/1389736685 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2368079ce0 0x7f23681b59d0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f23580069d0 tx=0x7f2358009e40).ready entity=mon.0 client_cookie=bb3a7245876165d4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.047 7f23677fe700 1 -- 172.21.2.117:0/1389736685 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235800b510 con 0x7f2368079ce0 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.047 7f23677fe700 1 -- 172.21.2.117:0/1389736685 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235800b890 con 0x7f2368079ce0 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.047 7f23677fe700 1 -- 172.21.2.117:0/1389736685 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235800bcf0 con 0x7f2368079ce0 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.047 7f236eba1700 1 -- 172.21.2.117:0/1389736685 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2368079ce0 msgr2=0x7f23681b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.047 7f236eba1700 1 --2- 172.21.2.117:0/1389736685 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2368079ce0 0x7f23681b59d0 secure :-1 s=READY pgs=3127 cs=0 l=1 rev1=1 rx=0x7f23580069d0 tx=0x7f2358009e40).stop 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.048 7f236eba1700 1 -- 172.21.2.117:0/1389736685 shutdown_connections 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.048 7f236eba1700 1 --2- 172.21.2.117:0/1389736685 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2368079ce0 0x7f23681b59d0 unknown :-1 s=CLOSED pgs=3127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.048 7f236eba1700 1 -- 172.21.2.117:0/1389736685 >> 172.21.2.117:0/1389736685 conn(0x7f23680d0910 msgr2=0x7f23680d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.048 7f236eba1700 1 -- 172.21.2.117:0/1389736685 shutdown_connections 2021-02-10T11:57:15.065 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.048 7f236eba1700 1 -- 172.21.2.117:0/1389736685 wait complete. 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.062 7f236eba1700 1 Processor -- start 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.062 7f236eba1700 1 -- start start 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23681c6940 0x7f23680feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 0x7f23680ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23680c3910 con 0x7f236809bf80 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2368098c30 con 0x7f23680ff0c0 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236eba1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2368091e90 con 0x7f23681c6940 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60350/0 (socket says 172.21.2.117:60350) 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 -- 172.21.2.117:0/2016604634 learned_addr learned my addr 172.21.2.117:0/2016604634 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236d1be700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 0x7f23680ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f2367fff700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23681c6940 0x7f23680feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 -- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23681c6940 msgr2=0x7f23680feaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23681c6940 0x7f23680feaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 -- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 msgr2=0x7f23680ff4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 0x7f23680ff4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236c9bd700 1 -- 172.21.2.117:0/2016604634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680ba880 con 0x7f236809bf80 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.063 7f236d1be700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 0x7f23680ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f236c9bd700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f235000b5b0 tx=0x7f235000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f235000bd40 con 0x7f236809bf80 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350006240 con 0x7f236809bf80 2021-02-10T11:57:15.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2350005690 con 0x7f236809bf80 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f236eba1700 1 -- 172.21.2.117:0/2016604634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23680ba880 con 0x7f236809bf80 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.064 7f236eba1700 1 -- 172.21.2.117:0/2016604634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23680bae10 con 0x7f236809bf80 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.065 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f23500054a0 con 0x7f236809bf80 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.065 7f23657fa700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.065 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2350032920 con 0x7f236809bf80 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.065 7f2367fff700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.066 7f2367fff700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.096 7f236eba1700 1 -- 172.21.2.117:0/2016604634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23680d0910 con 0x7f236809bf80 2021-02-10T11:57:15.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.098 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2350000c60 con 0x7f236809bf80 2021-02-10T11:57:15.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.229 7f236eba1700 1 -- 172.21.2.117:0/2016604634 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f23680d0910 con 0x7f236809bf80 2021-02-10T11:57:15.236 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.234 7f23657fa700 1 -- 172.21.2.117:0/2016604634 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v111) v1 ==== 184+0+0 (secure 0 0 0) 0x7f23500054a0 con 0x7f236809bf80 2021-02-10T11:57:15.239 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:57:15.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 msgr2=0x7f234c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 crc :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 msgr2=0x7f23681c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 rx=0x7f235000b5b0 tx=0x7f235000b950).stop 2021-02-10T11:57:15.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 shutdown_connections 2021-02-10T11:57:15.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f23680ff0c0 0x7f23680ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23681c6940 0x7f23680feaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f234c0186c0 0x7f234c01abc0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 --2- 172.21.2.117:0/2016604634 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f236809bf80 0x7f23681c6320 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 >> 172.21.2.117:0/2016604634 conn(0x7f236809e020 msgr2=0x7f23680fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:15.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 shutdown_connections 2021-02-10T11:57:15.242 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.238 7f236eba1700 1 -- 172.21.2.117:0/2016604634 wait complete. 2021-02-10T11:57:15.249 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.659 7f3348819700 1 Processor -- start 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.659 7f3348819700 1 -- start start 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3348819700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d18e0 0x7f33440ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3348819700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33440ede60 con 0x7f33440eb000 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3348819700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33440d1ce0 con 0x7f33440ef3f0 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3348819700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3344088020 con 0x7f33440d18e0 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3341e1b700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f33440eb000 0x7f33440eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60606/0 (socket says 172.21.2.117:60606) 2021-02-10T11:57:15.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f3341e1b700 1 -- 172.21.2.117:0/4226864442 learned_addr learned my addr 172.21.2.117:0/4226864442 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529392044 0 0) 0x7f332c0010b0 con 0x7f33440eb000 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.660 7f334161a700 1 -- 172.21.2.117:0/4226864442 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3330002fe0 con 0x7f33440eb000 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334261c700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d18e0 0x7f33440ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4090659360 0 0) 0x7f332c0010b0 con 0x7f33440eb000 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d18e0 msgr2=0x7f33440ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440d18e0 0x7f33440ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 >> v1:172.21.2.125:6789/0 conn(0x7f33440ef3f0 legacy=0x7f33441b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33441bb180 con 0x7f33440eb000 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f332c0015b0 con 0x7f33440eb000 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 >> v1:172.21.2.117:6789/0 conn(0x7f33440eb000 legacy=0x7f33440eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3330003b10 0x7f3330005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3330006800 0x7f3330008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3330009520 0x7f333000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333000c2d0 con 0x7f3330009520 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333000c4f0 con 0x7f3330006800 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333000c710 con 0x7f3330003b10 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3330009520 0x7f333000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3341e1b700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3330006800 0x7f3330008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334261c700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3330003b10 0x7f3330005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 -- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3330003b10 msgr2=0x7f3330005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3330003b10 0x7f3330005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 -- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3330006800 msgr2=0x7f3330008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3330006800 0x7f3330008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 -- 172.21.2.117:0/4226864442 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330050980 con 0x7f3330009520 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f3342e1d700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3330009520 0x7f333000ba20 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 rx=0x7f333c006e40 tx=0x7f333c009e30).ready entity=mon.0 client_cookie=9785d56448e48385 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c006a60 con 0x7f3330009520 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c00b300 con 0x7f3330009520 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.661 7f334161a700 1 -- 172.21.2.117:0/4226864442 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c00b820 con 0x7f3330009520 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 -- 172.21.2.117:0/4226864442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3330009520 msgr2=0x7f333000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 --2- 172.21.2.117:0/4226864442 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3330009520 0x7f333000ba20 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 rx=0x7f333c006e40 tx=0x7f333c009e30).stop 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3342e1d700 1 -- 172.21.2.117:0/4226864442 reap_dead start 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 -- 172.21.2.117:0/4226864442 shutdown_connections 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 -- 172.21.2.117:0/4226864442 >> 172.21.2.117:0/4226864442 conn(0x7f33440ced60 msgr2=0x7f33440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 -- 172.21.2.117:0/4226864442 shutdown_connections 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.662 7f3348819700 1 -- 172.21.2.117:0/4226864442 wait complete. 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 Processor -- start 2021-02-10T11:57:15.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 -- start start 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33441a4290 0x7f33441a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440ab280 0x7f33440ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440a7d10 con 0x7f33440a6fe0 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440a7a00 con 0x7f33441a4290 2021-02-10T11:57:15.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.677 7f3348819700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3344096370 con 0x7f33440ab280 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60368/0 (socket says 172.21.2.117:60368) 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 -- 172.21.2.117:0/2658712550 learned_addr learned my addr 172.21.2.117:0/2658712550 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 -- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440ab280 msgr2=0x7f33440ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440ab280 0x7f33440ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 -- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33441a4290 msgr2=0x7f33441a4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33441a4290 0x7f33441a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 -- 172.21.2.117:0/2658712550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33440a8290 con 0x7f33440a6fe0 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3342e1d700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7f333c00b910 tx=0x7f333c00bcb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c009650 con 0x7f33440a6fe0 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c000cc0 con 0x7f33440a6fe0 2021-02-10T11:57:15.685 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3348819700 1 -- 172.21.2.117:0/2658712550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33440a8290 con 0x7f33440a6fe0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f3348819700 1 -- 172.21.2.117:0/2658712550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33440a8820 con 0x7f33440a6fe0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.678 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f333c00c4e0 con 0x7f33440a6fe0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.679 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f333c009650 con 0x7f33440a6fe0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.679 7f333affd700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 0x7f332801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.679 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f333c032cc0 con 0x7f33440a6fe0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.680 7f334261c700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 0x7f332801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:15.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.680 7f334261c700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 0x7f332801abd0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:15.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.709 7f3348819700 1 -- 172.21.2.117:0/2658712550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344087290 con 0x7f33440a6fe0 2021-02-10T11:57:15.717 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.712 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f333c003020 con 0x7f33440a6fe0 2021-02-10T11:57:15.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.841 7f3348819700 1 -- 172.21.2.117:0/2658712550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33440ede60 con 0x7f33440a6fe0 2021-02-10T11:57:15.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.841 7f333affd700 1 -- 172.21.2.117:0/2658712550 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f333c009650 con 0x7f33440a6fe0 2021-02-10T11:57:15.844 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:15.844 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":161,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:11.517907","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8153":{"gid":8153,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/2333277299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2333277299},{"type":"v1","addr":"172.21.2.125:6835","nonce":2333277299}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:15.844 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 161 2021-02-10T11:57:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 msgr2=0x7f332801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 0x7f332801abd0 crc :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 msgr2=0x7f33440a73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:15.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7f333c00b910 tx=0x7f333c00bcb0).stop 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 shutdown_connections 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f33280186d0 0x7f332801abd0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f33440a6fe0 0x7f33440a73e0 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f33440ab280 0x7f33440ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 --2- 172.21.2.117:0/2658712550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f33441a4290 0x7f33441a4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 >> 172.21.2.117:0/2658712550 conn(0x7f33440874c0 msgr2=0x7f3344096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 shutdown_connections 2021-02-10T11:57:15.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:15.843 7f3348819700 1 -- 172.21.2.117:0/2658712550 wait complete. 2021-02-10T11:57:16.055 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 161, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:57:09.509260', 'modified': '2021-02-10 11:57:11.517907', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8153}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8153': {'gid': 8153, 'name': 'b', 'rank': 0, 'incarnation': 158, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/2333277299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 2333277299}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 2333277299}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:57:16.055 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:57:16.056 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:57:16.456 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:57:16.457 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:57:16.457 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:57:16.457 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8153, 2021-02-10T11:57:16.457 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:57:16.457 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9431260879999996, 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 161, 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 187, 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 187, 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.596828819000001 2021-02-10T11:57:16.458 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:57:16.461 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8153, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9431260879999996, "mdsmap_epoch": 161, "osdmap_epoch": 187, "osdmap_epoch_barrier": 187, "uptime": 12.596828819000001 } 2021-02-10T11:57:16.462 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:16.893 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 Processor -- start 2021-02-10T11:57:16.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 -- start start 2021-02-10T11:57:16.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 0x7f5e080ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 0x7f5e080eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e080ee3e0 0x7f5e080ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e080ede60 con 0x7f5e080ee3e0 2021-02-10T11:57:16.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e080eba20 con 0x7f5e080eb000 2021-02-10T11:57:16.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.874 7f5e0e33e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080d1ce0 con 0x7f5e080d18e0 2021-02-10T11:57:16.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e080ee3e0 0x7f5e080ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5e080ee3e0 0x7f5e080ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60376/0 (socket says 172.21.2.117:60376) 2021-02-10T11:57:16.895 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 -- 172.21.2.117:0/1733391277 learned_addr learned my addr 172.21.2.117:0/1733391277 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e07fff700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 0x7f5e080ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 -- 172.21.2.117:0/1733391277 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 msgr2=0x7f5e080ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 0x7f5e080ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 -- 172.21.2.117:0/1733391277 >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 msgr2=0x7f5e080eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e077fe700 1 --2- 172.21.2.117:0/1733391277 >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 0x7f5e080eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 --2- 172.21.2.117:0/1733391277 >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 0x7f5e080eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 -- 172.21.2.117:0/1733391277 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e080eee00 con 0x7f5e080ee3e0 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e07fff700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 0x7f5e080ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e077fe700 1 --2- 172.21.2.117:0/1733391277 >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 0x7f5e080eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.875 7f5e0c95b700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e080ee3e0 0x7f5e080ee7e0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 rx=0x7f5e00006f00 tx=0x7f5e00009f10).ready entity=mon.0 client_cookie=3a4a680becc0f68f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.876 7f5e06ffd700 1 -- 172.21.2.117:0/1733391277 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0000b3b0 con 0x7f5e080ee3e0 2021-02-10T11:57:16.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.876 7f5e06ffd700 1 -- 172.21.2.117:0/1733391277 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e00000d30 con 0x7f5e080ee3e0 2021-02-10T11:57:16.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.876 7f5e06ffd700 1 -- 172.21.2.117:0/1733391277 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e0000b820 con 0x7f5e080ee3e0 2021-02-10T11:57:16.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.876 7f5e0e33e700 1 -- 172.21.2.117:0/1733391277 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e080ee3e0 msgr2=0x7f5e080ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:16.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.876 7f5e0e33e700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e080ee3e0 0x7f5e080ee7e0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 rx=0x7f5e00006f00 tx=0x7f5e00009f10).stop 2021-02-10T11:57:16.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 -- 172.21.2.117:0/1733391277 shutdown_connections 2021-02-10T11:57:16.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 --2- 172.21.2.117:0/1733391277 >> v2:172.21.2.125:3300/0 conn(0x7f5e080eb000 0x7f5e080eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e080ee3e0 0x7f5e080ee7e0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 --2- 172.21.2.117:0/1733391277 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e080d18e0 0x7f5e080ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 -- 172.21.2.117:0/1733391277 >> 172.21.2.117:0/1733391277 conn(0x7f5e080ced60 msgr2=0x7f5e080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 -- 172.21.2.117:0/1733391277 shutdown_connections 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.877 7f5e0e33e700 1 -- 172.21.2.117:0/1733391277 wait complete. 2021-02-10T11:57:16.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.891 7f5e0e33e700 1 Processor -- start 2021-02-10T11:57:16.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.891 7f5e0e33e700 1 -- start start 2021-02-10T11:57:16.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 0x7f5e0808a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e0809e3f0 0x7f5e0809e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0808e380 con 0x7f5e0808b760 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0808c180 con 0x7f5e0808b260 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0e33e700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0808ad30 con 0x7f5e0809e3f0 2021-02-10T11:57:16.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e077fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60386/0 (socket says 172.21.2.117:60386) 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e077fe700 1 -- 172.21.2.117:0/29131167 learned_addr learned my addr 172.21.2.117:0/29131167 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e07fff700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 0x7f5e0808a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.892 7f5e0c95b700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e0809e3f0 0x7f5e0809e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 -- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e0809e3f0 msgr2=0x7f5e0809e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:16.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e0809e3f0 0x7f5e0809e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 -- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 msgr2=0x7f5e0808a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 0x7f5e0808a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 -- 172.21.2.117:0/29131167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0809ee10 con 0x7f5e0808b760 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e07fff700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 0x7f5e0808a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e077fe700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 rx=0x7f5dfc006f20 tx=0x7f5dfc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:16.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc00bd80 con 0x7f5e0808b760 2021-02-10T11:57:16.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc000c90 con 0x7f5e0808b760 2021-02-10T11:57:16.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5dfc0022b0 con 0x7f5e0808b760 2021-02-10T11:57:16.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0809ee10 con 0x7f5e0808b760 2021-02-10T11:57:16.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.893 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0809f3a0 con 0x7f5e0808b760 2021-02-10T11:57:16.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.894 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5dfc00a000 con 0x7f5e0808b760 2021-02-10T11:57:16.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.894 7f5e04ff9700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 0x7f5de801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:16.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.894 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5dfc032680 con 0x7f5e0808b760 2021-02-10T11:57:16.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.894 7f5e07fff700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 0x7f5de801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:16.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.895 7f5e07fff700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 0x7f5de801abc0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:16.927 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.924 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e08087290 con 0x7f5e0808b760 2021-02-10T11:57:16.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:16.927 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5dfc000960 con 0x7f5e0808b760 2021-02-10T11:57:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.050 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e080ede60 con 0x7f5e0808b760 2021-02-10T11:57:17.052 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.051 7f5e04ff9700 1 -- 172.21.2.117:0/29131167 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f5dfc00a000 con 0x7f5e0808b760 2021-02-10T11:57:17.053 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:17.053 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":187,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:10.511294","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:57:07.475412","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:57:08.504122","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:17.054 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 msgr2=0x7f5de801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 0x7f5de801abc0 crc :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 msgr2=0x7f5e0808bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 rx=0x7f5dfc006f20 tx=0x7f5dfc00b990).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 shutdown_connections 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5de80186c0 0x7f5de801abc0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e0808b760 0x7f5e0808bb60 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e0809e3f0 0x7f5e0809e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 --2- 172.21.2.117:0/29131167 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e0808b260 0x7f5e0808a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.052 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 >> 172.21.2.117:0/29131167 conn(0x7f5e0808e740 msgr2=0x7f5e081b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.053 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 shutdown_connections 2021-02-10T11:57:17.055 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.053 7f5e0e33e700 1 -- 172.21.2.117:0/29131167 wait complete. 2021-02-10T11:57:17.063 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:57:07.475412', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:57:08.504122', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:57:17.064 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:57:17.497 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.479 7f1467aa5700 1 Processor -- start 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.479 7f1467aa5700 1 -- start start 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f1467aa5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f14601b07f0 0x7f14600e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f1467aa5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14600e6170 0x7f14600e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f1467aa5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600e77f0 con 0x7f14600e6b90 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f1467aa5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600e5ce0 con 0x7f14600e6170 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f1467aa5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600e54b0 con 0x7f14601b07f0 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14600e6170 0x7f14600e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14600e6170 0x7f14600e6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60390/0 (socket says 172.21.2.117:60390) 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 -- 172.21.2.117:0/2300109623 learned_addr learned my addr 172.21.2.117:0/2300109623 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14658c1700 1 --2- 172.21.2.117:0/2300109623 >> v2:172.21.2.125:3300/0 conn(0x7f14601b07f0 0x7f14600e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 -- 172.21.2.117:0/2300109623 >> v2:172.21.2.125:3300/0 conn(0x7f14601b07f0 msgr2=0x7f14600e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 --2- 172.21.2.117:0/2300109623 >> v2:172.21.2.125:3300/0 conn(0x7f14601b07f0 0x7f14600e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 -- 172.21.2.117:0/2300109623 >> v1:172.21.2.117:6789/0 conn(0x7f14600e6b90 legacy=0x7f14600e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.480 7f14650c0700 1 -- 172.21.2.117:0/2300109623 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600e8d80 con 0x7f14600e6170 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f14650c0700 1 --2- 172.21.2.117:0/2300109623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14600e6170 0x7f14600e6570 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7f1458006a80 tx=0x7f1458009f00).ready entity=mon.0 client_cookie=52a2554e74952f8d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f14648bf700 1 -- 172.21.2.117:0/2300109623 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145800b5c0 con 0x7f14600e6170 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f14648bf700 1 -- 172.21.2.117:0/2300109623 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800b840 con 0x7f14600e6170 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f14648bf700 1 -- 172.21.2.117:0/2300109623 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145800bd20 con 0x7f14600e6170 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 -- 172.21.2.117:0/2300109623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14600e6170 msgr2=0x7f14600e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 --2- 172.21.2.117:0/2300109623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14600e6170 0x7f14600e6570 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 rx=0x7f1458006a80 tx=0x7f1458009f00).stop 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 -- 172.21.2.117:0/2300109623 shutdown_connections 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 --2- 172.21.2.117:0/2300109623 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14600e6170 0x7f14600e6570 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.499 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 --2- 172.21.2.117:0/2300109623 >> v2:172.21.2.125:3300/0 conn(0x7f14601b07f0 0x7f14600e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.481 7f1467aa5700 1 -- 172.21.2.117:0/2300109623 >> 172.21.2.117:0/2300109623 conn(0x7f14600ce500 msgr2=0x7f14600e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:17.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.482 7f1467aa5700 1 -- 172.21.2.117:0/2300109623 shutdown_connections 2021-02-10T11:57:17.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.482 7f1467aa5700 1 -- 172.21.2.117:0/2300109623 wait complete. 2021-02-10T11:57:17.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.496 7f1467aa5700 1 Processor -- start 2021-02-10T11:57:17.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.496 7f1467aa5700 1 -- start start 2021-02-10T11:57:17.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14600a7720 0x7f1460094af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 0x7f14600c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600a5c70 con 0x7f146019a340 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1460095110 con 0x7f146019a740 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f1467aa5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600c4060 con 0x7f14600a7720 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14660c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14660c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36548/0 (socket says 172.21.2.117:36548) 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14660c2700 1 -- 172.21.2.117:0/3133506922 learned_addr learned my addr 172.21.2.117:0/3133506922 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 0x7f14600c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14658c1700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14600a7720 0x7f1460094af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 -- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14600a7720 msgr2=0x7f1460094af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14600a7720 0x7f1460094af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 -- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 msgr2=0x7f14600c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 -- 172.21.2.117:0/3133506922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600c3bb0 con 0x7f146019a340 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14660c2700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.497 7f14650c0700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 0x7f14600c2e50 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7f1458006ed0 tx=0x7f145800bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145800c2f0 con 0x7f146019a340 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458006980 con 0x7f146019a340 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f145800c820 con 0x7f146019a340 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14600c3bb0 con 0x7f146019a340 2021-02-10T11:57:17.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14600c3bb0 con 0x7f146019a340 2021-02-10T11:57:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1458009570 con 0x7f146019a340 2021-02-10T11:57:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.498 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1458032980 con 0x7f146019a340 2021-02-10T11:57:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.499 7f14658c1700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:17.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.499 7f14658c1700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 0x7f144801abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:17.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.530 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146007e800 con 0x7f146019a340 2021-02-10T11:57:17.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.533 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f145800bd80 con 0x7f146019a340 2021-02-10T11:57:17.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.662 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f14600e77f0 con 0x7f146019a340 2021-02-10T11:57:17.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.662 7f14567fc700 1 -- 172.21.2.117:0/3133506922 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v187) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1458009570 con 0x7f146019a340 2021-02-10T11:57:17.665 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.664 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 msgr2=0x7f144801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 0x7f144801abc0 crc :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 msgr2=0x7f14600c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 0x7f14600c2e50 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 rx=0x7f1458006ed0 tx=0x7f145800bc40).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 shutdown_connections 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14480186c0 0x7f144801abc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f146019a340 0x7f14600c2e50 unknown :-1 s=CLOSED pgs=3136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f146019a740 0x7f14600c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 --2- 172.21.2.117:0/3133506922 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14600a7720 0x7f1460094af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 >> 172.21.2.117:0/3133506922 conn(0x7f14600a7210 msgr2=0x7f1460095680 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 shutdown_connections 2021-02-10T11:57:17.667 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:17.665 7f1467aa5700 1 -- 172.21.2.117:0/3133506922 wait complete. 2021-02-10T11:57:17.675 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.090 7f3ebc6f7700 1 Processor -- start 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.090 7f3ebc6f7700 1 -- start start 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.090 7f3ebc6f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 0x7f3eb40eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebc6f7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 0x7f3eb41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebc6f7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb40ede60 con 0x7f3eb40eb040 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebc6f7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb4088020 con 0x7f3eb40ef3f0 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebc6f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40d1d30 con 0x7f3eb40d18f0 2021-02-10T11:57:18.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eb9d12700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3eb40eb040 0x7f3eb40eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60654/0 (socket says 172.21.2.117:60654) 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eb9d12700 1 -- 172.21.2.117:0/3832021107 learned_addr learned my addr 172.21.2.117:0/3832021107 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebad14700 1 --2- 172.21.2.117:0/3832021107 >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 0x7f3eb41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eba513700 1 --2- 172.21.2.117:0/3832021107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 0x7f3eb40eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eb9511700 1 -- 172.21.2.117:0/3832021107 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885104555 0 0) 0x7f3ea8001180 con 0x7f3eb40eb040 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eb9511700 1 -- 172.21.2.117:0/3832021107 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ea0002fe0 con 0x7f3eb40eb040 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eba513700 1 -- 172.21.2.117:0/3832021107 >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 msgr2=0x7f3eb41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eba513700 1 --2- 172.21.2.117:0/3832021107 >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 0x7f3eb41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eba513700 1 -- 172.21.2.117:0/3832021107 >> v1:172.21.2.117:6789/0 conn(0x7f3eb40eb040 legacy=0x7f3eb40eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3eba513700 1 -- 172.21.2.117:0/3832021107 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb41b8fd0 con 0x7f3eb40d18f0 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.091 7f3ebad14700 1 --2- 172.21.2.117:0/3832021107 >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 0x7f3eb41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3eba513700 1 --2- 172.21.2.117:0/3832021107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 0x7f3eb40eaa20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3eb0006650 tx=0x7f3eb0009a80).ready entity=mon.2 client_cookie=21884146eaca9792 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3eb9511700 1 -- 172.21.2.117:0/3832021107 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb00009b0 con 0x7f3eb40d18f0 2021-02-10T11:57:18.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3eb9511700 1 -- 172.21.2.117:0/3832021107 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000c0c0 con 0x7f3eb40d18f0 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3eb9511700 1 -- 172.21.2.117:0/3832021107 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3eb000c510 con 0x7f3eb40d18f0 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 -- 172.21.2.117:0/3832021107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 msgr2=0x7f3eb40eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 --2- 172.21.2.117:0/3832021107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 0x7f3eb40eaa20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3eb0006650 tx=0x7f3eb0009a80).stop 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 -- 172.21.2.117:0/3832021107 shutdown_connections 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 --2- 172.21.2.117:0/3832021107 >> v2:172.21.2.125:3300/0 conn(0x7f3eb40ef3f0 0x7f3eb41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 --2- 172.21.2.117:0/3832021107 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40d18f0 0x7f3eb40eaa20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.092 7f3ebc6f7700 1 -- 172.21.2.117:0/3832021107 >> 172.21.2.117:0/3832021107 conn(0x7f3eb40cf060 msgr2=0x7f3eb40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.093 7f3ebc6f7700 1 -- 172.21.2.117:0/3832021107 shutdown_connections 2021-02-10T11:57:18.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.093 7f3ebc6f7700 1 -- 172.21.2.117:0/3832021107 wait complete. 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.107 7f3ebc6f7700 1 Processor -- start 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.107 7f3ebc6f7700 1 -- start start 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3eb41a7800 0x7f3eb41a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3eb40a2db0 0x7f3eb40a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b7e50 con 0x7f3eb40a2db0 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b8040 con 0x7f3eb41a7800 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebc6f7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40b8260 con 0x7f3eb40a2390 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.113 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3ebad14700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3eb40a2db0 0x7f3eb40a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58090/0 (socket says 172.21.2.117:58090) 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 -- 172.21.2.117:0/312757443 learned_addr learned my addr 172.21.2.117:0/312757443 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eba513700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3eb41a7800 0x7f3eb41a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 -- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3eb41a7800 msgr2=0x7f3eb41a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3eb41a7800 0x7f3eb41a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 -- 172.21.2.117:0/312757443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3eb40a2db0 msgr2=0x7f3eb40a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3eb40a2db0 0x7f3eb40a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.108 7f3eb9d12700 1 -- 172.21.2.117:0/312757443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40b8480 con 0x7f3eb40a2390 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3eb9d12700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3ea800a4d0 tx=0x7f3ea800a870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea8000ce0 con 0x7f3eb40a2390 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800ac60 con 0x7f3eb40a2390 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb40b8480 con 0x7f3eb40a2390 2021-02-10T11:57:18.114 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb40b8a10 con 0x7f3eb40a2390 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.109 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3ea8004620 con 0x7f3eb40a2390 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.110 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3ea8008020 con 0x7f3eb40a2390 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.110 7f3ea6ffd700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 0x7f3e9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.110 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3ea80328d0 con 0x7f3eb40a2390 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.110 7f3eba513700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 0x7f3e9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:18.115 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.110 7f3eba513700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 0x7f3e9c01abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:18.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.141 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb40daff0 con 0x7f3eb40a2390 2021-02-10T11:57:18.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.143 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3ea8000cc0 con 0x7f3eb40a2390 2021-02-10T11:57:18.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.270 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f3eb40ede60 con 0x7f3eb40a2390 2021-02-10T11:57:18.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.272 7f3ea6ffd700 1 -- 172.21.2.117:0/312757443 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v187) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3ea800ac60 con 0x7f3eb40a2390 2021-02-10T11:57:18.275 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:18.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 msgr2=0x7f3e9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 0x7f3e9c01abc0 crc :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 msgr2=0x7f3eb40a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:18.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3ea800a4d0 tx=0x7f3ea800a870).stop 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 shutdown_connections 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3e9c0186c0 0x7f3e9c01abc0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3eb40a2db0 0x7f3eb40a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3eb40a2390 0x7f3eb40a2790 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 --2- 172.21.2.117:0/312757443 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3eb41a7800 0x7f3eb41a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 >> 172.21.2.117:0/312757443 conn(0x7f3eb40bbeb0 msgr2=0x7f3eb408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:18.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 shutdown_connections 2021-02-10T11:57:18.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.275 7f3ebc6f7700 1 -- 172.21.2.117:0/312757443 wait complete. 2021-02-10T11:57:18.286 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:57:18.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.700 7f8dc226f700 1 Processor -- start 2021-02-10T11:57:19.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.700 7f8dc226f700 1 -- start start 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc226f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 0x7f8dbc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc226f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc226f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dbc0ef7f0 con 0x7f8dbc0eb040 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc226f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dbc0eded0 con 0x7f8dbc0d18f0 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc226f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc0d1cf0 con 0x7f8dbc0ef3f0 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc088c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dbbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 0x7f8dbc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc088c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58098/0 (socket says 172.21.2.117:58098) 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8dc088c700 1 -- 172.21.2.117:0/1072373559 learned_addr learned my addr 172.21.2.117:0/1072373559 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.701 7f8db3fff700 1 --1- 172.21.2.117:0/1072373559 >> v1:172.21.2.117:6789/0 conn(0x7f8dbc0eb040 0x7f8dbc0eb440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60668/0 (socket says 172.21.2.117:60668) 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc088c700 1 -- 172.21.2.117:0/1072373559 >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 msgr2=0x7f8dbc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:19.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc088c700 1 --2- 172.21.2.117:0/1072373559 >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 0x7f8dbc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc088c700 1 -- 172.21.2.117:0/1072373559 >> v1:172.21.2.117:6789/0 conn(0x7f8dbc0eb040 legacy=0x7f8dbc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc088c700 1 -- 172.21.2.117:0/1072373559 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc1b8fd0 con 0x7f8dbc0ef3f0 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dbbfff700 1 --2- 172.21.2.117:0/1072373559 >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 0x7f8dbc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc088c700 1 --2- 172.21.2.117:0/1072373559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8db4006720 tx=0x7f8db4009ba0).ready entity=mon.2 client_cookie=b4e98e84cf740835 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dbb7fe700 1 -- 172.21.2.117:0/1072373559 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db40022e0 con 0x7f8dbc0ef3f0 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dbb7fe700 1 -- 172.21.2.117:0/1072373559 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db4002560 con 0x7f8dbc0ef3f0 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dbb7fe700 1 -- 172.21.2.117:0/1072373559 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8db4002a80 con 0x7f8dbc0ef3f0 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc226f700 1 -- 172.21.2.117:0/1072373559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 msgr2=0x7f8dbc1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.702 7f8dc226f700 1 --2- 172.21.2.117:0/1072373559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f8db4006720 tx=0x7f8db4009ba0).stop 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 -- 172.21.2.117:0/1072373559 shutdown_connections 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 --2- 172.21.2.117:0/1072373559 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0ef3f0 0x7f8dbc1b79a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 --2- 172.21.2.117:0/1072373559 >> v2:172.21.2.117:3300/0 conn(0x7f8dbc0d18f0 0x7f8dbc0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 -- 172.21.2.117:0/1072373559 >> 172.21.2.117:0/1072373559 conn(0x7f8dbc0cf060 msgr2=0x7f8dbc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 -- 172.21.2.117:0/1072373559 shutdown_connections 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.703 7f8dc226f700 1 -- 172.21.2.117:0/1072373559 wait complete. 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.717 7f8dc226f700 1 Processor -- start 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.717 7f8dc226f700 1 -- start start 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8dbc08cf40 0x7f8dbc0a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0a9f90 0x7f8dbc0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc0aae30 con 0x7f8dbc08cf40 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc0aa9f0 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc226f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc0ab330 con 0x7f8dbc0a9f90 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36576/0 (socket says 172.21.2.117:36576) 2021-02-10T11:57:19.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 -- 172.21.2.117:0/3830257984 learned_addr learned my addr 172.21.2.117:0/3830257984 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8db3fff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8dbc08cf40 0x7f8dbc0a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dc088c700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0a9f90 0x7f8dbc0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 -- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0a9f90 msgr2=0x7f8dbc0aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0a9f90 0x7f8dbc0aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 -- 172.21.2.117:0/3830257984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8dbc08cf40 msgr2=0x7f8dbc0a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8dbc08cf40 0x7f8dbc0a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.718 7f8dbbfff700 1 -- 172.21.2.117:0/3830257984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc0ab550 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8dbbfff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f8da4006f40 tx=0x7f8da400b990).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da400bd80 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da4000c90 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8da4002220 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dbc0ab550 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dbc0abae0 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.012 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8da4006b00 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.719 7f8db97fa700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 0x7f8da001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.720 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8da4032910 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.720 7f8db3fff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 0x7f8da001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.720 7f8db3fff700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 0x7f8da001abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.015 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.750 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dbc0ede60 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.753 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8da4000940 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.884 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f8dbc0ede60 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.885 7f8db97fa700 1 -- 172.21.2.117:0/3830257984 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v187) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8da4006b00 con 0x7f8dbc1a48b0 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 msgr2=0x7f8da001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 0x7f8da001abc0 crc :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 msgr2=0x7f8dbc08c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7f8da4006f40 tx=0x7f8da400b990).stop 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 shutdown_connections 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f8da00186c0 0x7f8da001abc0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8dbc08cf40 0x7f8dbc0a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8dbc0a9f90 0x7f8dbc0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 --2- 172.21.2.117:0/3830257984 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f8dbc1a48b0 0x7f8dbc08c920 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 >> 172.21.2.117:0/3830257984 conn(0x7f8dbc092d00 msgr2=0x7f8dbc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.886 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 shutdown_connections 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:18.887 7f8dc226f700 1 -- 172.21.2.117:0/3830257984 wait complete. 2021-02-10T11:57:19.016 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:19.017 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:57:19.017 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:19.019 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:57:19.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.430 7f479da35700 1 Processor -- start 2021-02-10T11:57:19.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.430 7f479da35700 1 -- start start 2021-02-10T11:57:19.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f479da35700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 0x7f47981b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f479da35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.449 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f479da35700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4798085930 con 0x7f47980e5240 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f479da35700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47980e5cc0 con 0x7f47980e4e40 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f479da35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4798084840 con 0x7f4798085450 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f4797fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f4797fff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58110/0 (socket says 172.21.2.117:58110) 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f4797fff700 1 -- 172.21.2.117:0/2757598246 learned_addr learned my addr 172.21.2.117:0/2757598246 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.431 7f47977fe700 1 --2- 172.21.2.117:0/2757598246 >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 0x7f47981b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f4797fff700 1 -- 172.21.2.117:0/2757598246 >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 msgr2=0x7f47981b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f4797fff700 1 --2- 172.21.2.117:0/2757598246 >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 0x7f47981b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f4797fff700 1 -- 172.21.2.117:0/2757598246 >> v1:172.21.2.125:6789/0 conn(0x7f47980e5240 legacy=0x7f4798084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f4797fff700 1 -- 172.21.2.117:0/2757598246 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47981b0810 con 0x7f4798085450 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f47977fe700 1 --2- 172.21.2.117:0/2757598246 >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 0x7f47981b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:19.450 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f4797fff700 1 --2- 172.21.2.117:0/2757598246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f4790006aa0 tx=0x7f4790009f20).ready entity=mon.2 client_cookie=7d07e71f2f8fb61a server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f47967fc700 1 -- 172.21.2.117:0/2757598246 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4790000f90 con 0x7f4798085450 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f47967fc700 1 -- 172.21.2.117:0/2757598246 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790003030 con 0x7f4798085450 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f47967fc700 1 -- 172.21.2.117:0/2757598246 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f479000b7b0 con 0x7f4798085450 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f479da35700 1 -- 172.21.2.117:0/2757598246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 msgr2=0x7f4798086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.432 7f479da35700 1 --2- 172.21.2.117:0/2757598246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f4790006aa0 tx=0x7f4790009f20).stop 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 -- 172.21.2.117:0/2757598246 shutdown_connections 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 --2- 172.21.2.117:0/2757598246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4798085450 0x7f4798086ec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 --2- 172.21.2.117:0/2757598246 >> v2:172.21.2.125:3300/0 conn(0x7f47980e4e40 0x7f47981b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 -- 172.21.2.117:0/2757598246 >> 172.21.2.117:0/2757598246 conn(0x7f47980ce4e0 msgr2=0x7f47980e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 -- 172.21.2.117:0/2757598246 shutdown_connections 2021-02-10T11:57:19.451 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.433 7f479da35700 1 -- 172.21.2.117:0/2757598246 wait complete. 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 Processor -- start 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 -- start start 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 0x7f479819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4798189d00 0x7f479818a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479818d4b0 con 0x7f4798189d00 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479818ac30 con 0x7f479818a720 2021-02-10T11:57:19.453 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f479da35700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479819a480 con 0x7f479818d0b0 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f4797fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f4797fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36586/0 (socket says 172.21.2.117:36586) 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f4797fff700 1 -- 172.21.2.117:0/1903685274 learned_addr learned my addr 172.21.2.117:0/1903685274 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f4796ffd700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4798189d00 0x7f479818a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.448 7f47977fe700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 0x7f479819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 -- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 msgr2=0x7f479819e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 0x7f479819e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 -- 172.21.2.117:0/1903685274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4798189d00 msgr2=0x7f479818a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4798189d00 0x7f479818a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 -- 172.21.2.117:0/1903685274 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479818b7d0 con 0x7f479818a720 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f47977fe700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 0x7f479819e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f4797fff700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f4790009980 tx=0x7f479000bd40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47900013b0 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790003030 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f47900017a0 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f479da35700 1 -- 172.21.2.117:0/1903685274 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479818b7d0 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.449 7f479da35700 1 -- 172.21.2.117:0/1903685274 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479818bd60 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.450 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4790001000 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.450 7f478ffff700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 0x7f477c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.450 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4790032a60 con 0x7f479818a720 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.450 7f47977fe700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 0x7f477c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:19.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.450 7f47977fe700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 0x7f477c01abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:19.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.481 7f479da35700 1 -- 172.21.2.117:0/1903685274 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479810c260 con 0x7f479818a720 2021-02-10T11:57:19.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.485 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4790000df0 con 0x7f479818a720 2021-02-10T11:57:19.618 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.616 7f479da35700 1 -- 172.21.2.117:0/1903685274 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4798085930 con 0x7f479818a720 2021-02-10T11:57:19.619 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.617 7f478ffff700 1 -- 172.21.2.117:0/1903685274 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+2108 (secure 0 0 0) 0x7f4790001000 con 0x7f479818a720 2021-02-10T11:57:19.621 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:19.621 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":161,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:11.517907","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8153":{"gid":8153,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/2333277299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2333277299},{"type":"v1","addr":"172.21.2.125:6835","nonce":2333277299}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:19.621 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 161 2021-02-10T11:57:19.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 -- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 msgr2=0x7f477c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.622 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 0x7f477c01abc0 crc :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 -- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 msgr2=0x7f479818b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f4790009980 tx=0x7f479000bd40).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 -- 172.21.2.117:0/1903685274 shutdown_connections 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f477c0186c0 0x7f477c01abc0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4798189d00 0x7f479818a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f479818a720 0x7f479818b1b0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 --2- 172.21.2.117:0/1903685274 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f479818d0b0 0x7f479819e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.620 7f479da35700 1 -- 172.21.2.117:0/1903685274 >> 172.21.2.117:0/1903685274 conn(0x7f47981a60a0 msgr2=0x7f479810c490 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.621 7f479da35700 1 -- 172.21.2.117:0/1903685274 shutdown_connections 2021-02-10T11:57:19.623 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:19.621 7f479da35700 1 -- 172.21.2.117:0/1903685274 wait complete. 2021-02-10T11:57:19.833 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 161, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:57:09.509260', 'modified': '2021-02-10 11:57:11.517907', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8153}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8153': {'gid': 8153, 'name': 'b', 'rank': 0, 'incarnation': 158, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/2333277299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 2333277299}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 2333277299}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:57:19.833 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:57:19.833 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:57:20.230 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:57:20.230 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:57:20.230 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:57:20.230 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8153, 2021-02-10T11:57:20.230 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.716806528999999, 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 161, 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 187, 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 187, 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.370511465 2021-02-10T11:57:20.231 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:57:20.234 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8153, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.716806528999999, "mdsmap_epoch": 161, "osdmap_epoch": 187, "osdmap_epoch_barrier": 187, "uptime": 16.370511465 } 2021-02-10T11:57:20.234 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:57:20.234 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:57:20.234 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:57:20.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:20.235 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:20.570 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:57:20.915 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:57:20.916 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:57:21.259 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:57:21.260 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:57:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:57:21.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:21.261 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:57:21.596 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:57:21.596 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:21.598 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:57:21.944 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:57:21.944 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:57:22.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[192810]: starting ceph client 2021-02-10T11:57:22.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:57:22.009 7fd6af74cf80 -1 init, newargv = 0x55968bb9fb10 newargc=9 2021-02-10T11:57:22.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[192810]: starting fuse 2021-02-10T11:57:22.287 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:57:22.287 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:57:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:57:22.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:22.288 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:57:22.623 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:57:22.624 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:57:22.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:57:22.624 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:57:22.625 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:57:22.987 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.192810.asok 2021-02-10T11:57:22.990 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.192810.asok 2021-02-10T11:57:22.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:22.990 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.192810.asok status 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "192810", 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:57:23.384 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 13590, 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 13590 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3994883957 2021-02-10T11:57:23.385 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 3994883957 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.13590 172.21.2.117:0/3994883957", 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/3994883957", 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 161, 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 187, 2021-02-10T11:57:23.386 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:57:23.387 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:57:23.387 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:57:23.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:23.388 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:23.388 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:23.725 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:57:23.725 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:57:23.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:23.725 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:24.072 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:57:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.482 7f5e4fb54700 1 Processor -- start 2021-02-10T11:57:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.482 7f5e4fb54700 1 -- start start 2021-02-10T11:57:24.501 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4fb54700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 0x7f5e480eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4fb54700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5e480eb7f0 0x7f5e481b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4fb54700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e480ede60 con 0x7f5e480d18e0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4fb54700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e480ef7f0 con 0x7f5e480eb7f0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4fb54700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480d1ce0 con 0x7f5e480ef3f0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4d970700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5e480d18e0 0x7f5e480ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60704/0 (socket says 172.21.2.117:60704) 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4d970700 1 -- 172.21.2.117:0/2171506574 learned_addr learned my addr 172.21.2.117:0/2171506574 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4d16f700 1 --2- 172.21.2.117:0/2171506574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 0x7f5e480eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888290211 0 0) 0x7f5e44001180 con 0x7f5e480d18e0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e34002fe0 con 0x7f5e480d18e0 2021-02-10T11:57:24.502 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.483 7f5e4e171700 1 --2- 172.21.2.117:0/2171506574 >> v2:172.21.2.125:3300/0 conn(0x7f5e480eb7f0 0x7f5e481b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4d16f700 1 -- 172.21.2.117:0/2171506574 >> v2:172.21.2.125:3300/0 conn(0x7f5e480eb7f0 msgr2=0x7f5e481b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3084491122 0 0) 0x7f5e44001180 con 0x7f5e480d18e0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4d16f700 1 --2- 172.21.2.117:0/2171506574 >> v2:172.21.2.125:3300/0 conn(0x7f5e480eb7f0 0x7f5e481b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4d16f700 1 -- 172.21.2.117:0/2171506574 >> v1:172.21.2.117:6789/0 conn(0x7f5e480d18e0 legacy=0x7f5e480ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4d16f700 1 -- 172.21.2.117:0/2171506574 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e481b8f90 con 0x7f5e480ef3f0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4d16f700 1 --2- 172.21.2.117:0/2171506574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 0x7f5e480eb1d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5e3c006650 tx=0x7f5e3c009a80).ready entity=mon.2 client_cookie=72920e695b99e4c3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3c00c0c0 con 0x7f5e480ef3f0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c00c340 con 0x7f5e480ef3f0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4c96e700 1 -- 172.21.2.117:0/2171506574 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e3c00c860 con 0x7f5e480ef3f0 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4fb54700 1 -- 172.21.2.117:0/2171506574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 msgr2=0x7f5e480eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.484 7f5e4fb54700 1 --2- 172.21.2.117:0/2171506574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 0x7f5e480eb1d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5e3c006650 tx=0x7f5e3c009a80).stop 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 -- 172.21.2.117:0/2171506574 shutdown_connections 2021-02-10T11:57:24.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 --2- 172.21.2.117:0/2171506574 >> v2:172.21.2.125:3300/0 conn(0x7f5e480eb7f0 0x7f5e481b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 --2- 172.21.2.117:0/2171506574 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e480ef3f0 0x7f5e480eb1d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 -- 172.21.2.117:0/2171506574 >> 172.21.2.117:0/2171506574 conn(0x7f5e480ced60 msgr2=0x7f5e480cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 -- 172.21.2.117:0/2171506574 shutdown_connections 2021-02-10T11:57:24.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.485 7f5e4fb54700 1 -- 172.21.2.117:0/2171506574 wait complete. 2021-02-10T11:57:24.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 Processor -- start 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 -- start start 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e481a33d0 0x7f5e481a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e480b82f0 0x7f5e480b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480c0670 con 0x7f5e480b8d10 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480deb20 con 0x7f5e480b82f0 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4fb54700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480ded40 con 0x7f5e481a33d0 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4e171700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4e171700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60462/0 (socket says 172.21.2.117:60462) 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.500 7f5e4e171700 1 -- 172.21.2.117:0/491960151 learned_addr learned my addr 172.21.2.117:0/491960151 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4d970700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e481a33d0 0x7f5e481a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:24.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 -- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e481a33d0 msgr2=0x7f5e481a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e481a33d0 0x7f5e481a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 -- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e480b82f0 msgr2=0x7f5e480b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e480b82f0 0x7f5e480b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 -- 172.21.2.117:0/491960151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480def60 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4e171700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f5e4000b970 tx=0x7f5e4000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e40000f90 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40006aa0 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5e40001680 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e480def60 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.501 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e480def60 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.502 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5e40000df0 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.502 7f5e3a7fc700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 0x7f5e3001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.502 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5e40032a50 con 0x7f5e480b8d10 2021-02-10T11:57:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.503 7f5e4d970700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 0x7f5e3001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.503 7f5e4d970700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 0x7f5e3001abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:24.534 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.532 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48087290 con 0x7f5e480b8d10 2021-02-10T11:57:24.537 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.535 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5e4001e0c0 con 0x7f5e480b8d10 2021-02-10T11:57:24.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.660 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e480ede60 con 0x7f5e480b8d10 2021-02-10T11:57:24.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.660 7f5e3a7fc700 1 -- 172.21.2.117:0/491960151 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v187) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f5e4001e0c0 con 0x7f5e480b8d10 2021-02-10T11:57:24.663 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:24.663 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":187,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:10.511294","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:57:07.475412","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":41,"pool_name":"cephfs_data","create_time":"2021-02-10 11:57:08.504122","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"187","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:57:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 msgr2=0x7f5e3001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 0x7f5e3001abc0 crc :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 msgr2=0x7f5e480b9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:24.664 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 rx=0x7f5e4000b970 tx=0x7f5e4000bd10).stop 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 shutdown_connections 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5e300186c0 0x7f5e3001abc0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5e480b8d10 0x7f5e480b9110 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5e480b82f0 0x7f5e480b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 --2- 172.21.2.117:0/491960151 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5e481a33d0 0x7f5e481a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 >> 172.21.2.117:0/491960151 conn(0x7f5e480874c0 msgr2=0x7f5e48096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 shutdown_connections 2021-02-10T11:57:24.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:24.662 7f5e4fb54700 1 -- 172.21.2.117:0/491960151 wait complete. 2021-02-10T11:57:24.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 40, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:57:07.475412', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 41, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:57:08.504122', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:57:24.673 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:57:25.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.087 7fc38b6e4700 1 Processor -- start 2021-02-10T11:57:25.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.087 7fc38b6e4700 1 -- start start 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc38b6e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 0x7fc3840eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc38b6e4700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3840ef7f0 con 0x7fc3840d18f0 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc38b6e4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3840d1cf0 con 0x7fc3840eb830 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc38b6e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3840eded0 con 0x7fc3840ef3f0 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc389500700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc3840d18f0 0x7fc3840eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60718/0 (socket says 172.21.2.117:60718) 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc389500700 1 -- 172.21.2.117:0/27931883 learned_addr learned my addr 172.21.2.117:0/27931883 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.088 7fc388cff700 1 --2- 172.21.2.117:0/27931883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 0x7fc3840eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc377fff700 1 -- 172.21.2.117:0/27931883 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249691949 0 0) 0x7fc380001180 con 0x7fc3840d18f0 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc377fff700 1 -- 172.21.2.117:0/27931883 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370002fe0 con 0x7fc3840d18f0 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc377fff700 1 -- 172.21.2.117:0/27931883 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672567225 0 0) 0x7fc37c001400 con 0x7fc3840eb830 2021-02-10T11:57:25.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc388cff700 1 -- 172.21.2.117:0/27931883 >> v1:172.21.2.125:6789/0 conn(0x7fc3840eb830 legacy=0x7fc3841b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc388cff700 1 -- 172.21.2.117:0/27931883 >> v1:172.21.2.117:6789/0 conn(0x7fc3840d18f0 legacy=0x7fc3840eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc388cff700 1 -- 172.21.2.117:0/27931883 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3841bb1c0 con 0x7fc3840ef3f0 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc388cff700 1 --2- 172.21.2.117:0/27931883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 0x7fc3840eb210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fc378006650 tx=0x7fc378009a80).ready entity=mon.2 client_cookie=e04f5fe075050ef8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.089 7fc377fff700 1 -- 172.21.2.117:0/27931883 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3780009b0 con 0x7fc3840ef3f0 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc377fff700 1 -- 172.21.2.117:0/27931883 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37800c0c0 con 0x7fc3840ef3f0 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc377fff700 1 -- 172.21.2.117:0/27931883 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37800c560 con 0x7fc3840ef3f0 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc38b6e4700 1 -- 172.21.2.117:0/27931883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 msgr2=0x7fc3840eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc38b6e4700 1 --2- 172.21.2.117:0/27931883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 0x7fc3840eb210 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fc378006650 tx=0x7fc378009a80).stop 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc38b6e4700 1 -- 172.21.2.117:0/27931883 shutdown_connections 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc38b6e4700 1 --2- 172.21.2.117:0/27931883 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc3840ef3f0 0x7fc3840eb210 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.090 7fc38b6e4700 1 -- 172.21.2.117:0/27931883 >> 172.21.2.117:0/27931883 conn(0x7fc3840cf060 msgr2=0x7fc3840cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:25.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.091 7fc38b6e4700 1 -- 172.21.2.117:0/27931883 shutdown_connections 2021-02-10T11:57:25.109 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.091 7fc38b6e4700 1 -- 172.21.2.117:0/27931883 wait complete. 2021-02-10T11:57:25.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 Processor -- start 2021-02-10T11:57:25.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 -- start start 2021-02-10T11:57:25.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.110 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38407f210 0x7fc38418f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38407f610 0x7fc38418ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3840b6670 con 0x7fc3841a0370 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3840b6890 con 0x7fc38407f210 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.105 7fc38b6e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc384190590 con 0x7fc38407f610 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60476/0 (socket says 172.21.2.117:60476) 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 -- 172.21.2.117:0/3502244725 learned_addr learned my addr 172.21.2.117:0/3502244725 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389d01700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38407f610 0x7fc38418ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc388cff700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38407f210 0x7fc38418f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 -- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38407f610 msgr2=0x7fc38418ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38407f610 0x7fc38418ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 -- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38407f210 msgr2=0x7fc38418f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.111 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38407f210 0x7fc38418f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 -- 172.21.2.117:0/3502244725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3841907b0 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc389500700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 rx=0x7fc380000b80 tx=0x7fc38000a980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.106 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc38000ad70 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3800070d0 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3841907b0 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc3800047a0 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc384191310 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc3800043a0 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc375ffb700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 0x7fc36401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.107 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc380032930 con 0x7fc3841a0370 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.108 7fc388cff700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 0x7fc36401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.112 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.108 7fc388cff700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 0x7fc36401ac10 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.138 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3840f67e0 con 0x7fc3841a0370 2021-02-10T11:57:25.142 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.140 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc380001490 con 0x7fc3841a0370 2021-02-10T11:57:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.266 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fc3840f67e0 con 0x7fc3841a0370 2021-02-10T11:57:25.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.266 7fc375ffb700 1 -- 172.21.2.117:0/3502244725 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v187) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc3800070b0 con 0x7fc3841a0370 2021-02-10T11:57:25.269 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:25.270 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 msgr2=0x7fc36401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 0x7fc36401ac10 crc :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 msgr2=0x7fc38407ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 rx=0x7fc380000b80 tx=0x7fc38000a980).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 shutdown_connections 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38407f610 0x7fc38418ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38407f210 0x7fc38418f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc364018730 0x7fc36401ac10 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.268 7fc38b6e4700 1 --2- 172.21.2.117:0/3502244725 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc3841a0370 0x7fc38407ebf0 unknown :-1 s=CLOSED pgs=3145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.269 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 >> 172.21.2.117:0/3502244725 conn(0x7fc384090920 msgr2=0x7fc3840a9730 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.269 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 shutdown_connections 2021-02-10T11:57:25.271 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.269 7fc38b6e4700 1 -- 172.21.2.117:0/3502244725 wait complete. 2021-02-10T11:57:25.279 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:57:25.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.693 7f0792ff5700 1 Processor -- start 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.693 7f0792ff5700 1 -- start start 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 0x7f078c0eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c0ede60 con 0x7f078c0ee3e0 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c0eba60 con 0x7f078c0eb040 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0792ff5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c0d1cf0 con 0x7f078c0d18f0 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0791612700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0790e11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f078bfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 0x7f078c0eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f078bfff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 0x7f078c0eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36634/0 (socket says 172.21.2.117:36634) 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0791612700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60482/0 (socket says 172.21.2.117:60482) 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f0790e11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58162/0 (socket says 172.21.2.117:58162) 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.694 7f078bfff700 1 -- 172.21.2.117:0/2859842092 learned_addr learned my addr 172.21.2.117:0/2859842092 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 -- 172.21.2.117:0/2859842092 >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 msgr2=0x7f078c0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 --2- 172.21.2.117:0/2859842092 >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 0x7f078c0eb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 -- 172.21.2.117:0/2859842092 >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 msgr2=0x7f078c0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 --2- 172.21.2.117:0/2859842092 >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 -- 172.21.2.117:0/2859842092 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c0eee00 con 0x7f078c0d18f0 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0791612700 1 --2- 172.21.2.117:0/2859842092 >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0790e11700 1 --2- 172.21.2.117:0/2859842092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0774006650 tx=0x7f0774009a80).ready entity=mon.2 client_cookie=e692e347d0c707d0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f078b7fe700 1 -- 172.21.2.117:0/2859842092 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f07740009b0 con 0x7f078c0d18f0 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f078b7fe700 1 -- 172.21.2.117:0/2859842092 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077400c0c0 con 0x7f078c0d18f0 2021-02-10T11:57:25.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f078b7fe700 1 -- 172.21.2.117:0/2859842092 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077400c5a0 con 0x7f078c0d18f0 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0792ff5700 1 -- 172.21.2.117:0/2859842092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 msgr2=0x7f078c0eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.695 7f0792ff5700 1 --2- 172.21.2.117:0/2859842092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0774006650 tx=0x7f0774009a80).stop 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 -- 172.21.2.117:0/2859842092 shutdown_connections 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 --2- 172.21.2.117:0/2859842092 >> v2:172.21.2.125:3300/0 conn(0x7f078c0eb040 0x7f078c0eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 --2- 172.21.2.117:0/2859842092 >> v2:172.21.2.117:3300/0 conn(0x7f078c0ee3e0 0x7f078c0ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 --2- 172.21.2.117:0/2859842092 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c0d18f0 0x7f078c0eaa20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 -- 172.21.2.117:0/2859842092 >> 172.21.2.117:0/2859842092 conn(0x7f078c0cf060 msgr2=0x7f078c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 -- 172.21.2.117:0/2859842092 shutdown_connections 2021-02-10T11:57:25.715 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.696 7f0792ff5700 1 -- 172.21.2.117:0/2859842092 wait complete. 2021-02-10T11:57:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.711 7f0792ff5700 1 Processor -- start 2021-02-10T11:57:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.711 7f0792ff5700 1 -- start start 2021-02-10T11:57:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.711 7f0792ff5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c08a220 0x7f078c07c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.718 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0792ff5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 0x7f078c09dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0792ff5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0792ff5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c09e2a0 con 0x7f078c08ace0 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0792ff5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c1a7e50 con 0x7f078c08a8e0 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0792ff5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c1a8070 con 0x7f078c08a220 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0791612700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0791612700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60490/0 (socket says 172.21.2.117:60490) 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0791612700 1 -- 172.21.2.117:0/2235357215 learned_addr learned my addr 172.21.2.117:0/2235357215 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 0x7f078c09dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0790e11700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c08a220 0x7f078c07c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 -- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c08a220 msgr2=0x7f078c07c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c08a220 0x7f078c07c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 -- 172.21.2.117:0/2235357215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 msgr2=0x7f078c1a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.719 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 -- 172.21.2.117:0/2235357215 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c07d3d0 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f0791612700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.712 7f078bfff700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 0x7f078c09dc80 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f077c006ed0 tx=0x7f077c00b840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077c0063c0 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c00bc30 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f077c0022b0 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c07d3d0 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c07d3d0 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f077c000c90 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 0x7f077001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.713 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f077c032810 con 0x7f078c08a8e0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.714 7f0790e11700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 0x7f077001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:25.720 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.714 7f0790e11700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 0x7f077001abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:25.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.744 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c07cea0 con 0x7f078c08a8e0 2021-02-10T11:57:25.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.749 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f077c006340 con 0x7f078c08a8e0 2021-02-10T11:57:25.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.877 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f078c07cea0 con 0x7f078c08a8e0 2021-02-10T11:57:25.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.878 7f07897fa700 1 -- 172.21.2.117:0/2235357215 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v187) v1 ==== 104+0+10 (secure 0 0 0) 0x7f077c000c90 con 0x7f078c08a8e0 2021-02-10T11:57:25.882 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:25.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.881 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 msgr2=0x7f077001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.881 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 0x7f077001abc0 crc :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.881 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 msgr2=0x7f078c09dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:25.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.881 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 0x7f078c09dc80 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f077c006ed0 tx=0x7f077c00b840).stop 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 shutdown_connections 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f07700186c0 0x7f077001abc0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f078c08ace0 0x7f078c1a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f078c08a8e0 0x7f078c09dc80 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 --2- 172.21.2.117:0/2235357215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f078c08a220 0x7f078c07c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 >> 172.21.2.117:0/2235357215 conn(0x7f078c0a71e0 msgr2=0x7f078c0a64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 shutdown_connections 2021-02-10T11:57:25.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:25.882 7f0792ff5700 1 -- 172.21.2.117:0/2235357215 wait complete. 2021-02-10T11:57:25.892 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:57:26.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.307 7f58d23c9700 1 Processor -- start 2021-02-10T11:57:26.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.307 7f58d23c9700 1 -- start start 2021-02-10T11:57:26.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d23c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0d18f0 0x7f58cc0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d23c9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0ede60 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.327 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d23c9700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0d1cf0 con 0x7f58cc0eb040 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d23c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc088020 con 0x7f58cc0d18f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d09e6700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f58cc0ef3f0 0x7f58cc1b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60744/0 (socket says 172.21.2.117:60744) 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58d09e6700 1 -- 172.21.2.117:0/516514323 learned_addr learned my addr 172.21.2.117:0/516514323 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902422788 0 0) 0x7f58c4001500 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58caffd700 1 -- 172.21.2.117:0/516514323 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4002fe0 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.308 7f58cbfff700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0d18f0 0x7f58cc0eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3042329388 0 0) 0x7f58c4001500 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0d18f0 msgr2=0x7f58cc0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0d18f0 0x7f58cc0eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 >> v1:172.21.2.125:6789/0 conn(0x7f58cc0eb040 legacy=0x7f58cc0eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc1bb200 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f58c4001a00 con 0x7f58cc0ef3f0 2021-02-10T11:57:26.328 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 >> v1:172.21.2.117:6789/0 conn(0x7f58cc0ef3f0 legacy=0x7f58cc1b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58b4003a60 0x7f58b4005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58b4006770 0x7f58b4008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58b4009500 0x7f58b400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400c2d0 con 0x7f58b4009500 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400c4f0 con 0x7f58b4003a60 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58caffd700 1 -- 172.21.2.117:0/516514323 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400c710 con 0x7f58b4006770 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58cb7fe700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58b4006770 0x7f58b4008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58d09e6700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58b4009500 0x7f58b400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.309 7f58cbfff700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58b4003a60 0x7f58b4005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 -- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58b4006770 msgr2=0x7f58b4008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58b4006770 0x7f58b4008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 -- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58b4003a60 msgr2=0x7f58b4005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:26.329 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58b4003a60 0x7f58b4005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 -- 172.21.2.117:0/516514323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4050980 con 0x7f58b4009500 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58b4009500 0x7f58b400ba20 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f58c4001860 tx=0x7f58c400a310).ready entity=mon.0 client_cookie=85202ecdf3469b2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c40070c0 con 0x7f58b4009500 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c40073c0 con 0x7f58b4009500 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58caffd700 1 -- 172.21.2.117:0/516514323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c4004910 con 0x7f58b4009500 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d23c9700 1 -- 172.21.2.117:0/516514323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58b4009500 msgr2=0x7f58b400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d23c9700 1 --2- 172.21.2.117:0/516514323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58b4009500 0x7f58b400ba20 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 rx=0x7f58c4001860 tx=0x7f58c400a310).stop 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.310 7f58d09e6700 1 -- 172.21.2.117:0/516514323 reap_dead start 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.311 7f58d23c9700 1 -- 172.21.2.117:0/516514323 shutdown_connections 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.311 7f58d23c9700 1 -- 172.21.2.117:0/516514323 >> 172.21.2.117:0/516514323 conn(0x7f58cc0cf060 msgr2=0x7f58cc0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.311 7f58d23c9700 1 -- 172.21.2.117:0/516514323 shutdown_connections 2021-02-10T11:57:26.330 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.311 7f58d23c9700 1 -- 172.21.2.117:0/516514323 wait complete. 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.325 7f58d23c9700 1 Processor -- start 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 -- start start 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 0x7f58cc0dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58cc085ec0 0x7f58cc0862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0a67e0 con 0x7f58cc0a2040 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0a7120 con 0x7f58cc085ec0 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d23c9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc0a2a60 con 0x7f58cc0a6ce0 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d09e6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d09e6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60508/0 (socket says 172.21.2.117:60508) 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58d09e6700 1 -- 172.21.2.117:0/4064566451 learned_addr learned my addr 172.21.2.117:0/4064566451 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.326 7f58cbfff700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 0x7f58cc0dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 -- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 msgr2=0x7f58cc0dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 0x7f58cc0dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 -- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58cc085ec0 msgr2=0x7f58cc0862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58cc085ec0 0x7f58cc0862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 -- 172.21.2.117:0/4064566451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc1a2dc0 con 0x7f58cc0a2040 2021-02-10T11:57:26.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58cbfff700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 0x7f58cc0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d09e6700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 rx=0x7f58c400a940 tx=0x7f58c400ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c4004930 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c4004b70 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc1a2dc0 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc1a3350 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.327 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f58c4007020 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.328 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f58c4004720 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.328 7f58c8ff9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 0x7f58ac01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.328 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f58c4032c90 con 0x7f58cc0a2040 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.328 7f58cbfff700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 0x7f58ac01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.328 7f58cbfff700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 0x7f58ac01abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.359 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.358 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4002fd0 con 0x7f58cc0a2040 2021-02-10T11:57:26.362 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.360 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f58c4004b50 con 0x7f58cc0a2040 2021-02-10T11:57:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.487 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f58a4002ef0 con 0x7f58cc0a2040 2021-02-10T11:57:26.489 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.487 7f58c8ff9700 1 -- 172.21.2.117:0/4064566451 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v187) v1 ==== 100+0+10 (secure 0 0 0) 0x7f58c4004b50 con 0x7f58cc0a2040 2021-02-10T11:57:26.490 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 msgr2=0x7f58ac01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 0x7f58ac01abc0 crc :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 msgr2=0x7f58cc0a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 secure :-1 s=READY pgs=3148 cs=0 l=1 rev1=1 rx=0x7f58c400a940 tx=0x7f58c400ace0).stop 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 shutdown_connections 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f58ac0186c0 0x7f58ac01abc0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f58cc0a2040 0x7f58cc0a2440 unknown :-1 s=CLOSED pgs=3148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f58cc085ec0 0x7f58cc0862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 --2- 172.21.2.117:0/4064566451 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f58cc0a6ce0 0x7f58cc0dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 >> 172.21.2.117:0/4064566451 conn(0x7f58cc0b6540 msgr2=0x7f58cc08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 shutdown_connections 2021-02-10T11:57:26.492 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.489 7f58d23c9700 1 -- 172.21.2.117:0/4064566451 wait complete. 2021-02-10T11:57:26.500 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:57:26.500 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.911 7fb3b8f4b700 1 Processor -- start 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.911 7fb3b8f4b700 1 -- start start 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b8f4b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb3b41b6960 0x7fb3b41b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b8f4b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40ede60 con 0x7fb3b40ef3f0 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b8f4b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40eb9c0 con 0x7fb3b40d18e0 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b8f4b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b40ef7f0 con 0x7fb3b41b6960 2021-02-10T11:57:26.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b2312700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb3b40ef3f0 0x7fb3b40eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60762/0 (socket says 172.21.2.117:60762) 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b2312700 1 -- 172.21.2.117:0/1364960421 learned_addr learned my addr 172.21.2.117:0/1364960421 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.912 7fb3b3314700 1 --2- 172.21.2.117:0/1364960421 >> v2:172.21.2.125:3300/0 conn(0x7fb3b41b6960 0x7fb3b41b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b1b11700 1 -- 172.21.2.117:0/1364960421 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622273075 0 0) 0x7fb3a4000ce0 con 0x7fb3b40ef3f0 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b1b11700 1 -- 172.21.2.117:0/1364960421 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a0002fe0 con 0x7fb3b40ef3f0 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b3314700 1 -- 172.21.2.117:0/1364960421 >> v1:172.21.2.125:6789/0 conn(0x7fb3b40d18e0 legacy=0x7fb3b40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b3314700 1 -- 172.21.2.117:0/1364960421 >> v1:172.21.2.117:6789/0 conn(0x7fb3b40ef3f0 legacy=0x7fb3b40eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b3314700 1 -- 172.21.2.117:0/1364960421 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40d1d20 con 0x7fb3b41b6960 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b3314700 1 --2- 172.21.2.117:0/1364960421 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b41b6960 0x7fb3b41b8d70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7fb3ac006e50 tx=0x7fb3ac009e70).ready entity=mon.1 client_cookie=9dbd54dd7b2a4ddf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b1b11700 1 -- 172.21.2.117:0/1364960421 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac00b510 con 0x7fb3b41b6960 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b1b11700 1 -- 172.21.2.117:0/1364960421 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac006790 con 0x7fb3b41b6960 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b1b11700 1 -- 172.21.2.117:0/1364960421 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac00b940 con 0x7fb3b41b6960 2021-02-10T11:57:26.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b8f4b700 1 -- 172.21.2.117:0/1364960421 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b41b6960 msgr2=0x7fb3b41b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.913 7fb3b8f4b700 1 --2- 172.21.2.117:0/1364960421 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b41b6960 0x7fb3b41b8d70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7fb3ac006e50 tx=0x7fb3ac009e70).stop 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.914 7fb3b8f4b700 1 -- 172.21.2.117:0/1364960421 shutdown_connections 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.914 7fb3b8f4b700 1 --2- 172.21.2.117:0/1364960421 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b41b6960 0x7fb3b41b8d70 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.914 7fb3b8f4b700 1 -- 172.21.2.117:0/1364960421 >> 172.21.2.117:0/1364960421 conn(0x7fb3b40ced60 msgr2=0x7fb3b40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.914 7fb3b8f4b700 1 -- 172.21.2.117:0/1364960421 shutdown_connections 2021-02-10T11:57:26.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.914 7fb3b8f4b700 1 -- 172.21.2.117:0/1364960421 wait complete. 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.928 7fb3b8f4b700 1 Processor -- start 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.928 7fb3b8f4b700 1 -- start start 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 0x7fb3b40e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 0x7fb3b40e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4196810 con 0x7fb3b40e9960 2021-02-10T11:57:26.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4098ad0 con 0x7fb3b40e9560 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b8f4b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b40e8a40 con 0x7fb3b4196390 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b2b13700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 0x7fb3b40e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60520/0 (socket says 172.21.2.117:60520) 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 -- 172.21.2.117:0/294462915 learned_addr learned my addr 172.21.2.117:0/294462915 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b2312700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 0x7fb3b40e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 -- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 msgr2=0x7fb3b40e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 0x7fb3b40e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 -- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 msgr2=0x7fb3b40e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 0x7fb3b40e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b3314700 1 -- 172.21.2.117:0/294462915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40e8c60 con 0x7fb3b40e9960 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b2312700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 0x7fb3b40e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:26.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.929 7fb3b2b13700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 0x7fb3b40e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3b3314700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7fb3ac006c50 tx=0x7fb3ac00bd20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac0093d0 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac00be60 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb3ac00c570 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b40e8c60 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.930 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b419cd70 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.931 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb3ac0093d0 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.931 7fb3ab7fe700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.931 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb3ac0328e0 con 0x7fb3b40e9960 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.931 7fb3b2b13700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:26.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.931 7fb3b2b13700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 0x7fb39801abc0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:26.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.961 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b4087210 con 0x7fb3b40e9960 2021-02-10T11:57:26.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:26.964 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb3ac002ef0 con 0x7fb3b40e9960 2021-02-10T11:57:27.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.091 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3b40ede60 con 0x7fb3b40e9960 2021-02-10T11:57:27.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.091 7fb3ab7fe700 1 -- 172.21.2.117:0/294462915 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fb3ac0093d0 con 0x7fb3b40e9960 2021-02-10T11:57:27.094 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:27.094 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":161,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:11.517907","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8153":{"gid":8153,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/2333277299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2333277299},{"type":"v1","addr":"172.21.2.125:6835","nonce":2333277299}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:27.094 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 161 2021-02-10T11:57:27.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 msgr2=0x7fb39801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 0x7fb39801abc0 crc :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 msgr2=0x7fb3b40e8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 rx=0x7fb3ac006c50 tx=0x7fb3ac00bd20).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 shutdown_connections 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb3980186c0 0x7fb39801abc0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb3b40e9960 0x7fb3b40e8420 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb3b40e9560 0x7fb3b40e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 --2- 172.21.2.117:0/294462915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb3b4196390 0x7fb3b40e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 >> 172.21.2.117:0/294462915 conn(0x7fb3b419f610 msgr2=0x7fb3b41a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 shutdown_connections 2021-02-10T11:57:27.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:27.093 7fb3b8f4b700 1 -- 172.21.2.117:0/294462915 wait complete. 2021-02-10T11:57:27.305 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 161, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:57:09.509260', 'modified': '2021-02-10 11:57:11.517907', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8153}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8153': {'gid': 8153, 'name': 'b', 'rank': 0, 'incarnation': 158, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/2333277299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 2333277299}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 2333277299}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:57:27.305 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:57:27.306 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 8153, 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.190795147999999, 2021-02-10T11:57:27.704 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 161, 2021-02-10T11:57:27.705 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 187, 2021-02-10T11:57:27.705 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 187, 2021-02-10T11:57:27.705 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 23.844498577 2021-02-10T11:57:27.705 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:57:27.708 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 8153, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.190795147999999, "mdsmap_epoch": 161, "osdmap_epoch": 187, "osdmap_epoch_barrier": 187, "uptime": 23.844498577 } 2021-02-10T11:57:27.708 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:57:27.708 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:57:27.708 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:57:27.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:27.708 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:28.045 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:57:28.391 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:57:28.391 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:57:28.736 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:57:28.736 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:57:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:57:28.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:28.737 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:57:29.073 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:57:29.073 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:57:29.074 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:29.076 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:57:29.420 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:57:29.421 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:57:29.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[193450]: starting ceph client 2021-02-10T11:57:29.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:57:29.485 7f3ae89cef80 -1 init, newargv = 0x55cfd8f15f70 newargc=9 2021-02-10T11:57:29.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[193450]: starting fuse 2021-02-10T11:57:29.764 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:57:29.765 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:57:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:57:29.766 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:29.766 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:57:30.102 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:57:30.102 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:57:30.103 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:57:30.103 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:57:30.103 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:57:30.104 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:57:30.105 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:57:30.105 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:57:30.468 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.193450.asok 2021-02-10T11:57:30.470 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.193450.asok 2021-02-10T11:57:30.471 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:30.471 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.193450.asok status 2021-02-10T11:57:30.862 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:57:30.862 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:57:30.862 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "193450", 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 13626, 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:57:30.863 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 13626 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4281451823 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:57:30.864 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 4281451823 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.13626 172.21.2.117:0/4281451823", 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/4281451823", 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 161, 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 187, 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:57:30.865 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:57:30.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:30.867 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:30.867 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:31.203 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:57:31.203 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:57:31.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:31.203 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:31.551 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:57:31.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 Processor -- start 2021-02-10T11:57:31.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 -- start start 2021-02-10T11:57:31.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b381b07e0 0x7f3b380e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.984 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b380e6160 0x7f3b380e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b380e6b80 0x7f3b380e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b380e5cd0 con 0x7f3b380e6b80 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b380e75a0 con 0x7f3b381b07e0 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.966 7f3b3dd81700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b380e77e0 con 0x7f3b380e6160 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b380e6b80 0x7f3b380e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3b380e6b80 0x7f3b380e6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60538/0 (socket says 172.21.2.117:60538) 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 -- 172.21.2.117:0/2017538812 learned_addr learned my addr 172.21.2.117:0/2017538812 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 -- 172.21.2.117:0/2017538812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b380e6160 msgr2=0x7f3b380e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 --2- 172.21.2.117:0/2017538812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b380e6160 0x7f3b380e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 -- 172.21.2.117:0/2017538812 >> v2:172.21.2.125:3300/0 conn(0x7f3b381b07e0 msgr2=0x7f3b380e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 --2- 172.21.2.117:0/2017538812 >> v2:172.21.2.125:3300/0 conn(0x7f3b381b07e0 0x7f3b380e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 -- 172.21.2.117:0/2017538812 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b380e54a0 con 0x7f3b380e6b80 2021-02-10T11:57:31.985 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b37fff700 1 --2- 172.21.2.117:0/2017538812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b380e6b80 0x7f3b380e6f80 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 rx=0x7f3b30006f00 tx=0x7f3b30009f10).ready entity=mon.0 client_cookie=9bcce6c5635430f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b367fc700 1 -- 172.21.2.117:0/2017538812 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b3000b3b0 con 0x7f3b380e6b80 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b367fc700 1 -- 172.21.2.117:0/2017538812 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30000d30 con 0x7f3b380e6b80 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b367fc700 1 -- 172.21.2.117:0/2017538812 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b3000b820 con 0x7f3b380e6b80 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b3dd81700 1 -- 172.21.2.117:0/2017538812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b380e6b80 msgr2=0x7f3b380e6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.967 7f3b3dd81700 1 --2- 172.21.2.117:0/2017538812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b380e6b80 0x7f3b380e6f80 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 rx=0x7f3b30006f00 tx=0x7f3b30009f10).stop 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 -- 172.21.2.117:0/2017538812 shutdown_connections 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 --2- 172.21.2.117:0/2017538812 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b380e6b80 0x7f3b380e6f80 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 --2- 172.21.2.117:0/2017538812 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b380e6160 0x7f3b380e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 --2- 172.21.2.117:0/2017538812 >> v2:172.21.2.125:3300/0 conn(0x7f3b381b07e0 0x7f3b380e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 -- 172.21.2.117:0/2017538812 >> 172.21.2.117:0/2017538812 conn(0x7f3b380ce4f0 msgr2=0x7f3b380e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 -- 172.21.2.117:0/2017538812 shutdown_connections 2021-02-10T11:57:31.986 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.968 7f3b3dd81700 1 -- 172.21.2.117:0/2017538812 wait complete. 2021-02-10T11:57:31.987 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.982 7f3b3dd81700 1 Processor -- start 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 -- start start 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 0x7f3b380ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b3810d060 0x7f3b380ad660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381ab220 con 0x7f3b381064f0 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381060c0 con 0x7f3b381068f0 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b3dd81700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b381062e0 con 0x7f3b3810d060 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:31.988 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b377fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60542/0 (socket says 172.21.2.117:60542) 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b377fe700 1 -- 172.21.2.117:0/3325620320 learned_addr learned my addr 172.21.2.117:0/3325620320 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.983 7f3b36ffd700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 0x7f3b380ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 -- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b3810d060 msgr2=0x7f3b380ad660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b3810d060 0x7f3b380ad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 -- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 msgr2=0x7f3b380ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 0x7f3b380ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 -- 172.21.2.117:0/3325620320 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b380adc80 con 0x7f3b381064f0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b36ffd700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 0x7f3b380ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b377fe700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7f3b28006f00 tx=0x7f3b2800b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b28006350 con 0x7f3b381064f0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2800be00 con 0x7f3b381064f0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b380adc80 con 0x7f3b381064f0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3810dec0 con 0x7f3b381064f0 2021-02-10T11:57:31.989 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.984 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b28002020 con 0x7f3b381064f0 2021-02-10T11:57:31.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.985 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b28005310 con 0x7f3b381064f0 2021-02-10T11:57:31.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.985 7f3b2ffff700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 0x7f3b1c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:31.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.985 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b280326b0 con 0x7f3b381064f0 2021-02-10T11:57:31.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.985 7f3b36ffd700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 0x7f3b1c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:31.990 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:31.985 7f3b36ffd700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 0x7f3b1c01abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:32.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.016 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3810d6a0 con 0x7f3b381064f0 2021-02-10T11:57:32.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.019 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b28000960 con 0x7f3b381064f0 2021-02-10T11:57:32.155 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.154 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3b3810d6a0 con 0x7f3b381064f0 2021-02-10T11:57:32.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.698 7f3b2ffff700 1 -- 172.21.2.117:0/3325620320 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v162) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3b28005310 con 0x7f3b381064f0 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 msgr2=0x7f3b1c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 0x7f3b1c01abc0 crc :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 msgr2=0x7f3b3810ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 rx=0x7f3b28006f00 tx=0x7f3b2800b940).stop 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 shutdown_connections 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b3810d060 0x7f3b380ad660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b381068f0 0x7f3b380ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b1c0186c0 0x7f3b1c01abc0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:32.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 --2- 172.21.2.117:0/3325620320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b381064f0 0x7f3b3810ca40 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:32.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 >> 172.21.2.117:0/3325620320 conn(0x7f3b381998a0 msgr2=0x7f3b381a54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:32.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 shutdown_connections 2021-02-10T11:57:32.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:32.724 7f3b3dd81700 1 -- 172.21.2.117:0/3325620320 wait complete. 2021-02-10T11:57:32.734 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:57:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 Processor -- start 2021-02-10T11:57:33.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 -- start start 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff91c0d15e0 0x7ff91c0e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0e7580 con 0x7ff91c0e7180 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0e77f0 con 0x7ff91c1b07f0 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff923414700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91c0e5ce0 con 0x7ff91c0d15e0 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff920a2f700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff91c0e7180 0x7ff91c0e7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60798/0 (socket says 172.21.2.117:60798) 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.145 7ff920a2f700 1 -- 172.21.2.117:0/3124442420 learned_addr learned my addr 172.21.2.117:0/3124442420 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292083713 0 0) 0x7ff918001180 con 0x7ff91c0e7180 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff908002fe0 con 0x7ff91c0e7180 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff921a31700 1 --2- 172.21.2.117:0/3124442420 >> v2:172.21.2.125:3300/0 conn(0x7ff91c0d15e0 0x7ff91c0e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1898631456 0 0) 0x7ff918001180 con 0x7ff91c0e7180 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 >> v2:172.21.2.125:3300/0 conn(0x7ff91c0d15e0 msgr2=0x7ff91c0e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:33.165 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 --2- 172.21.2.117:0/3124442420 >> v2:172.21.2.125:3300/0 conn(0x7ff91c0d15e0 0x7ff91c0e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 >> v1:172.21.2.125:6789/0 conn(0x7ff91c1b07f0 legacy=0x7ff91c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0d19e0 con 0x7ff91c0e7180 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7ff918001680 con 0x7ff91c0e7180 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 >> v1:172.21.2.117:6789/0 conn(0x7ff91c0e7180 legacy=0x7ff91c0e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 0x7ff908006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9080068a0 0x7ff908008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff908009640 0x7ff90800bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90800c3b0 con 0x7ff908003b20 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90800c5d0 con 0x7ff908009640 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90800c7f0 con 0x7ff9080068a0 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff921a31700 1 --2- 172.21.2.117:0/3124442420 >> v2:172.21.2.125:3300/0 conn(0x7ff91c0d15e0 0x7ff91c0e8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.146 7ff921230700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 0x7ff908006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.166 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff920a2f700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9080068a0 0x7ff908008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921a31700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff908009640 0x7ff90800bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 -- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9080068a0 msgr2=0x7ff908008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff9080068a0 0x7ff908008dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 -- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff908009640 msgr2=0x7ff90800bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff908009640 0x7ff90800bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921a31700 1 -- 172.21.2.117:0/3124442420 reap_dead start 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 -- 172.21.2.117:0/3124442420 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90800ca90 con 0x7ff908003b20 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff921230700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 0x7ff908006000 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7ff910006ab0 tx=0x7ff91004d9f0).ready entity=mon.0 client_cookie=2274fa0d51c59566 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9100500c0 con 0x7ff908003b20 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9100066f0 con 0x7ff908003b20 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff90ffff700 1 -- 172.21.2.117:0/3124442420 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9100504f0 con 0x7ff908003b20 2021-02-10T11:57:33.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff923414700 1 -- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 msgr2=0x7ff908006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.147 7ff923414700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 0x7ff908006000 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 rx=0x7ff910006ab0 tx=0x7ff91004d9f0).stop 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.148 7ff923414700 1 -- 172.21.2.117:0/3124442420 shutdown_connections 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.148 7ff923414700 1 --2- 172.21.2.117:0/3124442420 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff908003b20 0x7ff908006000 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.148 7ff923414700 1 -- 172.21.2.117:0/3124442420 >> 172.21.2.117:0/3124442420 conn(0x7ff91c0ce500 msgr2=0x7ff91c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.148 7ff923414700 1 -- 172.21.2.117:0/3124442420 shutdown_connections 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.148 7ff923414700 1 -- 172.21.2.117:0/3124442420 wait complete. 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.162 7ff923414700 1 Processor -- start 2021-02-10T11:57:33.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.162 7ff923414700 1 -- start start 2021-02-10T11:57:33.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff91c10fd90 0x7ff91c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff91c0bce00 0x7ff91c0bd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0bd950 con 0x7ff91c091d90 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0bded0 con 0x7ff91c0bce00 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff923414700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0b21e0 con 0x7ff91c10fd90 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60560/0 (socket says 172.21.2.117:60560) 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 -- 172.21.2.117:0/235899997 learned_addr learned my addr 172.21.2.117:0/235899997 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 -- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff91c10fd90 msgr2=0x7ff91c110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff91c10fd90 0x7ff91c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 -- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff91c0bce00 msgr2=0x7ff91c0bd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.170 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff91c0bce00 0x7ff91c0bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.163 7ff921a31700 1 -- 172.21.2.117:0/235899997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0927b0 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff921a31700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 rx=0x7ff9140099a0 tx=0x7ff91400bc40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff914006030 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140062b0 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff9140056b0 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff923414700 1 -- 172.21.2.117:0/235899997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c0927b0 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff923414700 1 -- 172.21.2.117:0/235899997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c0927b0 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff914006910 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 0x7ff8fc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.164 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff914032c60 con 0x7ff91c091d90 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.165 7ff921230700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 0x7ff8fc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.171 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.165 7ff921230700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 0x7ff8fc01abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.195 7ff923414700 1 -- 172.21.2.117:0/235899997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c0bde90 con 0x7ff91c091d90 2021-02-10T11:57:33.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.198 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff91400bd80 con 0x7ff91c091d90 2021-02-10T11:57:33.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.331 7ff923414700 1 -- 172.21.2.117:0/235899997 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7ff91c0bde90 con 0x7ff8fc0186c0 2021-02-10T11:57:33.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.332 7ff90dffb700 1 -- 172.21.2.117:0/235899997 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7ff910006640 con 0x7ff8fc0186c0 2021-02-10T11:57:33.334 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.333 7ff923414700 1 -- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 msgr2=0x7ff8fc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.333 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 0x7ff8fc01abc0 crc :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.333 7ff923414700 1 -- 172.21.2.117:0/235899997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 msgr2=0x7ff91c092190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.333 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 rx=0x7ff9140099a0 tx=0x7ff91400bc40).stop 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 -- 172.21.2.117:0/235899997 shutdown_connections 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff8fc0186c0 0x7ff8fc01abc0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff91c091d90 0x7ff91c092190 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff91c0bce00 0x7ff91c0bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 --2- 172.21.2.117:0/235899997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff91c10fd90 0x7ff91c110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 -- 172.21.2.117:0/235899997 >> 172.21.2.117:0/235899997 conn(0x7ff91c08a780 msgr2=0x7ff91c0c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:33.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 -- 172.21.2.117:0/235899997 shutdown_connections 2021-02-10T11:57:33.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.334 7ff923414700 1 -- 172.21.2.117:0/235899997 wait complete. 2021-02-10T11:57:33.344 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:57:33.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.760 7f9befe03700 1 Processor -- start 2021-02-10T11:57:33.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.760 7f9befe03700 1 -- start start 2021-02-10T11:57:33.778 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9be80d18f0 0x7f9be80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80eb040 0x7f9be80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9be80ee3e0 0x7f9be80ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80edee0 con 0x7f9be80ee3e0 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be80eba60 con 0x7f9be80d18f0 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9befe03700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80d1cf0 con 0x7f9be80eb040 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9be80ee3e0 0x7f9be80ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9be80ee3e0 0x7f9be80ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60566/0 (socket says 172.21.2.117:60566) 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 -- 172.21.2.117:0/2307305667 learned_addr learned my addr 172.21.2.117:0/2307305667 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 -- 172.21.2.117:0/2307305667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80eb040 msgr2=0x7f9be80eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 --2- 172.21.2.117:0/2307305667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80eb040 0x7f9be80eb440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 -- 172.21.2.117:0/2307305667 >> v2:172.21.2.125:3300/0 conn(0x7f9be80d18f0 msgr2=0x7f9be80eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:33.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 --2- 172.21.2.117:0/2307305667 >> v2:172.21.2.125:3300/0 conn(0x7f9be80d18f0 0x7f9be80eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 -- 172.21.2.117:0/2307305667 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be80eee00 con 0x7f9be80ee3e0 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9bee420700 1 --2- 172.21.2.117:0/2307305667 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be80ee3e0 0x7f9be80ee7e0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7f9be0006f00 tx=0x7f9be0009f10).ready entity=mon.0 client_cookie=a4ccf9fd872fb203 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.761 7f9becc1d700 1 -- 172.21.2.117:0/2307305667 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be000b3b0 con 0x7f9be80ee3e0 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9becc1d700 1 -- 172.21.2.117:0/2307305667 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000b730 con 0x7f9be80ee3e0 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9becc1d700 1 -- 172.21.2.117:0/2307305667 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be000bc10 con 0x7f9be80ee3e0 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 -- 172.21.2.117:0/2307305667 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be80ee3e0 msgr2=0x7f9be80ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 --2- 172.21.2.117:0/2307305667 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be80ee3e0 0x7f9be80ee7e0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 rx=0x7f9be0006f00 tx=0x7f9be0009f10).stop 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 -- 172.21.2.117:0/2307305667 shutdown_connections 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 --2- 172.21.2.117:0/2307305667 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be80ee3e0 0x7f9be80ee7e0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 --2- 172.21.2.117:0/2307305667 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80eb040 0x7f9be80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 --2- 172.21.2.117:0/2307305667 >> v2:172.21.2.125:3300/0 conn(0x7f9be80d18f0 0x7f9be80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 -- 172.21.2.117:0/2307305667 >> 172.21.2.117:0/2307305667 conn(0x7f9be80cf060 msgr2=0x7f9be80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:33.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 -- 172.21.2.117:0/2307305667 shutdown_connections 2021-02-10T11:57:33.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.762 7f9befe03700 1 -- 172.21.2.117:0/2307305667 wait complete. 2021-02-10T11:57:33.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 Processor -- start 2021-02-10T11:57:33.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 -- start start 2021-02-10T11:57:33.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be81a4860 0x7f9be80a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80a65a0 0x7f9be80a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80a1820 con 0x7f9be809e160 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be80a69a0 con 0x7f9be81a4860 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9befe03700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be809dc80 con 0x7f9be80a65a0 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9bedc1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9bedc1f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60570/0 (socket says 172.21.2.117:60570) 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.777 7f9bedc1f700 1 -- 172.21.2.117:0/971500298 learned_addr learned my addr 172.21.2.117:0/971500298 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 -- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80a65a0 msgr2=0x7f9be80a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80a65a0 0x7f9be80a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 -- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be81a4860 msgr2=0x7f9be80a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be81a4860 0x7f9be80a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 -- 172.21.2.117:0/971500298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be80a23c0 con 0x7f9be809e160 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bedc1f700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7f9be400b5c0 tx=0x7f9be400b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be4000c40 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be400bd50 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9be40054d0 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9befe03700 1 -- 172.21.2.117:0/971500298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be80a23c0 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.778 7f9befe03700 1 -- 172.21.2.117:0/971500298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be80a23c0 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.779 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9be4006030 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.779 7f9bde7fc700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 0x7f9bd001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.779 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9be40326a0 con 0x7f9be809e160 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.779 7f9bed41e700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 0x7f9bd001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:33.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.780 7f9bed41e700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 0x7f9bd001abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:33.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.811 7f9befe03700 1 -- 172.21.2.117:0/971500298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be80ede60 con 0x7f9be809e160 2021-02-10T11:57:33.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.813 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9be4024020 con 0x7f9be809e160 2021-02-10T11:57:33.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.962 7f9befe03700 1 -- 172.21.2.117:0/971500298 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9be80ede60 con 0x7f9be809e160 2021-02-10T11:57:33.964 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.962 7f9bde7fc700 1 -- 172.21.2.117:0/971500298 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9be4024020 con 0x7f9be809e160 2021-02-10T11:57:33.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 msgr2=0x7f9bd001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 0x7f9bd001abc0 crc :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 msgr2=0x7f9be81a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:33.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 rx=0x7f9be400b5c0 tx=0x7f9be400b960).stop 2021-02-10T11:57:33.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 shutdown_connections 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9be80a65a0 0x7f9be80a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9be81a4860 0x7f9be80a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9bd00186c0 0x7f9bd001abc0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 --2- 172.21.2.117:0/971500298 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9be809e160 0x7f9be81a4240 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 >> 172.21.2.117:0/971500298 conn(0x7f9be809e660 msgr2=0x7f9be80bb930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 shutdown_connections 2021-02-10T11:57:33.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:33.964 7f9befe03700 1 -- 172.21.2.117:0/971500298 wait complete. 2021-02-10T11:57:33.974 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup create cephfs group_0000000000305114 2021-02-10T11:57:34.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 Processor -- start 2021-02-10T11:57:34.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 -- start start 2021-02-10T11:57:34.407 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 0x7fe9700eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700ede60 con 0x7fe9700ef3f0 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700d1cf0 con 0x7fe9700eb040 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9773c7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe970088020 con 0x7fe9700d18f0 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.389 7fe9759e4700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe9700ef3f0 0x7fe9701b79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60828/0 (socket says 172.21.2.117:60828) 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9759e4700 1 -- 172.21.2.117:0/3054722566 learned_addr learned my addr 172.21.2.117:0/3054722566 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9751e3700 1 --2- 172.21.2.117:0/3054722566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 0x7fe9700eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe963fff700 1 -- 172.21.2.117:0/3054722566 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878883467 0 0) 0x7fe968001500 con 0x7fe9700ef3f0 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe963fff700 1 -- 172.21.2.117:0/3054722566 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe95c002fe0 con 0x7fe9700ef3f0 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9751e3700 1 -- 172.21.2.117:0/3054722566 >> v1:172.21.2.125:6789/0 conn(0x7fe9700eb040 legacy=0x7fe9700eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9751e3700 1 -- 172.21.2.117:0/3054722566 >> v1:172.21.2.117:6789/0 conn(0x7fe9700ef3f0 legacy=0x7fe9701b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.408 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9751e3700 1 -- 172.21.2.117:0/3054722566 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9701bb200 con 0x7fe9700d18f0 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9751e3700 1 --2- 172.21.2.117:0/3054722566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 0x7fe9700eaa20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe96c006650 tx=0x7fe96c009a80).ready entity=mon.2 client_cookie=36809f6012bcecf4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe963fff700 1 -- 172.21.2.117:0/3054722566 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96c00c0c0 con 0x7fe9700d18f0 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe963fff700 1 -- 172.21.2.117:0/3054722566 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96c00c340 con 0x7fe9700d18f0 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe963fff700 1 -- 172.21.2.117:0/3054722566 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96c00c860 con 0x7fe9700d18f0 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.390 7fe9773c7700 1 -- 172.21.2.117:0/3054722566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 msgr2=0x7fe9700eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 --2- 172.21.2.117:0/3054722566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 0x7fe9700eaa20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe96c006650 tx=0x7fe96c009a80).stop 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 -- 172.21.2.117:0/3054722566 shutdown_connections 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 --2- 172.21.2.117:0/3054722566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700d18f0 0x7fe9700eaa20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 -- 172.21.2.117:0/3054722566 >> 172.21.2.117:0/3054722566 conn(0x7fe9700cf060 msgr2=0x7fe9700cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 -- 172.21.2.117:0/3054722566 shutdown_connections 2021-02-10T11:57:34.409 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.391 7fe9773c7700 1 -- 172.21.2.117:0/3054722566 wait complete. 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.405 7fe9773c7700 1 Processor -- start 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.405 7fe9773c7700 1 -- start start 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700a2030 0x7fe9700a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 0x7fe9700a2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:34.411 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9700a8430 con 0x7fe9700a7a10 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9700a8620 con 0x7fe9700a2a50 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9773c7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9700a8840 con 0x7fe9700a2030 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9759e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9759e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60584/0 (socket says 172.21.2.117:60584) 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.406 7fe9759e4700 1 -- 172.21.2.117:0/51876688 learned_addr learned my addr 172.21.2.117:0/51876688 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9749e2700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 0x7fe9700a2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 -- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700a2030 msgr2=0x7fe9700a2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700a2030 0x7fe9700a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 -- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 msgr2=0x7fe9700a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 0x7fe9700a2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 -- 172.21.2.117:0/51876688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700a8a60 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9749e2700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 0x7fe9700a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9759e4700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7fe968009980 tx=0x7fe96800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe96800be10 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968006080 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe9680065a0 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9773c7700 1 -- 172.21.2.117:0/51876688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9700a8a60 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.407 7fe9773c7700 1 -- 172.21.2.117:0/51876688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9700a8ff0 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.408 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe968006260 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.408 7fe961ffb700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.408 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe968032a40 con 0x7fe9700a7a10 2021-02-10T11:57:34.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.408 7fe9751e3700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:34.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.409 7fe9751e3700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 0x7fe95001abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:34.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.439 7fe9773c7700 1 -- 172.21.2.117:0/51876688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9700ab5a0 con 0x7fe9700a7a10 2021-02-10T11:57:34.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.442 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe96801cd70 con 0x7fe9700a7a10 2021-02-10T11:57:34.569 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.567 7fe9773c7700 1 -- 172.21.2.117:0/51876688 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000305114"}) v1 -- 0x7fe9700a8ff0 con 0x7fe9500186c0 2021-02-10T11:57:34.573 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:57:34.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:57:34.574 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:57:34.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:57:34.574 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:57:34.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:57:34.574 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:57:34.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:57:34.574 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:57:34.569 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:57:34.587 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.586 7fe961ffb700 1 -- 172.21.2.117:0/51876688 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fe96c000940 con 0x7fe9500186c0 2021-02-10T11:57:34.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.600 7fe9773c7700 1 -- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 msgr2=0x7fe95001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.600 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 0x7fe95001abc0 crc :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.600 7fe9773c7700 1 -- 172.21.2.117:0/51876688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 msgr2=0x7fe9700a7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:34.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.600 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 rx=0x7fe968009980 tx=0x7fe96800bcd0).stop 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 -- 172.21.2.117:0/51876688 shutdown_connections 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe9500186c0 0x7fe95001abc0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe9700a7a10 0x7fe9700a7e10 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe9700a2a50 0x7fe9700a2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 --2- 172.21.2.117:0/51876688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe9700a2030 0x7fe9700a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 -- 172.21.2.117:0/51876688 >> 172.21.2.117:0/51876688 conn(0x7fe9700ade10 msgr2=0x7fe970094d50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 -- 172.21.2.117:0/51876688 shutdown_connections 2021-02-10T11:57:34.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:34.601 7fe9773c7700 1 -- 172.21.2.117:0/51876688 wait complete. 2021-02-10T11:57:34.611 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000874672 --group_name group_0000000000305114 2021-02-10T11:57:35.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.025 7ff0b80fd700 1 Processor -- start 2021-02-10T11:57:35.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.025 7ff0b80fd700 1 -- start start 2021-02-10T11:57:35.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b80fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b80fd700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff0b00e7d60 0x7ff0b00e8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b80fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b00e77e0 con 0x7ff0b00e8780 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b80fd700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b00e92e0 con 0x7ff0b00e7d60 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b80fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00e5cd0 con 0x7ff0b01b07e0 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b5f19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b5f19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58268/0 (socket says 172.21.2.117:58268) 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b5f19700 1 -- 172.21.2.117:0/133852882 learned_addr learned my addr 172.21.2.117:0/133852882 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.026 7ff0b5718700 1 --2- 172.21.2.117:0/133852882 >> v2:172.21.2.125:3300/0 conn(0x7ff0b00e7d60 0x7ff0b00e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5718700 1 -- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 msgr2=0x7ff0b00e4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5718700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5718700 1 -- 172.21.2.117:0/133852882 >> v1:172.21.2.125:6789/0 conn(0x7ff0b00e8780 legacy=0x7ff0b00e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5718700 1 -- 172.21.2.117:0/133852882 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b00e54a0 con 0x7ff0b00e7d60 2021-02-10T11:57:35.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5f19700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b5718700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00e7d60 0x7ff0b00e8160 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7ff0a0006650 tx=0x7ff0a0009a80).ready entity=mon.1 client_cookie=36d181578dbfe552 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b4f17700 1 -- 172.21.2.117:0/133852882 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a00009b0 con 0x7ff0b00e7d60 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b4f17700 1 -- 172.21.2.117:0/133852882 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a000c0c0 con 0x7ff0b00e7d60 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b4f17700 1 -- 172.21.2.117:0/133852882 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0a000c550 con 0x7ff0b00e7d60 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b80fd700 1 -- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00e7d60 msgr2=0x7ff0b00e8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.027 7ff0b80fd700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00e7d60 0x7ff0b00e8160 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7ff0a0006650 tx=0x7ff0a0009a80).stop 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 -- 172.21.2.117:0/133852882 shutdown_connections 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00e7d60 0x7ff0b00e8160 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 --2- 172.21.2.117:0/133852882 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01b07e0 0x7ff0b00e4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 -- 172.21.2.117:0/133852882 >> 172.21.2.117:0/133852882 conn(0x7ff0b00ce4f0 msgr2=0x7ff0b00e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 -- 172.21.2.117:0/133852882 shutdown_connections 2021-02-10T11:57:35.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.028 7ff0b80fd700 1 -- 172.21.2.117:0/133852882 wait complete. 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 Processor -- start 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 -- start start 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00804b0 0x7ff0b01a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01a3460 0x7ff0b0195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0197250 con 0x7ff0b00800b0 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0195750 con 0x7ff0b00804b0 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b80fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b01962f0 con 0x7ff0b01a3460 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b5f19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.048 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b5f19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60592/0 (socket says 172.21.2.117:60592) 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.043 7ff0b5f19700 1 -- 172.21.2.117:0/791999844 learned_addr learned my addr 172.21.2.117:0/791999844 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 -- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01a3460 msgr2=0x7ff0b0195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01a3460 0x7ff0b0195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 -- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00804b0 msgr2=0x7ff0b01a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00804b0 0x7ff0b01a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 -- 172.21.2.117:0/791999844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b01964d0 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b5f19700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7ff0ac006ee0 tx=0x7ff0ac00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0ac00bd50 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac0052e0 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b01964d0 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff0ac005770 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.044 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b01964d0 con 0x7ff0b00800b0 2021-02-10T11:57:35.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.045 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff0ac0056d0 con 0x7ff0b00800b0 2021-02-10T11:57:35.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.045 7ff0a67fc700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 0x7ff09801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.045 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff0ac032930 con 0x7ff0b00800b0 2021-02-10T11:57:35.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.045 7ff0b5718700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 0x7ff09801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.050 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.046 7ff0b5718700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 0x7ff09801abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.078 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.076 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b007e8b0 con 0x7ff0b00800b0 2021-02-10T11:57:35.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.078 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff0ac0052e0 con 0x7ff0b00800b0 2021-02-10T11:57:35.206 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.204 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "group_name": "group_0000000000305114"}) v1 -- 0x7ff0b00e77e0 con 0x7ff098018730 2021-02-10T11:57:35.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.229 7ff0a67fc700 1 -- 172.21.2.117:0/791999844 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7ff0a00067b0 con 0x7ff098018730 2021-02-10T11:57:35.239 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 msgr2=0x7ff09801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 0x7ff09801abc0 crc :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 msgr2=0x7ff0b01a2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 rx=0x7ff0ac006ee0 tx=0x7ff0ac00b960).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 shutdown_connections 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff0b01a3460 0x7ff0b0195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff0b00804b0 0x7ff0b01a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff098018730 0x7ff09801abc0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 --2- 172.21.2.117:0/791999844 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff0b00800b0 0x7ff0b01a2820 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 >> 172.21.2.117:0/791999844 conn(0x7ff0b01998a0 msgr2=0x7ff0b01aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.237 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 shutdown_connections 2021-02-10T11:57:35.240 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.238 7ff0b80fd700 1 -- 172.21.2.117:0/791999844 wait complete. 2021-02-10T11:57:35.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000874672 alice --group_name group_0000000000305114 2021-02-10T11:57:35.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 Processor -- start 2021-02-10T11:57:35.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 -- start start 2021-02-10T11:57:35.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc38c0ef3f0 0x7fc38c0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fc38c0eb7f0 0x7fc38c1b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0ede60 con 0x7fc38c0d18e0 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38c0ef7f0 con 0x7fc38c0ef3f0 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc3921b5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38c0d1ce0 con 0x7fc38c0eb7f0 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc38affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fc38c0ef3f0 0x7fc38c0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.662 7fc38b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fc38c0d18e0 0x7fc38c0ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60850/0 (socket says 172.21.2.117:60850) 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38b7fe700 1 -- 172.21.2.117:0/2407091161 learned_addr learned my addr 172.21.2.117:0/2407091161 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:35.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38bfff700 1 --2- 172.21.2.117:0/2407091161 >> v2:172.21.2.125:3300/0 conn(0x7fc38c0eb7f0 0x7fc38c1b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38affd700 1 -- 172.21.2.117:0/2407091161 >> v2:172.21.2.125:3300/0 conn(0x7fc38c0eb7f0 msgr2=0x7fc38c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38affd700 1 --2- 172.21.2.117:0/2407091161 >> v2:172.21.2.125:3300/0 conn(0x7fc38c0eb7f0 0x7fc38c1b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38affd700 1 -- 172.21.2.117:0/2407091161 >> v1:172.21.2.117:6789/0 conn(0x7fc38c0d18e0 legacy=0x7fc38c0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38affd700 1 -- 172.21.2.117:0/2407091161 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c1b8f90 con 0x7fc38c0ef3f0 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38affd700 1 --2- 172.21.2.117:0/2407091161 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c0ef3f0 0x7fc38c0eb1d0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fc374006650 tx=0x7fc374009a80).ready entity=mon.0 client_cookie=2bcc9ff15eff8089 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38a7fc700 1 -- 172.21.2.117:0/2407091161 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37400c0c0 con 0x7fc38c0ef3f0 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38a7fc700 1 -- 172.21.2.117:0/2407091161 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37400c340 con 0x7fc38c0ef3f0 2021-02-10T11:57:35.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc38a7fc700 1 -- 172.21.2.117:0/2407091161 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37400c820 con 0x7fc38c0ef3f0 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc3921b5700 1 -- 172.21.2.117:0/2407091161 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c0ef3f0 msgr2=0x7fc38c0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.663 7fc3921b5700 1 --2- 172.21.2.117:0/2407091161 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c0ef3f0 0x7fc38c0eb1d0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 rx=0x7fc374006650 tx=0x7fc374009a80).stop 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 -- 172.21.2.117:0/2407091161 shutdown_connections 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 --2- 172.21.2.117:0/2407091161 >> v2:172.21.2.125:3300/0 conn(0x7fc38c0eb7f0 0x7fc38c1b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 --2- 172.21.2.117:0/2407091161 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c0ef3f0 0x7fc38c0eb1d0 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 -- 172.21.2.117:0/2407091161 >> 172.21.2.117:0/2407091161 conn(0x7fc38c0ced60 msgr2=0x7fc38c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 -- 172.21.2.117:0/2407091161 shutdown_connections 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.664 7fc3921b5700 1 -- 172.21.2.117:0/2407091161 wait complete. 2021-02-10T11:57:35.683 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.678 7fc3921b5700 1 Processor -- start 2021-02-10T11:57:35.684 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.678 7fc3921b5700 1 -- start start 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38c0901d0 0x7fc38c090e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38c091450 0x7fc38c091850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0905d0 con 0x7fc38c08f380 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c08f780 con 0x7fc38c0901d0 2021-02-10T11:57:35.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c1a5f10 con 0x7fc38c091450 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38bfff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38c091450 0x7fc38c091850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60610/0 (socket says 172.21.2.117:60610) 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 -- 172.21.2.117:0/1608488783 learned_addr learned my addr 172.21.2.117:0/1608488783 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38affd700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38c0901d0 0x7fc38c090e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 -- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38c091450 msgr2=0x7fc38c091850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38c091450 0x7fc38c091850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 -- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38c0901d0 msgr2=0x7fc38c090e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.687 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38c0901d0 0x7fc38c090e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 -- 172.21.2.117:0/1608488783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c1a83c0 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc38b7fe700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 rx=0x7fc37c001410 tx=0x7fc37c00a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37c00ad40 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c000c90 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fc37c004590 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.679 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c1a83c0 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.680 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c1a83c0 con 0x7fc38c08f380 2021-02-10T11:57:35.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.680 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fc37c0043a0 con 0x7fc38c08f380 2021-02-10T11:57:35.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.680 7fc383fff700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 0x7fc37001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:35.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.680 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fc37c0326c0 con 0x7fc38c08f380 2021-02-10T11:57:35.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.680 7fc38affd700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 0x7fc37001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:35.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.681 7fc38affd700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 0x7fc37001abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:35.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.712 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c07f0a0 con 0x7fc38c08f380 2021-02-10T11:57:35.716 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.714 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fc37c030da0 con 0x7fc38c08f380 2021-02-10T11:57:35.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.839 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000305114"}) v1 -- 0x7fc38c0ede60 con 0x7fc3700186c0 2021-02-10T11:57:35.846 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID alice with tenant None 2021-02-10T11:57:35.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.872 7fc383fff700 1 -- 172.21.2.117:0/1608488783 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fc374009080 con 0x7fc3700186c0 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 msgr2=0x7fc37001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 0x7fc37001abc0 crc :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 msgr2=0x7fc38c08fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 rx=0x7fc37c001410 tx=0x7fc37c00a950).stop 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 shutdown_connections 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fc38c091450 0x7fc38c091850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fc38c0901d0 0x7fc38c090e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fc3700186c0 0x7fc37001abc0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 --2- 172.21.2.117:0/1608488783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fc38c08f380 0x7fc38c08fbb0 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:35.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 >> 172.21.2.117:0/1608488783 conn(0x7fc38c197410 msgr2=0x7fc38c07f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:35.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 shutdown_connections 2021-02-10T11:57:35.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:35.905 7fc3921b5700 1 -- 172.21.2.117:0/1608488783 wait complete. 2021-02-10T11:57:35.915 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000874672 guest1 --group_name group_0000000000305114 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 Processor -- start 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 -- start start 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24240d18e0 0x7f24240ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240ede60 con 0x7f24240eb000 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24240d1ce0 con 0x7f24240ef3f0 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f242a97d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24240ef7f0 con 0x7f24240d18e0 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f2428f9a700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f24240ef3f0 0x7f24241b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:53792/0 (socket says 172.21.2.117:53792) 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f24237fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f24240eb000 0x7f24240eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60868/0 (socket says 172.21.2.117:60868) 2021-02-10T11:57:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f2428f9a700 1 -- 172.21.2.117:0/3620317090 learned_addr learned my addr 172.21.2.117:0/3620317090 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.330 7f2423fff700 1 --2- 172.21.2.117:0/3620317090 >> v2:172.21.2.125:3300/0 conn(0x7f24240d18e0 0x7f24240ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847548706 0 0) 0x7f2418000fb0 con 0x7f24240eb000 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c002fe0 con 0x7f24240eb000 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529591983 0 0) 0x7f241c000d30 con 0x7f24240ef3f0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c004370 con 0x7f24240ef3f0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2423fff700 1 -- 172.21.2.117:0/3620317090 >> v1:172.21.2.125:6789/0 conn(0x7f24240ef3f0 legacy=0x7f24241b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2423fff700 1 -- 172.21.2.117:0/3620317090 >> v1:172.21.2.117:6789/0 conn(0x7f24240eb000 legacy=0x7f24240eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2423fff700 1 -- 172.21.2.117:0/3620317090 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24241bb180 con 0x7f24240d18e0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2423fff700 1 --2- 172.21.2.117:0/3620317090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24240d18e0 0x7f24240ea9e0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f2414006650 tx=0x7f2414009a00).ready entity=mon.1 client_cookie=ba3745bd1caced7b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2414009df0 con 0x7f24240d18e0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24140009b0 con 0x7f24240d18e0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f2422ffd700 1 -- 172.21.2.117:0/3620317090 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241400c2b0 con 0x7f24240d18e0 2021-02-10T11:57:36.349 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f242a97d700 1 -- 172.21.2.117:0/3620317090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24240d18e0 msgr2=0x7f24240ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.331 7f242a97d700 1 --2- 172.21.2.117:0/3620317090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24240d18e0 0x7f24240ea9e0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f2414006650 tx=0x7f2414009a00).stop 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.332 7f242a97d700 1 -- 172.21.2.117:0/3620317090 shutdown_connections 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.332 7f242a97d700 1 --2- 172.21.2.117:0/3620317090 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24240d18e0 0x7f24240ea9e0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.332 7f242a97d700 1 -- 172.21.2.117:0/3620317090 >> 172.21.2.117:0/3620317090 conn(0x7f24240ced60 msgr2=0x7f24240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.332 7f242a97d700 1 -- 172.21.2.117:0/3620317090 shutdown_connections 2021-02-10T11:57:36.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.332 7f242a97d700 1 -- 172.21.2.117:0/3620317090 wait complete. 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.346 7f242a97d700 1 Processor -- start 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.346 7f242a97d700 1 -- start start 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24240a8300 0x7f2424083d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f242407fd20 0x7f2424080120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24241b8380 con 0x7f2424082260 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2424083800 con 0x7f242407fd20 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2424087170 con 0x7f24240a8300 2021-02-10T11:57:36.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60622/0 (socket says 172.21.2.117:60622) 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 -- 172.21.2.117:0/4136003185 learned_addr learned my addr 172.21.2.117:0/4136003185 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f24237fe700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f242407fd20 0x7f2424080120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2423fff700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24240a8300 0x7f2424083d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 -- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24240a8300 msgr2=0x7f2424083d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24240a8300 0x7f2424083d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 -- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f242407fd20 msgr2=0x7f2424080120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f242407fd20 0x7f2424080120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 -- 172.21.2.117:0/4136003185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424080740 con 0x7f2424082260 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2428f9a700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 rx=0x7f241c00a8f0 tx=0x7f241c00ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c000d30 con 0x7f2424082260 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c0070d0 con 0x7f2424082260 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 -- 172.21.2.117:0/4136003185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2424080740 con 0x7f2424082260 2021-02-10T11:57:36.353 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c0048b0 con 0x7f2424082260 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.347 7f242a97d700 1 -- 172.21.2.117:0/4136003185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2424080740 con 0x7f2424082260 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.348 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f241c00add0 con 0x7f2424082260 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.348 7f2420ff9700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 0x7f240401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.348 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f241c032af0 con 0x7f2424082260 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.348 7f2423fff700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 0x7f240401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:36.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.349 7f2423fff700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 0x7f240401abd0 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:36.382 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.380 7f242a97d700 1 -- 172.21.2.117:0/4136003185 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242407f0a0 con 0x7f2424082260 2021-02-10T11:57:36.384 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.383 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f241c007330 con 0x7f2424082260 2021-02-10T11:57:36.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.506 7f242a97d700 1 -- 172.21.2.117:0/4136003185 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000305114"}) v1 -- 0x7f2424083800 con 0x7f24040186d0 2021-02-10T11:57:36.512 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest1 with tenant None 2021-02-10T11:57:36.547 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.545 7f2420ff9700 1 -- 172.21.2.117:0/4136003185 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f2414006740 con 0x7f24040186d0 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 msgr2=0x7f240401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 0x7f240401abd0 crc :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 msgr2=0x7f2424082660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 rx=0x7f241c00a8f0 tx=0x7f241c00ac90).stop 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 shutdown_connections 2021-02-10T11:57:36.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24040186d0 0x7f240401abd0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2424082260 0x7f2424082660 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f242407fd20 0x7f2424080120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 --2- 172.21.2.117:0/4136003185 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24240a8300 0x7f2424083d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 >> 172.21.2.117:0/4136003185 conn(0x7f2424194230 msgr2=0x7f24241a8960 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 shutdown_connections 2021-02-10T11:57:36.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:36.572 7f242a97d700 1 -- 172.21.2.117:0/4136003185 wait complete. 2021-02-10T11:57:36.582 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorize cephfs subvolume_0000000000874672 guest2 --group_name group_0000000000305114 --access_level r 2021-02-10T11:57:37.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.003 7f4c084d3700 1 Processor -- start 2021-02-10T11:57:37.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c084d3700 1 -- start start 2021-02-10T11:57:37.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c084d3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c001b69a0 0x7f4c001b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c084d3700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000ede60 con 0x7f4c000d18f0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c084d3700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c000eba00 con 0x7f4c000ef3f0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c084d3700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c000d1cf0 con 0x7f4c001b69a0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c06af0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c001b69a0 0x7f4c001b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c06af0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4c001b69a0 0x7f4c001b8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60630/0 (socket says 172.21.2.117:60630) 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.004 7f4c06af0700 1 -- 172.21.2.117:0/1535572493 learned_addr learned my addr 172.21.2.117:0/1535572493 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c06af0700 1 -- 172.21.2.117:0/1535572493 >> v1:172.21.2.125:6789/0 conn(0x7f4c000ef3f0 legacy=0x7f4c000eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c06af0700 1 -- 172.21.2.117:0/1535572493 >> v1:172.21.2.117:6789/0 conn(0x7f4c000d18f0 legacy=0x7f4c000eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c06af0700 1 -- 172.21.2.117:0/1535572493 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c001bb3f0 con 0x7f4c001b69a0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c06af0700 1 --2- 172.21.2.117:0/1535572493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c001b69a0 0x7f4c001b8db0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 rx=0x7f4bf8006f00 tx=0x7f4bf8009f10).ready entity=mon.0 client_cookie=f473c311664c2471 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c052ed700 1 -- 172.21.2.117:0/1535572493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf800b3b0 con 0x7f4c001b69a0 2021-02-10T11:57:37.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c052ed700 1 -- 172.21.2.117:0/1535572493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf800b630 con 0x7f4c001b69a0 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c052ed700 1 -- 172.21.2.117:0/1535572493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bf800bb50 con 0x7f4c001b69a0 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c084d3700 1 -- 172.21.2.117:0/1535572493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c001b69a0 msgr2=0x7f4c001b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.005 7f4c084d3700 1 --2- 172.21.2.117:0/1535572493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c001b69a0 0x7f4c001b8db0 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 rx=0x7f4bf8006f00 tx=0x7f4bf8009f10).stop 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.006 7f4c084d3700 1 -- 172.21.2.117:0/1535572493 shutdown_connections 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.006 7f4c084d3700 1 --2- 172.21.2.117:0/1535572493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c001b69a0 0x7f4c001b8db0 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.006 7f4c084d3700 1 -- 172.21.2.117:0/1535572493 >> 172.21.2.117:0/1535572493 conn(0x7f4c000cf060 msgr2=0x7f4c000cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.006 7f4c084d3700 1 -- 172.21.2.117:0/1535572493 shutdown_connections 2021-02-10T11:57:37.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.006 7f4c084d3700 1 -- 172.21.2.117:0/1535572493 wait complete. 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 Processor -- start 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 -- start start 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c001c68c0 0x7f4c000df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c000958a0 0x7f4c00095ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c00099520 con 0x7f4c000dfc00 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000b21f0 con 0x7f4c000958a0 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c084d3700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c000b23d0 con 0x7f4c001c68c0 2021-02-10T11:57:37.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.021 7f4c05aee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60636/0 (socket says 172.21.2.117:60636) 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 -- 172.21.2.117:0/3755134786 learned_addr learned my addr 172.21.2.117:0/3755134786 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 -- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c001c68c0 msgr2=0x7f4c000df5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c001c68c0 0x7f4c000df5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 -- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c000958a0 msgr2=0x7f4c00095ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c000958a0 0x7f4c00095ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 -- 172.21.2.117:0/3755134786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c000b25f0 con 0x7f4c000dfc00 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c05aee700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7f4bfc00b5b0 tx=0x7f4bfc00b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bfc00bd40 con 0x7f4c000dfc00 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc002080 con 0x7f4c000dfc00 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4bfc002510 con 0x7f4c000dfc00 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c000b25f0 con 0x7f4c000dfc00 2021-02-10T11:57:37.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.022 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c000b25f0 con 0x7f4c000dfc00 2021-02-10T11:57:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.023 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4bfc002260 con 0x7f4c000dfc00 2021-02-10T11:57:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.023 7f4bf2ffd700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 0x7f4be801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.023 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4bfc0328f0 con 0x7f4c000dfc00 2021-02-10T11:57:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.023 7f4c062ef700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 0x7f4be801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.024 7f4c062ef700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 0x7f4be801abc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.056 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.054 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c000873c0 con 0x7f4c000dfc00 2021-02-10T11:57:37.058 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.057 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4bfc0009b0 con 0x7f4c000dfc00 2021-02-10T11:57:37.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.180 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"access_level": "r", "prefix": "fs subvolume authorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000305114"}) v1 -- 0x7f4c000873c0 con 0x7f4be8018700 2021-02-10T11:57:37.187 INFO:tasks.ceph.mgr.x.gibba025.stderr:Creating meta for ID guest2 with tenant None 2021-02-10T11:57:37.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.223 7f4bf2ffd700 1 -- 172.21.2.117:0/3755134786 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f4bf4006db0 con 0x7f4be8018700 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.245 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 msgr2=0x7f4be801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.245 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 0x7f4be801abc0 crc :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 msgr2=0x7f4c00095280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7f4bfc00b5b0 tx=0x7f4bfc00b950).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 shutdown_connections 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4be8018700 0x7f4be801abc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4c000dfc00 0x7f4c00095280 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4c000958a0 0x7f4c00095ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 --2- 172.21.2.117:0/3755134786 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4c001c68c0 0x7f4c000df5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 >> 172.21.2.117:0/3755134786 conn(0x7f4c000e9970 msgr2=0x7f4c0009fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 shutdown_connections 2021-02-10T11:57:37.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.246 7f4c084d3700 1 -- 172.21.2.117:0/3755134786 wait complete. 2021-02-10T11:57:37.256 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume authorized_list cephfs subvolume_0000000000874672 --group_name group_0000000000305114 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.669 7f176d0a5700 1 Processor -- start 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.669 7f176d0a5700 1 -- start start 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176d0a5700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f17680e5240 0x7f1768086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176d0a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 0x7f1768085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176d0a5700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1768085930 con 0x7f17680e4e40 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176d0a5700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1768085680 con 0x7f17680e5240 2021-02-10T11:57:37.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176d0a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17681b09e0 con 0x7f1768084c60 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f1766d9d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f17680e4e40 0x7f17681b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60890/0 (socket says 172.21.2.117:60890) 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f1766d9d700 1 -- 172.21.2.117:0/3170422497 learned_addr learned my addr 172.21.2.117:0/3170422497 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176759e700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 0x7f1768085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f176659c700 1 --2- 172.21.2.117:0/3170422497 >> v2:172.21.2.125:3300/0 conn(0x7f17680e5240 0x7f1768086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f1765d9b700 1 -- 172.21.2.117:0/3170422497 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424564270 0 0) 0x7f1758000ce0 con 0x7f17680e4e40 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.670 7f1765d9b700 1 -- 172.21.2.117:0/3170422497 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1754002fe0 con 0x7f17680e4e40 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176659c700 1 -- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 msgr2=0x7f1768085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176659c700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 0x7f1768085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176659c700 1 -- 172.21.2.117:0/3170422497 >> v1:172.21.2.117:6789/0 conn(0x7f17680e4e40 legacy=0x7f17681b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176659c700 1 -- 172.21.2.117:0/3170422497 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1768084840 con 0x7f17680e5240 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176759e700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 0x7f1768085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176659c700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17680e5240 0x7f1768086ec0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f1750006650 tx=0x7f1750009a80).ready entity=mon.1 client_cookie=350cfa49f9d4f11b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f1765d9b700 1 -- 172.21.2.117:0/3170422497 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175000c0c0 con 0x7f17680e5240 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f1765d9b700 1 -- 172.21.2.117:0/3170422497 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175000c340 con 0x7f17680e5240 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f1765d9b700 1 -- 172.21.2.117:0/3170422497 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f175000c790 con 0x7f17680e5240 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176d0a5700 1 -- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17680e5240 msgr2=0x7f1768086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.671 7f176d0a5700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17680e5240 0x7f1768086ec0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7f1750006650 tx=0x7f1750009a80).stop 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 -- 172.21.2.117:0/3170422497 shutdown_connections 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f1768084c60 0x7f1768085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 --2- 172.21.2.117:0/3170422497 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17680e5240 0x7f1768086ec0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 -- 172.21.2.117:0/3170422497 >> 172.21.2.117:0/3170422497 conn(0x7f17680ce4e0 msgr2=0x7f17680e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 -- 172.21.2.117:0/3170422497 shutdown_connections 2021-02-10T11:57:37.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.672 7f176d0a5700 1 -- 172.21.2.117:0/3170422497 wait complete. 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.686 7f176d0a5700 1 Processor -- start 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.686 7f176d0a5700 1 -- start start 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17680abe50 0x7f176808a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17681a7060 0x7f17681a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.692 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17681a7a80 con 0x7f176808a8a0 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17681a7ca0 con 0x7f17681a7060 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176d0a5700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17681a7ec0 con 0x7f17680abe50 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60646/0 (socket says 172.21.2.117:60646) 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 -- 172.21.2.117:0/2685174066 learned_addr learned my addr 172.21.2.117:0/2685174066 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176759e700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17681a7060 0x7f17681a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f1766d9d700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17680abe50 0x7f176808a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 -- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17680abe50 msgr2=0x7f176808a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17680abe50 0x7f176808a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 -- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17681a7060 msgr2=0x7f17681a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17681a7060 0x7f17681a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.687 7f176659c700 1 -- 172.21.2.117:0/2685174066 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17680b7030 con 0x7f176808a8a0 2021-02-10T11:57:37.693 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f176659c700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 rx=0x7f17500091f0 tx=0x7f175000b9c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f17500009b0 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175000bdb0 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17680b7030 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17680b75c0 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1750002030 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f17500052e0 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.688 7f175f7fe700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 0x7f174c01ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.689 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f17500328c0 con 0x7f176808a8a0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.689 7f1766d9d700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 0x7f174c01ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:37.694 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.689 7f1766d9d700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 0x7f174c01ac10 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:37.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.720 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176810c2b0 con 0x7f176808a8a0 2021-02-10T11:57:37.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.722 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f17500009b0 con 0x7f176808a8a0 2021-02-10T11:57:37.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.845 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume authorized_list", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "group_name": "group_0000000000305114"}) v1 -- 0x7f17681a7a80 con 0x7f174c018730 2021-02-10T11:57:37.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.850 7f175f7fe700 1 -- 172.21.2.117:0/2685174066 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+111 (crc 0 0 0) 0x7f1758007770 con 0x7f174c018730 2021-02-10T11:57:37.854 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: "guest2": "r" 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: "alice": "rw" 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: "guest1": "rw" 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:37.855 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 msgr2=0x7f174c01ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 0x7f174c01ac10 crc :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 msgr2=0x7f176808aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 rx=0x7f17500091f0 tx=0x7f175000b9c0).stop 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 shutdown_connections 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f174c018730 0x7f174c01ac10 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f176808a8a0 0x7f176808aca0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f17681a7060 0x7f17681a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 --2- 172.21.2.117:0/2685174066 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f17680abe50 0x7f176808a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:37.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 >> 172.21.2.117:0/2685174066 conn(0x7f17680acec0 msgr2=0x7f17680b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:37.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 shutdown_connections 2021-02-10T11:57:37.858 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:37.855 7f176d0a5700 1 -- 172.21.2.117:0/2685174066 wait complete. 2021-02-10T11:57:37.865 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000874672 alice --group_name group_0000000000305114 2021-02-10T11:57:38.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 Processor -- start 2021-02-10T11:57:38.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 -- start start 2021-02-10T11:57:38.299 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 0x7f23880e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23880e77f0 con 0x7f23880e8790 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23880e92f0 con 0x7f23880e7d70 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.281 7f238d068700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23880e5ce0 con 0x7f23881b07f0 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58334/0 (socket says 172.21.2.117:58334) 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f238659c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 0x7f23880e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 -- 172.21.2.117:0/950110779 learned_addr learned my addr 172.21.2.117:0/950110779 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 -- 172.21.2.117:0/950110779 >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 msgr2=0x7f23880e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 --2- 172.21.2.117:0/950110779 >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 0x7f23880e8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 -- 172.21.2.117:0/950110779 >> v1:172.21.2.125:6789/0 conn(0x7f23880e8790 legacy=0x7f23880e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 -- 172.21.2.117:0/950110779 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23880e54b0 con 0x7f23881b07f0 2021-02-10T11:57:38.300 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f238659c700 1 --2- 172.21.2.117:0/950110779 >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 0x7f23880e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2386d9d700 1 --2- 172.21.2.117:0/950110779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2370006700 tx=0x7f2370009b80).ready entity=mon.2 client_cookie=88f81cbf0e706440 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.282 7f2385d9b700 1 -- 172.21.2.117:0/950110779 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2370006340 con 0x7f23881b07f0 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f2385d9b700 1 -- 172.21.2.117:0/950110779 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237000c0c0 con 0x7f23881b07f0 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f2385d9b700 1 -- 172.21.2.117:0/950110779 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237000c5a0 con 0x7f23881b07f0 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 -- 172.21.2.117:0/950110779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 msgr2=0x7f23880e4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 --2- 172.21.2.117:0/950110779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2370006700 tx=0x7f2370009b80).stop 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 -- 172.21.2.117:0/950110779 shutdown_connections 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 --2- 172.21.2.117:0/950110779 >> v2:172.21.2.125:3300/0 conn(0x7f23880e7d70 0x7f23880e8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 --2- 172.21.2.117:0/950110779 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23881b07f0 0x7f23880e4e90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 -- 172.21.2.117:0/950110779 >> 172.21.2.117:0/950110779 conn(0x7f23880ce500 msgr2=0x7f23880e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 -- 172.21.2.117:0/950110779 shutdown_connections 2021-02-10T11:57:38.301 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.283 7f238d068700 1 -- 172.21.2.117:0/950110779 wait complete. 2021-02-10T11:57:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 Processor -- start 2021-02-10T11:57:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 -- start start 2021-02-10T11:57:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f238818cae0 0x7f23881a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.303 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23880b0050 0x7f238810cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23881a72b0 con 0x7f238818cee0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23881085d0 con 0x7f238818cae0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238d068700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238810d400 con 0x7f23880b0050 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60662/0 (socket says 172.21.2.117:60662) 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238659c700 1 -- 172.21.2.117:0/3794068076 learned_addr learned my addr 172.21.2.117:0/3794068076 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.298 7f238759e700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23880b0050 0x7f238810cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f2386d9d700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f238818cae0 0x7f23881a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 -- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23880b0050 msgr2=0x7f238810cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23880b0050 0x7f238810cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 -- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f238818cae0 msgr2=0x7f23881a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f238818cae0 0x7f23881a6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 -- 172.21.2.117:0/3794068076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238810d620 con 0x7f238818cee0 2021-02-10T11:57:38.304 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238659c700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f2378006f40 tx=0x7f237800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f237800bd80 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2378000c90 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238d068700 1 -- 172.21.2.117:0/3794068076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238810d620 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2378002220 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.299 7f238d068700 1 -- 172.21.2.117:0/3794068076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238818ae00 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.300 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f237800a000 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.300 7f237f7fe700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 0x7f236c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.300 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2378032580 con 0x7f238818cee0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.300 7f2386d9d700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 0x7f236c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.301 7f2386d9d700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 0x7f236c01abc0 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:38.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.331 7f238d068700 1 -- 172.21.2.117:0/3794068076 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2388080e20 con 0x7f238818cee0 2021-02-10T11:57:38.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.334 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f237802df10 con 0x7f238818cee0 2021-02-10T11:57:38.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.458 7f238d068700 1 -- 172.21.2.117:0/3794068076 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "alice", "group_name": "group_0000000000305114"}) v1 -- 0x7f238818ae00 con 0x7f236c0186c0 2021-02-10T11:57:38.496 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.494 7f237f7fe700 1 -- 172.21.2.117:0/3794068076 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f23700009b0 con 0x7f236c0186c0 2021-02-10T11:57:38.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 -- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 msgr2=0x7f236c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 0x7f236c01abc0 crc :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.526 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 -- 172.21.2.117:0/3794068076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 msgr2=0x7f23880afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 rx=0x7f2378006f40 tx=0x7f237800b990).stop 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 -- 172.21.2.117:0/3794068076 shutdown_connections 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f236c0186c0 0x7f236c01abc0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f238818cee0 0x7f23880afa30 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f23880b0050 0x7f238810cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 --2- 172.21.2.117:0/3794068076 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f238818cae0 0x7f23881a6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 -- 172.21.2.117:0/3794068076 >> 172.21.2.117:0/3794068076 conn(0x7f238818a7e0 msgr2=0x7f2388195e70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.524 7f238d068700 1 -- 172.21.2.117:0/3794068076 shutdown_connections 2021-02-10T11:57:38.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.525 7f238d068700 1 -- 172.21.2.117:0/3794068076 wait complete. 2021-02-10T11:57:38.534 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000874672 guest1 --group_name group_0000000000305114 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.947 7fb92e11f700 1 Processor -- start 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.947 7fb92e11f700 1 -- start start 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb92e11f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb9280ef3f0 0x7fb9280eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb92e11f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9280ede60 con 0x7fb9280eb7f0 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb92e11f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9280d1ce0 con 0x7fb9280d18e0 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb92e11f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280ef7f0 con 0x7fb9280ef3f0 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb927fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb9280eb7f0 0x7fb9281b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60922/0 (socket says 172.21.2.117:60922) 2021-02-10T11:57:38.967 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb927fff700 1 -- 172.21.2.117:0/2957114277 learned_addr learned my addr 172.21.2.117:0/2957114277 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb926ffd700 1 --2- 172.21.2.117:0/2957114277 >> v2:172.21.2.125:3300/0 conn(0x7fb9280ef3f0 0x7fb9280eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276802549 0 0) 0x7fb920001500 con 0x7fb9280eb7f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.948 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb914002fe0 con 0x7fb9280eb7f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb926ffd700 1 -- 172.21.2.117:0/2957114277 >> v1:172.21.2.125:6789/0 conn(0x7fb9280d18e0 legacy=0x7fb9280ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340990082 0 0) 0x7fb910001080 con 0x7fb9280d18e0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb926ffd700 1 -- 172.21.2.117:0/2957114277 >> v1:172.21.2.117:6789/0 conn(0x7fb9280eb7f0 legacy=0x7fb9281b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb926ffd700 1 -- 172.21.2.117:0/2957114277 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9281bb180 con 0x7fb9280ef3f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb926ffd700 1 --2- 172.21.2.117:0/2957114277 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9280ef3f0 0x7fb9280eb1d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fb918006650 tx=0x7fb918009a80).ready entity=mon.1 client_cookie=3e929a271b438967 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb91800c0c0 con 0x7fb9280ef3f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91800c340 con 0x7fb9280ef3f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb9267fc700 1 -- 172.21.2.117:0/2957114277 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb91800c860 con 0x7fb9280ef3f0 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb92e11f700 1 -- 172.21.2.117:0/2957114277 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9280ef3f0 msgr2=0x7fb9280eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.968 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.949 7fb92e11f700 1 --2- 172.21.2.117:0/2957114277 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9280ef3f0 0x7fb9280eb1d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7fb918006650 tx=0x7fb918009a80).stop 2021-02-10T11:57:38.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.950 7fb92e11f700 1 -- 172.21.2.117:0/2957114277 shutdown_connections 2021-02-10T11:57:38.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.950 7fb92e11f700 1 --2- 172.21.2.117:0/2957114277 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb9280ef3f0 0x7fb9280eb1d0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:38.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.950 7fb92e11f700 1 -- 172.21.2.117:0/2957114277 >> 172.21.2.117:0/2957114277 conn(0x7fb9280ced60 msgr2=0x7fb9280cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:38.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.950 7fb92e11f700 1 -- 172.21.2.117:0/2957114277 shutdown_connections 2021-02-10T11:57:38.969 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.950 7fb92e11f700 1 -- 172.21.2.117:0/2957114277 wait complete. 2021-02-10T11:57:38.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.964 7fb92e11f700 1 Processor -- start 2021-02-10T11:57:38.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.964 7fb92e11f700 1 -- start start 2021-02-10T11:57:38.970 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9280e1560 0x7fb9280e1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb92808b010 0x7fb92808b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280e2360 con 0x7fb9280e3280 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280e3eb0 con 0x7fb92808b010 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb92e11f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280e1f80 con 0x7fb9280e1560 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60676/0 (socket says 172.21.2.117:60676) 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 -- 172.21.2.117:0/1516204862 learned_addr learned my addr 172.21.2.117:0/1516204862 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 -- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9280e1560 msgr2=0x7fb9280e1960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9280e1560 0x7fb9280e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 -- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb92808b010 msgr2=0x7fb92808b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:38.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb92808b010 0x7fb92808b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.965 7fb9277fe700 1 -- 172.21.2.117:0/1516204862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92808ba30 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb9277fe700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7fb91000b580 tx=0x7fb91000b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb91000bd10 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910006030 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92808ba30 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92808bfc0 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb910005540 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb910006290 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 0x7fb90c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.966 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb9100326e0 con 0x7fb9280e3280 2021-02-10T11:57:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.967 7fb926ffd700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 0x7fb90c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.967 7fb926ffd700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 0x7fb90c01abc0 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:39.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:38.998 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92807f0a0 con 0x7fb9280e3280 2021-02-10T11:57:39.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.001 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb910000b70 con 0x7fb9280e3280 2021-02-10T11:57:39.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.127 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "guest1", "group_name": "group_0000000000305114"}) v1 -- 0x7fb9280ede60 con 0x7fb90c0186c0 2021-02-10T11:57:39.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.162 7fb91ffff700 1 -- 172.21.2.117:0/1516204862 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fb9180009b0 con 0x7fb90c0186c0 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 msgr2=0x7fb90c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 0x7fb90c01abc0 crc :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 msgr2=0x7fb9280e3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 rx=0x7fb91000b580 tx=0x7fb91000b920).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 shutdown_connections 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb92808b010 0x7fb92808b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb9280e1560 0x7fb9280e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb90c0186c0 0x7fb90c01abc0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 --2- 172.21.2.117:0/1516204862 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb9280e3280 0x7fb9280e3680 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.192 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 >> 172.21.2.117:0/1516204862 conn(0x7fb928197410 msgr2=0x7fb9280e2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.193 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 shutdown_connections 2021-02-10T11:57:39.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.193 7fb92e11f700 1 -- 172.21.2.117:0/1516204862 wait complete. 2021-02-10T11:57:39.204 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume deauthorize cephfs subvolume_0000000000874672 guest2 --group_name group_0000000000305114 2021-02-10T11:57:39.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.617 7f9ffae99700 1 Processor -- start 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ffae99700 1 -- start start 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ffae99700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9ff40d18e0 0x7f9ff40ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ffae99700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40ede60 con 0x7f9ff40eb000 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ffae99700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40d1ce0 con 0x7f9ff40ef3f0 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ffae99700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40ef7f0 con 0x7f9ff40d18e0 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ff3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9ff40eb000 0x7f9ff40eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60932/0 (socket says 172.21.2.117:60932) 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.618 7f9ff3fff700 1 -- 172.21.2.117:0/281204170 learned_addr learned my addr 172.21.2.117:0/281204170 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159440192 0 0) 0x7f9fe4001060 con 0x7f9ff40eb000 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> v2:172.21.2.125:3300/0 conn(0x7f9ff40d18e0 0x7f9ff40ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0002fe0 con 0x7f9ff40eb000 2021-02-10T11:57:39.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1570278967 0 0) 0x7f9fe4001500 con 0x7f9ff40eb000 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 >> v2:172.21.2.125:3300/0 conn(0x7f9ff40d18e0 msgr2=0x7f9ff40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 --2- 172.21.2.117:0/281204170 >> v2:172.21.2.125:3300/0 conn(0x7f9ff40d18e0 0x7f9ff40ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 >> v1:172.21.2.125:6789/0 conn(0x7f9ff40ef3f0 legacy=0x7f9ff41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff41bb180 con 0x7f9ff40eb000 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f9fe4001a00 con 0x7f9ff40eb000 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 >> v1:172.21.2.117:6789/0 conn(0x7f9ff40eb000 legacy=0x7f9ff40eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 0x7f9fe0006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fe00068a0 0x7f9fe0008dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fe00096c0 0x7f9fe000bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe000c430 con 0x7f9fe0003ae0 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe000c650 con 0x7f9fe00068a0 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0050880 con 0x7f9fe00096c0 2021-02-10T11:57:39.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.619 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> v2:172.21.2.125:3300/0 conn(0x7f9ff40d18e0 0x7f9ff40ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 0x7f9fe0006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff3fff700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fe00068a0 0x7f9fe0008dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff94b6700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fe00096c0 0x7f9fe000bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 -- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fe00096c0 msgr2=0x7f9fe000bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9fe00096c0 0x7f9fe000bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 -- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fe00068a0 msgr2=0x7f9fe0008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9fe00068a0 0x7f9fe0008dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 -- 172.21.2.117:0/281204170 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0050a70 con 0x7f9fe0003ae0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff94b6700 1 -- 172.21.2.117:0/281204170 reap_dead start 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff8cb5700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 0x7f9fe0006000 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f9fdc000b80 tx=0x7f9fdc009e80).ready entity=mon.0 client_cookie=855253df2ab9afc2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fdc006dc0 con 0x7f9fe0003ae0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc0063c0 con 0x7f9fe0003ae0 2021-02-10T11:57:39.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ff37fe700 1 -- 172.21.2.117:0/281204170 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fdc007260 con 0x7f9fe0003ae0 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ffae99700 1 -- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 msgr2=0x7f9fe0006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.620 7f9ffae99700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 0x7f9fe0006000 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 rx=0x7f9fdc000b80 tx=0x7f9fdc009e80).stop 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.621 7f9ffae99700 1 -- 172.21.2.117:0/281204170 shutdown_connections 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.621 7f9ffae99700 1 --2- 172.21.2.117:0/281204170 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9fe0003ae0 0x7f9fe0006000 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.621 7f9ffae99700 1 -- 172.21.2.117:0/281204170 >> 172.21.2.117:0/281204170 conn(0x7f9ff40ced60 msgr2=0x7f9ff40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.621 7f9ffae99700 1 -- 172.21.2.117:0/281204170 shutdown_connections 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.621 7f9ffae99700 1 -- 172.21.2.117:0/281204170 wait complete. 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 Processor -- start 2021-02-10T11:57:39.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 -- start start 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff409b630 0x7f9ff409bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff41a3880 0x7f9ff40b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff409c1f0 con 0x7f9ff41a2e60 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b5810 con 0x7f9ff41a3880 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ffae99700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40b5a30 con 0x7f9ff409b630 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ff3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ff3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60696/0 (socket says 172.21.2.117:60696) 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.636 7f9ff3fff700 1 -- 172.21.2.117:0/4218143764 learned_addr learned my addr 172.21.2.117:0/4218143764 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 -- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff409b630 msgr2=0x7f9ff409bbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff409b630 0x7f9ff409bbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 -- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff41a3880 msgr2=0x7f9ff40b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff41a3880 0x7f9ff40b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 -- 172.21.2.117:0/4218143764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff40b5c50 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff3fff700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f9fe400b990 tx=0x7f9fe4009cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe4009e00 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe40070d0 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9fe4004720 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff40b5c50 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.637 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff40b5c50 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.638 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9fe40072b0 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.638 7f9ff17fa700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.638 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9fe4032c70 con 0x7f9ff41a2e60 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.638 7f9ff8cb5700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:39.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.639 7f9ff8cb5700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:39.670 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.668 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40ede60 con 0x7f9ff41a2e60 2021-02-10T11:57:39.673 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.670 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9fe4007660 con 0x7f9ff41a2e60 2021-02-10T11:57:39.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.795 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume deauthorize", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "auth_id": "guest2", "group_name": "group_0000000000305114"}) v1 -- 0x7f9ff409c1f0 con 0x7f9fd80186c0 2021-02-10T11:57:39.835 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.832 7f9ff17fa700 1 -- 172.21.2.117:0/4218143764 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9fdc006250 con 0x7f9fd80186c0 2021-02-10T11:57:39.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 msgr2=0x7f9fd801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 crc :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 msgr2=0x7f9ff41a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:39.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 rx=0x7f9fe400b990 tx=0x7f9fe4009cc0).stop 2021-02-10T11:57:39.864 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 shutdown_connections 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9fd80186c0 0x7f9fd801abc0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ff41a2e60 0x7f9ff41a3260 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ff41a3880 0x7f9ff40b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 --2- 172.21.2.117:0/4218143764 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ff409b630 0x7f9ff409bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 >> 172.21.2.117:0/4218143764 conn(0x7f9ff4092880 msgr2=0x7f9ff40aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 shutdown_connections 2021-02-10T11:57:39.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:39.861 7f9ffae99700 1 -- 172.21.2.117:0/4218143764 wait complete. 2021-02-10T11:57:39.872 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume rm cephfs subvolume_0000000000874672 --group_name group_0000000000305114 2021-02-10T11:57:40.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.287 7f242f2fc700 1 Processor -- start 2021-02-10T11:57:40.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 -- start start 2021-02-10T11:57:40.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24280e4e40 0x7f24281b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f24280e5240 0x7f2428084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24281b0810 0x7f2428085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280e5cc0 con 0x7f24280e4e40 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2428085930 con 0x7f24280e5240 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242f2fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2428084840 con 0x7f24281b0810 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242d118700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24280e4e40 0x7f24281b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242d118700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f24280e4e40 0x7f24281b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60702/0 (socket says 172.21.2.117:60702) 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.288 7f242d118700 1 -- 172.21.2.117:0/1397308985 learned_addr learned my addr 172.21.2.117:0/1397308985 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 -- 172.21.2.117:0/1397308985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24281b0810 msgr2=0x7f2428085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 --2- 172.21.2.117:0/1397308985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24281b0810 0x7f2428085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 -- 172.21.2.117:0/1397308985 >> v2:172.21.2.125:3300/0 conn(0x7f24280e5240 msgr2=0x7f2428084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 --2- 172.21.2.117:0/1397308985 >> v2:172.21.2.125:3300/0 conn(0x7f24280e5240 0x7f2428084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 -- 172.21.2.117:0/1397308985 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428085eb0 con 0x7f24280e4e40 2021-02-10T11:57:40.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242d118700 1 --2- 172.21.2.117:0/1397308985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280e4e40 0x7f24281b01f0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 rx=0x7f241c006700 tx=0x7f241c009b80).ready entity=mon.0 client_cookie=4fbeed9ac0c8168e server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f241bfff700 1 -- 172.21.2.117:0/1397308985 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c006340 con 0x7f24280e4e40 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f241bfff700 1 -- 172.21.2.117:0/1397308985 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c00c0c0 con 0x7f24280e4e40 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f241bfff700 1 -- 172.21.2.117:0/1397308985 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c00c510 con 0x7f24280e4e40 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242f2fc700 1 -- 172.21.2.117:0/1397308985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280e4e40 msgr2=0x7f24281b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.289 7f242f2fc700 1 --2- 172.21.2.117:0/1397308985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280e4e40 0x7f24281b01f0 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 rx=0x7f241c006700 tx=0x7f241c009b80).stop 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 -- 172.21.2.117:0/1397308985 shutdown_connections 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 --2- 172.21.2.117:0/1397308985 >> v2:172.21.2.125:3300/0 conn(0x7f24280e5240 0x7f2428084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 --2- 172.21.2.117:0/1397308985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24281b0810 0x7f2428085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 --2- 172.21.2.117:0/1397308985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280e4e40 0x7f24281b01f0 unknown :-1 s=CLOSED pgs=3179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 -- 172.21.2.117:0/1397308985 >> 172.21.2.117:0/1397308985 conn(0x7f24280ce4e0 msgr2=0x7f24280e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 -- 172.21.2.117:0/1397308985 shutdown_connections 2021-02-10T11:57:40.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.290 7f242f2fc700 1 -- 172.21.2.117:0/1397308985 wait complete. 2021-02-10T11:57:40.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 Processor -- start 2021-02-10T11:57:40.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 -- start start 2021-02-10T11:57:40.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24280ad1b0 0x7f24280ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24280adbd0 0x7f242809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242818d0c0 con 0x7f24280973b0 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242809ae70 con 0x7f24280ad1b0 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242f2fc700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242809ba10 con 0x7f24280adbd0 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242d118700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242d118700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60708/0 (socket says 172.21.2.117:60708) 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.305 7f242d118700 1 -- 172.21.2.117:0/3280716356 learned_addr learned my addr 172.21.2.117:0/3280716356 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 -- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24280adbd0 msgr2=0x7f242809b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24280adbd0 0x7f242809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 -- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24280ad1b0 msgr2=0x7f24280ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24280ad1b0 0x7f24280ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 -- 172.21.2.117:0/3280716356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242809bbf0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242d118700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 rx=0x7f241c00b580 tx=0x7f241c00b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c00bd10 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c006030 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f241c0054d0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242809bbf0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.306 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242809bbf0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.307 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f241c0052e0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.307 7f2419ffb700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 0x7f240801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.307 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f241c0328b0 con 0x7f24280973b0 2021-02-10T11:57:40.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.307 7f242c917700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 0x7f240801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.308 7f242c917700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 0x7f240801abc0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.341 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.338 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2428085930 con 0x7f24280973b0 2021-02-10T11:57:40.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.341 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f241c0065b0 con 0x7f24280973b0 2021-02-10T11:57:40.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.465 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000874672", "target": ["mgr", ""], "group_name": "group_0000000000305114"}) v1 -- 0x7f2428085930 con 0x7f24080186c0 2021-02-10T11:57:40.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.476 7f2419ffb700 1 -- 172.21.2.117:0/3280716356 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f24240009b0 con 0x7f24080186c0 2021-02-10T11:57:40.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.483 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 msgr2=0x7f240801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.483 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 0x7f240801abc0 crc :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.483 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 msgr2=0x7f2428097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.486 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.483 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 rx=0x7f241c00b580 tx=0x7f241c00b920).stop 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 shutdown_connections 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f24280adbd0 0x7f242809b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f24280ad1b0 0x7f24280ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f24080186c0 0x7f240801abc0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 --2- 172.21.2.117:0/3280716356 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f24280973b0 0x7f2428097980 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 >> 172.21.2.117:0/3280716356 conn(0x7f242818cc70 msgr2=0x7f2428093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 shutdown_connections 2021-02-10T11:57:40.487 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.484 7f242f2fc700 1 -- 172.21.2.117:0/3280716356 wait complete. 2021-02-10T11:57:40.495 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolumegroup rm cephfs group_0000000000305114 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.909 7feca1bbd700 1 Processor -- start 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.909 7feca1bbd700 1 -- start start 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7feca1bbd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 0x7fec9c1b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7feca1bbd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0ede60 con 0x7fec9c0ef3f0 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7feca1bbd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0eba00 con 0x7fec9c0d18f0 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7feca1bbd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0d3170 con 0x7fec9c1b69a0 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7fec9affd700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fec9c0ef3f0 0x7fec9c0eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60966/0 (socket says 172.21.2.117:60966) 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7fec9affd700 1 -- 172.21.2.117:0/3240318704 learned_addr learned my addr 172.21.2.117:0/3240318704 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7fec9bfff700 1 --2- 172.21.2.117:0/3240318704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 0x7fec9c1b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.929 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699992702 0 0) 0x7fec8c001180 con 0x7fec9c0ef3f0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.910 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec88002fe0 con 0x7fec9c0ef3f0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050941625 0 0) 0x7fec84001080 con 0x7fec9c0d18f0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9bfff700 1 -- 172.21.2.117:0/3240318704 >> v1:172.21.2.125:6789/0 conn(0x7fec9c0d18f0 legacy=0x7fec9c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9bfff700 1 -- 172.21.2.117:0/3240318704 >> v1:172.21.2.117:6789/0 conn(0x7fec9c0ef3f0 legacy=0x7fec9c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9bfff700 1 -- 172.21.2.117:0/3240318704 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c0d1cf0 con 0x7fec9c1b69a0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9bfff700 1 --2- 172.21.2.117:0/3240318704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 0x7fec9c1b8db0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fec94006e50 tx=0x7fec94009df0).ready entity=mon.2 client_cookie=676c7b41e8701969 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9400b510 con 0x7fec9c1b69a0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94006790 con 0x7fec9c1b69a0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7fec9a7fc700 1 -- 172.21.2.117:0/3240318704 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9400b750 con 0x7fec9c1b69a0 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7feca1bbd700 1 -- 172.21.2.117:0/3240318704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 msgr2=0x7fec9c1b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.911 7feca1bbd700 1 --2- 172.21.2.117:0/3240318704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 0x7fec9c1b8db0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fec94006e50 tx=0x7fec94009df0).stop 2021-02-10T11:57:40.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.912 7feca1bbd700 1 -- 172.21.2.117:0/3240318704 shutdown_connections 2021-02-10T11:57:40.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.912 7feca1bbd700 1 --2- 172.21.2.117:0/3240318704 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1b69a0 0x7fec9c1b8db0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.912 7feca1bbd700 1 -- 172.21.2.117:0/3240318704 >> 172.21.2.117:0/3240318704 conn(0x7fec9c0cf060 msgr2=0x7fec9c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:40.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.912 7feca1bbd700 1 -- 172.21.2.117:0/3240318704 shutdown_connections 2021-02-10T11:57:40.931 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.912 7feca1bbd700 1 -- 172.21.2.117:0/3240318704 wait complete. 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.926 7feca1bbd700 1 Processor -- start 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.926 7feca1bbd700 1 -- start start 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1a7ed0 0x7fec9c08c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 0x7fec9c08cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c094280 con 0x7fec9c093be0 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c093fe0 con 0x7fec9c08c850 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c0ace00 con 0x7fec9c1a7ed0 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60722/0 (socket says 172.21.2.117:60722) 2021-02-10T11:57:40.933 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9b7fe700 1 -- 172.21.2.117:0/131587604 learned_addr learned my addr 172.21.2.117:0/131587604 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 0x7fec9c08cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9affd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1a7ed0 0x7fec9c08c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 -- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1a7ed0 msgr2=0x7fec9c08c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1a7ed0 0x7fec9c08c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 -- 172.21.2.117:0/131587604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 msgr2=0x7fec9c1a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 -- 172.21.2.117:0/131587604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c094800 con 0x7fec9c08c850 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9b7fe700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec9bfff700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 0x7fec9c08cc50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fec940069a0 tx=0x7fec9400bc60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec940093d0 con 0x7fec9c08c850 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9400bda0 con 0x7fec9c08c850 2021-02-10T11:57:40.934 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 -- 172.21.2.117:0/131587604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c094800 con 0x7fec9c08c850 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fec9400c570 con 0x7fec9c08c850 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.927 7feca1bbd700 1 -- 172.21.2.117:0/131587604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c094800 con 0x7fec9c08c850 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.928 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fec940093d0 con 0x7fec9c08c850 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.928 7fec93fff700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 0x7fec8001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.928 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fec94032940 con 0x7fec9c08c850 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.928 7fec9b7fe700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 0x7fec8001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:40.935 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.929 7fec9b7fe700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 0x7fec8001abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:40.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.960 7feca1bbd700 1 -- 172.21.2.117:0/131587604 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c0ede60 con 0x7fec9c08c850 2021-02-10T11:57:40.966 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:40.963 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fec94024f70 con 0x7fec9c08c850 2021-02-10T11:57:41.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.094 7feca1bbd700 1 -- 172.21.2.117:0/131587604 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolumegroup rm", "vol_name": "cephfs", "target": ["mgr", ""], "group_name": "group_0000000000305114"}) v1 -- 0x7fec9c093fe0 con 0x7fec800186c0 2021-02-10T11:57:41.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.099 7fec93fff700 1 -- 172.21.2.117:0/131587604 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fec84000bc0 con 0x7fec800186c0 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 msgr2=0x7fec8001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 0x7fec8001abc0 crc :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 msgr2=0x7fec9c08cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 0x7fec9c08cc50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 rx=0x7fec940069a0 tx=0x7fec9400bc60).stop 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 shutdown_connections 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fec9c08c850 0x7fec9c08cc50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fec9c1a7ed0 0x7fec9c08c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fec800186c0 0x7fec8001abc0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:41.107 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 --2- 172.21.2.117:0/131587604 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fec9c093be0 0x7fec9c1a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:41.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 >> 172.21.2.117:0/131587604 conn(0x7fec9c092d00 msgr2=0x7fec9c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:41.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 shutdown_connections 2021-02-10T11:57:41.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:41.104 7feca1bbd700 1 -- 172.21.2.117:0/131587604 wait complete. 2021-02-10T11:57:41.115 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:57:41.116 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:57:41.461 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:57:41.806 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:57:41.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:41.807 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:41.807 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:42.144 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:57:42.144 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:57:42.144 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:57:42.144 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:57:42.144 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:42.497 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[192810]: fuse finished with error 0 and tester_r 0 2021-02-10T11:57:42.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:42.499 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:42.499 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:42.834 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:57:42.835 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:57:42.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:57:42.835 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:43.171 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:43.171 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:43.171 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:43.507 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:57:43.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:57:43.507 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:57:43.507 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:57:43.507 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:43.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[193450]: fuse finished with error 0 and tester_r 0 2021-02-10T11:57:43.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:43.864 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:43.864 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:44.199 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:57:44.199 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:57:44.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:57:44.200 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:44.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:44.536 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:44.537 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:57:44.870 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:57:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:44.870 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:57:44.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:57:44.871 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:57:45.205 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:45.206 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:45.206 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:57:45.539 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:57:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:45.539 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:57:45.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:57:45.540 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:57:45.873 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 Processor -- start 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 -- start start 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9aa40d15e0 0x7f9aa40e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40e7580 con 0x7f9aa41b07f0 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40e77f0 con 0x7f9aa40e7180 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.287 7f9aaa4e6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa40e5ce0 con 0x7f9aa40d15e0 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa8b03700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f9aa40d15e0 0x7f9aa40e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa3fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9aa41b07f0 0x7f9aa40e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:60980/0 (socket says 172.21.2.117:60980) 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa3fff700 1 -- 172.21.2.117:0/3594396848 learned_addr learned my addr 172.21.2.117:0/3594396848 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa8b03700 1 -- 172.21.2.117:0/3594396848 >> v1:172.21.2.125:6789/0 conn(0x7f9aa40e7180 legacy=0x7f9aa40e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa8b03700 1 -- 172.21.2.117:0/3594396848 >> v1:172.21.2.117:6789/0 conn(0x7f9aa41b07f0 legacy=0x7f9aa40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa8b03700 1 -- 172.21.2.117:0/3594396848 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40d19e0 con 0x7f9aa40d15e0 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.288 7f9aa8b03700 1 --2- 172.21.2.117:0/3594396848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa40d15e0 0x7f9aa40e8560 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f9a9c006e30 tx=0x7f9a9c009e60).ready entity=mon.0 client_cookie=15a011ada9513c7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aa2ffd700 1 -- 172.21.2.117:0/3594396848 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a9c006a50 con 0x7f9aa40d15e0 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aa2ffd700 1 -- 172.21.2.117:0/3594396848 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00b300 con 0x7f9aa40d15e0 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aa2ffd700 1 -- 172.21.2.117:0/3594396848 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a9c00b7e0 con 0x7f9aa40d15e0 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aaa4e6700 1 -- 172.21.2.117:0/3594396848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa40d15e0 msgr2=0x7f9aa40e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aaa4e6700 1 --2- 172.21.2.117:0/3594396848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa40d15e0 0x7f9aa40e8560 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 rx=0x7f9a9c006e30 tx=0x7f9a9c009e60).stop 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aaa4e6700 1 -- 172.21.2.117:0/3594396848 shutdown_connections 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aaa4e6700 1 --2- 172.21.2.117:0/3594396848 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa40d15e0 0x7f9aa40e8560 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.289 7f9aaa4e6700 1 -- 172.21.2.117:0/3594396848 >> 172.21.2.117:0/3594396848 conn(0x7f9aa40ce500 msgr2=0x7f9aa40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:46.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.290 7f9aaa4e6700 1 -- 172.21.2.117:0/3594396848 shutdown_connections 2021-02-10T11:57:46.309 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.290 7f9aaa4e6700 1 -- 172.21.2.117:0/3594396848 wait complete. 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.304 7f9aaa4e6700 1 Processor -- start 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.304 7f9aaa4e6700 1 -- start start 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 0x7f9aa40bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 0x7f9aa40969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41c59d0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa408de90 con 0x7f9aa40bb840 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aaa4e6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4096fc0 con 0x7f9aa41c55d0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aa37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aa37fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60738/0 (socket says 172.21.2.117:60738) 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.305 7f9aa37fe700 1 -- 172.21.2.117:0/2449451246 learned_addr learned my addr 172.21.2.117:0/2449451246 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 -- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 msgr2=0x7f9aa40969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa8b03700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 0x7f9aa40969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa3fff700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 0x7f9aa40bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 0x7f9aa40969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 -- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 msgr2=0x7f9aa40bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 0x7f9aa40bbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 -- 172.21.2.117:0/2449451246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40971e0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa8b03700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 0x7f9aa40969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa3fff700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 0x7f9aa40bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa37fe700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f9a9800b5b0 tx=0x7f9a9800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a980009b0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9800bd40 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9a98006210 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.306 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa40971e0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.307 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa40971e0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.307 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9a98006230 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.307 7f9aa0ff9700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 0x7f9a8401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.307 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f9a98032950 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.308 7f9aa3fff700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 0x7f9a8401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.313 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.308 7f9aa3fff700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 0x7f9a8401abd0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.337 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa41c59d0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.340 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9a98006000 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.491 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.488 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9aa41c59d0 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.495 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.492 7f9aa0ff9700 1 -- 172.21.2.117:0/2449451246 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9a98006210 con 0x7f9aa41c4bb0 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 msgr2=0x7f9a8401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 0x7f9a8401abd0 crc :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 msgr2=0x7f9aa41c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f9a9800b5b0 tx=0x7f9a9800b950).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 shutdown_connections 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9a840186d0 0x7f9a8401abd0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9aa41c4bb0 0x7f9aa41c4fb0 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9aa41c55d0 0x7f9aa40969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 --2- 172.21.2.117:0/2449451246 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9aa40bb840 0x7f9aa40bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.496 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 >> 172.21.2.117:0/2449451246 conn(0x7f9aa407cb30 msgr2=0x7f9aa40bc590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.497 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 shutdown_connections 2021-02-10T11:57:46.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.497 7f9aaa4e6700 1 -- 172.21.2.117:0/2449451246 wait complete. 2021-02-10T11:57:46.508 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list' 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.917 7f38c8296700 1 Processor -- start 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.917 7f38c8296700 1 -- start start 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c8296700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00d18e0 0x7f38c00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c8296700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c00ef3f0 0x7f38c01b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c8296700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c00ede60 con 0x7f38c00eb000 2021-02-10T11:57:46.937 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c8296700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c0088020 con 0x7f38c00ef3f0 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c8296700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00d1ce0 con 0x7f38c00d18e0 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c00ef3f0 0x7f38c01b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f38c00ef3f0 0x7f38c01b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60744/0 (socket says 172.21.2.117:60744) 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 -- 172.21.2.117:0/3684222424 learned_addr learned my addr 172.21.2.117:0/3684222424 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 -- 172.21.2.117:0/3684222424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00d18e0 msgr2=0x7f38c00ea9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 --2- 172.21.2.117:0/3684222424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00d18e0 0x7f38c00ea9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 -- 172.21.2.117:0/3684222424 >> v1:172.21.2.125:6789/0 conn(0x7f38c00eb000 legacy=0x7f38c00eb400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.918 7f38c68b3700 1 -- 172.21.2.117:0/3684222424 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c01b8f90 con 0x7f38c00ef3f0 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c68b3700 1 --2- 172.21.2.117:0/3684222424 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00ef3f0 0x7f38c01b7960 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 rx=0x7f38b8006f00 tx=0x7f38b8009f10).ready entity=mon.0 client_cookie=9399c5435a66f2f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c50b0700 1 -- 172.21.2.117:0/3684222424 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38b800b3b0 con 0x7f38c00ef3f0 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c50b0700 1 -- 172.21.2.117:0/3684222424 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b800b630 con 0x7f38c00ef3f0 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c50b0700 1 -- 172.21.2.117:0/3684222424 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38b800bb50 con 0x7f38c00ef3f0 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c8296700 1 -- 172.21.2.117:0/3684222424 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00ef3f0 msgr2=0x7f38c01b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.919 7f38c8296700 1 --2- 172.21.2.117:0/3684222424 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00ef3f0 0x7f38c01b7960 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 rx=0x7f38b8006f00 tx=0x7f38b8009f10).stop 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 -- 172.21.2.117:0/3684222424 shutdown_connections 2021-02-10T11:57:46.939 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 --2- 172.21.2.117:0/3684222424 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c00ef3f0 0x7f38c01b7960 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:46.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 --2- 172.21.2.117:0/3684222424 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00d18e0 0x7f38c00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:46.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 -- 172.21.2.117:0/3684222424 >> 172.21.2.117:0/3684222424 conn(0x7f38c00ced60 msgr2=0x7f38c00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:46.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 -- 172.21.2.117:0/3684222424 shutdown_connections 2021-02-10T11:57:46.940 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.920 7f38c8296700 1 -- 172.21.2.117:0/3684222424 wait complete. 2021-02-10T11:57:46.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.934 7f38c8296700 1 Processor -- start 2021-02-10T11:57:46.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.934 7f38c8296700 1 -- start start 2021-02-10T11:57:46.941 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00e9560 0x7f38c00e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00e9960 0x7f38c00e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0196810 con 0x7f38c0196390 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0098ad0 con 0x7f38c00e9960 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c8296700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00e8a40 con 0x7f38c00e9560 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60750/0 (socket says 172.21.2.117:60750) 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 -- 172.21.2.117:0/1459953323 learned_addr learned my addr 172.21.2.117:0/1459953323 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 -- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00e9560 msgr2=0x7f38c00e7e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00e9560 0x7f38c00e7e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:46.942 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 -- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00e9960 msgr2=0x7f38c00e8420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00e9960 0x7f38c00e8420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 -- 172.21.2.117:0/1459953323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c00e8c60 con 0x7f38c0196390 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.935 7f38c60b2700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7f38bc00b5c0 tx=0x7f38bc00b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38bc006b80 con 0x7f38c0196390 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc00bd50 con 0x7f38c0196390 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38c8296700 1 -- 172.21.2.117:0/1459953323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c00e8c60 con 0x7f38c0196390 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38c8296700 1 -- 172.21.2.117:0/1459953323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c019cd70 con 0x7f38c0196390 2021-02-10T11:57:46.943 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f38bc0054d0 con 0x7f38c0196390 2021-02-10T11:57:46.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.936 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f38bc006030 con 0x7f38c0196390 2021-02-10T11:57:46.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.937 7f38b6ffd700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 0x7f38a801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:46.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.937 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f38bc032940 con 0x7f38c0196390 2021-02-10T11:57:46.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.937 7f38c58b1700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 0x7f38a801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:46.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.937 7f38c58b1700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 0x7f38a801abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:46.971 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.968 7f38c8296700 1 -- 172.21.2.117:0/1459953323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c0087210 con 0x7f38c0196390 2021-02-10T11:57:46.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:46.970 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f38bc000990 con 0x7f38c0196390 2021-02-10T11:57:47.106 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.103 7f38c8296700 1 -- 172.21.2.117:0/1459953323 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]} v 0) v1 -- 0x7f38c00ede60 con 0x7f38c0196390 2021-02-10T11:57:47.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.547 7f38b6ffd700 1 -- 172.21.2.117:0/1459953323 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_authorized_list"]}]=0 v915) v1 ==== 147+0+0 (secure 0 0 0) 0x7f38bc006030 con 0x7f38c0196390 2021-02-10T11:57:47.574 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.571 7f38c8296700 1 -- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 msgr2=0x7f38a801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:47.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.571 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 0x7f38a801abc0 crc :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:47.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.571 7f38c8296700 1 -- 172.21.2.117:0/1459953323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 msgr2=0x7f38c00e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:47.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.571 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 rx=0x7f38bc00b5c0 tx=0x7f38bc00b960).stop 2021-02-10T11:57:47.575 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 -- 172.21.2.117:0/1459953323 shutdown_connections 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f38c00e9960 0x7f38c00e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f38c00e9560 0x7f38c00e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f38a8018730 0x7f38a801abc0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 --2- 172.21.2.117:0/1459953323 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f38c0196390 0x7f38c00e8f40 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 -- 172.21.2.117:0/1459953323 >> 172.21.2.117:0/1459953323 conn(0x7f38c019f610 msgr2=0x7f38c01a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:47.576 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 -- 172.21.2.117:0/1459953323 shutdown_connections 2021-02-10T11:57:47.577 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.572 7f38c8296700 1 -- 172.21.2.117:0/1459953323 wait complete. 2021-02-10T11:57:47.583 INFO:tasks.cephfs_test_runner:test_subvolume_authorized_list (tasks.cephfs.test_volumes.TestVolumes) ... ok 2021-02-10T11:57:47.583 INFO:tasks.cephfs_test_runner:Starting test: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:57:47.584 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 Processor -- start 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 -- start start 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbdb80e4e40 0x7fbdb81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 0x7fbdb8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fbdb81b0810 0x7fbdb8085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb80e5cc0 con 0x7fbdb80e4e40 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb8085930 con 0x7fbdb81b0810 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.993 7fbdbf6d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8084840 con 0x7fbdb80e5240 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbdb80e4e40 0x7fbdb81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:48.013 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fbdb80e4e40 0x7fbdb81b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60756/0 (socket says 172.21.2.117:60756) 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 -- 172.21.2.117:0/3481403081 learned_addr learned my addr 172.21.2.117:0/3481403081 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbdced700 1 --2- 172.21.2.117:0/3481403081 >> v2:172.21.2.125:3300/0 conn(0x7fbdb81b0810 0x7fbdb8085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbcceb700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 0x7fbdb8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 -- 172.21.2.117:0/3481403081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 msgr2=0x7fbdb8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 0x7fbdb8084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 -- 172.21.2.117:0/3481403081 >> v2:172.21.2.125:3300/0 conn(0x7fbdb81b0810 msgr2=0x7fbdb8085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 --2- 172.21.2.117:0/3481403081 >> v2:172.21.2.125:3300/0 conn(0x7fbdb81b0810 0x7fbdb8085280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 -- 172.21.2.117:0/3481403081 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8085eb0 con 0x7fbdb80e4e40 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbcceb700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 0x7fbdb8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdbd4ec700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80e4e40 0x7fbdb81b01f0 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7fbdac006700 tx=0x7fbdac009b80).ready entity=mon.0 client_cookie=e29e8f40a52724bc server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.994 7fbdabfff700 1 -- 172.21.2.117:0/3481403081 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdac00c0c0 con 0x7fbdb80e4e40 2021-02-10T11:57:48.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdabfff700 1 -- 172.21.2.117:0/3481403081 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac00c380 con 0x7fbdb80e4e40 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdabfff700 1 -- 172.21.2.117:0/3481403081 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdac00c8a0 con 0x7fbdb80e4e40 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 -- 172.21.2.117:0/3481403081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80e4e40 msgr2=0x7fbdb81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80e4e40 0x7fbdb81b01f0 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 rx=0x7fbdac006700 tx=0x7fbdac009b80).stop 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 -- 172.21.2.117:0/3481403081 shutdown_connections 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 --2- 172.21.2.117:0/3481403081 >> v2:172.21.2.125:3300/0 conn(0x7fbdb81b0810 0x7fbdb8085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80e5240 0x7fbdb8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 --2- 172.21.2.117:0/3481403081 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80e4e40 0x7fbdb81b01f0 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.995 7fbdbf6d0700 1 -- 172.21.2.117:0/3481403081 >> 172.21.2.117:0/3481403081 conn(0x7fbdb80ce4e0 msgr2=0x7fbdb80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.996 7fbdbf6d0700 1 -- 172.21.2.117:0/3481403081 shutdown_connections 2021-02-10T11:57:48.015 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:47.996 7fbdbf6d0700 1 -- 172.21.2.117:0/3481403081 wait complete. 2021-02-10T11:57:48.016 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.010 7fbdbf6d0700 1 Processor -- start 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.010 7fbdbf6d0700 1 -- start start 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80a2010 0x7fbdb8092e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbdb80ade00 0x7fbdb80ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb80934a0 con 0x7fbdb80ae820 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb80a2410 con 0x7fbdb80ade00 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb80a2630 con 0x7fbdb80a2010 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60758/0 (socket says 172.21.2.117:60758) 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 -- 172.21.2.117:0/330139415 learned_addr learned my addr 172.21.2.117:0/330139415 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 -- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80a2010 msgr2=0x7fbdb8092e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:48.017 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80a2010 0x7fbdb8092e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 -- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbdb80ade00 msgr2=0x7fbdb80ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbdb80ade00 0x7fbdb80ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 -- 172.21.2.117:0/330139415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb80aec20 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbdced700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7fbdb00099a0 tx=0x7fbdb000bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdb00066f0 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb0003030 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fbdb00013e0 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb80aec20 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.011 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb80aec20 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.012 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fbdb00011f0 con 0x7fbdb80ae820 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.012 7fbda9ffb700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 0x7fbd9801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:48.018 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.012 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fbdb0032c60 con 0x7fbdb80ae820 2021-02-10T11:57:48.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.012 7fbdbd4ec700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 0x7fbd9801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:48.019 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.013 7fbdbd4ec700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 0x7fbd9801abd0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:48.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.043 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb8085930 con 0x7fbdb80ae820 2021-02-10T11:57:48.049 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.046 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fbdb000be30 con 0x7fbdb80ae820 2021-02-10T11:57:48.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.176 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7fbdb8085930 con 0x7fbdb80ae820 2021-02-10T11:57:48.555 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.552 7fbda9ffb700 1 -- 172.21.2.117:0/330139415 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v916) v1 ==== 160+0+0 (secure 0 0 0) 0x7fbdb001f850 con 0x7fbdb80ae820 2021-02-10T11:57:48.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 msgr2=0x7fbd9801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 0x7fbd9801abd0 crc :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 msgr2=0x7fbdb80a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 rx=0x7fbdb00099a0 tx=0x7fbdb000bcf0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 shutdown_connections 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fbd980186d0 0x7fbd9801abd0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fbdb80ae820 0x7fbdb80a2db0 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fbdb80ade00 0x7fbdb80ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 --2- 172.21.2.117:0/330139415 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fbdb80a2010 0x7fbdb8092e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.595 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 >> 172.21.2.117:0/330139415 conn(0x7fbdb80b5bd0 msgr2=0x7fbdb80c39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.596 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 shutdown_connections 2021-02-10T11:57:48.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:48.596 7fbdbf6d0700 1 -- 172.21.2.117:0/330139415 wait complete. 2021-02-10T11:57:48.607 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:57:49.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.020 7f5eeb058700 1 Processor -- start 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.020 7f5eeb058700 1 -- start start 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5eeb058700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 0x7f5ee40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5eeb058700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ee40eb830 0x7f5ee41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5eeb058700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee40ef7f0 con 0x7f5ee40d18f0 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5eeb058700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee40ede60 con 0x7f5ee40eb830 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5eeb058700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee40d1cf0 con 0x7f5ee40ef3f0 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5ee9675700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f5ee40eb830 0x7f5ee41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5ee8e74700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5ee40d18f0 0x7f5ee40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32784/0 (socket says 172.21.2.117:32784) 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5ee8e74700 1 -- 172.21.2.117:0/3891839345 learned_addr learned my addr 172.21.2.117:0/3891839345 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.021 7f5ee3fff700 1 --2- 172.21.2.117:0/3891839345 >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 0x7f5ee40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee9675700 1 -- 172.21.2.117:0/3891839345 >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 msgr2=0x7f5ee40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:49.041 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee9675700 1 --2- 172.21.2.117:0/3891839345 >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 0x7f5ee40eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee9675700 1 -- 172.21.2.117:0/3891839345 >> v1:172.21.2.117:6789/0 conn(0x7f5ee40d18f0 legacy=0x7f5ee40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee9675700 1 -- 172.21.2.117:0/3891839345 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee41b8fd0 con 0x7f5ee40eb830 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee3fff700 1 --2- 172.21.2.117:0/3891839345 >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 0x7f5ee40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee9675700 1 --2- 172.21.2.117:0/3891839345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee40eb830 0x7f5ee41b79a0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 rx=0x7f5edc006ab0 tx=0x7f5edc009b40).ready entity=mon.0 client_cookie=2e136f51439552f9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee37fe700 1 -- 172.21.2.117:0/3891839345 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5edc0022e0 con 0x7f5ee40eb830 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee37fe700 1 -- 172.21.2.117:0/3891839345 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5edc002660 con 0x7f5ee40eb830 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5ee37fe700 1 -- 172.21.2.117:0/3891839345 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5edc002a70 con 0x7f5ee40eb830 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5eeb058700 1 -- 172.21.2.117:0/3891839345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee40eb830 msgr2=0x7f5ee41b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.022 7f5eeb058700 1 --2- 172.21.2.117:0/3891839345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee40eb830 0x7f5ee41b79a0 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 rx=0x7f5edc006ab0 tx=0x7f5edc009b40).stop 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 -- 172.21.2.117:0/3891839345 shutdown_connections 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 --2- 172.21.2.117:0/3891839345 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee40eb830 0x7f5ee41b79a0 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.042 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 --2- 172.21.2.117:0/3891839345 >> v2:172.21.2.125:3300/0 conn(0x7f5ee40ef3f0 0x7f5ee40eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 -- 172.21.2.117:0/3891839345 >> 172.21.2.117:0/3891839345 conn(0x7f5ee40cf060 msgr2=0x7f5ee40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 -- 172.21.2.117:0/3891839345 shutdown_connections 2021-02-10T11:57:49.043 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.023 7f5eeb058700 1 -- 172.21.2.117:0/3891839345 wait complete. 2021-02-10T11:57:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 Processor -- start 2021-02-10T11:57:49.044 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 -- start start 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee4093be0 0x7f5ee41a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 0x7f5ee408c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee4094280 con 0x7f5ee408c850 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee4093fe0 con 0x7f5ee41a7ed0 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5eeb058700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee40ace00 con 0x7f5ee4093be0 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5ee3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 0x7f5ee408c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.038 7f5ee9675700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60776/0 (socket says 172.21.2.117:60776) 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 -- 172.21.2.117:0/4208990174 learned_addr learned my addr 172.21.2.117:0/4208990174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee3fff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 0x7f5ee408c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36924/0 (socket says 172.21.2.117:36924) 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee8e74700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee4093be0 0x7f5ee41a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.045 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 -- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee4093be0 msgr2=0x7f5ee41a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee4093be0 0x7f5ee41a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 -- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 msgr2=0x7f5ee408c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 0x7f5ee408c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 -- 172.21.2.117:0/4208990174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4094800 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee9675700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f5edc00b5e0 tx=0x7f5edc00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5edc002280 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5edc009070 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5edc0027c0 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee4094800 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.039 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee4094800 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.040 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5edc0054a0 con 0x7f5ee408c850 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.040 7f5ee17fa700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 0x7f5ec801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:49.046 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.040 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5edc032950 con 0x7f5ee408c850 2021-02-10T11:57:49.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.040 7f5ee8e74700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 0x7f5ec801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:49.047 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.041 7f5ee8e74700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 0x7f5ec801abc0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:49.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.071 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee40ede60 con 0x7f5ee408c850 2021-02-10T11:57:49.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.073 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5edc01f540 con 0x7f5ee408c850 2021-02-10T11:57:49.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.223 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5ee40ede60 con 0x7f5ee408c850 2021-02-10T11:57:49.527 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:57:49.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.227 7f5ee17fa700 1 -- 172.21.2.117:0/4208990174 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5edc0054a0 con 0x7f5ee408c850 2021-02-10T11:57:49.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 msgr2=0x7f5ec801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 0x7f5ec801abc0 crc :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 msgr2=0x7f5ee408cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 rx=0x7f5edc00b5e0 tx=0x7f5edc00b980).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 shutdown_connections 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5ec80186c0 0x7f5ec801abc0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ee408c850 0x7f5ee408cc50 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ee41a7ed0 0x7f5ee408c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 --2- 172.21.2.117:0/4208990174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ee4093be0 0x7f5ee41a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 >> 172.21.2.117:0/4208990174 conn(0x7f5ee4092d00 msgr2=0x7f5ee408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 shutdown_connections 2021-02-10T11:57:49.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:49.232 7f5eeb058700 1 -- 172.21.2.117:0/4208990174 wait complete. 2021-02-10T11:57:49.529 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:57:49.529 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:57:49.531 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:57:49.877 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:57:50.223 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:57:50.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:50.223 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:50.223 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:57:50.560 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:57:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:50.561 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:57:50.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:50.561 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:50.561 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:57:50.897 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:57:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:50.898 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:57:50.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:50.898 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:50.898 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:57:51.234 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:57:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:51.234 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:57:51.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:57:51.234 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:57:51.235 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:57:51.569 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:57:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:57:51.569 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:57:51.570 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json-pretty 2021-02-10T11:57:52.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 Processor -- start 2021-02-10T11:57:52.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 -- start start 2021-02-10T11:57:52.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8280d18e0 0x7fa8280ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280eb000 0x7fa8280eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8280ede60 con 0x7fa8280ee3e0 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8280eba90 con 0x7fa8280d18e0 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82ff69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280d1ce0 con 0x7fa8280eb000 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82dd85700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8280d18e0 0x7fa8280ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82dd85700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa8280d18e0 0x7fa8280ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60780/0 (socket says 172.21.2.117:60780) 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82dd85700 1 -- 172.21.2.117:0/3352332998 learned_addr learned my addr 172.21.2.117:0/3352332998 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.981 7fa82d584700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280eb000 0x7fa8280eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82dd85700 1 -- 172.21.2.117:0/3352332998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280eb000 msgr2=0x7fa8280eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82dd85700 1 --2- 172.21.2.117:0/3352332998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280eb000 0x7fa8280eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82dd85700 1 -- 172.21.2.117:0/3352332998 >> v1:172.21.2.125:6789/0 conn(0x7fa8280ee3e0 legacy=0x7fa8280ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82dd85700 1 -- 172.21.2.117:0/3352332998 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280eefd0 con 0x7fa8280d18e0 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82dd85700 1 --2- 172.21.2.117:0/3352332998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280d18e0 0x7fa8280ea9e0 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 rx=0x7fa824006700 tx=0x7fa824009b80).ready entity=mon.0 client_cookie=f98c28f1db7b7e10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82cd83700 1 -- 172.21.2.117:0/3352332998 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa824006340 con 0x7fa8280d18e0 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82cd83700 1 -- 172.21.2.117:0/3352332998 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82400c0c0 con 0x7fa8280d18e0 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82cd83700 1 -- 172.21.2.117:0/3352332998 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa82400c5a0 con 0x7fa8280d18e0 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82ff69700 1 -- 172.21.2.117:0/3352332998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280d18e0 msgr2=0x7fa8280ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.982 7fa82ff69700 1 --2- 172.21.2.117:0/3352332998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280d18e0 0x7fa8280ea9e0 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 rx=0x7fa824006700 tx=0x7fa824009b80).stop 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 -- 172.21.2.117:0/3352332998 shutdown_connections 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 --2- 172.21.2.117:0/3352332998 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280eb000 0x7fa8280eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 --2- 172.21.2.117:0/3352332998 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280d18e0 0x7fa8280ea9e0 unknown :-1 s=CLOSED pgs=3192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 -- 172.21.2.117:0/3352332998 >> 172.21.2.117:0/3352332998 conn(0x7fa8280ced60 msgr2=0x7fa8280cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 -- 172.21.2.117:0/3352332998 shutdown_connections 2021-02-10T11:57:52.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.983 7fa82ff69700 1 -- 172.21.2.117:0/3352332998 wait complete. 2021-02-10T11:57:52.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.997 7fa82ff69700 1 Processor -- start 2021-02-10T11:57:52.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 -- start start 2021-02-10T11:57:52.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 0x7fa8280c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 0x7fa8280c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 0x7fa8280967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828095e70 con 0x7fa8280963f0 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280e86d0 con 0x7fa8280959c0 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82ff69700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa828096e10 con 0x7fa8280c2910 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82dd85700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 0x7fa8280c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 0x7fa8280c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82dd85700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 0x7fa8280c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36938/0 (socket says 172.21.2.117:36938) 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82dd85700 1 -- 172.21.2.117:0/3703876399 learned_addr learned my addr 172.21.2.117:0/3703876399 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82e586700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 0x7fa8280967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 -- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 msgr2=0x7fa8280c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 0x7fa8280c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 -- 172.21.2.117:0/3703876399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 msgr2=0x7fa8280967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 0x7fa8280967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.998 7fa82d584700 1 -- 172.21.2.117:0/3703876399 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828097030 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa82e586700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 0x7fa8280967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa82d584700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 0x7fa8280c2d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa818009540 tx=0x7fa81800b8e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa8180009b0 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818006340 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa818002030 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa828097030 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa828097030 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:51.999 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa81800a000 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.000 7fa81e7fc700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.000 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa818032600 con 0x7fa8280c2910 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.000 7fa82dd85700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.000 7fa82dd85700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.031 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828087290 con 0x7fa8280c2910 2021-02-10T11:57:52.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.034 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa818006ae0 con 0x7fa8280c2910 2021-02-10T11:57:52.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.157 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa8280ede60 con 0x7fa8280c2910 2021-02-10T11:57:52.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.159 7fa81e7fc700 1 -- 172.21.2.117:0/3703876399 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v187) v1 ==== 81+0+30766 (secure 0 0 0) 0x7fa818000e20 con 0x7fa8280c2910 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "epoch": 187, 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "created": "2021-02-10 11:21:47.530737", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "modified": "2021-02-10 11:57:10.511294", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "last_up_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "last_in_change": "2021-02-10 11:22:18.312455", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_num": 5799936, 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_set": [ 2021-02-10T11:57:52.164 INFO:teuthology.orchestra.run.gibba017.stdout: "pglog_hardlimit", 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "purged_snapdirs", 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_deletes", 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "sortbitwise" 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_version": 5, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "full_ratio": 0.94999998807907104, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "nearfull_ratio": 0.85000002384185791, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_snapshot": "", 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_max": 41, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "max_osd": 8, 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "require_min_compat_client": "jewel", 2021-02-10T11:57:52.165 INFO:teuthology.orchestra.run.gibba017.stdout: "min_compat_client": "jewel", 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "require_osd_release": "nautilus", 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "pools": [ 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 1, 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "rbd", 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:22:23.728210", 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:57:52.166 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:57:52.167 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "16", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:57:52.168 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:57:52.169 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:57:52.170 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "rbd": {} 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 40, 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_metadata", 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:57:07.475412", 2021-02-10T11:57:52.171 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:57:52.172 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.173 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "186", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:57:52.174 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:57:52.175 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:57:52.176 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "options": { 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_bias": 4, 2021-02-10T11:57:52.177 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_min": 16, 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "recovery_priority": 5 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": "cephfs" 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "pool": 41, 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_name": "cephfs_data", 2021-02-10T11:57:52.178 INFO:teuthology.orchestra.run.gibba017.stdout: "create_time": "2021-02-10 11:57:08.504122", 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "flags": 1, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "flags_names": "hashpspool", 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "type": 1, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "size": 2, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "min_size": 1, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_rule": 0, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "object_hash": 2, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_autoscale_mode": "warn", 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num": 8, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num": 8, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_placement_num_target": 8, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_target": 8, 2021-02-10T11:57:52.179 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_num_pending": 8, 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "last_pg_merge_meta": { 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "source_pgid": "0.0", 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "ready_epoch": 0, 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_started": 0, 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "last_epoch_clean": 0, 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "source_version": "0'0", 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "target_version": "0'0" 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.180 INFO:teuthology.orchestra.run.gibba017.stdout: "last_change": "187", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend": "0", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_prenautilus": "0", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "last_force_op_resend_preluminous": "0", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "auid": 0, 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_mode": "selfmanaged", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_seq": 0, 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "snap_epoch": 0, 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "pool_snaps": [], 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps": "[]", 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_bytes": 0, 2021-02-10T11:57:52.181 INFO:teuthology.orchestra.run.gibba017.stdout: "quota_max_objects": 0, 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "tiers": [], 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "tier_of": -1, 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "read_tier": -1, 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "write_tier": -1, 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_mode": "none", 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_bytes": 0, 2021-02-10T11:57:52.182 INFO:teuthology.orchestra.run.gibba017.stdout: "target_max_objects": 0, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_target_full_ratio_micro": 800000, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_flush_age": 0, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "cache_min_evict_age": 0, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profile": "", 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_params": { 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "none" 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_period": 0, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_count": 0, 2021-02-10T11:57:52.183 INFO:teuthology.orchestra.run.gibba017.stdout: "use_gmt_hitset": true, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "min_read_recency_for_promote": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "min_write_recency_for_promote": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_grade_decay_rate": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "hit_set_search_last_n": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "grade_table": [], 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "stripe_width": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "expected_num_objects": 0, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "fast_read": false, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "options": {}, 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "application_metadata": { 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs": { 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: "data": "cephfs" 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.184 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "osds": [ 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "b2b85857-0d25-4760-8348-b4e0feb34706", 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.185 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 184, 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6805", 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.186 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6808", 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6811", 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.187 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6814", 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6822", 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.188 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6823", 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6817", 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.189 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6820", 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36018 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6808/36018", 2021-02-10T11:57:52.190 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6814/36018", 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6823/36018", 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6820/36018", 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.191 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "4bb369d2-72db-4361-97af-cd8850006a87", 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.192 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 184, 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6800", 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.193 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6802", 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6804", 2021-02-10T11:57:52.194 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6807", 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.195 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6816", 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6819", 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.196 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6810", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6813", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36019 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6802/36019", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6807/36019", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6819/36019", 2021-02-10T11:57:52.197 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6813/36019", 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "63367d7e-7197-4d2e-90b2-2b5cea7b4304", 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.198 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 185, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6801", 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.199 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6803", 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6806", 2021-02-10T11:57:52.200 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6809", 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6818", 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.201 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6821", 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.202 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6812", 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6815", 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36020 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6803/36020", 2021-02-10T11:57:52.203 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6809/36020", 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6821/36020", 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6815/36020", 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "2c9e4d21-9bfc-4107-ba32-853e096e8eb9", 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.204 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 185, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6824", 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.205 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6825", 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6826", 2021-02-10T11:57:52.206 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6827", 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.207 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6830", 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6831", 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.208 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6828", 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:6829", 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36021 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.209 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.117:6825/36021", 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.117:6827/36021", 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.117:6831/36021", 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.117:6829/36021", 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.210 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "be969e8d-b93d-43e0-811f-7fb5288876a7", 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 185, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.211 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6803", 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6806", 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.212 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6809", 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6813", 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.213 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.214 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6823", 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6827", 2021-02-10T11:57:52.215 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6817", 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.216 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6819", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36390 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6806/36390", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6813/36390", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6827/36390", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6819/36390", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.217 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "931344cc-ab1d-4eee-b067-08b04f0d8824", 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 177, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.218 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6812", 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6816", 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.219 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6820", 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6824", 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.220 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6832", 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6833", 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.221 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6828", 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.222 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6830", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36389 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6816/36389", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6824/36389", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6833/36389", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6830/36389", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.223 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "f0852634-0814-469a-9e5a-ae92485dd59e", 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 185, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.224 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6804", 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6807", 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.225 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6811", 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.226 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6815", 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6829", 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.227 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6831", 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6821", 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.228 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6826", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36388 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6807/36388", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6815/36388", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6831/36388", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6826/36388", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.229 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "uuid": "20bf9222-b6e5-4644-930f-5caff0fc379a", 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "up": 1, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "in": 1, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "weight": 1, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_affinity": 1, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_begin": 0, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "last_clean_end": 0, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "up_from": 13, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "up_thru": 185, 2021-02-10T11:57:52.230 INFO:teuthology.orchestra.run.gibba017.stdout: "down_at": 0, 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "lost_at": 0, 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addrs": { 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6802", 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.231 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6805", 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addrs": { 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6808", 2021-02-10T11:57:52.232 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6810", 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addrs": { 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.233 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6822", 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6825", 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addrs": { 2021-02-10T11:57:52.234 INFO:teuthology.orchestra.run.gibba017.stdout: "addrvec": [ 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v2", 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6814", 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.125:6818", 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 36387 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.235 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "public_addr": "172.21.2.125:6805/36387", 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "cluster_addr": "172.21.2.125:6810/36387", 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_back_addr": "172.21.2.125:6825/36387", 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "heartbeat_front_addr": "172.21.2.125:6818/36387", 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "state": [ 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "exists", 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "up" 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_xinfo": [ 2021-02-10T11:57:52.236 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 0, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 1, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.237 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 2, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.238 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 3, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 4, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.239 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 5, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 6, 2021-02-10T11:57:52.240 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": 7, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "down_stamp": "0.000000", 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_probability": 0, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "laggy_interval": 0, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "features": 4611087854035861503, 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: "old_weight": 0 2021-02-10T11:57:52.241 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: ], 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap": [], 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_upmap_items": [], 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "pg_temp": [], 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "primary_temp": [], 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklist": {}, 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "erasure_code_profiles": { 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "cephfs_data": { 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-device-class": "", 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-root": "default", 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "jerasure-per-chunk-alignment": "false", 2021-02-10T11:57:52.242 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "2", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "w": "8" 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "default": { 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "crush-failure-domain": "osd", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "k": "2", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "m": "1", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "plugin": "jerasure", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "ruleset-failure-domain": "osd", 2021-02-10T11:57:52.243 INFO:teuthology.orchestra.run.gibba017.stdout: "technique": "reed_sol_van" 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: "removed_snaps_queue": [], 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: "new_removed_snaps": [], 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: "new_purged_snaps": [], 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: "crush_node_flags": {}, 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout: "device_class_flags": {} 2021-02-10T11:57:52.244 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:57:52.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 msgr2=0x7fa81001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 crc :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 msgr2=0x7fa8280c2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 0x7fa8280c2d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa818009540 tx=0x7fa81800b8e0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 shutdown_connections 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa8100186c0 0x7fa81001abc0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa8280963f0 0x7fa8280967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa8280c2910 0x7fa8280c2d10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 --2- 172.21.2.117:0/3703876399 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa8280959c0 0x7fa8280c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 >> 172.21.2.117:0/3703876399 conn(0x7fa8280e8c80 msgr2=0x7fa828092070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 shutdown_connections 2021-02-10T11:57:52.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.162 7fa82ff69700 1 -- 172.21.2.117:0/3703876399 wait complete. 2021-02-10T11:57:52.247 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:52.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.657 7fb20f84c700 1 Processor -- start 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.657 7fb20f84c700 1 -- start start 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20f84c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2080e4e40 0x7fb2081b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20f84c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fb208085450 0x7fb208086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20f84c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208085930 con 0x7fb2080e5240 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20f84c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2080e5cc0 con 0x7fb2080e4e40 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20f84c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb208084840 con 0x7fb208085450 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2080e4e40 0x7fb2081b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb2080e4e40 0x7fb2081b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60794/0 (socket says 172.21.2.117:60794) 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 -- 172.21.2.117:0/970380329 learned_addr learned my addr 172.21.2.117:0/970380329 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 -- 172.21.2.117:0/970380329 >> v2:172.21.2.125:3300/0 conn(0x7fb208085450 msgr2=0x7fb208086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:52.677 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 --2- 172.21.2.117:0/970380329 >> v2:172.21.2.125:3300/0 conn(0x7fb208085450 0x7fb208086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 -- 172.21.2.117:0/970380329 >> v1:172.21.2.125:6789/0 conn(0x7fb2080e5240 legacy=0x7fb208084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 -- 172.21.2.117:0/970380329 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2081b0810 con 0x7fb2080e4e40 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.658 7fb20d668700 1 --2- 172.21.2.117:0/970380329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080e4e40 0x7fb2081b01f0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 rx=0x7fb1fc006700 tx=0x7fb1fc009b80).ready entity=mon.0 client_cookie=8efdbe8eadde5ee8 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb1fbfff700 1 -- 172.21.2.117:0/970380329 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1fc00c0c0 con 0x7fb2080e4e40 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb1fbfff700 1 -- 172.21.2.117:0/970380329 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1fc00c440 con 0x7fb2080e4e40 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb1fbfff700 1 -- 172.21.2.117:0/970380329 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb1fc00c960 con 0x7fb2080e4e40 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 -- 172.21.2.117:0/970380329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080e4e40 msgr2=0x7fb2081b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 --2- 172.21.2.117:0/970380329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080e4e40 0x7fb2081b01f0 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 rx=0x7fb1fc006700 tx=0x7fb1fc009b80).stop 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 -- 172.21.2.117:0/970380329 shutdown_connections 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 --2- 172.21.2.117:0/970380329 >> v2:172.21.2.125:3300/0 conn(0x7fb208085450 0x7fb208086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 --2- 172.21.2.117:0/970380329 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080e4e40 0x7fb2081b01f0 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.659 7fb20f84c700 1 -- 172.21.2.117:0/970380329 >> 172.21.2.117:0/970380329 conn(0x7fb2080ce4e0 msgr2=0x7fb2080e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:52.678 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.660 7fb20f84c700 1 -- 172.21.2.117:0/970380329 shutdown_connections 2021-02-10T11:57:52.679 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.660 7fb20f84c700 1 -- 172.21.2.117:0/970380329 wait complete. 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 Processor -- start 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 -- start start 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb20810e5c0 0x7fb20810dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb20810f130 0x7fb20810f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20810ec30 con 0x7fb2080b4520 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2081ad1e0 con 0x7fb20810f130 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.674 7fb20f84c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20810d740 con 0x7fb20810e5c0 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60800/0 (socket says 172.21.2.117:60800) 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 -- 172.21.2.117:0/4032167031 learned_addr learned my addr 172.21.2.117:0/4032167031 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 -- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb20810e5c0 msgr2=0x7fb20810dbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb20810e5c0 0x7fb20810dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 -- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb20810f130 msgr2=0x7fb20810f530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb20810f130 0x7fb20810f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 -- 172.21.2.117:0/4032167031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080a26b0 con 0x7fb2080b4520 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20ce67700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7fb20400b5b0 tx=0x7fb20400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb20400bd40 con 0x7fb2080b4520 2021-02-10T11:57:52.681 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb204000e20 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb2040062a0 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2080a26b0 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.675 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2080a26b0 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.676 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb204006020 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.676 7fb1f9ffb700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 0x7fb1e801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.676 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fb2040326f0 con 0x7fb2080b4520 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.677 7fb20d668700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 0x7fb1e801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:52.682 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.677 7fb20d668700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 0x7fb1e801abc0 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:52.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.707 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20810c250 con 0x7fb2080b4520 2021-02-10T11:57:52.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.710 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb204030d50 con 0x7fb2080b4520 2021-02-10T11:57:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.838 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2081ad1a0 con 0x7fb2080b4520 2021-02-10T11:57:52.841 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.838 7fb1f9ffb700 1 -- 172.21.2.117:0/4032167031 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fb20400bd40 con 0x7fb2080b4520 2021-02-10T11:57:52.842 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 162 2021-02-10T11:57:52.842 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:52.842 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":162,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:11.517907","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8153":{"gid":8153,"name":"b","rank":0,"incarnation":158,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/2333277299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":2333277299},{"type":"v1","addr":"172.21.2.125:6835","nonce":2333277299}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.840 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 msgr2=0x7fb1e801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.840 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 0x7fb1e801abc0 crc :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.840 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 msgr2=0x7fb2080b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:52.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.840 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 rx=0x7fb20400b5b0 tx=0x7fb20400b950).stop 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 shutdown_connections 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb1e80186c0 0x7fb1e801abc0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb2080b4520 0x7fb2080b4920 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb20810f130 0x7fb20810f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 --2- 172.21.2.117:0/4032167031 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb20810e5c0 0x7fb20810dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 >> 172.21.2.117:0/4032167031 conn(0x7fb2081aead0 msgr2=0x7fb2081a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:52.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 shutdown_connections 2021-02-10T11:57:52.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:52.841 7fb20f84c700 1 -- 172.21.2.117:0/4032167031 wait complete. 2021-02-10T11:57:52.852 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs fail cephfs 2021-02-10T11:57:53.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.260 7f288fbf0700 1 Processor -- start 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.260 7f288fbf0700 1 -- start start 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288fbf0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28880d15e0 0x7f28880e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288fbf0700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28880e7580 con 0x7f28881b07f0 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288fbf0700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28880e77f0 con 0x7f28880e7180 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288fbf0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28880e5ce0 con 0x7f28880d15e0 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288da0c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f28881b07f0 0x7f28880e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32826/0 (socket says 172.21.2.117:32826) 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288da0c700 1 -- 172.21.2.117:0/3094886465 learned_addr learned my addr 172.21.2.117:0/3094886465 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404882144 0 0) 0x7f28840010b0 con 0x7f28881b07f0 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2874002fe0 con 0x7f28881b07f0 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288e20d700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28880d15e0 0x7f28880e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2173587854 0 0) 0x7f28840010b0 con 0x7f28881b07f0 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28880d15e0 msgr2=0x7f28880e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f28880d15e0 0x7f28880e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 >> v1:172.21.2.125:6789/0 conn(0x7f28880e7180 legacy=0x7f28880e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28880d1a10 con 0x7f28881b07f0 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.261 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f28840015b0 con 0x7f28881b07f0 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 >> v1:172.21.2.117:6789/0 conn(0x7f28881b07f0 legacy=0x7f28880e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2874003a40 0x7f2874005f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28740067d0 0x7f2874008cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2874009590 0x7f287400bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287400c360 con 0x7f28740067d0 2021-02-10T11:57:53.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287400c580 con 0x7f2874009590 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287400c7a0 con 0x7f2874003a40 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28740067d0 0x7f2874008cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288da0c700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2874003a40 0x7f2874005f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288e20d700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2874009590 0x7f287400bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 -- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2874003a40 msgr2=0x7f2874005f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2874003a40 0x7f2874005f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 -- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2874009590 msgr2=0x7f287400bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2874009590 0x7f287400bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 -- 172.21.2.117:0/3094886465 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287400ca40 con 0x7f28740067d0 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288d20b700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28740067d0 0x7f2874008cf0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f287c04aad0 tx=0x7f287c04da50).ready entity=mon.0 client_cookie=6569ac7063cad8a4 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:53.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f287c04b280 con 0x7f28740067d0 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c04b4c0 con 0x7f28740067d0 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288ca0a700 1 -- 172.21.2.117:0/3094886465 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f287c04b890 con 0x7f28740067d0 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288fbf0700 1 -- 172.21.2.117:0/3094886465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28740067d0 msgr2=0x7f2874008cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288fbf0700 1 --2- 172.21.2.117:0/3094886465 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f28740067d0 0x7f2874008cf0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 rx=0x7f287c04aad0 tx=0x7f287c04da50).stop 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.262 7f288e20d700 1 -- 172.21.2.117:0/3094886465 reap_dead start 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.263 7f288fbf0700 1 -- 172.21.2.117:0/3094886465 shutdown_connections 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.263 7f288fbf0700 1 -- 172.21.2.117:0/3094886465 >> 172.21.2.117:0/3094886465 conn(0x7f28880ce500 msgr2=0x7f28880e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.263 7f288fbf0700 1 -- 172.21.2.117:0/3094886465 shutdown_connections 2021-02-10T11:57:53.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.263 7f288fbf0700 1 -- 172.21.2.117:0/3094886465 wait complete. 2021-02-10T11:57:53.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 Processor -- start 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 -- start start 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2888194c60 0x7f2888193570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2888193b90 0x7f2888193f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888110610 con 0x7f2888110080 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888195060 con 0x7f2888193b90 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288fbf0700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28881a26e0 con 0x7f2888194c60 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288da0c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288da0c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60818/0 (socket says 172.21.2.117:60818) 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.278 7f288da0c700 1 -- 172.21.2.117:0/2862581726 learned_addr learned my addr 172.21.2.117:0/2862581726 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:53.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 -- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2888194c60 msgr2=0x7f2888193570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:53.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2888194c60 0x7f2888193570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:53.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 -- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2888193b90 msgr2=0x7f2888193f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:53.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2888193b90 0x7f2888193f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:53.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 -- 172.21.2.117:0/2862581726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28881a2900 con 0x7f2888110080 2021-02-10T11:57:53.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288da0c700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f2884007650 tx=0x7f288400a880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f288400ac70 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28840016a0 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2884004330 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28881a2900 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.279 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28881a2900 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.280 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2884004140 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.280 7f287a7fc700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 0x7f287001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.280 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2884032570 con 0x7f2888110080 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.280 7f288d20b700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 0x7f287001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:53.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.280 7f288d20b700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 0x7f287001abc0 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:53.314 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.311 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288810c1b0 con 0x7f2888110080 2021-02-10T11:57:53.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.313 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2884001180 con 0x7f2888110080 2021-02-10T11:57:53.443 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:53.440 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f288810c1b0 con 0x7f2888110080 2021-02-10T11:57:54.241 INFO:tasks.ceph.mon.a.gibba017.stderr:2021-02-10 11:57:54.238 7fa041947700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2021-02-10T11:57:54.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.244 7f287a7fc700 1 -- 172.21.2.117:0/2862581726 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v163) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2884004140 con 0x7f2888110080 2021-02-10T11:57:54.262 INFO:teuthology.orchestra.run.gibba017.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2021-02-10T11:57:54.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 msgr2=0x7f287001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 0x7f287001abc0 crc :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.264 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 msgr2=0x7f2888194640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 rx=0x7f2884007650 tx=0x7f288400a880).stop 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 shutdown_connections 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2888193b90 0x7f2888193f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2888194c60 0x7f2888193570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f28700186c0 0x7f287001abc0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.265 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 --2- 172.21.2.117:0/2862581726 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2888110080 0x7f2888194640 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 >> 172.21.2.117:0/2862581726 conn(0x7f288808a780 msgr2=0x7f28880c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 shutdown_connections 2021-02-10T11:57:54.266 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.261 7f288fbf0700 1 -- 172.21.2.117:0/2862581726 wait complete. 2021-02-10T11:57:54.273 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:57:54.298 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.687 7f9eebb53700 1 Processor -- start 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 -- start start 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f9ee40eb7f0 0x7f9ee41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee40ede60 con 0x7f9ee40d18e0 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40ef7f0 con 0x7f9ee40eb7f0 2021-02-10T11:57:54.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.688 7f9eebb53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40d1ce0 con 0x7f9ee40ef3f0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58506/0 (socket says 172.21.2.117:58506) 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 -- 172.21.2.117:0/4024868298 learned_addr learned my addr 172.21.2.117:0/4024868298 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9eea170700 1 --2- 172.21.2.117:0/4024868298 >> v2:172.21.2.125:3300/0 conn(0x7f9ee40eb7f0 0x7f9ee41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 -- 172.21.2.117:0/4024868298 >> v2:172.21.2.125:3300/0 conn(0x7f9ee40eb7f0 msgr2=0x7f9ee41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 --2- 172.21.2.117:0/4024868298 >> v2:172.21.2.125:3300/0 conn(0x7f9ee40eb7f0 0x7f9ee41b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 -- 172.21.2.117:0/4024868298 >> v1:172.21.2.125:6789/0 conn(0x7f9ee40d18e0 legacy=0x7f9ee40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.689 7f9ee916e700 1 -- 172.21.2.117:0/4024868298 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee41b8f90 con 0x7f9ee40ef3f0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9ee916e700 1 --2- 172.21.2.117:0/4024868298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9ee0006700 tx=0x7f9ee0009b80).ready entity=mon.2 client_cookie=32178c3d8792b8f5 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9ee896d700 1 -- 172.21.2.117:0/4024868298 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee000c0c0 con 0x7f9ee40ef3f0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9ee896d700 1 -- 172.21.2.117:0/4024868298 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee000c340 con 0x7f9ee40ef3f0 2021-02-10T11:57:54.709 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9ee896d700 1 -- 172.21.2.117:0/4024868298 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee000c860 con 0x7f9ee40ef3f0 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9eebb53700 1 -- 172.21.2.117:0/4024868298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 msgr2=0x7f9ee40eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.690 7f9eebb53700 1 --2- 172.21.2.117:0/4024868298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f9ee0006700 tx=0x7f9ee0009b80).stop 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 -- 172.21.2.117:0/4024868298 shutdown_connections 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 --2- 172.21.2.117:0/4024868298 >> v2:172.21.2.125:3300/0 conn(0x7f9ee40eb7f0 0x7f9ee41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 --2- 172.21.2.117:0/4024868298 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee40ef3f0 0x7f9ee40eb1d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 -- 172.21.2.117:0/4024868298 >> 172.21.2.117:0/4024868298 conn(0x7f9ee40ced60 msgr2=0x7f9ee40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 -- 172.21.2.117:0/4024868298 shutdown_connections 2021-02-10T11:57:54.710 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.691 7f9eebb53700 1 -- 172.21.2.117:0/4024868298 wait complete. 2021-02-10T11:57:54.711 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.705 7f9eebb53700 1 Processor -- start 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.705 7f9eebb53700 1 -- start start 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ee41afd30 0x7f9ee40b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 0x7f9ee40e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 0x7f9ee40e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40e80e0 con 0x7f9ee40b43f0 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40e7e20 con 0x7f9ee41afd30 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eebb53700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee4096a70 con 0x7f9ee41b0130 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eea170700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 0x7f9ee40e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9ee916e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 0x7f9ee40e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9ee996f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ee41afd30 0x7f9ee40b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eea170700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 0x7f9ee40e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58514/0 (socket says 172.21.2.117:58514) 2021-02-10T11:57:54.712 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.706 7f9eea170700 1 -- 172.21.2.117:0/2320235228 learned_addr learned my addr 172.21.2.117:0/2320235228 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 -- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 msgr2=0x7f9ee40e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 0x7f9ee40e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 -- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ee41afd30 msgr2=0x7f9ee40b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ee41afd30 0x7f9ee40b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 -- 172.21.2.117:0/2320235228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40e8660 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ee916e700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 0x7f9ee40e71e0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7f9ee00096d0 tx=0x7f9ee000b8c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee000bcb0 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee0002030 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9ee00024c0 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee40e8660 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.707 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee40e8660 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.708 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f9ee0000c90 con 0x7f9ee40b43f0 2021-02-10T11:57:54.713 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.708 7f9ed67fc700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 0x7f9ecc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.708 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f9ee0032600 con 0x7f9ee40b43f0 2021-02-10T11:57:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.708 7f9ee996f700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 0x7f9ecc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:54.714 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.709 7f9ee996f700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 0x7f9ecc01abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:54.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.738 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee4087290 con 0x7f9ee40b43f0 2021-02-10T11:57:54.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.741 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9ee001dde0 con 0x7f9ee40b43f0 2021-02-10T11:57:54.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.868 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ee4087290 con 0x7f9ee40b43f0 2021-02-10T11:57:54.872 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.869 7f9ed67fc700 1 -- 172.21.2.117:0/2320235228 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+1763 (secure 0 0 0) 0x7f9ee001dde0 con 0x7f9ee40b43f0 2021-02-10T11:57:54.872 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:54.873 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":163,"default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13434,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3681031218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3681031218},{"type":"v1","addr":"172.21.2.117:6833","nonce":3681031218}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":163,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:57:09.509260","modified":"2021-02-10 11:57:54.239714","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":188,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":20}]} 2021-02-10T11:57:54.873 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 163 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 msgr2=0x7f9ecc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 0x7f9ecc01abc0 crc :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 msgr2=0x7f9ee40e71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 0x7f9ee40e71e0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 rx=0x7f9ee00096d0 tx=0x7f9ee000b8c0).stop 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 shutdown_connections 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9ecc0186c0 0x7f9ecc01abc0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9ee40b43f0 0x7f9ee40e71e0 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f9ee41b0130 0x7f9ee40e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 --2- 172.21.2.117:0/2320235228 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f9ee41afd30 0x7f9ee40b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:54.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 >> 172.21.2.117:0/2320235228 conn(0x7f9ee41a3620 msgr2=0x7f9ee40dd840 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:54.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 shutdown_connections 2021-02-10T11:57:54.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:54.871 7f9eebb53700 1 -- 172.21.2.117:0/2320235228 wait complete. 2021-02-10T11:57:54.883 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.296 7f712e546700 1 Processor -- start 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.296 7f712e546700 1 -- start start 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712e546700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712e546700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71280e8d80 0x7f71280e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712e546700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71280e77f0 con 0x7f71280e6170 2021-02-10T11:57:55.316 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712e546700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71280e5ce0 con 0x7f71280e8d80 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712e546700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71280e54b0 con 0x7f71281b07f0 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f7127fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f712cb63700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71280e8d80 0x7f71280e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f7127fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:36992/0 (socket says 172.21.2.117:36992) 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.297 7f7127fff700 1 -- 172.21.2.117:0/1074543343 learned_addr learned my addr 172.21.2.117:0/1074543343 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712cb63700 1 -- 172.21.2.117:0/1074543343 >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 msgr2=0x7f71280e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712cb63700 1 --2- 172.21.2.117:0/1074543343 >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712cb63700 1 -- 172.21.2.117:0/1074543343 >> v1:172.21.2.125:6789/0 conn(0x7f71280e6170 legacy=0x7f71280e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712cb63700 1 -- 172.21.2.117:0/1074543343 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280e9180 con 0x7f71280e8d80 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f7127fff700 1 --2- 172.21.2.117:0/1074543343 >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712cb63700 1 --2- 172.21.2.117:0/1074543343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71280e8d80 0x7f71280e6d60 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7f7120006e30 tx=0x7f7120009e60).ready entity=mon.0 client_cookie=11b67dfb01382f60 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:55.317 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f7126ffd700 1 -- 172.21.2.117:0/1074543343 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7120000f90 con 0x7f71280e8d80 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f7126ffd700 1 -- 172.21.2.117:0/1074543343 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712000b300 con 0x7f71280e8d80 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f7126ffd700 1 -- 172.21.2.117:0/1074543343 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f712000b7d0 con 0x7f71280e8d80 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712e546700 1 -- 172.21.2.117:0/1074543343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71280e8d80 msgr2=0x7f71280e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.298 7f712e546700 1 --2- 172.21.2.117:0/1074543343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71280e8d80 0x7f71280e6d60 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 rx=0x7f7120006e30 tx=0x7f7120009e60).stop 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 -- 172.21.2.117:0/1074543343 shutdown_connections 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 --2- 172.21.2.117:0/1074543343 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71280e8d80 0x7f71280e6d60 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 --2- 172.21.2.117:0/1074543343 >> v2:172.21.2.125:3300/0 conn(0x7f71281b07f0 0x7f71280e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 -- 172.21.2.117:0/1074543343 >> 172.21.2.117:0/1074543343 conn(0x7f71280ce500 msgr2=0x7f71280e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 -- 172.21.2.117:0/1074543343 shutdown_connections 2021-02-10T11:57:55.318 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.299 7f712e546700 1 -- 172.21.2.117:0/1074543343 wait complete. 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.313 7f712e546700 1 Processor -- start 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 -- start start 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 0x7f71281aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7128080250 0x7f7128195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128195f60 con 0x7f7128195b60 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128080650 con 0x7f71280aea50 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712e546700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71281afbf0 con 0x7f7128080250 2021-02-10T11:57:55.320 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712cb63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712cb63700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60846/0 (socket says 172.21.2.117:60846) 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f712cb63700 1 -- 172.21.2.117:0/690563304 learned_addr learned my addr 172.21.2.117:0/690563304 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.314 7f71277fe700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7128080250 0x7f7128195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 0x7f71281aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 -- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7128080250 msgr2=0x7f7128195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7128080250 0x7f7128195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 -- 172.21.2.117:0/690563304 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 msgr2=0x7f712810cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 -- 172.21.2.117:0/690563304 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712810d100 con 0x7f71280aea50 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f712cb63700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7127fff700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 0x7f71281aef70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f7118006b10 tx=0x7f711800b8c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71180065d0 con 0x7f71280aea50 2021-02-10T11:57:55.321 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711800bcb0 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7118006220 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f712e546700 1 -- 172.21.2.117:0/690563304 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712810d100 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.315 7f712e546700 1 -- 172.21.2.117:0/690563304 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712810d100 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.316 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7118006030 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.316 7f7124ff9700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 0x7f710801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.316 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f7118032680 con 0x7f71280aea50 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.316 7f71277fe700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 0x7f710801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:55.322 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.317 7f71277fe700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 0x7f710801abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:55.350 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.347 7f712e546700 1 -- 172.21.2.117:0/690563304 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712807e840 con 0x7f71280aea50 2021-02-10T11:57:55.354 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.352 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f71180065d0 con 0x7f71280aea50 2021-02-10T11:57:55.483 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:55.479 7f712e546700 1 -- 172.21.2.117:0/690563304 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f71280e77f0 con 0x7f71280aea50 2021-02-10T11:57:56.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.250 7f7124ff9700 1 -- 172.21.2.117:0/690563304 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v164) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7118006030 con 0x7f71280aea50 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 -- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 msgr2=0x7f710801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 0x7f710801abc0 crc :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 -- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 msgr2=0x7f71281aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 0x7f71281aef70 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f7118006b10 tx=0x7f711800b8c0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 -- 172.21.2.117:0/690563304 shutdown_connections 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71080186c0 0x7f710801abc0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f7128195b60 0x7f712810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f7128080250 0x7f7128195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 --2- 172.21.2.117:0/690563304 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71280aea50 0x7f71281aef70 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.302 7f712e546700 1 -- 172.21.2.117:0/690563304 >> 172.21.2.117:0/690563304 conn(0x7f71281998b0 msgr2=0x7f71281ab430 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.303 7f712e546700 1 -- 172.21.2.117:0/690563304 shutdown_connections 2021-02-10T11:57:56.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.303 7f712e546700 1 -- 172.21.2.117:0/690563304 wait complete. 2021-02-10T11:57:56.314 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.728 7f4daca66700 1 Processor -- start 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.728 7f4daca66700 1 -- start start 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4daca66700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80e5240 0x7f4da8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4daca66700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4da8084c60 0x7f4da8085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4daca66700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da8085930 con 0x7f4da80e4e40 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4daca66700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da8085680 con 0x7f4da8084c60 2021-02-10T11:57:56.749 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4daca66700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81b09e0 con 0x7f4da80e5240 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da685e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f4da80e4e40 0x7f4da81b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32870/0 (socket says 172.21.2.117:32870) 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da685e700 1 -- 172.21.2.117:0/2832479106 learned_addr learned my addr 172.21.2.117:0/2832479106 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609511783 0 0) 0x7f4d98000ce0 con 0x7f4da80e4e40 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94002fe0 con 0x7f4da80e4e40 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da705f700 1 --2- 172.21.2.117:0/2832479106 >> v2:172.21.2.125:3300/0 conn(0x7f4da8084c60 0x7f4da8085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da605d700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80e5240 0x7f4da8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 492182294 0 0) 0x7f4d98001180 con 0x7f4da80e4e40 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80e5240 msgr2=0x7f4da8086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:56.750 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80e5240 0x7f4da8086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 >> v2:172.21.2.125:3300/0 conn(0x7f4da8084c60 msgr2=0x7f4da8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 --2- 172.21.2.117:0/2832479106 >> v2:172.21.2.125:3300/0 conn(0x7f4da8084c60 0x7f4da8085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da8084840 con 0x7f4da80e4e40 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f4d98001680 con 0x7f4da80e4e40 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 >> v1:172.21.2.117:6789/0 conn(0x7f4da80e4e40 legacy=0x7f4da81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4d94003ac0 0x7f4d94005ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4d94006790 0x7f4d94008c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4d94009500 0x7f4d9400ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940502e0 con 0x7f4d94009500 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940504c0 con 0x7f4d94003ac0 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.729 7f4da585c700 1 -- 172.21.2.117:0/2832479106 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d940506e0 con 0x7f4d94006790 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4d94009500 0x7f4d9400ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.751 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da605d700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4d94006790 0x7f4d94008c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da685e700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4d94003ac0 0x7f4d94005ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 -- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4d94006790 msgr2=0x7f4d94008c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4d94006790 0x7f4d94008c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 -- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4d94003ac0 msgr2=0x7f4d94005ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4d94003ac0 0x7f4d94005ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 -- 172.21.2.117:0/2832479106 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94050980 con 0x7f4d94009500 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4d94009500 0x7f4d9400ba20 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 rx=0x7f4da00090f0 tx=0x7f4da000b050).ready entity=mon.0 client_cookie=acc444fae00b1ff6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da0000f90 con 0x7f4d94009500 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da0006240 con 0x7f4d94009500 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da585c700 1 -- 172.21.2.117:0/2832479106 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da00056b0 con 0x7f4d94009500 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4daca66700 1 -- 172.21.2.117:0/2832479106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4d94009500 msgr2=0x7f4d9400ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4daca66700 1 --2- 172.21.2.117:0/2832479106 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4d94009500 0x7f4d9400ba20 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 rx=0x7f4da00090f0 tx=0x7f4da000b050).stop 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.730 7f4da705f700 1 -- 172.21.2.117:0/2832479106 reap_dead start 2021-02-10T11:57:56.752 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.731 7f4daca66700 1 -- 172.21.2.117:0/2832479106 shutdown_connections 2021-02-10T11:57:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.731 7f4daca66700 1 -- 172.21.2.117:0/2832479106 >> 172.21.2.117:0/2832479106 conn(0x7f4da80ce4e0 msgr2=0x7f4da80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.731 7f4daca66700 1 -- 172.21.2.117:0/2832479106 shutdown_connections 2021-02-10T11:57:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.731 7f4daca66700 1 -- 172.21.2.117:0/2832479106 wait complete. 2021-02-10T11:57:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 Processor -- start 2021-02-10T11:57:56.753 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 -- start start 2021-02-10T11:57:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80bf1c0 0x7f4da80bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4da80c1d50 0x7f4da80c2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80c2b70 con 0x7f4da80c2770 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80a2920 con 0x7f4da80c1d50 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.746 7f4daca66700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80a2b40 con 0x7f4da80bf1c0 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60868/0 (socket says 172.21.2.117:60868) 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 -- 172.21.2.117:0/404094428 learned_addr learned my addr 172.21.2.117:0/404094428 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 -- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80bf1c0 msgr2=0x7f4da80bf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80bf1c0 0x7f4da80bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 -- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4da80c1d50 msgr2=0x7f4da80c2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4da80c1d50 0x7f4da80c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 -- 172.21.2.117:0/404094428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80a2d60 con 0x7f4da80c2770 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4da705f700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f4da00090e0 tx=0x7f4da000bd10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:56.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da0009910 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da0000df0 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4da00064c0 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.747 7f4daca66700 1 -- 172.21.2.117:0/404094428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da80a2d60 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.748 7f4daca66700 1 -- 172.21.2.117:0/404094428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da80a32f0 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.748 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4da0006030 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.748 7f4d9f7fe700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 0x7f4d8c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.748 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 6006+0+0 (secure 0 0 0) 0x7f4da00329b0 con 0x7f4da80c2770 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.749 7f4da685e700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 0x7f4d8c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.749 7f4da685e700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 0x7f4d8c01abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:56.781 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.778 7f4daca66700 1 -- 172.21.2.117:0/404094428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da810c2a0 con 0x7f4da80c2770 2021-02-10T11:57:56.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.781 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4da0009910 con 0x7f4da80c2770 2021-02-10T11:57:56.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:56.908 7f4daca66700 1 -- 172.21.2.117:0/404094428 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4da8085930 con 0x7f4da80c2770 2021-02-10T11:57:57.263 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.259 7f4d9f7fe700 1 -- 172.21.2.117:0/404094428 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_metadata", "prefix": "osd pool delete", "pool": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v189) v1 ==== 185+0+0 (secure 0 0 0) 0x7f4da001f910 con 0x7f4da80c2770 2021-02-10T11:57:57.277 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' removed 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 msgr2=0x7f4d8c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 0x7f4d8c01abc0 crc :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 msgr2=0x7f4da80a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f4da00090e0 tx=0x7f4da000bd10).stop 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 shutdown_connections 2021-02-10T11:57:57.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4d8c0186c0 0x7f4d8c01abc0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4da80c2770 0x7f4da80a2300 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4da80c1d50 0x7f4da80c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 --2- 172.21.2.117:0/404094428 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4da80bf1c0 0x7f4da80bf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 >> 172.21.2.117:0/404094428 conn(0x7f4da80aa1b0 msgr2=0x7f4da80c3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 shutdown_connections 2021-02-10T11:57:57.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.276 7f4daca66700 1 -- 172.21.2.117:0/404094428 wait complete. 2021-02-10T11:57:57.287 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2021-02-10T11:57:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.701 7f69bae0a700 1 Processor -- start 2021-02-10T11:57:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.701 7f69bae0a700 1 -- start start 2021-02-10T11:57:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69bae0a700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 0x7f69b41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.721 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69bae0a700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f69b40e5240 0x7f69b4084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69bae0a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b4085930 con 0x7f69b41b0810 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69bae0a700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b40e5cc0 con 0x7f69b40e5240 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69bae0a700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b4084840 con 0x7f69b40e4e40 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69b9427700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f69b41b0810 0x7f69b4085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32892/0 (socket says 172.21.2.117:32892) 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69b9427700 1 -- 172.21.2.117:0/4257583076 learned_addr learned my addr 172.21.2.117:0/4257583076 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69b3fff700 1 --2- 172.21.2.117:0/4257583076 >> v2:172.21.2.117:3300/0 conn(0x7f69b40e5240 0x7f69b4084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.702 7f69b8c26700 1 --2- 172.21.2.117:0/4257583076 >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 0x7f69b41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b3fff700 1 -- 172.21.2.117:0/4257583076 >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 msgr2=0x7f69b41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b3fff700 1 --2- 172.21.2.117:0/4257583076 >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 0x7f69b41b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b3fff700 1 -- 172.21.2.117:0/4257583076 >> v1:172.21.2.117:6789/0 conn(0x7f69b41b0810 legacy=0x7f69b4085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b3fff700 1 -- 172.21.2.117:0/4257583076 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4085280 con 0x7f69b40e5240 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b37fe700 1 -- 172.21.2.117:0/4257583076 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589349769 0 0) 0x7f69ac000f90 con 0x7f69b41b0810 2021-02-10T11:57:57.722 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b8c26700 1 --2- 172.21.2.117:0/4257583076 >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 0x7f69b41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b3fff700 1 --2- 172.21.2.117:0/4257583076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40e5240 0x7f69b4084c60 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f69a4006650 tx=0x7f69a4009a80).ready entity=mon.0 client_cookie=aadf9eb352dc3534 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b37fe700 1 -- 172.21.2.117:0/4257583076 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69a40009b0 con 0x7f69b40e5240 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b37fe700 1 -- 172.21.2.117:0/4257583076 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a400c0c0 con 0x7f69b40e5240 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69b37fe700 1 -- 172.21.2.117:0/4257583076 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69a400c520 con 0x7f69b40e5240 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69bae0a700 1 -- 172.21.2.117:0/4257583076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40e5240 msgr2=0x7f69b4084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.703 7f69bae0a700 1 --2- 172.21.2.117:0/4257583076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40e5240 0x7f69b4084c60 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 rx=0x7f69a4006650 tx=0x7f69a4009a80).stop 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 -- 172.21.2.117:0/4257583076 shutdown_connections 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 --2- 172.21.2.117:0/4257583076 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40e5240 0x7f69b4084c60 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 --2- 172.21.2.117:0/4257583076 >> v2:172.21.2.125:3300/0 conn(0x7f69b40e4e40 0x7f69b41b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 -- 172.21.2.117:0/4257583076 >> 172.21.2.117:0/4257583076 conn(0x7f69b40ce4e0 msgr2=0x7f69b40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 -- 172.21.2.117:0/4257583076 shutdown_connections 2021-02-10T11:57:57.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.704 7f69bae0a700 1 -- 172.21.2.117:0/4257583076 wait complete. 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 Processor -- start 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 -- start start 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 0x7f69b4191080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69b40b6c00 0x7f69b40b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4191ab0 con 0x7f69b40c14d0 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b41916a0 con 0x7f69b40c2000 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.719 7f69bae0a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4191880 con 0x7f69b40b6c00 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:57.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60878/0 (socket says 172.21.2.117:60878) 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 -- 172.21.2.117:0/1768271301 learned_addr learned my addr 172.21.2.117:0/1768271301 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 -- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69b40b6c00 msgr2=0x7f69b40b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69b40b6c00 0x7f69b40b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 -- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 msgr2=0x7f69b4191080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b8c26700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 0x7f69b4191080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 0x7f69b4191080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 -- 172.21.2.117:0/1768271301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40b7620 con 0x7f69b40c14d0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b8c26700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 0x7f69b4191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.720 7f69b3fff700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7f69a4009630 tx=0x7f69a400b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.721 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69a40093a0 con 0x7f69b40c14d0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.721 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a400bcf0 con 0x7f69b40c14d0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.721 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f69a4005560 con 0x7f69b40c14d0 2021-02-10T11:57:57.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.721 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b40b7620 con 0x7f69b40c14d0 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.721 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b40b7bb0 con 0x7f69b40c14d0 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.722 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f69a400a000 con 0x7f69b40c14d0 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.722 7f69b17fa700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.722 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5568+0+0 (secure 0 0 0) 0x7f69a4032580 con 0x7f69b40c14d0 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.722 7f69b8c26700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:57.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.723 7f69b8c26700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 0x7f699801abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:57.754 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.752 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b4191880 con 0x7f69b40c14d0 2021-02-10T11:57:57.757 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.754 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f69a4019180 con 0x7f69b40c14d0 2021-02-10T11:57:57.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:57.884 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f69b4191880 con 0x7f69b40c14d0 2021-02-10T11:57:58.268 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.265 7f69b17fa700 1 -- 172.21.2.117:0/1768271301 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"pool2": "cephfs_data", "prefix": "osd pool delete", "pool": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v190) v1 ==== 173+0+0 (secure 0 0 0) 0x7f69a4019180 con 0x7f69b40c14d0 2021-02-10T11:57:58.305 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' removed 2021-02-10T11:57:58.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 msgr2=0x7f699801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 0x7f699801abc0 crc :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 msgr2=0x7f69b40c18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 rx=0x7f69a4009630 tx=0x7f69a400b900).stop 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 shutdown_connections 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f69980186c0 0x7f699801abc0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f69b40c14d0 0x7f69b40c18d0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f69b40b6c00 0x7f69b40b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:58.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 --2- 172.21.2.117:0/1768271301 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f69b40c2000 0x7f69b4191080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 >> 172.21.2.117:0/1768271301 conn(0x7f69b40acec0 msgr2=0x7f69b40b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:58.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 shutdown_connections 2021-02-10T11:57:58.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.304 7f69bae0a700 1 -- 172.21.2.117:0/1768271301 wait complete. 2021-02-10T11:57:58.315 INFO:tasks.ceph.mds.a:Restarting daemon 2021-02-10T11:57:58.315 INFO:tasks.ceph.mds.a:Stopping old one... 2021-02-10T11:57:58.316 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T11:57:58.316 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:57:58.316 INFO:tasks.ceph.mds.b:Restarting daemon 2021-02-10T11:57:58.316 INFO:tasks.ceph.mds.b:Stopping old one... 2021-02-10T11:57:58.316 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T11:57:58.316 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T11:57:58.321 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T11:57:58.321 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2021-02-10T11:57:58.322 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T11:57:58.323 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2021-02-10T11:57:58.364 INFO:tasks.ceph.mds.a:Started 2021-02-10T11:57:58.364 DEBUG:teuthology.parallel:result is None 2021-02-10T11:57:58.365 INFO:tasks.ceph.mds.b:Started 2021-02-10T11:57:58.365 DEBUG:teuthology.parallel:result is None 2021-02-10T11:57:58.366 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd blacklist clear 2021-02-10T11:57:58.770 INFO:tasks.ceph.mds.b.gibba025.stdout:starting mds.b at 2021-02-10T11:57:58.777 INFO:tasks.ceph.mds.a.gibba017.stdout:starting mds.a at 2021-02-10T11:57:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.778 7fdf0f6aa700 1 Processor -- start 2021-02-10T11:57:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.778 7fdf0f6aa700 1 -- start start 2021-02-10T11:57:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdf080e4e40 0x7fdf081b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 0x7fdf08084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf08085930 con 0x7fdf081b0810 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf08084840 con 0x7fdf080e4e40 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf08084a30 con 0x7fdf080e5240 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdf080e4e40 0x7fdf081b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdf080e4e40 0x7fdf081b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60896/0 (socket says 172.21.2.117:60896) 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 -- 172.21.2.117:0/4285065320 learned_addr learned my addr 172.21.2.117:0/4285065320 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0ccc5700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 0x7fdf08084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 -- 172.21.2.117:0/4285065320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 msgr2=0x7fdf08084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 0x7fdf08084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 -- 172.21.2.117:0/4285065320 >> v1:172.21.2.117:6789/0 conn(0x7fdf081b0810 legacy=0x7fdf08085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 -- 172.21.2.117:0/4285065320 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf08085280 con 0x7fdf080e4e40 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0ccc5700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 0x7fdf08084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:57:58.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0d4c6700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080e4e40 0x7fdf081b01f0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 rx=0x7fdf00006a80 tx=0x7fdf00009f00).ready entity=mon.0 client_cookie=e3660412ef96603c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdefffff700 1 -- 172.21.2.117:0/4285065320 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf0000b5c0 con 0x7fdf080e4e40 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdefffff700 1 -- 172.21.2.117:0/4285065320 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0000b900 con 0x7fdf080e4e40 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdefffff700 1 -- 172.21.2.117:0/4285065320 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdf0000bde0 con 0x7fdf080e4e40 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 -- 172.21.2.117:0/4285065320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080e4e40 msgr2=0x7fdf081b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.779 7fdf0f6aa700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080e4e40 0x7fdf081b01f0 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 rx=0x7fdf00006a80 tx=0x7fdf00009f00).stop 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 -- 172.21.2.117:0/4285065320 shutdown_connections 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080e5240 0x7fdf08084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 --2- 172.21.2.117:0/4285065320 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080e4e40 0x7fdf081b01f0 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 -- 172.21.2.117:0/4285065320 >> 172.21.2.117:0/4285065320 conn(0x7fdf080ce4e0 msgr2=0x7fdf080e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 -- 172.21.2.117:0/4285065320 shutdown_connections 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.780 7fdf0f6aa700 1 -- 172.21.2.117:0/4285065320 wait complete. 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.795 7fdf0f6aa700 1 Processor -- start 2021-02-10T11:57:58.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.795 7fdf0f6aa700 1 -- start start 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdf081a58b0 0x7fdf08199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080b2020 0x7fdf0808caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0808c520 con 0x7fdf080b1600 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0808c230 con 0x7fdf081a58b0 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0819a430 con 0x7fdf080b2020 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60904/0 (socket says 172.21.2.117:60904) 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 -- 172.21.2.117:0/3699989504 learned_addr learned my addr 172.21.2.117:0/3699989504 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0d4c6700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdf081a58b0 0x7fdf08199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 -- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080b2020 msgr2=0x7fdf0808caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080b2020 0x7fdf0808caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 -- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdf081a58b0 msgr2=0x7fdf08199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:58.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdf081a58b0 0x7fdf08199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 -- 172.21.2.117:0/3699989504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0819e970 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0ccc5700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7fdef8006f20 tx=0x7fdef800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdef800bd80 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef8000c90 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0819e970 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdef80022b0 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.796 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0819e970 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.797 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdef800a000 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.797 7fdefdffb700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 0x7fdee801abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.797 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5186+0+0 (secure 0 0 0) 0x7fdef8032500 con 0x7fdf080b1600 2021-02-10T11:57:58.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.797 7fdf0d4c6700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 0x7fdee801abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:58.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.797 7fdf0d4c6700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 0x7fdee801abd0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:58.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.828 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf08085930 con 0x7fdf080b1600 2021-02-10T11:57:58.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.831 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdef8006bb0 con 0x7fdf080b1600 2021-02-10T11:57:58.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:58.971 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fdf08085930 con 0x7fdf080b1600 2021-02-10T11:57:59.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.304 7fdefdffb700 1 -- 172.21.2.117:0/3699989504 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v191) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdef800a000 con 0x7fdf080b1600 2021-02-10T11:57:59.341 INFO:teuthology.orchestra.run.gibba017.stderr: removed all blacklist entries 2021-02-10T11:57:59.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.339 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 msgr2=0x7fdee801abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.339 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 0x7fdee801abd0 crc :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.342 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.339 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 msgr2=0x7fdf080b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.339 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 rx=0x7fdef8006f20 tx=0x7fdef800b990).stop 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 shutdown_connections 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdee80186d0 0x7fdee801abd0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdf080b1600 0x7fdf080b1a00 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdf080b2020 0x7fdf0808caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 --2- 172.21.2.117:0/3699989504 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdf081a58b0 0x7fdf08199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.343 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 >> 172.21.2.117:0/3699989504 conn(0x7fdf0819f650 msgr2=0x7fdf0818c100 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:59.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 shutdown_connections 2021-02-10T11:57:59.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.340 7fdf0f6aa700 1 -- 172.21.2.117:0/3699989504 wait complete. 2021-02-10T11:57:59.351 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2', '3'] 2021-02-10T11:57:59.351 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth ls --format=json-pretty 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.759 7f92b7946700 1 Processor -- start 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.759 7f92b7946700 1 -- start start 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b7946700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92b01b6960 0x7f92b01b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b7946700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00ede60 con 0x7f92b00d18e0 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b7946700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00eb9c0 con 0x7f92b00ef3f0 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b7946700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b00ef7f0 con 0x7f92b01b6960 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5f63700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f92b01b6960 0x7f92b01b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5762700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f92b00d18e0 0x7f92b00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32928/0 (socket says 172.21.2.117:32928) 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5762700 1 -- 172.21.2.117:0/603924325 learned_addr learned my addr 172.21.2.117:0/603924325 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:57:59.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5f63700 1 -- 172.21.2.117:0/603924325 >> v1:172.21.2.125:6789/0 conn(0x7f92b00ef3f0 legacy=0x7f92b00eb1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5f63700 1 -- 172.21.2.117:0/603924325 >> v1:172.21.2.117:6789/0 conn(0x7f92b00d18e0 legacy=0x7f92b00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.760 7f92b5f63700 1 -- 172.21.2.117:0/603924325 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00d1d20 con 0x7f92b01b6960 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b5f63700 1 --2- 172.21.2.117:0/603924325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b01b6960 0x7f92b01b8d70 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 rx=0x7f92a8006e30 tx=0x7f92a8009e60).ready entity=mon.0 client_cookie=938d3423e881e14d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92a7fff700 1 -- 172.21.2.117:0/603924325 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a8006a50 con 0x7f92b01b6960 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92a7fff700 1 -- 172.21.2.117:0/603924325 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a800b300 con 0x7f92b01b6960 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92a7fff700 1 -- 172.21.2.117:0/603924325 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92a800b820 con 0x7f92b01b6960 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b7946700 1 -- 172.21.2.117:0/603924325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b01b6960 msgr2=0x7f92b01b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b7946700 1 --2- 172.21.2.117:0/603924325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b01b6960 0x7f92b01b8d70 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 rx=0x7f92a8006e30 tx=0x7f92a8009e60).stop 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b7946700 1 -- 172.21.2.117:0/603924325 shutdown_connections 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b7946700 1 --2- 172.21.2.117:0/603924325 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b01b6960 0x7f92b01b8d70 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.761 7f92b7946700 1 -- 172.21.2.117:0/603924325 >> 172.21.2.117:0/603924325 conn(0x7f92b00ced60 msgr2=0x7f92b00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.762 7f92b7946700 1 -- 172.21.2.117:0/603924325 shutdown_connections 2021-02-10T11:57:59.780 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.762 7f92b7946700 1 -- 172.21.2.117:0/603924325 wait complete. 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.776 7f92b7946700 1 Processor -- start 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.776 7f92b7946700 1 -- start start 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92b01c6930 0x7f92b00efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92b009a410 0x7f92b00f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00f0b30 con 0x7f92b00b1530 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00ed270 con 0x7f92b009a410 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b00b1c00 con 0x7f92b01c6930 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:59.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60916/0 (socket says 172.21.2.117:60916) 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 -- 172.21.2.117:0/1951175028 learned_addr learned my addr 172.21.2.117:0/1951175028 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 -- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92b01c6930 msgr2=0x7f92b00efdb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92b01c6930 0x7f92b00efdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 -- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92b009a410 msgr2=0x7f92b00f03d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92b009a410 0x7f92b00f03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 -- 172.21.2.117:0/1951175028 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b00f10b0 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b5762700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f92ac001c60 tx=0x7f92ac00a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac00acc0 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac004140 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f92ac004620 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.777 7f92b7946700 1 -- 172.21.2.117:0/1951175028 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b00f10b0 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.778 7f92b7946700 1 -- 172.21.2.117:0/1951175028 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b00f10b0 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.778 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f92ac000ce0 con 0x7f92b00b1530 2021-02-10T11:57:59.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.778 7f92a5ffb700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:57:59.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.778 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f92ac032400 con 0x7f92b00b1530 2021-02-10T11:57:59.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.780 7f92b4f61700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:57:59.784 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.780 7f92b4f61700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 0x7f929001ac10 crc :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:57:59.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.810 7f92b7946700 1 -- 172.21.2.117:0/1951175028 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b00ed270 con 0x7f92b00b1530 2021-02-10T11:57:59.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.812 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f92ac0009b0 con 0x7f92b00b1530 2021-02-10T11:57:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.941 7f92b7946700 1 -- 172.21.2.117:0/1951175028 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f92b00ed270 con 0x7f92b00b1530 2021-02-10T11:57:59.944 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.941 7f92a5ffb700 1 -- 172.21.2.117:0/1951175028 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v117) v1 ==== 80+0+4513 (secure 0 0 0) 0x7f92ac00acc0 con 0x7f92b00b1530 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: "auth_dump": [ 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.a", 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg9R0sAxAAT58NMqcTPXtYV6x8Ri1qQg==", 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.945 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mds.b", 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB/wSNg30IeLhAA9BClKWYTKwloZC/6i0lXpQ==", 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.946 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.0", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCHwSNgagntORAAwGONGT6awhjChST7OaENUg==", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.947 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.1", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCKwSNgQfRAOhAAv3wSupgi1hch4uKfKnZ64A==", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.2", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCNwSNgJgnMOhAAxOX6BFLnu51euDcMuUZOoQ==", 2021-02-10T11:57:59.948 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.3", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCQwSNgqH7zOhAABzJteQTFPtOy40c9s+tLwQ==", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.949 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.4", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCZwSNgeOdmLBAANZc2zWTNHSPNiF1PgLfRVA==", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.5", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCcwSNgcQ+pLBAA7bRyW/GTVKavEKFI8kUx4w==", 2021-02-10T11:57:59.950 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.6", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCfwSNgWTPfLBAADyBSqDbXA3VeOMZf9wLy1A==", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.951 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "osd.7", 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCiwSNgPB1VLRAAiVWXYufc+lu6GdwNtKePtA==", 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.0", 2021-02-10T11:57:59.952 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCAwSNgDlskHRAAUXwev3k3HGug+GWF9sZ4uw==", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.1", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNghnIXARAAx9hcWrUXUWj5BxMt7OuZfg==", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:59.953 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.2", 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNg7x2fFxAARqJinZjDmg4g97OdFMhQ6g==", 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.954 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.3", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQCBwSNgdjMeLhAAwxA6ToFM5PiUrMmJfLicLQ==", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow r", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow rw pool=cephfs_data" 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "client.admin", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB7wSNgSXr/CRAATOvCQT30fSAWSMANj5Jv1Q==", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "mgr": "allow *", 2021-02-10T11:57:59.955 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow *", 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.x", 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgFy39JxAAnjfX9kL5ISqM95Zliu5Jog==", 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:57:59.956 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "entity": "mgr.y", 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "key": "AQB+wSNgtBUMEBAAggrPp37SSLsoPjjhHOwgzg==", 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "caps": { 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "mds": "allow *", 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "mon": "allow profile mgr", 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: "osd": "allow *" 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout: ] 2021-02-10T11:57:59.957 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:57:59.958 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.943 7f92b7946700 1 -- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 msgr2=0x7f929001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.943 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 0x7f929001ac10 crc :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.943 7f92b7946700 1 -- 172.21.2.117:0/1951175028 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 msgr2=0x7f92b01c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.943 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f92ac001c60 tx=0x7f92ac00a8d0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 -- 172.21.2.117:0/1951175028 shutdown_connections 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f92b009a410 0x7f92b00f03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f92b01c6930 0x7f92b00efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f9290018730 0x7f929001ac10 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 --2- 172.21.2.117:0/1951175028 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f92b00b1530 0x7f92b01c6310 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 -- 172.21.2.117:0/1951175028 >> 172.21.2.117:0/1951175028 conn(0x7f92b0086d40 msgr2=0x7f92b00f5690 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 -- 172.21.2.117:0/1951175028 shutdown_connections 2021-02-10T11:57:59.959 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:57:59.944 7f92b7946700 1 -- 172.21.2.117:0/1951175028 wait complete. 2021-02-10T11:57:59.960 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:00.389 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.370 7f09c4a4d700 1 Processor -- start 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.370 7f09c4a4d700 1 -- start start 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09c4a4d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f09c00e4e40 0x7f09c01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09c4a4d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09c0085450 0x7f09c0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09c4a4d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c0085930 con 0x7f09c00e5240 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09c4a4d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c00e5cc0 con 0x7f09c0085450 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09c4a4d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c0084840 con 0x7f09c00e4e40 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09c0085450 0x7f09c0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f09c0085450 0x7f09c0086ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60928/0 (socket says 172.21.2.117:60928) 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 -- 172.21.2.117:0/2571856701 learned_addr learned my addr 172.21.2.117:0/2571856701 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 -- 172.21.2.117:0/2571856701 >> v2:172.21.2.125:3300/0 conn(0x7f09c00e4e40 msgr2=0x7f09c01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 --2- 172.21.2.117:0/2571856701 >> v2:172.21.2.125:3300/0 conn(0x7f09c00e4e40 0x7f09c01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.390 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 -- 172.21.2.117:0/2571856701 >> v1:172.21.2.125:6789/0 conn(0x7f09c00e5240 legacy=0x7f09c0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 -- 172.21.2.117:0/2571856701 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c01b0810 con 0x7f09c0085450 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.371 7f09bf05f700 1 --2- 172.21.2.117:0/2571856701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0085450 0x7f09c0086ec0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f09b8006f00 tx=0x7f09b8009f10).ready entity=mon.0 client_cookie=e45a78974ddfdd04 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09bd85c700 1 -- 172.21.2.117:0/2571856701 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800b3b0 con 0x7f09c0085450 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09bd85c700 1 -- 172.21.2.117:0/2571856701 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b800b730 con 0x7f09c0085450 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09bd85c700 1 -- 172.21.2.117:0/2571856701 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b800bc50 con 0x7f09c0085450 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 -- 172.21.2.117:0/2571856701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0085450 msgr2=0x7f09c0086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 --2- 172.21.2.117:0/2571856701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0085450 0x7f09c0086ec0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 rx=0x7f09b8006f00 tx=0x7f09b8009f10).stop 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 -- 172.21.2.117:0/2571856701 shutdown_connections 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 --2- 172.21.2.117:0/2571856701 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0085450 0x7f09c0086ec0 unknown :-1 s=CLOSED pgs=3217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 --2- 172.21.2.117:0/2571856701 >> v2:172.21.2.125:3300/0 conn(0x7f09c00e4e40 0x7f09c01b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 -- 172.21.2.117:0/2571856701 >> 172.21.2.117:0/2571856701 conn(0x7f09c00ce4e0 msgr2=0x7f09c00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 -- 172.21.2.117:0/2571856701 shutdown_connections 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.372 7f09c4a4d700 1 -- 172.21.2.117:0/2571856701 wait complete. 2021-02-10T11:58:00.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 Processor -- start 2021-02-10T11:58:00.392 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 -- start start 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c00b7ce0 0x7f09c00b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c0199a90 0x7f09c0199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0198af0 con 0x7f09c0199070 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0194c60 con 0x7f09c00b7ce0 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09c4a4d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0194e40 con 0x7f09c0199a90 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60934/0 (socket says 172.21.2.117:60934) 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 -- 172.21.2.117:0/1952756238 learned_addr learned my addr 172.21.2.117:0/1952756238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:00.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 -- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c0199a90 msgr2=0x7f09c0199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c0199a90 0x7f09c0199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 -- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c00b7ce0 msgr2=0x7f09c00b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c00b7ce0 0x7f09c00b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.387 7f09be05d700 1 -- 172.21.2.117:0/1952756238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0195060 con 0x7f09c0199070 2021-02-10T11:58:00.394 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09be05d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 rx=0x7f09b400b530 tx=0x7f09b400b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b400bcc0 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b4000e20 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f09b4006210 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c0195060 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c01955f0 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.388 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f09b4006020 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.389 7f09af7fe700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.389 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f09b40323e0 con 0x7f09c0199070 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.389 7f09be85e700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:00.395 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.389 7f09be85e700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:00.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.421 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c010c250 con 0x7f09c0199070 2021-02-10T11:58:00.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.424 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f09b4030d70 con 0x7f09c0199070 2021-02-10T11:58:00.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.547 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09c0085930 con 0x7f09c0199070 2021-02-10T11:58:00.550 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.547 7f09af7fe700 1 -- 172.21.2.117:0/1952756238 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f09b4008160 con 0x7f09c0199070 2021-02-10T11:58:00.551 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:00.551 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":191,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:57:59.301594","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 msgr2=0x7f09a401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 crc :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 msgr2=0x7f09c0199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.552 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 rx=0x7f09b400b530 tx=0x7f09b400b8d0).stop 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 shutdown_connections 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f09a40186c0 0x7f09a401abc0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f09c0199070 0x7f09c0199470 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f09c0199a90 0x7f09c0199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 --2- 172.21.2.117:0/1952756238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f09c00b7ce0 0x7f09c00b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 >> 172.21.2.117:0/1952756238 conn(0x7f09c01b38c0 msgr2=0x7f09c010c480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 shutdown_connections 2021-02-10T11:58:00.553 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.549 7f09c4a4d700 1 -- 172.21.2.117:0/1952756238 wait complete. 2021-02-10T11:58:00.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-10T11:58:00.561 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:58:00.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.975 7f91de635700 1 Processor -- start 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.975 7f91de635700 1 -- start start 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91de635700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f91d80ef3f0 0x7f91d80eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91de635700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d80ef7f0 con 0x7f91d80d18f0 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91de635700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d80ede60 con 0x7f91d80eb830 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91de635700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d80d1da0 con 0x7f91d80ef3f0 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91d7fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f91d80d18f0 0x7f91d80eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32956/0 (socket says 172.21.2.117:32956) 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91d7fff700 1 -- 172.21.2.117:0/1277008004 learned_addr learned my addr 172.21.2.117:0/1277008004 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738977112 0 0) 0x7f91c8000ce0 con 0x7f91d80d18f0 2021-02-10T11:58:00.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.976 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0002fe0 con 0x7f91d80d18f0 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d77fe700 1 --2- 172.21.2.117:0/1277008004 >> v2:172.21.2.125:3300/0 conn(0x7f91d80ef3f0 0x7f91d80eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1805961808 0 0) 0x7f91c80010e0 con 0x7f91d80d18f0 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 >> v2:172.21.2.125:3300/0 conn(0x7f91d80ef3f0 msgr2=0x7f91d80eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 --2- 172.21.2.117:0/1277008004 >> v2:172.21.2.125:3300/0 conn(0x7f91d80ef3f0 0x7f91d80eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 >> v1:172.21.2.125:6789/0 conn(0x7f91d80eb830 legacy=0x7f91d81b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d81bb1c0 con 0x7f91d80d18f0 2021-02-10T11:58:00.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f91c8001660 con 0x7f91d80d18f0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 >> v1:172.21.2.117:6789/0 conn(0x7f91d80d18f0 legacy=0x7f91d80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 0x7f91c0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00067b0 0x7f91c0008cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c00095b0 0x7f91c000bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c000c300 con 0x7f91c0003af0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c000c520 con 0x7f91c00095b0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c0050750 con 0x7f91c00067b0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 0x7f91c0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91dcc52700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c00095b0 0x7f91c000bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d77fe700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00067b0 0x7f91c0008cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 -- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00067b0 msgr2=0x7f91c0008cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:01.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91c00067b0 0x7f91c0008cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 -- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c00095b0 msgr2=0x7f91c000bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91c00095b0 0x7f91c000bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 -- 172.21.2.117:0/1277008004 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00509b0 con 0x7f91c0003af0 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91dcc52700 1 -- 172.21.2.117:0/1277008004 reap_dead start 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.977 7f91d7fff700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 0x7f91c0005fa0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7f91c80014e0 tx=0x7f91c8007d00).ready entity=mon.0 client_cookie=b820dc3d8a35ec56 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c80070c0 con 0x7f91c0003af0 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c80070c0 con 0x7f91c0003af0 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91d6ffd700 1 -- 172.21.2.117:0/1277008004 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91c80043a0 con 0x7f91c0003af0 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 -- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 msgr2=0x7f91c0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 0x7f91c0005fa0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 rx=0x7f91c80014e0 tx=0x7f91c8007d00).stop 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 -- 172.21.2.117:0/1277008004 shutdown_connections 2021-02-10T11:58:01.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 --2- 172.21.2.117:0/1277008004 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91c0003af0 0x7f91c0005fa0 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 -- 172.21.2.117:0/1277008004 >> 172.21.2.117:0/1277008004 conn(0x7f91d80cf060 msgr2=0x7f91d80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 -- 172.21.2.117:0/1277008004 shutdown_connections 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.978 7f91de635700 1 -- 172.21.2.117:0/1277008004 wait complete. 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 Processor -- start 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 -- start start 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91d8083990 0x7f91d80863d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91d81a4580 0x7f91d80dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8085e50 con 0x7f91d81a4180 2021-02-10T11:58:01.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8097270 con 0x7f91d8083990 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.993 7f91de635700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8097450 con 0x7f91d81a4580 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60950/0 (socket says 172.21.2.117:60950) 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 -- 172.21.2.117:0/1041931348 learned_addr learned my addr 172.21.2.117:0/1041931348 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91dcc52700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91d81a4580 0x7f91d80dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d7fff700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91d8083990 0x7f91d80863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 -- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91d81a4580 msgr2=0x7f91d80dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91d81a4580 0x7f91d80dc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 -- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91d8083990 msgr2=0x7f91d80863d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91d8083990 0x7f91d80863d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 -- 172.21.2.117:0/1041931348 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d809ed70 con 0x7f91d81a4180 2021-02-10T11:58:01.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d77fe700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 rx=0x7f91cc006f60 tx=0x7f91cc00b970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc0009b0 con 0x7f91d81a4180 2021-02-10T11:58:01.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc00bdf0 con 0x7f91d81a4180 2021-02-10T11:58:01.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f91cc002030 con 0x7f91d81a4180 2021-02-10T11:58:01.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91de635700 1 -- 172.21.2.117:0/1041931348 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d809ed70 con 0x7f91d81a4180 2021-02-10T11:58:01.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.994 7f91de635700 1 -- 172.21.2.117:0/1041931348 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d809ed70 con 0x7f91d81a4180 2021-02-10T11:58:01.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.995 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f91cc006b00 con 0x7f91d81a4180 2021-02-10T11:58:01.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.995 7f91d4ff9700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 0x7f91b801ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.995 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f91cc032450 con 0x7f91d81a4180 2021-02-10T11:58:01.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.995 7f91d7fff700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 0x7f91b801ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:00.996 7f91d7fff700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 0x7f91b801ac10 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.026 7f91de635700 1 -- 172.21.2.117:0/1041931348 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d80daff0 con 0x7f91d81a4180 2021-02-10T11:58:01.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.028 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f91cc006340 con 0x7f91d81a4180 2021-02-10T11:58:01.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.155 7f91de635700 1 -- 172.21.2.117:0/1041931348 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f91d809ed70 con 0x7f91d81a4180 2021-02-10T11:58:01.158 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.155 7f91d4ff9700 1 -- 172.21.2.117:0/1041931348 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v191) v1 ==== 92+0+10 (secure 0 0 0) 0x7f91cc00bdf0 con 0x7f91d81a4180 2021-02-10T11:58:01.159 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:01.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 -- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 msgr2=0x7f91b801ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 0x7f91b801ac10 crc :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 -- 172.21.2.117:0/1041931348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 msgr2=0x7f91d80dbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 rx=0x7f91cc006f60 tx=0x7f91cc00b970).stop 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 -- 172.21.2.117:0/1041931348 shutdown_connections 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f91b8018730 0x7f91b801ac10 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f91d81a4180 0x7f91d80dbe40 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f91d81a4580 0x7f91d80dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 --2- 172.21.2.117:0/1041931348 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f91d8083990 0x7f91d80863d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.157 7f91de635700 1 -- 172.21.2.117:0/1041931348 >> 172.21.2.117:0/1041931348 conn(0x7f91d80b6540 msgr2=0x7f91d808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.158 7f91de635700 1 -- 172.21.2.117:0/1041931348 shutdown_connections 2021-02-10T11:58:01.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.158 7f91de635700 1 -- 172.21.2.117:0/1041931348 wait complete. 2021-02-10T11:58:01.169 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --format=json-pretty osd lspools 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.578 7fb5731ea700 1 Processor -- start 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.578 7fb5731ea700 1 -- start start 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb5731ea700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb56c0d18e0 0x7fb56c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb5731ea700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0ede60 con 0x7fb56c0eb000 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb5731ea700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0d1ce0 con 0x7fb56c0ef3f0 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb5731ea700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c0ef7f0 con 0x7fb56c0d18e0 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb571006700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fb56c0d18e0 0x7fb56c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb570805700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fb56c0eb000 0x7fb56c0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:32976/0 (socket says 172.21.2.117:32976) 2021-02-10T11:58:01.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb570805700 1 -- 172.21.2.117:0/1156441771 learned_addr learned my addr 172.21.2.117:0/1156441771 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb571006700 1 -- 172.21.2.117:0/1156441771 >> v1:172.21.2.125:6789/0 conn(0x7fb56c0ef3f0 legacy=0x7fb56c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb571006700 1 -- 172.21.2.117:0/1156441771 >> v1:172.21.2.117:6789/0 conn(0x7fb56c0eb000 legacy=0x7fb56c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.579 7fb571006700 1 -- 172.21.2.117:0/1156441771 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c1bb180 con 0x7fb56c0d18e0 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb571006700 1 --2- 172.21.2.117:0/1156441771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c0d18e0 0x7fb56c0ea9e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 rx=0x7fb560006650 tx=0x7fb560009a80).ready entity=mon.0 client_cookie=3d7307bca036b5f7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb55ffff700 1 -- 172.21.2.117:0/1156441771 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb56000c0c0 con 0x7fb56c0d18e0 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb55ffff700 1 -- 172.21.2.117:0/1156441771 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56000c340 con 0x7fb56c0d18e0 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb55ffff700 1 -- 172.21.2.117:0/1156441771 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb56000c860 con 0x7fb56c0d18e0 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb5731ea700 1 -- 172.21.2.117:0/1156441771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c0d18e0 msgr2=0x7fb56c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb5731ea700 1 --2- 172.21.2.117:0/1156441771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c0d18e0 0x7fb56c0ea9e0 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 rx=0x7fb560006650 tx=0x7fb560009a80).stop 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb5731ea700 1 -- 172.21.2.117:0/1156441771 shutdown_connections 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb5731ea700 1 --2- 172.21.2.117:0/1156441771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c0d18e0 0x7fb56c0ea9e0 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.580 7fb5731ea700 1 -- 172.21.2.117:0/1156441771 >> 172.21.2.117:0/1156441771 conn(0x7fb56c0ced60 msgr2=0x7fb56c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:01.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.581 7fb5731ea700 1 -- 172.21.2.117:0/1156441771 shutdown_connections 2021-02-10T11:58:01.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.581 7fb5731ea700 1 -- 172.21.2.117:0/1156441771 wait complete. 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.595 7fb5731ea700 1 Processor -- start 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.595 7fb5731ea700 1 -- start start 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb56c0f2b10 0x7fb56c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb56c0b27a0 0x7fb56c097b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0b2ba0 con 0x7fb56c098170 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0f2f10 con 0x7fb56c0b27a0 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c098570 con 0x7fb56c0f2b10 2021-02-10T11:58:01.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60962/0 (socket says 172.21.2.117:60962) 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 -- 172.21.2.117:0/377531790 learned_addr learned my addr 172.21.2.117:0/377531790 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 -- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb56c0f2b10 msgr2=0x7fb56c0b2180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb56c0f2b10 0x7fb56c0b2180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 -- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb56c0b27a0 msgr2=0x7fb56c097b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb56c0b27a0 0x7fb56c097b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 -- 172.21.2.117:0/377531790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c098790 con 0x7fb56c098170 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb571807700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 rx=0x7fb564009940 tx=0x7fb56400bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb564000f90 con 0x7fb56c098170 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56400c2f0 con 0x7fb56c098170 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fb56400c750 con 0x7fb56c098170 2021-02-10T11:58:01.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 -- 172.21.2.117:0/377531790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c098790 con 0x7fb56c098170 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.596 7fb5731ea700 1 -- 172.21.2.117:0/377531790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c1c5760 con 0x7fb56c098170 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.597 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fb564001220 con 0x7fb56c098170 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.597 7fb55dffb700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.597 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5100+0+0 (secure 0 0 0) 0x7fb564032700 con 0x7fb56c098170 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.597 7fb571006700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:01.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.598 7fb571006700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 0x7fb54c01abc0 crc :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:01.631 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.629 7fb5731ea700 1 -- 172.21.2.117:0/377531790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c086140 con 0x7fb56c098170 2021-02-10T11:58:01.633 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.631 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fb564000f40 con 0x7fb56c098170 2021-02-10T11:58:01.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.753 7fb5731ea700 1 -- 172.21.2.117:0/377531790 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb56c0d18e0 con 0x7fb56c098170 2021-02-10T11:58:01.755 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.753 7fb55dffb700 1 -- 172.21.2.117:0/377531790 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v191) v1 ==== 84+0+64 (secure 0 0 0) 0x7fb564001220 con 0x7fb56c098170 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout: "poolnum": 1, 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout: "poolname": "rbd" 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:58:01.756 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.755 7fb5731ea700 1 -- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 msgr2=0x7fb54c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.755 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 0x7fb54c01abc0 crc :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.755 7fb5731ea700 1 -- 172.21.2.117:0/377531790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 msgr2=0x7fb56c1c4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.755 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 secure :-1 s=READY pgs=3224 cs=0 l=1 rev1=1 rx=0x7fb564009940 tx=0x7fb56400bcf0).stop 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 -- 172.21.2.117:0/377531790 shutdown_connections 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fb54c0186c0 0x7fb54c01abc0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fb56c098170 0x7fb56c1c4e50 unknown :-1 s=CLOSED pgs=3224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fb56c0b27a0 0x7fb56c097b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 --2- 172.21.2.117:0/377531790 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fb56c0f2b10 0x7fb56c0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:01.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 -- 172.21.2.117:0/377531790 >> 172.21.2.117:0/377531790 conn(0x7fb56c0f3230 msgr2=0x7fb56c0b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:01.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 -- 172.21.2.117:0/377531790 shutdown_connections 2021-02-10T11:58:01.759 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:01.756 7fb5731ea700 1 -- 172.21.2.117:0/377531790 wait complete. 2021-02-10T11:58:01.766 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2021-02-10T11:58:01.766 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_metadata 8 2021-02-10T11:58:02.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.180 7f14d03e4700 1 Processor -- start 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 -- start start 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14c80d18f0 0x7f14c80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c80eb040 0x7f14c80eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c80eba60 con 0x7f14c80ee3e0 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80d1cf0 con 0x7f14c80d18f0 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14d03e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80eefd0 con 0x7f14c80eb040 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14ce200700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14c80d18f0 0x7f14c80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14ce200700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14c80d18f0 0x7f14c80eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60970/0 (socket says 172.21.2.117:60970) 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.181 7f14ce200700 1 -- 172.21.2.117:0/3031634256 learned_addr learned my addr 172.21.2.117:0/3031634256 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14ce200700 1 -- 172.21.2.117:0/3031634256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c80eb040 msgr2=0x7f14c80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14ce200700 1 --2- 172.21.2.117:0/3031634256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c80eb040 0x7f14c80eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14ce200700 1 -- 172.21.2.117:0/3031634256 >> v1:172.21.2.125:6789/0 conn(0x7f14c80ee3e0 legacy=0x7f14c80ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:02.200 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14ce200700 1 -- 172.21.2.117:0/3031634256 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c80ef1f0 con 0x7f14c80d18f0 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14ce200700 1 --2- 172.21.2.117:0/3031634256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c80d18f0 0x7f14c80eaa20 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f14bc006700 tx=0x7f14bc009b00).ready entity=mon.0 client_cookie=4583039922b7a320 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14cd1fe700 1 -- 172.21.2.117:0/3031634256 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14bc00c0c0 con 0x7f14c80d18f0 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14cd1fe700 1 -- 172.21.2.117:0/3031634256 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc00c340 con 0x7f14c80d18f0 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14cd1fe700 1 -- 172.21.2.117:0/3031634256 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14bc00c820 con 0x7f14c80d18f0 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14d03e4700 1 -- 172.21.2.117:0/3031634256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c80d18f0 msgr2=0x7f14c80eaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.182 7f14d03e4700 1 --2- 172.21.2.117:0/3031634256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c80d18f0 0x7f14c80eaa20 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 rx=0x7f14bc006700 tx=0x7f14bc009b00).stop 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 -- 172.21.2.117:0/3031634256 shutdown_connections 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 --2- 172.21.2.117:0/3031634256 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c80eb040 0x7f14c80eb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 --2- 172.21.2.117:0/3031634256 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c80d18f0 0x7f14c80eaa20 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 -- 172.21.2.117:0/3031634256 >> 172.21.2.117:0/3031634256 conn(0x7f14c80cf060 msgr2=0x7f14c80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 -- 172.21.2.117:0/3031634256 shutdown_connections 2021-02-10T11:58:02.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.183 7f14d03e4700 1 -- 172.21.2.117:0/3031634256 wait complete. 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 Processor -- start 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 -- start start 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c81a3930 0x7f14c80a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14c80a13a0 0x7f14c80a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c807cca0 con 0x7f14c807c7c0 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c818fb70 con 0x7f14c80a13a0 2021-02-10T11:58:02.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.198 7f14d03e4700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80dc6d0 con 0x7f14c81a3930 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60976/0 (socket says 172.21.2.117:60976) 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 -- 172.21.2.117:0/1631998577 learned_addr learned my addr 172.21.2.117:0/1631998577 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 -- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c81a3930 msgr2=0x7f14c80a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c81a3930 0x7f14c80a0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 -- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14c80a13a0 msgr2=0x7f14c80a17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14c80a13a0 0x7f14c80a17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 -- 172.21.2.117:0/1631998577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c807d220 con 0x7f14c807c7c0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14ce200700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 rx=0x7f14bc006f20 tx=0x7f14bc00b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14bc00bd80 con 0x7f14c807c7c0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc000c90 con 0x7f14c807c7c0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c807d220 con 0x7f14c807c7c0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c807d7b0 con 0x7f14c807c7c0 2021-02-10T11:58:02.204 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.199 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f14bc0022b0 con 0x7f14c807c7c0 2021-02-10T11:58:02.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.200 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f14bc009100 con 0x7f14c807c7c0 2021-02-10T11:58:02.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.200 7f14baffd700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 0x7f14b001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:02.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.200 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5100+0+0 (secure 0 0 0) 0x7f14bc032430 con 0x7f14c807c7c0 2021-02-10T11:58:02.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.200 7f14cd9ff700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 0x7f14b001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:02.205 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.201 7f14cd9ff700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 0x7f14b001ac10 crc :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:02.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.231 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c80daff0 con 0x7f14c807c7c0 2021-02-10T11:58:02.235 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.233 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f14bc006390 con 0x7f14c807c7c0 2021-02-10T11:58:02.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.362 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f14c80daff0 con 0x7f14c807c7c0 2021-02-10T11:58:02.779 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.776 7f14baffd700 1 -- 172.21.2.117:0/1631998577 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v192) v1 ==== 133+0+0 (secure 0 0 0) 0x7f14bc009100 con 0x7f14c807c7c0 2021-02-10T11:58:02.780 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_metadata' created 2021-02-10T11:58:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 msgr2=0x7f14b001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:02.782 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 0x7f14b001ac10 crc :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 msgr2=0x7f14c81a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 rx=0x7f14bc006f20 tx=0x7f14bc00b990).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 shutdown_connections 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14c80a13a0 0x7f14c80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14c81a3930 0x7f14c80a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f14b0018730 0x7f14b001ac10 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 --2- 172.21.2.117:0/1631998577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14c807c7c0 0x7f14c81a3310 unknown :-1 s=CLOSED pgs=3226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 >> 172.21.2.117:0/1631998577 conn(0x7f14c8087090 msgr2=0x7f14c808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 shutdown_connections 2021-02-10T11:58:02.783 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:02.780 7f14d03e4700 1 -- 172.21.2.117:0/1631998577 wait complete. 2021-02-10T11:58:02.791 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool create cephfs_data 8 2021-02-10T11:58:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 Processor -- start 2021-02-10T11:58:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 -- start start 2021-02-10T11:58:03.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f978c0eb830 0x7f978c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0ef7f0 con 0x7f978c0d18f0 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0eded0 con 0x7f978c0eb830 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.205 7f97933fe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0d1cf0 con 0x7f978c0ef3f0 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f9790a19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f9791a1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f978c0eb830 0x7f978c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f9791a1b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f978c0eb830 0x7f978c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:37136/0 (socket says 172.21.2.117:37136) 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f979121a700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f978c0d18f0 0x7f978c0eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:54158/0 (socket says 172.21.2.117:54158) 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f9790a19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:58662/0 (socket says 172.21.2.117:58662) 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.206 7f979121a700 1 -- 172.21.2.117:0/3705132123 learned_addr learned my addr 172.21.2.117:0/3705132123 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9791a1b700 1 -- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 msgr2=0x7f978c0eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:03.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9791a1b700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9791a1b700 1 -- 172.21.2.117:0/3705132123 >> v1:172.21.2.125:6789/0 conn(0x7f978c0d18f0 legacy=0x7f978c0eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9791a1b700 1 -- 172.21.2.117:0/3705132123 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c1b8fd0 con 0x7f978c0eb830 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9790a19700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9791a1b700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c0eb830 0x7f978c1b79a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f9784006e50 tx=0x7f9784009e70).ready entity=mon.1 client_cookie=df6358e3e937d389 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9783fff700 1 -- 172.21.2.117:0/3705132123 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978400b510 con 0x7f978c0eb830 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9783fff700 1 -- 172.21.2.117:0/3705132123 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784006790 con 0x7f978c0eb830 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f9783fff700 1 -- 172.21.2.117:0/3705132123 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978400b940 con 0x7f978c0eb830 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f97933fe700 1 -- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c0eb830 msgr2=0x7f978c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.207 7f97933fe700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c0eb830 0x7f978c1b79a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f9784006e50 tx=0x7f9784009e70).stop 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.208 7f97933fe700 1 -- 172.21.2.117:0/3705132123 shutdown_connections 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.208 7f97933fe700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c0eb830 0x7f978c1b79a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.208 7f97933fe700 1 --2- 172.21.2.117:0/3705132123 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c0ef3f0 0x7f978c0eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.208 7f97933fe700 1 -- 172.21.2.117:0/3705132123 >> 172.21.2.117:0/3705132123 conn(0x7f978c0cf060 msgr2=0x7f978c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:03.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.208 7f97933fe700 1 -- 172.21.2.117:0/3705132123 shutdown_connections 2021-02-10T11:58:03.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.209 7f97933fe700 1 -- 172.21.2.117:0/3705132123 wait complete. 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.223 7f97933fe700 1 Processor -- start 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.223 7f97933fe700 1 -- start start 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.223 7f97933fe700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c097470 0x7f978c1a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.223 7f97933fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f97933fe700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c1a7c70 0x7f978c0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f97933fe700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b9260 con 0x7f978c1a7870 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f97933fe700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b9450 con 0x7f978c1a7c70 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f97933fe700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b9670 con 0x7f978c097470 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60988/0 (socket says 172.21.2.117:60988) 2021-02-10T11:58:03.229 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 -- 172.21.2.117:0/915444387 learned_addr learned my addr 172.21.2.117:0/915444387 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9791a1b700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c1a7c70 0x7f978c0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f979121a700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c097470 0x7f978c1a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 -- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c097470 msgr2=0x7f978c1a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c097470 0x7f978c1a7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 -- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c1a7c70 msgr2=0x7f978c0b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c1a7c70 0x7f978c0b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.224 7f9790a19700 1 -- 172.21.2.117:0/915444387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0a7920 con 0x7f978c1a7870 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f9790a19700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f9788006f40 tx=0x7f978800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f978800bd80 con 0x7f978c1a7870 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9788000c90 con 0x7f978c1a7870 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f9788002220 con 0x7f978c1a7870 2021-02-10T11:58:03.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f97933fe700 1 -- 172.21.2.117:0/915444387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0a7920 con 0x7f978c1a7870 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.225 7f97933fe700 1 -- 172.21.2.117:0/915444387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c0a7eb0 con 0x7f978c1a7870 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.226 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f978800a000 con 0x7f978c1a7870 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.226 7f9781ffb700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.226 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5454+0+0 (secure 0 0 0) 0x7f9788032500 con 0x7f978c1a7870 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.226 7f979121a700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:03.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.226 7f979121a700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 0x7f976c01abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:03.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.255 7f97933fe700 1 -- 172.21.2.117:0/915444387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0bbaf0 con 0x7f978c1a7870 2021-02-10T11:58:03.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.258 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9788000940 con 0x7f978c1a7870 2021-02-10T11:58:03.388 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.386 7f97933fe700 1 -- 172.21.2.117:0/915444387 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"} v 0) v1 -- 0x7f978c0bbaf0 con 0x7f978c1a7870 2021-02-10T11:58:03.787 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.785 7f9781ffb700 1 -- 172.21.2.117:0/915444387 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v194) v1 ==== 125+0+0 (secure 0 0 0) 0x7f978800a000 con 0x7f978c1a7870 2021-02-10T11:58:03.803 INFO:teuthology.orchestra.run.gibba017.stderr:pool 'cephfs_data' created 2021-02-10T11:58:03.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.803 7f97933fe700 1 -- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 msgr2=0x7f976c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.803 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 0x7f976c01abc0 crc :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.803 7f97933fe700 1 -- 172.21.2.117:0/915444387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 msgr2=0x7f978c0b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:03.805 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.803 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 secure :-1 s=READY pgs=3227 cs=0 l=1 rev1=1 rx=0x7f9788006f40 tx=0x7f978800b990).stop 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 -- 172.21.2.117:0/915444387 shutdown_connections 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f976c0186c0 0x7f976c01abc0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f978c1a7870 0x7f978c0b8620 unknown :-1 s=CLOSED pgs=3227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f978c1a7c70 0x7f978c0b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 --2- 172.21.2.117:0/915444387 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f978c097470 0x7f978c1a7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:03.806 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 -- 172.21.2.117:0/915444387 >> 172.21.2.117:0/915444387 conn(0x7f978c0bbea0 msgr2=0x7f978c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:03.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 -- 172.21.2.117:0/915444387 shutdown_connections 2021-02-10T11:58:03.807 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:03.804 7f97933fe700 1 -- 172.21.2.117:0/915444387 wait complete. 2021-02-10T11:58:03.814 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 Processor -- start 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 -- start start 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46481b07e0 0x7f46480e4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f46480e6160 0x7f46480e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480e77e0 con 0x7f46480e6b80 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46480e5cd0 con 0x7f46481b07e0 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.230 7f464db7c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46480e54a0 con 0x7f46480e6160 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46481b07e0 0x7f46480e4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f46481b07e0 0x7f46480e4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:60994/0 (socket says 172.21.2.117:60994) 2021-02-10T11:58:04.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 -- 172.21.2.117:0/441145040 learned_addr learned my addr 172.21.2.117:0/441145040 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 -- 172.21.2.117:0/441145040 >> v2:172.21.2.125:3300/0 conn(0x7f46480e6160 msgr2=0x7f46480e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 --2- 172.21.2.117:0/441145040 >> v2:172.21.2.125:3300/0 conn(0x7f46480e6160 0x7f46480e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 -- 172.21.2.117:0/441145040 >> v1:172.21.2.125:6789/0 conn(0x7f46480e6b80 legacy=0x7f46480e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 -- 172.21.2.117:0/441145040 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480e8d70 con 0x7f46481b07e0 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46477fe700 1 --2- 172.21.2.117:0/441145040 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46481b07e0 0x7f46480e4e80 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f4638006700 tx=0x7f4638009b80).ready entity=mon.0 client_cookie=ad741121085395a6 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46467fc700 1 -- 172.21.2.117:0/441145040 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f463800c0c0 con 0x7f46481b07e0 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46467fc700 1 -- 172.21.2.117:0/441145040 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463800c400 con 0x7f46481b07e0 2021-02-10T11:58:04.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f46467fc700 1 -- 172.21.2.117:0/441145040 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f463800c920 con 0x7f46481b07e0 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f464db7c700 1 -- 172.21.2.117:0/441145040 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46481b07e0 msgr2=0x7f46480e4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.231 7f464db7c700 1 --2- 172.21.2.117:0/441145040 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46481b07e0 0x7f46480e4e80 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f4638006700 tx=0x7f4638009b80).stop 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 -- 172.21.2.117:0/441145040 shutdown_connections 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 --2- 172.21.2.117:0/441145040 >> v2:172.21.2.125:3300/0 conn(0x7f46480e6160 0x7f46480e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 --2- 172.21.2.117:0/441145040 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f46481b07e0 0x7f46480e4e80 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 -- 172.21.2.117:0/441145040 >> 172.21.2.117:0/441145040 conn(0x7f46480ce4f0 msgr2=0x7f46480e4320 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 -- 172.21.2.117:0/441145040 shutdown_connections 2021-02-10T11:58:04.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.232 7f464db7c700 1 -- 172.21.2.117:0/441145040 wait complete. 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 Processor -- start 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 -- start start 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4648092d80 0x7f464810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4648093180 0x7f4648192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648196550 con 0x7f464810d300 2021-02-10T11:58:04.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648196130 con 0x7f4648092d80 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.247 7f464db7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4648197070 con 0x7f4648093180 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32768/0 (socket says 172.21.2.117:32768) 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 -- 172.21.2.117:0/1229928693 learned_addr learned my addr 172.21.2.117:0/1229928693 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f46477fe700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4648092d80 0x7f464810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4647fff700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4648093180 0x7f4648192a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 -- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4648093180 msgr2=0x7f4648192a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4648093180 0x7f4648192a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 -- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4648092d80 msgr2=0x7f464810cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4648092d80 0x7f464810cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 -- 172.21.2.117:0/1229928693 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648197290 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f4646ffd700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f4630006f00 tx=0x7f463000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4630006b90 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463000bd30 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.248 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f46300020d0 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f464db7c700 1 -- 172.21.2.117:0/1229928693 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4648197290 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f464db7c700 1 -- 172.21.2.117:0/1229928693 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4648193370 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4630007020 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f463ffff700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 0x7f462c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f46300326d0 con 0x7f464810d300 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.249 7f46477fe700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 0x7f462c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.250 7f46477fe700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 0x7f462c01abc0 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.280 7f464db7c700 1 -- 172.21.2.117:0/1229928693 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4648108ab0 con 0x7f464810d300 2021-02-10T11:58:04.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.283 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4630007ef0 con 0x7f464810d300 2021-02-10T11:58:04.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.412 7f464db7c700 1 -- 172.21.2.117:0/1229928693 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f46480e77e0 con 0x7f464810d300 2021-02-10T11:58:04.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.415 7f463ffff700 1 -- 172.21.2.117:0/1229928693 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"data": "cephfs_data", "prefix": "fs new", "force": true, "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 42 and data pool 43 v167) v1 ==== 189+0+0 (secure 0 0 0) 0x7f4630000940 con 0x7f464810d300 2021-02-10T11:58:04.421 INFO:teuthology.orchestra.run.gibba017.stderr:new fs with metadata pool 42 and data pool 43 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 msgr2=0x7f462c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 0x7f462c01abc0 crc :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 msgr2=0x7f4648196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 rx=0x7f4630006f00 tx=0x7f463000b940).stop 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 shutdown_connections 2021-02-10T11:58:04.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f462c018730 0x7f462c01abc0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f464810d300 0x7f4648196a50 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4648093180 0x7f4648192a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 --2- 172.21.2.117:0/1229928693 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4648092d80 0x7f464810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 >> 172.21.2.117:0/1229928693 conn(0x7f464819f010 msgr2=0x7f46481a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 shutdown_connections 2021-02-10T11:58:04.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.421 7f464db7c700 1 -- 172.21.2.117:0/1229928693 wait complete. 2021-02-10T11:58:04.431 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.841 7f73d8f37700 1 Processor -- start 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.841 7f73d8f37700 1 -- start start 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.842 7f73d8f37700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 0x7f73d40eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.842 7f73d8f37700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f73d40eb7f0 0x7f73d41b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.842 7f73d8f37700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d40ede60 con 0x7f73d40d18e0 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.842 7f73d8f37700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d40ef7f0 con 0x7f73d40eb7f0 2021-02-10T11:58:04.861 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.842 7f73d8f37700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40d1ce0 con 0x7f73d40ef3f0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2b13700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f73d40d18e0 0x7f73d40ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:54180/0 (socket says 172.21.2.117:54180) 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d3314700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f73d40eb7f0 0x7f73d41b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 0x7f73d40eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2b13700 1 -- 172.21.2.117:0/4022959492 learned_addr learned my addr 172.21.2.117:0/4022959492 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2312700 1 -- 172.21.2.117:0/4022959492 >> v2:172.21.2.125:3300/0 conn(0x7f73d40eb7f0 msgr2=0x7f73d41b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2312700 1 --2- 172.21.2.117:0/4022959492 >> v2:172.21.2.125:3300/0 conn(0x7f73d40eb7f0 0x7f73d41b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2312700 1 -- 172.21.2.117:0/4022959492 >> v1:172.21.2.125:6789/0 conn(0x7f73d40d18e0 legacy=0x7f73d40ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d2312700 1 -- 172.21.2.117:0/4022959492 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d41b8f90 con 0x7f73d40ef3f0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.843 7f73d1b11700 1 -- 172.21.2.117:0/4022959492 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617057838 0 0) 0x7f73c40010c0 con 0x7f73d40d18e0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d2312700 1 --2- 172.21.2.117:0/4022959492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 0x7f73d40eb1d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f73bc006650 tx=0x7f73bc009a80).ready entity=mon.2 client_cookie=eec330f4667d0b39 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d1b11700 1 -- 172.21.2.117:0/4022959492 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73bc00c0c0 con 0x7f73d40ef3f0 2021-02-10T11:58:04.862 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d1b11700 1 -- 172.21.2.117:0/4022959492 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc00c340 con 0x7f73d40ef3f0 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d1b11700 1 -- 172.21.2.117:0/4022959492 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73bc00c860 con 0x7f73d40ef3f0 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d8f37700 1 -- 172.21.2.117:0/4022959492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 msgr2=0x7f73d40eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.844 7f73d8f37700 1 --2- 172.21.2.117:0/4022959492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 0x7f73d40eb1d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f73bc006650 tx=0x7f73bc009a80).stop 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 -- 172.21.2.117:0/4022959492 shutdown_connections 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 --2- 172.21.2.117:0/4022959492 >> v2:172.21.2.125:3300/0 conn(0x7f73d40eb7f0 0x7f73d41b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 --2- 172.21.2.117:0/4022959492 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d40ef3f0 0x7f73d40eb1d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 -- 172.21.2.117:0/4022959492 >> 172.21.2.117:0/4022959492 conn(0x7f73d40ced60 msgr2=0x7f73d40cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 -- 172.21.2.117:0/4022959492 shutdown_connections 2021-02-10T11:58:04.863 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.845 7f73d8f37700 1 -- 172.21.2.117:0/4022959492 wait complete. 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.859 7f73d8f37700 1 Processor -- start 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.859 7f73d8f37700 1 -- start start 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d4090c40 0x7f73d4091040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73d408c1c0 0x7f73d408c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40906c0 con 0x7f73d41a3b50 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d41a3f50 con 0x7f73d408c1c0 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d8f37700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d4091e40 con 0x7f73d4090c40 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32778/0 (socket says 172.21.2.117:32778) 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 -- 172.21.2.117:0/1961649165 learned_addr learned my addr 172.21.2.117:0/1961649165 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 -- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d4090c40 msgr2=0x7f73d4091040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.865 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d4090c40 0x7f73d4091040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 -- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73d408c1c0 msgr2=0x7f73d408c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73d408c1c0 0x7f73d408c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 -- 172.21.2.117:0/1961649165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d408cbe0 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.860 7f73d2b13700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 rx=0x7f73c400a4f0 tx=0x7f73c400a890).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c400ac80 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40070d0 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f73c4004530 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d408cbe0 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d408d170 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f73c400ac80 con 0x7f73d41a3b50 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 0x7f73b801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:04.866 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.861 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f73c4032660 con 0x7f73d41a3b50 2021-02-10T11:58:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.862 7f73d2312700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 0x7f73b801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:04.867 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.862 7f73d2312700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 0x7f73b801abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:04.894 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.892 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d4087290 con 0x7f73d41a3b50 2021-02-10T11:58:04.896 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:04.894 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f73c40047f0 con 0x7f73d41a3b50 2021-02-10T11:58:05.021 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.019 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73d40ede60 con 0x7f73d41a3b50 2021-02-10T11:58:05.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.020 7f73cb7fe700 1 -- 172.21.2.117:0/1961649165 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) v1 ==== 74+0+14253 (secure 0 0 0) 0x7f73c400ac80 con 0x7f73d41a3b50 2021-02-10T11:58:05.022 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:05.022 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":195,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:04.413073","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.021 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 msgr2=0x7f73b801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.021 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 0x7f73b801abc0 crc :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 msgr2=0x7f73d4091820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 secure :-1 s=READY pgs=3230 cs=0 l=1 rev1=1 rx=0x7f73c400a4f0 tx=0x7f73c400a890).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 shutdown_connections 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f73d408c1c0 0x7f73d408c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f73d4090c40 0x7f73d4091040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f73b80186c0 0x7f73b801abc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 --2- 172.21.2.117:0/1961649165 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f73d41a3b50 0x7f73d4091820 unknown :-1 s=CLOSED pgs=3230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 >> 172.21.2.117:0/1961649165 conn(0x7f73d40874c0 msgr2=0x7f73d4199ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:05.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 shutdown_connections 2021-02-10T11:58:05.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.022 7f73d8f37700 1 -- 172.21.2.117:0/1961649165 wait complete. 2021-02-10T11:58:05.032 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:05.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 Processor -- start 2021-02-10T11:58:05.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 -- start start 2021-02-10T11:58:05.464 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fad241b6960 0x7fad241b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad240ede60 con 0x7fad240ef3f0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad240eb9c0 con 0x7fad240d18e0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad2c0fd700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad240ef7f0 con 0x7fad241b6960 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad29718700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fad240ef3f0 0x7fad240eb1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33036/0 (socket says 172.21.2.117:33036) 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.446 7fad29718700 1 -- 172.21.2.117:0/1236899914 learned_addr learned my addr 172.21.2.117:0/1236899914 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2a71a700 1 --2- 172.21.2.117:0/1236899914 >> v2:172.21.2.117:3300/0 conn(0x7fad241b6960 0x7fad241b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad28f17700 1 -- 172.21.2.117:0/1236899914 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770572345 0 0) 0x7fad20001180 con 0x7fad240ef3f0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad28f17700 1 -- 172.21.2.117:0/1236899914 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad10002fe0 con 0x7fad240ef3f0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2a71a700 1 -- 172.21.2.117:0/1236899914 >> v1:172.21.2.125:6789/0 conn(0x7fad240d18e0 legacy=0x7fad240ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2a71a700 1 -- 172.21.2.117:0/1236899914 >> v1:172.21.2.117:6789/0 conn(0x7fad240ef3f0 legacy=0x7fad240eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2a71a700 1 -- 172.21.2.117:0/1236899914 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad240d1d20 con 0x7fad241b6960 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2a71a700 1 --2- 172.21.2.117:0/1236899914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad241b6960 0x7fad241b8d70 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 rx=0x7fad1c006e30 tx=0x7fad1c009e60).ready entity=mon.0 client_cookie=19000def4125f107 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:05.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad28f17700 1 -- 172.21.2.117:0/1236899914 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad1c006a50 con 0x7fad241b6960 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad28f17700 1 -- 172.21.2.117:0/1236899914 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00b300 con 0x7fad241b6960 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad28f17700 1 -- 172.21.2.117:0/1236899914 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad1c00b7a0 con 0x7fad241b6960 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2c0fd700 1 -- 172.21.2.117:0/1236899914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad241b6960 msgr2=0x7fad241b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.447 7fad2c0fd700 1 --2- 172.21.2.117:0/1236899914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad241b6960 0x7fad241b8d70 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 rx=0x7fad1c006e30 tx=0x7fad1c009e60).stop 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.448 7fad2c0fd700 1 -- 172.21.2.117:0/1236899914 shutdown_connections 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.448 7fad2c0fd700 1 --2- 172.21.2.117:0/1236899914 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad241b6960 0x7fad241b8d70 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.448 7fad2c0fd700 1 -- 172.21.2.117:0/1236899914 >> 172.21.2.117:0/1236899914 conn(0x7fad240ced60 msgr2=0x7fad240cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.448 7fad2c0fd700 1 -- 172.21.2.117:0/1236899914 shutdown_connections 2021-02-10T11:58:05.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.448 7fad2c0fd700 1 -- 172.21.2.117:0/1236899914 wait complete. 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 Processor -- start 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 -- start start 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad240980c0 0x7fad2419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad2419dd40 0x7fad2419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad240b1f30 con 0x7fad240984c0 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419d830 con 0x7fad2419dd40 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad2c0fd700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad241a55e0 con 0x7fad240980c0 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32790/0 (socket says 172.21.2.117:32790) 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 -- 172.21.2.117:0/3713687174 learned_addr learned my addr 172.21.2.117:0/3713687174 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:05.468 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 -- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad240980c0 msgr2=0x7fad2419ca80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad240980c0 0x7fad2419ca80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 -- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad2419dd40 msgr2=0x7fad2419e140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad2419dd40 0x7fad2419e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.463 7fad29718700 1 -- 172.21.2.117:0/3713687174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2419e760 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad29718700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7fad2000a5a0 tx=0x7fad2000a940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad2000ad30 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad200070d0 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fad20004620 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2419e760 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2419e760 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fad200043a0 con 0x7fad240984c0 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 0x7fad0c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:05.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.464 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fad20032690 con 0x7fad240984c0 2021-02-10T11:58:05.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.465 7fad29f19700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 0x7fad0c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:05.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.465 7fad29f19700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 0x7fad0c01abc0 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:05.498 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.496 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad24087210 con 0x7fad240984c0 2021-02-10T11:58:05.500 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.498 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fad200009b0 con 0x7fad240984c0 2021-02-10T11:58:05.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.624 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad24087210 con 0x7fad240984c0 2021-02-10T11:58:05.626 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.625 7fad1a7fc700 1 -- 172.21.2.117:0/3713687174 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fad200043a0 con 0x7fad240984c0 2021-02-10T11:58:05.627 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:05.627 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:05.628 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 msgr2=0x7fad0c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 0x7fad0c01abc0 crc :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 msgr2=0x7fad2419d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 rx=0x7fad2000a5a0 tx=0x7fad2000a940).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 shutdown_connections 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fad0c0186c0 0x7fad0c01abc0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fad240984c0 0x7fad2419d0a0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fad2419dd40 0x7fad2419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.626 7fad2c0fd700 1 --2- 172.21.2.117:0/3713687174 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fad240980c0 0x7fad2419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.627 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 >> 172.21.2.117:0/3713687174 conn(0x7fad2419f610 msgr2=0x7fad241a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.627 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 shutdown_connections 2021-02-10T11:58:05.629 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:05.627 7fad2c0fd700 1 -- 172.21.2.117:0/3713687174 wait complete. 2021-02-10T11:58:05.637 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2021-02-10T11:58:06.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.052 7f77641fa700 1 Processor -- start 2021-02-10T11:58:06.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.052 7f77641fa700 1 -- start start 2021-02-10T11:58:06.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f77641fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c0d18f0 0x7f775c0eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f77641fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f775c0ef3f0 0x7f775c1b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f77641fa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c0ede60 con 0x7f775c0eb040 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f77641fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c088020 con 0x7f775c0ef3f0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f77641fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c0d1d30 con 0x7f775c0d18f0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f775c0ef3f0 0x7f775c1b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f775c0ef3f0 0x7f775c1b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32798/0 (socket says 172.21.2.117:32798) 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 -- 172.21.2.117:0/4166635652 learned_addr learned my addr 172.21.2.117:0/4166635652 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 -- 172.21.2.117:0/4166635652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c0d18f0 msgr2=0x7f775c0eaa20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 --2- 172.21.2.117:0/4166635652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c0d18f0 0x7f775c0eaa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 -- 172.21.2.117:0/4166635652 >> v1:172.21.2.125:6789/0 conn(0x7f775c0eb040 legacy=0x7f775c0eb440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 -- 172.21.2.117:0/4166635652 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c1b8fd0 con 0x7f775c0ef3f0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7762817700 1 --2- 172.21.2.117:0/4166635652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c0ef3f0 0x7f775c1b79a0 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f7754006f00 tx=0x7f7754009f10).ready entity=mon.0 client_cookie=8aee234904e6e827 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.053 7f7761014700 1 -- 172.21.2.117:0/4166635652 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775400b3b0 con 0x7f775c0ef3f0 2021-02-10T11:58:06.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f7761014700 1 -- 172.21.2.117:0/4166635652 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400b630 con 0x7f775c0ef3f0 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f7761014700 1 -- 172.21.2.117:0/4166635652 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775400bb10 con 0x7f775c0ef3f0 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 -- 172.21.2.117:0/4166635652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c0ef3f0 msgr2=0x7f775c1b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 --2- 172.21.2.117:0/4166635652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c0ef3f0 0x7f775c1b79a0 secure :-1 s=READY pgs=3234 cs=0 l=1 rev1=1 rx=0x7f7754006f00 tx=0x7f7754009f10).stop 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 -- 172.21.2.117:0/4166635652 shutdown_connections 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 --2- 172.21.2.117:0/4166635652 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c0ef3f0 0x7f775c1b79a0 unknown :-1 s=CLOSED pgs=3234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 --2- 172.21.2.117:0/4166635652 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c0d18f0 0x7f775c0eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 -- 172.21.2.117:0/4166635652 >> 172.21.2.117:0/4166635652 conn(0x7f775c0cf060 msgr2=0x7f775c0cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 -- 172.21.2.117:0/4166635652 shutdown_connections 2021-02-10T11:58:06.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.054 7f77641fa700 1 -- 172.21.2.117:0/4166635652 wait complete. 2021-02-10T11:58:06.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 Processor -- start 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 -- start start 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c07c7c0 0x7f775c1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775c0a13a0 0x7f775c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c07cca0 con 0x7f775c1a3930 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c18fb70 con 0x7f775c0a13a0 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f77641fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c0dc6d0 con 0x7f775c07c7c0 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f7761815700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f7761815700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32802/0 (socket says 172.21.2.117:32802) 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.069 7f7761815700 1 -- 172.21.2.117:0/68625802 learned_addr learned my addr 172.21.2.117:0/68625802 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:06.074 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 -- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c07c7c0 msgr2=0x7f775c1a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c07c7c0 0x7f775c1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 -- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775c0a13a0 msgr2=0x7f775c0a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775c0a13a0 0x7f775c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 -- 172.21.2.117:0/68625802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c07d220 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f7761815700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 rx=0x7f775000b5b0 tx=0x7f775000b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f775000bd40 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750006240 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7750005720 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f77641fa700 1 -- 172.21.2.117:0/68625802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c07d220 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.070 7f77641fa700 1 -- 172.21.2.117:0/68625802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c07d7b0 con 0x7f775c1a3930 2021-02-10T11:58:06.075 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.071 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f77500054a0 con 0x7f775c1a3930 2021-02-10T11:58:06.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.071 7f774effd700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 0x7f774401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.071 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7750032590 con 0x7f775c1a3930 2021-02-10T11:58:06.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.071 7f7762016700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 0x7f774401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.076 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.072 7f7762016700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 0x7f774401abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.102 7f77641fa700 1 -- 172.21.2.117:0/68625802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c18fb00 con 0x7f775c1a3930 2021-02-10T11:58:06.108 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.105 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7750000940 con 0x7f775c1a3930 2021-02-10T11:58:06.241 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.239 7f77641fa700 1 -- 172.21.2.117:0/68625802 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f775c18fb00 con 0x7f775c1a3930 2021-02-10T11:58:06.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.422 7f774effd700 1 -- 172.21.2.117:0/68625802 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v170) v1 ==== 118+0+0 (secure 0 0 0) 0x7f77500054a0 con 0x7f775c1a3930 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 msgr2=0x7f774401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 0x7f774401abc0 crc :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 msgr2=0x7f775c0a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 secure :-1 s=READY pgs=3235 cs=0 l=1 rev1=1 rx=0x7f775000b5b0 tx=0x7f775000b950).stop 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 shutdown_connections 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77440186c0 0x7f774401abc0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f775c1a3930 0x7f775c0a0d80 unknown :-1 s=CLOSED pgs=3235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775c0a13a0 0x7f775c0a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.460 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 --2- 172.21.2.117:0/68625802 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f775c07c7c0 0x7f775c1a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 >> 172.21.2.117:0/68625802 conn(0x7f775c087090 msgr2=0x7f775c08d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:06.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 shutdown_connections 2021-02-10T11:58:06.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.457 7f77641fa700 1 -- 172.21.2.117:0/68625802 wait complete. 2021-02-10T11:58:06.468 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 Processor -- start 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 -- start start 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fd3d00ef3f0 0x7fd3d00eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3d00eb7f0 0x7fd3d01b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00ede60 con 0x7fd3d00d18e0 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d00ef7f0 con 0x7fd3d00eb7f0 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.878 7fd3d5c56700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d00d1ce0 con 0x7fd3d00ef3f0 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3d00eb7f0 0x7fd3d01b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cf7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd3d00d18e0 0x7fd3d00ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33060/0 (socket says 172.21.2.117:33060) 2021-02-10T11:58:06.897 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cf7fe700 1 -- 172.21.2.117:0/95209283 learned_addr learned my addr 172.21.2.117:0/95209283 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 -- 172.21.2.117:0/95209283 >> v2:172.21.2.125:3300/0 conn(0x7fd3d00ef3f0 msgr2=0x7fd3d00eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 --2- 172.21.2.117:0/95209283 >> v2:172.21.2.125:3300/0 conn(0x7fd3d00ef3f0 0x7fd3d00eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 -- 172.21.2.117:0/95209283 >> v1:172.21.2.117:6789/0 conn(0x7fd3d00d18e0 legacy=0x7fd3d00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 -- 172.21.2.117:0/95209283 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d01b8f90 con 0x7fd3d00eb7f0 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.879 7fd3cffff700 1 --2- 172.21.2.117:0/95209283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d00eb7f0 0x7fd3d01b7960 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7fd3c8006e30 tx=0x7fd3c8009e60).ready entity=mon.0 client_cookie=cf55aac7452ad955 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3ce7fc700 1 -- 172.21.2.117:0/95209283 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c8006a50 con 0x7fd3d00eb7f0 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3ce7fc700 1 -- 172.21.2.117:0/95209283 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c800b300 con 0x7fd3d00eb7f0 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3ce7fc700 1 -- 172.21.2.117:0/95209283 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c800b820 con 0x7fd3d00eb7f0 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 -- 172.21.2.117:0/95209283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d00eb7f0 msgr2=0x7fd3d01b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 --2- 172.21.2.117:0/95209283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d00eb7f0 0x7fd3d01b7960 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 rx=0x7fd3c8006e30 tx=0x7fd3c8009e60).stop 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 -- 172.21.2.117:0/95209283 shutdown_connections 2021-02-10T11:58:06.898 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 --2- 172.21.2.117:0/95209283 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d00eb7f0 0x7fd3d01b7960 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:06.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 --2- 172.21.2.117:0/95209283 >> v2:172.21.2.125:3300/0 conn(0x7fd3d00ef3f0 0x7fd3d00eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 -- 172.21.2.117:0/95209283 >> 172.21.2.117:0/95209283 conn(0x7fd3d00ced60 msgr2=0x7fd3d00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:06.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 -- 172.21.2.117:0/95209283 shutdown_connections 2021-02-10T11:58:06.899 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.880 7fd3d5c56700 1 -- 172.21.2.117:0/95209283 wait complete. 2021-02-10T11:58:06.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.895 7fd3d5c56700 1 Processor -- start 2021-02-10T11:58:06.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.895 7fd3d5c56700 1 -- start start 2021-02-10T11:58:06.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3d00e9560 0x7fd3d00e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3d00e9960 0x7fd3d00e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d0196810 con 0x7fd3d0196390 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d0098ad0 con 0x7fd3d00e9960 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3d5c56700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00e8a40 con 0x7fd3d00e9560 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32816/0 (socket says 172.21.2.117:32816) 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 -- 172.21.2.117:0/861398384 learned_addr learned my addr 172.21.2.117:0/861398384 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 -- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3d00e9560 msgr2=0x7fd3d00e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3d00e9560 0x7fd3d00e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 -- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3d00e9960 msgr2=0x7fd3d00e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3d00e9960 0x7fd3d00e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 -- 172.21.2.117:0/861398384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00e8c60 con 0x7fd3d0196390 2021-02-10T11:58:06.901 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.896 7fd3cf7fe700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7fd3c0001410 tx=0x7fd3c000a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c000ad40 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c00009b0 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd3c0004620 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d00e8c60 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d019cd70 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd3c00043a0 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.897 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd3c0032570 con 0x7fd3d0196390 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.898 7fd3ceffd700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:06.902 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.898 7fd3ceffd700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 0x7fd3b401abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:06.930 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.928 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d0087210 con 0x7fd3d0196390 2021-02-10T11:58:06.932 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:06.931 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd3c0030d50 con 0x7fd3d0196390 2021-02-10T11:58:07.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.059 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d00ede60 con 0x7fd3d0196390 2021-02-10T11:58:07.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.059 7fd3c7fff700 1 -- 172.21.2.117:0/861398384 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fd3c000ad40 con 0x7fd3d0196390 2021-02-10T11:58:07.062 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:07.062 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13806,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3326472561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3326472561},{"type":"v1","addr":"172.21.2.117:6833","nonce":3326472561}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":166}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:58:04.413160","modified":"2021-02-10 11:58:06.419787","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9775":{"gid":9775,"name":"b","rank":0,"incarnation":167,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1289009861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1289009861},{"type":"v1","addr":"172.21.2.125:6835","nonce":1289009861}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:58:07.062 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 170 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 msgr2=0x7fd3b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 0x7fd3b401abc0 crc :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 msgr2=0x7fd3d00e8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 rx=0x7fd3c0001410 tx=0x7fd3c000a950).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 shutdown_connections 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3d00e9960 0x7fd3d00e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3d00e9560 0x7fd3d00e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3b40186c0 0x7fd3b401abc0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 --2- 172.21.2.117:0/861398384 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3d0196390 0x7fd3d00e8f40 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.063 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 >> 172.21.2.117:0/861398384 conn(0x7fd3d019f610 msgr2=0x7fd3d01a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:07.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 shutdown_connections 2021-02-10T11:58:07.064 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.061 7fd3d5c56700 1 -- 172.21.2.117:0/861398384 wait complete. 2021-02-10T11:58:07.072 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.483 7fe1e2cc1700 1 Processor -- start 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.483 7fe1e2cc1700 1 -- start start 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e2cc1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0d18e0 0x7fe1dc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e2cc1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fe1dc0eb000 0x7fe1dc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e2cc1700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc0ede60 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e2cc1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0eba20 con 0x7fe1dc0eb000 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e2cc1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0d1ce0 con 0x7fe1dc0d18e0 2021-02-10T11:58:07.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e12de700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fe1dc0ee3e0 0x7fe1dc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33074/0 (socket says 172.21.2.117:33074) 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e12de700 1 -- 172.21.2.117:0/4150933853 learned_addr learned my addr 172.21.2.117:0/4150933853 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350461241 0 0) 0x7fe1d4001180 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c8002fe0 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1e0add700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0d18e0 0x7fe1dc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1dbfff700 1 --2- 172.21.2.117:0/4150933853 >> v2:172.21.2.125:3300/0 conn(0x7fe1dc0eb000 0x7fe1dc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.484 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3269180870 0 0) 0x7fe1d4001180 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0d18e0 msgr2=0x7fe1dc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0d18e0 0x7fe1dc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 >> v2:172.21.2.125:3300/0 conn(0x7fe1dc0eb000 msgr2=0x7fe1dc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 --2- 172.21.2.117:0/4150933853 >> v2:172.21.2.125:3300/0 conn(0x7fe1dc0eb000 0x7fe1dc0eb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0eefd0 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fe1d4001680 con 0x7fe1dc0ee3e0 2021-02-10T11:58:07.504 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 >> v1:172.21.2.117:6789/0 conn(0x7fe1dc0ee3e0 legacy=0x7fe1dc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1c8003ae0 0x7fe1c8005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1c8006830 0x7fe1c8008d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1c8009520 0x7fe1c800ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c800c2f0 con 0x7fe1c8003ae0 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80504e0 con 0x7fe1c8009520 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8050700 con 0x7fe1c8006830 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1c8003ae0 0x7fe1c8005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e12de700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1c8009520 0x7fe1c800ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1dbfff700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1c8006830 0x7fe1c8008d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 -- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1c8006830 msgr2=0x7fe1c8008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1c8006830 0x7fe1c8008d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 -- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1c8009520 msgr2=0x7fe1c800ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1c8009520 0x7fe1c800ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 -- 172.21.2.117:0/4150933853 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80509a0 con 0x7fe1c8003ae0 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e0add700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1c8003ae0 0x7fe1c8005f90 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7fe1c4006a90 tx=0x7fe1c4009a90).ready entity=mon.0 client_cookie=556b1a06ff48793d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c400c0c0 con 0x7fe1c8003ae0 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c400c340 con 0x7fe1c8003ae0 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1db7fe700 1 -- 172.21.2.117:0/4150933853 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1c400c860 con 0x7fe1c8003ae0 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e2cc1700 1 -- 172.21.2.117:0/4150933853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1c8003ae0 msgr2=0x7fe1c8005f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e2cc1700 1 --2- 172.21.2.117:0/4150933853 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1c8003ae0 0x7fe1c8005f90 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7fe1c4006a90 tx=0x7fe1c4009a90).stop 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.485 7fe1e12de700 1 -- 172.21.2.117:0/4150933853 reap_dead start 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.486 7fe1e2cc1700 1 -- 172.21.2.117:0/4150933853 shutdown_connections 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.486 7fe1e2cc1700 1 -- 172.21.2.117:0/4150933853 >> 172.21.2.117:0/4150933853 conn(0x7fe1dc0ced60 msgr2=0x7fe1dc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.486 7fe1e2cc1700 1 -- 172.21.2.117:0/4150933853 shutdown_connections 2021-02-10T11:58:07.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.486 7fe1e2cc1700 1 -- 172.21.2.117:0/4150933853 wait complete. 2021-02-10T11:58:07.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 Processor -- start 2021-02-10T11:58:07.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 -- start start 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1dc0b77e0 0x7fe1dc0b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0b7310 0x7fe1dc0aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0ab8c0 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0ac460 con 0x7fe1dc0b77e0 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e2cc1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0ac680 con 0x7fe1dc0b7310 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e12de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e12de700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32838/0 (socket says 172.21.2.117:32838) 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.501 7fe1e12de700 1 -- 172.21.2.117:0/2398950637 learned_addr learned my addr 172.21.2.117:0/2398950637 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 -- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0b7310 msgr2=0x7fe1dc0aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0b7310 0x7fe1dc0aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 -- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1dc0b77e0 msgr2=0x7fe1dc0b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1dc0b77e0 0x7fe1dc0b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 -- 172.21.2.117:0/2398950637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0ac8a0 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e12de700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 rx=0x7fe1d4006fc0 tx=0x7fe1d400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d400bd80 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4007670 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc0ac8a0 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc0ace30 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.502 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fe1d4007b90 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.503 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fe1d40060d0 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.503 7fe1d97fa700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 0x7fe1c001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:07.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.503 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fe1d4032940 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.504 7fe1e0add700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 0x7fe1c001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:07.510 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.505 7fe1e0add700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 0x7fe1c001abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:07.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.533 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc087290 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.536 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fe1d4007670 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.662 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.660 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1dc0ede60 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.663 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.661 7fe1d97fa700 1 -- 172.21.2.117:0/2398950637 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7fe1d4006040 con 0x7fe1dc0ab2c0 2021-02-10T11:58:07.663 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:07.664 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:07.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 msgr2=0x7fe1c001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 0x7fe1c001abc0 crc :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 msgr2=0x7fe1dc0abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:07.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 rx=0x7fe1d4006fc0 tx=0x7fe1d400b990).stop 2021-02-10T11:58:07.665 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 shutdown_connections 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fe1c00186c0 0x7fe1c001abc0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fe1dc0ab2c0 0x7fe1dc0abe40 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fe1dc0b7310 0x7fe1dc0aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 --2- 172.21.2.117:0/2398950637 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fe1dc0b77e0 0x7fe1dc0b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 >> 172.21.2.117:0/2398950637 conn(0x7fe1dc0874c0 msgr2=0x7fe1dc0af480 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 shutdown_connections 2021-02-10T11:58:07.666 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:07.663 7fe1e2cc1700 1 -- 172.21.2.117:0/2398950637 wait complete. 2021-02-10T11:58:07.674 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.097 7f60d7901700 1 Processor -- start 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 -- start start 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d007ab40 0x7f60d00794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60d0079b10 0x7f60d0079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f60d007af40 0x7f60d007b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d007a5c0 con 0x7f60d0079b10 2021-02-10T11:58:08.116 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d00e5770 con 0x7f60d007af40 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d7901700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00d18c0 con 0x7f60d007ab40 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60d0079b10 0x7f60d0079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f60d0079b10 0x7f60d0079f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32844/0 (socket says 172.21.2.117:32844) 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 -- 172.21.2.117:0/826650664 learned_addr learned my addr 172.21.2.117:0/826650664 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 -- 172.21.2.117:0/826650664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d007ab40 msgr2=0x7f60d00794f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 --2- 172.21.2.117:0/826650664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d007ab40 0x7f60d00794f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 -- 172.21.2.117:0/826650664 >> v2:172.21.2.125:3300/0 conn(0x7f60d007af40 msgr2=0x7f60d007b340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.098 7f60d4f1c700 1 --2- 172.21.2.117:0/826650664 >> v2:172.21.2.125:3300/0 conn(0x7f60d007af40 0x7f60d007b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d4f1c700 1 -- 172.21.2.117:0/826650664 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d00d1cb0 con 0x7f60d0079b10 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d4f1c700 1 --2- 172.21.2.117:0/826650664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d0079b10 0x7f60d0079f10 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7f60c4006700 tx=0x7f60c4009b00).ready entity=mon.0 client_cookie=fe66afb0959a3341 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60c3fff700 1 -- 172.21.2.117:0/826650664 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60c400c0c0 con 0x7f60d0079b10 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60c3fff700 1 -- 172.21.2.117:0/826650664 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c400c340 con 0x7f60d0079b10 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60c3fff700 1 -- 172.21.2.117:0/826650664 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60c400c820 con 0x7f60d0079b10 2021-02-10T11:58:08.117 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 -- 172.21.2.117:0/826650664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d0079b10 msgr2=0x7f60d0079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 --2- 172.21.2.117:0/826650664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d0079b10 0x7f60d0079f10 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 rx=0x7f60c4006700 tx=0x7f60c4009b00).stop 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 -- 172.21.2.117:0/826650664 shutdown_connections 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 --2- 172.21.2.117:0/826650664 >> v2:172.21.2.125:3300/0 conn(0x7f60d007af40 0x7f60d007b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 --2- 172.21.2.117:0/826650664 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d0079b10 0x7f60d0079f10 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 --2- 172.21.2.117:0/826650664 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d007ab40 0x7f60d00794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.099 7f60d7901700 1 -- 172.21.2.117:0/826650664 >> 172.21.2.117:0/826650664 conn(0x7f60d00d0910 msgr2=0x7f60d00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.100 7f60d7901700 1 -- 172.21.2.117:0/826650664 shutdown_connections 2021-02-10T11:58:08.118 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.100 7f60d7901700 1 -- 172.21.2.117:0/826650664 wait complete. 2021-02-10T11:58:08.119 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.114 7f60d7901700 1 Processor -- start 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.114 7f60d7901700 1 -- start start 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d01a2ce0 0x7f60d00dc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60d00f73f0 0x7f60d00f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d00dbc20 con 0x7f60d009d610 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d01a30e0 con 0x7f60d00f73f0 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d7901700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d009e030 con 0x7f60d01a2ce0 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32852/0 (socket says 172.21.2.117:32852) 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 -- 172.21.2.117:0/1676811549 learned_addr learned my addr 172.21.2.117:0/1676811549 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d571d700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d01a2ce0 0x7f60d00dc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.120 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d5f1e700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60d00f73f0 0x7f60d00f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 -- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d01a2ce0 msgr2=0x7f60d00dc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d01a2ce0 0x7f60d00dc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 -- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60d00f73f0 msgr2=0x7f60d00f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60d00f73f0 0x7f60d00f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.115 7f60d4f1c700 1 -- 172.21.2.117:0/1676811549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d00f83b0 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60d4f1c700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f60c4006f20 tx=0x7f60c400b8e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60c400bcd0 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c4000c90 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f60c40022b0 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60d7901700 1 -- 172.21.2.117:0/1676811549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d00f83b0 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.116 7f60d7901700 1 -- 172.21.2.117:0/1676811549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d00f8100 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.117 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f60c4000c90 con 0x7f60d009d610 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.117 7f60c1ffb700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 0x7f60b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.121 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.117 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f60c40324c0 con 0x7f60d009d610 2021-02-10T11:58:08.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.117 7f60d571d700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 0x7f60b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.117 7f60d571d700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 0x7f60b001abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.148 7f60d7901700 1 -- 172.21.2.117:0/1676811549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d00da9a0 con 0x7f60d009d610 2021-02-10T11:58:08.152 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.150 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f60c4009100 con 0x7f60d009d610 2021-02-10T11:58:08.282 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.280 7f60d7901700 1 -- 172.21.2.117:0/1676811549 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f60d00e5770 con 0x7f60d009d610 2021-02-10T11:58:08.285 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.283 7f60c1ffb700 1 -- 172.21.2.117:0/1676811549 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v118) v1 ==== 184+0+0 (secure 0 0 0) 0x7f60c401f230 con 0x7f60d009d610 2021-02-10T11:58:08.289 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.0 2021-02-10T11:58:08.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 -- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 msgr2=0x7f60b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 0x7f60b001abc0 crc :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 -- 172.21.2.117:0/1676811549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 msgr2=0x7f60d009da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 rx=0x7f60c4006f20 tx=0x7f60c400b8e0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 -- 172.21.2.117:0/1676811549 shutdown_connections 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f60b00186c0 0x7f60b001abc0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f60d009d610 0x7f60d009da10 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f60d00f73f0 0x7f60d00f77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 --2- 172.21.2.117:0/1676811549 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f60d01a2ce0 0x7f60d00dc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.288 7f60d7901700 1 -- 172.21.2.117:0/1676811549 >> 172.21.2.117:0/1676811549 conn(0x7f60d019f690 msgr2=0x7f60d01a0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.289 7f60d7901700 1 -- 172.21.2.117:0/1676811549 shutdown_connections 2021-02-10T11:58:08.291 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.289 7f60d7901700 1 -- 172.21.2.117:0/1676811549 wait complete. 2021-02-10T11:58:08.299 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 Processor -- start 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 -- start start 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa7941b69b0 0x7fa7941b8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940ede70 con 0x7fa7940d1900 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7940eba10 con 0x7fa7940ef400 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa799cea700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7940d1d00 con 0x7fa7941b69b0 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa7937fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa7940d1900 0x7fa7940eaa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33108/0 (socket says 172.21.2.117:33108) 2021-02-10T11:58:08.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.723 7fa7937fe700 1 -- 172.21.2.117:0/3850258073 learned_addr learned my addr 172.21.2.117:0/3850258073 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.724 7fa7927fc700 1 -- 172.21.2.117:0/3850258073 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046411630 0 0) 0x7fa784001180 con 0x7fa7940d1900 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.724 7fa7927fc700 1 -- 172.21.2.117:0/3850258073 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c002fe0 con 0x7fa7940d1900 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.724 7fa793fff700 1 --2- 172.21.2.117:0/3850258073 >> v2:172.21.2.125:3300/0 conn(0x7fa7941b69b0 0x7fa7941b8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.724 7fa793fff700 1 -- 172.21.2.117:0/3850258073 >> v1:172.21.2.125:6789/0 conn(0x7fa7940ef400 legacy=0x7fa7940eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.724 7fa793fff700 1 -- 172.21.2.117:0/3850258073 >> v1:172.21.2.117:6789/0 conn(0x7fa7940d1900 legacy=0x7fa7940eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa793fff700 1 -- 172.21.2.117:0/3850258073 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7941bb400 con 0x7fa7941b69b0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa793fff700 1 --2- 172.21.2.117:0/3850258073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7941b69b0 0x7fa7941b8dc0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fa78c006e50 tx=0x7fa78c009e70).ready entity=mon.1 client_cookie=47ee3f98356a7a51 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa7927fc700 1 -- 172.21.2.117:0/3850258073 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa78c006a20 con 0x7fa7941b69b0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa7927fc700 1 -- 172.21.2.117:0/3850258073 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c006790 con 0x7fa7941b69b0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa7927fc700 1 -- 172.21.2.117:0/3850258073 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa78c00b700 con 0x7fa7941b69b0 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa799cea700 1 -- 172.21.2.117:0/3850258073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7941b69b0 msgr2=0x7fa7941b8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.725 7fa799cea700 1 --2- 172.21.2.117:0/3850258073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7941b69b0 0x7fa7941b8dc0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7fa78c006e50 tx=0x7fa78c009e70).stop 2021-02-10T11:58:08.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.726 7fa799cea700 1 -- 172.21.2.117:0/3850258073 shutdown_connections 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.726 7fa799cea700 1 --2- 172.21.2.117:0/3850258073 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7941b69b0 0x7fa7941b8dc0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.726 7fa799cea700 1 -- 172.21.2.117:0/3850258073 >> 172.21.2.117:0/3850258073 conn(0x7fa7940cf070 msgr2=0x7fa7940cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.726 7fa799cea700 1 -- 172.21.2.117:0/3850258073 shutdown_connections 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.726 7fa799cea700 1 -- 172.21.2.117:0/3850258073 wait complete. 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 Processor -- start 2021-02-10T11:58:08.745 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 -- start start 2021-02-10T11:58:08.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa79408a2b0 0x7fa794086840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.746 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa794087260 0x7fa794085530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794191c60 con 0x7fa794086e60 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79408a6b0 con 0x7fa79408a2b0 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.741 7fa799cea700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7940862c0 con 0x7fa794087260 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32866/0 (socket says 172.21.2.117:32866) 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 -- 172.21.2.117:0/1633744311 learned_addr learned my addr 172.21.2.117:0/1633744311 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 -- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa794087260 msgr2=0x7fa794085530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa794087260 0x7fa794085530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 -- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa79408a2b0 msgr2=0x7fa794086840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa79408a2b0 0x7fa794086840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 -- 172.21.2.117:0/1633744311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7941ab9c0 con 0x7fa794086e60 2021-02-10T11:58:08.747 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa792ffd700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7fa78800b5b0 tx=0x7fa78800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa788006240 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880054a0 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7880059c0 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.742 7fa799cea700 1 -- 172.21.2.117:0/1633744311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7941ab9c0 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.743 7fa799cea700 1 -- 172.21.2.117:0/1633744311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7941ab9c0 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.743 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa788006030 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.743 7fa77bfff700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 0x7fa77401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.743 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa788032740 con 0x7fa794086e60 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.743 7fa7937fe700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 0x7fa77401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:08.748 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.744 7fa7937fe700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 0x7fa77401abc0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:08.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.773 7fa799cea700 1 -- 172.21.2.117:0/1633744311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7940ede70 con 0x7fa794086e60 2021-02-10T11:58:08.777 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.776 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa788005680 con 0x7fa794086e60 2021-02-10T11:58:08.908 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.906 7fa799cea700 1 -- 172.21.2.117:0/1633744311 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa7941ab9c0 con 0x7fa794086e60 2021-02-10T11:58:08.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.910 7fa77bfff700 1 -- 172.21.2.117:0/1633744311 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v119) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa788006030 con 0x7fa794086e60 2021-02-10T11:58:08.915 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.1 2021-02-10T11:58:08.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 msgr2=0x7fa77401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 0x7fa77401abc0 crc :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 msgr2=0x7fa794084f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 rx=0x7fa78800b5b0 tx=0x7fa78800b950).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 shutdown_connections 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa7740186c0 0x7fa77401abc0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa794086e60 0x7fa794084f10 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa794087260 0x7fa794085530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 --2- 172.21.2.117:0/1633744311 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa79408a2b0 0x7fa794086840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:08.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 >> 172.21.2.117:0/1633744311 conn(0x7fa794191e10 msgr2=0x7fa79419ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:08.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 shutdown_connections 2021-02-10T11:58:08.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:08.915 7fa799cea700 1 -- 172.21.2.117:0/1633744311 wait complete. 2021-02-10T11:58:08.926 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:58:09.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 Processor -- start 2021-02-10T11:58:09.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 -- start start 2021-02-10T11:58:09.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 0x7f36c00794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36c0079b10 0x7f36c0079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c007a5c0 con 0x7f36c007af40 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c00e5770 con 0x7f36c0079b10 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36c5f5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00d18c0 con 0x7f36c007ab40 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36beffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36c0079b10 0x7f36c0079f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36beffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f36c0079b10 0x7f36c0079f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32872/0 (socket says 172.21.2.117:32872) 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36bffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f36c007af40 0x7f36c007b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33124/0 (socket says 172.21.2.117:33124) 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36beffd700 1 -- 172.21.2.117:0/2283793568 learned_addr learned my addr 172.21.2.117:0/2283793568 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.350 7f36bf7fe700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 0x7f36c00794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36be7fc700 1 -- 172.21.2.117:0/2283793568 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034513539 0 0) 0x7f36b8001480 con 0x7f36c007af40 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36beffd700 1 -- 172.21.2.117:0/2283793568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 msgr2=0x7f36c00794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36beffd700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 0x7f36c00794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36beffd700 1 -- 172.21.2.117:0/2283793568 >> v1:172.21.2.117:6789/0 conn(0x7f36c007af40 legacy=0x7f36c007b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36beffd700 1 -- 172.21.2.117:0/2283793568 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00d1cb0 con 0x7f36c0079b10 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36bf7fe700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 0x7f36c00794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36beffd700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c0079b10 0x7f36c0079f10 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 rx=0x7f36b0006700 tx=0x7f36b0009b80).ready entity=mon.0 client_cookie=e32de0ef25610d40 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36be7fc700 1 -- 172.21.2.117:0/2283793568 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b0006340 con 0x7f36c0079b10 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36be7fc700 1 -- 172.21.2.117:0/2283793568 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b000c0c0 con 0x7f36c0079b10 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36be7fc700 1 -- 172.21.2.117:0/2283793568 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b000c5e0 con 0x7f36c0079b10 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36c5f5b700 1 -- 172.21.2.117:0/2283793568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c0079b10 msgr2=0x7f36c0079f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.351 7f36c5f5b700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c0079b10 0x7f36c0079f10 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 rx=0x7f36b0006700 tx=0x7f36b0009b80).stop 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 -- 172.21.2.117:0/2283793568 shutdown_connections 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c0079b10 0x7f36c0079f10 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 --2- 172.21.2.117:0/2283793568 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c007ab40 0x7f36c00794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 -- 172.21.2.117:0/2283793568 >> 172.21.2.117:0/2283793568 conn(0x7f36c00d0910 msgr2=0x7f36c00d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:09.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 -- 172.21.2.117:0/2283793568 shutdown_connections 2021-02-10T11:58:09.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.352 7f36c5f5b700 1 -- 172.21.2.117:0/2283793568 wait complete. 2021-02-10T11:58:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 Processor -- start 2021-02-10T11:58:09.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 -- start start 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36c0100480 0x7f36c0100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c009bb90 0x7f36c009bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0192330 con 0x7f36c01ad610 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c01ae100 con 0x7f36c0100480 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36c5f5b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0100fb0 con 0x7f36c009bb90 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.367 7f36beffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32878/0 (socket says 172.21.2.117:32878) 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 -- 172.21.2.117:0/3095607771 learned_addr learned my addr 172.21.2.117:0/3095607771 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36bffff700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c009bb90 0x7f36c009bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36bf7fe700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36c0100480 0x7f36c0100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 -- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c009bb90 msgr2=0x7f36c009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c009bb90 0x7f36c009bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 -- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36c0100480 msgr2=0x7f36c0100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36c0100480 0x7f36c0100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 -- 172.21.2.117:0/3095607771 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c009c5b0 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36beffd700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f36b000b600 tx=0x7f36b000b9a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b0002030 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b0002230 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c009c5b0 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c009cb40 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.368 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f36b0002750 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.369 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f36b0000c90 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.369 7f36b7fff700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 0x7f36a401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.369 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f36b0032960 con 0x7f36c01ad610 2021-02-10T11:58:09.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.369 7f36bf7fe700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 0x7f36a401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.369 7f36bf7fe700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 0x7f36a401ac10 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:09.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.399 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c00daa50 con 0x7f36c01ad610 2021-02-10T11:58:09.404 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.402 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f36b00009b0 con 0x7f36c01ad610 2021-02-10T11:58:09.535 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.533 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f36c007a5c0 con 0x7f36c01ad610 2021-02-10T11:58:09.538 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.536 7f36b7fff700 1 -- 172.21.2.117:0/3095607771 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v120) v1 ==== 184+0+0 (secure 0 0 0) 0x7f36b0000c90 con 0x7f36c01ad610 2021-02-10T11:58:09.542 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.2 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 msgr2=0x7f36a401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 0x7f36a401ac10 crc :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 msgr2=0x7f36c01ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 rx=0x7f36b000b600 tx=0x7f36b000b9a0).stop 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 shutdown_connections 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f36a4018730 0x7f36a401ac10 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f36c01ad610 0x7f36c01ada10 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f36c009bb90 0x7f36c009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.541 7f36c5f5b700 1 --2- 172.21.2.117:0/3095607771 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f36c0100480 0x7f36c0100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.542 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 >> 172.21.2.117:0/3095607771 conn(0x7f36c01a3700 msgr2=0x7f36c00dac80 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:09.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.542 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 shutdown_connections 2021-02-10T11:58:09.545 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.542 7f36c5f5b700 1 -- 172.21.2.117:0/3095607771 wait complete. 2021-02-10T11:58:09.552 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph auth caps client.3 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.976 7f41926fa700 1 Processor -- start 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.976 7f41926fa700 1 -- start start 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f41926fa700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f418c07ab40 0x7f418c0794f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f41926fa700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f418c079b10 0x7f418c079f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f41926fa700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c0e5770 con 0x7f418c07af40 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f41926fa700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0d18c0 con 0x7f418c07ab40 2021-02-10T11:58:09.995 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f41926fa700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0d1c60 con 0x7f418c079b10 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f418c07ab40 0x7f418c0794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f418c07ab40 0x7f418c0794f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32884/0 (socket says 172.21.2.117:32884) 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 -- 172.21.2.117:0/1993475939 learned_addr learned my addr 172.21.2.117:0/1993475939 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 -- 172.21.2.117:0/1993475939 >> v2:172.21.2.125:3300/0 conn(0x7f418c079b10 msgr2=0x7f418c079f10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 --2- 172.21.2.117:0/1993475939 >> v2:172.21.2.125:3300/0 conn(0x7f418c079b10 0x7f418c079f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 -- 172.21.2.117:0/1993475939 >> v1:172.21.2.125:6789/0 conn(0x7f418c07af40 legacy=0x7f418c07b340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.977 7f418bfff700 1 -- 172.21.2.117:0/1993475939 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c07bb30 con 0x7f418c07ab40 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f418bfff700 1 --2- 172.21.2.117:0/1993475939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c07ab40 0x7f418c0794f0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f417c006700 tx=0x7f417c009b00).ready entity=mon.0 client_cookie=7d4ee0eb15e74f5b server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f418affd700 1 -- 172.21.2.117:0/1993475939 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f417c00c0c0 con 0x7f418c07ab40 2021-02-10T11:58:09.996 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f418affd700 1 -- 172.21.2.117:0/1993475939 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c00c340 con 0x7f418c07ab40 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f418affd700 1 -- 172.21.2.117:0/1993475939 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f417c00c820 con 0x7f418c07ab40 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f41926fa700 1 -- 172.21.2.117:0/1993475939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c07ab40 msgr2=0x7f418c0794f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.978 7f41926fa700 1 --2- 172.21.2.117:0/1993475939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c07ab40 0x7f418c0794f0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 rx=0x7f417c006700 tx=0x7f417c009b00).stop 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 -- 172.21.2.117:0/1993475939 shutdown_connections 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 --2- 172.21.2.117:0/1993475939 >> v2:172.21.2.125:3300/0 conn(0x7f418c079b10 0x7f418c079f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 --2- 172.21.2.117:0/1993475939 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c07ab40 0x7f418c0794f0 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 -- 172.21.2.117:0/1993475939 >> 172.21.2.117:0/1993475939 conn(0x7f418c0d0910 msgr2=0x7f418c0d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 -- 172.21.2.117:0/1993475939 shutdown_connections 2021-02-10T11:58:09.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.979 7f41926fa700 1 -- 172.21.2.117:0/1993475939 wait complete. 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.993 7f41926fa700 1 Processor -- start 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.993 7f41926fa700 1 -- start start 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f418c100750 0x7f418c098110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f418c0faa40 0x7f418c0fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0987b0 con 0x7f418c100b50 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c100210 con 0x7f418c100750 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f41926fa700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0fb600 con 0x7f418c0faa40 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:09.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32888/0 (socket says 172.21.2.117:32888) 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 -- 172.21.2.117:0/3430963533 learned_addr learned my addr 172.21.2.117:0/3430963533 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 -- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f418c0faa40 msgr2=0x7f418c0fafe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f418c0faa40 0x7f418c0fafe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 -- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f418c100750 msgr2=0x7f418c098110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f418c100750 0x7f418c098110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.994 7f418b7fe700 1 -- 172.21.2.117:0/3430963533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0fb820 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f418b7fe700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f418000b5c0 tx=0x7f418000b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f418000bd50 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4180006030 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f41800054d0 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f41926fa700 1 -- 172.21.2.117:0/3430963533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c0fb820 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f41926fa700 1 -- 172.21.2.117:0/3430963533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0ff1d0 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.995 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4180006210 con 0x7f418c100b50 2021-02-10T11:58:10.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.996 7f4188ff9700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 0x7f416c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.996 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4180032950 con 0x7f418c100b50 2021-02-10T11:58:10.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.996 7f418bfff700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 0x7f416c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:09.997 7f418bfff700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 0x7f416c01abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:10.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.027 7f41926fa700 1 -- 172.21.2.117:0/3430963533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c0e5770 con 0x7f418c100b50 2021-02-10T11:58:10.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.029 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4180000940 con 0x7f418c100b50 2021-02-10T11:58:10.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.157 7f41926fa700 1 -- 172.21.2.117:0/3430963533 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f418c0e5770 con 0x7f418c100b50 2021-02-10T11:58:10.162 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.160 7f4188ff9700 1 -- 172.21.2.117:0/3430963533 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.3", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.3 v121) v1 ==== 184+0+0 (secure 0 0 0) 0x7f41800060f0 con 0x7f418c100b50 2021-02-10T11:58:10.166 INFO:teuthology.orchestra.run.gibba017.stderr:updated caps for client.3 2021-02-10T11:58:10.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.165 7f41926fa700 1 -- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 msgr2=0x7f416c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.165 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 0x7f416c01abc0 crc :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.165 7f41926fa700 1 -- 172.21.2.117:0/3430963533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 msgr2=0x7f418c0fa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.167 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.165 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 rx=0x7f418000b5c0 tx=0x7f418000b960).stop 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 -- 172.21.2.117:0/3430963533 shutdown_connections 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f416c0186c0 0x7f416c01abc0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f418c100b50 0x7f418c0fa420 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f418c0faa40 0x7f418c0fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 --2- 172.21.2.117:0/3430963533 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f418c100750 0x7f418c098110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.168 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 -- 172.21.2.117:0/3430963533 >> 172.21.2.117:0/3430963533 conn(0x7f418c19f690 msgr2=0x7f418c1950e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:10.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 -- 172.21.2.117:0/3430963533 shutdown_connections 2021-02-10T11:58:10.169 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.166 7f41926fa700 1 -- 172.21.2.117:0/3430963533 wait complete. 2021-02-10T11:58:10.176 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:58:10.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.588 7fddc3f11700 1 Processor -- start 2021-02-10T11:58:10.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.588 7fddc3f11700 1 -- start start 2021-02-10T11:58:10.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc3f11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0d18e0 0x7fddbc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc3f11700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0ede60 con 0x7fddbc0eb000 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc3f11700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddbc0d1ce0 con 0x7fddbc0ef3f0 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc3f11700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc088020 con 0x7fddbc0d18e0 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc152c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fddbc0eb000 0x7fddbc0eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33146/0 (socket says 172.21.2.117:33146) 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc152c700 1 -- 172.21.2.117:0/1690584953 learned_addr learned my addr 172.21.2.117:0/1690584953 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162979374 0 0) 0x7fddb80010b0 con 0x7fddbc0eb000 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda8002fe0 con 0x7fddbc0eb000 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.589 7fddc1d2d700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0d18e0 0x7fddbc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3940816132 0 0) 0x7fddb80010b0 con 0x7fddbc0eb000 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0d18e0 msgr2=0x7fddbc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0d18e0 0x7fddbc0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 >> v1:172.21.2.125:6789/0 conn(0x7fddbc0ef3f0 legacy=0x7fddbc1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc1bb180 con 0x7fddbc0eb000 2021-02-10T11:58:10.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7fddb80015b0 con 0x7fddbc0eb000 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 >> v1:172.21.2.117:6789/0 conn(0x7fddbc0eb000 legacy=0x7fddbc0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda8003a80 0x7fdda8005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda8006730 0x7fdda8008c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda80094f0 0x7fdda800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda800c2c0 con 0x7fdda8006730 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda800c4e0 con 0x7fdda80094f0 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda800c700 con 0x7fdda8003a80 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda8006730 0x7fdda8008c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc1d2d700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda8003a80 0x7fdda8005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc252e700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda80094f0 0x7fdda800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 -- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda8003a80 msgr2=0x7fdda8005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdda8003a80 0x7fdda8005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 -- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda80094f0 msgr2=0x7fdda800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdda80094f0 0x7fdda800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.590 7fddc152c700 1 -- 172.21.2.117:0/1690584953 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8050970 con 0x7fdda8006730 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc152c700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda8006730 0x7fdda8008c50 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7fddb80017d0 tx=0x7fddb8007cb0).ready entity=mon.0 client_cookie=81153dcdaec655da server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb800a310 con 0x7fdda8006730 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb800a590 con 0x7fdda8006730 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc0d2b700 1 -- 172.21.2.117:0/1690584953 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb800a9e0 con 0x7fdda8006730 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc3f11700 1 -- 172.21.2.117:0/1690584953 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda8006730 msgr2=0x7fdda8008c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc3f11700 1 --2- 172.21.2.117:0/1690584953 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdda8006730 0x7fdda8008c50 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 rx=0x7fddb80017d0 tx=0x7fddb8007cb0).stop 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc252e700 1 -- 172.21.2.117:0/1690584953 reap_dead start 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc3f11700 1 -- 172.21.2.117:0/1690584953 shutdown_connections 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.591 7fddc3f11700 1 -- 172.21.2.117:0/1690584953 >> 172.21.2.117:0/1690584953 conn(0x7fddbc0ced60 msgr2=0x7fddbc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.592 7fddc3f11700 1 -- 172.21.2.117:0/1690584953 shutdown_connections 2021-02-10T11:58:10.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.592 7fddc3f11700 1 -- 172.21.2.117:0/1690584953 wait complete. 2021-02-10T11:58:10.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.606 7fddc3f11700 1 Processor -- start 2021-02-10T11:58:10.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.606 7fddc3f11700 1 -- start start 2021-02-10T11:58:10.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0e9490 0x7fddbc0e8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddbc0959c0 0x7fddbc095dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0ade90 con 0x7fddbc0963e0 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0e9890 con 0x7fddbc0959c0 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc3f11700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc096930 con 0x7fddbc0e9490 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32908/0 (socket says 172.21.2.117:32908) 2021-02-10T11:58:10.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 -- 172.21.2.117:0/3957434709 learned_addr learned my addr 172.21.2.117:0/3957434709 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 -- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0e9490 msgr2=0x7fddbc0e8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0e9490 0x7fddbc0e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 -- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddbc0959c0 msgr2=0x7fddbc095dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddbc0959c0 0x7fddbc095dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 -- 172.21.2.117:0/3957434709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc0974d0 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.607 7fddc252e700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 rx=0x7fddb400a8f0 tx=0x7fddb400ac90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb400c300 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb400c640 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc0974d0 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fddb400caa0 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc0974d0 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fddb400c300 con 0x7fddbc0963e0 2021-02-10T11:58:10.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.608 7fddae7fc700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 0x7fdda401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:10.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.609 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fddb4032ce0 con 0x7fddbc0963e0 2021-02-10T11:58:10.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.609 7fddc1d2d700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 0x7fdda401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:10.616 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.609 7fddc1d2d700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 0x7fdda401abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:10.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.638 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc087290 con 0x7fddbc0963e0 2021-02-10T11:58:10.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.641 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fddb400add0 con 0x7fddbc0963e0 2021-02-10T11:58:10.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.767 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddbc0ede60 con 0x7fddbc0963e0 2021-02-10T11:58:10.770 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.768 7fddae7fc700 1 -- 172.21.2.117:0/3957434709 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fddb401f8d0 con 0x7fddbc0963e0 2021-02-10T11:58:10.771 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 170 2021-02-10T11:58:10.771 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:10.771 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13806,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3326472561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3326472561},{"type":"v1","addr":"172.21.2.117:6833","nonce":3326472561}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":166}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:58:04.413160","modified":"2021-02-10 11:58:06.419787","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9775":{"gid":9775,"name":"b","rank":0,"incarnation":167,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1289009861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1289009861},{"type":"v1","addr":"172.21.2.125:6835","nonce":1289009861}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:58:10.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 msgr2=0x7fdda401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.772 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 0x7fdda401abc0 crc :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 msgr2=0x7fddbc096eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 rx=0x7fddb400a8f0 tx=0x7fddb400ac90).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 shutdown_connections 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdda40186c0 0x7fdda401abc0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fddbc0963e0 0x7fddbc096eb0 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fddbc0959c0 0x7fddbc095dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 --2- 172.21.2.117:0/3957434709 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fddbc0e9490 0x7fddbc0e8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 >> 172.21.2.117:0/3957434709 conn(0x7fddbc0c3fb0 msgr2=0x7fddbc0a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 shutdown_connections 2021-02-10T11:58:10.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:10.770 7fddc3f11700 1 -- 172.21.2.117:0/3957434709 wait complete. 2021-02-10T11:58:10.982 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 170, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:58:04.413160', 'modified': '2021-02-10 11:58:06.419787', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9775': {'gid': 9775, 'name': 'b', 'rank': 0, 'incarnation': 167, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1289009861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1289009861}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1289009861}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:58:10.983 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:58:10.983 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:58:11.382 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:58:11.383 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:58:11.383 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:58:11.383 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9775, 2021-02-10T11:58:11.383 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:58:11.383 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 6.9658085119999997, 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 170, 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 196, 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 196, 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 12.611480322 2021-02-10T11:58:11.384 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:58:11.387 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9775, "want_state": "up:active", "state": "up:active", "rank_uptime": 6.9658085119999997, "mdsmap_epoch": 170, "osdmap_epoch": 196, "osdmap_epoch_barrier": 196, "uptime": 12.611480322 } 2021-02-10T11:58:11.388 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:11.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.800 7f65b3db8700 1 Processor -- start 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.800 7f65b3db8700 1 -- start start 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b3db8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 0x7f65ac0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b3db8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 0x7f65ac0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b3db8700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac0ede60 con 0x7f65ac0ee3e0 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b3db8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ac0eba90 con 0x7f65ac0d18e0 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b3db8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0d1ce0 con 0x7f65ac0eb000 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b23d5700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7f65ac0ee3e0 0x7f65ac0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:54320/0 (socket says 172.21.2.117:54320) 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b23d5700 1 -- 172.21.2.117:0/4026338350 learned_addr learned my addr 172.21.2.117:0/4026338350 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b1bd4700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 0x7f65ac0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.801 7f65b13d3700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 0x7f65ac0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b13d3700 1 -- 172.21.2.117:0/4026338350 >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 msgr2=0x7f65ac0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:11.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b13d3700 1 --2- 172.21.2.117:0/4026338350 >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 0x7f65ac0ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b13d3700 1 -- 172.21.2.117:0/4026338350 >> v1:172.21.2.125:6789/0 conn(0x7f65ac0ee3e0 legacy=0x7f65ac0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b13d3700 1 -- 172.21.2.117:0/4026338350 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac0eefd0 con 0x7f65ac0eb000 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b0bd2700 1 -- 172.21.2.117:0/4026338350 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066470629 0 0) 0x7f65a4001510 con 0x7f65ac0ee3e0 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b1bd4700 1 --2- 172.21.2.117:0/4026338350 >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 0x7f65ac0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.802 7f65b13d3700 1 --2- 172.21.2.117:0/4026338350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 0x7f65ac0eb400 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f659c006650 tx=0x7f659c009a80).ready entity=mon.2 client_cookie=9ad8e273be953be9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b0bd2700 1 -- 172.21.2.117:0/4026338350 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f659c0009b0 con 0x7f65ac0eb000 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b0bd2700 1 -- 172.21.2.117:0/4026338350 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c00c0c0 con 0x7f65ac0eb000 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b0bd2700 1 -- 172.21.2.117:0/4026338350 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f659c00c5e0 con 0x7f65ac0eb000 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 -- 172.21.2.117:0/4026338350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 msgr2=0x7f65ac0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 --2- 172.21.2.117:0/4026338350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 0x7f65ac0eb400 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f659c006650 tx=0x7f659c009a80).stop 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 -- 172.21.2.117:0/4026338350 shutdown_connections 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 --2- 172.21.2.117:0/4026338350 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0eb000 0x7f65ac0eb400 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 --2- 172.21.2.117:0/4026338350 >> v2:172.21.2.125:3300/0 conn(0x7f65ac0d18e0 0x7f65ac0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.821 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.803 7f65b3db8700 1 -- 172.21.2.117:0/4026338350 >> 172.21.2.117:0/4026338350 conn(0x7f65ac0ced60 msgr2=0x7f65ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:11.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.804 7f65b3db8700 1 -- 172.21.2.117:0/4026338350 shutdown_connections 2021-02-10T11:58:11.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.804 7f65b3db8700 1 -- 172.21.2.117:0/4026338350 wait complete. 2021-02-10T11:58:11.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.818 7f65b3db8700 1 Processor -- start 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.818 7f65b3db8700 1 -- start start 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.818 7f65b3db8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 0x7f65ac1a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.818 7f65b3db8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b3db8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0b8d10 0x7f65ac0b9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b3db8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0c0670 con 0x7f65ac0b82f0 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b3db8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0deb20 con 0x7f65ac1a33d0 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b3db8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac0ded40 con 0x7f65ac0b8d10 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32916/0 (socket says 172.21.2.117:32916) 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 -- 172.21.2.117:0/2626297008 learned_addr learned my addr 172.21.2.117:0/2626297008 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 -- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0b8d10 msgr2=0x7f65ac0b9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0b8d10 0x7f65ac0b9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:11.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 -- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 msgr2=0x7f65ac1a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b1bd4700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 0x7f65ac1a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 0x7f65ac1a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b13d3700 1 -- 172.21.2.117:0/2626297008 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac0def60 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.819 7f65b1bd4700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 0x7f65ac1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65b13d3700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 rx=0x7f659c009630 tx=0x7f659c00b900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f659c0093a0 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c00bcf0 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f659c005560 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac0def60 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.820 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac0def60 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.821 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f659c00a000 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.821 7f65a27fc700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 0x7f659401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.821 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f659c0326b0 con 0x7f65ac0b82f0 2021-02-10T11:58:11.825 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.821 7f65b1bd4700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 0x7f659401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:11.826 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.822 7f65b1bd4700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 0x7f659401abc0 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:11.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.850 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac087290 con 0x7f65ac0b82f0 2021-02-10T11:58:11.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.853 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f659c020e50 con 0x7f65ac0b82f0 2021-02-10T11:58:11.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.977 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65ac0ede60 con 0x7f65ac0b82f0 2021-02-10T11:58:11.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.978 7f65a27fc700 1 -- 172.21.2.117:0/2626297008 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f659c020e50 con 0x7f65ac0b82f0 2021-02-10T11:58:11.980 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:11.980 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 msgr2=0x7f659401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 0x7f659401abc0 crc :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.981 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 msgr2=0x7f65ac0b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 rx=0x7f659c009630 tx=0x7f659c00b900).stop 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 shutdown_connections 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f65940186c0 0x7f659401abc0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f65ac0b82f0 0x7f65ac0b86f0 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f65ac0b8d10 0x7f65ac0b9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 --2- 172.21.2.117:0/2626297008 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f65ac1a33d0 0x7f65ac1a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.979 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 >> 172.21.2.117:0/2626297008 conn(0x7f65ac0874c0 msgr2=0x7f65ac096910 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.980 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 shutdown_connections 2021-02-10T11:58:11.982 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:11.980 7f65b3db8700 1 -- 172.21.2.117:0/2626297008 wait complete. 2021-02-10T11:58:11.990 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:58:02.381112', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:58:03.405092', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:58:11.991 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 Processor -- start 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 -- start start 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb41b69a0 0x7f2fb41b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40ede60 con 0x7f2fb40d18f0 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40eba00 con 0x7f2fb40ef3f0 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.406 7f2fbb12b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40d1cf0 con 0x7f2fb41b69a0 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.407 7f2fb8f47700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f2fb40d18f0 0x7f2fb40eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33174/0 (socket says 172.21.2.117:33174) 2021-02-10T11:58:12.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.407 7f2fb8f47700 1 -- 172.21.2.117:0/1705775860 learned_addr learned my addr 172.21.2.117:0/1705775860 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.407 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937530548 0 0) 0x7f2f9c000ce0 con 0x7f2fb40d18f0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.407 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa0002fe0 con 0x7f2fb40d18f0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.407 7f2fb9748700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb41b69a0 0x7f2fb41b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3888754046 0 0) 0x7f2f9c0010e0 con 0x7f2fb40d18f0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb41b69a0 msgr2=0x7f2fb41b8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb41b69a0 0x7f2fb41b8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 >> v1:172.21.2.125:6789/0 conn(0x7f2fb40ef3f0 legacy=0x7f2fb40eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb41bb3f0 con 0x7f2fb40d18f0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f2f9c001660 con 0x7f2fb40d18f0 2021-02-10T11:58:12.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 >> v1:172.21.2.117:6789/0 conn(0x7f2fb40d18f0 legacy=0x7f2fb40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fa0003a60 0x7f2fa0005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fa0006780 0x7f2fa0008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fa0009580 0x7f2fa000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa000c2c0 con 0x7f2fa0006780 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa000c4e0 con 0x7f2fa0003a60 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa0050710 con 0x7f2fa0009580 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fa0006780 0x7f2fa0008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb8f47700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fa0003a60 0x7f2fa0005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb9748700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fa0009580 0x7f2fa000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 -- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fa0009580 msgr2=0x7f2fa000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fa0009580 0x7f2fa000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 -- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fa0003a60 msgr2=0x7f2fa0005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:12.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fa0003a60 0x7f2fa0005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.408 7f2fb3fff700 1 -- 172.21.2.117:0/1705775860 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0050970 con 0x7f2fa0006780 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fb3fff700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fa0006780 0x7f2fa0008ca0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f2fa4006e30 tx=0x7f2fa4009de0).ready entity=mon.0 client_cookie=f7d4880ee512d1af server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fa400b300 con 0x7f2fa0006780 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400b580 con 0x7f2fa0006780 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fb37fe700 1 -- 172.21.2.117:0/1705775860 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fa400b9e0 con 0x7f2fa0006780 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 -- 172.21.2.117:0/1705775860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fa0006780 msgr2=0x7f2fa0008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 --2- 172.21.2.117:0/1705775860 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fa0006780 0x7f2fa0008ca0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 rx=0x7f2fa4006e30 tx=0x7f2fa4009de0).stop 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fb9748700 1 -- 172.21.2.117:0/1705775860 reap_dead start 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 -- 172.21.2.117:0/1705775860 shutdown_connections 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 -- 172.21.2.117:0/1705775860 >> 172.21.2.117:0/1705775860 conn(0x7f2fb40cf060 msgr2=0x7f2fb40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 -- 172.21.2.117:0/1705775860 shutdown_connections 2021-02-10T11:58:12.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.409 7f2fbb12b700 1 -- 172.21.2.117:0/1705775860 wait complete. 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.424 7f2fbb12b700 1 Processor -- start 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.424 7f2fbb12b700 1 -- start start 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb40a6ce0 0x7f2fb40dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fb40a2040 0x7f2fb40a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40a67e0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40a7120 con 0x7f2fb40a2040 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fbb12b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40a2a60 con 0x7f2fb40a6ce0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32938/0 (socket says 172.21.2.117:32938) 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 -- 172.21.2.117:0/3767068481 learned_addr learned my addr 172.21.2.117:0/3767068481 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 -- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb40a6ce0 msgr2=0x7f2fb40dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb40a6ce0 0x7f2fb40dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 -- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fb40a2040 msgr2=0x7f2fb40a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fb40a2040 0x7f2fb40a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.425 7f2fb3fff700 1 -- 172.21.2.117:0/3767068481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb41a2dc0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb3fff700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 rx=0x7f2fa400b960 tx=0x7f2fa400bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fa4000ce0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa4005a30 con 0x7f2fb4085ec0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2fa4006030 con 0x7f2fb4085ec0 2021-02-10T11:58:12.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb41a2dc0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb41a2dc0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2fa4000cc0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 0x7f2f9801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.426 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2fa4032cf0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.427 7f2fb8f47700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 0x7f2f9801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:12.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.427 7f2fb8f47700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 0x7f2f9801abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:12.458 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.456 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb40daff0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.459 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2fa4000fe0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.588 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f2fb40a70e0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.590 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.588 7f2fb17fa700 1 -- 172.21.2.117:0/3767068481 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v196) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2fa4005da0 con 0x7f2fb4085ec0 2021-02-10T11:58:12.591 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.590 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 msgr2=0x7f2f9801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.590 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 0x7f2f9801abc0 crc :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.590 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 msgr2=0x7f2fb40862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:12.592 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.590 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 rx=0x7f2fa400b960 tx=0x7f2fa400bd00).stop 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 shutdown_connections 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2f980186c0 0x7f2f9801abc0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2fb4085ec0 0x7f2fb40862c0 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2fb40a2040 0x7f2fb40a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 --2- 172.21.2.117:0/3767068481 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2fb40a6ce0 0x7f2fb40dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 >> 172.21.2.117:0/3767068481 conn(0x7f2fb40b6540 msgr2=0x7f2fb408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:12.593 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 shutdown_connections 2021-02-10T11:58:12.594 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:12.591 7f2fbb12b700 1 -- 172.21.2.117:0/3767068481 wait complete. 2021-02-10T11:58:12.601 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:58:13.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 Processor -- start 2021-02-10T11:58:13.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 -- start start 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf800d15e0 0x7fcf800e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf800e7580 con 0x7fcf800e7180 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf800e77f0 con 0x7fcf801b07f0 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf85412700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800e5ce0 con 0x7fcf800d15e0 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf7f87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf800d15e0 0x7fcf800e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.015 7fcf7f87e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcf800d15e0 0x7fcf800e8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32946/0 (socket says 172.21.2.117:32946) 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7f87e700 1 -- 172.21.2.117:0/3043502961 learned_addr learned my addr 172.21.2.117:0/3043502961 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7f87e700 1 -- 172.21.2.117:0/3043502961 >> v1:172.21.2.125:6789/0 conn(0x7fcf801b07f0 legacy=0x7fcf800e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:13.034 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7f87e700 1 -- 172.21.2.117:0/3043502961 >> v1:172.21.2.117:6789/0 conn(0x7fcf800e7180 legacy=0x7fcf800e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7f87e700 1 -- 172.21.2.117:0/3043502961 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800d19e0 con 0x7fcf800d15e0 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7f87e700 1 --2- 172.21.2.117:0/3043502961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800d15e0 0x7fcf800e8560 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fcf78006f00 tx=0x7fcf78009f10).ready entity=mon.0 client_cookie=16b3d0b193c580bf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7e07b700 1 -- 172.21.2.117:0/3043502961 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf7800b3b0 con 0x7fcf800d15e0 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7e07b700 1 -- 172.21.2.117:0/3043502961 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7800b630 con 0x7fcf800d15e0 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf7e07b700 1 -- 172.21.2.117:0/3043502961 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf7800bb50 con 0x7fcf800d15e0 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf85412700 1 -- 172.21.2.117:0/3043502961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800d15e0 msgr2=0x7fcf800e8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.016 7fcf85412700 1 --2- 172.21.2.117:0/3043502961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800d15e0 0x7fcf800e8560 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 rx=0x7fcf78006f00 tx=0x7fcf78009f10).stop 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.017 7fcf85412700 1 -- 172.21.2.117:0/3043502961 shutdown_connections 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.017 7fcf85412700 1 --2- 172.21.2.117:0/3043502961 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800d15e0 0x7fcf800e8560 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.017 7fcf85412700 1 -- 172.21.2.117:0/3043502961 >> 172.21.2.117:0/3043502961 conn(0x7fcf800ce500 msgr2=0x7fcf800e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:13.035 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.017 7fcf85412700 1 -- 172.21.2.117:0/3043502961 shutdown_connections 2021-02-10T11:58:13.036 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.017 7fcf85412700 1 -- 172.21.2.117:0/3043502961 wait complete. 2021-02-10T11:58:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 Processor -- start 2021-02-10T11:58:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 -- start start 2021-02-10T11:58:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf801c5370 0x7fcf801c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.037 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf800ab960 0x7fcf800abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800ab3e0 con 0x7fcf800ac380 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8007e130 con 0x7fcf800ab960 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf85412700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8007e350 con 0x7fcf801c5370 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf7f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf7f87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32948/0 (socket says 172.21.2.117:32948) 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.032 7fcf7f87e700 1 -- 172.21.2.117:0/340817237 learned_addr learned my addr 172.21.2.117:0/340817237 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 -- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf801c5370 msgr2=0x7fcf801c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf801c5370 0x7fcf801c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 -- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf800ab960 msgr2=0x7fcf800abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.038 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf800ab960 0x7fcf800abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 -- 172.21.2.117:0/340817237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800982f0 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf7f87e700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 rx=0x7fcf78009390 tx=0x7fcf7800bd40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf78003030 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7800c2f0 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcf7800c7d0 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf85412700 1 -- 172.21.2.117:0/340817237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf800982f0 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.033 7fcf85412700 1 -- 172.21.2.117:0/340817237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf80098880 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.034 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcf78001270 con 0x7fcf800ac380 2021-02-10T11:58:13.039 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.034 7fcf67fff700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 0x7fcf6001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.034 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcf780328b0 con 0x7fcf800ac380 2021-02-10T11:58:13.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.034 7fcf7f07d700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 0x7fcf6001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.040 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.035 7fcf7f07d700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 0x7fcf6001abc0 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.068 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.065 7fcf85412700 1 -- 172.21.2.117:0/340817237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8007c0a0 con 0x7fcf800ac380 2021-02-10T11:58:13.070 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.068 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcf78000cc0 con 0x7fcf800ac380 2021-02-10T11:58:13.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.197 7fcf85412700 1 -- 172.21.2.117:0/340817237 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcf800d1c50 con 0x7fcf800ac380 2021-02-10T11:58:13.199 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.197 7fcf67fff700 1 -- 172.21.2.117:0/340817237 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v196) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcf78001270 con 0x7fcf800ac380 2021-02-10T11:58:13.200 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:13.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 -- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 msgr2=0x7fcf6001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 0x7fcf6001abc0 crc :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 -- 172.21.2.117:0/340817237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 msgr2=0x7fcf800ac780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.201 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 rx=0x7fcf78009390 tx=0x7fcf7800bd40).stop 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 -- 172.21.2.117:0/340817237 shutdown_connections 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcf600186c0 0x7fcf6001abc0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcf800ac380 0x7fcf800ac780 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcf800ab960 0x7fcf800abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 --2- 172.21.2.117:0/340817237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcf801c5370 0x7fcf801c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 -- 172.21.2.117:0/340817237 >> 172.21.2.117:0/340817237 conn(0x7fcf80084ee0 msgr2=0x7fcf800873c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:13.202 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.199 7fcf85412700 1 -- 172.21.2.117:0/340817237 shutdown_connections 2021-02-10T11:58:13.203 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.200 7fcf85412700 1 -- 172.21.2.117:0/340817237 wait complete. 2021-02-10T11:58:13.210 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:58:13.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 Processor -- start 2021-02-10T11:58:13.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 -- start start 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 0x7f75980eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75980edee0 con 0x7f75980d18f0 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75980eba60 con 0x7f75980ee3e0 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f759db3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980d1cf0 con 0x7f75980eb040 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f7597fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f75977fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f7597fff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:37344/0 (socket says 172.21.2.117:37344) 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f75977fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32956/0 (socket says 172.21.2.117:32956) 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f7597fff700 1 -- 172.21.2.117:0/2973007258 learned_addr learned my addr 172.21.2.117:0/2973007258 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:13.643 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.624 7f7596ffd700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 0x7f75980eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 -- 172.21.2.117:0/2973007258 >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 msgr2=0x7f75980ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 -- 172.21.2.117:0/2973007258 >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 msgr2=0x7f75980eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 -- 172.21.2.117:0/2973007258 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980eee00 con 0x7f75980eb040 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7597fff700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f75977fe700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f7596ffd700 1 --2- 172.21.2.117:0/2973007258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 0x7f75980eb440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7580006650 tx=0x7f7580009a80).ready entity=mon.2 client_cookie=cd8bce7df5178dc0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f75967fc700 1 -- 172.21.2.117:0/2973007258 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f758000c0c0 con 0x7f75980eb040 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f75967fc700 1 -- 172.21.2.117:0/2973007258 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758000c3c0 con 0x7f75980eb040 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f75967fc700 1 -- 172.21.2.117:0/2973007258 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f758000c8e0 con 0x7f75980eb040 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f759db3f700 1 -- 172.21.2.117:0/2973007258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 msgr2=0x7f75980eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.625 7f759db3f700 1 --2- 172.21.2.117:0/2973007258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 0x7f75980eb440 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f7580006650 tx=0x7f7580009a80).stop 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 -- 172.21.2.117:0/2973007258 shutdown_connections 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.125:3300/0 conn(0x7f75980ee3e0 0x7f75980ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 --2- 172.21.2.117:0/2973007258 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980eb040 0x7f75980eb440 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 --2- 172.21.2.117:0/2973007258 >> v2:172.21.2.117:3300/0 conn(0x7f75980d18f0 0x7f75980eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 -- 172.21.2.117:0/2973007258 >> 172.21.2.117:0/2973007258 conn(0x7f75980cf060 msgr2=0x7f75980cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 -- 172.21.2.117:0/2973007258 shutdown_connections 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.626 7f759db3f700 1 -- 172.21.2.117:0/2973007258 wait complete. 2021-02-10T11:58:13.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.640 7f759db3f700 1 Processor -- start 2021-02-10T11:58:13.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.640 7f759db3f700 1 -- start start 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 0x7f75980dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7598085ec0 0x7f75980862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 0x7f75980a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980a67e0 con 0x7f75980a6ce0 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980a7120 con 0x7f7598085ec0 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f759db3f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980a2a60 con 0x7f75980a2040 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f75977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 0x7f75980dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f75977fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 0x7f75980dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32964/0 (socket says 172.21.2.117:32964) 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f75977fe700 1 -- 172.21.2.117:0/412847093 learned_addr learned my addr 172.21.2.117:0/412847093 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7597fff700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 0x7f75980a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7596ffd700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7598085ec0 0x7f75980862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7597fff700 1 -- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7598085ec0 msgr2=0x7f75980862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7597fff700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7598085ec0 0x7f75980862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7597fff700 1 -- 172.21.2.117:0/412847093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 msgr2=0x7f75980dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.641 7f7597fff700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 0x7f75980dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f7597fff700 1 -- 172.21.2.117:0/412847093 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75981a2dc0 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f7597fff700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 0x7f75980a2440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f7590006ed0 tx=0x7f759000b910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7590000c10 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759000bd90 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f759db3f700 1 -- 172.21.2.117:0/412847093 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75981a2dc0 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7590005330 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.642 7f759db3f700 1 -- 172.21.2.117:0/412847093 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75981a3350 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.643 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7590006030 con 0x7f75980a2040 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.643 7f758ffff700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 0x7f757c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:13.648 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.643 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7590032490 con 0x7f75980a2040 2021-02-10T11:58:13.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.643 7f75977fe700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 0x7f757c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:13.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.644 7f75977fe700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 0x7f757c01abc0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:13.676 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.674 7f759db3f700 1 -- 172.21.2.117:0/412847093 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75980daff0 con 0x7f75980a2040 2021-02-10T11:58:13.680 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.679 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7590000ac0 con 0x7f75980a2040 2021-02-10T11:58:13.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.807 7f759db3f700 1 -- 172.21.2.117:0/412847093 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f75980ede60 con 0x7f75980a2040 2021-02-10T11:58:13.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.808 7f758ffff700 1 -- 172.21.2.117:0/412847093 <== mon.2 v2:172.21.2.125:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v196) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7590006030 con 0x7f75980a2040 2021-02-10T11:58:13.810 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:13.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 -- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 msgr2=0x7f757c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 0x7f757c01abc0 crc :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 -- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 msgr2=0x7f75980a2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 0x7f75980a2440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f7590006ed0 tx=0x7f759000b910).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 -- 172.21.2.117:0/412847093 shutdown_connections 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f75980a2040 0x7f75980a2440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f7598085ec0 0x7f75980862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f757c0186c0 0x7f757c01abc0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 --2- 172.21.2.117:0/412847093 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f75980a6ce0 0x7f75980dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.809 7f759db3f700 1 -- 172.21.2.117:0/412847093 >> 172.21.2.117:0/412847093 conn(0x7f75980b6540 msgr2=0x7f759808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.810 7f759db3f700 1 -- 172.21.2.117:0/412847093 shutdown_connections 2021-02-10T11:58:13.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:13.810 7f759db3f700 1 -- 172.21.2.117:0/412847093 wait complete. 2021-02-10T11:58:13.820 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:58:13.820 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 Processor -- start 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 -- start start 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440d18e0 0x7fd3440ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3440ef3f0 0x7fd3441b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3440ede60 con 0x7fd3440eb000 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd344088020 con 0x7fd3440ef3f0 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.232 7fd3492cb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440d1ce0 con 0x7fd3440d18e0 2021-02-10T11:58:14.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3440ef3f0 0x7fd3441b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34287c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fd3440eb000 0x7fd3440eb400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33220/0 (socket says 172.21.2.117:33220) 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fd3440ef3f0 0x7fd3441b7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32970/0 (socket says 172.21.2.117:32970) 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34287c700 1 -- 172.21.2.117:0/2423905800 learned_addr learned my addr 172.21.2.117:0/2423905800 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34307d700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440d18e0 0x7fd3440ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 -- 172.21.2.117:0/2423905800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440d18e0 msgr2=0x7fd3440ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440d18e0 0x7fd3440ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34207b700 1 -- 172.21.2.117:0/2423905800 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208721967 0 0) 0x7fd3380009b0 con 0x7fd3440eb000 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 -- 172.21.2.117:0/2423905800 >> v1:172.21.2.117:6789/0 conn(0x7fd3440eb000 legacy=0x7fd3440eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 -- 172.21.2.117:0/2423905800 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3441b8f90 con 0x7fd3440ef3f0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34387e700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440ef3f0 0x7fd3441b7960 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 rx=0x7fd33c006e30 tx=0x7fd33c009de0).ready entity=mon.0 client_cookie=b2a3a3dd602dd07 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34207b700 1 -- 172.21.2.117:0/2423905800 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33c00b300 con 0x7fd3440ef3f0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34207b700 1 -- 172.21.2.117:0/2423905800 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00b580 con 0x7fd3440ef3f0 2021-02-10T11:58:14.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.233 7fd34207b700 1 -- 172.21.2.117:0/2423905800 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33c00baa0 con 0x7fd3440ef3f0 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 -- 172.21.2.117:0/2423905800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440ef3f0 msgr2=0x7fd3441b7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440ef3f0 0x7fd3441b7960 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 rx=0x7fd33c006e30 tx=0x7fd33c009de0).stop 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 -- 172.21.2.117:0/2423905800 shutdown_connections 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440ef3f0 0x7fd3441b7960 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 --2- 172.21.2.117:0/2423905800 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440d18e0 0x7fd3440ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 -- 172.21.2.117:0/2423905800 >> 172.21.2.117:0/2423905800 conn(0x7fd3440ced60 msgr2=0x7fd3440cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 -- 172.21.2.117:0/2423905800 shutdown_connections 2021-02-10T11:58:14.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.234 7fd3492cb700 1 -- 172.21.2.117:0/2423905800 wait complete. 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.249 7fd3492cb700 1 Processor -- start 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.249 7fd3492cb700 1 -- start start 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440e8340 0x7fd3440df850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 0x7fd3440e0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440ade90 con 0x7fd3440e93c0 2021-02-10T11:58:14.255 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440dfed0 con 0x7fd3441a3480 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd3492cb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3440b69b0 con 0x7fd3440e8340 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:32974/0 (socket says 172.21.2.117:32974) 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 -- 172.21.2.117:0/1385835493 learned_addr learned my addr 172.21.2.117:0/1385835493 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34387e700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 0x7fd3440e0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34307d700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440e8340 0x7fd3440df850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 -- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440e8340 msgr2=0x7fd3440df850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440e8340 0x7fd3440df850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 -- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 msgr2=0x7fd3440e0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 0x7fd3440e0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34287c700 1 -- 172.21.2.117:0/1385835493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3440e97c0 con 0x7fd3440e93c0 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.250 7fd34387e700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 0x7fd3440e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:14.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd34287c700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 rx=0x7fd338001c60 tx=0x7fd33800a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd33800acc0 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd338004350 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fd338004870 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3440e97c0 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.251 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3440e97c0 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.252 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fd338000c90 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.252 7fd32bfff700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.252 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fd3380326c0 con 0x7fd3440e93c0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.252 7fd34307d700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:14.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.252 7fd34307d700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 0x7fd32401abc0 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:14.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.282 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd344087290 con 0x7fd3440e93c0 2021-02-10T11:58:14.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.285 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fd338004140 con 0x7fd3440e93c0 2021-02-10T11:58:14.415 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.413 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd344087290 con 0x7fd3440e93c0 2021-02-10T11:58:14.416 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.414 7fd32bfff700 1 -- 172.21.2.117:0/1385835493 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+2108 (secure 0 0 0) 0x7fd338004ef0 con 0x7fd3440e93c0 2021-02-10T11:58:14.416 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:14.416 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13806,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3326472561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3326472561},{"type":"v1","addr":"172.21.2.117:6833","nonce":3326472561}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":166}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:58:04.413160","modified":"2021-02-10 11:58:06.419787","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9775":{"gid":9775,"name":"b","rank":0,"incarnation":167,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1289009861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1289009861},{"type":"v1","addr":"172.21.2.125:6835","nonce":1289009861}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:58:14.417 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 170 2021-02-10T11:58:14.417 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 msgr2=0x7fd32401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 0x7fd32401abc0 crc :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 msgr2=0x7fd3441a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 rx=0x7fd338001c60 tx=0x7fd33800a8d0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 shutdown_connections 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fd3240186c0 0x7fd32401abc0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fd3440e93c0 0x7fd3441a2e60 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fd3441a3480 0x7fd3440e0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 --2- 172.21.2.117:0/1385835493 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fd3440e8340 0x7fd3440df850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:14.418 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 >> 172.21.2.117:0/1385835493 conn(0x7fd3440c3fb0 msgr2=0x7fd3440a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:14.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 shutdown_connections 2021-02-10T11:58:14.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:14.416 7fd3492cb700 1 -- 172.21.2.117:0/1385835493 wait complete. 2021-02-10T11:58:14.630 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 170, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:58:04.413160', 'modified': '2021-02-10 11:58:06.419787', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9775': {'gid': 9775, 'name': 'b', 'rank': 0, 'incarnation': 167, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1289009861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1289009861}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1289009861}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:58:14.630 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:58:14.631 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:58:15.031 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9775, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 10.614879662, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 170, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 196, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 196, 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 16.260552748999999 2021-02-10T11:58:15.032 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:58:15.036 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9775, "want_state": "up:active", "state": "up:active", "rank_uptime": 10.614879662, "mdsmap_epoch": 170, "osdmap_epoch": 196, "osdmap_epoch_barrier": 196, "uptime": 16.260552748999999 } 2021-02-10T11:58:15.036 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:58:15.036 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2021-02-10T11:58:15.036 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2021-02-10T11:58:15.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:15.036 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:15.372 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:58:15.717 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:58:15.717 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:58:16.061 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:58:16.061 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:58:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:58:16.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:16.062 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:58:16.398 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2021-02-10T11:58:16.398 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:16.401 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:58:16.745 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:58:16.746 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:58:16.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:2021-02-10 11:58:16.811 7f20fa776f80 -1 init, newargv = 0x555c4bf89f90 newargc=9 2021-02-10T11:58:16.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[196948]: starting ceph client 2021-02-10T11:58:16.819 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[196948]: starting fuse 2021-02-10T11:58:17.089 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:58:17.090 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:58:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:58:17.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:17.092 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:58:17.427 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:58:17.428 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2021-02-10T11:58:17.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:58:17.428 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.0")) 2021-02-10T11:58:17.429 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:58:17.791 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.0.196948.asok 2021-02-10T11:58:17.794 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.196948.asok 2021-02-10T11:58:17.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:17.794 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.196948.asok status 2021-02-10T11:58:18.185 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "0", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "196948", 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:18.186 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 13965, 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 13965 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1294854258 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:18.187 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 1294854258 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.13965 172.21.2.117:0/1294854258", 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/1294854258", 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 170, 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 196, 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:58:18.188 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:58:18.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:18.190 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:18.190 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:18.527 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:18.527 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:18.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:18.528 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:18.875 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.285 7f326625d700 1 Processor -- start 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.285 7f326625d700 1 -- start start 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326625d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f32600d18e0 0x7f32600ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326625d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32600ede60 con 0x7f32600ef3f0 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326625d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32600d1ce0 con 0x7f32600eb000 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326625d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32600ef7f0 con 0x7f32600d18e0 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326487a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f32600ef3f0 0x7f32601b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33246/0 (socket says 172.21.2.117:33246) 2021-02-10T11:58:19.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f326487a700 1 -- 172.21.2.117:0/317028145 learned_addr learned my addr 172.21.2.117:0/317028145 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> v2:172.21.2.125:3300/0 conn(0x7f32600d18e0 0x7f32600ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.286 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494720671 0 0) 0x7f3258001500 con 0x7f32600ef3f0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3248002fe0 con 0x7f32600ef3f0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310333873 0 0) 0x7f3254000c60 con 0x7f32600eb000 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3248003210 con 0x7f32600eb000 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 183035883 0 0) 0x7f3258001500 con 0x7f32600ef3f0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 >> v2:172.21.2.125:3300/0 conn(0x7f32600d18e0 msgr2=0x7f32600ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 --2- 172.21.2.117:0/317028145 >> v2:172.21.2.125:3300/0 conn(0x7f32600d18e0 0x7f32600ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 >> v1:172.21.2.125:6789/0 conn(0x7f32600eb000 legacy=0x7f32600eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32601bb180 con 0x7f32600ef3f0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f3258001a00 con 0x7f32600ef3f0 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> v2:172.21.2.125:3300/0 conn(0x7f32600d18e0 0x7f32600ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:19.306 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 >> v1:172.21.2.117:6789/0 conn(0x7f32600ef3f0 legacy=0x7f32601b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 0x7f3248003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3248006830 0x7f3248008d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32480095a0 0x7f324800bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3248050320 con 0x7f32480038e0 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3248050540 con 0x7f32480095a0 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325effd700 1 -- 172.21.2.117:0/317028145 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3248050760 con 0x7f3248006830 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.287 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 0x7f3248003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f326487a700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32480095a0 0x7f324800bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 -- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3248006830 msgr2=0x7f3248008d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3248006830 0x7f3248008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 -- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32480095a0 msgr2=0x7f324800bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32480095a0 0x7f324800bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f326487a700 1 -- 172.21.2.117:0/317028145 reap_dead start 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 -- 172.21.2.117:0/317028145 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248050a00 con 0x7f32480038e0 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325ffff700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 0x7f3248003ce0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f3250006b00 tx=0x7f325000cbc0).ready entity=mon.0 client_cookie=417491799226d24 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32500068d0 con 0x7f32480038e0 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32500090d0 con 0x7f32480038e0 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f325effd700 1 -- 172.21.2.117:0/317028145 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32500095b0 con 0x7f32480038e0 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f326625d700 1 -- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 msgr2=0x7f3248003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.288 7f326625d700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 0x7f3248003ce0 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 rx=0x7f3250006b00 tx=0x7f325000cbc0).stop 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.289 7f326625d700 1 -- 172.21.2.117:0/317028145 shutdown_connections 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.289 7f326625d700 1 --2- 172.21.2.117:0/317028145 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f32480038e0 0x7f3248003ce0 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.289 7f326625d700 1 -- 172.21.2.117:0/317028145 >> 172.21.2.117:0/317028145 conn(0x7f32600ced60 msgr2=0x7f32600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.289 7f326625d700 1 -- 172.21.2.117:0/317028145 shutdown_connections 2021-02-10T11:58:19.308 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.289 7f326625d700 1 -- 172.21.2.117:0/317028145 wait complete. 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.303 7f326625d700 1 Processor -- start 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.303 7f326625d700 1 -- start start 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32600b2160 0x7f32600e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f326008bbe0 0x7f326008bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326008b660 con 0x7f326008c600 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326008ca00 con 0x7f32600b2160 2021-02-10T11:58:19.310 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326625d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32600e9a80 con 0x7f326008bbe0 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33008/0 (socket says 172.21.2.117:33008) 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 -- 172.21.2.117:0/4063067819 learned_addr learned my addr 172.21.2.117:0/4063067819 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 -- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f326008bbe0 msgr2=0x7f326008bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f326008bbe0 0x7f326008bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 -- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32600b2160 msgr2=0x7f32600e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32600b2160 0x7f32600e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.304 7f326487a700 1 -- 172.21.2.117:0/4063067819 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326008a8a0 con 0x7f326008c600 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f326487a700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7f325800b960 tx=0x7f325800bd00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3258007c70 con 0x7f326008c600 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32580067f0 con 0x7f326008c600 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f32580064d0 con 0x7f326008c600 2021-02-10T11:58:19.311 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f326625d700 1 -- 172.21.2.117:0/4063067819 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326008a8a0 con 0x7f326008c600 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f326625d700 1 -- 172.21.2.117:0/4063067819 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326008ae30 con 0x7f326008c600 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.305 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3258000d30 con 0x7f326008c600 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.306 7f325cff9700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 0x7f324001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.306 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3258032cd0 con 0x7f326008c600 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.306 7f325ffff700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 0x7f324001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:19.312 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.306 7f325ffff700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 0x7f324001abd0 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:19.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.335 7f326625d700 1 -- 172.21.2.117:0/4063067819 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32600ede60 con 0x7f326008c600 2021-02-10T11:58:19.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.338 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f32580062e0 con 0x7f326008c600 2021-02-10T11:58:19.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.461 7f326625d700 1 -- 172.21.2.117:0/4063067819 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32600ede60 con 0x7f326008c600 2021-02-10T11:58:19.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.462 7f325cff9700 1 -- 172.21.2.117:0/4063067819 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f3258000d30 con 0x7f326008c600 2021-02-10T11:58:19.464 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:19.464 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.463 7f326625d700 1 -- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 msgr2=0x7f324001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.463 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 0x7f324001abd0 crc :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.463 7f326625d700 1 -- 172.21.2.117:0/4063067819 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 msgr2=0x7f326008a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.463 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 rx=0x7f325800b960 tx=0x7f325800bd00).stop 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 -- 172.21.2.117:0/4063067819 shutdown_connections 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f32400186d0 0x7f324001abd0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f326008c600 0x7f326008a280 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f326008bbe0 0x7f326008bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:19.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 --2- 172.21.2.117:0/4063067819 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f32600b2160 0x7f32600e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:19.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 -- 172.21.2.117:0/4063067819 >> 172.21.2.117:0/4063067819 conn(0x7f32600bb5e0 msgr2=0x7f3260090e50 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:19.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 -- 172.21.2.117:0/4063067819 shutdown_connections 2021-02-10T11:58:19.467 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.464 7f326625d700 1 -- 172.21.2.117:0/4063067819 wait complete. 2021-02-10T11:58:19.474 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-10 11:22:23.728210', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 42, 'pool_name': 'cephfs_metadata', 'create_time': '2021-02-10 11:58:02.381112', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '195', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 43, 'pool_name': 'cephfs_data', 'create_time': '2021-02-10 11:58:03.405092', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'warn', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2021-02-10T11:58:19.475 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get rbd pg_num 2021-02-10T11:58:19.900 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.888 7fcb5eacf700 1 Processor -- start 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.888 7fcb5eacf700 1 -- start start 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.888 7fcb5eacf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb580d18f0 0x7fcb580eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.888 7fcb5eacf700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb580eb040 0x7fcb580eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.888 7fcb5eacf700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb580ede60 con 0x7fcb580ee3e0 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb5eacf700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb580eba60 con 0x7fcb580eb040 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb5eacf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580d1cf0 con 0x7fcb580d18f0 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb580eb040 0x7fcb580eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fcb580eb040 0x7fcb580eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33012/0 (socket says 172.21.2.117:33012) 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb5d0ec700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcb580ee3e0 0x7fcb580ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33262/0 (socket says 172.21.2.117:33262) 2021-02-10T11:58:20.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 -- 172.21.2.117:0/4144729195 learned_addr learned my addr 172.21.2.117:0/4144729195 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 -- 172.21.2.117:0/4144729195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb580d18f0 msgr2=0x7fcb580eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 --2- 172.21.2.117:0/4144729195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb580d18f0 0x7fcb580eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 -- 172.21.2.117:0/4144729195 >> v1:172.21.2.117:6789/0 conn(0x7fcb580ee3e0 legacy=0x7fcb580ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 -- 172.21.2.117:0/4144729195 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb580eefd0 con 0x7fcb580eb040 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.889 7fcb57fff700 1 --2- 172.21.2.117:0/4144729195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb580eb040 0x7fcb580eb440 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 rx=0x7fcb48006a80 tx=0x7fcb48009f00).ready entity=mon.0 client_cookie=1eb25da6257c2a2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb577fe700 1 -- 172.21.2.117:0/4144729195 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4800b5c0 con 0x7fcb580eb040 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb577fe700 1 -- 172.21.2.117:0/4144729195 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4800b8c0 con 0x7fcb580eb040 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb577fe700 1 -- 172.21.2.117:0/4144729195 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4800bde0 con 0x7fcb580eb040 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 -- 172.21.2.117:0/4144729195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb580eb040 msgr2=0x7fcb580eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 --2- 172.21.2.117:0/4144729195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb580eb040 0x7fcb580eb440 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 rx=0x7fcb48006a80 tx=0x7fcb48009f00).stop 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 -- 172.21.2.117:0/4144729195 shutdown_connections 2021-02-10T11:58:20.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 --2- 172.21.2.117:0/4144729195 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb580eb040 0x7fcb580eb440 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 --2- 172.21.2.117:0/4144729195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb580d18f0 0x7fcb580eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 -- 172.21.2.117:0/4144729195 >> 172.21.2.117:0/4144729195 conn(0x7fcb580cf060 msgr2=0x7fcb580cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 -- 172.21.2.117:0/4144729195 shutdown_connections 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.890 7fcb5eacf700 1 -- 172.21.2.117:0/4144729195 wait complete. 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 Processor -- start 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 -- start start 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb5809ed70 0x7fcb5809f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb5809f790 0x7fcb580b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5809fb90 con 0x7fcb581a4240 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580dc640 con 0x7fcb5809ed70 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5eacf700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58097270 con 0x7fcb5809f790 2021-02-10T11:58:20.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.905 7fcb5c8eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33014/0 (socket says 172.21.2.117:33014) 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 -- 172.21.2.117:0/4113277512 learned_addr learned my addr 172.21.2.117:0/4113277512 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 -- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb5809f790 msgr2=0x7fcb580b8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb5809f790 0x7fcb580b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 -- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb5809ed70 msgr2=0x7fcb5809f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb5809ed70 0x7fcb5809f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 -- 172.21.2.117:0/4113277512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb580b8780 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5c8eb700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 rx=0x7fcb40006f00 tx=0x7fcb4000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb4000bd30 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb40000c90 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.906 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb580b8780 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.907 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb580b8780 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.907 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcb400022a0 con 0x7fcb581a4240 2021-02-10T11:58:20.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.907 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcb40005310 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.907 7fcb557fa700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 0x7fcb3c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.907 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcb400328c0 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.908 7fcb57fff700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 0x7fcb3c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.908 7fcb57fff700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 0x7fcb3c01abc0 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.938 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5809fb90 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:19.941 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcb40006b90 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.067 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fcb5809fb90 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.067 7fcb557fa700 1 -- 172.21.2.117:0/4113277512 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v196) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcb4000bd30 con 0x7fcb581a4240 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.069 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 msgr2=0x7fcb3c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.069 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 0x7fcb3c01abc0 crc :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.069 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 msgr2=0x7fcb581a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.069 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 secure :-1 s=READY pgs=3271 cs=0 l=1 rev1=1 rx=0x7fcb40006f00 tx=0x7fcb4000b940).stop 2021-02-10T11:58:20.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 shutdown_connections 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcb5809f790 0x7fcb580b8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcb5809ed70 0x7fcb5809f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcb3c0186c0 0x7fcb3c01abc0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 --2- 172.21.2.117:0/4113277512 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcb581a4240 0x7fcb581a4640 unknown :-1 s=CLOSED pgs=3271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 >> 172.21.2.117:0/4113277512 conn(0x7fcb58087090 msgr2=0x7fcb5808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 shutdown_connections 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.070 7fcb5eacf700 1 -- 172.21.2.117:0/4113277512 wait complete. 2021-02-10T11:58:20.442 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:20.442 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2021-02-10T11:58:20.479 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:58:20.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 Processor -- start 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 -- start start 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fcbd81b69a0 0x7fcbd81b8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd80ede60 con 0x7fcbd80ef3f0 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbd80eba00 con 0x7fcbd80d18f0 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.893 7fcbdd02d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbd80d1cf0 con 0x7fcbd81b69a0 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd659c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fcbd80ef3f0 0x7fcbd80eb210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33274/0 (socket says 172.21.2.117:33274) 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd659c700 1 -- 172.21.2.117:0/1279334106 learned_addr learned my addr 172.21.2.117:0/1279334106 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd6d9d700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fcbd80d18f0 0x7fcbd80eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:54430/0 (socket says 172.21.2.117:54430) 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd759e700 1 --2- 172.21.2.117:0/1279334106 >> v2:172.21.2.125:3300/0 conn(0x7fcbd81b69a0 0x7fcbd81b8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798424734 0 0) 0x7fcbc00009b0 con 0x7fcbd80ef3f0 2021-02-10T11:58:20.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc4002fe0 con 0x7fcbd80ef3f0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261372407 0 0) 0x7fcbc8000c60 con 0x7fcbd80d18f0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc4004370 con 0x7fcbd80d18f0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd759e700 1 -- 172.21.2.117:0/1279334106 >> v1:172.21.2.125:6789/0 conn(0x7fcbd80d18f0 legacy=0x7fcbd80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd759e700 1 -- 172.21.2.117:0/1279334106 >> v1:172.21.2.117:6789/0 conn(0x7fcbd80ef3f0 legacy=0x7fcbd80eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd759e700 1 -- 172.21.2.117:0/1279334106 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd81bb3f0 con 0x7fcbd81b69a0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.894 7fcbd759e700 1 --2- 172.21.2.117:0/1279334106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81b69a0 0x7fcbd81b8db0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7fcbd0006e50 tx=0x7fcbd0009e70).ready entity=mon.1 client_cookie=5cb4901831ef9646 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd0006a20 con 0x7fcbd81b69a0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd0006790 con 0x7fcbd81b69a0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbd5d9b700 1 -- 172.21.2.117:0/1279334106 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd000b700 con 0x7fcbd81b69a0 2021-02-10T11:58:20.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 -- 172.21.2.117:0/1279334106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81b69a0 msgr2=0x7fcbd81b8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 --2- 172.21.2.117:0/1279334106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81b69a0 0x7fcbd81b8db0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7fcbd0006e50 tx=0x7fcbd0009e70).stop 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 -- 172.21.2.117:0/1279334106 shutdown_connections 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 --2- 172.21.2.117:0/1279334106 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81b69a0 0x7fcbd81b8db0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 -- 172.21.2.117:0/1279334106 >> 172.21.2.117:0/1279334106 conn(0x7fcbd80cf060 msgr2=0x7fcbd80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.895 7fcbdd02d700 1 -- 172.21.2.117:0/1279334106 shutdown_connections 2021-02-10T11:58:20.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.896 7fcbdd02d700 1 -- 172.21.2.117:0/1279334106 wait complete. 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 Processor -- start 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 -- start start 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd8097470 0x7fcbd81a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 0x7fcbd80b8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 0x7fcbd80b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80b9260 con 0x7fcbd81a7870 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80b9450 con 0x7fcbd81a7c70 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.910 7fcbdd02d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd80b9670 con 0x7fcbd8097470 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 0x7fcbd80b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.916 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 0x7fcbd80b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd659c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 0x7fcbd80b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33028/0 (socket says 172.21.2.117:33028) 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd659c700 1 -- 172.21.2.117:0/3395597238 learned_addr learned my addr 172.21.2.117:0/3395597238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd6d9d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd8097470 0x7fcbd81a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 -- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd8097470 msgr2=0x7fcbd81a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd8097470 0x7fcbd81a7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 -- 172.21.2.117:0/3395597238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 msgr2=0x7fcbd80b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 0x7fcbd80b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 -- 172.21.2.117:0/3395597238 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd80a7920 con 0x7fcbd81a7c70 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbd759e700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 0x7fcbd80b8c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fcbd0009930 tx=0x7fcbd000bc70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd000bdb0 con 0x7fcbd81a7c70 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd0009390 con 0x7fcbd81a7c70 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd80a7920 con 0x7fcbd81a7c70 2021-02-10T11:58:20.917 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fcbd0002220 con 0x7fcbd81a7c70 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.911 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd80a7920 con 0x7fcbd81a7c70 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.912 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fcbd00056b0 con 0x7fcbd81a7c70 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.912 7fcbcf7fe700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 0x7fcbbc01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.912 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fcbd00329b0 con 0x7fcbd81a7c70 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.912 7fcbd6d9d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 0x7fcbbc01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:20.918 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.913 7fcbd6d9d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 0x7fcbbc01abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:20.945 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.943 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd80bbaf0 con 0x7fcbd81a7c70 2021-02-10T11:58:20.949 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:20.947 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fcbd0027fd0 con 0x7fcbd81a7c70 2021-02-10T11:58:21.079 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.077 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7fcbd80bbaf0 con 0x7fcbd81a7c70 2021-02-10T11:58:21.080 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.078 7fcbcf7fe700 1 -- 172.21.2.117:0/3395597238 <== mon.1 v2:172.21.2.125:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v196) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcbd0027fd0 con 0x7fcbd81a7c70 2021-02-10T11:58:21.080 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:21.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.079 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 msgr2=0x7fcbbc01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.081 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.079 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 0x7fcbbc01abc0 crc :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.079 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 msgr2=0x7fcbd80b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.079 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 0x7fcbd80b8c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 rx=0x7fcbd0009930 tx=0x7fcbd000bc70).stop 2021-02-10T11:58:21.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 shutdown_connections 2021-02-10T11:58:21.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fcbbc0186c0 0x7fcbbc01abc0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.082 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fcbd81a7870 0x7fcbd80b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fcbd81a7c70 0x7fcbd80b8c40 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 --2- 172.21.2.117:0/3395597238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fcbd8097470 0x7fcbd81a7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 >> 172.21.2.117:0/3395597238 conn(0x7fcbd80bbea0 msgr2=0x7fcbd808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:21.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 shutdown_connections 2021-02-10T11:58:21.083 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.080 7fcbdd02d700 1 -- 172.21.2.117:0/3395597238 wait complete. 2021-02-10T11:58:21.091 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd pool get cephfs_data pg_num 2021-02-10T11:58:21.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 Processor -- start 2021-02-10T11:58:21.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 -- start start 2021-02-10T11:58:21.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc1b07f0 0x7f3fbc0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0e77f0 con 0x7f3fbc0e8790 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0e92f0 con 0x7f3fbc0e7d70 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fc2c76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0e5ce0 con 0x7f3fbc1b07f0 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fbbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fbbfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33038/0 (socket says 172.21.2.117:33038) 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.505 7f3fbbfff700 1 -- 172.21.2.117:0/2253190126 learned_addr learned my addr 172.21.2.117:0/2253190126 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbbfff700 1 -- 172.21.2.117:0/2253190126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc1b07f0 msgr2=0x7f3fbc0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbbfff700 1 --2- 172.21.2.117:0/2253190126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc1b07f0 0x7f3fbc0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbbfff700 1 -- 172.21.2.117:0/2253190126 >> v1:172.21.2.125:6789/0 conn(0x7f3fbc0e8790 legacy=0x7f3fbc0e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbbfff700 1 -- 172.21.2.117:0/2253190126 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0e54b0 con 0x7f3fbc0e7d70 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbbfff700 1 --2- 172.21.2.117:0/2253190126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f3fac006700 tx=0x7f3fac009b80).ready entity=mon.0 client_cookie=84123d8b02c63e32 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbb7fe700 1 -- 172.21.2.117:0/2253190126 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fac00c0c0 con 0x7f3fbc0e7d70 2021-02-10T11:58:21.524 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbb7fe700 1 -- 172.21.2.117:0/2253190126 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac00c400 con 0x7f3fbc0e7d70 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fbb7fe700 1 -- 172.21.2.117:0/2253190126 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fac00c8e0 con 0x7f3fbc0e7d70 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fc2c76700 1 -- 172.21.2.117:0/2253190126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0e7d70 msgr2=0x7f3fbc0e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.506 7f3fc2c76700 1 --2- 172.21.2.117:0/2253190126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 rx=0x7f3fac006700 tx=0x7f3fac009b80).stop 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 -- 172.21.2.117:0/2253190126 shutdown_connections 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 --2- 172.21.2.117:0/2253190126 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0e7d70 0x7f3fbc0e8170 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 --2- 172.21.2.117:0/2253190126 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc1b07f0 0x7f3fbc0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 -- 172.21.2.117:0/2253190126 >> 172.21.2.117:0/2253190126 conn(0x7f3fbc0ce500 msgr2=0x7f3fbc0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 -- 172.21.2.117:0/2253190126 shutdown_connections 2021-02-10T11:58:21.525 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.507 7f3fc2c76700 1 -- 172.21.2.117:0/2253190126 wait complete. 2021-02-10T11:58:21.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.521 7f3fc2c76700 1 Processor -- start 2021-02-10T11:58:21.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.521 7f3fc2c76700 1 -- start start 2021-02-10T11:58:21.527 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 0x7f3fbc0ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0ad460 0x7f3fbc1a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0a6730 con 0x7f3fbc0beb10 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1a4000 con 0x7f3fbc0bb2b0 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc2c76700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc1a4220 con 0x7f3fbc0ad460 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc0a92700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 0x7f3fbc0ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fbbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fbbfff700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:37424/0 (socket says 172.21.2.117:37424) 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fbbfff700 1 -- 172.21.2.117:0/3985904778 learned_addr learned my addr 172.21.2.117:0/3985904778 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:21.528 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.522 7f3fc1293700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0ad460 0x7f3fbc1a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 -- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0ad460 msgr2=0x7f3fbc1a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0ad460 0x7f3fbc1a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 -- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 msgr2=0x7f3fbc1a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 -- 172.21.2.117:0/3985904778 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc108c60 con 0x7f3fbc0beb10 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fbbfff700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:21.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc0a92700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 0x7f3fbc0ace40 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 rx=0x7f3fa400b510 tx=0x7f3fa400b8b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa400bca0 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa40063f0 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3fa40020d0 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc108c60 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.523 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc108c60 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.524 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3fa4005310 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.524 7f3fb97fa700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 0x7f3fa001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.524 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3fa4032630 con 0x7f3fbc0beb10 2021-02-10T11:58:21.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.524 7f3fbbfff700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 0x7f3fa001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:21.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.524 7f3fbbfff700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 0x7f3fa001abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:21.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.555 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0e77f0 con 0x7f3fbc0beb10 2021-02-10T11:58:21.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.557 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3fa4015b50 con 0x7f3fbc0beb10 2021-02-10T11:58:21.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.684 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f3fbc0e77f0 con 0x7f3fbc0beb10 2021-02-10T11:58:21.686 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.684 7f3fb97fa700 1 -- 172.21.2.117:0/3985904778 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v196) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3fa4015b50 con 0x7f3fbc0beb10 2021-02-10T11:58:21.687 INFO:teuthology.orchestra.run.gibba017.stdout:pg_num: 8 2021-02-10T11:58:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.686 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 msgr2=0x7f3fa001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.686 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 0x7f3fa001abc0 crc :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.686 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 msgr2=0x7f3fbc0ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:21.688 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.686 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 0x7f3fbc0ace40 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 rx=0x7f3fa400b510 tx=0x7f3fa400b8b0).stop 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 shutdown_connections 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3fbc0ad460 0x7f3fbc1a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3fbc0bb2b0 0x7f3fbc1a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3fa00186c0 0x7f3fa001abc0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 --2- 172.21.2.117:0/3985904778 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3fbc0beb10 0x7f3fbc0ace40 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:21.689 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 >> 172.21.2.117:0/3985904778 conn(0x7f3fbc0c3f30 msgr2=0x7f3fbc111070 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:21.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 shutdown_connections 2021-02-10T11:58:21.690 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:21.687 7f3fc2c76700 1 -- 172.21.2.117:0/3985904778 wait complete. 2021-02-10T11:58:21.697 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2021-02-10T11:58:21.697 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs dump --format=json 2021-02-10T11:58:22.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 Processor -- start 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 -- start start 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa600d18e0 0x7ffa600ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600ef3f0 0x7ffa601b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa600ede60 con 0x7ffa600eb000 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa600ef7f0 con 0x7ffa600d18e0 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa66d2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600d1ce0 con 0x7ffa600ef3f0 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa64b48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa600d18e0 0x7ffa600ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa64b48700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7ffa600d18e0 0x7ffa600ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33048/0 (socket says 172.21.2.117:33048) 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.107 7ffa64b48700 1 -- 172.21.2.117:0/1591907620 learned_addr learned my addr 172.21.2.117:0/1591907620 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa64b48700 1 -- 172.21.2.117:0/1591907620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600ef3f0 msgr2=0x7ffa601b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:22.126 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa64b48700 1 --2- 172.21.2.117:0/1591907620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600ef3f0 0x7ffa601b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa64b48700 1 -- 172.21.2.117:0/1591907620 >> v1:172.21.2.125:6789/0 conn(0x7ffa600eb000 legacy=0x7ffa600eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa64b48700 1 -- 172.21.2.117:0/1591907620 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa601b8f90 con 0x7ffa600d18e0 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa64b48700 1 --2- 172.21.2.117:0/1591907620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa600d18e0 0x7ffa600ea9e0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7ffa48006700 tx=0x7ffa48009b80).ready entity=mon.0 client_cookie=87800f37b2d02313 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa5f7fe700 1 -- 172.21.2.117:0/1591907620 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa480009b0 con 0x7ffa600d18e0 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa5f7fe700 1 -- 172.21.2.117:0/1591907620 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4800c0c0 con 0x7ffa600d18e0 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa5f7fe700 1 -- 172.21.2.117:0/1591907620 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa4800c5e0 con 0x7ffa600d18e0 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa66d2c700 1 -- 172.21.2.117:0/1591907620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa600d18e0 msgr2=0x7ffa600ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.108 7ffa66d2c700 1 --2- 172.21.2.117:0/1591907620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa600d18e0 0x7ffa600ea9e0 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7ffa48006700 tx=0x7ffa48009b80).stop 2021-02-10T11:58:22.127 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 -- 172.21.2.117:0/1591907620 shutdown_connections 2021-02-10T11:58:22.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 --2- 172.21.2.117:0/1591907620 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600ef3f0 0x7ffa601b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 --2- 172.21.2.117:0/1591907620 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa600d18e0 0x7ffa600ea9e0 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:22.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 -- 172.21.2.117:0/1591907620 >> 172.21.2.117:0/1591907620 conn(0x7ffa600ced60 msgr2=0x7ffa600cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:22.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 -- 172.21.2.117:0/1591907620 shutdown_connections 2021-02-10T11:58:22.128 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.109 7ffa66d2c700 1 -- 172.21.2.117:0/1591907620 wait complete. 2021-02-10T11:58:22.129 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 Processor -- start 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 -- start start 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa600980c0 0x7ffa6019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600984c0 0x7ffa6019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600b1f30 con 0x7ffa6019dd40 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6019d830 con 0x7ffa600980c0 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa66d2c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa601a55e0 con 0x7ffa600984c0 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa65349700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa65349700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33058/0 (socket says 172.21.2.117:33058) 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.124 7ffa65349700 1 -- 172.21.2.117:0/1676645054 learned_addr learned my addr 172.21.2.117:0/1676645054 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:22.130 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 -- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600984c0 msgr2=0x7ffa6019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600984c0 0x7ffa6019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 -- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa600980c0 msgr2=0x7ffa6019ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa600980c0 0x7ffa6019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 -- 172.21.2.117:0/1676645054 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6019e760 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa65349700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 rx=0x7ffa58009980 tx=0x7ffa5800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa58005af0 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa58005d70 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ffa580062b0 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6019e760 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.125 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6019ecf0 con 0x7ffa6019dd40 2021-02-10T11:58:22.131 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.126 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ffa58006030 con 0x7ffa6019dd40 2021-02-10T11:58:22.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.126 7ffa5d7fa700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 0x7ffa4401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:22.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.126 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ffa58032a50 con 0x7ffa6019dd40 2021-02-10T11:58:22.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.126 7ffa64b48700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 0x7ffa4401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:22.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.127 7ffa64b48700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 0x7ffa4401abc0 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:22.159 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.157 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa60087210 con 0x7ffa6019dd40 2021-02-10T11:58:22.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.161 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ffa58006570 con 0x7ffa6019dd40 2021-02-10T11:58:22.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.291 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa600ede60 con 0x7ffa6019dd40 2021-02-10T11:58:22.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.292 7ffa5d7fa700 1 -- 172.21.2.117:0/1676645054 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+2108 (secure 0 0 0) 0x7ffa58006030 con 0x7ffa6019dd40 2021-02-10T11:58:22.294 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:58:22.294 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":170,"default_fscid":21,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":13806,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.2.117:6833/3326472561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6832","nonce":3326472561},{"type":"v1","addr":"172.21.2.117:6833","nonce":3326472561}]},"export_targets":[],"features":4611087854035861503,"flags":0,"epoch":166}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2021-02-10 11:58:04.413160","modified":"2021-02-10 11:58:06.419787","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"-1 (unspecified)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9775":{"gid":9775,"name":"b","rank":0,"incarnation":167,"state":"up:active","state_seq":3,"addr":"172.21.2.125:6835/1289009861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6834","nonce":1289009861},{"type":"v1","addr":"172.21.2.125:6835","nonce":1289009861}]},"export_targets":[],"features":4611087854035861503,"flags":0}},"data_pools":[43],"metadata_pool":42,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":21}]} 2021-02-10T11:58:22.294 INFO:teuthology.orchestra.run.gibba017.stderr:dumped fsmap epoch 170 2021-02-10T11:58:22.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.293 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 msgr2=0x7ffa4401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:22.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.293 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 0x7ffa4401abc0 crc :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:22.295 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.293 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 msgr2=0x7ffa6019e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.293 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 rx=0x7ffa58009980 tx=0x7ffa5800bcd0).stop 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 shutdown_connections 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ffa440186c0 0x7ffa4401abc0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ffa6019dd40 0x7ffa6019e140 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ffa600984c0 0x7ffa6019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 --2- 172.21.2.117:0/1676645054 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ffa600980c0 0x7ffa6019ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:22.296 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 >> 172.21.2.117:0/1676645054 conn(0x7ffa6019f610 msgr2=0x7ffa601a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:22.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 shutdown_connections 2021-02-10T11:58:22.297 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:22.294 7ffa66d2c700 1 -- 172.21.2.117:0/1676645054 wait complete. 2021-02-10T11:58:22.506 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 170, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2021-02-10 11:58:04.413160', 'modified': '2021-02-10 11:58:06.419787', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '-1 (unspecified)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9775': {'gid': 9775, 'name': 'b', 'rank': 0, 'incarnation': 167, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.2.125:6835/1289009861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.2.125:6834', 'nonce': 1289009861}, {'type': 'v1', 'addr': '172.21.2.125:6835', 'nonce': 1289009861}]}, 'export_targets': [], 'features': 4611087854035861503, 'flags': 0}}, 'data_pools': [43], 'metadata_pool': 42, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2021-02-10T11:58:22.506 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2021-02-10T11:58:22.507 DEBUG:teuthology.orchestra.run.gibba025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2021-02-10T11:58:22.906 INFO:teuthology.orchestra.run.gibba025.stdout:{ 2021-02-10T11:58:22.906 INFO:teuthology.orchestra.run.gibba025.stdout: "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", 2021-02-10T11:58:22.906 INFO:teuthology.orchestra.run.gibba025.stdout: "whoami": 0, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "id": 9775, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "want_state": "up:active", 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "state": "up:active", 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "rank_uptime": 18.489487526000001, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "mdsmap_epoch": 170, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch": 196, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "osdmap_epoch_barrier": 196, 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout: "uptime": 24.135160602999999 2021-02-10T11:58:22.907 INFO:teuthology.orchestra.run.gibba025.stdout:} 2021-02-10T11:58:22.910 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "722ef60b-feda-4d6a-867f-a5d7f666055b", "whoami": 0, "id": 9775, "want_state": "up:active", "state": "up:active", "rank_uptime": 18.489487526000001, "mdsmap_epoch": 170, "osdmap_epoch": 196, "osdmap_epoch_barrier": 196, "uptime": 24.135160602999999 } 2021-02-10T11:58:22.910 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2021-02-10T11:58:22.910 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2021-02-10T11:58:22.911 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@gibba017.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2021-02-10T11:58:22.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:22.911 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec mkdir -p /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:58:23.246 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:58:23.591 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:58:23.591 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:58:23.936 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:58:23.936 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:58:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:58:23.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:23.937 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:58:24.272 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:58:24.273 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2021-02-10T11:58:24.273 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:58:24.275 DEBUG:teuthology.orchestra.run.gibba017:> sudo modprobe fuse 2021-02-10T11:58:24.620 INFO:teuthology.orchestra.run:Running command with timeout 30 2021-02-10T11:58:24.620 DEBUG:teuthology.orchestra.run.gibba017:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2021-02-10T11:58:24.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:2021-02-10 11:58:24.687 7f707ad5af80 -1 init, newargv = 0x55aa29a12770 newargc=9 2021-02-10T11:58:24.689 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[197615]: starting ceph client 2021-02-10T11:58:24.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[197615]: starting fuse 2021-02-10T11:58:24.963 INFO:teuthology.orchestra.run.gibba017.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2021-02-10T11:58:24.963 INFO:teuthology.orchestra.run.gibba017.stderr: /sys/fs/fuse/connections is already mounted on /sys/fs/fuse/connections 2021-02-10T11:58:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 32 2021-02-10T11:58:24.964 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:24.964 DEBUG:teuthology.orchestra.run.gibba017:> ls /sys/fs/fuse/connections 2021-02-10T11:58:25.300 INFO:teuthology.orchestra.run.gibba017.stdout:42 2021-02-10T11:58:25.301 INFO:teuthology.orchestra.run.gibba017.stdout:43 2021-02-10T11:58:25.301 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2021-02-10T11:58:25.301 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:25.301 DEBUG:teuthology.orchestra.run.gibba017:> sudo python3 -c ' 2021-02-10T11:58:25.301 DEBUG:teuthology.orchestra.run.gibba017:> import glob 2021-02-10T11:58:25.301 DEBUG:teuthology.orchestra.run.gibba017:> import re 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> import os 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> import subprocess 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> def find_socket(client_name): 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> files = glob.glob(asok_path) 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> # Given a non-glob path, it better be there 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> if "*" not in asok_path: 2021-02-10T11:58:25.302 DEBUG:teuthology.orchestra.run.gibba017:> assert(len(files) == 1) 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> return files[0] 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> for f in files: 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> if os.path.exists("/proc/{0}".format(pid)): 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> return f 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> print(find_socket("client.1")) 2021-02-10T11:58:25.303 DEBUG:teuthology.orchestra.run.gibba017:> ' 2021-02-10T11:58:25.690 INFO:teuthology.orchestra.run.gibba017.stdout:/var/run/ceph/ceph-client.1.197615.asok 2021-02-10T11:58:25.694 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.197615.asok 2021-02-10T11:58:25.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:25.695 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.197615.asok status 2021-02-10T11:58:26.090 INFO:teuthology.orchestra.run.gibba017.stdout:{ 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "metadata": { 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_sha1": "962fac13b1a781b8c7d25c1eed50d93813b8f5f0", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "ceph_version": "ceph version 14.2.16-280-g962fac1 (962fac13b1a781b8c7d25c1eed50d93813b8f5f0) nautilus (stable)", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "entity_id": "1", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "hostname": "gibba017", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "pid": "197615", 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "root": "/" 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_count": 0, 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "dentry_pinned_count": 0, 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "id": 14001, 2021-02-10T11:58:26.091 INFO:teuthology.orchestra.run.gibba017.stdout: "inst": { 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "name": { 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "client", 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "num": 14001 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 366328246 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": { 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "type": "v1", 2021-02-10T11:58:26.092 INFO:teuthology.orchestra.run.gibba017.stdout: "addr": "172.21.2.117:0", 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "nonce": 366328246 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: }, 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "inst_str": "client.14001 172.21.2.117:0/366328246", 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "addr_str": "172.21.2.117:0/366328246", 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "inode_count": 1, 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "mds_epoch": 170, 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch": 196, 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "osd_epoch_barrier": 0, 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout: "blacklisted": false 2021-02-10T11:58:26.093 INFO:teuthology.orchestra.run.gibba017.stdout:} 2021-02-10T11:58:26.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:26.095 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:26.095 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:58:26.434 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:26.434 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:58:26.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:26.434 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:58:26.783 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2021-02-10T11:58:27.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 Processor -- start 2021-02-10T11:58:27.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 -- start start 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b40e4e40 0x7f37b41b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40d1db0 con 0x7f37b4085450 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b4085930 con 0x7f37b40e5240 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37ba52f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40e5cc0 con 0x7f37b40e4e40 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37b8b4c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f37b4085450 0x7f37b40d05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33326/0 (socket says 172.21.2.117:33326) 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.198 7f37b8b4c700 1 -- 172.21.2.117:0/3322724997 learned_addr learned my addr 172.21.2.117:0/3322724997 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031014653 0 0) 0x7f37ac001500 con 0x7f37b4085450 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c002fe0 con 0x7f37b4085450 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1814491214 0 0) 0x7f37ac001270 con 0x7f37b4085450 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b40e4e40 msgr2=0x7f37b41b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b40e4e40 0x7f37b41b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 >> v1:172.21.2.125:6789/0 conn(0x7f37b40e5240 legacy=0x7f37b4084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:27.218 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4084840 con 0x7f37b4085450 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f37ac001960 con 0x7f37b4085450 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 >> v1:172.21.2.117:6789/0 conn(0x7f37b4085450 legacy=0x7f37b40d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f379c003a60 0x7f379c005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f379c006780 0x7f379c008ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f379c0095a0 0x7f379c00ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c00c2c0 con 0x7f379c006780 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c00c4e0 con 0x7f379c003a60 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c00c700 con 0x7f379c0095a0 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f379c006780 0x7f379c008ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 -- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f379c0095a0 msgr2=0x7f379c00ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f379c0095a0 0x7f379c00ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 -- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f379c003a60 msgr2=0x7f379c005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:27.219 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f379c003a60 0x7f379c005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 -- 172.21.2.117:0/3322724997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c00c9a0 con 0x7f379c006780 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b37fe700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f379c006780 0x7f379c008ca0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f37a8006ab0 tx=0x7f37a804fbc0).ready entity=mon.0 client_cookie=306cf8b2c2c431b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37a8006030 con 0x7f379c006780 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a80063b0 con 0x7f379c006780 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b2ffd700 1 -- 172.21.2.117:0/3322724997 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37a8005690 con 0x7f379c006780 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37ba52f700 1 -- 172.21.2.117:0/3322724997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f379c006780 msgr2=0x7f379c008ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37ba52f700 1 --2- 172.21.2.117:0/3322724997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f379c006780 0x7f379c008ca0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 rx=0x7f37a8006ab0 tx=0x7f37a804fbc0).stop 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.199 7f37b8b4c700 1 -- 172.21.2.117:0/3322724997 reap_dead start 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.200 7f37ba52f700 1 -- 172.21.2.117:0/3322724997 shutdown_connections 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.200 7f37ba52f700 1 -- 172.21.2.117:0/3322724997 >> 172.21.2.117:0/3322724997 conn(0x7f37b40ce4e0 msgr2=0x7f37b40e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.200 7f37ba52f700 1 -- 172.21.2.117:0/3322724997 shutdown_connections 2021-02-10T11:58:27.220 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.200 7f37ba52f700 1 -- 172.21.2.117:0/3322724997 wait complete. 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 Processor -- start 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 -- start start 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 0x7f37b410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37b408c290 0x7f37b41105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b408c8a0 con 0x7f37b4110be0 2021-02-10T11:58:27.222 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b41a4570 con 0x7f37b408c290 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37ba52f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b41a4790 con 0x7f37b41a4170 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37b8b4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37b8b4c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33090/0 (socket says 172.21.2.117:33090) 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.215 7f37b8b4c700 1 -- 172.21.2.117:0/885762215 learned_addr learned my addr 172.21.2.117:0/885762215 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b37fe700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37b408c290 0x7f37b41105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b3fff700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 0x7f37b410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 -- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 msgr2=0x7f37b410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 0x7f37b410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 -- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37b408c290 msgr2=0x7f37b41105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37b408c290 0x7f37b41105c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 -- 172.21.2.117:0/885762215 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40af350 con 0x7f37b4110be0 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b3fff700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 0x7f37b410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:58:27.223 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b8b4c700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 rx=0x7f37ac00a940 tx=0x7f37ac00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37ac007d00 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac004930 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37ba52f700 1 -- 172.21.2.117:0/885762215 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b40af350 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37ba52f700 1 -- 172.21.2.117:0/885762215 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b40af8e0 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.216 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f37ac004d00 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.217 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f37ac00ae20 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.217 7f37b0ff9700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 0x7f379401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.217 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f37ac032b10 con 0x7f37b4110be0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.217 7f37b3fff700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 0x7f379401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:27.224 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.218 7f37b3fff700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 0x7f379401abc0 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:27.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.248 7f37ba52f700 1 -- 172.21.2.117:0/885762215 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b410c2a0 con 0x7f37b4110be0 2021-02-10T11:58:27.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.250 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f37ac001270 con 0x7f37b4110be0 2021-02-10T11:58:27.391 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:27.388 7f37ba52f700 1 -- 172.21.2.117:0/885762215 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f37b4085930 con 0x7f37b4110be0 2021-02-10T11:58:28.398 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.395 7f37b0ff9700 1 -- 172.21.2.117:0/885762215 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v171) v1 ==== 136+0+0 (secure 0 0 0) 0x7f37ac004930 con 0x7f37b4110be0 2021-02-10T11:58:28.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.409 7f37ba52f700 1 -- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 msgr2=0x7f379401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:28.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.409 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 0x7f379401abc0 crc :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.412 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.409 7f37ba52f700 1 -- 172.21.2.117:0/885762215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 msgr2=0x7f37b40aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.409 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 rx=0x7f37ac00a940 tx=0x7f37ac00ace0).stop 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 -- 172.21.2.117:0/885762215 shutdown_connections 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f37940186c0 0x7f379401abc0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f37b4110be0 0x7f37b40aed30 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f37b408c290 0x7f37b41105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 --2- 172.21.2.117:0/885762215 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f37b41a4170 0x7f37b410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 -- 172.21.2.117:0/885762215 >> 172.21.2.117:0/885762215 conn(0x7f37b40a5e90 msgr2=0x7f37b40b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:28.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 -- 172.21.2.117:0/885762215 shutdown_connections 2021-02-10T11:58:28.414 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.410 7f37ba52f700 1 -- 172.21.2.117:0/885762215 wait complete. 2021-02-10T11:58:28.421 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs volume ls 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 Processor -- start 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 -- start start 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 0x7f5ce40e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 0x7f5ce40e6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40e77f0 con 0x7f5ce40e6b90 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40e5ce0 con 0x7f5ce41b07f0 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce92c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40e54b0 con 0x7f5ce40e6170 2021-02-10T11:58:28.851 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.831 7f5ce387e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f5ce40e6b90 0x7f5ce40e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33344/0 (socket says 172.21.2.117:33344) 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce387e700 1 -- 172.21.2.117:0/1285431125 learned_addr learned my addr 172.21.2.117:0/1285431125 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 --2- 172.21.2.117:0/1285431125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 0x7f5ce40e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce307d700 1 --2- 172.21.2.117:0/1285431125 >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 0x7f5ce40e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce207b700 1 -- 172.21.2.117:0/1285431125 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394135477 0 0) 0x7f5cdc001500 con 0x7f5ce40e6b90 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce207b700 1 -- 172.21.2.117:0/1285431125 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd0002fe0 con 0x7f5ce40e6b90 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 -- 172.21.2.117:0/1285431125 >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 msgr2=0x7f5ce40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 --2- 172.21.2.117:0/1285431125 >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 0x7f5ce40e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 -- 172.21.2.117:0/1285431125 >> v1:172.21.2.117:6789/0 conn(0x7f5ce40e6b90 legacy=0x7f5ce40e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 -- 172.21.2.117:0/1285431125 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40e8d80 con 0x7f5ce40e6170 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce307d700 1 --2- 172.21.2.117:0/1285431125 >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 0x7f5ce40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce287c700 1 --2- 172.21.2.117:0/1285431125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 0x7f5ce40e6570 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5cd4006650 tx=0x7f5cd4009a80).ready entity=mon.2 client_cookie=e76a5bc751b91c08 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce207b700 1 -- 172.21.2.117:0/1285431125 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c0c0 con 0x7f5ce40e6170 2021-02-10T11:58:28.852 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce207b700 1 -- 172.21.2.117:0/1285431125 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd400c300 con 0x7f5ce40e6170 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.832 7f5ce207b700 1 -- 172.21.2.117:0/1285431125 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cd400c820 con 0x7f5ce40e6170 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 -- 172.21.2.117:0/1285431125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 msgr2=0x7f5ce40e6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 --2- 172.21.2.117:0/1285431125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 0x7f5ce40e6570 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5cd4006650 tx=0x7f5cd4009a80).stop 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 -- 172.21.2.117:0/1285431125 shutdown_connections 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 --2- 172.21.2.117:0/1285431125 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40e6170 0x7f5ce40e6570 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 --2- 172.21.2.117:0/1285431125 >> v2:172.21.2.125:3300/0 conn(0x7f5ce41b07f0 0x7f5ce40e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 -- 172.21.2.117:0/1285431125 >> 172.21.2.117:0/1285431125 conn(0x7f5ce40ce500 msgr2=0x7f5ce40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 -- 172.21.2.117:0/1285431125 shutdown_connections 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.833 7f5ce92c8700 1 -- 172.21.2.117:0/1285431125 wait complete. 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 Processor -- start 2021-02-10T11:58:28.853 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 -- start start 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce418f200 0x7f5ce40bb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40935b0 0x7f5ce40939b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40a4f50 con 0x7f5ce41a6d60 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4094060 con 0x7f5ce418f200 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce92c8700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce41a7780 con 0x7f5ce40935b0 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce387e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33100/0 (socket says 172.21.2.117:33100) 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.848 7f5ce387e700 1 -- 172.21.2.117:0/3360170103 learned_addr learned my addr 172.21.2.117:0/3360170103 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:28.855 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 -- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40935b0 msgr2=0x7f5ce40939b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40935b0 0x7f5ce40939b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 -- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce418f200 msgr2=0x7f5ce40bb860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce418f200 0x7f5ce40bb860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 -- 172.21.2.117:0/3360170103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce41a7960 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce387e700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f5cdc00a940 tx=0x7f5cdc00ace0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cdc007020 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc007220 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f5cdc004720 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce41a7960 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.849 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40a3d50 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.850 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f5cdc007400 con 0x7f5ce41a6d60 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.850 7f5cdbfff700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 0x7f5cc801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:28.856 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.850 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f5cdc032a30 con 0x7f5ce41a6d60 2021-02-10T11:58:28.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.850 7f5ce307d700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 0x7f5cc801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:28.857 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.851 7f5ce307d700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 0x7f5cc801abc0 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:28.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.881 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce410c290 con 0x7f5ce41a6d60 2021-02-10T11:58:28.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:28.884 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f5cdc007a90 con 0x7f5ce41a6d60 2021-02-10T11:58:29.020 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.017 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs volume ls", "target": ["mgr", ""]}) v1 -- 0x7f5ce40a3d50 con 0x7f5cc80186c0 2021-02-10T11:58:29.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.019 7f5cdbfff700 1 -- 172.21.2.117:0/3360170103 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+41 (crc 0 0 0) 0x7f5ccc000940 con 0x7f5cc80186c0 2021-02-10T11:58:29.023 INFO:teuthology.orchestra.run.gibba017.stdout:[ 2021-02-10T11:58:29.024 INFO:teuthology.orchestra.run.gibba017.stdout: { 2021-02-10T11:58:29.024 INFO:teuthology.orchestra.run.gibba017.stdout: "name": "cephfs" 2021-02-10T11:58:29.024 INFO:teuthology.orchestra.run.gibba017.stdout: } 2021-02-10T11:58:29.024 INFO:teuthology.orchestra.run.gibba017.stdout:] 2021-02-10T11:58:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 msgr2=0x7f5cc801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 0x7f5cc801abc0 crc :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 msgr2=0x7f5ce41a7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 rx=0x7f5cdc00a940 tx=0x7f5cdc00ace0).stop 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 shutdown_connections 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f5cc80186c0 0x7f5cc801abc0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f5ce41a6d60 0x7f5ce41a7160 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f5ce40935b0 0x7f5ce40939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 --2- 172.21.2.117:0/3360170103 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f5ce418f200 0x7f5ce40bb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 >> 172.21.2.117:0/3360170103 conn(0x7f5ce40a7260 msgr2=0x7f5ce40bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.022 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 shutdown_connections 2021-02-10T11:58:29.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.023 7f5ce92c8700 1 -- 172.21.2.117:0/3360170103 wait complete. 2021-02-10T11:58:29.034 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config set mon mon_allow_pool_delete True 2021-02-10T11:58:29.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 Processor -- start 2021-02-10T11:58:29.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 -- start start 2021-02-10T11:58:29.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fdcb80d18f0 0x7fdcb80eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.469 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb80ef7f0 con 0x7fdcb80eb040 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80ede60 con 0x7fdcb80ef3f0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcbde18700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb80d1cf0 con 0x7fdcb80d18f0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcb7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcb7fff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33108/0 (socket says 172.21.2.117:33108) 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.450 7fdcb7fff700 1 -- 172.21.2.117:0/2352839865 learned_addr learned my addr 172.21.2.117:0/2352839865 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb7fff700 1 -- 172.21.2.117:0/2352839865 >> v2:172.21.2.125:3300/0 conn(0x7fdcb80d18f0 msgr2=0x7fdcb80eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb7fff700 1 --2- 172.21.2.117:0/2352839865 >> v2:172.21.2.125:3300/0 conn(0x7fdcb80d18f0 0x7fdcb80eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb7fff700 1 -- 172.21.2.117:0/2352839865 >> v1:172.21.2.117:6789/0 conn(0x7fdcb80eb040 legacy=0x7fdcb80eb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb7fff700 1 -- 172.21.2.117:0/2352839865 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb81b8fd0 con 0x7fdcb80ef3f0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb7fff700 1 --2- 172.21.2.117:0/2352839865 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 rx=0x7fdcb0006f00 tx=0x7fdcb0009f10).ready entity=mon.0 client_cookie=a95ebccff003bf0c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb67fc700 1 -- 172.21.2.117:0/2352839865 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcb000b3b0 con 0x7fdcb80ef3f0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb67fc700 1 -- 172.21.2.117:0/2352839865 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb000b730 con 0x7fdcb80ef3f0 2021-02-10T11:58:29.470 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcb67fc700 1 -- 172.21.2.117:0/2352839865 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdcb000bc50 con 0x7fdcb80ef3f0 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcbde18700 1 -- 172.21.2.117:0/2352839865 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb80ef3f0 msgr2=0x7fdcb81b79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.451 7fdcbde18700 1 --2- 172.21.2.117:0/2352839865 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 rx=0x7fdcb0006f00 tx=0x7fdcb0009f10).stop 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 -- 172.21.2.117:0/2352839865 shutdown_connections 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 --2- 172.21.2.117:0/2352839865 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb80ef3f0 0x7fdcb81b79a0 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 --2- 172.21.2.117:0/2352839865 >> v2:172.21.2.125:3300/0 conn(0x7fdcb80d18f0 0x7fdcb80eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 -- 172.21.2.117:0/2352839865 >> 172.21.2.117:0/2352839865 conn(0x7fdcb80cf060 msgr2=0x7fdcb80cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 -- 172.21.2.117:0/2352839865 shutdown_connections 2021-02-10T11:58:29.471 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.452 7fdcbde18700 1 -- 172.21.2.117:0/2352839865 wait complete. 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.466 7fdcbde18700 1 Processor -- start 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 -- start start 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 0x7fdcb80a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 0x7fdcb80a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb807cca0 con 0x7fdcb807c7c0 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb818fb70 con 0x7fdcb80a13a0 2021-02-10T11:58:29.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcbde18700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80dc6d0 con 0x7fdcb81a3930 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcb77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcb77fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33118/0 (socket says 172.21.2.117:33118) 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.467 7fdcb77fe700 1 -- 172.21.2.117:0/2147113249 learned_addr learned my addr 172.21.2.117:0/2147113249 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb7fff700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 0x7fdcb80a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb6ffd700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 0x7fdcb80a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 -- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 msgr2=0x7fdcb80a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 0x7fdcb80a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 -- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 msgr2=0x7fdcb80a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 0x7fdcb80a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 -- 172.21.2.117:0/2147113249 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb807d220 con 0x7fdcb807c7c0 2021-02-10T11:58:29.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb7fff700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 0x7fdcb80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb6ffd700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 0x7fdcb80a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcb77fe700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 rx=0x7fdca0006f00 tx=0x7fdca000b940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdca0006b90 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca000bd30 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb807d220 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb807d7b0 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.468 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fdca00020d0 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.469 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fdca0007000 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.469 7fdcaffff700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 0x7fdc9c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.469 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fdca00326a0 con 0x7fdcb807c7c0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.469 7fdcb6ffd700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 0x7fdc9c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:29.475 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.470 7fdcb6ffd700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 0x7fdc9c01abc0 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:29.503 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.500 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb818fb00 con 0x7fdcb807c7c0 2021-02-10T11:58:29.506 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.503 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fdca0000940 con 0x7fdcb807c7c0 2021-02-10T11:58:29.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.652 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdcb818fb00 con 0x7fdcb807c7c0 2021-02-10T11:58:29.655 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.652 7fdcaffff700 1 -- 172.21.2.117:0/2147113249 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdca0007000 con 0x7fdcb807c7c0 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 msgr2=0x7fdc9c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 0x7fdc9c01abc0 crc :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 msgr2=0x7fdcb81a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 secure :-1 s=READY pgs=3285 cs=0 l=1 rev1=1 rx=0x7fdca0006f00 tx=0x7fdca000b940).stop 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 shutdown_connections 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fdcb80a13a0 0x7fdcb80a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fdcb81a3930 0x7fdcb80a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.658 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fdc9c0186c0 0x7fdc9c01abc0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.654 7fdcbde18700 1 --2- 172.21.2.117:0/2147113249 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fdcb807c7c0 0x7fdcb81a3310 unknown :-1 s=CLOSED pgs=3285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:29.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.655 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 >> 172.21.2.117:0/2147113249 conn(0x7fdcb8087090 msgr2=0x7fdcb808d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:29.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.655 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 shutdown_connections 2021-02-10T11:58:29.659 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:29.655 7fdcbde18700 1 -- 172.21.2.117:0/2147113249 wait complete. 2021-02-10T11:58:29.666 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume create cephfs subvolume_0000000000815429 2021-02-10T11:58:30.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 Processor -- start 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 -- start start 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa6800eb040 0x7fa6800eb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800ede60 con 0x7fa6800ee3e0 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6800eba60 con 0x7fa6800eb040 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.079 7fa68712f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800d1cf0 con 0x7fa6800d18f0 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa684f4b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa6800eb040 0x7fa6800eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa68574c700 1 --1- >> v1:172.21.2.125:6789/0 conn(0x7fa6800ee3e0 0x7fa6800ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.125:6789/0 says I am v1:172.21.2.117:54532/0 (socket says 172.21.2.117:54532) 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa684f4b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:59034/0 (socket says 172.21.2.117:59034) 2021-02-10T11:58:30.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa68574c700 1 -- 172.21.2.117:0/2335972020 learned_addr learned my addr 172.21.2.117:0/2335972020 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 -- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 msgr2=0x7fa6800eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67f7fe700 1 -- 172.21.2.117:0/2335972020 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199907263 0 0) 0x7fa678001440 con 0x7fa6800ee3e0 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 -- 172.21.2.117:0/2335972020 >> v1:172.21.2.125:6789/0 conn(0x7fa6800ee3e0 legacy=0x7fa6800ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 -- 172.21.2.117:0/2335972020 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800eefd0 con 0x7fa6800eb040 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa684f4b700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.080 7fa67ffff700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800eb040 0x7fa6800eb440 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fa670006650 tx=0x7fa670009a80).ready entity=mon.1 client_cookie=2467b8d799925d6f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa67f7fe700 1 -- 172.21.2.117:0/2335972020 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa67000c0c0 con 0x7fa6800eb040 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa67f7fe700 1 -- 172.21.2.117:0/2335972020 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67000c340 con 0x7fa6800eb040 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa67f7fe700 1 -- 172.21.2.117:0/2335972020 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa67000c860 con 0x7fa6800eb040 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 -- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800eb040 msgr2=0x7fa6800eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.100 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800eb040 0x7fa6800eb440 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fa670006650 tx=0x7fa670009a80).stop 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 -- 172.21.2.117:0/2335972020 shutdown_connections 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800eb040 0x7fa6800eb440 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 --2- 172.21.2.117:0/2335972020 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa6800d18f0 0x7fa6800eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 -- 172.21.2.117:0/2335972020 >> 172.21.2.117:0/2335972020 conn(0x7fa6800cf060 msgr2=0x7fa6800cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 -- 172.21.2.117:0/2335972020 shutdown_connections 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.081 7fa68712f700 1 -- 172.21.2.117:0/2335972020 wait complete. 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 Processor -- start 2021-02-10T11:58:30.101 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 -- start start 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa680085ec0 0x7fa6800862c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800a2040 0x7fa6800a2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800a67e0 con 0x7fa6800a6ce0 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800a7120 con 0x7fa6800a2040 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa68712f700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6800a2a60 con 0x7fa680085ec0 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33132/0 (socket says 172.21.2.117:33132) 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 -- 172.21.2.117:0/3792112577 learned_addr learned my addr 172.21.2.117:0/3792112577 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:30.103 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 -- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa680085ec0 msgr2=0x7fa6800862c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa680085ec0 0x7fa6800862c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 -- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800a2040 msgr2=0x7fa6800a2440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800a2040 0x7fa6800a2440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.096 7fa684f4b700 1 -- 172.21.2.117:0/3792112577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6801a2dc0 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa684f4b700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 rx=0x7fa668006f40 tx=0x7fa66800b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa66800bd80 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668000c90 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa668002220 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa68712f700 1 -- 172.21.2.117:0/3792112577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801a2dc0 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa68712f700 1 -- 172.21.2.117:0/3792112577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801a2dc0 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa66800a000 con 0x7fa6800a6ce0 2021-02-10T11:58:30.104 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 0x7fa66401abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.097 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa668032780 con 0x7fa6800a6ce0 2021-02-10T11:58:30.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.098 7fa67ffff700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 0x7fa66401abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.105 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.098 7fa67ffff700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 0x7fa66401abd0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.132 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.129 7fa68712f700 1 -- 172.21.2.117:0/3792112577 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800daff0 con 0x7fa6800a6ce0 2021-02-10T11:58:30.134 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.131 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7fa668000c90 con 0x7fa6800a6ce0 2021-02-10T11:58:30.259 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.256 7fa68712f700 1 -- 172.21.2.117:0/3792112577 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000815429", "target": ["mgr", ""]}) v1 -- 0x7fa6800daff0 con 0x7fa6640186d0 2021-02-10T11:58:30.263 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:58:30.259 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:58:30.263 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:58:30.259 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:58:30.263 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:58:30.259 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:58:30.263 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:58:30.259 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:58:30.263 INFO:tasks.ceph.mgr.x.gibba025.stderr:2021-02-10 11:58:30.259 7f59c98e1700 -1 client.0 error registering admin socket command: (17) File exists 2021-02-10T11:58:30.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.287 7fa67d7fa700 1 -- 172.21.2.117:0/3792112577 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7fa670000d10 con 0x7fa6640186d0 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.289 7fa68712f700 1 -- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 msgr2=0x7fa66401abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 0x7fa66401abd0 crc :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 -- 172.21.2.117:0/3792112577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 msgr2=0x7fa6800dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 rx=0x7fa668006f40 tx=0x7fa66800b990).stop 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 -- 172.21.2.117:0/3792112577 shutdown_connections 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa6800a2040 0x7fa6800a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa680085ec0 0x7fa6800862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa6640186d0 0x7fa66401abd0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.293 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 --2- 172.21.2.117:0/3792112577 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa6800a6ce0 0x7fa6800dc420 unknown :-1 s=CLOSED pgs=3286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 -- 172.21.2.117:0/3792112577 >> 172.21.2.117:0/3792112577 conn(0x7fa6800b6540 msgr2=0x7fa68008d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:30.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 -- 172.21.2.117:0/3792112577 shutdown_connections 2021-02-10T11:58:30.294 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.290 7fa68712f700 1 -- 172.21.2.117:0/3792112577 wait complete. 2021-02-10T11:58:30.301 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs subvolume_0000000000815429 2021-02-10T11:58:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.715 7f2ca9fab700 1 Processor -- start 2021-02-10T11:58:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.715 7f2ca9fab700 1 -- start start 2021-02-10T11:58:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca9fab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca40ef3f0 0x7f2ca40eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca9fab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca40eb830 0x7f2ca41b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca9fab700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40ef7f0 con 0x7f2ca40d18f0 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca9fab700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca40eded0 con 0x7f2ca40ef3f0 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca9fab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40d1cf0 con 0x7f2ca40eb830 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca40ef3f0 0x7f2ca40eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f2ca40ef3f0 0x7f2ca40eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33136/0 (socket says 172.21.2.117:33136) 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 -- 172.21.2.117:0/213683924 learned_addr learned my addr 172.21.2.117:0/213683924 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca3fff700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca40eb830 0x7f2ca41b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 -- 172.21.2.117:0/213683924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca40eb830 msgr2=0x7f2ca41b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca40eb830 0x7f2ca41b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 -- 172.21.2.117:0/213683924 >> v1:172.21.2.125:6789/0 conn(0x7f2ca40d18f0 legacy=0x7f2ca40eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 -- 172.21.2.117:0/213683924 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca41b8fd0 con 0x7f2ca40ef3f0 2021-02-10T11:58:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.716 7f2ca2ffd700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca40ef3f0 0x7f2ca40eb210 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 rx=0x7f2c98006700 tx=0x7f2c98009b80).ready entity=mon.0 client_cookie=3e465117fbd37c1d server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca27fc700 1 -- 172.21.2.117:0/213683924 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800c0c0 con 0x7f2ca40ef3f0 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca27fc700 1 -- 172.21.2.117:0/213683924 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9800c340 con 0x7f2ca40ef3f0 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca27fc700 1 -- 172.21.2.117:0/213683924 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800c860 con 0x7f2ca40ef3f0 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 -- 172.21.2.117:0/213683924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca40ef3f0 msgr2=0x7f2ca40eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca40ef3f0 0x7f2ca40eb210 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 rx=0x7f2c98006700 tx=0x7f2c98009b80).stop 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 -- 172.21.2.117:0/213683924 shutdown_connections 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca40eb830 0x7f2ca41b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 --2- 172.21.2.117:0/213683924 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca40ef3f0 0x7f2ca40eb210 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 -- 172.21.2.117:0/213683924 >> 172.21.2.117:0/213683924 conn(0x7f2ca40cf060 msgr2=0x7f2ca40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 -- 172.21.2.117:0/213683924 shutdown_connections 2021-02-10T11:58:30.736 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.717 7f2ca9fab700 1 -- 172.21.2.117:0/213683924 wait complete. 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 Processor -- start 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 -- start start 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 0x7f2ca408c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 0x7f2ca40aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40aae30 con 0x7f2ca408cf40 2021-02-10T11:58:30.738 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40aa9f0 con 0x7f2ca40a9f90 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca9fab700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40ab330 con 0x7f2ca41a48b0 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca2ffd700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33142/0 (socket says 172.21.2.117:33142) 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.732 7f2ca2ffd700 1 -- 172.21.2.117:0/3620369178 learned_addr learned my addr 172.21.2.117:0/3620369178 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 -- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 msgr2=0x7f2ca408c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca37fe700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 0x7f2ca408c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 0x7f2ca408c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 -- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 msgr2=0x7f2ca40aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca3fff700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 0x7f2ca40aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 0x7f2ca40aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 -- 172.21.2.117:0/3620369178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40ab550 con 0x7f2ca408cf40 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca37fe700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 0x7f2ca408c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca3fff700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 0x7f2ca40aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:58:30.739 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca2ffd700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f2c9800b650 tx=0x7f2c9800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c9800bde0 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98009260 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2c98002220 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca40ab550 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.733 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca40abae0 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.734 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f2c98000c90 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.734 7f2c8bfff700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 0x7f2c8401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.734 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f2c980326b0 con 0x7f2ca408cf40 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.734 7f2ca37fe700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 0x7f2c8401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:58:30.740 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.735 7f2ca37fe700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 0x7f2c8401abc0 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:58:30.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.766 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca40ede60 con 0x7f2ca408cf40 2021-02-10T11:58:30.771 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.768 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f2c98000940 con 0x7f2ca408cf40 2021-02-10T11:58:30.900 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.897 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvolume_0000000000815429", "target": ["mgr", ""]}) v1 -- 0x7f2ca40abae0 con 0x7f2c840186c0 2021-02-10T11:58:30.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.900 7f2c8bfff700 1 -- 172.21.2.117:0/3620369178 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+82 (crc 0 0 0) 0x7f2c94000960 con 0x7f2c840186c0 2021-02-10T11:58:30.903 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5 2021-02-10T11:58:30.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 msgr2=0x7f2c8401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 0x7f2c8401abc0 crc :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 msgr2=0x7f2ca40a9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:58:30.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 rx=0x7f2c9800b650 tx=0x7f2c9800b9f0).stop 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 shutdown_connections 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f2c840186c0 0x7f2c8401abc0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f2ca408cf40 0x7f2ca40a9970 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f2ca40a9f90 0x7f2ca40aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 --2- 172.21.2.117:0/3620369178 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f2ca41a48b0 0x7f2ca408c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.901 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 >> 172.21.2.117:0/3620369178 conn(0x7f2ca4092d00 msgr2=0x7f2ca408d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.902 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 shutdown_connections 2021-02-10T11:58:30.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:58:30.902 7f2ca9fab700 1 -- 172.21.2.117:0/3620369178 wait complete. 2021-02-10T11:58:30.913 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvolume_0000000000815429 with 64 files each 1MB size under directory volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5 2021-02-10T11:58:30.913 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:30.913 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:30.913 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:31.249 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:31.249 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:31.249 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.0 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:31.636 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:31.636 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:31.636 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0286698 s, 36.6 MB/s 2021-02-10T11:58:31.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:31.638 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:31.638 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:31.974 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:31.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:31.974 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.1 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:32.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:32.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:32.351 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0267633 s, 39.2 MB/s 2021-02-10T11:58:32.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:32.352 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:32.352 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:32.689 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:32.690 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:32.690 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.2 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:33.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:33.068 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:33.068 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253413 s, 41.4 MB/s 2021-02-10T11:58:33.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:33.069 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:33.069 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:33.406 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:33.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:33.406 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.3 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:33.783 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0257425 s, 40.7 MB/s 2021-02-10T11:58:33.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:33.785 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:33.785 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:34.120 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:34.120 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:34.121 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.4 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:34.496 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:34.496 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:34.496 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240788 s, 43.5 MB/s 2021-02-10T11:58:34.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:34.497 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:34.497 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:34.834 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:34.834 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:34.835 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.5 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:35.210 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:35.210 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:35.210 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240325 s, 43.6 MB/s 2021-02-10T11:58:35.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:35.211 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:35.211 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:35.548 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:35.548 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:35.549 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.6 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:35.923 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:35.924 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:35.924 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229766 s, 45.6 MB/s 2021-02-10T11:58:35.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:35.925 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:35.925 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:36.261 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:36.261 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:36.262 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.7 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:36.637 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:36.638 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:36.638 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0256921 s, 40.8 MB/s 2021-02-10T11:58:36.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:36.639 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:36.639 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:36.975 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:36.975 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:36.975 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.8 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:37.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:37.351 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:37.351 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242347 s, 43.3 MB/s 2021-02-10T11:58:37.352 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:37.352 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:37.352 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:37.687 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:37.687 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:37.687 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.9 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:38.062 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:38.062 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:38.062 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239478 s, 43.8 MB/s 2021-02-10T11:58:38.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:38.063 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:38.063 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:38.399 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:38.399 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:38.400 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.10 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:38.775 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:38.776 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:38.776 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0241757 s, 43.4 MB/s 2021-02-10T11:58:38.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:38.777 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:38.777 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:39.112 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:39.113 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:39.113 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.11 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:39.487 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:39.487 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:39.487 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0250312 s, 41.9 MB/s 2021-02-10T11:58:39.488 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:39.488 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:39.488 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:39.825 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:39.825 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:39.825 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.12 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:40.200 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:40.200 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:40.200 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237293 s, 44.2 MB/s 2021-02-10T11:58:40.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:40.201 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:40.201 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:40.538 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:40.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:40.538 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.13 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:40.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:40.914 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:40.914 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244363 s, 42.9 MB/s 2021-02-10T11:58:40.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:40.915 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:40.915 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:41.251 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:41.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:41.252 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.14 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:41.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:41.628 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:41.628 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239185 s, 43.8 MB/s 2021-02-10T11:58:41.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:41.630 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:41.630 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:41.967 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:41.967 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:41.967 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.15 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:42.342 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:42.342 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:42.342 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0236146 s, 44.4 MB/s 2021-02-10T11:58:42.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:42.343 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:42.343 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:42.679 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:42.679 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:42.679 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.16 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:43.053 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:43.053 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:43.053 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0225647 s, 46.5 MB/s 2021-02-10T11:58:43.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:43.054 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:43.054 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:43.391 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:43.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:43.391 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.17 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:43.767 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:43.767 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:43.768 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024354 s, 43.1 MB/s 2021-02-10T11:58:43.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:43.769 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:43.769 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:44.105 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:44.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:44.106 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.18 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:44.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:44.478 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:44.478 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234184 s, 44.8 MB/s 2021-02-10T11:58:44.479 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:44.479 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:44.479 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:44.813 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:44.814 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:44.814 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.19 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:45.190 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:45.190 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:45.190 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243671 s, 43.0 MB/s 2021-02-10T11:58:45.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:45.191 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:45.191 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:45.527 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:45.527 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:45.527 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.20 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:45.903 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:45.903 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:45.903 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230016 s, 45.6 MB/s 2021-02-10T11:58:45.904 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:45.904 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:45.904 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:46.240 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:46.240 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:46.240 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.21 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:46.616 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:46.616 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:46.617 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240541 s, 43.6 MB/s 2021-02-10T11:58:46.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:46.618 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:46.618 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:46.954 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:46.954 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:46.954 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.22 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:47.328 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:47.329 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230718 s, 45.4 MB/s 2021-02-10T11:58:47.330 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:47.330 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:47.330 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:47.666 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:47.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:47.666 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.23 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:48.040 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:48.041 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0222282 s, 47.2 MB/s 2021-02-10T11:58:48.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:48.042 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:48.042 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:48.379 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:48.379 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:48.379 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.24 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:48.755 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:48.755 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:48.756 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0252332 s, 41.6 MB/s 2021-02-10T11:58:48.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:48.757 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:48.757 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:49.094 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:49.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:49.094 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.25 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:49.467 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:49.467 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:49.468 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234366 s, 44.7 MB/s 2021-02-10T11:58:49.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:49.469 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:49.469 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:49.804 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:49.805 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:49.805 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.26 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:50.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:50.181 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:50.181 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0245613 s, 42.7 MB/s 2021-02-10T11:58:50.182 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:50.182 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:50.182 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:50.517 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:50.518 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:50.518 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.27 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:50.847 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:58:50.850 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:58:50.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:50.895 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:50.895 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0251944 s, 41.6 MB/s 2021-02-10T11:58:50.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:50.896 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:50.896 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:51.188 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:51.344 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:51.344 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.28 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:51.760 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:51.760 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:51.760 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243067 s, 43.1 MB/s 2021-02-10T11:58:51.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:51.761 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:51.761 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:52.097 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:52.097 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:52.097 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.29 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:52.472 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:52.472 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:52.472 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234946 s, 44.6 MB/s 2021-02-10T11:58:52.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:52.473 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:52.473 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:52.809 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:52.809 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:52.809 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.30 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:53.184 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:53.184 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:53.184 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0230188 s, 45.6 MB/s 2021-02-10T11:58:53.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:53.185 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:53.185 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:53.521 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:53.522 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:53.522 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.31 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:53.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:53.894 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:53.894 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231668 s, 45.3 MB/s 2021-02-10T11:58:53.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:53.895 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:53.896 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:54.232 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:54.232 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:54.232 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.32 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:54.610 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249729 s, 42.0 MB/s 2021-02-10T11:58:54.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:54.611 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:54.611 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:54.951 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:54.951 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:54.951 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.33 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:55.327 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:55.327 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:55.327 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023963 s, 43.8 MB/s 2021-02-10T11:58:55.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:55.328 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:55.328 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:55.665 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:55.665 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:55.665 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.34 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:56.040 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:56.040 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:56.040 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0224148 s, 46.8 MB/s 2021-02-10T11:58:56.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:56.041 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:56.042 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:56.378 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:56.378 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:56.379 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.35 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:56.756 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:56.757 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0253778 s, 41.3 MB/s 2021-02-10T11:58:56.757 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:56.758 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:56.758 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:57.094 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:57.094 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:57.094 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.36 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:57.470 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:57.471 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:57.471 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024991 s, 42.0 MB/s 2021-02-10T11:58:57.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:57.472 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:57.472 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:57.808 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:57.808 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:57.809 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.37 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:58.183 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:58.183 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:58.183 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0238388 s, 44.0 MB/s 2021-02-10T11:58:58.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:58.184 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:58.184 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:58.520 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:58.520 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:58.520 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.38 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:58.893 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:58.893 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:58.893 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0240843 s, 43.5 MB/s 2021-02-10T11:58:58.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:58.894 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:58.894 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:59.231 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:59.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:59.232 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.39 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:58:59.607 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:58:59.608 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:58:59.608 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234621 s, 44.7 MB/s 2021-02-10T11:58:59.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:58:59.609 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:58:59.609 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:58:59.945 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:58:59.946 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:58:59.946 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.40 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:00.321 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:00.321 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:00.321 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.02349 s, 44.6 MB/s 2021-02-10T11:59:00.323 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:00.323 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:00.323 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:00.658 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:00.658 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:00.659 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.41 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:01.033 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:01.033 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:01.033 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233107 s, 45.0 MB/s 2021-02-10T11:59:01.034 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:01.034 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:01.035 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:01.371 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:01.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:01.371 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:01.757 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:01.758 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0215863 s, 48.6 MB/s 2021-02-10T11:59:01.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:01.760 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:01.760 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:02.099 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:02.099 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:02.100 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:02.475 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:02.475 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:02.475 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235461 s, 44.5 MB/s 2021-02-10T11:59:02.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:02.476 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:02.476 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:02.813 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:02.814 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:02.814 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.44 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:03.189 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237086 s, 44.2 MB/s 2021-02-10T11:59:03.190 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:03.190 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:03.190 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:03.526 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:03.526 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:03.526 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.45 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:03.899 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:03.899 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:03.899 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0243637 s, 43.0 MB/s 2021-02-10T11:59:03.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:03.900 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:03.900 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:04.236 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:04.236 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:04.236 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.46 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:04.612 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:04.612 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:04.612 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235453 s, 44.5 MB/s 2021-02-10T11:59:04.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:04.613 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:04.613 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:04.950 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:04.950 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:04.950 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.47 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:05.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:05.325 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:05.325 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023863 s, 43.9 MB/s 2021-02-10T11:59:05.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:05.327 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:05.327 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:05.664 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:05.664 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:05.664 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.48 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:06.037 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:06.038 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:06.038 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0222776 s, 47.1 MB/s 2021-02-10T11:59:06.039 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:06.039 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:06.039 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:06.375 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:06.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:06.375 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.49 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:06.749 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0234888 s, 44.6 MB/s 2021-02-10T11:59:06.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:06.751 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:06.751 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:07.086 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:07.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:07.086 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.50 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:07.461 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:07.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:07.462 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0235528 s, 44.5 MB/s 2021-02-10T11:59:07.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:07.463 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:07.463 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:07.798 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:07.798 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:07.798 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.51 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:08.173 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:08.174 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:08.174 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.023807 s, 44.0 MB/s 2021-02-10T11:59:08.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:08.175 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:08.175 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:08.510 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:08.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:08.510 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.52 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:08.881 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:08.881 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:08.881 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0229896 s, 45.6 MB/s 2021-02-10T11:59:08.882 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:08.882 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:08.882 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:09.219 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:09.219 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:09.220 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.53 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:09.594 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:09.595 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233126 s, 45.0 MB/s 2021-02-10T11:59:09.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:09.596 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:09.596 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:09.932 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:09.932 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:09.932 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.54 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:10.308 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231409 s, 45.3 MB/s 2021-02-10T11:59:10.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:10.309 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:10.309 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:10.645 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:10.645 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:10.645 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.55 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:11.020 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:11.020 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:11.020 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231447 s, 45.3 MB/s 2021-02-10T11:59:11.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:11.021 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:11.021 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:11.358 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:11.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:11.359 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.56 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:11.732 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:11.732 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:11.732 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0231289 s, 45.3 MB/s 2021-02-10T11:59:11.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:11.734 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:11.734 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:12.084 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:12.084 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:12.084 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.57 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:12.461 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:12.462 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:12.462 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0249349 s, 42.1 MB/s 2021-02-10T11:59:12.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:12.463 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:12.463 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:12.800 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:12.800 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:12.800 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.58 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:13.177 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:13.178 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0242179 s, 43.3 MB/s 2021-02-10T11:59:13.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:13.179 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:13.179 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:13.514 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:13.514 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:13.514 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.59 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:13.887 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:13.887 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:13.887 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.024468 s, 42.9 MB/s 2021-02-10T11:59:13.888 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:13.888 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:13.888 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:14.224 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:14.225 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:14.225 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.60 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:14.601 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:14.601 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:14.601 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0233678 s, 44.9 MB/s 2021-02-10T11:59:14.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:14.602 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:14.602 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:14.938 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:14.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:14.939 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.61 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:15.314 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:15.315 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:15.315 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0237981 s, 44.1 MB/s 2021-02-10T11:59:15.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:15.316 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:15.316 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:15.652 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:15.652 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:15.652 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.62 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:16.028 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:16.028 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:16.028 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0244386 s, 42.9 MB/s 2021-02-10T11:59:16.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:16.029 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:16.029 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:16.365 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:16.366 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:16.366 DEBUG:teuthology.orchestra.run.gibba017:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000815429/1756dda8-e8a2-4d37-a2f4-fd91bc3855d5/subvolume_file.63 bs=1M conv=fdatasync count=1 seek=0 2021-02-10T11:59:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records in 2021-02-10T11:59:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:1+0 records out 2021-02-10T11:59:16.742 INFO:teuthology.orchestra.run.gibba017.stderr:1048576 bytes (1.0 MB) copied, 0.0239423 s, 43.8 MB/s 2021-02-10T11:59:16.743 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot create cephfs subvolume_0000000000815429 snapshot_0000000000565169 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 Processor -- start 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 -- start start 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f99400e4e40 0x7f99401b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99400d1db0 con 0x7f9940085450 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9940085930 con 0x7f99400e5240 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.157 7f994664b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9940084840 con 0x7f99400e4e40 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f9944c68700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f9940085450 0x7f99400d05b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33410/0 (socket says 172.21.2.117:33410) 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f9944c68700 1 -- 172.21.2.117:0/124121530 learned_addr learned my addr 172.21.2.117:0/124121530 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993ffff700 1 --2- 172.21.2.117:0/124121530 >> v2:172.21.2.125:3300/0 conn(0x7f99400e4e40 0x7f99401b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.177 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993effd700 1 -- 172.21.2.117:0/124121530 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574184209 0 0) 0x7f9938001500 con 0x7f9940085450 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993effd700 1 -- 172.21.2.117:0/124121530 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928002fe0 con 0x7f9940085450 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993effd700 1 -- 172.21.2.117:0/124121530 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799634866 0 0) 0x7f9934000ff0 con 0x7f99400e5240 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993ffff700 1 -- 172.21.2.117:0/124121530 >> v1:172.21.2.125:6789/0 conn(0x7f99400e5240 legacy=0x7f9940084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993ffff700 1 -- 172.21.2.117:0/124121530 >> v1:172.21.2.117:6789/0 conn(0x7f9940085450 legacy=0x7f99400d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993ffff700 1 -- 172.21.2.117:0/124121530 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99401b0810 con 0x7f99400e4e40 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.158 7f993ffff700 1 --2- 172.21.2.117:0/124121530 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400e4e40 0x7f99401b01f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f9930006650 tx=0x7f9930009a80).ready entity=mon.1 client_cookie=4443743fbffd6579 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f993effd700 1 -- 172.21.2.117:0/124121530 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993000c0c0 con 0x7f99400e4e40 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f993effd700 1 -- 172.21.2.117:0/124121530 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993000c340 con 0x7f99400e4e40 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f993effd700 1 -- 172.21.2.117:0/124121530 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f993000c860 con 0x7f99400e4e40 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 -- 172.21.2.117:0/124121530 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400e4e40 msgr2=0x7f99401b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 --2- 172.21.2.117:0/124121530 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400e4e40 0x7f99401b01f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f9930006650 tx=0x7f9930009a80).stop 2021-02-10T11:59:17.178 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 -- 172.21.2.117:0/124121530 shutdown_connections 2021-02-10T11:59:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 --2- 172.21.2.117:0/124121530 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400e4e40 0x7f99401b01f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 -- 172.21.2.117:0/124121530 >> 172.21.2.117:0/124121530 conn(0x7f99400ce4e0 msgr2=0x7f99400e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 -- 172.21.2.117:0/124121530 shutdown_connections 2021-02-10T11:59:17.179 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.159 7f994664b700 1 -- 172.21.2.117:0/124121530 wait complete. 2021-02-10T11:59:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.174 7f994664b700 1 Processor -- start 2021-02-10T11:59:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.174 7f994664b700 1 -- start start 2021-02-10T11:59:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f994019d160 0x7f99400a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.180 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400a1060 0x7f994019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994019d660 con 0x7f9940187cf0 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99401b2530 con 0x7f99400a1060 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994019e200 con 0x7f994019d160 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33166/0 (socket says 172.21.2.117:33166) 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 -- 172.21.2.117:0/4056519036 learned_addr learned my addr 172.21.2.117:0/4056519036 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 -- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f994019d160 msgr2=0x7f99400a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f994019d160 0x7f99400a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 -- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400a1060 msgr2=0x7f994019dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400a1060 0x7f994019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 -- 172.21.2.117:0/4056519036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994019e420 con 0x7f9940187cf0 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993ffff700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f9930009630 tx=0x7f993000b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.181 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99300093a0 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993000bd70 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f99300054d0 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 -- 172.21.2.117:0/4056519036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994019e420 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.175 7f994664b700 1 -- 172.21.2.117:0/4056519036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994019e9b0 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.176 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f993000a000 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.176 7f993cff9700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 0x7f992001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.176 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f99300326b0 con 0x7f9940187cf0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.176 7f993f7fe700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 0x7f992001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.177 7f993f7fe700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 0x7f992001abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.210 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.207 7f994664b700 1 -- 172.21.2.117:0/4056519036 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994010c2e0 con 0x7f9940187cf0 2021-02-10T11:59:17.213 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.210 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f9930030d10 con 0x7f9940187cf0 2021-02-10T11:59:17.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.336 7f994664b700 1 -- 172.21.2.117:0/4056519036 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "subvolume_0000000000815429", "target": ["mgr", ""], "snap_name": "snapshot_0000000000565169"}) v1 -- 0x7f9940085930 con 0x7f99200186c0 2021-02-10T11:59:17.352 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.350 7f993cff9700 1 -- 172.21.2.117:0/4056519036 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f9934007e00 con 0x7f99200186c0 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.353 7f994664b700 1 -- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 msgr2=0x7f992001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.353 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 0x7f992001abc0 crc :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.353 7f994664b700 1 -- 172.21.2.117:0/4056519036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 msgr2=0x7f99401b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.353 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f9930009630 tx=0x7f993000b980).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 -- 172.21.2.117:0/4056519036 shutdown_connections 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f99400a1060 0x7f994019dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f994019d160 0x7f99400a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f99200186c0 0x7f992001abc0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 --2- 172.21.2.117:0/4056519036 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f9940187cf0 0x7f99401b1f10 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:17.357 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 -- 172.21.2.117:0/4056519036 >> 172.21.2.117:0/4056519036 conn(0x7f994010def0 msgr2=0x7f994019a710 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:17.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 -- 172.21.2.117:0/4056519036 shutdown_connections 2021-02-10T11:59:17.358 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.354 7f994664b700 1 -- 172.21.2.117:0/4056519036 wait complete. 2021-02-10T11:59:17.365 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume snapshot clone cephfs subvolume_0000000000815429 snapshot_0000000000565169 clone_0000000000240022 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.777 7f15812cb700 1 Processor -- start 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.777 7f15812cb700 1 -- start start 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f15812cb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 0x7f157c0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f15812cb700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c0ede60 con 0x7f157c0ef3f0 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f15812cb700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c0d1ce0 con 0x7f157c0eb000 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f15812cb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c088020 con 0x7f157c0d18e0 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f157b87e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f157c0ef3f0 0x7f157c1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33422/0 (socket says 172.21.2.117:33422) 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f157b87e700 1 -- 172.21.2.117:0/4124330342 learned_addr learned my addr 172.21.2.117:0/4124330342 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:17.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f157b07d700 1 --2- 172.21.2.117:0/4124330342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 0x7f157c0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.778 7f157a07b700 1 -- 172.21.2.117:0/4124330342 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200209643 0 0) 0x7f1574001500 con 0x7f157c0ef3f0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1568002fe0 con 0x7f157c0ef3f0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229982999 0 0) 0x7f156c000ff0 con 0x7f157c0eb000 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1568003210 con 0x7f157c0eb000 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157b07d700 1 -- 172.21.2.117:0/4124330342 >> v1:172.21.2.125:6789/0 conn(0x7f157c0eb000 legacy=0x7f157c0eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157b07d700 1 -- 172.21.2.117:0/4124330342 >> v1:172.21.2.117:6789/0 conn(0x7f157c0ef3f0 legacy=0x7f157c1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157b07d700 1 -- 172.21.2.117:0/4124330342 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c1bb180 con 0x7f157c0d18e0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157b07d700 1 --2- 172.21.2.117:0/4124330342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 0x7f157c0ea9e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1564006650 tx=0x7f1564009a00).ready entity=mon.2 client_cookie=197a542af66eb137 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15640009b0 con 0x7f157c0d18e0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156400c0c0 con 0x7f157c0d18e0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f157a07b700 1 -- 172.21.2.117:0/4124330342 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f156400c5a0 con 0x7f157c0d18e0 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f15812cb700 1 -- 172.21.2.117:0/4124330342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 msgr2=0x7f157c0ea9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:17.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.779 7f15812cb700 1 --2- 172.21.2.117:0/4124330342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 0x7f157c0ea9e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1564006650 tx=0x7f1564009a00).stop 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.780 7f15812cb700 1 -- 172.21.2.117:0/4124330342 shutdown_connections 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.780 7f15812cb700 1 --2- 172.21.2.117:0/4124330342 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0d18e0 0x7f157c0ea9e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.780 7f15812cb700 1 -- 172.21.2.117:0/4124330342 >> 172.21.2.117:0/4124330342 conn(0x7f157c0ced60 msgr2=0x7f157c0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.780 7f15812cb700 1 -- 172.21.2.117:0/4124330342 shutdown_connections 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.780 7f15812cb700 1 -- 172.21.2.117:0/4124330342 wait complete. 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 Processor -- start 2021-02-10T11:59:17.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 -- start start 2021-02-10T11:59:17.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f157c0e3280 0x7f157c0e3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0e1560 0x7f157c0e1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c0e2360 con 0x7f157c08b010 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c0e3eb0 con 0x7f157c0e3280 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f15812cb700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c0e1f80 con 0x7f157c0e1560 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f157b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f157b87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33174/0 (socket says 172.21.2.117:33174) 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.795 7f157b87e700 1 -- 172.21.2.117:0/1848594745 learned_addr learned my addr 172.21.2.117:0/1848594745 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:17.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 -- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0e1560 msgr2=0x7f157c0e1960 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0e1560 0x7f157c0e1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 -- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f157c0e3280 msgr2=0x7f157c0e3680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f157c0e3280 0x7f157c0e3680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 -- 172.21.2.117:0/1848594745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c08ba30 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f157b87e700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 rx=0x7f1574001c10 tx=0x7f157400aca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f15740070d0 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15740072d0 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f15812cb700 1 -- 172.21.2.117:0/1848594745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c08ba30 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f15812cb700 1 -- 172.21.2.117:0/1848594745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c08bfc0 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.796 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1574004720 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.797 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f157400ade0 con 0x7f157c08b010 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.797 7f1573fff700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 0x7f156001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:17.803 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.797 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f15740328c0 con 0x7f157c08b010 2021-02-10T11:59:17.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.797 7f157b07d700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 0x7f156001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:17.804 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.797 7f157b07d700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 0x7f156001abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:17.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.828 7f15812cb700 1 -- 172.21.2.117:0/1848594745 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c07f0a0 con 0x7f157c08b010 2021-02-10T11:59:17.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.830 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1574000f40 con 0x7f157c08b010 2021-02-10T11:59:17.963 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:17.959 7f15812cb700 1 -- 172.21.2.117:0/1848594745 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "subvolume_0000000000815429", "target": ["mgr", ""], "snap_name": "snapshot_0000000000565169", "target_sub_name": "clone_0000000000240022"}) v1 -- 0x7f157c08bfc0 con 0x7f15600186c0 2021-02-10T11:59:18.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.002 7f1573fff700 1 -- 172.21.2.117:0/1848594745 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x7f15640009b0 con 0x7f15600186c0 2021-02-10T11:59:18.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 -- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 msgr2=0x7f156001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 0x7f156001abc0 crc :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 -- 172.21.2.117:0/1848594745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 msgr2=0x7f157c08b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 rx=0x7f1574001c10 tx=0x7f157400aca0).stop 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 -- 172.21.2.117:0/1848594745 shutdown_connections 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15600186c0 0x7f156001abc0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f157c08b010 0x7f157c08b410 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f157c0e1560 0x7f157c0e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 --2- 172.21.2.117:0/1848594745 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f157c0e3280 0x7f157c0e3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 -- 172.21.2.117:0/1848594745 >> 172.21.2.117:0/1848594745 conn(0x7f157c197410 msgr2=0x7f157c0e2700 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.025 7f15812cb700 1 -- 172.21.2.117:0/1848594745 shutdown_connections 2021-02-10T11:59:18.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.026 7f15812cb700 1 -- 172.21.2.117:0/1848594745 wait complete. 2021-02-10T11:59:18.037 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph fs subvolume getpath cephfs clone_0000000000240022 2021-02-10T11:59:18.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 Processor -- start 2021-02-10T11:59:18.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 -- start start 2021-02-10T11:59:18.472 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71b00ef3f0 0x7f71b00eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b00ef7f0 con 0x7f71b00d18f0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b00ede60 con 0x7f71b00eb830 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.452 7f71b4b73700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b00d1da0 con 0x7f71b00ef3f0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71a605d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71b00ef3f0 0x7f71b00eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71ae85e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f71b00d18f0 0x7f71b00eaa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33432/0 (socket says 172.21.2.117:33432) 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71ae85e700 1 -- 172.21.2.117:0/972074507 learned_addr learned my addr 172.21.2.117:0/972074507 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71a605d700 1 -- 172.21.2.117:0/972074507 >> v1:172.21.2.125:6789/0 conn(0x7f71b00eb830 legacy=0x7f71b01b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71a605d700 1 -- 172.21.2.117:0/972074507 >> v1:172.21.2.117:6789/0 conn(0x7f71b00d18f0 legacy=0x7f71b00eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71a605d700 1 -- 172.21.2.117:0/972074507 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b01bb1c0 con 0x7f71b00ef3f0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71a605d700 1 --2- 172.21.2.117:0/972074507 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b00ef3f0 0x7f71b00eb210 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7f7198006650 tx=0x7f7198009a80).ready entity=mon.0 client_cookie=28047eb7de2533e3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71ae05d700 1 -- 172.21.2.117:0/972074507 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719800c0c0 con 0x7f71b00ef3f0 2021-02-10T11:59:18.473 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71ae05d700 1 -- 172.21.2.117:0/972074507 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719800c340 con 0x7f71b00ef3f0 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71ae05d700 1 -- 172.21.2.117:0/972074507 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f719800c860 con 0x7f71b00ef3f0 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71b4b73700 1 -- 172.21.2.117:0/972074507 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b00ef3f0 msgr2=0x7f71b00eb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.453 7f71b4b73700 1 --2- 172.21.2.117:0/972074507 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b00ef3f0 0x7f71b00eb210 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 rx=0x7f7198006650 tx=0x7f7198009a80).stop 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.454 7f71b4b73700 1 -- 172.21.2.117:0/972074507 shutdown_connections 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.454 7f71b4b73700 1 --2- 172.21.2.117:0/972074507 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b00ef3f0 0x7f71b00eb210 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.454 7f71b4b73700 1 -- 172.21.2.117:0/972074507 >> 172.21.2.117:0/972074507 conn(0x7f71b00cf060 msgr2=0x7f71b00cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.454 7f71b4b73700 1 -- 172.21.2.117:0/972074507 shutdown_connections 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.454 7f71b4b73700 1 -- 172.21.2.117:0/972074507 wait complete. 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.469 7f71b4b73700 1 Processor -- start 2021-02-10T11:59:18.474 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.469 7f71b4b73700 1 -- start start 2021-02-10T11:59:18.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 0x7f71b00c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:18.476 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71b01c2c80 0x7f71b01c3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00e24a0 con 0x7f71b01c36a0 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00a4b50 con 0x7f71b00c3570 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00c3f90 con 0x7f71b01c2c80 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33190/0 (socket says 172.21.2.117:33190) 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 -- 172.21.2.117:0/1032592982 learned_addr learned my addr 172.21.2.117:0/1032592982 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71a605d700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71b01c2c80 0x7f71b01c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71ae85e700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 0x7f71b00c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 -- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71b01c2c80 msgr2=0x7f71b01c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71b01c2c80 0x7f71b01c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 -- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 msgr2=0x7f71b00c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.477 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 0x7f71b00c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 -- 172.21.2.117:0/1032592982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b00a47e0 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71ae85e700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 0x7f71b00c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71af05f700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7f71a800b8e0 tx=0x7f71a800bc80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71a8000d30 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a800c2f0 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b00a47e0 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71a800c750 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.470 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b00a56b0 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.471 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f71a8001220 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.471 7f71a7fff700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 0x7f719401ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.471 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f71a8032af0 con 0x7f71b01c36a0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.471 7f71ae85e700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 0x7f719401ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:18.478 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.471 7f71ae85e700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 0x7f719401ac10 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:18.505 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.502 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b0087390 con 0x7f71b01c36a0 2021-02-10T11:59:18.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.504 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f71a8000cc0 con 0x7f71b01c36a0 2021-02-10T11:59:18.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.638 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "clone_0000000000240022", "target": ["mgr", ""]}) v1 -- 0x7f71b00d18f0 con 0x7f7194018730 2021-02-10T11:59:18.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.641 7f71a7fff700 1 -- 172.21.2.117:0/1032592982 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+78 (crc 0 0 0) 0x7f71a0000ef0 con 0x7f7194018730 2021-02-10T11:59:18.644 INFO:teuthology.orchestra.run.gibba017.stdout:/volumes/_nogroup/clone_0000000000240022/49774e2e-8b0c-4307-86b3-574de9960560 2021-02-10T11:59:18.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 msgr2=0x7f719401ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 0x7f719401ac10 crc :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 msgr2=0x7f71b00a5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 rx=0x7f71a800b8e0 tx=0x7f71a800bc80).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 shutdown_connections 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f7194018730 0x7f719401ac10 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71b01c36a0 0x7f71b00a5090 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71b01c2c80 0x7f71b01c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 --2- 172.21.2.117:0/1032592982 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71b00c3570 0x7f71b00c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 >> 172.21.2.117:0/1032592982 conn(0x7f71b00b8180 msgr2=0x7f71b00a7720 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 shutdown_connections 2021-02-10T11:59:18.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:18.643 7f71b4b73700 1 -- 172.21.2.117:0/1032592982 wait complete. 2021-02-10T11:59:18.655 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2021-02-10T11:59:18.655 DEBUG:teuthology.orchestra.run.gibba017:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:59:19.001 DEBUG:teuthology.orchestra.run.gibba025:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2021-02-10T11:59:19.346 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2021-02-10T11:59:19.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:19.346 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:19.346 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:19.683 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:19.683 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:19.683 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:59:19.684 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:59:19.684 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:20.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.gibba017.stderr:ceph-fuse[196948]: fuse finished with error 0 and tester_r 0 2021-02-10T11:59:20.040 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:20.040 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:20.040 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:20.377 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:59:20.378 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:59:20.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:59:20.378 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:20.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:20.714 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:20.715 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:59:21.053 INFO:teuthology.orchestra.run.gibba017.stdout:fuseblk 2021-02-10T11:59:21.053 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2021-02-10T11:59:21.054 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@gibba017.front.sepia.ceph.com... 2021-02-10T11:59:21.054 INFO:teuthology.orchestra.run:Running command with timeout 1800 2021-02-10T11:59:21.054 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:59:21.348 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:59:21.351 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:59:21.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.gibba017.stderr:ceph-fuse[197615]: fuse finished with error 0 and tester_r 0 2021-02-10T11:59:21.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:21.415 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:21.415 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:59:21.684 INFO:teuthology.orchestra.run.gibba017.stdout:ext2/ext3 2021-02-10T11:59:21.685 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2021-02-10T11:59:21.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:59:21.685 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:59:22.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:22.100 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:22.100 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:59:22.434 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:59:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:22.435 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:59:22.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:59:22.435 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:59:22.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:22.769 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:22.770 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:59:23.103 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:59:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:23.104 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:59:23.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2021-02-10T11:59:23.104 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:59:23.439 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph config rm mon mon_allow_pool_delete 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.853 7f40db485700 1 Processor -- start 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.853 7f40db485700 1 -- start start 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40db485700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 0x7f40d40e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40db485700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f40d40e8390 0x7f40d40e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40db485700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d40e7580 con 0x7f40d41b07f0 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40db485700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d40e5ce0 con 0x7f40d40e8390 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40db485700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40e77f0 con 0x7f40d40e7180 2021-02-10T11:59:23.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d8aa0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 0x7f40d40e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d8aa0700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 0x7f40d40e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:59108/0 (socket says 172.21.2.117:59108) 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d8aa0700 1 -- 172.21.2.117:0/1489270195 learned_addr learned my addr 172.21.2.117:0/1489270195 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d9aa2700 1 --2- 172.21.2.117:0/1489270195 >> v2:172.21.2.125:3300/0 conn(0x7f40d40e8390 0x7f40d40e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d9aa2700 1 -- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 msgr2=0x7f40d40e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d9aa2700 1 --2- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 0x7f40d40e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d9aa2700 1 -- 172.21.2.117:0/1489270195 >> v1:172.21.2.125:6789/0 conn(0x7f40d41b07f0 legacy=0x7f40d40e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.854 7f40d9aa2700 1 -- 172.21.2.117:0/1489270195 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d40e8db0 con 0x7f40d40e8390 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40d9aa2700 1 --2- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d40e8390 0x7f40d40e8790 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f40cc006e50 tx=0x7f40cc009e70).ready entity=mon.1 client_cookie=2fa6ae9d1189848f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40c7fff700 1 -- 172.21.2.117:0/1489270195 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40cc00b300 con 0x7f40d40e8390 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40c7fff700 1 -- 172.21.2.117:0/1489270195 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc003030 con 0x7f40d40e8390 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40c7fff700 1 -- 172.21.2.117:0/1489270195 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40cc00b770 con 0x7f40d40e8390 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40db485700 1 -- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d40e8390 msgr2=0x7f40d40e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:23.875 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.855 7f40db485700 1 --2- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d40e8390 0x7f40d40e8790 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7f40cc006e50 tx=0x7f40cc009e70).stop 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 -- 172.21.2.117:0/1489270195 shutdown_connections 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 --2- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d40e8390 0x7f40d40e8790 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 --2- 172.21.2.117:0/1489270195 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40e7180 0x7f40d40e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 -- 172.21.2.117:0/1489270195 >> 172.21.2.117:0/1489270195 conn(0x7f40d40ce500 msgr2=0x7f40d40e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 -- 172.21.2.117:0/1489270195 shutdown_connections 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.856 7f40db485700 1 -- 172.21.2.117:0/1489270195 wait complete. 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 Processor -- start 2021-02-10T11:59:23.876 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 -- start start 2021-02-10T11:59:23.877 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40ad6d0 0x7f40d40adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 0x7f40d41ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4093f70 con 0x7f40d41ad4b0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4197350 con 0x7f40d41aded0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40db485700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d4197570 con 0x7f40d40ad6d0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40d8aa0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40d8aa0700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33202/0 (socket says 172.21.2.117:33202) 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.871 7f40d8aa0700 1 -- 172.21.2.117:0/4235657536 learned_addr learned my addr 172.21.2.117:0/4235657536 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d92a1700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40ad6d0 0x7f40d40adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d9aa2700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 0x7f40d41ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 -- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40ad6d0 msgr2=0x7f40d40adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:23.878 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40ad6d0 0x7f40d40adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 -- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 msgr2=0x7f40d41ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 0x7f40d41ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 -- 172.21.2.117:0/4235657536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4197790 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d9aa2700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 0x7f40d41ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40d8aa0700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 rx=0x7f40c800b5e0 tx=0x7f40c800b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40c80009b0 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c800bd70 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40c80022b0 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40db485700 1 -- 172.21.2.117:0/4235657536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d4197790 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.872 7f40db485700 1 -- 172.21.2.117:0/4235657536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d4197d20 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.873 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40c8000c90 con 0x7f40d41ad4b0 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.873 7f40c5ffb700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 0x7f40b401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:23.879 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.873 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f40c80325d0 con 0x7f40d41ad4b0 2021-02-10T11:59:23.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.873 7f40d92a1700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 0x7f40b401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:23.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.874 7f40d92a1700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 0x7f40b401abc0 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:23.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.903 7f40db485700 1 -- 172.21.2.117:0/4235657536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d40e77f0 con 0x7f40d41ad4b0 2021-02-10T11:59:23.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:23.906 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40c8000960 con 0x7f40d41ad4b0 2021-02-10T11:59:24.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.054 7f40db485700 1 -- 172.21.2.117:0/4235657536 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f40d40e77f0 con 0x7f40d41ad4b0 2021-02-10T11:59:24.061 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.057 7f40c5ffb700 1 -- 172.21.2.117:0/4235657536 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f40c8000c90 con 0x7f40d41ad4b0 2021-02-10T11:59:24.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 msgr2=0x7f40b401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 0x7f40b401abc0 crc :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 msgr2=0x7f40d41ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 rx=0x7f40c800b5e0 tx=0x7f40c800b980).stop 2021-02-10T11:59:24.066 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 shutdown_connections 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40b40186c0 0x7f40b401abc0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40d41ad4b0 0x7f40d41ad8b0 unknown :-1 s=CLOSED pgs=3296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40d41aded0 0x7f40d41ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 --2- 172.21.2.117:0/4235657536 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40d40ad6d0 0x7f40d40adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 >> 172.21.2.117:0/4235657536 conn(0x7f40d41998b0 msgr2=0x7f40d41a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 shutdown_connections 2021-02-10T11:59:24.067 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.063 7f40db485700 1 -- 172.21.2.117:0/4235657536 wait complete. 2021-02-10T11:59:24.075 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath' 2021-02-10T11:59:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.487 7f77580af700 1 Processor -- start 2021-02-10T11:59:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.487 7f77580af700 1 -- start start 2021-02-10T11:59:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77580af700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77500ef3f0 0x7f77500eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77580af700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500ede60 con 0x7f77500d18e0 2021-02-10T11:59:24.507 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77580af700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77500d1ce0 con 0x7f77500eb7f0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77580af700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77500ef7f0 con 0x7f77500ef3f0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77556ca700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f77500ef3f0 0x7f77500eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f7755ecb700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f77500d18e0 0x7f77500ea9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33458/0 (socket says 172.21.2.117:33458) 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f7755ecb700 1 -- 172.21.2.117:0/207311710 learned_addr learned my addr 172.21.2.117:0/207311710 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f7754ec9700 1 -- 172.21.2.117:0/207311710 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503672514 0 0) 0x7f7744000ce0 con 0x7f77500d18e0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f7754ec9700 1 -- 172.21.2.117:0/207311710 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f773c002fe0 con 0x7f77500d18e0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77556ca700 1 -- 172.21.2.117:0/207311710 >> v1:172.21.2.125:6789/0 conn(0x7f77500eb7f0 legacy=0x7f77501b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77556ca700 1 -- 172.21.2.117:0/207311710 >> v1:172.21.2.117:6789/0 conn(0x7f77500d18e0 legacy=0x7f77500ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77556ca700 1 -- 172.21.2.117:0/207311710 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77501bb180 con 0x7f77500ef3f0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.488 7f77556ca700 1 --2- 172.21.2.117:0/207311710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500ef3f0 0x7f77500eb1d0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 rx=0x7f774c006650 tx=0x7f774c009a80).ready entity=mon.0 client_cookie=1183f493964e8e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f7754ec9700 1 -- 172.21.2.117:0/207311710 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f774c00c0c0 con 0x7f77500ef3f0 2021-02-10T11:59:24.508 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f7754ec9700 1 -- 172.21.2.117:0/207311710 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c00c340 con 0x7f77500ef3f0 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f7754ec9700 1 -- 172.21.2.117:0/207311710 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f774c00c860 con 0x7f77500ef3f0 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f77580af700 1 -- 172.21.2.117:0/207311710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500ef3f0 msgr2=0x7f77500eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f77580af700 1 --2- 172.21.2.117:0/207311710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500ef3f0 0x7f77500eb1d0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 rx=0x7f774c006650 tx=0x7f774c009a80).stop 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f77580af700 1 -- 172.21.2.117:0/207311710 shutdown_connections 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f77580af700 1 --2- 172.21.2.117:0/207311710 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500ef3f0 0x7f77500eb1d0 unknown :-1 s=CLOSED pgs=3298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.489 7f77580af700 1 -- 172.21.2.117:0/207311710 >> 172.21.2.117:0/207311710 conn(0x7f77500ced60 msgr2=0x7f77500cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.490 7f77580af700 1 -- 172.21.2.117:0/207311710 shutdown_connections 2021-02-10T11:59:24.509 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.490 7f77580af700 1 -- 172.21.2.117:0/207311710 wait complete. 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.504 7f77580af700 1 Processor -- start 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.504 7f77580af700 1 -- start start 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775008c6c0 0x7f775008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77500dd840 0x7f77500ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775008d0e0 con 0x7f77500de260 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775008d2d0 con 0x7f775008c6c0 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77580af700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77500de660 con 0x7f77500dd840 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77566cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:24.511 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77566cc700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33216/0 (socket says 172.21.2.117:33216) 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77566cc700 1 -- 172.21.2.117:0/1923546980 learned_addr learned my addr 172.21.2.117:0/1923546980 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f77556ca700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77500dd840 0x7f77500ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.505 7f7755ecb700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775008c6c0 0x7f775008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 -- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77500dd840 msgr2=0x7f77500ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77500dd840 0x7f77500ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 -- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775008c6c0 msgr2=0x7f775008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775008c6c0 0x7f775008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 -- 172.21.2.117:0/1923546980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77500de840 con 0x7f77500de260 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77566cc700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7f7748009940 tx=0x7f774800bcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:24.512 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7748000f90 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774800c2f0 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f774800c810 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77580af700 1 -- 172.21.2.117:0/1923546980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77500de840 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.506 7f77580af700 1 -- 172.21.2.117:0/1923546980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77500de840 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.507 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7748001220 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.507 7f77427fc700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 0x7f773801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.507 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7748032a70 con 0x7f77500de260 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.507 7f7755ecb700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 0x7f773801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:24.513 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.508 7f7755ecb700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 0x7f773801abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:24.541 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.538 7f77580af700 1 -- 172.21.2.117:0/1923546980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7750087280 con 0x7f77500de260 2021-02-10T11:59:24.544 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.541 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f7748000d30 con 0x7f77500de260 2021-02-10T11:59:24.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:24.671 7f77580af700 1 -- 172.21.2.117:0/1923546980 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]} v 0) v1 -- 0x7f77500ede60 con 0x7f77500de260 2021-02-10T11:59:25.413 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.409 7f77427fc700 1 -- 172.21.2.117:0/1923546980 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestVolumes.test_subvolume_clone_in_progress_getpath"]}]=0 v954) v1 ==== 157+0+0 (secure 0 0 0) 0x7f7748001220 con 0x7f77500de260 2021-02-10T11:59:25.445 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 msgr2=0x7f773801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 0x7f773801abc0 crc :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 msgr2=0x7f77500df020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 rx=0x7f7748009940 tx=0x7f774800bcf0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 shutdown_connections 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f77380186c0 0x7f773801abc0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f77500de260 0x7f77500df020 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f77500dd840 0x7f77500ddc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 --2- 172.21.2.117:0/1923546980 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f775008c6c0 0x7f775008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 >> 172.21.2.117:0/1923546980 conn(0x7f77500874b0 msgr2=0x7f77501a6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 shutdown_connections 2021-02-10T11:59:25.446 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:25.442 7f77580af700 1 -- 172.21.2.117:0/1923546980 wait complete. 2021-02-10T11:59:25.455 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) ... ERROR 2021-02-10T11:59:25.455 INFO:tasks.cephfs_test_runner: 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner: 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:Ran 21 tests in 2149.427s 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner: 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner:FAILED (errors=1, skipped=1) 2021-02-10T11:59:25.456 INFO:tasks.cephfs_test_runner: 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner:====================================================================== 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs/test_volumes.py", line 4052, in test_subvolume_clone_in_progress_getpath 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner: raise RuntimeError("expected fetching path of an pending clone to fail") 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner:RuntimeError: expected fetching path of an pending clone to fail 2021-02-10T11:59:25.457 INFO:tasks.cephfs_test_runner: 2021-02-10T11:59:25.457 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:59:25.600 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5092d15925bf408f8e34d475b216373d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_962fac13b1a781b8c7d25c1eed50d93813b8f5f0/qa/tasks/cephfs_test_runner.py", line 207, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestVolumes) 2021-02-10T11:59:25.601 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2021-02-10T11:59:25.610 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2021-02-10T11:59:25.625 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2021-02-10T11:59:25.625 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:25.625 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:25.626 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.0) 2021-02-10T11:59:25.964 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2021-02-10T11:59:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:25.965 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2021-02-10T11:59:25.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:25.966 DEBUG:teuthology.orchestra.run.gibba017:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:25.966 DEBUG:teuthology.orchestra.run.gibba017:> ' -- /home/ubuntu/cephtest/mnt.1) 2021-02-10T11:59:26.302 INFO:teuthology.orchestra.run.gibba017.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.1’: No such file or directory 2021-02-10T11:59:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:26.303 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2021-02-10T11:59:26.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:26.303 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:26.303 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.2) 2021-02-10T11:59:26.637 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.2’: No such file or directory 2021-02-10T11:59:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:26.637 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2021-02-10T11:59:26.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2021-02-10T11:59:26.638 DEBUG:teuthology.orchestra.run.gibba025:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2021-02-10T11:59:26.638 DEBUG:teuthology.orchestra.run.gibba025:> ' -- /home/ubuntu/cephtest/mnt.3) 2021-02-10T11:59:26.972 INFO:teuthology.orchestra.run.gibba025.stderr:stat: cannot read file system information for ‘/home/ubuntu/cephtest/mnt.3’: No such file or directory 2021-02-10T11:59:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T11:59:26.973 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3 2021-02-10T11:59:26.973 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-10T11:59:26.986 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2021-02-10T11:59:26.986 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:59:27.419 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.398 7f410077d700 1 Processor -- start 2021-02-10T11:59:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.398 7f410077d700 1 -- start start 2021-02-10T11:59:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f410077d700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f40f80e4e40 0x7f40f81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f410077d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 0x7f40f8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f410077d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f8085930 con 0x7f40f81b0810 2021-02-10T11:59:27.420 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f410077d700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f8084840 con 0x7f40f80e4e40 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f410077d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8084a30 con 0x7f40f80e5240 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fed9a700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f40f81b0810 0x7f40f8085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33470/0 (socket says 172.21.2.117:33470) 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fed9a700 1 -- 172.21.2.117:0/1441438184 learned_addr learned my addr 172.21.2.117:0/1441438184 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fe599700 1 --2- 172.21.2.117:0/1441438184 >> v2:172.21.2.125:3300/0 conn(0x7f40f80e4e40 0x7f40f81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fdd98700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 0x7f40f8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fd597700 1 -- 172.21.2.117:0/1441438184 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213411888 0 0) 0x7f40f0000f90 con 0x7f40f81b0810 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.399 7f40fd597700 1 -- 172.21.2.117:0/1441438184 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e4002fe0 con 0x7f40f81b0810 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fe599700 1 -- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 msgr2=0x7f40f8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:27.421 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fe599700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 0x7f40f8084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fe599700 1 -- 172.21.2.117:0/1441438184 >> v1:172.21.2.117:6789/0 conn(0x7f40f81b0810 legacy=0x7f40f8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fe599700 1 -- 172.21.2.117:0/1441438184 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8085280 con 0x7f40f80e4e40 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fdd98700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 0x7f40f8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.400 7f40fe599700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f80e4e40 0x7f40f81b01f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f40f4006650 tx=0x7f40f4009a80).ready entity=mon.1 client_cookie=6c35b589c36d8d7c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.401 7f40fd597700 1 -- 172.21.2.117:0/1441438184 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40f400c0c0 con 0x7f40f80e4e40 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.401 7f40fd597700 1 -- 172.21.2.117:0/1441438184 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f400c340 con 0x7f40f80e4e40 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.401 7f40fd597700 1 -- 172.21.2.117:0/1441438184 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40f400c860 con 0x7f40f80e4e40 2021-02-10T11:59:27.422 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.401 7f410077d700 1 -- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f80e4e40 msgr2=0x7f40f81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.401 7f410077d700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f80e4e40 0x7f40f81b01f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f40f4006650 tx=0x7f40f4009a80).stop 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 -- 172.21.2.117:0/1441438184 shutdown_connections 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f80e5240 0x7f40f8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 --2- 172.21.2.117:0/1441438184 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f80e4e40 0x7f40f81b01f0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 -- 172.21.2.117:0/1441438184 >> 172.21.2.117:0/1441438184 conn(0x7f40f80ce4e0 msgr2=0x7f40f80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 -- 172.21.2.117:0/1441438184 shutdown_connections 2021-02-10T11:59:27.423 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.402 7f410077d700 1 -- 172.21.2.117:0/1441438184 wait complete. 2021-02-10T11:59:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 Processor -- start 2021-02-10T11:59:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 -- start start 2021-02-10T11:59:27.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f81103a0 0x7f40f81107a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 0x7f40f818e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8092fd0 con 0x7f40f8092530 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f819bf70 con 0x7f40f8110dc0 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.417 7f410077d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f819c190 con 0x7f40f81103a0 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33224/0 (socket says 172.21.2.117:33224) 2021-02-10T11:59:27.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 -- 172.21.2.117:0/1958315371 learned_addr learned my addr 172.21.2.117:0/1958315371 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fdd98700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f81103a0 0x7f40f81107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fed9a700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 0x7f40f818e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 -- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f81103a0 msgr2=0x7f40f81107a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f81103a0 0x7f40f81107a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 -- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 msgr2=0x7f40f818e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 0x7f40f818e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 -- 172.21.2.117:0/1958315371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f818e9c0 con 0x7f40f8092530 2021-02-10T11:59:27.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fed9a700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 0x7f40f818e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.418 7f40fe599700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f40f400b580 tx=0x7f40f400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40f400bd10 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f40052e0 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f410077d700 1 -- 172.21.2.117:0/1958315371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f818e9c0 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f40f4005770 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f410077d700 1 -- 172.21.2.117:0/1958315371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f818e9c0 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f40f4006030 con 0x7f40f8092530 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 0x7f40e001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:27.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.419 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f40f40327f0 con 0x7f40f8092530 2021-02-10T11:59:27.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.420 7f40fdd98700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 0x7f40e001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:27.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.420 7f40fdd98700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 0x7f40e001ac10 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:27.452 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.448 7f410077d700 1 -- 172.21.2.117:0/1958315371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f810c280 con 0x7f40f8092530 2021-02-10T11:59:27.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.451 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f40f40063f0 con 0x7f40f8092530 2021-02-10T11:59:27.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.575 7f410077d700 1 -- 172.21.2.117:0/1958315371 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f8085930 con 0x7f40f8092530 2021-02-10T11:59:27.579 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.576 7f40eeffd700 1 -- 172.21.2.117:0/1958315371 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f40f4006030 con 0x7f40f8092530 2021-02-10T11:59:27.580 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:59:27.580 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:59:27.581 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 msgr2=0x7f40e001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 0x7f40e001ac10 crc :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 msgr2=0x7f40f8092930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 rx=0x7f40f400b580 tx=0x7f40f400b920).stop 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 shutdown_connections 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f40f8110dc0 0x7f40f818e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f40f81103a0 0x7f40f81107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.582 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f40e0018730 0x7f40e001ac10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 --2- 172.21.2.117:0/1958315371 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f40f8092530 0x7f40f8092930 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:27.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 >> 172.21.2.117:0/1958315371 conn(0x7f40f809d580 msgr2=0x7f40f80b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:27.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 shutdown_connections 2021-02-10T11:59:27.583 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:27.578 7f410077d700 1 -- 172.21.2.117:0/1958315371 wait complete. 2021-02-10T11:59:27.591 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:59:28.022 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 Processor -- start 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 -- start start 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc0d18e0 0x7f89cc0ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f89cc0eb000 0x7f89cc0eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc0ede60 con 0x7f89cc0ee3e0 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc0eba20 con 0x7f89cc0eb000 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89d092d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0d1ce0 con 0x7f89cc0d18e0 2021-02-10T11:59:28.023 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89c9e1b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f89cc0eb000 0x7f89cc0eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89cae1d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f89cc0ee3e0 0x7f89cc0ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33482/0 (socket says 172.21.2.117:33482) 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.003 7f89cae1d700 1 -- 172.21.2.117:0/2284198594 learned_addr learned my addr 172.21.2.117:0/2284198594 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c961a700 1 -- 172.21.2.117:0/2284198594 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959739528 0 0) 0x7f89c4001500 con 0x7f89cc0ee3e0 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c9e1b700 1 -- 172.21.2.117:0/2284198594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc0d18e0 msgr2=0x7f89cc0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c9e1b700 1 --2- 172.21.2.117:0/2284198594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc0d18e0 0x7f89cc0ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c9e1b700 1 -- 172.21.2.117:0/2284198594 >> v1:172.21.2.117:6789/0 conn(0x7f89cc0ee3e0 legacy=0x7f89cc0ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c9e1b700 1 -- 172.21.2.117:0/2284198594 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc0eefd0 con 0x7f89cc0eb000 2021-02-10T11:59:28.024 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c9e1b700 1 --2- 172.21.2.117:0/2284198594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0eb000 0x7f89cc0eb400 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f89c0006650 tx=0x7f89c0009a80).ready entity=mon.0 client_cookie=717e4e0ef075f9ff server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c961a700 1 -- 172.21.2.117:0/2284198594 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89c00009b0 con 0x7f89cc0eb000 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c961a700 1 -- 172.21.2.117:0/2284198594 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c000c0c0 con 0x7f89cc0eb000 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89c961a700 1 -- 172.21.2.117:0/2284198594 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89c000c520 con 0x7f89cc0eb000 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89d092d700 1 -- 172.21.2.117:0/2284198594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0eb000 msgr2=0x7f89cc0eb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.004 7f89d092d700 1 --2- 172.21.2.117:0/2284198594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0eb000 0x7f89cc0eb400 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 rx=0x7f89c0006650 tx=0x7f89c0009a80).stop 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 -- 172.21.2.117:0/2284198594 shutdown_connections 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 --2- 172.21.2.117:0/2284198594 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0eb000 0x7f89cc0eb400 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.025 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 --2- 172.21.2.117:0/2284198594 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc0d18e0 0x7f89cc0ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 -- 172.21.2.117:0/2284198594 >> 172.21.2.117:0/2284198594 conn(0x7f89cc0ced60 msgr2=0x7f89cc0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:28.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 -- 172.21.2.117:0/2284198594 shutdown_connections 2021-02-10T11:59:28.026 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.005 7f89d092d700 1 -- 172.21.2.117:0/2284198594 wait complete. 2021-02-10T11:59:28.027 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 Processor -- start 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 -- start start 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc1b1880 0x7f89cc1b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89cc1b22a0 0x7f89cc1b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0e0c40 con 0x7f89cc0e7550 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0e1150 con 0x7f89cc1b22a0 2021-02-10T11:59:28.028 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89d092d700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc0e7f70 con 0x7f89cc1b1880 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89cae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89cae1d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33242/0 (socket says 172.21.2.117:33242) 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.020 7f89cae1d700 1 -- 172.21.2.117:0/2207268353 learned_addr learned my addr 172.21.2.117:0/2207268353 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 -- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc1b1880 msgr2=0x7f89cc1b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc1b1880 0x7f89cc1b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 -- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89cc1b22a0 msgr2=0x7f89cc1b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89cc1b22a0 0x7f89cc1b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.029 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 -- 172.21.2.117:0/2207268353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc0e8190 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89cae1d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 rx=0x7f89c4001c10 tx=0x7f89c400ac60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89c4000f90 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c4007020 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f89c4007540 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89d092d700 1 -- 172.21.2.117:0/2207268353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc0e8190 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.021 7f89d092d700 1 -- 172.21.2.117:0/2207268353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc0e8190 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.022 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f89c4007200 con 0x7f89cc0e7550 2021-02-10T11:59:28.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.022 7f89baffd700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 0x7f89b001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.022 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f89c4032a40 con 0x7f89cc0e7550 2021-02-10T11:59:28.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.022 7f89ca61c700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 0x7f89b001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.031 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.023 7f89ca61c700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 0x7f89b001abc0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.057 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.053 7f89d092d700 1 -- 172.21.2.117:0/2207268353 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc0a1a80 con 0x7f89cc0e7550 2021-02-10T11:59:28.059 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.056 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f89c4000cc0 con 0x7f89cc0e7550 2021-02-10T11:59:28.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.188 7f89d092d700 1 -- 172.21.2.117:0/2207268353 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f89cc0a1a80 con 0x7f89b00186c0 2021-02-10T11:59:28.194 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.191 7f89baffd700 1 -- 172.21.2.117:0/2207268353 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101213 (crc 0 0 0) 0x7f89bc006a10 con 0x7f89b00186c0 2021-02-10T11:59:28.196 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:59:28.196 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T11:59:28.196 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.192 7f89d092d700 1 -- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 msgr2=0x7f89b001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.192 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 0x7f89b001abc0 crc :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.192 7f89d092d700 1 -- 172.21.2.117:0/2207268353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 msgr2=0x7f89cc0e7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.192 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 rx=0x7f89c4001c10 tx=0x7f89c400ac60).stop 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.192 7f89d092d700 1 -- 172.21.2.117:0/2207268353 shutdown_connections 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89b00186c0 0x7f89b001abc0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f89cc0e7550 0x7f89cc0e7950 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89cc1b22a0 0x7f89cc1b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.197 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 --2- 172.21.2.117:0/2207268353 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89cc1b1880 0x7f89cc1b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:28.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 -- 172.21.2.117:0/2207268353 >> 172.21.2.117:0/2207268353 conn(0x7f89cc1a7950 msgr2=0x7f89cc1b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:28.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 -- 172.21.2.117:0/2207268353 shutdown_connections 2021-02-10T11:59:28.198 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.193 7f89d092d700 1 -- 172.21.2.117:0/2207268353 wait complete. 2021-02-10T11:59:28.204 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":1316,"stamp":"2021-02-10 11:59:27.348989","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135065514,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":430,"num_write_kb":131999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":413,"ondisk_log_size":413,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672276,"kb_used_data":283600,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302444,"statfs":{"total":773094113280,"available":764213702656,"internally_reserved":8589934592,"allocated":290406400,"data_stored":279291899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":71767550,"num_objects":71,"num_object_clones":1,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":202,"num_write_kb":70132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"43.6","version":"196'22","reported_seq":"40","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.583889","last_change":"2021-02-10 11:58:04.419386","last_active":"2021-02-10 11:59:18.583889","last_peered":"2021-02-10 11:59:18.583889","last_clean":"2021-02-10 11:59:18.583889","last_became_active":"2021-02-10 11:58:04.419103","last_became_peered":"2021-02-10 11:58:04.419103","last_unstale":"2021-02-10 11:59:18.583889","last_undegraded":"2021-02-10 11:59:18.583889","last_fullsized":"2021-02-10 11:59:18.583889","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"196'8","reported_seq":"19","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:58:06.434973","last_change":"2021-02-10 11:58:02.783590","last_active":"2021-02-10 11:58:06.434973","last_peered":"2021-02-10 11:58:06.434973","last_clean":"2021-02-10 11:58:06.434973","last_became_active":"2021-02-10 11:58:02.783318","last_became_peered":"2021-02-10 11:58:02.783318","last_unstale":"2021-02-10 11:58:06.434973","last_undegraded":"2021-02-10 11:58:06.434973","last_fullsized":"2021-02-10 11:58:06.434973","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"196'14","reported_seq":"30","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.590538","last_change":"2021-02-10 11:58:04.419632","last_active":"2021-02-10 11:59:18.590538","last_peered":"2021-02-10 11:59:18.590538","last_clean":"2021-02-10 11:59:18.590538","last_became_active":"2021-02-10 11:58:04.419410","last_became_peered":"2021-02-10 11:58:04.419410","last_unstale":"2021-02-10 11:59:18.590538","last_undegraded":"2021-02-10 11:59:18.590538","last_fullsized":"2021-02-10 11:59:18.590538","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"196'16","reported_seq":"27","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:19.425612","last_change":"2021-02-10 11:58:02.783878","last_active":"2021-02-10 11:59:19.425612","last_peered":"2021-02-10 11:59:19.425612","last_clean":"2021-02-10 11:59:19.425612","last_became_active":"2021-02-10 11:58:02.783694","last_became_peered":"2021-02-10 11:58:02.783694","last_unstale":"2021-02-10 11:59:19.425612","last_undegraded":"2021-02-10 11:59:19.425612","last_fullsized":"2021-02-10 11:59:19.425612","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"196'23","reported_seq":"41","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.628470","last_change":"2021-02-10 11:58:04.421529","last_active":"2021-02-10 11:59:18.628470","last_peered":"2021-02-10 11:59:18.628470","last_clean":"2021-02-10 11:59:18.628470","last_became_active":"2021-02-10 11:58:04.421304","last_became_peered":"2021-02-10 11:58:04.421304","last_unstale":"2021-02-10 11:59:18.628470","last_undegraded":"2021-02-10 11:59:18.628470","last_fullsized":"2021-02-10 11:59:18.628470","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":21,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"196'219","reported_seq":"230","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:23.784131","last_change":"2021-02-10 11:58:02.784658","last_active":"2021-02-10 11:59:23.784131","last_peered":"2021-02-10 11:59:23.784131","last_clean":"2021-02-10 11:59:23.784131","last_became_active":"2021-02-10 11:58:02.784457","last_became_peered":"2021-02-10 11:58:02.784457","last_unstale":"2021-02-10 11:59:23.784131","last_undegraded":"2021-02-10 11:59:23.784131","last_fullsized":"2021-02-10 11:59:23.784131","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":219,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846498,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"196'20","reported_seq":"37","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.568376","last_change":"2021-02-10 11:58:04.421664","last_active":"2021-02-10 11:59:18.568376","last_peered":"2021-02-10 11:59:18.568376","last_clean":"2021-02-10 11:59:18.568376","last_became_active":"2021-02-10 11:58:04.421381","last_became_peered":"2021-02-10 11:58:04.421381","last_unstale":"2021-02-10 11:59:18.568376","last_undegraded":"2021-02-10 11:59:18.568376","last_fullsized":"2021-02-10 11:59:18.568376","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"196'6","reported_seq":"17","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:58:06.434322","last_change":"2021-02-10 11:58:03.787818","last_active":"2021-02-10 11:58:06.434322","last_peered":"2021-02-10 11:58:06.434322","last_clean":"2021-02-10 11:58:06.434322","last_became_active":"2021-02-10 11:58:03.787526","last_became_peered":"2021-02-10 11:58:03.787526","last_unstale":"2021-02-10 11:58:06.434322","last_undegraded":"2021-02-10 11:58:06.434322","last_fullsized":"2021-02-10 11:58:06.434322","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"196'25","reported_seq":"42","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.633888","last_change":"2021-02-10 11:58:04.419232","last_active":"2021-02-10 11:59:18.633888","last_peered":"2021-02-10 11:59:18.633888","last_clean":"2021-02-10 11:59:18.633888","last_became_active":"2021-02-10 11:58:04.419046","last_became_peered":"2021-02-10 11:58:04.419046","last_unstale":"2021-02-10 11:59:18.633888","last_undegraded":"2021-02-10 11:59:18.633888","last_fullsized":"2021-02-10 11:59:18.633888","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"195'1","reported_seq":"12","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.420239","last_change":"2021-02-10 11:58:03.788870","last_active":"2021-02-10 11:58:04.420239","last_peered":"2021-02-10 11:58:04.420239","last_clean":"2021-02-10 11:58:04.420239","last_became_active":"2021-02-10 11:58:03.788701","last_became_peered":"2021-02-10 11:58:03.788701","last_unstale":"2021-02-10 11:58:04.420239","last_undegraded":"2021-02-10 11:58:04.420239","last_fullsized":"2021-02-10 11:58:04.420239","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"196'19","reported_seq":"38","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.542369","last_change":"2021-02-10 11:58:04.419268","last_active":"2021-02-10 11:59:18.542369","last_peered":"2021-02-10 11:59:18.542369","last_clean":"2021-02-10 11:59:18.542369","last_became_active":"2021-02-10 11:58:04.419010","last_became_peered":"2021-02-10 11:58:04.419010","last_unstale":"2021-02-10 11:59:18.542369","last_undegraded":"2021-02-10 11:59:18.542369","last_fullsized":"2021-02-10 11:59:18.542369","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"196'2","reported_seq":"14","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:19.429309","last_change":"2021-02-10 11:58:03.789999","last_active":"2021-02-10 11:59:19.429309","last_peered":"2021-02-10 11:59:19.429309","last_clean":"2021-02-10 11:59:19.429309","last_became_active":"2021-02-10 11:58:03.789795","last_became_peered":"2021-02-10 11:58:03.789795","last_unstale":"2021-02-10 11:59:19.429309","last_undegraded":"2021-02-10 11:59:19.429309","last_fullsized":"2021-02-10 11:59:19.429309","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"196'11","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.607248","last_change":"2021-02-10 11:58:04.419297","last_active":"2021-02-10 11:59:18.607248","last_peered":"2021-02-10 11:59:18.607248","last_clean":"2021-02-10 11:59:18.607248","last_became_active":"2021-02-10 11:58:04.418997","last_became_peered":"2021-02-10 11:58:04.418997","last_unstale":"2021-02-10 11:59:18.607248","last_undegraded":"2021-02-10 11:59:18.607248","last_fullsized":"2021-02-10 11:59:18.607248","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"195'8","reported_seq":"19","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.421977","last_change":"2021-02-10 11:58:02.783688","last_active":"2021-02-10 11:58:04.421977","last_peered":"2021-02-10 11:58:04.421977","last_clean":"2021-02-10 11:58:04.421977","last_became_active":"2021-02-10 11:58:02.783528","last_became_peered":"2021-02-10 11:58:02.783528","last_unstale":"2021-02-10 11:58:04.421977","last_undegraded":"2021-02-10 11:58:04.421977","last_fullsized":"2021-02-10 11:58:04.421977","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"179","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.416354","last_change":"2021-02-10 11:22:24.884900","last_active":"2021-02-10 11:58:04.416354","last_peered":"2021-02-10 11:58:04.416354","last_clean":"2021-02-10 11:58:04.416354","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:58:04.416354","last_undegraded":"2021-02-10 11:58:04.416354","last_fullsized":"2021-02-10 11:58:04.416354","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"179","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.416410","last_change":"2021-02-10 11:22:24.885206","last_active":"2021-02-10 11:58:04.416410","last_peered":"2021-02-10 11:58:04.416410","last_clean":"2021-02-10 11:58:04.416410","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:58:04.416410","last_undegraded":"2021-02-10 11:58:04.416410","last_fullsized":"2021-02-10 11:58:04.416410","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"183","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.415349","last_change":"2021-02-10 11:22:24.885818","last_active":"2021-02-10 11:58:04.415349","last_peered":"2021-02-10 11:58:04.415349","last_clean":"2021-02-10 11:58:04.415349","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:58:04.415349","last_undegraded":"2021-02-10 11:58:04.415349","last_fullsized":"2021-02-10 11:58:04.415349","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.417604","last_change":"2021-02-10 11:22:24.886002","last_active":"2021-02-10 11:58:04.417604","last_peered":"2021-02-10 11:58:04.417604","last_clean":"2021-02-10 11:58:04.417604","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:58:04.417604","last_undegraded":"2021-02-10 11:58:04.417604","last_fullsized":"2021-02-10 11:58:04.417604","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.417584","last_change":"2021-02-10 11:22:24.885674","last_active":"2021-02-10 11:58:04.417584","last_peered":"2021-02-10 11:58:04.417584","last_clean":"2021-02-10 11:58:04.417584","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:58:04.417584","last_undegraded":"2021-02-10 11:58:04.417584","last_fullsized":"2021-02-10 11:58:04.417584","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"196'10","reported_seq":"25","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.553071","last_change":"2021-02-10 11:58:04.419612","last_active":"2021-02-10 11:59:18.553071","last_peered":"2021-02-10 11:59:18.553071","last_clean":"2021-02-10 11:59:18.553071","last_became_active":"2021-02-10 11:58:04.419401","last_became_peered":"2021-02-10 11:58:04.419401","last_unstale":"2021-02-10 11:59:18.553071","last_undegraded":"2021-02-10 11:59:18.553071","last_fullsized":"2021-02-10 11:59:18.553071","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"196'9","reported_seq":"20","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:19.425849","last_change":"2021-02-10 11:58:02.783468","last_active":"2021-02-10 11:59:19.425849","last_peered":"2021-02-10 11:59:19.425849","last_clean":"2021-02-10 11:59:19.425849","last_became_active":"2021-02-10 11:58:02.783300","last_became_peered":"2021-02-10 11:58:02.783300","last_unstale":"2021-02-10 11:59:19.425849","last_undegraded":"2021-02-10 11:59:19.425849","last_fullsized":"2021-02-10 11:59:19.425849","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"188","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.415654","last_change":"2021-02-10 11:22:24.885203","last_active":"2021-02-10 11:58:04.415654","last_peered":"2021-02-10 11:58:04.415654","last_clean":"2021-02-10 11:58:04.415654","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:58:04.415654","last_undegraded":"2021-02-10 11:58:04.415654","last_fullsized":"2021-02-10 11:58:04.415654","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.418155","last_change":"2021-02-10 11:22:24.885698","last_active":"2021-02-10 11:58:04.418155","last_peered":"2021-02-10 11:58:04.418155","last_clean":"2021-02-10 11:58:04.418155","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:58:04.418155","last_undegraded":"2021-02-10 11:58:04.418155","last_fullsized":"2021-02-10 11:58:04.418155","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"182","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.418367","last_change":"2021-02-10 11:22:24.886028","last_active":"2021-02-10 11:58:04.418367","last_peered":"2021-02-10 11:58:04.418367","last_clean":"2021-02-10 11:58:04.418367","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:58:04.418367","last_undegraded":"2021-02-10 11:58:04.418367","last_fullsized":"2021-02-10 11:58:04.418367","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":140,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":847359,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":290,"num_write_kb":920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1761280,"data_stored":1694718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269,"ondisk_log_size":269,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575297,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087940,"kb_used_data":39356,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283900,"statfs":{"total":96636764160,"available":95522713600,"internally_reserved":1073741824,"allocated":40300544,"data_stored":38903058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92100000000000004,"15min":0.96099999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33100000000000002},"max":{"1min":1.208,"5min":1.4219999999999999,"15min":1.534},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.61799999999999999,"5min":0.499,"15min":0.35899999999999999},"max":{"1min":1.212,"5min":1.304,"15min":1.498},"last":1.0489999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.81799999999999995,"15min":0.84399999999999997},"min":{"1min":0.443,"5min":0.38700000000000001,"15min":0.372},"max":{"1min":1.153,"5min":1.288,"15min":1.3400000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.84599999999999997,"15min":0.86299999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.35799999999999998},"max":{"1min":1.1140000000000001,"5min":1.411,"15min":1.4970000000000001},"last":0.875}]},{"osd":2,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82599999999999996,"15min":0.86799999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.39800000000000002},"max":{"1min":1.1020000000000001,"5min":1.1299999999999999,"15min":1.3680000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.121,"5min":1.26,"15min":1.288},"last":0.77000000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.871,"15min":0.873},"min":{"1min":0.5,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0740000000000001,"5min":1.2370000000000001,"15min":1.357},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.879,"5min":0.876,"15min":0.877},"min":{"1min":0.58799999999999997,"5min":0.48099999999999998,"15min":0.47899999999999998},"max":{"1min":1.238,"5min":1.266,"15min":1.278},"last":0.67700000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68600000000000005,"15min":0.67800000000000005},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.0269999999999999,"5min":1.2629999999999999,"15min":1.286},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72199999999999998,"15min":0.69799999999999995},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"max":{"1min":0.96899999999999997,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.56999999999999995}]},{"osd":5,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71699999999999997,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94099999999999995,"5min":1.135,"15min":1.1419999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77700000000000002,"15min":0.77100000000000002},"min":{"1min":0.57799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":0.90100000000000002,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.75600000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.56399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.58699999999999997,"5min":0.54000000000000004,"15min":0.433},"max":{"1min":1.02,"5min":1.167,"15min":1.3720000000000001},"last":0.64200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575297,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070264,"kb_used_data":21680,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301576,"statfs":{"total":96636764160,"available":95540813824,"internally_reserved":1073741824,"allocated":22200320,"data_stored":20808216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.86399999999999999,"15min":0.89800000000000002},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.5680000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91200000000000003,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.1040000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.499,"5min":0.39600000000000002,"15min":0.39000000000000001},"max":{"1min":1.1399999999999999,"5min":1.2390000000000001,"15min":1.363},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79400000000000004,"15min":0.82799999999999996},"min":{"1min":0.42199999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.1659999999999999,"5min":1.2789999999999999,"15min":1.45},"last":1.091}]},{"osd":2,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.41299999999999998,"15min":0.40300000000000002},"max":{"1min":1.2170000000000001,"5min":1.276,"15min":1.276},"last":1.028},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80200000000000005,"15min":0.81699999999999995},"min":{"1min":0.52800000000000002,"5min":0.505,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.90400000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.48499999999999999,"5min":0.442,"15min":0.441},"max":{"1min":1.2350000000000001,"5min":1.2889999999999999,"15min":1.3859999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.84499999999999997,"15min":0.84999999999999998},"min":{"1min":0.48399999999999999,"5min":0.443,"15min":0.379},"max":{"1min":1.2030000000000001,"5min":1.244,"15min":1.276},"last":1.1599999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.442},"max":{"1min":1.0349999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.54800000000000004,"5min":0.40100000000000002,"15min":0.36499999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.88500000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.55800000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94799999999999995,"5min":1.0049999999999999,"15min":1.153},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.56399999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.288,"5min":1.3660000000000001,"15min":1.6379999999999999},"last":0.70799999999999996}]},{"osd":7,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.65800000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0760000000000001,"5min":1.256,"15min":1.256},"last":0.749},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.81299999999999994},"min":{"1min":0.60199999999999998,"5min":0.51200000000000001,"15min":0.499},"max":{"1min":1.1279999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.74099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083832,"kb_used_data":35248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93288008,"statfs":{"total":96636764160,"available":95526920192,"internally_reserved":1073741824,"allocated":36093952,"data_stored":34707957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.40699999999999997,"5min":0.38700000000000001,"15min":0.36299999999999999},"max":{"1min":1.224,"5min":1.363,"15min":1.365},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.432,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2749999999999999,"5min":1.399,"15min":1.399},"last":0.75800000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.55400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.2430000000000001,"5min":1.3819999999999999,"15min":1.3979999999999999},"last":1.024},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.3},"last":0.876}]},{"osd":2,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.59799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.181,"5min":1.2330000000000001,"15min":1.2809999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.879,"15min":0.878},"min":{"1min":0.63,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.274,"15min":1.3500000000000001},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.878,"15min":0.878},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3660000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.2669999999999999},"last":0.89100000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73399999999999999,"15min":0.76600000000000001},"min":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.44800000000000001},"max":{"1min":1.101,"5min":1.103,"15min":1.343},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.55200000000000005,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.1279999999999999,"5min":1.292,"15min":1.292},"last":1.0589999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.75900000000000001},"min":{"1min":0.60399999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.278,"5min":1.278,"15min":1.3049999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.67200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.206,"5min":1.2230000000000001,"15min":1.302},"last":1.0780000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.085,"5min":1.2809999999999999,"15min":1.3220000000000001},"last":1.002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.77700000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96599999999999997,"5min":1.147,"15min":1.1870000000000001},"last":1.0449999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575296,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074420,"kb_used_data":25836,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297420,"statfs":{"total":96636764160,"available":95536558080,"internally_reserved":1073741824,"allocated":26456064,"data_stored":25068535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91800000000000004,"15min":0.93100000000000005},"min":{"1min":0.51300000000000001,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":1.3600000000000001,"5min":1.407,"15min":1.4099999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.046,"5min":0.94299999999999995,"15min":0.95399999999999996},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":1.3089999999999999,"5min":1.335,"15min":1.3879999999999999},"last":1.097}]},{"osd":1,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.84899999999999998,"15min":0.84999999999999998},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.372},"max":{"1min":1.222,"5min":1.2529999999999999,"15min":1.341},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.81499999999999995,"15min":0.82999999999999996},"min":{"1min":0.46200000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":1.119,"5min":1.181,"15min":1.2869999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.83899999999999997,"15min":0.85399999999999998},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.432},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4570000000000001},"last":0.98199999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83699999999999997,"15min":0.85699999999999998},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2569999999999999,"5min":1.321,"15min":1.3220000000000001},"last":0.89200000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.318},"max":{"1min":0.96399999999999997,"5min":1.0900000000000001,"15min":1.141},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96099999999999997,"5min":1.042,"15min":1.1279999999999999},"last":0.61399999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.438,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.113,"15min":1.204},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.0720000000000001,"15min":1.2729999999999999},"last":0.76600000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.44800000000000001,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.496,"5min":0.432,"15min":0.432},"max":{"1min":1.0249999999999999,"5min":1.165,"15min":1.321},"last":0.81399999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575297,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061300,"kb_used_data":12716,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310540,"statfs":{"total":96636764160,"available":95549992960,"internally_reserved":1073741824,"allocated":13021184,"data_stored":11639159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.78500000000000003,"15min":0.94399999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.496,"15min":0.502},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66200000000000003,"5min":0.82999999999999996,"15min":1.319},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.85099999999999998,"15min":0.91800000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.1599999999999999},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.378,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.89600000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.41899999999999998},"max":{"1min":0.92000000000000004,"5min":0.96799999999999997,"15min":1.077},"last":0.56200000000000006}]},{"osd":4,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.43099999999999999},"max":{"1min":1.212,"5min":1.3440000000000001,"15min":1.4039999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96799999999999997,"15min":0.96099999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.52100000000000002},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.82499999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.94399999999999995,"15min":0.94299999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48099999999999998},"max":{"1min":1.4690000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96699999999999997,"15min":0.97899999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.48299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.512},"last":0.878}]},{"osd":6,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0649999999999999,"15min":1.056},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.48999999999999999},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.093,"5min":1.077,"15min":1.079},"min":{"1min":0.69999999999999996,"5min":0.67400000000000004,"15min":0.53400000000000003},"max":{"1min":1.7490000000000001,"5min":1.7889999999999999,"15min":1.8100000000000001},"last":0.70599999999999996}]},{"osd":7,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.996,"15min":1.0009999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.54900000000000004},"max":{"1min":1.615,"5min":1.615,"15min":1.909},"last":0.754},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98499999999999999,"15min":0.99099999999999999},"min":{"1min":0.66700000000000004,"5min":0.63600000000000001,"15min":0.46800000000000003},"max":{"1min":1.538,"5min":1.538,"15min":1.5820000000000001},"last":0.77600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092032,"kb_used_data":43444,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279808,"statfs":{"total":96636764160,"available":95518523392,"internally_reserved":1073741824,"allocated":44486656,"data_stored":43096796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61699999999999999,"15min":0.60399999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32400000000000001},"max":{"1min":0.92900000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.629,"15min":0.62},"min":{"1min":0.44900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90000000000000002,"5min":0.92400000000000004,"15min":1.052},"last":0.56200000000000006}]},{"osd":2,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.49199999999999999,"15min":0.48699999999999999},"min":{"1min":0.372,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":0.73399999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.77400000000000002,"5min":0.877,"15min":1.077},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54600000000000004,"15min":0.53900000000000003},"min":{"1min":0.41999999999999998,"5min":0.35599999999999998,"15min":0.34200000000000003},"max":{"1min":0.71199999999999997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.0720000000000001,"15min":1.046},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.8919999999999999},"last":1.028},{"interface":"front","average":{"1min":1.143,"5min":1.05,"15min":1.03},"min":{"1min":0.76100000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.6080000000000001},"last":0.97599999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":0.94499999999999995,"15min":0.90800000000000003},"min":{"1min":0.623,"5min":0.53900000000000003,"15min":0.48799999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.391},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.98599999999999999,"15min":0.95899999999999996},"min":{"1min":0.72999999999999998,"5min":0.53200000000000003,"15min":0.48999999999999999},"max":{"1min":1.306,"5min":1.3939999999999999,"15min":1.7350000000000001},"last":0.91900000000000004}]},{"osd":6,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.079,"15min":1.044},"min":{"1min":0.60299999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.528,"5min":1.528,"15min":1.6910000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.0860000000000001,"15min":1.0549999999999999},"min":{"1min":0.70199999999999996,"5min":0.67900000000000005,"15min":0.64200000000000002},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.506},"last":0.88400000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:58:30 2021","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.004,"15min":0.98899999999999999},"min":{"1min":0.82999999999999996,"5min":0.65200000000000002,"15min":0.61799999999999999},"max":{"1min":1.4399999999999999,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0189999999999999,"15min":0.97499999999999998},"min":{"1min":0.65700000000000003,"5min":0.63900000000000001,"15min":0.54200000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.5529999999999999},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122756,"kb_used_data":74172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249084,"statfs":{"total":96636764160,"available":95487062016,"internally_reserved":1073741824,"allocated":75952128,"data_stored":74554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.90300000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.314},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53700000000000003,"15min":0.51700000000000002},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":0.443},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.63600000000000001,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":0.57099999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.97399999999999998,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.60199999999999998,"15min":0.47899999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.018,"5min":0.98299999999999998,"15min":0.96399999999999997},"min":{"1min":0.65600000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.85499999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.111,"15min":1.0920000000000001},"min":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.0840000000000001,"15min":1.071},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.77900000000000003}]},{"osd":6,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.97299999999999998,"15min":0.95699999999999996},"min":{"1min":0.59499999999999997,"5min":0.57799999999999996,"15min":0.51400000000000001},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":0.87},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1,"15min":0.99399999999999999},"min":{"1min":0.73999999999999999,"5min":0.66700000000000004,"15min":0.63600000000000001},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.86199999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":0.98499999999999999,"15min":0.98299999999999998},"min":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.98999999999999999},"min":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":0.96299999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575295,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079732,"kb_used_data":31148,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292108,"statfs":{"total":96636764160,"available":95531118592,"internally_reserved":1073741824,"allocated":31895552,"data_stored":30513630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60599999999999998,"15min":0.61099999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.087},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61199999999999999,"15min":0.59699999999999998},"min":{"1min":0.40899999999999997,"5min":0.39400000000000002,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.52700000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.495},"min":{"1min":0.40699999999999997,"5min":0.34499999999999997,"15min":0.32300000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.53700000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.0429999999999999},"last":0.57399999999999995}]},{"osd":2,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.374},"max":{"1min":0.85599999999999998,"5min":0.95399999999999996,"15min":1.018},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.41499999999999998,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.91900000000000004,"5min":0.97099999999999997,"15min":1.0309999999999999},"last":0.64000000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.93600000000000005,"15min":0.94599999999999995},"min":{"1min":0.54600000000000004,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2929999999999999,"5min":1.3999999999999999,"15min":1.5680000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.93600000000000005,"15min":0.94899999999999995},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.258,"5min":1.3360000000000001,"15min":1.4450000000000001},"last":0.99399999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.97699999999999998,"15min":0.98599999999999999},"min":{"1min":0.72599999999999998,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.248,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":1.232},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96399999999999997,"15min":0.97299999999999998},"min":{"1min":0.629,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.2569999999999999,"5min":1.3380000000000001,"15min":1.4790000000000001},"last":0.94399999999999995}]},{"osd":6,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.95799999999999996,"15min":0.97199999999999998},"min":{"1min":0.58099999999999996,"5min":0.502,"15min":0.502},"max":{"1min":1.242,"5min":1.3200000000000001,"15min":1.4830000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.98999999999999999,"15min":0.996},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.194,"5min":1.4470000000000001,"15min":1.5880000000000001},"last":1.133}]},{"osd":7,"last update":"Wed Feb 10 11:58:27 2021","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.97899999999999998,"15min":0.98299999999999998},"min":{"1min":0.67600000000000005,"5min":0.57499999999999996,"15min":0.56799999999999995},"max":{"1min":1.2569999999999999,"5min":1.431,"15min":1.5880000000000001},"last":1.056},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.97199999999999998,"15min":0.97799999999999998},"min":{"1min":0.63200000000000001,"5min":0.61699999999999999,"15min":0.59199999999999997},"max":{"1min":1.2769999999999999,"5min":1.3779999999999999,"15min":1.4510000000000001},"last":1.25}]}]}]}} 2021-02-10T11:59:28.207 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json 2021-02-10T11:59:28.638 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.618 7f142643b700 1 Processor -- start 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.618 7f142643b700 1 -- start start 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f142643b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14200ef3f0 0x7f14200eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f142643b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 0x7f14201b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f142643b700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14200ede60 con 0x7f14200d18e0 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f142643b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14200ef7f0 con 0x7f14200ef3f0 2021-02-10T11:59:28.639 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f142643b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200d1ce0 con 0x7f14200eb7f0 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f141f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14200ef3f0 0x7f14200eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f141f7fe700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f14200ef3f0 0x7f14200eb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33246/0 (socket says 172.21.2.117:33246) 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f141f7fe700 1 -- 172.21.2.117:0/549176773 learned_addr learned my addr 172.21.2.117:0/549176773 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.619 7f1424a58700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 0x7f14201b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141f7fe700 1 -- 172.21.2.117:0/549176773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 msgr2=0x7f14201b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141f7fe700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 0x7f14201b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141f7fe700 1 -- 172.21.2.117:0/549176773 >> v1:172.21.2.125:6789/0 conn(0x7f14200d18e0 legacy=0x7f14200ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:28.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141f7fe700 1 -- 172.21.2.117:0/549176773 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14201b8f90 con 0x7f14200ef3f0 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f1424a58700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 0x7f14201b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141f7fe700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200ef3f0 0x7f14200eb1d0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 rx=0x7f1414006700 tx=0x7f1414009b80).ready entity=mon.0 client_cookie=97993ba9bbd5a430 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141effd700 1 -- 172.21.2.117:0/549176773 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141400c0c0 con 0x7f14200ef3f0 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141effd700 1 -- 172.21.2.117:0/549176773 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141400c400 con 0x7f14200ef3f0 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f141effd700 1 -- 172.21.2.117:0/549176773 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141400c920 con 0x7f14200ef3f0 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f142643b700 1 -- 172.21.2.117:0/549176773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200ef3f0 msgr2=0x7f14200eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.620 7f142643b700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200ef3f0 0x7f14200eb1d0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 rx=0x7f1414006700 tx=0x7f1414009b80).stop 2021-02-10T11:59:28.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 -- 172.21.2.117:0/549176773 shutdown_connections 2021-02-10T11:59:28.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200eb7f0 0x7f14201b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 --2- 172.21.2.117:0/549176773 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200ef3f0 0x7f14200eb1d0 unknown :-1 s=CLOSED pgs=3305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 -- 172.21.2.117:0/549176773 >> 172.21.2.117:0/549176773 conn(0x7f14200ced60 msgr2=0x7f14200cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:28.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 -- 172.21.2.117:0/549176773 shutdown_connections 2021-02-10T11:59:28.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.621 7f142643b700 1 -- 172.21.2.117:0/549176773 wait complete. 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 Processor -- start 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 -- start start 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 0x7f1420095cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200e9930 0x7f14200e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.644 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200e1ec0 con 0x7f14200e8f10 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14200e1c40 con 0x7f14200e1840 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f142643b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14200e8990 con 0x7f14200e9930 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f141f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f141f7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33256/0 (socket says 172.21.2.117:33256) 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.636 7f141f7fe700 1 -- 172.21.2.117:0/4136524063 learned_addr learned my addr 172.21.2.117:0/4136524063 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 -- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200e9930 msgr2=0x7f14200e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f1424a58700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200e9930 0x7f14200e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.645 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200e9930 0x7f14200e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 -- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 msgr2=0x7f1420095cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141ffff700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 0x7f1420095cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 0x7f1420095cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 -- 172.21.2.117:0/4136524063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200e7800 con 0x7f14200e8f10 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141ffff700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 0x7f1420095cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141f7fe700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7f141400b580 tx=0x7f141400b920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f141400bd10 con 0x7f14200e8f10 2021-02-10T11:59:28.646 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14140052e0 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f142643b700 1 -- 172.21.2.117:0/4136524063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14200e7800 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1414005740 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.637 7f142643b700 1 -- 172.21.2.117:0/4136524063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14200e7800 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.638 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1414006240 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.638 7f141cff9700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 0x7f140001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.638 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1414032750 con 0x7f14200e8f10 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.638 7f141ffff700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 0x7f140001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:28.647 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.639 7f141ffff700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 0x7f140001abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:28.672 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.669 7f142643b700 1 -- 172.21.2.117:0/4136524063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1420087290 con 0x7f14200e8f10 2021-02-10T11:59:28.674 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.671 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f1414006030 con 0x7f14200e8f10 2021-02-10T11:59:28.798 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.795 7f142643b700 1 -- 172.21.2.117:0/4136524063 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14200ede60 con 0x7f14200e8f10 2021-02-10T11:59:28.799 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.796 7f141cff9700 1 -- 172.21.2.117:0/4136524063 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+14240 (secure 0 0 0) 0x7f1414006240 con 0x7f14200e8f10 2021-02-10T11:59:28.799 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:59:28.800 INFO:teuthology.orchestra.run.gibba017.stdout:{"epoch":196,"fsid":"722ef60b-feda-4d6a-867f-a5d7f666055b","created":"2021-02-10 11:21:47.530737","modified":"2021-02-10 11:58:05.414974","last_up_change":"2021-02-10 11:22:18.312455","last_in_change":"2021-02-10 11:22:18.312455","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":43,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-10 11:22:23.728210","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":42,"pool_name":"cephfs_metadata","create_time":"2021-02-10 11:58:02.381112","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"195","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":43,"pool_name":"cephfs_data","create_time":"2021-02-10 11:58:03.405092","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b2b85857-0d25-4760-8348-b4e0feb34706","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6805","nonce":36018},{"type":"v1","addr":"172.21.2.117:6808","nonce":36018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6811","nonce":36018},{"type":"v1","addr":"172.21.2.117:6814","nonce":36018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6822","nonce":36018},{"type":"v1","addr":"172.21.2.117:6823","nonce":36018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6817","nonce":36018},{"type":"v1","addr":"172.21.2.117:6820","nonce":36018}]},"public_addr":"172.21.2.117:6808/36018","cluster_addr":"172.21.2.117:6814/36018","heartbeat_back_addr":"172.21.2.117:6823/36018","heartbeat_front_addr":"172.21.2.117:6820/36018","state":["exists","up"]},{"osd":1,"uuid":"4bb369d2-72db-4361-97af-cd8850006a87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6800","nonce":36019},{"type":"v1","addr":"172.21.2.117:6802","nonce":36019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6804","nonce":36019},{"type":"v1","addr":"172.21.2.117:6807","nonce":36019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6816","nonce":36019},{"type":"v1","addr":"172.21.2.117:6819","nonce":36019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6810","nonce":36019},{"type":"v1","addr":"172.21.2.117:6813","nonce":36019}]},"public_addr":"172.21.2.117:6802/36019","cluster_addr":"172.21.2.117:6807/36019","heartbeat_back_addr":"172.21.2.117:6819/36019","heartbeat_front_addr":"172.21.2.117:6813/36019","state":["exists","up"]},{"osd":2,"uuid":"63367d7e-7197-4d2e-90b2-2b5cea7b4304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6801","nonce":36020},{"type":"v1","addr":"172.21.2.117:6803","nonce":36020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6806","nonce":36020},{"type":"v1","addr":"172.21.2.117:6809","nonce":36020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6818","nonce":36020},{"type":"v1","addr":"172.21.2.117:6821","nonce":36020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6812","nonce":36020},{"type":"v1","addr":"172.21.2.117:6815","nonce":36020}]},"public_addr":"172.21.2.117:6803/36020","cluster_addr":"172.21.2.117:6809/36020","heartbeat_back_addr":"172.21.2.117:6821/36020","heartbeat_front_addr":"172.21.2.117:6815/36020","state":["exists","up"]},{"osd":3,"uuid":"2c9e4d21-9bfc-4107-ba32-853e096e8eb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6824","nonce":36021},{"type":"v1","addr":"172.21.2.117:6825","nonce":36021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6826","nonce":36021},{"type":"v1","addr":"172.21.2.117:6827","nonce":36021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6830","nonce":36021},{"type":"v1","addr":"172.21.2.117:6831","nonce":36021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.117:6828","nonce":36021},{"type":"v1","addr":"172.21.2.117:6829","nonce":36021}]},"public_addr":"172.21.2.117:6825/36021","cluster_addr":"172.21.2.117:6827/36021","heartbeat_back_addr":"172.21.2.117:6831/36021","heartbeat_front_addr":"172.21.2.117:6829/36021","state":["exists","up"]},{"osd":4,"uuid":"be969e8d-b93d-43e0-811f-7fb5288876a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6803","nonce":36390},{"type":"v1","addr":"172.21.2.125:6806","nonce":36390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6809","nonce":36390},{"type":"v1","addr":"172.21.2.125:6813","nonce":36390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6823","nonce":36390},{"type":"v1","addr":"172.21.2.125:6827","nonce":36390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6817","nonce":36390},{"type":"v1","addr":"172.21.2.125:6819","nonce":36390}]},"public_addr":"172.21.2.125:6806/36390","cluster_addr":"172.21.2.125:6813/36390","heartbeat_back_addr":"172.21.2.125:6827/36390","heartbeat_front_addr":"172.21.2.125:6819/36390","state":["exists","up"]},{"osd":5,"uuid":"931344cc-ab1d-4eee-b067-08b04f0d8824","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6812","nonce":36389},{"type":"v1","addr":"172.21.2.125:6816","nonce":36389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6820","nonce":36389},{"type":"v1","addr":"172.21.2.125:6824","nonce":36389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6832","nonce":36389},{"type":"v1","addr":"172.21.2.125:6833","nonce":36389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6828","nonce":36389},{"type":"v1","addr":"172.21.2.125:6830","nonce":36389}]},"public_addr":"172.21.2.125:6816/36389","cluster_addr":"172.21.2.125:6824/36389","heartbeat_back_addr":"172.21.2.125:6833/36389","heartbeat_front_addr":"172.21.2.125:6830/36389","state":["exists","up"]},{"osd":6,"uuid":"f0852634-0814-469a-9e5a-ae92485dd59e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":194,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6804","nonce":36388},{"type":"v1","addr":"172.21.2.125:6807","nonce":36388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6811","nonce":36388},{"type":"v1","addr":"172.21.2.125:6815","nonce":36388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6829","nonce":36388},{"type":"v1","addr":"172.21.2.125:6831","nonce":36388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6821","nonce":36388},{"type":"v1","addr":"172.21.2.125:6826","nonce":36388}]},"public_addr":"172.21.2.125:6807/36388","cluster_addr":"172.21.2.125:6815/36388","heartbeat_back_addr":"172.21.2.125:6831/36388","heartbeat_front_addr":"172.21.2.125:6826/36388","state":["exists","up"]},{"osd":7,"uuid":"20bf9222-b6e5-4644-930f-5caff0fc379a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6802","nonce":36387},{"type":"v1","addr":"172.21.2.125:6805","nonce":36387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6808","nonce":36387},{"type":"v1","addr":"172.21.2.125:6810","nonce":36387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6822","nonce":36387},{"type":"v1","addr":"172.21.2.125:6825","nonce":36387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.125:6814","nonce":36387},{"type":"v1","addr":"172.21.2.125:6818","nonce":36387}]},"public_addr":"172.21.2.125:6805/36387","cluster_addr":"172.21.2.125:6810/36387","heartbeat_back_addr":"172.21.2.125:6825/36387","heartbeat_front_addr":"172.21.2.125:6818/36387","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854035861503,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-02-10T11:59:28.800 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.797 7f142643b700 1 -- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 msgr2=0x7f140001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.797 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 0x7f140001abc0 crc :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.797 7f142643b700 1 -- 172.21.2.117:0/4136524063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 msgr2=0x7f14200e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.797 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 rx=0x7f141400b580 tx=0x7f141400b920).stop 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 -- 172.21.2.117:0/4136524063 shutdown_connections 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f1400018730 0x7f140001abc0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f14200e8f10 0x7f14200e9310 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.801 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f14200e9930 0x7f14200e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 --2- 172.21.2.117:0/4136524063 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f14200e1840 0x7f1420095cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:28.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 -- 172.21.2.117:0/4136524063 >> 172.21.2.117:0/4136524063 conn(0x7f14201a3620 msgr2=0x7f14200963f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:28.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 -- 172.21.2.117:0/4136524063 shutdown_connections 2021-02-10T11:59:28.802 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:28.798 7f142643b700 1 -- 172.21.2.117:0/4136524063 wait complete. 2021-02-10T11:59:29.811 INFO:tasks.ceph:Scrubbing osd.0 2021-02-10T11:59:29.811 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:30.243 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 Processor -- start 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 -- start start 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7a80e4e40 0x7fa7a81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80e5240 0x7fa7a8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a8085930 con 0x7fa7a81b0810 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a8084840 con 0x7fa7a80e4e40 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.224 7fa7ad832700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8084a30 con 0x7fa7a80e5240 2021-02-10T11:59:30.244 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7a80e4e40 0x7fa7a81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa7a80e4e40 0x7fa7a81b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33260/0 (socket says 172.21.2.117:33260) 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 -- 172.21.2.117:0/936947488 learned_addr learned my addr 172.21.2.117:0/936947488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7d75700 1 --1- 172.21.2.117:0/936947488 >> v1:172.21.2.117:6789/0 conn(0x7fa7a81b0810 0x7fa7a8085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33514/0 (socket says 172.21.2.117:33514) 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 -- 172.21.2.117:0/936947488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80e5240 msgr2=0x7fa7a8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 --2- 172.21.2.117:0/936947488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80e5240 0x7fa7a8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 -- 172.21.2.117:0/936947488 >> v1:172.21.2.117:6789/0 conn(0x7fa7a81b0810 legacy=0x7fa7a8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.245 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 -- 172.21.2.117:0/936947488 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8085280 con 0x7fa7a80e4e40 2021-02-10T11:59:30.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a7574700 1 --2- 172.21.2.117:0/936947488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80e4e40 0x7fa7a81b01f0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 rx=0x7fa790006700 tx=0x7fa790009b00).ready entity=mon.0 client_cookie=c00c879c6d075c93 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a6572700 1 -- 172.21.2.117:0/936947488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa79000c0c0 con 0x7fa7a80e4e40 2021-02-10T11:59:30.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a6572700 1 -- 172.21.2.117:0/936947488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79000c340 con 0x7fa7a80e4e40 2021-02-10T11:59:30.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7a6572700 1 -- 172.21.2.117:0/936947488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa79000c820 con 0x7fa7a80e4e40 2021-02-10T11:59:30.246 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7ad832700 1 -- 172.21.2.117:0/936947488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80e4e40 msgr2=0x7fa7a81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.225 7fa7ad832700 1 --2- 172.21.2.117:0/936947488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80e4e40 0x7fa7a81b01f0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 rx=0x7fa790006700 tx=0x7fa790009b00).stop 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 -- 172.21.2.117:0/936947488 shutdown_connections 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 --2- 172.21.2.117:0/936947488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80e5240 0x7fa7a8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 --2- 172.21.2.117:0/936947488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80e4e40 0x7fa7a81b01f0 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 -- 172.21.2.117:0/936947488 >> 172.21.2.117:0/936947488 conn(0x7fa7a80ce4e0 msgr2=0x7fa7a80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:30.247 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 -- 172.21.2.117:0/936947488 shutdown_connections 2021-02-10T11:59:30.248 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.226 7fa7ad832700 1 -- 172.21.2.117:0/936947488 wait complete. 2021-02-10T11:59:30.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.240 7fa7ad832700 1 Processor -- start 2021-02-10T11:59:30.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 -- start start 2021-02-10T11:59:30.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7a80973b0 0x7fa7a8097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.249 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80ad1b0 0x7fa7a80ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 0x7fa7a809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a818d0c0 con 0x7fa7a80adbd0 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a809ae70 con 0x7fa7a80973b0 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7ad832700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a809ba10 con 0x7fa7a80ad1b0 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7a7d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 0x7fa7a809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.250 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7a7d75700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 0x7fa7a809b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33266/0 (socket says 172.21.2.117:33266) 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.241 7fa7a7d75700 1 -- 172.21.2.117:0/83443459 learned_addr learned my addr 172.21.2.117:0/83443459 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 -- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80ad1b0 msgr2=0x7fa7a80ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa7a80ad1b0 0x7fa7a80ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 -- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7a80973b0 msgr2=0x7fa7a8097980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa7a80973b0 0x7fa7a8097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 -- 172.21.2.117:0/83443459 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a809bbf0 con 0x7fa7a80adbd0 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7a7d75700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 0x7fa7a809b3f0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 rx=0x7fa7a0001b70 tx=0x7fa7a000ac50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.251 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7a0004930 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0004bb0 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7ad832700 1 -- 172.21.2.117:0/83443459 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a809bbf0 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa7ad832700 1 -- 172.21.2.117:0/83443459 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a809c180 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.242 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa7a00072a0 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.243 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa7a0004720 con 0x7fa7a80adbd0 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.243 7fa79ffff700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa78c0186c0 0x7fa78c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.252 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.243 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa7a0032a10 con 0x7fa7a80adbd0 2021-02-10T11:59:30.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.243 7fa7a7574700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa78c0186c0 0x7fa78c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.244 7fa7a7574700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa78c0186c0 0x7fa78c01abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.277 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.274 7fa7ad832700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7fa7a8098660 0x7fa7a8098a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.274 7fa7ad832700 1 -- 172.21.2.117:0/83443459 --> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7a8085930 con 0x7fa7a8098660 2021-02-10T11:59:30.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.274 7fa7a6d73700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7fa7a8098660 0x7fa7a8098a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.274 7fa7a6d73700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7fa7a8098660 0x7fa7a8098a60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.275 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== osd.0 v2:172.21.2.117:6805/36018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fa798006e40 con 0x7fa7a8098660 2021-02-10T11:59:30.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.281 7fa7ad832700 1 -- 172.21.2.117:0/83443459 --> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7fa7a8085930 con 0x7fa7a8098660 2021-02-10T11:59:30.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.281 7fa79ffff700 1 -- 172.21.2.117:0/83443459 <== osd.0 v2:172.21.2.117:6805/36018 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7fa798000db0 con 0x7fa7a8098660 2021-02-10T11:59:30.285 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 -- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7fa7a8098660 msgr2=0x7fa7a8098a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:6805/36018,v1:172.21.2.117:6808/36018] conn(0x7fa7a8098660 0x7fa7a8098a60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 -- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa78c0186c0 msgr2=0x7fa78c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa78c0186c0 0x7fa78c01abc0 crc :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 -- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 msgr2=0x7fa7a809b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.286 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 --2- 172.21.2.117:0/83443459 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa7a80adbd0 0x7fa7a809b3f0 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 rx=0x7fa7a0001b70 tx=0x7fa7a000ac50).stop 2021-02-10T11:59:30.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7a7d75700 1 -- 172.21.2.117:0/83443459 reap_dead start 2021-02-10T11:59:30.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 -- 172.21.2.117:0/83443459 shutdown_connections 2021-02-10T11:59:30.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.282 7fa7ad832700 1 -- 172.21.2.117:0/83443459 >> 172.21.2.117:0/83443459 conn(0x7fa7a818cc70 msgr2=0x7fa7a8093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:30.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.283 7fa7ad832700 1 -- 172.21.2.117:0/83443459 shutdown_connections 2021-02-10T11:59:30.287 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.283 7fa7ad832700 1 -- 172.21.2.117:0/83443459 wait complete. 2021-02-10T11:59:30.294 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 0 2021-02-10T11:59:30.723 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.702 7f3c0f694700 1 Processor -- start 2021-02-10T11:59:30.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 -- start start 2021-02-10T11:59:30.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 0x7f3c080e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3c080e8390 0x7f3c080e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.724 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c080e7580 con 0x7f3c081b07f0 2021-02-10T11:59:30.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080e5ce0 con 0x7f3c080e8390 2021-02-10T11:59:30.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0f694700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080e77f0 con 0x7f3c080e7180 2021-02-10T11:59:30.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0d4b0700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f3c081b07f0 0x7f3c080e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33526/0 (socket says 172.21.2.117:33526) 2021-02-10T11:59:30.725 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0ccaf700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 0x7f3c080e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.703 7f3c0d4b0700 1 -- 172.21.2.117:0/3177563469 learned_addr learned my addr 172.21.2.117:0/3177563469 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:30.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0dcb1700 1 --2- 172.21.2.117:0/3177563469 >> v2:172.21.2.125:3300/0 conn(0x7f3c080e8390 0x7f3c080e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.726 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0ccaf700 1 -- 172.21.2.117:0/3177563469 >> v2:172.21.2.125:3300/0 conn(0x7f3c080e8390 msgr2=0x7f3c080e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:30.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0ccaf700 1 --2- 172.21.2.117:0/3177563469 >> v2:172.21.2.125:3300/0 conn(0x7f3c080e8390 0x7f3c080e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0ccaf700 1 -- 172.21.2.117:0/3177563469 >> v1:172.21.2.117:6789/0 conn(0x7f3c081b07f0 legacy=0x7f3c080e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0ccaf700 1 -- 172.21.2.117:0/3177563469 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080e8db0 con 0x7f3c080e7180 2021-02-10T11:59:30.727 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3c0ccaf700 1 --2- 172.21.2.117:0/3177563469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 0x7f3c080e7d70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f3bf8006650 tx=0x7f3bf8009a80).ready entity=mon.2 client_cookie=a5711f0bb59892b7 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3bfffff700 1 -- 172.21.2.117:0/3177563469 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf800c0c0 con 0x7f3c080e7180 2021-02-10T11:59:30.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3bfffff700 1 -- 172.21.2.117:0/3177563469 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800c340 con 0x7f3c080e7180 2021-02-10T11:59:30.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.704 7f3bfffff700 1 -- 172.21.2.117:0/3177563469 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf800c860 con 0x7f3c080e7180 2021-02-10T11:59:30.728 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 -- 172.21.2.117:0/3177563469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 msgr2=0x7f3c080e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 --2- 172.21.2.117:0/3177563469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 0x7f3c080e7d70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f3bf8006650 tx=0x7f3bf8009a80).stop 2021-02-10T11:59:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 -- 172.21.2.117:0/3177563469 shutdown_connections 2021-02-10T11:59:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 --2- 172.21.2.117:0/3177563469 >> v2:172.21.2.125:3300/0 conn(0x7f3c080e8390 0x7f3c080e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.729 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 --2- 172.21.2.117:0/3177563469 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080e7180 0x7f3c080e7d70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.705 7f3c0f694700 1 -- 172.21.2.117:0/3177563469 >> 172.21.2.117:0/3177563469 conn(0x7f3c080ce500 msgr2=0x7f3c080e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.706 7f3c0f694700 1 -- 172.21.2.117:0/3177563469 shutdown_connections 2021-02-10T11:59:30.730 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.706 7f3c0f694700 1 -- 172.21.2.117:0/3177563469 wait complete. 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.720 7f3c0f694700 1 Processor -- start 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.720 7f3c0f694700 1 -- start start 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c081113b0 0x7f3c080c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080a29c0 0x7f3c0819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08110f70 con 0x7f3c080c1460 2021-02-10T11:59:30.732 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0819f110 con 0x7f3c081113b0 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0f694700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0819f330 con 0x7f3c080a29c0 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0ccaf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0ccaf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33284/0 (socket says 172.21.2.117:33284) 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0ccaf700 1 -- 172.21.2.117:0/3124229318 learned_addr learned my addr 172.21.2.117:0/3124229318 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0dcb1700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080a29c0 0x7f3c0819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.721 7f3c0d4b0700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c081113b0 0x7f3c080c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 -- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080a29c0 msgr2=0x7f3c0819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080a29c0 0x7f3c0819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 -- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c081113b0 msgr2=0x7f3c080c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c081113b0 0x7f3c080c0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 -- 172.21.2.117:0/3124229318 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08091c50 con 0x7f3c080c1460 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0ccaf700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f3bf8009650 tx=0x7f3bf800b960).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf800bd50 con 0x7f3c080c1460 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf8006030 con 0x7f3c080c1460 2021-02-10T11:59:30.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08091c50 con 0x7f3c080c1460 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3bf8005560 con 0x7f3c080c1460 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.722 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c080921e0 con 0x7f3c080c1460 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.723 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3bf80052e0 con 0x7f3c080c1460 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.723 7f3bfdffb700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 0x7f3be801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.723 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3bf8032550 con 0x7f3c080c1460 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.723 7f3c0d4b0700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 0x7f3be801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:30.735 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.723 7f3c0d4b0700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 0x7f3be801abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:30.756 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.753 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0810c290 con 0x7f3c080c1460 2021-02-10T11:59:30.758 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.755 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3bf80009c0 con 0x7f3c080c1460 2021-02-10T11:59:30.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.882 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mgr", ""]}) v1 -- 0x7f3c080921e0 con 0x7f3be80186c0 2021-02-10T11:59:30.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.884 7f3bfdffb700 1 -- 172.21.2.117:0/3124229318 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f3c04001080 con 0x7f3be80186c0 2021-02-10T11:59:30.887 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 0 to deep-scrub 2021-02-10T11:59:30.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.884 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 msgr2=0x7f3be801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.884 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 0x7f3be801abc0 crc :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.884 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 msgr2=0x7f3c0819e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:30.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.884 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 rx=0x7f3bf8009650 tx=0x7f3bf800b960).stop 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 shutdown_connections 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3be80186c0 0x7f3be801abc0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3c080c1460 0x7f3c0819e4d0 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3c080a29c0 0x7f3c0819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 --2- 172.21.2.117:0/3124229318 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3c081113b0 0x7f3c080c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 >> 172.21.2.117:0/3124229318 conn(0x7f3c080a3440 msgr2=0x7f3c0810de20 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 shutdown_connections 2021-02-10T11:59:30.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:30.885 7f3c0f694700 1 -- 172.21.2.117:0/3124229318 wait complete. 2021-02-10T11:59:30.897 INFO:tasks.ceph:Scrubbing osd.1 2021-02-10T11:59:30.897 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.309 7f2551a7c700 1 Processor -- start 2021-02-10T11:59:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.309 7f2551a7c700 1 -- start start 2021-02-10T11:59:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f2551a7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 0x7f254c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.331 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f2551a7c700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f254c084c60 0x7f254c085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f2551a7c700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c085930 con 0x7f254c0e4e40 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f2551a7c700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c085680 con 0x7f254c084c60 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f2551a7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1b09e0 con 0x7f254c0e5240 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f254b7fe700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f254c0e4e40 0x7f254c1b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33540/0 (socket says 172.21.2.117:33540) 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.310 7f254b7fe700 1 -- 172.21.2.117:0/1546580856 learned_addr learned my addr 172.21.2.117:0/1546580856 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577499329 0 0) 0x7f253c000ce0 con 0x7f254c0e4e40 2021-02-10T11:59:31.332 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2534002fe0 con 0x7f254c0e4e40 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254affd700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 0x7f254c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254bfff700 1 --2- 172.21.2.117:0/1546580856 >> v2:172.21.2.125:3300/0 conn(0x7f254c084c60 0x7f254c085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 792444598 0 0) 0x7f253c001180 con 0x7f254c0e4e40 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254bfff700 1 -- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 msgr2=0x7f254c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254bfff700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 0x7f254c086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254bfff700 1 -- 172.21.2.117:0/1546580856 >> v1:172.21.2.117:6789/0 conn(0x7f254c0e4e40 legacy=0x7f254c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254bfff700 1 -- 172.21.2.117:0/1546580856 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c084840 con 0x7f254c084c60 2021-02-10T11:59:31.333 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.311 7f254affd700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 0x7f254c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f254bfff700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c084c60 0x7f254c085060 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f2544006e50 tx=0x7f2544009e70).ready entity=mon.1 client_cookie=303c012ca5f4197 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f25440066f0 con 0x7f254c084c60 2021-02-10T11:59:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544003030 con 0x7f254c084c60 2021-02-10T11:59:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f2551a7c700 1 -- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c084c60 msgr2=0x7f254c085060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.334 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f254a7fc700 1 -- 172.21.2.117:0/1546580856 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f2544000f90 con 0x7f254c084c60 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.312 7f2551a7c700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c084c60 0x7f254c085060 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f2544006e50 tx=0x7f2544009e70).stop 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 -- 172.21.2.117:0/1546580856 shutdown_connections 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c084c60 0x7f254c085060 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 --2- 172.21.2.117:0/1546580856 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c0e5240 0x7f254c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 -- 172.21.2.117:0/1546580856 >> 172.21.2.117:0/1546580856 conn(0x7f254c0ce4e0 msgr2=0x7f254c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 -- 172.21.2.117:0/1546580856 shutdown_connections 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.313 7f2551a7c700 1 -- 172.21.2.117:0/1546580856 wait complete. 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.327 7f2551a7c700 1 Processor -- start 2021-02-10T11:59:31.335 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.327 7f2551a7c700 1 -- start start 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 0x7f254c08a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c08a8a0 0x7f254c08aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c1a7060 0x7f254c1a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1a7a80 con 0x7f254c0abe50 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1a7ca0 con 0x7f254c08a8a0 2021-02-10T11:59:31.336 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f2551a7c700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c1a7ec0 con 0x7f254c1a7060 2021-02-10T11:59:31.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 0x7f254c08a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 0x7f254c08a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33298/0 (socket says 172.21.2.117:33298) 2021-02-10T11:59:31.337 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 -- 172.21.2.117:0/2190719859 learned_addr learned my addr 172.21.2.117:0/2190719859 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:31.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 -- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c1a7060 msgr2=0x7f254c1a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254affd700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c08a8a0 0x7f254c08aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.338 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f254c1a7060 0x7f254c1a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 -- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c08a8a0 msgr2=0x7f254c08aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f254c08a8a0 0x7f254c08aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 -- 172.21.2.117:0/2190719859 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c0b7030 con 0x7f254c0abe50 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.328 7f254b7fe700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 0x7f254c08a280 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 rx=0x7f253c001ec0 tx=0x7f253c00a930).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f253c0070d0 con 0x7f254c0abe50 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c001180 con 0x7f254c0abe50 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f253c004830 con 0x7f254c0abe50 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c0b7030 con 0x7f254c0abe50 2021-02-10T11:59:31.339 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c0b75c0 con 0x7f254c0abe50 2021-02-10T11:59:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f253c0047c0 con 0x7f254c0abe50 2021-02-10T11:59:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f252c0186c0 0x7f252c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.329 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f253c0326b0 con 0x7f254c0abe50 2021-02-10T11:59:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.330 7f254affd700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f252c0186c0 0x7f252c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.340 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.330 7f254affd700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f252c0186c0 0x7f252c01abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.360 7f2551a7c700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7f254c0c43d0 0x7f254c0b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.360 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 --> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f254c0b75c0 con 0x7f254c0c43d0 2021-02-10T11:59:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.360 7f254bfff700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7f254c0c43d0 0x7f254c0b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.360 7f254bfff700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7f254c0c43d0 0x7f254c0b7e50 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.361 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== osd.1 v2:172.21.2.117:6800/36019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f2544006a20 con 0x7f254c0c43d0 2021-02-10T11:59:31.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.368 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 --> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f254c1a7ca0 con 0x7f254c0c43d0 2021-02-10T11:59:31.372 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.369 7f2533fff700 1 -- 172.21.2.117:0/2190719859 <== osd.1 v2:172.21.2.117:6800/36019 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f2544006db0 con 0x7f254c0c43d0 2021-02-10T11:59:31.373 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:31.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7f254c0c43d0 msgr2=0x7f254c0b7e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:6800/36019,v1:172.21.2.117:6802/36019] conn(0x7f254c0c43d0 0x7f254c0b7e50 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.373 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f252c0186c0 msgr2=0x7f252c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f252c0186c0 0x7f252c01abc0 crc :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 msgr2=0x7f254c08a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 --2- 172.21.2.117:0/2190719859 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f254c0abe50 0x7f254c08a280 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 rx=0x7f253c001ec0 tx=0x7f253c00a930).stop 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f254bfff700 1 -- 172.21.2.117:0/2190719859 reap_dead start 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 shutdown_connections 2021-02-10T11:59:31.374 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 >> 172.21.2.117:0/2190719859 conn(0x7f254c0acec0 msgr2=0x7f254c0b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:31.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 shutdown_connections 2021-02-10T11:59:31.375 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.370 7f2551a7c700 1 -- 172.21.2.117:0/2190719859 wait complete. 2021-02-10T11:59:31.381 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 1 2021-02-10T11:59:31.809 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.790 7f1561d33700 1 Processor -- start 2021-02-10T11:59:31.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.790 7f1561d33700 1 -- start start 2021-02-10T11:59:31.810 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f1561d33700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c1b07f0 0x7f155c0e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f1561d33700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155c0e7180 0x7f155c0e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f1561d33700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0e7580 con 0x7f155c0e7d70 2021-02-10T11:59:31.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f1561d33700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0e92f0 con 0x7f155c0e7180 2021-02-10T11:59:31.811 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f1561d33700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0e5ce0 con 0x7f155c1b07f0 2021-02-10T11:59:31.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155c0e7180 0x7f155c0e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f155c0e7180 0x7f155c0e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33306/0 (socket says 172.21.2.117:33306) 2021-02-10T11:59:31.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 -- 172.21.2.117:0/3658950499 learned_addr learned my addr 172.21.2.117:0/3658950499 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:31.812 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155b7fe700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c1b07f0 0x7f155c0e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 -- 172.21.2.117:0/3658950499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c1b07f0 msgr2=0x7f155c0e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:31.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c1b07f0 0x7f155c0e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 -- 172.21.2.117:0/3658950499 >> v1:172.21.2.125:6789/0 conn(0x7f155c0e7d70 legacy=0x7f155c0e8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:31.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 -- 172.21.2.117:0/3658950499 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c0e77f0 con 0x7f155c0e7180 2021-02-10T11:59:31.813 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155bfff700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c0e7180 0x7f155c0e8960 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f1554006f00 tx=0x7f1554009f10).ready entity=mon.0 client_cookie=7eb65193ec741b00 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.791 7f155a7fc700 1 -- 172.21.2.117:0/3658950499 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155400b3b0 con 0x7f155c0e7180 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f155a7fc700 1 -- 172.21.2.117:0/3658950499 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f155400b6b0 con 0x7f155c0e7180 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f155a7fc700 1 -- 172.21.2.117:0/3658950499 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f155400bb90 con 0x7f155c0e7180 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 -- 172.21.2.117:0/3658950499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c0e7180 msgr2=0x7f155c0e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c0e7180 0x7f155c0e8960 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 rx=0x7f1554006f00 tx=0x7f1554009f10).stop 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 -- 172.21.2.117:0/3658950499 shutdown_connections 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c0e7180 0x7f155c0e8960 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.814 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 --2- 172.21.2.117:0/3658950499 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c1b07f0 0x7f155c0e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 -- 172.21.2.117:0/3658950499 >> 172.21.2.117:0/3658950499 conn(0x7f155c0ce500 msgr2=0x7f155c0e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:31.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 -- 172.21.2.117:0/3658950499 shutdown_connections 2021-02-10T11:59:31.815 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.792 7f1561d33700 1 -- 172.21.2.117:0/3658950499 wait complete. 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 Processor -- start 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 -- start start 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155c0af5f0 0x7f155c08bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c08c380 0x7f155c199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c08cca0 con 0x7f155c08c780 2021-02-10T11:59:31.817 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c199c90 con 0x7f155c0af5f0 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.807 7f1561d33700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c199e70 con 0x7f155c08c380 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33312/0 (socket says 172.21.2.117:33312) 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 -- 172.21.2.117:0/1390989642 learned_addr learned my addr 172.21.2.117:0/1390989642 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 -- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c08c380 msgr2=0x7f155c199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c08c380 0x7f155c199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 -- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155c0af5f0 msgr2=0x7f155c08bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.818 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155c0af5f0 0x7f155c08bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 -- 172.21.2.117:0/1390989642 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c09d790 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f155bfff700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f155400ba30 tx=0x7f155400bdd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1554005820 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554005ba0 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f1561d33700 1 -- 172.21.2.117:0/1390989642 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c09d790 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f1554006030 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.808 7f1561d33700 1 -- 172.21.2.117:0/1390989642 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c09dd20 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.809 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f1554006030 con 0x7f155c08c780 2021-02-10T11:59:31.819 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.809 7f1553fff700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 0x7f154001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:31.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.809 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f1554032b70 con 0x7f155c08c780 2021-02-10T11:59:31.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.809 7f155b7fe700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 0x7f154001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:31.820 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.810 7f155b7fe700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 0x7f154001abd0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:31.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.841 7f1561d33700 1 -- 172.21.2.117:0/1390989642 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c106e10 con 0x7f155c08c780 2021-02-10T11:59:31.847 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.843 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f15540093f0 con 0x7f155c08c780 2021-02-10T11:59:31.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.969 7f1561d33700 1 -- 172.21.2.117:0/1390989642 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mgr", ""]}) v1 -- 0x7f155c106e10 con 0x7f15400186d0 2021-02-10T11:59:31.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.971 7f1553fff700 1 -- 172.21.2.117:0/1390989642 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f154c0009b0 con 0x7f15400186d0 2021-02-10T11:59:31.975 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 1 to deep-scrub 2021-02-10T11:59:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 -- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 msgr2=0x7f154001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 0x7f154001abd0 crc :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 -- 172.21.2.117:0/1390989642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 msgr2=0x7f155c199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 rx=0x7f155400ba30 tx=0x7f155400bdd0).stop 2021-02-10T11:59:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 -- 172.21.2.117:0/1390989642 shutdown_connections 2021-02-10T11:59:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f15400186d0 0x7f154001abd0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f155c08c780 0x7f155c199670 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:31.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f155c08c380 0x7f155c199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:31.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 --2- 172.21.2.117:0/1390989642 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f155c0af5f0 0x7f155c08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:31.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.973 7f1561d33700 1 -- 172.21.2.117:0/1390989642 >> 172.21.2.117:0/1390989642 conn(0x7f155c0953a0 msgr2=0x7f155c09e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:31.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.974 7f1561d33700 1 -- 172.21.2.117:0/1390989642 shutdown_connections 2021-02-10T11:59:31.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:31.974 7f1561d33700 1 -- 172.21.2.117:0/1390989642 wait complete. 2021-02-10T11:59:31.986 INFO:tasks.ceph:Scrubbing osd.2 2021-02-10T11:59:31.986 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:32.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.401 7f29e6421700 1 Processor -- start 2021-02-10T11:59:32.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.401 7f29e6421700 1 -- start start 2021-02-10T11:59:32.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e6421700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f29e00e5240 0x7f29e0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.424 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e6421700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e0084c60 0x7f29e0085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e6421700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e0085930 con 0x7f29e00e4e40 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e6421700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e0085680 con 0x7f29e00e5240 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e6421700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01b09e0 con 0x7f29e0084c60 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29dffff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f29e00e4e40 0x7f29e01b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33570/0 (socket says 172.21.2.117:33570) 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29dffff700 1 -- 172.21.2.117:0/1206001621 learned_addr learned my addr 172.21.2.117:0/1206001621 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241424615 0 0) 0x7f29d00009b0 con 0x7f29e00e4e40 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.402 7f29e4a3e700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e0084c60 0x7f29e0085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> v2:172.21.2.125:3300/0 conn(0x7f29e00e5240 0x7f29e0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.425 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c8002fe0 con 0x7f29e00e4e40 2021-02-10T11:59:32.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1968793135 0 0) 0x7f29d00010b0 con 0x7f29e00e4e40 2021-02-10T11:59:32.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e0084c60 msgr2=0x7f29e0085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:32.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e0084c60 0x7f29e0085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.426 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 >> v2:172.21.2.125:3300/0 conn(0x7f29e00e5240 msgr2=0x7f29e0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:32.427 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 --2- 172.21.2.117:0/1206001621 >> v2:172.21.2.125:3300/0 conn(0x7f29e00e5240 0x7f29e0086ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0084840 con 0x7f29e00e4e40 2021-02-10T11:59:32.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f29d00015b0 con 0x7f29e00e4e40 2021-02-10T11:59:32.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 >> v1:172.21.2.117:6789/0 conn(0x7f29e00e4e40 legacy=0x7f29e01b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.428 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29c8003af0 0x7f29c8005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29c804a7f0 0x7f29c804cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29c804d5b0 0x7f29c804fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c8050380 con 0x7f29c804a7f0 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80505a0 con 0x7f29c804d5b0 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29deffd700 1 -- 172.21.2.117:0/1206001621 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80507c0 con 0x7f29c8003af0 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> v2:172.21.2.125:3300/0 conn(0x7f29e00e5240 0x7f29e0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:32.429 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29c804a7f0 0x7f29c804cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29e4a3e700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29c804d5b0 0x7f29c804fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.403 7f29dffff700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29c8003af0 0x7f29c8005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 -- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29c8003af0 msgr2=0x7f29c8005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29c8003af0 0x7f29c8005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 -- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29c804d5b0 msgr2=0x7f29c804fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29c804d5b0 0x7f29c804fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.430 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 -- 172.21.2.117:0/1206001621 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8050a60 con 0x7f29c804a7f0 2021-02-10T11:59:32.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29dffff700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29c8003af0 0x7f29c8005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:32.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29df7fe700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29c804a7f0 0x7f29c804cd10 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f29d4000b80 tx=0x7f29d4009e80).ready entity=mon.0 client_cookie=be85d9b6111c22e0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29d400c5f0 con 0x7f29c804a7f0 2021-02-10T11:59:32.431 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d400c8f0 con 0x7f29c804a7f0 2021-02-10T11:59:32.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29deffd700 1 -- 172.21.2.117:0/1206001621 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29d4007070 con 0x7f29c804a7f0 2021-02-10T11:59:32.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29e6421700 1 -- 172.21.2.117:0/1206001621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29c804a7f0 msgr2=0x7f29c804cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29e6421700 1 --2- 172.21.2.117:0/1206001621 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29c804a7f0 0x7f29c804cd10 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f29d4000b80 tx=0x7f29d4009e80).stop 2021-02-10T11:59:32.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.404 7f29e4a3e700 1 -- 172.21.2.117:0/1206001621 reap_dead start 2021-02-10T11:59:32.432 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.405 7f29e6421700 1 -- 172.21.2.117:0/1206001621 shutdown_connections 2021-02-10T11:59:32.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.405 7f29e6421700 1 -- 172.21.2.117:0/1206001621 >> 172.21.2.117:0/1206001621 conn(0x7f29e00ce4e0 msgr2=0x7f29e00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:32.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.405 7f29e6421700 1 -- 172.21.2.117:0/1206001621 shutdown_connections 2021-02-10T11:59:32.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.405 7f29e6421700 1 -- 172.21.2.117:0/1206001621 wait complete. 2021-02-10T11:59:32.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.420 7f29e6421700 1 Processor -- start 2021-02-10T11:59:32.433 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.420 7f29e6421700 1 -- start start 2021-02-10T11:59:32.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 0x7f29e019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e010ffa0 0x7f29e01103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29e0090ae0 0x7f29e0090ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.436 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01109c0 con 0x7f29e00a0aa0 2021-02-10T11:59:32.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00a0ea0 con 0x7f29e0090ae0 2021-02-10T11:59:32.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e6421700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e01a0360 con 0x7f29e010ffa0 2021-02-10T11:59:32.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29dffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 0x7f29e019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29dffff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 0x7f29e019fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33328/0 (socket says 172.21.2.117:33328) 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29dffff700 1 -- 172.21.2.117:0/3490542985 learned_addr learned my addr 172.21.2.117:0/3490542985 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29df7fe700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e010ffa0 0x7f29e01103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.421 7f29e4a3e700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29e0090ae0 0x7f29e0090ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 -- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e010ffa0 msgr2=0x7f29e01103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f29e010ffa0 0x7f29e01103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 -- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29e0090ae0 msgr2=0x7f29e0090ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f29e0090ae0 0x7f29e0090ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 -- 172.21.2.117:0/3490542985 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0110d00 con 0x7f29e00a0aa0 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dffff700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 0x7f29e019fd40 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f29d0007650 tx=0x7f29d000a900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29d000acf0 con 0x7f29e00a0aa0 2021-02-10T11:59:32.439 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0004350 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29e6421700 1 -- 172.21.2.117:0/3490542985 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e0110d00 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f29d00048b0 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29e6421700 1 -- 172.21.2.117:0/3490542985 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e0110d00 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f29d0004530 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.422 7f29dcff9700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29c0018730 0x7f29c001ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.423 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f29d0032770 con 0x7f29e00a0aa0 2021-02-10T11:59:32.440 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.423 7f29df7fe700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29c0018730 0x7f29c001ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.441 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.423 7f29df7fe700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29c0018730 0x7f29c001ac10 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.454 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.451 7f29e6421700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f29e00c43d0 0x7f29e00627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.451 7f29e6421700 1 -- 172.21.2.117:0/3490542985 --> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29e01a04d0 con 0x7f29e00c43d0 2021-02-10T11:59:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.451 7f29e4a3e700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f29e00c43d0 0x7f29e00627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.452 7f29e4a3e700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f29e00c43d0 0x7f29e00627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.455 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.452 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== osd.2 v2:172.21.2.117:6801/36020 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f29d80056b0 con 0x7f29e00c43d0 2021-02-10T11:59:32.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.460 7f29e6421700 1 -- 172.21.2.117:0/3490542985 --> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f29e01109c0 con 0x7f29e00c43d0 2021-02-10T11:59:32.463 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.460 7f29dcff9700 1 -- 172.21.2.117:0/3490542985 <== osd.2 v2:172.21.2.117:6801/36020 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f29d80058c0 con 0x7f29e00c43d0 2021-02-10T11:59:32.464 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f29e00c43d0 msgr2=0x7f29e00627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:6801/36020,v1:172.21.2.117:6803/36020] conn(0x7f29e00c43d0 0x7f29e00627c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29c0018730 msgr2=0x7f29c001ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f29c0018730 0x7f29c001ac10 crc :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 msgr2=0x7f29e019fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.465 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 --2- 172.21.2.117:0/3490542985 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f29e00a0aa0 0x7f29e019fd40 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 rx=0x7f29d0007650 tx=0x7f29d000a900).stop 2021-02-10T11:59:32.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e4a3e700 1 -- 172.21.2.117:0/3490542985 reap_dead start 2021-02-10T11:59:32.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 shutdown_connections 2021-02-10T11:59:32.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 >> 172.21.2.117:0/3490542985 conn(0x7f29e00a5e90 msgr2=0x7f29e00b4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:32.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 shutdown_connections 2021-02-10T11:59:32.466 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.462 7f29e6421700 1 -- 172.21.2.117:0/3490542985 wait complete. 2021-02-10T11:59:32.473 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 2 2021-02-10T11:59:32.903 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.883 7f6c0ac81700 1 Processor -- start 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.883 7f6c0ac81700 1 -- start start 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.883 7f6c0ac81700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 0x7f6c040e4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.883 7f6c0ac81700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 0x7f6c040e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0ac81700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040e77f0 con 0x7f6c040e6170 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0ac81700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040e54b0 con 0x7f6c041b07f0 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0ac81700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040e9180 con 0x7f6c040e8d80 2021-02-10T11:59:32.904 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c03fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6c040e6170 0x7f6c040e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33588/0 (socket says 172.21.2.117:33588) 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c03fff700 1 -- 172.21.2.117:0/1724651835 learned_addr learned my addr 172.21.2.117:0/1724651835 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0929e700 1 --2- 172.21.2.117:0/1724651835 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 0x7f6c040e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c08a9d700 1 --2- 172.21.2.117:0/1724651835 >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 0x7f6c040e4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c037fe700 1 -- 172.21.2.117:0/1724651835 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643542014 0 0) 0x7f6bf40009b0 con 0x7f6c040e6170 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c037fe700 1 -- 172.21.2.117:0/1724651835 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0002fe0 con 0x7f6c040e6170 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0929e700 1 -- 172.21.2.117:0/1724651835 >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 msgr2=0x7f6c040e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0929e700 1 --2- 172.21.2.117:0/1724651835 >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 0x7f6c040e4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.905 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0929e700 1 -- 172.21.2.117:0/1724651835 >> v1:172.21.2.117:6789/0 conn(0x7f6c040e6170 legacy=0x7f6c040e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c0929e700 1 -- 172.21.2.117:0/1724651835 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040e7380 con 0x7f6c040e8d80 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.884 7f6c08a9d700 1 --2- 172.21.2.117:0/1724651835 >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 0x7f6c040e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c0929e700 1 --2- 172.21.2.117:0/1724651835 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 0x7f6c040e6d60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f6bfc006e50 tx=0x7f6bfc009e70).ready entity=mon.2 client_cookie=cff4b44054fc81f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c037fe700 1 -- 172.21.2.117:0/1724651835 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bfc006a20 con 0x7f6c040e8d80 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c037fe700 1 -- 172.21.2.117:0/1724651835 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc006790 con 0x7f6c040e8d80 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c037fe700 1 -- 172.21.2.117:0/1724651835 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bfc00b700 con 0x7f6c040e8d80 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c0ac81700 1 -- 172.21.2.117:0/1724651835 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 msgr2=0x7f6c040e6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.906 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.885 7f6c0ac81700 1 --2- 172.21.2.117:0/1724651835 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 0x7f6c040e6d60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f6bfc006e50 tx=0x7f6bfc009e70).stop 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 -- 172.21.2.117:0/1724651835 shutdown_connections 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 --2- 172.21.2.117:0/1724651835 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040e8d80 0x7f6c040e6d60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 --2- 172.21.2.117:0/1724651835 >> v2:172.21.2.125:3300/0 conn(0x7f6c041b07f0 0x7f6c040e4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 -- 172.21.2.117:0/1724651835 >> 172.21.2.117:0/1724651835 conn(0x7f6c040ce500 msgr2=0x7f6c040e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 -- 172.21.2.117:0/1724651835 shutdown_connections 2021-02-10T11:59:32.907 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.886 7f6c0ac81700 1 -- 172.21.2.117:0/1724651835 wait complete. 2021-02-10T11:59:32.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 Processor -- start 2021-02-10T11:59:32.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 -- start start 2021-02-10T11:59:32.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 0x7f6c041a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 0x7f6c041a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.909 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c040b15a0 0x7f6c04093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040b19a0 con 0x7f6c040b0a50 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040930b0 con 0x7f6c040b15a0 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.901 7f6c0ac81700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c041a3450 con 0x7f6c040a3a60 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c08a9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 0x7f6c041a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 0x7f6c041a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c08a9d700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 0x7f6c041a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:59256/0 (socket says 172.21.2.117:59256) 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c08a9d700 1 -- 172.21.2.117:0/2512970398 learned_addr learned my addr 172.21.2.117:0/2512970398 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:32.910 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c0929e700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c040b15a0 0x7f6c04093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 -- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 msgr2=0x7f6c041a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 0x7f6c041a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 -- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c040b15a0 msgr2=0x7f6c04093630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c040b15a0 0x7f6c04093630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 -- 172.21.2.117:0/2512970398 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c041a3630 con 0x7f6c040b0a50 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c03fff700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 0x7f6c041a2e30 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 rx=0x7f6bf400a4d0 tx=0x7f6bf400a870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bf400ac60 con 0x7f6c040b0a50 2021-02-10T11:59:32.911 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf4004140 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6bf4004620 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c041a3630 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.902 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c041a3630 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.903 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6bf4004320 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.903 7f6c017fa700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 0x7f6be801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.903 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6bf4032820 con 0x7f6c040b0a50 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.903 7f6c08a9d700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 0x7f6be801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:32.912 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.905 7f6c08a9d700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 0x7f6be801abc0 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:32.936 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.933 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c040b19a0 con 0x7f6c040b0a50 2021-02-10T11:59:32.938 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:32.936 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6bf4000e20 con 0x7f6c040b0a50 2021-02-10T11:59:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.065 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mgr", ""]}) v1 -- 0x7f6c040930b0 con 0x7f6be8018700 2021-02-10T11:59:33.069 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.066 7f6c017fa700 1 -- 172.21.2.117:0/2512970398 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f6bec0009b0 con 0x7f6be8018700 2021-02-10T11:59:33.070 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 2 to deep-scrub 2021-02-10T11:59:33.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 msgr2=0x7f6be801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 0x7f6be801abc0 crc :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 msgr2=0x7f6c041a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.071 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 0x7f6c041a2e30 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 rx=0x7f6bf400a4d0 tx=0x7f6bf400a870).stop 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 shutdown_connections 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6be8018700 0x7f6be801abc0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6c040b0a50 0x7f6c041a2e30 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6c040b15a0 0x7f6c04093630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 --2- 172.21.2.117:0/2512970398 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6c040a3a60 0x7f6c041a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 >> 172.21.2.117:0/2512970398 conn(0x7f6c040a7260 msgr2=0x7f6c040bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:33.072 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 shutdown_connections 2021-02-10T11:59:33.073 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.068 7f6c0ac81700 1 -- 172.21.2.117:0/2512970398 wait complete. 2021-02-10T11:59:33.079 INFO:tasks.ceph:Scrubbing osd.3 2021-02-10T11:59:33.080 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:33.514 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.494 7f42420f1700 1 Processor -- start 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 -- start start 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 0x7f423c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f423c0e5240 0x7f423c084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f423c1b0810 0x7f423c085280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423c087430 con 0x7f423c0e5240 2021-02-10T11:59:33.515 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423c0e5cc0 con 0x7f423c1b0810 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f42420f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c085930 con 0x7f423c0e4e40 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f423affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f423c0e5240 0x7f423c084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f423affd700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f423c0e5240 0x7f423c084c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33352/0 (socket says 172.21.2.117:33352) 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.495 7f423affd700 1 -- 172.21.2.117:0/3868132783 learned_addr learned my addr 172.21.2.117:0/3868132783 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423b7fe700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 0x7f423c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 -- 172.21.2.117:0/3868132783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 msgr2=0x7f423c1b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 0x7f423c1b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.516 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 -- 172.21.2.117:0/3868132783 >> v2:172.21.2.125:3300/0 conn(0x7f423c1b0810 msgr2=0x7f423c085280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 --2- 172.21.2.117:0/3868132783 >> v2:172.21.2.125:3300/0 conn(0x7f423c1b0810 0x7f423c085280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 -- 172.21.2.117:0/3868132783 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c084840 con 0x7f423c0e5240 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423b7fe700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 0x7f423c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423affd700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c0e5240 0x7f423c084c60 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f422c006700 tx=0x7f422c009b80).ready entity=mon.0 client_cookie=763d71ba9ef710f0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423a7fc700 1 -- 172.21.2.117:0/3868132783 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422c00c0c0 con 0x7f423c0e5240 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423a7fc700 1 -- 172.21.2.117:0/3868132783 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c00c340 con 0x7f423c0e5240 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f423a7fc700 1 -- 172.21.2.117:0/3868132783 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422c00c860 con 0x7f423c0e5240 2021-02-10T11:59:33.517 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f42420f1700 1 -- 172.21.2.117:0/3868132783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c0e5240 msgr2=0x7f423c084c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.497 7f42420f1700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c0e5240 0x7f423c084c60 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 rx=0x7f422c006700 tx=0x7f422c009b80).stop 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 -- 172.21.2.117:0/3868132783 shutdown_connections 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 --2- 172.21.2.117:0/3868132783 >> v2:172.21.2.125:3300/0 conn(0x7f423c1b0810 0x7f423c085280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c0e5240 0x7f423c084c60 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 --2- 172.21.2.117:0/3868132783 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0e4e40 0x7f423c1b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 -- 172.21.2.117:0/3868132783 >> 172.21.2.117:0/3868132783 conn(0x7f423c0ce4e0 msgr2=0x7f423c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 -- 172.21.2.117:0/3868132783 shutdown_connections 2021-02-10T11:59:33.518 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.498 7f42420f1700 1 -- 172.21.2.117:0/3868132783 wait complete. 2021-02-10T11:59:33.519 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.511 7f42420f1700 1 Processor -- start 2021-02-10T11:59:33.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 -- start start 2021-02-10T11:59:33.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 0x7f423c1915d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f423c191bf0 0x7f423c096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.520 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0973f0 0x7f423c0977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c195290 con 0x7f423c194e10 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0b1460 con 0x7f423c191bf0 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f42420f1700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0b1680 con 0x7f423c0973f0 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 0x7f423c1915d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 0x7f423c1915d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33356/0 (socket says 172.21.2.117:33356) 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 -- 172.21.2.117:0/3655447519 learned_addr learned my addr 172.21.2.117:0/3655447519 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 -- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0973f0 msgr2=0x7f423c0977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f423c0973f0 0x7f423c0977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:33.521 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 -- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f423c191bf0 msgr2=0x7f423c096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f423c191bf0 0x7f423c096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.512 7f423b7fe700 1 -- 172.21.2.117:0/3655447519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c0b18a0 con 0x7f423c194e10 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f423b7fe700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 0x7f423c1915d0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f4224006f20 tx=0x7f422400b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f422400bd80 con 0x7f423c194e10 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224000c90 con 0x7f423c194e10 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f42240022b0 con 0x7f423c194e10 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f42420f1700 1 -- 172.21.2.117:0/3655447519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c0b18a0 con 0x7f423c194e10 2021-02-10T11:59:33.522 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.513 7f42420f1700 1 -- 172.21.2.117:0/3655447519 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c0b1e30 con 0x7f423c194e10 2021-02-10T11:59:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.514 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f422400a000 con 0x7f423c194e10 2021-02-10T11:59:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.514 7f4233fff700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42200186c0 0x7f422001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.514 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4224032760 con 0x7f423c194e10 2021-02-10T11:59:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.514 7f423affd700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42200186c0 0x7f422001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.523 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.516 7f423affd700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42200186c0 0x7f422001abc0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:33.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.545 7f42420f1700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f423c0627c0 0x7f423c062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.548 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.545 7f42420f1700 1 -- 172.21.2.117:0/3655447519 --> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f423c0b1e30 con 0x7f423c0627c0 2021-02-10T11:59:33.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.545 7f423bfff700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f423c0627c0 0x7f423c062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.545 7f423bfff700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f423c0627c0 0x7f423c062bc0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:33.549 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.546 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== osd.3 v2:172.21.2.117:6824/36021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f423400b050 con 0x7f423c0627c0 2021-02-10T11:59:33.556 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.553 7f42420f1700 1 -- 172.21.2.117:0/3655447519 --> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f423c0b23d0 con 0x7f423c0627c0 2021-02-10T11:59:33.557 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.554 7f4233fff700 1 -- 172.21.2.117:0/3655447519 <== osd.3 v2:172.21.2.117:6824/36021 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f4234000f40 con 0x7f423c0627c0 2021-02-10T11:59:33.557 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:33.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f423c0627c0 msgr2=0x7f423c062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:6824/36021,v1:172.21.2.117:6825/36021] conn(0x7f423c0627c0 0x7f423c062bc0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.558 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42200186c0 msgr2=0x7f422001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f42200186c0 0x7f422001abc0 crc :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 msgr2=0x7f423c1915d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 --2- 172.21.2.117:0/3655447519 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f423c194e10 0x7f423c1915d0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 rx=0x7f4224006f20 tx=0x7f422400b990).stop 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f423bfff700 1 -- 172.21.2.117:0/3655447519 reap_dead start 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 shutdown_connections 2021-02-10T11:59:33.559 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 >> 172.21.2.117:0/3655447519 conn(0x7f423c18cc70 msgr2=0x7f423c093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:33.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 shutdown_connections 2021-02-10T11:59:33.560 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.556 7f42420f1700 1 -- 172.21.2.117:0/3655447519 wait complete. 2021-02-10T11:59:33.566 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 2021-02-10T11:59:33.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 Processor -- start 2021-02-10T11:59:33.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 -- start start 2021-02-10T11:59:33.997 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84780e7180 0x7f84780e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f84780e8390 0x7f84780e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84780e7580 con 0x7f84781b07f0 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84780e5ce0 con 0x7f84780e7180 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f8480520700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84780e77f0 con 0x7f84780e8390 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f847db3b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f84780e7180 0x7f84780e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f847e33c700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f84781b07f0 0x7f84780e4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33614/0 (socket says 172.21.2.117:33614) 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.978 7f847e33c700 1 -- 172.21.2.117:0/1049000516 learned_addr learned my addr 172.21.2.117:0/1049000516 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:33.998 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847d33a700 1 -- 172.21.2.117:0/1049000516 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104542114 0 0) 0x7f846c000ce0 con 0x7f84781b07f0 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847d33a700 1 -- 172.21.2.117:0/1049000516 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8464002fe0 con 0x7f84781b07f0 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847db3b700 1 -- 172.21.2.117:0/1049000516 >> v2:172.21.2.125:3300/0 conn(0x7f84780e8390 msgr2=0x7f84780e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847db3b700 1 --2- 172.21.2.117:0/1049000516 >> v2:172.21.2.125:3300/0 conn(0x7f84780e8390 0x7f84780e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847db3b700 1 -- 172.21.2.117:0/1049000516 >> v1:172.21.2.117:6789/0 conn(0x7f84781b07f0 legacy=0x7f84780e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847db3b700 1 -- 172.21.2.117:0/1049000516 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84780e8db0 con 0x7f84780e7180 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847db3b700 1 --2- 172.21.2.117:0/1049000516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780e7180 0x7f84780e7d70 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f8474006650 tx=0x7f8474009a80).ready entity=mon.0 client_cookie=451051aed0853561 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847d33a700 1 -- 172.21.2.117:0/1049000516 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f847400c0c0 con 0x7f84780e7180 2021-02-10T11:59:33.999 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847d33a700 1 -- 172.21.2.117:0/1049000516 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847400c340 con 0x7f84780e7180 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f847d33a700 1 -- 172.21.2.117:0/1049000516 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f847400c860 con 0x7f84780e7180 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f8480520700 1 -- 172.21.2.117:0/1049000516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780e7180 msgr2=0x7f84780e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.979 7f8480520700 1 --2- 172.21.2.117:0/1049000516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780e7180 0x7f84780e7d70 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 rx=0x7f8474006650 tx=0x7f8474009a80).stop 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 -- 172.21.2.117:0/1049000516 shutdown_connections 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 --2- 172.21.2.117:0/1049000516 >> v2:172.21.2.125:3300/0 conn(0x7f84780e8390 0x7f84780e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 --2- 172.21.2.117:0/1049000516 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780e7180 0x7f84780e7d70 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 -- 172.21.2.117:0/1049000516 >> 172.21.2.117:0/1049000516 conn(0x7f84780ce500 msgr2=0x7f84780e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:34.000 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 -- 172.21.2.117:0/1049000516 shutdown_connections 2021-02-10T11:59:34.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.980 7f8480520700 1 -- 172.21.2.117:0/1049000516 wait complete. 2021-02-10T11:59:34.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 Processor -- start 2021-02-10T11:59:34.001 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 -- start start 2021-02-10T11:59:34.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84780a3a60 0x7f84781a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84780b0a50 0x7f84781a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.002 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84780b19a0 con 0x7f84780b15a0 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84780930b0 con 0x7f84780b0a50 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.995 7f8480520700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84781a3450 con 0x7f84780a3a60 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33370/0 (socket says 172.21.2.117:33370) 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 -- 172.21.2.117:0/684795789 learned_addr learned my addr 172.21.2.117:0/684795789 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:34.003 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 -- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84780a3a60 msgr2=0x7f84781a2810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:34.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84780a3a60 0x7f84781a2810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 -- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84780b0a50 msgr2=0x7f84781a2e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:34.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84780b0a50 0x7f84781a2e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 -- 172.21.2.117:0/684795789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84781a3630 con 0x7f84780b15a0 2021-02-10T11:59:34.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f847eb3d700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 rx=0x7f8470009980 tx=0x7f847000bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8470003020 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8470005a80 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8470006030 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f8480520700 1 -- 172.21.2.117:0/684795789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84781a3630 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.996 7f8480520700 1 -- 172.21.2.117:0/684795789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8478093fb0 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.997 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8470006030 con 0x7f84780b15a0 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.997 7f846affd700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 0x7f846001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.997 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f84700328f0 con 0x7f84780b15a0 2021-02-10T11:59:34.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.998 7f847e33c700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 0x7f846001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:33.998 7f847e33c700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 0x7f846001abc0 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.030 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.028 7f8480520700 1 -- 172.21.2.117:0/684795789 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84780b19a0 con 0x7f84780b15a0 2021-02-10T11:59:34.033 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.031 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8470005c40 con 0x7f84780b15a0 2021-02-10T11:59:34.160 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.158 7f8480520700 1 -- 172.21.2.117:0/684795789 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mgr", ""]}) v1 -- 0x7f84781a3630 con 0x7f84600186c0 2021-02-10T11:59:34.161 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.159 7f846affd700 1 -- 172.21.2.117:0/684795789 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f846c0076f0 con 0x7f84600186c0 2021-02-10T11:59:34.161 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 3 to deep-scrub 2021-02-10T11:59:34.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.160 7f8480520700 1 -- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 msgr2=0x7f846001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 0x7f846001abc0 crc :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 -- 172.21.2.117:0/684795789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 msgr2=0x7f8478093630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 rx=0x7f8470009980 tx=0x7f847000bcd0).stop 2021-02-10T11:59:34.163 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 -- 172.21.2.117:0/684795789 shutdown_connections 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84600186c0 0x7f846001abc0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f84780b15a0 0x7f8478093630 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f84780b0a50 0x7f84781a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 --2- 172.21.2.117:0/684795789 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84780a3a60 0x7f84781a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 -- 172.21.2.117:0/684795789 >> 172.21.2.117:0/684795789 conn(0x7f84780a7260 msgr2=0x7f84780bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 -- 172.21.2.117:0/684795789 shutdown_connections 2021-02-10T11:59:34.164 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.161 7f8480520700 1 -- 172.21.2.117:0/684795789 wait complete. 2021-02-10T11:59:34.171 INFO:tasks.ceph:Scrubbing osd.4 2021-02-10T11:59:34.172 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:34.606 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 Processor -- start 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 -- start start 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b6c0e4e40 0x7f6b6c1b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c085450 0x7f6b6c086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c085930 con 0x7f6b6c0e5240 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c0e5cc0 con 0x7f6b6c0e4e40 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.588 7f6b72228700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c084840 con 0x7f6b6c085450 2021-02-10T11:59:34.607 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6b6c0e4e40 0x7f6b6c1b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b63fff700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6b6c0e5240 0x7f6b6c084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33628/0 (socket says 172.21.2.117:33628) 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b63fff700 1 -- 172.21.2.117:0/3434232402 learned_addr learned my addr 172.21.2.117:0/3434232402 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 -- 172.21.2.117:0/3434232402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c085450 msgr2=0x7f6b6c086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 --2- 172.21.2.117:0/3434232402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c085450 0x7f6b6c086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 -- 172.21.2.117:0/3434232402 >> v1:172.21.2.117:6789/0 conn(0x7f6b6c0e5240 legacy=0x7f6b6c084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6b7fe700 1 -- 172.21.2.117:0/3434232402 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699898930 0 0) 0x7f6b5c001500 con 0x7f6b6c0e5240 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 -- 172.21.2.117:0/3434232402 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c1b0810 con 0x7f6b6c0e4e40 2021-02-10T11:59:34.608 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6bfff700 1 --2- 172.21.2.117:0/3434232402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0e4e40 0x7f6b6c1b01f0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7f6b54006650 tx=0x7f6b54009a80).ready entity=mon.0 client_cookie=3dae2fe3f6dc4679 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6b7fe700 1 -- 172.21.2.117:0/3434232402 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b540009b0 con 0x7f6b6c0e4e40 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6b7fe700 1 -- 172.21.2.117:0/3434232402 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5400c0c0 con 0x7f6b6c0e4e40 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.589 7f6b6b7fe700 1 -- 172.21.2.117:0/3434232402 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5400c510 con 0x7f6b6c0e4e40 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 -- 172.21.2.117:0/3434232402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0e4e40 msgr2=0x7f6b6c1b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 --2- 172.21.2.117:0/3434232402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0e4e40 0x7f6b6c1b01f0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 rx=0x7f6b54006650 tx=0x7f6b54009a80).stop 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 -- 172.21.2.117:0/3434232402 shutdown_connections 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 --2- 172.21.2.117:0/3434232402 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c085450 0x7f6b6c086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 --2- 172.21.2.117:0/3434232402 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0e4e40 0x7f6b6c1b01f0 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.609 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 -- 172.21.2.117:0/3434232402 >> 172.21.2.117:0/3434232402 conn(0x7f6b6c0ce4e0 msgr2=0x7f6b6c0e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 -- 172.21.2.117:0/3434232402 shutdown_connections 2021-02-10T11:59:34.610 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.590 7f6b72228700 1 -- 172.21.2.117:0/3434232402 wait complete. 2021-02-10T11:59:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.604 7f6b72228700 1 Processor -- start 2021-02-10T11:59:34.611 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 -- start start 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 0x7f6b6c097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b6c0ad1b0 0x7f6b6c0ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c0adbd0 0x7f6b6c09b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c18d0c0 con 0x7f6b6c0973b0 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c09ae70 con 0x7f6b6c0ad1b0 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b72228700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c09ba10 con 0x7f6b6c0adbd0 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b6bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 0x7f6b6c097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.612 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b6bfff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 0x7f6b6c097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33382/0 (socket says 172.21.2.117:33382) 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.605 7f6b6bfff700 1 -- 172.21.2.117:0/2875516915 learned_addr learned my addr 172.21.2.117:0/2875516915 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b70845700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c0adbd0 0x7f6b6c09b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b63fff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b6c0ad1b0 0x7f6b6c0ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 -- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c0adbd0 msgr2=0x7f6b6c09b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c0adbd0 0x7f6b6c09b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 -- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b6c0ad1b0 msgr2=0x7f6b6c0ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6b6c0ad1b0 0x7f6b6c0ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.613 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 -- 172.21.2.117:0/2875516915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c09bbf0 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b70845700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6b6c0adbd0 0x7f6b6c09b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b6bfff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 0x7f6b6c097980 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 rx=0x7f6b5400b5e0 tx=0x7f6b5400b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b5400bd70 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54009290 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6b54002270 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b72228700 1 -- 172.21.2.117:0/2875516915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c09bbf0 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.606 7f6b72228700 1 -- 172.21.2.117:0/2875516915 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c09c180 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.607 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6b54000c90 con 0x7f6b6c0973b0 2021-02-10T11:59:34.614 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.607 7f6b697fa700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b500186c0 0x7f6b5001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.607 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6b54032540 con 0x7f6b6c0973b0 2021-02-10T11:59:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.607 7f6b63fff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b500186c0 0x7f6b5001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.615 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.608 7f6b63fff700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b500186c0 0x7f6b5001abc0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.638 7f6b72228700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f6b6c098660 0x7f6b6c098a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:34.640 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.638 7f6b72228700 1 -- 172.21.2.117:0/2875516915 --> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b6c085930 con 0x7f6b6c098660 2021-02-10T11:59:34.641 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.639 7f6b70845700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f6b6c098660 0x7f6b6c098a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:34.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.640 7f6b70845700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f6b6c098660 0x7f6b6c098a60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:34.642 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.640 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== osd.4 v2:172.21.2.125:6803/36390 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7f6b64006be0 con 0x7f6b6c098660 2021-02-10T11:59:34.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.647 7f6b72228700 1 -- 172.21.2.117:0/2875516915 --> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7f6b6c085930 con 0x7f6b6c098660 2021-02-10T11:59:34.649 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.647 7f6b697fa700 1 -- 172.21.2.117:0/2875516915 <== osd.4 v2:172.21.2.125:6803/36390 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7f6b64006be0 con 0x7f6b6c098660 2021-02-10T11:59:34.650 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 -- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f6b6c098660 msgr2=0x7f6b6c098a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6803/36390,v1:172.21.2.125:6806/36390] conn(0x7f6b6c098660 0x7f6b6c098a60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 -- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b500186c0 msgr2=0x7f6b5001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6b500186c0 0x7f6b5001abc0 crc :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 -- 172.21.2.117:0/2875516915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 msgr2=0x7f6b6c097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:34.651 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 --2- 172.21.2.117:0/2875516915 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6b6c0973b0 0x7f6b6c097980 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 rx=0x7f6b5400b5e0 tx=0x7f6b5400b980).stop 2021-02-10T11:59:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b70845700 1 -- 172.21.2.117:0/2875516915 reap_dead start 2021-02-10T11:59:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 -- 172.21.2.117:0/2875516915 shutdown_connections 2021-02-10T11:59:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.648 7f6b72228700 1 -- 172.21.2.117:0/2875516915 >> 172.21.2.117:0/2875516915 conn(0x7f6b6c18cc70 msgr2=0x7f6b6c093770 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.649 7f6b72228700 1 -- 172.21.2.117:0/2875516915 shutdown_connections 2021-02-10T11:59:34.652 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:34.649 7f6b72228700 1 -- 172.21.2.117:0/2875516915 wait complete. 2021-02-10T11:59:34.659 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 4 2021-02-10T11:59:35.089 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.070 7f71482f6700 1 Processor -- start 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.070 7f71482f6700 1 -- start start 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f71482f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f71482f6700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f71482f6700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71400e7580 con 0x7f71401b07f0 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f71482f6700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71400e5ce0 con 0x7f71400e8390 2021-02-10T11:59:35.090 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f71482f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71400e77f0 con 0x7f71400e7180 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7146913700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7145911700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7146913700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33396/0 (socket says 172.21.2.117:33396) 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7146913700 1 -- 172.21.2.117:0/1444309577 learned_addr learned my addr 172.21.2.117:0/1444309577 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7145911700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:59304/0 (socket says 172.21.2.117:59304) 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.071 7f7145110700 1 -- 172.21.2.117:0/1444309577 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773980724 0 0) 0x7f7134000ff0 con 0x7f71401b07f0 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145110700 1 -- 172.21.2.117:0/1444309577 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f712c002fe0 con 0x7f71401b07f0 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145911700 1 -- 172.21.2.117:0/1444309577 >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 msgr2=0x7f71400e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:35.091 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145911700 1 --2- 172.21.2.117:0/1444309577 >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145911700 1 -- 172.21.2.117:0/1444309577 >> v1:172.21.2.125:6789/0 conn(0x7f71401b07f0 legacy=0x7f71400e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145911700 1 -- 172.21.2.117:0/1444309577 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400e8db0 con 0x7f71400e7180 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7146913700 1 --2- 172.21.2.117:0/1444309577 >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145911700 1 --2- 172.21.2.117:0/1444309577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f713c006650 tx=0x7f713c009a80).ready entity=mon.2 client_cookie=f26cd629c51130f2 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145110700 1 -- 172.21.2.117:0/1444309577 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713c006390 con 0x7f71400e7180 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145110700 1 -- 172.21.2.117:0/1444309577 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c0009b0 con 0x7f71400e7180 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f7145110700 1 -- 172.21.2.117:0/1444309577 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f713c00c2b0 con 0x7f71400e7180 2021-02-10T11:59:35.092 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f71482f6700 1 -- 172.21.2.117:0/1444309577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 msgr2=0x7f71400e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.072 7f71482f6700 1 --2- 172.21.2.117:0/1444309577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f713c006650 tx=0x7f713c009a80).stop 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 -- 172.21.2.117:0/1444309577 shutdown_connections 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 --2- 172.21.2.117:0/1444309577 >> v2:172.21.2.117:3300/0 conn(0x7f71400e8390 0x7f71400e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 --2- 172.21.2.117:0/1444309577 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400e7180 0x7f71400e7d70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 -- 172.21.2.117:0/1444309577 >> 172.21.2.117:0/1444309577 conn(0x7f71400ce500 msgr2=0x7f71400e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 -- 172.21.2.117:0/1444309577 shutdown_connections 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.073 7f71482f6700 1 -- 172.21.2.117:0/1444309577 wait complete. 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.087 7f71482f6700 1 Processor -- start 2021-02-10T11:59:35.093 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.087 7f71482f6700 1 -- start start 2021-02-10T11:59:35.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71400a3a60 0x7f71401a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400b0a50 0x7f71401a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.095 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400b19a0 con 0x7f71400b15a0 2021-02-10T11:59:35.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400930b0 con 0x7f71400a3a60 2021-02-10T11:59:35.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f71482f6700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71401a3450 con 0x7f71400b0a50 2021-02-10T11:59:35.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f7146913700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.096 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f7146913700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33402/0 (socket says 172.21.2.117:33402) 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f7146913700 1 -- 172.21.2.117:0/2124333566 learned_addr learned my addr 172.21.2.117:0/2124333566 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f7146112700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71400a3a60 0x7f71401a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.088 7f7145911700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400b0a50 0x7f71401a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 -- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400b0a50 msgr2=0x7f71401a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400b0a50 0x7f71401a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.097 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 -- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71400a3a60 msgr2=0x7f71401a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71400a3a60 0x7f71401a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 -- 172.21.2.117:0/2124333566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71401a3630 con 0x7f71400b15a0 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7146913700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7f713800b940 tx=0x7f713800bce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f7138000f90 con 0x7f71400b15a0 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138003030 con 0x7f71400b15a0 2021-02-10T11:59:35.098 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f71380014f0 con 0x7f71400b15a0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f71482f6700 1 -- 172.21.2.117:0/2124333566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71401a3630 con 0x7f71400b15a0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.089 7f71482f6700 1 -- 172.21.2.117:0/2124333566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7140093fb0 con 0x7f71400b15a0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.090 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f7138001270 con 0x7f71400b15a0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.090 7f7132ffd700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 0x7f712801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.090 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f7138032aa0 con 0x7f71400b15a0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.091 7f7146112700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 0x7f712801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.099 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.092 7f7146112700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 0x7f712801abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.122 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.120 7f71482f6700 1 -- 172.21.2.117:0/2124333566 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71400b19a0 con 0x7f71400b15a0 2021-02-10T11:59:35.125 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.123 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f713800be20 con 0x7f71400b15a0 2021-02-10T11:59:35.253 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.251 7f71482f6700 1 -- 172.21.2.117:0/2124333566 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mgr", ""]}) v1 -- 0x7f71400930b0 con 0x7f71280186c0 2021-02-10T11:59:35.254 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.252 7f7132ffd700 1 -- 172.21.2.117:0/2124333566 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f71340009b0 con 0x7f71280186c0 2021-02-10T11:59:35.254 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 4 to deep-scrub 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 -- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 msgr2=0x7f712801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 0x7f712801abc0 crc :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 -- 172.21.2.117:0/2124333566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 msgr2=0x7f7140093630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 rx=0x7f713800b940 tx=0x7f713800bce0).stop 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 -- 172.21.2.117:0/2124333566 shutdown_connections 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f71280186c0 0x7f712801abc0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.256 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f71400b15a0 0x7f7140093630 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f71400b0a50 0x7f71401a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 --2- 172.21.2.117:0/2124333566 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f71400a3a60 0x7f71401a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.253 7f71482f6700 1 -- 172.21.2.117:0/2124333566 >> 172.21.2.117:0/2124333566 conn(0x7f71400a7260 msgr2=0x7f71400bd060 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:35.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.254 7f71482f6700 1 -- 172.21.2.117:0/2124333566 shutdown_connections 2021-02-10T11:59:35.257 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.254 7f71482f6700 1 -- 172.21.2.117:0/2124333566 wait complete. 2021-02-10T11:59:35.264 INFO:tasks.ceph:Scrubbing osd.5 2021-02-10T11:59:35.264 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:35.699 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.680 7fa2b52b9700 1 Processor -- start 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.680 7fa2b52b9700 1 -- start start 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2b52b9700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2b00e4e40 0x7fa2b01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2b52b9700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa2b00e5240 0x7fa2b0084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2b52b9700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b0085930 con 0x7fa2b01b0810 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2b52b9700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b00e5cc0 con 0x7fa2b00e4e40 2021-02-10T11:59:35.700 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2b52b9700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b0084840 con 0x7fa2b00e5240 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2af87e700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa2b01b0810 0x7fa2b0085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33658/0 (socket says 172.21.2.117:33658) 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2af87e700 1 -- 172.21.2.117:0/3366489660 learned_addr learned my addr 172.21.2.117:0/3366489660 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2af07d700 1 --2- 172.21.2.117:0/3366489660 >> v2:172.21.2.117:3300/0 conn(0x7fa2b00e4e40 0x7fa2b01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2ae87c700 1 --2- 172.21.2.117:0/3366489660 >> v2:172.21.2.125:3300/0 conn(0x7fa2b00e5240 0x7fa2b0084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2ae07b700 1 -- 172.21.2.117:0/3366489660 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854777604 0 0) 0x7fa2a8001500 con 0x7fa2b01b0810 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.681 7fa2ae07b700 1 -- 172.21.2.117:0/3366489660 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa298002fe0 con 0x7fa2b01b0810 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2af07d700 1 -- 172.21.2.117:0/3366489660 >> v2:172.21.2.125:3300/0 conn(0x7fa2b00e5240 msgr2=0x7fa2b0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:35.701 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2af07d700 1 --2- 172.21.2.117:0/3366489660 >> v2:172.21.2.125:3300/0 conn(0x7fa2b00e5240 0x7fa2b0084c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2af07d700 1 -- 172.21.2.117:0/3366489660 >> v1:172.21.2.117:6789/0 conn(0x7fa2b01b0810 legacy=0x7fa2b0085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2af07d700 1 -- 172.21.2.117:0/3366489660 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0085280 con 0x7fa2b00e4e40 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2af07d700 1 --2- 172.21.2.117:0/3366489660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b00e4e40 0x7fa2b01b01f0 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 rx=0x7fa2a4006650 tx=0x7fa2a4009a80).ready entity=mon.0 client_cookie=7b5d520769f55374 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2ae07b700 1 -- 172.21.2.117:0/3366489660 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a400c0c0 con 0x7fa2b00e4e40 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2ae07b700 1 -- 172.21.2.117:0/3366489660 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a400c340 con 0x7fa2b00e4e40 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2ae07b700 1 -- 172.21.2.117:0/3366489660 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a400c860 con 0x7fa2b00e4e40 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2b52b9700 1 -- 172.21.2.117:0/3366489660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b00e4e40 msgr2=0x7fa2b01b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.702 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.682 7fa2b52b9700 1 --2- 172.21.2.117:0/3366489660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b00e4e40 0x7fa2b01b01f0 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 rx=0x7fa2a4006650 tx=0x7fa2a4009a80).stop 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 -- 172.21.2.117:0/3366489660 shutdown_connections 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 --2- 172.21.2.117:0/3366489660 >> v2:172.21.2.125:3300/0 conn(0x7fa2b00e5240 0x7fa2b0084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 --2- 172.21.2.117:0/3366489660 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b00e4e40 0x7fa2b01b01f0 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 -- 172.21.2.117:0/3366489660 >> 172.21.2.117:0/3366489660 conn(0x7fa2b00ce4e0 msgr2=0x7fa2b00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 -- 172.21.2.117:0/3366489660 shutdown_connections 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.683 7fa2b52b9700 1 -- 172.21.2.117:0/3366489660 wait complete. 2021-02-10T11:59:35.703 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.697 7fa2b52b9700 1 Processor -- start 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 -- start start 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2b00a11f0 0x7fa2b00a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2b00a1c10 0x7fa2b00a2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 0x7fa2b010fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0199de0 con 0x7fa2b010f760 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00a0ca0 con 0x7fa2b00a1c10 2021-02-10T11:59:35.705 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2b52b9700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0110180 con 0x7fa2b00a11f0 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2af87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 0x7fa2b010fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2af87e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 0x7fa2b010fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33410/0 (socket says 172.21.2.117:33410) 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.698 7fa2af87e700 1 -- 172.21.2.117:0/2455183688 learned_addr learned my addr 172.21.2.117:0/2455183688 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af07d700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2b00a11f0 0x7fa2b00a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2ae87c700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2b00a1c10 0x7fa2b00a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 -- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2b00a11f0 msgr2=0x7fa2b00a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2b00a11f0 0x7fa2b00a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.706 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 -- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2b00a1c10 msgr2=0x7fa2b00a2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2b00a1c10 0x7fa2b00a2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 -- 172.21.2.117:0/2455183688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b01103a0 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2af87e700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 0x7fa2b010fb60 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7fa2a8009980 tx=0x7fa2a800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a800be10 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8006080 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2a80065a0 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b01103a0 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.699 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b0110930 con 0x7fa2b010f760 2021-02-10T11:59:35.707 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.700 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2a8006260 con 0x7fa2b010f760 2021-02-10T11:59:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.700 7fa297fff700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.700 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2a8032a30 con 0x7fa2b010f760 2021-02-10T11:59:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.700 7fa2af07d700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.708 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.701 7fa2af07d700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.730 7fa2b52b9700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7fa2b00c43d0 0x7fa2b010d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:35.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.730 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 --> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2b0110930 con 0x7fa2b00c43d0 2021-02-10T11:59:35.733 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.731 7fa2ae87c700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7fa2b00c43d0 0x7fa2b010d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:35.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.731 7fa2ae87c700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7fa2b00c43d0 0x7fa2b010d4b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:35.734 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.731 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== osd.5 v2:172.21.2.125:6812/36389 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fa2a0006a70 con 0x7fa2b00c43d0 2021-02-10T11:59:35.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.739 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 --> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7fa2b01a8260 con 0x7fa2b00c43d0 2021-02-10T11:59:35.741 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.739 7fa297fff700 1 -- 172.21.2.117:0/2455183688 <== osd.5 v2:172.21.2.125:6812/36389 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7fa2a0006e60 con 0x7fa2b00c43d0 2021-02-10T11:59:35.742 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:35.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.740 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7fa2b00c43d0 msgr2=0x7fa2b010d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.740 7fa2b52b9700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6812/36389,v1:172.21.2.125:6816/36389] conn(0x7fa2b00c43d0 0x7fa2b010d4b0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 msgr2=0x7fa29001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2900186c0 0x7fa29001abc0 crc :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:35.743 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 msgr2=0x7fa2b010fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 --2- 172.21.2.117:0/2455183688 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2b010f760 0x7fa2b010fb60 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 rx=0x7fa2a8009980 tx=0x7fa2a800bcd0).stop 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2af87e700 1 -- 172.21.2.117:0/2455183688 reap_dead start 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 shutdown_connections 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 >> 172.21.2.117:0/2455183688 conn(0x7fa2b009e8b0 msgr2=0x7fa2b010c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 shutdown_connections 2021-02-10T11:59:35.744 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:35.741 7fa2b52b9700 1 -- 172.21.2.117:0/2455183688 wait complete. 2021-02-10T11:59:35.751 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 2021-02-10T11:59:36.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 Processor -- start 2021-02-10T11:59:36.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 -- start start 2021-02-10T11:59:36.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89300e7180 0x7f89300e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300e7580 con 0x7f89300e8390 2021-02-10T11:59:36.182 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300e77f0 con 0x7f89301b07f0 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f8934d5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89300e8f80 con 0x7f89300e7180 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f892f314700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f89300e8390 0x7f89300e8790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33674/0 (socket says 172.21.2.117:33674) 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.162 7f892f314700 1 -- 172.21.2.117:0/3295390491 learned_addr learned my addr 172.21.2.117:0/3295390491 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892e312700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89300e7180 0x7f89300e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950005583 0 0) 0x7f8928000f90 con 0x7f89300e8390 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c002fe0 con 0x7f89300e8390 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.1 v1:172.21.2.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699392311 0 0) 0x7f8918001080 con 0x7f89301b07f0 2021-02-10T11:59:36.183 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> v1:172.21.2.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c003210 con 0x7f89301b07f0 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4034074897 0 0) 0x7f8928001460 con 0x7f89300e8390 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89300e7180 msgr2=0x7f89300e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f89300e7180 0x7f89300e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 >> v1:172.21.2.125:6789/0 conn(0x7f89301b07f0 legacy=0x7f89300e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> v1:172.21.2.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300e9170 con 0x7f89300e8390 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v1:172.21.2.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (unknown 984840417 0 0) 0x7f8928001960 con 0x7f89300e8390 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 >> v1:172.21.2.117:6789/0 conn(0x7f89300e8390 legacy=0x7f89300e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f891c003b90 0x7f891c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.184 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f891c006820 0x7f891c008d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f891c009530 0x7f891c00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c00c300 con 0x7f891c003b90 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891c0504f0 con 0x7f891c009530 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892eb13700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f891c003b90 0x7f891c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.163 7f892db11700 1 -- 172.21.2.117:0/3295390491 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c050710 con 0x7f891c006820 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892f314700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f891c009530 0x7f891c00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892e312700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f891c006820 0x7f891c008d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.185 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 -- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f891c006820 msgr2=0x7f891c008d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f891c006820 0x7f891c008d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 -- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f891c009530 msgr2=0x7f891c00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f891c009530 0x7f891c00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 -- 172.21.2.117:0/3295390491 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c0509b0 con 0x7f891c003b90 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892e312700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f891c006820 0x7f891c008d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892eb13700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f891c003b90 0x7f891c005fa0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f8918006ad0 tx=0x7f8918009b40).ready entity=mon.0 client_cookie=1ef856c708318d97 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918000c10 con 0x7f891c003b90 2021-02-10T11:59:36.186 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89180066d0 con 0x7f891c003b90 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892db11700 1 -- 172.21.2.117:0/3295390491 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f891800b2e0 con 0x7f891c003b90 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f8934d5a700 1 -- 172.21.2.117:0/3295390491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f891c003b90 msgr2=0x7f891c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f8934d5a700 1 --2- 172.21.2.117:0/3295390491 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f891c003b90 0x7f891c005fa0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 rx=0x7f8918006ad0 tx=0x7f8918009b40).stop 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.164 7f892f314700 1 -- 172.21.2.117:0/3295390491 reap_dead start 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.165 7f8934d5a700 1 -- 172.21.2.117:0/3295390491 shutdown_connections 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.165 7f8934d5a700 1 -- 172.21.2.117:0/3295390491 >> 172.21.2.117:0/3295390491 conn(0x7f89300ce500 msgr2=0x7f89300e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.165 7f8934d5a700 1 -- 172.21.2.117:0/3295390491 shutdown_connections 2021-02-10T11:59:36.187 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.165 7f8934d5a700 1 -- 172.21.2.117:0/3295390491 wait complete. 2021-02-10T11:59:36.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.179 7f8934d5a700 1 Processor -- start 2021-02-10T11:59:36.188 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 -- start start 2021-02-10T11:59:36.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89300bce00 0x7f89300bd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8930091d90 0x7f8930092190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300bd950 con 0x7f893010fd90 2021-02-10T11:59:36.189 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300bded0 con 0x7f89300bce00 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f8934d5a700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89300b21e0 con 0x7f8930091d90 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33432/0 (socket says 172.21.2.117:33432) 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 -- 172.21.2.117:0/856513550 learned_addr learned my addr 172.21.2.117:0/856513550 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 -- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8930091d90 msgr2=0x7f8930092190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8930091d90 0x7f8930092190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.190 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 -- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89300bce00 msgr2=0x7f89300bd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89300bce00 0x7f89300bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.180 7f892eb13700 1 -- 172.21.2.117:0/856513550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300927b0 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f892eb13700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7f891800b650 tx=0x7f891800b9f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918002e20 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918000e20 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8918006cf0 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f8934d5a700 1 -- 172.21.2.117:0/856513550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89300927b0 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f8934d5a700 1 -- 172.21.2.117:0/856513550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930092d40 con 0x7f893010fd90 2021-02-10T11:59:36.191 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8918006a70 con 0x7f893010fd90 2021-02-10T11:59:36.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.181 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f8918032780 con 0x7f893010fd90 2021-02-10T11:59:36.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.182 7f892e312700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.192 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.182 7f892e312700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 0x7f891401abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.214 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.212 7f8934d5a700 1 -- 172.21.2.117:0/856513550 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89300bde70 con 0x7f893010fd90 2021-02-10T11:59:36.217 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.215 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f891801f4f0 con 0x7f893010fd90 2021-02-10T11:59:36.344 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.342 7f8934d5a700 1 -- 172.21.2.117:0/856513550 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mgr", ""]}) v1 -- 0x7f8930092d40 con 0x7f89140186c0 2021-02-10T11:59:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.343 7f89277fe700 1 -- 172.21.2.117:0/856513550 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f8920006e70 con 0x7f89140186c0 2021-02-10T11:59:36.345 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 5 to deep-scrub 2021-02-10T11:59:36.346 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.344 7f8934d5a700 1 -- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 msgr2=0x7f891401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.344 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 0x7f891401abc0 crc :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.344 7f8934d5a700 1 -- 172.21.2.117:0/856513550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 msgr2=0x7f8930110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 rx=0x7f891800b650 tx=0x7f891800b9f0).stop 2021-02-10T11:59:36.347 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 -- 172.21.2.117:0/856513550 shutdown_connections 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f8930091d90 0x7f8930092190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f89300bce00 0x7f89300bd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f89140186c0 0x7f891401abc0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 --2- 172.21.2.117:0/856513550 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f893010fd90 0x7f8930110190 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 -- 172.21.2.117:0/856513550 >> 172.21.2.117:0/856513550 conn(0x7f893008a780 msgr2=0x7f89300c1850 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 -- 172.21.2.117:0/856513550 shutdown_connections 2021-02-10T11:59:36.348 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.345 7f8934d5a700 1 -- 172.21.2.117:0/856513550 wait complete. 2021-02-10T11:59:36.355 INFO:tasks.ceph:Scrubbing osd.6 2021-02-10T11:59:36.356 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:36.788 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.770 7fa2eee8b700 1 Processor -- start 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.770 7fa2eee8b700 1 -- start start 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2eee8b700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2e80e4e40 0x7fa2e81b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2eee8b700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7fa2e80e5240 0x7fa2e8084c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2eee8b700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8085930 con 0x7fa2e81b0810 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2eee8b700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e80e5cc0 con 0x7fa2e80e4e40 2021-02-10T11:59:36.789 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2eee8b700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e8084840 con 0x7fa2e80e5240 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2e80e4e40 0x7fa2e81b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ed4a8700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fa2e81b0810 0x7fa2e8085eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33690/0 (socket says 172.21.2.117:33690) 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fa2e80e4e40 0x7fa2e81b01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33440/0 (socket says 172.21.2.117:33440) 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ed4a8700 1 -- 172.21.2.117:0/3611580483 learned_addr learned my addr 172.21.2.117:0/3611580483 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 -- 172.21.2.117:0/3611580483 >> v2:172.21.2.125:3300/0 conn(0x7fa2e80e5240 msgr2=0x7fa2e8084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 --2- 172.21.2.117:0/3611580483 >> v2:172.21.2.125:3300/0 conn(0x7fa2e80e5240 0x7fa2e8084c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.790 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 -- 172.21.2.117:0/3611580483 >> v1:172.21.2.117:6789/0 conn(0x7fa2e81b0810 legacy=0x7fa2e8085eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.771 7fa2ecca7700 1 -- 172.21.2.117:0/3611580483 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8085280 con 0x7fa2e80e4e40 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2ecca7700 1 --2- 172.21.2.117:0/3611580483 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e80e4e40 0x7fa2e81b01f0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 rx=0x7fa2d8006650 tx=0x7fa2d8009a80).ready entity=mon.0 client_cookie=4fbf1ae5d802d0f server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2e77fe700 1 -- 172.21.2.117:0/3611580483 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2d800c0c0 con 0x7fa2e80e4e40 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2e77fe700 1 -- 172.21.2.117:0/3611580483 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d800c340 con 0x7fa2e80e4e40 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2e77fe700 1 -- 172.21.2.117:0/3611580483 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2d800c860 con 0x7fa2e80e4e40 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 -- 172.21.2.117:0/3611580483 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e80e4e40 msgr2=0x7fa2e81b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 --2- 172.21.2.117:0/3611580483 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e80e4e40 0x7fa2e81b01f0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 rx=0x7fa2d8006650 tx=0x7fa2d8009a80).stop 2021-02-10T11:59:36.791 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 -- 172.21.2.117:0/3611580483 shutdown_connections 2021-02-10T11:59:36.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 --2- 172.21.2.117:0/3611580483 >> v2:172.21.2.125:3300/0 conn(0x7fa2e80e5240 0x7fa2e8084c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 --2- 172.21.2.117:0/3611580483 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e80e4e40 0x7fa2e81b01f0 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.772 7fa2eee8b700 1 -- 172.21.2.117:0/3611580483 >> 172.21.2.117:0/3611580483 conn(0x7fa2e80ce4e0 msgr2=0x7fa2e80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:36.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.773 7fa2eee8b700 1 -- 172.21.2.117:0/3611580483 shutdown_connections 2021-02-10T11:59:36.792 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.773 7fa2eee8b700 1 -- 172.21.2.117:0/3611580483 wait complete. 2021-02-10T11:59:36.793 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 Processor -- start 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 -- start start 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2e81b5cf0 0x7fa2e80973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 0x7fa2e80979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2e8098680 0x7fa2e8098a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8098180 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b6220 con 0x7fa2e81b5cf0 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2eee8b700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b94a0 con 0x7fa2e8098680 2021-02-10T11:59:36.794 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2e7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 0x7fa2e80979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2e7fff700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 0x7fa2e80979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33448/0 (socket says 172.21.2.117:33448) 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.787 7fa2e7fff700 1 -- 172.21.2.117:0/335928488 learned_addr learned my addr 172.21.2.117:0/335928488 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 -- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2e8098680 msgr2=0x7fa2e8098a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fa2e8098680 0x7fa2e8098a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 -- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2e81b5cf0 msgr2=0x7fa2e80973b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fa2e81b5cf0 0x7fa2e80973b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 -- 172.21.2.117:0/335928488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80990a0 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.795 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e7fff700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 0x7fa2e80979d0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7fa2dc00b530 tx=0x7fa2dc00b8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2dc00bcc0 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc000c90 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e80990a0 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fa2dc0022a0 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.788 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e80ad4d0 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.789 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fa2dc005310 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.789 7fa2e57fa700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2c8018700 0x7fa2c801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.796 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.789 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fa2dc032900 con 0x7fa2e81a9cd0 2021-02-10T11:59:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.789 7fa2ecca7700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2c8018700 0x7fa2c801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.797 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.790 7fa2ecca7700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2c8018700 0x7fa2c801abc0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.822 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.820 7fa2eee8b700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7fa2e80add90 0x7fa2e80627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:36.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.820 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 --> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2e810c2f0 con 0x7fa2e80add90 2021-02-10T11:59:36.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.820 7fa2ed4a8700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7fa2e80add90 0x7fa2e80627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:36.823 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.821 7fa2ed4a8700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7fa2e80add90 0x7fa2e80627c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:36.824 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.821 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== osd.6 v2:172.21.2.125:6804/36388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fa2e0004720 con 0x7fa2e80add90 2021-02-10T11:59:36.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.829 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 --> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7fa2e8085930 con 0x7fa2e80add90 2021-02-10T11:59:36.831 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.829 7fa2e57fa700 1 -- 172.21.2.117:0/335928488 <== osd.6 v2:172.21.2.125:6804/36388 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7fa2e0004bf0 con 0x7fa2e80add90 2021-02-10T11:59:36.832 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:36.832 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7fa2e80add90 msgr2=0x7fa2e80627c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6804/36388,v1:172.21.2.125:6807/36388] conn(0x7fa2e80add90 0x7fa2e80627c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2c8018700 msgr2=0x7fa2c801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fa2c8018700 0x7fa2c801abc0 crc :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 msgr2=0x7fa2e80979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 --2- 172.21.2.117:0/335928488 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fa2e81a9cd0 0x7fa2e80979d0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7fa2dc00b530 tx=0x7fa2dc00b8d0).stop 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2ed4a8700 1 -- 172.21.2.117:0/335928488 reap_dead start 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 shutdown_connections 2021-02-10T11:59:36.833 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 >> 172.21.2.117:0/335928488 conn(0x7fa2e810df00 msgr2=0x7fa2e81a9440 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:36.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 shutdown_connections 2021-02-10T11:59:36.834 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:36.830 7fa2eee8b700 1 -- 172.21.2.117:0/335928488 wait complete. 2021-02-10T11:59:36.841 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 6 2021-02-10T11:59:37.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.252 7f3b57172700 1 Processor -- start 2021-02-10T11:59:37.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.252 7f3b57172700 1 -- start start 2021-02-10T11:59:37.272 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.252 7f3b57172700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b500e7180 0x7f3b500e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b57172700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 0x7f3b500e8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b57172700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b500e7580 con 0x7f3b501b07f0 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b57172700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b500e5ce0 con 0x7f3b500e7180 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b57172700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500e77f0 con 0x7f3b500e8390 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b4ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b500e7180 0x7f3b500e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b4ffff700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f3b500e7180 0x7f3b500e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3300/0 says I am v2:172.21.2.117:37838/0 (socket says 172.21.2.117:37838) 2021-02-10T11:59:37.273 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b4ffff700 1 -- 172.21.2.117:0/2897507785 learned_addr learned my addr 172.21.2.117:0/2897507785 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.253 7f3b5578f700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 0x7f3b500e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4ffff700 1 -- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 msgr2=0x7f3b500e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4ffff700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 0x7f3b500e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4ffff700 1 -- 172.21.2.117:0/2897507785 >> v1:172.21.2.125:6789/0 conn(0x7f3b501b07f0 legacy=0x7f3b500e4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4ffff700 1 -- 172.21.2.117:0/2897507785 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b500e8db0 con 0x7f3b500e7180 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b5578f700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 0x7f3b500e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-02-10T11:59:37.274 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4ffff700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b500e7180 0x7f3b500e7d70 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f3b40006a80 tx=0x7f3b40009f00).ready entity=mon.1 client_cookie=a7e18d99f2ab75e9 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4f7fe700 1 -- 172.21.2.117:0/2897507785 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4000b5c0 con 0x7f3b500e7180 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4f7fe700 1 -- 172.21.2.117:0/2897507785 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4000b8c0 con 0x7f3b500e7180 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b4f7fe700 1 -- 172.21.2.117:0/2897507785 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b4000bde0 con 0x7f3b500e7180 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b57172700 1 -- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b500e7180 msgr2=0x7f3b500e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.254 7f3b57172700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b500e7180 0x7f3b500e7d70 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7f3b40006a80 tx=0x7f3b40009f00).stop 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 -- 172.21.2.117:0/2897507785 shutdown_connections 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500e8390 0x7f3b500e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 --2- 172.21.2.117:0/2897507785 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b500e7180 0x7f3b500e7d70 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.275 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 -- 172.21.2.117:0/2897507785 >> 172.21.2.117:0/2897507785 conn(0x7f3b500ce500 msgr2=0x7f3b500e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:37.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 -- 172.21.2.117:0/2897507785 shutdown_connections 2021-02-10T11:59:37.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.255 7f3b57172700 1 -- 172.21.2.117:0/2897507785 wait complete. 2021-02-10T11:59:37.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 Processor -- start 2021-02-10T11:59:37.276 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 -- start start 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 0x7f3b500947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b50094e00 0x7f3b5008c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b50106e10 con 0x7f3b5019a820 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5008bf50 con 0x7f3b50094e00 2021-02-10T11:59:37.278 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.270 7f3b57172700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b500c4090 con 0x7f3b500af630 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33458/0 (socket says 172.21.2.117:33458) 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 -- 172.21.2.117:0/339956997 learned_addr learned my addr 172.21.2.117:0/339956997 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b4ffff700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 0x7f3b500947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b5578f700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b50094e00 0x7f3b5008c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 -- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 msgr2=0x7f3b500947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.279 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 0x7f3b500947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 -- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b50094e00 msgr2=0x7f3b5008c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b50094e00 0x7f3b5008c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 -- 172.21.2.117:0/339956997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5008caf0 con 0x7f3b5019a820 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b4ffff700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 0x7f3b500947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.271 7f3b54f8e700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 rx=0x7f3b3800b5b0 tx=0x7f3b3800b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b3800bd40 con 0x7f3b5019a820 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b38006020 con 0x7f3b5019a820 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b57172700 1 -- 172.21.2.117:0/339956997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5008caf0 con 0x7f3b5019a820 2021-02-10T11:59:37.280 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b57172700 1 -- 172.21.2.117:0/339956997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5008d080 con 0x7f3b5019a820 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3b380054a0 con 0x7f3b5019a820 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3b38006200 con 0x7f3b5019a820 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 0x7f3b3401abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.272 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f3b380326f0 con 0x7f3b5019a820 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.273 7f3b4ffff700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 0x7f3b3401abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.281 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.273 7f3b4ffff700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 0x7f3b3401abc0 crc :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.305 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.302 7f3b57172700 1 -- 172.21.2.117:0/339956997 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b50106e10 con 0x7f3b5019a820 2021-02-10T11:59:37.307 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.305 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3b380267b0 con 0x7f3b5019a820 2021-02-10T11:59:37.434 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.432 7f3b57172700 1 -- 172.21.2.117:0/339956997 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mgr", ""]}) v1 -- 0x7f3b5008d080 con 0x7f3b340186c0 2021-02-10T11:59:37.435 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.433 7f3b4d7fa700 1 -- 172.21.2.117:0/339956997 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f3b40000d30 con 0x7f3b340186c0 2021-02-10T11:59:37.435 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 6 to deep-scrub 2021-02-10T11:59:37.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.434 7f3b57172700 1 -- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 msgr2=0x7f3b3401abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.434 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 0x7f3b3401abc0 crc :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 -- 172.21.2.117:0/339956997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 msgr2=0x7f3b500af010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 rx=0x7f3b3800b5b0 tx=0x7f3b3800b950).stop 2021-02-10T11:59:37.437 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 -- 172.21.2.117:0/339956997 shutdown_connections 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f3b50094e00 0x7f3b5008c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f3b500af630 0x7f3b500947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f3b340186c0 0x7f3b3401abc0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 --2- 172.21.2.117:0/339956997 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f3b5019a820 0x7f3b500af010 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 -- 172.21.2.117:0/339956997 >> 172.21.2.117:0/339956997 conn(0x7f3b500953a0 msgr2=0x7f3b5009e970 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 -- 172.21.2.117:0/339956997 shutdown_connections 2021-02-10T11:59:37.438 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.435 7f3b57172700 1 -- 172.21.2.117:0/339956997 wait complete. 2021-02-10T11:59:37.446 INFO:tasks.ceph:Scrubbing osd.7 2021-02-10T11:59:37.446 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2021-02-10T11:59:37.880 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 Processor -- start 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 -- start start 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce800e5240 0x7fce80086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce80084c60 0x7fce80085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce80085930 con 0x7fce800e4e40 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce80085680 con 0x7fce800e5240 2021-02-10T11:59:37.881 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.861 7fce88490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce801b09e0 con 0x7fce80084c60 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7fce800e5240 0x7fce80086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce862ac700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7fce800e4e40 0x7fce801b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33716/0 (socket says 172.21.2.117:33716) 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce862ac700 1 -- 172.21.2.117:0/3154268237 learned_addr learned my addr 172.21.2.117:0/3154268237 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 -- 172.21.2.117:0/3154268237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce80084c60 msgr2=0x7fce80085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 --2- 172.21.2.117:0/3154268237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce80084c60 0x7fce80085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 -- 172.21.2.117:0/3154268237 >> v1:172.21.2.117:6789/0 conn(0x7fce800e4e40 legacy=0x7fce801b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.882 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 -- 172.21.2.117:0/3154268237 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80084840 con 0x7fce800e5240 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce85aab700 1 --2- 172.21.2.117:0/3154268237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce800e5240 0x7fce80086ec0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a80).ready entity=mon.0 client_cookie=48c04bd76e69d98 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.862 7fce852aa700 1 -- 172.21.2.117:0/3154268237 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c00c0c0 con 0x7fce800e5240 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce852aa700 1 -- 172.21.2.117:0/3154268237 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00c340 con 0x7fce800e5240 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce852aa700 1 -- 172.21.2.117:0/3154268237 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c00c820 con 0x7fce800e5240 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 -- 172.21.2.117:0/3154268237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce800e5240 msgr2=0x7fce80086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 --2- 172.21.2.117:0/3154268237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce800e5240 0x7fce80086ec0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 rx=0x7fce7c006650 tx=0x7fce7c009a80).stop 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 -- 172.21.2.117:0/3154268237 shutdown_connections 2021-02-10T11:59:37.883 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 --2- 172.21.2.117:0/3154268237 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce80084c60 0x7fce80085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 --2- 172.21.2.117:0/3154268237 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce800e5240 0x7fce80086ec0 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.863 7fce88490700 1 -- 172.21.2.117:0/3154268237 >> 172.21.2.117:0/3154268237 conn(0x7fce800ce4e0 msgr2=0x7fce800e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.864 7fce88490700 1 -- 172.21.2.117:0/3154268237 shutdown_connections 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.864 7fce88490700 1 -- 172.21.2.117:0/3154268237 wait complete. 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.878 7fce88490700 1 Processor -- start 2021-02-10T11:59:37.884 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.878 7fce88490700 1 -- start start 2021-02-10T11:59:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce8008be80 0x7fce80110b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 0x7fce80189db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.886 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce8018a3d0 0x7fce8008a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800a23d0 con 0x7fce801899b0 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8008b900 con 0x7fce8018a3d0 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce88490700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8018a7d0 con 0x7fce8008be80 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 0x7fce80189db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 0x7fce80189db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33472/0 (socket says 172.21.2.117:33472) 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 -- 172.21.2.117:0/2048855712 learned_addr learned my addr 172.21.2.117:0/2048855712 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce862ac700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce8008be80 0x7fce80110b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.887 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 -- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce8008be80 msgr2=0x7fce80110b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce8008be80 0x7fce80110b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 -- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce8018a3d0 msgr2=0x7fce8008a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7fce8018a3d0 0x7fce8008a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce85aab700 1 -- 172.21.2.117:0/2048855712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8008a8a0 con 0x7fce801899b0 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.879 7fce862ac700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7fce8008be80 0x7fce80110b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce85aab700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 0x7fce80189db0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7fce7c009630 tx=0x7fce7c00b980).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c00bd70 con 0x7fce801899b0 2021-02-10T11:59:37.888 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c000960 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce88490700 1 -- 172.21.2.117:0/2048855712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8008a8a0 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce88490700 1 -- 172.21.2.117:0/2048855712 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8008ae30 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.880 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7fce7c0054d0 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.881 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7fce7c00a000 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.881 7fce72ffd700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce68018730 0x7fce6801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.881 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7fce7c0328d0 con 0x7fce801899b0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.881 7fce862ac700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce68018730 0x7fce6801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.889 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.881 7fce862ac700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce68018730 0x7fce6801abc0 crc :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.911 7fce88490700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7fce800627c0 0x7fce80062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:37.913 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.911 7fce88490700 1 -- 172.21.2.117:0/2048855712 --> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce8008ae30 con 0x7fce800627c0 2021-02-10T11:59:37.914 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.912 7fce86aad700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7fce800627c0 0x7fce80062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:37.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.913 7fce86aad700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7fce800627c0 0x7fce80062bc0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:37.915 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.914 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== osd.7 v2:172.21.2.125:6802/36387 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+5151 (crc 0 0 0) 0x7fce78000f40 con 0x7fce800627c0 2021-02-10T11:59:37.922 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.920 7fce88490700 1 -- 172.21.2.117:0/2048855712 --> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] -- command(tid 2: {"prefix": "config set", "key": "osd_debug_deep_scrub_sleep", "value": "0"}) v1 -- 0x7fce8008b6c0 con 0x7fce800627c0 2021-02-10T11:59:37.923 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.921 7fce72ffd700 1 -- 172.21.2.117:0/2048855712 <== osd.7 v2:172.21.2.125:6802/36387 2 ==== command_reply(tid 2: 0 Set osd_debug_deep_scrub_sleep to 0) v1 ==== 43+0+0 (crc 0 0 0) 0x7fce78000f40 con 0x7fce800627c0 2021-02-10T11:59:37.924 INFO:teuthology.orchestra.run.gibba017.stderr:Set osd_debug_deep_scrub_sleep to 0 2021-02-10T11:59:37.924 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.921 7fce88490700 1 -- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7fce800627c0 msgr2=0x7fce80062bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.921 7fce88490700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6802/36387,v1:172.21.2.125:6805/36387] conn(0x7fce800627c0 0x7fce80062bc0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce68018730 msgr2=0x7fce6801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7fce68018730 0x7fce6801abc0 crc :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 msgr2=0x7fce80189db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 --2- 172.21.2.117:0/2048855712 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7fce801899b0 0x7fce80189db0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 rx=0x7fce7c009630 tx=0x7fce7c00b980).stop 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce86aad700 1 -- 172.21.2.117:0/2048855712 reap_dead start 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 shutdown_connections 2021-02-10T11:59:37.925 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 >> 172.21.2.117:0/2048855712 conn(0x7fce8019c310 msgr2=0x7fce8010c520 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:37.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 shutdown_connections 2021-02-10T11:59:37.926 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:37.922 7fce88490700 1 -- 172.21.2.117:0/2048855712 wait complete. 2021-02-10T11:59:37.932 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 7 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.343 7f6dc7219700 1 Processor -- start 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 -- start start 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6dc00e4e40 0x7f6dc01b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0085450 0x7f6dc0086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc0085930 con 0x7f6dc00e5240 2021-02-10T11:59:38.363 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc00e5cc0 con 0x7f6dc00e4e40 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc7219700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0084840 con 0x7f6dc0085450 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc5035700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f6dc00e4e40 0x7f6dc01b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc4834700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f6dc00e5240 0x7f6dc0084c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33732/0 (socket says 172.21.2.117:33732) 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.344 7f6dc4834700 1 -- 172.21.2.117:0/2284352736 learned_addr learned my addr 172.21.2.117:0/2284352736 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc5035700 1 -- 172.21.2.117:0/2284352736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0085450 msgr2=0x7f6dc0086ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc5035700 1 --2- 172.21.2.117:0/2284352736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0085450 0x7f6dc0086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc5035700 1 -- 172.21.2.117:0/2284352736 >> v1:172.21.2.117:6789/0 conn(0x7f6dc00e5240 legacy=0x7f6dc0084c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.364 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc5035700 1 -- 172.21.2.117:0/2284352736 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc01b0810 con 0x7f6dc00e4e40 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc5035700 1 --2- 172.21.2.117:0/2284352736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc00e4e40 0x7f6dc01b01f0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 rx=0x7f6dbc006650 tx=0x7f6dbc009a80).ready entity=mon.0 client_cookie=be561a92147f7647 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6db3fff700 1 -- 172.21.2.117:0/2284352736 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dbc0009b0 con 0x7f6dc00e4e40 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6db3fff700 1 -- 172.21.2.117:0/2284352736 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc00c0c0 con 0x7f6dc00e4e40 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6db3fff700 1 -- 172.21.2.117:0/2284352736 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6dbc00c5e0 con 0x7f6dc00e4e40 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc7219700 1 -- 172.21.2.117:0/2284352736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc00e4e40 msgr2=0x7f6dc01b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.345 7f6dc7219700 1 --2- 172.21.2.117:0/2284352736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc00e4e40 0x7f6dc01b01f0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 rx=0x7f6dbc006650 tx=0x7f6dbc009a80).stop 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 -- 172.21.2.117:0/2284352736 shutdown_connections 2021-02-10T11:59:38.365 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 --2- 172.21.2.117:0/2284352736 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0085450 0x7f6dc0086ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:38.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 --2- 172.21.2.117:0/2284352736 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc00e4e40 0x7f6dc01b01f0 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 -- 172.21.2.117:0/2284352736 >> 172.21.2.117:0/2284352736 conn(0x7f6dc00ce4e0 msgr2=0x7f6dc00e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:38.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 -- 172.21.2.117:0/2284352736 shutdown_connections 2021-02-10T11:59:38.366 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.346 7f6dc7219700 1 -- 172.21.2.117:0/2284352736 wait complete. 2021-02-10T11:59:38.367 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 Processor -- start 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 -- start start 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0092d20 0x7f6dc019d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dc019d920 0x7f6dc010dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0093120 con 0x7f6dc010e200 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc010d2f0 con 0x7f6dc019d920 2021-02-10T11:59:38.368 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.361 7f6dc7219700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00a4060 con 0x7f6dc0092d20 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33488/0 (socket says 172.21.2.117:33488) 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 -- 172.21.2.117:0/3395547543 learned_addr learned my addr 172.21.2.117:0/3395547543 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5035700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0092d20 0x7f6dc019d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc4834700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dc019d920 0x7f6dc010dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 -- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0092d20 msgr2=0x7f6dc019d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0092d20 0x7f6dc019d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 -- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dc019d920 msgr2=0x7f6dc010dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.369 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dc019d920 0x7f6dc010dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 -- 172.21.2.117:0/3395547543 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc010ec20 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6dc5836700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 rx=0x7f6db8009980 tx=0x7f6db800bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.362 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6db8003020 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db8005a80 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc010ec20 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f6db8006030 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc010ec20 con 0x7f6dc010e200 2021-02-10T11:59:38.370 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f6db8005c60 con 0x7f6dc010e200 2021-02-10T11:59:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6db1ffb700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 0x7f6da001abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.363 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f6db8032cb0 con 0x7f6dc010e200 2021-02-10T11:59:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.364 7f6dc5035700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 0x7f6da001abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.371 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.364 7f6dc5035700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 0x7f6da001abc0 crc :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:38.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.394 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc0093120 con 0x7f6dc010e200 2021-02-10T11:59:38.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.396 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f6db8005870 con 0x7f6dc010e200 2021-02-10T11:59:38.529 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.527 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mgr", ""]}) v1 -- 0x7f6dc0093120 con 0x7f6da0018730 2021-02-10T11:59:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.528 7f6db1ffb700 1 -- 172.21.2.117:0/3395547543 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (crc 0 0 0) 0x7f6dbc006420 con 0x7f6da0018730 2021-02-10T11:59:38.530 INFO:teuthology.orchestra.run.gibba017.stderr:instructed osd(s) 7 to deep-scrub 2021-02-10T11:59:38.531 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.529 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 msgr2=0x7f6da001abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.529 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 0x7f6da001abc0 crc :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.529 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 msgr2=0x7f6dc010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.532 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.529 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 rx=0x7f6db8009980 tx=0x7f6db800bcd0).stop 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 shutdown_connections 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f6da0018730 0x7f6da001abc0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f6dc010e200 0x7f6dc010e600 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f6dc019d920 0x7f6dc010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 --2- 172.21.2.117:0/3395547543 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f6dc0092d20 0x7f6dc019d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 >> 172.21.2.117:0/3395547543 conn(0x7f6dc019f650 msgr2=0x7f6dc019ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 shutdown_connections 2021-02-10T11:59:38.533 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.530 7f6dc7219700 1 -- 172.21.2.117:0/3395547543 wait complete. 2021-02-10T11:59:38.540 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:59:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 Processor -- start 2021-02-10T11:59:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 -- start start 2021-02-10T11:59:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df00d18e0 0x7f4df00ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f4df00eb000 0x7f4df00eb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.972 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4df00ee3e0 0x7f4df00ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 -- --> v2:172.21.2.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df00ede60 con 0x7f4df00ee3e0 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df00eba20 con 0x7f4df00eb000 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.953 7f4df71e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00d1ce0 con 0x7f4df00d18e0 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4df00ee3e0 0x7f4df00ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 --2- >> v2:172.21.2.117:3300/0 conn(0x7f4df00ee3e0 0x7f4df00ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33496/0 (socket says 172.21.2.117:33496) 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 -- 172.21.2.117:0/740256609 learned_addr learned my addr 172.21.2.117:0/740256609 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 -- 172.21.2.117:0/740256609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df00d18e0 msgr2=0x7f4df00ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5003700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df00d18e0 0x7f4df00ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.973 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df4802700 1 --2- 172.21.2.117:0/740256609 >> v2:172.21.2.125:3300/0 conn(0x7f4df00eb000 0x7f4df00eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df00d18e0 0x7f4df00ea9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 -- 172.21.2.117:0/740256609 >> v2:172.21.2.125:3300/0 conn(0x7f4df00eb000 msgr2=0x7f4df00eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 --2- 172.21.2.117:0/740256609 >> v2:172.21.2.125:3300/0 conn(0x7f4df00eb000 0x7f4df00eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 -- 172.21.2.117:0/740256609 --> v2:172.21.2.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df00eee00 con 0x7f4df00ee3e0 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.954 7f4df5804700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df00ee3e0 0x7f4df00ee7e0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f4de8006f00 tx=0x7f4de8009f10).ready entity=mon.0 client_cookie=7182d99cc52ffb10 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4de7fff700 1 -- 172.21.2.117:0/740256609 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de800b3b0 con 0x7f4df00ee3e0 2021-02-10T11:59:38.974 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4de7fff700 1 -- 172.21.2.117:0/740256609 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de800b730 con 0x7f4df00ee3e0 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4de7fff700 1 -- 172.21.2.117:0/740256609 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de800bc50 con 0x7f4df00ee3e0 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 -- 172.21.2.117:0/740256609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df00ee3e0 msgr2=0x7f4df00ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df00ee3e0 0x7f4df00ee7e0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 rx=0x7f4de8006f00 tx=0x7f4de8009f10).stop 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 -- 172.21.2.117:0/740256609 shutdown_connections 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 --2- 172.21.2.117:0/740256609 >> v2:172.21.2.125:3300/0 conn(0x7f4df00eb000 0x7f4df00eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df00ee3e0 0x7f4df00ee7e0 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 --2- 172.21.2.117:0/740256609 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df00d18e0 0x7f4df00ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:38.975 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 -- 172.21.2.117:0/740256609 >> 172.21.2.117:0/740256609 conn(0x7f4df00ced60 msgr2=0x7f4df00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:38.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 -- 172.21.2.117:0/740256609 shutdown_connections 2021-02-10T11:59:38.976 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.955 7f4df71e7700 1 -- 172.21.2.117:0/740256609 wait complete. 2021-02-10T11:59:38.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 Processor -- start 2021-02-10T11:59:38.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 -- start start 2021-02-10T11:59:38.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.977 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df008b760 0x7f4df008bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4df009e3f0 0x7f4df009e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df008e380 con 0x7f4df008b260 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df008c180 con 0x7f4df009e3f0 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df71e7700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df008ad30 con 0x7f4df008b760 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.970 7f4df5003700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33500/0 (socket says 172.21.2.117:33500) 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 -- 172.21.2.117:0/2097130921 learned_addr learned my addr 172.21.2.117:0/2097130921 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:38.978 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 -- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df008b760 msgr2=0x7f4df008bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df008b760 0x7f4df008bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 -- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4df009e3f0 msgr2=0x7f4df009e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4df009e3f0 0x7f4df009e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 -- 172.21.2.117:0/2097130921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df009ee10 con 0x7f4df008b260 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df5003700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 rx=0x7f4de0006f10 tx=0x7f4de000b990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de00009b0 con 0x7f4df008b260 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de0006370 con 0x7f4df008b260 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df009ee10 con 0x7f4df008b260 2021-02-10T11:59:38.979 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df009f3a0 con 0x7f4df008b260 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.971 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f4de0002030 con 0x7f4df008b260 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.972 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f4de00052e0 con 0x7f4df008b260 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.972 7f4de5ffb700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 0x7f4dd001abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.972 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f4de0032790 con 0x7f4df008b260 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.972 7f4df4802700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 0x7f4dd001abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:38.980 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:38.973 7f4df4802700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 0x7f4dd001abd0 crc :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:39.006 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.003 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0087290 con 0x7f4df008b260 2021-02-10T11:59:39.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.006 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f4de0000960 con 0x7f4df008b260 2021-02-10T11:59:39.140 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.138 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f4df009f3a0 con 0x7f4dd00186d0 2021-02-10T11:59:39.144 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.142 7f4de5ffb700 1 -- 172.21.2.117:0/2097130921 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101197 (crc 0 0 0) 0x7f4dec000960 con 0x7f4dd00186d0 2021-02-10T11:59:39.149 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:59:39.149 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T11:59:39.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 msgr2=0x7f4dd001abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:39.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 0x7f4dd001abd0 crc :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:39.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 msgr2=0x7f4df008a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:39.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 rx=0x7f4de0006f10 tx=0x7f4de000b990).stop 2021-02-10T11:59:39.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 shutdown_connections 2021-02-10T11:59:39.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f4df009e3f0 0x7f4df009e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:39.150 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f4df008b760 0x7f4df008bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:39.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f4dd00186d0 0x7f4dd001abd0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:39.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 --2- 172.21.2.117:0/2097130921 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f4df008b260 0x7f4df008a710 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:39.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 >> 172.21.2.117:0/2097130921 conn(0x7f4df008e740 msgr2=0x7f4df01b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:39.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 shutdown_connections 2021-02-10T11:59:39.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:39.147 7f4df71e7700 1 -- 172.21.2.117:0/2097130921 wait complete. 2021-02-10T11:59:39.158 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":1321,"stamp":"2021-02-10 11:59:37.351607","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135155186,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":436,"num_write_kb":132089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":418,"ondisk_log_size":418,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672508,"kb_used_data":283832,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302212,"statfs":{"total":773094113280,"available":764213465088,"internally_reserved":8589934592,"allocated":290643968,"data_stored":279529169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":89672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003382"},"pg_stats":[{"pgid":"43.6","version":"196'22","reported_seq":"40","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.583889","last_change":"2021-02-10 11:58:04.419386","last_active":"2021-02-10 11:59:18.583889","last_peered":"2021-02-10 11:59:18.583889","last_clean":"2021-02-10 11:59:18.583889","last_became_active":"2021-02-10 11:58:04.419103","last_became_peered":"2021-02-10 11:58:04.419103","last_unstale":"2021-02-10 11:59:18.583889","last_undegraded":"2021-02-10 11:59:18.583889","last_fullsized":"2021-02-10 11:59:18.583889","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"196'8","reported_seq":"19","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:58:06.434973","last_change":"2021-02-10 11:58:02.783590","last_active":"2021-02-10 11:58:06.434973","last_peered":"2021-02-10 11:58:06.434973","last_clean":"2021-02-10 11:58:06.434973","last_became_active":"2021-02-10 11:58:02.783318","last_became_peered":"2021-02-10 11:58:02.783318","last_unstale":"2021-02-10 11:58:06.434973","last_undegraded":"2021-02-10 11:58:06.434973","last_fullsized":"2021-02-10 11:58:06.434973","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"196'14","reported_seq":"30","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.590538","last_change":"2021-02-10 11:58:04.419632","last_active":"2021-02-10 11:59:18.590538","last_peered":"2021-02-10 11:59:18.590538","last_clean":"2021-02-10 11:59:18.590538","last_became_active":"2021-02-10 11:58:04.419410","last_became_peered":"2021-02-10 11:58:04.419410","last_unstale":"2021-02-10 11:59:18.590538","last_undegraded":"2021-02-10 11:59:18.590538","last_fullsized":"2021-02-10 11:59:18.590538","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"196'17","reported_seq":"28","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:23.776829","last_change":"2021-02-10 11:58:02.783878","last_active":"2021-02-10 11:59:23.776829","last_peered":"2021-02-10 11:59:23.776829","last_clean":"2021-02-10 11:59:23.776829","last_became_active":"2021-02-10 11:58:02.783694","last_became_peered":"2021-02-10 11:58:02.783694","last_unstale":"2021-02-10 11:59:23.776829","last_undegraded":"2021-02-10 11:59:23.776829","last_fullsized":"2021-02-10 11:59:23.776829","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"196'23","reported_seq":"41","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.628470","last_change":"2021-02-10 11:58:04.421529","last_active":"2021-02-10 11:59:18.628470","last_peered":"2021-02-10 11:59:18.628470","last_clean":"2021-02-10 11:59:18.628470","last_became_active":"2021-02-10 11:58:04.421304","last_became_peered":"2021-02-10 11:58:04.421304","last_unstale":"2021-02-10 11:59:18.628470","last_undegraded":"2021-02-10 11:59:18.628470","last_fullsized":"2021-02-10 11:59:18.628470","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":21,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"196'223","reported_seq":"234","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.776817","last_change":"2021-02-10 11:58:02.784658","last_active":"2021-02-10 11:59:33.776817","last_peered":"2021-02-10 11:59:33.776817","last_clean":"2021-02-10 11:59:33.776817","last_became_active":"2021-02-10 11:58:02.784457","last_became_peered":"2021-02-10 11:58:02.784457","last_unstale":"2021-02-10 11:59:33.776817","last_undegraded":"2021-02-10 11:59:33.776817","last_fullsized":"2021-02-10 11:59:33.776817","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":223,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":936170,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":223,"num_write_kb":982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"196'20","reported_seq":"37","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.568376","last_change":"2021-02-10 11:58:04.421664","last_active":"2021-02-10 11:59:18.568376","last_peered":"2021-02-10 11:59:18.568376","last_clean":"2021-02-10 11:59:18.568376","last_became_active":"2021-02-10 11:58:04.421381","last_became_peered":"2021-02-10 11:58:04.421381","last_unstale":"2021-02-10 11:59:18.568376","last_undegraded":"2021-02-10 11:59:18.568376","last_fullsized":"2021-02-10 11:59:18.568376","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"196'6","reported_seq":"17","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:58:06.434322","last_change":"2021-02-10 11:58:03.787818","last_active":"2021-02-10 11:58:06.434322","last_peered":"2021-02-10 11:58:06.434322","last_clean":"2021-02-10 11:58:06.434322","last_became_active":"2021-02-10 11:58:03.787526","last_became_peered":"2021-02-10 11:58:03.787526","last_unstale":"2021-02-10 11:58:06.434322","last_undegraded":"2021-02-10 11:58:06.434322","last_fullsized":"2021-02-10 11:58:06.434322","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"196'25","reported_seq":"42","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.633888","last_change":"2021-02-10 11:58:04.419232","last_active":"2021-02-10 11:59:18.633888","last_peered":"2021-02-10 11:59:18.633888","last_clean":"2021-02-10 11:59:18.633888","last_became_active":"2021-02-10 11:58:04.419046","last_became_peered":"2021-02-10 11:58:04.419046","last_unstale":"2021-02-10 11:59:18.633888","last_undegraded":"2021-02-10 11:59:18.633888","last_fullsized":"2021-02-10 11:59:18.633888","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"195'1","reported_seq":"15","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.706570","last_change":"2021-02-10 11:59:33.706603","last_active":"2021-02-10 11:59:33.706570","last_peered":"2021-02-10 11:59:33.706570","last_clean":"2021-02-10 11:59:33.706570","last_became_active":"2021-02-10 11:58:03.788701","last_became_peered":"2021-02-10 11:58:03.788701","last_unstale":"2021-02-10 11:59:33.706570","last_undegraded":"2021-02-10 11:59:33.706570","last_fullsized":"2021-02-10 11:59:33.706570","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'1","last_scrub_stamp":"2021-02-10 11:59:33.706566","last_deep_scrub":"195'1","last_deep_scrub_stamp":"2021-02-10 11:59:33.706566","last_clean_scrub_stamp":"2021-02-10 11:59:33.706566","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"196'19","reported_seq":"38","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.542369","last_change":"2021-02-10 11:58:04.419268","last_active":"2021-02-10 11:59:18.542369","last_peered":"2021-02-10 11:59:18.542369","last_clean":"2021-02-10 11:59:18.542369","last_became_active":"2021-02-10 11:58:04.419010","last_became_peered":"2021-02-10 11:58:04.419010","last_unstale":"2021-02-10 11:59:18.542369","last_undegraded":"2021-02-10 11:59:18.542369","last_fullsized":"2021-02-10 11:59:18.542369","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"196'2","reported_seq":"14","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:19.429309","last_change":"2021-02-10 11:58:03.789999","last_active":"2021-02-10 11:59:19.429309","last_peered":"2021-02-10 11:59:19.429309","last_clean":"2021-02-10 11:59:19.429309","last_became_active":"2021-02-10 11:58:03.789795","last_became_peered":"2021-02-10 11:58:03.789795","last_unstale":"2021-02-10 11:59:19.429309","last_undegraded":"2021-02-10 11:59:19.429309","last_fullsized":"2021-02-10 11:59:19.429309","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"196'11","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.607248","last_change":"2021-02-10 11:58:04.419297","last_active":"2021-02-10 11:59:18.607248","last_peered":"2021-02-10 11:59:18.607248","last_clean":"2021-02-10 11:59:18.607248","last_became_active":"2021-02-10 11:58:04.418997","last_became_peered":"2021-02-10 11:58:04.418997","last_unstale":"2021-02-10 11:59:18.607248","last_undegraded":"2021-02-10 11:59:18.607248","last_fullsized":"2021-02-10 11:59:18.607248","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"195'8","reported_seq":"19","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.421977","last_change":"2021-02-10 11:58:02.783688","last_active":"2021-02-10 11:58:04.421977","last_peered":"2021-02-10 11:58:04.421977","last_clean":"2021-02-10 11:58:04.421977","last_became_active":"2021-02-10 11:58:02.783528","last_became_peered":"2021-02-10 11:58:02.783528","last_unstale":"2021-02-10 11:58:04.421977","last_undegraded":"2021-02-10 11:58:04.421977","last_fullsized":"2021-02-10 11:58:04.421977","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"179","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.416354","last_change":"2021-02-10 11:22:24.884900","last_active":"2021-02-10 11:58:04.416354","last_peered":"2021-02-10 11:58:04.416354","last_clean":"2021-02-10 11:58:04.416354","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:58:04.416354","last_undegraded":"2021-02-10 11:58:04.416354","last_fullsized":"2021-02-10 11:58:04.416354","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"179","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.416410","last_change":"2021-02-10 11:22:24.885206","last_active":"2021-02-10 11:58:04.416410","last_peered":"2021-02-10 11:58:04.416410","last_clean":"2021-02-10 11:58:04.416410","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:58:04.416410","last_undegraded":"2021-02-10 11:58:04.416410","last_fullsized":"2021-02-10 11:58:04.416410","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"183","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.415349","last_change":"2021-02-10 11:22:24.885818","last_active":"2021-02-10 11:58:04.415349","last_peered":"2021-02-10 11:58:04.415349","last_clean":"2021-02-10 11:58:04.415349","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:58:04.415349","last_undegraded":"2021-02-10 11:58:04.415349","last_fullsized":"2021-02-10 11:58:04.415349","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.417604","last_change":"2021-02-10 11:22:24.886002","last_active":"2021-02-10 11:58:04.417604","last_peered":"2021-02-10 11:58:04.417604","last_clean":"2021-02-10 11:58:04.417604","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:58:04.417604","last_undegraded":"2021-02-10 11:58:04.417604","last_fullsized":"2021-02-10 11:58:04.417604","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.417584","last_change":"2021-02-10 11:22:24.885674","last_active":"2021-02-10 11:58:04.417584","last_peered":"2021-02-10 11:58:04.417584","last_clean":"2021-02-10 11:58:04.417584","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:58:04.417584","last_undegraded":"2021-02-10 11:58:04.417584","last_fullsized":"2021-02-10 11:58:04.417584","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"196'10","reported_seq":"25","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.553071","last_change":"2021-02-10 11:58:04.419612","last_active":"2021-02-10 11:59:18.553071","last_peered":"2021-02-10 11:59:18.553071","last_clean":"2021-02-10 11:59:18.553071","last_became_active":"2021-02-10 11:58:04.419401","last_became_peered":"2021-02-10 11:58:04.419401","last_unstale":"2021-02-10 11:59:18.553071","last_undegraded":"2021-02-10 11:59:18.553071","last_fullsized":"2021-02-10 11:59:18.553071","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"196'9","reported_seq":"20","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:19.425849","last_change":"2021-02-10 11:58:02.783468","last_active":"2021-02-10 11:59:19.425849","last_peered":"2021-02-10 11:59:19.425849","last_clean":"2021-02-10 11:59:19.425849","last_became_active":"2021-02-10 11:58:02.783300","last_became_peered":"2021-02-10 11:58:02.783300","last_unstale":"2021-02-10 11:59:19.425849","last_undegraded":"2021-02-10 11:59:19.425849","last_fullsized":"2021-02-10 11:59:19.425849","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:02.773071","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:02.773071","last_clean_scrub_stamp":"2021-02-10 11:58:02.773071","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"188","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.415654","last_change":"2021-02-10 11:22:24.885203","last_active":"2021-02-10 11:58:04.415654","last_peered":"2021-02-10 11:58:04.415654","last_clean":"2021-02-10 11:58:04.415654","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:58:04.415654","last_undegraded":"2021-02-10 11:58:04.415654","last_fullsized":"2021-02-10 11:58:04.415654","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"177","reported_epoch":"195","state":"active+clean","last_fresh":"2021-02-10 11:58:04.418155","last_change":"2021-02-10 11:22:24.885698","last_active":"2021-02-10 11:58:04.418155","last_peered":"2021-02-10 11:58:04.418155","last_clean":"2021-02-10 11:58:04.418155","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:58:04.418155","last_undegraded":"2021-02-10 11:58:04.418155","last_fullsized":"2021-02-10 11:58:04.418155","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:22:23.870088","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:22:23.870088","last_clean_scrub_stamp":"2021-02-10 11:22:23.870088","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"185","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.744224","last_change":"2021-02-10 11:59:33.744256","last_active":"2021-02-10 11:59:33.744224","last_peered":"2021-02-10 11:59:33.744224","last_clean":"2021-02-10 11:59:33.744224","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:59:33.744224","last_undegraded":"2021-02-10 11:59:33.744224","last_fullsized":"2021-02-10 11:59:33.744224","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:33.744222","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:33.744222","last_clean_scrub_stamp":"2021-02-10 11:59:33.744222","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":140,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":937031,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296,"num_write_kb":1010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1941504,"data_stored":1874062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575299,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087940,"kb_used_data":39356,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283900,"statfs":{"total":96636764160,"available":95522713600,"internally_reserved":1073741824,"allocated":40300544,"data_stored":38903058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92100000000000004,"15min":0.96099999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.33100000000000002},"max":{"1min":1.208,"5min":1.4219999999999999,"15min":1.534},"last":1.1839999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.61799999999999999,"5min":0.499,"15min":0.35899999999999999},"max":{"1min":1.212,"5min":1.304,"15min":1.498},"last":1.1619999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.81799999999999995,"15min":0.84399999999999997},"min":{"1min":0.443,"5min":0.38700000000000001,"15min":0.372},"max":{"1min":1.153,"5min":1.288,"15min":1.3400000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.84599999999999997,"15min":0.86299999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.35799999999999998},"max":{"1min":1.1140000000000001,"5min":1.411,"15min":1.4970000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.82599999999999996,"15min":0.86799999999999999},"min":{"1min":0.45900000000000002,"5min":0.41899999999999998,"15min":0.39800000000000002},"max":{"1min":1.1020000000000001,"5min":1.1299999999999999,"15min":1.3680000000000001},"last":1.129},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83299999999999996,"15min":0.86399999999999999},"min":{"1min":0.56599999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.121,"5min":1.26,"15min":1.288},"last":0.92300000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.871,"15min":0.873},"min":{"1min":0.5,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0740000000000001,"5min":1.2370000000000001,"15min":1.357},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.876,"15min":0.877},"min":{"1min":0.58799999999999997,"5min":0.48099999999999998,"15min":0.47899999999999998},"max":{"1min":1.238,"5min":1.266,"15min":1.278},"last":1.224}]},{"osd":4,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68600000000000005,"15min":0.67800000000000005},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.0269999999999999,"5min":1.2629999999999999,"15min":1.286},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.72199999999999998,"15min":0.69799999999999995},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.40000000000000002},"max":{"1min":0.96899999999999997,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.81899999999999995}]},{"osd":5,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.71699999999999997,"15min":0.70199999999999996},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94099999999999995,"5min":1.135,"15min":1.1419999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.754,"5min":0.77700000000000002,"15min":0.77100000000000002},"min":{"1min":0.57799999999999996,"5min":0.44,"15min":0.44},"max":{"1min":0.90100000000000002,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":1.1020000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:58:42 2021","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83899999999999997,"15min":0.84299999999999997},"min":{"1min":0.56399999999999995,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.997,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.58699999999999997,"5min":0.54000000000000004,"15min":0.433},"max":{"1min":1.02,"5min":1.167,"15min":1.3720000000000001},"last":1.073}]}]},{"osd":6,"up_from":13,"seq":55834575299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070408,"kb_used_data":21824,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301432,"statfs":{"total":96636764160,"available":95540666368,"internally_reserved":1073741824,"allocated":22347776,"data_stored":20955814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.86399999999999999,"15min":0.89800000000000002},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.5680000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91200000000000003,"15min":0.92700000000000005},"min":{"1min":0.52700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.82699999999999996}]},{"osd":1,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.499,"5min":0.39600000000000002,"15min":0.39000000000000001},"max":{"1min":1.1399999999999999,"5min":1.2390000000000001,"15min":1.363},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.79400000000000004,"15min":0.82799999999999996},"min":{"1min":0.42199999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.1659999999999999,"5min":1.2789999999999999,"15min":1.45},"last":0.65400000000000003}]},{"osd":2,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79800000000000004,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.41299999999999998,"15min":0.40300000000000002},"max":{"1min":1.2170000000000001,"5min":1.276,"15min":1.276},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80200000000000005,"15min":0.81699999999999995},"min":{"1min":0.52800000000000002,"5min":0.505,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.84299999999999997,"15min":0.85999999999999999},"min":{"1min":0.48499999999999999,"5min":0.442,"15min":0.441},"max":{"1min":1.2350000000000001,"5min":1.2889999999999999,"15min":1.3859999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.84499999999999997,"15min":0.84999999999999998},"min":{"1min":0.48399999999999999,"5min":0.443,"15min":0.379},"max":{"1min":1.2030000000000001,"5min":1.244,"15min":1.276},"last":0.64400000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72099999999999997,"15min":0.71699999999999997},"min":{"1min":0.47299999999999998,"5min":0.45200000000000001,"15min":0.442},"max":{"1min":1.0349999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.54800000000000004,"5min":0.40100000000000002,"15min":0.36499999999999999},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.51400000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.55800000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94799999999999995,"5min":1.0049999999999999,"15min":1.153},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79600000000000004,"15min":0.79400000000000004},"min":{"1min":0.56399999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.288,"5min":1.3660000000000001,"15min":1.6379999999999999},"last":0.748}]},{"osd":7,"last update":"Wed Feb 10 11:58:40 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.65800000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0760000000000001,"5min":1.256,"15min":1.256},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79400000000000004,"15min":0.81299999999999994},"min":{"1min":0.60199999999999998,"5min":0.51200000000000001,"15min":0.499},"max":{"1min":1.1279999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.80600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083832,"kb_used_data":35248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93288008,"statfs":{"total":96636764160,"available":95526920192,"internally_reserved":1073741824,"allocated":36093952,"data_stored":34707957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.40699999999999997,"5min":0.38700000000000001,"15min":0.36299999999999999},"max":{"1min":1.224,"5min":1.363,"15min":1.365},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.432,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2749999999999999,"5min":1.399,"15min":1.399},"last":0.52500000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.55400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.2430000000000001,"5min":1.3819999999999999,"15min":1.3979999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.3},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.59799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.181,"5min":1.2330000000000001,"15min":1.2809999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.879,"15min":0.878},"min":{"1min":0.63,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.274,"15min":1.3500000000000001},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.878,"15min":0.878},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3660000000000001},"last":0.627},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.2669999999999999},"last":0.50600000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73399999999999999,"15min":0.76600000000000001},"min":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.44800000000000001},"max":{"1min":1.101,"5min":1.103,"15min":1.343},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.55200000000000005,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.1279999999999999,"5min":1.292,"15min":1.292},"last":0.80400000000000005}]},{"osd":6,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.75900000000000001},"min":{"1min":0.60399999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.278,"5min":1.278,"15min":1.3049999999999999},"last":0.754},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.67200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.206,"5min":1.2230000000000001,"15min":1.302},"last":0.78800000000000003}]},{"osd":7,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.085,"5min":1.2809999999999999,"15min":1.3220000000000001},"last":0.745},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.77700000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96599999999999997,"5min":1.147,"15min":1.1870000000000001},"last":0.84999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074508,"kb_used_data":25924,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297332,"statfs":{"total":96636764160,"available":95536467968,"internally_reserved":1073741824,"allocated":26546176,"data_stored":25158207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91800000000000004,"15min":0.93100000000000005},"min":{"1min":0.51300000000000001,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":1.3600000000000001,"5min":1.407,"15min":1.4099999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.046,"5min":0.94299999999999995,"15min":0.95399999999999996},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":1.3089999999999999,"5min":1.335,"15min":1.3879999999999999},"last":1.238}]},{"osd":1,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.84899999999999998,"15min":0.84999999999999998},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.372},"max":{"1min":1.222,"5min":1.2529999999999999,"15min":1.341},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.81499999999999995,"15min":0.82999999999999996},"min":{"1min":0.46200000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":1.119,"5min":1.181,"15min":1.2869999999999999},"last":1.0720000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.83899999999999997,"15min":0.85399999999999998},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.432},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4570000000000001},"last":0.98499999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83699999999999997,"15min":0.85699999999999998},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2569999999999999,"5min":1.321,"15min":1.3220000000000001},"last":1.0209999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.318},"max":{"1min":0.96399999999999997,"5min":1.0900000000000001,"15min":1.141},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96099999999999997,"5min":1.042,"15min":1.1279999999999999},"last":0.79500000000000004}]},{"osd":6,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.438,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.113,"15min":1.204},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.0720000000000001,"15min":1.2729999999999999},"last":0.69299999999999995}]},{"osd":7,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.44800000000000001,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.496,"5min":0.432,"15min":0.432},"max":{"1min":1.0249999999999999,"5min":1.165,"15min":1.321},"last":0.96099999999999997}]}]},{"osd":0,"up_from":13,"seq":55834575299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061300,"kb_used_data":12716,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310540,"statfs":{"total":96636764160,"available":95549992960,"internally_reserved":1073741824,"allocated":13021184,"data_stored":11639159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.78500000000000003,"15min":0.94399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.496,"15min":0.502},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.66200000000000003,"5min":0.82999999999999996,"15min":1.319},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.85099999999999998,"15min":0.91800000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56999999999999995},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.35899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":1.1599999999999999},"last":0.54100000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.52000000000000002},"min":{"1min":0.378,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.89600000000000002},"last":0.503},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.63300000000000001,"15min":0.63200000000000001},"min":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.41899999999999998},"max":{"1min":0.92000000000000004,"5min":0.96799999999999997,"15min":1.077},"last":0.51800000000000002}]},{"osd":4,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.97399999999999998,"15min":0.96199999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.43099999999999999},"max":{"1min":1.212,"5min":1.3440000000000001,"15min":1.4039999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.96799999999999997,"15min":0.96099999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.52100000000000002},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.94399999999999995,"15min":0.94299999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.48099999999999998},"max":{"1min":1.4690000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":0.879},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96699999999999997,"15min":0.97899999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.48299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.512},"last":0.96699999999999997}]},{"osd":6,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0649999999999999,"15min":1.056},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.48999999999999999},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.014},{"interface":"front","average":{"1min":1.093,"5min":1.077,"15min":1.079},"min":{"1min":0.69999999999999996,"5min":0.67400000000000004,"15min":0.53400000000000003},"max":{"1min":1.7490000000000001,"5min":1.7889999999999999,"15min":1.8100000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Wed Feb 10 11:58:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":0.996,"15min":1.0009999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.54900000000000004},"max":{"1min":1.615,"5min":1.615,"15min":1.909},"last":1.034},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.98499999999999999,"15min":0.99099999999999999},"min":{"1min":0.66700000000000004,"5min":0.63600000000000001,"15min":0.46800000000000003},"max":{"1min":1.538,"5min":1.538,"15min":1.5820000000000001},"last":1.026}]}]},{"osd":1,"up_from":13,"seq":55834575298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092032,"kb_used_data":43444,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279808,"statfs":{"total":96636764160,"available":95518523392,"internally_reserved":1073741824,"allocated":44486656,"data_stored":43096796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73599999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.38},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.441,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90000000000000002,"5min":0.92400000000000004,"15min":1.052},"last":0.441}]},{"osd":2,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54200000000000004,"5min":1.071,"15min":1.071},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":0.61799999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.49299999999999999,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":1.077},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54500000000000004,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.34200000000000003},"max":{"1min":0.67800000000000005,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.55000000000000004}]},{"osd":4,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.0449999999999999},"min":{"1min":0.72799999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3100000000000001,"5min":1.46,"15min":1.8919999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.038,"15min":1.0249999999999999},"min":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2789999999999999,"5min":1.3799999999999999,"15min":1.6080000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.53900000000000003,"15min":0.48799999999999999},"max":{"1min":1.1519999999999999,"5min":1.3149999999999999,"15min":1.391},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.97899999999999998,"15min":0.95499999999999996},"min":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.48999999999999999},"max":{"1min":1.403,"5min":1.403,"15min":1.7350000000000001},"last":1.1240000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.0620000000000001,"15min":1.0389999999999999},"min":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4670000000000001,"5min":1.528,"15min":1.6910000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0669999999999999,"15min":1.048},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64200000000000002},"max":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.506},"last":1.202}]},{"osd":7,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.99099999999999999,"15min":0.98199999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.61799999999999999},"max":{"1min":1.1990000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.008},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.006,"15min":0.97199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.54200000000000004},"max":{"1min":1.2430000000000001,"5min":1.3959999999999999,"15min":1.5529999999999999},"last":1.0960000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122756,"kb_used_data":74172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249084,"statfs":{"total":96636764160,"available":95487062016,"internally_reserved":1073741824,"allocated":75952128,"data_stored":74554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.44500000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.55200000000000005,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.371,"15min":0.314},"max":{"1min":1.9279999999999999,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53700000000000003,"15min":0.51700000000000002},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.33600000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.63600000000000001,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":2.133,"5min":2.133,"15min":2.133},"last":0.76300000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.97399999999999998,"15min":0.98099999999999998},"min":{"1min":0.61499999999999999,"5min":0.60199999999999998,"15min":0.47899999999999998},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.018,"5min":0.98299999999999998,"15min":0.96399999999999997},"min":{"1min":0.65600000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.1469999999999998,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":0.89200000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.111,"15min":1.0920000000000001},"min":{"1min":0.68200000000000005,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.0840000000000001,"15min":1.071},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.874}]},{"osd":6,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.97299999999999998,"15min":0.95699999999999996},"min":{"1min":0.59499999999999997,"5min":0.57799999999999996,"15min":0.51400000000000001},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1,"15min":0.99399999999999999},"min":{"1min":0.73999999999999999,"5min":0.66700000000000004,"15min":0.63600000000000001},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":0.94599999999999995}]},{"osd":7,"last update":"Wed Feb 10 11:58:46 2021","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":0.98499999999999999,"15min":0.98299999999999998},"min":{"1min":0.71099999999999997,"5min":0.624,"15min":0.58999999999999997},"max":{"1min":2.335,"5min":2.335,"15min":2.335},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0209999999999999,"5min":0.97199999999999998,"15min":0.98999999999999999},"min":{"1min":0.60599999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":2.2709999999999999,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575297,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079732,"kb_used_data":31148,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292108,"statfs":{"total":96636764160,"available":95531118592,"internally_reserved":1073741824,"allocated":31895552,"data_stored":30513630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60399999999999998,"15min":0.61299999999999999},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":0.97699999999999998,"5min":1.006,"15min":1.006},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.32300000000000001},"max":{"1min":0.70399999999999996,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.33300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.91800000000000004,"15min":1.0429999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.374},"max":{"1min":0.71999999999999997,"5min":0.95399999999999996,"15min":1.018},"last":0.377},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":1.0309999999999999},"last":0.53800000000000003}]},{"osd":4,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.59299999999999997,"5min":0.45400000000000001,"15min":0.39200000000000002},"max":{"1min":1.3080000000000001,"5min":1.387,"15min":1.5680000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.94099999999999995,"15min":0.94499999999999995},"min":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2430000000000001,"5min":1.3360000000000001,"15min":1.4450000000000001},"last":0.64100000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.97999999999999998,"15min":0.98499999999999999},"min":{"1min":0.755,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.4790000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.97099999999999997,"15min":0.97299999999999998},"min":{"1min":0.64400000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.304,"5min":1.3200000000000001,"15min":1.4830000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0069999999999999,"15min":1},"min":{"1min":0.67200000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.5880000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.98999999999999999,"15min":0.98099999999999998},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.23,"5min":1.2589999999999999,"15min":1.5880000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.256,"5min":1.349,"15min":1.4510000000000001},"last":0.90400000000000003}]}]}]}} 2021-02-10T11:59:39.162 INFO:tasks.ceph:pgid 43.6 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.162 INFO:tasks.ceph:pgid 42.7 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.163 INFO:tasks.ceph:pgid 43.7 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.163 INFO:tasks.ceph:pgid 42.6 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.163 INFO:tasks.ceph:pgid 43.5 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.163 INFO:tasks.ceph:pgid 42.4 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.163 INFO:tasks.ceph:pgid 43.2 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 42.3 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 43.3 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 43.0 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 42.1 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 43.1 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.164 INFO:tasks.ceph:pgid 42.0 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 43.4 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.165 INFO:tasks.ceph:pgid 42.5 last_scrub_stamp 2021-02-10 11:58:02.773071 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=2, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.166 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.166 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-10 11:22:23.870088 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=22, tm_sec=23, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:39.166 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-10T11:59:51.724 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:59:51.727 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T11:59:59.167 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T11:59:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.576 7f31aef31700 1 Processor -- start 2021-02-10T11:59:59.596 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.576 7f31aef31700 1 -- start start 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31aef31700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 0x7f31a8086ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31aef31700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f31a8084c60 0x7f31a8085060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31aef31700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a8085930 con 0x7f31a80e4e40 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31aef31700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a8085680 con 0x7f31a8084c60 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31aef31700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a81b09e0 con 0x7f31a80e5240 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31acd4d700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7f31a80e4e40 0x7f31a81b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33758/0 (socket says 172.21.2.117:33758) 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31acd4d700 1 -- 172.21.2.117:0/3579377766 learned_addr learned my addr 172.21.2.117:0/3579377766 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T11:59:59.597 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31a7fff700 1 --2- 172.21.2.117:0/3579377766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 0x7f31a8086ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31ad54e700 1 --2- 172.21.2.117:0/3579377766 >> v2:172.21.2.125:3300/0 conn(0x7f31a8084c60 0x7f31a8085060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.577 7f31a77fe700 1 -- 172.21.2.117:0/3579377766 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873677584 0 0) 0x7f31900009b0 con 0x7f31a80e4e40 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a77fe700 1 -- 172.21.2.117:0/3579377766 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3194002fe0 con 0x7f31a80e4e40 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a7fff700 1 -- 172.21.2.117:0/3579377766 >> v2:172.21.2.125:3300/0 conn(0x7f31a8084c60 msgr2=0x7f31a8085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a7fff700 1 --2- 172.21.2.117:0/3579377766 >> v2:172.21.2.125:3300/0 conn(0x7f31a8084c60 0x7f31a8085060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a7fff700 1 -- 172.21.2.117:0/3579377766 >> v1:172.21.2.117:6789/0 conn(0x7f31a80e4e40 legacy=0x7f31a81b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a7fff700 1 -- 172.21.2.117:0/3579377766 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8084840 con 0x7f31a80e5240 2021-02-10T11:59:59.598 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a7fff700 1 --2- 172.21.2.117:0/3579377766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 0x7f31a8086ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f3198006650 tx=0x7f3198009a80).ready entity=mon.2 client_cookie=24d0052c96c7ed6c server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a77fe700 1 -- 172.21.2.117:0/3579377766 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319800c0c0 con 0x7f31a80e5240 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a77fe700 1 -- 172.21.2.117:0/3579377766 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319800c300 con 0x7f31a80e5240 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31a77fe700 1 -- 172.21.2.117:0/3579377766 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319800c820 con 0x7f31a80e5240 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31aef31700 1 -- 172.21.2.117:0/3579377766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 msgr2=0x7f31a8086ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.578 7f31aef31700 1 --2- 172.21.2.117:0/3579377766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 0x7f31a8086ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f3198006650 tx=0x7f3198009a80).stop 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 -- 172.21.2.117:0/3579377766 shutdown_connections 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 --2- 172.21.2.117:0/3579377766 >> v2:172.21.2.125:3300/0 conn(0x7f31a8084c60 0x7f31a8085060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.599 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 --2- 172.21.2.117:0/3579377766 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80e5240 0x7f31a8086ec0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 -- 172.21.2.117:0/3579377766 >> 172.21.2.117:0/3579377766 conn(0x7f31a80ce4e0 msgr2=0x7f31a80e4310 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 -- 172.21.2.117:0/3579377766 shutdown_connections 2021-02-10T11:59:59.600 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.579 7f31aef31700 1 -- 172.21.2.117:0/3579377766 wait complete. 2021-02-10T11:59:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.593 7f31aef31700 1 Processor -- start 2021-02-10T11:59:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.593 7f31aef31700 1 -- start start 2021-02-10T11:59:59.601 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a809f0c0 0x7f31a80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80a3df0 0x7f31a809f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80929d0 con 0x7f31a809ecc0 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8092f50 con 0x7f31a809f0c0 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31aef31700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a8093170 con 0x7f31a80a3df0 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33514/0 (socket says 172.21.2.117:33514) 2021-02-10T11:59:59.602 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 -- 172.21.2.117:0/1866919824 learned_addr learned my addr 172.21.2.117:0/1866919824 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 -- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80a3df0 msgr2=0x7f31a809f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80a3df0 0x7f31a809f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 -- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a809f0c0 msgr2=0x7f31a80a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a809f0c0 0x7f31a80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.594 7f31acd4d700 1 -- 172.21.2.117:0/1866919824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a809fe30 con 0x7f31a809ecc0 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31acd4d700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 rx=0x7f3190001c60 tx=0x7f319000a8d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f319000acc0 con 0x7f31a809ecc0 2021-02-10T11:59:59.603 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3190004140 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f3190004660 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31aef31700 1 -- 172.21.2.117:0/1866919824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a809fe30 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31aef31700 1 -- 172.21.2.117:0/1866919824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a80a03c0 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.595 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f3190004320 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.596 7f31a57fa700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 0x7f318c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.596 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f31900326c0 con 0x7f31a809ecc0 2021-02-10T11:59:59.604 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.596 7f31a7fff700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 0x7f318c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T11:59:59.605 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.596 7f31a7fff700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 0x7f318c01abc0 crc :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T11:59:59.630 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.627 7f31aef31700 1 -- 172.21.2.117:0/1866919824 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a810c280 con 0x7f31a809ecc0 2021-02-10T11:59:59.632 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.629 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f3190000940 con 0x7f31a809ecc0 2021-02-10T11:59:59.764 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.761 7f31aef31700 1 -- 172.21.2.117:0/1866919824 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7f31a80a03c0 con 0x7f318c0186c0 2021-02-10T11:59:59.769 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.766 7f31a57fa700 1 -- 172.21.2.117:0/1866919824 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101131 (crc 0 0 0) 0x7f3198006740 con 0x7f318c0186c0 2021-02-10T11:59:59.772 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T11:59:59.773 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T11:59:59.773 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 -- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 msgr2=0x7f318c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 0x7f318c01abc0 crc :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 -- 172.21.2.117:0/1866919824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 msgr2=0x7f31a80a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 rx=0x7f3190001c60 tx=0x7f319000a8d0).stop 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 -- 172.21.2.117:0/1866919824 shutdown_connections 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f31a80a3df0 0x7f31a809f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f31a809f0c0 0x7f31a80a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f318c0186c0 0x7f318c01abc0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.774 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 --2- 172.21.2.117:0/1866919824 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f31a809ecc0 0x7f31a80a31b0 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T11:59:59.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.770 7f31aef31700 1 -- 172.21.2.117:0/1866919824 >> 172.21.2.117:0/1866919824 conn(0x7f31a80a1510 msgr2=0x7f31a80c3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T11:59:59.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.771 7f31aef31700 1 -- 172.21.2.117:0/1866919824 shutdown_connections 2021-02-10T11:59:59.775 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 11:59:59.771 7f31aef31700 1 -- 172.21.2.117:0/1866919824 wait complete. 2021-02-10T11:59:59.782 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":1332,"stamp":"2021-02-10 11:59:59.356218","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135156737,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":438,"num_write_kb":132092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":485,"ondisk_log_size":485,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672516,"kb_used_data":283840,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302204,"statfs":{"total":773094113280,"available":764213456896,"internally_reserved":8589934592,"allocated":290652160,"data_stored":279540394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002763"},"pg_stats":[{"pgid":"43.6","version":"196'22","reported_seq":"40","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.583889","last_change":"2021-02-10 11:58:04.419386","last_active":"2021-02-10 11:59:18.583889","last_peered":"2021-02-10 11:59:18.583889","last_clean":"2021-02-10 11:59:18.583889","last_became_active":"2021-02-10 11:58:04.419103","last_became_peered":"2021-02-10 11:58:04.419103","last_unstale":"2021-02-10 11:59:18.583889","last_undegraded":"2021-02-10 11:59:18.583889","last_fullsized":"2021-02-10 11:59:18.583889","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"196'12","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:55.084592","last_change":"2021-02-10 11:59:55.084628","last_active":"2021-02-10 11:59:55.084592","last_peered":"2021-02-10 11:59:55.084592","last_clean":"2021-02-10 11:59:55.084592","last_became_active":"2021-02-10 11:58:02.783318","last_became_peered":"2021-02-10 11:58:02.783318","last_unstale":"2021-02-10 11:59:55.084592","last_undegraded":"2021-02-10 11:59:55.084592","last_fullsized":"2021-02-10 11:59:55.084592","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'12","last_scrub_stamp":"2021-02-10 11:59:55.084586","last_deep_scrub":"196'12","last_deep_scrub_stamp":"2021-02-10 11:59:55.084586","last_clean_scrub_stamp":"2021-02-10 11:59:55.084586","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"196'21","reported_seq":"40","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.691659","last_change":"2021-02-10 11:59:43.691685","last_active":"2021-02-10 11:59:43.691659","last_peered":"2021-02-10 11:59:43.691659","last_clean":"2021-02-10 11:59:43.691659","last_became_active":"2021-02-10 11:58:04.419410","last_became_peered":"2021-02-10 11:58:04.419410","last_unstale":"2021-02-10 11:59:43.691659","last_undegraded":"2021-02-10 11:59:43.691659","last_fullsized":"2021-02-10 11:59:43.691659","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'21","last_scrub_stamp":"2021-02-10 11:59:43.691655","last_deep_scrub":"196'21","last_deep_scrub_stamp":"2021-02-10 11:59:43.691655","last_clean_scrub_stamp":"2021-02-10 11:59:43.691655","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"196'19","reported_seq":"33","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:59.239107","last_change":"2021-02-10 11:59:59.239131","last_active":"2021-02-10 11:59:59.239107","last_peered":"2021-02-10 11:59:59.239107","last_clean":"2021-02-10 11:59:59.239107","last_became_active":"2021-02-10 11:58:02.783694","last_became_peered":"2021-02-10 11:58:02.783694","last_unstale":"2021-02-10 11:59:59.239107","last_undegraded":"2021-02-10 11:59:59.239107","last_fullsized":"2021-02-10 11:59:59.239107","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'19","last_scrub_stamp":"2021-02-10 11:59:59.239103","last_deep_scrub":"196'19","last_deep_scrub_stamp":"2021-02-10 11:59:59.239103","last_clean_scrub_stamp":"2021-02-10 11:59:59.239103","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"196'34","reported_seq":"55","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:46.758404","last_change":"2021-02-10 11:59:46.758429","last_active":"2021-02-10 11:59:46.758404","last_peered":"2021-02-10 11:59:46.758404","last_clean":"2021-02-10 11:59:46.758404","last_became_active":"2021-02-10 11:58:04.421304","last_became_peered":"2021-02-10 11:58:04.421304","last_unstale":"2021-02-10 11:59:46.758404","last_undegraded":"2021-02-10 11:59:46.758404","last_fullsized":"2021-02-10 11:59:46.758404","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'34","last_scrub_stamp":"2021-02-10 11:59:46.758385","last_deep_scrub":"196'34","last_deep_scrub_stamp":"2021-02-10 11:59:46.758385","last_clean_scrub_stamp":"2021-02-10 11:59:46.758385","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":21,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"196'228","reported_seq":"242","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.777094","last_change":"2021-02-10 11:59:42.278366","last_active":"2021-02-10 11:59:43.777094","last_peered":"2021-02-10 11:59:43.777094","last_clean":"2021-02-10 11:59:43.777094","last_became_active":"2021-02-10 11:58:02.784457","last_became_peered":"2021-02-10 11:58:02.784457","last_unstale":"2021-02-10 11:59:43.777094","last_undegraded":"2021-02-10 11:59:43.777094","last_fullsized":"2021-02-10 11:59:43.777094","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'226","last_scrub_stamp":"2021-02-10 11:59:42.278332","last_deep_scrub":"196'226","last_deep_scrub_stamp":"2021-02-10 11:59:42.278332","last_clean_scrub_stamp":"2021-02-10 11:59:42.278332","log_size":228,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":937721,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"196'27","reported_seq":"47","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:34.727689","last_change":"2021-02-10 11:59:34.727715","last_active":"2021-02-10 11:59:34.727689","last_peered":"2021-02-10 11:59:34.727689","last_clean":"2021-02-10 11:59:34.727689","last_became_active":"2021-02-10 11:58:04.421381","last_became_peered":"2021-02-10 11:58:04.421381","last_unstale":"2021-02-10 11:59:34.727689","last_undegraded":"2021-02-10 11:59:34.727689","last_fullsized":"2021-02-10 11:59:34.727689","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'27","last_scrub_stamp":"2021-02-10 11:59:34.727686","last_deep_scrub":"196'27","last_deep_scrub_stamp":"2021-02-10 11:59:34.727686","last_clean_scrub_stamp":"2021-02-10 11:59:34.727686","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"196'9","reported_seq":"23","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.312100","last_change":"2021-02-10 11:59:43.312125","last_active":"2021-02-10 11:59:43.312100","last_peered":"2021-02-10 11:59:43.312100","last_clean":"2021-02-10 11:59:43.312100","last_became_active":"2021-02-10 11:58:03.787526","last_became_peered":"2021-02-10 11:58:03.787526","last_unstale":"2021-02-10 11:59:43.312100","last_undegraded":"2021-02-10 11:59:43.312100","last_fullsized":"2021-02-10 11:59:43.312100","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2021-02-10 11:59:43.312097","last_deep_scrub":"196'9","last_deep_scrub_stamp":"2021-02-10 11:59:43.312097","last_clean_scrub_stamp":"2021-02-10 11:59:43.312097","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4620,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"196'31","reported_seq":"51","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:40.721440","last_change":"2021-02-10 11:59:40.721485","last_active":"2021-02-10 11:59:40.721440","last_peered":"2021-02-10 11:59:40.721440","last_clean":"2021-02-10 11:59:40.721440","last_became_active":"2021-02-10 11:58:04.419046","last_became_peered":"2021-02-10 11:58:04.419046","last_unstale":"2021-02-10 11:59:40.721440","last_undegraded":"2021-02-10 11:59:40.721440","last_fullsized":"2021-02-10 11:59:40.721440","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'31","last_scrub_stamp":"2021-02-10 11:59:40.721436","last_deep_scrub":"196'31","last_deep_scrub_stamp":"2021-02-10 11:59:40.721436","last_clean_scrub_stamp":"2021-02-10 11:59:40.721436","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"195'1","reported_seq":"15","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.706570","last_change":"2021-02-10 11:59:33.706603","last_active":"2021-02-10 11:59:33.706570","last_peered":"2021-02-10 11:59:33.706570","last_clean":"2021-02-10 11:59:33.706570","last_became_active":"2021-02-10 11:58:03.788701","last_became_peered":"2021-02-10 11:58:03.788701","last_unstale":"2021-02-10 11:59:33.706570","last_undegraded":"2021-02-10 11:59:33.706570","last_fullsized":"2021-02-10 11:59:33.706570","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'1","last_scrub_stamp":"2021-02-10 11:59:33.706566","last_deep_scrub":"195'1","last_deep_scrub_stamp":"2021-02-10 11:59:33.706566","last_clean_scrub_stamp":"2021-02-10 11:59:33.706566","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"196'27","reported_seq":"49","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:44.363939","last_change":"2021-02-10 11:59:44.363968","last_active":"2021-02-10 11:59:44.363939","last_peered":"2021-02-10 11:59:44.363939","last_clean":"2021-02-10 11:59:44.363939","last_became_active":"2021-02-10 11:58:04.419010","last_became_peered":"2021-02-10 11:58:04.419010","last_unstale":"2021-02-10 11:59:44.363939","last_undegraded":"2021-02-10 11:59:44.363939","last_fullsized":"2021-02-10 11:59:44.363939","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'27","last_scrub_stamp":"2021-02-10 11:59:44.363936","last_deep_scrub":"196'27","last_deep_scrub_stamp":"2021-02-10 11:59:44.363936","last_clean_scrub_stamp":"2021-02-10 11:59:44.363936","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"196'2","reported_seq":"17","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:39.723713","last_change":"2021-02-10 11:59:39.723757","last_active":"2021-02-10 11:59:39.723713","last_peered":"2021-02-10 11:59:39.723713","last_clean":"2021-02-10 11:59:39.723713","last_became_active":"2021-02-10 11:58:03.789795","last_became_peered":"2021-02-10 11:58:03.789795","last_unstale":"2021-02-10 11:59:39.723713","last_undegraded":"2021-02-10 11:59:39.723713","last_fullsized":"2021-02-10 11:59:39.723713","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'2","last_scrub_stamp":"2021-02-10 11:59:39.723708","last_deep_scrub":"196'2","last_deep_scrub_stamp":"2021-02-10 11:59:39.723708","last_clean_scrub_stamp":"2021-02-10 11:59:39.723708","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"196'11","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:18.607248","last_change":"2021-02-10 11:58:04.419297","last_active":"2021-02-10 11:59:18.607248","last_peered":"2021-02-10 11:59:18.607248","last_clean":"2021-02-10 11:59:18.607248","last_became_active":"2021-02-10 11:58:04.418997","last_became_peered":"2021-02-10 11:58:04.418997","last_unstale":"2021-02-10 11:59:18.607248","last_undegraded":"2021-02-10 11:59:18.607248","last_fullsized":"2021-02-10 11:59:18.607248","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:58:03.781579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:58:03.781579","last_clean_scrub_stamp":"2021-02-10 11:58:03.781579","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"196'12","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:46.251848","last_change":"2021-02-10 11:59:46.251887","last_active":"2021-02-10 11:59:46.251848","last_peered":"2021-02-10 11:59:46.251848","last_clean":"2021-02-10 11:59:46.251848","last_became_active":"2021-02-10 11:58:02.783528","last_became_peered":"2021-02-10 11:58:02.783528","last_unstale":"2021-02-10 11:59:46.251848","last_undegraded":"2021-02-10 11:59:46.251848","last_fullsized":"2021-02-10 11:59:46.251848","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'12","last_scrub_stamp":"2021-02-10 11:59:46.251844","last_deep_scrub":"196'12","last_deep_scrub_stamp":"2021-02-10 11:59:46.251844","last_clean_scrub_stamp":"2021-02-10 11:59:46.251844","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"182","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:42.316945","last_change":"2021-02-10 11:59:42.316985","last_active":"2021-02-10 11:59:42.316945","last_peered":"2021-02-10 11:59:42.316945","last_clean":"2021-02-10 11:59:42.316945","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:59:42.316945","last_undegraded":"2021-02-10 11:59:42.316945","last_fullsized":"2021-02-10 11:59:42.316945","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:42.316942","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:42.316942","last_clean_scrub_stamp":"2021-02-10 11:59:42.316942","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"182","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.345982","last_change":"2021-02-10 11:59:38.346013","last_active":"2021-02-10 11:59:38.345982","last_peered":"2021-02-10 11:59:38.345982","last_clean":"2021-02-10 11:59:38.345982","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:59:38.345982","last_undegraded":"2021-02-10 11:59:38.345982","last_fullsized":"2021-02-10 11:59:38.345982","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:38.345978","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:38.345978","last_clean_scrub_stamp":"2021-02-10 11:59:38.345978","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"186","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.162264","last_change":"2021-02-10 11:59:38.162298","last_active":"2021-02-10 11:59:38.162264","last_peered":"2021-02-10 11:59:38.162264","last_clean":"2021-02-10 11:59:38.162264","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:59:38.162264","last_undegraded":"2021-02-10 11:59:38.162264","last_fullsized":"2021-02-10 11:59:38.162264","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:38.162261","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:38.162261","last_clean_scrub_stamp":"2021-02-10 11:59:38.162261","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:35.057632","last_change":"2021-02-10 11:59:35.057663","last_active":"2021-02-10 11:59:35.057632","last_peered":"2021-02-10 11:59:35.057632","last_clean":"2021-02-10 11:59:35.057632","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:59:35.057632","last_undegraded":"2021-02-10 11:59:35.057632","last_fullsized":"2021-02-10 11:59:35.057632","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:35.057629","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:35.057629","last_clean_scrub_stamp":"2021-02-10 11:59:35.057629","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:41.931591","last_change":"2021-02-10 11:59:41.931628","last_active":"2021-02-10 11:59:41.931591","last_peered":"2021-02-10 11:59:41.931591","last_clean":"2021-02-10 11:59:41.931591","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:59:41.931591","last_undegraded":"2021-02-10 11:59:41.931591","last_fullsized":"2021-02-10 11:59:41.931591","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:41.931588","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:41.931588","last_clean_scrub_stamp":"2021-02-10 11:59:41.931588","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"196'16","reported_seq":"34","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.133110","last_change":"2021-02-10 11:59:38.133137","last_active":"2021-02-10 11:59:38.133110","last_peered":"2021-02-10 11:59:38.133110","last_clean":"2021-02-10 11:59:38.133110","last_became_active":"2021-02-10 11:58:04.419401","last_became_peered":"2021-02-10 11:58:04.419401","last_unstale":"2021-02-10 11:59:38.133110","last_undegraded":"2021-02-10 11:59:38.133110","last_fullsized":"2021-02-10 11:59:38.133110","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2021-02-10 11:59:38.133106","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2021-02-10 11:59:38.133106","last_clean_scrub_stamp":"2021-02-10 11:59:38.133106","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"196'13","reported_seq":"27","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:53.129763","last_change":"2021-02-10 11:59:53.129788","last_active":"2021-02-10 11:59:53.129763","last_peered":"2021-02-10 11:59:53.129763","last_clean":"2021-02-10 11:59:53.129763","last_became_active":"2021-02-10 11:58:02.783300","last_became_peered":"2021-02-10 11:58:02.783300","last_unstale":"2021-02-10 11:59:53.129763","last_undegraded":"2021-02-10 11:59:53.129763","last_fullsized":"2021-02-10 11:59:53.129763","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13","last_scrub_stamp":"2021-02-10 11:59:53.129758","last_deep_scrub":"196'13","last_deep_scrub_stamp":"2021-02-10 11:59:53.129758","last_clean_scrub_stamp":"2021-02-10 11:59:53.129758","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"191","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:40.295061","last_change":"2021-02-10 11:59:40.295091","last_active":"2021-02-10 11:59:40.295061","last_peered":"2021-02-10 11:59:40.295061","last_clean":"2021-02-10 11:59:40.295061","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:59:40.295061","last_undegraded":"2021-02-10 11:59:40.295061","last_fullsized":"2021-02-10 11:59:40.295061","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:40.295058","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:40.295058","last_clean_scrub_stamp":"2021-02-10 11:59:40.295058","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:48.048134","last_change":"2021-02-10 11:59:48.048165","last_active":"2021-02-10 11:59:48.048134","last_peered":"2021-02-10 11:59:48.048134","last_clean":"2021-02-10 11:59:48.048134","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:59:48.048134","last_undegraded":"2021-02-10 11:59:48.048134","last_fullsized":"2021-02-10 11:59:48.048134","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:48.048131","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:48.048131","last_clean_scrub_stamp":"2021-02-10 11:59:48.048131","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"185","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.744224","last_change":"2021-02-10 11:59:33.744256","last_active":"2021-02-10 11:59:33.744224","last_peered":"2021-02-10 11:59:33.744224","last_clean":"2021-02-10 11:59:33.744224","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:59:33.744224","last_undegraded":"2021-02-10 11:59:33.744224","last_fullsized":"2021-02-10 11:59:33.744224","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:33.744222","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:33.744222","last_clean_scrub_stamp":"2021-02-10 11:59:33.744222","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":140,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":938582,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298,"num_write_kb":1013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1941504,"data_stored":1877164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575304,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087940,"kb_used_data":39356,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283900,"statfs":{"total":96636764160,"available":95522713600,"internally_reserved":1073741824,"allocated":40300544,"data_stored":38903058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.91100000000000003,"15min":0.95899999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.33100000000000002},"max":{"1min":1.325,"5min":1.4219999999999999,"15min":1.534},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.95099999999999996,"15min":0.98899999999999999},"min":{"1min":0.54800000000000004,"5min":0.499,"15min":0.35899999999999999},"max":{"1min":1.296,"5min":1.304,"15min":1.498},"last":1.046}]},{"osd":1,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82399999999999995,"15min":0.84199999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.372},"max":{"1min":1.1479999999999999,"5min":1.288,"15min":1.3400000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.48899999999999999,"5min":0.45500000000000002,"15min":0.35799999999999998},"max":{"1min":1.2110000000000001,"5min":1.411,"15min":1.4970000000000001},"last":0.99199999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82099999999999995,"15min":0.86299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.3680000000000001},"last":1.012},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82099999999999995,"15min":0.85799999999999998},"min":{"1min":0.52600000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92300000000000004,"5min":1.2010000000000001,"15min":1.288},"last":1.0669999999999999}]},{"osd":3,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1240000000000001,"5min":1.2370000000000001,"15min":1.357},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.85999999999999999,"15min":0.874},"min":{"1min":0.61099999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"max":{"1min":1.224,"5min":1.266,"15min":1.278},"last":1.018}]},{"osd":4,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":1.0149999999999999,"5min":1.1200000000000001,"15min":1.286},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.501,"5min":0.42199999999999999,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.48999999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.1020000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.72299999999999998}]},{"osd":6,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.58699999999999997,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0940000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.63400000000000001,"5min":0.54000000000000004,"15min":0.433},"max":{"1min":1.171,"5min":1.171,"15min":1.3720000000000001},"last":0.64100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575304,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070416,"kb_used_data":21832,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301424,"statfs":{"total":96636764160,"available":95540658176,"internally_reserved":1073741824,"allocated":22355968,"data_stored":20965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.90300000000000002,"15min":0.90200000000000002},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.32000000000000001},"max":{"1min":1.407,"5min":1.407,"15min":1.5680000000000001},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.92600000000000005,"15min":0.92800000000000005},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2529999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0740000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.81499999999999995,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.39600000000000002,"15min":0.39000000000000001},"max":{"1min":1.2270000000000001,"5min":1.2390000000000001,"15min":1.363},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.82699999999999996,"15min":0.83299999999999996},"min":{"1min":0.57099999999999995,"5min":0.373,"15min":0.373},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.45},"last":1.0640000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.40300000000000002},"max":{"1min":1.0549999999999999,"5min":1.276,"15min":1.276},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.26,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":1.1020000000000001}]},{"osd":3,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.86599999999999999,"15min":0.85999999999999999},"min":{"1min":0.47999999999999998,"5min":0.442,"15min":0.441},"max":{"1min":1.2390000000000001,"5min":1.2889999999999999,"15min":1.3859999999999999},"last":0.997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.60599999999999998,"5min":0.443,"15min":0.379},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.94599999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0309999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":0.70999999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":1.256,"15min":1.256},"last":0.753}]},{"osd":5,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98499999999999999,"5min":1.0049999999999999,"15min":1.0720000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.78200000000000003,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1359999999999999,"5min":1.3660000000000001,"15min":1.6379999999999999},"last":0.92800000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.165,"5min":1.256,"15min":1.256},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78700000000000003,"15min":0.80500000000000005},"min":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.499},"max":{"1min":1.1779999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.87}]}]},{"osd":5,"up_from":13,"seq":55834575303,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083832,"kb_used_data":35248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93288008,"statfs":{"total":96636764160,"available":95526920192,"internally_reserved":1073741824,"allocated":36093952,"data_stored":34707957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.40699999999999997,"5min":0.38700000000000001,"15min":0.36299999999999999},"max":{"1min":1.224,"5min":1.363,"15min":1.365},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.432,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2749999999999999,"5min":1.399,"15min":1.399},"last":0.82199999999999995}]},{"osd":1,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.55400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.2430000000000001,"5min":1.3819999999999999,"15min":1.3979999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.3},"last":0.88600000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.59799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.181,"5min":1.2330000000000001,"15min":1.2809999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.879,"15min":0.878},"min":{"1min":0.63,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.274,"15min":1.3500000000000001},"last":0.84799999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.878,"15min":0.878},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3660000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.2669999999999999},"last":0.873}]},{"osd":4,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73399999999999999,"15min":0.76600000000000001},"min":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.44800000000000001},"max":{"1min":1.101,"5min":1.103,"15min":1.343},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.55200000000000005,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.1279999999999999,"5min":1.292,"15min":1.292},"last":0.69799999999999995}]},{"osd":6,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.75900000000000001},"min":{"1min":0.60399999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.278,"5min":1.278,"15min":1.3049999999999999},"last":0.622},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.67200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.206,"5min":1.2230000000000001,"15min":1.302},"last":0.89600000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.085,"5min":1.2809999999999999,"15min":1.3220000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.77700000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96599999999999997,"5min":1.147,"15min":1.1870000000000001},"last":0.78200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074508,"kb_used_data":25924,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297332,"statfs":{"total":96636764160,"available":95536467968,"internally_reserved":1073741824,"allocated":26546176,"data_stored":25159758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91800000000000004,"15min":0.93100000000000005},"min":{"1min":0.51300000000000001,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":1.3600000000000001,"5min":1.407,"15min":1.4099999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.046,"5min":0.94299999999999995,"15min":0.95399999999999996},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":1.3089999999999999,"5min":1.335,"15min":1.3879999999999999},"last":1.123}]},{"osd":1,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.84899999999999998,"15min":0.84999999999999998},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.372},"max":{"1min":1.222,"5min":1.2529999999999999,"15min":1.341},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.81499999999999995,"15min":0.82999999999999996},"min":{"1min":0.46200000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":1.119,"5min":1.181,"15min":1.2869999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.101},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.83899999999999997,"15min":0.85399999999999998},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.432},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4570000000000001},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83699999999999997,"15min":0.85699999999999998},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2569999999999999,"5min":1.321,"15min":1.3220000000000001},"last":0.98199999999999998}]},{"osd":5,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.318},"max":{"1min":0.96399999999999997,"5min":1.0900000000000001,"15min":1.141},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96099999999999997,"5min":1.042,"15min":1.1279999999999999},"last":0.78800000000000003}]},{"osd":6,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.438,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.113,"15min":1.204},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.0720000000000001,"15min":1.2729999999999999},"last":0.77600000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.44800000000000001,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.496,"5min":0.432,"15min":0.432},"max":{"1min":1.0249999999999999,"5min":1.165,"15min":1.321},"last":0.76200000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575304,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061300,"kb_used_data":12716,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310540,"statfs":{"total":96636764160,"available":95549992960,"internally_reserved":1073741824,"allocated":13021184,"data_stored":11639159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.94399999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78100000000000003,"5min":0.82999999999999996,"15min":1.319},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.85099999999999998,"15min":0.91800000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.39800000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.83999999999999997,"15min":1.1599999999999999},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.89600000000000002},"last":0.62},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.63400000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91300000000000003,"5min":0.96799999999999997,"15min":1.077},"last":0.60599999999999998}]},{"osd":4,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.97599999999999998,"15min":0.96299999999999997},"min":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":1.393,"5min":1.393,"15min":1.4039999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96899999999999997,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4219999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.94999999999999996},"min":{"1min":0.71299999999999997,"5min":0.504,"15min":0.48099999999999998},"max":{"1min":1.319,"5min":1.4690000000000001,"15min":1.5660000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.044,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.61599999999999999,"5min":0.54700000000000004,"15min":0.497},"max":{"1min":1.379,"5min":1.379,"15min":1.512},"last":0.96799999999999997}]},{"osd":6,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.64300000000000002,"5min":0.61799999999999999,"15min":0.48999999999999999},"max":{"1min":1.363,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.0669999999999999,"15min":1.079},"min":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.53400000000000003},"max":{"1min":1.3879999999999999,"5min":1.7490000000000001,"15min":1.8100000000000001},"last":1.036}]},{"osd":7,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.01,"15min":1.004},"min":{"1min":0.754,"5min":0.622,"15min":0.54900000000000004},"max":{"1min":1.417,"5min":1.615,"15min":1.909},"last":1.115},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":0.99299999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.46800000000000003},"max":{"1min":1.4590000000000001,"5min":1.538,"15min":1.538},"last":1.0660000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575303,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092032,"kb_used_data":43444,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279808,"statfs":{"total":96636764160,"available":95518523392,"internally_reserved":1073741824,"allocated":44486656,"data_stored":43096796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73599999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.441,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90000000000000002,"5min":0.92400000000000004,"15min":1.052},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54200000000000004,"5min":1.071,"15min":1.071},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":0.61799999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.747}]},{"osd":3,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.49299999999999999,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":1.077},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54500000000000004,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.34200000000000003},"max":{"1min":0.67800000000000005,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.55200000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.0449999999999999},"min":{"1min":0.72799999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3100000000000001,"5min":1.46,"15min":1.8919999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.038,"15min":1.0249999999999999},"min":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2789999999999999,"5min":1.3799999999999999,"15min":1.6080000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.53900000000000003,"15min":0.48799999999999999},"max":{"1min":1.1519999999999999,"5min":1.3149999999999999,"15min":1.391},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.97899999999999998,"15min":0.95499999999999996},"min":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.48999999999999999},"max":{"1min":1.403,"5min":1.403,"15min":1.7350000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.0620000000000001,"15min":1.0389999999999999},"min":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4670000000000001,"5min":1.528,"15min":1.6910000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0669999999999999,"15min":1.048},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64200000000000002},"max":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.506},"last":0.76900000000000002}]},{"osd":7,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.99099999999999999,"15min":0.98199999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.61799999999999999},"max":{"1min":1.1990000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.006,"15min":0.97199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.54200000000000004},"max":{"1min":1.2430000000000001,"5min":1.3959999999999999,"15min":1.5529999999999999},"last":0.73899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122756,"kb_used_data":74172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249084,"statfs":{"total":96636764160,"available":95487062016,"internally_reserved":1073741824,"allocated":75952128,"data_stored":74554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81200000000000006,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.44500000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90300000000000002,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.36199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.371,"15min":0.314},"max":{"1min":0.69599999999999995,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.52100000000000002},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.33600000000000002},"max":{"1min":0.82499999999999996,"5min":1.671,"15min":1.671},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61899999999999999},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":0.76700000000000002,"5min":2.133,"15min":2.133},"last":0.67100000000000004}]},{"osd":4,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.95999999999999996,"15min":0.97899999999999998},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.47899999999999998},"max":{"1min":1.276,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.97599999999999998,"15min":0.96299999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.45100000000000001},"max":{"1min":1.274,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1020000000000001,"15min":1.095},"min":{"1min":0.77500000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.397,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.071,"15min":1.0720000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.411,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.325}]},{"osd":6,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96699999999999997,"15min":0.96199999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.347,"5min":2.161,"15min":2.161},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.999,"15min":0.996},"min":{"1min":0.73299999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.228,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.284}]},{"osd":7,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98699999999999999},"min":{"1min":0.71299999999999997,"5min":0.624,"15min":0.58999999999999997},"max":{"1min":1.343,"5min":2.335,"15min":2.335},"last":1.214},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.97799999999999998,"15min":0.99199999999999999},"min":{"1min":0.73299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3580000000000001,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.2589999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575302,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079732,"kb_used_data":31148,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292108,"statfs":{"total":96636764160,"available":95531118592,"internally_reserved":1073741824,"allocated":31895552,"data_stored":30513630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60399999999999998,"15min":0.61299999999999999},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":0.97699999999999998,"5min":1.006,"15min":1.006},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.32300000000000001},"max":{"1min":0.70399999999999996,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.33300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.91800000000000004,"15min":1.0429999999999999},"last":0.64800000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.374},"max":{"1min":0.71999999999999997,"5min":0.95399999999999996,"15min":1.018},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":1.0309999999999999},"last":0.59499999999999997}]},{"osd":4,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.59299999999999997,"5min":0.45400000000000001,"15min":0.39200000000000002},"max":{"1min":1.3080000000000001,"5min":1.387,"15min":1.5680000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.94099999999999995,"15min":0.94499999999999995},"min":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2430000000000001,"5min":1.3360000000000001,"15min":1.4450000000000001},"last":1.0760000000000001}]},{"osd":5,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.97999999999999998,"15min":0.98499999999999999},"min":{"1min":0.755,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.4790000000000001},"last":1.0860000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.97099999999999997,"15min":0.97299999999999998},"min":{"1min":0.64400000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.304,"5min":1.3200000000000001,"15min":1.4830000000000001},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0069999999999999,"15min":1},"min":{"1min":0.67200000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.5880000000000001},"last":1.014}]},{"osd":7,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.98999999999999999,"15min":0.98099999999999998},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.23,"5min":1.2589999999999999,"15min":1.5880000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.256,"5min":1.349,"15min":1.4510000000000001},"last":1.0569999999999999}]}]}]}} 2021-02-10T11:59:59.784 INFO:tasks.ceph:pgid 43.6 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:59.785 INFO:tasks.ceph:pgid 43.1 last_scrub_stamp 2021-02-10 11:58:03.781579 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=58, tm_sec=3, tm_wday=2, tm_yday=41, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=10, tm_hour=11, tm_min=59, tm_sec=28, tm_wday=2, tm_yday=41, tm_isdst=0) 2021-02-10T11:59:59.785 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-10T12:00:19.786 DEBUG:teuthology.orchestra.run.gibba017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json 2021-02-10T12:00:20.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.205 7ff6b38c2700 1 Processor -- start 2021-02-10T12:00:20.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b38c2700 1 -- start start 2021-02-10T12:00:20.225 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b38c2700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7ff6ac0ef3f0 0x7ff6ac0eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b38c2700 1 -- --> v1:172.21.2.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac0ede60 con 0x7ff6ac0eb7f0 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b38c2700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ac0d1ce0 con 0x7ff6ac0d18e0 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b38c2700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6ac0ef7f0 con 0x7ff6ac0ef3f0 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b1edf700 1 --1- >> v1:172.21.2.117:6789/0 conn(0x7ff6ac0eb7f0 0x7ff6ac1b7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.2.117:6789/0 says I am v1:172.21.2.117:33776/0 (socket says 172.21.2.117:33776) 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.206 7ff6b1edf700 1 -- 172.21.2.117:0/3681404617 learned_addr learned my addr 172.21.2.117:0/3681404617 (peer_addr_for_me v1:172.21.2.117:0/0) 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 <== mon.0 v1:172.21.2.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540157105 0 0) 0x7ff6a4000f90 con 0x7ff6ac0eb7f0 2021-02-10T12:00:20.226 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 --> v1:172.21.2.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff698002fe0 con 0x7ff6ac0eb7f0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6b0edd700 1 --2- 172.21.2.117:0/3681404617 >> v2:172.21.2.125:3300/0 conn(0x7ff6ac0ef3f0 0x7ff6ac0eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6b0edd700 1 -- 172.21.2.117:0/3681404617 >> v1:172.21.2.125:6789/0 conn(0x7ff6ac0d18e0 legacy=0x7ff6ac0ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6b0edd700 1 -- 172.21.2.117:0/3681404617 >> v1:172.21.2.117:6789/0 conn(0x7ff6ac0eb7f0 legacy=0x7ff6ac1b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 <== mon.0 v1:172.21.2.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2752095970 0 0) 0x7ff6a4001500 con 0x7ff6ac0eb7f0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6b0edd700 1 -- 172.21.2.117:0/3681404617 --> v2:172.21.2.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac1bb180 con 0x7ff6ac0ef3f0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.207 7ff6b0edd700 1 --2- 172.21.2.117:0/3681404617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac0ef3f0 0x7ff6ac0eb1d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7ff69c006650 tx=0x7ff69c009a00).ready entity=mon.1 client_cookie=dca07d4d668653b3 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff69c009df0 con 0x7ff6ac0ef3f0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69c0009b0 con 0x7ff6ac0ef3f0 2021-02-10T12:00:20.227 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6a3fff700 1 -- 172.21.2.117:0/3681404617 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff69c00c2b0 con 0x7ff6ac0ef3f0 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6b38c2700 1 -- 172.21.2.117:0/3681404617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac0ef3f0 msgr2=0x7ff6ac0eb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6b38c2700 1 --2- 172.21.2.117:0/3681404617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac0ef3f0 0x7ff6ac0eb1d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7ff69c006650 tx=0x7ff69c009a00).stop 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6b38c2700 1 -- 172.21.2.117:0/3681404617 shutdown_connections 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.208 7ff6b38c2700 1 --2- 172.21.2.117:0/3681404617 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac0ef3f0 0x7ff6ac0eb1d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.209 7ff6b38c2700 1 -- 172.21.2.117:0/3681404617 >> 172.21.2.117:0/3681404617 conn(0x7ff6ac0ced60 msgr2=0x7ff6ac0cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.209 7ff6b38c2700 1 -- 172.21.2.117:0/3681404617 shutdown_connections 2021-02-10T12:00:20.228 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.209 7ff6b38c2700 1 -- 172.21.2.117:0/3681404617 wait complete. 2021-02-10T12:00:20.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.223 7ff6b38c2700 1 Processor -- start 2021-02-10T12:00:20.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.223 7ff6b38c2700 1 -- start start 2021-02-10T12:00:20.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 0x7ff6ac1ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 0x7ff6ac098ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.230 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac098560 con 0x7ff6ac1ac600 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac0982d0 con 0x7ff6ac1ac200 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b38c2700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac099100 con 0x7ff6ac1abd30 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b1edf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b1edf700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33532/0 (socket says 172.21.2.117:33532) 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b1edf700 1 -- 172.21.2.117:0/1056029238 learned_addr learned my addr 172.21.2.117:0/1056029238 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b0edd700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 0x7ff6ac098ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.224 7ff6b16de700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 0x7ff6ac1ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.231 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 -- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 msgr2=0x7ff6ac1ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 0x7ff6ac1ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 -- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 msgr2=0x7ff6ac098ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 0x7ff6ac098ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 -- 172.21.2.117:0/1056029238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac0992e0 con 0x7ff6ac1ac600 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b0edd700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 0x7ff6ac098ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b16de700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 0x7ff6ac1ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b1edf700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7ff6a4009980 tx=0x7ff6a400bcd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.232 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6a4004720 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a4004a60 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7ff6a4006030 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac0992e0 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.225 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ac0e61a0 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.226 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7ff6a4001270 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.226 7ff6a1ffb700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 0x7ff68c01abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.226 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7ff6a4032a40 con 0x7ff6ac1ac600 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.226 7ff6b16de700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 0x7ff68c01abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.233 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.227 7ff6b16de700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 0x7ff68c01abc0 crc :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.258 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.255 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ac087210 con 0x7ff6ac1ac600 2021-02-10T12:00:20.260 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.258 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7ff6a4000f40 con 0x7ff6ac1ac600 2021-02-10T12:00:20.393 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.391 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 --> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] -- command(tid 0: {"prefix": "pg dump", "target": ["mgr", ""], "format": "json"}) v1 -- 0x7ff6ac098560 con 0x7ff68c0186c0 2021-02-10T12:00:20.396 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.394 7ff6a1ffb700 1 -- 172.21.2.117:0/1056029238 <== mgr.4100 v2:172.21.2.125:6800/34993 1 ==== command_reply(tid 0: 0 dumped all) v1 ==== 18+0+101087 (crc 0 0 0) 0x7ff6a8000960 con 0x7ff68c0186c0 2021-02-10T12:00:20.398 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-02-10T12:00:20.398 INFO:teuthology.orchestra.run.gibba017.stderr:dumped all 2021-02-10T12:00:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.396 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 msgr2=0x7ff68c01abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.396 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 0x7ff68c01abc0 crc :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.399 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.396 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 msgr2=0x7ff6ac0a6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.396 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 rx=0x7ff6a4009980 tx=0x7ff6a400bcd0).stop 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 shutdown_connections 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7ff68c0186c0 0x7ff68c01abc0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7ff6ac1ac600 0x7ff6ac0a6c00 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7ff6ac1ac200 0x7ff6ac098ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 --2- 172.21.2.117:0/1056029238 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7ff6ac1abd30 0x7ff6ac1ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 >> 172.21.2.117:0/1056029238 conn(0x7ff6ac1916d0 msgr2=0x7ff6ac094b90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T12:00:20.400 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 shutdown_connections 2021-02-10T12:00:20.401 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.397 7ff6b38c2700 1 -- 172.21.2.117:0/1056029238 wait complete. 2021-02-10T12:00:20.407 INFO:teuthology.orchestra.run.gibba017.stdout:{"pg_ready":true,"pg_map":{"version":1342,"stamp":"2021-02-10 12:00:19.360555","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135156737,"num_objects":154,"num_object_clones":1,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":65,"num_read_kb":65537,"num_write":438,"num_write_kb":132092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":517,"ondisk_log_size":517,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8672516,"kb_used_data":283840,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746302204,"statfs":{"total":773094113280,"available":764213456896,"internally_reserved":8589934592,"allocated":290652160,"data_stored":279540394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002945"},"pg_stats":[{"pgid":"43.6","version":"196'44","reported_seq":"65","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 12:00:07.271958","last_change":"2021-02-10 12:00:07.271985","last_active":"2021-02-10 12:00:07.271958","last_peered":"2021-02-10 12:00:07.271958","last_clean":"2021-02-10 12:00:07.271958","last_became_active":"2021-02-10 11:58:04.419103","last_became_peered":"2021-02-10 11:58:04.419103","last_unstale":"2021-02-10 12:00:07.271958","last_undegraded":"2021-02-10 12:00:07.271958","last_fullsized":"2021-02-10 12:00:07.271958","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'44","last_scrub_stamp":"2021-02-10 12:00:07.271955","last_deep_scrub":"196'44","last_deep_scrub_stamp":"2021-02-10 12:00:07.271955","last_clean_scrub_stamp":"2021-02-10 12:00:07.271955","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23068672,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":9,"num_read_kb":9216,"num_write":22,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.7","version":"196'12","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:55.084592","last_change":"2021-02-10 11:59:55.084628","last_active":"2021-02-10 11:59:55.084592","last_peered":"2021-02-10 11:59:55.084592","last_clean":"2021-02-10 11:59:55.084592","last_became_active":"2021-02-10 11:58:02.783318","last_became_peered":"2021-02-10 11:58:02.783318","last_unstale":"2021-02-10 11:59:55.084592","last_undegraded":"2021-02-10 11:59:55.084592","last_fullsized":"2021-02-10 11:59:55.084592","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'12","last_scrub_stamp":"2021-02-10 11:59:55.084586","last_deep_scrub":"196'12","last_deep_scrub_stamp":"2021-02-10 11:59:55.084586","last_clean_scrub_stamp":"2021-02-10 11:59:55.084586","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.7","version":"196'21","reported_seq":"40","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.691659","last_change":"2021-02-10 11:59:43.691685","last_active":"2021-02-10 11:59:43.691659","last_peered":"2021-02-10 11:59:43.691659","last_clean":"2021-02-10 11:59:43.691659","last_became_active":"2021-02-10 11:58:04.419410","last_became_peered":"2021-02-10 11:58:04.419410","last_unstale":"2021-02-10 11:59:43.691659","last_undegraded":"2021-02-10 11:59:43.691659","last_fullsized":"2021-02-10 11:59:43.691659","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'21","last_scrub_stamp":"2021-02-10 11:59:43.691655","last_deep_scrub":"196'21","last_deep_scrub_stamp":"2021-02-10 11:59:43.691655","last_clean_scrub_stamp":"2021-02-10 11:59:43.691655","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14680064,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":7168,"num_write":14,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.6","version":"196'19","reported_seq":"33","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:59.239107","last_change":"2021-02-10 11:59:59.239131","last_active":"2021-02-10 11:59:59.239107","last_peered":"2021-02-10 11:59:59.239107","last_clean":"2021-02-10 11:59:59.239107","last_became_active":"2021-02-10 11:58:02.783694","last_became_peered":"2021-02-10 11:58:02.783694","last_unstale":"2021-02-10 11:59:59.239107","last_undegraded":"2021-02-10 11:59:59.239107","last_fullsized":"2021-02-10 11:59:59.239107","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'19","last_scrub_stamp":"2021-02-10 11:59:59.239103","last_deep_scrub":"196'19","last_deep_scrub_stamp":"2021-02-10 11:59:59.239103","last_clean_scrub_stamp":"2021-02-10 11:59:59.239103","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":386,"num_omap_keys":8,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"43.5","version":"196'34","reported_seq":"55","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:46.758404","last_change":"2021-02-10 11:59:46.758429","last_active":"2021-02-10 11:59:46.758404","last_peered":"2021-02-10 11:59:46.758404","last_clean":"2021-02-10 11:59:46.758404","last_became_active":"2021-02-10 11:58:04.421304","last_became_peered":"2021-02-10 11:58:04.421304","last_unstale":"2021-02-10 11:59:46.758404","last_undegraded":"2021-02-10 11:59:46.758404","last_fullsized":"2021-02-10 11:59:46.758404","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'34","last_scrub_stamp":"2021-02-10 11:59:46.758385","last_deep_scrub":"196'34","last_deep_scrub_stamp":"2021-02-10 11:59:46.758385","last_clean_scrub_stamp":"2021-02-10 11:59:46.758385","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16777506,"num_objects":18,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":21,"num_write_kb":16387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"42.4","version":"196'228","reported_seq":"242","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.777094","last_change":"2021-02-10 11:59:42.278366","last_active":"2021-02-10 11:59:43.777094","last_peered":"2021-02-10 11:59:43.777094","last_clean":"2021-02-10 11:59:43.777094","last_became_active":"2021-02-10 11:58:02.784457","last_became_peered":"2021-02-10 11:58:02.784457","last_unstale":"2021-02-10 11:59:43.777094","last_undegraded":"2021-02-10 11:59:43.777094","last_fullsized":"2021-02-10 11:59:43.777094","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'226","last_scrub_stamp":"2021-02-10 11:59:42.278332","last_deep_scrub":"196'226","last_deep_scrub_stamp":"2021-02-10 11:59:42.278332","last_clean_scrub_stamp":"2021-02-10 11:59:42.278332","log_size":228,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":937721,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":225,"num_write_kb":985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"43.2","version":"196'27","reported_seq":"47","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:34.727689","last_change":"2021-02-10 11:59:34.727715","last_active":"2021-02-10 11:59:34.727689","last_peered":"2021-02-10 11:59:34.727689","last_clean":"2021-02-10 11:59:34.727689","last_became_active":"2021-02-10 11:58:04.421381","last_became_peered":"2021-02-10 11:58:04.421381","last_unstale":"2021-02-10 11:59:34.727689","last_undegraded":"2021-02-10 11:59:34.727689","last_fullsized":"2021-02-10 11:59:34.727689","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'27","last_scrub_stamp":"2021-02-10 11:59:34.727686","last_deep_scrub":"196'27","last_deep_scrub_stamp":"2021-02-10 11:59:34.727686","last_clean_scrub_stamp":"2021-02-10 11:59:34.727686","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971520,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":20,"num_write_kb":20480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.3","version":"196'9","reported_seq":"23","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:43.312100","last_change":"2021-02-10 11:59:43.312125","last_active":"2021-02-10 11:59:43.312100","last_peered":"2021-02-10 11:59:43.312100","last_clean":"2021-02-10 11:59:43.312100","last_became_active":"2021-02-10 11:58:03.787526","last_became_peered":"2021-02-10 11:58:03.787526","last_unstale":"2021-02-10 11:59:43.312100","last_undegraded":"2021-02-10 11:59:43.312100","last_fullsized":"2021-02-10 11:59:43.312100","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'9","last_scrub_stamp":"2021-02-10 11:59:43.312097","last_deep_scrub":"196'9","last_deep_scrub_stamp":"2021-02-10 11:59:43.312097","last_clean_scrub_stamp":"2021-02-10 11:59:43.312097","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4620,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"43.3","version":"196'31","reported_seq":"51","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:40.721440","last_change":"2021-02-10 11:59:40.721485","last_active":"2021-02-10 11:59:40.721440","last_peered":"2021-02-10 11:59:40.721440","last_clean":"2021-02-10 11:59:40.721440","last_became_active":"2021-02-10 11:58:04.419046","last_became_peered":"2021-02-10 11:58:04.419046","last_unstale":"2021-02-10 11:59:40.721440","last_undegraded":"2021-02-10 11:59:40.721440","last_fullsized":"2021-02-10 11:59:40.721440","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'31","last_scrub_stamp":"2021-02-10 11:59:40.721436","last_deep_scrub":"196'31","last_deep_scrub_stamp":"2021-02-10 11:59:40.721436","last_clean_scrub_stamp":"2021-02-10 11:59:40.721436","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874505,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":8192,"num_write":25,"num_write_kb":18436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"42.2","version":"195'1","reported_seq":"15","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.706570","last_change":"2021-02-10 11:59:33.706603","last_active":"2021-02-10 11:59:33.706570","last_peered":"2021-02-10 11:59:33.706570","last_clean":"2021-02-10 11:59:33.706570","last_became_active":"2021-02-10 11:58:03.788701","last_became_peered":"2021-02-10 11:58:03.788701","last_unstale":"2021-02-10 11:59:33.706570","last_undegraded":"2021-02-10 11:59:33.706570","last_fullsized":"2021-02-10 11:59:33.706570","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'1","last_scrub_stamp":"2021-02-10 11:59:33.706566","last_deep_scrub":"195'1","last_deep_scrub_stamp":"2021-02-10 11:59:33.706566","last_clean_scrub_stamp":"2021-02-10 11:59:33.706566","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"43.0","version":"196'27","reported_seq":"49","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:44.363939","last_change":"2021-02-10 11:59:44.363968","last_active":"2021-02-10 11:59:44.363939","last_peered":"2021-02-10 11:59:44.363939","last_clean":"2021-02-10 11:59:44.363939","last_became_active":"2021-02-10 11:58:04.419010","last_became_peered":"2021-02-10 11:58:04.419010","last_unstale":"2021-02-10 11:59:44.363939","last_undegraded":"2021-02-10 11:59:44.363939","last_fullsized":"2021-02-10 11:59:44.363939","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'27","last_scrub_stamp":"2021-02-10 11:59:44.363936","last_deep_scrub":"196'27","last_deep_scrub_stamp":"2021-02-10 11:59:44.363936","last_clean_scrub_stamp":"2021-02-10 11:59:44.363936","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18874368,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":10240,"num_write":18,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"42.1","version":"196'2","reported_seq":"17","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:39.723713","last_change":"2021-02-10 11:59:39.723757","last_active":"2021-02-10 11:59:39.723713","last_peered":"2021-02-10 11:59:39.723713","last_clean":"2021-02-10 11:59:39.723713","last_became_active":"2021-02-10 11:58:03.789795","last_became_peered":"2021-02-10 11:58:03.789795","last_unstale":"2021-02-10 11:59:39.723713","last_undegraded":"2021-02-10 11:59:39.723713","last_fullsized":"2021-02-10 11:59:39.723713","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'2","last_scrub_stamp":"2021-02-10 11:59:39.723708","last_deep_scrub":"196'2","last_deep_scrub_stamp":"2021-02-10 11:59:39.723708","last_clean_scrub_stamp":"2021-02-10 11:59:39.723708","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"43.1","version":"196'21","reported_seq":"39","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 12:00:00.228786","last_change":"2021-02-10 12:00:00.228824","last_active":"2021-02-10 12:00:00.228786","last_peered":"2021-02-10 12:00:00.228786","last_clean":"2021-02-10 12:00:00.228786","last_became_active":"2021-02-10 11:58:04.418997","last_became_peered":"2021-02-10 11:58:04.418997","last_unstale":"2021-02-10 12:00:00.228786","last_undegraded":"2021-02-10 12:00:00.228786","last_fullsized":"2021-02-10 12:00:00.228786","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'21","last_scrub_stamp":"2021-02-10 12:00:00.228781","last_deep_scrub":"196'21","last_deep_scrub_stamp":"2021-02-10 12:00:00.228781","last_clean_scrub_stamp":"2021-02-10 12:00:00.228781","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"42.0","version":"196'12","reported_seq":"26","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:46.251848","last_change":"2021-02-10 11:59:46.251887","last_active":"2021-02-10 11:59:46.251848","last_peered":"2021-02-10 11:59:46.251848","last_clean":"2021-02-10 11:59:46.251848","last_became_active":"2021-02-10 11:58:02.783528","last_became_peered":"2021-02-10 11:58:02.783528","last_unstale":"2021-02-10 11:59:46.251848","last_undegraded":"2021-02-10 11:59:46.251848","last_fullsized":"2021-02-10 11:59:46.251848","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'12","last_scrub_stamp":"2021-02-10 11:59:46.251844","last_deep_scrub":"196'12","last_deep_scrub_stamp":"2021-02-10 11:59:46.251844","last_clean_scrub_stamp":"2021-02-10 11:59:46.251844","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"182","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:42.316945","last_change":"2021-02-10 11:59:42.316985","last_active":"2021-02-10 11:59:42.316945","last_peered":"2021-02-10 11:59:42.316945","last_clean":"2021-02-10 11:59:42.316945","last_became_active":"2021-02-10 11:22:24.884659","last_became_peered":"2021-02-10 11:22:24.884659","last_unstale":"2021-02-10 11:59:42.316945","last_undegraded":"2021-02-10 11:59:42.316945","last_fullsized":"2021-02-10 11:59:42.316945","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:42.316942","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:42.316942","last_clean_scrub_stamp":"2021-02-10 11:59:42.316942","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"182","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.345982","last_change":"2021-02-10 11:59:38.346013","last_active":"2021-02-10 11:59:38.345982","last_peered":"2021-02-10 11:59:38.345982","last_clean":"2021-02-10 11:59:38.345982","last_became_active":"2021-02-10 11:22:24.884979","last_became_peered":"2021-02-10 11:22:24.884979","last_unstale":"2021-02-10 11:59:38.345982","last_undegraded":"2021-02-10 11:59:38.345982","last_fullsized":"2021-02-10 11:59:38.345982","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:38.345978","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:38.345978","last_clean_scrub_stamp":"2021-02-10 11:59:38.345978","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"186","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.162264","last_change":"2021-02-10 11:59:38.162298","last_active":"2021-02-10 11:59:38.162264","last_peered":"2021-02-10 11:59:38.162264","last_clean":"2021-02-10 11:59:38.162264","last_became_active":"2021-02-10 11:22:24.885544","last_became_peered":"2021-02-10 11:22:24.885544","last_unstale":"2021-02-10 11:59:38.162264","last_undegraded":"2021-02-10 11:59:38.162264","last_fullsized":"2021-02-10 11:59:38.162264","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:38.162261","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:38.162261","last_clean_scrub_stamp":"2021-02-10 11:59:38.162261","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:35.057632","last_change":"2021-02-10 11:59:35.057663","last_active":"2021-02-10 11:59:35.057632","last_peered":"2021-02-10 11:59:35.057632","last_clean":"2021-02-10 11:59:35.057632","last_became_active":"2021-02-10 11:22:24.885765","last_became_peered":"2021-02-10 11:22:24.885765","last_unstale":"2021-02-10 11:59:35.057632","last_undegraded":"2021-02-10 11:59:35.057632","last_fullsized":"2021-02-10 11:59:35.057632","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:35.057629","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:35.057629","last_clean_scrub_stamp":"2021-02-10 11:59:35.057629","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:41.931591","last_change":"2021-02-10 11:59:41.931628","last_active":"2021-02-10 11:59:41.931591","last_peered":"2021-02-10 11:59:41.931591","last_clean":"2021-02-10 11:59:41.931591","last_became_active":"2021-02-10 11:22:24.885432","last_became_peered":"2021-02-10 11:22:24.885432","last_unstale":"2021-02-10 11:59:41.931591","last_undegraded":"2021-02-10 11:59:41.931591","last_fullsized":"2021-02-10 11:59:41.931591","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:41.931588","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:41.931588","last_clean_scrub_stamp":"2021-02-10 11:59:41.931588","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"43.4","version":"196'16","reported_seq":"34","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:38.133110","last_change":"2021-02-10 11:59:38.133137","last_active":"2021-02-10 11:59:38.133110","last_peered":"2021-02-10 11:59:38.133110","last_clean":"2021-02-10 11:59:38.133110","last_became_active":"2021-02-10 11:58:04.419401","last_became_peered":"2021-02-10 11:58:04.419401","last_unstale":"2021-02-10 11:59:38.133110","last_undegraded":"2021-02-10 11:59:38.133110","last_fullsized":"2021-02-10 11:59:38.133110","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'16","last_scrub_stamp":"2021-02-10 11:59:38.133106","last_deep_scrub":"196'16","last_deep_scrub_stamp":"2021-02-10 11:59:38.133106","last_clean_scrub_stamp":"2021-02-10 11:59:38.133106","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10485760,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":6144,"num_write":10,"num_write_kb":10240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"42.5","version":"196'13","reported_seq":"27","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:53.129763","last_change":"2021-02-10 11:59:53.129788","last_active":"2021-02-10 11:59:53.129763","last_peered":"2021-02-10 11:59:53.129763","last_clean":"2021-02-10 11:59:53.129763","last_became_active":"2021-02-10 11:58:02.783300","last_became_peered":"2021-02-10 11:58:02.783300","last_unstale":"2021-02-10 11:59:53.129763","last_undegraded":"2021-02-10 11:59:53.129763","last_fullsized":"2021-02-10 11:59:53.129763","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":192,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'13","last_scrub_stamp":"2021-02-10 11:59:53.129758","last_deep_scrub":"196'13","last_deep_scrub_stamp":"2021-02-10 11:59:53.129758","last_clean_scrub_stamp":"2021-02-10 11:59:53.129758","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"191","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:40.295061","last_change":"2021-02-10 11:59:40.295091","last_active":"2021-02-10 11:59:40.295061","last_peered":"2021-02-10 11:59:40.295061","last_clean":"2021-02-10 11:59:40.295061","last_became_active":"2021-02-10 11:22:24.884959","last_became_peered":"2021-02-10 11:22:24.884959","last_unstale":"2021-02-10 11:59:40.295061","last_undegraded":"2021-02-10 11:59:40.295061","last_fullsized":"2021-02-10 11:59:40.295061","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:40.295058","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:40.295058","last_clean_scrub_stamp":"2021-02-10 11:59:40.295058","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"180","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:48.048134","last_change":"2021-02-10 11:59:48.048165","last_active":"2021-02-10 11:59:48.048134","last_peered":"2021-02-10 11:59:48.048134","last_clean":"2021-02-10 11:59:48.048134","last_became_active":"2021-02-10 11:22:24.885418","last_became_peered":"2021-02-10 11:22:24.885418","last_unstale":"2021-02-10 11:59:48.048134","last_undegraded":"2021-02-10 11:59:48.048134","last_fullsized":"2021-02-10 11:59:48.048134","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:48.048131","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:48.048131","last_clean_scrub_stamp":"2021-02-10 11:59:48.048131","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"185","reported_epoch":"196","state":"active+clean","last_fresh":"2021-02-10 11:59:33.744224","last_change":"2021-02-10 11:59:33.744256","last_active":"2021-02-10 11:59:33.744224","last_peered":"2021-02-10 11:59:33.744224","last_clean":"2021-02-10 11:59:33.744224","last_became_active":"2021-02-10 11:22:24.885773","last_became_peered":"2021-02-10 11:22:24.885773","last_unstale":"2021-02-10 11:59:33.744224","last_undegraded":"2021-02-10 11:59:33.744224","last_fullsized":"2021-02-10 11:59:33.744224","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-10 11:59:33.744222","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-10 11:59:33.744222","last_clean_scrub_stamp":"2021-02-10 11:59:33.744222","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":43,"num_pg":8,"stat_sum":{"num_bytes":134218155,"num_objects":131,"num_object_clones":1,"num_object_copies":262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":64,"num_read_kb":65536,"num_write":140,"num_write_kb":131079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268460032,"data_stored":268436310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":16,"acting":16,"num_store_stats":8},{"poolid":42,"num_pg":8,"stat_sum":{"num_bytes":938582,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298,"num_write_kb":1013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5006,"num_omap_keys":18,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1941504,"data_stored":1877164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575308,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1087940,"kb_used_data":39356,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93283900,"statfs":{"total":96636764160,"available":95522713600,"internally_reserved":1073741824,"allocated":40300544,"data_stored":38903058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.91100000000000003,"15min":0.95899999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.33100000000000002},"max":{"1min":1.325,"5min":1.4219999999999999,"15min":1.534},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.95099999999999996,"15min":0.98899999999999999},"min":{"1min":0.54800000000000004,"5min":0.499,"15min":0.35899999999999999},"max":{"1min":1.296,"5min":1.304,"15min":1.498},"last":0.74099999999999999}]},{"osd":1,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82399999999999995,"15min":0.84199999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.372},"max":{"1min":1.1479999999999999,"5min":1.288,"15min":1.3400000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.83999999999999997,"15min":0.86199999999999999},"min":{"1min":0.48899999999999999,"5min":0.45500000000000002,"15min":0.35799999999999998},"max":{"1min":1.2110000000000001,"5min":1.411,"15min":1.4970000000000001},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82099999999999995,"15min":0.86299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.3680000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82099999999999995,"15min":0.85799999999999998},"min":{"1min":0.52600000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.92300000000000004,"5min":1.2010000000000001,"15min":1.288},"last":0.498}]},{"osd":3,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85999999999999999,"15min":0.873},"min":{"1min":0.52000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.1240000000000001,"5min":1.2370000000000001,"15min":1.357},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.85999999999999999,"15min":0.874},"min":{"1min":0.61099999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"max":{"1min":1.224,"5min":1.266,"15min":1.278},"last":0.55300000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67900000000000005,"15min":0.67500000000000004},"min":{"1min":0.443,"5min":0.376,"15min":0.376},"max":{"1min":1.0149999999999999,"5min":1.1200000000000001,"15min":1.286},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.69699999999999995},"min":{"1min":0.501,"5min":0.42199999999999999,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.46000000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.76900000000000002},"min":{"1min":0.48999999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.1020000000000001,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Wed Feb 10 11:59:46 2021","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.58699999999999997,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0940000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81299999999999994,"15min":0.81200000000000006},"min":{"1min":0.63400000000000001,"5min":0.54000000000000004,"15min":0.433},"max":{"1min":1.171,"5min":1.171,"15min":1.3720000000000001},"last":0.69499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1070416,"kb_used_data":21832,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93301424,"statfs":{"total":96636764160,"available":95540658176,"internally_reserved":1073741824,"allocated":22355968,"data_stored":20965488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.90300000000000002,"15min":0.90200000000000002},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.32000000000000001},"max":{"1min":1.407,"5min":1.407,"15min":1.5680000000000001},"last":0.627},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.92600000000000005,"15min":0.92800000000000005},"min":{"1min":0.42999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2529999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.81499999999999995,"15min":0.80600000000000005},"min":{"1min":0.54800000000000004,"5min":0.39600000000000002,"15min":0.39000000000000001},"max":{"1min":1.2270000000000001,"5min":1.2390000000000001,"15min":1.363},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.82699999999999996,"15min":0.83299999999999996},"min":{"1min":0.57099999999999995,"5min":0.373,"15min":0.373},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.45},"last":0.93600000000000005}]},{"osd":2,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.40300000000000002},"max":{"1min":1.0549999999999999,"5min":1.276,"15min":1.276},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.82099999999999995,"15min":0.81999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.433},"max":{"1min":1.26,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.86599999999999999,"15min":0.85999999999999999},"min":{"1min":0.47999999999999998,"5min":0.442,"15min":0.441},"max":{"1min":1.2390000000000001,"5min":1.2889999999999999,"15min":1.3859999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.60599999999999998,"5min":0.443,"15min":0.379},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.0349999999999999}]},{"osd":4,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71199999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0309999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.71299999999999997,"15min":0.70999999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":1.256,"15min":1.256},"last":1.1579999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98499999999999999,"5min":1.0049999999999999,"15min":1.0720000000000001},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.78200000000000003,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1359999999999999,"5min":1.3660000000000001,"15min":1.6379999999999999},"last":0.995}]},{"osd":7,"last update":"Wed Feb 10 11:59:41 2021","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.46300000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.165,"5min":1.256,"15min":1.256},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78700000000000003,"15min":0.80500000000000005},"min":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.499},"max":{"1min":1.1779999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":1.143}]}]},{"osd":5,"up_from":13,"seq":55834575307,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1083832,"kb_used_data":35248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93288008,"statfs":{"total":96636764160,"available":95526920192,"internally_reserved":1073741824,"allocated":36093952,"data_stored":34707957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.40699999999999997,"5min":0.38700000000000001,"15min":0.36299999999999999},"max":{"1min":1.224,"5min":1.363,"15min":1.365},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.82899999999999996,"15min":0.83499999999999996},"min":{"1min":0.432,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2749999999999999,"5min":1.399,"15min":1.399},"last":1.0600000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.84399999999999997,"15min":0.84699999999999998},"min":{"1min":0.55400000000000005,"5min":0.376,"15min":0.376},"max":{"1min":1.2430000000000001,"5min":1.3819999999999999,"15min":1.3979999999999999},"last":0.995},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.84499999999999997,"15min":0.85599999999999998},"min":{"1min":0.57999999999999996,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.3},"last":0.95299999999999996}]},{"osd":2,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.82299999999999995,"15min":0.82599999999999996},"min":{"1min":0.59799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.181,"5min":1.2330000000000001,"15min":1.2809999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.879,"15min":0.878},"min":{"1min":0.63,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2010000000000001,"5min":1.274,"15min":1.3500000000000001},"last":0.89700000000000002}]},{"osd":3,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.878,"15min":0.878},"min":{"1min":0.56399999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3660000000000001},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.83599999999999997,"15min":0.83999999999999997},"min":{"1min":0.625,"5min":0.433,"15min":0.41799999999999998},"max":{"1min":1.232,"5min":1.232,"15min":1.2669999999999999},"last":1.004}]},{"osd":4,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73399999999999999,"15min":0.76600000000000001},"min":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.44800000000000001},"max":{"1min":1.101,"5min":1.103,"15min":1.343},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.55200000000000005,"5min":0.47699999999999998,"15min":0.46999999999999997},"max":{"1min":1.1279999999999999,"5min":1.292,"15min":1.292},"last":0.83899999999999997}]},{"osd":6,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.75900000000000001},"min":{"1min":0.60399999999999998,"5min":0.438,"15min":0.438},"max":{"1min":1.278,"5min":1.278,"15min":1.3049999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.85499999999999998,"15min":0.86599999999999999},"min":{"1min":0.67200000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.206,"5min":1.2230000000000001,"15min":1.302},"last":0.80600000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:59:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.746,"15min":0.77700000000000002},"min":{"1min":0.501,"5min":0.376,"15min":0.376},"max":{"1min":1.085,"5min":1.2809999999999999,"15min":1.3220000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.73999999999999999,"15min":0.77700000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96599999999999997,"5min":1.147,"15min":1.1870000000000001},"last":0.72399999999999998}]}]},{"osd":4,"up_from":13,"seq":55834575307,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1074508,"kb_used_data":25924,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93297332,"statfs":{"total":96636764160,"available":95536467968,"internally_reserved":1073741824,"allocated":26546176,"data_stored":25159758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91800000000000004,"15min":0.93100000000000005},"min":{"1min":0.51300000000000001,"5min":0.379,"15min":0.34499999999999997},"max":{"1min":1.3600000000000001,"5min":1.407,"15min":1.4099999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.046,"5min":0.94299999999999995,"15min":0.95399999999999996},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.34599999999999997},"max":{"1min":1.3089999999999999,"5min":1.335,"15min":1.3879999999999999},"last":1.0820000000000001}]},{"osd":1,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.84899999999999998,"15min":0.84999999999999998},"min":{"1min":0.54500000000000004,"5min":0.435,"15min":0.372},"max":{"1min":1.222,"5min":1.2529999999999999,"15min":1.341},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.81499999999999995,"15min":0.82999999999999996},"min":{"1min":0.46200000000000002,"5min":0.40000000000000002,"15min":0.38800000000000001},"max":{"1min":1.119,"5min":1.181,"15min":1.2869999999999999},"last":0.90200000000000002}]},{"osd":2,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.81299999999999994,"15min":0.82699999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.83899999999999997,"15min":0.85399999999999998},"min":{"1min":0.57899999999999996,"5min":0.46800000000000003,"15min":0.432},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4570000000000001},"last":0.91300000000000003}]},{"osd":3,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.81999999999999995,"15min":0.82799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.83699999999999997,"15min":0.85699999999999998},"min":{"1min":0.60199999999999998,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2569999999999999,"5min":1.321,"15min":1.3220000000000001},"last":0.76800000000000002}]},{"osd":5,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.318},"max":{"1min":0.96399999999999997,"5min":1.0900000000000001,"15min":1.141},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96099999999999997,"5min":1.042,"15min":1.1279999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.70699999999999996,"15min":0.71999999999999997},"min":{"1min":0.51400000000000001,"5min":0.438,"15min":0.41699999999999998},"max":{"1min":0.93300000000000005,"5min":1.113,"15min":1.204},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.76400000000000001,"15min":0.77500000000000002},"min":{"1min":0.58999999999999997,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.0720000000000001,"15min":1.2729999999999999},"last":0.55300000000000005}]},{"osd":7,"last update":"Wed Feb 10 11:59:22 2021","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.748,"15min":0.75900000000000001},"min":{"1min":0.502,"5min":0.44800000000000001,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78800000000000003,"15min":0.81399999999999995},"min":{"1min":0.496,"5min":0.432,"15min":0.432},"max":{"1min":1.0249999999999999,"5min":1.165,"15min":1.321},"last":0.49399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575308,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1061300,"kb_used_data":12716,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93310540,"statfs":{"total":96636764160,"available":95549992960,"internally_reserved":1073741824,"allocated":13021184,"data_stored":11639159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.72499999999999998,"5min":0.78500000000000003,"15min":0.94399999999999995},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78100000000000003,"5min":0.82999999999999996,"15min":1.319},"last":0.624}]},{"osd":2,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.35199999999999998,"5min":0.33100000000000002,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.85099999999999998,"15min":0.91800000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.39800000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77100000000000002,"5min":0.83999999999999997,"15min":1.1599999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52500000000000002,"15min":0.52400000000000002},"min":{"1min":0.39000000000000001,"5min":0.377,"15min":0.33100000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.89600000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64100000000000001,"15min":0.63400000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.91300000000000003,"5min":0.96799999999999997,"15min":1.077},"last":0.69799999999999995}]},{"osd":4,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.97599999999999998,"15min":0.96299999999999997},"min":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.503},"max":{"1min":1.393,"5min":1.393,"15min":1.4039999999999999},"last":1.159},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96899999999999997,"15min":0.96299999999999997},"min":{"1min":0.59199999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.4219999999999999,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.95399999999999996,"15min":0.94999999999999996},"min":{"1min":0.71299999999999997,"5min":0.504,"15min":0.48099999999999998},"max":{"1min":1.319,"5min":1.4690000000000001,"15min":1.5660000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.044,"5min":0.98799999999999999,"15min":0.98199999999999998},"min":{"1min":0.61599999999999999,"5min":0.54700000000000004,"15min":0.497},"max":{"1min":1.379,"5min":1.379,"15min":1.512},"last":0.96599999999999997}]},{"osd":6,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.64300000000000002,"5min":0.61799999999999999,"15min":0.48999999999999999},"max":{"1min":1.363,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.147},{"interface":"front","average":{"1min":1.071,"5min":1.0669999999999999,"15min":1.079},"min":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.53400000000000003},"max":{"1min":1.3879999999999999,"5min":1.7490000000000001,"15min":1.8100000000000001},"last":1.139}]},{"osd":7,"last update":"Wed Feb 10 11:59:45 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.01,"15min":1.004},"min":{"1min":0.754,"5min":0.622,"15min":0.54900000000000004},"max":{"1min":1.417,"5min":1.615,"15min":1.909},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98999999999999999,"15min":0.99299999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.46800000000000003},"max":{"1min":1.4590000000000001,"5min":1.538,"15min":1.538},"last":1.089}]}]},{"osd":1,"up_from":13,"seq":55834575307,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1092032,"kb_used_data":43444,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93279808,"statfs":{"total":96636764160,"available":95518523392,"internally_reserved":1073741824,"allocated":44486656,"data_stored":43096796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.60699999999999998,"15min":0.60299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73599999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63200000000000001,"15min":0.62},"min":{"1min":0.441,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90000000000000002,"5min":0.92400000000000004,"15min":1.052},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.48599999999999999,"15min":0.48499999999999999},"min":{"1min":0.34399999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54200000000000004,"5min":1.071,"15min":1.071},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":0.61799999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66900000000000004}]},{"osd":3,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.49299999999999999,"5min":0.39100000000000001,"15min":0.35099999999999998},"max":{"1min":0.77100000000000002,"5min":0.877,"15min":1.077},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54500000000000004,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.34200000000000003},"max":{"1min":0.67800000000000005,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.59499999999999997}]},{"osd":4,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.073,"15min":1.0449999999999999},"min":{"1min":0.72799999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3100000000000001,"5min":1.46,"15min":1.8919999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.038,"15min":1.0249999999999999},"min":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2789999999999999,"5min":1.3799999999999999,"15min":1.6080000000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.53900000000000003,"15min":0.48799999999999999},"max":{"1min":1.1519999999999999,"5min":1.3149999999999999,"15min":1.391},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.97899999999999998,"15min":0.95499999999999996},"min":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.48999999999999999},"max":{"1min":1.403,"5min":1.403,"15min":1.7350000000000001},"last":0.93000000000000005}]},{"osd":6,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.0620000000000001,"15min":1.0389999999999999},"min":{"1min":0.54300000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4670000000000001,"5min":1.528,"15min":1.6910000000000001},"last":1.103},{"interface":"front","average":{"1min":0.97299999999999998,"5min":1.0669999999999999,"15min":1.048},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64200000000000002},"max":{"1min":1.4510000000000001,"5min":1.4930000000000001,"15min":1.506},"last":1.204}]},{"osd":7,"last update":"Wed Feb 10 11:59:32 2021","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.99099999999999999,"15min":0.98199999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.61799999999999999},"max":{"1min":1.1990000000000001,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.93300000000000005,"5min":1.006,"15min":0.97199999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.54200000000000004},"max":{"1min":1.2430000000000001,"5min":1.3959999999999999,"15min":1.5529999999999999},"last":0.93799999999999994}]}]},{"osd":2,"up_from":13,"seq":55834575308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1122756,"kb_used_data":74172,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93249084,"statfs":{"total":96636764160,"available":95487062016,"internally_reserved":1073741824,"allocated":75952128,"data_stored":74554548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61899999999999999,"15min":0.60699999999999998},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81200000000000006,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.44500000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.90300000000000002,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.60099999999999998}]},{"osd":1,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.54200000000000004},"min":{"1min":0.36199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69599999999999995,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.371,"15min":0.314},"max":{"1min":0.69599999999999995,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.52100000000000002},"min":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.33600000000000002},"max":{"1min":0.82499999999999996,"5min":1.671,"15min":1.671},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63700000000000001,"15min":0.61899999999999999},"min":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.40500000000000003},"max":{"1min":0.76700000000000002,"5min":2.133,"15min":2.133},"last":0.45000000000000001}]},{"osd":4,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.95999999999999996,"15min":0.97899999999999998},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.47899999999999998},"max":{"1min":1.276,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.97599999999999998,"15min":0.96299999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.45100000000000001},"max":{"1min":1.274,"5min":2.1469999999999998,"15min":2.1469999999999998},"last":1.087}]},{"osd":5,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1020000000000001,"15min":1.095},"min":{"1min":0.77500000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.397,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.071,"15min":1.0720000000000001},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.411,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.321}]},{"osd":6,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.96699999999999997,"15min":0.96199999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.347,"5min":2.161,"15min":2.161},"last":1.238},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.999,"15min":0.996},"min":{"1min":0.73299999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.228,"5min":2.2999999999999998,"15min":2.2999999999999998},"last":1.2869999999999999}]},{"osd":7,"last update":"Wed Feb 10 11:59:48 2021","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98699999999999999},"min":{"1min":0.71299999999999997,"5min":0.624,"15min":0.58999999999999997},"max":{"1min":1.343,"5min":2.335,"15min":2.335},"last":1.198},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.97799999999999998,"15min":0.99199999999999999},"min":{"1min":0.73299999999999998,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.3580000000000001,"5min":2.2709999999999999,"15min":2.2709999999999999},"last":1.115}]}]},{"osd":3,"up_from":13,"seq":55834575306,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1079732,"kb_used_data":31148,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93292108,"statfs":{"total":96636764160,"available":95531118592,"internally_reserved":1073741824,"allocated":31895552,"data_stored":30513630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60399999999999998,"15min":0.61299999999999999},"min":{"1min":0.37,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":0.97699999999999998,"5min":1.006,"15min":1.006},"last":0.624},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.436,"5min":0.40899999999999997,"15min":0.35999999999999999},"max":{"1min":0.871,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.503}]},{"osd":1,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.32300000000000001},"max":{"1min":0.70399999999999996,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.33300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.91800000000000004,"15min":1.0429999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.375,"15min":0.374},"max":{"1min":0.71999999999999997,"5min":0.95399999999999996,"15min":1.018},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.371,"15min":0.36099999999999999},"max":{"1min":0.86099999999999999,"5min":0.97099999999999997,"15min":1.0309999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.94499999999999995,"15min":0.94899999999999995},"min":{"1min":0.59299999999999997,"5min":0.45400000000000001,"15min":0.39200000000000002},"max":{"1min":1.3080000000000001,"5min":1.387,"15min":1.5680000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.94099999999999995,"15min":0.94499999999999995},"min":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.2430000000000001,"5min":1.3360000000000001,"15min":1.4450000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.97999999999999998,"15min":0.98499999999999999},"min":{"1min":0.755,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96699999999999997,"15min":0.96899999999999997},"min":{"1min":0.65700000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.4790000000000001},"last":1.0880000000000001}]},{"osd":6,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.97099999999999997,"15min":0.97299999999999998},"min":{"1min":0.64400000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.304,"5min":1.3200000000000001,"15min":1.4830000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0069999999999999,"15min":1},"min":{"1min":0.67200000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.5880000000000001},"last":1.0720000000000001}]},{"osd":7,"last update":"Wed Feb 10 11:59:28 2021","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.98999999999999999,"15min":0.98099999999999998},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.23,"5min":1.2589999999999999,"15min":1.5880000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.256,"5min":1.349,"15min":1.4510000000000001},"last":1.0109999999999999}]}]}]}} 2021-02-10T12:00:20.409 DEBUG:teuthology.orchestra.run.gibba017:> sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell mon.* injectargs -- --no-mon-health-to-clog 2021-02-10T12:00:20.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.817 7f8438f19700 1 Processor -- start 2021-02-10T12:00:20.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.817 7f8438f19700 1 -- start start 2021-02-10T12:00:20.837 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8438f19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8438f19700 1 --2- >> v2:172.21.2.125:3300/0 conn(0x7f84340f5900 0x7f84340f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8438f19700 1 -- --> v1:172.21.2.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84340f5380 con 0x7f84340e4f10 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8438f19700 1 -- --> v2:172.21.2.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84341b3f20 con 0x7f84340f5900 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8438f19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84340d1650 con 0x7f84340e5310 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f842a312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f842a312700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.125:3301/0 says I am v2:172.21.2.117:59448/0 (socket says 172.21.2.117:59448) 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f842a312700 1 -- 172.21.2.117:0/3198921145 learned_addr learned my addr 172.21.2.117:0/3198921145 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T12:00:20.838 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.818 7f8433314700 1 --2- 172.21.2.117:0/3198921145 >> v2:172.21.2.125:3300/0 conn(0x7f84340f5900 0x7f84340f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f842a312700 1 -- 172.21.2.117:0/3198921145 >> v2:172.21.2.125:3300/0 conn(0x7f84340f5900 msgr2=0x7f84340f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f842a312700 1 --2- 172.21.2.117:0/3198921145 >> v2:172.21.2.125:3300/0 conn(0x7f84340f5900 0x7f84340f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f842a312700 1 -- 172.21.2.117:0/3198921145 >> v1:172.21.2.125:6789/0 conn(0x7f84340e4f10 legacy=0x7f84341b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f842a312700 1 -- 172.21.2.117:0/3198921145 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84340d1870 con 0x7f84340e5310 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f842a312700 1 --2- 172.21.2.117:0/3198921145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f841c006700 tx=0x7f841c009b80).ready entity=mon.2 client_cookie=aff59ae68b457bcf server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f8432312700 1 -- 172.21.2.117:0/3198921145 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f841c00c0c0 con 0x7f84340e5310 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f8432312700 1 -- 172.21.2.117:0/3198921145 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c00c340 con 0x7f84340e5310 2021-02-10T12:00:20.839 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f8432312700 1 -- 172.21.2.117:0/3198921145 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f841c00c860 con 0x7f84340e5310 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f8438f19700 1 -- 172.21.2.117:0/3198921145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 msgr2=0x7f84340f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.819 7f8438f19700 1 --2- 172.21.2.117:0/3198921145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f841c006700 tx=0x7f841c009b80).stop 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 -- 172.21.2.117:0/3198921145 shutdown_connections 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 --2- 172.21.2.117:0/3198921145 >> v2:172.21.2.125:3300/0 conn(0x7f84340f5900 0x7f84340f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 --2- 172.21.2.117:0/3198921145 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f84340e5310 0x7f84340f4ce0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 -- 172.21.2.117:0/3198921145 >> 172.21.2.117:0/3198921145 conn(0x7f84340ce570 msgr2=0x7f84340e43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 -- 172.21.2.117:0/3198921145 shutdown_connections 2021-02-10T12:00:20.840 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.820 7f8438f19700 1 -- 172.21.2.117:0/3198921145 wait complete. 2021-02-10T12:00:20.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.835 7f8438f19700 1 Processor -- start 2021-02-10T12:00:20.842 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.835 7f8438f19700 1 -- start start 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.835 7f8438f19700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8438f19700 1 --2- >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8438f19700 1 --2- >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8438f19700 1 -- --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843407e4c0 con 0x7f8434101030 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8438f19700 1 -- --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8434101470 con 0x7f843407de80 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8438f19700 1 -- --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84340b2660 con 0x7f843407da80 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.843 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 --2- >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.2.117:3300/0 says I am v2:172.21.2.117:33540/0 (socket says 172.21.2.117:33540) 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 -- 172.21.2.117:0/380992900 learned_addr learned my addr 172.21.2.117:0/380992900 (peer_addr_for_me v2:172.21.2.117:0/0) 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 msgr2=0x7f843407e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f842a312700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8433314700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 msgr2=0x7f843407efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:20.844 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434101790 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f842a312700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8433314700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.836 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f842400b5b0 tx=0x7f842400b950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f842400bd40 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424006020 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84240054a0 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8434101790 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843407f8f0 con 0x7f8434101030 2021-02-10T12:00:20.845 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.837 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8424006200 con 0x7f8434101030 2021-02-10T12:00:20.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.838 7f842bfff700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:20.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.838 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 5920+0+0 (secure 0 0 0) 0x7f84240325a0 con 0x7f8434101030 2021-02-10T12:00:20.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.838 7f842a312700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:20.846 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.838 7f842a312700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 0x7f841801abc0 crc :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:20.870 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.868 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f843407cdc0 con 0x7f8434101030 2021-02-10T12:00:20.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.868 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+968 (secure 0 0 0) 0x7f8424000960 con 0x7f8434101030 2021-02-10T12:00:20.871 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.869 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84340f5380 con 0x7f8434101030 2021-02-10T12:00:20.874 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:20.872 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f8424000940 con 0x7f8434101030 2021-02-10T12:00:21.004 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.001 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f843407cdc0 con 0x7f8434101030 2021-02-10T12:00:21.005 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.003 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.0 v2:172.21.2.117:3300/0 8 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f84240061e0 con 0x7f8434101030 2021-02-10T12:00:21.007 INFO:teuthology.orchestra.run.gibba017.stderr:mon.a: injectargs:mon_health_to_clog = 'false' 2021-02-10T12:00:21.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.005 7f8438f19700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 msgr2=0x7f843407d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:21.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.005 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 rx=0x7f842400b5b0 tx=0x7f842400b950).stop 2021-02-10T12:00:21.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.005 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:21.007 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.005 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843407fe90 con 0x7f843407de80 2021-02-10T12:00:21.008 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.006 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:21.009 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.007 7f8432b13700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f843407f8f0 con 0x7f843407de80 2021-02-10T12:00:21.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.007 7f8432b13700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842401f370 con 0x7f843407de80 2021-02-10T12:00:21.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.008 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f842401fa90 tx=0x7f842401f9a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:21.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.008 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.1 v2:172.21.2.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f84240054a0 con 0x7f843407de80 2021-02-10T12:00:21.010 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.008 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.1 v2:172.21.2.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842400ebf0 con 0x7f843407de80 2021-02-10T12:00:21.011 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.008 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.1 v2:172.21.2.125:3300/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f8424012080 con 0x7f843407de80 2021-02-10T12:00:21.014 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.012 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.1 v2:172.21.2.125:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84240255f0 con 0x7f843407de80 2021-02-10T12:00:21.143 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.141 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f84340f5380 con 0x7f843407de80 2021-02-10T12:00:21.145 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.143 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.1 v2:172.21.2.125:3300/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f84240255f0 con 0x7f843407de80 2021-02-10T12:00:21.146 INFO:teuthology.orchestra.run.gibba017.stderr:mon.b: injectargs:mon_health_to_clog = 'false' 2021-02-10T12:00:21.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.144 7f8438f19700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 msgr2=0x7f843407efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:21.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.144 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 rx=0x7f842401fa90 tx=0x7f842401f9a0).stop 2021-02-10T12:00:21.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.145 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-02-10T12:00:21.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.145 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843407fe90 con 0x7f843407da80 2021-02-10T12:00:21.147 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.145 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-02-10T12:00:21.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.145 7f8432b13700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f843407cdc0 con 0x7f843407da80 2021-02-10T12:00:21.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.145 7f8432b13700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84240255f0 con 0x7f843407da80 2021-02-10T12:00:21.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.146 7f8432b13700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f84240135e0 tx=0x7f8424013870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-02-10T12:00:21.148 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.146 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.2 v2:172.21.2.125:3301/0 1 ==== mon_map magic: 0 v1 ==== 377+0+0 (secure 0 0 0) 0x7f8424022b60 con 0x7f843407da80 2021-02-10T12:00:21.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.146 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.2 v2:172.21.2.125:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842401bf80 con 0x7f843407da80 2021-02-10T12:00:21.149 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.146 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.2 v2:172.21.2.125:3301/0 3 ==== mgrmap(e 3) v1 ==== 33230+0+0 (secure 0 0 0) 0x7f842401d140 con 0x7f843407da80 2021-02-10T12:00:21.151 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.149 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.2 v2:172.21.2.125:3301/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111044 (secure 0 0 0) 0x7f84240212d0 con 0x7f843407da80 2021-02-10T12:00:21.283 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.280 7f8438f19700 1 -- 172.21.2.117:0/380992900 --> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] -- mon_command({"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"} v 0) v1 -- 0x7f84340f5380 con 0x7f843407da80 2021-02-10T12:00:21.284 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.282 7f842bfff700 1 -- 172.21.2.117:0/380992900 <== mon.2 v2:172.21.2.125:3301/0 5 ==== mon_command_ack([{"injected_args": ["--no-mon-health-to-clog"], "prefix": "injectargs"}]=0 injectargs:mon_health_to_clog = 'false' v0) v1 ==== 144+0+0 (secure 0 0 0) 0x7f8424020290 con 0x7f843407da80 2021-02-10T12:00:21.286 INFO:teuthology.orchestra.run.gibba017.stderr:mon.c: injectargs:mon_health_to_clog = 'false' 2021-02-10T12:00:21.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 msgr2=0x7f841801abc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:21.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 0x7f841801abc0 crc :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:21.288 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 msgr2=0x7f843407e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f84240135e0 tx=0x7f8424013870).stop 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 shutdown_connections 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3301/0,v1:172.21.2.125:6790/0] conn(0x7f843407da80 0x7f843407e990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:3300/0,v1:172.21.2.125:6789/0] conn(0x7f843407de80 0x7f843407efe0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.125:6800/34993,v1:172.21.2.125:6801/34993] conn(0x7f84180186c0 0x7f841801abc0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 --2- 172.21.2.117:0/380992900 >> [v2:172.21.2.117:3300/0,v1:172.21.2.117:6789/0] conn(0x7f8434101030 0x7f843407d460 unknown :-1 s=CLOSED pgs=3352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-02-10T12:00:21.289 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 >> 172.21.2.117:0/380992900 conn(0x7f8434095e30 msgr2=0x7f843409ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-02-10T12:00:21.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 shutdown_connections 2021-02-10T12:00:21.290 INFO:teuthology.orchestra.run.gibba017.stderr:2021-02-10 12:00:21.286 7f8438f19700 1 -- 172.21.2.117:0/380992900 wait complete. 2021-02-10T12:00:21.297 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-10T12:00:21.297 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2021-02-10T12:00:21.298 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.304 INFO:tasks.ceph.mds.a:Stopped 2021-02-10T12:00:21.304 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2021-02-10T12:00:21.304 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.311 INFO:tasks.ceph.mds.b:Stopped 2021-02-10T12:00:21.311 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-10T12:00:21.311 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-02-10T12:00:21.311 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.379 INFO:tasks.ceph.osd.0:Stopped 2021-02-10T12:00:21.379 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-02-10T12:00:21.380 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.444 INFO:tasks.ceph.osd.1:Stopped 2021-02-10T12:00:21.445 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-02-10T12:00:21.445 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.510 INFO:tasks.ceph.osd.2:Stopped 2021-02-10T12:00:21.510 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-02-10T12:00:21.510 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.571 INFO:tasks.ceph.osd.3:Stopped 2021-02-10T12:00:21.572 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2021-02-10T12:00:21.572 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.591 INFO:tasks.ceph.osd.4:Stopped 2021-02-10T12:00:21.591 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2021-02-10T12:00:21.591 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.657 INFO:tasks.ceph.osd.5:Stopped 2021-02-10T12:00:21.657 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2021-02-10T12:00:21.657 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.720 INFO:tasks.ceph.osd.6:Stopped 2021-02-10T12:00:21.721 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2021-02-10T12:00:21.721 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.782 INFO:tasks.ceph.osd.7:Stopped 2021-02-10T12:00:21.782 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-10T12:00:21.782 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2021-02-10T12:00:21.782 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.796 INFO:tasks.ceph.mgr.y:Stopped 2021-02-10T12:00:21.796 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2021-02-10T12:00:21.796 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.841 INFO:tasks.ceph.mgr.x:Stopped 2021-02-10T12:00:21.841 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-10T12:00:21.841 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-02-10T12:00:21.841 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.851 INFO:tasks.ceph.mon.a:Stopped 2021-02-10T12:00:21.851 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-02-10T12:00:21.851 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.888 INFO:tasks.ceph.mon.b:Stopped 2021-02-10T12:00:21.889 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-02-10T12:00:21.889 INFO:teuthology.orchestra.run:waiting for 300 2021-02-10T12:00:21.936 INFO:tasks.ceph.mon.c:Stopped 2021-02-10T12:00:21.937 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-10T12:00:21.937 DEBUG:teuthology.orchestra.run.gibba017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2021-02-10T12:00:22.126 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:22.130 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:22.284 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T12:00:22.285 DEBUG:teuthology.orchestra.run.gibba017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2021-02-10T12:00:22.758 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T12:00:22.758 DEBUG:teuthology.orchestra.run.gibba017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2021-02-10T12:00:23.166 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T12:00:23.167 DEBUG:teuthology.orchestra.run.gibba017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-02-10T12:00:23.576 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T12:00:23.576 DEBUG:teuthology.orchestra.run.gibba017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2021-02-10T12:00:23.976 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T12:00:23.977 DEBUG:teuthology.orchestra.run.gibba025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2021-02-10T12:00:24.444 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T12:00:24.444 DEBUG:teuthology.orchestra.run.gibba025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2021-02-10T12:00:24.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T12:00:24.850 DEBUG:teuthology.orchestra.run.gibba025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2021-02-10T12:00:25.250 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T12:00:25.250 DEBUG:teuthology.orchestra.run.gibba025:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2021-02-10T12:00:25.663 INFO:tasks.ceph:Archiving mon data... 2021-02-10T12:00:25.664 DEBUG:teuthology.misc:Transferring archived files from gibba017:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/data/mon.a.tgz 2021-02-10T12:00:25.664 DEBUG:teuthology.orchestra.run.gibba017:> mktemp 2021-02-10T12:00:26.000 INFO:teuthology.orchestra.run.gibba017.stdout:/tmp/tmp.TELQWrw5ku 2021-02-10T12:00:26.000 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TELQWrw5ku 2021-02-10T12:00:26.566 DEBUG:teuthology.orchestra.run.gibba017:> sudo chmod 0666 /tmp/tmp.TELQWrw5ku 2021-02-10T12:00:27.249 DEBUG:teuthology.orchestra.remote:gibba017:/tmp/tmp.TELQWrw5ku is 1MB 2021-02-10T12:00:27.644 DEBUG:teuthology.orchestra.run.gibba017:> rm -fr /tmp/tmp.TELQWrw5ku 2021-02-10T12:00:27.979 DEBUG:teuthology.misc:Transferring archived files from gibba025:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/data/mon.b.tgz 2021-02-10T12:00:27.980 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T12:00:28.315 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.1F35ZfJsDr 2021-02-10T12:00:28.315 DEBUG:teuthology.orchestra.run.gibba025:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1F35ZfJsDr 2021-02-10T12:00:28.888 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.1F35ZfJsDr 2021-02-10T12:00:29.567 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.1F35ZfJsDr is 1MB 2021-02-10T12:00:29.937 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.1F35ZfJsDr 2021-02-10T12:00:30.271 DEBUG:teuthology.misc:Transferring archived files from gibba025:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/data/mon.c.tgz 2021-02-10T12:00:30.271 DEBUG:teuthology.orchestra.run.gibba025:> mktemp 2021-02-10T12:00:30.606 INFO:teuthology.orchestra.run.gibba025.stdout:/tmp/tmp.NJktgMTtjy 2021-02-10T12:00:30.607 DEBUG:teuthology.orchestra.run.gibba025:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NJktgMTtjy 2021-02-10T12:00:31.170 DEBUG:teuthology.orchestra.run.gibba025:> sudo chmod 0666 /tmp/tmp.NJktgMTtjy 2021-02-10T12:00:31.849 DEBUG:teuthology.orchestra.remote:gibba025:/tmp/tmp.NJktgMTtjy is 1MB 2021-02-10T12:00:32.243 DEBUG:teuthology.orchestra.run.gibba025:> rm -fr /tmp/tmp.NJktgMTtjy 2021-02-10T12:00:32.577 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-10T12:00:32.578 DEBUG:teuthology.orchestra.run.gibba017:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T12:00:32.581 DEBUG:teuthology.orchestra.run.gibba025:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-10T12:00:32.924 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-10T12:00:32.925 DEBUG:teuthology.orchestra.run.gibba017:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T12:00:32.927 DEBUG:teuthology.orchestra.run.gibba025:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-10T12:00:33.270 INFO:teuthology.orchestra.run.gibba017.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T12:00:33.272 INFO:teuthology.orchestra.run.gibba025.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-10T12:00:33.277 INFO:tasks.ceph:Archiving crash dumps... 2021-02-10T12:00:33.279 DEBUG:teuthology.misc:Transferring archived files from gibba017:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba017/crash 2021-02-10T12:00:33.280 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T12:00:33.626 DEBUG:teuthology.misc:Transferring archived files from gibba025:/var/lib/ceph/crash to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba025/crash 2021-02-10T12:00:33.627 DEBUG:teuthology.orchestra.run.gibba025:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-10T12:00:33.971 INFO:tasks.ceph:Shutting down logrotate 2021-02-10T12:00:33.972 DEBUG:teuthology.orchestra.run.gibba017:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:33.976 DEBUG:teuthology.orchestra.run.gibba025:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:34.346 DEBUG:teuthology.orchestra.run.gibba017:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:34.690 DEBUG:teuthology.orchestra.run.gibba025:> sudo rm /etc/logrotate.d/ceph-test.conf 2021-02-10T12:00:35.033 INFO:tasks.ceph:Compressing logs... 2021-02-10T12:00:35.034 DEBUG:teuthology.orchestra.run.gibba017:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T12:00:35.038 DEBUG:teuthology.orchestra.run.gibba025:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T12:00:54.882 INFO:tasks.ceph:Archiving logs... 2021-02-10T12:00:54.883 DEBUG:teuthology.misc:Transferring archived files from gibba017:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba017/log 2021-02-10T12:00:54.884 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T12:00:58.905 DEBUG:teuthology.misc:Transferring archived files from gibba025:/var/log/ceph to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba025/log 2021-02-10T12:00:58.906 DEBUG:teuthology.orchestra.run.gibba025:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-10T12:01:02.732 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-10T12:01:02.744 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-10T12:01:02.744 DEBUG:teuthology.orchestra.run.gibba017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T12:01:02.748 DEBUG:teuthology.orchestra.run.gibba025:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-10T12:01:03.148 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-10T12:01:03.159 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-10T12:01:03.160 DEBUG:teuthology.orchestra.run.gibba017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T12:01:03.504 INFO:teuthology.orchestra.run.gibba017.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T12:01:03.505 INFO:teuthology.orchestra.run.gibba017.stdout:============================================================================== 2021-02-10T12:01:03.505 INFO:teuthology.orchestra.run.gibba017.stdout:+hv01.front.sepi 216.240.36.24 3 u 91 128 377 0.090 -0.432 3.019 2021-02-10T12:01:03.505 INFO:teuthology.orchestra.run.gibba017.stdout:*hv02.front.sepi 163.237.218.18 2 u 89 128 377 0.083 1.155 3.588 2021-02-10T12:01:03.505 INFO:teuthology.orchestra.run.gibba017.stdout:+hv03.front.sepi 74.6.168.72 3 u 88 128 377 0.077 1.175 2.970 2021-02-10T12:01:03.506 INFO:teuthology.orchestra.run.gibba017.stdout:-hv04.front.sepi 144.172.126.201 3 u 19 128 377 0.070 1.275 2.705 2021-02-10T12:01:03.506 DEBUG:teuthology.orchestra.run.gibba025:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-10T12:01:03.850 INFO:teuthology.orchestra.run.gibba025.stdout: remote refid st t when poll reach delay offset jitter 2021-02-10T12:01:03.851 INFO:teuthology.orchestra.run.gibba025.stdout:============================================================================== 2021-02-10T12:01:03.851 INFO:teuthology.orchestra.run.gibba025.stdout:+hv01.front.sepi 216.240.36.24 3 u 22 64 377 0.095 3.129 2.425 2021-02-10T12:01:03.851 INFO:teuthology.orchestra.run.gibba025.stdout:*hv02.front.sepi 163.237.218.18 2 u 23 64 377 0.070 3.309 2.358 2021-02-10T12:01:03.851 INFO:teuthology.orchestra.run.gibba025.stdout:-hv03.front.sepi 74.6.168.72 3 u 16 64 377 0.065 5.029 2.659 2021-02-10T12:01:03.851 INFO:teuthology.orchestra.run.gibba025.stdout:+hv04.front.sepi 144.172.126.201 3 u 63 64 377 0.066 0.595 2.265 2021-02-10T12:01:03.852 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-10T12:01:03.863 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-10T12:01:03.863 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-10T12:01:03.876 DEBUG:teuthology.orchestra.run.gibba017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T12:01:04.243 DEBUG:teuthology.orchestra.run.gibba025:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-10T12:01:04.643 DEBUG:teuthology.orchestra.run.gibba017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T12:01:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T12:01:04.989 DEBUG:teuthology.orchestra.run.gibba025:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\)' 2021-02-10T12:01:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T12:01:05.336 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-10T12:01:05.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-10T12:01:05.372 INFO:teuthology.task.internal:Duration was 3008.220671 seconds 2021-02-10T12:01:05.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-10T12:01:05.386 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-10T12:01:05.386 DEBUG:teuthology.orchestra.run.gibba017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T12:01:05.390 DEBUG:teuthology.orchestra.run.gibba025:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-10T12:01:05.748 INFO:teuthology.orchestra.run.gibba017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T12:01:05.749 INFO:teuthology.orchestra.run.gibba025.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-10T12:01:05.766 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-10T12:01:05.767 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba017.front.sepia.ceph.com 2021-02-10T12:01:05.767 DEBUG:teuthology.orchestra.run.gibba017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T12:01:06.113 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@gibba025.front.sepia.ceph.com 2021-02-10T12:01:06.114 DEBUG:teuthology.orchestra.run.gibba025:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-10T12:01:06.455 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-10T12:01:06.455 DEBUG:teuthology.orchestra.run.gibba017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T12:01:06.459 DEBUG:teuthology.orchestra.run.gibba025:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-10T12:01:06.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-10T12:01:06.839 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-10T12:01:06.840 DEBUG:teuthology.orchestra.run.gibba017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T12:01:07.187 DEBUG:teuthology.orchestra.run.gibba025:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-10T12:01:07.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-10T12:01:07.541 DEBUG:teuthology.orchestra.run.gibba017:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T12:01:07.546 DEBUG:teuthology.orchestra.run.gibba025:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-10T12:01:07.886 INFO:teuthology.orchestra.run.gibba017.stdout:kernel.core_pattern = core 2021-02-10T12:01:07.889 INFO:teuthology.orchestra.run.gibba025.stdout:kernel.core_pattern = core 2021-02-10T12:01:07.890 DEBUG:teuthology.orchestra.run.gibba017:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T12:01:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T12:01:08.225 DEBUG:teuthology.orchestra.run.gibba025:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-10T12:01:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-10T12:01:08.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-10T12:01:08.574 INFO:teuthology.task.internal:Transferring archived files... 2021-02-10T12:01:08.575 DEBUG:teuthology.misc:Transferring archived files from gibba017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba017 2021-02-10T12:01:08.575 DEBUG:teuthology.orchestra.run.gibba017:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T12:01:08.930 DEBUG:teuthology.misc:Transferring archived files from gibba025:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/khiremat-2021-02-10_10:13:45-fs-wip-khiremat-mgr-authid-bp-7-nautilus-distro-basic-gibba/5872722/remote/gibba025 2021-02-10T12:01:08.931 DEBUG:teuthology.orchestra.run.gibba025:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-10T12:01:09.283 INFO:teuthology.task.internal:Removing archive directory... 2021-02-10T12:01:09.283 DEBUG:teuthology.orchestra.run.gibba017:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T12:01:09.287 DEBUG:teuthology.orchestra.run.gibba025:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-10T12:01:09.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-10T12:01:09.636 INFO:teuthology.task.internal:Not uploading archives. 2021-02-10T12:01:09.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-10T12:01:09.650 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-10T12:01:09.651 DEBUG:teuthology.orchestra.run.gibba017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T12:01:09.655 DEBUG:teuthology.orchestra.run.gibba025:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-10T12:01:09.986 INFO:teuthology.orchestra.run.gibba017.stdout:393221 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 12:01 /home/ubuntu/cephtest 2021-02-10T12:01:09.988 INFO:teuthology.orchestra.run.gibba025.stdout:393221 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 12:01 /home/ubuntu/cephtest 2021-02-10T12:01:09.989 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-10T12:01:10.041 INFO:teuthology.nuke:Checking targets against current locks 2021-02-10T12:01:10.133 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_14fad31725ce3e5fc27890f2a8ef1f21c838d296/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs